2024-05-27T07:02:20.523 INFO:root:teuthology version: 0.0.1.dev291+ge7ae758 2024-05-27T07:02:20.524 DEBUG:teuthology.run:Teuthology command: teuthology --owner scheduled_vshankar@teuthology --description fs/workload/{0-centos_9.stream begin/{0-install 1-cephadm 2-logrotate 3-modules} clusters/1a11s-mds-1c-client-3node conf/{client mds mgr mon osd} mount/kclient/{base/{mount-syntax/{v1} mount overrides/{distro/stock/{centos_9.stream k-stock} ms-die-on-skipped}} ms_mode/legacy wsync/yes} objectstore-ec/bluestore-comp-ec-root omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts pg_health session_timeout} ranks/multi/{balancer/random export-check n/5 replication/default} standby-replay tasks/{0-subvolume/{with-no-extra-options} 1-check-counter 2-scrub/yes 3-snaps/yes 4-flush/yes 5-quiesce/no 6-workunit/suites/blogbench}} --name vshankar-2024-05-27_05:52:49-fs-wip-vshankar-testing-20240521.063558-debug-testing-default-smithi --archive /home/teuthworker/archive/vshankar-2024-05-27_05:52:49-fs-wip-vshankar-testing-20240521.063558-debug-testing-default-smithi/7727648 --verbose -- /home/teuthworker/archive/vshankar-2024-05-27_05:52:49-fs-wip-vshankar-testing-20240521.063558-debug-testing-default-smithi/7727648/orig.config.yaml 2024-05-27T07:02:20.585 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-05-27T07:02:20.679 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/vshankar-2024-05-27_05:52:49-fs-wip-vshankar-testing-20240521.063558-debug-testing-default-smithi/7727648 branch: wip-vshankar-testing-20240521.063558-debug description: fs/workload/{0-centos_9.stream begin/{0-install 1-cephadm 2-logrotate 3-modules} clusters/1a11s-mds-1c-client-3node conf/{client mds mgr mon osd} mount/kclient/{base/{mount-syntax/{v1} mount overrides/{distro/stock/{centos_9.stream k-stock} ms-die-on-skipped}} ms_mode/legacy wsync/yes} objectstore-ec/bluestore-comp-ec-root omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts pg_health session_timeout} ranks/multi/{balancer/random export-check n/5 replication/default} standby-replay tasks/{0-subvolume/{with-no-extra-options} 1-check-counter 2-scrub/yes 3-snaps/yes 4-flush/yes 5-quiesce/no 6-workunit/suites/blogbench}} email: vshankar@redhat.com first_in_suite: false job_id: '7727648' kernel: client: sha1: distro flavor: default kdb: 1 sha1: b9459a26b0888785d05c47deb35abc9d0354ab93 ktype: distro last_in_suite: false log-rotate: ceph-mds: 10G ceph-osd: 10G machine_type: smithi mgrmodules: sequential: - print: Enabling mgr modules - exec: mon.a: - ceph mgr module enable snap_schedule name: vshankar-2024-05-27_05:52:49-fs-wip-vshankar-testing-20240521.063558-debug-testing-default-smithi no_nested_subset: false openstack: - volumes: count: 4 size: 20 - machine: disk: 200 os_type: centos os_version: 9.stream overrides: admin_socket: branch: wip-vshankar-testing-20240521.063558-debug ceph: cephfs: ec_profile: - m=2 - k=2 - crush-failure-domain=osd max_mds: 5 session_timeout: 300 standby_replay: true conf: client: client mount timeout: 600 debug client: 20 debug ms: 1 rados mon op timeout: 900 rados osd op timeout: 900 global: ms die on skipped message: false mds: debug mds: 20 debug mds balancer: 20 debug ms: 1 mds bal fragment size max: 10000 mds bal merge size: 5 mds bal split bits: 3 mds bal split size: 100 mds debug frag: true mds debug scatterstat: true mds op complaint time: 180 mds verify scatter: true mds_export_ephemeral_random_max: 0.1 osd op complaint time: 180 rados mon op timeout: 900 rados osd op timeout: 900 mgr: cephsqlite lock renewal timeout: 900000 debug cephsqlite: 20 debug client: 20 debug mgr: 20 debug ms: 1 mon: debug mon: 20 debug ms: 1 debug paxos: 20 mon down mkfs grace: 300 mon op complaint time: 120 osd: bluestore block size: 96636764160 bluestore compression mode: aggressive bluestore fsck on mount: true debug bluefs: 20 debug bluestore: 20 debug ms: 1 debug osd: 20 debug rocksdb: 10 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd failsafe full ratio: 0.95 osd objectstore: bluestore osd op complaint time: 180 osd shutdown pgref assert: true osd_max_omap_entries_per_request: 10000 flavor: default fs: xfs log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - FS_DEGRADED - FS_INLINE_DATA_DEPRECATED - FS_WITH_FAILED_MDS - MDS_ALL_DOWN - MDS_DAMAGE - MDS_DEGRADED - MDS_FAILED - MDS_INSUFFICIENT_STANDBY - MDS_UP_LESS_THAN_MAX - filesystem is online with fewer MDS than max_mds - POOL_APP_NOT_ENABLED - do not have an application enabled - overall HEALTH_ - Replacing daemon - deprecated feature inline_data - overall HEALTH_ - \(OSD_DOWN\) - \(OSD_ - but it is still running - is not responding - PG_AVAILABILITY - PG_DEGRADED - Reduced data availability - Degraded data redundancy - slow metadata IO - SLOW_OPS - slow request sha1: 38f9d900c83c63799fbdbe61acc9a11b0d3554a6 subvols: create: 2 ceph-deploy: conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: {} ceph-fuse: client.0: mount_subvol_num: 0 check-counter: counters: mds: - mds.exported - mds.imported - mds.root_rsnaps - mds_server.req_mksnap_latency.avgcount - mds_server.req_rmsnap_latency.avgcount - mds.dir_split install: ceph: flavor: default sha1: 38f9d900c83c63799fbdbe61acc9a11b0d3554a6 kclient: client.0: mount_subvol_num: 1 mntopts: - ms_mode=legacy - wsync syntax: v1 selinux: allowlist: - scontext=system_u:system_r:logrotate_t:s0 - scontext=system_u:system_r:getty_t:s0 - scontext=system_u:system_r:getty_t:s0 thrashosds: bdev_inject_crash: 2 bdev_inject_crash_probability: 0.5 workunit: branch: wip-vshankar-testing-20240521.063558-debug sha1: 38f9d900c83c63799fbdbe61acc9a11b0d3554a6 owner: scheduled_vshankar@teuthology priority: 70 repo: https://git.ceph.com/ceph-ci.git roles: - - mon.a - mgr.x - mds.a - mds.d - mds.g - mds.j - osd.0 - osd.3 - osd.6 - osd.9 - client.0 - - mon.b - mgr.y - mds.b - mds.e - mds.h - mds.k - osd.1 - osd.4 - osd.7 - osd.10 - - mon.c - mgr.z - mds.c - mds.f - mds.i - mds.l - osd.2 - osd.5 - osd.8 - osd.11 seed: 2499 sha1: 38f9d900c83c63799fbdbe61acc9a11b0d3554a6 sleep_before_teardown: 0 subset: 573/1024 suite: fs suite_branch: wip-vshankar-testing-20240521.063558-debug suite_path: /home/teuthworker/src/git.ceph.com_ceph-c_38f9d900c83c63799fbdbe61acc9a11b0d3554a6/qa suite_relpath: qa suite_repo: https://git.ceph.com/ceph-ci.git suite_sha1: 38f9d900c83c63799fbdbe61acc9a11b0d3554a6 syslog: ignorelist: - WARNING*.*check_session_state - WARNING*.*__ceph_remove_cap targets: smithi052.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBAwYV+HsmvDZFyrGU0K+XVvXx0ZqaDIKarF0CWUlAgQ5gFiHnbSsG4WdBlIBU5ghUvKNj0kXudJWcMRCyV3iNwE= smithi105.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBBFT68/ui6KNdp0onmMMWlIB1x4O0XG1q4jtNPAvwvoO5OnHIaCeBlfv3MZk+yMI2x9K4apqYQTf3M9EVRUscr0= smithi173.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBPV2SXnYCstrOFWhaTR9g5+11cRybptxIb3dkvR2yv74cQ7iz7lhSB0kvQTHFHLUUjdvhxKzIBM3TO0hRNXR0WI= tasks: - install: extra_packages: deb: - python3-cephfs - cephfs-shell - cephfs-top - cephfs-mirror rpm: - python3-cephfs - cephfs-shell - cephfs-top - cephfs-mirror extra_system_packages: deb: - bison - flex - libelf-dev - libssl-dev - network-manager - iproute2 - util-linux - dump - indent - libaio-dev - libtool-bin - uuid-dev - xfslibs-dev - postgresql - postgresql-client - postgresql-common - postgresql-contrib rpm: - bison - flex - elfutils-libelf-devel - openssl-devel - NetworkManager - iproute - util-linux - libacl-devel - libaio-devel - libattr-devel - libtool - libuuid-devel - xfsdump - xfsprogs - xfsprogs-devel - libaio-devel - libtool - libuuid-devel - xfsprogs-devel - postgresql - postgresql-server - postgresql-contrib - cephadm: roleless: false - cephadm.shell: mon.a: - ceph orch status - ceph orch ps - ceph orch ls - ceph orch host ls - ceph orch device ls - cephadm.shell: mon.a: - ceph fs dump - ceph osd dump - fs.ready: timeout: 300 - sequential: - mgrmodules - kclient: null - exec: mon.a: - ceph fs set cephfs balance_automate false - ceph fs subvolumegroup pin cephfs qa random 0.10 - check-counter: {} - fwd_scrub: scrub_timeout: 900 sleep_between_iterations: 1 - exec: mon.a: - ceph config set mgr mgr/snap_schedule/log_level debug - ceph config set mgr mgr/snap_schedule/allow_m_granularity true - ceph config set mgr mgr/snap_schedule/dump_on_update true - ceph fs snap-schedule add --fs=cephfs --path=/ --snap_schedule=1m - ceph fs snap-schedule retention add --fs=cephfs --path=/ --retention-spec-or-period=6m3h - ceph fs snap-schedule status --fs=cephfs --path=/ - ceph fs snap-schedule list --fs=cephfs --path=/ --recursive=true - date +%s > START_TIME - full_sequential_finally: - exec: mon.a: - date +%s > END_TIME - START_TIME=$(cat START_TIME); END_TIME=$(cat END_TIME); DIFF_TIME=$((600-(END_TIME-START_TIME))); if [ "$DIFF_TIME" -gt 0 ]; then sleep "$DIFF_TIME"; fi - ceph fs snap-schedule status --fs=cephfs --path=/ - ceph fs snap-schedule list --fs=cephfs --path=/ --recursive=true - background_exec: mon.a: - while sleep 13; do ceph tell mds.cephfs:0 flush journal; done - workunit: clients: all: - suites/blogbench.sh teuthology: fragments_dropped: [] meta: {} postmerge: - local function is_kupstream() return yaml.ktype == 'upstream' end - local function is_kdistro() return yaml.ktype == 'distro' end - local function is_fuse() return false end - local function syntax_version() return yaml.overrides.kclient.syntax end teuthology_branch: main teuthology_sha1: e7ae7580c417d6ab27fda98fecadca5d9f85d63e timestamp: 2024-05-27_05:52:49 tube: smithi user: vshankar verbose: false worker_log: /home/teuthworker/archive/worker_logs/dispatcher.smithi.2698841 2024-05-27T07:02:20.679 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/git.ceph.com_ceph-c_38f9d900c83c63799fbdbe61acc9a11b0d3554a6/qa; will attempt to use it 2024-05-27T07:02:20.680 INFO:teuthology.run:Found tasks at /home/teuthworker/src/git.ceph.com_ceph-c_38f9d900c83c63799fbdbe61acc9a11b0d3554a6/qa/tasks 2024-05-27T07:02:20.680 INFO:teuthology.run_tasks:Running task internal.check_packages... 2024-05-27T07:02:20.682 INFO:teuthology.task.internal:Checking packages... 2024-05-27T07:02:20.704 INFO:teuthology.task.internal:Checking packages for os_type 'centos', flavor 'default' and ceph hash '38f9d900c83c63799fbdbe61acc9a11b0d3554a6' 2024-05-27T07:02:20.704 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-05-27T07:02:20.704 INFO:teuthology.packaging:ref: None 2024-05-27T07:02:20.704 INFO:teuthology.packaging:tag: None 2024-05-27T07:02:20.704 INFO:teuthology.packaging:branch: wip-vshankar-testing-20240521.063558-debug 2024-05-27T07:02:20.704 INFO:teuthology.packaging:sha1: 38f9d900c83c63799fbdbe61acc9a11b0d3554a6 2024-05-27T07:02:20.704 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&ref=wip-vshankar-testing-20240521.063558-debug 2024-05-27T07:02:20.976 INFO:teuthology.task.internal:Found packages for ceph version 19.0.0-3872.g38f9d900 2024-05-27T07:02:20.979 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2024-05-27T07:02:20.985 INFO:teuthology.task.internal:no buildpackages task found 2024-05-27T07:02:20.986 INFO:teuthology.run_tasks:Running task internal.save_config... 2024-05-27T07:02:21.020 INFO:teuthology.task.internal:Saving configuration 2024-05-27T07:02:21.042 INFO:teuthology.run_tasks:Running task internal.check_lock... 2024-05-27T07:02:21.049 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-05-27T07:02:21.074 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi052.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/vshankar-2024-05-27_05:52:49-fs-wip-vshankar-testing-20240521.063558-debug-testing-default-smithi/7727648', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-05-27 06:57:20.536741', 'locked_by': 'scheduled_vshankar@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBAwYV+HsmvDZFyrGU0K+XVvXx0ZqaDIKarF0CWUlAgQ5gFiHnbSsG4WdBlIBU5ghUvKNj0kXudJWcMRCyV3iNwE='} 2024-05-27T07:02:21.098 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi105.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/vshankar-2024-05-27_05:52:49-fs-wip-vshankar-testing-20240521.063558-debug-testing-default-smithi/7727648', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-05-27 06:57:20.534771', 'locked_by': 'scheduled_vshankar@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBBFT68/ui6KNdp0onmMMWlIB1x4O0XG1q4jtNPAvwvoO5OnHIaCeBlfv3MZk+yMI2x9K4apqYQTf3M9EVRUscr0='} 2024-05-27T07:02:21.120 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi173.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/vshankar-2024-05-27_05:52:49-fs-wip-vshankar-testing-20240521.063558-debug-testing-default-smithi/7727648', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-05-27 06:57:20.538384', 'locked_by': 'scheduled_vshankar@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBPV2SXnYCstrOFWhaTR9g5+11cRybptxIb3dkvR2yv74cQ7iz7lhSB0kvQTHFHLUUjdvhxKzIBM3TO0hRNXR0WI='} 2024-05-27T07:02:21.121 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2024-05-27T07:02:21.127 INFO:teuthology.task.internal:roles: ubuntu@smithi052.front.sepia.ceph.com - ['mon.a', 'mgr.x', 'mds.a', 'mds.d', 'mds.g', 'mds.j', 'osd.0', 'osd.3', 'osd.6', 'osd.9', 'client.0'] 2024-05-27T07:02:21.127 INFO:teuthology.task.internal:roles: ubuntu@smithi105.front.sepia.ceph.com - ['mon.b', 'mgr.y', 'mds.b', 'mds.e', 'mds.h', 'mds.k', 'osd.1', 'osd.4', 'osd.7', 'osd.10'] 2024-05-27T07:02:21.127 INFO:teuthology.task.internal:roles: ubuntu@smithi173.front.sepia.ceph.com - ['mon.c', 'mgr.z', 'mds.c', 'mds.f', 'mds.i', 'mds.l', 'osd.2', 'osd.5', 'osd.8', 'osd.11'] 2024-05-27T07:02:21.127 INFO:teuthology.run_tasks:Running task console_log... 2024-05-27T07:02:21.203 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7f70ef213ac0>, signals=[15]) 2024-05-27T07:02:21.203 INFO:teuthology.run_tasks:Running task internal.connect... 2024-05-27T07:02:21.209 INFO:teuthology.task.internal:Opening connections... 2024-05-27T07:02:21.210 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi052.front.sepia.ceph.com 2024-05-27T07:02:21.211 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi052.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-05-27T07:02:21.287 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi105.front.sepia.ceph.com 2024-05-27T07:02:21.288 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi105.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-05-27T07:02:21.402 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi173.front.sepia.ceph.com 2024-05-27T07:02:21.403 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi173.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-05-27T07:02:21.479 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2024-05-27T07:02:21.486 DEBUG:teuthology.orchestra.run.smithi052:> uname -m 2024-05-27T07:02:21.505 INFO:teuthology.orchestra.run.smithi052.stdout:x86_64 2024-05-27T07:02:21.505 DEBUG:teuthology.orchestra.run.smithi052:> cat /etc/os-release 2024-05-27T07:02:21.563 INFO:teuthology.orchestra.run.smithi052.stdout:NAME="CentOS Stream" 2024-05-27T07:02:21.563 INFO:teuthology.orchestra.run.smithi052.stdout:VERSION="9" 2024-05-27T07:02:21.563 INFO:teuthology.orchestra.run.smithi052.stdout:ID="centos" 2024-05-27T07:02:21.563 INFO:teuthology.orchestra.run.smithi052.stdout:ID_LIKE="rhel fedora" 2024-05-27T07:02:21.563 INFO:teuthology.orchestra.run.smithi052.stdout:VERSION_ID="9" 2024-05-27T07:02:21.563 INFO:teuthology.orchestra.run.smithi052.stdout:PLATFORM_ID="platform:el9" 2024-05-27T07:02:21.564 INFO:teuthology.orchestra.run.smithi052.stdout:PRETTY_NAME="CentOS Stream 9" 2024-05-27T07:02:21.564 INFO:teuthology.orchestra.run.smithi052.stdout:ANSI_COLOR="0;31" 2024-05-27T07:02:21.564 INFO:teuthology.orchestra.run.smithi052.stdout:LOGO="fedora-logo-icon" 2024-05-27T07:02:21.564 INFO:teuthology.orchestra.run.smithi052.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-05-27T07:02:21.564 INFO:teuthology.orchestra.run.smithi052.stdout:HOME_URL="https://centos.org/" 2024-05-27T07:02:21.564 INFO:teuthology.orchestra.run.smithi052.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-05-27T07:02:21.564 INFO:teuthology.orchestra.run.smithi052.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-05-27T07:02:21.564 INFO:teuthology.orchestra.run.smithi052.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-05-27T07:02:21.565 INFO:teuthology.lock.ops:Updating smithi052.front.sepia.ceph.com on lock server 2024-05-27T07:02:21.584 DEBUG:teuthology.orchestra.run.smithi105:> uname -m 2024-05-27T07:02:21.601 INFO:teuthology.orchestra.run.smithi105.stdout:x86_64 2024-05-27T07:02:21.602 DEBUG:teuthology.orchestra.run.smithi105:> cat /etc/os-release 2024-05-27T07:02:21.658 INFO:teuthology.orchestra.run.smithi105.stdout:NAME="CentOS Stream" 2024-05-27T07:02:21.658 INFO:teuthology.orchestra.run.smithi105.stdout:VERSION="9" 2024-05-27T07:02:21.658 INFO:teuthology.orchestra.run.smithi105.stdout:ID="centos" 2024-05-27T07:02:21.658 INFO:teuthology.orchestra.run.smithi105.stdout:ID_LIKE="rhel fedora" 2024-05-27T07:02:21.658 INFO:teuthology.orchestra.run.smithi105.stdout:VERSION_ID="9" 2024-05-27T07:02:21.658 INFO:teuthology.orchestra.run.smithi105.stdout:PLATFORM_ID="platform:el9" 2024-05-27T07:02:21.659 INFO:teuthology.orchestra.run.smithi105.stdout:PRETTY_NAME="CentOS Stream 9" 2024-05-27T07:02:21.659 INFO:teuthology.orchestra.run.smithi105.stdout:ANSI_COLOR="0;31" 2024-05-27T07:02:21.659 INFO:teuthology.orchestra.run.smithi105.stdout:LOGO="fedora-logo-icon" 2024-05-27T07:02:21.659 INFO:teuthology.orchestra.run.smithi105.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-05-27T07:02:21.659 INFO:teuthology.orchestra.run.smithi105.stdout:HOME_URL="https://centos.org/" 2024-05-27T07:02:21.659 INFO:teuthology.orchestra.run.smithi105.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-05-27T07:02:21.659 INFO:teuthology.orchestra.run.smithi105.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-05-27T07:02:21.659 INFO:teuthology.orchestra.run.smithi105.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-05-27T07:02:21.659 INFO:teuthology.lock.ops:Updating smithi105.front.sepia.ceph.com on lock server 2024-05-27T07:02:21.684 DEBUG:teuthology.orchestra.run.smithi173:> uname -m 2024-05-27T07:02:21.702 INFO:teuthology.orchestra.run.smithi173.stdout:x86_64 2024-05-27T07:02:21.706 DEBUG:teuthology.orchestra.run.smithi173:> cat /etc/os-release 2024-05-27T07:02:21.759 INFO:teuthology.orchestra.run.smithi173.stdout:NAME="CentOS Stream" 2024-05-27T07:02:21.759 INFO:teuthology.orchestra.run.smithi173.stdout:VERSION="9" 2024-05-27T07:02:21.759 INFO:teuthology.orchestra.run.smithi173.stdout:ID="centos" 2024-05-27T07:02:21.759 INFO:teuthology.orchestra.run.smithi173.stdout:ID_LIKE="rhel fedora" 2024-05-27T07:02:21.759 INFO:teuthology.orchestra.run.smithi173.stdout:VERSION_ID="9" 2024-05-27T07:02:21.760 INFO:teuthology.orchestra.run.smithi173.stdout:PLATFORM_ID="platform:el9" 2024-05-27T07:02:21.760 INFO:teuthology.orchestra.run.smithi173.stdout:PRETTY_NAME="CentOS Stream 9" 2024-05-27T07:02:21.760 INFO:teuthology.orchestra.run.smithi173.stdout:ANSI_COLOR="0;31" 2024-05-27T07:02:21.760 INFO:teuthology.orchestra.run.smithi173.stdout:LOGO="fedora-logo-icon" 2024-05-27T07:02:21.760 INFO:teuthology.orchestra.run.smithi173.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-05-27T07:02:21.760 INFO:teuthology.orchestra.run.smithi173.stdout:HOME_URL="https://centos.org/" 2024-05-27T07:02:21.760 INFO:teuthology.orchestra.run.smithi173.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-05-27T07:02:21.760 INFO:teuthology.orchestra.run.smithi173.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-05-27T07:02:21.760 INFO:teuthology.orchestra.run.smithi173.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-05-27T07:02:21.760 INFO:teuthology.lock.ops:Updating smithi173.front.sepia.ceph.com on lock server 2024-05-27T07:02:21.783 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2024-05-27T07:02:21.794 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2024-05-27T07:02:21.801 INFO:teuthology.task.internal:Checking for old test directory... 2024-05-27T07:02:21.801 DEBUG:teuthology.orchestra.run.smithi052:> test '!' -e /home/ubuntu/cephtest 2024-05-27T07:02:21.804 DEBUG:teuthology.orchestra.run.smithi105:> test '!' -e /home/ubuntu/cephtest 2024-05-27T07:02:21.806 DEBUG:teuthology.orchestra.run.smithi173:> test '!' -e /home/ubuntu/cephtest 2024-05-27T07:02:21.822 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2024-05-27T07:02:21.828 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2024-05-27T07:02:21.829 DEBUG:teuthology.orchestra.run.smithi052:> test -z $(ls -A /var/lib/ceph) 2024-05-27T07:02:21.861 DEBUG:teuthology.orchestra.run.smithi105:> test -z $(ls -A /var/lib/ceph) 2024-05-27T07:02:21.863 DEBUG:teuthology.orchestra.run.smithi173:> test -z $(ls -A /var/lib/ceph) 2024-05-27T07:02:21.971 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2024-05-27T07:02:22.051 INFO:teuthology.run_tasks:Running task kernel... 2024-05-27T07:02:22.066 INFO:teuthology.task.kernel:normalize config orig: {'client': {'sha1': 'distro'}, 'flavor': 'default', 'kdb': 1, 'sha1': 'b9459a26b0888785d05c47deb35abc9d0354ab93'} 2024-05-27T07:02:22.066 INFO:teuthology.task.kernel:normalize config final: {'client.0': {'sha1': 'distro'}} 2024-05-27T07:02:22.066 INFO:teuthology.task.kernel:config {'client.0': {'sha1': 'distro'}}, timeout 300 2024-05-27T07:02:22.066 DEBUG:teuthology.orchestra.run.smithi052:> test -f /run/.containerenv -o -f /.dockerenv 2024-05-27T07:02:22.083 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-05-27T07:02:22.084 DEBUG:teuthology.orchestra.run.smithi052:> uname -r 2024-05-27T07:02:22.142 INFO:teuthology.orchestra.run.smithi052.stdout:5.14.0-437.el9.x86_64 2024-05-27T07:02:22.143 INFO:teuthology.task.kernel:Running kernel on smithi052: 5.14.0-437.el9.x86_64 2024-05-27T07:02:22.143 DEBUG:teuthology.orchestra.run.smithi052:> sudo yum install -y kernel 2024-05-27T07:02:24.472 INFO:teuthology.orchestra.run.smithi052.stdout:CentOS Stream 9 - BaseOS 11 MB/s | 8.1 MB 00:00 2024-05-27T07:02:27.539 INFO:teuthology.orchestra.run.smithi052.stdout:CentOS Stream 9 - AppStream 9.8 MB/s | 19 MB 00:01 2024-05-27T07:02:33.856 INFO:teuthology.orchestra.run.smithi052.stdout:CentOS Stream 9 - CRB 2.4 MB/s | 6.2 MB 00:02 2024-05-27T07:02:35.323 INFO:teuthology.orchestra.run.smithi052.stdout:CentOS Stream 9 - Extras packages 64 kB/s | 17 kB 00:00 2024-05-27T07:02:39.344 INFO:teuthology.orchestra.run.smithi052.stdout:Extra Packages for Enterprise Linux 5.4 MB/s | 21 MB 00:03 2024-05-27T07:02:44.700 INFO:teuthology.orchestra.run.smithi052.stdout:lab-extras 33 kB/s | 1.7 kB 00:00 2024-05-27T07:02:46.689 INFO:teuthology.orchestra.run.smithi052.stdout:Package kernel-5.14.0-239.el9.x86_64 is already installed. 2024-05-27T07:02:46.689 INFO:teuthology.orchestra.run.smithi052.stdout:Package kernel-5.14.0-437.el9.x86_64 is already installed. 2024-05-27T07:02:46.787 INFO:teuthology.orchestra.run.smithi052.stdout:Dependencies resolved. 2024-05-27T07:02:46.807 INFO:teuthology.orchestra.run.smithi052.stdout:================================================================================ 2024-05-27T07:02:46.807 INFO:teuthology.orchestra.run.smithi052.stdout: Package Architecture Version Repository Size 2024-05-27T07:02:46.807 INFO:teuthology.orchestra.run.smithi052.stdout:================================================================================ 2024-05-27T07:02:46.807 INFO:teuthology.orchestra.run.smithi052.stdout:Installing: 2024-05-27T07:02:46.807 INFO:teuthology.orchestra.run.smithi052.stdout: kernel x86_64 5.14.0-447.el9 baseos 576 k 2024-05-27T07:02:46.807 INFO:teuthology.orchestra.run.smithi052.stdout:Installing dependencies: 2024-05-27T07:02:46.808 INFO:teuthology.orchestra.run.smithi052.stdout: kernel-core x86_64 5.14.0-447.el9 baseos 16 M 2024-05-27T07:02:46.808 INFO:teuthology.orchestra.run.smithi052.stdout: kernel-modules x86_64 5.14.0-447.el9 baseos 34 M 2024-05-27T07:02:46.808 INFO:teuthology.orchestra.run.smithi052.stdout: kernel-modules-core x86_64 5.14.0-447.el9 baseos 28 M 2024-05-27T07:02:46.808 INFO:teuthology.orchestra.run.smithi052.stdout: 2024-05-27T07:02:46.808 INFO:teuthology.orchestra.run.smithi052.stdout:Transaction Summary 2024-05-27T07:02:46.808 INFO:teuthology.orchestra.run.smithi052.stdout:================================================================================ 2024-05-27T07:02:46.808 INFO:teuthology.orchestra.run.smithi052.stdout:Install 4 Packages 2024-05-27T07:02:46.808 INFO:teuthology.orchestra.run.smithi052.stdout: 2024-05-27T07:02:46.808 INFO:teuthology.orchestra.run.smithi052.stdout:Total download size: 78 M 2024-05-27T07:02:46.808 INFO:teuthology.orchestra.run.smithi052.stdout:Installed size: 123 M 2024-05-27T07:02:46.808 INFO:teuthology.orchestra.run.smithi052.stdout:Downloading Packages: 2024-05-27T07:02:47.076 INFO:teuthology.orchestra.run.smithi052.stdout:(1/4): kernel-5.14.0-447.el9.x86_64.rpm 3.7 MB/s | 576 kB 00:00 2024-05-27T07:02:47.335 INFO:teuthology.orchestra.run.smithi052.stdout:(2/4): kernel-core-5.14.0-447.el9.x86_64.rpm 38 MB/s | 16 MB 00:00 2024-05-27T07:02:47.760 INFO:teuthology.orchestra.run.smithi052.stdout:(3/4): kernel-modules-5.14.0-447.el9.x86_64.rpm 41 MB/s | 34 MB 00:00 2024-05-27T07:02:48.019 INFO:teuthology.orchestra.run.smithi052.stdout:(4/4): kernel-modules-core-5.14.0-447.el9.x86_6 30 MB/s | 28 MB 00:00 2024-05-27T07:02:48.019 INFO:teuthology.orchestra.run.smithi052.stdout:-------------------------------------------------------------------------------- 2024-05-27T07:02:48.019 INFO:teuthology.orchestra.run.smithi052.stdout:Total 65 MB/s | 78 MB 00:01 2024-05-27T07:02:48.731 INFO:teuthology.orchestra.run.smithi052.stdout:Running transaction check 2024-05-27T07:02:49.134 INFO:teuthology.orchestra.run.smithi052.stdout:Transaction check succeeded. 2024-05-27T07:02:49.134 INFO:teuthology.orchestra.run.smithi052.stdout:Running transaction test 2024-05-27T07:02:50.085 INFO:teuthology.orchestra.run.smithi052.stdout:Transaction test succeeded. 2024-05-27T07:02:50.085 INFO:teuthology.orchestra.run.smithi052.stdout:Running transaction 2024-05-27T07:02:52.589 INFO:teuthology.orchestra.run.smithi052.stdout: Preparing : 1/1 2024-05-27T07:02:53.237 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : kernel-modules-core-5.14.0-447.el9.x86_64 1/4 2024-05-27T07:02:53.327 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : kernel-core-5.14.0-447.el9.x86_64 2/4 2024-05-27T07:02:56.116 INFO:teuthology.orchestra.run.smithi052.stdout: Running scriptlet: kernel-core-5.14.0-447.el9.x86_64 2/4 2024-05-27T07:02:56.239 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : kernel-modules-5.14.0-447.el9.x86_64 3/4 2024-05-27T07:03:01.324 INFO:teuthology.orchestra.run.smithi052.stdout: Running scriptlet: kernel-modules-5.14.0-447.el9.x86_64 3/4 2024-05-27T07:03:01.368 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : kernel-5.14.0-447.el9.x86_64 4/4 2024-05-27T07:03:06.470 INFO:teuthology.orchestra.run.smithi052.stdout: Running scriptlet: kernel-modules-core-5.14.0-447.el9.x86_64 4/4 2024-05-27T07:03:37.455 INFO:teuthology.orchestra.run.smithi052.stdout: Running scriptlet: kernel-core-5.14.0-447.el9.x86_64 4/4 2024-05-27T07:03:37.494 INFO:teuthology.orchestra.run.smithi052.stdout: Running scriptlet: kernel-modules-5.14.0-447.el9.x86_64 4/4 2024-05-27T07:03:37.841 INFO:teuthology.orchestra.run.smithi052.stdout: Running scriptlet: kernel-5.14.0-447.el9.x86_64 4/4 2024-05-27T07:03:37.841 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : kernel-5.14.0-447.el9.x86_64 1/4 2024-05-27T07:03:37.841 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : kernel-core-5.14.0-447.el9.x86_64 2/4 2024-05-27T07:03:37.841 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : kernel-modules-5.14.0-447.el9.x86_64 3/4 2024-05-27T07:03:38.643 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : kernel-modules-core-5.14.0-447.el9.x86_64 4/4 2024-05-27T07:03:38.643 INFO:teuthology.orchestra.run.smithi052.stdout: 2024-05-27T07:03:38.643 INFO:teuthology.orchestra.run.smithi052.stdout:Installed: 2024-05-27T07:03:38.643 INFO:teuthology.orchestra.run.smithi052.stdout: kernel-5.14.0-447.el9.x86_64 2024-05-27T07:03:38.643 INFO:teuthology.orchestra.run.smithi052.stdout: kernel-core-5.14.0-447.el9.x86_64 2024-05-27T07:03:38.643 INFO:teuthology.orchestra.run.smithi052.stdout: kernel-modules-5.14.0-447.el9.x86_64 2024-05-27T07:03:38.643 INFO:teuthology.orchestra.run.smithi052.stdout: kernel-modules-core-5.14.0-447.el9.x86_64 2024-05-27T07:03:38.644 INFO:teuthology.orchestra.run.smithi052.stdout: 2024-05-27T07:03:38.644 INFO:teuthology.orchestra.run.smithi052.stdout:Complete! 2024-05-27T07:03:38.969 DEBUG:teuthology.orchestra.run.smithi052:> rpm -q kernel | sort -rV | head -n 1 2024-05-27T07:03:39.084 INFO:teuthology.orchestra.run.smithi052.stdout:kernel-5.14.0-447.el9.x86_64 2024-05-27T07:03:39.084 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-447.el9.x86_64 2024-05-27T07:03:39.084 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.14.0-437.el9.x86_64 Expected: 5.14.0-447.el9.x86_64 2024-05-27T07:03:39.084 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-05-27T07:03:39.084 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-05-27T07:03:39.085 INFO:teuthology.task.kernel:Installing distro kernel on client.0... 2024-05-27T07:03:39.085 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi052.front.sepia.ceph.com, path=None, version=distro) 2024-05-27T07:03:39.085 DEBUG:teuthology.orchestra.run.smithi052:> rpm -q kernel | sort -rV | head -n 1 2024-05-27T07:03:39.135 INFO:teuthology.orchestra.run.smithi052.stdout:kernel-5.14.0-447.el9.x86_64 2024-05-27T07:03:39.136 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-447.el9.x86_64 2024-05-27T07:03:39.137 DEBUG:teuthology.orchestra.run.smithi052:> sudo rpm -qi grub2-tools 2024-05-27T07:03:39.170 INFO:teuthology.orchestra.run.smithi052.stdout:Name : grub2-tools 2024-05-27T07:03:39.170 INFO:teuthology.orchestra.run.smithi052.stdout:Epoch : 1 2024-05-27T07:03:39.170 INFO:teuthology.orchestra.run.smithi052.stdout:Version : 2.06 2024-05-27T07:03:39.170 INFO:teuthology.orchestra.run.smithi052.stdout:Release : 68.el9 2024-05-27T07:03:39.170 INFO:teuthology.orchestra.run.smithi052.stdout:Architecture: x86_64 2024-05-27T07:03:39.171 INFO:teuthology.orchestra.run.smithi052.stdout:Install Date: Fri 19 Apr 2024 09:32:39 PM UTC 2024-05-27T07:03:39.171 INFO:teuthology.orchestra.run.smithi052.stdout:Group : Unspecified 2024-05-27T07:03:39.171 INFO:teuthology.orchestra.run.smithi052.stdout:Size : 8264254 2024-05-27T07:03:39.171 INFO:teuthology.orchestra.run.smithi052.stdout:License : GPLv3+ 2024-05-27T07:03:39.171 INFO:teuthology.orchestra.run.smithi052.stdout:Signature : RSA/SHA256, Mon 07 Aug 2023 01:57:16 PM UTC, Key ID 05b555b38483c65d 2024-05-27T07:03:39.171 INFO:teuthology.orchestra.run.smithi052.stdout:Source RPM : grub2-2.06-68.el9.src.rpm 2024-05-27T07:03:39.171 INFO:teuthology.orchestra.run.smithi052.stdout:Build Date : Wed 02 Aug 2023 03:15:48 PM UTC 2024-05-27T07:03:39.171 INFO:teuthology.orchestra.run.smithi052.stdout:Build Host : x86-05.stream.rdu2.redhat.com 2024-05-27T07:03:39.171 INFO:teuthology.orchestra.run.smithi052.stdout:Packager : builder@centos.org 2024-05-27T07:03:39.171 INFO:teuthology.orchestra.run.smithi052.stdout:Vendor : CentOS 2024-05-27T07:03:39.171 INFO:teuthology.orchestra.run.smithi052.stdout:URL : http://www.gnu.org/software/grub/ 2024-05-27T07:03:39.171 INFO:teuthology.orchestra.run.smithi052.stdout:Summary : Support tools for GRUB. 2024-05-27T07:03:39.171 INFO:teuthology.orchestra.run.smithi052.stdout:Description : 2024-05-27T07:03:39.172 INFO:teuthology.orchestra.run.smithi052.stdout: 2024-05-27T07:03:39.172 INFO:teuthology.orchestra.run.smithi052.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2024-05-27T07:03:39.172 INFO:teuthology.orchestra.run.smithi052.stdout:customizable bootloader with modular architecture. It supports a rich 2024-05-27T07:03:39.172 INFO:teuthology.orchestra.run.smithi052.stdout:variety of kernel formats, file systems, computer architectures and 2024-05-27T07:03:39.172 INFO:teuthology.orchestra.run.smithi052.stdout:hardware devices. 2024-05-27T07:03:39.172 INFO:teuthology.orchestra.run.smithi052.stdout: 2024-05-27T07:03:39.172 INFO:teuthology.orchestra.run.smithi052.stdout:This subpackage provides tools for support of all platforms. 2024-05-27T07:03:39.173 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2024-05-27T07:03:39.173 INFO:teuthology.task.kernel:Updating grub on smithi052 to boot 5.14.0-447.el9.x86_64 2024-05-27T07:03:39.173 DEBUG:teuthology.orchestra.run.smithi052:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2024-05-27T07:03:39.813 INFO:teuthology.orchestra.run.smithi052.stderr:Generating grub configuration file ... 2024-05-27T07:03:41.342 INFO:teuthology.orchestra.run.smithi052.stderr:Adding boot menu entry for UEFI Firmware Settings ... 2024-05-27T07:03:41.371 INFO:teuthology.orchestra.run.smithi052.stderr:done 2024-05-27T07:03:41.373 DEBUG:teuthology.orchestra.run.smithi052:> mktemp 2024-05-27T07:03:41.388 INFO:teuthology.orchestra.run.smithi052.stdout:/tmp/tmp.ybDVHJJOXJ 2024-05-27T07:03:41.388 DEBUG:teuthology.orchestra.run.smithi052:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.ybDVHJJOXJ 2024-05-27T07:03:41.453 DEBUG:teuthology.orchestra.run.smithi052:> sudo chmod 0666 /tmp/tmp.ybDVHJJOXJ 2024-05-27T07:03:41.590 DEBUG:teuthology.orchestra.remote:smithi052:/tmp/tmp.ybDVHJJOXJ is 6KB 2024-05-27T07:03:41.638 DEBUG:teuthology.orchestra.run.smithi052:> rm -fr /tmp/tmp.ybDVHJJOXJ 2024-05-27T07:03:41.652 DEBUG:teuthology.orchestra.run.smithi052:> sudo /bin/ls /boot/loader/entries || true 2024-05-27T07:03:41.715 INFO:teuthology.orchestra.run.smithi052.stdout:7fc8828e06d0482694bcf4a447d13cb6-0-rescue.conf 2024-05-27T07:03:41.715 INFO:teuthology.orchestra.run.smithi052.stdout:7fc8828e06d0482694bcf4a447d13cb6-5.14.0-239.el9.x86_64.conf 2024-05-27T07:03:41.715 INFO:teuthology.orchestra.run.smithi052.stdout:7fc8828e06d0482694bcf4a447d13cb6-5.14.0-437.el9.x86_64.conf 2024-05-27T07:03:41.715 INFO:teuthology.orchestra.run.smithi052.stdout:7fc8828e06d0482694bcf4a447d13cb6-5.14.0-447.el9.x86_64.conf 2024-05-27T07:03:41.716 DEBUG:teuthology.orchestra.run.smithi052:> sudo grub2-set-default 7fc8828e06d0482694bcf4a447d13cb6-5.14.0-447.el9.x86_64 2024-05-27T07:03:41.872 DEBUG:teuthology.orchestra.run.smithi052:> sudo shutdown -r now 2024-05-27T07:04:11.875 INFO:teuthology.task.kernel:Checking client client.0 for new kernel version... 2024-05-27T07:04:11.876 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi052.front.sepia.ceph.com' 2024-05-27T07:04:11.877 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi052.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-05-27T07:05:11.879 DEBUG:teuthology.orchestra.remote:timed out 2024-05-27T07:05:20.880 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi052.front.sepia.ceph.com' 2024-05-27T07:05:20.881 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi052.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-05-27T07:05:21.223 DEBUG:teuthology.orchestra.run.smithi052:> true 2024-05-27T07:05:21.615 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi052.front.sepia.ceph.com' 2024-05-27T07:05:21.615 INFO:teuthology.task.kernel:Checking kernel version of client.0, want "5.14.0-447.el9.x86_64"... 2024-05-27T07:05:21.616 DEBUG:teuthology.orchestra.run.smithi052:> uname -r 2024-05-27T07:05:21.631 INFO:teuthology.orchestra.run.smithi052.stdout:5.14.0-447.el9.x86_64 2024-05-27T07:05:21.631 DEBUG:teuthology.task.kernel:current kernel version is 5.14.0-447.el9.x86_64 vs 5.14.0-447.el9.x86_64 2024-05-27T07:05:21.631 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-05-27T07:05:21.631 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2024-05-27T07:05:22.632 DEBUG:teuthology.parallel:result is None 2024-05-27T07:05:22.633 INFO:teuthology.run_tasks:Running task internal.base... 2024-05-27T07:05:22.642 INFO:teuthology.task.internal:Creating test directory... 2024-05-27T07:05:22.642 DEBUG:teuthology.orchestra.run.smithi052:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-05-27T07:05:22.644 DEBUG:teuthology.orchestra.run.smithi105:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-05-27T07:05:22.647 DEBUG:teuthology.orchestra.run.smithi173:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-05-27T07:05:22.665 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2024-05-27T07:05:22.736 INFO:teuthology.run_tasks:Running task internal.archive... 2024-05-27T07:05:22.750 INFO:teuthology.task.internal:Creating archive directory... 2024-05-27T07:05:22.750 DEBUG:teuthology.orchestra.run.smithi052:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-05-27T07:05:22.753 DEBUG:teuthology.orchestra.run.smithi105:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-05-27T07:05:22.755 DEBUG:teuthology.orchestra.run.smithi173:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-05-27T07:05:22.868 INFO:teuthology.run_tasks:Running task internal.coredump... 2024-05-27T07:05:22.877 INFO:teuthology.task.internal:Enabling coredump saving... 2024-05-27T07:05:22.877 DEBUG:teuthology.orchestra.run.smithi105:> test -f /run/.containerenv -o -f /.dockerenv 2024-05-27T07:05:22.893 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-05-27T07:05:22.894 DEBUG:teuthology.orchestra.run.smithi173:> test -f /run/.containerenv -o -f /.dockerenv 2024-05-27T07:05:22.910 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-05-27T07:05:22.911 DEBUG:teuthology.orchestra.run.smithi052:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-05-27T07:05:22.913 DEBUG:teuthology.orchestra.run.smithi105:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-05-27T07:05:22.937 DEBUG:teuthology.orchestra.run.smithi173:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-05-27T07:05:22.969 INFO:teuthology.orchestra.run.smithi105.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-05-27T07:05:22.984 INFO:teuthology.orchestra.run.smithi052.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-05-27T07:05:22.999 INFO:teuthology.orchestra.run.smithi173.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-05-27T07:05:23.006 INFO:teuthology.orchestra.run.smithi105.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-05-27T07:05:23.011 INFO:teuthology.orchestra.run.smithi052.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-05-27T07:05:23.018 INFO:teuthology.orchestra.run.smithi173.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-05-27T07:05:23.020 INFO:teuthology.run_tasks:Running task internal.sudo... 2024-05-27T07:05:23.032 INFO:teuthology.task.internal:Configuring sudo... 2024-05-27T07:05:23.032 DEBUG:teuthology.orchestra.run.smithi052:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-05-27T07:05:23.034 DEBUG:teuthology.orchestra.run.smithi105:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-05-27T07:05:23.050 DEBUG:teuthology.orchestra.run.smithi173:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-05-27T07:05:23.077 INFO:teuthology.run_tasks:Running task internal.syslog... 2024-05-27T07:05:23.086 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2024-05-27T07:05:23.086 DEBUG:teuthology.orchestra.run.smithi052:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-05-27T07:05:23.100 DEBUG:teuthology.orchestra.run.smithi105:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-05-27T07:05:23.116 DEBUG:teuthology.orchestra.run.smithi173:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-05-27T07:05:23.132 DEBUG:teuthology.orchestra.run.smithi052:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-05-27T07:05:23.195 DEBUG:teuthology.orchestra.run.smithi052:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-05-27T07:05:23.241 DEBUG:teuthology.orchestra.run.smithi052:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-05-27T07:05:23.323 DEBUG:teuthology.orchestra.run.smithi052:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-05-27T07:05:23.362 DEBUG:teuthology.orchestra.run.smithi052:> set -ex 2024-05-27T07:05:23.362 DEBUG:teuthology.orchestra.run.smithi052:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-05-27T07:05:23.428 DEBUG:teuthology.orchestra.run.smithi105:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-05-27T07:05:23.474 DEBUG:teuthology.orchestra.run.smithi105:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-05-27T07:05:23.530 DEBUG:teuthology.orchestra.run.smithi105:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-05-27T07:05:23.613 DEBUG:teuthology.orchestra.run.smithi105:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-05-27T07:05:23.656 DEBUG:teuthology.orchestra.run.smithi105:> set -ex 2024-05-27T07:05:23.656 DEBUG:teuthology.orchestra.run.smithi105:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-05-27T07:05:23.722 DEBUG:teuthology.orchestra.run.smithi173:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-05-27T07:05:23.767 DEBUG:teuthology.orchestra.run.smithi173:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-05-27T07:05:23.824 DEBUG:teuthology.orchestra.run.smithi173:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-05-27T07:05:23.907 DEBUG:teuthology.orchestra.run.smithi173:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-05-27T07:05:23.950 DEBUG:teuthology.orchestra.run.smithi173:> set -ex 2024-05-27T07:05:23.950 DEBUG:teuthology.orchestra.run.smithi173:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-05-27T07:05:24.017 DEBUG:teuthology.orchestra.run.smithi052:> sudo service rsyslog restart 2024-05-27T07:05:24.019 DEBUG:teuthology.orchestra.run.smithi105:> sudo service rsyslog restart 2024-05-27T07:05:24.022 DEBUG:teuthology.orchestra.run.smithi173:> sudo service rsyslog restart 2024-05-27T07:05:24.061 INFO:teuthology.orchestra.run.smithi105.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-05-27T07:05:24.082 INFO:teuthology.orchestra.run.smithi052.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-05-27T07:05:24.095 INFO:teuthology.orchestra.run.smithi173.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-05-27T07:05:24.516 INFO:teuthology.run_tasks:Running task internal.timer... 2024-05-27T07:05:24.525 INFO:teuthology.task.internal:Starting timer... 2024-05-27T07:05:24.526 INFO:teuthology.run_tasks:Running task pcp... 2024-05-27T07:05:24.543 INFO:teuthology.run_tasks:Running task selinux... 2024-05-27T07:05:24.553 DEBUG:teuthology.task:Applying overrides for task selinux: {'allowlist': ['scontext=system_u:system_r:logrotate_t:s0', 'scontext=system_u:system_r:getty_t:s0', 'scontext=system_u:system_r:getty_t:s0']} 2024-05-27T07:05:24.553 DEBUG:teuthology.orchestra.run.smithi052:> sudo service auditd rotate 2024-05-27T07:05:24.614 INFO:teuthology.orchestra.run.smithi052.stdout:Rotating logs: 2024-05-27T07:05:24.616 DEBUG:teuthology.orchestra.run.smithi105:> sudo service auditd rotate 2024-05-27T07:05:24.692 INFO:teuthology.orchestra.run.smithi105.stdout:Rotating logs: 2024-05-27T07:05:24.694 DEBUG:teuthology.orchestra.run.smithi173:> sudo service auditd rotate 2024-05-27T07:05:24.753 INFO:teuthology.orchestra.run.smithi173.stdout:Rotating logs: 2024-05-27T07:05:24.755 DEBUG:teuthology.task.selinux:Getting current SELinux state 2024-05-27T07:05:24.755 DEBUG:teuthology.orchestra.run.smithi052:> /usr/sbin/getenforce 2024-05-27T07:05:24.774 INFO:teuthology.orchestra.run.smithi052.stdout:Permissive 2024-05-27T07:05:24.774 DEBUG:teuthology.orchestra.run.smithi105:> /usr/sbin/getenforce 2024-05-27T07:05:24.801 INFO:teuthology.orchestra.run.smithi105.stdout:Permissive 2024-05-27T07:05:24.801 DEBUG:teuthology.orchestra.run.smithi173:> /usr/sbin/getenforce 2024-05-27T07:05:24.836 INFO:teuthology.orchestra.run.smithi173.stdout:Permissive 2024-05-27T07:05:24.837 DEBUG:teuthology.task.selinux:Existing SELinux modes: {'ubuntu@smithi052.front.sepia.ceph.com': 'permissive', 'ubuntu@smithi105.front.sepia.ceph.com': 'permissive', 'ubuntu@smithi173.front.sepia.ceph.com': 'permissive'} 2024-05-27T07:05:24.837 DEBUG:teuthology.orchestra.run.smithi052:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e scontext=system_u:system_r:logrotate_t:s0 -e scontext=system_u:system_r:getty_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-05-27T07:05:24.864 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-05-27T07:05:24.864 DEBUG:teuthology.orchestra.run.smithi105:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e scontext=system_u:system_r:logrotate_t:s0 -e scontext=system_u:system_r:getty_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-05-27T07:05:24.893 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-05-27T07:05:24.893 DEBUG:teuthology.orchestra.run.smithi173:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e scontext=system_u:system_r:logrotate_t:s0 -e scontext=system_u:system_r:getty_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-05-27T07:05:24.919 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-05-27T07:05:24.919 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2024-05-27T07:05:24.919 DEBUG:teuthology.orchestra.run.smithi052:> sudo /usr/sbin/setenforce permissive 2024-05-27T07:05:24.968 DEBUG:teuthology.orchestra.run.smithi105:> sudo /usr/sbin/setenforce permissive 2024-05-27T07:05:24.998 DEBUG:teuthology.orchestra.run.smithi173:> sudo /usr/sbin/setenforce permissive 2024-05-27T07:05:25.022 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2024-05-27T07:05:25.032 DEBUG:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main to origin/main 2024-05-27T07:05:25.063 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'hosts': 'testnodes', 'tasks': [{'set_fact': {'ran_from_cephlab_playbook': True}}]}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}, {'hosts': 'testnodes', 'become': True, 'tasks': [{'name': 'Touch /ceph-qa-ready', 'file': {'path': '/ceph-qa-ready', 'state': 'touch'}, 'when': 'ran_from_cephlab_playbook|bool'}]}] 2024-05-27T07:05:25.064 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit smithi052.front.sepia.ceph.com,smithi105.front.sepia.ceph.com,smithi173.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main/cephlab.yml 2024-05-27T07:10:21.658 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@smithi052.front.sepia.ceph.com'), Remote(name='ubuntu@smithi105.front.sepia.ceph.com'), Remote(name='ubuntu@smithi173.front.sepia.ceph.com')] 2024-05-27T07:10:21.660 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi052.front.sepia.ceph.com' 2024-05-27T07:10:21.661 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi052.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-05-27T07:10:21.738 DEBUG:teuthology.orchestra.run.smithi052:> true 2024-05-27T07:10:21.782 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi052.front.sepia.ceph.com' 2024-05-27T07:10:21.783 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi105.front.sepia.ceph.com' 2024-05-27T07:10:21.783 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi105.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-05-27T07:10:21.863 DEBUG:teuthology.orchestra.run.smithi105:> true 2024-05-27T07:10:21.937 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi105.front.sepia.ceph.com' 2024-05-27T07:10:21.937 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi173.front.sepia.ceph.com' 2024-05-27T07:10:21.938 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi173.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-05-27T07:10:22.014 DEBUG:teuthology.orchestra.run.smithi173:> true 2024-05-27T07:10:22.093 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi173.front.sepia.ceph.com' 2024-05-27T07:10:22.094 INFO:teuthology.run_tasks:Running task clock... 2024-05-27T07:10:22.105 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2024-05-27T07:10:22.106 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-05-27T07:10:22.106 DEBUG:teuthology.orchestra.run.smithi052:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-05-27T07:10:22.108 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-05-27T07:10:22.108 DEBUG:teuthology.orchestra.run.smithi105:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-05-27T07:10:22.111 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-05-27T07:10:22.111 DEBUG:teuthology.orchestra.run.smithi173:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-05-27T07:10:22.137 INFO:teuthology.orchestra.run.smithi052.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-05-27T07:10:22.137 INFO:teuthology.orchestra.run.smithi105.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-05-27T07:10:22.151 INFO:teuthology.orchestra.run.smithi105.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-05-27T07:10:22.152 INFO:teuthology.orchestra.run.smithi052.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-05-27T07:10:22.162 INFO:teuthology.orchestra.run.smithi173.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-05-27T07:10:22.174 INFO:teuthology.orchestra.run.smithi105.stderr:sudo: ntpd: command not found 2024-05-27T07:10:22.175 INFO:teuthology.orchestra.run.smithi173.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-05-27T07:10:22.177 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: ntpd: command not found 2024-05-27T07:10:22.184 INFO:teuthology.orchestra.run.smithi105.stdout:506 Cannot talk to daemon 2024-05-27T07:10:22.188 INFO:teuthology.orchestra.run.smithi052.stdout:506 Cannot talk to daemon 2024-05-27T07:10:22.198 INFO:teuthology.orchestra.run.smithi105.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-05-27T07:10:22.200 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: ntpd: command not found 2024-05-27T07:10:22.203 INFO:teuthology.orchestra.run.smithi052.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-05-27T07:10:22.210 INFO:teuthology.orchestra.run.smithi105.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-05-27T07:10:22.212 INFO:teuthology.orchestra.run.smithi173.stdout:506 Cannot talk to daemon 2024-05-27T07:10:22.217 INFO:teuthology.orchestra.run.smithi052.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-05-27T07:10:22.226 INFO:teuthology.orchestra.run.smithi173.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-05-27T07:10:22.239 INFO:teuthology.orchestra.run.smithi173.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-05-27T07:10:22.261 INFO:teuthology.orchestra.run.smithi105.stderr:bash: line 1: ntpq: command not found 2024-05-27T07:10:22.265 INFO:teuthology.orchestra.run.smithi105.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-05-27T07:10:22.265 INFO:teuthology.orchestra.run.smithi105.stdout:=============================================================================== 2024-05-27T07:10:22.265 INFO:teuthology.orchestra.run.smithi105.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-05-27T07:10:22.265 INFO:teuthology.orchestra.run.smithi105.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-05-27T07:10:22.265 INFO:teuthology.orchestra.run.smithi105.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-05-27T07:10:22.265 INFO:teuthology.orchestra.run.smithi105.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-05-27T07:10:22.277 INFO:teuthology.orchestra.run.smithi052.stderr:bash: line 1: ntpq: command not found 2024-05-27T07:10:22.280 INFO:teuthology.orchestra.run.smithi052.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-05-27T07:10:22.281 INFO:teuthology.orchestra.run.smithi052.stdout:=============================================================================== 2024-05-27T07:10:22.281 INFO:teuthology.orchestra.run.smithi052.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-05-27T07:10:22.281 INFO:teuthology.orchestra.run.smithi052.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-05-27T07:10:22.281 INFO:teuthology.orchestra.run.smithi052.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-05-27T07:10:22.281 INFO:teuthology.orchestra.run.smithi052.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-05-27T07:10:22.297 INFO:teuthology.orchestra.run.smithi173.stderr:bash: line 1: ntpq: command not found 2024-05-27T07:10:22.300 INFO:teuthology.orchestra.run.smithi173.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-05-27T07:10:22.301 INFO:teuthology.orchestra.run.smithi173.stdout:=============================================================================== 2024-05-27T07:10:22.301 INFO:teuthology.orchestra.run.smithi173.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-05-27T07:10:22.301 INFO:teuthology.orchestra.run.smithi173.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-05-27T07:10:22.301 INFO:teuthology.orchestra.run.smithi173.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-05-27T07:10:22.301 INFO:teuthology.orchestra.run.smithi173.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-05-27T07:10:22.302 INFO:teuthology.run_tasks:Running task install... 2024-05-27T07:10:22.310 DEBUG:teuthology.task.install:project ceph 2024-05-27T07:10:22.310 DEBUG:teuthology.task.install:INSTALL overrides: {'ceph': {'flavor': 'default', 'sha1': '38f9d900c83c63799fbdbe61acc9a11b0d3554a6'}} 2024-05-27T07:10:22.310 DEBUG:teuthology.task.install:config {'extra_packages': {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror']}, 'extra_system_packages': {'deb': ['bison', 'flex', 'libelf-dev', 'libssl-dev', 'network-manager', 'iproute2', 'util-linux', 'dump', 'indent', 'libaio-dev', 'libtool-bin', 'uuid-dev', 'xfslibs-dev', 'postgresql', 'postgresql-client', 'postgresql-common', 'postgresql-contrib'], 'rpm': ['bison', 'flex', 'elfutils-libelf-devel', 'openssl-devel', 'NetworkManager', 'iproute', 'util-linux', 'libacl-devel', 'libaio-devel', 'libattr-devel', 'libtool', 'libuuid-devel', 'xfsdump', 'xfsprogs', 'xfsprogs-devel', 'libaio-devel', 'libtool', 'libuuid-devel', 'xfsprogs-devel', 'postgresql', 'postgresql-server', 'postgresql-contrib']}, 'flavor': 'default', 'sha1': '38f9d900c83c63799fbdbe61acc9a11b0d3554a6'} 2024-05-27T07:10:22.310 INFO:teuthology.task.install:Using flavor: default 2024-05-27T07:10:22.316 DEBUG:teuthology.task.install:Package list is: {'deb': ['ceph', 'cephadm', 'ceph-mds', 'ceph-mgr', 'ceph-common', 'ceph-fuse', 'ceph-test', 'ceph-volume', 'radosgw', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'libcephfs2', 'libcephfs-dev', 'librados2', 'librbd1', 'rbd-fuse'], 'rpm': ['ceph-radosgw', 'ceph-test', 'ceph', 'ceph-base', 'cephadm', 'ceph-immutable-object-cache', 'ceph-mgr', 'ceph-mgr-dashboard', 'ceph-mgr-diskprediction-local', 'ceph-mgr-rook', 'ceph-mgr-cephadm', 'ceph-fuse', 'ceph-volume', 'librados-devel', 'libcephfs2', 'libcephfs-devel', 'librados2', 'librbd1', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'rbd-fuse', 'rbd-mirror', 'rbd-nbd']} 2024-05-27T07:10:22.316 INFO:teuthology.task.install:extra packages: {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror']} 2024-05-27T07:10:22.316 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror']}, 'extra_system_packages': {'deb': ['bison', 'flex', 'libelf-dev', 'libssl-dev', 'network-manager', 'iproute2', 'util-linux', 'dump', 'indent', 'libaio-dev', 'libtool-bin', 'uuid-dev', 'xfslibs-dev', 'postgresql', 'postgresql-client', 'postgresql-common', 'postgresql-contrib'], 'rpm': ['bison', 'flex', 'elfutils-libelf-devel', 'openssl-devel', 'NetworkManager', 'iproute', 'util-linux', 'libacl-devel', 'libaio-devel', 'libattr-devel', 'libtool', 'libuuid-devel', 'xfsdump', 'xfsprogs', 'xfsprogs-devel', 'libaio-devel', 'libtool', 'libuuid-devel', 'xfsprogs-devel', 'postgresql', 'postgresql-server', 'postgresql-contrib']}, 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '38f9d900c83c63799fbdbe61acc9a11b0d3554a6', 'tag': None, 'wait_for_package': False} 2024-05-27T07:10:22.317 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=38f9d900c83c63799fbdbe61acc9a11b0d3554a6 2024-05-27T07:10:22.319 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror']}, 'extra_system_packages': {'deb': ['bison', 'flex', 'libelf-dev', 'libssl-dev', 'network-manager', 'iproute2', 'util-linux', 'dump', 'indent', 'libaio-dev', 'libtool-bin', 'uuid-dev', 'xfslibs-dev', 'postgresql', 'postgresql-client', 'postgresql-common', 'postgresql-contrib'], 'rpm': ['bison', 'flex', 'elfutils-libelf-devel', 'openssl-devel', 'NetworkManager', 'iproute', 'util-linux', 'libacl-devel', 'libaio-devel', 'libattr-devel', 'libtool', 'libuuid-devel', 'xfsdump', 'xfsprogs', 'xfsprogs-devel', 'libaio-devel', 'libtool', 'libuuid-devel', 'xfsprogs-devel', 'postgresql', 'postgresql-server', 'postgresql-contrib']}, 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '38f9d900c83c63799fbdbe61acc9a11b0d3554a6', 'tag': None, 'wait_for_package': False} 2024-05-27T07:10:22.319 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=38f9d900c83c63799fbdbe61acc9a11b0d3554a6 2024-05-27T07:10:22.320 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror']}, 'extra_system_packages': {'deb': ['bison', 'flex', 'libelf-dev', 'libssl-dev', 'network-manager', 'iproute2', 'util-linux', 'dump', 'indent', 'libaio-dev', 'libtool-bin', 'uuid-dev', 'xfslibs-dev', 'postgresql', 'postgresql-client', 'postgresql-common', 'postgresql-contrib'], 'rpm': ['bison', 'flex', 'elfutils-libelf-devel', 'openssl-devel', 'NetworkManager', 'iproute', 'util-linux', 'libacl-devel', 'libaio-devel', 'libattr-devel', 'libtool', 'libuuid-devel', 'xfsdump', 'xfsprogs', 'xfsprogs-devel', 'libaio-devel', 'libtool', 'libuuid-devel', 'xfsprogs-devel', 'postgresql', 'postgresql-server', 'postgresql-contrib']}, 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '38f9d900c83c63799fbdbe61acc9a11b0d3554a6', 'tag': None, 'wait_for_package': False} 2024-05-27T07:10:22.320 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=38f9d900c83c63799fbdbe61acc9a11b0d3554a6 2024-05-27T07:10:22.491 INFO:teuthology.task.install.rpm:Pulling from https://1.chacra.ceph.com/r/ceph/wip-vshankar-testing-20240521.063558-debug/38f9d900c83c63799fbdbe61acc9a11b0d3554a6/centos/9/flavors/default/ 2024-05-27T07:10:22.491 INFO:teuthology.task.install.rpm:Package version is 19.0.0-3872.g38f9d900 2024-05-27T07:10:22.588 INFO:teuthology.task.install.rpm:Pulling from https://1.chacra.ceph.com/r/ceph/wip-vshankar-testing-20240521.063558-debug/38f9d900c83c63799fbdbe61acc9a11b0d3554a6/centos/9/flavors/default/ 2024-05-27T07:10:22.588 INFO:teuthology.task.install.rpm:Package version is 19.0.0-3872.g38f9d900 2024-05-27T07:10:22.591 INFO:teuthology.task.install.rpm:Pulling from https://1.chacra.ceph.com/r/ceph/wip-vshankar-testing-20240521.063558-debug/38f9d900c83c63799fbdbe61acc9a11b0d3554a6/centos/9/flavors/default/ 2024-05-27T07:10:22.591 INFO:teuthology.task.install.rpm:Package version is 19.0.0-3872.g38f9d900 2024-05-27T07:10:22.633 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://1.chacra.ceph.com/r/ceph/wip-vshankar-testing-20240521.063558-debug/38f9d900c83c63799fbdbe61acc9a11b0d3554a6/centos/9/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://1.chacra.ceph.com/r/ceph/wip-vshankar-testing-20240521.063558-debug/38f9d900c83c63799fbdbe61acc9a11b0d3554a6/centos/9/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://1.chacra.ceph.com/r/ceph/wip-vshankar-testing-20240521.063558-debug/38f9d900c83c63799fbdbe61acc9a11b0d3554a6/centos/9/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-05-27T07:10:22.634 DEBUG:teuthology.orchestra.run.smithi173:> set -ex 2024-05-27T07:10:22.634 DEBUG:teuthology.orchestra.run.smithi173:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-05-27T07:10:22.659 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd, python3-cephfs, cephfs-shell, cephfs-top, cephfs-mirror, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib on remote rpm x86_64 2024-05-27T07:10:22.659 DEBUG:teuthology.orchestra.run.smithi173:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/38f9d900c83c63799fbdbe61acc9a11b0d3554a6/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-05-27T07:10:22.709 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://1.chacra.ceph.com/r/ceph/wip-vshankar-testing-20240521.063558-debug/38f9d900c83c63799fbdbe61acc9a11b0d3554a6/centos/9/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://1.chacra.ceph.com/r/ceph/wip-vshankar-testing-20240521.063558-debug/38f9d900c83c63799fbdbe61acc9a11b0d3554a6/centos/9/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://1.chacra.ceph.com/r/ceph/wip-vshankar-testing-20240521.063558-debug/38f9d900c83c63799fbdbe61acc9a11b0d3554a6/centos/9/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-05-27T07:10:22.709 DEBUG:teuthology.orchestra.run.smithi105:> set -ex 2024-05-27T07:10:22.709 DEBUG:teuthology.orchestra.run.smithi105:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-05-27T07:10:22.724 DEBUG:teuthology.orchestra.run.smithi173:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-05-27T07:10:22.735 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd, python3-cephfs, cephfs-shell, cephfs-top, cephfs-mirror, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib on remote rpm x86_64 2024-05-27T07:10:22.736 DEBUG:teuthology.orchestra.run.smithi105:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/38f9d900c83c63799fbdbe61acc9a11b0d3554a6/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-05-27T07:10:22.798 DEBUG:teuthology.orchestra.run.smithi173:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-05-27T07:10:22.801 DEBUG:teuthology.orchestra.run.smithi105:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-05-27T07:10:22.811 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://1.chacra.ceph.com/r/ceph/wip-vshankar-testing-20240521.063558-debug/38f9d900c83c63799fbdbe61acc9a11b0d3554a6/centos/9/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://1.chacra.ceph.com/r/ceph/wip-vshankar-testing-20240521.063558-debug/38f9d900c83c63799fbdbe61acc9a11b0d3554a6/centos/9/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://1.chacra.ceph.com/r/ceph/wip-vshankar-testing-20240521.063558-debug/38f9d900c83c63799fbdbe61acc9a11b0d3554a6/centos/9/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-05-27T07:10:22.811 DEBUG:teuthology.orchestra.run.smithi052:> set -ex 2024-05-27T07:10:22.811 DEBUG:teuthology.orchestra.run.smithi052:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-05-27T07:10:22.840 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd, python3-cephfs, cephfs-shell, cephfs-top, cephfs-mirror, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib on remote rpm x86_64 2024-05-27T07:10:22.840 DEBUG:teuthology.orchestra.run.smithi052:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/38f9d900c83c63799fbdbe61acc9a11b0d3554a6/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-05-27T07:10:22.862 INFO:teuthology.orchestra.run.smithi173.stdout:check_obsoletes = 1 2024-05-27T07:10:22.864 DEBUG:teuthology.orchestra.run.smithi173:> sudo yum clean all 2024-05-27T07:10:22.877 DEBUG:teuthology.orchestra.run.smithi105:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-05-27T07:10:22.908 DEBUG:teuthology.orchestra.run.smithi052:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-05-27T07:10:22.941 INFO:teuthology.orchestra.run.smithi105.stdout:check_obsoletes = 1 2024-05-27T07:10:22.943 DEBUG:teuthology.orchestra.run.smithi105:> sudo yum clean all 2024-05-27T07:10:22.988 DEBUG:teuthology.orchestra.run.smithi052:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-05-27T07:10:23.054 INFO:teuthology.orchestra.run.smithi052.stdout:check_obsoletes = 1 2024-05-27T07:10:23.055 DEBUG:teuthology.orchestra.run.smithi052:> sudo yum clean all 2024-05-27T07:10:23.085 INFO:teuthology.orchestra.run.smithi173.stdout:45 files removed 2024-05-27T07:10:23.112 DEBUG:teuthology.orchestra.run.smithi173:> sudo yum -y install ceph-radosgw ceph-test ceph ceph-base cephadm ceph-immutable-object-cache ceph-mgr ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-rook ceph-mgr-cephadm ceph-fuse ceph-volume librados-devel libcephfs2 libcephfs-devel librados2 librbd1 python3-rados python3-rgw python3-cephfs python3-rbd rbd-fuse rbd-mirror rbd-nbd python3-cephfs cephfs-shell cephfs-top cephfs-mirror bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib 2024-05-27T07:10:23.168 INFO:teuthology.orchestra.run.smithi105.stdout:45 files removed 2024-05-27T07:10:23.194 DEBUG:teuthology.orchestra.run.smithi105:> sudo yum -y install ceph-radosgw ceph-test ceph ceph-base cephadm ceph-immutable-object-cache ceph-mgr ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-rook ceph-mgr-cephadm ceph-fuse ceph-volume librados-devel libcephfs2 libcephfs-devel librados2 librbd1 python3-rados python3-rgw python3-cephfs python3-rbd rbd-fuse rbd-mirror rbd-nbd python3-cephfs cephfs-shell cephfs-top cephfs-mirror bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib 2024-05-27T07:10:23.303 INFO:teuthology.orchestra.run.smithi052.stdout:45 files removed 2024-05-27T07:10:23.331 DEBUG:teuthology.orchestra.run.smithi052:> sudo yum -y install ceph-radosgw ceph-test ceph ceph-base cephadm ceph-immutable-object-cache ceph-mgr ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-rook ceph-mgr-cephadm ceph-fuse ceph-volume librados-devel libcephfs2 libcephfs-devel librados2 librbd1 python3-rados python3-rgw python3-cephfs python3-rbd rbd-fuse rbd-mirror rbd-nbd python3-cephfs cephfs-shell cephfs-top cephfs-mirror bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib 2024-05-27T07:10:23.690 INFO:teuthology.orchestra.run.smithi173.stdout:ceph packages for x86_64 263 kB/s | 84 kB 00:00 2024-05-27T07:10:23.849 INFO:teuthology.orchestra.run.smithi105.stdout:ceph packages for x86_64 212 kB/s | 84 kB 00:00 2024-05-27T07:10:24.090 INFO:teuthology.orchestra.run.smithi173.stdout:ceph noarch packages 33 kB/s | 12 kB 00:00 2024-05-27T07:10:24.095 INFO:teuthology.orchestra.run.smithi052.stdout:ceph packages for x86_64 175 kB/s | 84 kB 00:00 2024-05-27T07:10:24.516 INFO:teuthology.orchestra.run.smithi173.stdout:ceph source packages 4.8 kB/s | 1.9 kB 00:00 2024-05-27T07:10:24.537 INFO:teuthology.orchestra.run.smithi052.stdout:ceph noarch packages 30 kB/s | 12 kB 00:00 2024-05-27T07:10:24.850 INFO:teuthology.orchestra.run.smithi105.stdout:ceph noarch packages 12 kB/s | 12 kB 00:00 2024-05-27T07:10:24.863 INFO:teuthology.orchestra.run.smithi052.stdout:ceph source packages 6.8 kB/s | 1.9 kB 00:00 2024-05-27T07:10:25.381 INFO:teuthology.orchestra.run.smithi052.stdout:CentOS Stream 9 - BaseOS 17 MB/s | 8.1 MB 00:00 2024-05-27T07:10:25.409 INFO:teuthology.orchestra.run.smithi105.stdout:ceph source packages 3.6 kB/s | 1.9 kB 00:00 2024-05-27T07:10:25.876 INFO:teuthology.orchestra.run.smithi105.stdout:CentOS Stream 9 - BaseOS 19 MB/s | 8.1 MB 00:00 2024-05-27T07:10:26.391 INFO:teuthology.orchestra.run.smithi173.stdout:CentOS Stream 9 - BaseOS 4.4 MB/s | 8.1 MB 00:01 2024-05-27T07:10:27.308 INFO:teuthology.orchestra.run.smithi052.stdout:CentOS Stream 9 - AppStream 23 MB/s | 19 MB 00:00 2024-05-27T07:10:28.860 INFO:teuthology.orchestra.run.smithi105.stdout:CentOS Stream 9 - AppStream 9.9 MB/s | 19 MB 00:01 2024-05-27T07:10:29.925 INFO:teuthology.orchestra.run.smithi173.stdout:CentOS Stream 9 - AppStream 7.7 MB/s | 19 MB 00:02 2024-05-27T07:10:32.168 INFO:teuthology.orchestra.run.smithi052.stdout:CentOS Stream 9 - CRB 5.4 MB/s | 6.2 MB 00:01 2024-05-27T07:10:33.544 INFO:teuthology.orchestra.run.smithi052.stdout:CentOS Stream 9 - Extras packages 96 kB/s | 17 kB 00:00 2024-05-27T07:10:34.969 INFO:teuthology.orchestra.run.smithi105.stdout:CentOS Stream 9 - CRB 2.4 MB/s | 6.2 MB 00:02 2024-05-27T07:10:35.259 INFO:teuthology.orchestra.run.smithi173.stdout:CentOS Stream 9 - CRB 3.5 MB/s | 6.2 MB 00:01 2024-05-27T07:10:35.545 INFO:teuthology.orchestra.run.smithi052.stdout:Extra Packages for Enterprise Linux 11 MB/s | 21 MB 00:01 2024-05-27T07:10:36.436 INFO:teuthology.orchestra.run.smithi105.stdout:CentOS Stream 9 - Extras packages 54 kB/s | 17 kB 00:00 2024-05-27T07:10:36.768 INFO:teuthology.orchestra.run.smithi173.stdout:CentOS Stream 9 - Extras packages 47 kB/s | 17 kB 00:00 2024-05-27T07:10:37.832 INFO:teuthology.orchestra.run.smithi105.stdout:Extra Packages for Enterprise Linux 17 MB/s | 21 MB 00:01 2024-05-27T07:10:38.720 INFO:teuthology.orchestra.run.smithi173.stdout:Extra Packages for Enterprise Linux 12 MB/s | 21 MB 00:01 2024-05-27T07:10:40.912 INFO:teuthology.orchestra.run.smithi052.stdout:lab-extras 44 kB/s | 1.7 kB 00:00 2024-05-27T07:10:43.147 INFO:teuthology.orchestra.run.smithi105.stdout:lab-extras 8.7 kB/s | 1.7 kB 00:00 2024-05-27T07:10:43.189 INFO:teuthology.orchestra.run.smithi052.stdout:Package librados2-2:16.2.4-5.el9.x86_64 is already installed. 2024-05-27T07:10:43.189 INFO:teuthology.orchestra.run.smithi052.stdout:Package librbd1-2:16.2.4-5.el9.x86_64 is already installed. 2024-05-27T07:10:43.200 INFO:teuthology.orchestra.run.smithi052.stdout:Package NetworkManager-1:1.47.4-1.el9.x86_64 is already installed. 2024-05-27T07:10:43.201 INFO:teuthology.orchestra.run.smithi052.stdout:Package iproute-6.2.0-5.el9.x86_64 is already installed. 2024-05-27T07:10:43.202 INFO:teuthology.orchestra.run.smithi052.stdout:Package util-linux-2.37.4-18.el9.x86_64 is already installed. 2024-05-27T07:10:43.203 INFO:teuthology.orchestra.run.smithi052.stdout:Package libacl-devel-2.3.1-4.el9.x86_64 is already installed. 2024-05-27T07:10:43.205 INFO:teuthology.orchestra.run.smithi052.stdout:Package libattr-devel-2.5.1-3.el9.x86_64 is already installed. 2024-05-27T07:10:43.207 INFO:teuthology.orchestra.run.smithi052.stdout:Package xfsdump-3.1.12-4.el9.x86_64 is already installed. 2024-05-27T07:10:43.208 INFO:teuthology.orchestra.run.smithi052.stdout:Package xfsprogs-6.3.0-1.el9.x86_64 is already installed. 2024-05-27T07:10:43.274 INFO:teuthology.orchestra.run.smithi052.stdout:Dependencies resolved. 2024-05-27T07:10:43.307 INFO:teuthology.orchestra.run.smithi052.stdout:======================================================================================= 2024-05-27T07:10:43.307 INFO:teuthology.orchestra.run.smithi052.stdout: Package Arch Version Repository Size 2024-05-27T07:10:43.307 INFO:teuthology.orchestra.run.smithi052.stdout:======================================================================================= 2024-05-27T07:10:43.307 INFO:teuthology.orchestra.run.smithi052.stdout:Installing: 2024-05-27T07:10:43.307 INFO:teuthology.orchestra.run.smithi052.stdout: bison x86_64 3.7.4-5.el9 appstream 943 k 2024-05-27T07:10:43.307 INFO:teuthology.orchestra.run.smithi052.stdout: ceph x86_64 2:19.0.0-3872.g38f9d900.el9 ceph 6.6 k 2024-05-27T07:10:43.307 INFO:teuthology.orchestra.run.smithi052.stdout: ceph-base x86_64 2:19.0.0-3872.g38f9d900.el9 ceph 5.4 M 2024-05-27T07:10:43.307 INFO:teuthology.orchestra.run.smithi052.stdout: ceph-fuse x86_64 2:19.0.0-3872.g38f9d900.el9 ceph 863 k 2024-05-27T07:10:43.307 INFO:teuthology.orchestra.run.smithi052.stdout: ceph-immutable-object-cache x86_64 2:19.0.0-3872.g38f9d900.el9 ceph 147 k 2024-05-27T07:10:43.307 INFO:teuthology.orchestra.run.smithi052.stdout: ceph-mgr x86_64 2:19.0.0-3872.g38f9d900.el9 ceph 1.5 M 2024-05-27T07:10:43.307 INFO:teuthology.orchestra.run.smithi052.stdout: ceph-mgr-cephadm noarch 2:19.0.0-3872.g38f9d900.el9 ceph-noarch 143 k 2024-05-27T07:10:43.308 INFO:teuthology.orchestra.run.smithi052.stdout: ceph-mgr-dashboard noarch 2:19.0.0-3872.g38f9d900.el9 ceph-noarch 3.5 M 2024-05-27T07:10:43.308 INFO:teuthology.orchestra.run.smithi052.stdout: ceph-mgr-diskprediction-local noarch 2:19.0.0-3872.g38f9d900.el9 ceph-noarch 7.4 M 2024-05-27T07:10:43.308 INFO:teuthology.orchestra.run.smithi052.stdout: ceph-mgr-rook noarch 2:19.0.0-3872.g38f9d900.el9 ceph-noarch 49 k 2024-05-27T07:10:43.308 INFO:teuthology.orchestra.run.smithi052.stdout: ceph-radosgw x86_64 2:19.0.0-3872.g38f9d900.el9 ceph 10 M 2024-05-27T07:10:43.308 INFO:teuthology.orchestra.run.smithi052.stdout: ceph-test x86_64 2:19.0.0-3872.g38f9d900.el9 ceph 43 M 2024-05-27T07:10:43.308 INFO:teuthology.orchestra.run.smithi052.stdout: ceph-volume noarch 2:19.0.0-3872.g38f9d900.el9 ceph-noarch 273 k 2024-05-27T07:10:43.308 INFO:teuthology.orchestra.run.smithi052.stdout: cephadm noarch 2:19.0.0-3872.g38f9d900.el9 ceph-noarch 764 k 2024-05-27T07:10:43.308 INFO:teuthology.orchestra.run.smithi052.stdout: cephfs-mirror x86_64 2:19.0.0-3872.g38f9d900.el9 ceph 224 k 2024-05-27T07:10:43.308 INFO:teuthology.orchestra.run.smithi052.stdout: cephfs-shell x86_64 2:19.0.0-3872.g38f9d900.el9 ceph 27 k 2024-05-27T07:10:43.308 INFO:teuthology.orchestra.run.smithi052.stdout: cephfs-top noarch 2:19.0.0-3872.g38f9d900.el9 ceph-noarch 18 k 2024-05-27T07:10:43.308 INFO:teuthology.orchestra.run.smithi052.stdout: elfutils-libelf-devel x86_64 0.191-4.el9 appstream 22 k 2024-05-27T07:10:43.308 INFO:teuthology.orchestra.run.smithi052.stdout: flex x86_64 2.6.4-9.el9 appstream 313 k 2024-05-27T07:10:43.308 INFO:teuthology.orchestra.run.smithi052.stdout: libaio-devel x86_64 0.3.111-13.el9 appstream 11 k 2024-05-27T07:10:43.308 INFO:teuthology.orchestra.run.smithi052.stdout: libcephfs-devel x86_64 2:19.0.0-3872.g38f9d900.el9 ceph 32 k 2024-05-27T07:10:43.309 INFO:teuthology.orchestra.run.smithi052.stdout: libcephfs2 x86_64 2:19.0.0-3872.g38f9d900.el9 ceph 755 k 2024-05-27T07:10:43.309 INFO:teuthology.orchestra.run.smithi052.stdout: librados-devel x86_64 2:19.0.0-3872.g38f9d900.el9 ceph 127 k 2024-05-27T07:10:43.309 INFO:teuthology.orchestra.run.smithi052.stdout: libtool x86_64 2.4.6-45.el9 appstream 582 k 2024-05-27T07:10:43.309 INFO:teuthology.orchestra.run.smithi052.stdout: libuuid-devel x86_64 2.37.4-18.el9 appstream 25 k 2024-05-27T07:10:43.309 INFO:teuthology.orchestra.run.smithi052.stdout: openssl-devel x86_64 1:3.2.1-1.el9 appstream 4.4 M 2024-05-27T07:10:43.309 INFO:teuthology.orchestra.run.smithi052.stdout: postgresql x86_64 13.14-1.el9 appstream 1.6 M 2024-05-27T07:10:43.309 INFO:teuthology.orchestra.run.smithi052.stdout: postgresql-contrib x86_64 13.14-1.el9 appstream 911 k 2024-05-27T07:10:43.309 INFO:teuthology.orchestra.run.smithi052.stdout: postgresql-server x86_64 13.14-1.el9 appstream 5.8 M 2024-05-27T07:10:43.310 INFO:teuthology.orchestra.run.smithi052.stdout: python3-cephfs x86_64 2:19.0.0-3872.g38f9d900.el9 ceph 161 k 2024-05-27T07:10:43.310 INFO:teuthology.orchestra.run.smithi052.stdout: python3-rados x86_64 2:19.0.0-3872.g38f9d900.el9 ceph 320 k 2024-05-27T07:10:43.310 INFO:teuthology.orchestra.run.smithi052.stdout: python3-rbd x86_64 2:19.0.0-3872.g38f9d900.el9 ceph 300 k 2024-05-27T07:10:43.310 INFO:teuthology.orchestra.run.smithi052.stdout: python3-rgw x86_64 2:19.0.0-3872.g38f9d900.el9 ceph 99 k 2024-05-27T07:10:43.310 INFO:teuthology.orchestra.run.smithi052.stdout: rbd-fuse x86_64 2:19.0.0-3872.g38f9d900.el9 ceph 86 k 2024-05-27T07:10:43.311 INFO:teuthology.orchestra.run.smithi052.stdout: rbd-mirror x86_64 2:19.0.0-3872.g38f9d900.el9 ceph 3.0 M 2024-05-27T07:10:43.311 INFO:teuthology.orchestra.run.smithi052.stdout: rbd-nbd x86_64 2:19.0.0-3872.g38f9d900.el9 ceph 169 k 2024-05-27T07:10:43.311 INFO:teuthology.orchestra.run.smithi052.stdout: xfsprogs-devel x86_64 6.3.0-1.el9 appstream 94 k 2024-05-27T07:10:43.311 INFO:teuthology.orchestra.run.smithi052.stdout:Upgrading: 2024-05-27T07:10:43.311 INFO:teuthology.orchestra.run.smithi052.stdout: NetworkManager x86_64 1:1.47.90-1.el9 baseos 2.3 M 2024-05-27T07:10:43.311 INFO:teuthology.orchestra.run.smithi052.stdout: NetworkManager-libnm x86_64 1:1.47.90-1.el9 baseos 1.8 M 2024-05-27T07:10:43.311 INFO:teuthology.orchestra.run.smithi052.stdout: NetworkManager-team x86_64 1:1.47.90-1.el9 baseos 41 k 2024-05-27T07:10:43.311 INFO:teuthology.orchestra.run.smithi052.stdout: NetworkManager-tui x86_64 1:1.47.90-1.el9 baseos 248 k 2024-05-27T07:10:43.311 INFO:teuthology.orchestra.run.smithi052.stdout: elfutils-debuginfod-client x86_64 0.191-4.el9 baseos 37 k 2024-05-27T07:10:43.311 INFO:teuthology.orchestra.run.smithi052.stdout: elfutils-libelf x86_64 0.191-4.el9 baseos 207 k 2024-05-27T07:10:43.311 INFO:teuthology.orchestra.run.smithi052.stdout: elfutils-libs x86_64 0.191-4.el9 baseos 260 k 2024-05-27T07:10:43.312 INFO:teuthology.orchestra.run.smithi052.stdout: librados2 x86_64 2:19.0.0-3872.g38f9d900.el9 ceph 3.3 M 2024-05-27T07:10:43.312 INFO:teuthology.orchestra.run.smithi052.stdout: librbd1 x86_64 2:19.0.0-3872.g38f9d900.el9 ceph 3.1 M 2024-05-27T07:10:43.312 INFO:teuthology.orchestra.run.smithi052.stdout: openssl x86_64 1:3.2.1-1.el9 baseos 1.3 M 2024-05-27T07:10:43.312 INFO:teuthology.orchestra.run.smithi052.stdout: openssl-libs x86_64 1:3.2.1-1.el9 baseos 2.4 M 2024-05-27T07:10:43.312 INFO:teuthology.orchestra.run.smithi052.stdout:Installing dependencies: 2024-05-27T07:10:43.312 INFO:teuthology.orchestra.run.smithi052.stdout: abseil-cpp x86_64 20211102.0-3.el9 epel 548 k 2024-05-27T07:10:43.312 INFO:teuthology.orchestra.run.smithi052.stdout: automake noarch 1.16.2-8.el9 appstream 689 k 2024-05-27T07:10:43.312 INFO:teuthology.orchestra.run.smithi052.stdout: boost-program-options x86_64 1.75.0-8.el9 appstream 107 k 2024-05-27T07:10:43.312 INFO:teuthology.orchestra.run.smithi052.stdout: ceph-common x86_64 2:19.0.0-3872.g38f9d900.el9 ceph 20 M 2024-05-27T07:10:43.312 INFO:teuthology.orchestra.run.smithi052.stdout: ceph-grafana-dashboards noarch 2:19.0.0-3872.g38f9d900.el9 ceph-noarch 34 k 2024-05-27T07:10:43.312 INFO:teuthology.orchestra.run.smithi052.stdout: ceph-mds x86_64 2:19.0.0-3872.g38f9d900.el9 ceph 2.3 M 2024-05-27T07:10:43.312 INFO:teuthology.orchestra.run.smithi052.stdout: ceph-mgr-modules-core noarch 2:19.0.0-3872.g38f9d900.el9 ceph-noarch 274 k 2024-05-27T07:10:43.313 INFO:teuthology.orchestra.run.smithi052.stdout: ceph-mon x86_64 2:19.0.0-3872.g38f9d900.el9 ceph 4.9 M 2024-05-27T07:10:43.313 INFO:teuthology.orchestra.run.smithi052.stdout: ceph-osd x86_64 2:19.0.0-3872.g38f9d900.el9 ceph 18 M 2024-05-27T07:10:43.313 INFO:teuthology.orchestra.run.smithi052.stdout: ceph-prometheus-alerts noarch 2:19.0.0-3872.g38f9d900.el9 ceph-noarch 16 k 2024-05-27T07:10:43.313 INFO:teuthology.orchestra.run.smithi052.stdout: ceph-selinux x86_64 2:19.0.0-3872.g38f9d900.el9 ceph 25 k 2024-05-27T07:10:43.313 INFO:teuthology.orchestra.run.smithi052.stdout: flexiblas x86_64 3.0.4-8.el9 appstream 33 k 2024-05-27T07:10:43.313 INFO:teuthology.orchestra.run.smithi052.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 appstream 3.0 M 2024-05-27T07:10:43.313 INFO:teuthology.orchestra.run.smithi052.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 appstream 18 k 2024-05-27T07:10:43.313 INFO:teuthology.orchestra.run.smithi052.stdout: fuse x86_64 2.9.9-15.el9 baseos 80 k 2024-05-27T07:10:43.313 INFO:teuthology.orchestra.run.smithi052.stdout: gperftools-libs x86_64 2.9.1-2.el9 epel 309 k 2024-05-27T07:10:43.313 INFO:teuthology.orchestra.run.smithi052.stdout: grpc-data noarch 1.46.7-10.el9 epel 19 k 2024-05-27T07:10:43.313 INFO:teuthology.orchestra.run.smithi052.stdout: libarrow x86_64 9.0.0-11.el9 epel 4.4 M 2024-05-27T07:10:43.314 INFO:teuthology.orchestra.run.smithi052.stdout: libarrow-doc noarch 9.0.0-11.el9 epel 26 k 2024-05-27T07:10:43.314 INFO:teuthology.orchestra.run.smithi052.stdout: libcephsqlite x86_64 2:19.0.0-3872.g38f9d900.el9 ceph 157 k 2024-05-27T07:10:43.314 INFO:teuthology.orchestra.run.smithi052.stdout: libgfortran x86_64 11.4.1-3.el9 baseos 803 k 2024-05-27T07:10:43.314 INFO:teuthology.orchestra.run.smithi052.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2024-05-27T07:10:43.314 INFO:teuthology.orchestra.run.smithi052.stdout: libpmemobj x86_64 1.12.1-1.el9 appstream 160 k 2024-05-27T07:10:43.314 INFO:teuthology.orchestra.run.smithi052.stdout: libquadmath x86_64 11.4.1-3.el9 baseos 195 k 2024-05-27T07:10:43.314 INFO:teuthology.orchestra.run.smithi052.stdout: librabbitmq x86_64 0.11.0-7.el9 appstream 45 k 2024-05-27T07:10:43.314 INFO:teuthology.orchestra.run.smithi052.stdout: libradosstriper1 x86_64 2:19.0.0-3872.g38f9d900.el9 ceph 498 k 2024-05-27T07:10:43.314 INFO:teuthology.orchestra.run.smithi052.stdout: librdkafka x86_64 1.6.1-102.el9 appstream 662 k 2024-05-27T07:10:43.314 INFO:teuthology.orchestra.run.smithi052.stdout: librgw2 x86_64 2:19.0.0-3872.g38f9d900.el9 ceph 5.2 M 2024-05-27T07:10:43.314 INFO:teuthology.orchestra.run.smithi052.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2024-05-27T07:10:43.314 INFO:teuthology.orchestra.run.smithi052.stdout: libzstd-devel x86_64 1.5.1-2.el9 appstream 47 k 2024-05-27T07:10:43.315 INFO:teuthology.orchestra.run.smithi052.stdout: lttng-ust x86_64 2.12.0-6.el9 appstream 292 k 2024-05-27T07:10:43.315 INFO:teuthology.orchestra.run.smithi052.stdout: lua x86_64 5.4.4-4.el9 appstream 188 k 2024-05-27T07:10:43.315 INFO:teuthology.orchestra.run.smithi052.stdout: lua-devel x86_64 5.4.4-4.el9 crb 22 k 2024-05-27T07:10:43.315 INFO:teuthology.orchestra.run.smithi052.stdout: luarocks noarch 3.9.2-5.el9 epel 151 k 2024-05-27T07:10:43.315 INFO:teuthology.orchestra.run.smithi052.stdout: openblas x86_64 0.3.21-2.el9 appstream 35 k 2024-05-27T07:10:43.315 INFO:teuthology.orchestra.run.smithi052.stdout: openblas-openmp x86_64 0.3.21-2.el9 appstream 4.7 M 2024-05-27T07:10:43.315 INFO:teuthology.orchestra.run.smithi052.stdout: parquet-libs x86_64 9.0.0-11.el9 epel 839 k 2024-05-27T07:10:43.315 INFO:teuthology.orchestra.run.smithi052.stdout: postgresql-private-libs x86_64 13.14-1.el9 appstream 133 k 2024-05-27T07:10:43.315 INFO:teuthology.orchestra.run.smithi052.stdout: protobuf x86_64 3.14.0-13.el9 appstream 1.0 M 2024-05-27T07:10:43.315 INFO:teuthology.orchestra.run.smithi052.stdout: protobuf-compiler x86_64 3.14.0-13.el9 crb 863 k 2024-05-27T07:10:43.315 INFO:teuthology.orchestra.run.smithi052.stdout: python3-asyncssh noarch 2.13.2-5.el9 epel 548 k 2024-05-27T07:10:43.315 INFO:teuthology.orchestra.run.smithi052.stdout: python3-attrs noarch 20.3.0-7.el9 appstream 87 k 2024-05-27T07:10:43.316 INFO:teuthology.orchestra.run.smithi052.stdout: python3-babel noarch 2.9.1-2.el9 appstream 6.0 M 2024-05-27T07:10:43.316 INFO:teuthology.orchestra.run.smithi052.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2024-05-27T07:10:43.316 INFO:teuthology.orchestra.run.smithi052.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2024-05-27T07:10:43.316 INFO:teuthology.orchestra.run.smithi052.stdout: python3-ceph-argparse x86_64 2:19.0.0-3872.g38f9d900.el9 ceph 45 k 2024-05-27T07:10:43.316 INFO:teuthology.orchestra.run.smithi052.stdout: python3-ceph-common x86_64 2:19.0.0-3872.g38f9d900.el9 ceph 136 k 2024-05-27T07:10:43.316 INFO:teuthology.orchestra.run.smithi052.stdout: python3-certifi noarch 2021.10.8-2.el9 epel 15 k 2024-05-27T07:10:43.316 INFO:teuthology.orchestra.run.smithi052.stdout: python3-cffi x86_64 1.14.5-5.el9 baseos 253 k 2024-05-27T07:10:43.316 INFO:teuthology.orchestra.run.smithi052.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2024-05-27T07:10:43.316 INFO:teuthology.orchestra.run.smithi052.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2024-05-27T07:10:43.316 INFO:teuthology.orchestra.run.smithi052.stdout: python3-cmd2 noarch 2.4.2-2.el9 epel 303 k 2024-05-27T07:10:43.316 INFO:teuthology.orchestra.run.smithi052.stdout: python3-colorama noarch 0.4.6-3.el9 appstream 61 k 2024-05-27T07:10:43.316 INFO:teuthology.orchestra.run.smithi052.stdout: python3-cryptography x86_64 36.0.1-4.el9 baseos 1.2 M 2024-05-27T07:10:43.316 INFO:teuthology.orchestra.run.smithi052.stdout: python3-devel x86_64 3.9.18-3.el9 appstream 244 k 2024-05-27T07:10:43.317 INFO:teuthology.orchestra.run.smithi052.stdout: python3-google-auth noarch 1:2.29.0-1.el9 epel 206 k 2024-05-27T07:10:43.317 INFO:teuthology.orchestra.run.smithi052.stdout: python3-grpcio x86_64 1.46.7-10.el9 epel 2.0 M 2024-05-27T07:10:43.317 INFO:teuthology.orchestra.run.smithi052.stdout: python3-grpcio-tools x86_64 1.46.7-10.el9 epel 144 k 2024-05-27T07:10:43.317 INFO:teuthology.orchestra.run.smithi052.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2024-05-27T07:10:43.317 INFO:teuthology.orchestra.run.smithi052.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2024-05-27T07:10:43.317 INFO:teuthology.orchestra.run.smithi052.stdout: python3-jaraco-collections noarch 3.0.0-8.el9 epel 23 k 2024-05-27T07:10:43.317 INFO:teuthology.orchestra.run.smithi052.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2024-05-27T07:10:43.317 INFO:teuthology.orchestra.run.smithi052.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2024-05-27T07:10:43.317 INFO:teuthology.orchestra.run.smithi052.stdout: python3-jinja2 noarch 2.11.3-6.el9 appstream 248 k 2024-05-27T07:10:43.317 INFO:teuthology.orchestra.run.smithi052.stdout: python3-jsonpatch noarch 1.21-16.el9 appstream 26 k 2024-05-27T07:10:43.317 INFO:teuthology.orchestra.run.smithi052.stdout: python3-jsonpointer noarch 2.0-4.el9 appstream 19 k 2024-05-27T07:10:43.318 INFO:teuthology.orchestra.run.smithi052.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2024-05-27T07:10:43.318 INFO:teuthology.orchestra.run.smithi052.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2024-05-27T07:10:43.318 INFO:teuthology.orchestra.run.smithi052.stdout: python3-mako noarch 1.1.4-6.el9 appstream 172 k 2024-05-27T07:10:43.318 INFO:teuthology.orchestra.run.smithi052.stdout: python3-markupsafe x86_64 1.1.1-12.el9 appstream 35 k 2024-05-27T07:10:43.318 INFO:teuthology.orchestra.run.smithi052.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2024-05-27T07:10:43.318 INFO:teuthology.orchestra.run.smithi052.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2024-05-27T07:10:43.318 INFO:teuthology.orchestra.run.smithi052.stdout: python3-numpy x86_64 1:1.20.1-5.el9 appstream 5.1 M 2024-05-27T07:10:43.318 INFO:teuthology.orchestra.run.smithi052.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 appstream 250 k 2024-05-27T07:10:43.318 INFO:teuthology.orchestra.run.smithi052.stdout: python3-oauthlib noarch 3.1.1-5.el9 appstream 222 k 2024-05-27T07:10:43.318 INFO:teuthology.orchestra.run.smithi052.stdout: python3-packaging noarch 20.9-5.el9 appstream 77 k 2024-05-27T07:10:43.318 INFO:teuthology.orchestra.run.smithi052.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2024-05-27T07:10:43.318 INFO:teuthology.orchestra.run.smithi052.stdout: python3-ply noarch 3.11-14.el9 baseos 106 k 2024-05-27T07:10:43.319 INFO:teuthology.orchestra.run.smithi052.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2024-05-27T07:10:43.319 INFO:teuthology.orchestra.run.smithi052.stdout: python3-prettytable noarch 0.7.2-27.el9 appstream 42 k 2024-05-27T07:10:43.319 INFO:teuthology.orchestra.run.smithi052.stdout: python3-protobuf noarch 3.14.0-13.el9 appstream 269 k 2024-05-27T07:10:43.319 INFO:teuthology.orchestra.run.smithi052.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2024-05-27T07:10:43.319 INFO:teuthology.orchestra.run.smithi052.stdout: python3-pyasn1 noarch 0.4.8-6.el9 appstream 159 k 2024-05-27T07:10:43.319 INFO:teuthology.orchestra.run.smithi052.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 appstream 279 k 2024-05-27T07:10:43.319 INFO:teuthology.orchestra.run.smithi052.stdout: python3-pycparser noarch 2.20-6.el9 baseos 135 k 2024-05-27T07:10:43.319 INFO:teuthology.orchestra.run.smithi052.stdout: python3-pyperclip noarch 1.8.2-2.el9 epel 28 k 2024-05-27T07:10:43.319 INFO:teuthology.orchestra.run.smithi052.stdout: python3-pytz noarch 2021.1-5.el9 appstream 51 k 2024-05-27T07:10:43.319 INFO:teuthology.orchestra.run.smithi052.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2024-05-27T07:10:43.319 INFO:teuthology.orchestra.run.smithi052.stdout: python3-requests noarch 2.25.1-8.el9 baseos 125 k 2024-05-27T07:10:43.319 INFO:teuthology.orchestra.run.smithi052.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 appstream 54 k 2024-05-27T07:10:43.319 INFO:teuthology.orchestra.run.smithi052.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2024-05-27T07:10:43.320 INFO:teuthology.orchestra.run.smithi052.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2024-05-27T07:10:43.320 INFO:teuthology.orchestra.run.smithi052.stdout: python3-scipy x86_64 1.6.2-8.el9 appstream 16 M 2024-05-27T07:10:43.320 INFO:teuthology.orchestra.run.smithi052.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2024-05-27T07:10:43.320 INFO:teuthology.orchestra.run.smithi052.stdout: python3-toml noarch 0.10.2-6.el9 appstream 42 k 2024-05-27T07:10:43.320 INFO:teuthology.orchestra.run.smithi052.stdout: python3-typing-extensions noarch 4.4.0-2.el9 epel 51 k 2024-05-27T07:10:43.320 INFO:teuthology.orchestra.run.smithi052.stdout: python3-urllib3 noarch 1.26.5-5.el9 baseos 215 k 2024-05-27T07:10:43.320 INFO:teuthology.orchestra.run.smithi052.stdout: python3-wcwidth noarch 0.2.5-8.el9 appstream 44 k 2024-05-27T07:10:43.321 INFO:teuthology.orchestra.run.smithi052.stdout: python3-webob noarch 1.8.7-6.el9 epel 230 k 2024-05-27T07:10:43.321 INFO:teuthology.orchestra.run.smithi052.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2024-05-27T07:10:43.321 INFO:teuthology.orchestra.run.smithi052.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2024-05-27T07:10:43.321 INFO:teuthology.orchestra.run.smithi052.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2024-05-27T07:10:43.321 INFO:teuthology.orchestra.run.smithi052.stdout: qatlib x86_64 24.02.0-1.el9 appstream 221 k 2024-05-27T07:10:43.321 INFO:teuthology.orchestra.run.smithi052.stdout: qatzip-libs x86_64 1.2.0-1.el9 appstream 47 k 2024-05-27T07:10:43.321 INFO:teuthology.orchestra.run.smithi052.stdout: re2 x86_64 1:20211101-3.el9 epel 192 k 2024-05-27T07:10:43.321 INFO:teuthology.orchestra.run.smithi052.stdout: socat x86_64 1.7.4.1-5.el9 appstream 305 k 2024-05-27T07:10:43.321 INFO:teuthology.orchestra.run.smithi052.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2024-05-27T07:10:43.321 INFO:teuthology.orchestra.run.smithi052.stdout: uuid x86_64 1.6.2-55.el9 appstream 58 k 2024-05-27T07:10:43.321 INFO:teuthology.orchestra.run.smithi052.stdout: xmlstarlet x86_64 1.6.1-20.el9 appstream 64 k 2024-05-27T07:10:43.321 INFO:teuthology.orchestra.run.smithi052.stdout: zlib-devel x86_64 1.2.11-41.el9 appstream 45 k 2024-05-27T07:10:43.321 INFO:teuthology.orchestra.run.smithi052.stdout:Installing weak dependencies: 2024-05-27T07:10:43.322 INFO:teuthology.orchestra.run.smithi052.stdout: qatlib-service x86_64 24.02.0-1.el9 appstream 36 k 2024-05-27T07:10:43.322 INFO:teuthology.orchestra.run.smithi052.stdout: 2024-05-27T07:10:43.322 INFO:teuthology.orchestra.run.smithi052.stdout:Transaction Summary 2024-05-27T07:10:43.322 INFO:teuthology.orchestra.run.smithi052.stdout:======================================================================================= 2024-05-27T07:10:43.322 INFO:teuthology.orchestra.run.smithi052.stdout:Install 148 Packages 2024-05-27T07:10:43.322 INFO:teuthology.orchestra.run.smithi052.stdout:Upgrade 11 Packages 2024-05-27T07:10:43.322 INFO:teuthology.orchestra.run.smithi052.stdout: 2024-05-27T07:10:43.323 INFO:teuthology.orchestra.run.smithi052.stdout:Total download size: 219 M 2024-05-27T07:10:43.323 INFO:teuthology.orchestra.run.smithi052.stdout:Downloading Packages: 2024-05-27T07:10:43.928 INFO:teuthology.orchestra.run.smithi173.stdout:lab-extras 30 kB/s | 1.7 kB 00:00 2024-05-27T07:10:44.280 INFO:teuthology.orchestra.run.smithi052.stdout:(1/159): ceph-19.0.0-3872.g38f9d900.el9.x86_64. 39 kB/s | 6.6 kB 00:00 2024-05-27T07:10:44.497 INFO:teuthology.orchestra.run.smithi052.stdout:(2/159): ceph-fuse-19.0.0-3872.g38f9d900.el9.x8 3.9 MB/s | 863 kB 00:00 2024-05-27T07:10:44.555 INFO:teuthology.orchestra.run.smithi052.stdout:(3/159): ceph-immutable-object-cache-19.0.0-387 2.5 MB/s | 147 kB 00:00 2024-05-27T07:10:44.697 INFO:teuthology.orchestra.run.smithi052.stdout:(4/159): ceph-base-19.0.0-3872.g38f9d900.el9.x8 9.2 MB/s | 5.4 MB 00:00 2024-05-27T07:10:44.781 INFO:teuthology.orchestra.run.smithi052.stdout:(5/159): ceph-mds-19.0.0-3872.g38f9d900.el9.x86 10 MB/s | 2.3 MB 00:00 2024-05-27T07:10:44.823 INFO:teuthology.orchestra.run.smithi052.stdout:(6/159): ceph-mgr-19.0.0-3872.g38f9d900.el9.x86 12 MB/s | 1.5 MB 00:00 2024-05-27T07:10:45.257 INFO:teuthology.orchestra.run.smithi052.stdout:(7/159): ceph-mon-19.0.0-3872.g38f9d900.el9.x86 10 MB/s | 4.9 MB 00:00 2024-05-27T07:10:45.314 INFO:teuthology.orchestra.run.smithi105.stdout:Package librados2-2:16.2.4-5.el9.x86_64 is already installed. 2024-05-27T07:10:45.315 INFO:teuthology.orchestra.run.smithi105.stdout:Package librbd1-2:16.2.4-5.el9.x86_64 is already installed. 2024-05-27T07:10:45.325 INFO:teuthology.orchestra.run.smithi105.stdout:Package NetworkManager-1:1.47.4-1.el9.x86_64 is already installed. 2024-05-27T07:10:45.326 INFO:teuthology.orchestra.run.smithi105.stdout:Package iproute-6.2.0-5.el9.x86_64 is already installed. 2024-05-27T07:10:45.327 INFO:teuthology.orchestra.run.smithi105.stdout:Package util-linux-2.37.4-18.el9.x86_64 is already installed. 2024-05-27T07:10:45.328 INFO:teuthology.orchestra.run.smithi105.stdout:Package libacl-devel-2.3.1-4.el9.x86_64 is already installed. 2024-05-27T07:10:45.329 INFO:teuthology.orchestra.run.smithi105.stdout:Package libattr-devel-2.5.1-3.el9.x86_64 is already installed. 2024-05-27T07:10:45.331 INFO:teuthology.orchestra.run.smithi105.stdout:Package xfsdump-3.1.12-4.el9.x86_64 is already installed. 2024-05-27T07:10:45.332 INFO:teuthology.orchestra.run.smithi105.stdout:Package xfsprogs-6.3.0-1.el9.x86_64 is already installed. 2024-05-27T07:10:45.395 INFO:teuthology.orchestra.run.smithi105.stdout:Dependencies resolved. 2024-05-27T07:10:45.409 INFO:teuthology.orchestra.run.smithi105.stdout:======================================================================================= 2024-05-27T07:10:45.409 INFO:teuthology.orchestra.run.smithi105.stdout: Package Arch Version Repository Size 2024-05-27T07:10:45.409 INFO:teuthology.orchestra.run.smithi105.stdout:======================================================================================= 2024-05-27T07:10:45.409 INFO:teuthology.orchestra.run.smithi105.stdout:Installing: 2024-05-27T07:10:45.409 INFO:teuthology.orchestra.run.smithi105.stdout: bison x86_64 3.7.4-5.el9 appstream 943 k 2024-05-27T07:10:45.410 INFO:teuthology.orchestra.run.smithi105.stdout: ceph x86_64 2:19.0.0-3872.g38f9d900.el9 ceph 6.6 k 2024-05-27T07:10:45.410 INFO:teuthology.orchestra.run.smithi105.stdout: ceph-base x86_64 2:19.0.0-3872.g38f9d900.el9 ceph 5.4 M 2024-05-27T07:10:45.410 INFO:teuthology.orchestra.run.smithi105.stdout: ceph-fuse x86_64 2:19.0.0-3872.g38f9d900.el9 ceph 863 k 2024-05-27T07:10:45.410 INFO:teuthology.orchestra.run.smithi105.stdout: ceph-immutable-object-cache x86_64 2:19.0.0-3872.g38f9d900.el9 ceph 147 k 2024-05-27T07:10:45.410 INFO:teuthology.orchestra.run.smithi105.stdout: ceph-mgr x86_64 2:19.0.0-3872.g38f9d900.el9 ceph 1.5 M 2024-05-27T07:10:45.410 INFO:teuthology.orchestra.run.smithi105.stdout: ceph-mgr-cephadm noarch 2:19.0.0-3872.g38f9d900.el9 ceph-noarch 143 k 2024-05-27T07:10:45.410 INFO:teuthology.orchestra.run.smithi105.stdout: ceph-mgr-dashboard noarch 2:19.0.0-3872.g38f9d900.el9 ceph-noarch 3.5 M 2024-05-27T07:10:45.410 INFO:teuthology.orchestra.run.smithi105.stdout: ceph-mgr-diskprediction-local noarch 2:19.0.0-3872.g38f9d900.el9 ceph-noarch 7.4 M 2024-05-27T07:10:45.410 INFO:teuthology.orchestra.run.smithi105.stdout: ceph-mgr-rook noarch 2:19.0.0-3872.g38f9d900.el9 ceph-noarch 49 k 2024-05-27T07:10:45.410 INFO:teuthology.orchestra.run.smithi105.stdout: ceph-radosgw x86_64 2:19.0.0-3872.g38f9d900.el9 ceph 10 M 2024-05-27T07:10:45.410 INFO:teuthology.orchestra.run.smithi105.stdout: ceph-test x86_64 2:19.0.0-3872.g38f9d900.el9 ceph 43 M 2024-05-27T07:10:45.410 INFO:teuthology.orchestra.run.smithi105.stdout: ceph-volume noarch 2:19.0.0-3872.g38f9d900.el9 ceph-noarch 273 k 2024-05-27T07:10:45.411 INFO:teuthology.orchestra.run.smithi105.stdout: cephadm noarch 2:19.0.0-3872.g38f9d900.el9 ceph-noarch 764 k 2024-05-27T07:10:45.411 INFO:teuthology.orchestra.run.smithi105.stdout: cephfs-mirror x86_64 2:19.0.0-3872.g38f9d900.el9 ceph 224 k 2024-05-27T07:10:45.411 INFO:teuthology.orchestra.run.smithi105.stdout: cephfs-shell x86_64 2:19.0.0-3872.g38f9d900.el9 ceph 27 k 2024-05-27T07:10:45.411 INFO:teuthology.orchestra.run.smithi105.stdout: cephfs-top noarch 2:19.0.0-3872.g38f9d900.el9 ceph-noarch 18 k 2024-05-27T07:10:45.411 INFO:teuthology.orchestra.run.smithi105.stdout: elfutils-libelf-devel x86_64 0.191-4.el9 appstream 22 k 2024-05-27T07:10:45.411 INFO:teuthology.orchestra.run.smithi105.stdout: flex x86_64 2.6.4-9.el9 appstream 313 k 2024-05-27T07:10:45.411 INFO:teuthology.orchestra.run.smithi105.stdout: libaio-devel x86_64 0.3.111-13.el9 appstream 11 k 2024-05-27T07:10:45.411 INFO:teuthology.orchestra.run.smithi105.stdout: libcephfs-devel x86_64 2:19.0.0-3872.g38f9d900.el9 ceph 32 k 2024-05-27T07:10:45.411 INFO:teuthology.orchestra.run.smithi105.stdout: libcephfs2 x86_64 2:19.0.0-3872.g38f9d900.el9 ceph 755 k 2024-05-27T07:10:45.411 INFO:teuthology.orchestra.run.smithi105.stdout: librados-devel x86_64 2:19.0.0-3872.g38f9d900.el9 ceph 127 k 2024-05-27T07:10:45.411 INFO:teuthology.orchestra.run.smithi105.stdout: libtool x86_64 2.4.6-45.el9 appstream 582 k 2024-05-27T07:10:45.411 INFO:teuthology.orchestra.run.smithi105.stdout: libuuid-devel x86_64 2.37.4-18.el9 appstream 25 k 2024-05-27T07:10:45.412 INFO:teuthology.orchestra.run.smithi105.stdout: openssl-devel x86_64 1:3.2.1-1.el9 appstream 4.4 M 2024-05-27T07:10:45.412 INFO:teuthology.orchestra.run.smithi105.stdout: postgresql x86_64 13.14-1.el9 appstream 1.6 M 2024-05-27T07:10:45.412 INFO:teuthology.orchestra.run.smithi105.stdout: postgresql-contrib x86_64 13.14-1.el9 appstream 911 k 2024-05-27T07:10:45.412 INFO:teuthology.orchestra.run.smithi105.stdout: postgresql-server x86_64 13.14-1.el9 appstream 5.8 M 2024-05-27T07:10:45.412 INFO:teuthology.orchestra.run.smithi105.stdout: python3-cephfs x86_64 2:19.0.0-3872.g38f9d900.el9 ceph 161 k 2024-05-27T07:10:45.412 INFO:teuthology.orchestra.run.smithi105.stdout: python3-rados x86_64 2:19.0.0-3872.g38f9d900.el9 ceph 320 k 2024-05-27T07:10:45.412 INFO:teuthology.orchestra.run.smithi105.stdout: python3-rbd x86_64 2:19.0.0-3872.g38f9d900.el9 ceph 300 k 2024-05-27T07:10:45.412 INFO:teuthology.orchestra.run.smithi105.stdout: python3-rgw x86_64 2:19.0.0-3872.g38f9d900.el9 ceph 99 k 2024-05-27T07:10:45.412 INFO:teuthology.orchestra.run.smithi105.stdout: rbd-fuse x86_64 2:19.0.0-3872.g38f9d900.el9 ceph 86 k 2024-05-27T07:10:45.412 INFO:teuthology.orchestra.run.smithi105.stdout: rbd-mirror x86_64 2:19.0.0-3872.g38f9d900.el9 ceph 3.0 M 2024-05-27T07:10:45.413 INFO:teuthology.orchestra.run.smithi105.stdout: rbd-nbd x86_64 2:19.0.0-3872.g38f9d900.el9 ceph 169 k 2024-05-27T07:10:45.413 INFO:teuthology.orchestra.run.smithi105.stdout: xfsprogs-devel x86_64 6.3.0-1.el9 appstream 94 k 2024-05-27T07:10:45.413 INFO:teuthology.orchestra.run.smithi105.stdout:Upgrading: 2024-05-27T07:10:45.413 INFO:teuthology.orchestra.run.smithi105.stdout: NetworkManager x86_64 1:1.47.90-1.el9 baseos 2.3 M 2024-05-27T07:10:45.413 INFO:teuthology.orchestra.run.smithi105.stdout: NetworkManager-libnm x86_64 1:1.47.90-1.el9 baseos 1.8 M 2024-05-27T07:10:45.413 INFO:teuthology.orchestra.run.smithi105.stdout: NetworkManager-team x86_64 1:1.47.90-1.el9 baseos 41 k 2024-05-27T07:10:45.413 INFO:teuthology.orchestra.run.smithi105.stdout: NetworkManager-tui x86_64 1:1.47.90-1.el9 baseos 248 k 2024-05-27T07:10:45.413 INFO:teuthology.orchestra.run.smithi105.stdout: elfutils-debuginfod-client x86_64 0.191-4.el9 baseos 37 k 2024-05-27T07:10:45.413 INFO:teuthology.orchestra.run.smithi105.stdout: elfutils-libelf x86_64 0.191-4.el9 baseos 207 k 2024-05-27T07:10:45.413 INFO:teuthology.orchestra.run.smithi105.stdout: elfutils-libs x86_64 0.191-4.el9 baseos 260 k 2024-05-27T07:10:45.413 INFO:teuthology.orchestra.run.smithi105.stdout: librados2 x86_64 2:19.0.0-3872.g38f9d900.el9 ceph 3.3 M 2024-05-27T07:10:45.413 INFO:teuthology.orchestra.run.smithi105.stdout: librbd1 x86_64 2:19.0.0-3872.g38f9d900.el9 ceph 3.1 M 2024-05-27T07:10:45.414 INFO:teuthology.orchestra.run.smithi105.stdout: openssl x86_64 1:3.2.1-1.el9 baseos 1.3 M 2024-05-27T07:10:45.414 INFO:teuthology.orchestra.run.smithi105.stdout: openssl-libs x86_64 1:3.2.1-1.el9 baseos 2.4 M 2024-05-27T07:10:45.414 INFO:teuthology.orchestra.run.smithi105.stdout:Installing dependencies: 2024-05-27T07:10:45.414 INFO:teuthology.orchestra.run.smithi105.stdout: abseil-cpp x86_64 20211102.0-3.el9 epel 548 k 2024-05-27T07:10:45.414 INFO:teuthology.orchestra.run.smithi105.stdout: automake noarch 1.16.2-8.el9 appstream 689 k 2024-05-27T07:10:45.414 INFO:teuthology.orchestra.run.smithi105.stdout: boost-program-options x86_64 1.75.0-8.el9 appstream 107 k 2024-05-27T07:10:45.414 INFO:teuthology.orchestra.run.smithi105.stdout: ceph-common x86_64 2:19.0.0-3872.g38f9d900.el9 ceph 20 M 2024-05-27T07:10:45.414 INFO:teuthology.orchestra.run.smithi105.stdout: ceph-grafana-dashboards noarch 2:19.0.0-3872.g38f9d900.el9 ceph-noarch 34 k 2024-05-27T07:10:45.414 INFO:teuthology.orchestra.run.smithi105.stdout: ceph-mds x86_64 2:19.0.0-3872.g38f9d900.el9 ceph 2.3 M 2024-05-27T07:10:45.414 INFO:teuthology.orchestra.run.smithi105.stdout: ceph-mgr-modules-core noarch 2:19.0.0-3872.g38f9d900.el9 ceph-noarch 274 k 2024-05-27T07:10:45.414 INFO:teuthology.orchestra.run.smithi105.stdout: ceph-mon x86_64 2:19.0.0-3872.g38f9d900.el9 ceph 4.9 M 2024-05-27T07:10:45.414 INFO:teuthology.orchestra.run.smithi105.stdout: ceph-osd x86_64 2:19.0.0-3872.g38f9d900.el9 ceph 18 M 2024-05-27T07:10:45.415 INFO:teuthology.orchestra.run.smithi105.stdout: ceph-prometheus-alerts noarch 2:19.0.0-3872.g38f9d900.el9 ceph-noarch 16 k 2024-05-27T07:10:45.415 INFO:teuthology.orchestra.run.smithi105.stdout: ceph-selinux x86_64 2:19.0.0-3872.g38f9d900.el9 ceph 25 k 2024-05-27T07:10:45.415 INFO:teuthology.orchestra.run.smithi105.stdout: flexiblas x86_64 3.0.4-8.el9 appstream 33 k 2024-05-27T07:10:45.415 INFO:teuthology.orchestra.run.smithi105.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 appstream 3.0 M 2024-05-27T07:10:45.415 INFO:teuthology.orchestra.run.smithi105.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 appstream 18 k 2024-05-27T07:10:45.415 INFO:teuthology.orchestra.run.smithi105.stdout: fuse x86_64 2.9.9-15.el9 baseos 80 k 2024-05-27T07:10:45.415 INFO:teuthology.orchestra.run.smithi105.stdout: gperftools-libs x86_64 2.9.1-2.el9 epel 309 k 2024-05-27T07:10:45.415 INFO:teuthology.orchestra.run.smithi105.stdout: grpc-data noarch 1.46.7-10.el9 epel 19 k 2024-05-27T07:10:45.415 INFO:teuthology.orchestra.run.smithi105.stdout: libarrow x86_64 9.0.0-11.el9 epel 4.4 M 2024-05-27T07:10:45.415 INFO:teuthology.orchestra.run.smithi105.stdout: libarrow-doc noarch 9.0.0-11.el9 epel 26 k 2024-05-27T07:10:45.415 INFO:teuthology.orchestra.run.smithi105.stdout: libcephsqlite x86_64 2:19.0.0-3872.g38f9d900.el9 ceph 157 k 2024-05-27T07:10:45.415 INFO:teuthology.orchestra.run.smithi105.stdout: libgfortran x86_64 11.4.1-3.el9 baseos 803 k 2024-05-27T07:10:45.415 INFO:teuthology.orchestra.run.smithi105.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2024-05-27T07:10:45.416 INFO:teuthology.orchestra.run.smithi105.stdout: libpmemobj x86_64 1.12.1-1.el9 appstream 160 k 2024-05-27T07:10:45.416 INFO:teuthology.orchestra.run.smithi105.stdout: libquadmath x86_64 11.4.1-3.el9 baseos 195 k 2024-05-27T07:10:45.416 INFO:teuthology.orchestra.run.smithi105.stdout: librabbitmq x86_64 0.11.0-7.el9 appstream 45 k 2024-05-27T07:10:45.416 INFO:teuthology.orchestra.run.smithi105.stdout: libradosstriper1 x86_64 2:19.0.0-3872.g38f9d900.el9 ceph 498 k 2024-05-27T07:10:45.416 INFO:teuthology.orchestra.run.smithi105.stdout: librdkafka x86_64 1.6.1-102.el9 appstream 662 k 2024-05-27T07:10:45.416 INFO:teuthology.orchestra.run.smithi105.stdout: librgw2 x86_64 2:19.0.0-3872.g38f9d900.el9 ceph 5.2 M 2024-05-27T07:10:45.416 INFO:teuthology.orchestra.run.smithi105.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2024-05-27T07:10:45.416 INFO:teuthology.orchestra.run.smithi105.stdout: libzstd-devel x86_64 1.5.1-2.el9 appstream 47 k 2024-05-27T07:10:45.416 INFO:teuthology.orchestra.run.smithi105.stdout: lttng-ust x86_64 2.12.0-6.el9 appstream 292 k 2024-05-27T07:10:45.416 INFO:teuthology.orchestra.run.smithi105.stdout: lua x86_64 5.4.4-4.el9 appstream 188 k 2024-05-27T07:10:45.416 INFO:teuthology.orchestra.run.smithi105.stdout: lua-devel x86_64 5.4.4-4.el9 crb 22 k 2024-05-27T07:10:45.416 INFO:teuthology.orchestra.run.smithi105.stdout: luarocks noarch 3.9.2-5.el9 epel 151 k 2024-05-27T07:10:45.416 INFO:teuthology.orchestra.run.smithi105.stdout: openblas x86_64 0.3.21-2.el9 appstream 35 k 2024-05-27T07:10:45.417 INFO:teuthology.orchestra.run.smithi105.stdout: openblas-openmp x86_64 0.3.21-2.el9 appstream 4.7 M 2024-05-27T07:10:45.417 INFO:teuthology.orchestra.run.smithi105.stdout: parquet-libs x86_64 9.0.0-11.el9 epel 839 k 2024-05-27T07:10:45.417 INFO:teuthology.orchestra.run.smithi105.stdout: postgresql-private-libs x86_64 13.14-1.el9 appstream 133 k 2024-05-27T07:10:45.417 INFO:teuthology.orchestra.run.smithi105.stdout: protobuf x86_64 3.14.0-13.el9 appstream 1.0 M 2024-05-27T07:10:45.417 INFO:teuthology.orchestra.run.smithi105.stdout: protobuf-compiler x86_64 3.14.0-13.el9 crb 863 k 2024-05-27T07:10:45.417 INFO:teuthology.orchestra.run.smithi105.stdout: python3-asyncssh noarch 2.13.2-5.el9 epel 548 k 2024-05-27T07:10:45.417 INFO:teuthology.orchestra.run.smithi105.stdout: python3-attrs noarch 20.3.0-7.el9 appstream 87 k 2024-05-27T07:10:45.417 INFO:teuthology.orchestra.run.smithi105.stdout: python3-babel noarch 2.9.1-2.el9 appstream 6.0 M 2024-05-27T07:10:45.417 INFO:teuthology.orchestra.run.smithi105.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2024-05-27T07:10:45.417 INFO:teuthology.orchestra.run.smithi105.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2024-05-27T07:10:45.417 INFO:teuthology.orchestra.run.smithi105.stdout: python3-ceph-argparse x86_64 2:19.0.0-3872.g38f9d900.el9 ceph 45 k 2024-05-27T07:10:45.418 INFO:teuthology.orchestra.run.smithi105.stdout: python3-ceph-common x86_64 2:19.0.0-3872.g38f9d900.el9 ceph 136 k 2024-05-27T07:10:45.418 INFO:teuthology.orchestra.run.smithi105.stdout: python3-certifi noarch 2021.10.8-2.el9 epel 15 k 2024-05-27T07:10:45.418 INFO:teuthology.orchestra.run.smithi105.stdout: python3-cffi x86_64 1.14.5-5.el9 baseos 253 k 2024-05-27T07:10:45.418 INFO:teuthology.orchestra.run.smithi105.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2024-05-27T07:10:45.418 INFO:teuthology.orchestra.run.smithi105.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2024-05-27T07:10:45.418 INFO:teuthology.orchestra.run.smithi105.stdout: python3-cmd2 noarch 2.4.2-2.el9 epel 303 k 2024-05-27T07:10:45.418 INFO:teuthology.orchestra.run.smithi105.stdout: python3-colorama noarch 0.4.6-3.el9 appstream 61 k 2024-05-27T07:10:45.418 INFO:teuthology.orchestra.run.smithi105.stdout: python3-cryptography x86_64 36.0.1-4.el9 baseos 1.2 M 2024-05-27T07:10:45.418 INFO:teuthology.orchestra.run.smithi105.stdout: python3-devel x86_64 3.9.18-3.el9 appstream 244 k 2024-05-27T07:10:45.418 INFO:teuthology.orchestra.run.smithi105.stdout: python3-google-auth noarch 1:2.29.0-1.el9 epel 206 k 2024-05-27T07:10:45.418 INFO:teuthology.orchestra.run.smithi105.stdout: python3-grpcio x86_64 1.46.7-10.el9 epel 2.0 M 2024-05-27T07:10:45.418 INFO:teuthology.orchestra.run.smithi105.stdout: python3-grpcio-tools x86_64 1.46.7-10.el9 epel 144 k 2024-05-27T07:10:45.418 INFO:teuthology.orchestra.run.smithi105.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2024-05-27T07:10:45.419 INFO:teuthology.orchestra.run.smithi105.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2024-05-27T07:10:45.419 INFO:teuthology.orchestra.run.smithi105.stdout: python3-jaraco-collections noarch 3.0.0-8.el9 epel 23 k 2024-05-27T07:10:45.419 INFO:teuthology.orchestra.run.smithi105.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2024-05-27T07:10:45.419 INFO:teuthology.orchestra.run.smithi105.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2024-05-27T07:10:45.419 INFO:teuthology.orchestra.run.smithi105.stdout: python3-jinja2 noarch 2.11.3-6.el9 appstream 248 k 2024-05-27T07:10:45.419 INFO:teuthology.orchestra.run.smithi105.stdout: python3-jsonpatch noarch 1.21-16.el9 appstream 26 k 2024-05-27T07:10:45.419 INFO:teuthology.orchestra.run.smithi105.stdout: python3-jsonpointer noarch 2.0-4.el9 appstream 19 k 2024-05-27T07:10:45.419 INFO:teuthology.orchestra.run.smithi105.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2024-05-27T07:10:45.419 INFO:teuthology.orchestra.run.smithi105.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2024-05-27T07:10:45.419 INFO:teuthology.orchestra.run.smithi105.stdout: python3-mako noarch 1.1.4-6.el9 appstream 172 k 2024-05-27T07:10:45.419 INFO:teuthology.orchestra.run.smithi105.stdout: python3-markupsafe x86_64 1.1.1-12.el9 appstream 35 k 2024-05-27T07:10:45.419 INFO:teuthology.orchestra.run.smithi105.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2024-05-27T07:10:45.419 INFO:teuthology.orchestra.run.smithi105.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2024-05-27T07:10:45.419 INFO:teuthology.orchestra.run.smithi105.stdout: python3-numpy x86_64 1:1.20.1-5.el9 appstream 5.1 M 2024-05-27T07:10:45.420 INFO:teuthology.orchestra.run.smithi105.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 appstream 250 k 2024-05-27T07:10:45.420 INFO:teuthology.orchestra.run.smithi105.stdout: python3-oauthlib noarch 3.1.1-5.el9 appstream 222 k 2024-05-27T07:10:45.420 INFO:teuthology.orchestra.run.smithi105.stdout: python3-packaging noarch 20.9-5.el9 appstream 77 k 2024-05-27T07:10:45.420 INFO:teuthology.orchestra.run.smithi105.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2024-05-27T07:10:45.420 INFO:teuthology.orchestra.run.smithi105.stdout: python3-ply noarch 3.11-14.el9 baseos 106 k 2024-05-27T07:10:45.420 INFO:teuthology.orchestra.run.smithi105.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2024-05-27T07:10:45.420 INFO:teuthology.orchestra.run.smithi105.stdout: python3-prettytable noarch 0.7.2-27.el9 appstream 42 k 2024-05-27T07:10:45.420 INFO:teuthology.orchestra.run.smithi105.stdout: python3-protobuf noarch 3.14.0-13.el9 appstream 269 k 2024-05-27T07:10:45.420 INFO:teuthology.orchestra.run.smithi105.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2024-05-27T07:10:45.420 INFO:teuthology.orchestra.run.smithi105.stdout: python3-pyasn1 noarch 0.4.8-6.el9 appstream 159 k 2024-05-27T07:10:45.420 INFO:teuthology.orchestra.run.smithi105.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 appstream 279 k 2024-05-27T07:10:45.420 INFO:teuthology.orchestra.run.smithi105.stdout: python3-pycparser noarch 2.20-6.el9 baseos 135 k 2024-05-27T07:10:45.420 INFO:teuthology.orchestra.run.smithi105.stdout: python3-pyperclip noarch 1.8.2-2.el9 epel 28 k 2024-05-27T07:10:45.420 INFO:teuthology.orchestra.run.smithi105.stdout: python3-pytz noarch 2021.1-5.el9 appstream 51 k 2024-05-27T07:10:45.421 INFO:teuthology.orchestra.run.smithi105.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2024-05-27T07:10:45.421 INFO:teuthology.orchestra.run.smithi105.stdout: python3-requests noarch 2.25.1-8.el9 baseos 125 k 2024-05-27T07:10:45.421 INFO:teuthology.orchestra.run.smithi105.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 appstream 54 k 2024-05-27T07:10:45.421 INFO:teuthology.orchestra.run.smithi105.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2024-05-27T07:10:45.421 INFO:teuthology.orchestra.run.smithi105.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2024-05-27T07:10:45.421 INFO:teuthology.orchestra.run.smithi105.stdout: python3-scipy x86_64 1.6.2-8.el9 appstream 16 M 2024-05-27T07:10:45.421 INFO:teuthology.orchestra.run.smithi105.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2024-05-27T07:10:45.421 INFO:teuthology.orchestra.run.smithi105.stdout: python3-toml noarch 0.10.2-6.el9 appstream 42 k 2024-05-27T07:10:45.421 INFO:teuthology.orchestra.run.smithi105.stdout: python3-typing-extensions noarch 4.4.0-2.el9 epel 51 k 2024-05-27T07:10:45.421 INFO:teuthology.orchestra.run.smithi105.stdout: python3-urllib3 noarch 1.26.5-5.el9 baseos 215 k 2024-05-27T07:10:45.421 INFO:teuthology.orchestra.run.smithi105.stdout: python3-wcwidth noarch 0.2.5-8.el9 appstream 44 k 2024-05-27T07:10:45.421 INFO:teuthology.orchestra.run.smithi105.stdout: python3-webob noarch 1.8.7-6.el9 epel 230 k 2024-05-27T07:10:45.421 INFO:teuthology.orchestra.run.smithi105.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2024-05-27T07:10:45.421 INFO:teuthology.orchestra.run.smithi105.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2024-05-27T07:10:45.421 INFO:teuthology.orchestra.run.smithi105.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2024-05-27T07:10:45.422 INFO:teuthology.orchestra.run.smithi105.stdout: qatlib x86_64 24.02.0-1.el9 appstream 221 k 2024-05-27T07:10:45.422 INFO:teuthology.orchestra.run.smithi105.stdout: qatzip-libs x86_64 1.2.0-1.el9 appstream 47 k 2024-05-27T07:10:45.422 INFO:teuthology.orchestra.run.smithi105.stdout: re2 x86_64 1:20211101-3.el9 epel 192 k 2024-05-27T07:10:45.422 INFO:teuthology.orchestra.run.smithi105.stdout: socat x86_64 1.7.4.1-5.el9 appstream 305 k 2024-05-27T07:10:45.422 INFO:teuthology.orchestra.run.smithi105.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2024-05-27T07:10:45.422 INFO:teuthology.orchestra.run.smithi105.stdout: uuid x86_64 1.6.2-55.el9 appstream 58 k 2024-05-27T07:10:45.422 INFO:teuthology.orchestra.run.smithi105.stdout: xmlstarlet x86_64 1.6.1-20.el9 appstream 64 k 2024-05-27T07:10:45.422 INFO:teuthology.orchestra.run.smithi105.stdout: zlib-devel x86_64 1.2.11-41.el9 appstream 45 k 2024-05-27T07:10:45.422 INFO:teuthology.orchestra.run.smithi105.stdout:Installing weak dependencies: 2024-05-27T07:10:45.422 INFO:teuthology.orchestra.run.smithi105.stdout: qatlib-service x86_64 24.02.0-1.el9 appstream 36 k 2024-05-27T07:10:45.422 INFO:teuthology.orchestra.run.smithi105.stdout: 2024-05-27T07:10:45.422 INFO:teuthology.orchestra.run.smithi105.stdout:Transaction Summary 2024-05-27T07:10:45.422 INFO:teuthology.orchestra.run.smithi105.stdout:======================================================================================= 2024-05-27T07:10:45.423 INFO:teuthology.orchestra.run.smithi105.stdout:Install 148 Packages 2024-05-27T07:10:45.423 INFO:teuthology.orchestra.run.smithi105.stdout:Upgrade 11 Packages 2024-05-27T07:10:45.423 INFO:teuthology.orchestra.run.smithi105.stdout: 2024-05-27T07:10:45.423 INFO:teuthology.orchestra.run.smithi105.stdout:Total download size: 219 M 2024-05-27T07:10:45.423 INFO:teuthology.orchestra.run.smithi105.stdout:Downloading Packages: 2024-05-27T07:10:46.057 INFO:teuthology.orchestra.run.smithi052.stdout:(8/159): ceph-osd-19.0.0-3872.g38f9d900.el9.x86 14 MB/s | 18 MB 00:01 2024-05-27T07:10:46.062 INFO:teuthology.orchestra.run.smithi173.stdout:Package librados2-2:16.2.4-5.el9.x86_64 is already installed. 2024-05-27T07:10:46.063 INFO:teuthology.orchestra.run.smithi173.stdout:Package librbd1-2:16.2.4-5.el9.x86_64 is already installed. 2024-05-27T07:10:46.074 INFO:teuthology.orchestra.run.smithi173.stdout:Package NetworkManager-1:1.47.4-1.el9.x86_64 is already installed. 2024-05-27T07:10:46.075 INFO:teuthology.orchestra.run.smithi173.stdout:Package iproute-6.2.0-5.el9.x86_64 is already installed. 2024-05-27T07:10:46.075 INFO:teuthology.orchestra.run.smithi173.stdout:Package util-linux-2.37.4-18.el9.x86_64 is already installed. 2024-05-27T07:10:46.076 INFO:teuthology.orchestra.run.smithi173.stdout:Package libacl-devel-2.3.1-4.el9.x86_64 is already installed. 2024-05-27T07:10:46.078 INFO:teuthology.orchestra.run.smithi173.stdout:Package libattr-devel-2.5.1-3.el9.x86_64 is already installed. 2024-05-27T07:10:46.080 INFO:teuthology.orchestra.run.smithi173.stdout:Package xfsdump-3.1.12-4.el9.x86_64 is already installed. 2024-05-27T07:10:46.081 INFO:teuthology.orchestra.run.smithi173.stdout:Package xfsprogs-6.3.0-1.el9.x86_64 is already installed. 2024-05-27T07:10:46.108 INFO:teuthology.orchestra.run.smithi052.stdout:(9/159): ceph-selinux-19.0.0-3872.g38f9d900.el9 505 kB/s | 25 kB 00:00 2024-05-27T07:10:46.142 INFO:teuthology.orchestra.run.smithi173.stdout:Dependencies resolved. 2024-05-27T07:10:46.163 INFO:teuthology.orchestra.run.smithi173.stdout:======================================================================================= 2024-05-27T07:10:46.163 INFO:teuthology.orchestra.run.smithi173.stdout: Package Arch Version Repository Size 2024-05-27T07:10:46.163 INFO:teuthology.orchestra.run.smithi173.stdout:======================================================================================= 2024-05-27T07:10:46.163 INFO:teuthology.orchestra.run.smithi173.stdout:Installing: 2024-05-27T07:10:46.163 INFO:teuthology.orchestra.run.smithi173.stdout: bison x86_64 3.7.4-5.el9 appstream 943 k 2024-05-27T07:10:46.163 INFO:teuthology.orchestra.run.smithi173.stdout: ceph x86_64 2:19.0.0-3872.g38f9d900.el9 ceph 6.6 k 2024-05-27T07:10:46.163 INFO:teuthology.orchestra.run.smithi173.stdout: ceph-base x86_64 2:19.0.0-3872.g38f9d900.el9 ceph 5.4 M 2024-05-27T07:10:46.163 INFO:teuthology.orchestra.run.smithi173.stdout: ceph-fuse x86_64 2:19.0.0-3872.g38f9d900.el9 ceph 863 k 2024-05-27T07:10:46.163 INFO:teuthology.orchestra.run.smithi173.stdout: ceph-immutable-object-cache x86_64 2:19.0.0-3872.g38f9d900.el9 ceph 147 k 2024-05-27T07:10:46.163 INFO:teuthology.orchestra.run.smithi173.stdout: ceph-mgr x86_64 2:19.0.0-3872.g38f9d900.el9 ceph 1.5 M 2024-05-27T07:10:46.163 INFO:teuthology.orchestra.run.smithi173.stdout: ceph-mgr-cephadm noarch 2:19.0.0-3872.g38f9d900.el9 ceph-noarch 143 k 2024-05-27T07:10:46.164 INFO:teuthology.orchestra.run.smithi173.stdout: ceph-mgr-dashboard noarch 2:19.0.0-3872.g38f9d900.el9 ceph-noarch 3.5 M 2024-05-27T07:10:46.164 INFO:teuthology.orchestra.run.smithi173.stdout: ceph-mgr-diskprediction-local noarch 2:19.0.0-3872.g38f9d900.el9 ceph-noarch 7.4 M 2024-05-27T07:10:46.164 INFO:teuthology.orchestra.run.smithi173.stdout: ceph-mgr-rook noarch 2:19.0.0-3872.g38f9d900.el9 ceph-noarch 49 k 2024-05-27T07:10:46.164 INFO:teuthology.orchestra.run.smithi173.stdout: ceph-radosgw x86_64 2:19.0.0-3872.g38f9d900.el9 ceph 10 M 2024-05-27T07:10:46.164 INFO:teuthology.orchestra.run.smithi173.stdout: ceph-test x86_64 2:19.0.0-3872.g38f9d900.el9 ceph 43 M 2024-05-27T07:10:46.164 INFO:teuthology.orchestra.run.smithi173.stdout: ceph-volume noarch 2:19.0.0-3872.g38f9d900.el9 ceph-noarch 273 k 2024-05-27T07:10:46.164 INFO:teuthology.orchestra.run.smithi173.stdout: cephadm noarch 2:19.0.0-3872.g38f9d900.el9 ceph-noarch 764 k 2024-05-27T07:10:46.164 INFO:teuthology.orchestra.run.smithi173.stdout: cephfs-mirror x86_64 2:19.0.0-3872.g38f9d900.el9 ceph 224 k 2024-05-27T07:10:46.164 INFO:teuthology.orchestra.run.smithi173.stdout: cephfs-shell x86_64 2:19.0.0-3872.g38f9d900.el9 ceph 27 k 2024-05-27T07:10:46.164 INFO:teuthology.orchestra.run.smithi173.stdout: cephfs-top noarch 2:19.0.0-3872.g38f9d900.el9 ceph-noarch 18 k 2024-05-27T07:10:46.164 INFO:teuthology.orchestra.run.smithi173.stdout: elfutils-libelf-devel x86_64 0.191-4.el9 appstream 22 k 2024-05-27T07:10:46.164 INFO:teuthology.orchestra.run.smithi173.stdout: flex x86_64 2.6.4-9.el9 appstream 313 k 2024-05-27T07:10:46.165 INFO:teuthology.orchestra.run.smithi173.stdout: libaio-devel x86_64 0.3.111-13.el9 appstream 11 k 2024-05-27T07:10:46.165 INFO:teuthology.orchestra.run.smithi173.stdout: libcephfs-devel x86_64 2:19.0.0-3872.g38f9d900.el9 ceph 32 k 2024-05-27T07:10:46.165 INFO:teuthology.orchestra.run.smithi173.stdout: libcephfs2 x86_64 2:19.0.0-3872.g38f9d900.el9 ceph 755 k 2024-05-27T07:10:46.165 INFO:teuthology.orchestra.run.smithi173.stdout: librados-devel x86_64 2:19.0.0-3872.g38f9d900.el9 ceph 127 k 2024-05-27T07:10:46.165 INFO:teuthology.orchestra.run.smithi173.stdout: libtool x86_64 2.4.6-45.el9 appstream 582 k 2024-05-27T07:10:46.165 INFO:teuthology.orchestra.run.smithi173.stdout: libuuid-devel x86_64 2.37.4-18.el9 appstream 25 k 2024-05-27T07:10:46.165 INFO:teuthology.orchestra.run.smithi173.stdout: openssl-devel x86_64 1:3.2.1-1.el9 appstream 4.4 M 2024-05-27T07:10:46.165 INFO:teuthology.orchestra.run.smithi173.stdout: postgresql x86_64 13.14-1.el9 appstream 1.6 M 2024-05-27T07:10:46.165 INFO:teuthology.orchestra.run.smithi173.stdout: postgresql-contrib x86_64 13.14-1.el9 appstream 911 k 2024-05-27T07:10:46.165 INFO:teuthology.orchestra.run.smithi173.stdout: postgresql-server x86_64 13.14-1.el9 appstream 5.8 M 2024-05-27T07:10:46.165 INFO:teuthology.orchestra.run.smithi173.stdout: python3-cephfs x86_64 2:19.0.0-3872.g38f9d900.el9 ceph 161 k 2024-05-27T07:10:46.165 INFO:teuthology.orchestra.run.smithi173.stdout: python3-rados x86_64 2:19.0.0-3872.g38f9d900.el9 ceph 320 k 2024-05-27T07:10:46.165 INFO:teuthology.orchestra.run.smithi173.stdout: python3-rbd x86_64 2:19.0.0-3872.g38f9d900.el9 ceph 300 k 2024-05-27T07:10:46.166 INFO:teuthology.orchestra.run.smithi173.stdout: python3-rgw x86_64 2:19.0.0-3872.g38f9d900.el9 ceph 99 k 2024-05-27T07:10:46.166 INFO:teuthology.orchestra.run.smithi173.stdout: rbd-fuse x86_64 2:19.0.0-3872.g38f9d900.el9 ceph 86 k 2024-05-27T07:10:46.166 INFO:teuthology.orchestra.run.smithi173.stdout: rbd-mirror x86_64 2:19.0.0-3872.g38f9d900.el9 ceph 3.0 M 2024-05-27T07:10:46.166 INFO:teuthology.orchestra.run.smithi173.stdout: rbd-nbd x86_64 2:19.0.0-3872.g38f9d900.el9 ceph 169 k 2024-05-27T07:10:46.166 INFO:teuthology.orchestra.run.smithi173.stdout: xfsprogs-devel x86_64 6.3.0-1.el9 appstream 94 k 2024-05-27T07:10:46.166 INFO:teuthology.orchestra.run.smithi173.stdout:Upgrading: 2024-05-27T07:10:46.166 INFO:teuthology.orchestra.run.smithi173.stdout: NetworkManager x86_64 1:1.47.90-1.el9 baseos 2.3 M 2024-05-27T07:10:46.166 INFO:teuthology.orchestra.run.smithi173.stdout: NetworkManager-libnm x86_64 1:1.47.90-1.el9 baseos 1.8 M 2024-05-27T07:10:46.166 INFO:teuthology.orchestra.run.smithi173.stdout: NetworkManager-team x86_64 1:1.47.90-1.el9 baseos 41 k 2024-05-27T07:10:46.166 INFO:teuthology.orchestra.run.smithi173.stdout: NetworkManager-tui x86_64 1:1.47.90-1.el9 baseos 248 k 2024-05-27T07:10:46.166 INFO:teuthology.orchestra.run.smithi173.stdout: elfutils-debuginfod-client x86_64 0.191-4.el9 baseos 37 k 2024-05-27T07:10:46.166 INFO:teuthology.orchestra.run.smithi173.stdout: elfutils-libelf x86_64 0.191-4.el9 baseos 207 k 2024-05-27T07:10:46.167 INFO:teuthology.orchestra.run.smithi173.stdout: elfutils-libs x86_64 0.191-4.el9 baseos 260 k 2024-05-27T07:10:46.167 INFO:teuthology.orchestra.run.smithi173.stdout: librados2 x86_64 2:19.0.0-3872.g38f9d900.el9 ceph 3.3 M 2024-05-27T07:10:46.167 INFO:teuthology.orchestra.run.smithi173.stdout: librbd1 x86_64 2:19.0.0-3872.g38f9d900.el9 ceph 3.1 M 2024-05-27T07:10:46.167 INFO:teuthology.orchestra.run.smithi173.stdout: openssl x86_64 1:3.2.1-1.el9 baseos 1.3 M 2024-05-27T07:10:46.167 INFO:teuthology.orchestra.run.smithi173.stdout: openssl-libs x86_64 1:3.2.1-1.el9 baseos 2.4 M 2024-05-27T07:10:46.167 INFO:teuthology.orchestra.run.smithi173.stdout:Installing dependencies: 2024-05-27T07:10:46.167 INFO:teuthology.orchestra.run.smithi173.stdout: abseil-cpp x86_64 20211102.0-3.el9 epel 548 k 2024-05-27T07:10:46.167 INFO:teuthology.orchestra.run.smithi173.stdout: automake noarch 1.16.2-8.el9 appstream 689 k 2024-05-27T07:10:46.167 INFO:teuthology.orchestra.run.smithi173.stdout: boost-program-options x86_64 1.75.0-8.el9 appstream 107 k 2024-05-27T07:10:46.167 INFO:teuthology.orchestra.run.smithi173.stdout: ceph-common x86_64 2:19.0.0-3872.g38f9d900.el9 ceph 20 M 2024-05-27T07:10:46.167 INFO:teuthology.orchestra.run.smithi173.stdout: ceph-grafana-dashboards noarch 2:19.0.0-3872.g38f9d900.el9 ceph-noarch 34 k 2024-05-27T07:10:46.167 INFO:teuthology.orchestra.run.smithi173.stdout: ceph-mds x86_64 2:19.0.0-3872.g38f9d900.el9 ceph 2.3 M 2024-05-27T07:10:46.168 INFO:teuthology.orchestra.run.smithi173.stdout: ceph-mgr-modules-core noarch 2:19.0.0-3872.g38f9d900.el9 ceph-noarch 274 k 2024-05-27T07:10:46.168 INFO:teuthology.orchestra.run.smithi173.stdout: ceph-mon x86_64 2:19.0.0-3872.g38f9d900.el9 ceph 4.9 M 2024-05-27T07:10:46.168 INFO:teuthology.orchestra.run.smithi173.stdout: ceph-osd x86_64 2:19.0.0-3872.g38f9d900.el9 ceph 18 M 2024-05-27T07:10:46.168 INFO:teuthology.orchestra.run.smithi173.stdout: ceph-prometheus-alerts noarch 2:19.0.0-3872.g38f9d900.el9 ceph-noarch 16 k 2024-05-27T07:10:46.168 INFO:teuthology.orchestra.run.smithi173.stdout: ceph-selinux x86_64 2:19.0.0-3872.g38f9d900.el9 ceph 25 k 2024-05-27T07:10:46.168 INFO:teuthology.orchestra.run.smithi173.stdout: flexiblas x86_64 3.0.4-8.el9 appstream 33 k 2024-05-27T07:10:46.168 INFO:teuthology.orchestra.run.smithi173.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 appstream 3.0 M 2024-05-27T07:10:46.168 INFO:teuthology.orchestra.run.smithi173.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 appstream 18 k 2024-05-27T07:10:46.168 INFO:teuthology.orchestra.run.smithi173.stdout: fuse x86_64 2.9.9-15.el9 baseos 80 k 2024-05-27T07:10:46.168 INFO:teuthology.orchestra.run.smithi173.stdout: gperftools-libs x86_64 2.9.1-2.el9 epel 309 k 2024-05-27T07:10:46.168 INFO:teuthology.orchestra.run.smithi173.stdout: grpc-data noarch 1.46.7-10.el9 epel 19 k 2024-05-27T07:10:46.168 INFO:teuthology.orchestra.run.smithi173.stdout: libarrow x86_64 9.0.0-11.el9 epel 4.4 M 2024-05-27T07:10:46.168 INFO:teuthology.orchestra.run.smithi173.stdout: libarrow-doc noarch 9.0.0-11.el9 epel 26 k 2024-05-27T07:10:46.169 INFO:teuthology.orchestra.run.smithi173.stdout: libcephsqlite x86_64 2:19.0.0-3872.g38f9d900.el9 ceph 157 k 2024-05-27T07:10:46.169 INFO:teuthology.orchestra.run.smithi173.stdout: libgfortran x86_64 11.4.1-3.el9 baseos 803 k 2024-05-27T07:10:46.169 INFO:teuthology.orchestra.run.smithi173.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2024-05-27T07:10:46.169 INFO:teuthology.orchestra.run.smithi173.stdout: libpmemobj x86_64 1.12.1-1.el9 appstream 160 k 2024-05-27T07:10:46.169 INFO:teuthology.orchestra.run.smithi173.stdout: libquadmath x86_64 11.4.1-3.el9 baseos 195 k 2024-05-27T07:10:46.169 INFO:teuthology.orchestra.run.smithi173.stdout: librabbitmq x86_64 0.11.0-7.el9 appstream 45 k 2024-05-27T07:10:46.169 INFO:teuthology.orchestra.run.smithi173.stdout: libradosstriper1 x86_64 2:19.0.0-3872.g38f9d900.el9 ceph 498 k 2024-05-27T07:10:46.169 INFO:teuthology.orchestra.run.smithi173.stdout: librdkafka x86_64 1.6.1-102.el9 appstream 662 k 2024-05-27T07:10:46.169 INFO:teuthology.orchestra.run.smithi173.stdout: librgw2 x86_64 2:19.0.0-3872.g38f9d900.el9 ceph 5.2 M 2024-05-27T07:10:46.169 INFO:teuthology.orchestra.run.smithi173.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2024-05-27T07:10:46.169 INFO:teuthology.orchestra.run.smithi173.stdout: libzstd-devel x86_64 1.5.1-2.el9 appstream 47 k 2024-05-27T07:10:46.169 INFO:teuthology.orchestra.run.smithi173.stdout: lttng-ust x86_64 2.12.0-6.el9 appstream 292 k 2024-05-27T07:10:46.169 INFO:teuthology.orchestra.run.smithi173.stdout: lua x86_64 5.4.4-4.el9 appstream 188 k 2024-05-27T07:10:46.170 INFO:teuthology.orchestra.run.smithi173.stdout: lua-devel x86_64 5.4.4-4.el9 crb 22 k 2024-05-27T07:10:46.170 INFO:teuthology.orchestra.run.smithi173.stdout: luarocks noarch 3.9.2-5.el9 epel 151 k 2024-05-27T07:10:46.170 INFO:teuthology.orchestra.run.smithi173.stdout: openblas x86_64 0.3.21-2.el9 appstream 35 k 2024-05-27T07:10:46.170 INFO:teuthology.orchestra.run.smithi173.stdout: openblas-openmp x86_64 0.3.21-2.el9 appstream 4.7 M 2024-05-27T07:10:46.170 INFO:teuthology.orchestra.run.smithi173.stdout: parquet-libs x86_64 9.0.0-11.el9 epel 839 k 2024-05-27T07:10:46.170 INFO:teuthology.orchestra.run.smithi173.stdout: postgresql-private-libs x86_64 13.14-1.el9 appstream 133 k 2024-05-27T07:10:46.170 INFO:teuthology.orchestra.run.smithi173.stdout: protobuf x86_64 3.14.0-13.el9 appstream 1.0 M 2024-05-27T07:10:46.170 INFO:teuthology.orchestra.run.smithi173.stdout: protobuf-compiler x86_64 3.14.0-13.el9 crb 863 k 2024-05-27T07:10:46.170 INFO:teuthology.orchestra.run.smithi173.stdout: python3-asyncssh noarch 2.13.2-5.el9 epel 548 k 2024-05-27T07:10:46.170 INFO:teuthology.orchestra.run.smithi173.stdout: python3-attrs noarch 20.3.0-7.el9 appstream 87 k 2024-05-27T07:10:46.170 INFO:teuthology.orchestra.run.smithi173.stdout: python3-babel noarch 2.9.1-2.el9 appstream 6.0 M 2024-05-27T07:10:46.170 INFO:teuthology.orchestra.run.smithi173.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2024-05-27T07:10:46.171 INFO:teuthology.orchestra.run.smithi173.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2024-05-27T07:10:46.171 INFO:teuthology.orchestra.run.smithi173.stdout: python3-ceph-argparse x86_64 2:19.0.0-3872.g38f9d900.el9 ceph 45 k 2024-05-27T07:10:46.171 INFO:teuthology.orchestra.run.smithi173.stdout: python3-ceph-common x86_64 2:19.0.0-3872.g38f9d900.el9 ceph 136 k 2024-05-27T07:10:46.171 INFO:teuthology.orchestra.run.smithi173.stdout: python3-certifi noarch 2021.10.8-2.el9 epel 15 k 2024-05-27T07:10:46.171 INFO:teuthology.orchestra.run.smithi173.stdout: python3-cffi x86_64 1.14.5-5.el9 baseos 253 k 2024-05-27T07:10:46.171 INFO:teuthology.orchestra.run.smithi173.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2024-05-27T07:10:46.171 INFO:teuthology.orchestra.run.smithi173.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2024-05-27T07:10:46.171 INFO:teuthology.orchestra.run.smithi173.stdout: python3-cmd2 noarch 2.4.2-2.el9 epel 303 k 2024-05-27T07:10:46.171 INFO:teuthology.orchestra.run.smithi173.stdout: python3-colorama noarch 0.4.6-3.el9 appstream 61 k 2024-05-27T07:10:46.171 INFO:teuthology.orchestra.run.smithi173.stdout: python3-cryptography x86_64 36.0.1-4.el9 baseos 1.2 M 2024-05-27T07:10:46.171 INFO:teuthology.orchestra.run.smithi173.stdout: python3-devel x86_64 3.9.18-3.el9 appstream 244 k 2024-05-27T07:10:46.171 INFO:teuthology.orchestra.run.smithi173.stdout: python3-google-auth noarch 1:2.29.0-1.el9 epel 206 k 2024-05-27T07:10:46.171 INFO:teuthology.orchestra.run.smithi173.stdout: python3-grpcio x86_64 1.46.7-10.el9 epel 2.0 M 2024-05-27T07:10:46.172 INFO:teuthology.orchestra.run.smithi173.stdout: python3-grpcio-tools x86_64 1.46.7-10.el9 epel 144 k 2024-05-27T07:10:46.172 INFO:teuthology.orchestra.run.smithi173.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2024-05-27T07:10:46.172 INFO:teuthology.orchestra.run.smithi173.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2024-05-27T07:10:46.172 INFO:teuthology.orchestra.run.smithi173.stdout: python3-jaraco-collections noarch 3.0.0-8.el9 epel 23 k 2024-05-27T07:10:46.172 INFO:teuthology.orchestra.run.smithi173.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2024-05-27T07:10:46.172 INFO:teuthology.orchestra.run.smithi173.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2024-05-27T07:10:46.172 INFO:teuthology.orchestra.run.smithi173.stdout: python3-jinja2 noarch 2.11.3-6.el9 appstream 248 k 2024-05-27T07:10:46.172 INFO:teuthology.orchestra.run.smithi173.stdout: python3-jsonpatch noarch 1.21-16.el9 appstream 26 k 2024-05-27T07:10:46.172 INFO:teuthology.orchestra.run.smithi173.stdout: python3-jsonpointer noarch 2.0-4.el9 appstream 19 k 2024-05-27T07:10:46.172 INFO:teuthology.orchestra.run.smithi173.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2024-05-27T07:10:46.172 INFO:teuthology.orchestra.run.smithi173.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2024-05-27T07:10:46.172 INFO:teuthology.orchestra.run.smithi173.stdout: python3-mako noarch 1.1.4-6.el9 appstream 172 k 2024-05-27T07:10:46.173 INFO:teuthology.orchestra.run.smithi173.stdout: python3-markupsafe x86_64 1.1.1-12.el9 appstream 35 k 2024-05-27T07:10:46.173 INFO:teuthology.orchestra.run.smithi173.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2024-05-27T07:10:46.173 INFO:teuthology.orchestra.run.smithi173.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2024-05-27T07:10:46.173 INFO:teuthology.orchestra.run.smithi173.stdout: python3-numpy x86_64 1:1.20.1-5.el9 appstream 5.1 M 2024-05-27T07:10:46.173 INFO:teuthology.orchestra.run.smithi173.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 appstream 250 k 2024-05-27T07:10:46.173 INFO:teuthology.orchestra.run.smithi173.stdout: python3-oauthlib noarch 3.1.1-5.el9 appstream 222 k 2024-05-27T07:10:46.173 INFO:teuthology.orchestra.run.smithi173.stdout: python3-packaging noarch 20.9-5.el9 appstream 77 k 2024-05-27T07:10:46.173 INFO:teuthology.orchestra.run.smithi173.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2024-05-27T07:10:46.173 INFO:teuthology.orchestra.run.smithi173.stdout: python3-ply noarch 3.11-14.el9 baseos 106 k 2024-05-27T07:10:46.173 INFO:teuthology.orchestra.run.smithi173.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2024-05-27T07:10:46.173 INFO:teuthology.orchestra.run.smithi173.stdout: python3-prettytable noarch 0.7.2-27.el9 appstream 42 k 2024-05-27T07:10:46.173 INFO:teuthology.orchestra.run.smithi173.stdout: python3-protobuf noarch 3.14.0-13.el9 appstream 269 k 2024-05-27T07:10:46.173 INFO:teuthology.orchestra.run.smithi173.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2024-05-27T07:10:46.173 INFO:teuthology.orchestra.run.smithi173.stdout: python3-pyasn1 noarch 0.4.8-6.el9 appstream 159 k 2024-05-27T07:10:46.174 INFO:teuthology.orchestra.run.smithi173.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 appstream 279 k 2024-05-27T07:10:46.174 INFO:teuthology.orchestra.run.smithi173.stdout: python3-pycparser noarch 2.20-6.el9 baseos 135 k 2024-05-27T07:10:46.174 INFO:teuthology.orchestra.run.smithi173.stdout: python3-pyperclip noarch 1.8.2-2.el9 epel 28 k 2024-05-27T07:10:46.174 INFO:teuthology.orchestra.run.smithi173.stdout: python3-pytz noarch 2021.1-5.el9 appstream 51 k 2024-05-27T07:10:46.174 INFO:teuthology.orchestra.run.smithi173.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2024-05-27T07:10:46.174 INFO:teuthology.orchestra.run.smithi173.stdout: python3-requests noarch 2.25.1-8.el9 baseos 125 k 2024-05-27T07:10:46.174 INFO:teuthology.orchestra.run.smithi173.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 appstream 54 k 2024-05-27T07:10:46.174 INFO:teuthology.orchestra.run.smithi173.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2024-05-27T07:10:46.174 INFO:teuthology.orchestra.run.smithi173.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2024-05-27T07:10:46.174 INFO:teuthology.orchestra.run.smithi173.stdout: python3-scipy x86_64 1.6.2-8.el9 appstream 16 M 2024-05-27T07:10:46.174 INFO:teuthology.orchestra.run.smithi173.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2024-05-27T07:10:46.174 INFO:teuthology.orchestra.run.smithi173.stdout: python3-toml noarch 0.10.2-6.el9 appstream 42 k 2024-05-27T07:10:46.175 INFO:teuthology.orchestra.run.smithi173.stdout: python3-typing-extensions noarch 4.4.0-2.el9 epel 51 k 2024-05-27T07:10:46.175 INFO:teuthology.orchestra.run.smithi173.stdout: python3-urllib3 noarch 1.26.5-5.el9 baseos 215 k 2024-05-27T07:10:46.175 INFO:teuthology.orchestra.run.smithi173.stdout: python3-wcwidth noarch 0.2.5-8.el9 appstream 44 k 2024-05-27T07:10:46.175 INFO:teuthology.orchestra.run.smithi173.stdout: python3-webob noarch 1.8.7-6.el9 epel 230 k 2024-05-27T07:10:46.175 INFO:teuthology.orchestra.run.smithi173.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2024-05-27T07:10:46.175 INFO:teuthology.orchestra.run.smithi173.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2024-05-27T07:10:46.175 INFO:teuthology.orchestra.run.smithi173.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2024-05-27T07:10:46.175 INFO:teuthology.orchestra.run.smithi173.stdout: qatlib x86_64 24.02.0-1.el9 appstream 221 k 2024-05-27T07:10:46.175 INFO:teuthology.orchestra.run.smithi173.stdout: qatzip-libs x86_64 1.2.0-1.el9 appstream 47 k 2024-05-27T07:10:46.175 INFO:teuthology.orchestra.run.smithi173.stdout: re2 x86_64 1:20211101-3.el9 epel 192 k 2024-05-27T07:10:46.175 INFO:teuthology.orchestra.run.smithi173.stdout: socat x86_64 1.7.4.1-5.el9 appstream 305 k 2024-05-27T07:10:46.175 INFO:teuthology.orchestra.run.smithi173.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2024-05-27T07:10:46.175 INFO:teuthology.orchestra.run.smithi173.stdout: uuid x86_64 1.6.2-55.el9 appstream 58 k 2024-05-27T07:10:46.176 INFO:teuthology.orchestra.run.smithi173.stdout: xmlstarlet x86_64 1.6.1-20.el9 appstream 64 k 2024-05-27T07:10:46.176 INFO:teuthology.orchestra.run.smithi173.stdout: zlib-devel x86_64 1.2.11-41.el9 appstream 45 k 2024-05-27T07:10:46.176 INFO:teuthology.orchestra.run.smithi173.stdout:Installing weak dependencies: 2024-05-27T07:10:46.176 INFO:teuthology.orchestra.run.smithi173.stdout: qatlib-service x86_64 24.02.0-1.el9 appstream 36 k 2024-05-27T07:10:46.176 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-05-27T07:10:46.176 INFO:teuthology.orchestra.run.smithi173.stdout:Transaction Summary 2024-05-27T07:10:46.176 INFO:teuthology.orchestra.run.smithi173.stdout:======================================================================================= 2024-05-27T07:10:46.176 INFO:teuthology.orchestra.run.smithi173.stdout:Install 148 Packages 2024-05-27T07:10:46.176 INFO:teuthology.orchestra.run.smithi173.stdout:Upgrade 11 Packages 2024-05-27T07:10:46.176 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-05-27T07:10:46.177 INFO:teuthology.orchestra.run.smithi173.stdout:Total download size: 219 M 2024-05-27T07:10:46.177 INFO:teuthology.orchestra.run.smithi173.stdout:Downloading Packages: 2024-05-27T07:10:46.249 INFO:teuthology.orchestra.run.smithi052.stdout:(10/159): ceph-radosgw-19.0.0-3872.g38f9d900.el 10 MB/s | 10 MB 00:00 2024-05-27T07:10:46.333 INFO:teuthology.orchestra.run.smithi052.stdout:(11/159): cephfs-mirror-19.0.0-3872.g38f9d900.e 2.6 MB/s | 224 kB 00:00 2024-05-27T07:10:46.408 INFO:teuthology.orchestra.run.smithi052.stdout:(12/159): cephfs-shell-19.0.0-3872.g38f9d900.el 361 kB/s | 27 kB 00:00 2024-05-27T07:10:46.484 INFO:teuthology.orchestra.run.smithi052.stdout:(13/159): libcephfs-devel-19.0.0-3872.g38f9d900 423 kB/s | 32 kB 00:00 2024-05-27T07:10:46.634 INFO:teuthology.orchestra.run.smithi052.stdout:(14/159): libcephfs2-19.0.0-3872.g38f9d900.el9. 4.9 MB/s | 755 kB 00:00 2024-05-27T07:10:46.718 INFO:teuthology.orchestra.run.smithi052.stdout:(15/159): libcephsqlite-19.0.0-3872.g38f9d900.e 1.9 MB/s | 157 kB 00:00 2024-05-27T07:10:46.764 INFO:teuthology.orchestra.run.smithi105.stdout:(1/159): ceph-19.0.0-3872.g38f9d900.el9.x86_64. 12 kB/s | 6.6 kB 00:00 2024-05-27T07:10:46.801 INFO:teuthology.orchestra.run.smithi052.stdout:(16/159): librados-devel-19.0.0-3872.g38f9d900. 1.5 MB/s | 127 kB 00:00 2024-05-27T07:10:46.910 INFO:teuthology.orchestra.run.smithi052.stdout:(17/159): libradosstriper1-19.0.0-3872.g38f9d90 4.5 MB/s | 498 kB 00:00 2024-05-27T07:10:47.155 INFO:teuthology.orchestra.run.smithi173.stdout:(1/159): ceph-19.0.0-3872.g38f9d900.el9.x86_64. 15 kB/s | 6.6 kB 00:00 2024-05-27T07:10:47.660 INFO:teuthology.orchestra.run.smithi052.stdout:(18/159): librgw2-19.0.0-3872.g38f9d900.el9.x86 6.9 MB/s | 5.2 MB 00:00 2024-05-27T07:10:47.769 INFO:teuthology.orchestra.run.smithi052.stdout:(19/159): python3-ceph-argparse-19.0.0-3872.g38 418 kB/s | 45 kB 00:00 2024-05-27T07:10:47.839 INFO:teuthology.orchestra.run.smithi105.stdout:(2/159): ceph-fuse-19.0.0-3872.g38f9d900.el9.x8 803 kB/s | 863 kB 00:01 2024-05-27T07:10:47.855 INFO:teuthology.orchestra.run.smithi173.stdout:(2/159): ceph-fuse-19.0.0-3872.g38f9d900.el9.x8 1.2 MB/s | 863 kB 00:00 2024-05-27T07:10:47.869 INFO:teuthology.orchestra.run.smithi052.stdout:(20/159): python3-ceph-common-19.0.0-3872.g38f9 1.3 MB/s | 136 kB 00:00 2024-05-27T07:10:47.988 INFO:teuthology.orchestra.run.smithi173.stdout:(3/159): ceph-immutable-object-cache-19.0.0-387 1.1 MB/s | 147 kB 00:00 2024-05-27T07:10:48.056 INFO:teuthology.orchestra.run.smithi105.stdout:(3/159): ceph-immutable-object-cache-19.0.0-387 679 kB/s | 147 kB 00:00 2024-05-27T07:10:48.528 INFO:teuthology.orchestra.run.smithi052.stdout:(21/159): python3-cephfs-19.0.0-3872.g38f9d900. 245 kB/s | 161 kB 00:00 2024-05-27T07:10:48.551 INFO:teuthology.orchestra.run.smithi173.stdout:(4/159): ceph-mds-19.0.0-3872.g38f9d900.el9.x86 4.1 MB/s | 2.3 MB 00:00 2024-05-27T07:10:48.603 INFO:teuthology.orchestra.run.smithi052.stdout:(22/159): python3-rados-19.0.0-3872.g38f9d900.e 4.2 MB/s | 320 kB 00:00 2024-05-27T07:10:48.703 INFO:teuthology.orchestra.run.smithi052.stdout:(23/159): python3-rbd-19.0.0-3872.g38f9d900.el9 2.9 MB/s | 300 kB 00:00 2024-05-27T07:10:48.795 INFO:teuthology.orchestra.run.smithi052.stdout:(24/159): python3-rgw-19.0.0-3872.g38f9d900.el9 1.1 MB/s | 99 kB 00:00 2024-05-27T07:10:48.872 INFO:teuthology.orchestra.run.smithi173.stdout:(5/159): ceph-mgr-19.0.0-3872.g38f9d900.el9.x86 4.6 MB/s | 1.5 MB 00:00 2024-05-27T07:10:48.887 INFO:teuthology.orchestra.run.smithi052.stdout:(25/159): rbd-fuse-19.0.0-3872.g38f9d900.el9.x8 942 kB/s | 86 kB 00:00 2024-05-27T07:10:49.780 INFO:teuthology.orchestra.run.smithi173.stdout:(6/159): ceph-mon-19.0.0-3872.g38f9d900.el9.x86 5.4 MB/s | 4.9 MB 00:00 2024-05-27T07:10:49.804 INFO:teuthology.orchestra.run.smithi052.stdout:(26/159): rbd-mirror-19.0.0-3872.g38f9d900.el9. 3.3 MB/s | 3.0 MB 00:00 2024-05-27T07:10:49.856 INFO:teuthology.orchestra.run.smithi173.stdout:(7/159): ceph-base-19.0.0-3872.g38f9d900.el9.x8 1.7 MB/s | 5.4 MB 00:03 2024-05-27T07:10:49.871 INFO:teuthology.orchestra.run.smithi052.stdout:(27/159): rbd-nbd-19.0.0-3872.g38f9d900.el9.x86 2.5 MB/s | 169 kB 00:00 2024-05-27T07:10:49.938 INFO:teuthology.orchestra.run.smithi052.stdout:(28/159): ceph-grafana-dashboards-19.0.0-3872.g 514 kB/s | 34 kB 00:00 2024-05-27T07:10:50.013 INFO:teuthology.orchestra.run.smithi052.stdout:(29/159): ceph-mgr-cephadm-19.0.0-3872.g38f9d90 1.9 MB/s | 143 kB 00:00 2024-05-27T07:10:50.140 INFO:teuthology.orchestra.run.smithi105.stdout:(4/159): ceph-base-19.0.0-3872.g38f9d900.el9.x8 1.4 MB/s | 5.4 MB 00:03 2024-05-27T07:10:50.190 INFO:teuthology.orchestra.run.smithi105.stdout:(5/159): ceph-mds-19.0.0-3872.g38f9d900.el9.x86 1.1 MB/s | 2.3 MB 00:02 2024-05-27T07:10:50.430 INFO:teuthology.orchestra.run.smithi052.stdout:(30/159): ceph-test-19.0.0-3872.g38f9d900.el9.x 10 MB/s | 43 MB 00:04 2024-05-27T07:10:50.739 INFO:teuthology.orchestra.run.smithi173.stdout:(8/159): ceph-common-19.0.0-3872.g38f9d900.el9. 4.9 MB/s | 20 MB 00:04 2024-05-27T07:10:50.840 INFO:teuthology.orchestra.run.smithi173.stdout:(9/159): ceph-selinux-19.0.0-3872.g38f9d900.el9 252 kB/s | 25 kB 00:00 2024-05-27T07:10:50.905 INFO:teuthology.orchestra.run.smithi052.stdout:(31/159): ceph-mgr-dashboard-19.0.0-3872.g38f9d 4.0 MB/s | 3.5 MB 00:00 2024-05-27T07:10:51.047 INFO:teuthology.orchestra.run.smithi052.stdout:(32/159): ceph-mgr-modules-core-19.0.0-3872.g38 1.9 MB/s | 274 kB 00:00 2024-05-27T07:10:51.107 INFO:teuthology.orchestra.run.smithi105.stdout:(6/159): ceph-mgr-19.0.0-3872.g38f9d900.el9.x86 1.6 MB/s | 1.5 MB 00:00 2024-05-27T07:10:51.122 INFO:teuthology.orchestra.run.smithi052.stdout:(33/159): ceph-mgr-rook-19.0.0-3872.g38f9d900.e 657 kB/s | 49 kB 00:00 2024-05-27T07:10:51.198 INFO:teuthology.orchestra.run.smithi052.stdout:(34/159): ceph-prometheus-alerts-19.0.0-3872.g3 219 kB/s | 16 kB 00:00 2024-05-27T07:10:51.290 INFO:teuthology.orchestra.run.smithi052.stdout:(35/159): ceph-volume-19.0.0-3872.g38f9d900.el9 2.9 MB/s | 273 kB 00:00 2024-05-27T07:10:51.549 INFO:teuthology.orchestra.run.smithi052.stdout:(36/159): cephadm-19.0.0-3872.g38f9d900.el9.noa 2.9 MB/s | 764 kB 00:00 2024-05-27T07:10:51.624 INFO:teuthology.orchestra.run.smithi052.stdout:(37/159): cephfs-top-19.0.0-3872.g38f9d900.el9. 243 kB/s | 18 kB 00:00 2024-05-27T07:10:51.858 INFO:teuthology.orchestra.run.smithi052.stdout:(38/159): fuse-2.9.9-15.el9.x86_64.rpm 344 kB/s | 80 kB 00:00 2024-05-27T07:10:52.050 INFO:teuthology.orchestra.run.smithi052.stdout:(39/159): libgfortran-11.4.1-3.el9.x86_64.rpm 4.1 MB/s | 803 kB 00:00 2024-05-27T07:10:52.125 INFO:teuthology.orchestra.run.smithi052.stdout:(40/159): libquadmath-11.4.1-3.el9.x86_64.rpm 2.6 MB/s | 195 kB 00:00 2024-05-27T07:10:52.192 INFO:teuthology.orchestra.run.smithi052.stdout:(41/159): python3-cffi-1.14.5-5.el9.x86_64.rpm 3.7 MB/s | 253 kB 00:00 2024-05-27T07:10:52.334 INFO:teuthology.orchestra.run.smithi052.stdout:(42/159): ceph-mgr-diskprediction-local-19.0.0- 3.9 MB/s | 7.4 MB 00:01 2024-05-27T07:10:52.376 INFO:teuthology.orchestra.run.smithi052.stdout:(43/159): python3-cryptography-36.0.1-4.el9.x86 6.7 MB/s | 1.2 MB 00:00 2024-05-27T07:10:52.431 INFO:teuthology.orchestra.run.smithi173.stdout:(10/159): ceph-osd-19.0.0-3872.g38f9d900.el9.x8 6.7 MB/s | 18 MB 00:02 2024-05-27T07:10:52.443 INFO:teuthology.orchestra.run.smithi052.stdout:(44/159): python3-pycparser-2.20-6.el9.noarch.r 2.0 MB/s | 135 kB 00:00 2024-05-27T07:10:52.518 INFO:teuthology.orchestra.run.smithi052.stdout:(45/159): python3-requests-2.25.1-8.el9.noarch. 1.6 MB/s | 125 kB 00:00 2024-05-27T07:10:52.540 INFO:teuthology.orchestra.run.smithi173.stdout:(11/159): cephfs-mirror-19.0.0-3872.g38f9d900.e 2.0 MB/s | 224 kB 00:00 2024-05-27T07:10:52.593 INFO:teuthology.orchestra.run.smithi052.stdout:(46/159): python3-urllib3-1.26.5-5.el9.noarch.r 2.8 MB/s | 215 kB 00:00 2024-05-27T07:10:52.610 INFO:teuthology.orchestra.run.smithi052.stdout:(47/159): python3-ply-3.11-14.el9.noarch.rpm 386 kB/s | 106 kB 00:00 2024-05-27T07:10:52.648 INFO:teuthology.orchestra.run.smithi173.stdout:(12/159): ceph-radosgw-19.0.0-3872.g38f9d900.el 3.6 MB/s | 10 MB 00:02 2024-05-27T07:10:52.674 INFO:teuthology.orchestra.run.smithi173.stdout:(13/159): cephfs-shell-19.0.0-3872.g38f9d900.el 202 kB/s | 27 kB 00:00 2024-05-27T07:10:52.699 INFO:teuthology.orchestra.run.smithi173.stdout:(14/159): libcephfs-devel-19.0.0-3872.g38f9d900 636 kB/s | 32 kB 00:00 2024-05-27T07:10:52.766 INFO:teuthology.orchestra.run.smithi173.stdout:(15/159): libcephsqlite-19.0.0-3872.g38f9d900.e 2.3 MB/s | 157 kB 00:00 2024-05-27T07:10:52.816 INFO:teuthology.orchestra.run.smithi173.stdout:(16/159): libcephfs2-19.0.0-3872.g38f9d900.el9. 5.2 MB/s | 755 kB 00:00 2024-05-27T07:10:52.841 INFO:teuthology.orchestra.run.smithi173.stdout:(17/159): librados-devel-19.0.0-3872.g38f9d900. 1.6 MB/s | 127 kB 00:00 2024-05-27T07:10:52.932 INFO:teuthology.orchestra.run.smithi105.stdout:(7/159): ceph-common-19.0.0-3872.g38f9d900.el9. 2.9 MB/s | 20 MB 00:06 2024-05-27T07:10:52.958 INFO:teuthology.orchestra.run.smithi173.stdout:(18/159): libradosstriper1-19.0.0-3872.g38f9d90 3.4 MB/s | 498 kB 00:00 2024-05-27T07:10:53.034 INFO:teuthology.orchestra.run.smithi173.stdout:(19/159): python3-ceph-argparse-19.0.0-3872.g38 602 kB/s | 45 kB 00:00 2024-05-27T07:10:53.107 INFO:teuthology.orchestra.run.smithi105.stdout:(8/159): ceph-mon-19.0.0-3872.g38f9d900.el9.x86 1.7 MB/s | 4.9 MB 00:02 2024-05-27T07:10:53.125 INFO:teuthology.orchestra.run.smithi173.stdout:(20/159): python3-ceph-common-19.0.0-3872.g38f9 1.4 MB/s | 136 kB 00:00 2024-05-27T07:10:53.199 INFO:teuthology.orchestra.run.smithi105.stdout:(9/159): ceph-selinux-19.0.0-3872.g38f9d900.el9 275 kB/s | 25 kB 00:00 2024-05-27T07:10:53.209 INFO:teuthology.orchestra.run.smithi173.stdout:(21/159): python3-cephfs-19.0.0-3872.g38f9d900. 1.9 MB/s | 161 kB 00:00 2024-05-27T07:10:53.309 INFO:teuthology.orchestra.run.smithi173.stdout:(22/159): python3-rados-19.0.0-3872.g38f9d900.e 3.1 MB/s | 320 kB 00:00 2024-05-27T07:10:53.393 INFO:teuthology.orchestra.run.smithi173.stdout:(23/159): python3-rbd-19.0.0-3872.g38f9d900.el9 3.5 MB/s | 300 kB 00:00 2024-05-27T07:10:53.494 INFO:teuthology.orchestra.run.smithi173.stdout:(24/159): python3-rgw-19.0.0-3872.g38f9d900.el9 991 kB/s | 99 kB 00:00 2024-05-27T07:10:53.585 INFO:teuthology.orchestra.run.smithi173.stdout:(25/159): rbd-fuse-19.0.0-3872.g38f9d900.el9.x8 941 kB/s | 86 kB 00:00 2024-05-27T07:10:53.719 INFO:teuthology.orchestra.run.smithi052.stdout:(48/159): automake-1.16.2-8.el9.noarch.rpm 612 kB/s | 689 kB 00:01 2024-05-27T07:10:53.727 INFO:teuthology.orchestra.run.smithi173.stdout:(26/159): librgw2-19.0.0-3872.g38f9d900.el9.x86 5.8 MB/s | 5.2 MB 00:00 2024-05-27T07:10:53.794 INFO:teuthology.orchestra.run.smithi173.stdout:(27/159): rbd-nbd-19.0.0-3872.g38f9d900.el9.x86 2.5 MB/s | 169 kB 00:00 2024-05-27T07:10:53.802 INFO:teuthology.orchestra.run.smithi052.stdout:(49/159): bison-3.7.4-5.el9.x86_64.rpm 791 kB/s | 943 kB 00:01 2024-05-27T07:10:53.861 INFO:teuthology.orchestra.run.smithi173.stdout:(28/159): ceph-grafana-dashboards-19.0.0-3872.g 513 kB/s | 34 kB 00:00 2024-05-27T07:10:53.928 INFO:teuthology.orchestra.run.smithi173.stdout:(29/159): ceph-mgr-cephadm-19.0.0-3872.g38f9d90 2.1 MB/s | 143 kB 00:00 2024-05-27T07:10:54.054 INFO:teuthology.orchestra.run.smithi173.stdout:(30/159): rbd-mirror-19.0.0-3872.g38f9d900.el9. 6.5 MB/s | 3.0 MB 00:00 2024-05-27T07:10:54.495 INFO:teuthology.orchestra.run.smithi052.stdout:(50/159): elfutils-libelf-devel-0.191-4.el9.x86 31 kB/s | 22 kB 00:00 2024-05-27T07:10:54.570 INFO:teuthology.orchestra.run.smithi052.stdout:(51/159): boost-program-options-1.75.0-8.el9.x8 126 kB/s | 107 kB 00:00 2024-05-27T07:10:54.804 INFO:teuthology.orchestra.run.smithi173.stdout:(31/159): ceph-mgr-dashboard-19.0.0-3872.g38f9d 4.0 MB/s | 3.5 MB 00:00 2024-05-27T07:10:54.870 INFO:teuthology.orchestra.run.smithi052.stdout:(52/159): ceph-common-19.0.0-3872.g38f9d900.el9 1.8 MB/s | 20 MB 00:10 2024-05-27T07:10:54.929 INFO:teuthology.orchestra.run.smithi173.stdout:(32/159): ceph-mgr-modules-core-19.0.0-3872.g38 2.1 MB/s | 274 kB 00:00 2024-05-27T07:10:54.971 INFO:teuthology.orchestra.run.smithi173.stdout:(33/159): ceph-mgr-rook-19.0.0-3872.g38f9d900.e 1.2 MB/s | 49 kB 00:00 2024-05-27T07:10:54.991 INFO:teuthology.orchestra.run.smithi105.stdout:(10/159): ceph-radosgw-19.0.0-3872.g38f9d900.el 4.9 MB/s | 10 MB 00:02 2024-05-27T07:10:55.041 INFO:teuthology.orchestra.run.smithi105.stdout:(11/159): cephfs-mirror-19.0.0-3872.g38f9d900.e 4.4 MB/s | 224 kB 00:00 2024-05-27T07:10:55.063 INFO:teuthology.orchestra.run.smithi173.stdout:(34/159): ceph-mgr-diskprediction-local-19.0.0- 7.3 MB/s | 7.4 MB 00:01 2024-05-27T07:10:55.083 INFO:teuthology.orchestra.run.smithi105.stdout:(12/159): cephfs-shell-19.0.0-3872.g38f9d900.el 646 kB/s | 27 kB 00:00 2024-05-27T07:10:55.088 INFO:teuthology.orchestra.run.smithi173.stdout:(35/159): ceph-prometheus-alerts-19.0.0-3872.g3 140 kB/s | 16 kB 00:00 2024-05-27T07:10:55.125 INFO:teuthology.orchestra.run.smithi105.stdout:(13/159): libcephfs-devel-19.0.0-3872.g38f9d900 763 kB/s | 32 kB 00:00 2024-05-27T07:10:55.138 INFO:teuthology.orchestra.run.smithi173.stdout:(36/159): ceph-volume-19.0.0-3872.g38f9d900.el9 3.5 MB/s | 273 kB 00:00 2024-05-27T07:10:55.239 INFO:teuthology.orchestra.run.smithi173.stdout:(37/159): cephfs-top-19.0.0-3872.g38f9d900.el9. 182 kB/s | 18 kB 00:00 2024-05-27T07:10:55.259 INFO:teuthology.orchestra.run.smithi105.stdout:(14/159): libcephfs2-19.0.0-3872.g38f9d900.el9. 5.5 MB/s | 755 kB 00:00 2024-05-27T07:10:55.264 INFO:teuthology.orchestra.run.smithi173.stdout:(38/159): cephadm-19.0.0-3872.g38f9d900.el9.noa 4.3 MB/s | 764 kB 00:00 2024-05-27T07:10:55.309 INFO:teuthology.orchestra.run.smithi105.stdout:(15/159): libcephsqlite-19.0.0-3872.g38f9d900.e 3.1 MB/s | 157 kB 00:00 2024-05-27T07:10:55.360 INFO:teuthology.orchestra.run.smithi105.stdout:(16/159): librados-devel-19.0.0-3872.g38f9d900. 2.5 MB/s | 127 kB 00:00 2024-05-27T07:10:55.364 INFO:teuthology.orchestra.run.smithi173.stdout:(39/159): fuse-2.9.9-15.el9.x86_64.rpm 641 kB/s | 80 kB 00:00 2024-05-27T07:10:55.440 INFO:teuthology.orchestra.run.smithi173.stdout:(40/159): libquadmath-11.4.1-3.el9.x86_64.rpm 2.5 MB/s | 195 kB 00:00 2024-05-27T07:10:55.452 INFO:teuthology.orchestra.run.smithi105.stdout:(17/159): libradosstriper1-19.0.0-3872.g38f9d90 5.3 MB/s | 498 kB 00:00 2024-05-27T07:10:55.470 INFO:teuthology.orchestra.run.smithi052.stdout:(53/159): flexiblas-3.0.4-8.el9.x86_64.rpm 36 kB/s | 33 kB 00:00 2024-05-27T07:10:55.473 INFO:teuthology.orchestra.run.smithi173.stdout:(41/159): libgfortran-11.4.1-3.el9.x86_64.rpm 3.8 MB/s | 803 kB 00:00 2024-05-27T07:10:55.498 INFO:teuthology.orchestra.run.smithi173.stdout:(42/159): python3-cffi-1.14.5-5.el9.x86_64.rpm 4.2 MB/s | 253 kB 00:00 2024-05-27T07:10:55.541 INFO:teuthology.orchestra.run.smithi173.stdout:(43/159): python3-ply-3.11-14.el9.noarch.rpm 2.5 MB/s | 106 kB 00:00 2024-05-27T07:10:55.583 INFO:teuthology.orchestra.run.smithi173.stdout:(44/159): python3-cryptography-36.0.1-4.el9.x86 11 MB/s | 1.2 MB 00:00 2024-05-27T07:10:55.608 INFO:teuthology.orchestra.run.smithi173.stdout:(45/159): python3-pycparser-2.20-6.el9.noarch.r 2.0 MB/s | 135 kB 00:00 2024-05-27T07:10:55.633 INFO:teuthology.orchestra.run.smithi173.stdout:(46/159): python3-requests-2.25.1-8.el9.noarch. 2.4 MB/s | 125 kB 00:00 2024-05-27T07:10:55.658 INFO:teuthology.orchestra.run.smithi173.stdout:(47/159): python3-urllib3-1.26.5-5.el9.noarch.r 4.2 MB/s | 215 kB 00:00 2024-05-27T07:10:55.687 INFO:teuthology.orchestra.run.smithi052.stdout:(54/159): flex-2.6.4-9.el9.x86_64.rpm 263 kB/s | 313 kB 00:01 2024-05-27T07:10:55.985 INFO:teuthology.orchestra.run.smithi105.stdout:(18/159): librgw2-19.0.0-3872.g38f9d900.el9.x86 9.7 MB/s | 5.2 MB 00:00 2024-05-27T07:10:56.035 INFO:teuthology.orchestra.run.smithi105.stdout:(19/159): python3-ceph-argparse-19.0.0-3872.g38 907 kB/s | 45 kB 00:00 2024-05-27T07:10:56.084 INFO:teuthology.orchestra.run.smithi173.stdout:(48/159): automake-1.16.2-8.el9.noarch.rpm 1.5 MB/s | 689 kB 00:00 2024-05-27T07:10:56.085 INFO:teuthology.orchestra.run.smithi105.stdout:(20/159): python3-ceph-common-19.0.0-3872.g38f9 2.7 MB/s | 136 kB 00:00 2024-05-27T07:10:56.126 INFO:teuthology.orchestra.run.smithi173.stdout:(49/159): bison-3.7.4-5.el9.x86_64.rpm 2.0 MB/s | 943 kB 00:00 2024-05-27T07:10:56.136 INFO:teuthology.orchestra.run.smithi105.stdout:(21/159): python3-cephfs-19.0.0-3872.g38f9d900. 3.2 MB/s | 161 kB 00:00 2024-05-27T07:10:56.159 INFO:teuthology.orchestra.run.smithi173.stdout:(50/159): boost-program-options-1.75.0-8.el9.x8 1.4 MB/s | 107 kB 00:00 2024-05-27T07:10:56.162 INFO:teuthology.orchestra.run.smithi052.stdout:(55/159): flexiblas-openblas-openmp-3.0.4-8.el9 26 kB/s | 18 kB 00:00 2024-05-27T07:10:56.201 INFO:teuthology.orchestra.run.smithi173.stdout:(51/159): elfutils-libelf-devel-0.191-4.el9.x86 288 kB/s | 22 kB 00:00 2024-05-27T07:10:56.203 INFO:teuthology.orchestra.run.smithi105.stdout:(22/159): python3-rados-19.0.0-3872.g38f9d900.e 4.7 MB/s | 320 kB 00:00 2024-05-27T07:10:56.270 INFO:teuthology.orchestra.run.smithi105.stdout:(23/159): python3-rbd-19.0.0-3872.g38f9d900.el9 4.4 MB/s | 300 kB 00:00 2024-05-27T07:10:56.277 INFO:teuthology.orchestra.run.smithi173.stdout:(52/159): flexiblas-3.0.4-8.el9.x86_64.rpm 434 kB/s | 33 kB 00:00 2024-05-27T07:10:56.302 INFO:teuthology.orchestra.run.smithi173.stdout:(53/159): flex-2.6.4-9.el9.x86_64.rpm 2.1 MB/s | 313 kB 00:00 2024-05-27T07:10:56.320 INFO:teuthology.orchestra.run.smithi105.stdout:(24/159): python3-rgw-19.0.0-3872.g38f9d900.el9 1.9 MB/s | 99 kB 00:00 2024-05-27T07:10:56.346 INFO:teuthology.orchestra.run.smithi052.stdout:(56/159): flexiblas-netlib-3.0.4-8.el9.x86_64.r 2.1 MB/s | 3.0 MB 00:01 2024-05-27T07:10:56.362 INFO:teuthology.orchestra.run.smithi105.stdout:(25/159): rbd-fuse-19.0.0-3872.g38f9d900.el9.x8 2.0 MB/s | 86 kB 00:00 2024-05-27T07:10:56.363 INFO:teuthology.orchestra.run.smithi052.stdout:(57/159): libaio-devel-0.3.111-13.el9.x86_64.rp 16 kB/s | 11 kB 00:00 2024-05-27T07:10:56.377 INFO:teuthology.orchestra.run.smithi173.stdout:(54/159): flexiblas-openblas-openmp-3.0.4-8.el9 243 kB/s | 18 kB 00:00 2024-05-27T07:10:56.444 INFO:teuthology.orchestra.run.smithi173.stdout:(55/159): libaio-devel-0.3.111-13.el9.x86_64.rp 167 kB/s | 11 kB 00:00 2024-05-27T07:10:56.519 INFO:teuthology.orchestra.run.smithi173.stdout:(56/159): libpmemobj-1.12.1-1.el9.x86_64.rpm 2.1 MB/s | 160 kB 00:00 2024-05-27T07:10:56.586 INFO:teuthology.orchestra.run.smithi173.stdout:(57/159): librabbitmq-0.11.0-7.el9.x86_64.rpm 678 kB/s | 45 kB 00:00 2024-05-27T07:10:56.696 INFO:teuthology.orchestra.run.smithi105.stdout:(26/159): rbd-mirror-19.0.0-3872.g38f9d900.el9. 9.1 MB/s | 3.0 MB 00:00 2024-05-27T07:10:56.746 INFO:teuthology.orchestra.run.smithi105.stdout:(27/159): rbd-nbd-19.0.0-3872.g38f9d900.el9.x86 3.3 MB/s | 169 kB 00:00 2024-05-27T07:10:56.796 INFO:teuthology.orchestra.run.smithi105.stdout:(28/159): ceph-grafana-dashboards-19.0.0-3872.g 686 kB/s | 34 kB 00:00 2024-05-27T07:10:56.847 INFO:teuthology.orchestra.run.smithi105.stdout:(29/159): ceph-mgr-cephadm-19.0.0-3872.g38f9d90 2.8 MB/s | 143 kB 00:00 2024-05-27T07:10:57.079 INFO:teuthology.orchestra.run.smithi173.stdout:(58/159): ceph-test-19.0.0-3872.g38f9d900.el9.x 6.9 MB/s | 43 MB 00:06 2024-05-27T07:10:57.097 INFO:teuthology.orchestra.run.smithi052.stdout:(58/159): libpmemobj-1.12.1-1.el9.x86_64.rpm 171 kB/s | 160 kB 00:00 2024-05-27T07:10:57.103 INFO:teuthology.orchestra.run.smithi173.stdout:(59/159): librdkafka-1.6.1-102.el9.x86_64.rpm 1.3 MB/s | 662 kB 00:00 2024-05-27T07:10:57.114 INFO:teuthology.orchestra.run.smithi052.stdout:(59/159): librabbitmq-0.11.0-7.el9.x86_64.rpm 59 kB/s | 45 kB 00:00 2024-05-27T07:10:57.154 INFO:teuthology.orchestra.run.smithi173.stdout:(60/159): flexiblas-netlib-3.0.4-8.el9.x86_64.r 3.4 MB/s | 3.0 MB 00:00 2024-05-27T07:10:57.173 INFO:teuthology.orchestra.run.smithi173.stdout:(61/159): libuuid-devel-2.37.4-18.el9.x86_64.rp 380 kB/s | 25 kB 00:00 2024-05-27T07:10:57.197 INFO:teuthology.orchestra.run.smithi105.stdout:(30/159): ceph-mgr-dashboard-19.0.0-3872.g38f9d 10 MB/s | 3.5 MB 00:00 2024-05-27T07:10:57.221 INFO:teuthology.orchestra.run.smithi173.stdout:(62/159): libzstd-devel-1.5.1-2.el9.x86_64.rpm 709 kB/s | 47 kB 00:00 2024-05-27T07:10:57.305 INFO:teuthology.orchestra.run.smithi173.stdout:(63/159): lua-5.4.4-4.el9.x86_64.rpm 2.2 MB/s | 188 kB 00:00 2024-05-27T07:10:57.355 INFO:teuthology.orchestra.run.smithi173.stdout:(64/159): lttng-ust-2.12.0-6.el9.x86_64.rpm 1.6 MB/s | 292 kB 00:00 2024-05-27T07:10:57.372 INFO:teuthology.orchestra.run.smithi173.stdout:(65/159): openblas-0.3.21-2.el9.x86_64.rpm 527 kB/s | 35 kB 00:00 2024-05-27T07:10:57.531 INFO:teuthology.orchestra.run.smithi173.stdout:(66/159): libtool-2.4.6-45.el9.x86_64.rpm 1.3 MB/s | 582 kB 00:00 2024-05-27T07:10:57.572 INFO:teuthology.orchestra.run.smithi052.stdout:(60/159): librdkafka-1.6.1-102.el9.x86_64.rpm 548 kB/s | 662 kB 00:01 2024-05-27T07:10:57.864 INFO:teuthology.orchestra.run.smithi105.stdout:(31/159): ceph-mgr-diskprediction-local-19.0.0- 11 MB/s | 7.4 MB 00:00 2024-05-27T07:10:57.914 INFO:teuthology.orchestra.run.smithi105.stdout:(32/159): ceph-mgr-modules-core-19.0.0-3872.g38 5.4 MB/s | 274 kB 00:00 2024-05-27T07:10:57.931 INFO:teuthology.orchestra.run.smithi052.stdout:(61/159): libuuid-devel-2.37.4-18.el9.x86_64.rp 31 kB/s | 25 kB 00:00 2024-05-27T07:10:57.964 INFO:teuthology.orchestra.run.smithi105.stdout:(33/159): ceph-mgr-rook-19.0.0-3872.g38f9d900.e 989 kB/s | 49 kB 00:00 2024-05-27T07:10:57.989 INFO:teuthology.orchestra.run.smithi173.stdout:(67/159): postgresql-13.14-1.el9.x86_64.rpm 3.5 MB/s | 1.6 MB 00:00 2024-05-27T07:10:58.006 INFO:teuthology.orchestra.run.smithi105.stdout:(34/159): ceph-prometheus-alerts-19.0.0-3872.g3 396 kB/s | 16 kB 00:00 2024-05-27T07:10:58.057 INFO:teuthology.orchestra.run.smithi105.stdout:(35/159): ceph-volume-19.0.0-3872.g38f9d900.el9 5.3 MB/s | 273 kB 00:00 2024-05-27T07:10:58.157 INFO:teuthology.orchestra.run.smithi105.stdout:(36/159): cephadm-19.0.0-3872.g38f9d900.el9.noa 7.5 MB/s | 764 kB 00:00 2024-05-27T07:10:58.207 INFO:teuthology.orchestra.run.smithi105.stdout:(37/159): cephfs-top-19.0.0-3872.g38f9d900.el9. 365 kB/s | 18 kB 00:00 2024-05-27T07:10:58.264 INFO:teuthology.orchestra.run.smithi173.stdout:(68/159): postgresql-contrib-13.14-1.el9.x86_64 3.2 MB/s | 911 kB 00:00 2024-05-27T07:10:58.273 INFO:teuthology.orchestra.run.smithi052.stdout:(62/159): libtool-2.4.6-45.el9.x86_64.rpm 496 kB/s | 582 kB 00:01 2024-05-27T07:10:58.316 INFO:teuthology.orchestra.run.smithi105.stdout:(38/159): fuse-2.9.9-15.el9.x86_64.rpm 742 kB/s | 80 kB 00:00 2024-05-27T07:10:58.331 INFO:teuthology.orchestra.run.smithi173.stdout:(69/159): postgresql-private-libs-13.14-1.el9.x 2.0 MB/s | 133 kB 00:00 2024-05-27T07:10:58.348 INFO:teuthology.orchestra.run.smithi052.stdout:(63/159): libzstd-devel-1.5.1-2.el9.x86_64.rpm 61 kB/s | 47 kB 00:00 2024-05-27T07:10:58.398 INFO:teuthology.orchestra.run.smithi173.stdout:(70/159): openssl-devel-3.2.1-1.el9.x86_64.rpm 4.3 MB/s | 4.4 MB 00:01 2024-05-27T07:10:58.524 INFO:teuthology.orchestra.run.smithi173.stdout:(71/159): openblas-openmp-0.3.21-2.el9.x86_64.r 4.1 MB/s | 4.7 MB 00:01 2024-05-27T07:10:58.582 INFO:teuthology.orchestra.run.smithi173.stdout:(72/159): python3-attrs-20.3.0-7.el9.noarch.rpm 1.5 MB/s | 87 kB 00:00 2024-05-27T07:10:58.608 INFO:teuthology.orchestra.run.smithi105.stdout:(39/159): ceph-osd-19.0.0-3872.g38f9d900.el9.x8 2.4 MB/s | 18 MB 00:07 2024-05-27T07:10:58.642 INFO:teuthology.orchestra.run.smithi105.stdout:(40/159): libgfortran-11.4.1-3.el9.x86_64.rpm 2.4 MB/s | 803 kB 00:00 2024-05-27T07:10:58.682 INFO:teuthology.orchestra.run.smithi173.stdout:(73/159): protobuf-3.14.0-13.el9.x86_64.rpm 3.6 MB/s | 1.0 MB 00:00 2024-05-27T07:10:58.684 INFO:teuthology.orchestra.run.smithi105.stdout:(41/159): python3-cffi-1.14.5-5.el9.x86_64.rpm 5.9 MB/s | 253 kB 00:00 2024-05-27T07:10:58.759 INFO:teuthology.orchestra.run.smithi105.stdout:(42/159): python3-cryptography-36.0.1-4.el9.x86 16 MB/s | 1.2 MB 00:00 2024-05-27T07:10:58.784 INFO:teuthology.orchestra.run.smithi105.stdout:(43/159): libquadmath-11.4.1-3.el9.x86_64.rpm 1.1 MB/s | 195 kB 00:00 2024-05-27T07:10:58.810 INFO:teuthology.orchestra.run.smithi105.stdout:(44/159): python3-ply-3.11-14.el9.noarch.rpm 2.1 MB/s | 106 kB 00:00 2024-05-27T07:10:58.816 INFO:teuthology.orchestra.run.smithi173.stdout:(74/159): python3-colorama-0.4.6-3.el9.noarch.r 458 kB/s | 61 kB 00:00 2024-05-27T07:10:58.835 INFO:teuthology.orchestra.run.smithi105.stdout:(45/159): python3-pycparser-2.20-6.el9.noarch.r 2.6 MB/s | 135 kB 00:00 2024-05-27T07:10:58.860 INFO:teuthology.orchestra.run.smithi105.stdout:(46/159): python3-requests-2.25.1-8.el9.noarch. 2.4 MB/s | 125 kB 00:00 2024-05-27T07:10:58.882 INFO:teuthology.orchestra.run.smithi052.stdout:(64/159): lttng-ust-2.12.0-6.el9.x86_64.rpm 308 kB/s | 292 kB 00:00 2024-05-27T07:10:58.885 INFO:teuthology.orchestra.run.smithi105.stdout:(47/159): python3-urllib3-1.26.5-5.el9.noarch.r 4.2 MB/s | 215 kB 00:00 2024-05-27T07:10:58.966 INFO:teuthology.orchestra.run.smithi173.stdout:(75/159): python3-devel-3.9.18-3.el9.x86_64.rpm 1.6 MB/s | 244 kB 00:00 2024-05-27T07:10:59.090 INFO:teuthology.orchestra.run.smithi052.stdout:(65/159): openblas-0.3.21-2.el9.x86_64.rpm 47 kB/s | 35 kB 00:00 2024-05-27T07:10:59.117 INFO:teuthology.orchestra.run.smithi173.stdout:(76/159): python3-jinja2-2.11.3-6.el9.noarch.rp 1.6 MB/s | 248 kB 00:00 2024-05-27T07:10:59.192 INFO:teuthology.orchestra.run.smithi173.stdout:(77/159): python3-jsonpatch-1.21-16.el9.noarch. 342 kB/s | 26 kB 00:00 2024-05-27T07:10:59.249 INFO:teuthology.orchestra.run.smithi052.stdout:(66/159): lua-5.4.4-4.el9.x86_64.rpm 193 kB/s | 188 kB 00:00 2024-05-27T07:10:59.275 INFO:teuthology.orchestra.run.smithi173.stdout:(78/159): python3-jsonpointer-2.0-4.el9.noarch. 230 kB/s | 19 kB 00:00 2024-05-27T07:10:59.286 INFO:teuthology.orchestra.run.smithi105.stdout:(48/159): automake-1.16.2-8.el9.noarch.rpm 1.6 MB/s | 689 kB 00:00 2024-05-27T07:10:59.353 INFO:teuthology.orchestra.run.smithi105.stdout:(49/159): bison-3.7.4-5.el9.x86_64.rpm 2.0 MB/s | 943 kB 00:00 2024-05-27T07:10:59.359 INFO:teuthology.orchestra.run.smithi173.stdout:(79/159): python3-mako-1.1.4-6.el9.noarch.rpm 2.0 MB/s | 172 kB 00:00 2024-05-27T07:10:59.378 INFO:teuthology.orchestra.run.smithi105.stdout:(50/159): boost-program-options-1.75.0-8.el9.x8 1.1 MB/s | 107 kB 00:00 2024-05-27T07:10:59.403 INFO:teuthology.orchestra.run.smithi105.stdout:(51/159): elfutils-libelf-devel-0.191-4.el9.x86 431 kB/s | 22 kB 00:00 2024-05-27T07:10:59.454 INFO:teuthology.orchestra.run.smithi105.stdout:(52/159): flex-2.6.4-9.el9.x86_64.rpm 4.1 MB/s | 313 kB 00:00 2024-05-27T07:10:59.479 INFO:teuthology.orchestra.run.smithi105.stdout:(53/159): flexiblas-3.0.4-8.el9.x86_64.rpm 435 kB/s | 33 kB 00:00 2024-05-27T07:10:59.510 INFO:teuthology.orchestra.run.smithi173.stdout:(80/159): python3-markupsafe-1.1.1-12.el9.x86_6 231 kB/s | 35 kB 00:00 2024-05-27T07:10:59.529 INFO:teuthology.orchestra.run.smithi105.stdout:(54/159): flexiblas-openblas-openmp-3.0.4-8.el9 364 kB/s | 18 kB 00:00 2024-05-27T07:10:59.669 INFO:teuthology.orchestra.run.smithi173.stdout:(81/159): postgresql-server-13.14-1.el9.x86_64. 4.3 MB/s | 5.8 MB 00:01 2024-05-27T07:10:59.696 INFO:teuthology.orchestra.run.smithi105.stdout:(55/159): libaio-devel-0.3.111-13.el9.x86_64.rp 67 kB/s | 11 kB 00:00 2024-05-27T07:10:59.755 INFO:teuthology.orchestra.run.smithi105.stdout:(56/159): libpmemobj-1.12.1-1.el9.x86_64.rpm 2.7 MB/s | 160 kB 00:00 2024-05-27T07:10:59.760 INFO:teuthology.orchestra.run.smithi173.stdout:(82/159): python3-numpy-f2py-1.20.1-5.el9.x86_6 2.7 MB/s | 250 kB 00:00 2024-05-27T07:10:59.805 INFO:teuthology.orchestra.run.smithi105.stdout:(57/159): librabbitmq-0.11.0-7.el9.x86_64.rpm 908 kB/s | 45 kB 00:00 2024-05-27T07:10:59.819 INFO:teuthology.orchestra.run.smithi173.stdout:(83/159): python3-oauthlib-3.1.1-5.el9.noarch.r 3.7 MB/s | 222 kB 00:00 2024-05-27T07:10:59.855 INFO:teuthology.orchestra.run.smithi105.stdout:(58/159): flexiblas-netlib-3.0.4-8.el9.x86_64.r 7.4 MB/s | 3.0 MB 00:00 2024-05-27T07:10:59.878 INFO:teuthology.orchestra.run.smithi173.stdout:(84/159): python3-packaging-20.9-5.el9.noarch.r 1.3 MB/s | 77 kB 00:00 2024-05-27T07:10:59.931 INFO:teuthology.orchestra.run.smithi105.stdout:(59/159): librdkafka-1.6.1-102.el9.x86_64.rpm 5.2 MB/s | 662 kB 00:00 2024-05-27T07:10:59.944 INFO:teuthology.orchestra.run.smithi173.stdout:(85/159): python3-babel-2.9.1-2.el9.noarch.rpm 4.4 MB/s | 6.0 MB 00:01 2024-05-27T07:10:59.956 INFO:teuthology.orchestra.run.smithi105.stdout:(60/159): libtool-2.4.6-45.el9.x86_64.rpm 5.7 MB/s | 582 kB 00:00 2024-05-27T07:10:59.970 INFO:teuthology.orchestra.run.smithi173.stdout:(86/159): python3-prettytable-0.7.2-27.el9.noar 460 kB/s | 42 kB 00:00 2024-05-27T07:10:59.981 INFO:teuthology.orchestra.run.smithi105.stdout:(61/159): libuuid-devel-2.37.4-18.el9.x86_64.rp 507 kB/s | 25 kB 00:00 2024-05-27T07:11:00.015 INFO:teuthology.orchestra.run.smithi105.stdout:(62/159): libzstd-devel-1.5.1-2.el9.x86_64.rpm 809 kB/s | 47 kB 00:00 2024-05-27T07:11:00.028 INFO:teuthology.orchestra.run.smithi173.stdout:(87/159): python3-pyasn1-0.4.8-6.el9.noarch.rpm 2.7 MB/s | 159 kB 00:00 2024-05-27T07:11:00.040 INFO:teuthology.orchestra.run.smithi105.stdout:(63/159): lttng-ust-2.12.0-6.el9.x86_64.rpm 4.9 MB/s | 292 kB 00:00 2024-05-27T07:11:00.054 INFO:teuthology.orchestra.run.smithi173.stdout:(88/159): python3-protobuf-3.14.0-13.el9.noarch 2.4 MB/s | 269 kB 00:00 2024-05-27T07:11:00.073 INFO:teuthology.orchestra.run.smithi105.stdout:(64/159): lua-5.4.4-4.el9.x86_64.rpm 3.1 MB/s | 188 kB 00:00 2024-05-27T07:11:00.099 INFO:teuthology.orchestra.run.smithi105.stdout:(65/159): openblas-0.3.21-2.el9.x86_64.rpm 601 kB/s | 35 kB 00:00 2024-05-27T07:11:00.121 INFO:teuthology.orchestra.run.smithi173.stdout:(89/159): python3-pytz-2021.1-5.el9.noarch.rpm 765 kB/s | 51 kB 00:00 2024-05-27T07:11:00.146 INFO:teuthology.orchestra.run.smithi173.stdout:(90/159): python3-pyasn1-modules-0.4.8-6.el9.no 2.3 MB/s | 279 kB 00:00 2024-05-27T07:11:00.188 INFO:teuthology.orchestra.run.smithi173.stdout:(91/159): python3-requests-oauthlib-1.3.0-12.el 803 kB/s | 54 kB 00:00 2024-05-27T07:11:00.241 INFO:teuthology.orchestra.run.smithi052.stdout:(67/159): openblas-openmp-0.3.21-2.el9.x86_64.r 3.5 MB/s | 4.7 MB 00:01 2024-05-27T07:11:00.255 INFO:teuthology.orchestra.run.smithi173.stdout:(92/159): python3-toml-0.10.2-6.el9.noarch.rpm 624 kB/s | 42 kB 00:00 2024-05-27T07:11:00.330 INFO:teuthology.orchestra.run.smithi173.stdout:(93/159): python3-wcwidth-0.2.5-8.el9.noarch.rp 588 kB/s | 44 kB 00:00 2024-05-27T07:11:00.383 INFO:teuthology.orchestra.run.smithi052.stdout:(68/159): postgresql-13.14-1.el9.x86_64.rpm 1.4 MB/s | 1.6 MB 00:01 2024-05-27T07:11:00.447 INFO:teuthology.orchestra.run.smithi173.stdout:(94/159): qatlib-24.02.0-1.el9.x86_64.rpm 1.8 MB/s | 221 kB 00:00 2024-05-27T07:11:00.450 INFO:teuthology.orchestra.run.smithi052.stdout:(69/159): openssl-devel-3.2.1-1.el9.x86_64.rpm 3.3 MB/s | 4.4 MB 00:01 2024-05-27T07:11:00.474 INFO:teuthology.orchestra.run.smithi105.stdout:(66/159): openblas-openmp-0.3.21-2.el9.x86_64.r 12 MB/s | 4.7 MB 00:00 2024-05-27T07:11:00.506 INFO:teuthology.orchestra.run.smithi173.stdout:(95/159): qatlib-service-24.02.0-1.el9.x86_64.r 615 kB/s | 36 kB 00:00 2024-05-27T07:11:00.573 INFO:teuthology.orchestra.run.smithi173.stdout:(96/159): qatzip-libs-1.2.0-1.el9.x86_64.rpm 713 kB/s | 47 kB 00:00 2024-05-27T07:11:00.641 INFO:teuthology.orchestra.run.smithi105.stdout:(67/159): openssl-devel-3.2.1-1.el9.x86_64.rpm 8.2 MB/s | 4.4 MB 00:00 2024-05-27T07:11:00.675 INFO:teuthology.orchestra.run.smithi105.stdout:(68/159): postgresql-13.14-1.el9.x86_64.rpm 7.9 MB/s | 1.6 MB 00:00 2024-05-27T07:11:00.681 INFO:teuthology.orchestra.run.smithi173.stdout:(97/159): python3-numpy-1.20.1-5.el9.x86_64.rpm 4.4 MB/s | 5.1 MB 00:01 2024-05-27T07:11:00.706 INFO:teuthology.orchestra.run.smithi173.stdout:(98/159): socat-1.7.4.1-5.el9.x86_64.rpm 2.2 MB/s | 305 kB 00:00 2024-05-27T07:11:00.732 INFO:teuthology.orchestra.run.smithi173.stdout:(99/159): uuid-1.6.2-55.el9.x86_64.rpm 1.1 MB/s | 58 kB 00:00 2024-05-27T07:11:00.733 INFO:teuthology.orchestra.run.smithi105.stdout:(69/159): postgresql-contrib-13.14-1.el9.x86_64 9.7 MB/s | 911 kB 00:00 2024-05-27T07:11:00.759 INFO:teuthology.orchestra.run.smithi105.stdout:(70/159): postgresql-private-libs-13.14-1.el9.x 1.6 MB/s | 133 kB 00:00 2024-05-27T07:11:00.766 INFO:teuthology.orchestra.run.smithi173.stdout:(100/159): xfsprogs-devel-6.3.0-1.el9.x86_64.rp 1.6 MB/s | 94 kB 00:00 2024-05-27T07:11:00.791 INFO:teuthology.orchestra.run.smithi173.stdout:(101/159): xmlstarlet-1.6.1-20.el9.x86_64.rpm 1.1 MB/s | 64 kB 00:00 2024-05-27T07:11:00.824 INFO:teuthology.orchestra.run.smithi173.stdout:(102/159): zlib-devel-1.2.11-41.el9.x86_64.rpm 766 kB/s | 45 kB 00:00 2024-05-27T07:11:00.850 INFO:teuthology.orchestra.run.smithi173.stdout:(103/159): lua-devel-5.4.4-4.el9.x86_64.rpm 381 kB/s | 22 kB 00:00 2024-05-27T07:11:00.892 INFO:teuthology.orchestra.run.smithi105.stdout:(71/159): protobuf-3.14.0-13.el9.x86_64.rpm 7.6 MB/s | 1.0 MB 00:00 2024-05-27T07:11:00.960 INFO:teuthology.orchestra.run.smithi105.stdout:(72/159): python3-attrs-20.3.0-7.el9.noarch.rpm 1.3 MB/s | 87 kB 00:00 2024-05-27T07:11:00.992 INFO:teuthology.orchestra.run.smithi173.stdout:(104/159): protobuf-compiler-3.14.0-13.el9.x86_ 5.1 MB/s | 863 kB 00:00 2024-05-27T07:11:01.142 INFO:teuthology.orchestra.run.smithi173.stdout:(105/159): abseil-cpp-20211102.0-3.el9.x86_64.r 1.8 MB/s | 548 kB 00:00 2024-05-27T07:11:01.192 INFO:teuthology.orchestra.run.smithi173.stdout:(106/159): grpc-data-1.46.7-10.el9.noarch.rpm 388 kB/s | 19 kB 00:00 2024-05-27T07:11:01.259 INFO:teuthology.orchestra.run.smithi173.stdout:(107/159): gperftools-libs-2.9.1-2.el9.x86_64.r 1.1 MB/s | 309 kB 00:00 2024-05-27T07:11:01.268 INFO:teuthology.orchestra.run.smithi105.stdout:(73/159): postgresql-server-13.14-1.el9.x86_64. 11 MB/s | 5.8 MB 00:00 2024-05-27T07:11:01.318 INFO:teuthology.orchestra.run.smithi173.stdout:(108/159): libarrow-doc-9.0.0-11.el9.noarch.rpm 446 kB/s | 26 kB 00:00 2024-05-27T07:11:01.333 INFO:teuthology.orchestra.run.smithi052.stdout:(70/159): postgresql-private-libs-13.14-1.el9.x 140 kB/s | 133 kB 00:00 2024-05-27T07:11:01.343 INFO:teuthology.orchestra.run.smithi105.stdout:(74/159): python3-colorama-0.4.6-3.el9.noarch.r 818 kB/s | 61 kB 00:00 2024-05-27T07:11:01.385 INFO:teuthology.orchestra.run.smithi173.stdout:(109/159): liboath-2.6.7-2.el9.x86_64.rpm 725 kB/s | 49 kB 00:00 2024-05-27T07:11:01.452 INFO:teuthology.orchestra.run.smithi173.stdout:(110/159): libarrow-9.0.0-11.el9.x86_64.rpm 17 MB/s | 4.4 MB 00:00 2024-05-27T07:11:01.469 INFO:teuthology.orchestra.run.smithi173.stdout:(111/159): libunwind-1.6.2-1.el9.x86_64.rpm 807 kB/s | 67 kB 00:00 2024-05-27T07:11:01.511 INFO:teuthology.orchestra.run.smithi173.stdout:(112/159): luarocks-3.9.2-5.el9.noarch.rpm 2.5 MB/s | 151 kB 00:00 2024-05-27T07:11:01.567 INFO:teuthology.orchestra.run.smithi052.stdout:(71/159): postgresql-contrib-13.14-1.el9.x86_64 687 kB/s | 911 kB 00:01 2024-05-27T07:11:01.568 INFO:teuthology.orchestra.run.smithi105.stdout:(75/159): python3-babel-2.9.1-2.el9.noarch.rpm 9.8 MB/s | 6.0 MB 00:00 2024-05-27T07:11:01.586 INFO:teuthology.orchestra.run.smithi173.stdout:(113/159): python3-asyncssh-2.13.2-5.el9.noarch 7.1 MB/s | 548 kB 00:00 2024-05-27T07:11:01.644 INFO:teuthology.orchestra.run.smithi105.stdout:(76/159): python3-jinja2-2.11.3-6.el9.noarch.rp 3.2 MB/s | 248 kB 00:00 2024-05-27T07:11:01.653 INFO:teuthology.orchestra.run.smithi173.stdout:(114/159): parquet-libs-9.0.0-11.el9.x86_64.rpm 4.5 MB/s | 839 kB 00:00 2024-05-27T07:11:01.678 INFO:teuthology.orchestra.run.smithi173.stdout:(115/159): python3-bcrypt-3.2.2-1.el9.x86_64.rp 473 kB/s | 43 kB 00:00 2024-05-27T07:11:01.704 INFO:teuthology.orchestra.run.smithi173.stdout:(116/159): python3-cachetools-4.2.4-1.el9.noarc 638 kB/s | 32 kB 00:00 2024-05-27T07:11:01.719 INFO:teuthology.orchestra.run.smithi105.stdout:(77/159): python3-jsonpatch-1.21-16.el9.noarch. 341 kB/s | 26 kB 00:00 2024-05-27T07:11:01.729 INFO:teuthology.orchestra.run.smithi173.stdout:(117/159): python3-certifi-2021.10.8-2.el9.noar 298 kB/s | 15 kB 00:00 2024-05-27T07:11:01.778 INFO:teuthology.orchestra.run.smithi105.stdout:(78/159): python3-jsonpointer-2.0-4.el9.noarch. 328 kB/s | 19 kB 00:00 2024-05-27T07:11:01.779 INFO:teuthology.orchestra.run.smithi173.stdout:(118/159): python3-cheroot-8.6.0-4.el9.noarch.r 2.2 MB/s | 172 kB 00:00 2024-05-27T07:11:01.805 INFO:teuthology.orchestra.run.smithi173.stdout:(119/159): python3-cherrypy-18.6.1-2.el9.noarch 4.6 MB/s | 358 kB 00:00 2024-05-27T07:11:01.845 INFO:teuthology.orchestra.run.smithi105.stdout:(79/159): python3-mako-1.1.4-6.el9.noarch.rpm 2.5 MB/s | 172 kB 00:00 2024-05-27T07:11:01.863 INFO:teuthology.orchestra.run.smithi173.stdout:(120/159): python3-google-auth-2.29.0-1.el9.noa 3.4 MB/s | 206 kB 00:00 2024-05-27T07:11:01.888 INFO:teuthology.orchestra.run.smithi173.stdout:(121/159): python3-cmd2-2.4.2-2.el9.noarch.rpm 2.7 MB/s | 303 kB 00:00 2024-05-27T07:11:01.912 INFO:teuthology.orchestra.run.smithi105.stdout:(80/159): python3-markupsafe-1.1.1-12.el9.x86_6 520 kB/s | 35 kB 00:00 2024-05-27T07:11:01.955 INFO:teuthology.orchestra.run.smithi173.stdout:(122/159): python3-grpcio-1.46.7-10.el9.x86_64. 22 MB/s | 2.0 MB 00:00 2024-05-27T07:11:01.980 INFO:teuthology.orchestra.run.smithi173.stdout:(123/159): python3-grpcio-tools-1.46.7-10.el9.x 1.5 MB/s | 144 kB 00:00 2024-05-27T07:11:02.006 INFO:teuthology.orchestra.run.smithi173.stdout:(124/159): python3-jaraco-8.2.1-3.el9.noarch.rp 213 kB/s | 11 kB 00:00 2024-05-27T07:11:02.031 INFO:teuthology.orchestra.run.smithi173.stdout:(125/159): python3-jaraco-classes-3.2.1-5.el9.n 350 kB/s | 18 kB 00:00 2024-05-27T07:11:02.037 INFO:teuthology.orchestra.run.smithi105.stdout:(81/159): python3-devel-3.9.18-3.el9.x86_64.rpm 352 kB/s | 244 kB 00:00 2024-05-27T07:11:02.056 INFO:teuthology.orchestra.run.smithi173.stdout:(126/159): python3-jaraco-collections-3.0.0-8.e 460 kB/s | 23 kB 00:00 2024-05-27T07:11:02.081 INFO:teuthology.orchestra.run.smithi173.stdout:(127/159): python3-jaraco-functools-3.5.0-2.el9 389 kB/s | 19 kB 00:00 2024-05-27T07:11:02.107 INFO:teuthology.orchestra.run.smithi173.stdout:(128/159): python3-jaraco-text-3.2.0-6.el9.noar 390 kB/s | 20 kB 00:00 2024-05-27T07:11:02.137 INFO:teuthology.orchestra.run.smithi105.stdout:(82/159): python3-numpy-f2py-1.20.1-5.el9.x86_6 2.4 MB/s | 250 kB 00:00 2024-05-27T07:11:02.165 INFO:teuthology.orchestra.run.smithi173.stdout:(129/159): python3-kubernetes-26.1.0-2.el9.noar 12 MB/s | 1.0 MB 00:00 2024-05-27T07:11:02.191 INFO:teuthology.orchestra.run.smithi173.stdout:(130/159): python3-logutils-0.3.5-21.el9.noarch 553 kB/s | 46 kB 00:00 2024-05-27T07:11:02.196 INFO:teuthology.orchestra.run.smithi105.stdout:(83/159): python3-oauthlib-3.1.1-5.el9.noarch.r 3.7 MB/s | 222 kB 00:00 2024-05-27T07:11:02.241 INFO:teuthology.orchestra.run.smithi173.stdout:(131/159): python3-more-itertools-8.12.0-2.el9. 1.0 MB/s | 79 kB 00:00 2024-05-27T07:11:02.266 INFO:teuthology.orchestra.run.smithi173.stdout:(132/159): python3-natsort-7.1.1-5.el9.noarch.r 765 kB/s | 58 kB 00:00 2024-05-27T07:11:02.271 INFO:teuthology.orchestra.run.smithi105.stdout:(84/159): python3-packaging-20.9-5.el9.noarch.r 1.0 MB/s | 77 kB 00:00 2024-05-27T07:11:02.317 INFO:teuthology.orchestra.run.smithi173.stdout:(133/159): python3-portend-3.1.0-2.el9.noarch.r 329 kB/s | 16 kB 00:00 2024-05-27T07:11:02.342 INFO:teuthology.orchestra.run.smithi173.stdout:(134/159): python3-pecan-1.4.2-3.el9.noarch.rpm 2.6 MB/s | 272 kB 00:00 2024-05-27T07:11:02.355 INFO:teuthology.orchestra.run.smithi105.stdout:(85/159): python3-prettytable-0.7.2-27.el9.noar 508 kB/s | 42 kB 00:00 2024-05-27T07:11:02.401 INFO:teuthology.orchestra.run.smithi173.stdout:(135/159): python3-pyperclip-1.8.2-2.el9.noarch 485 kB/s | 28 kB 00:00 2024-05-27T07:11:02.409 INFO:teuthology.orchestra.run.smithi052.stdout:(72/159): postgresql-server-13.14-1.el9.x86_64. 3.0 MB/s | 5.8 MB 00:01 2024-05-27T07:11:02.426 INFO:teuthology.orchestra.run.smithi173.stdout:(136/159): python3-pyOpenSSL-21.0.0-1.el9.noarc 826 kB/s | 90 kB 00:00 2024-05-27T07:11:02.430 INFO:teuthology.orchestra.run.smithi105.stdout:(86/159): python3-protobuf-3.14.0-13.el9.noarch 3.5 MB/s | 269 kB 00:00 2024-05-27T07:11:02.451 INFO:teuthology.orchestra.run.smithi173.stdout:(137/159): python3-repoze-lru-0.7-16.el9.noarch 610 kB/s | 31 kB 00:00 2024-05-27T07:11:02.494 INFO:teuthology.orchestra.run.smithi173.stdout:(138/159): python3-routes-2.5.1-5.el9.noarch.rp 2.7 MB/s | 188 kB 00:00 2024-05-27T07:11:02.497 INFO:teuthology.orchestra.run.smithi105.stdout:(87/159): python3-pyasn1-0.4.8-6.el9.noarch.rpm 2.3 MB/s | 159 kB 00:00 2024-05-27T07:11:02.509 INFO:teuthology.orchestra.run.smithi052.stdout:(73/159): python3-attrs-20.3.0-7.el9.noarch.rpm 93 kB/s | 87 kB 00:00 2024-05-27T07:11:02.510 INFO:teuthology.orchestra.run.smithi173.stdout:(139/159): python3-rsa-4.9-2.el9.noarch.rpm 1.0 MB/s | 59 kB 00:00 2024-05-27T07:11:02.544 INFO:teuthology.orchestra.run.smithi173.stdout:(140/159): python3-tempora-5.0.0-2.el9.noarch.r 716 kB/s | 36 kB 00:00 2024-05-27T07:11:02.569 INFO:teuthology.orchestra.run.smithi173.stdout:(141/159): python3-typing-extensions-4.4.0-2.el 869 kB/s | 51 kB 00:00 2024-05-27T07:11:02.581 INFO:teuthology.orchestra.run.smithi105.stdout:(88/159): python3-pyasn1-modules-0.4.8-6.el9.no 3.3 MB/s | 279 kB 00:00 2024-05-27T07:11:02.611 INFO:teuthology.orchestra.run.smithi173.stdout:(142/159): python3-webob-1.8.7-6.el9.noarch.rpm 3.3 MB/s | 230 kB 00:00 2024-05-27T07:11:02.645 INFO:teuthology.orchestra.run.smithi173.stdout:(143/159): python3-websocket-client-1.2.3-2.el9 1.2 MB/s | 90 kB 00:00 2024-05-27T07:11:02.672 INFO:teuthology.orchestra.run.smithi105.stdout:(89/159): python3-pytz-2021.1-5.el9.noarch.rpm 557 kB/s | 51 kB 00:00 2024-05-27T07:11:02.703 INFO:teuthology.orchestra.run.smithi173.stdout:(144/159): python3-zc-lockfile-2.0-10.el9.noarc 343 kB/s | 20 kB 00:00 2024-05-27T07:11:02.729 INFO:teuthology.orchestra.run.smithi173.stdout:(145/159): python3-werkzeug-2.0.3-3.el9.1.noarc 3.6 MB/s | 427 kB 00:00 2024-05-27T07:11:02.762 INFO:teuthology.orchestra.run.smithi173.stdout:(146/159): re2-20211101-3.el9.x86_64.rpm 3.2 MB/s | 192 kB 00:00 2024-05-27T07:11:02.781 INFO:teuthology.orchestra.run.smithi105.stdout:(90/159): python3-numpy-1.20.1-5.el9.x86_64.rpm 5.9 MB/s | 5.1 MB 00:00 2024-05-27T07:11:02.807 INFO:teuthology.orchestra.run.smithi105.stdout:(91/159): python3-requests-oauthlib-1.3.0-12.el 401 kB/s | 54 kB 00:00 2024-05-27T07:11:02.857 INFO:teuthology.orchestra.run.smithi173.stdout:(147/159): thrift-0.15.0-2.el9.x86_64.rpm 12 MB/s | 1.6 MB 00:00 2024-05-27T07:11:02.874 INFO:teuthology.orchestra.run.smithi105.stdout:(92/159): python3-toml-0.10.2-6.el9.noarch.rpm 625 kB/s | 42 kB 00:00 2024-05-27T07:11:02.901 INFO:teuthology.orchestra.run.smithi052.stdout:(74/159): protobuf-3.14.0-13.el9.x86_64.rpm 659 kB/s | 1.0 MB 00:01 2024-05-27T07:11:02.932 INFO:teuthology.orchestra.run.smithi105.stdout:(93/159): python3-wcwidth-0.2.5-8.el9.noarch.rp 757 kB/s | 44 kB 00:00 2024-05-27T07:11:02.982 INFO:teuthology.orchestra.run.smithi105.stdout:(94/159): qatlib-24.02.0-1.el9.x86_64.rpm 4.3 MB/s | 221 kB 00:00 2024-05-27T07:11:03.033 INFO:teuthology.orchestra.run.smithi105.stdout:(95/159): qatlib-service-24.02.0-1.el9.x86_64.r 718 kB/s | 36 kB 00:00 2024-05-27T07:11:03.092 INFO:teuthology.orchestra.run.smithi105.stdout:(96/159): qatzip-libs-1.2.0-1.el9.x86_64.rpm 810 kB/s | 47 kB 00:00 2024-05-27T07:11:03.142 INFO:teuthology.orchestra.run.smithi105.stdout:(97/159): socat-1.7.4.1-5.el9.x86_64.rpm 6.0 MB/s | 305 kB 00:00 2024-05-27T07:11:03.192 INFO:teuthology.orchestra.run.smithi105.stdout:(98/159): uuid-1.6.2-55.el9.x86_64.rpm 1.1 MB/s | 58 kB 00:00 2024-05-27T07:11:03.243 INFO:teuthology.orchestra.run.smithi105.stdout:(99/159): xfsprogs-devel-6.3.0-1.el9.x86_64.rpm 1.8 MB/s | 94 kB 00:00 2024-05-27T07:11:03.271 INFO:teuthology.orchestra.run.smithi173.stdout:(148/159): librados2-19.0.0-3872.g38f9d900.el9. 6.5 MB/s | 3.3 MB 00:00 2024-05-27T07:11:03.285 INFO:teuthology.orchestra.run.smithi052.stdout:(75/159): python3-colorama-0.4.6-3.el9.noarch.r 79 kB/s | 61 kB 00:00 2024-05-27T07:11:03.293 INFO:teuthology.orchestra.run.smithi105.stdout:(100/159): xmlstarlet-1.6.1-20.el9.x86_64.rpm 1.2 MB/s | 64 kB 00:00 2024-05-27T07:11:03.330 INFO:teuthology.orchestra.run.smithi173.stdout:(149/159): librbd1-19.0.0-3872.g38f9d900.el9.x8 6.6 MB/s | 3.1 MB 00:00 2024-05-27T07:11:03.343 INFO:teuthology.orchestra.run.smithi105.stdout:(101/159): zlib-devel-1.2.11-41.el9.x86_64.rpm 897 kB/s | 45 kB 00:00 2024-05-27T07:11:03.427 INFO:teuthology.orchestra.run.smithi105.stdout:(102/159): lua-devel-5.4.4-4.el9.x86_64.rpm 268 kB/s | 22 kB 00:00 2024-05-27T07:11:03.472 INFO:teuthology.orchestra.run.smithi173.stdout:(150/159): NetworkManager-1.47.90-1.el9.x86_64. 12 MB/s | 2.3 MB 00:00 2024-05-27T07:11:03.552 INFO:teuthology.orchestra.run.smithi105.stdout:(103/159): protobuf-compiler-3.14.0-13.el9.x86_ 6.7 MB/s | 863 kB 00:00 2024-05-27T07:11:03.630 INFO:teuthology.orchestra.run.smithi173.stdout:(151/159): python3-scipy-1.6.2-8.el9.x86_64.rpm 4.7 MB/s | 16 MB 00:03 2024-05-27T07:11:03.655 INFO:teuthology.orchestra.run.smithi173.stdout:(152/159): NetworkManager-team-1.47.90-1.el9.x8 226 kB/s | 41 kB 00:00 2024-05-27T07:11:03.701 INFO:teuthology.orchestra.run.smithi052.stdout:(76/159): python3-devel-3.9.18-3.el9.x86_64.rpm 305 kB/s | 244 kB 00:00 2024-05-27T07:11:03.714 INFO:teuthology.orchestra.run.smithi173.stdout:(153/159): elfutils-debuginfod-client-0.191-4.e 630 kB/s | 37 kB 00:00 2024-05-27T07:11:03.773 INFO:teuthology.orchestra.run.smithi173.stdout:(154/159): NetworkManager-libnm-1.47.90-1.el9.x 4.1 MB/s | 1.8 MB 00:00 2024-05-27T07:11:03.777 INFO:teuthology.orchestra.run.smithi052.stdout:(77/159): python3-babel-2.9.1-2.el9.noarch.rpm 4.4 MB/s | 6.0 MB 00:01 2024-05-27T07:11:03.798 INFO:teuthology.orchestra.run.smithi173.stdout:(155/159): NetworkManager-tui-1.47.90-1.el9.x86 1.4 MB/s | 248 kB 00:00 2024-05-27T07:11:03.823 INFO:teuthology.orchestra.run.smithi173.stdout:(156/159): elfutils-libelf-0.191-4.el9.x86_64.r 1.9 MB/s | 207 kB 00:00 2024-05-27T07:11:03.836 INFO:teuthology.orchestra.run.smithi105.stdout:(104/159): abseil-cpp-20211102.0-3.el9.x86_64.r 1.9 MB/s | 548 kB 00:00 2024-05-27T07:11:03.848 INFO:teuthology.orchestra.run.smithi173.stdout:(157/159): elfutils-libs-0.191-4.el9.x86_64.rpm 3.4 MB/s | 260 kB 00:00 2024-05-27T07:11:03.936 INFO:teuthology.orchestra.run.smithi105.stdout:(105/159): gperftools-libs-2.9.1-2.el9.x86_64.r 3.0 MB/s | 309 kB 00:00 2024-05-27T07:11:04.020 INFO:teuthology.orchestra.run.smithi105.stdout:(106/159): grpc-data-1.46.7-10.el9.noarch.rpm 233 kB/s | 19 kB 00:00 2024-05-27T07:11:04.032 INFO:teuthology.orchestra.run.smithi173.stdout:(158/159): openssl-3.2.1-1.el9.x86_64.rpm 6.5 MB/s | 1.3 MB 00:00 2024-05-27T07:11:04.082 INFO:teuthology.orchestra.run.smithi173.stdout:(159/159): openssl-libs-3.2.1-1.el9.x86_64.rpm 9.3 MB/s | 2.4 MB 00:00 2024-05-27T07:11:04.086 INFO:teuthology.orchestra.run.smithi173.stdout:-------------------------------------------------------------------------------- 2024-05-27T07:11:04.086 INFO:teuthology.orchestra.run.smithi173.stdout:Total 12 MB/s | 219 MB 00:17 2024-05-27T07:11:04.219 INFO:teuthology.orchestra.run.smithi052.stdout:(78/159): python3-jinja2-2.11.3-6.el9.noarch.rp 266 kB/s | 248 kB 00:00 2024-05-27T07:11:04.437 INFO:teuthology.orchestra.run.smithi105.stdout:(107/159): python3-scipy-1.6.2-8.el9.x86_64.rpm 9.8 MB/s | 16 MB 00:01 2024-05-27T07:11:04.452 INFO:teuthology.orchestra.run.smithi052.stdout:(79/159): python3-jsonpatch-1.21-16.el9.noarch. 34 kB/s | 26 kB 00:00 2024-05-27T07:11:04.478 INFO:teuthology.orchestra.run.smithi052.stdout:(80/159): python3-jsonpointer-2.0-4.el9.noarch. 27 kB/s | 19 kB 00:00 2024-05-27T07:11:04.504 INFO:teuthology.orchestra.run.smithi105.stdout:(108/159): libarrow-9.0.0-11.el9.x86_64.rpm 9.1 MB/s | 4.4 MB 00:00 2024-05-27T07:11:04.562 INFO:teuthology.orchestra.run.smithi105.stdout:(109/159): liboath-2.6.7-2.el9.x86_64.rpm 836 kB/s | 49 kB 00:00 2024-05-27T07:11:04.596 INFO:teuthology.orchestra.run.smithi105.stdout:(110/159): libarrow-doc-9.0.0-11.el9.noarch.rpm 164 kB/s | 26 kB 00:00 2024-05-27T07:11:04.621 INFO:teuthology.orchestra.run.smithi105.stdout:(111/159): libunwind-1.6.2-1.el9.x86_64.rpm 1.1 MB/s | 67 kB 00:00 2024-05-27T07:11:04.705 INFO:teuthology.orchestra.run.smithi105.stdout:(112/159): parquet-libs-9.0.0-11.el9.x86_64.rpm 9.8 MB/s | 839 kB 00:00 2024-05-27T07:11:04.755 INFO:teuthology.orchestra.run.smithi105.stdout:(113/159): luarocks-3.9.2-5.el9.noarch.rpm 954 kB/s | 151 kB 00:00 2024-05-27T07:11:04.780 INFO:teuthology.orchestra.run.smithi105.stdout:(114/159): python3-asyncssh-2.13.2-5.el9.noarch 7.1 MB/s | 548 kB 00:00 2024-05-27T07:11:04.822 INFO:teuthology.orchestra.run.smithi105.stdout:(115/159): python3-bcrypt-3.2.2-1.el9.x86_64.rp 649 kB/s | 43 kB 00:00 2024-05-27T07:11:04.848 INFO:teuthology.orchestra.run.smithi105.stdout:(116/159): python3-cachetools-4.2.4-1.el9.noarc 481 kB/s | 32 kB 00:00 2024-05-27T07:11:04.881 INFO:teuthology.orchestra.run.smithi105.stdout:(117/159): python3-certifi-2021.10.8-2.el9.noar 256 kB/s | 15 kB 00:00 2024-05-27T07:11:04.907 INFO:teuthology.orchestra.run.smithi105.stdout:(118/159): python3-cheroot-8.6.0-4.el9.noarch.r 2.9 MB/s | 172 kB 00:00 2024-05-27T07:11:04.965 INFO:teuthology.orchestra.run.smithi105.stdout:(119/159): python3-cmd2-2.4.2-2.el9.noarch.rpm 5.1 MB/s | 303 kB 00:00 2024-05-27T07:11:04.990 INFO:teuthology.orchestra.run.smithi105.stdout:(120/159): python3-cherrypy-18.6.1-2.el9.noarch 3.2 MB/s | 358 kB 00:00 2024-05-27T07:11:05.024 INFO:teuthology.orchestra.run.smithi105.stdout:(121/159): python3-google-auth-2.29.0-1.el9.noa 3.4 MB/s | 206 kB 00:00 2024-05-27T07:11:05.044 INFO:teuthology.orchestra.run.smithi052.stdout:(81/159): python3-mako-1.1.4-6.el9.noarch.rpm 209 kB/s | 172 kB 00:00 2024-05-27T07:11:05.070 INFO:teuthology.orchestra.run.smithi052.stdout:(82/159): python3-markupsafe-1.1.1-12.el9.x86_6 56 kB/s | 35 kB 00:00 2024-05-27T07:11:05.082 INFO:teuthology.orchestra.run.smithi105.stdout:(122/159): python3-grpcio-tools-1.46.7-10.el9.x 2.4 MB/s | 144 kB 00:00 2024-05-27T07:11:05.150 INFO:teuthology.orchestra.run.smithi105.stdout:(123/159): python3-grpcio-1.46.7-10.el9.x86_64. 13 MB/s | 2.0 MB 00:00 2024-05-27T07:11:05.175 INFO:teuthology.orchestra.run.smithi105.stdout:(124/159): python3-jaraco-8.2.1-3.el9.noarch.rp 116 kB/s | 11 kB 00:00 2024-05-27T07:11:05.208 INFO:teuthology.orchestra.run.smithi105.stdout:(125/159): python3-jaraco-classes-3.2.1-5.el9.n 302 kB/s | 18 kB 00:00 2024-05-27T07:11:05.234 INFO:teuthology.orchestra.run.smithi105.stdout:(126/159): python3-jaraco-collections-3.0.0-8.e 394 kB/s | 23 kB 00:00 2024-05-27T07:11:05.259 INFO:teuthology.orchestra.run.smithi105.stdout:(127/159): python3-jaraco-functools-3.5.0-2.el9 386 kB/s | 19 kB 00:00 2024-05-27T07:11:05.276 INFO:teuthology.orchestra.run.smithi105.stdout:(128/159): python3-jaraco-text-3.2.0-6.el9.noar 467 kB/s | 20 kB 00:00 2024-05-27T07:11:05.292 INFO:teuthology.orchestra.run.smithi173.stdout:Running transaction check 2024-05-27T07:11:05.343 INFO:teuthology.orchestra.run.smithi105.stdout:(129/159): python3-kubernetes-26.1.0-2.el9.noar 12 MB/s | 1.0 MB 00:00 2024-05-27T07:11:05.360 INFO:teuthology.orchestra.run.smithi105.stdout:(130/159): python3-logutils-0.3.5-21.el9.noarch 554 kB/s | 46 kB 00:00 2024-05-27T07:11:05.402 INFO:teuthology.orchestra.run.smithi105.stdout:(131/159): python3-more-itertools-8.12.0-2.el9. 1.3 MB/s | 79 kB 00:00 2024-05-27T07:11:05.419 INFO:teuthology.orchestra.run.smithi105.stdout:(132/159): python3-natsort-7.1.1-5.el9.noarch.r 988 kB/s | 58 kB 00:00 2024-05-27T07:11:05.448 INFO:teuthology.orchestra.run.smithi173.stdout:Transaction check succeeded. 2024-05-27T07:11:05.448 INFO:teuthology.orchestra.run.smithi173.stdout:Running transaction test 2024-05-27T07:11:05.461 INFO:teuthology.orchestra.run.smithi105.stdout:(133/159): python3-pecan-1.4.2-3.el9.noarch.rpm 4.5 MB/s | 272 kB 00:00 2024-05-27T07:11:05.486 INFO:teuthology.orchestra.run.smithi105.stdout:(134/159): python3-portend-3.1.0-2.el9.noarch.r 245 kB/s | 16 kB 00:00 2024-05-27T07:11:05.520 INFO:teuthology.orchestra.run.smithi105.stdout:(135/159): python3-pyOpenSSL-21.0.0-1.el9.noarc 1.5 MB/s | 90 kB 00:00 2024-05-27T07:11:05.545 INFO:teuthology.orchestra.run.smithi105.stdout:(136/159): python3-pyperclip-1.8.2-2.el9.noarch 483 kB/s | 28 kB 00:00 2024-05-27T07:11:05.579 INFO:teuthology.orchestra.run.smithi105.stdout:(137/159): python3-repoze-lru-0.7-16.el9.noarch 525 kB/s | 31 kB 00:00 2024-05-27T07:11:05.604 INFO:teuthology.orchestra.run.smithi105.stdout:(138/159): python3-routes-2.5.1-5.el9.noarch.rp 3.1 MB/s | 188 kB 00:00 2024-05-27T07:11:05.638 INFO:teuthology.orchestra.run.smithi105.stdout:(139/159): python3-rsa-4.9-2.el9.noarch.rpm 1.0 MB/s | 59 kB 00:00 2024-05-27T07:11:05.663 INFO:teuthology.orchestra.run.smithi105.stdout:(140/159): python3-tempora-5.0.0-2.el9.noarch.r 610 kB/s | 36 kB 00:00 2024-05-27T07:11:05.713 INFO:teuthology.orchestra.run.smithi105.stdout:(141/159): python3-typing-extensions-4.4.0-2.el 677 kB/s | 51 kB 00:00 2024-05-27T07:11:05.720 INFO:teuthology.orchestra.run.smithi052.stdout:(83/159): python3-numpy-1.20.1-5.el9.x86_64.rpm 4.2 MB/s | 5.1 MB 00:01 2024-05-27T07:11:05.739 INFO:teuthology.orchestra.run.smithi105.stdout:(142/159): python3-webob-1.8.7-6.el9.noarch.rpm 3.0 MB/s | 230 kB 00:00 2024-05-27T07:11:05.782 INFO:teuthology.orchestra.run.smithi105.stdout:(143/159): python3-websocket-client-1.2.3-2.el9 1.3 MB/s | 90 kB 00:00 2024-05-27T07:11:05.806 INFO:teuthology.orchestra.run.smithi105.stdout:(144/159): python3-werkzeug-2.0.3-3.el9.1.noarc 6.2 MB/s | 427 kB 00:00 2024-05-27T07:11:05.839 INFO:teuthology.orchestra.run.smithi105.stdout:(145/159): python3-zc-lockfile-2.0-10.el9.noarc 341 kB/s | 20 kB 00:00 2024-05-27T07:11:05.865 INFO:teuthology.orchestra.run.smithi105.stdout:(146/159): re2-20211101-3.el9.x86_64.rpm 3.2 MB/s | 192 kB 00:00 2024-05-27T07:11:05.887 INFO:teuthology.orchestra.run.smithi052.stdout:(84/159): python3-oauthlib-3.1.1-5.el9.noarch.r 272 kB/s | 222 kB 00:00 2024-05-27T07:11:05.912 INFO:teuthology.orchestra.run.smithi052.stdout:(85/159): python3-numpy-f2py-1.20.1-5.el9.x86_6 289 kB/s | 250 kB 00:00 2024-05-27T07:11:05.940 INFO:teuthology.orchestra.run.smithi105.stdout:(147/159): thrift-0.15.0-2.el9.x86_64.rpm 16 MB/s | 1.6 MB 00:00 2024-05-27T07:11:06.282 INFO:teuthology.orchestra.run.smithi105.stdout:(148/159): librbd1-19.0.0-3872.g38f9d900.el9.x8 9.1 MB/s | 3.1 MB 00:00 2024-05-27T07:11:06.421 INFO:teuthology.orchestra.run.smithi052.stdout:(86/159): python3-packaging-20.9-5.el9.noarch.r 110 kB/s | 77 kB 00:00 2024-05-27T07:11:06.457 INFO:teuthology.orchestra.run.smithi105.stdout:(149/159): NetworkManager-1.47.90-1.el9.x86_64. 13 MB/s | 2.3 MB 00:00 2024-05-27T07:11:06.504 INFO:teuthology.orchestra.run.smithi052.stdout:(87/159): python3-prettytable-0.7.2-27.el9.noar 68 kB/s | 42 kB 00:00 2024-05-27T07:11:06.516 INFO:teuthology.orchestra.run.smithi105.stdout:(150/159): NetworkManager-libnm-1.47.90-1.el9.x 31 MB/s | 1.8 MB 00:00 2024-05-27T07:11:06.558 INFO:teuthology.orchestra.run.smithi105.stdout:(151/159): NetworkManager-team-1.47.90-1.el9.x8 1.0 MB/s | 41 kB 00:00 2024-05-27T07:11:06.601 INFO:teuthology.orchestra.run.smithi105.stdout:(152/159): NetworkManager-tui-1.47.90-1.el9.x86 5.8 MB/s | 248 kB 00:00 2024-05-27T07:11:06.642 INFO:teuthology.orchestra.run.smithi105.stdout:(153/159): elfutils-debuginfod-client-0.191-4.e 884 kB/s | 37 kB 00:00 2024-05-27T07:11:06.684 INFO:teuthology.orchestra.run.smithi105.stdout:(154/159): elfutils-libelf-0.191-4.el9.x86_64.r 4.9 MB/s | 207 kB 00:00 2024-05-27T07:11:06.726 INFO:teuthology.orchestra.run.smithi105.stdout:(155/159): elfutils-libs-0.191-4.el9.x86_64.rpm 6.1 MB/s | 260 kB 00:00 2024-05-27T07:11:06.754 INFO:teuthology.orchestra.run.smithi052.stdout:(88/159): python3-protobuf-3.14.0-13.el9.noarch 320 kB/s | 269 kB 00:00 2024-05-27T07:11:06.784 INFO:teuthology.orchestra.run.smithi105.stdout:(156/159): openssl-3.2.1-1.el9.x86_64.rpm 23 MB/s | 1.3 MB 00:00 2024-05-27T07:11:06.860 INFO:teuthology.orchestra.run.smithi105.stdout:(157/159): openssl-libs-3.2.1-1.el9.x86_64.rpm 32 MB/s | 2.4 MB 00:00 2024-05-27T07:11:06.943 INFO:teuthology.orchestra.run.smithi105.stdout:(158/159): librados2-19.0.0-3872.g38f9d900.el9. 3.1 MB/s | 3.3 MB 00:01 2024-05-27T07:11:07.188 INFO:teuthology.orchestra.run.smithi052.stdout:(89/159): python3-pyasn1-0.4.8-6.el9.noarch.rpm 207 kB/s | 159 kB 00:00 2024-05-27T07:11:07.207 INFO:teuthology.orchestra.run.smithi173.stdout:Transaction test succeeded. 2024-05-27T07:11:07.207 INFO:teuthology.orchestra.run.smithi173.stdout:Running transaction 2024-05-27T07:11:07.372 INFO:teuthology.orchestra.run.smithi052.stdout:(90/159): python3-pyasn1-modules-0.4.8-6.el9.no 321 kB/s | 279 kB 00:00 2024-05-27T07:11:07.431 INFO:teuthology.orchestra.run.smithi052.stdout:(91/159): python3-pytz-2021.1-5.el9.noarch.rpm 79 kB/s | 51 kB 00:00 2024-05-27T07:11:07.864 INFO:teuthology.orchestra.run.smithi052.stdout:(92/159): python3-requests-oauthlib-1.3.0-12.el 79 kB/s | 54 kB 00:00 2024-05-27T07:11:08.106 INFO:teuthology.orchestra.run.smithi052.stdout:(93/159): python3-toml-0.10.2-6.el9.noarch.rpm 64 kB/s | 42 kB 00:00 2024-05-27T07:11:08.427 INFO:teuthology.orchestra.run.smithi105.stdout:(159/159): ceph-test-19.0.0-3872.g38f9d900.el9. 2.8 MB/s | 43 MB 00:15 2024-05-27T07:11:08.431 INFO:teuthology.orchestra.run.smithi105.stdout:-------------------------------------------------------------------------------- 2024-05-27T07:11:08.431 INFO:teuthology.orchestra.run.smithi105.stdout:Total 9.5 MB/s | 219 MB 00:23 2024-05-27T07:11:08.525 INFO:teuthology.orchestra.run.smithi052.stdout:(94/159): python3-wcwidth-0.2.5-8.el9.noarch.rp 67 kB/s | 44 kB 00:00 2024-05-27T07:11:09.040 INFO:teuthology.orchestra.run.smithi052.stdout:(95/159): python3-scipy-1.6.2-8.el9.x86_64.rpm 9.7 MB/s | 16 MB 00:01 2024-05-27T07:11:09.065 INFO:teuthology.orchestra.run.smithi052.stdout:(96/159): qatlib-24.02.0-1.el9.x86_64.rpm 237 kB/s | 221 kB 00:00 2024-05-27T07:11:09.324 INFO:teuthology.orchestra.run.smithi052.stdout:(97/159): qatlib-service-24.02.0-1.el9.x86_64.r 45 kB/s | 36 kB 00:00 2024-05-27T07:11:09.592 INFO:teuthology.orchestra.run.smithi105.stdout:Running transaction check 2024-05-27T07:11:09.742 INFO:teuthology.orchestra.run.smithi052.stdout:(98/159): qatzip-libs-1.2.0-1.el9.x86_64.rpm 68 kB/s | 47 kB 00:00 2024-05-27T07:11:09.750 INFO:teuthology.orchestra.run.smithi105.stdout:Transaction check succeeded. 2024-05-27T07:11:09.750 INFO:teuthology.orchestra.run.smithi105.stdout:Running transaction test 2024-05-27T07:11:09.983 INFO:teuthology.orchestra.run.smithi052.stdout:(99/159): socat-1.7.4.1-5.el9.x86_64.rpm 332 kB/s | 305 kB 00:00 2024-05-27T07:11:10.033 INFO:teuthology.orchestra.run.smithi052.stdout:(100/159): uuid-1.6.2-55.el9.x86_64.rpm 88 kB/s | 58 kB 00:00 2024-05-27T07:11:10.475 INFO:teuthology.orchestra.run.smithi052.stdout:(101/159): xfsprogs-devel-6.3.0-1.el9.x86_64.rp 128 kB/s | 94 kB 00:00 2024-05-27T07:11:10.667 INFO:teuthology.orchestra.run.smithi052.stdout:(102/159): zlib-devel-1.2.11-41.el9.x86_64.rpm 71 kB/s | 45 kB 00:00 2024-05-27T07:11:10.692 INFO:teuthology.orchestra.run.smithi052.stdout:(103/159): xmlstarlet-1.6.1-20.el9.x86_64.rpm 93 kB/s | 64 kB 00:00 2024-05-27T07:11:10.717 INFO:teuthology.orchestra.run.smithi052.stdout:(104/159): lua-devel-5.4.4-4.el9.x86_64.rpm 92 kB/s | 22 kB 00:00 2024-05-27T07:11:11.001 INFO:teuthology.orchestra.run.smithi052.stdout:(105/159): protobuf-compiler-3.14.0-13.el9.x86_ 2.5 MB/s | 863 kB 00:00 2024-05-27T07:11:11.351 INFO:teuthology.orchestra.run.smithi052.stdout:(106/159): grpc-data-1.46.7-10.el9.noarch.rpm 56 kB/s | 19 kB 00:00 2024-05-27T07:11:11.377 INFO:teuthology.orchestra.run.smithi052.stdout:(107/159): gperftools-libs-2.9.1-2.el9.x86_64.r 469 kB/s | 309 kB 00:00 2024-05-27T07:11:11.402 INFO:teuthology.orchestra.run.smithi052.stdout:(108/159): abseil-cpp-20211102.0-3.el9.x86_64.r 773 kB/s | 548 kB 00:00 2024-05-27T07:11:11.477 INFO:teuthology.orchestra.run.smithi052.stdout:(109/159): libarrow-doc-9.0.0-11.el9.noarch.rpm 259 kB/s | 26 kB 00:00 2024-05-27T07:11:11.502 INFO:teuthology.orchestra.run.smithi052.stdout:(110/159): liboath-2.6.7-2.el9.x86_64.rpm 484 kB/s | 49 kB 00:00 2024-05-27T07:11:11.521 INFO:teuthology.orchestra.run.smithi105.stdout:Transaction test succeeded. 2024-05-27T07:11:11.522 INFO:teuthology.orchestra.run.smithi105.stdout:Running transaction 2024-05-27T07:11:11.586 INFO:teuthology.orchestra.run.smithi052.stdout:(111/159): libunwind-1.6.2-1.el9.x86_64.rpm 622 kB/s | 67 kB 00:00 2024-05-27T07:11:11.620 INFO:teuthology.orchestra.run.smithi052.stdout:(112/159): luarocks-3.9.2-5.el9.noarch.rpm 1.3 MB/s | 151 kB 00:00 2024-05-27T07:11:11.795 INFO:teuthology.orchestra.run.smithi052.stdout:(113/159): python3-asyncssh-2.13.2-5.el9.noarch 3.1 MB/s | 548 kB 00:00 2024-05-27T07:11:11.837 INFO:teuthology.orchestra.run.smithi052.stdout:(114/159): parquet-libs-9.0.0-11.el9.x86_64.rpm 3.3 MB/s | 839 kB 00:00 2024-05-27T07:11:11.896 INFO:teuthology.orchestra.run.smithi052.stdout:(115/159): python3-bcrypt-3.2.2-1.el9.x86_64.rp 434 kB/s | 43 kB 00:00 2024-05-27T07:11:11.937 INFO:teuthology.orchestra.run.smithi052.stdout:(116/159): python3-cachetools-4.2.4-1.el9.noarc 321 kB/s | 32 kB 00:00 2024-05-27T07:11:11.996 INFO:teuthology.orchestra.run.smithi052.stdout:(117/159): python3-certifi-2021.10.8-2.el9.noar 149 kB/s | 15 kB 00:00 2024-05-27T07:11:12.080 INFO:teuthology.orchestra.run.smithi052.stdout:(118/159): libarrow-9.0.0-11.el9.x86_64.rpm 6.1 MB/s | 4.4 MB 00:00 2024-05-27T07:11:12.105 INFO:teuthology.orchestra.run.smithi052.stdout:(119/159): python3-cheroot-8.6.0-4.el9.noarch.r 1.0 MB/s | 172 kB 00:00 2024-05-27T07:11:12.131 INFO:teuthology.orchestra.run.smithi052.stdout:(120/159): python3-cherrypy-18.6.1-2.el9.noarch 2.6 MB/s | 358 kB 00:00 2024-05-27T07:11:12.189 INFO:teuthology.orchestra.run.smithi052.stdout:(121/159): python3-cmd2-2.4.2-2.el9.noarch.rpm 2.7 MB/s | 303 kB 00:00 2024-05-27T07:11:12.214 INFO:teuthology.orchestra.run.smithi052.stdout:(122/159): python3-google-auth-2.29.0-1.el9.noa 1.8 MB/s | 206 kB 00:00 2024-05-27T07:11:12.290 INFO:teuthology.orchestra.run.smithi052.stdout:(123/159): python3-grpcio-tools-1.46.7-10.el9.x 1.4 MB/s | 144 kB 00:00 2024-05-27T07:11:12.315 INFO:teuthology.orchestra.run.smithi052.stdout:(124/159): python3-jaraco-8.2.1-3.el9.noarch.rp 107 kB/s | 11 kB 00:00 2024-05-27T07:11:12.407 INFO:teuthology.orchestra.run.smithi052.stdout:(125/159): python3-grpcio-1.46.7-10.el9.x86_64. 7.4 MB/s | 2.0 MB 00:00 2024-05-27T07:11:12.432 INFO:teuthology.orchestra.run.smithi052.stdout:(126/159): python3-jaraco-classes-3.2.1-5.el9.n 124 kB/s | 18 kB 00:00 2024-05-27T07:11:12.458 INFO:teuthology.orchestra.run.smithi052.stdout:(127/159): python3-jaraco-collections-3.0.0-8.e 163 kB/s | 23 kB 00:00 2024-05-27T07:11:12.508 INFO:teuthology.orchestra.run.smithi052.stdout:(128/159): python3-jaraco-functools-3.5.0-2.el9 194 kB/s | 19 kB 00:00 2024-05-27T07:11:12.558 INFO:teuthology.orchestra.run.smithi052.stdout:(129/159): python3-jaraco-text-3.2.0-6.el9.noar 196 kB/s | 20 kB 00:00 2024-05-27T07:11:12.600 INFO:teuthology.orchestra.run.smithi052.stdout:(130/159): python3-kubernetes-26.1.0-2.el9.noar 7.2 MB/s | 1.0 MB 00:00 2024-05-27T07:11:12.625 INFO:teuthology.orchestra.run.smithi052.stdout:(131/159): python3-logutils-0.3.5-21.el9.noarch 394 kB/s | 46 kB 00:00 2024-05-27T07:11:12.659 INFO:teuthology.orchestra.run.smithi052.stdout:(132/159): python3-more-itertools-8.12.0-2.el9. 785 kB/s | 79 kB 00:00 2024-05-27T07:11:12.701 INFO:teuthology.orchestra.run.smithi052.stdout:(133/159): python3-natsort-7.1.1-5.el9.noarch.r 574 kB/s | 58 kB 00:00 2024-05-27T07:11:12.726 INFO:teuthology.orchestra.run.smithi052.stdout:(134/159): python3-pecan-1.4.2-3.el9.noarch.rpm 2.6 MB/s | 272 kB 00:00 2024-05-27T07:11:12.751 INFO:teuthology.orchestra.run.smithi052.stdout:(135/159): python3-portend-3.1.0-2.el9.noarch.r 178 kB/s | 16 kB 00:00 2024-05-27T07:11:12.802 INFO:teuthology.orchestra.run.smithi052.stdout:(136/159): python3-pyOpenSSL-21.0.0-1.el9.noarc 894 kB/s | 90 kB 00:00 2024-05-27T07:11:12.827 INFO:teuthology.orchestra.run.smithi052.stdout:(137/159): python3-pyperclip-1.8.2-2.el9.noarch 281 kB/s | 28 kB 00:00 2024-05-27T07:11:12.852 INFO:teuthology.orchestra.run.smithi052.stdout:(138/159): python3-repoze-lru-0.7-16.el9.noarch 306 kB/s | 31 kB 00:00 2024-05-27T07:11:12.903 INFO:teuthology.orchestra.run.smithi052.stdout:(139/159): python3-routes-2.5.1-5.el9.noarch.rp 1.8 MB/s | 188 kB 00:00 2024-05-27T07:11:12.919 INFO:teuthology.orchestra.run.smithi052.stdout:(140/159): python3-rsa-4.9-2.el9.noarch.rpm 641 kB/s | 59 kB 00:00 2024-05-27T07:11:12.945 INFO:teuthology.orchestra.run.smithi052.stdout:(141/159): python3-tempora-5.0.0-2.el9.noarch.r 388 kB/s | 36 kB 00:00 2024-05-27T07:11:12.995 INFO:teuthology.orchestra.run.smithi052.stdout:(142/159): python3-typing-extensions-4.4.0-2.el 553 kB/s | 51 kB 00:00 2024-05-27T07:11:13.029 INFO:teuthology.orchestra.run.smithi052.stdout:(143/159): python3-webob-1.8.7-6.el9.noarch.rpm 2.1 MB/s | 230 kB 00:00 2024-05-27T07:11:13.054 INFO:teuthology.orchestra.run.smithi052.stdout:(144/159): python3-websocket-client-1.2.3-2.el9 824 kB/s | 90 kB 00:00 2024-05-27T07:11:13.104 INFO:teuthology.orchestra.run.smithi052.stdout:(145/159): python3-werkzeug-2.0.3-3.el9.1.noarc 3.8 MB/s | 427 kB 00:00 2024-05-27T07:11:13.121 INFO:teuthology.orchestra.run.smithi052.stdout:(146/159): python3-zc-lockfile-2.0-10.el9.noarc 216 kB/s | 20 kB 00:00 2024-05-27T07:11:13.165 INFO:teuthology.orchestra.run.smithi052.stdout:(147/159): re2-20211101-3.el9.x86_64.rpm 1.7 MB/s | 192 kB 00:00 2024-05-27T07:11:13.272 INFO:teuthology.orchestra.run.smithi052.stdout:(148/159): thrift-0.15.0-2.el9.x86_64.rpm 9.4 MB/s | 1.6 MB 00:00 2024-05-27T07:11:13.389 INFO:teuthology.orchestra.run.smithi052.stdout:(149/159): NetworkManager-1.47.90-1.el9.x86_64. 20 MB/s | 2.3 MB 00:00 2024-05-27T07:11:13.444 INFO:teuthology.orchestra.run.smithi173.stdout: Preparing : 1/1 2024-05-27T07:11:13.472 INFO:teuthology.orchestra.run.smithi052.stdout:(150/159): NetworkManager-libnm-1.47.90-1.el9.x 22 MB/s | 1.8 MB 00:00 2024-05-27T07:11:13.503 INFO:teuthology.orchestra.run.smithi173.stdout: Upgrading : openssl-libs-1:3.2.1-1.el9.x86_64 1/170 2024-05-27T07:11:13.514 INFO:teuthology.orchestra.run.smithi052.stdout:(151/159): NetworkManager-team-1.47.90-1.el9.x8 998 kB/s | 41 kB 00:00 2024-05-27T07:11:13.562 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 2/170 2024-05-27T07:11:13.564 INFO:teuthology.orchestra.run.smithi052.stdout:(152/159): NetworkManager-tui-1.47.90-1.el9.x86 4.9 MB/s | 248 kB 00:00 2024-05-27T07:11:13.606 INFO:teuthology.orchestra.run.smithi052.stdout:(153/159): elfutils-debuginfod-client-0.191-4.e 880 kB/s | 37 kB 00:00 2024-05-27T07:11:13.620 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 3/170 2024-05-27T07:11:13.654 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : thrift-0.15.0-2.el9.x86_64 4/170 2024-05-27T07:11:13.657 INFO:teuthology.orchestra.run.smithi052.stdout:(154/159): librados2-19.0.0-3872.g38f9d900.el9. 6.2 MB/s | 3.3 MB 00:00 2024-05-27T07:11:13.679 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 5/170 2024-05-27T07:11:13.682 INFO:teuthology.orchestra.run.smithi052.stdout:(155/159): elfutils-libelf-0.191-4.el9.x86_64.r 2.7 MB/s | 207 kB 00:00 2024-05-27T07:11:13.722 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 6/170 2024-05-27T07:11:13.749 INFO:teuthology.orchestra.run.smithi052.stdout:(156/159): librbd1-19.0.0-3872.g38f9d900.el9.x8 5.3 MB/s | 3.1 MB 00:00 2024-05-27T07:11:13.791 INFO:teuthology.orchestra.run.smithi052.stdout:(157/159): openssl-3.2.1-1.el9.x86_64.rpm 12 MB/s | 1.3 MB 00:00 2024-05-27T07:11:13.816 INFO:teuthology.orchestra.run.smithi052.stdout:(158/159): elfutils-libs-0.191-4.el9.x86_64.rpm 1.6 MB/s | 260 kB 00:00 2024-05-27T07:11:14.057 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 7/170 2024-05-27T07:11:14.125 INFO:teuthology.orchestra.run.smithi052.stdout:(159/159): openssl-libs-3.2.1-1.el9.x86_64.rpm 6.4 MB/s | 2.4 MB 00:00 2024-05-27T07:11:14.130 INFO:teuthology.orchestra.run.smithi052.stdout:-------------------------------------------------------------------------------- 2024-05-27T07:11:14.130 INFO:teuthology.orchestra.run.smithi052.stdout:Total 7.1 MB/s | 219 MB 00:30 2024-05-27T07:11:14.180 INFO:teuthology.orchestra.run.smithi173.stdout: Upgrading : librados2-2:19.0.0-3872.g38f9d900.el9.x86_64 8/170 2024-05-27T07:11:14.305 INFO:teuthology.orchestra.run.smithi173.stdout: Running scriptlet: librados2-2:19.0.0-3872.g38f9d900.el9.x86_64 8/170 2024-05-27T07:11:14.331 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : libcephfs2-2:19.0.0-3872.g38f9d900.el9.x86_64 9/170 2024-05-27T07:11:14.422 INFO:teuthology.orchestra.run.smithi173.stdout: Running scriptlet: libcephfs2-2:19.0.0-3872.g38f9d900.el9.x86_64 9/170 2024-05-27T07:11:14.452 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : python3-rados-2:19.0.0-3872.g38f9d900.el9.x86_64 10/170 2024-05-27T07:11:14.485 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : postgresql-private-libs-13.14-1.el9.x86_64 11/170 2024-05-27T07:11:14.692 INFO:teuthology.orchestra.run.smithi173.stdout: Upgrading : elfutils-libelf-0.191-4.el9.x86_64 12/170 2024-05-27T07:11:14.759 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : postgresql-13.14-1.el9.x86_64 13/170 2024-05-27T07:11:14.787 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : libcephsqlite-2:19.0.0-3872.g38f9d900.el9.x86_64 14/170 2024-05-27T07:11:14.891 INFO:teuthology.orchestra.run.smithi173.stdout: Running scriptlet: libcephsqlite-2:19.0.0-3872.g38f9d900.el9.x86_64 14/170 2024-05-27T07:11:14.921 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : libradosstriper1-2:19.0.0-3872.g38f9d900.el9.x86 15/170 2024-05-27T07:11:14.983 INFO:teuthology.orchestra.run.smithi173.stdout: Running scriptlet: libradosstriper1-2:19.0.0-3872.g38f9d900.el9.x86 15/170 2024-05-27T07:11:15.006 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : qatlib-service-24.02.0-1.el9.x86_64 16/170 2024-05-27T07:11:15.029 INFO:teuthology.orchestra.run.smithi173.stdout: Running scriptlet: qatlib-service-24.02.0-1.el9.x86_64 16/170 2024-05-27T07:11:15.223 INFO:teuthology.orchestra.run.smithi173.stdout: Running scriptlet: qatlib-24.02.0-1.el9.x86_64 17/170 2024-05-27T07:11:15.360 INFO:teuthology.orchestra.run.smithi052.stdout:Running transaction check 2024-05-27T07:11:15.385 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : qatlib-24.02.0-1.el9.x86_64 17/170 2024-05-27T07:11:15.541 INFO:teuthology.orchestra.run.smithi052.stdout:Transaction check succeeded. 2024-05-27T07:11:15.542 INFO:teuthology.orchestra.run.smithi052.stdout:Running transaction test 2024-05-27T07:11:15.561 INFO:teuthology.orchestra.run.smithi173.stdout: Upgrading : NetworkManager-libnm-1:1.47.90-1.el9.x86_64 18/170 2024-05-27T07:11:15.622 INFO:teuthology.orchestra.run.smithi173.stdout: Running scriptlet: NetworkManager-libnm-1:1.47.90-1.el9.x86_64 18/170 2024-05-27T07:11:15.710 INFO:teuthology.orchestra.run.smithi173.stdout: Running scriptlet: NetworkManager-1:1.47.90-1.el9.x86_64 19/170 2024-05-27T07:11:15.823 INFO:teuthology.orchestra.run.smithi173.stdout: Upgrading : NetworkManager-1:1.47.90-1.el9.x86_64 19/170 2024-05-27T07:11:16.021 INFO:teuthology.orchestra.run.smithi173.stdout: Running scriptlet: NetworkManager-1:1.47.90-1.el9.x86_64 19/170 2024-05-27T07:11:16.131 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : re2-1:20211101-3.el9.x86_64 20/170 2024-05-27T07:11:16.160 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : libarrow-9.0.0-11.el9.x86_64 21/170 2024-05-27T07:11:16.195 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : liboath-2.6.7-2.el9.x86_64 22/170 2024-05-27T07:11:16.260 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 23/170 2024-05-27T07:11:16.296 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 24/170 2024-05-27T07:11:16.341 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 25/170 2024-05-27T07:11:16.373 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : protobuf-3.14.0-13.el9.x86_64 26/170 2024-05-27T07:11:16.401 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : lua-5.4.4-4.el9.x86_64 27/170 2024-05-27T07:11:16.470 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 28/170 2024-05-27T07:11:16.524 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : python3-urllib3-1.26.5-5.el9.noarch 29/170 2024-05-27T07:11:16.554 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : python3-requests-2.25.1-8.el9.noarch 30/170 2024-05-27T07:11:16.598 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : libquadmath-11.4.1-3.el9.x86_64 31/170 2024-05-27T07:11:16.673 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : libgfortran-11.4.1-3.el9.x86_64 32/170 2024-05-27T07:11:16.708 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : python3-ceph-common-2:19.0.0-3872.g38f9d900.el9. 33/170 2024-05-27T07:11:16.743 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : python3-ceph-argparse-2:19.0.0-3872.g38f9d900.el 34/170 2024-05-27T07:11:16.781 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : python3-cephfs-2:19.0.0-3872.g38f9d900.el9.x86_6 35/170 2024-05-27T07:11:16.850 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : lua-devel-5.4.4-4.el9.x86_64 36/170 2024-05-27T07:11:16.896 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : luarocks-3.9.2-5.el9.noarch 37/170 2024-05-27T07:11:16.970 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : protobuf-compiler-3.14.0-13.el9.x86_64 38/170 2024-05-27T07:11:17.102 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 39/170 2024-05-27T07:11:17.173 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 40/170 2024-05-27T07:11:17.694 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : python3-rsa-4.9-2.el9.noarch 41/170 2024-05-27T07:11:17.777 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 42/170 2024-05-27T07:11:17.807 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : python3-jinja2-2.11.3-6.el9.noarch 43/170 2024-05-27T07:11:17.829 INFO:teuthology.orchestra.run.smithi052.stdout:Transaction test succeeded. 2024-05-27T07:11:17.829 INFO:teuthology.orchestra.run.smithi052.stdout:Running transaction 2024-05-27T07:11:17.843 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : qatzip-libs-1.2.0-1.el9.x86_64 44/170 2024-05-27T07:11:17.892 INFO:teuthology.orchestra.run.smithi173.stdout: Upgrading : elfutils-libs-0.191-4.el9.x86_64 45/170 2024-05-27T07:11:17.910 INFO:teuthology.orchestra.run.smithi105.stdout: Preparing : 1/1 2024-05-27T07:11:17.930 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : librados-devel-2:19.0.0-3872.g38f9d900.el9.x86_6 46/170 2024-05-27T07:11:17.980 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 47/170 2024-05-27T07:11:17.981 INFO:teuthology.orchestra.run.smithi105.stdout: Upgrading : openssl-libs-1:3.2.1-1.el9.x86_64 1/170 2024-05-27T07:11:18.020 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 2/170 2024-05-27T07:11:18.073 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 3/170 2024-05-27T07:11:18.107 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : thrift-0.15.0-2.el9.x86_64 4/170 2024-05-27T07:11:18.133 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 5/170 2024-05-27T07:11:18.138 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : socat-1.7.4.1-5.el9.x86_64 48/170 2024-05-27T07:11:18.174 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 6/170 2024-05-27T07:11:18.200 INFO:teuthology.orchestra.run.smithi173.stdout: Upgrading : openssl-1:3.2.1-1.el9.x86_64 49/170 2024-05-27T07:11:18.291 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 50/170 2024-05-27T07:11:18.356 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 51/170 2024-05-27T07:11:18.429 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 52/170 2024-05-27T07:11:18.461 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : python3-webob-1.8.7-6.el9.noarch 53/170 2024-05-27T07:11:18.497 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : python3-typing-extensions-4.4.0-2.el9.noarch 54/170 2024-05-27T07:11:18.501 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 7/170 2024-05-27T07:11:18.552 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 55/170 2024-05-27T07:11:18.597 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 56/170 2024-05-27T07:11:18.625 INFO:teuthology.orchestra.run.smithi105.stdout: Upgrading : librados2-2:19.0.0-3872.g38f9d900.el9.x86_64 8/170 2024-05-27T07:11:18.640 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : python3-pyperclip-1.8.2-2.el9.noarch 57/170 2024-05-27T07:11:18.682 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 58/170 2024-05-27T07:11:18.751 INFO:teuthology.orchestra.run.smithi105.stdout: Running scriptlet: librados2-2:19.0.0-3872.g38f9d900.el9.x86_64 8/170 2024-05-27T07:11:18.784 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : libcephfs2-2:19.0.0-3872.g38f9d900.el9.x86_64 9/170 2024-05-27T07:11:18.875 INFO:teuthology.orchestra.run.smithi105.stdout: Running scriptlet: libcephfs2-2:19.0.0-3872.g38f9d900.el9.x86_64 9/170 2024-05-27T07:11:18.901 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 59/170 2024-05-27T07:11:18.913 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : python3-rados-2:19.0.0-3872.g38f9d900.el9.x86_64 10/170 2024-05-27T07:11:18.946 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : postgresql-private-libs-13.14-1.el9.x86_64 11/170 2024-05-27T07:11:18.976 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 60/170 2024-05-27T07:11:19.017 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : python3-certifi-2021.10.8-2.el9.noarch 61/170 2024-05-27T07:11:19.112 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 62/170 2024-05-27T07:11:19.129 INFO:teuthology.orchestra.run.smithi105.stdout: Upgrading : elfutils-libelf-0.191-4.el9.x86_64 12/170 2024-05-27T07:11:19.143 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : python3-google-auth-1:2.29.0-1.el9.noarch 63/170 2024-05-27T07:11:19.179 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : postgresql-13.14-1.el9.x86_64 13/170 2024-05-27T07:11:19.183 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 64/170 2024-05-27T07:11:19.206 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : libcephsqlite-2:19.0.0-3872.g38f9d900.el9.x86_64 14/170 2024-05-27T07:11:19.217 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : gperftools-libs-2.9.1-2.el9.x86_64 65/170 2024-05-27T07:11:19.242 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : libarrow-doc-9.0.0-11.el9.noarch 66/170 2024-05-27T07:11:19.310 INFO:teuthology.orchestra.run.smithi105.stdout: Running scriptlet: libcephsqlite-2:19.0.0-3872.g38f9d900.el9.x86_64 14/170 2024-05-27T07:11:19.315 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : grpc-data-1.46.7-10.el9.noarch 67/170 2024-05-27T07:11:19.332 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : libradosstriper1-2:19.0.0-3872.g38f9d900.el9.x86 15/170 2024-05-27T07:11:19.403 INFO:teuthology.orchestra.run.smithi105.stdout: Running scriptlet: libradosstriper1-2:19.0.0-3872.g38f9d900.el9.x86 15/170 2024-05-27T07:11:19.425 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : qatlib-service-24.02.0-1.el9.x86_64 16/170 2024-05-27T07:11:19.450 INFO:teuthology.orchestra.run.smithi105.stdout: Running scriptlet: qatlib-service-24.02.0-1.el9.x86_64 16/170 2024-05-27T07:11:19.452 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : abseil-cpp-20211102.0-3.el9.x86_64 68/170 2024-05-27T07:11:19.491 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : python3-grpcio-1.46.7-10.el9.x86_64 69/170 2024-05-27T07:11:19.519 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : zlib-devel-1.2.11-41.el9.x86_64 70/170 2024-05-27T07:11:19.553 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 71/170 2024-05-27T07:11:19.593 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : uuid-1.6.2-55.el9.x86_64 72/170 2024-05-27T07:11:19.600 INFO:teuthology.orchestra.run.smithi105.stdout: Running scriptlet: qatlib-24.02.0-1.el9.x86_64 17/170 2024-05-27T07:11:19.624 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : python3-wcwidth-0.2.5-8.el9.noarch 73/170 2024-05-27T07:11:19.654 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 74/170 2024-05-27T07:11:19.689 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 75/170 2024-05-27T07:11:19.722 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 76/170 2024-05-27T07:11:19.763 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : qatlib-24.02.0-1.el9.x86_64 17/170 2024-05-27T07:11:19.764 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 77/170 2024-05-27T07:11:19.798 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 78/170 2024-05-27T07:11:19.886 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 79/170 2024-05-27T07:11:19.889 INFO:teuthology.orchestra.run.smithi105.stdout: Upgrading : NetworkManager-libnm-1:1.47.90-1.el9.x86_64 18/170 2024-05-27T07:11:19.938 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : python3-protobuf-3.14.0-13.el9.noarch 80/170 2024-05-27T07:11:19.966 INFO:teuthology.orchestra.run.smithi105.stdout: Running scriptlet: NetworkManager-libnm-1:1.47.90-1.el9.x86_64 18/170 2024-05-27T07:11:19.969 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : python3-grpcio-tools-1.46.7-10.el9.x86_64 81/170 2024-05-27T07:11:20.016 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 82/170 2024-05-27T07:11:20.079 INFO:teuthology.orchestra.run.smithi105.stdout: Running scriptlet: NetworkManager-1:1.47.90-1.el9.x86_64 19/170 2024-05-27T07:11:20.113 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : python3-packaging-20.9-5.el9.noarch 83/170 2024-05-27T07:11:20.162 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 84/170 2024-05-27T07:11:20.186 INFO:teuthology.orchestra.run.smithi105.stdout: Upgrading : NetworkManager-1:1.47.90-1.el9.x86_64 19/170 2024-05-27T07:11:20.378 INFO:teuthology.orchestra.run.smithi105.stdout: Running scriptlet: NetworkManager-1:1.47.90-1.el9.x86_64 19/170 2024-05-27T07:11:20.484 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : re2-1:20211101-3.el9.x86_64 20/170 2024-05-27T07:11:20.516 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : libarrow-9.0.0-11.el9.x86_64 21/170 2024-05-27T07:11:20.551 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : liboath-2.6.7-2.el9.x86_64 22/170 2024-05-27T07:11:20.615 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 23/170 2024-05-27T07:11:20.651 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 24/170 2024-05-27T07:11:20.694 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 25/170 2024-05-27T07:11:20.727 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : protobuf-3.14.0-13.el9.x86_64 26/170 2024-05-27T07:11:20.756 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : lua-5.4.4-4.el9.x86_64 27/170 2024-05-27T07:11:20.824 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 28/170 2024-05-27T07:11:20.877 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 85/170 2024-05-27T07:11:20.878 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : python3-urllib3-1.26.5-5.el9.noarch 29/170 2024-05-27T07:11:20.909 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : python3-requests-2.25.1-8.el9.noarch 30/170 2024-05-27T07:11:20.939 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 86/170 2024-05-27T07:11:20.944 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : libquadmath-11.4.1-3.el9.x86_64 31/170 2024-05-27T07:11:20.977 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 87/170 2024-05-27T07:11:21.019 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : libgfortran-11.4.1-3.el9.x86_64 32/170 2024-05-27T07:11:21.055 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : python3-ceph-common-2:19.0.0-3872.g38f9d900.el9. 33/170 2024-05-27T07:11:21.077 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 88/170 2024-05-27T07:11:21.098 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : python3-ceph-argparse-2:19.0.0-3872.g38f9d900.el 34/170 2024-05-27T07:11:21.127 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : python3-cephfs-2:19.0.0-3872.g38f9d900.el9.x86_6 35/170 2024-05-27T07:11:21.133 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : python3-devel-3.9.18-3.el9.x86_64 89/170 2024-05-27T07:11:21.176 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : python3-colorama-0.4.6-3.el9.noarch 90/170 2024-05-27T07:11:21.196 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : lua-devel-5.4.4-4.el9.x86_64 36/170 2024-05-27T07:11:21.241 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : luarocks-3.9.2-5.el9.noarch 37/170 2024-05-27T07:11:21.256 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : python3-attrs-20.3.0-7.el9.noarch 91/170 2024-05-27T07:11:21.289 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : python3-cmd2-2.4.2-2.el9.noarch 92/170 2024-05-27T07:11:21.316 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : protobuf-compiler-3.14.0-13.el9.x86_64 38/170 2024-05-27T07:11:21.455 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 39/170 2024-05-27T07:11:21.468 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : openblas-0.3.21-2.el9.x86_64 93/170 2024-05-27T07:11:21.495 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : openblas-openmp-0.3.21-2.el9.x86_64 94/170 2024-05-27T07:11:21.510 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 40/170 2024-05-27T07:11:21.582 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 95/170 2024-05-27T07:11:22.026 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : python3-rsa-4.9-2.el9.noarch 41/170 2024-05-27T07:11:22.114 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 42/170 2024-05-27T07:11:22.152 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : python3-jinja2-2.11.3-6.el9.noarch 43/170 2024-05-27T07:11:22.188 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : qatzip-libs-1.2.0-1.el9.x86_64 44/170 2024-05-27T07:11:22.234 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 96/170 2024-05-27T07:11:22.237 INFO:teuthology.orchestra.run.smithi105.stdout: Upgrading : elfutils-libs-0.191-4.el9.x86_64 45/170 2024-05-27T07:11:22.275 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : librados-devel-2:19.0.0-3872.g38f9d900.el9.x86_6 46/170 2024-05-27T07:11:22.317 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 47/170 2024-05-27T07:11:22.344 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 97/170 2024-05-27T07:11:22.460 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : socat-1.7.4.1-5.el9.x86_64 48/170 2024-05-27T07:11:22.520 INFO:teuthology.orchestra.run.smithi105.stdout: Upgrading : openssl-1:3.2.1-1.el9.x86_64 49/170 2024-05-27T07:11:22.609 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 50/170 2024-05-27T07:11:22.683 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 51/170 2024-05-27T07:11:22.756 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 52/170 2024-05-27T07:11:22.788 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : python3-webob-1.8.7-6.el9.noarch 53/170 2024-05-27T07:11:22.824 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : python3-typing-extensions-4.4.0-2.el9.noarch 54/170 2024-05-27T07:11:22.870 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 55/170 2024-05-27T07:11:22.907 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 56/170 2024-05-27T07:11:22.949 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : python3-pyperclip-1.8.2-2.el9.noarch 57/170 2024-05-27T07:11:23.000 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 58/170 2024-05-27T07:11:23.134 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 59/170 2024-05-27T07:11:23.178 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 60/170 2024-05-27T07:11:23.218 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : python3-certifi-2021.10.8-2.el9.noarch 61/170 2024-05-27T07:11:23.311 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 62/170 2024-05-27T07:11:23.352 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : python3-google-auth-1:2.29.0-1.el9.noarch 63/170 2024-05-27T07:11:23.420 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 64/170 2024-05-27T07:11:23.451 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : gperftools-libs-2.9.1-2.el9.x86_64 65/170 2024-05-27T07:11:23.476 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : libarrow-doc-9.0.0-11.el9.noarch 66/170 2024-05-27T07:11:23.557 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : grpc-data-1.46.7-10.el9.noarch 67/170 2024-05-27T07:11:23.562 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 98/170 2024-05-27T07:11:23.651 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 99/170 2024-05-27T07:11:23.679 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : libzstd-devel-1.5.1-2.el9.x86_64 100/170 2024-05-27T07:11:23.692 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : abseil-cpp-20211102.0-3.el9.x86_64 68/170 2024-05-27T07:11:23.712 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : libuuid-devel-2.37.4-18.el9.x86_64 101/170 2024-05-27T07:11:23.725 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : python3-grpcio-1.46.7-10.el9.x86_64 69/170 2024-05-27T07:11:23.752 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : zlib-devel-1.2.11-41.el9.x86_64 70/170 2024-05-27T07:11:23.787 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 71/170 2024-05-27T07:11:23.826 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : uuid-1.6.2-55.el9.x86_64 72/170 2024-05-27T07:11:23.857 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : python3-wcwidth-0.2.5-8.el9.noarch 73/170 2024-05-27T07:11:23.887 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 74/170 2024-05-27T07:11:23.922 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 75/170 2024-05-27T07:11:23.964 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 76/170 2024-05-27T07:11:24.005 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 77/170 2024-05-27T07:11:24.029 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 102/170 2024-05-27T07:11:24.039 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 78/170 2024-05-27T07:11:24.075 INFO:teuthology.orchestra.run.smithi173.stdout: Upgrading : librbd1-2:19.0.0-3872.g38f9d900.el9.x86_64 103/170 2024-05-27T07:11:24.125 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 79/170 2024-05-27T07:11:24.166 INFO:teuthology.orchestra.run.smithi173.stdout: Running scriptlet: librbd1-2:19.0.0-3872.g38f9d900.el9.x86_64 103/170 2024-05-27T07:11:24.169 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : python3-protobuf-3.14.0-13.el9.noarch 80/170 2024-05-27T07:11:24.198 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : python3-rbd-2:19.0.0-3872.g38f9d900.el9.x86_64 104/170 2024-05-27T07:11:24.201 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : python3-grpcio-tools-1.46.7-10.el9.x86_64 81/170 2024-05-27T07:11:24.227 INFO:teuthology.orchestra.run.smithi052.stdout: Preparing : 1/1 2024-05-27T07:11:24.233 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : boost-program-options-1.75.0-8.el9.x86_64 105/170 2024-05-27T07:11:24.248 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 82/170 2024-05-27T07:11:24.313 INFO:teuthology.orchestra.run.smithi052.stdout: Upgrading : openssl-libs-1:3.2.1-1.el9.x86_64 1/170 2024-05-27T07:11:24.346 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : python3-packaging-20.9-5.el9.noarch 83/170 2024-05-27T07:11:24.354 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 2/170 2024-05-27T07:11:24.394 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 84/170 2024-05-27T07:11:24.408 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 3/170 2024-05-27T07:11:24.456 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : thrift-0.15.0-2.el9.x86_64 4/170 2024-05-27T07:11:24.481 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 5/170 2024-05-27T07:11:24.539 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 6/170 2024-05-27T07:11:24.725 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : parquet-libs-9.0.0-11.el9.x86_64 106/170 2024-05-27T07:11:24.762 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : librgw2-2:19.0.0-3872.g38f9d900.el9.x86_64 107/170 2024-05-27T07:11:24.841 INFO:teuthology.orchestra.run.smithi173.stdout: Running scriptlet: librgw2-2:19.0.0-3872.g38f9d900.el9.x86_64 107/170 2024-05-27T07:11:24.864 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : python3-rgw-2:19.0.0-3872.g38f9d900.el9.x86_64 108/170 2024-05-27T07:11:24.894 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 7/170 2024-05-27T07:11:25.006 INFO:teuthology.orchestra.run.smithi052.stdout: Upgrading : librados2-2:19.0.0-3872.g38f9d900.el9.x86_64 8/170 2024-05-27T07:11:25.104 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 85/170 2024-05-27T07:11:25.149 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 86/170 2024-05-27T07:11:25.158 INFO:teuthology.orchestra.run.smithi052.stdout: Running scriptlet: librados2-2:19.0.0-3872.g38f9d900.el9.x86_64 8/170 2024-05-27T07:11:25.183 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : libcephfs2-2:19.0.0-3872.g38f9d900.el9.x86_64 9/170 2024-05-27T07:11:25.194 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 87/170 2024-05-27T07:11:25.278 INFO:teuthology.orchestra.run.smithi052.stdout: Running scriptlet: libcephfs2-2:19.0.0-3872.g38f9d900.el9.x86_64 9/170 2024-05-27T07:11:25.288 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 88/170 2024-05-27T07:11:25.312 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : python3-rados-2:19.0.0-3872.g38f9d900.el9.x86_64 10/170 2024-05-27T07:11:25.355 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : python3-devel-3.9.18-3.el9.x86_64 89/170 2024-05-27T07:11:25.367 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : postgresql-private-libs-13.14-1.el9.x86_64 11/170 2024-05-27T07:11:25.399 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : python3-colorama-0.4.6-3.el9.noarch 90/170 2024-05-27T07:11:25.477 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : python3-attrs-20.3.0-7.el9.noarch 91/170 2024-05-27T07:11:25.511 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : python3-cmd2-2.4.2-2.el9.noarch 92/170 2024-05-27T07:11:25.608 INFO:teuthology.orchestra.run.smithi052.stdout: Upgrading : elfutils-libelf-0.191-4.el9.x86_64 12/170 2024-05-27T07:11:25.669 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : postgresql-13.14-1.el9.x86_64 13/170 2024-05-27T07:11:25.687 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : openblas-0.3.21-2.el9.x86_64 93/170 2024-05-27T07:11:25.697 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : libcephsqlite-2:19.0.0-3872.g38f9d900.el9.x86_64 14/170 2024-05-27T07:11:25.717 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : openblas-openmp-0.3.21-2.el9.x86_64 94/170 2024-05-27T07:11:25.793 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 95/170 2024-05-27T07:11:25.804 INFO:teuthology.orchestra.run.smithi052.stdout: Running scriptlet: libcephsqlite-2:19.0.0-3872.g38f9d900.el9.x86_64 14/170 2024-05-27T07:11:25.831 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : libradosstriper1-2:19.0.0-3872.g38f9d900.el9.x86 15/170 2024-05-27T07:11:25.885 INFO:teuthology.orchestra.run.smithi052.stdout: Running scriptlet: libradosstriper1-2:19.0.0-3872.g38f9d900.el9.x86 15/170 2024-05-27T07:11:25.907 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : qatlib-service-24.02.0-1.el9.x86_64 16/170 2024-05-27T07:11:25.934 INFO:teuthology.orchestra.run.smithi052.stdout: Running scriptlet: qatlib-service-24.02.0-1.el9.x86_64 16/170 2024-05-27T07:11:26.118 INFO:teuthology.orchestra.run.smithi052.stdout: Running scriptlet: qatlib-24.02.0-1.el9.x86_64 17/170 2024-05-27T07:11:26.288 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : qatlib-24.02.0-1.el9.x86_64 17/170 2024-05-27T07:11:26.419 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 96/170 2024-05-27T07:11:26.454 INFO:teuthology.orchestra.run.smithi052.stdout: Upgrading : NetworkManager-libnm-1:1.47.90-1.el9.x86_64 18/170 2024-05-27T07:11:26.515 INFO:teuthology.orchestra.run.smithi052.stdout: Running scriptlet: NetworkManager-libnm-1:1.47.90-1.el9.x86_64 18/170 2024-05-27T07:11:26.539 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 97/170 2024-05-27T07:11:26.628 INFO:teuthology.orchestra.run.smithi052.stdout: Running scriptlet: NetworkManager-1:1.47.90-1.el9.x86_64 19/170 2024-05-27T07:11:26.707 INFO:teuthology.orchestra.run.smithi052.stdout: Upgrading : NetworkManager-1:1.47.90-1.el9.x86_64 19/170 2024-05-27T07:11:26.917 INFO:teuthology.orchestra.run.smithi052.stdout: Running scriptlet: NetworkManager-1:1.47.90-1.el9.x86_64 19/170 2024-05-27T07:11:27.025 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : re2-1:20211101-3.el9.x86_64 20/170 2024-05-27T07:11:27.053 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : libarrow-9.0.0-11.el9.x86_64 21/170 2024-05-27T07:11:27.089 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : liboath-2.6.7-2.el9.x86_64 22/170 2024-05-27T07:11:27.146 INFO:teuthology.orchestra.run.smithi173.stdout: Running scriptlet: ceph-common-2:19.0.0-3872.g38f9d900.el9.x86_64 109/170 2024-05-27T07:11:27.165 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 23/170 2024-05-27T07:11:27.178 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : ceph-common-2:19.0.0-3872.g38f9d900.el9.x86_64 109/170 2024-05-27T07:11:27.205 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 24/170 2024-05-27T07:11:27.249 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 25/170 2024-05-27T07:11:27.282 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : protobuf-3.14.0-13.el9.x86_64 26/170 2024-05-27T07:11:27.309 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : lua-5.4.4-4.el9.x86_64 27/170 2024-05-27T07:11:27.381 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 28/170 2024-05-27T07:11:27.433 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : python3-urllib3-1.26.5-5.el9.noarch 29/170 2024-05-27T07:11:27.462 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : python3-requests-2.25.1-8.el9.noarch 30/170 2024-05-27T07:11:27.497 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : libquadmath-11.4.1-3.el9.x86_64 31/170 2024-05-27T07:11:27.577 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : libgfortran-11.4.1-3.el9.x86_64 32/170 2024-05-27T07:11:27.608 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : python3-ceph-common-2:19.0.0-3872.g38f9d900.el9. 33/170 2024-05-27T07:11:27.652 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : python3-ceph-argparse-2:19.0.0-3872.g38f9d900.el 34/170 2024-05-27T07:11:27.682 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : python3-cephfs-2:19.0.0-3872.g38f9d900.el9.x86_6 35/170 2024-05-27T07:11:27.737 INFO:teuthology.orchestra.run.smithi173.stdout: Running scriptlet: ceph-common-2:19.0.0-3872.g38f9d900.el9.x86_64 109/170 2024-05-27T07:11:27.746 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 98/170 2024-05-27T07:11:27.754 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : lua-devel-5.4.4-4.el9.x86_64 36/170 2024-05-27T07:11:27.790 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : ceph-base-2:19.0.0-3872.g38f9d900.el9.x86_64 110/170 2024-05-27T07:11:27.803 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : luarocks-3.9.2-5.el9.noarch 37/170 2024-05-27T07:11:27.846 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 99/170 2024-05-27T07:11:27.874 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : libzstd-devel-1.5.1-2.el9.x86_64 100/170 2024-05-27T07:11:27.886 INFO:teuthology.orchestra.run.smithi173.stdout: Running scriptlet: ceph-base-2:19.0.0-3872.g38f9d900.el9.x86_64 110/170 2024-05-27T07:11:27.886 INFO:teuthology.orchestra.run.smithi173.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2024-05-27T07:11:27.886 INFO:teuthology.orchestra.run.smithi173.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2024-05-27T07:11:27.887 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-05-27T07:11:27.890 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : protobuf-compiler-3.14.0-13.el9.x86_64 38/170 2024-05-27T07:11:27.907 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : libuuid-devel-2.37.4-18.el9.x86_64 101/170 2024-05-27T07:11:27.916 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : ceph-selinux-2:19.0.0-3872.g38f9d900.el9.x86_64 111/170 2024-05-27T07:11:28.037 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 39/170 2024-05-27T07:11:28.099 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 40/170 2024-05-27T07:11:28.225 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 102/170 2024-05-27T07:11:28.278 INFO:teuthology.orchestra.run.smithi105.stdout: Upgrading : librbd1-2:19.0.0-3872.g38f9d900.el9.x86_64 103/170 2024-05-27T07:11:28.368 INFO:teuthology.orchestra.run.smithi105.stdout: Running scriptlet: librbd1-2:19.0.0-3872.g38f9d900.el9.x86_64 103/170 2024-05-27T07:11:28.399 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : python3-rbd-2:19.0.0-3872.g38f9d900.el9.x86_64 104/170 2024-05-27T07:11:28.435 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : boost-program-options-1.75.0-8.el9.x86_64 105/170 2024-05-27T07:11:28.665 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : python3-rsa-4.9-2.el9.noarch 41/170 2024-05-27T07:11:28.755 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 42/170 2024-05-27T07:11:28.788 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : python3-jinja2-2.11.3-6.el9.noarch 43/170 2024-05-27T07:11:28.834 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : qatzip-libs-1.2.0-1.el9.x86_64 44/170 2024-05-27T07:11:28.882 INFO:teuthology.orchestra.run.smithi052.stdout: Upgrading : elfutils-libs-0.191-4.el9.x86_64 45/170 2024-05-27T07:11:28.921 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : librados-devel-2:19.0.0-3872.g38f9d900.el9.x86_6 46/170 2024-05-27T07:11:28.924 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : parquet-libs-9.0.0-11.el9.x86_64 106/170 2024-05-27T07:11:28.963 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 47/170 2024-05-27T07:11:28.964 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : librgw2-2:19.0.0-3872.g38f9d900.el9.x86_64 107/170 2024-05-27T07:11:29.035 INFO:teuthology.orchestra.run.smithi105.stdout: Running scriptlet: librgw2-2:19.0.0-3872.g38f9d900.el9.x86_64 107/170 2024-05-27T07:11:29.057 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : python3-rgw-2:19.0.0-3872.g38f9d900.el9.x86_64 108/170 2024-05-27T07:11:29.118 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : socat-1.7.4.1-5.el9.x86_64 48/170 2024-05-27T07:11:29.198 INFO:teuthology.orchestra.run.smithi052.stdout: Upgrading : openssl-1:3.2.1-1.el9.x86_64 49/170 2024-05-27T07:11:29.295 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 50/170 2024-05-27T07:11:29.356 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 51/170 2024-05-27T07:11:29.429 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 52/170 2024-05-27T07:11:29.466 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : python3-webob-1.8.7-6.el9.noarch 53/170 2024-05-27T07:11:29.503 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : python3-typing-extensions-4.4.0-2.el9.noarch 54/170 2024-05-27T07:11:29.551 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 55/170 2024-05-27T07:11:29.585 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 56/170 2024-05-27T07:11:29.629 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : python3-pyperclip-1.8.2-2.el9.noarch 57/170 2024-05-27T07:11:29.679 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 58/170 2024-05-27T07:11:29.824 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 59/170 2024-05-27T07:11:29.864 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 60/170 2024-05-27T07:11:29.906 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : python3-certifi-2021.10.8-2.el9.noarch 61/170 2024-05-27T07:11:30.005 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 62/170 2024-05-27T07:11:30.038 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : python3-google-auth-1:2.29.0-1.el9.noarch 63/170 2024-05-27T07:11:30.071 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 64/170 2024-05-27T07:11:30.104 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : gperftools-libs-2.9.1-2.el9.x86_64 65/170 2024-05-27T07:11:30.128 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : libarrow-doc-9.0.0-11.el9.noarch 66/170 2024-05-27T07:11:30.229 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : grpc-data-1.46.7-10.el9.noarch 67/170 2024-05-27T07:11:30.365 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : abseil-cpp-20211102.0-3.el9.x86_64 68/170 2024-05-27T07:11:30.405 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : python3-grpcio-1.46.7-10.el9.x86_64 69/170 2024-05-27T07:11:30.433 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : zlib-devel-1.2.11-41.el9.x86_64 70/170 2024-05-27T07:11:30.474 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 71/170 2024-05-27T07:11:30.515 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : uuid-1.6.2-55.el9.x86_64 72/170 2024-05-27T07:11:30.555 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : python3-wcwidth-0.2.5-8.el9.noarch 73/170 2024-05-27T07:11:30.584 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 74/170 2024-05-27T07:11:30.618 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 75/170 2024-05-27T07:11:30.652 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 76/170 2024-05-27T07:11:30.686 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 77/170 2024-05-27T07:11:30.719 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 78/170 2024-05-27T07:11:30.812 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 79/170 2024-05-27T07:11:30.868 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : python3-protobuf-3.14.0-13.el9.noarch 80/170 2024-05-27T07:11:30.898 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : python3-grpcio-tools-1.46.7-10.el9.x86_64 81/170 2024-05-27T07:11:30.946 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 82/170 2024-05-27T07:11:31.058 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : python3-packaging-20.9-5.el9.noarch 83/170 2024-05-27T07:11:31.109 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 84/170 2024-05-27T07:11:31.310 INFO:teuthology.orchestra.run.smithi105.stdout: Running scriptlet: ceph-common-2:19.0.0-3872.g38f9d900.el9.x86_64 109/170 2024-05-27T07:11:31.346 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : ceph-common-2:19.0.0-3872.g38f9d900.el9.x86_64 109/170 2024-05-27T07:11:31.898 INFO:teuthology.orchestra.run.smithi105.stdout: Running scriptlet: ceph-common-2:19.0.0-3872.g38f9d900.el9.x86_64 109/170 2024-05-27T07:11:31.909 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 85/170 2024-05-27T07:11:31.949 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : ceph-base-2:19.0.0-3872.g38f9d900.el9.x86_64 110/170 2024-05-27T07:11:31.954 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 86/170 2024-05-27T07:11:31.988 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 87/170 2024-05-27T07:11:32.041 INFO:teuthology.orchestra.run.smithi105.stdout: Running scriptlet: ceph-base-2:19.0.0-3872.g38f9d900.el9.x86_64 110/170 2024-05-27T07:11:32.041 INFO:teuthology.orchestra.run.smithi105.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2024-05-27T07:11:32.041 INFO:teuthology.orchestra.run.smithi105.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2024-05-27T07:11:32.041 INFO:teuthology.orchestra.run.smithi105.stdout: 2024-05-27T07:11:32.075 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : ceph-selinux-2:19.0.0-3872.g38f9d900.el9.x86_64 111/170 2024-05-27T07:11:32.094 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 88/170 2024-05-27T07:11:32.145 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : python3-devel-3.9.18-3.el9.x86_64 89/170 2024-05-27T07:11:32.190 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : python3-colorama-0.4.6-3.el9.noarch 90/170 2024-05-27T07:11:32.281 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : python3-attrs-20.3.0-7.el9.noarch 91/170 2024-05-27T07:11:32.324 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : python3-cmd2-2.4.2-2.el9.noarch 92/170 2024-05-27T07:11:32.509 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : openblas-0.3.21-2.el9.x86_64 93/170 2024-05-27T07:11:32.538 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : openblas-openmp-0.3.21-2.el9.x86_64 94/170 2024-05-27T07:11:32.621 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 95/170 2024-05-27T07:11:33.330 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 96/170 2024-05-27T07:11:33.449 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 97/170 2024-05-27T07:11:34.807 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 98/170 2024-05-27T07:11:34.900 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 99/170 2024-05-27T07:11:34.929 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : libzstd-devel-1.5.1-2.el9.x86_64 100/170 2024-05-27T07:11:34.962 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : libuuid-devel-2.37.4-18.el9.x86_64 101/170 2024-05-27T07:11:35.297 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 102/170 2024-05-27T07:11:35.341 INFO:teuthology.orchestra.run.smithi052.stdout: Upgrading : librbd1-2:19.0.0-3872.g38f9d900.el9.x86_64 103/170 2024-05-27T07:11:35.433 INFO:teuthology.orchestra.run.smithi052.stdout: Running scriptlet: librbd1-2:19.0.0-3872.g38f9d900.el9.x86_64 103/170 2024-05-27T07:11:35.463 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : python3-rbd-2:19.0.0-3872.g38f9d900.el9.x86_64 104/170 2024-05-27T07:11:35.500 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : boost-program-options-1.75.0-8.el9.x86_64 105/170 2024-05-27T07:11:36.017 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : parquet-libs-9.0.0-11.el9.x86_64 106/170 2024-05-27T07:11:36.044 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : librgw2-2:19.0.0-3872.g38f9d900.el9.x86_64 107/170 2024-05-27T07:11:36.099 INFO:teuthology.orchestra.run.smithi052.stdout: Running scriptlet: librgw2-2:19.0.0-3872.g38f9d900.el9.x86_64 107/170 2024-05-27T07:11:36.129 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : python3-rgw-2:19.0.0-3872.g38f9d900.el9.x86_64 108/170 2024-05-27T07:11:38.499 INFO:teuthology.orchestra.run.smithi052.stdout: Running scriptlet: ceph-common-2:19.0.0-3872.g38f9d900.el9.x86_64 109/170 2024-05-27T07:11:38.534 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : ceph-common-2:19.0.0-3872.g38f9d900.el9.x86_64 109/170 2024-05-27T07:11:39.125 INFO:teuthology.orchestra.run.smithi052.stdout: Running scriptlet: ceph-common-2:19.0.0-3872.g38f9d900.el9.x86_64 109/170 2024-05-27T07:11:39.162 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : ceph-base-2:19.0.0-3872.g38f9d900.el9.x86_64 110/170 2024-05-27T07:11:39.250 INFO:teuthology.orchestra.run.smithi052.stdout: Running scriptlet: ceph-base-2:19.0.0-3872.g38f9d900.el9.x86_64 110/170 2024-05-27T07:11:39.251 INFO:teuthology.orchestra.run.smithi052.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2024-05-27T07:11:39.251 INFO:teuthology.orchestra.run.smithi052.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2024-05-27T07:11:39.251 INFO:teuthology.orchestra.run.smithi052.stdout: 2024-05-27T07:11:39.289 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : ceph-selinux-2:19.0.0-3872.g38f9d900.el9.x86_64 111/170 2024-05-27T07:11:39.721 INFO:teuthology.orchestra.run.smithi173.stdout: Running scriptlet: ceph-selinux-2:19.0.0-3872.g38f9d900.el9.x86_64 111/170 2024-05-27T07:11:39.721 INFO:teuthology.orchestra.run.smithi173.stdout:skipping the directory /sys 2024-05-27T07:11:39.721 INFO:teuthology.orchestra.run.smithi173.stdout:skipping the directory /proc 2024-05-27T07:11:39.721 INFO:teuthology.orchestra.run.smithi173.stdout:skipping the directory /mnt 2024-05-27T07:11:39.721 INFO:teuthology.orchestra.run.smithi173.stdout:skipping the directory /var/tmp 2024-05-27T07:11:39.721 INFO:teuthology.orchestra.run.smithi173.stdout:skipping the directory /home 2024-05-27T07:11:39.721 INFO:teuthology.orchestra.run.smithi173.stdout:skipping the directory /root 2024-05-27T07:11:39.722 INFO:teuthology.orchestra.run.smithi173.stdout:skipping the directory /tmp 2024-05-27T07:11:39.722 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-05-27T07:11:39.907 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : ceph-volume-2:19.0.0-3872.g38f9d900.el9.noarch 112/170 2024-05-27T07:11:39.925 INFO:teuthology.orchestra.run.smithi173.stdout: Running scriptlet: ceph-volume-2:19.0.0-3872.g38f9d900.el9.noarch 112/170 2024-05-27T07:11:39.925 INFO:teuthology.orchestra.run.smithi173.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-05-27T07:11:39.925 INFO:teuthology.orchestra.run.smithi173.stdout:Invalid unit name "ceph-volume@*.service" escaped as "ceph-volume@\x2a.service". 2024-05-27T07:11:39.926 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-05-27T07:11:41.510 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : ceph-osd-2:19.0.0-3872.g38f9d900.el9.x86_64 113/170 2024-05-27T07:11:41.537 INFO:teuthology.orchestra.run.smithi173.stdout: Running scriptlet: ceph-osd-2:19.0.0-3872.g38f9d900.el9.x86_64 113/170 2024-05-27T07:11:41.537 INFO:teuthology.orchestra.run.smithi173.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-05-27T07:11:41.537 INFO:teuthology.orchestra.run.smithi173.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2024-05-27T07:11:41.537 INFO:teuthology.orchestra.run.smithi173.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-05-27T07:11:41.537 INFO:teuthology.orchestra.run.smithi173.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-05-27T07:11:41.537 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-05-27T07:11:41.761 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : ceph-mds-2:19.0.0-3872.g38f9d900.el9.x86_64 114/170 2024-05-27T07:11:41.783 INFO:teuthology.orchestra.run.smithi173.stdout: Running scriptlet: ceph-mds-2:19.0.0-3872.g38f9d900.el9.x86_64 114/170 2024-05-27T07:11:41.784 INFO:teuthology.orchestra.run.smithi173.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-05-27T07:11:41.784 INFO:teuthology.orchestra.run.smithi173.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2024-05-27T07:11:41.784 INFO:teuthology.orchestra.run.smithi173.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-05-27T07:11:41.784 INFO:teuthology.orchestra.run.smithi173.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-05-27T07:11:41.784 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-05-27T07:11:42.253 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : ceph-mon-2:19.0.0-3872.g38f9d900.el9.x86_64 115/170 2024-05-27T07:11:42.276 INFO:teuthology.orchestra.run.smithi173.stdout: Running scriptlet: ceph-mon-2:19.0.0-3872.g38f9d900.el9.x86_64 115/170 2024-05-27T07:11:42.276 INFO:teuthology.orchestra.run.smithi173.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-05-27T07:11:42.276 INFO:teuthology.orchestra.run.smithi173.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2024-05-27T07:11:42.276 INFO:teuthology.orchestra.run.smithi173.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-05-27T07:11:42.277 INFO:teuthology.orchestra.run.smithi173.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-05-27T07:11:42.277 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-05-27T07:11:42.383 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : automake-1.16.2-8.el9.noarch 116/170 2024-05-27T07:11:42.449 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : python3-ply-3.11-14.el9.noarch 117/170 2024-05-27T07:11:42.504 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 118/170 2024-05-27T07:11:42.707 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 119/170 2024-05-27T07:11:42.762 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 120/170 2024-05-27T07:11:42.848 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 121/170 2024-05-27T07:11:42.945 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 122/170 2024-05-27T07:11:43.084 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 123/170 2024-05-27T07:11:43.121 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : python3-asyncssh-2.13.2-5.el9.noarch 124/170 2024-05-27T07:11:43.162 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 125/170 2024-05-27T07:11:43.197 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : fuse-2.9.9-15.el9.x86_64 126/170 2024-05-27T07:11:43.624 INFO:teuthology.orchestra.run.smithi173.stdout: Running scriptlet: cephadm-2:19.0.0-3872.g38f9d900.el9.noarch 127/170 2024-05-27T07:11:43.654 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : cephadm-2:19.0.0-3872.g38f9d900.el9.noarch 127/170 2024-05-27T07:11:43.686 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : ceph-prometheus-alerts-2:19.0.0-3872.g38f9d900.e 128/170 2024-05-27T07:11:43.710 INFO:teuthology.orchestra.run.smithi105.stdout: Running scriptlet: ceph-selinux-2:19.0.0-3872.g38f9d900.el9.x86_64 111/170 2024-05-27T07:11:43.710 INFO:teuthology.orchestra.run.smithi105.stdout:skipping the directory /sys 2024-05-27T07:11:43.710 INFO:teuthology.orchestra.run.smithi105.stdout:skipping the directory /proc 2024-05-27T07:11:43.710 INFO:teuthology.orchestra.run.smithi105.stdout:skipping the directory /mnt 2024-05-27T07:11:43.710 INFO:teuthology.orchestra.run.smithi105.stdout:skipping the directory /var/tmp 2024-05-27T07:11:43.710 INFO:teuthology.orchestra.run.smithi105.stdout:skipping the directory /home 2024-05-27T07:11:43.710 INFO:teuthology.orchestra.run.smithi105.stdout:skipping the directory /root 2024-05-27T07:11:43.710 INFO:teuthology.orchestra.run.smithi105.stdout:skipping the directory /tmp 2024-05-27T07:11:43.711 INFO:teuthology.orchestra.run.smithi105.stdout: 2024-05-27T07:11:43.748 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : ceph-grafana-dashboards-2:19.0.0-3872.g38f9d900. 129/170 2024-05-27T07:11:43.786 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : ceph-mgr-cephadm-2:19.0.0-3872.g38f9d900.el9.noa 130/170 2024-05-27T07:11:43.875 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : ceph-volume-2:19.0.0-3872.g38f9d900.el9.noarch 112/170 2024-05-27T07:11:43.890 INFO:teuthology.orchestra.run.smithi105.stdout: Running scriptlet: ceph-volume-2:19.0.0-3872.g38f9d900.el9.noarch 112/170 2024-05-27T07:11:43.890 INFO:teuthology.orchestra.run.smithi105.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-05-27T07:11:43.890 INFO:teuthology.orchestra.run.smithi105.stdout:Invalid unit name "ceph-volume@*.service" escaped as "ceph-volume@\x2a.service". 2024-05-27T07:11:43.890 INFO:teuthology.orchestra.run.smithi105.stdout: 2024-05-27T07:11:44.819 INFO:teuthology.orchestra.run.smithi173.stdout: Running scriptlet: ceph-mgr-cephadm-2:19.0.0-3872.g38f9d900.el9.noa 130/170 2024-05-27T07:11:44.854 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : ceph-mgr-dashboard-2:19.0.0-3872.g38f9d900.el9.n 131/170 2024-05-27T07:11:45.453 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : ceph-osd-2:19.0.0-3872.g38f9d900.el9.x86_64 113/170 2024-05-27T07:11:45.478 INFO:teuthology.orchestra.run.smithi105.stdout: Running scriptlet: ceph-osd-2:19.0.0-3872.g38f9d900.el9.x86_64 113/170 2024-05-27T07:11:45.479 INFO:teuthology.orchestra.run.smithi105.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-05-27T07:11:45.479 INFO:teuthology.orchestra.run.smithi105.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2024-05-27T07:11:45.479 INFO:teuthology.orchestra.run.smithi105.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-05-27T07:11:45.479 INFO:teuthology.orchestra.run.smithi105.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-05-27T07:11:45.479 INFO:teuthology.orchestra.run.smithi105.stdout: 2024-05-27T07:11:45.704 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : ceph-mds-2:19.0.0-3872.g38f9d900.el9.x86_64 114/170 2024-05-27T07:11:45.726 INFO:teuthology.orchestra.run.smithi105.stdout: Running scriptlet: ceph-mds-2:19.0.0-3872.g38f9d900.el9.x86_64 114/170 2024-05-27T07:11:45.726 INFO:teuthology.orchestra.run.smithi105.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-05-27T07:11:45.726 INFO:teuthology.orchestra.run.smithi105.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2024-05-27T07:11:45.726 INFO:teuthology.orchestra.run.smithi105.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-05-27T07:11:45.726 INFO:teuthology.orchestra.run.smithi105.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-05-27T07:11:45.726 INFO:teuthology.orchestra.run.smithi105.stdout: 2024-05-27T07:11:45.818 INFO:teuthology.orchestra.run.smithi173.stdout: Running scriptlet: ceph-mgr-dashboard-2:19.0.0-3872.g38f9d900.el9.n 131/170 2024-05-27T07:11:45.846 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : ceph-mgr-diskprediction-local-2:19.0.0-3872.g38f 132/170 2024-05-27T07:11:45.957 INFO:teuthology.orchestra.run.smithi173.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:19.0.0-3872.g38f 132/170 2024-05-27T07:11:46.121 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : ceph-mgr-modules-core-2:19.0.0-3872.g38f9d900.el 133/170 2024-05-27T07:11:46.157 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : ceph-mgr-2:19.0.0-3872.g38f9d900.el9.x86_64 134/170 2024-05-27T07:11:46.180 INFO:teuthology.orchestra.run.smithi173.stdout: Running scriptlet: ceph-mgr-2:19.0.0-3872.g38f9d900.el9.x86_64 134/170 2024-05-27T07:11:46.180 INFO:teuthology.orchestra.run.smithi173.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-05-27T07:11:46.180 INFO:teuthology.orchestra.run.smithi173.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2024-05-27T07:11:46.180 INFO:teuthology.orchestra.run.smithi173.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-05-27T07:11:46.180 INFO:teuthology.orchestra.run.smithi173.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-05-27T07:11:46.180 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-05-27T07:11:46.188 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : ceph-mon-2:19.0.0-3872.g38f9d900.el9.x86_64 115/170 2024-05-27T07:11:46.210 INFO:teuthology.orchestra.run.smithi105.stdout: Running scriptlet: ceph-mon-2:19.0.0-3872.g38f9d900.el9.x86_64 115/170 2024-05-27T07:11:46.210 INFO:teuthology.orchestra.run.smithi105.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-05-27T07:11:46.210 INFO:teuthology.orchestra.run.smithi105.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2024-05-27T07:11:46.210 INFO:teuthology.orchestra.run.smithi105.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-05-27T07:11:46.210 INFO:teuthology.orchestra.run.smithi105.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-05-27T07:11:46.210 INFO:teuthology.orchestra.run.smithi105.stdout: 2024-05-27T07:11:46.224 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : ceph-mgr-rook-2:19.0.0-3872.g38f9d900.el9.noarch 135/170 2024-05-27T07:11:46.235 INFO:teuthology.orchestra.run.smithi173.stdout: Running scriptlet: ceph-mgr-rook-2:19.0.0-3872.g38f9d900.el9.noarch 135/170 2024-05-27T07:11:46.310 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : automake-1.16.2-8.el9.noarch 116/170 2024-05-27T07:11:46.335 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : ceph-2:19.0.0-3872.g38f9d900.el9.x86_64 136/170 2024-05-27T07:11:46.367 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : python3-ply-3.11-14.el9.noarch 117/170 2024-05-27T07:11:46.389 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : ceph-fuse-2:19.0.0-3872.g38f9d900.el9.x86_64 137/170 2024-05-27T07:11:46.422 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 118/170 2024-05-27T07:11:46.606 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 119/170 2024-05-27T07:11:46.655 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 120/170 2024-05-27T07:11:46.733 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 121/170 2024-05-27T07:11:46.823 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 122/170 2024-05-27T07:11:46.964 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 123/170 2024-05-27T07:11:47.014 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : python3-asyncssh-2.13.2-5.el9.noarch 124/170 2024-05-27T07:11:47.046 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 125/170 2024-05-27T07:11:47.073 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : fuse-2.9.9-15.el9.x86_64 126/170 2024-05-27T07:11:47.321 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : libtool-2.4.6-45.el9.x86_64 138/170 2024-05-27T07:11:47.355 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : ceph-radosgw-2:19.0.0-3872.g38f9d900.el9.x86_64 139/170 2024-05-27T07:11:47.374 INFO:teuthology.orchestra.run.smithi173.stdout: Running scriptlet: ceph-radosgw-2:19.0.0-3872.g38f9d900.el9.x86_64 139/170 2024-05-27T07:11:47.375 INFO:teuthology.orchestra.run.smithi173.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-05-27T07:11:47.375 INFO:teuthology.orchestra.run.smithi173.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2024-05-27T07:11:47.375 INFO:teuthology.orchestra.run.smithi173.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-05-27T07:11:47.375 INFO:teuthology.orchestra.run.smithi173.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-05-27T07:11:47.375 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-05-27T07:11:47.419 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : ceph-immutable-object-cache-2:19.0.0-3872.g38f9d 140/170 2024-05-27T07:11:47.441 INFO:teuthology.orchestra.run.smithi173.stdout: Running scriptlet: ceph-immutable-object-cache-2:19.0.0-3872.g38f9d 140/170 2024-05-27T07:11:47.441 INFO:teuthology.orchestra.run.smithi173.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-05-27T07:11:47.441 INFO:teuthology.orchestra.run.smithi173.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2024-05-27T07:11:47.441 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-05-27T07:11:47.486 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : cephfs-mirror-2:19.0.0-3872.g38f9d900.el9.x86_64 141/170 2024-05-27T07:11:47.500 INFO:teuthology.orchestra.run.smithi105.stdout: Running scriptlet: cephadm-2:19.0.0-3872.g38f9d900.el9.noarch 127/170 2024-05-27T07:11:47.508 INFO:teuthology.orchestra.run.smithi173.stdout: Running scriptlet: cephfs-mirror-2:19.0.0-3872.g38f9d900.el9.x86_64 141/170 2024-05-27T07:11:47.508 INFO:teuthology.orchestra.run.smithi173.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-05-27T07:11:47.509 INFO:teuthology.orchestra.run.smithi173.stdout:Invalid unit name "cephfs-mirror@*.service" escaped as "cephfs-mirror@\x2a.service". 2024-05-27T07:11:47.509 INFO:teuthology.orchestra.run.smithi173.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/cephfs-mirror.target → /usr/lib/systemd/system/cephfs-mirror.target. 2024-05-27T07:11:47.509 INFO:teuthology.orchestra.run.smithi173.stdout:Created symlink /etc/systemd/system/ceph.target.wants/cephfs-mirror.target → /usr/lib/systemd/system/cephfs-mirror.target. 2024-05-27T07:11:47.509 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-05-27T07:11:47.530 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : cephadm-2:19.0.0-3872.g38f9d900.el9.noarch 127/170 2024-05-27T07:11:47.562 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : ceph-prometheus-alerts-2:19.0.0-3872.g38f9d900.e 128/170 2024-05-27T07:11:47.625 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : ceph-grafana-dashboards-2:19.0.0-3872.g38f9d900. 129/170 2024-05-27T07:11:47.653 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : ceph-mgr-cephadm-2:19.0.0-3872.g38f9d900.el9.noa 130/170 2024-05-27T07:11:47.829 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : rbd-mirror-2:19.0.0-3872.g38f9d900.el9.x86_64 142/170 2024-05-27T07:11:47.851 INFO:teuthology.orchestra.run.smithi173.stdout: Running scriptlet: rbd-mirror-2:19.0.0-3872.g38f9d900.el9.x86_64 142/170 2024-05-27T07:11:47.852 INFO:teuthology.orchestra.run.smithi173.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-05-27T07:11:47.852 INFO:teuthology.orchestra.run.smithi173.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2024-05-27T07:11:47.852 INFO:teuthology.orchestra.run.smithi173.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-05-27T07:11:47.852 INFO:teuthology.orchestra.run.smithi173.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-05-27T07:11:47.852 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-05-27T07:11:48.688 INFO:teuthology.orchestra.run.smithi105.stdout: Running scriptlet: ceph-mgr-cephadm-2:19.0.0-3872.g38f9d900.el9.noa 130/170 2024-05-27T07:11:48.744 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : ceph-mgr-dashboard-2:19.0.0-3872.g38f9d900.el9.n 131/170 2024-05-27T07:11:49.705 INFO:teuthology.orchestra.run.smithi105.stdout: Running scriptlet: ceph-mgr-dashboard-2:19.0.0-3872.g38f9d900.el9.n 131/170 2024-05-27T07:11:49.730 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : ceph-mgr-diskprediction-local-2:19.0.0-3872.g38f 132/170 2024-05-27T07:11:49.842 INFO:teuthology.orchestra.run.smithi105.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:19.0.0-3872.g38f 132/170 2024-05-27T07:11:50.014 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : ceph-mgr-modules-core-2:19.0.0-3872.g38f9d900.el 133/170 2024-05-27T07:11:50.041 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : ceph-mgr-2:19.0.0-3872.g38f9d900.el9.x86_64 134/170 2024-05-27T07:11:50.064 INFO:teuthology.orchestra.run.smithi105.stdout: Running scriptlet: ceph-mgr-2:19.0.0-3872.g38f9d900.el9.x86_64 134/170 2024-05-27T07:11:50.064 INFO:teuthology.orchestra.run.smithi105.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-05-27T07:11:50.064 INFO:teuthology.orchestra.run.smithi105.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2024-05-27T07:11:50.064 INFO:teuthology.orchestra.run.smithi105.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-05-27T07:11:50.064 INFO:teuthology.orchestra.run.smithi105.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-05-27T07:11:50.064 INFO:teuthology.orchestra.run.smithi105.stdout: 2024-05-27T07:11:50.108 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : ceph-mgr-rook-2:19.0.0-3872.g38f9d900.el9.noarch 135/170 2024-05-27T07:11:50.119 INFO:teuthology.orchestra.run.smithi105.stdout: Running scriptlet: ceph-mgr-rook-2:19.0.0-3872.g38f9d900.el9.noarch 135/170 2024-05-27T07:11:50.219 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : ceph-2:19.0.0-3872.g38f9d900.el9.x86_64 136/170 2024-05-27T07:11:50.273 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : ceph-fuse-2:19.0.0-3872.g38f9d900.el9.x86_64 137/170 2024-05-27T07:11:51.211 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : libtool-2.4.6-45.el9.x86_64 138/170 2024-05-27T07:11:51.252 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : ceph-radosgw-2:19.0.0-3872.g38f9d900.el9.x86_64 139/170 2024-05-27T07:11:51.275 INFO:teuthology.orchestra.run.smithi105.stdout: Running scriptlet: ceph-radosgw-2:19.0.0-3872.g38f9d900.el9.x86_64 139/170 2024-05-27T07:11:51.275 INFO:teuthology.orchestra.run.smithi105.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-05-27T07:11:51.275 INFO:teuthology.orchestra.run.smithi105.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2024-05-27T07:11:51.275 INFO:teuthology.orchestra.run.smithi105.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-05-27T07:11:51.275 INFO:teuthology.orchestra.run.smithi105.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-05-27T07:11:51.275 INFO:teuthology.orchestra.run.smithi105.stdout: 2024-05-27T07:11:51.320 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : ceph-immutable-object-cache-2:19.0.0-3872.g38f9d 140/170 2024-05-27T07:11:51.341 INFO:teuthology.orchestra.run.smithi105.stdout: Running scriptlet: ceph-immutable-object-cache-2:19.0.0-3872.g38f9d 140/170 2024-05-27T07:11:51.342 INFO:teuthology.orchestra.run.smithi105.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-05-27T07:11:51.342 INFO:teuthology.orchestra.run.smithi105.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2024-05-27T07:11:51.342 INFO:teuthology.orchestra.run.smithi105.stdout: 2024-05-27T07:11:51.387 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : cephfs-mirror-2:19.0.0-3872.g38f9d900.el9.x86_64 141/170 2024-05-27T07:11:51.409 INFO:teuthology.orchestra.run.smithi105.stdout: Running scriptlet: cephfs-mirror-2:19.0.0-3872.g38f9d900.el9.x86_64 141/170 2024-05-27T07:11:51.410 INFO:teuthology.orchestra.run.smithi105.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-05-27T07:11:51.410 INFO:teuthology.orchestra.run.smithi105.stdout:Invalid unit name "cephfs-mirror@*.service" escaped as "cephfs-mirror@\x2a.service". 2024-05-27T07:11:51.410 INFO:teuthology.orchestra.run.smithi105.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/cephfs-mirror.target → /usr/lib/systemd/system/cephfs-mirror.target. 2024-05-27T07:11:51.410 INFO:teuthology.orchestra.run.smithi105.stdout:Created symlink /etc/systemd/system/ceph.target.wants/cephfs-mirror.target → /usr/lib/systemd/system/cephfs-mirror.target. 2024-05-27T07:11:51.410 INFO:teuthology.orchestra.run.smithi105.stdout: 2024-05-27T07:11:51.730 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : rbd-mirror-2:19.0.0-3872.g38f9d900.el9.x86_64 142/170 2024-05-27T07:11:51.751 INFO:teuthology.orchestra.run.smithi105.stdout: Running scriptlet: rbd-mirror-2:19.0.0-3872.g38f9d900.el9.x86_64 142/170 2024-05-27T07:11:51.751 INFO:teuthology.orchestra.run.smithi105.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-05-27T07:11:51.751 INFO:teuthology.orchestra.run.smithi105.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2024-05-27T07:11:51.752 INFO:teuthology.orchestra.run.smithi105.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-05-27T07:11:51.752 INFO:teuthology.orchestra.run.smithi105.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-05-27T07:11:51.752 INFO:teuthology.orchestra.run.smithi105.stdout: 2024-05-27T07:11:51.850 INFO:teuthology.orchestra.run.smithi052.stdout: Running scriptlet: ceph-selinux-2:19.0.0-3872.g38f9d900.el9.x86_64 111/170 2024-05-27T07:11:51.850 INFO:teuthology.orchestra.run.smithi052.stdout:skipping the directory /sys 2024-05-27T07:11:51.850 INFO:teuthology.orchestra.run.smithi052.stdout:skipping the directory /proc 2024-05-27T07:11:51.850 INFO:teuthology.orchestra.run.smithi052.stdout:skipping the directory /mnt 2024-05-27T07:11:51.851 INFO:teuthology.orchestra.run.smithi052.stdout:skipping the directory /var/tmp 2024-05-27T07:11:51.851 INFO:teuthology.orchestra.run.smithi052.stdout:skipping the directory /home 2024-05-27T07:11:51.851 INFO:teuthology.orchestra.run.smithi052.stdout:skipping the directory /root 2024-05-27T07:11:51.851 INFO:teuthology.orchestra.run.smithi052.stdout:skipping the directory /tmp 2024-05-27T07:11:51.851 INFO:teuthology.orchestra.run.smithi052.stdout: 2024-05-27T07:11:52.029 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : ceph-volume-2:19.0.0-3872.g38f9d900.el9.noarch 112/170 2024-05-27T07:11:52.045 INFO:teuthology.orchestra.run.smithi052.stdout: Running scriptlet: ceph-volume-2:19.0.0-3872.g38f9d900.el9.noarch 112/170 2024-05-27T07:11:52.045 INFO:teuthology.orchestra.run.smithi052.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-05-27T07:11:52.046 INFO:teuthology.orchestra.run.smithi052.stdout:Invalid unit name "ceph-volume@*.service" escaped as "ceph-volume@\x2a.service". 2024-05-27T07:11:52.046 INFO:teuthology.orchestra.run.smithi052.stdout: 2024-05-27T07:11:52.108 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : ceph-test-2:19.0.0-3872.g38f9d900.el9.x86_64 143/170 2024-05-27T07:11:52.148 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : rbd-fuse-2:19.0.0-3872.g38f9d900.el9.x86_64 144/170 2024-05-27T07:11:52.189 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : rbd-nbd-2:19.0.0-3872.g38f9d900.el9.x86_64 145/170 2024-05-27T07:11:52.221 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : xfsprogs-devel-6.3.0-1.el9.x86_64 146/170 2024-05-27T07:11:52.248 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : elfutils-libelf-devel-0.191-4.el9.x86_64 147/170 2024-05-27T07:11:52.429 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : cephfs-shell-2:19.0.0-3872.g38f9d900.el9.x86_64 148/170 2024-05-27T07:11:52.470 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : postgresql-contrib-13.14-1.el9.x86_64 149/170 2024-05-27T07:11:52.502 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : libcephfs-devel-2:19.0.0-3872.g38f9d900.el9.x86_ 150/170 2024-05-27T07:11:52.543 INFO:teuthology.orchestra.run.smithi173.stdout: Upgrading : elfutils-debuginfod-client-0.191-4.el9.x86_64 151/170 2024-05-27T07:11:52.572 INFO:teuthology.orchestra.run.smithi173.stdout: Upgrading : NetworkManager-team-1:1.47.90-1.el9.x86_64 152/170 2024-05-27T07:11:52.610 INFO:teuthology.orchestra.run.smithi173.stdout: Upgrading : NetworkManager-tui-1:1.47.90-1.el9.x86_64 153/170 2024-05-27T07:11:53.166 INFO:teuthology.orchestra.run.smithi173.stdout: Running scriptlet: postgresql-server-13.14-1.el9.x86_64 154/170 2024-05-27T07:11:53.208 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : postgresql-server-13.14-1.el9.x86_64 154/170 2024-05-27T07:11:53.228 INFO:teuthology.orchestra.run.smithi173.stdout: Running scriptlet: postgresql-server-13.14-1.el9.x86_64 154/170 2024-05-27T07:11:53.680 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : ceph-osd-2:19.0.0-3872.g38f9d900.el9.x86_64 113/170 2024-05-27T07:11:53.708 INFO:teuthology.orchestra.run.smithi052.stdout: Running scriptlet: ceph-osd-2:19.0.0-3872.g38f9d900.el9.x86_64 113/170 2024-05-27T07:11:53.708 INFO:teuthology.orchestra.run.smithi052.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-05-27T07:11:53.708 INFO:teuthology.orchestra.run.smithi052.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2024-05-27T07:11:53.708 INFO:teuthology.orchestra.run.smithi052.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-05-27T07:11:53.709 INFO:teuthology.orchestra.run.smithi052.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-05-27T07:11:53.709 INFO:teuthology.orchestra.run.smithi052.stdout: 2024-05-27T07:11:53.947 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : ceph-mds-2:19.0.0-3872.g38f9d900.el9.x86_64 114/170 2024-05-27T07:11:53.972 INFO:teuthology.orchestra.run.smithi052.stdout: Running scriptlet: ceph-mds-2:19.0.0-3872.g38f9d900.el9.x86_64 114/170 2024-05-27T07:11:53.972 INFO:teuthology.orchestra.run.smithi052.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-05-27T07:11:53.972 INFO:teuthology.orchestra.run.smithi052.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2024-05-27T07:11:53.972 INFO:teuthology.orchestra.run.smithi052.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-05-27T07:11:53.972 INFO:teuthology.orchestra.run.smithi052.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-05-27T07:11:53.972 INFO:teuthology.orchestra.run.smithi052.stdout: 2024-05-27T07:11:54.069 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : cephfs-top-2:19.0.0-3872.g38f9d900.el9.noarch 155/170 2024-05-27T07:11:54.163 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : openssl-devel-1:3.2.1-1.el9.x86_64 156/170 2024-05-27T07:11:54.204 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : libaio-devel-0.3.111-13.el9.x86_64 157/170 2024-05-27T07:11:54.285 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : flex-2.6.4-9.el9.x86_64 158/170 2024-05-27T07:11:54.316 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : bison-3.7.4-5.el9.x86_64 159/170 2024-05-27T07:11:54.341 INFO:teuthology.orchestra.run.smithi173.stdout: Cleanup : NetworkManager-tui-1:1.47.4-1.el9.x86_64 160/170 2024-05-27T07:11:54.366 INFO:teuthology.orchestra.run.smithi173.stdout: Cleanup : elfutils-debuginfod-client-0.190-2.el9.x86_64 161/170 2024-05-27T07:11:54.394 INFO:teuthology.orchestra.run.smithi173.stdout: Cleanup : elfutils-libs-0.190-2.el9.x86_64 162/170 2024-05-27T07:11:54.394 INFO:teuthology.orchestra.run.smithi173.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 163/170 2024-05-27T07:11:54.456 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : ceph-mon-2:19.0.0-3872.g38f9d900.el9.x86_64 115/170 2024-05-27T07:11:54.481 INFO:teuthology.orchestra.run.smithi052.stdout: Running scriptlet: ceph-mon-2:19.0.0-3872.g38f9d900.el9.x86_64 115/170 2024-05-27T07:11:54.481 INFO:teuthology.orchestra.run.smithi052.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-05-27T07:11:54.481 INFO:teuthology.orchestra.run.smithi052.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2024-05-27T07:11:54.481 INFO:teuthology.orchestra.run.smithi052.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-05-27T07:11:54.481 INFO:teuthology.orchestra.run.smithi052.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-05-27T07:11:54.482 INFO:teuthology.orchestra.run.smithi052.stdout: 2024-05-27T07:11:54.485 INFO:teuthology.orchestra.run.smithi173.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 163/170 2024-05-27T07:11:54.527 INFO:teuthology.orchestra.run.smithi173.stdout: Cleanup : openssl-1:3.0.7-27.el9.x86_64 164/170 2024-05-27T07:11:54.527 INFO:teuthology.orchestra.run.smithi173.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 165/170 2024-05-27T07:11:54.611 INFO:teuthology.orchestra.run.smithi173.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 165/170 2024-05-27T07:11:54.636 INFO:teuthology.orchestra.run.smithi173.stdout: Cleanup : NetworkManager-team-1:1.47.4-1.el9.x86_64 166/170 2024-05-27T07:11:54.641 INFO:teuthology.orchestra.run.smithi173.stdout: Running scriptlet: NetworkManager-1:1.47.4-1.el9.x86_64 167/170 2024-05-27T07:11:54.641 INFO:teuthology.orchestra.run.smithi173.stdout: Cleanup : NetworkManager-1:1.47.4-1.el9.x86_64 167/170 2024-05-27T07:11:54.679 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : automake-1.16.2-8.el9.noarch 116/170 2024-05-27T07:11:54.746 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : python3-ply-3.11-14.el9.noarch 117/170 2024-05-27T07:11:54.810 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 118/170 2024-05-27T07:11:54.821 INFO:teuthology.orchestra.run.smithi173.stdout: Running scriptlet: NetworkManager-1:1.47.4-1.el9.x86_64 167/170 2024-05-27T07:11:54.821 INFO:teuthology.orchestra.run.smithi173.stdout: Cleanup : NetworkManager-libnm-1:1.47.4-1.el9.x86_64 168/170 2024-05-27T07:11:54.906 INFO:teuthology.orchestra.run.smithi173.stdout: Running scriptlet: NetworkManager-libnm-1:1.47.4-1.el9.x86_64 168/170 2024-05-27T07:11:54.939 INFO:teuthology.orchestra.run.smithi173.stdout: Cleanup : openssl-libs-1:3.0.7-27.el9.x86_64 169/170 2024-05-27T07:11:55.003 INFO:teuthology.orchestra.run.smithi173.stdout: Cleanup : elfutils-libelf-0.190-2.el9.x86_64 170/170 2024-05-27T07:11:55.007 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 119/170 2024-05-27T07:11:55.050 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 120/170 2024-05-27T07:11:55.132 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 121/170 2024-05-27T07:11:55.229 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 122/170 2024-05-27T07:11:55.390 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 123/170 2024-05-27T07:11:55.433 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : python3-asyncssh-2.13.2-5.el9.noarch 124/170 2024-05-27T07:11:55.466 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 125/170 2024-05-27T07:11:55.492 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : fuse-2.9.9-15.el9.x86_64 126/170 2024-05-27T07:11:55.890 INFO:teuthology.orchestra.run.smithi052.stdout: Running scriptlet: cephadm-2:19.0.0-3872.g38f9d900.el9.noarch 127/170 2024-05-27T07:11:55.916 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : cephadm-2:19.0.0-3872.g38f9d900.el9.noarch 127/170 2024-05-27T07:11:55.949 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : ceph-prometheus-alerts-2:19.0.0-3872.g38f9d900.e 128/170 2024-05-27T07:11:55.992 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : ceph-test-2:19.0.0-3872.g38f9d900.el9.x86_64 143/170 2024-05-27T07:11:56.013 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : ceph-grafana-dashboards-2:19.0.0-3872.g38f9d900. 129/170 2024-05-27T07:11:56.033 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : rbd-fuse-2:19.0.0-3872.g38f9d900.el9.x86_64 144/170 2024-05-27T07:11:56.039 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : ceph-mgr-cephadm-2:19.0.0-3872.g38f9d900.el9.noa 130/170 2024-05-27T07:11:56.074 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : rbd-nbd-2:19.0.0-3872.g38f9d900.el9.x86_64 145/170 2024-05-27T07:11:56.114 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : xfsprogs-devel-6.3.0-1.el9.x86_64 146/170 2024-05-27T07:11:56.141 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : elfutils-libelf-devel-0.191-4.el9.x86_64 147/170 2024-05-27T07:11:56.323 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : cephfs-shell-2:19.0.0-3872.g38f9d900.el9.x86_64 148/170 2024-05-27T07:11:56.371 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : postgresql-contrib-13.14-1.el9.x86_64 149/170 2024-05-27T07:11:56.403 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : libcephfs-devel-2:19.0.0-3872.g38f9d900.el9.x86_ 150/170 2024-05-27T07:11:56.435 INFO:teuthology.orchestra.run.smithi105.stdout: Upgrading : elfutils-debuginfod-client-0.191-4.el9.x86_64 151/170 2024-05-27T07:11:56.474 INFO:teuthology.orchestra.run.smithi105.stdout: Upgrading : NetworkManager-team-1:1.47.90-1.el9.x86_64 152/170 2024-05-27T07:11:56.519 INFO:teuthology.orchestra.run.smithi105.stdout: Upgrading : NetworkManager-tui-1:1.47.90-1.el9.x86_64 153/170 2024-05-27T07:11:57.065 INFO:teuthology.orchestra.run.smithi105.stdout: Running scriptlet: postgresql-server-13.14-1.el9.x86_64 154/170 2024-05-27T07:11:57.101 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : postgresql-server-13.14-1.el9.x86_64 154/170 2024-05-27T07:11:57.120 INFO:teuthology.orchestra.run.smithi105.stdout: Running scriptlet: postgresql-server-13.14-1.el9.x86_64 154/170 2024-05-27T07:11:57.144 INFO:teuthology.orchestra.run.smithi052.stdout: Running scriptlet: ceph-mgr-cephadm-2:19.0.0-3872.g38f9d900.el9.noa 130/170 2024-05-27T07:11:57.174 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : ceph-mgr-dashboard-2:19.0.0-3872.g38f9d900.el9.n 131/170 2024-05-27T07:11:57.945 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : cephfs-top-2:19.0.0-3872.g38f9d900.el9.noarch 155/170 2024-05-27T07:11:58.022 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : openssl-devel-1:3.2.1-1.el9.x86_64 156/170 2024-05-27T07:11:58.063 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : libaio-devel-0.3.111-13.el9.x86_64 157/170 2024-05-27T07:11:58.152 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : flex-2.6.4-9.el9.x86_64 158/170 2024-05-27T07:11:58.190 INFO:teuthology.orchestra.run.smithi052.stdout: Running scriptlet: ceph-mgr-dashboard-2:19.0.0-3872.g38f9d900.el9.n 131/170 2024-05-27T07:11:58.192 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : bison-3.7.4-5.el9.x86_64 159/170 2024-05-27T07:11:58.216 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : ceph-mgr-diskprediction-local-2:19.0.0-3872.g38f 132/170 2024-05-27T07:11:58.217 INFO:teuthology.orchestra.run.smithi105.stdout: Cleanup : NetworkManager-tui-1:1.47.4-1.el9.x86_64 160/170 2024-05-27T07:11:58.242 INFO:teuthology.orchestra.run.smithi105.stdout: Cleanup : elfutils-debuginfod-client-0.190-2.el9.x86_64 161/170 2024-05-27T07:11:58.269 INFO:teuthology.orchestra.run.smithi105.stdout: Cleanup : elfutils-libs-0.190-2.el9.x86_64 162/170 2024-05-27T07:11:58.269 INFO:teuthology.orchestra.run.smithi105.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 163/170 2024-05-27T07:11:58.337 INFO:teuthology.orchestra.run.smithi105.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 163/170 2024-05-27T07:11:58.337 INFO:teuthology.orchestra.run.smithi052.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:19.0.0-3872.g38f 132/170 2024-05-27T07:11:58.361 INFO:teuthology.orchestra.run.smithi105.stdout: Cleanup : openssl-1:3.0.7-27.el9.x86_64 164/170 2024-05-27T07:11:58.361 INFO:teuthology.orchestra.run.smithi105.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 165/170 2024-05-27T07:11:58.446 INFO:teuthology.orchestra.run.smithi105.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 165/170 2024-05-27T07:11:58.480 INFO:teuthology.orchestra.run.smithi105.stdout: Cleanup : NetworkManager-team-1:1.47.4-1.el9.x86_64 166/170 2024-05-27T07:11:58.484 INFO:teuthology.orchestra.run.smithi105.stdout: Running scriptlet: NetworkManager-1:1.47.4-1.el9.x86_64 167/170 2024-05-27T07:11:58.484 INFO:teuthology.orchestra.run.smithi105.stdout: Cleanup : NetworkManager-1:1.47.4-1.el9.x86_64 167/170 2024-05-27T07:11:58.515 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : ceph-mgr-modules-core-2:19.0.0-3872.g38f9d900.el 133/170 2024-05-27T07:11:58.543 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : ceph-mgr-2:19.0.0-3872.g38f9d900.el9.x86_64 134/170 2024-05-27T07:11:58.568 INFO:teuthology.orchestra.run.smithi052.stdout: Running scriptlet: ceph-mgr-2:19.0.0-3872.g38f9d900.el9.x86_64 134/170 2024-05-27T07:11:58.568 INFO:teuthology.orchestra.run.smithi052.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-05-27T07:11:58.568 INFO:teuthology.orchestra.run.smithi052.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2024-05-27T07:11:58.568 INFO:teuthology.orchestra.run.smithi052.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-05-27T07:11:58.568 INFO:teuthology.orchestra.run.smithi052.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-05-27T07:11:58.568 INFO:teuthology.orchestra.run.smithi052.stdout: 2024-05-27T07:11:58.619 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : ceph-mgr-rook-2:19.0.0-3872.g38f9d900.el9.noarch 135/170 2024-05-27T07:11:58.630 INFO:teuthology.orchestra.run.smithi052.stdout: Running scriptlet: ceph-mgr-rook-2:19.0.0-3872.g38f9d900.el9.noarch 135/170 2024-05-27T07:11:58.665 INFO:teuthology.orchestra.run.smithi105.stdout: Running scriptlet: NetworkManager-1:1.47.4-1.el9.x86_64 167/170 2024-05-27T07:11:58.665 INFO:teuthology.orchestra.run.smithi105.stdout: Cleanup : NetworkManager-libnm-1:1.47.4-1.el9.x86_64 168/170 2024-05-27T07:11:58.733 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : ceph-2:19.0.0-3872.g38f9d900.el9.x86_64 136/170 2024-05-27T07:11:58.748 INFO:teuthology.orchestra.run.smithi105.stdout: Running scriptlet: NetworkManager-libnm-1:1.47.4-1.el9.x86_64 168/170 2024-05-27T07:11:58.774 INFO:teuthology.orchestra.run.smithi105.stdout: Cleanup : openssl-libs-1:3.0.7-27.el9.x86_64 169/170 2024-05-27T07:11:58.789 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : ceph-fuse-2:19.0.0-3872.g38f9d900.el9.x86_64 137/170 2024-05-27T07:11:58.837 INFO:teuthology.orchestra.run.smithi105.stdout: Cleanup : elfutils-libelf-0.190-2.el9.x86_64 170/170 2024-05-27T07:11:59.797 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : libtool-2.4.6-45.el9.x86_64 138/170 2024-05-27T07:11:59.823 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : ceph-radosgw-2:19.0.0-3872.g38f9d900.el9.x86_64 139/170 2024-05-27T07:11:59.848 INFO:teuthology.orchestra.run.smithi052.stdout: Running scriptlet: ceph-radosgw-2:19.0.0-3872.g38f9d900.el9.x86_64 139/170 2024-05-27T07:11:59.848 INFO:teuthology.orchestra.run.smithi052.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-05-27T07:11:59.848 INFO:teuthology.orchestra.run.smithi052.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2024-05-27T07:11:59.848 INFO:teuthology.orchestra.run.smithi052.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-05-27T07:11:59.849 INFO:teuthology.orchestra.run.smithi052.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-05-27T07:11:59.849 INFO:teuthology.orchestra.run.smithi052.stdout: 2024-05-27T07:11:59.889 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : ceph-immutable-object-cache-2:19.0.0-3872.g38f9d 140/170 2024-05-27T07:11:59.913 INFO:teuthology.orchestra.run.smithi052.stdout: Running scriptlet: ceph-immutable-object-cache-2:19.0.0-3872.g38f9d 140/170 2024-05-27T07:11:59.913 INFO:teuthology.orchestra.run.smithi052.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-05-27T07:11:59.913 INFO:teuthology.orchestra.run.smithi052.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2024-05-27T07:11:59.913 INFO:teuthology.orchestra.run.smithi052.stdout: 2024-05-27T07:11:59.972 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : cephfs-mirror-2:19.0.0-3872.g38f9d900.el9.x86_64 141/170 2024-05-27T07:11:59.997 INFO:teuthology.orchestra.run.smithi052.stdout: Running scriptlet: cephfs-mirror-2:19.0.0-3872.g38f9d900.el9.x86_64 141/170 2024-05-27T07:11:59.997 INFO:teuthology.orchestra.run.smithi052.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-05-27T07:11:59.997 INFO:teuthology.orchestra.run.smithi052.stdout:Invalid unit name "cephfs-mirror@*.service" escaped as "cephfs-mirror@\x2a.service". 2024-05-27T07:11:59.997 INFO:teuthology.orchestra.run.smithi052.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/cephfs-mirror.target → /usr/lib/systemd/system/cephfs-mirror.target. 2024-05-27T07:11:59.997 INFO:teuthology.orchestra.run.smithi052.stdout:Created symlink /etc/systemd/system/ceph.target.wants/cephfs-mirror.target → /usr/lib/systemd/system/cephfs-mirror.target. 2024-05-27T07:11:59.997 INFO:teuthology.orchestra.run.smithi052.stdout: 2024-05-27T07:12:00.323 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : rbd-mirror-2:19.0.0-3872.g38f9d900.el9.x86_64 142/170 2024-05-27T07:12:00.348 INFO:teuthology.orchestra.run.smithi052.stdout: Running scriptlet: rbd-mirror-2:19.0.0-3872.g38f9d900.el9.x86_64 142/170 2024-05-27T07:12:00.348 INFO:teuthology.orchestra.run.smithi052.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-05-27T07:12:00.348 INFO:teuthology.orchestra.run.smithi052.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2024-05-27T07:12:00.348 INFO:teuthology.orchestra.run.smithi052.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-05-27T07:12:00.348 INFO:teuthology.orchestra.run.smithi052.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-05-27T07:12:00.348 INFO:teuthology.orchestra.run.smithi052.stdout: 2024-05-27T07:12:00.704 INFO:teuthology.orchestra.run.smithi173.stdout: Running scriptlet: elfutils-libelf-0.190-2.el9.x86_64 170/170 2024-05-27T07:12:00.704 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : ceph-2:19.0.0-3872.g38f9d900.el9.x86_64 1/170 2024-05-27T07:12:00.704 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : ceph-base-2:19.0.0-3872.g38f9d900.el9.x86_64 2/170 2024-05-27T07:12:00.704 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : ceph-common-2:19.0.0-3872.g38f9d900.el9.x86_64 3/170 2024-05-27T07:12:00.704 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : ceph-fuse-2:19.0.0-3872.g38f9d900.el9.x86_64 4/170 2024-05-27T07:12:00.704 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : ceph-immutable-object-cache-2:19.0.0-3872.g38f9d 5/170 2024-05-27T07:12:00.705 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : ceph-mds-2:19.0.0-3872.g38f9d900.el9.x86_64 6/170 2024-05-27T07:12:00.705 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : ceph-mgr-2:19.0.0-3872.g38f9d900.el9.x86_64 7/170 2024-05-27T07:12:00.705 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : ceph-mon-2:19.0.0-3872.g38f9d900.el9.x86_64 8/170 2024-05-27T07:12:00.705 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : ceph-osd-2:19.0.0-3872.g38f9d900.el9.x86_64 9/170 2024-05-27T07:12:00.705 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : ceph-radosgw-2:19.0.0-3872.g38f9d900.el9.x86_64 10/170 2024-05-27T07:12:00.705 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : ceph-selinux-2:19.0.0-3872.g38f9d900.el9.x86_64 11/170 2024-05-27T07:12:00.705 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : ceph-test-2:19.0.0-3872.g38f9d900.el9.x86_64 12/170 2024-05-27T07:12:00.705 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : cephfs-mirror-2:19.0.0-3872.g38f9d900.el9.x86_64 13/170 2024-05-27T07:12:00.705 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : cephfs-shell-2:19.0.0-3872.g38f9d900.el9.x86_64 14/170 2024-05-27T07:12:00.705 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : libcephfs-devel-2:19.0.0-3872.g38f9d900.el9.x86_ 15/170 2024-05-27T07:12:00.705 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : libcephfs2-2:19.0.0-3872.g38f9d900.el9.x86_64 16/170 2024-05-27T07:12:00.705 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : libcephsqlite-2:19.0.0-3872.g38f9d900.el9.x86_64 17/170 2024-05-27T07:12:00.705 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : librados-devel-2:19.0.0-3872.g38f9d900.el9.x86_6 18/170 2024-05-27T07:12:00.706 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : libradosstriper1-2:19.0.0-3872.g38f9d900.el9.x86 19/170 2024-05-27T07:12:00.706 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : librgw2-2:19.0.0-3872.g38f9d900.el9.x86_64 20/170 2024-05-27T07:12:00.706 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : python3-ceph-argparse-2:19.0.0-3872.g38f9d900.el 21/170 2024-05-27T07:12:00.706 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : python3-ceph-common-2:19.0.0-3872.g38f9d900.el9. 22/170 2024-05-27T07:12:00.706 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : python3-cephfs-2:19.0.0-3872.g38f9d900.el9.x86_6 23/170 2024-05-27T07:12:00.706 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : python3-rados-2:19.0.0-3872.g38f9d900.el9.x86_64 24/170 2024-05-27T07:12:00.706 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : python3-rbd-2:19.0.0-3872.g38f9d900.el9.x86_64 25/170 2024-05-27T07:12:00.706 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : python3-rgw-2:19.0.0-3872.g38f9d900.el9.x86_64 26/170 2024-05-27T07:12:00.706 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : rbd-fuse-2:19.0.0-3872.g38f9d900.el9.x86_64 27/170 2024-05-27T07:12:00.706 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : rbd-mirror-2:19.0.0-3872.g38f9d900.el9.x86_64 28/170 2024-05-27T07:12:00.706 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : rbd-nbd-2:19.0.0-3872.g38f9d900.el9.x86_64 29/170 2024-05-27T07:12:00.706 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : ceph-grafana-dashboards-2:19.0.0-3872.g38f9d900. 30/170 2024-05-27T07:12:00.718 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : ceph-mgr-cephadm-2:19.0.0-3872.g38f9d900.el9.noa 31/170 2024-05-27T07:12:00.718 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : ceph-mgr-dashboard-2:19.0.0-3872.g38f9d900.el9.n 32/170 2024-05-27T07:12:00.718 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : ceph-mgr-diskprediction-local-2:19.0.0-3872.g38f 33/170 2024-05-27T07:12:00.718 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : ceph-mgr-modules-core-2:19.0.0-3872.g38f9d900.el 34/170 2024-05-27T07:12:00.718 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : ceph-mgr-rook-2:19.0.0-3872.g38f9d900.el9.noarch 35/170 2024-05-27T07:12:00.719 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : ceph-prometheus-alerts-2:19.0.0-3872.g38f9d900.e 36/170 2024-05-27T07:12:00.719 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : ceph-volume-2:19.0.0-3872.g38f9d900.el9.noarch 37/170 2024-05-27T07:12:00.719 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : cephadm-2:19.0.0-3872.g38f9d900.el9.noarch 38/170 2024-05-27T07:12:00.719 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : cephfs-top-2:19.0.0-3872.g38f9d900.el9.noarch 39/170 2024-05-27T07:12:00.719 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : fuse-2.9.9-15.el9.x86_64 40/170 2024-05-27T07:12:00.719 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : libgfortran-11.4.1-3.el9.x86_64 41/170 2024-05-27T07:12:00.719 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : libquadmath-11.4.1-3.el9.x86_64 42/170 2024-05-27T07:12:00.719 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 43/170 2024-05-27T07:12:00.719 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 44/170 2024-05-27T07:12:00.719 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : python3-ply-3.11-14.el9.noarch 45/170 2024-05-27T07:12:00.719 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 46/170 2024-05-27T07:12:00.719 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : python3-requests-2.25.1-8.el9.noarch 47/170 2024-05-27T07:12:00.720 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : python3-urllib3-1.26.5-5.el9.noarch 48/170 2024-05-27T07:12:00.720 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : automake-1.16.2-8.el9.noarch 49/170 2024-05-27T07:12:00.720 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : bison-3.7.4-5.el9.x86_64 50/170 2024-05-27T07:12:00.720 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : boost-program-options-1.75.0-8.el9.x86_64 51/170 2024-05-27T07:12:00.720 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : elfutils-libelf-devel-0.191-4.el9.x86_64 52/170 2024-05-27T07:12:00.720 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : flex-2.6.4-9.el9.x86_64 53/170 2024-05-27T07:12:00.720 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 54/170 2024-05-27T07:12:00.720 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 55/170 2024-05-27T07:12:00.720 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 56/170 2024-05-27T07:12:00.720 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : libaio-devel-0.3.111-13.el9.x86_64 57/170 2024-05-27T07:12:00.720 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 58/170 2024-05-27T07:12:00.720 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 59/170 2024-05-27T07:12:00.721 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 60/170 2024-05-27T07:12:00.721 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : libtool-2.4.6-45.el9.x86_64 61/170 2024-05-27T07:12:00.721 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : libuuid-devel-2.37.4-18.el9.x86_64 62/170 2024-05-27T07:12:00.721 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : libzstd-devel-1.5.1-2.el9.x86_64 63/170 2024-05-27T07:12:00.721 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 64/170 2024-05-27T07:12:00.721 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : lua-5.4.4-4.el9.x86_64 65/170 2024-05-27T07:12:00.721 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : openblas-0.3.21-2.el9.x86_64 66/170 2024-05-27T07:12:00.721 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : openblas-openmp-0.3.21-2.el9.x86_64 67/170 2024-05-27T07:12:00.721 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : openssl-devel-1:3.2.1-1.el9.x86_64 68/170 2024-05-27T07:12:00.721 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : postgresql-13.14-1.el9.x86_64 69/170 2024-05-27T07:12:00.721 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : postgresql-contrib-13.14-1.el9.x86_64 70/170 2024-05-27T07:12:00.721 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : postgresql-private-libs-13.14-1.el9.x86_64 71/170 2024-05-27T07:12:00.722 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : postgresql-server-13.14-1.el9.x86_64 72/170 2024-05-27T07:12:00.722 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : protobuf-3.14.0-13.el9.x86_64 73/170 2024-05-27T07:12:00.722 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : python3-attrs-20.3.0-7.el9.noarch 74/170 2024-05-27T07:12:00.722 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 75/170 2024-05-27T07:12:00.722 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : python3-colorama-0.4.6-3.el9.noarch 76/170 2024-05-27T07:12:00.722 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : python3-devel-3.9.18-3.el9.x86_64 77/170 2024-05-27T07:12:00.722 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : python3-jinja2-2.11.3-6.el9.noarch 78/170 2024-05-27T07:12:00.722 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 79/170 2024-05-27T07:12:00.722 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 80/170 2024-05-27T07:12:00.722 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 81/170 2024-05-27T07:12:00.722 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 82/170 2024-05-27T07:12:00.722 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 83/170 2024-05-27T07:12:00.723 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 84/170 2024-05-27T07:12:00.723 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 85/170 2024-05-27T07:12:00.723 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : python3-packaging-20.9-5.el9.noarch 86/170 2024-05-27T07:12:00.723 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 87/170 2024-05-27T07:12:00.723 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : python3-protobuf-3.14.0-13.el9.noarch 88/170 2024-05-27T07:12:00.723 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 89/170 2024-05-27T07:12:00.723 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 90/170 2024-05-27T07:12:00.723 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 91/170 2024-05-27T07:12:00.723 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 92/170 2024-05-27T07:12:00.723 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 93/170 2024-05-27T07:12:00.723 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 94/170 2024-05-27T07:12:00.723 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : python3-wcwidth-0.2.5-8.el9.noarch 95/170 2024-05-27T07:12:00.723 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : qatlib-24.02.0-1.el9.x86_64 96/170 2024-05-27T07:12:00.724 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : qatlib-service-24.02.0-1.el9.x86_64 97/170 2024-05-27T07:12:00.724 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : qatzip-libs-1.2.0-1.el9.x86_64 98/170 2024-05-27T07:12:00.724 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : socat-1.7.4.1-5.el9.x86_64 99/170 2024-05-27T07:12:00.724 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : uuid-1.6.2-55.el9.x86_64 100/170 2024-05-27T07:12:00.724 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : xfsprogs-devel-6.3.0-1.el9.x86_64 101/170 2024-05-27T07:12:00.724 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 102/170 2024-05-27T07:12:00.724 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : zlib-devel-1.2.11-41.el9.x86_64 103/170 2024-05-27T07:12:00.724 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : lua-devel-5.4.4-4.el9.x86_64 104/170 2024-05-27T07:12:00.724 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : protobuf-compiler-3.14.0-13.el9.x86_64 105/170 2024-05-27T07:12:00.724 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : abseil-cpp-20211102.0-3.el9.x86_64 106/170 2024-05-27T07:12:00.724 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : gperftools-libs-2.9.1-2.el9.x86_64 107/170 2024-05-27T07:12:00.724 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : grpc-data-1.46.7-10.el9.noarch 108/170 2024-05-27T07:12:00.724 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : libarrow-9.0.0-11.el9.x86_64 109/170 2024-05-27T07:12:00.724 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : libarrow-doc-9.0.0-11.el9.noarch 110/170 2024-05-27T07:12:00.725 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 111/170 2024-05-27T07:12:00.725 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 112/170 2024-05-27T07:12:00.725 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : luarocks-3.9.2-5.el9.noarch 113/170 2024-05-27T07:12:00.725 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : parquet-libs-9.0.0-11.el9.x86_64 114/170 2024-05-27T07:12:00.725 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : python3-asyncssh-2.13.2-5.el9.noarch 115/170 2024-05-27T07:12:00.725 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 116/170 2024-05-27T07:12:00.725 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 117/170 2024-05-27T07:12:00.725 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : python3-certifi-2021.10.8-2.el9.noarch 118/170 2024-05-27T07:12:00.725 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 119/170 2024-05-27T07:12:00.725 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 120/170 2024-05-27T07:12:00.725 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : python3-cmd2-2.4.2-2.el9.noarch 121/170 2024-05-27T07:12:00.726 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : python3-google-auth-1:2.29.0-1.el9.noarch 122/170 2024-05-27T07:12:00.726 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : python3-grpcio-1.46.7-10.el9.x86_64 123/170 2024-05-27T07:12:00.726 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : python3-grpcio-tools-1.46.7-10.el9.x86_64 124/170 2024-05-27T07:12:00.726 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 125/170 2024-05-27T07:12:00.726 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 126/170 2024-05-27T07:12:00.726 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 127/170 2024-05-27T07:12:00.726 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 128/170 2024-05-27T07:12:00.726 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 129/170 2024-05-27T07:12:00.726 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 130/170 2024-05-27T07:12:00.726 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 131/170 2024-05-27T07:12:00.726 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 132/170 2024-05-27T07:12:00.726 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 133/170 2024-05-27T07:12:00.727 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 134/170 2024-05-27T07:12:00.727 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 135/170 2024-05-27T07:12:00.727 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 136/170 2024-05-27T07:12:00.727 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : python3-pyperclip-1.8.2-2.el9.noarch 137/170 2024-05-27T07:12:00.727 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 138/170 2024-05-27T07:12:00.727 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 139/170 2024-05-27T07:12:00.727 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 140/170 2024-05-27T07:12:00.727 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 141/170 2024-05-27T07:12:00.727 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : python3-typing-extensions-4.4.0-2.el9.noarch 142/170 2024-05-27T07:12:00.727 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : python3-webob-1.8.7-6.el9.noarch 143/170 2024-05-27T07:12:00.727 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 144/170 2024-05-27T07:12:00.727 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 145/170 2024-05-27T07:12:00.727 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 146/170 2024-05-27T07:12:00.728 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : re2-1:20211101-3.el9.x86_64 147/170 2024-05-27T07:12:00.728 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 148/170 2024-05-27T07:12:00.728 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : librados2-2:19.0.0-3872.g38f9d900.el9.x86_64 149/170 2024-05-27T07:12:00.728 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 150/170 2024-05-27T07:12:00.728 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : librbd1-2:19.0.0-3872.g38f9d900.el9.x86_64 151/170 2024-05-27T07:12:00.728 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 152/170 2024-05-27T07:12:00.728 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : NetworkManager-1:1.47.90-1.el9.x86_64 153/170 2024-05-27T07:12:00.728 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : NetworkManager-1:1.47.4-1.el9.x86_64 154/170 2024-05-27T07:12:00.728 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : NetworkManager-libnm-1:1.47.90-1.el9.x86_64 155/170 2024-05-27T07:12:00.728 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : NetworkManager-libnm-1:1.47.4-1.el9.x86_64 156/170 2024-05-27T07:12:00.728 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : NetworkManager-team-1:1.47.90-1.el9.x86_64 157/170 2024-05-27T07:12:00.728 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : NetworkManager-team-1:1.47.4-1.el9.x86_64 158/170 2024-05-27T07:12:00.728 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : NetworkManager-tui-1:1.47.90-1.el9.x86_64 159/170 2024-05-27T07:12:00.728 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : NetworkManager-tui-1:1.47.4-1.el9.x86_64 160/170 2024-05-27T07:12:00.729 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : elfutils-debuginfod-client-0.191-4.el9.x86_64 161/170 2024-05-27T07:12:00.729 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : elfutils-debuginfod-client-0.190-2.el9.x86_64 162/170 2024-05-27T07:12:00.729 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : elfutils-libelf-0.191-4.el9.x86_64 163/170 2024-05-27T07:12:00.729 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : elfutils-libelf-0.190-2.el9.x86_64 164/170 2024-05-27T07:12:00.729 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : elfutils-libs-0.191-4.el9.x86_64 165/170 2024-05-27T07:12:00.729 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : elfutils-libs-0.190-2.el9.x86_64 166/170 2024-05-27T07:12:00.729 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : openssl-1:3.2.1-1.el9.x86_64 167/170 2024-05-27T07:12:00.729 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : openssl-1:3.0.7-27.el9.x86_64 168/170 2024-05-27T07:12:00.729 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : openssl-libs-1:3.2.1-1.el9.x86_64 169/170 2024-05-27T07:12:03.444 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : openssl-libs-1:3.0.7-27.el9.x86_64 170/170 2024-05-27T07:12:03.444 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-05-27T07:12:03.444 INFO:teuthology.orchestra.run.smithi173.stdout:Upgraded: 2024-05-27T07:12:03.444 INFO:teuthology.orchestra.run.smithi173.stdout: NetworkManager-1:1.47.90-1.el9.x86_64 2024-05-27T07:12:03.444 INFO:teuthology.orchestra.run.smithi173.stdout: NetworkManager-libnm-1:1.47.90-1.el9.x86_64 2024-05-27T07:12:03.445 INFO:teuthology.orchestra.run.smithi173.stdout: NetworkManager-team-1:1.47.90-1.el9.x86_64 2024-05-27T07:12:03.445 INFO:teuthology.orchestra.run.smithi173.stdout: NetworkManager-tui-1:1.47.90-1.el9.x86_64 2024-05-27T07:12:03.445 INFO:teuthology.orchestra.run.smithi173.stdout: elfutils-debuginfod-client-0.191-4.el9.x86_64 2024-05-27T07:12:03.445 INFO:teuthology.orchestra.run.smithi173.stdout: elfutils-libelf-0.191-4.el9.x86_64 2024-05-27T07:12:03.445 INFO:teuthology.orchestra.run.smithi173.stdout: elfutils-libs-0.191-4.el9.x86_64 2024-05-27T07:12:03.445 INFO:teuthology.orchestra.run.smithi173.stdout: librados2-2:19.0.0-3872.g38f9d900.el9.x86_64 2024-05-27T07:12:03.445 INFO:teuthology.orchestra.run.smithi173.stdout: librbd1-2:19.0.0-3872.g38f9d900.el9.x86_64 2024-05-27T07:12:03.445 INFO:teuthology.orchestra.run.smithi173.stdout: openssl-1:3.2.1-1.el9.x86_64 2024-05-27T07:12:03.445 INFO:teuthology.orchestra.run.smithi173.stdout: openssl-libs-1:3.2.1-1.el9.x86_64 2024-05-27T07:12:03.445 INFO:teuthology.orchestra.run.smithi173.stdout:Installed: 2024-05-27T07:12:03.445 INFO:teuthology.orchestra.run.smithi173.stdout: abseil-cpp-20211102.0-3.el9.x86_64 2024-05-27T07:12:03.445 INFO:teuthology.orchestra.run.smithi173.stdout: automake-1.16.2-8.el9.noarch 2024-05-27T07:12:03.445 INFO:teuthology.orchestra.run.smithi173.stdout: bison-3.7.4-5.el9.x86_64 2024-05-27T07:12:03.445 INFO:teuthology.orchestra.run.smithi173.stdout: boost-program-options-1.75.0-8.el9.x86_64 2024-05-27T07:12:03.446 INFO:teuthology.orchestra.run.smithi173.stdout: ceph-2:19.0.0-3872.g38f9d900.el9.x86_64 2024-05-27T07:12:03.446 INFO:teuthology.orchestra.run.smithi173.stdout: ceph-base-2:19.0.0-3872.g38f9d900.el9.x86_64 2024-05-27T07:12:03.446 INFO:teuthology.orchestra.run.smithi173.stdout: ceph-common-2:19.0.0-3872.g38f9d900.el9.x86_64 2024-05-27T07:12:03.446 INFO:teuthology.orchestra.run.smithi173.stdout: ceph-fuse-2:19.0.0-3872.g38f9d900.el9.x86_64 2024-05-27T07:12:03.446 INFO:teuthology.orchestra.run.smithi173.stdout: ceph-grafana-dashboards-2:19.0.0-3872.g38f9d900.el9.noarch 2024-05-27T07:12:03.446 INFO:teuthology.orchestra.run.smithi173.stdout: ceph-immutable-object-cache-2:19.0.0-3872.g38f9d900.el9.x86_64 2024-05-27T07:12:03.446 INFO:teuthology.orchestra.run.smithi173.stdout: ceph-mds-2:19.0.0-3872.g38f9d900.el9.x86_64 2024-05-27T07:12:03.446 INFO:teuthology.orchestra.run.smithi173.stdout: ceph-mgr-2:19.0.0-3872.g38f9d900.el9.x86_64 2024-05-27T07:12:03.446 INFO:teuthology.orchestra.run.smithi173.stdout: ceph-mgr-cephadm-2:19.0.0-3872.g38f9d900.el9.noarch 2024-05-27T07:12:03.446 INFO:teuthology.orchestra.run.smithi173.stdout: ceph-mgr-dashboard-2:19.0.0-3872.g38f9d900.el9.noarch 2024-05-27T07:12:03.446 INFO:teuthology.orchestra.run.smithi173.stdout: ceph-mgr-diskprediction-local-2:19.0.0-3872.g38f9d900.el9.noarch 2024-05-27T07:12:03.446 INFO:teuthology.orchestra.run.smithi173.stdout: ceph-mgr-modules-core-2:19.0.0-3872.g38f9d900.el9.noarch 2024-05-27T07:12:03.447 INFO:teuthology.orchestra.run.smithi173.stdout: ceph-mgr-rook-2:19.0.0-3872.g38f9d900.el9.noarch 2024-05-27T07:12:03.447 INFO:teuthology.orchestra.run.smithi173.stdout: ceph-mon-2:19.0.0-3872.g38f9d900.el9.x86_64 2024-05-27T07:12:03.447 INFO:teuthology.orchestra.run.smithi173.stdout: ceph-osd-2:19.0.0-3872.g38f9d900.el9.x86_64 2024-05-27T07:12:03.447 INFO:teuthology.orchestra.run.smithi173.stdout: ceph-prometheus-alerts-2:19.0.0-3872.g38f9d900.el9.noarch 2024-05-27T07:12:03.447 INFO:teuthology.orchestra.run.smithi173.stdout: ceph-radosgw-2:19.0.0-3872.g38f9d900.el9.x86_64 2024-05-27T07:12:03.447 INFO:teuthology.orchestra.run.smithi173.stdout: ceph-selinux-2:19.0.0-3872.g38f9d900.el9.x86_64 2024-05-27T07:12:03.447 INFO:teuthology.orchestra.run.smithi173.stdout: ceph-test-2:19.0.0-3872.g38f9d900.el9.x86_64 2024-05-27T07:12:03.447 INFO:teuthology.orchestra.run.smithi173.stdout: ceph-volume-2:19.0.0-3872.g38f9d900.el9.noarch 2024-05-27T07:12:03.447 INFO:teuthology.orchestra.run.smithi173.stdout: cephadm-2:19.0.0-3872.g38f9d900.el9.noarch 2024-05-27T07:12:03.447 INFO:teuthology.orchestra.run.smithi173.stdout: cephfs-mirror-2:19.0.0-3872.g38f9d900.el9.x86_64 2024-05-27T07:12:03.447 INFO:teuthology.orchestra.run.smithi173.stdout: cephfs-shell-2:19.0.0-3872.g38f9d900.el9.x86_64 2024-05-27T07:12:03.447 INFO:teuthology.orchestra.run.smithi173.stdout: cephfs-top-2:19.0.0-3872.g38f9d900.el9.noarch 2024-05-27T07:12:03.447 INFO:teuthology.orchestra.run.smithi173.stdout: elfutils-libelf-devel-0.191-4.el9.x86_64 2024-05-27T07:12:03.448 INFO:teuthology.orchestra.run.smithi173.stdout: flex-2.6.4-9.el9.x86_64 2024-05-27T07:12:03.448 INFO:teuthology.orchestra.run.smithi173.stdout: flexiblas-3.0.4-8.el9.x86_64 2024-05-27T07:12:03.448 INFO:teuthology.orchestra.run.smithi173.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2024-05-27T07:12:03.448 INFO:teuthology.orchestra.run.smithi173.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2024-05-27T07:12:03.448 INFO:teuthology.orchestra.run.smithi173.stdout: fuse-2.9.9-15.el9.x86_64 2024-05-27T07:12:03.448 INFO:teuthology.orchestra.run.smithi173.stdout: gperftools-libs-2.9.1-2.el9.x86_64 2024-05-27T07:12:03.448 INFO:teuthology.orchestra.run.smithi173.stdout: grpc-data-1.46.7-10.el9.noarch 2024-05-27T07:12:03.448 INFO:teuthology.orchestra.run.smithi173.stdout: libaio-devel-0.3.111-13.el9.x86_64 2024-05-27T07:12:03.448 INFO:teuthology.orchestra.run.smithi173.stdout: libarrow-9.0.0-11.el9.x86_64 2024-05-27T07:12:03.448 INFO:teuthology.orchestra.run.smithi173.stdout: libarrow-doc-9.0.0-11.el9.noarch 2024-05-27T07:12:03.448 INFO:teuthology.orchestra.run.smithi173.stdout: libcephfs-devel-2:19.0.0-3872.g38f9d900.el9.x86_64 2024-05-27T07:12:03.448 INFO:teuthology.orchestra.run.smithi173.stdout: libcephfs2-2:19.0.0-3872.g38f9d900.el9.x86_64 2024-05-27T07:12:03.449 INFO:teuthology.orchestra.run.smithi173.stdout: libcephsqlite-2:19.0.0-3872.g38f9d900.el9.x86_64 2024-05-27T07:12:03.449 INFO:teuthology.orchestra.run.smithi173.stdout: libgfortran-11.4.1-3.el9.x86_64 2024-05-27T07:12:03.449 INFO:teuthology.orchestra.run.smithi173.stdout: liboath-2.6.7-2.el9.x86_64 2024-05-27T07:12:03.449 INFO:teuthology.orchestra.run.smithi173.stdout: libpmemobj-1.12.1-1.el9.x86_64 2024-05-27T07:12:03.449 INFO:teuthology.orchestra.run.smithi173.stdout: libquadmath-11.4.1-3.el9.x86_64 2024-05-27T07:12:03.449 INFO:teuthology.orchestra.run.smithi173.stdout: librabbitmq-0.11.0-7.el9.x86_64 2024-05-27T07:12:03.449 INFO:teuthology.orchestra.run.smithi173.stdout: librados-devel-2:19.0.0-3872.g38f9d900.el9.x86_64 2024-05-27T07:12:03.449 INFO:teuthology.orchestra.run.smithi173.stdout: libradosstriper1-2:19.0.0-3872.g38f9d900.el9.x86_64 2024-05-27T07:12:03.449 INFO:teuthology.orchestra.run.smithi173.stdout: librdkafka-1.6.1-102.el9.x86_64 2024-05-27T07:12:03.449 INFO:teuthology.orchestra.run.smithi173.stdout: librgw2-2:19.0.0-3872.g38f9d900.el9.x86_64 2024-05-27T07:12:03.449 INFO:teuthology.orchestra.run.smithi173.stdout: libtool-2.4.6-45.el9.x86_64 2024-05-27T07:12:03.449 INFO:teuthology.orchestra.run.smithi173.stdout: libunwind-1.6.2-1.el9.x86_64 2024-05-27T07:12:03.449 INFO:teuthology.orchestra.run.smithi173.stdout: libuuid-devel-2.37.4-18.el9.x86_64 2024-05-27T07:12:03.450 INFO:teuthology.orchestra.run.smithi173.stdout: libzstd-devel-1.5.1-2.el9.x86_64 2024-05-27T07:12:03.450 INFO:teuthology.orchestra.run.smithi173.stdout: lttng-ust-2.12.0-6.el9.x86_64 2024-05-27T07:12:03.450 INFO:teuthology.orchestra.run.smithi173.stdout: lua-5.4.4-4.el9.x86_64 2024-05-27T07:12:03.450 INFO:teuthology.orchestra.run.smithi173.stdout: lua-devel-5.4.4-4.el9.x86_64 2024-05-27T07:12:03.450 INFO:teuthology.orchestra.run.smithi173.stdout: luarocks-3.9.2-5.el9.noarch 2024-05-27T07:12:03.450 INFO:teuthology.orchestra.run.smithi173.stdout: openblas-0.3.21-2.el9.x86_64 2024-05-27T07:12:03.450 INFO:teuthology.orchestra.run.smithi173.stdout: openblas-openmp-0.3.21-2.el9.x86_64 2024-05-27T07:12:03.450 INFO:teuthology.orchestra.run.smithi173.stdout: openssl-devel-1:3.2.1-1.el9.x86_64 2024-05-27T07:12:03.450 INFO:teuthology.orchestra.run.smithi173.stdout: parquet-libs-9.0.0-11.el9.x86_64 2024-05-27T07:12:03.451 INFO:teuthology.orchestra.run.smithi173.stdout: postgresql-13.14-1.el9.x86_64 2024-05-27T07:12:03.451 INFO:teuthology.orchestra.run.smithi173.stdout: postgresql-contrib-13.14-1.el9.x86_64 2024-05-27T07:12:03.451 INFO:teuthology.orchestra.run.smithi173.stdout: postgresql-private-libs-13.14-1.el9.x86_64 2024-05-27T07:12:03.451 INFO:teuthology.orchestra.run.smithi173.stdout: postgresql-server-13.14-1.el9.x86_64 2024-05-27T07:12:03.451 INFO:teuthology.orchestra.run.smithi173.stdout: protobuf-3.14.0-13.el9.x86_64 2024-05-27T07:12:03.451 INFO:teuthology.orchestra.run.smithi173.stdout: protobuf-compiler-3.14.0-13.el9.x86_64 2024-05-27T07:12:03.451 INFO:teuthology.orchestra.run.smithi173.stdout: python3-asyncssh-2.13.2-5.el9.noarch 2024-05-27T07:12:03.451 INFO:teuthology.orchestra.run.smithi173.stdout: python3-attrs-20.3.0-7.el9.noarch 2024-05-27T07:12:03.451 INFO:teuthology.orchestra.run.smithi173.stdout: python3-babel-2.9.1-2.el9.noarch 2024-05-27T07:12:03.451 INFO:teuthology.orchestra.run.smithi173.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2024-05-27T07:12:03.451 INFO:teuthology.orchestra.run.smithi173.stdout: python3-cachetools-4.2.4-1.el9.noarch 2024-05-27T07:12:03.451 INFO:teuthology.orchestra.run.smithi173.stdout: python3-ceph-argparse-2:19.0.0-3872.g38f9d900.el9.x86_64 2024-05-27T07:12:03.451 INFO:teuthology.orchestra.run.smithi173.stdout: python3-ceph-common-2:19.0.0-3872.g38f9d900.el9.x86_64 2024-05-27T07:12:03.451 INFO:teuthology.orchestra.run.smithi173.stdout: python3-cephfs-2:19.0.0-3872.g38f9d900.el9.x86_64 2024-05-27T07:12:03.452 INFO:teuthology.orchestra.run.smithi173.stdout: python3-certifi-2021.10.8-2.el9.noarch 2024-05-27T07:12:03.452 INFO:teuthology.orchestra.run.smithi173.stdout: python3-cffi-1.14.5-5.el9.x86_64 2024-05-27T07:12:03.452 INFO:teuthology.orchestra.run.smithi173.stdout: python3-cheroot-8.6.0-4.el9.noarch 2024-05-27T07:12:03.452 INFO:teuthology.orchestra.run.smithi173.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2024-05-27T07:12:03.452 INFO:teuthology.orchestra.run.smithi173.stdout: python3-cmd2-2.4.2-2.el9.noarch 2024-05-27T07:12:03.452 INFO:teuthology.orchestra.run.smithi173.stdout: python3-colorama-0.4.6-3.el9.noarch 2024-05-27T07:12:03.452 INFO:teuthology.orchestra.run.smithi173.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2024-05-27T07:12:03.452 INFO:teuthology.orchestra.run.smithi173.stdout: python3-devel-3.9.18-3.el9.x86_64 2024-05-27T07:12:03.452 INFO:teuthology.orchestra.run.smithi173.stdout: python3-google-auth-1:2.29.0-1.el9.noarch 2024-05-27T07:12:03.452 INFO:teuthology.orchestra.run.smithi173.stdout: python3-grpcio-1.46.7-10.el9.x86_64 2024-05-27T07:12:03.452 INFO:teuthology.orchestra.run.smithi173.stdout: python3-grpcio-tools-1.46.7-10.el9.x86_64 2024-05-27T07:12:03.452 INFO:teuthology.orchestra.run.smithi173.stdout: python3-jaraco-8.2.1-3.el9.noarch 2024-05-27T07:12:03.453 INFO:teuthology.orchestra.run.smithi173.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2024-05-27T07:12:03.453 INFO:teuthology.orchestra.run.smithi173.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2024-05-27T07:12:03.453 INFO:teuthology.orchestra.run.smithi173.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2024-05-27T07:12:03.453 INFO:teuthology.orchestra.run.smithi173.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2024-05-27T07:12:03.453 INFO:teuthology.orchestra.run.smithi173.stdout: python3-jinja2-2.11.3-6.el9.noarch 2024-05-27T07:12:03.453 INFO:teuthology.orchestra.run.smithi173.stdout: python3-jsonpatch-1.21-16.el9.noarch 2024-05-27T07:12:03.453 INFO:teuthology.orchestra.run.smithi173.stdout: python3-jsonpointer-2.0-4.el9.noarch 2024-05-27T07:12:03.453 INFO:teuthology.orchestra.run.smithi173.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2024-05-27T07:12:03.453 INFO:teuthology.orchestra.run.smithi173.stdout: python3-logutils-0.3.5-21.el9.noarch 2024-05-27T07:12:03.453 INFO:teuthology.orchestra.run.smithi173.stdout: python3-mako-1.1.4-6.el9.noarch 2024-05-27T07:12:03.453 INFO:teuthology.orchestra.run.smithi173.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2024-05-27T07:12:03.453 INFO:teuthology.orchestra.run.smithi173.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2024-05-27T07:12:03.453 INFO:teuthology.orchestra.run.smithi173.stdout: python3-natsort-7.1.1-5.el9.noarch 2024-05-27T07:12:03.453 INFO:teuthology.orchestra.run.smithi173.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2024-05-27T07:12:03.454 INFO:teuthology.orchestra.run.smithi173.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2024-05-27T07:12:03.454 INFO:teuthology.orchestra.run.smithi173.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2024-05-27T07:12:03.454 INFO:teuthology.orchestra.run.smithi173.stdout: python3-packaging-20.9-5.el9.noarch 2024-05-27T07:12:03.454 INFO:teuthology.orchestra.run.smithi173.stdout: python3-pecan-1.4.2-3.el9.noarch 2024-05-27T07:12:03.454 INFO:teuthology.orchestra.run.smithi173.stdout: python3-ply-3.11-14.el9.noarch 2024-05-27T07:12:03.454 INFO:teuthology.orchestra.run.smithi173.stdout: python3-portend-3.1.0-2.el9.noarch 2024-05-27T07:12:03.454 INFO:teuthology.orchestra.run.smithi173.stdout: python3-prettytable-0.7.2-27.el9.noarch 2024-05-27T07:12:03.454 INFO:teuthology.orchestra.run.smithi173.stdout: python3-protobuf-3.14.0-13.el9.noarch 2024-05-27T07:12:03.454 INFO:teuthology.orchestra.run.smithi173.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2024-05-27T07:12:03.454 INFO:teuthology.orchestra.run.smithi173.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2024-05-27T07:12:03.454 INFO:teuthology.orchestra.run.smithi173.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2024-05-27T07:12:03.454 INFO:teuthology.orchestra.run.smithi173.stdout: python3-pycparser-2.20-6.el9.noarch 2024-05-27T07:12:03.454 INFO:teuthology.orchestra.run.smithi173.stdout: python3-pyperclip-1.8.2-2.el9.noarch 2024-05-27T07:12:03.455 INFO:teuthology.orchestra.run.smithi173.stdout: python3-pytz-2021.1-5.el9.noarch 2024-05-27T07:12:03.455 INFO:teuthology.orchestra.run.smithi173.stdout: python3-rados-2:19.0.0-3872.g38f9d900.el9.x86_64 2024-05-27T07:12:03.455 INFO:teuthology.orchestra.run.smithi173.stdout: python3-rbd-2:19.0.0-3872.g38f9d900.el9.x86_64 2024-05-27T07:12:03.455 INFO:teuthology.orchestra.run.smithi173.stdout: python3-repoze-lru-0.7-16.el9.noarch 2024-05-27T07:12:03.455 INFO:teuthology.orchestra.run.smithi173.stdout: python3-requests-2.25.1-8.el9.noarch 2024-05-27T07:12:03.455 INFO:teuthology.orchestra.run.smithi173.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2024-05-27T07:12:03.455 INFO:teuthology.orchestra.run.smithi173.stdout: python3-rgw-2:19.0.0-3872.g38f9d900.el9.x86_64 2024-05-27T07:12:03.455 INFO:teuthology.orchestra.run.smithi173.stdout: python3-routes-2.5.1-5.el9.noarch 2024-05-27T07:12:03.455 INFO:teuthology.orchestra.run.smithi173.stdout: python3-rsa-4.9-2.el9.noarch 2024-05-27T07:12:03.455 INFO:teuthology.orchestra.run.smithi173.stdout: python3-scipy-1.6.2-8.el9.x86_64 2024-05-27T07:12:03.455 INFO:teuthology.orchestra.run.smithi173.stdout: python3-tempora-5.0.0-2.el9.noarch 2024-05-27T07:12:03.455 INFO:teuthology.orchestra.run.smithi173.stdout: python3-toml-0.10.2-6.el9.noarch 2024-05-27T07:12:03.456 INFO:teuthology.orchestra.run.smithi173.stdout: python3-typing-extensions-4.4.0-2.el9.noarch 2024-05-27T07:12:03.456 INFO:teuthology.orchestra.run.smithi173.stdout: python3-urllib3-1.26.5-5.el9.noarch 2024-05-27T07:12:03.456 INFO:teuthology.orchestra.run.smithi173.stdout: python3-wcwidth-0.2.5-8.el9.noarch 2024-05-27T07:12:03.456 INFO:teuthology.orchestra.run.smithi173.stdout: python3-webob-1.8.7-6.el9.noarch 2024-05-27T07:12:03.456 INFO:teuthology.orchestra.run.smithi173.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2024-05-27T07:12:03.456 INFO:teuthology.orchestra.run.smithi173.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2024-05-27T07:12:03.456 INFO:teuthology.orchestra.run.smithi173.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2024-05-27T07:12:03.456 INFO:teuthology.orchestra.run.smithi173.stdout: qatlib-24.02.0-1.el9.x86_64 2024-05-27T07:12:03.456 INFO:teuthology.orchestra.run.smithi173.stdout: qatlib-service-24.02.0-1.el9.x86_64 2024-05-27T07:12:03.456 INFO:teuthology.orchestra.run.smithi173.stdout: qatzip-libs-1.2.0-1.el9.x86_64 2024-05-27T07:12:03.456 INFO:teuthology.orchestra.run.smithi173.stdout: rbd-fuse-2:19.0.0-3872.g38f9d900.el9.x86_64 2024-05-27T07:12:03.456 INFO:teuthology.orchestra.run.smithi173.stdout: rbd-mirror-2:19.0.0-3872.g38f9d900.el9.x86_64 2024-05-27T07:12:03.456 INFO:teuthology.orchestra.run.smithi173.stdout: rbd-nbd-2:19.0.0-3872.g38f9d900.el9.x86_64 2024-05-27T07:12:03.456 INFO:teuthology.orchestra.run.smithi173.stdout: re2-1:20211101-3.el9.x86_64 2024-05-27T07:12:03.457 INFO:teuthology.orchestra.run.smithi173.stdout: socat-1.7.4.1-5.el9.x86_64 2024-05-27T07:12:03.457 INFO:teuthology.orchestra.run.smithi173.stdout: thrift-0.15.0-2.el9.x86_64 2024-05-27T07:12:03.457 INFO:teuthology.orchestra.run.smithi173.stdout: uuid-1.6.2-55.el9.x86_64 2024-05-27T07:12:03.457 INFO:teuthology.orchestra.run.smithi173.stdout: xfsprogs-devel-6.3.0-1.el9.x86_64 2024-05-27T07:12:03.457 INFO:teuthology.orchestra.run.smithi173.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2024-05-27T07:12:03.457 INFO:teuthology.orchestra.run.smithi173.stdout: zlib-devel-1.2.11-41.el9.x86_64 2024-05-27T07:12:03.457 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-05-27T07:12:03.457 INFO:teuthology.orchestra.run.smithi173.stdout:Complete! 2024-05-27T07:12:03.890 DEBUG:teuthology.parallel:result is None 2024-05-27T07:12:04.589 INFO:teuthology.orchestra.run.smithi105.stdout: Running scriptlet: elfutils-libelf-0.190-2.el9.x86_64 170/170 2024-05-27T07:12:04.589 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : ceph-2:19.0.0-3872.g38f9d900.el9.x86_64 1/170 2024-05-27T07:12:04.589 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : ceph-base-2:19.0.0-3872.g38f9d900.el9.x86_64 2/170 2024-05-27T07:12:04.589 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : ceph-common-2:19.0.0-3872.g38f9d900.el9.x86_64 3/170 2024-05-27T07:12:04.589 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : ceph-fuse-2:19.0.0-3872.g38f9d900.el9.x86_64 4/170 2024-05-27T07:12:04.589 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : ceph-immutable-object-cache-2:19.0.0-3872.g38f9d 5/170 2024-05-27T07:12:04.589 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : ceph-mds-2:19.0.0-3872.g38f9d900.el9.x86_64 6/170 2024-05-27T07:12:04.589 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : ceph-mgr-2:19.0.0-3872.g38f9d900.el9.x86_64 7/170 2024-05-27T07:12:04.589 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : ceph-mon-2:19.0.0-3872.g38f9d900.el9.x86_64 8/170 2024-05-27T07:12:04.590 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : ceph-osd-2:19.0.0-3872.g38f9d900.el9.x86_64 9/170 2024-05-27T07:12:04.590 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : ceph-radosgw-2:19.0.0-3872.g38f9d900.el9.x86_64 10/170 2024-05-27T07:12:04.590 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : ceph-selinux-2:19.0.0-3872.g38f9d900.el9.x86_64 11/170 2024-05-27T07:12:04.590 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : ceph-test-2:19.0.0-3872.g38f9d900.el9.x86_64 12/170 2024-05-27T07:12:04.590 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : cephfs-mirror-2:19.0.0-3872.g38f9d900.el9.x86_64 13/170 2024-05-27T07:12:04.590 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : cephfs-shell-2:19.0.0-3872.g38f9d900.el9.x86_64 14/170 2024-05-27T07:12:04.590 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : libcephfs-devel-2:19.0.0-3872.g38f9d900.el9.x86_ 15/170 2024-05-27T07:12:04.590 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : libcephfs2-2:19.0.0-3872.g38f9d900.el9.x86_64 16/170 2024-05-27T07:12:04.590 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : libcephsqlite-2:19.0.0-3872.g38f9d900.el9.x86_64 17/170 2024-05-27T07:12:04.590 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : librados-devel-2:19.0.0-3872.g38f9d900.el9.x86_6 18/170 2024-05-27T07:12:04.590 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : libradosstriper1-2:19.0.0-3872.g38f9d900.el9.x86 19/170 2024-05-27T07:12:04.590 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : librgw2-2:19.0.0-3872.g38f9d900.el9.x86_64 20/170 2024-05-27T07:12:04.591 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : python3-ceph-argparse-2:19.0.0-3872.g38f9d900.el 21/170 2024-05-27T07:12:04.591 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : python3-ceph-common-2:19.0.0-3872.g38f9d900.el9. 22/170 2024-05-27T07:12:04.591 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : python3-cephfs-2:19.0.0-3872.g38f9d900.el9.x86_6 23/170 2024-05-27T07:12:04.591 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : python3-rados-2:19.0.0-3872.g38f9d900.el9.x86_64 24/170 2024-05-27T07:12:04.591 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : python3-rbd-2:19.0.0-3872.g38f9d900.el9.x86_64 25/170 2024-05-27T07:12:04.591 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : python3-rgw-2:19.0.0-3872.g38f9d900.el9.x86_64 26/170 2024-05-27T07:12:04.591 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : rbd-fuse-2:19.0.0-3872.g38f9d900.el9.x86_64 27/170 2024-05-27T07:12:04.591 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : rbd-mirror-2:19.0.0-3872.g38f9d900.el9.x86_64 28/170 2024-05-27T07:12:04.591 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : rbd-nbd-2:19.0.0-3872.g38f9d900.el9.x86_64 29/170 2024-05-27T07:12:04.591 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : ceph-grafana-dashboards-2:19.0.0-3872.g38f9d900. 30/170 2024-05-27T07:12:04.591 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : ceph-mgr-cephadm-2:19.0.0-3872.g38f9d900.el9.noa 31/170 2024-05-27T07:12:04.591 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : ceph-mgr-dashboard-2:19.0.0-3872.g38f9d900.el9.n 32/170 2024-05-27T07:12:04.592 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : ceph-mgr-diskprediction-local-2:19.0.0-3872.g38f 33/170 2024-05-27T07:12:04.592 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : ceph-mgr-modules-core-2:19.0.0-3872.g38f9d900.el 34/170 2024-05-27T07:12:04.592 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : ceph-mgr-rook-2:19.0.0-3872.g38f9d900.el9.noarch 35/170 2024-05-27T07:12:04.592 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : ceph-prometheus-alerts-2:19.0.0-3872.g38f9d900.e 36/170 2024-05-27T07:12:04.592 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : ceph-volume-2:19.0.0-3872.g38f9d900.el9.noarch 37/170 2024-05-27T07:12:04.592 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : cephadm-2:19.0.0-3872.g38f9d900.el9.noarch 38/170 2024-05-27T07:12:04.592 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : cephfs-top-2:19.0.0-3872.g38f9d900.el9.noarch 39/170 2024-05-27T07:12:04.592 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : fuse-2.9.9-15.el9.x86_64 40/170 2024-05-27T07:12:04.592 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : libgfortran-11.4.1-3.el9.x86_64 41/170 2024-05-27T07:12:04.592 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : libquadmath-11.4.1-3.el9.x86_64 42/170 2024-05-27T07:12:04.592 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 43/170 2024-05-27T07:12:04.592 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 44/170 2024-05-27T07:12:04.592 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : python3-ply-3.11-14.el9.noarch 45/170 2024-05-27T07:12:04.593 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 46/170 2024-05-27T07:12:04.593 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : python3-requests-2.25.1-8.el9.noarch 47/170 2024-05-27T07:12:04.593 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : python3-urllib3-1.26.5-5.el9.noarch 48/170 2024-05-27T07:12:04.593 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : automake-1.16.2-8.el9.noarch 49/170 2024-05-27T07:12:04.593 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : bison-3.7.4-5.el9.x86_64 50/170 2024-05-27T07:12:04.593 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : boost-program-options-1.75.0-8.el9.x86_64 51/170 2024-05-27T07:12:04.593 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : elfutils-libelf-devel-0.191-4.el9.x86_64 52/170 2024-05-27T07:12:04.593 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : flex-2.6.4-9.el9.x86_64 53/170 2024-05-27T07:12:04.593 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 54/170 2024-05-27T07:12:04.593 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 55/170 2024-05-27T07:12:04.593 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 56/170 2024-05-27T07:12:04.593 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : libaio-devel-0.3.111-13.el9.x86_64 57/170 2024-05-27T07:12:04.594 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 58/170 2024-05-27T07:12:04.594 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 59/170 2024-05-27T07:12:04.594 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 60/170 2024-05-27T07:12:04.594 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : libtool-2.4.6-45.el9.x86_64 61/170 2024-05-27T07:12:04.594 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : libuuid-devel-2.37.4-18.el9.x86_64 62/170 2024-05-27T07:12:04.594 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : libzstd-devel-1.5.1-2.el9.x86_64 63/170 2024-05-27T07:12:04.594 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 64/170 2024-05-27T07:12:04.594 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : lua-5.4.4-4.el9.x86_64 65/170 2024-05-27T07:12:04.594 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : openblas-0.3.21-2.el9.x86_64 66/170 2024-05-27T07:12:04.594 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : openblas-openmp-0.3.21-2.el9.x86_64 67/170 2024-05-27T07:12:04.594 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : openssl-devel-1:3.2.1-1.el9.x86_64 68/170 2024-05-27T07:12:04.594 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : postgresql-13.14-1.el9.x86_64 69/170 2024-05-27T07:12:04.595 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : postgresql-contrib-13.14-1.el9.x86_64 70/170 2024-05-27T07:12:04.595 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : postgresql-private-libs-13.14-1.el9.x86_64 71/170 2024-05-27T07:12:04.595 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : postgresql-server-13.14-1.el9.x86_64 72/170 2024-05-27T07:12:04.595 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : protobuf-3.14.0-13.el9.x86_64 73/170 2024-05-27T07:12:04.595 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : python3-attrs-20.3.0-7.el9.noarch 74/170 2024-05-27T07:12:04.595 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 75/170 2024-05-27T07:12:04.595 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : python3-colorama-0.4.6-3.el9.noarch 76/170 2024-05-27T07:12:04.595 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : python3-devel-3.9.18-3.el9.x86_64 77/170 2024-05-27T07:12:04.595 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : python3-jinja2-2.11.3-6.el9.noarch 78/170 2024-05-27T07:12:04.595 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 79/170 2024-05-27T07:12:04.595 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 80/170 2024-05-27T07:12:04.595 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 81/170 2024-05-27T07:12:04.595 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 82/170 2024-05-27T07:12:04.595 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 83/170 2024-05-27T07:12:04.596 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 84/170 2024-05-27T07:12:04.596 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 85/170 2024-05-27T07:12:04.596 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : python3-packaging-20.9-5.el9.noarch 86/170 2024-05-27T07:12:04.596 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 87/170 2024-05-27T07:12:04.596 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : python3-protobuf-3.14.0-13.el9.noarch 88/170 2024-05-27T07:12:04.596 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 89/170 2024-05-27T07:12:04.596 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 90/170 2024-05-27T07:12:04.596 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 91/170 2024-05-27T07:12:04.596 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 92/170 2024-05-27T07:12:04.596 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 93/170 2024-05-27T07:12:04.596 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 94/170 2024-05-27T07:12:04.596 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : python3-wcwidth-0.2.5-8.el9.noarch 95/170 2024-05-27T07:12:04.597 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : qatlib-24.02.0-1.el9.x86_64 96/170 2024-05-27T07:12:04.597 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : qatlib-service-24.02.0-1.el9.x86_64 97/170 2024-05-27T07:12:04.597 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : qatzip-libs-1.2.0-1.el9.x86_64 98/170 2024-05-27T07:12:04.597 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : socat-1.7.4.1-5.el9.x86_64 99/170 2024-05-27T07:12:04.597 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : uuid-1.6.2-55.el9.x86_64 100/170 2024-05-27T07:12:04.597 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : xfsprogs-devel-6.3.0-1.el9.x86_64 101/170 2024-05-27T07:12:04.597 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 102/170 2024-05-27T07:12:04.597 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : zlib-devel-1.2.11-41.el9.x86_64 103/170 2024-05-27T07:12:04.597 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : lua-devel-5.4.4-4.el9.x86_64 104/170 2024-05-27T07:12:04.597 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : protobuf-compiler-3.14.0-13.el9.x86_64 105/170 2024-05-27T07:12:04.597 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : abseil-cpp-20211102.0-3.el9.x86_64 106/170 2024-05-27T07:12:04.597 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : gperftools-libs-2.9.1-2.el9.x86_64 107/170 2024-05-27T07:12:04.597 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : grpc-data-1.46.7-10.el9.noarch 108/170 2024-05-27T07:12:04.598 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : libarrow-9.0.0-11.el9.x86_64 109/170 2024-05-27T07:12:04.598 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : libarrow-doc-9.0.0-11.el9.noarch 110/170 2024-05-27T07:12:04.598 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 111/170 2024-05-27T07:12:04.598 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 112/170 2024-05-27T07:12:04.598 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : luarocks-3.9.2-5.el9.noarch 113/170 2024-05-27T07:12:04.598 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : parquet-libs-9.0.0-11.el9.x86_64 114/170 2024-05-27T07:12:04.598 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : python3-asyncssh-2.13.2-5.el9.noarch 115/170 2024-05-27T07:12:04.598 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 116/170 2024-05-27T07:12:04.598 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 117/170 2024-05-27T07:12:04.598 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : python3-certifi-2021.10.8-2.el9.noarch 118/170 2024-05-27T07:12:04.598 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 119/170 2024-05-27T07:12:04.598 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 120/170 2024-05-27T07:12:04.599 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : python3-cmd2-2.4.2-2.el9.noarch 121/170 2024-05-27T07:12:04.599 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : python3-google-auth-1:2.29.0-1.el9.noarch 122/170 2024-05-27T07:12:04.599 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : python3-grpcio-1.46.7-10.el9.x86_64 123/170 2024-05-27T07:12:04.599 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : python3-grpcio-tools-1.46.7-10.el9.x86_64 124/170 2024-05-27T07:12:04.599 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 125/170 2024-05-27T07:12:04.599 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 126/170 2024-05-27T07:12:04.599 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 127/170 2024-05-27T07:12:04.599 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 128/170 2024-05-27T07:12:04.599 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 129/170 2024-05-27T07:12:04.599 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 130/170 2024-05-27T07:12:04.599 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 131/170 2024-05-27T07:12:04.599 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 132/170 2024-05-27T07:12:04.599 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 133/170 2024-05-27T07:12:04.600 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 134/170 2024-05-27T07:12:04.600 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 135/170 2024-05-27T07:12:04.600 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 136/170 2024-05-27T07:12:04.600 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : python3-pyperclip-1.8.2-2.el9.noarch 137/170 2024-05-27T07:12:04.600 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 138/170 2024-05-27T07:12:04.600 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 139/170 2024-05-27T07:12:04.600 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 140/170 2024-05-27T07:12:04.600 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 141/170 2024-05-27T07:12:04.600 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : python3-typing-extensions-4.4.0-2.el9.noarch 142/170 2024-05-27T07:12:04.600 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : python3-webob-1.8.7-6.el9.noarch 143/170 2024-05-27T07:12:04.600 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 144/170 2024-05-27T07:12:04.600 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 145/170 2024-05-27T07:12:04.600 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 146/170 2024-05-27T07:12:04.601 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : re2-1:20211101-3.el9.x86_64 147/170 2024-05-27T07:12:04.601 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 148/170 2024-05-27T07:12:04.601 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : librados2-2:19.0.0-3872.g38f9d900.el9.x86_64 149/170 2024-05-27T07:12:04.601 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 150/170 2024-05-27T07:12:04.601 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : librbd1-2:19.0.0-3872.g38f9d900.el9.x86_64 151/170 2024-05-27T07:12:04.601 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 152/170 2024-05-27T07:12:04.601 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : NetworkManager-1:1.47.90-1.el9.x86_64 153/170 2024-05-27T07:12:04.601 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : NetworkManager-1:1.47.4-1.el9.x86_64 154/170 2024-05-27T07:12:04.601 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : NetworkManager-libnm-1:1.47.90-1.el9.x86_64 155/170 2024-05-27T07:12:04.601 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : NetworkManager-libnm-1:1.47.4-1.el9.x86_64 156/170 2024-05-27T07:12:04.601 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : NetworkManager-team-1:1.47.90-1.el9.x86_64 157/170 2024-05-27T07:12:04.601 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : NetworkManager-team-1:1.47.4-1.el9.x86_64 158/170 2024-05-27T07:12:04.601 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : NetworkManager-tui-1:1.47.90-1.el9.x86_64 159/170 2024-05-27T07:12:04.602 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : NetworkManager-tui-1:1.47.4-1.el9.x86_64 160/170 2024-05-27T07:12:04.602 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : elfutils-debuginfod-client-0.191-4.el9.x86_64 161/170 2024-05-27T07:12:04.602 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : elfutils-debuginfod-client-0.190-2.el9.x86_64 162/170 2024-05-27T07:12:04.602 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : elfutils-libelf-0.191-4.el9.x86_64 163/170 2024-05-27T07:12:04.602 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : elfutils-libelf-0.190-2.el9.x86_64 164/170 2024-05-27T07:12:04.602 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : elfutils-libs-0.191-4.el9.x86_64 165/170 2024-05-27T07:12:04.602 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : elfutils-libs-0.190-2.el9.x86_64 166/170 2024-05-27T07:12:04.602 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : openssl-1:3.2.1-1.el9.x86_64 167/170 2024-05-27T07:12:04.602 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : openssl-1:3.0.7-27.el9.x86_64 168/170 2024-05-27T07:12:04.602 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : openssl-libs-1:3.2.1-1.el9.x86_64 169/170 2024-05-27T07:12:04.869 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : ceph-test-2:19.0.0-3872.g38f9d900.el9.x86_64 143/170 2024-05-27T07:12:04.911 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : rbd-fuse-2:19.0.0-3872.g38f9d900.el9.x86_64 144/170 2024-05-27T07:12:04.951 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : rbd-nbd-2:19.0.0-3872.g38f9d900.el9.x86_64 145/170 2024-05-27T07:12:04.983 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : xfsprogs-devel-6.3.0-1.el9.x86_64 146/170 2024-05-27T07:12:05.010 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : elfutils-libelf-devel-0.191-4.el9.x86_64 147/170 2024-05-27T07:12:05.229 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : cephfs-shell-2:19.0.0-3872.g38f9d900.el9.x86_64 148/170 2024-05-27T07:12:05.273 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : postgresql-contrib-13.14-1.el9.x86_64 149/170 2024-05-27T07:12:05.306 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : libcephfs-devel-2:19.0.0-3872.g38f9d900.el9.x86_ 150/170 2024-05-27T07:12:05.337 INFO:teuthology.orchestra.run.smithi052.stdout: Upgrading : elfutils-debuginfod-client-0.191-4.el9.x86_64 151/170 2024-05-27T07:12:05.367 INFO:teuthology.orchestra.run.smithi052.stdout: Upgrading : NetworkManager-team-1:1.47.90-1.el9.x86_64 152/170 2024-05-27T07:12:05.404 INFO:teuthology.orchestra.run.smithi052.stdout: Upgrading : NetworkManager-tui-1:1.47.90-1.el9.x86_64 153/170 2024-05-27T07:12:05.983 INFO:teuthology.orchestra.run.smithi052.stdout: Running scriptlet: postgresql-server-13.14-1.el9.x86_64 154/170 2024-05-27T07:12:06.018 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : postgresql-server-13.14-1.el9.x86_64 154/170 2024-05-27T07:12:06.040 INFO:teuthology.orchestra.run.smithi052.stdout: Running scriptlet: postgresql-server-13.14-1.el9.x86_64 154/170 2024-05-27T07:12:06.972 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : cephfs-top-2:19.0.0-3872.g38f9d900.el9.noarch 155/170 2024-05-27T07:12:07.073 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : openssl-devel-1:3.2.1-1.el9.x86_64 156/170 2024-05-27T07:12:07.115 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : libaio-devel-0.3.111-13.el9.x86_64 157/170 2024-05-27T07:12:07.216 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : flex-2.6.4-9.el9.x86_64 158/170 2024-05-27T07:12:07.251 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : bison-3.7.4-5.el9.x86_64 159/170 2024-05-27T07:12:07.275 INFO:teuthology.orchestra.run.smithi052.stdout: Cleanup : NetworkManager-tui-1:1.47.4-1.el9.x86_64 160/170 2024-05-27T07:12:07.301 INFO:teuthology.orchestra.run.smithi052.stdout: Cleanup : elfutils-debuginfod-client-0.190-2.el9.x86_64 161/170 2024-05-27T07:12:07.327 INFO:teuthology.orchestra.run.smithi052.stdout: Cleanup : elfutils-libs-0.190-2.el9.x86_64 162/170 2024-05-27T07:12:07.328 INFO:teuthology.orchestra.run.smithi052.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 163/170 2024-05-27T07:12:07.412 INFO:teuthology.orchestra.run.smithi052.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 163/170 2024-05-27T07:12:07.453 INFO:teuthology.orchestra.run.smithi052.stdout: Cleanup : openssl-1:3.0.7-27.el9.x86_64 164/170 2024-05-27T07:12:07.453 INFO:teuthology.orchestra.run.smithi052.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 165/170 2024-05-27T07:12:07.521 INFO:teuthology.orchestra.run.smithi052.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 165/170 2024-05-27T07:12:07.529 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : openssl-libs-1:3.0.7-27.el9.x86_64 170/170 2024-05-27T07:12:07.529 INFO:teuthology.orchestra.run.smithi105.stdout: 2024-05-27T07:12:07.529 INFO:teuthology.orchestra.run.smithi105.stdout:Upgraded: 2024-05-27T07:12:07.529 INFO:teuthology.orchestra.run.smithi105.stdout: NetworkManager-1:1.47.90-1.el9.x86_64 2024-05-27T07:12:07.529 INFO:teuthology.orchestra.run.smithi105.stdout: NetworkManager-libnm-1:1.47.90-1.el9.x86_64 2024-05-27T07:12:07.529 INFO:teuthology.orchestra.run.smithi105.stdout: NetworkManager-team-1:1.47.90-1.el9.x86_64 2024-05-27T07:12:07.529 INFO:teuthology.orchestra.run.smithi105.stdout: NetworkManager-tui-1:1.47.90-1.el9.x86_64 2024-05-27T07:12:07.530 INFO:teuthology.orchestra.run.smithi105.stdout: elfutils-debuginfod-client-0.191-4.el9.x86_64 2024-05-27T07:12:07.530 INFO:teuthology.orchestra.run.smithi105.stdout: elfutils-libelf-0.191-4.el9.x86_64 2024-05-27T07:12:07.530 INFO:teuthology.orchestra.run.smithi105.stdout: elfutils-libs-0.191-4.el9.x86_64 2024-05-27T07:12:07.530 INFO:teuthology.orchestra.run.smithi105.stdout: librados2-2:19.0.0-3872.g38f9d900.el9.x86_64 2024-05-27T07:12:07.530 INFO:teuthology.orchestra.run.smithi105.stdout: librbd1-2:19.0.0-3872.g38f9d900.el9.x86_64 2024-05-27T07:12:07.530 INFO:teuthology.orchestra.run.smithi105.stdout: openssl-1:3.2.1-1.el9.x86_64 2024-05-27T07:12:07.530 INFO:teuthology.orchestra.run.smithi105.stdout: openssl-libs-1:3.2.1-1.el9.x86_64 2024-05-27T07:12:07.530 INFO:teuthology.orchestra.run.smithi105.stdout:Installed: 2024-05-27T07:12:07.530 INFO:teuthology.orchestra.run.smithi105.stdout: abseil-cpp-20211102.0-3.el9.x86_64 2024-05-27T07:12:07.530 INFO:teuthology.orchestra.run.smithi105.stdout: automake-1.16.2-8.el9.noarch 2024-05-27T07:12:07.530 INFO:teuthology.orchestra.run.smithi105.stdout: bison-3.7.4-5.el9.x86_64 2024-05-27T07:12:07.530 INFO:teuthology.orchestra.run.smithi105.stdout: boost-program-options-1.75.0-8.el9.x86_64 2024-05-27T07:12:07.531 INFO:teuthology.orchestra.run.smithi105.stdout: ceph-2:19.0.0-3872.g38f9d900.el9.x86_64 2024-05-27T07:12:07.531 INFO:teuthology.orchestra.run.smithi105.stdout: ceph-base-2:19.0.0-3872.g38f9d900.el9.x86_64 2024-05-27T07:12:07.531 INFO:teuthology.orchestra.run.smithi105.stdout: ceph-common-2:19.0.0-3872.g38f9d900.el9.x86_64 2024-05-27T07:12:07.531 INFO:teuthology.orchestra.run.smithi105.stdout: ceph-fuse-2:19.0.0-3872.g38f9d900.el9.x86_64 2024-05-27T07:12:07.531 INFO:teuthology.orchestra.run.smithi105.stdout: ceph-grafana-dashboards-2:19.0.0-3872.g38f9d900.el9.noarch 2024-05-27T07:12:07.531 INFO:teuthology.orchestra.run.smithi105.stdout: ceph-immutable-object-cache-2:19.0.0-3872.g38f9d900.el9.x86_64 2024-05-27T07:12:07.531 INFO:teuthology.orchestra.run.smithi105.stdout: ceph-mds-2:19.0.0-3872.g38f9d900.el9.x86_64 2024-05-27T07:12:07.531 INFO:teuthology.orchestra.run.smithi105.stdout: ceph-mgr-2:19.0.0-3872.g38f9d900.el9.x86_64 2024-05-27T07:12:07.531 INFO:teuthology.orchestra.run.smithi105.stdout: ceph-mgr-cephadm-2:19.0.0-3872.g38f9d900.el9.noarch 2024-05-27T07:12:07.531 INFO:teuthology.orchestra.run.smithi105.stdout: ceph-mgr-dashboard-2:19.0.0-3872.g38f9d900.el9.noarch 2024-05-27T07:12:07.531 INFO:teuthology.orchestra.run.smithi105.stdout: ceph-mgr-diskprediction-local-2:19.0.0-3872.g38f9d900.el9.noarch 2024-05-27T07:12:07.531 INFO:teuthology.orchestra.run.smithi105.stdout: ceph-mgr-modules-core-2:19.0.0-3872.g38f9d900.el9.noarch 2024-05-27T07:12:07.531 INFO:teuthology.orchestra.run.smithi105.stdout: ceph-mgr-rook-2:19.0.0-3872.g38f9d900.el9.noarch 2024-05-27T07:12:07.532 INFO:teuthology.orchestra.run.smithi105.stdout: ceph-mon-2:19.0.0-3872.g38f9d900.el9.x86_64 2024-05-27T07:12:07.532 INFO:teuthology.orchestra.run.smithi105.stdout: ceph-osd-2:19.0.0-3872.g38f9d900.el9.x86_64 2024-05-27T07:12:07.532 INFO:teuthology.orchestra.run.smithi105.stdout: ceph-prometheus-alerts-2:19.0.0-3872.g38f9d900.el9.noarch 2024-05-27T07:12:07.532 INFO:teuthology.orchestra.run.smithi105.stdout: ceph-radosgw-2:19.0.0-3872.g38f9d900.el9.x86_64 2024-05-27T07:12:07.532 INFO:teuthology.orchestra.run.smithi105.stdout: ceph-selinux-2:19.0.0-3872.g38f9d900.el9.x86_64 2024-05-27T07:12:07.532 INFO:teuthology.orchestra.run.smithi105.stdout: ceph-test-2:19.0.0-3872.g38f9d900.el9.x86_64 2024-05-27T07:12:07.532 INFO:teuthology.orchestra.run.smithi105.stdout: ceph-volume-2:19.0.0-3872.g38f9d900.el9.noarch 2024-05-27T07:12:07.532 INFO:teuthology.orchestra.run.smithi105.stdout: cephadm-2:19.0.0-3872.g38f9d900.el9.noarch 2024-05-27T07:12:07.532 INFO:teuthology.orchestra.run.smithi105.stdout: cephfs-mirror-2:19.0.0-3872.g38f9d900.el9.x86_64 2024-05-27T07:12:07.532 INFO:teuthology.orchestra.run.smithi105.stdout: cephfs-shell-2:19.0.0-3872.g38f9d900.el9.x86_64 2024-05-27T07:12:07.532 INFO:teuthology.orchestra.run.smithi105.stdout: cephfs-top-2:19.0.0-3872.g38f9d900.el9.noarch 2024-05-27T07:12:07.532 INFO:teuthology.orchestra.run.smithi105.stdout: elfutils-libelf-devel-0.191-4.el9.x86_64 2024-05-27T07:12:07.533 INFO:teuthology.orchestra.run.smithi105.stdout: flex-2.6.4-9.el9.x86_64 2024-05-27T07:12:07.533 INFO:teuthology.orchestra.run.smithi105.stdout: flexiblas-3.0.4-8.el9.x86_64 2024-05-27T07:12:07.533 INFO:teuthology.orchestra.run.smithi105.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2024-05-27T07:12:07.533 INFO:teuthology.orchestra.run.smithi105.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2024-05-27T07:12:07.533 INFO:teuthology.orchestra.run.smithi105.stdout: fuse-2.9.9-15.el9.x86_64 2024-05-27T07:12:07.533 INFO:teuthology.orchestra.run.smithi105.stdout: gperftools-libs-2.9.1-2.el9.x86_64 2024-05-27T07:12:07.533 INFO:teuthology.orchestra.run.smithi105.stdout: grpc-data-1.46.7-10.el9.noarch 2024-05-27T07:12:07.534 INFO:teuthology.orchestra.run.smithi105.stdout: libaio-devel-0.3.111-13.el9.x86_64 2024-05-27T07:12:07.534 INFO:teuthology.orchestra.run.smithi105.stdout: libarrow-9.0.0-11.el9.x86_64 2024-05-27T07:12:07.534 INFO:teuthology.orchestra.run.smithi105.stdout: libarrow-doc-9.0.0-11.el9.noarch 2024-05-27T07:12:07.534 INFO:teuthology.orchestra.run.smithi105.stdout: libcephfs-devel-2:19.0.0-3872.g38f9d900.el9.x86_64 2024-05-27T07:12:07.534 INFO:teuthology.orchestra.run.smithi105.stdout: libcephfs2-2:19.0.0-3872.g38f9d900.el9.x86_64 2024-05-27T07:12:07.534 INFO:teuthology.orchestra.run.smithi105.stdout: libcephsqlite-2:19.0.0-3872.g38f9d900.el9.x86_64 2024-05-27T07:12:07.534 INFO:teuthology.orchestra.run.smithi105.stdout: libgfortran-11.4.1-3.el9.x86_64 2024-05-27T07:12:07.534 INFO:teuthology.orchestra.run.smithi105.stdout: liboath-2.6.7-2.el9.x86_64 2024-05-27T07:12:07.534 INFO:teuthology.orchestra.run.smithi105.stdout: libpmemobj-1.12.1-1.el9.x86_64 2024-05-27T07:12:07.534 INFO:teuthology.orchestra.run.smithi105.stdout: libquadmath-11.4.1-3.el9.x86_64 2024-05-27T07:12:07.535 INFO:teuthology.orchestra.run.smithi105.stdout: librabbitmq-0.11.0-7.el9.x86_64 2024-05-27T07:12:07.535 INFO:teuthology.orchestra.run.smithi105.stdout: librados-devel-2:19.0.0-3872.g38f9d900.el9.x86_64 2024-05-27T07:12:07.535 INFO:teuthology.orchestra.run.smithi105.stdout: libradosstriper1-2:19.0.0-3872.g38f9d900.el9.x86_64 2024-05-27T07:12:07.535 INFO:teuthology.orchestra.run.smithi105.stdout: librdkafka-1.6.1-102.el9.x86_64 2024-05-27T07:12:07.535 INFO:teuthology.orchestra.run.smithi105.stdout: librgw2-2:19.0.0-3872.g38f9d900.el9.x86_64 2024-05-27T07:12:07.535 INFO:teuthology.orchestra.run.smithi105.stdout: libtool-2.4.6-45.el9.x86_64 2024-05-27T07:12:07.535 INFO:teuthology.orchestra.run.smithi105.stdout: libunwind-1.6.2-1.el9.x86_64 2024-05-27T07:12:07.535 INFO:teuthology.orchestra.run.smithi105.stdout: libuuid-devel-2.37.4-18.el9.x86_64 2024-05-27T07:12:07.535 INFO:teuthology.orchestra.run.smithi105.stdout: libzstd-devel-1.5.1-2.el9.x86_64 2024-05-27T07:12:07.535 INFO:teuthology.orchestra.run.smithi105.stdout: lttng-ust-2.12.0-6.el9.x86_64 2024-05-27T07:12:07.535 INFO:teuthology.orchestra.run.smithi105.stdout: lua-5.4.4-4.el9.x86_64 2024-05-27T07:12:07.536 INFO:teuthology.orchestra.run.smithi105.stdout: lua-devel-5.4.4-4.el9.x86_64 2024-05-27T07:12:07.536 INFO:teuthology.orchestra.run.smithi105.stdout: luarocks-3.9.2-5.el9.noarch 2024-05-27T07:12:07.536 INFO:teuthology.orchestra.run.smithi105.stdout: openblas-0.3.21-2.el9.x86_64 2024-05-27T07:12:07.536 INFO:teuthology.orchestra.run.smithi105.stdout: openblas-openmp-0.3.21-2.el9.x86_64 2024-05-27T07:12:07.536 INFO:teuthology.orchestra.run.smithi105.stdout: openssl-devel-1:3.2.1-1.el9.x86_64 2024-05-27T07:12:07.536 INFO:teuthology.orchestra.run.smithi105.stdout: parquet-libs-9.0.0-11.el9.x86_64 2024-05-27T07:12:07.536 INFO:teuthology.orchestra.run.smithi105.stdout: postgresql-13.14-1.el9.x86_64 2024-05-27T07:12:07.536 INFO:teuthology.orchestra.run.smithi105.stdout: postgresql-contrib-13.14-1.el9.x86_64 2024-05-27T07:12:07.536 INFO:teuthology.orchestra.run.smithi105.stdout: postgresql-private-libs-13.14-1.el9.x86_64 2024-05-27T07:12:07.536 INFO:teuthology.orchestra.run.smithi105.stdout: postgresql-server-13.14-1.el9.x86_64 2024-05-27T07:12:07.536 INFO:teuthology.orchestra.run.smithi105.stdout: protobuf-3.14.0-13.el9.x86_64 2024-05-27T07:12:07.536 INFO:teuthology.orchestra.run.smithi105.stdout: protobuf-compiler-3.14.0-13.el9.x86_64 2024-05-27T07:12:07.536 INFO:teuthology.orchestra.run.smithi105.stdout: python3-asyncssh-2.13.2-5.el9.noarch 2024-05-27T07:12:07.537 INFO:teuthology.orchestra.run.smithi105.stdout: python3-attrs-20.3.0-7.el9.noarch 2024-05-27T07:12:07.537 INFO:teuthology.orchestra.run.smithi105.stdout: python3-babel-2.9.1-2.el9.noarch 2024-05-27T07:12:07.537 INFO:teuthology.orchestra.run.smithi105.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2024-05-27T07:12:07.537 INFO:teuthology.orchestra.run.smithi105.stdout: python3-cachetools-4.2.4-1.el9.noarch 2024-05-27T07:12:07.537 INFO:teuthology.orchestra.run.smithi105.stdout: python3-ceph-argparse-2:19.0.0-3872.g38f9d900.el9.x86_64 2024-05-27T07:12:07.537 INFO:teuthology.orchestra.run.smithi105.stdout: python3-ceph-common-2:19.0.0-3872.g38f9d900.el9.x86_64 2024-05-27T07:12:07.537 INFO:teuthology.orchestra.run.smithi105.stdout: python3-cephfs-2:19.0.0-3872.g38f9d900.el9.x86_64 2024-05-27T07:12:07.537 INFO:teuthology.orchestra.run.smithi105.stdout: python3-certifi-2021.10.8-2.el9.noarch 2024-05-27T07:12:07.537 INFO:teuthology.orchestra.run.smithi105.stdout: python3-cffi-1.14.5-5.el9.x86_64 2024-05-27T07:12:07.537 INFO:teuthology.orchestra.run.smithi105.stdout: python3-cheroot-8.6.0-4.el9.noarch 2024-05-27T07:12:07.537 INFO:teuthology.orchestra.run.smithi105.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2024-05-27T07:12:07.537 INFO:teuthology.orchestra.run.smithi105.stdout: python3-cmd2-2.4.2-2.el9.noarch 2024-05-27T07:12:07.537 INFO:teuthology.orchestra.run.smithi105.stdout: python3-colorama-0.4.6-3.el9.noarch 2024-05-27T07:12:07.538 INFO:teuthology.orchestra.run.smithi105.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2024-05-27T07:12:07.538 INFO:teuthology.orchestra.run.smithi105.stdout: python3-devel-3.9.18-3.el9.x86_64 2024-05-27T07:12:07.538 INFO:teuthology.orchestra.run.smithi105.stdout: python3-google-auth-1:2.29.0-1.el9.noarch 2024-05-27T07:12:07.538 INFO:teuthology.orchestra.run.smithi105.stdout: python3-grpcio-1.46.7-10.el9.x86_64 2024-05-27T07:12:07.538 INFO:teuthology.orchestra.run.smithi105.stdout: python3-grpcio-tools-1.46.7-10.el9.x86_64 2024-05-27T07:12:07.538 INFO:teuthology.orchestra.run.smithi105.stdout: python3-jaraco-8.2.1-3.el9.noarch 2024-05-27T07:12:07.538 INFO:teuthology.orchestra.run.smithi105.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2024-05-27T07:12:07.538 INFO:teuthology.orchestra.run.smithi105.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2024-05-27T07:12:07.538 INFO:teuthology.orchestra.run.smithi105.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2024-05-27T07:12:07.538 INFO:teuthology.orchestra.run.smithi105.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2024-05-27T07:12:07.538 INFO:teuthology.orchestra.run.smithi105.stdout: python3-jinja2-2.11.3-6.el9.noarch 2024-05-27T07:12:07.538 INFO:teuthology.orchestra.run.smithi105.stdout: python3-jsonpatch-1.21-16.el9.noarch 2024-05-27T07:12:07.539 INFO:teuthology.orchestra.run.smithi105.stdout: python3-jsonpointer-2.0-4.el9.noarch 2024-05-27T07:12:07.539 INFO:teuthology.orchestra.run.smithi105.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2024-05-27T07:12:07.539 INFO:teuthology.orchestra.run.smithi105.stdout: python3-logutils-0.3.5-21.el9.noarch 2024-05-27T07:12:07.539 INFO:teuthology.orchestra.run.smithi105.stdout: python3-mako-1.1.4-6.el9.noarch 2024-05-27T07:12:07.539 INFO:teuthology.orchestra.run.smithi105.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2024-05-27T07:12:07.539 INFO:teuthology.orchestra.run.smithi105.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2024-05-27T07:12:07.539 INFO:teuthology.orchestra.run.smithi105.stdout: python3-natsort-7.1.1-5.el9.noarch 2024-05-27T07:12:07.539 INFO:teuthology.orchestra.run.smithi105.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2024-05-27T07:12:07.539 INFO:teuthology.orchestra.run.smithi105.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2024-05-27T07:12:07.539 INFO:teuthology.orchestra.run.smithi105.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2024-05-27T07:12:07.539 INFO:teuthology.orchestra.run.smithi105.stdout: python3-packaging-20.9-5.el9.noarch 2024-05-27T07:12:07.539 INFO:teuthology.orchestra.run.smithi105.stdout: python3-pecan-1.4.2-3.el9.noarch 2024-05-27T07:12:07.539 INFO:teuthology.orchestra.run.smithi105.stdout: python3-ply-3.11-14.el9.noarch 2024-05-27T07:12:07.540 INFO:teuthology.orchestra.run.smithi105.stdout: python3-portend-3.1.0-2.el9.noarch 2024-05-27T07:12:07.540 INFO:teuthology.orchestra.run.smithi105.stdout: python3-prettytable-0.7.2-27.el9.noarch 2024-05-27T07:12:07.540 INFO:teuthology.orchestra.run.smithi105.stdout: python3-protobuf-3.14.0-13.el9.noarch 2024-05-27T07:12:07.540 INFO:teuthology.orchestra.run.smithi105.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2024-05-27T07:12:07.540 INFO:teuthology.orchestra.run.smithi105.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2024-05-27T07:12:07.540 INFO:teuthology.orchestra.run.smithi105.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2024-05-27T07:12:07.540 INFO:teuthology.orchestra.run.smithi105.stdout: python3-pycparser-2.20-6.el9.noarch 2024-05-27T07:12:07.540 INFO:teuthology.orchestra.run.smithi105.stdout: python3-pyperclip-1.8.2-2.el9.noarch 2024-05-27T07:12:07.540 INFO:teuthology.orchestra.run.smithi105.stdout: python3-pytz-2021.1-5.el9.noarch 2024-05-27T07:12:07.540 INFO:teuthology.orchestra.run.smithi105.stdout: python3-rados-2:19.0.0-3872.g38f9d900.el9.x86_64 2024-05-27T07:12:07.540 INFO:teuthology.orchestra.run.smithi105.stdout: python3-rbd-2:19.0.0-3872.g38f9d900.el9.x86_64 2024-05-27T07:12:07.540 INFO:teuthology.orchestra.run.smithi105.stdout: python3-repoze-lru-0.7-16.el9.noarch 2024-05-27T07:12:07.541 INFO:teuthology.orchestra.run.smithi105.stdout: python3-requests-2.25.1-8.el9.noarch 2024-05-27T07:12:07.541 INFO:teuthology.orchestra.run.smithi105.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2024-05-27T07:12:07.541 INFO:teuthology.orchestra.run.smithi105.stdout: python3-rgw-2:19.0.0-3872.g38f9d900.el9.x86_64 2024-05-27T07:12:07.541 INFO:teuthology.orchestra.run.smithi105.stdout: python3-routes-2.5.1-5.el9.noarch 2024-05-27T07:12:07.541 INFO:teuthology.orchestra.run.smithi105.stdout: python3-rsa-4.9-2.el9.noarch 2024-05-27T07:12:07.541 INFO:teuthology.orchestra.run.smithi105.stdout: python3-scipy-1.6.2-8.el9.x86_64 2024-05-27T07:12:07.541 INFO:teuthology.orchestra.run.smithi105.stdout: python3-tempora-5.0.0-2.el9.noarch 2024-05-27T07:12:07.541 INFO:teuthology.orchestra.run.smithi105.stdout: python3-toml-0.10.2-6.el9.noarch 2024-05-27T07:12:07.541 INFO:teuthology.orchestra.run.smithi105.stdout: python3-typing-extensions-4.4.0-2.el9.noarch 2024-05-27T07:12:07.541 INFO:teuthology.orchestra.run.smithi105.stdout: python3-urllib3-1.26.5-5.el9.noarch 2024-05-27T07:12:07.541 INFO:teuthology.orchestra.run.smithi105.stdout: python3-wcwidth-0.2.5-8.el9.noarch 2024-05-27T07:12:07.541 INFO:teuthology.orchestra.run.smithi105.stdout: python3-webob-1.8.7-6.el9.noarch 2024-05-27T07:12:07.541 INFO:teuthology.orchestra.run.smithi105.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2024-05-27T07:12:07.542 INFO:teuthology.orchestra.run.smithi105.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2024-05-27T07:12:07.542 INFO:teuthology.orchestra.run.smithi105.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2024-05-27T07:12:07.542 INFO:teuthology.orchestra.run.smithi105.stdout: qatlib-24.02.0-1.el9.x86_64 2024-05-27T07:12:07.542 INFO:teuthology.orchestra.run.smithi105.stdout: qatlib-service-24.02.0-1.el9.x86_64 2024-05-27T07:12:07.542 INFO:teuthology.orchestra.run.smithi105.stdout: qatzip-libs-1.2.0-1.el9.x86_64 2024-05-27T07:12:07.542 INFO:teuthology.orchestra.run.smithi105.stdout: rbd-fuse-2:19.0.0-3872.g38f9d900.el9.x86_64 2024-05-27T07:12:07.542 INFO:teuthology.orchestra.run.smithi105.stdout: rbd-mirror-2:19.0.0-3872.g38f9d900.el9.x86_64 2024-05-27T07:12:07.542 INFO:teuthology.orchestra.run.smithi105.stdout: rbd-nbd-2:19.0.0-3872.g38f9d900.el9.x86_64 2024-05-27T07:12:07.542 INFO:teuthology.orchestra.run.smithi105.stdout: re2-1:20211101-3.el9.x86_64 2024-05-27T07:12:07.542 INFO:teuthology.orchestra.run.smithi105.stdout: socat-1.7.4.1-5.el9.x86_64 2024-05-27T07:12:07.542 INFO:teuthology.orchestra.run.smithi105.stdout: thrift-0.15.0-2.el9.x86_64 2024-05-27T07:12:07.542 INFO:teuthology.orchestra.run.smithi105.stdout: uuid-1.6.2-55.el9.x86_64 2024-05-27T07:12:07.542 INFO:teuthology.orchestra.run.smithi105.stdout: xfsprogs-devel-6.3.0-1.el9.x86_64 2024-05-27T07:12:07.543 INFO:teuthology.orchestra.run.smithi105.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2024-05-27T07:12:07.543 INFO:teuthology.orchestra.run.smithi105.stdout: zlib-devel-1.2.11-41.el9.x86_64 2024-05-27T07:12:07.543 INFO:teuthology.orchestra.run.smithi105.stdout: 2024-05-27T07:12:07.543 INFO:teuthology.orchestra.run.smithi105.stdout:Complete! 2024-05-27T07:12:07.546 INFO:teuthology.orchestra.run.smithi052.stdout: Cleanup : NetworkManager-team-1:1.47.4-1.el9.x86_64 166/170 2024-05-27T07:12:07.552 INFO:teuthology.orchestra.run.smithi052.stdout: Running scriptlet: NetworkManager-1:1.47.4-1.el9.x86_64 167/170 2024-05-27T07:12:07.552 INFO:teuthology.orchestra.run.smithi052.stdout: Cleanup : NetworkManager-1:1.47.4-1.el9.x86_64 167/170 2024-05-27T07:12:07.739 INFO:teuthology.orchestra.run.smithi052.stdout: Running scriptlet: NetworkManager-1:1.47.4-1.el9.x86_64 167/170 2024-05-27T07:12:07.740 INFO:teuthology.orchestra.run.smithi052.stdout: Cleanup : NetworkManager-libnm-1:1.47.4-1.el9.x86_64 168/170 2024-05-27T07:12:07.832 INFO:teuthology.orchestra.run.smithi052.stdout: Running scriptlet: NetworkManager-libnm-1:1.47.4-1.el9.x86_64 168/170 2024-05-27T07:12:07.856 INFO:teuthology.orchestra.run.smithi052.stdout: Cleanup : openssl-libs-1:3.0.7-27.el9.x86_64 169/170 2024-05-27T07:12:07.925 INFO:teuthology.orchestra.run.smithi052.stdout: Cleanup : elfutils-libelf-0.190-2.el9.x86_64 170/170 2024-05-27T07:12:07.984 DEBUG:teuthology.parallel:result is None 2024-05-27T07:12:13.516 INFO:teuthology.orchestra.run.smithi052.stdout: Running scriptlet: elfutils-libelf-0.190-2.el9.x86_64 170/170 2024-05-27T07:12:13.517 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : ceph-2:19.0.0-3872.g38f9d900.el9.x86_64 1/170 2024-05-27T07:12:13.517 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : ceph-base-2:19.0.0-3872.g38f9d900.el9.x86_64 2/170 2024-05-27T07:12:13.517 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : ceph-common-2:19.0.0-3872.g38f9d900.el9.x86_64 3/170 2024-05-27T07:12:13.517 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : ceph-fuse-2:19.0.0-3872.g38f9d900.el9.x86_64 4/170 2024-05-27T07:12:13.517 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : ceph-immutable-object-cache-2:19.0.0-3872.g38f9d 5/170 2024-05-27T07:12:13.517 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : ceph-mds-2:19.0.0-3872.g38f9d900.el9.x86_64 6/170 2024-05-27T07:12:13.517 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : ceph-mgr-2:19.0.0-3872.g38f9d900.el9.x86_64 7/170 2024-05-27T07:12:13.517 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : ceph-mon-2:19.0.0-3872.g38f9d900.el9.x86_64 8/170 2024-05-27T07:12:13.517 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : ceph-osd-2:19.0.0-3872.g38f9d900.el9.x86_64 9/170 2024-05-27T07:12:13.517 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : ceph-radosgw-2:19.0.0-3872.g38f9d900.el9.x86_64 10/170 2024-05-27T07:12:13.517 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : ceph-selinux-2:19.0.0-3872.g38f9d900.el9.x86_64 11/170 2024-05-27T07:12:13.517 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : ceph-test-2:19.0.0-3872.g38f9d900.el9.x86_64 12/170 2024-05-27T07:12:13.517 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : cephfs-mirror-2:19.0.0-3872.g38f9d900.el9.x86_64 13/170 2024-05-27T07:12:13.518 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : cephfs-shell-2:19.0.0-3872.g38f9d900.el9.x86_64 14/170 2024-05-27T07:12:13.518 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : libcephfs-devel-2:19.0.0-3872.g38f9d900.el9.x86_ 15/170 2024-05-27T07:12:13.518 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : libcephfs2-2:19.0.0-3872.g38f9d900.el9.x86_64 16/170 2024-05-27T07:12:13.518 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : libcephsqlite-2:19.0.0-3872.g38f9d900.el9.x86_64 17/170 2024-05-27T07:12:13.518 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : librados-devel-2:19.0.0-3872.g38f9d900.el9.x86_6 18/170 2024-05-27T07:12:13.518 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : libradosstriper1-2:19.0.0-3872.g38f9d900.el9.x86 19/170 2024-05-27T07:12:13.518 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : librgw2-2:19.0.0-3872.g38f9d900.el9.x86_64 20/170 2024-05-27T07:12:13.518 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : python3-ceph-argparse-2:19.0.0-3872.g38f9d900.el 21/170 2024-05-27T07:12:13.518 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : python3-ceph-common-2:19.0.0-3872.g38f9d900.el9. 22/170 2024-05-27T07:12:13.518 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : python3-cephfs-2:19.0.0-3872.g38f9d900.el9.x86_6 23/170 2024-05-27T07:12:13.518 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : python3-rados-2:19.0.0-3872.g38f9d900.el9.x86_64 24/170 2024-05-27T07:12:13.518 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : python3-rbd-2:19.0.0-3872.g38f9d900.el9.x86_64 25/170 2024-05-27T07:12:13.519 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : python3-rgw-2:19.0.0-3872.g38f9d900.el9.x86_64 26/170 2024-05-27T07:12:13.519 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : rbd-fuse-2:19.0.0-3872.g38f9d900.el9.x86_64 27/170 2024-05-27T07:12:13.519 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : rbd-mirror-2:19.0.0-3872.g38f9d900.el9.x86_64 28/170 2024-05-27T07:12:13.519 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : rbd-nbd-2:19.0.0-3872.g38f9d900.el9.x86_64 29/170 2024-05-27T07:12:13.519 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : ceph-grafana-dashboards-2:19.0.0-3872.g38f9d900. 30/170 2024-05-27T07:12:13.519 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : ceph-mgr-cephadm-2:19.0.0-3872.g38f9d900.el9.noa 31/170 2024-05-27T07:12:13.519 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : ceph-mgr-dashboard-2:19.0.0-3872.g38f9d900.el9.n 32/170 2024-05-27T07:12:13.519 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : ceph-mgr-diskprediction-local-2:19.0.0-3872.g38f 33/170 2024-05-27T07:12:13.519 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : ceph-mgr-modules-core-2:19.0.0-3872.g38f9d900.el 34/170 2024-05-27T07:12:13.519 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : ceph-mgr-rook-2:19.0.0-3872.g38f9d900.el9.noarch 35/170 2024-05-27T07:12:13.519 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : ceph-prometheus-alerts-2:19.0.0-3872.g38f9d900.e 36/170 2024-05-27T07:12:13.519 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : ceph-volume-2:19.0.0-3872.g38f9d900.el9.noarch 37/170 2024-05-27T07:12:13.519 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : cephadm-2:19.0.0-3872.g38f9d900.el9.noarch 38/170 2024-05-27T07:12:13.520 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : cephfs-top-2:19.0.0-3872.g38f9d900.el9.noarch 39/170 2024-05-27T07:12:13.520 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : fuse-2.9.9-15.el9.x86_64 40/170 2024-05-27T07:12:13.520 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : libgfortran-11.4.1-3.el9.x86_64 41/170 2024-05-27T07:12:13.520 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : libquadmath-11.4.1-3.el9.x86_64 42/170 2024-05-27T07:12:13.520 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 43/170 2024-05-27T07:12:13.520 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 44/170 2024-05-27T07:12:13.520 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : python3-ply-3.11-14.el9.noarch 45/170 2024-05-27T07:12:13.520 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 46/170 2024-05-27T07:12:13.520 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : python3-requests-2.25.1-8.el9.noarch 47/170 2024-05-27T07:12:13.520 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : python3-urllib3-1.26.5-5.el9.noarch 48/170 2024-05-27T07:12:13.520 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : automake-1.16.2-8.el9.noarch 49/170 2024-05-27T07:12:13.520 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : bison-3.7.4-5.el9.x86_64 50/170 2024-05-27T07:12:13.521 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : boost-program-options-1.75.0-8.el9.x86_64 51/170 2024-05-27T07:12:13.521 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : elfutils-libelf-devel-0.191-4.el9.x86_64 52/170 2024-05-27T07:12:13.521 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : flex-2.6.4-9.el9.x86_64 53/170 2024-05-27T07:12:13.521 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 54/170 2024-05-27T07:12:13.521 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 55/170 2024-05-27T07:12:13.521 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 56/170 2024-05-27T07:12:13.521 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : libaio-devel-0.3.111-13.el9.x86_64 57/170 2024-05-27T07:12:13.521 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 58/170 2024-05-27T07:12:13.521 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 59/170 2024-05-27T07:12:13.521 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 60/170 2024-05-27T07:12:13.521 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : libtool-2.4.6-45.el9.x86_64 61/170 2024-05-27T07:12:13.521 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : libuuid-devel-2.37.4-18.el9.x86_64 62/170 2024-05-27T07:12:13.521 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : libzstd-devel-1.5.1-2.el9.x86_64 63/170 2024-05-27T07:12:13.522 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 64/170 2024-05-27T07:12:13.522 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : lua-5.4.4-4.el9.x86_64 65/170 2024-05-27T07:12:13.522 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : openblas-0.3.21-2.el9.x86_64 66/170 2024-05-27T07:12:13.522 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : openblas-openmp-0.3.21-2.el9.x86_64 67/170 2024-05-27T07:12:13.522 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : openssl-devel-1:3.2.1-1.el9.x86_64 68/170 2024-05-27T07:12:13.522 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : postgresql-13.14-1.el9.x86_64 69/170 2024-05-27T07:12:13.522 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : postgresql-contrib-13.14-1.el9.x86_64 70/170 2024-05-27T07:12:13.522 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : postgresql-private-libs-13.14-1.el9.x86_64 71/170 2024-05-27T07:12:13.522 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : postgresql-server-13.14-1.el9.x86_64 72/170 2024-05-27T07:12:13.522 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : protobuf-3.14.0-13.el9.x86_64 73/170 2024-05-27T07:12:13.522 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : python3-attrs-20.3.0-7.el9.noarch 74/170 2024-05-27T07:12:13.522 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 75/170 2024-05-27T07:12:13.523 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : python3-colorama-0.4.6-3.el9.noarch 76/170 2024-05-27T07:12:13.523 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : python3-devel-3.9.18-3.el9.x86_64 77/170 2024-05-27T07:12:13.523 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : python3-jinja2-2.11.3-6.el9.noarch 78/170 2024-05-27T07:12:13.523 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 79/170 2024-05-27T07:12:13.523 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 80/170 2024-05-27T07:12:13.523 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 81/170 2024-05-27T07:12:13.523 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 82/170 2024-05-27T07:12:13.523 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 83/170 2024-05-27T07:12:13.523 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 84/170 2024-05-27T07:12:13.523 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 85/170 2024-05-27T07:12:13.523 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : python3-packaging-20.9-5.el9.noarch 86/170 2024-05-27T07:12:13.523 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 87/170 2024-05-27T07:12:13.523 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : python3-protobuf-3.14.0-13.el9.noarch 88/170 2024-05-27T07:12:13.523 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 89/170 2024-05-27T07:12:13.524 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 90/170 2024-05-27T07:12:13.524 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 91/170 2024-05-27T07:12:13.524 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 92/170 2024-05-27T07:12:13.524 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 93/170 2024-05-27T07:12:13.524 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 94/170 2024-05-27T07:12:13.524 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : python3-wcwidth-0.2.5-8.el9.noarch 95/170 2024-05-27T07:12:13.524 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : qatlib-24.02.0-1.el9.x86_64 96/170 2024-05-27T07:12:13.524 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : qatlib-service-24.02.0-1.el9.x86_64 97/170 2024-05-27T07:12:13.524 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : qatzip-libs-1.2.0-1.el9.x86_64 98/170 2024-05-27T07:12:13.524 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : socat-1.7.4.1-5.el9.x86_64 99/170 2024-05-27T07:12:13.524 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : uuid-1.6.2-55.el9.x86_64 100/170 2024-05-27T07:12:13.524 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : xfsprogs-devel-6.3.0-1.el9.x86_64 101/170 2024-05-27T07:12:13.525 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 102/170 2024-05-27T07:12:13.525 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : zlib-devel-1.2.11-41.el9.x86_64 103/170 2024-05-27T07:12:13.525 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : lua-devel-5.4.4-4.el9.x86_64 104/170 2024-05-27T07:12:13.525 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : protobuf-compiler-3.14.0-13.el9.x86_64 105/170 2024-05-27T07:12:13.525 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : abseil-cpp-20211102.0-3.el9.x86_64 106/170 2024-05-27T07:12:13.525 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : gperftools-libs-2.9.1-2.el9.x86_64 107/170 2024-05-27T07:12:13.525 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : grpc-data-1.46.7-10.el9.noarch 108/170 2024-05-27T07:12:13.525 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : libarrow-9.0.0-11.el9.x86_64 109/170 2024-05-27T07:12:13.525 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : libarrow-doc-9.0.0-11.el9.noarch 110/170 2024-05-27T07:12:13.525 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 111/170 2024-05-27T07:12:13.525 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 112/170 2024-05-27T07:12:13.525 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : luarocks-3.9.2-5.el9.noarch 113/170 2024-05-27T07:12:13.525 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : parquet-libs-9.0.0-11.el9.x86_64 114/170 2024-05-27T07:12:13.525 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : python3-asyncssh-2.13.2-5.el9.noarch 115/170 2024-05-27T07:12:13.526 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 116/170 2024-05-27T07:12:13.526 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 117/170 2024-05-27T07:12:13.526 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : python3-certifi-2021.10.8-2.el9.noarch 118/170 2024-05-27T07:12:13.526 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 119/170 2024-05-27T07:12:13.526 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 120/170 2024-05-27T07:12:13.526 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : python3-cmd2-2.4.2-2.el9.noarch 121/170 2024-05-27T07:12:13.526 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : python3-google-auth-1:2.29.0-1.el9.noarch 122/170 2024-05-27T07:12:13.526 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : python3-grpcio-1.46.7-10.el9.x86_64 123/170 2024-05-27T07:12:13.526 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : python3-grpcio-tools-1.46.7-10.el9.x86_64 124/170 2024-05-27T07:12:13.526 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 125/170 2024-05-27T07:12:13.526 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 126/170 2024-05-27T07:12:13.526 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 127/170 2024-05-27T07:12:13.527 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 128/170 2024-05-27T07:12:13.527 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 129/170 2024-05-27T07:12:13.527 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 130/170 2024-05-27T07:12:13.527 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 131/170 2024-05-27T07:12:13.527 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 132/170 2024-05-27T07:12:13.527 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 133/170 2024-05-27T07:12:13.527 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 134/170 2024-05-27T07:12:13.527 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 135/170 2024-05-27T07:12:13.527 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 136/170 2024-05-27T07:12:13.527 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : python3-pyperclip-1.8.2-2.el9.noarch 137/170 2024-05-27T07:12:13.527 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 138/170 2024-05-27T07:12:13.527 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 139/170 2024-05-27T07:12:13.527 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 140/170 2024-05-27T07:12:13.528 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 141/170 2024-05-27T07:12:13.528 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : python3-typing-extensions-4.4.0-2.el9.noarch 142/170 2024-05-27T07:12:13.528 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : python3-webob-1.8.7-6.el9.noarch 143/170 2024-05-27T07:12:13.528 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 144/170 2024-05-27T07:12:13.528 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 145/170 2024-05-27T07:12:13.528 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 146/170 2024-05-27T07:12:13.528 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : re2-1:20211101-3.el9.x86_64 147/170 2024-05-27T07:12:13.528 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 148/170 2024-05-27T07:12:13.528 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : librados2-2:19.0.0-3872.g38f9d900.el9.x86_64 149/170 2024-05-27T07:12:13.528 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 150/170 2024-05-27T07:12:13.528 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : librbd1-2:19.0.0-3872.g38f9d900.el9.x86_64 151/170 2024-05-27T07:12:13.528 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 152/170 2024-05-27T07:12:13.528 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : NetworkManager-1:1.47.90-1.el9.x86_64 153/170 2024-05-27T07:12:13.529 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : NetworkManager-1:1.47.4-1.el9.x86_64 154/170 2024-05-27T07:12:13.529 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : NetworkManager-libnm-1:1.47.90-1.el9.x86_64 155/170 2024-05-27T07:12:13.529 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : NetworkManager-libnm-1:1.47.4-1.el9.x86_64 156/170 2024-05-27T07:12:13.529 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : NetworkManager-team-1:1.47.90-1.el9.x86_64 157/170 2024-05-27T07:12:13.529 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : NetworkManager-team-1:1.47.4-1.el9.x86_64 158/170 2024-05-27T07:12:13.529 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : NetworkManager-tui-1:1.47.90-1.el9.x86_64 159/170 2024-05-27T07:12:13.529 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : NetworkManager-tui-1:1.47.4-1.el9.x86_64 160/170 2024-05-27T07:12:13.529 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : elfutils-debuginfod-client-0.191-4.el9.x86_64 161/170 2024-05-27T07:12:13.529 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : elfutils-debuginfod-client-0.190-2.el9.x86_64 162/170 2024-05-27T07:12:13.529 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : elfutils-libelf-0.191-4.el9.x86_64 163/170 2024-05-27T07:12:13.529 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : elfutils-libelf-0.190-2.el9.x86_64 164/170 2024-05-27T07:12:13.529 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : elfutils-libs-0.191-4.el9.x86_64 165/170 2024-05-27T07:12:13.529 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : elfutils-libs-0.190-2.el9.x86_64 166/170 2024-05-27T07:12:13.529 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : openssl-1:3.2.1-1.el9.x86_64 167/170 2024-05-27T07:12:13.530 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : openssl-1:3.0.7-27.el9.x86_64 168/170 2024-05-27T07:12:13.530 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : openssl-libs-1:3.2.1-1.el9.x86_64 169/170 2024-05-27T07:12:16.095 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : openssl-libs-1:3.0.7-27.el9.x86_64 170/170 2024-05-27T07:12:16.095 INFO:teuthology.orchestra.run.smithi052.stdout: 2024-05-27T07:12:16.095 INFO:teuthology.orchestra.run.smithi052.stdout:Upgraded: 2024-05-27T07:12:16.095 INFO:teuthology.orchestra.run.smithi052.stdout: NetworkManager-1:1.47.90-1.el9.x86_64 2024-05-27T07:12:16.096 INFO:teuthology.orchestra.run.smithi052.stdout: NetworkManager-libnm-1:1.47.90-1.el9.x86_64 2024-05-27T07:12:16.096 INFO:teuthology.orchestra.run.smithi052.stdout: NetworkManager-team-1:1.47.90-1.el9.x86_64 2024-05-27T07:12:16.096 INFO:teuthology.orchestra.run.smithi052.stdout: NetworkManager-tui-1:1.47.90-1.el9.x86_64 2024-05-27T07:12:16.096 INFO:teuthology.orchestra.run.smithi052.stdout: elfutils-debuginfod-client-0.191-4.el9.x86_64 2024-05-27T07:12:16.096 INFO:teuthology.orchestra.run.smithi052.stdout: elfutils-libelf-0.191-4.el9.x86_64 2024-05-27T07:12:16.096 INFO:teuthology.orchestra.run.smithi052.stdout: elfutils-libs-0.191-4.el9.x86_64 2024-05-27T07:12:16.096 INFO:teuthology.orchestra.run.smithi052.stdout: librados2-2:19.0.0-3872.g38f9d900.el9.x86_64 2024-05-27T07:12:16.096 INFO:teuthology.orchestra.run.smithi052.stdout: librbd1-2:19.0.0-3872.g38f9d900.el9.x86_64 2024-05-27T07:12:16.096 INFO:teuthology.orchestra.run.smithi052.stdout: openssl-1:3.2.1-1.el9.x86_64 2024-05-27T07:12:16.096 INFO:teuthology.orchestra.run.smithi052.stdout: openssl-libs-1:3.2.1-1.el9.x86_64 2024-05-27T07:12:16.096 INFO:teuthology.orchestra.run.smithi052.stdout:Installed: 2024-05-27T07:12:16.096 INFO:teuthology.orchestra.run.smithi052.stdout: abseil-cpp-20211102.0-3.el9.x86_64 2024-05-27T07:12:16.096 INFO:teuthology.orchestra.run.smithi052.stdout: automake-1.16.2-8.el9.noarch 2024-05-27T07:12:16.097 INFO:teuthology.orchestra.run.smithi052.stdout: bison-3.7.4-5.el9.x86_64 2024-05-27T07:12:16.097 INFO:teuthology.orchestra.run.smithi052.stdout: boost-program-options-1.75.0-8.el9.x86_64 2024-05-27T07:12:16.097 INFO:teuthology.orchestra.run.smithi052.stdout: ceph-2:19.0.0-3872.g38f9d900.el9.x86_64 2024-05-27T07:12:16.097 INFO:teuthology.orchestra.run.smithi052.stdout: ceph-base-2:19.0.0-3872.g38f9d900.el9.x86_64 2024-05-27T07:12:16.097 INFO:teuthology.orchestra.run.smithi052.stdout: ceph-common-2:19.0.0-3872.g38f9d900.el9.x86_64 2024-05-27T07:12:16.097 INFO:teuthology.orchestra.run.smithi052.stdout: ceph-fuse-2:19.0.0-3872.g38f9d900.el9.x86_64 2024-05-27T07:12:16.097 INFO:teuthology.orchestra.run.smithi052.stdout: ceph-grafana-dashboards-2:19.0.0-3872.g38f9d900.el9.noarch 2024-05-27T07:12:16.097 INFO:teuthology.orchestra.run.smithi052.stdout: ceph-immutable-object-cache-2:19.0.0-3872.g38f9d900.el9.x86_64 2024-05-27T07:12:16.097 INFO:teuthology.orchestra.run.smithi052.stdout: ceph-mds-2:19.0.0-3872.g38f9d900.el9.x86_64 2024-05-27T07:12:16.097 INFO:teuthology.orchestra.run.smithi052.stdout: ceph-mgr-2:19.0.0-3872.g38f9d900.el9.x86_64 2024-05-27T07:12:16.097 INFO:teuthology.orchestra.run.smithi052.stdout: ceph-mgr-cephadm-2:19.0.0-3872.g38f9d900.el9.noarch 2024-05-27T07:12:16.097 INFO:teuthology.orchestra.run.smithi052.stdout: ceph-mgr-dashboard-2:19.0.0-3872.g38f9d900.el9.noarch 2024-05-27T07:12:16.098 INFO:teuthology.orchestra.run.smithi052.stdout: ceph-mgr-diskprediction-local-2:19.0.0-3872.g38f9d900.el9.noarch 2024-05-27T07:12:16.098 INFO:teuthology.orchestra.run.smithi052.stdout: ceph-mgr-modules-core-2:19.0.0-3872.g38f9d900.el9.noarch 2024-05-27T07:12:16.098 INFO:teuthology.orchestra.run.smithi052.stdout: ceph-mgr-rook-2:19.0.0-3872.g38f9d900.el9.noarch 2024-05-27T07:12:16.098 INFO:teuthology.orchestra.run.smithi052.stdout: ceph-mon-2:19.0.0-3872.g38f9d900.el9.x86_64 2024-05-27T07:12:16.098 INFO:teuthology.orchestra.run.smithi052.stdout: ceph-osd-2:19.0.0-3872.g38f9d900.el9.x86_64 2024-05-27T07:12:16.098 INFO:teuthology.orchestra.run.smithi052.stdout: ceph-prometheus-alerts-2:19.0.0-3872.g38f9d900.el9.noarch 2024-05-27T07:12:16.098 INFO:teuthology.orchestra.run.smithi052.stdout: ceph-radosgw-2:19.0.0-3872.g38f9d900.el9.x86_64 2024-05-27T07:12:16.098 INFO:teuthology.orchestra.run.smithi052.stdout: ceph-selinux-2:19.0.0-3872.g38f9d900.el9.x86_64 2024-05-27T07:12:16.098 INFO:teuthology.orchestra.run.smithi052.stdout: ceph-test-2:19.0.0-3872.g38f9d900.el9.x86_64 2024-05-27T07:12:16.098 INFO:teuthology.orchestra.run.smithi052.stdout: ceph-volume-2:19.0.0-3872.g38f9d900.el9.noarch 2024-05-27T07:12:16.098 INFO:teuthology.orchestra.run.smithi052.stdout: cephadm-2:19.0.0-3872.g38f9d900.el9.noarch 2024-05-27T07:12:16.099 INFO:teuthology.orchestra.run.smithi052.stdout: cephfs-mirror-2:19.0.0-3872.g38f9d900.el9.x86_64 2024-05-27T07:12:16.099 INFO:teuthology.orchestra.run.smithi052.stdout: cephfs-shell-2:19.0.0-3872.g38f9d900.el9.x86_64 2024-05-27T07:12:16.099 INFO:teuthology.orchestra.run.smithi052.stdout: cephfs-top-2:19.0.0-3872.g38f9d900.el9.noarch 2024-05-27T07:12:16.099 INFO:teuthology.orchestra.run.smithi052.stdout: elfutils-libelf-devel-0.191-4.el9.x86_64 2024-05-27T07:12:16.099 INFO:teuthology.orchestra.run.smithi052.stdout: flex-2.6.4-9.el9.x86_64 2024-05-27T07:12:16.099 INFO:teuthology.orchestra.run.smithi052.stdout: flexiblas-3.0.4-8.el9.x86_64 2024-05-27T07:12:16.099 INFO:teuthology.orchestra.run.smithi052.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2024-05-27T07:12:16.099 INFO:teuthology.orchestra.run.smithi052.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2024-05-27T07:12:16.099 INFO:teuthology.orchestra.run.smithi052.stdout: fuse-2.9.9-15.el9.x86_64 2024-05-27T07:12:16.100 INFO:teuthology.orchestra.run.smithi052.stdout: gperftools-libs-2.9.1-2.el9.x86_64 2024-05-27T07:12:16.100 INFO:teuthology.orchestra.run.smithi052.stdout: grpc-data-1.46.7-10.el9.noarch 2024-05-27T07:12:16.100 INFO:teuthology.orchestra.run.smithi052.stdout: libaio-devel-0.3.111-13.el9.x86_64 2024-05-27T07:12:16.100 INFO:teuthology.orchestra.run.smithi052.stdout: libarrow-9.0.0-11.el9.x86_64 2024-05-27T07:12:16.100 INFO:teuthology.orchestra.run.smithi052.stdout: libarrow-doc-9.0.0-11.el9.noarch 2024-05-27T07:12:16.100 INFO:teuthology.orchestra.run.smithi052.stdout: libcephfs-devel-2:19.0.0-3872.g38f9d900.el9.x86_64 2024-05-27T07:12:16.100 INFO:teuthology.orchestra.run.smithi052.stdout: libcephfs2-2:19.0.0-3872.g38f9d900.el9.x86_64 2024-05-27T07:12:16.100 INFO:teuthology.orchestra.run.smithi052.stdout: libcephsqlite-2:19.0.0-3872.g38f9d900.el9.x86_64 2024-05-27T07:12:16.100 INFO:teuthology.orchestra.run.smithi052.stdout: libgfortran-11.4.1-3.el9.x86_64 2024-05-27T07:12:16.100 INFO:teuthology.orchestra.run.smithi052.stdout: liboath-2.6.7-2.el9.x86_64 2024-05-27T07:12:16.100 INFO:teuthology.orchestra.run.smithi052.stdout: libpmemobj-1.12.1-1.el9.x86_64 2024-05-27T07:12:16.100 INFO:teuthology.orchestra.run.smithi052.stdout: libquadmath-11.4.1-3.el9.x86_64 2024-05-27T07:12:16.100 INFO:teuthology.orchestra.run.smithi052.stdout: librabbitmq-0.11.0-7.el9.x86_64 2024-05-27T07:12:16.101 INFO:teuthology.orchestra.run.smithi052.stdout: librados-devel-2:19.0.0-3872.g38f9d900.el9.x86_64 2024-05-27T07:12:16.101 INFO:teuthology.orchestra.run.smithi052.stdout: libradosstriper1-2:19.0.0-3872.g38f9d900.el9.x86_64 2024-05-27T07:12:16.101 INFO:teuthology.orchestra.run.smithi052.stdout: librdkafka-1.6.1-102.el9.x86_64 2024-05-27T07:12:16.101 INFO:teuthology.orchestra.run.smithi052.stdout: librgw2-2:19.0.0-3872.g38f9d900.el9.x86_64 2024-05-27T07:12:16.101 INFO:teuthology.orchestra.run.smithi052.stdout: libtool-2.4.6-45.el9.x86_64 2024-05-27T07:12:16.101 INFO:teuthology.orchestra.run.smithi052.stdout: libunwind-1.6.2-1.el9.x86_64 2024-05-27T07:12:16.101 INFO:teuthology.orchestra.run.smithi052.stdout: libuuid-devel-2.37.4-18.el9.x86_64 2024-05-27T07:12:16.101 INFO:teuthology.orchestra.run.smithi052.stdout: libzstd-devel-1.5.1-2.el9.x86_64 2024-05-27T07:12:16.101 INFO:teuthology.orchestra.run.smithi052.stdout: lttng-ust-2.12.0-6.el9.x86_64 2024-05-27T07:12:16.101 INFO:teuthology.orchestra.run.smithi052.stdout: lua-5.4.4-4.el9.x86_64 2024-05-27T07:12:16.101 INFO:teuthology.orchestra.run.smithi052.stdout: lua-devel-5.4.4-4.el9.x86_64 2024-05-27T07:12:16.101 INFO:teuthology.orchestra.run.smithi052.stdout: luarocks-3.9.2-5.el9.noarch 2024-05-27T07:12:16.101 INFO:teuthology.orchestra.run.smithi052.stdout: openblas-0.3.21-2.el9.x86_64 2024-05-27T07:12:16.102 INFO:teuthology.orchestra.run.smithi052.stdout: openblas-openmp-0.3.21-2.el9.x86_64 2024-05-27T07:12:16.102 INFO:teuthology.orchestra.run.smithi052.stdout: openssl-devel-1:3.2.1-1.el9.x86_64 2024-05-27T07:12:16.102 INFO:teuthology.orchestra.run.smithi052.stdout: parquet-libs-9.0.0-11.el9.x86_64 2024-05-27T07:12:16.102 INFO:teuthology.orchestra.run.smithi052.stdout: postgresql-13.14-1.el9.x86_64 2024-05-27T07:12:16.102 INFO:teuthology.orchestra.run.smithi052.stdout: postgresql-contrib-13.14-1.el9.x86_64 2024-05-27T07:12:16.102 INFO:teuthology.orchestra.run.smithi052.stdout: postgresql-private-libs-13.14-1.el9.x86_64 2024-05-27T07:12:16.102 INFO:teuthology.orchestra.run.smithi052.stdout: postgresql-server-13.14-1.el9.x86_64 2024-05-27T07:12:16.102 INFO:teuthology.orchestra.run.smithi052.stdout: protobuf-3.14.0-13.el9.x86_64 2024-05-27T07:12:16.102 INFO:teuthology.orchestra.run.smithi052.stdout: protobuf-compiler-3.14.0-13.el9.x86_64 2024-05-27T07:12:16.102 INFO:teuthology.orchestra.run.smithi052.stdout: python3-asyncssh-2.13.2-5.el9.noarch 2024-05-27T07:12:16.102 INFO:teuthology.orchestra.run.smithi052.stdout: python3-attrs-20.3.0-7.el9.noarch 2024-05-27T07:12:16.103 INFO:teuthology.orchestra.run.smithi052.stdout: python3-babel-2.9.1-2.el9.noarch 2024-05-27T07:12:16.103 INFO:teuthology.orchestra.run.smithi052.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2024-05-27T07:12:16.103 INFO:teuthology.orchestra.run.smithi052.stdout: python3-cachetools-4.2.4-1.el9.noarch 2024-05-27T07:12:16.103 INFO:teuthology.orchestra.run.smithi052.stdout: python3-ceph-argparse-2:19.0.0-3872.g38f9d900.el9.x86_64 2024-05-27T07:12:16.103 INFO:teuthology.orchestra.run.smithi052.stdout: python3-ceph-common-2:19.0.0-3872.g38f9d900.el9.x86_64 2024-05-27T07:12:16.103 INFO:teuthology.orchestra.run.smithi052.stdout: python3-cephfs-2:19.0.0-3872.g38f9d900.el9.x86_64 2024-05-27T07:12:16.103 INFO:teuthology.orchestra.run.smithi052.stdout: python3-certifi-2021.10.8-2.el9.noarch 2024-05-27T07:12:16.103 INFO:teuthology.orchestra.run.smithi052.stdout: python3-cffi-1.14.5-5.el9.x86_64 2024-05-27T07:12:16.103 INFO:teuthology.orchestra.run.smithi052.stdout: python3-cheroot-8.6.0-4.el9.noarch 2024-05-27T07:12:16.103 INFO:teuthology.orchestra.run.smithi052.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2024-05-27T07:12:16.103 INFO:teuthology.orchestra.run.smithi052.stdout: python3-cmd2-2.4.2-2.el9.noarch 2024-05-27T07:12:16.103 INFO:teuthology.orchestra.run.smithi052.stdout: python3-colorama-0.4.6-3.el9.noarch 2024-05-27T07:12:16.103 INFO:teuthology.orchestra.run.smithi052.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2024-05-27T07:12:16.104 INFO:teuthology.orchestra.run.smithi052.stdout: python3-devel-3.9.18-3.el9.x86_64 2024-05-27T07:12:16.104 INFO:teuthology.orchestra.run.smithi052.stdout: python3-google-auth-1:2.29.0-1.el9.noarch 2024-05-27T07:12:16.104 INFO:teuthology.orchestra.run.smithi052.stdout: python3-grpcio-1.46.7-10.el9.x86_64 2024-05-27T07:12:16.104 INFO:teuthology.orchestra.run.smithi052.stdout: python3-grpcio-tools-1.46.7-10.el9.x86_64 2024-05-27T07:12:16.104 INFO:teuthology.orchestra.run.smithi052.stdout: python3-jaraco-8.2.1-3.el9.noarch 2024-05-27T07:12:16.104 INFO:teuthology.orchestra.run.smithi052.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2024-05-27T07:12:16.104 INFO:teuthology.orchestra.run.smithi052.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2024-05-27T07:12:16.104 INFO:teuthology.orchestra.run.smithi052.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2024-05-27T07:12:16.104 INFO:teuthology.orchestra.run.smithi052.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2024-05-27T07:12:16.104 INFO:teuthology.orchestra.run.smithi052.stdout: python3-jinja2-2.11.3-6.el9.noarch 2024-05-27T07:12:16.104 INFO:teuthology.orchestra.run.smithi052.stdout: python3-jsonpatch-1.21-16.el9.noarch 2024-05-27T07:12:16.104 INFO:teuthology.orchestra.run.smithi052.stdout: python3-jsonpointer-2.0-4.el9.noarch 2024-05-27T07:12:16.104 INFO:teuthology.orchestra.run.smithi052.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2024-05-27T07:12:16.104 INFO:teuthology.orchestra.run.smithi052.stdout: python3-logutils-0.3.5-21.el9.noarch 2024-05-27T07:12:16.105 INFO:teuthology.orchestra.run.smithi052.stdout: python3-mako-1.1.4-6.el9.noarch 2024-05-27T07:12:16.105 INFO:teuthology.orchestra.run.smithi052.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2024-05-27T07:12:16.105 INFO:teuthology.orchestra.run.smithi052.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2024-05-27T07:12:16.105 INFO:teuthology.orchestra.run.smithi052.stdout: python3-natsort-7.1.1-5.el9.noarch 2024-05-27T07:12:16.105 INFO:teuthology.orchestra.run.smithi052.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2024-05-27T07:12:16.105 INFO:teuthology.orchestra.run.smithi052.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2024-05-27T07:12:16.105 INFO:teuthology.orchestra.run.smithi052.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2024-05-27T07:12:16.105 INFO:teuthology.orchestra.run.smithi052.stdout: python3-packaging-20.9-5.el9.noarch 2024-05-27T07:12:16.105 INFO:teuthology.orchestra.run.smithi052.stdout: python3-pecan-1.4.2-3.el9.noarch 2024-05-27T07:12:16.105 INFO:teuthology.orchestra.run.smithi052.stdout: python3-ply-3.11-14.el9.noarch 2024-05-27T07:12:16.105 INFO:teuthology.orchestra.run.smithi052.stdout: python3-portend-3.1.0-2.el9.noarch 2024-05-27T07:12:16.105 INFO:teuthology.orchestra.run.smithi052.stdout: python3-prettytable-0.7.2-27.el9.noarch 2024-05-27T07:12:16.105 INFO:teuthology.orchestra.run.smithi052.stdout: python3-protobuf-3.14.0-13.el9.noarch 2024-05-27T07:12:16.106 INFO:teuthology.orchestra.run.smithi052.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2024-05-27T07:12:16.106 INFO:teuthology.orchestra.run.smithi052.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2024-05-27T07:12:16.106 INFO:teuthology.orchestra.run.smithi052.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2024-05-27T07:12:16.106 INFO:teuthology.orchestra.run.smithi052.stdout: python3-pycparser-2.20-6.el9.noarch 2024-05-27T07:12:16.106 INFO:teuthology.orchestra.run.smithi052.stdout: python3-pyperclip-1.8.2-2.el9.noarch 2024-05-27T07:12:16.106 INFO:teuthology.orchestra.run.smithi052.stdout: python3-pytz-2021.1-5.el9.noarch 2024-05-27T07:12:16.106 INFO:teuthology.orchestra.run.smithi052.stdout: python3-rados-2:19.0.0-3872.g38f9d900.el9.x86_64 2024-05-27T07:12:16.106 INFO:teuthology.orchestra.run.smithi052.stdout: python3-rbd-2:19.0.0-3872.g38f9d900.el9.x86_64 2024-05-27T07:12:16.106 INFO:teuthology.orchestra.run.smithi052.stdout: python3-repoze-lru-0.7-16.el9.noarch 2024-05-27T07:12:16.106 INFO:teuthology.orchestra.run.smithi052.stdout: python3-requests-2.25.1-8.el9.noarch 2024-05-27T07:12:16.106 INFO:teuthology.orchestra.run.smithi052.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2024-05-27T07:12:16.106 INFO:teuthology.orchestra.run.smithi052.stdout: python3-rgw-2:19.0.0-3872.g38f9d900.el9.x86_64 2024-05-27T07:12:16.106 INFO:teuthology.orchestra.run.smithi052.stdout: python3-routes-2.5.1-5.el9.noarch 2024-05-27T07:12:16.107 INFO:teuthology.orchestra.run.smithi052.stdout: python3-rsa-4.9-2.el9.noarch 2024-05-27T07:12:16.107 INFO:teuthology.orchestra.run.smithi052.stdout: python3-scipy-1.6.2-8.el9.x86_64 2024-05-27T07:12:16.107 INFO:teuthology.orchestra.run.smithi052.stdout: python3-tempora-5.0.0-2.el9.noarch 2024-05-27T07:12:16.107 INFO:teuthology.orchestra.run.smithi052.stdout: python3-toml-0.10.2-6.el9.noarch 2024-05-27T07:12:16.107 INFO:teuthology.orchestra.run.smithi052.stdout: python3-typing-extensions-4.4.0-2.el9.noarch 2024-05-27T07:12:16.107 INFO:teuthology.orchestra.run.smithi052.stdout: python3-urllib3-1.26.5-5.el9.noarch 2024-05-27T07:12:16.107 INFO:teuthology.orchestra.run.smithi052.stdout: python3-wcwidth-0.2.5-8.el9.noarch 2024-05-27T07:12:16.107 INFO:teuthology.orchestra.run.smithi052.stdout: python3-webob-1.8.7-6.el9.noarch 2024-05-27T07:12:16.107 INFO:teuthology.orchestra.run.smithi052.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2024-05-27T07:12:16.107 INFO:teuthology.orchestra.run.smithi052.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2024-05-27T07:12:16.107 INFO:teuthology.orchestra.run.smithi052.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2024-05-27T07:12:16.107 INFO:teuthology.orchestra.run.smithi052.stdout: qatlib-24.02.0-1.el9.x86_64 2024-05-27T07:12:16.107 INFO:teuthology.orchestra.run.smithi052.stdout: qatlib-service-24.02.0-1.el9.x86_64 2024-05-27T07:12:16.107 INFO:teuthology.orchestra.run.smithi052.stdout: qatzip-libs-1.2.0-1.el9.x86_64 2024-05-27T07:12:16.107 INFO:teuthology.orchestra.run.smithi052.stdout: rbd-fuse-2:19.0.0-3872.g38f9d900.el9.x86_64 2024-05-27T07:12:16.108 INFO:teuthology.orchestra.run.smithi052.stdout: rbd-mirror-2:19.0.0-3872.g38f9d900.el9.x86_64 2024-05-27T07:12:16.108 INFO:teuthology.orchestra.run.smithi052.stdout: rbd-nbd-2:19.0.0-3872.g38f9d900.el9.x86_64 2024-05-27T07:12:16.108 INFO:teuthology.orchestra.run.smithi052.stdout: re2-1:20211101-3.el9.x86_64 2024-05-27T07:12:16.108 INFO:teuthology.orchestra.run.smithi052.stdout: socat-1.7.4.1-5.el9.x86_64 2024-05-27T07:12:16.108 INFO:teuthology.orchestra.run.smithi052.stdout: thrift-0.15.0-2.el9.x86_64 2024-05-27T07:12:16.108 INFO:teuthology.orchestra.run.smithi052.stdout: uuid-1.6.2-55.el9.x86_64 2024-05-27T07:12:16.108 INFO:teuthology.orchestra.run.smithi052.stdout: xfsprogs-devel-6.3.0-1.el9.x86_64 2024-05-27T07:12:16.108 INFO:teuthology.orchestra.run.smithi052.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2024-05-27T07:12:16.108 INFO:teuthology.orchestra.run.smithi052.stdout: zlib-devel-1.2.11-41.el9.x86_64 2024-05-27T07:12:16.108 INFO:teuthology.orchestra.run.smithi052.stdout: 2024-05-27T07:12:16.108 INFO:teuthology.orchestra.run.smithi052.stdout:Complete! 2024-05-27T07:12:16.551 DEBUG:teuthology.parallel:result is None 2024-05-27T07:12:16.552 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=38f9d900c83c63799fbdbe61acc9a11b0d3554a6 2024-05-27T07:12:16.700 DEBUG:teuthology.orchestra.run.smithi052:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-05-27T07:12:16.723 INFO:teuthology.orchestra.run.smithi052.stdout:19.0.0-3872.g38f9d900.el9 2024-05-27T07:12:16.724 INFO:teuthology.packaging:The installed version of ceph is 19.0.0-3872.g38f9d900.el9 2024-05-27T07:12:16.724 INFO:teuthology.task.install:The correct ceph version 19.0.0-3872.g38f9d900 is installed. 2024-05-27T07:12:16.726 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=38f9d900c83c63799fbdbe61acc9a11b0d3554a6 2024-05-27T07:12:16.868 DEBUG:teuthology.orchestra.run.smithi105:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-05-27T07:12:16.894 INFO:teuthology.orchestra.run.smithi105.stdout:19.0.0-3872.g38f9d900.el9 2024-05-27T07:12:16.894 INFO:teuthology.packaging:The installed version of ceph is 19.0.0-3872.g38f9d900.el9 2024-05-27T07:12:16.894 INFO:teuthology.task.install:The correct ceph version 19.0.0-3872.g38f9d900 is installed. 2024-05-27T07:12:16.896 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=38f9d900c83c63799fbdbe61acc9a11b0d3554a6 2024-05-27T07:12:17.124 DEBUG:teuthology.orchestra.run.smithi173:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-05-27T07:12:17.150 INFO:teuthology.orchestra.run.smithi173.stdout:19.0.0-3872.g38f9d900.el9 2024-05-27T07:12:17.150 INFO:teuthology.packaging:The installed version of ceph is 19.0.0-3872.g38f9d900.el9 2024-05-27T07:12:17.150 INFO:teuthology.task.install:The correct ceph version 19.0.0-3872.g38f9d900 is installed. 2024-05-27T07:12:17.152 INFO:teuthology.task.install.util:Shipping valgrind.supp... 2024-05-27T07:12:17.152 DEBUG:teuthology.orchestra.run.smithi052:> set -ex 2024-05-27T07:12:17.152 DEBUG:teuthology.orchestra.run.smithi052:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-05-27T07:12:17.181 DEBUG:teuthology.orchestra.run.smithi105:> set -ex 2024-05-27T07:12:17.181 DEBUG:teuthology.orchestra.run.smithi105:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-05-27T07:12:17.212 DEBUG:teuthology.orchestra.run.smithi173:> set -ex 2024-05-27T07:12:17.212 DEBUG:teuthology.orchestra.run.smithi173:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-05-27T07:12:17.242 INFO:teuthology.task.install.util:Shipping 'daemon-helper'... 2024-05-27T07:12:17.242 DEBUG:teuthology.orchestra.run.smithi052:> set -ex 2024-05-27T07:12:17.242 DEBUG:teuthology.orchestra.run.smithi052:> sudo dd of=/usr/bin/daemon-helper 2024-05-27T07:12:17.268 DEBUG:teuthology.orchestra.run.smithi052:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-05-27T07:12:17.334 DEBUG:teuthology.orchestra.run.smithi105:> set -ex 2024-05-27T07:12:17.335 DEBUG:teuthology.orchestra.run.smithi105:> sudo dd of=/usr/bin/daemon-helper 2024-05-27T07:12:17.362 DEBUG:teuthology.orchestra.run.smithi105:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-05-27T07:12:17.428 DEBUG:teuthology.orchestra.run.smithi173:> set -ex 2024-05-27T07:12:17.428 DEBUG:teuthology.orchestra.run.smithi173:> sudo dd of=/usr/bin/daemon-helper 2024-05-27T07:12:17.457 DEBUG:teuthology.orchestra.run.smithi173:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-05-27T07:12:17.525 INFO:teuthology.task.install.util:Shipping 'adjust-ulimits'... 2024-05-27T07:12:17.525 DEBUG:teuthology.orchestra.run.smithi052:> set -ex 2024-05-27T07:12:17.525 DEBUG:teuthology.orchestra.run.smithi052:> sudo dd of=/usr/bin/adjust-ulimits 2024-05-27T07:12:17.552 DEBUG:teuthology.orchestra.run.smithi052:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-05-27T07:12:17.618 DEBUG:teuthology.orchestra.run.smithi105:> set -ex 2024-05-27T07:12:17.618 DEBUG:teuthology.orchestra.run.smithi105:> sudo dd of=/usr/bin/adjust-ulimits 2024-05-27T07:12:17.645 DEBUG:teuthology.orchestra.run.smithi105:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-05-27T07:12:17.711 DEBUG:teuthology.orchestra.run.smithi173:> set -ex 2024-05-27T07:12:17.711 DEBUG:teuthology.orchestra.run.smithi173:> sudo dd of=/usr/bin/adjust-ulimits 2024-05-27T07:12:17.735 DEBUG:teuthology.orchestra.run.smithi173:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-05-27T07:12:17.798 INFO:teuthology.task.install.util:Shipping 'stdin-killer'... 2024-05-27T07:12:17.799 DEBUG:teuthology.orchestra.run.smithi052:> set -ex 2024-05-27T07:12:17.799 DEBUG:teuthology.orchestra.run.smithi052:> sudo dd of=/usr/bin/stdin-killer 2024-05-27T07:12:17.824 DEBUG:teuthology.orchestra.run.smithi052:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-05-27T07:12:17.890 DEBUG:teuthology.orchestra.run.smithi105:> set -ex 2024-05-27T07:12:17.890 DEBUG:teuthology.orchestra.run.smithi105:> sudo dd of=/usr/bin/stdin-killer 2024-05-27T07:12:17.916 DEBUG:teuthology.orchestra.run.smithi105:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-05-27T07:12:17.981 DEBUG:teuthology.orchestra.run.smithi173:> set -ex 2024-05-27T07:12:17.981 DEBUG:teuthology.orchestra.run.smithi173:> sudo dd of=/usr/bin/stdin-killer 2024-05-27T07:12:18.004 DEBUG:teuthology.orchestra.run.smithi173:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-05-27T07:12:18.068 INFO:teuthology.run_tasks:Running task cephadm... 2024-05-27T07:12:18.161 INFO:tasks.cephadm:Config: {'roleless': False, 'cephfs': {'ec_profile': ['m=2', 'k=2', 'crush-failure-domain=osd'], 'max_mds': 5, 'session_timeout': 300, 'standby_replay': True}, 'conf': {'client': {'client mount timeout': 600, 'debug client': 20, 'debug ms': 1, 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'global': {'ms die on skipped message': False}, 'mds': {'debug mds': 20, 'debug mds balancer': 20, 'debug ms': 1, 'mds bal fragment size max': 10000, 'mds bal merge size': 5, 'mds bal split bits': 3, 'mds bal split size': 100, 'mds debug frag': True, 'mds debug scatterstat': True, 'mds op complaint time': 180, 'mds verify scatter': True, 'mds_export_ephemeral_random_max': 0.1, 'osd op complaint time': 180, 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'mgr': {'cephsqlite lock renewal timeout': 900000, 'debug cephsqlite': 20, 'debug client': 20, 'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon down mkfs grace': 300, 'mon op complaint time': 120}, 'osd': {'bluestore block size': 96636764160, 'bluestore compression mode': 'aggressive', 'bluestore fsck on mount': True, 'debug bluefs': 20, 'debug bluestore': 20, 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore', 'osd op complaint time': 180, 'osd shutdown pgref assert': True, 'osd_max_omap_entries_per_request': 10000}}, 'flavor': 'default', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', 'FS_DEGRADED', 'FS_INLINE_DATA_DEPRECATED', 'FS_WITH_FAILED_MDS', 'MDS_ALL_DOWN', 'MDS_DAMAGE', 'MDS_DEGRADED', 'MDS_FAILED', 'MDS_INSUFFICIENT_STANDBY', 'MDS_UP_LESS_THAN_MAX', 'filesystem is online with fewer MDS than max_mds', 'POOL_APP_NOT_ENABLED', 'do not have an application enabled', 'overall HEALTH_', 'Replacing daemon', 'deprecated feature inline_data', 'overall HEALTH_', '\\(OSD_DOWN\\)', '\\(OSD_', 'but it is still running', 'is not responding', 'PG_AVAILABILITY', 'PG_DEGRADED', 'Reduced data availability', 'Degraded data redundancy', 'slow metadata IO', 'SLOW_OPS', 'slow request'], 'sha1': '38f9d900c83c63799fbdbe61acc9a11b0d3554a6', 'subvols': {'create': 2}} 2024-05-27T07:12:18.161 INFO:tasks.cephadm:Cluster image is quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:38f9d900c83c63799fbdbe61acc9a11b0d3554a6 2024-05-27T07:12:18.162 INFO:tasks.cephadm:Cluster fsid is 73df2636-1bf8-11ef-bc9b-c7b262605968 2024-05-27T07:12:18.162 INFO:tasks.cephadm:Choosing monitor IPs and ports... 2024-05-27T07:12:18.162 INFO:tasks.cephadm:Monitor IPs: {'mon.a': '172.21.15.52', 'mon.b': '172.21.15.105', 'mon.c': '172.21.15.173'} 2024-05-27T07:12:18.162 INFO:tasks.cephadm:First mon is mon.a on smithi052 2024-05-27T07:12:18.162 INFO:tasks.cephadm:First mgr is x 2024-05-27T07:12:18.162 INFO:tasks.cephadm:Normalizing hostnames... 2024-05-27T07:12:18.162 DEBUG:teuthology.orchestra.run.smithi052:> sudo hostname $(hostname -s) 2024-05-27T07:12:18.189 DEBUG:teuthology.orchestra.run.smithi105:> sudo hostname $(hostname -s) 2024-05-27T07:12:18.215 DEBUG:teuthology.orchestra.run.smithi173:> sudo hostname $(hostname -s) 2024-05-27T07:12:18.239 INFO:tasks.cephadm:Downloading "compiled" cephadm from cachra 2024-05-27T07:12:18.240 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=38f9d900c83c63799fbdbe61acc9a11b0d3554a6 2024-05-27T07:12:18.553 INFO:tasks.cephadm:builder_project result: [{'status': 'ready', 'sha1': '38f9d900c83c63799fbdbe61acc9a11b0d3554a6', 'extra': {'build_url': 'https://jenkins.ceph.com/job/ceph-dev-new-build/ARCH=x86_64,AVAILABLE_ARCH=x86_64,AVAILABLE_DIST=centos9,DIST=centos9,MACHINE_SIZE=gigantic/79587/', 'root_build_cause': 'SCMTRIGGER', 'version': '19.0.0-3872-g38f9d900', 'node_name': '172.21.5.35+adami05', 'job_name': 'ceph-dev-new-build/ARCH=x86_64,AVAILABLE_ARCH=x86_64,AVAILABLE_DIST=centos9,DIST=centos9,MACHINE_SIZE=gigantic', 'package_manager_version': '19.0.0-3872.g38f9d900'}, 'url': 'https://1.chacra.ceph.com/r/ceph/wip-vshankar-testing-20240521.063558-debug/38f9d900c83c63799fbdbe61acc9a11b0d3554a6/centos/9/flavors/default/', 'distro_codename': None, 'modified': '2024-05-21 08:08:15.155937', 'distro_version': '9', 'project': 'ceph', 'flavor': 'default', 'ref': 'wip-vshankar-testing-20240521.063558-debug', 'chacra_url': 'https://1.chacra.ceph.com/repos/ceph/wip-vshankar-testing-20240521.063558-debug/38f9d900c83c63799fbdbe61acc9a11b0d3554a6/centos/9/flavors/default/', 'archs': ['source', 'x86_64'], 'distro': 'centos'}, {'status': 'ready', 'sha1': '38f9d900c83c63799fbdbe61acc9a11b0d3554a6', 'extra': {'build_url': 'https://jenkins.ceph.com/job/ceph-dev-new-build/ARCH=x86_64,AVAILABLE_ARCH=x86_64,AVAILABLE_DIST=centos9,DIST=centos9,MACHINE_SIZE=gigantic/79583/', 'root_build_cause': 'SCMTRIGGER', 'version': '19.0.0-3872-g38f9d900', 'node_name': '172.21.5.34+adami04', 'job_name': 'ceph-dev-new-build/ARCH=x86_64,AVAILABLE_ARCH=x86_64,AVAILABLE_DIST=centos9,DIST=centos9,MACHINE_SIZE=gigantic', 'package_manager_version': '19.0.0-3872.g38f9d900'}, 'url': 'https://3.chacra.ceph.com/r/ceph/wip-vshankar-testing-20240521.063558-debug/38f9d900c83c63799fbdbe61acc9a11b0d3554a6/centos/9/flavors/default/', 'distro_codename': None, 'modified': '2024-05-21 07:54:08.730060', 'distro_version': '9', 'project': 'ceph', 'flavor': 'default', 'ref': 'wip-vshankar-testing-20240521.063558-debug', 'chacra_url': 'https://3.chacra.ceph.com/repos/ceph/wip-vshankar-testing-20240521.063558-debug/38f9d900c83c63799fbdbe61acc9a11b0d3554a6/centos/9/flavors/default/', 'archs': ['x86_64', 'source'], 'distro': 'centos'}] 2024-05-27T07:12:18.697 INFO:tasks.util.chacra:got chacra host 1.chacra.ceph.com, ref wip-vshankar-testing-20240521.063558-debug, sha1 38f9d900c83c63799fbdbe61acc9a11b0d3554a6 from https://shaman.ceph.com/api/search/?project=ceph&distros=centos%2F9%2Fx86_64&flavor=default&sha1=38f9d900c83c63799fbdbe61acc9a11b0d3554a6 2024-05-27T07:12:18.699 INFO:tasks.cephadm:Discovered cachra url: https://1.chacra.ceph.com/binaries/ceph/wip-vshankar-testing-20240521.063558-debug/38f9d900c83c63799fbdbe61acc9a11b0d3554a6/centos/9/x86_64/flavors/default/cephadm 2024-05-27T07:12:18.699 INFO:tasks.cephadm:Downloading cephadm from url: https://1.chacra.ceph.com/binaries/ceph/wip-vshankar-testing-20240521.063558-debug/38f9d900c83c63799fbdbe61acc9a11b0d3554a6/centos/9/x86_64/flavors/default/cephadm 2024-05-27T07:12:18.699 DEBUG:teuthology.orchestra.run.smithi052:> curl --silent -L https://1.chacra.ceph.com/binaries/ceph/wip-vshankar-testing-20240521.063558-debug/38f9d900c83c63799fbdbe61acc9a11b0d3554a6/centos/9/x86_64/flavors/default/cephadm > /home/ubuntu/cephtest/cephadm && ls -l /home/ubuntu/cephtest/cephadm 2024-05-27T07:12:19.679 INFO:teuthology.orchestra.run.smithi052.stdout:-rw-r--r--. 1 ubuntu ubuntu 782786 May 27 07:12 /home/ubuntu/cephtest/cephadm 2024-05-27T07:12:19.680 DEBUG:teuthology.orchestra.run.smithi105:> curl --silent -L https://1.chacra.ceph.com/binaries/ceph/wip-vshankar-testing-20240521.063558-debug/38f9d900c83c63799fbdbe61acc9a11b0d3554a6/centos/9/x86_64/flavors/default/cephadm > /home/ubuntu/cephtest/cephadm && ls -l /home/ubuntu/cephtest/cephadm 2024-05-27T07:12:20.226 INFO:teuthology.orchestra.run.smithi105.stdout:-rw-r--r--. 1 ubuntu ubuntu 782786 May 27 07:12 /home/ubuntu/cephtest/cephadm 2024-05-27T07:12:20.226 DEBUG:teuthology.orchestra.run.smithi173:> curl --silent -L https://1.chacra.ceph.com/binaries/ceph/wip-vshankar-testing-20240521.063558-debug/38f9d900c83c63799fbdbe61acc9a11b0d3554a6/centos/9/x86_64/flavors/default/cephadm > /home/ubuntu/cephtest/cephadm && ls -l /home/ubuntu/cephtest/cephadm 2024-05-27T07:12:20.825 INFO:teuthology.orchestra.run.smithi173.stdout:-rw-r--r--. 1 ubuntu ubuntu 782786 May 27 07:12 /home/ubuntu/cephtest/cephadm 2024-05-27T07:12:20.826 DEBUG:teuthology.orchestra.run.smithi052:> test -s /home/ubuntu/cephtest/cephadm && test $(stat -c%s /home/ubuntu/cephtest/cephadm) -gt 1000 && chmod +x /home/ubuntu/cephtest/cephadm 2024-05-27T07:12:20.846 DEBUG:teuthology.orchestra.run.smithi105:> test -s /home/ubuntu/cephtest/cephadm && test $(stat -c%s /home/ubuntu/cephtest/cephadm) -gt 1000 && chmod +x /home/ubuntu/cephtest/cephadm 2024-05-27T07:12:20.865 DEBUG:teuthology.orchestra.run.smithi173:> test -s /home/ubuntu/cephtest/cephadm && test $(stat -c%s /home/ubuntu/cephtest/cephadm) -gt 1000 && chmod +x /home/ubuntu/cephtest/cephadm 2024-05-27T07:12:20.899 INFO:tasks.cephadm:Pulling image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:38f9d900c83c63799fbdbe61acc9a11b0d3554a6 on all hosts... 2024-05-27T07:12:20.899 DEBUG:teuthology.orchestra.run.smithi052:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:38f9d900c83c63799fbdbe61acc9a11b0d3554a6 pull 2024-05-27T07:12:20.902 DEBUG:teuthology.orchestra.run.smithi105:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:38f9d900c83c63799fbdbe61acc9a11b0d3554a6 pull 2024-05-27T07:12:20.907 DEBUG:teuthology.orchestra.run.smithi173:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:38f9d900c83c63799fbdbe61acc9a11b0d3554a6 pull 2024-05-27T07:12:21.792 INFO:teuthology.orchestra.run.smithi052.stderr:Pulling container image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:38f9d900c83c63799fbdbe61acc9a11b0d3554a6... 2024-05-27T07:12:21.807 INFO:teuthology.orchestra.run.smithi173.stderr:Pulling container image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:38f9d900c83c63799fbdbe61acc9a11b0d3554a6... 2024-05-27T07:12:21.836 INFO:teuthology.orchestra.run.smithi105.stderr:Pulling container image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:38f9d900c83c63799fbdbe61acc9a11b0d3554a6... 2024-05-27T07:12:52.231 INFO:teuthology.orchestra.run.smithi052.stdout:{ 2024-05-27T07:12:52.231 INFO:teuthology.orchestra.run.smithi052.stdout: "ceph_version": "ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev)", 2024-05-27T07:12:52.231 INFO:teuthology.orchestra.run.smithi052.stdout: "image_id": "e36ea4fc8a5807e8d2701370e350a0de3856538797f7fba64cfce7e344bd6799", 2024-05-27T07:12:52.231 INFO:teuthology.orchestra.run.smithi052.stdout: "repo_digests": [ 2024-05-27T07:12:52.231 INFO:teuthology.orchestra.run.smithi052.stdout: "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6" 2024-05-27T07:12:52.231 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2024-05-27T07:12:52.232 INFO:teuthology.orchestra.run.smithi052.stdout:} 2024-05-27T07:13:18.852 INFO:teuthology.orchestra.run.smithi173.stdout:{ 2024-05-27T07:13:18.852 INFO:teuthology.orchestra.run.smithi173.stdout: "ceph_version": "ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev)", 2024-05-27T07:13:18.852 INFO:teuthology.orchestra.run.smithi173.stdout: "image_id": "e36ea4fc8a5807e8d2701370e350a0de3856538797f7fba64cfce7e344bd6799", 2024-05-27T07:13:18.852 INFO:teuthology.orchestra.run.smithi173.stdout: "repo_digests": [ 2024-05-27T07:13:18.852 INFO:teuthology.orchestra.run.smithi173.stdout: "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6" 2024-05-27T07:13:18.852 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-05-27T07:13:18.852 INFO:teuthology.orchestra.run.smithi173.stdout:} 2024-05-27T07:13:22.345 INFO:teuthology.orchestra.run.smithi105.stdout:{ 2024-05-27T07:13:22.345 INFO:teuthology.orchestra.run.smithi105.stdout: "ceph_version": "ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev)", 2024-05-27T07:13:22.345 INFO:teuthology.orchestra.run.smithi105.stdout: "image_id": "e36ea4fc8a5807e8d2701370e350a0de3856538797f7fba64cfce7e344bd6799", 2024-05-27T07:13:22.346 INFO:teuthology.orchestra.run.smithi105.stdout: "repo_digests": [ 2024-05-27T07:13:22.346 INFO:teuthology.orchestra.run.smithi105.stdout: "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6" 2024-05-27T07:13:22.346 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-05-27T07:13:22.346 INFO:teuthology.orchestra.run.smithi105.stdout:} 2024-05-27T07:13:22.371 DEBUG:teuthology.orchestra.run.smithi052:> sudo mkdir -p /etc/ceph 2024-05-27T07:13:22.405 DEBUG:teuthology.orchestra.run.smithi105:> sudo mkdir -p /etc/ceph 2024-05-27T07:13:22.434 DEBUG:teuthology.orchestra.run.smithi173:> sudo mkdir -p /etc/ceph 2024-05-27T07:13:22.465 DEBUG:teuthology.orchestra.run.smithi052:> sudo chmod 777 /etc/ceph 2024-05-27T07:13:22.497 DEBUG:teuthology.orchestra.run.smithi105:> sudo chmod 777 /etc/ceph 2024-05-27T07:13:22.520 DEBUG:teuthology.orchestra.run.smithi173:> sudo chmod 777 /etc/ceph 2024-05-27T07:13:22.547 INFO:tasks.cephadm:Writing seed config... 2024-05-27T07:13:22.548 INFO:tasks.cephadm: override: [client] client mount timeout = 600 2024-05-27T07:13:22.549 INFO:tasks.cephadm: override: [client] debug client = 20 2024-05-27T07:13:22.549 INFO:tasks.cephadm: override: [client] debug ms = 1 2024-05-27T07:13:22.549 INFO:tasks.cephadm: override: [client] rados mon op timeout = 900 2024-05-27T07:13:22.549 INFO:tasks.cephadm: override: [client] rados osd op timeout = 900 2024-05-27T07:13:22.549 INFO:tasks.cephadm: override: [global] ms die on skipped message = False 2024-05-27T07:13:22.549 INFO:tasks.cephadm: override: [mds] debug mds = 20 2024-05-27T07:13:22.549 INFO:tasks.cephadm: override: [mds] debug mds balancer = 20 2024-05-27T07:13:22.549 INFO:tasks.cephadm: override: [mds] debug ms = 1 2024-05-27T07:13:22.549 INFO:tasks.cephadm: override: [mds] mds bal fragment size max = 10000 2024-05-27T07:13:22.549 INFO:tasks.cephadm: override: [mds] mds bal merge size = 5 2024-05-27T07:13:22.549 INFO:tasks.cephadm: override: [mds] mds bal split bits = 3 2024-05-27T07:13:22.550 INFO:tasks.cephadm: override: [mds] mds bal split size = 100 2024-05-27T07:13:22.550 INFO:tasks.cephadm: override: [mds] mds debug frag = True 2024-05-27T07:13:22.550 INFO:tasks.cephadm: override: [mds] mds debug scatterstat = True 2024-05-27T07:13:22.550 INFO:tasks.cephadm: override: [mds] mds op complaint time = 180 2024-05-27T07:13:22.550 INFO:tasks.cephadm: override: [mds] mds verify scatter = True 2024-05-27T07:13:22.550 INFO:tasks.cephadm: override: [mds] mds_export_ephemeral_random_max = 0.1 2024-05-27T07:13:22.550 INFO:tasks.cephadm: override: [mds] osd op complaint time = 180 2024-05-27T07:13:22.550 INFO:tasks.cephadm: override: [mds] rados mon op timeout = 900 2024-05-27T07:13:22.550 INFO:tasks.cephadm: override: [mds] rados osd op timeout = 900 2024-05-27T07:13:22.550 INFO:tasks.cephadm: override: [mgr] cephsqlite lock renewal timeout = 900000 2024-05-27T07:13:22.550 INFO:tasks.cephadm: override: [mgr] debug cephsqlite = 20 2024-05-27T07:13:22.550 INFO:tasks.cephadm: override: [mgr] debug client = 20 2024-05-27T07:13:22.551 INFO:tasks.cephadm: override: [mgr] debug mgr = 20 2024-05-27T07:13:22.551 INFO:tasks.cephadm: override: [mgr] debug ms = 1 2024-05-27T07:13:22.551 INFO:tasks.cephadm: override: [mon] debug mon = 20 2024-05-27T07:13:22.551 INFO:tasks.cephadm: override: [mon] debug ms = 1 2024-05-27T07:13:22.551 INFO:tasks.cephadm: override: [mon] debug paxos = 20 2024-05-27T07:13:22.551 INFO:tasks.cephadm: override: [mon] mon down mkfs grace = 300 2024-05-27T07:13:22.551 INFO:tasks.cephadm: override: [mon] mon op complaint time = 120 2024-05-27T07:13:22.551 INFO:tasks.cephadm: override: [osd] bluestore block size = 96636764160 2024-05-27T07:13:22.551 INFO:tasks.cephadm: override: [osd] bluestore compression mode = aggressive 2024-05-27T07:13:22.551 INFO:tasks.cephadm: override: [osd] bluestore fsck on mount = True 2024-05-27T07:13:22.551 INFO:tasks.cephadm: override: [osd] debug bluefs = 20 2024-05-27T07:13:22.551 INFO:tasks.cephadm: override: [osd] debug bluestore = 20 2024-05-27T07:13:22.551 INFO:tasks.cephadm: override: [osd] debug ms = 1 2024-05-27T07:13:22.552 INFO:tasks.cephadm: override: [osd] debug osd = 20 2024-05-27T07:13:22.552 INFO:tasks.cephadm: override: [osd] debug rocksdb = 10 2024-05-27T07:13:22.552 INFO:tasks.cephadm: override: [osd] mon osd backfillfull_ratio = 0.85 2024-05-27T07:13:22.552 INFO:tasks.cephadm: override: [osd] mon osd full ratio = 0.9 2024-05-27T07:13:22.552 INFO:tasks.cephadm: override: [osd] mon osd nearfull ratio = 0.8 2024-05-27T07:13:22.552 INFO:tasks.cephadm: override: [osd] osd failsafe full ratio = 0.95 2024-05-27T07:13:22.552 INFO:tasks.cephadm: override: [osd] osd objectstore = bluestore 2024-05-27T07:13:22.552 INFO:tasks.cephadm: override: [osd] osd op complaint time = 180 2024-05-27T07:13:22.552 INFO:tasks.cephadm: override: [osd] osd shutdown pgref assert = True 2024-05-27T07:13:22.552 INFO:tasks.cephadm: override: [osd] osd_max_omap_entries_per_request = 10000 2024-05-27T07:13:22.553 DEBUG:teuthology.orchestra.run.smithi052:> set -ex 2024-05-27T07:13:22.553 DEBUG:teuthology.orchestra.run.smithi052:> dd of=/home/ubuntu/cephtest/seed.ceph.conf 2024-05-27T07:13:22.572 DEBUG:tasks.cephadm:Final config: [global] # make logging friendly to teuthology log_to_file = true log_to_stderr = false log to journald = false mon cluster log to file = true mon cluster log file level = debug mon clock drift allowed = 1.000 # replicate across OSDs, not hosts osd crush chooseleaf type = 0 #osd pool default size = 2 osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd # enable some debugging auth debug = true ms die on old message = true ms die on bug = true debug asserts on shutdown = true # adjust warnings mon max pg per osd = 10000# >= luminous mon pg warn max object skew = 0 mon osd allow primary affinity = true mon osd allow pg remap = true mon warn on legacy crush tunables = false mon warn on crush straw calc version zero = false mon warn on no sortbitwise = false mon warn on osd down out interval zero = false mon warn on too few osds = false mon_warn_on_pool_pg_num_not_power_of_two = false # disable pg_autoscaler by default for new pools osd_pool_default_pg_autoscale_mode = off # tests delete pools mon allow pool delete = true fsid = 73df2636-1bf8-11ef-bc9b-c7b262605968 ms die on skipped message = False [osd] osd scrub load threshold = 5.0 osd scrub max interval = 600 osd mclock profile = high_recovery_ops osd recover clone overlap = true osd recovery max chunk = 1048576 osd deep scrub update digest min age = 30 osd map max advance = 10 osd memory target autotune = true # debugging osd debug shutdown = true osd debug op order = true osd debug verify stray on activate = true osd debug pg log writeout = true osd debug verify cached snaps = true osd debug verify missing on start = true osd debug misdirected ops = true osd op queue = debug_random osd op queue cut off = debug_random osd shutdown pgref assert = True bdev debug aio = true osd sloppy crc = true bluestore block size = 96636764160 bluestore compression mode = aggressive bluestore fsck on mount = True debug bluefs = 20 debug bluestore = 20 debug ms = 1 debug osd = 20 debug rocksdb = 10 mon osd backfillfull_ratio = 0.85 mon osd full ratio = 0.9 mon osd nearfull ratio = 0.8 osd failsafe full ratio = 0.95 osd objectstore = bluestore osd op complaint time = 180 osd_max_omap_entries_per_request = 10000 [mgr] mon reweight min pgs per osd = 4 mon reweight min bytes per osd = 10 mgr/telemetry/nag = false cephsqlite lock renewal timeout = 900000 debug cephsqlite = 20 debug client = 20 debug mgr = 20 debug ms = 1 [mon] mon data avail warn = 5 mon mgr mkfs grace = 240 mon reweight min pgs per osd = 4 mon osd reporter subtree level = osd mon osd prime pg temp = true mon reweight min bytes per osd = 10 # rotate auth tickets quickly to exercise renewal paths auth mon ticket ttl = 660# 11m auth service ticket ttl = 240# 4m # don't complain about global id reclaim mon_warn_on_insecure_global_id_reclaim = false mon_warn_on_insecure_global_id_reclaim_allowed = false debug mon = 20 debug ms = 1 debug paxos = 20 mon down mkfs grace = 300 mon op complaint time = 120 [client.rgw] rgw cache enabled = true rgw enable ops log = true rgw enable usage log = true [client] client mount timeout = 600 debug client = 20 debug ms = 1 rados mon op timeout = 900 rados osd op timeout = 900 [mds] debug mds = 20 debug mds balancer = 20 debug ms = 1 mds bal fragment size max = 10000 mds bal merge size = 5 mds bal split bits = 3 mds bal split size = 100 mds debug frag = True mds debug scatterstat = True mds op complaint time = 180 mds verify scatter = True mds_export_ephemeral_random_max = 0.1 osd op complaint time = 180 rados mon op timeout = 900 rados osd op timeout = 900 2024-05-27T07:13:22.573 DEBUG:teuthology.orchestra.run.smithi052:mon.a> sudo journalctl -f -n 0 -u ceph-73df2636-1bf8-11ef-bc9b-c7b262605968@mon.a.service 2024-05-27T07:13:22.615 DEBUG:teuthology.orchestra.run.smithi052:mgr.x> sudo journalctl -f -n 0 -u ceph-73df2636-1bf8-11ef-bc9b-c7b262605968@mgr.x.service 2024-05-27T07:13:22.658 INFO:tasks.cephadm:Bootstrapping... 2024-05-27T07:13:22.659 DEBUG:teuthology.orchestra.run.smithi052:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:38f9d900c83c63799fbdbe61acc9a11b0d3554a6 -v bootstrap --fsid 73df2636-1bf8-11ef-bc9b-c7b262605968 --config /home/ubuntu/cephtest/seed.ceph.conf --output-config /etc/ceph/ceph.conf --output-keyring /etc/ceph/ceph.client.admin.keyring --output-pub-ssh-key /home/ubuntu/cephtest/ceph.pub --mon-id a --mgr-id x --orphan-initial-daemons --skip-monitoring-stack --mon-ip 172.21.15.52 --skip-admin-label && sudo chmod +r /etc/ceph/ceph.client.admin.keyring 2024-05-27T07:13:22.870 INFO:teuthology.orchestra.run.smithi052.stdout:-------------------------------------------------------------------------------- 2024-05-27T07:13:22.870 INFO:teuthology.orchestra.run.smithi052.stdout:cephadm ['--image', 'quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:38f9d900c83c63799fbdbe61acc9a11b0d3554a6', '-v', 'bootstrap', '--fsid', '73df2636-1bf8-11ef-bc9b-c7b262605968', '--config', '/home/ubuntu/cephtest/seed.ceph.conf', '--output-config', '/etc/ceph/ceph.conf', '--output-keyring', '/etc/ceph/ceph.client.admin.keyring', '--output-pub-ssh-key', '/home/ubuntu/cephtest/ceph.pub', '--mon-id', 'a', '--mgr-id', 'x', '--orphan-initial-daemons', '--skip-monitoring-stack', '--mon-ip', '172.21.15.52', '--skip-admin-label'] 2024-05-27T07:13:22.870 INFO:teuthology.orchestra.run.smithi052.stderr:Specifying an fsid for your cluster offers no advantages and may increase the likelihood of fsid conflicts. 2024-05-27T07:13:22.870 INFO:teuthology.orchestra.run.smithi052.stdout:Verifying podman|docker is present... 2024-05-27T07:13:22.905 INFO:teuthology.orchestra.run.smithi052.stdout:/bin/podman: stdout 5.0.2 2024-05-27T07:13:22.905 INFO:teuthology.orchestra.run.smithi052.stdout:Verifying lvm2 is present... 2024-05-27T07:13:22.905 INFO:teuthology.orchestra.run.smithi052.stdout:Verifying time synchronization is in place... 2024-05-27T07:13:22.913 INFO:teuthology.orchestra.run.smithi052.stdout:Non-zero exit code 1 from systemctl is-enabled chrony.service 2024-05-27T07:13:22.913 INFO:teuthology.orchestra.run.smithi052.stdout:systemctl: stderr Failed to get unit file state for chrony.service: No such file or directory 2024-05-27T07:13:22.920 INFO:teuthology.orchestra.run.smithi052.stdout:Non-zero exit code 3 from systemctl is-active chrony.service 2024-05-27T07:13:22.920 INFO:teuthology.orchestra.run.smithi052.stdout:systemctl: stdout inactive 2024-05-27T07:13:22.928 INFO:teuthology.orchestra.run.smithi052.stdout:systemctl: stdout enabled 2024-05-27T07:13:22.935 INFO:teuthology.orchestra.run.smithi052.stdout:systemctl: stdout active 2024-05-27T07:13:22.935 INFO:teuthology.orchestra.run.smithi052.stdout:Unit chronyd.service is enabled and running 2024-05-27T07:13:22.936 INFO:teuthology.orchestra.run.smithi052.stdout:Repeating the final host check... 2024-05-27T07:13:22.961 INFO:teuthology.orchestra.run.smithi052.stdout:/bin/podman: stdout 5.0.2 2024-05-27T07:13:22.962 INFO:teuthology.orchestra.run.smithi052.stdout:podman (/bin/podman) version 5.0.2 is present 2024-05-27T07:13:22.962 INFO:teuthology.orchestra.run.smithi052.stdout:systemctl is present 2024-05-27T07:13:22.962 INFO:teuthology.orchestra.run.smithi052.stdout:lvcreate is present 2024-05-27T07:13:22.969 INFO:teuthology.orchestra.run.smithi052.stdout:Non-zero exit code 1 from systemctl is-enabled chrony.service 2024-05-27T07:13:22.969 INFO:teuthology.orchestra.run.smithi052.stdout:systemctl: stderr Failed to get unit file state for chrony.service: No such file or directory 2024-05-27T07:13:22.976 INFO:teuthology.orchestra.run.smithi052.stdout:Non-zero exit code 3 from systemctl is-active chrony.service 2024-05-27T07:13:22.976 INFO:teuthology.orchestra.run.smithi052.stdout:systemctl: stdout inactive 2024-05-27T07:13:22.984 INFO:teuthology.orchestra.run.smithi052.stdout:systemctl: stdout enabled 2024-05-27T07:13:22.992 INFO:teuthology.orchestra.run.smithi052.stdout:systemctl: stdout active 2024-05-27T07:13:22.992 INFO:teuthology.orchestra.run.smithi052.stdout:Unit chronyd.service is enabled and running 2024-05-27T07:13:22.992 INFO:teuthology.orchestra.run.smithi052.stdout:Host looks OK 2024-05-27T07:13:22.992 INFO:teuthology.orchestra.run.smithi052.stdout:Cluster fsid: 73df2636-1bf8-11ef-bc9b-c7b262605968 2024-05-27T07:13:22.993 INFO:teuthology.orchestra.run.smithi052.stdout:Acquiring lock 139758584131104 on /run/cephadm/73df2636-1bf8-11ef-bc9b-c7b262605968.lock 2024-05-27T07:13:22.993 INFO:teuthology.orchestra.run.smithi052.stdout:Lock 139758584131104 acquired on /run/cephadm/73df2636-1bf8-11ef-bc9b-c7b262605968.lock 2024-05-27T07:13:22.993 INFO:teuthology.orchestra.run.smithi052.stdout:Verifying IP 172.21.15.52 port 3300 ... 2024-05-27T07:13:22.993 INFO:teuthology.orchestra.run.smithi052.stdout:Verifying IP 172.21.15.52 port 6789 ... 2024-05-27T07:13:22.994 INFO:teuthology.orchestra.run.smithi052.stdout:Base mon IP(s) is [172.21.15.52:3300, 172.21.15.52:6789], mon addrv is [v2:172.21.15.52:3300,v1:172.21.15.52:6789] 2024-05-27T07:13:22.997 INFO:teuthology.orchestra.run.smithi052.stdout:/sbin/ip: stdout default via 172.21.15.254 dev ens1f0 proto dhcp src 172.21.15.52 metric 100 2024-05-27T07:13:22.997 INFO:teuthology.orchestra.run.smithi052.stdout:/sbin/ip: stdout 172.21.0.0/20 dev ens1f0 proto kernel scope link src 172.21.15.52 metric 100 2024-05-27T07:13:23.000 INFO:teuthology.orchestra.run.smithi052.stdout:/sbin/ip: stdout ::1 dev lo proto kernel metric 256 pref medium 2024-05-27T07:13:23.001 INFO:teuthology.orchestra.run.smithi052.stdout:/sbin/ip: stdout fe80::/64 dev ens1f0 proto kernel metric 1024 pref medium 2024-05-27T07:13:23.001 INFO:teuthology.orchestra.run.smithi052.stdout:/sbin/ip: stdout default via fe80::327c:5e00:6487:71e0 dev ens1f0 proto ra metric 100 pref medium 2024-05-27T07:13:23.003 INFO:teuthology.orchestra.run.smithi052.stdout:/sbin/ip: stdout 1: lo: mtu 65536 state UNKNOWN qlen 1000 2024-05-27T07:13:23.003 INFO:teuthology.orchestra.run.smithi052.stdout:/sbin/ip: stdout inet6 ::1/128 scope host 2024-05-27T07:13:23.003 INFO:teuthology.orchestra.run.smithi052.stdout:/sbin/ip: stdout valid_lft forever preferred_lft forever 2024-05-27T07:13:23.003 INFO:teuthology.orchestra.run.smithi052.stdout:/sbin/ip: stdout 4: ens1f0: mtu 1500 state UP qlen 1000 2024-05-27T07:13:23.003 INFO:teuthology.orchestra.run.smithi052.stdout:/sbin/ip: stdout inet6 fe80::ec4:7aff:febd:15de/64 scope link noprefixroute 2024-05-27T07:13:23.003 INFO:teuthology.orchestra.run.smithi052.stdout:/sbin/ip: stdout valid_lft forever preferred_lft forever 2024-05-27T07:13:23.004 INFO:teuthology.orchestra.run.smithi052.stdout:Mon IP `172.21.15.52` is in CIDR network `172.21.0.0/20` 2024-05-27T07:13:23.004 INFO:teuthology.orchestra.run.smithi052.stdout:Mon IP `172.21.15.52` is in CIDR network `172.21.0.0/20` 2024-05-27T07:13:23.004 INFO:teuthology.orchestra.run.smithi052.stdout:Inferred mon public CIDR from local network configuration ['172.21.0.0/20', '172.21.0.0/20'] 2024-05-27T07:13:23.004 INFO:teuthology.orchestra.run.smithi052.stdout:Internal network (--cluster-network) has not been provided, OSD replication will default to the public_network 2024-05-27T07:13:23.005 INFO:teuthology.orchestra.run.smithi052.stdout:Pulling container image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:38f9d900c83c63799fbdbe61acc9a11b0d3554a6... 2024-05-27T07:13:26.665 INFO:teuthology.orchestra.run.smithi052.stdout:/bin/podman: stdout e36ea4fc8a5807e8d2701370e350a0de3856538797f7fba64cfce7e344bd6799 2024-05-27T07:13:26.666 INFO:teuthology.orchestra.run.smithi052.stdout:/bin/podman: stderr Trying to pull quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:38f9d900c83c63799fbdbe61acc9a11b0d3554a6... 2024-05-27T07:13:26.666 INFO:teuthology.orchestra.run.smithi052.stdout:/bin/podman: stderr Getting image source signatures 2024-05-27T07:13:26.666 INFO:teuthology.orchestra.run.smithi052.stdout:/bin/podman: stderr Copying blob sha256:cf46015854c6cdc81d3273f73f8f001fe4a1edc6ed77e464da8c9d5e01e9b108 2024-05-27T07:13:26.666 INFO:teuthology.orchestra.run.smithi052.stdout:/bin/podman: stderr Copying blob sha256:6ac36e7c57f227110df5cd65cfdca1cc05e0b17df6748637a6aa04c5c80f89c6 2024-05-27T07:13:26.666 INFO:teuthology.orchestra.run.smithi052.stdout:/bin/podman: stderr Copying config sha256:e36ea4fc8a5807e8d2701370e350a0de3856538797f7fba64cfce7e344bd6799 2024-05-27T07:13:26.666 INFO:teuthology.orchestra.run.smithi052.stdout:/bin/podman: stderr Writing manifest to image destination 2024-05-27T07:13:27.639 INFO:teuthology.orchestra.run.smithi052.stdout:ceph: stdout ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:13:27.639 INFO:teuthology.orchestra.run.smithi052.stdout:Ceph version: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:13:27.639 INFO:teuthology.orchestra.run.smithi052.stdout:Extracting ceph user uid/gid from container image... 2024-05-27T07:13:28.503 INFO:teuthology.orchestra.run.smithi052.stdout:stat: stdout 167 167 2024-05-27T07:13:28.503 INFO:teuthology.orchestra.run.smithi052.stdout:Creating initial keys... 2024-05-27T07:13:29.349 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph-authtool: stdout AQCYMlRmzOPuMhAArDIf5pPN5cofaPY+hLoAHg== 2024-05-27T07:13:30.222 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph-authtool: stdout AQCZMlRm4EBNKRAAizs7kmv70h53i6EeCVS8eg== 2024-05-27T07:13:31.093 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph-authtool: stdout AQCaMlRmkvQFIxAAHsCNXdi7CnPIvk/D8wNcgQ== 2024-05-27T07:13:31.094 INFO:teuthology.orchestra.run.smithi052.stdout:Creating initial monmap... 2024-05-27T07:13:31.924 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/monmaptool: stdout /usr/bin/monmaptool: monmap file /tmp/monmap 2024-05-27T07:13:31.924 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/monmaptool: stdout setting min_mon_release = quincy 2024-05-27T07:13:31.924 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/monmaptool: stdout /usr/bin/monmaptool: set fsid to 73df2636-1bf8-11ef-bc9b-c7b262605968 2024-05-27T07:13:31.924 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/monmaptool: stdout /usr/bin/monmaptool: writing epoch 0 to /tmp/monmap (1 monitors) 2024-05-27T07:13:31.924 INFO:teuthology.orchestra.run.smithi052.stdout:monmaptool for a [v2:172.21.15.52:3300,v1:172.21.15.52:6789] on /usr/bin/monmaptool: monmap file /tmp/monmap 2024-05-27T07:13:31.924 INFO:teuthology.orchestra.run.smithi052.stdout:setting min_mon_release = quincy 2024-05-27T07:13:31.924 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/monmaptool: set fsid to 73df2636-1bf8-11ef-bc9b-c7b262605968 2024-05-27T07:13:31.924 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/monmaptool: writing epoch 0 to /tmp/monmap (1 monitors) 2024-05-27T07:13:31.924 INFO:teuthology.orchestra.run.smithi052.stdout: 2024-05-27T07:13:31.924 INFO:teuthology.orchestra.run.smithi052.stdout:Creating mon... 2024-05-27T07:13:33.138 INFO:teuthology.orchestra.run.smithi052.stdout:create mon.a on 2024-05-27T07:13:33.443 INFO:teuthology.orchestra.run.smithi052.stdout:systemctl: stderr Removed "/etc/systemd/system/multi-user.target.wants/ceph.target". 2024-05-27T07:13:33.607 INFO:teuthology.orchestra.run.smithi052.stdout:systemctl: stderr Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /etc/systemd/system/ceph.target. 2024-05-27T07:13:33.788 INFO:teuthology.orchestra.run.smithi052.stdout:systemctl: stderr Created symlink /etc/systemd/system/multi-user.target.wants/ceph-73df2636-1bf8-11ef-bc9b-c7b262605968.target → /etc/systemd/system/ceph-73df2636-1bf8-11ef-bc9b-c7b262605968.target. 2024-05-27T07:13:33.788 INFO:teuthology.orchestra.run.smithi052.stdout:systemctl: stderr Created symlink /etc/systemd/system/ceph.target.wants/ceph-73df2636-1bf8-11ef-bc9b-c7b262605968.target → /etc/systemd/system/ceph-73df2636-1bf8-11ef-bc9b-c7b262605968.target. 2024-05-27T07:13:34.069 INFO:teuthology.orchestra.run.smithi052.stdout:Non-zero exit code 1 from systemctl reset-failed ceph-73df2636-1bf8-11ef-bc9b-c7b262605968@mon.a 2024-05-27T07:13:34.069 INFO:teuthology.orchestra.run.smithi052.stdout:systemctl: stderr Failed to reset failed state of unit ceph-73df2636-1bf8-11ef-bc9b-c7b262605968@mon.a.service: Unit ceph-73df2636-1bf8-11ef-bc9b-c7b262605968@mon.a.service not loaded. 2024-05-27T07:13:34.244 INFO:teuthology.orchestra.run.smithi052.stdout:systemctl: stderr Created symlink /etc/systemd/system/ceph-73df2636-1bf8-11ef-bc9b-c7b262605968.target.wants/ceph-73df2636-1bf8-11ef-bc9b-c7b262605968@mon.a.service → /etc/systemd/system/ceph-73df2636-1bf8-11ef-bc9b-c7b262605968@.service. 2024-05-27T07:13:34.770 INFO:teuthology.orchestra.run.smithi052.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2024-05-27T07:13:34.770 INFO:teuthology.orchestra.run.smithi052.stdout:systemctl: stdout disabled 2024-05-27T07:13:34.777 INFO:teuthology.orchestra.run.smithi052.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2024-05-27T07:13:34.777 INFO:teuthology.orchestra.run.smithi052.stdout:systemctl: stdout inactive 2024-05-27T07:13:34.778 INFO:teuthology.orchestra.run.smithi052.stdout:firewalld.service is not enabled 2024-05-27T07:13:34.778 INFO:teuthology.orchestra.run.smithi052.stdout:Not possible to enable service . firewalld.service is not available 2024-05-27T07:13:34.778 INFO:teuthology.orchestra.run.smithi052.stdout:Waiting for mon to start... 2024-05-27T07:13:34.778 INFO:teuthology.orchestra.run.smithi052.stdout:Waiting for mon... 2024-05-27T07:13:35.829 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout cluster: 2024-05-27T07:13:35.829 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout id: 73df2636-1bf8-11ef-bc9b-c7b262605968 2024-05-27T07:13:35.829 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout health: HEALTH_OK 2024-05-27T07:13:35.829 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout 2024-05-27T07:13:35.829 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout services: 2024-05-27T07:13:35.829 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout mon: 1 daemons, quorum a (age 0.584704s) 2024-05-27T07:13:35.829 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout mgr: no daemons active 2024-05-27T07:13:35.829 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout osd: 0 osds: 0 up, 0 in 2024-05-27T07:13:35.830 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout 2024-05-27T07:13:35.830 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout data: 2024-05-27T07:13:35.830 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout pools: 0 pools, 0 pgs 2024-05-27T07:13:35.830 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout objects: 0 objects, 0 B 2024-05-27T07:13:35.830 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout usage: 0 B used, 0 B / 0 B avail 2024-05-27T07:13:35.830 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout pgs: 2024-05-27T07:13:35.830 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout 2024-05-27T07:13:35.830 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:13:35.239+0000 7f5ace11f700 1 Processor -- start 2024-05-27T07:13:35.830 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:13:35.240+0000 7f5ace11f700 1 -- start start 2024-05-27T07:13:35.830 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:13:35.240+0000 7f5ace11f700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f5ac810a6d0 0x7f5ac810caf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:13:35.830 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:13:35.240+0000 7f5ace11f700 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x7f5ac807d810 con 0x7f5ac810a6d0 2024-05-27T07:13:35.831 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:13:35.240+0000 7f5ac77fe700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f5ac810a6d0 0x7f5ac810caf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:13:35.831 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:13:35.240+0000 7f5ac77fe700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f5ac810a6d0 0x7f5ac810caf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.52:56450/0 (socket says 172.21.15.52:56450) 2024-05-27T07:13:35.831 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:13:35.240+0000 7f5ac77fe700 1 -- 172.21.15.52:0/3864550976 learned_addr learned my addr 172.21.15.52:0/3864550976 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-05-27T07:13:35.831 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:13:35.240+0000 7f5ac77fe700 1 -- 172.21.15.52:0/3864550976 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5ac810d030 con 0x7f5ac810a6d0 2024-05-27T07:13:35.831 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:13:35.241+0000 7f5ac77fe700 1 --2- 172.21.15.52:0/3864550976 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f5ac810a6d0 0x7f5ac810caf0 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 crypto rx=0x7f5ab80098a0 tx=0x7f5ab8009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4e0f55a56943c902 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:13:35.831 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:13:35.241+0000 7f5ac67fc700 1 -- 172.21.15.52:0/3864550976 <== mon.0 v2:172.21.15.52:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f5ab8004160 con 0x7f5ac810a6d0 2024-05-27T07:13:35.831 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:13:35.241+0000 7f5ac67fc700 1 -- 172.21.15.52:0/3864550976 <== mon.0 v2:172.21.15.52:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5ab8004300 con 0x7f5ac810a6d0 2024-05-27T07:13:35.831 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:13:35.241+0000 7f5ac67fc700 1 -- 172.21.15.52:0/3864550976 <== mon.0 v2:172.21.15.52:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f5ab80036a0 con 0x7f5ac810a6d0 2024-05-27T07:13:35.831 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:13:35.241+0000 7f5ace11f700 1 -- 172.21.15.52:0/3864550976 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f5ac810a6d0 msgr2=0x7f5ac810caf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:13:35.831 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:13:35.241+0000 7f5ace11f700 1 --2- 172.21.15.52:0/3864550976 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f5ac810a6d0 0x7f5ac810caf0 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 crypto rx=0x7f5ab80098a0 tx=0x7f5ab8009bb0 comp rx=0 tx=0).stop 2024-05-27T07:13:35.831 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:13:35.242+0000 7f5ace11f700 1 -- 172.21.15.52:0/3864550976 shutdown_connections 2024-05-27T07:13:35.831 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:13:35.242+0000 7f5ace11f700 1 --2- 172.21.15.52:0/3864550976 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f5ac810a6d0 0x7f5ac810caf0 unknown :-1 s=CLOSED pgs=1 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:13:35.831 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:13:35.242+0000 7f5ace11f700 1 -- 172.21.15.52:0/3864550976 >> 172.21.15.52:0/3864550976 conn(0x7f5ac8104150 msgr2=0x7f5ac81065e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:13:35.832 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:13:35.242+0000 7f5ace11f700 1 -- 172.21.15.52:0/3864550976 shutdown_connections 2024-05-27T07:13:35.832 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:13:35.242+0000 7f5ace11f700 1 -- 172.21.15.52:0/3864550976 wait complete. 2024-05-27T07:13:35.832 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:13:35.242+0000 7f5ace11f700 1 Processor -- start 2024-05-27T07:13:35.832 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:13:35.243+0000 7f5ace11f700 1 -- start start 2024-05-27T07:13:35.832 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:13:35.243+0000 7f5ace11f700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f5ac810a6d0 0x7f5ac819d6c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:13:35.832 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:13:35.243+0000 7f5ace11f700 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x7f5ac810d3e0 con 0x7f5ac810a6d0 2024-05-27T07:13:35.832 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:13:35.243+0000 7f5ac77fe700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f5ac810a6d0 0x7f5ac819d6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:13:35.832 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:13:35.243+0000 7f5ac77fe700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f5ac810a6d0 0x7f5ac819d6c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.52:56454/0 (socket says 172.21.15.52:56454) 2024-05-27T07:13:35.832 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:13:35.243+0000 7f5ac77fe700 1 -- 172.21.15.52:0/151124590 learned_addr learned my addr 172.21.15.52:0/151124590 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-05-27T07:13:35.832 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:13:35.243+0000 7f5ac77fe700 1 -- 172.21.15.52:0/151124590 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5ac819dc00 con 0x7f5ac810a6d0 2024-05-27T07:13:35.832 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:13:35.243+0000 7f5ac77fe700 1 --2- 172.21.15.52:0/151124590 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f5ac810a6d0 0x7f5ac819d6c0 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 crypto rx=0x7f5ab8000c00 tx=0x7f5ab8004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:13:35.832 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:13:35.244+0000 7f5ac4ff9700 1 -- 172.21.15.52:0/151124590 <== mon.0 v2:172.21.15.52:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f5ab8003db0 con 0x7f5ac810a6d0 2024-05-27T07:13:35.832 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:13:35.244+0000 7f5ac4ff9700 1 -- 172.21.15.52:0/151124590 <== mon.0 v2:172.21.15.52:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5ab8011630 con 0x7f5ac810a6d0 2024-05-27T07:13:35.833 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:13:35.244+0000 7f5ace11f700 1 -- 172.21.15.52:0/151124590 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5ac819de90 con 0x7f5ac810a6d0 2024-05-27T07:13:35.833 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:13:35.244+0000 7f5ac4ff9700 1 -- 172.21.15.52:0/151124590 <== mon.0 v2:172.21.15.52:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f5ab8011950 con 0x7f5ac810a6d0 2024-05-27T07:13:35.833 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:13:35.244+0000 7f5ace11f700 1 -- 172.21.15.52:0/151124590 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5ac81a0aa0 con 0x7f5ac810a6d0 2024-05-27T07:13:35.833 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:13:35.244+0000 7f5ac4ff9700 1 -- 172.21.15.52:0/151124590 <== mon.0 v2:172.21.15.52:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7f5ab8011af0 con 0x7f5ac810a6d0 2024-05-27T07:13:35.833 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:13:35.244+0000 7f5ac4ff9700 1 -- 172.21.15.52:0/151124590 <== mon.0 v2:172.21.15.52:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7f5ab801c790 con 0x7f5ac810a6d0 2024-05-27T07:13:35.833 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:13:35.245+0000 7f5ace11f700 1 -- 172.21.15.52:0/151124590 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5ac8064e10 con 0x7f5ac810a6d0 2024-05-27T07:13:35.833 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:13:35.247+0000 7f5ac4ff9700 1 -- 172.21.15.52:0/151124590 <== mon.0 v2:172.21.15.52:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73389 (secure 0 0 0) 0x7f5ab8032020 con 0x7f5ac810a6d0 2024-05-27T07:13:35.833 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:13:35.320+0000 7f5ace11f700 1 -- 172.21.15.52:0/151124590 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_command({"prefix": "status"} v 0) -- 0x7f5ac802e8e0 con 0x7f5ac810a6d0 2024-05-27T07:13:35.833 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:13:35.321+0000 7f5ac4ff9700 1 -- 172.21.15.52:0/151124590 <== mon.0 v2:172.21.15.52:3300/0 7 ==== mon_command_ack([{"prefix": "status"}]=0 v0) ==== 54+0+317 (secure 0 0 0) 0x7f5ab8004260 con 0x7f5ac810a6d0 2024-05-27T07:13:35.833 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:13:35.322+0000 7f5ace11f700 1 -- 172.21.15.52:0/151124590 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f5ac810a6d0 msgr2=0x7f5ac819d6c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:13:35.833 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:13:35.322+0000 7f5ace11f700 1 --2- 172.21.15.52:0/151124590 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f5ac810a6d0 0x7f5ac819d6c0 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 crypto rx=0x7f5ab8000c00 tx=0x7f5ab8004750 comp rx=0 tx=0).stop 2024-05-27T07:13:35.833 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:13:35.322+0000 7f5ace11f700 1 -- 172.21.15.52:0/151124590 shutdown_connections 2024-05-27T07:13:35.833 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:13:35.322+0000 7f5ace11f700 1 --2- 172.21.15.52:0/151124590 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f5ac810a6d0 0x7f5ac819d6c0 unknown :-1 s=CLOSED pgs=2 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:13:35.833 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:13:35.322+0000 7f5ace11f700 1 -- 172.21.15.52:0/151124590 >> 172.21.15.52:0/151124590 conn(0x7f5ac8104150 msgr2=0x7f5ac8106330 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:13:35.834 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:13:35.322+0000 7f5ace11f700 1 -- 172.21.15.52:0/151124590 shutdown_connections 2024-05-27T07:13:35.834 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:13:35.322+0000 7f5ace11f700 1 -- 172.21.15.52:0/151124590 wait complete. 2024-05-27T07:13:35.834 INFO:teuthology.orchestra.run.smithi052.stdout:mon is available 2024-05-27T07:13:35.834 INFO:teuthology.orchestra.run.smithi052.stdout:Assimilating anything we can from ceph.conf... 2024-05-27T07:13:36.918 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:13:36 smithi052 ceph-mon[31583]: from='client.? 172.21.15.52:0/1711580040' entity='client.admin' cmd='[{"prefix": "config assimilate-conf"}]': finished 2024-05-27T07:13:36.927 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout 2024-05-27T07:13:36.927 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout [global] 2024-05-27T07:13:36.927 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout fsid = 73df2636-1bf8-11ef-bc9b-c7b262605968 2024-05-27T07:13:36.927 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout mon_cluster_log_file_level = debug 2024-05-27T07:13:36.927 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout mon_host = [v2:172.21.15.52:3300,v1:172.21.15.52:6789] 2024-05-27T07:13:36.927 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout mon_osd_allow_pg_remap = true 2024-05-27T07:13:36.927 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout mon_osd_allow_primary_affinity = true 2024-05-27T07:13:36.927 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout mon_warn_on_no_sortbitwise = false 2024-05-27T07:13:36.927 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout osd_crush_chooseleaf_type = 0 2024-05-27T07:13:36.927 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout 2024-05-27T07:13:36.927 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout [mgr] 2024-05-27T07:13:36.928 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout mgr/telemetry/nag = false 2024-05-27T07:13:36.928 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout 2024-05-27T07:13:36.928 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout [osd] 2024-05-27T07:13:36.928 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout mon_osd_backfillfull_ratio = 0.85 2024-05-27T07:13:36.928 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout mon_osd_full_ratio = 0.9 2024-05-27T07:13:36.928 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout mon_osd_nearfull_ratio = 0.8 2024-05-27T07:13:36.928 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout osd_map_max_advance = 10 2024-05-27T07:13:36.928 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout osd_sloppy_crc = true 2024-05-27T07:13:36.928 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:13:36.272+0000 7f6d5151c700 1 Processor -- start 2024-05-27T07:13:36.928 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:13:36.272+0000 7f6d5151c700 1 -- start start 2024-05-27T07:13:36.928 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:13:36.272+0000 7f6d5151c700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f6d4c07cc90 0x7f6d4c10eaf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:13:36.928 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:13:36.273+0000 7f6d5151c700 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x7f6d4c10f030 con 0x7f6d4c07cc90 2024-05-27T07:13:36.928 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:13:36.273+0000 7f6d4affd700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f6d4c07cc90 0x7f6d4c10eaf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:13:36.929 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:13:36.273+0000 7f6d4affd700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f6d4c07cc90 0x7f6d4c10eaf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.52:56470/0 (socket says 172.21.15.52:56470) 2024-05-27T07:13:36.929 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:13:36.273+0000 7f6d4affd700 1 -- 172.21.15.52:0/1765313257 learned_addr learned my addr 172.21.15.52:0/1765313257 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-05-27T07:13:36.929 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:13:36.273+0000 7f6d4affd700 1 -- 172.21.15.52:0/1765313257 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6d4c10f1b0 con 0x7f6d4c07cc90 2024-05-27T07:13:36.929 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:13:36.273+0000 7f6d4affd700 1 --2- 172.21.15.52:0/1765313257 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f6d4c07cc90 0x7f6d4c10eaf0 secure :-1 s=READY pgs=3 cs=0 l=1 rev1=1 crypto rx=0x7f6d3c0098a0 tx=0x7f6d3c009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1a4d2a6e97056db8 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:13:36.929 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:13:36.274+0000 7f6d49ffb700 1 -- 172.21.15.52:0/1765313257 <== mon.0 v2:172.21.15.52:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f6d3c004160 con 0x7f6d4c07cc90 2024-05-27T07:13:36.929 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:13:36.274+0000 7f6d49ffb700 1 -- 172.21.15.52:0/1765313257 <== mon.0 v2:172.21.15.52:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6d3c004300 con 0x7f6d4c07cc90 2024-05-27T07:13:36.929 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:13:36.274+0000 7f6d49ffb700 1 -- 172.21.15.52:0/1765313257 <== mon.0 v2:172.21.15.52:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f6d3c0036a0 con 0x7f6d4c07cc90 2024-05-27T07:13:36.929 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:13:36.274+0000 7f6d5151c700 1 -- 172.21.15.52:0/1765313257 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f6d4c07cc90 msgr2=0x7f6d4c10eaf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:13:36.929 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:13:36.274+0000 7f6d5151c700 1 --2- 172.21.15.52:0/1765313257 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f6d4c07cc90 0x7f6d4c10eaf0 secure :-1 s=READY pgs=3 cs=0 l=1 rev1=1 crypto rx=0x7f6d3c0098a0 tx=0x7f6d3c009bb0 comp rx=0 tx=0).stop 2024-05-27T07:13:36.929 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:13:36.274+0000 7f6d5151c700 1 -- 172.21.15.52:0/1765313257 shutdown_connections 2024-05-27T07:13:36.929 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:13:36.274+0000 7f6d5151c700 1 --2- 172.21.15.52:0/1765313257 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f6d4c07cc90 0x7f6d4c10eaf0 unknown :-1 s=CLOSED pgs=3 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:13:36.929 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:13:36.274+0000 7f6d5151c700 1 -- 172.21.15.52:0/1765313257 >> 172.21.15.52:0/1765313257 conn(0x7f6d4c1040e0 msgr2=0x7f6d4c106570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:13:36.930 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:13:36.274+0000 7f6d5151c700 1 -- 172.21.15.52:0/1765313257 shutdown_connections 2024-05-27T07:13:36.930 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:13:36.274+0000 7f6d5151c700 1 -- 172.21.15.52:0/1765313257 wait complete. 2024-05-27T07:13:36.930 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:13:36.275+0000 7f6d5151c700 1 Processor -- start 2024-05-27T07:13:36.930 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:13:36.275+0000 7f6d5151c700 1 -- start start 2024-05-27T07:13:36.930 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:13:36.275+0000 7f6d5151c700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f6d4c07cc90 0x7f6d4c19f840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:13:36.930 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:13:36.275+0000 7f6d5151c700 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x7f6d4c10f5c0 con 0x7f6d4c07cc90 2024-05-27T07:13:36.930 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:13:36.276+0000 7f6d4affd700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f6d4c07cc90 0x7f6d4c19f840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:13:36.930 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:13:36.276+0000 7f6d4affd700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f6d4c07cc90 0x7f6d4c19f840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.52:56478/0 (socket says 172.21.15.52:56478) 2024-05-27T07:13:36.930 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:13:36.276+0000 7f6d4affd700 1 -- 172.21.15.52:0/1711580040 learned_addr learned my addr 172.21.15.52:0/1711580040 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-05-27T07:13:36.931 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:13:36.276+0000 7f6d4affd700 1 -- 172.21.15.52:0/1711580040 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6d4c19fd80 con 0x7f6d4c07cc90 2024-05-27T07:13:36.931 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:13:36.276+0000 7f6d4affd700 1 --2- 172.21.15.52:0/1711580040 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f6d4c07cc90 0x7f6d4c19f840 secure :-1 s=READY pgs=4 cs=0 l=1 rev1=1 crypto rx=0x7f6d3c000c00 tx=0x7f6d3c004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:13:36.931 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:13:36.276+0000 7f6d43fff700 1 -- 172.21.15.52:0/1711580040 <== mon.0 v2:172.21.15.52:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f6d3c003db0 con 0x7f6d4c07cc90 2024-05-27T07:13:36.931 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:13:36.276+0000 7f6d43fff700 1 -- 172.21.15.52:0/1711580040 <== mon.0 v2:172.21.15.52:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6d3c011630 con 0x7f6d4c07cc90 2024-05-27T07:13:36.931 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:13:36.276+0000 7f6d5151c700 1 -- 172.21.15.52:0/1711580040 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6d4c1a0010 con 0x7f6d4c07cc90 2024-05-27T07:13:36.931 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:13:36.276+0000 7f6d43fff700 1 -- 172.21.15.52:0/1711580040 <== mon.0 v2:172.21.15.52:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f6d3c011950 con 0x7f6d4c07cc90 2024-05-27T07:13:36.931 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:13:36.276+0000 7f6d5151c700 1 -- 172.21.15.52:0/1711580040 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6d4c1a2c20 con 0x7f6d4c07cc90 2024-05-27T07:13:36.931 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:13:36.277+0000 7f6d43fff700 1 -- 172.21.15.52:0/1711580040 <== mon.0 v2:172.21.15.52:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7f6d3c011af0 con 0x7f6d4c07cc90 2024-05-27T07:13:36.931 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:13:36.277+0000 7f6d43fff700 1 -- 172.21.15.52:0/1711580040 <== mon.0 v2:172.21.15.52:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7f6d3c01c450 con 0x7f6d4c07cc90 2024-05-27T07:13:36.931 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:13:36.277+0000 7f6d5151c700 1 -- 172.21.15.52:0/1711580040 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6d4c064e10 con 0x7f6d4c07cc90 2024-05-27T07:13:36.932 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:13:36.280+0000 7f6d43fff700 1 -- 172.21.15.52:0/1711580040 <== mon.0 v2:172.21.15.52:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73389 (secure 0 0 0) 0x7f6d3c032020 con 0x7f6d4c07cc90 2024-05-27T07:13:36.932 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:13:36.351+0000 7f6d5151c700 1 -- 172.21.15.52:0/1711580040 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_command({"prefix": "config assimilate-conf"} v 0) -- 0x7f6d4c02e8e0 con 0x7f6d4c07cc90 2024-05-27T07:13:36.932 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:13:36.355+0000 7f6d43fff700 1 -- 172.21.15.52:0/1711580040 <== mon.0 v2:172.21.15.52:3300/0 7 ==== mon_command_ack([{"prefix": "config assimilate-conf"}]=0 v2) ==== 70+0+465 (secure 0 0 0) 0x7f6d3c004260 con 0x7f6d4c07cc90 2024-05-27T07:13:36.932 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:13:36.355+0000 7f6d5151c700 1 -- 172.21.15.52:0/1711580040 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f6d4c07cc90 msgr2=0x7f6d4c19f840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:13:36.932 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:13:36.356+0000 7f6d5151c700 1 --2- 172.21.15.52:0/1711580040 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f6d4c07cc90 0x7f6d4c19f840 secure :-1 s=READY pgs=4 cs=0 l=1 rev1=1 crypto rx=0x7f6d3c000c00 tx=0x7f6d3c004750 comp rx=0 tx=0).stop 2024-05-27T07:13:36.932 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:13:36.356+0000 7f6d5151c700 1 -- 172.21.15.52:0/1711580040 shutdown_connections 2024-05-27T07:13:36.932 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:13:36.356+0000 7f6d5151c700 1 --2- 172.21.15.52:0/1711580040 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f6d4c07cc90 0x7f6d4c19f840 unknown :-1 s=CLOSED pgs=4 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:13:36.932 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:13:36.356+0000 7f6d5151c700 1 -- 172.21.15.52:0/1711580040 >> 172.21.15.52:0/1711580040 conn(0x7f6d4c1040e0 msgr2=0x7f6d4c1062a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:13:36.932 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:13:36.356+0000 7f6d5151c700 1 -- 172.21.15.52:0/1711580040 shutdown_connections 2024-05-27T07:13:36.932 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:13:36.356+0000 7f6d5151c700 1 -- 172.21.15.52:0/1711580040 wait complete. 2024-05-27T07:13:36.932 INFO:teuthology.orchestra.run.smithi052.stdout:Generating new minimal ceph.conf... 2024-05-27T07:13:38.040 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:13:37 smithi052 ceph-mon[31583]: from='client.? 172.21.15.52:0/1296498342' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T07:13:38.048 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:13:37.378+0000 7f160f48d700 1 Processor -- start 2024-05-27T07:13:38.048 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:13:37.378+0000 7f160f48d700 1 -- start start 2024-05-27T07:13:38.048 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:13:37.378+0000 7f160f48d700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f1608108af0 0x7f1608108f00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:13:38.048 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:13:37.378+0000 7f160f48d700 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x7f16081094f0 con 0x7f1608108af0 2024-05-27T07:13:38.048 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:13:37.378+0000 7f160d229700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f1608108af0 0x7f1608108f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:13:38.048 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:13:37.378+0000 7f160d229700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f1608108af0 0x7f1608108f00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.52:56492/0 (socket says 172.21.15.52:56492) 2024-05-27T07:13:38.048 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:13:37.378+0000 7f160d229700 1 -- 172.21.15.52:0/4194656988 learned_addr learned my addr 172.21.15.52:0/4194656988 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-05-27T07:13:38.048 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:13:37.379+0000 7f160d229700 1 -- 172.21.15.52:0/4194656988 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1608109c20 con 0x7f1608108af0 2024-05-27T07:13:38.048 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:13:37.379+0000 7f160d229700 1 --2- 172.21.15.52:0/4194656988 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f1608108af0 0x7f1608108f00 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7f16040098a0 tx=0x7f1604009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c4725e374f4ac82 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:13:38.048 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:13:37.379+0000 7f15fffff700 1 -- 172.21.15.52:0/4194656988 <== mon.0 v2:172.21.15.52:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f1604004160 con 0x7f1608108af0 2024-05-27T07:13:38.048 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:13:37.379+0000 7f15fffff700 1 -- 172.21.15.52:0/4194656988 <== mon.0 v2:172.21.15.52:3300/0 2 ==== config(26 keys) ==== 1070+0+0 (secure 0 0 0) 0x7f1604004300 con 0x7f1608108af0 2024-05-27T07:13:38.049 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:13:37.379+0000 7f15fffff700 1 -- 172.21.15.52:0/4194656988 <== mon.0 v2:172.21.15.52:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f1604011af0 con 0x7f1608108af0 2024-05-27T07:13:38.049 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:13:37.380+0000 7f160f48d700 1 -- 172.21.15.52:0/4194656988 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f1608108af0 msgr2=0x7f1608108f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:13:38.049 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:13:37.380+0000 7f160f48d700 1 --2- 172.21.15.52:0/4194656988 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f1608108af0 0x7f1608108f00 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7f16040098a0 tx=0x7f1604009bb0 comp rx=0 tx=0).stop 2024-05-27T07:13:38.049 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:13:37.380+0000 7f160f48d700 1 -- 172.21.15.52:0/4194656988 shutdown_connections 2024-05-27T07:13:38.049 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:13:37.380+0000 7f160f48d700 1 --2- 172.21.15.52:0/4194656988 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f1608108af0 0x7f1608108f00 unknown :-1 s=CLOSED pgs=5 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:13:38.049 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:13:37.380+0000 7f160f48d700 1 -- 172.21.15.52:0/4194656988 >> 172.21.15.52:0/4194656988 conn(0x7f1608104150 msgr2=0x7f16081065e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:13:38.049 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:13:37.380+0000 7f160f48d700 1 -- 172.21.15.52:0/4194656988 shutdown_connections 2024-05-27T07:13:38.049 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:13:37.380+0000 7f160f48d700 1 -- 172.21.15.52:0/4194656988 wait complete. 2024-05-27T07:13:38.049 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:13:37.381+0000 7f160f48d700 1 Processor -- start 2024-05-27T07:13:38.049 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:13:37.381+0000 7f160f48d700 1 -- start start 2024-05-27T07:13:38.049 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:13:37.381+0000 7f160f48d700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f1608108af0 0x7f160819bd90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:13:38.049 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:13:37.381+0000 7f160f48d700 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x7f160807cb80 con 0x7f1608108af0 2024-05-27T07:13:38.049 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:13:37.381+0000 7f160d229700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f1608108af0 0x7f160819bd90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:13:38.050 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:13:37.381+0000 7f160d229700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f1608108af0 0x7f160819bd90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.52:56496/0 (socket says 172.21.15.52:56496) 2024-05-27T07:13:38.050 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:13:37.381+0000 7f160d229700 1 -- 172.21.15.52:0/1296498342 learned_addr learned my addr 172.21.15.52:0/1296498342 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-05-27T07:13:38.050 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:13:37.381+0000 7f160d229700 1 -- 172.21.15.52:0/1296498342 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f160819c2d0 con 0x7f1608108af0 2024-05-27T07:13:38.050 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:13:37.382+0000 7f160d229700 1 --2- 172.21.15.52:0/1296498342 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f1608108af0 0x7f160819bd90 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7f1604011890 tx=0x7f1604011970 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:13:38.050 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:13:37.382+0000 7f15fe7fc700 1 -- 172.21.15.52:0/1296498342 <== mon.0 v2:172.21.15.52:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f16040039e0 con 0x7f1608108af0 2024-05-27T07:13:38.050 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:13:37.382+0000 7f15fe7fc700 1 -- 172.21.15.52:0/1296498342 <== mon.0 v2:172.21.15.52:3300/0 2 ==== config(26 keys) ==== 1070+0+0 (secure 0 0 0) 0x7f1604003b80 con 0x7f1608108af0 2024-05-27T07:13:38.050 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:13:37.382+0000 7f15fe7fc700 1 -- 172.21.15.52:0/1296498342 <== mon.0 v2:172.21.15.52:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f1604024750 con 0x7f1608108af0 2024-05-27T07:13:38.050 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:13:37.382+0000 7f160f48d700 1 -- 172.21.15.52:0/1296498342 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f160819c560 con 0x7f1608108af0 2024-05-27T07:13:38.050 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:13:37.382+0000 7f160f48d700 1 -- 172.21.15.52:0/1296498342 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f160819f170 con 0x7f1608108af0 2024-05-27T07:13:38.050 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:13:37.382+0000 7f15fe7fc700 1 -- 172.21.15.52:0/1296498342 <== mon.0 v2:172.21.15.52:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7f1604022020 con 0x7f1608108af0 2024-05-27T07:13:38.050 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:13:37.382+0000 7f15fe7fc700 1 -- 172.21.15.52:0/1296498342 <== mon.0 v2:172.21.15.52:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7f160402dcf0 con 0x7f1608108af0 2024-05-27T07:13:38.050 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:13:37.383+0000 7f160f48d700 1 -- 172.21.15.52:0/1296498342 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1608064e10 con 0x7f1608108af0 2024-05-27T07:13:38.050 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:13:37.385+0000 7f15fe7fc700 1 -- 172.21.15.52:0/1296498342 <== mon.0 v2:172.21.15.52:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73389 (secure 0 0 0) 0x7f160401ab50 con 0x7f1608108af0 2024-05-27T07:13:38.051 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:13:37.456+0000 7f160f48d700 1 -- 172.21.15.52:0/1296498342 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_command({"prefix": "config generate-minimal-conf"} v 0) -- 0x7f1608194fa0 con 0x7f1608108af0 2024-05-27T07:13:38.051 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:13:37.456+0000 7f15fe7fc700 1 -- 172.21.15.52:0/1296498342 <== mon.0 v2:172.21.15.52:3300/0 7 ==== mon_command_ack([{"prefix": "config generate-minimal-conf"}]=0 v2) ==== 76+0+175 (secure 0 0 0) 0x7f160402d3f0 con 0x7f1608108af0 2024-05-27T07:13:38.051 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:13:37.457+0000 7f160f48d700 1 -- 172.21.15.52:0/1296498342 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f1608108af0 msgr2=0x7f160819bd90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:13:38.051 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:13:37.457+0000 7f160f48d700 1 --2- 172.21.15.52:0/1296498342 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f1608108af0 0x7f160819bd90 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7f1604011890 tx=0x7f1604011970 comp rx=0 tx=0).stop 2024-05-27T07:13:38.051 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:13:37.457+0000 7f160f48d700 1 -- 172.21.15.52:0/1296498342 shutdown_connections 2024-05-27T07:13:38.051 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:13:37.457+0000 7f160f48d700 1 --2- 172.21.15.52:0/1296498342 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f1608108af0 0x7f160819bd90 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:13:38.051 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:13:37.457+0000 7f160f48d700 1 -- 172.21.15.52:0/1296498342 >> 172.21.15.52:0/1296498342 conn(0x7f1608104150 msgr2=0x7f1608192060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:13:38.051 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:13:37.457+0000 7f160f48d700 1 -- 172.21.15.52:0/1296498342 shutdown_connections 2024-05-27T07:13:38.051 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:13:37.457+0000 7f160f48d700 1 -- 172.21.15.52:0/1296498342 wait complete. 2024-05-27T07:13:38.051 INFO:teuthology.orchestra.run.smithi052.stdout:Restarting the monitor... 2024-05-27T07:13:38.320 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:13:38 smithi052 systemd[1]: Stopping Ceph mon.a for 73df2636-1bf8-11ef-bc9b-c7b262605968... 2024-05-27T07:13:38.730 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:13:38 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mon-a[31579]: 2024-05-27T07:13:38.328+0000 7fe475502700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.a -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-05-27T07:13:38.730 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:13:38 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mon-a[31579]: 2024-05-27T07:13:38.328+0000 7fe475502700 -1 mon.a@0(leader) e1 *** Got Signal Terminated *** 2024-05-27T07:13:38.730 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:13:38 smithi052 podman[31827]: 2024-05-27 07:13:38.513516961 +0000 UTC m=+0.281956199 container died 121b4b6fe9e254c5416f6e128eb838c787379c40b78b85719fd361b688d4f544 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:38f9d900c83c63799fbdbe61acc9a11b0d3554a6, name=ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mon-a, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240521, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, GIT_CLEAN=True, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, ceph=True) 2024-05-27T07:13:39.230 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:13:38 smithi052 podman[31827]: 2024-05-27 07:13:38.853444734 +0000 UTC m=+0.621883971 container cleanup 121b4b6fe9e254c5416f6e128eb838c787379c40b78b85719fd361b688d4f544 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:38f9d900c83c63799fbdbe61acc9a11b0d3554a6, name=ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mon-a, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240521, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD) 2024-05-27T07:13:39.230 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:13:38 smithi052 bash[31827]: ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mon-a 2024-05-27T07:13:39.230 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:13:39 smithi052 podman[31838]: 2024-05-27 07:13:39.020811835 +0000 UTC m=+0.499828950 container remove 121b4b6fe9e254c5416f6e128eb838c787379c40b78b85719fd361b688d4f544 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:38f9d900c83c63799fbdbe61acc9a11b0d3554a6, name=ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mon-a, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.build-date=20240521, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, org.label-schema.vendor=CentOS, ceph=True, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True) 2024-05-27T07:13:39.564 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:13:39 smithi052 systemd[1]: ceph-73df2636-1bf8-11ef-bc9b-c7b262605968@mon.a.service: Deactivated successfully. 2024-05-27T07:13:39.564 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:13:39 smithi052 systemd[1]: Stopped Ceph mon.a for 73df2636-1bf8-11ef-bc9b-c7b262605968. 2024-05-27T07:13:39.564 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:13:39 smithi052 systemd[1]: Starting Ceph mon.a for 73df2636-1bf8-11ef-bc9b-c7b262605968... 2024-05-27T07:13:39.564 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:13:39 smithi052 podman[31936]: 2024-05-27 07:13:39.471514733 +0000 UTC m=+0.097546157 container create 2dc34a76e3c22634d5163db18519ee0c400f5934e46c39548ccebaef4627c41c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:38f9d900c83c63799fbdbe61acc9a11b0d3554a6, name=ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mon-a, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240521, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, maintainer=Guillaume Abrioux , ceph=True, GIT_BRANCH=HEAD, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image) 2024-05-27T07:13:39.564 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:13:39 smithi052 podman[31936]: 2024-05-27 07:13:39.391370413 +0000 UTC m=+0.017401851 image pull e36ea4fc8a5807e8d2701370e350a0de3856538797f7fba64cfce7e344bd6799 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:38f9d900c83c63799fbdbe61acc9a11b0d3554a6 2024-05-27T07:13:39.776 INFO:teuthology.orchestra.run.smithi052.stdout:Setting public_network to 172.21.0.0/20 in mon config section 2024-05-27T07:13:39.981 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:13:39 smithi052 podman[31936]: 2024-05-27 07:13:39.673797275 +0000 UTC m=+0.299828713 container init 2dc34a76e3c22634d5163db18519ee0c400f5934e46c39548ccebaef4627c41c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:38f9d900c83c63799fbdbe61acc9a11b0d3554a6, name=ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mon-a, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.build-date=20240521, GIT_CLEAN=True, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=) 2024-05-27T07:13:39.981 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:13:39 smithi052 podman[31936]: 2024-05-27 07:13:39.677495695 +0000 UTC m=+0.303527121 container start 2dc34a76e3c22634d5163db18519ee0c400f5934e46c39548ccebaef4627c41c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:38f9d900c83c63799fbdbe61acc9a11b0d3554a6, name=ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mon-a, ceph=True, org.label-schema.build-date=20240521, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, GIT_CLEAN=True, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, org.label-schema.name=CentOS Stream 8 Base Image) 2024-05-27T07:13:39.981 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:13:39 smithi052 ceph-mon[31955]: set uid:gid to 167:167 (ceph:ceph) 2024-05-27T07:13:39.981 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:13:39 smithi052 ceph-mon[31955]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev), process ceph-mon, pid 2 2024-05-27T07:13:39.981 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:13:39 smithi052 ceph-mon[31955]: pidfile_write: ignore empty --pid-file 2024-05-27T07:13:39.981 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:13:39 smithi052 ceph-mon[31955]: load: jerasure load: lrc 2024-05-27T07:13:39.981 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:13:39 smithi052 ceph-mon[31955]: rocksdb: RocksDB version: 7.9.2 2024-05-27T07:13:39.981 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:13:39 smithi052 ceph-mon[31955]: rocksdb: Git sha 0 2024-05-27T07:13:39.981 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:13:39 smithi052 ceph-mon[31955]: rocksdb: Compile date 2024-05-21 07:00:12 2024-05-27T07:13:39.981 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:13:39 smithi052 ceph-mon[31955]: rocksdb: DB SUMMARY 2024-05-27T07:13:39.982 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:13:39 smithi052 ceph-mon[31955]: rocksdb: DB Session ID: OK4L6G0KQSHT8A79WEQ9 2024-05-27T07:13:39.982 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:13:39 smithi052 ceph-mon[31955]: rocksdb: CURRENT file: CURRENT 2024-05-27T07:13:39.982 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:13:39 smithi052 ceph-mon[31955]: rocksdb: IDENTITY file: IDENTITY 2024-05-27T07:13:39.982 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:13:39 smithi052 ceph-mon[31955]: rocksdb: MANIFEST file: MANIFEST-000010 size: 179 Bytes 2024-05-27T07:13:39.982 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:13:39 smithi052 ceph-mon[31955]: rocksdb: SST files in /var/lib/ceph/mon/ceph-a/store.db dir, Total Num: 1, files: 000008.sst 2024-05-27T07:13:39.982 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:13:39 smithi052 ceph-mon[31955]: rocksdb: Write Ahead Log file in /var/lib/ceph/mon/ceph-a/store.db: 000009.log size: 105120 ; 2024-05-27T07:13:39.982 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:13:39 smithi052 ceph-mon[31955]: rocksdb: Options.error_if_exists: 0 2024-05-27T07:13:39.982 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:13:39 smithi052 ceph-mon[31955]: rocksdb: Options.create_if_missing: 0 2024-05-27T07:13:39.982 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:13:39 smithi052 ceph-mon[31955]: rocksdb: Options.paranoid_checks: 1 2024-05-27T07:13:39.982 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:13:39 smithi052 ceph-mon[31955]: rocksdb: Options.flush_verify_memtable_count: 1 2024-05-27T07:13:39.983 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:13:39 smithi052 ceph-mon[31955]: rocksdb: Options.track_and_verify_wals_in_manifest: 0 2024-05-27T07:13:39.983 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:13:39 smithi052 ceph-mon[31955]: rocksdb: Options.verify_sst_unique_id_in_manifest: 1 2024-05-27T07:13:39.983 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:13:39 smithi052 ceph-mon[31955]: rocksdb: Options.env: 0x55c72a514500 2024-05-27T07:13:39.983 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:13:39 smithi052 ceph-mon[31955]: rocksdb: Options.fs: PosixFileSystem 2024-05-27T07:13:39.983 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:13:39 smithi052 ceph-mon[31955]: rocksdb: Options.info_log: 0x55c72cab3fe0 2024-05-27T07:13:39.983 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:13:39 smithi052 ceph-mon[31955]: rocksdb: Options.max_file_opening_threads: 16 2024-05-27T07:13:39.984 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:13:39 smithi052 ceph-mon[31955]: rocksdb: Options.statistics: (nil) 2024-05-27T07:13:39.984 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:13:39 smithi052 ceph-mon[31955]: rocksdb: Options.use_fsync: 0 2024-05-27T07:13:39.984 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:13:39 smithi052 ceph-mon[31955]: rocksdb: Options.max_log_file_size: 0 2024-05-27T07:13:39.985 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:13:39 smithi052 ceph-mon[31955]: rocksdb: Options.max_manifest_file_size: 1073741824 2024-05-27T07:13:39.985 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:13:39 smithi052 ceph-mon[31955]: rocksdb: Options.log_file_time_to_roll: 0 2024-05-27T07:13:39.985 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:13:39 smithi052 ceph-mon[31955]: rocksdb: Options.keep_log_file_num: 1000 2024-05-27T07:13:39.985 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:13:39 smithi052 ceph-mon[31955]: rocksdb: Options.recycle_log_file_num: 0 2024-05-27T07:13:39.985 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:13:39 smithi052 ceph-mon[31955]: rocksdb: Options.allow_fallocate: 1 2024-05-27T07:13:39.985 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:13:39 smithi052 ceph-mon[31955]: rocksdb: Options.allow_mmap_reads: 0 2024-05-27T07:13:39.985 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:13:39 smithi052 ceph-mon[31955]: rocksdb: Options.allow_mmap_writes: 0 2024-05-27T07:13:39.985 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:13:39 smithi052 ceph-mon[31955]: rocksdb: Options.use_direct_reads: 0 2024-05-27T07:13:39.985 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:13:39 smithi052 ceph-mon[31955]: rocksdb: Options.use_direct_io_for_flush_and_compaction: 0 2024-05-27T07:13:39.985 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:13:39 smithi052 ceph-mon[31955]: rocksdb: Options.create_missing_column_families: 0 2024-05-27T07:13:39.985 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:13:39 smithi052 ceph-mon[31955]: rocksdb: Options.db_log_dir: 2024-05-27T07:13:39.985 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:13:39 smithi052 ceph-mon[31955]: rocksdb: Options.wal_dir: 2024-05-27T07:13:39.985 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:13:39 smithi052 ceph-mon[31955]: rocksdb: Options.table_cache_numshardbits: 6 2024-05-27T07:13:39.985 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:13:39 smithi052 ceph-mon[31955]: rocksdb: Options.WAL_ttl_seconds: 0 2024-05-27T07:13:39.985 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:13:39 smithi052 ceph-mon[31955]: rocksdb: Options.WAL_size_limit_MB: 0 2024-05-27T07:13:39.986 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:13:39 smithi052 ceph-mon[31955]: rocksdb: Options.max_write_batch_group_size_bytes: 1048576 2024-05-27T07:13:39.986 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:13:39 smithi052 ceph-mon[31955]: rocksdb: Options.manifest_preallocation_size: 4194304 2024-05-27T07:13:39.986 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:13:39 smithi052 ceph-mon[31955]: rocksdb: Options.is_fd_close_on_exec: 1 2024-05-27T07:13:39.986 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:13:39 smithi052 ceph-mon[31955]: rocksdb: Options.advise_random_on_open: 1 2024-05-27T07:13:39.986 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:13:39 smithi052 ceph-mon[31955]: rocksdb: Options.db_write_buffer_size: 0 2024-05-27T07:13:39.986 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:13:39 smithi052 ceph-mon[31955]: rocksdb: Options.write_buffer_manager: 0x55c72ca848c0 2024-05-27T07:13:39.986 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:13:39 smithi052 ceph-mon[31955]: rocksdb: Options.access_hint_on_compaction_start: 1 2024-05-27T07:13:39.986 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:13:39 smithi052 ceph-mon[31955]: rocksdb: Options.random_access_max_buffer_size: 1048576 2024-05-27T07:13:39.986 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:13:39 smithi052 ceph-mon[31955]: rocksdb: Options.use_adaptive_mutex: 0 2024-05-27T07:13:39.986 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:13:39 smithi052 ceph-mon[31955]: rocksdb: Options.rate_limiter: (nil) 2024-05-27T07:13:39.986 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:13:39 smithi052 ceph-mon[31955]: rocksdb: Options.sst_file_manager.rate_bytes_per_sec: 0 2024-05-27T07:13:39.986 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:13:39 smithi052 ceph-mon[31955]: rocksdb: Options.wal_recovery_mode: 2 2024-05-27T07:13:39.986 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:13:39 smithi052 ceph-mon[31955]: rocksdb: Options.enable_thread_tracking: 0 2024-05-27T07:13:39.986 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:13:39 smithi052 ceph-mon[31955]: rocksdb: Options.enable_pipelined_write: 0 2024-05-27T07:13:39.987 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:13:39 smithi052 ceph-mon[31955]: rocksdb: Options.unordered_write: 0 2024-05-27T07:13:39.987 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:13:39 smithi052 ceph-mon[31955]: rocksdb: Options.allow_concurrent_memtable_write: 1 2024-05-27T07:13:39.987 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:13:39 smithi052 ceph-mon[31955]: rocksdb: Options.enable_write_thread_adaptive_yield: 1 2024-05-27T07:13:39.987 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:13:39 smithi052 ceph-mon[31955]: rocksdb: Options.write_thread_max_yield_usec: 100 2024-05-27T07:13:39.987 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:13:39 smithi052 ceph-mon[31955]: rocksdb: Options.write_thread_slow_yield_usec: 3 2024-05-27T07:13:39.987 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:13:39 smithi052 ceph-mon[31955]: rocksdb: Options.row_cache: None 2024-05-27T07:13:39.987 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:13:39 smithi052 ceph-mon[31955]: rocksdb: Options.wal_filter: None 2024-05-27T07:13:39.987 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:13:39 smithi052 ceph-mon[31955]: rocksdb: Options.avoid_flush_during_recovery: 0 2024-05-27T07:13:39.987 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:13:39 smithi052 ceph-mon[31955]: rocksdb: Options.allow_ingest_behind: 0 2024-05-27T07:13:39.987 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:13:39 smithi052 ceph-mon[31955]: rocksdb: Options.two_write_queues: 0 2024-05-27T07:13:39.987 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:13:39 smithi052 ceph-mon[31955]: rocksdb: Options.manual_wal_flush: 0 2024-05-27T07:13:39.987 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:13:39 smithi052 ceph-mon[31955]: rocksdb: Options.wal_compression: 0 2024-05-27T07:13:39.987 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:13:39 smithi052 ceph-mon[31955]: rocksdb: Options.atomic_flush: 0 2024-05-27T07:13:39.987 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:13:39 smithi052 ceph-mon[31955]: rocksdb: Options.avoid_unnecessary_blocking_io: 0 2024-05-27T07:13:39.988 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:13:39 smithi052 ceph-mon[31955]: rocksdb: Options.persist_stats_to_disk: 0 2024-05-27T07:13:39.988 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:13:39 smithi052 ceph-mon[31955]: rocksdb: Options.write_dbid_to_manifest: 0 2024-05-27T07:13:39.988 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:13:39 smithi052 ceph-mon[31955]: rocksdb: Options.log_readahead_size: 0 2024-05-27T07:13:39.988 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:13:39 smithi052 ceph-mon[31955]: rocksdb: Options.file_checksum_gen_factory: Unknown 2024-05-27T07:13:39.988 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:13:39 smithi052 ceph-mon[31955]: rocksdb: Options.best_efforts_recovery: 0 2024-05-27T07:13:39.988 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:13:39 smithi052 ceph-mon[31955]: rocksdb: Options.max_bgerror_resume_count: 2147483647 2024-05-27T07:13:39.988 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:13:39 smithi052 ceph-mon[31955]: rocksdb: Options.bgerror_resume_retry_interval: 1000000 2024-05-27T07:13:39.988 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:13:39 smithi052 ceph-mon[31955]: rocksdb: Options.allow_data_in_errors: 0 2024-05-27T07:13:39.988 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:13:39 smithi052 ceph-mon[31955]: rocksdb: Options.db_host_id: __hostname__ 2024-05-27T07:13:39.988 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:13:39 smithi052 ceph-mon[31955]: rocksdb: Options.enforce_single_del_contracts: true 2024-05-27T07:13:39.988 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:13:39 smithi052 ceph-mon[31955]: rocksdb: Options.max_background_jobs: 2 2024-05-27T07:13:39.988 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:13:39 smithi052 ceph-mon[31955]: rocksdb: Options.max_background_compactions: -1 2024-05-27T07:13:39.988 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:13:39 smithi052 ceph-mon[31955]: rocksdb: Options.max_subcompactions: 1 2024-05-27T07:13:39.988 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:13:39 smithi052 ceph-mon[31955]: rocksdb: Options.avoid_flush_during_shutdown: 0 2024-05-27T07:13:39.988 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:13:39 smithi052 ceph-mon[31955]: rocksdb: Options.writable_file_max_buffer_size: 1048576 2024-05-27T07:13:39.989 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:13:39 smithi052 ceph-mon[31955]: rocksdb: Options.delayed_write_rate : 16777216 2024-05-27T07:13:39.989 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:13:39 smithi052 ceph-mon[31955]: rocksdb: Options.max_total_wal_size: 0 2024-05-27T07:13:39.989 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:13:39 smithi052 ceph-mon[31955]: rocksdb: Options.delete_obsolete_files_period_micros: 21600000000 2024-05-27T07:13:39.989 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:13:39 smithi052 ceph-mon[31955]: rocksdb: Options.stats_dump_period_sec: 600 2024-05-27T07:13:39.989 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:13:39 smithi052 ceph-mon[31955]: rocksdb: Options.stats_persist_period_sec: 600 2024-05-27T07:13:39.989 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:13:39 smithi052 ceph-mon[31955]: rocksdb: Options.stats_history_buffer_size: 1048576 2024-05-27T07:13:39.989 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:13:39 smithi052 ceph-mon[31955]: rocksdb: Options.max_open_files: -1 2024-05-27T07:13:39.989 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:13:39 smithi052 ceph-mon[31955]: rocksdb: Options.bytes_per_sync: 0 2024-05-27T07:13:39.989 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:13:39 smithi052 ceph-mon[31955]: rocksdb: Options.wal_bytes_per_sync: 0 2024-05-27T07:13:39.989 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:13:39 smithi052 ceph-mon[31955]: rocksdb: Options.strict_bytes_per_sync: 0 2024-05-27T07:13:39.989 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:13:39 smithi052 ceph-mon[31955]: rocksdb: Options.compaction_readahead_size: 0 2024-05-27T07:13:39.989 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:13:39 smithi052 ceph-mon[31955]: rocksdb: Options.max_background_flushes: -1 2024-05-27T07:13:39.989 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:13:39 smithi052 ceph-mon[31955]: rocksdb: Compression algorithms supported: 2024-05-27T07:13:39.989 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:13:39 smithi052 ceph-mon[31955]: rocksdb: kZSTDNotFinalCompression supported: 0 2024-05-27T07:13:39.989 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:13:39 smithi052 ceph-mon[31955]: rocksdb: kZSTD supported: 0 2024-05-27T07:13:39.990 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:13:39 smithi052 ceph-mon[31955]: rocksdb: kXpressCompression supported: 0 2024-05-27T07:13:39.990 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:13:39 smithi052 ceph-mon[31955]: rocksdb: kLZ4HCCompression supported: 1 2024-05-27T07:13:39.990 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:13:39 smithi052 ceph-mon[31955]: rocksdb: kZlibCompression supported: 1 2024-05-27T07:13:39.990 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:13:39 smithi052 ceph-mon[31955]: rocksdb: kSnappyCompression supported: 1 2024-05-27T07:13:39.990 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:13:39 smithi052 ceph-mon[31955]: rocksdb: kLZ4Compression supported: 1 2024-05-27T07:13:39.990 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:13:39 smithi052 ceph-mon[31955]: rocksdb: kBZip2Compression supported: 0 2024-05-27T07:13:39.990 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:13:39 smithi052 ceph-mon[31955]: rocksdb: Fast CRC32 supported: Supported on x86 2024-05-27T07:13:39.990 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:13:39 smithi052 ceph-mon[31955]: rocksdb: DMutex implementation: pthread_mutex_t 2024-05-27T07:13:39.990 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:13:39 smithi052 ceph-mon[31955]: rocksdb: [db/version_set.cc:5527] Recovering from manifest file: /var/lib/ceph/mon/ceph-a/store.db/MANIFEST-000010 2024-05-27T07:13:39.990 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:13:39 smithi052 ceph-mon[31955]: rocksdb: [db/column_family.cc:630] --------------- Options for column family [default]: 2024-05-27T07:13:39.990 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:13:39 smithi052 ceph-mon[31955]: rocksdb: Options.comparator: leveldb.BytewiseComparator 2024-05-27T07:13:39.990 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:13:39 smithi052 ceph-mon[31955]: rocksdb: Options.merge_operator: 2024-05-27T07:13:39.990 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:13:39 smithi052 ceph-mon[31955]: rocksdb: Options.compaction_filter: None 2024-05-27T07:13:39.990 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:13:39 smithi052 ceph-mon[31955]: rocksdb: Options.compaction_filter_factory: None 2024-05-27T07:13:39.991 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:13:39 smithi052 ceph-mon[31955]: rocksdb: Options.sst_partitioner_factory: None 2024-05-27T07:13:39.991 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:13:39 smithi052 ceph-mon[31955]: rocksdb: Options.memtable_factory: SkipListFactory 2024-05-27T07:13:39.991 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:13:39 smithi052 ceph-mon[31955]: rocksdb: Options.table_factory: BlockBasedTable 2024-05-27T07:13:39.991 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:13:39 smithi052 ceph-mon[31955]: rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x55c72cbba1c0) 2024-05-27T07:13:39.991 INFO:journalctl@ceph.mon.a.smithi052.stdout: cache_index_and_filter_blocks: 1 2024-05-27T07:13:39.991 INFO:journalctl@ceph.mon.a.smithi052.stdout: cache_index_and_filter_blocks_with_high_priority: 0 2024-05-27T07:13:39.991 INFO:journalctl@ceph.mon.a.smithi052.stdout: pin_l0_filter_and_index_blocks_in_cache: 0 2024-05-27T07:13:39.991 INFO:journalctl@ceph.mon.a.smithi052.stdout: pin_top_level_index_and_filter: 1 2024-05-27T07:13:39.991 INFO:journalctl@ceph.mon.a.smithi052.stdout: index_type: 0 2024-05-27T07:13:39.991 INFO:journalctl@ceph.mon.a.smithi052.stdout: data_block_index_type: 0 2024-05-27T07:13:39.991 INFO:journalctl@ceph.mon.a.smithi052.stdout: index_shortening: 1 2024-05-27T07:13:39.992 INFO:journalctl@ceph.mon.a.smithi052.stdout: data_block_hash_table_util_ratio: 0.750000 2024-05-27T07:13:39.992 INFO:journalctl@ceph.mon.a.smithi052.stdout: checksum: 4 2024-05-27T07:13:39.992 INFO:journalctl@ceph.mon.a.smithi052.stdout: no_block_cache: 0 2024-05-27T07:13:39.992 INFO:journalctl@ceph.mon.a.smithi052.stdout: block_cache: 0x55c72cab71f0 2024-05-27T07:13:39.992 INFO:journalctl@ceph.mon.a.smithi052.stdout: block_cache_name: BinnedLRUCache 2024-05-27T07:13:39.992 INFO:journalctl@ceph.mon.a.smithi052.stdout: block_cache_options: 2024-05-27T07:13:39.992 INFO:journalctl@ceph.mon.a.smithi052.stdout: capacity : 536870912 2024-05-27T07:13:39.992 INFO:journalctl@ceph.mon.a.smithi052.stdout: num_shard_bits : 4 2024-05-27T07:13:39.992 INFO:journalctl@ceph.mon.a.smithi052.stdout: strict_capacity_limit : 0 2024-05-27T07:13:39.992 INFO:journalctl@ceph.mon.a.smithi052.stdout: high_pri_pool_ratio: 0.000 2024-05-27T07:13:39.992 INFO:journalctl@ceph.mon.a.smithi052.stdout: block_cache_compressed: (nil) 2024-05-27T07:13:39.992 INFO:journalctl@ceph.mon.a.smithi052.stdout: persistent_cache: (nil) 2024-05-27T07:13:39.993 INFO:journalctl@ceph.mon.a.smithi052.stdout: block_size: 4096 2024-05-27T07:13:39.993 INFO:journalctl@ceph.mon.a.smithi052.stdout: block_size_deviation: 10 2024-05-27T07:13:39.993 INFO:journalctl@ceph.mon.a.smithi052.stdout: block_restart_interval: 16 2024-05-27T07:13:39.993 INFO:journalctl@ceph.mon.a.smithi052.stdout: index_block_restart_interval: 1 2024-05-27T07:13:39.993 INFO:journalctl@ceph.mon.a.smithi052.stdout: metadata_block_size: 4096 2024-05-27T07:13:39.993 INFO:journalctl@ceph.mon.a.smithi052.stdout: partition_filters: 0 2024-05-27T07:13:39.993 INFO:journalctl@ceph.mon.a.smithi052.stdout: use_delta_encoding: 1 2024-05-27T07:13:39.993 INFO:journalctl@ceph.mon.a.smithi052.stdout: filter_policy: bloomfilter 2024-05-27T07:13:39.993 INFO:journalctl@ceph.mon.a.smithi052.stdout: whole_key_filtering: 1 2024-05-27T07:13:39.993 INFO:journalctl@ceph.mon.a.smithi052.stdout: verify_compression: 0 2024-05-27T07:13:39.994 INFO:journalctl@ceph.mon.a.smithi052.stdout: read_amp_bytes_per_bit: 0 2024-05-27T07:13:39.994 INFO:journalctl@ceph.mon.a.smithi052.stdout: format_version: 5 2024-05-27T07:13:39.994 INFO:journalctl@ceph.mon.a.smithi052.stdout: enable_index_compression: 1 2024-05-27T07:13:39.994 INFO:journalctl@ceph.mon.a.smithi052.stdout: block_align: 0 2024-05-27T07:13:39.994 INFO:journalctl@ceph.mon.a.smithi052.stdout: max_auto_readahead_size: 262144 2024-05-27T07:13:39.994 INFO:journalctl@ceph.mon.a.smithi052.stdout: prepopulate_block_cache: 0 2024-05-27T07:13:39.994 INFO:journalctl@ceph.mon.a.smithi052.stdout: initial_auto_readahead_size: 8192 2024-05-27T07:13:39.994 INFO:journalctl@ceph.mon.a.smithi052.stdout: num_file_reads_for_auto_readahead: 2 2024-05-27T07:13:39.994 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:13:39 smithi052 ceph-mon[31955]: rocksdb: Options.write_buffer_size: 33554432 2024-05-27T07:13:39.994 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:13:39 smithi052 ceph-mon[31955]: rocksdb: Options.max_write_buffer_number: 2 2024-05-27T07:13:39.994 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:13:39 smithi052 ceph-mon[31955]: rocksdb: Options.compression: NoCompression 2024-05-27T07:13:39.994 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:13:39 smithi052 ceph-mon[31955]: rocksdb: Options.bottommost_compression: Disabled 2024-05-27T07:13:39.994 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:13:39 smithi052 ceph-mon[31955]: rocksdb: Options.prefix_extractor: nullptr 2024-05-27T07:13:39.994 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:13:39 smithi052 ceph-mon[31955]: rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr 2024-05-27T07:13:39.995 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:13:39 smithi052 ceph-mon[31955]: rocksdb: Options.num_levels: 7 2024-05-27T07:13:39.995 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:13:39 smithi052 ceph-mon[31955]: rocksdb: Options.min_write_buffer_number_to_merge: 1 2024-05-27T07:13:39.995 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:13:39 smithi052 ceph-mon[31955]: rocksdb: Options.max_write_buffer_number_to_maintain: 0 2024-05-27T07:13:39.995 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:13:39 smithi052 ceph-mon[31955]: rocksdb: Options.max_write_buffer_size_to_maintain: 0 2024-05-27T07:13:39.995 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:13:39 smithi052 ceph-mon[31955]: rocksdb: Options.bottommost_compression_opts.window_bits: -14 2024-05-27T07:13:39.995 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:13:39 smithi052 ceph-mon[31955]: rocksdb: Options.bottommost_compression_opts.level: 32767 2024-05-27T07:13:39.995 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:13:39 smithi052 ceph-mon[31955]: rocksdb: Options.bottommost_compression_opts.strategy: 0 2024-05-27T07:13:39.995 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:13:39 smithi052 ceph-mon[31955]: rocksdb: Options.bottommost_compression_opts.max_dict_bytes: 0 2024-05-27T07:13:39.995 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:13:39 smithi052 ceph-mon[31955]: rocksdb: Options.bottommost_compression_opts.zstd_max_train_bytes: 0 2024-05-27T07:13:39.995 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:13:39 smithi052 ceph-mon[31955]: rocksdb: Options.bottommost_compression_opts.parallel_threads: 1 2024-05-27T07:13:39.995 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:13:39 smithi052 ceph-mon[31955]: rocksdb: Options.bottommost_compression_opts.enabled: false 2024-05-27T07:13:39.995 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:13:39 smithi052 ceph-mon[31955]: rocksdb: Options.bottommost_compression_opts.max_dict_buffer_bytes: 0 2024-05-27T07:13:39.995 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:13:39 smithi052 ceph-mon[31955]: rocksdb: Options.bottommost_compression_opts.use_zstd_dict_trainer: true 2024-05-27T07:13:39.995 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:13:39 smithi052 ceph-mon[31955]: rocksdb: Options.compression_opts.window_bits: -14 2024-05-27T07:13:39.996 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:13:39 smithi052 ceph-mon[31955]: rocksdb: Options.compression_opts.level: 32767 2024-05-27T07:13:39.996 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:13:39 smithi052 ceph-mon[31955]: rocksdb: Options.compression_opts.strategy: 0 2024-05-27T07:13:39.996 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:13:39 smithi052 ceph-mon[31955]: rocksdb: Options.compression_opts.max_dict_bytes: 0 2024-05-27T07:13:39.996 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:13:39 smithi052 ceph-mon[31955]: rocksdb: Options.compression_opts.zstd_max_train_bytes: 0 2024-05-27T07:13:39.996 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:13:39 smithi052 ceph-mon[31955]: rocksdb: Options.compression_opts.use_zstd_dict_trainer: true 2024-05-27T07:13:39.996 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:13:39 smithi052 ceph-mon[31955]: rocksdb: Options.compression_opts.parallel_threads: 1 2024-05-27T07:13:39.996 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:13:39 smithi052 ceph-mon[31955]: rocksdb: Options.compression_opts.enabled: false 2024-05-27T07:13:39.996 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:13:39 smithi052 ceph-mon[31955]: rocksdb: Options.compression_opts.max_dict_buffer_bytes: 0 2024-05-27T07:13:39.996 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:13:39 smithi052 ceph-mon[31955]: rocksdb: Options.level0_file_num_compaction_trigger: 4 2024-05-27T07:13:39.996 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:13:39 smithi052 ceph-mon[31955]: rocksdb: Options.level0_slowdown_writes_trigger: 20 2024-05-27T07:13:39.996 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:13:39 smithi052 ceph-mon[31955]: rocksdb: Options.level0_stop_writes_trigger: 36 2024-05-27T07:13:39.996 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:13:39 smithi052 ceph-mon[31955]: rocksdb: Options.target_file_size_base: 67108864 2024-05-27T07:13:39.996 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:13:39 smithi052 ceph-mon[31955]: rocksdb: Options.target_file_size_multiplier: 1 2024-05-27T07:13:39.996 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:13:39 smithi052 ceph-mon[31955]: rocksdb: Options.max_bytes_for_level_base: 268435456 2024-05-27T07:13:39.997 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:13:39 smithi052 ceph-mon[31955]: rocksdb: Options.level_compaction_dynamic_level_bytes: 1 2024-05-27T07:13:39.997 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:13:39 smithi052 ceph-mon[31955]: rocksdb: Options.max_bytes_for_level_multiplier: 10.000000 2024-05-27T07:13:39.997 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:13:39 smithi052 ceph-mon[31955]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1 2024-05-27T07:13:39.997 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:13:39 smithi052 ceph-mon[31955]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1 2024-05-27T07:13:39.997 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:13:39 smithi052 ceph-mon[31955]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1 2024-05-27T07:13:39.997 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:13:39 smithi052 ceph-mon[31955]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1 2024-05-27T07:13:39.997 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:13:39 smithi052 ceph-mon[31955]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1 2024-05-27T07:13:39.997 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:13:39 smithi052 ceph-mon[31955]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1 2024-05-27T07:13:39.997 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:13:39 smithi052 ceph-mon[31955]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1 2024-05-27T07:13:39.997 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:13:39 smithi052 ceph-mon[31955]: rocksdb: Options.max_sequential_skip_in_iterations: 8 2024-05-27T07:13:39.997 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:13:39 smithi052 ceph-mon[31955]: rocksdb: Options.max_compaction_bytes: 1677721600 2024-05-27T07:13:39.997 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:13:39 smithi052 ceph-mon[31955]: rocksdb: Options.ignore_max_compaction_bytes_for_input: true 2024-05-27T07:13:39.997 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:13:39 smithi052 ceph-mon[31955]: rocksdb: Options.arena_block_size: 1048576 2024-05-27T07:13:39.997 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:13:39 smithi052 ceph-mon[31955]: rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736 2024-05-27T07:13:39.997 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:13:39 smithi052 ceph-mon[31955]: rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944 2024-05-27T07:13:39.998 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:13:39 smithi052 ceph-mon[31955]: rocksdb: Options.disable_auto_compactions: 0 2024-05-27T07:13:39.998 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:13:39 smithi052 ceph-mon[31955]: rocksdb: Options.compaction_style: kCompactionStyleLevel 2024-05-27T07:13:39.998 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:13:39 smithi052 ceph-mon[31955]: rocksdb: Options.compaction_pri: kMinOverlappingRatio 2024-05-27T07:13:39.998 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:13:39 smithi052 ceph-mon[31955]: rocksdb: Options.compaction_options_universal.size_ratio: 1 2024-05-27T07:13:39.998 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:13:39 smithi052 ceph-mon[31955]: rocksdb: Options.compaction_options_universal.min_merge_width: 2 2024-05-27T07:13:39.998 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:13:39 smithi052 ceph-mon[31955]: rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295 2024-05-27T07:13:39.998 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:13:39 smithi052 ceph-mon[31955]: rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200 2024-05-27T07:13:39.998 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:13:39 smithi052 ceph-mon[31955]: rocksdb: Options.compaction_options_universal.compression_size_percent: -1 2024-05-27T07:13:39.998 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:13:39 smithi052 ceph-mon[31955]: rocksdb: Options.compaction_options_universal.stop_style: kCompactionStopStyleTotalSize 2024-05-27T07:13:39.998 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:13:39 smithi052 ceph-mon[31955]: rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824 2024-05-27T07:13:39.998 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:13:39 smithi052 ceph-mon[31955]: rocksdb: Options.compaction_options_fifo.allow_compaction: 0 2024-05-27T07:13:39.998 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:13:39 smithi052 ceph-mon[31955]: rocksdb: Options.table_properties_collectors: CompactOnDeletionCollector (Sliding window size = 32768 Deletion trigger = 16384 Deletion ratio = 0); 2024-05-27T07:13:39.998 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:13:39 smithi052 ceph-mon[31955]: rocksdb: Options.inplace_update_support: 0 2024-05-27T07:13:39.999 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:13:39 smithi052 ceph-mon[31955]: rocksdb: Options.inplace_update_num_locks: 10000 2024-05-27T07:13:39.999 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:13:39 smithi052 ceph-mon[31955]: rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000 2024-05-27T07:13:39.999 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:13:39 smithi052 ceph-mon[31955]: rocksdb: Options.memtable_whole_key_filtering: 0 2024-05-27T07:13:39.999 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:13:39 smithi052 ceph-mon[31955]: rocksdb: Options.memtable_huge_page_size: 0 2024-05-27T07:13:39.999 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:13:39 smithi052 ceph-mon[31955]: rocksdb: Options.bloom_locality: 0 2024-05-27T07:13:39.999 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:13:39 smithi052 ceph-mon[31955]: rocksdb: Options.max_successive_merges: 0 2024-05-27T07:13:39.999 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:13:39 smithi052 ceph-mon[31955]: rocksdb: Options.optimize_filters_for_hits: 0 2024-05-27T07:13:39.999 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:13:39 smithi052 ceph-mon[31955]: rocksdb: Options.paranoid_file_checks: 0 2024-05-27T07:13:39.999 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:13:39 smithi052 ceph-mon[31955]: rocksdb: Options.force_consistency_checks: 1 2024-05-27T07:13:39.999 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:13:39 smithi052 ceph-mon[31955]: rocksdb: Options.report_bg_io_stats: 0 2024-05-27T07:13:39.999 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:13:39 smithi052 ceph-mon[31955]: rocksdb: Options.ttl: 2592000 2024-05-27T07:13:39.999 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:13:39 smithi052 ceph-mon[31955]: rocksdb: Options.periodic_compaction_seconds: 0 2024-05-27T07:13:40.000 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:13:39 smithi052 ceph-mon[31955]: rocksdb: Options.preclude_last_level_data_seconds: 0 2024-05-27T07:13:40.000 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:13:39 smithi052 ceph-mon[31955]: rocksdb: Options.preserve_internal_time_seconds: 0 2024-05-27T07:13:40.000 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:13:39 smithi052 ceph-mon[31955]: rocksdb: Options.enable_blob_files: false 2024-05-27T07:13:40.000 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:13:39 smithi052 ceph-mon[31955]: rocksdb: Options.min_blob_size: 0 2024-05-27T07:13:40.000 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:13:39 smithi052 ceph-mon[31955]: rocksdb: Options.blob_file_size: 268435456 2024-05-27T07:13:40.000 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:13:39 smithi052 ceph-mon[31955]: rocksdb: Options.blob_compression_type: NoCompression 2024-05-27T07:13:40.000 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:13:39 smithi052 ceph-mon[31955]: rocksdb: Options.enable_blob_garbage_collection: false 2024-05-27T07:13:40.000 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:13:39 smithi052 ceph-mon[31955]: rocksdb: Options.blob_garbage_collection_age_cutoff: 0.250000 2024-05-27T07:13:40.000 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:13:39 smithi052 ceph-mon[31955]: rocksdb: Options.blob_garbage_collection_force_threshold: 1.000000 2024-05-27T07:13:40.000 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:13:39 smithi052 ceph-mon[31955]: rocksdb: Options.blob_compaction_readahead_size: 0 2024-05-27T07:13:40.000 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:13:39 smithi052 ceph-mon[31955]: rocksdb: Options.blob_file_starting_level: 0 2024-05-27T07:13:40.000 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:13:39 smithi052 ceph-mon[31955]: rocksdb: Options.experimental_mempurge_threshold: 0.000000 2024-05-27T07:13:40.000 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:13:39 smithi052 ceph-mon[31955]: rocksdb: [db/version_set.cc:5566] Recovered from manifest file:/var/lib/ceph/mon/ceph-a/store.db/MANIFEST-000010 succeeded,manifest_file_number is 10, next_file_number is 12, last_sequence is 5, log_number is 5,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 5 2024-05-27T07:13:40.001 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:13:39 smithi052 ceph-mon[31955]: rocksdb: [db/version_set.cc:5581] Column family [default] (ID 0), log number is 5 2024-05-27T07:13:40.001 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:13:39 smithi052 ceph-mon[31955]: rocksdb: [db/db_impl/db_impl_open.cc:539] DB ID: 2d94a8e5-6bd2-4122-b597-1e9f7a0e7535 2024-05-27T07:13:40.001 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:13:39 smithi052 ceph-mon[31955]: rocksdb: EVENT_LOG_v1 {"time_micros": 1716794019718266, "job": 1, "event": "recovery_started", "wal_files": [9]} 2024-05-27T07:13:40.001 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:13:39 smithi052 ceph-mon[31955]: rocksdb: [db/db_impl/db_impl_open.cc:1043] Recovering log #9 mode 2 2024-05-27T07:13:40.001 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:13:39 smithi052 ceph-mon[31955]: rocksdb: EVENT_LOG_v1 {"time_micros": 1716794019719458, "cf_name": "default", "job": 1, "event": "table_file_creation", "file_number": 13, "file_size": 100287, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 8, "largest_seqno": 326, "table_properties": {"data_size": 98172, "index_size": 307, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 1, "index_value_is_delta_encoded": 1, "filter_size": 773, "raw_key_size": 14169, "raw_average_key_size": 49, "raw_value_size": 90456, "raw_average_value_size": 317, "num_data_blocks": 12, "num_entries": 285, "num_filter_entries": 285, "num_deletions": 3, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 0, "filter_policy": "bloomfilter", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "", "prefix_extractor_name": "nullptr", "property_collectors": "[CompactOnDeletionCollector]", "compression": "NoCompression", "compression_options": "window_bits=-14; level=32767; strategy=0; max_dict_bytes=0; zstd_max_train_bytes=0; enabled=0; max_dict_buffer_bytes=0; use_zstd_dict_trainer=1; ", "creation_time": 1716794019, "oldest_key_time": 0, "file_creation_time": 0, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "2d94a8e5-6bd2-4122-b597-1e9f7a0e7535", "db_session_id": "OK4L6G0KQSHT8A79WEQ9", "orig_file_number": 13, "seqno_to_time_mapping": "N/A"}} 2024-05-27T07:13:40.001 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:13:39 smithi052 ceph-mon[31955]: rocksdb: EVENT_LOG_v1 {"time_micros": 1716794019719545, "job": 1, "event": "recovery_finished"} 2024-05-27T07:13:40.001 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:13:39 smithi052 ceph-mon[31955]: rocksdb: [db/version_set.cc:5047] Creating manifest 15 2024-05-27T07:13:40.001 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:13:39 smithi052 ceph-mon[31955]: rocksdb: [file/delete_scheduler.cc:74] Deleted file /var/lib/ceph/mon/ceph-a/store.db/000009.log immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024-05-27T07:13:40.001 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:13:39 smithi052 ceph-mon[31955]: rocksdb: [db/db_impl/db_impl_open.cc:1987] SstFileManager instance 0x55c72ca9bc00 2024-05-27T07:13:40.001 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:13:39 smithi052 ceph-mon[31955]: rocksdb: DB pointer 0x55c73188e000 2024-05-27T07:13:40.001 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:13:39 smithi052 ceph-mon[31955]: rocksdb: [db/db_impl/db_impl.cc:1109] ------- DUMPING STATS ------- 2024-05-27T07:13:40.001 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:13:39 smithi052 ceph-mon[31955]: rocksdb: [db/db_impl/db_impl.cc:1111] 2024-05-27T07:13:40.001 INFO:journalctl@ceph.mon.a.smithi052.stdout: ** DB Stats ** 2024-05-27T07:13:40.002 INFO:journalctl@ceph.mon.a.smithi052.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-05-27T07:13:40.002 INFO:journalctl@ceph.mon.a.smithi052.stdout: Cumulative writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 GB, 0.00 MB/s 2024-05-27T07:13:40.002 INFO:journalctl@ceph.mon.a.smithi052.stdout: Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-05-27T07:13:40.002 INFO:journalctl@ceph.mon.a.smithi052.stdout: Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent 2024-05-27T07:13:40.002 INFO:journalctl@ceph.mon.a.smithi052.stdout: Interval writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 MB, 0.00 MB/s 2024-05-27T07:13:40.002 INFO:journalctl@ceph.mon.a.smithi052.stdout: Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-05-27T07:13:40.002 INFO:journalctl@ceph.mon.a.smithi052.stdout: Interval stall: 00:00:0.000 H:M:S, 0.0 percent 2024-05-27T07:13:40.002 INFO:journalctl@ceph.mon.a.smithi052.stdout: 2024-05-27T07:13:40.002 INFO:journalctl@ceph.mon.a.smithi052.stdout: ** Compaction Stats [default] ** 2024-05-27T07:13:40.002 INFO:journalctl@ceph.mon.a.smithi052.stdout: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-05-27T07:13:40.002 INFO:journalctl@ceph.mon.a.smithi052.stdout: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ 2024-05-27T07:13:40.002 INFO:journalctl@ceph.mon.a.smithi052.stdout: L0 2/0 99.80 KB 0.5 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 154.5 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-05-27T07:13:40.003 INFO:journalctl@ceph.mon.a.smithi052.stdout: Sum 2/0 99.80 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 154.5 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-05-27T07:13:40.003 INFO:journalctl@ceph.mon.a.smithi052.stdout: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 154.5 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-05-27T07:13:40.003 INFO:journalctl@ceph.mon.a.smithi052.stdout: 2024-05-27T07:13:40.003 INFO:journalctl@ceph.mon.a.smithi052.stdout: ** Compaction Stats [default] ** 2024-05-27T07:13:40.003 INFO:journalctl@ceph.mon.a.smithi052.stdout: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-05-27T07:13:40.003 INFO:journalctl@ceph.mon.a.smithi052.stdout: --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-05-27T07:13:40.003 INFO:journalctl@ceph.mon.a.smithi052.stdout: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 154.5 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-05-27T07:13:40.003 INFO:journalctl@ceph.mon.a.smithi052.stdout: 2024-05-27T07:13:40.003 INFO:journalctl@ceph.mon.a.smithi052.stdout: Blob file count: 0, total size: 0.0 GB, garbage size: 0.0 GB, space amp: 0.0 2024-05-27T07:13:40.003 INFO:journalctl@ceph.mon.a.smithi052.stdout: 2024-05-27T07:13:40.003 INFO:journalctl@ceph.mon.a.smithi052.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-05-27T07:13:40.003 INFO:journalctl@ceph.mon.a.smithi052.stdout: Flush(GB): cumulative 0.000, interval 0.000 2024-05-27T07:13:40.004 INFO:journalctl@ceph.mon.a.smithi052.stdout: AddFile(GB): cumulative 0.000, interval 0.000 2024-05-27T07:13:40.004 INFO:journalctl@ceph.mon.a.smithi052.stdout: AddFile(Total Files): cumulative 0, interval 0 2024-05-27T07:13:40.004 INFO:journalctl@ceph.mon.a.smithi052.stdout: AddFile(L0 Files): cumulative 0, interval 0 2024-05-27T07:13:40.004 INFO:journalctl@ceph.mon.a.smithi052.stdout: AddFile(Keys): cumulative 0, interval 0 2024-05-27T07:13:40.004 INFO:journalctl@ceph.mon.a.smithi052.stdout: Cumulative compaction: 0.00 GB write, 27.67 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-05-27T07:13:40.004 INFO:journalctl@ceph.mon.a.smithi052.stdout: Interval compaction: 0.00 GB write, 27.67 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-05-27T07:13:40.004 INFO:journalctl@ceph.mon.a.smithi052.stdout: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2024-05-27T07:13:40.004 INFO:journalctl@ceph.mon.a.smithi052.stdout: Block cache BinnedLRUCache@0x55c72cab71f0#2 capacity: 512.00 MB usage: 1.53 KB table_size: 0 occupancy: 18446744073709551615 collections: 1 last_copies: 0 last_secs: 8e-06 secs_since: 0 2024-05-27T07:13:40.004 INFO:journalctl@ceph.mon.a.smithi052.stdout: Block cache entry stats(count,size,portion): FilterBlock(2,1.03 KB,0.000196695%) IndexBlock(2,0.50 KB,9.53674e-05%) Misc(1,0.00 KB,0%) 2024-05-27T07:13:40.004 INFO:journalctl@ceph.mon.a.smithi052.stdout: 2024-05-27T07:13:40.004 INFO:journalctl@ceph.mon.a.smithi052.stdout: ** File Read Latency Histogram By Level [default] ** 2024-05-27T07:13:40.004 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:13:39 smithi052 ceph-mon[31955]: starting mon.a rank 0 at public addrs [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] at bind addrs [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] mon_data /var/lib/ceph/mon/ceph-a fsid 73df2636-1bf8-11ef-bc9b-c7b262605968 2024-05-27T07:13:40.005 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:13:39 smithi052 ceph-mon[31955]: mon.a@-1(???) e1 preinit fsid 73df2636-1bf8-11ef-bc9b-c7b262605968 2024-05-27T07:13:40.005 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:13:39 smithi052 ceph-mon[31955]: mon.a@-1(???).mds e1 new map 2024-05-27T07:13:40.005 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:13:39 smithi052 ceph-mon[31955]: mon.a@-1(???).mds e1 print_map 2024-05-27T07:13:40.005 INFO:journalctl@ceph.mon.a.smithi052.stdout: e1 2024-05-27T07:13:40.005 INFO:journalctl@ceph.mon.a.smithi052.stdout: enable_multiple, ever_enabled_multiple: 1,1 2024-05-27T07:13:40.005 INFO:journalctl@ceph.mon.a.smithi052.stdout: default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2,11=minor log segments,12=quiesce subvolumes} 2024-05-27T07:13:40.005 INFO:journalctl@ceph.mon.a.smithi052.stdout: legacy client fscid: -1 2024-05-27T07:13:40.005 INFO:journalctl@ceph.mon.a.smithi052.stdout: 2024-05-27T07:13:40.005 INFO:journalctl@ceph.mon.a.smithi052.stdout: No filesystems configured 2024-05-27T07:13:40.005 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:13:39 smithi052 ceph-mon[31955]: mon.a@-1(???).osd e1 crush map has features 3314932999778484224, adjusting msgr requires 2024-05-27T07:13:40.005 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:13:39 smithi052 ceph-mon[31955]: mon.a@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2024-05-27T07:13:40.005 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:13:39 smithi052 ceph-mon[31955]: mon.a@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2024-05-27T07:13:40.005 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:13:39 smithi052 ceph-mon[31955]: mon.a@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2024-05-27T07:13:40.006 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:13:39 smithi052 ceph-mon[31955]: mon.a@-1(???).paxosservice(auth 1..2) refresh upgraded, format 0 -> 3 2024-05-27T07:13:40.006 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:13:39 smithi052 ceph-mon[31955]: mon.a is new leader, mons a in quorum (ranks 0) 2024-05-27T07:13:40.006 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:13:39 smithi052 ceph-mon[31955]: monmap epoch 1 2024-05-27T07:13:40.006 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:13:39 smithi052 ceph-mon[31955]: fsid 73df2636-1bf8-11ef-bc9b-c7b262605968 2024-05-27T07:13:40.006 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:13:39 smithi052 ceph-mon[31955]: last_changed 2024-05-27T07:13:31.465220+0000 2024-05-27T07:13:40.006 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:13:39 smithi052 ceph-mon[31955]: created 2024-05-27T07:13:31.465220+0000 2024-05-27T07:13:40.006 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:13:39 smithi052 ceph-mon[31955]: min_mon_release 19 (squid) 2024-05-27T07:13:40.006 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:13:39 smithi052 ceph-mon[31955]: election_strategy: 1 2024-05-27T07:13:40.006 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:13:39 smithi052 ceph-mon[31955]: 0: [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] mon.a 2024-05-27T07:13:40.006 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:13:39 smithi052 ceph-mon[31955]: fsmap 2024-05-27T07:13:40.006 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:13:39 smithi052 ceph-mon[31955]: osdmap e1: 0 total, 0 up, 0 in 2024-05-27T07:13:40.006 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:13:39 smithi052 ceph-mon[31955]: mgrmap e1: no daemons active 2024-05-27T07:13:40.007 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:13:39 smithi052 bash[31936]: 2dc34a76e3c22634d5163db18519ee0c400f5934e46c39548ccebaef4627c41c 2024-05-27T07:13:40.007 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:13:39 smithi052 systemd[1]: Started Ceph mon.a for 73df2636-1bf8-11ef-bc9b-c7b262605968. 2024-05-27T07:13:40.876 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:13:40.237+0000 7f41a1762700 1 Processor -- start 2024-05-27T07:13:40.876 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:13:40.237+0000 7f41a1762700 1 -- start start 2024-05-27T07:13:40.876 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:13:40.237+0000 7f41a1762700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f419c109c10 0x7f419c10a020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:13:40.877 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:13:40.237+0000 7f41a1762700 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x7f419c10a610 con 0x7f419c109c10 2024-05-27T07:13:40.877 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:13:40.238+0000 7f419affd700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f419c109c10 0x7f419c10a020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:13:40.877 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:13:40.238+0000 7f419affd700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f419c109c10 0x7f419c10a020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.52:56498/0 (socket says 172.21.15.52:56498) 2024-05-27T07:13:40.877 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:13:40.238+0000 7f419affd700 1 -- 172.21.15.52:0/1372154262 learned_addr learned my addr 172.21.15.52:0/1372154262 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-05-27T07:13:40.877 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:13:40.238+0000 7f419affd700 1 -- 172.21.15.52:0/1372154262 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f419c10ad40 con 0x7f419c109c10 2024-05-27T07:13:40.877 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:13:40.238+0000 7f419affd700 1 --2- 172.21.15.52:0/1372154262 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f419c109c10 0x7f419c10a020 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 crypto rx=0x7f418c0098a0 tx=0x7f418c009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fd746dd257de4c77 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:13:40.877 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:13:40.238+0000 7f419a7fc700 1 -- 172.21.15.52:0/1372154262 <== mon.0 v2:172.21.15.52:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f418c004160 con 0x7f419c109c10 2024-05-27T07:13:40.877 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:13:40.238+0000 7f419a7fc700 1 -- 172.21.15.52:0/1372154262 <== mon.0 v2:172.21.15.52:3300/0 2 ==== config(26 keys) ==== 1070+0+0 (secure 0 0 0) 0x7f418c004300 con 0x7f419c109c10 2024-05-27T07:13:40.877 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:13:40.239+0000 7f419a7fc700 1 -- 172.21.15.52:0/1372154262 <== mon.0 v2:172.21.15.52:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f418c011af0 con 0x7f419c109c10 2024-05-27T07:13:40.877 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:13:40.239+0000 7f41a1762700 1 -- 172.21.15.52:0/1372154262 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f419c109c10 msgr2=0x7f419c10a020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:13:40.877 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:13:40.239+0000 7f41a1762700 1 --2- 172.21.15.52:0/1372154262 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f419c109c10 0x7f419c10a020 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 crypto rx=0x7f418c0098a0 tx=0x7f418c009bb0 comp rx=0 tx=0).stop 2024-05-27T07:13:40.877 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:13:40.239+0000 7f41a1762700 1 -- 172.21.15.52:0/1372154262 shutdown_connections 2024-05-27T07:13:40.878 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:13:40.239+0000 7f41a1762700 1 --2- 172.21.15.52:0/1372154262 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f419c109c10 0x7f419c10a020 unknown :-1 s=CLOSED pgs=1 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:13:40.878 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:13:40.239+0000 7f41a1762700 1 -- 172.21.15.52:0/1372154262 >> 172.21.15.52:0/1372154262 conn(0x7f419c1052f0 msgr2=0x7f419c107740 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:13:40.878 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:13:40.239+0000 7f41a1762700 1 -- 172.21.15.52:0/1372154262 shutdown_connections 2024-05-27T07:13:40.878 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:13:40.239+0000 7f41a1762700 1 -- 172.21.15.52:0/1372154262 wait complete. 2024-05-27T07:13:40.878 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:13:40.240+0000 7f41a1762700 1 Processor -- start 2024-05-27T07:13:40.878 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:13:40.240+0000 7f41a1762700 1 -- start start 2024-05-27T07:13:40.878 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:13:40.240+0000 7f41a1762700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f419c109c10 0x7f419c19e030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:13:40.878 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:13:40.240+0000 7f41a1762700 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x7f419c10e0f0 con 0x7f419c109c10 2024-05-27T07:13:40.878 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:13:40.241+0000 7f419affd700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f419c109c10 0x7f419c19e030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:13:40.878 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:13:40.241+0000 7f419affd700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f419c109c10 0x7f419c19e030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.52:56512/0 (socket says 172.21.15.52:56512) 2024-05-27T07:13:40.878 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:13:40.241+0000 7f419affd700 1 -- 172.21.15.52:0/585125781 learned_addr learned my addr 172.21.15.52:0/585125781 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-05-27T07:13:40.878 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:13:40.241+0000 7f419affd700 1 -- 172.21.15.52:0/585125781 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f419c19e570 con 0x7f419c109c10 2024-05-27T07:13:40.878 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:13:40.241+0000 7f419affd700 1 --2- 172.21.15.52:0/585125781 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f419c109c10 0x7f419c19e030 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 crypto rx=0x7f418c0090e0 tx=0x7f418c011940 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:13:40.879 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:13:40.241+0000 7f4198ff9700 1 -- 172.21.15.52:0/585125781 <== mon.0 v2:172.21.15.52:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f418c0039e0 con 0x7f419c109c10 2024-05-27T07:13:40.879 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:13:40.241+0000 7f4198ff9700 1 -- 172.21.15.52:0/585125781 <== mon.0 v2:172.21.15.52:3300/0 2 ==== config(26 keys) ==== 1070+0+0 (secure 0 0 0) 0x7f418c003b80 con 0x7f419c109c10 2024-05-27T07:13:40.879 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:13:40.241+0000 7f41a1762700 1 -- 172.21.15.52:0/585125781 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f419c19e800 con 0x7f419c109c10 2024-05-27T07:13:40.879 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:13:40.241+0000 7f4198ff9700 1 -- 172.21.15.52:0/585125781 <== mon.0 v2:172.21.15.52:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f418c024690 con 0x7f419c109c10 2024-05-27T07:13:40.879 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:13:40.241+0000 7f41a1762700 1 -- 172.21.15.52:0/585125781 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f419c1a1410 con 0x7f419c109c10 2024-05-27T07:13:40.879 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:13:40.242+0000 7f4198ff9700 1 -- 172.21.15.52:0/585125781 <== mon.0 v2:172.21.15.52:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7f418c022020 con 0x7f419c109c10 2024-05-27T07:13:40.879 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:13:40.242+0000 7f4198ff9700 1 -- 172.21.15.52:0/585125781 <== mon.0 v2:172.21.15.52:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7f418c02dc40 con 0x7f419c109c10 2024-05-27T07:13:40.879 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:13:40.242+0000 7f41a1762700 1 -- 172.21.15.52:0/585125781 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f419c064e10 con 0x7f419c109c10 2024-05-27T07:13:40.879 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:13:40.245+0000 7f4198ff9700 1 -- 172.21.15.52:0/585125781 <== mon.0 v2:172.21.15.52:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73389 (secure 0 0 0) 0x7f418c01a9c0 con 0x7f419c109c10 2024-05-27T07:13:40.879 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:13:40.315+0000 7f41a1762700 1 -- 172.21.15.52:0/585125781 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_command([{prefix=config set, name=public_network}] v 0) -- 0x7f419c1a15e0 con 0x7f419c109c10 2024-05-27T07:13:40.879 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:13:40.317+0000 7f4198ff9700 1 -- 172.21.15.52:0/585125781 <== mon.0 v2:172.21.15.52:3300/0 7 ==== mon_command_ack([{prefix=config set, name=public_network}]=0 v3)=0 v3) ==== 124+0+0 (secure 0 0 0) 0x7f418c02d3f0 con 0x7f419c109c10 2024-05-27T07:13:40.879 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:13:40.318+0000 7f41a1762700 1 -- 172.21.15.52:0/585125781 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f419c109c10 msgr2=0x7f419c19e030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:13:40.879 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:13:40.318+0000 7f41a1762700 1 --2- 172.21.15.52:0/585125781 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f419c109c10 0x7f419c19e030 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 crypto rx=0x7f418c0090e0 tx=0x7f418c011940 comp rx=0 tx=0).stop 2024-05-27T07:13:40.879 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:13:40.318+0000 7f41a1762700 1 -- 172.21.15.52:0/585125781 shutdown_connections 2024-05-27T07:13:40.880 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:13:40.318+0000 7f41a1762700 1 --2- 172.21.15.52:0/585125781 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f419c109c10 0x7f419c19e030 unknown :-1 s=CLOSED pgs=2 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:13:40.880 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:13:40.318+0000 7f41a1762700 1 -- 172.21.15.52:0/585125781 >> 172.21.15.52:0/585125781 conn(0x7f419c1052f0 msgr2=0x7f419c106c70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:13:40.880 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:13:40.318+0000 7f41a1762700 1 -- 172.21.15.52:0/585125781 shutdown_connections 2024-05-27T07:13:40.880 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:13:40.318+0000 7f41a1762700 1 -- 172.21.15.52:0/585125781 wait complete. 2024-05-27T07:13:40.880 INFO:teuthology.orchestra.run.smithi052.stdout:Wrote config to /etc/ceph/ceph.conf 2024-05-27T07:13:40.902 INFO:teuthology.orchestra.run.smithi052.stdout:Wrote keyring to /etc/ceph/ceph.client.admin.keyring 2024-05-27T07:13:40.903 INFO:teuthology.orchestra.run.smithi052.stdout:Creating mgr... 2024-05-27T07:13:40.903 INFO:teuthology.orchestra.run.smithi052.stdout:Verifying port 0.0.0.0:9283 ... 2024-05-27T07:13:40.903 INFO:teuthology.orchestra.run.smithi052.stdout:Verifying port 0.0.0.0:8765 ... 2024-05-27T07:13:41.164 INFO:teuthology.orchestra.run.smithi052.stdout:Non-zero exit code 1 from systemctl reset-failed ceph-73df2636-1bf8-11ef-bc9b-c7b262605968@mgr.x 2024-05-27T07:13:41.165 INFO:teuthology.orchestra.run.smithi052.stdout:systemctl: stderr Failed to reset failed state of unit ceph-73df2636-1bf8-11ef-bc9b-c7b262605968@mgr.x.service: Unit ceph-73df2636-1bf8-11ef-bc9b-c7b262605968@mgr.x.service not loaded. 2024-05-27T07:13:41.325 INFO:teuthology.orchestra.run.smithi052.stdout:systemctl: stderr Created symlink /etc/systemd/system/ceph-73df2636-1bf8-11ef-bc9b-c7b262605968.target.wants/ceph-73df2636-1bf8-11ef-bc9b-c7b262605968@mgr.x.service → /etc/systemd/system/ceph-73df2636-1bf8-11ef-bc9b-c7b262605968@.service. 2024-05-27T07:13:41.480 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:13:41 smithi052 ceph-mon[31955]: from='client.? 172.21.15.52:0/585125781' entity='client.admin' 2024-05-27T07:13:41.481 INFO:journalctl@ceph.mgr.x.smithi052.stdout:May 27 07:13:41 smithi052 systemd[1]: Starting Ceph mgr.x for 73df2636-1bf8-11ef-bc9b-c7b262605968... 2024-05-27T07:13:41.849 INFO:journalctl@ceph.mgr.x.smithi052.stdout:May 27 07:13:41 smithi052 podman[32169]: 2024-05-27 07:13:41.474469163 +0000 UTC m=+0.017475019 image pull e36ea4fc8a5807e8d2701370e350a0de3856538797f7fba64cfce7e344bd6799 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:38f9d900c83c63799fbdbe61acc9a11b0d3554a6 2024-05-27T07:13:41.849 INFO:journalctl@ceph.mgr.x.smithi052.stdout:May 27 07:13:41 smithi052 podman[32169]: 2024-05-27 07:13:41.573289359 +0000 UTC m=+0.116295244 container create 706e4038639624b167ca28ab0f525583a0c585bfd1afd594ada1c8c97e04d88a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:38f9d900c83c63799fbdbe61acc9a11b0d3554a6, name=ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mgr-x, CEPH_POINT_RELEASE=, ceph=True, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, GIT_CLEAN=True, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240521, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2) 2024-05-27T07:13:41.849 INFO:journalctl@ceph.mgr.x.smithi052.stdout:May 27 07:13:41 smithi052 podman[32169]: 2024-05-27 07:13:41.775354499 +0000 UTC m=+0.318360343 container init 706e4038639624b167ca28ab0f525583a0c585bfd1afd594ada1c8c97e04d88a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:38f9d900c83c63799fbdbe61acc9a11b0d3554a6, name=ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mgr-x, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240521, GIT_CLEAN=True, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2) 2024-05-27T07:13:41.849 INFO:journalctl@ceph.mgr.x.smithi052.stdout:May 27 07:13:41 smithi052 podman[32169]: 2024-05-27 07:13:41.778213588 +0000 UTC m=+0.321219439 container start 706e4038639624b167ca28ab0f525583a0c585bfd1afd594ada1c8c97e04d88a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:38f9d900c83c63799fbdbe61acc9a11b0d3554a6, name=ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mgr-x, GIT_CLEAN=True, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240521, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, GIT_BRANCH=HEAD, ceph=True, org.label-schema.vendor=CentOS, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, org.label-schema.schema-version=1.0) 2024-05-27T07:13:41.849 INFO:journalctl@ceph.mgr.x.smithi052.stdout:May 27 07:13:41 smithi052 bash[32169]: 706e4038639624b167ca28ab0f525583a0c585bfd1afd594ada1c8c97e04d88a 2024-05-27T07:13:41.862 INFO:teuthology.orchestra.run.smithi052.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2024-05-27T07:13:41.863 INFO:teuthology.orchestra.run.smithi052.stdout:systemctl: stdout disabled 2024-05-27T07:13:41.870 INFO:teuthology.orchestra.run.smithi052.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2024-05-27T07:13:41.870 INFO:teuthology.orchestra.run.smithi052.stdout:systemctl: stdout inactive 2024-05-27T07:13:41.871 INFO:teuthology.orchestra.run.smithi052.stdout:firewalld.service is not enabled 2024-05-27T07:13:41.871 INFO:teuthology.orchestra.run.smithi052.stdout:Not possible to enable service . firewalld.service is not available 2024-05-27T07:13:41.879 INFO:teuthology.orchestra.run.smithi052.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2024-05-27T07:13:41.879 INFO:teuthology.orchestra.run.smithi052.stdout:systemctl: stdout disabled 2024-05-27T07:13:41.887 INFO:teuthology.orchestra.run.smithi052.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2024-05-27T07:13:41.887 INFO:teuthology.orchestra.run.smithi052.stdout:systemctl: stdout inactive 2024-05-27T07:13:41.887 INFO:teuthology.orchestra.run.smithi052.stdout:firewalld.service is not enabled 2024-05-27T07:13:41.887 INFO:teuthology.orchestra.run.smithi052.stdout:Not possible to open ports <[9283, 8765]>. firewalld.service is not available 2024-05-27T07:13:41.887 INFO:teuthology.orchestra.run.smithi052.stdout:Waiting for mgr to start... 2024-05-27T07:13:41.888 INFO:teuthology.orchestra.run.smithi052.stdout:Waiting for mgr... 2024-05-27T07:13:42.134 INFO:journalctl@ceph.mgr.x.smithi052.stdout:May 27 07:13:41 smithi052 systemd[1]: Started Ceph mgr.x for 73df2636-1bf8-11ef-bc9b-c7b262605968. 2024-05-27T07:13:42.134 INFO:journalctl@ceph.mgr.x.smithi052.stdout:May 27 07:13:41 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mgr-x[32184]: 2024-05-27T07:13:41.950+0000 7fcdf660c200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-05-27T07:13:42.432 INFO:journalctl@ceph.mgr.x.smithi052.stdout:May 27 07:13:42 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mgr-x[32184]: 2024-05-27T07:13:42.149+0000 7fcdf660c200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-05-27T07:13:42.432 INFO:journalctl@ceph.mgr.x.smithi052.stdout:May 27 07:13:42 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mgr-x[32184]: 2024-05-27T07:13:42.257+0000 7fcdf660c200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-05-27T07:13:42.722 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:13:42 smithi052 ceph-mon[31955]: from='client.? 172.21.15.52:0/245276845' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-05-27T07:13:42.722 INFO:journalctl@ceph.mgr.x.smithi052.stdout:May 27 07:13:42 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mgr-x[32184]: 2024-05-27T07:13:42.597+0000 7fcdf660c200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-05-27T07:13:42.722 INFO:journalctl@ceph.mgr.x.smithi052.stdout:May 27 07:13:42 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mgr-x[32184]: 2024-05-27T07:13:42.687+0000 7fcdf660c200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-05-27T07:13:42.940 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout 2024-05-27T07:13:42.943 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout { 2024-05-27T07:13:42.943 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout "fsid": "73df2636-1bf8-11ef-bc9b-c7b262605968", 2024-05-27T07:13:42.943 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout "health": { 2024-05-27T07:13:42.943 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout "status": "HEALTH_OK", 2024-05-27T07:13:42.943 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout "checks": {}, 2024-05-27T07:13:42.943 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout "mutes": [] 2024-05-27T07:13:42.943 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout }, 2024-05-27T07:13:42.943 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout "election_epoch": 5, 2024-05-27T07:13:42.944 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout "quorum": [ 2024-05-27T07:13:42.944 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout 0 2024-05-27T07:13:42.944 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout ], 2024-05-27T07:13:42.944 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout "quorum_names": [ 2024-05-27T07:13:42.944 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout "a" 2024-05-27T07:13:42.944 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout ], 2024-05-27T07:13:42.944 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout "quorum_age": 2, 2024-05-27T07:13:42.944 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout "monmap": { 2024-05-27T07:13:42.944 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-05-27T07:13:42.944 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout "min_mon_release_name": "squid", 2024-05-27T07:13:42.945 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout "num_mons": 1 2024-05-27T07:13:42.945 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout }, 2024-05-27T07:13:42.945 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout "osdmap": { 2024-05-27T07:13:42.945 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-05-27T07:13:42.945 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout "num_osds": 0, 2024-05-27T07:13:42.945 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout "num_up_osds": 0, 2024-05-27T07:13:42.945 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout "osd_up_since": 0, 2024-05-27T07:13:42.952 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout "num_in_osds": 0, 2024-05-27T07:13:42.952 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout "osd_in_since": 0, 2024-05-27T07:13:42.952 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout "num_remapped_pgs": 0 2024-05-27T07:13:42.952 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout }, 2024-05-27T07:13:42.952 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout "pgmap": { 2024-05-27T07:13:42.952 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout "pgs_by_state": [], 2024-05-27T07:13:42.952 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout "num_pgs": 0, 2024-05-27T07:13:42.953 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout "num_pools": 0, 2024-05-27T07:13:42.953 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout "num_objects": 0, 2024-05-27T07:13:42.953 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout "data_bytes": 0, 2024-05-27T07:13:42.953 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout "bytes_used": 0, 2024-05-27T07:13:42.953 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout "bytes_avail": 0, 2024-05-27T07:13:42.953 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout "bytes_total": 0 2024-05-27T07:13:42.953 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout }, 2024-05-27T07:13:42.953 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout "fsmap": { 2024-05-27T07:13:42.953 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-05-27T07:13:42.953 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout "by_rank": [], 2024-05-27T07:13:42.953 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout "up:standby": 0 2024-05-27T07:13:42.953 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout }, 2024-05-27T07:13:42.953 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout "mgrmap": { 2024-05-27T07:13:42.953 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout "available": false, 2024-05-27T07:13:42.954 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout "num_standbys": 0, 2024-05-27T07:13:42.954 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout "modules": [ 2024-05-27T07:13:42.954 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout "iostat", 2024-05-27T07:13:42.954 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout "nfs", 2024-05-27T07:13:42.954 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout "restful" 2024-05-27T07:13:42.954 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout ], 2024-05-27T07:13:42.954 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout "services": {} 2024-05-27T07:13:42.954 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout }, 2024-05-27T07:13:42.954 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout "servicemap": { 2024-05-27T07:13:42.954 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-05-27T07:13:42.954 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout "modified": "2024-05-27T07:13:34.738650+0000", 2024-05-27T07:13:42.955 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout "services": {} 2024-05-27T07:13:42.955 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout }, 2024-05-27T07:13:42.955 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout "progress_events": {} 2024-05-27T07:13:42.955 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout } 2024-05-27T07:13:42.955 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:13:42.359+0000 7f815639f700 1 Processor -- start 2024-05-27T07:13:42.955 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:13:42.359+0000 7f815639f700 1 -- start start 2024-05-27T07:13:42.955 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:13:42.359+0000 7f815639f700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f815007ed70 0x7f815007d240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:13:42.955 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:13:42.359+0000 7f815639f700 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x7f815007d780 con 0x7f815007ed70 2024-05-27T07:13:42.955 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:13:42.360+0000 7f815539d700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f815007ed70 0x7f815007d240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:13:42.955 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:13:42.360+0000 7f815539d700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f815007ed70 0x7f815007d240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.52:39682/0 (socket says 172.21.15.52:39682) 2024-05-27T07:13:42.955 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:13:42.360+0000 7f815539d700 1 -- 172.21.15.52:0/3238622517 learned_addr learned my addr 172.21.15.52:0/3238622517 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-05-27T07:13:42.955 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:13:42.360+0000 7f815539d700 1 -- 172.21.15.52:0/3238622517 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f815007d900 con 0x7f815007ed70 2024-05-27T07:13:42.955 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:13:42.360+0000 7f815539d700 1 --2- 172.21.15.52:0/3238622517 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f815007ed70 0x7f815007d240 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7f814c0098a0 tx=0x7f814c009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3f98edbca3dd342b server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:13:42.955 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:13:42.360+0000 7f8147fff700 1 -- 172.21.15.52:0/3238622517 <== mon.0 v2:172.21.15.52:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f814c004160 con 0x7f815007ed70 2024-05-27T07:13:42.956 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:13:42.360+0000 7f8147fff700 1 -- 172.21.15.52:0/3238622517 <== mon.0 v2:172.21.15.52:3300/0 2 ==== config(26 keys) ==== 1070+0+0 (secure 0 0 0) 0x7f814c004300 con 0x7f815007ed70 2024-05-27T07:13:42.956 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:13:42.360+0000 7f8147fff700 1 -- 172.21.15.52:0/3238622517 <== mon.0 v2:172.21.15.52:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f814c011af0 con 0x7f815007ed70 2024-05-27T07:13:42.956 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:13:42.361+0000 7f815639f700 1 -- 172.21.15.52:0/3238622517 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f815007ed70 msgr2=0x7f815007d240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:13:42.956 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:13:42.361+0000 7f815639f700 1 --2- 172.21.15.52:0/3238622517 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f815007ed70 0x7f815007d240 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7f814c0098a0 tx=0x7f814c009bb0 comp rx=0 tx=0).stop 2024-05-27T07:13:42.956 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:13:42.361+0000 7f815639f700 1 -- 172.21.15.52:0/3238622517 shutdown_connections 2024-05-27T07:13:42.956 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:13:42.361+0000 7f815639f700 1 --2- 172.21.15.52:0/3238622517 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f815007ed70 0x7f815007d240 unknown :-1 s=CLOSED pgs=5 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:13:42.956 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:13:42.361+0000 7f815639f700 1 -- 172.21.15.52:0/3238622517 >> 172.21.15.52:0/3238622517 conn(0x7f81501050f0 msgr2=0x7f8150107580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:13:42.956 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:13:42.361+0000 7f815639f700 1 -- 172.21.15.52:0/3238622517 shutdown_connections 2024-05-27T07:13:42.956 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:13:42.361+0000 7f815639f700 1 -- 172.21.15.52:0/3238622517 wait complete. 2024-05-27T07:13:42.956 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:13:42.362+0000 7f815639f700 1 Processor -- start 2024-05-27T07:13:42.956 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:13:42.362+0000 7f815639f700 1 -- start start 2024-05-27T07:13:42.956 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:13:42.363+0000 7f815639f700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f815007ed70 0x7f815019ff80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:13:42.956 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:13:42.363+0000 7f815639f700 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x7f815010ffa0 con 0x7f815007ed70 2024-05-27T07:13:42.956 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:13:42.363+0000 7f815539d700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f815007ed70 0x7f815019ff80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:13:42.956 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:13:42.363+0000 7f815539d700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f815007ed70 0x7f815019ff80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.52:39698/0 (socket says 172.21.15.52:39698) 2024-05-27T07:13:42.957 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:13:42.363+0000 7f815539d700 1 -- 172.21.15.52:0/245276845 learned_addr learned my addr 172.21.15.52:0/245276845 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-05-27T07:13:42.957 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:13:42.363+0000 7f815539d700 1 -- 172.21.15.52:0/245276845 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f81501a04c0 con 0x7f815007ed70 2024-05-27T07:13:42.957 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:13:42.363+0000 7f815539d700 1 --2- 172.21.15.52:0/245276845 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f815007ed70 0x7f815019ff80 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7f814c009fb0 tx=0x7f814c011940 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:13:42.957 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:13:42.363+0000 7f81467fc700 1 -- 172.21.15.52:0/245276845 <== mon.0 v2:172.21.15.52:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f814c0036a0 con 0x7f815007ed70 2024-05-27T07:13:42.957 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:13:42.363+0000 7f81467fc700 1 -- 172.21.15.52:0/245276845 <== mon.0 v2:172.21.15.52:3300/0 2 ==== config(26 keys) ==== 1070+0+0 (secure 0 0 0) 0x7f814c01a430 con 0x7f815007ed70 2024-05-27T07:13:42.957 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:13:42.363+0000 7f815639f700 1 -- 172.21.15.52:0/245276845 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f81501a0750 con 0x7f815007ed70 2024-05-27T07:13:42.957 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:13:42.363+0000 7f81467fc700 1 -- 172.21.15.52:0/245276845 <== mon.0 v2:172.21.15.52:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f814c019550 con 0x7f815007ed70 2024-05-27T07:13:42.957 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:13:42.364+0000 7f815639f700 1 -- 172.21.15.52:0/245276845 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f81501a3360 con 0x7f815007ed70 2024-05-27T07:13:42.957 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:13:42.364+0000 7f81467fc700 1 -- 172.21.15.52:0/245276845 <== mon.0 v2:172.21.15.52:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7f814c019b90 con 0x7f815007ed70 2024-05-27T07:13:42.957 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:13:42.364+0000 7f81467fc700 1 -- 172.21.15.52:0/245276845 <== mon.0 v2:172.21.15.52:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7f814c024730 con 0x7f815007ed70 2024-05-27T07:13:42.957 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:13:42.364+0000 7f815639f700 1 -- 172.21.15.52:0/245276845 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8150064e10 con 0x7f815007ed70 2024-05-27T07:13:42.957 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:13:42.367+0000 7f81467fc700 1 -- 172.21.15.52:0/245276845 <== mon.0 v2:172.21.15.52:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73389 (secure 0 0 0) 0x7f814c043ec0 con 0x7f815007ed70 2024-05-27T07:13:42.957 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:13:42.441+0000 7f815639f700 1 -- 172.21.15.52:0/245276845 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) -- 0x7f81501a3650 con 0x7f815007ed70 2024-05-27T07:13:42.957 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:13:42.441+0000 7f81467fc700 1 -- 172.21.15.52:0/245276845 <== mon.0 v2:172.21.15.52:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) ==== 79+0+1239 (secure 0 0 0) 0x7f814c032020 con 0x7f815007ed70 2024-05-27T07:13:42.957 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:13:42.442+0000 7f815639f700 1 -- 172.21.15.52:0/245276845 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f815007ed70 msgr2=0x7f815019ff80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:13:42.958 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:13:42.442+0000 7f815639f700 1 --2- 172.21.15.52:0/245276845 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f815007ed70 0x7f815019ff80 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7f814c009fb0 tx=0x7f814c011940 comp rx=0 tx=0).stop 2024-05-27T07:13:42.958 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:13:42.442+0000 7f815639f700 1 -- 172.21.15.52:0/245276845 shutdown_connections 2024-05-27T07:13:42.958 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:13:42.442+0000 7f815639f700 1 --2- 172.21.15.52:0/245276845 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f815007ed70 0x7f815019ff80 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:13:42.958 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:13:42.442+0000 7f815639f700 1 -- 172.21.15.52:0/245276845 >> 172.21.15.52:0/245276845 conn(0x7f81501050f0 msgr2=0x7f81501059c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:13:42.958 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:13:42.442+0000 7f815639f700 1 -- 172.21.15.52:0/245276845 shutdown_connections 2024-05-27T07:13:42.958 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:13:42.442+0000 7f815639f700 1 -- 172.21.15.52:0/245276845 wait complete. 2024-05-27T07:13:42.958 INFO:teuthology.orchestra.run.smithi052.stdout:mgr not available, waiting (1/15)... 2024-05-27T07:13:44.230 INFO:journalctl@ceph.mgr.x.smithi052.stdout:May 27 07:13:43 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mgr-x[32184]: 2024-05-27T07:13:43.960+0000 7fcdf660c200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-05-27T07:13:44.230 INFO:journalctl@ceph.mgr.x.smithi052.stdout:May 27 07:13:44 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mgr-x[32184]: 2024-05-27T07:13:44.137+0000 7fcdf660c200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-05-27T07:13:44.609 INFO:journalctl@ceph.mgr.x.smithi052.stdout:May 27 07:13:44 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mgr-x[32184]: 2024-05-27T07:13:44.316+0000 7fcdf660c200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-05-27T07:13:44.609 INFO:journalctl@ceph.mgr.x.smithi052.stdout:May 27 07:13:44 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mgr-x[32184]: 2024-05-27T07:13:44.618+0000 7fcdf660c200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-05-27T07:13:44.916 INFO:journalctl@ceph.mgr.x.smithi052.stdout:May 27 07:13:44 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mgr-x[32184]: 2024-05-27T07:13:44.714+0000 7fcdf660c200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-05-27T07:13:44.917 INFO:journalctl@ceph.mgr.x.smithi052.stdout:May 27 07:13:44 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mgr-x[32184]: 2024-05-27T07:13:44.926+0000 7fcdf660c200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-05-27T07:13:45.230 INFO:journalctl@ceph.mgr.x.smithi052.stdout:May 27 07:13:45 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mgr-x[32184]: 2024-05-27T07:13:45.132+0000 7fcdf660c200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-05-27T07:13:45.730 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:13:45 smithi052 ceph-mon[31955]: from='client.? 172.21.15.52:0/1634677113' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-05-27T07:13:46.012 INFO:journalctl@ceph.mgr.x.smithi052.stdout:May 27 07:13:45 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mgr-x[32184]: 2024-05-27T07:13:45.786+0000 7fcdf660c200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-05-27T07:13:46.012 INFO:journalctl@ceph.mgr.x.smithi052.stdout:May 27 07:13:46 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mgr-x[32184]: 2024-05-27T07:13:46.021+0000 7fcdf660c200 -1 mgr[py] Module smb has missing NOTIFY_TYPES member 2024-05-27T07:13:46.041 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout 2024-05-27T07:13:46.042 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout { 2024-05-27T07:13:46.042 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout "fsid": "73df2636-1bf8-11ef-bc9b-c7b262605968", 2024-05-27T07:13:46.042 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout "health": { 2024-05-27T07:13:46.042 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout "status": "HEALTH_OK", 2024-05-27T07:13:46.042 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout "checks": {}, 2024-05-27T07:13:46.042 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout "mutes": [] 2024-05-27T07:13:46.042 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout }, 2024-05-27T07:13:46.042 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout "election_epoch": 5, 2024-05-27T07:13:46.042 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout "quorum": [ 2024-05-27T07:13:46.042 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout 0 2024-05-27T07:13:46.042 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout ], 2024-05-27T07:13:46.042 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout "quorum_names": [ 2024-05-27T07:13:46.043 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout "a" 2024-05-27T07:13:46.043 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout ], 2024-05-27T07:13:46.043 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout "quorum_age": 5, 2024-05-27T07:13:46.043 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout "monmap": { 2024-05-27T07:13:46.043 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-05-27T07:13:46.043 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout "min_mon_release_name": "squid", 2024-05-27T07:13:46.043 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout "num_mons": 1 2024-05-27T07:13:46.043 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout }, 2024-05-27T07:13:46.043 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout "osdmap": { 2024-05-27T07:13:46.043 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-05-27T07:13:46.043 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout "num_osds": 0, 2024-05-27T07:13:46.044 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout "num_up_osds": 0, 2024-05-27T07:13:46.044 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout "osd_up_since": 0, 2024-05-27T07:13:46.053 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout "num_in_osds": 0, 2024-05-27T07:13:46.053 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout "osd_in_since": 0, 2024-05-27T07:13:46.053 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout "num_remapped_pgs": 0 2024-05-27T07:13:46.053 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout }, 2024-05-27T07:13:46.053 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout "pgmap": { 2024-05-27T07:13:46.053 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout "pgs_by_state": [], 2024-05-27T07:13:46.053 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout "num_pgs": 0, 2024-05-27T07:13:46.054 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout "num_pools": 0, 2024-05-27T07:13:46.054 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout "num_objects": 0, 2024-05-27T07:13:46.054 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout "data_bytes": 0, 2024-05-27T07:13:46.054 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout "bytes_used": 0, 2024-05-27T07:13:46.054 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout "bytes_avail": 0, 2024-05-27T07:13:46.054 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout "bytes_total": 0 2024-05-27T07:13:46.054 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout }, 2024-05-27T07:13:46.054 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout "fsmap": { 2024-05-27T07:13:46.054 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-05-27T07:13:46.054 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout "by_rank": [], 2024-05-27T07:13:46.054 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout "up:standby": 0 2024-05-27T07:13:46.054 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout }, 2024-05-27T07:13:46.054 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout "mgrmap": { 2024-05-27T07:13:46.055 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout "available": false, 2024-05-27T07:13:46.055 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout "num_standbys": 0, 2024-05-27T07:13:46.055 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout "modules": [ 2024-05-27T07:13:46.055 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout "iostat", 2024-05-27T07:13:46.055 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout "nfs", 2024-05-27T07:13:46.055 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout "restful" 2024-05-27T07:13:46.055 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout ], 2024-05-27T07:13:46.055 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout "services": {} 2024-05-27T07:13:46.055 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout }, 2024-05-27T07:13:46.055 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout "servicemap": { 2024-05-27T07:13:46.056 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-05-27T07:13:46.056 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout "modified": "2024-05-27T07:13:34.738650+0000", 2024-05-27T07:13:46.056 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout "services": {} 2024-05-27T07:13:46.056 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout }, 2024-05-27T07:13:46.056 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout "progress_events": {} 2024-05-27T07:13:46.056 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout } 2024-05-27T07:13:46.056 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:13:45.379+0000 7ff91831f700 1 Processor -- start 2024-05-27T07:13:46.056 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:13:45.379+0000 7ff91831f700 1 -- start start 2024-05-27T07:13:46.056 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:13:45.379+0000 7ff91831f700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff91007cc90 0x7ff91010eaf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:13:46.056 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:13:45.379+0000 7ff91831f700 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x7ff91010f030 con 0x7ff91007cc90 2024-05-27T07:13:46.057 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:13:45.379+0000 7ff9160bb700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff91007cc90 0x7ff91010eaf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:13:46.057 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:13:45.379+0000 7ff9160bb700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff91007cc90 0x7ff91010eaf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.52:39710/0 (socket says 172.21.15.52:39710) 2024-05-27T07:13:46.057 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:13:45.379+0000 7ff9160bb700 1 -- 172.21.15.52:0/3131913423 learned_addr learned my addr 172.21.15.52:0/3131913423 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-05-27T07:13:46.057 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:13:45.380+0000 7ff9160bb700 1 -- 172.21.15.52:0/3131913423 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff91010f1b0 con 0x7ff91007cc90 2024-05-27T07:13:46.057 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:13:45.380+0000 7ff9160bb700 1 --2- 172.21.15.52:0/3131913423 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff91007cc90 0x7ff91010eaf0 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7ff90c0098a0 tx=0x7ff90c009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a233fb777cb78a9d server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:13:46.057 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:13:45.380+0000 7ff9150b9700 1 -- 172.21.15.52:0/3131913423 <== mon.0 v2:172.21.15.52:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7ff90c004160 con 0x7ff91007cc90 2024-05-27T07:13:46.057 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:13:45.380+0000 7ff9150b9700 1 -- 172.21.15.52:0/3131913423 <== mon.0 v2:172.21.15.52:3300/0 2 ==== config(26 keys) ==== 1070+0+0 (secure 0 0 0) 0x7ff90c004300 con 0x7ff91007cc90 2024-05-27T07:13:46.057 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:13:45.380+0000 7ff9150b9700 1 -- 172.21.15.52:0/3131913423 <== mon.0 v2:172.21.15.52:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7ff90c011af0 con 0x7ff91007cc90 2024-05-27T07:13:46.057 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:13:45.381+0000 7ff91831f700 1 -- 172.21.15.52:0/3131913423 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff91007cc90 msgr2=0x7ff91010eaf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:13:46.057 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:13:45.381+0000 7ff91831f700 1 --2- 172.21.15.52:0/3131913423 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff91007cc90 0x7ff91010eaf0 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7ff90c0098a0 tx=0x7ff90c009bb0 comp rx=0 tx=0).stop 2024-05-27T07:13:46.057 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:13:45.381+0000 7ff91831f700 1 -- 172.21.15.52:0/3131913423 shutdown_connections 2024-05-27T07:13:46.057 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:13:45.381+0000 7ff91831f700 1 --2- 172.21.15.52:0/3131913423 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff91007cc90 0x7ff91010eaf0 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:13:46.057 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:13:45.381+0000 7ff91831f700 1 -- 172.21.15.52:0/3131913423 >> 172.21.15.52:0/3131913423 conn(0x7ff9101040e0 msgr2=0x7ff910106570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:13:46.058 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:13:45.381+0000 7ff91831f700 1 -- 172.21.15.52:0/3131913423 shutdown_connections 2024-05-27T07:13:46.058 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:13:45.381+0000 7ff91831f700 1 -- 172.21.15.52:0/3131913423 wait complete. 2024-05-27T07:13:46.058 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:13:45.382+0000 7ff91831f700 1 Processor -- start 2024-05-27T07:13:46.058 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:13:45.382+0000 7ff91831f700 1 -- start start 2024-05-27T07:13:46.058 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:13:45.382+0000 7ff91831f700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff91007cc90 0x7ff9101a0180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:13:46.058 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:13:45.382+0000 7ff91831f700 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x7ff9101100c0 con 0x7ff91007cc90 2024-05-27T07:13:46.058 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:13:45.382+0000 7ff9160bb700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff91007cc90 0x7ff9101a0180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:13:46.058 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:13:45.382+0000 7ff9160bb700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff91007cc90 0x7ff9101a0180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.52:39726/0 (socket says 172.21.15.52:39726) 2024-05-27T07:13:46.058 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:13:45.382+0000 7ff9160bb700 1 -- 172.21.15.52:0/1634677113 learned_addr learned my addr 172.21.15.52:0/1634677113 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-05-27T07:13:46.058 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:13:45.382+0000 7ff9160bb700 1 -- 172.21.15.52:0/1634677113 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff9101a06c0 con 0x7ff91007cc90 2024-05-27T07:13:46.058 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:13:45.383+0000 7ff9160bb700 1 --2- 172.21.15.52:0/1634677113 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff91007cc90 0x7ff9101a0180 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7ff90c011ac0 tx=0x7ff90c011ba0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:13:46.058 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:13:45.383+0000 7ff9077fe700 1 -- 172.21.15.52:0/1634677113 <== mon.0 v2:172.21.15.52:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7ff90c003970 con 0x7ff91007cc90 2024-05-27T07:13:46.059 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:13:45.383+0000 7ff9077fe700 1 -- 172.21.15.52:0/1634677113 <== mon.0 v2:172.21.15.52:3300/0 2 ==== config(26 keys) ==== 1070+0+0 (secure 0 0 0) 0x7ff90c003b10 con 0x7ff91007cc90 2024-05-27T07:13:46.059 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:13:45.383+0000 7ff91831f700 1 -- 172.21.15.52:0/1634677113 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff9101a0950 con 0x7ff91007cc90 2024-05-27T07:13:46.059 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:13:45.383+0000 7ff9077fe700 1 -- 172.21.15.52:0/1634677113 <== mon.0 v2:172.21.15.52:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7ff90c024610 con 0x7ff91007cc90 2024-05-27T07:13:46.059 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:13:45.383+0000 7ff91831f700 1 -- 172.21.15.52:0/1634677113 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff9101a3560 con 0x7ff91007cc90 2024-05-27T07:13:46.059 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:13:45.383+0000 7ff9077fe700 1 -- 172.21.15.52:0/1634677113 <== mon.0 v2:172.21.15.52:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7ff90c024bd0 con 0x7ff91007cc90 2024-05-27T07:13:46.059 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:13:45.383+0000 7ff9077fe700 1 -- 172.21.15.52:0/1634677113 <== mon.0 v2:172.21.15.52:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7ff90c019700 con 0x7ff91007cc90 2024-05-27T07:13:46.059 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:13:45.384+0000 7ff91831f700 1 -- 172.21.15.52:0/1634677113 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff910064e10 con 0x7ff91007cc90 2024-05-27T07:13:46.059 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:13:45.386+0000 7ff9077fe700 1 -- 172.21.15.52:0/1634677113 <== mon.0 v2:172.21.15.52:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73389 (secure 0 0 0) 0x7ff90c043ec0 con 0x7ff91007cc90 2024-05-27T07:13:46.059 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:13:45.463+0000 7ff91831f700 1 -- 172.21.15.52:0/1634677113 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) -- 0x7ff9101a3730 con 0x7ff91007cc90 2024-05-27T07:13:46.059 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:13:45.463+0000 7ff9077fe700 1 -- 172.21.15.52:0/1634677113 <== mon.0 v2:172.21.15.52:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) ==== 79+0+1239 (secure 0 0 0) 0x7ff90c032030 con 0x7ff91007cc90 2024-05-27T07:13:46.059 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:13:45.464+0000 7ff91831f700 1 -- 172.21.15.52:0/1634677113 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff91007cc90 msgr2=0x7ff9101a0180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:13:46.059 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:13:45.464+0000 7ff91831f700 1 --2- 172.21.15.52:0/1634677113 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff91007cc90 0x7ff9101a0180 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7ff90c011ac0 tx=0x7ff90c011ba0 comp rx=0 tx=0).stop 2024-05-27T07:13:46.059 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:13:45.464+0000 7ff91831f700 1 -- 172.21.15.52:0/1634677113 shutdown_connections 2024-05-27T07:13:46.060 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:13:45.464+0000 7ff91831f700 1 --2- 172.21.15.52:0/1634677113 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff91007cc90 0x7ff9101a0180 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:13:46.060 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:13:45.464+0000 7ff91831f700 1 -- 172.21.15.52:0/1634677113 >> 172.21.15.52:0/1634677113 conn(0x7ff9101040e0 msgr2=0x7ff910104b30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:13:46.060 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:13:45.465+0000 7ff91831f700 1 -- 172.21.15.52:0/1634677113 shutdown_connections 2024-05-27T07:13:46.060 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:13:45.465+0000 7ff91831f700 1 -- 172.21.15.52:0/1634677113 wait complete. 2024-05-27T07:13:46.060 INFO:teuthology.orchestra.run.smithi052.stdout:mgr not available, waiting (2/15)... 2024-05-27T07:13:46.730 INFO:journalctl@ceph.mgr.x.smithi052.stdout:May 27 07:13:46 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mgr-x[32184]: 2024-05-27T07:13:46.472+0000 7fcdf660c200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-05-27T07:13:46.730 INFO:journalctl@ceph.mgr.x.smithi052.stdout:May 27 07:13:46 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mgr-x[32184]: 2024-05-27T07:13:46.574+0000 7fcdf660c200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-05-27T07:13:47.230 INFO:journalctl@ceph.mgr.x.smithi052.stdout:May 27 07:13:46 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mgr-x[32184]: 2024-05-27T07:13:46.985+0000 7fcdf660c200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-05-27T07:13:47.713 INFO:journalctl@ceph.mgr.x.smithi052.stdout:May 27 07:13:47 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mgr-x[32184]: 2024-05-27T07:13:47.362+0000 7fcdf660c200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-05-27T07:13:47.713 INFO:journalctl@ceph.mgr.x.smithi052.stdout:May 27 07:13:47 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mgr-x[32184]: 2024-05-27T07:13:47.462+0000 7fcdf660c200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-05-27T07:13:47.980 INFO:journalctl@ceph.mgr.x.smithi052.stdout:May 27 07:13:47 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mgr-x[32184]: 2024-05-27T07:13:47.722+0000 7fcdf660c200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-05-27T07:13:47.980 INFO:journalctl@ceph.mgr.x.smithi052.stdout:May 27 07:13:47 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mgr-x[32184]: 2024-05-27T07:13:47.864+0000 7fcdf660c200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-05-27T07:13:47.981 INFO:journalctl@ceph.mgr.x.smithi052.stdout:May 27 07:13:47 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mgr-x[32184]: 2024-05-27T07:13:47.967+0000 7fcdf660c200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-05-27T07:13:48.480 INFO:journalctl@ceph.mgr.x.smithi052.stdout:May 27 07:13:48 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mgr-x[32184]: 2024-05-27T07:13:48.314+0000 7fcdf660c200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-05-27T07:13:48.480 INFO:journalctl@ceph.mgr.x.smithi052.stdout:May 27 07:13:48 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mgr-x[32184]: 2024-05-27T07:13:48.411+0000 7fcdf660c200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-05-27T07:13:48.980 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:13:48 smithi052 ceph-mon[31955]: from='client.? 172.21.15.52:0/2685109243' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-05-27T07:13:49.224 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout 2024-05-27T07:13:49.224 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout { 2024-05-27T07:13:49.225 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout "fsid": "73df2636-1bf8-11ef-bc9b-c7b262605968", 2024-05-27T07:13:49.225 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout "health": { 2024-05-27T07:13:49.225 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout "status": "HEALTH_OK", 2024-05-27T07:13:49.225 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout "checks": {}, 2024-05-27T07:13:49.225 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout "mutes": [] 2024-05-27T07:13:49.225 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout }, 2024-05-27T07:13:49.225 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout "election_epoch": 5, 2024-05-27T07:13:49.225 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout "quorum": [ 2024-05-27T07:13:49.225 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout 0 2024-05-27T07:13:49.225 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout ], 2024-05-27T07:13:49.226 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout "quorum_names": [ 2024-05-27T07:13:49.226 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout "a" 2024-05-27T07:13:49.226 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout ], 2024-05-27T07:13:49.226 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout "quorum_age": 8, 2024-05-27T07:13:49.226 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout "monmap": { 2024-05-27T07:13:49.226 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-05-27T07:13:49.226 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout "min_mon_release_name": "squid", 2024-05-27T07:13:49.226 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout "num_mons": 1 2024-05-27T07:13:49.226 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout }, 2024-05-27T07:13:49.226 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout "osdmap": { 2024-05-27T07:13:49.226 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-05-27T07:13:49.226 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout "num_osds": 0, 2024-05-27T07:13:49.227 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout "num_up_osds": 0, 2024-05-27T07:13:49.227 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout "osd_up_since": 0, 2024-05-27T07:13:49.227 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout "num_in_osds": 0, 2024-05-27T07:13:49.227 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout "osd_in_since": 0, 2024-05-27T07:13:49.227 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout "num_remapped_pgs": 0 2024-05-27T07:13:49.227 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout }, 2024-05-27T07:13:49.227 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout "pgmap": { 2024-05-27T07:13:49.227 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout "pgs_by_state": [], 2024-05-27T07:13:49.227 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout "num_pgs": 0, 2024-05-27T07:13:49.227 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout "num_pools": 0, 2024-05-27T07:13:49.227 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout "num_objects": 0, 2024-05-27T07:13:49.227 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout "data_bytes": 0, 2024-05-27T07:13:49.227 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout "bytes_used": 0, 2024-05-27T07:13:49.227 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout "bytes_avail": 0, 2024-05-27T07:13:49.228 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout "bytes_total": 0 2024-05-27T07:13:49.228 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout }, 2024-05-27T07:13:49.228 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout "fsmap": { 2024-05-27T07:13:49.228 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-05-27T07:13:49.228 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout "by_rank": [], 2024-05-27T07:13:49.228 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout "up:standby": 0 2024-05-27T07:13:49.228 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout }, 2024-05-27T07:13:49.228 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout "mgrmap": { 2024-05-27T07:13:49.228 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout "available": false, 2024-05-27T07:13:49.228 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout "num_standbys": 0, 2024-05-27T07:13:49.228 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout "modules": [ 2024-05-27T07:13:49.228 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout "iostat", 2024-05-27T07:13:49.228 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout "nfs", 2024-05-27T07:13:49.229 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout "restful" 2024-05-27T07:13:49.229 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout ], 2024-05-27T07:13:49.229 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout "services": {} 2024-05-27T07:13:49.229 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout }, 2024-05-27T07:13:49.229 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout "servicemap": { 2024-05-27T07:13:49.229 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-05-27T07:13:49.229 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout "modified": "2024-05-27T07:13:34.738650+0000", 2024-05-27T07:13:49.229 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout "services": {} 2024-05-27T07:13:49.229 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout }, 2024-05-27T07:13:49.229 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout "progress_events": {} 2024-05-27T07:13:49.229 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout } 2024-05-27T07:13:49.229 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:13:48.535+0000 7fedbc9e6700 1 Processor -- start 2024-05-27T07:13:49.229 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:13:48.535+0000 7fedbc9e6700 1 -- start start 2024-05-27T07:13:49.229 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:13:48.536+0000 7fedbc9e6700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fedb807ccb0 0x7fedb810ead0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:13:49.230 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:13:48.536+0000 7fedbc9e6700 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x7fedb810f010 con 0x7fedb807ccb0 2024-05-27T07:13:49.230 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:13:48.536+0000 7fedb659c700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fedb807ccb0 0x7fedb810ead0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:13:49.230 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:13:48.536+0000 7fedb659c700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fedb807ccb0 0x7fedb810ead0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.52:39736/0 (socket says 172.21.15.52:39736) 2024-05-27T07:13:49.230 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:13:48.536+0000 7fedb659c700 1 -- 172.21.15.52:0/2484631981 learned_addr learned my addr 172.21.15.52:0/2484631981 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-05-27T07:13:49.230 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:13:48.536+0000 7fedb659c700 1 -- 172.21.15.52:0/2484631981 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fedb810f190 con 0x7fedb807ccb0 2024-05-27T07:13:49.230 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:13:48.536+0000 7fedb659c700 1 --2- 172.21.15.52:0/2484631981 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fedb807ccb0 0x7fedb810ead0 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7feda80098a0 tx=0x7feda8009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f0dfcdc7a7025b61 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:13:49.230 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:13:48.536+0000 7fedb559a700 1 -- 172.21.15.52:0/2484631981 <== mon.0 v2:172.21.15.52:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7feda8004160 con 0x7fedb807ccb0 2024-05-27T07:13:49.230 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:13:48.537+0000 7fedb559a700 1 -- 172.21.15.52:0/2484631981 <== mon.0 v2:172.21.15.52:3300/0 2 ==== config(26 keys) ==== 1070+0+0 (secure 0 0 0) 0x7feda8004300 con 0x7fedb807ccb0 2024-05-27T07:13:49.230 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:13:48.537+0000 7fedb559a700 1 -- 172.21.15.52:0/2484631981 <== mon.0 v2:172.21.15.52:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7feda8011af0 con 0x7fedb807ccb0 2024-05-27T07:13:49.230 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:13:48.537+0000 7fedbc9e6700 1 -- 172.21.15.52:0/2484631981 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fedb807ccb0 msgr2=0x7fedb810ead0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:13:49.230 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:13:48.537+0000 7fedbc9e6700 1 --2- 172.21.15.52:0/2484631981 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fedb807ccb0 0x7fedb810ead0 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7feda80098a0 tx=0x7feda8009bb0 comp rx=0 tx=0).stop 2024-05-27T07:13:49.230 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:13:48.537+0000 7fedbc9e6700 1 -- 172.21.15.52:0/2484631981 shutdown_connections 2024-05-27T07:13:49.230 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:13:48.537+0000 7fedbc9e6700 1 --2- 172.21.15.52:0/2484631981 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fedb807ccb0 0x7fedb810ead0 unknown :-1 s=CLOSED pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:13:49.231 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:13:48.537+0000 7fedbc9e6700 1 -- 172.21.15.52:0/2484631981 >> 172.21.15.52:0/2484631981 conn(0x7fedb81040e0 msgr2=0x7fedb8106570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:13:49.231 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:13:48.537+0000 7fedbc9e6700 1 -- 172.21.15.52:0/2484631981 shutdown_connections 2024-05-27T07:13:49.231 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:13:48.537+0000 7fedbc9e6700 1 -- 172.21.15.52:0/2484631981 wait complete. 2024-05-27T07:13:49.231 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:13:48.538+0000 7fedbc9e6700 1 Processor -- start 2024-05-27T07:13:49.231 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:13:48.538+0000 7fedbc9e6700 1 -- start start 2024-05-27T07:13:49.231 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:13:48.539+0000 7fedbc9e6700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fedb807ccb0 0x7fedb81a0130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:13:49.231 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:13:48.539+0000 7fedbc9e6700 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x7fedb81100f0 con 0x7fedb807ccb0 2024-05-27T07:13:49.231 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:13:48.539+0000 7fedb659c700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fedb807ccb0 0x7fedb81a0130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:13:49.231 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:13:48.539+0000 7fedb659c700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fedb807ccb0 0x7fedb81a0130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.52:39746/0 (socket says 172.21.15.52:39746) 2024-05-27T07:13:49.231 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:13:48.539+0000 7fedb659c700 1 -- 172.21.15.52:0/2685109243 learned_addr learned my addr 172.21.15.52:0/2685109243 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-05-27T07:13:49.231 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:13:48.539+0000 7fedb659c700 1 -- 172.21.15.52:0/2685109243 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fedb81a0670 con 0x7fedb807ccb0 2024-05-27T07:13:49.231 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:13:48.539+0000 7fedb659c700 1 --2- 172.21.15.52:0/2685109243 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fedb807ccb0 0x7fedb81a0130 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7feda8009fb0 tx=0x7feda8011940 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:13:49.231 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:13:48.539+0000 7feda77fe700 1 -- 172.21.15.52:0/2685109243 <== mon.0 v2:172.21.15.52:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7feda80036a0 con 0x7fedb807ccb0 2024-05-27T07:13:49.232 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:13:48.539+0000 7feda77fe700 1 -- 172.21.15.52:0/2685109243 <== mon.0 v2:172.21.15.52:3300/0 2 ==== config(26 keys) ==== 1070+0+0 (secure 0 0 0) 0x7feda801a430 con 0x7fedb807ccb0 2024-05-27T07:13:49.232 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:13:48.539+0000 7fedbc9e6700 1 -- 172.21.15.52:0/2685109243 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fedb81a0900 con 0x7fedb807ccb0 2024-05-27T07:13:49.232 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:13:48.539+0000 7feda77fe700 1 -- 172.21.15.52:0/2685109243 <== mon.0 v2:172.21.15.52:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7feda8019550 con 0x7fedb807ccb0 2024-05-27T07:13:49.232 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:13:48.540+0000 7feda77fe700 1 -- 172.21.15.52:0/2685109243 <== mon.0 v2:172.21.15.52:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7feda8019be0 con 0x7fedb807ccb0 2024-05-27T07:13:49.232 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:13:48.541+0000 7fedbc9e6700 1 -- 172.21.15.52:0/2685109243 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fedb81a3510 con 0x7fedb807ccb0 2024-05-27T07:13:49.232 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:13:48.541+0000 7feda77fe700 1 -- 172.21.15.52:0/2685109243 <== mon.0 v2:172.21.15.52:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7feda801a5e0 con 0x7fedb807ccb0 2024-05-27T07:13:49.232 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:13:48.541+0000 7fedbc9e6700 1 -- 172.21.15.52:0/2685109243 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fedb8064e10 con 0x7fedb807ccb0 2024-05-27T07:13:49.232 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:13:48.545+0000 7feda77fe700 1 -- 172.21.15.52:0/2685109243 <== mon.0 v2:172.21.15.52:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73389 (secure 0 0 0) 0x7feda8003840 con 0x7fedb807ccb0 2024-05-27T07:13:49.232 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:13:48.621+0000 7fedbc9e6700 1 -- 172.21.15.52:0/2685109243 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) -- 0x7fedb81a36e0 con 0x7fedb807ccb0 2024-05-27T07:13:49.232 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:13:48.622+0000 7feda77fe700 1 -- 172.21.15.52:0/2685109243 <== mon.0 v2:172.21.15.52:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) ==== 79+0+1239 (secure 0 0 0) 0x7feda8003a20 con 0x7fedb807ccb0 2024-05-27T07:13:49.232 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:13:48.623+0000 7fedbc9e6700 1 -- 172.21.15.52:0/2685109243 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fedb807ccb0 msgr2=0x7fedb81a0130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:13:49.232 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:13:48.623+0000 7fedbc9e6700 1 --2- 172.21.15.52:0/2685109243 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fedb807ccb0 0x7fedb81a0130 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7feda8009fb0 tx=0x7feda8011940 comp rx=0 tx=0).stop 2024-05-27T07:13:49.232 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:13:48.623+0000 7fedbc9e6700 1 -- 172.21.15.52:0/2685109243 shutdown_connections 2024-05-27T07:13:49.233 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:13:48.623+0000 7fedbc9e6700 1 --2- 172.21.15.52:0/2685109243 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fedb807ccb0 0x7fedb81a0130 unknown :-1 s=CLOSED pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:13:49.233 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:13:48.623+0000 7fedbc9e6700 1 -- 172.21.15.52:0/2685109243 >> 172.21.15.52:0/2685109243 conn(0x7fedb81040e0 msgr2=0x7fedb8104ae0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:13:49.233 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:13:48.623+0000 7fedbc9e6700 1 -- 172.21.15.52:0/2685109243 shutdown_connections 2024-05-27T07:13:49.233 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:13:48.623+0000 7fedbc9e6700 1 -- 172.21.15.52:0/2685109243 wait complete. 2024-05-27T07:13:49.233 INFO:teuthology.orchestra.run.smithi052.stdout:mgr not available, waiting (3/15)... 2024-05-27T07:13:49.480 INFO:journalctl@ceph.mgr.x.smithi052.stdout:May 27 07:13:49 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mgr-x[32184]: 2024-05-27T07:13:49.066+0000 7fcdf660c200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-05-27T07:13:49.480 INFO:journalctl@ceph.mgr.x.smithi052.stdout:May 27 07:13:49 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mgr-x[32184]: 2024-05-27T07:13:49.296+0000 7fcdf660c200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-05-27T07:13:49.980 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:13:49 smithi052 ceph-mon[31955]: Activating manager daemon x 2024-05-27T07:13:49.980 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:13:49 smithi052 ceph-mon[31955]: mgrmap e2: x(active, starting, since 0.00354507s) 2024-05-27T07:13:49.980 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:13:49 smithi052 ceph-mon[31955]: from='mgr.14100 172.21.15.52:0/228291737' entity='mgr.x' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-05-27T07:13:49.980 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:13:49 smithi052 ceph-mon[31955]: from='mgr.14100 172.21.15.52:0/228291737' entity='mgr.x' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-05-27T07:13:49.980 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:13:49 smithi052 ceph-mon[31955]: from='mgr.14100 172.21.15.52:0/228291737' entity='mgr.x' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-05-27T07:13:49.980 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:13:49 smithi052 ceph-mon[31955]: from='mgr.14100 172.21.15.52:0/228291737' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-05-27T07:13:49.980 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:13:49 smithi052 ceph-mon[31955]: from='mgr.14100 172.21.15.52:0/228291737' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-05-27T07:13:49.981 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:13:49 smithi052 ceph-mon[31955]: Manager daemon x is now available 2024-05-27T07:13:49.981 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:13:49 smithi052 ceph-mon[31955]: from='mgr.14100 172.21.15.52:0/228291737' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2024-05-27T07:13:49.981 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:13:49 smithi052 ceph-mon[31955]: from='mgr.14100 172.21.15.52:0/228291737' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2024-05-27T07:13:49.981 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:13:49 smithi052 ceph-mon[31955]: from='mgr.14100 172.21.15.52:0/228291737' entity='mgr.x' 2024-05-27T07:13:49.981 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:13:49 smithi052 ceph-mon[31955]: from='mgr.14100 172.21.15.52:0/228291737' entity='mgr.x' 2024-05-27T07:13:49.981 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:13:49 smithi052 ceph-mon[31955]: from='mgr.14100 172.21.15.52:0/228291737' entity='mgr.x' 2024-05-27T07:13:51.586 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:13:51 smithi052 ceph-mon[31955]: mgrmap e3: x(active, since 1.00814s) 2024-05-27T07:13:52.469 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout 2024-05-27T07:13:52.469 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout { 2024-05-27T07:13:52.469 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout "fsid": "73df2636-1bf8-11ef-bc9b-c7b262605968", 2024-05-27T07:13:52.469 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout "health": { 2024-05-27T07:13:52.469 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout "status": "HEALTH_OK", 2024-05-27T07:13:52.470 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout "checks": {}, 2024-05-27T07:13:52.470 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout "mutes": [] 2024-05-27T07:13:52.470 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout }, 2024-05-27T07:13:52.470 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout "election_epoch": 5, 2024-05-27T07:13:52.470 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout "quorum": [ 2024-05-27T07:13:52.470 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout 0 2024-05-27T07:13:52.470 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout ], 2024-05-27T07:13:52.470 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout "quorum_names": [ 2024-05-27T07:13:52.470 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout "a" 2024-05-27T07:13:52.470 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout ], 2024-05-27T07:13:52.470 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout "quorum_age": 12, 2024-05-27T07:13:52.470 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout "monmap": { 2024-05-27T07:13:52.470 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-05-27T07:13:52.471 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout "min_mon_release_name": "squid", 2024-05-27T07:13:52.471 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout "num_mons": 1 2024-05-27T07:13:52.471 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout }, 2024-05-27T07:13:52.471 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout "osdmap": { 2024-05-27T07:13:52.471 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-05-27T07:13:52.471 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout "num_osds": 0, 2024-05-27T07:13:52.471 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout "num_up_osds": 0, 2024-05-27T07:13:52.471 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout "osd_up_since": 0, 2024-05-27T07:13:52.479 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout "num_in_osds": 0, 2024-05-27T07:13:52.479 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout "osd_in_since": 0, 2024-05-27T07:13:52.479 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout "num_remapped_pgs": 0 2024-05-27T07:13:52.479 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout }, 2024-05-27T07:13:52.479 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout "pgmap": { 2024-05-27T07:13:52.479 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout "pgs_by_state": [], 2024-05-27T07:13:52.479 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout "num_pgs": 0, 2024-05-27T07:13:52.479 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout "num_pools": 0, 2024-05-27T07:13:52.479 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout "num_objects": 0, 2024-05-27T07:13:52.480 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout "data_bytes": 0, 2024-05-27T07:13:52.480 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout "bytes_used": 0, 2024-05-27T07:13:52.480 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout "bytes_avail": 0, 2024-05-27T07:13:52.480 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout "bytes_total": 0 2024-05-27T07:13:52.480 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout }, 2024-05-27T07:13:52.480 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout "fsmap": { 2024-05-27T07:13:52.480 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-05-27T07:13:52.480 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout "by_rank": [], 2024-05-27T07:13:52.480 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout "up:standby": 0 2024-05-27T07:13:52.480 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout }, 2024-05-27T07:13:52.480 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout "mgrmap": { 2024-05-27T07:13:52.480 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout "available": true, 2024-05-27T07:13:52.480 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout "num_standbys": 0, 2024-05-27T07:13:52.481 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout "modules": [ 2024-05-27T07:13:52.481 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout "iostat", 2024-05-27T07:13:52.481 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout "nfs", 2024-05-27T07:13:52.481 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout "restful" 2024-05-27T07:13:52.481 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout ], 2024-05-27T07:13:52.481 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout "services": {} 2024-05-27T07:13:52.481 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout }, 2024-05-27T07:13:52.481 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout "servicemap": { 2024-05-27T07:13:52.481 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-05-27T07:13:52.481 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout "modified": "2024-05-27T07:13:34.738650+0000", 2024-05-27T07:13:52.481 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout "services": {} 2024-05-27T07:13:52.481 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout }, 2024-05-27T07:13:52.481 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout "progress_events": {} 2024-05-27T07:13:52.482 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout } 2024-05-27T07:13:52.482 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:13:51.645+0000 7fcc8622d700 1 Processor -- start 2024-05-27T07:13:52.482 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:13:51.645+0000 7fcc8622d700 1 -- start start 2024-05-27T07:13:52.482 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:13:51.645+0000 7fcc8622d700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fcc80108b10 0x7fcc80108f20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:13:52.482 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:13:51.645+0000 7fcc8622d700 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x7fcc80109510 con 0x7fcc80108b10 2024-05-27T07:13:52.482 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:13:51.646+0000 7fcc7f7fe700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fcc80108b10 0x7fcc80108f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:13:52.482 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:13:51.646+0000 7fcc7f7fe700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fcc80108b10 0x7fcc80108f20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.52:51710/0 (socket says 172.21.15.52:51710) 2024-05-27T07:13:52.482 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:13:51.646+0000 7fcc7f7fe700 1 -- 172.21.15.52:0/2705909043 learned_addr learned my addr 172.21.15.52:0/2705909043 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-05-27T07:13:52.482 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:13:51.646+0000 7fcc7f7fe700 1 -- 172.21.15.52:0/2705909043 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcc80109c40 con 0x7fcc80108b10 2024-05-27T07:13:52.482 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:13:51.646+0000 7fcc7f7fe700 1 --2- 172.21.15.52:0/2705909043 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fcc80108b10 0x7fcc80108f20 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7fcc7000bb40 tx=0x7fcc7000be50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=99ff4a50a6d557ec server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:13:52.482 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:13:51.646+0000 7fcc7e7fc700 1 -- 172.21.15.52:0/2705909043 <== mon.0 v2:172.21.15.52:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fcc7000dbe0 con 0x7fcc80108b10 2024-05-27T07:13:52.482 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:13:51.646+0000 7fcc7e7fc700 1 -- 172.21.15.52:0/2705909043 <== mon.0 v2:172.21.15.52:3300/0 2 ==== config(26 keys) ==== 1070+0+0 (secure 0 0 0) 0x7fcc700044c0 con 0x7fcc80108b10 2024-05-27T07:13:52.483 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:13:51.646+0000 7fcc7e7fc700 1 -- 172.21.15.52:0/2705909043 <== mon.0 v2:172.21.15.52:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fcc70010470 con 0x7fcc80108b10 2024-05-27T07:13:52.483 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:13:51.647+0000 7fcc8622d700 1 -- 172.21.15.52:0/2705909043 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fcc80108b10 msgr2=0x7fcc80108f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:13:52.483 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:13:51.647+0000 7fcc8622d700 1 --2- 172.21.15.52:0/2705909043 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fcc80108b10 0x7fcc80108f20 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7fcc7000bb40 tx=0x7fcc7000be50 comp rx=0 tx=0).stop 2024-05-27T07:13:52.483 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:13:51.647+0000 7fcc8622d700 1 -- 172.21.15.52:0/2705909043 shutdown_connections 2024-05-27T07:13:52.483 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:13:51.647+0000 7fcc8622d700 1 --2- 172.21.15.52:0/2705909043 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fcc80108b10 0x7fcc80108f20 unknown :-1 s=CLOSED pgs=18 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:13:52.483 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:13:51.647+0000 7fcc8622d700 1 -- 172.21.15.52:0/2705909043 >> 172.21.15.52:0/2705909043 conn(0x7fcc801041b0 msgr2=0x7fcc80106600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:13:52.483 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:13:51.647+0000 7fcc8622d700 1 -- 172.21.15.52:0/2705909043 shutdown_connections 2024-05-27T07:13:52.483 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:13:51.647+0000 7fcc8622d700 1 -- 172.21.15.52:0/2705909043 wait complete. 2024-05-27T07:13:52.483 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:13:51.648+0000 7fcc8622d700 1 Processor -- start 2024-05-27T07:13:52.483 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:13:51.648+0000 7fcc8622d700 1 -- start start 2024-05-27T07:13:52.483 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:13:51.648+0000 7fcc8622d700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fcc80108b10 0x7fcc801a0180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:13:52.483 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:13:51.648+0000 7fcc8622d700 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x7fcc8007c9e0 con 0x7fcc80108b10 2024-05-27T07:13:52.484 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:13:51.649+0000 7fcc7f7fe700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fcc80108b10 0x7fcc801a0180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:13:52.484 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:13:51.649+0000 7fcc7f7fe700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fcc80108b10 0x7fcc801a0180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.52:51720/0 (socket says 172.21.15.52:51720) 2024-05-27T07:13:52.484 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:13:51.649+0000 7fcc7f7fe700 1 -- 172.21.15.52:0/2787666979 learned_addr learned my addr 172.21.15.52:0/2787666979 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-05-27T07:13:52.484 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:13:51.649+0000 7fcc7f7fe700 1 -- 172.21.15.52:0/2787666979 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcc801a06c0 con 0x7fcc80108b10 2024-05-27T07:13:52.484 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:13:51.649+0000 7fcc7f7fe700 1 --2- 172.21.15.52:0/2787666979 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fcc80108b10 0x7fcc801a0180 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7fcc7000b340 tx=0x7fcc70003dc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:13:52.484 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:13:51.649+0000 7fcc7cff9700 1 -- 172.21.15.52:0/2787666979 <== mon.0 v2:172.21.15.52:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fcc70010720 con 0x7fcc80108b10 2024-05-27T07:13:52.484 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:13:51.649+0000 7fcc7cff9700 1 -- 172.21.15.52:0/2787666979 <== mon.0 v2:172.21.15.52:3300/0 2 ==== config(26 keys) ==== 1070+0+0 (secure 0 0 0) 0x7fcc7001d070 con 0x7fcc80108b10 2024-05-27T07:13:52.484 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:13:51.649+0000 7fcc8622d700 1 -- 172.21.15.52:0/2787666979 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fcc801a0950 con 0x7fcc80108b10 2024-05-27T07:13:52.484 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:13:51.649+0000 7fcc7cff9700 1 -- 172.21.15.52:0/2787666979 <== mon.0 v2:172.21.15.52:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fcc70019ae0 con 0x7fcc80108b10 2024-05-27T07:13:52.484 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:13:51.649+0000 7fcc8622d700 1 -- 172.21.15.52:0/2787666979 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fcc801a3560 con 0x7fcc80108b10 2024-05-27T07:13:52.484 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:13:51.650+0000 7fcc7cff9700 1 -- 172.21.15.52:0/2787666979 <== mon.0 v2:172.21.15.52:3300/0 4 ==== mgrmap(e 4) ==== 50211+0+0 (secure 0 0 0) 0x7fcc70028060 con 0x7fcc80108b10 2024-05-27T07:13:52.484 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:13:51.650+0000 7fcc8622d700 1 -- 172.21.15.52:0/2787666979 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fcc80064e10 con 0x7fcc80108b10 2024-05-27T07:13:52.484 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:13:51.651+0000 7fcc7cff9700 1 --2- 172.21.15.52:0/2787666979 >> [v2:172.21.15.52:6800/587662512,v1:172.21.15.52:6801/587662512] conn(0x7fcc6803e470 0x7fcc68040980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:13:52.485 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:13:51.651+0000 7fcc7cff9700 1 -- 172.21.15.52:0/2787666979 <== mon.0 v2:172.21.15.52:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7fcc700538d0 con 0x7fcc80108b10 2024-05-27T07:13:52.485 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:13:51.651+0000 7fcc7effd700 1 --2- 172.21.15.52:0/2787666979 >> [v2:172.21.15.52:6800/587662512,v1:172.21.15.52:6801/587662512] conn(0x7fcc6803e470 0x7fcc68040980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:13:52.485 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:13:51.651+0000 7fcc7effd700 1 --2- 172.21.15.52:0/2787666979 >> [v2:172.21.15.52:6800/587662512,v1:172.21.15.52:6801/587662512] conn(0x7fcc6803e470 0x7fcc68040980 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7fcc74006fd0 tx=0x7fcc74006e40 comp rx=0 tx=0).ready entity=mgr.14100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:13:52.485 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:13:51.656+0000 7fcc7cff9700 1 -- 172.21.15.52:0/2787666979 <== mon.0 v2:172.21.15.52:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191963 (secure 0 0 0) 0x7fcc70015070 con 0x7fcc80108b10 2024-05-27T07:13:52.485 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:13:51.969+0000 7fcc8622d700 1 -- 172.21.15.52:0/2787666979 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) -- 0x7fcc80194f80 con 0x7fcc80108b10 2024-05-27T07:13:52.485 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:13:51.969+0000 7fcc7cff9700 1 -- 172.21.15.52:0/2787666979 <== mon.0 v2:172.21.15.52:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) ==== 79+0+1239 (secure 0 0 0) 0x7fcc700532f0 con 0x7fcc80108b10 2024-05-27T07:13:52.485 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:13:51.972+0000 7fcc8622d700 1 -- 172.21.15.52:0/2787666979 >> [v2:172.21.15.52:6800/587662512,v1:172.21.15.52:6801/587662512] conn(0x7fcc6803e470 msgr2=0x7fcc68040980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:13:52.485 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:13:51.972+0000 7fcc8622d700 1 --2- 172.21.15.52:0/2787666979 >> [v2:172.21.15.52:6800/587662512,v1:172.21.15.52:6801/587662512] conn(0x7fcc6803e470 0x7fcc68040980 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7fcc74006fd0 tx=0x7fcc74006e40 comp rx=0 tx=0).stop 2024-05-27T07:13:52.485 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:13:51.972+0000 7fcc8622d700 1 -- 172.21.15.52:0/2787666979 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fcc80108b10 msgr2=0x7fcc801a0180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:13:52.485 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:13:51.972+0000 7fcc8622d700 1 --2- 172.21.15.52:0/2787666979 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fcc80108b10 0x7fcc801a0180 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7fcc7000b340 tx=0x7fcc70003dc0 comp rx=0 tx=0).stop 2024-05-27T07:13:52.485 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:13:51.972+0000 7fcc8622d700 1 -- 172.21.15.52:0/2787666979 shutdown_connections 2024-05-27T07:13:52.485 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:13:51.972+0000 7fcc8622d700 1 --2- 172.21.15.52:0/2787666979 >> [v2:172.21.15.52:6800/587662512,v1:172.21.15.52:6801/587662512] conn(0x7fcc6803e470 0x7fcc68040980 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:13:52.485 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:13:51.972+0000 7fcc8622d700 1 --2- 172.21.15.52:0/2787666979 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fcc80108b10 0x7fcc801a0180 unknown :-1 s=CLOSED pgs=19 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:13:52.486 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:13:51.972+0000 7fcc8622d700 1 -- 172.21.15.52:0/2787666979 >> 172.21.15.52:0/2787666979 conn(0x7fcc801041b0 msgr2=0x7fcc80192040 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:13:52.486 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:13:51.972+0000 7fcc8622d700 1 -- 172.21.15.52:0/2787666979 shutdown_connections 2024-05-27T07:13:52.486 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:13:51.972+0000 7fcc8622d700 1 -- 172.21.15.52:0/2787666979 wait complete. 2024-05-27T07:13:52.486 INFO:teuthology.orchestra.run.smithi052.stdout:mgr is available 2024-05-27T07:13:52.598 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:13:52 smithi052 ceph-mon[31955]: mgrmap e4: x(active, since 2s) 2024-05-27T07:13:52.598 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:13:52 smithi052 ceph-mon[31955]: from='client.? 172.21.15.52:0/2787666979' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-05-27T07:13:53.444 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:13:53 smithi052 ceph-mon[31955]: from='client.? 172.21.15.52:0/1038237564' entity='client.admin' cmd=[{"prefix": "config assimilate-conf"}]: dispatch 2024-05-27T07:13:53.687 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout 2024-05-27T07:13:53.688 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout [global] 2024-05-27T07:13:53.688 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout fsid = 73df2636-1bf8-11ef-bc9b-c7b262605968 2024-05-27T07:13:53.688 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout mon_cluster_log_file_level = debug 2024-05-27T07:13:53.688 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout mon_osd_allow_pg_remap = true 2024-05-27T07:13:53.688 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout mon_osd_allow_primary_affinity = true 2024-05-27T07:13:53.688 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout mon_warn_on_no_sortbitwise = false 2024-05-27T07:13:53.688 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout osd_crush_chooseleaf_type = 0 2024-05-27T07:13:53.688 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout 2024-05-27T07:13:53.688 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout [mgr] 2024-05-27T07:13:53.688 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout mgr/telemetry/nag = false 2024-05-27T07:13:53.688 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout 2024-05-27T07:13:53.688 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout [osd] 2024-05-27T07:13:53.688 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout mon_osd_backfillfull_ratio = 0.85 2024-05-27T07:13:53.688 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout mon_osd_full_ratio = 0.9 2024-05-27T07:13:53.688 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout mon_osd_nearfull_ratio = 0.8 2024-05-27T07:13:53.689 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout osd_map_max_advance = 10 2024-05-27T07:13:53.689 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout osd_sloppy_crc = true 2024-05-27T07:13:53.689 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:13:52.922+0000 7f9ac81a1700 1 Processor -- start 2024-05-27T07:13:53.689 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:13:52.922+0000 7f9ac81a1700 1 -- start start 2024-05-27T07:13:53.689 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:13:52.923+0000 7f9ac81a1700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f9ac0108a80 0x7f9ac0108e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:13:53.689 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:13:52.923+0000 7f9ac81a1700 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x7f9ac0109480 con 0x7f9ac0108a80 2024-05-27T07:13:53.689 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:13:52.923+0000 7f9ac5f3d700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f9ac0108a80 0x7f9ac0108e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:13:53.689 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:13:52.923+0000 7f9ac5f3d700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f9ac0108a80 0x7f9ac0108e90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.52:51734/0 (socket says 172.21.15.52:51734) 2024-05-27T07:13:53.689 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:13:52.923+0000 7f9ac5f3d700 1 -- 172.21.15.52:0/504025183 learned_addr learned my addr 172.21.15.52:0/504025183 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-05-27T07:13:53.689 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:13:52.923+0000 7f9ac5f3d700 1 -- 172.21.15.52:0/504025183 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9ac0109bb0 con 0x7f9ac0108a80 2024-05-27T07:13:53.689 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:13:52.923+0000 7f9ac5f3d700 1 --2- 172.21.15.52:0/504025183 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f9ac0108a80 0x7f9ac0108e90 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7f9abc00bb40 tx=0x7f9abc00be50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5a1e6bbf9df2dc6 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:13:53.689 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:13:52.924+0000 7f9ac4f3b700 1 -- 172.21.15.52:0/504025183 <== mon.0 v2:172.21.15.52:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f9abc00dbe0 con 0x7f9ac0108a80 2024-05-27T07:13:53.689 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:13:52.924+0000 7f9ac4f3b700 1 -- 172.21.15.52:0/504025183 <== mon.0 v2:172.21.15.52:3300/0 2 ==== config(26 keys) ==== 1070+0+0 (secure 0 0 0) 0x7f9abc0044c0 con 0x7f9ac0108a80 2024-05-27T07:13:53.689 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:13:52.924+0000 7f9ac4f3b700 1 -- 172.21.15.52:0/504025183 <== mon.0 v2:172.21.15.52:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f9abc010470 con 0x7f9ac0108a80 2024-05-27T07:13:53.690 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:13:52.924+0000 7f9ac81a1700 1 -- 172.21.15.52:0/504025183 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f9ac0108a80 msgr2=0x7f9ac0108e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:13:53.690 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:13:52.924+0000 7f9ac81a1700 1 --2- 172.21.15.52:0/504025183 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f9ac0108a80 0x7f9ac0108e90 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7f9abc00bb40 tx=0x7f9abc00be50 comp rx=0 tx=0).stop 2024-05-27T07:13:53.690 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:13:52.924+0000 7f9ac81a1700 1 -- 172.21.15.52:0/504025183 shutdown_connections 2024-05-27T07:13:53.690 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:13:52.924+0000 7f9ac81a1700 1 --2- 172.21.15.52:0/504025183 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f9ac0108a80 0x7f9ac0108e90 unknown :-1 s=CLOSED pgs=20 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:13:53.690 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:13:52.924+0000 7f9ac81a1700 1 -- 172.21.15.52:0/504025183 >> 172.21.15.52:0/504025183 conn(0x7f9ac01040e0 msgr2=0x7f9ac0106570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:13:53.690 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:13:52.925+0000 7f9ac81a1700 1 -- 172.21.15.52:0/504025183 shutdown_connections 2024-05-27T07:13:53.690 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:13:52.925+0000 7f9ac81a1700 1 -- 172.21.15.52:0/504025183 wait complete. 2024-05-27T07:13:53.690 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:13:52.925+0000 7f9ac81a1700 1 Processor -- start 2024-05-27T07:13:53.690 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:13:52.925+0000 7f9ac81a1700 1 -- start start 2024-05-27T07:13:53.690 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:13:52.926+0000 7f9ac81a1700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f9ac0108a80 0x7f9ac019bcc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:13:53.690 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:13:52.926+0000 7f9ac81a1700 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x7f9ac007c150 con 0x7f9ac0108a80 2024-05-27T07:13:53.690 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:13:52.926+0000 7f9ac5f3d700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f9ac0108a80 0x7f9ac019bcc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:13:53.691 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:13:52.926+0000 7f9ac5f3d700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f9ac0108a80 0x7f9ac019bcc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.52:51750/0 (socket says 172.21.15.52:51750) 2024-05-27T07:13:53.691 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:13:52.926+0000 7f9ac5f3d700 1 -- 172.21.15.52:0/1038237564 learned_addr learned my addr 172.21.15.52:0/1038237564 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-05-27T07:13:53.691 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:13:52.926+0000 7f9ac5f3d700 1 -- 172.21.15.52:0/1038237564 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9ac019c200 con 0x7f9ac0108a80 2024-05-27T07:13:53.691 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:13:52.926+0000 7f9ac5f3d700 1 --2- 172.21.15.52:0/1038237564 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f9ac0108a80 0x7f9ac019bcc0 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7f9abc003c30 tx=0x7f9abc003d10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:13:53.691 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:13:52.926+0000 7f9ab6ffd700 1 -- 172.21.15.52:0/1038237564 <== mon.0 v2:172.21.15.52:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f9abc027070 con 0x7f9ac0108a80 2024-05-27T07:13:53.691 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:13:52.926+0000 7f9ac81a1700 1 -- 172.21.15.52:0/1038237564 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9ac019c490 con 0x7f9ac0108a80 2024-05-27T07:13:53.691 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:13:52.926+0000 7f9ab6ffd700 1 -- 172.21.15.52:0/1038237564 <== mon.0 v2:172.21.15.52:3300/0 2 ==== config(26 keys) ==== 1070+0+0 (secure 0 0 0) 0x7f9abc004300 con 0x7f9ac0108a80 2024-05-27T07:13:53.691 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:13:52.926+0000 7f9ab6ffd700 1 -- 172.21.15.52:0/1038237564 <== mon.0 v2:172.21.15.52:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f9abc0229d0 con 0x7f9ac0108a80 2024-05-27T07:13:53.691 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:13:52.926+0000 7f9ac81a1700 1 -- 172.21.15.52:0/1038237564 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9ac019f0a0 con 0x7f9ac0108a80 2024-05-27T07:13:53.691 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:13:52.927+0000 7f9ab6ffd700 1 -- 172.21.15.52:0/1038237564 <== mon.0 v2:172.21.15.52:3300/0 4 ==== mgrmap(e 4) ==== 50211+0+0 (secure 0 0 0) 0x7f9abc02a070 con 0x7f9ac0108a80 2024-05-27T07:13:53.691 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:13:52.927+0000 7f9ac81a1700 1 -- 172.21.15.52:0/1038237564 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9ac0064e10 con 0x7f9ac0108a80 2024-05-27T07:13:53.691 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:13:52.928+0000 7f9ab6ffd700 1 --2- 172.21.15.52:0/1038237564 >> [v2:172.21.15.52:6800/587662512,v1:172.21.15.52:6801/587662512] conn(0x7f9aac03e420 0x7f9aac040930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:13:53.691 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:13:52.928+0000 7f9ab6ffd700 1 -- 172.21.15.52:0/1038237564 <== mon.0 v2:172.21.15.52:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7f9abc01f070 con 0x7f9ac0108a80 2024-05-27T07:13:53.692 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:13:52.928+0000 7f9ac573c700 1 --2- 172.21.15.52:0/1038237564 >> [v2:172.21.15.52:6800/587662512,v1:172.21.15.52:6801/587662512] conn(0x7f9aac03e420 0x7f9aac040930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:13:53.692 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:13:52.928+0000 7f9ac573c700 1 --2- 172.21.15.52:0/1038237564 >> [v2:172.21.15.52:6800/587662512,v1:172.21.15.52:6801/587662512] conn(0x7f9aac03e420 0x7f9aac040930 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f9ab0006fd0 tx=0x7f9ab0006e40 comp rx=0 tx=0).ready entity=mgr.14100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:13:53.692 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:13:52.933+0000 7f9ab6ffd700 1 -- 172.21.15.52:0/1038237564 <== mon.0 v2:172.21.15.52:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191963 (secure 0 0 0) 0x7f9abc015070 con 0x7f9ac0108a80 2024-05-27T07:13:53.692 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:13:53.148+0000 7f9ac81a1700 1 -- 172.21.15.52:0/1038237564 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_command({"prefix": "config assimilate-conf"} v 0) -- 0x7f9ac01a08a0 con 0x7f9ac0108a80 2024-05-27T07:13:53.692 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:13:53.150+0000 7f9ab6ffd700 1 -- 172.21.15.52:0/1038237564 <== mon.0 v2:172.21.15.52:3300/0 7 ==== mon_command_ack([{"prefix": "config assimilate-conf"}]=0 v3) ==== 70+0+409 (secure 0 0 0) 0x7f9abc031030 con 0x7f9ac0108a80 2024-05-27T07:13:53.692 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:13:53.152+0000 7f9ac81a1700 1 -- 172.21.15.52:0/1038237564 >> [v2:172.21.15.52:6800/587662512,v1:172.21.15.52:6801/587662512] conn(0x7f9aac03e420 msgr2=0x7f9aac040930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:13:53.692 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:13:53.152+0000 7f9ac81a1700 1 --2- 172.21.15.52:0/1038237564 >> [v2:172.21.15.52:6800/587662512,v1:172.21.15.52:6801/587662512] conn(0x7f9aac03e420 0x7f9aac040930 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f9ab0006fd0 tx=0x7f9ab0006e40 comp rx=0 tx=0).stop 2024-05-27T07:13:53.692 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:13:53.152+0000 7f9ac81a1700 1 -- 172.21.15.52:0/1038237564 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f9ac0108a80 msgr2=0x7f9ac019bcc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:13:53.692 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:13:53.152+0000 7f9ac81a1700 1 --2- 172.21.15.52:0/1038237564 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f9ac0108a80 0x7f9ac019bcc0 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7f9abc003c30 tx=0x7f9abc003d10 comp rx=0 tx=0).stop 2024-05-27T07:13:53.692 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:13:53.152+0000 7f9ac81a1700 1 -- 172.21.15.52:0/1038237564 shutdown_connections 2024-05-27T07:13:53.692 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:13:53.152+0000 7f9ac81a1700 1 --2- 172.21.15.52:0/1038237564 >> [v2:172.21.15.52:6800/587662512,v1:172.21.15.52:6801/587662512] conn(0x7f9aac03e420 0x7f9aac040930 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:13:53.692 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:13:53.152+0000 7f9ac81a1700 1 --2- 172.21.15.52:0/1038237564 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f9ac0108a80 0x7f9ac019bcc0 unknown :-1 s=CLOSED pgs=21 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:13:53.692 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:13:53.152+0000 7f9ac81a1700 1 -- 172.21.15.52:0/1038237564 >> 172.21.15.52:0/1038237564 conn(0x7f9ac01040e0 msgr2=0x7f9ac0191f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:13:53.692 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:13:53.152+0000 7f9ac81a1700 1 -- 172.21.15.52:0/1038237564 shutdown_connections 2024-05-27T07:13:53.693 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:13:53.152+0000 7f9ac81a1700 1 -- 172.21.15.52:0/1038237564 wait complete. 2024-05-27T07:13:53.693 INFO:teuthology.orchestra.run.smithi052.stdout:Enabling cephadm module... 2024-05-27T07:13:54.657 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:13:54 smithi052 ceph-mon[31955]: from='client.? 172.21.15.52:0/4273111737' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "cephadm"}]: dispatch 2024-05-27T07:13:54.657 INFO:journalctl@ceph.mgr.x.smithi052.stdout:May 27 07:13:54 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mgr-x[32184]: ignoring --setuser ceph since I am not root 2024-05-27T07:13:54.658 INFO:journalctl@ceph.mgr.x.smithi052.stdout:May 27 07:13:54 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mgr-x[32184]: ignoring --setgroup ceph since I am not root 2024-05-27T07:13:54.658 INFO:journalctl@ceph.mgr.x.smithi052.stdout:May 27 07:13:54 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mgr-x[32184]: 2024-05-27T07:13:54.666+0000 7fcb6f962200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-05-27T07:13:54.967 INFO:journalctl@ceph.mgr.x.smithi052.stdout:May 27 07:13:54 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mgr-x[32184]: 2024-05-27T07:13:54.867+0000 7fcb6f962200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-05-27T07:13:54.967 INFO:journalctl@ceph.mgr.x.smithi052.stdout:May 27 07:13:54 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mgr-x[32184]: 2024-05-27T07:13:54.976+0000 7fcb6f962200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-05-27T07:13:55.023 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:13:54.123+0000 7f442b3ba700 1 Processor -- start 2024-05-27T07:13:55.028 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:13:54.123+0000 7f442b3ba700 1 -- start start 2024-05-27T07:13:55.028 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:13:54.123+0000 7f442b3ba700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f442407ccd0 0x7f442410c6b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:13:55.028 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:13:54.123+0000 7f442b3ba700 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x7f442410ccc0 con 0x7f442407ccd0 2024-05-27T07:13:55.028 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:13:54.123+0000 7f4429156700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f442407ccd0 0x7f442410c6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:13:55.028 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:13:54.123+0000 7f4429156700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f442407ccd0 0x7f442410c6b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.52:51752/0 (socket says 172.21.15.52:51752) 2024-05-27T07:13:55.029 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:13:54.123+0000 7f4429156700 1 -- 172.21.15.52:0/735011151 learned_addr learned my addr 172.21.15.52:0/735011151 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-05-27T07:13:55.029 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:13:54.124+0000 7f4429156700 1 -- 172.21.15.52:0/735011151 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f442410ce40 con 0x7f442407ccd0 2024-05-27T07:13:55.029 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:13:54.124+0000 7f4429156700 1 --2- 172.21.15.52:0/735011151 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f442407ccd0 0x7f442410c6b0 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7f44200098a0 tx=0x7f4420009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b1c27869e5d7d02b server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:13:55.029 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:13:54.124+0000 7f441bfff700 1 -- 172.21.15.52:0/735011151 <== mon.0 v2:172.21.15.52:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f4420004160 con 0x7f442407ccd0 2024-05-27T07:13:55.029 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:13:54.124+0000 7f441bfff700 1 -- 172.21.15.52:0/735011151 <== mon.0 v2:172.21.15.52:3300/0 2 ==== config(26 keys) ==== 1070+0+0 (secure 0 0 0) 0x7f4420004300 con 0x7f442407ccd0 2024-05-27T07:13:55.029 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:13:54.124+0000 7f441bfff700 1 -- 172.21.15.52:0/735011151 <== mon.0 v2:172.21.15.52:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f4420011af0 con 0x7f442407ccd0 2024-05-27T07:13:55.029 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:13:54.125+0000 7f442b3ba700 1 -- 172.21.15.52:0/735011151 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f442407ccd0 msgr2=0x7f442410c6b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:13:55.029 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:13:54.125+0000 7f442b3ba700 1 --2- 172.21.15.52:0/735011151 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f442407ccd0 0x7f442410c6b0 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7f44200098a0 tx=0x7f4420009bb0 comp rx=0 tx=0).stop 2024-05-27T07:13:55.029 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:13:54.125+0000 7f442b3ba700 1 -- 172.21.15.52:0/735011151 shutdown_connections 2024-05-27T07:13:55.029 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:13:54.125+0000 7f442b3ba700 1 --2- 172.21.15.52:0/735011151 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f442407ccd0 0x7f442410c6b0 unknown :-1 s=CLOSED pgs=22 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:13:55.029 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:13:54.125+0000 7f442b3ba700 1 -- 172.21.15.52:0/735011151 >> 172.21.15.52:0/735011151 conn(0x7f44241040e0 msgr2=0x7f4424106570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:13:55.029 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:13:54.125+0000 7f442b3ba700 1 -- 172.21.15.52:0/735011151 shutdown_connections 2024-05-27T07:13:55.030 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:13:54.125+0000 7f442b3ba700 1 -- 172.21.15.52:0/735011151 wait complete. 2024-05-27T07:13:55.030 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:13:54.126+0000 7f442b3ba700 1 Processor -- start 2024-05-27T07:13:55.030 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:13:54.126+0000 7f442b3ba700 1 -- start start 2024-05-27T07:13:55.030 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:13:54.126+0000 7f442b3ba700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f442407ccd0 0x7f442419df60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:13:55.030 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:13:54.126+0000 7f442b3ba700 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x7f442410dec0 con 0x7f442407ccd0 2024-05-27T07:13:55.030 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:13:54.126+0000 7f4429156700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f442407ccd0 0x7f442419df60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:13:55.030 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:13:54.127+0000 7f4429156700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f442407ccd0 0x7f442419df60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.52:51754/0 (socket says 172.21.15.52:51754) 2024-05-27T07:13:55.030 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:13:54.127+0000 7f4429156700 1 -- 172.21.15.52:0/4273111737 learned_addr learned my addr 172.21.15.52:0/4273111737 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-05-27T07:13:55.030 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:13:54.127+0000 7f4429156700 1 -- 172.21.15.52:0/4273111737 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f442419e4a0 con 0x7f442407ccd0 2024-05-27T07:13:55.030 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:13:54.127+0000 7f4429156700 1 --2- 172.21.15.52:0/4273111737 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f442407ccd0 0x7f442419df60 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f4420009fb0 tx=0x7f4420011940 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:13:55.030 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:13:54.127+0000 7f441a7fc700 1 -- 172.21.15.52:0/4273111737 <== mon.0 v2:172.21.15.52:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f44200036a0 con 0x7f442407ccd0 2024-05-27T07:13:55.030 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:13:54.127+0000 7f441a7fc700 1 -- 172.21.15.52:0/4273111737 <== mon.0 v2:172.21.15.52:3300/0 2 ==== config(26 keys) ==== 1070+0+0 (secure 0 0 0) 0x7f442001a430 con 0x7f442407ccd0 2024-05-27T07:13:55.030 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:13:54.127+0000 7f442b3ba700 1 -- 172.21.15.52:0/4273111737 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f442419e730 con 0x7f442407ccd0 2024-05-27T07:13:55.031 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:13:54.127+0000 7f441a7fc700 1 -- 172.21.15.52:0/4273111737 <== mon.0 v2:172.21.15.52:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f4420019550 con 0x7f442407ccd0 2024-05-27T07:13:55.031 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:13:54.127+0000 7f442b3ba700 1 -- 172.21.15.52:0/4273111737 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f44241a1340 con 0x7f442407ccd0 2024-05-27T07:13:55.031 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:13:54.128+0000 7f441a7fc700 1 -- 172.21.15.52:0/4273111737 <== mon.0 v2:172.21.15.52:3300/0 4 ==== mgrmap(e 4) ==== 50211+0+0 (secure 0 0 0) 0x7f4420019850 con 0x7f442407ccd0 2024-05-27T07:13:55.031 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:13:54.128+0000 7f442b3ba700 1 -- 172.21.15.52:0/4273111737 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4424064e10 con 0x7f442407ccd0 2024-05-27T07:13:55.031 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:13:54.129+0000 7f441a7fc700 1 --2- 172.21.15.52:0/4273111737 >> [v2:172.21.15.52:6800/587662512,v1:172.21.15.52:6801/587662512] conn(0x7f441003e3d0 0x7f44100408e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:13:55.031 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:13:54.129+0000 7f441a7fc700 1 -- 172.21.15.52:0/4273111737 <== mon.0 v2:172.21.15.52:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7f4420053960 con 0x7f442407ccd0 2024-05-27T07:13:55.031 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:13:54.129+0000 7f4428955700 1 --2- 172.21.15.52:0/4273111737 >> [v2:172.21.15.52:6800/587662512,v1:172.21.15.52:6801/587662512] conn(0x7f441003e3d0 0x7f44100408e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:13:55.031 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:13:54.129+0000 7f4428955700 1 --2- 172.21.15.52:0/4273111737 >> [v2:172.21.15.52:6800/587662512,v1:172.21.15.52:6801/587662512] conn(0x7f441003e3d0 0x7f44100408e0 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f4414006fd0 tx=0x7f4414006e40 comp rx=0 tx=0).ready entity=mgr.14100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:13:55.031 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:13:54.134+0000 7f441a7fc700 1 -- 172.21.15.52:0/4273111737 <== mon.0 v2:172.21.15.52:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191963 (secure 0 0 0) 0x7f4420053380 con 0x7f442407ccd0 2024-05-27T07:13:55.031 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:13:54.409+0000 7f442b3ba700 1 -- 172.21.15.52:0/4273111737 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_command({"prefix": "mgr module enable", "module": "cephadm"} v 0) -- 0x7f44241a1650 con 0x7f442407ccd0 2024-05-27T07:13:55.031 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:13:54.460+0000 7f441a7fc700 1 -- 172.21.15.52:0/4273111737 <== mon.0 v2:172.21.15.52:3300/0 7 ==== mon_command_ack([{"prefix": "mgr module enable", "module": "cephadm"}]=0 v5) ==== 86+0+0 (secure 0 0 0) 0x7f442001a5e0 con 0x7f442407ccd0 2024-05-27T07:13:55.031 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:13:54.462+0000 7f441a7fc700 1 -- 172.21.15.52:0/4273111737 <== mon.0 v2:172.21.15.52:3300/0 8 ==== mgrmap(e 5) ==== 50222+0+0 (secure 0 0 0) 0x7f442002d440 con 0x7f442407ccd0 2024-05-27T07:13:55.031 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:13:54.464+0000 7f442b3ba700 1 -- 172.21.15.52:0/4273111737 >> [v2:172.21.15.52:6800/587662512,v1:172.21.15.52:6801/587662512] conn(0x7f441003e3d0 msgr2=0x7f44100408e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:13:55.031 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:13:54.464+0000 7f442b3ba700 1 --2- 172.21.15.52:0/4273111737 >> [v2:172.21.15.52:6800/587662512,v1:172.21.15.52:6801/587662512] conn(0x7f441003e3d0 0x7f44100408e0 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f4414006fd0 tx=0x7f4414006e40 comp rx=0 tx=0).stop 2024-05-27T07:13:55.032 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:13:54.464+0000 7f442b3ba700 1 -- 172.21.15.52:0/4273111737 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f442407ccd0 msgr2=0x7f442419df60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:13:55.032 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:13:54.464+0000 7f442b3ba700 1 --2- 172.21.15.52:0/4273111737 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f442407ccd0 0x7f442419df60 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f4420009fb0 tx=0x7f4420011940 comp rx=0 tx=0).stop 2024-05-27T07:13:55.032 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:13:54.464+0000 7f442b3ba700 1 -- 172.21.15.52:0/4273111737 shutdown_connections 2024-05-27T07:13:55.032 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:13:54.464+0000 7f442b3ba700 1 --2- 172.21.15.52:0/4273111737 >> [v2:172.21.15.52:6800/587662512,v1:172.21.15.52:6801/587662512] conn(0x7f441003e3d0 0x7f44100408e0 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:13:55.032 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:13:54.464+0000 7f442b3ba700 1 --2- 172.21.15.52:0/4273111737 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f442407ccd0 0x7f442419df60 unknown :-1 s=CLOSED pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:13:55.032 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:13:54.464+0000 7f442b3ba700 1 -- 172.21.15.52:0/4273111737 >> 172.21.15.52:0/4273111737 conn(0x7f44241040e0 msgr2=0x7f4424104a90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:13:55.032 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:13:54.464+0000 7f442b3ba700 1 -- 172.21.15.52:0/4273111737 shutdown_connections 2024-05-27T07:13:55.032 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:13:54.464+0000 7f442b3ba700 1 -- 172.21.15.52:0/4273111737 wait complete. 2024-05-27T07:13:55.687 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:13:55 smithi052 ceph-mon[31955]: from='client.? 172.21.15.52:0/4273111737' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "cephadm"}]': finished 2024-05-27T07:13:55.687 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:13:55 smithi052 ceph-mon[31955]: mgrmap e5: x(active, since 5s) 2024-05-27T07:13:55.687 INFO:journalctl@ceph.mgr.x.smithi052.stdout:May 27 07:13:55 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mgr-x[32184]: 2024-05-27T07:13:55.311+0000 7fcb6f962200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-05-27T07:13:55.687 INFO:journalctl@ceph.mgr.x.smithi052.stdout:May 27 07:13:55 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mgr-x[32184]: 2024-05-27T07:13:55.403+0000 7fcb6f962200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-05-27T07:13:56.368 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout { 2024-05-27T07:13:56.369 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout "epoch": 5, 2024-05-27T07:13:56.369 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout "available": true, 2024-05-27T07:13:56.369 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout "active_name": "x", 2024-05-27T07:13:56.369 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout "num_standby": 0 2024-05-27T07:13:56.369 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout } 2024-05-27T07:13:56.369 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:13:55.489+0000 7f50ded61700 1 Processor -- start 2024-05-27T07:13:56.369 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:13:55.489+0000 7f50ded61700 1 -- start start 2024-05-27T07:13:56.369 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:13:55.490+0000 7f50ded61700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f50d807cc50 0x7f50d810c730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:13:56.369 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:13:55.490+0000 7f50ded61700 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x7f50d810cd00 con 0x7f50d807cc50 2024-05-27T07:13:56.369 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:13:55.490+0000 7f50dcafd700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f50d807cc50 0x7f50d810c730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:13:56.369 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:13:55.490+0000 7f50dcafd700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f50d807cc50 0x7f50d810c730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.52:51772/0 (socket says 172.21.15.52:51772) 2024-05-27T07:13:56.369 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:13:55.490+0000 7f50dcafd700 1 -- 172.21.15.52:0/3615225839 learned_addr learned my addr 172.21.15.52:0/3615225839 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-05-27T07:13:56.370 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:13:55.490+0000 7f50dcafd700 1 -- 172.21.15.52:0/3615225839 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f50d810ce80 con 0x7f50d807cc50 2024-05-27T07:13:56.370 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:13:55.490+0000 7f50dcafd700 1 --2- 172.21.15.52:0/3615225839 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f50d807cc50 0x7f50d810c730 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f50cc0098a0 tx=0x7f50cc009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8f88e7110bd2dcb0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:13:56.370 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:13:55.491+0000 7f50d77fe700 1 -- 172.21.15.52:0/3615225839 <== mon.0 v2:172.21.15.52:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f50cc004160 con 0x7f50d807cc50 2024-05-27T07:13:56.370 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:13:55.491+0000 7f50d77fe700 1 -- 172.21.15.52:0/3615225839 <== mon.0 v2:172.21.15.52:3300/0 2 ==== config(26 keys) ==== 1070+0+0 (secure 0 0 0) 0x7f50cc004300 con 0x7f50d807cc50 2024-05-27T07:13:56.370 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:13:55.491+0000 7f50d77fe700 1 -- 172.21.15.52:0/3615225839 <== mon.0 v2:172.21.15.52:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f50cc011af0 con 0x7f50d807cc50 2024-05-27T07:13:56.370 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:13:55.491+0000 7f50ded61700 1 -- 172.21.15.52:0/3615225839 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f50d807cc50 msgr2=0x7f50d810c730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:13:56.370 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:13:55.491+0000 7f50ded61700 1 --2- 172.21.15.52:0/3615225839 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f50d807cc50 0x7f50d810c730 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f50cc0098a0 tx=0x7f50cc009bb0 comp rx=0 tx=0).stop 2024-05-27T07:13:56.370 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:13:55.491+0000 7f50ded61700 1 -- 172.21.15.52:0/3615225839 shutdown_connections 2024-05-27T07:13:56.370 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:13:55.491+0000 7f50ded61700 1 --2- 172.21.15.52:0/3615225839 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f50d807cc50 0x7f50d810c730 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:13:56.370 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:13:55.491+0000 7f50ded61700 1 -- 172.21.15.52:0/3615225839 >> 172.21.15.52:0/3615225839 conn(0x7f50d81040e0 msgr2=0x7f50d8106570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:13:56.370 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:13:55.492+0000 7f50ded61700 1 -- 172.21.15.52:0/3615225839 shutdown_connections 2024-05-27T07:13:56.370 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:13:55.492+0000 7f50ded61700 1 -- 172.21.15.52:0/3615225839 wait complete. 2024-05-27T07:13:56.371 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:13:55.492+0000 7f50ded61700 1 Processor -- start 2024-05-27T07:13:56.371 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:13:55.493+0000 7f50ded61700 1 -- start start 2024-05-27T07:13:56.371 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:13:55.493+0000 7f50ded61700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f50d807cc50 0x7f50d819df40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:13:56.371 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:13:55.493+0000 7f50ded61700 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x7f50d810df00 con 0x7f50d807cc50 2024-05-27T07:13:56.371 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:13:55.493+0000 7f50dcafd700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f50d807cc50 0x7f50d819df40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:13:56.371 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:13:55.493+0000 7f50dcafd700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f50d807cc50 0x7f50d819df40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.52:51784/0 (socket says 172.21.15.52:51784) 2024-05-27T07:13:56.371 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:13:55.493+0000 7f50dcafd700 1 -- 172.21.15.52:0/1717100417 learned_addr learned my addr 172.21.15.52:0/1717100417 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-05-27T07:13:56.371 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:13:55.493+0000 7f50dcafd700 1 -- 172.21.15.52:0/1717100417 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f50d819e480 con 0x7f50d807cc50 2024-05-27T07:13:56.371 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:13:55.494+0000 7f50dcafd700 1 --2- 172.21.15.52:0/1717100417 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f50d807cc50 0x7f50d819df40 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f50cc009fb0 tx=0x7f50cc011940 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:13:56.371 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:13:55.494+0000 7f50d5ffb700 1 -- 172.21.15.52:0/1717100417 <== mon.0 v2:172.21.15.52:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f50cc0036a0 con 0x7f50d807cc50 2024-05-27T07:13:56.371 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:13:55.494+0000 7f50d5ffb700 1 -- 172.21.15.52:0/1717100417 <== mon.0 v2:172.21.15.52:3300/0 2 ==== config(26 keys) ==== 1070+0+0 (secure 0 0 0) 0x7f50cc01a430 con 0x7f50d807cc50 2024-05-27T07:13:56.371 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:13:55.494+0000 7f50ded61700 1 -- 172.21.15.52:0/1717100417 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f50d819e710 con 0x7f50d807cc50 2024-05-27T07:13:56.371 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:13:55.494+0000 7f50d5ffb700 1 -- 172.21.15.52:0/1717100417 <== mon.0 v2:172.21.15.52:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f50cc019550 con 0x7f50d807cc50 2024-05-27T07:13:56.371 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:13:55.495+0000 7f50ded61700 1 -- 172.21.15.52:0/1717100417 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f50d81a1320 con 0x7f50d807cc50 2024-05-27T07:13:56.372 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:13:55.495+0000 7f50d5ffb700 1 -- 172.21.15.52:0/1717100417 <== mon.0 v2:172.21.15.52:3300/0 4 ==== mgrmap(e 5) ==== 50222+0+0 (secure 0 0 0) 0x7f50cc019800 con 0x7f50d807cc50 2024-05-27T07:13:56.372 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:13:55.495+0000 7f50d5ffb700 1 --2- 172.21.15.52:0/1717100417 >> [v2:172.21.15.52:6800/587662512,v1:172.21.15.52:6801/587662512] conn(0x7f50c803e470 0x7f50c8040980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:13:56.372 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:13:55.495+0000 7f50d5ffb700 1 -- 172.21.15.52:0/1717100417 <== mon.0 v2:172.21.15.52:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7f50cc059060 con 0x7f50d807cc50 2024-05-27T07:13:56.372 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:13:55.495+0000 7f50d7fff700 1 -- 172.21.15.52:0/1717100417 >> [v2:172.21.15.52:6800/587662512,v1:172.21.15.52:6801/587662512] conn(0x7f50c803e470 msgr2=0x7f50c8040980 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.52:6800/587662512 2024-05-27T07:13:56.372 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:13:55.495+0000 7f50d7fff700 1 --2- 172.21.15.52:0/1717100417 >> [v2:172.21.15.52:6800/587662512,v1:172.21.15.52:6801/587662512] conn(0x7f50c803e470 0x7f50c8040980 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-05-27T07:13:56.372 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:13:55.496+0000 7f50ded61700 1 -- 172.21.15.52:0/1717100417 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f50d8064e10 con 0x7f50d807cc50 2024-05-27T07:13:56.372 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:13:55.501+0000 7f50d5ffb700 1 -- 172.21.15.52:0/1717100417 <== mon.0 v2:172.21.15.52:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191963 (secure 0 0 0) 0x7f50cc018350 con 0x7f50d807cc50 2024-05-27T07:13:56.372 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:13:55.696+0000 7f50d7fff700 1 -- 172.21.15.52:0/1717100417 >> [v2:172.21.15.52:6800/587662512,v1:172.21.15.52:6801/587662512] conn(0x7f50c803e470 msgr2=0x7f50c8040980 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.52:6800/587662512 2024-05-27T07:13:56.372 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:13:55.696+0000 7f50d7fff700 1 --2- 172.21.15.52:0/1717100417 >> [v2:172.21.15.52:6800/587662512,v1:172.21.15.52:6801/587662512] conn(0x7f50c803e470 0x7f50c8040980 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-05-27T07:13:56.372 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:13:55.775+0000 7f50ded61700 1 -- 172.21.15.52:0/1717100417 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_command({"prefix": "mgr stat"} v 0) -- 0x7f50d81a14f0 con 0x7f50d807cc50 2024-05-27T07:13:56.372 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:13:55.775+0000 7f50d5ffb700 1 -- 172.21.15.52:0/1717100417 <== mon.0 v2:172.21.15.52:3300/0 7 ==== mon_command_ack([{"prefix": "mgr stat"}]=0 v5) ==== 56+0+88 (secure 0 0 0) 0x7f50cc010e70 con 0x7f50d807cc50 2024-05-27T07:13:56.372 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:13:55.777+0000 7f50ded61700 1 -- 172.21.15.52:0/1717100417 >> [v2:172.21.15.52:6800/587662512,v1:172.21.15.52:6801/587662512] conn(0x7f50c803e470 msgr2=0x7f50c8040980 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-05-27T07:13:56.373 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:13:55.777+0000 7f50ded61700 1 --2- 172.21.15.52:0/1717100417 >> [v2:172.21.15.52:6800/587662512,v1:172.21.15.52:6801/587662512] conn(0x7f50c803e470 0x7f50c8040980 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:13:56.373 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:13:55.778+0000 7f50ded61700 1 -- 172.21.15.52:0/1717100417 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f50d807cc50 msgr2=0x7f50d819df40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:13:56.373 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:13:55.778+0000 7f50ded61700 1 --2- 172.21.15.52:0/1717100417 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f50d807cc50 0x7f50d819df40 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f50cc009fb0 tx=0x7f50cc011940 comp rx=0 tx=0).stop 2024-05-27T07:13:56.373 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:13:55.778+0000 7f50ded61700 1 -- 172.21.15.52:0/1717100417 shutdown_connections 2024-05-27T07:13:56.373 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:13:55.778+0000 7f50ded61700 1 --2- 172.21.15.52:0/1717100417 >> [v2:172.21.15.52:6800/587662512,v1:172.21.15.52:6801/587662512] conn(0x7f50c803e470 0x7f50c8040980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:13:56.373 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:13:55.778+0000 7f50ded61700 1 --2- 172.21.15.52:0/1717100417 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f50d807cc50 0x7f50d819df40 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:13:56.373 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:13:55.778+0000 7f50ded61700 1 -- 172.21.15.52:0/1717100417 >> 172.21.15.52:0/1717100417 conn(0x7f50d81040e0 msgr2=0x7f50d8104b00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:13:56.373 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:13:55.778+0000 7f50ded61700 1 -- 172.21.15.52:0/1717100417 shutdown_connections 2024-05-27T07:13:56.373 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:13:55.778+0000 7f50ded61700 1 -- 172.21.15.52:0/1717100417 wait complete. 2024-05-27T07:13:56.373 INFO:teuthology.orchestra.run.smithi052.stdout:Waiting for the mgr to restart... 2024-05-27T07:13:56.373 INFO:teuthology.orchestra.run.smithi052.stdout:Waiting for mgr epoch 5... 2024-05-27T07:13:56.672 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:13:56 smithi052 ceph-mon[31955]: from='client.? 172.21.15.52:0/1717100417' entity='client.admin' cmd=[{"prefix": "mgr stat"}]: dispatch 2024-05-27T07:13:56.980 INFO:journalctl@ceph.mgr.x.smithi052.stdout:May 27 07:13:56 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mgr-x[32184]: 2024-05-27T07:13:56.681+0000 7fcb6f962200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-05-27T07:13:56.980 INFO:journalctl@ceph.mgr.x.smithi052.stdout:May 27 07:13:56 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mgr-x[32184]: 2024-05-27T07:13:56.869+0000 7fcb6f962200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-05-27T07:13:57.344 INFO:journalctl@ceph.mgr.x.smithi052.stdout:May 27 07:13:57 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mgr-x[32184]: 2024-05-27T07:13:57.048+0000 7fcb6f962200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-05-27T07:13:57.345 INFO:journalctl@ceph.mgr.x.smithi052.stdout:May 27 07:13:57 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mgr-x[32184]: 2024-05-27T07:13:57.354+0000 7fcb6f962200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-05-27T07:13:57.654 INFO:journalctl@ceph.mgr.x.smithi052.stdout:May 27 07:13:57 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mgr-x[32184]: 2024-05-27T07:13:57.452+0000 7fcb6f962200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-05-27T07:13:57.655 INFO:journalctl@ceph.mgr.x.smithi052.stdout:May 27 07:13:57 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mgr-x[32184]: 2024-05-27T07:13:57.664+0000 7fcb6f962200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-05-27T07:13:57.980 INFO:journalctl@ceph.mgr.x.smithi052.stdout:May 27 07:13:57 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mgr-x[32184]: 2024-05-27T07:13:57.868+0000 7fcb6f962200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-05-27T07:13:58.980 INFO:journalctl@ceph.mgr.x.smithi052.stdout:May 27 07:13:58 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mgr-x[32184]: 2024-05-27T07:13:58.503+0000 7fcb6f962200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-05-27T07:13:58.980 INFO:journalctl@ceph.mgr.x.smithi052.stdout:May 27 07:13:58 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mgr-x[32184]: 2024-05-27T07:13:58.738+0000 7fcb6f962200 -1 mgr[py] Module smb has missing NOTIFY_TYPES member 2024-05-27T07:13:59.480 INFO:journalctl@ceph.mgr.x.smithi052.stdout:May 27 07:13:59 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mgr-x[32184]: 2024-05-27T07:13:59.186+0000 7fcb6f962200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-05-27T07:13:59.480 INFO:journalctl@ceph.mgr.x.smithi052.stdout:May 27 07:13:59 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mgr-x[32184]: 2024-05-27T07:13:59.288+0000 7fcb6f962200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-05-27T07:13:59.980 INFO:journalctl@ceph.mgr.x.smithi052.stdout:May 27 07:13:59 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mgr-x[32184]: 2024-05-27T07:13:59.675+0000 7fcb6f962200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-05-27T07:14:00.400 INFO:journalctl@ceph.mgr.x.smithi052.stdout:May 27 07:14:00 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mgr-x[32184]: 2024-05-27T07:14:00.053+0000 7fcb6f962200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-05-27T07:14:00.400 INFO:journalctl@ceph.mgr.x.smithi052.stdout:May 27 07:14:00 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mgr-x[32184]: 2024-05-27T07:14:00.151+0000 7fcb6f962200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-05-27T07:14:00.730 INFO:journalctl@ceph.mgr.x.smithi052.stdout:May 27 07:14:00 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mgr-x[32184]: 2024-05-27T07:14:00.409+0000 7fcb6f962200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-05-27T07:14:00.730 INFO:journalctl@ceph.mgr.x.smithi052.stdout:May 27 07:14:00 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mgr-x[32184]: 2024-05-27T07:14:00.550+0000 7fcb6f962200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-05-27T07:14:00.730 INFO:journalctl@ceph.mgr.x.smithi052.stdout:May 27 07:14:00 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mgr-x[32184]: 2024-05-27T07:14:00.652+0000 7fcb6f962200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-05-27T07:14:01.480 INFO:journalctl@ceph.mgr.x.smithi052.stdout:May 27 07:14:00 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mgr-x[32184]: 2024-05-27T07:14:00.993+0000 7fcb6f962200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-05-27T07:14:01.480 INFO:journalctl@ceph.mgr.x.smithi052.stdout:May 27 07:14:01 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mgr-x[32184]: 2024-05-27T07:14:01.091+0000 7fcb6f962200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-05-27T07:14:01.980 INFO:journalctl@ceph.mgr.x.smithi052.stdout:May 27 07:14:01 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mgr-x[32184]: 2024-05-27T07:14:01.727+0000 7fcb6f962200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-05-27T07:14:01.980 INFO:journalctl@ceph.mgr.x.smithi052.stdout:May 27 07:14:01 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mgr-x[32184]: 2024-05-27T07:14:01.958+0000 7fcb6f962200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-05-27T07:14:02.480 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:14:02 smithi052 ceph-mon[31955]: Active manager daemon x restarted 2024-05-27T07:14:02.480 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:14:02 smithi052 ceph-mon[31955]: Activating manager daemon x 2024-05-27T07:14:02.480 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:14:02 smithi052 ceph-mon[31955]: osdmap e2: 0 total, 0 up, 0 in 2024-05-27T07:14:02.480 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:14:02 smithi052 ceph-mon[31955]: mgrmap e6: x(active, starting, since 0.00406981s) 2024-05-27T07:14:02.480 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:14:02 smithi052 ceph-mon[31955]: from='mgr.14120 172.21.15.52:0/2490234989' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-05-27T07:14:02.481 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:14:02 smithi052 ceph-mon[31955]: from='mgr.14120 172.21.15.52:0/2490234989' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-05-27T07:14:02.481 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:14:02 smithi052 ceph-mon[31955]: from='mgr.14120 172.21.15.52:0/2490234989' entity='mgr.x' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-05-27T07:14:02.481 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:14:02 smithi052 ceph-mon[31955]: from='mgr.14120 172.21.15.52:0/2490234989' entity='mgr.x' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-05-27T07:14:02.481 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:14:02 smithi052 ceph-mon[31955]: from='mgr.14120 172.21.15.52:0/2490234989' entity='mgr.x' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-05-27T07:14:02.481 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:14:02 smithi052 ceph-mon[31955]: Manager daemon x is now available 2024-05-27T07:14:02.481 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:14:02 smithi052 ceph-mon[31955]: from='mgr.14120 172.21.15.52:0/2490234989' entity='mgr.x' 2024-05-27T07:14:02.481 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:14:02 smithi052 ceph-mon[31955]: from='mgr.14120 172.21.15.52:0/2490234989' entity='mgr.x' 2024-05-27T07:14:02.481 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:14:02 smithi052 ceph-mon[31955]: from='mgr.14120 172.21.15.52:0/2490234989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-27T07:14:02.481 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:14:02 smithi052 ceph-mon[31955]: from='mgr.14120 172.21.15.52:0/2490234989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-27T07:14:03.230 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:14:03 smithi052 ceph-mon[31955]: Found migration_current of "None". Setting to last migration. 2024-05-27T07:14:03.230 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:14:03 smithi052 ceph-mon[31955]: from='mgr.14120 172.21.15.52:0/2490234989' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2024-05-27T07:14:03.230 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:14:03 smithi052 ceph-mon[31955]: from='mgr.14120 172.21.15.52:0/2490234989' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2024-05-27T07:14:03.230 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:14:03 smithi052 ceph-mon[31955]: from='mgr.14120 172.21.15.52:0/2490234989' entity='mgr.x' 2024-05-27T07:14:03.230 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:14:03 smithi052 ceph-mon[31955]: from='mgr.14120 172.21.15.52:0/2490234989' entity='mgr.x' 2024-05-27T07:14:03.230 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:14:03 smithi052 ceph-mon[31955]: mgrmap e7: x(active, since 1.00654s) 2024-05-27T07:14:03.585 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout { 2024-05-27T07:14:03.585 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout "mgrmap_epoch": 7, 2024-05-27T07:14:03.585 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout "initialized": true 2024-05-27T07:14:03.585 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout } 2024-05-27T07:14:03.585 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:13:56.825+0000 7f641fa17700 1 Processor -- start 2024-05-27T07:14:03.585 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:13:56.825+0000 7f641fa17700 1 -- start start 2024-05-27T07:14:03.585 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:13:56.826+0000 7f641fa17700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f6418108aa0 0x7f6418108eb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:14:03.585 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:13:56.826+0000 7f641fa17700 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x7f64181094a0 con 0x7f6418108aa0 2024-05-27T07:14:03.586 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:13:56.826+0000 7f641d7b3700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f6418108aa0 0x7f6418108eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:14:03.586 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:13:56.826+0000 7f641d7b3700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f6418108aa0 0x7f6418108eb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.52:51796/0 (socket says 172.21.15.52:51796) 2024-05-27T07:14:03.586 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:13:56.826+0000 7f641d7b3700 1 -- 172.21.15.52:0/2775239397 learned_addr learned my addr 172.21.15.52:0/2775239397 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-05-27T07:14:03.586 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:13:56.826+0000 7f641d7b3700 1 -- 172.21.15.52:0/2775239397 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6418109bd0 con 0x7f6418108aa0 2024-05-27T07:14:03.586 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:13:56.826+0000 7f641d7b3700 1 --2- 172.21.15.52:0/2775239397 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f6418108aa0 0x7f6418108eb0 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f64140098a0 tx=0x7f6414009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=83152e39bacaa51 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:14:03.586 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:13:56.827+0000 7f640ffff700 1 -- 172.21.15.52:0/2775239397 <== mon.0 v2:172.21.15.52:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f6414004160 con 0x7f6418108aa0 2024-05-27T07:14:03.586 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:13:56.827+0000 7f640ffff700 1 -- 172.21.15.52:0/2775239397 <== mon.0 v2:172.21.15.52:3300/0 2 ==== config(26 keys) ==== 1070+0+0 (secure 0 0 0) 0x7f6414004300 con 0x7f6418108aa0 2024-05-27T07:14:03.586 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:13:56.827+0000 7f640ffff700 1 -- 172.21.15.52:0/2775239397 <== mon.0 v2:172.21.15.52:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f6414011af0 con 0x7f6418108aa0 2024-05-27T07:14:03.586 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:13:56.827+0000 7f641fa17700 1 -- 172.21.15.52:0/2775239397 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f6418108aa0 msgr2=0x7f6418108eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:14:03.586 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:13:56.827+0000 7f641fa17700 1 --2- 172.21.15.52:0/2775239397 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f6418108aa0 0x7f6418108eb0 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f64140098a0 tx=0x7f6414009bb0 comp rx=0 tx=0).stop 2024-05-27T07:14:03.586 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:13:56.828+0000 7f641fa17700 1 -- 172.21.15.52:0/2775239397 shutdown_connections 2024-05-27T07:14:03.586 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:13:56.828+0000 7f641fa17700 1 --2- 172.21.15.52:0/2775239397 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f6418108aa0 0x7f6418108eb0 unknown :-1 s=CLOSED pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:14:03.586 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:13:56.828+0000 7f641fa17700 1 -- 172.21.15.52:0/2775239397 >> 172.21.15.52:0/2775239397 conn(0x7f64181040e0 msgr2=0x7f6418106590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:14:03.587 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:13:56.828+0000 7f641fa17700 1 -- 172.21.15.52:0/2775239397 shutdown_connections 2024-05-27T07:14:03.587 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:13:56.828+0000 7f641fa17700 1 -- 172.21.15.52:0/2775239397 wait complete. 2024-05-27T07:14:03.587 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:13:56.829+0000 7f641fa17700 1 Processor -- start 2024-05-27T07:14:03.587 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:13:56.829+0000 7f641fa17700 1 -- start start 2024-05-27T07:14:03.587 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:13:56.829+0000 7f641fa17700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f6418108aa0 0x7f641819cec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:14:03.587 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:13:56.829+0000 7f641fa17700 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x7f641810cf80 con 0x7f6418108aa0 2024-05-27T07:14:03.587 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:13:56.829+0000 7f641d7b3700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f6418108aa0 0x7f641819cec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:14:03.587 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:13:56.829+0000 7f641d7b3700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f6418108aa0 0x7f641819cec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.52:51808/0 (socket says 172.21.15.52:51808) 2024-05-27T07:14:03.587 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:13:56.829+0000 7f641d7b3700 1 -- 172.21.15.52:0/2539359707 learned_addr learned my addr 172.21.15.52:0/2539359707 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-05-27T07:14:03.587 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:13:56.829+0000 7f641d7b3700 1 -- 172.21.15.52:0/2539359707 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f641819d400 con 0x7f6418108aa0 2024-05-27T07:14:03.587 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:13:56.830+0000 7f641d7b3700 1 --2- 172.21.15.52:0/2539359707 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f6418108aa0 0x7f641819cec0 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f6414011890 tx=0x7f6414011970 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:14:03.587 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:13:56.830+0000 7f640e7fc700 1 -- 172.21.15.52:0/2539359707 <== mon.0 v2:172.21.15.52:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f64140039e0 con 0x7f6418108aa0 2024-05-27T07:14:03.587 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:13:56.830+0000 7f640e7fc700 1 -- 172.21.15.52:0/2539359707 <== mon.0 v2:172.21.15.52:3300/0 2 ==== config(26 keys) ==== 1070+0+0 (secure 0 0 0) 0x7f6414003b80 con 0x7f6418108aa0 2024-05-27T07:14:03.588 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:13:56.830+0000 7f641fa17700 1 -- 172.21.15.52:0/2539359707 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f641819d690 con 0x7f6418108aa0 2024-05-27T07:14:03.588 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:13:56.830+0000 7f640e7fc700 1 -- 172.21.15.52:0/2539359707 <== mon.0 v2:172.21.15.52:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f64140246e0 con 0x7f6418108aa0 2024-05-27T07:14:03.588 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:13:56.830+0000 7f641fa17700 1 -- 172.21.15.52:0/2539359707 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f64181a02a0 con 0x7f6418108aa0 2024-05-27T07:14:03.588 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:13:56.831+0000 7f640e7fc700 1 -- 172.21.15.52:0/2539359707 <== mon.0 v2:172.21.15.52:3300/0 4 ==== mgrmap(e 5) ==== 50222+0+0 (secure 0 0 0) 0x7f6414022020 con 0x7f6418108aa0 2024-05-27T07:14:03.588 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:13:56.831+0000 7f640e7fc700 1 --2- 172.21.15.52:0/2539359707 >> [v2:172.21.15.52:6800/587662512,v1:172.21.15.52:6801/587662512] conn(0x7f640403e4c0 0x7f64040409d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:14:03.588 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:13:56.831+0000 7f640e7fc700 1 -- 172.21.15.52:0/2539359707 --> [v2:172.21.15.52:6800/587662512,v1:172.21.15.52:6801/587662512] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x7f64040410e0 con 0x7f640403e4c0 2024-05-27T07:14:03.588 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:13:56.831+0000 7f640e7fc700 1 -- 172.21.15.52:0/2539359707 <== mon.0 v2:172.21.15.52:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7f6414053420 con 0x7f6418108aa0 2024-05-27T07:14:03.588 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:13:56.831+0000 7f641cfb2700 1 -- 172.21.15.52:0/2539359707 >> [v2:172.21.15.52:6800/587662512,v1:172.21.15.52:6801/587662512] conn(0x7f640403e4c0 msgr2=0x7f64040409d0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.52:6800/587662512 2024-05-27T07:14:03.588 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:13:56.831+0000 7f641cfb2700 1 --2- 172.21.15.52:0/2539359707 >> [v2:172.21.15.52:6800/587662512,v1:172.21.15.52:6801/587662512] conn(0x7f640403e4c0 0x7f64040409d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-05-27T07:14:03.588 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:13:57.032+0000 7f641cfb2700 1 -- 172.21.15.52:0/2539359707 >> [v2:172.21.15.52:6800/587662512,v1:172.21.15.52:6801/587662512] conn(0x7f640403e4c0 msgr2=0x7f64040409d0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.52:6800/587662512 2024-05-27T07:14:03.588 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:13:57.032+0000 7f641cfb2700 1 --2- 172.21.15.52:0/2539359707 >> [v2:172.21.15.52:6800/587662512,v1:172.21.15.52:6801/587662512] conn(0x7f640403e4c0 0x7f64040409d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-05-27T07:14:03.588 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:13:57.433+0000 7f641cfb2700 1 -- 172.21.15.52:0/2539359707 >> [v2:172.21.15.52:6800/587662512,v1:172.21.15.52:6801/587662512] conn(0x7f640403e4c0 msgr2=0x7f64040409d0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.52:6800/587662512 2024-05-27T07:14:03.588 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:13:57.433+0000 7f641cfb2700 1 --2- 172.21.15.52:0/2539359707 >> [v2:172.21.15.52:6800/587662512,v1:172.21.15.52:6801/587662512] conn(0x7f640403e4c0 0x7f64040409d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-05-27T07:14:03.589 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:13:58.234+0000 7f641cfb2700 1 -- 172.21.15.52:0/2539359707 >> [v2:172.21.15.52:6800/587662512,v1:172.21.15.52:6801/587662512] conn(0x7f640403e4c0 msgr2=0x7f64040409d0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.52:6800/587662512 2024-05-27T07:14:03.589 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:13:58.234+0000 7f641cfb2700 1 --2- 172.21.15.52:0/2539359707 >> [v2:172.21.15.52:6800/587662512,v1:172.21.15.52:6801/587662512] conn(0x7f640403e4c0 0x7f64040409d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-05-27T07:14:03.589 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:13:59.835+0000 7f641cfb2700 1 -- 172.21.15.52:0/2539359707 >> [v2:172.21.15.52:6800/587662512,v1:172.21.15.52:6801/587662512] conn(0x7f640403e4c0 msgr2=0x7f64040409d0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.52:6800/587662512 2024-05-27T07:14:03.589 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:13:59.835+0000 7f641cfb2700 1 --2- 172.21.15.52:0/2539359707 >> [v2:172.21.15.52:6800/587662512,v1:172.21.15.52:6801/587662512] conn(0x7f640403e4c0 0x7f64040409d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 3.200000 2024-05-27T07:14:03.589 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:14:01.962+0000 7f640e7fc700 1 -- 172.21.15.52:0/2539359707 <== mon.0 v2:172.21.15.52:3300/0 6 ==== mgrmap(e 6) ==== 49989+0+0 (secure 0 0 0) 0x7f641402d3f0 con 0x7f6418108aa0 2024-05-27T07:14:03.589 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:14:01.962+0000 7f640e7fc700 1 -- 172.21.15.52:0/2539359707 >> [v2:172.21.15.52:6800/587662512,v1:172.21.15.52:6801/587662512] conn(0x7f640403e4c0 msgr2=0x7f64040409d0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-05-27T07:14:03.589 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:14:01.962+0000 7f640e7fc700 1 --2- 172.21.15.52:0/2539359707 >> [v2:172.21.15.52:6800/587662512,v1:172.21.15.52:6801/587662512] conn(0x7f640403e4c0 0x7f64040409d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:14:03.589 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:14:02.963+0000 7f640e7fc700 1 -- 172.21.15.52:0/2539359707 <== mon.0 v2:172.21.15.52:3300/0 7 ==== mgrmap(e 7) ==== 50116+0+0 (secure 0 0 0) 0x7f64140556e0 con 0x7f6418108aa0 2024-05-27T07:14:03.589 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:14:02.963+0000 7f640e7fc700 1 --2- 172.21.15.52:0/2539359707 >> [v2:172.21.15.52:6800/1631683163,v1:172.21.15.52:6801/1631683163] conn(0x7f6404041e00 0x7f6404044220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:14:03.589 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:14:02.963+0000 7f640e7fc700 1 -- 172.21.15.52:0/2539359707 --> [v2:172.21.15.52:6800/1631683163,v1:172.21.15.52:6801/1631683163] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x7f64040410e0 con 0x7f6404041e00 2024-05-27T07:14:03.589 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:14:02.964+0000 7f641cfb2700 1 --2- 172.21.15.52:0/2539359707 >> [v2:172.21.15.52:6800/1631683163,v1:172.21.15.52:6801/1631683163] conn(0x7f6404041e00 0x7f6404044220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:14:03.589 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:14:02.965+0000 7f641cfb2700 1 --2- 172.21.15.52:0/2539359707 >> [v2:172.21.15.52:6800/1631683163,v1:172.21.15.52:6801/1631683163] conn(0x7f6404041e00 0x7f6404044220 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 crypto rx=0x7f6408003a10 tx=0x7f64080092b0 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:14:03.590 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:14:02.965+0000 7f640e7fc700 1 -- 172.21.15.52:0/2539359707 <== mgr.14120 v2:172.21.15.52:6800/1631683163 1 ==== command_reply(tid 0: 0 ) ==== 8+0+9564 (secure 0 0 0) 0x7f64040410e0 con 0x7f6404041e00 2024-05-27T07:14:03.590 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:14:02.974+0000 7f641fa17700 1 -- 172.21.15.52:0/2539359707 --> [v2:172.21.15.52:6800/1631683163,v1:172.21.15.52:6801/1631683163] -- command(tid 1: {"prefix": "mgr_status"}) -- 0x7f6418064e10 con 0x7f6404041e00 2024-05-27T07:14:03.590 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:14:02.974+0000 7f640e7fc700 1 -- 172.21.15.52:0/2539359707 <== mgr.14120 v2:172.21.15.52:6800/1631683163 2 ==== command_reply(tid 1: 0 ) ==== 8+0+51 (secure 0 0 0) 0x7f6418064e10 con 0x7f6404041e00 2024-05-27T07:14:03.590 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:14:02.975+0000 7f641fa17700 1 -- 172.21.15.52:0/2539359707 >> [v2:172.21.15.52:6800/1631683163,v1:172.21.15.52:6801/1631683163] conn(0x7f6404041e00 msgr2=0x7f6404044220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:14:03.590 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:14:02.975+0000 7f641fa17700 1 --2- 172.21.15.52:0/2539359707 >> [v2:172.21.15.52:6800/1631683163,v1:172.21.15.52:6801/1631683163] conn(0x7f6404041e00 0x7f6404044220 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 crypto rx=0x7f6408003a10 tx=0x7f64080092b0 comp rx=0 tx=0).stop 2024-05-27T07:14:03.590 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:14:02.975+0000 7f641fa17700 1 -- 172.21.15.52:0/2539359707 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f6418108aa0 msgr2=0x7f641819cec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:14:03.590 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:14:02.975+0000 7f641fa17700 1 --2- 172.21.15.52:0/2539359707 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f6418108aa0 0x7f641819cec0 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f6414011890 tx=0x7f6414011970 comp rx=0 tx=0).stop 2024-05-27T07:14:03.590 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:14:02.975+0000 7f641fa17700 1 -- 172.21.15.52:0/2539359707 shutdown_connections 2024-05-27T07:14:03.590 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:14:02.975+0000 7f641fa17700 1 --2- 172.21.15.52:0/2539359707 >> [v2:172.21.15.52:6800/1631683163,v1:172.21.15.52:6801/1631683163] conn(0x7f6404041e00 0x7f6404044220 unknown :-1 s=CLOSED pgs=2 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:14:03.590 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:14:02.975+0000 7f641fa17700 1 --2- 172.21.15.52:0/2539359707 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f6418108aa0 0x7f641819cec0 unknown :-1 s=CLOSED pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:14:03.590 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:14:02.975+0000 7f641fa17700 1 --2- 172.21.15.52:0/2539359707 >> [v2:172.21.15.52:6800/587662512,v1:172.21.15.52:6801/587662512] conn(0x7f640403e4c0 0x7f64040409d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:14:03.590 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:14:02.975+0000 7f641fa17700 1 -- 172.21.15.52:0/2539359707 >> 172.21.15.52:0/2539359707 conn(0x7f64181040e0 msgr2=0x7f6418104a00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:14:03.591 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:14:02.975+0000 7f641fa17700 1 -- 172.21.15.52:0/2539359707 shutdown_connections 2024-05-27T07:14:03.591 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:14:02.975+0000 7f641fa17700 1 -- 172.21.15.52:0/2539359707 wait complete. 2024-05-27T07:14:03.591 INFO:teuthology.orchestra.run.smithi052.stdout:mgr epoch 5 is available 2024-05-27T07:14:03.591 INFO:teuthology.orchestra.run.smithi052.stdout:Setting orchestrator backend to cephadm... 2024-05-27T07:14:04.230 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:14:04 smithi052 ceph-mon[31955]: from='client.14124 -' entity='client.admin' cmd=[{"prefix": "get_command_descriptions"}]: dispatch 2024-05-27T07:14:04.230 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:14:04 smithi052 ceph-mon[31955]: from='client.14124 -' entity='client.admin' cmd=[{"prefix": "mgr_status"}]: dispatch 2024-05-27T07:14:04.230 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:14:04 smithi052 ceph-mon[31955]: [27/May/2024:07:14:03] ENGINE Bus STARTING 2024-05-27T07:14:04.230 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:14:04 smithi052 ceph-mon[31955]: [27/May/2024:07:14:03] ENGINE Serving on http://172.21.15.52:8765 2024-05-27T07:14:04.230 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:14:04 smithi052 ceph-mon[31955]: from='mgr.14120 172.21.15.52:0/2490234989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-27T07:14:04.967 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:14:04.005+0000 7fa5b78a6700 1 Processor -- start 2024-05-27T07:14:04.968 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:14:04.005+0000 7fa5b78a6700 1 -- start start 2024-05-27T07:14:04.968 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:14:04.005+0000 7fa5b78a6700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fa5b0109bf0 0x7fa5b010a000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:14:04.968 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:14:04.006+0000 7fa5b78a6700 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x7fa5b010a5f0 con 0x7fa5b0109bf0 2024-05-27T07:14:04.968 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:14:04.006+0000 7fa5b5642700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fa5b0109bf0 0x7fa5b010a000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:14:04.968 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:14:04.006+0000 7fa5b5642700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fa5b0109bf0 0x7fa5b010a000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.52:42494/0 (socket says 172.21.15.52:42494) 2024-05-27T07:14:04.968 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:14:04.006+0000 7fa5b5642700 1 -- 172.21.15.52:0/1048123822 learned_addr learned my addr 172.21.15.52:0/1048123822 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-05-27T07:14:04.968 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:14:04.006+0000 7fa5b5642700 1 -- 172.21.15.52:0/1048123822 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa5b010ad20 con 0x7fa5b0109bf0 2024-05-27T07:14:04.968 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:14:04.006+0000 7fa5b5642700 1 --2- 172.21.15.52:0/1048123822 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fa5b0109bf0 0x7fa5b010a000 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7fa5a0009e80 tx=0x7fa5a000b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5322b020c829286a server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:14:04.968 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:14:04.006+0000 7fa5affff700 1 -- 172.21.15.52:0/1048123822 <== mon.0 v2:172.21.15.52:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fa5a0017070 con 0x7fa5b0109bf0 2024-05-27T07:14:04.968 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:14:04.006+0000 7fa5affff700 1 -- 172.21.15.52:0/1048123822 <== mon.0 v2:172.21.15.52:3300/0 2 ==== config(26 keys) ==== 1070+0+0 (secure 0 0 0) 0x7fa5a000b8c0 con 0x7fa5b0109bf0 2024-05-27T07:14:04.969 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:14:04.006+0000 7fa5affff700 1 -- 172.21.15.52:0/1048123822 <== mon.0 v2:172.21.15.52:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fa5a0004490 con 0x7fa5b0109bf0 2024-05-27T07:14:04.969 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:14:04.007+0000 7fa5b78a6700 1 -- 172.21.15.52:0/1048123822 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fa5b0109bf0 msgr2=0x7fa5b010a000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:14:04.969 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:14:04.007+0000 7fa5b78a6700 1 --2- 172.21.15.52:0/1048123822 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fa5b0109bf0 0x7fa5b010a000 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7fa5a0009e80 tx=0x7fa5a000b330 comp rx=0 tx=0).stop 2024-05-27T07:14:04.969 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:14:04.007+0000 7fa5b78a6700 1 -- 172.21.15.52:0/1048123822 shutdown_connections 2024-05-27T07:14:04.969 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:14:04.007+0000 7fa5b78a6700 1 --2- 172.21.15.52:0/1048123822 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fa5b0109bf0 0x7fa5b010a000 unknown :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:14:04.969 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:14:04.007+0000 7fa5b78a6700 1 -- 172.21.15.52:0/1048123822 >> 172.21.15.52:0/1048123822 conn(0x7fa5b0105210 msgr2=0x7fa5b01076e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:14:04.969 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:14:04.007+0000 7fa5b78a6700 1 -- 172.21.15.52:0/1048123822 shutdown_connections 2024-05-27T07:14:04.969 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:14:04.007+0000 7fa5b78a6700 1 -- 172.21.15.52:0/1048123822 wait complete. 2024-05-27T07:14:04.970 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:14:04.008+0000 7fa5b78a6700 1 Processor -- start 2024-05-27T07:14:04.970 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:14:04.008+0000 7fa5b78a6700 1 -- start start 2024-05-27T07:14:04.970 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:14:04.008+0000 7fa5b78a6700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fa5b0109bf0 0x7fa5b019df60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:14:04.970 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:14:04.008+0000 7fa5b78a6700 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x7fa5b010e0d0 con 0x7fa5b0109bf0 2024-05-27T07:14:04.970 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:14:04.008+0000 7fa5b5642700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fa5b0109bf0 0x7fa5b019df60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:14:04.970 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:14:04.008+0000 7fa5b5642700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fa5b0109bf0 0x7fa5b019df60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.52:42502/0 (socket says 172.21.15.52:42502) 2024-05-27T07:14:04.970 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:14:04.008+0000 7fa5b5642700 1 -- 172.21.15.52:0/3530731868 learned_addr learned my addr 172.21.15.52:0/3530731868 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-05-27T07:14:04.970 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:14:04.009+0000 7fa5b5642700 1 -- 172.21.15.52:0/3530731868 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa5b019e4a0 con 0x7fa5b0109bf0 2024-05-27T07:14:04.971 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:14:04.009+0000 7fa5b5642700 1 --2- 172.21.15.52:0/3530731868 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fa5b0109bf0 0x7fa5b019df60 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7fa5a0009fd0 tx=0x7fa5a000bd50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:14:04.971 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:14:04.009+0000 7fa5ae7fc700 1 -- 172.21.15.52:0/3530731868 <== mon.0 v2:172.21.15.52:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fa5a0020020 con 0x7fa5b0109bf0 2024-05-27T07:14:04.971 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:14:04.009+0000 7fa5b78a6700 1 -- 172.21.15.52:0/3530731868 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa5b019e730 con 0x7fa5b0109bf0 2024-05-27T07:14:04.971 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:14:04.009+0000 7fa5ae7fc700 1 -- 172.21.15.52:0/3530731868 <== mon.0 v2:172.21.15.52:3300/0 2 ==== config(26 keys) ==== 1070+0+0 (secure 0 0 0) 0x7fa5a0012430 con 0x7fa5b0109bf0 2024-05-27T07:14:04.971 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:14:04.009+0000 7fa5ae7fc700 1 -- 172.21.15.52:0/3530731868 <== mon.0 v2:172.21.15.52:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fa5a0017040 con 0x7fa5b0109bf0 2024-05-27T07:14:04.971 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:14:04.009+0000 7fa5b78a6700 1 -- 172.21.15.52:0/3530731868 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa5b01a1340 con 0x7fa5b0109bf0 2024-05-27T07:14:04.971 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:14:04.010+0000 7fa5b78a6700 1 -- 172.21.15.52:0/3530731868 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa5b0064e10 con 0x7fa5b0109bf0 2024-05-27T07:14:04.971 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:14:04.010+0000 7fa5ae7fc700 1 -- 172.21.15.52:0/3530731868 <== mon.0 v2:172.21.15.52:3300/0 4 ==== mgrmap(e 7) ==== 50116+0+0 (secure 0 0 0) 0x7fa5a0023020 con 0x7fa5b0109bf0 2024-05-27T07:14:04.971 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:14:04.010+0000 7fa5ae7fc700 1 --2- 172.21.15.52:0/3530731868 >> [v2:172.21.15.52:6800/1631683163,v1:172.21.15.52:6801/1631683163] conn(0x7fa59803e310 0x7fa598040820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:14:04.971 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:14:04.010+0000 7fa5ae7fc700 1 -- 172.21.15.52:0/3530731868 <== mon.0 v2:172.21.15.52:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 940+0+0 (secure 0 0 0) 0x7fa5a000f050 con 0x7fa5b0109bf0 2024-05-27T07:14:04.971 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:14:04.011+0000 7fa5b4e41700 1 --2- 172.21.15.52:0/3530731868 >> [v2:172.21.15.52:6800/1631683163,v1:172.21.15.52:6801/1631683163] conn(0x7fa59803e310 0x7fa598040820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:14:04.971 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:14:04.011+0000 7fa5b4e41700 1 --2- 172.21.15.52:0/3530731868 >> [v2:172.21.15.52:6800/1631683163,v1:172.21.15.52:6801/1631683163] conn(0x7fa59803e310 0x7fa598040820 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7fa5a4006fd0 tx=0x7fa5a4006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:14:04.971 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:14:04.016+0000 7fa5ae7fc700 1 -- 172.21.15.52:0/3530731868 <== mon.0 v2:172.21.15.52:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191963 (secure 0 0 0) 0x7fa5a002a020 con 0x7fa5b0109bf0 2024-05-27T07:14:04.972 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:14:04.021+0000 7fa5ae7fc700 1 -- 172.21.15.52:0/3530731868 <== mon.0 v2:172.21.15.52:3300/0 7 ==== mgrmap(e 8) ==== 50222+0+0 (secure 0 0 0) 0x7fa5a0024ca0 con 0x7fa5b0109bf0 2024-05-27T07:14:04.972 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:14:04.247+0000 7fa5b78a6700 1 -- 172.21.15.52:0/3530731868 --> [v2:172.21.15.52:6800/1631683163,v1:172.21.15.52:6801/1631683163] -- mgr_command(tid 0: {"prefix": "orch set backend", "module_name": "cephadm", "target": ["mon-mgr", ""]}) -- 0x7fa5b01061a0 con 0x7fa59803e310 2024-05-27T07:14:04.972 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:14:04.253+0000 7fa5ae7fc700 1 -- 172.21.15.52:0/3530731868 <== mgr.14120 v2:172.21.15.52:6800/1631683163 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+0 (secure 0 0 0) 0x7fa5b01061a0 con 0x7fa59803e310 2024-05-27T07:14:04.972 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:14:04.255+0000 7fa5b78a6700 1 -- 172.21.15.52:0/3530731868 >> [v2:172.21.15.52:6800/1631683163,v1:172.21.15.52:6801/1631683163] conn(0x7fa59803e310 msgr2=0x7fa598040820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:14:04.972 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:14:04.255+0000 7fa5b78a6700 1 --2- 172.21.15.52:0/3530731868 >> [v2:172.21.15.52:6800/1631683163,v1:172.21.15.52:6801/1631683163] conn(0x7fa59803e310 0x7fa598040820 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7fa5a4006fd0 tx=0x7fa5a4006e40 comp rx=0 tx=0).stop 2024-05-27T07:14:04.972 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:14:04.255+0000 7fa5b78a6700 1 -- 172.21.15.52:0/3530731868 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fa5b0109bf0 msgr2=0x7fa5b019df60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:14:04.972 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:14:04.255+0000 7fa5b78a6700 1 --2- 172.21.15.52:0/3530731868 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fa5b0109bf0 0x7fa5b019df60 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7fa5a0009fd0 tx=0x7fa5a000bd50 comp rx=0 tx=0).stop 2024-05-27T07:14:04.972 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:14:04.256+0000 7fa5b78a6700 1 -- 172.21.15.52:0/3530731868 shutdown_connections 2024-05-27T07:14:04.972 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:14:04.256+0000 7fa5b78a6700 1 --2- 172.21.15.52:0/3530731868 >> [v2:172.21.15.52:6800/1631683163,v1:172.21.15.52:6801/1631683163] conn(0x7fa59803e310 0x7fa598040820 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:14:04.972 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:14:04.256+0000 7fa5b78a6700 1 --2- 172.21.15.52:0/3530731868 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fa5b0109bf0 0x7fa5b019df60 unknown :-1 s=CLOSED pgs=38 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:14:04.972 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:14:04.256+0000 7fa5b78a6700 1 -- 172.21.15.52:0/3530731868 >> 172.21.15.52:0/3530731868 conn(0x7fa5b0105210 msgr2=0x7fa5b0105a90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:14:04.972 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:14:04.256+0000 7fa5b78a6700 1 -- 172.21.15.52:0/3530731868 shutdown_connections 2024-05-27T07:14:04.973 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:14:04.256+0000 7fa5b78a6700 1 -- 172.21.15.52:0/3530731868 wait complete. 2024-05-27T07:14:05.230 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:14:05 smithi052 ceph-mon[31955]: [27/May/2024:07:14:03] ENGINE Serving on https://172.21.15.52:7150 2024-05-27T07:14:05.230 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:14:05 smithi052 ceph-mon[31955]: [27/May/2024:07:14:03] ENGINE Bus STARTED 2024-05-27T07:14:05.230 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:14:05 smithi052 ceph-mon[31955]: mgrmap e8: x(active, since 2s) 2024-05-27T07:14:05.230 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:14:05 smithi052 ceph-mon[31955]: from='client.14132 -' entity='client.admin' cmd=[{"prefix": "orch set backend", "module_name": "cephadm", "target": ["mon-mgr", ""]}]: dispatch 2024-05-27T07:14:05.230 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:14:05 smithi052 ceph-mon[31955]: from='mgr.14120 172.21.15.52:0/2490234989' entity='mgr.x' 2024-05-27T07:14:05.230 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:14:05 smithi052 ceph-mon[31955]: from='mgr.14120 172.21.15.52:0/2490234989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-27T07:14:06.164 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout value unchanged 2024-05-27T07:14:06.164 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:14:05.386+0000 7fb7483fd700 1 Processor -- start 2024-05-27T07:14:06.164 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:14:05.386+0000 7fb7483fd700 1 -- start start 2024-05-27T07:14:06.164 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:14:05.386+0000 7fb7483fd700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fb74007d6c0 0x7fb74010c760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:14:06.164 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:14:05.386+0000 7fb7483fd700 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x7fb74010cd30 con 0x7fb74007d6c0 2024-05-27T07:14:06.164 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:14:05.386+0000 7fb746199700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fb74007d6c0 0x7fb74010c760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:14:06.164 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:14:05.387+0000 7fb746199700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fb74007d6c0 0x7fb74010c760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.52:42504/0 (socket says 172.21.15.52:42504) 2024-05-27T07:14:06.164 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:14:05.387+0000 7fb746199700 1 -- 172.21.15.52:0/933818250 learned_addr learned my addr 172.21.15.52:0/933818250 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-05-27T07:14:06.164 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:14:05.387+0000 7fb746199700 1 -- 172.21.15.52:0/933818250 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb74010ceb0 con 0x7fb74007d6c0 2024-05-27T07:14:06.165 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:14:05.387+0000 7fb746199700 1 --2- 172.21.15.52:0/933818250 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fb74007d6c0 0x7fb74010c760 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7fb73c0098a0 tx=0x7fb73c009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=72a4872b54bd199 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:14:06.165 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:14:05.387+0000 7fb745197700 1 -- 172.21.15.52:0/933818250 <== mon.0 v2:172.21.15.52:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fb73c004160 con 0x7fb74007d6c0 2024-05-27T07:14:06.165 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:14:05.387+0000 7fb745197700 1 -- 172.21.15.52:0/933818250 <== mon.0 v2:172.21.15.52:3300/0 2 ==== config(26 keys) ==== 1070+0+0 (secure 0 0 0) 0x7fb73c004300 con 0x7fb74007d6c0 2024-05-27T07:14:06.165 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:14:05.387+0000 7fb745197700 1 -- 172.21.15.52:0/933818250 <== mon.0 v2:172.21.15.52:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fb73c011af0 con 0x7fb74007d6c0 2024-05-27T07:14:06.165 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:14:05.388+0000 7fb7483fd700 1 -- 172.21.15.52:0/933818250 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fb74007d6c0 msgr2=0x7fb74010c760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:14:06.165 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:14:05.388+0000 7fb7483fd700 1 --2- 172.21.15.52:0/933818250 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fb74007d6c0 0x7fb74010c760 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7fb73c0098a0 tx=0x7fb73c009bb0 comp rx=0 tx=0).stop 2024-05-27T07:14:06.165 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:14:05.388+0000 7fb7483fd700 1 -- 172.21.15.52:0/933818250 shutdown_connections 2024-05-27T07:14:06.165 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:14:05.388+0000 7fb7483fd700 1 --2- 172.21.15.52:0/933818250 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fb74007d6c0 0x7fb74010c760 unknown :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:14:06.165 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:14:05.388+0000 7fb7483fd700 1 -- 172.21.15.52:0/933818250 >> 172.21.15.52:0/933818250 conn(0x7fb740104150 msgr2=0x7fb7401065e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:14:06.165 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:14:05.388+0000 7fb7483fd700 1 -- 172.21.15.52:0/933818250 shutdown_connections 2024-05-27T07:14:06.165 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:14:05.388+0000 7fb7483fd700 1 -- 172.21.15.52:0/933818250 wait complete. 2024-05-27T07:14:06.165 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:14:05.389+0000 7fb7483fd700 1 Processor -- start 2024-05-27T07:14:06.166 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:14:05.389+0000 7fb7483fd700 1 -- start start 2024-05-27T07:14:06.166 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:14:05.389+0000 7fb7483fd700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fb74007d6c0 0x7fb74019df30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:14:06.166 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:14:05.389+0000 7fb7483fd700 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x7fb74010df30 con 0x7fb74007d6c0 2024-05-27T07:14:06.166 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:14:05.389+0000 7fb746199700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fb74007d6c0 0x7fb74019df30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:14:06.166 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:14:05.389+0000 7fb746199700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fb74007d6c0 0x7fb74019df30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.52:42520/0 (socket says 172.21.15.52:42520) 2024-05-27T07:14:06.166 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:14:05.389+0000 7fb746199700 1 -- 172.21.15.52:0/277498176 learned_addr learned my addr 172.21.15.52:0/277498176 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-05-27T07:14:06.166 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:14:05.390+0000 7fb746199700 1 -- 172.21.15.52:0/277498176 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb74019e470 con 0x7fb74007d6c0 2024-05-27T07:14:06.166 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:14:05.390+0000 7fb746199700 1 --2- 172.21.15.52:0/277498176 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fb74007d6c0 0x7fb74019df30 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7fb73c009fb0 tx=0x7fb73c011940 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:14:06.166 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:14:05.390+0000 7fb7377fe700 1 -- 172.21.15.52:0/277498176 <== mon.0 v2:172.21.15.52:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fb73c0036a0 con 0x7fb74007d6c0 2024-05-27T07:14:06.166 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:14:05.390+0000 7fb7377fe700 1 -- 172.21.15.52:0/277498176 <== mon.0 v2:172.21.15.52:3300/0 2 ==== config(26 keys) ==== 1070+0+0 (secure 0 0 0) 0x7fb73c01a430 con 0x7fb74007d6c0 2024-05-27T07:14:06.166 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:14:05.390+0000 7fb7483fd700 1 -- 172.21.15.52:0/277498176 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb74019e700 con 0x7fb74007d6c0 2024-05-27T07:14:06.166 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:14:05.390+0000 7fb7377fe700 1 -- 172.21.15.52:0/277498176 <== mon.0 v2:172.21.15.52:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fb73c019550 con 0x7fb74007d6c0 2024-05-27T07:14:06.167 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:14:05.390+0000 7fb7483fd700 1 -- 172.21.15.52:0/277498176 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb7401a1310 con 0x7fb74007d6c0 2024-05-27T07:14:06.167 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:14:05.391+0000 7fb7377fe700 1 -- 172.21.15.52:0/277498176 <== mon.0 v2:172.21.15.52:3300/0 4 ==== mgrmap(e 8) ==== 50222+0+0 (secure 0 0 0) 0x7fb73c019850 con 0x7fb74007d6c0 2024-05-27T07:14:06.167 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:14:05.391+0000 7fb7483fd700 1 -- 172.21.15.52:0/277498176 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb740064e10 con 0x7fb74007d6c0 2024-05-27T07:14:06.167 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:14:05.391+0000 7fb7377fe700 1 --2- 172.21.15.52:0/277498176 >> [v2:172.21.15.52:6800/1631683163,v1:172.21.15.52:6801/1631683163] conn(0x7fb72c03e420 0x7fb72c040930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:14:06.167 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:14:05.391+0000 7fb7377fe700 1 -- 172.21.15.52:0/277498176 <== mon.0 v2:172.21.15.52:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 940+0+0 (secure 0 0 0) 0x7fb73c053880 con 0x7fb74007d6c0 2024-05-27T07:14:06.167 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:14:05.392+0000 7fb745998700 1 --2- 172.21.15.52:0/277498176 >> [v2:172.21.15.52:6800/1631683163,v1:172.21.15.52:6801/1631683163] conn(0x7fb72c03e420 0x7fb72c040930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:14:06.167 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:14:05.392+0000 7fb745998700 1 --2- 172.21.15.52:0/277498176 >> [v2:172.21.15.52:6800/1631683163,v1:172.21.15.52:6801/1631683163] conn(0x7fb72c03e420 0x7fb72c040930 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7fb730006fd0 tx=0x7fb730006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:14:06.167 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:14:05.397+0000 7fb7377fe700 1 -- 172.21.15.52:0/277498176 <== mon.0 v2:172.21.15.52:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191963 (secure 0 0 0) 0x7fb73c01a5e0 con 0x7fb74007d6c0 2024-05-27T07:14:06.167 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:14:05.613+0000 7fb7483fd700 1 -- 172.21.15.52:0/277498176 --> [v2:172.21.15.52:6800/1631683163,v1:172.21.15.52:6801/1631683163] -- mgr_command(tid 0: {"prefix": "cephadm set-user", "user": "root", "target": ["mon-mgr", ""]}) -- 0x7fb7401051e0 con 0x7fb72c03e420 2024-05-27T07:14:06.167 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:14:05.614+0000 7fb7377fe700 1 -- 172.21.15.52:0/277498176 <== mgr.14120 v2:172.21.15.52:6800/1631683163 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+16 (secure 0 0 0) 0x7fb7401051e0 con 0x7fb72c03e420 2024-05-27T07:14:06.167 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:14:05.617+0000 7fb7483fd700 1 -- 172.21.15.52:0/277498176 >> [v2:172.21.15.52:6800/1631683163,v1:172.21.15.52:6801/1631683163] conn(0x7fb72c03e420 msgr2=0x7fb72c040930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:14:06.167 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:14:05.617+0000 7fb7483fd700 1 --2- 172.21.15.52:0/277498176 >> [v2:172.21.15.52:6800/1631683163,v1:172.21.15.52:6801/1631683163] conn(0x7fb72c03e420 0x7fb72c040930 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7fb730006fd0 tx=0x7fb730006e40 comp rx=0 tx=0).stop 2024-05-27T07:14:06.167 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:14:05.617+0000 7fb7483fd700 1 -- 172.21.15.52:0/277498176 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fb74007d6c0 msgr2=0x7fb74019df30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:14:06.167 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:14:05.617+0000 7fb7483fd700 1 --2- 172.21.15.52:0/277498176 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fb74007d6c0 0x7fb74019df30 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7fb73c009fb0 tx=0x7fb73c011940 comp rx=0 tx=0).stop 2024-05-27T07:14:06.168 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:14:05.617+0000 7fb7483fd700 1 -- 172.21.15.52:0/277498176 shutdown_connections 2024-05-27T07:14:06.168 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:14:05.617+0000 7fb7483fd700 1 --2- 172.21.15.52:0/277498176 >> [v2:172.21.15.52:6800/1631683163,v1:172.21.15.52:6801/1631683163] conn(0x7fb72c03e420 0x7fb72c040930 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:14:06.168 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:14:05.617+0000 7fb7483fd700 1 --2- 172.21.15.52:0/277498176 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fb74007d6c0 0x7fb74019df30 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:14:06.168 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:14:05.617+0000 7fb7483fd700 1 -- 172.21.15.52:0/277498176 >> 172.21.15.52:0/277498176 conn(0x7fb740104150 msgr2=0x7fb740104ab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:14:06.168 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:14:05.617+0000 7fb7483fd700 1 -- 172.21.15.52:0/277498176 shutdown_connections 2024-05-27T07:14:06.168 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:14:05.617+0000 7fb7483fd700 1 -- 172.21.15.52:0/277498176 wait complete. 2024-05-27T07:14:06.168 INFO:teuthology.orchestra.run.smithi052.stdout:Generating ssh key... 2024-05-27T07:14:06.980 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:14:06 smithi052 ceph-mon[31955]: from='client.14134 -' entity='client.admin' cmd=[{"prefix": "cephadm set-user", "user": "root", "target": ["mon-mgr", ""]}]: dispatch 2024-05-27T07:14:07.329 INFO:journalctl@ceph.mgr.x.smithi052.stdout:May 27 07:14:07 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mgr-x[32184]: Generating public/private rsa key pair. 2024-05-27T07:14:07.329 INFO:journalctl@ceph.mgr.x.smithi052.stdout:May 27 07:14:07 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mgr-x[32184]: Your identification has been saved in /tmp/tmp7f0ouu57/key. 2024-05-27T07:14:07.329 INFO:journalctl@ceph.mgr.x.smithi052.stdout:May 27 07:14:07 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mgr-x[32184]: Your public key has been saved in /tmp/tmp7f0ouu57/key.pub. 2024-05-27T07:14:07.330 INFO:journalctl@ceph.mgr.x.smithi052.stdout:May 27 07:14:07 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mgr-x[32184]: The key fingerprint is: 2024-05-27T07:14:07.330 INFO:journalctl@ceph.mgr.x.smithi052.stdout:May 27 07:14:07 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mgr-x[32184]: SHA256:E3BrGl2Hen2SM7kEGgY8eBybqpi60N46ZGqPfw5AsDw ceph-73df2636-1bf8-11ef-bc9b-c7b262605968 2024-05-27T07:14:07.330 INFO:journalctl@ceph.mgr.x.smithi052.stdout:May 27 07:14:07 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mgr-x[32184]: The key's randomart image is: 2024-05-27T07:14:07.330 INFO:journalctl@ceph.mgr.x.smithi052.stdout:May 27 07:14:07 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mgr-x[32184]: +---[RSA 3072]----+ 2024-05-27T07:14:07.330 INFO:journalctl@ceph.mgr.x.smithi052.stdout:May 27 07:14:07 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mgr-x[32184]: |. ++o. ... | 2024-05-27T07:14:07.330 INFO:journalctl@ceph.mgr.x.smithi052.stdout:May 27 07:14:07 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mgr-x[32184]: |o. . =*ooo. | 2024-05-27T07:14:07.330 INFO:journalctl@ceph.mgr.x.smithi052.stdout:May 27 07:14:07 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mgr-x[32184]: |.E .+o=+ o o | 2024-05-27T07:14:07.330 INFO:journalctl@ceph.mgr.x.smithi052.stdout:May 27 07:14:07 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mgr-x[32184]: |. . . +o.. O . | 2024-05-27T07:14:07.330 INFO:journalctl@ceph.mgr.x.smithi052.stdout:May 27 07:14:07 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mgr-x[32184]: | . . . S. . * | 2024-05-27T07:14:07.330 INFO:journalctl@ceph.mgr.x.smithi052.stdout:May 27 07:14:07 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mgr-x[32184]: | ++. . . | 2024-05-27T07:14:07.330 INFO:journalctl@ceph.mgr.x.smithi052.stdout:May 27 07:14:07 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mgr-x[32184]: |++o. | 2024-05-27T07:14:07.330 INFO:journalctl@ceph.mgr.x.smithi052.stdout:May 27 07:14:07 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mgr-x[32184]: |+oo... | 2024-05-27T07:14:07.331 INFO:journalctl@ceph.mgr.x.smithi052.stdout:May 27 07:14:07 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mgr-x[32184]: |+o==+. | 2024-05-27T07:14:07.331 INFO:journalctl@ceph.mgr.x.smithi052.stdout:May 27 07:14:07 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mgr-x[32184]: +----[SHA256]-----+ 2024-05-27T07:14:07.615 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:14:06.592+0000 7f3fcff77700 1 Processor -- start 2024-05-27T07:14:07.621 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:14:06.592+0000 7f3fcff77700 1 -- start start 2024-05-27T07:14:07.621 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:14:06.593+0000 7f3fcff77700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f3fc810b100 0x7f3fc810b510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:14:07.621 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:14:06.593+0000 7f3fcff77700 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x7f3fc810bae0 con 0x7f3fc810b100 2024-05-27T07:14:07.621 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:14:06.593+0000 7f3fcdd13700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f3fc810b100 0x7f3fc810b510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:14:07.621 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:14:06.593+0000 7f3fcdd13700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f3fc810b100 0x7f3fc810b510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.52:42536/0 (socket says 172.21.15.52:42536) 2024-05-27T07:14:07.622 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:14:06.593+0000 7f3fcdd13700 1 -- 172.21.15.52:0/2030640962 learned_addr learned my addr 172.21.15.52:0/2030640962 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-05-27T07:14:07.622 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:14:06.593+0000 7f3fcdd13700 1 -- 172.21.15.52:0/2030640962 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3fc810bc60 con 0x7f3fc810b100 2024-05-27T07:14:07.622 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:14:06.593+0000 7f3fcdd13700 1 --2- 172.21.15.52:0/2030640962 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f3fc810b100 0x7f3fc810b510 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f3fc40098a0 tx=0x7f3fc4009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=32ced02b844f267d server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:14:07.622 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:14:06.594+0000 7f3fccd11700 1 -- 172.21.15.52:0/2030640962 <== mon.0 v2:172.21.15.52:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f3fc4004160 con 0x7f3fc810b100 2024-05-27T07:14:07.622 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:14:06.594+0000 7f3fccd11700 1 -- 172.21.15.52:0/2030640962 <== mon.0 v2:172.21.15.52:3300/0 2 ==== config(26 keys) ==== 1070+0+0 (secure 0 0 0) 0x7f3fc4004300 con 0x7f3fc810b100 2024-05-27T07:14:07.622 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:14:06.594+0000 7f3fccd11700 1 -- 172.21.15.52:0/2030640962 <== mon.0 v2:172.21.15.52:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f3fc4011af0 con 0x7f3fc810b100 2024-05-27T07:14:07.622 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:14:06.594+0000 7f3fcff77700 1 -- 172.21.15.52:0/2030640962 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f3fc810b100 msgr2=0x7f3fc810b510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:14:07.622 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:14:06.594+0000 7f3fcff77700 1 --2- 172.21.15.52:0/2030640962 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f3fc810b100 0x7f3fc810b510 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f3fc40098a0 tx=0x7f3fc4009bb0 comp rx=0 tx=0).stop 2024-05-27T07:14:07.622 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:14:06.595+0000 7f3fcff77700 1 -- 172.21.15.52:0/2030640962 shutdown_connections 2024-05-27T07:14:07.622 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:14:06.595+0000 7f3fcff77700 1 --2- 172.21.15.52:0/2030640962 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f3fc810b100 0x7f3fc810b510 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:14:07.622 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:14:06.595+0000 7f3fcff77700 1 -- 172.21.15.52:0/2030640962 >> 172.21.15.52:0/2030640962 conn(0x7f3fc8103860 msgr2=0x7f3fc8105cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:14:07.622 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:14:06.595+0000 7f3fcff77700 1 -- 172.21.15.52:0/2030640962 shutdown_connections 2024-05-27T07:14:07.623 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:14:06.595+0000 7f3fcff77700 1 -- 172.21.15.52:0/2030640962 wait complete. 2024-05-27T07:14:07.623 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:14:06.595+0000 7f3fcff77700 1 Processor -- start 2024-05-27T07:14:07.623 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:14:06.595+0000 7f3fcff77700 1 -- start start 2024-05-27T07:14:07.623 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:14:06.596+0000 7f3fcff77700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f3fc810b100 0x7f3fc810a5f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:14:07.623 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:14:06.596+0000 7f3fcff77700 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x7f3fc81089b0 con 0x7f3fc810b100 2024-05-27T07:14:07.623 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:14:06.596+0000 7f3fcdd13700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f3fc810b100 0x7f3fc810a5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:14:07.623 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:14:06.596+0000 7f3fcdd13700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f3fc810b100 0x7f3fc810a5f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.52:42538/0 (socket says 172.21.15.52:42538) 2024-05-27T07:14:07.623 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:14:06.596+0000 7f3fcdd13700 1 -- 172.21.15.52:0/43727912 learned_addr learned my addr 172.21.15.52:0/43727912 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-05-27T07:14:07.623 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:14:06.596+0000 7f3fcdd13700 1 -- 172.21.15.52:0/43727912 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3fc810ab30 con 0x7f3fc810b100 2024-05-27T07:14:07.623 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:14:06.596+0000 7f3fcdd13700 1 --2- 172.21.15.52:0/43727912 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f3fc810b100 0x7f3fc810a5f0 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f3fc40117b0 tx=0x7f3fc4011b70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:14:07.623 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:14:06.596+0000 7f3fbeffd700 1 -- 172.21.15.52:0/43727912 <== mon.0 v2:172.21.15.52:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f3fc4003af0 con 0x7f3fc810b100 2024-05-27T07:14:07.623 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:14:06.596+0000 7f3fbeffd700 1 -- 172.21.15.52:0/43727912 <== mon.0 v2:172.21.15.52:3300/0 2 ==== config(26 keys) ==== 1070+0+0 (secure 0 0 0) 0x7f3fc4020070 con 0x7f3fc810b100 2024-05-27T07:14:07.623 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:14:06.596+0000 7f3fcff77700 1 -- 172.21.15.52:0/43727912 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3fc8108d80 con 0x7f3fc810b100 2024-05-27T07:14:07.624 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:14:06.597+0000 7f3fbeffd700 1 -- 172.21.15.52:0/43727912 <== mon.0 v2:172.21.15.52:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f3fc4019550 con 0x7f3fc810b100 2024-05-27T07:14:07.624 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:14:06.597+0000 7f3fcff77700 1 -- 172.21.15.52:0/43727912 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3fc8109260 con 0x7f3fc810b100 2024-05-27T07:14:07.624 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:14:06.597+0000 7f3fcff77700 1 -- 172.21.15.52:0/43727912 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3fc8064e10 con 0x7f3fc810b100 2024-05-27T07:14:07.624 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:14:06.598+0000 7f3fbeffd700 1 -- 172.21.15.52:0/43727912 <== mon.0 v2:172.21.15.52:3300/0 4 ==== mgrmap(e 8) ==== 50222+0+0 (secure 0 0 0) 0x7f3fc40197c0 con 0x7f3fc810b100 2024-05-27T07:14:07.624 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:14:06.598+0000 7f3fbeffd700 1 --2- 172.21.15.52:0/43727912 >> [v2:172.21.15.52:6800/1631683163,v1:172.21.15.52:6801/1631683163] conn(0x7f3fb403e4c0 0x7f3fb40409d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:14:07.624 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:14:06.598+0000 7f3fbeffd700 1 -- 172.21.15.52:0/43727912 <== mon.0 v2:172.21.15.52:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 940+0+0 (secure 0 0 0) 0x7f3fc40539b0 con 0x7f3fc810b100 2024-05-27T07:14:07.624 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:14:06.598+0000 7f3fcd512700 1 --2- 172.21.15.52:0/43727912 >> [v2:172.21.15.52:6800/1631683163,v1:172.21.15.52:6801/1631683163] conn(0x7f3fb403e4c0 0x7f3fb40409d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:14:07.624 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:14:06.598+0000 7f3fcd512700 1 --2- 172.21.15.52:0/43727912 >> [v2:172.21.15.52:6800/1631683163,v1:172.21.15.52:6801/1631683163] conn(0x7f3fb403e4c0 0x7f3fb40409d0 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f3fb8006fd0 tx=0x7f3fb8006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:14:07.624 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:14:06.603+0000 7f3fbeffd700 1 -- 172.21.15.52:0/43727912 <== mon.0 v2:172.21.15.52:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191963 (secure 0 0 0) 0x7f3fc402ade0 con 0x7f3fc810b100 2024-05-27T07:14:07.624 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:14:06.819+0000 7f3fcff77700 1 -- 172.21.15.52:0/43727912 --> [v2:172.21.15.52:6800/1631683163,v1:172.21.15.52:6801/1631683163] -- mgr_command(tid 0: {"prefix": "cephadm generate-key", "target": ["mon-mgr", ""]}) -- 0x7f3fc8109a70 con 0x7f3fb403e4c0 2024-05-27T07:14:07.624 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:14:07.038+0000 7f3fbeffd700 1 -- 172.21.15.52:0/43727912 <== mgr.14120 v2:172.21.15.52:6800/1631683163 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+0 (secure 0 0 0) 0x7f3fc8109a70 con 0x7f3fb403e4c0 2024-05-27T07:14:07.624 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:14:07.041+0000 7f3fcff77700 1 -- 172.21.15.52:0/43727912 >> [v2:172.21.15.52:6800/1631683163,v1:172.21.15.52:6801/1631683163] conn(0x7f3fb403e4c0 msgr2=0x7f3fb40409d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:14:07.624 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:14:07.041+0000 7f3fcff77700 1 --2- 172.21.15.52:0/43727912 >> [v2:172.21.15.52:6800/1631683163,v1:172.21.15.52:6801/1631683163] conn(0x7f3fb403e4c0 0x7f3fb40409d0 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f3fb8006fd0 tx=0x7f3fb8006e40 comp rx=0 tx=0).stop 2024-05-27T07:14:07.625 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:14:07.041+0000 7f3fcff77700 1 -- 172.21.15.52:0/43727912 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f3fc810b100 msgr2=0x7f3fc810a5f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:14:07.625 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:14:07.041+0000 7f3fcff77700 1 --2- 172.21.15.52:0/43727912 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f3fc810b100 0x7f3fc810a5f0 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f3fc40117b0 tx=0x7f3fc4011b70 comp rx=0 tx=0).stop 2024-05-27T07:14:07.625 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:14:07.041+0000 7f3fcff77700 1 -- 172.21.15.52:0/43727912 shutdown_connections 2024-05-27T07:14:07.625 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:14:07.041+0000 7f3fcff77700 1 --2- 172.21.15.52:0/43727912 >> [v2:172.21.15.52:6800/1631683163,v1:172.21.15.52:6801/1631683163] conn(0x7f3fb403e4c0 0x7f3fb40409d0 unknown :-1 s=CLOSED pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:14:07.625 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:14:07.041+0000 7f3fcff77700 1 --2- 172.21.15.52:0/43727912 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f3fc810b100 0x7f3fc810a5f0 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:14:07.625 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:14:07.041+0000 7f3fcff77700 1 -- 172.21.15.52:0/43727912 >> 172.21.15.52:0/43727912 conn(0x7f3fc8103860 msgr2=0x7f3fc8104240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:14:07.625 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:14:07.041+0000 7f3fcff77700 1 -- 172.21.15.52:0/43727912 shutdown_connections 2024-05-27T07:14:07.625 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:14:07.041+0000 7f3fcff77700 1 -- 172.21.15.52:0/43727912 wait complete. 2024-05-27T07:14:08.230 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:14:08 smithi052 ceph-mon[31955]: from='client.14136 -' entity='client.admin' cmd=[{"prefix": "cephadm generate-key", "target": ["mon-mgr", ""]}]: dispatch 2024-05-27T07:14:08.230 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:14:08 smithi052 ceph-mon[31955]: Generating ssh key... 2024-05-27T07:14:08.230 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:14:08 smithi052 ceph-mon[31955]: from='mgr.14120 172.21.15.52:0/2490234989' entity='mgr.x' 2024-05-27T07:14:08.230 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:14:08 smithi052 ceph-mon[31955]: from='mgr.14120 172.21.15.52:0/2490234989' entity='mgr.x' 2024-05-27T07:14:08.835 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQC8tboI03nnOKokdr00wKB0FvrqUHx08zkT/k0G/NBfqsAHGFHhJ3WHSvjyWs4J/6wdrcn7I/JdgB9WHq9Up6JGMwlkkyeu9e5fgMTv5QrrXYyNATTqZCZUMlJuiyBbkYLg49LVJiL8K2asXGcfiGYi3kFBvJf4gvfhJ8vqeVDSUpjAezyqZr/1HlV0MHPufOyl5irX2pxE2l/j6tBzu6Zk1ZMg63PqeX77ul/Tp0HSWVCvt19VGKUEXy9o2pYecB5BNGZEhtxO+1MCMLDomNcrRsGlpgxyj1JH7lVk3lBY/Xv4rMX6i2RJYj3XLFjdKmFSWFjlskRBsVcxRBjgC7vsnVpeabmvktcHTw+1di5HnJws0eBFY1OFWYUxQ2ICjOUP6SfpxAhbjKXFIMecFNGEvC9EdGSNBXIePJRYslarfbPuuxln18VF/rwNskWXxhut1aRWJqRD3baIAsVAIPoYx8GoN37YGEt6oXWBndiYespoelTAeRJu6RiepzpRwFk= ceph-73df2636-1bf8-11ef-bc9b-c7b262605968 2024-05-27T07:14:08.835 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:14:08.060+0000 7fa1779f3700 1 Processor -- start 2024-05-27T07:14:08.835 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:14:08.060+0000 7fa1779f3700 1 -- start start 2024-05-27T07:14:08.835 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:14:08.060+0000 7fa1779f3700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fa170108af0 0x7fa170108f00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:14:08.835 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:14:08.060+0000 7fa1779f3700 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x7fa1701094f0 con 0x7fa170108af0 2024-05-27T07:14:08.835 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:14:08.061+0000 7fa17578f700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fa170108af0 0x7fa170108f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:14:08.835 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:14:08.061+0000 7fa17578f700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fa170108af0 0x7fa170108f00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.52:42542/0 (socket says 172.21.15.52:42542) 2024-05-27T07:14:08.835 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:14:08.061+0000 7fa17578f700 1 -- 172.21.15.52:0/849640172 learned_addr learned my addr 172.21.15.52:0/849640172 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-05-27T07:14:08.836 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:14:08.061+0000 7fa17578f700 1 -- 172.21.15.52:0/849640172 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa170109c20 con 0x7fa170108af0 2024-05-27T07:14:08.836 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:14:08.061+0000 7fa17578f700 1 --2- 172.21.15.52:0/849640172 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fa170108af0 0x7fa170108f00 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7fa160009e80 tx=0x7fa16000b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c0e94c8f25a2b1f server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:14:08.836 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:14:08.061+0000 7fa16ffff700 1 -- 172.21.15.52:0/849640172 <== mon.0 v2:172.21.15.52:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fa160017070 con 0x7fa170108af0 2024-05-27T07:14:08.836 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:14:08.061+0000 7fa16ffff700 1 -- 172.21.15.52:0/849640172 <== mon.0 v2:172.21.15.52:3300/0 2 ==== config(26 keys) ==== 1070+0+0 (secure 0 0 0) 0x7fa16000b8c0 con 0x7fa170108af0 2024-05-27T07:14:08.836 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:14:08.061+0000 7fa16ffff700 1 -- 172.21.15.52:0/849640172 <== mon.0 v2:172.21.15.52:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fa160004490 con 0x7fa170108af0 2024-05-27T07:14:08.836 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:14:08.062+0000 7fa1779f3700 1 -- 172.21.15.52:0/849640172 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fa170108af0 msgr2=0x7fa170108f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:14:08.836 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:14:08.062+0000 7fa1779f3700 1 --2- 172.21.15.52:0/849640172 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fa170108af0 0x7fa170108f00 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7fa160009e80 tx=0x7fa16000b330 comp rx=0 tx=0).stop 2024-05-27T07:14:08.836 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:14:08.062+0000 7fa1779f3700 1 -- 172.21.15.52:0/849640172 shutdown_connections 2024-05-27T07:14:08.836 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:14:08.062+0000 7fa1779f3700 1 --2- 172.21.15.52:0/849640172 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fa170108af0 0x7fa170108f00 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:14:08.836 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:14:08.062+0000 7fa1779f3700 1 -- 172.21.15.52:0/849640172 >> 172.21.15.52:0/849640172 conn(0x7fa170104150 msgr2=0x7fa1701065e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:14:08.836 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:14:08.062+0000 7fa1779f3700 1 -- 172.21.15.52:0/849640172 shutdown_connections 2024-05-27T07:14:08.836 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:14:08.062+0000 7fa1779f3700 1 -- 172.21.15.52:0/849640172 wait complete. 2024-05-27T07:14:08.836 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:14:08.063+0000 7fa1779f3700 1 Processor -- start 2024-05-27T07:14:08.837 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:14:08.063+0000 7fa1779f3700 1 -- start start 2024-05-27T07:14:08.837 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:14:08.063+0000 7fa1779f3700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fa170108af0 0x7fa17019dfa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:14:08.837 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:14:08.063+0000 7fa1779f3700 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x7fa17010cfd0 con 0x7fa170108af0 2024-05-27T07:14:08.837 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:14:08.063+0000 7fa17578f700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fa170108af0 0x7fa17019dfa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:14:08.837 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:14:08.063+0000 7fa17578f700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fa170108af0 0x7fa17019dfa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.52:42558/0 (socket says 172.21.15.52:42558) 2024-05-27T07:14:08.837 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:14:08.063+0000 7fa17578f700 1 -- 172.21.15.52:0/2802178844 learned_addr learned my addr 172.21.15.52:0/2802178844 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-05-27T07:14:08.837 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:14:08.064+0000 7fa17578f700 1 -- 172.21.15.52:0/2802178844 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa17019e4e0 con 0x7fa170108af0 2024-05-27T07:14:08.837 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:14:08.064+0000 7fa17578f700 1 --2- 172.21.15.52:0/2802178844 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fa170108af0 0x7fa17019dfa0 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7fa160009fd0 tx=0x7fa16000bd50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:14:08.837 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:14:08.064+0000 7fa16e7fc700 1 -- 172.21.15.52:0/2802178844 <== mon.0 v2:172.21.15.52:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fa160020020 con 0x7fa170108af0 2024-05-27T07:14:08.837 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:14:08.064+0000 7fa1779f3700 1 -- 172.21.15.52:0/2802178844 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa17019e770 con 0x7fa170108af0 2024-05-27T07:14:08.837 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:14:08.064+0000 7fa16e7fc700 1 -- 172.21.15.52:0/2802178844 <== mon.0 v2:172.21.15.52:3300/0 2 ==== config(26 keys) ==== 1070+0+0 (secure 0 0 0) 0x7fa160012430 con 0x7fa170108af0 2024-05-27T07:14:08.837 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:14:08.064+0000 7fa16e7fc700 1 -- 172.21.15.52:0/2802178844 <== mon.0 v2:172.21.15.52:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fa160017040 con 0x7fa170108af0 2024-05-27T07:14:08.837 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:14:08.064+0000 7fa1779f3700 1 -- 172.21.15.52:0/2802178844 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa1701a1380 con 0x7fa170108af0 2024-05-27T07:14:08.837 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:14:08.065+0000 7fa1779f3700 1 -- 172.21.15.52:0/2802178844 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa170064e10 con 0x7fa170108af0 2024-05-27T07:14:08.838 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:14:08.065+0000 7fa16e7fc700 1 -- 172.21.15.52:0/2802178844 <== mon.0 v2:172.21.15.52:3300/0 4 ==== mgrmap(e 8) ==== 50222+0+0 (secure 0 0 0) 0x7fa160023020 con 0x7fa170108af0 2024-05-27T07:14:08.838 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:14:08.065+0000 7fa16e7fc700 1 --2- 172.21.15.52:0/2802178844 >> [v2:172.21.15.52:6800/1631683163,v1:172.21.15.52:6801/1631683163] conn(0x7fa15803e420 0x7fa158040930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:14:08.838 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:14:08.065+0000 7fa16e7fc700 1 -- 172.21.15.52:0/2802178844 <== mon.0 v2:172.21.15.52:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 940+0+0 (secure 0 0 0) 0x7fa16000f050 con 0x7fa170108af0 2024-05-27T07:14:08.838 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:14:08.065+0000 7fa174f8e700 1 --2- 172.21.15.52:0/2802178844 >> [v2:172.21.15.52:6800/1631683163,v1:172.21.15.52:6801/1631683163] conn(0x7fa15803e420 0x7fa158040930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:14:08.838 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:14:08.066+0000 7fa174f8e700 1 --2- 172.21.15.52:0/2802178844 >> [v2:172.21.15.52:6800/1631683163,v1:172.21.15.52:6801/1631683163] conn(0x7fa15803e420 0x7fa158040930 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7fa164006fd0 tx=0x7fa164006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:14:08.838 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:14:08.071+0000 7fa16e7fc700 1 -- 172.21.15.52:0/2802178844 <== mon.0 v2:172.21.15.52:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191963 (secure 0 0 0) 0x7fa16002a020 con 0x7fa170108af0 2024-05-27T07:14:08.838 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:14:08.288+0000 7fa1779f3700 1 -- 172.21.15.52:0/2802178844 --> [v2:172.21.15.52:6800/1631683163,v1:172.21.15.52:6801/1631683163] -- mgr_command(tid 0: {"prefix": "cephadm get-pub-key", "target": ["mon-mgr", ""]}) -- 0x7fa17002e950 con 0x7fa15803e420 2024-05-27T07:14:08.838 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:14:08.289+0000 7fa16e7fc700 1 -- 172.21.15.52:0/2802178844 <== mgr.14120 v2:172.21.15.52:6800/1631683163 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+595 (secure 0 0 0) 0x7fa17002e950 con 0x7fa15803e420 2024-05-27T07:14:08.838 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:14:08.291+0000 7fa1779f3700 1 -- 172.21.15.52:0/2802178844 >> [v2:172.21.15.52:6800/1631683163,v1:172.21.15.52:6801/1631683163] conn(0x7fa15803e420 msgr2=0x7fa158040930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:14:08.838 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:14:08.291+0000 7fa1779f3700 1 --2- 172.21.15.52:0/2802178844 >> [v2:172.21.15.52:6800/1631683163,v1:172.21.15.52:6801/1631683163] conn(0x7fa15803e420 0x7fa158040930 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7fa164006fd0 tx=0x7fa164006e40 comp rx=0 tx=0).stop 2024-05-27T07:14:08.838 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:14:08.291+0000 7fa1779f3700 1 -- 172.21.15.52:0/2802178844 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fa170108af0 msgr2=0x7fa17019dfa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:14:08.839 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:14:08.291+0000 7fa1779f3700 1 --2- 172.21.15.52:0/2802178844 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fa170108af0 0x7fa17019dfa0 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7fa160009fd0 tx=0x7fa16000bd50 comp rx=0 tx=0).stop 2024-05-27T07:14:08.839 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:14:08.291+0000 7fa1779f3700 1 -- 172.21.15.52:0/2802178844 shutdown_connections 2024-05-27T07:14:08.839 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:14:08.291+0000 7fa1779f3700 1 --2- 172.21.15.52:0/2802178844 >> [v2:172.21.15.52:6800/1631683163,v1:172.21.15.52:6801/1631683163] conn(0x7fa15803e420 0x7fa158040930 unknown :-1 s=CLOSED pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:14:08.839 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:14:08.291+0000 7fa1779f3700 1 --2- 172.21.15.52:0/2802178844 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fa170108af0 0x7fa17019dfa0 unknown :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:14:08.839 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:14:08.291+0000 7fa1779f3700 1 -- 172.21.15.52:0/2802178844 >> 172.21.15.52:0/2802178844 conn(0x7fa170104150 msgr2=0x7fa170194530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:14:08.839 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:14:08.291+0000 7fa1779f3700 1 -- 172.21.15.52:0/2802178844 shutdown_connections 2024-05-27T07:14:08.839 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:14:08.291+0000 7fa1779f3700 1 -- 172.21.15.52:0/2802178844 wait complete. 2024-05-27T07:14:08.839 INFO:teuthology.orchestra.run.smithi052.stdout:Wrote public SSH key to /home/ubuntu/cephtest/ceph.pub 2024-05-27T07:14:08.839 INFO:teuthology.orchestra.run.smithi052.stdout:Adding key to root@localhost authorized_keys... 2024-05-27T07:14:08.839 INFO:teuthology.orchestra.run.smithi052.stdout:Adding host smithi052... 2024-05-27T07:14:09.083 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:14:09 smithi052 ceph-mon[31955]: from='client.14138 -' entity='client.admin' cmd=[{"prefix": "cephadm get-pub-key", "target": ["mon-mgr", ""]}]: dispatch 2024-05-27T07:14:10.143 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:14:10 smithi052 ceph-mon[31955]: from='client.14140 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi052", "addr": "172.21.15.52", "target": ["mon-mgr", ""]}]: dispatch 2024-05-27T07:14:11.480 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:14:11 smithi052 ceph-mon[31955]: Deploying cephadm binary to smithi052 2024-05-27T07:14:14.385 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout Added host 'smithi052' with addr '172.21.15.52' 2024-05-27T07:14:14.385 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:14:09.297+0000 7f89049d7700 1 Processor -- start 2024-05-27T07:14:14.385 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:14:09.297+0000 7f89049d7700 1 -- start start 2024-05-27T07:14:14.385 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:14:09.297+0000 7f89049d7700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f890010b180 0x7f890010b590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:14:14.385 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:14:09.297+0000 7f89049d7700 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x7f890010bb60 con 0x7f890010b180 2024-05-27T07:14:14.386 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:14:09.298+0000 7f88fe59c700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f890010b180 0x7f890010b590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:14:14.386 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:14:09.298+0000 7f88fe59c700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f890010b180 0x7f890010b590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.52:42568/0 (socket says 172.21.15.52:42568) 2024-05-27T07:14:14.386 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:14:09.298+0000 7f88fe59c700 1 -- 172.21.15.52:0/1547843141 learned_addr learned my addr 172.21.15.52:0/1547843141 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-05-27T07:14:14.386 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:14:09.298+0000 7f88fe59c700 1 -- 172.21.15.52:0/1547843141 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f890010bce0 con 0x7f890010b180 2024-05-27T07:14:14.386 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:14:09.298+0000 7f88fe59c700 1 --2- 172.21.15.52:0/1547843141 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f890010b180 0x7f890010b590 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f88f00098a0 tx=0x7f88f0009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=688e752c9f2557db server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:14:14.386 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:14:09.298+0000 7f88fd59a700 1 -- 172.21.15.52:0/1547843141 <== mon.0 v2:172.21.15.52:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f88f0004160 con 0x7f890010b180 2024-05-27T07:14:14.386 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:14:09.298+0000 7f88fd59a700 1 -- 172.21.15.52:0/1547843141 <== mon.0 v2:172.21.15.52:3300/0 2 ==== config(26 keys) ==== 1070+0+0 (secure 0 0 0) 0x7f88f0004300 con 0x7f890010b180 2024-05-27T07:14:14.386 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:14:09.298+0000 7f88fd59a700 1 -- 172.21.15.52:0/1547843141 <== mon.0 v2:172.21.15.52:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f88f0011af0 con 0x7f890010b180 2024-05-27T07:14:14.386 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:14:09.299+0000 7f89049d7700 1 -- 172.21.15.52:0/1547843141 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f890010b180 msgr2=0x7f890010b590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:14:14.386 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:14:09.299+0000 7f89049d7700 1 --2- 172.21.15.52:0/1547843141 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f890010b180 0x7f890010b590 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f88f00098a0 tx=0x7f88f0009bb0 comp rx=0 tx=0).stop 2024-05-27T07:14:14.386 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:14:09.299+0000 7f89049d7700 1 -- 172.21.15.52:0/1547843141 shutdown_connections 2024-05-27T07:14:14.386 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:14:09.299+0000 7f89049d7700 1 --2- 172.21.15.52:0/1547843141 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f890010b180 0x7f890010b590 unknown :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:14:14.387 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:14:09.299+0000 7f89049d7700 1 -- 172.21.15.52:0/1547843141 >> 172.21.15.52:0/1547843141 conn(0x7f89001039a0 msgr2=0x7f8900105df0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:14:14.387 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:14:09.299+0000 7f89049d7700 1 -- 172.21.15.52:0/1547843141 shutdown_connections 2024-05-27T07:14:14.387 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:14:09.299+0000 7f89049d7700 1 -- 172.21.15.52:0/1547843141 wait complete. 2024-05-27T07:14:14.387 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:14:09.300+0000 7f89049d7700 1 Processor -- start 2024-05-27T07:14:14.387 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:14:09.300+0000 7f89049d7700 1 -- start start 2024-05-27T07:14:14.387 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:14:09.300+0000 7f89049d7700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f890010b180 0x7f890019e010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:14:14.387 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:14:09.300+0000 7f89049d7700 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x7f8900108a30 con 0x7f890010b180 2024-05-27T07:14:14.387 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:14:09.300+0000 7f88fe59c700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f890010b180 0x7f890019e010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:14:14.387 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:14:09.300+0000 7f88fe59c700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f890010b180 0x7f890019e010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.52:42576/0 (socket says 172.21.15.52:42576) 2024-05-27T07:14:14.387 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:14:09.300+0000 7f88fe59c700 1 -- 172.21.15.52:0/1565612895 learned_addr learned my addr 172.21.15.52:0/1565612895 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-05-27T07:14:14.387 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:14:09.301+0000 7f88fe59c700 1 -- 172.21.15.52:0/1565612895 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f890019e550 con 0x7f890010b180 2024-05-27T07:14:14.387 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:14:09.301+0000 7f88fe59c700 1 --2- 172.21.15.52:0/1565612895 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f890010b180 0x7f890019e010 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f88f0009fb0 tx=0x7f88f0011890 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:14:14.387 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:14:09.301+0000 7f88ef7fe700 1 -- 172.21.15.52:0/1565612895 <== mon.0 v2:172.21.15.52:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f88f0011bc0 con 0x7f890010b180 2024-05-27T07:14:14.388 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:14:09.301+0000 7f89049d7700 1 -- 172.21.15.52:0/1565612895 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f890019e7e0 con 0x7f890010b180 2024-05-27T07:14:14.388 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:14:09.301+0000 7f89049d7700 1 -- 172.21.15.52:0/1565612895 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f89001a13f0 con 0x7f890010b180 2024-05-27T07:14:14.388 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:14:09.301+0000 7f88ef7fe700 1 -- 172.21.15.52:0/1565612895 <== mon.0 v2:172.21.15.52:3300/0 2 ==== config(26 keys) ==== 1070+0+0 (secure 0 0 0) 0x7f88f0024440 con 0x7f890010b180 2024-05-27T07:14:14.388 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:14:09.301+0000 7f88ef7fe700 1 -- 172.21.15.52:0/1565612895 <== mon.0 v2:172.21.15.52:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f88f001a430 con 0x7f890010b180 2024-05-27T07:14:14.388 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:14:09.302+0000 7f88ef7fe700 1 -- 172.21.15.52:0/1565612895 <== mon.0 v2:172.21.15.52:3300/0 4 ==== mgrmap(e 8) ==== 50222+0+0 (secure 0 0 0) 0x7f88f001a5f0 con 0x7f890010b180 2024-05-27T07:14:14.388 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:14:09.302+0000 7f88ef7fe700 1 --2- 172.21.15.52:0/1565612895 >> [v2:172.21.15.52:6800/1631683163,v1:172.21.15.52:6801/1631683163] conn(0x7f88e803e420 0x7f88e8040930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:14:14.388 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:14:09.302+0000 7f88ef7fe700 1 -- 172.21.15.52:0/1565612895 <== mon.0 v2:172.21.15.52:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 940+0+0 (secure 0 0 0) 0x7f88f0054680 con 0x7f890010b180 2024-05-27T07:14:14.388 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:14:09.302+0000 7f89049d7700 1 -- 172.21.15.52:0/1565612895 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f88e0005360 con 0x7f890010b180 2024-05-27T07:14:14.388 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:14:09.302+0000 7f88fdd9b700 1 --2- 172.21.15.52:0/1565612895 >> [v2:172.21.15.52:6800/1631683163,v1:172.21.15.52:6801/1631683163] conn(0x7f88e803e420 0x7f88e8040930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:14:14.388 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:14:09.303+0000 7f88fdd9b700 1 --2- 172.21.15.52:0/1565612895 >> [v2:172.21.15.52:6800/1631683163,v1:172.21.15.52:6801/1631683163] conn(0x7f88e803e420 0x7f88e8040930 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7f88f4006fd0 tx=0x7f88f4006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:14:14.388 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:14:09.308+0000 7f88ef7fe700 1 -- 172.21.15.52:0/1565612895 <== mon.0 v2:172.21.15.52:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191963 (secure 0 0 0) 0x7f88f002c350 con 0x7f890010b180 2024-05-27T07:14:14.388 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:14:09.531+0000 7f89049d7700 1 -- 172.21.15.52:0/1565612895 --> [v2:172.21.15.52:6800/1631683163,v1:172.21.15.52:6801/1631683163] -- mgr_command(tid 0: {"prefix": "orch host add", "hostname": "smithi052", "addr": "172.21.15.52", "target": ["mon-mgr", ""]}) -- 0x7f88e0000bf0 con 0x7f88e803e420 2024-05-27T07:14:14.388 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:14:13.792+0000 7f88ef7fe700 1 -- 172.21.15.52:0/1565612895 <== mgr.14120 v2:172.21.15.52:6800/1631683163 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+48 (secure 0 0 0) 0x7f88e0000bf0 con 0x7f88e803e420 2024-05-27T07:14:14.389 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:14:13.795+0000 7f89049d7700 1 -- 172.21.15.52:0/1565612895 >> [v2:172.21.15.52:6800/1631683163,v1:172.21.15.52:6801/1631683163] conn(0x7f88e803e420 msgr2=0x7f88e8040930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:14:14.389 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:14:13.795+0000 7f89049d7700 1 --2- 172.21.15.52:0/1565612895 >> [v2:172.21.15.52:6800/1631683163,v1:172.21.15.52:6801/1631683163] conn(0x7f88e803e420 0x7f88e8040930 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7f88f4006fd0 tx=0x7f88f4006e40 comp rx=0 tx=0).stop 2024-05-27T07:14:14.389 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:14:13.795+0000 7f89049d7700 1 -- 172.21.15.52:0/1565612895 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f890010b180 msgr2=0x7f890019e010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:14:14.389 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:14:13.795+0000 7f89049d7700 1 --2- 172.21.15.52:0/1565612895 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f890010b180 0x7f890019e010 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f88f0009fb0 tx=0x7f88f0011890 comp rx=0 tx=0).stop 2024-05-27T07:14:14.389 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:14:13.795+0000 7f89049d7700 1 -- 172.21.15.52:0/1565612895 shutdown_connections 2024-05-27T07:14:14.389 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:14:13.795+0000 7f89049d7700 1 --2- 172.21.15.52:0/1565612895 >> [v2:172.21.15.52:6800/1631683163,v1:172.21.15.52:6801/1631683163] conn(0x7f88e803e420 0x7f88e8040930 unknown :-1 s=CLOSED pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:14:14.389 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:14:13.795+0000 7f89049d7700 1 --2- 172.21.15.52:0/1565612895 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f890010b180 0x7f890019e010 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:14:14.389 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:14:13.795+0000 7f89049d7700 1 -- 172.21.15.52:0/1565612895 >> 172.21.15.52:0/1565612895 conn(0x7f89001039a0 msgr2=0x7f89001053e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:14:14.389 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:14:13.795+0000 7f89049d7700 1 -- 172.21.15.52:0/1565612895 shutdown_connections 2024-05-27T07:14:14.389 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:14:13.795+0000 7f89049d7700 1 -- 172.21.15.52:0/1565612895 wait complete. 2024-05-27T07:14:14.389 INFO:teuthology.orchestra.run.smithi052.stdout:Deploying unmanaged mon service... 2024-05-27T07:14:14.918 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:14:14 smithi052 ceph-mon[31955]: from='mgr.14120 172.21.15.52:0/2490234989' entity='mgr.x' 2024-05-27T07:14:14.918 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:14:14 smithi052 ceph-mon[31955]: Added host smithi052 2024-05-27T07:14:14.918 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:14:14 smithi052 ceph-mon[31955]: from='mgr.14120 172.21.15.52:0/2490234989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-27T07:14:16.336 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:14:16 smithi052 ceph-mon[31955]: from='client.14142 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "unmanaged": true, "target": ["mon-mgr", ""]}]: dispatch 2024-05-27T07:14:16.336 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:14:16 smithi052 ceph-mon[31955]: Saving service mon spec with placement count:5 2024-05-27T07:14:16.336 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:14:16 smithi052 ceph-mon[31955]: from='mgr.14120 172.21.15.52:0/2490234989' entity='mgr.x' 2024-05-27T07:14:16.618 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout Scheduled mon update... 2024-05-27T07:14:16.618 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:14:14.915+0000 7ff2bd3e6700 1 Processor -- start 2024-05-27T07:14:16.618 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:14:14.915+0000 7ff2bd3e6700 1 -- start start 2024-05-27T07:14:16.618 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:14:14.915+0000 7ff2bd3e6700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff2b810a6f0 0x7ff2b810cb10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:14:16.619 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:14:14.915+0000 7ff2bd3e6700 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x7ff2b807d670 con 0x7ff2b810a6f0 2024-05-27T07:14:16.619 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:14:14.915+0000 7ff2b6ffd700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff2b810a6f0 0x7ff2b810cb10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:14:16.619 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:14:14.916+0000 7ff2b6ffd700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff2b810a6f0 0x7ff2b810cb10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.52:54914/0 (socket says 172.21.15.52:54914) 2024-05-27T07:14:16.619 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:14:14.916+0000 7ff2b6ffd700 1 -- 172.21.15.52:0/933622217 learned_addr learned my addr 172.21.15.52:0/933622217 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-05-27T07:14:16.619 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:14:14.916+0000 7ff2b6ffd700 1 -- 172.21.15.52:0/933622217 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff2b810d050 con 0x7ff2b810a6f0 2024-05-27T07:14:16.619 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:14:14.916+0000 7ff2b6ffd700 1 --2- 172.21.15.52:0/933622217 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff2b810a6f0 0x7ff2b810cb10 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7ff2a80098a0 tx=0x7ff2a8009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3954c716f4630fd server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:14:16.619 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:14:14.916+0000 7ff2b5ffb700 1 -- 172.21.15.52:0/933622217 <== mon.0 v2:172.21.15.52:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7ff2a8004160 con 0x7ff2b810a6f0 2024-05-27T07:14:16.619 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:14:14.916+0000 7ff2b5ffb700 1 -- 172.21.15.52:0/933622217 <== mon.0 v2:172.21.15.52:3300/0 2 ==== config(26 keys) ==== 1070+0+0 (secure 0 0 0) 0x7ff2a8004300 con 0x7ff2b810a6f0 2024-05-27T07:14:16.619 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:14:14.916+0000 7ff2b5ffb700 1 -- 172.21.15.52:0/933622217 <== mon.0 v2:172.21.15.52:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7ff2a8011af0 con 0x7ff2b810a6f0 2024-05-27T07:14:16.619 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:14:14.917+0000 7ff2bd3e6700 1 -- 172.21.15.52:0/933622217 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff2b810a6f0 msgr2=0x7ff2b810cb10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:14:16.619 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:14:14.917+0000 7ff2bd3e6700 1 --2- 172.21.15.52:0/933622217 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff2b810a6f0 0x7ff2b810cb10 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7ff2a80098a0 tx=0x7ff2a8009bb0 comp rx=0 tx=0).stop 2024-05-27T07:14:16.619 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:14:14.917+0000 7ff2bd3e6700 1 -- 172.21.15.52:0/933622217 shutdown_connections 2024-05-27T07:14:16.619 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:14:14.917+0000 7ff2bd3e6700 1 --2- 172.21.15.52:0/933622217 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff2b810a6f0 0x7ff2b810cb10 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:14:16.620 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:14:14.917+0000 7ff2bd3e6700 1 -- 172.21.15.52:0/933622217 >> 172.21.15.52:0/933622217 conn(0x7ff2b81041b0 msgr2=0x7ff2b8106600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:14:16.620 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:14:14.917+0000 7ff2bd3e6700 1 -- 172.21.15.52:0/933622217 shutdown_connections 2024-05-27T07:14:16.620 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:14:14.917+0000 7ff2bd3e6700 1 -- 172.21.15.52:0/933622217 wait complete. 2024-05-27T07:14:16.620 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:14:14.918+0000 7ff2bd3e6700 1 Processor -- start 2024-05-27T07:14:16.620 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:14:14.918+0000 7ff2bd3e6700 1 -- start start 2024-05-27T07:14:16.620 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:14:14.918+0000 7ff2bd3e6700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff2b810a6f0 0x7ff2b81a01e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:14:16.620 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:14:14.918+0000 7ff2bd3e6700 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x7ff2b810df50 con 0x7ff2b810a6f0 2024-05-27T07:14:16.620 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:14:14.918+0000 7ff2b6ffd700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff2b810a6f0 0x7ff2b81a01e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:14:16.620 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:14:14.918+0000 7ff2b6ffd700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff2b810a6f0 0x7ff2b81a01e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.52:54930/0 (socket says 172.21.15.52:54930) 2024-05-27T07:14:16.620 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:14:14.918+0000 7ff2b6ffd700 1 -- 172.21.15.52:0/1087167289 learned_addr learned my addr 172.21.15.52:0/1087167289 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-05-27T07:14:16.620 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:14:14.919+0000 7ff2b6ffd700 1 -- 172.21.15.52:0/1087167289 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff2b81a0720 con 0x7ff2b810a6f0 2024-05-27T07:14:16.620 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:14:14.919+0000 7ff2b6ffd700 1 --2- 172.21.15.52:0/1087167289 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff2b810a6f0 0x7ff2b81a01e0 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7ff2a80117b0 tx=0x7ff2a8011b70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:14:16.621 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:14:14.919+0000 7ff29ffff700 1 -- 172.21.15.52:0/1087167289 <== mon.0 v2:172.21.15.52:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7ff2a80039c0 con 0x7ff2b810a6f0 2024-05-27T07:14:16.621 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:14:14.919+0000 7ff2bd3e6700 1 -- 172.21.15.52:0/1087167289 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff2b81a09b0 con 0x7ff2b810a6f0 2024-05-27T07:14:16.621 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:14:14.919+0000 7ff29ffff700 1 -- 172.21.15.52:0/1087167289 <== mon.0 v2:172.21.15.52:3300/0 2 ==== config(26 keys) ==== 1070+0+0 (secure 0 0 0) 0x7ff2a8003b60 con 0x7ff2b810a6f0 2024-05-27T07:14:16.621 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:14:14.919+0000 7ff29ffff700 1 -- 172.21.15.52:0/1087167289 <== mon.0 v2:172.21.15.52:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7ff2a80196a0 con 0x7ff2b810a6f0 2024-05-27T07:14:16.621 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:14:14.919+0000 7ff2bd3e6700 1 -- 172.21.15.52:0/1087167289 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff2b81a35c0 con 0x7ff2b810a6f0 2024-05-27T07:14:16.621 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:14:14.920+0000 7ff2bd3e6700 1 -- 172.21.15.52:0/1087167289 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff2b8064e10 con 0x7ff2b810a6f0 2024-05-27T07:14:16.621 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:14:14.920+0000 7ff29ffff700 1 -- 172.21.15.52:0/1087167289 <== mon.0 v2:172.21.15.52:3300/0 4 ==== mgrmap(e 8) ==== 50222+0+0 (secure 0 0 0) 0x7ff2a8019960 con 0x7ff2b810a6f0 2024-05-27T07:14:16.621 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:14:14.920+0000 7ff29ffff700 1 --2- 172.21.15.52:0/1087167289 >> [v2:172.21.15.52:6800/1631683163,v1:172.21.15.52:6801/1631683163] conn(0x7ff2a003e470 0x7ff2a0040980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:14:16.621 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:14:14.920+0000 7ff29ffff700 1 -- 172.21.15.52:0/1087167289 <== mon.0 v2:172.21.15.52:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 940+0+0 (secure 0 0 0) 0x7ff2a8053920 con 0x7ff2b810a6f0 2024-05-27T07:14:16.621 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:14:14.920+0000 7ff2b67fc700 1 --2- 172.21.15.52:0/1087167289 >> [v2:172.21.15.52:6800/1631683163,v1:172.21.15.52:6801/1631683163] conn(0x7ff2a003e470 0x7ff2a0040980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:14:16.621 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:14:14.921+0000 7ff2b67fc700 1 --2- 172.21.15.52:0/1087167289 >> [v2:172.21.15.52:6800/1631683163,v1:172.21.15.52:6801/1631683163] conn(0x7ff2a003e470 0x7ff2a0040980 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7ff2ac006fd0 tx=0x7ff2ac006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:14:16.621 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:14:14.926+0000 7ff29ffff700 1 -- 172.21.15.52:0/1087167289 <== mon.0 v2:172.21.15.52:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191963 (secure 0 0 0) 0x7ff2a800b3e0 con 0x7ff2b810a6f0 2024-05-27T07:14:16.622 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:14:15.157+0000 7ff2bd3e6700 1 -- 172.21.15.52:0/1087167289 --> [v2:172.21.15.52:6800/1631683163,v1:172.21.15.52:6801/1631683163] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "mon", "unmanaged": true, "target": ["mon-mgr", ""]}) -- 0x7ff2b802e950 con 0x7ff2a003e470 2024-05-27T07:14:16.622 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:14:15.162+0000 7ff29ffff700 1 -- 172.21.15.52:0/1087167289 <== mgr.14120 v2:172.21.15.52:6800/1631683163 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+24 (secure 0 0 0) 0x7ff2b802e950 con 0x7ff2a003e470 2024-05-27T07:14:16.622 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:14:15.165+0000 7ff2bd3e6700 1 -- 172.21.15.52:0/1087167289 >> [v2:172.21.15.52:6800/1631683163,v1:172.21.15.52:6801/1631683163] conn(0x7ff2a003e470 msgr2=0x7ff2a0040980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:14:16.622 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:14:15.165+0000 7ff2bd3e6700 1 --2- 172.21.15.52:0/1087167289 >> [v2:172.21.15.52:6800/1631683163,v1:172.21.15.52:6801/1631683163] conn(0x7ff2a003e470 0x7ff2a0040980 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7ff2ac006fd0 tx=0x7ff2ac006e40 comp rx=0 tx=0).stop 2024-05-27T07:14:16.622 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:14:15.165+0000 7ff2bd3e6700 1 -- 172.21.15.52:0/1087167289 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff2b810a6f0 msgr2=0x7ff2b81a01e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:14:16.622 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:14:15.165+0000 7ff2bd3e6700 1 --2- 172.21.15.52:0/1087167289 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff2b810a6f0 0x7ff2b81a01e0 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7ff2a80117b0 tx=0x7ff2a8011b70 comp rx=0 tx=0).stop 2024-05-27T07:14:16.622 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:14:15.165+0000 7ff2bd3e6700 1 -- 172.21.15.52:0/1087167289 shutdown_connections 2024-05-27T07:14:16.622 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:14:15.165+0000 7ff2bd3e6700 1 --2- 172.21.15.52:0/1087167289 >> [v2:172.21.15.52:6800/1631683163,v1:172.21.15.52:6801/1631683163] conn(0x7ff2a003e470 0x7ff2a0040980 unknown :-1 s=CLOSED pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:14:16.622 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:14:15.165+0000 7ff2bd3e6700 1 --2- 172.21.15.52:0/1087167289 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff2b810a6f0 0x7ff2b81a01e0 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:14:16.622 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:14:15.165+0000 7ff2bd3e6700 1 -- 172.21.15.52:0/1087167289 >> 172.21.15.52:0/1087167289 conn(0x7ff2b81041b0 msgr2=0x7ff2b8108650 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:14:16.622 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:14:15.165+0000 7ff2bd3e6700 1 -- 172.21.15.52:0/1087167289 shutdown_connections 2024-05-27T07:14:16.622 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:14:15.165+0000 7ff2bd3e6700 1 -- 172.21.15.52:0/1087167289 wait complete. 2024-05-27T07:14:16.622 INFO:teuthology.orchestra.run.smithi052.stdout:Deploying unmanaged mgr service... 2024-05-27T07:14:17.480 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:14:17 smithi052 ceph-mon[31955]: from='mgr.14120 172.21.15.52:0/2490234989' entity='mgr.x' 2024-05-27T07:14:17.480 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:14:17 smithi052 ceph-mon[31955]: from='mgr.14120 172.21.15.52:0/2490234989' entity='mgr.x' 2024-05-27T07:14:17.850 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout Scheduled mgr update... 2024-05-27T07:14:17.851 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:14:17.057+0000 7f99d440e700 1 Processor -- start 2024-05-27T07:14:17.851 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:14:17.057+0000 7f99d440e700 1 -- start start 2024-05-27T07:14:17.851 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:14:17.057+0000 7f99d440e700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f99cc10a660 0x7f99cc10ca80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:14:17.851 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:14:17.057+0000 7f99d440e700 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x7f99cc07cde0 con 0x7f99cc10a660 2024-05-27T07:14:17.851 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:14:17.057+0000 7f99d21aa700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f99cc10a660 0x7f99cc10ca80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:14:17.851 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:14:17.057+0000 7f99d21aa700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f99cc10a660 0x7f99cc10ca80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.52:54942/0 (socket says 172.21.15.52:54942) 2024-05-27T07:14:17.851 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:14:17.057+0000 7f99d21aa700 1 -- 172.21.15.52:0/3841378345 learned_addr learned my addr 172.21.15.52:0/3841378345 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-05-27T07:14:17.851 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:14:17.058+0000 7f99d21aa700 1 -- 172.21.15.52:0/3841378345 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f99cc10cfc0 con 0x7f99cc10a660 2024-05-27T07:14:17.851 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:14:17.058+0000 7f99d21aa700 1 --2- 172.21.15.52:0/3841378345 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f99cc10a660 0x7f99cc10ca80 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f99c80098a0 tx=0x7f99c8009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=11779a9fc0b2e5d2 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:14:17.852 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:14:17.058+0000 7f99d11a8700 1 -- 172.21.15.52:0/3841378345 <== mon.0 v2:172.21.15.52:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f99c8004160 con 0x7f99cc10a660 2024-05-27T07:14:17.852 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:14:17.058+0000 7f99d11a8700 1 -- 172.21.15.52:0/3841378345 <== mon.0 v2:172.21.15.52:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f99c8004300 con 0x7f99cc10a660 2024-05-27T07:14:17.852 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:14:17.058+0000 7f99d11a8700 1 -- 172.21.15.52:0/3841378345 <== mon.0 v2:172.21.15.52:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f99c8011a20 con 0x7f99cc10a660 2024-05-27T07:14:17.852 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:14:17.059+0000 7f99d440e700 1 -- 172.21.15.52:0/3841378345 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f99cc10a660 msgr2=0x7f99cc10ca80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:14:17.852 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:14:17.059+0000 7f99d440e700 1 --2- 172.21.15.52:0/3841378345 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f99cc10a660 0x7f99cc10ca80 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f99c80098a0 tx=0x7f99c8009bb0 comp rx=0 tx=0).stop 2024-05-27T07:14:17.852 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:14:17.059+0000 7f99d440e700 1 -- 172.21.15.52:0/3841378345 shutdown_connections 2024-05-27T07:14:17.852 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:14:17.059+0000 7f99d440e700 1 --2- 172.21.15.52:0/3841378345 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f99cc10a660 0x7f99cc10ca80 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:14:17.852 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:14:17.059+0000 7f99d440e700 1 -- 172.21.15.52:0/3841378345 >> 172.21.15.52:0/3841378345 conn(0x7f99cc1040e0 msgr2=0x7f99cc106570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:14:17.852 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:14:17.059+0000 7f99d440e700 1 -- 172.21.15.52:0/3841378345 shutdown_connections 2024-05-27T07:14:17.852 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:14:17.059+0000 7f99d440e700 1 -- 172.21.15.52:0/3841378345 wait complete. 2024-05-27T07:14:17.852 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:14:17.060+0000 7f99d440e700 1 Processor -- start 2024-05-27T07:14:17.852 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:14:17.060+0000 7f99d440e700 1 -- start start 2024-05-27T07:14:17.852 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:14:17.060+0000 7f99d440e700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f99cc10a660 0x7f99cc19dee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:14:17.853 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:14:17.060+0000 7f99d440e700 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x7f99cc10df00 con 0x7f99cc10a660 2024-05-27T07:14:17.853 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:14:17.060+0000 7f99d21aa700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f99cc10a660 0x7f99cc19dee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:14:17.853 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:14:17.060+0000 7f99d21aa700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f99cc10a660 0x7f99cc19dee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.52:54956/0 (socket says 172.21.15.52:54956) 2024-05-27T07:14:17.853 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:14:17.060+0000 7f99d21aa700 1 -- 172.21.15.52:0/1578593677 learned_addr learned my addr 172.21.15.52:0/1578593677 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-05-27T07:14:17.853 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:14:17.060+0000 7f99d21aa700 1 -- 172.21.15.52:0/1578593677 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f99cc19e420 con 0x7f99cc10a660 2024-05-27T07:14:17.853 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:14:17.061+0000 7f99d21aa700 1 --2- 172.21.15.52:0/1578593677 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f99cc10a660 0x7f99cc19dee0 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f99c80119f0 tx=0x7f99c8011ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:14:17.853 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:14:17.061+0000 7f99c37fe700 1 -- 172.21.15.52:0/1578593677 <== mon.0 v2:172.21.15.52:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f99c80039e0 con 0x7f99cc10a660 2024-05-27T07:14:17.853 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:14:17.061+0000 7f99c37fe700 1 -- 172.21.15.52:0/1578593677 <== mon.0 v2:172.21.15.52:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f99c801b8f0 con 0x7f99cc10a660 2024-05-27T07:14:17.853 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:14:17.061+0000 7f99d440e700 1 -- 172.21.15.52:0/1578593677 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f99cc19e6b0 con 0x7f99cc10a660 2024-05-27T07:14:17.853 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:14:17.061+0000 7f99d440e700 1 -- 172.21.15.52:0/1578593677 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f99cc1a12c0 con 0x7f99cc10a660 2024-05-27T07:14:17.853 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:14:17.061+0000 7f99c37fe700 1 -- 172.21.15.52:0/1578593677 <== mon.0 v2:172.21.15.52:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f99c80237a0 con 0x7f99cc10a660 2024-05-27T07:14:17.853 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:14:17.062+0000 7f99d440e700 1 -- 172.21.15.52:0/1578593677 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f99cc064e10 con 0x7f99cc10a660 2024-05-27T07:14:17.854 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:14:17.062+0000 7f99c37fe700 1 -- 172.21.15.52:0/1578593677 <== mon.0 v2:172.21.15.52:3300/0 4 ==== mgrmap(e 8) ==== 50222+0+0 (secure 0 0 0) 0x7f99c802c3f0 con 0x7f99cc10a660 2024-05-27T07:14:17.854 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:14:17.062+0000 7f99c37fe700 1 --2- 172.21.15.52:0/1578593677 >> [v2:172.21.15.52:6800/1631683163,v1:172.21.15.52:6801/1631683163] conn(0x7f99b803e470 0x7f99b8040980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:14:17.854 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:14:17.062+0000 7f99c37fe700 1 -- 172.21.15.52:0/1578593677 <== mon.0 v2:172.21.15.52:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 940+0+0 (secure 0 0 0) 0x7f99c8053db0 con 0x7f99cc10a660 2024-05-27T07:14:17.854 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:14:17.062+0000 7f99d19a9700 1 --2- 172.21.15.52:0/1578593677 >> [v2:172.21.15.52:6800/1631683163,v1:172.21.15.52:6801/1631683163] conn(0x7f99b803e470 0x7f99b8040980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:14:17.854 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:14:17.063+0000 7f99d19a9700 1 --2- 172.21.15.52:0/1578593677 >> [v2:172.21.15.52:6800/1631683163,v1:172.21.15.52:6801/1631683163] conn(0x7f99b803e470 0x7f99b8040980 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7f99bc006fd0 tx=0x7f99bc006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:14:17.854 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:14:17.068+0000 7f99c37fe700 1 -- 172.21.15.52:0/1578593677 <== mon.0 v2:172.21.15.52:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191963 (secure 0 0 0) 0x7f99c801f030 con 0x7f99cc10a660 2024-05-27T07:14:17.854 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:14:17.296+0000 7f99d440e700 1 -- 172.21.15.52:0/1578593677 --> [v2:172.21.15.52:6800/1631683163,v1:172.21.15.52:6801/1631683163] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "mgr", "unmanaged": true, "target": ["mon-mgr", ""]}) -- 0x7f99cc1050e0 con 0x7f99b803e470 2024-05-27T07:14:17.854 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:14:17.301+0000 7f99c37fe700 1 -- 172.21.15.52:0/1578593677 <== mgr.14120 v2:172.21.15.52:6800/1631683163 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+24 (secure 0 0 0) 0x7f99cc1050e0 con 0x7f99b803e470 2024-05-27T07:14:17.854 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:14:17.303+0000 7f99d440e700 1 -- 172.21.15.52:0/1578593677 >> [v2:172.21.15.52:6800/1631683163,v1:172.21.15.52:6801/1631683163] conn(0x7f99b803e470 msgr2=0x7f99b8040980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:14:17.854 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:14:17.303+0000 7f99d440e700 1 --2- 172.21.15.52:0/1578593677 >> [v2:172.21.15.52:6800/1631683163,v1:172.21.15.52:6801/1631683163] conn(0x7f99b803e470 0x7f99b8040980 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7f99bc006fd0 tx=0x7f99bc006e40 comp rx=0 tx=0).stop 2024-05-27T07:14:17.854 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:14:17.303+0000 7f99d440e700 1 -- 172.21.15.52:0/1578593677 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f99cc10a660 msgr2=0x7f99cc19dee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:14:17.854 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:14:17.303+0000 7f99d440e700 1 --2- 172.21.15.52:0/1578593677 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f99cc10a660 0x7f99cc19dee0 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f99c80119f0 tx=0x7f99c8011ad0 comp rx=0 tx=0).stop 2024-05-27T07:14:17.855 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:14:17.303+0000 7f99d440e700 1 -- 172.21.15.52:0/1578593677 shutdown_connections 2024-05-27T07:14:17.855 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:14:17.303+0000 7f99d440e700 1 --2- 172.21.15.52:0/1578593677 >> [v2:172.21.15.52:6800/1631683163,v1:172.21.15.52:6801/1631683163] conn(0x7f99b803e470 0x7f99b8040980 unknown :-1 s=CLOSED pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:14:17.855 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:14:17.303+0000 7f99d440e700 1 --2- 172.21.15.52:0/1578593677 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f99cc10a660 0x7f99cc19dee0 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:14:17.855 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:14:17.303+0000 7f99d440e700 1 -- 172.21.15.52:0/1578593677 >> 172.21.15.52:0/1578593677 conn(0x7f99cc1040e0 msgr2=0x7f99cc1049d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:14:17.855 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:14:17.304+0000 7f99d440e700 1 -- 172.21.15.52:0/1578593677 shutdown_connections 2024-05-27T07:14:17.855 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:14:17.304+0000 7f99d440e700 1 -- 172.21.15.52:0/1578593677 wait complete. 2024-05-27T07:14:18.565 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:14:18 smithi052 ceph-mon[31955]: from='client.14144 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "unmanaged": true, "target": ["mon-mgr", ""]}]: dispatch 2024-05-27T07:14:18.565 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:14:18 smithi052 ceph-mon[31955]: Saving service mgr spec with placement count:2 2024-05-27T07:14:18.565 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:14:18 smithi052 ceph-mon[31955]: from='mgr.14120 172.21.15.52:0/2490234989' entity='mgr.x' 2024-05-27T07:14:19.300 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:14:18.332+0000 7fa697b09700 1 Processor -- start 2024-05-27T07:14:19.300 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:14:18.332+0000 7fa697b09700 1 -- start start 2024-05-27T07:14:19.300 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:14:18.333+0000 7fa697b09700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fa69007f5d0 0x7fa69007daf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:14:19.301 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:14:18.333+0000 7fa697b09700 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x7fa69007e030 con 0x7fa69007f5d0 2024-05-27T07:14:19.301 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:14:18.333+0000 7fa6958a5700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fa69007f5d0 0x7fa69007daf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:14:19.301 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:14:18.333+0000 7fa6958a5700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fa69007f5d0 0x7fa69007daf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.52:54958/0 (socket says 172.21.15.52:54958) 2024-05-27T07:14:19.301 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:14:18.333+0000 7fa6958a5700 1 -- 172.21.15.52:0/1238842291 learned_addr learned my addr 172.21.15.52:0/1238842291 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-05-27T07:14:19.301 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:14:18.333+0000 7fa6958a5700 1 -- 172.21.15.52:0/1238842291 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa69007e1b0 con 0x7fa69007f5d0 2024-05-27T07:14:19.301 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:14:18.333+0000 7fa6958a5700 1 --2- 172.21.15.52:0/1238842291 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fa69007f5d0 0x7fa69007daf0 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7fa68c0098a0 tx=0x7fa68c009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bef5ebf18ad5db06 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:14:19.301 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:14:18.335+0000 7fa6948a3700 1 -- 172.21.15.52:0/1238842291 <== mon.0 v2:172.21.15.52:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fa68c004160 con 0x7fa69007f5d0 2024-05-27T07:14:19.301 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:14:18.335+0000 7fa6948a3700 1 -- 172.21.15.52:0/1238842291 <== mon.0 v2:172.21.15.52:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fa68c004300 con 0x7fa69007f5d0 2024-05-27T07:14:19.301 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:14:18.335+0000 7fa6948a3700 1 -- 172.21.15.52:0/1238842291 <== mon.0 v2:172.21.15.52:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fa68c011a20 con 0x7fa69007f5d0 2024-05-27T07:14:19.301 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:14:18.335+0000 7fa697b09700 1 -- 172.21.15.52:0/1238842291 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fa69007f5d0 msgr2=0x7fa69007daf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:14:19.301 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:14:18.335+0000 7fa697b09700 1 --2- 172.21.15.52:0/1238842291 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fa69007f5d0 0x7fa69007daf0 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7fa68c0098a0 tx=0x7fa68c009bb0 comp rx=0 tx=0).stop 2024-05-27T07:14:19.301 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:14:18.335+0000 7fa697b09700 1 -- 172.21.15.52:0/1238842291 shutdown_connections 2024-05-27T07:14:19.302 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:14:18.335+0000 7fa697b09700 1 --2- 172.21.15.52:0/1238842291 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fa69007f5d0 0x7fa69007daf0 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:14:19.302 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:14:18.335+0000 7fa697b09700 1 -- 172.21.15.52:0/1238842291 >> 172.21.15.52:0/1238842291 conn(0x7fa690105250 msgr2=0x7fa690107700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:14:19.302 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:14:18.335+0000 7fa697b09700 1 -- 172.21.15.52:0/1238842291 shutdown_connections 2024-05-27T07:14:19.302 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:14:18.335+0000 7fa697b09700 1 -- 172.21.15.52:0/1238842291 wait complete. 2024-05-27T07:14:19.302 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:14:18.335+0000 7fa697b09700 1 Processor -- start 2024-05-27T07:14:19.302 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:14:18.335+0000 7fa697b09700 1 -- start start 2024-05-27T07:14:19.302 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:14:18.337+0000 7fa697b09700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fa69007f5d0 0x7fa6901a0110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:14:19.302 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:14:18.337+0000 7fa697b09700 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x7fa690110150 con 0x7fa69007f5d0 2024-05-27T07:14:19.302 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:14:18.337+0000 7fa6958a5700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fa69007f5d0 0x7fa6901a0110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:14:19.302 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:14:18.337+0000 7fa6958a5700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fa69007f5d0 0x7fa6901a0110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.52:54966/0 (socket says 172.21.15.52:54966) 2024-05-27T07:14:19.302 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:14:18.337+0000 7fa6958a5700 1 -- 172.21.15.52:0/788407650 learned_addr learned my addr 172.21.15.52:0/788407650 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-05-27T07:14:19.302 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:14:18.337+0000 7fa6958a5700 1 -- 172.21.15.52:0/788407650 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa6901a0650 con 0x7fa69007f5d0 2024-05-27T07:14:19.303 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:14:18.337+0000 7fa6958a5700 1 --2- 172.21.15.52:0/788407650 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fa69007f5d0 0x7fa6901a0110 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7fa68c000c00 tx=0x7fa68c011aa0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:14:19.303 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:14:18.337+0000 7fa686ffd700 1 -- 172.21.15.52:0/788407650 <== mon.0 v2:172.21.15.52:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fa68c003aa0 con 0x7fa69007f5d0 2024-05-27T07:14:19.303 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:14:18.337+0000 7fa686ffd700 1 -- 172.21.15.52:0/788407650 <== mon.0 v2:172.21.15.52:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fa68c01b8f0 con 0x7fa69007f5d0 2024-05-27T07:14:19.303 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:14:18.337+0000 7fa686ffd700 1 -- 172.21.15.52:0/788407650 <== mon.0 v2:172.21.15.52:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fa68c023890 con 0x7fa69007f5d0 2024-05-27T07:14:19.303 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:14:18.337+0000 7fa697b09700 1 -- 172.21.15.52:0/788407650 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa6901a08e0 con 0x7fa69007f5d0 2024-05-27T07:14:19.303 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:14:18.337+0000 7fa697b09700 1 -- 172.21.15.52:0/788407650 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa6901a34f0 con 0x7fa69007f5d0 2024-05-27T07:14:19.303 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:14:18.337+0000 7fa686ffd700 1 -- 172.21.15.52:0/788407650 <== mon.0 v2:172.21.15.52:3300/0 4 ==== mgrmap(e 8) ==== 50222+0+0 (secure 0 0 0) 0x7fa68c02c3f0 con 0x7fa69007f5d0 2024-05-27T07:14:19.303 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:14:18.339+0000 7fa697b09700 1 -- 172.21.15.52:0/788407650 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa690064e10 con 0x7fa69007f5d0 2024-05-27T07:14:19.303 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:14:18.339+0000 7fa686ffd700 1 --2- 172.21.15.52:0/788407650 >> [v2:172.21.15.52:6800/1631683163,v1:172.21.15.52:6801/1631683163] conn(0x7fa67c03e4c0 0x7fa67c0409d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:14:19.303 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:14:18.339+0000 7fa686ffd700 1 -- 172.21.15.52:0/788407650 <== mon.0 v2:172.21.15.52:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 940+0+0 (secure 0 0 0) 0x7fa68c053df0 con 0x7fa69007f5d0 2024-05-27T07:14:19.303 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:14:18.339+0000 7fa6950a4700 1 --2- 172.21.15.52:0/788407650 >> [v2:172.21.15.52:6800/1631683163,v1:172.21.15.52:6801/1631683163] conn(0x7fa67c03e4c0 0x7fa67c0409d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:14:19.303 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:14:18.339+0000 7fa6950a4700 1 --2- 172.21.15.52:0/788407650 >> [v2:172.21.15.52:6800/1631683163,v1:172.21.15.52:6801/1631683163] conn(0x7fa67c03e4c0 0x7fa67c0409d0 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7fa680006fd0 tx=0x7fa680006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:14:19.304 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:14:18.345+0000 7fa686ffd700 1 -- 172.21.15.52:0/788407650 <== mon.0 v2:172.21.15.52:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191963 (secure 0 0 0) 0x7fa68c01f030 con 0x7fa69007f5d0 2024-05-27T07:14:19.304 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:14:18.574+0000 7fa697b09700 1 -- 172.21.15.52:0/788407650 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_command([{prefix=config set, name=mgr/cephadm/container_init}] v 0) -- 0x7fa6901a3bc0 con 0x7fa69007f5d0 2024-05-27T07:14:19.304 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:14:18.577+0000 7fa686ffd700 1 -- 172.21.15.52:0/788407650 <== mon.0 v2:172.21.15.52:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/cephadm/container_init}]=0 v7)=0 v7) ==== 142+0+0 (secure 0 0 0) 0x7fa68c017e90 con 0x7fa69007f5d0 2024-05-27T07:14:19.304 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:14:18.580+0000 7fa697b09700 1 -- 172.21.15.52:0/788407650 >> [v2:172.21.15.52:6800/1631683163,v1:172.21.15.52:6801/1631683163] conn(0x7fa67c03e4c0 msgr2=0x7fa67c0409d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:14:19.304 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:14:18.580+0000 7fa697b09700 1 --2- 172.21.15.52:0/788407650 >> [v2:172.21.15.52:6800/1631683163,v1:172.21.15.52:6801/1631683163] conn(0x7fa67c03e4c0 0x7fa67c0409d0 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7fa680006fd0 tx=0x7fa680006e40 comp rx=0 tx=0).stop 2024-05-27T07:14:19.304 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:14:18.580+0000 7fa697b09700 1 -- 172.21.15.52:0/788407650 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fa69007f5d0 msgr2=0x7fa6901a0110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:14:19.304 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:14:18.580+0000 7fa697b09700 1 --2- 172.21.15.52:0/788407650 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fa69007f5d0 0x7fa6901a0110 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7fa68c000c00 tx=0x7fa68c011aa0 comp rx=0 tx=0).stop 2024-05-27T07:14:19.304 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:14:18.580+0000 7fa697b09700 1 -- 172.21.15.52:0/788407650 shutdown_connections 2024-05-27T07:14:19.304 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:14:18.580+0000 7fa697b09700 1 --2- 172.21.15.52:0/788407650 >> [v2:172.21.15.52:6800/1631683163,v1:172.21.15.52:6801/1631683163] conn(0x7fa67c03e4c0 0x7fa67c0409d0 unknown :-1 s=CLOSED pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:14:19.304 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:14:18.580+0000 7fa697b09700 1 --2- 172.21.15.52:0/788407650 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fa69007f5d0 0x7fa6901a0110 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:14:19.304 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:14:18.580+0000 7fa697b09700 1 -- 172.21.15.52:0/788407650 >> 172.21.15.52:0/788407650 conn(0x7fa690105250 msgr2=0x7fa690105b30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:14:19.304 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:14:18.580+0000 7fa697b09700 1 -- 172.21.15.52:0/788407650 shutdown_connections 2024-05-27T07:14:19.304 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:14:18.580+0000 7fa697b09700 1 -- 172.21.15.52:0/788407650 wait complete. 2024-05-27T07:14:19.837 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:14:19 smithi052 ceph-mon[31955]: from='client.? 172.21.15.52:0/788407650' entity='client.admin' 2024-05-27T07:14:20.843 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:14:20 smithi052 ceph-mon[31955]: from='mgr.14120 172.21.15.52:0/2490234989' entity='mgr.x' 2024-05-27T07:14:20.844 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:14:20 smithi052 ceph-mon[31955]: from='client.? 172.21.15.52:0/1727533575' entity='client.admin' 2024-05-27T07:14:20.851 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:14:20.043+0000 7f2d8d89f700 1 Processor -- start 2024-05-27T07:14:20.851 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:14:20.043+0000 7f2d8d89f700 1 -- start start 2024-05-27T07:14:20.851 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:14:20.043+0000 7f2d8d89f700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f2d8810a660 0x7f2d8810ca80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:14:20.851 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:14:20.043+0000 7f2d8d89f700 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x7f2d8807cde0 con 0x7f2d8810a660 2024-05-27T07:14:20.852 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:14:20.043+0000 7f2d86ffd700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f2d8810a660 0x7f2d8810ca80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:14:20.852 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:14:20.043+0000 7f2d86ffd700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f2d8810a660 0x7f2d8810ca80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.52:54982/0 (socket says 172.21.15.52:54982) 2024-05-27T07:14:20.852 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:14:20.043+0000 7f2d86ffd700 1 -- 172.21.15.52:0/228174361 learned_addr learned my addr 172.21.15.52:0/228174361 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-05-27T07:14:20.852 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:14:20.044+0000 7f2d86ffd700 1 -- 172.21.15.52:0/228174361 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2d8810cfc0 con 0x7f2d8810a660 2024-05-27T07:14:20.852 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:14:20.044+0000 7f2d86ffd700 1 --2- 172.21.15.52:0/228174361 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f2d8810a660 0x7f2d8810ca80 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f2d780098a0 tx=0x7f2d78009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=753a42f5331cbbf5 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:14:20.852 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:14:20.044+0000 7f2d85ffb700 1 -- 172.21.15.52:0/228174361 <== mon.0 v2:172.21.15.52:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f2d78004160 con 0x7f2d8810a660 2024-05-27T07:14:20.852 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:14:20.044+0000 7f2d85ffb700 1 -- 172.21.15.52:0/228174361 <== mon.0 v2:172.21.15.52:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f2d78004300 con 0x7f2d8810a660 2024-05-27T07:14:20.852 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:14:20.044+0000 7f2d85ffb700 1 -- 172.21.15.52:0/228174361 <== mon.0 v2:172.21.15.52:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f2d78011a20 con 0x7f2d8810a660 2024-05-27T07:14:20.852 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:14:20.045+0000 7f2d8d89f700 1 -- 172.21.15.52:0/228174361 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f2d8810a660 msgr2=0x7f2d8810ca80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:14:20.852 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:14:20.045+0000 7f2d8d89f700 1 --2- 172.21.15.52:0/228174361 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f2d8810a660 0x7f2d8810ca80 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f2d780098a0 tx=0x7f2d78009bb0 comp rx=0 tx=0).stop 2024-05-27T07:14:20.853 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:14:20.045+0000 7f2d8d89f700 1 -- 172.21.15.52:0/228174361 shutdown_connections 2024-05-27T07:14:20.853 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:14:20.045+0000 7f2d8d89f700 1 --2- 172.21.15.52:0/228174361 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f2d8810a660 0x7f2d8810ca80 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:14:20.853 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:14:20.045+0000 7f2d8d89f700 1 -- 172.21.15.52:0/228174361 >> 172.21.15.52:0/228174361 conn(0x7f2d881040e0 msgr2=0x7f2d88106570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:14:20.853 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:14:20.045+0000 7f2d8d89f700 1 -- 172.21.15.52:0/228174361 shutdown_connections 2024-05-27T07:14:20.853 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:14:20.045+0000 7f2d8d89f700 1 -- 172.21.15.52:0/228174361 wait complete. 2024-05-27T07:14:20.853 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:14:20.046+0000 7f2d8d89f700 1 Processor -- start 2024-05-27T07:14:20.853 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:14:20.046+0000 7f2d8d89f700 1 -- start start 2024-05-27T07:14:20.853 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:14:20.046+0000 7f2d8d89f700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f2d8810a660 0x7f2d8819dee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:14:20.853 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:14:20.046+0000 7f2d8d89f700 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x7f2d8810df00 con 0x7f2d8810a660 2024-05-27T07:14:20.853 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:14:20.046+0000 7f2d86ffd700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f2d8810a660 0x7f2d8819dee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:14:20.853 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:14:20.046+0000 7f2d86ffd700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f2d8810a660 0x7f2d8819dee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.52:54998/0 (socket says 172.21.15.52:54998) 2024-05-27T07:14:20.853 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:14:20.046+0000 7f2d86ffd700 1 -- 172.21.15.52:0/1727533575 learned_addr learned my addr 172.21.15.52:0/1727533575 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-05-27T07:14:20.853 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:14:20.047+0000 7f2d86ffd700 1 -- 172.21.15.52:0/1727533575 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2d8819e420 con 0x7f2d8810a660 2024-05-27T07:14:20.854 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:14:20.047+0000 7f2d86ffd700 1 --2- 172.21.15.52:0/1727533575 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f2d8810a660 0x7f2d8819dee0 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f2d78000c00 tx=0x7f2d78011aa0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:14:20.854 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:14:20.047+0000 7f2d8c89d700 1 -- 172.21.15.52:0/1727533575 <== mon.0 v2:172.21.15.52:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f2d78003aa0 con 0x7f2d8810a660 2024-05-27T07:14:20.854 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:14:20.047+0000 7f2d8c89d700 1 -- 172.21.15.52:0/1727533575 <== mon.0 v2:172.21.15.52:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f2d7801b8f0 con 0x7f2d8810a660 2024-05-27T07:14:20.854 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:14:20.047+0000 7f2d8d89f700 1 -- 172.21.15.52:0/1727533575 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2d8819e6b0 con 0x7f2d8810a660 2024-05-27T07:14:20.854 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:14:20.047+0000 7f2d8c89d700 1 -- 172.21.15.52:0/1727533575 <== mon.0 v2:172.21.15.52:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f2d78023890 con 0x7f2d8810a660 2024-05-27T07:14:20.854 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:14:20.047+0000 7f2d8d89f700 1 -- 172.21.15.52:0/1727533575 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2d881a12c0 con 0x7f2d8810a660 2024-05-27T07:14:20.854 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:14:20.048+0000 7f2d8d89f700 1 -- 172.21.15.52:0/1727533575 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2d88064e10 con 0x7f2d8810a660 2024-05-27T07:14:20.854 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:14:20.048+0000 7f2d8c89d700 1 -- 172.21.15.52:0/1727533575 <== mon.0 v2:172.21.15.52:3300/0 4 ==== mgrmap(e 8) ==== 50222+0+0 (secure 0 0 0) 0x7f2d7802c3f0 con 0x7f2d8810a660 2024-05-27T07:14:20.854 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:14:20.049+0000 7f2d8c89d700 1 --2- 172.21.15.52:0/1727533575 >> [v2:172.21.15.52:6800/1631683163,v1:172.21.15.52:6801/1631683163] conn(0x7f2d7403e470 0x7f2d74040980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:14:20.854 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:14:20.049+0000 7f2d8c89d700 1 -- 172.21.15.52:0/1727533575 <== mon.0 v2:172.21.15.52:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 940+0+0 (secure 0 0 0) 0x7f2d78053dc0 con 0x7f2d8810a660 2024-05-27T07:14:20.855 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:14:20.049+0000 7f2d867fc700 1 --2- 172.21.15.52:0/1727533575 >> [v2:172.21.15.52:6800/1631683163,v1:172.21.15.52:6801/1631683163] conn(0x7f2d7403e470 0x7f2d74040980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:14:20.855 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:14:20.049+0000 7f2d867fc700 1 --2- 172.21.15.52:0/1727533575 >> [v2:172.21.15.52:6800/1631683163,v1:172.21.15.52:6801/1631683163] conn(0x7f2d7403e470 0x7f2d74040980 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7f2d70006fd0 tx=0x7f2d70006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:14:20.855 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:14:20.054+0000 7f2d8c89d700 1 -- 172.21.15.52:0/1727533575 <== mon.0 v2:172.21.15.52:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191963 (secure 0 0 0) 0x7f2d7801f030 con 0x7f2d8810a660 2024-05-27T07:14:20.855 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:14:20.273+0000 7f2d8d89f700 1 -- 172.21.15.52:0/1727533575 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_command([{prefix=config set, name=mgr/dashboard/ssl_server_port}] v 0) -- 0x7f2d881a15b0 con 0x7f2d8810a660 2024-05-27T07:14:20.855 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:14:20.276+0000 7f2d8c89d700 1 -- 172.21.15.52:0/1727533575 <== mon.0 v2:172.21.15.52:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/dashboard/ssl_server_port}]=0 v8)=0 v8) ==== 130+0+0 (secure 0 0 0) 0x7f2d7801bda0 con 0x7f2d8810a660 2024-05-27T07:14:20.855 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:14:20.278+0000 7f2d8d89f700 1 -- 172.21.15.52:0/1727533575 >> [v2:172.21.15.52:6800/1631683163,v1:172.21.15.52:6801/1631683163] conn(0x7f2d7403e470 msgr2=0x7f2d74040980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:14:20.855 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:14:20.278+0000 7f2d8d89f700 1 --2- 172.21.15.52:0/1727533575 >> [v2:172.21.15.52:6800/1631683163,v1:172.21.15.52:6801/1631683163] conn(0x7f2d7403e470 0x7f2d74040980 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7f2d70006fd0 tx=0x7f2d70006e40 comp rx=0 tx=0).stop 2024-05-27T07:14:20.855 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:14:20.278+0000 7f2d8d89f700 1 -- 172.21.15.52:0/1727533575 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f2d8810a660 msgr2=0x7f2d8819dee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:14:20.855 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:14:20.278+0000 7f2d8d89f700 1 --2- 172.21.15.52:0/1727533575 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f2d8810a660 0x7f2d8819dee0 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f2d78000c00 tx=0x7f2d78011aa0 comp rx=0 tx=0).stop 2024-05-27T07:14:20.855 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:14:20.279+0000 7f2d8d89f700 1 -- 172.21.15.52:0/1727533575 shutdown_connections 2024-05-27T07:14:20.855 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:14:20.279+0000 7f2d8d89f700 1 --2- 172.21.15.52:0/1727533575 >> [v2:172.21.15.52:6800/1631683163,v1:172.21.15.52:6801/1631683163] conn(0x7f2d7403e470 0x7f2d74040980 unknown :-1 s=CLOSED pgs=15 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:14:20.855 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:14:20.279+0000 7f2d8d89f700 1 --2- 172.21.15.52:0/1727533575 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f2d8810a660 0x7f2d8819dee0 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:14:20.855 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:14:20.279+0000 7f2d8d89f700 1 -- 172.21.15.52:0/1727533575 >> 172.21.15.52:0/1727533575 conn(0x7f2d881040e0 msgr2=0x7f2d881049d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:14:20.855 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:14:20.279+0000 7f2d8d89f700 1 -- 172.21.15.52:0/1727533575 shutdown_connections 2024-05-27T07:14:20.856 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:14:20.279+0000 7f2d8d89f700 1 -- 172.21.15.52:0/1727533575 wait complete. 2024-05-27T07:14:20.856 INFO:teuthology.orchestra.run.smithi052.stdout:Enabling the dashboard module... 2024-05-27T07:14:22.230 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:14:21 smithi052 ceph-mon[31955]: from='mgr.14120 172.21.15.52:0/2490234989' entity='mgr.x' 2024-05-27T07:14:22.230 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:14:21 smithi052 ceph-mon[31955]: from='client.? 172.21.15.52:0/1970322495' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "dashboard"}]: dispatch 2024-05-27T07:14:22.230 INFO:journalctl@ceph.mgr.x.smithi052.stdout:May 27 07:14:22 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mgr-x[32184]: ignoring --setuser ceph since I am not root 2024-05-27T07:14:22.230 INFO:journalctl@ceph.mgr.x.smithi052.stdout:May 27 07:14:22 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mgr-x[32184]: ignoring --setgroup ceph since I am not root 2024-05-27T07:14:22.230 INFO:journalctl@ceph.mgr.x.smithi052.stdout:May 27 07:14:22 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mgr-x[32184]: 2024-05-27T07:14:22.172+0000 7f8ad4168200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-05-27T07:14:22.538 INFO:journalctl@ceph.mgr.x.smithi052.stdout:May 27 07:14:22 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mgr-x[32184]: 2024-05-27T07:14:22.370+0000 7f8ad4168200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-05-27T07:14:22.538 INFO:journalctl@ceph.mgr.x.smithi052.stdout:May 27 07:14:22 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mgr-x[32184]: 2024-05-27T07:14:22.478+0000 7f8ad4168200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-05-27T07:14:22.545 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:14:21.389+0000 7fa19d165700 1 Processor -- start 2024-05-27T07:14:22.545 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:14:21.389+0000 7fa19d165700 1 -- start start 2024-05-27T07:14:22.545 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:14:21.389+0000 7fa19d165700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fa19810a660 0x7fa19810ca80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:14:22.545 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:14:21.389+0000 7fa19d165700 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x7fa19807cde0 con 0x7fa19810a660 2024-05-27T07:14:22.545 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:14:21.389+0000 7fa196d9d700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fa19810a660 0x7fa19810ca80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:14:22.546 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:14:21.390+0000 7fa196d9d700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fa19810a660 0x7fa19810ca80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.52:44534/0 (socket says 172.21.15.52:44534) 2024-05-27T07:14:22.546 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:14:21.390+0000 7fa196d9d700 1 -- 172.21.15.52:0/3318305149 learned_addr learned my addr 172.21.15.52:0/3318305149 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-05-27T07:14:22.546 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:14:21.390+0000 7fa196d9d700 1 -- 172.21.15.52:0/3318305149 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa19810cfc0 con 0x7fa19810a660 2024-05-27T07:14:22.546 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:14:21.390+0000 7fa196d9d700 1 --2- 172.21.15.52:0/3318305149 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fa19810a660 0x7fa19810ca80 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7fa1880098a0 tx=0x7fa188009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cd5d8cdcbc22f061 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:14:22.546 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:14:21.390+0000 7fa195d9b700 1 -- 172.21.15.52:0/3318305149 <== mon.0 v2:172.21.15.52:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fa188004160 con 0x7fa19810a660 2024-05-27T07:14:22.546 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:14:21.390+0000 7fa195d9b700 1 -- 172.21.15.52:0/3318305149 <== mon.0 v2:172.21.15.52:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fa188004300 con 0x7fa19810a660 2024-05-27T07:14:22.546 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:14:21.390+0000 7fa195d9b700 1 -- 172.21.15.52:0/3318305149 <== mon.0 v2:172.21.15.52:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fa188011a20 con 0x7fa19810a660 2024-05-27T07:14:22.546 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:14:21.391+0000 7fa19d165700 1 -- 172.21.15.52:0/3318305149 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fa19810a660 msgr2=0x7fa19810ca80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:14:22.546 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:14:21.391+0000 7fa19d165700 1 --2- 172.21.15.52:0/3318305149 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fa19810a660 0x7fa19810ca80 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7fa1880098a0 tx=0x7fa188009bb0 comp rx=0 tx=0).stop 2024-05-27T07:14:22.546 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:14:21.391+0000 7fa19d165700 1 -- 172.21.15.52:0/3318305149 shutdown_connections 2024-05-27T07:14:22.546 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:14:21.391+0000 7fa19d165700 1 --2- 172.21.15.52:0/3318305149 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fa19810a660 0x7fa19810ca80 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:14:22.547 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:14:21.391+0000 7fa19d165700 1 -- 172.21.15.52:0/3318305149 >> 172.21.15.52:0/3318305149 conn(0x7fa1981040e0 msgr2=0x7fa198106570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:14:22.547 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:14:21.391+0000 7fa19d165700 1 -- 172.21.15.52:0/3318305149 shutdown_connections 2024-05-27T07:14:22.547 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:14:21.391+0000 7fa19d165700 1 -- 172.21.15.52:0/3318305149 wait complete. 2024-05-27T07:14:22.547 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:14:21.392+0000 7fa19d165700 1 Processor -- start 2024-05-27T07:14:22.547 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:14:21.392+0000 7fa19d165700 1 -- start start 2024-05-27T07:14:22.547 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:14:21.393+0000 7fa19d165700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fa19810a660 0x7fa19819dee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:14:22.547 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:14:21.393+0000 7fa19d165700 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x7fa19810df00 con 0x7fa19810a660 2024-05-27T07:14:22.547 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:14:21.393+0000 7fa196d9d700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fa19810a660 0x7fa19819dee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:14:22.547 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:14:21.393+0000 7fa196d9d700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fa19810a660 0x7fa19819dee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.52:44540/0 (socket says 172.21.15.52:44540) 2024-05-27T07:14:22.547 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:14:21.393+0000 7fa196d9d700 1 -- 172.21.15.52:0/1970322495 learned_addr learned my addr 172.21.15.52:0/1970322495 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-05-27T07:14:22.547 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:14:21.393+0000 7fa196d9d700 1 -- 172.21.15.52:0/1970322495 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa19819e420 con 0x7fa19810a660 2024-05-27T07:14:22.547 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:14:21.393+0000 7fa196d9d700 1 --2- 172.21.15.52:0/1970322495 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fa19810a660 0x7fa19819dee0 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7fa188000c00 tx=0x7fa188011aa0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:14:22.547 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:14:21.394+0000 7fa17ffff700 1 -- 172.21.15.52:0/1970322495 <== mon.0 v2:172.21.15.52:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fa188003aa0 con 0x7fa19810a660 2024-05-27T07:14:22.547 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:14:21.394+0000 7fa19d165700 1 -- 172.21.15.52:0/1970322495 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa19819e6b0 con 0x7fa19810a660 2024-05-27T07:14:22.548 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:14:21.394+0000 7fa17ffff700 1 -- 172.21.15.52:0/1970322495 <== mon.0 v2:172.21.15.52:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fa18801b8f0 con 0x7fa19810a660 2024-05-27T07:14:22.548 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:14:21.394+0000 7fa17ffff700 1 -- 172.21.15.52:0/1970322495 <== mon.0 v2:172.21.15.52:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fa188023890 con 0x7fa19810a660 2024-05-27T07:14:22.548 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:14:21.394+0000 7fa19d165700 1 -- 172.21.15.52:0/1970322495 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa1981a12c0 con 0x7fa19810a660 2024-05-27T07:14:22.548 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:14:21.395+0000 7fa17ffff700 1 -- 172.21.15.52:0/1970322495 <== mon.0 v2:172.21.15.52:3300/0 4 ==== mgrmap(e 8) ==== 50222+0+0 (secure 0 0 0) 0x7fa18802c3f0 con 0x7fa19810a660 2024-05-27T07:14:22.548 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:14:21.395+0000 7fa19d165700 1 -- 172.21.15.52:0/1970322495 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa198064e10 con 0x7fa19810a660 2024-05-27T07:14:22.548 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:14:21.395+0000 7fa17ffff700 1 --2- 172.21.15.52:0/1970322495 >> [v2:172.21.15.52:6800/1631683163,v1:172.21.15.52:6801/1631683163] conn(0x7fa18003e4c0 0x7fa1800409d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:14:22.548 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:14:21.395+0000 7fa17ffff700 1 -- 172.21.15.52:0/1970322495 <== mon.0 v2:172.21.15.52:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 940+0+0 (secure 0 0 0) 0x7fa188053da0 con 0x7fa19810a660 2024-05-27T07:14:22.548 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:14:21.395+0000 7fa19659c700 1 --2- 172.21.15.52:0/1970322495 >> [v2:172.21.15.52:6800/1631683163,v1:172.21.15.52:6801/1631683163] conn(0x7fa18003e4c0 0x7fa1800409d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:14:22.548 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:14:21.396+0000 7fa19659c700 1 --2- 172.21.15.52:0/1970322495 >> [v2:172.21.15.52:6800/1631683163,v1:172.21.15.52:6801/1631683163] conn(0x7fa18003e4c0 0x7fa1800409d0 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7fa18c006fd0 tx=0x7fa18c006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:14:22.548 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:14:21.401+0000 7fa17ffff700 1 -- 172.21.15.52:0/1970322495 <== mon.0 v2:172.21.15.52:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191963 (secure 0 0 0) 0x7fa18801f030 con 0x7fa19810a660 2024-05-27T07:14:22.548 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:14:21.690+0000 7fa19d165700 1 -- 172.21.15.52:0/1970322495 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_command({"prefix": "mgr module enable", "module": "dashboard"} v 0) -- 0x7fa1981a1900 con 0x7fa19810a660 2024-05-27T07:14:22.548 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:14:21.955+0000 7fa17ffff700 1 -- 172.21.15.52:0/1970322495 <== mon.0 v2:172.21.15.52:3300/0 7 ==== mon_command_ack([{"prefix": "mgr module enable", "module": "dashboard"}]=0 v9) ==== 88+0+0 (secure 0 0 0) 0x7fa18800f2e0 con 0x7fa19810a660 2024-05-27T07:14:22.548 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:14:21.957+0000 7fa17ffff700 1 -- 172.21.15.52:0/1970322495 <== mon.0 v2:172.21.15.52:3300/0 8 ==== mgrmap(e 9) ==== 50235+0+0 (secure 0 0 0) 0x7fa188016bd0 con 0x7fa19810a660 2024-05-27T07:14:22.549 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:14:21.958+0000 7fa19d165700 1 -- 172.21.15.52:0/1970322495 >> [v2:172.21.15.52:6800/1631683163,v1:172.21.15.52:6801/1631683163] conn(0x7fa18003e4c0 msgr2=0x7fa1800409d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:14:22.549 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:14:21.958+0000 7fa19d165700 1 --2- 172.21.15.52:0/1970322495 >> [v2:172.21.15.52:6800/1631683163,v1:172.21.15.52:6801/1631683163] conn(0x7fa18003e4c0 0x7fa1800409d0 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7fa18c006fd0 tx=0x7fa18c006e40 comp rx=0 tx=0).stop 2024-05-27T07:14:22.549 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:14:21.958+0000 7fa19d165700 1 -- 172.21.15.52:0/1970322495 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fa19810a660 msgr2=0x7fa19819dee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:14:22.549 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:14:21.958+0000 7fa19d165700 1 --2- 172.21.15.52:0/1970322495 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fa19810a660 0x7fa19819dee0 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7fa188000c00 tx=0x7fa188011aa0 comp rx=0 tx=0).stop 2024-05-27T07:14:22.549 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:14:21.958+0000 7fa19d165700 1 -- 172.21.15.52:0/1970322495 shutdown_connections 2024-05-27T07:14:22.549 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:14:21.958+0000 7fa19d165700 1 --2- 172.21.15.52:0/1970322495 >> [v2:172.21.15.52:6800/1631683163,v1:172.21.15.52:6801/1631683163] conn(0x7fa18003e4c0 0x7fa1800409d0 unknown :-1 s=CLOSED pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:14:22.549 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:14:21.958+0000 7fa19d165700 1 --2- 172.21.15.52:0/1970322495 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fa19810a660 0x7fa19819dee0 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:14:22.549 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:14:21.958+0000 7fa19d165700 1 -- 172.21.15.52:0/1970322495 >> 172.21.15.52:0/1970322495 conn(0x7fa1981040e0 msgr2=0x7fa1981049d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:14:22.549 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:14:21.958+0000 7fa19d165700 1 -- 172.21.15.52:0/1970322495 shutdown_connections 2024-05-27T07:14:22.549 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:14:21.958+0000 7fa19d165700 1 -- 172.21.15.52:0/1970322495 wait complete. 2024-05-27T07:14:23.082 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:14:22 smithi052 ceph-mon[31955]: from='client.? 172.21.15.52:0/1970322495' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "dashboard"}]': finished 2024-05-27T07:14:23.083 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:14:22 smithi052 ceph-mon[31955]: mgrmap e9: x(active, since 19s) 2024-05-27T07:14:23.083 INFO:journalctl@ceph.mgr.x.smithi052.stdout:May 27 07:14:22 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mgr-x[32184]: 2024-05-27T07:14:22.807+0000 7f8ad4168200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-05-27T07:14:23.083 INFO:journalctl@ceph.mgr.x.smithi052.stdout:May 27 07:14:22 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mgr-x[32184]: 2024-05-27T07:14:22.900+0000 7f8ad4168200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-05-27T07:14:24.230 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:14:23 smithi052 ceph-mon[31955]: from='client.? 172.21.15.52:0/2211208749' entity='client.admin' cmd=[{"prefix": "mgr stat"}]: dispatch 2024-05-27T07:14:24.230 INFO:journalctl@ceph.mgr.x.smithi052.stdout:May 27 07:14:24 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mgr-x[32184]: 2024-05-27T07:14:24.195+0000 7f8ad4168200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-05-27T07:14:24.300 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout { 2024-05-27T07:14:24.301 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout "epoch": 9, 2024-05-27T07:14:24.301 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout "available": true, 2024-05-27T07:14:24.301 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout "active_name": "x", 2024-05-27T07:14:24.301 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout "num_standby": 0 2024-05-27T07:14:24.301 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout } 2024-05-27T07:14:24.301 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:14:23.456+0000 7f4951562700 1 Processor -- start 2024-05-27T07:14:24.301 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:14:23.456+0000 7f4951562700 1 -- start start 2024-05-27T07:14:24.301 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:14:23.456+0000 7f4951562700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f494c07f600 0x7f494c07dad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:14:24.301 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:14:23.456+0000 7f4951562700 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x7f494c07e010 con 0x7f494c07f600 2024-05-27T07:14:24.301 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:14:23.457+0000 7f494affd700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f494c07f600 0x7f494c07dad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:14:24.301 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:14:23.457+0000 7f494affd700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f494c07f600 0x7f494c07dad0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.52:44558/0 (socket says 172.21.15.52:44558) 2024-05-27T07:14:24.301 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:14:23.457+0000 7f494affd700 1 -- 172.21.15.52:0/2132988595 learned_addr learned my addr 172.21.15.52:0/2132988595 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-05-27T07:14:24.302 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:14:23.457+0000 7f494affd700 1 -- 172.21.15.52:0/2132988595 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f494c07e190 con 0x7f494c07f600 2024-05-27T07:14:24.302 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:14:23.457+0000 7f494affd700 1 --2- 172.21.15.52:0/2132988595 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f494c07f600 0x7f494c07dad0 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f493c0098a0 tx=0x7f493c009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1d399db56e5bd698 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:14:24.302 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:14:23.457+0000 7f4949ffb700 1 -- 172.21.15.52:0/2132988595 <== mon.0 v2:172.21.15.52:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f493c004160 con 0x7f494c07f600 2024-05-27T07:14:24.302 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:14:23.457+0000 7f4949ffb700 1 -- 172.21.15.52:0/2132988595 <== mon.0 v2:172.21.15.52:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f493c004300 con 0x7f494c07f600 2024-05-27T07:14:24.302 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:14:23.457+0000 7f4949ffb700 1 -- 172.21.15.52:0/2132988595 <== mon.0 v2:172.21.15.52:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f493c011a20 con 0x7f494c07f600 2024-05-27T07:14:24.302 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:14:23.458+0000 7f4951562700 1 -- 172.21.15.52:0/2132988595 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f494c07f600 msgr2=0x7f494c07dad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:14:24.302 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:14:23.458+0000 7f4951562700 1 --2- 172.21.15.52:0/2132988595 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f494c07f600 0x7f494c07dad0 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f493c0098a0 tx=0x7f493c009bb0 comp rx=0 tx=0).stop 2024-05-27T07:14:24.302 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:14:23.458+0000 7f4951562700 1 -- 172.21.15.52:0/2132988595 shutdown_connections 2024-05-27T07:14:24.302 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:14:23.458+0000 7f4951562700 1 --2- 172.21.15.52:0/2132988595 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f494c07f600 0x7f494c07dad0 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:14:24.302 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:14:23.458+0000 7f4951562700 1 -- 172.21.15.52:0/2132988595 >> 172.21.15.52:0/2132988595 conn(0x7f494c1052d0 msgr2=0x7f494c107720 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:14:24.302 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:14:23.459+0000 7f4951562700 1 -- 172.21.15.52:0/2132988595 shutdown_connections 2024-05-27T07:14:24.302 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:14:23.459+0000 7f4951562700 1 -- 172.21.15.52:0/2132988595 wait complete. 2024-05-27T07:14:24.302 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:14:23.459+0000 7f4951562700 1 Processor -- start 2024-05-27T07:14:24.303 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:14:23.459+0000 7f4951562700 1 -- start start 2024-05-27T07:14:24.303 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:14:23.460+0000 7f4951562700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f494c07f600 0x7f494c1a0150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:14:24.303 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:14:23.460+0000 7f4951562700 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x7f494c1101d0 con 0x7f494c07f600 2024-05-27T07:14:24.303 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:14:23.460+0000 7f494affd700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f494c07f600 0x7f494c1a0150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:14:24.303 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:14:23.460+0000 7f494affd700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f494c07f600 0x7f494c1a0150 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.52:44568/0 (socket says 172.21.15.52:44568) 2024-05-27T07:14:24.303 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:14:23.460+0000 7f494affd700 1 -- 172.21.15.52:0/2211208749 learned_addr learned my addr 172.21.15.52:0/2211208749 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-05-27T07:14:24.303 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:14:23.460+0000 7f494affd700 1 -- 172.21.15.52:0/2211208749 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f494c1a0690 con 0x7f494c07f600 2024-05-27T07:14:24.303 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:14:23.460+0000 7f494affd700 1 --2- 172.21.15.52:0/2211208749 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f494c07f600 0x7f494c1a0150 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f493c004660 tx=0x7f493c011aa0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:14:24.303 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:14:23.461+0000 7f4933fff700 1 -- 172.21.15.52:0/2211208749 <== mon.0 v2:172.21.15.52:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f493c003a60 con 0x7f494c07f600 2024-05-27T07:14:24.303 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:14:23.461+0000 7f4951562700 1 -- 172.21.15.52:0/2211208749 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f494c1a0920 con 0x7f494c07f600 2024-05-27T07:14:24.303 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:14:23.461+0000 7f4933fff700 1 -- 172.21.15.52:0/2211208749 <== mon.0 v2:172.21.15.52:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f493c01b8f0 con 0x7f494c07f600 2024-05-27T07:14:24.303 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:14:23.461+0000 7f4933fff700 1 -- 172.21.15.52:0/2211208749 <== mon.0 v2:172.21.15.52:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f493c023890 con 0x7f494c07f600 2024-05-27T07:14:24.303 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:14:23.461+0000 7f4951562700 1 -- 172.21.15.52:0/2211208749 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f494c1a3530 con 0x7f494c07f600 2024-05-27T07:14:24.304 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:14:23.462+0000 7f4951562700 1 -- 172.21.15.52:0/2211208749 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f494c064e10 con 0x7f494c07f600 2024-05-27T07:14:24.304 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:14:23.462+0000 7f4933fff700 1 -- 172.21.15.52:0/2211208749 <== mon.0 v2:172.21.15.52:3300/0 4 ==== mgrmap(e 9) ==== 50235+0+0 (secure 0 0 0) 0x7f493c02c3f0 con 0x7f494c07f600 2024-05-27T07:14:24.304 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:14:23.463+0000 7f4933fff700 1 --2- 172.21.15.52:0/2211208749 >> [v2:172.21.15.52:6800/1631683163,v1:172.21.15.52:6801/1631683163] conn(0x7f493403e470 0x7f4934040980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:14:24.304 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:14:23.463+0000 7f4933fff700 1 -- 172.21.15.52:0/2211208749 <== mon.0 v2:172.21.15.52:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 940+0+0 (secure 0 0 0) 0x7f493c053fb0 con 0x7f494c07f600 2024-05-27T07:14:24.304 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:14:23.463+0000 7f494a7fc700 1 -- 172.21.15.52:0/2211208749 >> [v2:172.21.15.52:6800/1631683163,v1:172.21.15.52:6801/1631683163] conn(0x7f493403e470 msgr2=0x7f4934040980 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.52:6800/1631683163 2024-05-27T07:14:24.304 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:14:23.463+0000 7f494a7fc700 1 --2- 172.21.15.52:0/2211208749 >> [v2:172.21.15.52:6800/1631683163,v1:172.21.15.52:6801/1631683163] conn(0x7f493403e470 0x7f4934040980 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-05-27T07:14:24.304 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:14:23.468+0000 7f4933fff700 1 -- 172.21.15.52:0/2211208749 <== mon.0 v2:172.21.15.52:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191963 (secure 0 0 0) 0x7f493c01f030 con 0x7f494c07f600 2024-05-27T07:14:24.304 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:14:23.664+0000 7f494a7fc700 1 -- 172.21.15.52:0/2211208749 >> [v2:172.21.15.52:6800/1631683163,v1:172.21.15.52:6801/1631683163] conn(0x7f493403e470 msgr2=0x7f4934040980 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.52:6800/1631683163 2024-05-27T07:14:24.304 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:14:23.664+0000 7f494a7fc700 1 --2- 172.21.15.52:0/2211208749 >> [v2:172.21.15.52:6800/1631683163,v1:172.21.15.52:6801/1631683163] conn(0x7f493403e470 0x7f4934040980 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-05-27T07:14:24.304 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:14:23.756+0000 7f4951562700 1 -- 172.21.15.52:0/2211208749 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_command({"prefix": "mgr stat"} v 0) -- 0x7f494c1a3820 con 0x7f494c07f600 2024-05-27T07:14:24.304 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:14:23.756+0000 7f4933fff700 1 -- 172.21.15.52:0/2211208749 <== mon.0 v2:172.21.15.52:3300/0 7 ==== mon_command_ack([{"prefix": "mgr stat"}]=0 v9) ==== 56+0+88 (secure 0 0 0) 0x7f493c00f2e0 con 0x7f494c07f600 2024-05-27T07:14:24.304 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:14:23.759+0000 7f4951562700 1 -- 172.21.15.52:0/2211208749 >> [v2:172.21.15.52:6800/1631683163,v1:172.21.15.52:6801/1631683163] conn(0x7f493403e470 msgr2=0x7f4934040980 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-05-27T07:14:24.305 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:14:23.759+0000 7f4951562700 1 --2- 172.21.15.52:0/2211208749 >> [v2:172.21.15.52:6800/1631683163,v1:172.21.15.52:6801/1631683163] conn(0x7f493403e470 0x7f4934040980 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:14:24.305 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:14:23.759+0000 7f4951562700 1 -- 172.21.15.52:0/2211208749 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f494c07f600 msgr2=0x7f494c1a0150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:14:24.305 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:14:23.759+0000 7f4951562700 1 --2- 172.21.15.52:0/2211208749 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f494c07f600 0x7f494c1a0150 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f493c004660 tx=0x7f493c011aa0 comp rx=0 tx=0).stop 2024-05-27T07:14:24.305 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:14:23.759+0000 7f4951562700 1 -- 172.21.15.52:0/2211208749 shutdown_connections 2024-05-27T07:14:24.305 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:14:23.759+0000 7f4951562700 1 --2- 172.21.15.52:0/2211208749 >> [v2:172.21.15.52:6800/1631683163,v1:172.21.15.52:6801/1631683163] conn(0x7f493403e470 0x7f4934040980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:14:24.305 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:14:23.759+0000 7f4951562700 1 --2- 172.21.15.52:0/2211208749 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f494c07f600 0x7f494c1a0150 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:14:24.305 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:14:23.759+0000 7f4951562700 1 -- 172.21.15.52:0/2211208749 >> 172.21.15.52:0/2211208749 conn(0x7f494c1052d0 msgr2=0x7f494c106b90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:14:24.305 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:14:23.759+0000 7f4951562700 1 -- 172.21.15.52:0/2211208749 shutdown_connections 2024-05-27T07:14:24.305 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:14:23.759+0000 7f4951562700 1 -- 172.21.15.52:0/2211208749 wait complete. 2024-05-27T07:14:24.305 INFO:teuthology.orchestra.run.smithi052.stdout:Waiting for the mgr to restart... 2024-05-27T07:14:24.305 INFO:teuthology.orchestra.run.smithi052.stdout:Waiting for mgr epoch 9... 2024-05-27T07:14:24.544 INFO:journalctl@ceph.mgr.x.smithi052.stdout:May 27 07:14:24 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mgr-x[32184]: 2024-05-27T07:14:24.391+0000 7f8ad4168200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-05-27T07:14:24.884 INFO:journalctl@ceph.mgr.x.smithi052.stdout:May 27 07:14:24 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mgr-x[32184]: 2024-05-27T07:14:24.578+0000 7f8ad4168200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-05-27T07:14:24.884 INFO:journalctl@ceph.mgr.x.smithi052.stdout:May 27 07:14:24 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mgr-x[32184]: 2024-05-27T07:14:24.892+0000 7f8ad4168200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-05-27T07:14:25.196 INFO:journalctl@ceph.mgr.x.smithi052.stdout:May 27 07:14:24 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mgr-x[32184]: 2024-05-27T07:14:24.992+0000 7f8ad4168200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-05-27T07:14:25.480 INFO:journalctl@ceph.mgr.x.smithi052.stdout:May 27 07:14:25 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mgr-x[32184]: 2024-05-27T07:14:25.205+0000 7f8ad4168200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-05-27T07:14:25.480 INFO:journalctl@ceph.mgr.x.smithi052.stdout:May 27 07:14:25 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mgr-x[32184]: 2024-05-27T07:14:25.411+0000 7f8ad4168200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-05-27T07:14:26.480 INFO:journalctl@ceph.mgr.x.smithi052.stdout:May 27 07:14:26 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mgr-x[32184]: 2024-05-27T07:14:26.046+0000 7f8ad4168200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-05-27T07:14:26.480 INFO:journalctl@ceph.mgr.x.smithi052.stdout:May 27 07:14:26 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mgr-x[32184]: 2024-05-27T07:14:26.283+0000 7f8ad4168200 -1 mgr[py] Module smb has missing NOTIFY_TYPES member 2024-05-27T07:14:27.224 INFO:journalctl@ceph.mgr.x.smithi052.stdout:May 27 07:14:26 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mgr-x[32184]: 2024-05-27T07:14:26.740+0000 7f8ad4168200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-05-27T07:14:27.224 INFO:journalctl@ceph.mgr.x.smithi052.stdout:May 27 07:14:26 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mgr-x[32184]: 2024-05-27T07:14:26.843+0000 7f8ad4168200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-05-27T07:14:27.224 INFO:journalctl@ceph.mgr.x.smithi052.stdout:May 27 07:14:27 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mgr-x[32184]: 2024-05-27T07:14:27.232+0000 7f8ad4168200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-05-27T07:14:27.968 INFO:journalctl@ceph.mgr.x.smithi052.stdout:May 27 07:14:27 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mgr-x[32184]: 2024-05-27T07:14:27.613+0000 7f8ad4168200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-05-27T07:14:27.968 INFO:journalctl@ceph.mgr.x.smithi052.stdout:May 27 07:14:27 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mgr-x[32184]: 2024-05-27T07:14:27.714+0000 7f8ad4168200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-05-27T07:14:27.968 INFO:journalctl@ceph.mgr.x.smithi052.stdout:May 27 07:14:27 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mgr-x[32184]: 2024-05-27T07:14:27.977+0000 7f8ad4168200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-05-27T07:14:28.230 INFO:journalctl@ceph.mgr.x.smithi052.stdout:May 27 07:14:28 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mgr-x[32184]: 2024-05-27T07:14:28.120+0000 7f8ad4168200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-05-27T07:14:28.230 INFO:journalctl@ceph.mgr.x.smithi052.stdout:May 27 07:14:28 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mgr-x[32184]: 2024-05-27T07:14:28.225+0000 7f8ad4168200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-05-27T07:14:28.980 INFO:journalctl@ceph.mgr.x.smithi052.stdout:May 27 07:14:28 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mgr-x[32184]: 2024-05-27T07:14:28.567+0000 7f8ad4168200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-05-27T07:14:28.980 INFO:journalctl@ceph.mgr.x.smithi052.stdout:May 27 07:14:28 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mgr-x[32184]: 2024-05-27T07:14:28.668+0000 7f8ad4168200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-05-27T07:14:29.586 INFO:journalctl@ceph.mgr.x.smithi052.stdout:May 27 07:14:29 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mgr-x[32184]: 2024-05-27T07:14:29.308+0000 7f8ad4168200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-05-27T07:14:29.587 INFO:journalctl@ceph.mgr.x.smithi052.stdout:May 27 07:14:29 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mgr-x[32184]: 2024-05-27T07:14:29.542+0000 7f8ad4168200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-05-27T07:14:29.980 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:14:29 smithi052 ceph-mon[31955]: Active manager daemon x restarted 2024-05-27T07:14:29.980 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:14:29 smithi052 ceph-mon[31955]: Activating manager daemon x 2024-05-27T07:14:29.980 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:14:29 smithi052 ceph-mon[31955]: osdmap e3: 0 total, 0 up, 0 in 2024-05-27T07:14:29.980 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:14:29 smithi052 ceph-mon[31955]: mgrmap e10: x(active, starting, since 0.00425479s) 2024-05-27T07:14:29.980 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:14:29 smithi052 ceph-mon[31955]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-05-27T07:14:29.981 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:14:29 smithi052 ceph-mon[31955]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-05-27T07:14:29.981 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:14:29 smithi052 ceph-mon[31955]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-05-27T07:14:29.981 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:14:29 smithi052 ceph-mon[31955]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-05-27T07:14:29.981 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:14:29 smithi052 ceph-mon[31955]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-05-27T07:14:29.981 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:14:29 smithi052 ceph-mon[31955]: Manager daemon x is now available 2024-05-27T07:14:29.981 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:14:29 smithi052 ceph-mon[31955]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-27T07:14:30.730 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:14:30 smithi052 ceph-mon[31955]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2024-05-27T07:14:30.730 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:14:30 smithi052 ceph-mon[31955]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2024-05-27T07:14:30.730 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:14:30 smithi052 ceph-mon[31955]: mgrmap e11: x(active, since 1.00753s) 2024-05-27T07:14:31.015 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout { 2024-05-27T07:14:31.015 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout "mgrmap_epoch": 11, 2024-05-27T07:14:31.015 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout "initialized": true 2024-05-27T07:14:31.015 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout } 2024-05-27T07:14:31.015 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:14:24.759+0000 7f9f2d723700 1 Processor -- start 2024-05-27T07:14:31.015 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:14:24.759+0000 7f9f2d723700 1 -- start start 2024-05-27T07:14:31.015 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:14:24.759+0000 7f9f2d723700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f9f2810a020 0x7f9f2810a430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:14:31.015 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:14:24.759+0000 7f9f2d723700 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x7f9f2810aa00 con 0x7f9f2810a020 2024-05-27T07:14:31.015 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:14:24.759+0000 7f9f26ffd700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f9f2810a020 0x7f9f2810a430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:14:31.016 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:14:24.759+0000 7f9f26ffd700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f9f2810a020 0x7f9f2810a430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.52:44582/0 (socket says 172.21.15.52:44582) 2024-05-27T07:14:31.016 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:14:24.759+0000 7f9f26ffd700 1 -- 172.21.15.52:0/1591005197 learned_addr learned my addr 172.21.15.52:0/1591005197 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-05-27T07:14:31.016 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:14:24.759+0000 7f9f26ffd700 1 -- 172.21.15.52:0/1591005197 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9f2810ab80 con 0x7f9f2810a020 2024-05-27T07:14:31.016 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:14:24.760+0000 7f9f26ffd700 1 --2- 172.21.15.52:0/1591005197 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f9f2810a020 0x7f9f2810a430 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f9f180098a0 tx=0x7f9f18009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1b79c4d3c0ebbf1e server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:14:31.016 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:14:24.760+0000 7f9f267fc700 1 -- 172.21.15.52:0/1591005197 <== mon.0 v2:172.21.15.52:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f9f18004160 con 0x7f9f2810a020 2024-05-27T07:14:31.016 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:14:24.760+0000 7f9f267fc700 1 -- 172.21.15.52:0/1591005197 <== mon.0 v2:172.21.15.52:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f9f18004300 con 0x7f9f2810a020 2024-05-27T07:14:31.016 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:14:24.760+0000 7f9f267fc700 1 -- 172.21.15.52:0/1591005197 <== mon.0 v2:172.21.15.52:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f9f18011a20 con 0x7f9f2810a020 2024-05-27T07:14:31.016 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:14:24.761+0000 7f9f2d723700 1 -- 172.21.15.52:0/1591005197 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f9f2810a020 msgr2=0x7f9f2810a430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:14:31.016 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:14:24.761+0000 7f9f2d723700 1 --2- 172.21.15.52:0/1591005197 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f9f2810a020 0x7f9f2810a430 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f9f180098a0 tx=0x7f9f18009bb0 comp rx=0 tx=0).stop 2024-05-27T07:14:31.016 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:14:24.761+0000 7f9f2d723700 1 -- 172.21.15.52:0/1591005197 shutdown_connections 2024-05-27T07:14:31.016 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:14:24.761+0000 7f9f2d723700 1 --2- 172.21.15.52:0/1591005197 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f9f2810a020 0x7f9f2810a430 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:14:31.016 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:14:24.761+0000 7f9f2d723700 1 -- 172.21.15.52:0/1591005197 >> 172.21.15.52:0/1591005197 conn(0x7f9f281038e0 msgr2=0x7f9f28105d30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:14:31.016 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:14:24.761+0000 7f9f2d723700 1 -- 172.21.15.52:0/1591005197 shutdown_connections 2024-05-27T07:14:31.017 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:14:24.761+0000 7f9f2d723700 1 -- 172.21.15.52:0/1591005197 wait complete. 2024-05-27T07:14:31.017 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:14:24.761+0000 7f9f2d723700 1 Processor -- start 2024-05-27T07:14:31.017 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:14:24.762+0000 7f9f2d723700 1 -- start start 2024-05-27T07:14:31.017 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:14:24.762+0000 7f9f2d723700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f9f2810a020 0x7f9f2819ce00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:14:31.017 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:14:24.762+0000 7f9f2d723700 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x7f9f2810cf80 con 0x7f9f2810a020 2024-05-27T07:14:31.017 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:14:24.762+0000 7f9f26ffd700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f9f2810a020 0x7f9f2819ce00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:14:31.017 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:14:24.762+0000 7f9f26ffd700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f9f2810a020 0x7f9f2819ce00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.52:44594/0 (socket says 172.21.15.52:44594) 2024-05-27T07:14:31.017 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:14:24.762+0000 7f9f26ffd700 1 -- 172.21.15.52:0/2108714421 learned_addr learned my addr 172.21.15.52:0/2108714421 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-05-27T07:14:31.017 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:14:24.762+0000 7f9f26ffd700 1 -- 172.21.15.52:0/2108714421 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9f2819d340 con 0x7f9f2810a020 2024-05-27T07:14:31.017 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:14:24.763+0000 7f9f26ffd700 1 --2- 172.21.15.52:0/2108714421 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f9f2810a020 0x7f9f2819ce00 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f9f180119f0 tx=0x7f9f18011ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:14:31.017 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:14:24.763+0000 7f9f24ff9700 1 -- 172.21.15.52:0/2108714421 <== mon.0 v2:172.21.15.52:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f9f180039e0 con 0x7f9f2810a020 2024-05-27T07:14:31.017 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:14:24.763+0000 7f9f24ff9700 1 -- 172.21.15.52:0/2108714421 <== mon.0 v2:172.21.15.52:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f9f1801b8f0 con 0x7f9f2810a020 2024-05-27T07:14:31.017 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:14:24.763+0000 7f9f2d723700 1 -- 172.21.15.52:0/2108714421 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9f2819d5d0 con 0x7f9f2810a020 2024-05-27T07:14:31.017 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:14:24.763+0000 7f9f24ff9700 1 -- 172.21.15.52:0/2108714421 <== mon.0 v2:172.21.15.52:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f9f180237a0 con 0x7f9f2810a020 2024-05-27T07:14:31.018 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:14:24.763+0000 7f9f2d723700 1 -- 172.21.15.52:0/2108714421 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9f281a01e0 con 0x7f9f2810a020 2024-05-27T07:14:31.018 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:14:24.764+0000 7f9f24ff9700 1 -- 172.21.15.52:0/2108714421 <== mon.0 v2:172.21.15.52:3300/0 4 ==== mgrmap(e 9) ==== 50235+0+0 (secure 0 0 0) 0x7f9f18023ac0 con 0x7f9f2810a020 2024-05-27T07:14:31.018 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:14:24.764+0000 7f9f24ff9700 1 --2- 172.21.15.52:0/2108714421 >> [v2:172.21.15.52:6800/1631683163,v1:172.21.15.52:6801/1631683163] conn(0x7f9f1403e470 0x7f9f14040980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:14:31.018 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:14:24.764+0000 7f9f24ff9700 1 -- 172.21.15.52:0/2108714421 --> [v2:172.21.15.52:6800/1631683163,v1:172.21.15.52:6801/1631683163] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x7f9f14041090 con 0x7f9f1403e470 2024-05-27T07:14:31.018 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:14:24.765+0000 7f9f24ff9700 1 -- 172.21.15.52:0/2108714421 <== mon.0 v2:172.21.15.52:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 940+0+0 (secure 0 0 0) 0x7f9f18053e80 con 0x7f9f2810a020 2024-05-27T07:14:31.018 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:14:24.764+0000 7f9f1e7fc700 1 -- 172.21.15.52:0/2108714421 >> [v2:172.21.15.52:6800/1631683163,v1:172.21.15.52:6801/1631683163] conn(0x7f9f1403e470 msgr2=0x7f9f14040980 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.52:6800/1631683163 2024-05-27T07:14:31.018 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:14:24.765+0000 7f9f1e7fc700 1 --2- 172.21.15.52:0/2108714421 >> [v2:172.21.15.52:6800/1631683163,v1:172.21.15.52:6801/1631683163] conn(0x7f9f1403e470 0x7f9f14040980 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-05-27T07:14:31.018 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:14:24.966+0000 7f9f1e7fc700 1 -- 172.21.15.52:0/2108714421 >> [v2:172.21.15.52:6800/1631683163,v1:172.21.15.52:6801/1631683163] conn(0x7f9f1403e470 msgr2=0x7f9f14040980 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.52:6800/1631683163 2024-05-27T07:14:31.018 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:14:24.966+0000 7f9f1e7fc700 1 --2- 172.21.15.52:0/2108714421 >> [v2:172.21.15.52:6800/1631683163,v1:172.21.15.52:6801/1631683163] conn(0x7f9f1403e470 0x7f9f14040980 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-05-27T07:14:31.018 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:14:25.367+0000 7f9f1e7fc700 1 -- 172.21.15.52:0/2108714421 >> [v2:172.21.15.52:6800/1631683163,v1:172.21.15.52:6801/1631683163] conn(0x7f9f1403e470 msgr2=0x7f9f14040980 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.52:6800/1631683163 2024-05-27T07:14:31.018 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:14:25.367+0000 7f9f1e7fc700 1 --2- 172.21.15.52:0/2108714421 >> [v2:172.21.15.52:6800/1631683163,v1:172.21.15.52:6801/1631683163] conn(0x7f9f1403e470 0x7f9f14040980 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-05-27T07:14:31.019 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:14:26.168+0000 7f9f1e7fc700 1 -- 172.21.15.52:0/2108714421 >> [v2:172.21.15.52:6800/1631683163,v1:172.21.15.52:6801/1631683163] conn(0x7f9f1403e470 msgr2=0x7f9f14040980 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.52:6800/1631683163 2024-05-27T07:14:31.019 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:14:26.168+0000 7f9f1e7fc700 1 --2- 172.21.15.52:0/2108714421 >> [v2:172.21.15.52:6800/1631683163,v1:172.21.15.52:6801/1631683163] conn(0x7f9f1403e470 0x7f9f14040980 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-05-27T07:14:31.019 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:14:27.769+0000 7f9f1e7fc700 1 -- 172.21.15.52:0/2108714421 >> [v2:172.21.15.52:6800/1631683163,v1:172.21.15.52:6801/1631683163] conn(0x7f9f1403e470 msgr2=0x7f9f14040980 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.52:6800/1631683163 2024-05-27T07:14:31.019 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:14:27.769+0000 7f9f1e7fc700 1 --2- 172.21.15.52:0/2108714421 >> [v2:172.21.15.52:6800/1631683163,v1:172.21.15.52:6801/1631683163] conn(0x7f9f1403e470 0x7f9f14040980 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 3.200000 2024-05-27T07:14:31.019 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:14:29.546+0000 7f9f24ff9700 1 -- 172.21.15.52:0/2108714421 <== mon.0 v2:172.21.15.52:3300/0 6 ==== mgrmap(e 10) ==== 50002+0+0 (secure 0 0 0) 0x7f9f1802c3f0 con 0x7f9f2810a020 2024-05-27T07:14:31.019 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:14:29.547+0000 7f9f24ff9700 1 -- 172.21.15.52:0/2108714421 >> [v2:172.21.15.52:6800/1631683163,v1:172.21.15.52:6801/1631683163] conn(0x7f9f1403e470 msgr2=0x7f9f14040980 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-05-27T07:14:31.019 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:14:29.547+0000 7f9f24ff9700 1 --2- 172.21.15.52:0/2108714421 >> [v2:172.21.15.52:6800/1631683163,v1:172.21.15.52:6801/1631683163] conn(0x7f9f1403e470 0x7f9f14040980 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:14:31.019 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:14:30.548+0000 7f9f24ff9700 1 -- 172.21.15.52:0/2108714421 <== mon.0 v2:172.21.15.52:3300/0 7 ==== mgrmap(e 11) ==== 50129+0+0 (secure 0 0 0) 0x7f9f18017b60 con 0x7f9f2810a020 2024-05-27T07:14:31.019 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:14:30.548+0000 7f9f24ff9700 1 --2- 172.21.15.52:0/2108714421 >> [v2:172.21.15.52:6800/3513075805,v1:172.21.15.52:6801/3513075805] conn(0x7f9f18053a10 0x7f9f14043f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:14:31.019 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:14:30.549+0000 7f9f24ff9700 1 -- 172.21.15.52:0/2108714421 --> [v2:172.21.15.52:6800/3513075805,v1:172.21.15.52:6801/3513075805] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x7f9f1800bd30 con 0x7f9f18053a10 2024-05-27T07:14:31.019 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:14:30.549+0000 7f9f1e7fc700 1 --2- 172.21.15.52:0/2108714421 >> [v2:172.21.15.52:6800/3513075805,v1:172.21.15.52:6801/3513075805] conn(0x7f9f18053a10 0x7f9f14043f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:14:31.019 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:14:30.550+0000 7f9f1e7fc700 1 --2- 172.21.15.52:0/2108714421 >> [v2:172.21.15.52:6800/3513075805,v1:172.21.15.52:6801/3513075805] conn(0x7f9f18053a10 0x7f9f14043f70 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 crypto rx=0x7f9f10003a10 tx=0x7f9f100092b0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:14:31.019 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:14:30.551+0000 7f9f24ff9700 1 -- 172.21.15.52:0/2108714421 <== mgr.14152 v2:172.21.15.52:6800/3513075805 1 ==== command_reply(tid 0: 0 ) ==== 8+0+9564 (secure 0 0 0) 0x7f9f1800bd30 con 0x7f9f18053a10 2024-05-27T07:14:31.020 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:14:30.559+0000 7f9f2d723700 1 -- 172.21.15.52:0/2108714421 --> [v2:172.21.15.52:6800/3513075805,v1:172.21.15.52:6801/3513075805] -- command(tid 1: {"prefix": "mgr_status"}) -- 0x7f9f28064e10 con 0x7f9f18053a10 2024-05-27T07:14:31.020 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:14:30.560+0000 7f9f24ff9700 1 -- 172.21.15.52:0/2108714421 <== mgr.14152 v2:172.21.15.52:6800/3513075805 2 ==== command_reply(tid 1: 0 ) ==== 8+0+52 (secure 0 0 0) 0x7f9f28064e10 con 0x7f9f18053a10 2024-05-27T07:14:31.020 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:14:30.560+0000 7f9f2d723700 1 -- 172.21.15.52:0/2108714421 >> [v2:172.21.15.52:6800/3513075805,v1:172.21.15.52:6801/3513075805] conn(0x7f9f18053a10 msgr2=0x7f9f14043f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:14:31.020 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:14:30.560+0000 7f9f2d723700 1 --2- 172.21.15.52:0/2108714421 >> [v2:172.21.15.52:6800/3513075805,v1:172.21.15.52:6801/3513075805] conn(0x7f9f18053a10 0x7f9f14043f70 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 crypto rx=0x7f9f10003a10 tx=0x7f9f100092b0 comp rx=0 tx=0).stop 2024-05-27T07:14:31.020 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:14:30.560+0000 7f9f2d723700 1 -- 172.21.15.52:0/2108714421 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f9f2810a020 msgr2=0x7f9f2819ce00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:14:31.020 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:14:30.560+0000 7f9f2d723700 1 --2- 172.21.15.52:0/2108714421 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f9f2810a020 0x7f9f2819ce00 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f9f180119f0 tx=0x7f9f18011ad0 comp rx=0 tx=0).stop 2024-05-27T07:14:31.020 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:14:30.560+0000 7f9f2d723700 1 -- 172.21.15.52:0/2108714421 shutdown_connections 2024-05-27T07:14:31.020 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:14:30.560+0000 7f9f2d723700 1 --2- 172.21.15.52:0/2108714421 >> [v2:172.21.15.52:6800/3513075805,v1:172.21.15.52:6801/3513075805] conn(0x7f9f18053a10 0x7f9f14043f70 unknown :-1 s=CLOSED pgs=1 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:14:31.020 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:14:30.560+0000 7f9f2d723700 1 --2- 172.21.15.52:0/2108714421 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f9f2810a020 0x7f9f2819ce00 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:14:31.020 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:14:30.560+0000 7f9f2d723700 1 --2- 172.21.15.52:0/2108714421 >> [v2:172.21.15.52:6800/1631683163,v1:172.21.15.52:6801/1631683163] conn(0x7f9f1403e470 0x7f9f14040980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:14:31.020 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:14:30.560+0000 7f9f2d723700 1 -- 172.21.15.52:0/2108714421 >> 172.21.15.52:0/2108714421 conn(0x7f9f281038e0 msgr2=0x7f9f28104110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:14:31.020 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:14:30.560+0000 7f9f2d723700 1 -- 172.21.15.52:0/2108714421 shutdown_connections 2024-05-27T07:14:31.020 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:14:30.560+0000 7f9f2d723700 1 -- 172.21.15.52:0/2108714421 wait complete. 2024-05-27T07:14:31.021 INFO:teuthology.orchestra.run.smithi052.stdout:mgr epoch 9 is available 2024-05-27T07:14:31.021 INFO:teuthology.orchestra.run.smithi052.stdout:Generating a dashboard self-signed certificate... 2024-05-27T07:14:31.980 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:14:31 smithi052 ceph-mon[31955]: from='client.14156 -' entity='client.admin' cmd=[{"prefix": "get_command_descriptions"}]: dispatch 2024-05-27T07:14:31.980 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:14:31 smithi052 ceph-mon[31955]: from='client.14156 -' entity='client.admin' cmd=[{"prefix": "mgr_status"}]: dispatch 2024-05-27T07:14:31.980 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:14:31 smithi052 ceph-mon[31955]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' 2024-05-27T07:14:32.647 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:14:32 smithi052 ceph-mon[31955]: [27/May/2024:07:14:31] ENGINE Bus STARTING 2024-05-27T07:14:32.647 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:14:32 smithi052 ceph-mon[31955]: mgrmap e12: x(active, since 2s) 2024-05-27T07:14:32.647 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:14:32 smithi052 ceph-mon[31955]: [27/May/2024:07:14:31] ENGINE Serving on http://172.21.15.52:8765 2024-05-27T07:14:32.647 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:14:32 smithi052 ceph-mon[31955]: [27/May/2024:07:14:31] ENGINE Serving on https://172.21.15.52:7150 2024-05-27T07:14:32.647 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:14:32 smithi052 ceph-mon[31955]: [27/May/2024:07:14:31] ENGINE Bus STARTED 2024-05-27T07:14:32.647 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:14:32 smithi052 ceph-mon[31955]: from='client.14164 -' entity='client.admin' cmd=[{"prefix": "dashboard create-self-signed-cert", "target": ["mon-mgr", ""]}]: dispatch 2024-05-27T07:14:32.648 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:14:32 smithi052 ceph-mon[31955]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' 2024-05-27T07:14:32.648 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:14:32 smithi052 ceph-mon[31955]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' 2024-05-27T07:14:32.654 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout Self-signed certificate created 2024-05-27T07:14:32.654 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:14:31.785+0000 7f6cf9116700 1 Processor -- start 2024-05-27T07:14:32.654 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:14:31.785+0000 7f6cf9116700 1 -- start start 2024-05-27T07:14:32.654 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:14:31.785+0000 7f6cf9116700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f6cf407ccd0 0x7f6cf410c6b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:14:32.654 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:14:31.785+0000 7f6cf9116700 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x7f6cf410ccc0 con 0x7f6cf407ccd0 2024-05-27T07:14:32.654 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:14:31.785+0000 7f6cf2d9d700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f6cf407ccd0 0x7f6cf410c6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:14:32.654 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:14:31.785+0000 7f6cf2d9d700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f6cf407ccd0 0x7f6cf410c6b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.52:51558/0 (socket says 172.21.15.52:51558) 2024-05-27T07:14:32.654 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:14:31.785+0000 7f6cf2d9d700 1 -- 172.21.15.52:0/2325665904 learned_addr learned my addr 172.21.15.52:0/2325665904 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-05-27T07:14:32.654 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:14:31.786+0000 7f6cf2d9d700 1 -- 172.21.15.52:0/2325665904 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6cf410ce40 con 0x7f6cf407ccd0 2024-05-27T07:14:32.654 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:14:31.786+0000 7f6cf2d9d700 1 --2- 172.21.15.52:0/2325665904 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f6cf407ccd0 0x7f6cf410c6b0 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f6ce40098a0 tx=0x7f6ce4009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4e92eb5ee4eafda8 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:14:32.654 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:14:31.786+0000 7f6cf1d9b700 1 -- 172.21.15.52:0/2325665904 <== mon.0 v2:172.21.15.52:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f6ce4004160 con 0x7f6cf407ccd0 2024-05-27T07:14:32.655 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:14:31.786+0000 7f6cf1d9b700 1 -- 172.21.15.52:0/2325665904 <== mon.0 v2:172.21.15.52:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f6ce4004300 con 0x7f6cf407ccd0 2024-05-27T07:14:32.655 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:14:31.786+0000 7f6cf1d9b700 1 -- 172.21.15.52:0/2325665904 <== mon.0 v2:172.21.15.52:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f6ce4011a20 con 0x7f6cf407ccd0 2024-05-27T07:14:32.655 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:14:31.787+0000 7f6cf9116700 1 -- 172.21.15.52:0/2325665904 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f6cf407ccd0 msgr2=0x7f6cf410c6b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:14:32.655 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:14:31.787+0000 7f6cf9116700 1 --2- 172.21.15.52:0/2325665904 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f6cf407ccd0 0x7f6cf410c6b0 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f6ce40098a0 tx=0x7f6ce4009bb0 comp rx=0 tx=0).stop 2024-05-27T07:14:32.655 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:14:31.787+0000 7f6cf9116700 1 -- 172.21.15.52:0/2325665904 shutdown_connections 2024-05-27T07:14:32.655 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:14:31.787+0000 7f6cf9116700 1 --2- 172.21.15.52:0/2325665904 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f6cf407ccd0 0x7f6cf410c6b0 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:14:32.655 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:14:31.787+0000 7f6cf9116700 1 -- 172.21.15.52:0/2325665904 >> 172.21.15.52:0/2325665904 conn(0x7f6cf41040e0 msgr2=0x7f6cf4106570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:14:32.655 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:14:31.787+0000 7f6cf9116700 1 -- 172.21.15.52:0/2325665904 shutdown_connections 2024-05-27T07:14:32.655 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:14:31.787+0000 7f6cf9116700 1 -- 172.21.15.52:0/2325665904 wait complete. 2024-05-27T07:14:32.655 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:14:31.788+0000 7f6cf9116700 1 Processor -- start 2024-05-27T07:14:32.655 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:14:31.788+0000 7f6cf9116700 1 -- start start 2024-05-27T07:14:32.655 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:14:31.788+0000 7f6cf9116700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f6cf407ccd0 0x7f6cf4109e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:14:32.655 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:14:31.788+0000 7f6cf9116700 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x7f6cf410df00 con 0x7f6cf407ccd0 2024-05-27T07:14:32.656 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:14:31.788+0000 7f6cf2d9d700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f6cf407ccd0 0x7f6cf4109e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:14:32.656 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:14:31.788+0000 7f6cf2d9d700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f6cf407ccd0 0x7f6cf4109e20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.52:51572/0 (socket says 172.21.15.52:51572) 2024-05-27T07:14:32.656 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:14:31.788+0000 7f6cf2d9d700 1 -- 172.21.15.52:0/1723799528 learned_addr learned my addr 172.21.15.52:0/1723799528 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-05-27T07:14:32.656 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:14:31.789+0000 7f6cf2d9d700 1 -- 172.21.15.52:0/1723799528 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6cf410a360 con 0x7f6cf407ccd0 2024-05-27T07:14:32.656 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:14:31.789+0000 7f6cf2d9d700 1 --2- 172.21.15.52:0/1723799528 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f6cf407ccd0 0x7f6cf4109e20 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f6ce40119f0 tx=0x7f6ce4011ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:14:32.656 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:14:31.789+0000 7f6cdbfff700 1 -- 172.21.15.52:0/1723799528 <== mon.0 v2:172.21.15.52:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f6ce40039e0 con 0x7f6cf407ccd0 2024-05-27T07:14:32.656 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:14:31.789+0000 7f6cdbfff700 1 -- 172.21.15.52:0/1723799528 <== mon.0 v2:172.21.15.52:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f6ce401b8f0 con 0x7f6cf407ccd0 2024-05-27T07:14:32.656 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:14:31.789+0000 7f6cdbfff700 1 -- 172.21.15.52:0/1723799528 <== mon.0 v2:172.21.15.52:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f6ce40237a0 con 0x7f6cf407ccd0 2024-05-27T07:14:32.656 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:14:31.789+0000 7f6cf9116700 1 -- 172.21.15.52:0/1723799528 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6cf41085b0 con 0x7f6cf407ccd0 2024-05-27T07:14:32.656 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:14:31.789+0000 7f6cf9116700 1 -- 172.21.15.52:0/1723799528 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6cf4108a90 con 0x7f6cf407ccd0 2024-05-27T07:14:32.656 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:14:31.790+0000 7f6cdbfff700 1 -- 172.21.15.52:0/1723799528 <== mon.0 v2:172.21.15.52:3300/0 4 ==== mgrmap(e 12) ==== 50235+0+0 (secure 0 0 0) 0x7f6ce40193d0 con 0x7f6cf407ccd0 2024-05-27T07:14:32.656 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:14:31.790+0000 7f6cf9116700 1 -- 172.21.15.52:0/1723799528 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6cf4064e10 con 0x7f6cf407ccd0 2024-05-27T07:14:32.656 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:14:31.791+0000 7f6cdbfff700 1 --2- 172.21.15.52:0/1723799528 >> [v2:172.21.15.52:6800/3513075805,v1:172.21.15.52:6801/3513075805] conn(0x7f6cdc03e470 0x7f6cdc040980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:14:32.657 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:14:31.791+0000 7f6cdbfff700 1 -- 172.21.15.52:0/1723799528 <== mon.0 v2:172.21.15.52:3300/0 5 ==== osd_map(3..3 src has 1..3) ==== 1155+0+0 (secure 0 0 0) 0x7f6ce4054020 con 0x7f6cf407ccd0 2024-05-27T07:14:32.657 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:14:31.791+0000 7f6cf259c700 1 --2- 172.21.15.52:0/1723799528 >> [v2:172.21.15.52:6800/3513075805,v1:172.21.15.52:6801/3513075805] conn(0x7f6cdc03e470 0x7f6cdc040980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:14:32.657 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:14:31.791+0000 7f6cf259c700 1 --2- 172.21.15.52:0/1723799528 >> [v2:172.21.15.52:6800/3513075805,v1:172.21.15.52:6801/3513075805] conn(0x7f6cdc03e470 0x7f6cdc040980 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f6ce8006fd0 tx=0x7f6ce8006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:14:32.657 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:14:31.796+0000 7f6cdbfff700 1 -- 172.21.15.52:0/1723799528 <== mon.0 v2:172.21.15.52:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191963 (secure 0 0 0) 0x7f6ce401f030 con 0x7f6cf407ccd0 2024-05-27T07:14:32.657 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:14:32.027+0000 7f6cf9116700 1 -- 172.21.15.52:0/1723799528 --> [v2:172.21.15.52:6800/3513075805,v1:172.21.15.52:6801/3513075805] -- mgr_command(tid 0: {"prefix": "dashboard create-self-signed-cert", "target": ["mon-mgr", ""]}) -- 0x7f6cf41091c0 con 0x7f6cdc03e470 2024-05-27T07:14:32.657 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:14:32.097+0000 7f6cdbfff700 1 -- 172.21.15.52:0/1723799528 <== mgr.14152 v2:172.21.15.52:6800/3513075805 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+32 (secure 0 0 0) 0x7f6cf41091c0 con 0x7f6cdc03e470 2024-05-27T07:14:32.657 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:14:32.101+0000 7f6cf9116700 1 -- 172.21.15.52:0/1723799528 >> [v2:172.21.15.52:6800/3513075805,v1:172.21.15.52:6801/3513075805] conn(0x7f6cdc03e470 msgr2=0x7f6cdc040980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:14:32.657 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:14:32.101+0000 7f6cf9116700 1 --2- 172.21.15.52:0/1723799528 >> [v2:172.21.15.52:6800/3513075805,v1:172.21.15.52:6801/3513075805] conn(0x7f6cdc03e470 0x7f6cdc040980 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f6ce8006fd0 tx=0x7f6ce8006e40 comp rx=0 tx=0).stop 2024-05-27T07:14:32.657 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:14:32.101+0000 7f6cf9116700 1 -- 172.21.15.52:0/1723799528 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f6cf407ccd0 msgr2=0x7f6cf4109e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:14:32.657 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:14:32.101+0000 7f6cf9116700 1 --2- 172.21.15.52:0/1723799528 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f6cf407ccd0 0x7f6cf4109e20 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f6ce40119f0 tx=0x7f6ce4011ad0 comp rx=0 tx=0).stop 2024-05-27T07:14:32.657 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:14:32.101+0000 7f6cf9116700 1 -- 172.21.15.52:0/1723799528 shutdown_connections 2024-05-27T07:14:32.657 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:14:32.101+0000 7f6cf9116700 1 --2- 172.21.15.52:0/1723799528 >> [v2:172.21.15.52:6800/3513075805,v1:172.21.15.52:6801/3513075805] conn(0x7f6cdc03e470 0x7f6cdc040980 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:14:32.657 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:14:32.101+0000 7f6cf9116700 1 --2- 172.21.15.52:0/1723799528 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f6cf407ccd0 0x7f6cf4109e20 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:14:32.658 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:14:32.101+0000 7f6cf9116700 1 -- 172.21.15.52:0/1723799528 >> 172.21.15.52:0/1723799528 conn(0x7f6cf41040e0 msgr2=0x7f6cf41049e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:14:32.658 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:14:32.101+0000 7f6cf9116700 1 -- 172.21.15.52:0/1723799528 shutdown_connections 2024-05-27T07:14:32.658 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:14:32.101+0000 7f6cf9116700 1 -- 172.21.15.52:0/1723799528 wait complete. 2024-05-27T07:14:32.658 INFO:teuthology.orchestra.run.smithi052.stdout:Creating initial admin user... 2024-05-27T07:14:33.882 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:14:33 smithi052 ceph-mon[31955]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' 2024-05-27T07:14:33.882 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:14:33 smithi052 ceph-mon[31955]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' 2024-05-27T07:14:34.793 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout {"username": "admin", "password": "$2b$12$nxsED0wcNwGTf84a7X1.ZeIBvsFSms.Euheu/b7UTd9gA4Ji4m3fW", "roles": ["administrator"], "name": null, "email": null, "lastUpdate": 1716794073, "enabled": true, "pwdExpirationDate": null, "pwdUpdateRequired": true} 2024-05-27T07:14:34.794 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:14:33.129+0000 7f79d9fc0700 1 Processor -- start 2024-05-27T07:14:34.794 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:14:33.129+0000 7f79d9fc0700 1 -- start start 2024-05-27T07:14:34.794 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:14:33.129+0000 7f79d9fc0700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f79d407cc90 0x7f79d410c6f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:14:34.794 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:14:33.129+0000 7f79d9fc0700 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x7f79d410ccc0 con 0x7f79d407cc90 2024-05-27T07:14:34.794 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:14:33.130+0000 7f79d37fe700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f79d407cc90 0x7f79d410c6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:14:34.794 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:14:33.130+0000 7f79d37fe700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f79d407cc90 0x7f79d410c6f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.52:51584/0 (socket says 172.21.15.52:51584) 2024-05-27T07:14:34.794 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:14:33.130+0000 7f79d37fe700 1 -- 172.21.15.52:0/3155928861 learned_addr learned my addr 172.21.15.52:0/3155928861 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-05-27T07:14:34.794 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:14:33.130+0000 7f79d37fe700 1 -- 172.21.15.52:0/3155928861 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f79d410ce40 con 0x7f79d407cc90 2024-05-27T07:14:34.794 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:14:33.130+0000 7f79d37fe700 1 --2- 172.21.15.52:0/3155928861 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f79d407cc90 0x7f79d410c6f0 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f79c40098a0 tx=0x7f79c4009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=55e833b0a793a13c server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:14:34.794 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:14:33.130+0000 7f79d27fc700 1 -- 172.21.15.52:0/3155928861 <== mon.0 v2:172.21.15.52:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f79c4004160 con 0x7f79d407cc90 2024-05-27T07:14:34.794 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:14:33.130+0000 7f79d27fc700 1 -- 172.21.15.52:0/3155928861 <== mon.0 v2:172.21.15.52:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f79c4004300 con 0x7f79d407cc90 2024-05-27T07:14:34.795 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:14:33.130+0000 7f79d27fc700 1 -- 172.21.15.52:0/3155928861 <== mon.0 v2:172.21.15.52:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f79c4011a20 con 0x7f79d407cc90 2024-05-27T07:14:34.795 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:14:33.131+0000 7f79d9fc0700 1 -- 172.21.15.52:0/3155928861 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f79d407cc90 msgr2=0x7f79d410c6f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:14:34.795 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:14:33.131+0000 7f79d9fc0700 1 --2- 172.21.15.52:0/3155928861 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f79d407cc90 0x7f79d410c6f0 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f79c40098a0 tx=0x7f79c4009bb0 comp rx=0 tx=0).stop 2024-05-27T07:14:34.795 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:14:33.131+0000 7f79d9fc0700 1 -- 172.21.15.52:0/3155928861 shutdown_connections 2024-05-27T07:14:34.795 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:14:33.131+0000 7f79d9fc0700 1 --2- 172.21.15.52:0/3155928861 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f79d407cc90 0x7f79d410c6f0 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:14:34.795 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:14:33.131+0000 7f79d9fc0700 1 -- 172.21.15.52:0/3155928861 >> 172.21.15.52:0/3155928861 conn(0x7f79d41040e0 msgr2=0x7f79d4106570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:14:34.795 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:14:33.131+0000 7f79d9fc0700 1 -- 172.21.15.52:0/3155928861 shutdown_connections 2024-05-27T07:14:34.795 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:14:33.131+0000 7f79d9fc0700 1 -- 172.21.15.52:0/3155928861 wait complete. 2024-05-27T07:14:34.795 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:14:33.132+0000 7f79d9fc0700 1 Processor -- start 2024-05-27T07:14:34.795 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:14:33.132+0000 7f79d9fc0700 1 -- start start 2024-05-27T07:14:34.795 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:14:33.132+0000 7f79d9fc0700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f79d407cc90 0x7f79d419dea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:14:34.795 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:14:33.132+0000 7f79d9fc0700 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x7f79d410df00 con 0x7f79d407cc90 2024-05-27T07:14:34.795 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:14:33.133+0000 7f79d37fe700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f79d407cc90 0x7f79d419dea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:14:34.796 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:14:33.133+0000 7f79d37fe700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f79d407cc90 0x7f79d419dea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.52:51590/0 (socket says 172.21.15.52:51590) 2024-05-27T07:14:34.796 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:14:33.133+0000 7f79d37fe700 1 -- 172.21.15.52:0/2619990183 learned_addr learned my addr 172.21.15.52:0/2619990183 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-05-27T07:14:34.796 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:14:33.133+0000 7f79d37fe700 1 -- 172.21.15.52:0/2619990183 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f79d419e3e0 con 0x7f79d407cc90 2024-05-27T07:14:34.796 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:14:33.133+0000 7f79d37fe700 1 --2- 172.21.15.52:0/2619990183 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f79d407cc90 0x7f79d419dea0 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f79c4004660 tx=0x7f79c4011aa0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:14:34.796 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:14:33.133+0000 7f79d0ff9700 1 -- 172.21.15.52:0/2619990183 <== mon.0 v2:172.21.15.52:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f79c4003aa0 con 0x7f79d407cc90 2024-05-27T07:14:34.796 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:14:33.133+0000 7f79d0ff9700 1 -- 172.21.15.52:0/2619990183 <== mon.0 v2:172.21.15.52:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f79c401b8f0 con 0x7f79d407cc90 2024-05-27T07:14:34.796 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:14:33.133+0000 7f79d9fc0700 1 -- 172.21.15.52:0/2619990183 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f79d419e670 con 0x7f79d407cc90 2024-05-27T07:14:34.796 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:14:33.134+0000 7f79d0ff9700 1 -- 172.21.15.52:0/2619990183 <== mon.0 v2:172.21.15.52:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f79c4023900 con 0x7f79d407cc90 2024-05-27T07:14:34.796 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:14:33.134+0000 7f79d9fc0700 1 -- 172.21.15.52:0/2619990183 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f79d41a1280 con 0x7f79d407cc90 2024-05-27T07:14:34.796 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:14:33.134+0000 7f79d0ff9700 1 -- 172.21.15.52:0/2619990183 <== mon.0 v2:172.21.15.52:3300/0 4 ==== mgrmap(e 12) ==== 50235+0+0 (secure 0 0 0) 0x7f79c4023b70 con 0x7f79d407cc90 2024-05-27T07:14:34.796 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:14:33.135+0000 7f79d0ff9700 1 --2- 172.21.15.52:0/2619990183 >> [v2:172.21.15.52:6800/3513075805,v1:172.21.15.52:6801/3513075805] conn(0x7f79bc03e520 0x7f79bc040a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:14:34.796 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:14:33.135+0000 7f79d0ff9700 1 -- 172.21.15.52:0/2619990183 <== mon.0 v2:172.21.15.52:3300/0 5 ==== osd_map(3..3 src has 1..3) ==== 1155+0+0 (secure 0 0 0) 0x7f79c4054e90 con 0x7f79d407cc90 2024-05-27T07:14:34.797 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:14:33.135+0000 7f79d9fc0700 1 -- 172.21.15.52:0/2619990183 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f79d4064e10 con 0x7f79d407cc90 2024-05-27T07:14:34.797 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:14:33.135+0000 7f79d2ffd700 1 --2- 172.21.15.52:0/2619990183 >> [v2:172.21.15.52:6800/3513075805,v1:172.21.15.52:6801/3513075805] conn(0x7f79bc03e520 0x7f79bc040a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:14:34.797 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:14:33.135+0000 7f79d2ffd700 1 --2- 172.21.15.52:0/2619990183 >> [v2:172.21.15.52:6800/3513075805,v1:172.21.15.52:6801/3513075805] conn(0x7f79bc03e520 0x7f79bc040a30 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f79c8006fd0 tx=0x7f79c8006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:14:34.797 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:14:33.140+0000 7f79d0ff9700 1 -- 172.21.15.52:0/2619990183 <== mon.0 v2:172.21.15.52:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191963 (secure 0 0 0) 0x7f79c401f030 con 0x7f79d407cc90 2024-05-27T07:14:34.797 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:14:33.378+0000 7f79d9fc0700 1 -- 172.21.15.52:0/2619990183 --> [v2:172.21.15.52:6800/3513075805,v1:172.21.15.52:6801/3513075805] -- mgr_command(tid 0: {"prefix": "dashboard ac-user-create", "username": "admin", "rolename": "administrator", "force_password": true, "pwd_update_required": true, "target": ["mon-mgr", ""]}) -- 0x7f79d41050f0 con 0x7f79bc03e520 2024-05-27T07:14:34.797 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:14:33.625+0000 7f79d0ff9700 1 -- 172.21.15.52:0/2619990183 <== mgr.14152 v2:172.21.15.52:6800/3513075805 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+252 (secure 0 0 0) 0x7f79d41050f0 con 0x7f79bc03e520 2024-05-27T07:14:34.797 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:14:33.627+0000 7f79d9fc0700 1 -- 172.21.15.52:0/2619990183 >> [v2:172.21.15.52:6800/3513075805,v1:172.21.15.52:6801/3513075805] conn(0x7f79bc03e520 msgr2=0x7f79bc040a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:14:34.797 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:14:33.627+0000 7f79d9fc0700 1 --2- 172.21.15.52:0/2619990183 >> [v2:172.21.15.52:6800/3513075805,v1:172.21.15.52:6801/3513075805] conn(0x7f79bc03e520 0x7f79bc040a30 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f79c8006fd0 tx=0x7f79c8006e40 comp rx=0 tx=0).stop 2024-05-27T07:14:34.797 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:14:33.627+0000 7f79d9fc0700 1 -- 172.21.15.52:0/2619990183 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f79d407cc90 msgr2=0x7f79d419dea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:14:34.797 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:14:33.627+0000 7f79d9fc0700 1 --2- 172.21.15.52:0/2619990183 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f79d407cc90 0x7f79d419dea0 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f79c4004660 tx=0x7f79c4011aa0 comp rx=0 tx=0).stop 2024-05-27T07:14:34.797 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:14:33.627+0000 7f79d9fc0700 1 -- 172.21.15.52:0/2619990183 shutdown_connections 2024-05-27T07:14:34.797 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:14:33.627+0000 7f79d9fc0700 1 --2- 172.21.15.52:0/2619990183 >> [v2:172.21.15.52:6800/3513075805,v1:172.21.15.52:6801/3513075805] conn(0x7f79bc03e520 0x7f79bc040a30 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:14:34.798 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:14:33.627+0000 7f79d9fc0700 1 --2- 172.21.15.52:0/2619990183 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f79d407cc90 0x7f79d419dea0 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:14:34.798 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:14:33.627+0000 7f79d9fc0700 1 -- 172.21.15.52:0/2619990183 >> 172.21.15.52:0/2619990183 conn(0x7f79d41040e0 msgr2=0x7f79d41049e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:14:34.798 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:14:33.627+0000 7f79d9fc0700 1 -- 172.21.15.52:0/2619990183 shutdown_connections 2024-05-27T07:14:34.798 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:14:33.627+0000 7f79d9fc0700 1 -- 172.21.15.52:0/2619990183 wait complete. 2024-05-27T07:14:34.798 INFO:teuthology.orchestra.run.smithi052.stdout:Fetching dashboard port number... 2024-05-27T07:14:34.895 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:14:34 smithi052 ceph-mon[31955]: from='client.14166 -' entity='client.admin' cmd=[{"prefix": "dashboard ac-user-create", "username": "admin", "rolename": "administrator", "force_password": true, "pwd_update_required": true, "target": ["mon-mgr", ""]}]: dispatch 2024-05-27T07:14:36.009 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:14:35 smithi052 ceph-mon[31955]: from='client.? 172.21.15.52:0/2564227453' entity='client.admin' cmd=[{"prefix": "config get", "who": "mgr", "key": "mgr/dashboard/ssl_server_port"}]: dispatch 2024-05-27T07:14:36.017 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout 8443 2024-05-27T07:14:36.017 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:14:35.249+0000 7f2edd8b0700 1 Processor -- start 2024-05-27T07:14:36.017 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:14:35.249+0000 7f2edd8b0700 1 -- start start 2024-05-27T07:14:36.017 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:14:35.249+0000 7f2edd8b0700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f2ed807ccb0 0x7f2ed810c6d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:14:36.017 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:14:35.249+0000 7f2edd8b0700 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x7f2ed810ccc0 con 0x7f2ed807ccb0 2024-05-27T07:14:36.017 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:14:35.250+0000 7f2ed6ffd700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f2ed807ccb0 0x7f2ed810c6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:14:36.017 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:14:35.250+0000 7f2ed6ffd700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f2ed807ccb0 0x7f2ed810c6d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.52:51606/0 (socket says 172.21.15.52:51606) 2024-05-27T07:14:36.017 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:14:35.250+0000 7f2ed6ffd700 1 -- 172.21.15.52:0/4185216735 learned_addr learned my addr 172.21.15.52:0/4185216735 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-05-27T07:14:36.017 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:14:35.250+0000 7f2ed6ffd700 1 -- 172.21.15.52:0/4185216735 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2ed810ce40 con 0x7f2ed807ccb0 2024-05-27T07:14:36.017 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:14:35.250+0000 7f2ed6ffd700 1 --2- 172.21.15.52:0/4185216735 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f2ed807ccb0 0x7f2ed810c6d0 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f2ec8009e80 tx=0x7f2ec800b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e3f6f74fc5f4e773 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:14:36.018 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:14:35.250+0000 7f2ed5ffb700 1 -- 172.21.15.52:0/4185216735 <== mon.0 v2:172.21.15.52:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f2ec8017070 con 0x7f2ed807ccb0 2024-05-27T07:14:36.018 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:14:35.250+0000 7f2ed5ffb700 1 -- 172.21.15.52:0/4185216735 <== mon.0 v2:172.21.15.52:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f2ec800b8c0 con 0x7f2ed807ccb0 2024-05-27T07:14:36.018 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:14:35.250+0000 7f2ed5ffb700 1 -- 172.21.15.52:0/4185216735 <== mon.0 v2:172.21.15.52:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f2ec8004490 con 0x7f2ed807ccb0 2024-05-27T07:14:36.018 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:14:35.251+0000 7f2edd8b0700 1 -- 172.21.15.52:0/4185216735 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f2ed807ccb0 msgr2=0x7f2ed810c6d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:14:36.018 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:14:35.251+0000 7f2edd8b0700 1 --2- 172.21.15.52:0/4185216735 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f2ed807ccb0 0x7f2ed810c6d0 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f2ec8009e80 tx=0x7f2ec800b330 comp rx=0 tx=0).stop 2024-05-27T07:14:36.018 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:14:35.251+0000 7f2edd8b0700 1 -- 172.21.15.52:0/4185216735 shutdown_connections 2024-05-27T07:14:36.018 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:14:35.251+0000 7f2edd8b0700 1 --2- 172.21.15.52:0/4185216735 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f2ed807ccb0 0x7f2ed810c6d0 unknown :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:14:36.018 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:14:35.251+0000 7f2edd8b0700 1 -- 172.21.15.52:0/4185216735 >> 172.21.15.52:0/4185216735 conn(0x7f2ed81040e0 msgr2=0x7f2ed8106570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:14:36.018 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:14:35.251+0000 7f2edd8b0700 1 -- 172.21.15.52:0/4185216735 shutdown_connections 2024-05-27T07:14:36.018 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:14:35.251+0000 7f2edd8b0700 1 -- 172.21.15.52:0/4185216735 wait complete. 2024-05-27T07:14:36.018 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:14:35.252+0000 7f2edd8b0700 1 Processor -- start 2024-05-27T07:14:36.018 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:14:35.252+0000 7f2edd8b0700 1 -- start start 2024-05-27T07:14:36.018 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:14:35.252+0000 7f2edd8b0700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f2ed807ccb0 0x7f2ed819df50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:14:36.018 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:14:35.252+0000 7f2edd8b0700 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x7f2ed810df00 con 0x7f2ed807ccb0 2024-05-27T07:14:36.019 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:14:35.253+0000 7f2ed6ffd700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f2ed807ccb0 0x7f2ed819df50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:14:36.019 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:14:35.253+0000 7f2ed6ffd700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f2ed807ccb0 0x7f2ed819df50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.52:51614/0 (socket says 172.21.15.52:51614) 2024-05-27T07:14:36.019 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:14:35.253+0000 7f2ed6ffd700 1 -- 172.21.15.52:0/2564227453 learned_addr learned my addr 172.21.15.52:0/2564227453 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-05-27T07:14:36.019 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:14:35.253+0000 7f2ed6ffd700 1 -- 172.21.15.52:0/2564227453 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2ed819e490 con 0x7f2ed807ccb0 2024-05-27T07:14:36.019 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:14:35.253+0000 7f2ed6ffd700 1 --2- 172.21.15.52:0/2564227453 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f2ed807ccb0 0x7f2ed819df50 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f2ec800bca0 tx=0x7f2ec800bd80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:14:36.019 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:14:35.253+0000 7f2edc8ae700 1 -- 172.21.15.52:0/2564227453 <== mon.0 v2:172.21.15.52:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f2ec8020020 con 0x7f2ed807ccb0 2024-05-27T07:14:36.019 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:14:35.253+0000 7f2edc8ae700 1 -- 172.21.15.52:0/2564227453 <== mon.0 v2:172.21.15.52:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f2ec8003b50 con 0x7f2ed807ccb0 2024-05-27T07:14:36.019 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:14:35.253+0000 7f2edc8ae700 1 -- 172.21.15.52:0/2564227453 <== mon.0 v2:172.21.15.52:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f2ec8017040 con 0x7f2ed807ccb0 2024-05-27T07:14:36.019 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:14:35.253+0000 7f2edd8b0700 1 -- 172.21.15.52:0/2564227453 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2ed819e720 con 0x7f2ed807ccb0 2024-05-27T07:14:36.019 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:14:35.253+0000 7f2edd8b0700 1 -- 172.21.15.52:0/2564227453 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2ed81a1330 con 0x7f2ed807ccb0 2024-05-27T07:14:36.019 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:14:35.254+0000 7f2edc8ae700 1 -- 172.21.15.52:0/2564227453 <== mon.0 v2:172.21.15.52:3300/0 4 ==== mgrmap(e 12) ==== 50235+0+0 (secure 0 0 0) 0x7f2ec8023020 con 0x7f2ed807ccb0 2024-05-27T07:14:36.019 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:14:35.254+0000 7f2edd8b0700 1 -- 172.21.15.52:0/2564227453 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2ed8064e10 con 0x7f2ed807ccb0 2024-05-27T07:14:36.020 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:14:35.255+0000 7f2edc8ae700 1 --2- 172.21.15.52:0/2564227453 >> [v2:172.21.15.52:6800/3513075805,v1:172.21.15.52:6801/3513075805] conn(0x7f2ec003e520 0x7f2ec0040a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:14:36.020 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:14:35.255+0000 7f2edc8ae700 1 -- 172.21.15.52:0/2564227453 <== mon.0 v2:172.21.15.52:3300/0 5 ==== osd_map(3..3 src has 1..3) ==== 1155+0+0 (secure 0 0 0) 0x7f2ec800f050 con 0x7f2ed807ccb0 2024-05-27T07:14:36.020 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:14:35.255+0000 7f2ed67fc700 1 --2- 172.21.15.52:0/2564227453 >> [v2:172.21.15.52:6800/3513075805,v1:172.21.15.52:6801/3513075805] conn(0x7f2ec003e520 0x7f2ec0040a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:14:36.020 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:14:35.255+0000 7f2ed67fc700 1 --2- 172.21.15.52:0/2564227453 >> [v2:172.21.15.52:6800/3513075805,v1:172.21.15.52:6801/3513075805] conn(0x7f2ec003e520 0x7f2ec0040a30 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f2ecc006fd0 tx=0x7f2ecc006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:14:36.020 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:14:35.260+0000 7f2edc8ae700 1 -- 172.21.15.52:0/2564227453 <== mon.0 v2:172.21.15.52:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191963 (secure 0 0 0) 0x7f2ec802a020 con 0x7f2ed807ccb0 2024-05-27T07:14:36.020 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:14:35.472+0000 7f2edd8b0700 1 -- 172.21.15.52:0/2564227453 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_command({"prefix": "config get", "who": "mgr", "key": "mgr/dashboard/ssl_server_port"} v 0) -- 0x7f2ed81a1500 con 0x7f2ed807ccb0 2024-05-27T07:14:36.020 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:14:35.473+0000 7f2edc8ae700 1 -- 172.21.15.52:0/2564227453 <== mon.0 v2:172.21.15.52:3300/0 7 ==== mon_command_ack([{"prefix": "config get", "who": "mgr", "key": "mgr/dashboard/ssl_server_port"}]=0 v8) ==== 112+0+5 (secure 0 0 0) 0x7f2ed81a1500 con 0x7f2ed807ccb0 2024-05-27T07:14:36.020 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:14:35.475+0000 7f2edd8b0700 1 -- 172.21.15.52:0/2564227453 >> [v2:172.21.15.52:6800/3513075805,v1:172.21.15.52:6801/3513075805] conn(0x7f2ec003e520 msgr2=0x7f2ec0040a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:14:36.020 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:14:35.475+0000 7f2edd8b0700 1 --2- 172.21.15.52:0/2564227453 >> [v2:172.21.15.52:6800/3513075805,v1:172.21.15.52:6801/3513075805] conn(0x7f2ec003e520 0x7f2ec0040a30 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f2ecc006fd0 tx=0x7f2ecc006e40 comp rx=0 tx=0).stop 2024-05-27T07:14:36.020 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:14:35.475+0000 7f2edd8b0700 1 -- 172.21.15.52:0/2564227453 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f2ed807ccb0 msgr2=0x7f2ed819df50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:14:36.020 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:14:35.475+0000 7f2edd8b0700 1 --2- 172.21.15.52:0/2564227453 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f2ed807ccb0 0x7f2ed819df50 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f2ec800bca0 tx=0x7f2ec800bd80 comp rx=0 tx=0).stop 2024-05-27T07:14:36.021 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:14:35.475+0000 7f2edd8b0700 1 -- 172.21.15.52:0/2564227453 shutdown_connections 2024-05-27T07:14:36.021 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:14:35.475+0000 7f2edd8b0700 1 --2- 172.21.15.52:0/2564227453 >> [v2:172.21.15.52:6800/3513075805,v1:172.21.15.52:6801/3513075805] conn(0x7f2ec003e520 0x7f2ec0040a30 unknown :-1 s=CLOSED pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:14:36.021 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:14:35.475+0000 7f2edd8b0700 1 --2- 172.21.15.52:0/2564227453 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f2ed807ccb0 0x7f2ed819df50 unknown :-1 s=CLOSED pgs=75 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:14:36.021 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:14:35.475+0000 7f2edd8b0700 1 -- 172.21.15.52:0/2564227453 >> 172.21.15.52:0/2564227453 conn(0x7f2ed81040e0 msgr2=0x7f2ed8104a30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:14:36.021 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:14:35.475+0000 7f2edd8b0700 1 -- 172.21.15.52:0/2564227453 shutdown_connections 2024-05-27T07:14:36.021 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:14:35.475+0000 7f2edd8b0700 1 -- 172.21.15.52:0/2564227453 wait complete. 2024-05-27T07:14:36.024 INFO:teuthology.orchestra.run.smithi052.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2024-05-27T07:14:36.025 INFO:teuthology.orchestra.run.smithi052.stdout:systemctl: stdout disabled 2024-05-27T07:14:36.032 INFO:teuthology.orchestra.run.smithi052.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2024-05-27T07:14:36.032 INFO:teuthology.orchestra.run.smithi052.stdout:systemctl: stdout inactive 2024-05-27T07:14:36.033 INFO:teuthology.orchestra.run.smithi052.stdout:firewalld.service is not enabled 2024-05-27T07:14:36.033 INFO:teuthology.orchestra.run.smithi052.stdout:Not possible to open ports <[8443]>. firewalld.service is not available 2024-05-27T07:14:36.034 INFO:teuthology.orchestra.run.smithi052.stdout:Ceph Dashboard is now available at: 2024-05-27T07:14:36.034 INFO:teuthology.orchestra.run.smithi052.stdout: 2024-05-27T07:14:36.034 INFO:teuthology.orchestra.run.smithi052.stdout: URL: https://smithi052.front.sepia.ceph.com:8443/ 2024-05-27T07:14:36.034 INFO:teuthology.orchestra.run.smithi052.stdout: User: admin 2024-05-27T07:14:36.034 INFO:teuthology.orchestra.run.smithi052.stdout: Password: otxaunj73n 2024-05-27T07:14:36.034 INFO:teuthology.orchestra.run.smithi052.stdout: 2024-05-27T07:14:36.035 INFO:teuthology.orchestra.run.smithi052.stdout:Saving cluster configuration to /var/lib/ceph/73df2636-1bf8-11ef-bc9b-c7b262605968/config directory 2024-05-27T07:14:36.035 INFO:teuthology.orchestra.run.smithi052.stdout:Enabling autotune for osd_memory_target 2024-05-27T07:14:36.941 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:14:36 smithi052 ceph-mon[31955]: mgrmap e13: x(active, since 6s) 2024-05-27T07:14:37.206 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:14:36.429+0000 7f186e910700 1 Processor -- start 2024-05-27T07:14:37.206 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:14:36.429+0000 7f186e910700 1 -- start start 2024-05-27T07:14:37.207 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:14:36.429+0000 7f186e910700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f1868109bf0 0x7f186810a000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:14:37.207 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:14:36.430+0000 7f186e910700 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x7f186810a5f0 con 0x7f1868109bf0 2024-05-27T07:14:37.207 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:14:36.430+0000 7f1867fff700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f1868109bf0 0x7f186810a000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:14:37.207 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:14:36.430+0000 7f1867fff700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f1868109bf0 0x7f186810a000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.52:51628/0 (socket says 172.21.15.52:51628) 2024-05-27T07:14:37.207 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:14:36.430+0000 7f1867fff700 1 -- 172.21.15.52:0/763848737 learned_addr learned my addr 172.21.15.52:0/763848737 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-05-27T07:14:37.207 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:14:36.430+0000 7f1867fff700 1 -- 172.21.15.52:0/763848737 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f186810ad20 con 0x7f1868109bf0 2024-05-27T07:14:37.207 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:14:36.430+0000 7f1867fff700 1 --2- 172.21.15.52:0/763848737 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f1868109bf0 0x7f186810a000 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f18580098a0 tx=0x7f1858009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6996f7bec66bcd1e server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:14:37.207 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:14:36.430+0000 7f1866ffd700 1 -- 172.21.15.52:0/763848737 <== mon.0 v2:172.21.15.52:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f1858004160 con 0x7f1868109bf0 2024-05-27T07:14:37.207 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:14:36.430+0000 7f1866ffd700 1 -- 172.21.15.52:0/763848737 <== mon.0 v2:172.21.15.52:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f1858004300 con 0x7f1868109bf0 2024-05-27T07:14:37.207 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:14:36.430+0000 7f1866ffd700 1 -- 172.21.15.52:0/763848737 <== mon.0 v2:172.21.15.52:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f1858011a20 con 0x7f1868109bf0 2024-05-27T07:14:37.207 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:14:36.431+0000 7f186e910700 1 -- 172.21.15.52:0/763848737 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f1868109bf0 msgr2=0x7f186810a000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:14:37.208 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:14:36.431+0000 7f186e910700 1 --2- 172.21.15.52:0/763848737 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f1868109bf0 0x7f186810a000 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f18580098a0 tx=0x7f1858009bb0 comp rx=0 tx=0).stop 2024-05-27T07:14:37.208 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:14:36.431+0000 7f186e910700 1 -- 172.21.15.52:0/763848737 shutdown_connections 2024-05-27T07:14:37.208 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:14:36.431+0000 7f186e910700 1 --2- 172.21.15.52:0/763848737 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f1868109bf0 0x7f186810a000 unknown :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:14:37.208 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:14:36.431+0000 7f186e910700 1 -- 172.21.15.52:0/763848737 >> 172.21.15.52:0/763848737 conn(0x7f1868105210 msgr2=0x7f18681076e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:14:37.208 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:14:36.431+0000 7f186e910700 1 -- 172.21.15.52:0/763848737 shutdown_connections 2024-05-27T07:14:37.208 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:14:36.431+0000 7f186e910700 1 -- 172.21.15.52:0/763848737 wait complete. 2024-05-27T07:14:37.208 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:14:36.432+0000 7f186e910700 1 Processor -- start 2024-05-27T07:14:37.208 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:14:36.432+0000 7f186e910700 1 -- start start 2024-05-27T07:14:37.208 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:14:36.432+0000 7f186e910700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f1868109bf0 0x7f186807f4a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:14:37.208 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:14:36.432+0000 7f186e910700 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x7f186810e0d0 con 0x7f1868109bf0 2024-05-27T07:14:37.208 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:14:36.432+0000 7f1867fff700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f1868109bf0 0x7f186807f4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:14:37.208 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:14:36.433+0000 7f1867fff700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f1868109bf0 0x7f186807f4a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.52:51632/0 (socket says 172.21.15.52:51632) 2024-05-27T07:14:37.209 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:14:36.433+0000 7f1867fff700 1 -- 172.21.15.52:0/539292402 learned_addr learned my addr 172.21.15.52:0/539292402 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-05-27T07:14:37.209 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:14:36.433+0000 7f1867fff700 1 -- 172.21.15.52:0/539292402 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f186807f9e0 con 0x7f1868109bf0 2024-05-27T07:14:37.209 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:14:36.433+0000 7f1867fff700 1 --2- 172.21.15.52:0/539292402 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f1868109bf0 0x7f186807f4a0 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f1858003e70 tx=0x7f1858011aa0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:14:37.209 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:14:36.433+0000 7f18657fa700 1 -- 172.21.15.52:0/539292402 <== mon.0 v2:172.21.15.52:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f1858003af0 con 0x7f1868109bf0 2024-05-27T07:14:37.209 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:14:36.433+0000 7f18657fa700 1 -- 172.21.15.52:0/539292402 <== mon.0 v2:172.21.15.52:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f185801a8e0 con 0x7f1868109bf0 2024-05-27T07:14:37.209 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:14:36.433+0000 7f186e910700 1 -- 172.21.15.52:0/539292402 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f186807dc30 con 0x7f1868109bf0 2024-05-27T07:14:37.209 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:14:36.433+0000 7f18657fa700 1 -- 172.21.15.52:0/539292402 <== mon.0 v2:172.21.15.52:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f1858024870 con 0x7f1868109bf0 2024-05-27T07:14:37.209 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:14:36.433+0000 7f186e910700 1 -- 172.21.15.52:0/539292402 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f186807e110 con 0x7f1868109bf0 2024-05-27T07:14:37.209 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:14:36.434+0000 7f186e910700 1 -- 172.21.15.52:0/539292402 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1868064e10 con 0x7f1868109bf0 2024-05-27T07:14:37.209 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:14:36.434+0000 7f18657fa700 1 -- 172.21.15.52:0/539292402 <== mon.0 v2:172.21.15.52:3300/0 4 ==== mgrmap(e 13) ==== 50278+0+0 (secure 0 0 0) 0x7f1858022020 con 0x7f1868109bf0 2024-05-27T07:14:37.209 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:14:36.435+0000 7f18657fa700 1 --2- 172.21.15.52:0/539292402 >> [v2:172.21.15.52:6800/3513075805,v1:172.21.15.52:6801/3513075805] conn(0x7f185003e510 0x7f1850040a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:14:37.209 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:14:36.435+0000 7f18657fa700 1 -- 172.21.15.52:0/539292402 <== mon.0 v2:172.21.15.52:3300/0 5 ==== osd_map(3..3 src has 1..3) ==== 1155+0+0 (secure 0 0 0) 0x7f1858053b20 con 0x7f1868109bf0 2024-05-27T07:14:37.209 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:14:36.435+0000 7f18677fe700 1 --2- 172.21.15.52:0/539292402 >> [v2:172.21.15.52:6800/3513075805,v1:172.21.15.52:6801/3513075805] conn(0x7f185003e510 0x7f1850040a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:14:37.209 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:14:36.435+0000 7f18677fe700 1 --2- 172.21.15.52:0/539292402 >> [v2:172.21.15.52:6800/3513075805,v1:172.21.15.52:6801/3513075805] conn(0x7f185003e510 0x7f1850040a20 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f185c006fd0 tx=0x7f185c006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:14:37.210 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:14:36.440+0000 7f18657fa700 1 -- 172.21.15.52:0/539292402 <== mon.0 v2:172.21.15.52:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191963 (secure 0 0 0) 0x7f185802b300 con 0x7f1868109bf0 2024-05-27T07:14:37.210 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:14:36.654+0000 7f186e910700 1 -- 172.21.15.52:0/539292402 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_command([{prefix=config set, name=osd_memory_target_autotune}] v 0) -- 0x7f186807e830 con 0x7f1868109bf0 2024-05-27T07:14:37.210 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:14:36.655+0000 7f18657fa700 1 -- 172.21.15.52:0/539292402 <== mon.0 v2:172.21.15.52:3300/0 7 ==== mon_command_ack([{prefix=config set, name=osd_memory_target_autotune}]=0 v8)=0 v8) ==== 127+0+0 (secure 0 0 0) 0x7f185800e260 con 0x7f1868109bf0 2024-05-27T07:14:37.210 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:14:36.658+0000 7f186e910700 1 -- 172.21.15.52:0/539292402 >> [v2:172.21.15.52:6800/3513075805,v1:172.21.15.52:6801/3513075805] conn(0x7f185003e510 msgr2=0x7f1850040a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:14:37.210 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:14:36.658+0000 7f186e910700 1 --2- 172.21.15.52:0/539292402 >> [v2:172.21.15.52:6800/3513075805,v1:172.21.15.52:6801/3513075805] conn(0x7f185003e510 0x7f1850040a20 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f185c006fd0 tx=0x7f185c006e40 comp rx=0 tx=0).stop 2024-05-27T07:14:37.210 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:14:36.658+0000 7f186e910700 1 -- 172.21.15.52:0/539292402 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f1868109bf0 msgr2=0x7f186807f4a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:14:37.210 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:14:36.658+0000 7f186e910700 1 --2- 172.21.15.52:0/539292402 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f1868109bf0 0x7f186807f4a0 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f1858003e70 tx=0x7f1858011aa0 comp rx=0 tx=0).stop 2024-05-27T07:14:37.210 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:14:36.658+0000 7f186e910700 1 -- 172.21.15.52:0/539292402 shutdown_connections 2024-05-27T07:14:37.210 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:14:36.658+0000 7f186e910700 1 --2- 172.21.15.52:0/539292402 >> [v2:172.21.15.52:6800/3513075805,v1:172.21.15.52:6801/3513075805] conn(0x7f185003e510 0x7f1850040a20 unknown :-1 s=CLOSED pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:14:37.210 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:14:36.658+0000 7f186e910700 1 --2- 172.21.15.52:0/539292402 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f1868109bf0 0x7f186807f4a0 unknown :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:14:37.210 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:14:36.658+0000 7f186e910700 1 -- 172.21.15.52:0/539292402 >> 172.21.15.52:0/539292402 conn(0x7f1868105210 msgr2=0x7f1868105a90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:14:37.210 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:14:36.658+0000 7f186e910700 1 -- 172.21.15.52:0/539292402 shutdown_connections 2024-05-27T07:14:37.210 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:14:36.658+0000 7f186e910700 1 -- 172.21.15.52:0/539292402 wait complete. 2024-05-27T07:14:38.605 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:14:37.704+0000 7f44a984b700 1 Processor -- start 2024-05-27T07:14:38.605 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:14:37.704+0000 7f44a984b700 1 -- start start 2024-05-27T07:14:38.605 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:14:37.705+0000 7f44a984b700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f44a407cc90 0x7f44a410c6f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:14:38.605 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:14:37.705+0000 7f44a984b700 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x7f44a410ccc0 con 0x7f44a407cc90 2024-05-27T07:14:38.605 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:14:37.705+0000 7f44a2ffd700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f44a407cc90 0x7f44a410c6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:14:38.605 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:14:37.705+0000 7f44a2ffd700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f44a407cc90 0x7f44a410c6f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.52:51644/0 (socket says 172.21.15.52:51644) 2024-05-27T07:14:38.605 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:14:37.705+0000 7f44a2ffd700 1 -- 172.21.15.52:0/3059638771 learned_addr learned my addr 172.21.15.52:0/3059638771 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-05-27T07:14:38.605 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:14:37.705+0000 7f44a2ffd700 1 -- 172.21.15.52:0/3059638771 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f44a410ce40 con 0x7f44a407cc90 2024-05-27T07:14:38.605 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:14:37.705+0000 7f44a2ffd700 1 --2- 172.21.15.52:0/3059638771 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f44a407cc90 0x7f44a410c6f0 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f44940098a0 tx=0x7f4494009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=630f9b599f03623a server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:14:38.606 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:14:37.705+0000 7f44a1ffb700 1 -- 172.21.15.52:0/3059638771 <== mon.0 v2:172.21.15.52:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f4494004160 con 0x7f44a407cc90 2024-05-27T07:14:38.606 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:14:37.706+0000 7f44a1ffb700 1 -- 172.21.15.52:0/3059638771 <== mon.0 v2:172.21.15.52:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f4494004300 con 0x7f44a407cc90 2024-05-27T07:14:38.606 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:14:37.706+0000 7f44a1ffb700 1 -- 172.21.15.52:0/3059638771 <== mon.0 v2:172.21.15.52:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f4494011a20 con 0x7f44a407cc90 2024-05-27T07:14:38.606 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:14:37.706+0000 7f44a984b700 1 -- 172.21.15.52:0/3059638771 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f44a407cc90 msgr2=0x7f44a410c6f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:14:38.606 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:14:37.706+0000 7f44a984b700 1 --2- 172.21.15.52:0/3059638771 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f44a407cc90 0x7f44a410c6f0 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f44940098a0 tx=0x7f4494009bb0 comp rx=0 tx=0).stop 2024-05-27T07:14:38.606 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:14:37.706+0000 7f44a984b700 1 -- 172.21.15.52:0/3059638771 shutdown_connections 2024-05-27T07:14:38.606 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:14:37.706+0000 7f44a984b700 1 --2- 172.21.15.52:0/3059638771 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f44a407cc90 0x7f44a410c6f0 unknown :-1 s=CLOSED pgs=78 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:14:38.606 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:14:37.706+0000 7f44a984b700 1 -- 172.21.15.52:0/3059638771 >> 172.21.15.52:0/3059638771 conn(0x7f44a41040e0 msgr2=0x7f44a4106570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:14:38.606 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:14:37.706+0000 7f44a984b700 1 -- 172.21.15.52:0/3059638771 shutdown_connections 2024-05-27T07:14:38.606 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:14:37.707+0000 7f44a984b700 1 -- 172.21.15.52:0/3059638771 wait complete. 2024-05-27T07:14:38.606 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:14:37.707+0000 7f44a984b700 1 Processor -- start 2024-05-27T07:14:38.606 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:14:37.707+0000 7f44a984b700 1 -- start start 2024-05-27T07:14:38.607 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:14:37.708+0000 7f44a984b700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f44a407cc90 0x7f44a419dec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:14:38.607 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:14:37.708+0000 7f44a984b700 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x7f44a410df00 con 0x7f44a407cc90 2024-05-27T07:14:38.607 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:14:37.708+0000 7f44a2ffd700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f44a407cc90 0x7f44a419dec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:14:38.607 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:14:37.708+0000 7f44a2ffd700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f44a407cc90 0x7f44a419dec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.52:51652/0 (socket says 172.21.15.52:51652) 2024-05-27T07:14:38.607 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:14:37.708+0000 7f44a2ffd700 1 -- 172.21.15.52:0/2731997199 learned_addr learned my addr 172.21.15.52:0/2731997199 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-05-27T07:14:38.607 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:14:37.708+0000 7f44a2ffd700 1 -- 172.21.15.52:0/2731997199 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f44a419e400 con 0x7f44a407cc90 2024-05-27T07:14:38.607 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:14:37.708+0000 7f44a2ffd700 1 --2- 172.21.15.52:0/2731997199 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f44a407cc90 0x7f44a419dec0 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f4494006af0 tx=0x7f4494011aa0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:14:38.607 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:14:37.708+0000 7f44a8849700 1 -- 172.21.15.52:0/2731997199 <== mon.0 v2:172.21.15.52:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f4494003b20 con 0x7f44a407cc90 2024-05-27T07:14:38.607 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:14:37.708+0000 7f44a984b700 1 -- 172.21.15.52:0/2731997199 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f44a419e690 con 0x7f44a407cc90 2024-05-27T07:14:38.607 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:14:37.708+0000 7f44a8849700 1 -- 172.21.15.52:0/2731997199 <== mon.0 v2:172.21.15.52:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f449401b8f0 con 0x7f44a407cc90 2024-05-27T07:14:38.607 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:14:37.708+0000 7f44a8849700 1 -- 172.21.15.52:0/2731997199 <== mon.0 v2:172.21.15.52:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f44940238c0 con 0x7f44a407cc90 2024-05-27T07:14:38.607 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:14:37.708+0000 7f44a984b700 1 -- 172.21.15.52:0/2731997199 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f44a41a12a0 con 0x7f44a407cc90 2024-05-27T07:14:38.608 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:14:37.710+0000 7f44a984b700 1 -- 172.21.15.52:0/2731997199 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f44a4064e10 con 0x7f44a407cc90 2024-05-27T07:14:38.608 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:14:37.710+0000 7f44a8849700 1 -- 172.21.15.52:0/2731997199 <== mon.0 v2:172.21.15.52:3300/0 4 ==== mgrmap(e 13) ==== 50278+0+0 (secure 0 0 0) 0x7f4494023b70 con 0x7f44a407cc90 2024-05-27T07:14:38.608 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:14:37.710+0000 7f44a8849700 1 --2- 172.21.15.52:0/2731997199 >> [v2:172.21.15.52:6800/3513075805,v1:172.21.15.52:6801/3513075805] conn(0x7f449003e510 0x7f4490040a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:14:38.608 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:14:37.710+0000 7f44a8849700 1 -- 172.21.15.52:0/2731997199 <== mon.0 v2:172.21.15.52:3300/0 5 ==== osd_map(3..3 src has 1..3) ==== 1155+0+0 (secure 0 0 0) 0x7f4494054460 con 0x7f44a407cc90 2024-05-27T07:14:38.608 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:14:37.710+0000 7f44a27fc700 1 --2- 172.21.15.52:0/2731997199 >> [v2:172.21.15.52:6800/3513075805,v1:172.21.15.52:6801/3513075805] conn(0x7f449003e510 0x7f4490040a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:14:38.608 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:14:37.711+0000 7f44a27fc700 1 --2- 172.21.15.52:0/2731997199 >> [v2:172.21.15.52:6800/3513075805,v1:172.21.15.52:6801/3513075805] conn(0x7f449003e510 0x7f4490040a20 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7f448c006fd0 tx=0x7f448c006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:14:38.608 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:14:37.715+0000 7f44a8849700 1 -- 172.21.15.52:0/2731997199 <== mon.0 v2:172.21.15.52:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191963 (secure 0 0 0) 0x7f449401f030 con 0x7f44a407cc90 2024-05-27T07:14:38.608 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:14:38.048+0000 7f44a984b700 1 -- 172.21.15.52:0/2731997199 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_command([{prefix=config-key set, key=mgr/dashboard/cluster/status}] v 0) -- 0x7f44a41a1470 con 0x7f44a407cc90 2024-05-27T07:14:38.608 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:14:38.050+0000 7f44a8849700 1 -- 172.21.15.52:0/2731997199 <== mon.0 v2:172.21.15.52:3300/0 7 ==== mon_command_ack([{prefix=config-key set, key=mgr/dashboard/cluster/status}]=0 set mgr/dashboard/cluster/status v28)=0 set mgr/dashboard/cluster/status v28) ==== 153+0+0 (secure 0 0 0) 0x7f449401baa0 con 0x7f44a407cc90 2024-05-27T07:14:38.608 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:14:38.053+0000 7f44a984b700 1 -- 172.21.15.52:0/2731997199 >> [v2:172.21.15.52:6800/3513075805,v1:172.21.15.52:6801/3513075805] conn(0x7f449003e510 msgr2=0x7f4490040a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:14:38.608 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:14:38.053+0000 7f44a984b700 1 --2- 172.21.15.52:0/2731997199 >> [v2:172.21.15.52:6800/3513075805,v1:172.21.15.52:6801/3513075805] conn(0x7f449003e510 0x7f4490040a20 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7f448c006fd0 tx=0x7f448c006e40 comp rx=0 tx=0).stop 2024-05-27T07:14:38.608 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:14:38.053+0000 7f44a984b700 1 -- 172.21.15.52:0/2731997199 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f44a407cc90 msgr2=0x7f44a419dec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:14:38.608 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:14:38.053+0000 7f44a984b700 1 --2- 172.21.15.52:0/2731997199 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f44a407cc90 0x7f44a419dec0 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f4494006af0 tx=0x7f4494011aa0 comp rx=0 tx=0).stop 2024-05-27T07:14:38.609 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:14:38.053+0000 7f44a984b700 1 -- 172.21.15.52:0/2731997199 shutdown_connections 2024-05-27T07:14:38.609 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:14:38.053+0000 7f44a984b700 1 --2- 172.21.15.52:0/2731997199 >> [v2:172.21.15.52:6800/3513075805,v1:172.21.15.52:6801/3513075805] conn(0x7f449003e510 0x7f4490040a20 unknown :-1 s=CLOSED pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:14:38.609 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:14:38.053+0000 7f44a984b700 1 --2- 172.21.15.52:0/2731997199 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f44a407cc90 0x7f44a419dec0 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:14:38.609 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:14:38.053+0000 7f44a984b700 1 -- 172.21.15.52:0/2731997199 >> 172.21.15.52:0/2731997199 conn(0x7f44a41040e0 msgr2=0x7f44a4104970 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:14:38.609 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:14:38.053+0000 7f44a984b700 1 -- 172.21.15.52:0/2731997199 shutdown_connections 2024-05-27T07:14:38.609 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr 2024-05-27T07:14:38.053+0000 7f44a984b700 1 -- 172.21.15.52:0/2731997199 wait complete. 2024-05-27T07:14:38.609 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr set mgr/dashboard/cluster/status 2024-05-27T07:14:38.609 INFO:teuthology.orchestra.run.smithi052.stdout:You can access the Ceph CLI as following in case of multi-cluster or non-default config: 2024-05-27T07:14:38.609 INFO:teuthology.orchestra.run.smithi052.stdout: 2024-05-27T07:14:38.609 INFO:teuthology.orchestra.run.smithi052.stdout: sudo /home/ubuntu/cephtest/cephadm shell --fsid 73df2636-1bf8-11ef-bc9b-c7b262605968 -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring 2024-05-27T07:14:38.609 INFO:teuthology.orchestra.run.smithi052.stdout: 2024-05-27T07:14:38.609 INFO:teuthology.orchestra.run.smithi052.stdout:Or, if you are only running a single cluster on this host: 2024-05-27T07:14:38.609 INFO:teuthology.orchestra.run.smithi052.stdout: 2024-05-27T07:14:38.610 INFO:teuthology.orchestra.run.smithi052.stdout: sudo /home/ubuntu/cephtest/cephadm shell 2024-05-27T07:14:38.610 INFO:teuthology.orchestra.run.smithi052.stdout: 2024-05-27T07:14:38.610 INFO:teuthology.orchestra.run.smithi052.stdout:Please consider enabling telemetry to help improve Ceph: 2024-05-27T07:14:38.610 INFO:teuthology.orchestra.run.smithi052.stdout: 2024-05-27T07:14:38.610 INFO:teuthology.orchestra.run.smithi052.stdout: ceph telemetry on 2024-05-27T07:14:38.610 INFO:teuthology.orchestra.run.smithi052.stdout: 2024-05-27T07:14:38.610 INFO:teuthology.orchestra.run.smithi052.stdout:For more information see: 2024-05-27T07:14:38.610 INFO:teuthology.orchestra.run.smithi052.stdout: 2024-05-27T07:14:38.610 INFO:teuthology.orchestra.run.smithi052.stdout: https://docs.ceph.com/en/latest/mgr/telemetry/ 2024-05-27T07:14:38.610 INFO:teuthology.orchestra.run.smithi052.stdout: 2024-05-27T07:14:38.610 INFO:teuthology.orchestra.run.smithi052.stdout:Bootstrap complete. 2024-05-27T07:14:38.648 INFO:tasks.cephadm:Fetching config... 2024-05-27T07:14:38.648 DEBUG:teuthology.orchestra.run.smithi052:> set -ex 2024-05-27T07:14:38.648 DEBUG:teuthology.orchestra.run.smithi052:> dd if=/etc/ceph/ceph.conf of=/dev/stdout 2024-05-27T07:14:38.666 INFO:tasks.cephadm:Fetching client.admin keyring... 2024-05-27T07:14:38.666 DEBUG:teuthology.orchestra.run.smithi052:> set -ex 2024-05-27T07:14:38.666 DEBUG:teuthology.orchestra.run.smithi052:> dd if=/etc/ceph/ceph.client.admin.keyring of=/dev/stdout 2024-05-27T07:14:38.723 INFO:tasks.cephadm:Fetching mon keyring... 2024-05-27T07:14:38.723 DEBUG:teuthology.orchestra.run.smithi052:> set -ex 2024-05-27T07:14:38.723 DEBUG:teuthology.orchestra.run.smithi052:> sudo dd if=/var/lib/ceph/73df2636-1bf8-11ef-bc9b-c7b262605968/mon.a/keyring of=/dev/stdout 2024-05-27T07:14:38.789 INFO:tasks.cephadm:Fetching pub ssh key... 2024-05-27T07:14:38.790 DEBUG:teuthology.orchestra.run.smithi052:> set -ex 2024-05-27T07:14:38.790 DEBUG:teuthology.orchestra.run.smithi052:> dd if=/home/ubuntu/cephtest/ceph.pub of=/dev/stdout 2024-05-27T07:14:38.845 INFO:tasks.cephadm:Installing pub ssh key for root users... 2024-05-27T07:14:38.846 DEBUG:teuthology.orchestra.run.smithi052:> sudo install -d -m 0700 /root/.ssh && echo 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQC8tboI03nnOKokdr00wKB0FvrqUHx08zkT/k0G/NBfqsAHGFHhJ3WHSvjyWs4J/6wdrcn7I/JdgB9WHq9Up6JGMwlkkyeu9e5fgMTv5QrrXYyNATTqZCZUMlJuiyBbkYLg49LVJiL8K2asXGcfiGYi3kFBvJf4gvfhJ8vqeVDSUpjAezyqZr/1HlV0MHPufOyl5irX2pxE2l/j6tBzu6Zk1ZMg63PqeX77ul/Tp0HSWVCvt19VGKUEXy9o2pYecB5BNGZEhtxO+1MCMLDomNcrRsGlpgxyj1JH7lVk3lBY/Xv4rMX6i2RJYj3XLFjdKmFSWFjlskRBsVcxRBjgC7vsnVpeabmvktcHTw+1di5HnJws0eBFY1OFWYUxQ2ICjOUP6SfpxAhbjKXFIMecFNGEvC9EdGSNBXIePJRYslarfbPuuxln18VF/rwNskWXxhut1aRWJqRD3baIAsVAIPoYx8GoN37YGEt6oXWBndiYespoelTAeRJu6RiepzpRwFk= ceph-73df2636-1bf8-11ef-bc9b-c7b262605968' | sudo tee -a /root/.ssh/authorized_keys && sudo chmod 0600 /root/.ssh/authorized_keys 2024-05-27T07:14:38.920 INFO:teuthology.orchestra.run.smithi052.stdout:ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQC8tboI03nnOKokdr00wKB0FvrqUHx08zkT/k0G/NBfqsAHGFHhJ3WHSvjyWs4J/6wdrcn7I/JdgB9WHq9Up6JGMwlkkyeu9e5fgMTv5QrrXYyNATTqZCZUMlJuiyBbkYLg49LVJiL8K2asXGcfiGYi3kFBvJf4gvfhJ8vqeVDSUpjAezyqZr/1HlV0MHPufOyl5irX2pxE2l/j6tBzu6Zk1ZMg63PqeX77ul/Tp0HSWVCvt19VGKUEXy9o2pYecB5BNGZEhtxO+1MCMLDomNcrRsGlpgxyj1JH7lVk3lBY/Xv4rMX6i2RJYj3XLFjdKmFSWFjlskRBsVcxRBjgC7vsnVpeabmvktcHTw+1di5HnJws0eBFY1OFWYUxQ2ICjOUP6SfpxAhbjKXFIMecFNGEvC9EdGSNBXIePJRYslarfbPuuxln18VF/rwNskWXxhut1aRWJqRD3baIAsVAIPoYx8GoN37YGEt6oXWBndiYespoelTAeRJu6RiepzpRwFk= ceph-73df2636-1bf8-11ef-bc9b-c7b262605968 2024-05-27T07:14:38.932 DEBUG:teuthology.orchestra.run.smithi105:> sudo install -d -m 0700 /root/.ssh && echo 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQC8tboI03nnOKokdr00wKB0FvrqUHx08zkT/k0G/NBfqsAHGFHhJ3WHSvjyWs4J/6wdrcn7I/JdgB9WHq9Up6JGMwlkkyeu9e5fgMTv5QrrXYyNATTqZCZUMlJuiyBbkYLg49LVJiL8K2asXGcfiGYi3kFBvJf4gvfhJ8vqeVDSUpjAezyqZr/1HlV0MHPufOyl5irX2pxE2l/j6tBzu6Zk1ZMg63PqeX77ul/Tp0HSWVCvt19VGKUEXy9o2pYecB5BNGZEhtxO+1MCMLDomNcrRsGlpgxyj1JH7lVk3lBY/Xv4rMX6i2RJYj3XLFjdKmFSWFjlskRBsVcxRBjgC7vsnVpeabmvktcHTw+1di5HnJws0eBFY1OFWYUxQ2ICjOUP6SfpxAhbjKXFIMecFNGEvC9EdGSNBXIePJRYslarfbPuuxln18VF/rwNskWXxhut1aRWJqRD3baIAsVAIPoYx8GoN37YGEt6oXWBndiYespoelTAeRJu6RiepzpRwFk= ceph-73df2636-1bf8-11ef-bc9b-c7b262605968' | sudo tee -a /root/.ssh/authorized_keys && sudo chmod 0600 /root/.ssh/authorized_keys 2024-05-27T07:14:38.971 INFO:teuthology.orchestra.run.smithi105.stdout:ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQC8tboI03nnOKokdr00wKB0FvrqUHx08zkT/k0G/NBfqsAHGFHhJ3WHSvjyWs4J/6wdrcn7I/JdgB9WHq9Up6JGMwlkkyeu9e5fgMTv5QrrXYyNATTqZCZUMlJuiyBbkYLg49LVJiL8K2asXGcfiGYi3kFBvJf4gvfhJ8vqeVDSUpjAezyqZr/1HlV0MHPufOyl5irX2pxE2l/j6tBzu6Zk1ZMg63PqeX77ul/Tp0HSWVCvt19VGKUEXy9o2pYecB5BNGZEhtxO+1MCMLDomNcrRsGlpgxyj1JH7lVk3lBY/Xv4rMX6i2RJYj3XLFjdKmFSWFjlskRBsVcxRBjgC7vsnVpeabmvktcHTw+1di5HnJws0eBFY1OFWYUxQ2ICjOUP6SfpxAhbjKXFIMecFNGEvC9EdGSNBXIePJRYslarfbPuuxln18VF/rwNskWXxhut1aRWJqRD3baIAsVAIPoYx8GoN37YGEt6oXWBndiYespoelTAeRJu6RiepzpRwFk= ceph-73df2636-1bf8-11ef-bc9b-c7b262605968 2024-05-27T07:14:38.982 DEBUG:teuthology.orchestra.run.smithi173:> sudo install -d -m 0700 /root/.ssh && echo 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQC8tboI03nnOKokdr00wKB0FvrqUHx08zkT/k0G/NBfqsAHGFHhJ3WHSvjyWs4J/6wdrcn7I/JdgB9WHq9Up6JGMwlkkyeu9e5fgMTv5QrrXYyNATTqZCZUMlJuiyBbkYLg49LVJiL8K2asXGcfiGYi3kFBvJf4gvfhJ8vqeVDSUpjAezyqZr/1HlV0MHPufOyl5irX2pxE2l/j6tBzu6Zk1ZMg63PqeX77ul/Tp0HSWVCvt19VGKUEXy9o2pYecB5BNGZEhtxO+1MCMLDomNcrRsGlpgxyj1JH7lVk3lBY/Xv4rMX6i2RJYj3XLFjdKmFSWFjlskRBsVcxRBjgC7vsnVpeabmvktcHTw+1di5HnJws0eBFY1OFWYUxQ2ICjOUP6SfpxAhbjKXFIMecFNGEvC9EdGSNBXIePJRYslarfbPuuxln18VF/rwNskWXxhut1aRWJqRD3baIAsVAIPoYx8GoN37YGEt6oXWBndiYespoelTAeRJu6RiepzpRwFk= ceph-73df2636-1bf8-11ef-bc9b-c7b262605968' | sudo tee -a /root/.ssh/authorized_keys && sudo chmod 0600 /root/.ssh/authorized_keys 2024-05-27T07:14:39.021 INFO:teuthology.orchestra.run.smithi173.stdout:ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQC8tboI03nnOKokdr00wKB0FvrqUHx08zkT/k0G/NBfqsAHGFHhJ3WHSvjyWs4J/6wdrcn7I/JdgB9WHq9Up6JGMwlkkyeu9e5fgMTv5QrrXYyNATTqZCZUMlJuiyBbkYLg49LVJiL8K2asXGcfiGYi3kFBvJf4gvfhJ8vqeVDSUpjAezyqZr/1HlV0MHPufOyl5irX2pxE2l/j6tBzu6Zk1ZMg63PqeX77ul/Tp0HSWVCvt19VGKUEXy9o2pYecB5BNGZEhtxO+1MCMLDomNcrRsGlpgxyj1JH7lVk3lBY/Xv4rMX6i2RJYj3XLFjdKmFSWFjlskRBsVcxRBjgC7vsnVpeabmvktcHTw+1di5HnJws0eBFY1OFWYUxQ2ICjOUP6SfpxAhbjKXFIMecFNGEvC9EdGSNBXIePJRYslarfbPuuxln18VF/rwNskWXxhut1aRWJqRD3baIAsVAIPoYx8GoN37YGEt6oXWBndiYespoelTAeRJu6RiepzpRwFk= ceph-73df2636-1bf8-11ef-bc9b-c7b262605968 2024-05-27T07:14:39.101 DEBUG:teuthology.orchestra.run.smithi052:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:38f9d900c83c63799fbdbe61acc9a11b0d3554a6 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 73df2636-1bf8-11ef-bc9b-c7b262605968 -- ceph config set mgr mgr/cephadm/allow_ptrace true 2024-05-27T07:14:39.230 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:14:39 smithi052 ceph-mon[31955]: from='client.? 172.21.15.52:0/2731997199' entity='client.admin' 2024-05-27T07:14:39.354 INFO:teuthology.orchestra.run.smithi052.stderr:Inferring config /var/lib/ceph/73df2636-1bf8-11ef-bc9b-c7b262605968/mon.a/config 2024-05-27T07:14:41.844 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:14:41 smithi052 ceph-mon[31955]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' 2024-05-27T07:14:41.845 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:14:41 smithi052 ceph-mon[31955]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' 2024-05-27T07:14:41.845 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:14:41 smithi052 ceph-mon[31955]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd/host:smithi052", "name": "osd_memory_target"}]: dispatch 2024-05-27T07:14:41.845 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:14:41 smithi052 ceph-mon[31955]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' 2024-05-27T07:14:41.845 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:14:41 smithi052 ceph-mon[31955]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-27T07:14:41.845 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:14:41 smithi052 ceph-mon[31955]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' 2024-05-27T07:14:41.882 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:14:41.890+0000 7f12dd4b9700 1 -- 172.21.15.52:0/676695162 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f12d8102240 msgr2=0x7f12d8102650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:14:41.882 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:14:41.891+0000 7f12dd4b9700 1 --2- 172.21.15.52:0/676695162 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f12d8102240 0x7f12d8102650 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f12c8009910 tx=0x7f12c8009c20 comp rx=0 tx=0).stop 2024-05-27T07:14:41.882 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:14:41.891+0000 7f12dd4b9700 1 -- 172.21.15.52:0/676695162 shutdown_connections 2024-05-27T07:14:41.882 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:14:41.891+0000 7f12dd4b9700 1 --2- 172.21.15.52:0/676695162 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f12d8102240 0x7f12d8102650 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:14:41.883 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:14:41.891+0000 7f12dd4b9700 1 -- 172.21.15.52:0/676695162 >> 172.21.15.52:0/676695162 conn(0x7f12d80fd880 msgr2=0x7f12d80ffd30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:14:41.883 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:14:41.891+0000 7f12dd4b9700 1 -- 172.21.15.52:0/676695162 shutdown_connections 2024-05-27T07:14:41.883 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:14:41.891+0000 7f12dd4b9700 1 -- 172.21.15.52:0/676695162 wait complete. 2024-05-27T07:14:41.883 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:14:41.891+0000 7f12dd4b9700 1 Processor -- start 2024-05-27T07:14:41.884 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:14:41.892+0000 7f12dd4b9700 1 -- start start 2024-05-27T07:14:41.884 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:14:41.892+0000 7f12dd4b9700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f12d8102240 0x7f12d80fe6a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:14:41.884 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:14:41.892+0000 7f12dd4b9700 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x7f12d8108200 con 0x7f12d8102240 2024-05-27T07:14:41.884 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:14:41.892+0000 7f12d6ffd700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f12d8102240 0x7f12d80fe6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:14:41.884 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:14:41.892+0000 7f12d6ffd700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f12d8102240 0x7f12d80fe6a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.52:58646/0 (socket says 172.21.15.52:58646) 2024-05-27T07:14:41.884 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:14:41.892+0000 7f12d6ffd700 1 -- 172.21.15.52:0/2384527857 learned_addr learned my addr 172.21.15.52:0/2384527857 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-05-27T07:14:41.885 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:14:41.893+0000 7f12d6ffd700 1 -- 172.21.15.52:0/2384527857 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f12d80febe0 con 0x7f12d8102240 2024-05-27T07:14:41.885 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:14:41.893+0000 7f12d6ffd700 1 --2- 172.21.15.52:0/2384527857 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f12d8102240 0x7f12d80fe6a0 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f12c80057d0 tx=0x7f12c8005800 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:14:41.885 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:14:41.893+0000 7f12bffff700 1 -- 172.21.15.52:0/2384527857 <== mon.0 v2:172.21.15.52:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f12c8027070 con 0x7f12d8102240 2024-05-27T07:14:41.885 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:14:41.893+0000 7f12bffff700 1 -- 172.21.15.52:0/2384527857 <== mon.0 v2:172.21.15.52:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f12c800be10 con 0x7f12d8102240 2024-05-27T07:14:41.885 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:14:41.893+0000 7f12dd4b9700 1 -- 172.21.15.52:0/2384527857 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f12d819c4c0 con 0x7f12d8102240 2024-05-27T07:14:41.885 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:14:41.893+0000 7f12bffff700 1 -- 172.21.15.52:0/2384527857 <== mon.0 v2:172.21.15.52:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f12c8022dd0 con 0x7f12d8102240 2024-05-27T07:14:41.885 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:14:41.893+0000 7f12dd4b9700 1 -- 172.21.15.52:0/2384527857 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f12d819c940 con 0x7f12d8102240 2024-05-27T07:14:41.886 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:14:41.894+0000 7f12dd4b9700 1 -- 172.21.15.52:0/2384527857 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f12d8108980 con 0x7f12d8102240 2024-05-27T07:14:41.886 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:14:41.894+0000 7f12bffff700 1 -- 172.21.15.52:0/2384527857 <== mon.0 v2:172.21.15.52:3300/0 4 ==== mgrmap(e 13) ==== 50278+0+0 (secure 0 0 0) 0x7f12c802a070 con 0x7f12d8102240 2024-05-27T07:14:41.886 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:14:41.895+0000 7f12bffff700 1 --2- 172.21.15.52:0/2384527857 >> [v2:172.21.15.52:6800/3513075805,v1:172.21.15.52:6801/3513075805] conn(0x7f12c003e4c0 0x7f12c00409d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:14:41.886 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:14:41.895+0000 7f12bffff700 1 -- 172.21.15.52:0/2384527857 <== mon.0 v2:172.21.15.52:3300/0 5 ==== osd_map(3..3 src has 1..3) ==== 1155+0+0 (secure 0 0 0) 0x7f12c801e030 con 0x7f12d8102240 2024-05-27T07:14:41.886 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:14:41.895+0000 7f12d67fc700 1 --2- 172.21.15.52:0/2384527857 >> [v2:172.21.15.52:6800/3513075805,v1:172.21.15.52:6801/3513075805] conn(0x7f12c003e4c0 0x7f12c00409d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:14:41.886 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:14:41.895+0000 7f12d67fc700 1 --2- 172.21.15.52:0/2384527857 >> [v2:172.21.15.52:6800/3513075805,v1:172.21.15.52:6801/3513075805] conn(0x7f12c003e4c0 0x7f12c00409d0 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7f12cc006fd0 tx=0x7f12cc006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:14:41.891 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:14:41.900+0000 7f12bffff700 1 -- 172.21.15.52:0/2384527857 <== mon.0 v2:172.21.15.52:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191963 (secure 0 0 0) 0x7f12c8013070 con 0x7f12d8102240 2024-05-27T07:14:42.174 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:14:42.183+0000 7f12dd4b9700 1 -- 172.21.15.52:0/2384527857 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_command([{prefix=config set, name=mgr/cephadm/allow_ptrace}] v 0) -- 0x7f12d819cbc0 con 0x7f12d8102240 2024-05-27T07:14:42.177 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:14:42.186+0000 7f12bffff700 1 -- 172.21.15.52:0/2384527857 <== mon.0 v2:172.21.15.52:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/cephadm/allow_ptrace}]=0 v9)=0 v9) ==== 125+0+0 (secure 0 0 0) 0x7f12c80292c0 con 0x7f12d8102240 2024-05-27T07:14:42.180 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:14:42.189+0000 7f12dd4b9700 1 -- 172.21.15.52:0/2384527857 >> [v2:172.21.15.52:6800/3513075805,v1:172.21.15.52:6801/3513075805] conn(0x7f12c003e4c0 msgr2=0x7f12c00409d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:14:42.180 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:14:42.189+0000 7f12dd4b9700 1 --2- 172.21.15.52:0/2384527857 >> [v2:172.21.15.52:6800/3513075805,v1:172.21.15.52:6801/3513075805] conn(0x7f12c003e4c0 0x7f12c00409d0 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7f12cc006fd0 tx=0x7f12cc006e40 comp rx=0 tx=0).stop 2024-05-27T07:14:42.180 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:14:42.189+0000 7f12dd4b9700 1 -- 172.21.15.52:0/2384527857 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f12d8102240 msgr2=0x7f12d80fe6a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:14:42.180 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:14:42.189+0000 7f12dd4b9700 1 --2- 172.21.15.52:0/2384527857 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f12d8102240 0x7f12d80fe6a0 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f12c80057d0 tx=0x7f12c8005800 comp rx=0 tx=0).stop 2024-05-27T07:14:42.180 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:14:42.189+0000 7f12dd4b9700 1 -- 172.21.15.52:0/2384527857 shutdown_connections 2024-05-27T07:14:42.180 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:14:42.189+0000 7f12dd4b9700 1 --2- 172.21.15.52:0/2384527857 >> [v2:172.21.15.52:6800/3513075805,v1:172.21.15.52:6801/3513075805] conn(0x7f12c003e4c0 0x7f12c00409d0 unknown :-1 s=CLOSED pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:14:42.180 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:14:42.189+0000 7f12dd4b9700 1 --2- 172.21.15.52:0/2384527857 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f12d8102240 0x7f12d80fe6a0 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:14:42.181 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:14:42.189+0000 7f12dd4b9700 1 -- 172.21.15.52:0/2384527857 >> 172.21.15.52:0/2384527857 conn(0x7f12d80fd880 msgr2=0x7f12d80fee90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:14:42.181 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:14:42.189+0000 7f12dd4b9700 1 -- 172.21.15.52:0/2384527857 shutdown_connections 2024-05-27T07:14:42.181 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:14:42.189+0000 7f12dd4b9700 1 -- 172.21.15.52:0/2384527857 wait complete. 2024-05-27T07:14:43.245 INFO:tasks.cephadm:Distributing conf and client.admin keyring to all hosts + 0755 2024-05-27T07:14:43.245 DEBUG:teuthology.orchestra.run.smithi052:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:38f9d900c83c63799fbdbe61acc9a11b0d3554a6 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 73df2636-1bf8-11ef-bc9b-c7b262605968 -- ceph orch client-keyring set client.admin '*' --mode 0755 2024-05-27T07:14:43.480 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:14:43 smithi052 ceph-mon[31955]: from='client.? 172.21.15.52:0/2384527857' entity='client.admin' 2024-05-27T07:14:43.480 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:14:43 smithi052 ceph-mon[31955]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-27T07:14:43.480 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:14:43 smithi052 ceph-mon[31955]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' 2024-05-27T07:14:43.497 INFO:teuthology.orchestra.run.smithi052.stderr:Inferring config /var/lib/ceph/73df2636-1bf8-11ef-bc9b-c7b262605968/mon.a/config 2024-05-27T07:14:45.024 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:14:45.033+0000 7ff3c7727700 1 -- 172.21.15.52:0/1392688472 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff3c0102b00 msgr2=0x7ff3c0102f10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:14:45.025 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:14:45.033+0000 7ff3c7727700 1 --2- 172.21.15.52:0/1392688472 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff3c0102b00 0x7ff3c0102f10 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7ff3bc009910 tx=0x7ff3bc009c20 comp rx=0 tx=0).stop 2024-05-27T07:14:45.025 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:14:45.034+0000 7ff3c7727700 1 -- 172.21.15.52:0/1392688472 shutdown_connections 2024-05-27T07:14:45.025 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:14:45.034+0000 7ff3c7727700 1 --2- 172.21.15.52:0/1392688472 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff3c0102b00 0x7ff3c0102f10 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:14:45.025 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:14:45.034+0000 7ff3c7727700 1 -- 172.21.15.52:0/1392688472 >> 172.21.15.52:0/1392688472 conn(0x7ff3c00fe120 msgr2=0x7ff3c01005f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:14:45.025 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:14:45.034+0000 7ff3c7727700 1 -- 172.21.15.52:0/1392688472 shutdown_connections 2024-05-27T07:14:45.025 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:14:45.034+0000 7ff3c7727700 1 -- 172.21.15.52:0/1392688472 wait complete. 2024-05-27T07:14:45.027 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:14:45.034+0000 7ff3c7727700 1 Processor -- start 2024-05-27T07:14:45.027 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:14:45.034+0000 7ff3c7727700 1 -- start start 2024-05-27T07:14:45.027 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:14:45.035+0000 7ff3c7727700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff3c0102b00 0x7ff3c006b780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:14:45.027 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:14:45.035+0000 7ff3c7727700 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x7ff3c0108ac0 con 0x7ff3c0102b00 2024-05-27T07:14:45.027 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:14:45.035+0000 7ff3c54c3700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff3c0102b00 0x7ff3c006b780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:14:45.027 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:14:45.035+0000 7ff3c54c3700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff3c0102b00 0x7ff3c006b780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.52:58662/0 (socket says 172.21.15.52:58662) 2024-05-27T07:14:45.027 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:14:45.035+0000 7ff3c54c3700 1 -- 172.21.15.52:0/2887235647 learned_addr learned my addr 172.21.15.52:0/2887235647 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-05-27T07:14:45.027 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:14:45.035+0000 7ff3c54c3700 1 -- 172.21.15.52:0/2887235647 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff3c006bcc0 con 0x7ff3c0102b00 2024-05-27T07:14:45.028 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:14:45.035+0000 7ff3c54c3700 1 --2- 172.21.15.52:0/2887235647 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff3c0102b00 0x7ff3c006b780 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7ff3bc0057d0 tx=0x7ff3bc005800 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:14:45.028 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:14:45.035+0000 7ff3b67fc700 1 -- 172.21.15.52:0/2887235647 <== mon.0 v2:172.21.15.52:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7ff3bc027070 con 0x7ff3c0102b00 2024-05-27T07:14:45.028 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:14:45.035+0000 7ff3b67fc700 1 -- 172.21.15.52:0/2887235647 <== mon.0 v2:172.21.15.52:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ff3bc00be10 con 0x7ff3c0102b00 2024-05-27T07:14:45.028 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:14:45.035+0000 7ff3c7727700 1 -- 172.21.15.52:0/2887235647 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff3c019c4b0 con 0x7ff3c0102b00 2024-05-27T07:14:45.028 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:14:45.035+0000 7ff3b67fc700 1 -- 172.21.15.52:0/2887235647 <== mon.0 v2:172.21.15.52:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7ff3bc022dd0 con 0x7ff3c0102b00 2024-05-27T07:14:45.028 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:14:45.035+0000 7ff3c7727700 1 -- 172.21.15.52:0/2887235647 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff3c019c930 con 0x7ff3c0102b00 2024-05-27T07:14:45.028 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:14:45.036+0000 7ff3b67fc700 1 -- 172.21.15.52:0/2887235647 <== mon.0 v2:172.21.15.52:3300/0 4 ==== mgrmap(e 13) ==== 50278+0+0 (secure 0 0 0) 0x7ff3bc02a070 con 0x7ff3c0102b00 2024-05-27T07:14:45.029 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:14:45.037+0000 7ff3c7727700 1 -- 172.21.15.52:0/2887235647 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff3c0109240 con 0x7ff3c0102b00 2024-05-27T07:14:45.029 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:14:45.037+0000 7ff3b67fc700 1 --2- 172.21.15.52:0/2887235647 >> [v2:172.21.15.52:6800/3513075805,v1:172.21.15.52:6801/3513075805] conn(0x7ff3ac03e510 0x7ff3ac040a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:14:45.029 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:14:45.037+0000 7ff3b67fc700 1 -- 172.21.15.52:0/2887235647 <== mon.0 v2:172.21.15.52:3300/0 5 ==== osd_map(3..3 src has 1..3) ==== 1155+0+0 (secure 0 0 0) 0x7ff3bc01e030 con 0x7ff3c0102b00 2024-05-27T07:14:45.029 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:14:45.037+0000 7ff3c4cc2700 1 --2- 172.21.15.52:0/2887235647 >> [v2:172.21.15.52:6800/3513075805,v1:172.21.15.52:6801/3513075805] conn(0x7ff3ac03e510 0x7ff3ac040a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:14:45.029 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:14:45.037+0000 7ff3c4cc2700 1 --2- 172.21.15.52:0/2887235647 >> [v2:172.21.15.52:6800/3513075805,v1:172.21.15.52:6801/3513075805] conn(0x7ff3ac03e510 0x7ff3ac040a20 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7ff3b0006fd0 tx=0x7ff3b0006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:14:45.033 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:14:45.042+0000 7ff3b67fc700 1 -- 172.21.15.52:0/2887235647 <== mon.0 v2:172.21.15.52:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191963 (secure 0 0 0) 0x7ff3bc013070 con 0x7ff3c0102b00 2024-05-27T07:14:45.250 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:14:45.260+0000 7ff3c7727700 1 -- 172.21.15.52:0/2887235647 --> [v2:172.21.15.52:6800/3513075805,v1:172.21.15.52:6801/3513075805] -- mgr_command(tid 0: {"prefix": "orch client-keyring set", "entity": "client.admin", "placement": "*", "mode": "0755", "target": ["mon-mgr", ""]}) -- 0x7ff3c002e880 con 0x7ff3ac03e510 2024-05-27T07:14:45.254 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:14:45.264+0000 7ff3b67fc700 1 -- 172.21.15.52:0/2887235647 <== mgr.14152 v2:172.21.15.52:6800/3513075805 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+0 (secure 0 0 0) 0x7ff3c002e880 con 0x7ff3ac03e510 2024-05-27T07:14:45.257 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:14:45.266+0000 7ff3c7727700 1 -- 172.21.15.52:0/2887235647 >> [v2:172.21.15.52:6800/3513075805,v1:172.21.15.52:6801/3513075805] conn(0x7ff3ac03e510 msgr2=0x7ff3ac040a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:14:45.257 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:14:45.266+0000 7ff3c7727700 1 --2- 172.21.15.52:0/2887235647 >> [v2:172.21.15.52:6800/3513075805,v1:172.21.15.52:6801/3513075805] conn(0x7ff3ac03e510 0x7ff3ac040a20 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7ff3b0006fd0 tx=0x7ff3b0006e40 comp rx=0 tx=0).stop 2024-05-27T07:14:45.257 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:14:45.266+0000 7ff3c7727700 1 -- 172.21.15.52:0/2887235647 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff3c0102b00 msgr2=0x7ff3c006b780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:14:45.257 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:14:45.266+0000 7ff3c7727700 1 --2- 172.21.15.52:0/2887235647 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff3c0102b00 0x7ff3c006b780 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7ff3bc0057d0 tx=0x7ff3bc005800 comp rx=0 tx=0).stop 2024-05-27T07:14:45.257 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:14:45.266+0000 7ff3c7727700 1 -- 172.21.15.52:0/2887235647 shutdown_connections 2024-05-27T07:14:45.257 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:14:45.266+0000 7ff3c7727700 1 --2- 172.21.15.52:0/2887235647 >> [v2:172.21.15.52:6800/3513075805,v1:172.21.15.52:6801/3513075805] conn(0x7ff3ac03e510 0x7ff3ac040a20 unknown :-1 s=CLOSED pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:14:45.257 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:14:45.266+0000 7ff3c7727700 1 --2- 172.21.15.52:0/2887235647 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff3c0102b00 0x7ff3c006b780 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:14:45.257 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:14:45.266+0000 7ff3c7727700 1 -- 172.21.15.52:0/2887235647 >> 172.21.15.52:0/2887235647 conn(0x7ff3c00fe120 msgr2=0x7ff3c00fee40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:14:45.257 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:14:45.266+0000 7ff3c7727700 1 -- 172.21.15.52:0/2887235647 shutdown_connections 2024-05-27T07:14:45.257 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:14:45.266+0000 7ff3c7727700 1 -- 172.21.15.52:0/2887235647 wait complete. 2024-05-27T07:14:46.009 INFO:tasks.cephadm:Writing (initial) conf and keyring to smithi105 2024-05-27T07:14:46.009 DEBUG:teuthology.orchestra.run.smithi105:> set -ex 2024-05-27T07:14:46.009 DEBUG:teuthology.orchestra.run.smithi105:> dd of=/etc/ceph/ceph.conf 2024-05-27T07:14:46.028 DEBUG:teuthology.orchestra.run.smithi105:> set -ex 2024-05-27T07:14:46.028 DEBUG:teuthology.orchestra.run.smithi105:> dd of=/etc/ceph/ceph.client.admin.keyring 2024-05-27T07:14:46.085 INFO:tasks.cephadm:Adding host smithi105 to orchestrator... 2024-05-27T07:14:46.085 DEBUG:teuthology.orchestra.run.smithi052:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:38f9d900c83c63799fbdbe61acc9a11b0d3554a6 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 73df2636-1bf8-11ef-bc9b-c7b262605968 -- ceph orch host add smithi105 2024-05-27T07:14:46.334 INFO:teuthology.orchestra.run.smithi052.stderr:Inferring config /var/lib/ceph/73df2636-1bf8-11ef-bc9b-c7b262605968/mon.a/config 2024-05-27T07:14:46.553 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:14:46 smithi052 ceph-mon[31955]: from='client.14176 -' entity='client.admin' cmd=[{"prefix": "orch client-keyring set", "entity": "client.admin", "placement": "*", "mode": "0755", "target": ["mon-mgr", ""]}]: dispatch 2024-05-27T07:14:46.553 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:14:46 smithi052 ceph-mon[31955]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' 2024-05-27T07:14:46.553 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:14:46 smithi052 ceph-mon[31955]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-27T07:14:46.553 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:14:46 smithi052 ceph-mon[31955]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T07:14:46.553 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:14:46 smithi052 ceph-mon[31955]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-27T07:14:46.553 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:14:46 smithi052 ceph-mon[31955]: Updating smithi052:/etc/ceph/ceph.conf 2024-05-27T07:14:46.553 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:14:46 smithi052 ceph-mon[31955]: Updating smithi052:/var/lib/ceph/73df2636-1bf8-11ef-bc9b-c7b262605968/config/ceph.conf 2024-05-27T07:14:46.553 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:14:46 smithi052 ceph-mon[31955]: Updating smithi052:/etc/ceph/ceph.client.admin.keyring 2024-05-27T07:14:47.609 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:14:47 smithi052 ceph-mon[31955]: Updating smithi052:/var/lib/ceph/73df2636-1bf8-11ef-bc9b-c7b262605968/config/ceph.client.admin.keyring 2024-05-27T07:14:47.609 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:14:47 smithi052 ceph-mon[31955]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' 2024-05-27T07:14:47.609 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:14:47 smithi052 ceph-mon[31955]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' 2024-05-27T07:14:47.609 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:14:47 smithi052 ceph-mon[31955]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' 2024-05-27T07:14:47.803 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:14:47.812+0000 7f9767809700 1 -- 172.21.15.52:0/1113646322 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f9760104d10 msgr2=0x7f9760105120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:14:47.803 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:14:47.812+0000 7f9767809700 1 --2- 172.21.15.52:0/1113646322 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f9760104d10 0x7f9760105120 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f975c009910 tx=0x7f975c009c20 comp rx=0 tx=0).stop 2024-05-27T07:14:47.804 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:14:47.812+0000 7f9767809700 1 -- 172.21.15.52:0/1113646322 shutdown_connections 2024-05-27T07:14:47.804 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:14:47.812+0000 7f9767809700 1 --2- 172.21.15.52:0/1113646322 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f9760104d10 0x7f9760105120 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:14:47.804 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:14:47.812+0000 7f9767809700 1 -- 172.21.15.52:0/1113646322 >> 172.21.15.52:0/1113646322 conn(0x7f97600fe560 msgr2=0x7f97601009b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:14:47.804 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:14:47.812+0000 7f9767809700 1 -- 172.21.15.52:0/1113646322 shutdown_connections 2024-05-27T07:14:47.804 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:14:47.812+0000 7f9767809700 1 -- 172.21.15.52:0/1113646322 wait complete. 2024-05-27T07:14:47.804 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:14:47.813+0000 7f9767809700 1 Processor -- start 2024-05-27T07:14:47.804 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:14:47.813+0000 7f9767809700 1 -- start start 2024-05-27T07:14:47.805 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:14:47.813+0000 7f9767809700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f9760104d10 0x7f97601a1760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:14:47.805 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:14:47.813+0000 7f9767809700 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x7f976010e0a0 con 0x7f9760104d10 2024-05-27T07:14:47.805 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:14:47.813+0000 7f97655a5700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f9760104d10 0x7f97601a1760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:14:47.806 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:14:47.813+0000 7f97655a5700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f9760104d10 0x7f97601a1760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.52:58676/0 (socket says 172.21.15.52:58676) 2024-05-27T07:14:47.806 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:14:47.813+0000 7f97655a5700 1 -- 172.21.15.52:0/837670031 learned_addr learned my addr 172.21.15.52:0/837670031 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-05-27T07:14:47.806 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:14:47.814+0000 7f97655a5700 1 -- 172.21.15.52:0/837670031 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f97601a1ca0 con 0x7f9760104d10 2024-05-27T07:14:47.806 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:14:47.814+0000 7f97655a5700 1 --2- 172.21.15.52:0/837670031 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f9760104d10 0x7f97601a1760 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f975c0092e0 tx=0x7f975c005780 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:14:47.806 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:14:47.814+0000 7f97567fc700 1 -- 172.21.15.52:0/837670031 <== mon.0 v2:172.21.15.52:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f975c026070 con 0x7f9760104d10 2024-05-27T07:14:47.806 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:14:47.814+0000 7f97567fc700 1 -- 172.21.15.52:0/837670031 <== mon.0 v2:172.21.15.52:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f975c00b6f0 con 0x7f9760104d10 2024-05-27T07:14:47.806 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:14:47.814+0000 7f9767809700 1 -- 172.21.15.52:0/837670031 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f97601a1f90 con 0x7f9760104d10 2024-05-27T07:14:47.806 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:14:47.814+0000 7f97567fc700 1 -- 172.21.15.52:0/837670031 <== mon.0 v2:172.21.15.52:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f975c0037f0 con 0x7f9760104d10 2024-05-27T07:14:47.806 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:14:47.814+0000 7f9767809700 1 -- 172.21.15.52:0/837670031 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f976019b980 con 0x7f9760104d10 2024-05-27T07:14:47.806 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:14:47.815+0000 7f97567fc700 1 -- 172.21.15.52:0/837670031 <== mon.0 v2:172.21.15.52:3300/0 4 ==== mgrmap(e 13) ==== 50278+0+0 (secure 0 0 0) 0x7f975c029070 con 0x7f9760104d10 2024-05-27T07:14:47.806 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:14:47.815+0000 7f9767809700 1 -- 172.21.15.52:0/837670031 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f976006c450 con 0x7f9760104d10 2024-05-27T07:14:47.807 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:14:47.815+0000 7f97567fc700 1 --2- 172.21.15.52:0/837670031 >> [v2:172.21.15.52:6800/3513075805,v1:172.21.15.52:6801/3513075805] conn(0x7f974c03e570 0x7f974c040a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:14:47.807 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:14:47.815+0000 7f97567fc700 1 -- 172.21.15.52:0/837670031 <== mon.0 v2:172.21.15.52:3300/0 5 ==== osd_map(3..3 src has 1..3) ==== 1155+0+0 (secure 0 0 0) 0x7f975c01e070 con 0x7f9760104d10 2024-05-27T07:14:47.807 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:14:47.815+0000 7f9764da4700 1 --2- 172.21.15.52:0/837670031 >> [v2:172.21.15.52:6800/3513075805,v1:172.21.15.52:6801/3513075805] conn(0x7f974c03e570 0x7f974c040a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:14:47.807 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:14:47.816+0000 7f9764da4700 1 --2- 172.21.15.52:0/837670031 >> [v2:172.21.15.52:6800/3513075805,v1:172.21.15.52:6801/3513075805] conn(0x7f974c03e570 0x7f974c040a80 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7f9750006fd0 tx=0x7f9750006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:14:47.811 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:14:47.820+0000 7f97567fc700 1 -- 172.21.15.52:0/837670031 <== mon.0 v2:172.21.15.52:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191963 (secure 0 0 0) 0x7f975c013070 con 0x7f9760104d10 2024-05-27T07:14:48.031 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:14:48.041+0000 7f9767809700 1 -- 172.21.15.52:0/837670031 --> [v2:172.21.15.52:6800/3513075805,v1:172.21.15.52:6801/3513075805] -- mgr_command(tid 0: {"prefix": "orch host add", "hostname": "smithi105", "target": ["mon-mgr", ""]}) -- 0x7f976019ceb0 con 0x7f974c03e570 2024-05-27T07:14:48.730 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:14:48 smithi052 ceph-mon[31955]: from='client.14178 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi105", "target": ["mon-mgr", ""]}]: dispatch 2024-05-27T07:14:49.730 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:14:49 smithi052 ceph-mon[31955]: Deploying cephadm binary to smithi105 2024-05-27T07:14:50.980 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:14:50 smithi052 ceph-mon[31955]: pgmap v3: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-27T07:14:52.352 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:14:52.361+0000 7f97567fc700 1 -- 172.21.15.52:0/837670031 <== mgr.14152 v2:172.21.15.52:6800/3513075805 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+49 (secure 0 0 0) 0x7f976019ceb0 con 0x7f974c03e570 2024-05-27T07:14:52.352 INFO:teuthology.orchestra.run.smithi052.stdout:Added host 'smithi105' with addr '172.21.15.105' 2024-05-27T07:14:52.356 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:14:52.364+0000 7f9767809700 1 -- 172.21.15.52:0/837670031 >> [v2:172.21.15.52:6800/3513075805,v1:172.21.15.52:6801/3513075805] conn(0x7f974c03e570 msgr2=0x7f974c040a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:14:52.356 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:14:52.364+0000 7f9767809700 1 --2- 172.21.15.52:0/837670031 >> [v2:172.21.15.52:6800/3513075805,v1:172.21.15.52:6801/3513075805] conn(0x7f974c03e570 0x7f974c040a80 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7f9750006fd0 tx=0x7f9750006e40 comp rx=0 tx=0).stop 2024-05-27T07:14:52.356 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:14:52.364+0000 7f9767809700 1 -- 172.21.15.52:0/837670031 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f9760104d10 msgr2=0x7f97601a1760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:14:52.356 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:14:52.365+0000 7f9767809700 1 --2- 172.21.15.52:0/837670031 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f9760104d10 0x7f97601a1760 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f975c0092e0 tx=0x7f975c005780 comp rx=0 tx=0).stop 2024-05-27T07:14:52.356 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:14:52.365+0000 7f9767809700 1 -- 172.21.15.52:0/837670031 shutdown_connections 2024-05-27T07:14:52.356 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:14:52.365+0000 7f9767809700 1 --2- 172.21.15.52:0/837670031 >> [v2:172.21.15.52:6800/3513075805,v1:172.21.15.52:6801/3513075805] conn(0x7f974c03e570 0x7f974c040a80 unknown :-1 s=CLOSED pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:14:52.356 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:14:52.365+0000 7f9767809700 1 --2- 172.21.15.52:0/837670031 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f9760104d10 0x7f97601a1760 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:14:52.356 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:14:52.365+0000 7f9767809700 1 -- 172.21.15.52:0/837670031 >> 172.21.15.52:0/837670031 conn(0x7f97600fe560 msgr2=0x7f9760109ea0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:14:52.356 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:14:52.365+0000 7f9767809700 1 -- 172.21.15.52:0/837670031 shutdown_connections 2024-05-27T07:14:52.356 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:14:52.365+0000 7f9767809700 1 -- 172.21.15.52:0/837670031 wait complete. 2024-05-27T07:14:52.797 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:14:52 smithi052 ceph-mon[31955]: pgmap v4: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-27T07:14:52.798 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:14:52 smithi052 ceph-mon[31955]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' 2024-05-27T07:14:52.798 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:14:52 smithi052 ceph-mon[31955]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-27T07:14:53.084 DEBUG:teuthology.orchestra.run.smithi052:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:38f9d900c83c63799fbdbe61acc9a11b0d3554a6 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 73df2636-1bf8-11ef-bc9b-c7b262605968 -- ceph orch host ls --format=json 2024-05-27T07:14:53.330 INFO:teuthology.orchestra.run.smithi052.stderr:Inferring config /var/lib/ceph/73df2636-1bf8-11ef-bc9b-c7b262605968/mon.a/config 2024-05-27T07:14:54.205 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:14:53 smithi052 ceph-mon[31955]: Added host smithi105 2024-05-27T07:14:54.205 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:14:53 smithi052 ceph-mon[31955]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' 2024-05-27T07:14:54.205 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:14:53 smithi052 ceph-mon[31955]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' 2024-05-27T07:14:54.867 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:14:54.876+0000 7fe719e68700 1 -- 172.21.15.52:0/3383482473 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fe714104aa0 msgr2=0x7fe714101f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:14:54.867 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:14:54.876+0000 7fe719e68700 1 --2- 172.21.15.52:0/3383482473 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fe714104aa0 0x7fe714101f80 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7fe704009910 tx=0x7fe704009c20 comp rx=0 tx=0).stop 2024-05-27T07:14:54.867 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:14:54.876+0000 7fe719e68700 1 -- 172.21.15.52:0/3383482473 shutdown_connections 2024-05-27T07:14:54.867 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:14:54.876+0000 7fe719e68700 1 --2- 172.21.15.52:0/3383482473 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fe714104aa0 0x7fe714101f80 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:14:54.867 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:14:54.876+0000 7fe719e68700 1 -- 172.21.15.52:0/3383482473 >> 172.21.15.52:0/3383482473 conn(0x7fe7140fde20 msgr2=0x7fe714100240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:14:54.867 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:14:54.876+0000 7fe719e68700 1 -- 172.21.15.52:0/3383482473 shutdown_connections 2024-05-27T07:14:54.867 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:14:54.876+0000 7fe719e68700 1 -- 172.21.15.52:0/3383482473 wait complete. 2024-05-27T07:14:54.868 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:14:54.877+0000 7fe719e68700 1 Processor -- start 2024-05-27T07:14:54.868 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:14:54.877+0000 7fe719e68700 1 -- start start 2024-05-27T07:14:54.868 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:14:54.877+0000 7fe719e68700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fe714104aa0 0x7fe7141a1760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:14:54.868 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:14:54.877+0000 7fe719e68700 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x7fe714110330 con 0x7fe714104aa0 2024-05-27T07:14:54.869 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:14:54.877+0000 7fe7137fe700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fe714104aa0 0x7fe7141a1760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:14:54.869 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:14:54.877+0000 7fe7137fe700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fe714104aa0 0x7fe7141a1760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.52:58746/0 (socket says 172.21.15.52:58746) 2024-05-27T07:14:54.869 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:14:54.877+0000 7fe7137fe700 1 -- 172.21.15.52:0/1757356704 learned_addr learned my addr 172.21.15.52:0/1757356704 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-05-27T07:14:54.869 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:14:54.878+0000 7fe7137fe700 1 -- 172.21.15.52:0/1757356704 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe7141a1ca0 con 0x7fe714104aa0 2024-05-27T07:14:54.869 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:14:54.878+0000 7fe7137fe700 1 --2- 172.21.15.52:0/1757356704 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fe714104aa0 0x7fe7141a1760 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7fe7040056d0 tx=0x7fe7040057b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:14:54.870 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:14:54.878+0000 7fe710ff9700 1 -- 172.21.15.52:0/1757356704 <== mon.0 v2:172.21.15.52:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fe704027070 con 0x7fe714104aa0 2024-05-27T07:14:54.870 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:14:54.878+0000 7fe710ff9700 1 -- 172.21.15.52:0/1757356704 <== mon.0 v2:172.21.15.52:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fe70400be10 con 0x7fe714104aa0 2024-05-27T07:14:54.870 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:14:54.878+0000 7fe710ff9700 1 -- 172.21.15.52:0/1757356704 <== mon.0 v2:172.21.15.52:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fe704022d00 con 0x7fe714104aa0 2024-05-27T07:14:54.871 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:14:54.878+0000 7fe719e68700 1 -- 172.21.15.52:0/1757356704 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe7141a1f30 con 0x7fe714104aa0 2024-05-27T07:14:54.871 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:14:54.878+0000 7fe719e68700 1 -- 172.21.15.52:0/1757356704 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe71419b980 con 0x7fe714104aa0 2024-05-27T07:14:54.871 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:14:54.879+0000 7fe719e68700 1 -- 172.21.15.52:0/1757356704 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe7141031a0 con 0x7fe714104aa0 2024-05-27T07:14:54.871 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:14:54.879+0000 7fe710ff9700 1 -- 172.21.15.52:0/1757356704 <== mon.0 v2:172.21.15.52:3300/0 4 ==== mgrmap(e 13) ==== 50278+0+0 (secure 0 0 0) 0x7fe70402a070 con 0x7fe714104aa0 2024-05-27T07:14:54.871 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:14:54.880+0000 7fe710ff9700 1 --2- 172.21.15.52:0/1757356704 >> [v2:172.21.15.52:6800/3513075805,v1:172.21.15.52:6801/3513075805] conn(0x7fe6fc03e510 0x7fe6fc040a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:14:54.871 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:14:54.880+0000 7fe710ff9700 1 -- 172.21.15.52:0/1757356704 <== mon.0 v2:172.21.15.52:3300/0 5 ==== osd_map(3..3 src has 1..3) ==== 1155+0+0 (secure 0 0 0) 0x7fe70401e030 con 0x7fe714104aa0 2024-05-27T07:14:54.871 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:14:54.880+0000 7fe712ffd700 1 --2- 172.21.15.52:0/1757356704 >> [v2:172.21.15.52:6800/3513075805,v1:172.21.15.52:6801/3513075805] conn(0x7fe6fc03e510 0x7fe6fc040a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:14:54.871 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:14:54.880+0000 7fe712ffd700 1 --2- 172.21.15.52:0/1757356704 >> [v2:172.21.15.52:6800/3513075805,v1:172.21.15.52:6801/3513075805] conn(0x7fe6fc03e510 0x7fe6fc040a20 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7fe708006fd0 tx=0x7fe708006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:14:54.875 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:14:54.885+0000 7fe710ff9700 1 -- 172.21.15.52:0/1757356704 <== mon.0 v2:172.21.15.52:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191963 (secure 0 0 0) 0x7fe704013070 con 0x7fe714104aa0 2024-05-27T07:14:55.096 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:14:55.105+0000 7fe719e68700 1 -- 172.21.15.52:0/1757356704 --> [v2:172.21.15.52:6800/3513075805,v1:172.21.15.52:6801/3513075805] -- mgr_command(tid 0: {"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7fe71419ceb0 con 0x7fe6fc03e510 2024-05-27T07:14:55.096 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:14:54 smithi052 ceph-mon[31955]: pgmap v5: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-27T07:14:55.097 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:14:55.107+0000 7fe710ff9700 1 -- 172.21.15.52:0/1757356704 <== mgr.14152 v2:172.21.15.52:6800/3513075805 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+160 (secure 0 0 0) 0x7fe71419ceb0 con 0x7fe6fc03e510 2024-05-27T07:14:55.098 INFO:teuthology.orchestra.run.smithi052.stdout: 2024-05-27T07:14:55.098 INFO:teuthology.orchestra.run.smithi052.stdout:[{"addr": "172.21.15.52", "hostname": "smithi052", "labels": [], "status": ""}, {"addr": "172.21.15.105", "hostname": "smithi105", "labels": [], "status": ""}] 2024-05-27T07:14:55.101 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:14:55.110+0000 7fe719e68700 1 -- 172.21.15.52:0/1757356704 >> [v2:172.21.15.52:6800/3513075805,v1:172.21.15.52:6801/3513075805] conn(0x7fe6fc03e510 msgr2=0x7fe6fc040a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:14:55.101 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:14:55.110+0000 7fe719e68700 1 --2- 172.21.15.52:0/1757356704 >> [v2:172.21.15.52:6800/3513075805,v1:172.21.15.52:6801/3513075805] conn(0x7fe6fc03e510 0x7fe6fc040a20 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7fe708006fd0 tx=0x7fe708006e40 comp rx=0 tx=0).stop 2024-05-27T07:14:55.101 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:14:55.110+0000 7fe719e68700 1 -- 172.21.15.52:0/1757356704 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fe714104aa0 msgr2=0x7fe7141a1760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:14:55.101 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:14:55.110+0000 7fe719e68700 1 --2- 172.21.15.52:0/1757356704 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fe714104aa0 0x7fe7141a1760 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7fe7040056d0 tx=0x7fe7040057b0 comp rx=0 tx=0).stop 2024-05-27T07:14:55.101 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:14:55.110+0000 7fe719e68700 1 -- 172.21.15.52:0/1757356704 shutdown_connections 2024-05-27T07:14:55.101 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:14:55.110+0000 7fe719e68700 1 --2- 172.21.15.52:0/1757356704 >> [v2:172.21.15.52:6800/3513075805,v1:172.21.15.52:6801/3513075805] conn(0x7fe6fc03e510 0x7fe6fc040a20 unknown :-1 s=CLOSED pgs=15 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:14:55.101 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:14:55.110+0000 7fe719e68700 1 --2- 172.21.15.52:0/1757356704 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fe714104aa0 0x7fe7141a1760 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:14:55.101 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:14:55.110+0000 7fe719e68700 1 -- 172.21.15.52:0/1757356704 >> 172.21.15.52:0/1757356704 conn(0x7fe7140fde20 msgr2=0x7fe71410b6c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:14:55.101 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:14:55.110+0000 7fe719e68700 1 -- 172.21.15.52:0/1757356704 shutdown_connections 2024-05-27T07:14:55.101 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:14:55.110+0000 7fe719e68700 1 -- 172.21.15.52:0/1757356704 wait complete. 2024-05-27T07:14:55.671 INFO:tasks.cephadm:Writing (initial) conf and keyring to smithi173 2024-05-27T07:14:55.671 DEBUG:teuthology.orchestra.run.smithi173:> set -ex 2024-05-27T07:14:55.671 DEBUG:teuthology.orchestra.run.smithi173:> dd of=/etc/ceph/ceph.conf 2024-05-27T07:14:55.690 DEBUG:teuthology.orchestra.run.smithi173:> set -ex 2024-05-27T07:14:55.691 DEBUG:teuthology.orchestra.run.smithi173:> dd of=/etc/ceph/ceph.client.admin.keyring 2024-05-27T07:14:55.748 INFO:tasks.cephadm:Adding host smithi173 to orchestrator... 2024-05-27T07:14:55.748 DEBUG:teuthology.orchestra.run.smithi052:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:38f9d900c83c63799fbdbe61acc9a11b0d3554a6 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 73df2636-1bf8-11ef-bc9b-c7b262605968 -- ceph orch host add smithi173 2024-05-27T07:14:56.006 INFO:teuthology.orchestra.run.smithi052.stderr:Inferring config /var/lib/ceph/73df2636-1bf8-11ef-bc9b-c7b262605968/mon.a/config 2024-05-27T07:14:56.322 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:14:56 smithi052 ceph-mon[31955]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' 2024-05-27T07:14:57.151 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:14:57 smithi052 ceph-mon[31955]: from='client.14180 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-05-27T07:14:57.151 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:14:57 smithi052 ceph-mon[31955]: pgmap v6: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-27T07:14:57.447 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:14:57.456+0000 7fd653caf700 1 -- 172.21.15.52:0/1681815111 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fd64c108ab0 msgr2=0x7fd64c108ec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:14:57.447 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:14:57.456+0000 7fd653caf700 1 --2- 172.21.15.52:0/1681815111 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fd64c108ab0 0x7fd64c108ec0 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7fd648009910 tx=0x7fd648009c20 comp rx=0 tx=0).stop 2024-05-27T07:14:57.447 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:14:57.456+0000 7fd653caf700 1 -- 172.21.15.52:0/1681815111 shutdown_connections 2024-05-27T07:14:57.447 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:14:57.456+0000 7fd653caf700 1 --2- 172.21.15.52:0/1681815111 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fd64c108ab0 0x7fd64c108ec0 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:14:57.447 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:14:57.456+0000 7fd653caf700 1 -- 172.21.15.52:0/1681815111 >> 172.21.15.52:0/1681815111 conn(0x7fd64c0fe5a0 msgr2=0x7fd64c1009f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:14:57.447 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:14:57.456+0000 7fd653caf700 1 -- 172.21.15.52:0/1681815111 shutdown_connections 2024-05-27T07:14:57.447 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:14:57.456+0000 7fd653caf700 1 -- 172.21.15.52:0/1681815111 wait complete. 2024-05-27T07:14:57.448 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:14:57.457+0000 7fd653caf700 1 Processor -- start 2024-05-27T07:14:57.448 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:14:57.457+0000 7fd653caf700 1 -- start start 2024-05-27T07:14:57.448 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:14:57.457+0000 7fd653caf700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fd64c108ab0 0x7fd64c19a7f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:14:57.448 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:14:57.457+0000 7fd653caf700 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x7fd64c10d000 con 0x7fd64c108ab0 2024-05-27T07:14:57.449 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:14:57.457+0000 7fd651a4b700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fd64c108ab0 0x7fd64c19a7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:14:57.449 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:14:57.458+0000 7fd651a4b700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fd64c108ab0 0x7fd64c19a7f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.52:58760/0 (socket says 172.21.15.52:58760) 2024-05-27T07:14:57.449 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:14:57.458+0000 7fd651a4b700 1 -- 172.21.15.52:0/2713511699 learned_addr learned my addr 172.21.15.52:0/2713511699 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-05-27T07:14:57.449 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:14:57.458+0000 7fd651a4b700 1 -- 172.21.15.52:0/2713511699 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd64c06ca00 con 0x7fd64c108ab0 2024-05-27T07:14:57.449 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:14:57.458+0000 7fd651a4b700 1 --2- 172.21.15.52:0/2713511699 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fd64c108ab0 0x7fd64c19a7f0 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7fd6480056d0 tx=0x7fd6480057b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:14:57.449 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:14:57.458+0000 7fd63effd700 1 -- 172.21.15.52:0/2713511699 <== mon.0 v2:172.21.15.52:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fd648027070 con 0x7fd64c108ab0 2024-05-27T07:14:57.450 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:14:57.458+0000 7fd63effd700 1 -- 172.21.15.52:0/2713511699 <== mon.0 v2:172.21.15.52:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fd64800be10 con 0x7fd64c108ab0 2024-05-27T07:14:57.450 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:14:57.458+0000 7fd653caf700 1 -- 172.21.15.52:0/2713511699 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd64c19adf0 con 0x7fd64c108ab0 2024-05-27T07:14:57.450 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:14:57.458+0000 7fd63effd700 1 -- 172.21.15.52:0/2713511699 <== mon.0 v2:172.21.15.52:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fd648022d20 con 0x7fd64c108ab0 2024-05-27T07:14:57.450 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:14:57.458+0000 7fd653caf700 1 -- 172.21.15.52:0/2713511699 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd64c19b2b0 con 0x7fd64c108ab0 2024-05-27T07:14:57.450 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:14:57.459+0000 7fd653caf700 1 -- 172.21.15.52:0/2713511699 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd64c19b5f0 con 0x7fd64c108ab0 2024-05-27T07:14:57.450 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:14:57.459+0000 7fd63effd700 1 -- 172.21.15.52:0/2713511699 <== mon.0 v2:172.21.15.52:3300/0 4 ==== mgrmap(e 13) ==== 50278+0+0 (secure 0 0 0) 0x7fd64802a070 con 0x7fd64c108ab0 2024-05-27T07:14:57.451 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:14:57.459+0000 7fd63effd700 1 --2- 172.21.15.52:0/2713511699 >> [v2:172.21.15.52:6800/3513075805,v1:172.21.15.52:6801/3513075805] conn(0x7fd63803e570 0x7fd638040a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:14:57.451 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:14:57.460+0000 7fd63effd700 1 -- 172.21.15.52:0/2713511699 <== mon.0 v2:172.21.15.52:3300/0 5 ==== osd_map(3..3 src has 1..3) ==== 1155+0+0 (secure 0 0 0) 0x7fd64801e030 con 0x7fd64c108ab0 2024-05-27T07:14:57.451 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:14:57.460+0000 7fd65124a700 1 --2- 172.21.15.52:0/2713511699 >> [v2:172.21.15.52:6800/3513075805,v1:172.21.15.52:6801/3513075805] conn(0x7fd63803e570 0x7fd638040a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:14:57.451 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:14:57.460+0000 7fd65124a700 1 --2- 172.21.15.52:0/2713511699 >> [v2:172.21.15.52:6800/3513075805,v1:172.21.15.52:6801/3513075805] conn(0x7fd63803e570 0x7fd638040a80 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7fd640006fd0 tx=0x7fd640006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:14:57.455 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:14:57.465+0000 7fd63effd700 1 -- 172.21.15.52:0/2713511699 <== mon.0 v2:172.21.15.52:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191963 (secure 0 0 0) 0x7fd64c06ca00 con 0x7fd64c108ab0 2024-05-27T07:14:57.675 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:14:57.684+0000 7fd653caf700 1 -- 172.21.15.52:0/2713511699 --> [v2:172.21.15.52:6800/3513075805,v1:172.21.15.52:6801/3513075805] -- mgr_command(tid 0: {"prefix": "orch host add", "hostname": "smithi173", "target": ["mon-mgr", ""]}) -- 0x7fd64c06c0e0 con 0x7fd63803e570 2024-05-27T07:14:59.230 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:14:58 smithi052 ceph-mon[31955]: pgmap v7: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-27T07:14:59.230 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:14:58 smithi052 ceph-mon[31955]: from='client.14182 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi173", "target": ["mon-mgr", ""]}]: dispatch 2024-05-27T07:14:59.231 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:14:58 smithi052 ceph-mon[31955]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' 2024-05-27T07:14:59.231 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:14:58 smithi052 ceph-mon[31955]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' 2024-05-27T07:14:59.231 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:14:58 smithi052 ceph-mon[31955]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' 2024-05-27T07:14:59.231 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:14:58 smithi052 ceph-mon[31955]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' 2024-05-27T07:14:59.231 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:14:58 smithi052 ceph-mon[31955]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd/host:smithi105", "name": "osd_memory_target"}]: dispatch 2024-05-27T07:14:59.232 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:14:58 smithi052 ceph-mon[31955]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T07:14:59.232 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:14:58 smithi052 ceph-mon[31955]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-27T07:14:59.232 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:14:58 smithi052 ceph-mon[31955]: Updating smithi105:/etc/ceph/ceph.conf 2024-05-27T07:15:00.230 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:14:59 smithi052 ceph-mon[31955]: Updating smithi105:/var/lib/ceph/73df2636-1bf8-11ef-bc9b-c7b262605968/config/ceph.conf 2024-05-27T07:15:00.230 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:14:59 smithi052 ceph-mon[31955]: Deploying cephadm binary to smithi173 2024-05-27T07:15:00.230 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:14:59 smithi052 ceph-mon[31955]: Updating smithi105:/etc/ceph/ceph.client.admin.keyring 2024-05-27T07:15:00.230 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:14:59 smithi052 ceph-mon[31955]: Updating smithi105:/var/lib/ceph/73df2636-1bf8-11ef-bc9b-c7b262605968/config/ceph.client.admin.keyring 2024-05-27T07:15:01.230 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:15:00 smithi052 ceph-mon[31955]: pgmap v8: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-27T07:15:02.004 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:15:02.013+0000 7fd63effd700 1 -- 172.21.15.52:0/2713511699 <== mgr.14152 v2:172.21.15.52:6800/3513075805 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+49 (secure 0 0 0) 0x7fd64c06c0e0 con 0x7fd63803e570 2024-05-27T07:15:02.004 INFO:teuthology.orchestra.run.smithi052.stdout:Added host 'smithi173' with addr '172.21.15.173' 2024-05-27T07:15:02.007 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:15:02.016+0000 7fd653caf700 1 -- 172.21.15.52:0/2713511699 >> [v2:172.21.15.52:6800/3513075805,v1:172.21.15.52:6801/3513075805] conn(0x7fd63803e570 msgr2=0x7fd638040a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:15:02.007 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:15:02.016+0000 7fd653caf700 1 --2- 172.21.15.52:0/2713511699 >> [v2:172.21.15.52:6800/3513075805,v1:172.21.15.52:6801/3513075805] conn(0x7fd63803e570 0x7fd638040a80 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7fd640006fd0 tx=0x7fd640006e40 comp rx=0 tx=0).stop 2024-05-27T07:15:02.007 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:15:02.016+0000 7fd653caf700 1 -- 172.21.15.52:0/2713511699 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fd64c108ab0 msgr2=0x7fd64c19a7f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:15:02.007 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:15:02.016+0000 7fd653caf700 1 --2- 172.21.15.52:0/2713511699 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fd64c108ab0 0x7fd64c19a7f0 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7fd6480056d0 tx=0x7fd6480057b0 comp rx=0 tx=0).stop 2024-05-27T07:15:02.007 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:15:02.016+0000 7fd653caf700 1 -- 172.21.15.52:0/2713511699 shutdown_connections 2024-05-27T07:15:02.007 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:15:02.016+0000 7fd653caf700 1 --2- 172.21.15.52:0/2713511699 >> [v2:172.21.15.52:6800/3513075805,v1:172.21.15.52:6801/3513075805] conn(0x7fd63803e570 0x7fd638040a80 unknown :-1 s=CLOSED pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:15:02.007 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:15:02.016+0000 7fd653caf700 1 --2- 172.21.15.52:0/2713511699 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fd64c108ab0 0x7fd64c19a7f0 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:15:02.007 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:15:02.016+0000 7fd653caf700 1 -- 172.21.15.52:0/2713511699 >> 172.21.15.52:0/2713511699 conn(0x7fd64c0fe5a0 msgr2=0x7fd64c0ff620 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:15:02.007 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:15:02.016+0000 7fd653caf700 1 -- 172.21.15.52:0/2713511699 shutdown_connections 2024-05-27T07:15:02.007 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:15:02.016+0000 7fd653caf700 1 -- 172.21.15.52:0/2713511699 wait complete. 2024-05-27T07:15:02.560 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:15:02 smithi052 ceph-mon[31955]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' 2024-05-27T07:15:02.560 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:15:02 smithi052 ceph-mon[31955]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' 2024-05-27T07:15:02.560 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:15:02 smithi052 ceph-mon[31955]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' 2024-05-27T07:15:02.561 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:15:02 smithi052 ceph-mon[31955]: pgmap v9: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-27T07:15:02.561 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:15:02 smithi052 ceph-mon[31955]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' 2024-05-27T07:15:02.561 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:15:02 smithi052 ceph-mon[31955]: Added host smithi173 2024-05-27T07:15:02.561 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:15:02 smithi052 ceph-mon[31955]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-27T07:15:02.562 DEBUG:teuthology.orchestra.run.smithi052:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:38f9d900c83c63799fbdbe61acc9a11b0d3554a6 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 73df2636-1bf8-11ef-bc9b-c7b262605968 -- ceph orch host ls --format=json 2024-05-27T07:15:02.810 INFO:teuthology.orchestra.run.smithi052.stderr:Inferring config /var/lib/ceph/73df2636-1bf8-11ef-bc9b-c7b262605968/mon.a/config 2024-05-27T07:15:03.711 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:15:03 smithi052 ceph-mon[31955]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' 2024-05-27T07:15:03.711 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:15:03 smithi052 ceph-mon[31955]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' 2024-05-27T07:15:04.181 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:15:04.190+0000 7f93b51fa700 1 -- 172.21.15.52:0/1701157674 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f93b0108a80 msgr2=0x7f93b0108e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:15:04.181 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:15:04.190+0000 7f93b51fa700 1 --2- 172.21.15.52:0/1701157674 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f93b0108a80 0x7f93b0108e90 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f93a0009910 tx=0x7f93a0009c20 comp rx=0 tx=0).stop 2024-05-27T07:15:04.181 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:15:04.190+0000 7f93b51fa700 1 -- 172.21.15.52:0/1701157674 shutdown_connections 2024-05-27T07:15:04.181 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:15:04.190+0000 7f93b51fa700 1 --2- 172.21.15.52:0/1701157674 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f93b0108a80 0x7f93b0108e90 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:15:04.181 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:15:04.190+0000 7f93b51fa700 1 -- 172.21.15.52:0/1701157674 >> 172.21.15.52:0/1701157674 conn(0x7f93b00785a0 msgr2=0x7f93b0078a00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:15:04.182 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:15:04.190+0000 7f93b51fa700 1 -- 172.21.15.52:0/1701157674 shutdown_connections 2024-05-27T07:15:04.182 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:15:04.190+0000 7f93b51fa700 1 -- 172.21.15.52:0/1701157674 wait complete. 2024-05-27T07:15:04.183 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:15:04.191+0000 7f93b51fa700 1 Processor -- start 2024-05-27T07:15:04.183 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:15:04.191+0000 7f93b51fa700 1 -- start start 2024-05-27T07:15:04.183 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:15:04.191+0000 7f93b51fa700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f93b0108a80 0x7f93b019e520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:15:04.183 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:15:04.191+0000 7f93b51fa700 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x7f93b010cfd0 con 0x7f93b0108a80 2024-05-27T07:15:04.183 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:15:04.192+0000 7f93aed9d700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f93b0108a80 0x7f93b019e520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:15:04.183 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:15:04.192+0000 7f93aed9d700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f93b0108a80 0x7f93b019e520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.52:57664/0 (socket says 172.21.15.52:57664) 2024-05-27T07:15:04.184 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:15:04.192+0000 7f93aed9d700 1 -- 172.21.15.52:0/3985971589 learned_addr learned my addr 172.21.15.52:0/3985971589 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-05-27T07:15:04.184 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:15:04.192+0000 7f93aed9d700 1 -- 172.21.15.52:0/3985971589 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f93b019ea60 con 0x7f93b0108a80 2024-05-27T07:15:04.184 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:15:04.192+0000 7f93aed9d700 1 --2- 172.21.15.52:0/3985971589 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f93b0108a80 0x7f93b019e520 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f93a00056d0 tx=0x7f93a00057b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:15:04.184 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:15:04.192+0000 7f9397fff700 1 -- 172.21.15.52:0/3985971589 <== mon.0 v2:172.21.15.52:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f93a0027070 con 0x7f93b0108a80 2024-05-27T07:15:04.184 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:15:04.192+0000 7f9397fff700 1 -- 172.21.15.52:0/3985971589 <== mon.0 v2:172.21.15.52:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f93a000be10 con 0x7f93b0108a80 2024-05-27T07:15:04.184 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:15:04.192+0000 7f9397fff700 1 -- 172.21.15.52:0/3985971589 <== mon.0 v2:172.21.15.52:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f93a0022d20 con 0x7f93b0108a80 2024-05-27T07:15:04.184 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:15:04.192+0000 7f93b51fa700 1 -- 172.21.15.52:0/3985971589 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f93b019ed50 con 0x7f93b0108a80 2024-05-27T07:15:04.184 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:15:04.193+0000 7f93b51fa700 1 -- 172.21.15.52:0/3985971589 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f93b0198700 con 0x7f93b0108a80 2024-05-27T07:15:04.185 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:15:04.194+0000 7f9397fff700 1 -- 172.21.15.52:0/3985971589 <== mon.0 v2:172.21.15.52:3300/0 4 ==== mgrmap(e 13) ==== 50278+0+0 (secure 0 0 0) 0x7f93a002a070 con 0x7f93b0108a80 2024-05-27T07:15:04.185 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:15:04.194+0000 7f93b51fa700 1 -- 172.21.15.52:0/3985971589 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f93b0104190 con 0x7f93b0108a80 2024-05-27T07:15:04.185 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:15:04.194+0000 7f9397fff700 1 --2- 172.21.15.52:0/3985971589 >> [v2:172.21.15.52:6800/3513075805,v1:172.21.15.52:6801/3513075805] conn(0x7f939803e570 0x7f9398040a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:15:04.185 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:15:04.194+0000 7f9397fff700 1 -- 172.21.15.52:0/3985971589 <== mon.0 v2:172.21.15.52:3300/0 5 ==== osd_map(3..3 src has 1..3) ==== 1155+0+0 (secure 0 0 0) 0x7f93a001e030 con 0x7f93b0108a80 2024-05-27T07:15:04.185 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:15:04.194+0000 7f93ae59c700 1 --2- 172.21.15.52:0/3985971589 >> [v2:172.21.15.52:6800/3513075805,v1:172.21.15.52:6801/3513075805] conn(0x7f939803e570 0x7f9398040a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:15:04.185 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:15:04.195+0000 7f93ae59c700 1 --2- 172.21.15.52:0/3985971589 >> [v2:172.21.15.52:6800/3513075805,v1:172.21.15.52:6801/3513075805] conn(0x7f939803e570 0x7f9398040a80 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7f93a4006fd0 tx=0x7f93a4006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:15:04.190 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:15:04.199+0000 7f9397fff700 1 -- 172.21.15.52:0/3985971589 <== mon.0 v2:172.21.15.52:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191963 (secure 0 0 0) 0x7f93a0013070 con 0x7f93b0108a80 2024-05-27T07:15:04.411 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:15:04.420+0000 7f93b51fa700 1 -- 172.21.15.52:0/3985971589 --> [v2:172.21.15.52:6800/3513075805,v1:172.21.15.52:6801/3513075805] -- mgr_command(tid 0: {"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f93b0199c10 con 0x7f939803e570 2024-05-27T07:15:04.412 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:15:04.421+0000 7f9397fff700 1 -- 172.21.15.52:0/3985971589 <== mgr.14152 v2:172.21.15.52:6800/3513075805 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+240 (secure 0 0 0) 0x7f93b0199c10 con 0x7f939803e570 2024-05-27T07:15:04.412 INFO:teuthology.orchestra.run.smithi052.stdout: 2024-05-27T07:15:04.412 INFO:teuthology.orchestra.run.smithi052.stdout:[{"addr": "172.21.15.52", "hostname": "smithi052", "labels": [], "status": ""}, {"addr": "172.21.15.105", "hostname": "smithi105", "labels": [], "status": ""}, {"addr": "172.21.15.173", "hostname": "smithi173", "labels": [], "status": ""}] 2024-05-27T07:15:04.419 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:15:04.424+0000 7f93b51fa700 1 -- 172.21.15.52:0/3985971589 >> [v2:172.21.15.52:6800/3513075805,v1:172.21.15.52:6801/3513075805] conn(0x7f939803e570 msgr2=0x7f9398040a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:15:04.419 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:15:04.424+0000 7f93b51fa700 1 --2- 172.21.15.52:0/3985971589 >> [v2:172.21.15.52:6800/3513075805,v1:172.21.15.52:6801/3513075805] conn(0x7f939803e570 0x7f9398040a80 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7f93a4006fd0 tx=0x7f93a4006e40 comp rx=0 tx=0).stop 2024-05-27T07:15:04.419 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:15:04.424+0000 7f93b51fa700 1 -- 172.21.15.52:0/3985971589 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f93b0108a80 msgr2=0x7f93b019e520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:15:04.419 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:15:04.424+0000 7f93b51fa700 1 --2- 172.21.15.52:0/3985971589 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f93b0108a80 0x7f93b019e520 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f93a00056d0 tx=0x7f93a00057b0 comp rx=0 tx=0).stop 2024-05-27T07:15:04.419 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:15:04.424+0000 7f93b51fa700 1 -- 172.21.15.52:0/3985971589 shutdown_connections 2024-05-27T07:15:04.419 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:15:04.424+0000 7f93b51fa700 1 --2- 172.21.15.52:0/3985971589 >> [v2:172.21.15.52:6800/3513075805,v1:172.21.15.52:6801/3513075805] conn(0x7f939803e570 0x7f9398040a80 unknown :-1 s=CLOSED pgs=17 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:15:04.419 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:15:04.424+0000 7f93b51fa700 1 --2- 172.21.15.52:0/3985971589 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f93b0108a80 0x7f93b019e520 unknown :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:15:04.419 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:15:04.424+0000 7f93b51fa700 1 -- 172.21.15.52:0/3985971589 >> 172.21.15.52:0/3985971589 conn(0x7f93b00785a0 msgr2=0x7f93b00ff910 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:15:04.419 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:15:04.424+0000 7f93b51fa700 1 -- 172.21.15.52:0/3985971589 shutdown_connections 2024-05-27T07:15:04.419 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:15:04.424+0000 7f93b51fa700 1 -- 172.21.15.52:0/3985971589 wait complete. 2024-05-27T07:15:04.692 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:15:04 smithi052 ceph-mon[31955]: pgmap v10: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-27T07:15:04.919 INFO:tasks.cephadm:Setting crush tunables to default 2024-05-27T07:15:04.919 DEBUG:teuthology.orchestra.run.smithi052:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:38f9d900c83c63799fbdbe61acc9a11b0d3554a6 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 73df2636-1bf8-11ef-bc9b-c7b262605968 -- ceph osd crush tunables default 2024-05-27T07:15:05.159 INFO:teuthology.orchestra.run.smithi052.stderr:Inferring config /var/lib/ceph/73df2636-1bf8-11ef-bc9b-c7b262605968/mon.a/config 2024-05-27T07:15:05.865 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:15:05 smithi052 ceph-mon[31955]: from='client.14184 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-05-27T07:15:05.865 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:15:05 smithi052 ceph-mon[31955]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' 2024-05-27T07:15:06.569 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:15:06.578+0000 7fb2c61a5700 1 -- 172.21.15.52:0/2849700588 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fb2c0104d40 msgr2=0x7fb2c0105150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:15:06.570 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:15:06.578+0000 7fb2c61a5700 1 --2- 172.21.15.52:0/2849700588 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fb2c0104d40 0x7fb2c0105150 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7fb2b0009910 tx=0x7fb2b0009c20 comp rx=0 tx=0).stop 2024-05-27T07:15:06.570 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:15:06.578+0000 7fb2c61a5700 1 -- 172.21.15.52:0/2849700588 shutdown_connections 2024-05-27T07:15:06.570 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:15:06.579+0000 7fb2c61a5700 1 --2- 172.21.15.52:0/2849700588 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fb2c0104d40 0x7fb2c0105150 unknown :-1 s=CLOSED pgs=92 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:15:06.570 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:15:06.579+0000 7fb2c61a5700 1 -- 172.21.15.52:0/2849700588 >> 172.21.15.52:0/2849700588 conn(0x7fb2c00785a0 msgr2=0x7fb2c0078a00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:15:06.570 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:15:06.579+0000 7fb2c61a5700 1 -- 172.21.15.52:0/2849700588 shutdown_connections 2024-05-27T07:15:06.570 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:15:06.579+0000 7fb2c61a5700 1 -- 172.21.15.52:0/2849700588 wait complete. 2024-05-27T07:15:06.571 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:15:06.579+0000 7fb2c61a5700 1 Processor -- start 2024-05-27T07:15:06.571 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:15:06.579+0000 7fb2c61a5700 1 -- start start 2024-05-27T07:15:06.571 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:15:06.580+0000 7fb2c61a5700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fb2c0104d40 0x7fb2c019f570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:15:06.572 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:15:06.580+0000 7fb2c61a5700 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x7fb2c010e130 con 0x7fb2c0104d40 2024-05-27T07:15:06.572 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:15:06.580+0000 7fb2bf7fe700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fb2c0104d40 0x7fb2c019f570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:15:06.572 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:15:06.580+0000 7fb2bf7fe700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fb2c0104d40 0x7fb2c019f570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.52:57670/0 (socket says 172.21.15.52:57670) 2024-05-27T07:15:06.572 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:15:06.580+0000 7fb2bf7fe700 1 -- 172.21.15.52:0/1478112303 learned_addr learned my addr 172.21.15.52:0/1478112303 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-05-27T07:15:06.572 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:15:06.580+0000 7fb2bf7fe700 1 -- 172.21.15.52:0/1478112303 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb2c019fab0 con 0x7fb2c0104d40 2024-05-27T07:15:06.572 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:15:06.580+0000 7fb2bf7fe700 1 --2- 172.21.15.52:0/1478112303 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fb2c0104d40 0x7fb2c019f570 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7fb2b00056d0 tx=0x7fb2b00057b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:15:06.572 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:15:06.580+0000 7fb2bcff9700 1 -- 172.21.15.52:0/1478112303 <== mon.0 v2:172.21.15.52:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fb2b0026070 con 0x7fb2c0104d40 2024-05-27T07:15:06.572 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:15:06.581+0000 7fb2c61a5700 1 -- 172.21.15.52:0/1478112303 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb2c019fd40 con 0x7fb2c0104d40 2024-05-27T07:15:06.572 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:15:06.581+0000 7fb2bcff9700 1 -- 172.21.15.52:0/1478112303 <== mon.0 v2:172.21.15.52:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fb2b0004370 con 0x7fb2c0104d40 2024-05-27T07:15:06.572 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:15:06.581+0000 7fb2bcff9700 1 -- 172.21.15.52:0/1478112303 <== mon.0 v2:172.21.15.52:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fb2b00039b0 con 0x7fb2c0104d40 2024-05-27T07:15:06.572 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:15:06.581+0000 7fb2c61a5700 1 -- 172.21.15.52:0/1478112303 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb2c01997b0 con 0x7fb2c0104d40 2024-05-27T07:15:06.573 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:15:06.582+0000 7fb2c61a5700 1 -- 172.21.15.52:0/1478112303 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb2c006b6c0 con 0x7fb2c0104d40 2024-05-27T07:15:06.573 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:15:06.582+0000 7fb2bcff9700 1 -- 172.21.15.52:0/1478112303 <== mon.0 v2:172.21.15.52:3300/0 4 ==== mgrmap(e 13) ==== 50278+0+0 (secure 0 0 0) 0x7fb2b0029070 con 0x7fb2c0104d40 2024-05-27T07:15:06.573 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:15:06.582+0000 7fb2bcff9700 1 --2- 172.21.15.52:0/1478112303 >> [v2:172.21.15.52:6800/3513075805,v1:172.21.15.52:6801/3513075805] conn(0x7fb2a803e510 0x7fb2a8040a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:15:06.573 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:15:06.582+0000 7fb2bcff9700 1 -- 172.21.15.52:0/1478112303 <== mon.0 v2:172.21.15.52:3300/0 5 ==== osd_map(3..3 src has 1..3) ==== 1155+0+0 (secure 0 0 0) 0x7fb2b001e070 con 0x7fb2c0104d40 2024-05-27T07:15:06.574 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:15:06.582+0000 7fb2beffd700 1 --2- 172.21.15.52:0/1478112303 >> [v2:172.21.15.52:6800/3513075805,v1:172.21.15.52:6801/3513075805] conn(0x7fb2a803e510 0x7fb2a8040a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:15:06.574 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:15:06.583+0000 7fb2beffd700 1 --2- 172.21.15.52:0/1478112303 >> [v2:172.21.15.52:6800/3513075805,v1:172.21.15.52:6801/3513075805] conn(0x7fb2a803e510 0x7fb2a8040a20 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7fb2b4006fd0 tx=0x7fb2b4006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:15:06.578 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:15:06.587+0000 7fb2bcff9700 1 -- 172.21.15.52:0/1478112303 <== mon.0 v2:172.21.15.52:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191963 (secure 0 0 0) 0x7fb2b0013070 con 0x7fb2c0104d40 2024-05-27T07:15:06.801 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:15:06.810+0000 7fb2c61a5700 1 -- 172.21.15.52:0/1478112303 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_command({"prefix": "osd crush tunables", "profile": "default"} v 0) -- 0x7fb2c019acc0 con 0x7fb2c0104d40 2024-05-27T07:15:06.980 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:15:06 smithi052 ceph-mon[31955]: pgmap v11: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-27T07:15:07.677 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:15:07.686+0000 7fb2bcff9700 1 -- 172.21.15.52:0/1478112303 <== mon.0 v2:172.21.15.52:3300/0 7 ==== mon_command_ack([{"prefix": "osd crush tunables", "profile": "default"}]=0 adjusted tunables profile to default v4) ==== 124+0+0 (secure 0 0 0) 0x7fb2b0015350 con 0x7fb2c0104d40 2024-05-27T07:15:07.681 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:15:07.690+0000 7fb2c61a5700 1 -- 172.21.15.52:0/1478112303 >> [v2:172.21.15.52:6800/3513075805,v1:172.21.15.52:6801/3513075805] conn(0x7fb2a803e510 msgr2=0x7fb2a8040a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:15:07.681 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:15:07.690+0000 7fb2c61a5700 1 --2- 172.21.15.52:0/1478112303 >> [v2:172.21.15.52:6800/3513075805,v1:172.21.15.52:6801/3513075805] conn(0x7fb2a803e510 0x7fb2a8040a20 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7fb2b4006fd0 tx=0x7fb2b4006e40 comp rx=0 tx=0).stop 2024-05-27T07:15:07.681 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:15:07.690+0000 7fb2c61a5700 1 -- 172.21.15.52:0/1478112303 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fb2c0104d40 msgr2=0x7fb2c019f570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:15:07.681 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:15:07.690+0000 7fb2c61a5700 1 --2- 172.21.15.52:0/1478112303 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fb2c0104d40 0x7fb2c019f570 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7fb2b00056d0 tx=0x7fb2b00057b0 comp rx=0 tx=0).stop 2024-05-27T07:15:07.681 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:15:07.690+0000 7fb2c61a5700 1 -- 172.21.15.52:0/1478112303 shutdown_connections 2024-05-27T07:15:07.681 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:15:07.690+0000 7fb2c61a5700 1 --2- 172.21.15.52:0/1478112303 >> [v2:172.21.15.52:6800/3513075805,v1:172.21.15.52:6801/3513075805] conn(0x7fb2a803e510 0x7fb2a8040a20 unknown :-1 s=CLOSED pgs=18 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:15:07.681 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:15:07.690+0000 7fb2c61a5700 1 --2- 172.21.15.52:0/1478112303 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fb2c0104d40 0x7fb2c019f570 unknown :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:15:07.681 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:15:07.690+0000 7fb2c61a5700 1 -- 172.21.15.52:0/1478112303 >> 172.21.15.52:0/1478112303 conn(0x7fb2c00785a0 msgr2=0x7fb2c0105aa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:15:07.681 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:15:07.690+0000 7fb2c61a5700 1 -- 172.21.15.52:0/1478112303 shutdown_connections 2024-05-27T07:15:07.681 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:15:07.690+0000 7fb2c61a5700 1 -- 172.21.15.52:0/1478112303 wait complete. 2024-05-27T07:15:07.682 INFO:teuthology.orchestra.run.smithi052.stderr:adjusted tunables profile to default 2024-05-27T07:15:07.980 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:15:07 smithi052 ceph-mon[31955]: from='client.? 172.21.15.52:0/1478112303' entity='client.admin' cmd=[{"prefix": "osd crush tunables", "profile": "default"}]: dispatch 2024-05-27T07:15:08.337 INFO:tasks.cephadm:Adding mon.a on smithi052 2024-05-27T07:15:08.337 INFO:tasks.cephadm:Adding mon.b on smithi105 2024-05-27T07:15:08.337 INFO:tasks.cephadm:Adding mon.c on smithi173 2024-05-27T07:15:08.337 DEBUG:teuthology.orchestra.run.smithi173:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:38f9d900c83c63799fbdbe61acc9a11b0d3554a6 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 73df2636-1bf8-11ef-bc9b-c7b262605968 -- ceph orch apply mon '3;smithi052:172.21.15.52=a;smithi105:172.21.15.105=b;smithi173:172.21.15.173=c' 2024-05-27T07:15:08.572 INFO:teuthology.orchestra.run.smithi173.stderr:Inferring config /var/lib/ceph/73df2636-1bf8-11ef-bc9b-c7b262605968/config/ceph.conf 2024-05-27T07:15:08.980 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:15:08 smithi052 ceph-mon[31955]: pgmap v12: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-27T07:15:08.980 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:15:08 smithi052 ceph-mon[31955]: from='client.? 172.21.15.52:0/1478112303' entity='client.admin' cmd='[{"prefix": "osd crush tunables", "profile": "default"}]': finished 2024-05-27T07:15:08.980 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:15:08 smithi052 ceph-mon[31955]: osdmap e4: 0 total, 0 up, 0 in 2024-05-27T07:15:08.980 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:15:08 smithi052 ceph-mon[31955]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' 2024-05-27T07:15:08.980 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:15:08 smithi052 ceph-mon[31955]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' 2024-05-27T07:15:08.981 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:15:08 smithi052 ceph-mon[31955]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' 2024-05-27T07:15:08.981 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:15:08 smithi052 ceph-mon[31955]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' 2024-05-27T07:15:08.981 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:15:08 smithi052 ceph-mon[31955]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd/host:smithi173", "name": "osd_memory_target"}]: dispatch 2024-05-27T07:15:08.981 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:15:08 smithi052 ceph-mon[31955]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T07:15:08.981 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:15:08 smithi052 ceph-mon[31955]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-27T07:15:08.981 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:15:08 smithi052 ceph-mon[31955]: Updating smithi173:/etc/ceph/ceph.conf 2024-05-27T07:15:08.981 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:15:08 smithi052 ceph-mon[31955]: Updating smithi173:/var/lib/ceph/73df2636-1bf8-11ef-bc9b-c7b262605968/config/ceph.conf 2024-05-27T07:15:08.981 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:15:08 smithi052 ceph-mon[31955]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' 2024-05-27T07:15:08.981 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:15:08 smithi052 ceph-mon[31955]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' 2024-05-27T07:15:08.981 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:15:08 smithi052 ceph-mon[31955]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' 2024-05-27T07:15:09.980 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:15:09 smithi052 ceph-mon[31955]: Updating smithi173:/etc/ceph/ceph.client.admin.keyring 2024-05-27T07:15:09.980 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:15:09 smithi052 ceph-mon[31955]: Updating smithi173:/var/lib/ceph/73df2636-1bf8-11ef-bc9b-c7b262605968/config/ceph.client.admin.keyring 2024-05-27T07:15:10.082 INFO:teuthology.orchestra.run.smithi173.stderr:2024-05-27T07:15:10.091+0000 7f1b046c9700 1 -- 172.21.15.173:0/3069769146 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f1afc109c30 msgr2=0x7f1afc10a040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:15:10.082 INFO:teuthology.orchestra.run.smithi173.stderr:2024-05-27T07:15:10.091+0000 7f1b046c9700 1 --2- 172.21.15.173:0/3069769146 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f1afc109c30 0x7f1afc10a040 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f1af8009910 tx=0x7f1af8009c20 comp rx=0 tx=0).stop 2024-05-27T07:15:10.083 INFO:teuthology.orchestra.run.smithi173.stderr:2024-05-27T07:15:10.091+0000 7f1b046c9700 1 -- 172.21.15.173:0/3069769146 shutdown_connections 2024-05-27T07:15:10.083 INFO:teuthology.orchestra.run.smithi173.stderr:2024-05-27T07:15:10.091+0000 7f1b046c9700 1 --2- 172.21.15.173:0/3069769146 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f1afc109c30 0x7f1afc10a040 unknown :-1 s=CLOSED pgs=94 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:15:10.083 INFO:teuthology.orchestra.run.smithi173.stderr:2024-05-27T07:15:10.091+0000 7f1b046c9700 1 -- 172.21.15.173:0/3069769146 >> 172.21.15.173:0/3069769146 conn(0x7f1afc0ff650 msgr2=0x7f1afc101aa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:15:10.083 INFO:teuthology.orchestra.run.smithi173.stderr:2024-05-27T07:15:10.091+0000 7f1b046c9700 1 -- 172.21.15.173:0/3069769146 shutdown_connections 2024-05-27T07:15:10.083 INFO:teuthology.orchestra.run.smithi173.stderr:2024-05-27T07:15:10.091+0000 7f1b046c9700 1 -- 172.21.15.173:0/3069769146 wait complete. 2024-05-27T07:15:10.083 INFO:teuthology.orchestra.run.smithi173.stderr:2024-05-27T07:15:10.091+0000 7f1b046c9700 1 Processor -- start 2024-05-27T07:15:10.083 INFO:teuthology.orchestra.run.smithi173.stderr:2024-05-27T07:15:10.091+0000 7f1b046c9700 1 -- start start 2024-05-27T07:15:10.083 INFO:teuthology.orchestra.run.smithi173.stderr:2024-05-27T07:15:10.092+0000 7f1b046c9700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f1afc109c30 0x7f1afc19f5e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:15:10.083 INFO:teuthology.orchestra.run.smithi173.stderr:2024-05-27T07:15:10.092+0000 7f1b046c9700 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x7f1afc10e110 con 0x7f1afc109c30 2024-05-27T07:15:10.085 INFO:teuthology.orchestra.run.smithi173.stderr:2024-05-27T07:15:10.092+0000 7f1b02465700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f1afc109c30 0x7f1afc19f5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:15:10.085 INFO:teuthology.orchestra.run.smithi173.stderr:2024-05-27T07:15:10.092+0000 7f1b02465700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f1afc109c30 0x7f1afc19f5e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.173:57568/0 (socket says 172.21.15.173:57568) 2024-05-27T07:15:10.085 INFO:teuthology.orchestra.run.smithi173.stderr:2024-05-27T07:15:10.092+0000 7f1b02465700 1 -- 172.21.15.173:0/2826181800 learned_addr learned my addr 172.21.15.173:0/2826181800 (peer_addr_for_me v2:172.21.15.173:0/0) 2024-05-27T07:15:10.085 INFO:teuthology.orchestra.run.smithi173.stderr:2024-05-27T07:15:10.092+0000 7f1b02465700 1 -- 172.21.15.173:0/2826181800 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1afc19fb20 con 0x7f1afc109c30 2024-05-27T07:15:10.085 INFO:teuthology.orchestra.run.smithi173.stderr:2024-05-27T07:15:10.093+0000 7f1b02465700 1 --2- 172.21.15.173:0/2826181800 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f1afc109c30 0x7f1afc19f5e0 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f1af80092e0 tx=0x7f1af8005780 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:15:10.085 INFO:teuthology.orchestra.run.smithi173.stderr:2024-05-27T07:15:10.093+0000 7f1af37fe700 1 -- 172.21.15.173:0/2826181800 <== mon.0 v2:172.21.15.52:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f1af8026070 con 0x7f1afc109c30 2024-05-27T07:15:10.085 INFO:teuthology.orchestra.run.smithi173.stderr:2024-05-27T07:15:10.093+0000 7f1b046c9700 1 -- 172.21.15.173:0/2826181800 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1afc19fd50 con 0x7f1afc109c30 2024-05-27T07:15:10.085 INFO:teuthology.orchestra.run.smithi173.stderr:2024-05-27T07:15:10.093+0000 7f1af37fe700 1 -- 172.21.15.173:0/2826181800 <== mon.0 v2:172.21.15.52:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f1af800b750 con 0x7f1afc109c30 2024-05-27T07:15:10.085 INFO:teuthology.orchestra.run.smithi173.stderr:2024-05-27T07:15:10.093+0000 7f1b046c9700 1 -- 172.21.15.173:0/2826181800 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1afc1997c0 con 0x7f1afc109c30 2024-05-27T07:15:10.085 INFO:teuthology.orchestra.run.smithi173.stderr:2024-05-27T07:15:10.093+0000 7f1af37fe700 1 -- 172.21.15.173:0/2826181800 <== mon.0 v2:172.21.15.52:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f1af8003890 con 0x7f1afc109c30 2024-05-27T07:15:10.085 INFO:teuthology.orchestra.run.smithi173.stderr:2024-05-27T07:15:10.094+0000 7f1b046c9700 1 -- 172.21.15.173:0/2826181800 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1afc105200 con 0x7f1afc109c30 2024-05-27T07:15:10.086 INFO:teuthology.orchestra.run.smithi173.stderr:2024-05-27T07:15:10.094+0000 7f1af37fe700 1 -- 172.21.15.173:0/2826181800 <== mon.0 v2:172.21.15.52:3300/0 4 ==== mgrmap(e 13) ==== 50278+0+0 (secure 0 0 0) 0x7f1af8029070 con 0x7f1afc109c30 2024-05-27T07:15:10.086 INFO:teuthology.orchestra.run.smithi173.stderr:2024-05-27T07:15:10.094+0000 7f1af37fe700 1 --2- 172.21.15.173:0/2826181800 >> [v2:172.21.15.52:6800/3513075805,v1:172.21.15.52:6801/3513075805] conn(0x7f1ae803e510 0x7f1ae8040a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:15:10.086 INFO:teuthology.orchestra.run.smithi173.stderr:2024-05-27T07:15:10.094+0000 7f1af37fe700 1 -- 172.21.15.173:0/2826181800 <== mon.0 v2:172.21.15.52:3300/0 5 ==== osd_map(4..4 src has 1..4) ==== 1155+0+0 (secure 0 0 0) 0x7f1af801e070 con 0x7f1afc109c30 2024-05-27T07:15:10.086 INFO:teuthology.orchestra.run.smithi173.stderr:2024-05-27T07:15:10.094+0000 7f1b01c64700 1 --2- 172.21.15.173:0/2826181800 >> [v2:172.21.15.52:6800/3513075805,v1:172.21.15.52:6801/3513075805] conn(0x7f1ae803e510 0x7f1ae8040a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:15:10.086 INFO:teuthology.orchestra.run.smithi173.stderr:2024-05-27T07:15:10.095+0000 7f1b01c64700 1 --2- 172.21.15.173:0/2826181800 >> [v2:172.21.15.52:6800/3513075805,v1:172.21.15.52:6801/3513075805] conn(0x7f1ae803e510 0x7f1ae8040a20 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7f1aec006fd0 tx=0x7f1aec006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:15:10.091 INFO:teuthology.orchestra.run.smithi173.stderr:2024-05-27T07:15:10.100+0000 7f1af37fe700 1 -- 172.21.15.173:0/2826181800 <== mon.0 v2:172.21.15.52:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191963 (secure 0 0 0) 0x7f1af8013070 con 0x7f1afc109c30 2024-05-27T07:15:10.304 INFO:teuthology.orchestra.run.smithi173.stderr:2024-05-27T07:15:10.314+0000 7f1b046c9700 1 -- 172.21.15.173:0/2826181800 --> [v2:172.21.15.52:6800/3513075805,v1:172.21.15.52:6801/3513075805] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "mon", "placement": "3;smithi052:172.21.15.52=a;smithi105:172.21.15.105=b;smithi173:172.21.15.173=c", "target": ["mon-mgr", ""]}) -- 0x7f1afc19acd0 con 0x7f1ae803e510 2024-05-27T07:15:10.312 INFO:teuthology.orchestra.run.smithi173.stderr:2024-05-27T07:15:10.321+0000 7f1af37fe700 1 -- 172.21.15.173:0/2826181800 <== mgr.14152 v2:172.21.15.52:6800/3513075805 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+24 (secure 0 0 0) 0x7f1afc19acd0 con 0x7f1ae803e510 2024-05-27T07:15:10.313 INFO:teuthology.orchestra.run.smithi173.stdout:Scheduled mon update... 2024-05-27T07:15:10.314 INFO:teuthology.orchestra.run.smithi173.stderr:2024-05-27T07:15:10.323+0000 7f1b046c9700 1 -- 172.21.15.173:0/2826181800 >> [v2:172.21.15.52:6800/3513075805,v1:172.21.15.52:6801/3513075805] conn(0x7f1ae803e510 msgr2=0x7f1ae8040a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:15:10.314 INFO:teuthology.orchestra.run.smithi173.stderr:2024-05-27T07:15:10.323+0000 7f1b046c9700 1 --2- 172.21.15.173:0/2826181800 >> [v2:172.21.15.52:6800/3513075805,v1:172.21.15.52:6801/3513075805] conn(0x7f1ae803e510 0x7f1ae8040a20 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7f1aec006fd0 tx=0x7f1aec006e40 comp rx=0 tx=0).stop 2024-05-27T07:15:10.315 INFO:teuthology.orchestra.run.smithi173.stderr:2024-05-27T07:15:10.323+0000 7f1b046c9700 1 -- 172.21.15.173:0/2826181800 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f1afc109c30 msgr2=0x7f1afc19f5e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:15:10.315 INFO:teuthology.orchestra.run.smithi173.stderr:2024-05-27T07:15:10.323+0000 7f1b046c9700 1 --2- 172.21.15.173:0/2826181800 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f1afc109c30 0x7f1afc19f5e0 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f1af80092e0 tx=0x7f1af8005780 comp rx=0 tx=0).stop 2024-05-27T07:15:10.315 INFO:teuthology.orchestra.run.smithi173.stderr:2024-05-27T07:15:10.323+0000 7f1b046c9700 1 -- 172.21.15.173:0/2826181800 shutdown_connections 2024-05-27T07:15:10.315 INFO:teuthology.orchestra.run.smithi173.stderr:2024-05-27T07:15:10.323+0000 7f1b046c9700 1 --2- 172.21.15.173:0/2826181800 >> [v2:172.21.15.52:6800/3513075805,v1:172.21.15.52:6801/3513075805] conn(0x7f1ae803e510 0x7f1ae8040a20 unknown :-1 s=CLOSED pgs=19 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:15:10.315 INFO:teuthology.orchestra.run.smithi173.stderr:2024-05-27T07:15:10.323+0000 7f1b046c9700 1 --2- 172.21.15.173:0/2826181800 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f1afc109c30 0x7f1afc19f5e0 unknown :-1 s=CLOSED pgs=95 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:15:10.315 INFO:teuthology.orchestra.run.smithi173.stderr:2024-05-27T07:15:10.323+0000 7f1b046c9700 1 -- 172.21.15.173:0/2826181800 >> 172.21.15.173:0/2826181800 conn(0x7f1afc0ff650 msgr2=0x7f1afc100540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:15:10.315 INFO:teuthology.orchestra.run.smithi173.stderr:2024-05-27T07:15:10.323+0000 7f1b046c9700 1 -- 172.21.15.173:0/2826181800 shutdown_connections 2024-05-27T07:15:10.315 INFO:teuthology.orchestra.run.smithi173.stderr:2024-05-27T07:15:10.323+0000 7f1b046c9700 1 -- 172.21.15.173:0/2826181800 wait complete. 2024-05-27T07:15:10.920 DEBUG:teuthology.orchestra.run.smithi105:mon.b> sudo journalctl -f -n 0 -u ceph-73df2636-1bf8-11ef-bc9b-c7b262605968@mon.b.service 2024-05-27T07:15:10.922 DEBUG:teuthology.orchestra.run.smithi173:mon.c> sudo journalctl -f -n 0 -u ceph-73df2636-1bf8-11ef-bc9b-c7b262605968@mon.c.service 2024-05-27T07:15:10.925 INFO:tasks.cephadm:Waiting for 3 mons in monmap... 2024-05-27T07:15:10.925 DEBUG:teuthology.orchestra.run.smithi173:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:38f9d900c83c63799fbdbe61acc9a11b0d3554a6 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 73df2636-1bf8-11ef-bc9b-c7b262605968 -- ceph mon dump -f json 2024-05-27T07:15:10.980 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:15:10 smithi052 ceph-mon[31955]: pgmap v14: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-27T07:15:10.980 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:15:10 smithi052 ceph-mon[31955]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' 2024-05-27T07:15:10.980 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:15:10 smithi052 ceph-mon[31955]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-27T07:15:10.980 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:15:10 smithi052 ceph-mon[31955]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T07:15:10.980 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:15:10 smithi052 ceph-mon[31955]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-27T07:15:10.980 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:15:10 smithi052 ceph-mon[31955]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' 2024-05-27T07:15:10.980 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:15:10 smithi052 ceph-mon[31955]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-05-27T07:15:10.980 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:15:10 smithi052 ceph-mon[31955]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T07:15:11.655 INFO:teuthology.orchestra.run.smithi173.stderr:Inferring config /var/lib/ceph/73df2636-1bf8-11ef-bc9b-c7b262605968/config/ceph.conf 2024-05-27T07:15:11.980 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:15:11 smithi052 ceph-mon[31955]: from='client.14188 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "placement": "3;smithi052:172.21.15.52=a;smithi105:172.21.15.105=b;smithi173:172.21.15.173=c", "target": ["mon-mgr", ""]}]: dispatch 2024-05-27T07:15:11.980 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:15:11 smithi052 ceph-mon[31955]: Saving service mon spec with placement smithi052:172.21.15.52=a;smithi105:172.21.15.105=b;smithi173:172.21.15.173=c;count:3 2024-05-27T07:15:11.980 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:15:11 smithi052 ceph-mon[31955]: Deploying daemon mon.c on smithi173 2024-05-27T07:15:12.980 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:15:12 smithi052 ceph-mon[31955]: pgmap v15: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-27T07:15:13.568 INFO:teuthology.orchestra.run.smithi173.stderr:2024-05-27T07:15:13.576+0000 7f986ec26700 1 -- 172.21.15.173:0/1868000122 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f9868108a50 msgr2=0x7f9868108e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:15:13.568 INFO:teuthology.orchestra.run.smithi173.stderr:2024-05-27T07:15:13.576+0000 7f986ec26700 1 --2- 172.21.15.173:0/1868000122 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f9868108a50 0x7f9868108e60 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7f985c009910 tx=0x7f985c009c20 comp rx=0 tx=0).stop 2024-05-27T07:15:13.568 INFO:teuthology.orchestra.run.smithi173.stderr:2024-05-27T07:15:13.576+0000 7f986ec26700 1 -- 172.21.15.173:0/1868000122 shutdown_connections 2024-05-27T07:15:13.568 INFO:teuthology.orchestra.run.smithi173.stderr:2024-05-27T07:15:13.576+0000 7f986ec26700 1 --2- 172.21.15.173:0/1868000122 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f9868108a50 0x7f9868108e60 unknown :-1 s=CLOSED pgs=96 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:15:13.568 INFO:teuthology.orchestra.run.smithi173.stderr:2024-05-27T07:15:13.576+0000 7f986ec26700 1 -- 172.21.15.173:0/1868000122 >> 172.21.15.173:0/1868000122 conn(0x7f98680fe560 msgr2=0x7f98681009b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:15:13.568 INFO:teuthology.orchestra.run.smithi173.stderr:2024-05-27T07:15:13.576+0000 7f986ec26700 1 -- 172.21.15.173:0/1868000122 shutdown_connections 2024-05-27T07:15:13.568 INFO:teuthology.orchestra.run.smithi173.stderr:2024-05-27T07:15:13.576+0000 7f986ec26700 1 -- 172.21.15.173:0/1868000122 wait complete. 2024-05-27T07:15:13.568 INFO:teuthology.orchestra.run.smithi173.stderr:2024-05-27T07:15:13.577+0000 7f986ec26700 1 Processor -- start 2024-05-27T07:15:13.568 INFO:teuthology.orchestra.run.smithi173.stderr:2024-05-27T07:15:13.577+0000 7f986ec26700 1 -- start start 2024-05-27T07:15:13.568 INFO:teuthology.orchestra.run.smithi173.stderr:2024-05-27T07:15:13.577+0000 7f986ec26700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f9868108a50 0x7f986819f5e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:15:13.568 INFO:teuthology.orchestra.run.smithi173.stderr:2024-05-27T07:15:13.577+0000 7f986ec26700 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x7f986810cfa0 con 0x7f9868108a50 2024-05-27T07:15:13.569 INFO:teuthology.orchestra.run.smithi173.stderr:2024-05-27T07:15:13.577+0000 7f986c9c2700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f9868108a50 0x7f986819f5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:15:13.569 INFO:teuthology.orchestra.run.smithi173.stderr:2024-05-27T07:15:13.577+0000 7f986c9c2700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f9868108a50 0x7f986819f5e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.173:57596/0 (socket says 172.21.15.173:57596) 2024-05-27T07:15:13.570 INFO:teuthology.orchestra.run.smithi173.stderr:2024-05-27T07:15:13.577+0000 7f986c9c2700 1 -- 172.21.15.173:0/53244174 learned_addr learned my addr 172.21.15.173:0/53244174 (peer_addr_for_me v2:172.21.15.173:0/0) 2024-05-27T07:15:13.570 INFO:teuthology.orchestra.run.smithi173.stderr:2024-05-27T07:15:13.578+0000 7f986c9c2700 1 -- 172.21.15.173:0/53244174 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f986819fb20 con 0x7f9868108a50 2024-05-27T07:15:13.570 INFO:teuthology.orchestra.run.smithi173.stderr:2024-05-27T07:15:13.578+0000 7f986c9c2700 1 --2- 172.21.15.173:0/53244174 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f9868108a50 0x7f986819f5e0 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f985c0056d0 tx=0x7f985c0057b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:15:13.570 INFO:teuthology.orchestra.run.smithi173.stderr:2024-05-27T07:15:13.578+0000 7f9865ffb700 1 -- 172.21.15.173:0/53244174 <== mon.0 v2:172.21.15.52:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f985c027070 con 0x7f9868108a50 2024-05-27T07:15:13.570 INFO:teuthology.orchestra.run.smithi173.stderr:2024-05-27T07:15:13.578+0000 7f986ec26700 1 -- 172.21.15.173:0/53244174 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f986819fe10 con 0x7f9868108a50 2024-05-27T07:15:13.570 INFO:teuthology.orchestra.run.smithi173.stderr:2024-05-27T07:15:13.578+0000 7f9865ffb700 1 -- 172.21.15.173:0/53244174 <== mon.0 v2:172.21.15.52:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f985c00be10 con 0x7f9868108a50 2024-05-27T07:15:13.570 INFO:teuthology.orchestra.run.smithi173.stderr:2024-05-27T07:15:13.578+0000 7f9865ffb700 1 -- 172.21.15.173:0/53244174 <== mon.0 v2:172.21.15.52:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f985c022d20 con 0x7f9868108a50 2024-05-27T07:15:13.570 INFO:teuthology.orchestra.run.smithi173.stderr:2024-05-27T07:15:13.578+0000 7f986ec26700 1 -- 172.21.15.173:0/53244174 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9868199940 con 0x7f9868108a50 2024-05-27T07:15:13.571 INFO:teuthology.orchestra.run.smithi173.stderr:2024-05-27T07:15:13.579+0000 7f986ec26700 1 -- 172.21.15.173:0/53244174 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9868104160 con 0x7f9868108a50 2024-05-27T07:15:13.571 INFO:teuthology.orchestra.run.smithi173.stderr:2024-05-27T07:15:13.579+0000 7f9865ffb700 1 -- 172.21.15.173:0/53244174 <== mon.0 v2:172.21.15.52:3300/0 4 ==== mgrmap(e 13) ==== 50278+0+0 (secure 0 0 0) 0x7f985c02a070 con 0x7f9868108a50 2024-05-27T07:15:13.571 INFO:teuthology.orchestra.run.smithi173.stderr:2024-05-27T07:15:13.580+0000 7f9865ffb700 1 --2- 172.21.15.173:0/53244174 >> [v2:172.21.15.52:6800/3513075805,v1:172.21.15.52:6801/3513075805] conn(0x7f985803e510 0x7f9858040a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:15:13.571 INFO:teuthology.orchestra.run.smithi173.stderr:2024-05-27T07:15:13.580+0000 7f9865ffb700 1 -- 172.21.15.173:0/53244174 <== mon.0 v2:172.21.15.52:3300/0 5 ==== osd_map(4..4 src has 1..4) ==== 1155+0+0 (secure 0 0 0) 0x7f985c01e030 con 0x7f9868108a50 2024-05-27T07:15:13.571 INFO:teuthology.orchestra.run.smithi173.stderr:2024-05-27T07:15:13.580+0000 7f9867fff700 1 --2- 172.21.15.173:0/53244174 >> [v2:172.21.15.52:6800/3513075805,v1:172.21.15.52:6801/3513075805] conn(0x7f985803e510 0x7f9858040a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:15:13.571 INFO:teuthology.orchestra.run.smithi173.stderr:2024-05-27T07:15:13.580+0000 7f9867fff700 1 --2- 172.21.15.173:0/53244174 >> [v2:172.21.15.52:6800/3513075805,v1:172.21.15.52:6801/3513075805] conn(0x7f985803e510 0x7f9858040a20 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7f9854006fd0 tx=0x7f9854006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:15:13.576 INFO:teuthology.orchestra.run.smithi173.stderr:2024-05-27T07:15:13.585+0000 7f9865ffb700 1 -- 172.21.15.173:0/53244174 <== mon.0 v2:172.21.15.52:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191963 (secure 0 0 0) 0x7f985c013070 con 0x7f9868108a50 2024-05-27T07:15:13.878 INFO:teuthology.orchestra.run.smithi173.stderr:2024-05-27T07:15:13.887+0000 7f986ec26700 1 -- 172.21.15.173:0/53244174 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) -- 0x7f98681a0040 con 0x7f9868108a50 2024-05-27T07:15:13.879 INFO:teuthology.orchestra.run.smithi173.stderr:2024-05-27T07:15:13.888+0000 7f9865ffb700 1 -- 172.21.15.173:0/53244174 <== mon.0 v2:172.21.15.52:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) ==== 95+0+745 (secure 0 0 0) 0x7f985c031090 con 0x7f9868108a50 2024-05-27T07:15:13.879 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-05-27T07:15:13.879 INFO:teuthology.orchestra.run.smithi173.stdout:{"epoch":1,"fsid":"73df2636-1bf8-11ef-bc9b-c7b262605968","modified":"2024-05-27T07:13:31.465220Z","created":"2024-05-27T07:13:31.465220Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.52:3300","nonce":0},{"type":"v1","addr":"172.21.15.52:6789","nonce":0}]},"addr":"172.21.15.52:6789/0","public_addr":"172.21.15.52:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-05-27T07:15:13.882 INFO:teuthology.orchestra.run.smithi173.stderr:2024-05-27T07:15:13.891+0000 7f986ec26700 1 -- 172.21.15.173:0/53244174 >> [v2:172.21.15.52:6800/3513075805,v1:172.21.15.52:6801/3513075805] conn(0x7f985803e510 msgr2=0x7f9858040a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:15:13.882 INFO:teuthology.orchestra.run.smithi173.stderr:2024-05-27T07:15:13.891+0000 7f986ec26700 1 --2- 172.21.15.173:0/53244174 >> [v2:172.21.15.52:6800/3513075805,v1:172.21.15.52:6801/3513075805] conn(0x7f985803e510 0x7f9858040a20 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7f9854006fd0 tx=0x7f9854006e40 comp rx=0 tx=0).stop 2024-05-27T07:15:13.882 INFO:teuthology.orchestra.run.smithi173.stderr:2024-05-27T07:15:13.891+0000 7f986ec26700 1 -- 172.21.15.173:0/53244174 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f9868108a50 msgr2=0x7f986819f5e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:15:13.882 INFO:teuthology.orchestra.run.smithi173.stderr:2024-05-27T07:15:13.891+0000 7f986ec26700 1 --2- 172.21.15.173:0/53244174 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f9868108a50 0x7f986819f5e0 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f985c0056d0 tx=0x7f985c0057b0 comp rx=0 tx=0).stop 2024-05-27T07:15:13.882 INFO:teuthology.orchestra.run.smithi173.stderr:2024-05-27T07:15:13.891+0000 7f986ec26700 1 -- 172.21.15.173:0/53244174 shutdown_connections 2024-05-27T07:15:13.882 INFO:teuthology.orchestra.run.smithi173.stderr:2024-05-27T07:15:13.891+0000 7f986ec26700 1 --2- 172.21.15.173:0/53244174 >> [v2:172.21.15.52:6800/3513075805,v1:172.21.15.52:6801/3513075805] conn(0x7f985803e510 0x7f9858040a20 unknown :-1 s=CLOSED pgs=20 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:15:13.882 INFO:teuthology.orchestra.run.smithi173.stderr:2024-05-27T07:15:13.891+0000 7f986ec26700 1 --2- 172.21.15.173:0/53244174 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f9868108a50 0x7f986819f5e0 unknown :-1 s=CLOSED pgs=97 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:15:13.882 INFO:teuthology.orchestra.run.smithi173.stderr:2024-05-27T07:15:13.891+0000 7f986ec26700 1 -- 172.21.15.173:0/53244174 >> 172.21.15.173:0/53244174 conn(0x7f98680fe560 msgr2=0x7f9868106530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:15:13.882 INFO:teuthology.orchestra.run.smithi173.stderr:2024-05-27T07:15:13.891+0000 7f986ec26700 1 -- 172.21.15.173:0/53244174 shutdown_connections 2024-05-27T07:15:13.882 INFO:teuthology.orchestra.run.smithi173.stderr:2024-05-27T07:15:13.891+0000 7f986ec26700 1 -- 172.21.15.173:0/53244174 wait complete. 2024-05-27T07:15:13.883 INFO:teuthology.orchestra.run.smithi173.stderr:dumped monmap epoch 1 2024-05-27T07:15:14.980 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:15:14 smithi052 ceph-mon[31955]: pgmap v16: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-27T07:15:14.980 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:15:14 smithi052 ceph-mon[31955]: from='client.? 172.21.15.173:0/53244174' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-05-27T07:15:15.603 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:15:15 smithi173 podman[36981]: 2024-05-27 07:15:15.463527646 +0000 UTC m=+0.017402773 image pull e36ea4fc8a5807e8d2701370e350a0de3856538797f7fba64cfce7e344bd6799 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6 2024-05-27T07:15:16.052 INFO:tasks.cephadm:Waiting for 3 mons in monmap... 2024-05-27T07:15:16.052 DEBUG:teuthology.orchestra.run.smithi173:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:38f9d900c83c63799fbdbe61acc9a11b0d3554a6 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 73df2636-1bf8-11ef-bc9b-c7b262605968 -- ceph mon dump -f json 2024-05-27T07:15:16.076 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:15:15 smithi173 podman[36981]: 2024-05-27 07:15:15.728697696 +0000 UTC m=+0.282572817 container init de9961ba152193f89fc37e84f5f5e9e9c83b488327cbf64a307dcc95c6bd88d9 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mon-c, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, org.label-schema.build-date=20240521, CEPH_POINT_RELEASE=, ceph=True) 2024-05-27T07:15:16.076 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:15:15 smithi173 podman[36981]: 2024-05-27 07:15:15.73201874 +0000 UTC m=+0.285893870 container start de9961ba152193f89fc37e84f5f5e9e9c83b488327cbf64a307dcc95c6bd88d9 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mon-c, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, org.label-schema.build-date=20240521, io.buildah.version=1.33.5, GIT_CLEAN=True, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-05-27T07:15:16.076 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:15:15 smithi173 ceph-mon[37000]: set uid:gid to 167:167 (ceph:ceph) 2024-05-27T07:15:16.076 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:15:15 smithi173 ceph-mon[37000]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev), process ceph-mon, pid 2 2024-05-27T07:15:16.076 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:15:15 smithi173 ceph-mon[37000]: pidfile_write: ignore empty --pid-file 2024-05-27T07:15:16.076 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:15:15 smithi173 ceph-mon[37000]: load: jerasure load: lrc 2024-05-27T07:15:16.076 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:15:15 smithi173 ceph-mon[37000]: rocksdb: RocksDB version: 7.9.2 2024-05-27T07:15:16.076 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:15:15 smithi173 ceph-mon[37000]: rocksdb: Git sha 0 2024-05-27T07:15:16.077 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:15:15 smithi173 ceph-mon[37000]: rocksdb: Compile date 2024-05-21 07:00:12 2024-05-27T07:15:16.077 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:15:15 smithi173 ceph-mon[37000]: rocksdb: DB SUMMARY 2024-05-27T07:15:16.077 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:15:15 smithi173 ceph-mon[37000]: rocksdb: DB Session ID: NCW1JYWGA3RXP6ZCA6C3 2024-05-27T07:15:16.077 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:15:15 smithi173 ceph-mon[37000]: rocksdb: CURRENT file: CURRENT 2024-05-27T07:15:16.077 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:15:15 smithi173 ceph-mon[37000]: rocksdb: IDENTITY file: IDENTITY 2024-05-27T07:15:16.077 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:15:15 smithi173 ceph-mon[37000]: rocksdb: MANIFEST file: MANIFEST-000005 size: 59 Bytes 2024-05-27T07:15:16.077 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:15:15 smithi173 ceph-mon[37000]: rocksdb: SST files in /var/lib/ceph/mon/ceph-c/store.db dir, Total Num: 0, files: 2024-05-27T07:15:16.077 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:15:15 smithi173 ceph-mon[37000]: rocksdb: Write Ahead Log file in /var/lib/ceph/mon/ceph-c/store.db: 000004.log size: 511 ; 2024-05-27T07:15:16.077 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:15:15 smithi173 ceph-mon[37000]: rocksdb: Options.error_if_exists: 0 2024-05-27T07:15:16.077 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:15:15 smithi173 ceph-mon[37000]: rocksdb: Options.create_if_missing: 0 2024-05-27T07:15:16.078 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:15:15 smithi173 ceph-mon[37000]: rocksdb: Options.paranoid_checks: 1 2024-05-27T07:15:16.078 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:15:15 smithi173 ceph-mon[37000]: rocksdb: Options.flush_verify_memtable_count: 1 2024-05-27T07:15:16.078 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:15:15 smithi173 ceph-mon[37000]: rocksdb: Options.track_and_verify_wals_in_manifest: 0 2024-05-27T07:15:16.078 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:15:15 smithi173 ceph-mon[37000]: rocksdb: Options.verify_sst_unique_id_in_manifest: 1 2024-05-27T07:15:16.078 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:15:15 smithi173 ceph-mon[37000]: rocksdb: Options.env: 0x55db9bec0500 2024-05-27T07:15:16.078 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:15:15 smithi173 ceph-mon[37000]: rocksdb: Options.fs: PosixFileSystem 2024-05-27T07:15:16.078 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:15:15 smithi173 ceph-mon[37000]: rocksdb: Options.info_log: 0x55db9cf11fe0 2024-05-27T07:15:16.078 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:15:15 smithi173 ceph-mon[37000]: rocksdb: Options.max_file_opening_threads: 16 2024-05-27T07:15:16.078 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:15:15 smithi173 ceph-mon[37000]: rocksdb: Options.statistics: (nil) 2024-05-27T07:15:16.078 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:15:15 smithi173 ceph-mon[37000]: rocksdb: Options.use_fsync: 0 2024-05-27T07:15:16.078 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:15:15 smithi173 ceph-mon[37000]: rocksdb: Options.max_log_file_size: 0 2024-05-27T07:15:16.079 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:15:15 smithi173 ceph-mon[37000]: rocksdb: Options.max_manifest_file_size: 1073741824 2024-05-27T07:15:16.079 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:15:15 smithi173 ceph-mon[37000]: rocksdb: Options.log_file_time_to_roll: 0 2024-05-27T07:15:16.079 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:15:15 smithi173 ceph-mon[37000]: rocksdb: Options.keep_log_file_num: 1000 2024-05-27T07:15:16.079 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:15:15 smithi173 ceph-mon[37000]: rocksdb: Options.recycle_log_file_num: 0 2024-05-27T07:15:16.079 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:15:15 smithi173 ceph-mon[37000]: rocksdb: Options.allow_fallocate: 1 2024-05-27T07:15:16.079 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:15:15 smithi173 ceph-mon[37000]: rocksdb: Options.allow_mmap_reads: 0 2024-05-27T07:15:16.079 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:15:15 smithi173 ceph-mon[37000]: rocksdb: Options.allow_mmap_writes: 0 2024-05-27T07:15:16.079 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:15:15 smithi173 ceph-mon[37000]: rocksdb: Options.use_direct_reads: 0 2024-05-27T07:15:16.079 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:15:15 smithi173 ceph-mon[37000]: rocksdb: Options.use_direct_io_for_flush_and_compaction: 0 2024-05-27T07:15:16.079 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:15:15 smithi173 ceph-mon[37000]: rocksdb: Options.create_missing_column_families: 0 2024-05-27T07:15:16.079 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:15:15 smithi173 ceph-mon[37000]: rocksdb: Options.db_log_dir: 2024-05-27T07:15:16.079 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:15:15 smithi173 ceph-mon[37000]: rocksdb: Options.wal_dir: 2024-05-27T07:15:16.080 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:15:15 smithi173 ceph-mon[37000]: rocksdb: Options.table_cache_numshardbits: 6 2024-05-27T07:15:16.080 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:15:15 smithi173 ceph-mon[37000]: rocksdb: Options.WAL_ttl_seconds: 0 2024-05-27T07:15:16.080 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:15:15 smithi173 ceph-mon[37000]: rocksdb: Options.WAL_size_limit_MB: 0 2024-05-27T07:15:16.080 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:15:15 smithi173 ceph-mon[37000]: rocksdb: Options.max_write_batch_group_size_bytes: 1048576 2024-05-27T07:15:16.080 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:15:15 smithi173 ceph-mon[37000]: rocksdb: Options.manifest_preallocation_size: 4194304 2024-05-27T07:15:16.080 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:15:15 smithi173 ceph-mon[37000]: rocksdb: Options.is_fd_close_on_exec: 1 2024-05-27T07:15:16.080 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:15:15 smithi173 ceph-mon[37000]: rocksdb: Options.advise_random_on_open: 1 2024-05-27T07:15:16.080 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:15:15 smithi173 ceph-mon[37000]: rocksdb: Options.db_write_buffer_size: 0 2024-05-27T07:15:16.080 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:15:15 smithi173 ceph-mon[37000]: rocksdb: Options.write_buffer_manager: 0x55db9cee28c0 2024-05-27T07:15:16.080 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:15:15 smithi173 ceph-mon[37000]: rocksdb: Options.access_hint_on_compaction_start: 1 2024-05-27T07:15:16.081 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:15:15 smithi173 ceph-mon[37000]: rocksdb: Options.random_access_max_buffer_size: 1048576 2024-05-27T07:15:16.081 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:15:15 smithi173 ceph-mon[37000]: rocksdb: Options.use_adaptive_mutex: 0 2024-05-27T07:15:16.081 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:15:15 smithi173 ceph-mon[37000]: rocksdb: Options.rate_limiter: (nil) 2024-05-27T07:15:16.081 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:15:15 smithi173 ceph-mon[37000]: rocksdb: Options.sst_file_manager.rate_bytes_per_sec: 0 2024-05-27T07:15:16.081 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:15:15 smithi173 ceph-mon[37000]: rocksdb: Options.wal_recovery_mode: 2 2024-05-27T07:15:16.081 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:15:15 smithi173 ceph-mon[37000]: rocksdb: Options.enable_thread_tracking: 0 2024-05-27T07:15:16.081 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:15:15 smithi173 ceph-mon[37000]: rocksdb: Options.enable_pipelined_write: 0 2024-05-27T07:15:16.081 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:15:15 smithi173 ceph-mon[37000]: rocksdb: Options.unordered_write: 0 2024-05-27T07:15:16.081 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:15:15 smithi173 ceph-mon[37000]: rocksdb: Options.allow_concurrent_memtable_write: 1 2024-05-27T07:15:16.081 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:15:15 smithi173 ceph-mon[37000]: rocksdb: Options.enable_write_thread_adaptive_yield: 1 2024-05-27T07:15:16.081 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:15:15 smithi173 ceph-mon[37000]: rocksdb: Options.write_thread_max_yield_usec: 100 2024-05-27T07:15:16.081 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:15:15 smithi173 ceph-mon[37000]: rocksdb: Options.write_thread_slow_yield_usec: 3 2024-05-27T07:15:16.081 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:15:15 smithi173 ceph-mon[37000]: rocksdb: Options.row_cache: None 2024-05-27T07:15:16.082 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:15:15 smithi173 ceph-mon[37000]: rocksdb: Options.wal_filter: None 2024-05-27T07:15:16.083 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:15:15 smithi173 ceph-mon[37000]: rocksdb: Options.avoid_flush_during_recovery: 0 2024-05-27T07:15:16.083 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:15:15 smithi173 ceph-mon[37000]: rocksdb: Options.allow_ingest_behind: 0 2024-05-27T07:15:16.083 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:15:15 smithi173 ceph-mon[37000]: rocksdb: Options.two_write_queues: 0 2024-05-27T07:15:16.083 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:15:15 smithi173 ceph-mon[37000]: rocksdb: Options.manual_wal_flush: 0 2024-05-27T07:15:16.083 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:15:15 smithi173 ceph-mon[37000]: rocksdb: Options.wal_compression: 0 2024-05-27T07:15:16.084 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:15:15 smithi173 ceph-mon[37000]: rocksdb: Options.atomic_flush: 0 2024-05-27T07:15:16.084 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:15:15 smithi173 ceph-mon[37000]: rocksdb: Options.avoid_unnecessary_blocking_io: 0 2024-05-27T07:15:16.084 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:15:15 smithi173 ceph-mon[37000]: rocksdb: Options.persist_stats_to_disk: 0 2024-05-27T07:15:16.084 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:15:15 smithi173 ceph-mon[37000]: rocksdb: Options.write_dbid_to_manifest: 0 2024-05-27T07:15:16.084 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:15:15 smithi173 ceph-mon[37000]: rocksdb: Options.log_readahead_size: 0 2024-05-27T07:15:16.084 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:15:15 smithi173 ceph-mon[37000]: rocksdb: Options.file_checksum_gen_factory: Unknown 2024-05-27T07:15:16.084 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:15:15 smithi173 ceph-mon[37000]: rocksdb: Options.best_efforts_recovery: 0 2024-05-27T07:15:16.084 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:15:15 smithi173 ceph-mon[37000]: rocksdb: Options.max_bgerror_resume_count: 2147483647 2024-05-27T07:15:16.084 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:15:15 smithi173 ceph-mon[37000]: rocksdb: Options.bgerror_resume_retry_interval: 1000000 2024-05-27T07:15:16.084 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:15:15 smithi173 ceph-mon[37000]: rocksdb: Options.allow_data_in_errors: 0 2024-05-27T07:15:16.084 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:15:15 smithi173 ceph-mon[37000]: rocksdb: Options.db_host_id: __hostname__ 2024-05-27T07:15:16.084 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:15:15 smithi173 ceph-mon[37000]: rocksdb: Options.enforce_single_del_contracts: true 2024-05-27T07:15:16.085 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:15:15 smithi173 ceph-mon[37000]: rocksdb: Options.max_background_jobs: 2 2024-05-27T07:15:16.085 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:15:15 smithi173 ceph-mon[37000]: rocksdb: Options.max_background_compactions: -1 2024-05-27T07:15:16.085 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:15:15 smithi173 ceph-mon[37000]: rocksdb: Options.max_subcompactions: 1 2024-05-27T07:15:16.085 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:15:15 smithi173 ceph-mon[37000]: rocksdb: Options.avoid_flush_during_shutdown: 0 2024-05-27T07:15:16.085 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:15:15 smithi173 ceph-mon[37000]: rocksdb: Options.writable_file_max_buffer_size: 1048576 2024-05-27T07:15:16.085 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:15:15 smithi173 ceph-mon[37000]: rocksdb: Options.delayed_write_rate : 16777216 2024-05-27T07:15:16.085 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:15:15 smithi173 ceph-mon[37000]: rocksdb: Options.max_total_wal_size: 0 2024-05-27T07:15:16.085 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:15:15 smithi173 ceph-mon[37000]: rocksdb: Options.delete_obsolete_files_period_micros: 21600000000 2024-05-27T07:15:16.085 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:15:15 smithi173 ceph-mon[37000]: rocksdb: Options.stats_dump_period_sec: 600 2024-05-27T07:15:16.085 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:15:15 smithi173 ceph-mon[37000]: rocksdb: Options.stats_persist_period_sec: 600 2024-05-27T07:15:16.085 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:15:15 smithi173 ceph-mon[37000]: rocksdb: Options.stats_history_buffer_size: 1048576 2024-05-27T07:15:16.085 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:15:15 smithi173 ceph-mon[37000]: rocksdb: Options.max_open_files: -1 2024-05-27T07:15:16.085 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:15:15 smithi173 ceph-mon[37000]: rocksdb: Options.bytes_per_sync: 0 2024-05-27T07:15:16.086 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:15:15 smithi173 ceph-mon[37000]: rocksdb: Options.wal_bytes_per_sync: 0 2024-05-27T07:15:16.086 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:15:15 smithi173 ceph-mon[37000]: rocksdb: Options.strict_bytes_per_sync: 0 2024-05-27T07:15:16.086 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:15:15 smithi173 ceph-mon[37000]: rocksdb: Options.compaction_readahead_size: 0 2024-05-27T07:15:16.086 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:15:15 smithi173 ceph-mon[37000]: rocksdb: Options.max_background_flushes: -1 2024-05-27T07:15:16.086 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:15:15 smithi173 ceph-mon[37000]: rocksdb: Compression algorithms supported: 2024-05-27T07:15:16.086 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:15:15 smithi173 ceph-mon[37000]: rocksdb: kZSTDNotFinalCompression supported: 0 2024-05-27T07:15:16.086 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:15:15 smithi173 ceph-mon[37000]: rocksdb: kZSTD supported: 0 2024-05-27T07:15:16.086 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:15:15 smithi173 ceph-mon[37000]: rocksdb: kXpressCompression supported: 0 2024-05-27T07:15:16.086 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:15:15 smithi173 ceph-mon[37000]: rocksdb: kLZ4HCCompression supported: 1 2024-05-27T07:15:16.086 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:15:15 smithi173 ceph-mon[37000]: rocksdb: kZlibCompression supported: 1 2024-05-27T07:15:16.086 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:15:15 smithi173 ceph-mon[37000]: rocksdb: kSnappyCompression supported: 1 2024-05-27T07:15:16.086 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:15:15 smithi173 ceph-mon[37000]: rocksdb: kLZ4Compression supported: 1 2024-05-27T07:15:16.086 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:15:15 smithi173 ceph-mon[37000]: rocksdb: kBZip2Compression supported: 0 2024-05-27T07:15:16.086 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:15:15 smithi173 ceph-mon[37000]: rocksdb: Fast CRC32 supported: Supported on x86 2024-05-27T07:15:16.087 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:15:15 smithi173 ceph-mon[37000]: rocksdb: DMutex implementation: pthread_mutex_t 2024-05-27T07:15:16.087 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:15:15 smithi173 ceph-mon[37000]: rocksdb: [db/version_set.cc:5527] Recovering from manifest file: /var/lib/ceph/mon/ceph-c/store.db/MANIFEST-000005 2024-05-27T07:15:16.087 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:15:15 smithi173 ceph-mon[37000]: rocksdb: [db/column_family.cc:630] --------------- Options for column family [default]: 2024-05-27T07:15:16.087 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:15:15 smithi173 ceph-mon[37000]: rocksdb: Options.comparator: leveldb.BytewiseComparator 2024-05-27T07:15:16.087 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:15:15 smithi173 ceph-mon[37000]: rocksdb: Options.merge_operator: 2024-05-27T07:15:16.087 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:15:15 smithi173 ceph-mon[37000]: rocksdb: Options.compaction_filter: None 2024-05-27T07:15:16.087 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:15:15 smithi173 ceph-mon[37000]: rocksdb: Options.compaction_filter_factory: None 2024-05-27T07:15:16.087 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:15:15 smithi173 ceph-mon[37000]: rocksdb: Options.sst_partitioner_factory: None 2024-05-27T07:15:16.087 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:15:15 smithi173 ceph-mon[37000]: rocksdb: Options.memtable_factory: SkipListFactory 2024-05-27T07:15:16.087 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:15:15 smithi173 ceph-mon[37000]: rocksdb: Options.table_factory: BlockBasedTable 2024-05-27T07:15:16.087 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:15:15 smithi173 ceph-mon[37000]: rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x55db9d0181c0) 2024-05-27T07:15:16.087 INFO:journalctl@ceph.mon.c.smithi173.stdout: cache_index_and_filter_blocks: 1 2024-05-27T07:15:16.088 INFO:journalctl@ceph.mon.c.smithi173.stdout: cache_index_and_filter_blocks_with_high_priority: 0 2024-05-27T07:15:16.088 INFO:journalctl@ceph.mon.c.smithi173.stdout: pin_l0_filter_and_index_blocks_in_cache: 0 2024-05-27T07:15:16.088 INFO:journalctl@ceph.mon.c.smithi173.stdout: pin_top_level_index_and_filter: 1 2024-05-27T07:15:16.088 INFO:journalctl@ceph.mon.c.smithi173.stdout: index_type: 0 2024-05-27T07:15:16.088 INFO:journalctl@ceph.mon.c.smithi173.stdout: data_block_index_type: 0 2024-05-27T07:15:16.088 INFO:journalctl@ceph.mon.c.smithi173.stdout: index_shortening: 1 2024-05-27T07:15:16.088 INFO:journalctl@ceph.mon.c.smithi173.stdout: data_block_hash_table_util_ratio: 0.750000 2024-05-27T07:15:16.088 INFO:journalctl@ceph.mon.c.smithi173.stdout: checksum: 4 2024-05-27T07:15:16.088 INFO:journalctl@ceph.mon.c.smithi173.stdout: no_block_cache: 0 2024-05-27T07:15:16.088 INFO:journalctl@ceph.mon.c.smithi173.stdout: block_cache: 0x55db9cf151f0 2024-05-27T07:15:16.088 INFO:journalctl@ceph.mon.c.smithi173.stdout: block_cache_name: BinnedLRUCache 2024-05-27T07:15:16.088 INFO:journalctl@ceph.mon.c.smithi173.stdout: block_cache_options: 2024-05-27T07:15:16.088 INFO:journalctl@ceph.mon.c.smithi173.stdout: capacity : 536870912 2024-05-27T07:15:16.089 INFO:journalctl@ceph.mon.c.smithi173.stdout: num_shard_bits : 4 2024-05-27T07:15:16.089 INFO:journalctl@ceph.mon.c.smithi173.stdout: strict_capacity_limit : 0 2024-05-27T07:15:16.089 INFO:journalctl@ceph.mon.c.smithi173.stdout: high_pri_pool_ratio: 0.000 2024-05-27T07:15:16.089 INFO:journalctl@ceph.mon.c.smithi173.stdout: block_cache_compressed: (nil) 2024-05-27T07:15:16.089 INFO:journalctl@ceph.mon.c.smithi173.stdout: persistent_cache: (nil) 2024-05-27T07:15:16.089 INFO:journalctl@ceph.mon.c.smithi173.stdout: block_size: 4096 2024-05-27T07:15:16.089 INFO:journalctl@ceph.mon.c.smithi173.stdout: block_size_deviation: 10 2024-05-27T07:15:16.089 INFO:journalctl@ceph.mon.c.smithi173.stdout: block_restart_interval: 16 2024-05-27T07:15:16.089 INFO:journalctl@ceph.mon.c.smithi173.stdout: index_block_restart_interval: 1 2024-05-27T07:15:16.089 INFO:journalctl@ceph.mon.c.smithi173.stdout: metadata_block_size: 4096 2024-05-27T07:15:16.089 INFO:journalctl@ceph.mon.c.smithi173.stdout: partition_filters: 0 2024-05-27T07:15:16.089 INFO:journalctl@ceph.mon.c.smithi173.stdout: use_delta_encoding: 1 2024-05-27T07:15:16.090 INFO:journalctl@ceph.mon.c.smithi173.stdout: filter_policy: bloomfilter 2024-05-27T07:15:16.090 INFO:journalctl@ceph.mon.c.smithi173.stdout: whole_key_filtering: 1 2024-05-27T07:15:16.090 INFO:journalctl@ceph.mon.c.smithi173.stdout: verify_compression: 0 2024-05-27T07:15:16.090 INFO:journalctl@ceph.mon.c.smithi173.stdout: read_amp_bytes_per_bit: 0 2024-05-27T07:15:16.090 INFO:journalctl@ceph.mon.c.smithi173.stdout: format_version: 5 2024-05-27T07:15:16.090 INFO:journalctl@ceph.mon.c.smithi173.stdout: enable_index_compression: 1 2024-05-27T07:15:16.090 INFO:journalctl@ceph.mon.c.smithi173.stdout: block_align: 0 2024-05-27T07:15:16.090 INFO:journalctl@ceph.mon.c.smithi173.stdout: max_auto_readahead_size: 262144 2024-05-27T07:15:16.090 INFO:journalctl@ceph.mon.c.smithi173.stdout: prepopulate_block_cache: 0 2024-05-27T07:15:16.090 INFO:journalctl@ceph.mon.c.smithi173.stdout: initial_auto_readahead_size: 8192 2024-05-27T07:15:16.090 INFO:journalctl@ceph.mon.c.smithi173.stdout: num_file_reads_for_auto_readahead: 2 2024-05-27T07:15:16.091 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:15:15 smithi173 ceph-mon[37000]: rocksdb: Options.write_buffer_size: 33554432 2024-05-27T07:15:16.091 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:15:15 smithi173 ceph-mon[37000]: rocksdb: Options.max_write_buffer_number: 2 2024-05-27T07:15:16.091 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:15:15 smithi173 ceph-mon[37000]: rocksdb: Options.compression: NoCompression 2024-05-27T07:15:16.091 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:15:15 smithi173 ceph-mon[37000]: rocksdb: Options.bottommost_compression: Disabled 2024-05-27T07:15:16.091 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:15:15 smithi173 ceph-mon[37000]: rocksdb: Options.prefix_extractor: nullptr 2024-05-27T07:15:16.091 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:15:15 smithi173 ceph-mon[37000]: rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr 2024-05-27T07:15:16.091 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:15:15 smithi173 ceph-mon[37000]: rocksdb: Options.num_levels: 7 2024-05-27T07:15:16.091 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:15:15 smithi173 ceph-mon[37000]: rocksdb: Options.min_write_buffer_number_to_merge: 1 2024-05-27T07:15:16.091 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:15:15 smithi173 ceph-mon[37000]: rocksdb: Options.max_write_buffer_number_to_maintain: 0 2024-05-27T07:15:16.091 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:15:15 smithi173 ceph-mon[37000]: rocksdb: Options.max_write_buffer_size_to_maintain: 0 2024-05-27T07:15:16.091 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:15:15 smithi173 ceph-mon[37000]: rocksdb: Options.bottommost_compression_opts.window_bits: -14 2024-05-27T07:15:16.091 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:15:15 smithi173 ceph-mon[37000]: rocksdb: Options.bottommost_compression_opts.level: 32767 2024-05-27T07:15:16.091 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:15:15 smithi173 ceph-mon[37000]: rocksdb: Options.bottommost_compression_opts.strategy: 0 2024-05-27T07:15:16.092 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:15:15 smithi173 ceph-mon[37000]: rocksdb: Options.bottommost_compression_opts.max_dict_bytes: 0 2024-05-27T07:15:16.092 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:15:15 smithi173 ceph-mon[37000]: rocksdb: Options.bottommost_compression_opts.zstd_max_train_bytes: 0 2024-05-27T07:15:16.092 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:15:15 smithi173 ceph-mon[37000]: rocksdb: Options.bottommost_compression_opts.parallel_threads: 1 2024-05-27T07:15:16.092 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:15:15 smithi173 ceph-mon[37000]: rocksdb: Options.bottommost_compression_opts.enabled: false 2024-05-27T07:15:16.092 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:15:15 smithi173 ceph-mon[37000]: rocksdb: Options.bottommost_compression_opts.max_dict_buffer_bytes: 0 2024-05-27T07:15:16.092 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:15:15 smithi173 ceph-mon[37000]: rocksdb: Options.bottommost_compression_opts.use_zstd_dict_trainer: true 2024-05-27T07:15:16.092 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:15:15 smithi173 ceph-mon[37000]: rocksdb: Options.compression_opts.window_bits: -14 2024-05-27T07:15:16.092 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:15:15 smithi173 ceph-mon[37000]: rocksdb: Options.compression_opts.level: 32767 2024-05-27T07:15:16.092 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:15:15 smithi173 ceph-mon[37000]: rocksdb: Options.compression_opts.strategy: 0 2024-05-27T07:15:16.092 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:15:15 smithi173 ceph-mon[37000]: rocksdb: Options.compression_opts.max_dict_bytes: 0 2024-05-27T07:15:16.092 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:15:15 smithi173 ceph-mon[37000]: rocksdb: Options.compression_opts.zstd_max_train_bytes: 0 2024-05-27T07:15:16.092 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:15:15 smithi173 ceph-mon[37000]: rocksdb: Options.compression_opts.use_zstd_dict_trainer: true 2024-05-27T07:15:16.092 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:15:15 smithi173 ceph-mon[37000]: rocksdb: Options.compression_opts.parallel_threads: 1 2024-05-27T07:15:16.093 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:15:15 smithi173 ceph-mon[37000]: rocksdb: Options.compression_opts.enabled: false 2024-05-27T07:15:16.093 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:15:15 smithi173 ceph-mon[37000]: rocksdb: Options.compression_opts.max_dict_buffer_bytes: 0 2024-05-27T07:15:16.093 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:15:15 smithi173 ceph-mon[37000]: rocksdb: Options.level0_file_num_compaction_trigger: 4 2024-05-27T07:15:16.093 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:15:15 smithi173 ceph-mon[37000]: rocksdb: Options.level0_slowdown_writes_trigger: 20 2024-05-27T07:15:16.093 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:15:15 smithi173 ceph-mon[37000]: rocksdb: Options.level0_stop_writes_trigger: 36 2024-05-27T07:15:16.093 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:15:15 smithi173 ceph-mon[37000]: rocksdb: Options.target_file_size_base: 67108864 2024-05-27T07:15:16.093 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:15:15 smithi173 ceph-mon[37000]: rocksdb: Options.target_file_size_multiplier: 1 2024-05-27T07:15:16.093 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:15:15 smithi173 ceph-mon[37000]: rocksdb: Options.max_bytes_for_level_base: 268435456 2024-05-27T07:15:16.093 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:15:15 smithi173 ceph-mon[37000]: rocksdb: Options.level_compaction_dynamic_level_bytes: 1 2024-05-27T07:15:16.093 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:15:15 smithi173 ceph-mon[37000]: rocksdb: Options.max_bytes_for_level_multiplier: 10.000000 2024-05-27T07:15:16.093 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:15:15 smithi173 ceph-mon[37000]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1 2024-05-27T07:15:16.093 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:15:15 smithi173 ceph-mon[37000]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1 2024-05-27T07:15:16.093 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:15:15 smithi173 ceph-mon[37000]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1 2024-05-27T07:15:16.093 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:15:15 smithi173 ceph-mon[37000]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1 2024-05-27T07:15:16.094 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:15:15 smithi173 ceph-mon[37000]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1 2024-05-27T07:15:16.094 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:15:15 smithi173 ceph-mon[37000]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1 2024-05-27T07:15:16.094 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:15:15 smithi173 ceph-mon[37000]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1 2024-05-27T07:15:16.094 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:15:15 smithi173 ceph-mon[37000]: rocksdb: Options.max_sequential_skip_in_iterations: 8 2024-05-27T07:15:16.094 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:15:15 smithi173 ceph-mon[37000]: rocksdb: Options.max_compaction_bytes: 1677721600 2024-05-27T07:15:16.094 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:15:15 smithi173 ceph-mon[37000]: rocksdb: Options.ignore_max_compaction_bytes_for_input: true 2024-05-27T07:15:16.094 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:15:15 smithi173 ceph-mon[37000]: rocksdb: Options.arena_block_size: 1048576 2024-05-27T07:15:16.094 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:15:15 smithi173 ceph-mon[37000]: rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736 2024-05-27T07:15:16.094 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:15:15 smithi173 ceph-mon[37000]: rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944 2024-05-27T07:15:16.094 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:15:15 smithi173 ceph-mon[37000]: rocksdb: Options.disable_auto_compactions: 0 2024-05-27T07:15:16.094 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:15:15 smithi173 ceph-mon[37000]: rocksdb: Options.compaction_style: kCompactionStyleLevel 2024-05-27T07:15:16.094 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:15:15 smithi173 ceph-mon[37000]: rocksdb: Options.compaction_pri: kMinOverlappingRatio 2024-05-27T07:15:16.094 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:15:15 smithi173 ceph-mon[37000]: rocksdb: Options.compaction_options_universal.size_ratio: 1 2024-05-27T07:15:16.095 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:15:15 smithi173 ceph-mon[37000]: rocksdb: Options.compaction_options_universal.min_merge_width: 2 2024-05-27T07:15:16.095 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:15:15 smithi173 ceph-mon[37000]: rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295 2024-05-27T07:15:16.095 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:15:15 smithi173 ceph-mon[37000]: rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200 2024-05-27T07:15:16.095 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:15:15 smithi173 ceph-mon[37000]: rocksdb: Options.compaction_options_universal.compression_size_percent: -1 2024-05-27T07:15:16.095 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:15:15 smithi173 ceph-mon[37000]: rocksdb: Options.compaction_options_universal.stop_style: kCompactionStopStyleTotalSize 2024-05-27T07:15:16.095 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:15:15 smithi173 ceph-mon[37000]: rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824 2024-05-27T07:15:16.095 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:15:15 smithi173 ceph-mon[37000]: rocksdb: Options.compaction_options_fifo.allow_compaction: 0 2024-05-27T07:15:16.095 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:15:15 smithi173 ceph-mon[37000]: rocksdb: Options.table_properties_collectors: CompactOnDeletionCollector (Sliding window size = 32768 Deletion trigger = 16384 Deletion ratio = 0); 2024-05-27T07:15:16.095 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:15:15 smithi173 ceph-mon[37000]: rocksdb: Options.inplace_update_support: 0 2024-05-27T07:15:16.095 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:15:15 smithi173 ceph-mon[37000]: rocksdb: Options.inplace_update_num_locks: 10000 2024-05-27T07:15:16.095 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:15:15 smithi173 ceph-mon[37000]: rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000 2024-05-27T07:15:16.095 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:15:15 smithi173 ceph-mon[37000]: rocksdb: Options.memtable_whole_key_filtering: 0 2024-05-27T07:15:16.095 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:15:15 smithi173 ceph-mon[37000]: rocksdb: Options.memtable_huge_page_size: 0 2024-05-27T07:15:16.095 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:15:15 smithi173 ceph-mon[37000]: rocksdb: Options.bloom_locality: 0 2024-05-27T07:15:16.096 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:15:15 smithi173 ceph-mon[37000]: rocksdb: Options.max_successive_merges: 0 2024-05-27T07:15:16.096 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:15:15 smithi173 ceph-mon[37000]: rocksdb: Options.optimize_filters_for_hits: 0 2024-05-27T07:15:16.096 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:15:15 smithi173 ceph-mon[37000]: rocksdb: Options.paranoid_file_checks: 0 2024-05-27T07:15:16.096 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:15:15 smithi173 ceph-mon[37000]: rocksdb: Options.force_consistency_checks: 1 2024-05-27T07:15:16.096 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:15:15 smithi173 ceph-mon[37000]: rocksdb: Options.report_bg_io_stats: 0 2024-05-27T07:15:16.096 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:15:15 smithi173 ceph-mon[37000]: rocksdb: Options.ttl: 2592000 2024-05-27T07:15:16.096 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:15:15 smithi173 ceph-mon[37000]: rocksdb: Options.periodic_compaction_seconds: 0 2024-05-27T07:15:16.096 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:15:15 smithi173 ceph-mon[37000]: rocksdb: Options.preclude_last_level_data_seconds: 0 2024-05-27T07:15:16.096 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:15:15 smithi173 ceph-mon[37000]: rocksdb: Options.preserve_internal_time_seconds: 0 2024-05-27T07:15:16.096 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:15:15 smithi173 ceph-mon[37000]: rocksdb: Options.enable_blob_files: false 2024-05-27T07:15:16.096 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:15:15 smithi173 ceph-mon[37000]: rocksdb: Options.min_blob_size: 0 2024-05-27T07:15:16.096 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:15:15 smithi173 ceph-mon[37000]: rocksdb: Options.blob_file_size: 268435456 2024-05-27T07:15:16.096 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:15:15 smithi173 ceph-mon[37000]: rocksdb: Options.blob_compression_type: NoCompression 2024-05-27T07:15:16.097 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:15:15 smithi173 ceph-mon[37000]: rocksdb: Options.enable_blob_garbage_collection: false 2024-05-27T07:15:16.097 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:15:15 smithi173 ceph-mon[37000]: rocksdb: Options.blob_garbage_collection_age_cutoff: 0.250000 2024-05-27T07:15:16.097 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:15:15 smithi173 ceph-mon[37000]: rocksdb: Options.blob_garbage_collection_force_threshold: 1.000000 2024-05-27T07:15:16.097 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:15:15 smithi173 ceph-mon[37000]: rocksdb: Options.blob_compaction_readahead_size: 0 2024-05-27T07:15:16.097 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:15:15 smithi173 ceph-mon[37000]: rocksdb: Options.blob_file_starting_level: 0 2024-05-27T07:15:16.097 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:15:15 smithi173 ceph-mon[37000]: rocksdb: Options.experimental_mempurge_threshold: 0.000000 2024-05-27T07:15:16.097 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:15:15 smithi173 ceph-mon[37000]: rocksdb: [db/version_set.cc:5566] Recovered from manifest file:/var/lib/ceph/mon/ceph-c/store.db/MANIFEST-000005 succeeded,manifest_file_number is 5, next_file_number is 7, last_sequence is 0, log_number is 0,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 0 2024-05-27T07:15:16.097 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:15:15 smithi173 ceph-mon[37000]: rocksdb: [db/version_set.cc:5581] Column family [default] (ID 0), log number is 0 2024-05-27T07:15:16.097 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:15:15 smithi173 ceph-mon[37000]: rocksdb: [db/db_impl/db_impl_open.cc:539] DB ID: 625bf53d-8885-4142-869c-57ae49e585db 2024-05-27T07:15:16.097 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:15:15 smithi173 ceph-mon[37000]: rocksdb: EVENT_LOG_v1 {"time_micros": 1716794115783014, "job": 1, "event": "recovery_started", "wal_files": [4]} 2024-05-27T07:15:16.097 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:15:15 smithi173 ceph-mon[37000]: rocksdb: [db/db_impl/db_impl_open.cc:1043] Recovering log #4 mode 2 2024-05-27T07:15:16.097 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:15:15 smithi173 ceph-mon[37000]: rocksdb: EVENT_LOG_v1 {"time_micros": 1716794115783434, "cf_name": "default", "job": 1, "event": "table_file_creation", "file_number": 8, "file_size": 1648, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 1, "largest_seqno": 5, "table_properties": {"data_size": 523, "index_size": 31, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 1, "index_value_is_delta_encoded": 1, "filter_size": 69, "raw_key_size": 115, "raw_average_key_size": 23, "raw_value_size": 401, "raw_average_value_size": 80, "num_data_blocks": 1, "num_entries": 5, "num_filter_entries": 5, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 0, "filter_policy": "bloomfilter", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "", "prefix_extractor_name": "nullptr", "property_collectors": "[CompactOnDeletionCollector]", "compression": "NoCompression", "compression_options": "window_bits=-14; level=32767; strategy=0; max_dict_bytes=0; zstd_max_train_bytes=0; enabled=0; max_dict_buffer_bytes=0; use_zstd_dict_trainer=1; ", "creation_time": 1716794115, "oldest_key_time": 0, "file_creation_time": 0, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "625bf53d-8885-4142-869c-57ae49e585db", "db_session_id": "NCW1JYWGA3RXP6ZCA6C3", "orig_file_number": 8, "seqno_to_time_mapping": "N/A"}} 2024-05-27T07:15:16.097 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:15:15 smithi173 ceph-mon[37000]: rocksdb: EVENT_LOG_v1 {"time_micros": 1716794115783514, "job": 1, "event": "recovery_finished"} 2024-05-27T07:15:16.097 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:15:15 smithi173 ceph-mon[37000]: rocksdb: [db/version_set.cc:5047] Creating manifest 10 2024-05-27T07:15:16.098 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:15:15 smithi173 ceph-mon[37000]: rocksdb: [file/delete_scheduler.cc:74] Deleted file /var/lib/ceph/mon/ceph-c/store.db/000004.log immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024-05-27T07:15:16.098 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:15:15 smithi173 ceph-mon[37000]: rocksdb: [db/db_impl/db_impl_open.cc:1987] SstFileManager instance 0x55db9cef9c00 2024-05-27T07:15:16.098 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:15:15 smithi173 ceph-mon[37000]: rocksdb: DB pointer 0x55dba1cec000 2024-05-27T07:15:16.098 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:15:15 smithi173 ceph-mon[37000]: mon.c does not exist in monmap, will attempt to join an existing cluster 2024-05-27T07:15:16.098 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:15:15 smithi173 ceph-mon[37000]: rocksdb: [db/db_impl/db_impl.cc:1109] ------- DUMPING STATS ------- 2024-05-27T07:15:16.098 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:15:15 smithi173 ceph-mon[37000]: rocksdb: [db/db_impl/db_impl.cc:1111] 2024-05-27T07:15:16.098 INFO:journalctl@ceph.mon.c.smithi173.stdout: ** DB Stats ** 2024-05-27T07:15:16.098 INFO:journalctl@ceph.mon.c.smithi173.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-05-27T07:15:16.098 INFO:journalctl@ceph.mon.c.smithi173.stdout: Cumulative writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 GB, 0.00 MB/s 2024-05-27T07:15:16.098 INFO:journalctl@ceph.mon.c.smithi173.stdout: Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-05-27T07:15:16.098 INFO:journalctl@ceph.mon.c.smithi173.stdout: Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent 2024-05-27T07:15:16.099 INFO:journalctl@ceph.mon.c.smithi173.stdout: Interval writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 MB, 0.00 MB/s 2024-05-27T07:15:16.099 INFO:journalctl@ceph.mon.c.smithi173.stdout: Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-05-27T07:15:16.099 INFO:journalctl@ceph.mon.c.smithi173.stdout: Interval stall: 00:00:0.000 H:M:S, 0.0 percent 2024-05-27T07:15:16.099 INFO:journalctl@ceph.mon.c.smithi173.stdout: 2024-05-27T07:15:16.099 INFO:journalctl@ceph.mon.c.smithi173.stdout: ** Compaction Stats [default] ** 2024-05-27T07:15:16.099 INFO:journalctl@ceph.mon.c.smithi173.stdout: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-05-27T07:15:16.099 INFO:journalctl@ceph.mon.c.smithi173.stdout: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ 2024-05-27T07:15:16.099 INFO:journalctl@ceph.mon.c.smithi173.stdout: L0 1/0 1.61 KB 0.2 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 3.9 0.00 0.00 1 0.000 0 0 0.0 0.0 2024-05-27T07:15:16.099 INFO:journalctl@ceph.mon.c.smithi173.stdout: Sum 1/0 1.61 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 3.9 0.00 0.00 1 0.000 0 0 0.0 0.0 2024-05-27T07:15:16.099 INFO:journalctl@ceph.mon.c.smithi173.stdout: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 3.9 0.00 0.00 1 0.000 0 0 0.0 0.0 2024-05-27T07:15:16.099 INFO:journalctl@ceph.mon.c.smithi173.stdout: 2024-05-27T07:15:16.099 INFO:journalctl@ceph.mon.c.smithi173.stdout: ** Compaction Stats [default] ** 2024-05-27T07:15:16.099 INFO:journalctl@ceph.mon.c.smithi173.stdout: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-05-27T07:15:16.099 INFO:journalctl@ceph.mon.c.smithi173.stdout: --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-05-27T07:15:16.099 INFO:journalctl@ceph.mon.c.smithi173.stdout: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 3.9 0.00 0.00 1 0.000 0 0 0.0 0.0 2024-05-27T07:15:16.100 INFO:journalctl@ceph.mon.c.smithi173.stdout: 2024-05-27T07:15:16.100 INFO:journalctl@ceph.mon.c.smithi173.stdout: Blob file count: 0, total size: 0.0 GB, garbage size: 0.0 GB, space amp: 0.0 2024-05-27T07:15:16.100 INFO:journalctl@ceph.mon.c.smithi173.stdout: 2024-05-27T07:15:16.100 INFO:journalctl@ceph.mon.c.smithi173.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-05-27T07:15:16.100 INFO:journalctl@ceph.mon.c.smithi173.stdout: Flush(GB): cumulative 0.000, interval 0.000 2024-05-27T07:15:16.100 INFO:journalctl@ceph.mon.c.smithi173.stdout: AddFile(GB): cumulative 0.000, interval 0.000 2024-05-27T07:15:16.100 INFO:journalctl@ceph.mon.c.smithi173.stdout: AddFile(Total Files): cumulative 0, interval 0 2024-05-27T07:15:16.100 INFO:journalctl@ceph.mon.c.smithi173.stdout: AddFile(L0 Files): cumulative 0, interval 0 2024-05-27T07:15:16.100 INFO:journalctl@ceph.mon.c.smithi173.stdout: AddFile(Keys): cumulative 0, interval 0 2024-05-27T07:15:16.100 INFO:journalctl@ceph.mon.c.smithi173.stdout: Cumulative compaction: 0.00 GB write, 0.12 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-05-27T07:15:16.100 INFO:journalctl@ceph.mon.c.smithi173.stdout: Interval compaction: 0.00 GB write, 0.12 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-05-27T07:15:16.100 INFO:journalctl@ceph.mon.c.smithi173.stdout: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2024-05-27T07:15:16.100 INFO:journalctl@ceph.mon.c.smithi173.stdout: Block cache BinnedLRUCache@0x55db9cf151f0#2 capacity: 512.00 MB usage: 0.23 KB table_size: 0 occupancy: 18446744073709551615 collections: 1 last_copies: 0 last_secs: 1e-05 secs_since: 0 2024-05-27T07:15:16.101 INFO:journalctl@ceph.mon.c.smithi173.stdout: Block cache entry stats(count,size,portion): FilterBlock(1,0.12 KB,2.23517e-05%) IndexBlock(1,0.11 KB,2.08616e-05%) Misc(1,0.00 KB,0%) 2024-05-27T07:15:16.101 INFO:journalctl@ceph.mon.c.smithi173.stdout: 2024-05-27T07:15:16.101 INFO:journalctl@ceph.mon.c.smithi173.stdout: ** File Read Latency Histogram By Level [default] ** 2024-05-27T07:15:16.101 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:15:15 smithi173 ceph-mon[37000]: using public_addr v2:172.21.15.173:0/0 -> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] 2024-05-27T07:15:16.101 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:15:15 smithi173 ceph-mon[37000]: starting mon.c rank -1 at public addrs [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] at bind addrs [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] mon_data /var/lib/ceph/mon/ceph-c fsid 73df2636-1bf8-11ef-bc9b-c7b262605968 2024-05-27T07:15:16.101 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:15:15 smithi173 ceph-mon[37000]: mon.c@-1(???) e0 preinit fsid 73df2636-1bf8-11ef-bc9b-c7b262605968 2024-05-27T07:15:16.101 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:15:15 smithi173 ceph-mon[37000]: mon.c@-1(synchronizing).mds e1 new map 2024-05-27T07:15:16.101 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:15:15 smithi173 ceph-mon[37000]: mon.c@-1(synchronizing).mds e1 print_map 2024-05-27T07:15:16.101 INFO:journalctl@ceph.mon.c.smithi173.stdout: e1 2024-05-27T07:15:16.101 INFO:journalctl@ceph.mon.c.smithi173.stdout: enable_multiple, ever_enabled_multiple: 1,1 2024-05-27T07:15:16.101 INFO:journalctl@ceph.mon.c.smithi173.stdout: default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2,11=minor log segments,12=quiesce subvolumes} 2024-05-27T07:15:16.101 INFO:journalctl@ceph.mon.c.smithi173.stdout: legacy client fscid: -1 2024-05-27T07:15:16.101 INFO:journalctl@ceph.mon.c.smithi173.stdout: 2024-05-27T07:15:16.102 INFO:journalctl@ceph.mon.c.smithi173.stdout: No filesystems configured 2024-05-27T07:15:16.102 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:15:15 smithi173 ceph-mon[37000]: mon.c@-1(synchronizing).osd e0 _set_cache_ratios kv ratio 0.25 inc ratio 0.375 full ratio 0.375 2024-05-27T07:15:16.102 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:15:15 smithi173 ceph-mon[37000]: mon.c@-1(synchronizing).osd e0 register_cache_with_pcm pcm target: 2147483648 pcm max: 1020054732 pcm min: 134217728 inc_osd_cache size: 1 2024-05-27T07:15:16.102 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:15:15 smithi173 ceph-mon[37000]: mon.c@-1(synchronizing).osd e1 e1: 0 total, 0 up, 0 in 2024-05-27T07:15:16.102 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:15:15 smithi173 ceph-mon[37000]: mon.c@-1(synchronizing).osd e2 e2: 0 total, 0 up, 0 in 2024-05-27T07:15:16.102 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:15:15 smithi173 ceph-mon[37000]: mon.c@-1(synchronizing).osd e3 e3: 0 total, 0 up, 0 in 2024-05-27T07:15:16.102 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:15:15 smithi173 ceph-mon[37000]: mon.c@-1(synchronizing).osd e4 e4: 0 total, 0 up, 0 in 2024-05-27T07:15:16.102 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:15:15 smithi173 ceph-mon[37000]: mon.c@-1(synchronizing).osd e4 crush map has features 3314932999778484224, adjusting msgr requires 2024-05-27T07:15:16.102 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:15:15 smithi173 ceph-mon[37000]: mon.c@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2024-05-27T07:15:16.102 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:15:15 smithi173 ceph-mon[37000]: mon.c@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2024-05-27T07:15:16.102 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:15:15 smithi173 ceph-mon[37000]: mon.c@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2024-05-27T07:15:16.102 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:15:15 smithi173 ceph-mon[37000]: pgmap v4: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-27T07:15:16.102 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:15:15 smithi173 ceph-mon[37000]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' 2024-05-27T07:15:16.103 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:15:15 smithi173 ceph-mon[37000]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-27T07:15:16.103 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:15:15 smithi173 ceph-mon[37000]: Added host smithi105 2024-05-27T07:15:16.103 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:15:15 smithi173 ceph-mon[37000]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' 2024-05-27T07:15:16.103 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:15:15 smithi173 ceph-mon[37000]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' 2024-05-27T07:15:16.103 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:15:15 smithi173 ceph-mon[37000]: pgmap v5: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-27T07:15:16.103 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:15:15 smithi173 ceph-mon[37000]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' 2024-05-27T07:15:16.103 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:15:15 smithi173 ceph-mon[37000]: from='client.14180 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-05-27T07:15:16.103 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:15:15 smithi173 ceph-mon[37000]: pgmap v6: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-27T07:15:16.103 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:15:15 smithi173 ceph-mon[37000]: pgmap v7: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-27T07:15:16.103 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:15:15 smithi173 ceph-mon[37000]: from='client.14182 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi173", "target": ["mon-mgr", ""]}]: dispatch 2024-05-27T07:15:16.103 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:15:15 smithi173 ceph-mon[37000]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' 2024-05-27T07:15:16.103 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:15:15 smithi173 ceph-mon[37000]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' 2024-05-27T07:15:16.103 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:15:15 smithi173 bash[36981]: de9961ba152193f89fc37e84f5f5e9e9c83b488327cbf64a307dcc95c6bd88d9 2024-05-27T07:15:16.104 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:15:15 smithi173 ceph-mon[37000]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' 2024-05-27T07:15:16.104 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:15:15 smithi173 ceph-mon[37000]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' 2024-05-27T07:15:16.104 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:15:15 smithi173 ceph-mon[37000]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd/host:smithi105", "name": "osd_memory_target"}]: dispatch 2024-05-27T07:15:16.104 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:15:15 smithi173 ceph-mon[37000]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T07:15:16.104 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:15:15 smithi173 ceph-mon[37000]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-27T07:15:16.104 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:15:15 smithi173 ceph-mon[37000]: Updating smithi105:/etc/ceph/ceph.conf 2024-05-27T07:15:16.104 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:15:15 smithi173 ceph-mon[37000]: Updating smithi105:/var/lib/ceph/73df2636-1bf8-11ef-bc9b-c7b262605968/config/ceph.conf 2024-05-27T07:15:16.104 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:15:15 smithi173 ceph-mon[37000]: Deploying cephadm binary to smithi173 2024-05-27T07:15:16.104 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:15:15 smithi173 ceph-mon[37000]: Updating smithi105:/etc/ceph/ceph.client.admin.keyring 2024-05-27T07:15:16.104 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:15:15 smithi173 ceph-mon[37000]: Updating smithi105:/var/lib/ceph/73df2636-1bf8-11ef-bc9b-c7b262605968/config/ceph.client.admin.keyring 2024-05-27T07:15:16.104 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:15:15 smithi173 ceph-mon[37000]: pgmap v8: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-27T07:15:16.104 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:15:15 smithi173 ceph-mon[37000]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' 2024-05-27T07:15:16.104 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:15:15 smithi173 ceph-mon[37000]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' 2024-05-27T07:15:16.104 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:15:15 smithi173 ceph-mon[37000]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' 2024-05-27T07:15:16.104 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:15:15 smithi173 ceph-mon[37000]: pgmap v9: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-27T07:15:16.104 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:15:15 smithi173 ceph-mon[37000]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' 2024-05-27T07:15:16.105 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:15:15 smithi173 ceph-mon[37000]: Added host smithi173 2024-05-27T07:15:16.105 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:15:15 smithi173 ceph-mon[37000]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-27T07:15:16.105 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:15:15 smithi173 ceph-mon[37000]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' 2024-05-27T07:15:16.105 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:15:15 smithi173 ceph-mon[37000]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' 2024-05-27T07:15:16.105 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:15:15 smithi173 ceph-mon[37000]: pgmap v10: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-27T07:15:16.105 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:15:15 smithi173 ceph-mon[37000]: from='client.14184 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-05-27T07:15:16.105 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:15:15 smithi173 ceph-mon[37000]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' 2024-05-27T07:15:16.105 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:15:15 smithi173 ceph-mon[37000]: pgmap v11: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-27T07:15:16.105 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:15:15 smithi173 ceph-mon[37000]: from='client.? 172.21.15.52:0/1478112303' entity='client.admin' cmd=[{"prefix": "osd crush tunables", "profile": "default"}]: dispatch 2024-05-27T07:15:16.105 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:15:15 smithi173 ceph-mon[37000]: pgmap v12: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-27T07:15:16.105 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:15:15 smithi173 ceph-mon[37000]: from='client.? 172.21.15.52:0/1478112303' entity='client.admin' cmd='[{"prefix": "osd crush tunables", "profile": "default"}]': finished 2024-05-27T07:15:16.105 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:15:15 smithi173 ceph-mon[37000]: osdmap e4: 0 total, 0 up, 0 in 2024-05-27T07:15:16.105 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:15:15 smithi173 ceph-mon[37000]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' 2024-05-27T07:15:16.105 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:15:15 smithi173 ceph-mon[37000]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' 2024-05-27T07:15:16.105 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:15:15 smithi173 ceph-mon[37000]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' 2024-05-27T07:15:16.106 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:15:15 smithi173 ceph-mon[37000]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' 2024-05-27T07:15:16.106 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:15:15 smithi173 ceph-mon[37000]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd/host:smithi173", "name": "osd_memory_target"}]: dispatch 2024-05-27T07:15:16.106 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:15:15 smithi173 ceph-mon[37000]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T07:15:16.106 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:15:15 smithi173 ceph-mon[37000]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-27T07:15:16.106 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:15:15 smithi173 ceph-mon[37000]: Updating smithi173:/etc/ceph/ceph.conf 2024-05-27T07:15:16.106 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:15:15 smithi173 ceph-mon[37000]: Updating smithi173:/var/lib/ceph/73df2636-1bf8-11ef-bc9b-c7b262605968/config/ceph.conf 2024-05-27T07:15:16.106 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:15:15 smithi173 ceph-mon[37000]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' 2024-05-27T07:15:16.106 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:15:15 smithi173 ceph-mon[37000]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' 2024-05-27T07:15:16.106 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:15:15 smithi173 ceph-mon[37000]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' 2024-05-27T07:15:16.106 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:15:15 smithi173 ceph-mon[37000]: Updating smithi173:/etc/ceph/ceph.client.admin.keyring 2024-05-27T07:15:16.106 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:15:15 smithi173 ceph-mon[37000]: Updating smithi173:/var/lib/ceph/73df2636-1bf8-11ef-bc9b-c7b262605968/config/ceph.client.admin.keyring 2024-05-27T07:15:16.106 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:15:15 smithi173 ceph-mon[37000]: pgmap v14: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-27T07:15:16.106 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:15:15 smithi173 ceph-mon[37000]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' 2024-05-27T07:15:16.107 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:15:15 smithi173 ceph-mon[37000]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-27T07:15:16.107 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:15:15 smithi173 ceph-mon[37000]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T07:15:16.107 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:15:15 smithi173 ceph-mon[37000]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-27T07:15:16.107 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:15:15 smithi173 ceph-mon[37000]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' 2024-05-27T07:15:16.107 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:15:15 smithi173 ceph-mon[37000]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-05-27T07:15:16.107 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:15:15 smithi173 ceph-mon[37000]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T07:15:16.107 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:15:15 smithi173 ceph-mon[37000]: from='client.14188 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "placement": "3;smithi052:172.21.15.52=a;smithi105:172.21.15.105=b;smithi173:172.21.15.173=c", "target": ["mon-mgr", ""]}]: dispatch 2024-05-27T07:15:16.107 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:15:15 smithi173 ceph-mon[37000]: Saving service mon spec with placement smithi052:172.21.15.52=a;smithi105:172.21.15.105=b;smithi173:172.21.15.173=c;count:3 2024-05-27T07:15:16.107 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:15:15 smithi173 ceph-mon[37000]: Deploying daemon mon.c on smithi173 2024-05-27T07:15:16.107 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:15:15 smithi173 ceph-mon[37000]: pgmap v15: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-27T07:15:16.107 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:15:15 smithi173 ceph-mon[37000]: pgmap v16: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-27T07:15:16.107 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:15:15 smithi173 ceph-mon[37000]: from='client.? 172.21.15.173:0/53244174' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-05-27T07:15:16.107 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:15:15 smithi173 ceph-mon[37000]: mon.c@-1(synchronizing).paxosservice(auth 1..3) refresh upgraded, format 0 -> 3 2024-05-27T07:15:16.107 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:15:15 smithi173 systemd[1]: Started Ceph mon.c for 73df2636-1bf8-11ef-bc9b-c7b262605968. 2024-05-27T07:15:16.286 INFO:teuthology.orchestra.run.smithi173.stderr:Inferring config /var/lib/ceph/73df2636-1bf8-11ef-bc9b-c7b262605968/mon.c/config 2024-05-27T07:15:20.882 INFO:teuthology.orchestra.run.smithi173.stderr:2024-05-27T07:15:20.891+0000 7fc7529ad700 1 -- 172.21.15.173:0/1822446079 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fc7340035c0 msgr2=0x7fc734005aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:15:20.883 INFO:teuthology.orchestra.run.smithi173.stderr:2024-05-27T07:15:20.891+0000 7fc7529ad700 1 --2- 172.21.15.173:0/1822446079 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fc7340035c0 0x7fc734005aa0 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7fc74c06bdb0 tx=0x7fc740013e90 comp rx=0 tx=0).stop 2024-05-27T07:15:20.883 INFO:teuthology.orchestra.run.smithi173.stderr:2024-05-27T07:15:20.891+0000 7fc7529ad700 1 -- 172.21.15.173:0/1822446079 shutdown_connections 2024-05-27T07:15:20.883 INFO:teuthology.orchestra.run.smithi173.stderr:2024-05-27T07:15:20.891+0000 7fc7529ad700 1 --2- 172.21.15.173:0/1822446079 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fc7340035c0 0x7fc734005aa0 unknown :-1 s=CLOSED pgs=100 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:15:20.883 INFO:teuthology.orchestra.run.smithi173.stderr:2024-05-27T07:15:20.891+0000 7fc7529ad700 1 -- 172.21.15.173:0/1822446079 >> 172.21.15.173:0/1822446079 conn(0x7fc74c0797d0 msgr2=0x7fc74c079c10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:15:20.883 INFO:teuthology.orchestra.run.smithi173.stderr:2024-05-27T07:15:20.891+0000 7fc7529ad700 1 -- 172.21.15.173:0/1822446079 shutdown_connections 2024-05-27T07:15:20.883 INFO:teuthology.orchestra.run.smithi173.stderr:2024-05-27T07:15:20.891+0000 7fc7529ad700 1 -- 172.21.15.173:0/1822446079 wait complete. 2024-05-27T07:15:20.884 INFO:teuthology.orchestra.run.smithi173.stderr:2024-05-27T07:15:20.892+0000 7fc7529ad700 1 Processor -- start 2024-05-27T07:15:20.884 INFO:teuthology.orchestra.run.smithi173.stderr:2024-05-27T07:15:20.892+0000 7fc7529ad700 1 -- start start 2024-05-27T07:15:20.884 INFO:teuthology.orchestra.run.smithi173.stderr:2024-05-27T07:15:20.893+0000 7fc7529ad700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fc7340035c0 0x7fc74c19f690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:15:20.884 INFO:teuthology.orchestra.run.smithi173.stderr:2024-05-27T07:15:20.893+0000 7fc7529ad700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fc74c19fbd0 0x7fc74c1998b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:15:20.884 INFO:teuthology.orchestra.run.smithi173.stderr:2024-05-27T07:15:20.893+0000 7fc7529ad700 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x7fc74c10bc00 con 0x7fc7340035c0 2024-05-27T07:15:20.885 INFO:teuthology.orchestra.run.smithi173.stderr:2024-05-27T07:15:20.893+0000 7fc7529ad700 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7fc74c10ba80 con 0x7fc74c19fbd0 2024-05-27T07:15:20.885 INFO:teuthology.orchestra.run.smithi173.stderr:2024-05-27T07:15:20.893+0000 7fc74b7fe700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fc74c19fbd0 0x7fc74c1998b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:15:20.885 INFO:teuthology.orchestra.run.smithi173.stderr:2024-05-27T07:15:20.893+0000 7fc74b7fe700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fc74c19fbd0 0x7fc74c1998b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.173:3300/0 says I am v2:172.21.15.173:54620/0 (socket says 172.21.15.173:54620) 2024-05-27T07:15:20.885 INFO:teuthology.orchestra.run.smithi173.stderr:2024-05-27T07:15:20.893+0000 7fc74b7fe700 1 -- 172.21.15.173:0/4091862757 learned_addr learned my addr 172.21.15.173:0/4091862757 (peer_addr_for_me v2:172.21.15.173:0/0) 2024-05-27T07:15:20.885 INFO:teuthology.orchestra.run.smithi173.stderr:2024-05-27T07:15:20.893+0000 7fc74b7fe700 1 -- 172.21.15.173:0/4091862757 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fc74c19fbd0 msgr2=0x7fc74c1998b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2024-05-27T07:15:20.885 INFO:teuthology.orchestra.run.smithi173.stderr:2024-05-27T07:15:20.893+0000 7fc74bfff700 1 --2- 172.21.15.173:0/4091862757 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fc7340035c0 0x7fc74c19f690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:15:20.885 INFO:teuthology.orchestra.run.smithi173.stderr:2024-05-27T07:15:20.893+0000 7fc74b7fe700 1 -- 172.21.15.173:0/4091862757 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fc74c19fbd0 msgr2=0x7fc74c1998b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-05-27T07:15:20.885 INFO:teuthology.orchestra.run.smithi173.stderr:2024-05-27T07:15:20.893+0000 7fc74b7fe700 1 --2- 172.21.15.173:0/4091862757 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fc74c19fbd0 0x7fc74c1998b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-05-27T07:15:20.885 INFO:teuthology.orchestra.run.smithi173.stderr:2024-05-27T07:15:20.893+0000 7fc74b7fe700 1 --2- 172.21.15.173:0/4091862757 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fc74c19fbd0 0x7fc74c1998b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-05-27T07:15:20.885 INFO:teuthology.orchestra.run.smithi173.stderr:2024-05-27T07:15:20.894+0000 7fc74bfff700 1 -- 172.21.15.173:0/4091862757 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fc74c19fbd0 msgr2=0x7fc74c1998b0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-05-27T07:15:20.885 INFO:teuthology.orchestra.run.smithi173.stderr:2024-05-27T07:15:20.894+0000 7fc74bfff700 1 --2- 172.21.15.173:0/4091862757 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fc74c19fbd0 0x7fc74c1998b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:15:20.885 INFO:teuthology.orchestra.run.smithi173.stderr:2024-05-27T07:15:20.894+0000 7fc74bfff700 1 -- 172.21.15.173:0/4091862757 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc74c1a01a0 con 0x7fc7340035c0 2024-05-27T07:15:20.887 INFO:teuthology.orchestra.run.smithi173.stderr:2024-05-27T07:15:20.894+0000 7fc74bfff700 1 --2- 172.21.15.173:0/4091862757 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fc7340035c0 0x7fc74c19f690 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7fc74001c000 tx=0x7fc740005500 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:15:20.887 INFO:teuthology.orchestra.run.smithi173.stderr:2024-05-27T07:15:20.894+0000 7fc7497fa700 1 -- 172.21.15.173:0/4091862757 <== mon.0 v2:172.21.15.52:3300/0 1 ==== mon_map magic: 0 ==== 309+0+0 (secure 0 0 0) 0x7fc740021040 con 0x7fc7340035c0 2024-05-27T07:15:20.887 INFO:teuthology.orchestra.run.smithi173.stderr:2024-05-27T07:15:20.894+0000 7fc7497fa700 1 -- 172.21.15.173:0/4091862757 <== mon.0 v2:172.21.15.52:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fc740026410 con 0x7fc7340035c0 2024-05-27T07:15:20.887 INFO:teuthology.orchestra.run.smithi173.stderr:2024-05-27T07:15:20.894+0000 7fc7529ad700 1 -- 172.21.15.173:0/4091862757 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc74c199e50 con 0x7fc7340035c0 2024-05-27T07:15:20.887 INFO:teuthology.orchestra.run.smithi173.stderr:2024-05-27T07:15:20.894+0000 7fc7497fa700 1 -- 172.21.15.173:0/4091862757 <== mon.0 v2:172.21.15.52:3300/0 3 ==== mon_map magic: 0 ==== 309+0+0 (secure 0 0 0) 0x7fc74001c610 con 0x7fc7340035c0 2024-05-27T07:15:20.887 INFO:teuthology.orchestra.run.smithi173.stderr:2024-05-27T07:15:20.894+0000 7fc7529ad700 1 -- 172.21.15.173:0/4091862757 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc74c19a3e0 con 0x7fc7340035c0 2024-05-27T07:15:20.887 INFO:teuthology.orchestra.run.smithi173.stderr:2024-05-27T07:15:20.895+0000 7fc7497fa700 1 -- 172.21.15.173:0/4091862757 <== mon.0 v2:172.21.15.52:3300/0 4 ==== mgrmap(e 13) ==== 50278+0+0 (secure 0 0 0) 0x7fc74002d020 con 0x7fc7340035c0 2024-05-27T07:15:20.887 INFO:teuthology.orchestra.run.smithi173.stderr:2024-05-27T07:15:20.895+0000 7fc7529ad700 1 -- 172.21.15.173:0/4091862757 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc74c06c5b0 con 0x7fc7340035c0 2024-05-27T07:15:20.887 INFO:teuthology.orchestra.run.smithi173.stderr:2024-05-27T07:15:20.896+0000 7fc7497fa700 1 --2- 172.21.15.173:0/4091862757 >> [v2:172.21.15.52:6800/3513075805,v1:172.21.15.52:6801/3513075805] conn(0x7fc73803e300 0x7fc738040810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:15:20.887 INFO:teuthology.orchestra.run.smithi173.stderr:2024-05-27T07:15:20.896+0000 7fc7497fa700 1 -- 172.21.15.173:0/4091862757 <== mon.0 v2:172.21.15.52:3300/0 5 ==== osd_map(4..4 src has 1..4) ==== 1155+0+0 (secure 0 0 0) 0x7fc740019070 con 0x7fc7340035c0 2024-05-27T07:15:20.888 INFO:teuthology.orchestra.run.smithi173.stderr:2024-05-27T07:15:20.896+0000 7fc74b7fe700 1 --2- 172.21.15.173:0/4091862757 >> [v2:172.21.15.52:6800/3513075805,v1:172.21.15.52:6801/3513075805] conn(0x7fc73803e300 0x7fc738040810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:15:20.888 INFO:teuthology.orchestra.run.smithi173.stderr:2024-05-27T07:15:20.896+0000 7fc74b7fe700 1 --2- 172.21.15.173:0/4091862757 >> [v2:172.21.15.52:6800/3513075805,v1:172.21.15.52:6801/3513075805] conn(0x7fc73803e300 0x7fc738040810 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7fc74c109920 tx=0x7fc73c008040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:15:20.892 INFO:teuthology.orchestra.run.smithi173.stderr:2024-05-27T07:15:20.902+0000 7fc7497fa700 1 -- 172.21.15.173:0/4091862757 <== mon.0 v2:172.21.15.52:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191963 (secure 0 0 0) 0x7fc740034020 con 0x7fc7340035c0 2024-05-27T07:15:21.187 INFO:teuthology.orchestra.run.smithi173.stderr:2024-05-27T07:15:21.196+0000 7fc7529ad700 1 -- 172.21.15.173:0/4091862757 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) -- 0x7fc74c19ade0 con 0x7fc7340035c0 2024-05-27T07:15:21.188 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:15:20 smithi173 ceph-mon[37000]: pgmap v17: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-27T07:15:21.188 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:15:20 smithi173 ceph-mon[37000]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-05-27T07:15:21.188 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:15:20 smithi173 ceph-mon[37000]: mon.a calling monitor election 2024-05-27T07:15:21.188 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:15:20 smithi173 ceph-mon[37000]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-05-27T07:15:21.188 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:15:20 smithi173 ceph-mon[37000]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-05-27T07:15:21.188 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:15:20 smithi173 ceph-mon[37000]: pgmap v18: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-27T07:15:21.188 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:15:20 smithi173 ceph-mon[37000]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-05-27T07:15:21.188 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:15:20 smithi173 ceph-mon[37000]: mon.c calling monitor election 2024-05-27T07:15:21.188 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:15:20 smithi173 ceph-mon[37000]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-05-27T07:15:21.188 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:15:20 smithi173 ceph-mon[37000]: pgmap v19: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-27T07:15:21.189 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:15:20 smithi173 ceph-mon[37000]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-05-27T07:15:21.189 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:15:20 smithi173 ceph-mon[37000]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-05-27T07:15:21.189 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:15:20 smithi173 ceph-mon[37000]: mon.a is new leader, mons a,c in quorum (ranks 0,1) 2024-05-27T07:15:21.189 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:15:20 smithi173 ceph-mon[37000]: monmap epoch 2 2024-05-27T07:15:21.189 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:15:20 smithi173 ceph-mon[37000]: fsid 73df2636-1bf8-11ef-bc9b-c7b262605968 2024-05-27T07:15:21.189 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:15:20 smithi173 ceph-mon[37000]: last_changed 2024-05-27T07:15:15.809004+0000 2024-05-27T07:15:21.189 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:15:20 smithi173 ceph-mon[37000]: created 2024-05-27T07:13:31.465220+0000 2024-05-27T07:15:21.189 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:15:20 smithi173 ceph-mon[37000]: min_mon_release 19 (squid) 2024-05-27T07:15:21.189 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:15:20 smithi173 ceph-mon[37000]: election_strategy: 1 2024-05-27T07:15:21.189 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:15:20 smithi173 ceph-mon[37000]: 0: [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] mon.a 2024-05-27T07:15:21.189 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:15:20 smithi173 ceph-mon[37000]: 1: [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] mon.c 2024-05-27T07:15:21.189 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:15:20 smithi173 ceph-mon[37000]: fsmap 2024-05-27T07:15:21.189 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:15:20 smithi173 ceph-mon[37000]: osdmap e4: 0 total, 0 up, 0 in 2024-05-27T07:15:21.190 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:15:20 smithi173 ceph-mon[37000]: mgrmap e13: x(active, since 51s) 2024-05-27T07:15:21.190 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:15:20 smithi173 ceph-mon[37000]: overall HEALTH_OK 2024-05-27T07:15:21.190 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:15:20 smithi173 ceph-mon[37000]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' 2024-05-27T07:15:21.190 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:15:20 smithi173 ceph-mon[37000]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' 2024-05-27T07:15:21.190 INFO:teuthology.orchestra.run.smithi173.stderr:2024-05-27T07:15:21.197+0000 7fc7497fa700 1 -- 172.21.15.173:0/4091862757 <== mon.0 v2:172.21.15.52:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 2 v2) ==== 95+0+1015 (secure 0 0 0) 0x7fc740025c30 con 0x7fc7340035c0 2024-05-27T07:15:21.190 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-05-27T07:15:21.190 INFO:teuthology.orchestra.run.smithi173.stdout:{"epoch":2,"fsid":"73df2636-1bf8-11ef-bc9b-c7b262605968","modified":"2024-05-27T07:15:15.809004Z","created":"2024-05-27T07:13:31.465220Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.52:3300","nonce":0},{"type":"v1","addr":"172.21.15.52:6789","nonce":0}]},"addr":"172.21.15.52:6789/0","public_addr":"172.21.15.52:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:3300","nonce":0},{"type":"v1","addr":"172.21.15.173:6789","nonce":0}]},"addr":"172.21.15.173:6789/0","public_addr":"172.21.15.173:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0,1]} 2024-05-27T07:15:21.191 INFO:teuthology.orchestra.run.smithi173.stderr:2024-05-27T07:15:21.200+0000 7fc7529ad700 1 -- 172.21.15.173:0/4091862757 >> [v2:172.21.15.52:6800/3513075805,v1:172.21.15.52:6801/3513075805] conn(0x7fc73803e300 msgr2=0x7fc738040810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:15:21.191 INFO:teuthology.orchestra.run.smithi173.stderr:2024-05-27T07:15:21.200+0000 7fc7529ad700 1 --2- 172.21.15.173:0/4091862757 >> [v2:172.21.15.52:6800/3513075805,v1:172.21.15.52:6801/3513075805] conn(0x7fc73803e300 0x7fc738040810 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7fc74c109920 tx=0x7fc73c008040 comp rx=0 tx=0).stop 2024-05-27T07:15:21.191 INFO:teuthology.orchestra.run.smithi173.stderr:2024-05-27T07:15:21.200+0000 7fc7529ad700 1 -- 172.21.15.173:0/4091862757 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fc7340035c0 msgr2=0x7fc74c19f690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:15:21.191 INFO:teuthology.orchestra.run.smithi173.stderr:2024-05-27T07:15:21.200+0000 7fc7529ad700 1 --2- 172.21.15.173:0/4091862757 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fc7340035c0 0x7fc74c19f690 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7fc74001c000 tx=0x7fc740005500 comp rx=0 tx=0).stop 2024-05-27T07:15:21.191 INFO:teuthology.orchestra.run.smithi173.stderr:2024-05-27T07:15:21.200+0000 7fc7529ad700 1 -- 172.21.15.173:0/4091862757 shutdown_connections 2024-05-27T07:15:21.192 INFO:teuthology.orchestra.run.smithi173.stderr:2024-05-27T07:15:21.200+0000 7fc7529ad700 1 --2- 172.21.15.173:0/4091862757 >> [v2:172.21.15.52:6800/3513075805,v1:172.21.15.52:6801/3513075805] conn(0x7fc73803e300 0x7fc738040810 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:15:21.192 INFO:teuthology.orchestra.run.smithi173.stderr:2024-05-27T07:15:21.200+0000 7fc7529ad700 1 --2- 172.21.15.173:0/4091862757 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fc7340035c0 0x7fc74c19f690 unknown :-1 s=CLOSED pgs=101 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:15:21.192 INFO:teuthology.orchestra.run.smithi173.stderr:2024-05-27T07:15:21.200+0000 7fc7529ad700 1 --2- 172.21.15.173:0/4091862757 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fc74c19fbd0 0x7fc74c1998b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:15:21.192 INFO:teuthology.orchestra.run.smithi173.stderr:2024-05-27T07:15:21.200+0000 7fc7529ad700 1 -- 172.21.15.173:0/4091862757 >> 172.21.15.173:0/4091862757 conn(0x7fc74c0797d0 msgr2=0x7fc74c101cd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:15:21.192 INFO:teuthology.orchestra.run.smithi173.stderr:2024-05-27T07:15:21.201+0000 7fc7529ad700 1 -- 172.21.15.173:0/4091862757 shutdown_connections 2024-05-27T07:15:21.192 INFO:teuthology.orchestra.run.smithi173.stderr:2024-05-27T07:15:21.201+0000 7fc7529ad700 1 -- 172.21.15.173:0/4091862757 wait complete. 2024-05-27T07:15:21.192 INFO:teuthology.orchestra.run.smithi173.stderr:dumped monmap epoch 2 2024-05-27T07:15:21.231 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:15:20 smithi052 ceph-mon[31955]: pgmap v17: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-27T07:15:21.231 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:15:20 smithi052 ceph-mon[31955]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-05-27T07:15:21.231 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:15:20 smithi052 ceph-mon[31955]: mon.a calling monitor election 2024-05-27T07:15:21.231 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:15:20 smithi052 ceph-mon[31955]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-05-27T07:15:21.231 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:15:20 smithi052 ceph-mon[31955]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-05-27T07:15:21.231 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:15:20 smithi052 ceph-mon[31955]: pgmap v18: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-27T07:15:21.231 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:15:20 smithi052 ceph-mon[31955]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-05-27T07:15:21.231 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:15:20 smithi052 ceph-mon[31955]: mon.c calling monitor election 2024-05-27T07:15:21.231 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:15:20 smithi052 ceph-mon[31955]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-05-27T07:15:21.231 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:15:20 smithi052 ceph-mon[31955]: pgmap v19: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-27T07:15:21.231 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:15:20 smithi052 ceph-mon[31955]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-05-27T07:15:21.232 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:15:20 smithi052 ceph-mon[31955]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-05-27T07:15:21.232 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:15:20 smithi052 ceph-mon[31955]: mon.a is new leader, mons a,c in quorum (ranks 0,1) 2024-05-27T07:15:21.232 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:15:20 smithi052 ceph-mon[31955]: monmap epoch 2 2024-05-27T07:15:21.232 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:15:20 smithi052 ceph-mon[31955]: fsid 73df2636-1bf8-11ef-bc9b-c7b262605968 2024-05-27T07:15:21.232 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:15:20 smithi052 ceph-mon[31955]: last_changed 2024-05-27T07:15:15.809004+0000 2024-05-27T07:15:21.232 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:15:20 smithi052 ceph-mon[31955]: created 2024-05-27T07:13:31.465220+0000 2024-05-27T07:15:21.232 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:15:20 smithi052 ceph-mon[31955]: min_mon_release 19 (squid) 2024-05-27T07:15:21.232 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:15:20 smithi052 ceph-mon[31955]: election_strategy: 1 2024-05-27T07:15:21.232 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:15:20 smithi052 ceph-mon[31955]: 0: [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] mon.a 2024-05-27T07:15:21.232 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:15:20 smithi052 ceph-mon[31955]: 1: [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] mon.c 2024-05-27T07:15:21.232 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:15:20 smithi052 ceph-mon[31955]: fsmap 2024-05-27T07:15:21.232 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:15:20 smithi052 ceph-mon[31955]: osdmap e4: 0 total, 0 up, 0 in 2024-05-27T07:15:21.232 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:15:20 smithi052 ceph-mon[31955]: mgrmap e13: x(active, since 51s) 2024-05-27T07:15:21.232 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:15:20 smithi052 ceph-mon[31955]: overall HEALTH_OK 2024-05-27T07:15:21.233 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:15:20 smithi052 ceph-mon[31955]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' 2024-05-27T07:15:21.233 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:15:20 smithi052 ceph-mon[31955]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' 2024-05-27T07:15:22.133 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:15:21 smithi052 ceph-mon[31955]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' 2024-05-27T07:15:22.133 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:15:21 smithi052 ceph-mon[31955]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-05-27T07:15:22.133 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:15:21 smithi052 ceph-mon[31955]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T07:15:22.133 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:15:21 smithi052 ceph-mon[31955]: Deploying daemon mon.b on smithi105 2024-05-27T07:15:22.133 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:15:21 smithi052 ceph-mon[31955]: from='client.? 172.21.15.173:0/4091862757' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-05-27T07:15:22.134 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:15:21 smithi052 ceph-mon[31955]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-05-27T07:15:22.352 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:15:21 smithi173 ceph-mon[37000]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' 2024-05-27T07:15:22.353 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:15:21 smithi173 ceph-mon[37000]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-05-27T07:15:22.353 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:15:21 smithi173 ceph-mon[37000]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T07:15:22.353 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:15:21 smithi173 ceph-mon[37000]: Deploying daemon mon.b on smithi105 2024-05-27T07:15:22.353 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:15:21 smithi173 ceph-mon[37000]: from='client.? 172.21.15.173:0/4091862757' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-05-27T07:15:22.353 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:15:21 smithi173 ceph-mon[37000]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-05-27T07:15:22.649 INFO:tasks.cephadm:Waiting for 3 mons in monmap... 2024-05-27T07:15:22.649 DEBUG:teuthology.orchestra.run.smithi173:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:38f9d900c83c63799fbdbe61acc9a11b0d3554a6 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 73df2636-1bf8-11ef-bc9b-c7b262605968 -- ceph mon dump -f json 2024-05-27T07:15:22.893 INFO:teuthology.orchestra.run.smithi173.stderr:Inferring config /var/lib/ceph/73df2636-1bf8-11ef-bc9b-c7b262605968/mon.c/config 2024-05-27T07:15:23.014 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:15:22 smithi173 ceph-mon[37000]: pgmap v20: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-27T07:15:23.230 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:15:22 smithi052 ceph-mon[31955]: pgmap v20: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-27T07:15:24.394 INFO:teuthology.orchestra.run.smithi173.stderr:2024-05-27T07:15:24.403+0000 7fdfa4baa700 1 -- 172.21.15.173:0/3493240246 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fdfa0108c30 msgr2=0x7fdfa0109040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:15:24.395 INFO:teuthology.orchestra.run.smithi173.stderr:2024-05-27T07:15:24.403+0000 7fdfa4baa700 1 --2- 172.21.15.173:0/3493240246 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fdfa0108c30 0x7fdfa0109040 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7fdf94009910 tx=0x7fdf94009c20 comp rx=0 tx=0).stop 2024-05-27T07:15:24.395 INFO:teuthology.orchestra.run.smithi173.stderr:2024-05-27T07:15:24.403+0000 7fdfa4baa700 1 -- 172.21.15.173:0/3493240246 shutdown_connections 2024-05-27T07:15:24.395 INFO:teuthology.orchestra.run.smithi173.stderr:2024-05-27T07:15:24.403+0000 7fdfa4baa700 1 --2- 172.21.15.173:0/3493240246 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fdfa0108c30 0x7fdfa0109040 unknown :-1 s=CLOSED pgs=102 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:15:24.395 INFO:teuthology.orchestra.run.smithi173.stderr:2024-05-27T07:15:24.403+0000 7fdfa4baa700 1 -- 172.21.15.173:0/3493240246 >> 172.21.15.173:0/3493240246 conn(0x7fdfa00797d0 msgr2=0x7fdfa0079c10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:15:24.395 INFO:teuthology.orchestra.run.smithi173.stderr:2024-05-27T07:15:24.403+0000 7fdfa4baa700 1 -- 172.21.15.173:0/3493240246 shutdown_connections 2024-05-27T07:15:24.395 INFO:teuthology.orchestra.run.smithi173.stderr:2024-05-27T07:15:24.403+0000 7fdfa4baa700 1 -- 172.21.15.173:0/3493240246 wait complete. 2024-05-27T07:15:24.397 INFO:teuthology.orchestra.run.smithi173.stderr:2024-05-27T07:15:24.404+0000 7fdfa4baa700 1 Processor -- start 2024-05-27T07:15:24.397 INFO:teuthology.orchestra.run.smithi173.stderr:2024-05-27T07:15:24.404+0000 7fdfa4baa700 1 -- start start 2024-05-27T07:15:24.397 INFO:teuthology.orchestra.run.smithi173.stderr:2024-05-27T07:15:24.404+0000 7fdfa4baa700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fdfa0108c30 0x7fdfa019e780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:15:24.397 INFO:teuthology.orchestra.run.smithi173.stderr:2024-05-27T07:15:24.404+0000 7fdfa4baa700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fdfa019ecc0 0x7fdfa01989a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:15:24.397 INFO:teuthology.orchestra.run.smithi173.stderr:2024-05-27T07:15:24.404+0000 7fdfa4baa700 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x7fdfa010d110 con 0x7fdfa0108c30 2024-05-27T07:15:24.397 INFO:teuthology.orchestra.run.smithi173.stderr:2024-05-27T07:15:24.404+0000 7fdfa4baa700 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7fdfa010cf90 con 0x7fdfa019ecc0 2024-05-27T07:15:24.397 INFO:teuthology.orchestra.run.smithi173.stderr:2024-05-27T07:15:24.405+0000 7fdf9e59c700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fdfa0108c30 0x7fdfa019e780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:15:24.398 INFO:teuthology.orchestra.run.smithi173.stderr:2024-05-27T07:15:24.405+0000 7fdf9e59c700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fdfa0108c30 0x7fdfa019e780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.173:38250/0 (socket says 172.21.15.173:38250) 2024-05-27T07:15:24.398 INFO:teuthology.orchestra.run.smithi173.stderr:2024-05-27T07:15:24.405+0000 7fdf9e59c700 1 -- 172.21.15.173:0/3304214027 learned_addr learned my addr 172.21.15.173:0/3304214027 (peer_addr_for_me v2:172.21.15.173:0/0) 2024-05-27T07:15:24.398 INFO:teuthology.orchestra.run.smithi173.stderr:2024-05-27T07:15:24.405+0000 7fdf9dd9b700 1 --2- 172.21.15.173:0/3304214027 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fdfa019ecc0 0x7fdfa01989a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:15:24.398 INFO:teuthology.orchestra.run.smithi173.stderr:2024-05-27T07:15:24.405+0000 7fdf9dd9b700 1 -- 172.21.15.173:0/3304214027 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fdfa019ecc0 msgr2=0x7fdfa01989a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2024-05-27T07:15:24.398 INFO:teuthology.orchestra.run.smithi173.stderr:2024-05-27T07:15:24.405+0000 7fdf9dd9b700 1 -- 172.21.15.173:0/3304214027 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fdfa019ecc0 msgr2=0x7fdfa01989a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-05-27T07:15:24.398 INFO:teuthology.orchestra.run.smithi173.stderr:2024-05-27T07:15:24.405+0000 7fdf9dd9b700 1 --2- 172.21.15.173:0/3304214027 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fdfa019ecc0 0x7fdfa01989a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-05-27T07:15:24.398 INFO:teuthology.orchestra.run.smithi173.stderr:2024-05-27T07:15:24.405+0000 7fdf9dd9b700 1 --2- 172.21.15.173:0/3304214027 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fdfa019ecc0 0x7fdfa01989a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-05-27T07:15:24.398 INFO:teuthology.orchestra.run.smithi173.stderr:2024-05-27T07:15:24.405+0000 7fdf9e59c700 1 -- 172.21.15.173:0/3304214027 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fdfa019ecc0 msgr2=0x7fdfa01989a0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-05-27T07:15:24.398 INFO:teuthology.orchestra.run.smithi173.stderr:2024-05-27T07:15:24.405+0000 7fdf9e59c700 1 --2- 172.21.15.173:0/3304214027 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fdfa019ecc0 0x7fdfa01989a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:15:24.398 INFO:teuthology.orchestra.run.smithi173.stderr:2024-05-27T07:15:24.405+0000 7fdf9e59c700 1 -- 172.21.15.173:0/3304214027 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdfa019f290 con 0x7fdfa0108c30 2024-05-27T07:15:24.398 INFO:teuthology.orchestra.run.smithi173.stderr:2024-05-27T07:15:24.405+0000 7fdf9e59c700 1 --2- 172.21.15.173:0/3304214027 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fdfa0108c30 0x7fdfa019e780 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7fdf94006010 tx=0x7fdf94005030 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:15:24.398 INFO:teuthology.orchestra.run.smithi173.stderr:2024-05-27T07:15:24.405+0000 7fdf8f7fe700 1 -- 172.21.15.173:0/3304214027 <== mon.0 v2:172.21.15.52:3300/0 1 ==== mon_map magic: 0 ==== 309+0+0 (secure 0 0 0) 0x7fdf94005dc0 con 0x7fdfa0108c30 2024-05-27T07:15:24.399 INFO:teuthology.orchestra.run.smithi173.stderr:2024-05-27T07:15:24.405+0000 7fdf8f7fe700 1 -- 172.21.15.173:0/3304214027 <== mon.0 v2:172.21.15.52:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fdf9400be10 con 0x7fdfa0108c30 2024-05-27T07:15:24.399 INFO:teuthology.orchestra.run.smithi173.stderr:2024-05-27T07:15:24.405+0000 7fdfa4baa700 1 -- 172.21.15.173:0/3304214027 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fdfa0198ee0 con 0x7fdfa0108c30 2024-05-27T07:15:24.399 INFO:teuthology.orchestra.run.smithi173.stderr:2024-05-27T07:15:24.405+0000 7fdf8f7fe700 1 -- 172.21.15.173:0/3304214027 <== mon.0 v2:172.21.15.52:3300/0 3 ==== mon_map magic: 0 ==== 309+0+0 (secure 0 0 0) 0x7fdf94022e00 con 0x7fdfa0108c30 2024-05-27T07:15:24.399 INFO:teuthology.orchestra.run.smithi173.stderr:2024-05-27T07:15:24.405+0000 7fdfa4baa700 1 -- 172.21.15.173:0/3304214027 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fdfa0199470 con 0x7fdfa0108c30 2024-05-27T07:15:24.399 INFO:teuthology.orchestra.run.smithi173.stderr:2024-05-27T07:15:24.406+0000 7fdfa4baa700 1 -- 172.21.15.173:0/3304214027 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fdfa0104200 con 0x7fdfa0108c30 2024-05-27T07:15:24.400 INFO:teuthology.orchestra.run.smithi173.stderr:2024-05-27T07:15:24.407+0000 7fdf8f7fe700 1 -- 172.21.15.173:0/3304214027 <== mon.0 v2:172.21.15.52:3300/0 4 ==== mgrmap(e 13) ==== 50278+0+0 (secure 0 0 0) 0x7fdf9402a070 con 0x7fdfa0108c30 2024-05-27T07:15:24.400 INFO:teuthology.orchestra.run.smithi173.stderr:2024-05-27T07:15:24.407+0000 7fdf8f7fe700 1 --2- 172.21.15.173:0/3304214027 >> [v2:172.21.15.52:6800/3513075805,v1:172.21.15.52:6801/3513075805] conn(0x7fdf8803e2b0 0x7fdf880407c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:15:24.400 INFO:teuthology.orchestra.run.smithi173.stderr:2024-05-27T07:15:24.407+0000 7fdf8f7fe700 1 -- 172.21.15.173:0/3304214027 <== mon.0 v2:172.21.15.52:3300/0 5 ==== osd_map(4..4 src has 1..4) ==== 1155+0+0 (secure 0 0 0) 0x7fdf9401e030 con 0x7fdfa0108c30 2024-05-27T07:15:24.400 INFO:teuthology.orchestra.run.smithi173.stderr:2024-05-27T07:15:24.407+0000 7fdf9dd9b700 1 --2- 172.21.15.173:0/3304214027 >> [v2:172.21.15.52:6800/3513075805,v1:172.21.15.52:6801/3513075805] conn(0x7fdf8803e2b0 0x7fdf880407c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:15:24.400 INFO:teuthology.orchestra.run.smithi173.stderr:2024-05-27T07:15:24.407+0000 7fdf9dd9b700 1 --2- 172.21.15.173:0/3304214027 >> [v2:172.21.15.52:6800/3513075805,v1:172.21.15.52:6801/3513075805] conn(0x7fdf8803e2b0 0x7fdf880407c0 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7fdfa0102410 tx=0x7fdf90008040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:15:24.403 INFO:teuthology.orchestra.run.smithi173.stderr:2024-05-27T07:15:24.412+0000 7fdf8f7fe700 1 -- 172.21.15.173:0/3304214027 <== mon.0 v2:172.21.15.52:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191963 (secure 0 0 0) 0x7fdf94013070 con 0x7fdfa0108c30 2024-05-27T07:15:24.693 INFO:teuthology.orchestra.run.smithi173.stderr:2024-05-27T07:15:24.702+0000 7fdfa4baa700 1 -- 172.21.15.173:0/3304214027 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) -- 0x7fdfa0199ed0 con 0x7fdfa0108c30 2024-05-27T07:15:24.694 INFO:teuthology.orchestra.run.smithi173.stderr:2024-05-27T07:15:24.703+0000 7fdf8f7fe700 1 -- 172.21.15.173:0/3304214027 <== mon.0 v2:172.21.15.52:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 2 v2) ==== 95+0+1015 (secure 0 0 0) 0x7fdf9400b960 con 0x7fdfa0108c30 2024-05-27T07:15:24.694 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-05-27T07:15:24.694 INFO:teuthology.orchestra.run.smithi173.stdout:{"epoch":2,"fsid":"73df2636-1bf8-11ef-bc9b-c7b262605968","modified":"2024-05-27T07:15:15.809004Z","created":"2024-05-27T07:13:31.465220Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.52:3300","nonce":0},{"type":"v1","addr":"172.21.15.52:6789","nonce":0}]},"addr":"172.21.15.52:6789/0","public_addr":"172.21.15.52:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:3300","nonce":0},{"type":"v1","addr":"172.21.15.173:6789","nonce":0}]},"addr":"172.21.15.173:6789/0","public_addr":"172.21.15.173:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0,1]} 2024-05-27T07:15:24.696 INFO:teuthology.orchestra.run.smithi173.stderr:2024-05-27T07:15:24.705+0000 7fdfa4baa700 1 -- 172.21.15.173:0/3304214027 >> [v2:172.21.15.52:6800/3513075805,v1:172.21.15.52:6801/3513075805] conn(0x7fdf8803e2b0 msgr2=0x7fdf880407c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:15:24.696 INFO:teuthology.orchestra.run.smithi173.stderr:2024-05-27T07:15:24.705+0000 7fdfa4baa700 1 --2- 172.21.15.173:0/3304214027 >> [v2:172.21.15.52:6800/3513075805,v1:172.21.15.52:6801/3513075805] conn(0x7fdf8803e2b0 0x7fdf880407c0 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7fdfa0102410 tx=0x7fdf90008040 comp rx=0 tx=0).stop 2024-05-27T07:15:24.696 INFO:teuthology.orchestra.run.smithi173.stderr:2024-05-27T07:15:24.705+0000 7fdfa4baa700 1 -- 172.21.15.173:0/3304214027 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fdfa0108c30 msgr2=0x7fdfa019e780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:15:24.696 INFO:teuthology.orchestra.run.smithi173.stderr:2024-05-27T07:15:24.705+0000 7fdfa4baa700 1 --2- 172.21.15.173:0/3304214027 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fdfa0108c30 0x7fdfa019e780 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7fdf94006010 tx=0x7fdf94005030 comp rx=0 tx=0).stop 2024-05-27T07:15:24.697 INFO:teuthology.orchestra.run.smithi173.stderr:2024-05-27T07:15:24.705+0000 7fdfa4baa700 1 -- 172.21.15.173:0/3304214027 shutdown_connections 2024-05-27T07:15:24.697 INFO:teuthology.orchestra.run.smithi173.stderr:2024-05-27T07:15:24.705+0000 7fdfa4baa700 1 --2- 172.21.15.173:0/3304214027 >> [v2:172.21.15.52:6800/3513075805,v1:172.21.15.52:6801/3513075805] conn(0x7fdf8803e2b0 0x7fdf880407c0 unknown :-1 s=CLOSED pgs=30 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:15:24.697 INFO:teuthology.orchestra.run.smithi173.stderr:2024-05-27T07:15:24.705+0000 7fdfa4baa700 1 --2- 172.21.15.173:0/3304214027 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fdfa0108c30 0x7fdfa019e780 unknown :-1 s=CLOSED pgs=103 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:15:24.697 INFO:teuthology.orchestra.run.smithi173.stderr:2024-05-27T07:15:24.705+0000 7fdfa4baa700 1 --2- 172.21.15.173:0/3304214027 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fdfa019ecc0 0x7fdfa01989a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:15:24.697 INFO:teuthology.orchestra.run.smithi173.stderr:2024-05-27T07:15:24.705+0000 7fdfa4baa700 1 -- 172.21.15.173:0/3304214027 >> 172.21.15.173:0/3304214027 conn(0x7fdfa00797d0 msgr2=0x7fdfa00ffda0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:15:24.697 INFO:teuthology.orchestra.run.smithi173.stderr:2024-05-27T07:15:24.705+0000 7fdfa4baa700 1 -- 172.21.15.173:0/3304214027 shutdown_connections 2024-05-27T07:15:24.697 INFO:teuthology.orchestra.run.smithi173.stderr:2024-05-27T07:15:24.705+0000 7fdfa4baa700 1 -- 172.21.15.173:0/3304214027 wait complete. 2024-05-27T07:15:24.697 INFO:teuthology.orchestra.run.smithi173.stderr:dumped monmap epoch 2 2024-05-27T07:15:24.884 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:15:24 smithi105 podman[35950]: 2024-05-27 07:15:24.860395946 +0000 UTC m=+0.342492167 container start ac140bd844088e241552e0adb16cd5f33c4259c1c6d8721e0edfbb1ae48164bf (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mon-b, org.label-schema.schema-version=1.0, ceph=True, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, org.label-schema.build-date=20240521, GIT_CLEAN=True, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2) 2024-05-27T07:15:25.019 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:15:24 smithi173 ceph-mon[37000]: pgmap v21: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-27T07:15:25.019 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:15:24 smithi173 ceph-mon[37000]: from='client.? 172.21.15.173:0/3304214027' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-05-27T07:15:25.191 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:15:24 smithi105 ceph-mon[35977]: set uid:gid to 167:167 (ceph:ceph) 2024-05-27T07:15:25.191 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:15:24 smithi105 ceph-mon[35977]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev), process ceph-mon, pid 2 2024-05-27T07:15:25.191 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:15:24 smithi105 ceph-mon[35977]: pidfile_write: ignore empty --pid-file 2024-05-27T07:15:25.191 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:15:24 smithi105 ceph-mon[35977]: load: jerasure load: lrc 2024-05-27T07:15:25.191 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:15:24 smithi105 ceph-mon[35977]: rocksdb: RocksDB version: 7.9.2 2024-05-27T07:15:25.191 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:15:24 smithi105 ceph-mon[35977]: rocksdb: Git sha 0 2024-05-27T07:15:25.192 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:15:24 smithi105 ceph-mon[35977]: rocksdb: Compile date 2024-05-21 07:00:12 2024-05-27T07:15:25.192 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:15:24 smithi105 ceph-mon[35977]: rocksdb: DB SUMMARY 2024-05-27T07:15:25.192 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:15:24 smithi105 ceph-mon[35977]: rocksdb: DB Session ID: UD1MKFBY1JTXT4OBNR14 2024-05-27T07:15:25.192 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:15:24 smithi105 ceph-mon[35977]: rocksdb: CURRENT file: CURRENT 2024-05-27T07:15:25.192 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:15:24 smithi105 ceph-mon[35977]: rocksdb: IDENTITY file: IDENTITY 2024-05-27T07:15:25.192 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:15:24 smithi105 ceph-mon[35977]: rocksdb: MANIFEST file: MANIFEST-000005 size: 59 Bytes 2024-05-27T07:15:25.192 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:15:24 smithi105 ceph-mon[35977]: rocksdb: SST files in /var/lib/ceph/mon/ceph-b/store.db dir, Total Num: 0, files: 2024-05-27T07:15:25.192 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:15:24 smithi105 ceph-mon[35977]: rocksdb: Write Ahead Log file in /var/lib/ceph/mon/ceph-b/store.db: 000004.log size: 636 ; 2024-05-27T07:15:25.192 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:15:24 smithi105 ceph-mon[35977]: rocksdb: Options.error_if_exists: 0 2024-05-27T07:15:25.192 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:15:24 smithi105 ceph-mon[35977]: rocksdb: Options.create_if_missing: 0 2024-05-27T07:15:25.192 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:15:24 smithi105 ceph-mon[35977]: rocksdb: Options.paranoid_checks: 1 2024-05-27T07:15:25.193 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:15:24 smithi105 ceph-mon[35977]: rocksdb: Options.flush_verify_memtable_count: 1 2024-05-27T07:15:25.193 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:15:24 smithi105 ceph-mon[35977]: rocksdb: Options.track_and_verify_wals_in_manifest: 0 2024-05-27T07:15:25.193 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:15:24 smithi105 ceph-mon[35977]: rocksdb: Options.verify_sst_unique_id_in_manifest: 1 2024-05-27T07:15:25.193 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:15:24 smithi105 ceph-mon[35977]: rocksdb: Options.env: 0x55ff365aa500 2024-05-27T07:15:25.193 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:15:24 smithi105 ceph-mon[35977]: rocksdb: Options.fs: PosixFileSystem 2024-05-27T07:15:25.193 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:15:24 smithi105 ceph-mon[35977]: rocksdb: Options.info_log: 0x55ff37419fe0 2024-05-27T07:15:25.193 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:15:24 smithi105 ceph-mon[35977]: rocksdb: Options.max_file_opening_threads: 16 2024-05-27T07:15:25.193 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:15:24 smithi105 ceph-mon[35977]: rocksdb: Options.statistics: (nil) 2024-05-27T07:15:25.193 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:15:24 smithi105 ceph-mon[35977]: rocksdb: Options.use_fsync: 0 2024-05-27T07:15:25.193 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:15:24 smithi105 ceph-mon[35977]: rocksdb: Options.max_log_file_size: 0 2024-05-27T07:15:25.193 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:15:24 smithi105 ceph-mon[35977]: rocksdb: Options.max_manifest_file_size: 1073741824 2024-05-27T07:15:25.193 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:15:24 smithi105 ceph-mon[35977]: rocksdb: Options.log_file_time_to_roll: 0 2024-05-27T07:15:25.193 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:15:24 smithi105 ceph-mon[35977]: rocksdb: Options.keep_log_file_num: 1000 2024-05-27T07:15:25.194 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:15:24 smithi105 ceph-mon[35977]: rocksdb: Options.recycle_log_file_num: 0 2024-05-27T07:15:25.194 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:15:24 smithi105 ceph-mon[35977]: rocksdb: Options.allow_fallocate: 1 2024-05-27T07:15:25.194 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:15:24 smithi105 ceph-mon[35977]: rocksdb: Options.allow_mmap_reads: 0 2024-05-27T07:15:25.194 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:15:24 smithi105 ceph-mon[35977]: rocksdb: Options.allow_mmap_writes: 0 2024-05-27T07:15:25.194 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:15:24 smithi105 ceph-mon[35977]: rocksdb: Options.use_direct_reads: 0 2024-05-27T07:15:25.194 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:15:24 smithi105 ceph-mon[35977]: rocksdb: Options.use_direct_io_for_flush_and_compaction: 0 2024-05-27T07:15:25.194 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:15:24 smithi105 ceph-mon[35977]: rocksdb: Options.create_missing_column_families: 0 2024-05-27T07:15:25.194 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:15:24 smithi105 ceph-mon[35977]: rocksdb: Options.db_log_dir: 2024-05-27T07:15:25.194 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:15:24 smithi105 ceph-mon[35977]: rocksdb: Options.wal_dir: 2024-05-27T07:15:25.194 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:15:24 smithi105 ceph-mon[35977]: rocksdb: Options.table_cache_numshardbits: 6 2024-05-27T07:15:25.194 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:15:24 smithi105 ceph-mon[35977]: rocksdb: Options.WAL_ttl_seconds: 0 2024-05-27T07:15:25.195 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:15:24 smithi105 ceph-mon[35977]: rocksdb: Options.WAL_size_limit_MB: 0 2024-05-27T07:15:25.195 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:15:24 smithi105 ceph-mon[35977]: rocksdb: Options.max_write_batch_group_size_bytes: 1048576 2024-05-27T07:15:25.195 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:15:24 smithi105 ceph-mon[35977]: rocksdb: Options.manifest_preallocation_size: 4194304 2024-05-27T07:15:25.195 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:15:24 smithi105 ceph-mon[35977]: rocksdb: Options.is_fd_close_on_exec: 1 2024-05-27T07:15:25.195 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:15:24 smithi105 ceph-mon[35977]: rocksdb: Options.advise_random_on_open: 1 2024-05-27T07:15:25.195 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:15:24 smithi105 ceph-mon[35977]: rocksdb: Options.db_write_buffer_size: 0 2024-05-27T07:15:25.195 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:15:24 smithi105 ceph-mon[35977]: rocksdb: Options.write_buffer_manager: 0x55ff373ea8c0 2024-05-27T07:15:25.195 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:15:24 smithi105 ceph-mon[35977]: rocksdb: Options.access_hint_on_compaction_start: 1 2024-05-27T07:15:25.195 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:15:24 smithi105 ceph-mon[35977]: rocksdb: Options.random_access_max_buffer_size: 1048576 2024-05-27T07:15:25.195 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:15:24 smithi105 ceph-mon[35977]: rocksdb: Options.use_adaptive_mutex: 0 2024-05-27T07:15:25.195 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:15:24 smithi105 ceph-mon[35977]: rocksdb: Options.rate_limiter: (nil) 2024-05-27T07:15:25.195 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:15:24 smithi105 ceph-mon[35977]: rocksdb: Options.sst_file_manager.rate_bytes_per_sec: 0 2024-05-27T07:15:25.195 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:15:24 smithi105 ceph-mon[35977]: rocksdb: Options.wal_recovery_mode: 2 2024-05-27T07:15:25.196 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:15:24 smithi105 ceph-mon[35977]: rocksdb: Options.enable_thread_tracking: 0 2024-05-27T07:15:25.196 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:15:24 smithi105 ceph-mon[35977]: rocksdb: Options.enable_pipelined_write: 0 2024-05-27T07:15:25.196 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:15:24 smithi105 ceph-mon[35977]: rocksdb: Options.unordered_write: 0 2024-05-27T07:15:25.196 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:15:24 smithi105 ceph-mon[35977]: rocksdb: Options.allow_concurrent_memtable_write: 1 2024-05-27T07:15:25.196 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:15:24 smithi105 ceph-mon[35977]: rocksdb: Options.enable_write_thread_adaptive_yield: 1 2024-05-27T07:15:25.196 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:15:24 smithi105 ceph-mon[35977]: rocksdb: Options.write_thread_max_yield_usec: 100 2024-05-27T07:15:25.196 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:15:24 smithi105 ceph-mon[35977]: rocksdb: Options.write_thread_slow_yield_usec: 3 2024-05-27T07:15:25.196 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:15:24 smithi105 ceph-mon[35977]: rocksdb: Options.row_cache: None 2024-05-27T07:15:25.196 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:15:24 smithi105 ceph-mon[35977]: rocksdb: Options.wal_filter: None 2024-05-27T07:15:25.196 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:15:24 smithi105 ceph-mon[35977]: rocksdb: Options.avoid_flush_during_recovery: 0 2024-05-27T07:15:25.196 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:15:24 smithi105 ceph-mon[35977]: rocksdb: Options.allow_ingest_behind: 0 2024-05-27T07:15:25.196 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:15:24 smithi105 ceph-mon[35977]: rocksdb: Options.two_write_queues: 0 2024-05-27T07:15:25.197 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:15:24 smithi105 ceph-mon[35977]: rocksdb: Options.manual_wal_flush: 0 2024-05-27T07:15:25.197 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:15:24 smithi105 ceph-mon[35977]: rocksdb: Options.wal_compression: 0 2024-05-27T07:15:25.197 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:15:24 smithi105 ceph-mon[35977]: rocksdb: Options.atomic_flush: 0 2024-05-27T07:15:25.197 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:15:24 smithi105 ceph-mon[35977]: rocksdb: Options.avoid_unnecessary_blocking_io: 0 2024-05-27T07:15:25.197 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:15:24 smithi105 ceph-mon[35977]: rocksdb: Options.persist_stats_to_disk: 0 2024-05-27T07:15:25.197 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:15:24 smithi105 ceph-mon[35977]: rocksdb: Options.write_dbid_to_manifest: 0 2024-05-27T07:15:25.197 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:15:24 smithi105 ceph-mon[35977]: rocksdb: Options.log_readahead_size: 0 2024-05-27T07:15:25.197 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:15:24 smithi105 ceph-mon[35977]: rocksdb: Options.file_checksum_gen_factory: Unknown 2024-05-27T07:15:25.197 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:15:24 smithi105 ceph-mon[35977]: rocksdb: Options.best_efforts_recovery: 0 2024-05-27T07:15:25.197 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:15:24 smithi105 ceph-mon[35977]: rocksdb: Options.max_bgerror_resume_count: 2147483647 2024-05-27T07:15:25.197 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:15:24 smithi105 ceph-mon[35977]: rocksdb: Options.bgerror_resume_retry_interval: 1000000 2024-05-27T07:15:25.197 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:15:24 smithi105 ceph-mon[35977]: rocksdb: Options.allow_data_in_errors: 0 2024-05-27T07:15:25.197 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:15:24 smithi105 ceph-mon[35977]: rocksdb: Options.db_host_id: __hostname__ 2024-05-27T07:15:25.197 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:15:24 smithi105 ceph-mon[35977]: rocksdb: Options.enforce_single_del_contracts: true 2024-05-27T07:15:25.199 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:15:24 smithi105 ceph-mon[35977]: rocksdb: Options.max_background_jobs: 2 2024-05-27T07:15:25.199 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:15:24 smithi105 ceph-mon[35977]: rocksdb: Options.max_background_compactions: -1 2024-05-27T07:15:25.199 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:15:24 smithi105 ceph-mon[35977]: rocksdb: Options.max_subcompactions: 1 2024-05-27T07:15:25.199 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:15:24 smithi105 ceph-mon[35977]: rocksdb: Options.avoid_flush_during_shutdown: 0 2024-05-27T07:15:25.199 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:15:24 smithi105 ceph-mon[35977]: rocksdb: Options.writable_file_max_buffer_size: 1048576 2024-05-27T07:15:25.199 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:15:24 smithi105 ceph-mon[35977]: rocksdb: Options.delayed_write_rate : 16777216 2024-05-27T07:15:25.200 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:15:24 smithi105 ceph-mon[35977]: rocksdb: Options.max_total_wal_size: 0 2024-05-27T07:15:25.200 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:15:24 smithi105 ceph-mon[35977]: rocksdb: Options.delete_obsolete_files_period_micros: 21600000000 2024-05-27T07:15:25.200 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:15:24 smithi105 ceph-mon[35977]: rocksdb: Options.stats_dump_period_sec: 600 2024-05-27T07:15:25.200 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:15:24 smithi105 ceph-mon[35977]: rocksdb: Options.stats_persist_period_sec: 600 2024-05-27T07:15:25.200 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:15:24 smithi105 ceph-mon[35977]: rocksdb: Options.stats_history_buffer_size: 1048576 2024-05-27T07:15:25.200 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:15:24 smithi105 ceph-mon[35977]: rocksdb: Options.max_open_files: -1 2024-05-27T07:15:25.200 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:15:24 smithi105 ceph-mon[35977]: rocksdb: Options.bytes_per_sync: 0 2024-05-27T07:15:25.200 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:15:24 smithi105 ceph-mon[35977]: rocksdb: Options.wal_bytes_per_sync: 0 2024-05-27T07:15:25.200 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:15:24 smithi105 ceph-mon[35977]: rocksdb: Options.strict_bytes_per_sync: 0 2024-05-27T07:15:25.200 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:15:24 smithi105 ceph-mon[35977]: rocksdb: Options.compaction_readahead_size: 0 2024-05-27T07:15:25.200 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:15:24 smithi105 ceph-mon[35977]: rocksdb: Options.max_background_flushes: -1 2024-05-27T07:15:25.200 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:15:24 smithi105 ceph-mon[35977]: rocksdb: Compression algorithms supported: 2024-05-27T07:15:25.201 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:15:24 smithi105 ceph-mon[35977]: rocksdb: kZSTDNotFinalCompression supported: 0 2024-05-27T07:15:25.201 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:15:24 smithi105 ceph-mon[35977]: rocksdb: kZSTD supported: 0 2024-05-27T07:15:25.201 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:15:24 smithi105 ceph-mon[35977]: rocksdb: kXpressCompression supported: 0 2024-05-27T07:15:25.201 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:15:24 smithi105 ceph-mon[35977]: rocksdb: kLZ4HCCompression supported: 1 2024-05-27T07:15:25.201 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:15:24 smithi105 ceph-mon[35977]: rocksdb: kZlibCompression supported: 1 2024-05-27T07:15:25.201 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:15:24 smithi105 ceph-mon[35977]: rocksdb: kSnappyCompression supported: 1 2024-05-27T07:15:25.201 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:15:24 smithi105 ceph-mon[35977]: rocksdb: kLZ4Compression supported: 1 2024-05-27T07:15:25.201 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:15:24 smithi105 ceph-mon[35977]: rocksdb: kBZip2Compression supported: 0 2024-05-27T07:15:25.201 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:15:24 smithi105 ceph-mon[35977]: rocksdb: Fast CRC32 supported: Supported on x86 2024-05-27T07:15:25.201 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:15:24 smithi105 ceph-mon[35977]: rocksdb: DMutex implementation: pthread_mutex_t 2024-05-27T07:15:25.201 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:15:24 smithi105 ceph-mon[35977]: rocksdb: [db/version_set.cc:5527] Recovering from manifest file: /var/lib/ceph/mon/ceph-b/store.db/MANIFEST-000005 2024-05-27T07:15:25.201 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:15:24 smithi105 ceph-mon[35977]: rocksdb: [db/column_family.cc:630] --------------- Options for column family [default]: 2024-05-27T07:15:25.201 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:15:24 smithi105 ceph-mon[35977]: rocksdb: Options.comparator: leveldb.BytewiseComparator 2024-05-27T07:15:25.202 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:15:24 smithi105 ceph-mon[35977]: rocksdb: Options.merge_operator: 2024-05-27T07:15:25.202 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:15:24 smithi105 ceph-mon[35977]: rocksdb: Options.compaction_filter: None 2024-05-27T07:15:25.202 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:15:24 smithi105 ceph-mon[35977]: rocksdb: Options.compaction_filter_factory: None 2024-05-27T07:15:25.202 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:15:24 smithi105 ceph-mon[35977]: rocksdb: Options.sst_partitioner_factory: None 2024-05-27T07:15:25.202 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:15:24 smithi105 ceph-mon[35977]: rocksdb: Options.memtable_factory: SkipListFactory 2024-05-27T07:15:25.202 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:15:24 smithi105 ceph-mon[35977]: rocksdb: Options.table_factory: BlockBasedTable 2024-05-27T07:15:25.202 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:15:24 smithi105 ceph-mon[35977]: rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x55ff375201c0) 2024-05-27T07:15:25.202 INFO:journalctl@ceph.mon.b.smithi105.stdout: cache_index_and_filter_blocks: 1 2024-05-27T07:15:25.202 INFO:journalctl@ceph.mon.b.smithi105.stdout: cache_index_and_filter_blocks_with_high_priority: 0 2024-05-27T07:15:25.202 INFO:journalctl@ceph.mon.b.smithi105.stdout: pin_l0_filter_and_index_blocks_in_cache: 0 2024-05-27T07:15:25.202 INFO:journalctl@ceph.mon.b.smithi105.stdout: pin_top_level_index_and_filter: 1 2024-05-27T07:15:25.202 INFO:journalctl@ceph.mon.b.smithi105.stdout: index_type: 0 2024-05-27T07:15:25.202 INFO:journalctl@ceph.mon.b.smithi105.stdout: data_block_index_type: 0 2024-05-27T07:15:25.203 INFO:journalctl@ceph.mon.b.smithi105.stdout: index_shortening: 1 2024-05-27T07:15:25.203 INFO:journalctl@ceph.mon.b.smithi105.stdout: data_block_hash_table_util_ratio: 0.750000 2024-05-27T07:15:25.203 INFO:journalctl@ceph.mon.b.smithi105.stdout: checksum: 4 2024-05-27T07:15:25.203 INFO:journalctl@ceph.mon.b.smithi105.stdout: no_block_cache: 0 2024-05-27T07:15:25.203 INFO:journalctl@ceph.mon.b.smithi105.stdout: block_cache: 0x55ff3741d1f0 2024-05-27T07:15:25.203 INFO:journalctl@ceph.mon.b.smithi105.stdout: block_cache_name: BinnedLRUCache 2024-05-27T07:15:25.203 INFO:journalctl@ceph.mon.b.smithi105.stdout: block_cache_options: 2024-05-27T07:15:25.203 INFO:journalctl@ceph.mon.b.smithi105.stdout: capacity : 536870912 2024-05-27T07:15:25.203 INFO:journalctl@ceph.mon.b.smithi105.stdout: num_shard_bits : 4 2024-05-27T07:15:25.203 INFO:journalctl@ceph.mon.b.smithi105.stdout: strict_capacity_limit : 0 2024-05-27T07:15:25.203 INFO:journalctl@ceph.mon.b.smithi105.stdout: high_pri_pool_ratio: 0.000 2024-05-27T07:15:25.204 INFO:journalctl@ceph.mon.b.smithi105.stdout: block_cache_compressed: (nil) 2024-05-27T07:15:25.204 INFO:journalctl@ceph.mon.b.smithi105.stdout: persistent_cache: (nil) 2024-05-27T07:15:25.204 INFO:journalctl@ceph.mon.b.smithi105.stdout: block_size: 4096 2024-05-27T07:15:25.204 INFO:journalctl@ceph.mon.b.smithi105.stdout: block_size_deviation: 10 2024-05-27T07:15:25.204 INFO:journalctl@ceph.mon.b.smithi105.stdout: block_restart_interval: 16 2024-05-27T07:15:25.204 INFO:journalctl@ceph.mon.b.smithi105.stdout: index_block_restart_interval: 1 2024-05-27T07:15:25.204 INFO:journalctl@ceph.mon.b.smithi105.stdout: metadata_block_size: 4096 2024-05-27T07:15:25.204 INFO:journalctl@ceph.mon.b.smithi105.stdout: partition_filters: 0 2024-05-27T07:15:25.204 INFO:journalctl@ceph.mon.b.smithi105.stdout: use_delta_encoding: 1 2024-05-27T07:15:25.204 INFO:journalctl@ceph.mon.b.smithi105.stdout: filter_policy: bloomfilter 2024-05-27T07:15:25.204 INFO:journalctl@ceph.mon.b.smithi105.stdout: whole_key_filtering: 1 2024-05-27T07:15:25.204 INFO:journalctl@ceph.mon.b.smithi105.stdout: verify_compression: 0 2024-05-27T07:15:25.205 INFO:journalctl@ceph.mon.b.smithi105.stdout: read_amp_bytes_per_bit: 0 2024-05-27T07:15:25.205 INFO:journalctl@ceph.mon.b.smithi105.stdout: format_version: 5 2024-05-27T07:15:25.205 INFO:journalctl@ceph.mon.b.smithi105.stdout: enable_index_compression: 1 2024-05-27T07:15:25.205 INFO:journalctl@ceph.mon.b.smithi105.stdout: block_align: 0 2024-05-27T07:15:25.205 INFO:journalctl@ceph.mon.b.smithi105.stdout: max_auto_readahead_size: 262144 2024-05-27T07:15:25.205 INFO:journalctl@ceph.mon.b.smithi105.stdout: prepopulate_block_cache: 0 2024-05-27T07:15:25.205 INFO:journalctl@ceph.mon.b.smithi105.stdout: initial_auto_readahead_size: 8192 2024-05-27T07:15:25.205 INFO:journalctl@ceph.mon.b.smithi105.stdout: num_file_reads_for_auto_readahead: 2 2024-05-27T07:15:25.205 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:15:24 smithi105 ceph-mon[35977]: rocksdb: Options.write_buffer_size: 33554432 2024-05-27T07:15:25.205 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:15:24 smithi105 ceph-mon[35977]: rocksdb: Options.max_write_buffer_number: 2 2024-05-27T07:15:25.205 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:15:24 smithi105 ceph-mon[35977]: rocksdb: Options.compression: NoCompression 2024-05-27T07:15:25.205 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:15:24 smithi105 ceph-mon[35977]: rocksdb: Options.bottommost_compression: Disabled 2024-05-27T07:15:25.205 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:15:24 smithi105 ceph-mon[35977]: rocksdb: Options.prefix_extractor: nullptr 2024-05-27T07:15:25.206 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:15:24 smithi105 ceph-mon[35977]: rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr 2024-05-27T07:15:25.206 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:15:24 smithi105 ceph-mon[35977]: rocksdb: Options.num_levels: 7 2024-05-27T07:15:25.206 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:15:24 smithi105 ceph-mon[35977]: rocksdb: Options.min_write_buffer_number_to_merge: 1 2024-05-27T07:15:25.206 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:15:24 smithi105 ceph-mon[35977]: rocksdb: Options.max_write_buffer_number_to_maintain: 0 2024-05-27T07:15:25.206 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:15:24 smithi105 ceph-mon[35977]: rocksdb: Options.max_write_buffer_size_to_maintain: 0 2024-05-27T07:15:25.206 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:15:24 smithi105 ceph-mon[35977]: rocksdb: Options.bottommost_compression_opts.window_bits: -14 2024-05-27T07:15:25.206 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:15:24 smithi105 ceph-mon[35977]: rocksdb: Options.bottommost_compression_opts.level: 32767 2024-05-27T07:15:25.206 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:15:24 smithi105 ceph-mon[35977]: rocksdb: Options.bottommost_compression_opts.strategy: 0 2024-05-27T07:15:25.206 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:15:24 smithi105 ceph-mon[35977]: rocksdb: Options.bottommost_compression_opts.max_dict_bytes: 0 2024-05-27T07:15:25.206 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:15:24 smithi105 ceph-mon[35977]: rocksdb: Options.bottommost_compression_opts.zstd_max_train_bytes: 0 2024-05-27T07:15:25.206 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:15:24 smithi105 ceph-mon[35977]: rocksdb: Options.bottommost_compression_opts.parallel_threads: 1 2024-05-27T07:15:25.207 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:15:24 smithi105 ceph-mon[35977]: rocksdb: Options.bottommost_compression_opts.enabled: false 2024-05-27T07:15:25.207 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:15:24 smithi105 ceph-mon[35977]: rocksdb: Options.bottommost_compression_opts.max_dict_buffer_bytes: 0 2024-05-27T07:15:25.207 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:15:24 smithi105 ceph-mon[35977]: rocksdb: Options.bottommost_compression_opts.use_zstd_dict_trainer: true 2024-05-27T07:15:25.207 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:15:24 smithi105 ceph-mon[35977]: rocksdb: Options.compression_opts.window_bits: -14 2024-05-27T07:15:25.207 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:15:24 smithi105 ceph-mon[35977]: rocksdb: Options.compression_opts.level: 32767 2024-05-27T07:15:25.207 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:15:24 smithi105 ceph-mon[35977]: rocksdb: Options.compression_opts.strategy: 0 2024-05-27T07:15:25.207 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:15:24 smithi105 ceph-mon[35977]: rocksdb: Options.compression_opts.max_dict_bytes: 0 2024-05-27T07:15:25.207 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:15:24 smithi105 ceph-mon[35977]: rocksdb: Options.compression_opts.zstd_max_train_bytes: 0 2024-05-27T07:15:25.207 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:15:24 smithi105 ceph-mon[35977]: rocksdb: Options.compression_opts.use_zstd_dict_trainer: true 2024-05-27T07:15:25.207 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:15:24 smithi105 ceph-mon[35977]: rocksdb: Options.compression_opts.parallel_threads: 1 2024-05-27T07:15:25.207 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:15:24 smithi105 ceph-mon[35977]: rocksdb: Options.compression_opts.enabled: false 2024-05-27T07:15:25.208 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:15:24 smithi105 ceph-mon[35977]: rocksdb: Options.compression_opts.max_dict_buffer_bytes: 0 2024-05-27T07:15:25.208 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:15:24 smithi105 ceph-mon[35977]: rocksdb: Options.level0_file_num_compaction_trigger: 4 2024-05-27T07:15:25.208 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:15:24 smithi105 ceph-mon[35977]: rocksdb: Options.level0_slowdown_writes_trigger: 20 2024-05-27T07:15:25.208 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:15:24 smithi105 ceph-mon[35977]: rocksdb: Options.level0_stop_writes_trigger: 36 2024-05-27T07:15:25.208 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:15:24 smithi105 ceph-mon[35977]: rocksdb: Options.target_file_size_base: 67108864 2024-05-27T07:15:25.208 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:15:24 smithi105 ceph-mon[35977]: rocksdb: Options.target_file_size_multiplier: 1 2024-05-27T07:15:25.208 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:15:24 smithi105 ceph-mon[35977]: rocksdb: Options.max_bytes_for_level_base: 268435456 2024-05-27T07:15:25.208 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:15:24 smithi105 ceph-mon[35977]: rocksdb: Options.level_compaction_dynamic_level_bytes: 1 2024-05-27T07:15:25.208 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:15:24 smithi105 ceph-mon[35977]: rocksdb: Options.max_bytes_for_level_multiplier: 10.000000 2024-05-27T07:15:25.208 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:15:24 smithi105 ceph-mon[35977]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1 2024-05-27T07:15:25.208 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:15:24 smithi105 ceph-mon[35977]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1 2024-05-27T07:15:25.208 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:15:24 smithi105 ceph-mon[35977]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1 2024-05-27T07:15:25.209 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:15:24 smithi105 ceph-mon[35977]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1 2024-05-27T07:15:25.209 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:15:24 smithi105 ceph-mon[35977]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1 2024-05-27T07:15:25.209 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:15:24 smithi105 ceph-mon[35977]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1 2024-05-27T07:15:25.209 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:15:24 smithi105 ceph-mon[35977]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1 2024-05-27T07:15:25.209 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:15:24 smithi105 ceph-mon[35977]: rocksdb: Options.max_sequential_skip_in_iterations: 8 2024-05-27T07:15:25.209 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:15:24 smithi105 ceph-mon[35977]: rocksdb: Options.max_compaction_bytes: 1677721600 2024-05-27T07:15:25.209 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:15:24 smithi105 ceph-mon[35977]: rocksdb: Options.ignore_max_compaction_bytes_for_input: true 2024-05-27T07:15:25.209 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:15:24 smithi105 ceph-mon[35977]: rocksdb: Options.arena_block_size: 1048576 2024-05-27T07:15:25.209 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:15:24 smithi105 ceph-mon[35977]: rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736 2024-05-27T07:15:25.209 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:15:24 smithi105 ceph-mon[35977]: rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944 2024-05-27T07:15:25.209 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:15:24 smithi105 ceph-mon[35977]: rocksdb: Options.disable_auto_compactions: 0 2024-05-27T07:15:25.209 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:15:24 smithi105 ceph-mon[35977]: rocksdb: Options.compaction_style: kCompactionStyleLevel 2024-05-27T07:15:25.210 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:15:24 smithi105 ceph-mon[35977]: rocksdb: Options.compaction_pri: kMinOverlappingRatio 2024-05-27T07:15:25.210 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:15:24 smithi105 ceph-mon[35977]: rocksdb: Options.compaction_options_universal.size_ratio: 1 2024-05-27T07:15:25.210 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:15:24 smithi105 ceph-mon[35977]: rocksdb: Options.compaction_options_universal.min_merge_width: 2 2024-05-27T07:15:25.210 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:15:24 smithi105 ceph-mon[35977]: rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295 2024-05-27T07:15:25.210 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:15:24 smithi105 ceph-mon[35977]: rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200 2024-05-27T07:15:25.210 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:15:24 smithi105 ceph-mon[35977]: rocksdb: Options.compaction_options_universal.compression_size_percent: -1 2024-05-27T07:15:25.210 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:15:24 smithi105 ceph-mon[35977]: rocksdb: Options.compaction_options_universal.stop_style: kCompactionStopStyleTotalSize 2024-05-27T07:15:25.210 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:15:24 smithi105 ceph-mon[35977]: rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824 2024-05-27T07:15:25.210 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:15:24 smithi105 ceph-mon[35977]: rocksdb: Options.compaction_options_fifo.allow_compaction: 0 2024-05-27T07:15:25.210 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:15:24 smithi105 ceph-mon[35977]: rocksdb: Options.table_properties_collectors: CompactOnDeletionCollector (Sliding window size = 32768 Deletion trigger = 16384 Deletion ratio = 0); 2024-05-27T07:15:25.210 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:15:24 smithi105 ceph-mon[35977]: rocksdb: Options.inplace_update_support: 0 2024-05-27T07:15:25.210 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:15:24 smithi105 ceph-mon[35977]: rocksdb: Options.inplace_update_num_locks: 10000 2024-05-27T07:15:25.211 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:15:24 smithi105 ceph-mon[35977]: rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000 2024-05-27T07:15:25.211 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:15:24 smithi105 ceph-mon[35977]: rocksdb: Options.memtable_whole_key_filtering: 0 2024-05-27T07:15:25.211 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:15:24 smithi105 ceph-mon[35977]: rocksdb: Options.memtable_huge_page_size: 0 2024-05-27T07:15:25.211 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:15:24 smithi105 ceph-mon[35977]: rocksdb: Options.bloom_locality: 0 2024-05-27T07:15:25.211 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:15:24 smithi105 ceph-mon[35977]: rocksdb: Options.max_successive_merges: 0 2024-05-27T07:15:25.211 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:15:24 smithi105 ceph-mon[35977]: rocksdb: Options.optimize_filters_for_hits: 0 2024-05-27T07:15:25.211 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:15:24 smithi105 ceph-mon[35977]: rocksdb: Options.paranoid_file_checks: 0 2024-05-27T07:15:25.211 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:15:24 smithi105 ceph-mon[35977]: rocksdb: Options.force_consistency_checks: 1 2024-05-27T07:15:25.211 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:15:24 smithi105 ceph-mon[35977]: rocksdb: Options.report_bg_io_stats: 0 2024-05-27T07:15:25.211 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:15:24 smithi105 ceph-mon[35977]: rocksdb: Options.ttl: 2592000 2024-05-27T07:15:25.211 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:15:24 smithi105 ceph-mon[35977]: rocksdb: Options.periodic_compaction_seconds: 0 2024-05-27T07:15:25.211 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:15:24 smithi105 ceph-mon[35977]: rocksdb: Options.preclude_last_level_data_seconds: 0 2024-05-27T07:15:25.212 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:15:24 smithi105 ceph-mon[35977]: rocksdb: Options.preserve_internal_time_seconds: 0 2024-05-27T07:15:25.212 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:15:24 smithi105 ceph-mon[35977]: rocksdb: Options.enable_blob_files: false 2024-05-27T07:15:25.212 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:15:24 smithi105 ceph-mon[35977]: rocksdb: Options.min_blob_size: 0 2024-05-27T07:15:25.212 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:15:24 smithi105 ceph-mon[35977]: rocksdb: Options.blob_file_size: 268435456 2024-05-27T07:15:25.212 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:15:24 smithi105 ceph-mon[35977]: rocksdb: Options.blob_compression_type: NoCompression 2024-05-27T07:15:25.212 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:15:24 smithi105 ceph-mon[35977]: rocksdb: Options.enable_blob_garbage_collection: false 2024-05-27T07:15:25.212 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:15:24 smithi105 ceph-mon[35977]: rocksdb: Options.blob_garbage_collection_age_cutoff: 0.250000 2024-05-27T07:15:25.212 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:15:24 smithi105 ceph-mon[35977]: rocksdb: Options.blob_garbage_collection_force_threshold: 1.000000 2024-05-27T07:15:25.212 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:15:24 smithi105 ceph-mon[35977]: rocksdb: Options.blob_compaction_readahead_size: 0 2024-05-27T07:15:25.212 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:15:24 smithi105 ceph-mon[35977]: rocksdb: Options.blob_file_starting_level: 0 2024-05-27T07:15:25.212 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:15:24 smithi105 ceph-mon[35977]: rocksdb: Options.experimental_mempurge_threshold: 0.000000 2024-05-27T07:15:25.212 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:15:24 smithi105 ceph-mon[35977]: rocksdb: [db/version_set.cc:5566] Recovered from manifest file:/var/lib/ceph/mon/ceph-b/store.db/MANIFEST-000005 succeeded,manifest_file_number is 5, next_file_number is 7, last_sequence is 0, log_number is 0,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 0 2024-05-27T07:15:25.212 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:15:24 smithi105 ceph-mon[35977]: rocksdb: [db/version_set.cc:5581] Column family [default] (ID 0), log number is 0 2024-05-27T07:15:25.213 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:15:24 smithi105 ceph-mon[35977]: rocksdb: [db/db_impl/db_impl_open.cc:539] DB ID: cfa3df19-4061-4571-bf42-f547387b6c81 2024-05-27T07:15:25.213 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:15:24 smithi105 ceph-mon[35977]: rocksdb: EVENT_LOG_v1 {"time_micros": 1716794124901269, "job": 1, "event": "recovery_started", "wal_files": [4]} 2024-05-27T07:15:25.213 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:15:24 smithi105 ceph-mon[35977]: rocksdb: [db/db_impl/db_impl_open.cc:1043] Recovering log #4 mode 2 2024-05-27T07:15:25.213 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:15:24 smithi105 ceph-mon[35977]: rocksdb: EVENT_LOG_v1 {"time_micros": 1716794124901700, "cf_name": "default", "job": 1, "event": "table_file_creation", "file_number": 8, "file_size": 1773, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 1, "largest_seqno": 5, "table_properties": {"data_size": 648, "index_size": 31, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 1, "index_value_is_delta_encoded": 1, "filter_size": 69, "raw_key_size": 115, "raw_average_key_size": 23, "raw_value_size": 526, "raw_average_value_size": 105, "num_data_blocks": 1, "num_entries": 5, "num_filter_entries": 5, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 0, "filter_policy": "bloomfilter", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "", "prefix_extractor_name": "nullptr", "property_collectors": "[CompactOnDeletionCollector]", "compression": "NoCompression", "compression_options": "window_bits=-14; level=32767; strategy=0; max_dict_bytes=0; zstd_max_train_bytes=0; enabled=0; max_dict_buffer_bytes=0; use_zstd_dict_trainer=1; ", "creation_time": 1716794124, "oldest_key_time": 0, "file_creation_time": 0, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "cfa3df19-4061-4571-bf42-f547387b6c81", "db_session_id": "UD1MKFBY1JTXT4OBNR14", "orig_file_number": 8, "seqno_to_time_mapping": "N/A"}} 2024-05-27T07:15:25.213 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:15:24 smithi105 ceph-mon[35977]: rocksdb: EVENT_LOG_v1 {"time_micros": 1716794124901793, "job": 1, "event": "recovery_finished"} 2024-05-27T07:15:25.213 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:15:24 smithi105 ceph-mon[35977]: rocksdb: [db/version_set.cc:5047] Creating manifest 10 2024-05-27T07:15:25.213 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:15:24 smithi105 ceph-mon[35977]: rocksdb: [file/delete_scheduler.cc:74] Deleted file /var/lib/ceph/mon/ceph-b/store.db/000004.log immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024-05-27T07:15:25.213 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:15:24 smithi105 ceph-mon[35977]: rocksdb: [db/db_impl/db_impl_open.cc:1987] SstFileManager instance 0x55ff37401c00 2024-05-27T07:15:25.213 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:15:24 smithi105 ceph-mon[35977]: rocksdb: DB pointer 0x55ff3c1f4000 2024-05-27T07:15:25.213 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:15:24 smithi105 ceph-mon[35977]: rocksdb: [db/db_impl/db_impl.cc:1109] ------- DUMPING STATS ------- 2024-05-27T07:15:25.213 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:15:24 smithi105 ceph-mon[35977]: rocksdb: [db/db_impl/db_impl.cc:1111] 2024-05-27T07:15:25.213 INFO:journalctl@ceph.mon.b.smithi105.stdout: ** DB Stats ** 2024-05-27T07:15:25.213 INFO:journalctl@ceph.mon.b.smithi105.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-05-27T07:15:25.214 INFO:journalctl@ceph.mon.b.smithi105.stdout: Cumulative writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 GB, 0.00 MB/s 2024-05-27T07:15:25.214 INFO:journalctl@ceph.mon.b.smithi105.stdout: Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-05-27T07:15:25.214 INFO:journalctl@ceph.mon.b.smithi105.stdout: Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent 2024-05-27T07:15:25.214 INFO:journalctl@ceph.mon.b.smithi105.stdout: Interval writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 MB, 0.00 MB/s 2024-05-27T07:15:25.214 INFO:journalctl@ceph.mon.b.smithi105.stdout: Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-05-27T07:15:25.214 INFO:journalctl@ceph.mon.b.smithi105.stdout: Interval stall: 00:00:0.000 H:M:S, 0.0 percent 2024-05-27T07:15:25.214 INFO:journalctl@ceph.mon.b.smithi105.stdout: 2024-05-27T07:15:25.214 INFO:journalctl@ceph.mon.b.smithi105.stdout: ** Compaction Stats [default] ** 2024-05-27T07:15:25.214 INFO:journalctl@ceph.mon.b.smithi105.stdout: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-05-27T07:15:25.214 INFO:journalctl@ceph.mon.b.smithi105.stdout: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ 2024-05-27T07:15:25.214 INFO:journalctl@ceph.mon.b.smithi105.stdout: L0 1/0 1.73 KB 0.2 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 4.2 0.00 0.00 1 0.000 0 0 0.0 0.0 2024-05-27T07:15:25.214 INFO:journalctl@ceph.mon.b.smithi105.stdout: Sum 1/0 1.73 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 4.2 0.00 0.00 1 0.000 0 0 0.0 0.0 2024-05-27T07:15:25.214 INFO:journalctl@ceph.mon.b.smithi105.stdout: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 4.2 0.00 0.00 1 0.000 0 0 0.0 0.0 2024-05-27T07:15:25.214 INFO:journalctl@ceph.mon.b.smithi105.stdout: 2024-05-27T07:15:25.215 INFO:journalctl@ceph.mon.b.smithi105.stdout: ** Compaction Stats [default] ** 2024-05-27T07:15:25.215 INFO:journalctl@ceph.mon.b.smithi105.stdout: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-05-27T07:15:25.215 INFO:journalctl@ceph.mon.b.smithi105.stdout: --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-05-27T07:15:25.215 INFO:journalctl@ceph.mon.b.smithi105.stdout: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 4.2 0.00 0.00 1 0.000 0 0 0.0 0.0 2024-05-27T07:15:25.215 INFO:journalctl@ceph.mon.b.smithi105.stdout: 2024-05-27T07:15:25.215 INFO:journalctl@ceph.mon.b.smithi105.stdout: Blob file count: 0, total size: 0.0 GB, garbage size: 0.0 GB, space amp: 0.0 2024-05-27T07:15:25.215 INFO:journalctl@ceph.mon.b.smithi105.stdout: 2024-05-27T07:15:25.215 INFO:journalctl@ceph.mon.b.smithi105.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-05-27T07:15:25.215 INFO:journalctl@ceph.mon.b.smithi105.stdout: Flush(GB): cumulative 0.000, interval 0.000 2024-05-27T07:15:25.215 INFO:journalctl@ceph.mon.b.smithi105.stdout: AddFile(GB): cumulative 0.000, interval 0.000 2024-05-27T07:15:25.215 INFO:journalctl@ceph.mon.b.smithi105.stdout: AddFile(Total Files): cumulative 0, interval 0 2024-05-27T07:15:25.215 INFO:journalctl@ceph.mon.b.smithi105.stdout: AddFile(L0 Files): cumulative 0, interval 0 2024-05-27T07:15:25.215 INFO:journalctl@ceph.mon.b.smithi105.stdout: AddFile(Keys): cumulative 0, interval 0 2024-05-27T07:15:25.215 INFO:journalctl@ceph.mon.b.smithi105.stdout: Cumulative compaction: 0.00 GB write, 0.63 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-05-27T07:15:25.215 INFO:journalctl@ceph.mon.b.smithi105.stdout: Interval compaction: 0.00 GB write, 0.63 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-05-27T07:15:25.216 INFO:journalctl@ceph.mon.b.smithi105.stdout: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2024-05-27T07:15:25.216 INFO:journalctl@ceph.mon.b.smithi105.stdout: Block cache BinnedLRUCache@0x55ff3741d1f0#2 capacity: 512.00 MB usage: 0.23 KB table_size: 0 occupancy: 18446744073709551615 collections: 1 last_copies: 0 last_secs: 1e-05 secs_since: 0 2024-05-27T07:15:25.216 INFO:journalctl@ceph.mon.b.smithi105.stdout: Block cache entry stats(count,size,portion): FilterBlock(1,0.12 KB,2.23517e-05%) IndexBlock(1,0.11 KB,2.08616e-05%) Misc(1,0.00 KB,0%) 2024-05-27T07:15:25.216 INFO:journalctl@ceph.mon.b.smithi105.stdout: 2024-05-27T07:15:25.216 INFO:journalctl@ceph.mon.b.smithi105.stdout: ** File Read Latency Histogram By Level [default] ** 2024-05-27T07:15:25.216 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:15:24 smithi105 ceph-mon[35977]: mon.b does not exist in monmap, will attempt to join an existing cluster 2024-05-27T07:15:25.216 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:15:24 smithi105 ceph-mon[35977]: using public_addr v2:172.21.15.105:0/0 -> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] 2024-05-27T07:15:25.216 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:15:24 smithi105 ceph-mon[35977]: starting mon.b rank -1 at public addrs [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] at bind addrs [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] mon_data /var/lib/ceph/mon/ceph-b fsid 73df2636-1bf8-11ef-bc9b-c7b262605968 2024-05-27T07:15:25.216 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:15:24 smithi105 ceph-mon[35977]: mon.b@-1(???) e0 preinit fsid 73df2636-1bf8-11ef-bc9b-c7b262605968 2024-05-27T07:15:25.216 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:15:24 smithi105 ceph-mon[35977]: mon.b@-1(synchronizing).mds e1 new map 2024-05-27T07:15:25.216 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:15:24 smithi105 ceph-mon[35977]: mon.b@-1(synchronizing).mds e1 print_map 2024-05-27T07:15:25.216 INFO:journalctl@ceph.mon.b.smithi105.stdout: e1 2024-05-27T07:15:25.216 INFO:journalctl@ceph.mon.b.smithi105.stdout: enable_multiple, ever_enabled_multiple: 1,1 2024-05-27T07:15:25.216 INFO:journalctl@ceph.mon.b.smithi105.stdout: default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2,11=minor log segments,12=quiesce subvolumes} 2024-05-27T07:15:25.216 INFO:journalctl@ceph.mon.b.smithi105.stdout: legacy client fscid: -1 2024-05-27T07:15:25.217 INFO:journalctl@ceph.mon.b.smithi105.stdout: 2024-05-27T07:15:25.217 INFO:journalctl@ceph.mon.b.smithi105.stdout: No filesystems configured 2024-05-27T07:15:25.217 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:15:24 smithi105 ceph-mon[35977]: mon.b@-1(synchronizing).osd e0 _set_cache_ratios kv ratio 0.25 inc ratio 0.375 full ratio 0.375 2024-05-27T07:15:25.217 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:15:24 smithi105 ceph-mon[35977]: mon.b@-1(synchronizing).osd e0 register_cache_with_pcm pcm target: 2147483648 pcm max: 1020054732 pcm min: 134217728 inc_osd_cache size: 1 2024-05-27T07:15:25.217 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:15:24 smithi105 ceph-mon[35977]: mon.b@-1(synchronizing).osd e1 e1: 0 total, 0 up, 0 in 2024-05-27T07:15:25.217 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:15:24 smithi105 ceph-mon[35977]: mon.b@-1(synchronizing).osd e2 e2: 0 total, 0 up, 0 in 2024-05-27T07:15:25.217 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:15:24 smithi105 ceph-mon[35977]: mon.b@-1(synchronizing).osd e3 e3: 0 total, 0 up, 0 in 2024-05-27T07:15:25.217 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:15:24 smithi105 ceph-mon[35977]: mon.b@-1(synchronizing).osd e4 e4: 0 total, 0 up, 0 in 2024-05-27T07:15:25.217 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:15:24 smithi105 ceph-mon[35977]: mon.b@-1(synchronizing).osd e4 crush map has features 3314932999778484224, adjusting msgr requires 2024-05-27T07:15:25.217 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:15:24 smithi105 ceph-mon[35977]: mon.b@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2024-05-27T07:15:25.217 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:15:24 smithi105 ceph-mon[35977]: mon.b@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2024-05-27T07:15:25.217 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:15:24 smithi105 ceph-mon[35977]: mon.b@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2024-05-27T07:15:25.217 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:15:24 smithi105 ceph-mon[35977]: pgmap v4: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-27T07:15:25.218 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:15:24 smithi105 ceph-mon[35977]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' 2024-05-27T07:15:25.218 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:15:24 smithi105 ceph-mon[35977]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-27T07:15:25.218 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:15:24 smithi105 ceph-mon[35977]: Added host smithi105 2024-05-27T07:15:25.218 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:15:24 smithi105 ceph-mon[35977]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' 2024-05-27T07:15:25.218 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:15:24 smithi105 ceph-mon[35977]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' 2024-05-27T07:15:25.218 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:15:24 smithi105 ceph-mon[35977]: pgmap v5: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-27T07:15:25.218 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:15:24 smithi105 ceph-mon[35977]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' 2024-05-27T07:15:25.218 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:15:24 smithi105 ceph-mon[35977]: from='client.14180 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-05-27T07:15:25.218 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:15:24 smithi105 ceph-mon[35977]: pgmap v6: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-27T07:15:25.218 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:15:24 smithi105 ceph-mon[35977]: pgmap v7: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-27T07:15:25.218 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:15:24 smithi105 ceph-mon[35977]: from='client.14182 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi173", "target": ["mon-mgr", ""]}]: dispatch 2024-05-27T07:15:25.218 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:15:24 smithi105 ceph-mon[35977]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' 2024-05-27T07:15:25.218 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:15:24 smithi105 ceph-mon[35977]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' 2024-05-27T07:15:25.218 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:15:24 smithi105 ceph-mon[35977]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' 2024-05-27T07:15:25.219 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:15:24 smithi105 ceph-mon[35977]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' 2024-05-27T07:15:25.219 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:15:24 smithi105 ceph-mon[35977]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd/host:smithi105", "name": "osd_memory_target"}]: dispatch 2024-05-27T07:15:25.219 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:15:24 smithi105 ceph-mon[35977]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T07:15:25.219 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:15:24 smithi105 ceph-mon[35977]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-27T07:15:25.219 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:15:24 smithi105 ceph-mon[35977]: Updating smithi105:/etc/ceph/ceph.conf 2024-05-27T07:15:25.219 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:15:24 smithi105 ceph-mon[35977]: Updating smithi105:/var/lib/ceph/73df2636-1bf8-11ef-bc9b-c7b262605968/config/ceph.conf 2024-05-27T07:15:25.219 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:15:24 smithi105 ceph-mon[35977]: Deploying cephadm binary to smithi173 2024-05-27T07:15:25.219 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:15:24 smithi105 ceph-mon[35977]: Updating smithi105:/etc/ceph/ceph.client.admin.keyring 2024-05-27T07:15:25.219 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:15:24 smithi105 ceph-mon[35977]: Updating smithi105:/var/lib/ceph/73df2636-1bf8-11ef-bc9b-c7b262605968/config/ceph.client.admin.keyring 2024-05-27T07:15:25.219 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:15:24 smithi105 ceph-mon[35977]: pgmap v8: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-27T07:15:25.219 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:15:24 smithi105 ceph-mon[35977]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' 2024-05-27T07:15:25.219 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:15:24 smithi105 ceph-mon[35977]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' 2024-05-27T07:15:25.219 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:15:24 smithi105 ceph-mon[35977]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' 2024-05-27T07:15:25.219 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:15:24 smithi105 ceph-mon[35977]: pgmap v9: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-27T07:15:25.219 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:15:24 smithi105 ceph-mon[35977]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' 2024-05-27T07:15:25.220 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:15:24 smithi105 ceph-mon[35977]: Added host smithi173 2024-05-27T07:15:25.220 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:15:24 smithi105 ceph-mon[35977]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-27T07:15:25.220 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:15:24 smithi105 ceph-mon[35977]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' 2024-05-27T07:15:25.220 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:15:24 smithi105 ceph-mon[35977]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' 2024-05-27T07:15:25.220 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:15:24 smithi105 ceph-mon[35977]: pgmap v10: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-27T07:15:25.220 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:15:24 smithi105 ceph-mon[35977]: from='client.14184 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-05-27T07:15:25.220 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:15:24 smithi105 ceph-mon[35977]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' 2024-05-27T07:15:25.220 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:15:24 smithi105 ceph-mon[35977]: pgmap v11: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-27T07:15:25.220 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:15:24 smithi105 ceph-mon[35977]: from='client.? 172.21.15.52:0/1478112303' entity='client.admin' cmd=[{"prefix": "osd crush tunables", "profile": "default"}]: dispatch 2024-05-27T07:15:25.220 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:15:24 smithi105 ceph-mon[35977]: pgmap v12: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-27T07:15:25.220 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:15:24 smithi105 ceph-mon[35977]: from='client.? 172.21.15.52:0/1478112303' entity='client.admin' cmd='[{"prefix": "osd crush tunables", "profile": "default"}]': finished 2024-05-27T07:15:25.220 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:15:24 smithi105 ceph-mon[35977]: osdmap e4: 0 total, 0 up, 0 in 2024-05-27T07:15:25.220 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:15:24 smithi105 ceph-mon[35977]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' 2024-05-27T07:15:25.220 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:15:24 smithi105 ceph-mon[35977]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' 2024-05-27T07:15:25.221 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:15:24 smithi105 ceph-mon[35977]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' 2024-05-27T07:15:25.221 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:15:24 smithi105 ceph-mon[35977]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' 2024-05-27T07:15:25.221 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:15:24 smithi105 ceph-mon[35977]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd/host:smithi173", "name": "osd_memory_target"}]: dispatch 2024-05-27T07:15:25.221 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:15:24 smithi105 ceph-mon[35977]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T07:15:25.221 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:15:24 smithi105 ceph-mon[35977]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-27T07:15:25.221 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:15:24 smithi105 ceph-mon[35977]: Updating smithi173:/etc/ceph/ceph.conf 2024-05-27T07:15:25.221 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:15:24 smithi105 ceph-mon[35977]: Updating smithi173:/var/lib/ceph/73df2636-1bf8-11ef-bc9b-c7b262605968/config/ceph.conf 2024-05-27T07:15:25.221 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:15:24 smithi105 ceph-mon[35977]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' 2024-05-27T07:15:25.221 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:15:24 smithi105 ceph-mon[35977]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' 2024-05-27T07:15:25.221 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:15:24 smithi105 ceph-mon[35977]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' 2024-05-27T07:15:25.221 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:15:24 smithi105 ceph-mon[35977]: Updating smithi173:/etc/ceph/ceph.client.admin.keyring 2024-05-27T07:15:25.221 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:15:24 smithi105 ceph-mon[35977]: Updating smithi173:/var/lib/ceph/73df2636-1bf8-11ef-bc9b-c7b262605968/config/ceph.client.admin.keyring 2024-05-27T07:15:25.221 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:15:24 smithi105 ceph-mon[35977]: pgmap v14: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-27T07:15:25.221 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:15:24 smithi105 ceph-mon[35977]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' 2024-05-27T07:15:25.221 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:15:24 smithi105 ceph-mon[35977]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-27T07:15:25.222 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:15:24 smithi105 ceph-mon[35977]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T07:15:25.222 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:15:24 smithi105 ceph-mon[35977]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-27T07:15:25.222 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:15:24 smithi105 ceph-mon[35977]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' 2024-05-27T07:15:25.222 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:15:24 smithi105 ceph-mon[35977]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-05-27T07:15:25.222 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:15:24 smithi105 ceph-mon[35977]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T07:15:25.222 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:15:24 smithi105 ceph-mon[35977]: from='client.14188 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "placement": "3;smithi052:172.21.15.52=a;smithi105:172.21.15.105=b;smithi173:172.21.15.173=c", "target": ["mon-mgr", ""]}]: dispatch 2024-05-27T07:15:25.222 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:15:24 smithi105 ceph-mon[35977]: Saving service mon spec with placement smithi052:172.21.15.52=a;smithi105:172.21.15.105=b;smithi173:172.21.15.173=c;count:3 2024-05-27T07:15:25.222 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:15:24 smithi105 ceph-mon[35977]: Deploying daemon mon.c on smithi173 2024-05-27T07:15:25.222 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:15:24 smithi105 ceph-mon[35977]: pgmap v15: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-27T07:15:25.222 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:15:24 smithi105 ceph-mon[35977]: pgmap v16: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-27T07:15:25.222 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:15:24 smithi105 ceph-mon[35977]: from='client.? 172.21.15.173:0/53244174' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-05-27T07:15:25.222 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:15:24 smithi105 ceph-mon[35977]: pgmap v17: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-27T07:15:25.222 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:15:24 smithi105 ceph-mon[35977]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-05-27T07:15:25.223 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:15:24 smithi105 ceph-mon[35977]: mon.a calling monitor election 2024-05-27T07:15:25.223 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:15:24 smithi105 ceph-mon[35977]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-05-27T07:15:25.223 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:15:24 smithi105 ceph-mon[35977]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-05-27T07:15:25.223 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:15:24 smithi105 ceph-mon[35977]: pgmap v18: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-27T07:15:25.223 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:15:24 smithi105 ceph-mon[35977]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-05-27T07:15:25.223 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:15:24 smithi105 ceph-mon[35977]: mon.c calling monitor election 2024-05-27T07:15:25.223 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:15:24 smithi105 ceph-mon[35977]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-05-27T07:15:25.223 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:15:24 smithi105 ceph-mon[35977]: pgmap v19: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-27T07:15:25.223 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:15:24 smithi105 ceph-mon[35977]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-05-27T07:15:25.223 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:15:24 smithi105 ceph-mon[35977]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-05-27T07:15:25.223 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:15:24 smithi105 ceph-mon[35977]: mon.a is new leader, mons a,c in quorum (ranks 0,1) 2024-05-27T07:15:25.223 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:15:24 smithi105 ceph-mon[35977]: monmap epoch 2 2024-05-27T07:15:25.223 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:15:24 smithi105 ceph-mon[35977]: fsid 73df2636-1bf8-11ef-bc9b-c7b262605968 2024-05-27T07:15:25.223 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:15:24 smithi105 ceph-mon[35977]: last_changed 2024-05-27T07:15:15.809004+0000 2024-05-27T07:15:25.224 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:15:24 smithi105 ceph-mon[35977]: created 2024-05-27T07:13:31.465220+0000 2024-05-27T07:15:25.224 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:15:24 smithi105 ceph-mon[35977]: min_mon_release 19 (squid) 2024-05-27T07:15:25.224 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:15:24 smithi105 ceph-mon[35977]: election_strategy: 1 2024-05-27T07:15:25.224 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:15:24 smithi105 ceph-mon[35977]: 0: [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] mon.a 2024-05-27T07:15:25.224 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:15:24 smithi105 ceph-mon[35977]: 1: [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] mon.c 2024-05-27T07:15:25.224 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:15:24 smithi105 ceph-mon[35977]: fsmap 2024-05-27T07:15:25.224 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:15:24 smithi105 ceph-mon[35977]: osdmap e4: 0 total, 0 up, 0 in 2024-05-27T07:15:25.224 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:15:24 smithi105 ceph-mon[35977]: mgrmap e13: x(active, since 51s) 2024-05-27T07:15:25.224 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:15:24 smithi105 ceph-mon[35977]: overall HEALTH_OK 2024-05-27T07:15:25.224 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:15:24 smithi105 ceph-mon[35977]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' 2024-05-27T07:15:25.224 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:15:24 smithi105 ceph-mon[35977]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' 2024-05-27T07:15:25.224 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:15:24 smithi105 ceph-mon[35977]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' 2024-05-27T07:15:25.224 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:15:24 smithi105 ceph-mon[35977]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-05-27T07:15:25.225 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:15:24 smithi105 ceph-mon[35977]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T07:15:25.225 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:15:24 smithi105 ceph-mon[35977]: Deploying daemon mon.b on smithi105 2024-05-27T07:15:25.225 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:15:24 smithi105 ceph-mon[35977]: from='client.? 172.21.15.173:0/4091862757' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-05-27T07:15:25.225 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:15:24 smithi105 ceph-mon[35977]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-05-27T07:15:25.225 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:15:24 smithi105 ceph-mon[35977]: pgmap v20: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-27T07:15:25.225 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:15:24 smithi105 ceph-mon[35977]: pgmap v21: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-27T07:15:25.225 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:15:24 smithi105 ceph-mon[35977]: from='client.? 172.21.15.173:0/3304214027' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-05-27T07:15:25.225 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:15:24 smithi105 ceph-mon[35977]: mon.b@-1(synchronizing).paxosservice(auth 1..3) refresh upgraded, format 0 -> 3 2024-05-27T07:15:25.225 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:15:24 smithi105 bash[35950]: ac140bd844088e241552e0adb16cd5f33c4259c1c6d8721e0edfbb1ae48164bf 2024-05-27T07:15:25.225 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:15:24 smithi105 systemd[1]: Started Ceph mon.b for 73df2636-1bf8-11ef-bc9b-c7b262605968. 2024-05-27T07:15:25.230 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:15:24 smithi052 ceph-mon[31955]: pgmap v21: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-27T07:15:25.230 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:15:24 smithi052 ceph-mon[31955]: from='client.? 172.21.15.173:0/3304214027' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-05-27T07:15:26.324 INFO:tasks.cephadm:Waiting for 3 mons in monmap... 2024-05-27T07:15:26.324 DEBUG:teuthology.orchestra.run.smithi173:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:38f9d900c83c63799fbdbe61acc9a11b0d3554a6 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 73df2636-1bf8-11ef-bc9b-c7b262605968 -- ceph mon dump -f json 2024-05-27T07:15:26.565 INFO:teuthology.orchestra.run.smithi173.stderr:Inferring config /var/lib/ceph/73df2636-1bf8-11ef-bc9b-c7b262605968/mon.c/config 2024-05-27T07:15:30.353 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:15:29 smithi173 ceph-mon[37000]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-05-27T07:15:30.353 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:15:29 smithi173 ceph-mon[37000]: mon.a calling monitor election 2024-05-27T07:15:30.353 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:15:29 smithi173 ceph-mon[37000]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-05-27T07:15:30.353 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:15:29 smithi173 ceph-mon[37000]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-05-27T07:15:30.353 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:15:29 smithi173 ceph-mon[37000]: mon.c calling monitor election 2024-05-27T07:15:30.353 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:15:29 smithi173 ceph-mon[37000]: pgmap v22: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-27T07:15:30.353 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:15:29 smithi173 ceph-mon[37000]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-05-27T07:15:30.353 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:15:29 smithi173 ceph-mon[37000]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-05-27T07:15:30.353 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:15:29 smithi173 ceph-mon[37000]: mon.b calling monitor election 2024-05-27T07:15:30.354 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:15:29 smithi173 ceph-mon[37000]: pgmap v23: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-27T07:15:30.354 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:15:29 smithi173 ceph-mon[37000]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-05-27T07:15:30.354 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:15:29 smithi173 ceph-mon[37000]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-05-27T07:15:30.354 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:15:29 smithi173 ceph-mon[37000]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-05-27T07:15:30.354 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:15:29 smithi173 ceph-mon[37000]: mon.a is new leader, mons a,c,b in quorum (ranks 0,1,2) 2024-05-27T07:15:30.354 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:15:29 smithi173 ceph-mon[37000]: monmap epoch 3 2024-05-27T07:15:30.354 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:15:29 smithi173 ceph-mon[37000]: fsid 73df2636-1bf8-11ef-bc9b-c7b262605968 2024-05-27T07:15:30.354 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:15:29 smithi173 ceph-mon[37000]: last_changed 2024-05-27T07:15:24.926772+0000 2024-05-27T07:15:30.354 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:15:29 smithi173 ceph-mon[37000]: created 2024-05-27T07:13:31.465220+0000 2024-05-27T07:15:30.354 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:15:29 smithi173 ceph-mon[37000]: min_mon_release 19 (squid) 2024-05-27T07:15:30.354 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:15:29 smithi173 ceph-mon[37000]: election_strategy: 1 2024-05-27T07:15:30.354 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:15:29 smithi173 ceph-mon[37000]: 0: [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] mon.a 2024-05-27T07:15:30.354 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:15:29 smithi173 ceph-mon[37000]: 1: [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] mon.c 2024-05-27T07:15:30.354 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:15:29 smithi173 ceph-mon[37000]: 2: [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] mon.b 2024-05-27T07:15:30.355 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:15:29 smithi173 ceph-mon[37000]: fsmap 2024-05-27T07:15:30.355 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:15:29 smithi173 ceph-mon[37000]: osdmap e4: 0 total, 0 up, 0 in 2024-05-27T07:15:30.355 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:15:29 smithi173 ceph-mon[37000]: mgrmap e13: x(active, since 60s) 2024-05-27T07:15:30.355 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:15:29 smithi173 ceph-mon[37000]: overall HEALTH_OK 2024-05-27T07:15:30.355 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:15:29 smithi173 ceph-mon[37000]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' 2024-05-27T07:15:30.355 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:15:29 smithi173 ceph-mon[37000]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' 2024-05-27T07:15:30.355 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:15:29 smithi173 ceph-mon[37000]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' 2024-05-27T07:15:30.355 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:15:29 smithi173 ceph-mon[37000]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' 2024-05-27T07:15:30.441 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:15:29 smithi105 ceph-mon[35977]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-05-27T07:15:30.441 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:15:29 smithi105 ceph-mon[35977]: mon.a calling monitor election 2024-05-27T07:15:30.441 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:15:29 smithi105 ceph-mon[35977]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-05-27T07:15:30.441 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:15:29 smithi105 ceph-mon[35977]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-05-27T07:15:30.441 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:15:29 smithi105 ceph-mon[35977]: mon.c calling monitor election 2024-05-27T07:15:30.441 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:15:29 smithi105 ceph-mon[35977]: pgmap v22: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-27T07:15:30.441 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:15:29 smithi105 ceph-mon[35977]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-05-27T07:15:30.441 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:15:29 smithi105 ceph-mon[35977]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-05-27T07:15:30.441 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:15:29 smithi105 ceph-mon[35977]: mon.b calling monitor election 2024-05-27T07:15:30.441 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:15:29 smithi105 ceph-mon[35977]: pgmap v23: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-27T07:15:30.442 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:15:29 smithi105 ceph-mon[35977]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-05-27T07:15:30.442 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:15:29 smithi105 ceph-mon[35977]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-05-27T07:15:30.442 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:15:29 smithi105 ceph-mon[35977]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-05-27T07:15:30.442 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:15:29 smithi105 ceph-mon[35977]: mon.a is new leader, mons a,c,b in quorum (ranks 0,1,2) 2024-05-27T07:15:30.442 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:15:29 smithi105 ceph-mon[35977]: monmap epoch 3 2024-05-27T07:15:30.442 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:15:29 smithi105 ceph-mon[35977]: fsid 73df2636-1bf8-11ef-bc9b-c7b262605968 2024-05-27T07:15:30.442 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:15:29 smithi105 ceph-mon[35977]: last_changed 2024-05-27T07:15:24.926772+0000 2024-05-27T07:15:30.442 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:15:29 smithi105 ceph-mon[35977]: created 2024-05-27T07:13:31.465220+0000 2024-05-27T07:15:30.442 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:15:29 smithi105 ceph-mon[35977]: min_mon_release 19 (squid) 2024-05-27T07:15:30.442 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:15:29 smithi105 ceph-mon[35977]: election_strategy: 1 2024-05-27T07:15:30.442 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:15:29 smithi105 ceph-mon[35977]: 0: [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] mon.a 2024-05-27T07:15:30.442 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:15:29 smithi105 ceph-mon[35977]: 1: [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] mon.c 2024-05-27T07:15:30.442 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:15:29 smithi105 ceph-mon[35977]: 2: [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] mon.b 2024-05-27T07:15:30.442 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:15:29 smithi105 ceph-mon[35977]: fsmap 2024-05-27T07:15:30.443 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:15:29 smithi105 ceph-mon[35977]: osdmap e4: 0 total, 0 up, 0 in 2024-05-27T07:15:30.443 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:15:29 smithi105 ceph-mon[35977]: mgrmap e13: x(active, since 60s) 2024-05-27T07:15:30.443 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:15:29 smithi105 ceph-mon[35977]: overall HEALTH_OK 2024-05-27T07:15:30.443 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:15:29 smithi105 ceph-mon[35977]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' 2024-05-27T07:15:30.443 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:15:29 smithi105 ceph-mon[35977]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' 2024-05-27T07:15:30.443 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:15:29 smithi105 ceph-mon[35977]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' 2024-05-27T07:15:30.443 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:15:29 smithi105 ceph-mon[35977]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' 2024-05-27T07:15:30.480 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:15:29 smithi052 ceph-mon[31955]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-05-27T07:15:30.480 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:15:29 smithi052 ceph-mon[31955]: mon.a calling monitor election 2024-05-27T07:15:30.480 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:15:29 smithi052 ceph-mon[31955]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-05-27T07:15:30.480 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:15:29 smithi052 ceph-mon[31955]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-05-27T07:15:30.480 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:15:29 smithi052 ceph-mon[31955]: mon.c calling monitor election 2024-05-27T07:15:30.480 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:15:29 smithi052 ceph-mon[31955]: pgmap v22: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-27T07:15:30.481 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:15:29 smithi052 ceph-mon[31955]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-05-27T07:15:30.481 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:15:29 smithi052 ceph-mon[31955]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-05-27T07:15:30.481 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:15:29 smithi052 ceph-mon[31955]: mon.b calling monitor election 2024-05-27T07:15:30.481 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:15:29 smithi052 ceph-mon[31955]: pgmap v23: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-27T07:15:30.481 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:15:29 smithi052 ceph-mon[31955]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-05-27T07:15:30.481 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:15:29 smithi052 ceph-mon[31955]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-05-27T07:15:30.481 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:15:29 smithi052 ceph-mon[31955]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-05-27T07:15:30.481 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:15:29 smithi052 ceph-mon[31955]: mon.a is new leader, mons a,c,b in quorum (ranks 0,1,2) 2024-05-27T07:15:30.481 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:15:29 smithi052 ceph-mon[31955]: monmap epoch 3 2024-05-27T07:15:30.481 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:15:29 smithi052 ceph-mon[31955]: fsid 73df2636-1bf8-11ef-bc9b-c7b262605968 2024-05-27T07:15:30.481 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:15:29 smithi052 ceph-mon[31955]: last_changed 2024-05-27T07:15:24.926772+0000 2024-05-27T07:15:30.481 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:15:29 smithi052 ceph-mon[31955]: created 2024-05-27T07:13:31.465220+0000 2024-05-27T07:15:30.481 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:15:29 smithi052 ceph-mon[31955]: min_mon_release 19 (squid) 2024-05-27T07:15:30.481 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:15:29 smithi052 ceph-mon[31955]: election_strategy: 1 2024-05-27T07:15:30.482 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:15:29 smithi052 ceph-mon[31955]: 0: [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] mon.a 2024-05-27T07:15:30.482 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:15:29 smithi052 ceph-mon[31955]: 1: [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] mon.c 2024-05-27T07:15:30.482 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:15:29 smithi052 ceph-mon[31955]: 2: [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] mon.b 2024-05-27T07:15:30.482 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:15:29 smithi052 ceph-mon[31955]: fsmap 2024-05-27T07:15:30.482 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:15:29 smithi052 ceph-mon[31955]: osdmap e4: 0 total, 0 up, 0 in 2024-05-27T07:15:30.482 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:15:29 smithi052 ceph-mon[31955]: mgrmap e13: x(active, since 60s) 2024-05-27T07:15:30.482 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:15:29 smithi052 ceph-mon[31955]: overall HEALTH_OK 2024-05-27T07:15:30.482 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:15:29 smithi052 ceph-mon[31955]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' 2024-05-27T07:15:30.482 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:15:29 smithi052 ceph-mon[31955]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' 2024-05-27T07:15:30.482 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:15:29 smithi052 ceph-mon[31955]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' 2024-05-27T07:15:30.482 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:15:29 smithi052 ceph-mon[31955]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' 2024-05-27T07:15:31.020 INFO:teuthology.orchestra.run.smithi173.stderr:2024-05-27T07:15:31.029+0000 7fc160e41700 1 -- 172.21.15.173:0/3351518033 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fc1440035c0 msgr2=0x7fc144005aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:15:31.020 INFO:teuthology.orchestra.run.smithi173.stderr:2024-05-27T07:15:31.029+0000 7fc160e41700 1 --2- 172.21.15.173:0/3351518033 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fc1440035c0 0x7fc144005aa0 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7fc15c102a60 tx=0x7fc14c00feb0 comp rx=0 tx=0).stop 2024-05-27T07:15:31.020 INFO:teuthology.orchestra.run.smithi173.stderr:2024-05-27T07:15:31.029+0000 7fc160e41700 1 -- 172.21.15.173:0/3351518033 shutdown_connections 2024-05-27T07:15:31.020 INFO:teuthology.orchestra.run.smithi173.stderr:2024-05-27T07:15:31.029+0000 7fc160e41700 1 --2- 172.21.15.173:0/3351518033 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fc1440035c0 0x7fc144005aa0 unknown :-1 s=CLOSED pgs=106 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:15:31.020 INFO:teuthology.orchestra.run.smithi173.stderr:2024-05-27T07:15:31.029+0000 7fc160e41700 1 -- 172.21.15.173:0/3351518033 >> 172.21.15.173:0/3351518033 conn(0x7fc15c0797d0 msgr2=0x7fc15c079c10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:15:31.021 INFO:teuthology.orchestra.run.smithi173.stderr:2024-05-27T07:15:31.029+0000 7fc160e41700 1 -- 172.21.15.173:0/3351518033 shutdown_connections 2024-05-27T07:15:31.021 INFO:teuthology.orchestra.run.smithi173.stderr:2024-05-27T07:15:31.029+0000 7fc160e41700 1 -- 172.21.15.173:0/3351518033 wait complete. 2024-05-27T07:15:31.023 INFO:teuthology.orchestra.run.smithi173.stderr:2024-05-27T07:15:31.030+0000 7fc160e41700 1 Processor -- start 2024-05-27T07:15:31.023 INFO:teuthology.orchestra.run.smithi173.stderr:2024-05-27T07:15:31.030+0000 7fc160e41700 1 -- start start 2024-05-27T07:15:31.023 INFO:teuthology.orchestra.run.smithi173.stderr:2024-05-27T07:15:31.031+0000 7fc160e41700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fc1440035c0 0x7fc15c19e780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:15:31.023 INFO:teuthology.orchestra.run.smithi173.stderr:2024-05-27T07:15:31.031+0000 7fc160e41700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fc15c19ecc0 0x7fc15c1989a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:15:31.023 INFO:teuthology.orchestra.run.smithi173.stderr:2024-05-27T07:15:31.031+0000 7fc160e41700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fc15c198ee0 0x7fc15c1992f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:15:31.023 INFO:teuthology.orchestra.run.smithi173.stderr:2024-05-27T07:15:31.031+0000 7fc160e41700 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x7fc15c10aae0 con 0x7fc15c19ecc0 2024-05-27T07:15:31.023 INFO:teuthology.orchestra.run.smithi173.stderr:2024-05-27T07:15:31.031+0000 7fc160e41700 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7fc15c10a960 con 0x7fc15c198ee0 2024-05-27T07:15:31.023 INFO:teuthology.orchestra.run.smithi173.stderr:2024-05-27T07:15:31.031+0000 7fc160e41700 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7fc15c10ac60 con 0x7fc1440035c0 2024-05-27T07:15:31.024 INFO:teuthology.orchestra.run.smithi173.stderr:2024-05-27T07:15:31.031+0000 7fc15a59c700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fc1440035c0 0x7fc15c19e780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:15:31.024 INFO:teuthology.orchestra.run.smithi173.stderr:2024-05-27T07:15:31.031+0000 7fc15a59c700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fc1440035c0 0x7fc15c19e780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.173:3300/0 says I am v2:172.21.15.173:55856/0 (socket says 172.21.15.173:55856) 2024-05-27T07:15:31.024 INFO:teuthology.orchestra.run.smithi173.stderr:2024-05-27T07:15:31.031+0000 7fc15a59c700 1 -- 172.21.15.173:0/3412253802 learned_addr learned my addr 172.21.15.173:0/3412253802 (peer_addr_for_me v2:172.21.15.173:0/0) 2024-05-27T07:15:31.024 INFO:teuthology.orchestra.run.smithi173.stderr:2024-05-27T07:15:31.031+0000 7fc159d9b700 1 --2- 172.21.15.173:0/3412253802 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fc15c19ecc0 0x7fc15c1989a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:15:31.024 INFO:teuthology.orchestra.run.smithi173.stderr:2024-05-27T07:15:31.031+0000 7fc15ad9d700 1 --2- 172.21.15.173:0/3412253802 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fc15c198ee0 0x7fc15c1992f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:15:31.024 INFO:teuthology.orchestra.run.smithi173.stderr:2024-05-27T07:15:31.031+0000 7fc15a59c700 1 -- 172.21.15.173:0/3412253802 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fc15c198ee0 msgr2=0x7fc15c1992f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:15:31.024 INFO:teuthology.orchestra.run.smithi173.stderr:2024-05-27T07:15:31.031+0000 7fc15a59c700 1 --2- 172.21.15.173:0/3412253802 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fc15c198ee0 0x7fc15c1992f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:15:31.024 INFO:teuthology.orchestra.run.smithi173.stderr:2024-05-27T07:15:31.031+0000 7fc15a59c700 1 -- 172.21.15.173:0/3412253802 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fc15c19ecc0 msgr2=0x7fc15c1989a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:15:31.024 INFO:teuthology.orchestra.run.smithi173.stderr:2024-05-27T07:15:31.031+0000 7fc15a59c700 1 --2- 172.21.15.173:0/3412253802 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fc15c19ecc0 0x7fc15c1989a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:15:31.024 INFO:teuthology.orchestra.run.smithi173.stderr:2024-05-27T07:15:31.031+0000 7fc15a59c700 1 -- 172.21.15.173:0/3412253802 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc15c06bcd0 con 0x7fc1440035c0 2024-05-27T07:15:31.024 INFO:teuthology.orchestra.run.smithi173.stderr:2024-05-27T07:15:31.031+0000 7fc15a59c700 1 --2- 172.21.15.173:0/3412253802 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fc1440035c0 0x7fc15c19e780 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7fc14c001eb0 tx=0x7fc14c0050a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:15:31.024 INFO:teuthology.orchestra.run.smithi173.stderr:2024-05-27T07:15:31.032+0000 7fc14b7fe700 1 -- 172.21.15.173:0/3412253802 <== mon.1 v2:172.21.15.173:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc14c029030 con 0x7fc1440035c0 2024-05-27T07:15:31.025 INFO:teuthology.orchestra.run.smithi173.stderr:2024-05-27T07:15:31.032+0000 7fc160e41700 1 -- 172.21.15.173:0/3412253802 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc15c06bf60 con 0x7fc1440035c0 2024-05-27T07:15:31.025 INFO:teuthology.orchestra.run.smithi173.stderr:2024-05-27T07:15:31.032+0000 7fc14b7fe700 1 -- 172.21.15.173:0/3412253802 <== mon.1 v2:172.21.15.173:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fc14c011b90 con 0x7fc1440035c0 2024-05-27T07:15:31.025 INFO:teuthology.orchestra.run.smithi173.stderr:2024-05-27T07:15:31.032+0000 7fc14b7fe700 1 -- 172.21.15.173:0/3412253802 <== mon.1 v2:172.21.15.173:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc14c003ba0 con 0x7fc1440035c0 2024-05-27T07:15:31.025 INFO:teuthology.orchestra.run.smithi173.stderr:2024-05-27T07:15:31.032+0000 7fc160e41700 1 -- 172.21.15.173:0/3412253802 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc15c06c570 con 0x7fc1440035c0 2024-05-27T07:15:31.026 INFO:teuthology.orchestra.run.smithi173.stderr:2024-05-27T07:15:31.033+0000 7fc14b7fe700 1 -- 172.21.15.173:0/3412253802 <== mon.1 v2:172.21.15.173:3300/0 4 ==== mgrmap(e 13) ==== 50278+0+0 (secure 0 0 0) 0x7fc14c02c070 con 0x7fc1440035c0 2024-05-27T07:15:31.026 INFO:teuthology.orchestra.run.smithi173.stderr:2024-05-27T07:15:31.033+0000 7fc14b7fe700 1 --2- 172.21.15.173:0/3412253802 >> [v2:172.21.15.52:6800/3513075805,v1:172.21.15.52:6801/3513075805] conn(0x7fc13c03e9a0 0x7fc13c040eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:15:31.026 INFO:teuthology.orchestra.run.smithi173.stderr:2024-05-27T07:15:31.033+0000 7fc14b7fe700 1 -- 172.21.15.173:0/3412253802 <== mon.1 v2:172.21.15.173:3300/0 5 ==== osd_map(4..4 src has 1..4) ==== 1155+0+0 (secure 0 0 0) 0x7fc14c0558c0 con 0x7fc1440035c0 2024-05-27T07:15:31.026 INFO:teuthology.orchestra.run.smithi173.stderr:2024-05-27T07:15:31.033+0000 7fc160e41700 1 -- 172.21.15.173:0/3412253802 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc15c064e10 con 0x7fc1440035c0 2024-05-27T07:15:31.026 INFO:teuthology.orchestra.run.smithi173.stderr:2024-05-27T07:15:31.033+0000 7fc159d9b700 1 --2- 172.21.15.173:0/3412253802 >> [v2:172.21.15.52:6800/3513075805,v1:172.21.15.52:6801/3513075805] conn(0x7fc13c03e9a0 0x7fc13c040eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:15:31.026 INFO:teuthology.orchestra.run.smithi173.stderr:2024-05-27T07:15:31.034+0000 7fc159d9b700 1 --2- 172.21.15.173:0/3412253802 >> [v2:172.21.15.52:6800/3513075805,v1:172.21.15.52:6801/3513075805] conn(0x7fc13c03e9a0 0x7fc13c040eb0 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7fc15c108600 tx=0x7fc150009210 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:15:31.030 INFO:teuthology.orchestra.run.smithi173.stderr:2024-05-27T07:15:31.039+0000 7fc14b7fe700 1 -- 172.21.15.173:0/3412253802 <== mon.1 v2:172.21.15.173:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191963 (secure 0 0 0) 0x7fc14c033070 con 0x7fc1440035c0 2024-05-27T07:15:31.325 INFO:teuthology.orchestra.run.smithi173.stderr:2024-05-27T07:15:31.334+0000 7fc160e41700 1 -- 172.21.15.173:0/3412253802 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) -- 0x7fc15c06c7d0 con 0x7fc1440035c0 2024-05-27T07:15:31.325 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:15:31 smithi173 ceph-mon[37000]: pgmap v24: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-27T07:15:31.325 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:15:31 smithi173 ceph-mon[37000]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-27T07:15:31.325 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:15:31 smithi173 ceph-mon[37000]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-05-27T07:15:31.326 INFO:teuthology.orchestra.run.smithi173.stderr:2024-05-27T07:15:31.335+0000 7fc14b7fe700 1 -- 172.21.15.173:0/3412253802 <== mon.1 v2:172.21.15.173:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 3 v3) ==== 95+0+1285 (secure 0 0 0) 0x7fc14c0103d0 con 0x7fc1440035c0 2024-05-27T07:15:31.326 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-05-27T07:15:31.326 INFO:teuthology.orchestra.run.smithi173.stdout:{"epoch":3,"fsid":"73df2636-1bf8-11ef-bc9b-c7b262605968","modified":"2024-05-27T07:15:24.926772Z","created":"2024-05-27T07:13:31.465220Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.52:3300","nonce":0},{"type":"v1","addr":"172.21.15.52:6789","nonce":0}]},"addr":"172.21.15.52:6789/0","public_addr":"172.21.15.52:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:3300","nonce":0},{"type":"v1","addr":"172.21.15.173:6789","nonce":0}]},"addr":"172.21.15.173:6789/0","public_addr":"172.21.15.173:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:3300","nonce":0},{"type":"v1","addr":"172.21.15.105:6789","nonce":0}]},"addr":"172.21.15.105:6789/0","public_addr":"172.21.15.105:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0,1,2]} 2024-05-27T07:15:31.328 INFO:teuthology.orchestra.run.smithi173.stderr:2024-05-27T07:15:31.337+0000 7fc160e41700 1 -- 172.21.15.173:0/3412253802 >> [v2:172.21.15.52:6800/3513075805,v1:172.21.15.52:6801/3513075805] conn(0x7fc13c03e9a0 msgr2=0x7fc13c040eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:15:31.328 INFO:teuthology.orchestra.run.smithi173.stderr:2024-05-27T07:15:31.337+0000 7fc160e41700 1 --2- 172.21.15.173:0/3412253802 >> [v2:172.21.15.52:6800/3513075805,v1:172.21.15.52:6801/3513075805] conn(0x7fc13c03e9a0 0x7fc13c040eb0 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7fc15c108600 tx=0x7fc150009210 comp rx=0 tx=0).stop 2024-05-27T07:15:31.328 INFO:teuthology.orchestra.run.smithi173.stderr:2024-05-27T07:15:31.337+0000 7fc160e41700 1 -- 172.21.15.173:0/3412253802 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fc1440035c0 msgr2=0x7fc15c19e780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:15:31.328 INFO:teuthology.orchestra.run.smithi173.stderr:2024-05-27T07:15:31.337+0000 7fc160e41700 1 --2- 172.21.15.173:0/3412253802 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fc1440035c0 0x7fc15c19e780 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7fc14c001eb0 tx=0x7fc14c0050a0 comp rx=0 tx=0).stop 2024-05-27T07:15:31.329 INFO:teuthology.orchestra.run.smithi173.stderr:2024-05-27T07:15:31.337+0000 7fc160e41700 1 -- 172.21.15.173:0/3412253802 shutdown_connections 2024-05-27T07:15:31.329 INFO:teuthology.orchestra.run.smithi173.stderr:2024-05-27T07:15:31.337+0000 7fc160e41700 1 --2- 172.21.15.173:0/3412253802 >> [v2:172.21.15.52:6800/3513075805,v1:172.21.15.52:6801/3513075805] conn(0x7fc13c03e9a0 0x7fc13c040eb0 unknown :-1 s=CLOSED pgs=38 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:15:31.329 INFO:teuthology.orchestra.run.smithi173.stderr:2024-05-27T07:15:31.337+0000 7fc160e41700 1 --2- 172.21.15.173:0/3412253802 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fc15c198ee0 0x7fc15c1992f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:15:31.329 INFO:teuthology.orchestra.run.smithi173.stderr:2024-05-27T07:15:31.337+0000 7fc160e41700 1 --2- 172.21.15.173:0/3412253802 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fc1440035c0 0x7fc15c19e780 unknown :-1 s=CLOSED pgs=5 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:15:31.329 INFO:teuthology.orchestra.run.smithi173.stderr:2024-05-27T07:15:31.337+0000 7fc160e41700 1 --2- 172.21.15.173:0/3412253802 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fc15c19ecc0 0x7fc15c1989a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:15:31.329 INFO:teuthology.orchestra.run.smithi173.stderr:2024-05-27T07:15:31.337+0000 7fc160e41700 1 -- 172.21.15.173:0/3412253802 >> 172.21.15.173:0/3412253802 conn(0x7fc15c0797d0 msgr2=0x7fc15c101d20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:15:31.329 INFO:teuthology.orchestra.run.smithi173.stderr:2024-05-27T07:15:31.337+0000 7fc160e41700 1 -- 172.21.15.173:0/3412253802 shutdown_connections 2024-05-27T07:15:31.329 INFO:teuthology.orchestra.run.smithi173.stderr:2024-05-27T07:15:31.337+0000 7fc160e41700 1 -- 172.21.15.173:0/3412253802 wait complete. 2024-05-27T07:15:31.329 INFO:teuthology.orchestra.run.smithi173.stderr:dumped monmap epoch 3 2024-05-27T07:15:31.441 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:15:31 smithi105 ceph-mon[35977]: pgmap v24: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-27T07:15:31.441 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:15:31 smithi105 ceph-mon[35977]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-27T07:15:31.441 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:15:31 smithi105 ceph-mon[35977]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-05-27T07:15:31.480 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:15:31 smithi052 ceph-mon[31955]: pgmap v24: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-27T07:15:31.480 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:15:31 smithi052 ceph-mon[31955]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-27T07:15:31.480 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:15:31 smithi052 ceph-mon[31955]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-05-27T07:15:32.017 INFO:tasks.cephadm:Generating final ceph.conf file... 2024-05-27T07:15:32.017 DEBUG:teuthology.orchestra.run.smithi052:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:38f9d900c83c63799fbdbe61acc9a11b0d3554a6 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 73df2636-1bf8-11ef-bc9b-c7b262605968 -- ceph config generate-minimal-conf 2024-05-27T07:15:32.275 INFO:teuthology.orchestra.run.smithi052.stderr:Inferring config /var/lib/ceph/73df2636-1bf8-11ef-bc9b-c7b262605968/mon.a/config 2024-05-27T07:15:32.352 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:15:32 smithi173 ceph-mon[37000]: from='client.? 172.21.15.173:0/3412253802' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-05-27T07:15:32.353 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:15:32 smithi173 ceph-mon[37000]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' 2024-05-27T07:15:32.353 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:15:32 smithi173 ceph-mon[37000]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' 2024-05-27T07:15:32.353 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:15:32 smithi173 ceph-mon[37000]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' 2024-05-27T07:15:32.353 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:15:32 smithi173 ceph-mon[37000]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' 2024-05-27T07:15:32.353 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:15:32 smithi173 ceph-mon[37000]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T07:15:32.353 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:15:32 smithi173 ceph-mon[37000]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-27T07:15:32.353 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:15:32 smithi173 ceph-mon[37000]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' 2024-05-27T07:15:32.353 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:15:32 smithi173 ceph-mon[37000]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' 2024-05-27T07:15:32.353 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:15:32 smithi173 ceph-mon[37000]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' 2024-05-27T07:15:32.353 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:15:32 smithi173 ceph-mon[37000]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' 2024-05-27T07:15:32.353 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:15:32 smithi173 ceph-mon[37000]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' 2024-05-27T07:15:32.354 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:15:32 smithi173 ceph-mon[37000]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' 2024-05-27T07:15:32.354 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:15:32 smithi173 ceph-mon[37000]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' 2024-05-27T07:15:32.374 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:15:32 smithi052 ceph-mon[31955]: from='client.? 172.21.15.173:0/3412253802' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-05-27T07:15:32.374 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:15:32 smithi052 ceph-mon[31955]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' 2024-05-27T07:15:32.374 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:15:32 smithi052 ceph-mon[31955]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' 2024-05-27T07:15:32.374 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:15:32 smithi052 ceph-mon[31955]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' 2024-05-27T07:15:32.374 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:15:32 smithi052 ceph-mon[31955]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' 2024-05-27T07:15:32.374 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:15:32 smithi052 ceph-mon[31955]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T07:15:32.375 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:15:32 smithi052 ceph-mon[31955]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-27T07:15:32.375 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:15:32 smithi052 ceph-mon[31955]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' 2024-05-27T07:15:32.375 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:15:32 smithi052 ceph-mon[31955]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' 2024-05-27T07:15:32.375 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:15:32 smithi052 ceph-mon[31955]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' 2024-05-27T07:15:32.375 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:15:32 smithi052 ceph-mon[31955]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' 2024-05-27T07:15:32.375 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:15:32 smithi052 ceph-mon[31955]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' 2024-05-27T07:15:32.375 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:15:32 smithi052 ceph-mon[31955]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' 2024-05-27T07:15:32.375 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:15:32 smithi052 ceph-mon[31955]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' 2024-05-27T07:15:32.441 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:15:32 smithi105 ceph-mon[35977]: from='client.? 172.21.15.173:0/3412253802' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-05-27T07:15:32.441 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:15:32 smithi105 ceph-mon[35977]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' 2024-05-27T07:15:32.441 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:15:32 smithi105 ceph-mon[35977]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' 2024-05-27T07:15:32.441 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:15:32 smithi105 ceph-mon[35977]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' 2024-05-27T07:15:32.441 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:15:32 smithi105 ceph-mon[35977]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' 2024-05-27T07:15:32.441 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:15:32 smithi105 ceph-mon[35977]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T07:15:32.441 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:15:32 smithi105 ceph-mon[35977]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-27T07:15:32.441 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:15:32 smithi105 ceph-mon[35977]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' 2024-05-27T07:15:32.441 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:15:32 smithi105 ceph-mon[35977]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' 2024-05-27T07:15:32.441 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:15:32 smithi105 ceph-mon[35977]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' 2024-05-27T07:15:32.441 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:15:32 smithi105 ceph-mon[35977]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' 2024-05-27T07:15:32.442 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:15:32 smithi105 ceph-mon[35977]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' 2024-05-27T07:15:32.442 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:15:32 smithi105 ceph-mon[35977]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' 2024-05-27T07:15:32.442 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:15:32 smithi105 ceph-mon[35977]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' 2024-05-27T07:15:33.230 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:15:33 smithi052 ceph-mon[31955]: Updating smithi052:/etc/ceph/ceph.conf 2024-05-27T07:15:33.230 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:15:33 smithi052 ceph-mon[31955]: Updating smithi105:/etc/ceph/ceph.conf 2024-05-27T07:15:33.230 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:15:33 smithi052 ceph-mon[31955]: Updating smithi173:/etc/ceph/ceph.conf 2024-05-27T07:15:33.230 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:15:33 smithi052 ceph-mon[31955]: pgmap v25: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-27T07:15:33.231 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:15:33 smithi052 ceph-mon[31955]: Updating smithi052:/var/lib/ceph/73df2636-1bf8-11ef-bc9b-c7b262605968/config/ceph.conf 2024-05-27T07:15:33.231 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:15:33 smithi052 ceph-mon[31955]: Updating smithi105:/var/lib/ceph/73df2636-1bf8-11ef-bc9b-c7b262605968/config/ceph.conf 2024-05-27T07:15:33.231 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:15:33 smithi052 ceph-mon[31955]: Updating smithi173:/var/lib/ceph/73df2636-1bf8-11ef-bc9b-c7b262605968/config/ceph.conf 2024-05-27T07:15:33.231 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:15:33 smithi052 ceph-mon[31955]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' 2024-05-27T07:15:33.231 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:15:33 smithi052 ceph-mon[31955]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' 2024-05-27T07:15:33.231 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:15:33 smithi052 ceph-mon[31955]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' 2024-05-27T07:15:33.231 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:15:33 smithi052 ceph-mon[31955]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' 2024-05-27T07:15:33.231 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:15:33 smithi052 ceph-mon[31955]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-05-27T07:15:33.231 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:15:33 smithi052 ceph-mon[31955]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-05-27T07:15:33.231 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:15:33 smithi052 ceph-mon[31955]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T07:15:33.352 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:15:33 smithi173 ceph-mon[37000]: Updating smithi052:/etc/ceph/ceph.conf 2024-05-27T07:15:33.353 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:15:33 smithi173 ceph-mon[37000]: Updating smithi105:/etc/ceph/ceph.conf 2024-05-27T07:15:33.353 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:15:33 smithi173 ceph-mon[37000]: Updating smithi173:/etc/ceph/ceph.conf 2024-05-27T07:15:33.353 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:15:33 smithi173 ceph-mon[37000]: pgmap v25: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-27T07:15:33.353 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:15:33 smithi173 ceph-mon[37000]: Updating smithi052:/var/lib/ceph/73df2636-1bf8-11ef-bc9b-c7b262605968/config/ceph.conf 2024-05-27T07:15:33.353 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:15:33 smithi173 ceph-mon[37000]: Updating smithi105:/var/lib/ceph/73df2636-1bf8-11ef-bc9b-c7b262605968/config/ceph.conf 2024-05-27T07:15:33.353 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:15:33 smithi173 ceph-mon[37000]: Updating smithi173:/var/lib/ceph/73df2636-1bf8-11ef-bc9b-c7b262605968/config/ceph.conf 2024-05-27T07:15:33.353 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:15:33 smithi173 ceph-mon[37000]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' 2024-05-27T07:15:33.353 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:15:33 smithi173 ceph-mon[37000]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' 2024-05-27T07:15:33.353 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:15:33 smithi173 ceph-mon[37000]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' 2024-05-27T07:15:33.353 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:15:33 smithi173 ceph-mon[37000]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' 2024-05-27T07:15:33.353 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:15:33 smithi173 ceph-mon[37000]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-05-27T07:15:33.353 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:15:33 smithi173 ceph-mon[37000]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-05-27T07:15:33.353 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:15:33 smithi173 ceph-mon[37000]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T07:15:33.441 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:15:33 smithi105 ceph-mon[35977]: Updating smithi052:/etc/ceph/ceph.conf 2024-05-27T07:15:33.441 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:15:33 smithi105 ceph-mon[35977]: Updating smithi105:/etc/ceph/ceph.conf 2024-05-27T07:15:33.441 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:15:33 smithi105 ceph-mon[35977]: Updating smithi173:/etc/ceph/ceph.conf 2024-05-27T07:15:33.441 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:15:33 smithi105 ceph-mon[35977]: pgmap v25: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-27T07:15:33.441 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:15:33 smithi105 ceph-mon[35977]: Updating smithi052:/var/lib/ceph/73df2636-1bf8-11ef-bc9b-c7b262605968/config/ceph.conf 2024-05-27T07:15:33.441 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:15:33 smithi105 ceph-mon[35977]: Updating smithi105:/var/lib/ceph/73df2636-1bf8-11ef-bc9b-c7b262605968/config/ceph.conf 2024-05-27T07:15:33.441 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:15:33 smithi105 ceph-mon[35977]: Updating smithi173:/var/lib/ceph/73df2636-1bf8-11ef-bc9b-c7b262605968/config/ceph.conf 2024-05-27T07:15:33.441 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:15:33 smithi105 ceph-mon[35977]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' 2024-05-27T07:15:33.441 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:15:33 smithi105 ceph-mon[35977]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' 2024-05-27T07:15:33.441 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:15:33 smithi105 ceph-mon[35977]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' 2024-05-27T07:15:33.442 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:15:33 smithi105 ceph-mon[35977]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' 2024-05-27T07:15:33.442 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:15:33 smithi105 ceph-mon[35977]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-05-27T07:15:33.442 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:15:33 smithi105 ceph-mon[35977]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-05-27T07:15:33.442 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:15:33 smithi105 ceph-mon[35977]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T07:15:34.352 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:15:34 smithi173 ceph-mon[37000]: Reconfiguring mon.a (unknown last config time)... 2024-05-27T07:15:34.353 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:15:34 smithi173 ceph-mon[37000]: Reconfiguring daemon mon.a on smithi052 2024-05-27T07:15:34.353 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:15:34 smithi173 ceph-mon[37000]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' 2024-05-27T07:15:34.353 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:15:34 smithi173 ceph-mon[37000]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' 2024-05-27T07:15:34.353 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:15:34 smithi173 ceph-mon[37000]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-05-27T07:15:34.353 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:15:34 smithi173 ceph-mon[37000]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-05-27T07:15:34.353 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:15:34 smithi173 ceph-mon[37000]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T07:15:34.378 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:15:34 smithi052 ceph-mon[31955]: Reconfiguring mon.a (unknown last config time)... 2024-05-27T07:15:34.378 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:15:34 smithi052 ceph-mon[31955]: Reconfiguring daemon mon.a on smithi052 2024-05-27T07:15:34.378 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:15:34 smithi052 ceph-mon[31955]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' 2024-05-27T07:15:34.378 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:15:34 smithi052 ceph-mon[31955]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' 2024-05-27T07:15:34.378 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:15:34 smithi052 ceph-mon[31955]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-05-27T07:15:34.378 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:15:34 smithi052 ceph-mon[31955]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-05-27T07:15:34.378 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:15:34 smithi052 ceph-mon[31955]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T07:15:34.397 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:15:34 smithi105 ceph-mon[35977]: Reconfiguring mon.a (unknown last config time)... 2024-05-27T07:15:34.397 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:15:34 smithi105 ceph-mon[35977]: Reconfiguring daemon mon.a on smithi052 2024-05-27T07:15:34.397 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:15:34 smithi105 ceph-mon[35977]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' 2024-05-27T07:15:34.397 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:15:34 smithi105 ceph-mon[35977]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' 2024-05-27T07:15:34.397 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:15:34 smithi105 ceph-mon[35977]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-05-27T07:15:34.397 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:15:34 smithi105 ceph-mon[35977]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-05-27T07:15:34.397 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:15:34 smithi105 ceph-mon[35977]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T07:15:34.758 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:15:34.767+0000 7f76fc839700 1 -- 172.21.15.52:0/2857709669 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f76f81034b0 msgr2=0x7f76f81038c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:15:34.759 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:15:34.767+0000 7f76fc839700 1 --2- 172.21.15.52:0/2857709669 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f76f81034b0 0x7f76f81038c0 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f76e800b0d0 tx=0x7f76e800b3e0 comp rx=0 tx=0).stop 2024-05-27T07:15:34.759 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:15:34.767+0000 7f76fc839700 1 -- 172.21.15.52:0/2857709669 shutdown_connections 2024-05-27T07:15:34.759 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:15:34.767+0000 7f76fc839700 1 --2- 172.21.15.52:0/2857709669 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f76f8108730 0x7f76f8110090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:15:34.759 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:15:34.767+0000 7f76fc839700 1 --2- 172.21.15.52:0/2857709669 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f76f81034b0 0x7f76f81038c0 unknown :-1 s=CLOSED pgs=107 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:15:34.759 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:15:34.767+0000 7f76fc839700 1 --2- 172.21.15.52:0/2857709669 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f76f8107de0 0x7f76f81081f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:15:34.759 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:15:34.767+0000 7f76fc839700 1 -- 172.21.15.52:0/2857709669 >> 172.21.15.52:0/2857709669 conn(0x7f76f80fedb0 msgr2=0x7f76f81011d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:15:34.759 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:15:34.767+0000 7f76fc839700 1 -- 172.21.15.52:0/2857709669 shutdown_connections 2024-05-27T07:15:34.759 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:15:34.767+0000 7f76fc839700 1 -- 172.21.15.52:0/2857709669 wait complete. 2024-05-27T07:15:34.761 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:15:34.768+0000 7f76fc839700 1 Processor -- start 2024-05-27T07:15:34.761 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:15:34.768+0000 7f76fc839700 1 -- start start 2024-05-27T07:15:34.761 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:15:34.768+0000 7f76fc839700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f76f81034b0 0x7f76f819bc60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:15:34.761 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:15:34.768+0000 7f76fc839700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f76f8107de0 0x7f76f819c1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:15:34.761 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:15:34.768+0000 7f76fc839700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f76f8108730 0x7f76f819c700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:15:34.761 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:15:34.768+0000 7f76fc839700 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x7f76f8113360 con 0x7f76f8108730 2024-05-27T07:15:34.761 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:15:34.768+0000 7f76fc839700 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f76f81131e0 con 0x7f76f8107de0 2024-05-27T07:15:34.761 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:15:34.768+0000 7f76fc839700 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f76f81134e0 con 0x7f76f81034b0 2024-05-27T07:15:34.762 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:15:34.768+0000 7f76f6d9d700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f76f8108730 0x7f76f819c700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:15:34.762 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:15:34.768+0000 7f76f6d9d700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f76f8108730 0x7f76f819c700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.52:34514/0 (socket says 172.21.15.52:34514) 2024-05-27T07:15:34.762 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:15:34.768+0000 7f76f6d9d700 1 -- 172.21.15.52:0/1446107143 learned_addr learned my addr 172.21.15.52:0/1446107143 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-05-27T07:15:34.762 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:15:34.768+0000 7f76f5d9b700 1 --2- 172.21.15.52:0/1446107143 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f76f8107de0 0x7f76f819c1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:15:34.762 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:15:34.768+0000 7f76f659c700 1 --2- 172.21.15.52:0/1446107143 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f76f81034b0 0x7f76f819bc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:15:34.762 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:15:34.768+0000 7f76f6d9d700 1 -- 172.21.15.52:0/1446107143 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f76f81034b0 msgr2=0x7f76f819bc60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:15:34.762 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:15:34.768+0000 7f76f6d9d700 1 --2- 172.21.15.52:0/1446107143 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f76f81034b0 0x7f76f819bc60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:15:34.762 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:15:34.768+0000 7f76f6d9d700 1 -- 172.21.15.52:0/1446107143 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f76f8107de0 msgr2=0x7f76f819c1a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:15:34.762 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:15:34.768+0000 7f76f6d9d700 1 --2- 172.21.15.52:0/1446107143 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f76f8107de0 0x7f76f819c1a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:15:34.762 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:15:34.768+0000 7f76f6d9d700 1 -- 172.21.15.52:0/1446107143 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f76f81a6fb0 con 0x7f76f8108730 2024-05-27T07:15:34.762 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:15:34.769+0000 7f76f6d9d700 1 --2- 172.21.15.52:0/1446107143 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f76f8108730 0x7f76f819c700 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7f76f000d980 tx=0x7f76f000dc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:15:34.762 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:15:34.769+0000 7f76e77fe700 1 -- 172.21.15.52:0/1446107143 <== mon.0 v2:172.21.15.52:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f76f0017070 con 0x7f76f8108730 2024-05-27T07:15:34.763 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:15:34.769+0000 7f76fc839700 1 -- 172.21.15.52:0/1446107143 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f76f81a72a0 con 0x7f76f8108730 2024-05-27T07:15:34.763 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:15:34.769+0000 7f76e77fe700 1 -- 172.21.15.52:0/1446107143 <== mon.0 v2:172.21.15.52:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f76f0005020 con 0x7f76f8108730 2024-05-27T07:15:34.763 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:15:34.769+0000 7f76e77fe700 1 -- 172.21.15.52:0/1446107143 <== mon.0 v2:172.21.15.52:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f76f00099b0 con 0x7f76f8108730 2024-05-27T07:15:34.763 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:15:34.769+0000 7f76fc839700 1 -- 172.21.15.52:0/1446107143 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f76f81a7790 con 0x7f76f8108730 2024-05-27T07:15:34.763 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:15:34.770+0000 7f76fc839700 1 -- 172.21.15.52:0/1446107143 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f76f80512b0 con 0x7f76f8108730 2024-05-27T07:15:34.764 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:15:34.770+0000 7f76e77fe700 1 -- 172.21.15.52:0/1446107143 <== mon.0 v2:172.21.15.52:3300/0 4 ==== mgrmap(e 13) ==== 50278+0+0 (secure 0 0 0) 0x7f76f0009b90 con 0x7f76f8108730 2024-05-27T07:15:34.764 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:15:34.770+0000 7f76e77fe700 1 --2- 172.21.15.52:0/1446107143 >> [v2:172.21.15.52:6800/3513075805,v1:172.21.15.52:6801/3513075805] conn(0x7f76e003e750 0x7f76e0040c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:15:34.764 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:15:34.770+0000 7f76e77fe700 1 -- 172.21.15.52:0/1446107143 <== mon.0 v2:172.21.15.52:3300/0 5 ==== osd_map(4..4 src has 1..4) ==== 1155+0+0 (secure 0 0 0) 0x7f76f0052970 con 0x7f76f8108730 2024-05-27T07:15:34.764 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:15:34.771+0000 7f76f659c700 1 --2- 172.21.15.52:0/1446107143 >> [v2:172.21.15.52:6800/3513075805,v1:172.21.15.52:6801/3513075805] conn(0x7f76e003e750 0x7f76e0040c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:15:34.764 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:15:34.771+0000 7f76f659c700 1 --2- 172.21.15.52:0/1446107143 >> [v2:172.21.15.52:6800/3513075805,v1:172.21.15.52:6801/3513075805] conn(0x7f76e003e750 0x7f76e0040c60 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f76e8009fd0 tx=0x7f76e800b040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:15:34.767 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:15:34.776+0000 7f76e77fe700 1 -- 172.21.15.52:0/1446107143 <== mon.0 v2:172.21.15.52:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191963 (secure 0 0 0) 0x7f76f0020020 con 0x7f76f8108730 2024-05-27T07:15:34.981 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:15:34.989+0000 7f76fc839700 1 -- 172.21.15.52:0/1446107143 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_command({"prefix": "config generate-minimal-conf"} v 0) -- 0x7f76f802e5e0 con 0x7f76f8108730 2024-05-27T07:15:34.981 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:15:34.989+0000 7f76e77fe700 1 -- 172.21.15.52:0/1446107143 <== mon.0 v2:172.21.15.52:3300/0 7 ==== mon_command_ack([{"prefix": "config generate-minimal-conf"}]=0 v9) ==== 76+0+275 (secure 0 0 0) 0x7f76f00051d0 con 0x7f76f8108730 2024-05-27T07:15:34.981 INFO:teuthology.orchestra.run.smithi052.stdout:# minimal ceph.conf for 73df2636-1bf8-11ef-bc9b-c7b262605968 2024-05-27T07:15:34.981 INFO:teuthology.orchestra.run.smithi052.stdout:[global] 2024-05-27T07:15:34.981 INFO:teuthology.orchestra.run.smithi052.stdout: fsid = 73df2636-1bf8-11ef-bc9b-c7b262605968 2024-05-27T07:15:34.981 INFO:teuthology.orchestra.run.smithi052.stdout: mon_host = [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] 2024-05-27T07:15:34.984 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:15:34.992+0000 7f76fc839700 1 -- 172.21.15.52:0/1446107143 >> [v2:172.21.15.52:6800/3513075805,v1:172.21.15.52:6801/3513075805] conn(0x7f76e003e750 msgr2=0x7f76e0040c60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:15:34.984 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:15:34.992+0000 7f76fc839700 1 --2- 172.21.15.52:0/1446107143 >> [v2:172.21.15.52:6800/3513075805,v1:172.21.15.52:6801/3513075805] conn(0x7f76e003e750 0x7f76e0040c60 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f76e8009fd0 tx=0x7f76e800b040 comp rx=0 tx=0).stop 2024-05-27T07:15:34.984 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:15:34.992+0000 7f76fc839700 1 -- 172.21.15.52:0/1446107143 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f76f8108730 msgr2=0x7f76f819c700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:15:34.984 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:15:34.992+0000 7f76fc839700 1 --2- 172.21.15.52:0/1446107143 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f76f8108730 0x7f76f819c700 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7f76f000d980 tx=0x7f76f000dc90 comp rx=0 tx=0).stop 2024-05-27T07:15:34.984 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:15:34.992+0000 7f76fc839700 1 -- 172.21.15.52:0/1446107143 shutdown_connections 2024-05-27T07:15:34.984 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:15:34.992+0000 7f76fc839700 1 --2- 172.21.15.52:0/1446107143 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f76f8108730 0x7f76f819c700 unknown :-1 s=CLOSED pgs=108 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:15:34.984 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:15:34.992+0000 7f76fc839700 1 --2- 172.21.15.52:0/1446107143 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f76f81034b0 0x7f76f819bc60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:15:34.984 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:15:34.992+0000 7f76fc839700 1 --2- 172.21.15.52:0/1446107143 >> [v2:172.21.15.52:6800/3513075805,v1:172.21.15.52:6801/3513075805] conn(0x7f76e003e750 0x7f76e0040c60 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:15:34.984 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:15:34.992+0000 7f76fc839700 1 --2- 172.21.15.52:0/1446107143 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f76f8107de0 0x7f76f819c1a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:15:34.984 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:15:34.992+0000 7f76fc839700 1 -- 172.21.15.52:0/1446107143 >> 172.21.15.52:0/1446107143 conn(0x7f76f80fedb0 msgr2=0x7f76f8109690 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:15:34.985 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:15:34.992+0000 7f76fc839700 1 -- 172.21.15.52:0/1446107143 shutdown_connections 2024-05-27T07:15:34.985 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:15:34.992+0000 7f76fc839700 1 -- 172.21.15.52:0/1446107143 wait complete. 2024-05-27T07:15:35.291 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:15:35 smithi052 ceph-mon[31955]: pgmap v26: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-27T07:15:35.291 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:15:35 smithi052 ceph-mon[31955]: Reconfiguring mon.b (monmap changed)... 2024-05-27T07:15:35.292 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:15:35 smithi052 ceph-mon[31955]: Reconfiguring daemon mon.b on smithi105 2024-05-27T07:15:35.292 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:15:35 smithi052 ceph-mon[31955]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' 2024-05-27T07:15:35.292 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:15:35 smithi052 ceph-mon[31955]: from='client.? 172.21.15.52:0/1446107143' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T07:15:35.352 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:15:35 smithi173 ceph-mon[37000]: pgmap v26: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-27T07:15:35.352 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:15:35 smithi173 ceph-mon[37000]: Reconfiguring mon.b (monmap changed)... 2024-05-27T07:15:35.353 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:15:35 smithi173 ceph-mon[37000]: Reconfiguring daemon mon.b on smithi105 2024-05-27T07:15:35.353 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:15:35 smithi173 ceph-mon[37000]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' 2024-05-27T07:15:35.353 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:15:35 smithi173 ceph-mon[37000]: from='client.? 172.21.15.52:0/1446107143' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T07:15:35.404 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:15:35 smithi105 ceph-mon[35977]: pgmap v26: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-27T07:15:35.404 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:15:35 smithi105 ceph-mon[35977]: Reconfiguring mon.b (monmap changed)... 2024-05-27T07:15:35.404 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:15:35 smithi105 ceph-mon[35977]: Reconfiguring daemon mon.b on smithi105 2024-05-27T07:15:35.404 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:15:35 smithi105 ceph-mon[35977]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' 2024-05-27T07:15:35.405 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:15:35 smithi105 ceph-mon[35977]: from='client.? 172.21.15.52:0/1446107143' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T07:15:35.567 INFO:tasks.cephadm:Distributing (final) config and client.admin keyring... 2024-05-27T07:15:35.567 DEBUG:teuthology.orchestra.run.smithi052:> set -ex 2024-05-27T07:15:35.567 DEBUG:teuthology.orchestra.run.smithi052:> sudo dd of=/etc/ceph/ceph.conf 2024-05-27T07:15:35.596 DEBUG:teuthology.orchestra.run.smithi052:> set -ex 2024-05-27T07:15:35.596 DEBUG:teuthology.orchestra.run.smithi052:> sudo dd of=/etc/ceph/ceph.client.admin.keyring 2024-05-27T07:15:35.662 DEBUG:teuthology.orchestra.run.smithi105:> set -ex 2024-05-27T07:15:35.662 DEBUG:teuthology.orchestra.run.smithi105:> sudo dd of=/etc/ceph/ceph.conf 2024-05-27T07:15:35.689 DEBUG:teuthology.orchestra.run.smithi105:> set -ex 2024-05-27T07:15:35.689 DEBUG:teuthology.orchestra.run.smithi105:> sudo dd of=/etc/ceph/ceph.client.admin.keyring 2024-05-27T07:15:35.753 DEBUG:teuthology.orchestra.run.smithi173:> set -ex 2024-05-27T07:15:35.753 DEBUG:teuthology.orchestra.run.smithi173:> sudo dd of=/etc/ceph/ceph.conf 2024-05-27T07:15:35.782 DEBUG:teuthology.orchestra.run.smithi173:> set -ex 2024-05-27T07:15:35.782 DEBUG:teuthology.orchestra.run.smithi173:> sudo dd of=/etc/ceph/ceph.client.admin.keyring 2024-05-27T07:15:35.846 INFO:tasks.cephadm:Adding mgr.x on smithi052 2024-05-27T07:15:35.846 INFO:tasks.cephadm:Adding mgr.y on smithi105 2024-05-27T07:15:35.846 INFO:tasks.cephadm:Adding mgr.z on smithi173 2024-05-27T07:15:35.846 DEBUG:teuthology.orchestra.run.smithi173:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:38f9d900c83c63799fbdbe61acc9a11b0d3554a6 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 73df2636-1bf8-11ef-bc9b-c7b262605968 -- ceph orch apply mgr '3;smithi052=x;smithi105=y;smithi173=z' 2024-05-27T07:15:36.353 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:15:36 smithi173 ceph-mon[37000]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' 2024-05-27T07:15:36.353 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:15:36 smithi173 ceph-mon[37000]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' 2024-05-27T07:15:36.353 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:15:36 smithi173 ceph-mon[37000]: Reconfiguring mon.c (monmap changed)... 2024-05-27T07:15:36.353 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:15:36 smithi173 ceph-mon[37000]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-05-27T07:15:36.353 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:15:36 smithi173 ceph-mon[37000]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-05-27T07:15:36.353 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:15:36 smithi173 ceph-mon[37000]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T07:15:36.353 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:15:36 smithi173 ceph-mon[37000]: Reconfiguring daemon mon.c on smithi173 2024-05-27T07:15:36.353 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:15:36 smithi173 ceph-mon[37000]: pgmap v27: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-27T07:15:36.691 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:15:36 smithi105 ceph-mon[35977]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' 2024-05-27T07:15:36.691 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:15:36 smithi105 ceph-mon[35977]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' 2024-05-27T07:15:36.691 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:15:36 smithi105 ceph-mon[35977]: Reconfiguring mon.c (monmap changed)... 2024-05-27T07:15:36.691 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:15:36 smithi105 ceph-mon[35977]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-05-27T07:15:36.691 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:15:36 smithi105 ceph-mon[35977]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-05-27T07:15:36.691 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:15:36 smithi105 ceph-mon[35977]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T07:15:36.691 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:15:36 smithi105 ceph-mon[35977]: Reconfiguring daemon mon.c on smithi173 2024-05-27T07:15:36.691 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:15:36 smithi105 ceph-mon[35977]: pgmap v27: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-27T07:15:36.730 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:15:36 smithi052 ceph-mon[31955]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' 2024-05-27T07:15:36.730 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:15:36 smithi052 ceph-mon[31955]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' 2024-05-27T07:15:36.730 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:15:36 smithi052 ceph-mon[31955]: Reconfiguring mon.c (monmap changed)... 2024-05-27T07:15:36.730 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:15:36 smithi052 ceph-mon[31955]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-05-27T07:15:36.730 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:15:36 smithi052 ceph-mon[31955]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-05-27T07:15:36.730 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:15:36 smithi052 ceph-mon[31955]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T07:15:36.730 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:15:36 smithi052 ceph-mon[31955]: Reconfiguring daemon mon.c on smithi173 2024-05-27T07:15:36.731 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:15:36 smithi052 ceph-mon[31955]: pgmap v27: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-27T07:15:36.852 INFO:teuthology.orchestra.run.smithi173.stderr:Inferring config /var/lib/ceph/73df2636-1bf8-11ef-bc9b-c7b262605968/mon.c/config 2024-05-27T07:15:37.988 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:15:37 smithi173 ceph-mon[37000]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' 2024-05-27T07:15:37.988 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:15:37 smithi173 ceph-mon[37000]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' 2024-05-27T07:15:37.988 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:15:37 smithi173 ceph-mon[37000]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-27T07:15:37.988 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:15:37 smithi173 ceph-mon[37000]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T07:15:37.988 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:15:37 smithi173 ceph-mon[37000]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-27T07:15:37.989 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:15:37 smithi173 ceph-mon[37000]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' 2024-05-27T07:15:38.101 INFO:teuthology.orchestra.run.smithi173.stderr:2024-05-27T07:15:38.110+0000 7fa968ce0700 1 -- 172.21.15.173:0/1366229072 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fa964104350 msgr2=0x7fa964104760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:15:38.101 INFO:teuthology.orchestra.run.smithi173.stderr:2024-05-27T07:15:38.110+0000 7fa968ce0700 1 --2- 172.21.15.173:0/1366229072 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fa964104350 0x7fa964104760 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7fa94c009990 tx=0x7fa94c009ca0 comp rx=0 tx=0).stop 2024-05-27T07:15:38.101 INFO:teuthology.orchestra.run.smithi173.stderr:2024-05-27T07:15:38.110+0000 7fa968ce0700 1 -- 172.21.15.173:0/1366229072 shutdown_connections 2024-05-27T07:15:38.101 INFO:teuthology.orchestra.run.smithi173.stderr:2024-05-27T07:15:38.110+0000 7fa968ce0700 1 --2- 172.21.15.173:0/1366229072 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fa964104cf0 0x7fa96410b5a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:15:38.101 INFO:teuthology.orchestra.run.smithi173.stderr:2024-05-27T07:15:38.110+0000 7fa968ce0700 1 --2- 172.21.15.173:0/1366229072 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fa964103220 0x7fa964103630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:15:38.101 INFO:teuthology.orchestra.run.smithi173.stderr:2024-05-27T07:15:38.110+0000 7fa968ce0700 1 --2- 172.21.15.173:0/1366229072 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fa964104350 0x7fa964104760 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:15:38.101 INFO:teuthology.orchestra.run.smithi173.stderr:2024-05-27T07:15:38.110+0000 7fa968ce0700 1 -- 172.21.15.173:0/1366229072 >> 172.21.15.173:0/1366229072 conn(0x7fa9640fe880 msgr2=0x7fa964100d10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:15:38.101 INFO:teuthology.orchestra.run.smithi173.stderr:2024-05-27T07:15:38.110+0000 7fa968ce0700 1 -- 172.21.15.173:0/1366229072 shutdown_connections 2024-05-27T07:15:38.101 INFO:teuthology.orchestra.run.smithi173.stderr:2024-05-27T07:15:38.110+0000 7fa968ce0700 1 -- 172.21.15.173:0/1366229072 wait complete. 2024-05-27T07:15:38.104 INFO:teuthology.orchestra.run.smithi173.stderr:2024-05-27T07:15:38.110+0000 7fa968ce0700 1 Processor -- start 2024-05-27T07:15:38.104 INFO:teuthology.orchestra.run.smithi173.stderr:2024-05-27T07:15:38.110+0000 7fa968ce0700 1 -- start start 2024-05-27T07:15:38.104 INFO:teuthology.orchestra.run.smithi173.stderr:2024-05-27T07:15:38.111+0000 7fa968ce0700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fa964103220 0x7fa9641998c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:15:38.104 INFO:teuthology.orchestra.run.smithi173.stderr:2024-05-27T07:15:38.111+0000 7fa968ce0700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fa964104350 0x7fa964199e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:15:38.104 INFO:teuthology.orchestra.run.smithi173.stderr:2024-05-27T07:15:38.111+0000 7fa968ce0700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fa964104cf0 0x7fa9641a0da0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:15:38.104 INFO:teuthology.orchestra.run.smithi173.stderr:2024-05-27T07:15:38.111+0000 7fa968ce0700 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x7fa96410ed50 con 0x7fa964104cf0 2024-05-27T07:15:38.104 INFO:teuthology.orchestra.run.smithi173.stderr:2024-05-27T07:15:38.111+0000 7fa968ce0700 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7fa96410ebd0 con 0x7fa964103220 2024-05-27T07:15:38.105 INFO:teuthology.orchestra.run.smithi173.stderr:2024-05-27T07:15:38.111+0000 7fa968ce0700 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7fa96410eed0 con 0x7fa964104350 2024-05-27T07:15:38.105 INFO:teuthology.orchestra.run.smithi173.stderr:2024-05-27T07:15:38.111+0000 7fa95bfff700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fa964104350 0x7fa964199e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:15:38.105 INFO:teuthology.orchestra.run.smithi173.stderr:2024-05-27T07:15:38.111+0000 7fa95bfff700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fa964104350 0x7fa964199e00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.173:3300/0 says I am v2:172.21.15.173:34718/0 (socket says 172.21.15.173:34718) 2024-05-27T07:15:38.105 INFO:teuthology.orchestra.run.smithi173.stderr:2024-05-27T07:15:38.111+0000 7fa95bfff700 1 -- 172.21.15.173:0/3499476237 learned_addr learned my addr 172.21.15.173:0/3499476237 (peer_addr_for_me v2:172.21.15.173:0/0) 2024-05-27T07:15:38.105 INFO:teuthology.orchestra.run.smithi173.stderr:2024-05-27T07:15:38.111+0000 7fa96259c700 1 --2- 172.21.15.173:0/3499476237 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fa964103220 0x7fa9641998c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:15:38.105 INFO:teuthology.orchestra.run.smithi173.stderr:2024-05-27T07:15:38.111+0000 7fa962d9d700 1 --2- 172.21.15.173:0/3499476237 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fa964104cf0 0x7fa9641a0da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:15:38.105 INFO:teuthology.orchestra.run.smithi173.stderr:2024-05-27T07:15:38.111+0000 7fa95bfff700 1 -- 172.21.15.173:0/3499476237 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fa964103220 msgr2=0x7fa9641998c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:15:38.105 INFO:teuthology.orchestra.run.smithi173.stderr:2024-05-27T07:15:38.111+0000 7fa95bfff700 1 --2- 172.21.15.173:0/3499476237 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fa964103220 0x7fa9641998c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:15:38.105 INFO:teuthology.orchestra.run.smithi173.stderr:2024-05-27T07:15:38.111+0000 7fa95bfff700 1 -- 172.21.15.173:0/3499476237 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fa964104cf0 msgr2=0x7fa9641a0da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:15:38.105 INFO:teuthology.orchestra.run.smithi173.stderr:2024-05-27T07:15:38.111+0000 7fa95bfff700 1 --2- 172.21.15.173:0/3499476237 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fa964104cf0 0x7fa9641a0da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:15:38.105 INFO:teuthology.orchestra.run.smithi173.stderr:2024-05-27T07:15:38.111+0000 7fa95bfff700 1 -- 172.21.15.173:0/3499476237 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa9641a1420 con 0x7fa964104350 2024-05-27T07:15:38.105 INFO:teuthology.orchestra.run.smithi173.stderr:2024-05-27T07:15:38.111+0000 7fa95bfff700 1 --2- 172.21.15.173:0/3499476237 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fa964104350 0x7fa964199e00 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7fa94c00b760 tx=0x7fa94c004790 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:15:38.106 INFO:teuthology.orchestra.run.smithi173.stderr:2024-05-27T07:15:38.111+0000 7fa95b7fe700 1 -- 172.21.15.173:0/3499476237 <== mon.1 v2:172.21.15.173:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa94c028070 con 0x7fa964104350 2024-05-27T07:15:38.106 INFO:teuthology.orchestra.run.smithi173.stderr:2024-05-27T07:15:38.112+0000 7fa95b7fe700 1 -- 172.21.15.173:0/3499476237 <== mon.1 v2:172.21.15.173:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fa94c01ad40 con 0x7fa964104350 2024-05-27T07:15:38.106 INFO:teuthology.orchestra.run.smithi173.stderr:2024-05-27T07:15:38.112+0000 7fa968ce0700 1 -- 172.21.15.173:0/3499476237 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa9640759e0 con 0x7fa964104350 2024-05-27T07:15:38.106 INFO:teuthology.orchestra.run.smithi173.stderr:2024-05-27T07:15:38.112+0000 7fa95b7fe700 1 -- 172.21.15.173:0/3499476237 <== mon.1 v2:172.21.15.173:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa94c00f460 con 0x7fa964104350 2024-05-27T07:15:38.106 INFO:teuthology.orchestra.run.smithi173.stderr:2024-05-27T07:15:38.112+0000 7fa968ce0700 1 -- 172.21.15.173:0/3499476237 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa964075f40 con 0x7fa964104350 2024-05-27T07:15:38.106 INFO:teuthology.orchestra.run.smithi173.stderr:2024-05-27T07:15:38.112+0000 7fa968ce0700 1 -- 172.21.15.173:0/3499476237 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa9640512b0 con 0x7fa964104350 2024-05-27T07:15:38.106 INFO:teuthology.orchestra.run.smithi173.stderr:2024-05-27T07:15:38.113+0000 7fa95b7fe700 1 -- 172.21.15.173:0/3499476237 <== mon.1 v2:172.21.15.173:3300/0 4 ==== mgrmap(e 13) ==== 50278+0+0 (secure 0 0 0) 0x7fa94c02b070 con 0x7fa964104350 2024-05-27T07:15:38.106 INFO:teuthology.orchestra.run.smithi173.stderr:2024-05-27T07:15:38.113+0000 7fa95b7fe700 1 --2- 172.21.15.173:0/3499476237 >> [v2:172.21.15.52:6800/3513075805,v1:172.21.15.52:6801/3513075805] conn(0x7fa95003e750 0x7fa950040c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:15:38.106 INFO:teuthology.orchestra.run.smithi173.stderr:2024-05-27T07:15:38.113+0000 7fa95b7fe700 1 -- 172.21.15.173:0/3499476237 <== mon.1 v2:172.21.15.173:3300/0 5 ==== osd_map(4..4 src has 1..4) ==== 1155+0+0 (secure 0 0 0) 0x7fa94c0559a0 con 0x7fa964104350 2024-05-27T07:15:38.106 INFO:teuthology.orchestra.run.smithi173.stderr:2024-05-27T07:15:38.113+0000 7fa96259c700 1 --2- 172.21.15.173:0/3499476237 >> [v2:172.21.15.52:6800/3513075805,v1:172.21.15.52:6801/3513075805] conn(0x7fa95003e750 0x7fa950040c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:15:38.107 INFO:teuthology.orchestra.run.smithi173.stderr:2024-05-27T07:15:38.114+0000 7fa96259c700 1 --2- 172.21.15.173:0/3499476237 >> [v2:172.21.15.52:6800/3513075805,v1:172.21.15.52:6801/3513075805] conn(0x7fa95003e750 0x7fa950040c60 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7fa964104250 tx=0x7fa954009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:15:38.109 INFO:teuthology.orchestra.run.smithi173.stderr:2024-05-27T07:15:38.118+0000 7fa95b7fe700 1 -- 172.21.15.173:0/3499476237 <== mon.1 v2:172.21.15.173:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191963 (secure 0 0 0) 0x7fa94c013070 con 0x7fa964104350 2024-05-27T07:15:38.190 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:15:37 smithi105 ceph-mon[35977]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' 2024-05-27T07:15:38.190 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:15:37 smithi105 ceph-mon[35977]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' 2024-05-27T07:15:38.190 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:15:37 smithi105 ceph-mon[35977]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-27T07:15:38.190 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:15:37 smithi105 ceph-mon[35977]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T07:15:38.191 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:15:37 smithi105 ceph-mon[35977]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-27T07:15:38.191 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:15:37 smithi105 ceph-mon[35977]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' 2024-05-27T07:15:38.230 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:15:37 smithi052 ceph-mon[31955]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' 2024-05-27T07:15:38.230 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:15:37 smithi052 ceph-mon[31955]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' 2024-05-27T07:15:38.230 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:15:37 smithi052 ceph-mon[31955]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-27T07:15:38.230 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:15:37 smithi052 ceph-mon[31955]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T07:15:38.230 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:15:37 smithi052 ceph-mon[31955]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-27T07:15:38.230 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:15:37 smithi052 ceph-mon[31955]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' 2024-05-27T07:15:38.322 INFO:teuthology.orchestra.run.smithi173.stderr:2024-05-27T07:15:38.331+0000 7fa968ce0700 1 -- 172.21.15.173:0/3499476237 --> [v2:172.21.15.52:6800/3513075805,v1:172.21.15.52:6801/3513075805] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "mgr", "placement": "3;smithi052=x;smithi105=y;smithi173=z", "target": ["mon-mgr", ""]}) -- 0x7fa964076260 con 0x7fa95003e750 2024-05-27T07:15:38.329 INFO:teuthology.orchestra.run.smithi173.stderr:2024-05-27T07:15:38.338+0000 7fa95b7fe700 1 -- 172.21.15.173:0/3499476237 <== mgr.14152 v2:172.21.15.52:6800/3513075805 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+24 (secure 0 0 0) 0x7fa964076260 con 0x7fa95003e750 2024-05-27T07:15:38.329 INFO:teuthology.orchestra.run.smithi173.stdout:Scheduled mgr update... 2024-05-27T07:15:38.332 INFO:teuthology.orchestra.run.smithi173.stderr:2024-05-27T07:15:38.341+0000 7fa968ce0700 1 -- 172.21.15.173:0/3499476237 >> [v2:172.21.15.52:6800/3513075805,v1:172.21.15.52:6801/3513075805] conn(0x7fa95003e750 msgr2=0x7fa950040c60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:15:38.332 INFO:teuthology.orchestra.run.smithi173.stderr:2024-05-27T07:15:38.341+0000 7fa968ce0700 1 --2- 172.21.15.173:0/3499476237 >> [v2:172.21.15.52:6800/3513075805,v1:172.21.15.52:6801/3513075805] conn(0x7fa95003e750 0x7fa950040c60 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7fa964104250 tx=0x7fa954009260 comp rx=0 tx=0).stop 2024-05-27T07:15:38.332 INFO:teuthology.orchestra.run.smithi173.stderr:2024-05-27T07:15:38.341+0000 7fa968ce0700 1 -- 172.21.15.173:0/3499476237 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fa964104350 msgr2=0x7fa964199e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:15:38.332 INFO:teuthology.orchestra.run.smithi173.stderr:2024-05-27T07:15:38.341+0000 7fa968ce0700 1 --2- 172.21.15.173:0/3499476237 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fa964104350 0x7fa964199e00 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7fa94c00b760 tx=0x7fa94c004790 comp rx=0 tx=0).stop 2024-05-27T07:15:38.332 INFO:teuthology.orchestra.run.smithi173.stderr:2024-05-27T07:15:38.341+0000 7fa968ce0700 1 -- 172.21.15.173:0/3499476237 shutdown_connections 2024-05-27T07:15:38.332 INFO:teuthology.orchestra.run.smithi173.stderr:2024-05-27T07:15:38.341+0000 7fa968ce0700 1 --2- 172.21.15.173:0/3499476237 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fa964104cf0 0x7fa9641a0da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:15:38.332 INFO:teuthology.orchestra.run.smithi173.stderr:2024-05-27T07:15:38.341+0000 7fa968ce0700 1 --2- 172.21.15.173:0/3499476237 >> [v2:172.21.15.52:6800/3513075805,v1:172.21.15.52:6801/3513075805] conn(0x7fa95003e750 0x7fa950040c60 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:15:38.332 INFO:teuthology.orchestra.run.smithi173.stderr:2024-05-27T07:15:38.341+0000 7fa968ce0700 1 --2- 172.21.15.173:0/3499476237 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fa964103220 0x7fa9641998c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:15:38.332 INFO:teuthology.orchestra.run.smithi173.stderr:2024-05-27T07:15:38.341+0000 7fa968ce0700 1 --2- 172.21.15.173:0/3499476237 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fa964104350 0x7fa964199e00 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:15:38.332 INFO:teuthology.orchestra.run.smithi173.stderr:2024-05-27T07:15:38.341+0000 7fa968ce0700 1 -- 172.21.15.173:0/3499476237 >> 172.21.15.173:0/3499476237 conn(0x7fa9640fe880 msgr2=0x7fa964107580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:15:38.332 INFO:teuthology.orchestra.run.smithi173.stderr:2024-05-27T07:15:38.341+0000 7fa968ce0700 1 -- 172.21.15.173:0/3499476237 shutdown_connections 2024-05-27T07:15:38.333 INFO:teuthology.orchestra.run.smithi173.stderr:2024-05-27T07:15:38.341+0000 7fa968ce0700 1 -- 172.21.15.173:0/3499476237 wait complete. 2024-05-27T07:15:38.909 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:15:38 smithi173 ceph-mon[37000]: pgmap v28: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-27T07:15:38.909 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:15:38 smithi173 ceph-mon[37000]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' 2024-05-27T07:15:38.909 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:15:38 smithi173 ceph-mon[37000]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-27T07:15:38.909 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:15:38 smithi173 ceph-mon[37000]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T07:15:38.909 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:15:38 smithi173 ceph-mon[37000]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-27T07:15:38.909 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:15:38 smithi173 ceph-mon[37000]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' 2024-05-27T07:15:38.909 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:15:38 smithi173 ceph-mon[37000]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.z", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-05-27T07:15:38.909 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:15:38 smithi173 ceph-mon[37000]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.z", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-05-27T07:15:38.910 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:15:38 smithi173 ceph-mon[37000]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-05-27T07:15:38.910 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:15:38 smithi173 ceph-mon[37000]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T07:15:38.937 DEBUG:teuthology.orchestra.run.smithi105:mgr.y> sudo journalctl -f -n 0 -u ceph-73df2636-1bf8-11ef-bc9b-c7b262605968@mgr.y.service 2024-05-27T07:15:38.939 DEBUG:teuthology.orchestra.run.smithi173:mgr.z> sudo journalctl -f -n 0 -u ceph-73df2636-1bf8-11ef-bc9b-c7b262605968@mgr.z.service 2024-05-27T07:15:38.941 INFO:tasks.cephadm:Deploying OSDs... 2024-05-27T07:15:38.942 DEBUG:teuthology.orchestra.run.smithi052:> set -ex 2024-05-27T07:15:38.942 DEBUG:teuthology.orchestra.run.smithi052:> dd if=/scratch_devs of=/dev/stdout 2024-05-27T07:15:38.963 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-05-27T07:15:38.963 DEBUG:teuthology.orchestra.run.smithi052:> stat /dev/vg_nvme/lv_1 2024-05-27T07:15:39.023 INFO:teuthology.orchestra.run.smithi052.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-05-27T07:15:39.023 INFO:teuthology.orchestra.run.smithi052.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-05-27T07:15:39.023 INFO:teuthology.orchestra.run.smithi052.stdout:Device: 5h/5d Inode: 668 Links: 1 2024-05-27T07:15:39.023 INFO:teuthology.orchestra.run.smithi052.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-05-27T07:15:39.023 INFO:teuthology.orchestra.run.smithi052.stdout:Context: system_u:object_r:device_t:s0 2024-05-27T07:15:39.023 INFO:teuthology.orchestra.run.smithi052.stdout:Access: 2024-05-27 07:14:41.852566937 +0000 2024-05-27T07:15:39.023 INFO:teuthology.orchestra.run.smithi052.stdout:Modify: 2024-05-27 07:09:10.347295651 +0000 2024-05-27T07:15:39.023 INFO:teuthology.orchestra.run.smithi052.stdout:Change: 2024-05-27 07:09:10.347295651 +0000 2024-05-27T07:15:39.023 INFO:teuthology.orchestra.run.smithi052.stdout: Birth: 2024-05-27 07:09:10.347295651 +0000 2024-05-27T07:15:39.024 DEBUG:teuthology.orchestra.run.smithi052:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-05-27T07:15:39.092 INFO:teuthology.orchestra.run.smithi052.stderr:1+0 records in 2024-05-27T07:15:39.093 INFO:teuthology.orchestra.run.smithi052.stderr:1+0 records out 2024-05-27T07:15:39.093 INFO:teuthology.orchestra.run.smithi052.stderr:512 bytes copied, 0.000128957 s, 4.0 MB/s 2024-05-27T07:15:39.094 DEBUG:teuthology.orchestra.run.smithi052:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-05-27T07:15:39.153 DEBUG:teuthology.orchestra.run.smithi052:> stat /dev/vg_nvme/lv_2 2024-05-27T07:15:39.191 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:15:38 smithi105 ceph-mon[35977]: pgmap v28: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-27T07:15:39.191 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:15:38 smithi105 ceph-mon[35977]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' 2024-05-27T07:15:39.191 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:15:38 smithi105 ceph-mon[35977]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-27T07:15:39.191 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:15:38 smithi105 ceph-mon[35977]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T07:15:39.191 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:15:38 smithi105 ceph-mon[35977]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-27T07:15:39.191 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:15:38 smithi105 ceph-mon[35977]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' 2024-05-27T07:15:39.191 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:15:38 smithi105 ceph-mon[35977]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.z", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-05-27T07:15:39.192 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:15:38 smithi105 ceph-mon[35977]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.z", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-05-27T07:15:39.192 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:15:38 smithi105 ceph-mon[35977]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-05-27T07:15:39.192 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:15:38 smithi105 ceph-mon[35977]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T07:15:39.211 INFO:teuthology.orchestra.run.smithi052.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-05-27T07:15:39.211 INFO:teuthology.orchestra.run.smithi052.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-05-27T07:15:39.211 INFO:teuthology.orchestra.run.smithi052.stdout:Device: 5h/5d Inode: 686 Links: 1 2024-05-27T07:15:39.211 INFO:teuthology.orchestra.run.smithi052.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-05-27T07:15:39.211 INFO:teuthology.orchestra.run.smithi052.stdout:Context: system_u:object_r:device_t:s0 2024-05-27T07:15:39.211 INFO:teuthology.orchestra.run.smithi052.stdout:Access: 2024-05-27 07:14:41.889566513 +0000 2024-05-27T07:15:39.211 INFO:teuthology.orchestra.run.smithi052.stdout:Modify: 2024-05-27 07:09:10.677291891 +0000 2024-05-27T07:15:39.211 INFO:teuthology.orchestra.run.smithi052.stdout:Change: 2024-05-27 07:09:10.677291891 +0000 2024-05-27T07:15:39.211 INFO:teuthology.orchestra.run.smithi052.stdout: Birth: 2024-05-27 07:09:10.677291891 +0000 2024-05-27T07:15:39.211 DEBUG:teuthology.orchestra.run.smithi052:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-05-27T07:15:39.230 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:15:38 smithi052 ceph-mon[31955]: pgmap v28: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-27T07:15:39.230 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:15:38 smithi052 ceph-mon[31955]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' 2024-05-27T07:15:39.230 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:15:38 smithi052 ceph-mon[31955]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-27T07:15:39.230 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:15:38 smithi052 ceph-mon[31955]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T07:15:39.230 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:15:38 smithi052 ceph-mon[31955]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-27T07:15:39.230 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:15:38 smithi052 ceph-mon[31955]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' 2024-05-27T07:15:39.230 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:15:38 smithi052 ceph-mon[31955]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.z", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-05-27T07:15:39.231 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:15:38 smithi052 ceph-mon[31955]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.z", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-05-27T07:15:39.231 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:15:38 smithi052 ceph-mon[31955]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-05-27T07:15:39.231 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:15:38 smithi052 ceph-mon[31955]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T07:15:39.256 INFO:teuthology.orchestra.run.smithi052.stderr:1+0 records in 2024-05-27T07:15:39.256 INFO:teuthology.orchestra.run.smithi052.stderr:1+0 records out 2024-05-27T07:15:39.256 INFO:teuthology.orchestra.run.smithi052.stderr:512 bytes copied, 0.00012059 s, 4.2 MB/s 2024-05-27T07:15:39.257 DEBUG:teuthology.orchestra.run.smithi052:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-05-27T07:15:39.315 DEBUG:teuthology.orchestra.run.smithi052:> stat /dev/vg_nvme/lv_3 2024-05-27T07:15:39.372 INFO:teuthology.orchestra.run.smithi052.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-05-27T07:15:39.372 INFO:teuthology.orchestra.run.smithi052.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-05-27T07:15:39.372 INFO:teuthology.orchestra.run.smithi052.stdout:Device: 5h/5d Inode: 703 Links: 1 2024-05-27T07:15:39.372 INFO:teuthology.orchestra.run.smithi052.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-05-27T07:15:39.372 INFO:teuthology.orchestra.run.smithi052.stdout:Context: system_u:object_r:device_t:s0 2024-05-27T07:15:39.372 INFO:teuthology.orchestra.run.smithi052.stdout:Access: 2024-05-27 07:14:41.930566043 +0000 2024-05-27T07:15:39.372 INFO:teuthology.orchestra.run.smithi052.stdout:Modify: 2024-05-27 07:09:11.008288121 +0000 2024-05-27T07:15:39.373 INFO:teuthology.orchestra.run.smithi052.stdout:Change: 2024-05-27 07:09:11.008288121 +0000 2024-05-27T07:15:39.373 INFO:teuthology.orchestra.run.smithi052.stdout: Birth: 2024-05-27 07:09:11.008288121 +0000 2024-05-27T07:15:39.373 DEBUG:teuthology.orchestra.run.smithi052:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-05-27T07:15:39.437 INFO:teuthology.orchestra.run.smithi052.stderr:1+0 records in 2024-05-27T07:15:39.438 INFO:teuthology.orchestra.run.smithi052.stderr:1+0 records out 2024-05-27T07:15:39.438 INFO:teuthology.orchestra.run.smithi052.stderr:512 bytes copied, 0.000113046 s, 4.5 MB/s 2024-05-27T07:15:39.439 DEBUG:teuthology.orchestra.run.smithi052:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-05-27T07:15:39.496 DEBUG:teuthology.orchestra.run.smithi052:> stat /dev/vg_nvme/lv_4 2024-05-27T07:15:39.552 INFO:teuthology.orchestra.run.smithi052.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-05-27T07:15:39.553 INFO:teuthology.orchestra.run.smithi052.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-05-27T07:15:39.553 INFO:teuthology.orchestra.run.smithi052.stdout:Device: 5h/5d Inode: 724 Links: 1 2024-05-27T07:15:39.553 INFO:teuthology.orchestra.run.smithi052.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-05-27T07:15:39.553 INFO:teuthology.orchestra.run.smithi052.stdout:Context: system_u:object_r:device_t:s0 2024-05-27T07:15:39.553 INFO:teuthology.orchestra.run.smithi052.stdout:Access: 2024-05-27 07:14:41.975565527 +0000 2024-05-27T07:15:39.553 INFO:teuthology.orchestra.run.smithi052.stdout:Modify: 2024-05-27 07:09:11.341284330 +0000 2024-05-27T07:15:39.553 INFO:teuthology.orchestra.run.smithi052.stdout:Change: 2024-05-27 07:09:11.341284330 +0000 2024-05-27T07:15:39.553 INFO:teuthology.orchestra.run.smithi052.stdout: Birth: 2024-05-27 07:09:11.341284330 +0000 2024-05-27T07:15:39.553 DEBUG:teuthology.orchestra.run.smithi052:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-05-27T07:15:39.618 INFO:teuthology.orchestra.run.smithi052.stderr:1+0 records in 2024-05-27T07:15:39.618 INFO:teuthology.orchestra.run.smithi052.stderr:1+0 records out 2024-05-27T07:15:39.618 INFO:teuthology.orchestra.run.smithi052.stderr:512 bytes copied, 9.3597e-05 s, 5.5 MB/s 2024-05-27T07:15:39.620 DEBUG:teuthology.orchestra.run.smithi052:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-05-27T07:15:39.677 DEBUG:teuthology.orchestra.run.smithi105:> set -ex 2024-05-27T07:15:39.677 DEBUG:teuthology.orchestra.run.smithi105:> dd if=/scratch_devs of=/dev/stdout 2024-05-27T07:15:39.696 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-05-27T07:15:39.696 DEBUG:teuthology.orchestra.run.smithi105:> stat /dev/vg_nvme/lv_1 2024-05-27T07:15:39.753 INFO:teuthology.orchestra.run.smithi105.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-05-27T07:15:39.753 INFO:teuthology.orchestra.run.smithi105.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-05-27T07:15:39.753 INFO:teuthology.orchestra.run.smithi105.stdout:Device: 5h/5d Inode: 666 Links: 1 2024-05-27T07:15:39.753 INFO:teuthology.orchestra.run.smithi105.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-05-27T07:15:39.753 INFO:teuthology.orchestra.run.smithi105.stdout:Context: system_u:object_r:device_t:s0 2024-05-27T07:15:39.753 INFO:teuthology.orchestra.run.smithi105.stdout:Access: 2024-05-27 07:15:24.333679051 +0000 2024-05-27T07:15:39.754 INFO:teuthology.orchestra.run.smithi105.stdout:Modify: 2024-05-27 07:09:26.436341299 +0000 2024-05-27T07:15:39.754 INFO:teuthology.orchestra.run.smithi105.stdout:Change: 2024-05-27 07:09:26.436341299 +0000 2024-05-27T07:15:39.754 INFO:teuthology.orchestra.run.smithi105.stdout: Birth: 2024-05-27 07:09:26.436341299 +0000 2024-05-27T07:15:39.754 DEBUG:teuthology.orchestra.run.smithi105:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-05-27T07:15:39.819 INFO:teuthology.orchestra.run.smithi105.stderr:1+0 records in 2024-05-27T07:15:39.819 INFO:teuthology.orchestra.run.smithi105.stderr:1+0 records out 2024-05-27T07:15:39.819 INFO:teuthology.orchestra.run.smithi105.stderr:512 bytes copied, 0.000284685 s, 1.8 MB/s 2024-05-27T07:15:39.820 DEBUG:teuthology.orchestra.run.smithi105:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-05-27T07:15:39.878 DEBUG:teuthology.orchestra.run.smithi105:> stat /dev/vg_nvme/lv_2 2024-05-27T07:15:39.935 INFO:teuthology.orchestra.run.smithi105.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-05-27T07:15:39.935 INFO:teuthology.orchestra.run.smithi105.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-05-27T07:15:39.935 INFO:teuthology.orchestra.run.smithi105.stdout:Device: 5h/5d Inode: 683 Links: 1 2024-05-27T07:15:39.935 INFO:teuthology.orchestra.run.smithi105.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-05-27T07:15:39.935 INFO:teuthology.orchestra.run.smithi105.stdout:Context: system_u:object_r:device_t:s0 2024-05-27T07:15:39.935 INFO:teuthology.orchestra.run.smithi105.stdout:Access: 2024-05-27 07:15:24.333679051 +0000 2024-05-27T07:15:39.935 INFO:teuthology.orchestra.run.smithi105.stdout:Modify: 2024-05-27 07:09:26.766334621 +0000 2024-05-27T07:15:39.935 INFO:teuthology.orchestra.run.smithi105.stdout:Change: 2024-05-27 07:09:26.766334621 +0000 2024-05-27T07:15:39.935 INFO:teuthology.orchestra.run.smithi105.stdout: Birth: 2024-05-27 07:09:26.766334621 +0000 2024-05-27T07:15:39.936 DEBUG:teuthology.orchestra.run.smithi105:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-05-27T07:15:39.999 INFO:teuthology.orchestra.run.smithi105.stderr:1+0 records in 2024-05-27T07:15:39.999 INFO:teuthology.orchestra.run.smithi105.stderr:1+0 records out 2024-05-27T07:15:39.999 INFO:teuthology.orchestra.run.smithi105.stderr:512 bytes copied, 0.000268191 s, 1.9 MB/s 2024-05-27T07:15:40.001 DEBUG:teuthology.orchestra.run.smithi105:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-05-27T07:15:40.056 DEBUG:teuthology.orchestra.run.smithi105:> stat /dev/vg_nvme/lv_3 2024-05-27T07:15:40.112 INFO:teuthology.orchestra.run.smithi105.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-05-27T07:15:40.112 INFO:teuthology.orchestra.run.smithi105.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-05-27T07:15:40.112 INFO:teuthology.orchestra.run.smithi105.stdout:Device: 5h/5d Inode: 703 Links: 1 2024-05-27T07:15:40.112 INFO:teuthology.orchestra.run.smithi105.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-05-27T07:15:40.112 INFO:teuthology.orchestra.run.smithi105.stdout:Context: system_u:object_r:device_t:s0 2024-05-27T07:15:40.112 INFO:teuthology.orchestra.run.smithi105.stdout:Access: 2024-05-27 07:15:24.333679051 +0000 2024-05-27T07:15:40.112 INFO:teuthology.orchestra.run.smithi105.stdout:Modify: 2024-05-27 07:09:27.103327801 +0000 2024-05-27T07:15:40.112 INFO:teuthology.orchestra.run.smithi105.stdout:Change: 2024-05-27 07:09:27.103327801 +0000 2024-05-27T07:15:40.112 INFO:teuthology.orchestra.run.smithi105.stdout: Birth: 2024-05-27 07:09:27.103327801 +0000 2024-05-27T07:15:40.113 DEBUG:teuthology.orchestra.run.smithi105:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-05-27T07:15:40.171 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:15:39 smithi173 ceph-mon[37000]: from='client.24112 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "placement": "3;smithi052=x;smithi105=y;smithi173=z", "target": ["mon-mgr", ""]}]: dispatch 2024-05-27T07:15:40.171 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:15:39 smithi173 ceph-mon[37000]: Saving service mgr spec with placement smithi052=x;smithi105=y;smithi173=z;count:3 2024-05-27T07:15:40.171 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:15:39 smithi173 ceph-mon[37000]: Deploying daemon mgr.z on smithi173 2024-05-27T07:15:40.172 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:15:39 smithi105 ceph-mon[35977]: from='client.24112 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "placement": "3;smithi052=x;smithi105=y;smithi173=z", "target": ["mon-mgr", ""]}]: dispatch 2024-05-27T07:15:40.173 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:15:39 smithi105 ceph-mon[35977]: Saving service mgr spec with placement smithi052=x;smithi105=y;smithi173=z;count:3 2024-05-27T07:15:40.173 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:15:39 smithi105 ceph-mon[35977]: Deploying daemon mgr.z on smithi173 2024-05-27T07:15:40.175 INFO:teuthology.orchestra.run.smithi105.stderr:1+0 records in 2024-05-27T07:15:40.175 INFO:teuthology.orchestra.run.smithi105.stderr:1+0 records out 2024-05-27T07:15:40.175 INFO:teuthology.orchestra.run.smithi105.stderr:512 bytes copied, 0.000263136 s, 1.9 MB/s 2024-05-27T07:15:40.176 DEBUG:teuthology.orchestra.run.smithi105:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-05-27T07:15:40.230 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:15:39 smithi052 ceph-mon[31955]: from='client.24112 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "placement": "3;smithi052=x;smithi105=y;smithi173=z", "target": ["mon-mgr", ""]}]: dispatch 2024-05-27T07:15:40.230 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:15:39 smithi052 ceph-mon[31955]: Saving service mgr spec with placement smithi052=x;smithi105=y;smithi173=z;count:3 2024-05-27T07:15:40.230 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:15:39 smithi052 ceph-mon[31955]: Deploying daemon mgr.z on smithi173 2024-05-27T07:15:40.232 DEBUG:teuthology.orchestra.run.smithi105:> stat /dev/vg_nvme/lv_4 2024-05-27T07:15:40.287 INFO:teuthology.orchestra.run.smithi105.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-05-27T07:15:40.287 INFO:teuthology.orchestra.run.smithi105.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-05-27T07:15:40.288 INFO:teuthology.orchestra.run.smithi105.stdout:Device: 5h/5d Inode: 721 Links: 1 2024-05-27T07:15:40.288 INFO:teuthology.orchestra.run.smithi105.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-05-27T07:15:40.288 INFO:teuthology.orchestra.run.smithi105.stdout:Context: system_u:object_r:device_t:s0 2024-05-27T07:15:40.288 INFO:teuthology.orchestra.run.smithi105.stdout:Access: 2024-05-27 07:15:24.334679031 +0000 2024-05-27T07:15:40.288 INFO:teuthology.orchestra.run.smithi105.stdout:Modify: 2024-05-27 07:09:27.442320942 +0000 2024-05-27T07:15:40.288 INFO:teuthology.orchestra.run.smithi105.stdout:Change: 2024-05-27 07:09:27.442320942 +0000 2024-05-27T07:15:40.288 INFO:teuthology.orchestra.run.smithi105.stdout: Birth: 2024-05-27 07:09:27.442320942 +0000 2024-05-27T07:15:40.288 DEBUG:teuthology.orchestra.run.smithi105:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-05-27T07:15:40.349 INFO:teuthology.orchestra.run.smithi105.stderr:1+0 records in 2024-05-27T07:15:40.350 INFO:teuthology.orchestra.run.smithi105.stderr:1+0 records out 2024-05-27T07:15:40.350 INFO:teuthology.orchestra.run.smithi105.stderr:512 bytes copied, 0.000200193 s, 2.6 MB/s 2024-05-27T07:15:40.351 DEBUG:teuthology.orchestra.run.smithi105:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-05-27T07:15:40.407 DEBUG:teuthology.orchestra.run.smithi173:> set -ex 2024-05-27T07:15:40.407 DEBUG:teuthology.orchestra.run.smithi173:> dd if=/scratch_devs of=/dev/stdout 2024-05-27T07:15:40.423 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-05-27T07:15:40.423 DEBUG:teuthology.orchestra.run.smithi173:> stat /dev/vg_nvme/lv_1 2024-05-27T07:15:40.475 INFO:journalctl@ceph.mgr.z.smithi173.stdout:May 27 07:15:40 smithi173 systemd[1]: Starting Ceph mgr.z for 73df2636-1bf8-11ef-bc9b-c7b262605968... 2024-05-27T07:15:40.480 INFO:teuthology.orchestra.run.smithi173.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-05-27T07:15:40.480 INFO:teuthology.orchestra.run.smithi173.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-05-27T07:15:40.480 INFO:teuthology.orchestra.run.smithi173.stdout:Device: 5h/5d Inode: 666 Links: 1 2024-05-27T07:15:40.480 INFO:teuthology.orchestra.run.smithi173.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-05-27T07:15:40.480 INFO:teuthology.orchestra.run.smithi173.stdout:Context: system_u:object_r:device_t:s0 2024-05-27T07:15:40.480 INFO:teuthology.orchestra.run.smithi173.stdout:Access: 2024-05-27 07:15:40.345181782 +0000 2024-05-27T07:15:40.480 INFO:teuthology.orchestra.run.smithi173.stdout:Modify: 2024-05-27 07:09:42.096823198 +0000 2024-05-27T07:15:40.480 INFO:teuthology.orchestra.run.smithi173.stdout:Change: 2024-05-27 07:09:42.096823198 +0000 2024-05-27T07:15:40.480 INFO:teuthology.orchestra.run.smithi173.stdout: Birth: 2024-05-27 07:09:42.096823198 +0000 2024-05-27T07:15:40.481 DEBUG:teuthology.orchestra.run.smithi173:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-05-27T07:15:40.545 INFO:teuthology.orchestra.run.smithi173.stderr:1+0 records in 2024-05-27T07:15:40.545 INFO:teuthology.orchestra.run.smithi173.stderr:1+0 records out 2024-05-27T07:15:40.545 INFO:teuthology.orchestra.run.smithi173.stderr:512 bytes copied, 0.000193935 s, 2.6 MB/s 2024-05-27T07:15:40.547 DEBUG:teuthology.orchestra.run.smithi173:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-05-27T07:15:40.602 DEBUG:teuthology.orchestra.run.smithi173:> stat /dev/vg_nvme/lv_2 2024-05-27T07:15:40.657 INFO:teuthology.orchestra.run.smithi173.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-05-27T07:15:40.658 INFO:teuthology.orchestra.run.smithi173.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-05-27T07:15:40.658 INFO:teuthology.orchestra.run.smithi173.stdout:Device: 5h/5d Inode: 682 Links: 1 2024-05-27T07:15:40.658 INFO:teuthology.orchestra.run.smithi173.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-05-27T07:15:40.658 INFO:teuthology.orchestra.run.smithi173.stdout:Context: system_u:object_r:device_t:s0 2024-05-27T07:15:40.658 INFO:teuthology.orchestra.run.smithi173.stdout:Access: 2024-05-27 07:15:40.345181782 +0000 2024-05-27T07:15:40.658 INFO:teuthology.orchestra.run.smithi173.stdout:Modify: 2024-05-27 07:09:42.433815744 +0000 2024-05-27T07:15:40.658 INFO:teuthology.orchestra.run.smithi173.stdout:Change: 2024-05-27 07:09:42.433815744 +0000 2024-05-27T07:15:40.658 INFO:teuthology.orchestra.run.smithi173.stdout: Birth: 2024-05-27 07:09:42.433815744 +0000 2024-05-27T07:15:40.658 DEBUG:teuthology.orchestra.run.smithi173:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-05-27T07:15:40.721 INFO:teuthology.orchestra.run.smithi173.stderr:1+0 records in 2024-05-27T07:15:40.721 INFO:teuthology.orchestra.run.smithi173.stderr:1+0 records out 2024-05-27T07:15:40.721 INFO:teuthology.orchestra.run.smithi173.stderr:512 bytes copied, 0.000245858 s, 2.1 MB/s 2024-05-27T07:15:40.723 DEBUG:teuthology.orchestra.run.smithi173:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-05-27T07:15:40.744 INFO:journalctl@ceph.mgr.z.smithi173.stdout:May 27 07:15:40 smithi173 podman[38797]: 2024-05-27 07:15:40.533999486 +0000 UTC m=+0.018297902 image pull e36ea4fc8a5807e8d2701370e350a0de3856538797f7fba64cfce7e344bd6799 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6 2024-05-27T07:15:40.744 INFO:journalctl@ceph.mgr.z.smithi173.stdout:May 27 07:15:40 smithi173 podman[38797]: 2024-05-27 07:15:40.669545017 +0000 UTC m=+0.153843424 container create db6f11578c162527f37009af50bca17a42cbf6043df42591b22f1f62ea94ee02 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mgr-z, org.label-schema.schema-version=1.0, ceph=True, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, org.label-schema.build-date=20240521, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, org.label-schema.license=GPLv2) 2024-05-27T07:15:40.758 DEBUG:teuthology.orchestra.run.smithi173:> stat /dev/vg_nvme/lv_3 2024-05-27T07:15:40.813 INFO:teuthology.orchestra.run.smithi173.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-05-27T07:15:40.813 INFO:teuthology.orchestra.run.smithi173.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-05-27T07:15:40.813 INFO:teuthology.orchestra.run.smithi173.stdout:Device: 5h/5d Inode: 701 Links: 1 2024-05-27T07:15:40.813 INFO:teuthology.orchestra.run.smithi173.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-05-27T07:15:40.813 INFO:teuthology.orchestra.run.smithi173.stdout:Context: system_u:object_r:device_t:s0 2024-05-27T07:15:40.813 INFO:teuthology.orchestra.run.smithi173.stdout:Access: 2024-05-27 07:15:40.346181760 +0000 2024-05-27T07:15:40.813 INFO:teuthology.orchestra.run.smithi173.stdout:Modify: 2024-05-27 07:09:42.764808422 +0000 2024-05-27T07:15:40.813 INFO:teuthology.orchestra.run.smithi173.stdout:Change: 2024-05-27 07:09:42.764808422 +0000 2024-05-27T07:15:40.813 INFO:teuthology.orchestra.run.smithi173.stdout: Birth: 2024-05-27 07:09:42.764808422 +0000 2024-05-27T07:15:40.814 DEBUG:teuthology.orchestra.run.smithi173:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-05-27T07:15:40.878 INFO:teuthology.orchestra.run.smithi173.stderr:1+0 records in 2024-05-27T07:15:40.878 INFO:teuthology.orchestra.run.smithi173.stderr:1+0 records out 2024-05-27T07:15:40.879 INFO:teuthology.orchestra.run.smithi173.stderr:512 bytes copied, 0.00020162 s, 2.5 MB/s 2024-05-27T07:15:40.879 DEBUG:teuthology.orchestra.run.smithi173:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-05-27T07:15:40.935 DEBUG:teuthology.orchestra.run.smithi173:> stat /dev/vg_nvme/lv_4 2024-05-27T07:15:40.992 INFO:teuthology.orchestra.run.smithi173.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-05-27T07:15:40.992 INFO:teuthology.orchestra.run.smithi173.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-05-27T07:15:40.992 INFO:teuthology.orchestra.run.smithi173.stdout:Device: 5h/5d Inode: 720 Links: 1 2024-05-27T07:15:40.992 INFO:teuthology.orchestra.run.smithi173.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-05-27T07:15:40.993 INFO:teuthology.orchestra.run.smithi173.stdout:Context: system_u:object_r:device_t:s0 2024-05-27T07:15:40.993 INFO:teuthology.orchestra.run.smithi173.stdout:Access: 2024-05-27 07:15:40.346181760 +0000 2024-05-27T07:15:40.993 INFO:teuthology.orchestra.run.smithi173.stdout:Modify: 2024-05-27 07:09:43.098801034 +0000 2024-05-27T07:15:40.993 INFO:teuthology.orchestra.run.smithi173.stdout:Change: 2024-05-27 07:09:43.098801034 +0000 2024-05-27T07:15:40.993 INFO:teuthology.orchestra.run.smithi173.stdout: Birth: 2024-05-27 07:09:43.098801034 +0000 2024-05-27T07:15:40.993 DEBUG:teuthology.orchestra.run.smithi173:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-05-27T07:15:41.049 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:15:40 smithi173 ceph-mon[37000]: pgmap v29: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-27T07:15:41.049 INFO:journalctl@ceph.mgr.z.smithi173.stdout:May 27 07:15:40 smithi173 podman[38797]: 2024-05-27 07:15:40.881086963 +0000 UTC m=+0.365385372 container init db6f11578c162527f37009af50bca17a42cbf6043df42591b22f1f62ea94ee02 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mgr-z, org.label-schema.build-date=20240521, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, GIT_CLEAN=True, ceph=True, io.buildah.version=1.33.5) 2024-05-27T07:15:41.049 INFO:journalctl@ceph.mgr.z.smithi173.stdout:May 27 07:15:40 smithi173 podman[38797]: 2024-05-27 07:15:40.883985185 +0000 UTC m=+0.368283595 container start db6f11578c162527f37009af50bca17a42cbf6043df42591b22f1f62ea94ee02 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mgr-z, io.buildah.version=1.33.5, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, org.label-schema.build-date=20240521, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, GIT_BRANCH=HEAD, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=) 2024-05-27T07:15:41.049 INFO:journalctl@ceph.mgr.z.smithi173.stdout:May 27 07:15:40 smithi173 ceph-mgr[38946]: -- 172.21.15.173:0/2121138191 <== mon.1 v2:172.21.15.173:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55931fafc5a0 con 0x559324886480 2024-05-27T07:15:41.049 INFO:journalctl@ceph.mgr.z.smithi173.stdout:May 27 07:15:40 smithi173 bash[38797]: db6f11578c162527f37009af50bca17a42cbf6043df42591b22f1f62ea94ee02 2024-05-27T07:15:41.049 INFO:journalctl@ceph.mgr.z.smithi173.stdout:May 27 07:15:40 smithi173 systemd[1]: Started Ceph mgr.z for 73df2636-1bf8-11ef-bc9b-c7b262605968. 2024-05-27T07:15:41.058 INFO:teuthology.orchestra.run.smithi173.stderr:1+0 records in 2024-05-27T07:15:41.059 INFO:teuthology.orchestra.run.smithi173.stderr:1+0 records out 2024-05-27T07:15:41.059 INFO:teuthology.orchestra.run.smithi173.stderr:512 bytes copied, 0.000223085 s, 2.3 MB/s 2024-05-27T07:15:41.061 DEBUG:teuthology.orchestra.run.smithi173:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-05-27T07:15:41.117 INFO:tasks.cephadm:Deploying osd.0 on smithi052 with /dev/vg_nvme/lv_4... 2024-05-27T07:15:41.117 DEBUG:teuthology.orchestra.run.smithi052:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:38f9d900c83c63799fbdbe61acc9a11b0d3554a6 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 73df2636-1bf8-11ef-bc9b-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_4 2024-05-27T07:15:41.142 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:15:40 smithi052 ceph-mon[31955]: pgmap v29: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-27T07:15:41.190 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:15:40 smithi105 ceph-mon[35977]: pgmap v29: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-27T07:15:41.343 INFO:journalctl@ceph.mgr.z.smithi173.stdout:May 27 07:15:41 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mgr-z[38891]: 2024-05-27T07:15:41.058+0000 7f407e904200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-05-27T07:15:41.343 INFO:journalctl@ceph.mgr.z.smithi173.stdout:May 27 07:15:41 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mgr-z[38891]: 2024-05-27T07:15:41.249+0000 7f407e904200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-05-27T07:15:41.344 INFO:journalctl@ceph.mgr.z.smithi173.stdout:May 27 07:15:41 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mgr-z[38891]: 2024-05-27T07:15:41.352+0000 7f407e904200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-05-27T07:15:41.365 INFO:teuthology.orchestra.run.smithi052.stderr:Inferring config /var/lib/ceph/73df2636-1bf8-11ef-bc9b-c7b262605968/mon.a/config 2024-05-27T07:15:42.031 INFO:journalctl@ceph.mgr.z.smithi173.stdout:May 27 07:15:41 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mgr-z[38891]: 2024-05-27T07:15:41.662+0000 7f407e904200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-05-27T07:15:42.031 INFO:journalctl@ceph.mgr.z.smithi173.stdout:May 27 07:15:41 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mgr-z[38891]: 2024-05-27T07:15:41.746+0000 7f407e904200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-05-27T07:15:42.102 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:15:42 smithi105 ceph-mon[35977]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' 2024-05-27T07:15:42.102 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:15:42 smithi105 ceph-mon[35977]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' 2024-05-27T07:15:42.102 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:15:42 smithi105 ceph-mon[35977]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' 2024-05-27T07:15:42.102 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:15:42 smithi105 ceph-mon[35977]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-05-27T07:15:42.102 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:15:42 smithi105 ceph-mon[35977]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-05-27T07:15:42.102 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:15:42 smithi105 ceph-mon[35977]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-05-27T07:15:42.102 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:15:42 smithi105 ceph-mon[35977]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T07:15:42.352 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:15:42 smithi173 ceph-mon[37000]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' 2024-05-27T07:15:42.353 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:15:42 smithi173 ceph-mon[37000]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' 2024-05-27T07:15:42.353 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:15:42 smithi173 ceph-mon[37000]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' 2024-05-27T07:15:42.353 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:15:42 smithi173 ceph-mon[37000]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-05-27T07:15:42.353 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:15:42 smithi173 ceph-mon[37000]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-05-27T07:15:42.353 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:15:42 smithi173 ceph-mon[37000]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-05-27T07:15:42.353 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:15:42 smithi173 ceph-mon[37000]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T07:15:42.436 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:15:42 smithi052 ceph-mon[31955]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' 2024-05-27T07:15:42.436 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:15:42 smithi052 ceph-mon[31955]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' 2024-05-27T07:15:42.436 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:15:42 smithi052 ceph-mon[31955]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' 2024-05-27T07:15:42.436 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:15:42 smithi052 ceph-mon[31955]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-05-27T07:15:42.436 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:15:42 smithi052 ceph-mon[31955]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-05-27T07:15:42.436 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:15:42 smithi052 ceph-mon[31955]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-05-27T07:15:42.436 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:15:42 smithi052 ceph-mon[31955]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T07:15:43.115 INFO:teuthology.orchestra.run.smithi052.stderr:--> Zapping: /dev/vg_nvme/lv_4 2024-05-27T07:15:43.116 INFO:teuthology.orchestra.run.smithi052.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_4 bs=1M count=10 conv=fsync 2024-05-27T07:15:43.116 INFO:teuthology.orchestra.run.smithi052.stderr: stderr: 10+0 records in 2024-05-27T07:15:43.116 INFO:teuthology.orchestra.run.smithi052.stderr:10+0 records out 2024-05-27T07:15:43.116 INFO:teuthology.orchestra.run.smithi052.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0081117 s, 1.3 GB/s 2024-05-27T07:15:43.116 INFO:teuthology.orchestra.run.smithi052.stderr:--> Zapping successful for: 2024-05-27T07:15:43.247 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:15:43 smithi173 ceph-mon[37000]: Deploying daemon mgr.y on smithi105 2024-05-27T07:15:43.248 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:15:43 smithi173 ceph-mon[37000]: pgmap v30: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-27T07:15:43.248 INFO:journalctl@ceph.mgr.z.smithi173.stdout:May 27 07:15:42 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mgr-z[38891]: 2024-05-27T07:15:42.924+0000 7f407e904200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-05-27T07:15:43.248 INFO:journalctl@ceph.mgr.z.smithi173.stdout:May 27 07:15:43 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mgr-z[38891]: 2024-05-27T07:15:43.088+0000 7f407e904200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-05-27T07:15:43.248 INFO:journalctl@ceph.mgr.z.smithi173.stdout:May 27 07:15:43 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mgr-z[38891]: 2024-05-27T07:15:43.256+0000 7f407e904200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-05-27T07:15:43.344 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:15:43 smithi105 ceph-mon[35977]: Deploying daemon mgr.y on smithi105 2024-05-27T07:15:43.344 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:15:43 smithi105 ceph-mon[35977]: pgmap v30: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-27T07:15:43.474 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:15:43 smithi052 ceph-mon[31955]: Deploying daemon mgr.y on smithi105 2024-05-27T07:15:43.474 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:15:43 smithi052 ceph-mon[31955]: pgmap v30: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-27T07:15:43.681 DEBUG:teuthology.orchestra.run.smithi052:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:38f9d900c83c63799fbdbe61acc9a11b0d3554a6 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 73df2636-1bf8-11ef-bc9b-c7b262605968 -- ceph orch daemon add osd smithi052:vg_nvme/lv_4 2024-05-27T07:15:43.801 INFO:journalctl@ceph.mgr.y.smithi105.stdout:May 27 07:15:43 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mgr-y[37393]: 2024-05-27T07:15:43.621+0000 7f9c21ace200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-05-27T07:15:43.801 INFO:journalctl@ceph.mgr.y.smithi105.stdout:May 27 07:15:43 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mgr-y[37393]: 2024-05-27T07:15:43.810+0000 7f9c21ace200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-05-27T07:15:43.825 INFO:journalctl@ceph.mgr.z.smithi173.stdout:May 27 07:15:43 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mgr-z[38891]: 2024-05-27T07:15:43.537+0000 7f407e904200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-05-27T07:15:43.825 INFO:journalctl@ceph.mgr.z.smithi173.stdout:May 27 07:15:43 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mgr-z[38891]: 2024-05-27T07:15:43.627+0000 7f407e904200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-05-27T07:15:43.825 INFO:journalctl@ceph.mgr.z.smithi173.stdout:May 27 07:15:43 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mgr-z[38891]: 2024-05-27T07:15:43.834+0000 7f407e904200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-05-27T07:15:43.968 INFO:teuthology.orchestra.run.smithi052.stderr:Inferring config /var/lib/ceph/73df2636-1bf8-11ef-bc9b-c7b262605968/mon.a/config 2024-05-27T07:15:44.102 INFO:journalctl@ceph.mgr.z.smithi173.stdout:May 27 07:15:44 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mgr-z[38891]: 2024-05-27T07:15:44.031+0000 7f407e904200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-05-27T07:15:44.190 INFO:journalctl@ceph.mgr.y.smithi105.stdout:May 27 07:15:43 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mgr-y[37393]: 2024-05-27T07:15:43.919+0000 7f9c21ace200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-05-27T07:15:44.568 INFO:journalctl@ceph.mgr.y.smithi105.stdout:May 27 07:15:44 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mgr-y[37393]: 2024-05-27T07:15:44.248+0000 7f9c21ace200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-05-27T07:15:44.568 INFO:journalctl@ceph.mgr.y.smithi105.stdout:May 27 07:15:44 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mgr-y[37393]: 2024-05-27T07:15:44.345+0000 7f9c21ace200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-05-27T07:15:44.655 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:15:44 smithi052 ceph-mon[31955]: pgmap v31: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-27T07:15:44.656 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:15:44 smithi052 ceph-mon[31955]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' 2024-05-27T07:15:44.656 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:15:44 smithi052 ceph-mon[31955]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' 2024-05-27T07:15:44.656 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:15:44 smithi052 ceph-mon[31955]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' 2024-05-27T07:15:44.656 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:15:44 smithi052 ceph-mon[31955]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' 2024-05-27T07:15:44.656 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:15:44 smithi052 ceph-mon[31955]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-27T07:15:44.853 INFO:journalctl@ceph.mgr.z.smithi173.stdout:May 27 07:15:44 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mgr-z[38891]: 2024-05-27T07:15:44.676+0000 7f407e904200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-05-27T07:15:44.853 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:15:44 smithi173 ceph-mon[37000]: pgmap v31: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-27T07:15:44.853 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:15:44 smithi173 ceph-mon[37000]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' 2024-05-27T07:15:44.853 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:15:44 smithi173 ceph-mon[37000]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' 2024-05-27T07:15:44.853 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:15:44 smithi173 ceph-mon[37000]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' 2024-05-27T07:15:44.853 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:15:44 smithi173 ceph-mon[37000]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' 2024-05-27T07:15:44.853 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:15:44 smithi173 ceph-mon[37000]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-27T07:15:44.940 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:15:44 smithi105 ceph-mon[35977]: pgmap v31: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-27T07:15:44.941 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:15:44 smithi105 ceph-mon[35977]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' 2024-05-27T07:15:44.941 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:15:44 smithi105 ceph-mon[35977]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' 2024-05-27T07:15:44.941 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:15:44 smithi105 ceph-mon[35977]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' 2024-05-27T07:15:44.941 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:15:44 smithi105 ceph-mon[35977]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' 2024-05-27T07:15:44.941 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:15:44 smithi105 ceph-mon[35977]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-27T07:15:45.319 INFO:journalctl@ceph.mgr.z.smithi173.stdout:May 27 07:15:44 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mgr-z[38891]: 2024-05-27T07:15:44.900+0000 7f407e904200 -1 mgr[py] Module smb has missing NOTIFY_TYPES member 2024-05-27T07:15:45.424 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:15:45.432+0000 7f15daed4700 1 -- 172.21.15.52:0/3644400165 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f15d4107780 msgr2=0x7f15d4109ba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:15:45.424 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:15:45.432+0000 7f15daed4700 1 --2- 172.21.15.52:0/3644400165 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f15d4107780 0x7f15d4109ba0 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7f15c0009990 tx=0x7f15c0009ca0 comp rx=0 tx=0).stop 2024-05-27T07:15:45.424 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:15:45.432+0000 7f15daed4700 1 -- 172.21.15.52:0/3644400165 shutdown_connections 2024-05-27T07:15:45.424 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:15:45.432+0000 7f15daed4700 1 --2- 172.21.15.52:0/3644400165 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f15d410a0e0 0x7f15d410c500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:15:45.424 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:15:45.433+0000 7f15daed4700 1 --2- 172.21.15.52:0/3644400165 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f15d4104e20 0x7f15d4107240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:15:45.425 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:15:45.433+0000 7f15daed4700 1 --2- 172.21.15.52:0/3644400165 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f15d4107780 0x7f15d4109ba0 unknown :-1 s=CLOSED pgs=109 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:15:45.425 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:15:45.433+0000 7f15daed4700 1 -- 172.21.15.52:0/3644400165 >> 172.21.15.52:0/3644400165 conn(0x7f15d40fe860 msgr2=0x7f15d4100d10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:15:45.425 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:15:45.433+0000 7f15daed4700 1 -- 172.21.15.52:0/3644400165 shutdown_connections 2024-05-27T07:15:45.425 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:15:45.433+0000 7f15daed4700 1 -- 172.21.15.52:0/3644400165 wait complete. 2024-05-27T07:15:45.427 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:15:45.433+0000 7f15daed4700 1 Processor -- start 2024-05-27T07:15:45.427 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:15:45.433+0000 7f15daed4700 1 -- start start 2024-05-27T07:15:45.427 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:15:45.433+0000 7f15daed4700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f15d4104e20 0x7f15d4104680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:15:45.427 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:15:45.434+0000 7f15daed4700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f15d4107780 0x7f15d4102d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:15:45.427 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:15:45.434+0000 7f15daed4700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f15d410a0e0 0x7f15d4103270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:15:45.427 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:15:45.434+0000 7f15daed4700 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x7f15d410fd80 con 0x7f15d4104e20 2024-05-27T07:15:45.427 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:15:45.434+0000 7f15daed4700 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f15d410fc00 con 0x7f15d4107780 2024-05-27T07:15:45.427 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:15:45.434+0000 7f15daed4700 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f15d410ff00 con 0x7f15d410a0e0 2024-05-27T07:15:45.427 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:15:45.434+0000 7f15d8c70700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f15d4104e20 0x7f15d4104680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:15:45.427 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:15:45.434+0000 7f15d3fff700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f15d4107780 0x7f15d4102d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:15:45.427 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:15:45.434+0000 7f15d8c70700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f15d4104e20 0x7f15d4104680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.52:34252/0 (socket says 172.21.15.52:34252) 2024-05-27T07:15:45.427 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:15:45.434+0000 7f15d8c70700 1 -- 172.21.15.52:0/1177631558 learned_addr learned my addr 172.21.15.52:0/1177631558 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-05-27T07:15:45.428 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:15:45.434+0000 7f15d3fff700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f15d4107780 0x7f15d4102d30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.52:56728/0 (socket says 172.21.15.52:56728) 2024-05-27T07:15:45.428 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:15:45.434+0000 7f15d9471700 1 --2- 172.21.15.52:0/1177631558 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f15d410a0e0 0x7f15d4103270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:15:45.428 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:15:45.434+0000 7f15d8c70700 1 -- 172.21.15.52:0/1177631558 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f15d410a0e0 msgr2=0x7f15d4103270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:15:45.428 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:15:45.434+0000 7f15d8c70700 1 --2- 172.21.15.52:0/1177631558 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f15d410a0e0 0x7f15d4103270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:15:45.428 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:15:45.434+0000 7f15d8c70700 1 -- 172.21.15.52:0/1177631558 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f15d4107780 msgr2=0x7f15d4102d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:15:45.428 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:15:45.434+0000 7f15d8c70700 1 --2- 172.21.15.52:0/1177631558 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f15d4107780 0x7f15d4102d30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:15:45.428 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:15:45.434+0000 7f15d8c70700 1 -- 172.21.15.52:0/1177631558 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f15d41039d0 con 0x7f15d4104e20 2024-05-27T07:15:45.428 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:15:45.434+0000 7f15d9471700 1 --2- 172.21.15.52:0/1177631558 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f15d410a0e0 0x7f15d4103270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-27T07:15:45.428 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:15:45.434+0000 7f15d8c70700 1 --2- 172.21.15.52:0/1177631558 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f15d4104e20 0x7f15d4104680 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7f15c800e970 tx=0x7f15c800ec80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:15:45.428 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:15:45.434+0000 7f15d1ffb700 1 -- 172.21.15.52:0/1177631558 <== mon.0 v2:172.21.15.52:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f15c8004a30 con 0x7f15d4104e20 2024-05-27T07:15:45.428 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:15:45.435+0000 7f15d1ffb700 1 -- 172.21.15.52:0/1177631558 <== mon.0 v2:172.21.15.52:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f15c800cc90 con 0x7f15d4104e20 2024-05-27T07:15:45.428 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:15:45.435+0000 7f15daed4700 1 -- 172.21.15.52:0/1177631558 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f15d41a49a0 con 0x7f15d4104e20 2024-05-27T07:15:45.429 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:15:45.435+0000 7f15d1ffb700 1 -- 172.21.15.52:0/1177631558 <== mon.0 v2:172.21.15.52:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f15c8005610 con 0x7f15d4104e20 2024-05-27T07:15:45.429 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:15:45.435+0000 7f15daed4700 1 -- 172.21.15.52:0/1177631558 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f15d41a4e80 con 0x7f15d4104e20 2024-05-27T07:15:45.429 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:15:45.436+0000 7f15d1ffb700 1 -- 172.21.15.52:0/1177631558 <== mon.0 v2:172.21.15.52:3300/0 4 ==== mgrmap(e 13) ==== 50278+0+0 (secure 0 0 0) 0x7f15c8010430 con 0x7f15d4104e20 2024-05-27T07:15:45.429 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:15:45.436+0000 7f15d1ffb700 1 --2- 172.21.15.52:0/1177631558 >> [v2:172.21.15.52:6800/3513075805,v1:172.21.15.52:6801/3513075805] conn(0x7f15c403e750 0x7f15c4040c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:15:45.430 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:15:45.436+0000 7f15d1ffb700 1 -- 172.21.15.52:0/1177631558 <== mon.0 v2:172.21.15.52:3300/0 5 ==== osd_map(4..4 src has 1..4) ==== 1155+0+0 (secure 0 0 0) 0x7f15c8052c50 con 0x7f15d4104e20 2024-05-27T07:15:45.430 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:15:45.436+0000 7f15d3fff700 1 --2- 172.21.15.52:0/1177631558 >> [v2:172.21.15.52:6800/3513075805,v1:172.21.15.52:6801/3513075805] conn(0x7f15c403e750 0x7f15c4040c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:15:45.430 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:15:45.436+0000 7f15daed4700 1 -- 172.21.15.52:0/1177631558 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f15d40512b0 con 0x7f15d4104e20 2024-05-27T07:15:45.430 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:15:45.437+0000 7f15d3fff700 1 --2- 172.21.15.52:0/1177631558 >> [v2:172.21.15.52:6800/3513075805,v1:172.21.15.52:6801/3513075805] conn(0x7f15c403e750 0x7f15c4040c60 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f15d4104030 tx=0x7f15c000b720 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:15:45.433 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:15:45.442+0000 7f15d1ffb700 1 -- 172.21.15.52:0/1177631558 <== mon.0 v2:172.21.15.52:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191963 (secure 0 0 0) 0x7f15c800a250 con 0x7f15d4104e20 2024-05-27T07:15:45.598 INFO:journalctl@ceph.mgr.z.smithi173.stdout:May 27 07:15:45 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mgr-z[38891]: 2024-05-27T07:15:45.329+0000 7f407e904200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-05-27T07:15:45.598 INFO:journalctl@ceph.mgr.z.smithi173.stdout:May 27 07:15:45 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mgr-z[38891]: 2024-05-27T07:15:45.423+0000 7f407e904200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-05-27T07:15:45.661 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:15:45.670+0000 7f15daed4700 1 -- 172.21.15.52:0/1177631558 --> [v2:172.21.15.52:6800/3513075805,v1:172.21.15.52:6801/3513075805] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi052:vg_nvme/lv_4", "target": ["mon-mgr", ""]}) -- 0x7f15d4063bf0 con 0x7f15c403e750 2024-05-27T07:15:45.662 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:15:45 smithi052 ceph-mon[31955]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' 2024-05-27T07:15:45.662 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:15:45 smithi052 ceph-mon[31955]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' 2024-05-27T07:15:45.662 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:15:45 smithi052 ceph-mon[31955]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' 2024-05-27T07:15:45.662 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:15:45 smithi052 ceph-mon[31955]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' 2024-05-27T07:15:45.662 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:15:45 smithi052 ceph-mon[31955]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' 2024-05-27T07:15:45.662 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:15:45 smithi052 ceph-mon[31955]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T07:15:45.662 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:15:45 smithi052 ceph-mon[31955]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-27T07:15:45.662 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:15:45 smithi052 ceph-mon[31955]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' 2024-05-27T07:15:45.662 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:15:45 smithi052 ceph-mon[31955]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-05-27T07:15:45.662 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:15:45 smithi052 ceph-mon[31955]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-05-27T07:15:45.662 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:15:45 smithi052 ceph-mon[31955]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T07:15:45.852 INFO:journalctl@ceph.mgr.z.smithi173.stdout:May 27 07:15:45 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mgr-z[38891]: 2024-05-27T07:15:45.792+0000 7f407e904200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-05-27T07:15:45.853 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:15:45 smithi173 ceph-mon[37000]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' 2024-05-27T07:15:45.853 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:15:45 smithi173 ceph-mon[37000]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' 2024-05-27T07:15:45.853 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:15:45 smithi173 ceph-mon[37000]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' 2024-05-27T07:15:45.853 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:15:45 smithi173 ceph-mon[37000]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' 2024-05-27T07:15:45.853 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:15:45 smithi173 ceph-mon[37000]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' 2024-05-27T07:15:45.853 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:15:45 smithi173 ceph-mon[37000]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T07:15:45.853 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:15:45 smithi173 ceph-mon[37000]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-27T07:15:45.853 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:15:45 smithi173 ceph-mon[37000]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' 2024-05-27T07:15:45.853 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:15:45 smithi173 ceph-mon[37000]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-05-27T07:15:45.853 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:15:45 smithi173 ceph-mon[37000]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-05-27T07:15:45.853 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:15:45 smithi173 ceph-mon[37000]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T07:15:45.912 INFO:journalctl@ceph.mgr.y.smithi105.stdout:May 27 07:15:45 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mgr-y[37393]: 2024-05-27T07:15:45.582+0000 7f9c21ace200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-05-27T07:15:45.912 INFO:journalctl@ceph.mgr.y.smithi105.stdout:May 27 07:15:45 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mgr-y[37393]: 2024-05-27T07:15:45.751+0000 7f9c21ace200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-05-27T07:15:45.912 INFO:journalctl@ceph.mgr.y.smithi105.stdout:May 27 07:15:45 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mgr-y[37393]: 2024-05-27T07:15:45.921+0000 7f9c21ace200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-05-27T07:15:45.912 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:15:45 smithi105 ceph-mon[35977]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' 2024-05-27T07:15:45.912 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:15:45 smithi105 ceph-mon[35977]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' 2024-05-27T07:15:45.913 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:15:45 smithi105 ceph-mon[35977]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' 2024-05-27T07:15:45.913 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:15:45 smithi105 ceph-mon[35977]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' 2024-05-27T07:15:45.913 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:15:45 smithi105 ceph-mon[35977]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' 2024-05-27T07:15:45.913 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:15:45 smithi105 ceph-mon[35977]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T07:15:45.913 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:15:45 smithi105 ceph-mon[35977]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-27T07:15:45.913 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:15:45 smithi105 ceph-mon[35977]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' 2024-05-27T07:15:45.913 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:15:45 smithi105 ceph-mon[35977]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-05-27T07:15:45.913 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:15:45 smithi105 ceph-mon[35977]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-05-27T07:15:45.913 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:15:45 smithi105 ceph-mon[35977]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T07:15:46.459 INFO:journalctl@ceph.mgr.z.smithi173.stdout:May 27 07:15:46 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mgr-z[38891]: 2024-05-27T07:15:46.141+0000 7f407e904200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-05-27T07:15:46.460 INFO:journalctl@ceph.mgr.z.smithi173.stdout:May 27 07:15:46 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mgr-z[38891]: 2024-05-27T07:15:46.233+0000 7f407e904200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-05-27T07:15:46.492 INFO:journalctl@ceph.mgr.y.smithi105.stdout:May 27 07:15:46 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mgr-y[37393]: 2024-05-27T07:15:46.210+0000 7f9c21ace200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-05-27T07:15:46.492 INFO:journalctl@ceph.mgr.y.smithi105.stdout:May 27 07:15:46 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mgr-y[37393]: 2024-05-27T07:15:46.300+0000 7f9c21ace200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-05-27T07:15:46.493 INFO:journalctl@ceph.mgr.y.smithi105.stdout:May 27 07:15:46 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mgr-y[37393]: 2024-05-27T07:15:46.502+0000 7f9c21ace200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-05-27T07:15:46.707 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:15:46 smithi052 ceph-mon[31955]: Reconfiguring mgr.x (unknown last config time)... 2024-05-27T07:15:46.707 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:15:46 smithi052 ceph-mon[31955]: Reconfiguring daemon mgr.x on smithi052 2024-05-27T07:15:46.707 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:15:46 smithi052 ceph-mon[31955]: pgmap v32: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-27T07:15:46.707 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:15:46 smithi052 ceph-mon[31955]: from='client.14223 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi052:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-05-27T07:15:46.707 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:15:46 smithi052 ceph-mon[31955]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-05-27T07:15:46.707 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:15:46 smithi052 ceph-mon[31955]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-05-27T07:15:46.707 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:15:46 smithi052 ceph-mon[31955]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T07:15:46.853 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:15:46 smithi173 ceph-mon[37000]: Reconfiguring mgr.x (unknown last config time)... 2024-05-27T07:15:46.853 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:15:46 smithi173 ceph-mon[37000]: Reconfiguring daemon mgr.x on smithi052 2024-05-27T07:15:46.853 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:15:46 smithi173 ceph-mon[37000]: pgmap v32: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-27T07:15:46.853 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:15:46 smithi173 ceph-mon[37000]: from='client.14223 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi052:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-05-27T07:15:46.854 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:15:46 smithi173 ceph-mon[37000]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-05-27T07:15:46.854 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:15:46 smithi173 ceph-mon[37000]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-05-27T07:15:46.854 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:15:46 smithi173 ceph-mon[37000]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T07:15:46.854 INFO:journalctl@ceph.mgr.z.smithi173.stdout:May 27 07:15:46 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mgr-z[38891]: 2024-05-27T07:15:46.468+0000 7f407e904200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-05-27T07:15:46.854 INFO:journalctl@ceph.mgr.z.smithi173.stdout:May 27 07:15:46 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mgr-z[38891]: 2024-05-27T07:15:46.598+0000 7f407e904200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-05-27T07:15:46.855 INFO:journalctl@ceph.mgr.z.smithi173.stdout:May 27 07:15:46 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mgr-z[38891]: 2024-05-27T07:15:46.692+0000 7f407e904200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-05-27T07:15:46.940 INFO:journalctl@ceph.mgr.y.smithi105.stdout:May 27 07:15:46 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mgr-y[37393]: 2024-05-27T07:15:46.694+0000 7f9c21ace200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-05-27T07:15:46.941 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:15:46 smithi105 ceph-mon[35977]: Reconfiguring mgr.x (unknown last config time)... 2024-05-27T07:15:46.941 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:15:46 smithi105 ceph-mon[35977]: Reconfiguring daemon mgr.x on smithi052 2024-05-27T07:15:46.941 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:15:46 smithi105 ceph-mon[35977]: pgmap v32: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-27T07:15:46.941 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:15:46 smithi105 ceph-mon[35977]: from='client.14223 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi052:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-05-27T07:15:46.941 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:15:46 smithi105 ceph-mon[35977]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-05-27T07:15:46.941 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:15:46 smithi105 ceph-mon[35977]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-05-27T07:15:46.941 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:15:46 smithi105 ceph-mon[35977]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T07:15:47.353 INFO:journalctl@ceph.mgr.z.smithi173.stdout:May 27 07:15:47 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mgr-z[38891]: 2024-05-27T07:15:47.026+0000 7f407e904200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-05-27T07:15:47.353 INFO:journalctl@ceph.mgr.z.smithi173.stdout:May 27 07:15:47 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mgr-z[38891]: 2024-05-27T07:15:47.114+0000 7f407e904200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-05-27T07:15:47.690 INFO:journalctl@ceph.mgr.y.smithi105.stdout:May 27 07:15:47 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mgr-y[37393]: 2024-05-27T07:15:47.298+0000 7f9c21ace200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-05-27T07:15:47.690 INFO:journalctl@ceph.mgr.y.smithi105.stdout:May 27 07:15:47 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mgr-y[37393]: 2024-05-27T07:15:47.518+0000 7f9c21ace200 -1 mgr[py] Module smb has missing NOTIFY_TYPES member 2024-05-27T07:15:48.025 INFO:journalctl@ceph.mgr.y.smithi105.stdout:May 27 07:15:47 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mgr-y[37393]: 2024-05-27T07:15:47.940+0000 7f9c21ace200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-05-27T07:15:48.025 INFO:journalctl@ceph.mgr.y.smithi105.stdout:May 27 07:15:48 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mgr-y[37393]: 2024-05-27T07:15:48.034+0000 7f9c21ace200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-05-27T07:15:48.026 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:15:47 smithi105 ceph-mon[35977]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' 2024-05-27T07:15:48.026 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:15:47 smithi105 ceph-mon[35977]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' 2024-05-27T07:15:48.026 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:15:47 smithi105 ceph-mon[35977]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-27T07:15:48.026 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:15:47 smithi105 ceph-mon[35977]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T07:15:48.026 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:15:47 smithi105 ceph-mon[35977]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-27T07:15:48.026 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:15:47 smithi105 ceph-mon[35977]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' 2024-05-27T07:15:48.068 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:15:47 smithi052 ceph-mon[31955]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' 2024-05-27T07:15:48.068 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:15:47 smithi052 ceph-mon[31955]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' 2024-05-27T07:15:48.068 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:15:47 smithi052 ceph-mon[31955]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-27T07:15:48.068 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:15:47 smithi052 ceph-mon[31955]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T07:15:48.068 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:15:47 smithi052 ceph-mon[31955]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-27T07:15:48.068 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:15:47 smithi052 ceph-mon[31955]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' 2024-05-27T07:15:48.102 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:15:47 smithi173 ceph-mon[37000]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' 2024-05-27T07:15:48.103 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:15:47 smithi173 ceph-mon[37000]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' 2024-05-27T07:15:48.103 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:15:47 smithi173 ceph-mon[37000]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-27T07:15:48.103 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:15:47 smithi173 ceph-mon[37000]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T07:15:48.103 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:15:47 smithi173 ceph-mon[37000]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-27T07:15:48.103 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:15:47 smithi173 ceph-mon[37000]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' 2024-05-27T07:15:48.103 INFO:journalctl@ceph.mgr.z.smithi173.stdout:May 27 07:15:47 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mgr-z[38891]: 2024-05-27T07:15:47.701+0000 7f407e904200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-05-27T07:15:48.103 INFO:journalctl@ceph.mgr.z.smithi173.stdout:May 27 07:15:47 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mgr-z[38891]: 2024-05-27T07:15:47.916+0000 7f407e904200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-05-27T07:15:48.690 INFO:journalctl@ceph.mgr.y.smithi105.stdout:May 27 07:15:48 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mgr-y[37393]: 2024-05-27T07:15:48.403+0000 7f9c21ace200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-05-27T07:15:48.821 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:15:48.830+0000 7f15d1ffb700 1 -- 172.21.15.52:0/1177631558 <== mon.0 v2:172.21.15.52:3300/0 7 ==== mgrmap(e 14) ==== 99937+0+0 (secure 0 0 0) 0x7f15c801f340 con 0x7f15d4104e20 2024-05-27T07:15:49.083 INFO:journalctl@ceph.mgr.y.smithi105.stdout:May 27 07:15:48 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mgr-y[37393]: 2024-05-27T07:15:48.756+0000 7f9c21ace200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-05-27T07:15:49.084 INFO:journalctl@ceph.mgr.y.smithi105.stdout:May 27 07:15:48 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mgr-y[37393]: 2024-05-27T07:15:48.850+0000 7f9c21ace200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-05-27T07:15:49.084 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:15:48 smithi105 ceph-mon[35977]: pgmap v33: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-27T07:15:49.084 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:15:48 smithi105 ceph-mon[35977]: Standby manager daemon z started 2024-05-27T07:15:49.084 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:15:48 smithi105 ceph-mon[35977]: from='mgr.? 172.21.15.173:0/3975457533' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/crt"}]: dispatch 2024-05-27T07:15:49.084 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:15:48 smithi105 ceph-mon[35977]: from='mgr.? 172.21.15.173:0/3975457533' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-05-27T07:15:49.084 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:15:48 smithi105 ceph-mon[35977]: from='mgr.? 172.21.15.173:0/3975457533' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/key"}]: dispatch 2024-05-27T07:15:49.084 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:15:48 smithi105 ceph-mon[35977]: from='mgr.? 172.21.15.173:0/3975457533' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-05-27T07:15:49.102 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:15:48 smithi173 ceph-mon[37000]: pgmap v33: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-27T07:15:49.103 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:15:48 smithi173 ceph-mon[37000]: Standby manager daemon z started 2024-05-27T07:15:49.103 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:15:48 smithi173 ceph-mon[37000]: from='mgr.? 172.21.15.173:0/3975457533' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/crt"}]: dispatch 2024-05-27T07:15:49.103 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:15:48 smithi173 ceph-mon[37000]: from='mgr.? 172.21.15.173:0/3975457533' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-05-27T07:15:49.103 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:15:48 smithi173 ceph-mon[37000]: from='mgr.? 172.21.15.173:0/3975457533' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/key"}]: dispatch 2024-05-27T07:15:49.103 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:15:48 smithi173 ceph-mon[37000]: from='mgr.? 172.21.15.173:0/3975457533' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-05-27T07:15:49.224 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:15:48 smithi052 ceph-mon[31955]: pgmap v33: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-27T07:15:49.224 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:15:48 smithi052 ceph-mon[31955]: Standby manager daemon z started 2024-05-27T07:15:49.224 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:15:48 smithi052 ceph-mon[31955]: from='mgr.? 172.21.15.173:0/3975457533' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/crt"}]: dispatch 2024-05-27T07:15:49.224 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:15:48 smithi052 ceph-mon[31955]: from='mgr.? 172.21.15.173:0/3975457533' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-05-27T07:15:49.224 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:15:48 smithi052 ceph-mon[31955]: from='mgr.? 172.21.15.173:0/3975457533' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/key"}]: dispatch 2024-05-27T07:15:49.224 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:15:48 smithi052 ceph-mon[31955]: from='mgr.? 172.21.15.173:0/3975457533' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-05-27T07:15:49.440 INFO:journalctl@ceph.mgr.y.smithi105.stdout:May 27 07:15:49 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mgr-y[37393]: 2024-05-27T07:15:49.092+0000 7f9c21ace200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-05-27T07:15:49.440 INFO:journalctl@ceph.mgr.y.smithi105.stdout:May 27 07:15:49 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mgr-y[37393]: 2024-05-27T07:15:49.225+0000 7f9c21ace200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-05-27T07:15:49.441 INFO:journalctl@ceph.mgr.y.smithi105.stdout:May 27 07:15:49 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mgr-y[37393]: 2024-05-27T07:15:49.319+0000 7f9c21ace200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-05-27T07:15:49.940 INFO:journalctl@ceph.mgr.y.smithi105.stdout:May 27 07:15:49 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mgr-y[37393]: 2024-05-27T07:15:49.642+0000 7f9c21ace200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-05-27T07:15:49.940 INFO:journalctl@ceph.mgr.y.smithi105.stdout:May 27 07:15:49 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mgr-y[37393]: 2024-05-27T07:15:49.732+0000 7f9c21ace200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-05-27T07:15:49.941 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:15:49 smithi105 ceph-mon[35977]: mgrmap e14: x(active, since 79s), standbys: z 2024-05-27T07:15:49.941 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:15:49 smithi105 ceph-mon[35977]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "z", "id": "z"}]: dispatch 2024-05-27T07:15:49.941 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:15:49 smithi105 ceph-mon[35977]: from='client.? 172.21.15.52:0/605874614' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "347b8a89-9dbe-4848-83b0-fd0e9374f33d"}]: dispatch 2024-05-27T07:15:49.941 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:15:49 smithi105 ceph-mon[35977]: from='client.? 172.21.15.52:0/605874614' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "347b8a89-9dbe-4848-83b0-fd0e9374f33d"}]': finished 2024-05-27T07:15:49.941 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:15:49 smithi105 ceph-mon[35977]: osdmap e5: 1 total, 0 up, 1 in 2024-05-27T07:15:49.941 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:15:49 smithi105 ceph-mon[35977]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-05-27T07:15:49.941 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:15:49 smithi105 ceph-mon[35977]: from='client.? 172.21.15.52:0/3540304091' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-05-27T07:15:50.103 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:15:49 smithi173 ceph-mon[37000]: mgrmap e14: x(active, since 79s), standbys: z 2024-05-27T07:15:50.103 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:15:49 smithi173 ceph-mon[37000]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "z", "id": "z"}]: dispatch 2024-05-27T07:15:50.103 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:15:49 smithi173 ceph-mon[37000]: from='client.? 172.21.15.52:0/605874614' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "347b8a89-9dbe-4848-83b0-fd0e9374f33d"}]: dispatch 2024-05-27T07:15:50.103 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:15:49 smithi173 ceph-mon[37000]: from='client.? 172.21.15.52:0/605874614' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "347b8a89-9dbe-4848-83b0-fd0e9374f33d"}]': finished 2024-05-27T07:15:50.103 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:15:49 smithi173 ceph-mon[37000]: osdmap e5: 1 total, 0 up, 1 in 2024-05-27T07:15:50.103 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:15:49 smithi173 ceph-mon[37000]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-05-27T07:15:50.103 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:15:49 smithi173 ceph-mon[37000]: from='client.? 172.21.15.52:0/3540304091' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-05-27T07:15:50.230 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:15:49 smithi052 ceph-mon[31955]: mgrmap e14: x(active, since 79s), standbys: z 2024-05-27T07:15:50.230 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:15:49 smithi052 ceph-mon[31955]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "z", "id": "z"}]: dispatch 2024-05-27T07:15:50.230 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:15:49 smithi052 ceph-mon[31955]: from='client.? 172.21.15.52:0/605874614' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "347b8a89-9dbe-4848-83b0-fd0e9374f33d"}]: dispatch 2024-05-27T07:15:50.230 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:15:49 smithi052 ceph-mon[31955]: from='client.? 172.21.15.52:0/605874614' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "347b8a89-9dbe-4848-83b0-fd0e9374f33d"}]': finished 2024-05-27T07:15:50.230 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:15:49 smithi052 ceph-mon[31955]: osdmap e5: 1 total, 0 up, 1 in 2024-05-27T07:15:50.230 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:15:49 smithi052 ceph-mon[31955]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-05-27T07:15:50.230 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:15:49 smithi052 ceph-mon[31955]: from='client.? 172.21.15.52:0/3540304091' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-05-27T07:15:50.690 INFO:journalctl@ceph.mgr.y.smithi105.stdout:May 27 07:15:50 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mgr-y[37393]: 2024-05-27T07:15:50.331+0000 7f9c21ace200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-05-27T07:15:50.690 INFO:journalctl@ceph.mgr.y.smithi105.stdout:May 27 07:15:50 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mgr-y[37393]: 2024-05-27T07:15:50.545+0000 7f9c21ace200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-05-27T07:15:50.902 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:15:50.911+0000 7f15d1ffb700 1 -- 172.21.15.52:0/1177631558 <== mon.0 v2:172.21.15.52:3300/0 8 ==== mgrmap(e 15) ==== 149596+0+0 (secure 0 0 0) 0x7f15c801f7c0 con 0x7f15d4104e20 2024-05-27T07:15:51.190 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:15:50 smithi105 ceph-mon[35977]: pgmap v35: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-27T07:15:51.191 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:15:50 smithi105 ceph-mon[35977]: Standby manager daemon y started 2024-05-27T07:15:51.191 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:15:50 smithi105 ceph-mon[35977]: from='mgr.? 172.21.15.105:0/2397884874' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/crt"}]: dispatch 2024-05-27T07:15:51.191 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:15:50 smithi105 ceph-mon[35977]: from='mgr.? 172.21.15.105:0/2397884874' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-05-27T07:15:51.191 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:15:50 smithi105 ceph-mon[35977]: from='mgr.? 172.21.15.105:0/2397884874' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/key"}]: dispatch 2024-05-27T07:15:51.191 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:15:50 smithi105 ceph-mon[35977]: from='mgr.? 172.21.15.105:0/2397884874' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-05-27T07:15:51.230 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:15:50 smithi052 ceph-mon[31955]: pgmap v35: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-27T07:15:51.230 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:15:50 smithi052 ceph-mon[31955]: Standby manager daemon y started 2024-05-27T07:15:51.230 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:15:50 smithi052 ceph-mon[31955]: from='mgr.? 172.21.15.105:0/2397884874' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/crt"}]: dispatch 2024-05-27T07:15:51.230 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:15:50 smithi052 ceph-mon[31955]: from='mgr.? 172.21.15.105:0/2397884874' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-05-27T07:15:51.230 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:15:50 smithi052 ceph-mon[31955]: from='mgr.? 172.21.15.105:0/2397884874' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/key"}]: dispatch 2024-05-27T07:15:51.230 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:15:50 smithi052 ceph-mon[31955]: from='mgr.? 172.21.15.105:0/2397884874' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-05-27T07:15:51.352 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:15:50 smithi173 ceph-mon[37000]: pgmap v35: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-27T07:15:51.353 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:15:50 smithi173 ceph-mon[37000]: Standby manager daemon y started 2024-05-27T07:15:51.353 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:15:50 smithi173 ceph-mon[37000]: from='mgr.? 172.21.15.105:0/2397884874' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/crt"}]: dispatch 2024-05-27T07:15:51.353 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:15:50 smithi173 ceph-mon[37000]: from='mgr.? 172.21.15.105:0/2397884874' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-05-27T07:15:51.353 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:15:50 smithi173 ceph-mon[37000]: from='mgr.? 172.21.15.105:0/2397884874' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/key"}]: dispatch 2024-05-27T07:15:51.353 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:15:50 smithi173 ceph-mon[37000]: from='mgr.? 172.21.15.105:0/2397884874' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-05-27T07:15:52.168 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:15:51 smithi173 ceph-mon[37000]: mgrmap e15: x(active, since 81s), standbys: z, y 2024-05-27T07:15:52.169 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:15:51 smithi173 ceph-mon[37000]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-05-27T07:15:52.190 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:15:51 smithi105 ceph-mon[35977]: mgrmap e15: x(active, since 81s), standbys: z, y 2024-05-27T07:15:52.190 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:15:51 smithi105 ceph-mon[35977]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-05-27T07:15:52.230 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:15:51 smithi052 ceph-mon[31955]: mgrmap e15: x(active, since 81s), standbys: z, y 2024-05-27T07:15:52.230 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:15:51 smithi052 ceph-mon[31955]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-05-27T07:15:53.190 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:15:52 smithi105 ceph-mon[35977]: pgmap v36: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-27T07:15:53.230 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:15:52 smithi052 ceph-mon[31955]: pgmap v36: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-27T07:15:53.352 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:15:52 smithi173 ceph-mon[37000]: pgmap v36: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-27T07:15:55.011 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:15:54 smithi052 ceph-mon[31955]: pgmap v37: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-27T07:15:55.190 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:15:54 smithi105 ceph-mon[35977]: pgmap v37: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-27T07:15:55.352 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:15:54 smithi173 ceph-mon[37000]: pgmap v37: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-27T07:15:56.929 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:15:56 smithi052 ceph-mon[31955]: pgmap v38: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-27T07:15:56.929 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:15:56 smithi052 ceph-mon[31955]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2024-05-27T07:15:56.929 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:15:56 smithi052 ceph-mon[31955]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T07:15:56.930 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:15:56 smithi052 ceph-mon[31955]: Deploying daemon osd.0 on smithi052 2024-05-27T07:15:57.190 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:15:56 smithi105 ceph-mon[35977]: pgmap v38: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-27T07:15:57.191 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:15:56 smithi105 ceph-mon[35977]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2024-05-27T07:15:57.191 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:15:56 smithi105 ceph-mon[35977]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T07:15:57.191 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:15:56 smithi105 ceph-mon[35977]: Deploying daemon osd.0 on smithi052 2024-05-27T07:15:57.352 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:15:56 smithi173 ceph-mon[37000]: pgmap v38: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-27T07:15:57.353 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:15:56 smithi173 ceph-mon[37000]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2024-05-27T07:15:57.353 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:15:56 smithi173 ceph-mon[37000]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T07:15:57.353 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:15:56 smithi173 ceph-mon[37000]: Deploying daemon osd.0 on smithi052 2024-05-27T07:15:59.128 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:15:58 smithi052 ceph-mon[31955]: pgmap v39: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-27T07:15:59.190 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:15:58 smithi105 ceph-mon[35977]: pgmap v39: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-27T07:15:59.352 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:15:58 smithi173 ceph-mon[37000]: pgmap v39: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-27T07:16:01.184 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:16:00 smithi052 ceph-mon[31955]: pgmap v40: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-27T07:16:01.190 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:16:00 smithi105 ceph-mon[35977]: pgmap v40: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-27T07:16:01.352 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:16:00 smithi173 ceph-mon[37000]: pgmap v40: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-27T07:16:02.480 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:16:02 smithi052 ceph-mon[31955]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' 2024-05-27T07:16:02.480 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:16:02 smithi052 ceph-mon[31955]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' 2024-05-27T07:16:02.480 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:16:02 smithi052 ceph-mon[31955]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-27T07:16:02.480 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:16:02 smithi052 ceph-mon[31955]: pgmap v41: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-27T07:16:02.602 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:16:02 smithi173 ceph-mon[37000]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' 2024-05-27T07:16:02.603 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:16:02 smithi173 ceph-mon[37000]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' 2024-05-27T07:16:02.603 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:16:02 smithi173 ceph-mon[37000]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-27T07:16:02.603 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:16:02 smithi173 ceph-mon[37000]: pgmap v41: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-27T07:16:02.693 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:16:02 smithi105 ceph-mon[35977]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' 2024-05-27T07:16:02.693 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:16:02 smithi105 ceph-mon[35977]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' 2024-05-27T07:16:02.693 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:16:02 smithi105 ceph-mon[35977]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-27T07:16:02.693 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:16:02 smithi105 ceph-mon[35977]: pgmap v41: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-27T07:16:04.417 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:16:04.426+0000 7f15d1ffb700 1 -- 172.21.15.52:0/1177631558 <== mgr.14152 v2:172.21.15.52:6800/3513075805 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+37 (secure 0 0 0) 0x7f15d4063bf0 con 0x7f15c403e750 2024-05-27T07:16:04.417 INFO:teuthology.orchestra.run.smithi052.stdout:Created osd(s) 0 on host 'smithi052' 2024-05-27T07:16:04.419 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:16:04.428+0000 7f15daed4700 1 -- 172.21.15.52:0/1177631558 >> [v2:172.21.15.52:6800/3513075805,v1:172.21.15.52:6801/3513075805] conn(0x7f15c403e750 msgr2=0x7f15c4040c60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:16:04.419 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:16:04.428+0000 7f15daed4700 1 --2- 172.21.15.52:0/1177631558 >> [v2:172.21.15.52:6800/3513075805,v1:172.21.15.52:6801/3513075805] conn(0x7f15c403e750 0x7f15c4040c60 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f15d4104030 tx=0x7f15c000b720 comp rx=0 tx=0).stop 2024-05-27T07:16:04.419 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:16:04.428+0000 7f15daed4700 1 -- 172.21.15.52:0/1177631558 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f15d4104e20 msgr2=0x7f15d4104680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:16:04.419 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:16:04.428+0000 7f15daed4700 1 --2- 172.21.15.52:0/1177631558 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f15d4104e20 0x7f15d4104680 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7f15c800e970 tx=0x7f15c800ec80 comp rx=0 tx=0).stop 2024-05-27T07:16:04.419 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:16:04.428+0000 7f15daed4700 1 -- 172.21.15.52:0/1177631558 shutdown_connections 2024-05-27T07:16:04.420 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:16:04.428+0000 7f15daed4700 1 --2- 172.21.15.52:0/1177631558 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f15d410a0e0 0x7f15d4103270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:16:04.420 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:16:04.428+0000 7f15daed4700 1 --2- 172.21.15.52:0/1177631558 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f15d4104e20 0x7f15d4104680 unknown :-1 s=CLOSED pgs=110 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:16:04.420 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:16:04.428+0000 7f15daed4700 1 --2- 172.21.15.52:0/1177631558 >> [v2:172.21.15.52:6800/3513075805,v1:172.21.15.52:6801/3513075805] conn(0x7f15c403e750 0x7f15c4040c60 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:16:04.420 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:16:04.428+0000 7f15daed4700 1 --2- 172.21.15.52:0/1177631558 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f15d4107780 0x7f15d4102d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:16:04.420 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:16:04.428+0000 7f15daed4700 1 -- 172.21.15.52:0/1177631558 >> 172.21.15.52:0/1177631558 conn(0x7f15d40fe860 msgr2=0x7f15d41058f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:16:04.420 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:16:04.429+0000 7f15daed4700 1 -- 172.21.15.52:0/1177631558 shutdown_connections 2024-05-27T07:16:04.420 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:16:04.429+0000 7f15daed4700 1 -- 172.21.15.52:0/1177631558 wait complete. 2024-05-27T07:16:04.942 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:16:04 smithi105 ceph-mon[35977]: pgmap v42: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-27T07:16:04.942 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:16:04 smithi105 ceph-mon[35977]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' 2024-05-27T07:16:04.942 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:16:04 smithi105 ceph-mon[35977]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' 2024-05-27T07:16:04.942 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:16:04 smithi105 ceph-mon[35977]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' 2024-05-27T07:16:04.942 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:16:04 smithi105 ceph-mon[35977]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' 2024-05-27T07:16:04.962 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:16:04 smithi052 ceph-mon[31955]: pgmap v42: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-27T07:16:04.962 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:16:04 smithi052 ceph-mon[31955]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' 2024-05-27T07:16:04.962 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:16:04 smithi052 ceph-mon[31955]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' 2024-05-27T07:16:04.962 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:16:04 smithi052 ceph-mon[31955]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' 2024-05-27T07:16:04.962 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:16:04 smithi052 ceph-mon[31955]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' 2024-05-27T07:16:04.993 DEBUG:teuthology.orchestra.run.smithi052:osd.0> sudo journalctl -f -n 0 -u ceph-73df2636-1bf8-11ef-bc9b-c7b262605968@osd.0.service 2024-05-27T07:16:04.996 INFO:tasks.cephadm:Deploying osd.1 on smithi105 with /dev/vg_nvme/lv_4... 2024-05-27T07:16:04.996 DEBUG:teuthology.orchestra.run.smithi105:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:38f9d900c83c63799fbdbe61acc9a11b0d3554a6 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 73df2636-1bf8-11ef-bc9b-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_4 2024-05-27T07:16:05.102 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:16:04 smithi173 ceph-mon[37000]: pgmap v42: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-27T07:16:05.103 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:16:04 smithi173 ceph-mon[37000]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' 2024-05-27T07:16:05.103 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:16:04 smithi173 ceph-mon[37000]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' 2024-05-27T07:16:05.103 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:16:04 smithi173 ceph-mon[37000]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' 2024-05-27T07:16:05.103 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:16:04 smithi173 ceph-mon[37000]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' 2024-05-27T07:16:05.234 INFO:teuthology.orchestra.run.smithi105.stderr:Inferring config /var/lib/ceph/73df2636-1bf8-11ef-bc9b-c7b262605968/mon.b/config 2024-05-27T07:16:05.480 INFO:journalctl@ceph.osd.0.smithi052.stdout:May 27 07:16:05 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-osd-0[47137]: 2024-05-27T07:16:05.248+0000 7f6ca70387c0 -1 osd.0 0 log_to_monitors true 2024-05-27T07:16:05.927 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:16:05 smithi052 ceph-mon[31955]: from='osd.0 [v2:172.21.15.52:6802/1626508899,v1:172.21.15.52:6803/1626508899]' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-05-27T07:16:05.940 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:16:05 smithi105 ceph-mon[35977]: from='osd.0 [v2:172.21.15.52:6802/1626508899,v1:172.21.15.52:6803/1626508899]' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-05-27T07:16:06.102 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:16:05 smithi173 ceph-mon[37000]: from='osd.0 [v2:172.21.15.52:6802/1626508899,v1:172.21.15.52:6803/1626508899]' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-05-27T07:16:06.940 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:16:06 smithi105 ceph-mon[35977]: pgmap v43: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-27T07:16:06.941 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:16:06 smithi105 ceph-mon[35977]: from='osd.0 [v2:172.21.15.52:6802/1626508899,v1:172.21.15.52:6803/1626508899]' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2024-05-27T07:16:06.941 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:16:06 smithi105 ceph-mon[35977]: osdmap e6: 1 total, 0 up, 1 in 2024-05-27T07:16:06.941 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:16:06 smithi105 ceph-mon[35977]: from='osd.0 [v2:172.21.15.52:6802/1626508899,v1:172.21.15.52:6803/1626508899]' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi052", "root=default"]}]: dispatch 2024-05-27T07:16:06.941 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:16:06 smithi105 ceph-mon[35977]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-05-27T07:16:06.964 INFO:journalctl@ceph.osd.0.smithi052.stdout:May 27 07:16:06 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-osd-0[47137]: 2024-05-27T07:16:06.634+0000 7f6c9c69c700 -1 osd.0 0 waiting for initial osdmap 2024-05-27T07:16:06.964 INFO:journalctl@ceph.osd.0.smithi052.stdout:May 27 07:16:06 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-osd-0[47137]: 2024-05-27T07:16:06.636+0000 7f6c98451700 -1 osd.0 7 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-05-27T07:16:06.964 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:16:06 smithi052 ceph-mon[31955]: pgmap v43: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-27T07:16:06.964 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:16:06 smithi052 ceph-mon[31955]: from='osd.0 [v2:172.21.15.52:6802/1626508899,v1:172.21.15.52:6803/1626508899]' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2024-05-27T07:16:06.964 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:16:06 smithi052 ceph-mon[31955]: osdmap e6: 1 total, 0 up, 1 in 2024-05-27T07:16:06.965 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:16:06 smithi052 ceph-mon[31955]: from='osd.0 [v2:172.21.15.52:6802/1626508899,v1:172.21.15.52:6803/1626508899]' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi052", "root=default"]}]: dispatch 2024-05-27T07:16:06.965 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:16:06 smithi052 ceph-mon[31955]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-05-27T07:16:07.102 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:16:06 smithi173 ceph-mon[37000]: pgmap v43: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-27T07:16:07.103 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:16:06 smithi173 ceph-mon[37000]: from='osd.0 [v2:172.21.15.52:6802/1626508899,v1:172.21.15.52:6803/1626508899]' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2024-05-27T07:16:07.103 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:16:06 smithi173 ceph-mon[37000]: osdmap e6: 1 total, 0 up, 1 in 2024-05-27T07:16:07.103 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:16:06 smithi173 ceph-mon[37000]: from='osd.0 [v2:172.21.15.52:6802/1626508899,v1:172.21.15.52:6803/1626508899]' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi052", "root=default"]}]: dispatch 2024-05-27T07:16:07.103 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:16:06 smithi173 ceph-mon[37000]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-05-27T07:16:07.130 INFO:teuthology.orchestra.run.smithi105.stderr:--> Zapping: /dev/vg_nvme/lv_4 2024-05-27T07:16:07.130 INFO:teuthology.orchestra.run.smithi105.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_4 bs=1M count=10 conv=fsync 2024-05-27T07:16:07.130 INFO:teuthology.orchestra.run.smithi105.stderr: stderr: 10+0 records in 2024-05-27T07:16:07.130 INFO:teuthology.orchestra.run.smithi105.stderr:10+0 records out 2024-05-27T07:16:07.130 INFO:teuthology.orchestra.run.smithi105.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0114283 s, 918 MB/s 2024-05-27T07:16:07.130 INFO:teuthology.orchestra.run.smithi105.stderr:--> Zapping successful for: 2024-05-27T07:16:07.690 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:16:07 smithi105 ceph-mon[35977]: from='osd.0 [v2:172.21.15.52:6802/1626508899,v1:172.21.15.52:6803/1626508899]' entity='osd.0' cmd='[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi052", "root=default"]}]': finished 2024-05-27T07:16:07.691 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:16:07 smithi105 ceph-mon[35977]: osdmap e7: 1 total, 0 up, 1 in 2024-05-27T07:16:07.691 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:16:07 smithi105 ceph-mon[35977]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-05-27T07:16:07.691 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:16:07 smithi105 ceph-mon[35977]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-05-27T07:16:07.691 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:16:07 smithi105 ceph-mon[35977]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' 2024-05-27T07:16:07.691 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:16:07 smithi105 ceph-mon[35977]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' 2024-05-27T07:16:07.691 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:16:07 smithi105 ceph-mon[35977]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-05-27T07:16:07.691 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:16:07 smithi105 ceph-mon[35977]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' 2024-05-27T07:16:07.691 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:16:07 smithi105 ceph-mon[35977]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T07:16:07.691 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:16:07 smithi105 ceph-mon[35977]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-27T07:16:07.691 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:16:07 smithi105 ceph-mon[35977]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' 2024-05-27T07:16:07.721 DEBUG:teuthology.orchestra.run.smithi105:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:38f9d900c83c63799fbdbe61acc9a11b0d3554a6 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 73df2636-1bf8-11ef-bc9b-c7b262605968 -- ceph orch daemon add osd smithi105:vg_nvme/lv_4 2024-05-27T07:16:07.980 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:16:07 smithi052 ceph-mon[31955]: from='osd.0 [v2:172.21.15.52:6802/1626508899,v1:172.21.15.52:6803/1626508899]' entity='osd.0' cmd='[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi052", "root=default"]}]': finished 2024-05-27T07:16:07.980 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:16:07 smithi052 ceph-mon[31955]: osdmap e7: 1 total, 0 up, 1 in 2024-05-27T07:16:07.980 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:16:07 smithi052 ceph-mon[31955]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-05-27T07:16:07.980 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:16:07 smithi052 ceph-mon[31955]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-05-27T07:16:07.980 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:16:07 smithi052 ceph-mon[31955]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' 2024-05-27T07:16:07.980 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:16:07 smithi052 ceph-mon[31955]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' 2024-05-27T07:16:07.980 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:16:07 smithi052 ceph-mon[31955]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-05-27T07:16:07.981 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:16:07 smithi052 ceph-mon[31955]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' 2024-05-27T07:16:07.981 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:16:07 smithi052 ceph-mon[31955]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T07:16:07.981 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:16:07 smithi052 ceph-mon[31955]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-27T07:16:07.981 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:16:07 smithi052 ceph-mon[31955]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' 2024-05-27T07:16:07.982 INFO:teuthology.orchestra.run.smithi105.stderr:Inferring config /var/lib/ceph/73df2636-1bf8-11ef-bc9b-c7b262605968/mon.b/config 2024-05-27T07:16:08.102 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:16:07 smithi173 ceph-mon[37000]: from='osd.0 [v2:172.21.15.52:6802/1626508899,v1:172.21.15.52:6803/1626508899]' entity='osd.0' cmd='[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi052", "root=default"]}]': finished 2024-05-27T07:16:08.103 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:16:07 smithi173 ceph-mon[37000]: osdmap e7: 1 total, 0 up, 1 in 2024-05-27T07:16:08.103 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:16:07 smithi173 ceph-mon[37000]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-05-27T07:16:08.103 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:16:07 smithi173 ceph-mon[37000]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-05-27T07:16:08.103 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:16:07 smithi173 ceph-mon[37000]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' 2024-05-27T07:16:08.103 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:16:07 smithi173 ceph-mon[37000]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' 2024-05-27T07:16:08.103 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:16:07 smithi173 ceph-mon[37000]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-05-27T07:16:08.103 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:16:07 smithi173 ceph-mon[37000]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' 2024-05-27T07:16:08.103 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:16:07 smithi173 ceph-mon[37000]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T07:16:08.103 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:16:07 smithi173 ceph-mon[37000]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-27T07:16:08.103 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:16:07 smithi173 ceph-mon[37000]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' 2024-05-27T07:16:08.690 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:16:08 smithi105 ceph-mon[35977]: purged_snaps scrub starts 2024-05-27T07:16:08.691 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:16:08 smithi105 ceph-mon[35977]: purged_snaps scrub ok 2024-05-27T07:16:08.691 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:16:08 smithi105 ceph-mon[35977]: pgmap v46: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-27T07:16:08.691 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:16:08 smithi105 ceph-mon[35977]: Detected new or changed devices on smithi052 2024-05-27T07:16:08.691 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:16:08 smithi105 ceph-mon[35977]: Adjusting osd_memory_target on smithi052 to 17093M 2024-05-27T07:16:08.691 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:16:08 smithi105 ceph-mon[35977]: osd.0 [v2:172.21.15.52:6802/1626508899,v1:172.21.15.52:6803/1626508899] boot 2024-05-27T07:16:08.691 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:16:08 smithi105 ceph-mon[35977]: osdmap e8: 1 total, 1 up, 1 in 2024-05-27T07:16:08.691 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:16:08 smithi105 ceph-mon[35977]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-05-27T07:16:08.691 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:16:08 smithi105 ceph-mon[35977]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-27T07:16:08.691 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:16:08 smithi105 ceph-mon[35977]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T07:16:08.691 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:16:08 smithi105 ceph-mon[35977]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-27T07:16:08.691 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:16:08 smithi105 ceph-mon[35977]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' 2024-05-27T07:16:08.980 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:16:08 smithi052 ceph-mon[31955]: purged_snaps scrub starts 2024-05-27T07:16:08.980 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:16:08 smithi052 ceph-mon[31955]: purged_snaps scrub ok 2024-05-27T07:16:08.980 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:16:08 smithi052 ceph-mon[31955]: pgmap v46: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-27T07:16:08.980 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:16:08 smithi052 ceph-mon[31955]: Detected new or changed devices on smithi052 2024-05-27T07:16:08.980 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:16:08 smithi052 ceph-mon[31955]: Adjusting osd_memory_target on smithi052 to 17093M 2024-05-27T07:16:08.980 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:16:08 smithi052 ceph-mon[31955]: osd.0 [v2:172.21.15.52:6802/1626508899,v1:172.21.15.52:6803/1626508899] boot 2024-05-27T07:16:08.980 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:16:08 smithi052 ceph-mon[31955]: osdmap e8: 1 total, 1 up, 1 in 2024-05-27T07:16:08.980 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:16:08 smithi052 ceph-mon[31955]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-05-27T07:16:08.980 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:16:08 smithi052 ceph-mon[31955]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-27T07:16:08.981 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:16:08 smithi052 ceph-mon[31955]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T07:16:08.981 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:16:08 smithi052 ceph-mon[31955]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-27T07:16:08.981 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:16:08 smithi052 ceph-mon[31955]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' 2024-05-27T07:16:09.004 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:16:08 smithi173 ceph-mon[37000]: purged_snaps scrub starts 2024-05-27T07:16:09.004 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:16:08 smithi173 ceph-mon[37000]: purged_snaps scrub ok 2024-05-27T07:16:09.004 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:16:08 smithi173 ceph-mon[37000]: pgmap v46: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-27T07:16:09.004 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:16:08 smithi173 ceph-mon[37000]: Detected new or changed devices on smithi052 2024-05-27T07:16:09.004 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:16:08 smithi173 ceph-mon[37000]: Adjusting osd_memory_target on smithi052 to 17093M 2024-05-27T07:16:09.004 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:16:08 smithi173 ceph-mon[37000]: osd.0 [v2:172.21.15.52:6802/1626508899,v1:172.21.15.52:6803/1626508899] boot 2024-05-27T07:16:09.004 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:16:08 smithi173 ceph-mon[37000]: osdmap e8: 1 total, 1 up, 1 in 2024-05-27T07:16:09.005 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:16:08 smithi173 ceph-mon[37000]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-05-27T07:16:09.005 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:16:08 smithi173 ceph-mon[37000]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-27T07:16:09.005 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:16:08 smithi173 ceph-mon[37000]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T07:16:09.005 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:16:08 smithi173 ceph-mon[37000]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-27T07:16:09.005 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:16:08 smithi173 ceph-mon[37000]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' 2024-05-27T07:16:09.543 INFO:teuthology.orchestra.run.smithi105.stderr:2024-05-27T07:16:09.550+0000 7ff3fd914700 1 -- 172.21.15.105:0/3852347291 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7ff3f81058f0 msgr2=0x7ff3f8105d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:16:09.543 INFO:teuthology.orchestra.run.smithi105.stderr:2024-05-27T07:16:09.550+0000 7ff3fd914700 1 --2- 172.21.15.105:0/3852347291 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7ff3f81058f0 0x7ff3f8105d00 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7ff3e800b0a0 tx=0x7ff3e800b3b0 comp rx=0 tx=0).stop 2024-05-27T07:16:09.543 INFO:teuthology.orchestra.run.smithi105.stderr:2024-05-27T07:16:09.550+0000 7ff3fd914700 1 -- 172.21.15.105:0/3852347291 shutdown_connections 2024-05-27T07:16:09.544 INFO:teuthology.orchestra.run.smithi105.stderr:2024-05-27T07:16:09.550+0000 7ff3fd914700 1 --2- 172.21.15.105:0/3852347291 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7ff3f8102c90 0x7ff3f810c8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:16:09.544 INFO:teuthology.orchestra.run.smithi105.stderr:2024-05-27T07:16:09.550+0000 7ff3fd914700 1 --2- 172.21.15.105:0/3852347291 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7ff3f81058f0 0x7ff3f8105d00 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:16:09.544 INFO:teuthology.orchestra.run.smithi105.stderr:2024-05-27T07:16:09.550+0000 7ff3fd914700 1 --2- 172.21.15.105:0/3852347291 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff3f8102340 0x7ff3f8102750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:16:09.544 INFO:teuthology.orchestra.run.smithi105.stderr:2024-05-27T07:16:09.550+0000 7ff3fd914700 1 -- 172.21.15.105:0/3852347291 >> 172.21.15.105:0/3852347291 conn(0x7ff3f80fe010 msgr2=0x7ff3f81004c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:16:09.544 INFO:teuthology.orchestra.run.smithi105.stderr:2024-05-27T07:16:09.550+0000 7ff3fd914700 1 -- 172.21.15.105:0/3852347291 shutdown_connections 2024-05-27T07:16:09.544 INFO:teuthology.orchestra.run.smithi105.stderr:2024-05-27T07:16:09.550+0000 7ff3fd914700 1 -- 172.21.15.105:0/3852347291 wait complete. 2024-05-27T07:16:09.544 INFO:teuthology.orchestra.run.smithi105.stderr:2024-05-27T07:16:09.551+0000 7ff3fd914700 1 Processor -- start 2024-05-27T07:16:09.544 INFO:teuthology.orchestra.run.smithi105.stderr:2024-05-27T07:16:09.551+0000 7ff3fd914700 1 -- start start 2024-05-27T07:16:09.544 INFO:teuthology.orchestra.run.smithi105.stderr:2024-05-27T07:16:09.551+0000 7ff3fd914700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7ff3f8102340 0x7ff3f819a950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:16:09.544 INFO:teuthology.orchestra.run.smithi105.stderr:2024-05-27T07:16:09.551+0000 7ff3fd914700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7ff3f8102c90 0x7ff3f819ae90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:16:09.544 INFO:teuthology.orchestra.run.smithi105.stderr:2024-05-27T07:16:09.551+0000 7ff3fd914700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff3f81058f0 0x7ff3f81a1e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:16:09.544 INFO:teuthology.orchestra.run.smithi105.stderr:2024-05-27T07:16:09.552+0000 7ff3fd914700 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x7ff3f810fde0 con 0x7ff3f81058f0 2024-05-27T07:16:09.544 INFO:teuthology.orchestra.run.smithi105.stderr:2024-05-27T07:16:09.552+0000 7ff3fd914700 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7ff3f810fc60 con 0x7ff3f8102c90 2024-05-27T07:16:09.544 INFO:teuthology.orchestra.run.smithi105.stderr:2024-05-27T07:16:09.552+0000 7ff3fd914700 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7ff3f810ff60 con 0x7ff3f8102340 2024-05-27T07:16:09.545 INFO:teuthology.orchestra.run.smithi105.stderr:2024-05-27T07:16:09.552+0000 7ff3f67fc700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7ff3f8102c90 0x7ff3f819ae90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:16:09.545 INFO:teuthology.orchestra.run.smithi105.stderr:2024-05-27T07:16:09.552+0000 7ff3f67fc700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7ff3f8102c90 0x7ff3f819ae90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.105:40628/0 (socket says 172.21.15.105:40628) 2024-05-27T07:16:09.545 INFO:teuthology.orchestra.run.smithi105.stderr:2024-05-27T07:16:09.552+0000 7ff3f67fc700 1 -- 172.21.15.105:0/1711842 learned_addr learned my addr 172.21.15.105:0/1711842 (peer_addr_for_me v2:172.21.15.105:0/0) 2024-05-27T07:16:09.545 INFO:teuthology.orchestra.run.smithi105.stderr:2024-05-27T07:16:09.552+0000 7ff3f6ffd700 1 --2- 172.21.15.105:0/1711842 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7ff3f8102340 0x7ff3f819a950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:16:09.545 INFO:teuthology.orchestra.run.smithi105.stderr:2024-05-27T07:16:09.552+0000 7ff3f77fe700 1 --2- 172.21.15.105:0/1711842 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff3f81058f0 0x7ff3f81a1e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:16:09.545 INFO:teuthology.orchestra.run.smithi105.stderr:2024-05-27T07:16:09.552+0000 7ff3f67fc700 1 -- 172.21.15.105:0/1711842 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7ff3f8102340 msgr2=0x7ff3f819a950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:16:09.545 INFO:teuthology.orchestra.run.smithi105.stderr:2024-05-27T07:16:09.552+0000 7ff3f67fc700 1 --2- 172.21.15.105:0/1711842 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7ff3f8102340 0x7ff3f819a950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:16:09.545 INFO:teuthology.orchestra.run.smithi105.stderr:2024-05-27T07:16:09.552+0000 7ff3f67fc700 1 -- 172.21.15.105:0/1711842 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff3f81058f0 msgr2=0x7ff3f81a1e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:16:09.545 INFO:teuthology.orchestra.run.smithi105.stderr:2024-05-27T07:16:09.552+0000 7ff3f67fc700 1 --2- 172.21.15.105:0/1711842 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff3f81058f0 0x7ff3f81a1e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:16:09.545 INFO:teuthology.orchestra.run.smithi105.stderr:2024-05-27T07:16:09.552+0000 7ff3f67fc700 1 -- 172.21.15.105:0/1711842 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff3f81a24b0 con 0x7ff3f8102c90 2024-05-27T07:16:09.545 INFO:teuthology.orchestra.run.smithi105.stderr:2024-05-27T07:16:09.552+0000 7ff3f67fc700 1 --2- 172.21.15.105:0/1711842 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7ff3f8102c90 0x7ff3f819ae90 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7ff3ec00d980 tx=0x7ff3ec00dc90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:16:09.545 INFO:teuthology.orchestra.run.smithi105.stderr:2024-05-27T07:16:09.552+0000 7ff3fc912700 1 -- 172.21.15.105:0/1711842 <== mon.2 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff3ec016070 con 0x7ff3f8102c90 2024-05-27T07:16:09.545 INFO:teuthology.orchestra.run.smithi105.stderr:2024-05-27T07:16:09.552+0000 7ff3fc912700 1 -- 172.21.15.105:0/1711842 <== mon.2 v2:172.21.15.105:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ff3ec004830 con 0x7ff3f8102c90 2024-05-27T07:16:09.546 INFO:teuthology.orchestra.run.smithi105.stderr:2024-05-27T07:16:09.552+0000 7ff3fd914700 1 -- 172.21.15.105:0/1711842 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff3f81a27a0 con 0x7ff3f8102c90 2024-05-27T07:16:09.546 INFO:teuthology.orchestra.run.smithi105.stderr:2024-05-27T07:16:09.552+0000 7ff3fc912700 1 -- 172.21.15.105:0/1711842 <== mon.2 v2:172.21.15.105:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff3ec005540 con 0x7ff3f8102c90 2024-05-27T07:16:09.546 INFO:teuthology.orchestra.run.smithi105.stderr:2024-05-27T07:16:09.553+0000 7ff3fd914700 1 -- 172.21.15.105:0/1711842 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff3f81a2d30 con 0x7ff3f8102c90 2024-05-27T07:16:09.546 INFO:teuthology.orchestra.run.smithi105.stderr:2024-05-27T07:16:09.554+0000 7ff3fd914700 1 -- 172.21.15.105:0/1711842 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff3f80512b0 con 0x7ff3f8102c90 2024-05-27T07:16:09.546 INFO:teuthology.orchestra.run.smithi105.stderr:2024-05-27T07:16:09.555+0000 7ff3fc912700 1 -- 172.21.15.105:0/1711842 <== mon.2 v2:172.21.15.105:3300/0 4 ==== mgrmap(e 15) ==== 149596+0+0 (secure 0 0 0) 0x7ff3ec004220 con 0x7ff3f8102c90 2024-05-27T07:16:09.546 INFO:teuthology.orchestra.run.smithi105.stderr:2024-05-27T07:16:09.556+0000 7ff3fc912700 1 --2- 172.21.15.105:0/1711842 >> [v2:172.21.15.52:6800/3513075805,v1:172.21.15.52:6801/3513075805] conn(0x7ff3e00b2870 0x7ff3e00b4d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:16:09.547 INFO:teuthology.orchestra.run.smithi105.stderr:2024-05-27T07:16:09.556+0000 7ff3fc912700 1 -- 172.21.15.105:0/1711842 <== mon.2 v2:172.21.15.105:3300/0 5 ==== osd_map(9..9 src has 1..9) ==== 1767+0+0 (secure 0 0 0) 0x7ff3ec0e0a50 con 0x7ff3f8102c90 2024-05-27T07:16:09.547 INFO:teuthology.orchestra.run.smithi105.stderr:2024-05-27T07:16:09.556+0000 7ff3f6ffd700 1 --2- 172.21.15.105:0/1711842 >> [v2:172.21.15.52:6800/3513075805,v1:172.21.15.52:6801/3513075805] conn(0x7ff3e00b2870 0x7ff3e00b4d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:16:09.547 INFO:teuthology.orchestra.run.smithi105.stderr:2024-05-27T07:16:09.557+0000 7ff3f6ffd700 1 --2- 172.21.15.105:0/1711842 >> [v2:172.21.15.52:6800/3513075805,v1:172.21.15.52:6801/3513075805] conn(0x7ff3e00b2870 0x7ff3e00b4d80 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7ff3e80062a0 tx=0x7ff3e8006210 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:16:09.551 INFO:teuthology.orchestra.run.smithi105.stderr:2024-05-27T07:16:09.560+0000 7ff3fc912700 1 -- 172.21.15.105:0/1711842 <== mon.2 v2:172.21.15.105:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191963 (secure 0 0 0) 0x7ff3ec01f020 con 0x7ff3f8102c90 2024-05-27T07:16:09.766 INFO:teuthology.orchestra.run.smithi105.stderr:2024-05-27T07:16:09.775+0000 7ff3fd914700 1 -- 172.21.15.105:0/1711842 --> [v2:172.21.15.52:6800/3513075805,v1:172.21.15.52:6801/3513075805] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi105:vg_nvme/lv_4", "target": ["mon-mgr", ""]}) -- 0x7ff3f8063bf0 con 0x7ff3e00b2870 2024-05-27T07:16:09.940 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:16:09 smithi105 ceph-mon[35977]: osdmap e9: 1 total, 1 up, 1 in 2024-05-27T07:16:09.980 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:16:09 smithi052 ceph-mon[31955]: osdmap e9: 1 total, 1 up, 1 in 2024-05-27T07:16:10.030 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:16:09 smithi173 ceph-mon[37000]: osdmap e9: 1 total, 1 up, 1 in 2024-05-27T07:16:10.940 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:16:10 smithi105 ceph-mon[35977]: pgmap v49: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-05-27T07:16:10.940 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:16:10 smithi105 ceph-mon[35977]: from='client.24158 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi105:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-05-27T07:16:10.941 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:16:10 smithi105 ceph-mon[35977]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-05-27T07:16:10.941 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:16:10 smithi105 ceph-mon[35977]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-05-27T07:16:10.941 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:16:10 smithi105 ceph-mon[35977]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T07:16:10.980 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:16:10 smithi052 ceph-mon[31955]: pgmap v49: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-05-27T07:16:10.980 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:16:10 smithi052 ceph-mon[31955]: from='client.24158 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi105:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-05-27T07:16:10.980 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:16:10 smithi052 ceph-mon[31955]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-05-27T07:16:10.980 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:16:10 smithi052 ceph-mon[31955]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-05-27T07:16:10.980 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:16:10 smithi052 ceph-mon[31955]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T07:16:11.102 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:16:10 smithi173 ceph-mon[37000]: pgmap v49: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-05-27T07:16:11.103 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:16:10 smithi173 ceph-mon[37000]: from='client.24158 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi105:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-05-27T07:16:11.103 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:16:10 smithi173 ceph-mon[37000]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-05-27T07:16:11.103 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:16:10 smithi173 ceph-mon[37000]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-05-27T07:16:11.103 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:16:10 smithi173 ceph-mon[37000]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T07:16:12.941 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:16:12 smithi105 ceph-mon[35977]: pgmap v50: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-05-27T07:16:12.941 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:16:12 smithi105 ceph-mon[35977]: from='client.? 172.21.15.105:0/1302677319' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "7a3623d4-e087-40cb-9336-d4a5c52203d4"}]: dispatch 2024-05-27T07:16:12.941 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:16:12 smithi105 ceph-mon[35977]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "7a3623d4-e087-40cb-9336-d4a5c52203d4"}]: dispatch 2024-05-27T07:16:12.941 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:16:12 smithi105 ceph-mon[35977]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "7a3623d4-e087-40cb-9336-d4a5c52203d4"}]': finished 2024-05-27T07:16:12.941 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:16:12 smithi105 ceph-mon[35977]: osdmap e10: 2 total, 1 up, 2 in 2024-05-27T07:16:12.941 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:16:12 smithi105 ceph-mon[35977]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-05-27T07:16:12.980 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:16:12 smithi052 ceph-mon[31955]: pgmap v50: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-05-27T07:16:12.980 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:16:12 smithi052 ceph-mon[31955]: from='client.? 172.21.15.105:0/1302677319' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "7a3623d4-e087-40cb-9336-d4a5c52203d4"}]: dispatch 2024-05-27T07:16:12.980 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:16:12 smithi052 ceph-mon[31955]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "7a3623d4-e087-40cb-9336-d4a5c52203d4"}]: dispatch 2024-05-27T07:16:12.980 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:16:12 smithi052 ceph-mon[31955]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "7a3623d4-e087-40cb-9336-d4a5c52203d4"}]': finished 2024-05-27T07:16:12.980 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:16:12 smithi052 ceph-mon[31955]: osdmap e10: 2 total, 1 up, 2 in 2024-05-27T07:16:12.980 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:16:12 smithi052 ceph-mon[31955]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-05-27T07:16:13.103 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:16:12 smithi173 ceph-mon[37000]: pgmap v50: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-05-27T07:16:13.103 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:16:12 smithi173 ceph-mon[37000]: from='client.? 172.21.15.105:0/1302677319' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "7a3623d4-e087-40cb-9336-d4a5c52203d4"}]: dispatch 2024-05-27T07:16:13.103 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:16:12 smithi173 ceph-mon[37000]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "7a3623d4-e087-40cb-9336-d4a5c52203d4"}]: dispatch 2024-05-27T07:16:13.103 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:16:12 smithi173 ceph-mon[37000]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "7a3623d4-e087-40cb-9336-d4a5c52203d4"}]': finished 2024-05-27T07:16:13.103 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:16:12 smithi173 ceph-mon[37000]: osdmap e10: 2 total, 1 up, 2 in 2024-05-27T07:16:13.103 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:16:12 smithi173 ceph-mon[37000]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-05-27T07:16:13.940 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:16:13 smithi105 ceph-mon[35977]: from='client.? 172.21.15.105:0/3197955508' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-05-27T07:16:13.980 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:16:13 smithi052 ceph-mon[31955]: from='client.? 172.21.15.105:0/3197955508' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-05-27T07:16:14.102 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:16:13 smithi173 ceph-mon[37000]: from='client.? 172.21.15.105:0/3197955508' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-05-27T07:16:14.940 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:16:14 smithi105 ceph-mon[35977]: pgmap v52: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-05-27T07:16:14.980 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:16:14 smithi052 ceph-mon[31955]: pgmap v52: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-05-27T07:16:15.102 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:16:14 smithi173 ceph-mon[37000]: pgmap v52: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-05-27T07:16:16.940 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:16:16 smithi105 ceph-mon[35977]: pgmap v53: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-05-27T07:16:16.980 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:16:16 smithi052 ceph-mon[31955]: pgmap v53: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-05-27T07:16:17.102 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:16:16 smithi173 ceph-mon[37000]: pgmap v53: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-05-27T07:16:18.690 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:16:18 smithi105 ceph-mon[35977]: pgmap v54: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-05-27T07:16:18.980 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:16:18 smithi052 ceph-mon[31955]: pgmap v54: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-05-27T07:16:19.104 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:16:18 smithi173 ceph-mon[37000]: pgmap v54: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-05-27T07:16:19.940 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:16:19 smithi105 ceph-mon[35977]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2024-05-27T07:16:19.940 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:16:19 smithi105 ceph-mon[35977]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T07:16:19.980 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:16:19 smithi052 ceph-mon[31955]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2024-05-27T07:16:19.980 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:16:19 smithi052 ceph-mon[31955]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T07:16:20.102 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:16:19 smithi173 ceph-mon[37000]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2024-05-27T07:16:20.102 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:16:19 smithi173 ceph-mon[37000]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T07:16:20.940 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:16:20 smithi105 ceph-mon[35977]: Deploying daemon osd.1 on smithi105 2024-05-27T07:16:20.941 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:16:20 smithi105 ceph-mon[35977]: pgmap v55: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-05-27T07:16:20.980 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:16:20 smithi052 ceph-mon[31955]: Deploying daemon osd.1 on smithi105 2024-05-27T07:16:20.980 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:16:20 smithi052 ceph-mon[31955]: pgmap v55: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-05-27T07:16:21.102 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:16:20 smithi173 ceph-mon[37000]: Deploying daemon osd.1 on smithi105 2024-05-27T07:16:21.102 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:16:20 smithi173 ceph-mon[37000]: pgmap v55: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-05-27T07:16:22.891 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:16:22 smithi105 ceph-mon[35977]: pgmap v56: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-05-27T07:16:22.980 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:16:22 smithi052 ceph-mon[31955]: pgmap v56: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-05-27T07:16:23.102 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:16:22 smithi173 ceph-mon[37000]: pgmap v56: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-05-27T07:16:24.941 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:16:24 smithi105 ceph-mon[35977]: pgmap v57: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-05-27T07:16:24.979 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:16:24 smithi052 ceph-mon[31955]: pgmap v57: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-05-27T07:16:25.102 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:16:24 smithi173 ceph-mon[37000]: pgmap v57: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-05-27T07:16:25.980 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:16:25 smithi052 ceph-mon[31955]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' 2024-05-27T07:16:25.980 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:16:25 smithi052 ceph-mon[31955]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' 2024-05-27T07:16:25.980 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:16:25 smithi052 ceph-mon[31955]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-27T07:16:26.024 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:16:25 smithi105 ceph-mon[35977]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' 2024-05-27T07:16:26.025 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:16:25 smithi105 ceph-mon[35977]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' 2024-05-27T07:16:26.025 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:16:25 smithi105 ceph-mon[35977]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-27T07:16:26.102 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:16:25 smithi173 ceph-mon[37000]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' 2024-05-27T07:16:26.103 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:16:25 smithi173 ceph-mon[37000]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' 2024-05-27T07:16:26.103 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:16:25 smithi173 ceph-mon[37000]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-27T07:16:26.940 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:16:26 smithi105 ceph-mon[35977]: pgmap v58: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-05-27T07:16:26.980 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:16:26 smithi052 ceph-mon[31955]: pgmap v58: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-05-27T07:16:27.102 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:16:26 smithi173 ceph-mon[37000]: pgmap v58: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-05-27T07:16:28.352 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:16:28 smithi173 ceph-mon[37000]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' 2024-05-27T07:16:28.353 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:16:28 smithi173 ceph-mon[37000]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' 2024-05-27T07:16:28.353 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:16:28 smithi173 ceph-mon[37000]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T07:16:28.353 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:16:28 smithi173 ceph-mon[37000]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-27T07:16:28.353 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:16:28 smithi173 ceph-mon[37000]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' 2024-05-27T07:16:28.353 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:16:28 smithi173 ceph-mon[37000]: pgmap v59: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-05-27T07:16:28.416 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:16:28 smithi105 ceph-mon[35977]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' 2024-05-27T07:16:28.417 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:16:28 smithi105 ceph-mon[35977]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' 2024-05-27T07:16:28.417 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:16:28 smithi105 ceph-mon[35977]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T07:16:28.417 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:16:28 smithi105 ceph-mon[35977]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-27T07:16:28.417 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:16:28 smithi105 ceph-mon[35977]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' 2024-05-27T07:16:28.417 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:16:28 smithi105 ceph-mon[35977]: pgmap v59: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-05-27T07:16:28.480 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:16:28 smithi052 ceph-mon[31955]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' 2024-05-27T07:16:28.480 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:16:28 smithi052 ceph-mon[31955]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' 2024-05-27T07:16:28.480 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:16:28 smithi052 ceph-mon[31955]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T07:16:28.480 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:16:28 smithi052 ceph-mon[31955]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-27T07:16:28.480 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:16:28 smithi052 ceph-mon[31955]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' 2024-05-27T07:16:28.480 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:16:28 smithi052 ceph-mon[31955]: pgmap v59: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-05-27T07:16:28.482 INFO:teuthology.orchestra.run.smithi105.stderr:2024-05-27T07:16:28.492+0000 7ff3fc912700 1 -- 172.21.15.105:0/1711842 <== mgr.14152 v2:172.21.15.52:6800/3513075805 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+37 (secure 0 0 0) 0x7ff3f8063bf0 con 0x7ff3e00b2870 2024-05-27T07:16:28.483 INFO:teuthology.orchestra.run.smithi105.stdout:Created osd(s) 1 on host 'smithi105' 2024-05-27T07:16:28.485 INFO:teuthology.orchestra.run.smithi105.stderr:2024-05-27T07:16:28.494+0000 7ff3fd914700 1 -- 172.21.15.105:0/1711842 >> [v2:172.21.15.52:6800/3513075805,v1:172.21.15.52:6801/3513075805] conn(0x7ff3e00b2870 msgr2=0x7ff3e00b4d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:16:28.485 INFO:teuthology.orchestra.run.smithi105.stderr:2024-05-27T07:16:28.494+0000 7ff3fd914700 1 --2- 172.21.15.105:0/1711842 >> [v2:172.21.15.52:6800/3513075805,v1:172.21.15.52:6801/3513075805] conn(0x7ff3e00b2870 0x7ff3e00b4d80 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7ff3e80062a0 tx=0x7ff3e8006210 comp rx=0 tx=0).stop 2024-05-27T07:16:28.485 INFO:teuthology.orchestra.run.smithi105.stderr:2024-05-27T07:16:28.494+0000 7ff3fd914700 1 -- 172.21.15.105:0/1711842 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7ff3f8102c90 msgr2=0x7ff3f819ae90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:16:28.485 INFO:teuthology.orchestra.run.smithi105.stderr:2024-05-27T07:16:28.494+0000 7ff3fd914700 1 --2- 172.21.15.105:0/1711842 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7ff3f8102c90 0x7ff3f819ae90 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7ff3ec00d980 tx=0x7ff3ec00dc90 comp rx=0 tx=0).stop 2024-05-27T07:16:28.485 INFO:teuthology.orchestra.run.smithi105.stderr:2024-05-27T07:16:28.494+0000 7ff3fd914700 1 -- 172.21.15.105:0/1711842 shutdown_connections 2024-05-27T07:16:28.485 INFO:teuthology.orchestra.run.smithi105.stderr:2024-05-27T07:16:28.494+0000 7ff3fd914700 1 --2- 172.21.15.105:0/1711842 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff3f81058f0 0x7ff3f81a1e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:16:28.485 INFO:teuthology.orchestra.run.smithi105.stderr:2024-05-27T07:16:28.494+0000 7ff3fd914700 1 --2- 172.21.15.105:0/1711842 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7ff3f8102340 0x7ff3f819a950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:16:28.485 INFO:teuthology.orchestra.run.smithi105.stderr:2024-05-27T07:16:28.494+0000 7ff3fd914700 1 --2- 172.21.15.105:0/1711842 >> [v2:172.21.15.52:6800/3513075805,v1:172.21.15.52:6801/3513075805] conn(0x7ff3e00b2870 0x7ff3e00b4d80 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:16:28.486 INFO:teuthology.orchestra.run.smithi105.stderr:2024-05-27T07:16:28.494+0000 7ff3fd914700 1 --2- 172.21.15.105:0/1711842 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7ff3f8102c90 0x7ff3f819ae90 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:16:28.486 INFO:teuthology.orchestra.run.smithi105.stderr:2024-05-27T07:16:28.494+0000 7ff3fd914700 1 -- 172.21.15.105:0/1711842 >> 172.21.15.105:0/1711842 conn(0x7ff3f80fe010 msgr2=0x7ff3f8108890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:16:28.486 INFO:teuthology.orchestra.run.smithi105.stderr:2024-05-27T07:16:28.494+0000 7ff3fd914700 1 -- 172.21.15.105:0/1711842 shutdown_connections 2024-05-27T07:16:28.486 INFO:teuthology.orchestra.run.smithi105.stderr:2024-05-27T07:16:28.494+0000 7ff3fd914700 1 -- 172.21.15.105:0/1711842 wait complete. 2024-05-27T07:16:29.122 DEBUG:teuthology.orchestra.run.smithi105:osd.1> sudo journalctl -f -n 0 -u ceph-73df2636-1bf8-11ef-bc9b-c7b262605968@osd.1.service 2024-05-27T07:16:29.125 INFO:tasks.cephadm:Deploying osd.2 on smithi173 with /dev/vg_nvme/lv_4... 2024-05-27T07:16:29.125 DEBUG:teuthology.orchestra.run.smithi173:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:38f9d900c83c63799fbdbe61acc9a11b0d3554a6 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 73df2636-1bf8-11ef-bc9b-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_4 2024-05-27T07:16:29.375 INFO:teuthology.orchestra.run.smithi173.stderr:Inferring config /var/lib/ceph/73df2636-1bf8-11ef-bc9b-c7b262605968/mon.c/config 2024-05-27T07:16:29.640 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:16:29 smithi105 ceph-mon[35977]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' 2024-05-27T07:16:29.640 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:16:29 smithi105 ceph-mon[35977]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' 2024-05-27T07:16:29.640 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:16:29 smithi105 ceph-mon[35977]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-27T07:16:29.640 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:16:29 smithi105 ceph-mon[35977]: from='osd.1 [v2:172.21.15.105:6800/2900093195,v1:172.21.15.105:6801/2900093195]' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-05-27T07:16:29.640 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:16:29 smithi105 ceph-mon[35977]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-05-27T07:16:29.730 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:16:29 smithi052 ceph-mon[31955]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' 2024-05-27T07:16:29.730 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:16:29 smithi052 ceph-mon[31955]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' 2024-05-27T07:16:29.730 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:16:29 smithi052 ceph-mon[31955]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-27T07:16:29.730 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:16:29 smithi052 ceph-mon[31955]: from='osd.1 [v2:172.21.15.105:6800/2900093195,v1:172.21.15.105:6801/2900093195]' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-05-27T07:16:29.730 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:16:29 smithi052 ceph-mon[31955]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-05-27T07:16:29.783 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:16:29 smithi173 ceph-mon[37000]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' 2024-05-27T07:16:29.783 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:16:29 smithi173 ceph-mon[37000]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' 2024-05-27T07:16:29.783 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:16:29 smithi173 ceph-mon[37000]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-27T07:16:29.783 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:16:29 smithi173 ceph-mon[37000]: from='osd.1 [v2:172.21.15.105:6800/2900093195,v1:172.21.15.105:6801/2900093195]' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-05-27T07:16:29.783 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:16:29 smithi173 ceph-mon[37000]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-05-27T07:16:30.690 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:16:30 smithi105 ceph-mon[35977]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2024-05-27T07:16:30.691 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:16:30 smithi105 ceph-mon[35977]: from='osd.1 [v2:172.21.15.105:6800/2900093195,v1:172.21.15.105:6801/2900093195]' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi105", "root=default"]}]: dispatch 2024-05-27T07:16:30.691 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:16:30 smithi105 ceph-mon[35977]: osdmap e11: 2 total, 1 up, 2 in 2024-05-27T07:16:30.691 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:16:30 smithi105 ceph-mon[35977]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-05-27T07:16:30.691 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:16:30 smithi105 ceph-mon[35977]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi105", "root=default"]}]: dispatch 2024-05-27T07:16:30.691 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:16:30 smithi105 ceph-mon[35977]: pgmap v61: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-05-27T07:16:30.730 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:16:30 smithi052 ceph-mon[31955]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2024-05-27T07:16:30.730 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:16:30 smithi052 ceph-mon[31955]: from='osd.1 [v2:172.21.15.105:6800/2900093195,v1:172.21.15.105:6801/2900093195]' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi105", "root=default"]}]: dispatch 2024-05-27T07:16:30.730 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:16:30 smithi052 ceph-mon[31955]: osdmap e11: 2 total, 1 up, 2 in 2024-05-27T07:16:30.730 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:16:30 smithi052 ceph-mon[31955]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-05-27T07:16:30.730 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:16:30 smithi052 ceph-mon[31955]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi105", "root=default"]}]: dispatch 2024-05-27T07:16:30.730 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:16:30 smithi052 ceph-mon[31955]: pgmap v61: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-05-27T07:16:30.798 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:16:30 smithi173 ceph-mon[37000]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2024-05-27T07:16:30.798 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:16:30 smithi173 ceph-mon[37000]: from='osd.1 [v2:172.21.15.105:6800/2900093195,v1:172.21.15.105:6801/2900093195]' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi105", "root=default"]}]: dispatch 2024-05-27T07:16:30.798 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:16:30 smithi173 ceph-mon[37000]: osdmap e11: 2 total, 1 up, 2 in 2024-05-27T07:16:30.798 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:16:30 smithi173 ceph-mon[37000]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-05-27T07:16:30.798 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:16:30 smithi173 ceph-mon[37000]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi105", "root=default"]}]: dispatch 2024-05-27T07:16:30.798 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:16:30 smithi173 ceph-mon[37000]: pgmap v61: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-05-27T07:16:31.191 INFO:journalctl@ceph.osd.1.smithi105.stdout:May 27 07:16:30 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-osd-1[40586]: 2024-05-27T07:16:30.937+0000 7f854b508700 -1 osd.1 0 waiting for initial osdmap 2024-05-27T07:16:31.192 INFO:journalctl@ceph.osd.1.smithi105.stdout:May 27 07:16:30 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-osd-1[40586]: 2024-05-27T07:16:30.941+0000 7f8542ab4700 -1 osd.1 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-05-27T07:16:31.251 INFO:teuthology.orchestra.run.smithi173.stderr:--> Zapping: /dev/vg_nvme/lv_4 2024-05-27T07:16:31.251 INFO:teuthology.orchestra.run.smithi173.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_4 bs=1M count=10 conv=fsync 2024-05-27T07:16:31.251 INFO:teuthology.orchestra.run.smithi173.stderr: stderr: 10+0 records in 2024-05-27T07:16:31.252 INFO:teuthology.orchestra.run.smithi173.stderr:10+0 records out 2024-05-27T07:16:31.252 INFO:teuthology.orchestra.run.smithi173.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0114622 s, 915 MB/s 2024-05-27T07:16:31.252 INFO:teuthology.orchestra.run.smithi173.stderr:--> Zapping successful for: 2024-05-27T07:16:31.546 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:16:31 smithi173 ceph-mon[37000]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi105", "root=default"]}]': finished 2024-05-27T07:16:31.546 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:16:31 smithi173 ceph-mon[37000]: osdmap e12: 2 total, 1 up, 2 in 2024-05-27T07:16:31.546 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:16:31 smithi173 ceph-mon[37000]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-05-27T07:16:31.546 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:16:31 smithi173 ceph-mon[37000]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-05-27T07:16:31.546 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:16:31 smithi173 ceph-mon[37000]: from='osd.1 ' entity='osd.1' 2024-05-27T07:16:31.730 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:16:31 smithi052 ceph-mon[31955]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi105", "root=default"]}]': finished 2024-05-27T07:16:31.730 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:16:31 smithi052 ceph-mon[31955]: osdmap e12: 2 total, 1 up, 2 in 2024-05-27T07:16:31.730 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:16:31 smithi052 ceph-mon[31955]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-05-27T07:16:31.730 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:16:31 smithi052 ceph-mon[31955]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-05-27T07:16:31.730 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:16:31 smithi052 ceph-mon[31955]: from='osd.1 ' entity='osd.1' 2024-05-27T07:16:31.798 DEBUG:teuthology.orchestra.run.smithi173:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:38f9d900c83c63799fbdbe61acc9a11b0d3554a6 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 73df2636-1bf8-11ef-bc9b-c7b262605968 -- ceph orch daemon add osd smithi173:vg_nvme/lv_4 2024-05-27T07:16:31.859 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:16:31 smithi105 ceph-mon[35977]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi105", "root=default"]}]': finished 2024-05-27T07:16:31.859 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:16:31 smithi105 ceph-mon[35977]: osdmap e12: 2 total, 1 up, 2 in 2024-05-27T07:16:31.859 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:16:31 smithi105 ceph-mon[35977]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-05-27T07:16:31.859 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:16:31 smithi105 ceph-mon[35977]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-05-27T07:16:31.859 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:16:31 smithi105 ceph-mon[35977]: from='osd.1 ' entity='osd.1' 2024-05-27T07:16:32.045 INFO:teuthology.orchestra.run.smithi173.stderr:Inferring config /var/lib/ceph/73df2636-1bf8-11ef-bc9b-c7b262605968/mon.c/config 2024-05-27T07:16:32.730 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:16:32 smithi052 ceph-mon[31955]: purged_snaps scrub starts 2024-05-27T07:16:32.730 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:16:32 smithi052 ceph-mon[31955]: purged_snaps scrub ok 2024-05-27T07:16:32.730 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:16:32 smithi052 ceph-mon[31955]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-05-27T07:16:32.730 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:16:32 smithi052 ceph-mon[31955]: pgmap v63: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-05-27T07:16:32.730 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:16:32 smithi052 ceph-mon[31955]: osd.1 [v2:172.21.15.105:6800/2900093195,v1:172.21.15.105:6801/2900093195] boot 2024-05-27T07:16:32.730 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:16:32 smithi052 ceph-mon[31955]: osdmap e13: 2 total, 2 up, 2 in 2024-05-27T07:16:32.730 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:16:32 smithi052 ceph-mon[31955]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-05-27T07:16:32.730 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:16:32 smithi052 ceph-mon[31955]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' 2024-05-27T07:16:32.731 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:16:32 smithi052 ceph-mon[31955]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' 2024-05-27T07:16:32.731 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:16:32 smithi052 ceph-mon[31955]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-05-27T07:16:32.731 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:16:32 smithi052 ceph-mon[31955]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' 2024-05-27T07:16:32.731 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:16:32 smithi052 ceph-mon[31955]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T07:16:32.731 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:16:32 smithi052 ceph-mon[31955]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-27T07:16:32.731 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:16:32 smithi052 ceph-mon[31955]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' 2024-05-27T07:16:32.851 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:16:32 smithi173 ceph-mon[37000]: purged_snaps scrub starts 2024-05-27T07:16:32.851 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:16:32 smithi173 ceph-mon[37000]: purged_snaps scrub ok 2024-05-27T07:16:32.851 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:16:32 smithi173 ceph-mon[37000]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-05-27T07:16:32.851 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:16:32 smithi173 ceph-mon[37000]: pgmap v63: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-05-27T07:16:32.851 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:16:32 smithi173 ceph-mon[37000]: osd.1 [v2:172.21.15.105:6800/2900093195,v1:172.21.15.105:6801/2900093195] boot 2024-05-27T07:16:32.851 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:16:32 smithi173 ceph-mon[37000]: osdmap e13: 2 total, 2 up, 2 in 2024-05-27T07:16:32.851 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:16:32 smithi173 ceph-mon[37000]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-05-27T07:16:32.851 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:16:32 smithi173 ceph-mon[37000]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' 2024-05-27T07:16:32.851 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:16:32 smithi173 ceph-mon[37000]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' 2024-05-27T07:16:32.851 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:16:32 smithi173 ceph-mon[37000]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-05-27T07:16:32.851 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:16:32 smithi173 ceph-mon[37000]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' 2024-05-27T07:16:32.852 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:16:32 smithi173 ceph-mon[37000]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T07:16:32.852 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:16:32 smithi173 ceph-mon[37000]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-27T07:16:32.852 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:16:32 smithi173 ceph-mon[37000]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' 2024-05-27T07:16:32.941 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:16:32 smithi105 ceph-mon[35977]: purged_snaps scrub starts 2024-05-27T07:16:32.941 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:16:32 smithi105 ceph-mon[35977]: purged_snaps scrub ok 2024-05-27T07:16:32.941 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:16:32 smithi105 ceph-mon[35977]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-05-27T07:16:32.941 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:16:32 smithi105 ceph-mon[35977]: pgmap v63: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-05-27T07:16:32.941 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:16:32 smithi105 ceph-mon[35977]: osd.1 [v2:172.21.15.105:6800/2900093195,v1:172.21.15.105:6801/2900093195] boot 2024-05-27T07:16:32.941 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:16:32 smithi105 ceph-mon[35977]: osdmap e13: 2 total, 2 up, 2 in 2024-05-27T07:16:32.941 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:16:32 smithi105 ceph-mon[35977]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-05-27T07:16:32.941 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:16:32 smithi105 ceph-mon[35977]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' 2024-05-27T07:16:32.941 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:16:32 smithi105 ceph-mon[35977]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' 2024-05-27T07:16:32.941 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:16:32 smithi105 ceph-mon[35977]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-05-27T07:16:32.941 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:16:32 smithi105 ceph-mon[35977]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' 2024-05-27T07:16:32.942 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:16:32 smithi105 ceph-mon[35977]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T07:16:32.942 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:16:32 smithi105 ceph-mon[35977]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-27T07:16:32.942 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:16:32 smithi105 ceph-mon[35977]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' 2024-05-27T07:16:33.581 INFO:teuthology.orchestra.run.smithi173.stderr:2024-05-27T07:16:33.587+0000 7f8792f9f700 1 -- 172.21.15.173:0/3926728913 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f878c06cce0 msgr2=0x7f878c106cf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:16:33.581 INFO:teuthology.orchestra.run.smithi173.stderr:2024-05-27T07:16:33.587+0000 7f8792f9f700 1 --2- 172.21.15.173:0/3926728913 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f878c06cce0 0x7f878c106cf0 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f8788009990 tx=0x7f8788009ca0 comp rx=0 tx=0).stop 2024-05-27T07:16:33.581 INFO:teuthology.orchestra.run.smithi173.stderr:2024-05-27T07:16:33.587+0000 7f8792f9f700 1 -- 172.21.15.173:0/3926728913 shutdown_connections 2024-05-27T07:16:33.581 INFO:teuthology.orchestra.run.smithi173.stderr:2024-05-27T07:16:33.587+0000 7f8792f9f700 1 --2- 172.21.15.173:0/3926728913 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f878c109e10 0x7f878c10c320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:16:33.581 INFO:teuthology.orchestra.run.smithi173.stderr:2024-05-27T07:16:33.587+0000 7f8792f9f700 1 --2- 172.21.15.173:0/3926728913 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f878c06cce0 0x7f878c106cf0 unknown :-1 s=CLOSED pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:16:33.581 INFO:teuthology.orchestra.run.smithi173.stderr:2024-05-27T07:16:33.587+0000 7f8792f9f700 1 --2- 172.21.15.173:0/3926728913 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f878c1072c0 0x7f878c109780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:16:33.582 INFO:teuthology.orchestra.run.smithi173.stderr:2024-05-27T07:16:33.587+0000 7f8792f9f700 1 -- 172.21.15.173:0/3926728913 >> 172.21.15.173:0/3926728913 conn(0x7f878c0fe6a0 msgr2=0x7f878c100b50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:16:33.582 INFO:teuthology.orchestra.run.smithi173.stderr:2024-05-27T07:16:33.587+0000 7f8792f9f700 1 -- 172.21.15.173:0/3926728913 shutdown_connections 2024-05-27T07:16:33.582 INFO:teuthology.orchestra.run.smithi173.stderr:2024-05-27T07:16:33.587+0000 7f8792f9f700 1 -- 172.21.15.173:0/3926728913 wait complete. 2024-05-27T07:16:33.582 INFO:teuthology.orchestra.run.smithi173.stderr:2024-05-27T07:16:33.588+0000 7f8792f9f700 1 Processor -- start 2024-05-27T07:16:33.582 INFO:teuthology.orchestra.run.smithi173.stderr:2024-05-27T07:16:33.588+0000 7f8792f9f700 1 -- start start 2024-05-27T07:16:33.582 INFO:teuthology.orchestra.run.smithi173.stderr:2024-05-27T07:16:33.588+0000 7f8792f9f700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f878c06cce0 0x7f878c19a780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:16:33.582 INFO:teuthology.orchestra.run.smithi173.stderr:2024-05-27T07:16:33.588+0000 7f8792f9f700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f878c1072c0 0x7f878c19acc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:16:33.582 INFO:teuthology.orchestra.run.smithi173.stderr:2024-05-27T07:16:33.588+0000 7f8792f9f700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f878c109e10 0x7f878c1a1c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:16:33.583 INFO:teuthology.orchestra.run.smithi173.stderr:2024-05-27T07:16:33.588+0000 7f8792f9f700 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x7f878c10fc70 con 0x7f878c109e10 2024-05-27T07:16:33.583 INFO:teuthology.orchestra.run.smithi173.stderr:2024-05-27T07:16:33.588+0000 7f8792f9f700 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f878c10faf0 con 0x7f878c1072c0 2024-05-27T07:16:33.583 INFO:teuthology.orchestra.run.smithi173.stderr:2024-05-27T07:16:33.588+0000 7f8792f9f700 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f878c10fdf0 con 0x7f878c06cce0 2024-05-27T07:16:33.583 INFO:teuthology.orchestra.run.smithi173.stderr:2024-05-27T07:16:33.588+0000 7f8791f9d700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f878c06cce0 0x7f878c19a780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:16:33.583 INFO:teuthology.orchestra.run.smithi173.stderr:2024-05-27T07:16:33.588+0000 7f8791f9d700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f878c06cce0 0x7f878c19a780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.173:3300/0 says I am v2:172.21.15.173:47012/0 (socket says 172.21.15.173:47012) 2024-05-27T07:16:33.583 INFO:teuthology.orchestra.run.smithi173.stderr:2024-05-27T07:16:33.588+0000 7f8791f9d700 1 -- 172.21.15.173:0/409482919 learned_addr learned my addr 172.21.15.173:0/409482919 (peer_addr_for_me v2:172.21.15.173:0/0) 2024-05-27T07:16:33.583 INFO:teuthology.orchestra.run.smithi173.stderr:2024-05-27T07:16:33.589+0000 7f879179c700 1 --2- 172.21.15.173:0/409482919 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f878c1072c0 0x7f878c19acc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:16:33.583 INFO:teuthology.orchestra.run.smithi173.stderr:2024-05-27T07:16:33.589+0000 7f879279e700 1 --2- 172.21.15.173:0/409482919 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f878c109e10 0x7f878c1a1c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:16:33.583 INFO:teuthology.orchestra.run.smithi173.stderr:2024-05-27T07:16:33.589+0000 7f8791f9d700 1 -- 172.21.15.173:0/409482919 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f878c1072c0 msgr2=0x7f878c19acc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:16:33.583 INFO:teuthology.orchestra.run.smithi173.stderr:2024-05-27T07:16:33.589+0000 7f8791f9d700 1 --2- 172.21.15.173:0/409482919 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f878c1072c0 0x7f878c19acc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:16:33.583 INFO:teuthology.orchestra.run.smithi173.stderr:2024-05-27T07:16:33.589+0000 7f8791f9d700 1 -- 172.21.15.173:0/409482919 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f878c109e10 msgr2=0x7f878c1a1c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:16:33.584 INFO:teuthology.orchestra.run.smithi173.stderr:2024-05-27T07:16:33.589+0000 7f8791f9d700 1 --2- 172.21.15.173:0/409482919 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f878c109e10 0x7f878c1a1c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:16:33.584 INFO:teuthology.orchestra.run.smithi173.stderr:2024-05-27T07:16:33.589+0000 7f8791f9d700 1 -- 172.21.15.173:0/409482919 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f878c1a22e0 con 0x7f878c06cce0 2024-05-27T07:16:33.584 INFO:teuthology.orchestra.run.smithi173.stderr:2024-05-27T07:16:33.589+0000 7f8791f9d700 1 --2- 172.21.15.173:0/409482919 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f878c06cce0 0x7f878c19a780 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7f87880047b0 tx=0x7f8788005e80 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:16:33.584 INFO:teuthology.orchestra.run.smithi173.stderr:2024-05-27T07:16:33.589+0000 7f8782ffd700 1 -- 172.21.15.173:0/409482919 <== mon.1 v2:172.21.15.173:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8788027070 con 0x7f878c06cce0 2024-05-27T07:16:33.584 INFO:teuthology.orchestra.run.smithi173.stderr:2024-05-27T07:16:33.589+0000 7f8782ffd700 1 -- 172.21.15.173:0/409482919 <== mon.1 v2:172.21.15.173:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f87880041c0 con 0x7f878c06cce0 2024-05-27T07:16:33.584 INFO:teuthology.orchestra.run.smithi173.stderr:2024-05-27T07:16:33.589+0000 7f8792f9f700 1 -- 172.21.15.173:0/409482919 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f878c1a2570 con 0x7f878c06cce0 2024-05-27T07:16:33.584 INFO:teuthology.orchestra.run.smithi173.stderr:2024-05-27T07:16:33.589+0000 7f8782ffd700 1 -- 172.21.15.173:0/409482919 <== mon.1 v2:172.21.15.173:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f878800f460 con 0x7f878c06cce0 2024-05-27T07:16:33.584 INFO:teuthology.orchestra.run.smithi173.stderr:2024-05-27T07:16:33.589+0000 7f8792f9f700 1 -- 172.21.15.173:0/409482919 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f878c1a2aa0 con 0x7f878c06cce0 2024-05-27T07:16:33.585 INFO:teuthology.orchestra.run.smithi173.stderr:2024-05-27T07:16:33.590+0000 7f8792f9f700 1 -- 172.21.15.173:0/409482919 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f878c0512b0 con 0x7f878c06cce0 2024-05-27T07:16:33.585 INFO:teuthology.orchestra.run.smithi173.stderr:2024-05-27T07:16:33.592+0000 7f8782ffd700 1 -- 172.21.15.173:0/409482919 <== mon.1 v2:172.21.15.173:3300/0 4 ==== mgrmap(e 15) ==== 149596+0+0 (secure 0 0 0) 0x7f878802a070 con 0x7f878c06cce0 2024-05-27T07:16:33.585 INFO:teuthology.orchestra.run.smithi173.stderr:2024-05-27T07:16:33.592+0000 7f8782ffd700 1 --2- 172.21.15.173:0/409482919 >> [v2:172.21.15.52:6800/3513075805,v1:172.21.15.52:6801/3513075805] conn(0x7f87780b2870 0x7f87780b4d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:16:33.585 INFO:teuthology.orchestra.run.smithi173.stderr:2024-05-27T07:16:33.592+0000 7f8782ffd700 1 -- 172.21.15.173:0/409482919 <== mon.1 v2:172.21.15.173:3300/0 5 ==== osd_map(14..14 src has 1..14) ==== 2309+0+0 (secure 0 0 0) 0x7f87880e1bf0 con 0x7f878c06cce0 2024-05-27T07:16:33.585 INFO:teuthology.orchestra.run.smithi173.stderr:2024-05-27T07:16:33.593+0000 7f879179c700 1 --2- 172.21.15.173:0/409482919 >> [v2:172.21.15.52:6800/3513075805,v1:172.21.15.52:6801/3513075805] conn(0x7f87780b2870 0x7f87780b4d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:16:33.585 INFO:teuthology.orchestra.run.smithi173.stderr:2024-05-27T07:16:33.593+0000 7f879179c700 1 --2- 172.21.15.173:0/409482919 >> [v2:172.21.15.52:6800/3513075805,v1:172.21.15.52:6801/3513075805] conn(0x7f87780b2870 0x7f87780b4d80 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f877c005950 tx=0x7f877c00a300 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:16:33.588 INFO:teuthology.orchestra.run.smithi173.stderr:2024-05-27T07:16:33.597+0000 7f8782ffd700 1 -- 172.21.15.173:0/409482919 <== mon.1 v2:172.21.15.173:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191963 (secure 0 0 0) 0x7f8788013070 con 0x7f878c06cce0 2024-05-27T07:16:33.730 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:16:33 smithi052 ceph-mon[31955]: Detected new or changed devices on smithi105 2024-05-27T07:16:33.730 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:16:33 smithi052 ceph-mon[31955]: Adjusting osd_memory_target on smithi105 to 17088M 2024-05-27T07:16:33.801 INFO:teuthology.orchestra.run.smithi173.stderr:2024-05-27T07:16:33.810+0000 7f8792f9f700 1 -- 172.21.15.173:0/409482919 --> [v2:172.21.15.52:6800/3513075805,v1:172.21.15.52:6801/3513075805] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi173:vg_nvme/lv_4", "target": ["mon-mgr", ""]}) -- 0x7f878c063bf0 con 0x7f87780b2870 2024-05-27T07:16:33.801 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:16:33 smithi173 ceph-mon[37000]: Detected new or changed devices on smithi105 2024-05-27T07:16:33.801 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:16:33 smithi173 ceph-mon[37000]: Adjusting osd_memory_target on smithi105 to 17088M 2024-05-27T07:16:33.940 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:16:33 smithi105 ceph-mon[35977]: Detected new or changed devices on smithi105 2024-05-27T07:16:33.941 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:16:33 smithi105 ceph-mon[35977]: Adjusting osd_memory_target on smithi105 to 17088M 2024-05-27T07:16:34.730 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:16:34 smithi052 ceph-mon[31955]: osdmap e14: 2 total, 2 up, 2 in 2024-05-27T07:16:34.730 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:16:34 smithi052 ceph-mon[31955]: pgmap v66: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-05-27T07:16:34.730 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:16:34 smithi052 ceph-mon[31955]: from='client.24166 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi173:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-05-27T07:16:34.730 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:16:34 smithi052 ceph-mon[31955]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-05-27T07:16:34.730 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:16:34 smithi052 ceph-mon[31955]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-05-27T07:16:34.730 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:16:34 smithi052 ceph-mon[31955]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T07:16:34.749 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:16:34 smithi173 ceph-mon[37000]: osdmap e14: 2 total, 2 up, 2 in 2024-05-27T07:16:34.750 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:16:34 smithi173 ceph-mon[37000]: pgmap v66: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-05-27T07:16:34.750 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:16:34 smithi173 ceph-mon[37000]: from='client.24166 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi173:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-05-27T07:16:34.750 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:16:34 smithi173 ceph-mon[37000]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-05-27T07:16:34.750 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:16:34 smithi173 ceph-mon[37000]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-05-27T07:16:34.750 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:16:34 smithi173 ceph-mon[37000]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T07:16:34.940 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:16:34 smithi105 ceph-mon[35977]: osdmap e14: 2 total, 2 up, 2 in 2024-05-27T07:16:34.940 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:16:34 smithi105 ceph-mon[35977]: pgmap v66: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-05-27T07:16:34.941 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:16:34 smithi105 ceph-mon[35977]: from='client.24166 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi173:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-05-27T07:16:34.941 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:16:34 smithi105 ceph-mon[35977]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-05-27T07:16:34.941 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:16:34 smithi105 ceph-mon[35977]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-05-27T07:16:34.941 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:16:34 smithi105 ceph-mon[35977]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T07:16:36.940 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:16:36 smithi105 ceph-mon[35977]: pgmap v67: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-05-27T07:16:36.941 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:16:36 smithi105 ceph-mon[35977]: from='client.? 172.21.15.173:0/2635935295' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "41a9df5b-882a-47da-bb26-7f4d26ccf6c1"}]: dispatch 2024-05-27T07:16:36.941 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:16:36 smithi105 ceph-mon[35977]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "41a9df5b-882a-47da-bb26-7f4d26ccf6c1"}]: dispatch 2024-05-27T07:16:36.941 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:16:36 smithi105 ceph-mon[35977]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "41a9df5b-882a-47da-bb26-7f4d26ccf6c1"}]': finished 2024-05-27T07:16:36.941 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:16:36 smithi105 ceph-mon[35977]: osdmap e15: 3 total, 2 up, 3 in 2024-05-27T07:16:36.941 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:16:36 smithi105 ceph-mon[35977]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-05-27T07:16:36.980 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:16:36 smithi052 ceph-mon[31955]: pgmap v67: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-05-27T07:16:36.980 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:16:36 smithi052 ceph-mon[31955]: from='client.? 172.21.15.173:0/2635935295' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "41a9df5b-882a-47da-bb26-7f4d26ccf6c1"}]: dispatch 2024-05-27T07:16:36.980 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:16:36 smithi052 ceph-mon[31955]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "41a9df5b-882a-47da-bb26-7f4d26ccf6c1"}]: dispatch 2024-05-27T07:16:36.980 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:16:36 smithi052 ceph-mon[31955]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "41a9df5b-882a-47da-bb26-7f4d26ccf6c1"}]': finished 2024-05-27T07:16:36.980 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:16:36 smithi052 ceph-mon[31955]: osdmap e15: 3 total, 2 up, 3 in 2024-05-27T07:16:36.980 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:16:36 smithi052 ceph-mon[31955]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-05-27T07:16:37.102 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:16:36 smithi173 ceph-mon[37000]: pgmap v67: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-05-27T07:16:37.103 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:16:36 smithi173 ceph-mon[37000]: from='client.? 172.21.15.173:0/2635935295' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "41a9df5b-882a-47da-bb26-7f4d26ccf6c1"}]: dispatch 2024-05-27T07:16:37.103 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:16:36 smithi173 ceph-mon[37000]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "41a9df5b-882a-47da-bb26-7f4d26ccf6c1"}]: dispatch 2024-05-27T07:16:37.103 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:16:36 smithi173 ceph-mon[37000]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "41a9df5b-882a-47da-bb26-7f4d26ccf6c1"}]': finished 2024-05-27T07:16:37.103 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:16:36 smithi173 ceph-mon[37000]: osdmap e15: 3 total, 2 up, 3 in 2024-05-27T07:16:37.103 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:16:36 smithi173 ceph-mon[37000]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-05-27T07:16:37.940 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:16:37 smithi105 ceph-mon[35977]: from='client.? 172.21.15.173:0/3234847600' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-05-27T07:16:37.980 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:16:37 smithi052 ceph-mon[31955]: from='client.? 172.21.15.173:0/3234847600' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-05-27T07:16:38.102 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:16:37 smithi173 ceph-mon[37000]: from='client.? 172.21.15.173:0/3234847600' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-05-27T07:16:38.940 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:16:38 smithi105 ceph-mon[35977]: pgmap v69: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-05-27T07:16:38.980 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:16:38 smithi052 ceph-mon[31955]: pgmap v69: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-05-27T07:16:39.004 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:16:38 smithi173 ceph-mon[37000]: pgmap v69: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-05-27T07:16:40.728 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:16:40 smithi173 ceph-mon[37000]: pgmap v70: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-05-27T07:16:40.940 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:16:40 smithi105 ceph-mon[35977]: pgmap v70: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-05-27T07:16:40.980 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:16:40 smithi052 ceph-mon[31955]: pgmap v70: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-05-27T07:16:42.851 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:16:42 smithi173 ceph-mon[37000]: pgmap v71: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-05-27T07:16:42.940 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:16:42 smithi105 ceph-mon[35977]: pgmap v71: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-05-27T07:16:42.980 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:16:42 smithi052 ceph-mon[31955]: pgmap v71: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-05-27T07:16:43.852 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:16:43 smithi173 ceph-mon[37000]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2024-05-27T07:16:43.852 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:16:43 smithi173 ceph-mon[37000]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T07:16:43.939 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:16:43 smithi105 ceph-mon[35977]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2024-05-27T07:16:43.939 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:16:43 smithi105 ceph-mon[35977]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T07:16:43.980 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:16:43 smithi052 ceph-mon[31955]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2024-05-27T07:16:43.980 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:16:43 smithi052 ceph-mon[31955]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T07:16:44.852 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:16:44 smithi173 ceph-mon[37000]: Deploying daemon osd.2 on smithi173 2024-05-27T07:16:44.853 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:16:44 smithi173 ceph-mon[37000]: pgmap v72: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-05-27T07:16:44.940 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:16:44 smithi105 ceph-mon[35977]: Deploying daemon osd.2 on smithi173 2024-05-27T07:16:44.940 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:16:44 smithi105 ceph-mon[35977]: pgmap v72: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-05-27T07:16:44.980 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:16:44 smithi052 ceph-mon[31955]: Deploying daemon osd.2 on smithi173 2024-05-27T07:16:44.980 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:16:44 smithi052 ceph-mon[31955]: pgmap v72: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-05-27T07:16:46.853 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:16:46 smithi173 ceph-mon[37000]: pgmap v73: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-05-27T07:16:46.940 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:16:46 smithi105 ceph-mon[35977]: pgmap v73: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-05-27T07:16:46.980 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:16:46 smithi052 ceph-mon[31955]: pgmap v73: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-05-27T07:16:48.900 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:16:48 smithi173 ceph-mon[37000]: pgmap v74: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-05-27T07:16:48.940 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:16:48 smithi105 ceph-mon[35977]: pgmap v74: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-05-27T07:16:48.980 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:16:48 smithi052 ceph-mon[31955]: pgmap v74: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-05-27T07:16:50.102 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:16:49 smithi173 ceph-mon[37000]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' 2024-05-27T07:16:50.103 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:16:49 smithi173 ceph-mon[37000]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' 2024-05-27T07:16:50.103 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:16:49 smithi173 ceph-mon[37000]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-27T07:16:50.230 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:16:49 smithi052 ceph-mon[31955]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' 2024-05-27T07:16:50.230 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:16:49 smithi052 ceph-mon[31955]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' 2024-05-27T07:16:50.230 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:16:49 smithi052 ceph-mon[31955]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-27T07:16:50.440 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:16:49 smithi105 ceph-mon[35977]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' 2024-05-27T07:16:50.440 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:16:49 smithi105 ceph-mon[35977]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' 2024-05-27T07:16:50.441 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:16:49 smithi105 ceph-mon[35977]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-27T07:16:51.136 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:16:50 smithi173 ceph-mon[37000]: pgmap v75: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-05-27T07:16:51.136 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:16:50 smithi173 ceph-mon[37000]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' 2024-05-27T07:16:51.136 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:16:50 smithi173 ceph-mon[37000]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' 2024-05-27T07:16:51.136 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:16:50 smithi173 ceph-mon[37000]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T07:16:51.136 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:16:50 smithi173 ceph-mon[37000]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-27T07:16:51.136 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:16:50 smithi173 ceph-mon[37000]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' 2024-05-27T07:16:51.230 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:16:50 smithi052 ceph-mon[31955]: pgmap v75: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-05-27T07:16:51.230 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:16:50 smithi052 ceph-mon[31955]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' 2024-05-27T07:16:51.230 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:16:50 smithi052 ceph-mon[31955]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' 2024-05-27T07:16:51.230 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:16:50 smithi052 ceph-mon[31955]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T07:16:51.230 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:16:50 smithi052 ceph-mon[31955]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-27T07:16:51.230 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:16:50 smithi052 ceph-mon[31955]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' 2024-05-27T07:16:51.440 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:16:50 smithi105 ceph-mon[35977]: pgmap v75: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-05-27T07:16:51.440 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:16:50 smithi105 ceph-mon[35977]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' 2024-05-27T07:16:51.440 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:16:50 smithi105 ceph-mon[35977]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' 2024-05-27T07:16:51.441 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:16:50 smithi105 ceph-mon[35977]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T07:16:51.441 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:16:50 smithi105 ceph-mon[35977]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-27T07:16:51.441 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:16:50 smithi105 ceph-mon[35977]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' 2024-05-27T07:16:52.674 INFO:teuthology.orchestra.run.smithi173.stderr:2024-05-27T07:16:52.683+0000 7f8782ffd700 1 -- 172.21.15.173:0/409482919 <== mgr.14152 v2:172.21.15.52:6800/3513075805 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+37 (secure 0 0 0) 0x7f878c063bf0 con 0x7f87780b2870 2024-05-27T07:16:52.675 INFO:teuthology.orchestra.run.smithi173.stdout:Created osd(s) 2 on host 'smithi173' 2024-05-27T07:16:52.677 INFO:teuthology.orchestra.run.smithi173.stderr:2024-05-27T07:16:52.686+0000 7f8792f9f700 1 -- 172.21.15.173:0/409482919 >> [v2:172.21.15.52:6800/3513075805,v1:172.21.15.52:6801/3513075805] conn(0x7f87780b2870 msgr2=0x7f87780b4d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:16:52.677 INFO:teuthology.orchestra.run.smithi173.stderr:2024-05-27T07:16:52.686+0000 7f8792f9f700 1 --2- 172.21.15.173:0/409482919 >> [v2:172.21.15.52:6800/3513075805,v1:172.21.15.52:6801/3513075805] conn(0x7f87780b2870 0x7f87780b4d80 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f877c005950 tx=0x7f877c00a300 comp rx=0 tx=0).stop 2024-05-27T07:16:52.677 INFO:teuthology.orchestra.run.smithi173.stderr:2024-05-27T07:16:52.686+0000 7f8792f9f700 1 -- 172.21.15.173:0/409482919 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f878c06cce0 msgr2=0x7f878c19a780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:16:52.677 INFO:teuthology.orchestra.run.smithi173.stderr:2024-05-27T07:16:52.686+0000 7f8792f9f700 1 --2- 172.21.15.173:0/409482919 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f878c06cce0 0x7f878c19a780 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7f87880047b0 tx=0x7f8788005e80 comp rx=0 tx=0).stop 2024-05-27T07:16:52.677 INFO:teuthology.orchestra.run.smithi173.stderr:2024-05-27T07:16:52.686+0000 7f8792f9f700 1 -- 172.21.15.173:0/409482919 shutdown_connections 2024-05-27T07:16:52.677 INFO:teuthology.orchestra.run.smithi173.stderr:2024-05-27T07:16:52.686+0000 7f8792f9f700 1 --2- 172.21.15.173:0/409482919 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f878c109e10 0x7f878c1a1c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:16:52.677 INFO:teuthology.orchestra.run.smithi173.stderr:2024-05-27T07:16:52.686+0000 7f8792f9f700 1 --2- 172.21.15.173:0/409482919 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f878c06cce0 0x7f878c19a780 unknown :-1 s=CLOSED pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:16:52.677 INFO:teuthology.orchestra.run.smithi173.stderr:2024-05-27T07:16:52.686+0000 7f8792f9f700 1 --2- 172.21.15.173:0/409482919 >> [v2:172.21.15.52:6800/3513075805,v1:172.21.15.52:6801/3513075805] conn(0x7f87780b2870 0x7f87780b4d80 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:16:52.677 INFO:teuthology.orchestra.run.smithi173.stderr:2024-05-27T07:16:52.686+0000 7f8792f9f700 1 --2- 172.21.15.173:0/409482919 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f878c1072c0 0x7f878c19acc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:16:52.677 INFO:teuthology.orchestra.run.smithi173.stderr:2024-05-27T07:16:52.686+0000 7f8792f9f700 1 -- 172.21.15.173:0/409482919 >> 172.21.15.173:0/409482919 conn(0x7f878c0fe6a0 msgr2=0x7f878c1053a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:16:52.677 INFO:teuthology.orchestra.run.smithi173.stderr:2024-05-27T07:16:52.686+0000 7f8792f9f700 1 -- 172.21.15.173:0/409482919 shutdown_connections 2024-05-27T07:16:52.678 INFO:teuthology.orchestra.run.smithi173.stderr:2024-05-27T07:16:52.686+0000 7f8792f9f700 1 -- 172.21.15.173:0/409482919 wait complete. 2024-05-27T07:16:53.230 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:16:52 smithi052 ceph-mon[31955]: pgmap v76: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-05-27T07:16:53.230 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:16:52 smithi052 ceph-mon[31955]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' 2024-05-27T07:16:53.230 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:16:52 smithi052 ceph-mon[31955]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' 2024-05-27T07:16:53.230 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:16:52 smithi052 ceph-mon[31955]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-27T07:16:53.230 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:16:52 smithi052 ceph-mon[31955]: from='osd.2 [v2:172.21.15.173:6800/2251189314,v1:172.21.15.173:6801/2251189314]' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-05-27T07:16:53.230 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:16:52 smithi052 ceph-mon[31955]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-05-27T07:16:53.353 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:16:52 smithi173 ceph-mon[37000]: pgmap v76: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-05-27T07:16:53.353 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:16:52 smithi173 ceph-mon[37000]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' 2024-05-27T07:16:53.353 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:16:52 smithi173 ceph-mon[37000]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' 2024-05-27T07:16:53.353 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:16:52 smithi173 ceph-mon[37000]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-27T07:16:53.353 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:16:52 smithi173 ceph-mon[37000]: from='osd.2 [v2:172.21.15.173:6800/2251189314,v1:172.21.15.173:6801/2251189314]' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-05-27T07:16:53.353 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:16:52 smithi173 ceph-mon[37000]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-05-27T07:16:53.400 DEBUG:teuthology.orchestra.run.smithi173:osd.2> sudo journalctl -f -n 0 -u ceph-73df2636-1bf8-11ef-bc9b-c7b262605968@osd.2.service 2024-05-27T07:16:53.402 INFO:tasks.cephadm:Deploying osd.3 on smithi052 with /dev/vg_nvme/lv_3... 2024-05-27T07:16:53.402 DEBUG:teuthology.orchestra.run.smithi052:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:38f9d900c83c63799fbdbe61acc9a11b0d3554a6 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 73df2636-1bf8-11ef-bc9b-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_3 2024-05-27T07:16:53.440 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:16:52 smithi105 ceph-mon[35977]: pgmap v76: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-05-27T07:16:53.440 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:16:52 smithi105 ceph-mon[35977]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' 2024-05-27T07:16:53.441 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:16:52 smithi105 ceph-mon[35977]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' 2024-05-27T07:16:53.441 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:16:52 smithi105 ceph-mon[35977]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-27T07:16:53.441 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:16:52 smithi105 ceph-mon[35977]: from='osd.2 [v2:172.21.15.173:6800/2251189314,v1:172.21.15.173:6801/2251189314]' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-05-27T07:16:53.441 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:16:52 smithi105 ceph-mon[35977]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-05-27T07:16:53.658 INFO:teuthology.orchestra.run.smithi052.stderr:Inferring config /var/lib/ceph/73df2636-1bf8-11ef-bc9b-c7b262605968/mon.a/config 2024-05-27T07:16:54.746 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:16:54 smithi173 ceph-mon[37000]: pgmap v77: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-05-27T07:16:54.746 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:16:54 smithi173 ceph-mon[37000]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2024-05-27T07:16:54.746 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:16:54 smithi173 ceph-mon[37000]: osdmap e16: 3 total, 2 up, 3 in 2024-05-27T07:16:54.746 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:16:54 smithi173 ceph-mon[37000]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-05-27T07:16:54.747 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:16:54 smithi173 ceph-mon[37000]: from='osd.2 [v2:172.21.15.173:6800/2251189314,v1:172.21.15.173:6801/2251189314]' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi173", "root=default"]}]: dispatch 2024-05-27T07:16:54.747 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:16:54 smithi173 ceph-mon[37000]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi173", "root=default"]}]: dispatch 2024-05-27T07:16:54.913 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:16:54 smithi052 ceph-mon[31955]: pgmap v77: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-05-27T07:16:54.913 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:16:54 smithi052 ceph-mon[31955]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2024-05-27T07:16:54.913 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:16:54 smithi052 ceph-mon[31955]: osdmap e16: 3 total, 2 up, 3 in 2024-05-27T07:16:54.913 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:16:54 smithi052 ceph-mon[31955]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-05-27T07:16:54.913 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:16:54 smithi052 ceph-mon[31955]: from='osd.2 [v2:172.21.15.173:6800/2251189314,v1:172.21.15.173:6801/2251189314]' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi173", "root=default"]}]: dispatch 2024-05-27T07:16:54.913 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:16:54 smithi052 ceph-mon[31955]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi173", "root=default"]}]: dispatch 2024-05-27T07:16:54.940 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:16:54 smithi105 ceph-mon[35977]: pgmap v77: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-05-27T07:16:54.940 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:16:54 smithi105 ceph-mon[35977]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2024-05-27T07:16:54.940 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:16:54 smithi105 ceph-mon[35977]: osdmap e16: 3 total, 2 up, 3 in 2024-05-27T07:16:54.940 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:16:54 smithi105 ceph-mon[35977]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-05-27T07:16:54.941 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:16:54 smithi105 ceph-mon[35977]: from='osd.2 [v2:172.21.15.173:6800/2251189314,v1:172.21.15.173:6801/2251189314]' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi173", "root=default"]}]: dispatch 2024-05-27T07:16:54.941 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:16:54 smithi105 ceph-mon[35977]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi173", "root=default"]}]: dispatch 2024-05-27T07:16:55.353 INFO:journalctl@ceph.osd.2.smithi173.stdout:May 27 07:16:55 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-osd-2[42253]: 2024-05-27T07:16:55.085+0000 7fe7d48fa700 -1 osd.2 0 waiting for initial osdmap 2024-05-27T07:16:55.353 INFO:journalctl@ceph.osd.2.smithi173.stdout:May 27 07:16:55 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-osd-2[42253]: 2024-05-27T07:16:55.090+0000 7fe7cbea6700 -1 osd.2 17 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-05-27T07:16:55.706 INFO:teuthology.orchestra.run.smithi052.stderr:--> Zapping: /dev/vg_nvme/lv_3 2024-05-27T07:16:55.706 INFO:teuthology.orchestra.run.smithi052.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_3 bs=1M count=10 conv=fsync 2024-05-27T07:16:55.706 INFO:teuthology.orchestra.run.smithi052.stderr: stderr: 10+0 records in 2024-05-27T07:16:55.706 INFO:teuthology.orchestra.run.smithi052.stderr:10+0 records out 2024-05-27T07:16:55.706 INFO:teuthology.orchestra.run.smithi052.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.00829989 s, 1.3 GB/s 2024-05-27T07:16:55.706 INFO:teuthology.orchestra.run.smithi052.stderr:--> Zapping successful for: 2024-05-27T07:16:55.940 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:16:55 smithi105 ceph-mon[35977]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi173", "root=default"]}]': finished 2024-05-27T07:16:55.940 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:16:55 smithi105 ceph-mon[35977]: osdmap e17: 3 total, 2 up, 3 in 2024-05-27T07:16:55.940 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:16:55 smithi105 ceph-mon[35977]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-05-27T07:16:55.941 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:16:55 smithi105 ceph-mon[35977]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-05-27T07:16:55.941 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:16:55 smithi105 ceph-mon[35977]: from='osd.2 ' entity='osd.2' 2024-05-27T07:16:55.980 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:16:55 smithi052 ceph-mon[31955]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi173", "root=default"]}]': finished 2024-05-27T07:16:55.980 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:16:55 smithi052 ceph-mon[31955]: osdmap e17: 3 total, 2 up, 3 in 2024-05-27T07:16:55.980 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:16:55 smithi052 ceph-mon[31955]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-05-27T07:16:55.980 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:16:55 smithi052 ceph-mon[31955]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-05-27T07:16:55.980 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:16:55 smithi052 ceph-mon[31955]: from='osd.2 ' entity='osd.2' 2024-05-27T07:16:56.080 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:16:55 smithi173 ceph-mon[37000]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi173", "root=default"]}]': finished 2024-05-27T07:16:56.080 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:16:55 smithi173 ceph-mon[37000]: osdmap e17: 3 total, 2 up, 3 in 2024-05-27T07:16:56.080 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:16:55 smithi173 ceph-mon[37000]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-05-27T07:16:56.080 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:16:55 smithi173 ceph-mon[37000]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-05-27T07:16:56.080 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:16:55 smithi173 ceph-mon[37000]: from='osd.2 ' entity='osd.2' 2024-05-27T07:16:56.311 DEBUG:teuthology.orchestra.run.smithi052:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:38f9d900c83c63799fbdbe61acc9a11b0d3554a6 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 73df2636-1bf8-11ef-bc9b-c7b262605968 -- ceph orch daemon add osd smithi052:vg_nvme/lv_3 2024-05-27T07:16:56.561 INFO:teuthology.orchestra.run.smithi052.stderr:Inferring config /var/lib/ceph/73df2636-1bf8-11ef-bc9b-c7b262605968/mon.a/config 2024-05-27T07:16:56.853 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:16:56 smithi173 ceph-mon[37000]: purged_snaps scrub starts 2024-05-27T07:16:56.853 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:16:56 smithi173 ceph-mon[37000]: purged_snaps scrub ok 2024-05-27T07:16:56.853 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:16:56 smithi173 ceph-mon[37000]: pgmap v80: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-05-27T07:16:56.853 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:16:56 smithi173 ceph-mon[37000]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-05-27T07:16:56.853 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:16:56 smithi173 ceph-mon[37000]: osd.2 [v2:172.21.15.173:6800/2251189314,v1:172.21.15.173:6801/2251189314] boot 2024-05-27T07:16:56.853 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:16:56 smithi173 ceph-mon[37000]: osdmap e18: 3 total, 3 up, 3 in 2024-05-27T07:16:56.853 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:16:56 smithi173 ceph-mon[37000]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-05-27T07:16:56.853 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:16:56 smithi173 ceph-mon[37000]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' 2024-05-27T07:16:56.853 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:16:56 smithi173 ceph-mon[37000]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' 2024-05-27T07:16:56.853 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:16:56 smithi173 ceph-mon[37000]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-05-27T07:16:56.853 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:16:56 smithi173 ceph-mon[37000]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' 2024-05-27T07:16:56.854 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:16:56 smithi173 ceph-mon[37000]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T07:16:56.854 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:16:56 smithi173 ceph-mon[37000]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-27T07:16:56.854 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:16:56 smithi173 ceph-mon[37000]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' 2024-05-27T07:16:56.912 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:16:56 smithi052 ceph-mon[31955]: purged_snaps scrub starts 2024-05-27T07:16:56.912 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:16:56 smithi052 ceph-mon[31955]: purged_snaps scrub ok 2024-05-27T07:16:56.912 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:16:56 smithi052 ceph-mon[31955]: pgmap v80: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-05-27T07:16:56.912 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:16:56 smithi052 ceph-mon[31955]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-05-27T07:16:56.912 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:16:56 smithi052 ceph-mon[31955]: osd.2 [v2:172.21.15.173:6800/2251189314,v1:172.21.15.173:6801/2251189314] boot 2024-05-27T07:16:56.912 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:16:56 smithi052 ceph-mon[31955]: osdmap e18: 3 total, 3 up, 3 in 2024-05-27T07:16:56.912 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:16:56 smithi052 ceph-mon[31955]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-05-27T07:16:56.912 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:16:56 smithi052 ceph-mon[31955]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' 2024-05-27T07:16:56.913 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:16:56 smithi052 ceph-mon[31955]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' 2024-05-27T07:16:56.913 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:16:56 smithi052 ceph-mon[31955]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-05-27T07:16:56.913 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:16:56 smithi052 ceph-mon[31955]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' 2024-05-27T07:16:56.913 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:16:56 smithi052 ceph-mon[31955]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T07:16:56.913 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:16:56 smithi052 ceph-mon[31955]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-27T07:16:56.913 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:16:56 smithi052 ceph-mon[31955]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' 2024-05-27T07:16:56.940 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:16:56 smithi105 ceph-mon[35977]: purged_snaps scrub starts 2024-05-27T07:16:56.941 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:16:56 smithi105 ceph-mon[35977]: purged_snaps scrub ok 2024-05-27T07:16:56.941 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:16:56 smithi105 ceph-mon[35977]: pgmap v80: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-05-27T07:16:56.941 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:16:56 smithi105 ceph-mon[35977]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-05-27T07:16:56.941 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:16:56 smithi105 ceph-mon[35977]: osd.2 [v2:172.21.15.173:6800/2251189314,v1:172.21.15.173:6801/2251189314] boot 2024-05-27T07:16:56.941 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:16:56 smithi105 ceph-mon[35977]: osdmap e18: 3 total, 3 up, 3 in 2024-05-27T07:16:56.941 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:16:56 smithi105 ceph-mon[35977]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-05-27T07:16:56.941 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:16:56 smithi105 ceph-mon[35977]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' 2024-05-27T07:16:56.941 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:16:56 smithi105 ceph-mon[35977]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' 2024-05-27T07:16:56.941 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:16:56 smithi105 ceph-mon[35977]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-05-27T07:16:56.941 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:16:56 smithi105 ceph-mon[35977]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' 2024-05-27T07:16:56.941 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:16:56 smithi105 ceph-mon[35977]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T07:16:56.941 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:16:56 smithi105 ceph-mon[35977]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-27T07:16:56.941 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:16:56 smithi105 ceph-mon[35977]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' 2024-05-27T07:16:57.917 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:16:57 smithi052 ceph-mon[31955]: Detected new or changed devices on smithi173 2024-05-27T07:16:57.917 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:16:57 smithi052 ceph-mon[31955]: Adjusting osd_memory_target on smithi173 to 17088M 2024-05-27T07:16:57.917 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:16:57 smithi052 ceph-mon[31955]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' cmd=[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]: dispatch 2024-05-27T07:16:57.940 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:16:57 smithi105 ceph-mon[35977]: Detected new or changed devices on smithi173 2024-05-27T07:16:57.940 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:16:57 smithi105 ceph-mon[35977]: Adjusting osd_memory_target on smithi173 to 17088M 2024-05-27T07:16:57.941 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:16:57 smithi105 ceph-mon[35977]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' cmd=[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]: dispatch 2024-05-27T07:16:58.103 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:16:57 smithi173 ceph-mon[37000]: Detected new or changed devices on smithi173 2024-05-27T07:16:58.103 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:16:57 smithi173 ceph-mon[37000]: Adjusting osd_memory_target on smithi173 to 17088M 2024-05-27T07:16:58.103 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:16:57 smithi173 ceph-mon[37000]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' cmd=[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]: dispatch 2024-05-27T07:16:58.159 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:16:58.168+0000 7ffa519c7700 1 -- 172.21.15.52:0/1967923005 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ffa4c105e00 msgr2=0x7ffa4c10c6b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:16:58.159 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:16:58.168+0000 7ffa519c7700 1 --2- 172.21.15.52:0/1967923005 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ffa4c105e00 0x7ffa4c10c6b0 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7ffa44009990 tx=0x7ffa44009ca0 comp rx=0 tx=0).stop 2024-05-27T07:16:58.159 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:16:58.168+0000 7ffa519c7700 1 -- 172.21.15.52:0/1967923005 shutdown_connections 2024-05-27T07:16:58.160 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:16:58.168+0000 7ffa519c7700 1 --2- 172.21.15.52:0/1967923005 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ffa4c105e00 0x7ffa4c10c6b0 unknown :-1 s=CLOSED pgs=118 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:16:58.160 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:16:58.168+0000 7ffa519c7700 1 --2- 172.21.15.52:0/1967923005 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7ffa4c104330 0x7ffa4c104740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:16:58.160 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:16:58.168+0000 7ffa519c7700 1 --2- 172.21.15.52:0/1967923005 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7ffa4c105460 0x7ffa4c105870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:16:58.160 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:16:58.168+0000 7ffa519c7700 1 -- 172.21.15.52:0/1967923005 >> 172.21.15.52:0/1967923005 conn(0x7ffa4c0ff9b0 msgr2=0x7ffa4c101e20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:16:58.160 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:16:58.168+0000 7ffa519c7700 1 -- 172.21.15.52:0/1967923005 shutdown_connections 2024-05-27T07:16:58.160 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:16:58.168+0000 7ffa519c7700 1 -- 172.21.15.52:0/1967923005 wait complete. 2024-05-27T07:16:58.160 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:16:58.169+0000 7ffa519c7700 1 Processor -- start 2024-05-27T07:16:58.162 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:16:58.169+0000 7ffa519c7700 1 -- start start 2024-05-27T07:16:58.162 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:16:58.170+0000 7ffa519c7700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ffa4c104330 0x7ffa4c19a900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:16:58.162 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:16:58.170+0000 7ffa519c7700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7ffa4c105460 0x7ffa4c19ae40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:16:58.162 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:16:58.170+0000 7ffa519c7700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7ffa4c105e00 0x7ffa4c1a1de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:16:58.162 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:16:58.170+0000 7ffa519c7700 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x7ffa4c10fe10 con 0x7ffa4c104330 2024-05-27T07:16:58.162 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:16:58.170+0000 7ffa519c7700 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7ffa4c10fc90 con 0x7ffa4c105e00 2024-05-27T07:16:58.162 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:16:58.170+0000 7ffa519c7700 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7ffa4c10ff90 con 0x7ffa4c105460 2024-05-27T07:16:58.162 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:16:58.170+0000 7ffa4affd700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ffa4c104330 0x7ffa4c19a900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:16:58.163 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:16:58.170+0000 7ffa4affd700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ffa4c104330 0x7ffa4c19a900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.52:43334/0 (socket says 172.21.15.52:43334) 2024-05-27T07:16:58.163 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:16:58.170+0000 7ffa4affd700 1 -- 172.21.15.52:0/1437887350 learned_addr learned my addr 172.21.15.52:0/1437887350 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-05-27T07:16:58.163 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:16:58.170+0000 7ffa4a7fc700 1 --2- 172.21.15.52:0/1437887350 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7ffa4c105460 0x7ffa4c19ae40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:16:58.163 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:16:58.170+0000 7ffa4b7fe700 1 --2- 172.21.15.52:0/1437887350 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7ffa4c105e00 0x7ffa4c1a1de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:16:58.163 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:16:58.170+0000 7ffa4affd700 1 -- 172.21.15.52:0/1437887350 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7ffa4c105460 msgr2=0x7ffa4c19ae40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:16:58.163 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:16:58.170+0000 7ffa4affd700 1 --2- 172.21.15.52:0/1437887350 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7ffa4c105460 0x7ffa4c19ae40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:16:58.163 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:16:58.170+0000 7ffa4affd700 1 -- 172.21.15.52:0/1437887350 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7ffa4c105e00 msgr2=0x7ffa4c1a1de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:16:58.163 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:16:58.170+0000 7ffa4affd700 1 --2- 172.21.15.52:0/1437887350 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7ffa4c105e00 0x7ffa4c1a1de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:16:58.163 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:16:58.170+0000 7ffa4affd700 1 -- 172.21.15.52:0/1437887350 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ffa4c1a2460 con 0x7ffa4c104330 2024-05-27T07:16:58.163 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:16:58.170+0000 7ffa4affd700 1 --2- 172.21.15.52:0/1437887350 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ffa4c104330 0x7ffa4c19a900 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7ffa3c00b990 tx=0x7ffa3c00bca0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:16:58.163 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:16:58.171+0000 7ffa509c5700 1 -- 172.21.15.52:0/1437887350 <== mon.0 v2:172.21.15.52:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffa3c00d890 con 0x7ffa4c104330 2024-05-27T07:16:58.163 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:16:58.171+0000 7ffa509c5700 1 -- 172.21.15.52:0/1437887350 <== mon.0 v2:172.21.15.52:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ffa3c00e460 con 0x7ffa4c104330 2024-05-27T07:16:58.163 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:16:58.171+0000 7ffa509c5700 1 -- 172.21.15.52:0/1437887350 <== mon.0 v2:172.21.15.52:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffa3c017660 con 0x7ffa4c104330 2024-05-27T07:16:58.163 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:16:58.171+0000 7ffa519c7700 1 -- 172.21.15.52:0/1437887350 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ffa4c1a2750 con 0x7ffa4c104330 2024-05-27T07:16:58.164 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:16:58.171+0000 7ffa519c7700 1 -- 172.21.15.52:0/1437887350 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ffa4c1a2cc0 con 0x7ffa4c104330 2024-05-27T07:16:58.164 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:16:58.172+0000 7ffa519c7700 1 -- 172.21.15.52:0/1437887350 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ffa4c0512b0 con 0x7ffa4c104330 2024-05-27T07:16:58.164 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:16:58.174+0000 7ffa509c5700 1 -- 172.21.15.52:0/1437887350 <== mon.0 v2:172.21.15.52:3300/0 4 ==== mgrmap(e 15) ==== 149596+0+0 (secure 0 0 0) 0x7ffa3c00e610 con 0x7ffa4c104330 2024-05-27T07:16:58.165 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:16:58.174+0000 7ffa509c5700 1 --2- 172.21.15.52:0/1437887350 >> [v2:172.21.15.52:6800/3513075805,v1:172.21.15.52:6801/3513075805] conn(0x7ffa340b27f0 0x7ffa340b4d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:16:58.165 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:16:58.175+0000 7ffa4a7fc700 1 --2- 172.21.15.52:0/1437887350 >> [v2:172.21.15.52:6800/3513075805,v1:172.21.15.52:6801/3513075805] conn(0x7ffa340b27f0 0x7ffa340b4d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:16:58.165 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:16:58.175+0000 7ffa509c5700 1 -- 172.21.15.52:0/1437887350 <== mon.0 v2:172.21.15.52:3300/0 5 ==== osd_map(19..19 src has 1..19) ==== 3226+0+0 (secure 0 0 0) 0x7ffa3c0e0af0 con 0x7ffa4c104330 2024-05-27T07:16:58.165 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:16:58.175+0000 7ffa4a7fc700 1 --2- 172.21.15.52:0/1437887350 >> [v2:172.21.15.52:6800/3513075805,v1:172.21.15.52:6801/3513075805] conn(0x7ffa340b27f0 0x7ffa340b4d00 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7ffa4c19b9b0 tx=0x7ffa4000b410 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:16:58.169 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:16:58.178+0000 7ffa509c5700 1 -- 172.21.15.52:0/1437887350 <== mon.0 v2:172.21.15.52:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191963 (secure 0 0 0) 0x7ffa3c053df0 con 0x7ffa4c104330 2024-05-27T07:16:58.392 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:16:58.402+0000 7ffa519c7700 1 -- 172.21.15.52:0/1437887350 --> [v2:172.21.15.52:6800/3513075805,v1:172.21.15.52:6801/3513075805] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi052:vg_nvme/lv_3", "target": ["mon-mgr", ""]}) -- 0x7ffa4c063bf0 con 0x7ffa340b27f0 2024-05-27T07:16:58.927 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:16:58 smithi052 ceph-mon[31955]: pgmap v82: 0 pgs: ; 0 B data, 88 MiB used, 268 GiB / 268 GiB avail 2024-05-27T07:16:58.927 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:16:58 smithi052 ceph-mon[31955]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' cmd='[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]': finished 2024-05-27T07:16:58.927 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:16:58 smithi052 ceph-mon[31955]: osdmap e19: 3 total, 3 up, 3 in 2024-05-27T07:16:58.927 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:16:58 smithi052 ceph-mon[31955]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' cmd=[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]: dispatch 2024-05-27T07:16:58.928 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:16:58 smithi052 ceph-mon[31955]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-05-27T07:16:58.928 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:16:58 smithi052 ceph-mon[31955]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-05-27T07:16:58.928 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:16:58 smithi052 ceph-mon[31955]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T07:16:58.928 INFO:journalctl@ceph.osd.0.smithi052.stdout:May 27 07:16:58 smithi052 sudo[51437]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-05-27T07:16:58.928 INFO:journalctl@ceph.osd.0.smithi052.stdout:May 27 07:16:58 smithi052 sudo[51437]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-05-27T07:16:58.928 INFO:journalctl@ceph.osd.0.smithi052.stdout:May 27 07:16:58 smithi052 sudo[51437]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-05-27T07:16:58.928 INFO:journalctl@ceph.osd.0.smithi052.stdout:May 27 07:16:58 smithi052 sudo[51437]: pam_unix(sudo:session): session closed for user root 2024-05-27T07:16:58.928 INFO:journalctl@ceph.osd.0.smithi052.stdout:May 27 07:16:58 smithi052 sudo[51440]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-05-27T07:16:58.928 INFO:journalctl@ceph.osd.0.smithi052.stdout:May 27 07:16:58 smithi052 sudo[51440]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-05-27T07:16:58.928 INFO:journalctl@ceph.osd.0.smithi052.stdout:May 27 07:16:58 smithi052 sudo[51440]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-05-27T07:16:58.928 INFO:journalctl@ceph.osd.0.smithi052.stdout:May 27 07:16:58 smithi052 sudo[51440]: pam_unix(sudo:session): session closed for user root 2024-05-27T07:16:58.940 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:16:58 smithi105 ceph-mon[35977]: pgmap v82: 0 pgs: ; 0 B data, 88 MiB used, 268 GiB / 268 GiB avail 2024-05-27T07:16:58.940 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:16:58 smithi105 ceph-mon[35977]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' cmd='[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]': finished 2024-05-27T07:16:58.940 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:16:58 smithi105 ceph-mon[35977]: osdmap e19: 3 total, 3 up, 3 in 2024-05-27T07:16:58.940 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:16:58 smithi105 ceph-mon[35977]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' cmd=[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]: dispatch 2024-05-27T07:16:58.941 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:16:58 smithi105 ceph-mon[35977]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-05-27T07:16:58.941 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:16:58 smithi105 ceph-mon[35977]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-05-27T07:16:58.941 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:16:58 smithi105 ceph-mon[35977]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T07:16:58.941 INFO:journalctl@ceph.osd.1.smithi105.stdout:May 27 07:16:58 smithi105 sudo[44270]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-05-27T07:16:58.941 INFO:journalctl@ceph.osd.1.smithi105.stdout:May 27 07:16:58 smithi105 sudo[44270]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-05-27T07:16:58.941 INFO:journalctl@ceph.osd.1.smithi105.stdout:May 27 07:16:58 smithi105 sudo[44270]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-05-27T07:16:58.941 INFO:journalctl@ceph.osd.1.smithi105.stdout:May 27 07:16:58 smithi105 sudo[44270]: pam_unix(sudo:session): session closed for user root 2024-05-27T07:16:58.941 INFO:journalctl@ceph.osd.1.smithi105.stdout:May 27 07:16:58 smithi105 sudo[44273]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-05-27T07:16:58.941 INFO:journalctl@ceph.osd.1.smithi105.stdout:May 27 07:16:58 smithi105 sudo[44273]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-05-27T07:16:58.941 INFO:journalctl@ceph.osd.1.smithi105.stdout:May 27 07:16:58 smithi105 sudo[44273]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-05-27T07:16:58.941 INFO:journalctl@ceph.osd.1.smithi105.stdout:May 27 07:16:58 smithi105 sudo[44273]: pam_unix(sudo:session): session closed for user root 2024-05-27T07:16:58.968 INFO:journalctl@ceph.osd.2.smithi173.stdout:May 27 07:16:58 smithi173 sudo[46136]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-05-27T07:16:58.969 INFO:journalctl@ceph.osd.2.smithi173.stdout:May 27 07:16:58 smithi173 sudo[46136]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-05-27T07:16:58.969 INFO:journalctl@ceph.osd.2.smithi173.stdout:May 27 07:16:58 smithi173 sudo[46136]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-05-27T07:16:58.969 INFO:journalctl@ceph.osd.2.smithi173.stdout:May 27 07:16:58 smithi173 sudo[46136]: pam_unix(sudo:session): session closed for user root 2024-05-27T07:16:58.969 INFO:journalctl@ceph.osd.2.smithi173.stdout:May 27 07:16:58 smithi173 sudo[46139]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-05-27T07:16:58.969 INFO:journalctl@ceph.osd.2.smithi173.stdout:May 27 07:16:58 smithi173 sudo[46139]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-05-27T07:16:58.969 INFO:journalctl@ceph.osd.2.smithi173.stdout:May 27 07:16:58 smithi173 sudo[46139]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-05-27T07:16:58.969 INFO:journalctl@ceph.osd.2.smithi173.stdout:May 27 07:16:58 smithi173 sudo[46139]: pam_unix(sudo:session): session closed for user root 2024-05-27T07:16:58.969 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:16:58 smithi173 ceph-mon[37000]: pgmap v82: 0 pgs: ; 0 B data, 88 MiB used, 268 GiB / 268 GiB avail 2024-05-27T07:16:58.969 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:16:58 smithi173 ceph-mon[37000]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' cmd='[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]': finished 2024-05-27T07:16:58.969 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:16:58 smithi173 ceph-mon[37000]: osdmap e19: 3 total, 3 up, 3 in 2024-05-27T07:16:58.969 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:16:58 smithi173 ceph-mon[37000]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' cmd=[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]: dispatch 2024-05-27T07:16:58.969 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:16:58 smithi173 ceph-mon[37000]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-05-27T07:16:58.970 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:16:58 smithi173 ceph-mon[37000]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-05-27T07:16:58.970 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:16:58 smithi173 ceph-mon[37000]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T07:16:59.178 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:16:58 smithi052 sudo[51481]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-05-27T07:16:59.178 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:16:58 smithi052 sudo[51481]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-05-27T07:16:59.178 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:16:58 smithi052 sudo[51481]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-05-27T07:16:59.178 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:16:58 smithi052 sudo[51481]: pam_unix(sudo:session): session closed for user root 2024-05-27T07:16:59.178 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:16:58 smithi052 sudo[51499]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-05-27T07:16:59.178 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:16:58 smithi052 sudo[51499]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-05-27T07:16:59.178 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:16:58 smithi052 sudo[51499]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-05-27T07:16:59.178 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:16:58 smithi052 sudo[51499]: pam_unix(sudo:session): session closed for user root 2024-05-27T07:16:59.352 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:16:58 smithi173 sudo[46142]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-05-27T07:16:59.352 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:16:58 smithi173 sudo[46142]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-05-27T07:16:59.352 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:16:58 smithi173 sudo[46142]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-05-27T07:16:59.353 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:16:58 smithi173 sudo[46142]: pam_unix(sudo:session): session closed for user root 2024-05-27T07:16:59.353 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:16:59 smithi173 sudo[46145]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-05-27T07:16:59.353 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:16:59 smithi173 sudo[46145]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-05-27T07:16:59.353 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:16:59 smithi173 sudo[46145]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-05-27T07:16:59.353 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:16:59 smithi173 sudo[46145]: pam_unix(sudo:session): session closed for user root 2024-05-27T07:16:59.440 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:16:59 smithi105 sudo[44276]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-05-27T07:16:59.441 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:16:59 smithi105 sudo[44276]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-05-27T07:16:59.441 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:16:59 smithi105 sudo[44276]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-05-27T07:16:59.441 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:16:59 smithi105 sudo[44276]: pam_unix(sudo:session): session closed for user root 2024-05-27T07:16:59.441 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:16:59 smithi105 sudo[44279]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-05-27T07:16:59.441 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:16:59 smithi105 sudo[44279]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-05-27T07:16:59.441 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:16:59 smithi105 sudo[44279]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-05-27T07:16:59.441 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:16:59 smithi105 sudo[44279]: pam_unix(sudo:session): session closed for user root 2024-05-27T07:16:59.929 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:16:59.937+0000 7ffa509c5700 1 -- 172.21.15.52:0/1437887350 <== mon.0 v2:172.21.15.52:3300/0 7 ==== mgrmap(e 16) ==== 149652+0+0 (secure 0 0 0) 0x7ffa3c0a5860 con 0x7ffa4c104330 2024-05-27T07:16:59.929 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:16:59 smithi052 ceph-mon[31955]: from='client.14286 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi052:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-05-27T07:16:59.930 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:16:59 smithi052 ceph-mon[31955]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' cmd='[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]': finished 2024-05-27T07:16:59.930 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:16:59 smithi052 ceph-mon[31955]: osdmap e20: 3 total, 3 up, 3 in 2024-05-27T07:16:59.930 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:16:59 smithi052 ceph-mon[31955]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' 2024-05-27T07:16:59.930 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:16:59 smithi052 ceph-mon[31955]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' 2024-05-27T07:16:59.930 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:16:59 smithi052 ceph-mon[31955]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' 2024-05-27T07:16:59.930 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:16:59 smithi052 ceph-mon[31955]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-05-27T07:16:59.930 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:16:59 smithi052 ceph-mon[31955]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-05-27T07:16:59.930 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:16:59 smithi052 ceph-mon[31955]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-05-27T07:16:59.930 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:16:59 smithi052 ceph-mon[31955]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-05-27T07:16:59.930 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:16:59 smithi052 ceph-mon[31955]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-05-27T07:16:59.931 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:16:59 smithi052 ceph-mon[31955]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-05-27T07:16:59.931 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:16:59 smithi052 ceph-mon[31955]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-05-27T07:16:59.931 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:16:59 smithi052 ceph-mon[31955]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-05-27T07:16:59.931 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:16:59 smithi052 ceph-mon[31955]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-05-27T07:16:59.931 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:16:59 smithi052 ceph-mon[31955]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-05-27T07:16:59.931 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:16:59 smithi052 ceph-mon[31955]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-05-27T07:16:59.931 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:16:59 smithi052 ceph-mon[31955]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-05-27T07:16:59.931 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:16:59 smithi052 ceph-mon[31955]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-05-27T07:16:59.931 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:16:59 smithi052 ceph-mon[31955]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-05-27T07:16:59.931 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:16:59 smithi052 ceph-mon[31955]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-05-27T07:16:59.940 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:16:59 smithi105 ceph-mon[35977]: from='client.14286 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi052:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-05-27T07:16:59.940 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:16:59 smithi105 ceph-mon[35977]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' cmd='[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]': finished 2024-05-27T07:16:59.941 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:16:59 smithi105 ceph-mon[35977]: osdmap e20: 3 total, 3 up, 3 in 2024-05-27T07:16:59.941 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:16:59 smithi105 ceph-mon[35977]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' 2024-05-27T07:16:59.941 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:16:59 smithi105 ceph-mon[35977]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' 2024-05-27T07:16:59.941 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:16:59 smithi105 ceph-mon[35977]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' 2024-05-27T07:16:59.941 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:16:59 smithi105 ceph-mon[35977]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-05-27T07:16:59.941 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:16:59 smithi105 ceph-mon[35977]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-05-27T07:16:59.941 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:16:59 smithi105 ceph-mon[35977]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-05-27T07:16:59.941 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:16:59 smithi105 ceph-mon[35977]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-05-27T07:16:59.941 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:16:59 smithi105 ceph-mon[35977]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-05-27T07:16:59.941 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:16:59 smithi105 ceph-mon[35977]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-05-27T07:16:59.941 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:16:59 smithi105 ceph-mon[35977]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-05-27T07:16:59.941 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:16:59 smithi105 ceph-mon[35977]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-05-27T07:16:59.941 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:16:59 smithi105 ceph-mon[35977]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-05-27T07:16:59.941 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:16:59 smithi105 ceph-mon[35977]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-05-27T07:16:59.942 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:16:59 smithi105 ceph-mon[35977]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-05-27T07:16:59.942 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:16:59 smithi105 ceph-mon[35977]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-05-27T07:16:59.942 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:16:59 smithi105 ceph-mon[35977]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-05-27T07:16:59.942 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:16:59 smithi105 ceph-mon[35977]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-05-27T07:16:59.942 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:16:59 smithi105 ceph-mon[35977]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-05-27T07:17:00.103 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:16:59 smithi173 ceph-mon[37000]: from='client.14286 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi052:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-05-27T07:17:00.103 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:16:59 smithi173 ceph-mon[37000]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' cmd='[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]': finished 2024-05-27T07:17:00.103 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:16:59 smithi173 ceph-mon[37000]: osdmap e20: 3 total, 3 up, 3 in 2024-05-27T07:17:00.103 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:16:59 smithi173 ceph-mon[37000]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' 2024-05-27T07:17:00.103 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:16:59 smithi173 ceph-mon[37000]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' 2024-05-27T07:17:00.103 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:16:59 smithi173 ceph-mon[37000]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' 2024-05-27T07:17:00.103 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:16:59 smithi173 ceph-mon[37000]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-05-27T07:17:00.103 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:16:59 smithi173 ceph-mon[37000]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-05-27T07:17:00.103 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:16:59 smithi173 ceph-mon[37000]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-05-27T07:17:00.103 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:16:59 smithi173 ceph-mon[37000]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-05-27T07:17:00.103 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:16:59 smithi173 ceph-mon[37000]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-05-27T07:17:00.104 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:16:59 smithi173 ceph-mon[37000]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-05-27T07:17:00.104 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:16:59 smithi173 ceph-mon[37000]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-05-27T07:17:00.104 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:16:59 smithi173 ceph-mon[37000]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-05-27T07:17:00.104 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:16:59 smithi173 ceph-mon[37000]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-05-27T07:17:00.104 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:16:59 smithi173 ceph-mon[37000]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-05-27T07:17:00.104 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:16:59 smithi173 ceph-mon[37000]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-05-27T07:17:00.104 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:16:59 smithi173 ceph-mon[37000]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-05-27T07:17:00.104 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:16:59 smithi173 ceph-mon[37000]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-05-27T07:17:00.104 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:16:59 smithi173 ceph-mon[37000]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-05-27T07:17:00.104 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:16:59 smithi173 ceph-mon[37000]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-05-27T07:17:00.940 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:17:00 smithi105 ceph-mon[35977]: pgmap v85: 1 pgs: 1 unknown; 0 B data, 88 MiB used, 268 GiB / 268 GiB avail 2024-05-27T07:17:00.940 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:17:00 smithi105 ceph-mon[35977]: osdmap e21: 3 total, 3 up, 3 in 2024-05-27T07:17:00.941 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:17:00 smithi105 ceph-mon[35977]: mgrmap e16: x(active, since 2m), standbys: z, y 2024-05-27T07:17:00.980 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:17:00 smithi052 ceph-mon[31955]: pgmap v85: 1 pgs: 1 unknown; 0 B data, 88 MiB used, 268 GiB / 268 GiB avail 2024-05-27T07:17:00.980 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:17:00 smithi052 ceph-mon[31955]: osdmap e21: 3 total, 3 up, 3 in 2024-05-27T07:17:00.980 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:17:00 smithi052 ceph-mon[31955]: mgrmap e16: x(active, since 2m), standbys: z, y 2024-05-27T07:17:01.102 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:17:00 smithi173 ceph-mon[37000]: pgmap v85: 1 pgs: 1 unknown; 0 B data, 88 MiB used, 268 GiB / 268 GiB avail 2024-05-27T07:17:01.102 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:17:00 smithi173 ceph-mon[37000]: osdmap e21: 3 total, 3 up, 3 in 2024-05-27T07:17:01.102 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:17:00 smithi173 ceph-mon[37000]: mgrmap e16: x(active, since 2m), standbys: z, y 2024-05-27T07:17:01.730 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:17:01 smithi052 ceph-mon[31955]: from='client.? 172.21.15.52:0/2217226182' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "f1245439-42a5-46c5-a998-e0a78f9df3fb"}]: dispatch 2024-05-27T07:17:01.730 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:17:01 smithi052 ceph-mon[31955]: from='client.? 172.21.15.52:0/2217226182' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "f1245439-42a5-46c5-a998-e0a78f9df3fb"}]': finished 2024-05-27T07:17:01.730 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:17:01 smithi052 ceph-mon[31955]: osdmap e22: 4 total, 3 up, 4 in 2024-05-27T07:17:01.730 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:17:01 smithi052 ceph-mon[31955]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-05-27T07:17:01.940 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:17:01 smithi105 ceph-mon[35977]: from='client.? 172.21.15.52:0/2217226182' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "f1245439-42a5-46c5-a998-e0a78f9df3fb"}]: dispatch 2024-05-27T07:17:01.940 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:17:01 smithi105 ceph-mon[35977]: from='client.? 172.21.15.52:0/2217226182' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "f1245439-42a5-46c5-a998-e0a78f9df3fb"}]': finished 2024-05-27T07:17:01.940 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:17:01 smithi105 ceph-mon[35977]: osdmap e22: 4 total, 3 up, 4 in 2024-05-27T07:17:01.940 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:17:01 smithi105 ceph-mon[35977]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-05-27T07:17:02.102 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:17:01 smithi173 ceph-mon[37000]: from='client.? 172.21.15.52:0/2217226182' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "f1245439-42a5-46c5-a998-e0a78f9df3fb"}]: dispatch 2024-05-27T07:17:02.103 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:17:01 smithi173 ceph-mon[37000]: from='client.? 172.21.15.52:0/2217226182' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "f1245439-42a5-46c5-a998-e0a78f9df3fb"}]': finished 2024-05-27T07:17:02.103 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:17:01 smithi173 ceph-mon[37000]: osdmap e22: 4 total, 3 up, 4 in 2024-05-27T07:17:02.103 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:17:01 smithi173 ceph-mon[37000]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-05-27T07:17:02.940 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:17:02 smithi105 ceph-mon[35977]: pgmap v88: 1 pgs: 1 unknown; 0 B data, 88 MiB used, 268 GiB / 268 GiB avail 2024-05-27T07:17:02.940 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:17:02 smithi105 ceph-mon[35977]: from='client.? 172.21.15.52:0/2476279139' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-05-27T07:17:02.980 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:17:02 smithi052 ceph-mon[31955]: pgmap v88: 1 pgs: 1 unknown; 0 B data, 88 MiB used, 268 GiB / 268 GiB avail 2024-05-27T07:17:02.980 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:17:02 smithi052 ceph-mon[31955]: from='client.? 172.21.15.52:0/2476279139' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-05-27T07:17:03.102 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:17:02 smithi173 ceph-mon[37000]: pgmap v88: 1 pgs: 1 unknown; 0 B data, 88 MiB used, 268 GiB / 268 GiB avail 2024-05-27T07:17:03.103 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:17:02 smithi173 ceph-mon[37000]: from='client.? 172.21.15.52:0/2476279139' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-05-27T07:17:04.940 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:17:04 smithi105 ceph-mon[35977]: pgmap v89: 1 pgs: 1 active+clean; 577 KiB data, 88 MiB used, 268 GiB / 268 GiB avail 2024-05-27T07:17:04.979 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:17:04 smithi052 ceph-mon[31955]: pgmap v89: 1 pgs: 1 active+clean; 577 KiB data, 88 MiB used, 268 GiB / 268 GiB avail 2024-05-27T07:17:05.102 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:17:04 smithi173 ceph-mon[37000]: pgmap v89: 1 pgs: 1 active+clean; 577 KiB data, 88 MiB used, 268 GiB / 268 GiB avail 2024-05-27T07:17:06.940 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:17:06 smithi105 ceph-mon[35977]: pgmap v90: 1 pgs: 1 active+clean; 577 KiB data, 88 MiB used, 268 GiB / 268 GiB avail 2024-05-27T07:17:06.980 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:17:06 smithi052 ceph-mon[31955]: pgmap v90: 1 pgs: 1 active+clean; 577 KiB data, 88 MiB used, 268 GiB / 268 GiB avail 2024-05-27T07:17:07.102 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:17:06 smithi173 ceph-mon[37000]: pgmap v90: 1 pgs: 1 active+clean; 577 KiB data, 88 MiB used, 268 GiB / 268 GiB avail 2024-05-27T07:17:08.948 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:17:08 smithi052 ceph-mon[31955]: pgmap v91: 1 pgs: 1 active+clean; 577 KiB data, 88 MiB used, 268 GiB / 268 GiB avail 2024-05-27T07:17:08.948 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:17:08 smithi052 ceph-mon[31955]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2024-05-27T07:17:08.948 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:17:08 smithi052 ceph-mon[31955]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T07:17:09.102 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:17:08 smithi173 ceph-mon[37000]: pgmap v91: 1 pgs: 1 active+clean; 577 KiB data, 88 MiB used, 268 GiB / 268 GiB avail 2024-05-27T07:17:09.102 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:17:08 smithi173 ceph-mon[37000]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2024-05-27T07:17:09.103 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:17:08 smithi173 ceph-mon[37000]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T07:17:09.190 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:17:08 smithi105 ceph-mon[35977]: pgmap v91: 1 pgs: 1 active+clean; 577 KiB data, 88 MiB used, 268 GiB / 268 GiB avail 2024-05-27T07:17:09.190 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:17:08 smithi105 ceph-mon[35977]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2024-05-27T07:17:09.190 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:17:08 smithi105 ceph-mon[35977]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T07:17:09.846 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:17:09 smithi052 ceph-mon[31955]: Deploying daemon osd.3 on smithi052 2024-05-27T07:17:10.003 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:17:09 smithi173 ceph-mon[37000]: Deploying daemon osd.3 on smithi052 2024-05-27T07:17:10.190 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:17:09 smithi105 ceph-mon[35977]: Deploying daemon osd.3 on smithi052 2024-05-27T07:17:10.854 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:17:10 smithi052 ceph-mon[31955]: pgmap v92: 1 pgs: 1 active+clean; 577 KiB data, 88 MiB used, 268 GiB / 268 GiB avail 2024-05-27T07:17:11.020 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:17:10 smithi173 ceph-mon[37000]: pgmap v92: 1 pgs: 1 active+clean; 577 KiB data, 88 MiB used, 268 GiB / 268 GiB avail 2024-05-27T07:17:11.190 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:17:10 smithi105 ceph-mon[35977]: pgmap v92: 1 pgs: 1 active+clean; 577 KiB data, 88 MiB used, 268 GiB / 268 GiB avail 2024-05-27T07:17:12.980 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:17:12 smithi052 ceph-mon[31955]: pgmap v93: 1 pgs: 1 active+clean; 577 KiB data, 88 MiB used, 268 GiB / 268 GiB avail 2024-05-27T07:17:12.992 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:17:12 smithi105 ceph-mon[35977]: pgmap v93: 1 pgs: 1 active+clean; 577 KiB data, 88 MiB used, 268 GiB / 268 GiB avail 2024-05-27T07:17:13.102 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:17:12 smithi173 ceph-mon[37000]: pgmap v93: 1 pgs: 1 active+clean; 577 KiB data, 88 MiB used, 268 GiB / 268 GiB avail 2024-05-27T07:17:15.230 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:17:14 smithi052 ceph-mon[31955]: pgmap v94: 1 pgs: 1 active+clean; 577 KiB data, 88 MiB used, 268 GiB / 268 GiB avail 2024-05-27T07:17:15.230 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:17:14 smithi052 ceph-mon[31955]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' 2024-05-27T07:17:15.230 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:17:14 smithi052 ceph-mon[31955]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' 2024-05-27T07:17:15.231 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:17:14 smithi052 ceph-mon[31955]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-27T07:17:15.352 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:17:14 smithi173 ceph-mon[37000]: pgmap v94: 1 pgs: 1 active+clean; 577 KiB data, 88 MiB used, 268 GiB / 268 GiB avail 2024-05-27T07:17:15.352 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:17:14 smithi173 ceph-mon[37000]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' 2024-05-27T07:17:15.353 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:17:14 smithi173 ceph-mon[37000]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' 2024-05-27T07:17:15.353 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:17:14 smithi173 ceph-mon[37000]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-27T07:17:15.440 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:17:14 smithi105 ceph-mon[35977]: pgmap v94: 1 pgs: 1 active+clean; 577 KiB data, 88 MiB used, 268 GiB / 268 GiB avail 2024-05-27T07:17:15.440 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:17:14 smithi105 ceph-mon[35977]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' 2024-05-27T07:17:15.441 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:17:14 smithi105 ceph-mon[35977]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' 2024-05-27T07:17:15.441 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:17:14 smithi105 ceph-mon[35977]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-27T07:17:17.158 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:17:16 smithi052 ceph-mon[31955]: pgmap v95: 1 pgs: 1 active+clean; 577 KiB data, 88 MiB used, 268 GiB / 268 GiB avail 2024-05-27T07:17:17.203 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:17:17.212+0000 7ffa509c5700 1 -- 172.21.15.52:0/1437887350 <== mgr.14152 v2:172.21.15.52:6800/3513075805 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+37 (secure 0 0 0) 0x7ffa4c063bf0 con 0x7ffa340b27f0 2024-05-27T07:17:17.203 INFO:teuthology.orchestra.run.smithi052.stdout:Created osd(s) 3 on host 'smithi052' 2024-05-27T07:17:17.205 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:17:17.214+0000 7ffa519c7700 1 -- 172.21.15.52:0/1437887350 >> [v2:172.21.15.52:6800/3513075805,v1:172.21.15.52:6801/3513075805] conn(0x7ffa340b27f0 msgr2=0x7ffa340b4d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:17:17.205 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:17:17.214+0000 7ffa519c7700 1 --2- 172.21.15.52:0/1437887350 >> [v2:172.21.15.52:6800/3513075805,v1:172.21.15.52:6801/3513075805] conn(0x7ffa340b27f0 0x7ffa340b4d00 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7ffa4c19b9b0 tx=0x7ffa4000b410 comp rx=0 tx=0).stop 2024-05-27T07:17:17.206 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:17:17.214+0000 7ffa519c7700 1 -- 172.21.15.52:0/1437887350 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ffa4c104330 msgr2=0x7ffa4c19a900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:17:17.206 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:17:17.215+0000 7ffa519c7700 1 --2- 172.21.15.52:0/1437887350 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ffa4c104330 0x7ffa4c19a900 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7ffa3c00b990 tx=0x7ffa3c00bca0 comp rx=0 tx=0).stop 2024-05-27T07:17:17.206 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:17:17.215+0000 7ffa519c7700 1 -- 172.21.15.52:0/1437887350 shutdown_connections 2024-05-27T07:17:17.206 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:17:17.215+0000 7ffa519c7700 1 --2- 172.21.15.52:0/1437887350 >> [v2:172.21.15.52:6800/3513075805,v1:172.21.15.52:6801/3513075805] conn(0x7ffa340b27f0 0x7ffa340b4d00 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:17:17.206 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:17:17.215+0000 7ffa519c7700 1 --2- 172.21.15.52:0/1437887350 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7ffa4c105e00 0x7ffa4c1a1de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:17:17.207 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:17:17.215+0000 7ffa519c7700 1 --2- 172.21.15.52:0/1437887350 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ffa4c104330 0x7ffa4c19a900 unknown :-1 s=CLOSED pgs=119 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:17:17.207 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:17:17.215+0000 7ffa519c7700 1 --2- 172.21.15.52:0/1437887350 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7ffa4c105460 0x7ffa4c19ae40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:17:17.207 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:17:17.215+0000 7ffa519c7700 1 -- 172.21.15.52:0/1437887350 >> 172.21.15.52:0/1437887350 conn(0x7ffa4c0ff9b0 msgr2=0x7ffa4c108690 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:17:17.207 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:17:17.215+0000 7ffa519c7700 1 -- 172.21.15.52:0/1437887350 shutdown_connections 2024-05-27T07:17:17.207 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:17:17.215+0000 7ffa519c7700 1 -- 172.21.15.52:0/1437887350 wait complete. 2024-05-27T07:17:17.352 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:17:16 smithi173 ceph-mon[37000]: pgmap v95: 1 pgs: 1 active+clean; 577 KiB data, 88 MiB used, 268 GiB / 268 GiB avail 2024-05-27T07:17:17.440 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:17:16 smithi105 ceph-mon[35977]: pgmap v95: 1 pgs: 1 active+clean; 577 KiB data, 88 MiB used, 268 GiB / 268 GiB avail 2024-05-27T07:17:17.891 DEBUG:teuthology.orchestra.run.smithi052:osd.3> sudo journalctl -f -n 0 -u ceph-73df2636-1bf8-11ef-bc9b-c7b262605968@osd.3.service 2024-05-27T07:17:17.893 INFO:tasks.cephadm:Deploying osd.4 on smithi105 with /dev/vg_nvme/lv_3... 2024-05-27T07:17:17.893 DEBUG:teuthology.orchestra.run.smithi105:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:38f9d900c83c63799fbdbe61acc9a11b0d3554a6 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 73df2636-1bf8-11ef-bc9b-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_3 2024-05-27T07:17:18.134 INFO:teuthology.orchestra.run.smithi105.stderr:Inferring config /var/lib/ceph/73df2636-1bf8-11ef-bc9b-c7b262605968/mon.b/config 2024-05-27T07:17:18.373 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:17:18 smithi052 ceph-mon[31955]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' 2024-05-27T07:17:18.374 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:17:18 smithi052 ceph-mon[31955]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' 2024-05-27T07:17:18.374 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:17:18 smithi052 ceph-mon[31955]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' 2024-05-27T07:17:18.374 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:17:18 smithi052 ceph-mon[31955]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' 2024-05-27T07:17:18.374 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:17:18 smithi052 ceph-mon[31955]: pgmap v96: 1 pgs: 1 active+clean; 577 KiB data, 88 MiB used, 268 GiB / 268 GiB avail 2024-05-27T07:17:18.374 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:17:18 smithi052 ceph-mon[31955]: from='osd.3 [v2:172.21.15.52:6810/57920794,v1:172.21.15.52:6811/57920794]' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-05-27T07:17:18.477 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:17:18 smithi105 ceph-mon[35977]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' 2024-05-27T07:17:18.477 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:17:18 smithi105 ceph-mon[35977]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' 2024-05-27T07:17:18.477 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:17:18 smithi105 ceph-mon[35977]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' 2024-05-27T07:17:18.477 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:17:18 smithi105 ceph-mon[35977]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' 2024-05-27T07:17:18.477 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:17:18 smithi105 ceph-mon[35977]: pgmap v96: 1 pgs: 1 active+clean; 577 KiB data, 88 MiB used, 268 GiB / 268 GiB avail 2024-05-27T07:17:18.477 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:17:18 smithi105 ceph-mon[35977]: from='osd.3 [v2:172.21.15.52:6810/57920794,v1:172.21.15.52:6811/57920794]' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-05-27T07:17:18.602 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:17:18 smithi173 ceph-mon[37000]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' 2024-05-27T07:17:18.602 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:17:18 smithi173 ceph-mon[37000]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' 2024-05-27T07:17:18.603 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:17:18 smithi173 ceph-mon[37000]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' 2024-05-27T07:17:18.603 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:17:18 smithi173 ceph-mon[37000]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' 2024-05-27T07:17:18.603 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:17:18 smithi173 ceph-mon[37000]: pgmap v96: 1 pgs: 1 active+clean; 577 KiB data, 88 MiB used, 268 GiB / 268 GiB avail 2024-05-27T07:17:18.603 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:17:18 smithi173 ceph-mon[37000]: from='osd.3 [v2:172.21.15.52:6810/57920794,v1:172.21.15.52:6811/57920794]' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-05-27T07:17:19.730 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:17:19 smithi052 ceph-mon[31955]: from='osd.3 [v2:172.21.15.52:6810/57920794,v1:172.21.15.52:6811/57920794]' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2024-05-27T07:17:19.730 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:17:19 smithi052 ceph-mon[31955]: osdmap e23: 4 total, 3 up, 4 in 2024-05-27T07:17:19.730 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:17:19 smithi052 ceph-mon[31955]: from='osd.3 [v2:172.21.15.52:6810/57920794,v1:172.21.15.52:6811/57920794]' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi052", "root=default"]}]: dispatch 2024-05-27T07:17:19.730 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:17:19 smithi052 ceph-mon[31955]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-05-27T07:17:19.852 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:17:19 smithi173 ceph-mon[37000]: from='osd.3 [v2:172.21.15.52:6810/57920794,v1:172.21.15.52:6811/57920794]' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2024-05-27T07:17:19.852 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:17:19 smithi173 ceph-mon[37000]: osdmap e23: 4 total, 3 up, 4 in 2024-05-27T07:17:19.853 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:17:19 smithi173 ceph-mon[37000]: from='osd.3 [v2:172.21.15.52:6810/57920794,v1:172.21.15.52:6811/57920794]' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi052", "root=default"]}]: dispatch 2024-05-27T07:17:19.853 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:17:19 smithi173 ceph-mon[37000]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-05-27T07:17:19.940 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:17:19 smithi105 ceph-mon[35977]: from='osd.3 [v2:172.21.15.52:6810/57920794,v1:172.21.15.52:6811/57920794]' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2024-05-27T07:17:19.941 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:17:19 smithi105 ceph-mon[35977]: osdmap e23: 4 total, 3 up, 4 in 2024-05-27T07:17:19.941 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:17:19 smithi105 ceph-mon[35977]: from='osd.3 [v2:172.21.15.52:6810/57920794,v1:172.21.15.52:6811/57920794]' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi052", "root=default"]}]: dispatch 2024-05-27T07:17:19.941 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:17:19 smithi105 ceph-mon[35977]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-05-27T07:17:20.077 INFO:teuthology.orchestra.run.smithi105.stderr:--> Zapping: /dev/vg_nvme/lv_3 2024-05-27T07:17:20.077 INFO:teuthology.orchestra.run.smithi105.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_3 bs=1M count=10 conv=fsync 2024-05-27T07:17:20.078 INFO:teuthology.orchestra.run.smithi105.stderr: stderr: 10+0 records in 2024-05-27T07:17:20.078 INFO:teuthology.orchestra.run.smithi105.stderr:10+0 records out 2024-05-27T07:17:20.078 INFO:teuthology.orchestra.run.smithi105.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0113139 s, 927 MB/s 2024-05-27T07:17:20.078 INFO:teuthology.orchestra.run.smithi105.stderr:--> Zapping successful for: 2024-05-27T07:17:20.236 INFO:journalctl@ceph.osd.3.smithi052.stdout:May 27 07:17:19 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-osd-3[53606]: 2024-05-27T07:17:19.990+0000 7f53295c6700 -1 osd.3 0 waiting for initial osdmap 2024-05-27T07:17:20.236 INFO:journalctl@ceph.osd.3.smithi052.stdout:May 27 07:17:19 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-osd-3[53606]: 2024-05-27T07:17:19.998+0000 7f5321373700 -1 osd.3 25 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-05-27T07:17:20.690 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:17:20 smithi105 ceph-mon[35977]: from='osd.3 [v2:172.21.15.52:6810/57920794,v1:172.21.15.52:6811/57920794]' entity='osd.3' cmd='[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi052", "root=default"]}]': finished 2024-05-27T07:17:20.691 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:17:20 smithi105 ceph-mon[35977]: osdmap e24: 4 total, 3 up, 4 in 2024-05-27T07:17:20.691 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:17:20 smithi105 ceph-mon[35977]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-05-27T07:17:20.691 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:17:20 smithi105 ceph-mon[35977]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-05-27T07:17:20.691 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:17:20 smithi105 ceph-mon[35977]: pgmap v99: 1 pgs: 1 active+clean; 577 KiB data, 88 MiB used, 268 GiB / 268 GiB avail 2024-05-27T07:17:20.691 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:17:20 smithi105 ceph-mon[35977]: osdmap e25: 4 total, 3 up, 4 in 2024-05-27T07:17:20.691 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:17:20 smithi105 ceph-mon[35977]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-05-27T07:17:20.691 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:17:20 smithi105 ceph-mon[35977]: from='osd.3 [v2:172.21.15.52:6810/57920794,v1:172.21.15.52:6811/57920794]' entity='osd.3' 2024-05-27T07:17:20.722 DEBUG:teuthology.orchestra.run.smithi105:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:38f9d900c83c63799fbdbe61acc9a11b0d3554a6 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 73df2636-1bf8-11ef-bc9b-c7b262605968 -- ceph orch daemon add osd smithi105:vg_nvme/lv_3 2024-05-27T07:17:20.802 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:17:20 smithi052 ceph-mon[31955]: from='osd.3 [v2:172.21.15.52:6810/57920794,v1:172.21.15.52:6811/57920794]' entity='osd.3' cmd='[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi052", "root=default"]}]': finished 2024-05-27T07:17:20.803 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:17:20 smithi052 ceph-mon[31955]: osdmap e24: 4 total, 3 up, 4 in 2024-05-27T07:17:20.803 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:17:20 smithi052 ceph-mon[31955]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-05-27T07:17:20.803 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:17:20 smithi052 ceph-mon[31955]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-05-27T07:17:20.803 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:17:20 smithi052 ceph-mon[31955]: pgmap v99: 1 pgs: 1 active+clean; 577 KiB data, 88 MiB used, 268 GiB / 268 GiB avail 2024-05-27T07:17:20.803 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:17:20 smithi052 ceph-mon[31955]: osdmap e25: 4 total, 3 up, 4 in 2024-05-27T07:17:20.803 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:17:20 smithi052 ceph-mon[31955]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-05-27T07:17:20.803 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:17:20 smithi052 ceph-mon[31955]: from='osd.3 [v2:172.21.15.52:6810/57920794,v1:172.21.15.52:6811/57920794]' entity='osd.3' 2024-05-27T07:17:20.852 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:17:20 smithi173 ceph-mon[37000]: from='osd.3 [v2:172.21.15.52:6810/57920794,v1:172.21.15.52:6811/57920794]' entity='osd.3' cmd='[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi052", "root=default"]}]': finished 2024-05-27T07:17:20.852 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:17:20 smithi173 ceph-mon[37000]: osdmap e24: 4 total, 3 up, 4 in 2024-05-27T07:17:20.852 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:17:20 smithi173 ceph-mon[37000]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-05-27T07:17:20.852 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:17:20 smithi173 ceph-mon[37000]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-05-27T07:17:20.852 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:17:20 smithi173 ceph-mon[37000]: pgmap v99: 1 pgs: 1 active+clean; 577 KiB data, 88 MiB used, 268 GiB / 268 GiB avail 2024-05-27T07:17:20.853 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:17:20 smithi173 ceph-mon[37000]: osdmap e25: 4 total, 3 up, 4 in 2024-05-27T07:17:20.853 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:17:20 smithi173 ceph-mon[37000]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-05-27T07:17:20.853 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:17:20 smithi173 ceph-mon[37000]: from='osd.3 [v2:172.21.15.52:6810/57920794,v1:172.21.15.52:6811/57920794]' entity='osd.3' 2024-05-27T07:17:20.954 INFO:teuthology.orchestra.run.smithi105.stderr:Inferring config /var/lib/ceph/73df2636-1bf8-11ef-bc9b-c7b262605968/mon.b/config 2024-05-27T07:17:21.691 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:17:21 smithi105 ceph-mon[35977]: purged_snaps scrub starts 2024-05-27T07:17:21.691 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:17:21 smithi105 ceph-mon[35977]: purged_snaps scrub ok 2024-05-27T07:17:21.691 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:17:21 smithi105 ceph-mon[35977]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-05-27T07:17:21.691 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:17:21 smithi105 ceph-mon[35977]: osd.3 [v2:172.21.15.52:6810/57920794,v1:172.21.15.52:6811/57920794] boot 2024-05-27T07:17:21.691 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:17:21 smithi105 ceph-mon[35977]: osdmap e26: 4 total, 4 up, 4 in 2024-05-27T07:17:21.691 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:17:21 smithi105 ceph-mon[35977]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-05-27T07:17:21.691 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:17:21 smithi105 ceph-mon[35977]: Detected new or changed devices on smithi052 2024-05-27T07:17:21.691 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:17:21 smithi105 ceph-mon[35977]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' 2024-05-27T07:17:21.691 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:17:21 smithi105 ceph-mon[35977]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' 2024-05-27T07:17:21.692 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:17:21 smithi105 ceph-mon[35977]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-05-27T07:17:21.692 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:17:21 smithi105 ceph-mon[35977]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-05-27T07:17:21.692 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:17:21 smithi105 ceph-mon[35977]: Adjusting osd_memory_target on smithi052 to 8546M 2024-05-27T07:17:21.692 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:17:21 smithi105 ceph-mon[35977]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' 2024-05-27T07:17:21.692 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:17:21 smithi105 ceph-mon[35977]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T07:17:21.692 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:17:21 smithi105 ceph-mon[35977]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-27T07:17:21.692 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:17:21 smithi105 ceph-mon[35977]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' 2024-05-27T07:17:21.692 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:17:21 smithi105 ceph-mon[35977]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-27T07:17:21.692 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:17:21 smithi105 ceph-mon[35977]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T07:17:21.692 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:17:21 smithi105 ceph-mon[35977]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-27T07:17:21.692 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:17:21 smithi105 ceph-mon[35977]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' 2024-05-27T07:17:21.852 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:17:21 smithi173 ceph-mon[37000]: purged_snaps scrub starts 2024-05-27T07:17:21.853 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:17:21 smithi173 ceph-mon[37000]: purged_snaps scrub ok 2024-05-27T07:17:21.853 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:17:21 smithi173 ceph-mon[37000]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-05-27T07:17:21.853 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:17:21 smithi173 ceph-mon[37000]: osd.3 [v2:172.21.15.52:6810/57920794,v1:172.21.15.52:6811/57920794] boot 2024-05-27T07:17:21.853 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:17:21 smithi173 ceph-mon[37000]: osdmap e26: 4 total, 4 up, 4 in 2024-05-27T07:17:21.853 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:17:21 smithi173 ceph-mon[37000]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-05-27T07:17:21.853 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:17:21 smithi173 ceph-mon[37000]: Detected new or changed devices on smithi052 2024-05-27T07:17:21.853 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:17:21 smithi173 ceph-mon[37000]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' 2024-05-27T07:17:21.853 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:17:21 smithi173 ceph-mon[37000]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' 2024-05-27T07:17:21.853 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:17:21 smithi173 ceph-mon[37000]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-05-27T07:17:21.853 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:17:21 smithi173 ceph-mon[37000]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-05-27T07:17:21.853 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:17:21 smithi173 ceph-mon[37000]: Adjusting osd_memory_target on smithi052 to 8546M 2024-05-27T07:17:21.853 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:17:21 smithi173 ceph-mon[37000]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' 2024-05-27T07:17:21.854 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:17:21 smithi173 ceph-mon[37000]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T07:17:21.854 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:17:21 smithi173 ceph-mon[37000]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-27T07:17:21.854 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:17:21 smithi173 ceph-mon[37000]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' 2024-05-27T07:17:21.854 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:17:21 smithi173 ceph-mon[37000]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-27T07:17:21.854 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:17:21 smithi173 ceph-mon[37000]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T07:17:21.854 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:17:21 smithi173 ceph-mon[37000]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-27T07:17:21.854 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:17:21 smithi173 ceph-mon[37000]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' 2024-05-27T07:17:21.980 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:17:21 smithi052 ceph-mon[31955]: purged_snaps scrub starts 2024-05-27T07:17:21.980 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:17:21 smithi052 ceph-mon[31955]: purged_snaps scrub ok 2024-05-27T07:17:21.980 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:17:21 smithi052 ceph-mon[31955]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-05-27T07:17:21.980 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:17:21 smithi052 ceph-mon[31955]: osd.3 [v2:172.21.15.52:6810/57920794,v1:172.21.15.52:6811/57920794] boot 2024-05-27T07:17:21.980 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:17:21 smithi052 ceph-mon[31955]: osdmap e26: 4 total, 4 up, 4 in 2024-05-27T07:17:21.980 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:17:21 smithi052 ceph-mon[31955]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-05-27T07:17:21.981 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:17:21 smithi052 ceph-mon[31955]: Detected new or changed devices on smithi052 2024-05-27T07:17:21.981 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:17:21 smithi052 ceph-mon[31955]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' 2024-05-27T07:17:21.981 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:17:21 smithi052 ceph-mon[31955]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' 2024-05-27T07:17:21.981 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:17:21 smithi052 ceph-mon[31955]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-05-27T07:17:21.981 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:17:21 smithi052 ceph-mon[31955]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-05-27T07:17:21.981 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:17:21 smithi052 ceph-mon[31955]: Adjusting osd_memory_target on smithi052 to 8546M 2024-05-27T07:17:21.981 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:17:21 smithi052 ceph-mon[31955]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' 2024-05-27T07:17:21.981 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:17:21 smithi052 ceph-mon[31955]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T07:17:21.981 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:17:21 smithi052 ceph-mon[31955]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-27T07:17:21.981 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:17:21 smithi052 ceph-mon[31955]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' 2024-05-27T07:17:21.981 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:17:21 smithi052 ceph-mon[31955]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-27T07:17:21.981 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:17:21 smithi052 ceph-mon[31955]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T07:17:21.981 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:17:21 smithi052 ceph-mon[31955]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-27T07:17:21.981 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:17:21 smithi052 ceph-mon[31955]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' 2024-05-27T07:17:22.594 INFO:teuthology.orchestra.run.smithi105.stderr:2024-05-27T07:17:22.603+0000 7f470cd6a700 1 -- 172.21.15.105:0/2879863675 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f470810a0f0 msgr2=0x7f470810c510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:17:22.594 INFO:teuthology.orchestra.run.smithi105.stderr:2024-05-27T07:17:22.603+0000 7f470cd6a700 1 --2- 172.21.15.105:0/2879863675 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f470810a0f0 0x7f470810c510 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7f470000b0a0 tx=0x7f470000b3b0 comp rx=0 tx=0).stop 2024-05-27T07:17:22.594 INFO:teuthology.orchestra.run.smithi105.stderr:2024-05-27T07:17:22.603+0000 7f470cd6a700 1 -- 172.21.15.105:0/2879863675 shutdown_connections 2024-05-27T07:17:22.594 INFO:teuthology.orchestra.run.smithi105.stderr:2024-05-27T07:17:22.603+0000 7f470cd6a700 1 --2- 172.21.15.105:0/2879863675 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f470810a0f0 0x7f470810c510 unknown :-1 s=CLOSED pgs=17 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:17:22.594 INFO:teuthology.orchestra.run.smithi105.stderr:2024-05-27T07:17:22.603+0000 7f470cd6a700 1 --2- 172.21.15.105:0/2879863675 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f4708104e30 0x7f4708107250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:17:22.595 INFO:teuthology.orchestra.run.smithi105.stderr:2024-05-27T07:17:22.603+0000 7f470cd6a700 1 --2- 172.21.15.105:0/2879863675 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f4708107790 0x7f4708109bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:17:22.595 INFO:teuthology.orchestra.run.smithi105.stderr:2024-05-27T07:17:22.603+0000 7f470cd6a700 1 -- 172.21.15.105:0/2879863675 >> 172.21.15.105:0/2879863675 conn(0x7f47080fe870 msgr2=0x7f4708100d20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:17:22.595 INFO:teuthology.orchestra.run.smithi105.stderr:2024-05-27T07:17:22.603+0000 7f470cd6a700 1 -- 172.21.15.105:0/2879863675 shutdown_connections 2024-05-27T07:17:22.595 INFO:teuthology.orchestra.run.smithi105.stderr:2024-05-27T07:17:22.603+0000 7f470cd6a700 1 -- 172.21.15.105:0/2879863675 wait complete. 2024-05-27T07:17:22.597 INFO:teuthology.orchestra.run.smithi105.stderr:2024-05-27T07:17:22.604+0000 7f470cd6a700 1 Processor -- start 2024-05-27T07:17:22.597 INFO:teuthology.orchestra.run.smithi105.stderr:2024-05-27T07:17:22.604+0000 7f470cd6a700 1 -- start start 2024-05-27T07:17:22.597 INFO:teuthology.orchestra.run.smithi105.stderr:2024-05-27T07:17:22.604+0000 7f470cd6a700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f4708104e30 0x7f470819a910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:17:22.597 INFO:teuthology.orchestra.run.smithi105.stderr:2024-05-27T07:17:22.604+0000 7f470cd6a700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f4708107790 0x7f470819ae50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:17:22.597 INFO:teuthology.orchestra.run.smithi105.stderr:2024-05-27T07:17:22.604+0000 7f470cd6a700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f470810a0f0 0x7f47081a1df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:17:22.597 INFO:teuthology.orchestra.run.smithi105.stderr:2024-05-27T07:17:22.604+0000 7f470cd6a700 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x7f470810fe50 con 0x7f4708104e30 2024-05-27T07:17:22.597 INFO:teuthology.orchestra.run.smithi105.stderr:2024-05-27T07:17:22.604+0000 7f470cd6a700 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f470810fcd0 con 0x7f470810a0f0 2024-05-27T07:17:22.597 INFO:teuthology.orchestra.run.smithi105.stderr:2024-05-27T07:17:22.604+0000 7f470cd6a700 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f470810ffd0 con 0x7f4708107790 2024-05-27T07:17:22.597 INFO:teuthology.orchestra.run.smithi105.stderr:2024-05-27T07:17:22.604+0000 7f4706d9d700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f470810a0f0 0x7f47081a1df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:17:22.597 INFO:teuthology.orchestra.run.smithi105.stderr:2024-05-27T07:17:22.604+0000 7f4706d9d700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f470810a0f0 0x7f47081a1df0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.105:60630/0 (socket says 172.21.15.105:60630) 2024-05-27T07:17:22.597 INFO:teuthology.orchestra.run.smithi105.stderr:2024-05-27T07:17:22.604+0000 7f4706d9d700 1 -- 172.21.15.105:0/8001890 learned_addr learned my addr 172.21.15.105:0/8001890 (peer_addr_for_me v2:172.21.15.105:0/0) 2024-05-27T07:17:22.598 INFO:teuthology.orchestra.run.smithi105.stderr:2024-05-27T07:17:22.604+0000 7f46fffff700 1 --2- 172.21.15.105:0/8001890 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f4708107790 0x7f470819ae50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:17:22.598 INFO:teuthology.orchestra.run.smithi105.stderr:2024-05-27T07:17:22.604+0000 7f470659c700 1 --2- 172.21.15.105:0/8001890 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f4708104e30 0x7f470819a910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:17:22.598 INFO:teuthology.orchestra.run.smithi105.stderr:2024-05-27T07:17:22.605+0000 7f4706d9d700 1 -- 172.21.15.105:0/8001890 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f4708107790 msgr2=0x7f470819ae50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:17:22.598 INFO:teuthology.orchestra.run.smithi105.stderr:2024-05-27T07:17:22.605+0000 7f4706d9d700 1 --2- 172.21.15.105:0/8001890 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f4708107790 0x7f470819ae50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:17:22.598 INFO:teuthology.orchestra.run.smithi105.stderr:2024-05-27T07:17:22.605+0000 7f4706d9d700 1 -- 172.21.15.105:0/8001890 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f4708104e30 msgr2=0x7f470819a910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:17:22.598 INFO:teuthology.orchestra.run.smithi105.stderr:2024-05-27T07:17:22.605+0000 7f4706d9d700 1 --2- 172.21.15.105:0/8001890 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f4708104e30 0x7f470819a910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:17:22.598 INFO:teuthology.orchestra.run.smithi105.stderr:2024-05-27T07:17:22.605+0000 7f4706d9d700 1 -- 172.21.15.105:0/8001890 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f47081a2470 con 0x7f470810a0f0 2024-05-27T07:17:22.598 INFO:teuthology.orchestra.run.smithi105.stderr:2024-05-27T07:17:22.605+0000 7f4706d9d700 1 --2- 172.21.15.105:0/8001890 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f470810a0f0 0x7f47081a1df0 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7f4700009fd0 tx=0x7f4700004da0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:17:22.598 INFO:teuthology.orchestra.run.smithi105.stderr:2024-05-27T07:17:22.605+0000 7f46ff7fe700 1 -- 172.21.15.105:0/8001890 <== mon.2 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4700021070 con 0x7f470810a0f0 2024-05-27T07:17:22.598 INFO:teuthology.orchestra.run.smithi105.stderr:2024-05-27T07:17:22.605+0000 7f470cd6a700 1 -- 172.21.15.105:0/8001890 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f47081a2700 con 0x7f470810a0f0 2024-05-27T07:17:22.598 INFO:teuthology.orchestra.run.smithi105.stderr:2024-05-27T07:17:22.605+0000 7f46ff7fe700 1 -- 172.21.15.105:0/8001890 <== mon.2 v2:172.21.15.105:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f4700007b80 con 0x7f470810a0f0 2024-05-27T07:17:22.598 INFO:teuthology.orchestra.run.smithi105.stderr:2024-05-27T07:17:22.605+0000 7f46ff7fe700 1 -- 172.21.15.105:0/8001890 <== mon.2 v2:172.21.15.105:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4700005280 con 0x7f470810a0f0 2024-05-27T07:17:22.598 INFO:teuthology.orchestra.run.smithi105.stderr:2024-05-27T07:17:22.605+0000 7f470cd6a700 1 -- 172.21.15.105:0/8001890 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f47081a2cc0 con 0x7f470810a0f0 2024-05-27T07:17:22.599 INFO:teuthology.orchestra.run.smithi105.stderr:2024-05-27T07:17:22.606+0000 7f470cd6a700 1 -- 172.21.15.105:0/8001890 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f47080512b0 con 0x7f470810a0f0 2024-05-27T07:17:22.599 INFO:teuthology.orchestra.run.smithi105.stderr:2024-05-27T07:17:22.607+0000 7f46ff7fe700 1 -- 172.21.15.105:0/8001890 <== mon.2 v2:172.21.15.105:3300/0 4 ==== mgrmap(e 16) ==== 149652+0+0 (secure 0 0 0) 0x7f4700024070 con 0x7f470810a0f0 2024-05-27T07:17:22.599 INFO:teuthology.orchestra.run.smithi105.stderr:2024-05-27T07:17:22.608+0000 7f46ff7fe700 1 --2- 172.21.15.105:0/8001890 >> [v2:172.21.15.52:6800/3513075805,v1:172.21.15.52:6801/3513075805] conn(0x7f46f40b2880 0x7f46f40b4d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:17:22.599 INFO:teuthology.orchestra.run.smithi105.stderr:2024-05-27T07:17:22.608+0000 7f46ff7fe700 1 -- 172.21.15.105:0/8001890 <== mon.2 v2:172.21.15.105:3300/0 5 ==== osd_map(27..27 src has 1..27) ==== 3669+0+0 (secure 0 0 0) 0x7f470000f020 con 0x7f470810a0f0 2024-05-27T07:17:22.599 INFO:teuthology.orchestra.run.smithi105.stderr:2024-05-27T07:17:22.608+0000 7f470659c700 1 --2- 172.21.15.105:0/8001890 >> [v2:172.21.15.52:6800/3513075805,v1:172.21.15.52:6801/3513075805] conn(0x7f46f40b2880 0x7f46f40b4d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:17:22.600 INFO:teuthology.orchestra.run.smithi105.stderr:2024-05-27T07:17:22.609+0000 7f470659c700 1 --2- 172.21.15.105:0/8001890 >> [v2:172.21.15.52:6800/3513075805,v1:172.21.15.52:6801/3513075805] conn(0x7f46f40b2880 0x7f46f40b4d90 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f470806c020 tx=0x7f46f8009190 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:17:22.604 INFO:teuthology.orchestra.run.smithi105.stderr:2024-05-27T07:17:22.613+0000 7f46ff7fe700 1 -- 172.21.15.105:0/8001890 <== mon.2 v2:172.21.15.105:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191963 (secure 0 0 0) 0x7f470002b020 con 0x7f470810a0f0 2024-05-27T07:17:22.819 INFO:teuthology.orchestra.run.smithi105.stderr:2024-05-27T07:17:22.828+0000 7f470cd6a700 1 -- 172.21.15.105:0/8001890 --> [v2:172.21.15.52:6800/3513075805,v1:172.21.15.52:6801/3513075805] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi105:vg_nvme/lv_3", "target": ["mon-mgr", ""]}) -- 0x7f4708063bf0 con 0x7f46f40b2880 2024-05-27T07:17:23.102 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:17:22 smithi173 ceph-mon[37000]: pgmap v102: 1 pgs: 1 active+clean; 577 KiB data, 119 MiB used, 357 GiB / 358 GiB avail 2024-05-27T07:17:23.102 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:17:22 smithi173 ceph-mon[37000]: osdmap e27: 4 total, 4 up, 4 in 2024-05-27T07:17:23.190 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:17:22 smithi105 ceph-mon[35977]: pgmap v102: 1 pgs: 1 active+clean; 577 KiB data, 119 MiB used, 357 GiB / 358 GiB avail 2024-05-27T07:17:23.190 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:17:22 smithi105 ceph-mon[35977]: osdmap e27: 4 total, 4 up, 4 in 2024-05-27T07:17:23.229 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:17:22 smithi052 ceph-mon[31955]: pgmap v102: 1 pgs: 1 active+clean; 577 KiB data, 119 MiB used, 357 GiB / 358 GiB avail 2024-05-27T07:17:23.230 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:17:22 smithi052 ceph-mon[31955]: osdmap e27: 4 total, 4 up, 4 in 2024-05-27T07:17:24.102 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:17:23 smithi173 ceph-mon[37000]: osdmap e28: 4 total, 4 up, 4 in 2024-05-27T07:17:24.102 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:17:23 smithi173 ceph-mon[37000]: from='client.24236 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi105:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-05-27T07:17:24.102 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:17:23 smithi173 ceph-mon[37000]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-05-27T07:17:24.102 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:17:23 smithi173 ceph-mon[37000]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-05-27T07:17:24.103 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:17:23 smithi173 ceph-mon[37000]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T07:17:24.131 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:17:23 smithi052 ceph-mon[31955]: osdmap e28: 4 total, 4 up, 4 in 2024-05-27T07:17:24.131 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:17:23 smithi052 ceph-mon[31955]: from='client.24236 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi105:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-05-27T07:17:24.131 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:17:23 smithi052 ceph-mon[31955]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-05-27T07:17:24.131 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:17:23 smithi052 ceph-mon[31955]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-05-27T07:17:24.132 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:17:23 smithi052 ceph-mon[31955]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T07:17:24.190 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:17:23 smithi105 ceph-mon[35977]: osdmap e28: 4 total, 4 up, 4 in 2024-05-27T07:17:24.190 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:17:23 smithi105 ceph-mon[35977]: from='client.24236 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi105:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-05-27T07:17:24.191 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:17:23 smithi105 ceph-mon[35977]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-05-27T07:17:24.191 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:17:23 smithi105 ceph-mon[35977]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-05-27T07:17:24.191 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:17:23 smithi105 ceph-mon[35977]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T07:17:24.940 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:17:24 smithi105 ceph-mon[35977]: pgmap v105: 1 pgs: 1 peering; 577 KiB data, 119 MiB used, 357 GiB / 358 GiB avail 2024-05-27T07:17:25.102 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:17:24 smithi173 ceph-mon[37000]: pgmap v105: 1 pgs: 1 peering; 577 KiB data, 119 MiB used, 357 GiB / 358 GiB avail 2024-05-27T07:17:25.229 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:17:24 smithi052 ceph-mon[31955]: pgmap v105: 1 pgs: 1 peering; 577 KiB data, 119 MiB used, 357 GiB / 358 GiB avail 2024-05-27T07:17:26.102 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:17:25 smithi173 ceph-mon[37000]: from='client.? 172.21.15.105:0/1140061504' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "2592d1cc-1726-4359-83d8-a2e6314b54ec"}]: dispatch 2024-05-27T07:17:26.102 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:17:25 smithi173 ceph-mon[37000]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "2592d1cc-1726-4359-83d8-a2e6314b54ec"}]: dispatch 2024-05-27T07:17:26.102 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:17:25 smithi173 ceph-mon[37000]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "2592d1cc-1726-4359-83d8-a2e6314b54ec"}]': finished 2024-05-27T07:17:26.103 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:17:25 smithi173 ceph-mon[37000]: osdmap e29: 5 total, 4 up, 5 in 2024-05-27T07:17:26.103 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:17:25 smithi173 ceph-mon[37000]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-05-27T07:17:26.190 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:17:25 smithi105 ceph-mon[35977]: from='client.? 172.21.15.105:0/1140061504' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "2592d1cc-1726-4359-83d8-a2e6314b54ec"}]: dispatch 2024-05-27T07:17:26.190 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:17:25 smithi105 ceph-mon[35977]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "2592d1cc-1726-4359-83d8-a2e6314b54ec"}]: dispatch 2024-05-27T07:17:26.190 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:17:25 smithi105 ceph-mon[35977]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "2592d1cc-1726-4359-83d8-a2e6314b54ec"}]': finished 2024-05-27T07:17:26.191 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:17:25 smithi105 ceph-mon[35977]: osdmap e29: 5 total, 4 up, 5 in 2024-05-27T07:17:26.191 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:17:25 smithi105 ceph-mon[35977]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-05-27T07:17:26.230 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:17:25 smithi052 ceph-mon[31955]: from='client.? 172.21.15.105:0/1140061504' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "2592d1cc-1726-4359-83d8-a2e6314b54ec"}]: dispatch 2024-05-27T07:17:26.230 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:17:25 smithi052 ceph-mon[31955]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "2592d1cc-1726-4359-83d8-a2e6314b54ec"}]: dispatch 2024-05-27T07:17:26.230 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:17:25 smithi052 ceph-mon[31955]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "2592d1cc-1726-4359-83d8-a2e6314b54ec"}]': finished 2024-05-27T07:17:26.230 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:17:25 smithi052 ceph-mon[31955]: osdmap e29: 5 total, 4 up, 5 in 2024-05-27T07:17:26.230 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:17:25 smithi052 ceph-mon[31955]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-05-27T07:17:27.102 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:17:26 smithi173 ceph-mon[37000]: pgmap v106: 1 pgs: 1 peering; 577 KiB data, 119 MiB used, 357 GiB / 358 GiB avail 2024-05-27T07:17:27.102 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:17:26 smithi173 ceph-mon[37000]: from='client.? 172.21.15.105:0/2293831320' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-05-27T07:17:27.190 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:17:26 smithi105 ceph-mon[35977]: pgmap v106: 1 pgs: 1 peering; 577 KiB data, 119 MiB used, 357 GiB / 358 GiB avail 2024-05-27T07:17:27.190 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:17:26 smithi105 ceph-mon[35977]: from='client.? 172.21.15.105:0/2293831320' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-05-27T07:17:27.230 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:17:26 smithi052 ceph-mon[31955]: pgmap v106: 1 pgs: 1 peering; 577 KiB data, 119 MiB used, 357 GiB / 358 GiB avail 2024-05-27T07:17:27.230 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:17:26 smithi052 ceph-mon[31955]: from='client.? 172.21.15.105:0/2293831320' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-05-27T07:17:29.102 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:17:28 smithi173 ceph-mon[37000]: pgmap v108: 1 pgs: 1 active+clean; 577 KiB data, 119 MiB used, 357 GiB / 358 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-05-27T07:17:29.190 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:17:28 smithi105 ceph-mon[35977]: pgmap v108: 1 pgs: 1 active+clean; 577 KiB data, 119 MiB used, 357 GiB / 358 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-05-27T07:17:29.230 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:17:28 smithi052 ceph-mon[31955]: pgmap v108: 1 pgs: 1 active+clean; 577 KiB data, 119 MiB used, 357 GiB / 358 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-05-27T07:17:30.865 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:17:30 smithi105 ceph-mon[35977]: pgmap v109: 1 pgs: 1 active+clean; 577 KiB data, 119 MiB used, 357 GiB / 358 GiB avail; 74 KiB/s, 0 objects/s recovering 2024-05-27T07:17:31.102 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:17:30 smithi173 ceph-mon[37000]: pgmap v109: 1 pgs: 1 active+clean; 577 KiB data, 119 MiB used, 357 GiB / 358 GiB avail; 74 KiB/s, 0 objects/s recovering 2024-05-27T07:17:31.230 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:17:30 smithi052 ceph-mon[31955]: pgmap v109: 1 pgs: 1 active+clean; 577 KiB data, 119 MiB used, 357 GiB / 358 GiB avail; 74 KiB/s, 0 objects/s recovering 2024-05-27T07:17:32.940 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:17:32 smithi105 ceph-mon[35977]: pgmap v110: 1 pgs: 1 active+clean; 577 KiB data, 119 MiB used, 357 GiB / 358 GiB avail; 65 KiB/s, 0 objects/s recovering 2024-05-27T07:17:33.102 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:17:32 smithi173 ceph-mon[37000]: pgmap v110: 1 pgs: 1 active+clean; 577 KiB data, 119 MiB used, 357 GiB / 358 GiB avail; 65 KiB/s, 0 objects/s recovering 2024-05-27T07:17:33.230 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:17:32 smithi052 ceph-mon[31955]: pgmap v110: 1 pgs: 1 active+clean; 577 KiB data, 119 MiB used, 357 GiB / 358 GiB avail; 65 KiB/s, 0 objects/s recovering 2024-05-27T07:17:34.102 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:17:33 smithi173 ceph-mon[37000]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2024-05-27T07:17:34.102 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:17:33 smithi173 ceph-mon[37000]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T07:17:34.103 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:17:33 smithi173 ceph-mon[37000]: Deploying daemon osd.4 on smithi105 2024-05-27T07:17:34.190 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:17:33 smithi105 ceph-mon[35977]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2024-05-27T07:17:34.191 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:17:33 smithi105 ceph-mon[35977]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T07:17:34.191 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:17:33 smithi105 ceph-mon[35977]: Deploying daemon osd.4 on smithi105 2024-05-27T07:17:34.230 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:17:33 smithi052 ceph-mon[31955]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2024-05-27T07:17:34.230 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:17:33 smithi052 ceph-mon[31955]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T07:17:34.230 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:17:33 smithi052 ceph-mon[31955]: Deploying daemon osd.4 on smithi105 2024-05-27T07:17:34.940 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:17:34 smithi105 ceph-mon[35977]: pgmap v111: 1 pgs: 1 active+clean; 577 KiB data, 119 MiB used, 357 GiB / 358 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-05-27T07:17:35.102 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:17:34 smithi173 ceph-mon[37000]: pgmap v111: 1 pgs: 1 active+clean; 577 KiB data, 119 MiB used, 357 GiB / 358 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-05-27T07:17:35.230 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:17:34 smithi052 ceph-mon[31955]: pgmap v111: 1 pgs: 1 active+clean; 577 KiB data, 119 MiB used, 357 GiB / 358 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-05-27T07:17:37.102 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:17:36 smithi173 ceph-mon[37000]: pgmap v112: 1 pgs: 1 active+clean; 577 KiB data, 119 MiB used, 357 GiB / 358 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-05-27T07:17:37.190 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:17:36 smithi105 ceph-mon[35977]: pgmap v112: 1 pgs: 1 active+clean; 577 KiB data, 119 MiB used, 357 GiB / 358 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-05-27T07:17:37.230 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:17:36 smithi052 ceph-mon[31955]: pgmap v112: 1 pgs: 1 active+clean; 577 KiB data, 119 MiB used, 357 GiB / 358 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-05-27T07:17:39.102 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:17:38 smithi173 ceph-mon[37000]: pgmap v113: 1 pgs: 1 active+clean; 577 KiB data, 119 MiB used, 357 GiB / 358 GiB avail; 48 KiB/s, 0 objects/s recovering 2024-05-27T07:17:39.102 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:17:38 smithi173 ceph-mon[37000]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' 2024-05-27T07:17:39.103 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:17:38 smithi173 ceph-mon[37000]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' 2024-05-27T07:17:39.103 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:17:38 smithi173 ceph-mon[37000]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-27T07:17:39.190 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:17:38 smithi105 ceph-mon[35977]: pgmap v113: 1 pgs: 1 active+clean; 577 KiB data, 119 MiB used, 357 GiB / 358 GiB avail; 48 KiB/s, 0 objects/s recovering 2024-05-27T07:17:39.190 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:17:38 smithi105 ceph-mon[35977]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' 2024-05-27T07:17:39.190 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:17:38 smithi105 ceph-mon[35977]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' 2024-05-27T07:17:39.190 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:17:38 smithi105 ceph-mon[35977]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-27T07:17:39.229 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:17:38 smithi052 ceph-mon[31955]: pgmap v113: 1 pgs: 1 active+clean; 577 KiB data, 119 MiB used, 357 GiB / 358 GiB avail; 48 KiB/s, 0 objects/s recovering 2024-05-27T07:17:39.230 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:17:38 smithi052 ceph-mon[31955]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' 2024-05-27T07:17:39.230 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:17:38 smithi052 ceph-mon[31955]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' 2024-05-27T07:17:39.230 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:17:38 smithi052 ceph-mon[31955]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-27T07:17:41.096 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:17:40 smithi105 ceph-mon[35977]: pgmap v114: 1 pgs: 1 active+clean; 577 KiB data, 119 MiB used, 357 GiB / 358 GiB avail 2024-05-27T07:17:41.102 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:17:40 smithi173 ceph-mon[37000]: pgmap v114: 1 pgs: 1 active+clean; 577 KiB data, 119 MiB used, 357 GiB / 358 GiB avail 2024-05-27T07:17:41.229 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:17:40 smithi052 ceph-mon[31955]: pgmap v114: 1 pgs: 1 active+clean; 577 KiB data, 119 MiB used, 357 GiB / 358 GiB avail 2024-05-27T07:17:42.230 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:17:41 smithi052 ceph-mon[31955]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' 2024-05-27T07:17:42.230 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:17:41 smithi052 ceph-mon[31955]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' 2024-05-27T07:17:42.230 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:17:41 smithi052 ceph-mon[31955]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T07:17:42.230 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:17:41 smithi052 ceph-mon[31955]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-27T07:17:42.230 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:17:41 smithi052 ceph-mon[31955]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' 2024-05-27T07:17:42.352 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:17:41 smithi173 ceph-mon[37000]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' 2024-05-27T07:17:42.352 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:17:41 smithi173 ceph-mon[37000]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' 2024-05-27T07:17:42.352 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:17:41 smithi173 ceph-mon[37000]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T07:17:42.352 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:17:41 smithi173 ceph-mon[37000]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-27T07:17:42.352 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:17:41 smithi173 ceph-mon[37000]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' 2024-05-27T07:17:42.359 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:17:41 smithi105 ceph-mon[35977]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' 2024-05-27T07:17:42.359 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:17:41 smithi105 ceph-mon[35977]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' 2024-05-27T07:17:42.360 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:17:41 smithi105 ceph-mon[35977]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T07:17:42.360 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:17:41 smithi105 ceph-mon[35977]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-27T07:17:42.360 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:17:41 smithi105 ceph-mon[35977]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' 2024-05-27T07:17:42.439 INFO:teuthology.orchestra.run.smithi105.stderr:2024-05-27T07:17:42.448+0000 7f46ff7fe700 1 -- 172.21.15.105:0/8001890 <== mgr.14152 v2:172.21.15.52:6800/3513075805 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+37 (secure 0 0 0) 0x7f4708063bf0 con 0x7f46f40b2880 2024-05-27T07:17:42.439 INFO:teuthology.orchestra.run.smithi105.stdout:Created osd(s) 4 on host 'smithi105' 2024-05-27T07:17:42.441 INFO:teuthology.orchestra.run.smithi105.stderr:2024-05-27T07:17:42.450+0000 7f470cd6a700 1 -- 172.21.15.105:0/8001890 >> [v2:172.21.15.52:6800/3513075805,v1:172.21.15.52:6801/3513075805] conn(0x7f46f40b2880 msgr2=0x7f46f40b4d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:17:42.441 INFO:teuthology.orchestra.run.smithi105.stderr:2024-05-27T07:17:42.450+0000 7f470cd6a700 1 --2- 172.21.15.105:0/8001890 >> [v2:172.21.15.52:6800/3513075805,v1:172.21.15.52:6801/3513075805] conn(0x7f46f40b2880 0x7f46f40b4d90 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f470806c020 tx=0x7f46f8009190 comp rx=0 tx=0).stop 2024-05-27T07:17:42.441 INFO:teuthology.orchestra.run.smithi105.stderr:2024-05-27T07:17:42.450+0000 7f470cd6a700 1 -- 172.21.15.105:0/8001890 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f470810a0f0 msgr2=0x7f47081a1df0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:17:42.441 INFO:teuthology.orchestra.run.smithi105.stderr:2024-05-27T07:17:42.450+0000 7f470cd6a700 1 --2- 172.21.15.105:0/8001890 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f470810a0f0 0x7f47081a1df0 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7f4700009fd0 tx=0x7f4700004da0 comp rx=0 tx=0).stop 2024-05-27T07:17:42.441 INFO:teuthology.orchestra.run.smithi105.stderr:2024-05-27T07:17:42.450+0000 7f470cd6a700 1 -- 172.21.15.105:0/8001890 shutdown_connections 2024-05-27T07:17:42.442 INFO:teuthology.orchestra.run.smithi105.stderr:2024-05-27T07:17:42.450+0000 7f470cd6a700 1 --2- 172.21.15.105:0/8001890 >> [v2:172.21.15.52:6800/3513075805,v1:172.21.15.52:6801/3513075805] conn(0x7f46f40b2880 0x7f46f40b4d90 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:17:42.442 INFO:teuthology.orchestra.run.smithi105.stderr:2024-05-27T07:17:42.450+0000 7f470cd6a700 1 --2- 172.21.15.105:0/8001890 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f470810a0f0 0x7f47081a1df0 unknown :-1 s=CLOSED pgs=18 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:17:42.442 INFO:teuthology.orchestra.run.smithi105.stderr:2024-05-27T07:17:42.450+0000 7f470cd6a700 1 --2- 172.21.15.105:0/8001890 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f4708104e30 0x7f470819a910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:17:42.442 INFO:teuthology.orchestra.run.smithi105.stderr:2024-05-27T07:17:42.450+0000 7f470cd6a700 1 --2- 172.21.15.105:0/8001890 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f4708107790 0x7f470819ae50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:17:42.442 INFO:teuthology.orchestra.run.smithi105.stderr:2024-05-27T07:17:42.450+0000 7f470cd6a700 1 -- 172.21.15.105:0/8001890 >> 172.21.15.105:0/8001890 conn(0x7f47080fe870 msgr2=0x7f4708105900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:17:42.442 INFO:teuthology.orchestra.run.smithi105.stderr:2024-05-27T07:17:42.451+0000 7f470cd6a700 1 -- 172.21.15.105:0/8001890 shutdown_connections 2024-05-27T07:17:42.442 INFO:teuthology.orchestra.run.smithi105.stderr:2024-05-27T07:17:42.451+0000 7f470cd6a700 1 -- 172.21.15.105:0/8001890 wait complete. 2024-05-27T07:17:43.097 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:17:42 smithi105 ceph-mon[35977]: pgmap v115: 1 pgs: 1 active+clean; 577 KiB data, 119 MiB used, 357 GiB / 358 GiB avail 2024-05-27T07:17:43.098 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:17:42 smithi105 ceph-mon[35977]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' 2024-05-27T07:17:43.098 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:17:42 smithi105 ceph-mon[35977]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' 2024-05-27T07:17:43.098 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:17:42 smithi105 ceph-mon[35977]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-27T07:17:43.098 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:17:42 smithi105 ceph-mon[35977]: from='osd.4 [v2:172.21.15.105:6808/3441134818,v1:172.21.15.105:6809/3441134818]' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-05-27T07:17:43.098 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:17:42 smithi105 ceph-mon[35977]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-05-27T07:17:43.123 DEBUG:teuthology.orchestra.run.smithi105:osd.4> sudo journalctl -f -n 0 -u ceph-73df2636-1bf8-11ef-bc9b-c7b262605968@osd.4.service 2024-05-27T07:17:43.125 INFO:tasks.cephadm:Deploying osd.5 on smithi173 with /dev/vg_nvme/lv_3... 2024-05-27T07:17:43.126 DEBUG:teuthology.orchestra.run.smithi173:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:38f9d900c83c63799fbdbe61acc9a11b0d3554a6 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 73df2636-1bf8-11ef-bc9b-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_3 2024-05-27T07:17:43.352 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:17:42 smithi173 ceph-mon[37000]: pgmap v115: 1 pgs: 1 active+clean; 577 KiB data, 119 MiB used, 357 GiB / 358 GiB avail 2024-05-27T07:17:43.352 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:17:42 smithi173 ceph-mon[37000]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' 2024-05-27T07:17:43.352 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:17:42 smithi173 ceph-mon[37000]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' 2024-05-27T07:17:43.352 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:17:42 smithi173 ceph-mon[37000]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-27T07:17:43.352 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:17:42 smithi173 ceph-mon[37000]: from='osd.4 [v2:172.21.15.105:6808/3441134818,v1:172.21.15.105:6809/3441134818]' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-05-27T07:17:43.352 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:17:42 smithi173 ceph-mon[37000]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-05-27T07:17:43.370 INFO:teuthology.orchestra.run.smithi173.stderr:Inferring config /var/lib/ceph/73df2636-1bf8-11ef-bc9b-c7b262605968/mon.c/config 2024-05-27T07:17:43.480 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:17:42 smithi052 ceph-mon[31955]: pgmap v115: 1 pgs: 1 active+clean; 577 KiB data, 119 MiB used, 357 GiB / 358 GiB avail 2024-05-27T07:17:43.480 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:17:42 smithi052 ceph-mon[31955]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' 2024-05-27T07:17:43.480 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:17:42 smithi052 ceph-mon[31955]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' 2024-05-27T07:17:43.480 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:17:42 smithi052 ceph-mon[31955]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-27T07:17:43.480 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:17:42 smithi052 ceph-mon[31955]: from='osd.4 [v2:172.21.15.105:6808/3441134818,v1:172.21.15.105:6809/3441134818]' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-05-27T07:17:43.480 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:17:42 smithi052 ceph-mon[31955]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-05-27T07:17:44.636 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:17:44 smithi173 ceph-mon[37000]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2024-05-27T07:17:44.636 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:17:44 smithi173 ceph-mon[37000]: osdmap e30: 5 total, 4 up, 5 in 2024-05-27T07:17:44.636 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:17:44 smithi173 ceph-mon[37000]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-05-27T07:17:44.636 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:17:44 smithi173 ceph-mon[37000]: from='osd.4 [v2:172.21.15.105:6808/3441134818,v1:172.21.15.105:6809/3441134818]' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi105", "root=default"]}]: dispatch 2024-05-27T07:17:44.636 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:17:44 smithi173 ceph-mon[37000]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi105", "root=default"]}]: dispatch 2024-05-27T07:17:44.636 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:17:44 smithi173 ceph-mon[37000]: pgmap v117: 1 pgs: 1 active+clean; 577 KiB data, 119 MiB used, 357 GiB / 358 GiB avail 2024-05-27T07:17:44.661 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:17:44 smithi105 ceph-mon[35977]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2024-05-27T07:17:44.661 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:17:44 smithi105 ceph-mon[35977]: osdmap e30: 5 total, 4 up, 5 in 2024-05-27T07:17:44.661 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:17:44 smithi105 ceph-mon[35977]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-05-27T07:17:44.661 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:17:44 smithi105 ceph-mon[35977]: from='osd.4 [v2:172.21.15.105:6808/3441134818,v1:172.21.15.105:6809/3441134818]' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi105", "root=default"]}]: dispatch 2024-05-27T07:17:44.661 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:17:44 smithi105 ceph-mon[35977]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi105", "root=default"]}]: dispatch 2024-05-27T07:17:44.661 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:17:44 smithi105 ceph-mon[35977]: pgmap v117: 1 pgs: 1 active+clean; 577 KiB data, 119 MiB used, 357 GiB / 358 GiB avail 2024-05-27T07:17:44.662 INFO:journalctl@ceph.osd.4.smithi105.stdout:May 27 07:17:44 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-osd-4[47069]: 2024-05-27T07:17:44.419+0000 7f8a412e6700 -1 osd.4 0 waiting for initial osdmap 2024-05-27T07:17:44.662 INFO:journalctl@ceph.osd.4.smithi105.stdout:May 27 07:17:44 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-osd-4[47069]: 2024-05-27T07:17:44.427+0000 7f8a3d09b700 -1 osd.4 31 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-05-27T07:17:44.730 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:17:44 smithi052 ceph-mon[31955]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2024-05-27T07:17:44.730 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:17:44 smithi052 ceph-mon[31955]: osdmap e30: 5 total, 4 up, 5 in 2024-05-27T07:17:44.730 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:17:44 smithi052 ceph-mon[31955]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-05-27T07:17:44.730 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:17:44 smithi052 ceph-mon[31955]: from='osd.4 [v2:172.21.15.105:6808/3441134818,v1:172.21.15.105:6809/3441134818]' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi105", "root=default"]}]: dispatch 2024-05-27T07:17:44.730 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:17:44 smithi052 ceph-mon[31955]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi105", "root=default"]}]: dispatch 2024-05-27T07:17:44.730 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:17:44 smithi052 ceph-mon[31955]: pgmap v117: 1 pgs: 1 active+clean; 577 KiB data, 119 MiB used, 357 GiB / 358 GiB avail 2024-05-27T07:17:45.203 INFO:teuthology.orchestra.run.smithi173.stderr:--> Zapping: /dev/vg_nvme/lv_3 2024-05-27T07:17:45.203 INFO:teuthology.orchestra.run.smithi173.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_3 bs=1M count=10 conv=fsync 2024-05-27T07:17:45.203 INFO:teuthology.orchestra.run.smithi173.stderr: stderr: 10+0 records in 2024-05-27T07:17:45.203 INFO:teuthology.orchestra.run.smithi173.stderr:10+0 records out 2024-05-27T07:17:45.203 INFO:teuthology.orchestra.run.smithi173.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0112894 s, 929 MB/s 2024-05-27T07:17:45.203 INFO:teuthology.orchestra.run.smithi173.stderr:--> Zapping successful for: 2024-05-27T07:17:45.690 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:17:45 smithi105 ceph-mon[35977]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi105", "root=default"]}]': finished 2024-05-27T07:17:45.691 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:17:45 smithi105 ceph-mon[35977]: osdmap e31: 5 total, 4 up, 5 in 2024-05-27T07:17:45.691 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:17:45 smithi105 ceph-mon[35977]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-05-27T07:17:45.691 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:17:45 smithi105 ceph-mon[35977]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-05-27T07:17:45.729 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:17:45 smithi052 ceph-mon[31955]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi105", "root=default"]}]': finished 2024-05-27T07:17:45.730 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:17:45 smithi052 ceph-mon[31955]: osdmap e31: 5 total, 4 up, 5 in 2024-05-27T07:17:45.730 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:17:45 smithi052 ceph-mon[31955]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-05-27T07:17:45.730 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:17:45 smithi052 ceph-mon[31955]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-05-27T07:17:45.850 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:17:45 smithi173 ceph-mon[37000]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi105", "root=default"]}]': finished 2024-05-27T07:17:45.851 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:17:45 smithi173 ceph-mon[37000]: osdmap e31: 5 total, 4 up, 5 in 2024-05-27T07:17:45.851 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:17:45 smithi173 ceph-mon[37000]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-05-27T07:17:45.851 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:17:45 smithi173 ceph-mon[37000]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-05-27T07:17:45.879 DEBUG:teuthology.orchestra.run.smithi173:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:38f9d900c83c63799fbdbe61acc9a11b0d3554a6 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 73df2636-1bf8-11ef-bc9b-c7b262605968 -- ceph orch daemon add osd smithi173:vg_nvme/lv_3 2024-05-27T07:17:46.113 INFO:teuthology.orchestra.run.smithi173.stderr:Inferring config /var/lib/ceph/73df2636-1bf8-11ef-bc9b-c7b262605968/mon.c/config 2024-05-27T07:17:46.496 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:17:46 smithi173 ceph-mon[37000]: purged_snaps scrub starts 2024-05-27T07:17:46.496 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:17:46 smithi173 ceph-mon[37000]: purged_snaps scrub ok 2024-05-27T07:17:46.496 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:17:46 smithi173 ceph-mon[37000]: osd.4 [v2:172.21.15.105:6808/3441134818,v1:172.21.15.105:6809/3441134818] boot 2024-05-27T07:17:46.496 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:17:46 smithi173 ceph-mon[37000]: osdmap e32: 5 total, 5 up, 5 in 2024-05-27T07:17:46.496 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:17:46 smithi173 ceph-mon[37000]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-05-27T07:17:46.496 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:17:46 smithi173 ceph-mon[37000]: pgmap v120: 1 pgs: 1 active+clean; 577 KiB data, 145 MiB used, 447 GiB / 447 GiB avail 2024-05-27T07:17:46.690 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:17:46 smithi105 ceph-mon[35977]: purged_snaps scrub starts 2024-05-27T07:17:46.690 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:17:46 smithi105 ceph-mon[35977]: purged_snaps scrub ok 2024-05-27T07:17:46.691 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:17:46 smithi105 ceph-mon[35977]: osd.4 [v2:172.21.15.105:6808/3441134818,v1:172.21.15.105:6809/3441134818] boot 2024-05-27T07:17:46.691 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:17:46 smithi105 ceph-mon[35977]: osdmap e32: 5 total, 5 up, 5 in 2024-05-27T07:17:46.691 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:17:46 smithi105 ceph-mon[35977]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-05-27T07:17:46.691 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:17:46 smithi105 ceph-mon[35977]: pgmap v120: 1 pgs: 1 active+clean; 577 KiB data, 145 MiB used, 447 GiB / 447 GiB avail 2024-05-27T07:17:46.730 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:17:46 smithi052 ceph-mon[31955]: purged_snaps scrub starts 2024-05-27T07:17:46.730 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:17:46 smithi052 ceph-mon[31955]: purged_snaps scrub ok 2024-05-27T07:17:46.730 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:17:46 smithi052 ceph-mon[31955]: osd.4 [v2:172.21.15.105:6808/3441134818,v1:172.21.15.105:6809/3441134818] boot 2024-05-27T07:17:46.730 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:17:46 smithi052 ceph-mon[31955]: osdmap e32: 5 total, 5 up, 5 in 2024-05-27T07:17:46.730 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:17:46 smithi052 ceph-mon[31955]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-05-27T07:17:46.730 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:17:46 smithi052 ceph-mon[31955]: pgmap v120: 1 pgs: 1 active+clean; 577 KiB data, 145 MiB used, 447 GiB / 447 GiB avail 2024-05-27T07:17:47.516 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:17:47 smithi173 ceph-mon[37000]: osdmap e33: 5 total, 5 up, 5 in 2024-05-27T07:17:47.516 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:17:47 smithi173 ceph-mon[37000]: Detected new or changed devices on smithi105 2024-05-27T07:17:47.516 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:17:47 smithi173 ceph-mon[37000]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' 2024-05-27T07:17:47.516 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:17:47 smithi173 ceph-mon[37000]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' 2024-05-27T07:17:47.516 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:17:47 smithi173 ceph-mon[37000]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-05-27T07:17:47.516 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:17:47 smithi173 ceph-mon[37000]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-05-27T07:17:47.516 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:17:47 smithi173 ceph-mon[37000]: Adjusting osd_memory_target on smithi105 to 8544M 2024-05-27T07:17:47.516 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:17:47 smithi173 ceph-mon[37000]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' 2024-05-27T07:17:47.516 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:17:47 smithi173 ceph-mon[37000]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T07:17:47.516 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:17:47 smithi173 ceph-mon[37000]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-27T07:17:47.517 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:17:47 smithi173 ceph-mon[37000]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' 2024-05-27T07:17:47.690 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:17:47 smithi105 ceph-mon[35977]: osdmap e33: 5 total, 5 up, 5 in 2024-05-27T07:17:47.690 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:17:47 smithi105 ceph-mon[35977]: Detected new or changed devices on smithi105 2024-05-27T07:17:47.691 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:17:47 smithi105 ceph-mon[35977]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' 2024-05-27T07:17:47.694 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:17:47 smithi105 ceph-mon[35977]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' 2024-05-27T07:17:47.694 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:17:47 smithi105 ceph-mon[35977]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-05-27T07:17:47.694 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:17:47 smithi105 ceph-mon[35977]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-05-27T07:17:47.694 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:17:47 smithi105 ceph-mon[35977]: Adjusting osd_memory_target on smithi105 to 8544M 2024-05-27T07:17:47.694 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:17:47 smithi105 ceph-mon[35977]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' 2024-05-27T07:17:47.694 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:17:47 smithi105 ceph-mon[35977]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T07:17:47.694 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:17:47 smithi105 ceph-mon[35977]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-27T07:17:47.694 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:17:47 smithi105 ceph-mon[35977]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' 2024-05-27T07:17:47.730 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:17:47 smithi052 ceph-mon[31955]: osdmap e33: 5 total, 5 up, 5 in 2024-05-27T07:17:47.730 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:17:47 smithi052 ceph-mon[31955]: Detected new or changed devices on smithi105 2024-05-27T07:17:47.730 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:17:47 smithi052 ceph-mon[31955]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' 2024-05-27T07:17:47.730 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:17:47 smithi052 ceph-mon[31955]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' 2024-05-27T07:17:47.730 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:17:47 smithi052 ceph-mon[31955]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-05-27T07:17:47.730 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:17:47 smithi052 ceph-mon[31955]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-05-27T07:17:47.730 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:17:47 smithi052 ceph-mon[31955]: Adjusting osd_memory_target on smithi105 to 8544M 2024-05-27T07:17:47.730 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:17:47 smithi052 ceph-mon[31955]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' 2024-05-27T07:17:47.730 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:17:47 smithi052 ceph-mon[31955]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T07:17:47.731 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:17:47 smithi052 ceph-mon[31955]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-27T07:17:47.731 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:17:47 smithi052 ceph-mon[31955]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' 2024-05-27T07:17:47.740 INFO:teuthology.orchestra.run.smithi173.stderr:2024-05-27T07:17:47.750+0000 7fe701b47700 1 -- 172.21.15.173:0/4026032862 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fe6fc10a100 msgr2=0x7fe6fc10c520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:17:47.740 INFO:teuthology.orchestra.run.smithi173.stderr:2024-05-27T07:17:47.750+0000 7fe701b47700 1 --2- 172.21.15.173:0/4026032862 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fe6fc10a100 0x7fe6fc10c520 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7fe6f400b0a0 tx=0x7fe6f400b3b0 comp rx=0 tx=0).stop 2024-05-27T07:17:47.741 INFO:teuthology.orchestra.run.smithi173.stderr:2024-05-27T07:17:47.750+0000 7fe701b47700 1 -- 172.21.15.173:0/4026032862 shutdown_connections 2024-05-27T07:17:47.741 INFO:teuthology.orchestra.run.smithi173.stderr:2024-05-27T07:17:47.750+0000 7fe701b47700 1 --2- 172.21.15.173:0/4026032862 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fe6fc10a100 0x7fe6fc10c520 unknown :-1 s=CLOSED pgs=20 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:17:47.741 INFO:teuthology.orchestra.run.smithi173.stderr:2024-05-27T07:17:47.750+0000 7fe701b47700 1 --2- 172.21.15.173:0/4026032862 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fe6fc104e40 0x7fe6fc107260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:17:47.741 INFO:teuthology.orchestra.run.smithi173.stderr:2024-05-27T07:17:47.750+0000 7fe701b47700 1 --2- 172.21.15.173:0/4026032862 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fe6fc1077a0 0x7fe6fc109bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:17:47.741 INFO:teuthology.orchestra.run.smithi173.stderr:2024-05-27T07:17:47.750+0000 7fe701b47700 1 -- 172.21.15.173:0/4026032862 >> 172.21.15.173:0/4026032862 conn(0x7fe6fc0fe880 msgr2=0x7fe6fc100d30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:17:47.741 INFO:teuthology.orchestra.run.smithi173.stderr:2024-05-27T07:17:47.750+0000 7fe701b47700 1 -- 172.21.15.173:0/4026032862 shutdown_connections 2024-05-27T07:17:47.741 INFO:teuthology.orchestra.run.smithi173.stderr:2024-05-27T07:17:47.750+0000 7fe701b47700 1 -- 172.21.15.173:0/4026032862 wait complete. 2024-05-27T07:17:47.743 INFO:teuthology.orchestra.run.smithi173.stderr:2024-05-27T07:17:47.751+0000 7fe701b47700 1 Processor -- start 2024-05-27T07:17:47.743 INFO:teuthology.orchestra.run.smithi173.stderr:2024-05-27T07:17:47.751+0000 7fe701b47700 1 -- start start 2024-05-27T07:17:47.743 INFO:teuthology.orchestra.run.smithi173.stderr:2024-05-27T07:17:47.751+0000 7fe701b47700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fe6fc104e40 0x7fe6fc19a8c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:17:47.743 INFO:teuthology.orchestra.run.smithi173.stderr:2024-05-27T07:17:47.751+0000 7fe701b47700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fe6fc1077a0 0x7fe6fc19ae00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:17:47.743 INFO:teuthology.orchestra.run.smithi173.stderr:2024-05-27T07:17:47.751+0000 7fe701b47700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fe6fc10a100 0x7fe6fc1a1da0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:17:47.744 INFO:teuthology.orchestra.run.smithi173.stderr:2024-05-27T07:17:47.751+0000 7fe701b47700 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x7fe6fc10fe10 con 0x7fe6fc1077a0 2024-05-27T07:17:47.744 INFO:teuthology.orchestra.run.smithi173.stderr:2024-05-27T07:17:47.751+0000 7fe701b47700 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7fe6fc10fc90 con 0x7fe6fc104e40 2024-05-27T07:17:47.744 INFO:teuthology.orchestra.run.smithi173.stderr:2024-05-27T07:17:47.751+0000 7fe701b47700 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7fe6fc10ff90 con 0x7fe6fc10a100 2024-05-27T07:17:47.744 INFO:teuthology.orchestra.run.smithi173.stderr:2024-05-27T07:17:47.751+0000 7fe6fbfff700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fe6fc10a100 0x7fe6fc1a1da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:17:47.744 INFO:teuthology.orchestra.run.smithi173.stderr:2024-05-27T07:17:47.751+0000 7fe6fbfff700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fe6fc10a100 0x7fe6fc1a1da0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.173:3300/0 says I am v2:172.21.15.173:51154/0 (socket says 172.21.15.173:51154) 2024-05-27T07:17:47.744 INFO:teuthology.orchestra.run.smithi173.stderr:2024-05-27T07:17:47.751+0000 7fe6fbfff700 1 -- 172.21.15.173:0/1700888403 learned_addr learned my addr 172.21.15.173:0/1700888403 (peer_addr_for_me v2:172.21.15.173:0/0) 2024-05-27T07:17:47.744 INFO:teuthology.orchestra.run.smithi173.stderr:2024-05-27T07:17:47.751+0000 7fe6faffd700 1 --2- 172.21.15.173:0/1700888403 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fe6fc1077a0 0x7fe6fc19ae00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:17:47.744 INFO:teuthology.orchestra.run.smithi173.stderr:2024-05-27T07:17:47.751+0000 7fe6fb7fe700 1 --2- 172.21.15.173:0/1700888403 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fe6fc104e40 0x7fe6fc19a8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:17:47.744 INFO:teuthology.orchestra.run.smithi173.stderr:2024-05-27T07:17:47.752+0000 7fe6fbfff700 1 -- 172.21.15.173:0/1700888403 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fe6fc104e40 msgr2=0x7fe6fc19a8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:17:47.744 INFO:teuthology.orchestra.run.smithi173.stderr:2024-05-27T07:17:47.752+0000 7fe6fbfff700 1 --2- 172.21.15.173:0/1700888403 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fe6fc104e40 0x7fe6fc19a8c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:17:47.744 INFO:teuthology.orchestra.run.smithi173.stderr:2024-05-27T07:17:47.752+0000 7fe6fbfff700 1 -- 172.21.15.173:0/1700888403 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fe6fc1077a0 msgr2=0x7fe6fc19ae00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:17:47.745 INFO:teuthology.orchestra.run.smithi173.stderr:2024-05-27T07:17:47.752+0000 7fe6fbfff700 1 --2- 172.21.15.173:0/1700888403 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fe6fc1077a0 0x7fe6fc19ae00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:17:47.745 INFO:teuthology.orchestra.run.smithi173.stderr:2024-05-27T07:17:47.752+0000 7fe6fbfff700 1 -- 172.21.15.173:0/1700888403 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe6fc1a2420 con 0x7fe6fc10a100 2024-05-27T07:17:47.745 INFO:teuthology.orchestra.run.smithi173.stderr:2024-05-27T07:17:47.752+0000 7fe6fb7fe700 1 --2- 172.21.15.173:0/1700888403 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fe6fc104e40 0x7fe6fc19a8c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-27T07:17:47.745 INFO:teuthology.orchestra.run.smithi173.stderr:2024-05-27T07:17:47.752+0000 7fe6fbfff700 1 --2- 172.21.15.173:0/1700888403 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fe6fc10a100 0x7fe6fc1a1da0 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7fe6f40062a0 tx=0x7fe6f4004e40 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:17:47.745 INFO:teuthology.orchestra.run.smithi173.stderr:2024-05-27T07:17:47.752+0000 7fe6f8ff9700 1 -- 172.21.15.173:0/1700888403 <== mon.1 v2:172.21.15.173:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe6f4021070 con 0x7fe6fc10a100 2024-05-27T07:17:47.745 INFO:teuthology.orchestra.run.smithi173.stderr:2024-05-27T07:17:47.752+0000 7fe6f8ff9700 1 -- 172.21.15.173:0/1700888403 <== mon.1 v2:172.21.15.173:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fe6f4007c80 con 0x7fe6fc10a100 2024-05-27T07:17:47.745 INFO:teuthology.orchestra.run.smithi173.stderr:2024-05-27T07:17:47.752+0000 7fe701b47700 1 -- 172.21.15.173:0/1700888403 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe6fc1a26b0 con 0x7fe6fc10a100 2024-05-27T07:17:47.745 INFO:teuthology.orchestra.run.smithi173.stderr:2024-05-27T07:17:47.752+0000 7fe6f8ff9700 1 -- 172.21.15.173:0/1700888403 <== mon.1 v2:172.21.15.173:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe6f4004850 con 0x7fe6fc10a100 2024-05-27T07:17:47.745 INFO:teuthology.orchestra.run.smithi173.stderr:2024-05-27T07:17:47.752+0000 7fe701b47700 1 -- 172.21.15.173:0/1700888403 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe6fc1a2cc0 con 0x7fe6fc10a100 2024-05-27T07:17:47.745 INFO:teuthology.orchestra.run.smithi173.stderr:2024-05-27T07:17:47.753+0000 7fe701b47700 1 -- 172.21.15.173:0/1700888403 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe6fc0512b0 con 0x7fe6fc10a100 2024-05-27T07:17:47.746 INFO:teuthology.orchestra.run.smithi173.stderr:2024-05-27T07:17:47.755+0000 7fe6f8ff9700 1 -- 172.21.15.173:0/1700888403 <== mon.1 v2:172.21.15.173:3300/0 4 ==== mgrmap(e 16) ==== 149652+0+0 (secure 0 0 0) 0x7fe6f4003a00 con 0x7fe6fc10a100 2024-05-27T07:17:47.746 INFO:teuthology.orchestra.run.smithi173.stderr:2024-05-27T07:17:47.755+0000 7fe6f8ff9700 1 --2- 172.21.15.173:0/1700888403 >> [v2:172.21.15.52:6800/3513075805,v1:172.21.15.52:6801/3513075805] conn(0x7fe6e80b2900 0x7fe6e80b4e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:17:47.746 INFO:teuthology.orchestra.run.smithi173.stderr:2024-05-27T07:17:47.755+0000 7fe6f8ff9700 1 -- 172.21.15.173:0/1700888403 <== mon.1 v2:172.21.15.173:3300/0 5 ==== osd_map(33..33 src has 1..33) ==== 4101+0+0 (secure 0 0 0) 0x7fe6f400f020 con 0x7fe6fc10a100 2024-05-27T07:17:47.746 INFO:teuthology.orchestra.run.smithi173.stderr:2024-05-27T07:17:47.756+0000 7fe6fb7fe700 1 --2- 172.21.15.173:0/1700888403 >> [v2:172.21.15.52:6800/3513075805,v1:172.21.15.52:6801/3513075805] conn(0x7fe6e80b2900 0x7fe6e80b4e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:17:47.747 INFO:teuthology.orchestra.run.smithi173.stderr:2024-05-27T07:17:47.756+0000 7fe6fb7fe700 1 --2- 172.21.15.173:0/1700888403 >> [v2:172.21.15.52:6800/3513075805,v1:172.21.15.52:6801/3513075805] conn(0x7fe6e80b2900 0x7fe6e80b4e10 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7fe6fc076400 tx=0x7fe6e4008040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:17:47.749 INFO:teuthology.orchestra.run.smithi173.stderr:2024-05-27T07:17:47.759+0000 7fe6f8ff9700 1 -- 172.21.15.173:0/1700888403 <== mon.1 v2:172.21.15.173:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191963 (secure 0 0 0) 0x7fe6f402a020 con 0x7fe6fc10a100 2024-05-27T07:17:47.961 INFO:teuthology.orchestra.run.smithi173.stderr:2024-05-27T07:17:47.971+0000 7fe701b47700 1 -- 172.21.15.173:0/1700888403 --> [v2:172.21.15.52:6800/3513075805,v1:172.21.15.52:6801/3513075805] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi173:vg_nvme/lv_3", "target": ["mon-mgr", ""]}) -- 0x7fe6fc063bf0 con 0x7fe6e80b2900 2024-05-27T07:17:48.684 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:17:48 smithi173 ceph-mon[37000]: pgmap v122: 1 pgs: 1 active+clean; 577 KiB data, 146 MiB used, 447 GiB / 447 GiB avail 2024-05-27T07:17:48.684 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:17:48 smithi173 ceph-mon[37000]: from='client.24238 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi173:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-05-27T07:17:48.684 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:17:48 smithi173 ceph-mon[37000]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-05-27T07:17:48.684 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:17:48 smithi173 ceph-mon[37000]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-05-27T07:17:48.684 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:17:48 smithi173 ceph-mon[37000]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T07:17:48.690 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:17:48 smithi105 ceph-mon[35977]: pgmap v122: 1 pgs: 1 active+clean; 577 KiB data, 146 MiB used, 447 GiB / 447 GiB avail 2024-05-27T07:17:48.690 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:17:48 smithi105 ceph-mon[35977]: from='client.24238 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi173:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-05-27T07:17:48.691 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:17:48 smithi105 ceph-mon[35977]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-05-27T07:17:48.691 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:17:48 smithi105 ceph-mon[35977]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-05-27T07:17:48.691 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:17:48 smithi105 ceph-mon[35977]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T07:17:48.730 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:17:48 smithi052 ceph-mon[31955]: pgmap v122: 1 pgs: 1 active+clean; 577 KiB data, 146 MiB used, 447 GiB / 447 GiB avail 2024-05-27T07:17:48.730 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:17:48 smithi052 ceph-mon[31955]: from='client.24238 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi173:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-05-27T07:17:48.730 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:17:48 smithi052 ceph-mon[31955]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-05-27T07:17:48.730 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:17:48 smithi052 ceph-mon[31955]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-05-27T07:17:48.730 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:17:48 smithi052 ceph-mon[31955]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T07:17:50.940 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:17:50 smithi105 ceph-mon[35977]: pgmap v123: 1 pgs: 1 active+clean; 577 KiB data, 146 MiB used, 447 GiB / 447 GiB avail 2024-05-27T07:17:50.980 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:17:50 smithi052 ceph-mon[31955]: pgmap v123: 1 pgs: 1 active+clean; 577 KiB data, 146 MiB used, 447 GiB / 447 GiB avail 2024-05-27T07:17:51.102 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:17:50 smithi173 ceph-mon[37000]: pgmap v123: 1 pgs: 1 active+clean; 577 KiB data, 146 MiB used, 447 GiB / 447 GiB avail 2024-05-27T07:17:51.940 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:17:51 smithi105 ceph-mon[35977]: from='client.? 172.21.15.173:0/3303699834' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "27e98ee2-4a92-4972-b2a3-959d0ed90981"}]: dispatch 2024-05-27T07:17:51.941 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:17:51 smithi105 ceph-mon[35977]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "27e98ee2-4a92-4972-b2a3-959d0ed90981"}]: dispatch 2024-05-27T07:17:51.941 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:17:51 smithi105 ceph-mon[35977]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "27e98ee2-4a92-4972-b2a3-959d0ed90981"}]': finished 2024-05-27T07:17:51.941 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:17:51 smithi105 ceph-mon[35977]: osdmap e34: 6 total, 5 up, 6 in 2024-05-27T07:17:51.941 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:17:51 smithi105 ceph-mon[35977]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-05-27T07:17:51.941 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:17:51 smithi105 ceph-mon[35977]: from='client.? 172.21.15.173:0/1987703770' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-05-27T07:17:51.979 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:17:51 smithi052 ceph-mon[31955]: from='client.? 172.21.15.173:0/3303699834' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "27e98ee2-4a92-4972-b2a3-959d0ed90981"}]: dispatch 2024-05-27T07:17:51.980 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:17:51 smithi052 ceph-mon[31955]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "27e98ee2-4a92-4972-b2a3-959d0ed90981"}]: dispatch 2024-05-27T07:17:51.980 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:17:51 smithi052 ceph-mon[31955]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "27e98ee2-4a92-4972-b2a3-959d0ed90981"}]': finished 2024-05-27T07:17:51.980 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:17:51 smithi052 ceph-mon[31955]: osdmap e34: 6 total, 5 up, 6 in 2024-05-27T07:17:51.980 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:17:51 smithi052 ceph-mon[31955]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-05-27T07:17:51.980 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:17:51 smithi052 ceph-mon[31955]: from='client.? 172.21.15.173:0/1987703770' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-05-27T07:17:52.102 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:17:51 smithi173 ceph-mon[37000]: from='client.? 172.21.15.173:0/3303699834' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "27e98ee2-4a92-4972-b2a3-959d0ed90981"}]: dispatch 2024-05-27T07:17:52.102 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:17:51 smithi173 ceph-mon[37000]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "27e98ee2-4a92-4972-b2a3-959d0ed90981"}]: dispatch 2024-05-27T07:17:52.102 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:17:51 smithi173 ceph-mon[37000]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "27e98ee2-4a92-4972-b2a3-959d0ed90981"}]': finished 2024-05-27T07:17:52.102 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:17:51 smithi173 ceph-mon[37000]: osdmap e34: 6 total, 5 up, 6 in 2024-05-27T07:17:52.103 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:17:51 smithi173 ceph-mon[37000]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-05-27T07:17:52.103 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:17:51 smithi173 ceph-mon[37000]: from='client.? 172.21.15.173:0/1987703770' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-05-27T07:17:53.102 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:17:52 smithi173 ceph-mon[37000]: pgmap v125: 1 pgs: 1 active+clean; 577 KiB data, 146 MiB used, 447 GiB / 447 GiB avail 2024-05-27T07:17:53.190 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:17:52 smithi105 ceph-mon[35977]: pgmap v125: 1 pgs: 1 active+clean; 577 KiB data, 146 MiB used, 447 GiB / 447 GiB avail 2024-05-27T07:17:53.230 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:17:52 smithi052 ceph-mon[31955]: pgmap v125: 1 pgs: 1 active+clean; 577 KiB data, 146 MiB used, 447 GiB / 447 GiB avail 2024-05-27T07:17:55.031 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:17:54 smithi052 ceph-mon[31955]: pgmap v126: 1 pgs: 1 active+clean; 577 KiB data, 146 MiB used, 447 GiB / 447 GiB avail 2024-05-27T07:17:55.103 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:17:54 smithi173 ceph-mon[37000]: pgmap v126: 1 pgs: 1 active+clean; 577 KiB data, 146 MiB used, 447 GiB / 447 GiB avail 2024-05-27T07:17:55.190 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:17:54 smithi105 ceph-mon[35977]: pgmap v126: 1 pgs: 1 active+clean; 577 KiB data, 146 MiB used, 447 GiB / 447 GiB avail 2024-05-27T07:17:57.046 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:17:56 smithi173 ceph-mon[37000]: pgmap v127: 1 pgs: 1 active+clean; 577 KiB data, 146 MiB used, 447 GiB / 447 GiB avail 2024-05-27T07:17:57.190 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:17:56 smithi105 ceph-mon[35977]: pgmap v127: 1 pgs: 1 active+clean; 577 KiB data, 146 MiB used, 447 GiB / 447 GiB avail 2024-05-27T07:17:57.230 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:17:56 smithi052 ceph-mon[31955]: pgmap v127: 1 pgs: 1 active+clean; 577 KiB data, 146 MiB used, 447 GiB / 447 GiB avail 2024-05-27T07:17:58.940 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:17:58 smithi173 ceph-mon[37000]: pgmap v128: 1 pgs: 1 active+clean; 577 KiB data, 146 MiB used, 447 GiB / 447 GiB avail 2024-05-27T07:17:58.940 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:17:58 smithi173 ceph-mon[37000]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2024-05-27T07:17:58.941 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:17:58 smithi173 ceph-mon[37000]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T07:17:58.941 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:17:58 smithi173 ceph-mon[37000]: Deploying daemon osd.5 on smithi173 2024-05-27T07:17:59.190 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:17:58 smithi105 ceph-mon[35977]: pgmap v128: 1 pgs: 1 active+clean; 577 KiB data, 146 MiB used, 447 GiB / 447 GiB avail 2024-05-27T07:17:59.190 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:17:58 smithi105 ceph-mon[35977]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2024-05-27T07:17:59.190 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:17:58 smithi105 ceph-mon[35977]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T07:17:59.190 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:17:58 smithi105 ceph-mon[35977]: Deploying daemon osd.5 on smithi173 2024-05-27T07:17:59.230 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:17:58 smithi052 ceph-mon[31955]: pgmap v128: 1 pgs: 1 active+clean; 577 KiB data, 146 MiB used, 447 GiB / 447 GiB avail 2024-05-27T07:17:59.230 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:17:58 smithi052 ceph-mon[31955]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2024-05-27T07:17:59.230 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:17:58 smithi052 ceph-mon[31955]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T07:17:59.230 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:17:58 smithi052 ceph-mon[31955]: Deploying daemon osd.5 on smithi173 2024-05-27T07:18:01.044 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:18:00 smithi173 ceph-mon[37000]: pgmap v129: 1 pgs: 1 active+clean; 577 KiB data, 146 MiB used, 447 GiB / 447 GiB avail 2024-05-27T07:18:01.190 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:18:00 smithi105 ceph-mon[35977]: pgmap v129: 1 pgs: 1 active+clean; 577 KiB data, 146 MiB used, 447 GiB / 447 GiB avail 2024-05-27T07:18:01.230 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:18:00 smithi052 ceph-mon[31955]: pgmap v129: 1 pgs: 1 active+clean; 577 KiB data, 146 MiB used, 447 GiB / 447 GiB avail 2024-05-27T07:18:03.029 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:18:02 smithi173 ceph-mon[37000]: pgmap v130: 1 pgs: 1 active+clean; 577 KiB data, 146 MiB used, 447 GiB / 447 GiB avail 2024-05-27T07:18:03.191 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:18:02 smithi105 ceph-mon[35977]: pgmap v130: 1 pgs: 1 active+clean; 577 KiB data, 146 MiB used, 447 GiB / 447 GiB avail 2024-05-27T07:18:03.229 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:18:02 smithi052 ceph-mon[31955]: pgmap v130: 1 pgs: 1 active+clean; 577 KiB data, 146 MiB used, 447 GiB / 447 GiB avail 2024-05-27T07:18:05.102 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:18:04 smithi173 ceph-mon[37000]: pgmap v131: 1 pgs: 1 active+clean; 577 KiB data, 146 MiB used, 447 GiB / 447 GiB avail 2024-05-27T07:18:05.102 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:18:04 smithi173 ceph-mon[37000]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' 2024-05-27T07:18:05.103 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:18:04 smithi173 ceph-mon[37000]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' 2024-05-27T07:18:05.103 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:18:04 smithi173 ceph-mon[37000]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-27T07:18:05.190 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:18:04 smithi105 ceph-mon[35977]: pgmap v131: 1 pgs: 1 active+clean; 577 KiB data, 146 MiB used, 447 GiB / 447 GiB avail 2024-05-27T07:18:05.190 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:18:04 smithi105 ceph-mon[35977]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' 2024-05-27T07:18:05.190 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:18:04 smithi105 ceph-mon[35977]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' 2024-05-27T07:18:05.190 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:18:04 smithi105 ceph-mon[35977]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-27T07:18:05.229 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:18:04 smithi052 ceph-mon[31955]: pgmap v131: 1 pgs: 1 active+clean; 577 KiB data, 146 MiB used, 447 GiB / 447 GiB avail 2024-05-27T07:18:05.230 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:18:04 smithi052 ceph-mon[31955]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' 2024-05-27T07:18:05.230 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:18:04 smithi052 ceph-mon[31955]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' 2024-05-27T07:18:05.230 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:18:04 smithi052 ceph-mon[31955]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-27T07:18:07.190 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:18:06 smithi105 ceph-mon[35977]: pgmap v132: 1 pgs: 1 active+clean; 577 KiB data, 146 MiB used, 447 GiB / 447 GiB avail 2024-05-27T07:18:07.229 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:18:06 smithi052 ceph-mon[31955]: pgmap v132: 1 pgs: 1 active+clean; 577 KiB data, 146 MiB used, 447 GiB / 447 GiB avail 2024-05-27T07:18:07.270 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:18:06 smithi173 ceph-mon[37000]: pgmap v132: 1 pgs: 1 active+clean; 577 KiB data, 146 MiB used, 447 GiB / 447 GiB avail 2024-05-27T07:18:07.333 INFO:teuthology.orchestra.run.smithi173.stderr:2024-05-27T07:18:07.342+0000 7fe6f8ff9700 1 -- 172.21.15.173:0/1700888403 <== mgr.14152 v2:172.21.15.52:6800/3513075805 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+37 (secure 0 0 0) 0x7fe6fc063bf0 con 0x7fe6e80b2900 2024-05-27T07:18:07.334 INFO:teuthology.orchestra.run.smithi173.stdout:Created osd(s) 5 on host 'smithi173' 2024-05-27T07:18:07.335 INFO:teuthology.orchestra.run.smithi173.stderr:2024-05-27T07:18:07.345+0000 7fe701b47700 1 -- 172.21.15.173:0/1700888403 >> [v2:172.21.15.52:6800/3513075805,v1:172.21.15.52:6801/3513075805] conn(0x7fe6e80b2900 msgr2=0x7fe6e80b4e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:18:07.336 INFO:teuthology.orchestra.run.smithi173.stderr:2024-05-27T07:18:07.345+0000 7fe701b47700 1 --2- 172.21.15.173:0/1700888403 >> [v2:172.21.15.52:6800/3513075805,v1:172.21.15.52:6801/3513075805] conn(0x7fe6e80b2900 0x7fe6e80b4e10 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7fe6fc076400 tx=0x7fe6e4008040 comp rx=0 tx=0).stop 2024-05-27T07:18:07.336 INFO:teuthology.orchestra.run.smithi173.stderr:2024-05-27T07:18:07.345+0000 7fe701b47700 1 -- 172.21.15.173:0/1700888403 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fe6fc10a100 msgr2=0x7fe6fc1a1da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:18:07.336 INFO:teuthology.orchestra.run.smithi173.stderr:2024-05-27T07:18:07.345+0000 7fe701b47700 1 --2- 172.21.15.173:0/1700888403 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fe6fc10a100 0x7fe6fc1a1da0 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7fe6f40062a0 tx=0x7fe6f4004e40 comp rx=0 tx=0).stop 2024-05-27T07:18:07.336 INFO:teuthology.orchestra.run.smithi173.stderr:2024-05-27T07:18:07.345+0000 7fe701b47700 1 -- 172.21.15.173:0/1700888403 shutdown_connections 2024-05-27T07:18:07.336 INFO:teuthology.orchestra.run.smithi173.stderr:2024-05-27T07:18:07.345+0000 7fe701b47700 1 --2- 172.21.15.173:0/1700888403 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fe6fc10a100 0x7fe6fc1a1da0 unknown :-1 s=CLOSED pgs=21 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:18:07.336 INFO:teuthology.orchestra.run.smithi173.stderr:2024-05-27T07:18:07.345+0000 7fe701b47700 1 --2- 172.21.15.173:0/1700888403 >> [v2:172.21.15.52:6800/3513075805,v1:172.21.15.52:6801/3513075805] conn(0x7fe6e80b2900 0x7fe6e80b4e10 unknown :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:18:07.336 INFO:teuthology.orchestra.run.smithi173.stderr:2024-05-27T07:18:07.345+0000 7fe701b47700 1 --2- 172.21.15.173:0/1700888403 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fe6fc104e40 0x7fe6fc19a8c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:18:07.336 INFO:teuthology.orchestra.run.smithi173.stderr:2024-05-27T07:18:07.345+0000 7fe701b47700 1 --2- 172.21.15.173:0/1700888403 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fe6fc1077a0 0x7fe6fc19ae00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:18:07.336 INFO:teuthology.orchestra.run.smithi173.stderr:2024-05-27T07:18:07.345+0000 7fe701b47700 1 -- 172.21.15.173:0/1700888403 >> 172.21.15.173:0/1700888403 conn(0x7fe6fc0fe880 msgr2=0x7fe6fc105910 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:18:07.336 INFO:teuthology.orchestra.run.smithi173.stderr:2024-05-27T07:18:07.345+0000 7fe701b47700 1 -- 172.21.15.173:0/1700888403 shutdown_connections 2024-05-27T07:18:07.336 INFO:teuthology.orchestra.run.smithi173.stderr:2024-05-27T07:18:07.345+0000 7fe701b47700 1 -- 172.21.15.173:0/1700888403 wait complete. 2024-05-27T07:18:07.941 DEBUG:teuthology.orchestra.run.smithi173:osd.5> sudo journalctl -f -n 0 -u ceph-73df2636-1bf8-11ef-bc9b-c7b262605968@osd.5.service 2024-05-27T07:18:07.943 INFO:tasks.cephadm:Deploying osd.6 on smithi052 with /dev/vg_nvme/lv_2... 2024-05-27T07:18:07.943 DEBUG:teuthology.orchestra.run.smithi052:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:38f9d900c83c63799fbdbe61acc9a11b0d3554a6 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 73df2636-1bf8-11ef-bc9b-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_2 2024-05-27T07:18:08.216 INFO:teuthology.orchestra.run.smithi052.stderr:Inferring config /var/lib/ceph/73df2636-1bf8-11ef-bc9b-c7b262605968/mon.a/config 2024-05-27T07:18:08.602 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:18:08 smithi173 ceph-mon[37000]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' 2024-05-27T07:18:08.602 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:18:08 smithi173 ceph-mon[37000]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' 2024-05-27T07:18:08.603 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:18:08 smithi173 ceph-mon[37000]: pgmap v133: 1 pgs: 1 active+clean; 577 KiB data, 146 MiB used, 447 GiB / 447 GiB avail 2024-05-27T07:18:08.603 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:18:08 smithi173 ceph-mon[37000]: from='osd.5 [v2:172.21.15.173:6808/2989239729,v1:172.21.15.173:6809/2989239729]' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-05-27T07:18:08.603 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:18:08 smithi173 ceph-mon[37000]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-05-27T07:18:08.603 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:18:08 smithi173 ceph-mon[37000]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' 2024-05-27T07:18:08.603 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:18:08 smithi173 ceph-mon[37000]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' 2024-05-27T07:18:08.615 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:18:08 smithi052 ceph-mon[31955]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' 2024-05-27T07:18:08.615 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:18:08 smithi052 ceph-mon[31955]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' 2024-05-27T07:18:08.615 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:18:08 smithi052 ceph-mon[31955]: pgmap v133: 1 pgs: 1 active+clean; 577 KiB data, 146 MiB used, 447 GiB / 447 GiB avail 2024-05-27T07:18:08.616 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:18:08 smithi052 ceph-mon[31955]: from='osd.5 [v2:172.21.15.173:6808/2989239729,v1:172.21.15.173:6809/2989239729]' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-05-27T07:18:08.616 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:18:08 smithi052 ceph-mon[31955]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-05-27T07:18:08.616 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:18:08 smithi052 ceph-mon[31955]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' 2024-05-27T07:18:08.616 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:18:08 smithi052 ceph-mon[31955]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' 2024-05-27T07:18:08.690 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:18:08 smithi105 ceph-mon[35977]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' 2024-05-27T07:18:08.691 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:18:08 smithi105 ceph-mon[35977]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' 2024-05-27T07:18:08.691 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:18:08 smithi105 ceph-mon[35977]: pgmap v133: 1 pgs: 1 active+clean; 577 KiB data, 146 MiB used, 447 GiB / 447 GiB avail 2024-05-27T07:18:08.691 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:18:08 smithi105 ceph-mon[35977]: from='osd.5 [v2:172.21.15.173:6808/2989239729,v1:172.21.15.173:6809/2989239729]' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-05-27T07:18:08.691 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:18:08 smithi105 ceph-mon[35977]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-05-27T07:18:08.691 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:18:08 smithi105 ceph-mon[35977]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' 2024-05-27T07:18:08.691 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:18:08 smithi105 ceph-mon[35977]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' 2024-05-27T07:18:09.602 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:18:09 smithi173 ceph-mon[37000]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2024-05-27T07:18:09.602 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:18:09 smithi173 ceph-mon[37000]: osdmap e35: 6 total, 5 up, 6 in 2024-05-27T07:18:09.603 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:18:09 smithi173 ceph-mon[37000]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-05-27T07:18:09.603 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:18:09 smithi173 ceph-mon[37000]: from='osd.5 [v2:172.21.15.173:6808/2989239729,v1:172.21.15.173:6809/2989239729]' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi173", "root=default"]}]: dispatch 2024-05-27T07:18:09.603 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:18:09 smithi173 ceph-mon[37000]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi173", "root=default"]}]: dispatch 2024-05-27T07:18:09.603 INFO:journalctl@ceph.osd.5.smithi173.stdout:May 27 07:18:09 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-osd-5[49009]: 2024-05-27T07:18:09.338+0000 7f331ee3b700 -1 osd.5 0 waiting for initial osdmap 2024-05-27T07:18:09.603 INFO:journalctl@ceph.osd.5.smithi173.stdout:May 27 07:18:09 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-osd-5[49009]: 2024-05-27T07:18:09.344+0000 7f331abf0700 -1 osd.5 36 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-05-27T07:18:09.641 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:18:09 smithi052 ceph-mon[31955]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2024-05-27T07:18:09.641 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:18:09 smithi052 ceph-mon[31955]: osdmap e35: 6 total, 5 up, 6 in 2024-05-27T07:18:09.641 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:18:09 smithi052 ceph-mon[31955]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-05-27T07:18:09.641 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:18:09 smithi052 ceph-mon[31955]: from='osd.5 [v2:172.21.15.173:6808/2989239729,v1:172.21.15.173:6809/2989239729]' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi173", "root=default"]}]: dispatch 2024-05-27T07:18:09.641 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:18:09 smithi052 ceph-mon[31955]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi173", "root=default"]}]: dispatch 2024-05-27T07:18:09.690 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:18:09 smithi105 ceph-mon[35977]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2024-05-27T07:18:09.690 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:18:09 smithi105 ceph-mon[35977]: osdmap e35: 6 total, 5 up, 6 in 2024-05-27T07:18:09.690 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:18:09 smithi105 ceph-mon[35977]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-05-27T07:18:09.690 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:18:09 smithi105 ceph-mon[35977]: from='osd.5 [v2:172.21.15.173:6808/2989239729,v1:172.21.15.173:6809/2989239729]' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi173", "root=default"]}]: dispatch 2024-05-27T07:18:09.690 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:18:09 smithi105 ceph-mon[35977]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi173", "root=default"]}]: dispatch 2024-05-27T07:18:10.347 INFO:teuthology.orchestra.run.smithi052.stderr:--> Zapping: /dev/vg_nvme/lv_2 2024-05-27T07:18:10.347 INFO:teuthology.orchestra.run.smithi052.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_2 bs=1M count=10 conv=fsync 2024-05-27T07:18:10.347 INFO:teuthology.orchestra.run.smithi052.stderr: stderr: 10+0 records in 2024-05-27T07:18:10.347 INFO:teuthology.orchestra.run.smithi052.stderr:10+0 records out 2024-05-27T07:18:10.347 INFO:teuthology.orchestra.run.smithi052.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.00834419 s, 1.3 GB/s 2024-05-27T07:18:10.347 INFO:teuthology.orchestra.run.smithi052.stderr:--> Zapping successful for: 2024-05-27T07:18:10.602 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:18:10 smithi173 ceph-mon[37000]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi173", "root=default"]}]': finished 2024-05-27T07:18:10.602 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:18:10 smithi173 ceph-mon[37000]: osdmap e36: 6 total, 5 up, 6 in 2024-05-27T07:18:10.603 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:18:10 smithi173 ceph-mon[37000]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-05-27T07:18:10.603 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:18:10 smithi173 ceph-mon[37000]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-05-27T07:18:10.603 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:18:10 smithi173 ceph-mon[37000]: pgmap v136: 1 pgs: 1 active+clean; 577 KiB data, 146 MiB used, 447 GiB / 447 GiB avail 2024-05-27T07:18:10.603 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:18:10 smithi173 ceph-mon[37000]: osd.5 [v2:172.21.15.173:6808/2989239729,v1:172.21.15.173:6809/2989239729] boot 2024-05-27T07:18:10.603 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:18:10 smithi173 ceph-mon[37000]: osdmap e37: 6 total, 6 up, 6 in 2024-05-27T07:18:10.603 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:18:10 smithi173 ceph-mon[37000]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-05-27T07:18:10.671 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:18:10 smithi052 ceph-mon[31955]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi173", "root=default"]}]': finished 2024-05-27T07:18:10.671 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:18:10 smithi052 ceph-mon[31955]: osdmap e36: 6 total, 5 up, 6 in 2024-05-27T07:18:10.671 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:18:10 smithi052 ceph-mon[31955]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-05-27T07:18:10.671 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:18:10 smithi052 ceph-mon[31955]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-05-27T07:18:10.671 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:18:10 smithi052 ceph-mon[31955]: pgmap v136: 1 pgs: 1 active+clean; 577 KiB data, 146 MiB used, 447 GiB / 447 GiB avail 2024-05-27T07:18:10.671 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:18:10 smithi052 ceph-mon[31955]: osd.5 [v2:172.21.15.173:6808/2989239729,v1:172.21.15.173:6809/2989239729] boot 2024-05-27T07:18:10.671 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:18:10 smithi052 ceph-mon[31955]: osdmap e37: 6 total, 6 up, 6 in 2024-05-27T07:18:10.671 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:18:10 smithi052 ceph-mon[31955]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-05-27T07:18:10.690 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:18:10 smithi105 ceph-mon[35977]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi173", "root=default"]}]': finished 2024-05-27T07:18:10.690 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:18:10 smithi105 ceph-mon[35977]: osdmap e36: 6 total, 5 up, 6 in 2024-05-27T07:18:10.690 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:18:10 smithi105 ceph-mon[35977]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-05-27T07:18:10.691 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:18:10 smithi105 ceph-mon[35977]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-05-27T07:18:10.691 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:18:10 smithi105 ceph-mon[35977]: pgmap v136: 1 pgs: 1 active+clean; 577 KiB data, 146 MiB used, 447 GiB / 447 GiB avail 2024-05-27T07:18:10.691 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:18:10 smithi105 ceph-mon[35977]: osd.5 [v2:172.21.15.173:6808/2989239729,v1:172.21.15.173:6809/2989239729] boot 2024-05-27T07:18:10.691 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:18:10 smithi105 ceph-mon[35977]: osdmap e37: 6 total, 6 up, 6 in 2024-05-27T07:18:10.691 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:18:10 smithi105 ceph-mon[35977]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-05-27T07:18:11.110 DEBUG:teuthology.orchestra.run.smithi052:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:38f9d900c83c63799fbdbe61acc9a11b0d3554a6 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 73df2636-1bf8-11ef-bc9b-c7b262605968 -- ceph orch daemon add osd smithi052:vg_nvme/lv_2 2024-05-27T07:18:11.358 INFO:teuthology.orchestra.run.smithi052.stderr:Inferring config /var/lib/ceph/73df2636-1bf8-11ef-bc9b-c7b262605968/mon.a/config 2024-05-27T07:18:11.852 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:18:11 smithi173 ceph-mon[37000]: purged_snaps scrub starts 2024-05-27T07:18:11.853 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:18:11 smithi173 ceph-mon[37000]: purged_snaps scrub ok 2024-05-27T07:18:11.853 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:18:11 smithi173 ceph-mon[37000]: osdmap e38: 6 total, 6 up, 6 in 2024-05-27T07:18:11.853 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:18:11 smithi173 ceph-mon[37000]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' 2024-05-27T07:18:11.853 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:18:11 smithi173 ceph-mon[37000]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' 2024-05-27T07:18:11.853 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:18:11 smithi173 ceph-mon[37000]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-05-27T07:18:11.854 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:18:11 smithi173 ceph-mon[37000]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-05-27T07:18:11.854 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:18:11 smithi173 ceph-mon[37000]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' 2024-05-27T07:18:11.854 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:18:11 smithi173 ceph-mon[37000]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T07:18:11.854 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:18:11 smithi173 ceph-mon[37000]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-27T07:18:11.854 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:18:11 smithi173 ceph-mon[37000]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' 2024-05-27T07:18:12.190 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:18:11 smithi105 ceph-mon[35977]: purged_snaps scrub starts 2024-05-27T07:18:12.191 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:18:11 smithi105 ceph-mon[35977]: purged_snaps scrub ok 2024-05-27T07:18:12.191 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:18:11 smithi105 ceph-mon[35977]: osdmap e38: 6 total, 6 up, 6 in 2024-05-27T07:18:12.191 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:18:11 smithi105 ceph-mon[35977]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' 2024-05-27T07:18:12.191 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:18:11 smithi105 ceph-mon[35977]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' 2024-05-27T07:18:12.191 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:18:11 smithi105 ceph-mon[35977]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-05-27T07:18:12.191 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:18:11 smithi105 ceph-mon[35977]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-05-27T07:18:12.191 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:18:11 smithi105 ceph-mon[35977]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' 2024-05-27T07:18:12.191 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:18:11 smithi105 ceph-mon[35977]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T07:18:12.191 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:18:11 smithi105 ceph-mon[35977]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-27T07:18:12.191 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:18:11 smithi105 ceph-mon[35977]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' 2024-05-27T07:18:12.230 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:18:11 smithi052 ceph-mon[31955]: purged_snaps scrub starts 2024-05-27T07:18:12.230 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:18:11 smithi052 ceph-mon[31955]: purged_snaps scrub ok 2024-05-27T07:18:12.230 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:18:11 smithi052 ceph-mon[31955]: osdmap e38: 6 total, 6 up, 6 in 2024-05-27T07:18:12.230 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:18:11 smithi052 ceph-mon[31955]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' 2024-05-27T07:18:12.230 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:18:11 smithi052 ceph-mon[31955]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' 2024-05-27T07:18:12.230 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:18:11 smithi052 ceph-mon[31955]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-05-27T07:18:12.230 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:18:11 smithi052 ceph-mon[31955]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-05-27T07:18:12.230 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:18:11 smithi052 ceph-mon[31955]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' 2024-05-27T07:18:12.230 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:18:11 smithi052 ceph-mon[31955]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T07:18:12.231 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:18:11 smithi052 ceph-mon[31955]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-27T07:18:12.231 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:18:11 smithi052 ceph-mon[31955]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' 2024-05-27T07:18:12.814 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:18:12 smithi052 ceph-mon[31955]: pgmap v139: 1 pgs: 1 active+clean; 577 KiB data, 172 MiB used, 536 GiB / 536 GiB avail 2024-05-27T07:18:12.814 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:18:12 smithi052 ceph-mon[31955]: Detected new or changed devices on smithi173 2024-05-27T07:18:12.815 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:18:12 smithi052 ceph-mon[31955]: Adjusting osd_memory_target on smithi173 to 8544M 2024-05-27T07:18:12.815 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:18:12 smithi052 ceph-mon[31955]: osdmap e39: 6 total, 6 up, 6 in 2024-05-27T07:18:12.815 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:18:12 smithi052 ceph-mon[31955]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-27T07:18:12.815 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:18:12 smithi052 ceph-mon[31955]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T07:18:12.815 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:18:12 smithi052 ceph-mon[31955]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-27T07:18:12.815 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:18:12 smithi052 ceph-mon[31955]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' 2024-05-27T07:18:13.052 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:18:13.059+0000 7f3d35ee3700 1 -- 172.21.15.52:0/1229697694 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f3d30104360 msgr2=0x7f3d30104770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:18:13.052 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:18:13.059+0000 7f3d35ee3700 1 --2- 172.21.15.52:0/1229697694 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f3d30104360 0x7f3d30104770 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7f3d24009990 tx=0x7f3d24009ca0 comp rx=0 tx=0).stop 2024-05-27T07:18:13.053 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:18:13.060+0000 7f3d35ee3700 1 -- 172.21.15.52:0/1229697694 shutdown_connections 2024-05-27T07:18:13.053 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:18:13.060+0000 7f3d35ee3700 1 --2- 172.21.15.52:0/1229697694 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f3d30104d00 0x7f3d3010b5b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:18:13.053 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:18:13.060+0000 7f3d35ee3700 1 --2- 172.21.15.52:0/1229697694 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f3d30103230 0x7f3d30103640 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:18:13.053 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:18:13.060+0000 7f3d35ee3700 1 --2- 172.21.15.52:0/1229697694 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f3d30104360 0x7f3d30104770 unknown :-1 s=CLOSED pgs=130 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:18:13.053 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:18:13.060+0000 7f3d35ee3700 1 -- 172.21.15.52:0/1229697694 >> 172.21.15.52:0/1229697694 conn(0x7f3d300fe870 msgr2=0x7f3d30100d20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:18:13.053 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:18:13.060+0000 7f3d35ee3700 1 -- 172.21.15.52:0/1229697694 shutdown_connections 2024-05-27T07:18:13.053 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:18:13.060+0000 7f3d35ee3700 1 -- 172.21.15.52:0/1229697694 wait complete. 2024-05-27T07:18:13.053 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:18:13.060+0000 7f3d35ee3700 1 Processor -- start 2024-05-27T07:18:13.053 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:18:13.061+0000 7f3d35ee3700 1 -- start start 2024-05-27T07:18:13.053 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:18:13.061+0000 7f3d35ee3700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f3d30103230 0x7f3d3019a9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:18:13.053 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:18:13.061+0000 7f3d35ee3700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f3d30104360 0x7f3d3019aee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:18:13.053 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:18:13.061+0000 7f3d35ee3700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f3d30104d00 0x7f3d301a1e80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:18:13.054 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:18:13.061+0000 7f3d35ee3700 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x7f3d3010ed10 con 0x7f3d30103230 2024-05-27T07:18:13.054 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:18:13.061+0000 7f3d35ee3700 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f3d3010eb90 con 0x7f3d30104d00 2024-05-27T07:18:13.054 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:18:13.061+0000 7f3d35ee3700 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f3d3010ee90 con 0x7f3d30104360 2024-05-27T07:18:13.054 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:18:13.061+0000 7f3d2f7fe700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f3d30103230 0x7f3d3019a9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:18:13.054 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:18:13.061+0000 7f3d2f7fe700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f3d30103230 0x7f3d3019a9a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.52:46754/0 (socket says 172.21.15.52:46754) 2024-05-27T07:18:13.054 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:18:13.061+0000 7f3d2f7fe700 1 -- 172.21.15.52:0/1645291891 learned_addr learned my addr 172.21.15.52:0/1645291891 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-05-27T07:18:13.054 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:18:13.061+0000 7f3d2effd700 1 --2- 172.21.15.52:0/1645291891 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f3d30104360 0x7f3d3019aee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:18:13.054 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:18:13.061+0000 7f3d2ffff700 1 --2- 172.21.15.52:0/1645291891 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f3d30104d00 0x7f3d301a1e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:18:13.054 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:18:13.061+0000 7f3d2f7fe700 1 -- 172.21.15.52:0/1645291891 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f3d30104360 msgr2=0x7f3d3019aee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:18:13.054 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:18:13.061+0000 7f3d2f7fe700 1 --2- 172.21.15.52:0/1645291891 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f3d30104360 0x7f3d3019aee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:18:13.054 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:18:13.061+0000 7f3d2f7fe700 1 -- 172.21.15.52:0/1645291891 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f3d30104d00 msgr2=0x7f3d301a1e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:18:13.055 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:18:13.061+0000 7f3d2f7fe700 1 --2- 172.21.15.52:0/1645291891 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f3d30104d00 0x7f3d301a1e80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:18:13.055 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:18:13.061+0000 7f3d2f7fe700 1 -- 172.21.15.52:0/1645291891 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3d301a2470 con 0x7f3d30103230 2024-05-27T07:18:13.055 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:18:13.061+0000 7f3d2effd700 1 --2- 172.21.15.52:0/1645291891 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f3d30104360 0x7f3d3019aee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-27T07:18:13.055 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:18:13.061+0000 7f3d2ffff700 1 --2- 172.21.15.52:0/1645291891 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f3d30104d00 0x7f3d301a1e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-27T07:18:13.055 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:18:13.062+0000 7f3d2f7fe700 1 --2- 172.21.15.52:0/1645291891 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f3d30103230 0x7f3d3019a9a0 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7f3d2000b4d0 tx=0x7f3d2000b890 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:18:13.055 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:18:13.062+0000 7f3d2cff9700 1 -- 172.21.15.52:0/1645291891 <== mon.0 v2:172.21.15.52:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3d20015020 con 0x7f3d30103230 2024-05-27T07:18:13.055 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:18:13.062+0000 7f3d35ee3700 1 -- 172.21.15.52:0/1645291891 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3d301a2760 con 0x7f3d30103230 2024-05-27T07:18:13.055 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:18:13.062+0000 7f3d2cff9700 1 -- 172.21.15.52:0/1645291891 <== mon.0 v2:172.21.15.52:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f3d20011ce0 con 0x7f3d30103230 2024-05-27T07:18:13.055 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:18:13.062+0000 7f3d2cff9700 1 -- 172.21.15.52:0/1645291891 <== mon.0 v2:172.21.15.52:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3d20010cf0 con 0x7f3d30103230 2024-05-27T07:18:13.055 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:18:13.062+0000 7f3d35ee3700 1 -- 172.21.15.52:0/1645291891 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3d301a2d70 con 0x7f3d30103230 2024-05-27T07:18:13.056 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:18:13.063+0000 7f3d35ee3700 1 -- 172.21.15.52:0/1645291891 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3d300512b0 con 0x7f3d30103230 2024-05-27T07:18:13.056 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:18:13.065+0000 7f3d2cff9700 1 -- 172.21.15.52:0/1645291891 <== mon.0 v2:172.21.15.52:3300/0 4 ==== mgrmap(e 16) ==== 149652+0+0 (secure 0 0 0) 0x7f3d20018020 con 0x7f3d30103230 2024-05-27T07:18:13.056 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:18:13.065+0000 7f3d2cff9700 1 --2- 172.21.15.52:0/1645291891 >> [v2:172.21.15.52:6800/3513075805,v1:172.21.15.52:6801/3513075805] conn(0x7f3d180b2880 0x7f3d180b4d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:18:13.056 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:18:13.065+0000 7f3d2effd700 1 --2- 172.21.15.52:0/1645291891 >> [v2:172.21.15.52:6800/3513075805,v1:172.21.15.52:6801/3513075805] conn(0x7f3d180b2880 0x7f3d180b4d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:18:13.057 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:18:13.065+0000 7f3d2cff9700 1 -- 172.21.15.52:0/1645291891 <== mon.0 v2:172.21.15.52:3300/0 5 ==== osd_map(39..39 src has 1..39) ==== 4533+0+0 (secure 0 0 0) 0x7f3d200e1200 con 0x7f3d30103230 2024-05-27T07:18:13.057 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:18:13.066+0000 7f3d2effd700 1 --2- 172.21.15.52:0/1645291891 >> [v2:172.21.15.52:6800/3513075805,v1:172.21.15.52:6801/3513075805] conn(0x7f3d180b2880 0x7f3d180b4d90 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f3d24003890 tx=0x7f3d2401a040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:18:13.059 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:18:13.069+0000 7f3d2cff9700 1 -- 172.21.15.52:0/1645291891 <== mon.0 v2:172.21.15.52:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191963 (secure 0 0 0) 0x7f3d2001f020 con 0x7f3d30103230 2024-05-27T07:18:13.092 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:18:12 smithi105 ceph-mon[35977]: pgmap v139: 1 pgs: 1 active+clean; 577 KiB data, 172 MiB used, 536 GiB / 536 GiB avail 2024-05-27T07:18:13.093 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:18:12 smithi105 ceph-mon[35977]: Detected new or changed devices on smithi173 2024-05-27T07:18:13.093 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:18:12 smithi105 ceph-mon[35977]: Adjusting osd_memory_target on smithi173 to 8544M 2024-05-27T07:18:13.093 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:18:12 smithi105 ceph-mon[35977]: osdmap e39: 6 total, 6 up, 6 in 2024-05-27T07:18:13.093 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:18:12 smithi105 ceph-mon[35977]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-27T07:18:13.093 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:18:12 smithi105 ceph-mon[35977]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T07:18:13.093 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:18:12 smithi105 ceph-mon[35977]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-27T07:18:13.093 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:18:12 smithi105 ceph-mon[35977]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' 2024-05-27T07:18:13.102 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:18:12 smithi173 ceph-mon[37000]: pgmap v139: 1 pgs: 1 active+clean; 577 KiB data, 172 MiB used, 536 GiB / 536 GiB avail 2024-05-27T07:18:13.102 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:18:12 smithi173 ceph-mon[37000]: Detected new or changed devices on smithi173 2024-05-27T07:18:13.102 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:18:12 smithi173 ceph-mon[37000]: Adjusting osd_memory_target on smithi173 to 8544M 2024-05-27T07:18:13.102 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:18:12 smithi173 ceph-mon[37000]: osdmap e39: 6 total, 6 up, 6 in 2024-05-27T07:18:13.103 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:18:12 smithi173 ceph-mon[37000]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-27T07:18:13.103 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:18:12 smithi173 ceph-mon[37000]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T07:18:13.103 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:18:12 smithi173 ceph-mon[37000]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-27T07:18:13.103 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:18:12 smithi173 ceph-mon[37000]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' 2024-05-27T07:18:13.285 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:18:13.295+0000 7f3d35ee3700 1 -- 172.21.15.52:0/1645291891 --> [v2:172.21.15.52:6800/3513075805,v1:172.21.15.52:6801/3513075805] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi052:vg_nvme/lv_2", "target": ["mon-mgr", ""]}) -- 0x7f3d301a2fd0 con 0x7f3d180b2880 2024-05-27T07:18:14.076 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:18:13 smithi052 ceph-mon[31955]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-05-27T07:18:14.077 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:18:13 smithi052 ceph-mon[31955]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-05-27T07:18:14.077 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:18:13 smithi052 ceph-mon[31955]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T07:18:14.102 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:18:13 smithi173 ceph-mon[37000]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-05-27T07:18:14.102 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:18:13 smithi173 ceph-mon[37000]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-05-27T07:18:14.102 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:18:13 smithi173 ceph-mon[37000]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T07:18:14.105 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:18:13 smithi105 ceph-mon[35977]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-05-27T07:18:14.105 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:18:13 smithi105 ceph-mon[35977]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-05-27T07:18:14.106 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:18:13 smithi105 ceph-mon[35977]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T07:18:15.035 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:18:14 smithi052 ceph-mon[31955]: from='client.14370 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi052:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-05-27T07:18:15.035 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:18:14 smithi052 ceph-mon[31955]: pgmap v141: 1 pgs: 1 active+clean; 577 KiB data, 173 MiB used, 536 GiB / 536 GiB avail; 135 KiB/s, 0 objects/s recovering 2024-05-27T07:18:15.102 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:18:14 smithi173 ceph-mon[37000]: from='client.14370 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi052:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-05-27T07:18:15.102 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:18:14 smithi173 ceph-mon[37000]: pgmap v141: 1 pgs: 1 active+clean; 577 KiB data, 173 MiB used, 536 GiB / 536 GiB avail; 135 KiB/s, 0 objects/s recovering 2024-05-27T07:18:15.190 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:18:14 smithi105 ceph-mon[35977]: from='client.14370 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi052:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-05-27T07:18:15.190 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:18:14 smithi105 ceph-mon[35977]: pgmap v141: 1 pgs: 1 active+clean; 577 KiB data, 173 MiB used, 536 GiB / 536 GiB avail; 135 KiB/s, 0 objects/s recovering 2024-05-27T07:18:17.102 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:18:16 smithi173 ceph-mon[37000]: pgmap v142: 1 pgs: 1 active+clean; 577 KiB data, 173 MiB used, 536 GiB / 536 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-05-27T07:18:17.102 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:18:16 smithi173 ceph-mon[37000]: from='client.? 172.21.15.52:0/3945292141' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "2b0ec0cb-f438-4b2c-8642-0778ed374e9a"}]: dispatch 2024-05-27T07:18:17.102 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:18:16 smithi173 ceph-mon[37000]: from='client.? 172.21.15.52:0/3945292141' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "2b0ec0cb-f438-4b2c-8642-0778ed374e9a"}]': finished 2024-05-27T07:18:17.103 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:18:16 smithi173 ceph-mon[37000]: osdmap e40: 7 total, 6 up, 7 in 2024-05-27T07:18:17.103 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:18:16 smithi173 ceph-mon[37000]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-05-27T07:18:17.103 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:18:16 smithi173 ceph-mon[37000]: from='client.? 172.21.15.52:0/744810945' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-05-27T07:18:17.190 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:18:16 smithi105 ceph-mon[35977]: pgmap v142: 1 pgs: 1 active+clean; 577 KiB data, 173 MiB used, 536 GiB / 536 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-05-27T07:18:17.190 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:18:16 smithi105 ceph-mon[35977]: from='client.? 172.21.15.52:0/3945292141' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "2b0ec0cb-f438-4b2c-8642-0778ed374e9a"}]: dispatch 2024-05-27T07:18:17.191 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:18:16 smithi105 ceph-mon[35977]: from='client.? 172.21.15.52:0/3945292141' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "2b0ec0cb-f438-4b2c-8642-0778ed374e9a"}]': finished 2024-05-27T07:18:17.191 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:18:16 smithi105 ceph-mon[35977]: osdmap e40: 7 total, 6 up, 7 in 2024-05-27T07:18:17.191 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:18:16 smithi105 ceph-mon[35977]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-05-27T07:18:17.191 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:18:16 smithi105 ceph-mon[35977]: from='client.? 172.21.15.52:0/744810945' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-05-27T07:18:17.230 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:18:16 smithi052 ceph-mon[31955]: pgmap v142: 1 pgs: 1 active+clean; 577 KiB data, 173 MiB used, 536 GiB / 536 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-05-27T07:18:17.230 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:18:16 smithi052 ceph-mon[31955]: from='client.? 172.21.15.52:0/3945292141' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "2b0ec0cb-f438-4b2c-8642-0778ed374e9a"}]: dispatch 2024-05-27T07:18:17.230 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:18:16 smithi052 ceph-mon[31955]: from='client.? 172.21.15.52:0/3945292141' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "2b0ec0cb-f438-4b2c-8642-0778ed374e9a"}]': finished 2024-05-27T07:18:17.230 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:18:16 smithi052 ceph-mon[31955]: osdmap e40: 7 total, 6 up, 7 in 2024-05-27T07:18:17.230 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:18:16 smithi052 ceph-mon[31955]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-05-27T07:18:17.230 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:18:16 smithi052 ceph-mon[31955]: from='client.? 172.21.15.52:0/744810945' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-05-27T07:18:19.102 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:18:18 smithi173 ceph-mon[37000]: pgmap v144: 1 pgs: 1 active+clean; 577 KiB data, 173 MiB used, 536 GiB / 536 GiB avail; 85 KiB/s, 0 objects/s recovering 2024-05-27T07:18:19.190 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:18:18 smithi105 ceph-mon[35977]: pgmap v144: 1 pgs: 1 active+clean; 577 KiB data, 173 MiB used, 536 GiB / 536 GiB avail; 85 KiB/s, 0 objects/s recovering 2024-05-27T07:18:19.229 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:18:18 smithi052 ceph-mon[31955]: pgmap v144: 1 pgs: 1 active+clean; 577 KiB data, 173 MiB used, 536 GiB / 536 GiB avail; 85 KiB/s, 0 objects/s recovering 2024-05-27T07:18:21.102 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:18:20 smithi173 ceph-mon[37000]: pgmap v145: 1 pgs: 1 active+clean; 577 KiB data, 173 MiB used, 536 GiB / 536 GiB avail; 72 KiB/s, 0 objects/s recovering 2024-05-27T07:18:21.119 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:18:20 smithi052 ceph-mon[31955]: pgmap v145: 1 pgs: 1 active+clean; 577 KiB data, 173 MiB used, 536 GiB / 536 GiB avail; 72 KiB/s, 0 objects/s recovering 2024-05-27T07:18:21.190 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:18:20 smithi105 ceph-mon[35977]: pgmap v145: 1 pgs: 1 active+clean; 577 KiB data, 173 MiB used, 536 GiB / 536 GiB avail; 72 KiB/s, 0 objects/s recovering 2024-05-27T07:18:22.980 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:18:22 smithi052 ceph-mon[31955]: pgmap v146: 1 pgs: 1 active+clean; 577 KiB data, 173 MiB used, 536 GiB / 536 GiB avail; 59 KiB/s, 0 objects/s recovering 2024-05-27T07:18:23.102 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:18:22 smithi173 ceph-mon[37000]: pgmap v146: 1 pgs: 1 active+clean; 577 KiB data, 173 MiB used, 536 GiB / 536 GiB avail; 59 KiB/s, 0 objects/s recovering 2024-05-27T07:18:23.190 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:18:22 smithi105 ceph-mon[35977]: pgmap v146: 1 pgs: 1 active+clean; 577 KiB data, 173 MiB used, 536 GiB / 536 GiB avail; 59 KiB/s, 0 objects/s recovering 2024-05-27T07:18:24.091 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:18:23 smithi052 ceph-mon[31955]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.6"}]: dispatch 2024-05-27T07:18:24.091 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:18:23 smithi052 ceph-mon[31955]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T07:18:24.102 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:18:23 smithi173 ceph-mon[37000]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.6"}]: dispatch 2024-05-27T07:18:24.102 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:18:23 smithi173 ceph-mon[37000]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T07:18:24.190 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:18:23 smithi105 ceph-mon[35977]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.6"}]: dispatch 2024-05-27T07:18:24.191 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:18:23 smithi105 ceph-mon[35977]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T07:18:25.102 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:18:24 smithi173 ceph-mon[37000]: Deploying daemon osd.6 on smithi052 2024-05-27T07:18:25.102 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:18:24 smithi173 ceph-mon[37000]: pgmap v147: 1 pgs: 1 active+clean; 577 KiB data, 173 MiB used, 536 GiB / 536 GiB avail 2024-05-27T07:18:25.186 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:18:24 smithi052 ceph-mon[31955]: Deploying daemon osd.6 on smithi052 2024-05-27T07:18:25.186 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:18:24 smithi052 ceph-mon[31955]: pgmap v147: 1 pgs: 1 active+clean; 577 KiB data, 173 MiB used, 536 GiB / 536 GiB avail 2024-05-27T07:18:25.190 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:18:24 smithi105 ceph-mon[35977]: Deploying daemon osd.6 on smithi052 2024-05-27T07:18:25.190 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:18:24 smithi105 ceph-mon[35977]: pgmap v147: 1 pgs: 1 active+clean; 577 KiB data, 173 MiB used, 536 GiB / 536 GiB avail 2024-05-27T07:18:27.101 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:18:26 smithi052 ceph-mon[31955]: pgmap v148: 1 pgs: 1 active+clean; 577 KiB data, 173 MiB used, 536 GiB / 536 GiB avail 2024-05-27T07:18:27.102 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:18:26 smithi173 ceph-mon[37000]: pgmap v148: 1 pgs: 1 active+clean; 577 KiB data, 173 MiB used, 536 GiB / 536 GiB avail 2024-05-27T07:18:27.190 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:18:26 smithi105 ceph-mon[35977]: pgmap v148: 1 pgs: 1 active+clean; 577 KiB data, 173 MiB used, 536 GiB / 536 GiB avail 2024-05-27T07:18:28.943 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:18:28 smithi052 ceph-mon[31955]: pgmap v149: 1 pgs: 1 active+clean; 577 KiB data, 173 MiB used, 536 GiB / 536 GiB avail 2024-05-27T07:18:29.102 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:18:28 smithi173 ceph-mon[37000]: pgmap v149: 1 pgs: 1 active+clean; 577 KiB data, 173 MiB used, 536 GiB / 536 GiB avail 2024-05-27T07:18:29.190 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:18:28 smithi105 ceph-mon[35977]: pgmap v149: 1 pgs: 1 active+clean; 577 KiB data, 173 MiB used, 536 GiB / 536 GiB avail 2024-05-27T07:18:30.480 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:18:30 smithi052 ceph-mon[31955]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' 2024-05-27T07:18:30.480 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:18:30 smithi052 ceph-mon[31955]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-27T07:18:30.480 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:18:30 smithi052 ceph-mon[31955]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' 2024-05-27T07:18:30.480 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:18:30 smithi052 ceph-mon[31955]: pgmap v150: 1 pgs: 1 active+clean; 577 KiB data, 173 MiB used, 536 GiB / 536 GiB avail 2024-05-27T07:18:30.602 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:18:30 smithi173 ceph-mon[37000]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' 2024-05-27T07:18:30.602 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:18:30 smithi173 ceph-mon[37000]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-27T07:18:30.602 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:18:30 smithi173 ceph-mon[37000]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' 2024-05-27T07:18:30.603 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:18:30 smithi173 ceph-mon[37000]: pgmap v150: 1 pgs: 1 active+clean; 577 KiB data, 173 MiB used, 536 GiB / 536 GiB avail 2024-05-27T07:18:30.690 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:18:30 smithi105 ceph-mon[35977]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' 2024-05-27T07:18:30.690 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:18:30 smithi105 ceph-mon[35977]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-27T07:18:30.690 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:18:30 smithi105 ceph-mon[35977]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' 2024-05-27T07:18:30.690 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:18:30 smithi105 ceph-mon[35977]: pgmap v150: 1 pgs: 1 active+clean; 577 KiB data, 173 MiB used, 536 GiB / 536 GiB avail 2024-05-27T07:18:32.841 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:18:32 smithi052 ceph-mon[31955]: pgmap v151: 1 pgs: 1 active+clean; 577 KiB data, 173 MiB used, 536 GiB / 536 GiB avail 2024-05-27T07:18:32.841 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:18:32 smithi052 ceph-mon[31955]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' 2024-05-27T07:18:32.841 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:18:32 smithi052 ceph-mon[31955]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' 2024-05-27T07:18:32.841 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:18:32 smithi052 ceph-mon[31955]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T07:18:32.841 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:18:32 smithi052 ceph-mon[31955]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-27T07:18:32.841 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:18:32 smithi052 ceph-mon[31955]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' 2024-05-27T07:18:32.940 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:18:32 smithi105 ceph-mon[35977]: pgmap v151: 1 pgs: 1 active+clean; 577 KiB data, 173 MiB used, 536 GiB / 536 GiB avail 2024-05-27T07:18:32.940 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:18:32 smithi105 ceph-mon[35977]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' 2024-05-27T07:18:32.941 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:18:32 smithi105 ceph-mon[35977]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' 2024-05-27T07:18:32.941 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:18:32 smithi105 ceph-mon[35977]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T07:18:32.941 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:18:32 smithi105 ceph-mon[35977]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-27T07:18:32.941 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:18:32 smithi105 ceph-mon[35977]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' 2024-05-27T07:18:33.102 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:18:32 smithi173 ceph-mon[37000]: pgmap v151: 1 pgs: 1 active+clean; 577 KiB data, 173 MiB used, 536 GiB / 536 GiB avail 2024-05-27T07:18:33.102 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:18:32 smithi173 ceph-mon[37000]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' 2024-05-27T07:18:33.102 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:18:32 smithi173 ceph-mon[37000]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' 2024-05-27T07:18:33.103 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:18:32 smithi173 ceph-mon[37000]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T07:18:33.103 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:18:32 smithi173 ceph-mon[37000]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-27T07:18:33.103 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:18:32 smithi173 ceph-mon[37000]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' 2024-05-27T07:18:33.306 INFO:teuthology.orchestra.run.smithi052.stdout:Created osd(s) 6 on host 'smithi052' 2024-05-27T07:18:33.307 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:18:33.314+0000 7f3d2cff9700 1 -- 172.21.15.52:0/1645291891 <== mgr.14152 v2:172.21.15.52:6800/3513075805 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+37 (secure 0 0 0) 0x7f3d301a2fd0 con 0x7f3d180b2880 2024-05-27T07:18:33.308 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:18:33.317+0000 7f3d35ee3700 1 -- 172.21.15.52:0/1645291891 >> [v2:172.21.15.52:6800/3513075805,v1:172.21.15.52:6801/3513075805] conn(0x7f3d180b2880 msgr2=0x7f3d180b4d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:18:33.308 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:18:33.317+0000 7f3d35ee3700 1 --2- 172.21.15.52:0/1645291891 >> [v2:172.21.15.52:6800/3513075805,v1:172.21.15.52:6801/3513075805] conn(0x7f3d180b2880 0x7f3d180b4d90 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f3d24003890 tx=0x7f3d2401a040 comp rx=0 tx=0).stop 2024-05-27T07:18:33.309 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:18:33.317+0000 7f3d35ee3700 1 -- 172.21.15.52:0/1645291891 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f3d30103230 msgr2=0x7f3d3019a9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:18:33.309 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:18:33.317+0000 7f3d35ee3700 1 --2- 172.21.15.52:0/1645291891 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f3d30103230 0x7f3d3019a9a0 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7f3d2000b4d0 tx=0x7f3d2000b890 comp rx=0 tx=0).stop 2024-05-27T07:18:33.309 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:18:33.317+0000 7f3d35ee3700 1 -- 172.21.15.52:0/1645291891 shutdown_connections 2024-05-27T07:18:33.309 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:18:33.317+0000 7f3d35ee3700 1 --2- 172.21.15.52:0/1645291891 >> [v2:172.21.15.52:6800/3513075805,v1:172.21.15.52:6801/3513075805] conn(0x7f3d180b2880 0x7f3d180b4d90 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:18:33.309 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:18:33.317+0000 7f3d35ee3700 1 --2- 172.21.15.52:0/1645291891 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f3d30104d00 0x7f3d301a1e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:18:33.309 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:18:33.317+0000 7f3d35ee3700 1 --2- 172.21.15.52:0/1645291891 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f3d30103230 0x7f3d3019a9a0 unknown :-1 s=CLOSED pgs=131 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:18:33.309 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:18:33.317+0000 7f3d35ee3700 1 --2- 172.21.15.52:0/1645291891 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f3d30104360 0x7f3d3019aee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:18:33.309 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:18:33.317+0000 7f3d35ee3700 1 -- 172.21.15.52:0/1645291891 >> 172.21.15.52:0/1645291891 conn(0x7f3d300fe870 msgr2=0x7f3d3006b600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:18:33.309 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:18:33.317+0000 7f3d35ee3700 1 -- 172.21.15.52:0/1645291891 shutdown_connections 2024-05-27T07:18:33.309 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:18:33.317+0000 7f3d35ee3700 1 -- 172.21.15.52:0/1645291891 wait complete. 2024-05-27T07:18:33.930 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:18:33 smithi052 ceph-mon[31955]: from='osd.6 [v2:172.21.15.52:6818/4244423065,v1:172.21.15.52:6819/4244423065]' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-05-27T07:18:33.930 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:18:33 smithi052 ceph-mon[31955]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' 2024-05-27T07:18:33.930 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:18:33 smithi052 ceph-mon[31955]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' 2024-05-27T07:18:33.930 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:18:33 smithi052 ceph-mon[31955]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-27T07:18:33.940 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:18:33 smithi105 ceph-mon[35977]: from='osd.6 [v2:172.21.15.52:6818/4244423065,v1:172.21.15.52:6819/4244423065]' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-05-27T07:18:33.940 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:18:33 smithi105 ceph-mon[35977]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' 2024-05-27T07:18:33.940 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:18:33 smithi105 ceph-mon[35977]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' 2024-05-27T07:18:33.940 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:18:33 smithi105 ceph-mon[35977]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-27T07:18:33.958 DEBUG:teuthology.orchestra.run.smithi052:osd.6> sudo journalctl -f -n 0 -u ceph-73df2636-1bf8-11ef-bc9b-c7b262605968@osd.6.service 2024-05-27T07:18:33.961 INFO:tasks.cephadm:Deploying osd.7 on smithi105 with /dev/vg_nvme/lv_2... 2024-05-27T07:18:33.961 DEBUG:teuthology.orchestra.run.smithi105:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:38f9d900c83c63799fbdbe61acc9a11b0d3554a6 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 73df2636-1bf8-11ef-bc9b-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_2 2024-05-27T07:18:34.102 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:18:33 smithi173 ceph-mon[37000]: from='osd.6 [v2:172.21.15.52:6818/4244423065,v1:172.21.15.52:6819/4244423065]' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-05-27T07:18:34.102 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:18:33 smithi173 ceph-mon[37000]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' 2024-05-27T07:18:34.102 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:18:33 smithi173 ceph-mon[37000]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' 2024-05-27T07:18:34.102 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:18:33 smithi173 ceph-mon[37000]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-27T07:18:34.204 INFO:teuthology.orchestra.run.smithi105.stderr:Inferring config /var/lib/ceph/73df2636-1bf8-11ef-bc9b-c7b262605968/mon.b/config 2024-05-27T07:18:34.940 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:18:34 smithi105 ceph-mon[35977]: pgmap v152: 1 pgs: 1 active+clean; 577 KiB data, 173 MiB used, 536 GiB / 536 GiB avail 2024-05-27T07:18:34.940 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:18:34 smithi105 ceph-mon[35977]: from='osd.6 [v2:172.21.15.52:6818/4244423065,v1:172.21.15.52:6819/4244423065]' entity='osd.6' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]': finished 2024-05-27T07:18:34.940 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:18:34 smithi105 ceph-mon[35977]: osdmap e41: 7 total, 6 up, 7 in 2024-05-27T07:18:34.940 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:18:34 smithi105 ceph-mon[35977]: from='osd.6 [v2:172.21.15.52:6818/4244423065,v1:172.21.15.52:6819/4244423065]' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi052", "root=default"]}]: dispatch 2024-05-27T07:18:34.940 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:18:34 smithi105 ceph-mon[35977]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-05-27T07:18:34.979 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:18:34 smithi052 ceph-mon[31955]: pgmap v152: 1 pgs: 1 active+clean; 577 KiB data, 173 MiB used, 536 GiB / 536 GiB avail 2024-05-27T07:18:34.979 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:18:34 smithi052 ceph-mon[31955]: from='osd.6 [v2:172.21.15.52:6818/4244423065,v1:172.21.15.52:6819/4244423065]' entity='osd.6' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]': finished 2024-05-27T07:18:34.980 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:18:34 smithi052 ceph-mon[31955]: osdmap e41: 7 total, 6 up, 7 in 2024-05-27T07:18:34.980 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:18:34 smithi052 ceph-mon[31955]: from='osd.6 [v2:172.21.15.52:6818/4244423065,v1:172.21.15.52:6819/4244423065]' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi052", "root=default"]}]: dispatch 2024-05-27T07:18:34.980 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:18:34 smithi052 ceph-mon[31955]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-05-27T07:18:34.980 INFO:journalctl@ceph.osd.6.smithi052.stdout:May 27 07:18:34 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-osd-6[60312]: 2024-05-27T07:18:34.663+0000 7f362c095700 -1 osd.6 0 waiting for initial osdmap 2024-05-27T07:18:34.980 INFO:journalctl@ceph.osd.6.smithi052.stdout:May 27 07:18:34 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-osd-6[60312]: 2024-05-27T07:18:34.673+0000 7f3627e4a700 -1 osd.6 42 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-05-27T07:18:35.102 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:18:34 smithi173 ceph-mon[37000]: pgmap v152: 1 pgs: 1 active+clean; 577 KiB data, 173 MiB used, 536 GiB / 536 GiB avail 2024-05-27T07:18:35.103 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:18:34 smithi173 ceph-mon[37000]: from='osd.6 [v2:172.21.15.52:6818/4244423065,v1:172.21.15.52:6819/4244423065]' entity='osd.6' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]': finished 2024-05-27T07:18:35.103 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:18:34 smithi173 ceph-mon[37000]: osdmap e41: 7 total, 6 up, 7 in 2024-05-27T07:18:35.103 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:18:34 smithi173 ceph-mon[37000]: from='osd.6 [v2:172.21.15.52:6818/4244423065,v1:172.21.15.52:6819/4244423065]' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi052", "root=default"]}]: dispatch 2024-05-27T07:18:35.103 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:18:34 smithi173 ceph-mon[37000]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-05-27T07:18:35.940 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:18:35 smithi105 ceph-mon[35977]: from='osd.6 [v2:172.21.15.52:6818/4244423065,v1:172.21.15.52:6819/4244423065]' entity='osd.6' cmd='[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi052", "root=default"]}]': finished 2024-05-27T07:18:35.941 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:18:35 smithi105 ceph-mon[35977]: osdmap e42: 7 total, 6 up, 7 in 2024-05-27T07:18:35.941 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:18:35 smithi105 ceph-mon[35977]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-05-27T07:18:35.941 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:18:35 smithi105 ceph-mon[35977]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-05-27T07:18:35.980 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:18:35 smithi052 ceph-mon[31955]: from='osd.6 [v2:172.21.15.52:6818/4244423065,v1:172.21.15.52:6819/4244423065]' entity='osd.6' cmd='[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi052", "root=default"]}]': finished 2024-05-27T07:18:35.980 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:18:35 smithi052 ceph-mon[31955]: osdmap e42: 7 total, 6 up, 7 in 2024-05-27T07:18:35.980 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:18:35 smithi052 ceph-mon[31955]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-05-27T07:18:35.980 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:18:35 smithi052 ceph-mon[31955]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-05-27T07:18:36.102 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:18:35 smithi173 ceph-mon[37000]: from='osd.6 [v2:172.21.15.52:6818/4244423065,v1:172.21.15.52:6819/4244423065]' entity='osd.6' cmd='[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi052", "root=default"]}]': finished 2024-05-27T07:18:36.102 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:18:35 smithi173 ceph-mon[37000]: osdmap e42: 7 total, 6 up, 7 in 2024-05-27T07:18:36.102 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:18:35 smithi173 ceph-mon[37000]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-05-27T07:18:36.102 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:18:35 smithi173 ceph-mon[37000]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-05-27T07:18:36.326 INFO:teuthology.orchestra.run.smithi105.stderr:--> Zapping: /dev/vg_nvme/lv_2 2024-05-27T07:18:36.326 INFO:teuthology.orchestra.run.smithi105.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_2 bs=1M count=10 conv=fsync 2024-05-27T07:18:36.326 INFO:teuthology.orchestra.run.smithi105.stderr: stderr: 10+0 records in 2024-05-27T07:18:36.326 INFO:teuthology.orchestra.run.smithi105.stderr:10+0 records out 2024-05-27T07:18:36.326 INFO:teuthology.orchestra.run.smithi105.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0114384 s, 917 MB/s 2024-05-27T07:18:36.326 INFO:teuthology.orchestra.run.smithi105.stderr:--> Zapping successful for: 2024-05-27T07:18:36.940 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:18:36 smithi105 ceph-mon[35977]: purged_snaps scrub starts 2024-05-27T07:18:36.940 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:18:36 smithi105 ceph-mon[35977]: purged_snaps scrub ok 2024-05-27T07:18:36.940 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:18:36 smithi105 ceph-mon[35977]: pgmap v155: 1 pgs: 1 active+clean; 577 KiB data, 173 MiB used, 536 GiB / 536 GiB avail 2024-05-27T07:18:36.940 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:18:36 smithi105 ceph-mon[35977]: osd.6 [v2:172.21.15.52:6818/4244423065,v1:172.21.15.52:6819/4244423065] boot 2024-05-27T07:18:36.941 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:18:36 smithi105 ceph-mon[35977]: osdmap e43: 7 total, 7 up, 7 in 2024-05-27T07:18:36.941 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:18:36 smithi105 ceph-mon[35977]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-05-27T07:18:36.980 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:18:36 smithi052 ceph-mon[31955]: purged_snaps scrub starts 2024-05-27T07:18:36.980 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:18:36 smithi052 ceph-mon[31955]: purged_snaps scrub ok 2024-05-27T07:18:36.980 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:18:36 smithi052 ceph-mon[31955]: pgmap v155: 1 pgs: 1 active+clean; 577 KiB data, 173 MiB used, 536 GiB / 536 GiB avail 2024-05-27T07:18:36.980 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:18:36 smithi052 ceph-mon[31955]: osd.6 [v2:172.21.15.52:6818/4244423065,v1:172.21.15.52:6819/4244423065] boot 2024-05-27T07:18:36.980 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:18:36 smithi052 ceph-mon[31955]: osdmap e43: 7 total, 7 up, 7 in 2024-05-27T07:18:36.980 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:18:36 smithi052 ceph-mon[31955]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-05-27T07:18:37.078 DEBUG:teuthology.orchestra.run.smithi105:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:38f9d900c83c63799fbdbe61acc9a11b0d3554a6 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 73df2636-1bf8-11ef-bc9b-c7b262605968 -- ceph orch daemon add osd smithi105:vg_nvme/lv_2 2024-05-27T07:18:37.102 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:18:36 smithi173 ceph-mon[37000]: purged_snaps scrub starts 2024-05-27T07:18:37.102 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:18:36 smithi173 ceph-mon[37000]: purged_snaps scrub ok 2024-05-27T07:18:37.102 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:18:36 smithi173 ceph-mon[37000]: pgmap v155: 1 pgs: 1 active+clean; 577 KiB data, 173 MiB used, 536 GiB / 536 GiB avail 2024-05-27T07:18:37.102 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:18:36 smithi173 ceph-mon[37000]: osd.6 [v2:172.21.15.52:6818/4244423065,v1:172.21.15.52:6819/4244423065] boot 2024-05-27T07:18:37.103 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:18:36 smithi173 ceph-mon[37000]: osdmap e43: 7 total, 7 up, 7 in 2024-05-27T07:18:37.103 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:18:36 smithi173 ceph-mon[37000]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-05-27T07:18:37.320 INFO:teuthology.orchestra.run.smithi105.stderr:Inferring config /var/lib/ceph/73df2636-1bf8-11ef-bc9b-c7b262605968/mon.b/config 2024-05-27T07:18:37.690 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:18:37 smithi105 ceph-mon[35977]: osdmap e44: 7 total, 7 up, 7 in 2024-05-27T07:18:37.690 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:18:37 smithi105 ceph-mon[35977]: Detected new or changed devices on smithi052 2024-05-27T07:18:37.691 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:18:37 smithi105 ceph-mon[35977]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' 2024-05-27T07:18:37.691 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:18:37 smithi105 ceph-mon[35977]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' 2024-05-27T07:18:37.691 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:18:37 smithi105 ceph-mon[35977]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-05-27T07:18:37.691 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:18:37 smithi105 ceph-mon[35977]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-05-27T07:18:37.691 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:18:37 smithi105 ceph-mon[35977]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-05-27T07:18:37.691 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:18:37 smithi105 ceph-mon[35977]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' 2024-05-27T07:18:37.691 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:18:37 smithi105 ceph-mon[35977]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T07:18:37.691 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:18:37 smithi105 ceph-mon[35977]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-27T07:18:37.691 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:18:37 smithi105 ceph-mon[35977]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' 2024-05-27T07:18:37.729 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:18:37 smithi052 ceph-mon[31955]: osdmap e44: 7 total, 7 up, 7 in 2024-05-27T07:18:37.730 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:18:37 smithi052 ceph-mon[31955]: Detected new or changed devices on smithi052 2024-05-27T07:18:37.730 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:18:37 smithi052 ceph-mon[31955]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' 2024-05-27T07:18:37.730 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:18:37 smithi052 ceph-mon[31955]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' 2024-05-27T07:18:37.730 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:18:37 smithi052 ceph-mon[31955]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-05-27T07:18:37.730 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:18:37 smithi052 ceph-mon[31955]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-05-27T07:18:37.730 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:18:37 smithi052 ceph-mon[31955]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-05-27T07:18:37.730 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:18:37 smithi052 ceph-mon[31955]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' 2024-05-27T07:18:37.730 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:18:37 smithi052 ceph-mon[31955]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T07:18:37.730 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:18:37 smithi052 ceph-mon[31955]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-27T07:18:37.730 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:18:37 smithi052 ceph-mon[31955]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' 2024-05-27T07:18:38.102 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:18:37 smithi173 ceph-mon[37000]: osdmap e44: 7 total, 7 up, 7 in 2024-05-27T07:18:38.102 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:18:37 smithi173 ceph-mon[37000]: Detected new or changed devices on smithi052 2024-05-27T07:18:38.102 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:18:37 smithi173 ceph-mon[37000]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' 2024-05-27T07:18:38.102 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:18:37 smithi173 ceph-mon[37000]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' 2024-05-27T07:18:38.102 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:18:37 smithi173 ceph-mon[37000]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-05-27T07:18:38.103 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:18:37 smithi173 ceph-mon[37000]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-05-27T07:18:38.103 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:18:37 smithi173 ceph-mon[37000]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-05-27T07:18:38.103 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:18:37 smithi173 ceph-mon[37000]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' 2024-05-27T07:18:38.103 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:18:37 smithi173 ceph-mon[37000]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T07:18:38.103 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:18:37 smithi173 ceph-mon[37000]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-27T07:18:38.103 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:18:37 smithi173 ceph-mon[37000]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' 2024-05-27T07:18:38.809 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:18:38 smithi105 ceph-mon[35977]: Adjusting osd_memory_target on smithi052 to 5697M 2024-05-27T07:18:38.810 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:18:38 smithi105 ceph-mon[35977]: pgmap v158: 1 pgs: 1 active+clean; 577 KiB data, 204 MiB used, 626 GiB / 626 GiB avail 2024-05-27T07:18:38.979 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:18:38 smithi052 ceph-mon[31955]: Adjusting osd_memory_target on smithi052 to 5697M 2024-05-27T07:18:38.980 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:18:38 smithi052 ceph-mon[31955]: pgmap v158: 1 pgs: 1 active+clean; 577 KiB data, 204 MiB used, 626 GiB / 626 GiB avail 2024-05-27T07:18:39.048 INFO:teuthology.orchestra.run.smithi105.stderr:2024-05-27T07:18:39.058+0000 7f197d626700 1 -- 172.21.15.105:0/2093412152 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f1978104360 msgr2=0x7f1978104770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:18:39.049 INFO:teuthology.orchestra.run.smithi105.stderr:2024-05-27T07:18:39.058+0000 7f197d626700 1 --2- 172.21.15.105:0/2093412152 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f1978104360 0x7f1978104770 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f196c009990 tx=0x7f196c009ca0 comp rx=0 tx=0).stop 2024-05-27T07:18:39.049 INFO:teuthology.orchestra.run.smithi105.stderr:2024-05-27T07:18:39.058+0000 7f197d626700 1 -- 172.21.15.105:0/2093412152 shutdown_connections 2024-05-27T07:18:39.049 INFO:teuthology.orchestra.run.smithi105.stderr:2024-05-27T07:18:39.058+0000 7f197d626700 1 --2- 172.21.15.105:0/2093412152 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f1978104d00 0x7f197810b5b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:18:39.049 INFO:teuthology.orchestra.run.smithi105.stderr:2024-05-27T07:18:39.058+0000 7f197d626700 1 --2- 172.21.15.105:0/2093412152 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f1978103230 0x7f1978103640 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:18:39.049 INFO:teuthology.orchestra.run.smithi105.stderr:2024-05-27T07:18:39.058+0000 7f197d626700 1 --2- 172.21.15.105:0/2093412152 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f1978104360 0x7f1978104770 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:18:39.049 INFO:teuthology.orchestra.run.smithi105.stderr:2024-05-27T07:18:39.058+0000 7f197d626700 1 -- 172.21.15.105:0/2093412152 >> 172.21.15.105:0/2093412152 conn(0x7f19780fe870 msgr2=0x7f1978100d20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:18:39.049 INFO:teuthology.orchestra.run.smithi105.stderr:2024-05-27T07:18:39.058+0000 7f197d626700 1 -- 172.21.15.105:0/2093412152 shutdown_connections 2024-05-27T07:18:39.049 INFO:teuthology.orchestra.run.smithi105.stderr:2024-05-27T07:18:39.058+0000 7f197d626700 1 -- 172.21.15.105:0/2093412152 wait complete. 2024-05-27T07:18:39.052 INFO:teuthology.orchestra.run.smithi105.stderr:2024-05-27T07:18:39.058+0000 7f197d626700 1 Processor -- start 2024-05-27T07:18:39.052 INFO:teuthology.orchestra.run.smithi105.stderr:2024-05-27T07:18:39.059+0000 7f197d626700 1 -- start start 2024-05-27T07:18:39.052 INFO:teuthology.orchestra.run.smithi105.stderr:2024-05-27T07:18:39.059+0000 7f197d626700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f1978103230 0x7f1978199860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:18:39.052 INFO:teuthology.orchestra.run.smithi105.stderr:2024-05-27T07:18:39.059+0000 7f197d626700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f1978104360 0x7f1978199da0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:18:39.052 INFO:teuthology.orchestra.run.smithi105.stderr:2024-05-27T07:18:39.059+0000 7f197d626700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f1978104d00 0x7f19781a0d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:18:39.052 INFO:teuthology.orchestra.run.smithi105.stderr:2024-05-27T07:18:39.059+0000 7f197d626700 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x7f197810ec90 con 0x7f1978104d00 2024-05-27T07:18:39.052 INFO:teuthology.orchestra.run.smithi105.stderr:2024-05-27T07:18:39.059+0000 7f197d626700 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f197810eb10 con 0x7f1978104360 2024-05-27T07:18:39.052 INFO:teuthology.orchestra.run.smithi105.stderr:2024-05-27T07:18:39.059+0000 7f197d626700 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f197810ee10 con 0x7f1978103230 2024-05-27T07:18:39.052 INFO:teuthology.orchestra.run.smithi105.stderr:2024-05-27T07:18:39.059+0000 7f19767fc700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f1978104360 0x7f1978199da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:18:39.053 INFO:teuthology.orchestra.run.smithi105.stderr:2024-05-27T07:18:39.059+0000 7f19767fc700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f1978104360 0x7f1978199da0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.105:36242/0 (socket says 172.21.15.105:36242) 2024-05-27T07:18:39.053 INFO:teuthology.orchestra.run.smithi105.stderr:2024-05-27T07:18:39.059+0000 7f19767fc700 1 -- 172.21.15.105:0/43987685 learned_addr learned my addr 172.21.15.105:0/43987685 (peer_addr_for_me v2:172.21.15.105:0/0) 2024-05-27T07:18:39.053 INFO:teuthology.orchestra.run.smithi105.stderr:2024-05-27T07:18:39.059+0000 7f1976ffd700 1 --2- 172.21.15.105:0/43987685 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f1978103230 0x7f1978199860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:18:39.053 INFO:teuthology.orchestra.run.smithi105.stderr:2024-05-27T07:18:39.059+0000 7f19777fe700 1 --2- 172.21.15.105:0/43987685 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f1978104d00 0x7f19781a0d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:18:39.053 INFO:teuthology.orchestra.run.smithi105.stderr:2024-05-27T07:18:39.059+0000 7f19767fc700 1 -- 172.21.15.105:0/43987685 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f1978103230 msgr2=0x7f1978199860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:18:39.053 INFO:teuthology.orchestra.run.smithi105.stderr:2024-05-27T07:18:39.059+0000 7f19767fc700 1 --2- 172.21.15.105:0/43987685 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f1978103230 0x7f1978199860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:18:39.053 INFO:teuthology.orchestra.run.smithi105.stderr:2024-05-27T07:18:39.059+0000 7f19767fc700 1 -- 172.21.15.105:0/43987685 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f1978104d00 msgr2=0x7f19781a0d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:18:39.053 INFO:teuthology.orchestra.run.smithi105.stderr:2024-05-27T07:18:39.059+0000 7f19767fc700 1 --2- 172.21.15.105:0/43987685 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f1978104d00 0x7f19781a0d40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:18:39.053 INFO:teuthology.orchestra.run.smithi105.stderr:2024-05-27T07:18:39.059+0000 7f19767fc700 1 -- 172.21.15.105:0/43987685 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f197806b600 con 0x7f1978104360 2024-05-27T07:18:39.053 INFO:teuthology.orchestra.run.smithi105.stderr:2024-05-27T07:18:39.059+0000 7f19767fc700 1 --2- 172.21.15.105:0/43987685 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f1978104360 0x7f1978199da0 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f196c006010 tx=0x7f196c0047b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:18:39.053 INFO:teuthology.orchestra.run.smithi105.stderr:2024-05-27T07:18:39.060+0000 7f195ffff700 1 -- 172.21.15.105:0/43987685 <== mon.2 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f196c027070 con 0x7f1978104360 2024-05-27T07:18:39.053 INFO:teuthology.orchestra.run.smithi105.stderr:2024-05-27T07:18:39.060+0000 7f197d626700 1 -- 172.21.15.105:0/43987685 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f197806b890 con 0x7f1978104360 2024-05-27T07:18:39.053 INFO:teuthology.orchestra.run.smithi105.stderr:2024-05-27T07:18:39.060+0000 7f195ffff700 1 -- 172.21.15.105:0/43987685 <== mon.2 v2:172.21.15.105:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f196c0040c0 con 0x7f1978104360 2024-05-27T07:18:39.054 INFO:teuthology.orchestra.run.smithi105.stderr:2024-05-27T07:18:39.060+0000 7f195ffff700 1 -- 172.21.15.105:0/43987685 <== mon.2 v2:172.21.15.105:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f196c022d90 con 0x7f1978104360 2024-05-27T07:18:39.054 INFO:teuthology.orchestra.run.smithi105.stderr:2024-05-27T07:18:39.060+0000 7f197d626700 1 -- 172.21.15.105:0/43987685 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f197806bdc0 con 0x7f1978104360 2024-05-27T07:18:39.054 INFO:teuthology.orchestra.run.smithi105.stderr:2024-05-27T07:18:39.061+0000 7f197d626700 1 -- 172.21.15.105:0/43987685 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f19780512b0 con 0x7f1978104360 2024-05-27T07:18:39.055 INFO:teuthology.orchestra.run.smithi105.stderr:2024-05-27T07:18:39.062+0000 7f195ffff700 1 -- 172.21.15.105:0/43987685 <== mon.2 v2:172.21.15.105:3300/0 4 ==== mgrmap(e 16) ==== 149652+0+0 (secure 0 0 0) 0x7f196c02a070 con 0x7f1978104360 2024-05-27T07:18:39.055 INFO:teuthology.orchestra.run.smithi105.stderr:2024-05-27T07:18:39.063+0000 7f195ffff700 1 --2- 172.21.15.105:0/43987685 >> [v2:172.21.15.52:6800/3513075805,v1:172.21.15.52:6801/3513075805] conn(0x7f19600b2880 0x7f19600b4d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:18:39.055 INFO:teuthology.orchestra.run.smithi105.stderr:2024-05-27T07:18:39.063+0000 7f195ffff700 1 -- 172.21.15.105:0/43987685 <== mon.2 v2:172.21.15.105:3300/0 5 ==== osd_map(44..44 src has 1..44) ==== 4965+0+0 (secure 0 0 0) 0x7f196c0e2620 con 0x7f1978104360 2024-05-27T07:18:39.055 INFO:teuthology.orchestra.run.smithi105.stderr:2024-05-27T07:18:39.063+0000 7f1976ffd700 1 --2- 172.21.15.105:0/43987685 >> [v2:172.21.15.52:6800/3513075805,v1:172.21.15.52:6801/3513075805] conn(0x7f19600b2880 0x7f19600b4d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:18:39.055 INFO:teuthology.orchestra.run.smithi105.stderr:2024-05-27T07:18:39.064+0000 7f1976ffd700 1 --2- 172.21.15.105:0/43987685 >> [v2:172.21.15.52:6800/3513075805,v1:172.21.15.52:6801/3513075805] conn(0x7f19600b2880 0x7f19600b4d90 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f1978104260 tx=0x7f196800a380 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:18:39.057 INFO:teuthology.orchestra.run.smithi105.stderr:2024-05-27T07:18:39.067+0000 7f195ffff700 1 -- 172.21.15.105:0/43987685 <== mon.2 v2:172.21.15.105:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191963 (secure 0 0 0) 0x7f196c013070 con 0x7f1978104360 2024-05-27T07:18:39.102 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:18:38 smithi173 ceph-mon[37000]: Adjusting osd_memory_target on smithi052 to 5697M 2024-05-27T07:18:39.102 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:18:38 smithi173 ceph-mon[37000]: pgmap v158: 1 pgs: 1 active+clean; 577 KiB data, 204 MiB used, 626 GiB / 626 GiB avail 2024-05-27T07:18:39.269 INFO:teuthology.orchestra.run.smithi105.stderr:2024-05-27T07:18:39.279+0000 7f197d626700 1 -- 172.21.15.105:0/43987685 --> [v2:172.21.15.52:6800/3513075805,v1:172.21.15.52:6801/3513075805] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi105:vg_nvme/lv_2", "target": ["mon-mgr", ""]}) -- 0x7f197806c020 con 0x7f19600b2880 2024-05-27T07:18:39.942 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:18:39 smithi105 ceph-mon[35977]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-05-27T07:18:39.942 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:18:39 smithi105 ceph-mon[35977]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-05-27T07:18:39.942 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:18:39 smithi105 ceph-mon[35977]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T07:18:39.979 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:18:39 smithi052 ceph-mon[31955]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-05-27T07:18:39.980 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:18:39 smithi052 ceph-mon[31955]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-05-27T07:18:39.980 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:18:39 smithi052 ceph-mon[31955]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T07:18:40.003 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:18:39 smithi173 ceph-mon[37000]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-05-27T07:18:40.003 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:18:39 smithi173 ceph-mon[37000]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-05-27T07:18:40.003 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:18:39 smithi173 ceph-mon[37000]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T07:18:40.940 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:18:40 smithi105 ceph-mon[35977]: from='client.24296 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi105:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-05-27T07:18:40.940 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:18:40 smithi105 ceph-mon[35977]: pgmap v159: 1 pgs: 1 active+clean; 577 KiB data, 204 MiB used, 626 GiB / 626 GiB avail 2024-05-27T07:18:40.979 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:18:40 smithi052 ceph-mon[31955]: from='client.24296 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi105:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-05-27T07:18:40.980 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:18:40 smithi052 ceph-mon[31955]: pgmap v159: 1 pgs: 1 active+clean; 577 KiB data, 204 MiB used, 626 GiB / 626 GiB avail 2024-05-27T07:18:41.023 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:18:40 smithi173 ceph-mon[37000]: from='client.24296 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi105:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-05-27T07:18:41.023 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:18:40 smithi173 ceph-mon[37000]: pgmap v159: 1 pgs: 1 active+clean; 577 KiB data, 204 MiB used, 626 GiB / 626 GiB avail 2024-05-27T07:18:42.940 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:18:42 smithi105 ceph-mon[35977]: pgmap v160: 1 pgs: 1 active+clean; 577 KiB data, 204 MiB used, 626 GiB / 626 GiB avail 2024-05-27T07:18:42.941 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:18:42 smithi105 ceph-mon[35977]: from='client.? 172.21.15.105:0/3410419168' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "cb2a7a6b-8015-43ea-865a-be144e45c51a"}]: dispatch 2024-05-27T07:18:42.941 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:18:42 smithi105 ceph-mon[35977]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "cb2a7a6b-8015-43ea-865a-be144e45c51a"}]: dispatch 2024-05-27T07:18:42.941 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:18:42 smithi105 ceph-mon[35977]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "cb2a7a6b-8015-43ea-865a-be144e45c51a"}]': finished 2024-05-27T07:18:42.941 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:18:42 smithi105 ceph-mon[35977]: osdmap e45: 8 total, 7 up, 8 in 2024-05-27T07:18:42.941 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:18:42 smithi105 ceph-mon[35977]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-05-27T07:18:42.980 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:18:42 smithi052 ceph-mon[31955]: pgmap v160: 1 pgs: 1 active+clean; 577 KiB data, 204 MiB used, 626 GiB / 626 GiB avail 2024-05-27T07:18:42.980 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:18:42 smithi052 ceph-mon[31955]: from='client.? 172.21.15.105:0/3410419168' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "cb2a7a6b-8015-43ea-865a-be144e45c51a"}]: dispatch 2024-05-27T07:18:42.980 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:18:42 smithi052 ceph-mon[31955]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "cb2a7a6b-8015-43ea-865a-be144e45c51a"}]: dispatch 2024-05-27T07:18:42.980 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:18:42 smithi052 ceph-mon[31955]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "cb2a7a6b-8015-43ea-865a-be144e45c51a"}]': finished 2024-05-27T07:18:42.980 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:18:42 smithi052 ceph-mon[31955]: osdmap e45: 8 total, 7 up, 8 in 2024-05-27T07:18:42.980 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:18:42 smithi052 ceph-mon[31955]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-05-27T07:18:43.102 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:18:42 smithi173 ceph-mon[37000]: pgmap v160: 1 pgs: 1 active+clean; 577 KiB data, 204 MiB used, 626 GiB / 626 GiB avail 2024-05-27T07:18:43.102 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:18:42 smithi173 ceph-mon[37000]: from='client.? 172.21.15.105:0/3410419168' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "cb2a7a6b-8015-43ea-865a-be144e45c51a"}]: dispatch 2024-05-27T07:18:43.102 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:18:42 smithi173 ceph-mon[37000]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "cb2a7a6b-8015-43ea-865a-be144e45c51a"}]: dispatch 2024-05-27T07:18:43.103 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:18:42 smithi173 ceph-mon[37000]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "cb2a7a6b-8015-43ea-865a-be144e45c51a"}]': finished 2024-05-27T07:18:43.103 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:18:42 smithi173 ceph-mon[37000]: osdmap e45: 8 total, 7 up, 8 in 2024-05-27T07:18:43.103 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:18:42 smithi173 ceph-mon[37000]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-05-27T07:18:43.940 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:18:43 smithi105 ceph-mon[35977]: from='client.? 172.21.15.105:0/3096970588' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-05-27T07:18:43.979 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:18:43 smithi052 ceph-mon[31955]: from='client.? 172.21.15.105:0/3096970588' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-05-27T07:18:44.102 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:18:43 smithi173 ceph-mon[37000]: from='client.? 172.21.15.105:0/3096970588' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-05-27T07:18:44.940 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:18:44 smithi105 ceph-mon[35977]: pgmap v162: 1 pgs: 1 active+clean; 577 KiB data, 204 MiB used, 626 GiB / 626 GiB avail 2024-05-27T07:18:44.979 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:18:44 smithi052 ceph-mon[31955]: pgmap v162: 1 pgs: 1 active+clean; 577 KiB data, 204 MiB used, 626 GiB / 626 GiB avail 2024-05-27T07:18:45.102 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:18:44 smithi173 ceph-mon[37000]: pgmap v162: 1 pgs: 1 active+clean; 577 KiB data, 204 MiB used, 626 GiB / 626 GiB avail 2024-05-27T07:18:46.940 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:18:46 smithi105 ceph-mon[35977]: pgmap v163: 1 pgs: 1 active+clean; 577 KiB data, 204 MiB used, 626 GiB / 626 GiB avail 2024-05-27T07:18:46.979 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:18:46 smithi052 ceph-mon[31955]: pgmap v163: 1 pgs: 1 active+clean; 577 KiB data, 204 MiB used, 626 GiB / 626 GiB avail 2024-05-27T07:18:47.102 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:18:46 smithi173 ceph-mon[37000]: pgmap v163: 1 pgs: 1 active+clean; 577 KiB data, 204 MiB used, 626 GiB / 626 GiB avail 2024-05-27T07:18:48.898 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:18:48 smithi105 ceph-mon[35977]: pgmap v164: 1 pgs: 1 active+clean; 577 KiB data, 204 MiB used, 626 GiB / 626 GiB avail 2024-05-27T07:18:48.980 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:18:48 smithi052 ceph-mon[31955]: pgmap v164: 1 pgs: 1 active+clean; 577 KiB data, 204 MiB used, 626 GiB / 626 GiB avail 2024-05-27T07:18:49.102 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:18:48 smithi173 ceph-mon[37000]: pgmap v164: 1 pgs: 1 active+clean; 577 KiB data, 204 MiB used, 626 GiB / 626 GiB avail 2024-05-27T07:18:49.940 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:18:49 smithi105 ceph-mon[35977]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.7"}]: dispatch 2024-05-27T07:18:49.940 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:18:49 smithi105 ceph-mon[35977]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T07:18:49.980 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:18:49 smithi052 ceph-mon[31955]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.7"}]: dispatch 2024-05-27T07:18:49.980 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:18:49 smithi052 ceph-mon[31955]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T07:18:50.102 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:18:49 smithi173 ceph-mon[37000]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.7"}]: dispatch 2024-05-27T07:18:50.102 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:18:49 smithi173 ceph-mon[37000]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T07:18:50.940 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:18:50 smithi105 ceph-mon[35977]: pgmap v165: 1 pgs: 1 active+clean; 577 KiB data, 204 MiB used, 626 GiB / 626 GiB avail 2024-05-27T07:18:50.940 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:18:50 smithi105 ceph-mon[35977]: Deploying daemon osd.7 on smithi105 2024-05-27T07:18:50.979 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:18:50 smithi052 ceph-mon[31955]: pgmap v165: 1 pgs: 1 active+clean; 577 KiB data, 204 MiB used, 626 GiB / 626 GiB avail 2024-05-27T07:18:50.980 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:18:50 smithi052 ceph-mon[31955]: Deploying daemon osd.7 on smithi105 2024-05-27T07:18:51.102 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:18:50 smithi173 ceph-mon[37000]: pgmap v165: 1 pgs: 1 active+clean; 577 KiB data, 204 MiB used, 626 GiB / 626 GiB avail 2024-05-27T07:18:51.102 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:18:50 smithi173 ceph-mon[37000]: Deploying daemon osd.7 on smithi105 2024-05-27T07:18:52.875 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:18:52 smithi105 ceph-mon[35977]: pgmap v166: 1 pgs: 1 active+clean; 577 KiB data, 204 MiB used, 626 GiB / 626 GiB avail 2024-05-27T07:18:52.979 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:18:52 smithi052 ceph-mon[31955]: pgmap v166: 1 pgs: 1 active+clean; 577 KiB data, 204 MiB used, 626 GiB / 626 GiB avail 2024-05-27T07:18:53.102 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:18:52 smithi173 ceph-mon[37000]: pgmap v166: 1 pgs: 1 active+clean; 577 KiB data, 204 MiB used, 626 GiB / 626 GiB avail 2024-05-27T07:18:54.979 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:18:54 smithi052 ceph-mon[31955]: pgmap v167: 1 pgs: 1 active+clean; 577 KiB data, 204 MiB used, 626 GiB / 626 GiB avail 2024-05-27T07:18:55.062 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:18:54 smithi105 ceph-mon[35977]: pgmap v167: 1 pgs: 1 active+clean; 577 KiB data, 204 MiB used, 626 GiB / 626 GiB avail 2024-05-27T07:18:55.102 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:18:54 smithi173 ceph-mon[37000]: pgmap v167: 1 pgs: 1 active+clean; 577 KiB data, 204 MiB used, 626 GiB / 626 GiB avail 2024-05-27T07:18:56.690 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:18:56 smithi105 ceph-mon[35977]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' 2024-05-27T07:18:56.690 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:18:56 smithi105 ceph-mon[35977]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' 2024-05-27T07:18:56.690 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:18:56 smithi105 ceph-mon[35977]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-27T07:18:56.691 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:18:56 smithi105 ceph-mon[35977]: pgmap v168: 1 pgs: 1 active+clean; 577 KiB data, 204 MiB used, 626 GiB / 626 GiB avail 2024-05-27T07:18:56.729 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:18:56 smithi052 ceph-mon[31955]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' 2024-05-27T07:18:56.729 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:18:56 smithi052 ceph-mon[31955]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' 2024-05-27T07:18:56.730 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:18:56 smithi052 ceph-mon[31955]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-27T07:18:56.730 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:18:56 smithi052 ceph-mon[31955]: pgmap v168: 1 pgs: 1 active+clean; 577 KiB data, 204 MiB used, 626 GiB / 626 GiB avail 2024-05-27T07:18:56.852 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:18:56 smithi173 ceph-mon[37000]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' 2024-05-27T07:18:56.852 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:18:56 smithi173 ceph-mon[37000]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' 2024-05-27T07:18:56.852 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:18:56 smithi173 ceph-mon[37000]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-27T07:18:56.852 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:18:56 smithi173 ceph-mon[37000]: pgmap v168: 1 pgs: 1 active+clean; 577 KiB data, 204 MiB used, 626 GiB / 626 GiB avail 2024-05-27T07:18:58.898 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:18:58 smithi105 ceph-mon[35977]: pgmap v169: 1 pgs: 1 active+clean; 577 KiB data, 204 MiB used, 626 GiB / 626 GiB avail 2024-05-27T07:18:58.979 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:18:58 smithi052 ceph-mon[31955]: pgmap v169: 1 pgs: 1 active+clean; 577 KiB data, 204 MiB used, 626 GiB / 626 GiB avail 2024-05-27T07:18:58.986 INFO:teuthology.orchestra.run.smithi105.stderr:2024-05-27T07:18:58.996+0000 7f195ffff700 1 -- 172.21.15.105:0/43987685 <== mgr.14152 v2:172.21.15.52:6800/3513075805 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+37 (secure 0 0 0) 0x7f197806c020 con 0x7f19600b2880 2024-05-27T07:18:58.987 INFO:teuthology.orchestra.run.smithi105.stdout:Created osd(s) 7 on host 'smithi105' 2024-05-27T07:18:58.990 INFO:teuthology.orchestra.run.smithi105.stderr:2024-05-27T07:18:58.999+0000 7f197d626700 1 -- 172.21.15.105:0/43987685 >> [v2:172.21.15.52:6800/3513075805,v1:172.21.15.52:6801/3513075805] conn(0x7f19600b2880 msgr2=0x7f19600b4d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:18:58.990 INFO:teuthology.orchestra.run.smithi105.stderr:2024-05-27T07:18:58.999+0000 7f197d626700 1 --2- 172.21.15.105:0/43987685 >> [v2:172.21.15.52:6800/3513075805,v1:172.21.15.52:6801/3513075805] conn(0x7f19600b2880 0x7f19600b4d90 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f1978104260 tx=0x7f196800a380 comp rx=0 tx=0).stop 2024-05-27T07:18:58.990 INFO:teuthology.orchestra.run.smithi105.stderr:2024-05-27T07:18:58.999+0000 7f197d626700 1 -- 172.21.15.105:0/43987685 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f1978104360 msgr2=0x7f1978199da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:18:58.990 INFO:teuthology.orchestra.run.smithi105.stderr:2024-05-27T07:18:58.999+0000 7f197d626700 1 --2- 172.21.15.105:0/43987685 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f1978104360 0x7f1978199da0 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f196c006010 tx=0x7f196c0047b0 comp rx=0 tx=0).stop 2024-05-27T07:18:58.990 INFO:teuthology.orchestra.run.smithi105.stderr:2024-05-27T07:18:58.999+0000 7f197d626700 1 -- 172.21.15.105:0/43987685 shutdown_connections 2024-05-27T07:18:58.990 INFO:teuthology.orchestra.run.smithi105.stderr:2024-05-27T07:18:58.999+0000 7f197d626700 1 --2- 172.21.15.105:0/43987685 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f1978104d00 0x7f19781a0d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:18:58.991 INFO:teuthology.orchestra.run.smithi105.stderr:2024-05-27T07:18:58.999+0000 7f197d626700 1 --2- 172.21.15.105:0/43987685 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f1978103230 0x7f1978199860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:18:58.991 INFO:teuthology.orchestra.run.smithi105.stderr:2024-05-27T07:18:58.999+0000 7f197d626700 1 --2- 172.21.15.105:0/43987685 >> [v2:172.21.15.52:6800/3513075805,v1:172.21.15.52:6801/3513075805] conn(0x7f19600b2880 0x7f19600b4d90 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:18:58.991 INFO:teuthology.orchestra.run.smithi105.stderr:2024-05-27T07:18:58.999+0000 7f197d626700 1 --2- 172.21.15.105:0/43987685 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f1978104360 0x7f1978199da0 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:18:58.991 INFO:teuthology.orchestra.run.smithi105.stderr:2024-05-27T07:18:58.999+0000 7f197d626700 1 -- 172.21.15.105:0/43987685 >> 172.21.15.105:0/43987685 conn(0x7f19780fe870 msgr2=0x7f1978107590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:18:58.991 INFO:teuthology.orchestra.run.smithi105.stderr:2024-05-27T07:18:59.000+0000 7f197d626700 1 -- 172.21.15.105:0/43987685 shutdown_connections 2024-05-27T07:18:58.991 INFO:teuthology.orchestra.run.smithi105.stderr:2024-05-27T07:18:59.000+0000 7f197d626700 1 -- 172.21.15.105:0/43987685 wait complete. 2024-05-27T07:18:59.102 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:18:58 smithi173 ceph-mon[37000]: pgmap v169: 1 pgs: 1 active+clean; 577 KiB data, 204 MiB used, 626 GiB / 626 GiB avail 2024-05-27T07:18:59.586 DEBUG:teuthology.orchestra.run.smithi105:osd.7> sudo journalctl -f -n 0 -u ceph-73df2636-1bf8-11ef-bc9b-c7b262605968@osd.7.service 2024-05-27T07:18:59.588 INFO:tasks.cephadm:Deploying osd.8 on smithi173 with /dev/vg_nvme/lv_2... 2024-05-27T07:18:59.588 DEBUG:teuthology.orchestra.run.smithi173:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:38f9d900c83c63799fbdbe61acc9a11b0d3554a6 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 73df2636-1bf8-11ef-bc9b-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_2 2024-05-27T07:18:59.833 INFO:teuthology.orchestra.run.smithi173.stderr:Inferring config /var/lib/ceph/73df2636-1bf8-11ef-bc9b-c7b262605968/mon.c/config 2024-05-27T07:19:00.190 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:18:59 smithi105 ceph-mon[35977]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' 2024-05-27T07:19:00.190 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:18:59 smithi105 ceph-mon[35977]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' 2024-05-27T07:19:00.190 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:18:59 smithi105 ceph-mon[35977]: from='osd.7 [v2:172.21.15.105:6816/2063611552,v1:172.21.15.105:6817/2063611552]' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-05-27T07:19:00.190 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:18:59 smithi105 ceph-mon[35977]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-05-27T07:19:00.190 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:18:59 smithi105 ceph-mon[35977]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' 2024-05-27T07:19:00.190 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:18:59 smithi105 ceph-mon[35977]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' 2024-05-27T07:19:00.210 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:18:59 smithi173 ceph-mon[37000]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' 2024-05-27T07:19:00.210 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:18:59 smithi173 ceph-mon[37000]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' 2024-05-27T07:19:00.210 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:18:59 smithi173 ceph-mon[37000]: from='osd.7 [v2:172.21.15.105:6816/2063611552,v1:172.21.15.105:6817/2063611552]' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-05-27T07:19:00.210 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:18:59 smithi173 ceph-mon[37000]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-05-27T07:19:00.211 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:18:59 smithi173 ceph-mon[37000]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' 2024-05-27T07:19:00.211 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:18:59 smithi173 ceph-mon[37000]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' 2024-05-27T07:19:00.230 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:18:59 smithi052 ceph-mon[31955]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' 2024-05-27T07:19:00.230 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:18:59 smithi052 ceph-mon[31955]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' 2024-05-27T07:19:00.230 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:18:59 smithi052 ceph-mon[31955]: from='osd.7 [v2:172.21.15.105:6816/2063611552,v1:172.21.15.105:6817/2063611552]' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-05-27T07:19:00.230 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:18:59 smithi052 ceph-mon[31955]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-05-27T07:19:00.230 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:18:59 smithi052 ceph-mon[31955]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' 2024-05-27T07:19:00.230 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:18:59 smithi052 ceph-mon[31955]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' 2024-05-27T07:19:01.221 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:19:00 smithi173 ceph-mon[37000]: pgmap v170: 1 pgs: 1 active+clean; 577 KiB data, 204 MiB used, 626 GiB / 626 GiB avail 2024-05-27T07:19:01.221 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:19:00 smithi173 ceph-mon[37000]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]': finished 2024-05-27T07:19:01.221 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:19:00 smithi173 ceph-mon[37000]: osdmap e46: 8 total, 7 up, 8 in 2024-05-27T07:19:01.222 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:19:00 smithi173 ceph-mon[37000]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-05-27T07:19:01.222 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:19:00 smithi173 ceph-mon[37000]: from='osd.7 [v2:172.21.15.105:6816/2063611552,v1:172.21.15.105:6817/2063611552]' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi105", "root=default"]}]: dispatch 2024-05-27T07:19:01.222 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:19:00 smithi173 ceph-mon[37000]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi105", "root=default"]}]: dispatch 2024-05-27T07:19:01.229 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:19:00 smithi052 ceph-mon[31955]: pgmap v170: 1 pgs: 1 active+clean; 577 KiB data, 204 MiB used, 626 GiB / 626 GiB avail 2024-05-27T07:19:01.230 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:19:00 smithi052 ceph-mon[31955]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]': finished 2024-05-27T07:19:01.230 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:19:00 smithi052 ceph-mon[31955]: osdmap e46: 8 total, 7 up, 8 in 2024-05-27T07:19:01.230 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:19:00 smithi052 ceph-mon[31955]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-05-27T07:19:01.230 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:19:00 smithi052 ceph-mon[31955]: from='osd.7 [v2:172.21.15.105:6816/2063611552,v1:172.21.15.105:6817/2063611552]' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi105", "root=default"]}]: dispatch 2024-05-27T07:19:01.230 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:19:00 smithi052 ceph-mon[31955]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi105", "root=default"]}]: dispatch 2024-05-27T07:19:01.259 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:19:00 smithi105 ceph-mon[35977]: pgmap v170: 1 pgs: 1 active+clean; 577 KiB data, 204 MiB used, 626 GiB / 626 GiB avail 2024-05-27T07:19:01.259 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:19:00 smithi105 ceph-mon[35977]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]': finished 2024-05-27T07:19:01.259 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:19:00 smithi105 ceph-mon[35977]: osdmap e46: 8 total, 7 up, 8 in 2024-05-27T07:19:01.259 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:19:00 smithi105 ceph-mon[35977]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-05-27T07:19:01.259 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:19:00 smithi105 ceph-mon[35977]: from='osd.7 [v2:172.21.15.105:6816/2063611552,v1:172.21.15.105:6817/2063611552]' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi105", "root=default"]}]: dispatch 2024-05-27T07:19:01.259 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:19:00 smithi105 ceph-mon[35977]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi105", "root=default"]}]: dispatch 2024-05-27T07:19:01.568 INFO:journalctl@ceph.osd.7.smithi105.stdout:May 27 07:19:01 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-osd-7[53596]: 2024-05-27T07:19:01.389+0000 7f5ef6841700 -1 osd.7 0 waiting for initial osdmap 2024-05-27T07:19:01.569 INFO:journalctl@ceph.osd.7.smithi105.stdout:May 27 07:19:01 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-osd-7[53596]: 2024-05-27T07:19:01.398+0000 7f5eee5ee700 -1 osd.7 47 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-05-27T07:19:01.963 INFO:teuthology.orchestra.run.smithi173.stderr:--> Zapping: /dev/vg_nvme/lv_2 2024-05-27T07:19:01.963 INFO:teuthology.orchestra.run.smithi173.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_2 bs=1M count=10 conv=fsync 2024-05-27T07:19:01.963 INFO:teuthology.orchestra.run.smithi173.stderr: stderr: 10+0 records in 2024-05-27T07:19:01.963 INFO:teuthology.orchestra.run.smithi173.stderr:10+0 records out 2024-05-27T07:19:01.963 INFO:teuthology.orchestra.run.smithi173.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.0137774 s, 761 MB/s 2024-05-27T07:19:01.963 INFO:teuthology.orchestra.run.smithi173.stderr:--> Zapping successful for: 2024-05-27T07:19:02.230 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:19:01 smithi052 ceph-mon[31955]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi105", "root=default"]}]': finished 2024-05-27T07:19:02.230 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:19:01 smithi052 ceph-mon[31955]: osdmap e47: 8 total, 7 up, 8 in 2024-05-27T07:19:02.230 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:19:01 smithi052 ceph-mon[31955]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-05-27T07:19:02.230 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:19:01 smithi052 ceph-mon[31955]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-05-27T07:19:02.230 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:19:01 smithi052 ceph-mon[31955]: from='osd.7 ' entity='osd.7' 2024-05-27T07:19:02.230 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:19:01 smithi052 ceph-mon[31955]: osd.7 [v2:172.21.15.105:6816/2063611552,v1:172.21.15.105:6817/2063611552] boot 2024-05-27T07:19:02.230 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:19:01 smithi052 ceph-mon[31955]: osdmap e48: 8 total, 8 up, 8 in 2024-05-27T07:19:02.230 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:19:01 smithi052 ceph-mon[31955]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-05-27T07:19:02.264 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:19:01 smithi173 ceph-mon[37000]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi105", "root=default"]}]': finished 2024-05-27T07:19:02.264 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:19:01 smithi173 ceph-mon[37000]: osdmap e47: 8 total, 7 up, 8 in 2024-05-27T07:19:02.265 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:19:01 smithi173 ceph-mon[37000]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-05-27T07:19:02.265 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:19:01 smithi173 ceph-mon[37000]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-05-27T07:19:02.265 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:19:01 smithi173 ceph-mon[37000]: from='osd.7 ' entity='osd.7' 2024-05-27T07:19:02.265 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:19:01 smithi173 ceph-mon[37000]: osd.7 [v2:172.21.15.105:6816/2063611552,v1:172.21.15.105:6817/2063611552] boot 2024-05-27T07:19:02.265 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:19:01 smithi173 ceph-mon[37000]: osdmap e48: 8 total, 8 up, 8 in 2024-05-27T07:19:02.265 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:19:01 smithi173 ceph-mon[37000]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-05-27T07:19:02.381 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:19:01 smithi105 ceph-mon[35977]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi105", "root=default"]}]': finished 2024-05-27T07:19:02.382 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:19:01 smithi105 ceph-mon[35977]: osdmap e47: 8 total, 7 up, 8 in 2024-05-27T07:19:02.382 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:19:01 smithi105 ceph-mon[35977]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-05-27T07:19:02.382 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:19:01 smithi105 ceph-mon[35977]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-05-27T07:19:02.382 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:19:01 smithi105 ceph-mon[35977]: from='osd.7 ' entity='osd.7' 2024-05-27T07:19:02.382 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:19:01 smithi105 ceph-mon[35977]: osd.7 [v2:172.21.15.105:6816/2063611552,v1:172.21.15.105:6817/2063611552] boot 2024-05-27T07:19:02.382 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:19:01 smithi105 ceph-mon[35977]: osdmap e48: 8 total, 8 up, 8 in 2024-05-27T07:19:02.382 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:19:01 smithi105 ceph-mon[35977]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-05-27T07:19:02.601 DEBUG:teuthology.orchestra.run.smithi173:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:38f9d900c83c63799fbdbe61acc9a11b0d3554a6 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 73df2636-1bf8-11ef-bc9b-c7b262605968 -- ceph orch daemon add osd smithi173:vg_nvme/lv_2 2024-05-27T07:19:02.840 INFO:teuthology.orchestra.run.smithi173.stderr:Inferring config /var/lib/ceph/73df2636-1bf8-11ef-bc9b-c7b262605968/mon.c/config 2024-05-27T07:19:03.026 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:19:02 smithi105 ceph-mon[35977]: purged_snaps scrub starts 2024-05-27T07:19:03.026 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:19:02 smithi105 ceph-mon[35977]: purged_snaps scrub ok 2024-05-27T07:19:03.026 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:19:02 smithi105 ceph-mon[35977]: pgmap v173: 1 pgs: 1 active+clean; 577 KiB data, 204 MiB used, 626 GiB / 626 GiB avail 2024-05-27T07:19:03.026 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:19:02 smithi105 ceph-mon[35977]: osdmap e49: 8 total, 8 up, 8 in 2024-05-27T07:19:03.229 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:19:02 smithi052 ceph-mon[31955]: purged_snaps scrub starts 2024-05-27T07:19:03.230 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:19:02 smithi052 ceph-mon[31955]: purged_snaps scrub ok 2024-05-27T07:19:03.230 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:19:02 smithi052 ceph-mon[31955]: pgmap v173: 1 pgs: 1 active+clean; 577 KiB data, 204 MiB used, 626 GiB / 626 GiB avail 2024-05-27T07:19:03.230 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:19:02 smithi052 ceph-mon[31955]: osdmap e49: 8 total, 8 up, 8 in 2024-05-27T07:19:03.235 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:19:02 smithi173 ceph-mon[37000]: purged_snaps scrub starts 2024-05-27T07:19:03.235 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:19:02 smithi173 ceph-mon[37000]: purged_snaps scrub ok 2024-05-27T07:19:03.235 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:19:02 smithi173 ceph-mon[37000]: pgmap v173: 1 pgs: 1 active+clean; 577 KiB data, 204 MiB used, 626 GiB / 626 GiB avail 2024-05-27T07:19:03.235 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:19:02 smithi173 ceph-mon[37000]: osdmap e49: 8 total, 8 up, 8 in 2024-05-27T07:19:04.216 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:19:04 smithi173 ceph-mon[37000]: Detected new or changed devices on smithi105 2024-05-27T07:19:04.217 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:19:04 smithi173 ceph-mon[37000]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' 2024-05-27T07:19:04.217 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:19:04 smithi173 ceph-mon[37000]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' 2024-05-27T07:19:04.217 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:19:04 smithi173 ceph-mon[37000]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-05-27T07:19:04.217 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:19:04 smithi173 ceph-mon[37000]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-05-27T07:19:04.217 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:19:04 smithi173 ceph-mon[37000]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-05-27T07:19:04.217 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:19:04 smithi173 ceph-mon[37000]: Adjusting osd_memory_target on smithi105 to 5696M 2024-05-27T07:19:04.217 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:19:04 smithi173 ceph-mon[37000]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' 2024-05-27T07:19:04.217 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:19:04 smithi173 ceph-mon[37000]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T07:19:04.217 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:19:04 smithi173 ceph-mon[37000]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-27T07:19:04.217 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:19:04 smithi173 ceph-mon[37000]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' 2024-05-27T07:19:04.217 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:19:04 smithi173 ceph-mon[37000]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-27T07:19:04.217 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:19:04 smithi173 ceph-mon[37000]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T07:19:04.218 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:19:04 smithi173 ceph-mon[37000]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-27T07:19:04.218 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:19:04 smithi173 ceph-mon[37000]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' 2024-05-27T07:19:04.218 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:19:04 smithi173 ceph-mon[37000]: pgmap v176: 1 pgs: 1 active+clean; 577 KiB data, 270 MiB used, 715 GiB / 715 GiB avail 2024-05-27T07:19:04.440 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:19:04 smithi105 ceph-mon[35977]: Detected new or changed devices on smithi105 2024-05-27T07:19:04.440 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:19:04 smithi105 ceph-mon[35977]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' 2024-05-27T07:19:04.440 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:19:04 smithi105 ceph-mon[35977]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' 2024-05-27T07:19:04.441 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:19:04 smithi105 ceph-mon[35977]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-05-27T07:19:04.441 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:19:04 smithi105 ceph-mon[35977]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-05-27T07:19:04.441 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:19:04 smithi105 ceph-mon[35977]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-05-27T07:19:04.441 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:19:04 smithi105 ceph-mon[35977]: Adjusting osd_memory_target on smithi105 to 5696M 2024-05-27T07:19:04.441 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:19:04 smithi105 ceph-mon[35977]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' 2024-05-27T07:19:04.441 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:19:04 smithi105 ceph-mon[35977]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T07:19:04.441 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:19:04 smithi105 ceph-mon[35977]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-27T07:19:04.441 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:19:04 smithi105 ceph-mon[35977]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' 2024-05-27T07:19:04.441 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:19:04 smithi105 ceph-mon[35977]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-27T07:19:04.441 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:19:04 smithi105 ceph-mon[35977]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T07:19:04.441 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:19:04 smithi105 ceph-mon[35977]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-27T07:19:04.441 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:19:04 smithi105 ceph-mon[35977]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' 2024-05-27T07:19:04.441 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:19:04 smithi105 ceph-mon[35977]: pgmap v176: 1 pgs: 1 active+clean; 577 KiB data, 270 MiB used, 715 GiB / 715 GiB avail 2024-05-27T07:19:04.463 INFO:teuthology.orchestra.run.smithi173.stderr:2024-05-27T07:19:04.471+0000 7f6d393e6700 1 -- 172.21.15.173:0/3089284045 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f6d34104e20 msgr2=0x7f6d34107240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:19:04.463 INFO:teuthology.orchestra.run.smithi173.stderr:2024-05-27T07:19:04.471+0000 7f6d393e6700 1 --2- 172.21.15.173:0/3089284045 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f6d34104e20 0x7f6d34107240 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f6d24009990 tx=0x7f6d24009ca0 comp rx=0 tx=0).stop 2024-05-27T07:19:04.463 INFO:teuthology.orchestra.run.smithi173.stderr:2024-05-27T07:19:04.471+0000 7f6d393e6700 1 -- 172.21.15.173:0/3089284045 shutdown_connections 2024-05-27T07:19:04.463 INFO:teuthology.orchestra.run.smithi173.stderr:2024-05-27T07:19:04.471+0000 7f6d393e6700 1 --2- 172.21.15.173:0/3089284045 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f6d3410a0e0 0x7f6d3410c500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:19:04.464 INFO:teuthology.orchestra.run.smithi173.stderr:2024-05-27T07:19:04.471+0000 7f6d393e6700 1 --2- 172.21.15.173:0/3089284045 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f6d34104e20 0x7f6d34107240 unknown :-1 s=CLOSED pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:19:04.464 INFO:teuthology.orchestra.run.smithi173.stderr:2024-05-27T07:19:04.471+0000 7f6d393e6700 1 --2- 172.21.15.173:0/3089284045 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f6d34107780 0x7f6d34109ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:19:04.464 INFO:teuthology.orchestra.run.smithi173.stderr:2024-05-27T07:19:04.471+0000 7f6d393e6700 1 -- 172.21.15.173:0/3089284045 >> 172.21.15.173:0/3089284045 conn(0x7f6d340fe860 msgr2=0x7f6d34100d10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:19:04.464 INFO:teuthology.orchestra.run.smithi173.stderr:2024-05-27T07:19:04.471+0000 7f6d393e6700 1 -- 172.21.15.173:0/3089284045 shutdown_connections 2024-05-27T07:19:04.464 INFO:teuthology.orchestra.run.smithi173.stderr:2024-05-27T07:19:04.471+0000 7f6d393e6700 1 -- 172.21.15.173:0/3089284045 wait complete. 2024-05-27T07:19:04.464 INFO:teuthology.orchestra.run.smithi173.stderr:2024-05-27T07:19:04.472+0000 7f6d393e6700 1 Processor -- start 2024-05-27T07:19:04.466 INFO:teuthology.orchestra.run.smithi173.stderr:2024-05-27T07:19:04.472+0000 7f6d393e6700 1 -- start start 2024-05-27T07:19:04.466 INFO:teuthology.orchestra.run.smithi173.stderr:2024-05-27T07:19:04.472+0000 7f6d393e6700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f6d34104e20 0x7f6d34104680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:19:04.466 INFO:teuthology.orchestra.run.smithi173.stderr:2024-05-27T07:19:04.472+0000 7f6d393e6700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f6d34107780 0x7f6d34102d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:19:04.466 INFO:teuthology.orchestra.run.smithi173.stderr:2024-05-27T07:19:04.472+0000 7f6d393e6700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f6d3410a0e0 0x7f6d34103270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:19:04.467 INFO:teuthology.orchestra.run.smithi173.stderr:2024-05-27T07:19:04.472+0000 7f6d393e6700 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x7f6d3410fd60 con 0x7f6d34104e20 2024-05-27T07:19:04.467 INFO:teuthology.orchestra.run.smithi173.stderr:2024-05-27T07:19:04.472+0000 7f6d393e6700 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f6d3410fbe0 con 0x7f6d3410a0e0 2024-05-27T07:19:04.467 INFO:teuthology.orchestra.run.smithi173.stderr:2024-05-27T07:19:04.472+0000 7f6d393e6700 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f6d3410fee0 con 0x7f6d34107780 2024-05-27T07:19:04.467 INFO:teuthology.orchestra.run.smithi173.stderr:2024-05-27T07:19:04.472+0000 7f6d2a7fc700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f6d34107780 0x7f6d34102d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:19:04.467 INFO:teuthology.orchestra.run.smithi173.stderr:2024-05-27T07:19:04.472+0000 7f6d2a7fc700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f6d34107780 0x7f6d34102d30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.173:3300/0 says I am v2:172.21.15.173:59408/0 (socket says 172.21.15.173:59408) 2024-05-27T07:19:04.467 INFO:teuthology.orchestra.run.smithi173.stderr:2024-05-27T07:19:04.473+0000 7f6d2a7fc700 1 -- 172.21.15.173:0/4096077160 learned_addr learned my addr 172.21.15.173:0/4096077160 (peer_addr_for_me v2:172.21.15.173:0/0) 2024-05-27T07:19:04.467 INFO:teuthology.orchestra.run.smithi173.stderr:2024-05-27T07:19:04.473+0000 7f6d337fe700 1 --2- 172.21.15.173:0/4096077160 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f6d3410a0e0 0x7f6d34103270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:19:04.467 INFO:teuthology.orchestra.run.smithi173.stderr:2024-05-27T07:19:04.473+0000 7f6d32ffd700 1 --2- 172.21.15.173:0/4096077160 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f6d34104e20 0x7f6d34104680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:19:04.467 INFO:teuthology.orchestra.run.smithi173.stderr:2024-05-27T07:19:04.473+0000 7f6d2a7fc700 1 -- 172.21.15.173:0/4096077160 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f6d3410a0e0 msgr2=0x7f6d34103270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:19:04.467 INFO:teuthology.orchestra.run.smithi173.stderr:2024-05-27T07:19:04.473+0000 7f6d2a7fc700 1 --2- 172.21.15.173:0/4096077160 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f6d3410a0e0 0x7f6d34103270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:19:04.468 INFO:teuthology.orchestra.run.smithi173.stderr:2024-05-27T07:19:04.473+0000 7f6d2a7fc700 1 -- 172.21.15.173:0/4096077160 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f6d34104e20 msgr2=0x7f6d34104680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:19:04.468 INFO:teuthology.orchestra.run.smithi173.stderr:2024-05-27T07:19:04.473+0000 7f6d2a7fc700 1 --2- 172.21.15.173:0/4096077160 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f6d34104e20 0x7f6d34104680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:19:04.468 INFO:teuthology.orchestra.run.smithi173.stderr:2024-05-27T07:19:04.473+0000 7f6d2a7fc700 1 -- 172.21.15.173:0/4096077160 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6d341039d0 con 0x7f6d34107780 2024-05-27T07:19:04.468 INFO:teuthology.orchestra.run.smithi173.stderr:2024-05-27T07:19:04.473+0000 7f6d2a7fc700 1 --2- 172.21.15.173:0/4096077160 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f6d34107780 0x7f6d34102d30 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7f6d1c00d980 tx=0x7f6d1c00dc90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:19:04.468 INFO:teuthology.orchestra.run.smithi173.stderr:2024-05-27T07:19:04.473+0000 7f6d30ff9700 1 -- 172.21.15.173:0/4096077160 <== mon.1 v2:172.21.15.173:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6d1c016070 con 0x7f6d34107780 2024-05-27T07:19:04.468 INFO:teuthology.orchestra.run.smithi173.stderr:2024-05-27T07:19:04.473+0000 7f6d30ff9700 1 -- 172.21.15.173:0/4096077160 <== mon.1 v2:172.21.15.173:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f6d1c004830 con 0x7f6d34107780 2024-05-27T07:19:04.468 INFO:teuthology.orchestra.run.smithi173.stderr:2024-05-27T07:19:04.473+0000 7f6d393e6700 1 -- 172.21.15.173:0/4096077160 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6d341a49a0 con 0x7f6d34107780 2024-05-27T07:19:04.468 INFO:teuthology.orchestra.run.smithi173.stderr:2024-05-27T07:19:04.473+0000 7f6d30ff9700 1 -- 172.21.15.173:0/4096077160 <== mon.1 v2:172.21.15.173:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6d1c0054c0 con 0x7f6d34107780 2024-05-27T07:19:04.468 INFO:teuthology.orchestra.run.smithi173.stderr:2024-05-27T07:19:04.473+0000 7f6d393e6700 1 -- 172.21.15.173:0/4096077160 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6d341a4e80 con 0x7f6d34107780 2024-05-27T07:19:04.468 INFO:teuthology.orchestra.run.smithi173.stderr:2024-05-27T07:19:04.474+0000 7f6d393e6700 1 -- 172.21.15.173:0/4096077160 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6d3402e5e0 con 0x7f6d34107780 2024-05-27T07:19:04.469 INFO:teuthology.orchestra.run.smithi173.stderr:2024-05-27T07:19:04.476+0000 7f6d30ff9700 1 -- 172.21.15.173:0/4096077160 <== mon.1 v2:172.21.15.173:3300/0 4 ==== mgrmap(e 16) ==== 149652+0+0 (secure 0 0 0) 0x7f6d1c004220 con 0x7f6d34107780 2024-05-27T07:19:04.469 INFO:teuthology.orchestra.run.smithi173.stderr:2024-05-27T07:19:04.477+0000 7f6d30ff9700 1 --2- 172.21.15.173:0/4096077160 >> [v2:172.21.15.52:6800/3513075805,v1:172.21.15.52:6801/3513075805] conn(0x7f6d200b2880 0x7f6d200b4d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:19:04.469 INFO:teuthology.orchestra.run.smithi173.stderr:2024-05-27T07:19:04.477+0000 7f6d30ff9700 1 -- 172.21.15.173:0/4096077160 <== mon.1 v2:172.21.15.173:3300/0 5 ==== osd_map(50..50 src has 1..50) ==== 5397+0+0 (secure 0 0 0) 0x7f6d1c0e1a70 con 0x7f6d34107780 2024-05-27T07:19:04.470 INFO:teuthology.orchestra.run.smithi173.stderr:2024-05-27T07:19:04.477+0000 7f6d32ffd700 1 --2- 172.21.15.173:0/4096077160 >> [v2:172.21.15.52:6800/3513075805,v1:172.21.15.52:6801/3513075805] conn(0x7f6d200b2880 0x7f6d200b4d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:19:04.470 INFO:teuthology.orchestra.run.smithi173.stderr:2024-05-27T07:19:04.477+0000 7f6d32ffd700 1 --2- 172.21.15.173:0/4096077160 >> [v2:172.21.15.52:6800/3513075805,v1:172.21.15.52:6801/3513075805] conn(0x7f6d200b2880 0x7f6d200b4d90 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f6d240038a0 tx=0x7f6d2401a040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:19:04.472 INFO:teuthology.orchestra.run.smithi173.stderr:2024-05-27T07:19:04.480+0000 7f6d30ff9700 1 -- 172.21.15.173:0/4096077160 <== mon.1 v2:172.21.15.173:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191963 (secure 0 0 0) 0x7f6d1c01f020 con 0x7f6d34107780 2024-05-27T07:19:04.480 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:19:04 smithi052 ceph-mon[31955]: Detected new or changed devices on smithi105 2024-05-27T07:19:04.480 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:19:04 smithi052 ceph-mon[31955]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' 2024-05-27T07:19:04.480 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:19:04 smithi052 ceph-mon[31955]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' 2024-05-27T07:19:04.480 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:19:04 smithi052 ceph-mon[31955]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-05-27T07:19:04.480 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:19:04 smithi052 ceph-mon[31955]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-05-27T07:19:04.480 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:19:04 smithi052 ceph-mon[31955]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-05-27T07:19:04.480 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:19:04 smithi052 ceph-mon[31955]: Adjusting osd_memory_target on smithi105 to 5696M 2024-05-27T07:19:04.480 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:19:04 smithi052 ceph-mon[31955]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' 2024-05-27T07:19:04.481 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:19:04 smithi052 ceph-mon[31955]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T07:19:04.481 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:19:04 smithi052 ceph-mon[31955]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-27T07:19:04.481 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:19:04 smithi052 ceph-mon[31955]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' 2024-05-27T07:19:04.481 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:19:04 smithi052 ceph-mon[31955]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-27T07:19:04.481 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:19:04 smithi052 ceph-mon[31955]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T07:19:04.481 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:19:04 smithi052 ceph-mon[31955]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-27T07:19:04.481 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:19:04 smithi052 ceph-mon[31955]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' 2024-05-27T07:19:04.481 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:19:04 smithi052 ceph-mon[31955]: pgmap v176: 1 pgs: 1 active+clean; 577 KiB data, 270 MiB used, 715 GiB / 715 GiB avail 2024-05-27T07:19:04.685 INFO:teuthology.orchestra.run.smithi173.stderr:2024-05-27T07:19:04.693+0000 7f6d393e6700 1 -- 172.21.15.173:0/4096077160 --> [v2:172.21.15.52:6800/3513075805,v1:172.21.15.52:6801/3513075805] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi173:vg_nvme/lv_2", "target": ["mon-mgr", ""]}) -- 0x7f6d34063bf0 con 0x7f6d200b2880 2024-05-27T07:19:05.432 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:19:05 smithi173 ceph-mon[37000]: osdmap e50: 8 total, 8 up, 8 in 2024-05-27T07:19:05.433 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:19:05 smithi173 ceph-mon[37000]: from='client.24310 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi173:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-05-27T07:19:05.433 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:19:05 smithi173 ceph-mon[37000]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-05-27T07:19:05.433 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:19:05 smithi173 ceph-mon[37000]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-05-27T07:19:05.433 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:19:05 smithi173 ceph-mon[37000]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T07:19:05.441 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:19:05 smithi105 ceph-mon[35977]: osdmap e50: 8 total, 8 up, 8 in 2024-05-27T07:19:05.441 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:19:05 smithi105 ceph-mon[35977]: from='client.24310 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi173:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-05-27T07:19:05.441 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:19:05 smithi105 ceph-mon[35977]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-05-27T07:19:05.441 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:19:05 smithi105 ceph-mon[35977]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-05-27T07:19:05.441 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:19:05 smithi105 ceph-mon[35977]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T07:19:05.479 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:19:05 smithi052 ceph-mon[31955]: osdmap e50: 8 total, 8 up, 8 in 2024-05-27T07:19:05.479 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:19:05 smithi052 ceph-mon[31955]: from='client.24310 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi173:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-05-27T07:19:05.480 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:19:05 smithi052 ceph-mon[31955]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-05-27T07:19:05.480 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:19:05 smithi052 ceph-mon[31955]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-05-27T07:19:05.480 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:19:05 smithi052 ceph-mon[31955]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T07:19:06.440 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:19:06 smithi105 ceph-mon[35977]: pgmap v178: 1 pgs: 1 active+clean; 577 KiB data, 270 MiB used, 715 GiB / 715 GiB avail 2024-05-27T07:19:06.443 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:19:06 smithi173 ceph-mon[37000]: pgmap v178: 1 pgs: 1 active+clean; 577 KiB data, 270 MiB used, 715 GiB / 715 GiB avail 2024-05-27T07:19:06.479 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:19:06 smithi052 ceph-mon[31955]: pgmap v178: 1 pgs: 1 active+clean; 577 KiB data, 270 MiB used, 715 GiB / 715 GiB avail 2024-05-27T07:19:07.649 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:19:07 smithi173 ceph-mon[37000]: from='client.? 172.21.15.173:0/630631735' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "c2fc565d-a0b9-4396-a209-bd884cddcfe3"}]: dispatch 2024-05-27T07:19:07.649 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:19:07 smithi173 ceph-mon[37000]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "c2fc565d-a0b9-4396-a209-bd884cddcfe3"}]: dispatch 2024-05-27T07:19:07.649 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:19:07 smithi173 ceph-mon[37000]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "c2fc565d-a0b9-4396-a209-bd884cddcfe3"}]': finished 2024-05-27T07:19:07.650 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:19:07 smithi173 ceph-mon[37000]: osdmap e51: 9 total, 8 up, 9 in 2024-05-27T07:19:07.650 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:19:07 smithi173 ceph-mon[37000]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-05-27T07:19:07.690 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:19:07 smithi105 ceph-mon[35977]: from='client.? 172.21.15.173:0/630631735' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "c2fc565d-a0b9-4396-a209-bd884cddcfe3"}]: dispatch 2024-05-27T07:19:07.690 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:19:07 smithi105 ceph-mon[35977]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "c2fc565d-a0b9-4396-a209-bd884cddcfe3"}]: dispatch 2024-05-27T07:19:07.690 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:19:07 smithi105 ceph-mon[35977]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "c2fc565d-a0b9-4396-a209-bd884cddcfe3"}]': finished 2024-05-27T07:19:07.691 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:19:07 smithi105 ceph-mon[35977]: osdmap e51: 9 total, 8 up, 9 in 2024-05-27T07:19:07.691 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:19:07 smithi105 ceph-mon[35977]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-05-27T07:19:07.729 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:19:07 smithi052 ceph-mon[31955]: from='client.? 172.21.15.173:0/630631735' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "c2fc565d-a0b9-4396-a209-bd884cddcfe3"}]: dispatch 2024-05-27T07:19:07.730 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:19:07 smithi052 ceph-mon[31955]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "c2fc565d-a0b9-4396-a209-bd884cddcfe3"}]: dispatch 2024-05-27T07:19:07.730 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:19:07 smithi052 ceph-mon[31955]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "c2fc565d-a0b9-4396-a209-bd884cddcfe3"}]': finished 2024-05-27T07:19:07.730 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:19:07 smithi052 ceph-mon[31955]: osdmap e51: 9 total, 8 up, 9 in 2024-05-27T07:19:07.730 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:19:07 smithi052 ceph-mon[31955]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-05-27T07:19:08.690 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:19:08 smithi105 ceph-mon[35977]: pgmap v180: 1 pgs: 1 active+clean; 577 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 102 KiB/s, 0 objects/s recovering 2024-05-27T07:19:08.690 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:19:08 smithi105 ceph-mon[35977]: from='client.? 172.21.15.173:0/4262506414' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-05-27T07:19:08.729 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:19:08 smithi052 ceph-mon[31955]: pgmap v180: 1 pgs: 1 active+clean; 577 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 102 KiB/s, 0 objects/s recovering 2024-05-27T07:19:08.730 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:19:08 smithi052 ceph-mon[31955]: from='client.? 172.21.15.173:0/4262506414' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-05-27T07:19:08.852 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:19:08 smithi173 ceph-mon[37000]: pgmap v180: 1 pgs: 1 active+clean; 577 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 102 KiB/s, 0 objects/s recovering 2024-05-27T07:19:08.852 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:19:08 smithi173 ceph-mon[37000]: from='client.? 172.21.15.173:0/4262506414' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-05-27T07:19:10.940 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:19:10 smithi105 ceph-mon[35977]: pgmap v181: 1 pgs: 1 active+clean; 577 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 87 KiB/s, 0 objects/s recovering 2024-05-27T07:19:10.968 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:19:10 smithi173 ceph-mon[37000]: pgmap v181: 1 pgs: 1 active+clean; 577 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 87 KiB/s, 0 objects/s recovering 2024-05-27T07:19:10.980 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:19:10 smithi052 ceph-mon[31955]: pgmap v181: 1 pgs: 1 active+clean; 577 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 87 KiB/s, 0 objects/s recovering 2024-05-27T07:19:12.852 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:19:12 smithi173 ceph-mon[37000]: pgmap v182: 1 pgs: 1 active+clean; 577 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 72 KiB/s, 0 objects/s recovering 2024-05-27T07:19:12.940 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:19:12 smithi105 ceph-mon[35977]: pgmap v182: 1 pgs: 1 active+clean; 577 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 72 KiB/s, 0 objects/s recovering 2024-05-27T07:19:12.979 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:19:12 smithi052 ceph-mon[31955]: pgmap v182: 1 pgs: 1 active+clean; 577 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 72 KiB/s, 0 objects/s recovering 2024-05-27T07:19:14.940 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:19:14 smithi105 ceph-mon[35977]: pgmap v183: 1 pgs: 1 active+clean; 577 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 61 KiB/s, 0 objects/s recovering 2024-05-27T07:19:14.979 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:19:14 smithi052 ceph-mon[31955]: pgmap v183: 1 pgs: 1 active+clean; 577 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 61 KiB/s, 0 objects/s recovering 2024-05-27T07:19:15.102 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:19:14 smithi173 ceph-mon[37000]: pgmap v183: 1 pgs: 1 active+clean; 577 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 61 KiB/s, 0 objects/s recovering 2024-05-27T07:19:15.852 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:19:15 smithi173 ceph-mon[37000]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.8"}]: dispatch 2024-05-27T07:19:15.852 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:19:15 smithi173 ceph-mon[37000]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T07:19:15.852 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:19:15 smithi173 ceph-mon[37000]: Deploying daemon osd.8 on smithi173 2024-05-27T07:19:15.923 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:19:15 smithi105 ceph-mon[35977]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.8"}]: dispatch 2024-05-27T07:19:15.923 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:19:15 smithi105 ceph-mon[35977]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T07:19:15.924 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:19:15 smithi105 ceph-mon[35977]: Deploying daemon osd.8 on smithi173 2024-05-27T07:19:15.980 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:19:15 smithi052 ceph-mon[31955]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.8"}]: dispatch 2024-05-27T07:19:15.980 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:19:15 smithi052 ceph-mon[31955]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T07:19:15.980 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:19:15 smithi052 ceph-mon[31955]: Deploying daemon osd.8 on smithi173 2024-05-27T07:19:16.940 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:19:16 smithi105 ceph-mon[35977]: pgmap v184: 1 pgs: 1 active+clean; 577 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-05-27T07:19:16.960 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:19:16 smithi173 ceph-mon[37000]: pgmap v184: 1 pgs: 1 active+clean; 577 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-05-27T07:19:16.979 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:19:16 smithi052 ceph-mon[31955]: pgmap v184: 1 pgs: 1 active+clean; 577 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-05-27T07:19:18.940 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:19:18 smithi105 ceph-mon[35977]: pgmap v185: 1 pgs: 1 active+clean; 577 KiB data, 235 MiB used, 715 GiB / 715 GiB avail 2024-05-27T07:19:18.980 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:19:18 smithi052 ceph-mon[31955]: pgmap v185: 1 pgs: 1 active+clean; 577 KiB data, 235 MiB used, 715 GiB / 715 GiB avail 2024-05-27T07:19:19.102 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:19:18 smithi173 ceph-mon[37000]: pgmap v185: 1 pgs: 1 active+clean; 577 KiB data, 235 MiB used, 715 GiB / 715 GiB avail 2024-05-27T07:19:20.925 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:19:20 smithi173 ceph-mon[37000]: pgmap v186: 1 pgs: 1 active+clean; 577 KiB data, 235 MiB used, 715 GiB / 715 GiB avail 2024-05-27T07:19:20.925 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:19:20 smithi173 ceph-mon[37000]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' 2024-05-27T07:19:20.925 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:19:20 smithi173 ceph-mon[37000]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-27T07:19:20.925 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:19:20 smithi173 ceph-mon[37000]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' 2024-05-27T07:19:20.940 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:19:20 smithi105 ceph-mon[35977]: pgmap v186: 1 pgs: 1 active+clean; 577 KiB data, 235 MiB used, 715 GiB / 715 GiB avail 2024-05-27T07:19:20.940 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:19:20 smithi105 ceph-mon[35977]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' 2024-05-27T07:19:20.940 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:19:20 smithi105 ceph-mon[35977]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-27T07:19:20.941 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:19:20 smithi105 ceph-mon[35977]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' 2024-05-27T07:19:20.979 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:19:20 smithi052 ceph-mon[31955]: pgmap v186: 1 pgs: 1 active+clean; 577 KiB data, 235 MiB used, 715 GiB / 715 GiB avail 2024-05-27T07:19:20.979 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:19:20 smithi052 ceph-mon[31955]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' 2024-05-27T07:19:20.980 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:19:20 smithi052 ceph-mon[31955]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-27T07:19:20.980 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:19:20 smithi052 ceph-mon[31955]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' 2024-05-27T07:19:22.852 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:19:22 smithi173 ceph-mon[37000]: pgmap v187: 1 pgs: 1 active+clean; 577 KiB data, 235 MiB used, 715 GiB / 715 GiB avail 2024-05-27T07:19:22.940 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:19:22 smithi105 ceph-mon[35977]: pgmap v187: 1 pgs: 1 active+clean; 577 KiB data, 235 MiB used, 715 GiB / 715 GiB avail 2024-05-27T07:19:22.979 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:19:22 smithi052 ceph-mon[31955]: pgmap v187: 1 pgs: 1 active+clean; 577 KiB data, 235 MiB used, 715 GiB / 715 GiB avail 2024-05-27T07:19:24.009 INFO:teuthology.orchestra.run.smithi173.stderr:2024-05-27T07:19:24.018+0000 7f6d30ff9700 1 -- 172.21.15.173:0/4096077160 <== mgr.14152 v2:172.21.15.52:6800/3513075805 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+37 (secure 0 0 0) 0x7f6d34063bf0 con 0x7f6d200b2880 2024-05-27T07:19:24.009 INFO:teuthology.orchestra.run.smithi173.stdout:Created osd(s) 8 on host 'smithi173' 2024-05-27T07:19:24.011 INFO:teuthology.orchestra.run.smithi173.stderr:2024-05-27T07:19:24.021+0000 7f6d393e6700 1 -- 172.21.15.173:0/4096077160 >> [v2:172.21.15.52:6800/3513075805,v1:172.21.15.52:6801/3513075805] conn(0x7f6d200b2880 msgr2=0x7f6d200b4d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:19:24.011 INFO:teuthology.orchestra.run.smithi173.stderr:2024-05-27T07:19:24.021+0000 7f6d393e6700 1 --2- 172.21.15.173:0/4096077160 >> [v2:172.21.15.52:6800/3513075805,v1:172.21.15.52:6801/3513075805] conn(0x7f6d200b2880 0x7f6d200b4d90 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f6d240038a0 tx=0x7f6d2401a040 comp rx=0 tx=0).stop 2024-05-27T07:19:24.011 INFO:teuthology.orchestra.run.smithi173.stderr:2024-05-27T07:19:24.021+0000 7f6d393e6700 1 -- 172.21.15.173:0/4096077160 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f6d34107780 msgr2=0x7f6d34102d30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:19:24.012 INFO:teuthology.orchestra.run.smithi173.stderr:2024-05-27T07:19:24.021+0000 7f6d393e6700 1 --2- 172.21.15.173:0/4096077160 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f6d34107780 0x7f6d34102d30 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7f6d1c00d980 tx=0x7f6d1c00dc90 comp rx=0 tx=0).stop 2024-05-27T07:19:24.012 INFO:teuthology.orchestra.run.smithi173.stderr:2024-05-27T07:19:24.021+0000 7f6d393e6700 1 -- 172.21.15.173:0/4096077160 shutdown_connections 2024-05-27T07:19:24.012 INFO:teuthology.orchestra.run.smithi173.stderr:2024-05-27T07:19:24.021+0000 7f6d393e6700 1 --2- 172.21.15.173:0/4096077160 >> [v2:172.21.15.52:6800/3513075805,v1:172.21.15.52:6801/3513075805] conn(0x7f6d200b2880 0x7f6d200b4d90 unknown :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:19:24.012 INFO:teuthology.orchestra.run.smithi173.stderr:2024-05-27T07:19:24.021+0000 7f6d393e6700 1 --2- 172.21.15.173:0/4096077160 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f6d3410a0e0 0x7f6d34103270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:19:24.012 INFO:teuthology.orchestra.run.smithi173.stderr:2024-05-27T07:19:24.021+0000 7f6d393e6700 1 --2- 172.21.15.173:0/4096077160 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f6d34104e20 0x7f6d34104680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:19:24.012 INFO:teuthology.orchestra.run.smithi173.stderr:2024-05-27T07:19:24.021+0000 7f6d393e6700 1 --2- 172.21.15.173:0/4096077160 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f6d34107780 0x7f6d34102d30 unknown :-1 s=CLOSED pgs=30 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:19:24.012 INFO:teuthology.orchestra.run.smithi173.stderr:2024-05-27T07:19:24.021+0000 7f6d393e6700 1 -- 172.21.15.173:0/4096077160 >> 172.21.15.173:0/4096077160 conn(0x7f6d340fe860 msgr2=0x7f6d341058f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:19:24.012 INFO:teuthology.orchestra.run.smithi173.stderr:2024-05-27T07:19:24.021+0000 7f6d393e6700 1 -- 172.21.15.173:0/4096077160 shutdown_connections 2024-05-27T07:19:24.012 INFO:teuthology.orchestra.run.smithi173.stderr:2024-05-27T07:19:24.021+0000 7f6d393e6700 1 -- 172.21.15.173:0/4096077160 wait complete. 2024-05-27T07:19:24.683 DEBUG:teuthology.orchestra.run.smithi173:osd.8> sudo journalctl -f -n 0 -u ceph-73df2636-1bf8-11ef-bc9b-c7b262605968@osd.8.service 2024-05-27T07:19:24.685 INFO:tasks.cephadm:Deploying osd.9 on smithi052 with /dev/vg_nvme/lv_1... 2024-05-27T07:19:24.685 DEBUG:teuthology.orchestra.run.smithi052:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:38f9d900c83c63799fbdbe61acc9a11b0d3554a6 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 73df2636-1bf8-11ef-bc9b-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_1 2024-05-27T07:19:24.950 INFO:teuthology.orchestra.run.smithi052.stderr:Inferring config /var/lib/ceph/73df2636-1bf8-11ef-bc9b-c7b262605968/mon.a/config 2024-05-27T07:19:25.317 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:19:25 smithi052 ceph-mon[31955]: pgmap v188: 1 pgs: 1 active+clean; 577 KiB data, 235 MiB used, 715 GiB / 715 GiB avail 2024-05-27T07:19:25.317 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:19:25 smithi052 ceph-mon[31955]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' 2024-05-27T07:19:25.317 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:19:25 smithi052 ceph-mon[31955]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' 2024-05-27T07:19:25.318 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:19:25 smithi052 ceph-mon[31955]: from='osd.8 [v2:172.21.15.173:6816/4131399852,v1:172.21.15.173:6817/4131399852]' entity='osd.8' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]: dispatch 2024-05-27T07:19:25.318 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:19:25 smithi052 ceph-mon[31955]: from='osd.8 ' entity='osd.8' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]: dispatch 2024-05-27T07:19:25.318 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:19:25 smithi052 ceph-mon[31955]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' 2024-05-27T07:19:25.318 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:19:25 smithi052 ceph-mon[31955]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' 2024-05-27T07:19:25.353 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:19:25 smithi173 ceph-mon[37000]: pgmap v188: 1 pgs: 1 active+clean; 577 KiB data, 235 MiB used, 715 GiB / 715 GiB avail 2024-05-27T07:19:25.354 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:19:25 smithi173 ceph-mon[37000]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' 2024-05-27T07:19:25.354 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:19:25 smithi173 ceph-mon[37000]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' 2024-05-27T07:19:25.354 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:19:25 smithi173 ceph-mon[37000]: from='osd.8 [v2:172.21.15.173:6816/4131399852,v1:172.21.15.173:6817/4131399852]' entity='osd.8' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]: dispatch 2024-05-27T07:19:25.354 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:19:25 smithi173 ceph-mon[37000]: from='osd.8 ' entity='osd.8' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]: dispatch 2024-05-27T07:19:25.354 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:19:25 smithi173 ceph-mon[37000]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' 2024-05-27T07:19:25.354 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:19:25 smithi173 ceph-mon[37000]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' 2024-05-27T07:19:25.440 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:19:25 smithi105 ceph-mon[35977]: pgmap v188: 1 pgs: 1 active+clean; 577 KiB data, 235 MiB used, 715 GiB / 715 GiB avail 2024-05-27T07:19:25.440 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:19:25 smithi105 ceph-mon[35977]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' 2024-05-27T07:19:25.440 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:19:25 smithi105 ceph-mon[35977]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' 2024-05-27T07:19:25.440 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:19:25 smithi105 ceph-mon[35977]: from='osd.8 [v2:172.21.15.173:6816/4131399852,v1:172.21.15.173:6817/4131399852]' entity='osd.8' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]: dispatch 2024-05-27T07:19:25.440 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:19:25 smithi105 ceph-mon[35977]: from='osd.8 ' entity='osd.8' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]: dispatch 2024-05-27T07:19:25.441 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:19:25 smithi105 ceph-mon[35977]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' 2024-05-27T07:19:25.441 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:19:25 smithi105 ceph-mon[35977]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' 2024-05-27T07:19:26.352 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:19:26 smithi173 ceph-mon[37000]: from='osd.8 ' entity='osd.8' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]': finished 2024-05-27T07:19:26.352 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:19:26 smithi173 ceph-mon[37000]: osdmap e52: 9 total, 8 up, 9 in 2024-05-27T07:19:26.352 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:19:26 smithi173 ceph-mon[37000]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-05-27T07:19:26.352 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:19:26 smithi173 ceph-mon[37000]: from='osd.8 [v2:172.21.15.173:6816/4131399852,v1:172.21.15.173:6817/4131399852]' entity='osd.8' cmd=[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi173", "root=default"]}]: dispatch 2024-05-27T07:19:26.352 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:19:26 smithi173 ceph-mon[37000]: from='osd.8 ' entity='osd.8' cmd=[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi173", "root=default"]}]: dispatch 2024-05-27T07:19:26.353 INFO:journalctl@ceph.osd.8.smithi173.stdout:May 27 07:19:26 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-osd-8[55969]: 2024-05-27T07:19:26.021+0000 7f425dd32700 -1 osd.8 0 waiting for initial osdmap 2024-05-27T07:19:26.353 INFO:journalctl@ceph.osd.8.smithi173.stdout:May 27 07:19:26 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-osd-8[55969]: 2024-05-27T07:19:26.031+0000 7f4259ae7700 -1 osd.8 53 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-05-27T07:19:26.363 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:19:26 smithi052 ceph-mon[31955]: from='osd.8 ' entity='osd.8' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]': finished 2024-05-27T07:19:26.363 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:19:26 smithi052 ceph-mon[31955]: osdmap e52: 9 total, 8 up, 9 in 2024-05-27T07:19:26.364 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:19:26 smithi052 ceph-mon[31955]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-05-27T07:19:26.364 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:19:26 smithi052 ceph-mon[31955]: from='osd.8 [v2:172.21.15.173:6816/4131399852,v1:172.21.15.173:6817/4131399852]' entity='osd.8' cmd=[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi173", "root=default"]}]: dispatch 2024-05-27T07:19:26.364 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:19:26 smithi052 ceph-mon[31955]: from='osd.8 ' entity='osd.8' cmd=[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi173", "root=default"]}]: dispatch 2024-05-27T07:19:26.440 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:19:26 smithi105 ceph-mon[35977]: from='osd.8 ' entity='osd.8' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]': finished 2024-05-27T07:19:26.440 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:19:26 smithi105 ceph-mon[35977]: osdmap e52: 9 total, 8 up, 9 in 2024-05-27T07:19:26.441 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:19:26 smithi105 ceph-mon[35977]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-05-27T07:19:26.441 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:19:26 smithi105 ceph-mon[35977]: from='osd.8 [v2:172.21.15.173:6816/4131399852,v1:172.21.15.173:6817/4131399852]' entity='osd.8' cmd=[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi173", "root=default"]}]: dispatch 2024-05-27T07:19:26.441 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:19:26 smithi105 ceph-mon[35977]: from='osd.8 ' entity='osd.8' cmd=[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi173", "root=default"]}]: dispatch 2024-05-27T07:19:27.130 INFO:teuthology.orchestra.run.smithi052.stderr:--> Zapping: /dev/vg_nvme/lv_1 2024-05-27T07:19:27.130 INFO:teuthology.orchestra.run.smithi052.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_1 bs=1M count=10 conv=fsync 2024-05-27T07:19:27.130 INFO:teuthology.orchestra.run.smithi052.stderr: stderr: 10+0 records in 2024-05-27T07:19:27.130 INFO:teuthology.orchestra.run.smithi052.stderr:10+0 records out 2024-05-27T07:19:27.131 INFO:teuthology.orchestra.run.smithi052.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.00897279 s, 1.2 GB/s 2024-05-27T07:19:27.131 INFO:teuthology.orchestra.run.smithi052.stderr: stderr: 2024-05-27T07:19:27.131 INFO:teuthology.orchestra.run.smithi052.stderr:--> Zapping successful for: 2024-05-27T07:19:27.352 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:19:27 smithi173 ceph-mon[37000]: pgmap v190: 1 pgs: 1 active+clean; 577 KiB data, 235 MiB used, 715 GiB / 715 GiB avail 2024-05-27T07:19:27.352 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:19:27 smithi173 ceph-mon[37000]: from='osd.8 ' entity='osd.8' cmd='[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi173", "root=default"]}]': finished 2024-05-27T07:19:27.352 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:19:27 smithi173 ceph-mon[37000]: osdmap e53: 9 total, 8 up, 9 in 2024-05-27T07:19:27.353 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:19:27 smithi173 ceph-mon[37000]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-05-27T07:19:27.353 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:19:27 smithi173 ceph-mon[37000]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-05-27T07:19:27.440 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:19:27 smithi105 ceph-mon[35977]: pgmap v190: 1 pgs: 1 active+clean; 577 KiB data, 235 MiB used, 715 GiB / 715 GiB avail 2024-05-27T07:19:27.440 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:19:27 smithi105 ceph-mon[35977]: from='osd.8 ' entity='osd.8' cmd='[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi173", "root=default"]}]': finished 2024-05-27T07:19:27.440 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:19:27 smithi105 ceph-mon[35977]: osdmap e53: 9 total, 8 up, 9 in 2024-05-27T07:19:27.440 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:19:27 smithi105 ceph-mon[35977]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-05-27T07:19:27.440 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:19:27 smithi105 ceph-mon[35977]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-05-27T07:19:27.447 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:19:27 smithi052 ceph-mon[31955]: pgmap v190: 1 pgs: 1 active+clean; 577 KiB data, 235 MiB used, 715 GiB / 715 GiB avail 2024-05-27T07:19:27.447 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:19:27 smithi052 ceph-mon[31955]: from='osd.8 ' entity='osd.8' cmd='[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi173", "root=default"]}]': finished 2024-05-27T07:19:27.448 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:19:27 smithi052 ceph-mon[31955]: osdmap e53: 9 total, 8 up, 9 in 2024-05-27T07:19:27.448 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:19:27 smithi052 ceph-mon[31955]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-05-27T07:19:27.448 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:19:27 smithi052 ceph-mon[31955]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-05-27T07:19:27.838 DEBUG:teuthology.orchestra.run.smithi052:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:38f9d900c83c63799fbdbe61acc9a11b0d3554a6 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 73df2636-1bf8-11ef-bc9b-c7b262605968 -- ceph orch daemon add osd smithi052:vg_nvme/lv_1 2024-05-27T07:19:28.095 INFO:teuthology.orchestra.run.smithi052.stderr:Inferring config /var/lib/ceph/73df2636-1bf8-11ef-bc9b-c7b262605968/mon.a/config 2024-05-27T07:19:28.194 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:19:28 smithi173 ceph-mon[37000]: purged_snaps scrub starts 2024-05-27T07:19:28.195 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:19:28 smithi173 ceph-mon[37000]: purged_snaps scrub ok 2024-05-27T07:19:28.195 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:19:28 smithi173 ceph-mon[37000]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-05-27T07:19:28.195 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:19:28 smithi173 ceph-mon[37000]: osd.8 [v2:172.21.15.173:6816/4131399852,v1:172.21.15.173:6817/4131399852] boot 2024-05-27T07:19:28.195 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:19:28 smithi173 ceph-mon[37000]: osdmap e54: 9 total, 9 up, 9 in 2024-05-27T07:19:28.195 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:19:28 smithi173 ceph-mon[37000]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-05-27T07:19:28.218 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:19:28 smithi052 ceph-mon[31955]: purged_snaps scrub starts 2024-05-27T07:19:28.218 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:19:28 smithi052 ceph-mon[31955]: purged_snaps scrub ok 2024-05-27T07:19:28.218 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:19:28 smithi052 ceph-mon[31955]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-05-27T07:19:28.219 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:19:28 smithi052 ceph-mon[31955]: osd.8 [v2:172.21.15.173:6816/4131399852,v1:172.21.15.173:6817/4131399852] boot 2024-05-27T07:19:28.219 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:19:28 smithi052 ceph-mon[31955]: osdmap e54: 9 total, 9 up, 9 in 2024-05-27T07:19:28.219 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:19:28 smithi052 ceph-mon[31955]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-05-27T07:19:28.440 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:19:28 smithi105 ceph-mon[35977]: purged_snaps scrub starts 2024-05-27T07:19:28.440 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:19:28 smithi105 ceph-mon[35977]: purged_snaps scrub ok 2024-05-27T07:19:28.441 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:19:28 smithi105 ceph-mon[35977]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-05-27T07:19:28.441 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:19:28 smithi105 ceph-mon[35977]: osd.8 [v2:172.21.15.173:6816/4131399852,v1:172.21.15.173:6817/4131399852] boot 2024-05-27T07:19:28.441 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:19:28 smithi105 ceph-mon[35977]: osdmap e54: 9 total, 9 up, 9 in 2024-05-27T07:19:28.441 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:19:28 smithi105 ceph-mon[35977]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-05-27T07:19:29.288 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:19:29 smithi052 ceph-mon[31955]: pgmap v193: 1 pgs: 1 peering; 577 KiB data, 235 MiB used, 715 GiB / 715 GiB avail 2024-05-27T07:19:29.289 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:19:29 smithi052 ceph-mon[31955]: osdmap e55: 9 total, 9 up, 9 in 2024-05-27T07:19:29.289 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:19:29 smithi052 ceph-mon[31955]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' 2024-05-27T07:19:29.289 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:19:29 smithi052 ceph-mon[31955]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' 2024-05-27T07:19:29.289 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:19:29 smithi052 ceph-mon[31955]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-05-27T07:19:29.289 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:19:29 smithi052 ceph-mon[31955]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-05-27T07:19:29.289 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:19:29 smithi052 ceph-mon[31955]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-05-27T07:19:29.289 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:19:29 smithi052 ceph-mon[31955]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' 2024-05-27T07:19:29.289 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:19:29 smithi052 ceph-mon[31955]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T07:19:29.289 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:19:29 smithi052 ceph-mon[31955]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-27T07:19:29.289 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:19:29 smithi052 ceph-mon[31955]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' 2024-05-27T07:19:29.289 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:19:29 smithi052 ceph-mon[31955]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-27T07:19:29.290 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:19:29 smithi052 ceph-mon[31955]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T07:19:29.290 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:19:29 smithi052 ceph-mon[31955]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-27T07:19:29.290 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:19:29 smithi052 ceph-mon[31955]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' 2024-05-27T07:19:29.352 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:19:29 smithi173 ceph-mon[37000]: pgmap v193: 1 pgs: 1 peering; 577 KiB data, 235 MiB used, 715 GiB / 715 GiB avail 2024-05-27T07:19:29.352 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:19:29 smithi173 ceph-mon[37000]: osdmap e55: 9 total, 9 up, 9 in 2024-05-27T07:19:29.352 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:19:29 smithi173 ceph-mon[37000]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' 2024-05-27T07:19:29.352 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:19:29 smithi173 ceph-mon[37000]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' 2024-05-27T07:19:29.352 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:19:29 smithi173 ceph-mon[37000]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-05-27T07:19:29.353 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:19:29 smithi173 ceph-mon[37000]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-05-27T07:19:29.353 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:19:29 smithi173 ceph-mon[37000]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-05-27T07:19:29.353 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:19:29 smithi173 ceph-mon[37000]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' 2024-05-27T07:19:29.353 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:19:29 smithi173 ceph-mon[37000]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T07:19:29.353 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:19:29 smithi173 ceph-mon[37000]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-27T07:19:29.353 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:19:29 smithi173 ceph-mon[37000]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' 2024-05-27T07:19:29.353 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:19:29 smithi173 ceph-mon[37000]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-27T07:19:29.353 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:19:29 smithi173 ceph-mon[37000]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T07:19:29.353 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:19:29 smithi173 ceph-mon[37000]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-27T07:19:29.353 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:19:29 smithi173 ceph-mon[37000]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' 2024-05-27T07:19:29.440 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:19:29 smithi105 ceph-mon[35977]: pgmap v193: 1 pgs: 1 peering; 577 KiB data, 235 MiB used, 715 GiB / 715 GiB avail 2024-05-27T07:19:29.440 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:19:29 smithi105 ceph-mon[35977]: osdmap e55: 9 total, 9 up, 9 in 2024-05-27T07:19:29.441 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:19:29 smithi105 ceph-mon[35977]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' 2024-05-27T07:19:29.441 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:19:29 smithi105 ceph-mon[35977]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' 2024-05-27T07:19:29.441 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:19:29 smithi105 ceph-mon[35977]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-05-27T07:19:29.441 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:19:29 smithi105 ceph-mon[35977]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-05-27T07:19:29.441 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:19:29 smithi105 ceph-mon[35977]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-05-27T07:19:29.441 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:19:29 smithi105 ceph-mon[35977]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' 2024-05-27T07:19:29.441 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:19:29 smithi105 ceph-mon[35977]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T07:19:29.441 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:19:29 smithi105 ceph-mon[35977]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-27T07:19:29.441 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:19:29 smithi105 ceph-mon[35977]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' 2024-05-27T07:19:29.441 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:19:29 smithi105 ceph-mon[35977]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-27T07:19:29.441 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:19:29 smithi105 ceph-mon[35977]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T07:19:29.441 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:19:29 smithi105 ceph-mon[35977]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-27T07:19:29.441 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:19:29 smithi105 ceph-mon[35977]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' 2024-05-27T07:19:29.679 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:19:29.688+0000 7f35dc564700 1 -- 172.21.15.52:0/4096405268 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f35d410c1e0 msgr2=0x7f35d410e6f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:19:29.679 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:19:29.688+0000 7f35dc564700 1 --2- 172.21.15.52:0/4096405268 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f35d410c1e0 0x7f35d410e6f0 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7f35cc00b0a0 tx=0x7f35cc00b3b0 comp rx=0 tx=0).stop 2024-05-27T07:19:29.679 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:19:29.688+0000 7f35dc564700 1 -- 172.21.15.52:0/4096405268 shutdown_connections 2024-05-27T07:19:29.679 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:19:29.688+0000 7f35dc564700 1 --2- 172.21.15.52:0/4096405268 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f35d410c1e0 0x7f35d410e6f0 unknown :-1 s=CLOSED pgs=139 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:19:29.679 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:19:29.688+0000 7f35dc564700 1 --2- 172.21.15.52:0/4096405268 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f35d406ccd0 0x7f35d41092b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:19:29.679 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:19:29.688+0000 7f35dc564700 1 --2- 172.21.15.52:0/4096405268 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f35d41097f0 0x7f35d410bc10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:19:29.679 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:19:29.688+0000 7f35dc564700 1 -- 172.21.15.52:0/4096405268 >> 172.21.15.52:0/4096405268 conn(0x7f35d40fe860 msgr2=0x7f35d4100d10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:19:29.679 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:19:29.688+0000 7f35dc564700 1 -- 172.21.15.52:0/4096405268 shutdown_connections 2024-05-27T07:19:29.680 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:19:29.688+0000 7f35dc564700 1 -- 172.21.15.52:0/4096405268 wait complete. 2024-05-27T07:19:29.681 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:19:29.689+0000 7f35dc564700 1 Processor -- start 2024-05-27T07:19:29.682 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:19:29.689+0000 7f35dc564700 1 -- start start 2024-05-27T07:19:29.682 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:19:29.689+0000 7f35dc564700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f35d406ccd0 0x7f35d419cab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:19:29.682 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:19:29.689+0000 7f35dc564700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f35d41097f0 0x7f35d419cff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:19:29.682 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:19:29.689+0000 7f35dc564700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f35d410c1e0 0x7f35d41a3f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:19:29.682 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:19:29.689+0000 7f35dc564700 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x7f35d4111fd0 con 0x7f35d406ccd0 2024-05-27T07:19:29.682 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:19:29.689+0000 7f35dc564700 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f35d4111e50 con 0x7f35d41097f0 2024-05-27T07:19:29.682 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:19:29.689+0000 7f35dc564700 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f35d4112150 con 0x7f35d410c1e0 2024-05-27T07:19:29.682 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:19:29.689+0000 7f35da300700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f35d406ccd0 0x7f35d419cab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:19:29.682 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:19:29.689+0000 7f35da300700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f35d406ccd0 0x7f35d419cab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.52:51038/0 (socket says 172.21.15.52:51038) 2024-05-27T07:19:29.682 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:19:29.689+0000 7f35da300700 1 -- 172.21.15.52:0/74070166 learned_addr learned my addr 172.21.15.52:0/74070166 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-05-27T07:19:29.682 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:19:29.690+0000 7f35d9aff700 1 --2- 172.21.15.52:0/74070166 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f35d41097f0 0x7f35d419cff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:19:29.683 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:19:29.690+0000 7f35dab01700 1 --2- 172.21.15.52:0/74070166 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f35d410c1e0 0x7f35d41a3f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:19:29.683 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:19:29.690+0000 7f35da300700 1 -- 172.21.15.52:0/74070166 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f35d410c1e0 msgr2=0x7f35d41a3f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:19:29.683 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:19:29.690+0000 7f35da300700 1 --2- 172.21.15.52:0/74070166 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f35d410c1e0 0x7f35d41a3f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:19:29.683 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:19:29.690+0000 7f35da300700 1 -- 172.21.15.52:0/74070166 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f35d41097f0 msgr2=0x7f35d419cff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:19:29.683 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:19:29.690+0000 7f35da300700 1 --2- 172.21.15.52:0/74070166 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f35d41097f0 0x7f35d419cff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:19:29.683 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:19:29.690+0000 7f35da300700 1 -- 172.21.15.52:0/74070166 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f35d41a4610 con 0x7f35d406ccd0 2024-05-27T07:19:29.683 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:19:29.690+0000 7f35da300700 1 --2- 172.21.15.52:0/74070166 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f35d406ccd0 0x7f35d419cab0 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7f35d000c8f0 tx=0x7f35d000ccb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:19:29.683 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:19:29.690+0000 7f35cb7fe700 1 -- 172.21.15.52:0/74070166 <== mon.0 v2:172.21.15.52:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f35d0007b00 con 0x7f35d406ccd0 2024-05-27T07:19:29.683 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:19:29.690+0000 7f35cb7fe700 1 -- 172.21.15.52:0/74070166 <== mon.0 v2:172.21.15.52:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f35d0007ca0 con 0x7f35d406ccd0 2024-05-27T07:19:29.683 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:19:29.690+0000 7f35cb7fe700 1 -- 172.21.15.52:0/74070166 <== mon.0 v2:172.21.15.52:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f35d0018660 con 0x7f35d406ccd0 2024-05-27T07:19:29.683 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:19:29.690+0000 7f35dc564700 1 -- 172.21.15.52:0/74070166 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f35d41a4900 con 0x7f35d406ccd0 2024-05-27T07:19:29.683 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:19:29.690+0000 7f35dc564700 1 -- 172.21.15.52:0/74070166 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f35d41a4e40 con 0x7f35d406ccd0 2024-05-27T07:19:29.683 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:19:29.691+0000 7f35dc564700 1 -- 172.21.15.52:0/74070166 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f35d40512b0 con 0x7f35d406ccd0 2024-05-27T07:19:29.684 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:19:29.693+0000 7f35cb7fe700 1 -- 172.21.15.52:0/74070166 <== mon.0 v2:172.21.15.52:3300/0 4 ==== mgrmap(e 16) ==== 149652+0+0 (secure 0 0 0) 0x7f35d000f450 con 0x7f35d406ccd0 2024-05-27T07:19:29.684 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:19:29.694+0000 7f35cb7fe700 1 --2- 172.21.15.52:0/74070166 >> [v2:172.21.15.52:6800/3513075805,v1:172.21.15.52:6801/3513075805] conn(0x7f35c00b2880 0x7f35c00b4d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:19:29.684 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:19:29.694+0000 7f35d9aff700 1 --2- 172.21.15.52:0/74070166 >> [v2:172.21.15.52:6800/3513075805,v1:172.21.15.52:6801/3513075805] conn(0x7f35c00b2880 0x7f35c00b4d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:19:29.684 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:19:29.694+0000 7f35cb7fe700 1 -- 172.21.15.52:0/74070166 <== mon.0 v2:172.21.15.52:3300/0 5 ==== osd_map(56..56 src has 1..56) ==== 5829+0+0 (secure 0 0 0) 0x7f35d00e19e0 con 0x7f35d406ccd0 2024-05-27T07:19:29.685 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:19:29.694+0000 7f35d9aff700 1 --2- 172.21.15.52:0/74070166 >> [v2:172.21.15.52:6800/3513075805,v1:172.21.15.52:6801/3513075805] conn(0x7f35c00b2880 0x7f35c00b4d90 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f35c4005950 tx=0x7f35c4005ed0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:19:29.687 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:19:29.697+0000 7f35cb7fe700 1 -- 172.21.15.52:0/74070166 <== mon.0 v2:172.21.15.52:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191963 (secure 0 0 0) 0x7f35d0014030 con 0x7f35d406ccd0 2024-05-27T07:19:29.915 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:19:29.925+0000 7f35dc564700 1 -- 172.21.15.52:0/74070166 --> [v2:172.21.15.52:6800/3513075805,v1:172.21.15.52:6801/3513075805] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi052:vg_nvme/lv_1", "target": ["mon-mgr", ""]}) -- 0x7f35d4063bf0 con 0x7f35c00b2880 2024-05-27T07:19:30.229 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:19:30 smithi052 ceph-mon[31955]: Detected new or changed devices on smithi173 2024-05-27T07:19:30.230 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:19:30 smithi052 ceph-mon[31955]: Adjusting osd_memory_target on smithi173 to 5696M 2024-05-27T07:19:30.230 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:19:30 smithi052 ceph-mon[31955]: osdmap e56: 9 total, 9 up, 9 in 2024-05-27T07:19:30.230 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:19:30 smithi052 ceph-mon[31955]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-05-27T07:19:30.230 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:19:30 smithi052 ceph-mon[31955]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-05-27T07:19:30.230 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:19:30 smithi052 ceph-mon[31955]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T07:19:30.352 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:19:30 smithi173 ceph-mon[37000]: Detected new or changed devices on smithi173 2024-05-27T07:19:30.353 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:19:30 smithi173 ceph-mon[37000]: Adjusting osd_memory_target on smithi173 to 5696M 2024-05-27T07:19:30.353 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:19:30 smithi173 ceph-mon[37000]: osdmap e56: 9 total, 9 up, 9 in 2024-05-27T07:19:30.353 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:19:30 smithi173 ceph-mon[37000]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-05-27T07:19:30.353 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:19:30 smithi173 ceph-mon[37000]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-05-27T07:19:30.353 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:19:30 smithi173 ceph-mon[37000]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T07:19:30.440 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:19:30 smithi105 ceph-mon[35977]: Detected new or changed devices on smithi173 2024-05-27T07:19:30.440 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:19:30 smithi105 ceph-mon[35977]: Adjusting osd_memory_target on smithi173 to 5696M 2024-05-27T07:19:30.440 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:19:30 smithi105 ceph-mon[35977]: osdmap e56: 9 total, 9 up, 9 in 2024-05-27T07:19:30.441 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:19:30 smithi105 ceph-mon[35977]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-05-27T07:19:30.441 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:19:30 smithi105 ceph-mon[35977]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-05-27T07:19:30.441 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:19:30 smithi105 ceph-mon[35977]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T07:19:31.229 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:19:31 smithi052 ceph-mon[31955]: pgmap v196: 1 pgs: 1 peering; 577 KiB data, 262 MiB used, 804 GiB / 805 GiB avail 2024-05-27T07:19:31.230 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:19:31 smithi052 ceph-mon[31955]: from='client.14433 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi052:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-05-27T07:19:31.352 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:19:31 smithi173 ceph-mon[37000]: pgmap v196: 1 pgs: 1 peering; 577 KiB data, 262 MiB used, 804 GiB / 805 GiB avail 2024-05-27T07:19:31.352 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:19:31 smithi173 ceph-mon[37000]: from='client.14433 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi052:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-05-27T07:19:31.440 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:19:31 smithi105 ceph-mon[35977]: pgmap v196: 1 pgs: 1 peering; 577 KiB data, 262 MiB used, 804 GiB / 805 GiB avail 2024-05-27T07:19:31.440 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:19:31 smithi105 ceph-mon[35977]: from='client.14433 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi052:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-05-27T07:19:33.352 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:19:33 smithi173 ceph-mon[37000]: pgmap v197: 1 pgs: 1 peering; 577 KiB data, 263 MiB used, 804 GiB / 805 GiB avail 2024-05-27T07:19:33.352 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:19:33 smithi173 ceph-mon[37000]: from='client.? 172.21.15.52:0/2945043049' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "78fa7f17-1dad-4262-be40-31fa82e75520"}]: dispatch 2024-05-27T07:19:33.352 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:19:33 smithi173 ceph-mon[37000]: from='client.? 172.21.15.52:0/2945043049' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "78fa7f17-1dad-4262-be40-31fa82e75520"}]': finished 2024-05-27T07:19:33.353 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:19:33 smithi173 ceph-mon[37000]: osdmap e57: 10 total, 9 up, 10 in 2024-05-27T07:19:33.353 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:19:33 smithi173 ceph-mon[37000]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-05-27T07:19:33.440 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:19:33 smithi105 ceph-mon[35977]: pgmap v197: 1 pgs: 1 peering; 577 KiB data, 263 MiB used, 804 GiB / 805 GiB avail 2024-05-27T07:19:33.440 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:19:33 smithi105 ceph-mon[35977]: from='client.? 172.21.15.52:0/2945043049' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "78fa7f17-1dad-4262-be40-31fa82e75520"}]: dispatch 2024-05-27T07:19:33.440 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:19:33 smithi105 ceph-mon[35977]: from='client.? 172.21.15.52:0/2945043049' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "78fa7f17-1dad-4262-be40-31fa82e75520"}]': finished 2024-05-27T07:19:33.440 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:19:33 smithi105 ceph-mon[35977]: osdmap e57: 10 total, 9 up, 10 in 2024-05-27T07:19:33.440 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:19:33 smithi105 ceph-mon[35977]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-05-27T07:19:33.480 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:19:33 smithi052 ceph-mon[31955]: pgmap v197: 1 pgs: 1 peering; 577 KiB data, 263 MiB used, 804 GiB / 805 GiB avail 2024-05-27T07:19:33.480 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:19:33 smithi052 ceph-mon[31955]: from='client.? 172.21.15.52:0/2945043049' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "78fa7f17-1dad-4262-be40-31fa82e75520"}]: dispatch 2024-05-27T07:19:33.480 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:19:33 smithi052 ceph-mon[31955]: from='client.? 172.21.15.52:0/2945043049' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "78fa7f17-1dad-4262-be40-31fa82e75520"}]': finished 2024-05-27T07:19:33.480 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:19:33 smithi052 ceph-mon[31955]: osdmap e57: 10 total, 9 up, 10 in 2024-05-27T07:19:33.480 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:19:33 smithi052 ceph-mon[31955]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-05-27T07:19:34.352 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:19:34 smithi173 ceph-mon[37000]: from='client.? 172.21.15.52:0/3838988391' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-05-27T07:19:34.440 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:19:34 smithi105 ceph-mon[35977]: from='client.? 172.21.15.52:0/3838988391' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-05-27T07:19:34.479 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:19:34 smithi052 ceph-mon[31955]: from='client.? 172.21.15.52:0/3838988391' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-05-27T07:19:35.352 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:19:35 smithi173 ceph-mon[37000]: pgmap v199: 1 pgs: 1 active+clean; 577 KiB data, 263 MiB used, 804 GiB / 805 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-05-27T07:19:35.440 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:19:35 smithi105 ceph-mon[35977]: pgmap v199: 1 pgs: 1 active+clean; 577 KiB data, 263 MiB used, 804 GiB / 805 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-05-27T07:19:35.479 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:19:35 smithi052 ceph-mon[31955]: pgmap v199: 1 pgs: 1 active+clean; 577 KiB data, 263 MiB used, 804 GiB / 805 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-05-27T07:19:37.229 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:19:37 smithi052 ceph-mon[31955]: pgmap v200: 1 pgs: 1 active+clean; 577 KiB data, 263 MiB used, 804 GiB / 805 GiB avail; 76 KiB/s, 0 objects/s recovering 2024-05-27T07:19:37.352 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:19:37 smithi173 ceph-mon[37000]: pgmap v200: 1 pgs: 1 active+clean; 577 KiB data, 263 MiB used, 804 GiB / 805 GiB avail; 76 KiB/s, 0 objects/s recovering 2024-05-27T07:19:37.440 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:19:37 smithi105 ceph-mon[35977]: pgmap v200: 1 pgs: 1 active+clean; 577 KiB data, 263 MiB used, 804 GiB / 805 GiB avail; 76 KiB/s, 0 objects/s recovering 2024-05-27T07:19:39.352 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:19:39 smithi173 ceph-mon[37000]: pgmap v201: 1 pgs: 1 active+clean; 577 KiB data, 263 MiB used, 804 GiB / 805 GiB avail; 67 KiB/s, 0 objects/s recovering 2024-05-27T07:19:39.359 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:19:39 smithi052 ceph-mon[31955]: pgmap v201: 1 pgs: 1 active+clean; 577 KiB data, 263 MiB used, 804 GiB / 805 GiB avail; 67 KiB/s, 0 objects/s recovering 2024-05-27T07:19:39.440 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:19:39 smithi105 ceph-mon[35977]: pgmap v201: 1 pgs: 1 active+clean; 577 KiB data, 263 MiB used, 804 GiB / 805 GiB avail; 67 KiB/s, 0 objects/s recovering 2024-05-27T07:19:41.352 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:19:41 smithi173 ceph-mon[37000]: pgmap v202: 1 pgs: 1 active+clean; 577 KiB data, 263 MiB used, 804 GiB / 805 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-05-27T07:19:41.352 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:19:41 smithi173 ceph-mon[37000]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.9"}]: dispatch 2024-05-27T07:19:41.352 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:19:41 smithi173 ceph-mon[37000]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T07:19:41.440 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:19:41 smithi052 ceph-mon[31955]: pgmap v202: 1 pgs: 1 active+clean; 577 KiB data, 263 MiB used, 804 GiB / 805 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-05-27T07:19:41.440 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:19:41 smithi052 ceph-mon[31955]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.9"}]: dispatch 2024-05-27T07:19:41.440 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:19:41 smithi052 ceph-mon[31955]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T07:19:41.440 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:19:41 smithi105 ceph-mon[35977]: pgmap v202: 1 pgs: 1 active+clean; 577 KiB data, 263 MiB used, 804 GiB / 805 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-05-27T07:19:41.440 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:19:41 smithi105 ceph-mon[35977]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.9"}]: dispatch 2024-05-27T07:19:41.440 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:19:41 smithi105 ceph-mon[35977]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T07:19:42.352 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:19:42 smithi173 ceph-mon[37000]: Deploying daemon osd.9 on smithi052 2024-05-27T07:19:42.440 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:19:42 smithi105 ceph-mon[35977]: Deploying daemon osd.9 on smithi052 2024-05-27T07:19:42.479 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:19:42 smithi052 ceph-mon[31955]: Deploying daemon osd.9 on smithi052 2024-05-27T07:19:43.230 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:19:43 smithi052 ceph-mon[31955]: pgmap v203: 1 pgs: 1 active+clean; 577 KiB data, 263 MiB used, 804 GiB / 805 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-05-27T07:19:43.352 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:19:43 smithi173 ceph-mon[37000]: pgmap v203: 1 pgs: 1 active+clean; 577 KiB data, 263 MiB used, 804 GiB / 805 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-05-27T07:19:43.440 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:19:43 smithi105 ceph-mon[35977]: pgmap v203: 1 pgs: 1 active+clean; 577 KiB data, 263 MiB used, 804 GiB / 805 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-05-27T07:19:45.230 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:19:45 smithi052 ceph-mon[31955]: pgmap v204: 1 pgs: 1 active+clean; 577 KiB data, 263 MiB used, 804 GiB / 805 GiB avail 2024-05-27T07:19:45.352 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:19:45 smithi173 ceph-mon[37000]: pgmap v204: 1 pgs: 1 active+clean; 577 KiB data, 263 MiB used, 804 GiB / 805 GiB avail 2024-05-27T07:19:45.440 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:19:45 smithi105 ceph-mon[35977]: pgmap v204: 1 pgs: 1 active+clean; 577 KiB data, 263 MiB used, 804 GiB / 805 GiB avail 2024-05-27T07:19:47.229 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:19:47 smithi052 ceph-mon[31955]: pgmap v205: 1 pgs: 1 active+clean; 577 KiB data, 263 MiB used, 804 GiB / 805 GiB avail 2024-05-27T07:19:47.352 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:19:47 smithi173 ceph-mon[37000]: pgmap v205: 1 pgs: 1 active+clean; 577 KiB data, 263 MiB used, 804 GiB / 805 GiB avail 2024-05-27T07:19:47.442 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:19:47 smithi105 ceph-mon[35977]: pgmap v205: 1 pgs: 1 active+clean; 577 KiB data, 263 MiB used, 804 GiB / 805 GiB avail 2024-05-27T07:19:48.440 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:19:48 smithi105 ceph-mon[35977]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' 2024-05-27T07:19:48.440 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:19:48 smithi105 ceph-mon[35977]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' 2024-05-27T07:19:48.440 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:19:48 smithi105 ceph-mon[35977]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-27T07:19:48.440 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:19:48 smithi105 ceph-mon[35977]: pgmap v206: 1 pgs: 1 active+clean; 577 KiB data, 263 MiB used, 804 GiB / 805 GiB avail 2024-05-27T07:19:48.479 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:19:48 smithi052 ceph-mon[31955]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' 2024-05-27T07:19:48.480 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:19:48 smithi052 ceph-mon[31955]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' 2024-05-27T07:19:48.480 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:19:48 smithi052 ceph-mon[31955]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-27T07:19:48.480 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:19:48 smithi052 ceph-mon[31955]: pgmap v206: 1 pgs: 1 active+clean; 577 KiB data, 263 MiB used, 804 GiB / 805 GiB avail 2024-05-27T07:19:48.602 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:19:48 smithi173 ceph-mon[37000]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' 2024-05-27T07:19:48.602 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:19:48 smithi173 ceph-mon[37000]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' 2024-05-27T07:19:48.602 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:19:48 smithi173 ceph-mon[37000]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-27T07:19:48.602 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:19:48 smithi173 ceph-mon[37000]: pgmap v206: 1 pgs: 1 active+clean; 577 KiB data, 263 MiB used, 804 GiB / 805 GiB avail 2024-05-27T07:19:50.940 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:19:50 smithi105 ceph-mon[35977]: pgmap v207: 1 pgs: 1 active+clean; 577 KiB data, 263 MiB used, 804 GiB / 805 GiB avail 2024-05-27T07:19:50.940 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:19:50 smithi105 ceph-mon[35977]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' 2024-05-27T07:19:50.940 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:19:50 smithi105 ceph-mon[35977]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' 2024-05-27T07:19:50.941 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:19:50 smithi105 ceph-mon[35977]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T07:19:50.941 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:19:50 smithi105 ceph-mon[35977]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-27T07:19:50.941 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:19:50 smithi105 ceph-mon[35977]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' 2024-05-27T07:19:50.958 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:19:50 smithi052 ceph-mon[31955]: pgmap v207: 1 pgs: 1 active+clean; 577 KiB data, 263 MiB used, 804 GiB / 805 GiB avail 2024-05-27T07:19:50.959 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:19:50 smithi052 ceph-mon[31955]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' 2024-05-27T07:19:50.959 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:19:50 smithi052 ceph-mon[31955]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' 2024-05-27T07:19:50.959 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:19:50 smithi052 ceph-mon[31955]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T07:19:50.959 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:19:50 smithi052 ceph-mon[31955]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-27T07:19:50.959 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:19:50 smithi052 ceph-mon[31955]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' 2024-05-27T07:19:51.033 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:19:51.042+0000 7f35cb7fe700 1 -- 172.21.15.52:0/74070166 <== mgr.14152 v2:172.21.15.52:6800/3513075805 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+37 (secure 0 0 0) 0x7f35d4063bf0 con 0x7f35c00b2880 2024-05-27T07:19:51.033 INFO:teuthology.orchestra.run.smithi052.stdout:Created osd(s) 9 on host 'smithi052' 2024-05-27T07:19:51.035 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:19:51.045+0000 7f35dc564700 1 -- 172.21.15.52:0/74070166 >> [v2:172.21.15.52:6800/3513075805,v1:172.21.15.52:6801/3513075805] conn(0x7f35c00b2880 msgr2=0x7f35c00b4d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:19:51.035 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:19:51.045+0000 7f35dc564700 1 --2- 172.21.15.52:0/74070166 >> [v2:172.21.15.52:6800/3513075805,v1:172.21.15.52:6801/3513075805] conn(0x7f35c00b2880 0x7f35c00b4d90 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f35c4005950 tx=0x7f35c4005ed0 comp rx=0 tx=0).stop 2024-05-27T07:19:51.035 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:19:51.045+0000 7f35dc564700 1 -- 172.21.15.52:0/74070166 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f35d406ccd0 msgr2=0x7f35d419cab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:19:51.035 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:19:51.045+0000 7f35dc564700 1 --2- 172.21.15.52:0/74070166 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f35d406ccd0 0x7f35d419cab0 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7f35d000c8f0 tx=0x7f35d000ccb0 comp rx=0 tx=0).stop 2024-05-27T07:19:51.036 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:19:51.045+0000 7f35dc564700 1 -- 172.21.15.52:0/74070166 shutdown_connections 2024-05-27T07:19:51.036 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:19:51.045+0000 7f35dc564700 1 --2- 172.21.15.52:0/74070166 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f35d410c1e0 0x7f35d41a3f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:19:51.036 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:19:51.045+0000 7f35dc564700 1 --2- 172.21.15.52:0/74070166 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f35d406ccd0 0x7f35d419cab0 unknown :-1 s=CLOSED pgs=140 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:19:51.036 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:19:51.045+0000 7f35dc564700 1 --2- 172.21.15.52:0/74070166 >> [v2:172.21.15.52:6800/3513075805,v1:172.21.15.52:6801/3513075805] conn(0x7f35c00b2880 0x7f35c00b4d90 unknown :-1 s=CLOSED pgs=97 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:19:51.036 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:19:51.045+0000 7f35dc564700 1 --2- 172.21.15.52:0/74070166 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f35d41097f0 0x7f35d419cff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:19:51.036 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:19:51.045+0000 7f35dc564700 1 -- 172.21.15.52:0/74070166 >> 172.21.15.52:0/74070166 conn(0x7f35d40fe860 msgr2=0x7f35d410a6f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:19:51.036 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:19:51.045+0000 7f35dc564700 1 -- 172.21.15.52:0/74070166 shutdown_connections 2024-05-27T07:19:51.036 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:19:51.045+0000 7f35dc564700 1 -- 172.21.15.52:0/74070166 wait complete. 2024-05-27T07:19:51.102 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:19:50 smithi173 ceph-mon[37000]: pgmap v207: 1 pgs: 1 active+clean; 577 KiB data, 263 MiB used, 804 GiB / 805 GiB avail 2024-05-27T07:19:51.102 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:19:50 smithi173 ceph-mon[37000]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' 2024-05-27T07:19:51.102 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:19:50 smithi173 ceph-mon[37000]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' 2024-05-27T07:19:51.102 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:19:50 smithi173 ceph-mon[37000]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T07:19:51.102 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:19:50 smithi173 ceph-mon[37000]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-27T07:19:51.103 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:19:50 smithi173 ceph-mon[37000]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' 2024-05-27T07:19:51.715 DEBUG:teuthology.orchestra.run.smithi052:osd.9> sudo journalctl -f -n 0 -u ceph-73df2636-1bf8-11ef-bc9b-c7b262605968@osd.9.service 2024-05-27T07:19:51.718 INFO:tasks.cephadm:Deploying osd.10 on smithi105 with /dev/vg_nvme/lv_1... 2024-05-27T07:19:51.718 DEBUG:teuthology.orchestra.run.smithi105:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:38f9d900c83c63799fbdbe61acc9a11b0d3554a6 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 73df2636-1bf8-11ef-bc9b-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_1 2024-05-27T07:19:51.963 INFO:teuthology.orchestra.run.smithi105.stderr:Inferring config /var/lib/ceph/73df2636-1bf8-11ef-bc9b-c7b262605968/mon.b/config 2024-05-27T07:19:52.084 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:19:52 smithi105 ceph-mon[35977]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' 2024-05-27T07:19:52.085 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:19:52 smithi105 ceph-mon[35977]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' 2024-05-27T07:19:52.085 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:19:52 smithi105 ceph-mon[35977]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-27T07:19:52.085 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:19:52 smithi105 ceph-mon[35977]: from='osd.9 [v2:172.21.15.52:6826/479119778,v1:172.21.15.52:6827/479119778]' entity='osd.9' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["9"]}]: dispatch 2024-05-27T07:19:52.352 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:19:52 smithi173 ceph-mon[37000]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' 2024-05-27T07:19:52.352 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:19:52 smithi173 ceph-mon[37000]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' 2024-05-27T07:19:52.352 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:19:52 smithi173 ceph-mon[37000]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-27T07:19:52.352 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:19:52 smithi173 ceph-mon[37000]: from='osd.9 [v2:172.21.15.52:6826/479119778,v1:172.21.15.52:6827/479119778]' entity='osd.9' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["9"]}]: dispatch 2024-05-27T07:19:52.480 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:19:52 smithi052 ceph-mon[31955]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' 2024-05-27T07:19:52.480 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:19:52 smithi052 ceph-mon[31955]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' 2024-05-27T07:19:52.480 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:19:52 smithi052 ceph-mon[31955]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-27T07:19:52.480 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:19:52 smithi052 ceph-mon[31955]: from='osd.9 [v2:172.21.15.52:6826/479119778,v1:172.21.15.52:6827/479119778]' entity='osd.9' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["9"]}]: dispatch 2024-05-27T07:19:53.229 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:19:53 smithi052 ceph-mon[31955]: pgmap v208: 1 pgs: 1 active+clean; 577 KiB data, 263 MiB used, 804 GiB / 805 GiB avail 2024-05-27T07:19:53.230 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:19:53 smithi052 ceph-mon[31955]: from='osd.9 [v2:172.21.15.52:6826/479119778,v1:172.21.15.52:6827/479119778]' entity='osd.9' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["9"]}]': finished 2024-05-27T07:19:53.230 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:19:53 smithi052 ceph-mon[31955]: osdmap e58: 10 total, 9 up, 10 in 2024-05-27T07:19:53.230 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:19:53 smithi052 ceph-mon[31955]: from='osd.9 [v2:172.21.15.52:6826/479119778,v1:172.21.15.52:6827/479119778]' entity='osd.9' cmd=[{"prefix": "osd crush create-or-move", "id": 9, "weight":0.0873, "args": ["host=smithi052", "root=default"]}]: dispatch 2024-05-27T07:19:53.230 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:19:53 smithi052 ceph-mon[31955]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-05-27T07:19:53.265 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:19:53 smithi105 ceph-mon[35977]: pgmap v208: 1 pgs: 1 active+clean; 577 KiB data, 263 MiB used, 804 GiB / 805 GiB avail 2024-05-27T07:19:53.266 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:19:53 smithi105 ceph-mon[35977]: from='osd.9 [v2:172.21.15.52:6826/479119778,v1:172.21.15.52:6827/479119778]' entity='osd.9' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["9"]}]': finished 2024-05-27T07:19:53.266 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:19:53 smithi105 ceph-mon[35977]: osdmap e58: 10 total, 9 up, 10 in 2024-05-27T07:19:53.266 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:19:53 smithi105 ceph-mon[35977]: from='osd.9 [v2:172.21.15.52:6826/479119778,v1:172.21.15.52:6827/479119778]' entity='osd.9' cmd=[{"prefix": "osd crush create-or-move", "id": 9, "weight":0.0873, "args": ["host=smithi052", "root=default"]}]: dispatch 2024-05-27T07:19:53.266 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:19:53 smithi105 ceph-mon[35977]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-05-27T07:19:53.352 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:19:53 smithi173 ceph-mon[37000]: pgmap v208: 1 pgs: 1 active+clean; 577 KiB data, 263 MiB used, 804 GiB / 805 GiB avail 2024-05-27T07:19:53.352 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:19:53 smithi173 ceph-mon[37000]: from='osd.9 [v2:172.21.15.52:6826/479119778,v1:172.21.15.52:6827/479119778]' entity='osd.9' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["9"]}]': finished 2024-05-27T07:19:53.352 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:19:53 smithi173 ceph-mon[37000]: osdmap e58: 10 total, 9 up, 10 in 2024-05-27T07:19:53.352 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:19:53 smithi173 ceph-mon[37000]: from='osd.9 [v2:172.21.15.52:6826/479119778,v1:172.21.15.52:6827/479119778]' entity='osd.9' cmd=[{"prefix": "osd crush create-or-move", "id": 9, "weight":0.0873, "args": ["host=smithi052", "root=default"]}]: dispatch 2024-05-27T07:19:53.352 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:19:53 smithi173 ceph-mon[37000]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-05-27T07:19:53.527 INFO:journalctl@ceph.osd.9.smithi052.stdout:May 27 07:19:53 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-osd-9[67134]: 2024-05-27T07:19:53.522+0000 7f1e44d1a700 -1 osd.9 0 waiting for initial osdmap 2024-05-27T07:19:53.528 INFO:journalctl@ceph.osd.9.smithi052.stdout:May 27 07:19:53 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-osd-9[67134]: 2024-05-27T07:19:53.536+0000 7f1e3cac7700 -1 osd.9 59 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-05-27T07:19:53.961 INFO:teuthology.orchestra.run.smithi105.stderr:--> Zapping: /dev/vg_nvme/lv_1 2024-05-27T07:19:53.961 INFO:teuthology.orchestra.run.smithi105.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_1 bs=1M count=10 conv=fsync 2024-05-27T07:19:53.961 INFO:teuthology.orchestra.run.smithi105.stderr: stderr: 10+0 records in 2024-05-27T07:19:53.961 INFO:teuthology.orchestra.run.smithi105.stderr:10+0 records out 2024-05-27T07:19:53.961 INFO:teuthology.orchestra.run.smithi105.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.0112928 s, 929 MB/s 2024-05-27T07:19:53.961 INFO:teuthology.orchestra.run.smithi105.stderr:--> Zapping successful for: 2024-05-27T07:19:54.265 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:19:54 smithi105 ceph-mon[35977]: from='osd.9 [v2:172.21.15.52:6826/479119778,v1:172.21.15.52:6827/479119778]' entity='osd.9' cmd='[{"prefix": "osd crush create-or-move", "id": 9, "weight":0.0873, "args": ["host=smithi052", "root=default"]}]': finished 2024-05-27T07:19:54.265 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:19:54 smithi105 ceph-mon[35977]: osdmap e59: 10 total, 9 up, 10 in 2024-05-27T07:19:54.265 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:19:54 smithi105 ceph-mon[35977]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-05-27T07:19:54.265 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:19:54 smithi105 ceph-mon[35977]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-05-27T07:19:54.265 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:19:54 smithi105 ceph-mon[35977]: from='osd.9 [v2:172.21.15.52:6826/479119778,v1:172.21.15.52:6827/479119778]' entity='osd.9' 2024-05-27T07:19:54.352 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:19:54 smithi173 ceph-mon[37000]: from='osd.9 [v2:172.21.15.52:6826/479119778,v1:172.21.15.52:6827/479119778]' entity='osd.9' cmd='[{"prefix": "osd crush create-or-move", "id": 9, "weight":0.0873, "args": ["host=smithi052", "root=default"]}]': finished 2024-05-27T07:19:54.352 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:19:54 smithi173 ceph-mon[37000]: osdmap e59: 10 total, 9 up, 10 in 2024-05-27T07:19:54.353 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:19:54 smithi173 ceph-mon[37000]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-05-27T07:19:54.353 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:19:54 smithi173 ceph-mon[37000]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-05-27T07:19:54.353 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:19:54 smithi173 ceph-mon[37000]: from='osd.9 [v2:172.21.15.52:6826/479119778,v1:172.21.15.52:6827/479119778]' entity='osd.9' 2024-05-27T07:19:54.454 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:19:54 smithi052 ceph-mon[31955]: from='osd.9 [v2:172.21.15.52:6826/479119778,v1:172.21.15.52:6827/479119778]' entity='osd.9' cmd='[{"prefix": "osd crush create-or-move", "id": 9, "weight":0.0873, "args": ["host=smithi052", "root=default"]}]': finished 2024-05-27T07:19:54.454 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:19:54 smithi052 ceph-mon[31955]: osdmap e59: 10 total, 9 up, 10 in 2024-05-27T07:19:54.454 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:19:54 smithi052 ceph-mon[31955]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-05-27T07:19:54.454 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:19:54 smithi052 ceph-mon[31955]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-05-27T07:19:54.454 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:19:54 smithi052 ceph-mon[31955]: from='osd.9 [v2:172.21.15.52:6826/479119778,v1:172.21.15.52:6827/479119778]' entity='osd.9' 2024-05-27T07:19:54.651 DEBUG:teuthology.orchestra.run.smithi105:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:38f9d900c83c63799fbdbe61acc9a11b0d3554a6 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 73df2636-1bf8-11ef-bc9b-c7b262605968 -- ceph orch daemon add osd smithi105:vg_nvme/lv_1 2024-05-27T07:19:54.894 INFO:teuthology.orchestra.run.smithi105.stderr:Inferring config /var/lib/ceph/73df2636-1bf8-11ef-bc9b-c7b262605968/mon.b/config 2024-05-27T07:19:55.230 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:19:55 smithi052 ceph-mon[31955]: purged_snaps scrub starts 2024-05-27T07:19:55.230 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:19:55 smithi052 ceph-mon[31955]: purged_snaps scrub ok 2024-05-27T07:19:55.230 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:19:55 smithi052 ceph-mon[31955]: pgmap v211: 1 pgs: 1 active+clean; 577 KiB data, 263 MiB used, 804 GiB / 805 GiB avail 2024-05-27T07:19:55.230 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:19:55 smithi052 ceph-mon[31955]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-05-27T07:19:55.230 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:19:55 smithi052 ceph-mon[31955]: osd.9 [v2:172.21.15.52:6826/479119778,v1:172.21.15.52:6827/479119778] boot 2024-05-27T07:19:55.230 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:19:55 smithi052 ceph-mon[31955]: osdmap e60: 10 total, 10 up, 10 in 2024-05-27T07:19:55.230 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:19:55 smithi052 ceph-mon[31955]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-05-27T07:19:55.230 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:19:55 smithi052 ceph-mon[31955]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' 2024-05-27T07:19:55.230 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:19:55 smithi052 ceph-mon[31955]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' 2024-05-27T07:19:55.230 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:19:55 smithi052 ceph-mon[31955]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-05-27T07:19:55.231 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:19:55 smithi052 ceph-mon[31955]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-05-27T07:19:55.231 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:19:55 smithi052 ceph-mon[31955]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-05-27T07:19:55.231 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:19:55 smithi052 ceph-mon[31955]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-05-27T07:19:55.231 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:19:55 smithi052 ceph-mon[31955]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' 2024-05-27T07:19:55.231 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:19:55 smithi052 ceph-mon[31955]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T07:19:55.231 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:19:55 smithi052 ceph-mon[31955]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-27T07:19:55.231 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:19:55 smithi052 ceph-mon[31955]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' 2024-05-27T07:19:55.352 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:19:55 smithi173 ceph-mon[37000]: purged_snaps scrub starts 2024-05-27T07:19:55.352 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:19:55 smithi173 ceph-mon[37000]: purged_snaps scrub ok 2024-05-27T07:19:55.353 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:19:55 smithi173 ceph-mon[37000]: pgmap v211: 1 pgs: 1 active+clean; 577 KiB data, 263 MiB used, 804 GiB / 805 GiB avail 2024-05-27T07:19:55.353 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:19:55 smithi173 ceph-mon[37000]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-05-27T07:19:55.353 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:19:55 smithi173 ceph-mon[37000]: osd.9 [v2:172.21.15.52:6826/479119778,v1:172.21.15.52:6827/479119778] boot 2024-05-27T07:19:55.353 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:19:55 smithi173 ceph-mon[37000]: osdmap e60: 10 total, 10 up, 10 in 2024-05-27T07:19:55.353 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:19:55 smithi173 ceph-mon[37000]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-05-27T07:19:55.353 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:19:55 smithi173 ceph-mon[37000]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' 2024-05-27T07:19:55.353 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:19:55 smithi173 ceph-mon[37000]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' 2024-05-27T07:19:55.353 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:19:55 smithi173 ceph-mon[37000]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-05-27T07:19:55.353 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:19:55 smithi173 ceph-mon[37000]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-05-27T07:19:55.353 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:19:55 smithi173 ceph-mon[37000]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-05-27T07:19:55.353 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:19:55 smithi173 ceph-mon[37000]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-05-27T07:19:55.354 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:19:55 smithi173 ceph-mon[37000]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' 2024-05-27T07:19:55.354 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:19:55 smithi173 ceph-mon[37000]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T07:19:55.354 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:19:55 smithi173 ceph-mon[37000]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-27T07:19:55.354 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:19:55 smithi173 ceph-mon[37000]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' 2024-05-27T07:19:55.354 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:19:55 smithi105 ceph-mon[35977]: purged_snaps scrub starts 2024-05-27T07:19:55.354 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:19:55 smithi105 ceph-mon[35977]: purged_snaps scrub ok 2024-05-27T07:19:55.354 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:19:55 smithi105 ceph-mon[35977]: pgmap v211: 1 pgs: 1 active+clean; 577 KiB data, 263 MiB used, 804 GiB / 805 GiB avail 2024-05-27T07:19:55.355 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:19:55 smithi105 ceph-mon[35977]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-05-27T07:19:55.355 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:19:55 smithi105 ceph-mon[35977]: osd.9 [v2:172.21.15.52:6826/479119778,v1:172.21.15.52:6827/479119778] boot 2024-05-27T07:19:55.355 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:19:55 smithi105 ceph-mon[35977]: osdmap e60: 10 total, 10 up, 10 in 2024-05-27T07:19:55.355 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:19:55 smithi105 ceph-mon[35977]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-05-27T07:19:55.355 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:19:55 smithi105 ceph-mon[35977]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' 2024-05-27T07:19:55.355 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:19:55 smithi105 ceph-mon[35977]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' 2024-05-27T07:19:55.355 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:19:55 smithi105 ceph-mon[35977]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-05-27T07:19:55.355 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:19:55 smithi105 ceph-mon[35977]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-05-27T07:19:55.355 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:19:55 smithi105 ceph-mon[35977]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-05-27T07:19:55.356 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:19:55 smithi105 ceph-mon[35977]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-05-27T07:19:55.356 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:19:55 smithi105 ceph-mon[35977]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' 2024-05-27T07:19:55.356 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:19:55 smithi105 ceph-mon[35977]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T07:19:55.356 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:19:55 smithi105 ceph-mon[35977]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-27T07:19:55.356 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:19:55 smithi105 ceph-mon[35977]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' 2024-05-27T07:19:56.335 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:19:56 smithi105 ceph-mon[35977]: Detected new or changed devices on smithi052 2024-05-27T07:19:56.335 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:19:56 smithi105 ceph-mon[35977]: Adjusting osd_memory_target on smithi052 to 4273M 2024-05-27T07:19:56.352 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:19:56 smithi173 ceph-mon[37000]: Detected new or changed devices on smithi052 2024-05-27T07:19:56.352 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:19:56 smithi173 ceph-mon[37000]: Adjusting osd_memory_target on smithi052 to 4273M 2024-05-27T07:19:56.433 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:19:56 smithi052 ceph-mon[31955]: Detected new or changed devices on smithi052 2024-05-27T07:19:56.433 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:19:56 smithi052 ceph-mon[31955]: Adjusting osd_memory_target on smithi052 to 4273M 2024-05-27T07:19:56.673 INFO:teuthology.orchestra.run.smithi105.stderr:2024-05-27T07:19:56.682+0000 7f6f70df8700 1 -- 172.21.15.105:0/946088913 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f6f6c1049d0 msgr2=0x7f6f6c10c6e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:19:56.673 INFO:teuthology.orchestra.run.smithi105.stderr:2024-05-27T07:19:56.682+0000 7f6f70df8700 1 --2- 172.21.15.105:0/946088913 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f6f6c1049d0 0x7f6f6c10c6e0 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7f6f5c00b0a0 tx=0x7f6f5c00b3b0 comp rx=0 tx=0).stop 2024-05-27T07:19:56.673 INFO:teuthology.orchestra.run.smithi105.stderr:2024-05-27T07:19:56.682+0000 7f6f70df8700 1 -- 172.21.15.105:0/946088913 shutdown_connections 2024-05-27T07:19:56.673 INFO:teuthology.orchestra.run.smithi105.stderr:2024-05-27T07:19:56.682+0000 7f6f70df8700 1 --2- 172.21.15.105:0/946088913 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f6f6c1049d0 0x7f6f6c10c6e0 unknown :-1 s=CLOSED pgs=35 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:19:56.673 INFO:teuthology.orchestra.run.smithi105.stderr:2024-05-27T07:19:56.682+0000 7f6f70df8700 1 --2- 172.21.15.105:0/946088913 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f6f6c1059a0 0x7f6f6c105db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:19:56.674 INFO:teuthology.orchestra.run.smithi105.stderr:2024-05-27T07:19:56.682+0000 7f6f70df8700 1 --2- 172.21.15.105:0/946088913 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f6f6c103f50 0x7f6f6c104360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:19:56.674 INFO:teuthology.orchestra.run.smithi105.stderr:2024-05-27T07:19:56.682+0000 7f6f70df8700 1 -- 172.21.15.105:0/946088913 >> 172.21.15.105:0/946088913 conn(0x7f6f6c0ff9c0 msgr2=0x7f6f6c101e50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:19:56.674 INFO:teuthology.orchestra.run.smithi105.stderr:2024-05-27T07:19:56.682+0000 7f6f70df8700 1 -- 172.21.15.105:0/946088913 shutdown_connections 2024-05-27T07:19:56.674 INFO:teuthology.orchestra.run.smithi105.stderr:2024-05-27T07:19:56.682+0000 7f6f70df8700 1 -- 172.21.15.105:0/946088913 wait complete. 2024-05-27T07:19:56.680 INFO:teuthology.orchestra.run.smithi105.stderr:2024-05-27T07:19:56.683+0000 7f6f70df8700 1 Processor -- start 2024-05-27T07:19:56.681 INFO:teuthology.orchestra.run.smithi105.stderr:2024-05-27T07:19:56.683+0000 7f6f70df8700 1 -- start start 2024-05-27T07:19:56.681 INFO:teuthology.orchestra.run.smithi105.stderr:2024-05-27T07:19:56.683+0000 7f6f70df8700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f6f6c103f50 0x7f6f6c19a9e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:19:56.681 INFO:teuthology.orchestra.run.smithi105.stderr:2024-05-27T07:19:56.683+0000 7f6f70df8700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f6f6c1049d0 0x7f6f6c19af20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:19:56.681 INFO:teuthology.orchestra.run.smithi105.stderr:2024-05-27T07:19:56.683+0000 7f6f70df8700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f6f6c1059a0 0x7f6f6c1a1ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:19:56.681 INFO:teuthology.orchestra.run.smithi105.stderr:2024-05-27T07:19:56.683+0000 7f6f70df8700 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x7f6f6c10ff30 con 0x7f6f6c103f50 2024-05-27T07:19:56.682 INFO:teuthology.orchestra.run.smithi105.stderr:2024-05-27T07:19:56.683+0000 7f6f70df8700 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f6f6c10fdb0 con 0x7f6f6c1049d0 2024-05-27T07:19:56.682 INFO:teuthology.orchestra.run.smithi105.stderr:2024-05-27T07:19:56.683+0000 7f6f70df8700 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f6f6c1100b0 con 0x7f6f6c1059a0 2024-05-27T07:19:56.682 INFO:teuthology.orchestra.run.smithi105.stderr:2024-05-27T07:19:56.683+0000 7f6f69d9b700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f6f6c1049d0 0x7f6f6c19af20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:19:56.682 INFO:teuthology.orchestra.run.smithi105.stderr:2024-05-27T07:19:56.683+0000 7f6f69d9b700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f6f6c1049d0 0x7f6f6c19af20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.105:32966/0 (socket says 172.21.15.105:32966) 2024-05-27T07:19:56.682 INFO:teuthology.orchestra.run.smithi105.stderr:2024-05-27T07:19:56.683+0000 7f6f69d9b700 1 -- 172.21.15.105:0/904741557 learned_addr learned my addr 172.21.15.105:0/904741557 (peer_addr_for_me v2:172.21.15.105:0/0) 2024-05-27T07:19:56.683 INFO:teuthology.orchestra.run.smithi105.stderr:2024-05-27T07:19:56.683+0000 7f6f6a59c700 1 --2- 172.21.15.105:0/904741557 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f6f6c103f50 0x7f6f6c19a9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:19:56.683 INFO:teuthology.orchestra.run.smithi105.stderr:2024-05-27T07:19:56.684+0000 7f6f6ad9d700 1 --2- 172.21.15.105:0/904741557 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f6f6c1059a0 0x7f6f6c1a1ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:19:56.683 INFO:teuthology.orchestra.run.smithi105.stderr:2024-05-27T07:19:56.684+0000 7f6f69d9b700 1 -- 172.21.15.105:0/904741557 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f6f6c1059a0 msgr2=0x7f6f6c1a1ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:19:56.683 INFO:teuthology.orchestra.run.smithi105.stderr:2024-05-27T07:19:56.684+0000 7f6f69d9b700 1 --2- 172.21.15.105:0/904741557 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f6f6c1059a0 0x7f6f6c1a1ec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:19:56.683 INFO:teuthology.orchestra.run.smithi105.stderr:2024-05-27T07:19:56.684+0000 7f6f69d9b700 1 -- 172.21.15.105:0/904741557 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f6f6c103f50 msgr2=0x7f6f6c19a9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:19:56.684 INFO:teuthology.orchestra.run.smithi105.stderr:2024-05-27T07:19:56.684+0000 7f6f69d9b700 1 --2- 172.21.15.105:0/904741557 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f6f6c103f50 0x7f6f6c19a9e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:19:56.684 INFO:teuthology.orchestra.run.smithi105.stderr:2024-05-27T07:19:56.684+0000 7f6f69d9b700 1 -- 172.21.15.105:0/904741557 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6f6c1a2540 con 0x7f6f6c1049d0 2024-05-27T07:19:56.684 INFO:teuthology.orchestra.run.smithi105.stderr:2024-05-27T07:19:56.684+0000 7f6f6a59c700 1 --2- 172.21.15.105:0/904741557 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f6f6c103f50 0x7f6f6c19a9e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-27T07:19:56.684 INFO:teuthology.orchestra.run.smithi105.stderr:2024-05-27T07:19:56.684+0000 7f6f6ad9d700 1 --2- 172.21.15.105:0/904741557 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f6f6c1059a0 0x7f6f6c1a1ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-27T07:19:56.684 INFO:teuthology.orchestra.run.smithi105.stderr:2024-05-27T07:19:56.684+0000 7f6f69d9b700 1 --2- 172.21.15.105:0/904741557 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f6f6c1049d0 0x7f6f6c19af20 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f6f6000d980 tx=0x7f6f6000dc90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:19:56.684 INFO:teuthology.orchestra.run.smithi105.stderr:2024-05-27T07:19:56.684+0000 7f6f5b7fe700 1 -- 172.21.15.105:0/904741557 <== mon.2 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6f60016070 con 0x7f6f6c1049d0 2024-05-27T07:19:56.685 INFO:teuthology.orchestra.run.smithi105.stderr:2024-05-27T07:19:56.684+0000 7f6f5b7fe700 1 -- 172.21.15.105:0/904741557 <== mon.2 v2:172.21.15.105:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f6f60004830 con 0x7f6f6c1049d0 2024-05-27T07:19:56.685 INFO:teuthology.orchestra.run.smithi105.stderr:2024-05-27T07:19:56.684+0000 7f6f70df8700 1 -- 172.21.15.105:0/904741557 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6f6c1a2830 con 0x7f6f6c1049d0 2024-05-27T07:19:56.685 INFO:teuthology.orchestra.run.smithi105.stderr:2024-05-27T07:19:56.684+0000 7f6f5b7fe700 1 -- 172.21.15.105:0/904741557 <== mon.2 v2:172.21.15.105:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6f600054c0 con 0x7f6f6c1049d0 2024-05-27T07:19:56.685 INFO:teuthology.orchestra.run.smithi105.stderr:2024-05-27T07:19:56.684+0000 7f6f70df8700 1 -- 172.21.15.105:0/904741557 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6f6c1a2d70 con 0x7f6f6c1049d0 2024-05-27T07:19:56.685 INFO:teuthology.orchestra.run.smithi105.stderr:2024-05-27T07:19:56.685+0000 7f6f70df8700 1 -- 172.21.15.105:0/904741557 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6f6c0512b0 con 0x7f6f6c1049d0 2024-05-27T07:19:56.685 INFO:teuthology.orchestra.run.smithi105.stderr:2024-05-27T07:19:56.687+0000 7f6f5b7fe700 1 -- 172.21.15.105:0/904741557 <== mon.2 v2:172.21.15.105:3300/0 4 ==== mgrmap(e 16) ==== 149652+0+0 (secure 0 0 0) 0x7f6f60004220 con 0x7f6f6c1049d0 2024-05-27T07:19:56.686 INFO:teuthology.orchestra.run.smithi105.stderr:2024-05-27T07:19:56.687+0000 7f6f5b7fe700 1 --2- 172.21.15.105:0/904741557 >> [v2:172.21.15.52:6800/3513075805,v1:172.21.15.52:6801/3513075805] conn(0x7f6f540b2880 0x7f6f540b4d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:19:56.686 INFO:teuthology.orchestra.run.smithi105.stderr:2024-05-27T07:19:56.687+0000 7f6f5b7fe700 1 -- 172.21.15.105:0/904741557 <== mon.2 v2:172.21.15.105:3300/0 5 ==== osd_map(61..61 src has 1..61) ==== 6261+0+0 (secure 0 0 0) 0x7f6f600e18e0 con 0x7f6f6c1049d0 2024-05-27T07:19:56.686 INFO:teuthology.orchestra.run.smithi105.stderr:2024-05-27T07:19:56.687+0000 7f6f6a59c700 1 --2- 172.21.15.105:0/904741557 >> [v2:172.21.15.52:6800/3513075805,v1:172.21.15.52:6801/3513075805] conn(0x7f6f540b2880 0x7f6f540b4d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:19:56.686 INFO:teuthology.orchestra.run.smithi105.stderr:2024-05-27T07:19:56.688+0000 7f6f6a59c700 1 --2- 172.21.15.105:0/904741557 >> [v2:172.21.15.52:6800/3513075805,v1:172.21.15.52:6801/3513075805] conn(0x7f6f540b2880 0x7f6f540b4d90 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7f6f6c105390 tx=0x7f6f6400b410 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:19:56.687 INFO:teuthology.orchestra.run.smithi105.stderr:2024-05-27T07:19:56.692+0000 7f6f5b7fe700 1 -- 172.21.15.105:0/904741557 <== mon.2 v2:172.21.15.105:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191963 (secure 0 0 0) 0x7f6f6001f020 con 0x7f6f6c1049d0 2024-05-27T07:19:56.894 INFO:teuthology.orchestra.run.smithi105.stderr:2024-05-27T07:19:56.904+0000 7f6f70df8700 1 -- 172.21.15.105:0/904741557 --> [v2:172.21.15.52:6800/3513075805,v1:172.21.15.52:6801/3513075805] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi105:vg_nvme/lv_1", "target": ["mon-mgr", ""]}) -- 0x7f6f6c02ea50 con 0x7f6f540b2880 2024-05-27T07:19:57.352 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:19:57 smithi173 ceph-mon[37000]: pgmap v213: 1 pgs: 1 active+clean; 577 KiB data, 294 MiB used, 894 GiB / 894 GiB avail 2024-05-27T07:19:57.352 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:19:57 smithi173 ceph-mon[37000]: osdmap e61: 10 total, 10 up, 10 in 2024-05-27T07:19:57.352 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:19:57 smithi173 ceph-mon[37000]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-05-27T07:19:57.352 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:19:57 smithi173 ceph-mon[37000]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-05-27T07:19:57.352 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:19:57 smithi173 ceph-mon[37000]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T07:19:57.409 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:19:57 smithi052 ceph-mon[31955]: pgmap v213: 1 pgs: 1 active+clean; 577 KiB data, 294 MiB used, 894 GiB / 894 GiB avail 2024-05-27T07:19:57.409 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:19:57 smithi052 ceph-mon[31955]: osdmap e61: 10 total, 10 up, 10 in 2024-05-27T07:19:57.409 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:19:57 smithi052 ceph-mon[31955]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-05-27T07:19:57.409 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:19:57 smithi052 ceph-mon[31955]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-05-27T07:19:57.409 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:19:57 smithi052 ceph-mon[31955]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T07:19:57.440 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:19:57 smithi105 ceph-mon[35977]: pgmap v213: 1 pgs: 1 active+clean; 577 KiB data, 294 MiB used, 894 GiB / 894 GiB avail 2024-05-27T07:19:57.440 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:19:57 smithi105 ceph-mon[35977]: osdmap e61: 10 total, 10 up, 10 in 2024-05-27T07:19:57.440 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:19:57 smithi105 ceph-mon[35977]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-05-27T07:19:57.440 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:19:57 smithi105 ceph-mon[35977]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-05-27T07:19:57.440 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:19:57 smithi105 ceph-mon[35977]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T07:19:58.352 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:19:58 smithi173 ceph-mon[37000]: from='client.24365 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi105:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-05-27T07:19:58.440 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:19:58 smithi105 ceph-mon[35977]: from='client.24365 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi105:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-05-27T07:19:58.479 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:19:58 smithi052 ceph-mon[31955]: from='client.24365 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi105:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-05-27T07:19:59.190 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:19:59 smithi105 ceph-mon[35977]: pgmap v215: 1 pgs: 1 active+clean; 577 KiB data, 294 MiB used, 894 GiB / 894 GiB avail 2024-05-27T07:19:59.352 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:19:59 smithi173 ceph-mon[37000]: pgmap v215: 1 pgs: 1 active+clean; 577 KiB data, 294 MiB used, 894 GiB / 894 GiB avail 2024-05-27T07:19:59.480 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:19:59 smithi052 ceph-mon[31955]: pgmap v215: 1 pgs: 1 active+clean; 577 KiB data, 294 MiB used, 894 GiB / 894 GiB avail 2024-05-27T07:20:00.352 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:20:00 smithi173 ceph-mon[37000]: from='client.? 172.21.15.105:0/1830231694' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "e6f8e753-e15c-45b5-8ee9-d50f384559f9"}]: dispatch 2024-05-27T07:20:00.352 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:20:00 smithi173 ceph-mon[37000]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "e6f8e753-e15c-45b5-8ee9-d50f384559f9"}]: dispatch 2024-05-27T07:20:00.352 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:20:00 smithi173 ceph-mon[37000]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "e6f8e753-e15c-45b5-8ee9-d50f384559f9"}]': finished 2024-05-27T07:20:00.352 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:20:00 smithi173 ceph-mon[37000]: osdmap e62: 11 total, 10 up, 11 in 2024-05-27T07:20:00.353 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:20:00 smithi173 ceph-mon[37000]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-05-27T07:20:00.353 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:20:00 smithi173 ceph-mon[37000]: overall HEALTH_OK 2024-05-27T07:20:00.440 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:20:00 smithi105 ceph-mon[35977]: from='client.? 172.21.15.105:0/1830231694' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "e6f8e753-e15c-45b5-8ee9-d50f384559f9"}]: dispatch 2024-05-27T07:20:00.440 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:20:00 smithi105 ceph-mon[35977]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "e6f8e753-e15c-45b5-8ee9-d50f384559f9"}]: dispatch 2024-05-27T07:20:00.440 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:20:00 smithi105 ceph-mon[35977]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "e6f8e753-e15c-45b5-8ee9-d50f384559f9"}]': finished 2024-05-27T07:20:00.440 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:20:00 smithi105 ceph-mon[35977]: osdmap e62: 11 total, 10 up, 11 in 2024-05-27T07:20:00.440 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:20:00 smithi105 ceph-mon[35977]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-05-27T07:20:00.441 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:20:00 smithi105 ceph-mon[35977]: overall HEALTH_OK 2024-05-27T07:20:00.479 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:20:00 smithi052 ceph-mon[31955]: from='client.? 172.21.15.105:0/1830231694' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "e6f8e753-e15c-45b5-8ee9-d50f384559f9"}]: dispatch 2024-05-27T07:20:00.480 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:20:00 smithi052 ceph-mon[31955]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "e6f8e753-e15c-45b5-8ee9-d50f384559f9"}]: dispatch 2024-05-27T07:20:00.480 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:20:00 smithi052 ceph-mon[31955]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "e6f8e753-e15c-45b5-8ee9-d50f384559f9"}]': finished 2024-05-27T07:20:00.480 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:20:00 smithi052 ceph-mon[31955]: osdmap e62: 11 total, 10 up, 11 in 2024-05-27T07:20:00.480 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:20:00 smithi052 ceph-mon[31955]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-05-27T07:20:00.480 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:20:00 smithi052 ceph-mon[31955]: overall HEALTH_OK 2024-05-27T07:20:01.352 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:20:01 smithi173 ceph-mon[37000]: pgmap v216: 1 pgs: 1 active+clean; 577 KiB data, 294 MiB used, 894 GiB / 894 GiB avail 2024-05-27T07:20:01.352 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:20:01 smithi173 ceph-mon[37000]: from='client.? 172.21.15.105:0/833098660' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-05-27T07:20:01.440 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:20:01 smithi105 ceph-mon[35977]: pgmap v216: 1 pgs: 1 active+clean; 577 KiB data, 294 MiB used, 894 GiB / 894 GiB avail 2024-05-27T07:20:01.440 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:20:01 smithi105 ceph-mon[35977]: from='client.? 172.21.15.105:0/833098660' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-05-27T07:20:01.479 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:20:01 smithi052 ceph-mon[31955]: pgmap v216: 1 pgs: 1 active+clean; 577 KiB data, 294 MiB used, 894 GiB / 894 GiB avail 2024-05-27T07:20:01.480 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:20:01 smithi052 ceph-mon[31955]: from='client.? 172.21.15.105:0/833098660' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-05-27T07:20:03.352 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:20:03 smithi173 ceph-mon[37000]: pgmap v218: 1 pgs: 1 active+clean; 577 KiB data, 294 MiB used, 894 GiB / 894 GiB avail 2024-05-27T07:20:03.423 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:20:03 smithi105 ceph-mon[35977]: pgmap v218: 1 pgs: 1 active+clean; 577 KiB data, 294 MiB used, 894 GiB / 894 GiB avail 2024-05-27T07:20:03.479 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:20:03 smithi052 ceph-mon[31955]: pgmap v218: 1 pgs: 1 active+clean; 577 KiB data, 294 MiB used, 894 GiB / 894 GiB avail 2024-05-27T07:20:05.286 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:20:05 smithi105 ceph-mon[35977]: pgmap v219: 1 pgs: 1 active+clean; 577 KiB data, 294 MiB used, 894 GiB / 894 GiB avail 2024-05-27T07:20:05.352 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:20:05 smithi173 ceph-mon[37000]: pgmap v219: 1 pgs: 1 active+clean; 577 KiB data, 294 MiB used, 894 GiB / 894 GiB avail 2024-05-27T07:20:05.479 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:20:05 smithi052 ceph-mon[31955]: pgmap v219: 1 pgs: 1 active+clean; 577 KiB data, 294 MiB used, 894 GiB / 894 GiB avail 2024-05-27T07:20:07.230 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:20:07 smithi052 ceph-mon[31955]: pgmap v220: 1 pgs: 1 active+clean; 577 KiB data, 294 MiB used, 894 GiB / 894 GiB avail 2024-05-27T07:20:07.230 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:20:07 smithi052 ceph-mon[31955]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.10"}]: dispatch 2024-05-27T07:20:07.230 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:20:07 smithi052 ceph-mon[31955]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T07:20:07.352 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:20:07 smithi173 ceph-mon[37000]: pgmap v220: 1 pgs: 1 active+clean; 577 KiB data, 294 MiB used, 894 GiB / 894 GiB avail 2024-05-27T07:20:07.352 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:20:07 smithi173 ceph-mon[37000]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.10"}]: dispatch 2024-05-27T07:20:07.352 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:20:07 smithi173 ceph-mon[37000]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T07:20:07.440 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:20:07 smithi105 ceph-mon[35977]: pgmap v220: 1 pgs: 1 active+clean; 577 KiB data, 294 MiB used, 894 GiB / 894 GiB avail 2024-05-27T07:20:07.440 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:20:07 smithi105 ceph-mon[35977]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.10"}]: dispatch 2024-05-27T07:20:07.440 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:20:07 smithi105 ceph-mon[35977]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T07:20:08.190 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:20:08 smithi105 ceph-mon[35977]: Deploying daemon osd.10 on smithi105 2024-05-27T07:20:08.352 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:20:08 smithi173 ceph-mon[37000]: Deploying daemon osd.10 on smithi105 2024-05-27T07:20:08.479 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:20:08 smithi052 ceph-mon[31955]: Deploying daemon osd.10 on smithi105 2024-05-27T07:20:09.190 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:20:09 smithi105 ceph-mon[35977]: pgmap v221: 1 pgs: 1 active+clean; 577 KiB data, 294 MiB used, 894 GiB / 894 GiB avail 2024-05-27T07:20:09.352 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:20:09 smithi173 ceph-mon[37000]: pgmap v221: 1 pgs: 1 active+clean; 577 KiB data, 294 MiB used, 894 GiB / 894 GiB avail 2024-05-27T07:20:09.479 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:20:09 smithi052 ceph-mon[31955]: pgmap v221: 1 pgs: 1 active+clean; 577 KiB data, 294 MiB used, 894 GiB / 894 GiB avail 2024-05-27T07:20:11.352 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:20:11 smithi173 ceph-mon[37000]: pgmap v222: 1 pgs: 1 active+clean; 577 KiB data, 294 MiB used, 894 GiB / 894 GiB avail 2024-05-27T07:20:11.365 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:20:11 smithi105 ceph-mon[35977]: pgmap v222: 1 pgs: 1 active+clean; 577 KiB data, 294 MiB used, 894 GiB / 894 GiB avail 2024-05-27T07:20:11.479 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:20:11 smithi052 ceph-mon[31955]: pgmap v222: 1 pgs: 1 active+clean; 577 KiB data, 294 MiB used, 894 GiB / 894 GiB avail 2024-05-27T07:20:13.282 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:20:13 smithi105 ceph-mon[35977]: pgmap v223: 1 pgs: 1 active+clean; 577 KiB data, 294 MiB used, 894 GiB / 894 GiB avail 2024-05-27T07:20:13.283 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:20:13 smithi105 ceph-mon[35977]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' 2024-05-27T07:20:13.283 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:20:13 smithi105 ceph-mon[35977]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' 2024-05-27T07:20:13.283 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:20:13 smithi105 ceph-mon[35977]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-27T07:20:13.352 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:20:13 smithi173 ceph-mon[37000]: pgmap v223: 1 pgs: 1 active+clean; 577 KiB data, 294 MiB used, 894 GiB / 894 GiB avail 2024-05-27T07:20:13.352 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:20:13 smithi173 ceph-mon[37000]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' 2024-05-27T07:20:13.352 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:20:13 smithi173 ceph-mon[37000]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' 2024-05-27T07:20:13.352 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:20:13 smithi173 ceph-mon[37000]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-27T07:20:13.479 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:20:13 smithi052 ceph-mon[31955]: pgmap v223: 1 pgs: 1 active+clean; 577 KiB data, 294 MiB used, 894 GiB / 894 GiB avail 2024-05-27T07:20:13.480 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:20:13 smithi052 ceph-mon[31955]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' 2024-05-27T07:20:13.480 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:20:13 smithi052 ceph-mon[31955]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' 2024-05-27T07:20:13.480 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:20:13 smithi052 ceph-mon[31955]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-27T07:20:15.190 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:20:15 smithi105 ceph-mon[35977]: pgmap v224: 1 pgs: 1 active+clean; 577 KiB data, 294 MiB used, 894 GiB / 894 GiB avail 2024-05-27T07:20:15.352 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:20:15 smithi173 ceph-mon[37000]: pgmap v224: 1 pgs: 1 active+clean; 577 KiB data, 294 MiB used, 894 GiB / 894 GiB avail 2024-05-27T07:20:15.482 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:20:15 smithi052 ceph-mon[31955]: pgmap v224: 1 pgs: 1 active+clean; 577 KiB data, 294 MiB used, 894 GiB / 894 GiB avail 2024-05-27T07:20:16.614 INFO:teuthology.orchestra.run.smithi105.stdout:Created osd(s) 10 on host 'smithi105' 2024-05-27T07:20:16.614 INFO:teuthology.orchestra.run.smithi105.stderr:2024-05-27T07:20:16.623+0000 7f6f5b7fe700 1 -- 172.21.15.105:0/904741557 <== mgr.14152 v2:172.21.15.52:6800/3513075805 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+38 (secure 0 0 0) 0x7f6f6c02ea50 con 0x7f6f540b2880 2024-05-27T07:20:16.616 INFO:teuthology.orchestra.run.smithi105.stderr:2024-05-27T07:20:16.626+0000 7f6f70df8700 1 -- 172.21.15.105:0/904741557 >> [v2:172.21.15.52:6800/3513075805,v1:172.21.15.52:6801/3513075805] conn(0x7f6f540b2880 msgr2=0x7f6f540b4d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:20:16.616 INFO:teuthology.orchestra.run.smithi105.stderr:2024-05-27T07:20:16.626+0000 7f6f70df8700 1 --2- 172.21.15.105:0/904741557 >> [v2:172.21.15.52:6800/3513075805,v1:172.21.15.52:6801/3513075805] conn(0x7f6f540b2880 0x7f6f540b4d90 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7f6f6c105390 tx=0x7f6f6400b410 comp rx=0 tx=0).stop 2024-05-27T07:20:16.616 INFO:teuthology.orchestra.run.smithi105.stderr:2024-05-27T07:20:16.626+0000 7f6f70df8700 1 -- 172.21.15.105:0/904741557 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f6f6c1049d0 msgr2=0x7f6f6c19af20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:20:16.616 INFO:teuthology.orchestra.run.smithi105.stderr:2024-05-27T07:20:16.626+0000 7f6f70df8700 1 --2- 172.21.15.105:0/904741557 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f6f6c1049d0 0x7f6f6c19af20 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f6f6000d980 tx=0x7f6f6000dc90 comp rx=0 tx=0).stop 2024-05-27T07:20:16.616 INFO:teuthology.orchestra.run.smithi105.stderr:2024-05-27T07:20:16.626+0000 7f6f70df8700 1 -- 172.21.15.105:0/904741557 shutdown_connections 2024-05-27T07:20:16.617 INFO:teuthology.orchestra.run.smithi105.stderr:2024-05-27T07:20:16.626+0000 7f6f70df8700 1 --2- 172.21.15.105:0/904741557 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f6f6c1059a0 0x7f6f6c1a1ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:20:16.617 INFO:teuthology.orchestra.run.smithi105.stderr:2024-05-27T07:20:16.626+0000 7f6f70df8700 1 --2- 172.21.15.105:0/904741557 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f6f6c103f50 0x7f6f6c19a9e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:20:16.617 INFO:teuthology.orchestra.run.smithi105.stderr:2024-05-27T07:20:16.626+0000 7f6f70df8700 1 --2- 172.21.15.105:0/904741557 >> [v2:172.21.15.52:6800/3513075805,v1:172.21.15.52:6801/3513075805] conn(0x7f6f540b2880 0x7f6f540b4d90 unknown :-1 s=CLOSED pgs=103 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:20:16.617 INFO:teuthology.orchestra.run.smithi105.stderr:2024-05-27T07:20:16.626+0000 7f6f70df8700 1 --2- 172.21.15.105:0/904741557 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f6f6c1049d0 0x7f6f6c19af20 unknown :-1 s=CLOSED pgs=36 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:20:16.617 INFO:teuthology.orchestra.run.smithi105.stderr:2024-05-27T07:20:16.626+0000 7f6f70df8700 1 -- 172.21.15.105:0/904741557 >> 172.21.15.105:0/904741557 conn(0x7f6f6c0ff9c0 msgr2=0x7f6f6c1086c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:20:16.617 INFO:teuthology.orchestra.run.smithi105.stderr:2024-05-27T07:20:16.626+0000 7f6f70df8700 1 -- 172.21.15.105:0/904741557 shutdown_connections 2024-05-27T07:20:16.617 INFO:teuthology.orchestra.run.smithi105.stderr:2024-05-27T07:20:16.626+0000 7f6f70df8700 1 -- 172.21.15.105:0/904741557 wait complete. 2024-05-27T07:20:16.852 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:20:16 smithi173 ceph-mon[37000]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' 2024-05-27T07:20:16.852 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:20:16 smithi173 ceph-mon[37000]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' 2024-05-27T07:20:16.852 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:20:16 smithi173 ceph-mon[37000]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T07:20:16.852 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:20:16 smithi173 ceph-mon[37000]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-27T07:20:16.852 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:20:16 smithi173 ceph-mon[37000]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' 2024-05-27T07:20:16.852 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:20:16 smithi173 ceph-mon[37000]: pgmap v225: 1 pgs: 1 active+clean; 577 KiB data, 294 MiB used, 894 GiB / 894 GiB avail 2024-05-27T07:20:16.853 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:20:16 smithi173 ceph-mon[37000]: from='osd.10 [v2:172.21.15.105:6824/4032386628,v1:172.21.15.105:6825/4032386628]' entity='osd.10' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]: dispatch 2024-05-27T07:20:16.853 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:20:16 smithi173 ceph-mon[37000]: from='osd.10 ' entity='osd.10' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]: dispatch 2024-05-27T07:20:16.940 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:20:16 smithi105 ceph-mon[35977]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' 2024-05-27T07:20:16.940 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:20:16 smithi105 ceph-mon[35977]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' 2024-05-27T07:20:16.940 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:20:16 smithi105 ceph-mon[35977]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T07:20:16.940 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:20:16 smithi105 ceph-mon[35977]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-27T07:20:16.940 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:20:16 smithi105 ceph-mon[35977]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' 2024-05-27T07:20:16.940 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:20:16 smithi105 ceph-mon[35977]: pgmap v225: 1 pgs: 1 active+clean; 577 KiB data, 294 MiB used, 894 GiB / 894 GiB avail 2024-05-27T07:20:16.940 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:20:16 smithi105 ceph-mon[35977]: from='osd.10 [v2:172.21.15.105:6824/4032386628,v1:172.21.15.105:6825/4032386628]' entity='osd.10' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]: dispatch 2024-05-27T07:20:16.940 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:20:16 smithi105 ceph-mon[35977]: from='osd.10 ' entity='osd.10' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]: dispatch 2024-05-27T07:20:16.979 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:20:16 smithi052 ceph-mon[31955]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' 2024-05-27T07:20:16.980 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:20:16 smithi052 ceph-mon[31955]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' 2024-05-27T07:20:16.980 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:20:16 smithi052 ceph-mon[31955]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T07:20:16.980 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:20:16 smithi052 ceph-mon[31955]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-27T07:20:16.980 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:20:16 smithi052 ceph-mon[31955]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' 2024-05-27T07:20:16.980 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:20:16 smithi052 ceph-mon[31955]: pgmap v225: 1 pgs: 1 active+clean; 577 KiB data, 294 MiB used, 894 GiB / 894 GiB avail 2024-05-27T07:20:16.980 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:20:16 smithi052 ceph-mon[31955]: from='osd.10 [v2:172.21.15.105:6824/4032386628,v1:172.21.15.105:6825/4032386628]' entity='osd.10' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]: dispatch 2024-05-27T07:20:16.980 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:20:16 smithi052 ceph-mon[31955]: from='osd.10 ' entity='osd.10' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]: dispatch 2024-05-27T07:20:17.271 DEBUG:teuthology.orchestra.run.smithi105:osd.10> sudo journalctl -f -n 0 -u ceph-73df2636-1bf8-11ef-bc9b-c7b262605968@osd.10.service 2024-05-27T07:20:17.273 INFO:tasks.cephadm:Deploying osd.11 on smithi173 with /dev/vg_nvme/lv_1... 2024-05-27T07:20:17.273 DEBUG:teuthology.orchestra.run.smithi173:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:38f9d900c83c63799fbdbe61acc9a11b0d3554a6 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 73df2636-1bf8-11ef-bc9b-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_1 2024-05-27T07:20:17.522 INFO:teuthology.orchestra.run.smithi173.stderr:Inferring config /var/lib/ceph/73df2636-1bf8-11ef-bc9b-c7b262605968/mon.c/config 2024-05-27T07:20:17.852 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:20:17 smithi173 ceph-mon[37000]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' 2024-05-27T07:20:17.852 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:20:17 smithi173 ceph-mon[37000]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' 2024-05-27T07:20:17.852 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:20:17 smithi173 ceph-mon[37000]: from='osd.10 ' entity='osd.10' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]': finished 2024-05-27T07:20:17.852 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:20:17 smithi173 ceph-mon[37000]: osdmap e63: 11 total, 10 up, 11 in 2024-05-27T07:20:17.853 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:20:17 smithi173 ceph-mon[37000]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-05-27T07:20:17.853 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:20:17 smithi173 ceph-mon[37000]: from='osd.10 [v2:172.21.15.105:6824/4032386628,v1:172.21.15.105:6825/4032386628]' entity='osd.10' cmd=[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi105", "root=default"]}]: dispatch 2024-05-27T07:20:17.853 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:20:17 smithi173 ceph-mon[37000]: from='osd.10 ' entity='osd.10' cmd=[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi105", "root=default"]}]: dispatch 2024-05-27T07:20:17.853 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:20:17 smithi173 ceph-mon[37000]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-27T07:20:17.940 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:20:17 smithi105 ceph-mon[35977]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' 2024-05-27T07:20:17.940 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:20:17 smithi105 ceph-mon[35977]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' 2024-05-27T07:20:17.940 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:20:17 smithi105 ceph-mon[35977]: from='osd.10 ' entity='osd.10' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]': finished 2024-05-27T07:20:17.940 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:20:17 smithi105 ceph-mon[35977]: osdmap e63: 11 total, 10 up, 11 in 2024-05-27T07:20:17.941 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:20:17 smithi105 ceph-mon[35977]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-05-27T07:20:17.941 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:20:17 smithi105 ceph-mon[35977]: from='osd.10 [v2:172.21.15.105:6824/4032386628,v1:172.21.15.105:6825/4032386628]' entity='osd.10' cmd=[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi105", "root=default"]}]: dispatch 2024-05-27T07:20:17.941 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:20:17 smithi105 ceph-mon[35977]: from='osd.10 ' entity='osd.10' cmd=[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi105", "root=default"]}]: dispatch 2024-05-27T07:20:17.941 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:20:17 smithi105 ceph-mon[35977]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-27T07:20:17.941 INFO:journalctl@ceph.osd.10.smithi105.stdout:May 27 07:20:17 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-osd-10[60334]: 2024-05-27T07:20:17.608+0000 7f16325d7700 -1 osd.10 0 waiting for initial osdmap 2024-05-27T07:20:17.941 INFO:journalctl@ceph.osd.10.smithi105.stdout:May 27 07:20:17 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-osd-10[60334]: 2024-05-27T07:20:17.619+0000 7f162e38c700 -1 osd.10 64 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-05-27T07:20:17.979 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:20:17 smithi052 ceph-mon[31955]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' 2024-05-27T07:20:17.979 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:20:17 smithi052 ceph-mon[31955]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' 2024-05-27T07:20:17.980 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:20:17 smithi052 ceph-mon[31955]: from='osd.10 ' entity='osd.10' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]': finished 2024-05-27T07:20:17.980 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:20:17 smithi052 ceph-mon[31955]: osdmap e63: 11 total, 10 up, 11 in 2024-05-27T07:20:17.980 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:20:17 smithi052 ceph-mon[31955]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-05-27T07:20:17.980 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:20:17 smithi052 ceph-mon[31955]: from='osd.10 [v2:172.21.15.105:6824/4032386628,v1:172.21.15.105:6825/4032386628]' entity='osd.10' cmd=[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi105", "root=default"]}]: dispatch 2024-05-27T07:20:17.980 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:20:17 smithi052 ceph-mon[31955]: from='osd.10 ' entity='osd.10' cmd=[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi105", "root=default"]}]: dispatch 2024-05-27T07:20:17.980 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:20:17 smithi052 ceph-mon[31955]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-27T07:20:18.690 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:20:18 smithi105 ceph-mon[35977]: from='osd.10 ' entity='osd.10' cmd='[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi105", "root=default"]}]': finished 2024-05-27T07:20:18.690 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:20:18 smithi105 ceph-mon[35977]: osdmap e64: 11 total, 10 up, 11 in 2024-05-27T07:20:18.690 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:20:18 smithi105 ceph-mon[35977]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-05-27T07:20:18.690 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:20:18 smithi105 ceph-mon[35977]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-05-27T07:20:18.690 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:20:18 smithi105 ceph-mon[35977]: pgmap v228: 1 pgs: 1 active+clean; 577 KiB data, 294 MiB used, 894 GiB / 894 GiB avail 2024-05-27T07:20:18.852 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:20:18 smithi173 ceph-mon[37000]: from='osd.10 ' entity='osd.10' cmd='[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi105", "root=default"]}]': finished 2024-05-27T07:20:18.852 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:20:18 smithi173 ceph-mon[37000]: osdmap e64: 11 total, 10 up, 11 in 2024-05-27T07:20:18.852 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:20:18 smithi173 ceph-mon[37000]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-05-27T07:20:18.852 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:20:18 smithi173 ceph-mon[37000]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-05-27T07:20:18.852 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:20:18 smithi173 ceph-mon[37000]: pgmap v228: 1 pgs: 1 active+clean; 577 KiB data, 294 MiB used, 894 GiB / 894 GiB avail 2024-05-27T07:20:18.980 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:20:18 smithi052 ceph-mon[31955]: from='osd.10 ' entity='osd.10' cmd='[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi105", "root=default"]}]': finished 2024-05-27T07:20:18.980 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:20:18 smithi052 ceph-mon[31955]: osdmap e64: 11 total, 10 up, 11 in 2024-05-27T07:20:18.980 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:20:18 smithi052 ceph-mon[31955]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-05-27T07:20:18.980 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:20:18 smithi052 ceph-mon[31955]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-05-27T07:20:18.980 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:20:18 smithi052 ceph-mon[31955]: pgmap v228: 1 pgs: 1 active+clean; 577 KiB data, 294 MiB used, 894 GiB / 894 GiB avail 2024-05-27T07:20:19.575 INFO:teuthology.orchestra.run.smithi173.stderr:--> Zapping: /dev/vg_nvme/lv_1 2024-05-27T07:20:19.575 INFO:teuthology.orchestra.run.smithi173.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_1 bs=1M count=10 conv=fsync 2024-05-27T07:20:19.575 INFO:teuthology.orchestra.run.smithi173.stderr: stderr: 10+0 records in 2024-05-27T07:20:19.575 INFO:teuthology.orchestra.run.smithi173.stderr:10+0 records out 2024-05-27T07:20:19.575 INFO:teuthology.orchestra.run.smithi173.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.011308 s, 927 MB/s 2024-05-27T07:20:19.575 INFO:teuthology.orchestra.run.smithi173.stderr:--> Zapping successful for: 2024-05-27T07:20:19.852 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:20:19 smithi173 ceph-mon[37000]: purged_snaps scrub starts 2024-05-27T07:20:19.852 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:20:19 smithi173 ceph-mon[37000]: purged_snaps scrub ok 2024-05-27T07:20:19.852 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:20:19 smithi173 ceph-mon[37000]: osd.10 [v2:172.21.15.105:6824/4032386628,v1:172.21.15.105:6825/4032386628] boot 2024-05-27T07:20:19.852 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:20:19 smithi173 ceph-mon[37000]: osdmap e65: 11 total, 11 up, 11 in 2024-05-27T07:20:19.852 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:20:19 smithi173 ceph-mon[37000]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-05-27T07:20:19.881 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:20:19 smithi105 ceph-mon[35977]: purged_snaps scrub starts 2024-05-27T07:20:19.881 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:20:19 smithi105 ceph-mon[35977]: purged_snaps scrub ok 2024-05-27T07:20:19.881 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:20:19 smithi105 ceph-mon[35977]: osd.10 [v2:172.21.15.105:6824/4032386628,v1:172.21.15.105:6825/4032386628] boot 2024-05-27T07:20:19.881 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:20:19 smithi105 ceph-mon[35977]: osdmap e65: 11 total, 11 up, 11 in 2024-05-27T07:20:19.881 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:20:19 smithi105 ceph-mon[35977]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-05-27T07:20:19.979 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:20:19 smithi052 ceph-mon[31955]: purged_snaps scrub starts 2024-05-27T07:20:19.980 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:20:19 smithi052 ceph-mon[31955]: purged_snaps scrub ok 2024-05-27T07:20:19.980 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:20:19 smithi052 ceph-mon[31955]: osd.10 [v2:172.21.15.105:6824/4032386628,v1:172.21.15.105:6825/4032386628] boot 2024-05-27T07:20:19.980 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:20:19 smithi052 ceph-mon[31955]: osdmap e65: 11 total, 11 up, 11 in 2024-05-27T07:20:19.980 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:20:19 smithi052 ceph-mon[31955]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-05-27T07:20:20.198 DEBUG:teuthology.orchestra.run.smithi173:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:38f9d900c83c63799fbdbe61acc9a11b0d3554a6 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 73df2636-1bf8-11ef-bc9b-c7b262605968 -- ceph orch daemon add osd smithi173:vg_nvme/lv_1 2024-05-27T07:20:20.433 INFO:teuthology.orchestra.run.smithi173.stderr:Inferring config /var/lib/ceph/73df2636-1bf8-11ef-bc9b-c7b262605968/mon.c/config 2024-05-27T07:20:20.788 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:20:20 smithi173 ceph-mon[37000]: osdmap e66: 11 total, 11 up, 11 in 2024-05-27T07:20:20.788 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:20:20 smithi173 ceph-mon[37000]: pgmap v231: 1 pgs: 1 active+clean; 577 KiB data, 322 MiB used, 983 GiB / 983 GiB avail 2024-05-27T07:20:20.788 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:20:20 smithi173 ceph-mon[37000]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' 2024-05-27T07:20:20.788 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:20:20 smithi173 ceph-mon[37000]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' 2024-05-27T07:20:20.789 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:20:20 smithi173 ceph-mon[37000]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-05-27T07:20:20.789 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:20:20 smithi173 ceph-mon[37000]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-05-27T07:20:20.789 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:20:20 smithi173 ceph-mon[37000]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-05-27T07:20:20.789 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:20:20 smithi173 ceph-mon[37000]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-05-27T07:20:20.789 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:20:20 smithi173 ceph-mon[37000]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' 2024-05-27T07:20:20.789 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:20:20 smithi173 ceph-mon[37000]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T07:20:20.789 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:20:20 smithi173 ceph-mon[37000]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-27T07:20:20.789 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:20:20 smithi173 ceph-mon[37000]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' 2024-05-27T07:20:20.940 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:20:20 smithi105 ceph-mon[35977]: osdmap e66: 11 total, 11 up, 11 in 2024-05-27T07:20:20.940 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:20:20 smithi105 ceph-mon[35977]: pgmap v231: 1 pgs: 1 active+clean; 577 KiB data, 322 MiB used, 983 GiB / 983 GiB avail 2024-05-27T07:20:20.940 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:20:20 smithi105 ceph-mon[35977]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' 2024-05-27T07:20:20.940 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:20:20 smithi105 ceph-mon[35977]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' 2024-05-27T07:20:20.940 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:20:20 smithi105 ceph-mon[35977]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-05-27T07:20:20.940 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:20:20 smithi105 ceph-mon[35977]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-05-27T07:20:20.941 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:20:20 smithi105 ceph-mon[35977]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-05-27T07:20:20.941 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:20:20 smithi105 ceph-mon[35977]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-05-27T07:20:20.941 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:20:20 smithi105 ceph-mon[35977]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' 2024-05-27T07:20:20.941 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:20:20 smithi105 ceph-mon[35977]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T07:20:20.941 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:20:20 smithi105 ceph-mon[35977]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-27T07:20:20.941 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:20:20 smithi105 ceph-mon[35977]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' 2024-05-27T07:20:20.980 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:20:20 smithi052 ceph-mon[31955]: osdmap e66: 11 total, 11 up, 11 in 2024-05-27T07:20:20.980 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:20:20 smithi052 ceph-mon[31955]: pgmap v231: 1 pgs: 1 active+clean; 577 KiB data, 322 MiB used, 983 GiB / 983 GiB avail 2024-05-27T07:20:20.980 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:20:20 smithi052 ceph-mon[31955]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' 2024-05-27T07:20:20.980 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:20:20 smithi052 ceph-mon[31955]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' 2024-05-27T07:20:20.980 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:20:20 smithi052 ceph-mon[31955]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-05-27T07:20:20.980 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:20:20 smithi052 ceph-mon[31955]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-05-27T07:20:20.980 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:20:20 smithi052 ceph-mon[31955]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-05-27T07:20:20.980 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:20:20 smithi052 ceph-mon[31955]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-05-27T07:20:20.980 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:20:20 smithi052 ceph-mon[31955]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' 2024-05-27T07:20:20.980 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:20:20 smithi052 ceph-mon[31955]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T07:20:20.980 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:20:20 smithi052 ceph-mon[31955]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-27T07:20:20.980 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:20:20 smithi052 ceph-mon[31955]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' 2024-05-27T07:20:21.857 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:20:21 smithi173 ceph-mon[37000]: Detected new or changed devices on smithi105 2024-05-27T07:20:21.857 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:20:21 smithi173 ceph-mon[37000]: Adjusting osd_memory_target on smithi105 to 4272M 2024-05-27T07:20:21.857 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:20:21 smithi173 ceph-mon[37000]: osdmap e67: 11 total, 11 up, 11 in 2024-05-27T07:20:21.940 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:20:21 smithi105 ceph-mon[35977]: Detected new or changed devices on smithi105 2024-05-27T07:20:21.940 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:20:21 smithi105 ceph-mon[35977]: Adjusting osd_memory_target on smithi105 to 4272M 2024-05-27T07:20:21.940 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:20:21 smithi105 ceph-mon[35977]: osdmap e67: 11 total, 11 up, 11 in 2024-05-27T07:20:21.979 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:20:21 smithi052 ceph-mon[31955]: Detected new or changed devices on smithi105 2024-05-27T07:20:21.980 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:20:21 smithi052 ceph-mon[31955]: Adjusting osd_memory_target on smithi105 to 4272M 2024-05-27T07:20:21.980 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:20:21 smithi052 ceph-mon[31955]: osdmap e67: 11 total, 11 up, 11 in 2024-05-27T07:20:22.160 INFO:teuthology.orchestra.run.smithi173.stderr:2024-05-27T07:20:22.167+0000 7ff1f71a3700 1 -- 172.21.15.173:0/194756392 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7ff1f0104d00 msgr2=0x7ff1f010b5b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:20:22.424 INFO:teuthology.orchestra.run.smithi173.stderr:2024-05-27T07:20:22.167+0000 7ff1f71a3700 1 --2- 172.21.15.173:0/194756392 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7ff1f0104d00 0x7ff1f010b5b0 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7ff1e800b0a0 tx=0x7ff1e800b3b0 comp rx=0 tx=0).stop 2024-05-27T07:20:22.424 INFO:teuthology.orchestra.run.smithi173.stderr:2024-05-27T07:20:22.167+0000 7ff1f71a3700 1 -- 172.21.15.173:0/194756392 shutdown_connections 2024-05-27T07:20:22.424 INFO:teuthology.orchestra.run.smithi173.stderr:2024-05-27T07:20:22.167+0000 7ff1f71a3700 1 --2- 172.21.15.173:0/194756392 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7ff1f0104d00 0x7ff1f010b5b0 unknown :-1 s=CLOSED pgs=38 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:20:22.424 INFO:teuthology.orchestra.run.smithi173.stderr:2024-05-27T07:20:22.167+0000 7ff1f71a3700 1 --2- 172.21.15.173:0/194756392 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff1f0103230 0x7ff1f0103640 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:20:22.425 INFO:teuthology.orchestra.run.smithi173.stderr:2024-05-27T07:20:22.167+0000 7ff1f71a3700 1 --2- 172.21.15.173:0/194756392 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7ff1f0104360 0x7ff1f0104770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:20:22.425 INFO:teuthology.orchestra.run.smithi173.stderr:2024-05-27T07:20:22.167+0000 7ff1f71a3700 1 -- 172.21.15.173:0/194756392 >> 172.21.15.173:0/194756392 conn(0x7ff1f00fe870 msgr2=0x7ff1f0100d20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:20:22.425 INFO:teuthology.orchestra.run.smithi173.stderr:2024-05-27T07:20:22.168+0000 7ff1f71a3700 1 -- 172.21.15.173:0/194756392 shutdown_connections 2024-05-27T07:20:22.425 INFO:teuthology.orchestra.run.smithi173.stderr:2024-05-27T07:20:22.168+0000 7ff1f71a3700 1 -- 172.21.15.173:0/194756392 wait complete. 2024-05-27T07:20:22.425 INFO:teuthology.orchestra.run.smithi173.stderr:2024-05-27T07:20:22.168+0000 7ff1f71a3700 1 Processor -- start 2024-05-27T07:20:22.425 INFO:teuthology.orchestra.run.smithi173.stderr:2024-05-27T07:20:22.168+0000 7ff1f71a3700 1 -- start start 2024-05-27T07:20:22.425 INFO:teuthology.orchestra.run.smithi173.stderr:2024-05-27T07:20:22.168+0000 7ff1f71a3700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7ff1f0103230 0x7ff1f019a8a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:20:22.425 INFO:teuthology.orchestra.run.smithi173.stderr:2024-05-27T07:20:22.168+0000 7ff1f71a3700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7ff1f0104360 0x7ff1f019ade0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:20:22.425 INFO:teuthology.orchestra.run.smithi173.stderr:2024-05-27T07:20:22.168+0000 7ff1f71a3700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff1f0104d00 0x7ff1f01a1d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:20:22.425 INFO:teuthology.orchestra.run.smithi173.stderr:2024-05-27T07:20:22.168+0000 7ff1f71a3700 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x7ff1f010ec90 con 0x7ff1f0104d00 2024-05-27T07:20:22.425 INFO:teuthology.orchestra.run.smithi173.stderr:2024-05-27T07:20:22.168+0000 7ff1f71a3700 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7ff1f010eb10 con 0x7ff1f0103230 2024-05-27T07:20:22.426 INFO:teuthology.orchestra.run.smithi173.stderr:2024-05-27T07:20:22.168+0000 7ff1f71a3700 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7ff1f010ee10 con 0x7ff1f0104360 2024-05-27T07:20:22.426 INFO:teuthology.orchestra.run.smithi173.stderr:2024-05-27T07:20:22.168+0000 7ff1effff700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7ff1f0104360 0x7ff1f019ade0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:20:22.426 INFO:teuthology.orchestra.run.smithi173.stderr:2024-05-27T07:20:22.168+0000 7ff1effff700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7ff1f0104360 0x7ff1f019ade0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.173:3300/0 says I am v2:172.21.15.173:54076/0 (socket says 172.21.15.173:54076) 2024-05-27T07:20:22.426 INFO:teuthology.orchestra.run.smithi173.stderr:2024-05-27T07:20:22.168+0000 7ff1effff700 1 -- 172.21.15.173:0/1021042946 learned_addr learned my addr 172.21.15.173:0/1021042946 (peer_addr_for_me v2:172.21.15.173:0/0) 2024-05-27T07:20:22.426 INFO:teuthology.orchestra.run.smithi173.stderr:2024-05-27T07:20:22.169+0000 7ff1f4f3f700 1 --2- 172.21.15.173:0/1021042946 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7ff1f0103230 0x7ff1f019a8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:20:22.426 INFO:teuthology.orchestra.run.smithi173.stderr:2024-05-27T07:20:22.169+0000 7ff1f5740700 1 --2- 172.21.15.173:0/1021042946 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff1f0104d00 0x7ff1f01a1d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:20:22.426 INFO:teuthology.orchestra.run.smithi173.stderr:2024-05-27T07:20:22.169+0000 7ff1effff700 1 -- 172.21.15.173:0/1021042946 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7ff1f0103230 msgr2=0x7ff1f019a8a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:20:22.426 INFO:teuthology.orchestra.run.smithi173.stderr:2024-05-27T07:20:22.169+0000 7ff1effff700 1 --2- 172.21.15.173:0/1021042946 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7ff1f0103230 0x7ff1f019a8a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:20:22.426 INFO:teuthology.orchestra.run.smithi173.stderr:2024-05-27T07:20:22.169+0000 7ff1effff700 1 -- 172.21.15.173:0/1021042946 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff1f0104d00 msgr2=0x7ff1f01a1d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:20:22.426 INFO:teuthology.orchestra.run.smithi173.stderr:2024-05-27T07:20:22.169+0000 7ff1effff700 1 --2- 172.21.15.173:0/1021042946 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff1f0104d00 0x7ff1f01a1d80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:20:22.427 INFO:teuthology.orchestra.run.smithi173.stderr:2024-05-27T07:20:22.169+0000 7ff1effff700 1 -- 172.21.15.173:0/1021042946 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff1f01a2400 con 0x7ff1f0104360 2024-05-27T07:20:22.427 INFO:teuthology.orchestra.run.smithi173.stderr:2024-05-27T07:20:22.169+0000 7ff1f5740700 1 --2- 172.21.15.173:0/1021042946 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff1f0104d00 0x7ff1f01a1d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-27T07:20:22.427 INFO:teuthology.orchestra.run.smithi173.stderr:2024-05-27T07:20:22.169+0000 7ff1f4f3f700 1 --2- 172.21.15.173:0/1021042946 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7ff1f0103230 0x7ff1f019a8a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-27T07:20:22.427 INFO:teuthology.orchestra.run.smithi173.stderr:2024-05-27T07:20:22.169+0000 7ff1effff700 1 --2- 172.21.15.173:0/1021042946 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7ff1f0104360 0x7ff1f019ade0 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7ff1dc00d980 tx=0x7ff1dc00dd40 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:20:22.427 INFO:teuthology.orchestra.run.smithi173.stderr:2024-05-27T07:20:22.169+0000 7ff1edffb700 1 -- 172.21.15.173:0/1021042946 <== mon.1 v2:172.21.15.173:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff1dc016070 con 0x7ff1f0104360 2024-05-27T07:20:22.427 INFO:teuthology.orchestra.run.smithi173.stderr:2024-05-27T07:20:22.169+0000 7ff1f71a3700 1 -- 172.21.15.173:0/1021042946 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff1f01a26f0 con 0x7ff1f0104360 2024-05-27T07:20:22.427 INFO:teuthology.orchestra.run.smithi173.stderr:2024-05-27T07:20:22.169+0000 7ff1edffb700 1 -- 172.21.15.173:0/1021042946 <== mon.1 v2:172.21.15.173:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ff1dc0044d0 con 0x7ff1f0104360 2024-05-27T07:20:22.427 INFO:teuthology.orchestra.run.smithi173.stderr:2024-05-27T07:20:22.169+0000 7ff1edffb700 1 -- 172.21.15.173:0/1021042946 <== mon.1 v2:172.21.15.173:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff1dc004d00 con 0x7ff1f0104360 2024-05-27T07:20:22.427 INFO:teuthology.orchestra.run.smithi173.stderr:2024-05-27T07:20:22.169+0000 7ff1f71a3700 1 -- 172.21.15.173:0/1021042946 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff1f01a2c30 con 0x7ff1f0104360 2024-05-27T07:20:22.429 INFO:teuthology.orchestra.run.smithi173.stderr:2024-05-27T07:20:22.171+0000 7ff1f71a3700 1 -- 172.21.15.173:0/1021042946 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff1f00512b0 con 0x7ff1f0104360 2024-05-27T07:20:22.429 INFO:teuthology.orchestra.run.smithi173.stderr:2024-05-27T07:20:22.172+0000 7ff1edffb700 1 -- 172.21.15.173:0/1021042946 <== mon.1 v2:172.21.15.173:3300/0 4 ==== mgrmap(e 16) ==== 149652+0+0 (secure 0 0 0) 0x7ff1dc004020 con 0x7ff1f0104360 2024-05-27T07:20:22.429 INFO:teuthology.orchestra.run.smithi173.stderr:2024-05-27T07:20:22.173+0000 7ff1edffb700 1 --2- 172.21.15.173:0/1021042946 >> [v2:172.21.15.52:6800/3513075805,v1:172.21.15.52:6801/3513075805] conn(0x7ff1e00b2880 0x7ff1e00b4d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:20:22.429 INFO:teuthology.orchestra.run.smithi173.stderr:2024-05-27T07:20:22.173+0000 7ff1edffb700 1 -- 172.21.15.173:0/1021042946 <== mon.1 v2:172.21.15.173:3300/0 5 ==== osd_map(67..67 src has 1..67) ==== 6694+0+0 (secure 0 0 0) 0x7ff1dc0e2000 con 0x7ff1f0104360 2024-05-27T07:20:22.429 INFO:teuthology.orchestra.run.smithi173.stderr:2024-05-27T07:20:22.173+0000 7ff1f4f3f700 1 --2- 172.21.15.173:0/1021042946 >> [v2:172.21.15.52:6800/3513075805,v1:172.21.15.52:6801/3513075805] conn(0x7ff1e00b2880 0x7ff1e00b4d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:20:22.429 INFO:teuthology.orchestra.run.smithi173.stderr:2024-05-27T07:20:22.173+0000 7ff1f4f3f700 1 --2- 172.21.15.173:0/1021042946 >> [v2:172.21.15.52:6800/3513075805,v1:172.21.15.52:6801/3513075805] conn(0x7ff1e00b2880 0x7ff1e00b4d90 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7ff1f0104260 tx=0x7ff1e4005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:20:22.429 INFO:teuthology.orchestra.run.smithi173.stderr:2024-05-27T07:20:22.176+0000 7ff1edffb700 1 -- 172.21.15.173:0/1021042946 <== mon.1 v2:172.21.15.173:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191963 (secure 0 0 0) 0x7ff1dc01f020 con 0x7ff1f0104360 2024-05-27T07:20:22.430 INFO:teuthology.orchestra.run.smithi173.stderr:2024-05-27T07:20:22.393+0000 7ff1f71a3700 1 -- 172.21.15.173:0/1021042946 --> [v2:172.21.15.52:6800/3513075805,v1:172.21.15.52:6801/3513075805] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi173:vg_nvme/lv_1", "target": ["mon-mgr", ""]}) -- 0x7ff1f0100870 con 0x7ff1e00b2880 2024-05-27T07:20:22.852 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:20:22 smithi173 ceph-mon[37000]: pgmap v233: 1 pgs: 1 active+clean; 577 KiB data, 322 MiB used, 983 GiB / 983 GiB avail 2024-05-27T07:20:22.852 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:20:22 smithi173 ceph-mon[37000]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-05-27T07:20:22.852 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:20:22 smithi173 ceph-mon[37000]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-05-27T07:20:22.852 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:20:22 smithi173 ceph-mon[37000]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T07:20:22.940 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:20:22 smithi105 ceph-mon[35977]: pgmap v233: 1 pgs: 1 active+clean; 577 KiB data, 322 MiB used, 983 GiB / 983 GiB avail 2024-05-27T07:20:22.940 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:20:22 smithi105 ceph-mon[35977]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-05-27T07:20:22.940 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:20:22 smithi105 ceph-mon[35977]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-05-27T07:20:22.940 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:20:22 smithi105 ceph-mon[35977]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T07:20:22.980 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:20:22 smithi052 ceph-mon[31955]: pgmap v233: 1 pgs: 1 active+clean; 577 KiB data, 322 MiB used, 983 GiB / 983 GiB avail 2024-05-27T07:20:22.980 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:20:22 smithi052 ceph-mon[31955]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-05-27T07:20:22.980 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:20:22 smithi052 ceph-mon[31955]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-05-27T07:20:22.980 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:20:22 smithi052 ceph-mon[31955]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T07:20:23.941 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:20:23 smithi105 ceph-mon[35977]: from='client.24379 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi173:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-05-27T07:20:23.979 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:20:23 smithi052 ceph-mon[31955]: from='client.24379 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi173:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-05-27T07:20:24.095 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:20:23 smithi173 ceph-mon[37000]: from='client.24379 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi173:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-05-27T07:20:24.852 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:20:24 smithi173 ceph-mon[37000]: pgmap v234: 1 pgs: 1 active+clean; 577 KiB data, 322 MiB used, 983 GiB / 983 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-05-27T07:20:24.940 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:20:24 smithi105 ceph-mon[35977]: pgmap v234: 1 pgs: 1 active+clean; 577 KiB data, 322 MiB used, 983 GiB / 983 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-05-27T07:20:24.979 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:20:24 smithi052 ceph-mon[31955]: pgmap v234: 1 pgs: 1 active+clean; 577 KiB data, 322 MiB used, 983 GiB / 983 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-05-27T07:20:25.940 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:20:25 smithi105 ceph-mon[35977]: from='client.? 172.21.15.173:0/3196982722' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "164572bf-67ed-4e8e-b32d-d7a2e8da607b"}]: dispatch 2024-05-27T07:20:25.940 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:20:25 smithi105 ceph-mon[35977]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "164572bf-67ed-4e8e-b32d-d7a2e8da607b"}]: dispatch 2024-05-27T07:20:25.940 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:20:25 smithi105 ceph-mon[35977]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "164572bf-67ed-4e8e-b32d-d7a2e8da607b"}]': finished 2024-05-27T07:20:25.940 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:20:25 smithi105 ceph-mon[35977]: osdmap e68: 12 total, 11 up, 12 in 2024-05-27T07:20:25.940 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:20:25 smithi105 ceph-mon[35977]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-05-27T07:20:25.979 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:20:25 smithi052 ceph-mon[31955]: from='client.? 172.21.15.173:0/3196982722' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "164572bf-67ed-4e8e-b32d-d7a2e8da607b"}]: dispatch 2024-05-27T07:20:25.979 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:20:25 smithi052 ceph-mon[31955]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "164572bf-67ed-4e8e-b32d-d7a2e8da607b"}]: dispatch 2024-05-27T07:20:25.979 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:20:25 smithi052 ceph-mon[31955]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "164572bf-67ed-4e8e-b32d-d7a2e8da607b"}]': finished 2024-05-27T07:20:25.980 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:20:25 smithi052 ceph-mon[31955]: osdmap e68: 12 total, 11 up, 12 in 2024-05-27T07:20:25.980 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:20:25 smithi052 ceph-mon[31955]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-05-27T07:20:26.102 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:20:25 smithi173 ceph-mon[37000]: from='client.? 172.21.15.173:0/3196982722' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "164572bf-67ed-4e8e-b32d-d7a2e8da607b"}]: dispatch 2024-05-27T07:20:26.102 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:20:25 smithi173 ceph-mon[37000]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "164572bf-67ed-4e8e-b32d-d7a2e8da607b"}]: dispatch 2024-05-27T07:20:26.103 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:20:25 smithi173 ceph-mon[37000]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "164572bf-67ed-4e8e-b32d-d7a2e8da607b"}]': finished 2024-05-27T07:20:26.103 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:20:25 smithi173 ceph-mon[37000]: osdmap e68: 12 total, 11 up, 12 in 2024-05-27T07:20:26.103 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:20:25 smithi173 ceph-mon[37000]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-05-27T07:20:26.893 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:20:26 smithi052 ceph-mon[31955]: pgmap v236: 1 pgs: 1 active+clean; 577 KiB data, 322 MiB used, 983 GiB / 983 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-05-27T07:20:26.893 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:20:26 smithi052 ceph-mon[31955]: from='client.? 172.21.15.173:0/3823835398' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-05-27T07:20:26.940 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:20:26 smithi105 ceph-mon[35977]: pgmap v236: 1 pgs: 1 active+clean; 577 KiB data, 322 MiB used, 983 GiB / 983 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-05-27T07:20:26.940 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:20:26 smithi105 ceph-mon[35977]: from='client.? 172.21.15.173:0/3823835398' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-05-27T07:20:27.102 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:20:26 smithi173 ceph-mon[37000]: pgmap v236: 1 pgs: 1 active+clean; 577 KiB data, 322 MiB used, 983 GiB / 983 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-05-27T07:20:27.102 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:20:26 smithi173 ceph-mon[37000]: from='client.? 172.21.15.173:0/3823835398' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-05-27T07:20:28.940 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:20:28 smithi105 ceph-mon[35977]: pgmap v237: 1 pgs: 1 active+clean; 577 KiB data, 322 MiB used, 983 GiB / 983 GiB avail; 72 KiB/s, 0 objects/s recovering 2024-05-27T07:20:28.979 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:20:28 smithi052 ceph-mon[31955]: pgmap v237: 1 pgs: 1 active+clean; 577 KiB data, 322 MiB used, 983 GiB / 983 GiB avail; 72 KiB/s, 0 objects/s recovering 2024-05-27T07:20:28.988 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:20:28 smithi173 ceph-mon[37000]: pgmap v237: 1 pgs: 1 active+clean; 577 KiB data, 322 MiB used, 983 GiB / 983 GiB avail; 72 KiB/s, 0 objects/s recovering 2024-05-27T07:20:30.664 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:20:30 smithi173 ceph-mon[37000]: pgmap v238: 1 pgs: 1 active+clean; 577 KiB data, 322 MiB used, 983 GiB / 983 GiB avail; 64 KiB/s, 0 objects/s recovering 2024-05-27T07:20:30.940 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:20:30 smithi105 ceph-mon[35977]: pgmap v238: 1 pgs: 1 active+clean; 577 KiB data, 322 MiB used, 983 GiB / 983 GiB avail; 64 KiB/s, 0 objects/s recovering 2024-05-27T07:20:30.979 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:20:30 smithi052 ceph-mon[31955]: pgmap v238: 1 pgs: 1 active+clean; 577 KiB data, 322 MiB used, 983 GiB / 983 GiB avail; 64 KiB/s, 0 objects/s recovering 2024-05-27T07:20:32.940 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:20:32 smithi105 ceph-mon[35977]: pgmap v239: 1 pgs: 1 active+clean; 577 KiB data, 322 MiB used, 983 GiB / 983 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-05-27T07:20:32.979 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:20:32 smithi052 ceph-mon[31955]: pgmap v239: 1 pgs: 1 active+clean; 577 KiB data, 322 MiB used, 983 GiB / 983 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-05-27T07:20:32.987 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:20:32 smithi173 ceph-mon[37000]: pgmap v239: 1 pgs: 1 active+clean; 577 KiB data, 322 MiB used, 983 GiB / 983 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-05-27T07:20:33.940 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:20:33 smithi105 ceph-mon[35977]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.11"}]: dispatch 2024-05-27T07:20:33.940 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:20:33 smithi105 ceph-mon[35977]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T07:20:33.940 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:20:33 smithi105 ceph-mon[35977]: Deploying daemon osd.11 on smithi173 2024-05-27T07:20:33.950 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:20:33 smithi173 ceph-mon[37000]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.11"}]: dispatch 2024-05-27T07:20:33.950 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:20:33 smithi173 ceph-mon[37000]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T07:20:33.950 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:20:33 smithi173 ceph-mon[37000]: Deploying daemon osd.11 on smithi173 2024-05-27T07:20:33.979 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:20:33 smithi052 ceph-mon[31955]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.11"}]: dispatch 2024-05-27T07:20:33.979 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:20:33 smithi052 ceph-mon[31955]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T07:20:33.979 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:20:33 smithi052 ceph-mon[31955]: Deploying daemon osd.11 on smithi173 2024-05-27T07:20:34.852 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:20:34 smithi173 ceph-mon[37000]: pgmap v240: 1 pgs: 1 active+clean; 577 KiB data, 322 MiB used, 983 GiB / 983 GiB avail 2024-05-27T07:20:34.940 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:20:34 smithi105 ceph-mon[35977]: pgmap v240: 1 pgs: 1 active+clean; 577 KiB data, 322 MiB used, 983 GiB / 983 GiB avail 2024-05-27T07:20:34.979 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:20:34 smithi052 ceph-mon[31955]: pgmap v240: 1 pgs: 1 active+clean; 577 KiB data, 322 MiB used, 983 GiB / 983 GiB avail 2024-05-27T07:20:36.711 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:20:36 smithi173 ceph-mon[37000]: pgmap v241: 1 pgs: 1 active+clean; 577 KiB data, 322 MiB used, 983 GiB / 983 GiB avail 2024-05-27T07:20:36.940 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:20:36 smithi105 ceph-mon[35977]: pgmap v241: 1 pgs: 1 active+clean; 577 KiB data, 322 MiB used, 983 GiB / 983 GiB avail 2024-05-27T07:20:36.979 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:20:36 smithi052 ceph-mon[31955]: pgmap v241: 1 pgs: 1 active+clean; 577 KiB data, 322 MiB used, 983 GiB / 983 GiB avail 2024-05-27T07:20:38.852 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:20:38 smithi173 ceph-mon[37000]: pgmap v242: 1 pgs: 1 active+clean; 577 KiB data, 322 MiB used, 983 GiB / 983 GiB avail 2024-05-27T07:20:38.940 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:20:38 smithi105 ceph-mon[35977]: pgmap v242: 1 pgs: 1 active+clean; 577 KiB data, 322 MiB used, 983 GiB / 983 GiB avail 2024-05-27T07:20:38.979 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:20:38 smithi052 ceph-mon[31955]: pgmap v242: 1 pgs: 1 active+clean; 577 KiB data, 322 MiB used, 983 GiB / 983 GiB avail 2024-05-27T07:20:39.979 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:20:39 smithi052 ceph-mon[31955]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' 2024-05-27T07:20:39.980 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:20:39 smithi052 ceph-mon[31955]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-27T07:20:39.980 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:20:39 smithi052 ceph-mon[31955]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' 2024-05-27T07:20:40.102 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:20:39 smithi173 ceph-mon[37000]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' 2024-05-27T07:20:40.102 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:20:39 smithi173 ceph-mon[37000]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-27T07:20:40.102 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:20:39 smithi173 ceph-mon[37000]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' 2024-05-27T07:20:40.190 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:20:39 smithi105 ceph-mon[35977]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' 2024-05-27T07:20:40.190 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:20:39 smithi105 ceph-mon[35977]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-27T07:20:40.190 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:20:39 smithi105 ceph-mon[35977]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' 2024-05-27T07:20:40.979 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:20:40 smithi052 ceph-mon[31955]: pgmap v243: 1 pgs: 1 active+clean; 577 KiB data, 322 MiB used, 983 GiB / 983 GiB avail 2024-05-27T07:20:40.991 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:20:40 smithi173 ceph-mon[37000]: pgmap v243: 1 pgs: 1 active+clean; 577 KiB data, 322 MiB used, 983 GiB / 983 GiB avail 2024-05-27T07:20:41.190 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:20:40 smithi105 ceph-mon[35977]: pgmap v243: 1 pgs: 1 active+clean; 577 KiB data, 322 MiB used, 983 GiB / 983 GiB avail 2024-05-27T07:20:42.355 INFO:teuthology.orchestra.run.smithi173.stdout:Created osd(s) 11 on host 'smithi173' 2024-05-27T07:20:42.356 INFO:teuthology.orchestra.run.smithi173.stderr:2024-05-27T07:20:42.364+0000 7ff1edffb700 1 -- 172.21.15.173:0/1021042946 <== mgr.14152 v2:172.21.15.52:6800/3513075805 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+38 (secure 0 0 0) 0x7ff1f0100870 con 0x7ff1e00b2880 2024-05-27T07:20:42.358 INFO:teuthology.orchestra.run.smithi173.stderr:2024-05-27T07:20:42.367+0000 7ff1f71a3700 1 -- 172.21.15.173:0/1021042946 >> [v2:172.21.15.52:6800/3513075805,v1:172.21.15.52:6801/3513075805] conn(0x7ff1e00b2880 msgr2=0x7ff1e00b4d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:20:42.358 INFO:teuthology.orchestra.run.smithi173.stderr:2024-05-27T07:20:42.367+0000 7ff1f71a3700 1 --2- 172.21.15.173:0/1021042946 >> [v2:172.21.15.52:6800/3513075805,v1:172.21.15.52:6801/3513075805] conn(0x7ff1e00b2880 0x7ff1e00b4d90 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7ff1f0104260 tx=0x7ff1e4005f90 comp rx=0 tx=0).stop 2024-05-27T07:20:42.358 INFO:teuthology.orchestra.run.smithi173.stderr:2024-05-27T07:20:42.367+0000 7ff1f71a3700 1 -- 172.21.15.173:0/1021042946 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7ff1f0104360 msgr2=0x7ff1f019ade0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:20:42.359 INFO:teuthology.orchestra.run.smithi173.stderr:2024-05-27T07:20:42.367+0000 7ff1f71a3700 1 --2- 172.21.15.173:0/1021042946 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7ff1f0104360 0x7ff1f019ade0 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7ff1dc00d980 tx=0x7ff1dc00dd40 comp rx=0 tx=0).stop 2024-05-27T07:20:42.359 INFO:teuthology.orchestra.run.smithi173.stderr:2024-05-27T07:20:42.368+0000 7ff1f71a3700 1 -- 172.21.15.173:0/1021042946 shutdown_connections 2024-05-27T07:20:42.359 INFO:teuthology.orchestra.run.smithi173.stderr:2024-05-27T07:20:42.368+0000 7ff1f71a3700 1 --2- 172.21.15.173:0/1021042946 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff1f0104d00 0x7ff1f01a1d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:20:42.359 INFO:teuthology.orchestra.run.smithi173.stderr:2024-05-27T07:20:42.368+0000 7ff1f71a3700 1 --2- 172.21.15.173:0/1021042946 >> [v2:172.21.15.52:6800/3513075805,v1:172.21.15.52:6801/3513075805] conn(0x7ff1e00b2880 0x7ff1e00b4d90 unknown :-1 s=CLOSED pgs=109 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:20:42.359 INFO:teuthology.orchestra.run.smithi173.stderr:2024-05-27T07:20:42.368+0000 7ff1f71a3700 1 --2- 172.21.15.173:0/1021042946 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7ff1f0103230 0x7ff1f019a8a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:20:42.359 INFO:teuthology.orchestra.run.smithi173.stderr:2024-05-27T07:20:42.368+0000 7ff1f71a3700 1 --2- 172.21.15.173:0/1021042946 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7ff1f0104360 0x7ff1f019ade0 unknown :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:20:42.360 INFO:teuthology.orchestra.run.smithi173.stderr:2024-05-27T07:20:42.368+0000 7ff1f71a3700 1 -- 172.21.15.173:0/1021042946 >> 172.21.15.173:0/1021042946 conn(0x7ff1f00fe870 msgr2=0x7ff1f0100160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:20:42.360 INFO:teuthology.orchestra.run.smithi173.stderr:2024-05-27T07:20:42.368+0000 7ff1f71a3700 1 -- 172.21.15.173:0/1021042946 shutdown_connections 2024-05-27T07:20:42.360 INFO:teuthology.orchestra.run.smithi173.stderr:2024-05-27T07:20:42.368+0000 7ff1f71a3700 1 -- 172.21.15.173:0/1021042946 wait complete. 2024-05-27T07:20:42.979 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:20:42 smithi052 ceph-mon[31955]: pgmap v244: 1 pgs: 1 active+clean; 577 KiB data, 322 MiB used, 983 GiB / 983 GiB avail 2024-05-27T07:20:42.980 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:20:42 smithi052 ceph-mon[31955]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' 2024-05-27T07:20:42.980 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:20:42 smithi052 ceph-mon[31955]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' 2024-05-27T07:20:42.980 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:20:42 smithi052 ceph-mon[31955]: from='osd.11 [v2:172.21.15.173:6824/1816308401,v1:172.21.15.173:6825/1816308401]' entity='osd.11' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]: dispatch 2024-05-27T07:20:42.980 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:20:42 smithi052 ceph-mon[31955]: from='osd.11 ' entity='osd.11' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]: dispatch 2024-05-27T07:20:42.989 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:20:42 smithi173 ceph-mon[37000]: pgmap v244: 1 pgs: 1 active+clean; 577 KiB data, 322 MiB used, 983 GiB / 983 GiB avail 2024-05-27T07:20:42.989 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:20:42 smithi173 ceph-mon[37000]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' 2024-05-27T07:20:42.990 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:20:42 smithi173 ceph-mon[37000]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' 2024-05-27T07:20:42.990 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:20:42 smithi173 ceph-mon[37000]: from='osd.11 [v2:172.21.15.173:6824/1816308401,v1:172.21.15.173:6825/1816308401]' entity='osd.11' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]: dispatch 2024-05-27T07:20:42.990 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:20:42 smithi173 ceph-mon[37000]: from='osd.11 ' entity='osd.11' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]: dispatch 2024-05-27T07:20:43.019 DEBUG:teuthology.orchestra.run.smithi173:osd.11> sudo journalctl -f -n 0 -u ceph-73df2636-1bf8-11ef-bc9b-c7b262605968@osd.11.service 2024-05-27T07:20:43.021 INFO:tasks.cephadm:Waiting for 12 OSDs to come up... 2024-05-27T07:20:43.021 DEBUG:teuthology.orchestra.run.smithi052:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:38f9d900c83c63799fbdbe61acc9a11b0d3554a6 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 73df2636-1bf8-11ef-bc9b-c7b262605968 -- ceph osd stat -f json 2024-05-27T07:20:43.190 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:20:42 smithi105 ceph-mon[35977]: pgmap v244: 1 pgs: 1 active+clean; 577 KiB data, 322 MiB used, 983 GiB / 983 GiB avail 2024-05-27T07:20:43.190 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:20:42 smithi105 ceph-mon[35977]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' 2024-05-27T07:20:43.190 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:20:42 smithi105 ceph-mon[35977]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' 2024-05-27T07:20:43.190 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:20:42 smithi105 ceph-mon[35977]: from='osd.11 [v2:172.21.15.173:6824/1816308401,v1:172.21.15.173:6825/1816308401]' entity='osd.11' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]: dispatch 2024-05-27T07:20:43.191 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:20:42 smithi105 ceph-mon[35977]: from='osd.11 ' entity='osd.11' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]: dispatch 2024-05-27T07:20:43.286 INFO:teuthology.orchestra.run.smithi052.stderr:Inferring config /var/lib/ceph/73df2636-1bf8-11ef-bc9b-c7b262605968/mon.a/config 2024-05-27T07:20:44.352 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:20:44 smithi173 ceph-mon[37000]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' 2024-05-27T07:20:44.352 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:20:44 smithi173 ceph-mon[37000]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' 2024-05-27T07:20:44.353 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:20:44 smithi173 ceph-mon[37000]: from='osd.11 ' entity='osd.11' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]': finished 2024-05-27T07:20:44.353 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:20:44 smithi173 ceph-mon[37000]: osdmap e69: 12 total, 11 up, 12 in 2024-05-27T07:20:44.353 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:20:44 smithi173 ceph-mon[37000]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-05-27T07:20:44.353 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:20:44 smithi173 ceph-mon[37000]: from='osd.11 [v2:172.21.15.173:6824/1816308401,v1:172.21.15.173:6825/1816308401]' entity='osd.11' cmd=[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi173", "root=default"]}]: dispatch 2024-05-27T07:20:44.353 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:20:44 smithi173 ceph-mon[37000]: from='osd.11 ' entity='osd.11' cmd=[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi173", "root=default"]}]: dispatch 2024-05-27T07:20:44.353 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:20:44 smithi173 ceph-mon[37000]: pgmap v246: 1 pgs: 1 active+clean; 577 KiB data, 322 MiB used, 983 GiB / 983 GiB avail 2024-05-27T07:20:44.440 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:20:44 smithi105 ceph-mon[35977]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' 2024-05-27T07:20:44.440 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:20:44 smithi105 ceph-mon[35977]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' 2024-05-27T07:20:44.440 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:20:44 smithi105 ceph-mon[35977]: from='osd.11 ' entity='osd.11' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]': finished 2024-05-27T07:20:44.440 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:20:44 smithi105 ceph-mon[35977]: osdmap e69: 12 total, 11 up, 12 in 2024-05-27T07:20:44.440 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:20:44 smithi105 ceph-mon[35977]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-05-27T07:20:44.440 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:20:44 smithi105 ceph-mon[35977]: from='osd.11 [v2:172.21.15.173:6824/1816308401,v1:172.21.15.173:6825/1816308401]' entity='osd.11' cmd=[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi173", "root=default"]}]: dispatch 2024-05-27T07:20:44.441 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:20:44 smithi105 ceph-mon[35977]: from='osd.11 ' entity='osd.11' cmd=[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi173", "root=default"]}]: dispatch 2024-05-27T07:20:44.441 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:20:44 smithi105 ceph-mon[35977]: pgmap v246: 1 pgs: 1 active+clean; 577 KiB data, 322 MiB used, 983 GiB / 983 GiB avail 2024-05-27T07:20:44.480 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:20:44 smithi052 ceph-mon[31955]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' 2024-05-27T07:20:44.480 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:20:44 smithi052 ceph-mon[31955]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' 2024-05-27T07:20:44.480 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:20:44 smithi052 ceph-mon[31955]: from='osd.11 ' entity='osd.11' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]': finished 2024-05-27T07:20:44.480 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:20:44 smithi052 ceph-mon[31955]: osdmap e69: 12 total, 11 up, 12 in 2024-05-27T07:20:44.480 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:20:44 smithi052 ceph-mon[31955]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-05-27T07:20:44.480 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:20:44 smithi052 ceph-mon[31955]: from='osd.11 [v2:172.21.15.173:6824/1816308401,v1:172.21.15.173:6825/1816308401]' entity='osd.11' cmd=[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi173", "root=default"]}]: dispatch 2024-05-27T07:20:44.480 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:20:44 smithi052 ceph-mon[31955]: from='osd.11 ' entity='osd.11' cmd=[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi173", "root=default"]}]: dispatch 2024-05-27T07:20:44.480 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:20:44 smithi052 ceph-mon[31955]: pgmap v246: 1 pgs: 1 active+clean; 577 KiB data, 322 MiB used, 983 GiB / 983 GiB avail 2024-05-27T07:20:45.051 INFO:journalctl@ceph.osd.11.smithi173.stdout:May 27 07:20:44 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-osd-11[63021]: 2024-05-27T07:20:44.774+0000 7f9d4b518700 -1 osd.11 0 waiting for initial osdmap 2024-05-27T07:20:45.052 INFO:journalctl@ceph.osd.11.smithi173.stdout:May 27 07:20:44 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-osd-11[63021]: 2024-05-27T07:20:44.788+0000 7f9d432c5700 -1 osd.11 70 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-05-27T07:20:45.259 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:20:45.266+0000 7f89e9d65700 1 -- 172.21.15.52:0/600197730 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f89e4104360 msgr2=0x7f89e4104770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:20:45.259 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:20:45.266+0000 7f89e9d65700 1 --2- 172.21.15.52:0/600197730 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f89e4104360 0x7f89e4104770 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7f89d4009990 tx=0x7f89d4009ca0 comp rx=0 tx=0).stop 2024-05-27T07:20:45.259 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:20:45.266+0000 7f89e9d65700 1 -- 172.21.15.52:0/600197730 shutdown_connections 2024-05-27T07:20:45.259 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:20:45.266+0000 7f89e9d65700 1 --2- 172.21.15.52:0/600197730 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f89e4105e30 0x7f89e410c6e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:20:45.260 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:20:45.266+0000 7f89e9d65700 1 --2- 172.21.15.52:0/600197730 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f89e4104360 0x7f89e4104770 unknown :-1 s=CLOSED pgs=148 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:20:45.260 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:20:45.266+0000 7f89e9d65700 1 --2- 172.21.15.52:0/600197730 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f89e4105490 0x7f89e41058a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:20:45.260 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:20:45.266+0000 7f89e9d65700 1 -- 172.21.15.52:0/600197730 >> 172.21.15.52:0/600197730 conn(0x7f89e40ff9c0 msgr2=0x7f89e4101e50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:20:45.260 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:20:45.266+0000 7f89e9d65700 1 -- 172.21.15.52:0/600197730 shutdown_connections 2024-05-27T07:20:45.260 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:20:45.266+0000 7f89e9d65700 1 -- 172.21.15.52:0/600197730 wait complete. 2024-05-27T07:20:45.260 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:20:45.267+0000 7f89e9d65700 1 Processor -- start 2024-05-27T07:20:45.260 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:20:45.267+0000 7f89e9d65700 1 -- start start 2024-05-27T07:20:45.260 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:20:45.267+0000 7f89e9d65700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f89e4104360 0x7f89e419a8e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:20:45.260 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:20:45.267+0000 7f89e9d65700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f89e4105490 0x7f89e419ae20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:20:45.260 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:20:45.267+0000 7f89e9d65700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f89e4105e30 0x7f89e41a1dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:20:45.261 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:20:45.267+0000 7f89e9d65700 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x7f89e410fda0 con 0x7f89e4104360 2024-05-27T07:20:45.261 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:20:45.267+0000 7f89e9d65700 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f89e410fc20 con 0x7f89e4105e30 2024-05-27T07:20:45.261 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:20:45.267+0000 7f89e9d65700 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f89e410ff20 con 0x7f89e4105490 2024-05-27T07:20:45.261 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:20:45.267+0000 7f89e37fe700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f89e4104360 0x7f89e419a8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:20:45.261 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:20:45.267+0000 7f89e37fe700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f89e4104360 0x7f89e419a8e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.52:40072/0 (socket says 172.21.15.52:40072) 2024-05-27T07:20:45.261 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:20:45.267+0000 7f89e37fe700 1 -- 172.21.15.52:0/1243184838 learned_addr learned my addr 172.21.15.52:0/1243184838 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-05-27T07:20:45.261 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:20:45.267+0000 7f89e3fff700 1 --2- 172.21.15.52:0/1243184838 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f89e4105e30 0x7f89e41a1dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:20:45.262 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:20:45.267+0000 7f89e2ffd700 1 --2- 172.21.15.52:0/1243184838 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f89e4105490 0x7f89e419ae20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:20:45.262 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:20:45.268+0000 7f89e37fe700 1 -- 172.21.15.52:0/1243184838 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f89e4105490 msgr2=0x7f89e419ae20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:20:45.262 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:20:45.268+0000 7f89e37fe700 1 --2- 172.21.15.52:0/1243184838 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f89e4105490 0x7f89e419ae20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:20:45.262 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:20:45.268+0000 7f89e37fe700 1 -- 172.21.15.52:0/1243184838 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f89e4105e30 msgr2=0x7f89e41a1dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:20:45.262 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:20:45.268+0000 7f89e37fe700 1 --2- 172.21.15.52:0/1243184838 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f89e4105e30 0x7f89e41a1dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:20:45.262 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:20:45.268+0000 7f89e37fe700 1 -- 172.21.15.52:0/1243184838 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f89e41a2440 con 0x7f89e4104360 2024-05-27T07:20:45.262 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:20:45.268+0000 7f89e3fff700 1 --2- 172.21.15.52:0/1243184838 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f89e4105e30 0x7f89e41a1dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-27T07:20:45.262 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:20:45.268+0000 7f89e2ffd700 1 --2- 172.21.15.52:0/1243184838 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f89e4105490 0x7f89e419ae20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-27T07:20:45.262 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:20:45.268+0000 7f89e37fe700 1 --2- 172.21.15.52:0/1243184838 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f89e4104360 0x7f89e419a8e0 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7f89d400b6c0 tx=0x7f89d4009580 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:20:45.262 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:20:45.268+0000 7f89e0ff9700 1 -- 172.21.15.52:0/1243184838 <== mon.0 v2:172.21.15.52:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f89d4026070 con 0x7f89e4104360 2024-05-27T07:20:45.262 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:20:45.268+0000 7f89e9d65700 1 -- 172.21.15.52:0/1243184838 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f89e41a26d0 con 0x7f89e4104360 2024-05-27T07:20:45.263 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:20:45.268+0000 7f89e0ff9700 1 -- 172.21.15.52:0/1243184838 <== mon.0 v2:172.21.15.52:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f89d4021410 con 0x7f89e4104360 2024-05-27T07:20:45.263 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:20:45.268+0000 7f89e0ff9700 1 -- 172.21.15.52:0/1243184838 <== mon.0 v2:172.21.15.52:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f89d4017650 con 0x7f89e4104360 2024-05-27T07:20:45.263 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:20:45.268+0000 7f89e9d65700 1 -- 172.21.15.52:0/1243184838 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f89e41a2b20 con 0x7f89e4104360 2024-05-27T07:20:45.264 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:20:45.269+0000 7f89e9d65700 1 -- 172.21.15.52:0/1243184838 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f89e40512b0 con 0x7f89e4104360 2024-05-27T07:20:45.264 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:20:45.271+0000 7f89e0ff9700 1 -- 172.21.15.52:0/1243184838 <== mon.0 v2:172.21.15.52:3300/0 4 ==== mgrmap(e 16) ==== 149652+0+0 (secure 0 0 0) 0x7f89d4029070 con 0x7f89e4104360 2024-05-27T07:20:45.264 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:20:45.272+0000 7f89e0ff9700 1 --2- 172.21.15.52:0/1243184838 >> [v2:172.21.15.52:6800/3513075805,v1:172.21.15.52:6801/3513075805] conn(0x7f89cc0b2880 0x7f89cc0b4d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:20:45.264 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:20:45.272+0000 7f89e0ff9700 1 -- 172.21.15.52:0/1243184838 <== mon.0 v2:172.21.15.52:3300/0 5 ==== osd_map(70..70 src has 1..70) ==== 6847+0+0 (secure 0 0 0) 0x7f89d40e2f80 con 0x7f89e4104360 2024-05-27T07:20:45.264 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:20:45.272+0000 7f89e2ffd700 1 --2- 172.21.15.52:0/1243184838 >> [v2:172.21.15.52:6800/3513075805,v1:172.21.15.52:6801/3513075805] conn(0x7f89cc0b2880 0x7f89cc0b4d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:20:45.264 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:20:45.272+0000 7f89e2ffd700 1 --2- 172.21.15.52:0/1243184838 >> [v2:172.21.15.52:6800/3513075805,v1:172.21.15.52:6801/3513075805] conn(0x7f89cc0b2880 0x7f89cc0b4d90 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7f89e419b990 tx=0x7f89d8009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:20:45.265 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:20:45.275+0000 7f89e0ff9700 1 -- 172.21.15.52:0/1243184838 <== mon.0 v2:172.21.15.52:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191963 (secure 0 0 0) 0x7f89d4013070 con 0x7f89e4104360 2024-05-27T07:20:45.413 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:20:45 smithi052 ceph-mon[31955]: from='osd.11 ' entity='osd.11' cmd='[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi173", "root=default"]}]': finished 2024-05-27T07:20:45.414 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:20:45 smithi052 ceph-mon[31955]: osdmap e70: 12 total, 11 up, 12 in 2024-05-27T07:20:45.414 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:20:45 smithi052 ceph-mon[31955]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-05-27T07:20:45.414 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:20:45 smithi052 ceph-mon[31955]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-05-27T07:20:45.414 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:20:45 smithi052 ceph-mon[31955]: from='osd.11 ' entity='osd.11' 2024-05-27T07:20:45.486 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:20:45.495+0000 7f89e9d65700 1 -- 172.21.15.52:0/1243184838 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_command({"prefix": "osd stat", "format": "json"} v 0) -- 0x7f89e402e5e0 con 0x7f89e4104360 2024-05-27T07:20:45.486 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:20:45.496+0000 7f89e0ff9700 1 -- 172.21.15.52:0/1243184838 <== mon.0 v2:172.21.15.52:3300/0 7 ==== mon_command_ack([{"prefix": "osd stat", "format": "json"}]=0 v70) ==== 74+0+133 (secure 0 0 0) 0x7f89d40ab2f0 con 0x7f89e4104360 2024-05-27T07:20:45.486 INFO:teuthology.orchestra.run.smithi052.stdout: 2024-05-27T07:20:45.489 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:20:45.498+0000 7f89e9d65700 1 -- 172.21.15.52:0/1243184838 >> [v2:172.21.15.52:6800/3513075805,v1:172.21.15.52:6801/3513075805] conn(0x7f89cc0b2880 msgr2=0x7f89cc0b4d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:20:45.489 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:20:45.498+0000 7f89e9d65700 1 --2- 172.21.15.52:0/1243184838 >> [v2:172.21.15.52:6800/3513075805,v1:172.21.15.52:6801/3513075805] conn(0x7f89cc0b2880 0x7f89cc0b4d90 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7f89e419b990 tx=0x7f89d8009260 comp rx=0 tx=0).stop 2024-05-27T07:20:45.489 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:20:45.498+0000 7f89e9d65700 1 -- 172.21.15.52:0/1243184838 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f89e4104360 msgr2=0x7f89e419a8e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:20:45.489 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:20:45.498+0000 7f89e9d65700 1 --2- 172.21.15.52:0/1243184838 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f89e4104360 0x7f89e419a8e0 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7f89d400b6c0 tx=0x7f89d4009580 comp rx=0 tx=0).stop 2024-05-27T07:20:45.489 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:20:45.498+0000 7f89e9d65700 1 -- 172.21.15.52:0/1243184838 shutdown_connections 2024-05-27T07:20:45.489 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:20:45.498+0000 7f89e9d65700 1 --2- 172.21.15.52:0/1243184838 >> [v2:172.21.15.52:6800/3513075805,v1:172.21.15.52:6801/3513075805] conn(0x7f89cc0b2880 0x7f89cc0b4d90 unknown :-1 s=CLOSED pgs=113 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:20:45.489 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:20:45.498+0000 7f89e9d65700 1 --2- 172.21.15.52:0/1243184838 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f89e4105e30 0x7f89e41a1dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:20:45.489 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:20:45.498+0000 7f89e9d65700 1 --2- 172.21.15.52:0/1243184838 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f89e4104360 0x7f89e419a8e0 unknown :-1 s=CLOSED pgs=149 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:20:45.489 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:20:45.498+0000 7f89e9d65700 1 --2- 172.21.15.52:0/1243184838 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f89e4105490 0x7f89e419ae20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:20:45.489 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:20:45.498+0000 7f89e9d65700 1 -- 172.21.15.52:0/1243184838 >> 172.21.15.52:0/1243184838 conn(0x7f89e40ff9c0 msgr2=0x7f89e41086c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:20:45.489 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:20:45.498+0000 7f89e9d65700 1 -- 172.21.15.52:0/1243184838 shutdown_connections 2024-05-27T07:20:45.489 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:20:45.498+0000 7f89e9d65700 1 -- 172.21.15.52:0/1243184838 wait complete. 2024-05-27T07:20:45.602 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:20:45 smithi173 ceph-mon[37000]: from='osd.11 ' entity='osd.11' cmd='[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi173", "root=default"]}]': finished 2024-05-27T07:20:45.602 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:20:45 smithi173 ceph-mon[37000]: osdmap e70: 12 total, 11 up, 12 in 2024-05-27T07:20:45.602 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:20:45 smithi173 ceph-mon[37000]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-05-27T07:20:45.602 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:20:45 smithi173 ceph-mon[37000]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-05-27T07:20:45.602 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:20:45 smithi173 ceph-mon[37000]: from='osd.11 ' entity='osd.11' 2024-05-27T07:20:45.691 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:20:45 smithi105 ceph-mon[35977]: from='osd.11 ' entity='osd.11' cmd='[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi173", "root=default"]}]': finished 2024-05-27T07:20:45.691 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:20:45 smithi105 ceph-mon[35977]: osdmap e70: 12 total, 11 up, 12 in 2024-05-27T07:20:45.691 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:20:45 smithi105 ceph-mon[35977]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-05-27T07:20:45.691 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:20:45 smithi105 ceph-mon[35977]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-05-27T07:20:45.691 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:20:45 smithi105 ceph-mon[35977]: from='osd.11 ' entity='osd.11' 2024-05-27T07:20:46.175 INFO:teuthology.orchestra.run.smithi052.stdout:{"epoch":70,"num_osds":12,"num_up_osds":11,"osd_up_since":1716794418,"num_in_osds":12,"osd_in_since":1716794425,"num_remapped_pgs":0} 2024-05-27T07:20:46.408 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:20:46 smithi052 ceph-mon[31955]: purged_snaps scrub starts 2024-05-27T07:20:46.408 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:20:46 smithi052 ceph-mon[31955]: purged_snaps scrub ok 2024-05-27T07:20:46.408 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:20:46 smithi052 ceph-mon[31955]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-05-27T07:20:46.408 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:20:46 smithi052 ceph-mon[31955]: from='client.? 172.21.15.52:0/1243184838' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-05-27T07:20:46.408 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:20:46 smithi052 ceph-mon[31955]: pgmap v248: 1 pgs: 1 active+clean; 577 KiB data, 322 MiB used, 983 GiB / 983 GiB avail 2024-05-27T07:20:46.408 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:20:46 smithi052 ceph-mon[31955]: osd.11 [v2:172.21.15.173:6824/1816308401,v1:172.21.15.173:6825/1816308401] boot 2024-05-27T07:20:46.408 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:20:46 smithi052 ceph-mon[31955]: osdmap e71: 12 total, 12 up, 12 in 2024-05-27T07:20:46.408 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:20:46 smithi052 ceph-mon[31955]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-05-27T07:20:46.517 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:20:46 smithi173 ceph-mon[37000]: purged_snaps scrub starts 2024-05-27T07:20:46.517 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:20:46 smithi173 ceph-mon[37000]: purged_snaps scrub ok 2024-05-27T07:20:46.517 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:20:46 smithi173 ceph-mon[37000]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-05-27T07:20:46.518 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:20:46 smithi173 ceph-mon[37000]: from='client.? 172.21.15.52:0/1243184838' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-05-27T07:20:46.518 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:20:46 smithi173 ceph-mon[37000]: pgmap v248: 1 pgs: 1 active+clean; 577 KiB data, 322 MiB used, 983 GiB / 983 GiB avail 2024-05-27T07:20:46.518 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:20:46 smithi173 ceph-mon[37000]: osd.11 [v2:172.21.15.173:6824/1816308401,v1:172.21.15.173:6825/1816308401] boot 2024-05-27T07:20:46.518 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:20:46 smithi173 ceph-mon[37000]: osdmap e71: 12 total, 12 up, 12 in 2024-05-27T07:20:46.518 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:20:46 smithi173 ceph-mon[37000]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-05-27T07:20:46.622 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:20:46 smithi105 ceph-mon[35977]: purged_snaps scrub starts 2024-05-27T07:20:46.623 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:20:46 smithi105 ceph-mon[35977]: purged_snaps scrub ok 2024-05-27T07:20:46.623 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:20:46 smithi105 ceph-mon[35977]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-05-27T07:20:46.623 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:20:46 smithi105 ceph-mon[35977]: from='client.? 172.21.15.52:0/1243184838' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-05-27T07:20:46.623 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:20:46 smithi105 ceph-mon[35977]: pgmap v248: 1 pgs: 1 active+clean; 577 KiB data, 322 MiB used, 983 GiB / 983 GiB avail 2024-05-27T07:20:46.623 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:20:46 smithi105 ceph-mon[35977]: osd.11 [v2:172.21.15.173:6824/1816308401,v1:172.21.15.173:6825/1816308401] boot 2024-05-27T07:20:46.623 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:20:46 smithi105 ceph-mon[35977]: osdmap e71: 12 total, 12 up, 12 in 2024-05-27T07:20:46.623 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:20:46 smithi105 ceph-mon[35977]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-05-27T07:20:47.176 DEBUG:teuthology.orchestra.run.smithi052:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:38f9d900c83c63799fbdbe61acc9a11b0d3554a6 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 73df2636-1bf8-11ef-bc9b-c7b262605968 -- ceph osd stat -f json 2024-05-27T07:20:47.430 INFO:teuthology.orchestra.run.smithi052.stderr:Inferring config /var/lib/ceph/73df2636-1bf8-11ef-bc9b-c7b262605968/mon.a/config 2024-05-27T07:20:47.817 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:20:47 smithi052 ceph-mon[31955]: Detected new or changed devices on smithi173 2024-05-27T07:20:47.817 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:20:47 smithi052 ceph-mon[31955]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' 2024-05-27T07:20:47.818 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:20:47 smithi052 ceph-mon[31955]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' 2024-05-27T07:20:47.818 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:20:47 smithi052 ceph-mon[31955]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-05-27T07:20:47.818 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:20:47 smithi052 ceph-mon[31955]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-05-27T07:20:47.818 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:20:47 smithi052 ceph-mon[31955]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-05-27T07:20:47.818 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:20:47 smithi052 ceph-mon[31955]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-05-27T07:20:47.818 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:20:47 smithi052 ceph-mon[31955]: Adjusting osd_memory_target on smithi173 to 4272M 2024-05-27T07:20:47.818 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:20:47 smithi052 ceph-mon[31955]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' 2024-05-27T07:20:47.818 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:20:47 smithi052 ceph-mon[31955]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T07:20:47.818 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:20:47 smithi052 ceph-mon[31955]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-27T07:20:47.818 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:20:47 smithi052 ceph-mon[31955]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' 2024-05-27T07:20:47.818 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:20:47 smithi052 ceph-mon[31955]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-27T07:20:47.819 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:20:47 smithi052 ceph-mon[31955]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T07:20:47.819 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:20:47 smithi052 ceph-mon[31955]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-27T07:20:47.819 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:20:47 smithi052 ceph-mon[31955]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' 2024-05-27T07:20:47.852 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:20:47 smithi173 ceph-mon[37000]: Detected new or changed devices on smithi173 2024-05-27T07:20:47.853 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:20:47 smithi173 ceph-mon[37000]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' 2024-05-27T07:20:47.853 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:20:47 smithi173 ceph-mon[37000]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' 2024-05-27T07:20:47.853 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:20:47 smithi173 ceph-mon[37000]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-05-27T07:20:47.853 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:20:47 smithi173 ceph-mon[37000]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-05-27T07:20:47.853 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:20:47 smithi173 ceph-mon[37000]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-05-27T07:20:47.853 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:20:47 smithi173 ceph-mon[37000]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-05-27T07:20:47.853 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:20:47 smithi173 ceph-mon[37000]: Adjusting osd_memory_target on smithi173 to 4272M 2024-05-27T07:20:47.853 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:20:47 smithi173 ceph-mon[37000]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' 2024-05-27T07:20:47.853 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:20:47 smithi173 ceph-mon[37000]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T07:20:47.853 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:20:47 smithi173 ceph-mon[37000]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-27T07:20:47.853 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:20:47 smithi173 ceph-mon[37000]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' 2024-05-27T07:20:47.854 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:20:47 smithi173 ceph-mon[37000]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-27T07:20:47.854 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:20:47 smithi173 ceph-mon[37000]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T07:20:47.854 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:20:47 smithi173 ceph-mon[37000]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-27T07:20:47.854 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:20:47 smithi173 ceph-mon[37000]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' 2024-05-27T07:20:47.940 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:20:47 smithi105 ceph-mon[35977]: Detected new or changed devices on smithi173 2024-05-27T07:20:47.940 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:20:47 smithi105 ceph-mon[35977]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' 2024-05-27T07:20:47.940 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:20:47 smithi105 ceph-mon[35977]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' 2024-05-27T07:20:47.940 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:20:47 smithi105 ceph-mon[35977]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-05-27T07:20:47.941 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:20:47 smithi105 ceph-mon[35977]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-05-27T07:20:47.941 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:20:47 smithi105 ceph-mon[35977]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-05-27T07:20:47.941 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:20:47 smithi105 ceph-mon[35977]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-05-27T07:20:47.941 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:20:47 smithi105 ceph-mon[35977]: Adjusting osd_memory_target on smithi173 to 4272M 2024-05-27T07:20:47.941 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:20:47 smithi105 ceph-mon[35977]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' 2024-05-27T07:20:47.941 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:20:47 smithi105 ceph-mon[35977]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T07:20:47.941 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:20:47 smithi105 ceph-mon[35977]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-27T07:20:47.941 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:20:47 smithi105 ceph-mon[35977]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' 2024-05-27T07:20:47.941 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:20:47 smithi105 ceph-mon[35977]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-27T07:20:47.941 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:20:47 smithi105 ceph-mon[35977]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T07:20:47.941 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:20:47 smithi105 ceph-mon[35977]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-27T07:20:47.941 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:20:47 smithi105 ceph-mon[35977]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' 2024-05-27T07:20:48.852 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:20:48 smithi173 ceph-mon[37000]: osdmap e72: 12 total, 12 up, 12 in 2024-05-27T07:20:48.852 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:20:48 smithi173 ceph-mon[37000]: pgmap v251: 1 pgs: 1 active+clean; 577 KiB data, 354 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-27T07:20:48.940 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:20:48 smithi105 ceph-mon[35977]: osdmap e72: 12 total, 12 up, 12 in 2024-05-27T07:20:48.940 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:20:48 smithi105 ceph-mon[35977]: pgmap v251: 1 pgs: 1 active+clean; 577 KiB data, 354 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-27T07:20:48.966 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:20:48 smithi052 ceph-mon[31955]: osdmap e72: 12 total, 12 up, 12 in 2024-05-27T07:20:48.966 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:20:48 smithi052 ceph-mon[31955]: pgmap v251: 1 pgs: 1 active+clean; 577 KiB data, 354 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-27T07:20:49.223 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:20:49.232+0000 7f6cc1f55700 1 -- 172.21.15.52:0/2689432982 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f6cbc1049d0 msgr2=0x7f6cbc10c6e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:20:49.224 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:20:49.232+0000 7f6cc1f55700 1 --2- 172.21.15.52:0/2689432982 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f6cbc1049d0 0x7f6cbc10c6e0 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7f6cb400b0a0 tx=0x7f6cb400b3b0 comp rx=0 tx=0).stop 2024-05-27T07:20:49.224 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:20:49.233+0000 7f6cc1f55700 1 -- 172.21.15.52:0/2689432982 shutdown_connections 2024-05-27T07:20:49.224 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:20:49.233+0000 7f6cc1f55700 1 --2- 172.21.15.52:0/2689432982 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f6cbc1049d0 0x7f6cbc10c6e0 unknown :-1 s=CLOSED pgs=150 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:20:49.224 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:20:49.233+0000 7f6cc1f55700 1 --2- 172.21.15.52:0/2689432982 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f6cbc1059a0 0x7f6cbc105db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:20:49.224 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:20:49.233+0000 7f6cc1f55700 1 --2- 172.21.15.52:0/2689432982 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f6cbc103f50 0x7f6cbc104360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:20:49.224 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:20:49.233+0000 7f6cc1f55700 1 -- 172.21.15.52:0/2689432982 >> 172.21.15.52:0/2689432982 conn(0x7f6cbc0ff9c0 msgr2=0x7f6cbc101e50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:20:49.224 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:20:49.233+0000 7f6cc1f55700 1 -- 172.21.15.52:0/2689432982 shutdown_connections 2024-05-27T07:20:49.224 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:20:49.233+0000 7f6cc1f55700 1 -- 172.21.15.52:0/2689432982 wait complete. 2024-05-27T07:20:49.224 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:20:49.233+0000 7f6cc1f55700 1 Processor -- start 2024-05-27T07:20:49.224 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:20:49.234+0000 7f6cc1f55700 1 -- start start 2024-05-27T07:20:49.226 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:20:49.234+0000 7f6cc1f55700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f6cbc103f50 0x7f6cbc19cb80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:20:49.226 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:20:49.234+0000 7f6cc1f55700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f6cbc1049d0 0x7f6cbc19d0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:20:49.226 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:20:49.234+0000 7f6cc1f55700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f6cbc1059a0 0x7f6cbc1a4060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:20:49.227 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:20:49.234+0000 7f6cc1f55700 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x7f6cbc10fee0 con 0x7f6cbc1049d0 2024-05-27T07:20:49.227 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:20:49.234+0000 7f6cc1f55700 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f6cbc10fd60 con 0x7f6cbc1059a0 2024-05-27T07:20:49.227 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:20:49.234+0000 7f6cc1f55700 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f6cbc110060 con 0x7f6cbc103f50 2024-05-27T07:20:49.227 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:20:49.234+0000 7f6cbaffd700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f6cbc1049d0 0x7f6cbc19d0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:20:49.227 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:20:49.234+0000 7f6cbaffd700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f6cbc1049d0 0x7f6cbc19d0c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.52:40076/0 (socket says 172.21.15.52:40076) 2024-05-27T07:20:49.227 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:20:49.234+0000 7f6cbaffd700 1 -- 172.21.15.52:0/2808016305 learned_addr learned my addr 172.21.15.52:0/2808016305 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-05-27T07:20:49.227 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:20:49.234+0000 7f6cbb7fe700 1 --2- 172.21.15.52:0/2808016305 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f6cbc103f50 0x7f6cbc19cb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:20:49.227 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:20:49.234+0000 7f6cbbfff700 1 --2- 172.21.15.52:0/2808016305 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f6cbc1059a0 0x7f6cbc1a4060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:20:49.227 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:20:49.235+0000 7f6cbaffd700 1 -- 172.21.15.52:0/2808016305 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f6cbc103f50 msgr2=0x7f6cbc19cb80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:20:49.227 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:20:49.235+0000 7f6cbaffd700 1 --2- 172.21.15.52:0/2808016305 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f6cbc103f50 0x7f6cbc19cb80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:20:49.227 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:20:49.235+0000 7f6cbaffd700 1 -- 172.21.15.52:0/2808016305 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f6cbc1059a0 msgr2=0x7f6cbc1a4060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:20:49.227 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:20:49.235+0000 7f6cbaffd700 1 --2- 172.21.15.52:0/2808016305 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f6cbc1059a0 0x7f6cbc1a4060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:20:49.228 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:20:49.235+0000 7f6cbaffd700 1 -- 172.21.15.52:0/2808016305 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6cbc1a46e0 con 0x7f6cbc1049d0 2024-05-27T07:20:49.228 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:20:49.235+0000 7f6cbaffd700 1 --2- 172.21.15.52:0/2808016305 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f6cbc1049d0 0x7f6cbc19d0c0 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7f6ca400d950 tx=0x7f6ca400dc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:20:49.228 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:20:49.235+0000 7f6cb8ff9700 1 -- 172.21.15.52:0/2808016305 <== mon.0 v2:172.21.15.52:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6ca4016070 con 0x7f6cbc1049d0 2024-05-27T07:20:49.228 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:20:49.235+0000 7f6cb8ff9700 1 -- 172.21.15.52:0/2808016305 <== mon.0 v2:172.21.15.52:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f6ca4009df0 con 0x7f6cbc1049d0 2024-05-27T07:20:49.228 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:20:49.235+0000 7f6cb8ff9700 1 -- 172.21.15.52:0/2808016305 <== mon.0 v2:172.21.15.52:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6ca400f3c0 con 0x7f6cbc1049d0 2024-05-27T07:20:49.228 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:20:49.235+0000 7f6cc1f55700 1 -- 172.21.15.52:0/2808016305 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6cbc1a49d0 con 0x7f6cbc1049d0 2024-05-27T07:20:49.228 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:20:49.235+0000 7f6cc1f55700 1 -- 172.21.15.52:0/2808016305 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6cbc1a4f10 con 0x7f6cbc1049d0 2024-05-27T07:20:49.228 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:20:49.236+0000 7f6cc1f55700 1 -- 172.21.15.52:0/2808016305 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6cbc0512b0 con 0x7f6cbc1049d0 2024-05-27T07:20:49.229 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:20:49.238+0000 7f6cb8ff9700 1 -- 172.21.15.52:0/2808016305 <== mon.0 v2:172.21.15.52:3300/0 4 ==== mgrmap(e 16) ==== 149652+0+0 (secure 0 0 0) 0x7f6ca4019070 con 0x7f6cbc1049d0 2024-05-27T07:20:49.229 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:20:49.239+0000 7f6cb8ff9700 1 --2- 172.21.15.52:0/2808016305 >> [v2:172.21.15.52:6800/3513075805,v1:172.21.15.52:6801/3513075805] conn(0x7f6ca80b2880 0x7f6ca80b4d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:20:49.229 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:20:49.239+0000 7f6cbb7fe700 1 --2- 172.21.15.52:0/2808016305 >> [v2:172.21.15.52:6800/3513075805,v1:172.21.15.52:6801/3513075805] conn(0x7f6ca80b2880 0x7f6ca80b4d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:20:49.229 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:20:49.239+0000 7f6cb8ff9700 1 -- 172.21.15.52:0/2808016305 <== mon.0 v2:172.21.15.52:3300/0 5 ==== osd_map(72..72 src has 1..72) ==== 7127+0+0 (secure 0 0 0) 0x7f6ca40e1a00 con 0x7f6cbc1049d0 2024-05-27T07:20:49.229 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:20:49.239+0000 7f6cbb7fe700 1 --2- 172.21.15.52:0/2808016305 >> [v2:172.21.15.52:6800/3513075805,v1:172.21.15.52:6801/3513075805] conn(0x7f6ca80b2880 0x7f6ca80b4d90 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7f6cbc105390 tx=0x7f6cac00b410 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:20:49.232 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:20:49.242+0000 7f6cb8ff9700 1 -- 172.21.15.52:0/2808016305 <== mon.0 v2:172.21.15.52:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191963 (secure 0 0 0) 0x7f6ca40a9b20 con 0x7f6cbc1049d0 2024-05-27T07:20:49.446 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:20:49.456+0000 7f6cc1f55700 1 -- 172.21.15.52:0/2808016305 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_command({"prefix": "osd stat", "format": "json"} v 0) -- 0x7f6cbc1a5200 con 0x7f6cbc1049d0 2024-05-27T07:20:49.446 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:20:49.456+0000 7f6cb8ff9700 1 -- 172.21.15.52:0/2808016305 <== mon.0 v2:172.21.15.52:3300/0 7 ==== mon_command_ack([{"prefix": "osd stat", "format": "json"}]=0 v72) ==== 74+0+133 (secure 0 0 0) 0x7f6ca40a9420 con 0x7f6cbc1049d0 2024-05-27T07:20:49.447 INFO:teuthology.orchestra.run.smithi052.stdout: 2024-05-27T07:20:49.449 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:20:49.459+0000 7f6cc1f55700 1 -- 172.21.15.52:0/2808016305 >> [v2:172.21.15.52:6800/3513075805,v1:172.21.15.52:6801/3513075805] conn(0x7f6ca80b2880 msgr2=0x7f6ca80b4d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:20:49.449 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:20:49.459+0000 7f6cc1f55700 1 --2- 172.21.15.52:0/2808016305 >> [v2:172.21.15.52:6800/3513075805,v1:172.21.15.52:6801/3513075805] conn(0x7f6ca80b2880 0x7f6ca80b4d90 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7f6cbc105390 tx=0x7f6cac00b410 comp rx=0 tx=0).stop 2024-05-27T07:20:49.449 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:20:49.459+0000 7f6cc1f55700 1 -- 172.21.15.52:0/2808016305 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f6cbc1049d0 msgr2=0x7f6cbc19d0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:20:49.449 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:20:49.459+0000 7f6cc1f55700 1 --2- 172.21.15.52:0/2808016305 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f6cbc1049d0 0x7f6cbc19d0c0 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7f6ca400d950 tx=0x7f6ca400dc60 comp rx=0 tx=0).stop 2024-05-27T07:20:49.450 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:20:49.459+0000 7f6cc1f55700 1 -- 172.21.15.52:0/2808016305 shutdown_connections 2024-05-27T07:20:49.450 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:20:49.459+0000 7f6cc1f55700 1 --2- 172.21.15.52:0/2808016305 >> [v2:172.21.15.52:6800/3513075805,v1:172.21.15.52:6801/3513075805] conn(0x7f6ca80b2880 0x7f6ca80b4d90 unknown :-1 s=CLOSED pgs=116 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:20:49.450 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:20:49.459+0000 7f6cc1f55700 1 --2- 172.21.15.52:0/2808016305 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f6cbc1059a0 0x7f6cbc1a4060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:20:49.450 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:20:49.459+0000 7f6cc1f55700 1 --2- 172.21.15.52:0/2808016305 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f6cbc103f50 0x7f6cbc19cb80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:20:49.450 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:20:49.459+0000 7f6cc1f55700 1 --2- 172.21.15.52:0/2808016305 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f6cbc1049d0 0x7f6cbc19d0c0 unknown :-1 s=CLOSED pgs=151 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:20:49.450 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:20:49.459+0000 7f6cc1f55700 1 -- 172.21.15.52:0/2808016305 >> 172.21.15.52:0/2808016305 conn(0x7f6cbc0ff9c0 msgr2=0x7f6cbc076af0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:20:49.450 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:20:49.459+0000 7f6cc1f55700 1 -- 172.21.15.52:0/2808016305 shutdown_connections 2024-05-27T07:20:49.450 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:20:49.459+0000 7f6cc1f55700 1 -- 172.21.15.52:0/2808016305 wait complete. 2024-05-27T07:20:49.852 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:20:49 smithi173 ceph-mon[37000]: from='client.? 172.21.15.52:0/2808016305' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-05-27T07:20:49.940 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:20:49 smithi105 ceph-mon[35977]: from='client.? 172.21.15.52:0/2808016305' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-05-27T07:20:49.979 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:20:49 smithi052 ceph-mon[31955]: from='client.? 172.21.15.52:0/2808016305' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-05-27T07:20:50.105 INFO:teuthology.orchestra.run.smithi052.stdout:{"epoch":72,"num_osds":12,"num_up_osds":12,"osd_up_since":1716794445,"num_in_osds":12,"osd_in_since":1716794425,"num_remapped_pgs":0} 2024-05-27T07:20:50.106 DEBUG:teuthology.orchestra.run.smithi052:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:38f9d900c83c63799fbdbe61acc9a11b0d3554a6 shell --fsid 73df2636-1bf8-11ef-bc9b-c7b262605968 -- ceph osd dump --format=json 2024-05-27T07:20:50.357 INFO:teuthology.orchestra.run.smithi052.stderr:Inferring config /var/lib/ceph/73df2636-1bf8-11ef-bc9b-c7b262605968/mon.a/config 2024-05-27T07:20:50.729 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:20:50 smithi052 ceph-mon[31955]: pgmap v252: 1 pgs: 1 active+clean; 577 KiB data, 354 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-27T07:20:50.940 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:20:50 smithi105 ceph-mon[35977]: pgmap v252: 1 pgs: 1 active+clean; 577 KiB data, 354 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-27T07:20:51.102 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:20:50 smithi173 ceph-mon[37000]: pgmap v252: 1 pgs: 1 active+clean; 577 KiB data, 354 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-27T07:20:52.003 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:20:52.012+0000 7f016d829700 1 -- 172.21.15.52:0/1734252298 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f01681034b0 msgr2=0x7f01681038c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:20:52.003 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:20:52.012+0000 7f016d829700 1 --2- 172.21.15.52:0/1734252298 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f01681034b0 0x7f01681038c0 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7f015800b0d0 tx=0x7f015800b3e0 comp rx=0 tx=0).stop 2024-05-27T07:20:52.003 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:20:52.012+0000 7f016d829700 1 -- 172.21.15.52:0/1734252298 shutdown_connections 2024-05-27T07:20:52.003 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:20:52.012+0000 7f016d829700 1 --2- 172.21.15.52:0/1734252298 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f0168108730 0x7f0168110090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:20:52.004 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:20:52.012+0000 7f016d829700 1 --2- 172.21.15.52:0/1734252298 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f01681034b0 0x7f01681038c0 unknown :-1 s=CLOSED pgs=152 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:20:52.004 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:20:52.012+0000 7f016d829700 1 --2- 172.21.15.52:0/1734252298 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f0168107de0 0x7f01681081f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:20:52.004 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:20:52.012+0000 7f016d829700 1 -- 172.21.15.52:0/1734252298 >> 172.21.15.52:0/1734252298 conn(0x7f01680fede0 msgr2=0x7f0168101200 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:20:52.004 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:20:52.012+0000 7f016d829700 1 -- 172.21.15.52:0/1734252298 shutdown_connections 2024-05-27T07:20:52.004 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:20:52.012+0000 7f016d829700 1 -- 172.21.15.52:0/1734252298 wait complete. 2024-05-27T07:20:52.004 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:20:52.013+0000 7f016d829700 1 Processor -- start 2024-05-27T07:20:52.004 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:20:52.013+0000 7f016d829700 1 -- start start 2024-05-27T07:20:52.006 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:20:52.014+0000 7f016d829700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f01681034b0 0x7f016819bd50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:20:52.006 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:20:52.014+0000 7f016d829700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f0168107de0 0x7f016819c290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:20:52.006 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:20:52.014+0000 7f016d829700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f0168108730 0x7f016819c7f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:20:52.006 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:20:52.014+0000 7f016d829700 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x7f0168113430 con 0x7f0168107de0 2024-05-27T07:20:52.006 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:20:52.014+0000 7f016d829700 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f01681132b0 con 0x7f01681034b0 2024-05-27T07:20:52.006 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:20:52.014+0000 7f016d829700 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f01681135b0 con 0x7f0168108730 2024-05-27T07:20:52.006 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:20:52.014+0000 7f01667fc700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f0168107de0 0x7f016819c290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:20:52.006 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:20:52.014+0000 7f01667fc700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f0168107de0 0x7f016819c290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.52:47006/0 (socket says 172.21.15.52:47006) 2024-05-27T07:20:52.006 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:20:52.014+0000 7f01667fc700 1 -- 172.21.15.52:0/1759416754 learned_addr learned my addr 172.21.15.52:0/1759416754 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-05-27T07:20:52.006 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:20:52.014+0000 7f0166ffd700 1 --2- 172.21.15.52:0/1759416754 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f01681034b0 0x7f016819bd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:20:52.007 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:20:52.014+0000 7f01677fe700 1 --2- 172.21.15.52:0/1759416754 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f0168108730 0x7f016819c7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:20:52.007 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:20:52.014+0000 7f01667fc700 1 -- 172.21.15.52:0/1759416754 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f0168108730 msgr2=0x7f016819c7f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:20:52.007 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:20:52.014+0000 7f01667fc700 1 --2- 172.21.15.52:0/1759416754 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f0168108730 0x7f016819c7f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:20:52.007 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:20:52.014+0000 7f01667fc700 1 -- 172.21.15.52:0/1759416754 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f01681034b0 msgr2=0x7f016819bd50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:20:52.007 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:20:52.014+0000 7f01667fc700 1 --2- 172.21.15.52:0/1759416754 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f01681034b0 0x7f016819bd50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:20:52.007 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:20:52.014+0000 7f01667fc700 1 -- 172.21.15.52:0/1759416754 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f01681a70a0 con 0x7f0168107de0 2024-05-27T07:20:52.007 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:20:52.014+0000 7f01677fe700 1 --2- 172.21.15.52:0/1759416754 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f0168108730 0x7f016819c7f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-27T07:20:52.007 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:20:52.014+0000 7f01667fc700 1 --2- 172.21.15.52:0/1759416754 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f0168107de0 0x7f016819c290 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7f015000d980 tx=0x7f015000dc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:20:52.007 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:20:52.014+0000 7f016c827700 1 -- 172.21.15.52:0/1759416754 <== mon.0 v2:172.21.15.52:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0150017070 con 0x7f0168107de0 2024-05-27T07:20:52.007 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:20:52.015+0000 7f016d829700 1 -- 172.21.15.52:0/1759416754 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f01681a7390 con 0x7f0168107de0 2024-05-27T07:20:52.007 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:20:52.015+0000 7f016c827700 1 -- 172.21.15.52:0/1759416754 <== mon.0 v2:172.21.15.52:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f0150009de0 con 0x7f0168107de0 2024-05-27T07:20:52.007 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:20:52.015+0000 7f016c827700 1 -- 172.21.15.52:0/1759416754 <== mon.0 v2:172.21.15.52:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f015000f3c0 con 0x7f0168107de0 2024-05-27T07:20:52.007 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:20:52.015+0000 7f016d829700 1 -- 172.21.15.52:0/1759416754 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f01681a78d0 con 0x7f0168107de0 2024-05-27T07:20:52.007 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:20:52.016+0000 7f016d829700 1 -- 172.21.15.52:0/1759416754 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f01680512b0 con 0x7f0168107de0 2024-05-27T07:20:52.008 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:20:52.017+0000 7f016c827700 1 -- 172.21.15.52:0/1759416754 <== mon.0 v2:172.21.15.52:3300/0 4 ==== mgrmap(e 16) ==== 149652+0+0 (secure 0 0 0) 0x7f0150009930 con 0x7f0168107de0 2024-05-27T07:20:52.009 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:20:52.019+0000 7f016c827700 1 --2- 172.21.15.52:0/1759416754 >> [v2:172.21.15.52:6800/3513075805,v1:172.21.15.52:6801/3513075805] conn(0x7f01540b2880 0x7f01540b4d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:20:52.009 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:20:52.019+0000 7f0166ffd700 1 --2- 172.21.15.52:0/1759416754 >> [v2:172.21.15.52:6800/3513075805,v1:172.21.15.52:6801/3513075805] conn(0x7f01540b2880 0x7f01540b4d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:20:52.009 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:20:52.019+0000 7f016c827700 1 -- 172.21.15.52:0/1759416754 <== mon.0 v2:172.21.15.52:3300/0 5 ==== osd_map(72..72 src has 1..72) ==== 7127+0+0 (secure 0 0 0) 0x7f01500e1840 con 0x7f0168107de0 2024-05-27T07:20:52.009 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:20:52.019+0000 7f0166ffd700 1 --2- 172.21.15.52:0/1759416754 >> [v2:172.21.15.52:6800/3513075805,v1:172.21.15.52:6801/3513075805] conn(0x7f01540b2880 0x7f01540b4d90 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7f0158009900 tx=0x7f015800b040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:20:52.012 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:20:52.022+0000 7f016c827700 1 -- 172.21.15.52:0/1759416754 <== mon.0 v2:172.21.15.52:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191963 (secure 0 0 0) 0x7f01500a9a00 con 0x7f0168107de0 2024-05-27T07:20:52.242 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:20:52.252+0000 7f016d829700 1 -- 172.21.15.52:0/1759416754 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f016802e5e0 con 0x7f0168107de0 2024-05-27T07:20:52.243 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:20:52.253+0000 7f016c827700 1 -- 172.21.15.52:0/1759416754 <== mon.0 v2:172.21.15.52:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v72) ==== 74+0+18454 (secure 0 0 0) 0x7f01500a9300 con 0x7f0168107de0 2024-05-27T07:20:52.244 INFO:teuthology.orchestra.run.smithi052.stdout: 2024-05-27T07:20:52.244 INFO:teuthology.orchestra.run.smithi052.stdout:{"epoch":72,"fsid":"73df2636-1bf8-11ef-bc9b-c7b262605968","created":"2024-05-27T07:13:34.738032+0000","modified":"2024-05-27T07:20:47.599599+0000","last_up_change":"2024-05-27T07:20:45.777597+0000","last_in_change":"2024-05-27T07:20:25.314823+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-05-27T07:16:57.644034+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"347b8a89-9dbe-4848-83b0-fd0e9374f33d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.52:6802","nonce":1626508899},{"type":"v1","addr":"172.21.15.52:6803","nonce":1626508899}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.52:6804","nonce":1626508899},{"type":"v1","addr":"172.21.15.52:6805","nonce":1626508899}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.52:6808","nonce":1626508899},{"type":"v1","addr":"172.21.15.52:6809","nonce":1626508899}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.52:6806","nonce":1626508899},{"type":"v1","addr":"172.21.15.52:6807","nonce":1626508899}]},"public_addr":"172.21.15.52:6803/1626508899","cluster_addr":"172.21.15.52:6805/1626508899","heartbeat_back_addr":"172.21.15.52:6809/1626508899","heartbeat_front_addr":"172.21.15.52:6807/1626508899","state":["exists","up"]},{"osd":1,"uuid":"7a3623d4-e087-40cb-9336-d4a5c52203d4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":38,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6800","nonce":2900093195},{"type":"v1","addr":"172.21.15.105:6801","nonce":2900093195}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6802","nonce":2900093195},{"type":"v1","addr":"172.21.15.105:6803","nonce":2900093195}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6806","nonce":2900093195},{"type":"v1","addr":"172.21.15.105:6807","nonce":2900093195}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6804","nonce":2900093195},{"type":"v1","addr":"172.21.15.105:6805","nonce":2900093195}]},"public_addr":"172.21.15.105:6801/2900093195","cluster_addr":"172.21.15.105:6803/2900093195","heartbeat_back_addr":"172.21.15.105:6807/2900093195","heartbeat_front_addr":"172.21.15.105:6805/2900093195","state":["exists","up"]},{"osd":2,"uuid":"41a9df5b-882a-47da-bb26-7f4d26ccf6c1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6800","nonce":2251189314},{"type":"v1","addr":"172.21.15.173:6801","nonce":2251189314}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6802","nonce":2251189314},{"type":"v1","addr":"172.21.15.173:6803","nonce":2251189314}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6806","nonce":2251189314},{"type":"v1","addr":"172.21.15.173:6807","nonce":2251189314}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6804","nonce":2251189314},{"type":"v1","addr":"172.21.15.173:6805","nonce":2251189314}]},"public_addr":"172.21.15.173:6801/2251189314","cluster_addr":"172.21.15.173:6803/2251189314","heartbeat_back_addr":"172.21.15.173:6807/2251189314","heartbeat_front_addr":"172.21.15.173:6805/2251189314","state":["exists","up"]},{"osd":3,"uuid":"f1245439-42a5-46c5-a998-e0a78f9df3fb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":26,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.52:6810","nonce":57920794},{"type":"v1","addr":"172.21.15.52:6811","nonce":57920794}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.52:6812","nonce":57920794},{"type":"v1","addr":"172.21.15.52:6813","nonce":57920794}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.52:6816","nonce":57920794},{"type":"v1","addr":"172.21.15.52:6817","nonce":57920794}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.52:6814","nonce":57920794},{"type":"v1","addr":"172.21.15.52:6815","nonce":57920794}]},"public_addr":"172.21.15.52:6811/57920794","cluster_addr":"172.21.15.52:6813/57920794","heartbeat_back_addr":"172.21.15.52:6817/57920794","heartbeat_front_addr":"172.21.15.52:6815/57920794","state":["exists","up"]},{"osd":4,"uuid":"2592d1cc-1726-4359-83d8-a2e6314b54ec","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":32,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6808","nonce":3441134818},{"type":"v1","addr":"172.21.15.105:6809","nonce":3441134818}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6810","nonce":3441134818},{"type":"v1","addr":"172.21.15.105:6811","nonce":3441134818}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6814","nonce":3441134818},{"type":"v1","addr":"172.21.15.105:6815","nonce":3441134818}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6812","nonce":3441134818},{"type":"v1","addr":"172.21.15.105:6813","nonce":3441134818}]},"public_addr":"172.21.15.105:6809/3441134818","cluster_addr":"172.21.15.105:6811/3441134818","heartbeat_back_addr":"172.21.15.105:6815/3441134818","heartbeat_front_addr":"172.21.15.105:6813/3441134818","state":["exists","up"]},{"osd":5,"uuid":"27e98ee2-4a92-4972-b2a3-959d0ed90981","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":37,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6808","nonce":2989239729},{"type":"v1","addr":"172.21.15.173:6809","nonce":2989239729}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6810","nonce":2989239729},{"type":"v1","addr":"172.21.15.173:6811","nonce":2989239729}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6814","nonce":2989239729},{"type":"v1","addr":"172.21.15.173:6815","nonce":2989239729}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6812","nonce":2989239729},{"type":"v1","addr":"172.21.15.173:6813","nonce":2989239729}]},"public_addr":"172.21.15.173:6809/2989239729","cluster_addr":"172.21.15.173:6811/2989239729","heartbeat_back_addr":"172.21.15.173:6815/2989239729","heartbeat_front_addr":"172.21.15.173:6813/2989239729","state":["exists","up"]},{"osd":6,"uuid":"2b0ec0cb-f438-4b2c-8642-0778ed374e9a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":43,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.52:6818","nonce":4244423065},{"type":"v1","addr":"172.21.15.52:6819","nonce":4244423065}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.52:6820","nonce":4244423065},{"type":"v1","addr":"172.21.15.52:6821","nonce":4244423065}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.52:6824","nonce":4244423065},{"type":"v1","addr":"172.21.15.52:6825","nonce":4244423065}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.52:6822","nonce":4244423065},{"type":"v1","addr":"172.21.15.52:6823","nonce":4244423065}]},"public_addr":"172.21.15.52:6819/4244423065","cluster_addr":"172.21.15.52:6821/4244423065","heartbeat_back_addr":"172.21.15.52:6825/4244423065","heartbeat_front_addr":"172.21.15.52:6823/4244423065","state":["exists","up"]},{"osd":7,"uuid":"cb2a7a6b-8015-43ea-865a-be144e45c51a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":48,"up_thru":66,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6816","nonce":2063611552},{"type":"v1","addr":"172.21.15.105:6817","nonce":2063611552}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6818","nonce":2063611552},{"type":"v1","addr":"172.21.15.105:6819","nonce":2063611552}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6822","nonce":2063611552},{"type":"v1","addr":"172.21.15.105:6823","nonce":2063611552}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6820","nonce":2063611552},{"type":"v1","addr":"172.21.15.105:6821","nonce":2063611552}]},"public_addr":"172.21.15.105:6817/2063611552","cluster_addr":"172.21.15.105:6819/2063611552","heartbeat_back_addr":"172.21.15.105:6823/2063611552","heartbeat_front_addr":"172.21.15.105:6821/2063611552","state":["exists","up"]},{"osd":8,"uuid":"c2fc565d-a0b9-4396-a209-bd884cddcfe3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":54,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6816","nonce":4131399852},{"type":"v1","addr":"172.21.15.173:6817","nonce":4131399852}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6818","nonce":4131399852},{"type":"v1","addr":"172.21.15.173:6819","nonce":4131399852}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6822","nonce":4131399852},{"type":"v1","addr":"172.21.15.173:6823","nonce":4131399852}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6820","nonce":4131399852},{"type":"v1","addr":"172.21.15.173:6821","nonce":4131399852}]},"public_addr":"172.21.15.173:6817/4131399852","cluster_addr":"172.21.15.173:6819/4131399852","heartbeat_back_addr":"172.21.15.173:6823/4131399852","heartbeat_front_addr":"172.21.15.173:6821/4131399852","state":["exists","up"]},{"osd":9,"uuid":"78fa7f17-1dad-4262-be40-31fa82e75520","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":60,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.52:6826","nonce":479119778},{"type":"v1","addr":"172.21.15.52:6827","nonce":479119778}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.52:6828","nonce":479119778},{"type":"v1","addr":"172.21.15.52:6829","nonce":479119778}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.52:6832","nonce":479119778},{"type":"v1","addr":"172.21.15.52:6833","nonce":479119778}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.52:6830","nonce":479119778},{"type":"v1","addr":"172.21.15.52:6831","nonce":479119778}]},"public_addr":"172.21.15.52:6827/479119778","cluster_addr":"172.21.15.52:6829/479119778","heartbeat_back_addr":"172.21.15.52:6833/479119778","heartbeat_front_addr":"172.21.15.52:6831/479119778","state":["exists","up"]},{"osd":10,"uuid":"e6f8e753-e15c-45b5-8ee9-d50f384559f9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":65,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6824","nonce":4032386628},{"type":"v1","addr":"172.21.15.105:6825","nonce":4032386628}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6826","nonce":4032386628},{"type":"v1","addr":"172.21.15.105:6827","nonce":4032386628}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6830","nonce":4032386628},{"type":"v1","addr":"172.21.15.105:6831","nonce":4032386628}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6828","nonce":4032386628},{"type":"v1","addr":"172.21.15.105:6829","nonce":4032386628}]},"public_addr":"172.21.15.105:6825/4032386628","cluster_addr":"172.21.15.105:6827/4032386628","heartbeat_back_addr":"172.21.15.105:6831/4032386628","heartbeat_front_addr":"172.21.15.105:6829/4032386628","state":["exists","up"]},{"osd":11,"uuid":"164572bf-67ed-4e8e-b32d-d7a2e8da607b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":71,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6824","nonce":1816308401},{"type":"v1","addr":"172.21.15.173:6825","nonce":1816308401}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6826","nonce":1816308401},{"type":"v1","addr":"172.21.15.173:6827","nonce":1816308401}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6830","nonce":1816308401},{"type":"v1","addr":"172.21.15.173:6831","nonce":1816308401}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6828","nonce":1816308401},{"type":"v1","addr":"172.21.15.173:6829","nonce":1816308401}]},"public_addr":"172.21.15.173:6825/1816308401","cluster_addr":"172.21.15.173:6827/1816308401","heartbeat_back_addr":"172.21.15.173:6831/1816308401","heartbeat_front_addr":"172.21.15.173:6829/1816308401","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-27T07:16:06.279220+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-27T07:16:29.668167+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-27T07:16:53.903808+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-27T07:17:18.787372+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-27T07:17:43.480784+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-27T07:18:08.816030+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-27T07:18:34.076148+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-27T07:19:00.283468+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-27T07:19:25.403023+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-27T07:19:52.139173+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-27T07:20:17.530304+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-27T07:20:43.665632+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.52:6801/1631683163":"2024-05-28T07:14:29.544633+0000","172.21.15.52:0/2776144659":"2024-05-28T07:14:29.544633+0000","172.21.15.52:6800/1631683163":"2024-05-28T07:14:29.544633+0000","172.21.15.52:0/3675529097":"2024-05-28T07:14:01.960550+0000","172.21.15.52:0/727808107":"2024-05-28T07:14:01.960550+0000","172.21.15.52:0/3839436833":"2024-05-28T07:14:29.544633+0000","172.21.15.52:6800/587662512":"2024-05-28T07:14:01.960550+0000","172.21.15.52:6801/587662512":"2024-05-28T07:14:01.960550+0000","172.21.15.52:0/853867465":"2024-05-28T07:14:01.960550+0000","172.21.15.52:0/3480918279":"2024-05-28T07:14:29.544633+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-05-27T07:20:52.246 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:20:52.255+0000 7f016d829700 1 -- 172.21.15.52:0/1759416754 >> [v2:172.21.15.52:6800/3513075805,v1:172.21.15.52:6801/3513075805] conn(0x7f01540b2880 msgr2=0x7f01540b4d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:20:52.246 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:20:52.255+0000 7f016d829700 1 --2- 172.21.15.52:0/1759416754 >> [v2:172.21.15.52:6800/3513075805,v1:172.21.15.52:6801/3513075805] conn(0x7f01540b2880 0x7f01540b4d90 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7f0158009900 tx=0x7f015800b040 comp rx=0 tx=0).stop 2024-05-27T07:20:52.246 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:20:52.255+0000 7f016d829700 1 -- 172.21.15.52:0/1759416754 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f0168107de0 msgr2=0x7f016819c290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:20:52.246 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:20:52.255+0000 7f016d829700 1 --2- 172.21.15.52:0/1759416754 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f0168107de0 0x7f016819c290 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7f015000d980 tx=0x7f015000dc90 comp rx=0 tx=0).stop 2024-05-27T07:20:52.246 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:20:52.255+0000 7f016d829700 1 -- 172.21.15.52:0/1759416754 shutdown_connections 2024-05-27T07:20:52.246 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:20:52.255+0000 7f016d829700 1 --2- 172.21.15.52:0/1759416754 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f0168108730 0x7f016819c7f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:20:52.246 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:20:52.255+0000 7f016d829700 1 --2- 172.21.15.52:0/1759416754 >> [v2:172.21.15.52:6800/3513075805,v1:172.21.15.52:6801/3513075805] conn(0x7f01540b2880 0x7f01540b4d90 unknown :-1 s=CLOSED pgs=117 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:20:52.246 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:20:52.255+0000 7f016d829700 1 --2- 172.21.15.52:0/1759416754 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f01681034b0 0x7f016819bd50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:20:52.246 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:20:52.255+0000 7f016d829700 1 --2- 172.21.15.52:0/1759416754 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f0168107de0 0x7f016819c290 unknown :-1 s=CLOSED pgs=153 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:20:52.247 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:20:52.255+0000 7f016d829700 1 -- 172.21.15.52:0/1759416754 >> 172.21.15.52:0/1759416754 conn(0x7f01680fede0 msgr2=0x7f0168109690 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:20:52.247 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:20:52.256+0000 7f016d829700 1 -- 172.21.15.52:0/1759416754 shutdown_connections 2024-05-27T07:20:52.247 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:20:52.256+0000 7f016d829700 1 -- 172.21.15.52:0/1759416754 wait complete. 2024-05-27T07:20:52.940 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:20:52 smithi105 ceph-mon[35977]: pgmap v253: 1 pgs: 1 active+clean; 577 KiB data, 354 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-27T07:20:52.940 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:20:52 smithi105 ceph-mon[35977]: from='client.? 172.21.15.52:0/1759416754' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-05-27T07:20:52.955 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:20:52 smithi052 ceph-mon[31955]: pgmap v253: 1 pgs: 1 active+clean; 577 KiB data, 354 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-27T07:20:52.955 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:20:52 smithi052 ceph-mon[31955]: from='client.? 172.21.15.52:0/1759416754' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-05-27T07:20:52.986 INFO:tasks.cephadm.ceph_manager.ceph:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-05-27T07:16:57.644034+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-05-27T07:20:52.987 DEBUG:teuthology.orchestra.run.smithi052:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:38f9d900c83c63799fbdbe61acc9a11b0d3554a6 shell --fsid 73df2636-1bf8-11ef-bc9b-c7b262605968 -- ceph osd pool get .mgr pg_num 2024-05-27T07:20:53.102 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:20:52 smithi173 ceph-mon[37000]: pgmap v253: 1 pgs: 1 active+clean; 577 KiB data, 354 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-27T07:20:53.102 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:20:52 smithi173 ceph-mon[37000]: from='client.? 172.21.15.52:0/1759416754' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-05-27T07:20:53.235 INFO:teuthology.orchestra.run.smithi052.stderr:Inferring config /var/lib/ceph/73df2636-1bf8-11ef-bc9b-c7b262605968/mon.a/config 2024-05-27T07:20:54.711 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:20:54 smithi052 ceph-mon[31955]: pgmap v254: 1 pgs: 1 active+clean; 577 KiB data, 354 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-27T07:20:54.940 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:20:54 smithi105 ceph-mon[35977]: pgmap v254: 1 pgs: 1 active+clean; 577 KiB data, 354 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-27T07:20:54.965 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:20:54.974+0000 7f21c6236700 1 -- 172.21.15.52:0/3319608027 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f21c01034b0 msgr2=0x7f21c01038c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:20:54.965 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:20:54.974+0000 7f21c6236700 1 --2- 172.21.15.52:0/3319608027 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f21c01034b0 0x7f21c01038c0 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7f21b000b0d0 tx=0x7f21b000b3e0 comp rx=0 tx=0).stop 2024-05-27T07:20:54.965 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:20:54.974+0000 7f21c6236700 1 -- 172.21.15.52:0/3319608027 shutdown_connections 2024-05-27T07:20:54.965 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:20:54.974+0000 7f21c6236700 1 --2- 172.21.15.52:0/3319608027 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f21c0108730 0x7f21c0110090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:20:54.965 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:20:54.974+0000 7f21c6236700 1 --2- 172.21.15.52:0/3319608027 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f21c01034b0 0x7f21c01038c0 unknown :-1 s=CLOSED pgs=154 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:20:54.965 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:20:54.974+0000 7f21c6236700 1 --2- 172.21.15.52:0/3319608027 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f21c0107de0 0x7f21c01081f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:20:54.965 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:20:54.974+0000 7f21c6236700 1 -- 172.21.15.52:0/3319608027 >> 172.21.15.52:0/3319608027 conn(0x7f21c00fede0 msgr2=0x7f21c0101200 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:20:54.965 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:20:54.974+0000 7f21c6236700 1 -- 172.21.15.52:0/3319608027 shutdown_connections 2024-05-27T07:20:54.965 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:20:54.975+0000 7f21c6236700 1 -- 172.21.15.52:0/3319608027 wait complete. 2024-05-27T07:20:54.967 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:20:54.975+0000 7f21c6236700 1 Processor -- start 2024-05-27T07:20:54.967 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:20:54.975+0000 7f21c6236700 1 -- start start 2024-05-27T07:20:54.967 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:20:54.976+0000 7f21c6236700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f21c01034b0 0x7f21c00ff2c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:20:54.967 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:20:54.976+0000 7f21c6236700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f21c0107de0 0x7f21c00ff800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:20:54.967 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:20:54.976+0000 7f21c6236700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f21c0108730 0x7f21c00ffd40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:20:54.967 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:20:54.976+0000 7f21c6236700 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x7f21c0113360 con 0x7f21c01034b0 2024-05-27T07:20:54.967 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:20:54.976+0000 7f21c6236700 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f21c01131e0 con 0x7f21c0107de0 2024-05-27T07:20:54.967 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:20:54.976+0000 7f21c6236700 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f21c01134e0 con 0x7f21c0108730 2024-05-27T07:20:54.968 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:20:54.976+0000 7f21bf7fe700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f21c01034b0 0x7f21c00ff2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:20:54.968 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:20:54.976+0000 7f21bf7fe700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f21c01034b0 0x7f21c00ff2c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.52:47012/0 (socket says 172.21.15.52:47012) 2024-05-27T07:20:54.968 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:20:54.976+0000 7f21bf7fe700 1 -- 172.21.15.52:0/625654154 learned_addr learned my addr 172.21.15.52:0/625654154 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-05-27T07:20:54.968 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:20:54.976+0000 7f21beffd700 1 --2- 172.21.15.52:0/625654154 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f21c0107de0 0x7f21c00ff800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:20:54.968 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:20:54.976+0000 7f21bffff700 1 --2- 172.21.15.52:0/625654154 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f21c0108730 0x7f21c00ffd40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:20:54.968 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:20:54.976+0000 7f21bf7fe700 1 -- 172.21.15.52:0/625654154 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f21c0108730 msgr2=0x7f21c00ffd40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:20:54.968 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:20:54.976+0000 7f21bf7fe700 1 --2- 172.21.15.52:0/625654154 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f21c0108730 0x7f21c00ffd40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:20:54.968 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:20:54.976+0000 7f21bf7fe700 1 -- 172.21.15.52:0/625654154 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f21c0107de0 msgr2=0x7f21c00ff800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:20:54.968 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:20:54.976+0000 7f21bf7fe700 1 --2- 172.21.15.52:0/625654154 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f21c0107de0 0x7f21c00ff800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:20:54.968 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:20:54.976+0000 7f21bf7fe700 1 -- 172.21.15.52:0/625654154 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f21c01a91c0 con 0x7f21c01034b0 2024-05-27T07:20:54.968 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:20:54.976+0000 7f21bffff700 1 --2- 172.21.15.52:0/625654154 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f21c0108730 0x7f21c00ffd40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-27T07:20:54.968 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:20:54.976+0000 7f21bf7fe700 1 --2- 172.21.15.52:0/625654154 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f21c01034b0 0x7f21c00ff2c0 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7f21b000d010 tx=0x7f21b0007a20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:20:54.968 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:20:54.977+0000 7f21bcff9700 1 -- 172.21.15.52:0/625654154 <== mon.0 v2:172.21.15.52:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f21b001f070 con 0x7f21c01034b0 2024-05-27T07:20:54.968 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:20:54.977+0000 7f21bcff9700 1 -- 172.21.15.52:0/625654154 <== mon.0 v2:172.21.15.52:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f21b0003c00 con 0x7f21c01034b0 2024-05-27T07:20:54.968 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:20:54.977+0000 7f21bcff9700 1 -- 172.21.15.52:0/625654154 <== mon.0 v2:172.21.15.52:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f21b001b6a0 con 0x7f21c01034b0 2024-05-27T07:20:54.969 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:20:54.977+0000 7f21c6236700 1 -- 172.21.15.52:0/625654154 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f21c01a9450 con 0x7f21c01034b0 2024-05-27T07:20:54.969 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:20:54.977+0000 7f21c6236700 1 -- 172.21.15.52:0/625654154 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f21c01a9900 con 0x7f21c01034b0 2024-05-27T07:20:54.969 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:20:54.978+0000 7f21c6236700 1 -- 172.21.15.52:0/625654154 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f21c00512b0 con 0x7f21c01034b0 2024-05-27T07:20:54.969 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:20:54.979+0000 7f21bcff9700 1 -- 172.21.15.52:0/625654154 <== mon.0 v2:172.21.15.52:3300/0 4 ==== mgrmap(e 16) ==== 149652+0+0 (secure 0 0 0) 0x7f21b0022070 con 0x7f21c01034b0 2024-05-27T07:20:54.970 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:20:54.980+0000 7f21bcff9700 1 --2- 172.21.15.52:0/625654154 >> [v2:172.21.15.52:6800/3513075805,v1:172.21.15.52:6801/3513075805] conn(0x7f21ac0b2900 0x7f21ac0b4e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:20:54.970 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:20:54.980+0000 7f21beffd700 1 --2- 172.21.15.52:0/625654154 >> [v2:172.21.15.52:6800/3513075805,v1:172.21.15.52:6801/3513075805] conn(0x7f21ac0b2900 0x7f21ac0b4e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:20:54.970 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:20:54.980+0000 7f21bcff9700 1 -- 172.21.15.52:0/625654154 <== mon.0 v2:172.21.15.52:3300/0 5 ==== osd_map(72..72 src has 1..72) ==== 7127+0+0 (secure 0 0 0) 0x7f21b000f020 con 0x7f21c01034b0 2024-05-27T07:20:54.971 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:20:54.981+0000 7f21beffd700 1 --2- 172.21.15.52:0/625654154 >> [v2:172.21.15.52:6800/3513075805,v1:172.21.15.52:6801/3513075805] conn(0x7f21ac0b2900 0x7f21ac0b4e10 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f21c0102280 tx=0x7f21a800a300 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:20:54.975 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:20:54.985+0000 7f21bcff9700 1 -- 172.21.15.52:0/625654154 <== mon.0 v2:172.21.15.52:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191963 (secure 0 0 0) 0x7f21b0029020 con 0x7f21c01034b0 2024-05-27T07:20:55.102 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:20:54 smithi173 ceph-mon[37000]: pgmap v254: 1 pgs: 1 active+clean; 577 KiB data, 354 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-27T07:20:55.187 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:20:55.197+0000 7f21c6236700 1 -- 172.21.15.52:0/625654154 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f21c002e5e0 con 0x7f21c01034b0 2024-05-27T07:20:55.188 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:20:55.198+0000 7f21bcff9700 1 -- 172.21.15.52:0/625654154 <== mon.0 v2:172.21.15.52:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v72) ==== 93+0+10 (secure 0 0 0) 0x7f21b00a6020 con 0x7f21c01034b0 2024-05-27T07:20:55.188 INFO:teuthology.orchestra.run.smithi052.stdout:pg_num: 1 2024-05-27T07:20:55.190 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:20:55.200+0000 7f21c6236700 1 -- 172.21.15.52:0/625654154 >> [v2:172.21.15.52:6800/3513075805,v1:172.21.15.52:6801/3513075805] conn(0x7f21ac0b2900 msgr2=0x7f21ac0b4e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:20:55.190 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:20:55.200+0000 7f21c6236700 1 --2- 172.21.15.52:0/625654154 >> [v2:172.21.15.52:6800/3513075805,v1:172.21.15.52:6801/3513075805] conn(0x7f21ac0b2900 0x7f21ac0b4e10 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f21c0102280 tx=0x7f21a800a300 comp rx=0 tx=0).stop 2024-05-27T07:20:55.190 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:20:55.200+0000 7f21c6236700 1 -- 172.21.15.52:0/625654154 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f21c01034b0 msgr2=0x7f21c00ff2c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:20:55.190 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:20:55.200+0000 7f21c6236700 1 --2- 172.21.15.52:0/625654154 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f21c01034b0 0x7f21c00ff2c0 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7f21b000d010 tx=0x7f21b0007a20 comp rx=0 tx=0).stop 2024-05-27T07:20:55.191 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:20:55.200+0000 7f21c6236700 1 -- 172.21.15.52:0/625654154 shutdown_connections 2024-05-27T07:20:55.191 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:20:55.200+0000 7f21c6236700 1 --2- 172.21.15.52:0/625654154 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f21c0108730 0x7f21c00ffd40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:20:55.191 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:20:55.200+0000 7f21c6236700 1 --2- 172.21.15.52:0/625654154 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f21c01034b0 0x7f21c00ff2c0 unknown :-1 s=CLOSED pgs=155 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:20:55.191 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:20:55.200+0000 7f21c6236700 1 --2- 172.21.15.52:0/625654154 >> [v2:172.21.15.52:6800/3513075805,v1:172.21.15.52:6801/3513075805] conn(0x7f21ac0b2900 0x7f21ac0b4e10 unknown :-1 s=CLOSED pgs=118 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:20:55.191 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:20:55.200+0000 7f21c6236700 1 --2- 172.21.15.52:0/625654154 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f21c0107de0 0x7f21c00ff800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:20:55.191 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:20:55.200+0000 7f21c6236700 1 -- 172.21.15.52:0/625654154 >> 172.21.15.52:0/625654154 conn(0x7f21c00fede0 msgr2=0x7f21c0106390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:20:55.191 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:20:55.200+0000 7f21c6236700 1 -- 172.21.15.52:0/625654154 shutdown_connections 2024-05-27T07:20:55.191 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:20:55.200+0000 7f21c6236700 1 -- 172.21.15.52:0/625654154 wait complete. 2024-05-27T07:20:55.883 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:20:55 smithi052 ceph-mon[31955]: from='client.? 172.21.15.52:0/625654154' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-05-27T07:20:55.921 INFO:tasks.cephadm:Adding mds.a on smithi052 2024-05-27T07:20:55.922 INFO:tasks.cephadm:Adding mds.d on smithi052 2024-05-27T07:20:55.922 INFO:tasks.cephadm:Adding mds.g on smithi052 2024-05-27T07:20:55.922 INFO:tasks.cephadm:Adding mds.j on smithi052 2024-05-27T07:20:55.922 INFO:tasks.cephadm:Adding mds.b on smithi105 2024-05-27T07:20:55.922 INFO:tasks.cephadm:Adding mds.e on smithi105 2024-05-27T07:20:55.922 INFO:tasks.cephadm:Adding mds.h on smithi105 2024-05-27T07:20:55.923 INFO:tasks.cephadm:Adding mds.k on smithi105 2024-05-27T07:20:55.923 INFO:tasks.cephadm:Adding mds.c on smithi173 2024-05-27T07:20:55.923 INFO:tasks.cephadm:Adding mds.f on smithi173 2024-05-27T07:20:55.923 INFO:tasks.cephadm:Adding mds.i on smithi173 2024-05-27T07:20:55.923 INFO:tasks.cephadm:Adding mds.l on smithi173 2024-05-27T07:20:55.923 DEBUG:teuthology.orchestra.run.smithi173:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:38f9d900c83c63799fbdbe61acc9a11b0d3554a6 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 73df2636-1bf8-11ef-bc9b-c7b262605968 -- ceph orch apply mds all '12;smithi052=a;smithi052=d;smithi052=g;smithi052=j;smithi105=b;smithi105=e;smithi105=h;smithi105=k;smithi173=c;smithi173=f;smithi173=i;smithi173=l' 2024-05-27T07:20:55.940 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:20:55 smithi105 ceph-mon[35977]: from='client.? 172.21.15.52:0/625654154' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-05-27T07:20:55.950 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:20:55 smithi173 ceph-mon[37000]: from='client.? 172.21.15.52:0/625654154' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-05-27T07:20:56.174 INFO:teuthology.orchestra.run.smithi173.stderr:Inferring config /var/lib/ceph/73df2636-1bf8-11ef-bc9b-c7b262605968/mon.c/config 2024-05-27T07:20:56.852 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:20:56 smithi173 ceph-mon[37000]: pgmap v255: 1 pgs: 1 active+clean; 577 KiB data, 354 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-27T07:20:56.940 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:20:56 smithi105 ceph-mon[35977]: pgmap v255: 1 pgs: 1 active+clean; 577 KiB data, 354 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-27T07:20:56.979 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:20:56 smithi052 ceph-mon[31955]: pgmap v255: 1 pgs: 1 active+clean; 577 KiB data, 354 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-27T07:20:57.868 INFO:teuthology.orchestra.run.smithi173.stderr:2024-05-27T07:20:57.878+0000 7f6656deb700 1 -- 172.21.15.173:0/153795573 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f66501059a0 msgr2=0x7f6650105db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:20:57.869 INFO:teuthology.orchestra.run.smithi173.stderr:2024-05-27T07:20:57.878+0000 7f6656deb700 1 --2- 172.21.15.173:0/153795573 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f66501059a0 0x7f6650105db0 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f664c00b0a0 tx=0x7f664c00b3b0 comp rx=0 tx=0).stop 2024-05-27T07:20:57.869 INFO:teuthology.orchestra.run.smithi173.stderr:2024-05-27T07:20:57.878+0000 7f6656deb700 1 -- 172.21.15.173:0/153795573 shutdown_connections 2024-05-27T07:20:57.869 INFO:teuthology.orchestra.run.smithi173.stderr:2024-05-27T07:20:57.878+0000 7f6656deb700 1 --2- 172.21.15.173:0/153795573 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f66501049d0 0x7f665010c6e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:20:57.869 INFO:teuthology.orchestra.run.smithi173.stderr:2024-05-27T07:20:57.878+0000 7f6656deb700 1 --2- 172.21.15.173:0/153795573 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f66501059a0 0x7f6650105db0 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:20:57.869 INFO:teuthology.orchestra.run.smithi173.stderr:2024-05-27T07:20:57.878+0000 7f6656deb700 1 --2- 172.21.15.173:0/153795573 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f6650103f50 0x7f6650104360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:20:57.869 INFO:teuthology.orchestra.run.smithi173.stderr:2024-05-27T07:20:57.878+0000 7f6656deb700 1 -- 172.21.15.173:0/153795573 >> 172.21.15.173:0/153795573 conn(0x7f66500ff9c0 msgr2=0x7f6650101e50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:20:57.869 INFO:teuthology.orchestra.run.smithi173.stderr:2024-05-27T07:20:57.878+0000 7f6656deb700 1 -- 172.21.15.173:0/153795573 shutdown_connections 2024-05-27T07:20:57.869 INFO:teuthology.orchestra.run.smithi173.stderr:2024-05-27T07:20:57.878+0000 7f6656deb700 1 -- 172.21.15.173:0/153795573 wait complete. 2024-05-27T07:20:57.871 INFO:teuthology.orchestra.run.smithi173.stderr:2024-05-27T07:20:57.879+0000 7f6656deb700 1 Processor -- start 2024-05-27T07:20:57.871 INFO:teuthology.orchestra.run.smithi173.stderr:2024-05-27T07:20:57.879+0000 7f6656deb700 1 -- start start 2024-05-27T07:20:57.871 INFO:teuthology.orchestra.run.smithi173.stderr:2024-05-27T07:20:57.879+0000 7f6656deb700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f6650103f50 0x7f6650078560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:20:57.871 INFO:teuthology.orchestra.run.smithi173.stderr:2024-05-27T07:20:57.879+0000 7f6656deb700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f66501049d0 0x7f6650076c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:20:57.871 INFO:teuthology.orchestra.run.smithi173.stderr:2024-05-27T07:20:57.879+0000 7f6656deb700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f66501059a0 0x7f6650077150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:20:57.871 INFO:teuthology.orchestra.run.smithi173.stderr:2024-05-27T07:20:57.879+0000 7f6656deb700 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x7f665010fec0 con 0x7f66501049d0 2024-05-27T07:20:57.871 INFO:teuthology.orchestra.run.smithi173.stderr:2024-05-27T07:20:57.879+0000 7f6656deb700 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f665010fd40 con 0x7f66501059a0 2024-05-27T07:20:57.872 INFO:teuthology.orchestra.run.smithi173.stderr:2024-05-27T07:20:57.879+0000 7f6656deb700 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f6650110040 con 0x7f6650103f50 2024-05-27T07:20:57.872 INFO:teuthology.orchestra.run.smithi173.stderr:2024-05-27T07:20:57.879+0000 7f6654b87700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f6650103f50 0x7f6650078560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:20:57.872 INFO:teuthology.orchestra.run.smithi173.stderr:2024-05-27T07:20:57.879+0000 7f6654b87700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f6650103f50 0x7f6650078560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.173:3300/0 says I am v2:172.21.15.173:45482/0 (socket says 172.21.15.173:45482) 2024-05-27T07:20:57.872 INFO:teuthology.orchestra.run.smithi173.stderr:2024-05-27T07:20:57.879+0000 7f6654b87700 1 -- 172.21.15.173:0/3722081862 learned_addr learned my addr 172.21.15.173:0/3722081862 (peer_addr_for_me v2:172.21.15.173:0/0) 2024-05-27T07:20:57.872 INFO:teuthology.orchestra.run.smithi173.stderr:2024-05-27T07:20:57.879+0000 7f6647fff700 1 --2- 172.21.15.173:0/3722081862 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f66501049d0 0x7f6650076c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:20:57.872 INFO:teuthology.orchestra.run.smithi173.stderr:2024-05-27T07:20:57.879+0000 7f6655388700 1 --2- 172.21.15.173:0/3722081862 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f66501059a0 0x7f6650077150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:20:57.872 INFO:teuthology.orchestra.run.smithi173.stderr:2024-05-27T07:20:57.879+0000 7f6654b87700 1 -- 172.21.15.173:0/3722081862 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f66501059a0 msgr2=0x7f6650077150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:20:57.872 INFO:teuthology.orchestra.run.smithi173.stderr:2024-05-27T07:20:57.879+0000 7f6654b87700 1 --2- 172.21.15.173:0/3722081862 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f66501059a0 0x7f6650077150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:20:57.872 INFO:teuthology.orchestra.run.smithi173.stderr:2024-05-27T07:20:57.879+0000 7f6654b87700 1 -- 172.21.15.173:0/3722081862 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f66501049d0 msgr2=0x7f6650076c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:20:57.872 INFO:teuthology.orchestra.run.smithi173.stderr:2024-05-27T07:20:57.879+0000 7f6654b87700 1 --2- 172.21.15.173:0/3722081862 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f66501049d0 0x7f6650076c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:20:57.872 INFO:teuthology.orchestra.run.smithi173.stderr:2024-05-27T07:20:57.880+0000 7f6654b87700 1 -- 172.21.15.173:0/3722081862 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f665019e8d0 con 0x7f6650103f50 2024-05-27T07:20:57.872 INFO:teuthology.orchestra.run.smithi173.stderr:2024-05-27T07:20:57.880+0000 7f6654b87700 1 --2- 172.21.15.173:0/3722081862 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f6650103f50 0x7f6650078560 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f664c000f80 tx=0x7f664c004e40 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:20:57.873 INFO:teuthology.orchestra.run.smithi173.stderr:2024-05-27T07:20:57.880+0000 7f6645ffb700 1 -- 172.21.15.173:0/3722081862 <== mon.1 v2:172.21.15.173:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f664c021070 con 0x7f6650103f50 2024-05-27T07:20:57.873 INFO:teuthology.orchestra.run.smithi173.stderr:2024-05-27T07:20:57.880+0000 7f6645ffb700 1 -- 172.21.15.173:0/3722081862 <== mon.1 v2:172.21.15.173:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f664c007c00 con 0x7f6650103f50 2024-05-27T07:20:57.873 INFO:teuthology.orchestra.run.smithi173.stderr:2024-05-27T07:20:57.880+0000 7f6656deb700 1 -- 172.21.15.173:0/3722081862 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6650077750 con 0x7f6650103f50 2024-05-27T07:20:57.873 INFO:teuthology.orchestra.run.smithi173.stderr:2024-05-27T07:20:57.880+0000 7f6645ffb700 1 -- 172.21.15.173:0/3722081862 <== mon.1 v2:172.21.15.173:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f664c0052a0 con 0x7f6650103f50 2024-05-27T07:20:57.873 INFO:teuthology.orchestra.run.smithi173.stderr:2024-05-27T07:20:57.880+0000 7f6656deb700 1 -- 172.21.15.173:0/3722081862 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f66501a9490 con 0x7f6650103f50 2024-05-27T07:20:57.873 INFO:teuthology.orchestra.run.smithi173.stderr:2024-05-27T07:20:57.881+0000 7f6656deb700 1 -- 172.21.15.173:0/3722081862 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6650110680 con 0x7f6650103f50 2024-05-27T07:20:57.874 INFO:teuthology.orchestra.run.smithi173.stderr:2024-05-27T07:20:57.883+0000 7f6645ffb700 1 -- 172.21.15.173:0/3722081862 <== mon.1 v2:172.21.15.173:3300/0 4 ==== mgrmap(e 16) ==== 149652+0+0 (secure 0 0 0) 0x7f664c024070 con 0x7f6650103f50 2024-05-27T07:20:57.874 INFO:teuthology.orchestra.run.smithi173.stderr:2024-05-27T07:20:57.883+0000 7f6645ffb700 1 --2- 172.21.15.173:0/3722081862 >> [v2:172.21.15.52:6800/3513075805,v1:172.21.15.52:6801/3513075805] conn(0x7f66380b2900 0x7f66380b4e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:20:57.874 INFO:teuthology.orchestra.run.smithi173.stderr:2024-05-27T07:20:57.883+0000 7f6645ffb700 1 -- 172.21.15.173:0/3722081862 <== mon.1 v2:172.21.15.173:3300/0 5 ==== osd_map(72..72 src has 1..72) ==== 7127+0+0 (secure 0 0 0) 0x7f664c00f020 con 0x7f6650103f50 2024-05-27T07:20:57.874 INFO:teuthology.orchestra.run.smithi173.stderr:2024-05-27T07:20:57.883+0000 7f6647fff700 1 --2- 172.21.15.173:0/3722081862 >> [v2:172.21.15.52:6800/3513075805,v1:172.21.15.52:6801/3513075805] conn(0x7f66380b2900 0x7f66380b4e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:20:57.874 INFO:teuthology.orchestra.run.smithi173.stderr:2024-05-27T07:20:57.884+0000 7f6647fff700 1 --2- 172.21.15.173:0/3722081862 >> [v2:172.21.15.52:6800/3513075805,v1:172.21.15.52:6801/3513075805] conn(0x7f66380b2900 0x7f66380b4e10 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7f6640005950 tx=0x7f664000a300 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:20:57.880 INFO:teuthology.orchestra.run.smithi173.stderr:2024-05-27T07:20:57.889+0000 7f6645ffb700 1 -- 172.21.15.173:0/3722081862 <== mon.1 v2:172.21.15.173:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191963 (secure 0 0 0) 0x7f664c02b020 con 0x7f6650103f50 2024-05-27T07:20:58.091 INFO:teuthology.orchestra.run.smithi173.stderr:2024-05-27T07:20:58.100+0000 7f6656deb700 1 -- 172.21.15.173:0/3722081862 --> [v2:172.21.15.52:6800/3513075805,v1:172.21.15.52:6801/3513075805] -- mgr_command(tid 0: {"prefix": "orch apply mds", "fs_name": "all", "placement": "12;smithi052=a;smithi052=d;smithi052=g;smithi052=j;smithi105=b;smithi105=e;smithi105=h;smithi105=k;smithi173=c;smithi173=f;smithi173=i;smithi173=l", "target": ["mon-mgr", ""]}) -- 0x7f6650063bf0 con 0x7f66380b2900 2024-05-27T07:20:58.099 INFO:teuthology.orchestra.run.smithi173.stderr:2024-05-27T07:20:58.109+0000 7f6645ffb700 1 -- 172.21.15.173:0/3722081862 <== mgr.14152 v2:172.21.15.52:6800/3513075805 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+28 (secure 0 0 0) 0x7f6650063bf0 con 0x7f66380b2900 2024-05-27T07:20:58.100 INFO:teuthology.orchestra.run.smithi173.stdout:Scheduled mds.all update... 2024-05-27T07:20:58.102 INFO:teuthology.orchestra.run.smithi173.stderr:2024-05-27T07:20:58.112+0000 7f6656deb700 1 -- 172.21.15.173:0/3722081862 >> [v2:172.21.15.52:6800/3513075805,v1:172.21.15.52:6801/3513075805] conn(0x7f66380b2900 msgr2=0x7f66380b4e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:20:58.102 INFO:teuthology.orchestra.run.smithi173.stderr:2024-05-27T07:20:58.112+0000 7f6656deb700 1 --2- 172.21.15.173:0/3722081862 >> [v2:172.21.15.52:6800/3513075805,v1:172.21.15.52:6801/3513075805] conn(0x7f66380b2900 0x7f66380b4e10 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7f6640005950 tx=0x7f664000a300 comp rx=0 tx=0).stop 2024-05-27T07:20:58.102 INFO:teuthology.orchestra.run.smithi173.stderr:2024-05-27T07:20:58.112+0000 7f6656deb700 1 -- 172.21.15.173:0/3722081862 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f6650103f50 msgr2=0x7f6650078560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:20:58.103 INFO:teuthology.orchestra.run.smithi173.stderr:2024-05-27T07:20:58.112+0000 7f6656deb700 1 --2- 172.21.15.173:0/3722081862 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f6650103f50 0x7f6650078560 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f664c000f80 tx=0x7f664c004e40 comp rx=0 tx=0).stop 2024-05-27T07:20:58.103 INFO:teuthology.orchestra.run.smithi173.stderr:2024-05-27T07:20:58.112+0000 7f6656deb700 1 -- 172.21.15.173:0/3722081862 shutdown_connections 2024-05-27T07:20:58.103 INFO:teuthology.orchestra.run.smithi173.stderr:2024-05-27T07:20:58.112+0000 7f6656deb700 1 --2- 172.21.15.173:0/3722081862 >> [v2:172.21.15.52:6800/3513075805,v1:172.21.15.52:6801/3513075805] conn(0x7f66380b2900 0x7f66380b4e10 unknown :-1 s=CLOSED pgs=119 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:20:58.103 INFO:teuthology.orchestra.run.smithi173.stderr:2024-05-27T07:20:58.112+0000 7f6656deb700 1 --2- 172.21.15.173:0/3722081862 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f66501059a0 0x7f6650077150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:20:58.103 INFO:teuthology.orchestra.run.smithi173.stderr:2024-05-27T07:20:58.112+0000 7f6656deb700 1 --2- 172.21.15.173:0/3722081862 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f6650103f50 0x7f6650078560 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:20:58.103 INFO:teuthology.orchestra.run.smithi173.stderr:2024-05-27T07:20:58.112+0000 7f6656deb700 1 --2- 172.21.15.173:0/3722081862 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f66501049d0 0x7f6650076c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:20:58.103 INFO:teuthology.orchestra.run.smithi173.stderr:2024-05-27T07:20:58.112+0000 7f6656deb700 1 -- 172.21.15.173:0/3722081862 >> 172.21.15.173:0/3722081862 conn(0x7f66500ff9c0 msgr2=0x7f66501086c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:20:58.103 INFO:teuthology.orchestra.run.smithi173.stderr:2024-05-27T07:20:58.112+0000 7f6656deb700 1 -- 172.21.15.173:0/3722081862 shutdown_connections 2024-05-27T07:20:58.103 INFO:teuthology.orchestra.run.smithi173.stderr:2024-05-27T07:20:58.112+0000 7f6656deb700 1 -- 172.21.15.173:0/3722081862 wait complete. 2024-05-27T07:20:58.701 DEBUG:teuthology.orchestra.run.smithi052:mds.a> sudo journalctl -f -n 0 -u ceph-73df2636-1bf8-11ef-bc9b-c7b262605968@mds.a.service 2024-05-27T07:20:58.705 DEBUG:teuthology.orchestra.run.smithi052:mds.d> sudo journalctl -f -n 0 -u ceph-73df2636-1bf8-11ef-bc9b-c7b262605968@mds.d.service 2024-05-27T07:20:58.751 DEBUG:teuthology.orchestra.run.smithi052:mds.g> sudo journalctl -f -n 0 -u ceph-73df2636-1bf8-11ef-bc9b-c7b262605968@mds.g.service 2024-05-27T07:20:58.795 DEBUG:teuthology.orchestra.run.smithi052:mds.j> sudo journalctl -f -n 0 -u ceph-73df2636-1bf8-11ef-bc9b-c7b262605968@mds.j.service 2024-05-27T07:20:58.839 DEBUG:teuthology.orchestra.run.smithi105:mds.b> sudo journalctl -f -n 0 -u ceph-73df2636-1bf8-11ef-bc9b-c7b262605968@mds.b.service 2024-05-27T07:20:58.842 DEBUG:teuthology.orchestra.run.smithi105:mds.e> sudo journalctl -f -n 0 -u ceph-73df2636-1bf8-11ef-bc9b-c7b262605968@mds.e.service 2024-05-27T07:20:58.885 DEBUG:teuthology.orchestra.run.smithi105:mds.h> sudo journalctl -f -n 0 -u ceph-73df2636-1bf8-11ef-bc9b-c7b262605968@mds.h.service 2024-05-27T07:20:58.928 DEBUG:teuthology.orchestra.run.smithi105:mds.k> sudo journalctl -f -n 0 -u ceph-73df2636-1bf8-11ef-bc9b-c7b262605968@mds.k.service 2024-05-27T07:20:58.971 DEBUG:teuthology.orchestra.run.smithi173:mds.c> sudo journalctl -f -n 0 -u ceph-73df2636-1bf8-11ef-bc9b-c7b262605968@mds.c.service 2024-05-27T07:20:58.973 DEBUG:teuthology.orchestra.run.smithi173:mds.f> sudo journalctl -f -n 0 -u ceph-73df2636-1bf8-11ef-bc9b-c7b262605968@mds.f.service 2024-05-27T07:20:59.016 DEBUG:teuthology.orchestra.run.smithi173:mds.i> sudo journalctl -f -n 0 -u ceph-73df2636-1bf8-11ef-bc9b-c7b262605968@mds.i.service 2024-05-27T07:20:59.059 DEBUG:teuthology.orchestra.run.smithi173:mds.l> sudo journalctl -f -n 0 -u ceph-73df2636-1bf8-11ef-bc9b-c7b262605968@mds.l.service 2024-05-27T07:20:59.102 INFO:tasks.cephadm:Setting up CephFS filesystem(s)... 2024-05-27T07:20:59.102 DEBUG:teuthology.orchestra.run.smithi052:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:38f9d900c83c63799fbdbe61acc9a11b0d3554a6 shell --fsid 73df2636-1bf8-11ef-bc9b-c7b262605968 -- ceph fs dump --format=json 2024-05-27T07:20:59.190 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:20:59 smithi105 ceph-mon[35977]: pgmap v256: 1 pgs: 1 active+clean; 577 KiB data, 354 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-27T07:20:59.190 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:20:59 smithi105 ceph-mon[35977]: from='client.24421 -' entity='client.admin' cmd=[{"prefix": "orch apply mds", "fs_name": "all", "placement": "12;smithi052=a;smithi052=d;smithi052=g;smithi052=j;smithi105=b;smithi105=e;smithi105=h;smithi105=k;smithi173=c;smithi173=f;smithi173=i;smithi173=l", "target": ["mon-mgr", ""]}]: dispatch 2024-05-27T07:20:59.190 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:20:59 smithi105 ceph-mon[35977]: Saving service mds.all spec with placement smithi052=a;smithi052=d;smithi052=g;smithi052=j;smithi105=b;smithi105=e;smithi105=h;smithi105=k;smithi173=c;smithi173=f;smithi173=i;smithi173=l;count:12 2024-05-27T07:20:59.190 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:20:59 smithi105 ceph-mon[35977]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' 2024-05-27T07:20:59.190 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:20:59 smithi105 ceph-mon[35977]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-27T07:20:59.190 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:20:59 smithi105 ceph-mon[35977]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T07:20:59.191 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:20:59 smithi105 ceph-mon[35977]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-27T07:20:59.191 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:20:59 smithi105 ceph-mon[35977]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' 2024-05-27T07:20:59.191 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:20:59 smithi105 ceph-mon[35977]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.l", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-05-27T07:20:59.191 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:20:59 smithi105 ceph-mon[35977]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.l", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-05-27T07:20:59.191 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:20:59 smithi105 ceph-mon[35977]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T07:20:59.350 INFO:teuthology.orchestra.run.smithi052.stderr:Inferring config /var/lib/ceph/73df2636-1bf8-11ef-bc9b-c7b262605968/mon.a/config 2024-05-27T07:20:59.352 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:20:59 smithi173 ceph-mon[37000]: pgmap v256: 1 pgs: 1 active+clean; 577 KiB data, 354 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-27T07:20:59.352 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:20:59 smithi173 ceph-mon[37000]: from='client.24421 -' entity='client.admin' cmd=[{"prefix": "orch apply mds", "fs_name": "all", "placement": "12;smithi052=a;smithi052=d;smithi052=g;smithi052=j;smithi105=b;smithi105=e;smithi105=h;smithi105=k;smithi173=c;smithi173=f;smithi173=i;smithi173=l", "target": ["mon-mgr", ""]}]: dispatch 2024-05-27T07:20:59.352 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:20:59 smithi173 ceph-mon[37000]: Saving service mds.all spec with placement smithi052=a;smithi052=d;smithi052=g;smithi052=j;smithi105=b;smithi105=e;smithi105=h;smithi105=k;smithi173=c;smithi173=f;smithi173=i;smithi173=l;count:12 2024-05-27T07:20:59.352 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:20:59 smithi173 ceph-mon[37000]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' 2024-05-27T07:20:59.352 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:20:59 smithi173 ceph-mon[37000]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-27T07:20:59.352 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:20:59 smithi173 ceph-mon[37000]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T07:20:59.352 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:20:59 smithi173 ceph-mon[37000]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-27T07:20:59.352 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:20:59 smithi173 ceph-mon[37000]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' 2024-05-27T07:20:59.352 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:20:59 smithi173 ceph-mon[37000]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.l", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-05-27T07:20:59.352 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:20:59 smithi173 ceph-mon[37000]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.l", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-05-27T07:20:59.353 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:20:59 smithi173 ceph-mon[37000]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T07:20:59.473 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:20:59 smithi052 ceph-mon[31955]: pgmap v256: 1 pgs: 1 active+clean; 577 KiB data, 354 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-27T07:20:59.474 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:20:59 smithi052 ceph-mon[31955]: from='client.24421 -' entity='client.admin' cmd=[{"prefix": "orch apply mds", "fs_name": "all", "placement": "12;smithi052=a;smithi052=d;smithi052=g;smithi052=j;smithi105=b;smithi105=e;smithi105=h;smithi105=k;smithi173=c;smithi173=f;smithi173=i;smithi173=l", "target": ["mon-mgr", ""]}]: dispatch 2024-05-27T07:20:59.474 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:20:59 smithi052 ceph-mon[31955]: Saving service mds.all spec with placement smithi052=a;smithi052=d;smithi052=g;smithi052=j;smithi105=b;smithi105=e;smithi105=h;smithi105=k;smithi173=c;smithi173=f;smithi173=i;smithi173=l;count:12 2024-05-27T07:20:59.474 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:20:59 smithi052 ceph-mon[31955]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' 2024-05-27T07:20:59.474 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:20:59 smithi052 ceph-mon[31955]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-27T07:20:59.474 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:20:59 smithi052 ceph-mon[31955]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T07:20:59.474 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:20:59 smithi052 ceph-mon[31955]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-27T07:20:59.474 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:20:59 smithi052 ceph-mon[31955]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' 2024-05-27T07:20:59.474 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:20:59 smithi052 ceph-mon[31955]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.l", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-05-27T07:20:59.474 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:20:59 smithi052 ceph-mon[31955]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.l", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-05-27T07:20:59.474 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:20:59 smithi052 ceph-mon[31955]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T07:21:00.414 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:21:00 smithi173 ceph-mon[37000]: Deploying daemon mds.l on smithi173 2024-05-27T07:21:00.440 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:21:00 smithi105 ceph-mon[35977]: Deploying daemon mds.l on smithi173 2024-05-27T07:21:00.467 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:21:00 smithi052 ceph-mon[31955]: Deploying daemon mds.l on smithi173 2024-05-27T07:21:01.068 INFO:journalctl@ceph.mds.l.smithi173.stdout:May 27 07:21:00 smithi173 systemd[1]: Starting Ceph mds.l for 73df2636-1bf8-11ef-bc9b-c7b262605968... 2024-05-27T07:21:01.075 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:21:01.085+0000 7fafa39de700 1 -- 172.21.15.52:0/3481385628 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7faf9c108f00 msgr2=0x7faf9c109310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:21:01.075 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:21:01.085+0000 7fafa39de700 1 --2- 172.21.15.52:0/3481385628 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7faf9c108f00 0x7faf9c109310 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7faf900099c0 tx=0x7faf90009cd0 comp rx=0 tx=0).stop 2024-05-27T07:21:01.075 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:21:01.085+0000 7fafa39de700 1 -- 172.21.15.52:0/3481385628 shutdown_connections 2024-05-27T07:21:01.075 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:21:01.085+0000 7fafa39de700 1 --2- 172.21.15.52:0/3481385628 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7faf9c109850 0x7faf9c1111b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:21:01.075 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:21:01.085+0000 7fafa39de700 1 --2- 172.21.15.52:0/3481385628 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7faf9c1045d0 0x7faf9c1049e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:21:01.076 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:21:01.085+0000 7fafa39de700 1 --2- 172.21.15.52:0/3481385628 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7faf9c108f00 0x7faf9c109310 unknown :-1 s=CLOSED pgs=156 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:21:01.076 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:21:01.085+0000 7fafa39de700 1 -- 172.21.15.52:0/3481385628 >> 172.21.15.52:0/3481385628 conn(0x7faf9c0ffee0 msgr2=0x7faf9c102330 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:21:01.076 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:21:01.085+0000 7fafa39de700 1 -- 172.21.15.52:0/3481385628 shutdown_connections 2024-05-27T07:21:01.076 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:21:01.085+0000 7fafa39de700 1 -- 172.21.15.52:0/3481385628 wait complete. 2024-05-27T07:21:01.077 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:21:01.086+0000 7fafa39de700 1 Processor -- start 2024-05-27T07:21:01.078 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:21:01.086+0000 7fafa39de700 1 -- start start 2024-05-27T07:21:01.078 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:21:01.086+0000 7fafa39de700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7faf9c1045d0 0x7faf9c19aeb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:21:01.078 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:21:01.086+0000 7fafa39de700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7faf9c108f00 0x7faf9c19b450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:21:01.078 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:21:01.086+0000 7fafa39de700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7faf9c109850 0x7faf9c19f6e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:21:01.078 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:21:01.086+0000 7fafa39de700 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x7faf9c114460 con 0x7faf9c1045d0 2024-05-27T07:21:01.078 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:21:01.086+0000 7fafa39de700 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7faf9c1142e0 con 0x7faf9c108f00 2024-05-27T07:21:01.078 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:21:01.086+0000 7fafa39de700 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7faf9c1145e0 con 0x7faf9c109850 2024-05-27T07:21:01.078 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:21:01.086+0000 7fafa177a700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7faf9c1045d0 0x7faf9c19aeb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:21:01.078 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:21:01.086+0000 7fafa177a700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7faf9c1045d0 0x7faf9c19aeb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.52:54908/0 (socket says 172.21.15.52:54908) 2024-05-27T07:21:01.078 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:21:01.086+0000 7fafa177a700 1 -- 172.21.15.52:0/1329050680 learned_addr learned my addr 172.21.15.52:0/1329050680 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-05-27T07:21:01.078 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:21:01.087+0000 7fafa0f79700 1 --2- 172.21.15.52:0/1329050680 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7faf9c108f00 0x7faf9c19b450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:21:01.079 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:21:01.087+0000 7fafa1f7b700 1 --2- 172.21.15.52:0/1329050680 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7faf9c109850 0x7faf9c19f6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:21:01.079 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:21:01.087+0000 7fafa177a700 1 -- 172.21.15.52:0/1329050680 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7faf9c109850 msgr2=0x7faf9c19f6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:21:01.079 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:21:01.087+0000 7fafa177a700 1 --2- 172.21.15.52:0/1329050680 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7faf9c109850 0x7faf9c19f6e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:21:01.079 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:21:01.087+0000 7fafa177a700 1 -- 172.21.15.52:0/1329050680 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7faf9c108f00 msgr2=0x7faf9c19b450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:21:01.079 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:21:01.087+0000 7fafa177a700 1 --2- 172.21.15.52:0/1329050680 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7faf9c108f00 0x7faf9c19b450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:21:01.079 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:21:01.087+0000 7fafa177a700 1 -- 172.21.15.52:0/1329050680 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7faf9c19fd80 con 0x7faf9c1045d0 2024-05-27T07:21:01.079 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:21:01.087+0000 7fafa0f79700 1 --2- 172.21.15.52:0/1329050680 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7faf9c108f00 0x7faf9c19b450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-27T07:21:01.079 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:21:01.087+0000 7fafa177a700 1 --2- 172.21.15.52:0/1329050680 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7faf9c1045d0 0x7faf9c19aeb0 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7faf9800d950 tx=0x7faf9800dc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:21:01.079 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:21:01.087+0000 7faf8e7fc700 1 -- 172.21.15.52:0/1329050680 <== mon.0 v2:172.21.15.52:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faf98016070 con 0x7faf9c1045d0 2024-05-27T07:21:01.079 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:21:01.087+0000 7faf8e7fc700 1 -- 172.21.15.52:0/1329050680 <== mon.0 v2:172.21.15.52:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7faf9800fd40 con 0x7faf9c1045d0 2024-05-27T07:21:01.079 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:21:01.087+0000 7fafa39de700 1 -- 172.21.15.52:0/1329050680 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7faf9c1a0070 con 0x7faf9c1045d0 2024-05-27T07:21:01.080 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:21:01.087+0000 7faf8e7fc700 1 -- 172.21.15.52:0/1329050680 <== mon.0 v2:172.21.15.52:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faf9800e3b0 con 0x7faf9c1045d0 2024-05-27T07:21:01.080 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:21:01.088+0000 7fafa39de700 1 -- 172.21.15.52:0/1329050680 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7faf9c076af0 con 0x7faf9c1045d0 2024-05-27T07:21:01.080 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:21:01.088+0000 7fafa39de700 1 -- 172.21.15.52:0/1329050680 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7faf9c105a80 con 0x7faf9c1045d0 2024-05-27T07:21:01.080 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:21:01.090+0000 7faf8e7fc700 1 -- 172.21.15.52:0/1329050680 <== mon.0 v2:172.21.15.52:3300/0 4 ==== mgrmap(e 16) ==== 149652+0+0 (secure 0 0 0) 0x7faf98019070 con 0x7faf9c1045d0 2024-05-27T07:21:01.082 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:21:01.091+0000 7faf8e7fc700 1 --2- 172.21.15.52:0/1329050680 >> [v2:172.21.15.52:6800/3513075805,v1:172.21.15.52:6801/3513075805] conn(0x7faf880b2900 0x7faf880b4e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:21:01.082 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:21:01.092+0000 7fafa0f79700 1 --2- 172.21.15.52:0/1329050680 >> [v2:172.21.15.52:6800/3513075805,v1:172.21.15.52:6801/3513075805] conn(0x7faf880b2900 0x7faf880b4e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:21:01.082 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:21:01.092+0000 7faf8e7fc700 1 -- 172.21.15.52:0/1329050680 <== mon.0 v2:172.21.15.52:3300/0 5 ==== osd_map(72..72 src has 1..72) ==== 7127+0+0 (secure 0 0 0) 0x7faf980e1750 con 0x7faf9c1045d0 2024-05-27T07:21:01.082 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:21:01.092+0000 7fafa0f79700 1 --2- 172.21.15.52:0/1329050680 >> [v2:172.21.15.52:6800/3513075805,v1:172.21.15.52:6801/3513075805] conn(0x7faf880b2900 0x7faf880b4e10 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7faf90006010 tx=0x7faf90000bc0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:21:01.085 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:21:01.095+0000 7faf8e7fc700 1 -- 172.21.15.52:0/1329050680 <== mon.0 v2:172.21.15.52:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191963 (secure 0 0 0) 0x7faf980a9910 con 0x7faf9c1045d0 2024-05-27T07:21:01.348 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:21:01 smithi173 ceph-mon[37000]: pgmap v257: 1 pgs: 1 active+clean; 577 KiB data, 354 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-27T07:21:01.348 INFO:journalctl@ceph.mds.l.smithi173.stdout:May 27 07:21:01 smithi173 podman[67414]: 2024-05-27 07:21:01.077972562 +0000 UTC m=+0.111969855 container create 4752e4a243d5c61dcd9047e25c945c56cbb10391dc631051e83ed8e7a39f7d44 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-l, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , GIT_CLEAN=True, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, ceph=True, org.label-schema.build-date=20240521, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=) 2024-05-27T07:21:01.348 INFO:journalctl@ceph.mds.l.smithi173.stdout:May 27 07:21:01 smithi173 podman[67414]: 2024-05-27 07:21:00.982949944 +0000 UTC m=+0.016947237 image pull e36ea4fc8a5807e8d2701370e350a0de3856538797f7fba64cfce7e344bd6799 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6 2024-05-27T07:21:01.349 INFO:journalctl@ceph.mds.l.smithi173.stdout:May 27 07:21:01 smithi173 podman[67414]: 2024-05-27 07:21:01.322902855 +0000 UTC m=+0.356900147 container init 4752e4a243d5c61dcd9047e25c945c56cbb10391dc631051e83ed8e7a39f7d44 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-l, org.label-schema.vendor=CentOS, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, GIT_CLEAN=True, ceph=True, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240521, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux ) 2024-05-27T07:21:01.349 INFO:journalctl@ceph.mds.l.smithi173.stdout:May 27 07:21:01 smithi173 podman[67414]: 2024-05-27 07:21:01.325489739 +0000 UTC m=+0.359487034 container start 4752e4a243d5c61dcd9047e25c945c56cbb10391dc631051e83ed8e7a39f7d44 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-l, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, org.label-schema.build-date=20240521, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, GIT_CLEAN=True, ceph=True, org.label-schema.vendor=CentOS) 2024-05-27T07:21:01.349 INFO:journalctl@ceph.mds.l.smithi173.stdout:May 27 07:21:01 smithi173 ceph-mds[67433]: -- 172.21.15.173:0/3413865302 <== mon.1 v2:172.21.15.173:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x56350b3efa40 con 0x56350b437a80 2024-05-27T07:21:01.355 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:21:01.365+0000 7fafa39de700 1 -- 172.21.15.52:0/1329050680 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7faf9c0512b0 con 0x7faf9c1045d0 2024-05-27T07:21:01.355 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:21:01 smithi052 ceph-mon[31955]: pgmap v257: 1 pgs: 1 active+clean; 577 KiB data, 354 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-27T07:21:01.357 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:21:01.367+0000 7faf8e7fc700 1 -- 172.21.15.52:0/1329050680 <== mon.0 v2:172.21.15.52:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 2 v2) ==== 93+0+1396 (secure 0 0 0) 0x7faf980a9210 con 0x7faf9c1045d0 2024-05-27T07:21:01.357 INFO:teuthology.orchestra.run.smithi052.stdout: 2024-05-27T07:21:01.357 INFO:teuthology.orchestra.run.smithi052.stdout:{"epoch":2,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":24433,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.173:6833/3013589220","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6832","nonce":3013589220},{"type":"v1","addr":"172.21.15.173:6833","nonce":3013589220}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2}],"filesystems":[]} 2024-05-27T07:21:01.360 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:21:01.370+0000 7fafa39de700 1 -- 172.21.15.52:0/1329050680 >> [v2:172.21.15.52:6800/3513075805,v1:172.21.15.52:6801/3513075805] conn(0x7faf880b2900 msgr2=0x7faf880b4e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:21:01.360 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:21:01.370+0000 7fafa39de700 1 --2- 172.21.15.52:0/1329050680 >> [v2:172.21.15.52:6800/3513075805,v1:172.21.15.52:6801/3513075805] conn(0x7faf880b2900 0x7faf880b4e10 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7faf90006010 tx=0x7faf90000bc0 comp rx=0 tx=0).stop 2024-05-27T07:21:01.360 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:21:01.370+0000 7fafa39de700 1 -- 172.21.15.52:0/1329050680 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7faf9c1045d0 msgr2=0x7faf9c19aeb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:21:01.360 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:21:01.370+0000 7fafa39de700 1 --2- 172.21.15.52:0/1329050680 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7faf9c1045d0 0x7faf9c19aeb0 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7faf9800d950 tx=0x7faf9800dc60 comp rx=0 tx=0).stop 2024-05-27T07:21:01.360 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:21:01.370+0000 7fafa39de700 1 -- 172.21.15.52:0/1329050680 shutdown_connections 2024-05-27T07:21:01.360 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:21:01.370+0000 7fafa39de700 1 --2- 172.21.15.52:0/1329050680 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7faf9c109850 0x7faf9c19f6e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:21:01.360 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:21:01.370+0000 7fafa39de700 1 --2- 172.21.15.52:0/1329050680 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7faf9c1045d0 0x7faf9c19aeb0 unknown :-1 s=CLOSED pgs=157 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:21:01.360 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:21:01.370+0000 7fafa39de700 1 --2- 172.21.15.52:0/1329050680 >> [v2:172.21.15.52:6800/3513075805,v1:172.21.15.52:6801/3513075805] conn(0x7faf880b2900 0x7faf880b4e10 unknown :-1 s=CLOSED pgs=120 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:21:01.361 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:21:01.370+0000 7fafa39de700 1 --2- 172.21.15.52:0/1329050680 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7faf9c108f00 0x7faf9c19b450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:21:01.361 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:21:01.370+0000 7fafa39de700 1 -- 172.21.15.52:0/1329050680 >> 172.21.15.52:0/1329050680 conn(0x7faf9c0ffee0 msgr2=0x7faf9c10a7b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:21:01.361 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:21:01.370+0000 7fafa39de700 1 -- 172.21.15.52:0/1329050680 shutdown_connections 2024-05-27T07:21:01.361 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:21:01.370+0000 7fafa39de700 1 -- 172.21.15.52:0/1329050680 wait complete. 2024-05-27T07:21:01.361 INFO:teuthology.orchestra.run.smithi052.stderr:dumped fsmap epoch 2 2024-05-27T07:21:01.440 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:21:01 smithi105 ceph-mon[35977]: pgmap v257: 1 pgs: 1 active+clean; 577 KiB data, 354 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-27T07:21:01.602 INFO:journalctl@ceph.mds.l.smithi173.stdout:May 27 07:21:01 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-l[67429]: starting mds.l at 2024-05-27T07:21:01.602 INFO:journalctl@ceph.mds.l.smithi173.stdout:May 27 07:21:01 smithi173 bash[67414]: 4752e4a243d5c61dcd9047e25c945c56cbb10391dc631051e83ed8e7a39f7d44 2024-05-27T07:21:01.602 INFO:journalctl@ceph.mds.l.smithi173.stdout:May 27 07:21:01 smithi173 systemd[1]: Started Ceph mds.l for 73df2636-1bf8-11ef-bc9b-c7b262605968. 2024-05-27T07:21:02.440 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:21:02 smithi105 ceph-mon[35977]: mds.? [v2:172.21.15.173:6832/3013589220,v1:172.21.15.173:6833/3013589220] up:boot 2024-05-27T07:21:02.441 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:21:02 smithi105 ceph-mon[35977]: fsmap 1 up:standby 2024-05-27T07:21:02.441 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:21:02 smithi105 ceph-mon[35977]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "l"}]: dispatch 2024-05-27T07:21:02.441 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:21:02 smithi105 ceph-mon[35977]: from='client.? 172.21.15.52:0/1329050680' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-27T07:21:02.441 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:21:02 smithi105 ceph-mon[35977]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' 2024-05-27T07:21:02.441 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:21:02 smithi105 ceph-mon[35977]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' 2024-05-27T07:21:02.441 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:21:02 smithi105 ceph-mon[35977]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' 2024-05-27T07:21:02.441 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:21:02 smithi105 ceph-mon[35977]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.k", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-05-27T07:21:02.441 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:21:02 smithi105 ceph-mon[35977]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.k", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-05-27T07:21:02.441 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:21:02 smithi105 ceph-mon[35977]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T07:21:02.441 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:21:02 smithi105 ceph-mon[35977]: Deploying daemon mds.k on smithi105 2024-05-27T07:21:02.441 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:21:02 smithi105 ceph-mon[35977]: pgmap v258: 1 pgs: 1 active+clean; 577 KiB data, 354 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-27T07:21:02.729 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:21:02 smithi052 ceph-mon[31955]: mds.? [v2:172.21.15.173:6832/3013589220,v1:172.21.15.173:6833/3013589220] up:boot 2024-05-27T07:21:02.730 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:21:02 smithi052 ceph-mon[31955]: fsmap 1 up:standby 2024-05-27T07:21:02.730 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:21:02 smithi052 ceph-mon[31955]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "l"}]: dispatch 2024-05-27T07:21:02.730 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:21:02 smithi052 ceph-mon[31955]: from='client.? 172.21.15.52:0/1329050680' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-27T07:21:02.730 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:21:02 smithi052 ceph-mon[31955]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' 2024-05-27T07:21:02.730 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:21:02 smithi052 ceph-mon[31955]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' 2024-05-27T07:21:02.730 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:21:02 smithi052 ceph-mon[31955]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' 2024-05-27T07:21:02.730 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:21:02 smithi052 ceph-mon[31955]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.k", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-05-27T07:21:02.730 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:21:02 smithi052 ceph-mon[31955]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.k", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-05-27T07:21:02.730 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:21:02 smithi052 ceph-mon[31955]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T07:21:02.730 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:21:02 smithi052 ceph-mon[31955]: Deploying daemon mds.k on smithi105 2024-05-27T07:21:02.730 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:21:02 smithi052 ceph-mon[31955]: pgmap v258: 1 pgs: 1 active+clean; 577 KiB data, 354 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-27T07:21:02.852 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:21:02 smithi173 ceph-mon[37000]: mds.? [v2:172.21.15.173:6832/3013589220,v1:172.21.15.173:6833/3013589220] up:boot 2024-05-27T07:21:02.852 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:21:02 smithi173 ceph-mon[37000]: fsmap 1 up:standby 2024-05-27T07:21:02.852 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:21:02 smithi173 ceph-mon[37000]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "l"}]: dispatch 2024-05-27T07:21:02.852 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:21:02 smithi173 ceph-mon[37000]: from='client.? 172.21.15.52:0/1329050680' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-27T07:21:02.853 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:21:02 smithi173 ceph-mon[37000]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' 2024-05-27T07:21:02.853 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:21:02 smithi173 ceph-mon[37000]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' 2024-05-27T07:21:02.853 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:21:02 smithi173 ceph-mon[37000]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' 2024-05-27T07:21:02.853 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:21:02 smithi173 ceph-mon[37000]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.k", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-05-27T07:21:02.853 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:21:02 smithi173 ceph-mon[37000]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.k", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-05-27T07:21:02.853 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:21:02 smithi173 ceph-mon[37000]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T07:21:02.853 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:21:02 smithi173 ceph-mon[37000]: Deploying daemon mds.k on smithi105 2024-05-27T07:21:02.853 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:21:02 smithi173 ceph-mon[37000]: pgmap v258: 1 pgs: 1 active+clean; 577 KiB data, 354 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-27T07:21:03.940 INFO:journalctl@ceph.mds.k.smithi105.stdout:May 27 07:21:03 smithi105 systemd[1]: Starting Ceph mds.k for 73df2636-1bf8-11ef-bc9b-c7b262605968... 2024-05-27T07:21:03.998 DEBUG:teuthology.orchestra.run.smithi052:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:38f9d900c83c63799fbdbe61acc9a11b0d3554a6 shell --fsid 73df2636-1bf8-11ef-bc9b-c7b262605968 -- ceph fs dump --format=json 2024-05-27T07:21:04.253 INFO:journalctl@ceph.mds.k.smithi105.stdout:May 27 07:21:04 smithi105 podman[64599]: 2024-05-27 07:21:03.902456828 +0000 UTC m=+0.017462487 image pull e36ea4fc8a5807e8d2701370e350a0de3856538797f7fba64cfce7e344bd6799 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6 2024-05-27T07:21:04.253 INFO:journalctl@ceph.mds.k.smithi105.stdout:May 27 07:21:04 smithi105 podman[64599]: 2024-05-27 07:21:04.02793807 +0000 UTC m=+0.142943769 container create 5d00409929d19fe1d05590684327698fc49e3239f33e4fc0ffad523c2ae62ab3 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-k, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, org.label-schema.build-date=20240521, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, ceph=True, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-05-27T07:21:04.253 INFO:journalctl@ceph.mds.k.smithi105.stdout:May 27 07:21:04 smithi105 podman[64599]: 2024-05-27 07:21:04.263782508 +0000 UTC m=+0.378788151 container init 5d00409929d19fe1d05590684327698fc49e3239f33e4fc0ffad523c2ae62ab3 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-k, CEPH_POINT_RELEASE=, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240521, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, io.buildah.version=1.33.5) 2024-05-27T07:21:04.260 INFO:teuthology.orchestra.run.smithi052.stderr:Inferring config /var/lib/ceph/73df2636-1bf8-11ef-bc9b-c7b262605968/mon.a/config 2024-05-27T07:21:04.673 INFO:journalctl@ceph.mds.k.smithi105.stdout:May 27 07:21:04 smithi105 podman[64599]: 2024-05-27 07:21:04.266430696 +0000 UTC m=+0.381436345 container start 5d00409929d19fe1d05590684327698fc49e3239f33e4fc0ffad523c2ae62ab3 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-k, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240521, org.label-schema.license=GPLv2, GIT_CLEAN=True, org.label-schema.schema-version=1.0, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, io.buildah.version=1.33.5) 2024-05-27T07:21:04.673 INFO:journalctl@ceph.mds.k.smithi105.stdout:May 27 07:21:04 smithi105 ceph-mds[64618]: -- 172.21.15.105:0/3725254005 <== mon.2 v2:172.21.15.105:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55b021673a40 con 0x55b0216bb180 2024-05-27T07:21:04.673 INFO:journalctl@ceph.mds.k.smithi105.stdout:May 27 07:21:04 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-k[64614]: starting mds.k at 2024-05-27T07:21:04.673 INFO:journalctl@ceph.mds.k.smithi105.stdout:May 27 07:21:04 smithi105 bash[64599]: 5d00409929d19fe1d05590684327698fc49e3239f33e4fc0ffad523c2ae62ab3 2024-05-27T07:21:04.673 INFO:journalctl@ceph.mds.k.smithi105.stdout:May 27 07:21:04 smithi105 systemd[1]: Started Ceph mds.k for 73df2636-1bf8-11ef-bc9b-c7b262605968. 2024-05-27T07:21:04.940 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:21:04 smithi105 ceph-mon[35977]: pgmap v259: 1 pgs: 1 active+clean; 577 KiB data, 354 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-27T07:21:04.940 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:21:04 smithi105 ceph-mon[35977]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' 2024-05-27T07:21:04.940 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:21:04 smithi105 ceph-mon[35977]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' 2024-05-27T07:21:04.940 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:21:04 smithi105 ceph-mon[35977]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' 2024-05-27T07:21:04.941 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:21:04 smithi105 ceph-mon[35977]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.j", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-05-27T07:21:04.941 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:21:04 smithi105 ceph-mon[35977]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.j", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-05-27T07:21:04.941 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:21:04 smithi105 ceph-mon[35977]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T07:21:04.979 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:21:04 smithi052 ceph-mon[31955]: pgmap v259: 1 pgs: 1 active+clean; 577 KiB data, 354 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-27T07:21:04.979 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:21:04 smithi052 ceph-mon[31955]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' 2024-05-27T07:21:04.979 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:21:04 smithi052 ceph-mon[31955]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' 2024-05-27T07:21:04.979 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:21:04 smithi052 ceph-mon[31955]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' 2024-05-27T07:21:04.980 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:21:04 smithi052 ceph-mon[31955]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.j", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-05-27T07:21:04.980 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:21:04 smithi052 ceph-mon[31955]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.j", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-05-27T07:21:04.980 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:21:04 smithi052 ceph-mon[31955]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T07:21:05.102 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:21:04 smithi173 ceph-mon[37000]: pgmap v259: 1 pgs: 1 active+clean; 577 KiB data, 354 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-27T07:21:05.102 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:21:04 smithi173 ceph-mon[37000]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' 2024-05-27T07:21:05.102 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:21:04 smithi173 ceph-mon[37000]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' 2024-05-27T07:21:05.102 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:21:04 smithi173 ceph-mon[37000]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' 2024-05-27T07:21:05.102 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:21:04 smithi173 ceph-mon[37000]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.j", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-05-27T07:21:05.102 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:21:04 smithi173 ceph-mon[37000]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.j", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-05-27T07:21:05.103 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:21:04 smithi173 ceph-mon[37000]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T07:21:05.940 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:21:05 smithi105 ceph-mon[35977]: Deploying daemon mds.j on smithi052 2024-05-27T07:21:05.940 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:21:05 smithi105 ceph-mon[35977]: mds.? [v2:172.21.15.105:6832/2800678412,v1:172.21.15.105:6833/2800678412] up:boot 2024-05-27T07:21:05.940 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:21:05 smithi105 ceph-mon[35977]: fsmap 2 up:standby 2024-05-27T07:21:05.940 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:21:05 smithi105 ceph-mon[35977]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "k"}]: dispatch 2024-05-27T07:21:05.954 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:21:05.964+0000 7f3c2297e700 1 -- 172.21.15.52:0/2849327042 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f3c1c108730 msgr2=0x7f3c1c110090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:21:05.954 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:21:05.964+0000 7f3c2297e700 1 --2- 172.21.15.52:0/2849327042 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f3c1c108730 0x7f3c1c110090 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7f3c1400d360 tx=0x7f3c1400d670 comp rx=0 tx=0).stop 2024-05-27T07:21:05.954 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:21:05.964+0000 7f3c2297e700 1 -- 172.21.15.52:0/2849327042 shutdown_connections 2024-05-27T07:21:05.955 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:21:05.964+0000 7f3c2297e700 1 --2- 172.21.15.52:0/2849327042 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f3c1c108730 0x7f3c1c110090 unknown :-1 s=CLOSED pgs=158 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:21:05.955 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:21:05.964+0000 7f3c2297e700 1 --2- 172.21.15.52:0/2849327042 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f3c1c1034b0 0x7f3c1c1038c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:21:05.955 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:21:05.964+0000 7f3c2297e700 1 --2- 172.21.15.52:0/2849327042 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f3c1c107de0 0x7f3c1c1081f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:21:05.955 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:21:05.964+0000 7f3c2297e700 1 -- 172.21.15.52:0/2849327042 >> 172.21.15.52:0/2849327042 conn(0x7f3c1c0fedb0 msgr2=0x7f3c1c1011d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:21:05.955 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:21:05 smithi052 ceph-mon[31955]: Deploying daemon mds.j on smithi052 2024-05-27T07:21:05.955 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:21:05 smithi052 ceph-mon[31955]: mds.? [v2:172.21.15.105:6832/2800678412,v1:172.21.15.105:6833/2800678412] up:boot 2024-05-27T07:21:05.955 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:21:05 smithi052 ceph-mon[31955]: fsmap 2 up:standby 2024-05-27T07:21:05.955 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:21:05 smithi052 ceph-mon[31955]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "k"}]: dispatch 2024-05-27T07:21:05.957 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:21:05.964+0000 7f3c2297e700 1 -- 172.21.15.52:0/2849327042 shutdown_connections 2024-05-27T07:21:05.957 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:21:05.964+0000 7f3c2297e700 1 -- 172.21.15.52:0/2849327042 wait complete. 2024-05-27T07:21:05.958 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:21:05.965+0000 7f3c2297e700 1 Processor -- start 2024-05-27T07:21:05.958 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:21:05.965+0000 7f3c2297e700 1 -- start start 2024-05-27T07:21:05.958 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:21:05.965+0000 7f3c2297e700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f3c1c1034b0 0x7f3c1c19cf40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:21:05.958 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:21:05.965+0000 7f3c2297e700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f3c1c107de0 0x7f3c1c19d4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:21:05.958 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:21:05.965+0000 7f3c2297e700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f3c1c108730 0x7f3c1c1a1770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:21:05.958 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:21:05.965+0000 7f3c2297e700 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x7f3c1c1122c0 con 0x7f3c1c1034b0 2024-05-27T07:21:05.958 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:21:05.965+0000 7f3c2297e700 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f3c1c112140 con 0x7f3c1c108730 2024-05-27T07:21:05.958 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:21:05.965+0000 7f3c2297e700 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f3c1c112440 con 0x7f3c1c107de0 2024-05-27T07:21:05.958 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:21:05.966+0000 7f3c1bfff700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f3c1c1034b0 0x7f3c1c19cf40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:21:05.958 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:21:05.966+0000 7f3c1bfff700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f3c1c1034b0 0x7f3c1c19cf40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.52:54930/0 (socket says 172.21.15.52:54930) 2024-05-27T07:21:05.958 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:21:05.966+0000 7f3c1bfff700 1 -- 172.21.15.52:0/3359927994 learned_addr learned my addr 172.21.15.52:0/3359927994 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-05-27T07:21:05.959 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:21:05.966+0000 7f3c20f1b700 1 --2- 172.21.15.52:0/3359927994 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f3c1c108730 0x7f3c1c1a1770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:21:05.959 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:21:05.966+0000 7f3c13fff700 1 --2- 172.21.15.52:0/3359927994 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f3c1c107de0 0x7f3c1c19d4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:21:05.959 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:21:05.966+0000 7f3c1bfff700 1 -- 172.21.15.52:0/3359927994 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f3c1c107de0 msgr2=0x7f3c1c19d4e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:21:05.959 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:21:05.966+0000 7f3c1bfff700 1 --2- 172.21.15.52:0/3359927994 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f3c1c107de0 0x7f3c1c19d4e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:21:05.959 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:21:05.966+0000 7f3c1bfff700 1 -- 172.21.15.52:0/3359927994 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f3c1c108730 msgr2=0x7f3c1c1a1770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:21:05.959 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:21:05.966+0000 7f3c1bfff700 1 --2- 172.21.15.52:0/3359927994 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f3c1c108730 0x7f3c1c1a1770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:21:05.959 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:21:05.966+0000 7f3c1bfff700 1 -- 172.21.15.52:0/3359927994 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3c1c1a1e10 con 0x7f3c1c1034b0 2024-05-27T07:21:05.959 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:21:05.966+0000 7f3c1bfff700 1 --2- 172.21.15.52:0/3359927994 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f3c1c1034b0 0x7f3c1c19cf40 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7f3c0c00e9a0 tx=0x7f3c0c00ecb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:21:05.959 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:21:05.966+0000 7f3c19ffb700 1 -- 172.21.15.52:0/3359927994 <== mon.0 v2:172.21.15.52:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3c0c00cbd0 con 0x7f3c1c1034b0 2024-05-27T07:21:05.959 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:21:05.966+0000 7f3c2297e700 1 -- 172.21.15.52:0/3359927994 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3c1c1a2100 con 0x7f3c1c1034b0 2024-05-27T07:21:05.959 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:21:05.966+0000 7f3c19ffb700 1 -- 172.21.15.52:0/3359927994 <== mon.0 v2:172.21.15.52:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f3c0c00cd70 con 0x7f3c1c1034b0 2024-05-27T07:21:05.959 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:21:05.966+0000 7f3c19ffb700 1 -- 172.21.15.52:0/3359927994 <== mon.0 v2:172.21.15.52:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3c0c018680 con 0x7f3c1c1034b0 2024-05-27T07:21:05.959 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:21:05.966+0000 7f3c2297e700 1 -- 172.21.15.52:0/3359927994 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3c1c1a9a20 con 0x7f3c1c1034b0 2024-05-27T07:21:05.960 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:21:05.967+0000 7f3c2297e700 1 -- 172.21.15.52:0/3359927994 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3c1c0512b0 con 0x7f3c1c1034b0 2024-05-27T07:21:05.960 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:21:05.969+0000 7f3c19ffb700 1 -- 172.21.15.52:0/3359927994 <== mon.0 v2:172.21.15.52:3300/0 4 ==== mgrmap(e 16) ==== 149652+0+0 (secure 0 0 0) 0x7f3c0c010430 con 0x7f3c1c1034b0 2024-05-27T07:21:05.960 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:21:05.970+0000 7f3c19ffb700 1 --2- 172.21.15.52:0/3359927994 >> [v2:172.21.15.52:6800/3513075805,v1:172.21.15.52:6801/3513075805] conn(0x7f3c080b2880 0x7f3c080b4d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:21:05.960 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:21:05.970+0000 7f3c13fff700 1 --2- 172.21.15.52:0/3359927994 >> [v2:172.21.15.52:6800/3513075805,v1:172.21.15.52:6801/3513075805] conn(0x7f3c080b2880 0x7f3c080b4d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:21:05.961 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:21:05.970+0000 7f3c19ffb700 1 -- 172.21.15.52:0/3359927994 <== mon.0 v2:172.21.15.52:3300/0 5 ==== osd_map(72..72 src has 1..72) ==== 7127+0+0 (secure 0 0 0) 0x7f3c0c0e2240 con 0x7f3c1c1034b0 2024-05-27T07:21:05.961 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:21:05.970+0000 7f3c13fff700 1 --2- 172.21.15.52:0/3359927994 >> [v2:172.21.15.52:6800/3513075805,v1:172.21.15.52:6801/3513075805] conn(0x7f3c080b2880 0x7f3c080b4d90 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7f3c1c1027e0 tx=0x7f3c04009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:21:05.963 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:21:05.973+0000 7f3c19ffb700 1 -- 172.21.15.52:0/3359927994 <== mon.0 v2:172.21.15.52:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191963 (secure 0 0 0) 0x7f3c0c0aa400 con 0x7f3c1c1034b0 2024-05-27T07:21:06.102 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:21:05 smithi173 ceph-mon[37000]: Deploying daemon mds.j on smithi052 2024-05-27T07:21:06.102 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:21:05 smithi173 ceph-mon[37000]: mds.? [v2:172.21.15.105:6832/2800678412,v1:172.21.15.105:6833/2800678412] up:boot 2024-05-27T07:21:06.102 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:21:05 smithi173 ceph-mon[37000]: fsmap 2 up:standby 2024-05-27T07:21:06.102 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:21:05 smithi173 ceph-mon[37000]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "k"}]: dispatch 2024-05-27T07:21:06.233 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:21:06.243+0000 7f3c2297e700 1 -- 172.21.15.52:0/3359927994 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f3c1c19e2a0 con 0x7f3c1c1034b0 2024-05-27T07:21:06.233 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:21:06.243+0000 7f3c19ffb700 1 -- 172.21.15.52:0/3359927994 <== mon.0 v2:172.21.15.52:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 3 v3) ==== 93+0+2224 (secure 0 0 0) 0x7f3c0c0a9d00 con 0x7f3c1c1034b0 2024-05-27T07:21:06.234 INFO:teuthology.orchestra.run.smithi052.stdout: 2024-05-27T07:21:06.234 INFO:teuthology.orchestra.run.smithi052.stdout:{"epoch":3,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":24433,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.173:6833/3013589220","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6832","nonce":3013589220},{"type":"v1","addr":"172.21.15.173:6833","nonce":3013589220}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24452,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.105:6833/2800678412","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6832","nonce":2800678412},{"type":"v1","addr":"172.21.15.105:6833","nonce":2800678412}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[]} 2024-05-27T07:21:06.236 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:21:06.246+0000 7f3c2297e700 1 -- 172.21.15.52:0/3359927994 >> [v2:172.21.15.52:6800/3513075805,v1:172.21.15.52:6801/3513075805] conn(0x7f3c080b2880 msgr2=0x7f3c080b4d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:21:06.236 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:21:06.246+0000 7f3c2297e700 1 --2- 172.21.15.52:0/3359927994 >> [v2:172.21.15.52:6800/3513075805,v1:172.21.15.52:6801/3513075805] conn(0x7f3c080b2880 0x7f3c080b4d90 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7f3c1c1027e0 tx=0x7f3c04009260 comp rx=0 tx=0).stop 2024-05-27T07:21:06.236 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:21:06.246+0000 7f3c2297e700 1 -- 172.21.15.52:0/3359927994 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f3c1c1034b0 msgr2=0x7f3c1c19cf40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:21:06.236 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:21:06.246+0000 7f3c2297e700 1 --2- 172.21.15.52:0/3359927994 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f3c1c1034b0 0x7f3c1c19cf40 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7f3c0c00e9a0 tx=0x7f3c0c00ecb0 comp rx=0 tx=0).stop 2024-05-27T07:21:06.236 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:21:06.246+0000 7f3c2297e700 1 -- 172.21.15.52:0/3359927994 shutdown_connections 2024-05-27T07:21:06.236 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:21:06.246+0000 7f3c2297e700 1 --2- 172.21.15.52:0/3359927994 >> [v2:172.21.15.52:6800/3513075805,v1:172.21.15.52:6801/3513075805] conn(0x7f3c080b2880 0x7f3c080b4d90 unknown :-1 s=CLOSED pgs=123 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:21:06.236 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:21:06.246+0000 7f3c2297e700 1 --2- 172.21.15.52:0/3359927994 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f3c1c108730 0x7f3c1c1a1770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:21:06.237 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:21:06.246+0000 7f3c2297e700 1 --2- 172.21.15.52:0/3359927994 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f3c1c1034b0 0x7f3c1c19cf40 unknown :-1 s=CLOSED pgs=159 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:21:06.237 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:21:06.246+0000 7f3c2297e700 1 --2- 172.21.15.52:0/3359927994 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f3c1c107de0 0x7f3c1c19d4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:21:06.237 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:21:06.246+0000 7f3c2297e700 1 -- 172.21.15.52:0/3359927994 >> 172.21.15.52:0/3359927994 conn(0x7f3c1c0fedb0 msgr2=0x7f3c1c107550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:21:06.237 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:21:06.246+0000 7f3c2297e700 1 -- 172.21.15.52:0/3359927994 shutdown_connections 2024-05-27T07:21:06.237 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:21:06.246+0000 7f3c2297e700 1 -- 172.21.15.52:0/3359927994 wait complete. 2024-05-27T07:21:06.238 INFO:teuthology.orchestra.run.smithi052.stderr:dumped fsmap epoch 3 2024-05-27T07:21:06.940 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:21:06 smithi105 ceph-mon[35977]: pgmap v260: 1 pgs: 1 active+clean; 577 KiB data, 354 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-27T07:21:06.940 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:21:06 smithi105 ceph-mon[35977]: from='client.? 172.21.15.52:0/3359927994' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-27T07:21:06.979 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:21:06 smithi052 ceph-mon[31955]: pgmap v260: 1 pgs: 1 active+clean; 577 KiB data, 354 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-27T07:21:06.979 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:21:06 smithi052 ceph-mon[31955]: from='client.? 172.21.15.52:0/3359927994' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-27T07:21:07.102 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:21:06 smithi173 ceph-mon[37000]: pgmap v260: 1 pgs: 1 active+clean; 577 KiB data, 354 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-27T07:21:07.102 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:21:06 smithi173 ceph-mon[37000]: from='client.? 172.21.15.52:0/3359927994' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-27T07:21:07.729 INFO:journalctl@ceph.mds.j.smithi052.stdout:May 27 07:21:07 smithi052 systemd[1]: Starting Ceph mds.j for 73df2636-1bf8-11ef-bc9b-c7b262605968... 2024-05-27T07:21:08.144 INFO:journalctl@ceph.mds.j.smithi052.stdout:May 27 07:21:08 smithi052 podman[72706]: 2024-05-27 07:21:07.906376979 +0000 UTC m=+0.017711717 image pull e36ea4fc8a5807e8d2701370e350a0de3856538797f7fba64cfce7e344bd6799 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6 2024-05-27T07:21:08.144 INFO:journalctl@ceph.mds.j.smithi052.stdout:May 27 07:21:08 smithi052 podman[72706]: 2024-05-27 07:21:08.02221713 +0000 UTC m=+0.133551848 container create 107b05d83164e934771756afdb36f84df595611dba00940359c933ea5ddee931 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-j, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, org.label-schema.build-date=20240521, CEPH_POINT_RELEASE=, GIT_CLEAN=True, ceph=True) 2024-05-27T07:21:08.693 INFO:journalctl@ceph.mds.j.smithi052.stdout:May 27 07:21:08 smithi052 podman[72706]: 2024-05-27 07:21:08.275408436 +0000 UTC m=+0.386743169 container init 107b05d83164e934771756afdb36f84df595611dba00940359c933ea5ddee931 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-j, org.label-schema.license=GPLv2, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, GIT_CLEAN=True, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, org.label-schema.build-date=20240521, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image) 2024-05-27T07:21:08.693 INFO:journalctl@ceph.mds.j.smithi052.stdout:May 27 07:21:08 smithi052 podman[72706]: 2024-05-27 07:21:08.278502658 +0000 UTC m=+0.389837384 container start 107b05d83164e934771756afdb36f84df595611dba00940359c933ea5ddee931 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-j, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240521, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, ceph=True, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90) 2024-05-27T07:21:08.693 INFO:journalctl@ceph.mds.j.smithi052.stdout:May 27 07:21:08 smithi052 ceph-mds[72725]: -- 172.21.15.52:0/2500967417 <== mon.0 v2:172.21.15.52:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55b5c929fa40 con 0x55b5c92e7180 2024-05-27T07:21:08.693 INFO:journalctl@ceph.mds.j.smithi052.stdout:May 27 07:21:08 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-j[72721]: starting mds.j at 2024-05-27T07:21:08.693 INFO:journalctl@ceph.mds.j.smithi052.stdout:May 27 07:21:08 smithi052 bash[72706]: 107b05d83164e934771756afdb36f84df595611dba00940359c933ea5ddee931 2024-05-27T07:21:08.694 INFO:journalctl@ceph.mds.j.smithi052.stdout:May 27 07:21:08 smithi052 systemd[1]: Started Ceph mds.j for 73df2636-1bf8-11ef-bc9b-c7b262605968. 2024-05-27T07:21:08.979 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:21:08 smithi052 ceph-mon[31955]: pgmap v261: 1 pgs: 1 active+clean; 577 KiB data, 354 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-27T07:21:08.979 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:21:08 smithi052 ceph-mon[31955]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' 2024-05-27T07:21:08.979 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:21:08 smithi052 ceph-mon[31955]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' 2024-05-27T07:21:08.980 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:21:08 smithi052 ceph-mon[31955]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' 2024-05-27T07:21:08.980 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:21:08 smithi052 ceph-mon[31955]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.d", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-05-27T07:21:08.980 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:21:08 smithi052 ceph-mon[31955]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.d", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-05-27T07:21:08.980 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:21:08 smithi052 ceph-mon[31955]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T07:21:09.102 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:21:08 smithi173 ceph-mon[37000]: pgmap v261: 1 pgs: 1 active+clean; 577 KiB data, 354 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-27T07:21:09.102 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:21:08 smithi173 ceph-mon[37000]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' 2024-05-27T07:21:09.102 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:21:08 smithi173 ceph-mon[37000]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' 2024-05-27T07:21:09.103 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:21:08 smithi173 ceph-mon[37000]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' 2024-05-27T07:21:09.103 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:21:08 smithi173 ceph-mon[37000]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.d", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-05-27T07:21:09.103 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:21:08 smithi173 ceph-mon[37000]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.d", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-05-27T07:21:09.103 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:21:08 smithi173 ceph-mon[37000]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T07:21:09.190 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:21:08 smithi105 ceph-mon[35977]: pgmap v261: 1 pgs: 1 active+clean; 577 KiB data, 354 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-27T07:21:09.190 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:21:08 smithi105 ceph-mon[35977]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' 2024-05-27T07:21:09.191 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:21:08 smithi105 ceph-mon[35977]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' 2024-05-27T07:21:09.191 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:21:08 smithi105 ceph-mon[35977]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' 2024-05-27T07:21:09.191 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:21:08 smithi105 ceph-mon[35977]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.d", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-05-27T07:21:09.191 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:21:08 smithi105 ceph-mon[35977]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.d", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-05-27T07:21:09.191 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:21:08 smithi105 ceph-mon[35977]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T07:21:09.818 DEBUG:teuthology.orchestra.run.smithi052:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:38f9d900c83c63799fbdbe61acc9a11b0d3554a6 shell --fsid 73df2636-1bf8-11ef-bc9b-c7b262605968 -- ceph fs dump --format=json 2024-05-27T07:21:09.940 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:21:09 smithi052 ceph-mon[31955]: Deploying daemon mds.d on smithi052 2024-05-27T07:21:09.940 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:21:09 smithi052 ceph-mon[31955]: mds.? [v2:172.21.15.52:6834/2584733778,v1:172.21.15.52:6835/2584733778] up:boot 2024-05-27T07:21:09.940 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:21:09 smithi052 ceph-mon[31955]: fsmap 3 up:standby 2024-05-27T07:21:09.940 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:21:09 smithi052 ceph-mon[31955]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "j"}]: dispatch 2024-05-27T07:21:10.075 INFO:teuthology.orchestra.run.smithi052.stderr:Inferring config /var/lib/ceph/73df2636-1bf8-11ef-bc9b-c7b262605968/mon.a/config 2024-05-27T07:21:10.102 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:21:09 smithi173 ceph-mon[37000]: Deploying daemon mds.d on smithi052 2024-05-27T07:21:10.102 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:21:09 smithi173 ceph-mon[37000]: mds.? [v2:172.21.15.52:6834/2584733778,v1:172.21.15.52:6835/2584733778] up:boot 2024-05-27T07:21:10.102 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:21:09 smithi173 ceph-mon[37000]: fsmap 3 up:standby 2024-05-27T07:21:10.102 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:21:09 smithi173 ceph-mon[37000]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "j"}]: dispatch 2024-05-27T07:21:10.190 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:21:09 smithi105 ceph-mon[35977]: Deploying daemon mds.d on smithi052 2024-05-27T07:21:10.190 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:21:09 smithi105 ceph-mon[35977]: mds.? [v2:172.21.15.52:6834/2584733778,v1:172.21.15.52:6835/2584733778] up:boot 2024-05-27T07:21:10.190 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:21:09 smithi105 ceph-mon[35977]: fsmap 3 up:standby 2024-05-27T07:21:10.190 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:21:09 smithi105 ceph-mon[35977]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "j"}]: dispatch 2024-05-27T07:21:10.980 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:21:10 smithi052 ceph-mon[31955]: pgmap v262: 1 pgs: 1 active+clean; 577 KiB data, 354 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-27T07:21:11.004 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:21:10 smithi173 ceph-mon[37000]: pgmap v262: 1 pgs: 1 active+clean; 577 KiB data, 354 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-27T07:21:11.190 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:21:10 smithi105 ceph-mon[35977]: pgmap v262: 1 pgs: 1 active+clean; 577 KiB data, 354 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-27T07:21:11.866 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:21:11.876+0000 7f8ab3f7a700 1 -- 172.21.15.52:0/775526724 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f8aac108730 msgr2=0x7f8aac110090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:21:11.867 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:21:11.876+0000 7f8ab3f7a700 1 --2- 172.21.15.52:0/775526724 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f8aac108730 0x7f8aac110090 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7f8aa40099c0 tx=0x7f8aa4009cd0 comp rx=0 tx=0).stop 2024-05-27T07:21:11.867 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:21:11.876+0000 7f8ab3f7a700 1 -- 172.21.15.52:0/775526724 shutdown_connections 2024-05-27T07:21:11.867 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:21:11.876+0000 7f8ab3f7a700 1 --2- 172.21.15.52:0/775526724 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f8aac108730 0x7f8aac110090 unknown :-1 s=CLOSED pgs=162 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:21:11.867 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:21:11.876+0000 7f8ab3f7a700 1 --2- 172.21.15.52:0/775526724 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f8aac1034b0 0x7f8aac1038c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:21:11.867 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:21:11.876+0000 7f8ab3f7a700 1 --2- 172.21.15.52:0/775526724 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f8aac107de0 0x7f8aac1081f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:21:11.867 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:21:11.876+0000 7f8ab3f7a700 1 -- 172.21.15.52:0/775526724 >> 172.21.15.52:0/775526724 conn(0x7f8aac0fede0 msgr2=0x7f8aac101200 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:21:11.867 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:21:11.876+0000 7f8ab3f7a700 1 -- 172.21.15.52:0/775526724 shutdown_connections 2024-05-27T07:21:11.867 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:21:11.876+0000 7f8ab3f7a700 1 -- 172.21.15.52:0/775526724 wait complete. 2024-05-27T07:21:11.869 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:21:11.877+0000 7f8ab3f7a700 1 Processor -- start 2024-05-27T07:21:11.869 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:21:11.877+0000 7f8ab3f7a700 1 -- start start 2024-05-27T07:21:11.869 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:21:11.877+0000 7f8ab3f7a700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f8aac1034b0 0x7f8aac19bca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:21:11.869 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:21:11.877+0000 7f8ab3f7a700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f8aac107de0 0x7f8aac19c1e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:21:11.869 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:21:11.877+0000 7f8ab3f7a700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f8aac108730 0x7f8aac19c760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:21:11.869 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:21:11.877+0000 7f8ab3f7a700 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x7f8aac1133e0 con 0x7f8aac108730 2024-05-27T07:21:11.869 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:21:11.877+0000 7f8ab3f7a700 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f8aac113260 con 0x7f8aac1034b0 2024-05-27T07:21:11.869 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:21:11.877+0000 7f8ab3f7a700 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f8aac113560 con 0x7f8aac107de0 2024-05-27T07:21:11.869 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:21:11.877+0000 7f8ab2517700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f8aac108730 0x7f8aac19c760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:21:11.869 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:21:11.877+0000 7f8ab1d16700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f8aac1034b0 0x7f8aac19bca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:21:11.869 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:21:11.877+0000 7f8ab2517700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f8aac108730 0x7f8aac19c760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.52:44610/0 (socket says 172.21.15.52:44610) 2024-05-27T07:21:11.870 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:21:11.878+0000 7f8ab2517700 1 -- 172.21.15.52:0/1771241895 learned_addr learned my addr 172.21.15.52:0/1771241895 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-05-27T07:21:11.870 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:21:11.878+0000 7f8ab1d16700 1 --2- 172.21.15.52:0/1771241895 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f8aac1034b0 0x7f8aac19bca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.52:55198/0 (socket says 172.21.15.52:55198) 2024-05-27T07:21:11.870 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:21:11.878+0000 7f8ab1515700 1 --2- 172.21.15.52:0/1771241895 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f8aac107de0 0x7f8aac19c1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:21:11.870 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:21:11.878+0000 7f8ab2517700 1 -- 172.21.15.52:0/1771241895 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f8aac107de0 msgr2=0x7f8aac19c1e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:21:11.870 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:21:11.878+0000 7f8ab2517700 1 --2- 172.21.15.52:0/1771241895 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f8aac107de0 0x7f8aac19c1e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:21:11.870 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:21:11.878+0000 7f8ab2517700 1 -- 172.21.15.52:0/1771241895 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f8aac1034b0 msgr2=0x7f8aac19bca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:21:11.870 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:21:11.878+0000 7f8ab2517700 1 --2- 172.21.15.52:0/1771241895 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f8aac1034b0 0x7f8aac19bca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:21:11.870 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:21:11.878+0000 7f8ab2517700 1 -- 172.21.15.52:0/1771241895 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8aac1a6f80 con 0x7f8aac108730 2024-05-27T07:21:11.871 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:21:11.878+0000 7f8ab2517700 1 --2- 172.21.15.52:0/1771241895 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f8aac108730 0x7f8aac19c760 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7f8aa4009990 tx=0x7f8aa40047b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:21:11.871 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:21:11.878+0000 7f8aa2ffd700 1 -- 172.21.15.52:0/1771241895 <== mon.0 v2:172.21.15.52:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8aa4027070 con 0x7f8aac108730 2024-05-27T07:21:11.871 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:21:11.878+0000 7f8ab3f7a700 1 -- 172.21.15.52:0/1771241895 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8aac1a7210 con 0x7f8aac108730 2024-05-27T07:21:11.871 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:21:11.878+0000 7f8aa2ffd700 1 -- 172.21.15.52:0/1771241895 <== mon.0 v2:172.21.15.52:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f8aa4003fa0 con 0x7f8aac108730 2024-05-27T07:21:11.871 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:21:11.878+0000 7f8aa2ffd700 1 -- 172.21.15.52:0/1771241895 <== mon.0 v2:172.21.15.52:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8aa4022d10 con 0x7f8aac108730 2024-05-27T07:21:11.871 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:21:11.878+0000 7f8ab3f7a700 1 -- 172.21.15.52:0/1771241895 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8aac1a76c0 con 0x7f8aac108730 2024-05-27T07:21:11.872 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:21:11.880+0000 7f8ab3f7a700 1 -- 172.21.15.52:0/1771241895 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8aac0512b0 con 0x7f8aac108730 2024-05-27T07:21:11.872 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:21:11.881+0000 7f8aa2ffd700 1 -- 172.21.15.52:0/1771241895 <== mon.0 v2:172.21.15.52:3300/0 4 ==== mgrmap(e 16) ==== 149652+0+0 (secure 0 0 0) 0x7f8aa402a070 con 0x7f8aac108730 2024-05-27T07:21:11.872 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:21:11.882+0000 7f8aa2ffd700 1 --2- 172.21.15.52:0/1771241895 >> [v2:172.21.15.52:6800/3513075805,v1:172.21.15.52:6801/3513075805] conn(0x7f8a980b2880 0x7f8a980b4d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:21:11.872 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:21:11.882+0000 7f8ab1d16700 1 --2- 172.21.15.52:0/1771241895 >> [v2:172.21.15.52:6800/3513075805,v1:172.21.15.52:6801/3513075805] conn(0x7f8a980b2880 0x7f8a980b4d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:21:11.872 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:21:11.882+0000 7f8aa2ffd700 1 -- 172.21.15.52:0/1771241895 <== mon.0 v2:172.21.15.52:3300/0 5 ==== osd_map(72..72 src has 1..72) ==== 7127+0+0 (secure 0 0 0) 0x7f8aa40e3120 con 0x7f8aac108730 2024-05-27T07:21:11.873 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:21:11.882+0000 7f8ab1d16700 1 --2- 172.21.15.52:0/1771241895 >> [v2:172.21.15.52:6800/3513075805,v1:172.21.15.52:6801/3513075805] conn(0x7f8a980b2880 0x7f8a980b4d90 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7f8aa800a400 tx=0x7f8aa800d040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:21:11.875 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:21:11.885+0000 7f8aa2ffd700 1 -- 172.21.15.52:0/1771241895 <== mon.0 v2:172.21.15.52:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191963 (secure 0 0 0) 0x7f8aa4031080 con 0x7f8aac108730 2024-05-27T07:21:12.143 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:21:12.152+0000 7f8ab3f7a700 1 -- 172.21.15.52:0/1771241895 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f8aac02e5e0 con 0x7f8aac108730 2024-05-27T07:21:12.143 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:21:12.153+0000 7f8aa2ffd700 1 -- 172.21.15.52:0/1771241895 <== mon.0 v2:172.21.15.52:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 4 v4) ==== 93+0+3049 (secure 0 0 0) 0x7f8aa40b1070 con 0x7f8aac108730 2024-05-27T07:21:12.143 INFO:teuthology.orchestra.run.smithi052.stdout: 2024-05-27T07:21:12.144 INFO:teuthology.orchestra.run.smithi052.stdout:{"epoch":4,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14556,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.52:6835/2584733778","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.52:6834","nonce":2584733778},{"type":"v1","addr":"172.21.15.52:6835","nonce":2584733778}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":24433,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.173:6833/3013589220","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6832","nonce":3013589220},{"type":"v1","addr":"172.21.15.173:6833","nonce":3013589220}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24452,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.105:6833/2800678412","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6832","nonce":2800678412},{"type":"v1","addr":"172.21.15.105:6833","nonce":2800678412}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[]} 2024-05-27T07:21:12.146 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:21:12.155+0000 7f8ab3f7a700 1 -- 172.21.15.52:0/1771241895 >> [v2:172.21.15.52:6800/3513075805,v1:172.21.15.52:6801/3513075805] conn(0x7f8a980b2880 msgr2=0x7f8a980b4d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:21:12.146 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:21:12.155+0000 7f8ab3f7a700 1 --2- 172.21.15.52:0/1771241895 >> [v2:172.21.15.52:6800/3513075805,v1:172.21.15.52:6801/3513075805] conn(0x7f8a980b2880 0x7f8a980b4d90 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7f8aa800a400 tx=0x7f8aa800d040 comp rx=0 tx=0).stop 2024-05-27T07:21:12.146 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:21:12.155+0000 7f8ab3f7a700 1 -- 172.21.15.52:0/1771241895 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f8aac108730 msgr2=0x7f8aac19c760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:21:12.146 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:21:12.155+0000 7f8ab3f7a700 1 --2- 172.21.15.52:0/1771241895 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f8aac108730 0x7f8aac19c760 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7f8aa4009990 tx=0x7f8aa40047b0 comp rx=0 tx=0).stop 2024-05-27T07:21:12.146 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:21:12.155+0000 7f8ab3f7a700 1 -- 172.21.15.52:0/1771241895 shutdown_connections 2024-05-27T07:21:12.146 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:21:12.155+0000 7f8ab3f7a700 1 --2- 172.21.15.52:0/1771241895 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f8aac108730 0x7f8aac19c760 unknown :-1 s=CLOSED pgs=163 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:21:12.146 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:21:12.155+0000 7f8ab3f7a700 1 --2- 172.21.15.52:0/1771241895 >> [v2:172.21.15.52:6800/3513075805,v1:172.21.15.52:6801/3513075805] conn(0x7f8a980b2880 0x7f8a980b4d90 unknown :-1 s=CLOSED pgs=125 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:21:12.146 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:21:12.155+0000 7f8ab3f7a700 1 --2- 172.21.15.52:0/1771241895 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f8aac1034b0 0x7f8aac19bca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:21:12.146 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:21:12.155+0000 7f8ab3f7a700 1 --2- 172.21.15.52:0/1771241895 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f8aac107de0 0x7f8aac19c1e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:21:12.146 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:21:12.155+0000 7f8ab3f7a700 1 -- 172.21.15.52:0/1771241895 >> 172.21.15.52:0/1771241895 conn(0x7f8aac0fede0 msgr2=0x7f8aac107550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:21:12.146 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:21:12.156+0000 7f8ab3f7a700 1 -- 172.21.15.52:0/1771241895 shutdown_connections 2024-05-27T07:21:12.147 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:21:12.156+0000 7f8ab3f7a700 1 -- 172.21.15.52:0/1771241895 wait complete. 2024-05-27T07:21:12.147 INFO:teuthology.orchestra.run.smithi052.stderr:dumped fsmap epoch 4 2024-05-27T07:21:12.960 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:21:12 smithi052 ceph-mon[31955]: pgmap v263: 1 pgs: 1 active+clean; 577 KiB data, 354 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-27T07:21:12.960 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:21:12 smithi052 ceph-mon[31955]: from='client.? 172.21.15.52:0/1771241895' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-27T07:21:12.960 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:21:12 smithi052 ceph-mon[31955]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' 2024-05-27T07:21:12.960 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:21:12 smithi052 ceph-mon[31955]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' 2024-05-27T07:21:12.960 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:21:12 smithi052 ceph-mon[31955]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' 2024-05-27T07:21:12.960 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:21:12 smithi052 ceph-mon[31955]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.e", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-05-27T07:21:12.961 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:21:12 smithi052 ceph-mon[31955]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.e", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-05-27T07:21:12.961 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:21:12 smithi052 ceph-mon[31955]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T07:21:13.027 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:21:12 smithi105 ceph-mon[35977]: pgmap v263: 1 pgs: 1 active+clean; 577 KiB data, 354 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-27T07:21:13.027 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:21:12 smithi105 ceph-mon[35977]: from='client.? 172.21.15.52:0/1771241895' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-27T07:21:13.027 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:21:12 smithi105 ceph-mon[35977]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' 2024-05-27T07:21:13.027 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:21:12 smithi105 ceph-mon[35977]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' 2024-05-27T07:21:13.027 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:21:12 smithi105 ceph-mon[35977]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' 2024-05-27T07:21:13.027 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:21:12 smithi105 ceph-mon[35977]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.e", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-05-27T07:21:13.027 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:21:12 smithi105 ceph-mon[35977]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.e", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-05-27T07:21:13.028 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:21:12 smithi105 ceph-mon[35977]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T07:21:13.102 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:21:12 smithi173 ceph-mon[37000]: pgmap v263: 1 pgs: 1 active+clean; 577 KiB data, 354 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-27T07:21:13.102 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:21:12 smithi173 ceph-mon[37000]: from='client.? 172.21.15.52:0/1771241895' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-27T07:21:13.102 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:21:12 smithi173 ceph-mon[37000]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' 2024-05-27T07:21:13.102 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:21:12 smithi173 ceph-mon[37000]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' 2024-05-27T07:21:13.102 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:21:12 smithi173 ceph-mon[37000]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' 2024-05-27T07:21:13.102 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:21:12 smithi173 ceph-mon[37000]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.e", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-05-27T07:21:13.103 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:21:12 smithi173 ceph-mon[37000]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.e", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-05-27T07:21:13.103 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:21:12 smithi173 ceph-mon[37000]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T07:21:13.980 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:21:13 smithi052 ceph-mon[31955]: Deploying daemon mds.e on smithi105 2024-05-27T07:21:13.980 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:21:13 smithi052 ceph-mon[31955]: mds.? [v2:172.21.15.52:6836/675445303,v1:172.21.15.52:6837/675445303] up:boot 2024-05-27T07:21:13.980 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:21:13 smithi052 ceph-mon[31955]: fsmap 4 up:standby 2024-05-27T07:21:13.980 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:21:13 smithi052 ceph-mon[31955]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "d"}]: dispatch 2024-05-27T07:21:14.102 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:21:13 smithi173 ceph-mon[37000]: Deploying daemon mds.e on smithi105 2024-05-27T07:21:14.102 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:21:13 smithi173 ceph-mon[37000]: mds.? [v2:172.21.15.52:6836/675445303,v1:172.21.15.52:6837/675445303] up:boot 2024-05-27T07:21:14.102 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:21:13 smithi173 ceph-mon[37000]: fsmap 4 up:standby 2024-05-27T07:21:14.102 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:21:13 smithi173 ceph-mon[37000]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "d"}]: dispatch 2024-05-27T07:21:14.154 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:21:13 smithi105 ceph-mon[35977]: Deploying daemon mds.e on smithi105 2024-05-27T07:21:14.154 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:21:13 smithi105 ceph-mon[35977]: mds.? [v2:172.21.15.52:6836/675445303,v1:172.21.15.52:6837/675445303] up:boot 2024-05-27T07:21:14.154 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:21:13 smithi105 ceph-mon[35977]: fsmap 4 up:standby 2024-05-27T07:21:14.154 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:21:13 smithi105 ceph-mon[35977]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "d"}]: dispatch 2024-05-27T07:21:14.690 INFO:journalctl@ceph.mds.e.smithi105.stdout:May 27 07:21:14 smithi105 systemd[1]: Starting Ceph mds.e for 73df2636-1bf8-11ef-bc9b-c7b262605968... 2024-05-27T07:21:14.979 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:21:14 smithi052 ceph-mon[31955]: pgmap v264: 1 pgs: 1 active+clean; 577 KiB data, 354 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-27T07:21:14.981 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:21:14 smithi105 ceph-mon[35977]: pgmap v264: 1 pgs: 1 active+clean; 577 KiB data, 354 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-27T07:21:14.981 INFO:journalctl@ceph.mds.e.smithi105.stdout:May 27 07:21:14 smithi105 podman[64916]: 2024-05-27 07:21:14.74518248 +0000 UTC m=+0.018942352 image pull e36ea4fc8a5807e8d2701370e350a0de3856538797f7fba64cfce7e344bd6799 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6 2024-05-27T07:21:14.981 INFO:journalctl@ceph.mds.e.smithi105.stdout:May 27 07:21:14 smithi105 podman[64916]: 2024-05-27 07:21:14.858482825 +0000 UTC m=+0.132242674 container create 04d00bde09b404f47e49ec06cf03c8343ec4e1627082b900a48012676de9f248 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-e, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240521, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, GIT_BRANCH=HEAD) 2024-05-27T07:21:14.991 DEBUG:teuthology.orchestra.run.smithi052:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:38f9d900c83c63799fbdbe61acc9a11b0d3554a6 shell --fsid 73df2636-1bf8-11ef-bc9b-c7b262605968 -- ceph fs dump --format=json 2024-05-27T07:21:15.102 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:21:14 smithi173 ceph-mon[37000]: pgmap v264: 1 pgs: 1 active+clean; 577 KiB data, 354 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-27T07:21:15.259 INFO:teuthology.orchestra.run.smithi052.stderr:Inferring config /var/lib/ceph/73df2636-1bf8-11ef-bc9b-c7b262605968/mon.a/config 2024-05-27T07:21:15.440 INFO:journalctl@ceph.mds.e.smithi105.stdout:May 27 07:21:15 smithi105 podman[64916]: 2024-05-27 07:21:15.094496558 +0000 UTC m=+0.368256421 container init 04d00bde09b404f47e49ec06cf03c8343ec4e1627082b900a48012676de9f248 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-e, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240521, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, org.label-schema.license=GPLv2, GIT_CLEAN=True, org.label-schema.vendor=CentOS, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-05-27T07:21:15.440 INFO:journalctl@ceph.mds.e.smithi105.stdout:May 27 07:21:15 smithi105 podman[64916]: 2024-05-27 07:21:15.09708603 +0000 UTC m=+0.370845892 container start 04d00bde09b404f47e49ec06cf03c8343ec4e1627082b900a48012676de9f248 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-e, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , ceph=True, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240521, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS) 2024-05-27T07:21:15.440 INFO:journalctl@ceph.mds.e.smithi105.stdout:May 27 07:21:15 smithi105 ceph-mds[64936]: -- 172.21.15.105:0/366067688 <== mon.2 v2:172.21.15.105:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55b89b349a40 con 0x55b89b391a80 2024-05-27T07:21:15.441 INFO:journalctl@ceph.mds.e.smithi105.stdout:May 27 07:21:15 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-e[64932]: starting mds.e at 2024-05-27T07:21:15.441 INFO:journalctl@ceph.mds.e.smithi105.stdout:May 27 07:21:15 smithi105 bash[64916]: 04d00bde09b404f47e49ec06cf03c8343ec4e1627082b900a48012676de9f248 2024-05-27T07:21:15.441 INFO:journalctl@ceph.mds.e.smithi105.stdout:May 27 07:21:15 smithi105 systemd[1]: Started Ceph mds.e for 73df2636-1bf8-11ef-bc9b-c7b262605968. 2024-05-27T07:21:16.436 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:21:16 smithi052 ceph-mon[31955]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' 2024-05-27T07:21:16.436 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:21:16 smithi052 ceph-mon[31955]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' 2024-05-27T07:21:16.436 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:21:16 smithi052 ceph-mon[31955]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' 2024-05-27T07:21:16.436 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:21:16 smithi052 ceph-mon[31955]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.f", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-05-27T07:21:16.436 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:21:16 smithi052 ceph-mon[31955]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.f", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-05-27T07:21:16.436 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:21:16 smithi052 ceph-mon[31955]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T07:21:16.436 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:21:16 smithi052 ceph-mon[31955]: Deploying daemon mds.f on smithi173 2024-05-27T07:21:16.436 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:21:16 smithi052 ceph-mon[31955]: pgmap v265: 1 pgs: 1 active+clean; 577 KiB data, 354 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-27T07:21:16.437 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:21:16 smithi052 ceph-mon[31955]: mds.? [v2:172.21.15.105:6834/3298498384,v1:172.21.15.105:6835/3298498384] up:boot 2024-05-27T07:21:16.437 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:21:16 smithi052 ceph-mon[31955]: fsmap 5 up:standby 2024-05-27T07:21:16.437 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:21:16 smithi052 ceph-mon[31955]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "e"}]: dispatch 2024-05-27T07:21:16.440 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:21:16 smithi105 ceph-mon[35977]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' 2024-05-27T07:21:16.440 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:21:16 smithi105 ceph-mon[35977]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' 2024-05-27T07:21:16.440 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:21:16 smithi105 ceph-mon[35977]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' 2024-05-27T07:21:16.440 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:21:16 smithi105 ceph-mon[35977]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.f", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-05-27T07:21:16.440 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:21:16 smithi105 ceph-mon[35977]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.f", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-05-27T07:21:16.440 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:21:16 smithi105 ceph-mon[35977]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T07:21:16.440 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:21:16 smithi105 ceph-mon[35977]: Deploying daemon mds.f on smithi173 2024-05-27T07:21:16.440 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:21:16 smithi105 ceph-mon[35977]: pgmap v265: 1 pgs: 1 active+clean; 577 KiB data, 354 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-27T07:21:16.440 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:21:16 smithi105 ceph-mon[35977]: mds.? [v2:172.21.15.105:6834/3298498384,v1:172.21.15.105:6835/3298498384] up:boot 2024-05-27T07:21:16.440 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:21:16 smithi105 ceph-mon[35977]: fsmap 5 up:standby 2024-05-27T07:21:16.440 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:21:16 smithi105 ceph-mon[35977]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "e"}]: dispatch 2024-05-27T07:21:16.588 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:21:16 smithi173 ceph-mon[37000]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' 2024-05-27T07:21:16.589 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:21:16 smithi173 ceph-mon[37000]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' 2024-05-27T07:21:16.589 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:21:16 smithi173 ceph-mon[37000]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' 2024-05-27T07:21:16.589 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:21:16 smithi173 ceph-mon[37000]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.f", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-05-27T07:21:16.589 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:21:16 smithi173 ceph-mon[37000]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.f", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-05-27T07:21:16.589 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:21:16 smithi173 ceph-mon[37000]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T07:21:16.589 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:21:16 smithi173 ceph-mon[37000]: Deploying daemon mds.f on smithi173 2024-05-27T07:21:16.589 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:21:16 smithi173 ceph-mon[37000]: pgmap v265: 1 pgs: 1 active+clean; 577 KiB data, 354 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-27T07:21:16.589 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:21:16 smithi173 ceph-mon[37000]: mds.? [v2:172.21.15.105:6834/3298498384,v1:172.21.15.105:6835/3298498384] up:boot 2024-05-27T07:21:16.589 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:21:16 smithi173 ceph-mon[37000]: fsmap 5 up:standby 2024-05-27T07:21:16.589 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:21:16 smithi173 ceph-mon[37000]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "e"}]: dispatch 2024-05-27T07:21:16.956 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:21:16.966+0000 7ff2c2475700 1 -- 172.21.15.52:0/1839167423 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff2bc1034b0 msgr2=0x7ff2bc1038c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:21:16.956 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:21:16.966+0000 7ff2c2475700 1 --2- 172.21.15.52:0/1839167423 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff2bc1034b0 0x7ff2bc1038c0 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7ff2ac0099c0 tx=0x7ff2ac009cd0 comp rx=0 tx=0).stop 2024-05-27T07:21:16.956 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:21:16.966+0000 7ff2c2475700 1 -- 172.21.15.52:0/1839167423 shutdown_connections 2024-05-27T07:21:16.957 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:21:16.966+0000 7ff2c2475700 1 --2- 172.21.15.52:0/1839167423 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7ff2bc108730 0x7ff2bc110090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:21:16.957 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:21:16.966+0000 7ff2c2475700 1 --2- 172.21.15.52:0/1839167423 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff2bc1034b0 0x7ff2bc1038c0 unknown :-1 s=CLOSED pgs=166 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:21:16.957 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:21:16.966+0000 7ff2c2475700 1 --2- 172.21.15.52:0/1839167423 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7ff2bc107de0 0x7ff2bc1081f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:21:16.957 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:21:16.966+0000 7ff2c2475700 1 -- 172.21.15.52:0/1839167423 >> 172.21.15.52:0/1839167423 conn(0x7ff2bc0fedb0 msgr2=0x7ff2bc1011d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:21:16.957 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:21:16.966+0000 7ff2c2475700 1 -- 172.21.15.52:0/1839167423 shutdown_connections 2024-05-27T07:21:16.957 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:21:16.966+0000 7ff2c2475700 1 -- 172.21.15.52:0/1839167423 wait complete. 2024-05-27T07:21:16.959 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:21:16.967+0000 7ff2c2475700 1 Processor -- start 2024-05-27T07:21:16.959 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:21:16.967+0000 7ff2c2475700 1 -- start start 2024-05-27T07:21:16.959 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:21:16.967+0000 7ff2c2475700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff2bc1034b0 0x7ff2bc0ff290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:21:16.959 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:21:16.967+0000 7ff2c2475700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7ff2bc107de0 0x7ff2bc0ff7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:21:16.959 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:21:16.967+0000 7ff2c2475700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7ff2bc108730 0x7ff2bc0ffd10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:21:16.959 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:21:16.967+0000 7ff2c2475700 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x7ff2bc1133b0 con 0x7ff2bc1034b0 2024-05-27T07:21:16.959 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:21:16.967+0000 7ff2c2475700 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7ff2bc113230 con 0x7ff2bc107de0 2024-05-27T07:21:16.959 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:21:16.967+0000 7ff2c2475700 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7ff2bc113530 con 0x7ff2bc108730 2024-05-27T07:21:16.959 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:21:16.967+0000 7ff2bbfff700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff2bc1034b0 0x7ff2bc0ff290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:21:16.959 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:21:16.967+0000 7ff2bbfff700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff2bc1034b0 0x7ff2bc0ff290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.52:44644/0 (socket says 172.21.15.52:44644) 2024-05-27T07:21:16.960 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:21:16.967+0000 7ff2bbfff700 1 -- 172.21.15.52:0/1964658534 learned_addr learned my addr 172.21.15.52:0/1964658534 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-05-27T07:21:16.960 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:21:16.967+0000 7ff2b3fff700 1 --2- 172.21.15.52:0/1964658534 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7ff2bc107de0 0x7ff2bc0ff7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:21:16.960 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:21:16.967+0000 7ff2c0a12700 1 --2- 172.21.15.52:0/1964658534 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7ff2bc108730 0x7ff2bc0ffd10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:21:16.960 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:21:16.967+0000 7ff2bbfff700 1 -- 172.21.15.52:0/1964658534 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7ff2bc108730 msgr2=0x7ff2bc0ffd10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:21:16.960 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:21:16.967+0000 7ff2bbfff700 1 --2- 172.21.15.52:0/1964658534 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7ff2bc108730 0x7ff2bc0ffd10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:21:16.960 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:21:16.967+0000 7ff2bbfff700 1 -- 172.21.15.52:0/1964658534 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7ff2bc107de0 msgr2=0x7ff2bc0ff7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:21:16.960 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:21:16.967+0000 7ff2bbfff700 1 --2- 172.21.15.52:0/1964658534 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7ff2bc107de0 0x7ff2bc0ff7d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:21:16.960 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:21:16.967+0000 7ff2bbfff700 1 -- 172.21.15.52:0/1964658534 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff2bc1a9190 con 0x7ff2bc1034b0 2024-05-27T07:21:16.960 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:21:16.968+0000 7ff2bbfff700 1 --2- 172.21.15.52:0/1964658534 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff2bc1034b0 0x7ff2bc0ff290 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7ff2ac009990 tx=0x7ff2ac0047b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:21:16.961 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:21:16.968+0000 7ff2b9ffb700 1 -- 172.21.15.52:0/1964658534 <== mon.0 v2:172.21.15.52:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff2ac026070 con 0x7ff2bc1034b0 2024-05-27T07:21:16.961 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:21:16.968+0000 7ff2c2475700 1 -- 172.21.15.52:0/1964658534 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff2bc1a9420 con 0x7ff2bc1034b0 2024-05-27T07:21:16.961 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:21:16.968+0000 7ff2b9ffb700 1 -- 172.21.15.52:0/1964658534 <== mon.0 v2:172.21.15.52:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ff2ac003f20 con 0x7ff2bc1034b0 2024-05-27T07:21:16.961 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:21:16.968+0000 7ff2b9ffb700 1 -- 172.21.15.52:0/1964658534 <== mon.0 v2:172.21.15.52:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff2ac00fcc0 con 0x7ff2bc1034b0 2024-05-27T07:21:16.961 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:21:16.968+0000 7ff2c2475700 1 -- 172.21.15.52:0/1964658534 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff2bc1a98d0 con 0x7ff2bc1034b0 2024-05-27T07:21:16.961 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:21:16.969+0000 7ff2c2475700 1 -- 172.21.15.52:0/1964658534 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff2bc0512b0 con 0x7ff2bc1034b0 2024-05-27T07:21:16.961 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:21:16.971+0000 7ff2b9ffb700 1 -- 172.21.15.52:0/1964658534 <== mon.0 v2:172.21.15.52:3300/0 4 ==== mgrmap(e 16) ==== 149652+0+0 (secure 0 0 0) 0x7ff2ac029070 con 0x7ff2bc1034b0 2024-05-27T07:21:16.962 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:21:16.972+0000 7ff2b9ffb700 1 --2- 172.21.15.52:0/1964658534 >> [v2:172.21.15.52:6800/3513075805,v1:172.21.15.52:6801/3513075805] conn(0x7ff2a80b2880 0x7ff2a80b4d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:21:16.962 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:21:16.972+0000 7ff2b3fff700 1 --2- 172.21.15.52:0/1964658534 >> [v2:172.21.15.52:6800/3513075805,v1:172.21.15.52:6801/3513075805] conn(0x7ff2a80b2880 0x7ff2a80b4d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:21:16.962 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:21:16.972+0000 7ff2b9ffb700 1 -- 172.21.15.52:0/1964658534 <== mon.0 v2:172.21.15.52:3300/0 5 ==== osd_map(72..72 src has 1..72) ==== 7127+0+0 (secure 0 0 0) 0x7ff2ac0e2a20 con 0x7ff2bc1034b0 2024-05-27T07:21:16.963 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:21:16.973+0000 7ff2b3fff700 1 --2- 172.21.15.52:0/1964658534 >> [v2:172.21.15.52:6800/3513075805,v1:172.21.15.52:6801/3513075805] conn(0x7ff2a80b2880 0x7ff2a80b4d90 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7ff2bc1086a0 tx=0x7ff2a400b410 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:21:16.966 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:21:16.976+0000 7ff2b9ffb700 1 -- 172.21.15.52:0/1964658534 <== mon.0 v2:172.21.15.52:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191963 (secure 0 0 0) 0x7ff2ac05edf0 con 0x7ff2bc1034b0 2024-05-27T07:21:17.229 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:21:17.239+0000 7ff2c2475700 1 -- 172.21.15.52:0/1964658534 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7ff2bc02e5e0 con 0x7ff2bc1034b0 2024-05-27T07:21:17.230 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:21:17.240+0000 7ff2b9ffb700 1 -- 172.21.15.52:0/1964658534 <== mon.0 v2:172.21.15.52:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 6 v6) ==== 93+0+4699 (secure 0 0 0) 0x7ff2ac013070 con 0x7ff2bc1034b0 2024-05-27T07:21:17.231 INFO:teuthology.orchestra.run.smithi052.stdout: 2024-05-27T07:21:17.231 INFO:teuthology.orchestra.run.smithi052.stdout:{"epoch":6,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14556,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.52:6835/2584733778","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.52:6834","nonce":2584733778},{"type":"v1","addr":"172.21.15.52:6835","nonce":2584733778}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14568,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.52:6837/675445303","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.52:6836","nonce":675445303},{"type":"v1","addr":"172.21.15.52:6837","nonce":675445303}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":24433,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.173:6833/3013589220","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6832","nonce":3013589220},{"type":"v1","addr":"172.21.15.173:6833","nonce":3013589220}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24452,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.105:6833/2800678412","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6832","nonce":2800678412},{"type":"v1","addr":"172.21.15.105:6833","nonce":2800678412}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24473,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.105:6835/3298498384","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6834","nonce":3298498384},{"type":"v1","addr":"172.21.15.105:6835","nonce":3298498384}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6}],"filesystems":[]} 2024-05-27T07:21:17.233 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:21:17.242+0000 7ff2c2475700 1 -- 172.21.15.52:0/1964658534 >> [v2:172.21.15.52:6800/3513075805,v1:172.21.15.52:6801/3513075805] conn(0x7ff2a80b2880 msgr2=0x7ff2a80b4d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:21:17.233 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:21:17.242+0000 7ff2c2475700 1 --2- 172.21.15.52:0/1964658534 >> [v2:172.21.15.52:6800/3513075805,v1:172.21.15.52:6801/3513075805] conn(0x7ff2a80b2880 0x7ff2a80b4d90 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7ff2bc1086a0 tx=0x7ff2a400b410 comp rx=0 tx=0).stop 2024-05-27T07:21:17.233 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:21:17.242+0000 7ff2c2475700 1 -- 172.21.15.52:0/1964658534 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff2bc1034b0 msgr2=0x7ff2bc0ff290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:21:17.234 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:21:17.242+0000 7ff2c2475700 1 --2- 172.21.15.52:0/1964658534 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff2bc1034b0 0x7ff2bc0ff290 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7ff2ac009990 tx=0x7ff2ac0047b0 comp rx=0 tx=0).stop 2024-05-27T07:21:17.234 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:21:17.242+0000 7ff2c2475700 1 -- 172.21.15.52:0/1964658534 shutdown_connections 2024-05-27T07:21:17.234 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:21:17.242+0000 7ff2c2475700 1 --2- 172.21.15.52:0/1964658534 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7ff2bc108730 0x7ff2bc0ffd10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:21:17.234 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:21:17.242+0000 7ff2c2475700 1 --2- 172.21.15.52:0/1964658534 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff2bc1034b0 0x7ff2bc0ff290 unknown :-1 s=CLOSED pgs=167 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:21:17.234 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:21:17.242+0000 7ff2c2475700 1 --2- 172.21.15.52:0/1964658534 >> [v2:172.21.15.52:6800/3513075805,v1:172.21.15.52:6801/3513075805] conn(0x7ff2a80b2880 0x7ff2a80b4d90 unknown :-1 s=CLOSED pgs=128 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:21:17.234 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:21:17.242+0000 7ff2c2475700 1 --2- 172.21.15.52:0/1964658534 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7ff2bc107de0 0x7ff2bc0ff7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:21:17.235 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:21:17.242+0000 7ff2c2475700 1 -- 172.21.15.52:0/1964658534 >> 172.21.15.52:0/1964658534 conn(0x7ff2bc0fedb0 msgr2=0x7ff2bc06b970 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:21:17.235 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:21:17.243+0000 7ff2c2475700 1 -- 172.21.15.52:0/1964658534 shutdown_connections 2024-05-27T07:21:17.235 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:21:17.243+0000 7ff2c2475700 1 -- 172.21.15.52:0/1964658534 wait complete. 2024-05-27T07:21:17.235 INFO:teuthology.orchestra.run.smithi052.stderr:dumped fsmap epoch 6 2024-05-27T07:21:17.529 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:21:17 smithi052 ceph-mon[31955]: from='client.? 172.21.15.52:0/1964658534' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-27T07:21:17.574 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:21:17 smithi173 ceph-mon[37000]: from='client.? 172.21.15.52:0/1964658534' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-27T07:21:17.575 INFO:journalctl@ceph.mds.f.smithi173.stdout:May 27 07:21:17 smithi173 systemd[1]: Starting Ceph mds.f for 73df2636-1bf8-11ef-bc9b-c7b262605968... 2024-05-27T07:21:17.690 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:21:17 smithi105 ceph-mon[35977]: from='client.? 172.21.15.52:0/1964658534' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-27T07:21:17.852 INFO:journalctl@ceph.mds.f.smithi173.stdout:May 27 07:21:17 smithi173 podman[67730]: 2024-05-27 07:21:17.487306823 +0000 UTC m=+0.017074015 image pull e36ea4fc8a5807e8d2701370e350a0de3856538797f7fba64cfce7e344bd6799 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6 2024-05-27T07:21:17.852 INFO:journalctl@ceph.mds.f.smithi173.stdout:May 27 07:21:17 smithi173 podman[67730]: 2024-05-27 07:21:17.611055191 +0000 UTC m=+0.140822360 container create e053aecb986487270fcbb179981fb8e9a24093408f8a147f12a6f004f4318f3d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-f, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, GIT_CLEAN=True, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240521) 2024-05-27T07:21:18.352 INFO:journalctl@ceph.mds.f.smithi173.stdout:May 27 07:21:17 smithi173 podman[67730]: 2024-05-27 07:21:17.898431664 +0000 UTC m=+0.428198840 container init e053aecb986487270fcbb179981fb8e9a24093408f8a147f12a6f004f4318f3d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-f, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, org.label-schema.vendor=CentOS, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, ceph=True, org.label-schema.build-date=20240521, GIT_CLEAN=True, org.label-schema.license=GPLv2) 2024-05-27T07:21:18.352 INFO:journalctl@ceph.mds.f.smithi173.stdout:May 27 07:21:17 smithi173 podman[67730]: 2024-05-27 07:21:17.901027815 +0000 UTC m=+0.430795002 container start e053aecb986487270fcbb179981fb8e9a24093408f8a147f12a6f004f4318f3d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-f, GIT_CLEAN=True, org.label-schema.vendor=CentOS, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240521, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, org.label-schema.license=GPLv2) 2024-05-27T07:21:18.352 INFO:journalctl@ceph.mds.f.smithi173.stdout:May 27 07:21:17 smithi173 ceph-mds[67749]: -- 172.21.15.173:0/4229557556 <== mon.1 v2:172.21.15.173:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55578d159a40 con 0x55578d1a1180 2024-05-27T07:21:18.352 INFO:journalctl@ceph.mds.f.smithi173.stdout:May 27 07:21:17 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-f[67745]: starting mds.f at 2024-05-27T07:21:18.352 INFO:journalctl@ceph.mds.f.smithi173.stdout:May 27 07:21:17 smithi173 bash[67730]: e053aecb986487270fcbb179981fb8e9a24093408f8a147f12a6f004f4318f3d 2024-05-27T07:21:18.353 INFO:journalctl@ceph.mds.f.smithi173.stdout:May 27 07:21:18 smithi173 systemd[1]: Started Ceph mds.f for 73df2636-1bf8-11ef-bc9b-c7b262605968. 2024-05-27T07:21:19.352 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:21:19 smithi173 ceph-mon[37000]: pgmap v266: 1 pgs: 1 active+clean; 577 KiB data, 354 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-27T07:21:19.353 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:21:19 smithi173 ceph-mon[37000]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' 2024-05-27T07:21:19.353 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:21:19 smithi173 ceph-mon[37000]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' 2024-05-27T07:21:19.353 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:21:19 smithi173 ceph-mon[37000]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' 2024-05-27T07:21:19.353 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:21:19 smithi173 ceph-mon[37000]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.c", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-05-27T07:21:19.353 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:21:19 smithi173 ceph-mon[37000]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.c", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-05-27T07:21:19.353 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:21:19 smithi173 ceph-mon[37000]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T07:21:19.354 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:21:19 smithi173 ceph-mon[37000]: Deploying daemon mds.c on smithi173 2024-05-27T07:21:19.354 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:21:19 smithi173 ceph-mon[37000]: mds.? [v2:172.21.15.173:6834/1619486437,v1:172.21.15.173:6835/1619486437] up:boot 2024-05-27T07:21:19.354 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:21:19 smithi173 ceph-mon[37000]: fsmap 6 up:standby 2024-05-27T07:21:19.354 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:21:19 smithi173 ceph-mon[37000]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2024-05-27T07:21:19.440 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:21:19 smithi105 ceph-mon[35977]: pgmap v266: 1 pgs: 1 active+clean; 577 KiB data, 354 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-27T07:21:19.440 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:21:19 smithi105 ceph-mon[35977]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' 2024-05-27T07:21:19.441 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:21:19 smithi105 ceph-mon[35977]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' 2024-05-27T07:21:19.441 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:21:19 smithi105 ceph-mon[35977]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' 2024-05-27T07:21:19.441 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:21:19 smithi105 ceph-mon[35977]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.c", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-05-27T07:21:19.441 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:21:19 smithi105 ceph-mon[35977]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.c", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-05-27T07:21:19.441 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:21:19 smithi105 ceph-mon[35977]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T07:21:19.441 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:21:19 smithi105 ceph-mon[35977]: Deploying daemon mds.c on smithi173 2024-05-27T07:21:19.441 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:21:19 smithi105 ceph-mon[35977]: mds.? [v2:172.21.15.173:6834/1619486437,v1:172.21.15.173:6835/1619486437] up:boot 2024-05-27T07:21:19.441 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:21:19 smithi105 ceph-mon[35977]: fsmap 6 up:standby 2024-05-27T07:21:19.441 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:21:19 smithi105 ceph-mon[35977]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2024-05-27T07:21:19.480 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:21:19 smithi052 ceph-mon[31955]: pgmap v266: 1 pgs: 1 active+clean; 577 KiB data, 354 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-27T07:21:19.480 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:21:19 smithi052 ceph-mon[31955]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' 2024-05-27T07:21:19.480 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:21:19 smithi052 ceph-mon[31955]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' 2024-05-27T07:21:19.480 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:21:19 smithi052 ceph-mon[31955]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' 2024-05-27T07:21:19.480 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:21:19 smithi052 ceph-mon[31955]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.c", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-05-27T07:21:19.480 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:21:19 smithi052 ceph-mon[31955]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.c", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-05-27T07:21:19.480 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:21:19 smithi052 ceph-mon[31955]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T07:21:19.480 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:21:19 smithi052 ceph-mon[31955]: Deploying daemon mds.c on smithi173 2024-05-27T07:21:19.480 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:21:19 smithi052 ceph-mon[31955]: mds.? [v2:172.21.15.173:6834/1619486437,v1:172.21.15.173:6835/1619486437] up:boot 2024-05-27T07:21:19.480 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:21:19 smithi052 ceph-mon[31955]: fsmap 6 up:standby 2024-05-27T07:21:19.481 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:21:19 smithi052 ceph-mon[31955]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2024-05-27T07:21:19.855 DEBUG:teuthology.orchestra.run.smithi052:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:38f9d900c83c63799fbdbe61acc9a11b0d3554a6 shell --fsid 73df2636-1bf8-11ef-bc9b-c7b262605968 -- ceph fs dump --format=json 2024-05-27T07:21:20.121 INFO:teuthology.orchestra.run.smithi052.stderr:Inferring config /var/lib/ceph/73df2636-1bf8-11ef-bc9b-c7b262605968/mon.a/config 2024-05-27T07:21:20.352 INFO:journalctl@ceph.mds.c.smithi173.stdout:May 27 07:21:20 smithi173 systemd[1]: Starting Ceph mds.c for 73df2636-1bf8-11ef-bc9b-c7b262605968... 2024-05-27T07:21:20.679 INFO:journalctl@ceph.mds.c.smithi173.stdout:May 27 07:21:20 smithi173 podman[68035]: 2024-05-27 07:21:20.278821144 +0000 UTC m=+0.017847917 image pull e36ea4fc8a5807e8d2701370e350a0de3856538797f7fba64cfce7e344bd6799 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6 2024-05-27T07:21:20.680 INFO:journalctl@ceph.mds.c.smithi173.stdout:May 27 07:21:20 smithi173 podman[68035]: 2024-05-27 07:21:20.420037227 +0000 UTC m=+0.159063996 container create 2c67bd7fd298e151c6edf858968c857239929a098508aee6f757811727548049 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-c, org.label-schema.schema-version=1.0, GIT_CLEAN=True, org.label-schema.build-date=20240521, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, org.label-schema.vendor=CentOS) 2024-05-27T07:21:20.680 INFO:journalctl@ceph.mds.c.smithi173.stdout:May 27 07:21:20 smithi173 podman[68035]: 2024-05-27 07:21:20.690029123 +0000 UTC m=+0.429055888 container init 2c67bd7fd298e151c6edf858968c857239929a098508aee6f757811727548049 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-c, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240521, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-05-27T07:21:21.049 INFO:journalctl@ceph.mds.c.smithi173.stdout:May 27 07:21:20 smithi173 podman[68035]: 2024-05-27 07:21:20.692778865 +0000 UTC m=+0.431805637 container start 2c67bd7fd298e151c6edf858968c857239929a098508aee6f757811727548049 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-c, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, GIT_CLEAN=True, ceph=True, org.label-schema.build-date=20240521, org.label-schema.vendor=CentOS) 2024-05-27T07:21:21.050 INFO:journalctl@ceph.mds.c.smithi173.stdout:May 27 07:21:20 smithi173 ceph-mds[68054]: -- 172.21.15.173:0/1488459523 <== mon.1 v2:172.21.15.173:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55d315f15a40 con 0x55d315f5d600 2024-05-27T07:21:21.050 INFO:journalctl@ceph.mds.c.smithi173.stdout:May 27 07:21:20 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-c[68050]: starting mds.c at 2024-05-27T07:21:21.050 INFO:journalctl@ceph.mds.c.smithi173.stdout:May 27 07:21:20 smithi173 bash[68035]: 2c67bd7fd298e151c6edf858968c857239929a098508aee6f757811727548049 2024-05-27T07:21:21.050 INFO:journalctl@ceph.mds.c.smithi173.stdout:May 27 07:21:20 smithi173 systemd[1]: Started Ceph mds.c for 73df2636-1bf8-11ef-bc9b-c7b262605968. 2024-05-27T07:21:21.352 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:21:21 smithi173 ceph-mon[37000]: pgmap v267: 1 pgs: 1 active+clean; 577 KiB data, 354 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-27T07:21:21.352 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:21:21 smithi173 ceph-mon[37000]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' 2024-05-27T07:21:21.352 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:21:21 smithi173 ceph-mon[37000]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' 2024-05-27T07:21:21.352 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:21:21 smithi173 ceph-mon[37000]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' 2024-05-27T07:21:21.353 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:21:21 smithi173 ceph-mon[37000]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.h", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-05-27T07:21:21.353 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:21:21 smithi173 ceph-mon[37000]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.h", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-05-27T07:21:21.353 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:21:21 smithi173 ceph-mon[37000]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T07:21:21.377 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:21:21 smithi052 ceph-mon[31955]: pgmap v267: 1 pgs: 1 active+clean; 577 KiB data, 354 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-27T07:21:21.377 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:21:21 smithi052 ceph-mon[31955]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' 2024-05-27T07:21:21.377 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:21:21 smithi052 ceph-mon[31955]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' 2024-05-27T07:21:21.377 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:21:21 smithi052 ceph-mon[31955]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' 2024-05-27T07:21:21.377 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:21:21 smithi052 ceph-mon[31955]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.h", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-05-27T07:21:21.377 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:21:21 smithi052 ceph-mon[31955]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.h", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-05-27T07:21:21.377 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:21:21 smithi052 ceph-mon[31955]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T07:21:21.440 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:21:21 smithi105 ceph-mon[35977]: pgmap v267: 1 pgs: 1 active+clean; 577 KiB data, 354 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-27T07:21:21.440 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:21:21 smithi105 ceph-mon[35977]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' 2024-05-27T07:21:21.440 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:21:21 smithi105 ceph-mon[35977]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' 2024-05-27T07:21:21.440 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:21:21 smithi105 ceph-mon[35977]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' 2024-05-27T07:21:21.440 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:21:21 smithi105 ceph-mon[35977]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.h", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-05-27T07:21:21.440 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:21:21 smithi105 ceph-mon[35977]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.h", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-05-27T07:21:21.440 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:21:21 smithi105 ceph-mon[35977]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T07:21:21.712 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:21:21.721+0000 7fb08021e700 1 -- 172.21.15.52:0/1233366258 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fb078108f00 msgr2=0x7fb078109310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:21:21.712 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:21:21.721+0000 7fb08021e700 1 --2- 172.21.15.52:0/1233366258 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fb078108f00 0x7fb078109310 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7fb0680099c0 tx=0x7fb068009cd0 comp rx=0 tx=0).stop 2024-05-27T07:21:21.712 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:21:21.722+0000 7fb08021e700 1 -- 172.21.15.52:0/1233366258 shutdown_connections 2024-05-27T07:21:21.712 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:21:21.722+0000 7fb08021e700 1 --2- 172.21.15.52:0/1233366258 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fb078109850 0x7fb0781111b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:21:21.712 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:21:21.722+0000 7fb08021e700 1 --2- 172.21.15.52:0/1233366258 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fb0781045d0 0x7fb0781049e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:21:21.713 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:21:21.722+0000 7fb08021e700 1 --2- 172.21.15.52:0/1233366258 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fb078108f00 0x7fb078109310 unknown :-1 s=CLOSED pgs=168 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:21:21.713 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:21:21.722+0000 7fb08021e700 1 -- 172.21.15.52:0/1233366258 >> 172.21.15.52:0/1233366258 conn(0x7fb0780ffe60 msgr2=0x7fb0781022b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:21:21.713 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:21:21.722+0000 7fb08021e700 1 -- 172.21.15.52:0/1233366258 shutdown_connections 2024-05-27T07:21:21.713 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:21:21.722+0000 7fb08021e700 1 -- 172.21.15.52:0/1233366258 wait complete. 2024-05-27T07:21:21.715 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:21:21.722+0000 7fb08021e700 1 Processor -- start 2024-05-27T07:21:21.715 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:21:21.722+0000 7fb08021e700 1 -- start start 2024-05-27T07:21:21.715 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:21:21.723+0000 7fb08021e700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fb0781045d0 0x7fb078075ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:21:21.715 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:21:21.723+0000 7fb08021e700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fb078108f00 0x7fb078075ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:21:21.715 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:21:21.723+0000 7fb08021e700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fb078109850 0x7fb078076530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:21:21.715 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:21:21.723+0000 7fb08021e700 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x7fb0781144b0 con 0x7fb078108f00 2024-05-27T07:21:21.715 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:21:21.723+0000 7fb08021e700 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7fb078114330 con 0x7fb078109850 2024-05-27T07:21:21.715 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:21:21.723+0000 7fb08021e700 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7fb078114630 con 0x7fb0781045d0 2024-05-27T07:21:21.715 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:21:21.723+0000 7fb07d7b9700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fb078108f00 0x7fb078075ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:21:21.715 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:21:21.723+0000 7fb07dfba700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fb0781045d0 0x7fb078075ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:21:21.716 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:21:21.723+0000 7fb07d7b9700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fb078108f00 0x7fb078075ff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.52:36488/0 (socket says 172.21.15.52:36488) 2024-05-27T07:21:21.716 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:21:21.723+0000 7fb07d7b9700 1 -- 172.21.15.52:0/3841978640 learned_addr learned my addr 172.21.15.52:0/3841978640 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-05-27T07:21:21.716 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:21:21.723+0000 7fb07e7bb700 1 --2- 172.21.15.52:0/3841978640 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fb078109850 0x7fb078076530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:21:21.716 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:21:21.723+0000 7fb07d7b9700 1 -- 172.21.15.52:0/3841978640 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fb0781045d0 msgr2=0x7fb078075ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:21:21.716 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:21:21.723+0000 7fb07d7b9700 1 --2- 172.21.15.52:0/3841978640 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fb0781045d0 0x7fb078075ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:21:21.716 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:21:21.723+0000 7fb07d7b9700 1 -- 172.21.15.52:0/3841978640 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fb078109850 msgr2=0x7fb078076530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:21:21.716 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:21:21.723+0000 7fb07d7b9700 1 --2- 172.21.15.52:0/3841978640 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fb078109850 0x7fb078076530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:21:21.716 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:21:21.723+0000 7fb07d7b9700 1 -- 172.21.15.52:0/3841978640 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb0781a9190 con 0x7fb078108f00 2024-05-27T07:21:21.716 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:21:21.723+0000 7fb07d7b9700 1 --2- 172.21.15.52:0/3841978640 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fb078108f00 0x7fb078075ff0 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7fb068006010 tx=0x7fb0680047b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:21:21.716 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:21:21.724+0000 7fb06effd700 1 -- 172.21.15.52:0/3841978640 <== mon.0 v2:172.21.15.52:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb068027070 con 0x7fb078108f00 2024-05-27T07:21:21.716 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:21:21.724+0000 7fb08021e700 1 -- 172.21.15.52:0/3841978640 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb0781a9420 con 0x7fb078108f00 2024-05-27T07:21:21.716 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:21:21.724+0000 7fb06effd700 1 -- 172.21.15.52:0/3841978640 <== mon.0 v2:172.21.15.52:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fb068003f20 con 0x7fb078108f00 2024-05-27T07:21:21.716 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:21:21.724+0000 7fb06effd700 1 -- 172.21.15.52:0/3841978640 <== mon.0 v2:172.21.15.52:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb068022c60 con 0x7fb078108f00 2024-05-27T07:21:21.716 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:21:21.724+0000 7fb08021e700 1 -- 172.21.15.52:0/3841978640 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb0781a98d0 con 0x7fb078108f00 2024-05-27T07:21:21.717 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:21:21.725+0000 7fb08021e700 1 -- 172.21.15.52:0/3841978640 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb0780512b0 con 0x7fb078108f00 2024-05-27T07:21:21.717 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:21:21.726+0000 7fb06effd700 1 -- 172.21.15.52:0/3841978640 <== mon.0 v2:172.21.15.52:3300/0 4 ==== mgrmap(e 16) ==== 149652+0+0 (secure 0 0 0) 0x7fb06802a070 con 0x7fb078108f00 2024-05-27T07:21:21.717 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:21:21.727+0000 7fb06effd700 1 --2- 172.21.15.52:0/3841978640 >> [v2:172.21.15.52:6800/3513075805,v1:172.21.15.52:6801/3513075805] conn(0x7fb0640b2880 0x7fb0640b4d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:21:21.717 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:21:21.727+0000 7fb07dfba700 1 --2- 172.21.15.52:0/3841978640 >> [v2:172.21.15.52:6800/3513075805,v1:172.21.15.52:6801/3513075805] conn(0x7fb0640b2880 0x7fb0640b4d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:21:21.717 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:21:21.727+0000 7fb06effd700 1 -- 172.21.15.52:0/3841978640 <== mon.0 v2:172.21.15.52:3300/0 5 ==== osd_map(72..72 src has 1..72) ==== 7127+0+0 (secure 0 0 0) 0x7fb068022e00 con 0x7fb078108f00 2024-05-27T07:21:21.718 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:21:21.728+0000 7fb07dfba700 1 --2- 172.21.15.52:0/3841978640 >> [v2:172.21.15.52:6800/3513075805,v1:172.21.15.52:6801/3513075805] conn(0x7fb0640b2880 0x7fb0640b4d90 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7fb074005950 tx=0x7fb07400b410 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:21:21.721 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:21:21.731+0000 7fb06effd700 1 -- 172.21.15.52:0/3841978640 <== mon.0 v2:172.21.15.52:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191963 (secure 0 0 0) 0x7fb068031080 con 0x7fb078108f00 2024-05-27T07:21:21.988 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:21:21.998+0000 7fb08021e700 1 -- 172.21.15.52:0/3841978640 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fb07802e5e0 con 0x7fb078108f00 2024-05-27T07:21:21.988 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:21:21.998+0000 7fb06effd700 1 -- 172.21.15.52:0/3841978640 <== mon.0 v2:172.21.15.52:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 8 v8) ==== 93+0+6355 (secure 0 0 0) 0x7fb068013070 con 0x7fb078108f00 2024-05-27T07:21:21.989 INFO:teuthology.orchestra.run.smithi052.stdout: 2024-05-27T07:21:21.989 INFO:teuthology.orchestra.run.smithi052.stdout:{"epoch":8,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14556,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.52:6835/2584733778","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.52:6834","nonce":2584733778},{"type":"v1","addr":"172.21.15.52:6835","nonce":2584733778}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14568,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.52:6837/675445303","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.52:6836","nonce":675445303},{"type":"v1","addr":"172.21.15.52:6837","nonce":675445303}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":24433,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.173:6833/3013589220","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6832","nonce":3013589220},{"type":"v1","addr":"172.21.15.173:6833","nonce":3013589220}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24452,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.105:6833/2800678412","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6832","nonce":2800678412},{"type":"v1","addr":"172.21.15.105:6833","nonce":2800678412}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24460,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.173:6835/1619486437","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6834","nonce":1619486437},{"type":"v1","addr":"172.21.15.173:6835","nonce":1619486437}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":7},{"gid":24466,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.173:6837/1396363827","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6836","nonce":1396363827},{"type":"v1","addr":"172.21.15.173:6837","nonce":1396363827}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":8},{"gid":24473,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.105:6835/3298498384","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6834","nonce":3298498384},{"type":"v1","addr":"172.21.15.105:6835","nonce":3298498384}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6}],"filesystems":[]} 2024-05-27T07:21:21.991 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:21:22.001+0000 7fb08021e700 1 -- 172.21.15.52:0/3841978640 >> [v2:172.21.15.52:6800/3513075805,v1:172.21.15.52:6801/3513075805] conn(0x7fb0640b2880 msgr2=0x7fb0640b4d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:21:21.991 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:21:22.001+0000 7fb08021e700 1 --2- 172.21.15.52:0/3841978640 >> [v2:172.21.15.52:6800/3513075805,v1:172.21.15.52:6801/3513075805] conn(0x7fb0640b2880 0x7fb0640b4d90 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7fb074005950 tx=0x7fb07400b410 comp rx=0 tx=0).stop 2024-05-27T07:21:21.991 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:21:22.001+0000 7fb08021e700 1 -- 172.21.15.52:0/3841978640 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fb078108f00 msgr2=0x7fb078075ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:21:21.991 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:21:22.001+0000 7fb08021e700 1 --2- 172.21.15.52:0/3841978640 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fb078108f00 0x7fb078075ff0 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7fb068006010 tx=0x7fb0680047b0 comp rx=0 tx=0).stop 2024-05-27T07:21:21.991 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:21:22.001+0000 7fb08021e700 1 -- 172.21.15.52:0/3841978640 shutdown_connections 2024-05-27T07:21:21.991 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:21:22.001+0000 7fb08021e700 1 --2- 172.21.15.52:0/3841978640 >> [v2:172.21.15.52:6800/3513075805,v1:172.21.15.52:6801/3513075805] conn(0x7fb0640b2880 0x7fb0640b4d90 unknown :-1 s=CLOSED pgs=131 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:21:21.991 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:21:22.001+0000 7fb08021e700 1 --2- 172.21.15.52:0/3841978640 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fb078109850 0x7fb078076530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:21:21.992 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:21:22.001+0000 7fb08021e700 1 --2- 172.21.15.52:0/3841978640 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fb0781045d0 0x7fb078075ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:21:21.992 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:21:22.001+0000 7fb08021e700 1 --2- 172.21.15.52:0/3841978640 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fb078108f00 0x7fb078075ff0 unknown :-1 s=CLOSED pgs=169 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:21:21.992 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:21:22.001+0000 7fb08021e700 1 -- 172.21.15.52:0/3841978640 >> 172.21.15.52:0/3841978640 conn(0x7fb0780ffe60 msgr2=0x7fb078107150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:21:21.992 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:21:22.001+0000 7fb08021e700 1 -- 172.21.15.52:0/3841978640 shutdown_connections 2024-05-27T07:21:21.992 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:21:22.001+0000 7fb08021e700 1 -- 172.21.15.52:0/3841978640 wait complete. 2024-05-27T07:21:21.992 INFO:teuthology.orchestra.run.smithi052.stderr:dumped fsmap epoch 8 2024-05-27T07:21:22.308 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:21:22 smithi052 ceph-mon[31955]: Deploying daemon mds.h on smithi105 2024-05-27T07:21:22.308 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:21:22 smithi052 ceph-mon[31955]: mds.? [v2:172.21.15.173:6836/1396363827,v1:172.21.15.173:6837/1396363827] up:boot 2024-05-27T07:21:22.309 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:21:22 smithi052 ceph-mon[31955]: fsmap 7 up:standby 2024-05-27T07:21:22.309 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:21:22 smithi052 ceph-mon[31955]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2024-05-27T07:21:22.309 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:21:22 smithi052 ceph-mon[31955]: from='client.? 172.21.15.52:0/3841978640' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-27T07:21:22.352 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:21:22 smithi173 ceph-mon[37000]: Deploying daemon mds.h on smithi105 2024-05-27T07:21:22.352 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:21:22 smithi173 ceph-mon[37000]: mds.? [v2:172.21.15.173:6836/1396363827,v1:172.21.15.173:6837/1396363827] up:boot 2024-05-27T07:21:22.352 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:21:22 smithi173 ceph-mon[37000]: fsmap 7 up:standby 2024-05-27T07:21:22.352 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:21:22 smithi173 ceph-mon[37000]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2024-05-27T07:21:22.352 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:21:22 smithi173 ceph-mon[37000]: from='client.? 172.21.15.52:0/3841978640' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-27T07:21:22.440 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:21:22 smithi105 ceph-mon[35977]: Deploying daemon mds.h on smithi105 2024-05-27T07:21:22.440 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:21:22 smithi105 ceph-mon[35977]: mds.? [v2:172.21.15.173:6836/1396363827,v1:172.21.15.173:6837/1396363827] up:boot 2024-05-27T07:21:22.440 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:21:22 smithi105 ceph-mon[35977]: fsmap 7 up:standby 2024-05-27T07:21:22.441 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:21:22 smithi105 ceph-mon[35977]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2024-05-27T07:21:22.441 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:21:22 smithi105 ceph-mon[35977]: from='client.? 172.21.15.52:0/3841978640' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-27T07:21:23.190 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:21:23 smithi105 ceph-mon[35977]: pgmap v268: 1 pgs: 1 active+clean; 577 KiB data, 354 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-27T07:21:23.352 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:21:23 smithi173 ceph-mon[37000]: pgmap v268: 1 pgs: 1 active+clean; 577 KiB data, 354 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-27T07:21:23.479 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:21:23 smithi052 ceph-mon[31955]: pgmap v268: 1 pgs: 1 active+clean; 577 KiB data, 354 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-27T07:21:23.940 INFO:journalctl@ceph.mds.h.smithi105.stdout:May 27 07:21:23 smithi105 systemd[1]: Started Ceph mds.h for 73df2636-1bf8-11ef-bc9b-c7b262605968. 2024-05-27T07:21:24.634 DEBUG:teuthology.orchestra.run.smithi052:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:38f9d900c83c63799fbdbe61acc9a11b0d3554a6 shell --fsid 73df2636-1bf8-11ef-bc9b-c7b262605968 -- ceph fs dump --format=json 2024-05-27T07:21:24.900 INFO:teuthology.orchestra.run.smithi052.stderr:Inferring config /var/lib/ceph/73df2636-1bf8-11ef-bc9b-c7b262605968/mon.a/config 2024-05-27T07:21:24.980 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:21:24 smithi052 ceph-mon[31955]: pgmap v269: 1 pgs: 1 active+clean; 577 KiB data, 354 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-27T07:21:24.980 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:21:24 smithi052 ceph-mon[31955]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' 2024-05-27T07:21:24.980 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:21:24 smithi052 ceph-mon[31955]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' 2024-05-27T07:21:24.980 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:21:24 smithi052 ceph-mon[31955]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' 2024-05-27T07:21:24.980 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:21:24 smithi052 ceph-mon[31955]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.i", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-05-27T07:21:24.980 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:21:24 smithi052 ceph-mon[31955]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.i", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-05-27T07:21:24.980 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:21:24 smithi052 ceph-mon[31955]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T07:21:24.980 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:21:24 smithi052 ceph-mon[31955]: Deploying daemon mds.i on smithi173 2024-05-27T07:21:24.980 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:21:24 smithi052 ceph-mon[31955]: mds.? [v2:172.21.15.105:6836/191393149,v1:172.21.15.105:6837/191393149] up:boot 2024-05-27T07:21:24.980 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:21:24 smithi052 ceph-mon[31955]: fsmap 8 up:standby 2024-05-27T07:21:24.981 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:21:24 smithi052 ceph-mon[31955]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2024-05-27T07:21:25.102 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:21:24 smithi173 ceph-mon[37000]: pgmap v269: 1 pgs: 1 active+clean; 577 KiB data, 354 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-27T07:21:25.103 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:21:24 smithi173 ceph-mon[37000]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' 2024-05-27T07:21:25.103 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:21:24 smithi173 ceph-mon[37000]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' 2024-05-27T07:21:25.103 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:21:24 smithi173 ceph-mon[37000]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' 2024-05-27T07:21:25.103 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:21:24 smithi173 ceph-mon[37000]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.i", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-05-27T07:21:25.103 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:21:24 smithi173 ceph-mon[37000]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.i", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-05-27T07:21:25.103 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:21:24 smithi173 ceph-mon[37000]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T07:21:25.103 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:21:24 smithi173 ceph-mon[37000]: Deploying daemon mds.i on smithi173 2024-05-27T07:21:25.103 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:21:24 smithi173 ceph-mon[37000]: mds.? [v2:172.21.15.105:6836/191393149,v1:172.21.15.105:6837/191393149] up:boot 2024-05-27T07:21:25.103 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:21:24 smithi173 ceph-mon[37000]: fsmap 8 up:standby 2024-05-27T07:21:25.103 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:21:24 smithi173 ceph-mon[37000]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2024-05-27T07:21:25.190 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:21:24 smithi105 ceph-mon[35977]: pgmap v269: 1 pgs: 1 active+clean; 577 KiB data, 354 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-27T07:21:25.190 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:21:24 smithi105 ceph-mon[35977]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' 2024-05-27T07:21:25.190 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:21:24 smithi105 ceph-mon[35977]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' 2024-05-27T07:21:25.190 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:21:24 smithi105 ceph-mon[35977]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' 2024-05-27T07:21:25.190 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:21:24 smithi105 ceph-mon[35977]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.i", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-05-27T07:21:25.191 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:21:24 smithi105 ceph-mon[35977]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.i", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-05-27T07:21:25.191 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:21:24 smithi105 ceph-mon[35977]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T07:21:25.191 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:21:24 smithi105 ceph-mon[35977]: Deploying daemon mds.i on smithi173 2024-05-27T07:21:25.191 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:21:24 smithi105 ceph-mon[35977]: mds.? [v2:172.21.15.105:6836/191393149,v1:172.21.15.105:6837/191393149] up:boot 2024-05-27T07:21:25.191 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:21:24 smithi105 ceph-mon[35977]: fsmap 8 up:standby 2024-05-27T07:21:25.191 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:21:24 smithi105 ceph-mon[35977]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2024-05-27T07:21:25.852 INFO:journalctl@ceph.mds.i.smithi173.stdout:May 27 07:21:25 smithi173 systemd[1]: Starting Ceph mds.i for 73df2636-1bf8-11ef-bc9b-c7b262605968... 2024-05-27T07:21:26.225 INFO:journalctl@ceph.mds.i.smithi173.stdout:May 27 07:21:25 smithi173 podman[68338]: 2024-05-27 07:21:25.854992571 +0000 UTC m=+0.017743995 image pull e36ea4fc8a5807e8d2701370e350a0de3856538797f7fba64cfce7e344bd6799 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6 2024-05-27T07:21:26.225 INFO:journalctl@ceph.mds.i.smithi173.stdout:May 27 07:21:25 smithi173 podman[68338]: 2024-05-27 07:21:25.985593503 +0000 UTC m=+0.148344915 container create 6c1bde6811ef6919ac682726a379a32a04e27699f36ca1ade9e3d442c7d97415 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-i, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, ceph=True, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, org.label-schema.build-date=20240521, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, GIT_CLEAN=True, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, CEPH_POINT_RELEASE=) 2024-05-27T07:21:26.225 INFO:journalctl@ceph.mds.i.smithi173.stdout:May 27 07:21:26 smithi173 podman[68338]: 2024-05-27 07:21:26.199292665 +0000 UTC m=+0.362044074 container init 6c1bde6811ef6919ac682726a379a32a04e27699f36ca1ade9e3d442c7d97415 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-i, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, org.label-schema.build-date=20240521, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True) 2024-05-27T07:21:26.225 INFO:journalctl@ceph.mds.i.smithi173.stdout:May 27 07:21:26 smithi173 podman[68338]: 2024-05-27 07:21:26.202169756 +0000 UTC m=+0.364921184 container start 6c1bde6811ef6919ac682726a379a32a04e27699f36ca1ade9e3d442c7d97415 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-i, org.label-schema.schema-version=1.0, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, ceph=True, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, org.label-schema.build-date=20240521, io.buildah.version=1.33.5, GIT_CLEAN=True, org.label-schema.license=GPLv2, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux ) 2024-05-27T07:21:26.487 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:21:26.490+0000 7f29065bf700 1 -- 172.21.15.52:0/2316585111 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f2900107de0 msgr2=0x7f29001081f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:21:26.488 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:21:26.490+0000 7f29065bf700 1 --2- 172.21.15.52:0/2316585111 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f2900107de0 0x7f29001081f0 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7f28f40099c0 tx=0x7f28f4009cd0 comp rx=0 tx=0).stop 2024-05-27T07:21:26.488 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:21:26.490+0000 7f29065bf700 1 -- 172.21.15.52:0/2316585111 shutdown_connections 2024-05-27T07:21:26.488 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:21:26.490+0000 7f29065bf700 1 --2- 172.21.15.52:0/2316585111 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f2900108730 0x7f2900110090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:21:26.488 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:21:26.490+0000 7f29065bf700 1 --2- 172.21.15.52:0/2316585111 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f29001034b0 0x7f29001038c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:21:26.488 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:21:26.490+0000 7f29065bf700 1 --2- 172.21.15.52:0/2316585111 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f2900107de0 0x7f29001081f0 unknown :-1 s=CLOSED pgs=170 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:21:26.488 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:21:26.490+0000 7f29065bf700 1 -- 172.21.15.52:0/2316585111 >> 172.21.15.52:0/2316585111 conn(0x7f29000fedb0 msgr2=0x7f29001011d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:21:26.488 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:21:26.490+0000 7f29065bf700 1 -- 172.21.15.52:0/2316585111 shutdown_connections 2024-05-27T07:21:26.488 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:21:26.490+0000 7f29065bf700 1 -- 172.21.15.52:0/2316585111 wait complete. 2024-05-27T07:21:26.488 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:21:26.491+0000 7f29065bf700 1 Processor -- start 2024-05-27T07:21:26.488 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:21:26.491+0000 7f29065bf700 1 -- start start 2024-05-27T07:21:26.489 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:21:26.491+0000 7f29065bf700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f29001034b0 0x7f290019bc40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:21:26.489 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:21:26.491+0000 7f29065bf700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f2900107de0 0x7f290019c180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:21:26.489 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:21:26.491+0000 7f29065bf700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f2900108730 0x7f290019c700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:21:26.489 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:21:26.491+0000 7f29065bf700 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x7f2900113340 con 0x7f29001034b0 2024-05-27T07:21:26.489 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:21:26.491+0000 7f29065bf700 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f29001131c0 con 0x7f2900107de0 2024-05-27T07:21:26.489 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:21:26.491+0000 7f29065bf700 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f29001134c0 con 0x7f2900108730 2024-05-27T07:21:26.489 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:21:26.492+0000 7f28fffff700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f29001034b0 0x7f290019bc40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:21:26.489 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:21:26.492+0000 7f28fffff700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f29001034b0 0x7f290019bc40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.52:36506/0 (socket says 172.21.15.52:36506) 2024-05-27T07:21:26.489 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:21:26.492+0000 7f28fffff700 1 -- 172.21.15.52:0/2965886468 learned_addr learned my addr 172.21.15.52:0/2965886468 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-05-27T07:21:26.489 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:21:26.492+0000 7f2904b5c700 1 --2- 172.21.15.52:0/2965886468 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f2900108730 0x7f290019c700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:21:26.489 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:21:26.492+0000 7f28ff7fe700 1 --2- 172.21.15.52:0/2965886468 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f2900107de0 0x7f290019c180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:21:26.489 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:21:26.492+0000 7f28fffff700 1 -- 172.21.15.52:0/2965886468 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f2900108730 msgr2=0x7f290019c700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:21:26.490 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:21:26.492+0000 7f28fffff700 1 --2- 172.21.15.52:0/2965886468 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f2900108730 0x7f290019c700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:21:26.490 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:21:26.492+0000 7f28fffff700 1 -- 172.21.15.52:0/2965886468 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f2900107de0 msgr2=0x7f290019c180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:21:26.490 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:21:26.492+0000 7f28fffff700 1 --2- 172.21.15.52:0/2965886468 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f2900107de0 0x7f290019c180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:21:26.490 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:21:26.492+0000 7f28fffff700 1 -- 172.21.15.52:0/2965886468 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f29001a6fb0 con 0x7f29001034b0 2024-05-27T07:21:26.490 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:21:26.492+0000 7f2904b5c700 1 --2- 172.21.15.52:0/2965886468 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f2900108730 0x7f290019c700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-27T07:21:26.490 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:21:26.492+0000 7f28fffff700 1 --2- 172.21.15.52:0/2965886468 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f29001034b0 0x7f290019bc40 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7f28f000d950 tx=0x7f28f000dd10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:21:26.490 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:21:26.492+0000 7f28fd7fa700 1 -- 172.21.15.52:0/2965886468 <== mon.0 v2:172.21.15.52:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f28f0016070 con 0x7f29001034b0 2024-05-27T07:21:26.490 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:21:26.492+0000 7f29065bf700 1 -- 172.21.15.52:0/2965886468 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f29001a72a0 con 0x7f29001034b0 2024-05-27T07:21:26.490 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:21:26.492+0000 7f28fd7fa700 1 -- 172.21.15.52:0/2965886468 <== mon.0 v2:172.21.15.52:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f28f0010460 con 0x7f29001034b0 2024-05-27T07:21:26.490 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:21:26.492+0000 7f28fd7fa700 1 -- 172.21.15.52:0/2965886468 <== mon.0 v2:172.21.15.52:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f28f000f620 con 0x7f29001034b0 2024-05-27T07:21:26.490 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:21:26.492+0000 7f29065bf700 1 -- 172.21.15.52:0/2965886468 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f29001a7800 con 0x7f29001034b0 2024-05-27T07:21:26.490 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:21:26.494+0000 7f29065bf700 1 -- 172.21.15.52:0/2965886468 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f29000512b0 con 0x7f29001034b0 2024-05-27T07:21:26.490 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:21:26.495+0000 7f28fd7fa700 1 -- 172.21.15.52:0/2965886468 <== mon.0 v2:172.21.15.52:3300/0 4 ==== mgrmap(e 16) ==== 149652+0+0 (secure 0 0 0) 0x7f28f0019070 con 0x7f29001034b0 2024-05-27T07:21:26.491 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:21:26.496+0000 7f28fd7fa700 1 --2- 172.21.15.52:0/2965886468 >> [v2:172.21.15.52:6800/3513075805,v1:172.21.15.52:6801/3513075805] conn(0x7f28e80b2900 0x7f28e80b4e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:21:26.491 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:21:26.496+0000 7f28ff7fe700 1 --2- 172.21.15.52:0/2965886468 >> [v2:172.21.15.52:6800/3513075805,v1:172.21.15.52:6801/3513075805] conn(0x7f28e80b2900 0x7f28e80b4e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:21:26.491 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:21:26.497+0000 7f28fd7fa700 1 -- 172.21.15.52:0/2965886468 <== mon.0 v2:172.21.15.52:3300/0 5 ==== osd_map(72..72 src has 1..72) ==== 7127+0+0 (secure 0 0 0) 0x7f28f00e19b0 con 0x7f29001034b0 2024-05-27T07:21:26.491 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:21:26.497+0000 7f28ff7fe700 1 --2- 172.21.15.52:0/2965886468 >> [v2:172.21.15.52:6800/3513075805,v1:172.21.15.52:6801/3513075805] conn(0x7f28e80b2900 0x7f28e80b4e10 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7f28f4000c00 tx=0x7f28f401a040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:21:26.492 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:21:26.501+0000 7f28fd7fa700 1 -- 172.21.15.52:0/2965886468 <== mon.0 v2:172.21.15.52:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191963 (secure 0 0 0) 0x7f28f00a9b70 con 0x7f29001034b0 2024-05-27T07:21:26.602 INFO:journalctl@ceph.mds.i.smithi173.stdout:May 27 07:21:26 smithi173 ceph-mds[68357]: -- 172.21.15.173:0/3910106499 <== mon.1 v2:172.21.15.173:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55adf5053a40 con 0x55adf509b180 2024-05-27T07:21:26.602 INFO:journalctl@ceph.mds.i.smithi173.stdout:May 27 07:21:26 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-i[68353]: starting mds.i at 2024-05-27T07:21:26.602 INFO:journalctl@ceph.mds.i.smithi173.stdout:May 27 07:21:26 smithi173 bash[68338]: 6c1bde6811ef6919ac682726a379a32a04e27699f36ca1ade9e3d442c7d97415 2024-05-27T07:21:26.602 INFO:journalctl@ceph.mds.i.smithi173.stdout:May 27 07:21:26 smithi173 systemd[1]: Started Ceph mds.i for 73df2636-1bf8-11ef-bc9b-c7b262605968. 2024-05-27T07:21:26.764 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:21:26.773+0000 7f29065bf700 1 -- 172.21.15.52:0/2965886468 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f290002e7a0 con 0x7f29001034b0 2024-05-27T07:21:26.764 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:21:26 smithi052 ceph-mon[31955]: pgmap v270: 1 pgs: 1 active+clean; 577 KiB data, 354 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-27T07:21:26.764 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:21:26 smithi052 ceph-mon[31955]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' 2024-05-27T07:21:26.764 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:21:26 smithi052 ceph-mon[31955]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' 2024-05-27T07:21:26.764 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:21:26 smithi052 ceph-mon[31955]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' 2024-05-27T07:21:26.765 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:21:26 smithi052 ceph-mon[31955]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.g", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-05-27T07:21:26.765 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:21:26 smithi052 ceph-mon[31955]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.g", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-05-27T07:21:26.765 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:21:26 smithi052 ceph-mon[31955]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T07:21:26.765 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:21:26.773+0000 7f28fd7fa700 1 -- 172.21.15.52:0/2965886468 <== mon.0 v2:172.21.15.52:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 10 v10) ==== 94+0+8007 (secure 0 0 0) 0x7f28f00a9470 con 0x7f29001034b0 2024-05-27T07:21:26.765 INFO:teuthology.orchestra.run.smithi052.stdout: 2024-05-27T07:21:26.766 INFO:teuthology.orchestra.run.smithi052.stdout:{"epoch":10,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14556,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.52:6835/2584733778","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.52:6834","nonce":2584733778},{"type":"v1","addr":"172.21.15.52:6835","nonce":2584733778}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14568,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.52:6837/675445303","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.52:6836","nonce":675445303},{"type":"v1","addr":"172.21.15.52:6837","nonce":675445303}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":24433,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.173:6833/3013589220","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6832","nonce":3013589220},{"type":"v1","addr":"172.21.15.173:6833","nonce":3013589220}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24452,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.105:6833/2800678412","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6832","nonce":2800678412},{"type":"v1","addr":"172.21.15.105:6833","nonce":2800678412}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24460,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.173:6835/1619486437","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6834","nonce":1619486437},{"type":"v1","addr":"172.21.15.173:6835","nonce":1619486437}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":7},{"gid":24466,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.173:6837/1396363827","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6836","nonce":1396363827},{"type":"v1","addr":"172.21.15.173:6837","nonce":1396363827}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":8},{"gid":24473,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.105:6835/3298498384","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6834","nonce":3298498384},{"type":"v1","addr":"172.21.15.105:6835","nonce":3298498384}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6},{"gid":24475,"name":"i","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.173:6839/564778293","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6838","nonce":564778293},{"type":"v1","addr":"172.21.15.173:6839","nonce":564778293}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":10},{"gid":24497,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.105:6837/191393149","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6836","nonce":191393149},{"type":"v1","addr":"172.21.15.105:6837","nonce":191393149}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":9}],"filesystems":[]} 2024-05-27T07:21:26.767 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:21:26.776+0000 7f29065bf700 1 -- 172.21.15.52:0/2965886468 >> [v2:172.21.15.52:6800/3513075805,v1:172.21.15.52:6801/3513075805] conn(0x7f28e80b2900 msgr2=0x7f28e80b4e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:21:26.767 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:21:26.776+0000 7f29065bf700 1 --2- 172.21.15.52:0/2965886468 >> [v2:172.21.15.52:6800/3513075805,v1:172.21.15.52:6801/3513075805] conn(0x7f28e80b2900 0x7f28e80b4e10 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7f28f4000c00 tx=0x7f28f401a040 comp rx=0 tx=0).stop 2024-05-27T07:21:26.767 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:21:26.776+0000 7f29065bf700 1 -- 172.21.15.52:0/2965886468 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f29001034b0 msgr2=0x7f290019bc40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:21:26.767 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:21:26.776+0000 7f29065bf700 1 --2- 172.21.15.52:0/2965886468 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f29001034b0 0x7f290019bc40 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7f28f000d950 tx=0x7f28f000dd10 comp rx=0 tx=0).stop 2024-05-27T07:21:26.768 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:21:26.776+0000 7f29065bf700 1 -- 172.21.15.52:0/2965886468 shutdown_connections 2024-05-27T07:21:26.768 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:21:26.776+0000 7f29065bf700 1 --2- 172.21.15.52:0/2965886468 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f2900108730 0x7f290019c700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:21:26.768 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:21:26.776+0000 7f29065bf700 1 --2- 172.21.15.52:0/2965886468 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f29001034b0 0x7f290019bc40 unknown :-1 s=CLOSED pgs=171 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:21:26.768 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:21:26.776+0000 7f29065bf700 1 --2- 172.21.15.52:0/2965886468 >> [v2:172.21.15.52:6800/3513075805,v1:172.21.15.52:6801/3513075805] conn(0x7f28e80b2900 0x7f28e80b4e10 unknown :-1 s=CLOSED pgs=133 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:21:26.768 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:21:26.776+0000 7f29065bf700 1 --2- 172.21.15.52:0/2965886468 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f2900107de0 0x7f290019c180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:21:26.768 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:21:26.776+0000 7f29065bf700 1 -- 172.21.15.52:0/2965886468 >> 172.21.15.52:0/2965886468 conn(0x7f29000fedb0 msgr2=0x7f2900109690 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:21:26.768 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:21:26.776+0000 7f29065bf700 1 -- 172.21.15.52:0/2965886468 shutdown_connections 2024-05-27T07:21:26.768 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:21:26.776+0000 7f29065bf700 1 -- 172.21.15.52:0/2965886468 wait complete. 2024-05-27T07:21:26.768 INFO:teuthology.orchestra.run.smithi052.stderr:dumped fsmap epoch 10 2024-05-27T07:21:27.102 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:21:26 smithi173 ceph-mon[37000]: pgmap v270: 1 pgs: 1 active+clean; 577 KiB data, 354 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-27T07:21:27.102 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:21:26 smithi173 ceph-mon[37000]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' 2024-05-27T07:21:27.102 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:21:26 smithi173 ceph-mon[37000]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' 2024-05-27T07:21:27.102 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:21:26 smithi173 ceph-mon[37000]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' 2024-05-27T07:21:27.102 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:21:26 smithi173 ceph-mon[37000]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.g", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-05-27T07:21:27.102 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:21:26 smithi173 ceph-mon[37000]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.g", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-05-27T07:21:27.103 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:21:26 smithi173 ceph-mon[37000]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T07:21:27.190 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:21:26 smithi105 ceph-mon[35977]: pgmap v270: 1 pgs: 1 active+clean; 577 KiB data, 354 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-27T07:21:27.190 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:21:26 smithi105 ceph-mon[35977]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' 2024-05-27T07:21:27.190 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:21:26 smithi105 ceph-mon[35977]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' 2024-05-27T07:21:27.190 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:21:26 smithi105 ceph-mon[35977]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' 2024-05-27T07:21:27.190 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:21:26 smithi105 ceph-mon[35977]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.g", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-05-27T07:21:27.191 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:21:26 smithi105 ceph-mon[35977]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.g", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-05-27T07:21:27.191 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:21:26 smithi105 ceph-mon[35977]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T07:21:27.979 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:21:27 smithi052 ceph-mon[31955]: Deploying daemon mds.g on smithi052 2024-05-27T07:21:27.979 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:21:27 smithi052 ceph-mon[31955]: mds.? [v2:172.21.15.173:6838/564778293,v1:172.21.15.173:6839/564778293] up:boot 2024-05-27T07:21:27.979 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:21:27 smithi052 ceph-mon[31955]: fsmap 9 up:standby 2024-05-27T07:21:27.980 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:21:27 smithi052 ceph-mon[31955]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "i"}]: dispatch 2024-05-27T07:21:27.980 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:21:27 smithi052 ceph-mon[31955]: from='client.? 172.21.15.52:0/2965886468' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-27T07:21:28.102 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:21:27 smithi173 ceph-mon[37000]: Deploying daemon mds.g on smithi052 2024-05-27T07:21:28.102 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:21:27 smithi173 ceph-mon[37000]: mds.? [v2:172.21.15.173:6838/564778293,v1:172.21.15.173:6839/564778293] up:boot 2024-05-27T07:21:28.102 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:21:27 smithi173 ceph-mon[37000]: fsmap 9 up:standby 2024-05-27T07:21:28.102 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:21:27 smithi173 ceph-mon[37000]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "i"}]: dispatch 2024-05-27T07:21:28.102 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:21:27 smithi173 ceph-mon[37000]: from='client.? 172.21.15.52:0/2965886468' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-27T07:21:28.190 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:21:27 smithi105 ceph-mon[35977]: Deploying daemon mds.g on smithi052 2024-05-27T07:21:28.190 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:21:27 smithi105 ceph-mon[35977]: mds.? [v2:172.21.15.173:6838/564778293,v1:172.21.15.173:6839/564778293] up:boot 2024-05-27T07:21:28.190 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:21:27 smithi105 ceph-mon[35977]: fsmap 9 up:standby 2024-05-27T07:21:28.190 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:21:27 smithi105 ceph-mon[35977]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "i"}]: dispatch 2024-05-27T07:21:28.190 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:21:27 smithi105 ceph-mon[35977]: from='client.? 172.21.15.52:0/2965886468' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-27T07:21:28.879 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:21:28 smithi052 ceph-mon[31955]: pgmap v271: 1 pgs: 1 active+clean; 577 KiB data, 354 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-27T07:21:29.102 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:21:28 smithi173 ceph-mon[37000]: pgmap v271: 1 pgs: 1 active+clean; 577 KiB data, 354 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-27T07:21:29.190 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:21:28 smithi105 ceph-mon[35977]: pgmap v271: 1 pgs: 1 active+clean; 577 KiB data, 354 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-27T07:21:29.230 INFO:journalctl@ceph.mds.g.smithi052.stdout:May 27 07:21:29 smithi052 systemd[1]: Starting Ceph mds.g for 73df2636-1bf8-11ef-bc9b-c7b262605968... 2024-05-27T07:21:29.429 DEBUG:teuthology.orchestra.run.smithi052:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:38f9d900c83c63799fbdbe61acc9a11b0d3554a6 shell --fsid 73df2636-1bf8-11ef-bc9b-c7b262605968 -- ceph fs dump --format=json 2024-05-27T07:21:29.552 INFO:journalctl@ceph.mds.g.smithi052.stdout:May 27 07:21:29 smithi052 podman[73952]: 2024-05-27 07:21:29.207534376 +0000 UTC m=+0.017680109 image pull e36ea4fc8a5807e8d2701370e350a0de3856538797f7fba64cfce7e344bd6799 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6 2024-05-27T07:21:29.552 INFO:journalctl@ceph.mds.g.smithi052.stdout:May 27 07:21:29 smithi052 podman[73952]: 2024-05-27 07:21:29.312169454 +0000 UTC m=+0.122315172 container create e7a40afa0af4a8507385ba0e0c7d06747373a30d90fb7a16b4533f563f7fe719 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-g, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240521, GIT_CLEAN=True, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0) 2024-05-27T07:21:29.552 INFO:journalctl@ceph.mds.g.smithi052.stdout:May 27 07:21:29 smithi052 podman[73952]: 2024-05-27 07:21:29.523412819 +0000 UTC m=+0.333558550 container init e7a40afa0af4a8507385ba0e0c7d06747373a30d90fb7a16b4533f563f7fe719 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-g, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, ceph=True, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, GIT_BRANCH=HEAD, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, org.label-schema.build-date=20240521, org.label-schema.name=CentOS Stream 8 Base Image) 2024-05-27T07:21:29.552 INFO:journalctl@ceph.mds.g.smithi052.stdout:May 27 07:21:29 smithi052 podman[73952]: 2024-05-27 07:21:29.526789304 +0000 UTC m=+0.336935029 container start e7a40afa0af4a8507385ba0e0c7d06747373a30d90fb7a16b4533f563f7fe719 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-g, org.label-schema.vendor=CentOS, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, org.label-schema.build-date=20240521, CEPH_POINT_RELEASE=, ceph=True) 2024-05-27T07:21:29.697 INFO:teuthology.orchestra.run.smithi052.stderr:Inferring config /var/lib/ceph/73df2636-1bf8-11ef-bc9b-c7b262605968/mon.a/config 2024-05-27T07:21:29.820 INFO:journalctl@ceph.mds.g.smithi052.stdout:May 27 07:21:29 smithi052 ceph-mds[73989]: -- 172.21.15.52:0/434339259 <== mon.0 v2:172.21.15.52:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55a765647a40 con 0x55a76568f600 2024-05-27T07:21:29.821 INFO:journalctl@ceph.mds.g.smithi052.stdout:May 27 07:21:29 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-g[73967]: starting mds.g at 2024-05-27T07:21:29.821 INFO:journalctl@ceph.mds.g.smithi052.stdout:May 27 07:21:29 smithi052 bash[73952]: e7a40afa0af4a8507385ba0e0c7d06747373a30d90fb7a16b4533f563f7fe719 2024-05-27T07:21:29.821 INFO:journalctl@ceph.mds.g.smithi052.stdout:May 27 07:21:29 smithi052 systemd[1]: Started Ceph mds.g for 73df2636-1bf8-11ef-bc9b-c7b262605968. 2024-05-27T07:21:30.821 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:21:30 smithi052 ceph-mon[31955]: pgmap v272: 1 pgs: 1 active+clean; 577 KiB data, 354 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-27T07:21:30.821 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:21:30 smithi052 ceph-mon[31955]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' 2024-05-27T07:21:30.821 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:21:30 smithi052 ceph-mon[31955]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' 2024-05-27T07:21:30.821 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:21:30 smithi052 ceph-mon[31955]: mds.? [v2:172.21.15.52:6838/4258549559,v1:172.21.15.52:6839/4258549559] up:boot 2024-05-27T07:21:30.821 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:21:30 smithi052 ceph-mon[31955]: fsmap 10 up:standby 2024-05-27T07:21:30.822 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:21:30 smithi052 ceph-mon[31955]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2024-05-27T07:21:30.822 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:21:30 smithi052 ceph-mon[31955]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' 2024-05-27T07:21:30.822 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:21:30 smithi052 ceph-mon[31955]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.a", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-05-27T07:21:30.822 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:21:30 smithi052 ceph-mon[31955]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.a", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-05-27T07:21:30.822 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:21:30 smithi052 ceph-mon[31955]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T07:21:30.822 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:21:30 smithi052 ceph-mon[31955]: Deploying daemon mds.a on smithi052 2024-05-27T07:21:30.940 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:21:30 smithi105 ceph-mon[35977]: pgmap v272: 1 pgs: 1 active+clean; 577 KiB data, 354 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-27T07:21:30.940 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:21:30 smithi105 ceph-mon[35977]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' 2024-05-27T07:21:30.940 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:21:30 smithi105 ceph-mon[35977]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' 2024-05-27T07:21:30.940 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:21:30 smithi105 ceph-mon[35977]: mds.? [v2:172.21.15.52:6838/4258549559,v1:172.21.15.52:6839/4258549559] up:boot 2024-05-27T07:21:30.940 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:21:30 smithi105 ceph-mon[35977]: fsmap 10 up:standby 2024-05-27T07:21:30.940 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:21:30 smithi105 ceph-mon[35977]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2024-05-27T07:21:30.941 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:21:30 smithi105 ceph-mon[35977]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' 2024-05-27T07:21:30.941 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:21:30 smithi105 ceph-mon[35977]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.a", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-05-27T07:21:30.941 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:21:30 smithi105 ceph-mon[35977]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.a", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-05-27T07:21:30.941 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:21:30 smithi105 ceph-mon[35977]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T07:21:30.941 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:21:30 smithi105 ceph-mon[35977]: Deploying daemon mds.a on smithi052 2024-05-27T07:21:31.102 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:21:30 smithi173 ceph-mon[37000]: pgmap v272: 1 pgs: 1 active+clean; 577 KiB data, 354 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-27T07:21:31.103 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:21:30 smithi173 ceph-mon[37000]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' 2024-05-27T07:21:31.103 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:21:30 smithi173 ceph-mon[37000]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' 2024-05-27T07:21:31.103 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:21:30 smithi173 ceph-mon[37000]: mds.? [v2:172.21.15.52:6838/4258549559,v1:172.21.15.52:6839/4258549559] up:boot 2024-05-27T07:21:31.103 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:21:30 smithi173 ceph-mon[37000]: fsmap 10 up:standby 2024-05-27T07:21:31.103 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:21:30 smithi173 ceph-mon[37000]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2024-05-27T07:21:31.103 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:21:30 smithi173 ceph-mon[37000]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' 2024-05-27T07:21:31.103 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:21:30 smithi173 ceph-mon[37000]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.a", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-05-27T07:21:31.103 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:21:30 smithi173 ceph-mon[37000]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.a", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-05-27T07:21:31.103 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:21:30 smithi173 ceph-mon[37000]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T07:21:31.103 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:21:30 smithi173 ceph-mon[37000]: Deploying daemon mds.a on smithi052 2024-05-27T07:21:31.569 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:21:31.368+0000 7f90007dc700 1 -- 172.21.15.52:0/2652513518 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f8ff8108f00 msgr2=0x7f8ff8109310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:21:31.569 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:21:31.368+0000 7f90007dc700 1 --2- 172.21.15.52:0/2652513518 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f8ff8108f00 0x7f8ff8109310 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7f8fec0099c0 tx=0x7f8fec009cd0 comp rx=0 tx=0).stop 2024-05-27T07:21:31.569 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:21:31.368+0000 7f90007dc700 1 -- 172.21.15.52:0/2652513518 shutdown_connections 2024-05-27T07:21:31.569 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:21:31.368+0000 7f90007dc700 1 --2- 172.21.15.52:0/2652513518 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f8ff8109850 0x7f8ff81111b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:21:31.569 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:21:31.368+0000 7f90007dc700 1 --2- 172.21.15.52:0/2652513518 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f8ff81045d0 0x7f8ff81049e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:21:31.570 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:21:31.368+0000 7f90007dc700 1 --2- 172.21.15.52:0/2652513518 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f8ff8108f00 0x7f8ff8109310 unknown :-1 s=CLOSED pgs=174 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:21:31.570 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:21:31.368+0000 7f90007dc700 1 -- 172.21.15.52:0/2652513518 >> 172.21.15.52:0/2652513518 conn(0x7f8ff80ffee0 msgr2=0x7f8ff8102330 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:21:31.570 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:21:31.368+0000 7f90007dc700 1 -- 172.21.15.52:0/2652513518 shutdown_connections 2024-05-27T07:21:31.570 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:21:31.368+0000 7f90007dc700 1 -- 172.21.15.52:0/2652513518 wait complete. 2024-05-27T07:21:31.570 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:21:31.369+0000 7f90007dc700 1 Processor -- start 2024-05-27T07:21:31.570 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:21:31.369+0000 7f90007dc700 1 -- start start 2024-05-27T07:21:31.570 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:21:31.369+0000 7f90007dc700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f8ff81045d0 0x7f8ff819aeb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:21:31.570 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:21:31.369+0000 7f90007dc700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f8ff8108f00 0x7f8ff819b450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:21:31.570 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:21:31.369+0000 7f90007dc700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f8ff8109850 0x7f8ff819f6e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:21:31.570 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:21:31.369+0000 7f90007dc700 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x7f8ff8114520 con 0x7f8ff8109850 2024-05-27T07:21:31.570 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:21:31.369+0000 7f90007dc700 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f8ff81143a0 con 0x7f8ff8108f00 2024-05-27T07:21:31.571 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:21:31.369+0000 7f90007dc700 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f8ff81146a0 con 0x7f8ff81045d0 2024-05-27T07:21:31.571 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:21:31.370+0000 7f8ffed79700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f8ff8109850 0x7f8ff819f6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:21:31.571 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:21:31.370+0000 7f8ffed79700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f8ff8109850 0x7f8ff819f6e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.52:40448/0 (socket says 172.21.15.52:40448) 2024-05-27T07:21:31.571 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:21:31.370+0000 7f8ffed79700 1 -- 172.21.15.52:0/3562714883 learned_addr learned my addr 172.21.15.52:0/3562714883 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-05-27T07:21:31.571 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:21:31.370+0000 7f8ffe578700 1 --2- 172.21.15.52:0/3562714883 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f8ff81045d0 0x7f8ff819aeb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:21:31.571 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:21:31.370+0000 7f8ffdd77700 1 --2- 172.21.15.52:0/3562714883 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f8ff8108f00 0x7f8ff819b450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:21:31.571 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:21:31.370+0000 7f8ffed79700 1 -- 172.21.15.52:0/3562714883 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f8ff81045d0 msgr2=0x7f8ff819aeb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:21:31.571 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:21:31.370+0000 7f8ffed79700 1 --2- 172.21.15.52:0/3562714883 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f8ff81045d0 0x7f8ff819aeb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:21:31.572 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:21:31.370+0000 7f8ffed79700 1 -- 172.21.15.52:0/3562714883 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f8ff8108f00 msgr2=0x7f8ff819b450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:21:31.572 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:21:31.370+0000 7f8ffed79700 1 --2- 172.21.15.52:0/3562714883 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f8ff8108f00 0x7f8ff819b450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:21:31.572 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:21:31.370+0000 7f8ffed79700 1 -- 172.21.15.52:0/3562714883 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8ff819fd80 con 0x7f8ff8109850 2024-05-27T07:21:31.572 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:21:31.370+0000 7f8ffed79700 1 --2- 172.21.15.52:0/3562714883 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f8ff8109850 0x7f8ff819f6e0 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7f8ff000b550 tx=0x7f8ff000b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:21:31.572 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:21:31.370+0000 7f8ff77fe700 1 -- 172.21.15.52:0/3562714883 <== mon.0 v2:172.21.15.52:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8ff0015020 con 0x7f8ff8109850 2024-05-27T07:21:31.572 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:21:31.370+0000 7f90007dc700 1 -- 172.21.15.52:0/3562714883 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8ff81a0070 con 0x7f8ff8109850 2024-05-27T07:21:31.572 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:21:31.370+0000 7f8ff77fe700 1 -- 172.21.15.52:0/3562714883 <== mon.0 v2:172.21.15.52:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f8ff0011ce0 con 0x7f8ff8109850 2024-05-27T07:21:31.572 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:21:31.370+0000 7f8ff77fe700 1 -- 172.21.15.52:0/3562714883 <== mon.0 v2:172.21.15.52:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8ff0010d40 con 0x7f8ff8109850 2024-05-27T07:21:31.572 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:21:31.370+0000 7f90007dc700 1 -- 172.21.15.52:0/3562714883 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8ff81a7990 con 0x7f8ff8109850 2024-05-27T07:21:31.572 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:21:31.372+0000 7f90007dc700 1 -- 172.21.15.52:0/3562714883 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8ff80512b0 con 0x7f8ff8109850 2024-05-27T07:21:31.572 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:21:31.373+0000 7f8ff77fe700 1 -- 172.21.15.52:0/3562714883 <== mon.0 v2:172.21.15.52:3300/0 4 ==== mgrmap(e 16) ==== 149652+0+0 (secure 0 0 0) 0x7f8ff0018020 con 0x7f8ff8109850 2024-05-27T07:21:31.572 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:21:31.374+0000 7f8ff77fe700 1 --2- 172.21.15.52:0/3562714883 >> [v2:172.21.15.52:6800/3513075805,v1:172.21.15.52:6801/3513075805] conn(0x7f8fe00b2880 0x7f8fe00b4d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:21:31.572 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:21:31.374+0000 7f8ffe578700 1 --2- 172.21.15.52:0/3562714883 >> [v2:172.21.15.52:6800/3513075805,v1:172.21.15.52:6801/3513075805] conn(0x7f8fe00b2880 0x7f8fe00b4d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:21:31.573 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:21:31.374+0000 7f8ff77fe700 1 -- 172.21.15.52:0/3562714883 <== mon.0 v2:172.21.15.52:3300/0 5 ==== osd_map(72..72 src has 1..72) ==== 7127+0+0 (secure 0 0 0) 0x7f8ff00e15a0 con 0x7f8ff8109850 2024-05-27T07:21:31.573 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:21:31.374+0000 7f8ffe578700 1 --2- 172.21.15.52:0/3562714883 >> [v2:172.21.15.52:6800/3513075805,v1:172.21.15.52:6801/3513075805] conn(0x7f8fe00b2880 0x7f8fe00b4d90 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7f8fe8009d10 tx=0x7f8fe8009040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:21:31.573 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:21:31.379+0000 7f8ff77fe700 1 -- 172.21.15.52:0/3562714883 <== mon.0 v2:172.21.15.52:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191963 (secure 0 0 0) 0x7f8ff001f020 con 0x7f8ff8109850 2024-05-27T07:21:31.643 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:21:31.652+0000 7f90007dc700 1 -- 172.21.15.52:0/3562714883 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f8ff819c210 con 0x7f8ff8109850 2024-05-27T07:21:31.643 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:21:31.653+0000 7f8ff77fe700 1 -- 172.21.15.52:0/3562714883 <== mon.0 v2:172.21.15.52:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 11 v11) ==== 94+0+8833 (secure 0 0 0) 0x7f8ff00a9740 con 0x7f8ff8109850 2024-05-27T07:21:31.644 INFO:teuthology.orchestra.run.smithi052.stdout: 2024-05-27T07:21:31.644 INFO:teuthology.orchestra.run.smithi052.stdout:{"epoch":11,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14556,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.52:6835/2584733778","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.52:6834","nonce":2584733778},{"type":"v1","addr":"172.21.15.52:6835","nonce":2584733778}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14568,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.52:6837/675445303","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.52:6836","nonce":675445303},{"type":"v1","addr":"172.21.15.52:6837","nonce":675445303}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14619,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.52:6839/4258549559","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.52:6838","nonce":4258549559},{"type":"v1","addr":"172.21.15.52:6839","nonce":4258549559}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":24433,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.173:6833/3013589220","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6832","nonce":3013589220},{"type":"v1","addr":"172.21.15.173:6833","nonce":3013589220}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24452,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.105:6833/2800678412","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6832","nonce":2800678412},{"type":"v1","addr":"172.21.15.105:6833","nonce":2800678412}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24460,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.173:6835/1619486437","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6834","nonce":1619486437},{"type":"v1","addr":"172.21.15.173:6835","nonce":1619486437}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":7},{"gid":24466,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.173:6837/1396363827","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6836","nonce":1396363827},{"type":"v1","addr":"172.21.15.173:6837","nonce":1396363827}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":8},{"gid":24473,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.105:6835/3298498384","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6834","nonce":3298498384},{"type":"v1","addr":"172.21.15.105:6835","nonce":3298498384}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6},{"gid":24475,"name":"i","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.173:6839/564778293","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6838","nonce":564778293},{"type":"v1","addr":"172.21.15.173:6839","nonce":564778293}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":10},{"gid":24497,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.105:6837/191393149","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6836","nonce":191393149},{"type":"v1","addr":"172.21.15.105:6837","nonce":191393149}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":9}],"filesystems":[]} 2024-05-27T07:21:31.646 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:21:31.655+0000 7f90007dc700 1 -- 172.21.15.52:0/3562714883 >> [v2:172.21.15.52:6800/3513075805,v1:172.21.15.52:6801/3513075805] conn(0x7f8fe00b2880 msgr2=0x7f8fe00b4d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:21:31.646 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:21:31.655+0000 7f90007dc700 1 --2- 172.21.15.52:0/3562714883 >> [v2:172.21.15.52:6800/3513075805,v1:172.21.15.52:6801/3513075805] conn(0x7f8fe00b2880 0x7f8fe00b4d90 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7f8fe8009d10 tx=0x7f8fe8009040 comp rx=0 tx=0).stop 2024-05-27T07:21:31.646 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:21:31.655+0000 7f90007dc700 1 -- 172.21.15.52:0/3562714883 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f8ff8109850 msgr2=0x7f8ff819f6e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:21:31.646 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:21:31.655+0000 7f90007dc700 1 --2- 172.21.15.52:0/3562714883 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f8ff8109850 0x7f8ff819f6e0 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7f8ff000b550 tx=0x7f8ff000b910 comp rx=0 tx=0).stop 2024-05-27T07:21:31.646 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:21:31.655+0000 7f90007dc700 1 -- 172.21.15.52:0/3562714883 shutdown_connections 2024-05-27T07:21:31.646 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:21:31.655+0000 7f90007dc700 1 --2- 172.21.15.52:0/3562714883 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f8ff8109850 0x7f8ff819f6e0 unknown :-1 s=CLOSED pgs=175 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:21:31.646 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:21:31.655+0000 7f90007dc700 1 --2- 172.21.15.52:0/3562714883 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f8ff81045d0 0x7f8ff819aeb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:21:31.647 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:21:31.655+0000 7f90007dc700 1 --2- 172.21.15.52:0/3562714883 >> [v2:172.21.15.52:6800/3513075805,v1:172.21.15.52:6801/3513075805] conn(0x7f8fe00b2880 0x7f8fe00b4d90 unknown :-1 s=CLOSED pgs=136 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:21:31.647 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:21:31.655+0000 7f90007dc700 1 --2- 172.21.15.52:0/3562714883 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f8ff8108f00 0x7f8ff819b450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:21:31.647 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:21:31.655+0000 7f90007dc700 1 -- 172.21.15.52:0/3562714883 >> 172.21.15.52:0/3562714883 conn(0x7f8ff80ffee0 msgr2=0x7f8ff810a7b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:21:31.647 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:21:31.655+0000 7f90007dc700 1 -- 172.21.15.52:0/3562714883 shutdown_connections 2024-05-27T07:21:31.647 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:21:31.655+0000 7f90007dc700 1 -- 172.21.15.52:0/3562714883 wait complete. 2024-05-27T07:21:31.647 INFO:teuthology.orchestra.run.smithi052.stderr:dumped fsmap epoch 11 2024-05-27T07:21:31.940 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:21:31 smithi105 ceph-mon[35977]: from='client.? 172.21.15.52:0/3562714883' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-27T07:21:31.979 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:21:31 smithi052 ceph-mon[31955]: from='client.? 172.21.15.52:0/3562714883' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-27T07:21:32.102 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:21:31 smithi173 ceph-mon[37000]: from='client.? 172.21.15.52:0/3562714883' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-27T07:21:32.940 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:21:32 smithi105 ceph-mon[35977]: pgmap v273: 1 pgs: 1 active+clean; 577 KiB data, 354 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-27T07:21:32.974 INFO:journalctl@ceph.mds.a.smithi052.stdout:May 27 07:21:32 smithi052 systemd[1]: Starting Ceph mds.a for 73df2636-1bf8-11ef-bc9b-c7b262605968... 2024-05-27T07:21:32.975 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:21:32 smithi052 ceph-mon[31955]: pgmap v273: 1 pgs: 1 active+clean; 577 KiB data, 354 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-27T07:21:33.102 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:21:32 smithi173 ceph-mon[37000]: pgmap v273: 1 pgs: 1 active+clean; 577 KiB data, 354 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-27T07:21:33.579 INFO:journalctl@ceph.mds.a.smithi052.stdout:May 27 07:21:33 smithi052 podman[74413]: 2024-05-27 07:21:33.240654526 +0000 UTC m=+0.017708101 image pull e36ea4fc8a5807e8d2701370e350a0de3856538797f7fba64cfce7e344bd6799 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6 2024-05-27T07:21:33.579 INFO:journalctl@ceph.mds.a.smithi052.stdout:May 27 07:21:33 smithi052 podman[74413]: 2024-05-27 07:21:33.359380099 +0000 UTC m=+0.136433658 container create eda21eb53157d0b0ae4ffa197709e0eea08539346544e8fa541a514e350a59cf (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-a, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, org.label-schema.schema-version=1.0, ceph=True, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, org.label-schema.vendor=CentOS, GIT_CLEAN=True, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240521, GIT_BRANCH=HEAD) 2024-05-27T07:21:33.580 INFO:journalctl@ceph.mds.a.smithi052.stdout:May 27 07:21:33 smithi052 podman[74413]: 2024-05-27 07:21:33.586855277 +0000 UTC m=+0.363908838 container init eda21eb53157d0b0ae4ffa197709e0eea08539346544e8fa541a514e350a59cf (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-a, GIT_CLEAN=True, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240521, GIT_BRANCH=HEAD, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, org.label-schema.schema-version=1.0, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2) 2024-05-27T07:21:33.580 INFO:journalctl@ceph.mds.a.smithi052.stdout:May 27 07:21:33 smithi052 podman[74413]: 2024-05-27 07:21:33.589862 +0000 UTC m=+0.366915563 container start eda21eb53157d0b0ae4ffa197709e0eea08539346544e8fa541a514e350a59cf (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-a, org.label-schema.build-date=20240521, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_CLEAN=True, CEPH_POINT_RELEASE=, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux ) 2024-05-27T07:21:33.980 INFO:journalctl@ceph.mds.a.smithi052.stdout:May 27 07:21:33 smithi052 ceph-mds[74432]: -- 172.21.15.52:0/361969090 <== mon.0 v2:172.21.15.52:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x5629e1753a40 con 0x5629e179b180 2024-05-27T07:21:33.980 INFO:journalctl@ceph.mds.a.smithi052.stdout:May 27 07:21:33 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-a[74428]: starting mds.a at 2024-05-27T07:21:33.980 INFO:journalctl@ceph.mds.a.smithi052.stdout:May 27 07:21:33 smithi052 bash[74413]: eda21eb53157d0b0ae4ffa197709e0eea08539346544e8fa541a514e350a59cf 2024-05-27T07:21:33.980 INFO:journalctl@ceph.mds.a.smithi052.stdout:May 27 07:21:33 smithi052 systemd[1]: Started Ceph mds.a for 73df2636-1bf8-11ef-bc9b-c7b262605968. 2024-05-27T07:21:34.940 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:21:34 smithi105 ceph-mon[35977]: pgmap v274: 1 pgs: 1 active+clean; 577 KiB data, 354 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-27T07:21:34.940 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:21:34 smithi105 ceph-mon[35977]: mds.? [v2:172.21.15.52:6840/2878618188,v1:172.21.15.52:6841/2878618188] up:boot 2024-05-27T07:21:34.940 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:21:34 smithi105 ceph-mon[35977]: fsmap 11 up:standby 2024-05-27T07:21:34.941 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:21:34 smithi105 ceph-mon[35977]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "a"}]: dispatch 2024-05-27T07:21:34.941 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:21:34 smithi105 ceph-mon[35977]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' 2024-05-27T07:21:34.941 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:21:34 smithi105 ceph-mon[35977]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' 2024-05-27T07:21:34.941 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:21:34 smithi105 ceph-mon[35977]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' 2024-05-27T07:21:34.941 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:21:34 smithi105 ceph-mon[35977]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.b", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-05-27T07:21:34.941 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:21:34 smithi105 ceph-mon[35977]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.b", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-05-27T07:21:34.941 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:21:34 smithi105 ceph-mon[35977]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T07:21:34.941 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:21:34 smithi105 ceph-mon[35977]: Deploying daemon mds.b on smithi105 2024-05-27T07:21:34.980 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:21:34 smithi052 ceph-mon[31955]: pgmap v274: 1 pgs: 1 active+clean; 577 KiB data, 354 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-27T07:21:34.980 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:21:34 smithi052 ceph-mon[31955]: mds.? [v2:172.21.15.52:6840/2878618188,v1:172.21.15.52:6841/2878618188] up:boot 2024-05-27T07:21:34.980 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:21:34 smithi052 ceph-mon[31955]: fsmap 11 up:standby 2024-05-27T07:21:34.980 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:21:34 smithi052 ceph-mon[31955]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "a"}]: dispatch 2024-05-27T07:21:34.980 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:21:34 smithi052 ceph-mon[31955]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' 2024-05-27T07:21:34.980 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:21:34 smithi052 ceph-mon[31955]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' 2024-05-27T07:21:34.980 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:21:34 smithi052 ceph-mon[31955]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' 2024-05-27T07:21:34.980 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:21:34 smithi052 ceph-mon[31955]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.b", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-05-27T07:21:34.980 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:21:34 smithi052 ceph-mon[31955]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.b", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-05-27T07:21:34.980 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:21:34 smithi052 ceph-mon[31955]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T07:21:34.980 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:21:34 smithi052 ceph-mon[31955]: Deploying daemon mds.b on smithi105 2024-05-27T07:21:35.102 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:21:34 smithi173 ceph-mon[37000]: pgmap v274: 1 pgs: 1 active+clean; 577 KiB data, 354 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-27T07:21:35.102 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:21:34 smithi173 ceph-mon[37000]: mds.? [v2:172.21.15.52:6840/2878618188,v1:172.21.15.52:6841/2878618188] up:boot 2024-05-27T07:21:35.102 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:21:34 smithi173 ceph-mon[37000]: fsmap 11 up:standby 2024-05-27T07:21:35.102 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:21:34 smithi173 ceph-mon[37000]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "a"}]: dispatch 2024-05-27T07:21:35.102 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:21:34 smithi173 ceph-mon[37000]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' 2024-05-27T07:21:35.102 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:21:34 smithi173 ceph-mon[37000]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' 2024-05-27T07:21:35.103 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:21:34 smithi173 ceph-mon[37000]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' 2024-05-27T07:21:35.103 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:21:34 smithi173 ceph-mon[37000]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.b", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-05-27T07:21:35.103 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:21:34 smithi173 ceph-mon[37000]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.b", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-05-27T07:21:35.103 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:21:34 smithi173 ceph-mon[37000]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T07:21:35.103 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:21:34 smithi173 ceph-mon[37000]: Deploying daemon mds.b on smithi105 2024-05-27T07:21:35.166 DEBUG:teuthology.orchestra.run.smithi052:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:38f9d900c83c63799fbdbe61acc9a11b0d3554a6 shell --fsid 73df2636-1bf8-11ef-bc9b-c7b262605968 -- ceph fs dump --format=json 2024-05-27T07:21:35.433 INFO:teuthology.orchestra.run.smithi052.stderr:Inferring config /var/lib/ceph/73df2636-1bf8-11ef-bc9b-c7b262605968/mon.a/config 2024-05-27T07:21:36.190 INFO:journalctl@ceph.mds.b.smithi105.stdout:May 27 07:21:35 smithi105 systemd[1]: Starting Ceph mds.b for 73df2636-1bf8-11ef-bc9b-c7b262605968... 2024-05-27T07:21:36.471 INFO:journalctl@ceph.mds.b.smithi105.stdout:May 27 07:21:36 smithi105 podman[65529]: 2024-05-27 07:21:36.11546346 +0000 UTC m=+0.016868644 image pull e36ea4fc8a5807e8d2701370e350a0de3856538797f7fba64cfce7e344bd6799 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6 2024-05-27T07:21:36.472 INFO:journalctl@ceph.mds.b.smithi105.stdout:May 27 07:21:36 smithi105 podman[65529]: 2024-05-27 07:21:36.237372907 +0000 UTC m=+0.138778103 container create 94f6b4e4fe5671bd78bfffb952250e5a090ae24c024db4e59a43aaa435caab4d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-b, ceph=True, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240521, CEPH_POINT_RELEASE=, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, GIT_CLEAN=True, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0) 2024-05-27T07:21:36.810 INFO:journalctl@ceph.mds.b.smithi105.stdout:May 27 07:21:36 smithi105 podman[65529]: 2024-05-27 07:21:36.482453037 +0000 UTC m=+0.383858212 container init 94f6b4e4fe5671bd78bfffb952250e5a090ae24c024db4e59a43aaa435caab4d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-b, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, ceph=True, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, org.label-schema.schema-version=1.0, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.build-date=20240521, GIT_BRANCH=HEAD, io.buildah.version=1.33.5) 2024-05-27T07:21:36.810 INFO:journalctl@ceph.mds.b.smithi105.stdout:May 27 07:21:36 smithi105 podman[65529]: 2024-05-27 07:21:36.485367059 +0000 UTC m=+0.386772242 container start 94f6b4e4fe5671bd78bfffb952250e5a090ae24c024db4e59a43aaa435caab4d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-b, maintainer=Guillaume Abrioux , GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240521, org.label-schema.schema-version=1.0, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, ceph=True, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0) 2024-05-27T07:21:36.810 INFO:journalctl@ceph.mds.b.smithi105.stdout:May 27 07:21:36 smithi105 ceph-mds[65547]: -- 172.21.15.105:0/1000192777 <== mon.2 v2:172.21.15.105:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55ee47bc7a40 con 0x55ee47c0f180 2024-05-27T07:21:36.811 INFO:journalctl@ceph.mds.b.smithi105.stdout:May 27 07:21:36 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-b[65543]: starting mds.b at 2024-05-27T07:21:36.811 INFO:journalctl@ceph.mds.b.smithi105.stdout:May 27 07:21:36 smithi105 bash[65529]: 94f6b4e4fe5671bd78bfffb952250e5a090ae24c024db4e59a43aaa435caab4d 2024-05-27T07:21:36.811 INFO:journalctl@ceph.mds.b.smithi105.stdout:May 27 07:21:36 smithi105 systemd[1]: Started Ceph mds.b for 73df2636-1bf8-11ef-bc9b-c7b262605968. 2024-05-27T07:21:37.082 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:21:37.090+0000 7fe8fbc92700 1 -- 172.21.15.52:0/1309077630 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fe8f4107e20 msgr2=0x7fe8f4108230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:21:37.082 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:21:37.090+0000 7fe8fbc92700 1 --2- 172.21.15.52:0/1309077630 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fe8f4107e20 0x7fe8f4108230 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7fe8e40099c0 tx=0x7fe8e4009cd0 comp rx=0 tx=0).stop 2024-05-27T07:21:37.082 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:21:37.090+0000 7fe8fbc92700 1 -- 172.21.15.52:0/1309077630 shutdown_connections 2024-05-27T07:21:37.082 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:21:37.090+0000 7fe8fbc92700 1 --2- 172.21.15.52:0/1309077630 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fe8f4108770 0x7fe8f41100d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:21:37.082 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:21:37.090+0000 7fe8fbc92700 1 --2- 172.21.15.52:0/1309077630 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fe8f4103540 0x7fe8f4103950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:21:37.082 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:21:37.090+0000 7fe8fbc92700 1 --2- 172.21.15.52:0/1309077630 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fe8f4107e20 0x7fe8f4108230 unknown :-1 s=CLOSED pgs=178 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:21:37.082 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:21:37.090+0000 7fe8fbc92700 1 -- 172.21.15.52:0/1309077630 >> 172.21.15.52:0/1309077630 conn(0x7fe8f4078df0 msgr2=0x7fe8f4079230 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:21:37.082 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:21:37.090+0000 7fe8fbc92700 1 -- 172.21.15.52:0/1309077630 shutdown_connections 2024-05-27T07:21:37.082 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:21:37.090+0000 7fe8fbc92700 1 -- 172.21.15.52:0/1309077630 wait complete. 2024-05-27T07:21:37.083 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:21:36 smithi052 ceph-mon[31955]: pgmap v275: 1 pgs: 1 active+clean; 577 KiB data, 354 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-27T07:21:37.083 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:21:36 smithi052 ceph-mon[31955]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' 2024-05-27T07:21:37.083 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:21:36 smithi052 ceph-mon[31955]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' 2024-05-27T07:21:37.083 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:21:36 smithi052 ceph-mon[31955]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' 2024-05-27T07:21:37.083 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:21:36 smithi052 ceph-mon[31955]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' 2024-05-27T07:21:37.083 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:21:36 smithi052 ceph-mon[31955]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' 2024-05-27T07:21:37.085 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:21:37.091+0000 7fe8fbc92700 1 Processor -- start 2024-05-27T07:21:37.085 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:21:37.092+0000 7fe8fbc92700 1 -- start start 2024-05-27T07:21:37.085 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:21:37.092+0000 7fe8fbc92700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fe8f4103540 0x7fe8f4199d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:21:37.085 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:21:37.092+0000 7fe8fbc92700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fe8f4107e20 0x7fe8f419a2b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:21:37.085 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:21:37.092+0000 7fe8fbc92700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fe8f4108770 0x7fe8f419e540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:21:37.085 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:21:37.092+0000 7fe8fbc92700 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x7fe8f4113380 con 0x7fe8f4103540 2024-05-27T07:21:37.085 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:21:37.092+0000 7fe8fbc92700 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7fe8f4113200 con 0x7fe8f4108770 2024-05-27T07:21:37.085 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:21:37.092+0000 7fe8fbc92700 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7fe8f4113500 con 0x7fe8f4107e20 2024-05-27T07:21:37.085 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:21:37.092+0000 7fe8f9a2e700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fe8f4103540 0x7fe8f4199d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:21:37.086 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:21:37.092+0000 7fe8f9a2e700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fe8f4103540 0x7fe8f4199d10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.52:40486/0 (socket says 172.21.15.52:40486) 2024-05-27T07:21:37.086 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:21:37.092+0000 7fe8f9a2e700 1 -- 172.21.15.52:0/2325299485 learned_addr learned my addr 172.21.15.52:0/2325299485 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-05-27T07:21:37.086 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:21:37.092+0000 7fe8f922d700 1 --2- 172.21.15.52:0/2325299485 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fe8f4107e20 0x7fe8f419a2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:21:37.086 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:21:37.092+0000 7fe8fa22f700 1 --2- 172.21.15.52:0/2325299485 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fe8f4108770 0x7fe8f419e540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:21:37.086 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:21:37.093+0000 7fe8f922d700 1 -- 172.21.15.52:0/2325299485 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fe8f4108770 msgr2=0x7fe8f419e540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:21:37.086 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:21:37.093+0000 7fe8f922d700 1 --2- 172.21.15.52:0/2325299485 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fe8f4108770 0x7fe8f419e540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:21:37.086 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:21:37.093+0000 7fe8f922d700 1 -- 172.21.15.52:0/2325299485 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fe8f4103540 msgr2=0x7fe8f4199d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:21:37.086 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:21:37.093+0000 7fe8f922d700 1 --2- 172.21.15.52:0/2325299485 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fe8f4103540 0x7fe8f4199d10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:21:37.086 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:21:37.093+0000 7fe8f922d700 1 -- 172.21.15.52:0/2325299485 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe8f419ebe0 con 0x7fe8f4107e20 2024-05-27T07:21:37.086 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:21:37.093+0000 7fe8f9a2e700 1 --2- 172.21.15.52:0/2325299485 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fe8f4103540 0x7fe8f4199d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-27T07:21:37.086 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:21:37.093+0000 7fe8fa22f700 1 --2- 172.21.15.52:0/2325299485 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fe8f4108770 0x7fe8f419e540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-27T07:21:37.086 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:21:37.093+0000 7fe8f922d700 1 --2- 172.21.15.52:0/2325299485 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fe8f4107e20 0x7fe8f419a2b0 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7fe8e40098e0 tx=0x7fe8e4004270 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:21:37.086 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:21:37.093+0000 7fe8eaffd700 1 -- 172.21.15.52:0/2325299485 <== mon.1 v2:172.21.15.173:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe8e4026070 con 0x7fe8f4107e20 2024-05-27T07:21:37.087 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:21:37.093+0000 7fe8eaffd700 1 -- 172.21.15.52:0/2325299485 <== mon.1 v2:172.21.15.173:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fe8e4003c70 con 0x7fe8f4107e20 2024-05-27T07:21:37.087 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:21:37.093+0000 7fe8fbc92700 1 -- 172.21.15.52:0/2325299485 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe8f419ee70 con 0x7fe8f4107e20 2024-05-27T07:21:37.087 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:21:37.093+0000 7fe8eaffd700 1 -- 172.21.15.52:0/2325299485 <== mon.1 v2:172.21.15.173:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe8e400f9c0 con 0x7fe8f4107e20 2024-05-27T07:21:37.087 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:21:37.093+0000 7fe8fbc92700 1 -- 172.21.15.52:0/2325299485 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe8f419f350 con 0x7fe8f4107e20 2024-05-27T07:21:37.087 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:21:37.094+0000 7fe8fbc92700 1 -- 172.21.15.52:0/2325299485 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe8f40512b0 con 0x7fe8f4107e20 2024-05-27T07:21:37.088 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:21:37.097+0000 7fe8eaffd700 1 -- 172.21.15.52:0/2325299485 <== mon.1 v2:172.21.15.173:3300/0 4 ==== mgrmap(e 16) ==== 149652+0+0 (secure 0 0 0) 0x7fe8e4029070 con 0x7fe8f4107e20 2024-05-27T07:21:37.089 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:21:37.098+0000 7fe8eaffd700 1 --2- 172.21.15.52:0/2325299485 >> [v2:172.21.15.52:6800/3513075805,v1:172.21.15.52:6801/3513075805] conn(0x7fe8e00b2880 0x7fe8e00b4d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:21:37.089 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:21:37.098+0000 7fe8f9a2e700 1 --2- 172.21.15.52:0/2325299485 >> [v2:172.21.15.52:6800/3513075805,v1:172.21.15.52:6801/3513075805] conn(0x7fe8e00b2880 0x7fe8e00b4d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:21:37.089 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:21:37.098+0000 7fe8eaffd700 1 -- 172.21.15.52:0/2325299485 <== mon.1 v2:172.21.15.173:3300/0 5 ==== osd_map(72..72 src has 1..72) ==== 7127+0+0 (secure 0 0 0) 0x7fe8e40e27a0 con 0x7fe8f4107e20 2024-05-27T07:21:37.089 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:21:37.098+0000 7fe8f9a2e700 1 --2- 172.21.15.52:0/2325299485 >> [v2:172.21.15.52:6800/3513075805,v1:172.21.15.52:6801/3513075805] conn(0x7fe8e00b2880 0x7fe8e00b4d90 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7fe8f0011550 tx=0x7fe8f0009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:21:37.091 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:21:37.100+0000 7fe8eaffd700 1 -- 172.21.15.52:0/2325299485 <== mon.1 v2:172.21.15.173:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191963 (secure 0 0 0) 0x7fe8e4013070 con 0x7fe8f4107e20 2024-05-27T07:21:37.102 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:21:36 smithi173 ceph-mon[37000]: pgmap v275: 1 pgs: 1 active+clean; 577 KiB data, 354 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-27T07:21:37.102 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:21:36 smithi173 ceph-mon[37000]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' 2024-05-27T07:21:37.102 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:21:36 smithi173 ceph-mon[37000]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' 2024-05-27T07:21:37.102 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:21:36 smithi173 ceph-mon[37000]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' 2024-05-27T07:21:37.102 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:21:36 smithi173 ceph-mon[37000]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' 2024-05-27T07:21:37.102 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:21:36 smithi173 ceph-mon[37000]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' 2024-05-27T07:21:37.190 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:21:36 smithi105 ceph-mon[35977]: pgmap v275: 1 pgs: 1 active+clean; 577 KiB data, 354 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-27T07:21:37.190 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:21:36 smithi105 ceph-mon[35977]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' 2024-05-27T07:21:37.190 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:21:36 smithi105 ceph-mon[35977]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' 2024-05-27T07:21:37.191 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:21:36 smithi105 ceph-mon[35977]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' 2024-05-27T07:21:37.191 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:21:36 smithi105 ceph-mon[35977]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' 2024-05-27T07:21:37.191 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:21:36 smithi105 ceph-mon[35977]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' 2024-05-27T07:21:37.364 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:21:37.374+0000 7fe8fbc92700 1 -- 172.21.15.52:0/2325299485 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fe8f410b200 con 0x7fe8f4107e20 2024-05-27T07:21:37.365 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:21:37.374+0000 7fe8eaffd700 1 -- 172.21.15.52:0/2325299485 <== mon.1 v2:172.21.15.173:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 13 v13) ==== 94+0+10485 (secure 0 0 0) 0x7fe8e40aa950 con 0x7fe8f4107e20 2024-05-27T07:21:37.366 INFO:teuthology.orchestra.run.smithi052.stdout: 2024-05-27T07:21:37.366 INFO:teuthology.orchestra.run.smithi052.stdout:{"epoch":13,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14556,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.52:6835/2584733778","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.52:6834","nonce":2584733778},{"type":"v1","addr":"172.21.15.52:6835","nonce":2584733778}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14568,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.52:6837/675445303","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.52:6836","nonce":675445303},{"type":"v1","addr":"172.21.15.52:6837","nonce":675445303}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14619,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.52:6839/4258549559","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.52:6838","nonce":4258549559},{"type":"v1","addr":"172.21.15.52:6839","nonce":4258549559}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14631,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.52:6841/2878618188","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.52:6840","nonce":2878618188},{"type":"v1","addr":"172.21.15.52:6841","nonce":2878618188}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24433,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.173:6833/3013589220","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6832","nonce":3013589220},{"type":"v1","addr":"172.21.15.173:6833","nonce":3013589220}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24452,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.105:6833/2800678412","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6832","nonce":2800678412},{"type":"v1","addr":"172.21.15.105:6833","nonce":2800678412}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24460,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.173:6835/1619486437","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6834","nonce":1619486437},{"type":"v1","addr":"172.21.15.173:6835","nonce":1619486437}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":7},{"gid":24466,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.173:6837/1396363827","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6836","nonce":1396363827},{"type":"v1","addr":"172.21.15.173:6837","nonce":1396363827}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":8},{"gid":24473,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.105:6835/3298498384","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6834","nonce":3298498384},{"type":"v1","addr":"172.21.15.105:6835","nonce":3298498384}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6},{"gid":24475,"name":"i","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.173:6839/564778293","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6838","nonce":564778293},{"type":"v1","addr":"172.21.15.173:6839","nonce":564778293}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":10},{"gid":24497,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.105:6837/191393149","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6836","nonce":191393149},{"type":"v1","addr":"172.21.15.105:6837","nonce":191393149}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":9},{"gid":24527,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.105:6839/500348151","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6838","nonce":500348151},{"type":"v1","addr":"172.21.15.105:6839","nonce":500348151}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":13}],"filesystems":[]} 2024-05-27T07:21:37.368 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:21:37.378+0000 7fe8fbc92700 1 -- 172.21.15.52:0/2325299485 >> [v2:172.21.15.52:6800/3513075805,v1:172.21.15.52:6801/3513075805] conn(0x7fe8e00b2880 msgr2=0x7fe8e00b4d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:21:37.368 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:21:37.378+0000 7fe8fbc92700 1 --2- 172.21.15.52:0/2325299485 >> [v2:172.21.15.52:6800/3513075805,v1:172.21.15.52:6801/3513075805] conn(0x7fe8e00b2880 0x7fe8e00b4d90 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7fe8f0011550 tx=0x7fe8f0009260 comp rx=0 tx=0).stop 2024-05-27T07:21:37.368 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:21:37.378+0000 7fe8fbc92700 1 -- 172.21.15.52:0/2325299485 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fe8f4107e20 msgr2=0x7fe8f419a2b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:21:37.368 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:21:37.378+0000 7fe8fbc92700 1 --2- 172.21.15.52:0/2325299485 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fe8f4107e20 0x7fe8f419a2b0 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7fe8e40098e0 tx=0x7fe8e4004270 comp rx=0 tx=0).stop 2024-05-27T07:21:37.368 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:21:37.378+0000 7fe8fbc92700 1 -- 172.21.15.52:0/2325299485 shutdown_connections 2024-05-27T07:21:37.369 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:21:37.378+0000 7fe8fbc92700 1 --2- 172.21.15.52:0/2325299485 >> [v2:172.21.15.52:6800/3513075805,v1:172.21.15.52:6801/3513075805] conn(0x7fe8e00b2880 0x7fe8e00b4d90 unknown :-1 s=CLOSED pgs=139 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:21:37.369 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:21:37.378+0000 7fe8fbc92700 1 --2- 172.21.15.52:0/2325299485 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fe8f4108770 0x7fe8f419e540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:21:37.369 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:21:37.378+0000 7fe8fbc92700 1 --2- 172.21.15.52:0/2325299485 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fe8f4103540 0x7fe8f4199d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:21:37.369 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:21:37.378+0000 7fe8fbc92700 1 --2- 172.21.15.52:0/2325299485 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fe8f4107e20 0x7fe8f419a2b0 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:21:37.369 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:21:37.378+0000 7fe8fbc92700 1 -- 172.21.15.52:0/2325299485 >> 172.21.15.52:0/2325299485 conn(0x7fe8f4078df0 msgr2=0x7fe8f4100f40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:21:37.369 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:21:37.378+0000 7fe8fbc92700 1 -- 172.21.15.52:0/2325299485 shutdown_connections 2024-05-27T07:21:37.369 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:21:37.378+0000 7fe8fbc92700 1 -- 172.21.15.52:0/2325299485 wait complete. 2024-05-27T07:21:37.370 INFO:teuthology.orchestra.run.smithi052.stderr:dumped fsmap epoch 13 2024-05-27T07:21:37.889 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:21:37 smithi105 ceph-mon[35977]: mds.? [v2:172.21.15.105:6838/500348151,v1:172.21.15.105:6839/500348151] up:boot 2024-05-27T07:21:37.889 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:21:37 smithi105 ceph-mon[35977]: fsmap 12 up:standby 2024-05-27T07:21:37.889 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:21:37 smithi105 ceph-mon[35977]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "b"}]: dispatch 2024-05-27T07:21:37.889 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:21:37 smithi105 ceph-mon[35977]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-27T07:21:37.890 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:21:37 smithi105 ceph-mon[35977]: from='client.? 172.21.15.52:0/2325299485' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-27T07:21:37.892 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:21:37 smithi173 ceph-mon[37000]: mds.? [v2:172.21.15.105:6838/500348151,v1:172.21.15.105:6839/500348151] up:boot 2024-05-27T07:21:37.892 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:21:37 smithi173 ceph-mon[37000]: fsmap 12 up:standby 2024-05-27T07:21:37.892 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:21:37 smithi173 ceph-mon[37000]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "b"}]: dispatch 2024-05-27T07:21:37.892 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:21:37 smithi173 ceph-mon[37000]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-27T07:21:37.892 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:21:37 smithi173 ceph-mon[37000]: from='client.? 172.21.15.52:0/2325299485' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-27T07:21:38.052 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:21:37 smithi052 ceph-mon[31955]: mds.? [v2:172.21.15.105:6838/500348151,v1:172.21.15.105:6839/500348151] up:boot 2024-05-27T07:21:38.052 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:21:37 smithi052 ceph-mon[31955]: fsmap 12 up:standby 2024-05-27T07:21:38.052 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:21:37 smithi052 ceph-mon[31955]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "b"}]: dispatch 2024-05-27T07:21:38.052 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:21:37 smithi052 ceph-mon[31955]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-27T07:21:38.053 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:21:37 smithi052 ceph-mon[31955]: from='client.? 172.21.15.52:0/2325299485' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-27T07:21:38.080 DEBUG:teuthology.orchestra.run.smithi052:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:38f9d900c83c63799fbdbe61acc9a11b0d3554a6 shell --fsid 73df2636-1bf8-11ef-bc9b-c7b262605968 -- ceph --format=json-pretty osd lspools 2024-05-27T07:21:38.849 INFO:teuthology.orchestra.run.smithi052.stderr:Inferring config /var/lib/ceph/73df2636-1bf8-11ef-bc9b-c7b262605968/mon.a/config 2024-05-27T07:21:38.974 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:21:38 smithi052 ceph-mon[31955]: pgmap v276: 1 pgs: 1 active+clean; 577 KiB data, 354 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-27T07:21:38.974 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:21:38 smithi052 ceph-mon[31955]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' 2024-05-27T07:21:38.974 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:21:38 smithi052 ceph-mon[31955]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' 2024-05-27T07:21:38.974 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:21:38 smithi052 ceph-mon[31955]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' 2024-05-27T07:21:38.974 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:21:38 smithi052 ceph-mon[31955]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' 2024-05-27T07:21:39.102 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:21:38 smithi173 ceph-mon[37000]: pgmap v276: 1 pgs: 1 active+clean; 577 KiB data, 354 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-27T07:21:39.102 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:21:38 smithi173 ceph-mon[37000]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' 2024-05-27T07:21:39.102 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:21:38 smithi173 ceph-mon[37000]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' 2024-05-27T07:21:39.102 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:21:38 smithi173 ceph-mon[37000]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' 2024-05-27T07:21:39.102 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:21:38 smithi173 ceph-mon[37000]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' 2024-05-27T07:21:39.116 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:21:38 smithi105 ceph-mon[35977]: pgmap v276: 1 pgs: 1 active+clean; 577 KiB data, 354 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-27T07:21:39.116 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:21:38 smithi105 ceph-mon[35977]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' 2024-05-27T07:21:39.116 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:21:38 smithi105 ceph-mon[35977]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' 2024-05-27T07:21:39.116 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:21:38 smithi105 ceph-mon[35977]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' 2024-05-27T07:21:39.116 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:21:38 smithi105 ceph-mon[35977]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' 2024-05-27T07:21:40.510 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:21:40.519+0000 7f5612b89700 1 -- 172.21.15.52:0/1168493688 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f560c108730 msgr2=0x7f560c110090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:21:40.510 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:21:40.519+0000 7f5612b89700 1 --2- 172.21.15.52:0/1168493688 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f560c108730 0x7f560c110090 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7f56040099c0 tx=0x7f5604009cd0 comp rx=0 tx=0).stop 2024-05-27T07:21:40.510 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:21:40.519+0000 7f5612b89700 1 -- 172.21.15.52:0/1168493688 shutdown_connections 2024-05-27T07:21:40.510 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:21:40.519+0000 7f5612b89700 1 --2- 172.21.15.52:0/1168493688 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f560c108730 0x7f560c110090 unknown :-1 s=CLOSED pgs=179 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:21:40.510 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:21:40.519+0000 7f5612b89700 1 --2- 172.21.15.52:0/1168493688 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f560c1034b0 0x7f560c1038c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:21:40.510 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:21:40.519+0000 7f5612b89700 1 --2- 172.21.15.52:0/1168493688 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f560c107de0 0x7f560c1081f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:21:40.510 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:21:40.519+0000 7f5612b89700 1 -- 172.21.15.52:0/1168493688 >> 172.21.15.52:0/1168493688 conn(0x7f560c0fede0 msgr2=0x7f560c101200 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:21:40.510 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:21:40.519+0000 7f5612b89700 1 -- 172.21.15.52:0/1168493688 shutdown_connections 2024-05-27T07:21:40.510 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:21:40.519+0000 7f5612b89700 1 -- 172.21.15.52:0/1168493688 wait complete. 2024-05-27T07:21:40.510 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:21:40.520+0000 7f5612b89700 1 Processor -- start 2024-05-27T07:21:40.510 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:21:40.520+0000 7f5612b89700 1 -- start start 2024-05-27T07:21:40.512 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:21:40.520+0000 7f5612b89700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f560c1034b0 0x7f560c0ff2c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:21:40.512 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:21:40.520+0000 7f5612b89700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f560c107de0 0x7f560c0ff800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:21:40.512 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:21:40.520+0000 7f5612b89700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f560c108730 0x7f560c0ffd40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:21:40.512 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:21:40.520+0000 7f5612b89700 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x7f560c113360 con 0x7f560c107de0 2024-05-27T07:21:40.512 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:21:40.520+0000 7f5612b89700 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f560c1131e0 con 0x7f560c108730 2024-05-27T07:21:40.512 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:21:40.520+0000 7f5612b89700 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f560c1134e0 con 0x7f560c1034b0 2024-05-27T07:21:40.513 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:21:40.520+0000 7f560bfff700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f560c107de0 0x7f560c0ff800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:21:40.513 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:21:40.520+0000 7f560bfff700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f560c107de0 0x7f560c0ff800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.52:40506/0 (socket says 172.21.15.52:40506) 2024-05-27T07:21:40.513 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:21:40.520+0000 7f560bfff700 1 -- 172.21.15.52:0/676168259 learned_addr learned my addr 172.21.15.52:0/676168259 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-05-27T07:21:40.513 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:21:40.520+0000 7f5610925700 1 --2- 172.21.15.52:0/676168259 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f560c1034b0 0x7f560c0ff2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:21:40.513 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:21:40.521+0000 7f5611126700 1 --2- 172.21.15.52:0/676168259 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f560c108730 0x7f560c0ffd40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:21:40.513 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:21:40.521+0000 7f560bfff700 1 -- 172.21.15.52:0/676168259 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f560c1034b0 msgr2=0x7f560c0ff2c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:21:40.513 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:21:40.521+0000 7f560bfff700 1 --2- 172.21.15.52:0/676168259 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f560c1034b0 0x7f560c0ff2c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:21:40.513 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:21:40.521+0000 7f560bfff700 1 -- 172.21.15.52:0/676168259 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f560c108730 msgr2=0x7f560c0ffd40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:21:40.513 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:21:40.521+0000 7f560bfff700 1 --2- 172.21.15.52:0/676168259 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f560c108730 0x7f560c0ffd40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:21:40.513 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:21:40.521+0000 7f560bfff700 1 -- 172.21.15.52:0/676168259 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f560c1a9180 con 0x7f560c107de0 2024-05-27T07:21:40.513 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:21:40.521+0000 7f560bfff700 1 --2- 172.21.15.52:0/676168259 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f560c107de0 0x7f560c0ff800 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7f55fc007f10 tx=0x7f55fc00d3b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:21:40.514 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:21:40.521+0000 7f5609ffb700 1 -- 172.21.15.52:0/676168259 <== mon.0 v2:172.21.15.52:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f55fc00dd20 con 0x7f560c107de0 2024-05-27T07:21:40.514 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:21:40.521+0000 7f5609ffb700 1 -- 172.21.15.52:0/676168259 <== mon.0 v2:172.21.15.52:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f55fc018070 con 0x7f560c107de0 2024-05-27T07:21:40.514 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:21:40.521+0000 7f5612b89700 1 -- 172.21.15.52:0/676168259 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f560c1a9470 con 0x7f560c107de0 2024-05-27T07:21:40.514 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:21:40.521+0000 7f5609ffb700 1 -- 172.21.15.52:0/676168259 <== mon.0 v2:172.21.15.52:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f55fc0125b0 con 0x7f560c107de0 2024-05-27T07:21:40.514 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:21:40.522+0000 7f5612b89700 1 -- 172.21.15.52:0/676168259 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f560c1a9a00 con 0x7f560c107de0 2024-05-27T07:21:40.514 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:21:40.523+0000 7f5612b89700 1 -- 172.21.15.52:0/676168259 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f560c0512b0 con 0x7f560c107de0 2024-05-27T07:21:40.515 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:21:40.524+0000 7f5609ffb700 1 -- 172.21.15.52:0/676168259 <== mon.0 v2:172.21.15.52:3300/0 4 ==== mgrmap(e 16) ==== 149652+0+0 (secure 0 0 0) 0x7f55fc0133f0 con 0x7f560c107de0 2024-05-27T07:21:40.515 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:21:40.525+0000 7f5609ffb700 1 --2- 172.21.15.52:0/676168259 >> [v2:172.21.15.52:6800/3513075805,v1:172.21.15.52:6801/3513075805] conn(0x7f55f80b2900 0x7f55f80b4e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:21:40.515 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:21:40.525+0000 7f5610925700 1 --2- 172.21.15.52:0/676168259 >> [v2:172.21.15.52:6800/3513075805,v1:172.21.15.52:6801/3513075805] conn(0x7f55f80b2900 0x7f55f80b4e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:21:40.515 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:21:40.525+0000 7f5609ffb700 1 -- 172.21.15.52:0/676168259 <== mon.0 v2:172.21.15.52:3300/0 5 ==== osd_map(72..72 src has 1..72) ==== 7127+0+0 (secure 0 0 0) 0x7f55fc0e1a40 con 0x7f560c107de0 2024-05-27T07:21:40.516 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:21:40.526+0000 7f5610925700 1 --2- 172.21.15.52:0/676168259 >> [v2:172.21.15.52:6800/3513075805,v1:172.21.15.52:6801/3513075805] conn(0x7f55f80b2900 0x7f55f80b4e10 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7f55f4005950 tx=0x7f55f400b410 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:21:40.520 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:21:40.530+0000 7f5609ffb700 1 -- 172.21.15.52:0/676168259 <== mon.0 v2:172.21.15.52:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191963 (secure 0 0 0) 0x7f55fc010070 con 0x7f560c107de0 2024-05-27T07:21:40.794 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:21:40.804+0000 7f5612b89700 1 -- 172.21.15.52:0/676168259 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_command({"prefix": "osd lspools", "format": "json-pretty"} v 0) -- 0x7f560c02e5e0 con 0x7f560c107de0 2024-05-27T07:21:40.794 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:21:40 smithi052 ceph-mon[31955]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' 2024-05-27T07:21:40.794 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:21:40 smithi052 ceph-mon[31955]: pgmap v277: 1 pgs: 1 active+clean; 577 KiB data, 354 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-27T07:21:40.795 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:21:40.804+0000 7f5609ffb700 1 -- 172.21.15.52:0/676168259 <== mon.0 v2:172.21.15.52:3300/0 7 ==== mon_command_ack([{"prefix": "osd lspools", "format": "json-pretty"}]=0 v72) ==== 84+0+65 (secure 0 0 0) 0x7f55fc0a9b60 con 0x7f560c107de0 2024-05-27T07:21:40.795 INFO:teuthology.orchestra.run.smithi052.stdout: 2024-05-27T07:21:40.795 INFO:teuthology.orchestra.run.smithi052.stdout:[ 2024-05-27T07:21:40.795 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-05-27T07:21:40.795 INFO:teuthology.orchestra.run.smithi052.stdout: "poolnum": 1, 2024-05-27T07:21:40.795 INFO:teuthology.orchestra.run.smithi052.stdout: "poolname": ".mgr" 2024-05-27T07:21:40.795 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-05-27T07:21:40.795 INFO:teuthology.orchestra.run.smithi052.stdout:] 2024-05-27T07:21:40.797 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:21:40.807+0000 7f5612b89700 1 -- 172.21.15.52:0/676168259 >> [v2:172.21.15.52:6800/3513075805,v1:172.21.15.52:6801/3513075805] conn(0x7f55f80b2900 msgr2=0x7f55f80b4e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:21:40.797 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:21:40.807+0000 7f5612b89700 1 --2- 172.21.15.52:0/676168259 >> [v2:172.21.15.52:6800/3513075805,v1:172.21.15.52:6801/3513075805] conn(0x7f55f80b2900 0x7f55f80b4e10 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7f55f4005950 tx=0x7f55f400b410 comp rx=0 tx=0).stop 2024-05-27T07:21:40.797 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:21:40.807+0000 7f5612b89700 1 -- 172.21.15.52:0/676168259 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f560c107de0 msgr2=0x7f560c0ff800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:21:40.797 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:21:40.807+0000 7f5612b89700 1 --2- 172.21.15.52:0/676168259 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f560c107de0 0x7f560c0ff800 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7f55fc007f10 tx=0x7f55fc00d3b0 comp rx=0 tx=0).stop 2024-05-27T07:21:40.797 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:21:40.807+0000 7f5612b89700 1 -- 172.21.15.52:0/676168259 shutdown_connections 2024-05-27T07:21:40.797 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:21:40.807+0000 7f5612b89700 1 --2- 172.21.15.52:0/676168259 >> [v2:172.21.15.52:6800/3513075805,v1:172.21.15.52:6801/3513075805] conn(0x7f55f80b2900 0x7f55f80b4e10 unknown :-1 s=CLOSED pgs=140 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:21:40.797 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:21:40.807+0000 7f5612b89700 1 --2- 172.21.15.52:0/676168259 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f560c108730 0x7f560c0ffd40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:21:40.797 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:21:40.807+0000 7f5612b89700 1 --2- 172.21.15.52:0/676168259 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f560c1034b0 0x7f560c0ff2c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:21:40.797 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:21:40.807+0000 7f5612b89700 1 --2- 172.21.15.52:0/676168259 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f560c107de0 0x7f560c0ff800 unknown :-1 s=CLOSED pgs=180 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:21:40.798 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:21:40.807+0000 7f5612b89700 1 -- 172.21.15.52:0/676168259 >> 172.21.15.52:0/676168259 conn(0x7f560c0fede0 msgr2=0x7f560c075d20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:21:40.798 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:21:40.807+0000 7f5612b89700 1 -- 172.21.15.52:0/676168259 shutdown_connections 2024-05-27T07:21:40.798 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:21:40.807+0000 7f5612b89700 1 -- 172.21.15.52:0/676168259 wait complete. 2024-05-27T07:21:40.940 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:21:40 smithi105 ceph-mon[35977]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' 2024-05-27T07:21:40.940 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:21:40 smithi105 ceph-mon[35977]: pgmap v277: 1 pgs: 1 active+clean; 577 KiB data, 354 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-27T07:21:41.102 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:21:40 smithi173 ceph-mon[37000]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' 2024-05-27T07:21:41.102 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:21:40 smithi173 ceph-mon[37000]: pgmap v277: 1 pgs: 1 active+clean; 577 KiB data, 354 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-27T07:21:41.528 DEBUG:tasks.cephfs.filesystem:Creating filesystem 'cephfs' 2024-05-27T07:21:41.528 DEBUG:teuthology.orchestra.run.smithi052:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:38f9d900c83c63799fbdbe61acc9a11b0d3554a6 shell --fsid 73df2636-1bf8-11ef-bc9b-c7b262605968 -- ceph osd pool create cephfs_metadata --pg_num_min 64 2024-05-27T07:21:41.797 INFO:teuthology.orchestra.run.smithi052.stderr:Inferring config /var/lib/ceph/73df2636-1bf8-11ef-bc9b-c7b262605968/mon.a/config 2024-05-27T07:21:41.920 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:21:41 smithi052 ceph-mon[31955]: from='client.? 172.21.15.52:0/676168259' entity='client.admin' cmd=[{"prefix": "osd lspools", "format": "json-pretty"}]: dispatch 2024-05-27T07:21:41.920 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:21:41 smithi052 ceph-mon[31955]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' 2024-05-27T07:21:41.920 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:21:41 smithi052 ceph-mon[31955]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' 2024-05-27T07:21:41.940 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:21:41 smithi105 ceph-mon[35977]: from='client.? 172.21.15.52:0/676168259' entity='client.admin' cmd=[{"prefix": "osd lspools", "format": "json-pretty"}]: dispatch 2024-05-27T07:21:41.940 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:21:41 smithi105 ceph-mon[35977]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' 2024-05-27T07:21:41.940 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:21:41 smithi105 ceph-mon[35977]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' 2024-05-27T07:21:42.003 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:21:41 smithi173 ceph-mon[37000]: from='client.? 172.21.15.52:0/676168259' entity='client.admin' cmd=[{"prefix": "osd lspools", "format": "json-pretty"}]: dispatch 2024-05-27T07:21:42.003 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:21:41 smithi173 ceph-mon[37000]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' 2024-05-27T07:21:42.003 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:21:41 smithi173 ceph-mon[37000]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' 2024-05-27T07:21:42.922 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:21:42 smithi052 ceph-mon[31955]: pgmap v278: 1 pgs: 1 active+clean; 577 KiB data, 354 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-27T07:21:42.923 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:21:42 smithi052 ceph-mon[31955]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T07:21:42.923 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:21:42 smithi052 ceph-mon[31955]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-27T07:21:42.923 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:21:42 smithi052 ceph-mon[31955]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' 2024-05-27T07:21:42.923 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:21:42 smithi052 ceph-mon[31955]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-27T07:21:42.940 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:21:42 smithi105 ceph-mon[35977]: pgmap v278: 1 pgs: 1 active+clean; 577 KiB data, 354 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-27T07:21:42.940 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:21:42 smithi105 ceph-mon[35977]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T07:21:42.940 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:21:42 smithi105 ceph-mon[35977]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-27T07:21:42.940 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:21:42 smithi105 ceph-mon[35977]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' 2024-05-27T07:21:42.940 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:21:42 smithi105 ceph-mon[35977]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-27T07:21:43.102 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:21:42 smithi173 ceph-mon[37000]: pgmap v278: 1 pgs: 1 active+clean; 577 KiB data, 354 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-27T07:21:43.102 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:21:42 smithi173 ceph-mon[37000]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T07:21:43.102 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:21:42 smithi173 ceph-mon[37000]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-27T07:21:43.102 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:21:42 smithi173 ceph-mon[37000]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' 2024-05-27T07:21:43.102 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:21:42 smithi173 ceph-mon[37000]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-27T07:21:43.676 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:21:43.686+0000 7fa1d2f93700 1 -- 172.21.15.52:0/3252122399 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fa1cc1034b0 msgr2=0x7fa1cc1038c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:21:43.676 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:21:43.686+0000 7fa1d2f93700 1 --2- 172.21.15.52:0/3252122399 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fa1cc1034b0 0x7fa1cc1038c0 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7fa1c00099c0 tx=0x7fa1c0009cd0 comp rx=0 tx=0).stop 2024-05-27T07:21:43.676 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:21:43.686+0000 7fa1d2f93700 1 -- 172.21.15.52:0/3252122399 shutdown_connections 2024-05-27T07:21:43.676 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:21:43.686+0000 7fa1d2f93700 1 --2- 172.21.15.52:0/3252122399 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fa1cc108730 0x7fa1cc110090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:21:43.676 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:21:43.686+0000 7fa1d2f93700 1 --2- 172.21.15.52:0/3252122399 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fa1cc1034b0 0x7fa1cc1038c0 unknown :-1 s=CLOSED pgs=181 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:21:43.677 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:21:43.686+0000 7fa1d2f93700 1 --2- 172.21.15.52:0/3252122399 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fa1cc107de0 0x7fa1cc1081f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:21:43.677 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:21:43.686+0000 7fa1d2f93700 1 -- 172.21.15.52:0/3252122399 >> 172.21.15.52:0/3252122399 conn(0x7fa1cc0fedb0 msgr2=0x7fa1cc1011d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:21:43.677 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:21:43.686+0000 7fa1d2f93700 1 -- 172.21.15.52:0/3252122399 shutdown_connections 2024-05-27T07:21:43.677 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:21:43.686+0000 7fa1d2f93700 1 -- 172.21.15.52:0/3252122399 wait complete. 2024-05-27T07:21:43.679 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:21:43.686+0000 7fa1d2f93700 1 Processor -- start 2024-05-27T07:21:43.679 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:21:43.687+0000 7fa1d2f93700 1 -- start start 2024-05-27T07:21:43.679 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:21:43.687+0000 7fa1d2f93700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fa1cc1034b0 0x7fa1cc19bce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:21:43.679 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:21:43.687+0000 7fa1d2f93700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fa1cc107de0 0x7fa1cc19c220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:21:43.679 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:21:43.687+0000 7fa1d2f93700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fa1cc108730 0x7fa1cc19c760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:21:43.679 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:21:43.687+0000 7fa1d2f93700 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x7fa1cc1133e0 con 0x7fa1cc108730 2024-05-27T07:21:43.679 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:21:43.687+0000 7fa1d2f93700 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7fa1cc113260 con 0x7fa1cc1034b0 2024-05-27T07:21:43.679 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:21:43.687+0000 7fa1d2f93700 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7fa1cc113560 con 0x7fa1cc107de0 2024-05-27T07:21:43.679 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:21:43.687+0000 7fa1d1530700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fa1cc108730 0x7fa1cc19c760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:21:43.679 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:21:43.687+0000 7fa1d1530700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fa1cc108730 0x7fa1cc19c760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.52:54416/0 (socket says 172.21.15.52:54416) 2024-05-27T07:21:43.680 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:21:43.687+0000 7fa1d1530700 1 -- 172.21.15.52:0/2248019265 learned_addr learned my addr 172.21.15.52:0/2248019265 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-05-27T07:21:43.680 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:21:43.687+0000 7fa1d0d2f700 1 --2- 172.21.15.52:0/2248019265 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fa1cc1034b0 0x7fa1cc19bce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:21:43.680 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:21:43.687+0000 7fa1cbfff700 1 --2- 172.21.15.52:0/2248019265 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fa1cc107de0 0x7fa1cc19c220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:21:43.680 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:21:43.687+0000 7fa1d1530700 1 -- 172.21.15.52:0/2248019265 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fa1cc107de0 msgr2=0x7fa1cc19c220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:21:43.680 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:21:43.687+0000 7fa1d1530700 1 --2- 172.21.15.52:0/2248019265 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fa1cc107de0 0x7fa1cc19c220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:21:43.680 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:21:43.687+0000 7fa1d1530700 1 -- 172.21.15.52:0/2248019265 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fa1cc1034b0 msgr2=0x7fa1cc19bce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:21:43.680 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:21:43.687+0000 7fa1d1530700 1 --2- 172.21.15.52:0/2248019265 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fa1cc1034b0 0x7fa1cc19bce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:21:43.680 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:21:43.687+0000 7fa1d1530700 1 -- 172.21.15.52:0/2248019265 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa1cc1a7000 con 0x7fa1cc108730 2024-05-27T07:21:43.680 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:21:43.688+0000 7fa1d1530700 1 --2- 172.21.15.52:0/2248019265 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fa1cc108730 0x7fa1cc19c760 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7fa1c400b970 tx=0x7fa1c400bc80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:21:43.681 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:21:43.688+0000 7fa1c9ffb700 1 -- 172.21.15.52:0/2248019265 <== mon.0 v2:172.21.15.52:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa1c400d860 con 0x7fa1cc108730 2024-05-27T07:21:43.681 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:21:43.688+0000 7fa1d2f93700 1 -- 172.21.15.52:0/2248019265 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa1cc1a72f0 con 0x7fa1cc108730 2024-05-27T07:21:43.681 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:21:43.688+0000 7fa1c9ffb700 1 -- 172.21.15.52:0/2248019265 <== mon.0 v2:172.21.15.52:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fa1c400e430 con 0x7fa1cc108730 2024-05-27T07:21:43.681 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:21:43.688+0000 7fa1c9ffb700 1 -- 172.21.15.52:0/2248019265 <== mon.0 v2:172.21.15.52:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa1c4017660 con 0x7fa1cc108730 2024-05-27T07:21:43.681 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:21:43.688+0000 7fa1d2f93700 1 -- 172.21.15.52:0/2248019265 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa1cc1a7830 con 0x7fa1cc108730 2024-05-27T07:21:43.681 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:21:43.689+0000 7fa1d2f93700 1 -- 172.21.15.52:0/2248019265 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa1cc0512b0 con 0x7fa1cc108730 2024-05-27T07:21:43.681 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:21:43.691+0000 7fa1c9ffb700 1 -- 172.21.15.52:0/2248019265 <== mon.0 v2:172.21.15.52:3300/0 4 ==== mgrmap(e 16) ==== 149652+0+0 (secure 0 0 0) 0x7fa1c400da00 con 0x7fa1cc108730 2024-05-27T07:21:43.682 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:21:43.692+0000 7fa1c9ffb700 1 --2- 172.21.15.52:0/2248019265 >> [v2:172.21.15.52:6800/3513075805,v1:172.21.15.52:6801/3513075805] conn(0x7fa1bc0b2880 0x7fa1bc0b4d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:21:43.682 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:21:43.692+0000 7fa1d0d2f700 1 --2- 172.21.15.52:0/2248019265 >> [v2:172.21.15.52:6800/3513075805,v1:172.21.15.52:6801/3513075805] conn(0x7fa1bc0b2880 0x7fa1bc0b4d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:21:43.682 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:21:43.692+0000 7fa1c9ffb700 1 -- 172.21.15.52:0/2248019265 <== mon.0 v2:172.21.15.52:3300/0 5 ==== osd_map(72..72 src has 1..72) ==== 7127+0+0 (secure 0 0 0) 0x7fa1c40e1b90 con 0x7fa1cc108730 2024-05-27T07:21:43.682 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:21:43.692+0000 7fa1d0d2f700 1 --2- 172.21.15.52:0/2248019265 >> [v2:172.21.15.52:6800/3513075805,v1:172.21.15.52:6801/3513075805] conn(0x7fa1bc0b2880 0x7fa1bc0b4d90 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7fa1c0009990 tx=0x7fa1c0000bc0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:21:43.685 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:21:43.696+0000 7fa1c9ffb700 1 -- 172.21.15.52:0/2248019265 <== mon.0 v2:172.21.15.52:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191963 (secure 0 0 0) 0x7fa1c4053df0 con 0x7fa1cc108730 2024-05-27T07:21:43.903 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:21:43.912+0000 7fa1d2f93700 1 -- 172.21.15.52:0/2248019265 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64} v 0) -- 0x7fa1cc02e5e0 con 0x7fa1cc108730 2024-05-27T07:21:44.538 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:21:44.547+0000 7fa1c9ffb700 1 -- 172.21.15.52:0/2248019265 <== mon.0 v2:172.21.15.52:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]=0 pool 'cephfs_metadata' created v73) ==== 138+0+0 (secure 0 0 0) 0x7fa1c4013030 con 0x7fa1cc108730 2024-05-27T07:21:44.541 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:21:44.550+0000 7fa1d2f93700 1 -- 172.21.15.52:0/2248019265 >> [v2:172.21.15.52:6800/3513075805,v1:172.21.15.52:6801/3513075805] conn(0x7fa1bc0b2880 msgr2=0x7fa1bc0b4d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:21:44.541 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:21:44.550+0000 7fa1d2f93700 1 --2- 172.21.15.52:0/2248019265 >> [v2:172.21.15.52:6800/3513075805,v1:172.21.15.52:6801/3513075805] conn(0x7fa1bc0b2880 0x7fa1bc0b4d90 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7fa1c0009990 tx=0x7fa1c0000bc0 comp rx=0 tx=0).stop 2024-05-27T07:21:44.541 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:21:44.550+0000 7fa1d2f93700 1 -- 172.21.15.52:0/2248019265 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fa1cc108730 msgr2=0x7fa1cc19c760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:21:44.541 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:21:44.550+0000 7fa1d2f93700 1 --2- 172.21.15.52:0/2248019265 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fa1cc108730 0x7fa1cc19c760 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7fa1c400b970 tx=0x7fa1c400bc80 comp rx=0 tx=0).stop 2024-05-27T07:21:44.541 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:21:44.551+0000 7fa1d2f93700 1 -- 172.21.15.52:0/2248019265 shutdown_connections 2024-05-27T07:21:44.541 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:21:44.551+0000 7fa1d2f93700 1 --2- 172.21.15.52:0/2248019265 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fa1cc108730 0x7fa1cc19c760 unknown :-1 s=CLOSED pgs=182 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:21:44.541 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:21:44.551+0000 7fa1d2f93700 1 --2- 172.21.15.52:0/2248019265 >> [v2:172.21.15.52:6800/3513075805,v1:172.21.15.52:6801/3513075805] conn(0x7fa1bc0b2880 0x7fa1bc0b4d90 unknown :-1 s=CLOSED pgs=141 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:21:44.541 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:21:44.551+0000 7fa1d2f93700 1 --2- 172.21.15.52:0/2248019265 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fa1cc1034b0 0x7fa1cc19bce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:21:44.542 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:21:44.551+0000 7fa1d2f93700 1 --2- 172.21.15.52:0/2248019265 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fa1cc107de0 0x7fa1cc19c220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:21:44.542 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:21:44.551+0000 7fa1d2f93700 1 -- 172.21.15.52:0/2248019265 >> 172.21.15.52:0/2248019265 conn(0x7fa1cc0fedb0 msgr2=0x7fa1cc1011d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:21:44.542 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:21:44.551+0000 7fa1d2f93700 1 -- 172.21.15.52:0/2248019265 shutdown_connections 2024-05-27T07:21:44.542 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:21:44.551+0000 7fa1d2f93700 1 -- 172.21.15.52:0/2248019265 wait complete. 2024-05-27T07:21:44.545 INFO:teuthology.orchestra.run.smithi052.stderr:pool 'cephfs_metadata' created 2024-05-27T07:21:44.852 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:21:44 smithi173 ceph-mon[37000]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' 2024-05-27T07:21:44.852 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:21:44 smithi173 ceph-mon[37000]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' 2024-05-27T07:21:44.853 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:21:44 smithi173 ceph-mon[37000]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' 2024-05-27T07:21:44.853 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:21:44 smithi173 ceph-mon[37000]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' 2024-05-27T07:21:44.853 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:21:44 smithi173 ceph-mon[37000]: pgmap v279: 1 pgs: 1 active+clean; 577 KiB data, 354 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-27T07:21:44.853 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:21:44 smithi173 ceph-mon[37000]: from='client.? 172.21.15.52:0/2248019265' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]: dispatch 2024-05-27T07:21:44.904 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:21:44 smithi052 ceph-mon[31955]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' 2024-05-27T07:21:44.904 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:21:44 smithi052 ceph-mon[31955]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' 2024-05-27T07:21:44.904 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:21:44 smithi052 ceph-mon[31955]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' 2024-05-27T07:21:44.904 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:21:44 smithi052 ceph-mon[31955]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' 2024-05-27T07:21:44.904 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:21:44 smithi052 ceph-mon[31955]: pgmap v279: 1 pgs: 1 active+clean; 577 KiB data, 354 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-27T07:21:44.904 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:21:44 smithi052 ceph-mon[31955]: from='client.? 172.21.15.52:0/2248019265' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]: dispatch 2024-05-27T07:21:44.940 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:21:44 smithi105 ceph-mon[35977]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' 2024-05-27T07:21:44.940 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:21:44 smithi105 ceph-mon[35977]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' 2024-05-27T07:21:44.940 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:21:44 smithi105 ceph-mon[35977]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' 2024-05-27T07:21:44.940 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:21:44 smithi105 ceph-mon[35977]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' 2024-05-27T07:21:44.940 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:21:44 smithi105 ceph-mon[35977]: pgmap v279: 1 pgs: 1 active+clean; 577 KiB data, 354 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-27T07:21:44.940 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:21:44 smithi105 ceph-mon[35977]: from='client.? 172.21.15.52:0/2248019265' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]: dispatch 2024-05-27T07:21:45.225 DEBUG:teuthology.orchestra.run.smithi052:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:38f9d900c83c63799fbdbe61acc9a11b0d3554a6 shell --fsid 73df2636-1bf8-11ef-bc9b-c7b262605968 -- ceph osd pool create cephfs_data 64 --pg_num_min 64 --target_size_ratio 0.05 2024-05-27T07:21:45.474 INFO:teuthology.orchestra.run.smithi052.stderr:Inferring config /var/lib/ceph/73df2636-1bf8-11ef-bc9b-c7b262605968/mon.a/config 2024-05-27T07:21:45.816 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:21:45 smithi052 ceph-mon[31955]: from='client.? 172.21.15.52:0/2248019265' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]': finished 2024-05-27T07:21:45.816 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:21:45 smithi052 ceph-mon[31955]: osdmap e73: 12 total, 12 up, 12 in 2024-05-27T07:21:45.816 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:21:45 smithi052 ceph-mon[31955]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' 2024-05-27T07:21:45.816 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:21:45 smithi052 ceph-mon[31955]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' 2024-05-27T07:21:45.816 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:21:45 smithi052 ceph-mon[31955]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T07:21:45.816 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:21:45 smithi052 ceph-mon[31955]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-27T07:21:45.817 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:21:45 smithi052 ceph-mon[31955]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' 2024-05-27T07:21:45.842 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:21:45 smithi105 ceph-mon[35977]: from='client.? 172.21.15.52:0/2248019265' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]': finished 2024-05-27T07:21:45.842 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:21:45 smithi105 ceph-mon[35977]: osdmap e73: 12 total, 12 up, 12 in 2024-05-27T07:21:45.842 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:21:45 smithi105 ceph-mon[35977]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' 2024-05-27T07:21:45.842 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:21:45 smithi105 ceph-mon[35977]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' 2024-05-27T07:21:45.842 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:21:45 smithi105 ceph-mon[35977]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T07:21:45.842 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:21:45 smithi105 ceph-mon[35977]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-27T07:21:45.842 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:21:45 smithi105 ceph-mon[35977]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' 2024-05-27T07:21:45.852 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:21:45 smithi173 ceph-mon[37000]: from='client.? 172.21.15.52:0/2248019265' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]': finished 2024-05-27T07:21:45.852 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:21:45 smithi173 ceph-mon[37000]: osdmap e73: 12 total, 12 up, 12 in 2024-05-27T07:21:45.852 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:21:45 smithi173 ceph-mon[37000]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' 2024-05-27T07:21:45.852 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:21:45 smithi173 ceph-mon[37000]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' 2024-05-27T07:21:45.852 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:21:45 smithi173 ceph-mon[37000]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T07:21:45.852 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:21:45 smithi173 ceph-mon[37000]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-27T07:21:45.853 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:21:45 smithi173 ceph-mon[37000]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' 2024-05-27T07:21:46.833 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:21:46 smithi052 ceph-mon[31955]: osdmap e74: 12 total, 12 up, 12 in 2024-05-27T07:21:46.833 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:21:46 smithi052 ceph-mon[31955]: pgmap v282: 33 pgs: 14 creating+peering, 1 active+clean, 18 unknown; 577 KiB data, 354 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-27T07:21:46.833 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:21:46 smithi052 ceph-mon[31955]: osdmap e75: 12 total, 12 up, 12 in 2024-05-27T07:21:46.852 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:21:46 smithi173 ceph-mon[37000]: osdmap e74: 12 total, 12 up, 12 in 2024-05-27T07:21:46.852 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:21:46 smithi173 ceph-mon[37000]: pgmap v282: 33 pgs: 14 creating+peering, 1 active+clean, 18 unknown; 577 KiB data, 354 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-27T07:21:46.852 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:21:46 smithi173 ceph-mon[37000]: osdmap e75: 12 total, 12 up, 12 in 2024-05-27T07:21:46.852 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:21:46 smithi105 ceph-mon[35977]: osdmap e74: 12 total, 12 up, 12 in 2024-05-27T07:21:46.853 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:21:46 smithi105 ceph-mon[35977]: pgmap v282: 33 pgs: 14 creating+peering, 1 active+clean, 18 unknown; 577 KiB data, 354 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-27T07:21:46.853 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:21:46 smithi105 ceph-mon[35977]: osdmap e75: 12 total, 12 up, 12 in 2024-05-27T07:21:47.099 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:21:47.108+0000 7fa0879b3700 1 -- 172.21.15.52:0/3021397449 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fa0801045b0 msgr2=0x7fa0801049c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:21:47.099 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:21:47.108+0000 7fa0879b3700 1 --2- 172.21.15.52:0/3021397449 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fa0801045b0 0x7fa0801049c0 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7fa07c0099c0 tx=0x7fa07c009cd0 comp rx=0 tx=0).stop 2024-05-27T07:21:47.099 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:21:47.108+0000 7fa0879b3700 1 -- 172.21.15.52:0/3021397449 shutdown_connections 2024-05-27T07:21:47.099 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:21:47.108+0000 7fa0879b3700 1 --2- 172.21.15.52:0/3021397449 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fa080109830 0x7fa080111190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:21:47.099 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:21:47.108+0000 7fa0879b3700 1 --2- 172.21.15.52:0/3021397449 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fa0801045b0 0x7fa0801049c0 unknown :-1 s=CLOSED pgs=183 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:21:47.099 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:21:47.108+0000 7fa0879b3700 1 --2- 172.21.15.52:0/3021397449 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fa080108ee0 0x7fa0801092f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:21:47.099 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:21:47.108+0000 7fa0879b3700 1 -- 172.21.15.52:0/3021397449 >> 172.21.15.52:0/3021397449 conn(0x7fa0800ffec0 msgr2=0x7fa080102310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:21:47.099 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:21:47.108+0000 7fa0879b3700 1 -- 172.21.15.52:0/3021397449 shutdown_connections 2024-05-27T07:21:47.099 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:21:47.108+0000 7fa0879b3700 1 -- 172.21.15.52:0/3021397449 wait complete. 2024-05-27T07:21:47.101 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:21:47.109+0000 7fa0879b3700 1 Processor -- start 2024-05-27T07:21:47.101 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:21:47.109+0000 7fa0879b3700 1 -- start start 2024-05-27T07:21:47.101 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:21:47.110+0000 7fa0879b3700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fa0801045b0 0x7fa080078760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:21:47.101 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:21:47.110+0000 7fa0879b3700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fa080108ee0 0x7fa080076bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:21:47.101 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:21:47.110+0000 7fa0879b3700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fa080109830 0x7fa0800771a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:21:47.101 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:21:47.110+0000 7fa0879b3700 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x7fa0801144d0 con 0x7fa080108ee0 2024-05-27T07:21:47.101 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:21:47.110+0000 7fa0879b3700 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7fa080114350 con 0x7fa0801045b0 2024-05-27T07:21:47.102 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:21:47.110+0000 7fa0879b3700 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7fa080114650 con 0x7fa080109830 2024-05-27T07:21:47.102 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:21:47.110+0000 7fa084f4e700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fa080108ee0 0x7fa080076bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:21:47.102 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:21:47.110+0000 7fa084f4e700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fa080108ee0 0x7fa080076bc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.52:54430/0 (socket says 172.21.15.52:54430) 2024-05-27T07:21:47.102 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:21:47.110+0000 7fa084f4e700 1 -- 172.21.15.52:0/1426102038 learned_addr learned my addr 172.21.15.52:0/1426102038 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-05-27T07:21:47.102 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:21:47.110+0000 7fa085f50700 1 --2- 172.21.15.52:0/1426102038 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fa080109830 0x7fa0800771a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:21:47.102 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:21:47.110+0000 7fa08574f700 1 --2- 172.21.15.52:0/1426102038 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fa0801045b0 0x7fa080078760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:21:47.102 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:21:47.110+0000 7fa084f4e700 1 -- 172.21.15.52:0/1426102038 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fa080109830 msgr2=0x7fa0800771a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:21:47.102 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:21:47.110+0000 7fa084f4e700 1 --2- 172.21.15.52:0/1426102038 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fa080109830 0x7fa0800771a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:21:47.102 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:21:47.110+0000 7fa084f4e700 1 -- 172.21.15.52:0/1426102038 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fa0801045b0 msgr2=0x7fa080078760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:21:47.102 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:21:47.110+0000 7fa084f4e700 1 --2- 172.21.15.52:0/1426102038 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fa0801045b0 0x7fa080078760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:21:47.102 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:21:47.110+0000 7fa084f4e700 1 -- 172.21.15.52:0/1426102038 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa080077940 con 0x7fa080108ee0 2024-05-27T07:21:47.102 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:21:47.110+0000 7fa08574f700 1 --2- 172.21.15.52:0/1426102038 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fa0801045b0 0x7fa080078760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-27T07:21:47.103 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:21:47.110+0000 7fa084f4e700 1 --2- 172.21.15.52:0/1426102038 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fa080108ee0 0x7fa080076bc0 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7fa07400b500 tx=0x7fa07400b810 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:21:47.103 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:21:47.111+0000 7fa0727fc700 1 -- 172.21.15.52:0/1426102038 <== mon.0 v2:172.21.15.52:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa074015020 con 0x7fa080108ee0 2024-05-27T07:21:47.103 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:21:47.111+0000 7fa0879b3700 1 -- 172.21.15.52:0/1426102038 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa0801ad870 con 0x7fa080108ee0 2024-05-27T07:21:47.103 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:21:47.111+0000 7fa0727fc700 1 -- 172.21.15.52:0/1426102038 <== mon.0 v2:172.21.15.52:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fa074011ce0 con 0x7fa080108ee0 2024-05-27T07:21:47.103 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:21:47.111+0000 7fa0727fc700 1 -- 172.21.15.52:0/1426102038 <== mon.0 v2:172.21.15.52:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa074010d40 con 0x7fa080108ee0 2024-05-27T07:21:47.103 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:21:47.111+0000 7fa0879b3700 1 -- 172.21.15.52:0/1426102038 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa0801ade00 con 0x7fa080108ee0 2024-05-27T07:21:47.103 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:21:47.113+0000 7fa0879b3700 1 -- 172.21.15.52:0/1426102038 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa080105a60 con 0x7fa080108ee0 2024-05-27T07:21:47.104 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:21:47.114+0000 7fa0727fc700 1 -- 172.21.15.52:0/1426102038 <== mon.0 v2:172.21.15.52:3300/0 4 ==== mgrmap(e 16) ==== 149652+0+0 (secure 0 0 0) 0x7fa074018020 con 0x7fa080108ee0 2024-05-27T07:21:47.105 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:21:47.115+0000 7fa0727fc700 1 --2- 172.21.15.52:0/1426102038 >> [v2:172.21.15.52:6800/3513075805,v1:172.21.15.52:6801/3513075805] conn(0x7fa06c0b2900 0x7fa06c0b4e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:21:47.105 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:21:47.115+0000 7fa08574f700 1 --2- 172.21.15.52:0/1426102038 >> [v2:172.21.15.52:6800/3513075805,v1:172.21.15.52:6801/3513075805] conn(0x7fa06c0b2900 0x7fa06c0b4e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:21:47.105 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:21:47.115+0000 7fa0727fc700 1 -- 172.21.15.52:0/1426102038 <== mon.0 v2:172.21.15.52:3300/0 5 ==== osd_map(75..75 src has 1..75) ==== 7497+0+0 (secure 0 0 0) 0x7fa0740e15a0 con 0x7fa080108ee0 2024-05-27T07:21:47.106 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:21:47.116+0000 7fa08574f700 1 --2- 172.21.15.52:0/1426102038 >> [v2:172.21.15.52:6800/3513075805,v1:172.21.15.52:6801/3513075805] conn(0x7fa06c0b2900 0x7fa06c0b4e10 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7fa07c009fd0 tx=0x7fa07c01a040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:21:47.110 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:21:47.120+0000 7fa0727fc700 1 -- 172.21.15.52:0/1426102038 <== mon.0 v2:172.21.15.52:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191963 (secure 0 0 0) 0x7fa07401f020 con 0x7fa080108ee0 2024-05-27T07:21:47.326 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:21:47.336+0000 7fa0879b3700 1 -- 172.21.15.52:0/1426102038 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.05} v 0) -- 0x7fa08002e5e0 con 0x7fa080108ee0 2024-05-27T07:21:47.557 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:21:47.567+0000 7fa0727fc700 1 -- 172.21.15.52:0/1426102038 <== mon.0 v2:172.21.15.52:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.05}]=0 pool 'cephfs_data' created v76) ==== 171+0+0 (secure 0 0 0) 0x7fa0740a95f0 con 0x7fa080108ee0 2024-05-27T07:21:47.559 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:21:47.569+0000 7fa0879b3700 1 -- 172.21.15.52:0/1426102038 >> [v2:172.21.15.52:6800/3513075805,v1:172.21.15.52:6801/3513075805] conn(0x7fa06c0b2900 msgr2=0x7fa06c0b4e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:21:47.560 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:21:47.569+0000 7fa0879b3700 1 --2- 172.21.15.52:0/1426102038 >> [v2:172.21.15.52:6800/3513075805,v1:172.21.15.52:6801/3513075805] conn(0x7fa06c0b2900 0x7fa06c0b4e10 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7fa07c009fd0 tx=0x7fa07c01a040 comp rx=0 tx=0).stop 2024-05-27T07:21:47.560 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:21:47.569+0000 7fa0879b3700 1 -- 172.21.15.52:0/1426102038 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fa080108ee0 msgr2=0x7fa080076bc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:21:47.560 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:21:47.569+0000 7fa0879b3700 1 --2- 172.21.15.52:0/1426102038 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fa080108ee0 0x7fa080076bc0 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7fa07400b500 tx=0x7fa07400b810 comp rx=0 tx=0).stop 2024-05-27T07:21:47.560 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:21:47.569+0000 7fa0879b3700 1 -- 172.21.15.52:0/1426102038 shutdown_connections 2024-05-27T07:21:47.560 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:21:47.569+0000 7fa0879b3700 1 --2- 172.21.15.52:0/1426102038 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fa080109830 0x7fa0800771a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:21:47.560 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:21:47.569+0000 7fa0879b3700 1 --2- 172.21.15.52:0/1426102038 >> [v2:172.21.15.52:6800/3513075805,v1:172.21.15.52:6801/3513075805] conn(0x7fa06c0b2900 0x7fa06c0b4e10 unknown :-1 s=CLOSED pgs=142 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:21:47.560 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:21:47.569+0000 7fa0879b3700 1 --2- 172.21.15.52:0/1426102038 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fa0801045b0 0x7fa080078760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:21:47.560 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:21:47.569+0000 7fa0879b3700 1 --2- 172.21.15.52:0/1426102038 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fa080108ee0 0x7fa080076bc0 unknown :-1 s=CLOSED pgs=184 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:21:47.560 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:21:47.569+0000 7fa0879b3700 1 -- 172.21.15.52:0/1426102038 >> 172.21.15.52:0/1426102038 conn(0x7fa0800ffec0 msgr2=0x7fa080102e50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:21:47.561 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:21:47.571+0000 7fa0879b3700 1 -- 172.21.15.52:0/1426102038 shutdown_connections 2024-05-27T07:21:47.561 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:21:47.571+0000 7fa0879b3700 1 -- 172.21.15.52:0/1426102038 wait complete. 2024-05-27T07:21:47.563 INFO:teuthology.orchestra.run.smithi052.stderr:pool 'cephfs_data' created 2024-05-27T07:21:47.729 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:21:47 smithi052 ceph-mon[31955]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-05-27T07:21:47.729 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:21:47 smithi052 ceph-mon[31955]: from='client.? 172.21.15.52:0/1426102038' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.05}]: dispatch 2024-05-27T07:21:47.852 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:21:47 smithi173 ceph-mon[37000]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-05-27T07:21:47.852 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:21:47 smithi173 ceph-mon[37000]: from='client.? 172.21.15.52:0/1426102038' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.05}]: dispatch 2024-05-27T07:21:47.940 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:21:47 smithi105 ceph-mon[35977]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-05-27T07:21:47.940 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:21:47 smithi105 ceph-mon[35977]: from='client.? 172.21.15.52:0/1426102038' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.05}]: dispatch 2024-05-27T07:21:48.260 DEBUG:teuthology.orchestra.run.smithi052:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:38f9d900c83c63799fbdbe61acc9a11b0d3554a6 shell --fsid 73df2636-1bf8-11ef-bc9b-c7b262605968 -- ceph fs new cephfs cephfs_metadata cephfs_data 2024-05-27T07:21:48.513 INFO:teuthology.orchestra.run.smithi052.stderr:Inferring config /var/lib/ceph/73df2636-1bf8-11ef-bc9b-c7b262605968/mon.a/config 2024-05-27T07:21:48.576 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:21:48 smithi052 ceph-mon[31955]: from='client.? 172.21.15.52:0/1426102038' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.05}]': finished 2024-05-27T07:21:48.577 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:21:48 smithi052 ceph-mon[31955]: osdmap e76: 12 total, 12 up, 12 in 2024-05-27T07:21:48.577 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:21:48 smithi052 ceph-mon[31955]: pgmap v285: 97 pgs: 1 creating+activating, 14 creating+peering, 12 active+clean, 70 unknown; 577 KiB data, 354 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-27T07:21:48.577 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:21:48 smithi052 ceph-mon[31955]: osdmap e77: 12 total, 12 up, 12 in 2024-05-27T07:21:48.852 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:21:48 smithi173 ceph-mon[37000]: from='client.? 172.21.15.52:0/1426102038' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.05}]': finished 2024-05-27T07:21:48.852 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:21:48 smithi173 ceph-mon[37000]: osdmap e76: 12 total, 12 up, 12 in 2024-05-27T07:21:48.853 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:21:48 smithi173 ceph-mon[37000]: pgmap v285: 97 pgs: 1 creating+activating, 14 creating+peering, 12 active+clean, 70 unknown; 577 KiB data, 354 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-27T07:21:48.853 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:21:48 smithi173 ceph-mon[37000]: osdmap e77: 12 total, 12 up, 12 in 2024-05-27T07:21:48.940 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:21:48 smithi105 ceph-mon[35977]: from='client.? 172.21.15.52:0/1426102038' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.05}]': finished 2024-05-27T07:21:48.940 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:21:48 smithi105 ceph-mon[35977]: osdmap e76: 12 total, 12 up, 12 in 2024-05-27T07:21:48.940 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:21:48 smithi105 ceph-mon[35977]: pgmap v285: 97 pgs: 1 creating+activating, 14 creating+peering, 12 active+clean, 70 unknown; 577 KiB data, 354 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-27T07:21:48.940 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:21:48 smithi105 ceph-mon[35977]: osdmap e77: 12 total, 12 up, 12 in 2024-05-27T07:21:50.243 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:21:50.252+0000 7f9e5e27e700 1 -- 172.21.15.52:0/28722903 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f9e58107de0 msgr2=0x7f9e581081f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:21:50.243 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:21:50.252+0000 7f9e5e27e700 1 --2- 172.21.15.52:0/28722903 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f9e58107de0 0x7f9e581081f0 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7f9e400099c0 tx=0x7f9e40009cd0 comp rx=0 tx=0).stop 2024-05-27T07:21:50.243 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:21:50.252+0000 7f9e5e27e700 1 -- 172.21.15.52:0/28722903 shutdown_connections 2024-05-27T07:21:50.243 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:21:50.252+0000 7f9e5e27e700 1 --2- 172.21.15.52:0/28722903 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f9e58108730 0x7f9e58110090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:21:50.243 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:21:50.252+0000 7f9e5e27e700 1 --2- 172.21.15.52:0/28722903 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f9e581034b0 0x7f9e581038c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:21:50.243 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:21:50.252+0000 7f9e5e27e700 1 --2- 172.21.15.52:0/28722903 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f9e58107de0 0x7f9e581081f0 unknown :-1 s=CLOSED pgs=185 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:21:50.244 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:21:50.252+0000 7f9e5e27e700 1 -- 172.21.15.52:0/28722903 >> 172.21.15.52:0/28722903 conn(0x7f9e580fedb0 msgr2=0x7f9e581011d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:21:50.244 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:21:50.252+0000 7f9e5e27e700 1 -- 172.21.15.52:0/28722903 shutdown_connections 2024-05-27T07:21:50.244 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:21:50.252+0000 7f9e5e27e700 1 -- 172.21.15.52:0/28722903 wait complete. 2024-05-27T07:21:50.244 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:21:50.252+0000 7f9e5e27e700 1 Processor -- start 2024-05-27T07:21:50.244 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:21:50.252+0000 7f9e5e27e700 1 -- start start 2024-05-27T07:21:50.244 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:21:50.253+0000 7f9e5e27e700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f9e581034b0 0x7f9e5819ae20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:21:50.244 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:21:50.253+0000 7f9e5e27e700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f9e58107de0 0x7f9e5819b3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:21:50.244 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:21:50.253+0000 7f9e5e27e700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f9e58108730 0x7f9e5819f650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:21:50.244 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:21:50.253+0000 7f9e5e27e700 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x7f9e58113340 con 0x7f9e58108730 2024-05-27T07:21:50.244 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:21:50.253+0000 7f9e5e27e700 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f9e581131c0 con 0x7f9e581034b0 2024-05-27T07:21:50.244 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:21:50.253+0000 7f9e5e27e700 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f9e581134c0 con 0x7f9e58107de0 2024-05-27T07:21:50.244 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:21:50.253+0000 7f9e5c81b700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f9e58108730 0x7f9e5819f650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:21:50.244 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:21:50.253+0000 7f9e57fff700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f9e581034b0 0x7f9e5819ae20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:21:50.245 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:21:50.253+0000 7f9e4ffff700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f9e58107de0 0x7f9e5819b3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:21:50.245 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:21:50.253+0000 7f9e5c81b700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f9e58108730 0x7f9e5819f650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.52:54446/0 (socket says 172.21.15.52:54446) 2024-05-27T07:21:50.245 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:21:50.253+0000 7f9e57fff700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f9e581034b0 0x7f9e5819ae20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.52:33446/0 (socket says 172.21.15.52:33446) 2024-05-27T07:21:50.245 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:21:50.253+0000 7f9e5c81b700 1 -- 172.21.15.52:0/1760160210 learned_addr learned my addr 172.21.15.52:0/1760160210 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-05-27T07:21:50.246 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:21:50.253+0000 7f9e5c81b700 1 -- 172.21.15.52:0/1760160210 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f9e58107de0 msgr2=0x7f9e5819b3c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:21:50.246 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:21:50.253+0000 7f9e5c81b700 1 --2- 172.21.15.52:0/1760160210 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f9e58107de0 0x7f9e5819b3c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:21:50.246 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:21:50.253+0000 7f9e5c81b700 1 -- 172.21.15.52:0/1760160210 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f9e581034b0 msgr2=0x7f9e5819ae20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:21:50.246 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:21:50.253+0000 7f9e5c81b700 1 --2- 172.21.15.52:0/1760160210 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f9e581034b0 0x7f9e5819ae20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:21:50.246 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:21:50.253+0000 7f9e5c81b700 1 -- 172.21.15.52:0/1760160210 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9e5819fcf0 con 0x7f9e58108730 2024-05-27T07:21:50.246 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:21:50.254+0000 7f9e5c81b700 1 --2- 172.21.15.52:0/1760160210 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f9e58108730 0x7f9e5819f650 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7f9e5000ada0 tx=0x7f9e5000c330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:21:50.246 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:21:50.254+0000 7f9e55ffb700 1 -- 172.21.15.52:0/1760160210 <== mon.0 v2:172.21.15.52:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9e50017020 con 0x7f9e58108730 2024-05-27T07:21:50.246 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:21:50.254+0000 7f9e5e27e700 1 -- 172.21.15.52:0/1760160210 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9e5819ffe0 con 0x7f9e58108730 2024-05-27T07:21:50.246 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:21:50.254+0000 7f9e55ffb700 1 -- 172.21.15.52:0/1760160210 <== mon.0 v2:172.21.15.52:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f9e5000cd80 con 0x7f9e58108730 2024-05-27T07:21:50.246 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:21:50.254+0000 7f9e55ffb700 1 -- 172.21.15.52:0/1760160210 <== mon.0 v2:172.21.15.52:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9e50005260 con 0x7f9e58108730 2024-05-27T07:21:50.246 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:21:50.254+0000 7f9e5e27e700 1 -- 172.21.15.52:0/1760160210 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9e581a7900 con 0x7f9e58108730 2024-05-27T07:21:50.246 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:21:50.255+0000 7f9e5e27e700 1 -- 172.21.15.52:0/1760160210 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9e580512b0 con 0x7f9e58108730 2024-05-27T07:21:50.247 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:21:50.257+0000 7f9e55ffb700 1 -- 172.21.15.52:0/1760160210 <== mon.0 v2:172.21.15.52:3300/0 4 ==== mgrmap(e 16) ==== 149652+0+0 (secure 0 0 0) 0x7f9e5000a090 con 0x7f9e58108730 2024-05-27T07:21:50.247 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:21:50.257+0000 7f9e55ffb700 1 --2- 172.21.15.52:0/1760160210 >> [v2:172.21.15.52:6800/3513075805,v1:172.21.15.52:6801/3513075805] conn(0x7f9e440b2900 0x7f9e440b4e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:21:50.248 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:21:50.258+0000 7f9e57fff700 1 --2- 172.21.15.52:0/1760160210 >> [v2:172.21.15.52:6800/3513075805,v1:172.21.15.52:6801/3513075805] conn(0x7f9e440b2900 0x7f9e440b4e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:21:50.248 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:21:50.258+0000 7f9e55ffb700 1 -- 172.21.15.52:0/1760160210 <== mon.0 v2:172.21.15.52:3300/0 5 ==== osd_map(78..78 src has 1..78) ==== 7879+0+0 (secure 0 0 0) 0x7f9e500e1c60 con 0x7f9e58108730 2024-05-27T07:21:50.248 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:21:50.258+0000 7f9e57fff700 1 --2- 172.21.15.52:0/1760160210 >> [v2:172.21.15.52:6800/3513075805,v1:172.21.15.52:6801/3513075805] conn(0x7f9e440b2900 0x7f9e440b4e10 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7f9e48005950 tx=0x7f9e4800a300 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:21:50.251 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:21:50.261+0000 7f9e55ffb700 1 -- 172.21.15.52:0/1760160210 <== mon.0 v2:172.21.15.52:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191963 (secure 0 0 0) 0x7f9e50021020 con 0x7f9e58108730 2024-05-27T07:21:50.510 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:21:50.520+0000 7f9e5e27e700 1 -- 172.21.15.52:0/1760160210 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_command({"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"} v 0) -- 0x7f9e5802e5e0 con 0x7f9e58108730 2024-05-27T07:21:50.562 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:21:50.572+0000 7f9e55ffb700 1 -- 172.21.15.52:0/1760160210 <== mon.0 v2:172.21.15.52:3300/0 7 ==== mon_command_ack([{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]=0 new fs with metadata pool 2 and data pool 3 v14) ==== 172+0+0 (secure 0 0 0) 0x7f9e500a9bc0 con 0x7f9e58108730 2024-05-27T07:21:50.565 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:21:50.575+0000 7f9e5e27e700 1 -- 172.21.15.52:0/1760160210 >> [v2:172.21.15.52:6800/3513075805,v1:172.21.15.52:6801/3513075805] conn(0x7f9e440b2900 msgr2=0x7f9e440b4e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:21:50.565 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:21:50.575+0000 7f9e5e27e700 1 --2- 172.21.15.52:0/1760160210 >> [v2:172.21.15.52:6800/3513075805,v1:172.21.15.52:6801/3513075805] conn(0x7f9e440b2900 0x7f9e440b4e10 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7f9e48005950 tx=0x7f9e4800a300 comp rx=0 tx=0).stop 2024-05-27T07:21:50.565 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:21:50.575+0000 7f9e5e27e700 1 -- 172.21.15.52:0/1760160210 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f9e58108730 msgr2=0x7f9e5819f650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:21:50.565 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:21:50.575+0000 7f9e5e27e700 1 --2- 172.21.15.52:0/1760160210 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f9e58108730 0x7f9e5819f650 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7f9e5000ada0 tx=0x7f9e5000c330 comp rx=0 tx=0).stop 2024-05-27T07:21:50.565 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:21:50.575+0000 7f9e5e27e700 1 -- 172.21.15.52:0/1760160210 shutdown_connections 2024-05-27T07:21:50.566 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:21:50.575+0000 7f9e5e27e700 1 --2- 172.21.15.52:0/1760160210 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f9e58108730 0x7f9e5819f650 unknown :-1 s=CLOSED pgs=186 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:21:50.566 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:21:50.575+0000 7f9e5e27e700 1 --2- 172.21.15.52:0/1760160210 >> [v2:172.21.15.52:6800/3513075805,v1:172.21.15.52:6801/3513075805] conn(0x7f9e440b2900 0x7f9e440b4e10 unknown :-1 s=CLOSED pgs=143 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:21:50.566 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:21:50.575+0000 7f9e5e27e700 1 --2- 172.21.15.52:0/1760160210 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f9e581034b0 0x7f9e5819ae20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:21:50.566 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:21:50.575+0000 7f9e5e27e700 1 --2- 172.21.15.52:0/1760160210 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f9e58107de0 0x7f9e5819b3c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:21:50.566 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:21:50.575+0000 7f9e5e27e700 1 -- 172.21.15.52:0/1760160210 >> 172.21.15.52:0/1760160210 conn(0x7f9e580fedb0 msgr2=0x7f9e5806b980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:21:50.566 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:21:50.575+0000 7f9e5e27e700 1 -- 172.21.15.52:0/1760160210 shutdown_connections 2024-05-27T07:21:50.566 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:21:50.575+0000 7f9e5e27e700 1 -- 172.21.15.52:0/1760160210 wait complete. 2024-05-27T07:21:50.566 INFO:teuthology.orchestra.run.smithi052.stderr:new fs with metadata pool 2 and data pool 3 2024-05-27T07:21:50.852 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:21:50 smithi173 ceph-mon[37000]: osdmap e78: 12 total, 12 up, 12 in 2024-05-27T07:21:50.852 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:21:50 smithi173 ceph-mon[37000]: pgmap v288: 97 pgs: 1 creating+activating, 15 creating+peering, 33 active+clean, 48 unknown; 577 KiB data, 354 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-27T07:21:50.852 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:21:50 smithi173 ceph-mon[37000]: from='client.? 172.21.15.52:0/1760160210' entity='client.admin' cmd=[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]: dispatch 2024-05-27T07:21:50.852 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:21:50 smithi173 ceph-mon[37000]: daemon mds.b assigned to filesystem cephfs as rank 0 2024-05-27T07:21:50.891 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:21:50 smithi052 ceph-mon[31955]: osdmap e78: 12 total, 12 up, 12 in 2024-05-27T07:21:50.891 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:21:50 smithi052 ceph-mon[31955]: pgmap v288: 97 pgs: 1 creating+activating, 15 creating+peering, 33 active+clean, 48 unknown; 577 KiB data, 354 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-27T07:21:50.891 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:21:50 smithi052 ceph-mon[31955]: from='client.? 172.21.15.52:0/1760160210' entity='client.admin' cmd=[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]: dispatch 2024-05-27T07:21:50.891 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:21:50 smithi052 ceph-mon[31955]: daemon mds.b assigned to filesystem cephfs as rank 0 2024-05-27T07:21:50.940 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:21:50 smithi105 ceph-mon[35977]: osdmap e78: 12 total, 12 up, 12 in 2024-05-27T07:21:50.940 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:21:50 smithi105 ceph-mon[35977]: pgmap v288: 97 pgs: 1 creating+activating, 15 creating+peering, 33 active+clean, 48 unknown; 577 KiB data, 354 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-27T07:21:50.940 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:21:50 smithi105 ceph-mon[35977]: from='client.? 172.21.15.52:0/1760160210' entity='client.admin' cmd=[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]: dispatch 2024-05-27T07:21:50.940 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:21:50 smithi105 ceph-mon[35977]: daemon mds.b assigned to filesystem cephfs as rank 0 2024-05-27T07:21:51.204 DEBUG:tasks.cephfs.filesystem:EC profile is ['m=2', 'k=2', 'crush-failure-domain=osd'] 2024-05-27T07:21:51.205 DEBUG:teuthology.orchestra.run.smithi052:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:38f9d900c83c63799fbdbe61acc9a11b0d3554a6 shell --fsid 73df2636-1bf8-11ef-bc9b-c7b262605968 -- ceph osd erasure-code-profile set cephfs_data_ec m=2 k=2 crush-failure-domain=osd 2024-05-27T07:21:51.458 INFO:teuthology.orchestra.run.smithi052.stderr:Inferring config /var/lib/ceph/73df2636-1bf8-11ef-bc9b-c7b262605968/mon.a/config 2024-05-27T07:21:51.823 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:21:51 smithi105 ceph-mon[35977]: from='client.? 172.21.15.52:0/1760160210' entity='client.admin' cmd='[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]': finished 2024-05-27T07:21:51.824 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:21:51 smithi105 ceph-mon[35977]: osdmap e79: 12 total, 12 up, 12 in 2024-05-27T07:21:51.824 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:21:51 smithi105 ceph-mon[35977]: fsmap cephfs:1 {0=b=up:creating} 11 up:standby 2024-05-27T07:21:51.824 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:21:51 smithi105 ceph-mon[35977]: daemon mds.b is now active in filesystem cephfs as rank 0 2024-05-27T07:21:51.852 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:21:51 smithi173 ceph-mon[37000]: from='client.? 172.21.15.52:0/1760160210' entity='client.admin' cmd='[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]': finished 2024-05-27T07:21:51.852 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:21:51 smithi173 ceph-mon[37000]: osdmap e79: 12 total, 12 up, 12 in 2024-05-27T07:21:51.852 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:21:51 smithi173 ceph-mon[37000]: fsmap cephfs:1 {0=b=up:creating} 11 up:standby 2024-05-27T07:21:51.852 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:21:51 smithi173 ceph-mon[37000]: daemon mds.b is now active in filesystem cephfs as rank 0 2024-05-27T07:21:51.860 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:21:51 smithi052 ceph-mon[31955]: from='client.? 172.21.15.52:0/1760160210' entity='client.admin' cmd='[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]': finished 2024-05-27T07:21:51.860 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:21:51 smithi052 ceph-mon[31955]: osdmap e79: 12 total, 12 up, 12 in 2024-05-27T07:21:51.861 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:21:51 smithi052 ceph-mon[31955]: fsmap cephfs:1 {0=b=up:creating} 11 up:standby 2024-05-27T07:21:51.861 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:21:51 smithi052 ceph-mon[31955]: daemon mds.b is now active in filesystem cephfs as rank 0 2024-05-27T07:21:52.190 INFO:journalctl@ceph.mds.b.smithi105.stdout:May 27 07:21:51 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-b[65543]: 2024-05-27T07:21:51.832+0000 7f33fa6cb700 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-05-27T07:21:52.615 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:21:52 smithi052 ceph-mon[31955]: mds.? [v2:172.21.15.105:6838/500348151,v1:172.21.15.105:6839/500348151] up:active 2024-05-27T07:21:52.616 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:21:52 smithi052 ceph-mon[31955]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-05-27T07:21:52.616 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:21:52 smithi052 ceph-mon[31955]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-05-27T07:21:52.616 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:21:52 smithi052 ceph-mon[31955]: pgmap v290: 97 pgs: 4 creating+peering, 75 active+clean, 18 unknown; 577 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 0 op/s 2024-05-27T07:21:52.852 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:21:52 smithi173 ceph-mon[37000]: mds.? [v2:172.21.15.105:6838/500348151,v1:172.21.15.105:6839/500348151] up:active 2024-05-27T07:21:52.852 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:21:52 smithi173 ceph-mon[37000]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-05-27T07:21:52.852 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:21:52 smithi173 ceph-mon[37000]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-05-27T07:21:52.852 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:21:52 smithi173 ceph-mon[37000]: pgmap v290: 97 pgs: 4 creating+peering, 75 active+clean, 18 unknown; 577 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 0 op/s 2024-05-27T07:21:52.940 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:21:52 smithi105 ceph-mon[35977]: mds.? [v2:172.21.15.105:6838/500348151,v1:172.21.15.105:6839/500348151] up:active 2024-05-27T07:21:52.940 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:21:52 smithi105 ceph-mon[35977]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-05-27T07:21:52.940 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:21:52 smithi105 ceph-mon[35977]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-05-27T07:21:52.940 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:21:52 smithi105 ceph-mon[35977]: pgmap v290: 97 pgs: 4 creating+peering, 75 active+clean, 18 unknown; 577 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 0 op/s 2024-05-27T07:21:53.153 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:21:53.159+0000 7fc897735700 1 -- 172.21.15.52:0/631364741 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fc890076e30 msgr2=0x7fc890077240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:21:53.153 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:21:53.159+0000 7fc897735700 1 --2- 172.21.15.52:0/631364741 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fc890076e30 0x7fc890077240 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7fc88c009990 tx=0x7fc88c009ca0 comp rx=0 tx=0).stop 2024-05-27T07:21:53.153 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:21:53.159+0000 7fc897735700 1 -- 172.21.15.52:0/631364741 shutdown_connections 2024-05-27T07:21:53.153 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:21:53.159+0000 7fc897735700 1 --2- 172.21.15.52:0/631364741 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fc890110b70 0x7fc890112fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:21:53.154 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:21:53.159+0000 7fc897735700 1 --2- 172.21.15.52:0/631364741 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fc890076e30 0x7fc890077240 unknown :-1 s=CLOSED pgs=187 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:21:53.154 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:21:53.159+0000 7fc897735700 1 --2- 172.21.15.52:0/631364741 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fc890077780 0x7fc890110630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:21:53.154 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:21:53.159+0000 7fc897735700 1 -- 172.21.15.52:0/631364741 >> 172.21.15.52:0/631364741 conn(0x7fc8900ffe90 msgr2=0x7fc8901022e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:21:53.154 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:21:53.159+0000 7fc897735700 1 -- 172.21.15.52:0/631364741 shutdown_connections 2024-05-27T07:21:53.154 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:21:53.159+0000 7fc897735700 1 -- 172.21.15.52:0/631364741 wait complete. 2024-05-27T07:21:53.154 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:21:53.160+0000 7fc897735700 1 Processor -- start 2024-05-27T07:21:53.155 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:21:53.160+0000 7fc897735700 1 -- start start 2024-05-27T07:21:53.155 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:21:53.160+0000 7fc897735700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fc890076e30 0x7fc89019d010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:21:53.155 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:21:53.161+0000 7fc897735700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fc890077780 0x7fc89019d5b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:21:53.155 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:21:53.161+0000 7fc897735700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fc890110b70 0x7fc8901a1840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:21:53.155 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:21:53.161+0000 7fc897735700 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x7fc890116660 con 0x7fc890076e30 2024-05-27T07:21:53.155 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:21:53.161+0000 7fc897735700 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7fc8901164e0 con 0x7fc890077780 2024-05-27T07:21:53.155 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:21:53.161+0000 7fc897735700 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7fc8901167e0 con 0x7fc890110b70 2024-05-27T07:21:53.156 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:21:53.161+0000 7fc8954d1700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fc890076e30 0x7fc89019d010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:21:53.156 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:21:53.161+0000 7fc8954d1700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fc890076e30 0x7fc89019d010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.52:50368/0 (socket says 172.21.15.52:50368) 2024-05-27T07:21:53.156 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:21:53.161+0000 7fc8954d1700 1 -- 172.21.15.52:0/2854547998 learned_addr learned my addr 172.21.15.52:0/2854547998 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-05-27T07:21:53.156 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:21:53.161+0000 7fc895cd2700 1 --2- 172.21.15.52:0/2854547998 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fc890110b70 0x7fc8901a1840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:21:53.156 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:21:53.161+0000 7fc894cd0700 1 --2- 172.21.15.52:0/2854547998 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fc890077780 0x7fc89019d5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:21:53.156 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:21:53.161+0000 7fc8954d1700 1 -- 172.21.15.52:0/2854547998 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fc890110b70 msgr2=0x7fc8901a1840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:21:53.157 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:21:53.161+0000 7fc8954d1700 1 --2- 172.21.15.52:0/2854547998 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fc890110b70 0x7fc8901a1840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:21:53.157 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:21:53.161+0000 7fc8954d1700 1 -- 172.21.15.52:0/2854547998 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fc890077780 msgr2=0x7fc89019d5b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:21:53.157 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:21:53.161+0000 7fc8954d1700 1 --2- 172.21.15.52:0/2854547998 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fc890077780 0x7fc89019d5b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:21:53.157 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:21:53.161+0000 7fc8954d1700 1 -- 172.21.15.52:0/2854547998 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc8901a1ee0 con 0x7fc890076e30 2024-05-27T07:21:53.157 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:21:53.161+0000 7fc8954d1700 1 --2- 172.21.15.52:0/2854547998 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fc890076e30 0x7fc89019d010 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7fc88c000c00 tx=0x7fc88c0047b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:21:53.157 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:21:53.161+0000 7fc8867fc700 1 -- 172.21.15.52:0/2854547998 <== mon.0 v2:172.21.15.52:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc88c027070 con 0x7fc890076e30 2024-05-27T07:21:53.157 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:21:53.161+0000 7fc8867fc700 1 -- 172.21.15.52:0/2854547998 <== mon.0 v2:172.21.15.52:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fc88c003f20 con 0x7fc890076e30 2024-05-27T07:21:53.158 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:21:53.161+0000 7fc897735700 1 -- 172.21.15.52:0/2854547998 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc8901a2170 con 0x7fc890076e30 2024-05-27T07:21:53.158 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:21:53.161+0000 7fc8867fc700 1 -- 172.21.15.52:0/2854547998 <== mon.0 v2:172.21.15.52:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc88c022cf0 con 0x7fc890076e30 2024-05-27T07:21:53.158 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:21:53.161+0000 7fc897735700 1 -- 172.21.15.52:0/2854547998 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc8901a9af0 con 0x7fc890076e30 2024-05-27T07:21:53.158 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:21:53.162+0000 7fc897735700 1 -- 172.21.15.52:0/2854547998 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc8900512b0 con 0x7fc890076e30 2024-05-27T07:21:53.160 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:21:53.164+0000 7fc8867fc700 1 -- 172.21.15.52:0/2854547998 <== mon.0 v2:172.21.15.52:3300/0 4 ==== mgrmap(e 16) ==== 149652+0+0 (secure 0 0 0) 0x7fc88c02a070 con 0x7fc890076e30 2024-05-27T07:21:53.160 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:21:53.165+0000 7fc8867fc700 1 --2- 172.21.15.52:0/2854547998 >> [v2:172.21.15.52:6800/3513075805,v1:172.21.15.52:6801/3513075805] conn(0x7fc87c0b2900 0x7fc87c0b4e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:21:53.160 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:21:53.165+0000 7fc894cd0700 1 --2- 172.21.15.52:0/2854547998 >> [v2:172.21.15.52:6800/3513075805,v1:172.21.15.52:6801/3513075805] conn(0x7fc87c0b2900 0x7fc87c0b4e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:21:53.160 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:21:53.165+0000 7fc8867fc700 1 -- 172.21.15.52:0/2854547998 <== mon.0 v2:172.21.15.52:3300/0 5 ==== osd_map(79..79 src has 1..79) ==== 7979+0+0 (secure 0 0 0) 0x7fc88c0e32b0 con 0x7fc890076e30 2024-05-27T07:21:53.160 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:21:53.166+0000 7fc894cd0700 1 --2- 172.21.15.52:0/2854547998 >> [v2:172.21.15.52:6800/3513075805,v1:172.21.15.52:6801/3513075805] conn(0x7fc87c0b2900 0x7fc87c0b4e10 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7fc880005fd0 tx=0x7fc880005f60 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:21:53.161 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:21:53.169+0000 7fc8867fc700 1 -- 172.21.15.52:0/2854547998 <== mon.0 v2:172.21.15.52:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191963 (secure 0 0 0) 0x7fc88c013070 con 0x7fc890076e30 2024-05-27T07:21:53.384 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:21:53.393+0000 7fc897735700 1 -- 172.21.15.52:0/2854547998 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_command({"prefix": "osd erasure-code-profile set", "name": "cephfs_data_ec", "profile": ["m=2", "k=2", "crush-failure-domain=osd"]} v 0) -- 0x7fc89019e370 con 0x7fc890076e30 2024-05-27T07:21:53.585 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:21:53.595+0000 7fc8867fc700 1 -- 172.21.15.52:0/2854547998 <== mon.0 v2:172.21.15.52:3300/0 7 ==== mon_command_ack([{"prefix": "osd erasure-code-profile set", "name": "cephfs_data_ec", "profile": ["m=2", "k=2", "crush-failure-domain=osd"]}]=0 v80) ==== 157+0+0 (secure 0 0 0) 0x7fc88c0ab110 con 0x7fc890076e30 2024-05-27T07:21:53.589 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:21:53.599+0000 7fc897735700 1 -- 172.21.15.52:0/2854547998 >> [v2:172.21.15.52:6800/3513075805,v1:172.21.15.52:6801/3513075805] conn(0x7fc87c0b2900 msgr2=0x7fc87c0b4e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:21:53.589 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:21:53.599+0000 7fc897735700 1 --2- 172.21.15.52:0/2854547998 >> [v2:172.21.15.52:6800/3513075805,v1:172.21.15.52:6801/3513075805] conn(0x7fc87c0b2900 0x7fc87c0b4e10 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7fc880005fd0 tx=0x7fc880005f60 comp rx=0 tx=0).stop 2024-05-27T07:21:53.590 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:21:53.599+0000 7fc897735700 1 -- 172.21.15.52:0/2854547998 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fc890076e30 msgr2=0x7fc89019d010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:21:53.590 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:21:53.599+0000 7fc897735700 1 --2- 172.21.15.52:0/2854547998 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fc890076e30 0x7fc89019d010 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7fc88c000c00 tx=0x7fc88c0047b0 comp rx=0 tx=0).stop 2024-05-27T07:21:53.590 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:21:53.599+0000 7fc897735700 1 -- 172.21.15.52:0/2854547998 shutdown_connections 2024-05-27T07:21:53.590 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:21:53.599+0000 7fc897735700 1 --2- 172.21.15.52:0/2854547998 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fc890110b70 0x7fc8901a1840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:21:53.590 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:21:53.599+0000 7fc897735700 1 --2- 172.21.15.52:0/2854547998 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fc890076e30 0x7fc89019d010 unknown :-1 s=CLOSED pgs=188 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:21:53.590 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:21:53.599+0000 7fc897735700 1 --2- 172.21.15.52:0/2854547998 >> [v2:172.21.15.52:6800/3513075805,v1:172.21.15.52:6801/3513075805] conn(0x7fc87c0b2900 0x7fc87c0b4e10 unknown :-1 s=CLOSED pgs=144 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:21:53.590 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:21:53.599+0000 7fc897735700 1 --2- 172.21.15.52:0/2854547998 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fc890077780 0x7fc89019d5b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:21:53.590 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:21:53.599+0000 7fc897735700 1 -- 172.21.15.52:0/2854547998 >> 172.21.15.52:0/2854547998 conn(0x7fc8900ffe90 msgr2=0x7fc890100520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:21:53.591 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:21:53.599+0000 7fc897735700 1 -- 172.21.15.52:0/2854547998 shutdown_connections 2024-05-27T07:21:53.591 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:21:53.600+0000 7fc897735700 1 -- 172.21.15.52:0/2854547998 wait complete. 2024-05-27T07:21:53.852 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:21:53 smithi173 ceph-mon[37000]: Health check cleared: POOL_APP_NOT_ENABLED (was: 2 pool(s) do not have an application enabled) 2024-05-27T07:21:53.852 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:21:53 smithi173 ceph-mon[37000]: Cluster is now healthy 2024-05-27T07:21:53.852 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:21:53 smithi173 ceph-mon[37000]: from='client.? 172.21.15.52:0/2854547998' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "cephfs_data_ec", "profile": ["m=2", "k=2", "crush-failure-domain=osd"]}]: dispatch 2024-05-27T07:21:53.940 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:21:53 smithi105 ceph-mon[35977]: Health check cleared: POOL_APP_NOT_ENABLED (was: 2 pool(s) do not have an application enabled) 2024-05-27T07:21:53.940 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:21:53 smithi105 ceph-mon[35977]: Cluster is now healthy 2024-05-27T07:21:53.941 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:21:53 smithi105 ceph-mon[35977]: from='client.? 172.21.15.52:0/2854547998' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "cephfs_data_ec", "profile": ["m=2", "k=2", "crush-failure-domain=osd"]}]: dispatch 2024-05-27T07:21:53.948 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:21:53 smithi052 ceph-mon[31955]: Health check cleared: POOL_APP_NOT_ENABLED (was: 2 pool(s) do not have an application enabled) 2024-05-27T07:21:53.948 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:21:53 smithi052 ceph-mon[31955]: Cluster is now healthy 2024-05-27T07:21:53.948 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:21:53 smithi052 ceph-mon[31955]: from='client.? 172.21.15.52:0/2854547998' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "cephfs_data_ec", "profile": ["m=2", "k=2", "crush-failure-domain=osd"]}]: dispatch 2024-05-27T07:21:54.403 DEBUG:teuthology.orchestra.run.smithi052:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:38f9d900c83c63799fbdbe61acc9a11b0d3554a6 shell --fsid 73df2636-1bf8-11ef-bc9b-c7b262605968 -- ceph osd pool create cephfs_data_ec erasure cephfs_data_ec --pg_num_min 64 --target_size_ratio 0.9 2024-05-27T07:21:54.661 INFO:teuthology.orchestra.run.smithi052.stderr:Inferring config /var/lib/ceph/73df2636-1bf8-11ef-bc9b-c7b262605968/mon.a/config 2024-05-27T07:21:54.729 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:21:54 smithi052 ceph-mon[31955]: from='client.? 172.21.15.52:0/2854547998' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "cephfs_data_ec", "profile": ["m=2", "k=2", "crush-failure-domain=osd"]}]': finished 2024-05-27T07:21:54.729 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:21:54 smithi052 ceph-mon[31955]: osdmap e80: 12 total, 12 up, 12 in 2024-05-27T07:21:54.730 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:21:54 smithi052 ceph-mon[31955]: pgmap v292: 97 pgs: 97 active+clean; 577 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s wr, 5 op/s 2024-05-27T07:21:54.852 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:21:54 smithi173 ceph-mon[37000]: from='client.? 172.21.15.52:0/2854547998' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "cephfs_data_ec", "profile": ["m=2", "k=2", "crush-failure-domain=osd"]}]': finished 2024-05-27T07:21:54.852 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:21:54 smithi173 ceph-mon[37000]: osdmap e80: 12 total, 12 up, 12 in 2024-05-27T07:21:54.852 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:21:54 smithi173 ceph-mon[37000]: pgmap v292: 97 pgs: 97 active+clean; 577 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s wr, 5 op/s 2024-05-27T07:21:54.940 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:21:54 smithi105 ceph-mon[35977]: from='client.? 172.21.15.52:0/2854547998' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "cephfs_data_ec", "profile": ["m=2", "k=2", "crush-failure-domain=osd"]}]': finished 2024-05-27T07:21:54.940 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:21:54 smithi105 ceph-mon[35977]: osdmap e80: 12 total, 12 up, 12 in 2024-05-27T07:21:54.940 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:21:54 smithi105 ceph-mon[35977]: pgmap v292: 97 pgs: 97 active+clean; 577 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s wr, 5 op/s 2024-05-27T07:21:56.499 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:21:56.508+0000 7f350a7d9700 1 -- 172.21.15.52:0/1997847196 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f3504108ef0 msgr2=0x7f3504109300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:21:56.499 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:21:56.508+0000 7f350a7d9700 1 --2- 172.21.15.52:0/1997847196 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f3504108ef0 0x7f3504109300 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7f34ec0099c0 tx=0x7f34ec009cd0 comp rx=0 tx=0).stop 2024-05-27T07:21:56.500 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:21:56.508+0000 7f350a7d9700 1 -- 172.21.15.52:0/1997847196 shutdown_connections 2024-05-27T07:21:56.500 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:21:56.508+0000 7f350a7d9700 1 --2- 172.21.15.52:0/1997847196 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f3504109840 0x7f35041111a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:21:56.500 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:21:56.508+0000 7f350a7d9700 1 --2- 172.21.15.52:0/1997847196 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f35041045c0 0x7f35041049d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:21:56.500 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:21:56.508+0000 7f350a7d9700 1 --2- 172.21.15.52:0/1997847196 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f3504108ef0 0x7f3504109300 unknown :-1 s=CLOSED pgs=189 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:21:56.500 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:21:56.508+0000 7f350a7d9700 1 -- 172.21.15.52:0/1997847196 >> 172.21.15.52:0/1997847196 conn(0x7f35040ffe90 msgr2=0x7f35041022e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:21:56.500 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:21:56.509+0000 7f350a7d9700 1 -- 172.21.15.52:0/1997847196 shutdown_connections 2024-05-27T07:21:56.500 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:21:56.509+0000 7f350a7d9700 1 -- 172.21.15.52:0/1997847196 wait complete. 2024-05-27T07:21:56.500 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:21:56.509+0000 7f350a7d9700 1 Processor -- start 2024-05-27T07:21:56.500 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:21:56.510+0000 7f350a7d9700 1 -- start start 2024-05-27T07:21:56.502 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:21:56.510+0000 7f350a7d9700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f35041045c0 0x7f350419d0b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:21:56.502 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:21:56.510+0000 7f350a7d9700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f3504108ef0 0x7f350419d650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:21:56.502 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:21:56.510+0000 7f350a7d9700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f3504109840 0x7f35041a18e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:21:56.502 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:21:56.510+0000 7f350a7d9700 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x7f35041144a0 con 0x7f3504108ef0 2024-05-27T07:21:56.502 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:21:56.510+0000 7f350a7d9700 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f3504114320 con 0x7f35041045c0 2024-05-27T07:21:56.502 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:21:56.510+0000 7f350a7d9700 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f3504114620 con 0x7f3504109840 2024-05-27T07:21:56.502 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:21:56.510+0000 7f34fbfff700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f3504108ef0 0x7f350419d650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:21:56.502 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:21:56.510+0000 7f34fbfff700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f3504108ef0 0x7f350419d650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.52:50392/0 (socket says 172.21.15.52:50392) 2024-05-27T07:21:56.503 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:21:56.510+0000 7f34fbfff700 1 -- 172.21.15.52:0/3546674888 learned_addr learned my addr 172.21.15.52:0/3546674888 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-05-27T07:21:56.503 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:21:56.510+0000 7f3503fff700 1 --2- 172.21.15.52:0/3546674888 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f35041045c0 0x7f350419d0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:21:56.503 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:21:56.510+0000 7f3508d76700 1 --2- 172.21.15.52:0/3546674888 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f3504109840 0x7f35041a18e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:21:56.503 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:21:56.510+0000 7f34fbfff700 1 -- 172.21.15.52:0/3546674888 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f3504109840 msgr2=0x7f35041a18e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:21:56.503 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:21:56.510+0000 7f34fbfff700 1 --2- 172.21.15.52:0/3546674888 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f3504109840 0x7f35041a18e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:21:56.503 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:21:56.510+0000 7f34fbfff700 1 -- 172.21.15.52:0/3546674888 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f35041045c0 msgr2=0x7f350419d0b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:21:56.503 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:21:56.510+0000 7f34fbfff700 1 --2- 172.21.15.52:0/3546674888 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f35041045c0 0x7f350419d0b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:21:56.503 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:21:56.510+0000 7f34fbfff700 1 -- 172.21.15.52:0/3546674888 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f35041a1ed0 con 0x7f3504108ef0 2024-05-27T07:21:56.503 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:21:56.510+0000 7f3508d76700 1 --2- 172.21.15.52:0/3546674888 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f3504109840 0x7f35041a18e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-27T07:21:56.503 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:21:56.511+0000 7f34fbfff700 1 --2- 172.21.15.52:0/3546674888 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f3504108ef0 0x7f350419d650 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7f34ec009f50 tx=0x7f34ec0037c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:21:56.503 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:21:56.511+0000 7f3501ffb700 1 -- 172.21.15.52:0/3546674888 <== mon.0 v2:172.21.15.52:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f34ec027070 con 0x7f3504108ef0 2024-05-27T07:21:56.503 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:21:56.511+0000 7f350a7d9700 1 -- 172.21.15.52:0/3546674888 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f35041a2160 con 0x7f3504108ef0 2024-05-27T07:21:56.504 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:21:56.511+0000 7f3501ffb700 1 -- 172.21.15.52:0/3546674888 <== mon.0 v2:172.21.15.52:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f34ec004090 con 0x7f3504108ef0 2024-05-27T07:21:56.504 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:21:56.511+0000 7f3501ffb700 1 -- 172.21.15.52:0/3546674888 <== mon.0 v2:172.21.15.52:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f34ec022e40 con 0x7f3504108ef0 2024-05-27T07:21:56.504 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:21:56.511+0000 7f350a7d9700 1 -- 172.21.15.52:0/3546674888 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f35041a2690 con 0x7f3504108ef0 2024-05-27T07:21:56.504 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:21:56.512+0000 7f350a7d9700 1 -- 172.21.15.52:0/3546674888 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f35040512b0 con 0x7f3504108ef0 2024-05-27T07:21:56.504 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:21:56.514+0000 7f3501ffb700 1 -- 172.21.15.52:0/3546674888 <== mon.0 v2:172.21.15.52:3300/0 4 ==== mgrmap(e 16) ==== 149652+0+0 (secure 0 0 0) 0x7f34ec02a070 con 0x7f3504108ef0 2024-05-27T07:21:56.504 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:21:56.515+0000 7f3501ffb700 1 --2- 172.21.15.52:0/3546674888 >> [v2:172.21.15.52:6800/3513075805,v1:172.21.15.52:6801/3513075805] conn(0x7f34f00b2900 0x7f34f00b4e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:21:56.505 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:21:56.515+0000 7f3503fff700 1 --2- 172.21.15.52:0/3546674888 >> [v2:172.21.15.52:6800/3513075805,v1:172.21.15.52:6801/3513075805] conn(0x7f34f00b2900 0x7f34f00b4e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:21:56.505 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:21:56.515+0000 7f3501ffb700 1 -- 172.21.15.52:0/3546674888 <== mon.0 v2:172.21.15.52:3300/0 5 ==== osd_map(80..80 src has 1..80) ==== 8277+0+0 (secure 0 0 0) 0x7f34ec0e3360 con 0x7f3504108ef0 2024-05-27T07:21:56.505 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:21:56.515+0000 7f3503fff700 1 --2- 172.21.15.52:0/3546674888 >> [v2:172.21.15.52:6800/3513075805,v1:172.21.15.52:6801/3513075805] conn(0x7f34f00b2900 0x7f34f00b4e10 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7f34f400ac50 tx=0x7f34f400a380 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:21:56.508 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:21:56.518+0000 7f3501ffb700 1 -- 172.21.15.52:0/3546674888 <== mon.0 v2:172.21.15.52:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191963 (secure 0 0 0) 0x7f34ec013070 con 0x7f3504108ef0 2024-05-27T07:21:56.725 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:21:56.735+0000 7f350a7d9700 1 -- 172.21.15.52:0/3546674888 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_data_ec", "pool_type": "erasure", "erasure_code_profile": "cephfs_data_ec", "pg_num_min": 64, "target_size_ratio": 0.9} v 0) -- 0x7f350419e410 con 0x7f3504108ef0 2024-05-27T07:21:56.940 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:21:56 smithi105 ceph-mon[35977]: pgmap v293: 97 pgs: 97 active+clean; 579 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s wr, 7 op/s 2024-05-27T07:21:56.979 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:21:56 smithi052 ceph-mon[31955]: pgmap v293: 97 pgs: 97 active+clean; 579 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s wr, 7 op/s 2024-05-27T07:21:57.102 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:21:56 smithi173 ceph-mon[37000]: pgmap v293: 97 pgs: 97 active+clean; 579 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s wr, 7 op/s 2024-05-27T07:21:57.940 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:21:57 smithi105 ceph-mon[35977]: from='client.? 172.21.15.52:0/3546674888' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_data_ec", "pool_type": "erasure", "erasure_code_profile": "cephfs_data_ec", "pg_num_min": 64, "target_size_ratio": 0.9}]: dispatch 2024-05-27T07:21:57.980 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:21:57 smithi052 ceph-mon[31955]: from='client.? 172.21.15.52:0/3546674888' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_data_ec", "pool_type": "erasure", "erasure_code_profile": "cephfs_data_ec", "pg_num_min": 64, "target_size_ratio": 0.9}]: dispatch 2024-05-27T07:21:58.102 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:21:57 smithi173 ceph-mon[37000]: from='client.? 172.21.15.52:0/3546674888' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_data_ec", "pool_type": "erasure", "erasure_code_profile": "cephfs_data_ec", "pg_num_min": 64, "target_size_ratio": 0.9}]: dispatch 2024-05-27T07:21:58.695 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:21:58.705+0000 7f3501ffb700 1 -- 172.21.15.52:0/3546674888 <== mon.0 v2:172.21.15.52:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_data_ec", "pool_type": "erasure", "erasure_code_profile": "cephfs_data_ec", "pg_num_min": 64, "target_size_ratio": 0.9}]=0 pool 'cephfs_data_ec' created v82) ==== 228+0+0 (secure 0 0 0) 0x7f34ec0ab080 con 0x7f3504108ef0 2024-05-27T07:21:58.699 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:21:58.709+0000 7f350a7d9700 1 -- 172.21.15.52:0/3546674888 >> [v2:172.21.15.52:6800/3513075805,v1:172.21.15.52:6801/3513075805] conn(0x7f34f00b2900 msgr2=0x7f34f00b4e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:21:58.700 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:21:58.709+0000 7f350a7d9700 1 --2- 172.21.15.52:0/3546674888 >> [v2:172.21.15.52:6800/3513075805,v1:172.21.15.52:6801/3513075805] conn(0x7f34f00b2900 0x7f34f00b4e10 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7f34f400ac50 tx=0x7f34f400a380 comp rx=0 tx=0).stop 2024-05-27T07:21:58.700 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:21:58.709+0000 7f350a7d9700 1 -- 172.21.15.52:0/3546674888 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f3504108ef0 msgr2=0x7f350419d650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:21:58.700 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:21:58.709+0000 7f350a7d9700 1 --2- 172.21.15.52:0/3546674888 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f3504108ef0 0x7f350419d650 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7f34ec009f50 tx=0x7f34ec0037c0 comp rx=0 tx=0).stop 2024-05-27T07:21:58.700 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:21:58.710+0000 7f350a7d9700 1 -- 172.21.15.52:0/3546674888 shutdown_connections 2024-05-27T07:21:58.700 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:21:58.710+0000 7f350a7d9700 1 --2- 172.21.15.52:0/3546674888 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f3504109840 0x7f35041a18e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:21:58.700 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:21:58.710+0000 7f350a7d9700 1 --2- 172.21.15.52:0/3546674888 >> [v2:172.21.15.52:6800/3513075805,v1:172.21.15.52:6801/3513075805] conn(0x7f34f00b2900 0x7f34f00b4e10 unknown :-1 s=CLOSED pgs=145 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:21:58.700 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:21:58.710+0000 7f350a7d9700 1 --2- 172.21.15.52:0/3546674888 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f35041045c0 0x7f350419d0b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:21:58.700 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:21:58.710+0000 7f350a7d9700 1 --2- 172.21.15.52:0/3546674888 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f3504108ef0 0x7f350419d650 unknown :-1 s=CLOSED pgs=190 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:21:58.700 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:21:58.710+0000 7f350a7d9700 1 -- 172.21.15.52:0/3546674888 >> 172.21.15.52:0/3546674888 conn(0x7f35040ffe90 msgr2=0x7f350410a7a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:21:58.701 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:21:58.710+0000 7f350a7d9700 1 -- 172.21.15.52:0/3546674888 shutdown_connections 2024-05-27T07:21:58.701 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:21:58.710+0000 7f350a7d9700 1 -- 172.21.15.52:0/3546674888 wait complete. 2024-05-27T07:21:58.707 INFO:teuthology.orchestra.run.smithi052.stderr:pool 'cephfs_data_ec' created 2024-05-27T07:21:58.940 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:21:58 smithi105 ceph-mon[35977]: pgmap v294: 97 pgs: 97 active+clean; 579 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s wr, 5 op/s 2024-05-27T07:21:58.940 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:21:58 smithi105 ceph-mon[35977]: osdmap e81: 12 total, 12 up, 12 in 2024-05-27T07:21:58.975 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:21:58 smithi052 ceph-mon[31955]: pgmap v294: 97 pgs: 97 active+clean; 579 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s wr, 5 op/s 2024-05-27T07:21:58.975 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:21:58 smithi052 ceph-mon[31955]: osdmap e81: 12 total, 12 up, 12 in 2024-05-27T07:21:59.102 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:21:58 smithi173 ceph-mon[37000]: pgmap v294: 97 pgs: 97 active+clean; 579 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s wr, 5 op/s 2024-05-27T07:21:59.102 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:21:58 smithi173 ceph-mon[37000]: osdmap e81: 12 total, 12 up, 12 in 2024-05-27T07:21:59.421 DEBUG:teuthology.orchestra.run.smithi052:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:38f9d900c83c63799fbdbe61acc9a11b0d3554a6 shell --fsid 73df2636-1bf8-11ef-bc9b-c7b262605968 -- ceph osd pool set cephfs_data_ec allow_ec_overwrites true 2024-05-27T07:21:59.679 INFO:teuthology.orchestra.run.smithi052.stderr:Inferring config /var/lib/ceph/73df2636-1bf8-11ef-bc9b-c7b262605968/mon.a/config 2024-05-27T07:21:59.962 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:21:59 smithi052 ceph-mon[31955]: from='client.? 172.21.15.52:0/3546674888' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_data_ec", "pool_type": "erasure", "erasure_code_profile": "cephfs_data_ec", "pg_num_min": 64, "target_size_ratio": 0.9}]': finished 2024-05-27T07:21:59.962 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:21:59 smithi052 ceph-mon[31955]: osdmap e82: 12 total, 12 up, 12 in 2024-05-27T07:22:00.102 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:21:59 smithi173 ceph-mon[37000]: from='client.? 172.21.15.52:0/3546674888' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_data_ec", "pool_type": "erasure", "erasure_code_profile": "cephfs_data_ec", "pg_num_min": 64, "target_size_ratio": 0.9}]': finished 2024-05-27T07:22:00.102 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:21:59 smithi173 ceph-mon[37000]: osdmap e82: 12 total, 12 up, 12 in 2024-05-27T07:22:00.190 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:21:59 smithi105 ceph-mon[35977]: from='client.? 172.21.15.52:0/3546674888' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_data_ec", "pool_type": "erasure", "erasure_code_profile": "cephfs_data_ec", "pg_num_min": 64, "target_size_ratio": 0.9}]': finished 2024-05-27T07:22:00.190 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:21:59 smithi105 ceph-mon[35977]: osdmap e82: 12 total, 12 up, 12 in 2024-05-27T07:22:00.978 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:22:00 smithi052 ceph-mon[31955]: pgmap v297: 129 pgs: 32 unknown, 97 active+clean; 579 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail; 1016 B/s wr, 3 op/s 2024-05-27T07:22:00.978 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:22:00 smithi052 ceph-mon[31955]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-05-27T07:22:00.979 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:22:00 smithi052 ceph-mon[31955]: osdmap e83: 12 total, 12 up, 12 in 2024-05-27T07:22:01.102 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:22:00 smithi173 ceph-mon[37000]: pgmap v297: 129 pgs: 32 unknown, 97 active+clean; 579 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail; 1016 B/s wr, 3 op/s 2024-05-27T07:22:01.102 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:22:00 smithi173 ceph-mon[37000]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-05-27T07:22:01.102 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:22:00 smithi173 ceph-mon[37000]: osdmap e83: 12 total, 12 up, 12 in 2024-05-27T07:22:01.190 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:22:00 smithi105 ceph-mon[35977]: pgmap v297: 129 pgs: 32 unknown, 97 active+clean; 579 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail; 1016 B/s wr, 3 op/s 2024-05-27T07:22:01.190 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:22:00 smithi105 ceph-mon[35977]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-05-27T07:22:01.190 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:22:00 smithi105 ceph-mon[35977]: osdmap e83: 12 total, 12 up, 12 in 2024-05-27T07:22:01.210 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:22:01.219+0000 7f25ca807700 1 -- 172.21.15.52:0/294514090 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f25c4108730 msgr2=0x7f25c4110090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:22:01.210 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:22:01.219+0000 7f25ca807700 1 --2- 172.21.15.52:0/294514090 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f25c4108730 0x7f25c4110090 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7f25bc00b0d0 tx=0x7f25bc00b3e0 comp rx=0 tx=0).stop 2024-05-27T07:22:01.210 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:22:01.219+0000 7f25ca807700 1 -- 172.21.15.52:0/294514090 shutdown_connections 2024-05-27T07:22:01.210 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:22:01.219+0000 7f25ca807700 1 --2- 172.21.15.52:0/294514090 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f25c4108730 0x7f25c4110090 unknown :-1 s=CLOSED pgs=191 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:22:01.210 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:22:01.219+0000 7f25ca807700 1 --2- 172.21.15.52:0/294514090 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f25c41034b0 0x7f25c41038c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:22:01.210 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:22:01.220+0000 7f25ca807700 1 --2- 172.21.15.52:0/294514090 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f25c4107de0 0x7f25c41081f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:22:01.210 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:22:01.220+0000 7f25ca807700 1 -- 172.21.15.52:0/294514090 >> 172.21.15.52:0/294514090 conn(0x7f25c40fede0 msgr2=0x7f25c4101200 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:22:01.210 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:22:01.220+0000 7f25ca807700 1 -- 172.21.15.52:0/294514090 shutdown_connections 2024-05-27T07:22:01.211 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:22:01.220+0000 7f25ca807700 1 -- 172.21.15.52:0/294514090 wait complete. 2024-05-27T07:22:01.212 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:22:01.220+0000 7f25ca807700 1 Processor -- start 2024-05-27T07:22:01.212 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:22:01.220+0000 7f25ca807700 1 -- start start 2024-05-27T07:22:01.212 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:22:01.221+0000 7f25ca807700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f25c41034b0 0x7f25c419adc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:22:01.213 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:22:01.221+0000 7f25ca807700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f25c4107de0 0x7f25c419b360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:22:01.213 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:22:01.221+0000 7f25ca807700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f25c4108730 0x7f25c419f5f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:22:01.213 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:22:01.221+0000 7f25ca807700 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x7f25c4113360 con 0x7f25c41034b0 2024-05-27T07:22:01.213 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:22:01.221+0000 7f25ca807700 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f25c41131e0 con 0x7f25c4108730 2024-05-27T07:22:01.213 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:22:01.221+0000 7f25ca807700 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f25c41134e0 con 0x7f25c4107de0 2024-05-27T07:22:01.213 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:22:01.221+0000 7f25c3fff700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f25c41034b0 0x7f25c419adc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:22:01.213 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:22:01.221+0000 7f25c3fff700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f25c41034b0 0x7f25c419adc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.52:56948/0 (socket says 172.21.15.52:56948) 2024-05-27T07:22:01.213 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:22:01.221+0000 7f25c3fff700 1 -- 172.21.15.52:0/2383323944 learned_addr learned my addr 172.21.15.52:0/2383323944 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-05-27T07:22:01.213 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:22:01.221+0000 7f25c8da4700 1 --2- 172.21.15.52:0/2383323944 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f25c4108730 0x7f25c419f5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:22:01.213 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:22:01.221+0000 7f25bbfff700 1 --2- 172.21.15.52:0/2383323944 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f25c4107de0 0x7f25c419b360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:22:01.213 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:22:01.221+0000 7f25c3fff700 1 -- 172.21.15.52:0/2383323944 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f25c4107de0 msgr2=0x7f25c419b360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:22:01.213 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:22:01.221+0000 7f25c3fff700 1 --2- 172.21.15.52:0/2383323944 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f25c4107de0 0x7f25c419b360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:22:01.213 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:22:01.221+0000 7f25c3fff700 1 -- 172.21.15.52:0/2383323944 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f25c4108730 msgr2=0x7f25c419f5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:22:01.214 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:22:01.221+0000 7f25c3fff700 1 --2- 172.21.15.52:0/2383323944 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f25c4108730 0x7f25c419f5f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:22:01.214 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:22:01.221+0000 7f25c3fff700 1 -- 172.21.15.52:0/2383323944 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f25c419fc90 con 0x7f25c41034b0 2024-05-27T07:22:01.214 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:22:01.221+0000 7f25c3fff700 1 --2- 172.21.15.52:0/2383323944 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f25c41034b0 0x7f25c419adc0 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7f25b400e9a0 tx=0x7f25b400ecb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:22:01.214 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:22:01.222+0000 7f25c1ffb700 1 -- 172.21.15.52:0/2383323944 <== mon.0 v2:172.21.15.52:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f25b400cbd0 con 0x7f25c41034b0 2024-05-27T07:22:01.214 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:22:01.222+0000 7f25ca807700 1 -- 172.21.15.52:0/2383323944 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f25c419ff80 con 0x7f25c41034b0 2024-05-27T07:22:01.214 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:22:01.222+0000 7f25c1ffb700 1 -- 172.21.15.52:0/2383323944 <== mon.0 v2:172.21.15.52:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f25b400cd70 con 0x7f25c41034b0 2024-05-27T07:22:01.214 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:22:01.222+0000 7f25c1ffb700 1 -- 172.21.15.52:0/2383323944 <== mon.0 v2:172.21.15.52:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f25b4018680 con 0x7f25c41034b0 2024-05-27T07:22:01.214 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:22:01.222+0000 7f25ca807700 1 -- 172.21.15.52:0/2383323944 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f25c41a78a0 con 0x7f25c41034b0 2024-05-27T07:22:01.214 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:22:01.223+0000 7f25ca807700 1 -- 172.21.15.52:0/2383323944 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f25c40512b0 con 0x7f25c41034b0 2024-05-27T07:22:01.215 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:22:01.224+0000 7f25c1ffb700 1 -- 172.21.15.52:0/2383323944 <== mon.0 v2:172.21.15.52:3300/0 4 ==== mgrmap(e 16) ==== 149652+0+0 (secure 0 0 0) 0x7f25b4010430 con 0x7f25c41034b0 2024-05-27T07:22:01.215 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:22:01.225+0000 7f25c1ffb700 1 --2- 172.21.15.52:0/2383323944 >> [v2:172.21.15.52:6800/3513075805,v1:172.21.15.52:6801/3513075805] conn(0x7f25b00b2880 0x7f25b00b4d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:22:01.215 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:22:01.225+0000 7f25bbfff700 1 --2- 172.21.15.52:0/2383323944 >> [v2:172.21.15.52:6800/3513075805,v1:172.21.15.52:6801/3513075805] conn(0x7f25b00b2880 0x7f25b00b4d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:22:01.215 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:22:01.225+0000 7f25c1ffb700 1 -- 172.21.15.52:0/2383323944 <== mon.0 v2:172.21.15.52:3300/0 5 ==== osd_map(84..84 src has 1..84) ==== 8770+0+0 (secure 0 0 0) 0x7f25b40e2900 con 0x7f25c41034b0 2024-05-27T07:22:01.215 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:22:01.226+0000 7f25bbfff700 1 --2- 172.21.15.52:0/2383323944 >> [v2:172.21.15.52:6800/3513075805,v1:172.21.15.52:6801/3513075805] conn(0x7f25b00b2880 0x7f25b00b4d90 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7f25ac005950 tx=0x7f25ac0058e0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:22:01.219 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:22:01.229+0000 7f25c1ffb700 1 -- 172.21.15.52:0/2383323944 <== mon.0 v2:172.21.15.52:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191963 (secure 0 0 0) 0x7f25b40aa400 con 0x7f25c41034b0 2024-05-27T07:22:01.438 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:22:01.448+0000 7f25ca807700 1 -- 172.21.15.52:0/2383323944 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_command({"prefix": "osd pool set", "pool": "cephfs_data_ec", "var": "allow_ec_overwrites", "val": "true"} v 0) -- 0x7f25c419c120 con 0x7f25c41034b0 2024-05-27T07:22:01.715 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:22:01.724+0000 7f25c1ffb700 1 -- 172.21.15.52:0/2383323944 <== mon.0 v2:172.21.15.52:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool set", "pool": "cephfs_data_ec", "var": "allow_ec_overwrites", "val": "true"}]=0 set pool 4 allow_ec_overwrites to true v85) ==== 169+0+0 (secure 0 0 0) 0x7f25b40a9d00 con 0x7f25c41034b0 2024-05-27T07:22:01.718 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:22:01.728+0000 7f25ca807700 1 -- 172.21.15.52:0/2383323944 >> [v2:172.21.15.52:6800/3513075805,v1:172.21.15.52:6801/3513075805] conn(0x7f25b00b2880 msgr2=0x7f25b00b4d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:22:01.718 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:22:01.728+0000 7f25ca807700 1 --2- 172.21.15.52:0/2383323944 >> [v2:172.21.15.52:6800/3513075805,v1:172.21.15.52:6801/3513075805] conn(0x7f25b00b2880 0x7f25b00b4d90 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7f25ac005950 tx=0x7f25ac0058e0 comp rx=0 tx=0).stop 2024-05-27T07:22:01.718 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:22:01.728+0000 7f25ca807700 1 -- 172.21.15.52:0/2383323944 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f25c41034b0 msgr2=0x7f25c419adc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:22:01.719 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:22:01.728+0000 7f25ca807700 1 --2- 172.21.15.52:0/2383323944 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f25c41034b0 0x7f25c419adc0 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7f25b400e9a0 tx=0x7f25b400ecb0 comp rx=0 tx=0).stop 2024-05-27T07:22:01.719 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:22:01.728+0000 7f25ca807700 1 -- 172.21.15.52:0/2383323944 shutdown_connections 2024-05-27T07:22:01.719 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:22:01.728+0000 7f25ca807700 1 --2- 172.21.15.52:0/2383323944 >> [v2:172.21.15.52:6800/3513075805,v1:172.21.15.52:6801/3513075805] conn(0x7f25b00b2880 0x7f25b00b4d90 unknown :-1 s=CLOSED pgs=146 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:22:01.719 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:22:01.728+0000 7f25ca807700 1 --2- 172.21.15.52:0/2383323944 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f25c4108730 0x7f25c419f5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:22:01.719 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:22:01.728+0000 7f25ca807700 1 --2- 172.21.15.52:0/2383323944 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f25c41034b0 0x7f25c419adc0 unknown :-1 s=CLOSED pgs=192 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:22:01.719 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:22:01.728+0000 7f25ca807700 1 --2- 172.21.15.52:0/2383323944 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f25c4107de0 0x7f25c419b360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:22:01.719 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:22:01.728+0000 7f25ca807700 1 -- 172.21.15.52:0/2383323944 >> 172.21.15.52:0/2383323944 conn(0x7f25c40fede0 msgr2=0x7f25c4075d40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:22:01.719 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:22:01.728+0000 7f25ca807700 1 -- 172.21.15.52:0/2383323944 shutdown_connections 2024-05-27T07:22:01.719 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:22:01.728+0000 7f25ca807700 1 -- 172.21.15.52:0/2383323944 wait complete. 2024-05-27T07:22:01.720 INFO:teuthology.orchestra.run.smithi052.stderr:set pool 4 allow_ec_overwrites to true 2024-05-27T07:22:01.979 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:22:01 smithi052 ceph-mon[31955]: osdmap e84: 12 total, 12 up, 12 in 2024-05-27T07:22:01.980 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:22:01 smithi052 ceph-mon[31955]: from='client.? 172.21.15.52:0/2383323944' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "cephfs_data_ec", "var": "allow_ec_overwrites", "val": "true"}]: dispatch 2024-05-27T07:22:02.102 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:22:01 smithi173 ceph-mon[37000]: osdmap e84: 12 total, 12 up, 12 in 2024-05-27T07:22:02.102 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:22:01 smithi173 ceph-mon[37000]: from='client.? 172.21.15.52:0/2383323944' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "cephfs_data_ec", "var": "allow_ec_overwrites", "val": "true"}]: dispatch 2024-05-27T07:22:02.190 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:22:01 smithi105 ceph-mon[35977]: osdmap e84: 12 total, 12 up, 12 in 2024-05-27T07:22:02.190 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:22:01 smithi105 ceph-mon[35977]: from='client.? 172.21.15.52:0/2383323944' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "cephfs_data_ec", "var": "allow_ec_overwrites", "val": "true"}]: dispatch 2024-05-27T07:22:02.380 DEBUG:teuthology.orchestra.run.smithi052:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:38f9d900c83c63799fbdbe61acc9a11b0d3554a6 shell --fsid 73df2636-1bf8-11ef-bc9b-c7b262605968 -- ceph fs add_data_pool cephfs cephfs_data_ec 2024-05-27T07:22:02.632 INFO:teuthology.orchestra.run.smithi052.stderr:Inferring config /var/lib/ceph/73df2636-1bf8-11ef-bc9b-c7b262605968/mon.a/config 2024-05-27T07:22:02.979 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:22:02 smithi052 ceph-mon[31955]: pgmap v300: 129 pgs: 9 unknown, 120 active+clean; 579 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-27T07:22:02.979 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:22:02 smithi052 ceph-mon[31955]: from='client.? 172.21.15.52:0/2383323944' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "cephfs_data_ec", "var": "allow_ec_overwrites", "val": "true"}]': finished 2024-05-27T07:22:02.979 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:22:02 smithi052 ceph-mon[31955]: osdmap e85: 12 total, 12 up, 12 in 2024-05-27T07:22:03.102 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:22:02 smithi173 ceph-mon[37000]: pgmap v300: 129 pgs: 9 unknown, 120 active+clean; 579 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-27T07:22:03.102 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:22:02 smithi173 ceph-mon[37000]: from='client.? 172.21.15.52:0/2383323944' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "cephfs_data_ec", "var": "allow_ec_overwrites", "val": "true"}]': finished 2024-05-27T07:22:03.102 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:22:02 smithi173 ceph-mon[37000]: osdmap e85: 12 total, 12 up, 12 in 2024-05-27T07:22:03.190 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:22:02 smithi105 ceph-mon[35977]: pgmap v300: 129 pgs: 9 unknown, 120 active+clean; 579 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-27T07:22:03.190 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:22:02 smithi105 ceph-mon[35977]: from='client.? 172.21.15.52:0/2383323944' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "cephfs_data_ec", "var": "allow_ec_overwrites", "val": "true"}]': finished 2024-05-27T07:22:03.190 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:22:02 smithi105 ceph-mon[35977]: osdmap e85: 12 total, 12 up, 12 in 2024-05-27T07:22:04.324 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:22:04.334+0000 7fa9bcfd4700 1 -- 172.21.15.52:0/3951536636 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fa9b8108730 msgr2=0x7fa9b8110090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:22:04.324 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:22:04.334+0000 7fa9bcfd4700 1 --2- 172.21.15.52:0/3951536636 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fa9b8108730 0x7fa9b8110090 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7fa9b00099c0 tx=0x7fa9b0009cd0 comp rx=0 tx=0).stop 2024-05-27T07:22:04.324 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:22:04.334+0000 7fa9bcfd4700 1 -- 172.21.15.52:0/3951536636 shutdown_connections 2024-05-27T07:22:04.324 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:22:04.334+0000 7fa9bcfd4700 1 --2- 172.21.15.52:0/3951536636 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fa9b8108730 0x7fa9b8110090 unknown :-1 s=CLOSED pgs=193 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:22:04.324 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:22:04.334+0000 7fa9bcfd4700 1 --2- 172.21.15.52:0/3951536636 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fa9b81034b0 0x7fa9b81038c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:22:04.325 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:22:04.334+0000 7fa9bcfd4700 1 --2- 172.21.15.52:0/3951536636 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fa9b8107de0 0x7fa9b81081f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:22:04.325 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:22:04.334+0000 7fa9bcfd4700 1 -- 172.21.15.52:0/3951536636 >> 172.21.15.52:0/3951536636 conn(0x7fa9b80fedb0 msgr2=0x7fa9b81011d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:22:04.325 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:22:04.334+0000 7fa9bcfd4700 1 -- 172.21.15.52:0/3951536636 shutdown_connections 2024-05-27T07:22:04.325 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:22:04.334+0000 7fa9bcfd4700 1 -- 172.21.15.52:0/3951536636 wait complete. 2024-05-27T07:22:04.327 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:22:04.335+0000 7fa9bcfd4700 1 Processor -- start 2024-05-27T07:22:04.327 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:22:04.335+0000 7fa9bcfd4700 1 -- start start 2024-05-27T07:22:04.327 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:22:04.335+0000 7fa9bcfd4700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fa9b81034b0 0x7fa9b819ad60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:22:04.327 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:22:04.335+0000 7fa9bcfd4700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fa9b8107de0 0x7fa9b819b300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:22:04.327 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:22:04.335+0000 7fa9bcfd4700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fa9b8108730 0x7fa9b819f590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:22:04.327 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:22:04.335+0000 7fa9bcfd4700 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x7fa9b81144b0 con 0x7fa9b81034b0 2024-05-27T07:22:04.327 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:22:04.335+0000 7fa9bcfd4700 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7fa9b8114330 con 0x7fa9b8108730 2024-05-27T07:22:04.327 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:22:04.335+0000 7fa9bcfd4700 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7fa9b8114630 con 0x7fa9b8107de0 2024-05-27T07:22:04.327 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:22:04.335+0000 7fa9b659c700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fa9b81034b0 0x7fa9b819ad60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:22:04.327 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:22:04.335+0000 7fa9b659c700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fa9b81034b0 0x7fa9b819ad60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.52:56954/0 (socket says 172.21.15.52:56954) 2024-05-27T07:22:04.327 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:22:04.335+0000 7fa9b659c700 1 -- 172.21.15.52:0/1596529046 learned_addr learned my addr 172.21.15.52:0/1596529046 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-05-27T07:22:04.327 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:22:04.335+0000 7fa9b5d9b700 1 --2- 172.21.15.52:0/1596529046 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fa9b8107de0 0x7fa9b819b300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:22:04.328 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:22:04.335+0000 7fa9b6d9d700 1 --2- 172.21.15.52:0/1596529046 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fa9b8108730 0x7fa9b819f590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:22:04.328 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:22:04.336+0000 7fa9b659c700 1 -- 172.21.15.52:0/1596529046 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fa9b8107de0 msgr2=0x7fa9b819b300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:22:04.328 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:22:04.336+0000 7fa9b659c700 1 --2- 172.21.15.52:0/1596529046 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fa9b8107de0 0x7fa9b819b300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:22:04.328 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:22:04.336+0000 7fa9b659c700 1 -- 172.21.15.52:0/1596529046 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fa9b8108730 msgr2=0x7fa9b819f590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:22:04.328 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:22:04.336+0000 7fa9b659c700 1 --2- 172.21.15.52:0/1596529046 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fa9b8108730 0x7fa9b819f590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:22:04.328 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:22:04.336+0000 7fa9b659c700 1 -- 172.21.15.52:0/1596529046 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa9b819fc30 con 0x7fa9b81034b0 2024-05-27T07:22:04.328 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:22:04.336+0000 7fa9b659c700 1 --2- 172.21.15.52:0/1596529046 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fa9b81034b0 0x7fa9b819ad60 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7fa9a800c920 tx=0x7fa9a800cc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:22:04.328 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:22:04.336+0000 7fa9a77fe700 1 -- 172.21.15.52:0/1596529046 <== mon.0 v2:172.21.15.52:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa9a8004a30 con 0x7fa9b81034b0 2024-05-27T07:22:04.328 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:22:04.336+0000 7fa9bcfd4700 1 -- 172.21.15.52:0/1596529046 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa9b819ff20 con 0x7fa9b81034b0 2024-05-27T07:22:04.328 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:22:04.336+0000 7fa9a77fe700 1 -- 172.21.15.52:0/1596529046 <== mon.0 v2:172.21.15.52:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fa9a80054d0 con 0x7fa9b81034b0 2024-05-27T07:22:04.328 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:22:04.336+0000 7fa9a77fe700 1 -- 172.21.15.52:0/1596529046 <== mon.0 v2:172.21.15.52:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa9a8007bd0 con 0x7fa9b81034b0 2024-05-27T07:22:04.328 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:22:04.336+0000 7fa9bcfd4700 1 -- 172.21.15.52:0/1596529046 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa9b81a7840 con 0x7fa9b81034b0 2024-05-27T07:22:04.330 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:22:04.337+0000 7fa9bcfd4700 1 -- 172.21.15.52:0/1596529046 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa9b80512b0 con 0x7fa9b81034b0 2024-05-27T07:22:04.330 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:22:04.339+0000 7fa9a77fe700 1 -- 172.21.15.52:0/1596529046 <== mon.0 v2:172.21.15.52:3300/0 4 ==== mgrmap(e 16) ==== 149652+0+0 (secure 0 0 0) 0x7fa9a8004bd0 con 0x7fa9b81034b0 2024-05-27T07:22:04.330 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:22:04.340+0000 7fa9a77fe700 1 --2- 172.21.15.52:0/1596529046 >> [v2:172.21.15.52:6800/3513075805,v1:172.21.15.52:6801/3513075805] conn(0x7fa9a00b2880 0x7fa9a00b4d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:22:04.330 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:22:04.340+0000 7fa9b5d9b700 1 --2- 172.21.15.52:0/1596529046 >> [v2:172.21.15.52:6800/3513075805,v1:172.21.15.52:6801/3513075805] conn(0x7fa9a00b2880 0x7fa9a00b4d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:22:04.330 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:22:04.340+0000 7fa9a77fe700 1 -- 172.21.15.52:0/1596529046 <== mon.0 v2:172.21.15.52:3300/0 5 ==== osd_map(85..85 src has 1..85) ==== 8770+0+0 (secure 0 0 0) 0x7fa9a80e1f40 con 0x7fa9b81034b0 2024-05-27T07:22:04.330 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:22:04.340+0000 7fa9b5d9b700 1 --2- 172.21.15.52:0/1596529046 >> [v2:172.21.15.52:6800/3513075805,v1:172.21.15.52:6801/3513075805] conn(0x7fa9a00b2880 0x7fa9a00b4d90 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7fa9ac009550 tx=0x7fa9ac0091b0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:22:04.336 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:22:04.344+0000 7fa9a77fe700 1 -- 172.21.15.52:0/1596529046 <== mon.0 v2:172.21.15.52:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191963 (secure 0 0 0) 0x7fa9a8014030 con 0x7fa9b81034b0 2024-05-27T07:22:04.601 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:22:04.611+0000 7fa9bcfd4700 1 -- 172.21.15.52:0/1596529046 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_command({"prefix": "fs add_data_pool", "fs_name": "cephfs", "pool": "cephfs_data_ec"} v 0) -- 0x7fa9b819c0c0 con 0x7fa9b81034b0 2024-05-27T07:22:04.719 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:22:04.729+0000 7fa9a77fe700 1 -- 172.21.15.52:0/1596529046 <== mon.0 v2:172.21.15.52:3300/0 7 ==== mon_command_ack([{"prefix": "fs add_data_pool", "fs_name": "cephfs", "pool": "cephfs_data_ec"}]=0 added data pool 4 to fsmap v17) ==== 137+0+0 (secure 0 0 0) 0x7fa9a80a9a40 con 0x7fa9b81034b0 2024-05-27T07:22:04.722 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:22:04.732+0000 7fa9bcfd4700 1 -- 172.21.15.52:0/1596529046 >> [v2:172.21.15.52:6800/3513075805,v1:172.21.15.52:6801/3513075805] conn(0x7fa9a00b2880 msgr2=0x7fa9a00b4d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:22:04.722 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:22:04.732+0000 7fa9bcfd4700 1 --2- 172.21.15.52:0/1596529046 >> [v2:172.21.15.52:6800/3513075805,v1:172.21.15.52:6801/3513075805] conn(0x7fa9a00b2880 0x7fa9a00b4d90 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7fa9ac009550 tx=0x7fa9ac0091b0 comp rx=0 tx=0).stop 2024-05-27T07:22:04.722 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:22:04.732+0000 7fa9bcfd4700 1 -- 172.21.15.52:0/1596529046 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fa9b81034b0 msgr2=0x7fa9b819ad60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:22:04.723 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:22:04.732+0000 7fa9bcfd4700 1 --2- 172.21.15.52:0/1596529046 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fa9b81034b0 0x7fa9b819ad60 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7fa9a800c920 tx=0x7fa9a800cc30 comp rx=0 tx=0).stop 2024-05-27T07:22:04.723 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:22:04.732+0000 7fa9bcfd4700 1 -- 172.21.15.52:0/1596529046 shutdown_connections 2024-05-27T07:22:04.723 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:22:04.732+0000 7fa9bcfd4700 1 --2- 172.21.15.52:0/1596529046 >> [v2:172.21.15.52:6800/3513075805,v1:172.21.15.52:6801/3513075805] conn(0x7fa9a00b2880 0x7fa9a00b4d90 unknown :-1 s=CLOSED pgs=147 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:22:04.723 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:22:04.732+0000 7fa9bcfd4700 1 --2- 172.21.15.52:0/1596529046 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fa9b8108730 0x7fa9b819f590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:22:04.723 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:22:04.732+0000 7fa9bcfd4700 1 --2- 172.21.15.52:0/1596529046 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fa9b81034b0 0x7fa9b819ad60 unknown :-1 s=CLOSED pgs=194 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:22:04.723 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:22:04.732+0000 7fa9bcfd4700 1 --2- 172.21.15.52:0/1596529046 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fa9b8107de0 0x7fa9b819b300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:22:04.723 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:22:04.732+0000 7fa9bcfd4700 1 -- 172.21.15.52:0/1596529046 >> 172.21.15.52:0/1596529046 conn(0x7fa9b80fedb0 msgr2=0x7fa9b8107520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:22:04.723 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:22:04.732+0000 7fa9bcfd4700 1 -- 172.21.15.52:0/1596529046 shutdown_connections 2024-05-27T07:22:04.723 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:22:04.732+0000 7fa9bcfd4700 1 -- 172.21.15.52:0/1596529046 wait complete. 2024-05-27T07:22:04.723 INFO:teuthology.orchestra.run.smithi052.stderr:added data pool 4 to fsmap 2024-05-27T07:22:04.979 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:22:04 smithi052 ceph-mon[31955]: pgmap v302: 129 pgs: 2 unknown, 127 active+clean; 579 KiB data, 356 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-27T07:22:04.979 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:22:04 smithi052 ceph-mon[31955]: from='client.? 172.21.15.52:0/1596529046' entity='client.admin' cmd=[{"prefix": "fs add_data_pool", "fs_name": "cephfs", "pool": "cephfs_data_ec"}]: dispatch 2024-05-27T07:22:05.102 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:22:04 smithi173 ceph-mon[37000]: pgmap v302: 129 pgs: 2 unknown, 127 active+clean; 579 KiB data, 356 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-27T07:22:05.102 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:22:04 smithi173 ceph-mon[37000]: from='client.? 172.21.15.52:0/1596529046' entity='client.admin' cmd=[{"prefix": "fs add_data_pool", "fs_name": "cephfs", "pool": "cephfs_data_ec"}]: dispatch 2024-05-27T07:22:05.190 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:22:04 smithi105 ceph-mon[35977]: pgmap v302: 129 pgs: 2 unknown, 127 active+clean; 579 KiB data, 356 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-27T07:22:05.190 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:22:04 smithi105 ceph-mon[35977]: from='client.? 172.21.15.52:0/1596529046' entity='client.admin' cmd=[{"prefix": "fs add_data_pool", "fs_name": "cephfs", "pool": "cephfs_data_ec"}]: dispatch 2024-05-27T07:22:05.691 DEBUG:teuthology.orchestra.run.smithi052:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:38f9d900c83c63799fbdbe61acc9a11b0d3554a6 shell --fsid 73df2636-1bf8-11ef-bc9b-c7b262605968 -- ceph fs dump --format=json 2024-05-27T07:22:05.950 INFO:teuthology.orchestra.run.smithi052.stderr:Inferring config /var/lib/ceph/73df2636-1bf8-11ef-bc9b-c7b262605968/mon.a/config 2024-05-27T07:22:05.979 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:22:05 smithi052 ceph-mon[31955]: from='client.? 172.21.15.52:0/1596529046' entity='client.admin' cmd='[{"prefix": "fs add_data_pool", "fs_name": "cephfs", "pool": "cephfs_data_ec"}]': finished 2024-05-27T07:22:05.979 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:22:05 smithi052 ceph-mon[31955]: osdmap e86: 12 total, 12 up, 12 in 2024-05-27T07:22:05.979 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:22:05 smithi052 ceph-mon[31955]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-05-27T07:22:06.102 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:22:05 smithi173 ceph-mon[37000]: from='client.? 172.21.15.52:0/1596529046' entity='client.admin' cmd='[{"prefix": "fs add_data_pool", "fs_name": "cephfs", "pool": "cephfs_data_ec"}]': finished 2024-05-27T07:22:06.102 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:22:05 smithi173 ceph-mon[37000]: osdmap e86: 12 total, 12 up, 12 in 2024-05-27T07:22:06.102 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:22:05 smithi173 ceph-mon[37000]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-05-27T07:22:06.190 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:22:05 smithi105 ceph-mon[35977]: from='client.? 172.21.15.52:0/1596529046' entity='client.admin' cmd='[{"prefix": "fs add_data_pool", "fs_name": "cephfs", "pool": "cephfs_data_ec"}]': finished 2024-05-27T07:22:06.190 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:22:05 smithi105 ceph-mon[35977]: osdmap e86: 12 total, 12 up, 12 in 2024-05-27T07:22:06.190 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:22:05 smithi105 ceph-mon[35977]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-05-27T07:22:07.102 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:22:06 smithi173 ceph-mon[37000]: pgmap v304: 129 pgs: 129 active+clean; 579 KiB data, 356 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-27T07:22:07.102 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:22:06 smithi173 ceph-mon[37000]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-05-27T07:22:07.102 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:22:06 smithi173 ceph-mon[37000]: Cluster is now healthy 2024-05-27T07:22:07.190 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:22:06 smithi105 ceph-mon[35977]: pgmap v304: 129 pgs: 129 active+clean; 579 KiB data, 356 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-27T07:22:07.190 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:22:06 smithi105 ceph-mon[35977]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-05-27T07:22:07.190 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:22:06 smithi105 ceph-mon[35977]: Cluster is now healthy 2024-05-27T07:22:07.214 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:22:06 smithi052 ceph-mon[31955]: pgmap v304: 129 pgs: 129 active+clean; 579 KiB data, 356 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-27T07:22:07.214 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:22:06 smithi052 ceph-mon[31955]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-05-27T07:22:07.214 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:22:06 smithi052 ceph-mon[31955]: Cluster is now healthy 2024-05-27T07:22:07.736 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:22:07.746+0000 7f894f416700 1 -- 172.21.15.52:0/3054408845 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f8948107de0 msgr2=0x7f89481081f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:22:07.737 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:22:07.746+0000 7f894f416700 1 --2- 172.21.15.52:0/3054408845 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f8948107de0 0x7f89481081f0 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7f89380099c0 tx=0x7f8938009cd0 comp rx=0 tx=0).stop 2024-05-27T07:22:07.737 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:22:07.746+0000 7f894f416700 1 -- 172.21.15.52:0/3054408845 shutdown_connections 2024-05-27T07:22:07.737 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:22:07.746+0000 7f894f416700 1 --2- 172.21.15.52:0/3054408845 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f8948108730 0x7f8948110090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:22:07.737 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:22:07.746+0000 7f894f416700 1 --2- 172.21.15.52:0/3054408845 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f89481034b0 0x7f89481038c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:22:07.737 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:22:07.746+0000 7f894f416700 1 --2- 172.21.15.52:0/3054408845 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f8948107de0 0x7f89481081f0 unknown :-1 s=CLOSED pgs=195 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:22:07.737 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:22:07.746+0000 7f894f416700 1 -- 172.21.15.52:0/3054408845 >> 172.21.15.52:0/3054408845 conn(0x7f89480fedb0 msgr2=0x7f89481011d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:22:07.737 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:22:07.746+0000 7f894f416700 1 -- 172.21.15.52:0/3054408845 shutdown_connections 2024-05-27T07:22:07.737 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:22:07.746+0000 7f894f416700 1 -- 172.21.15.52:0/3054408845 wait complete. 2024-05-27T07:22:07.739 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:22:07.747+0000 7f894f416700 1 Processor -- start 2024-05-27T07:22:07.739 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:22:07.747+0000 7f894f416700 1 -- start start 2024-05-27T07:22:07.739 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:22:07.747+0000 7f894f416700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f89481034b0 0x7f894819bbe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:22:07.739 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:22:07.747+0000 7f894f416700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f8948107de0 0x7f894819c120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:22:07.740 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:22:07.747+0000 7f894f416700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f8948108730 0x7f894819c6a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:22:07.740 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:22:07.748+0000 7f894f416700 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x7f8948113340 con 0x7f89481034b0 2024-05-27T07:22:07.740 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:22:07.748+0000 7f894f416700 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f89481131c0 con 0x7f8948107de0 2024-05-27T07:22:07.740 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:22:07.748+0000 7f894f416700 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f89481134c0 con 0x7f8948108730 2024-05-27T07:22:07.740 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:22:07.748+0000 7f894d1b2700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f89481034b0 0x7f894819bbe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:22:07.740 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:22:07.748+0000 7f894d1b2700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f89481034b0 0x7f894819bbe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.52:56958/0 (socket says 172.21.15.52:56958) 2024-05-27T07:22:07.740 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:22:07.748+0000 7f894d1b2700 1 -- 172.21.15.52:0/2951330680 learned_addr learned my addr 172.21.15.52:0/2951330680 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-05-27T07:22:07.740 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:22:07.748+0000 7f894c9b1700 1 --2- 172.21.15.52:0/2951330680 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f8948107de0 0x7f894819c120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:22:07.740 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:22:07.748+0000 7f894d9b3700 1 --2- 172.21.15.52:0/2951330680 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f8948108730 0x7f894819c6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:22:07.740 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:22:07.748+0000 7f894d1b2700 1 -- 172.21.15.52:0/2951330680 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f8948108730 msgr2=0x7f894819c6a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:22:07.740 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:22:07.748+0000 7f894d1b2700 1 --2- 172.21.15.52:0/2951330680 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f8948108730 0x7f894819c6a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:22:07.740 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:22:07.748+0000 7f894d1b2700 1 -- 172.21.15.52:0/2951330680 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f8948107de0 msgr2=0x7f894819c120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:22:07.741 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:22:07.748+0000 7f894d1b2700 1 --2- 172.21.15.52:0/2951330680 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f8948107de0 0x7f894819c120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:22:07.741 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:22:07.748+0000 7f894d1b2700 1 -- 172.21.15.52:0/2951330680 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f89481a6ec0 con 0x7f89481034b0 2024-05-27T07:22:07.741 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:22:07.748+0000 7f894d1b2700 1 --2- 172.21.15.52:0/2951330680 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f89481034b0 0x7f894819bbe0 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7f894400c9b0 tx=0x7f894400ccc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:22:07.741 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:22:07.749+0000 7f893e7fc700 1 -- 172.21.15.52:0/2951330680 <== mon.0 v2:172.21.15.52:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8944007b00 con 0x7f89481034b0 2024-05-27T07:22:07.741 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:22:07.749+0000 7f894f416700 1 -- 172.21.15.52:0/2951330680 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f89481a71b0 con 0x7f89481034b0 2024-05-27T07:22:07.741 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:22:07.749+0000 7f893e7fc700 1 -- 172.21.15.52:0/2951330680 <== mon.0 v2:172.21.15.52:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f894400ce50 con 0x7f89481034b0 2024-05-27T07:22:07.741 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:22:07.749+0000 7f894f416700 1 -- 172.21.15.52:0/2951330680 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f89481a7710 con 0x7f89481034b0 2024-05-27T07:22:07.741 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:22:07.749+0000 7f893e7fc700 1 -- 172.21.15.52:0/2951330680 <== mon.0 v2:172.21.15.52:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8944018660 con 0x7f89481034b0 2024-05-27T07:22:07.742 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:22:07.750+0000 7f894f416700 1 -- 172.21.15.52:0/2951330680 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f89480512b0 con 0x7f89481034b0 2024-05-27T07:22:07.743 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:22:07.751+0000 7f893e7fc700 1 -- 172.21.15.52:0/2951330680 <== mon.0 v2:172.21.15.52:3300/0 4 ==== mgrmap(e 16) ==== 149652+0+0 (secure 0 0 0) 0x7f894400f450 con 0x7f89481034b0 2024-05-27T07:22:07.743 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:22:07.752+0000 7f893e7fc700 1 --2- 172.21.15.52:0/2951330680 >> [v2:172.21.15.52:6800/3513075805,v1:172.21.15.52:6801/3513075805] conn(0x7f89340b2900 0x7f89340b4e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:22:07.743 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:22:07.752+0000 7f894c9b1700 1 --2- 172.21.15.52:0/2951330680 >> [v2:172.21.15.52:6800/3513075805,v1:172.21.15.52:6801/3513075805] conn(0x7f89340b2900 0x7f89340b4e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:22:07.743 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:22:07.752+0000 7f893e7fc700 1 -- 172.21.15.52:0/2951330680 <== mon.0 v2:172.21.15.52:3300/0 5 ==== osd_map(86..86 src has 1..86) ==== 8802+0+0 (secure 0 0 0) 0x7f89440e2360 con 0x7f89481034b0 2024-05-27T07:22:07.743 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:22:07.753+0000 7f894c9b1700 1 --2- 172.21.15.52:0/2951330680 >> [v2:172.21.15.52:6800/3513075805,v1:172.21.15.52:6801/3513075805] conn(0x7f89340b2900 0x7f89340b4e10 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7f8938006010 tx=0x7f8938000bc0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:22:07.747 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:22:07.757+0000 7f893e7fc700 1 -- 172.21.15.52:0/2951330680 <== mon.0 v2:172.21.15.52:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191963 (secure 0 0 0) 0x7f8944014030 con 0x7f89481034b0 2024-05-27T07:22:08.006 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:22:08.016+0000 7f894f416700 1 -- 172.21.15.52:0/2951330680 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f894806bc10 con 0x7f89481034b0 2024-05-27T07:22:08.007 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:22:08.017+0000 7f893e7fc700 1 -- 172.21.15.52:0/2951330680 <== mon.0 v2:172.21.15.52:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 17 v17) ==== 94+0+11803 (secure 0 0 0) 0x7f89440a9de0 con 0x7f89481034b0 2024-05-27T07:22:08.007 INFO:teuthology.orchestra.run.smithi052.stdout: 2024-05-27T07:22:08.007 INFO:teuthology.orchestra.run.smithi052.stdout:{"epoch":17,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14556,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.52:6835/2584733778","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.52:6834","nonce":2584733778},{"type":"v1","addr":"172.21.15.52:6835","nonce":2584733778}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14568,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.52:6837/675445303","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.52:6836","nonce":675445303},{"type":"v1","addr":"172.21.15.52:6837","nonce":675445303}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14619,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.52:6839/4258549559","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.52:6838","nonce":4258549559},{"type":"v1","addr":"172.21.15.52:6839","nonce":4258549559}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14631,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.52:6841/2878618188","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.52:6840","nonce":2878618188},{"type":"v1","addr":"172.21.15.52:6841","nonce":2878618188}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24433,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.173:6833/3013589220","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6832","nonce":3013589220},{"type":"v1","addr":"172.21.15.173:6833","nonce":3013589220}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24452,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.105:6833/2800678412","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6832","nonce":2800678412},{"type":"v1","addr":"172.21.15.105:6833","nonce":2800678412}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24460,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.173:6835/1619486437","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6834","nonce":1619486437},{"type":"v1","addr":"172.21.15.173:6835","nonce":1619486437}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":7},{"gid":24466,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.173:6837/1396363827","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6836","nonce":1396363827},{"type":"v1","addr":"172.21.15.173:6837","nonce":1396363827}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":8},{"gid":24473,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.105:6835/3298498384","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6834","nonce":3298498384},{"type":"v1","addr":"172.21.15.105:6835","nonce":3298498384}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6},{"gid":24475,"name":"i","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.173:6839/564778293","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6838","nonce":564778293},{"type":"v1","addr":"172.21.15.173:6839","nonce":564778293}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":10},{"gid":24497,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.105:6837/191393149","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6836","nonce":191393149},{"type":"v1","addr":"172.21.15.105:6837","nonce":191393149}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":9}],"filesystems":[{"mdsmap":{"epoch":17,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-05-27T07:21:50.521783+0000","modified":"2024-05-27T07:22:04.612487+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":24527},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24527":{"gid":24527,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":5,"addr":"172.21.15.105:6839/500348151","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6838","nonce":500348151},{"type":"v1","addr":"172.21.15.105:6839","nonce":500348151}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":1,"qdb_leader":24527,"qdb_cluster":[24527]},"id":1}]} 2024-05-27T07:22:08.010 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:22:08.019+0000 7f894f416700 1 -- 172.21.15.52:0/2951330680 >> [v2:172.21.15.52:6800/3513075805,v1:172.21.15.52:6801/3513075805] conn(0x7f89340b2900 msgr2=0x7f89340b4e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:22:08.010 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:22:08.019+0000 7f894f416700 1 --2- 172.21.15.52:0/2951330680 >> [v2:172.21.15.52:6800/3513075805,v1:172.21.15.52:6801/3513075805] conn(0x7f89340b2900 0x7f89340b4e10 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7f8938006010 tx=0x7f8938000bc0 comp rx=0 tx=0).stop 2024-05-27T07:22:08.010 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:22:08.019+0000 7f894f416700 1 -- 172.21.15.52:0/2951330680 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f89481034b0 msgr2=0x7f894819bbe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:22:08.010 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:22:08.019+0000 7f894f416700 1 --2- 172.21.15.52:0/2951330680 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f89481034b0 0x7f894819bbe0 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7f894400c9b0 tx=0x7f894400ccc0 comp rx=0 tx=0).stop 2024-05-27T07:22:08.010 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:22:08.019+0000 7f894f416700 1 -- 172.21.15.52:0/2951330680 shutdown_connections 2024-05-27T07:22:08.010 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:22:08.019+0000 7f894f416700 1 --2- 172.21.15.52:0/2951330680 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f8948108730 0x7f894819c6a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:22:08.010 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:22:08.019+0000 7f894f416700 1 --2- 172.21.15.52:0/2951330680 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f89481034b0 0x7f894819bbe0 unknown :-1 s=CLOSED pgs=196 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:22:08.010 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:22:08.019+0000 7f894f416700 1 --2- 172.21.15.52:0/2951330680 >> [v2:172.21.15.52:6800/3513075805,v1:172.21.15.52:6801/3513075805] conn(0x7f89340b2900 0x7f89340b4e10 unknown :-1 s=CLOSED pgs=148 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:22:08.010 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:22:08.019+0000 7f894f416700 1 --2- 172.21.15.52:0/2951330680 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f8948107de0 0x7f894819c120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:22:08.011 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:22:08.019+0000 7f894f416700 1 -- 172.21.15.52:0/2951330680 >> 172.21.15.52:0/2951330680 conn(0x7f89480fedb0 msgr2=0x7f8948109690 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:22:08.011 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:22:08.020+0000 7f894f416700 1 -- 172.21.15.52:0/2951330680 shutdown_connections 2024-05-27T07:22:08.011 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:22:08.020+0000 7f894f416700 1 -- 172.21.15.52:0/2951330680 wait complete. 2024-05-27T07:22:08.011 INFO:teuthology.orchestra.run.smithi052.stderr:dumped fsmap epoch 17 2024-05-27T07:22:08.721 DEBUG:teuthology.orchestra.run.smithi052:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:38f9d900c83c63799fbdbe61acc9a11b0d3554a6 shell --fsid 73df2636-1bf8-11ef-bc9b-c7b262605968 -- ceph osd dump --format=json 2024-05-27T07:22:08.973 INFO:teuthology.orchestra.run.smithi052.stderr:Inferring config /var/lib/ceph/73df2636-1bf8-11ef-bc9b-c7b262605968/mon.a/config 2024-05-27T07:22:08.979 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:22:08 smithi052 ceph-mon[31955]: pgmap v305: 129 pgs: 129 active+clean; 579 KiB data, 356 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-27T07:22:08.979 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:22:08 smithi052 ceph-mon[31955]: from='client.? 172.21.15.52:0/2951330680' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-27T07:22:09.102 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:22:08 smithi173 ceph-mon[37000]: pgmap v305: 129 pgs: 129 active+clean; 579 KiB data, 356 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-27T07:22:09.102 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:22:08 smithi173 ceph-mon[37000]: from='client.? 172.21.15.52:0/2951330680' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-27T07:22:09.190 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:22:08 smithi105 ceph-mon[35977]: pgmap v305: 129 pgs: 129 active+clean; 579 KiB data, 356 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-27T07:22:09.190 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:22:08 smithi105 ceph-mon[35977]: from='client.? 172.21.15.52:0/2951330680' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-27T07:22:10.755 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:22:10.764+0000 7f33c21e2700 1 -- 172.21.15.52:0/2664009075 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f33bc1092f0 msgr2=0x7f33bc109700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:22:10.755 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:22:10.764+0000 7f33c21e2700 1 --2- 172.21.15.52:0/2664009075 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f33bc1092f0 0x7f33bc109700 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7f33ac009990 tx=0x7f33ac009ca0 comp rx=0 tx=0).stop 2024-05-27T07:22:10.755 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:22:10.764+0000 7f33c21e2700 1 -- 172.21.15.52:0/2664009075 shutdown_connections 2024-05-27T07:22:10.755 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:22:10.764+0000 7f33c21e2700 1 --2- 172.21.15.52:0/2664009075 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f33bc1030d0 0x7f33bc111160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:22:10.755 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:22:10.764+0000 7f33c21e2700 1 --2- 172.21.15.52:0/2664009075 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f33bc1092f0 0x7f33bc109700 unknown :-1 s=CLOSED pgs=197 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:22:10.755 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:22:10.764+0000 7f33c21e2700 1 --2- 172.21.15.52:0/2664009075 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f33bc102780 0x7f33bc102b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:22:10.755 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:22:10.764+0000 7f33c21e2700 1 -- 172.21.15.52:0/2664009075 >> 172.21.15.52:0/2664009075 conn(0x7f33bc0fe4d0 msgr2=0x7f33bc100920 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:22:10.755 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:22:10.764+0000 7f33c21e2700 1 -- 172.21.15.52:0/2664009075 shutdown_connections 2024-05-27T07:22:10.755 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:22:10.764+0000 7f33c21e2700 1 -- 172.21.15.52:0/2664009075 wait complete. 2024-05-27T07:22:10.756 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:22:10 smithi052 ceph-mon[31955]: pgmap v306: 129 pgs: 129 active+clean; 579 KiB data, 356 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-27T07:22:10.757 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:22:10.765+0000 7f33c21e2700 1 Processor -- start 2024-05-27T07:22:10.757 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:22:10.765+0000 7f33c21e2700 1 -- start start 2024-05-27T07:22:10.758 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:22:10.766+0000 7f33c21e2700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f33bc102780 0x7f33bc19af70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:22:10.758 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:22:10.766+0000 7f33c21e2700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f33bc1030d0 0x7f33bc19b510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:22:10.758 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:22:10.766+0000 7f33c21e2700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f33bc1092f0 0x7f33bc19f7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:22:10.758 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:22:10.766+0000 7f33c21e2700 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x7f33bc114580 con 0x7f33bc1092f0 2024-05-27T07:22:10.758 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:22:10.766+0000 7f33c21e2700 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f33bc114400 con 0x7f33bc1030d0 2024-05-27T07:22:10.758 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:22:10.766+0000 7f33c21e2700 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f33bc114700 con 0x7f33bc102780 2024-05-27T07:22:10.758 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:22:10.766+0000 7f33bbfff700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f33bc1092f0 0x7f33bc19f7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:22:10.758 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:22:10.766+0000 7f33bbfff700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f33bc1092f0 0x7f33bc19f7a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.52:37586/0 (socket says 172.21.15.52:37586) 2024-05-27T07:22:10.758 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:22:10.766+0000 7f33bbfff700 1 -- 172.21.15.52:0/2431155280 learned_addr learned my addr 172.21.15.52:0/2431155280 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-05-27T07:22:10.758 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:22:10.766+0000 7f33baffd700 1 --2- 172.21.15.52:0/2431155280 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f33bc1030d0 0x7f33bc19b510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:22:10.759 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:22:10.766+0000 7f33bb7fe700 1 --2- 172.21.15.52:0/2431155280 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f33bc102780 0x7f33bc19af70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:22:10.759 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:22:10.766+0000 7f33bbfff700 1 -- 172.21.15.52:0/2431155280 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f33bc102780 msgr2=0x7f33bc19af70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:22:10.759 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:22:10.766+0000 7f33bbfff700 1 --2- 172.21.15.52:0/2431155280 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f33bc102780 0x7f33bc19af70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:22:10.759 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:22:10.766+0000 7f33bbfff700 1 -- 172.21.15.52:0/2431155280 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f33bc1030d0 msgr2=0x7f33bc19b510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:22:10.759 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:22:10.766+0000 7f33bbfff700 1 --2- 172.21.15.52:0/2431155280 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f33bc1030d0 0x7f33bc19b510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:22:10.759 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:22:10.766+0000 7f33bbfff700 1 -- 172.21.15.52:0/2431155280 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f33bc19fd90 con 0x7f33bc1092f0 2024-05-27T07:22:10.759 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:22:10.766+0000 7f33bbfff700 1 --2- 172.21.15.52:0/2431155280 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f33bc1092f0 0x7f33bc19f7a0 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7f33b4007ee0 tx=0x7f33b400d3b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:22:10.759 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:22:10.766+0000 7f33b8ff9700 1 -- 172.21.15.52:0/2431155280 <== mon.0 v2:172.21.15.52:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f33b4004bb0 con 0x7f33bc1092f0 2024-05-27T07:22:10.759 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:22:10.767+0000 7f33b8ff9700 1 -- 172.21.15.52:0/2431155280 <== mon.0 v2:172.21.15.52:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f33b4004d50 con 0x7f33bc1092f0 2024-05-27T07:22:10.759 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:22:10.767+0000 7f33c21e2700 1 -- 172.21.15.52:0/2431155280 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f33bc1a0080 con 0x7f33bc1092f0 2024-05-27T07:22:10.759 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:22:10.767+0000 7f33b8ff9700 1 -- 172.21.15.52:0/2431155280 <== mon.0 v2:172.21.15.52:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f33b4005920 con 0x7f33bc1092f0 2024-05-27T07:22:10.759 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:22:10.767+0000 7f33c21e2700 1 -- 172.21.15.52:0/2431155280 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f33bc1a0570 con 0x7f33bc1092f0 2024-05-27T07:22:10.759 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:22:10.768+0000 7f33c21e2700 1 -- 172.21.15.52:0/2431155280 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f33bc0512b0 con 0x7f33bc1092f0 2024-05-27T07:22:10.760 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:22:10.770+0000 7f33b8ff9700 1 -- 172.21.15.52:0/2431155280 <== mon.0 v2:172.21.15.52:3300/0 4 ==== mgrmap(e 16) ==== 149652+0+0 (secure 0 0 0) 0x7f33b401a020 con 0x7f33bc1092f0 2024-05-27T07:22:10.761 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:22:10.771+0000 7f33b8ff9700 1 --2- 172.21.15.52:0/2431155280 >> [v2:172.21.15.52:6800/3513075805,v1:172.21.15.52:6801/3513075805] conn(0x7f33a40b2880 0x7f33a40b4d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:22:10.761 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:22:10.771+0000 7f33bb7fe700 1 --2- 172.21.15.52:0/2431155280 >> [v2:172.21.15.52:6800/3513075805,v1:172.21.15.52:6801/3513075805] conn(0x7f33a40b2880 0x7f33a40b4d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:22:10.761 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:22:10.771+0000 7f33b8ff9700 1 -- 172.21.15.52:0/2431155280 <== mon.0 v2:172.21.15.52:3300/0 5 ==== osd_map(86..86 src has 1..86) ==== 8802+0+0 (secure 0 0 0) 0x7f33b40e2110 con 0x7f33bc1092f0 2024-05-27T07:22:10.762 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:22:10.772+0000 7f33bb7fe700 1 --2- 172.21.15.52:0/2431155280 >> [v2:172.21.15.52:6800/3513075805,v1:172.21.15.52:6801/3513075805] conn(0x7f33a40b2880 0x7f33a40b4d90 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7f33ac006010 tx=0x7f33ac005960 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:22:10.765 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:22:10.775+0000 7f33b8ff9700 1 -- 172.21.15.52:0/2431155280 <== mon.0 v2:172.21.15.52:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191963 (secure 0 0 0) 0x7f33b4010070 con 0x7f33bc1092f0 2024-05-27T07:22:10.978 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:22:10.988+0000 7f33c21e2700 1 -- 172.21.15.52:0/2431155280 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f33bc19c2d0 con 0x7f33bc1092f0 2024-05-27T07:22:10.979 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:22:10.990+0000 7f33b8ff9700 1 -- 172.21.15.52:0/2431155280 <== mon.0 v2:172.21.15.52:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v86) ==== 74+0+24126 (secure 0 0 0) 0x7f33b40a9b90 con 0x7f33bc1092f0 2024-05-27T07:22:10.980 INFO:teuthology.orchestra.run.smithi052.stdout: 2024-05-27T07:22:10.980 INFO:teuthology.orchestra.run.smithi052.stdout:{"epoch":86,"fsid":"73df2636-1bf8-11ef-bc9b-c7b262605968","created":"2024-05-27T07:13:34.738032+0000","modified":"2024-05-27T07:22:04.612333+0000","last_up_change":"2024-05-27T07:20:45.777597+0000","last_in_change":"2024-05-27T07:20:25.314823+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-05-27T07:16:57.644034+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-05-27T07:21:43.917748+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"79","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-05-27T07:21:47.341876+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"79","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-05-27T07:21:57.707611+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"86","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"347b8a89-9dbe-4848-83b0-fd0e9374f33d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":82,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.52:6802","nonce":1626508899},{"type":"v1","addr":"172.21.15.52:6803","nonce":1626508899}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.52:6804","nonce":1626508899},{"type":"v1","addr":"172.21.15.52:6805","nonce":1626508899}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.52:6808","nonce":1626508899},{"type":"v1","addr":"172.21.15.52:6809","nonce":1626508899}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.52:6806","nonce":1626508899},{"type":"v1","addr":"172.21.15.52:6807","nonce":1626508899}]},"public_addr":"172.21.15.52:6803/1626508899","cluster_addr":"172.21.15.52:6805/1626508899","heartbeat_back_addr":"172.21.15.52:6809/1626508899","heartbeat_front_addr":"172.21.15.52:6807/1626508899","state":["exists","up"]},{"osd":1,"uuid":"7a3623d4-e087-40cb-9336-d4a5c52203d4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":82,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6800","nonce":2900093195},{"type":"v1","addr":"172.21.15.105:6801","nonce":2900093195}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6802","nonce":2900093195},{"type":"v1","addr":"172.21.15.105:6803","nonce":2900093195}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6806","nonce":2900093195},{"type":"v1","addr":"172.21.15.105:6807","nonce":2900093195}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6804","nonce":2900093195},{"type":"v1","addr":"172.21.15.105:6805","nonce":2900093195}]},"public_addr":"172.21.15.105:6801/2900093195","cluster_addr":"172.21.15.105:6803/2900093195","heartbeat_back_addr":"172.21.15.105:6807/2900093195","heartbeat_front_addr":"172.21.15.105:6805/2900093195","state":["exists","up"]},{"osd":2,"uuid":"41a9df5b-882a-47da-bb26-7f4d26ccf6c1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":82,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6800","nonce":2251189314},{"type":"v1","addr":"172.21.15.173:6801","nonce":2251189314}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6802","nonce":2251189314},{"type":"v1","addr":"172.21.15.173:6803","nonce":2251189314}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6806","nonce":2251189314},{"type":"v1","addr":"172.21.15.173:6807","nonce":2251189314}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6804","nonce":2251189314},{"type":"v1","addr":"172.21.15.173:6805","nonce":2251189314}]},"public_addr":"172.21.15.173:6801/2251189314","cluster_addr":"172.21.15.173:6803/2251189314","heartbeat_back_addr":"172.21.15.173:6807/2251189314","heartbeat_front_addr":"172.21.15.173:6805/2251189314","state":["exists","up"]},{"osd":3,"uuid":"f1245439-42a5-46c5-a998-e0a78f9df3fb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":26,"up_thru":82,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.52:6810","nonce":57920794},{"type":"v1","addr":"172.21.15.52:6811","nonce":57920794}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.52:6812","nonce":57920794},{"type":"v1","addr":"172.21.15.52:6813","nonce":57920794}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.52:6816","nonce":57920794},{"type":"v1","addr":"172.21.15.52:6817","nonce":57920794}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.52:6814","nonce":57920794},{"type":"v1","addr":"172.21.15.52:6815","nonce":57920794}]},"public_addr":"172.21.15.52:6811/57920794","cluster_addr":"172.21.15.52:6813/57920794","heartbeat_back_addr":"172.21.15.52:6817/57920794","heartbeat_front_addr":"172.21.15.52:6815/57920794","state":["exists","up"]},{"osd":4,"uuid":"2592d1cc-1726-4359-83d8-a2e6314b54ec","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":32,"up_thru":82,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6808","nonce":3441134818},{"type":"v1","addr":"172.21.15.105:6809","nonce":3441134818}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6810","nonce":3441134818},{"type":"v1","addr":"172.21.15.105:6811","nonce":3441134818}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6814","nonce":3441134818},{"type":"v1","addr":"172.21.15.105:6815","nonce":3441134818}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6812","nonce":3441134818},{"type":"v1","addr":"172.21.15.105:6813","nonce":3441134818}]},"public_addr":"172.21.15.105:6809/3441134818","cluster_addr":"172.21.15.105:6811/3441134818","heartbeat_back_addr":"172.21.15.105:6815/3441134818","heartbeat_front_addr":"172.21.15.105:6813/3441134818","state":["exists","up"]},{"osd":5,"uuid":"27e98ee2-4a92-4972-b2a3-959d0ed90981","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":37,"up_thru":82,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6808","nonce":2989239729},{"type":"v1","addr":"172.21.15.173:6809","nonce":2989239729}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6810","nonce":2989239729},{"type":"v1","addr":"172.21.15.173:6811","nonce":2989239729}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6814","nonce":2989239729},{"type":"v1","addr":"172.21.15.173:6815","nonce":2989239729}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6812","nonce":2989239729},{"type":"v1","addr":"172.21.15.173:6813","nonce":2989239729}]},"public_addr":"172.21.15.173:6809/2989239729","cluster_addr":"172.21.15.173:6811/2989239729","heartbeat_back_addr":"172.21.15.173:6815/2989239729","heartbeat_front_addr":"172.21.15.173:6813/2989239729","state":["exists","up"]},{"osd":6,"uuid":"2b0ec0cb-f438-4b2c-8642-0778ed374e9a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":43,"up_thru":82,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.52:6818","nonce":4244423065},{"type":"v1","addr":"172.21.15.52:6819","nonce":4244423065}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.52:6820","nonce":4244423065},{"type":"v1","addr":"172.21.15.52:6821","nonce":4244423065}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.52:6824","nonce":4244423065},{"type":"v1","addr":"172.21.15.52:6825","nonce":4244423065}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.52:6822","nonce":4244423065},{"type":"v1","addr":"172.21.15.52:6823","nonce":4244423065}]},"public_addr":"172.21.15.52:6819/4244423065","cluster_addr":"172.21.15.52:6821/4244423065","heartbeat_back_addr":"172.21.15.52:6825/4244423065","heartbeat_front_addr":"172.21.15.52:6823/4244423065","state":["exists","up"]},{"osd":7,"uuid":"cb2a7a6b-8015-43ea-865a-be144e45c51a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":48,"up_thru":82,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6816","nonce":2063611552},{"type":"v1","addr":"172.21.15.105:6817","nonce":2063611552}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6818","nonce":2063611552},{"type":"v1","addr":"172.21.15.105:6819","nonce":2063611552}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6822","nonce":2063611552},{"type":"v1","addr":"172.21.15.105:6823","nonce":2063611552}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6820","nonce":2063611552},{"type":"v1","addr":"172.21.15.105:6821","nonce":2063611552}]},"public_addr":"172.21.15.105:6817/2063611552","cluster_addr":"172.21.15.105:6819/2063611552","heartbeat_back_addr":"172.21.15.105:6823/2063611552","heartbeat_front_addr":"172.21.15.105:6821/2063611552","state":["exists","up"]},{"osd":8,"uuid":"c2fc565d-a0b9-4396-a209-bd884cddcfe3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":54,"up_thru":82,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6816","nonce":4131399852},{"type":"v1","addr":"172.21.15.173:6817","nonce":4131399852}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6818","nonce":4131399852},{"type":"v1","addr":"172.21.15.173:6819","nonce":4131399852}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6822","nonce":4131399852},{"type":"v1","addr":"172.21.15.173:6823","nonce":4131399852}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6820","nonce":4131399852},{"type":"v1","addr":"172.21.15.173:6821","nonce":4131399852}]},"public_addr":"172.21.15.173:6817/4131399852","cluster_addr":"172.21.15.173:6819/4131399852","heartbeat_back_addr":"172.21.15.173:6823/4131399852","heartbeat_front_addr":"172.21.15.173:6821/4131399852","state":["exists","up"]},{"osd":9,"uuid":"78fa7f17-1dad-4262-be40-31fa82e75520","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":60,"up_thru":82,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.52:6826","nonce":479119778},{"type":"v1","addr":"172.21.15.52:6827","nonce":479119778}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.52:6828","nonce":479119778},{"type":"v1","addr":"172.21.15.52:6829","nonce":479119778}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.52:6832","nonce":479119778},{"type":"v1","addr":"172.21.15.52:6833","nonce":479119778}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.52:6830","nonce":479119778},{"type":"v1","addr":"172.21.15.52:6831","nonce":479119778}]},"public_addr":"172.21.15.52:6827/479119778","cluster_addr":"172.21.15.52:6829/479119778","heartbeat_back_addr":"172.21.15.52:6833/479119778","heartbeat_front_addr":"172.21.15.52:6831/479119778","state":["exists","up"]},{"osd":10,"uuid":"e6f8e753-e15c-45b5-8ee9-d50f384559f9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":65,"up_thru":76,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6824","nonce":4032386628},{"type":"v1","addr":"172.21.15.105:6825","nonce":4032386628}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6826","nonce":4032386628},{"type":"v1","addr":"172.21.15.105:6827","nonce":4032386628}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6830","nonce":4032386628},{"type":"v1","addr":"172.21.15.105:6831","nonce":4032386628}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6828","nonce":4032386628},{"type":"v1","addr":"172.21.15.105:6829","nonce":4032386628}]},"public_addr":"172.21.15.105:6825/4032386628","cluster_addr":"172.21.15.105:6827/4032386628","heartbeat_back_addr":"172.21.15.105:6831/4032386628","heartbeat_front_addr":"172.21.15.105:6829/4032386628","state":["exists","up"]},{"osd":11,"uuid":"164572bf-67ed-4e8e-b32d-d7a2e8da607b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":71,"up_thru":82,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6824","nonce":1816308401},{"type":"v1","addr":"172.21.15.173:6825","nonce":1816308401}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6826","nonce":1816308401},{"type":"v1","addr":"172.21.15.173:6827","nonce":1816308401}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6830","nonce":1816308401},{"type":"v1","addr":"172.21.15.173:6831","nonce":1816308401}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6828","nonce":1816308401},{"type":"v1","addr":"172.21.15.173:6829","nonce":1816308401}]},"public_addr":"172.21.15.173:6825/1816308401","cluster_addr":"172.21.15.173:6827/1816308401","heartbeat_back_addr":"172.21.15.173:6831/1816308401","heartbeat_front_addr":"172.21.15.173:6829/1816308401","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-27T07:16:06.279220+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-27T07:16:29.668167+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-27T07:16:53.903808+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-27T07:17:18.787372+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-27T07:17:43.480784+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-27T07:18:08.816030+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-27T07:18:34.076148+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-27T07:19:00.283468+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-27T07:19:25.403023+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-27T07:19:52.139173+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-27T07:20:17.530304+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-27T07:20:43.665632+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.52:6801/1631683163":"2024-05-28T07:14:29.544633+0000","172.21.15.52:0/2776144659":"2024-05-28T07:14:29.544633+0000","172.21.15.52:6800/1631683163":"2024-05-28T07:14:29.544633+0000","172.21.15.52:0/3675529097":"2024-05-28T07:14:01.960550+0000","172.21.15.52:0/727808107":"2024-05-28T07:14:01.960550+0000","172.21.15.52:0/3839436833":"2024-05-28T07:14:29.544633+0000","172.21.15.52:6800/587662512":"2024-05-28T07:14:01.960550+0000","172.21.15.52:6801/587662512":"2024-05-28T07:14:01.960550+0000","172.21.15.52:0/853867465":"2024-05-28T07:14:01.960550+0000","172.21.15.52:0/3480918279":"2024-05-28T07:14:29.544633+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-05-27T07:22:10.982 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:22:10.992+0000 7f33c21e2700 1 -- 172.21.15.52:0/2431155280 >> [v2:172.21.15.52:6800/3513075805,v1:172.21.15.52:6801/3513075805] conn(0x7f33a40b2880 msgr2=0x7f33a40b4d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:22:10.982 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:22:10.992+0000 7f33c21e2700 1 --2- 172.21.15.52:0/2431155280 >> [v2:172.21.15.52:6800/3513075805,v1:172.21.15.52:6801/3513075805] conn(0x7f33a40b2880 0x7f33a40b4d90 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7f33ac006010 tx=0x7f33ac005960 comp rx=0 tx=0).stop 2024-05-27T07:22:10.983 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:22:10.992+0000 7f33c21e2700 1 -- 172.21.15.52:0/2431155280 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f33bc1092f0 msgr2=0x7f33bc19f7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:22:10.983 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:22:10.992+0000 7f33c21e2700 1 --2- 172.21.15.52:0/2431155280 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f33bc1092f0 0x7f33bc19f7a0 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7f33b4007ee0 tx=0x7f33b400d3b0 comp rx=0 tx=0).stop 2024-05-27T07:22:10.983 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:22:10.992+0000 7f33c21e2700 1 -- 172.21.15.52:0/2431155280 shutdown_connections 2024-05-27T07:22:10.983 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:22:10.992+0000 7f33c21e2700 1 --2- 172.21.15.52:0/2431155280 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f33bc1092f0 0x7f33bc19f7a0 unknown :-1 s=CLOSED pgs=198 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:22:10.983 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:22:10.992+0000 7f33c21e2700 1 --2- 172.21.15.52:0/2431155280 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f33bc102780 0x7f33bc19af70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:22:10.983 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:22:10.992+0000 7f33c21e2700 1 --2- 172.21.15.52:0/2431155280 >> [v2:172.21.15.52:6800/3513075805,v1:172.21.15.52:6801/3513075805] conn(0x7f33a40b2880 0x7f33a40b4d90 unknown :-1 s=CLOSED pgs=149 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:22:10.983 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:22:10.992+0000 7f33c21e2700 1 --2- 172.21.15.52:0/2431155280 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f33bc1030d0 0x7f33bc19b510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:22:10.983 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:22:10.992+0000 7f33c21e2700 1 -- 172.21.15.52:0/2431155280 >> 172.21.15.52:0/2431155280 conn(0x7f33bc0fe4d0 msgr2=0x7f33bc107090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:22:10.983 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:22:10.992+0000 7f33c21e2700 1 -- 172.21.15.52:0/2431155280 shutdown_connections 2024-05-27T07:22:10.983 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:22:10.992+0000 7f33c21e2700 1 -- 172.21.15.52:0/2431155280 wait complete. 2024-05-27T07:22:11.102 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:22:10 smithi173 ceph-mon[37000]: pgmap v306: 129 pgs: 129 active+clean; 579 KiB data, 356 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-27T07:22:11.190 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:22:10 smithi105 ceph-mon[35977]: pgmap v306: 129 pgs: 129 active+clean; 579 KiB data, 356 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-27T07:22:11.769 DEBUG:teuthology.orchestra.run.smithi052:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:38f9d900c83c63799fbdbe61acc9a11b0d3554a6 shell --fsid 73df2636-1bf8-11ef-bc9b-c7b262605968 -- ceph osd dump --format=json 2024-05-27T07:22:12.003 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:22:11 smithi173 ceph-mon[37000]: from='client.? 172.21.15.52:0/2431155280' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-05-27T07:22:12.025 INFO:teuthology.orchestra.run.smithi052.stderr:Inferring config /var/lib/ceph/73df2636-1bf8-11ef-bc9b-c7b262605968/mon.a/config 2024-05-27T07:22:12.147 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:22:11 smithi052 ceph-mon[31955]: from='client.? 172.21.15.52:0/2431155280' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-05-27T07:22:12.190 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:22:11 smithi105 ceph-mon[35977]: from='client.? 172.21.15.52:0/2431155280' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-05-27T07:22:13.102 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:22:12 smithi173 ceph-mon[37000]: pgmap v307: 129 pgs: 129 active+clean; 579 KiB data, 356 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-27T07:22:13.190 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:22:12 smithi105 ceph-mon[35977]: pgmap v307: 129 pgs: 129 active+clean; 579 KiB data, 356 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-27T07:22:13.229 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:22:12 smithi052 ceph-mon[31955]: pgmap v307: 129 pgs: 129 active+clean; 579 KiB data, 356 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-27T07:22:13.790 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:22:13.800+0000 7f1b099ad700 1 -- 172.21.15.52:0/2200880824 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f1b04109840 msgr2=0x7f1b041111a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:22:13.791 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:22:13.800+0000 7f1b099ad700 1 --2- 172.21.15.52:0/2200880824 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f1b04109840 0x7f1b041111a0 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7f1afc00b0d0 tx=0x7f1afc00b3e0 comp rx=0 tx=0).stop 2024-05-27T07:22:13.791 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:22:13.800+0000 7f1b099ad700 1 -- 172.21.15.52:0/2200880824 shutdown_connections 2024-05-27T07:22:13.791 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:22:13.800+0000 7f1b099ad700 1 --2- 172.21.15.52:0/2200880824 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f1b04109840 0x7f1b041111a0 unknown :-1 s=CLOSED pgs=199 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:22:13.791 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:22:13.800+0000 7f1b099ad700 1 --2- 172.21.15.52:0/2200880824 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f1b041045c0 0x7f1b041049d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:22:13.791 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:22:13.800+0000 7f1b099ad700 1 --2- 172.21.15.52:0/2200880824 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f1b04108ef0 0x7f1b04109300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:22:13.791 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:22:13.800+0000 7f1b099ad700 1 -- 172.21.15.52:0/2200880824 >> 172.21.15.52:0/2200880824 conn(0x7f1b040ffe90 msgr2=0x7f1b041022e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:22:13.791 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:22:13.800+0000 7f1b099ad700 1 -- 172.21.15.52:0/2200880824 shutdown_connections 2024-05-27T07:22:13.791 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:22:13.800+0000 7f1b099ad700 1 -- 172.21.15.52:0/2200880824 wait complete. 2024-05-27T07:22:13.791 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:22:13.801+0000 7f1b099ad700 1 Processor -- start 2024-05-27T07:22:13.793 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:22:13.801+0000 7f1b099ad700 1 -- start start 2024-05-27T07:22:13.793 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:22:13.801+0000 7f1b099ad700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f1b041045c0 0x7f1b0419ae10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:22:13.793 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:22:13.801+0000 7f1b099ad700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f1b04108ef0 0x7f1b0419b3b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:22:13.793 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:22:13.801+0000 7f1b099ad700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f1b04109840 0x7f1b0419f640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:22:13.793 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:22:13.801+0000 7f1b099ad700 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x7f1b04114500 con 0x7f1b041045c0 2024-05-27T07:22:13.793 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:22:13.801+0000 7f1b099ad700 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f1b04114380 con 0x7f1b04108ef0 2024-05-27T07:22:13.793 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:22:13.801+0000 7f1b099ad700 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f1b04114680 con 0x7f1b04109840 2024-05-27T07:22:13.794 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:22:13.801+0000 7f1b02ffd700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f1b041045c0 0x7f1b0419ae10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:22:13.794 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:22:13.801+0000 7f1b02ffd700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f1b041045c0 0x7f1b0419ae10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.52:37600/0 (socket says 172.21.15.52:37600) 2024-05-27T07:22:13.794 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:22:13.801+0000 7f1b02ffd700 1 -- 172.21.15.52:0/3907844014 learned_addr learned my addr 172.21.15.52:0/3907844014 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-05-27T07:22:13.794 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:22:13.802+0000 7f1b027fc700 1 --2- 172.21.15.52:0/3907844014 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f1b04108ef0 0x7f1b0419b3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:22:13.794 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:22:13.802+0000 7f1b037fe700 1 --2- 172.21.15.52:0/3907844014 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f1b04109840 0x7f1b0419f640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:22:13.794 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:22:13.802+0000 7f1b02ffd700 1 -- 172.21.15.52:0/3907844014 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f1b04109840 msgr2=0x7f1b0419f640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:22:13.794 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:22:13.802+0000 7f1b02ffd700 1 --2- 172.21.15.52:0/3907844014 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f1b04109840 0x7f1b0419f640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:22:13.794 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:22:13.802+0000 7f1b02ffd700 1 -- 172.21.15.52:0/3907844014 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f1b04108ef0 msgr2=0x7f1b0419b3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:22:13.794 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:22:13.802+0000 7f1b02ffd700 1 --2- 172.21.15.52:0/3907844014 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f1b04108ef0 0x7f1b0419b3b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:22:13.794 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:22:13.802+0000 7f1b02ffd700 1 -- 172.21.15.52:0/3907844014 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1b0419fce0 con 0x7f1b041045c0 2024-05-27T07:22:13.794 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:22:13.802+0000 7f1b02ffd700 1 --2- 172.21.15.52:0/3907844014 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f1b041045c0 0x7f1b0419ae10 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7f1af400e9c0 tx=0x7f1af400ecd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:22:13.795 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:22:13.802+0000 7f1b089ab700 1 -- 172.21.15.52:0/3907844014 <== mon.0 v2:172.21.15.52:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1af400cbd0 con 0x7f1b041045c0 2024-05-27T07:22:13.795 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:22:13.802+0000 7f1b089ab700 1 -- 172.21.15.52:0/3907844014 <== mon.0 v2:172.21.15.52:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f1af400cd70 con 0x7f1b041045c0 2024-05-27T07:22:13.795 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:22:13.802+0000 7f1b089ab700 1 -- 172.21.15.52:0/3907844014 <== mon.0 v2:172.21.15.52:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1af4005610 con 0x7f1b041045c0 2024-05-27T07:22:13.795 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:22:13.802+0000 7f1b099ad700 1 -- 172.21.15.52:0/3907844014 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1b0419ffd0 con 0x7f1b041045c0 2024-05-27T07:22:13.795 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:22:13.802+0000 7f1b099ad700 1 -- 172.21.15.52:0/3907844014 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1b041a78f0 con 0x7f1b041045c0 2024-05-27T07:22:13.795 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:22:13.803+0000 7f1b099ad700 1 -- 172.21.15.52:0/3907844014 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1b040512b0 con 0x7f1b041045c0 2024-05-27T07:22:13.796 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:22:13.805+0000 7f1b089ab700 1 -- 172.21.15.52:0/3907844014 <== mon.0 v2:172.21.15.52:3300/0 4 ==== mgrmap(e 16) ==== 149652+0+0 (secure 0 0 0) 0x7f1af4004750 con 0x7f1b041045c0 2024-05-27T07:22:13.796 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:22:13.805+0000 7f1b089ab700 1 --2- 172.21.15.52:0/3907844014 >> [v2:172.21.15.52:6800/3513075805,v1:172.21.15.52:6801/3513075805] conn(0x7f1aec0b2900 0x7f1aec0b4e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:22:13.796 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:22:13.806+0000 7f1b027fc700 1 --2- 172.21.15.52:0/3907844014 >> [v2:172.21.15.52:6800/3513075805,v1:172.21.15.52:6801/3513075805] conn(0x7f1aec0b2900 0x7f1aec0b4e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:22:13.796 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:22:13.806+0000 7f1b089ab700 1 -- 172.21.15.52:0/3907844014 <== mon.0 v2:172.21.15.52:3300/0 5 ==== osd_map(86..86 src has 1..86) ==== 8802+0+0 (secure 0 0 0) 0x7f1af40e28e0 con 0x7f1b041045c0 2024-05-27T07:22:13.796 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:22:13.806+0000 7f1b027fc700 1 --2- 172.21.15.52:0/3907844014 >> [v2:172.21.15.52:6800/3513075805,v1:172.21.15.52:6801/3513075805] conn(0x7f1aec0b2900 0x7f1aec0b4e10 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7f1b04103430 tx=0x7f1af8009190 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:22:13.799 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:22:13.809+0000 7f1b089ab700 1 -- 172.21.15.52:0/3907844014 <== mon.0 v2:172.21.15.52:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191963 (secure 0 0 0) 0x7f1af4054df0 con 0x7f1b041045c0 2024-05-27T07:22:14.014 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:22:14.024+0000 7f1b099ad700 1 -- 172.21.15.52:0/3907844014 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f1b0419c1a0 con 0x7f1b041045c0 2024-05-27T07:22:14.016 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:22:14.026+0000 7f1b089ab700 1 -- 172.21.15.52:0/3907844014 <== mon.0 v2:172.21.15.52:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v86) ==== 74+0+24126 (secure 0 0 0) 0x7f1b0419c1a0 con 0x7f1b041045c0 2024-05-27T07:22:14.016 INFO:teuthology.orchestra.run.smithi052.stdout: 2024-05-27T07:22:14.016 INFO:teuthology.orchestra.run.smithi052.stdout:{"epoch":86,"fsid":"73df2636-1bf8-11ef-bc9b-c7b262605968","created":"2024-05-27T07:13:34.738032+0000","modified":"2024-05-27T07:22:04.612333+0000","last_up_change":"2024-05-27T07:20:45.777597+0000","last_in_change":"2024-05-27T07:20:25.314823+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-05-27T07:16:57.644034+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-05-27T07:21:43.917748+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"79","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-05-27T07:21:47.341876+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"79","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-05-27T07:21:57.707611+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"86","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"347b8a89-9dbe-4848-83b0-fd0e9374f33d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":82,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.52:6802","nonce":1626508899},{"type":"v1","addr":"172.21.15.52:6803","nonce":1626508899}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.52:6804","nonce":1626508899},{"type":"v1","addr":"172.21.15.52:6805","nonce":1626508899}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.52:6808","nonce":1626508899},{"type":"v1","addr":"172.21.15.52:6809","nonce":1626508899}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.52:6806","nonce":1626508899},{"type":"v1","addr":"172.21.15.52:6807","nonce":1626508899}]},"public_addr":"172.21.15.52:6803/1626508899","cluster_addr":"172.21.15.52:6805/1626508899","heartbeat_back_addr":"172.21.15.52:6809/1626508899","heartbeat_front_addr":"172.21.15.52:6807/1626508899","state":["exists","up"]},{"osd":1,"uuid":"7a3623d4-e087-40cb-9336-d4a5c52203d4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":82,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6800","nonce":2900093195},{"type":"v1","addr":"172.21.15.105:6801","nonce":2900093195}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6802","nonce":2900093195},{"type":"v1","addr":"172.21.15.105:6803","nonce":2900093195}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6806","nonce":2900093195},{"type":"v1","addr":"172.21.15.105:6807","nonce":2900093195}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6804","nonce":2900093195},{"type":"v1","addr":"172.21.15.105:6805","nonce":2900093195}]},"public_addr":"172.21.15.105:6801/2900093195","cluster_addr":"172.21.15.105:6803/2900093195","heartbeat_back_addr":"172.21.15.105:6807/2900093195","heartbeat_front_addr":"172.21.15.105:6805/2900093195","state":["exists","up"]},{"osd":2,"uuid":"41a9df5b-882a-47da-bb26-7f4d26ccf6c1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":82,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6800","nonce":2251189314},{"type":"v1","addr":"172.21.15.173:6801","nonce":2251189314}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6802","nonce":2251189314},{"type":"v1","addr":"172.21.15.173:6803","nonce":2251189314}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6806","nonce":2251189314},{"type":"v1","addr":"172.21.15.173:6807","nonce":2251189314}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6804","nonce":2251189314},{"type":"v1","addr":"172.21.15.173:6805","nonce":2251189314}]},"public_addr":"172.21.15.173:6801/2251189314","cluster_addr":"172.21.15.173:6803/2251189314","heartbeat_back_addr":"172.21.15.173:6807/2251189314","heartbeat_front_addr":"172.21.15.173:6805/2251189314","state":["exists","up"]},{"osd":3,"uuid":"f1245439-42a5-46c5-a998-e0a78f9df3fb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":26,"up_thru":82,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.52:6810","nonce":57920794},{"type":"v1","addr":"172.21.15.52:6811","nonce":57920794}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.52:6812","nonce":57920794},{"type":"v1","addr":"172.21.15.52:6813","nonce":57920794}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.52:6816","nonce":57920794},{"type":"v1","addr":"172.21.15.52:6817","nonce":57920794}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.52:6814","nonce":57920794},{"type":"v1","addr":"172.21.15.52:6815","nonce":57920794}]},"public_addr":"172.21.15.52:6811/57920794","cluster_addr":"172.21.15.52:6813/57920794","heartbeat_back_addr":"172.21.15.52:6817/57920794","heartbeat_front_addr":"172.21.15.52:6815/57920794","state":["exists","up"]},{"osd":4,"uuid":"2592d1cc-1726-4359-83d8-a2e6314b54ec","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":32,"up_thru":82,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6808","nonce":3441134818},{"type":"v1","addr":"172.21.15.105:6809","nonce":3441134818}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6810","nonce":3441134818},{"type":"v1","addr":"172.21.15.105:6811","nonce":3441134818}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6814","nonce":3441134818},{"type":"v1","addr":"172.21.15.105:6815","nonce":3441134818}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6812","nonce":3441134818},{"type":"v1","addr":"172.21.15.105:6813","nonce":3441134818}]},"public_addr":"172.21.15.105:6809/3441134818","cluster_addr":"172.21.15.105:6811/3441134818","heartbeat_back_addr":"172.21.15.105:6815/3441134818","heartbeat_front_addr":"172.21.15.105:6813/3441134818","state":["exists","up"]},{"osd":5,"uuid":"27e98ee2-4a92-4972-b2a3-959d0ed90981","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":37,"up_thru":82,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6808","nonce":2989239729},{"type":"v1","addr":"172.21.15.173:6809","nonce":2989239729}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6810","nonce":2989239729},{"type":"v1","addr":"172.21.15.173:6811","nonce":2989239729}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6814","nonce":2989239729},{"type":"v1","addr":"172.21.15.173:6815","nonce":2989239729}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6812","nonce":2989239729},{"type":"v1","addr":"172.21.15.173:6813","nonce":2989239729}]},"public_addr":"172.21.15.173:6809/2989239729","cluster_addr":"172.21.15.173:6811/2989239729","heartbeat_back_addr":"172.21.15.173:6815/2989239729","heartbeat_front_addr":"172.21.15.173:6813/2989239729","state":["exists","up"]},{"osd":6,"uuid":"2b0ec0cb-f438-4b2c-8642-0778ed374e9a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":43,"up_thru":82,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.52:6818","nonce":4244423065},{"type":"v1","addr":"172.21.15.52:6819","nonce":4244423065}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.52:6820","nonce":4244423065},{"type":"v1","addr":"172.21.15.52:6821","nonce":4244423065}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.52:6824","nonce":4244423065},{"type":"v1","addr":"172.21.15.52:6825","nonce":4244423065}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.52:6822","nonce":4244423065},{"type":"v1","addr":"172.21.15.52:6823","nonce":4244423065}]},"public_addr":"172.21.15.52:6819/4244423065","cluster_addr":"172.21.15.52:6821/4244423065","heartbeat_back_addr":"172.21.15.52:6825/4244423065","heartbeat_front_addr":"172.21.15.52:6823/4244423065","state":["exists","up"]},{"osd":7,"uuid":"cb2a7a6b-8015-43ea-865a-be144e45c51a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":48,"up_thru":82,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6816","nonce":2063611552},{"type":"v1","addr":"172.21.15.105:6817","nonce":2063611552}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6818","nonce":2063611552},{"type":"v1","addr":"172.21.15.105:6819","nonce":2063611552}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6822","nonce":2063611552},{"type":"v1","addr":"172.21.15.105:6823","nonce":2063611552}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6820","nonce":2063611552},{"type":"v1","addr":"172.21.15.105:6821","nonce":2063611552}]},"public_addr":"172.21.15.105:6817/2063611552","cluster_addr":"172.21.15.105:6819/2063611552","heartbeat_back_addr":"172.21.15.105:6823/2063611552","heartbeat_front_addr":"172.21.15.105:6821/2063611552","state":["exists","up"]},{"osd":8,"uuid":"c2fc565d-a0b9-4396-a209-bd884cddcfe3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":54,"up_thru":82,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6816","nonce":4131399852},{"type":"v1","addr":"172.21.15.173:6817","nonce":4131399852}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6818","nonce":4131399852},{"type":"v1","addr":"172.21.15.173:6819","nonce":4131399852}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6822","nonce":4131399852},{"type":"v1","addr":"172.21.15.173:6823","nonce":4131399852}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6820","nonce":4131399852},{"type":"v1","addr":"172.21.15.173:6821","nonce":4131399852}]},"public_addr":"172.21.15.173:6817/4131399852","cluster_addr":"172.21.15.173:6819/4131399852","heartbeat_back_addr":"172.21.15.173:6823/4131399852","heartbeat_front_addr":"172.21.15.173:6821/4131399852","state":["exists","up"]},{"osd":9,"uuid":"78fa7f17-1dad-4262-be40-31fa82e75520","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":60,"up_thru":82,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.52:6826","nonce":479119778},{"type":"v1","addr":"172.21.15.52:6827","nonce":479119778}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.52:6828","nonce":479119778},{"type":"v1","addr":"172.21.15.52:6829","nonce":479119778}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.52:6832","nonce":479119778},{"type":"v1","addr":"172.21.15.52:6833","nonce":479119778}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.52:6830","nonce":479119778},{"type":"v1","addr":"172.21.15.52:6831","nonce":479119778}]},"public_addr":"172.21.15.52:6827/479119778","cluster_addr":"172.21.15.52:6829/479119778","heartbeat_back_addr":"172.21.15.52:6833/479119778","heartbeat_front_addr":"172.21.15.52:6831/479119778","state":["exists","up"]},{"osd":10,"uuid":"e6f8e753-e15c-45b5-8ee9-d50f384559f9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":65,"up_thru":76,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6824","nonce":4032386628},{"type":"v1","addr":"172.21.15.105:6825","nonce":4032386628}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6826","nonce":4032386628},{"type":"v1","addr":"172.21.15.105:6827","nonce":4032386628}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6830","nonce":4032386628},{"type":"v1","addr":"172.21.15.105:6831","nonce":4032386628}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6828","nonce":4032386628},{"type":"v1","addr":"172.21.15.105:6829","nonce":4032386628}]},"public_addr":"172.21.15.105:6825/4032386628","cluster_addr":"172.21.15.105:6827/4032386628","heartbeat_back_addr":"172.21.15.105:6831/4032386628","heartbeat_front_addr":"172.21.15.105:6829/4032386628","state":["exists","up"]},{"osd":11,"uuid":"164572bf-67ed-4e8e-b32d-d7a2e8da607b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":71,"up_thru":82,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6824","nonce":1816308401},{"type":"v1","addr":"172.21.15.173:6825","nonce":1816308401}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6826","nonce":1816308401},{"type":"v1","addr":"172.21.15.173:6827","nonce":1816308401}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6830","nonce":1816308401},{"type":"v1","addr":"172.21.15.173:6831","nonce":1816308401}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6828","nonce":1816308401},{"type":"v1","addr":"172.21.15.173:6829","nonce":1816308401}]},"public_addr":"172.21.15.173:6825/1816308401","cluster_addr":"172.21.15.173:6827/1816308401","heartbeat_back_addr":"172.21.15.173:6831/1816308401","heartbeat_front_addr":"172.21.15.173:6829/1816308401","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-27T07:16:06.279220+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-27T07:16:29.668167+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-27T07:16:53.903808+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-27T07:17:18.787372+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-27T07:17:43.480784+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-27T07:18:08.816030+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-27T07:18:34.076148+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-27T07:19:00.283468+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-27T07:19:25.403023+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-27T07:19:52.139173+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-27T07:20:17.530304+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-27T07:20:43.665632+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.52:6801/1631683163":"2024-05-28T07:14:29.544633+0000","172.21.15.52:0/2776144659":"2024-05-28T07:14:29.544633+0000","172.21.15.52:6800/1631683163":"2024-05-28T07:14:29.544633+0000","172.21.15.52:0/3675529097":"2024-05-28T07:14:01.960550+0000","172.21.15.52:0/727808107":"2024-05-28T07:14:01.960550+0000","172.21.15.52:0/3839436833":"2024-05-28T07:14:29.544633+0000","172.21.15.52:6800/587662512":"2024-05-28T07:14:01.960550+0000","172.21.15.52:6801/587662512":"2024-05-28T07:14:01.960550+0000","172.21.15.52:0/853867465":"2024-05-28T07:14:01.960550+0000","172.21.15.52:0/3480918279":"2024-05-28T07:14:29.544633+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-05-27T07:22:14.018 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:22:14.028+0000 7f1b099ad700 1 -- 172.21.15.52:0/3907844014 >> [v2:172.21.15.52:6800/3513075805,v1:172.21.15.52:6801/3513075805] conn(0x7f1aec0b2900 msgr2=0x7f1aec0b4e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:22:14.019 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:22:14.028+0000 7f1b099ad700 1 --2- 172.21.15.52:0/3907844014 >> [v2:172.21.15.52:6800/3513075805,v1:172.21.15.52:6801/3513075805] conn(0x7f1aec0b2900 0x7f1aec0b4e10 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7f1b04103430 tx=0x7f1af8009190 comp rx=0 tx=0).stop 2024-05-27T07:22:14.019 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:22:14.028+0000 7f1b099ad700 1 -- 172.21.15.52:0/3907844014 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f1b041045c0 msgr2=0x7f1b0419ae10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:22:14.019 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:22:14.028+0000 7f1b099ad700 1 --2- 172.21.15.52:0/3907844014 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f1b041045c0 0x7f1b0419ae10 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7f1af400e9c0 tx=0x7f1af400ecd0 comp rx=0 tx=0).stop 2024-05-27T07:22:14.019 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:22:14.028+0000 7f1b099ad700 1 -- 172.21.15.52:0/3907844014 shutdown_connections 2024-05-27T07:22:14.019 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:22:14.028+0000 7f1b099ad700 1 --2- 172.21.15.52:0/3907844014 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f1b04109840 0x7f1b0419f640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:22:14.019 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:22:14.028+0000 7f1b099ad700 1 --2- 172.21.15.52:0/3907844014 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f1b041045c0 0x7f1b0419ae10 unknown :-1 s=CLOSED pgs=200 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:22:14.019 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:22:14.028+0000 7f1b099ad700 1 --2- 172.21.15.52:0/3907844014 >> [v2:172.21.15.52:6800/3513075805,v1:172.21.15.52:6801/3513075805] conn(0x7f1aec0b2900 0x7f1aec0b4e10 unknown :-1 s=CLOSED pgs=150 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:22:14.019 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:22:14.028+0000 7f1b099ad700 1 --2- 172.21.15.52:0/3907844014 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f1b04108ef0 0x7f1b0419b3b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:22:14.019 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:22:14.028+0000 7f1b099ad700 1 -- 172.21.15.52:0/3907844014 >> 172.21.15.52:0/3907844014 conn(0x7f1b040ffe90 msgr2=0x7f1b04108660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:22:14.019 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:22:14.028+0000 7f1b099ad700 1 -- 172.21.15.52:0/3907844014 shutdown_connections 2024-05-27T07:22:14.019 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:22:14.028+0000 7f1b099ad700 1 -- 172.21.15.52:0/3907844014 wait complete. 2024-05-27T07:22:14.779 DEBUG:teuthology.orchestra.run.smithi052:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:38f9d900c83c63799fbdbe61acc9a11b0d3554a6 shell --fsid 73df2636-1bf8-11ef-bc9b-c7b262605968 -- ceph fs dump --format=json 2024-05-27T07:22:14.979 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:22:14 smithi052 ceph-mon[31955]: pgmap v308: 129 pgs: 129 active+clean; 579 KiB data, 356 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-27T07:22:14.979 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:22:14 smithi052 ceph-mon[31955]: from='client.? 172.21.15.52:0/3907844014' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-05-27T07:22:15.036 INFO:teuthology.orchestra.run.smithi052.stderr:Inferring config /var/lib/ceph/73df2636-1bf8-11ef-bc9b-c7b262605968/mon.a/config 2024-05-27T07:22:15.102 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:22:14 smithi173 ceph-mon[37000]: pgmap v308: 129 pgs: 129 active+clean; 579 KiB data, 356 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-27T07:22:15.102 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:22:14 smithi173 ceph-mon[37000]: from='client.? 172.21.15.52:0/3907844014' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-05-27T07:22:15.190 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:22:14 smithi105 ceph-mon[35977]: pgmap v308: 129 pgs: 129 active+clean; 579 KiB data, 356 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-27T07:22:15.190 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:22:14 smithi105 ceph-mon[35977]: from='client.? 172.21.15.52:0/3907844014' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-05-27T07:22:16.687 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:22:16.693+0000 7f99da17a700 1 -- 172.21.15.52:0/1796415681 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f99d4108730 msgr2=0x7f99d4110090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:22:16.688 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:22:16.693+0000 7f99da17a700 1 --2- 172.21.15.52:0/1796415681 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f99d4108730 0x7f99d4110090 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7f99cc00b0d0 tx=0x7f99cc00b3e0 comp rx=0 tx=0).stop 2024-05-27T07:22:16.688 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:22:16.693+0000 7f99da17a700 1 -- 172.21.15.52:0/1796415681 shutdown_connections 2024-05-27T07:22:16.688 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:22:16.693+0000 7f99da17a700 1 --2- 172.21.15.52:0/1796415681 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f99d4108730 0x7f99d4110090 unknown :-1 s=CLOSED pgs=201 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:22:16.688 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:22:16.693+0000 7f99da17a700 1 --2- 172.21.15.52:0/1796415681 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f99d41034b0 0x7f99d41038c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:22:16.688 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:22:16.693+0000 7f99da17a700 1 --2- 172.21.15.52:0/1796415681 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f99d4107de0 0x7f99d41081f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:22:16.688 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:22:16.693+0000 7f99da17a700 1 -- 172.21.15.52:0/1796415681 >> 172.21.15.52:0/1796415681 conn(0x7f99d40fedb0 msgr2=0x7f99d41011d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:22:16.688 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:22:16.693+0000 7f99da17a700 1 -- 172.21.15.52:0/1796415681 shutdown_connections 2024-05-27T07:22:16.688 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:22:16.693+0000 7f99da17a700 1 -- 172.21.15.52:0/1796415681 wait complete. 2024-05-27T07:22:16.688 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:22:16.694+0000 7f99da17a700 1 Processor -- start 2024-05-27T07:22:16.688 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:22:16.694+0000 7f99da17a700 1 -- start start 2024-05-27T07:22:16.688 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:22:16.694+0000 7f99da17a700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f99d41034b0 0x7f99d4199d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:22:16.689 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:22:16.694+0000 7f99da17a700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f99d4107de0 0x7f99d419a2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:22:16.689 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:22:16.694+0000 7f99da17a700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f99d4108730 0x7f99d419e580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:22:16.689 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:22:16.694+0000 7f99da17a700 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x7f99d41133e0 con 0x7f99d4107de0 2024-05-27T07:22:16.689 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:22:16.694+0000 7f99da17a700 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f99d4113260 con 0x7f99d4108730 2024-05-27T07:22:16.689 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:22:16.694+0000 7f99da17a700 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f99d4113560 con 0x7f99d41034b0 2024-05-27T07:22:16.689 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:22:16.695+0000 7f99d2ffd700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f99d4107de0 0x7f99d419a2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:22:16.689 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:22:16.695+0000 7f99d2ffd700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f99d4107de0 0x7f99d419a2f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.52:37626/0 (socket says 172.21.15.52:37626) 2024-05-27T07:22:16.689 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:22:16.695+0000 7f99d2ffd700 1 -- 172.21.15.52:0/1942577413 learned_addr learned my addr 172.21.15.52:0/1942577413 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-05-27T07:22:16.689 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:22:16.695+0000 7f99d3fff700 1 --2- 172.21.15.52:0/1942577413 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f99d4108730 0x7f99d419e580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:22:16.689 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:22:16.695+0000 7f99d37fe700 1 --2- 172.21.15.52:0/1942577413 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f99d41034b0 0x7f99d4199d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:22:16.689 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:22:16.695+0000 7f99d3fff700 1 -- 172.21.15.52:0/1942577413 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f99d41034b0 msgr2=0x7f99d4199d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:22:16.689 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:22:16.695+0000 7f99d3fff700 1 --2- 172.21.15.52:0/1942577413 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f99d41034b0 0x7f99d4199d50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:22:16.690 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:22:16.695+0000 7f99d3fff700 1 -- 172.21.15.52:0/1942577413 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f99d4107de0 msgr2=0x7f99d419a2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:22:16.690 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:22:16.695+0000 7f99d3fff700 1 --2- 172.21.15.52:0/1942577413 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f99d4107de0 0x7f99d419a2f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:22:16.690 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:22:16.695+0000 7f99d3fff700 1 -- 172.21.15.52:0/1942577413 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f99d419eb70 con 0x7f99d4108730 2024-05-27T07:22:16.690 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:22:16.695+0000 7f99d3fff700 1 --2- 172.21.15.52:0/1942577413 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f99d4108730 0x7f99d419e580 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f99cc009fd0 tx=0x7f99cc007a20 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:22:16.690 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:22:16.696+0000 7f99d0ff9700 1 -- 172.21.15.52:0/1942577413 <== mon.2 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f99cc021070 con 0x7f99d4108730 2024-05-27T07:22:16.690 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:22:16.696+0000 7f99da17a700 1 -- 172.21.15.52:0/1942577413 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f99d419ee00 con 0x7f99d4108730 2024-05-27T07:22:16.690 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:22:16.696+0000 7f99da17a700 1 -- 172.21.15.52:0/1942577413 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f99d419f390 con 0x7f99d4108730 2024-05-27T07:22:16.690 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:22:16.696+0000 7f99d0ff9700 1 -- 172.21.15.52:0/1942577413 <== mon.2 v2:172.21.15.105:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f99cc0075c0 con 0x7f99d4108730 2024-05-27T07:22:16.690 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:22:16.696+0000 7f99d0ff9700 1 -- 172.21.15.52:0/1942577413 <== mon.2 v2:172.21.15.105:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f99cc01b420 con 0x7f99d4108730 2024-05-27T07:22:16.690 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:22:16.697+0000 7f99da17a700 1 -- 172.21.15.52:0/1942577413 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f99c0005330 con 0x7f99d4108730 2024-05-27T07:22:16.691 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:22:16.699+0000 7f99d0ff9700 1 -- 172.21.15.52:0/1942577413 <== mon.2 v2:172.21.15.105:3300/0 4 ==== mgrmap(e 16) ==== 149652+0+0 (secure 0 0 0) 0x7f99cc024070 con 0x7f99d4108730 2024-05-27T07:22:16.691 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:22:16.699+0000 7f99d0ff9700 1 --2- 172.21.15.52:0/1942577413 >> [v2:172.21.15.52:6800/3513075805,v1:172.21.15.52:6801/3513075805] conn(0x7f99bc0b2880 0x7f99bc0b4d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:22:16.691 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:22:16.700+0000 7f99d0ff9700 1 -- 172.21.15.52:0/1942577413 <== mon.2 v2:172.21.15.105:3300/0 5 ==== osd_map(86..86 src has 1..86) ==== 8802+0+0 (secure 0 0 0) 0x7f99cc0e33f0 con 0x7f99d4108730 2024-05-27T07:22:16.691 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:22:16.700+0000 7f99d37fe700 1 --2- 172.21.15.52:0/1942577413 >> [v2:172.21.15.52:6800/3513075805,v1:172.21.15.52:6801/3513075805] conn(0x7f99bc0b2880 0x7f99bc0b4d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:22:16.691 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:22:16.700+0000 7f99d37fe700 1 --2- 172.21.15.52:0/1942577413 >> [v2:172.21.15.52:6800/3513075805,v1:172.21.15.52:6801/3513075805] conn(0x7f99bc0b2880 0x7f99bc0b4d90 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7f99c4009a10 tx=0x7f99c4009190 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:22:16.693 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:22:16.703+0000 7f99d0ff9700 1 -- 172.21.15.52:0/1942577413 <== mon.2 v2:172.21.15.105:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191963 (secure 0 0 0) 0x7f99cc02b030 con 0x7f99d4108730 2024-05-27T07:22:16.956 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:22:16.966+0000 7f99da17a700 1 -- 172.21.15.52:0/1942577413 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f99c0005620 con 0x7f99d4108730 2024-05-27T07:22:16.957 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:22:16.967+0000 7f99d0ff9700 1 -- 172.21.15.52:0/1942577413 <== mon.2 v2:172.21.15.105:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 17 v17) ==== 94+0+11803 (secure 0 0 0) 0x7f99cc007e20 con 0x7f99d4108730 2024-05-27T07:22:16.958 INFO:teuthology.orchestra.run.smithi052.stdout: 2024-05-27T07:22:16.958 INFO:teuthology.orchestra.run.smithi052.stdout:{"epoch":17,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14556,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.52:6835/2584733778","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.52:6834","nonce":2584733778},{"type":"v1","addr":"172.21.15.52:6835","nonce":2584733778}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14568,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.52:6837/675445303","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.52:6836","nonce":675445303},{"type":"v1","addr":"172.21.15.52:6837","nonce":675445303}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14619,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.52:6839/4258549559","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.52:6838","nonce":4258549559},{"type":"v1","addr":"172.21.15.52:6839","nonce":4258549559}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14631,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.52:6841/2878618188","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.52:6840","nonce":2878618188},{"type":"v1","addr":"172.21.15.52:6841","nonce":2878618188}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24433,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.173:6833/3013589220","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6832","nonce":3013589220},{"type":"v1","addr":"172.21.15.173:6833","nonce":3013589220}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24452,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.105:6833/2800678412","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6832","nonce":2800678412},{"type":"v1","addr":"172.21.15.105:6833","nonce":2800678412}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24460,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.173:6835/1619486437","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6834","nonce":1619486437},{"type":"v1","addr":"172.21.15.173:6835","nonce":1619486437}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":7},{"gid":24466,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.173:6837/1396363827","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6836","nonce":1396363827},{"type":"v1","addr":"172.21.15.173:6837","nonce":1396363827}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":8},{"gid":24473,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.105:6835/3298498384","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6834","nonce":3298498384},{"type":"v1","addr":"172.21.15.105:6835","nonce":3298498384}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6},{"gid":24475,"name":"i","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.173:6839/564778293","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6838","nonce":564778293},{"type":"v1","addr":"172.21.15.173:6839","nonce":564778293}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":10},{"gid":24497,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.105:6837/191393149","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6836","nonce":191393149},{"type":"v1","addr":"172.21.15.105:6837","nonce":191393149}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":9}],"filesystems":[{"mdsmap":{"epoch":17,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-05-27T07:21:50.521783+0000","modified":"2024-05-27T07:22:04.612487+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":24527},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24527":{"gid":24527,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":5,"addr":"172.21.15.105:6839/500348151","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6838","nonce":500348151},{"type":"v1","addr":"172.21.15.105:6839","nonce":500348151}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":1,"qdb_leader":24527,"qdb_cluster":[24527]},"id":1}]} 2024-05-27T07:22:16.960 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:22:16.970+0000 7f99da17a700 1 -- 172.21.15.52:0/1942577413 >> [v2:172.21.15.52:6800/3513075805,v1:172.21.15.52:6801/3513075805] conn(0x7f99bc0b2880 msgr2=0x7f99bc0b4d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:22:16.960 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:22:16.970+0000 7f99da17a700 1 --2- 172.21.15.52:0/1942577413 >> [v2:172.21.15.52:6800/3513075805,v1:172.21.15.52:6801/3513075805] conn(0x7f99bc0b2880 0x7f99bc0b4d90 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7f99c4009a10 tx=0x7f99c4009190 comp rx=0 tx=0).stop 2024-05-27T07:22:16.960 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:22:16.970+0000 7f99da17a700 1 -- 172.21.15.52:0/1942577413 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f99d4108730 msgr2=0x7f99d419e580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:22:16.960 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:22:16.970+0000 7f99da17a700 1 --2- 172.21.15.52:0/1942577413 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f99d4108730 0x7f99d419e580 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f99cc009fd0 tx=0x7f99cc007a20 comp rx=0 tx=0).stop 2024-05-27T07:22:16.960 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:22:16.970+0000 7f99da17a700 1 -- 172.21.15.52:0/1942577413 shutdown_connections 2024-05-27T07:22:16.960 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:22:16.970+0000 7f99da17a700 1 --2- 172.21.15.52:0/1942577413 >> [v2:172.21.15.52:6800/3513075805,v1:172.21.15.52:6801/3513075805] conn(0x7f99bc0b2880 0x7f99bc0b4d90 unknown :-1 s=CLOSED pgs=151 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:22:16.960 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:22:16.970+0000 7f99da17a700 1 --2- 172.21.15.52:0/1942577413 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f99d4108730 0x7f99d419e580 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:22:16.960 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:22:16.970+0000 7f99da17a700 1 --2- 172.21.15.52:0/1942577413 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f99d41034b0 0x7f99d4199d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:22:16.961 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:22:16.970+0000 7f99da17a700 1 --2- 172.21.15.52:0/1942577413 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f99d4107de0 0x7f99d419a2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:22:16.961 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:22:16.970+0000 7f99da17a700 1 -- 172.21.15.52:0/1942577413 >> 172.21.15.52:0/1942577413 conn(0x7f99d40fedb0 msgr2=0x7f99d4107580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:22:16.961 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:22:16.970+0000 7f99da17a700 1 -- 172.21.15.52:0/1942577413 shutdown_connections 2024-05-27T07:22:16.961 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:22:16.970+0000 7f99da17a700 1 -- 172.21.15.52:0/1942577413 wait complete. 2024-05-27T07:22:16.961 INFO:teuthology.orchestra.run.smithi052.stderr:dumped fsmap epoch 17 2024-05-27T07:22:17.102 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:22:16 smithi173 ceph-mon[37000]: pgmap v309: 129 pgs: 129 active+clean; 579 KiB data, 356 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-27T07:22:17.190 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:22:16 smithi105 ceph-mon[35977]: pgmap v309: 129 pgs: 129 active+clean; 579 KiB data, 356 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-27T07:22:17.229 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:22:16 smithi052 ceph-mon[31955]: pgmap v309: 129 pgs: 129 active+clean; 579 KiB data, 356 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-27T07:22:17.755 DEBUG:teuthology.orchestra.run.smithi052:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:38f9d900c83c63799fbdbe61acc9a11b0d3554a6 shell --fsid 73df2636-1bf8-11ef-bc9b-c7b262605968 -- ceph osd dump --format=json 2024-05-27T07:22:17.979 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:22:17 smithi052 ceph-mon[31955]: from='client.? 172.21.15.52:0/1942577413' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-27T07:22:18.014 INFO:teuthology.orchestra.run.smithi052.stderr:Inferring config /var/lib/ceph/73df2636-1bf8-11ef-bc9b-c7b262605968/mon.a/config 2024-05-27T07:22:18.102 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:22:17 smithi173 ceph-mon[37000]: from='client.? 172.21.15.52:0/1942577413' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-27T07:22:18.190 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:22:17 smithi105 ceph-mon[35977]: from='client.? 172.21.15.52:0/1942577413' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-27T07:22:19.102 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:22:18 smithi173 ceph-mon[37000]: pgmap v310: 129 pgs: 129 active+clean; 579 KiB data, 356 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-27T07:22:19.150 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:22:18 smithi052 ceph-mon[31955]: pgmap v310: 129 pgs: 129 active+clean; 579 KiB data, 356 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-27T07:22:19.190 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:22:18 smithi105 ceph-mon[35977]: pgmap v310: 129 pgs: 129 active+clean; 579 KiB data, 356 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-27T07:22:19.779 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:22:19.789+0000 7f347366a700 1 -- 172.21.15.52:0/1946526316 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f346c1034b0 msgr2=0x7f346c1038c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:22:19.779 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:22:19.789+0000 7f347366a700 1 --2- 172.21.15.52:0/1946526316 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f346c1034b0 0x7f346c1038c0 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7f34680099c0 tx=0x7f3468009cd0 comp rx=0 tx=0).stop 2024-05-27T07:22:19.779 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:22:19.789+0000 7f347366a700 1 -- 172.21.15.52:0/1946526316 shutdown_connections 2024-05-27T07:22:19.780 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:22:19.789+0000 7f347366a700 1 --2- 172.21.15.52:0/1946526316 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f346c108730 0x7f346c110090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:22:19.780 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:22:19.789+0000 7f347366a700 1 --2- 172.21.15.52:0/1946526316 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f346c1034b0 0x7f346c1038c0 unknown :-1 s=CLOSED pgs=202 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:22:19.780 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:22:19.789+0000 7f347366a700 1 --2- 172.21.15.52:0/1946526316 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f346c107de0 0x7f346c1081f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:22:19.780 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:22:19.789+0000 7f347366a700 1 -- 172.21.15.52:0/1946526316 >> 172.21.15.52:0/1946526316 conn(0x7f346c0fede0 msgr2=0x7f346c101200 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:22:19.780 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:22:19.789+0000 7f347366a700 1 -- 172.21.15.52:0/1946526316 shutdown_connections 2024-05-27T07:22:19.780 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:22:19.789+0000 7f347366a700 1 -- 172.21.15.52:0/1946526316 wait complete. 2024-05-27T07:22:19.780 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:22:19.789+0000 7f347366a700 1 Processor -- start 2024-05-27T07:22:19.780 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:22:19.789+0000 7f347366a700 1 -- start start 2024-05-27T07:22:19.782 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:22:19.790+0000 7f347366a700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f346c1034b0 0x7f346c19aec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:22:19.782 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:22:19.790+0000 7f347366a700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f346c107de0 0x7f346c19b460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:22:19.782 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:22:19.790+0000 7f347366a700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f346c108730 0x7f346c19f6f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:22:19.782 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:22:19.790+0000 7f347366a700 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x7f346c1144b0 con 0x7f346c107de0 2024-05-27T07:22:19.782 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:22:19.790+0000 7f347366a700 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f346c114330 con 0x7f346c1034b0 2024-05-27T07:22:19.782 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:22:19.790+0000 7f347366a700 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f346c114630 con 0x7f346c108730 2024-05-27T07:22:19.782 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:22:19.790+0000 7f3470c05700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f346c107de0 0x7f346c19b460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:22:19.783 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:22:19.790+0000 7f3470c05700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f346c107de0 0x7f346c19b460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.52:37648/0 (socket says 172.21.15.52:37648) 2024-05-27T07:22:19.783 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:22:19.790+0000 7f3470c05700 1 -- 172.21.15.52:0/3401220157 learned_addr learned my addr 172.21.15.52:0/3401220157 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-05-27T07:22:19.783 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:22:19.790+0000 7f3471406700 1 --2- 172.21.15.52:0/3401220157 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f346c1034b0 0x7f346c19aec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:22:19.783 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:22:19.790+0000 7f3471c07700 1 --2- 172.21.15.52:0/3401220157 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f346c108730 0x7f346c19f6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:22:19.783 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:22:19.790+0000 7f3470c05700 1 -- 172.21.15.52:0/3401220157 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f346c108730 msgr2=0x7f346c19f6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:22:19.783 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:22:19.790+0000 7f3470c05700 1 --2- 172.21.15.52:0/3401220157 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f346c108730 0x7f346c19f6f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:22:19.783 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:22:19.790+0000 7f3470c05700 1 -- 172.21.15.52:0/3401220157 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f346c1034b0 msgr2=0x7f346c19aec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:22:19.783 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:22:19.790+0000 7f3470c05700 1 --2- 172.21.15.52:0/3401220157 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f346c1034b0 0x7f346c19aec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:22:19.783 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:22:19.790+0000 7f3470c05700 1 -- 172.21.15.52:0/3401220157 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f346c19fce0 con 0x7f346c107de0 2024-05-27T07:22:19.783 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:22:19.790+0000 7f3471406700 1 --2- 172.21.15.52:0/3401220157 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f346c1034b0 0x7f346c19aec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-27T07:22:19.783 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:22:19.790+0000 7f3471c07700 1 --2- 172.21.15.52:0/3401220157 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f346c108730 0x7f346c19f6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-27T07:22:19.783 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:22:19.790+0000 7f3470c05700 1 --2- 172.21.15.52:0/3401220157 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f346c107de0 0x7f346c19b460 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7f345c00d980 tx=0x7f345c00dd40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:22:19.784 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:22:19.791+0000 7f34627fc700 1 -- 172.21.15.52:0/3401220157 <== mon.0 v2:172.21.15.52:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f345c017070 con 0x7f346c107de0 2024-05-27T07:22:19.784 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:22:19.791+0000 7f34627fc700 1 -- 172.21.15.52:0/3401220157 <== mon.0 v2:172.21.15.52:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f345c010460 con 0x7f346c107de0 2024-05-27T07:22:19.784 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:22:19.791+0000 7f347366a700 1 -- 172.21.15.52:0/3401220157 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f346c19ffd0 con 0x7f346c107de0 2024-05-27T07:22:19.784 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:22:19.791+0000 7f34627fc700 1 -- 172.21.15.52:0/3401220157 <== mon.0 v2:172.21.15.52:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f345c00de60 con 0x7f346c107de0 2024-05-27T07:22:19.784 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:22:19.791+0000 7f347366a700 1 -- 172.21.15.52:0/3401220157 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f346c1a79a0 con 0x7f346c107de0 2024-05-27T07:22:19.784 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:22:19.792+0000 7f347366a700 1 -- 172.21.15.52:0/3401220157 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f346c0512b0 con 0x7f346c107de0 2024-05-27T07:22:19.785 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:22:19.794+0000 7f34627fc700 1 -- 172.21.15.52:0/3401220157 <== mon.0 v2:172.21.15.52:3300/0 4 ==== mgrmap(e 16) ==== 149652+0+0 (secure 0 0 0) 0x7f345c010610 con 0x7f346c107de0 2024-05-27T07:22:19.785 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:22:19.794+0000 7f34627fc700 1 --2- 172.21.15.52:0/3401220157 >> [v2:172.21.15.52:6800/3513075805,v1:172.21.15.52:6801/3513075805] conn(0x7f34580b2900 0x7f34580b4e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:22:19.785 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:22:19.794+0000 7f3471406700 1 --2- 172.21.15.52:0/3401220157 >> [v2:172.21.15.52:6800/3513075805,v1:172.21.15.52:6801/3513075805] conn(0x7f34580b2900 0x7f34580b4e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:22:19.785 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:22:19.794+0000 7f34627fc700 1 -- 172.21.15.52:0/3401220157 <== mon.0 v2:172.21.15.52:3300/0 5 ==== osd_map(86..86 src has 1..86) ==== 8802+0+0 (secure 0 0 0) 0x7f345c0e1f90 con 0x7f346c107de0 2024-05-27T07:22:19.785 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:22:19.795+0000 7f3471406700 1 --2- 172.21.15.52:0/3401220157 >> [v2:172.21.15.52:6800/3513075805,v1:172.21.15.52:6801/3513075805] conn(0x7f34580b2900 0x7f34580b4e10 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7f3468000c00 tx=0x7f346801a040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:22:19.788 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:22:19.798+0000 7f34627fc700 1 -- 172.21.15.52:0/3401220157 <== mon.0 v2:172.21.15.52:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191963 (secure 0 0 0) 0x7f345c0a9af0 con 0x7f346c107de0 2024-05-27T07:22:20.005 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:22:20.015+0000 7f347366a700 1 -- 172.21.15.52:0/3401220157 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f346c02e5e0 con 0x7f346c107de0 2024-05-27T07:22:20.007 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:22:20.017+0000 7f34627fc700 1 -- 172.21.15.52:0/3401220157 <== mon.0 v2:172.21.15.52:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v86) ==== 74+0+24126 (secure 0 0 0) 0x7f345c0a93f0 con 0x7f346c107de0 2024-05-27T07:22:20.007 INFO:teuthology.orchestra.run.smithi052.stdout: 2024-05-27T07:22:20.008 INFO:teuthology.orchestra.run.smithi052.stdout:{"epoch":86,"fsid":"73df2636-1bf8-11ef-bc9b-c7b262605968","created":"2024-05-27T07:13:34.738032+0000","modified":"2024-05-27T07:22:04.612333+0000","last_up_change":"2024-05-27T07:20:45.777597+0000","last_in_change":"2024-05-27T07:20:25.314823+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-05-27T07:16:57.644034+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-05-27T07:21:43.917748+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"79","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-05-27T07:21:47.341876+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"79","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-05-27T07:21:57.707611+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"86","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"347b8a89-9dbe-4848-83b0-fd0e9374f33d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":82,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.52:6802","nonce":1626508899},{"type":"v1","addr":"172.21.15.52:6803","nonce":1626508899}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.52:6804","nonce":1626508899},{"type":"v1","addr":"172.21.15.52:6805","nonce":1626508899}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.52:6808","nonce":1626508899},{"type":"v1","addr":"172.21.15.52:6809","nonce":1626508899}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.52:6806","nonce":1626508899},{"type":"v1","addr":"172.21.15.52:6807","nonce":1626508899}]},"public_addr":"172.21.15.52:6803/1626508899","cluster_addr":"172.21.15.52:6805/1626508899","heartbeat_back_addr":"172.21.15.52:6809/1626508899","heartbeat_front_addr":"172.21.15.52:6807/1626508899","state":["exists","up"]},{"osd":1,"uuid":"7a3623d4-e087-40cb-9336-d4a5c52203d4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":82,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6800","nonce":2900093195},{"type":"v1","addr":"172.21.15.105:6801","nonce":2900093195}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6802","nonce":2900093195},{"type":"v1","addr":"172.21.15.105:6803","nonce":2900093195}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6806","nonce":2900093195},{"type":"v1","addr":"172.21.15.105:6807","nonce":2900093195}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6804","nonce":2900093195},{"type":"v1","addr":"172.21.15.105:6805","nonce":2900093195}]},"public_addr":"172.21.15.105:6801/2900093195","cluster_addr":"172.21.15.105:6803/2900093195","heartbeat_back_addr":"172.21.15.105:6807/2900093195","heartbeat_front_addr":"172.21.15.105:6805/2900093195","state":["exists","up"]},{"osd":2,"uuid":"41a9df5b-882a-47da-bb26-7f4d26ccf6c1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":82,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6800","nonce":2251189314},{"type":"v1","addr":"172.21.15.173:6801","nonce":2251189314}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6802","nonce":2251189314},{"type":"v1","addr":"172.21.15.173:6803","nonce":2251189314}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6806","nonce":2251189314},{"type":"v1","addr":"172.21.15.173:6807","nonce":2251189314}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6804","nonce":2251189314},{"type":"v1","addr":"172.21.15.173:6805","nonce":2251189314}]},"public_addr":"172.21.15.173:6801/2251189314","cluster_addr":"172.21.15.173:6803/2251189314","heartbeat_back_addr":"172.21.15.173:6807/2251189314","heartbeat_front_addr":"172.21.15.173:6805/2251189314","state":["exists","up"]},{"osd":3,"uuid":"f1245439-42a5-46c5-a998-e0a78f9df3fb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":26,"up_thru":82,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.52:6810","nonce":57920794},{"type":"v1","addr":"172.21.15.52:6811","nonce":57920794}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.52:6812","nonce":57920794},{"type":"v1","addr":"172.21.15.52:6813","nonce":57920794}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.52:6816","nonce":57920794},{"type":"v1","addr":"172.21.15.52:6817","nonce":57920794}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.52:6814","nonce":57920794},{"type":"v1","addr":"172.21.15.52:6815","nonce":57920794}]},"public_addr":"172.21.15.52:6811/57920794","cluster_addr":"172.21.15.52:6813/57920794","heartbeat_back_addr":"172.21.15.52:6817/57920794","heartbeat_front_addr":"172.21.15.52:6815/57920794","state":["exists","up"]},{"osd":4,"uuid":"2592d1cc-1726-4359-83d8-a2e6314b54ec","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":32,"up_thru":82,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6808","nonce":3441134818},{"type":"v1","addr":"172.21.15.105:6809","nonce":3441134818}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6810","nonce":3441134818},{"type":"v1","addr":"172.21.15.105:6811","nonce":3441134818}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6814","nonce":3441134818},{"type":"v1","addr":"172.21.15.105:6815","nonce":3441134818}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6812","nonce":3441134818},{"type":"v1","addr":"172.21.15.105:6813","nonce":3441134818}]},"public_addr":"172.21.15.105:6809/3441134818","cluster_addr":"172.21.15.105:6811/3441134818","heartbeat_back_addr":"172.21.15.105:6815/3441134818","heartbeat_front_addr":"172.21.15.105:6813/3441134818","state":["exists","up"]},{"osd":5,"uuid":"27e98ee2-4a92-4972-b2a3-959d0ed90981","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":37,"up_thru":82,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6808","nonce":2989239729},{"type":"v1","addr":"172.21.15.173:6809","nonce":2989239729}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6810","nonce":2989239729},{"type":"v1","addr":"172.21.15.173:6811","nonce":2989239729}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6814","nonce":2989239729},{"type":"v1","addr":"172.21.15.173:6815","nonce":2989239729}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6812","nonce":2989239729},{"type":"v1","addr":"172.21.15.173:6813","nonce":2989239729}]},"public_addr":"172.21.15.173:6809/2989239729","cluster_addr":"172.21.15.173:6811/2989239729","heartbeat_back_addr":"172.21.15.173:6815/2989239729","heartbeat_front_addr":"172.21.15.173:6813/2989239729","state":["exists","up"]},{"osd":6,"uuid":"2b0ec0cb-f438-4b2c-8642-0778ed374e9a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":43,"up_thru":82,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.52:6818","nonce":4244423065},{"type":"v1","addr":"172.21.15.52:6819","nonce":4244423065}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.52:6820","nonce":4244423065},{"type":"v1","addr":"172.21.15.52:6821","nonce":4244423065}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.52:6824","nonce":4244423065},{"type":"v1","addr":"172.21.15.52:6825","nonce":4244423065}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.52:6822","nonce":4244423065},{"type":"v1","addr":"172.21.15.52:6823","nonce":4244423065}]},"public_addr":"172.21.15.52:6819/4244423065","cluster_addr":"172.21.15.52:6821/4244423065","heartbeat_back_addr":"172.21.15.52:6825/4244423065","heartbeat_front_addr":"172.21.15.52:6823/4244423065","state":["exists","up"]},{"osd":7,"uuid":"cb2a7a6b-8015-43ea-865a-be144e45c51a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":48,"up_thru":82,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6816","nonce":2063611552},{"type":"v1","addr":"172.21.15.105:6817","nonce":2063611552}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6818","nonce":2063611552},{"type":"v1","addr":"172.21.15.105:6819","nonce":2063611552}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6822","nonce":2063611552},{"type":"v1","addr":"172.21.15.105:6823","nonce":2063611552}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6820","nonce":2063611552},{"type":"v1","addr":"172.21.15.105:6821","nonce":2063611552}]},"public_addr":"172.21.15.105:6817/2063611552","cluster_addr":"172.21.15.105:6819/2063611552","heartbeat_back_addr":"172.21.15.105:6823/2063611552","heartbeat_front_addr":"172.21.15.105:6821/2063611552","state":["exists","up"]},{"osd":8,"uuid":"c2fc565d-a0b9-4396-a209-bd884cddcfe3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":54,"up_thru":82,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6816","nonce":4131399852},{"type":"v1","addr":"172.21.15.173:6817","nonce":4131399852}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6818","nonce":4131399852},{"type":"v1","addr":"172.21.15.173:6819","nonce":4131399852}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6822","nonce":4131399852},{"type":"v1","addr":"172.21.15.173:6823","nonce":4131399852}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6820","nonce":4131399852},{"type":"v1","addr":"172.21.15.173:6821","nonce":4131399852}]},"public_addr":"172.21.15.173:6817/4131399852","cluster_addr":"172.21.15.173:6819/4131399852","heartbeat_back_addr":"172.21.15.173:6823/4131399852","heartbeat_front_addr":"172.21.15.173:6821/4131399852","state":["exists","up"]},{"osd":9,"uuid":"78fa7f17-1dad-4262-be40-31fa82e75520","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":60,"up_thru":82,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.52:6826","nonce":479119778},{"type":"v1","addr":"172.21.15.52:6827","nonce":479119778}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.52:6828","nonce":479119778},{"type":"v1","addr":"172.21.15.52:6829","nonce":479119778}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.52:6832","nonce":479119778},{"type":"v1","addr":"172.21.15.52:6833","nonce":479119778}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.52:6830","nonce":479119778},{"type":"v1","addr":"172.21.15.52:6831","nonce":479119778}]},"public_addr":"172.21.15.52:6827/479119778","cluster_addr":"172.21.15.52:6829/479119778","heartbeat_back_addr":"172.21.15.52:6833/479119778","heartbeat_front_addr":"172.21.15.52:6831/479119778","state":["exists","up"]},{"osd":10,"uuid":"e6f8e753-e15c-45b5-8ee9-d50f384559f9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":65,"up_thru":76,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6824","nonce":4032386628},{"type":"v1","addr":"172.21.15.105:6825","nonce":4032386628}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6826","nonce":4032386628},{"type":"v1","addr":"172.21.15.105:6827","nonce":4032386628}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6830","nonce":4032386628},{"type":"v1","addr":"172.21.15.105:6831","nonce":4032386628}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6828","nonce":4032386628},{"type":"v1","addr":"172.21.15.105:6829","nonce":4032386628}]},"public_addr":"172.21.15.105:6825/4032386628","cluster_addr":"172.21.15.105:6827/4032386628","heartbeat_back_addr":"172.21.15.105:6831/4032386628","heartbeat_front_addr":"172.21.15.105:6829/4032386628","state":["exists","up"]},{"osd":11,"uuid":"164572bf-67ed-4e8e-b32d-d7a2e8da607b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":71,"up_thru":82,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6824","nonce":1816308401},{"type":"v1","addr":"172.21.15.173:6825","nonce":1816308401}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6826","nonce":1816308401},{"type":"v1","addr":"172.21.15.173:6827","nonce":1816308401}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6830","nonce":1816308401},{"type":"v1","addr":"172.21.15.173:6831","nonce":1816308401}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6828","nonce":1816308401},{"type":"v1","addr":"172.21.15.173:6829","nonce":1816308401}]},"public_addr":"172.21.15.173:6825/1816308401","cluster_addr":"172.21.15.173:6827/1816308401","heartbeat_back_addr":"172.21.15.173:6831/1816308401","heartbeat_front_addr":"172.21.15.173:6829/1816308401","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-27T07:16:06.279220+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-27T07:16:29.668167+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-27T07:16:53.903808+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-27T07:17:18.787372+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-27T07:17:43.480784+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-27T07:18:08.816030+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-27T07:18:34.076148+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-27T07:19:00.283468+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-27T07:19:25.403023+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-27T07:19:52.139173+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-27T07:20:17.530304+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-27T07:20:43.665632+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.52:6801/1631683163":"2024-05-28T07:14:29.544633+0000","172.21.15.52:0/2776144659":"2024-05-28T07:14:29.544633+0000","172.21.15.52:6800/1631683163":"2024-05-28T07:14:29.544633+0000","172.21.15.52:0/3675529097":"2024-05-28T07:14:01.960550+0000","172.21.15.52:0/727808107":"2024-05-28T07:14:01.960550+0000","172.21.15.52:0/3839436833":"2024-05-28T07:14:29.544633+0000","172.21.15.52:6800/587662512":"2024-05-28T07:14:01.960550+0000","172.21.15.52:6801/587662512":"2024-05-28T07:14:01.960550+0000","172.21.15.52:0/853867465":"2024-05-28T07:14:01.960550+0000","172.21.15.52:0/3480918279":"2024-05-28T07:14:29.544633+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-05-27T07:22:20.010 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:22:20.020+0000 7f347366a700 1 -- 172.21.15.52:0/3401220157 >> [v2:172.21.15.52:6800/3513075805,v1:172.21.15.52:6801/3513075805] conn(0x7f34580b2900 msgr2=0x7f34580b4e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:22:20.011 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:22:20.020+0000 7f347366a700 1 --2- 172.21.15.52:0/3401220157 >> [v2:172.21.15.52:6800/3513075805,v1:172.21.15.52:6801/3513075805] conn(0x7f34580b2900 0x7f34580b4e10 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7f3468000c00 tx=0x7f346801a040 comp rx=0 tx=0).stop 2024-05-27T07:22:20.011 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:22:20.020+0000 7f347366a700 1 -- 172.21.15.52:0/3401220157 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f346c107de0 msgr2=0x7f346c19b460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:22:20.011 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:22:20.020+0000 7f347366a700 1 --2- 172.21.15.52:0/3401220157 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f346c107de0 0x7f346c19b460 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7f345c00d980 tx=0x7f345c00dd40 comp rx=0 tx=0).stop 2024-05-27T07:22:20.011 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:22:20.020+0000 7f347366a700 1 -- 172.21.15.52:0/3401220157 shutdown_connections 2024-05-27T07:22:20.011 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:22:20.020+0000 7f347366a700 1 --2- 172.21.15.52:0/3401220157 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f346c108730 0x7f346c19f6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:22:20.011 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:22:20.020+0000 7f347366a700 1 --2- 172.21.15.52:0/3401220157 >> [v2:172.21.15.52:6800/3513075805,v1:172.21.15.52:6801/3513075805] conn(0x7f34580b2900 0x7f34580b4e10 unknown :-1 s=CLOSED pgs=152 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:22:20.011 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:22:20.020+0000 7f347366a700 1 --2- 172.21.15.52:0/3401220157 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f346c1034b0 0x7f346c19aec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:22:20.011 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:22:20.020+0000 7f347366a700 1 --2- 172.21.15.52:0/3401220157 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f346c107de0 0x7f346c19b460 unknown :-1 s=CLOSED pgs=203 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:22:20.011 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:22:20.020+0000 7f347366a700 1 -- 172.21.15.52:0/3401220157 >> 172.21.15.52:0/3401220157 conn(0x7f346c0fede0 msgr2=0x7f346c101200 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:22:20.011 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:22:20.020+0000 7f347366a700 1 -- 172.21.15.52:0/3401220157 shutdown_connections 2024-05-27T07:22:20.011 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:22:20.020+0000 7f347366a700 1 -- 172.21.15.52:0/3401220157 wait complete. 2024-05-27T07:22:20.868 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 17, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-05-27T07:21:50.521783+0000', 'modified': '2024-05-27T07:22:04.612487+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 24527}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_24527': {'gid': 24527, 'name': 'b', 'rank': 0, 'incarnation': 14, 'state': 'up:active', 'state_seq': 5, 'addr': '172.21.15.105:6839/500348151', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.105:6838', 'nonce': 500348151}, {'type': 'v1', 'addr': '172.21.15.105:6839', 'nonce': 500348151}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 1, 'qdb_leader': 24527, 'qdb_cluster': [24527]} 2024-05-27T07:22:20.868 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2024-05-27T07:22:20.868 DEBUG:teuthology.orchestra.run.smithi052:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:38f9d900c83c63799fbdbe61acc9a11b0d3554a6 shell --fsid 73df2636-1bf8-11ef-bc9b-c7b262605968 -- ceph tell mds.b status 2024-05-27T07:22:21.102 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:22:20 smithi173 ceph-mon[37000]: pgmap v311: 129 pgs: 129 active+clean; 579 KiB data, 356 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-27T07:22:21.102 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:22:20 smithi173 ceph-mon[37000]: from='client.? 172.21.15.52:0/3401220157' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-05-27T07:22:21.121 INFO:teuthology.orchestra.run.smithi052.stderr:Inferring config /var/lib/ceph/73df2636-1bf8-11ef-bc9b-c7b262605968/mon.a/config 2024-05-27T07:22:21.190 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:22:20 smithi105 ceph-mon[35977]: pgmap v311: 129 pgs: 129 active+clean; 579 KiB data, 356 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-27T07:22:21.190 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:22:20 smithi105 ceph-mon[35977]: from='client.? 172.21.15.52:0/3401220157' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-05-27T07:22:21.229 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:22:20 smithi052 ceph-mon[31955]: pgmap v311: 129 pgs: 129 active+clean; 579 KiB data, 356 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-27T07:22:21.229 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:22:20 smithi052 ceph-mon[31955]: from='client.? 172.21.15.52:0/3401220157' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-05-27T07:22:22.753 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:22:22.762+0000 7fa17793b700 1 -- 172.21.15.52:0/1087806617 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fa170107e00 msgr2=0x7fa170108210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:22:22.753 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:22:22.762+0000 7fa17793b700 1 --2- 172.21.15.52:0/1087806617 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fa170107e00 0x7fa170108210 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7fa1600099c0 tx=0x7fa160009cd0 comp rx=0 tx=0).stop 2024-05-27T07:22:22.753 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:22:22.762+0000 7fa17793b700 1 -- 172.21.15.52:0/1087806617 shutdown_connections 2024-05-27T07:22:22.753 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:22:22.762+0000 7fa17793b700 1 --2- 172.21.15.52:0/1087806617 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fa170108750 0x7fa1701100b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:22:22.753 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:22:22.762+0000 7fa17793b700 1 --2- 172.21.15.52:0/1087806617 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fa1701034d0 0x7fa1701038e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:22:22.753 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:22:22.762+0000 7fa17793b700 1 --2- 172.21.15.52:0/1087806617 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fa170107e00 0x7fa170108210 unknown :-1 s=CLOSED pgs=204 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:22:22.753 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:22:22.762+0000 7fa17793b700 1 -- 172.21.15.52:0/1087806617 >> 172.21.15.52:0/1087806617 conn(0x7fa1700fef20 msgr2=0x7fa170101340 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:22:22.754 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:22:22.762+0000 7fa17793b700 1 -- 172.21.15.52:0/1087806617 shutdown_connections 2024-05-27T07:22:22.754 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:22:22.762+0000 7fa17793b700 1 -- 172.21.15.52:0/1087806617 wait complete. 2024-05-27T07:22:22.754 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:22:22.763+0000 7fa17793b700 1 Processor -- start 2024-05-27T07:22:22.754 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:22:22.763+0000 7fa17793b700 1 -- start start 2024-05-27T07:22:22.755 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:22:22.763+0000 7fa17793b700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fa1701034d0 0x7fa17019bce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:22:22.756 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:22:22.763+0000 7fa17793b700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fa170107e00 0x7fa17019c220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:22:22.756 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:22:22.763+0000 7fa17793b700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fa170108750 0x7fa17019c780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:22:22.756 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:22:22.763+0000 7fa17793b700 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x7fa170113360 con 0x7fa170107e00 2024-05-27T07:22:22.756 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:22:22.763+0000 7fa17793b700 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7fa1701131e0 con 0x7fa1701034d0 2024-05-27T07:22:22.756 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:22:22.763+0000 7fa17793b700 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7fa1701134e0 con 0x7fa170108750 2024-05-27T07:22:22.756 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:22:22.763+0000 7fa174ed6700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fa170107e00 0x7fa17019c220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:22:22.756 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:22:22.763+0000 7fa175ed8700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fa170108750 0x7fa17019c780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:22:22.756 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:22:22.763+0000 7fa174ed6700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fa170107e00 0x7fa17019c220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.52:60488/0 (socket says 172.21.15.52:60488) 2024-05-27T07:22:22.756 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:22:22.763+0000 7fa174ed6700 1 -- 172.21.15.52:0/1745825753 learned_addr learned my addr 172.21.15.52:0/1745825753 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-05-27T07:22:22.757 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:22:22.763+0000 7fa1756d7700 1 --2- 172.21.15.52:0/1745825753 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fa1701034d0 0x7fa17019bce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:22:22.757 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:22:22.764+0000 7fa174ed6700 1 -- 172.21.15.52:0/1745825753 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fa170108750 msgr2=0x7fa17019c780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:22:22.757 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:22:22.764+0000 7fa174ed6700 1 --2- 172.21.15.52:0/1745825753 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fa170108750 0x7fa17019c780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:22:22.757 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:22:22.764+0000 7fa174ed6700 1 -- 172.21.15.52:0/1745825753 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fa1701034d0 msgr2=0x7fa17019bce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:22:22.757 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:22:22.764+0000 7fa174ed6700 1 --2- 172.21.15.52:0/1745825753 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fa1701034d0 0x7fa17019bce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:22:22.757 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:22:22.764+0000 7fa174ed6700 1 -- 172.21.15.52:0/1745825753 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa1701a7030 con 0x7fa170107e00 2024-05-27T07:22:22.757 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:22:22.764+0000 7fa174ed6700 1 --2- 172.21.15.52:0/1745825753 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fa170107e00 0x7fa17019c220 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7fa160006010 tx=0x7fa1600047b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:22:22.757 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:22:22.764+0000 7fa1667fc700 1 -- 172.21.15.52:0/1745825753 <== mon.0 v2:172.21.15.52:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa160027070 con 0x7fa170107e00 2024-05-27T07:22:22.757 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:22:22.764+0000 7fa17793b700 1 -- 172.21.15.52:0/1745825753 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa1701a72c0 con 0x7fa170107e00 2024-05-27T07:22:22.757 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:22:22.764+0000 7fa1667fc700 1 -- 172.21.15.52:0/1745825753 <== mon.0 v2:172.21.15.52:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fa160003f20 con 0x7fa170107e00 2024-05-27T07:22:22.757 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:22:22.764+0000 7fa1667fc700 1 -- 172.21.15.52:0/1745825753 <== mon.0 v2:172.21.15.52:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa160022c60 con 0x7fa170107e00 2024-05-27T07:22:22.757 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:22:22.764+0000 7fa17793b700 1 -- 172.21.15.52:0/1745825753 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa1701a7720 con 0x7fa170107e00 2024-05-27T07:22:22.758 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:22:22.768+0000 7fa1667fc700 1 -- 172.21.15.52:0/1745825753 <== mon.0 v2:172.21.15.52:3300/0 4 ==== mgrmap(e 16) ==== 149652+0+0 (secure 0 0 0) 0x7fa16002a070 con 0x7fa170107e00 2024-05-27T07:22:22.761 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:22:22.769+0000 7fa186350b80 1 Processor -- start 2024-05-27T07:22:22.761 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:22:22.769+0000 7fa1667fc700 1 --2- 172.21.15.52:0/1745825753 >> [v2:172.21.15.52:6800/3513075805,v1:172.21.15.52:6801/3513075805] conn(0x7fa15c0b2880 0x7fa15c0b4d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:22:22.761 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:22:22.769+0000 7fa186350b80 1 -- start start 2024-05-27T07:22:22.761 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:22:22.769+0000 7fa175ed8700 1 --2- 172.21.15.52:0/1745825753 >> [v2:172.21.15.52:6800/3513075805,v1:172.21.15.52:6801/3513075805] conn(0x7fa15c0b2880 0x7fa15c0b4d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:22:22.761 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:22:22.770+0000 7fa1667fc700 1 -- 172.21.15.52:0/1745825753 <== mon.0 v2:172.21.15.52:3300/0 5 ==== osd_map(86..86 src has 1..86) ==== 8802+0+0 (secure 0 0 0) 0x7fa160022e00 con 0x7fa170107e00 2024-05-27T07:22:22.761 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:22:22.770+0000 7fa186350b80 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x559388d54df0 0x559388edcb50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:22:22.761 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:22:22.770+0000 7fa186350b80 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x559388edd090 0x559388ed6e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:22:22.761 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:22:22.770+0000 7fa186350b80 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x559388ed7340 0x559388ed7750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:22:22.761 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:22:22.770+0000 7fa186350b80 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x559388e01c40 con 0x559388ed7340 2024-05-27T07:22:22.762 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:22:22.770+0000 7fa186350b80 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x559388df17e0 con 0x559388edd090 2024-05-27T07:22:22.762 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:22:22.770+0000 7fa186350b80 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x559388df5820 con 0x559388d54df0 2024-05-27T07:22:22.762 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:22:22.770+0000 7fa1756d7700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x559388d54df0 0x559388edcb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:22:22.762 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:22:22.770+0000 7fa175ed8700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x559388ed7340 0x559388ed7750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:22:22.762 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:22:22.770+0000 7fa1756d7700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x559388d54df0 0x559388edcb50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.173:3300/0 says I am v2:172.21.15.52:44044/0 (socket says 172.21.15.52:44044) 2024-05-27T07:22:22.762 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:22:22.770+0000 7fa1756d7700 1 -- 172.21.15.52:0/1512018212 learned_addr learned my addr 172.21.15.52:0/1512018212 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-05-27T07:22:22.762 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:22:22.770+0000 7fa174ed6700 1 --2- 172.21.15.52:0/1512018212 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x559388edd090 0x559388ed6e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:22:22.762 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:22:22.770+0000 7fa175ed8700 1 --2- 172.21.15.52:0/1745825753 >> [v2:172.21.15.52:6800/3513075805,v1:172.21.15.52:6801/3513075805] conn(0x7fa15c0b2880 0x7fa15c0b4d90 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7fa17006d110 tx=0x7fa16800af20 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:22:22.762 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:22:22.770+0000 7fa1756d7700 1 -- 172.21.15.52:0/1512018212 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x559388edd090 msgr2=0x559388ed6e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:22:22.762 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:22:22.770+0000 7fa1756d7700 1 --2- 172.21.15.52:0/1512018212 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x559388edd090 0x559388ed6e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:22:22.762 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:22:22.770+0000 7fa1756d7700 1 -- 172.21.15.52:0/1512018212 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x559388ed7340 msgr2=0x559388ed7750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:22:22.763 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:22:22.770+0000 7fa1756d7700 1 --2- 172.21.15.52:0/1512018212 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x559388ed7340 0x559388ed7750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:22:22.763 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:22:22.770+0000 7fa1756d7700 1 -- 172.21.15.52:0/1512018212 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x559388d41880 con 0x559388d54df0 2024-05-27T07:22:22.763 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:22:22.770+0000 7fa174ed6700 1 --2- 172.21.15.52:0/1512018212 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x559388edd090 0x559388ed6e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-27T07:22:22.763 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:22:22.771+0000 7fa1756d7700 1 --2- 172.21.15.52:0/1512018212 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x559388d54df0 0x559388edcb50 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7fa16c009990 tx=0x7fa16c009ca0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=8c8190f26aa25944 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:22:22.763 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:22:22.771+0000 7fa15b7fe700 1 -- 172.21.15.52:0/1512018212 <== mon.1 v2:172.21.15.173:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa16c004780 con 0x559388d54df0 2024-05-27T07:22:22.763 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:22:22.771+0000 7fa15b7fe700 1 -- 172.21.15.52:0/1512018212 <== mon.1 v2:172.21.15.173:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fa16c01a070 con 0x559388d54df0 2024-05-27T07:22:22.763 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:22:22.771+0000 7fa15b7fe700 1 -- 172.21.15.52:0/1512018212 <== mon.1 v2:172.21.15.173:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa16c015770 con 0x559388d54df0 2024-05-27T07:22:22.765 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:22:22.772+0000 7fa186350b80 1 -- 172.21.15.52:0/1512018212 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x559388d54df0 msgr2=0x559388edcb50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:22:22.765 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:22:22.772+0000 7fa186350b80 1 --2- 172.21.15.52:0/1512018212 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x559388d54df0 0x559388edcb50 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7fa16c009990 tx=0x7fa16c009ca0 comp rx=0 tx=0).stop 2024-05-27T07:22:22.765 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:22:22.772+0000 7fa186350b80 1 -- 172.21.15.52:0/1512018212 shutdown_connections 2024-05-27T07:22:22.765 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:22:22.772+0000 7fa186350b80 1 --2- 172.21.15.52:0/1512018212 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x559388ed7340 0x559388ed7750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:22:22.765 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:22:22.772+0000 7fa186350b80 1 --2- 172.21.15.52:0/1512018212 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x559388d54df0 0x559388edcb50 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:22:22.766 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:22:22.772+0000 7fa186350b80 1 --2- 172.21.15.52:0/1512018212 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x559388edd090 0x559388ed6e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:22:22.766 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:22:22.772+0000 7fa186350b80 1 -- 172.21.15.52:0/1512018212 >> 172.21.15.52:0/1512018212 conn(0x559388dafaa0 msgr2=0x559388ecd5f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:22:22.766 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:22:22.772+0000 7fa186350b80 1 -- 172.21.15.52:0/1512018212 shutdown_connections 2024-05-27T07:22:22.766 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:22:22.772+0000 7fa186350b80 1 -- 172.21.15.52:0/1512018212 wait complete. 2024-05-27T07:22:22.766 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:22:22.772+0000 7fa186350b80 1 -- start start 2024-05-27T07:22:22.766 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:22:22.772+0000 7fa186350b80 1 Processor -- start 2024-05-27T07:22:22.766 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:22:22.773+0000 7fa186350b80 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x559388d54df0 0x559388ee0710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:22:22.766 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:22:22.773+0000 7fa186350b80 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x559388ed7340 0x559388ee0c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:22:22.766 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:22:22.773+0000 7fa186350b80 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x559388edd090 0x559388ee1190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:22:22.766 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:22:22.773+0000 7fa186350b80 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x559388cf7f50 con 0x559388ed7340 2024-05-27T07:22:22.766 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:22:22.773+0000 7fa186350b80 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x559388d93770 con 0x559388edd090 2024-05-27T07:22:22.766 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:22:22.773+0000 7fa186350b80 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x559388ee3ac0 con 0x559388d54df0 2024-05-27T07:22:22.766 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:22:22.773+0000 7fa186350b80 10 client.0 _ll_register_callbacks cb 0x559388d8c8d0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T07:22:22.767 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:22:22.773+0000 7fa1756d7700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x559388d54df0 0x559388ee0710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:22:22.767 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:22:22.773+0000 7fa174ed6700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x559388ed7340 0x559388ee0c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:22:22.767 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:22:22.773+0000 7fa1756d7700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x559388d54df0 0x559388ee0710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.173:3300/0 says I am v2:172.21.15.52:44060/0 (socket says 172.21.15.52:44060) 2024-05-27T07:22:22.767 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:22:22.773+0000 7fa1756d7700 1 -- 172.21.15.52:0/1427096405 learned_addr learned my addr 172.21.15.52:0/1427096405 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-05-27T07:22:22.767 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:22:22.773+0000 7fa174ed6700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x559388ed7340 0x559388ee0c50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.52:60504/0 (socket says 172.21.15.52:60504) 2024-05-27T07:22:22.767 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:22:22.773+0000 7fa175ed8700 1 --2- 172.21.15.52:0/1427096405 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x559388edd090 0x559388ee1190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:22:22.767 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:22:22.773+0000 7fa1756d7700 1 -- 172.21.15.52:0/1427096405 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x559388edd090 msgr2=0x559388ee1190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:22:22.767 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:22:22.773+0000 7fa1756d7700 1 --2- 172.21.15.52:0/1427096405 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x559388edd090 0x559388ee1190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:22:22.767 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:22:22.773+0000 7fa1756d7700 1 -- 172.21.15.52:0/1427096405 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x559388ed7340 msgr2=0x559388ee0c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:22:22.767 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:22:22.773+0000 7fa1756d7700 1 --2- 172.21.15.52:0/1427096405 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x559388ed7340 0x559388ee0c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:22:22.767 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:22:22.773+0000 7fa1756d7700 1 -- 172.21.15.52:0/1427096405 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({osdmap=0}) -- 0x559388d90640 con 0x559388d54df0 2024-05-27T07:22:22.767 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:22:22.773+0000 7fa174ed6700 1 --2- 172.21.15.52:0/1427096405 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x559388ed7340 0x559388ee0c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-27T07:22:22.767 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:22:22.774+0000 7fa1756d7700 1 --2- 172.21.15.52:0/1427096405 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x559388d54df0 0x559388ee0710 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7fa16c007e30 tx=0x7fa16c0077b0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=87329a1c797f77f6 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:22:22.768 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:22:22.774+0000 7fa158ff9700 10 client.0 ms_handle_connect on v2:172.21.15.173:3300/0 2024-05-27T07:22:22.769 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:22:22.777+0000 7fa158ff9700 1 -- 172.21.15.52:0/1427096405 <== mon.1 v2:172.21.15.173:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa16c023070 con 0x559388d54df0 2024-05-27T07:22:22.769 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:22:22.778+0000 7fa158ff9700 1 -- 172.21.15.52:0/1427096405 <== mon.1 v2:172.21.15.173:3300/0 2 ==== osd_map(86..86 src has 1..86) ==== 8802+0+0 (secure 0 0 0) 0x7fa16c015480 con 0x559388d54df0 2024-05-27T07:22:22.769 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:22:22.778+0000 7fa186350b80 1 -- 172.21.15.52:0/1427096405 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fa16c01a070 con 0x559388d54df0 2024-05-27T07:22:22.769 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:22:22.778+0000 7fa158ff9700 1 -- 172.21.15.52:0/1427096405 <== mon.1 v2:172.21.15.173:3300/0 3 ==== mon_get_version_reply(handle=1 version=17) ==== 24+0+0 (secure 0 0 0) 0x7fa16c02f050 con 0x559388d54df0 2024-05-27T07:22:22.769 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:22:22.778+0000 7fa186350b80 10 client.24562 fetch_fsmap learned FSMap version 17 2024-05-27T07:22:22.769 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:22:22.778+0000 7fa186350b80 1 -- 172.21.15.52:0/1427096405 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({config=0+,fsmap=17,monmap=4+}) -- 0x559388eee4f0 con 0x559388d54df0 2024-05-27T07:22:22.770 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:22:22.778+0000 7fa158ff9700 1 -- 172.21.15.52:0/1427096405 <== mon.1 v2:172.21.15.173:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fa16c01a070 con 0x559388d54df0 2024-05-27T07:22:22.770 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:22:22.778+0000 7fa158ff9700 1 -- 172.21.15.52:0/1427096405 <== mon.1 v2:172.21.15.173:3300/0 5 ==== fsmap(e 17) ==== 8342+0+0 (secure 0 0 0) 0x7fa16c005070 con 0x559388d54df0 2024-05-27T07:22:22.770 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:22:22.778+0000 7fa186350b80 10 client.24562 fetch_fsmap finished waiting for FSMap version 17 2024-05-27T07:22:22.770 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:22:22.778+0000 7fa186350b80 10 client.24562 resolve_mds: resolved name 'b' to daemon mds.b 2024-05-27T07:22:22.770 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:22:22.778+0000 7fa186350b80 20 client.24562 populate_metadata read hostname 'smithi052' 2024-05-27T07:22:22.770 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:22:22.778+0000 7fa186350b80 1 --2- 172.21.15.52:0/1427096405 >> [v2:172.21.15.105:6838/500348151,v1:172.21.15.105:6839/500348151] conn(0x559388ef0720 0x559388ef1b90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:22:22.770 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:22:22.778+0000 7fa186350b80 4 client.24562 mds_command: new command op to 24527 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-27T07:22:22.770 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:22:22.778+0000 7fa186350b80 1 -- 172.21.15.52:0/1427096405 --> [v2:172.21.15.105:6838/500348151,v1:172.21.15.105:6839/500348151] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x559388e9e9a0 con 0x559388ef0720 2024-05-27T07:22:22.770 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:22:22.779+0000 7fa174ed6700 1 --2- 172.21.15.52:0/1427096405 >> [v2:172.21.15.105:6838/500348151,v1:172.21.15.105:6839/500348151] conn(0x559388ef0720 0x559388ef1b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:22:22.770 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:22:22.779+0000 7fa174ed6700 1 --2- 172.21.15.52:0/1427096405 >> [v2:172.21.15.105:6838/500348151,v1:172.21.15.105:6839/500348151] conn(0x559388ef0720 0x559388ef1b90 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=91d0bfd7f8855d8d server_cookie=85b1a8a68eaa484f in_seq=0 out_seq=0 2024-05-27T07:22:22.770 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:22:22.779+0000 7fa158ff9700 10 client.24562 ms_handle_connect on v2:172.21.15.105:6838/500348151 2024-05-27T07:22:22.771 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:22:22.780+0000 7fa158ff9700 1 -- 172.21.15.52:0/1427096405 <== mds.0 v2:172.21.15.105:6838/500348151 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24041 (crc 0 0 0) 0x559388e9e9a0 con 0x559388ef0720 2024-05-27T07:22:22.772 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:22:22.780+0000 7fa158ff9700 10 client.24562 handle_command_reply: tid=0 2024-05-27T07:22:22.772 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:22:22.780+0000 7fa186350b80 1 client.24562 shutdown 2024-05-27T07:22:22.772 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:22:22.780+0000 7fa186350b80 1 -- 172.21.15.52:0/1427096405 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x559388d54df0 msgr2=0x559388ee0710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:22:22.772 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:22:22.780+0000 7fa186350b80 1 --2- 172.21.15.52:0/1427096405 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x559388d54df0 0x559388ee0710 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7fa16c007e30 tx=0x7fa16c0077b0 comp rx=0 tx=0).stop 2024-05-27T07:22:22.772 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:22:22.780+0000 7fa186350b80 1 -- 172.21.15.52:0/1427096405 shutdown_connections 2024-05-27T07:22:22.772 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:22:22.781+0000 7fa186350b80 1 --2- 172.21.15.52:0/1427096405 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x559388edd090 0x559388ee1190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:22:22.772 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:22:22.781+0000 7fa186350b80 1 --2- 172.21.15.52:0/1427096405 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x559388d54df0 0x559388ee0710 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:22:22.772 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:22:22.781+0000 7fa186350b80 1 --2- 172.21.15.52:0/1427096405 >> [v2:172.21.15.105:6838/500348151,v1:172.21.15.105:6839/500348151] conn(0x559388ef0720 0x559388ef1b90 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:22:22.772 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:22:22.781+0000 7fa186350b80 1 --2- 172.21.15.52:0/1427096405 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x559388ed7340 0x559388ee0c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:22:22.772 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:22:22.781+0000 7fa158ff9700 0 client.24562 ms_handle_reset on v2:172.21.15.105:6838/500348151 2024-05-27T07:22:22.772 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:22:22.781+0000 7fa186350b80 1 -- 172.21.15.52:0/1427096405 >> 172.21.15.52:0/1427096405 conn(0x559388dafaa0 msgr2=0x559388ed19e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:22:22.772 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:22:22.781+0000 7fa186350b80 1 -- 172.21.15.52:0/1427096405 shutdown_connections 2024-05-27T07:22:22.772 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:22:22.781+0000 7fa186350b80 1 -- 172.21.15.52:0/1427096405 wait complete. 2024-05-27T07:22:22.773 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:22:22.781+0000 7fa186350b80 20 client.24562 trim_cache size 0 max 16384 2024-05-27T07:22:22.793 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:22:22.802+0000 7fa186350b80 1 Processor -- start 2024-05-27T07:22:22.793 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:22:22.802+0000 7fa186350b80 1 -- start start 2024-05-27T07:22:22.793 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:22:22.802+0000 7fa186350b80 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x559388d54df0 0x559388ef0050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:22:22.793 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:22:22.802+0000 7fa186350b80 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x559388ed7340 0x559388eecd20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:22:22.794 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:22:22.802+0000 7fa186350b80 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x559388edd090 0x559388ee0fd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:22:22.794 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:22:22.802+0000 7fa186350b80 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x559388eefed0 con 0x559388edd090 2024-05-27T07:22:22.794 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:22:22.802+0000 7fa186350b80 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x559388ef05a0 con 0x559388d54df0 2024-05-27T07:22:22.794 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:22:22.802+0000 7fa186350b80 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x559388ef0c70 con 0x559388ed7340 2024-05-27T07:22:22.794 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:22:22.803+0000 7fa175ed8700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x559388edd090 0x559388ee0fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:22:22.794 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:22:22.803+0000 7fa175ed8700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x559388edd090 0x559388ee0fd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.52:60508/0 (socket says 172.21.15.52:60508) 2024-05-27T07:22:22.794 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:22:22.803+0000 7fa175ed8700 1 -- 172.21.15.52:0/2893402227 learned_addr learned my addr 172.21.15.52:0/2893402227 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-05-27T07:22:22.794 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:22:22.803+0000 7fa1756d7700 1 --2- 172.21.15.52:0/2893402227 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x559388d54df0 0x559388ef0050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:22:22.794 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:22:22.803+0000 7fa174ed6700 1 --2- 172.21.15.52:0/2893402227 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x559388ed7340 0x559388eecd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:22:22.794 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:22:22.803+0000 7fa175ed8700 1 -- 172.21.15.52:0/2893402227 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x559388ed7340 msgr2=0x559388eecd20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:22:22.794 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:22:22.803+0000 7fa175ed8700 1 --2- 172.21.15.52:0/2893402227 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x559388ed7340 0x559388eecd20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:22:22.794 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:22:22.803+0000 7fa175ed8700 1 -- 172.21.15.52:0/2893402227 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x559388d54df0 msgr2=0x559388ef0050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:22:22.795 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:22:22.803+0000 7fa175ed8700 1 --2- 172.21.15.52:0/2893402227 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x559388d54df0 0x559388ef0050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:22:22.795 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:22:22.803+0000 7fa175ed8700 1 -- 172.21.15.52:0/2893402227 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x559388edeca0 con 0x559388edd090 2024-05-27T07:22:22.795 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:22:22.803+0000 7fa1756d7700 1 --2- 172.21.15.52:0/2893402227 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x559388d54df0 0x559388ef0050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-27T07:22:22.795 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:22:22.803+0000 7fa175ed8700 1 --2- 172.21.15.52:0/2893402227 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x559388edd090 0x559388ee0fd0 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7fa16800ffb0 tx=0x7fa16801a000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5fb554cb74d00e18 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:22:22.795 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:22:22.803+0000 7fa15affd700 1 -- 172.21.15.52:0/2893402227 <== mon.0 v2:172.21.15.52:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa16802c070 con 0x559388edd090 2024-05-27T07:22:22.795 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:22:22.803+0000 7fa15affd700 1 -- 172.21.15.52:0/2893402227 <== mon.0 v2:172.21.15.52:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fa168027b10 con 0x559388edd090 2024-05-27T07:22:22.795 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:22:22.803+0000 7fa15affd700 1 -- 172.21.15.52:0/2893402227 <== mon.0 v2:172.21.15.52:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa168018070 con 0x559388edd090 2024-05-27T07:22:22.799 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:22:22.804+0000 7fa186350b80 1 -- 172.21.15.52:0/2893402227 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x559388edd090 msgr2=0x559388ee0fd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:22:22.799 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:22:22.804+0000 7fa186350b80 1 --2- 172.21.15.52:0/2893402227 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x559388edd090 0x559388ee0fd0 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7fa16800ffb0 tx=0x7fa16801a000 comp rx=0 tx=0).stop 2024-05-27T07:22:22.799 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:22:22.804+0000 7fa186350b80 1 -- 172.21.15.52:0/2893402227 shutdown_connections 2024-05-27T07:22:22.799 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:22:22.804+0000 7fa186350b80 1 --2- 172.21.15.52:0/2893402227 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x559388edd090 0x559388ee0fd0 unknown :-1 s=CLOSED pgs=206 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:22:22.799 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:22:22.804+0000 7fa186350b80 1 --2- 172.21.15.52:0/2893402227 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x559388d54df0 0x559388ef0050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:22:22.799 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:22:22.804+0000 7fa186350b80 1 --2- 172.21.15.52:0/2893402227 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x559388ed7340 0x559388eecd20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:22:22.799 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:22:22.804+0000 7fa186350b80 1 -- 172.21.15.52:0/2893402227 >> 172.21.15.52:0/2893402227 conn(0x559388dafaa0 msgr2=0x559388ed7750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:22:22.799 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:22:22.804+0000 7fa186350b80 1 -- 172.21.15.52:0/2893402227 shutdown_connections 2024-05-27T07:22:22.799 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:22:22.804+0000 7fa186350b80 1 -- 172.21.15.52:0/2893402227 wait complete. 2024-05-27T07:22:22.799 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:22:22.805+0000 7fa186350b80 1 -- start start 2024-05-27T07:22:22.799 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:22:22.805+0000 7fa186350b80 1 Processor -- start 2024-05-27T07:22:22.800 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:22:22.805+0000 7fa186350b80 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x559388d54df0 0x559388efb700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:22:22.800 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:22:22.805+0000 7fa186350b80 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x559388ed7340 0x559388eebdb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:22:22.800 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:22:22.805+0000 7fa186350b80 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x559388edd090 0x559388eec2f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:22:22.800 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:22:22.805+0000 7fa186350b80 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x559388ef1a10 con 0x559388ed7340 2024-05-27T07:22:22.800 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:22:22.805+0000 7fa186350b80 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x559388ef1340 con 0x559388edd090 2024-05-27T07:22:22.800 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:22:22.805+0000 7fa186350b80 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x559388ef20e0 con 0x559388d54df0 2024-05-27T07:22:22.800 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:22:22.805+0000 7fa186350b80 10 client.0 _ll_register_callbacks cb 0x559388d8c8d0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T07:22:22.800 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:22:22.805+0000 7fa174ed6700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x559388ed7340 0x559388eebdb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:22:22.800 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:22:22.805+0000 7fa1756d7700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x559388d54df0 0x559388efb700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:22:22.800 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:22:22.805+0000 7fa174ed6700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x559388ed7340 0x559388eebdb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.52:60516/0 (socket says 172.21.15.52:60516) 2024-05-27T07:22:22.800 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:22:22.805+0000 7fa174ed6700 1 -- 172.21.15.52:0/3137368447 learned_addr learned my addr 172.21.15.52:0/3137368447 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-05-27T07:22:22.800 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:22:22.806+0000 7fa175ed8700 1 --2- 172.21.15.52:0/3137368447 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x559388edd090 0x559388eec2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:22:22.800 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:22:22.806+0000 7fa1756d7700 1 -- 172.21.15.52:0/3137368447 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x559388edd090 msgr2=0x559388eec2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:22:22.801 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:22:22.806+0000 7fa1756d7700 1 --2- 172.21.15.52:0/3137368447 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x559388edd090 0x559388eec2f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:22:22.801 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:22:22.806+0000 7fa1756d7700 1 -- 172.21.15.52:0/3137368447 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x559388ed7340 msgr2=0x559388eebdb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:22:22.801 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:22:22.806+0000 7fa1756d7700 1 --2- 172.21.15.52:0/3137368447 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x559388ed7340 0x559388eebdb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:22:22.801 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:22:22.806+0000 7fa1756d7700 1 -- 172.21.15.52:0/3137368447 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({osdmap=0}) -- 0x559388eeca90 con 0x559388d54df0 2024-05-27T07:22:22.801 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:22:22.806+0000 7fa1756d7700 1 --2- 172.21.15.52:0/3137368447 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x559388d54df0 0x559388efb700 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7fa16c0098e0 tx=0x7fa16c005960 comp rx=0 tx=0).ready entity=mon.1 client_cookie=b121f82695d03183 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:22:22.801 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:22:22.806+0000 7fa158ff9700 10 client.0 ms_handle_connect on v2:172.21.15.173:3300/0 2024-05-27T07:22:22.801 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:22:22.806+0000 7fa158ff9700 1 -- 172.21.15.52:0/3137368447 <== mon.1 v2:172.21.15.173:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa16c023040 con 0x559388d54df0 2024-05-27T07:22:22.801 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:22:22.806+0000 7fa158ff9700 1 -- 172.21.15.52:0/3137368447 <== mon.1 v2:172.21.15.173:3300/0 2 ==== osd_map(86..86 src has 1..86) ==== 8802+0+0 (secure 0 0 0) 0x7fa16c005e70 con 0x559388d54df0 2024-05-27T07:22:22.801 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:22:22.806+0000 7fa186350b80 1 -- 172.21.15.52:0/3137368447 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fa168027b10 con 0x559388d54df0 2024-05-27T07:22:22.801 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:22:22.806+0000 7fa158ff9700 1 -- 172.21.15.52:0/3137368447 <== mon.1 v2:172.21.15.173:3300/0 3 ==== mon_get_version_reply(handle=1 version=17) ==== 24+0+0 (secure 0 0 0) 0x7fa16c007e60 con 0x559388d54df0 2024-05-27T07:22:22.801 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:22:22.806+0000 7fa186350b80 10 client.24568 fetch_fsmap learned FSMap version 17 2024-05-27T07:22:22.801 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:22:22.806+0000 7fa186350b80 1 -- 172.21.15.52:0/3137368447 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({config=0+,fsmap=17,monmap=4+}) -- 0x559388f5beb0 con 0x559388d54df0 2024-05-27T07:22:22.801 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:22:22.807+0000 7fa158ff9700 1 -- 172.21.15.52:0/3137368447 <== mon.1 v2:172.21.15.173:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fa168027b10 con 0x559388d54df0 2024-05-27T07:22:22.802 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:22:22.807+0000 7fa158ff9700 1 -- 172.21.15.52:0/3137368447 <== mon.1 v2:172.21.15.173:3300/0 5 ==== fsmap(e 17) ==== 8342+0+0 (secure 0 0 0) 0x7fa16c028440 con 0x559388d54df0 2024-05-27T07:22:22.802 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:22:22.807+0000 7fa186350b80 10 client.24568 fetch_fsmap finished waiting for FSMap version 17 2024-05-27T07:22:22.802 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:22:22.807+0000 7fa186350b80 10 client.24568 resolve_mds: resolved name 'b' to daemon mds.b 2024-05-27T07:22:22.802 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:22:22.807+0000 7fa186350b80 20 client.24568 populate_metadata read hostname 'smithi052' 2024-05-27T07:22:22.802 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:22:22.807+0000 7fa186350b80 1 --2- 172.21.15.52:0/3137368447 >> [v2:172.21.15.105:6838/500348151,v1:172.21.15.105:6839/500348151] conn(0x559388f5e150 0x559388f5f5c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:22:22.802 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:22:22.807+0000 7fa186350b80 4 client.24568 mds_command: new command op to 24527 tid=0 multi_id=0 [{"prefix": "status"}] 2024-05-27T07:22:22.802 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:22:22.807+0000 7fa186350b80 1 -- 172.21.15.52:0/3137368447 --> [v2:172.21.15.105:6838/500348151,v1:172.21.15.105:6839/500348151] -- command(tid 0: {"prefix": "status"}) -- 0x559388e03b90 con 0x559388f5e150 2024-05-27T07:22:22.802 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:22:22.807+0000 7fa174ed6700 1 --2- 172.21.15.52:0/3137368447 >> [v2:172.21.15.105:6838/500348151,v1:172.21.15.105:6839/500348151] conn(0x559388f5e150 0x559388f5f5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:22:22.802 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:22:22.808+0000 7fa158ff9700 10 client.24568 ms_handle_connect on v2:172.21.15.105:6838/500348151 2024-05-27T07:22:22.802 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:22:22.808+0000 7fa174ed6700 1 --2- 172.21.15.52:0/3137368447 >> [v2:172.21.15.105:6838/500348151,v1:172.21.15.105:6839/500348151] conn(0x559388f5e150 0x559388f5f5c0 crc :-1 s=READY pgs=14 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=ec90a13a64534f4 server_cookie=277547216b4d5ff5 in_seq=0 out_seq=0 2024-05-27T07:22:22.802 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:22:22.808+0000 7fa158ff9700 1 -- 172.21.15.52:0/3137368447 <== mds.0 v2:172.21.15.105:6838/500348151 1 ==== command_reply(tid 0: 0 ) ==== 8+0+332 (crc 0 0 0) 0x559388e03b90 con 0x559388f5e150 2024-05-27T07:22:22.803 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:22:22.808+0000 7fa158ff9700 10 client.24568 handle_command_reply: tid=0 2024-05-27T07:22:22.803 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:22:22.808+0000 7fa186350b80 1 client.24568 shutdown 2024-05-27T07:22:22.803 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:22:22.808+0000 7fa186350b80 1 -- 172.21.15.52:0/3137368447 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x559388d54df0 msgr2=0x559388efb700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:22:22.803 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:22:22.808+0000 7fa186350b80 1 --2- 172.21.15.52:0/3137368447 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x559388d54df0 0x559388efb700 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7fa16c0098e0 tx=0x7fa16c005960 comp rx=0 tx=0).stop 2024-05-27T07:22:22.803 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:22:22.808+0000 7fa186350b80 1 -- 172.21.15.52:0/3137368447 shutdown_connections 2024-05-27T07:22:22.803 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:22:22.808+0000 7fa186350b80 1 --2- 172.21.15.52:0/3137368447 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x559388edd090 0x559388eec2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:22:22.803 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:22:22.808+0000 7fa186350b80 1 --2- 172.21.15.52:0/3137368447 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x559388d54df0 0x559388efb700 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:22:22.803 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:22:22.808+0000 7fa186350b80 1 --2- 172.21.15.52:0/3137368447 >> [v2:172.21.15.105:6838/500348151,v1:172.21.15.105:6839/500348151] conn(0x559388f5e150 0x559388f5f5c0 crc :-1 s=READY pgs=14 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:22:22.803 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:22:22.808+0000 7fa186350b80 1 --2- 172.21.15.52:0/3137368447 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x559388ed7340 0x559388eebdb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:22:22.803 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:22:22.808+0000 7fa158ff9700 0 client.24568 ms_handle_reset on v2:172.21.15.105:6838/500348151 2024-05-27T07:22:22.803 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:22:22.808+0000 7fa186350b80 1 -- 172.21.15.52:0/3137368447 >> 172.21.15.52:0/3137368447 conn(0x559388dafaa0 msgr2=0x559388ed7750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:22:22.803 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:22:22.808+0000 7fa186350b80 1 -- 172.21.15.52:0/3137368447 shutdown_connections 2024-05-27T07:22:22.803 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:22:22.808+0000 7fa186350b80 1 -- 172.21.15.52:0/3137368447 wait complete. 2024-05-27T07:22:22.804 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:22:22.808+0000 7fa186350b80 20 client.24568 trim_cache size 0 max 16384 2024-05-27T07:22:22.804 INFO:teuthology.orchestra.run.smithi052.stdout:{ 2024-05-27T07:22:22.804 INFO:teuthology.orchestra.run.smithi052.stdout: "cluster_fsid": "73df2636-1bf8-11ef-bc9b-c7b262605968", 2024-05-27T07:22:22.804 INFO:teuthology.orchestra.run.smithi052.stdout: "whoami": 0, 2024-05-27T07:22:22.804 INFO:teuthology.orchestra.run.smithi052.stdout: "id": 24527, 2024-05-27T07:22:22.804 INFO:teuthology.orchestra.run.smithi052.stdout: "want_state": "up:active", 2024-05-27T07:22:22.804 INFO:teuthology.orchestra.run.smithi052.stdout: "state": "up:active", 2024-05-27T07:22:22.804 INFO:teuthology.orchestra.run.smithi052.stdout: "fs_name": "cephfs", 2024-05-27T07:22:22.804 INFO:teuthology.orchestra.run.smithi052.stdout: "rank_uptime": 32.233639752999999, 2024-05-27T07:22:22.805 INFO:teuthology.orchestra.run.smithi052.stdout: "mdsmap_epoch": 17, 2024-05-27T07:22:22.805 INFO:teuthology.orchestra.run.smithi052.stdout: "osdmap_epoch": 86, 2024-05-27T07:22:22.805 INFO:teuthology.orchestra.run.smithi052.stdout: "osdmap_epoch_barrier": 85, 2024-05-27T07:22:22.805 INFO:teuthology.orchestra.run.smithi052.stdout: "uptime": 46.287818369999997 2024-05-27T07:22:22.805 INFO:teuthology.orchestra.run.smithi052.stdout:} 2024-05-27T07:22:22.806 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:22:22.809+0000 7fa17793b700 1 -- 172.21.15.52:0/1745825753 >> [v2:172.21.15.52:6800/3513075805,v1:172.21.15.52:6801/3513075805] conn(0x7fa15c0b2880 msgr2=0x7fa15c0b4d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:22:22.806 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:22:22.809+0000 7fa17793b700 1 --2- 172.21.15.52:0/1745825753 >> [v2:172.21.15.52:6800/3513075805,v1:172.21.15.52:6801/3513075805] conn(0x7fa15c0b2880 0x7fa15c0b4d90 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7fa17006d110 tx=0x7fa16800af20 comp rx=0 tx=0).stop 2024-05-27T07:22:22.806 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:22:22.809+0000 7fa17793b700 1 -- 172.21.15.52:0/1745825753 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fa170107e00 msgr2=0x7fa17019c220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:22:22.806 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:22:22.809+0000 7fa17793b700 1 --2- 172.21.15.52:0/1745825753 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fa170107e00 0x7fa17019c220 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7fa160006010 tx=0x7fa1600047b0 comp rx=0 tx=0).stop 2024-05-27T07:22:22.806 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:22:22.809+0000 7fa17793b700 1 -- 172.21.15.52:0/1745825753 shutdown_connections 2024-05-27T07:22:22.806 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:22:22.809+0000 7fa17793b700 1 --2- 172.21.15.52:0/1745825753 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fa170108750 0x7fa17019c780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:22:22.806 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:22:22.809+0000 7fa17793b700 1 --2- 172.21.15.52:0/1745825753 >> [v2:172.21.15.52:6800/3513075805,v1:172.21.15.52:6801/3513075805] conn(0x7fa15c0b2880 0x7fa15c0b4d90 unknown :-1 s=CLOSED pgs=153 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:22:22.806 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:22:22.809+0000 7fa17793b700 1 --2- 172.21.15.52:0/1745825753 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fa1701034d0 0x7fa17019bce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:22:22.806 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:22:22.809+0000 7fa17793b700 1 --2- 172.21.15.52:0/1745825753 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fa170107e00 0x7fa17019c220 unknown :-1 s=CLOSED pgs=205 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:22:22.806 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:22:22.809+0000 7fa17793b700 1 -- 172.21.15.52:0/1745825753 >> 172.21.15.52:0/1745825753 conn(0x7fa1700fef20 msgr2=0x7fa1701096b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:22:22.806 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:22:22.809+0000 7fa17793b700 1 -- 172.21.15.52:0/1745825753 shutdown_connections 2024-05-27T07:22:22.806 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:22:22.809+0000 7fa17793b700 1 -- 172.21.15.52:0/1745825753 wait complete. 2024-05-27T07:22:23.102 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:22:22 smithi173 ceph-mon[37000]: pgmap v312: 129 pgs: 129 active+clean; 579 KiB data, 356 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-27T07:22:23.190 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:22:22 smithi105 ceph-mon[35977]: pgmap v312: 129 pgs: 129 active+clean; 579 KiB data, 356 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-27T07:22:23.229 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:22:22 smithi052 ceph-mon[31955]: pgmap v312: 129 pgs: 129 active+clean; 579 KiB data, 356 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-27T07:22:23.486 INFO:tasks.cephfs.mount:cephfs_mntpt = None 2024-05-27T07:22:23.486 INFO:tasks.cephfs.mount:hostfs_mntpt = /home/ubuntu/cephtest/mnt.admin 2024-05-27T07:22:23.487 DEBUG:teuthology.orchestra.run.smithi052:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:38f9d900c83c63799fbdbe61acc9a11b0d3554a6 shell --fsid 73df2636-1bf8-11ef-bc9b-c7b262605968 -- ceph fs ls 2024-05-27T07:22:23.742 INFO:teuthology.orchestra.run.smithi052.stderr:Inferring config /var/lib/ceph/73df2636-1bf8-11ef-bc9b-c7b262605968/mon.a/config 2024-05-27T07:22:24.849 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:22:24 smithi052 ceph-mon[31955]: pgmap v313: 129 pgs: 129 active+clean; 579 KiB data, 356 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-27T07:22:25.102 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:22:24 smithi173 ceph-mon[37000]: pgmap v313: 129 pgs: 129 active+clean; 579 KiB data, 356 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-27T07:22:25.190 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:22:24 smithi105 ceph-mon[35977]: pgmap v313: 129 pgs: 129 active+clean; 579 KiB data, 356 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-27T07:22:25.440 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:22:25.450+0000 7f87908c3700 1 -- 172.21.15.52:0/1009231810 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f878c108ef0 msgr2=0x7f878c109300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:22:25.440 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:22:25.450+0000 7f87908c3700 1 --2- 172.21.15.52:0/1009231810 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f878c108ef0 0x7f878c109300 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7f87740099c0 tx=0x7f8774009cd0 comp rx=0 tx=0).stop 2024-05-27T07:22:25.440 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:22:25.450+0000 7f87908c3700 1 -- 172.21.15.52:0/1009231810 shutdown_connections 2024-05-27T07:22:25.441 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:22:25.450+0000 7f87908c3700 1 --2- 172.21.15.52:0/1009231810 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f878c109840 0x7f878c1111a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:22:25.441 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:22:25.450+0000 7f87908c3700 1 --2- 172.21.15.52:0/1009231810 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f878c1045c0 0x7f878c1049d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:22:25.441 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:22:25.450+0000 7f87908c3700 1 --2- 172.21.15.52:0/1009231810 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f878c108ef0 0x7f878c109300 unknown :-1 s=CLOSED pgs=207 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:22:25.441 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:22:25.450+0000 7f87908c3700 1 -- 172.21.15.52:0/1009231810 >> 172.21.15.52:0/1009231810 conn(0x7f878c0ffe90 msgr2=0x7f878c1022e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:22:25.441 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:22:25.450+0000 7f87908c3700 1 -- 172.21.15.52:0/1009231810 shutdown_connections 2024-05-27T07:22:25.441 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:22:25.450+0000 7f87908c3700 1 -- 172.21.15.52:0/1009231810 wait complete. 2024-05-27T07:22:25.442 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:22:25.451+0000 7f87908c3700 1 Processor -- start 2024-05-27T07:22:25.443 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:22:25.451+0000 7f87908c3700 1 -- start start 2024-05-27T07:22:25.443 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:22:25.451+0000 7f87908c3700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f878c1045c0 0x7f878c076bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:22:25.443 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:22:25.451+0000 7f87908c3700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f878c108ef0 0x7f878c077100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:22:25.443 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:22:25.451+0000 7f87908c3700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f878c109840 0x7f878c077640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:22:25.443 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:22:25.451+0000 7f87908c3700 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x7f878c114510 con 0x7f878c109840 2024-05-27T07:22:25.443 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:22:25.451+0000 7f87908c3700 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f878c114390 con 0x7f878c108ef0 2024-05-27T07:22:25.443 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:22:25.451+0000 7f87908c3700 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f878c114690 con 0x7f878c1045c0 2024-05-27T07:22:25.443 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:22:25.451+0000 7f878ad9d700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f878c109840 0x7f878c077640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:22:25.443 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:22:25.451+0000 7f878ad9d700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f878c109840 0x7f878c077640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.52:60526/0 (socket says 172.21.15.52:60526) 2024-05-27T07:22:25.443 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:22:25.451+0000 7f878ad9d700 1 -- 172.21.15.52:0/3120972168 learned_addr learned my addr 172.21.15.52:0/3120972168 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-05-27T07:22:25.443 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:22:25.452+0000 7f878a59c700 1 --2- 172.21.15.52:0/3120972168 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f878c1045c0 0x7f878c076bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:22:25.443 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:22:25.452+0000 7f8789d9b700 1 --2- 172.21.15.52:0/3120972168 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f878c108ef0 0x7f878c077100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:22:25.443 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:22:25.452+0000 7f878ad9d700 1 -- 172.21.15.52:0/3120972168 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f878c1045c0 msgr2=0x7f878c076bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:22:25.444 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:22:25.452+0000 7f878ad9d700 1 --2- 172.21.15.52:0/3120972168 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f878c1045c0 0x7f878c076bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:22:25.444 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:22:25.452+0000 7f878ad9d700 1 -- 172.21.15.52:0/3120972168 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f878c108ef0 msgr2=0x7f878c077100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:22:25.444 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:22:25.452+0000 7f878ad9d700 1 --2- 172.21.15.52:0/3120972168 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f878c108ef0 0x7f878c077100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:22:25.444 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:22:25.452+0000 7f878ad9d700 1 -- 172.21.15.52:0/3120972168 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f878c1a9220 con 0x7f878c109840 2024-05-27T07:22:25.444 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:22:25.452+0000 7f878ad9d700 1 --2- 172.21.15.52:0/3120972168 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f878c109840 0x7f878c077640 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7f8784007d70 tx=0x7f878400a340 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:22:25.444 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:22:25.452+0000 7f87837fe700 1 -- 172.21.15.52:0/3120972168 <== mon.0 v2:172.21.15.52:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8784016020 con 0x7f878c109840 2024-05-27T07:22:25.444 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:22:25.452+0000 7f87908c3700 1 -- 172.21.15.52:0/3120972168 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f878c1a9510 con 0x7f878c109840 2024-05-27T07:22:25.444 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:22:25.452+0000 7f87837fe700 1 -- 172.21.15.52:0/3120972168 <== mon.0 v2:172.21.15.52:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f878400ad40 con 0x7f878c109840 2024-05-27T07:22:25.444 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:22:25.452+0000 7f87837fe700 1 -- 172.21.15.52:0/3120972168 <== mon.0 v2:172.21.15.52:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f87840115e0 con 0x7f878c109840 2024-05-27T07:22:25.444 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:22:25.452+0000 7f87908c3700 1 -- 172.21.15.52:0/3120972168 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f878c1a9a20 con 0x7f878c109840 2024-05-27T07:22:25.445 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:22:25.453+0000 7f87908c3700 1 -- 172.21.15.52:0/3120972168 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f876c005330 con 0x7f878c109840 2024-05-27T07:22:25.445 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:22:25.455+0000 7f87837fe700 1 -- 172.21.15.52:0/3120972168 <== mon.0 v2:172.21.15.52:3300/0 4 ==== mgrmap(e 16) ==== 149652+0+0 (secure 0 0 0) 0x7f8784019020 con 0x7f878c109840 2024-05-27T07:22:25.446 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:22:25.456+0000 7f87837fe700 1 --2- 172.21.15.52:0/3120972168 >> [v2:172.21.15.52:6800/3513075805,v1:172.21.15.52:6801/3513075805] conn(0x7f87780b2880 0x7f87780b4d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:22:25.446 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:22:25.456+0000 7f878a59c700 1 --2- 172.21.15.52:0/3120972168 >> [v2:172.21.15.52:6800/3513075805,v1:172.21.15.52:6801/3513075805] conn(0x7f87780b2880 0x7f87780b4d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:22:25.446 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:22:25.456+0000 7f87837fe700 1 -- 172.21.15.52:0/3120972168 <== mon.0 v2:172.21.15.52:3300/0 5 ==== osd_map(86..86 src has 1..86) ==== 8802+0+0 (secure 0 0 0) 0x7f87840e27b0 con 0x7f878c109840 2024-05-27T07:22:25.446 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:22:25.457+0000 7f878a59c700 1 --2- 172.21.15.52:0/3120972168 >> [v2:172.21.15.52:6800/3513075805,v1:172.21.15.52:6801/3513075805] conn(0x7f87780b2880 0x7f87780b4d90 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7f877c006fd0 tx=0x7f877c008040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:22:25.450 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:22:25.460+0000 7f87837fe700 1 -- 172.21.15.52:0/3120972168 <== mon.0 v2:172.21.15.52:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191963 (secure 0 0 0) 0x7f8784020020 con 0x7f878c109840 2024-05-27T07:22:25.714 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:22:25.724+0000 7f87908c3700 1 -- 172.21.15.52:0/3120972168 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_command({"prefix": "fs ls"} v 0) -- 0x7f876c0058f0 con 0x7f878c109840 2024-05-27T07:22:25.714 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:22:25.725+0000 7f87837fe700 1 -- 172.21.15.52:0/3120972168 <== mon.0 v2:172.21.15.52:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v17) ==== 53+0+88 (secure 0 0 0) 0x7f8784012b40 con 0x7f878c109840 2024-05-27T07:22:25.715 INFO:teuthology.orchestra.run.smithi052.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data cephfs_data_ec ] 2024-05-27T07:22:25.717 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:22:25.727+0000 7f87908c3700 1 -- 172.21.15.52:0/3120972168 >> [v2:172.21.15.52:6800/3513075805,v1:172.21.15.52:6801/3513075805] conn(0x7f87780b2880 msgr2=0x7f87780b4d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:22:25.717 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:22:25.727+0000 7f87908c3700 1 --2- 172.21.15.52:0/3120972168 >> [v2:172.21.15.52:6800/3513075805,v1:172.21.15.52:6801/3513075805] conn(0x7f87780b2880 0x7f87780b4d90 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7f877c006fd0 tx=0x7f877c008040 comp rx=0 tx=0).stop 2024-05-27T07:22:25.717 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:22:25.727+0000 7f87908c3700 1 -- 172.21.15.52:0/3120972168 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f878c109840 msgr2=0x7f878c077640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:22:25.717 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:22:25.727+0000 7f87908c3700 1 --2- 172.21.15.52:0/3120972168 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f878c109840 0x7f878c077640 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7f8784007d70 tx=0x7f878400a340 comp rx=0 tx=0).stop 2024-05-27T07:22:25.717 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:22:25.727+0000 7f87908c3700 1 -- 172.21.15.52:0/3120972168 shutdown_connections 2024-05-27T07:22:25.717 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:22:25.727+0000 7f87908c3700 1 --2- 172.21.15.52:0/3120972168 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f878c109840 0x7f878c077640 unknown :-1 s=CLOSED pgs=208 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:22:25.717 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:22:25.727+0000 7f87908c3700 1 --2- 172.21.15.52:0/3120972168 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f878c1045c0 0x7f878c076bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:22:25.717 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:22:25.727+0000 7f87908c3700 1 --2- 172.21.15.52:0/3120972168 >> [v2:172.21.15.52:6800/3513075805,v1:172.21.15.52:6801/3513075805] conn(0x7f87780b2880 0x7f87780b4d90 unknown :-1 s=CLOSED pgs=154 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:22:25.717 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:22:25.727+0000 7f87908c3700 1 --2- 172.21.15.52:0/3120972168 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f878c108ef0 0x7f878c077100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:22:25.717 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:22:25.727+0000 7f87908c3700 1 -- 172.21.15.52:0/3120972168 >> 172.21.15.52:0/3120972168 conn(0x7f878c0ffe90 msgr2=0x7f878c10a7a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:22:25.717 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:22:25.727+0000 7f87908c3700 1 -- 172.21.15.52:0/3120972168 shutdown_connections 2024-05-27T07:22:25.718 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:22:25.727+0000 7f87908c3700 1 -- 172.21.15.52:0/3120972168 wait complete. 2024-05-27T07:22:25.979 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:22:25 smithi052 ceph-mon[31955]: from='client.? 172.21.15.52:0/3120972168' entity='client.admin' cmd=[{"prefix": "fs ls"}]: dispatch 2024-05-27T07:22:26.102 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:22:25 smithi173 ceph-mon[37000]: from='client.? 172.21.15.52:0/3120972168' entity='client.admin' cmd=[{"prefix": "fs ls"}]: dispatch 2024-05-27T07:22:26.190 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:22:25 smithi105 ceph-mon[35977]: from='client.? 172.21.15.52:0/3120972168' entity='client.admin' cmd=[{"prefix": "fs ls"}]: dispatch 2024-05-27T07:22:26.385 INFO:tasks.cephfs.mount:Mounting Ceph FS cephfs; just confirmed its presence on cluster 2024-05-27T07:22:26.385 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-05-27T07:22:26.385 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi052.front.sepia.ceph.com 2024-05-27T07:22:26.385 INFO:tasks.cephfs.mount:self.client.name = client.admin 2024-05-27T07:22:26.385 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.admin 2024-05-27T07:22:26.386 INFO:tasks.cephfs.mount:self.cephfs_name = cephfs 2024-05-27T07:22:26.386 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2024-05-27T07:22:26.386 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-05-27T07:22:26.386 INFO:tasks.cephfs.mount:Setting the 'None' netns for '/home/ubuntu/cephtest/mnt.admin' 2024-05-27T07:22:26.386 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-27T07:22:26.386 DEBUG:teuthology.orchestra.run.smithi052:> ip addr 2024-05-27T07:22:26.405 INFO:teuthology.orchestra.run.smithi052.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-05-27T07:22:26.405 INFO:teuthology.orchestra.run.smithi052.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-05-27T07:22:26.405 INFO:teuthology.orchestra.run.smithi052.stdout: inet 127.0.0.1/8 scope host lo 2024-05-27T07:22:26.405 INFO:teuthology.orchestra.run.smithi052.stdout: valid_lft forever preferred_lft forever 2024-05-27T07:22:26.405 INFO:teuthology.orchestra.run.smithi052.stdout: inet6 ::1/128 scope host 2024-05-27T07:22:26.405 INFO:teuthology.orchestra.run.smithi052.stdout: valid_lft forever preferred_lft forever 2024-05-27T07:22:26.405 INFO:teuthology.orchestra.run.smithi052.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-05-27T07:22:26.406 INFO:teuthology.orchestra.run.smithi052.stdout: link/ether 0c:c4:7a:6b:fd:d2 brd ff:ff:ff:ff:ff:ff 2024-05-27T07:22:26.406 INFO:teuthology.orchestra.run.smithi052.stdout: altname enp7s0f0 2024-05-27T07:22:26.406 INFO:teuthology.orchestra.run.smithi052.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-05-27T07:22:26.406 INFO:teuthology.orchestra.run.smithi052.stdout: link/ether 0c:c4:7a:6b:fd:d3 brd ff:ff:ff:ff:ff:ff 2024-05-27T07:22:26.406 INFO:teuthology.orchestra.run.smithi052.stdout: altname enp7s0f1 2024-05-27T07:22:26.406 INFO:teuthology.orchestra.run.smithi052.stdout:4: ens1f0: mtu 1500 qdisc mq state UP group default qlen 1000 2024-05-27T07:22:26.407 INFO:teuthology.orchestra.run.smithi052.stdout: link/ether 0c:c4:7a:bd:15:de brd ff:ff:ff:ff:ff:ff 2024-05-27T07:22:26.407 INFO:teuthology.orchestra.run.smithi052.stdout: altname enp1s0f0 2024-05-27T07:22:26.407 INFO:teuthology.orchestra.run.smithi052.stdout: inet 172.21.15.52/20 brd 172.21.15.255 scope global dynamic noprefixroute ens1f0 2024-05-27T07:22:26.407 INFO:teuthology.orchestra.run.smithi052.stdout: valid_lft 42170sec preferred_lft 42170sec 2024-05-27T07:22:26.407 INFO:teuthology.orchestra.run.smithi052.stdout: inet6 fe80::ec4:7aff:febd:15de/64 scope link noprefixroute 2024-05-27T07:22:26.407 INFO:teuthology.orchestra.run.smithi052.stdout: valid_lft forever preferred_lft forever 2024-05-27T07:22:26.407 INFO:teuthology.orchestra.run.smithi052.stdout:5: ens1f1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-05-27T07:22:26.407 INFO:teuthology.orchestra.run.smithi052.stdout: link/ether 0c:c4:7a:bd:15:df brd ff:ff:ff:ff:ff:ff 2024-05-27T07:22:26.407 INFO:teuthology.orchestra.run.smithi052.stdout: altname enp1s0f1 2024-05-27T07:22:26.408 INFO:tasks.cephfs.mount:Setuping the 'ceph-brx' with 192.168.255.254/16 2024-05-27T07:22:26.409 DEBUG:teuthology.orchestra.run.smithi052:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-05-27T07:22:26.409 DEBUG:teuthology.orchestra.run.smithi052:> set -e 2024-05-27T07:22:26.409 DEBUG:teuthology.orchestra.run.smithi052:> sudo ip link add name ceph-brx type bridge 2024-05-27T07:22:26.409 DEBUG:teuthology.orchestra.run.smithi052:> sudo ip addr flush dev ceph-brx 2024-05-27T07:22:26.409 DEBUG:teuthology.orchestra.run.smithi052:> sudo ip link set ceph-brx up 2024-05-27T07:22:26.410 DEBUG:teuthology.orchestra.run.smithi052:> sudo ip addr add 192.168.255.254/16 brd 192.168.255.255 dev ceph-brx 2024-05-27T07:22:26.410 DEBUG:teuthology.orchestra.run.smithi052:> ') 2024-05-27T07:22:26.499 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:22:26 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-05-27T07:22:26.630 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:22:26 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-05-27T07:22:26.638 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-27T07:22:26.638 DEBUG:teuthology.orchestra.run.smithi052:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-05-27T07:22:26.664 INFO:teuthology.orchestra.run.smithi052.stdout:1 2024-05-27T07:22:26.666 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-27T07:22:26.666 DEBUG:teuthology.orchestra.run.smithi052:> route 2024-05-27T07:22:26.736 INFO:teuthology.orchestra.run.smithi052.stdout:Kernel IP routing table 2024-05-27T07:22:26.737 INFO:teuthology.orchestra.run.smithi052.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-05-27T07:22:26.737 INFO:teuthology.orchestra.run.smithi052.stdout:default _gateway 0.0.0.0 UG 100 0 0 ens1f0 2024-05-27T07:22:26.737 INFO:teuthology.orchestra.run.smithi052.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 ens1f0 2024-05-27T07:22:26.737 INFO:teuthology.orchestra.run.smithi052.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-05-27T07:22:26.737 DEBUG:teuthology.orchestra.run.smithi052:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-05-27T07:22:26.737 DEBUG:teuthology.orchestra.run.smithi052:> set -e 2024-05-27T07:22:26.737 DEBUG:teuthology.orchestra.run.smithi052:> sudo iptables -A FORWARD -o ens1f0 -i ceph-brx -j ACCEPT 2024-05-27T07:22:26.738 DEBUG:teuthology.orchestra.run.smithi052:> sudo iptables -A FORWARD -i ens1f0 -o ceph-brx -j ACCEPT 2024-05-27T07:22:26.738 DEBUG:teuthology.orchestra.run.smithi052:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o ens1f0 -j MASQUERADE 2024-05-27T07:22:26.738 DEBUG:teuthology.orchestra.run.smithi052:> ') 2024-05-27T07:22:26.828 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:22:26 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-05-27T07:22:26.941 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:22:26 smithi052 ceph-mon[31955]: pgmap v314: 129 pgs: 129 active+clean; 579 KiB data, 356 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-27T07:22:27.037 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:22:27 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-05-27T07:22:27.044 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-27T07:22:27.044 DEBUG:teuthology.orchestra.run.smithi052:> ip netns list 2024-05-27T07:22:27.060 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-27T07:22:27.060 DEBUG:teuthology.orchestra.run.smithi052:> ip netns list-id 2024-05-27T07:22:27.102 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:22:26 smithi173 ceph-mon[37000]: pgmap v314: 129 pgs: 129 active+clean; 579 KiB data, 356 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-27T07:22:27.117 DEBUG:teuthology.orchestra.run.smithi052:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-05-27T07:22:27.117 DEBUG:teuthology.orchestra.run.smithi052:> set -e 2024-05-27T07:22:27.117 DEBUG:teuthology.orchestra.run.smithi052:> sudo ip netns add ceph-ns--home-ubuntu-cephtest-mnt.admin 2024-05-27T07:22:27.117 DEBUG:teuthology.orchestra.run.smithi052:> sudo ip netns set ceph-ns--home-ubuntu-cephtest-mnt.admin 0 2024-05-27T07:22:27.117 DEBUG:teuthology.orchestra.run.smithi052:> ') 2024-05-27T07:22:27.190 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:22:26 smithi105 ceph-mon[35977]: pgmap v314: 129 pgs: 129 active+clean; 579 KiB data, 356 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-27T07:22:27.207 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:22:27 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-05-27T07:22:27.232 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:22:27 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-05-27T07:22:27.239 INFO:tasks.cephfs.mount:Setuping the netns 'ceph-ns--home-ubuntu-cephtest-mnt.admin' with 192.168.0.1/16 2024-05-27T07:22:27.239 DEBUG:teuthology.orchestra.run.smithi052:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-05-27T07:22:27.239 DEBUG:teuthology.orchestra.run.smithi052:> set -e 2024-05-27T07:22:27.239 DEBUG:teuthology.orchestra.run.smithi052:> sudo ip link add veth0 netns ceph-ns--home-ubuntu-cephtest-mnt.admin type veth peer name brx.0 2024-05-27T07:22:27.239 DEBUG:teuthology.orchestra.run.smithi052:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.admin ip addr add 192.168.0.1/16 brd 192.168.255.255 dev veth0 2024-05-27T07:22:27.239 DEBUG:teuthology.orchestra.run.smithi052:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.admin ip link set veth0 up 2024-05-27T07:22:27.239 DEBUG:teuthology.orchestra.run.smithi052:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.admin ip link set lo up 2024-05-27T07:22:27.239 DEBUG:teuthology.orchestra.run.smithi052:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.admin ip route add default via 192.168.255.254 2024-05-27T07:22:27.239 DEBUG:teuthology.orchestra.run.smithi052:> ') 2024-05-27T07:22:27.290 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:22:27 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-05-27T07:22:27.389 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:22:27 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-05-27T07:22:27.396 DEBUG:teuthology.orchestra.run.smithi052:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-05-27T07:22:27.396 DEBUG:teuthology.orchestra.run.smithi052:> set -e 2024-05-27T07:22:27.396 DEBUG:teuthology.orchestra.run.smithi052:> sudo ip link set brx.0 up 2024-05-27T07:22:27.396 DEBUG:teuthology.orchestra.run.smithi052:> sudo ip link set dev brx.0 master ceph-brx 2024-05-27T07:22:27.396 DEBUG:teuthology.orchestra.run.smithi052:> ') 2024-05-27T07:22:27.447 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:22:27 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-05-27T07:22:27.471 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:22:27 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-05-27T07:22:27.478 INFO:tasks.cephfs.fuse_mount:Client client.admin config is {} 2024-05-27T07:22:27.478 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-05-27T07:22:27.478 DEBUG:teuthology.orchestra.run.smithi052:> mkdir -p -v /home/ubuntu/cephtest/mnt.admin 2024-05-27T07:22:27.494 INFO:teuthology.orchestra.run.smithi052.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.admin' 2024-05-27T07:22:27.494 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-05-27T07:22:27.494 DEBUG:teuthology.orchestra.run.smithi052:> chmod 0000 /home/ubuntu/cephtest/mnt.admin 2024-05-27T07:22:27.551 DEBUG:teuthology.orchestra.run.smithi052:> sudo modprobe fuse 2024-05-27T07:22:27.623 DEBUG:teuthology.orchestra.run.smithi052:> cat /proc/self/mounts | awk '{print $2}' 2024-05-27T07:22:27.676 INFO:teuthology.orchestra.run.smithi052.stdout:/proc 2024-05-27T07:22:27.676 INFO:teuthology.orchestra.run.smithi052.stdout:/sys 2024-05-27T07:22:27.676 INFO:teuthology.orchestra.run.smithi052.stdout:/dev 2024-05-27T07:22:27.676 INFO:teuthology.orchestra.run.smithi052.stdout:/sys/kernel/security 2024-05-27T07:22:27.676 INFO:teuthology.orchestra.run.smithi052.stdout:/dev/shm 2024-05-27T07:22:27.676 INFO:teuthology.orchestra.run.smithi052.stdout:/dev/pts 2024-05-27T07:22:27.676 INFO:teuthology.orchestra.run.smithi052.stdout:/run 2024-05-27T07:22:27.676 INFO:teuthology.orchestra.run.smithi052.stdout:/sys/fs/cgroup 2024-05-27T07:22:27.676 INFO:teuthology.orchestra.run.smithi052.stdout:/sys/fs/pstore 2024-05-27T07:22:27.677 INFO:teuthology.orchestra.run.smithi052.stdout:/sys/fs/bpf 2024-05-27T07:22:27.677 INFO:teuthology.orchestra.run.smithi052.stdout:/ 2024-05-27T07:22:27.677 INFO:teuthology.orchestra.run.smithi052.stdout:/sys/fs/selinux 2024-05-27T07:22:27.677 INFO:teuthology.orchestra.run.smithi052.stdout:/proc/sys/fs/binfmt_misc 2024-05-27T07:22:27.677 INFO:teuthology.orchestra.run.smithi052.stdout:/dev/hugepages 2024-05-27T07:22:27.677 INFO:teuthology.orchestra.run.smithi052.stdout:/dev/mqueue 2024-05-27T07:22:27.677 INFO:teuthology.orchestra.run.smithi052.stdout:/sys/kernel/debug 2024-05-27T07:22:27.677 INFO:teuthology.orchestra.run.smithi052.stdout:/sys/kernel/tracing 2024-05-27T07:22:27.677 INFO:teuthology.orchestra.run.smithi052.stdout:/sys/fs/fuse/connections 2024-05-27T07:22:27.677 INFO:teuthology.orchestra.run.smithi052.stdout:/sys/kernel/config 2024-05-27T07:22:27.677 INFO:teuthology.orchestra.run.smithi052.stdout:/run/credentials/systemd-sysctl.service 2024-05-27T07:22:27.677 INFO:teuthology.orchestra.run.smithi052.stdout:/run/credentials/systemd-tmpfiles-setup-dev.service 2024-05-27T07:22:27.678 INFO:teuthology.orchestra.run.smithi052.stdout:/run/credentials/systemd-tmpfiles-setup.service 2024-05-27T07:22:27.678 INFO:teuthology.orchestra.run.smithi052.stdout:/var/lib/nfs/rpc_pipefs 2024-05-27T07:22:27.678 INFO:teuthology.orchestra.run.smithi052.stdout:/run/user/1000 2024-05-27T07:22:27.678 INFO:teuthology.orchestra.run.smithi052.stdout:/var/lib/ceph 2024-05-27T07:22:27.678 INFO:teuthology.orchestra.run.smithi052.stdout:/var/lib/containers/storage/overlay 2024-05-27T07:22:27.678 INFO:teuthology.orchestra.run.smithi052.stdout:/var/lib/containers/storage/overlay/339b609a774606614a4357911c7efdf1485294f41086dc57156ce1506c859e82/merged 2024-05-27T07:22:27.678 INFO:teuthology.orchestra.run.smithi052.stdout:/var/lib/containers/storage/overlay/156afeab7561c4315f8c0269174cc6d7fc51ff200896f8ecdd8453475afa51f5/merged 2024-05-27T07:22:27.678 INFO:teuthology.orchestra.run.smithi052.stdout:/run/user/0 2024-05-27T07:22:27.678 INFO:teuthology.orchestra.run.smithi052.stdout:/proc/sys/fs/binfmt_misc 2024-05-27T07:22:27.678 INFO:teuthology.orchestra.run.smithi052.stdout:/var/lib/containers/storage/overlay/57343f17079f73a0c532f85d4aa22f6ba25f238945cd9874c632b74b3232ec25/merged 2024-05-27T07:22:27.678 INFO:teuthology.orchestra.run.smithi052.stdout:/var/lib/containers/storage/overlay/8032bf106ed313012e44c3d322c9802484683cfca36223e8fd3f0691555a7f4d/merged 2024-05-27T07:22:27.678 INFO:teuthology.orchestra.run.smithi052.stdout:/var/lib/containers/storage/overlay/0df7e3d814ba0a1e066e632896b4b3d19bc0a7b31de4a6efbe598517d3d4b3ee/merged 2024-05-27T07:22:27.679 INFO:teuthology.orchestra.run.smithi052.stdout:/var/lib/containers/storage/overlay/eb022041ddf436dc94838d9f6739734727769c7ddd3193bcc904b0f0d41551fd/merged 2024-05-27T07:22:27.679 INFO:teuthology.orchestra.run.smithi052.stdout:/var/lib/containers/storage/overlay/cf9b7bbfe621da9b2ded22d846b075a387466a2407402eeef036b38c7cdf5f9b/merged 2024-05-27T07:22:27.679 INFO:teuthology.orchestra.run.smithi052.stdout:/var/lib/containers/storage/overlay/b91f91b35c929c0ad0e1b2ed644718b1fe511cf643ff91b3ef9788305b5c35ae/merged 2024-05-27T07:22:27.679 INFO:teuthology.orchestra.run.smithi052.stdout:/var/lib/containers/storage/overlay/c61b48a893b2f74d05d41cb09909654b29f7079e8b1da82f2515c15858f8da97/merged 2024-05-27T07:22:27.679 INFO:teuthology.orchestra.run.smithi052.stdout:/var/lib/containers/storage/overlay/053ea3d29cab1736b9b72846a69904185921866998ecb426b6ddfb230e1a0c00/merged 2024-05-27T07:22:27.679 INFO:teuthology.orchestra.run.smithi052.stdout:/run/netns 2024-05-27T07:22:27.679 INFO:teuthology.orchestra.run.smithi052.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.admin 2024-05-27T07:22:27.679 INFO:teuthology.orchestra.run.smithi052.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.admin 2024-05-27T07:22:27.679 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-27T07:22:27.679 DEBUG:teuthology.orchestra.run.smithi052:> ls /sys/fs/fuse/connections 2024-05-27T07:22:27.732 INFO:tasks.cephfs.fuse_mount:Pre-mount connections: [] 2024-05-27T07:22:27.732 DEBUG:teuthology.orchestra.run.smithi052:> (cd /home/ubuntu/cephtest && exec sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.admin sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --admin-socket '/var/run/ceph/$cluster-$name.$pid.asok' /home/ubuntu/cephtest/mnt.admin --id admin --client_fs=cephfs) 2024-05-27T07:22:27.774 DEBUG:teuthology.orchestra.run.smithi052:> sudo modprobe fuse 2024-05-27T07:22:27.800 DEBUG:teuthology.orchestra.run.smithi052:> cat /proc/self/mounts | awk '{print $2}' 2024-05-27T07:22:27.858 INFO:teuthology.orchestra.run.smithi052.stdout:/proc 2024-05-27T07:22:27.858 INFO:teuthology.orchestra.run.smithi052.stdout:/sys 2024-05-27T07:22:27.858 INFO:teuthology.orchestra.run.smithi052.stdout:/dev 2024-05-27T07:22:27.858 INFO:teuthology.orchestra.run.smithi052.stdout:/sys/kernel/security 2024-05-27T07:22:27.858 INFO:teuthology.orchestra.run.smithi052.stdout:/dev/shm 2024-05-27T07:22:27.858 INFO:teuthology.orchestra.run.smithi052.stdout:/dev/pts 2024-05-27T07:22:27.858 INFO:teuthology.orchestra.run.smithi052.stdout:/run 2024-05-27T07:22:27.858 INFO:teuthology.orchestra.run.smithi052.stdout:/sys/fs/cgroup 2024-05-27T07:22:27.858 INFO:teuthology.orchestra.run.smithi052.stdout:/sys/fs/pstore 2024-05-27T07:22:27.858 INFO:teuthology.orchestra.run.smithi052.stdout:/sys/fs/bpf 2024-05-27T07:22:27.859 INFO:teuthology.orchestra.run.smithi052.stdout:/ 2024-05-27T07:22:27.859 INFO:teuthology.orchestra.run.smithi052.stdout:/sys/fs/selinux 2024-05-27T07:22:27.859 INFO:teuthology.orchestra.run.smithi052.stdout:/proc/sys/fs/binfmt_misc 2024-05-27T07:22:27.859 INFO:teuthology.orchestra.run.smithi052.stdout:/dev/hugepages 2024-05-27T07:22:27.859 INFO:teuthology.orchestra.run.smithi052.stdout:/dev/mqueue 2024-05-27T07:22:27.859 INFO:teuthology.orchestra.run.smithi052.stdout:/sys/kernel/debug 2024-05-27T07:22:27.859 INFO:teuthology.orchestra.run.smithi052.stdout:/sys/kernel/tracing 2024-05-27T07:22:27.859 INFO:teuthology.orchestra.run.smithi052.stdout:/sys/fs/fuse/connections 2024-05-27T07:22:27.859 INFO:teuthology.orchestra.run.smithi052.stdout:/sys/kernel/config 2024-05-27T07:22:27.859 INFO:teuthology.orchestra.run.smithi052.stdout:/run/credentials/systemd-sysctl.service 2024-05-27T07:22:27.859 INFO:teuthology.orchestra.run.smithi052.stdout:/run/credentials/systemd-tmpfiles-setup-dev.service 2024-05-27T07:22:27.859 INFO:teuthology.orchestra.run.smithi052.stdout:/run/credentials/systemd-tmpfiles-setup.service 2024-05-27T07:22:27.859 INFO:teuthology.orchestra.run.smithi052.stdout:/var/lib/nfs/rpc_pipefs 2024-05-27T07:22:27.860 INFO:teuthology.orchestra.run.smithi052.stdout:/run/user/1000 2024-05-27T07:22:27.860 INFO:teuthology.orchestra.run.smithi052.stdout:/var/lib/ceph 2024-05-27T07:22:27.860 INFO:teuthology.orchestra.run.smithi052.stdout:/var/lib/containers/storage/overlay 2024-05-27T07:22:27.860 INFO:teuthology.orchestra.run.smithi052.stdout:/var/lib/containers/storage/overlay/339b609a774606614a4357911c7efdf1485294f41086dc57156ce1506c859e82/merged 2024-05-27T07:22:27.860 INFO:teuthology.orchestra.run.smithi052.stdout:/var/lib/containers/storage/overlay/156afeab7561c4315f8c0269174cc6d7fc51ff200896f8ecdd8453475afa51f5/merged 2024-05-27T07:22:27.860 INFO:teuthology.orchestra.run.smithi052.stdout:/run/user/0 2024-05-27T07:22:27.860 INFO:teuthology.orchestra.run.smithi052.stdout:/proc/sys/fs/binfmt_misc 2024-05-27T07:22:27.860 INFO:teuthology.orchestra.run.smithi052.stdout:/var/lib/containers/storage/overlay/57343f17079f73a0c532f85d4aa22f6ba25f238945cd9874c632b74b3232ec25/merged 2024-05-27T07:22:27.860 INFO:teuthology.orchestra.run.smithi052.stdout:/var/lib/containers/storage/overlay/8032bf106ed313012e44c3d322c9802484683cfca36223e8fd3f0691555a7f4d/merged 2024-05-27T07:22:27.860 INFO:teuthology.orchestra.run.smithi052.stdout:/var/lib/containers/storage/overlay/0df7e3d814ba0a1e066e632896b4b3d19bc0a7b31de4a6efbe598517d3d4b3ee/merged 2024-05-27T07:22:27.860 INFO:teuthology.orchestra.run.smithi052.stdout:/var/lib/containers/storage/overlay/eb022041ddf436dc94838d9f6739734727769c7ddd3193bcc904b0f0d41551fd/merged 2024-05-27T07:22:27.860 INFO:teuthology.orchestra.run.smithi052.stdout:/var/lib/containers/storage/overlay/cf9b7bbfe621da9b2ded22d846b075a387466a2407402eeef036b38c7cdf5f9b/merged 2024-05-27T07:22:27.860 INFO:teuthology.orchestra.run.smithi052.stdout:/var/lib/containers/storage/overlay/b91f91b35c929c0ad0e1b2ed644718b1fe511cf643ff91b3ef9788305b5c35ae/merged 2024-05-27T07:22:27.861 INFO:teuthology.orchestra.run.smithi052.stdout:/var/lib/containers/storage/overlay/c61b48a893b2f74d05d41cb09909654b29f7079e8b1da82f2515c15858f8da97/merged 2024-05-27T07:22:27.861 INFO:teuthology.orchestra.run.smithi052.stdout:/var/lib/containers/storage/overlay/053ea3d29cab1736b9b72846a69904185921866998ecb426b6ddfb230e1a0c00/merged 2024-05-27T07:22:27.861 INFO:teuthology.orchestra.run.smithi052.stdout:/run/netns 2024-05-27T07:22:27.861 INFO:teuthology.orchestra.run.smithi052.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.admin 2024-05-27T07:22:27.861 INFO:teuthology.orchestra.run.smithi052.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.admin 2024-05-27T07:22:27.861 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-27T07:22:27.861 DEBUG:teuthology.orchestra.run.smithi052:> ls /sys/fs/fuse/connections 2024-05-27T07:22:27.892 INFO:tasks.cephfs.fuse_mount.ceph-fuse.admin.smithi052.stderr:2024-05-27T07:22:27.902+0000 7f33cee3c480 -1 init, newargv = 0x7f33bc0046a0 newargc=15 2024-05-27T07:22:27.892 INFO:tasks.cephfs.fuse_mount.ceph-fuse.admin.smithi052.stderr:ceph-fuse[78281]: starting ceph client 2024-05-27T07:22:27.899 INFO:tasks.cephfs.fuse_mount.ceph-fuse.admin.smithi052.stderr:ceph-fuse[78281]: starting fuse 2024-05-27T07:22:27.908 INFO:teuthology.orchestra.run.smithi052.stdout:119 2024-05-27T07:22:27.908 INFO:tasks.cephfs.fuse_mount:Post-mount connections: [119] 2024-05-27T07:22:27.908 DEBUG:teuthology.orchestra.run.smithi052:> sudo stdin-killer -- python3 -c ' 2024-05-27T07:22:27.908 DEBUG:teuthology.orchestra.run.smithi052:> import glob 2024-05-27T07:22:27.908 DEBUG:teuthology.orchestra.run.smithi052:> import re 2024-05-27T07:22:27.909 DEBUG:teuthology.orchestra.run.smithi052:> import os 2024-05-27T07:22:27.909 DEBUG:teuthology.orchestra.run.smithi052:> import subprocess 2024-05-27T07:22:27.909 DEBUG:teuthology.orchestra.run.smithi052:> 2024-05-27T07:22:27.909 DEBUG:teuthology.orchestra.run.smithi052:> def _find_admin_socket(client_name): 2024-05-27T07:22:27.909 DEBUG:teuthology.orchestra.run.smithi052:> asok_path = "/var/run/ceph/ceph-client.admin.*.asok" 2024-05-27T07:22:27.909 DEBUG:teuthology.orchestra.run.smithi052:> files = glob.glob(asok_path) 2024-05-27T07:22:27.909 DEBUG:teuthology.orchestra.run.smithi052:> mountpoint = "/home/ubuntu/cephtest/mnt.admin" 2024-05-27T07:22:27.909 DEBUG:teuthology.orchestra.run.smithi052:> 2024-05-27T07:22:27.909 DEBUG:teuthology.orchestra.run.smithi052:> # Given a non-glob path, it better be there 2024-05-27T07:22:27.909 DEBUG:teuthology.orchestra.run.smithi052:> if "*" not in asok_path: 2024-05-27T07:22:27.909 DEBUG:teuthology.orchestra.run.smithi052:> assert(len(files) == 1) 2024-05-27T07:22:27.909 DEBUG:teuthology.orchestra.run.smithi052:> return files[0] 2024-05-27T07:22:27.909 DEBUG:teuthology.orchestra.run.smithi052:> 2024-05-27T07:22:27.909 DEBUG:teuthology.orchestra.run.smithi052:> for f in files: 2024-05-27T07:22:27.909 DEBUG:teuthology.orchestra.run.smithi052:> pid = re.match(".*\.(\d+)\.asok$", f).group(1) 2024-05-27T07:22:27.910 DEBUG:teuthology.orchestra.run.smithi052:> if os.path.exists("/proc/{0}".format(pid)): 2024-05-27T07:22:27.910 DEBUG:teuthology.orchestra.run.smithi052:> with open("/proc/{0}/cmdline".format(pid), '"'"'r'"'"') as proc_f: 2024-05-27T07:22:27.910 DEBUG:teuthology.orchestra.run.smithi052:> contents = proc_f.read() 2024-05-27T07:22:27.910 DEBUG:teuthology.orchestra.run.smithi052:> if mountpoint in contents: 2024-05-27T07:22:27.910 DEBUG:teuthology.orchestra.run.smithi052:> return f 2024-05-27T07:22:27.910 DEBUG:teuthology.orchestra.run.smithi052:> raise RuntimeError("Client socket {0} not found".format(client_name)) 2024-05-27T07:22:27.910 DEBUG:teuthology.orchestra.run.smithi052:> 2024-05-27T07:22:27.910 DEBUG:teuthology.orchestra.run.smithi052:> print(_find_admin_socket("client.admin")) 2024-05-27T07:22:27.910 DEBUG:teuthology.orchestra.run.smithi052:> ' 2024-05-27T07:22:28.029 INFO:teuthology.orchestra.run.smithi052.stdout:/var/run/ceph/ceph-client.admin.78281.asok 2024-05-27T07:22:28.032 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:22:28 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-05-27T07:22:28.040 INFO:tasks.cephfs.fuse_mount:Found client admin socket at /var/run/ceph/ceph-client.admin.78281.asok 2024-05-27T07:22:28.040 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-27T07:22:28.041 DEBUG:teuthology.orchestra.run.smithi052:> sudo ceph --admin-daemon /var/run/ceph/ceph-client.admin.78281.asok status 2024-05-27T07:22:28.189 INFO:teuthology.orchestra.run.smithi052.stdout:{ 2024-05-27T07:22:28.189 INFO:teuthology.orchestra.run.smithi052.stdout: "metadata": { 2024-05-27T07:22:28.189 INFO:teuthology.orchestra.run.smithi052.stdout: "ceph_sha1": "38f9d900c83c63799fbdbe61acc9a11b0d3554a6", 2024-05-27T07:22:28.190 INFO:teuthology.orchestra.run.smithi052.stdout: "ceph_version": "ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev)", 2024-05-27T07:22:28.190 INFO:teuthology.orchestra.run.smithi052.stdout: "entity_id": "admin", 2024-05-27T07:22:28.190 INFO:teuthology.orchestra.run.smithi052.stdout: "hostname": "smithi052", 2024-05-27T07:22:28.190 INFO:teuthology.orchestra.run.smithi052.stdout: "mount_point": "/home/ubuntu/cephtest/mnt.admin", 2024-05-27T07:22:28.190 INFO:teuthology.orchestra.run.smithi052.stdout: "pid": "78281", 2024-05-27T07:22:28.190 INFO:teuthology.orchestra.run.smithi052.stdout: "root": "/" 2024-05-27T07:22:28.190 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-05-27T07:22:28.190 INFO:teuthology.orchestra.run.smithi052.stdout: "dentry_count": 0, 2024-05-27T07:22:28.190 INFO:teuthology.orchestra.run.smithi052.stdout: "dentry_pinned_count": 0, 2024-05-27T07:22:28.191 INFO:teuthology.orchestra.run.smithi052.stdout: "id": 14751, 2024-05-27T07:22:28.191 INFO:teuthology.orchestra.run.smithi052.stdout: "inst": { 2024-05-27T07:22:28.191 INFO:teuthology.orchestra.run.smithi052.stdout: "name": { 2024-05-27T07:22:28.191 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "client", 2024-05-27T07:22:28.191 INFO:teuthology.orchestra.run.smithi052.stdout: "num": 14751 2024-05-27T07:22:28.191 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-05-27T07:22:28.191 INFO:teuthology.orchestra.run.smithi052.stdout: "addr": { 2024-05-27T07:22:28.192 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "v1", 2024-05-27T07:22:28.192 INFO:teuthology.orchestra.run.smithi052.stdout: "addr": "192.168.0.1:0", 2024-05-27T07:22:28.192 INFO:teuthology.orchestra.run.smithi052.stdout: "nonce": 3333368950 2024-05-27T07:22:28.192 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-05-27T07:22:28.192 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-05-27T07:22:28.192 INFO:teuthology.orchestra.run.smithi052.stdout: "addr": { 2024-05-27T07:22:28.192 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "v1", 2024-05-27T07:22:28.192 INFO:teuthology.orchestra.run.smithi052.stdout: "addr": "192.168.0.1:0", 2024-05-27T07:22:28.192 INFO:teuthology.orchestra.run.smithi052.stdout: "nonce": 3333368950 2024-05-27T07:22:28.192 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-05-27T07:22:28.192 INFO:teuthology.orchestra.run.smithi052.stdout: "inst_str": "client.14751 192.168.0.1:0/3333368950", 2024-05-27T07:22:28.192 INFO:teuthology.orchestra.run.smithi052.stdout: "addr_str": "192.168.0.1:0/3333368950", 2024-05-27T07:22:28.192 INFO:teuthology.orchestra.run.smithi052.stdout: "inode_count": 1, 2024-05-27T07:22:28.193 INFO:teuthology.orchestra.run.smithi052.stdout: "mds_epoch": 17, 2024-05-27T07:22:28.193 INFO:teuthology.orchestra.run.smithi052.stdout: "osd_epoch": 86, 2024-05-27T07:22:28.193 INFO:teuthology.orchestra.run.smithi052.stdout: "osd_epoch_barrier": 0, 2024-05-27T07:22:28.193 INFO:teuthology.orchestra.run.smithi052.stdout: "blocklisted": false, 2024-05-27T07:22:28.193 INFO:teuthology.orchestra.run.smithi052.stdout: "fs_name": "cephfs" 2024-05-27T07:22:28.193 INFO:teuthology.orchestra.run.smithi052.stdout:} 2024-05-27T07:22:28.198 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-27T07:22:28.198 DEBUG:teuthology.orchestra.run.smithi052:> stat --file-system '--printf=%T 2024-05-27T07:22:28.198 DEBUG:teuthology.orchestra.run.smithi052:> ' -- /home/ubuntu/cephtest/mnt.admin 2024-05-27T07:22:28.215 INFO:teuthology.orchestra.run.smithi052.stdout:fuseblk 2024-05-27T07:22:28.215 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.admin 2024-05-27T07:22:28.215 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-27T07:22:28.215 DEBUG:teuthology.orchestra.run.smithi052:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.admin 2024-05-27T07:22:28.282 DEBUG:teuthology.orchestra.run.smithi052:> (cd /home/ubuntu/cephtest/mnt.admin && exec stdin-killer --timeout=900 -- bash -c 'setfattr -n ceph.dir.layout.pool -v cephfs_data_ec . && getfattr -n ceph.dir.layout .') 2024-05-27T07:22:28.375 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:22:28 stdin-killer INFO: expiration expected; waiting 900 seconds for command to complete 2024-05-27T07:22:28.401 INFO:teuthology.orchestra.run.smithi052.stdout:# file: . 2024-05-27T07:22:28.401 INFO:teuthology.orchestra.run.smithi052.stdout:ceph.dir.layout="stripe_unit=4194304 stripe_count=1 object_size=4194304 pool=cephfs_data_ec" 2024-05-27T07:22:28.401 INFO:teuthology.orchestra.run.smithi052.stdout: 2024-05-27T07:22:28.401 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:22:28 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-05-27T07:22:28.407 DEBUG:teuthology.orchestra.run.smithi052:> set -ex 2024-05-27T07:22:28.408 DEBUG:teuthology.orchestra.run.smithi052:> dd if=/proc/self/mounts of=/dev/stdout 2024-05-27T07:22:28.426 DEBUG:teuthology.orchestra.run.smithi052:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:38f9d900c83c63799fbdbe61acc9a11b0d3554a6 shell --fsid 73df2636-1bf8-11ef-bc9b-c7b262605968 -- ceph osd blocklist ls 2024-05-27T07:22:28.718 INFO:teuthology.orchestra.run.smithi052.stderr:Inferring config /var/lib/ceph/73df2636-1bf8-11ef-bc9b-c7b262605968/mon.a/config 2024-05-27T07:22:29.102 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:22:28 smithi173 ceph-mon[37000]: pgmap v315: 129 pgs: 129 active+clean; 579 KiB data, 356 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-27T07:22:29.106 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:22:28 smithi052 ceph-mon[31955]: pgmap v315: 129 pgs: 129 active+clean; 579 KiB data, 356 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-27T07:22:29.190 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:22:28 smithi105 ceph-mon[35977]: pgmap v315: 129 pgs: 129 active+clean; 579 KiB data, 356 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-27T07:22:29.871 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:22:29 smithi052 ceph-mon[31955]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.f", "id": [8, 11]}]: dispatch 2024-05-27T07:22:29.871 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:22:29 smithi052 ceph-mon[31955]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.10", "id": [2, 4]}]: dispatch 2024-05-27T07:22:29.871 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:22:29 smithi052 ceph-mon[31955]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.0", "id": [5, 9]}]: dispatch 2024-05-27T07:22:29.871 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:22:29 smithi052 ceph-mon[31955]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.1b", "id": [4, 8]}]: dispatch 2024-05-27T07:22:29.871 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:22:29 smithi052 ceph-mon[31955]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.2c", "id": [4, 9]}]: dispatch 2024-05-27T07:22:29.871 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:22:29 smithi052 ceph-mon[31955]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.3c", "id": [4, 9]}]: dispatch 2024-05-27T07:22:29.872 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:22:29 smithi052 ceph-mon[31955]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.e", "id": [9, 11]}]: dispatch 2024-05-27T07:22:29.872 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:22:29 smithi052 ceph-mon[31955]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.12", "id": [5, 10]}]: dispatch 2024-05-27T07:22:29.872 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:22:29 smithi052 ceph-mon[31955]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.19", "id": [9, 8]}]: dispatch 2024-05-27T07:22:29.872 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:22:29 smithi052 ceph-mon[31955]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1f", "id": [9, 8]}]: dispatch 2024-05-27T07:22:30.102 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:22:29 smithi173 ceph-mon[37000]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.f", "id": [8, 11]}]: dispatch 2024-05-27T07:22:30.102 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:22:29 smithi173 ceph-mon[37000]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.10", "id": [2, 4]}]: dispatch 2024-05-27T07:22:30.102 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:22:29 smithi173 ceph-mon[37000]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.0", "id": [5, 9]}]: dispatch 2024-05-27T07:22:30.102 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:22:29 smithi173 ceph-mon[37000]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.1b", "id": [4, 8]}]: dispatch 2024-05-27T07:22:30.102 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:22:29 smithi173 ceph-mon[37000]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.2c", "id": [4, 9]}]: dispatch 2024-05-27T07:22:30.102 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:22:29 smithi173 ceph-mon[37000]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.3c", "id": [4, 9]}]: dispatch 2024-05-27T07:22:30.102 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:22:29 smithi173 ceph-mon[37000]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.e", "id": [9, 11]}]: dispatch 2024-05-27T07:22:30.102 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:22:29 smithi173 ceph-mon[37000]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.12", "id": [5, 10]}]: dispatch 2024-05-27T07:22:30.103 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:22:29 smithi173 ceph-mon[37000]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.19", "id": [9, 8]}]: dispatch 2024-05-27T07:22:30.103 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:22:29 smithi173 ceph-mon[37000]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1f", "id": [9, 8]}]: dispatch 2024-05-27T07:22:30.190 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:22:29 smithi105 ceph-mon[35977]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.f", "id": [8, 11]}]: dispatch 2024-05-27T07:22:30.190 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:22:29 smithi105 ceph-mon[35977]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.10", "id": [2, 4]}]: dispatch 2024-05-27T07:22:30.190 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:22:29 smithi105 ceph-mon[35977]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.0", "id": [5, 9]}]: dispatch 2024-05-27T07:22:30.190 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:22:29 smithi105 ceph-mon[35977]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.1b", "id": [4, 8]}]: dispatch 2024-05-27T07:22:30.190 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:22:29 smithi105 ceph-mon[35977]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.2c", "id": [4, 9]}]: dispatch 2024-05-27T07:22:30.190 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:22:29 smithi105 ceph-mon[35977]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.3c", "id": [4, 9]}]: dispatch 2024-05-27T07:22:30.190 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:22:29 smithi105 ceph-mon[35977]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.e", "id": [9, 11]}]: dispatch 2024-05-27T07:22:30.191 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:22:29 smithi105 ceph-mon[35977]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.12", "id": [5, 10]}]: dispatch 2024-05-27T07:22:30.191 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:22:29 smithi105 ceph-mon[35977]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.19", "id": [9, 8]}]: dispatch 2024-05-27T07:22:30.191 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:22:29 smithi105 ceph-mon[35977]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1f", "id": [9, 8]}]: dispatch 2024-05-27T07:22:30.435 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:22:30.445+0000 7f8a1e9a5700 1 -- 172.21.15.52:0/2833544066 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f8a181045e0 msgr2=0x7f8a181049f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:22:30.435 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:22:30.445+0000 7f8a1e9a5700 1 --2- 172.21.15.52:0/2833544066 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f8a181045e0 0x7f8a181049f0 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7f8a080099c0 tx=0x7f8a08009cd0 comp rx=0 tx=0).stop 2024-05-27T07:22:30.435 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:22:30.445+0000 7f8a1e9a5700 1 -- 172.21.15.52:0/2833544066 shutdown_connections 2024-05-27T07:22:30.435 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:22:30.445+0000 7f8a1e9a5700 1 --2- 172.21.15.52:0/2833544066 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f8a18109860 0x7f8a181111c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:22:30.436 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:22:30.445+0000 7f8a1e9a5700 1 --2- 172.21.15.52:0/2833544066 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f8a181045e0 0x7f8a181049f0 unknown :-1 s=CLOSED pgs=211 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:22:30.436 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:22:30.445+0000 7f8a1e9a5700 1 --2- 172.21.15.52:0/2833544066 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f8a18108f10 0x7f8a18109320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:22:30.436 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:22:30.445+0000 7f8a1e9a5700 1 -- 172.21.15.52:0/2833544066 >> 172.21.15.52:0/2833544066 conn(0x7f8a18100030 msgr2=0x7f8a18102450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:22:30.436 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:22:30.445+0000 7f8a1e9a5700 1 -- 172.21.15.52:0/2833544066 shutdown_connections 2024-05-27T07:22:30.436 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:22:30.445+0000 7f8a1e9a5700 1 -- 172.21.15.52:0/2833544066 wait complete. 2024-05-27T07:22:30.438 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:22:30.446+0000 7f8a1e9a5700 1 Processor -- start 2024-05-27T07:22:30.438 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:22:30.446+0000 7f8a1e9a5700 1 -- start start 2024-05-27T07:22:30.438 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:22:30.446+0000 7f8a1e9a5700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f8a181045e0 0x7f8a18075ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:22:30.438 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:22:30.446+0000 7f8a1e9a5700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f8a18108f10 0x7f8a18075ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:22:30.438 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:22:30.446+0000 7f8a1e9a5700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f8a18109860 0x7f8a18076530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:22:30.438 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:22:30.446+0000 7f8a1e9a5700 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x7f8a181144e0 con 0x7f8a18109860 2024-05-27T07:22:30.438 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:22:30.446+0000 7f8a1e9a5700 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f8a18114360 con 0x7f8a18108f10 2024-05-27T07:22:30.439 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:22:30.446+0000 7f8a1e9a5700 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f8a18114660 con 0x7f8a181045e0 2024-05-27T07:22:30.439 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:22:30.446+0000 7f8a1cf42700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f8a18109860 0x7f8a18076530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:22:30.439 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:22:30.446+0000 7f8a1cf42700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f8a18109860 0x7f8a18076530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.52:60544/0 (socket says 172.21.15.52:60544) 2024-05-27T07:22:30.439 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:22:30.446+0000 7f8a1cf42700 1 -- 172.21.15.52:0/3456750030 learned_addr learned my addr 172.21.15.52:0/3456750030 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-05-27T07:22:30.439 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:22:30.446+0000 7f8a17fff700 1 --2- 172.21.15.52:0/3456750030 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f8a181045e0 0x7f8a18075ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:22:30.439 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:22:30.446+0000 7f8a0ffff700 1 --2- 172.21.15.52:0/3456750030 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f8a18108f10 0x7f8a18075ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:22:30.439 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:22:30.447+0000 7f8a1cf42700 1 -- 172.21.15.52:0/3456750030 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f8a181045e0 msgr2=0x7f8a18075ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:22:30.439 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:22:30.447+0000 7f8a1cf42700 1 --2- 172.21.15.52:0/3456750030 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f8a181045e0 0x7f8a18075ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:22:30.439 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:22:30.447+0000 7f8a1cf42700 1 -- 172.21.15.52:0/3456750030 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f8a18108f10 msgr2=0x7f8a18075ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:22:30.439 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:22:30.447+0000 7f8a1cf42700 1 --2- 172.21.15.52:0/3456750030 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f8a18108f10 0x7f8a18075ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:22:30.439 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:22:30.447+0000 7f8a1cf42700 1 -- 172.21.15.52:0/3456750030 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8a181a9160 con 0x7f8a18109860 2024-05-27T07:22:30.440 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:22:30.447+0000 7f8a0ffff700 1 --2- 172.21.15.52:0/3456750030 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f8a18108f10 0x7f8a18075ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-27T07:22:30.440 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:22:30.447+0000 7f8a1cf42700 1 --2- 172.21.15.52:0/3456750030 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f8a18109860 0x7f8a18076530 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7f8a10007ee0 tx=0x7f8a1000d3b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:22:30.440 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:22:30.447+0000 7f8a15ffb700 1 -- 172.21.15.52:0/3456750030 <== mon.0 v2:172.21.15.52:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8a1000dbd0 con 0x7f8a18109860 2024-05-27T07:22:30.440 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:22:30.447+0000 7f8a15ffb700 1 -- 172.21.15.52:0/3456750030 <== mon.0 v2:172.21.15.52:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f8a1000dd70 con 0x7f8a18109860 2024-05-27T07:22:30.440 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:22:30.447+0000 7f8a1e9a5700 1 -- 172.21.15.52:0/3456750030 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8a181a9450 con 0x7f8a18109860 2024-05-27T07:22:30.440 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:22:30.447+0000 7f8a15ffb700 1 -- 172.21.15.52:0/3456750030 <== mon.0 v2:172.21.15.52:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8a10005990 con 0x7f8a18109860 2024-05-27T07:22:30.440 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:22:30.447+0000 7f8a1e9a5700 1 -- 172.21.15.52:0/3456750030 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8a181a9940 con 0x7f8a18109860 2024-05-27T07:22:30.441 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:22:30.448+0000 7f8a1e9a5700 1 -- 172.21.15.52:0/3456750030 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8a180512b0 con 0x7f8a18109860 2024-05-27T07:22:30.441 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:22:30.450+0000 7f8a15ffb700 1 -- 172.21.15.52:0/3456750030 <== mon.0 v2:172.21.15.52:3300/0 4 ==== mgrmap(e 16) ==== 149652+0+0 (secure 0 0 0) 0x7f8a1001a020 con 0x7f8a18109860 2024-05-27T07:22:30.441 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:22:30.450+0000 7f8a15ffb700 1 --2- 172.21.15.52:0/3456750030 >> [v2:172.21.15.52:6800/3513075805,v1:172.21.15.52:6801/3513075805] conn(0x7f8a040b2880 0x7f8a040b4d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:22:30.441 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:22:30.451+0000 7f8a17fff700 1 --2- 172.21.15.52:0/3456750030 >> [v2:172.21.15.52:6800/3513075805,v1:172.21.15.52:6801/3513075805] conn(0x7f8a040b2880 0x7f8a040b4d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:22:30.441 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:22:30.451+0000 7f8a15ffb700 1 -- 172.21.15.52:0/3456750030 <== mon.0 v2:172.21.15.52:3300/0 5 ==== osd_map(87..87 src has 1..87) ==== 9092+0+0 (secure 0 0 0) 0x7f8a100e1cb0 con 0x7f8a18109860 2024-05-27T07:22:30.441 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:22:30.451+0000 7f8a17fff700 1 --2- 172.21.15.52:0/3456750030 >> [v2:172.21.15.52:6800/3513075805,v1:172.21.15.52:6801/3513075805] conn(0x7f8a040b2880 0x7f8a040b4d90 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7f8a0800b580 tx=0x7f8a080036e0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:22:30.445 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:22:30.455+0000 7f8a15ffb700 1 -- 172.21.15.52:0/3456750030 <== mon.0 v2:172.21.15.52:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191963 (secure 0 0 0) 0x7f8a10010070 con 0x7f8a18109860 2024-05-27T07:22:30.681 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:22:30.690+0000 7f8a1e9a5700 1 -- 172.21.15.52:0/3456750030 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) -- 0x7f8a1802e5e0 con 0x7f8a18109860 2024-05-27T07:22:30.777 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:22:30.691+0000 7f8a15ffb700 1 -- 172.21.15.52:0/3456750030 <== mon.0 v2:172.21.15.52:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 10 entries v87) ==== 81+0+588 (secure 0 0 0) 0x7f8a100a9670 con 0x7f8a18109860 2024-05-27T07:22:30.778 INFO:teuthology.orchestra.run.smithi052.stdout:172.21.15.52:6801/1631683163 2024-05-28T07:14:29.544633+0000 2024-05-27T07:22:30.778 INFO:teuthology.orchestra.run.smithi052.stdout:172.21.15.52:0/2776144659 2024-05-28T07:14:29.544633+0000 2024-05-27T07:22:30.778 INFO:teuthology.orchestra.run.smithi052.stdout:172.21.15.52:6800/1631683163 2024-05-28T07:14:29.544633+0000 2024-05-27T07:22:30.778 INFO:teuthology.orchestra.run.smithi052.stdout:172.21.15.52:0/3675529097 2024-05-28T07:14:01.960550+0000 2024-05-27T07:22:30.778 INFO:teuthology.orchestra.run.smithi052.stdout:172.21.15.52:0/727808107 2024-05-28T07:14:01.960550+0000 2024-05-27T07:22:30.778 INFO:teuthology.orchestra.run.smithi052.stdout:172.21.15.52:0/3839436833 2024-05-28T07:14:29.544633+0000 2024-05-27T07:22:30.778 INFO:teuthology.orchestra.run.smithi052.stdout:172.21.15.52:6800/587662512 2024-05-28T07:14:01.960550+0000 2024-05-27T07:22:30.778 INFO:teuthology.orchestra.run.smithi052.stdout:172.21.15.52:6801/587662512 2024-05-28T07:14:01.960550+0000 2024-05-27T07:22:30.778 INFO:teuthology.orchestra.run.smithi052.stdout:172.21.15.52:0/853867465 2024-05-28T07:14:01.960550+0000 2024-05-27T07:22:30.778 INFO:teuthology.orchestra.run.smithi052.stdout:172.21.15.52:0/3480918279 2024-05-28T07:14:29.544633+0000 2024-05-27T07:22:30.779 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:22:30.693+0000 7f8a1e9a5700 1 -- 172.21.15.52:0/3456750030 >> [v2:172.21.15.52:6800/3513075805,v1:172.21.15.52:6801/3513075805] conn(0x7f8a040b2880 msgr2=0x7f8a040b4d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:22:30.779 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:22:30.693+0000 7f8a1e9a5700 1 --2- 172.21.15.52:0/3456750030 >> [v2:172.21.15.52:6800/3513075805,v1:172.21.15.52:6801/3513075805] conn(0x7f8a040b2880 0x7f8a040b4d90 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7f8a0800b580 tx=0x7f8a080036e0 comp rx=0 tx=0).stop 2024-05-27T07:22:30.779 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:22:30.693+0000 7f8a1e9a5700 1 -- 172.21.15.52:0/3456750030 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f8a18109860 msgr2=0x7f8a18076530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:22:30.779 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:22:30.693+0000 7f8a1e9a5700 1 --2- 172.21.15.52:0/3456750030 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f8a18109860 0x7f8a18076530 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7f8a10007ee0 tx=0x7f8a1000d3b0 comp rx=0 tx=0).stop 2024-05-27T07:22:30.779 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:22:30.693+0000 7f8a1e9a5700 1 -- 172.21.15.52:0/3456750030 shutdown_connections 2024-05-27T07:22:30.779 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:22:30.693+0000 7f8a1e9a5700 1 --2- 172.21.15.52:0/3456750030 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f8a18109860 0x7f8a18076530 unknown :-1 s=CLOSED pgs=212 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:22:30.779 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:22:30.693+0000 7f8a1e9a5700 1 --2- 172.21.15.52:0/3456750030 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f8a181045e0 0x7f8a18075ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:22:30.780 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:22:30.693+0000 7f8a1e9a5700 1 --2- 172.21.15.52:0/3456750030 >> [v2:172.21.15.52:6800/3513075805,v1:172.21.15.52:6801/3513075805] conn(0x7f8a040b2880 0x7f8a040b4d90 unknown :-1 s=CLOSED pgs=155 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:22:30.780 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:22:30.693+0000 7f8a1e9a5700 1 --2- 172.21.15.52:0/3456750030 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f8a18108f10 0x7f8a18075ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:22:30.780 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:22:30.694+0000 7f8a1e9a5700 1 -- 172.21.15.52:0/3456750030 >> 172.21.15.52:0/3456750030 conn(0x7f8a18100030 msgr2=0x7f8a1810a7c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:22:30.780 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:22:30.694+0000 7f8a1e9a5700 1 -- 172.21.15.52:0/3456750030 shutdown_connections 2024-05-27T07:22:30.780 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:22:30.694+0000 7f8a1e9a5700 1 -- 172.21.15.52:0/3456750030 wait complete. 2024-05-27T07:22:30.780 INFO:teuthology.orchestra.run.smithi052.stderr:listed 10 entries 2024-05-27T07:22:30.979 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:22:30 smithi052 ceph-mon[31955]: pgmap v316: 129 pgs: 129 active+clean; 581 KiB data, 356 MiB used, 1.0 TiB / 1.0 TiB avail; 341 B/s wr, 0 op/s 2024-05-27T07:22:30.979 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:22:30 smithi052 ceph-mon[31955]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.f", "id": [8, 11]}]': finished 2024-05-27T07:22:30.979 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:22:30 smithi052 ceph-mon[31955]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.10", "id": [2, 4]}]': finished 2024-05-27T07:22:30.980 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:22:30 smithi052 ceph-mon[31955]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.0", "id": [5, 9]}]': finished 2024-05-27T07:22:30.980 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:22:30 smithi052 ceph-mon[31955]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.1b", "id": [4, 8]}]': finished 2024-05-27T07:22:30.980 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:22:30 smithi052 ceph-mon[31955]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.2c", "id": [4, 9]}]': finished 2024-05-27T07:22:30.980 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:22:30 smithi052 ceph-mon[31955]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.3c", "id": [4, 9]}]': finished 2024-05-27T07:22:30.980 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:22:30 smithi052 ceph-mon[31955]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.e", "id": [9, 11]}]': finished 2024-05-27T07:22:30.980 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:22:30 smithi052 ceph-mon[31955]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.12", "id": [5, 10]}]': finished 2024-05-27T07:22:30.980 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:22:30 smithi052 ceph-mon[31955]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.19", "id": [9, 8]}]': finished 2024-05-27T07:22:30.980 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:22:30 smithi052 ceph-mon[31955]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1f", "id": [9, 8]}]': finished 2024-05-27T07:22:30.980 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:22:30 smithi052 ceph-mon[31955]: osdmap e87: 12 total, 12 up, 12 in 2024-05-27T07:22:30.980 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:22:30 smithi052 ceph-mon[31955]: from='client.? 172.21.15.52:0/3456750030' entity='client.admin' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-05-27T07:22:30.980 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:22:30 smithi052 ceph-mon[31955]: osdmap e88: 12 total, 12 up, 12 in 2024-05-27T07:22:31.102 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:22:30 smithi173 ceph-mon[37000]: pgmap v316: 129 pgs: 129 active+clean; 581 KiB data, 356 MiB used, 1.0 TiB / 1.0 TiB avail; 341 B/s wr, 0 op/s 2024-05-27T07:22:31.102 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:22:30 smithi173 ceph-mon[37000]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.f", "id": [8, 11]}]': finished 2024-05-27T07:22:31.102 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:22:30 smithi173 ceph-mon[37000]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.10", "id": [2, 4]}]': finished 2024-05-27T07:22:31.103 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:22:30 smithi173 ceph-mon[37000]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.0", "id": [5, 9]}]': finished 2024-05-27T07:22:31.103 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:22:30 smithi173 ceph-mon[37000]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.1b", "id": [4, 8]}]': finished 2024-05-27T07:22:31.103 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:22:30 smithi173 ceph-mon[37000]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.2c", "id": [4, 9]}]': finished 2024-05-27T07:22:31.103 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:22:30 smithi173 ceph-mon[37000]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.3c", "id": [4, 9]}]': finished 2024-05-27T07:22:31.103 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:22:30 smithi173 ceph-mon[37000]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.e", "id": [9, 11]}]': finished 2024-05-27T07:22:31.103 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:22:30 smithi173 ceph-mon[37000]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.12", "id": [5, 10]}]': finished 2024-05-27T07:22:31.103 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:22:30 smithi173 ceph-mon[37000]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.19", "id": [9, 8]}]': finished 2024-05-27T07:22:31.103 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:22:30 smithi173 ceph-mon[37000]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1f", "id": [9, 8]}]': finished 2024-05-27T07:22:31.103 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:22:30 smithi173 ceph-mon[37000]: osdmap e87: 12 total, 12 up, 12 in 2024-05-27T07:22:31.103 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:22:30 smithi173 ceph-mon[37000]: from='client.? 172.21.15.52:0/3456750030' entity='client.admin' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-05-27T07:22:31.103 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:22:30 smithi173 ceph-mon[37000]: osdmap e88: 12 total, 12 up, 12 in 2024-05-27T07:22:31.190 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:22:30 smithi105 ceph-mon[35977]: pgmap v316: 129 pgs: 129 active+clean; 581 KiB data, 356 MiB used, 1.0 TiB / 1.0 TiB avail; 341 B/s wr, 0 op/s 2024-05-27T07:22:31.190 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:22:30 smithi105 ceph-mon[35977]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.f", "id": [8, 11]}]': finished 2024-05-27T07:22:31.190 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:22:30 smithi105 ceph-mon[35977]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.10", "id": [2, 4]}]': finished 2024-05-27T07:22:31.190 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:22:30 smithi105 ceph-mon[35977]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.0", "id": [5, 9]}]': finished 2024-05-27T07:22:31.190 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:22:30 smithi105 ceph-mon[35977]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.1b", "id": [4, 8]}]': finished 2024-05-27T07:22:31.194 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:22:30 smithi105 ceph-mon[35977]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.2c", "id": [4, 9]}]': finished 2024-05-27T07:22:31.194 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:22:30 smithi105 ceph-mon[35977]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.3c", "id": [4, 9]}]': finished 2024-05-27T07:22:31.194 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:22:30 smithi105 ceph-mon[35977]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.e", "id": [9, 11]}]': finished 2024-05-27T07:22:31.194 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:22:30 smithi105 ceph-mon[35977]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.12", "id": [5, 10]}]': finished 2024-05-27T07:22:31.194 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:22:30 smithi105 ceph-mon[35977]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.19", "id": [9, 8]}]': finished 2024-05-27T07:22:31.194 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:22:30 smithi105 ceph-mon[35977]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1f", "id": [9, 8]}]': finished 2024-05-27T07:22:31.194 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:22:30 smithi105 ceph-mon[35977]: osdmap e87: 12 total, 12 up, 12 in 2024-05-27T07:22:31.194 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:22:30 smithi105 ceph-mon[35977]: from='client.? 172.21.15.52:0/3456750030' entity='client.admin' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-05-27T07:22:31.194 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:22:30 smithi105 ceph-mon[35977]: osdmap e88: 12 total, 12 up, 12 in 2024-05-27T07:22:31.386 DEBUG:teuthology.orchestra.run.smithi052:> set -ex 2024-05-27T07:22:31.386 DEBUG:teuthology.orchestra.run.smithi052:> dd if=/proc/self/mounts of=/dev/stdout 2024-05-27T07:22:31.405 DEBUG:teuthology.orchestra.run.smithi052:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:38f9d900c83c63799fbdbe61acc9a11b0d3554a6 shell --fsid 73df2636-1bf8-11ef-bc9b-c7b262605968 -- ceph osd blocklist ls 2024-05-27T07:22:31.697 INFO:teuthology.orchestra.run.smithi052.stderr:Inferring config /var/lib/ceph/73df2636-1bf8-11ef-bc9b-c7b262605968/mon.a/config 2024-05-27T07:22:33.102 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:22:32 smithi173 ceph-mon[37000]: pgmap v319: 129 pgs: 5 peering, 124 active+clean; 581 KiB data, 356 MiB used, 1.0 TiB / 1.0 TiB avail; 511 B/s wr, 0 op/s; 2 B/s, 0 objects/s recovering 2024-05-27T07:22:33.151 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:22:32 smithi052 ceph-mon[31955]: pgmap v319: 129 pgs: 5 peering, 124 active+clean; 581 KiB data, 356 MiB used, 1.0 TiB / 1.0 TiB avail; 511 B/s wr, 0 op/s; 2 B/s, 0 objects/s recovering 2024-05-27T07:22:33.190 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:22:32 smithi105 ceph-mon[35977]: pgmap v319: 129 pgs: 5 peering, 124 active+clean; 581 KiB data, 356 MiB used, 1.0 TiB / 1.0 TiB avail; 511 B/s wr, 0 op/s; 2 B/s, 0 objects/s recovering 2024-05-27T07:22:33.407 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:22:33.416+0000 7f356f6fd700 1 -- 172.21.15.52:0/716183115 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f3568107de0 msgr2=0x7f35681081f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:22:33.407 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:22:33.416+0000 7f356f6fd700 1 --2- 172.21.15.52:0/716183115 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f3568107de0 0x7f35681081f0 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7f35580099c0 tx=0x7f3558009cd0 comp rx=0 tx=0).stop 2024-05-27T07:22:33.407 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:22:33.416+0000 7f356f6fd700 1 -- 172.21.15.52:0/716183115 shutdown_connections 2024-05-27T07:22:33.407 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:22:33.416+0000 7f356f6fd700 1 --2- 172.21.15.52:0/716183115 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f3568108730 0x7f3568110090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:22:33.407 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:22:33.416+0000 7f356f6fd700 1 --2- 172.21.15.52:0/716183115 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f35681034b0 0x7f35681038c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:22:33.407 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:22:33.416+0000 7f356f6fd700 1 --2- 172.21.15.52:0/716183115 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f3568107de0 0x7f35681081f0 unknown :-1 s=CLOSED pgs=213 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:22:33.407 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:22:33.417+0000 7f356f6fd700 1 -- 172.21.15.52:0/716183115 >> 172.21.15.52:0/716183115 conn(0x7f35680fede0 msgr2=0x7f3568101200 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:22:33.407 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:22:33.417+0000 7f356f6fd700 1 -- 172.21.15.52:0/716183115 shutdown_connections 2024-05-27T07:22:33.408 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:22:33.417+0000 7f356f6fd700 1 -- 172.21.15.52:0/716183115 wait complete. 2024-05-27T07:22:33.409 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:22:33.417+0000 7f356f6fd700 1 Processor -- start 2024-05-27T07:22:33.409 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:22:33.417+0000 7f356f6fd700 1 -- start start 2024-05-27T07:22:33.410 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:22:33.418+0000 7f356f6fd700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f35681034b0 0x7f356819bd40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:22:33.410 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:22:33.418+0000 7f356f6fd700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f3568107de0 0x7f356819c280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:22:33.410 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:22:33.418+0000 7f356f6fd700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f3568108730 0x7f356819c7e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:22:33.410 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:22:33.418+0000 7f356f6fd700 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x7f3568113400 con 0x7f3568108730 2024-05-27T07:22:33.410 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:22:33.418+0000 7f356f6fd700 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f3568113280 con 0x7f3568107de0 2024-05-27T07:22:33.410 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:22:33.418+0000 7f356f6fd700 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f3568113580 con 0x7f35681034b0 2024-05-27T07:22:33.410 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:22:33.418+0000 7f356dc9a700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f3568108730 0x7f356819c7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:22:33.410 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:22:33.418+0000 7f356dc9a700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f3568108730 0x7f356819c7e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.52:39276/0 (socket says 172.21.15.52:39276) 2024-05-27T07:22:33.410 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:22:33.418+0000 7f356d499700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f35681034b0 0x7f356819bd40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:22:33.410 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:22:33.418+0000 7f356dc9a700 1 -- 172.21.15.52:0/1726929725 learned_addr learned my addr 172.21.15.52:0/1726929725 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-05-27T07:22:33.410 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:22:33.418+0000 7f356cc98700 1 --2- 172.21.15.52:0/1726929725 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f3568107de0 0x7f356819c280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:22:33.411 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:22:33.418+0000 7f356d499700 1 -- 172.21.15.52:0/1726929725 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f3568107de0 msgr2=0x7f356819c280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:22:33.411 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:22:33.418+0000 7f356d499700 1 --2- 172.21.15.52:0/1726929725 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f3568107de0 0x7f356819c280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:22:33.411 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:22:33.418+0000 7f356d499700 1 -- 172.21.15.52:0/1726929725 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f3568108730 msgr2=0x7f356819c7e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:22:33.411 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:22:33.418+0000 7f356d499700 1 --2- 172.21.15.52:0/1726929725 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f3568108730 0x7f356819c7e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:22:33.411 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:22:33.418+0000 7f356d499700 1 -- 172.21.15.52:0/1726929725 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f35681a7090 con 0x7f35681034b0 2024-05-27T07:22:33.411 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:22:33.418+0000 7f356dc9a700 1 --2- 172.21.15.52:0/1726929725 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f3568108730 0x7f356819c7e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-27T07:22:33.411 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:22:33.419+0000 7f356d499700 1 --2- 172.21.15.52:0/1726929725 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f35681034b0 0x7f356819bd40 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f356400ada0 tx=0x7f356400c330 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:22:33.411 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:22:33.419+0000 7f355e7fc700 1 -- 172.21.15.52:0/1726929725 <== mon.1 v2:172.21.15.173:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3564017020 con 0x7f35681034b0 2024-05-27T07:22:33.411 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:22:33.419+0000 7f356f6fd700 1 -- 172.21.15.52:0/1726929725 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f35681a7380 con 0x7f35681034b0 2024-05-27T07:22:33.411 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:22:33.419+0000 7f356f6fd700 1 -- 172.21.15.52:0/1726929725 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f35681a78e0 con 0x7f35681034b0 2024-05-27T07:22:33.411 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:22:33.419+0000 7f355e7fc700 1 -- 172.21.15.52:0/1726929725 <== mon.1 v2:172.21.15.173:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f356400cd80 con 0x7f35681034b0 2024-05-27T07:22:33.412 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:22:33.419+0000 7f355e7fc700 1 -- 172.21.15.52:0/1726929725 <== mon.1 v2:172.21.15.173:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3564005280 con 0x7f35681034b0 2024-05-27T07:22:33.412 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:22:33.420+0000 7f356f6fd700 1 -- 172.21.15.52:0/1726929725 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f354c005330 con 0x7f35681034b0 2024-05-27T07:22:33.412 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:22:33.422+0000 7f355e7fc700 1 -- 172.21.15.52:0/1726929725 <== mon.1 v2:172.21.15.173:3300/0 4 ==== mgrmap(e 16) ==== 149652+0+0 (secure 0 0 0) 0x7f356400a090 con 0x7f35681034b0 2024-05-27T07:22:33.413 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:22:33.423+0000 7f355e7fc700 1 --2- 172.21.15.52:0/1726929725 >> [v2:172.21.15.52:6800/3513075805,v1:172.21.15.52:6801/3513075805] conn(0x7f35540b2880 0x7f35540b4d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:22:33.413 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:22:33.423+0000 7f356cc98700 1 --2- 172.21.15.52:0/1726929725 >> [v2:172.21.15.52:6800/3513075805,v1:172.21.15.52:6801/3513075805] conn(0x7f35540b2880 0x7f35540b4d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:22:33.413 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:22:33.423+0000 7f355e7fc700 1 -- 172.21.15.52:0/1726929725 <== mon.1 v2:172.21.15.173:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 9092+0+0 (secure 0 0 0) 0x7f35640e1ff0 con 0x7f35681034b0 2024-05-27T07:22:33.413 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:22:33.423+0000 7f356cc98700 1 --2- 172.21.15.52:0/1726929725 >> [v2:172.21.15.52:6800/3513075805,v1:172.21.15.52:6801/3513075805] conn(0x7f35540b2880 0x7f35540b4d90 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7f3558006010 tx=0x7f355800b540 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:22:33.416 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:22:33.427+0000 7f355e7fc700 1 -- 172.21.15.52:0/1726929725 <== mon.1 v2:172.21.15.173:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191963 (secure 0 0 0) 0x7f3564021020 con 0x7f35681034b0 2024-05-27T07:22:33.650 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:22:33.659+0000 7f356f6fd700 1 -- 172.21.15.52:0/1726929725 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) -- 0x7f354c005620 con 0x7f35681034b0 2024-05-27T07:22:33.651 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:22:33.659+0000 7f355e7fc700 1 -- 172.21.15.52:0/1726929725 <== mon.1 v2:172.21.15.173:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 10 entries v88) ==== 81+0+588 (secure 0 0 0) 0x7f3564004ad0 con 0x7f35681034b0 2024-05-27T07:22:33.651 INFO:teuthology.orchestra.run.smithi052.stdout:172.21.15.52:6801/1631683163 2024-05-28T07:14:29.544633+0000 2024-05-27T07:22:33.651 INFO:teuthology.orchestra.run.smithi052.stdout:172.21.15.52:0/2776144659 2024-05-28T07:14:29.544633+0000 2024-05-27T07:22:33.651 INFO:teuthology.orchestra.run.smithi052.stdout:172.21.15.52:6800/1631683163 2024-05-28T07:14:29.544633+0000 2024-05-27T07:22:33.651 INFO:teuthology.orchestra.run.smithi052.stdout:172.21.15.52:0/3675529097 2024-05-28T07:14:01.960550+0000 2024-05-27T07:22:33.651 INFO:teuthology.orchestra.run.smithi052.stdout:172.21.15.52:0/727808107 2024-05-28T07:14:01.960550+0000 2024-05-27T07:22:33.651 INFO:teuthology.orchestra.run.smithi052.stdout:172.21.15.52:0/3839436833 2024-05-28T07:14:29.544633+0000 2024-05-27T07:22:33.651 INFO:teuthology.orchestra.run.smithi052.stdout:172.21.15.52:6800/587662512 2024-05-28T07:14:01.960550+0000 2024-05-27T07:22:33.651 INFO:teuthology.orchestra.run.smithi052.stdout:172.21.15.52:6801/587662512 2024-05-28T07:14:01.960550+0000 2024-05-27T07:22:33.651 INFO:teuthology.orchestra.run.smithi052.stdout:172.21.15.52:0/853867465 2024-05-28T07:14:01.960550+0000 2024-05-27T07:22:33.652 INFO:teuthology.orchestra.run.smithi052.stdout:172.21.15.52:0/3480918279 2024-05-28T07:14:29.544633+0000 2024-05-27T07:22:33.652 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:22:33.662+0000 7f356f6fd700 1 -- 172.21.15.52:0/1726929725 >> [v2:172.21.15.52:6800/3513075805,v1:172.21.15.52:6801/3513075805] conn(0x7f35540b2880 msgr2=0x7f35540b4d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:22:33.652 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:22:33.662+0000 7f356f6fd700 1 --2- 172.21.15.52:0/1726929725 >> [v2:172.21.15.52:6800/3513075805,v1:172.21.15.52:6801/3513075805] conn(0x7f35540b2880 0x7f35540b4d90 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7f3558006010 tx=0x7f355800b540 comp rx=0 tx=0).stop 2024-05-27T07:22:33.652 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:22:33.662+0000 7f356f6fd700 1 -- 172.21.15.52:0/1726929725 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f35681034b0 msgr2=0x7f356819bd40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:22:33.652 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:22:33.662+0000 7f356f6fd700 1 --2- 172.21.15.52:0/1726929725 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f35681034b0 0x7f356819bd40 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f356400ada0 tx=0x7f356400c330 comp rx=0 tx=0).stop 2024-05-27T07:22:33.653 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:22:33.662+0000 7f356f6fd700 1 -- 172.21.15.52:0/1726929725 shutdown_connections 2024-05-27T07:22:33.653 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:22:33.662+0000 7f356f6fd700 1 --2- 172.21.15.52:0/1726929725 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f3568108730 0x7f356819c7e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:22:33.653 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:22:33.662+0000 7f356f6fd700 1 --2- 172.21.15.52:0/1726929725 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f35681034b0 0x7f356819bd40 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:22:33.653 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:22:33.662+0000 7f356f6fd700 1 --2- 172.21.15.52:0/1726929725 >> [v2:172.21.15.52:6800/3513075805,v1:172.21.15.52:6801/3513075805] conn(0x7f35540b2880 0x7f35540b4d90 unknown :-1 s=CLOSED pgs=156 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:22:33.653 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:22:33.662+0000 7f356f6fd700 1 --2- 172.21.15.52:0/1726929725 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f3568107de0 0x7f356819c280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:22:33.653 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:22:33.662+0000 7f356f6fd700 1 -- 172.21.15.52:0/1726929725 >> 172.21.15.52:0/1726929725 conn(0x7f35680fede0 msgr2=0x7f3568109690 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:22:33.653 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:22:33.662+0000 7f356f6fd700 1 -- 172.21.15.52:0/1726929725 shutdown_connections 2024-05-27T07:22:33.653 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:22:33.662+0000 7f356f6fd700 1 -- 172.21.15.52:0/1726929725 wait complete. 2024-05-27T07:22:33.653 INFO:teuthology.orchestra.run.smithi052.stderr:listed 10 entries 2024-05-27T07:22:33.979 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:22:33 smithi052 ceph-mon[31955]: from='client.? 172.21.15.52:0/1726929725' entity='client.admin' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-05-27T07:22:34.102 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:22:33 smithi173 ceph-mon[37000]: from='client.? 172.21.15.52:0/1726929725' entity='client.admin' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-05-27T07:22:34.190 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:22:33 smithi105 ceph-mon[35977]: from='client.? 172.21.15.52:0/1726929725' entity='client.admin' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-05-27T07:22:34.319 INFO:tasks.cephfs.fuse_mount:Running fusermount -u on ubuntu@smithi052.front.sepia.ceph.com... 2024-05-27T07:22:34.320 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-27T07:22:34.320 DEBUG:teuthology.orchestra.run.smithi052:> sudo fusermount -u /home/ubuntu/cephtest/mnt.admin 2024-05-27T07:22:34.347 INFO:tasks.cephfs.fuse_mount.ceph-fuse.admin.smithi052.stderr:ceph-fuse[78281]: fuse finished with error 0 and tester_r 0 2024-05-27T07:22:34.349 INFO:teuthology.orchestra.run:waiting for 300 2024-05-27T07:22:35.102 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:22:34 smithi173 ceph-mon[37000]: pgmap v320: 129 pgs: 5 peering, 124 active+clean; 581 KiB data, 356 MiB used, 1.0 TiB / 1.0 TiB avail; 511 B/s wr, 0 op/s; 2 B/s, 0 objects/s recovering 2024-05-27T07:22:35.190 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:22:34 smithi105 ceph-mon[35977]: pgmap v320: 129 pgs: 5 peering, 124 active+clean; 581 KiB data, 356 MiB used, 1.0 TiB / 1.0 TiB avail; 511 B/s wr, 0 op/s; 2 B/s, 0 objects/s recovering 2024-05-27T07:22:35.229 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:22:34 smithi052 ceph-mon[31955]: pgmap v320: 129 pgs: 5 peering, 124 active+clean; 581 KiB data, 356 MiB used, 1.0 TiB / 1.0 TiB avail; 511 B/s wr, 0 op/s; 2 B/s, 0 objects/s recovering 2024-05-27T07:22:36.979 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:22:36 smithi052 ceph-mon[31955]: pgmap v321: 129 pgs: 129 active+clean; 581 KiB data, 356 MiB used, 1.0 TiB / 1.0 TiB avail; 639 B/s wr, 0 op/s; 2 B/s, 0 objects/s recovering 2024-05-27T07:22:37.102 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:22:36 smithi173 ceph-mon[37000]: pgmap v321: 129 pgs: 129 active+clean; 581 KiB data, 356 MiB used, 1.0 TiB / 1.0 TiB avail; 639 B/s wr, 0 op/s; 2 B/s, 0 objects/s recovering 2024-05-27T07:22:37.190 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:22:36 smithi105 ceph-mon[35977]: pgmap v321: 129 pgs: 129 active+clean; 581 KiB data, 356 MiB used, 1.0 TiB / 1.0 TiB avail; 639 B/s wr, 0 op/s; 2 B/s, 0 objects/s recovering 2024-05-27T07:22:39.102 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:22:38 smithi173 ceph-mon[37000]: pgmap v322: 129 pgs: 129 active+clean; 581 KiB data, 356 MiB used, 1.0 TiB / 1.0 TiB avail; 127 B/s wr, 0 op/s; 2 B/s, 0 objects/s recovering 2024-05-27T07:22:39.190 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:22:38 smithi105 ceph-mon[35977]: pgmap v322: 129 pgs: 129 active+clean; 581 KiB data, 356 MiB used, 1.0 TiB / 1.0 TiB avail; 127 B/s wr, 0 op/s; 2 B/s, 0 objects/s recovering 2024-05-27T07:22:39.229 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:22:38 smithi052 ceph-mon[31955]: pgmap v322: 129 pgs: 129 active+clean; 581 KiB data, 356 MiB used, 1.0 TiB / 1.0 TiB avail; 127 B/s wr, 0 op/s; 2 B/s, 0 objects/s recovering 2024-05-27T07:22:40.451 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi052.front.sepia.ceph.com 2024-05-27T07:22:40.451 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-27T07:22:40.451 DEBUG:teuthology.orchestra.run.smithi052:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.admin) 2024-05-27T07:22:40.483 DEBUG:teuthology.orchestra.run.smithi052:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:38f9d900c83c63799fbdbe61acc9a11b0d3554a6 shell --fsid 73df2636-1bf8-11ef-bc9b-c7b262605968 -- ceph osd dump --format=json 2024-05-27T07:22:40.784 INFO:teuthology.orchestra.run.smithi052.stderr:Inferring config /var/lib/ceph/73df2636-1bf8-11ef-bc9b-c7b262605968/mon.a/config 2024-05-27T07:22:41.102 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:22:40 smithi173 ceph-mon[37000]: pgmap v323: 129 pgs: 129 active+clean; 581 KiB data, 356 MiB used, 1.0 TiB / 1.0 TiB avail; 103 B/s wr, 0 op/s; 2 B/s, 0 objects/s recovering 2024-05-27T07:22:41.154 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:22:40 smithi052 ceph-mon[31955]: pgmap v323: 129 pgs: 129 active+clean; 581 KiB data, 356 MiB used, 1.0 TiB / 1.0 TiB avail; 103 B/s wr, 0 op/s; 2 B/s, 0 objects/s recovering 2024-05-27T07:22:41.190 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:22:40 smithi105 ceph-mon[35977]: pgmap v323: 129 pgs: 129 active+clean; 581 KiB data, 356 MiB used, 1.0 TiB / 1.0 TiB avail; 103 B/s wr, 0 op/s; 2 B/s, 0 objects/s recovering 2024-05-27T07:22:42.428 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:22:42.438+0000 7f9b49dc6700 1 -- 172.21.15.52:0/830436371 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f9b441045c0 msgr2=0x7f9b441049d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:22:42.428 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:22:42.438+0000 7f9b49dc6700 1 --2- 172.21.15.52:0/830436371 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f9b441045c0 0x7f9b441049d0 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7f9b340099c0 tx=0x7f9b34009cd0 comp rx=0 tx=0).stop 2024-05-27T07:22:42.428 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:22:42.438+0000 7f9b49dc6700 1 -- 172.21.15.52:0/830436371 shutdown_connections 2024-05-27T07:22:42.428 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:22:42.438+0000 7f9b49dc6700 1 --2- 172.21.15.52:0/830436371 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f9b44109840 0x7f9b441111a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:22:42.429 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:22:42.438+0000 7f9b49dc6700 1 --2- 172.21.15.52:0/830436371 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f9b441045c0 0x7f9b441049d0 unknown :-1 s=CLOSED pgs=214 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:22:42.429 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:22:42.438+0000 7f9b49dc6700 1 --2- 172.21.15.52:0/830436371 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f9b44108ef0 0x7f9b44109300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:22:42.429 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:22:42.438+0000 7f9b49dc6700 1 -- 172.21.15.52:0/830436371 >> 172.21.15.52:0/830436371 conn(0x7f9b440ffe90 msgr2=0x7f9b441022e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:22:42.429 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:22:42.438+0000 7f9b49dc6700 1 -- 172.21.15.52:0/830436371 shutdown_connections 2024-05-27T07:22:42.429 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:22:42.438+0000 7f9b49dc6700 1 -- 172.21.15.52:0/830436371 wait complete. 2024-05-27T07:22:42.429 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:22:42.438+0000 7f9b49dc6700 1 Processor -- start 2024-05-27T07:22:42.430 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:22:42.439+0000 7f9b49dc6700 1 -- start start 2024-05-27T07:22:42.431 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:22:42.439+0000 7f9b49dc6700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f9b441045c0 0x7f9b44076bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:22:42.431 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:22:42.439+0000 7f9b49dc6700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f9b44108ef0 0x7f9b44077100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:22:42.431 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:22:42.439+0000 7f9b49dc6700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f9b44109840 0x7f9b44077640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:22:42.431 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:22:42.439+0000 7f9b49dc6700 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x7f9b44114470 con 0x7f9b441045c0 2024-05-27T07:22:42.431 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:22:42.439+0000 7f9b49dc6700 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f9b441142f0 con 0x7f9b44108ef0 2024-05-27T07:22:42.431 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:22:42.439+0000 7f9b49dc6700 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f9b441145f0 con 0x7f9b44109840 2024-05-27T07:22:42.431 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:22:42.439+0000 7f9b437fe700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f9b441045c0 0x7f9b44076bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:22:42.431 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:22:42.439+0000 7f9b437fe700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f9b441045c0 0x7f9b44076bc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.52:60006/0 (socket says 172.21.15.52:60006) 2024-05-27T07:22:42.431 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:22:42.439+0000 7f9b437fe700 1 -- 172.21.15.52:0/345140363 learned_addr learned my addr 172.21.15.52:0/345140363 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-05-27T07:22:42.431 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:22:42.439+0000 7f9b43fff700 1 --2- 172.21.15.52:0/345140363 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f9b44109840 0x7f9b44077640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:22:42.431 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:22:42.439+0000 7f9b42ffd700 1 --2- 172.21.15.52:0/345140363 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f9b44108ef0 0x7f9b44077100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:22:42.432 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:22:42.440+0000 7f9b437fe700 1 -- 172.21.15.52:0/345140363 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f9b44109840 msgr2=0x7f9b44077640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:22:42.432 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:22:42.440+0000 7f9b437fe700 1 --2- 172.21.15.52:0/345140363 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f9b44109840 0x7f9b44077640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:22:42.432 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:22:42.440+0000 7f9b437fe700 1 -- 172.21.15.52:0/345140363 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f9b44108ef0 msgr2=0x7f9b44077100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:22:42.432 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:22:42.440+0000 7f9b437fe700 1 --2- 172.21.15.52:0/345140363 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f9b44108ef0 0x7f9b44077100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:22:42.432 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:22:42.440+0000 7f9b437fe700 1 -- 172.21.15.52:0/345140363 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9b441a9200 con 0x7f9b441045c0 2024-05-27T07:22:42.432 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:22:42.440+0000 7f9b437fe700 1 --2- 172.21.15.52:0/345140363 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f9b441045c0 0x7f9b44076bc0 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7f9b340098e0 tx=0x7f9b340057c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:22:42.432 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:22:42.440+0000 7f9b40ff9700 1 -- 172.21.15.52:0/345140363 <== mon.0 v2:172.21.15.52:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9b34027070 con 0x7f9b441045c0 2024-05-27T07:22:42.432 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:22:42.440+0000 7f9b49dc6700 1 -- 172.21.15.52:0/345140363 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9b441a9490 con 0x7f9b441045c0 2024-05-27T07:22:42.432 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:22:42.440+0000 7f9b40ff9700 1 -- 172.21.15.52:0/345140363 <== mon.0 v2:172.21.15.52:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f9b34003f20 con 0x7f9b441045c0 2024-05-27T07:22:42.433 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:22:42.440+0000 7f9b40ff9700 1 -- 172.21.15.52:0/345140363 <== mon.0 v2:172.21.15.52:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9b34022ca0 con 0x7f9b441045c0 2024-05-27T07:22:42.433 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:22:42.440+0000 7f9b49dc6700 1 -- 172.21.15.52:0/345140363 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9b441a9940 con 0x7f9b441045c0 2024-05-27T07:22:42.433 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:22:42.441+0000 7f9b49dc6700 1 -- 172.21.15.52:0/345140363 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9b24005330 con 0x7f9b441045c0 2024-05-27T07:22:42.433 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:22:42.443+0000 7f9b40ff9700 1 -- 172.21.15.52:0/345140363 <== mon.0 v2:172.21.15.52:3300/0 4 ==== mgrmap(e 16) ==== 149652+0+0 (secure 0 0 0) 0x7f9b3402a070 con 0x7f9b441045c0 2024-05-27T07:22:42.434 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:22:42.444+0000 7f9b40ff9700 1 --2- 172.21.15.52:0/345140363 >> [v2:172.21.15.52:6800/3513075805,v1:172.21.15.52:6801/3513075805] conn(0x7f9b300b2900 0x7f9b300b4e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:22:42.434 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:22:42.444+0000 7f9b42ffd700 1 --2- 172.21.15.52:0/345140363 >> [v2:172.21.15.52:6800/3513075805,v1:172.21.15.52:6801/3513075805] conn(0x7f9b300b2900 0x7f9b300b4e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:22:42.434 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:22:42.444+0000 7f9b40ff9700 1 -- 172.21.15.52:0/345140363 <== mon.0 v2:172.21.15.52:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 9092+0+0 (secure 0 0 0) 0x7f9b340e37f0 con 0x7f9b441045c0 2024-05-27T07:22:42.434 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:22:42.444+0000 7f9b42ffd700 1 --2- 172.21.15.52:0/345140363 >> [v2:172.21.15.52:6800/3513075805,v1:172.21.15.52:6801/3513075805] conn(0x7f9b300b2900 0x7f9b300b4e10 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7f9b44107700 tx=0x7f9b2c00b410 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:22:42.438 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:22:42.448+0000 7f9b40ff9700 1 -- 172.21.15.52:0/345140363 <== mon.0 v2:172.21.15.52:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191963 (secure 0 0 0) 0x7f9b34031080 con 0x7f9b441045c0 2024-05-27T07:22:42.658 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:22:42.667+0000 7f9b49dc6700 1 -- 172.21.15.52:0/345140363 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f9b240058f0 con 0x7f9b441045c0 2024-05-27T07:22:42.659 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:22:42.669+0000 7f9b40ff9700 1 -- 172.21.15.52:0/345140363 <== mon.0 v2:172.21.15.52:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v88) ==== 74+0+24654 (secure 0 0 0) 0x7f9b34013070 con 0x7f9b441045c0 2024-05-27T07:22:42.660 INFO:teuthology.orchestra.run.smithi052.stdout: 2024-05-27T07:22:42.660 INFO:teuthology.orchestra.run.smithi052.stdout:{"epoch":88,"fsid":"73df2636-1bf8-11ef-bc9b-c7b262605968","created":"2024-05-27T07:13:34.738032+0000","modified":"2024-05-27T07:22:30.789020+0000","last_up_change":"2024-05-27T07:20:45.777597+0000","last_in_change":"2024-05-27T07:20:25.314823+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-05-27T07:16:57.644034+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-05-27T07:21:43.917748+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"79","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-05-27T07:21:47.341876+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"79","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-05-27T07:21:57.707611+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"86","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"347b8a89-9dbe-4848-83b0-fd0e9374f33d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":82,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.52:6802","nonce":1626508899},{"type":"v1","addr":"172.21.15.52:6803","nonce":1626508899}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.52:6804","nonce":1626508899},{"type":"v1","addr":"172.21.15.52:6805","nonce":1626508899}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.52:6808","nonce":1626508899},{"type":"v1","addr":"172.21.15.52:6809","nonce":1626508899}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.52:6806","nonce":1626508899},{"type":"v1","addr":"172.21.15.52:6807","nonce":1626508899}]},"public_addr":"172.21.15.52:6803/1626508899","cluster_addr":"172.21.15.52:6805/1626508899","heartbeat_back_addr":"172.21.15.52:6809/1626508899","heartbeat_front_addr":"172.21.15.52:6807/1626508899","state":["exists","up"]},{"osd":1,"uuid":"7a3623d4-e087-40cb-9336-d4a5c52203d4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":82,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6800","nonce":2900093195},{"type":"v1","addr":"172.21.15.105:6801","nonce":2900093195}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6802","nonce":2900093195},{"type":"v1","addr":"172.21.15.105:6803","nonce":2900093195}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6806","nonce":2900093195},{"type":"v1","addr":"172.21.15.105:6807","nonce":2900093195}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6804","nonce":2900093195},{"type":"v1","addr":"172.21.15.105:6805","nonce":2900093195}]},"public_addr":"172.21.15.105:6801/2900093195","cluster_addr":"172.21.15.105:6803/2900093195","heartbeat_back_addr":"172.21.15.105:6807/2900093195","heartbeat_front_addr":"172.21.15.105:6805/2900093195","state":["exists","up"]},{"osd":2,"uuid":"41a9df5b-882a-47da-bb26-7f4d26ccf6c1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6800","nonce":2251189314},{"type":"v1","addr":"172.21.15.173:6801","nonce":2251189314}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6802","nonce":2251189314},{"type":"v1","addr":"172.21.15.173:6803","nonce":2251189314}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6806","nonce":2251189314},{"type":"v1","addr":"172.21.15.173:6807","nonce":2251189314}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6804","nonce":2251189314},{"type":"v1","addr":"172.21.15.173:6805","nonce":2251189314}]},"public_addr":"172.21.15.173:6801/2251189314","cluster_addr":"172.21.15.173:6803/2251189314","heartbeat_back_addr":"172.21.15.173:6807/2251189314","heartbeat_front_addr":"172.21.15.173:6805/2251189314","state":["exists","up"]},{"osd":3,"uuid":"f1245439-42a5-46c5-a998-e0a78f9df3fb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":26,"up_thru":82,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.52:6810","nonce":57920794},{"type":"v1","addr":"172.21.15.52:6811","nonce":57920794}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.52:6812","nonce":57920794},{"type":"v1","addr":"172.21.15.52:6813","nonce":57920794}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.52:6816","nonce":57920794},{"type":"v1","addr":"172.21.15.52:6817","nonce":57920794}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.52:6814","nonce":57920794},{"type":"v1","addr":"172.21.15.52:6815","nonce":57920794}]},"public_addr":"172.21.15.52:6811/57920794","cluster_addr":"172.21.15.52:6813/57920794","heartbeat_back_addr":"172.21.15.52:6817/57920794","heartbeat_front_addr":"172.21.15.52:6815/57920794","state":["exists","up"]},{"osd":4,"uuid":"2592d1cc-1726-4359-83d8-a2e6314b54ec","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":32,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6808","nonce":3441134818},{"type":"v1","addr":"172.21.15.105:6809","nonce":3441134818}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6810","nonce":3441134818},{"type":"v1","addr":"172.21.15.105:6811","nonce":3441134818}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6814","nonce":3441134818},{"type":"v1","addr":"172.21.15.105:6815","nonce":3441134818}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6812","nonce":3441134818},{"type":"v1","addr":"172.21.15.105:6813","nonce":3441134818}]},"public_addr":"172.21.15.105:6809/3441134818","cluster_addr":"172.21.15.105:6811/3441134818","heartbeat_back_addr":"172.21.15.105:6815/3441134818","heartbeat_front_addr":"172.21.15.105:6813/3441134818","state":["exists","up"]},{"osd":5,"uuid":"27e98ee2-4a92-4972-b2a3-959d0ed90981","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":37,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6808","nonce":2989239729},{"type":"v1","addr":"172.21.15.173:6809","nonce":2989239729}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6810","nonce":2989239729},{"type":"v1","addr":"172.21.15.173:6811","nonce":2989239729}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6814","nonce":2989239729},{"type":"v1","addr":"172.21.15.173:6815","nonce":2989239729}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6812","nonce":2989239729},{"type":"v1","addr":"172.21.15.173:6813","nonce":2989239729}]},"public_addr":"172.21.15.173:6809/2989239729","cluster_addr":"172.21.15.173:6811/2989239729","heartbeat_back_addr":"172.21.15.173:6815/2989239729","heartbeat_front_addr":"172.21.15.173:6813/2989239729","state":["exists","up"]},{"osd":6,"uuid":"2b0ec0cb-f438-4b2c-8642-0778ed374e9a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":43,"up_thru":82,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.52:6818","nonce":4244423065},{"type":"v1","addr":"172.21.15.52:6819","nonce":4244423065}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.52:6820","nonce":4244423065},{"type":"v1","addr":"172.21.15.52:6821","nonce":4244423065}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.52:6824","nonce":4244423065},{"type":"v1","addr":"172.21.15.52:6825","nonce":4244423065}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.52:6822","nonce":4244423065},{"type":"v1","addr":"172.21.15.52:6823","nonce":4244423065}]},"public_addr":"172.21.15.52:6819/4244423065","cluster_addr":"172.21.15.52:6821/4244423065","heartbeat_back_addr":"172.21.15.52:6825/4244423065","heartbeat_front_addr":"172.21.15.52:6823/4244423065","state":["exists","up"]},{"osd":7,"uuid":"cb2a7a6b-8015-43ea-865a-be144e45c51a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":48,"up_thru":82,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6816","nonce":2063611552},{"type":"v1","addr":"172.21.15.105:6817","nonce":2063611552}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6818","nonce":2063611552},{"type":"v1","addr":"172.21.15.105:6819","nonce":2063611552}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6822","nonce":2063611552},{"type":"v1","addr":"172.21.15.105:6823","nonce":2063611552}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6820","nonce":2063611552},{"type":"v1","addr":"172.21.15.105:6821","nonce":2063611552}]},"public_addr":"172.21.15.105:6817/2063611552","cluster_addr":"172.21.15.105:6819/2063611552","heartbeat_back_addr":"172.21.15.105:6823/2063611552","heartbeat_front_addr":"172.21.15.105:6821/2063611552","state":["exists","up"]},{"osd":8,"uuid":"c2fc565d-a0b9-4396-a209-bd884cddcfe3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":54,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6816","nonce":4131399852},{"type":"v1","addr":"172.21.15.173:6817","nonce":4131399852}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6818","nonce":4131399852},{"type":"v1","addr":"172.21.15.173:6819","nonce":4131399852}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6822","nonce":4131399852},{"type":"v1","addr":"172.21.15.173:6823","nonce":4131399852}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6820","nonce":4131399852},{"type":"v1","addr":"172.21.15.173:6821","nonce":4131399852}]},"public_addr":"172.21.15.173:6817/4131399852","cluster_addr":"172.21.15.173:6819/4131399852","heartbeat_back_addr":"172.21.15.173:6823/4131399852","heartbeat_front_addr":"172.21.15.173:6821/4131399852","state":["exists","up"]},{"osd":9,"uuid":"78fa7f17-1dad-4262-be40-31fa82e75520","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":60,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.52:6826","nonce":479119778},{"type":"v1","addr":"172.21.15.52:6827","nonce":479119778}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.52:6828","nonce":479119778},{"type":"v1","addr":"172.21.15.52:6829","nonce":479119778}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.52:6832","nonce":479119778},{"type":"v1","addr":"172.21.15.52:6833","nonce":479119778}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.52:6830","nonce":479119778},{"type":"v1","addr":"172.21.15.52:6831","nonce":479119778}]},"public_addr":"172.21.15.52:6827/479119778","cluster_addr":"172.21.15.52:6829/479119778","heartbeat_back_addr":"172.21.15.52:6833/479119778","heartbeat_front_addr":"172.21.15.52:6831/479119778","state":["exists","up"]},{"osd":10,"uuid":"e6f8e753-e15c-45b5-8ee9-d50f384559f9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":65,"up_thru":76,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6824","nonce":4032386628},{"type":"v1","addr":"172.21.15.105:6825","nonce":4032386628}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6826","nonce":4032386628},{"type":"v1","addr":"172.21.15.105:6827","nonce":4032386628}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6830","nonce":4032386628},{"type":"v1","addr":"172.21.15.105:6831","nonce":4032386628}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6828","nonce":4032386628},{"type":"v1","addr":"172.21.15.105:6829","nonce":4032386628}]},"public_addr":"172.21.15.105:6825/4032386628","cluster_addr":"172.21.15.105:6827/4032386628","heartbeat_back_addr":"172.21.15.105:6831/4032386628","heartbeat_front_addr":"172.21.15.105:6829/4032386628","state":["exists","up"]},{"osd":11,"uuid":"164572bf-67ed-4e8e-b32d-d7a2e8da607b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":71,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6824","nonce":1816308401},{"type":"v1","addr":"172.21.15.173:6825","nonce":1816308401}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6826","nonce":1816308401},{"type":"v1","addr":"172.21.15.173:6827","nonce":1816308401}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6830","nonce":1816308401},{"type":"v1","addr":"172.21.15.173:6831","nonce":1816308401}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6828","nonce":1816308401},{"type":"v1","addr":"172.21.15.173:6829","nonce":1816308401}]},"public_addr":"172.21.15.173:6825/1816308401","cluster_addr":"172.21.15.173:6827/1816308401","heartbeat_back_addr":"172.21.15.173:6831/1816308401","heartbeat_front_addr":"172.21.15.173:6829/1816308401","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-27T07:16:06.279220+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-27T07:16:29.668167+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-27T07:16:53.903808+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-27T07:17:18.787372+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-27T07:17:43.480784+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-27T07:18:08.816030+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-27T07:18:34.076148+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-27T07:19:00.283468+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-27T07:19:25.403023+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-27T07:19:52.139173+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-27T07:20:17.530304+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-27T07:20:43.665632+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.f","mappings":[{"from":8,"to":11}]},{"pgid":"2.10","mappings":[{"from":2,"to":4}]},{"pgid":"3.0","mappings":[{"from":5,"to":9}]},{"pgid":"3.1b","mappings":[{"from":4,"to":8}]},{"pgid":"3.2c","mappings":[{"from":4,"to":9}]},{"pgid":"3.3c","mappings":[{"from":4,"to":9}]},{"pgid":"4.e","mappings":[{"from":9,"to":11}]},{"pgid":"4.12","mappings":[{"from":5,"to":10}]},{"pgid":"4.19","mappings":[{"from":9,"to":8}]},{"pgid":"4.1f","mappings":[{"from":9,"to":8}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.52:6801/1631683163":"2024-05-28T07:14:29.544633+0000","172.21.15.52:0/2776144659":"2024-05-28T07:14:29.544633+0000","172.21.15.52:6800/1631683163":"2024-05-28T07:14:29.544633+0000","172.21.15.52:0/3675529097":"2024-05-28T07:14:01.960550+0000","172.21.15.52:0/727808107":"2024-05-28T07:14:01.960550+0000","172.21.15.52:0/3839436833":"2024-05-28T07:14:29.544633+0000","172.21.15.52:6800/587662512":"2024-05-28T07:14:01.960550+0000","172.21.15.52:6801/587662512":"2024-05-28T07:14:01.960550+0000","172.21.15.52:0/853867465":"2024-05-28T07:14:01.960550+0000","172.21.15.52:0/3480918279":"2024-05-28T07:14:29.544633+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-05-27T07:22:42.661 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:22:42.671+0000 7f9b49dc6700 1 -- 172.21.15.52:0/345140363 >> [v2:172.21.15.52:6800/3513075805,v1:172.21.15.52:6801/3513075805] conn(0x7f9b300b2900 msgr2=0x7f9b300b4e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:22:42.661 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:22:42.671+0000 7f9b49dc6700 1 --2- 172.21.15.52:0/345140363 >> [v2:172.21.15.52:6800/3513075805,v1:172.21.15.52:6801/3513075805] conn(0x7f9b300b2900 0x7f9b300b4e10 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7f9b44107700 tx=0x7f9b2c00b410 comp rx=0 tx=0).stop 2024-05-27T07:22:42.662 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:22:42.671+0000 7f9b49dc6700 1 -- 172.21.15.52:0/345140363 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f9b441045c0 msgr2=0x7f9b44076bc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:22:42.662 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:22:42.671+0000 7f9b49dc6700 1 --2- 172.21.15.52:0/345140363 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f9b441045c0 0x7f9b44076bc0 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7f9b340098e0 tx=0x7f9b340057c0 comp rx=0 tx=0).stop 2024-05-27T07:22:42.662 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:22:42.671+0000 7f9b49dc6700 1 -- 172.21.15.52:0/345140363 shutdown_connections 2024-05-27T07:22:42.662 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:22:42.671+0000 7f9b49dc6700 1 --2- 172.21.15.52:0/345140363 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f9b44109840 0x7f9b44077640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:22:42.662 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:22:42.671+0000 7f9b49dc6700 1 --2- 172.21.15.52:0/345140363 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f9b441045c0 0x7f9b44076bc0 unknown :-1 s=CLOSED pgs=215 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:22:42.662 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:22:42.671+0000 7f9b49dc6700 1 --2- 172.21.15.52:0/345140363 >> [v2:172.21.15.52:6800/3513075805,v1:172.21.15.52:6801/3513075805] conn(0x7f9b300b2900 0x7f9b300b4e10 unknown :-1 s=CLOSED pgs=157 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:22:42.662 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:22:42.671+0000 7f9b49dc6700 1 --2- 172.21.15.52:0/345140363 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f9b44108ef0 0x7f9b44077100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:22:42.662 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:22:42.671+0000 7f9b49dc6700 1 -- 172.21.15.52:0/345140363 >> 172.21.15.52:0/345140363 conn(0x7f9b440ffe90 msgr2=0x7f9b44102e20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:22:42.662 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:22:42.672+0000 7f9b49dc6700 1 -- 172.21.15.52:0/345140363 shutdown_connections 2024-05-27T07:22:42.662 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:22:42.672+0000 7f9b49dc6700 1 -- 172.21.15.52:0/345140363 wait complete. 2024-05-27T07:22:43.017 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:22:42 smithi052 ceph-mon[31955]: pgmap v324: 129 pgs: 129 active+clean; 581 KiB data, 356 MiB used, 1.0 TiB / 1.0 TiB avail; 94 B/s wr, 0 op/s; 2 B/s, 0 objects/s recovering 2024-05-27T07:22:43.017 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:22:42 smithi052 ceph-mon[31955]: from='client.? 172.21.15.52:0/345140363' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-05-27T07:22:43.102 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:22:42 smithi173 ceph-mon[37000]: pgmap v324: 129 pgs: 129 active+clean; 581 KiB data, 356 MiB used, 1.0 TiB / 1.0 TiB avail; 94 B/s wr, 0 op/s; 2 B/s, 0 objects/s recovering 2024-05-27T07:22:43.102 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:22:42 smithi173 ceph-mon[37000]: from='client.? 172.21.15.52:0/345140363' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-05-27T07:22:43.190 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:22:42 smithi105 ceph-mon[35977]: pgmap v324: 129 pgs: 129 active+clean; 581 KiB data, 356 MiB used, 1.0 TiB / 1.0 TiB avail; 94 B/s wr, 0 op/s; 2 B/s, 0 objects/s recovering 2024-05-27T07:22:43.190 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:22:42 smithi105 ceph-mon[35977]: from='client.? 172.21.15.52:0/345140363' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-05-27T07:22:43.351 DEBUG:teuthology.orchestra.run.smithi052:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:38f9d900c83c63799fbdbe61acc9a11b0d3554a6 shell --fsid 73df2636-1bf8-11ef-bc9b-c7b262605968 -- ceph osd dump --format=json 2024-05-27T07:22:43.606 INFO:teuthology.orchestra.run.smithi052.stderr:Inferring config /var/lib/ceph/73df2636-1bf8-11ef-bc9b-c7b262605968/mon.a/config 2024-05-27T07:22:45.084 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:22:44 smithi052 ceph-mon[31955]: pgmap v325: 129 pgs: 129 active+clean; 581 KiB data, 356 MiB used, 1.0 TiB / 1.0 TiB avail; 85 B/s wr, 0 op/s; 0 B/s, 0 objects/s recovering 2024-05-27T07:22:45.102 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:22:44 smithi173 ceph-mon[37000]: pgmap v325: 129 pgs: 129 active+clean; 581 KiB data, 356 MiB used, 1.0 TiB / 1.0 TiB avail; 85 B/s wr, 0 op/s; 0 B/s, 0 objects/s recovering 2024-05-27T07:22:45.190 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:22:44 smithi105 ceph-mon[35977]: pgmap v325: 129 pgs: 129 active+clean; 581 KiB data, 356 MiB used, 1.0 TiB / 1.0 TiB avail; 85 B/s wr, 0 op/s; 0 B/s, 0 objects/s recovering 2024-05-27T07:22:45.308 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:22:45.318+0000 7fa56e925700 1 -- 172.21.15.52:0/507789114 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fa568108f10 msgr2=0x7fa568109320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:22:45.308 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:22:45.318+0000 7fa56e925700 1 --2- 172.21.15.52:0/507789114 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fa568108f10 0x7fa568109320 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7fa5600099c0 tx=0x7fa560009cd0 comp rx=0 tx=0).stop 2024-05-27T07:22:45.309 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:22:45.318+0000 7fa56e925700 1 -- 172.21.15.52:0/507789114 shutdown_connections 2024-05-27T07:22:45.309 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:22:45.318+0000 7fa56e925700 1 --2- 172.21.15.52:0/507789114 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fa568109860 0x7fa5681111c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:22:45.309 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:22:45.318+0000 7fa56e925700 1 --2- 172.21.15.52:0/507789114 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fa5681045e0 0x7fa5681049f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:22:45.309 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:22:45.318+0000 7fa56e925700 1 --2- 172.21.15.52:0/507789114 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fa568108f10 0x7fa568109320 unknown :-1 s=CLOSED pgs=216 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:22:45.309 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:22:45.318+0000 7fa56e925700 1 -- 172.21.15.52:0/507789114 >> 172.21.15.52:0/507789114 conn(0x7fa5680fffd0 msgr2=0x7fa5681023f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:22:45.309 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:22:45.318+0000 7fa56e925700 1 -- 172.21.15.52:0/507789114 shutdown_connections 2024-05-27T07:22:45.309 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:22:45.318+0000 7fa56e925700 1 -- 172.21.15.52:0/507789114 wait complete. 2024-05-27T07:22:45.311 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:22:45.319+0000 7fa56e925700 1 Processor -- start 2024-05-27T07:22:45.311 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:22:45.319+0000 7fa56e925700 1 -- start start 2024-05-27T07:22:45.311 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:22:45.320+0000 7fa56e925700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fa5681045e0 0x7fa568075ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:22:45.311 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:22:45.320+0000 7fa56e925700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fa568108f10 0x7fa568075ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:22:45.311 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:22:45.320+0000 7fa56e925700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fa568109860 0x7fa568076530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:22:45.311 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:22:45.320+0000 7fa56e925700 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x7fa568114470 con 0x7fa5681045e0 2024-05-27T07:22:45.312 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:22:45.320+0000 7fa56e925700 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7fa5681142f0 con 0x7fa568109860 2024-05-27T07:22:45.312 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:22:45.320+0000 7fa56cec2700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fa568109860 0x7fa568076530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:22:45.312 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:22:45.320+0000 7fa567fff700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fa5681045e0 0x7fa568075ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:22:45.312 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:22:45.320+0000 7fa56e925700 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7fa5681145f0 con 0x7fa568108f10 2024-05-27T07:22:45.312 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:22:45.320+0000 7fa56cec2700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fa568109860 0x7fa568076530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.52:44008/0 (socket says 172.21.15.52:44008) 2024-05-27T07:22:45.312 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:22:45.320+0000 7fa56cec2700 1 -- 172.21.15.52:0/2819804616 learned_addr learned my addr 172.21.15.52:0/2819804616 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-05-27T07:22:45.312 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:22:45.320+0000 7fa5677fe700 1 --2- 172.21.15.52:0/2819804616 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fa568108f10 0x7fa568075ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:22:45.312 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:22:45.320+0000 7fa56cec2700 1 -- 172.21.15.52:0/2819804616 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fa568108f10 msgr2=0x7fa568075ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:22:45.312 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:22:45.320+0000 7fa56cec2700 1 --2- 172.21.15.52:0/2819804616 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fa568108f10 0x7fa568075ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:22:45.312 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:22:45.320+0000 7fa56cec2700 1 -- 172.21.15.52:0/2819804616 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fa5681045e0 msgr2=0x7fa568075ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:22:45.312 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:22:45.320+0000 7fa56cec2700 1 --2- 172.21.15.52:0/2819804616 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fa5681045e0 0x7fa568075ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:22:45.312 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:22:45.320+0000 7fa56cec2700 1 -- 172.21.15.52:0/2819804616 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa5681a91f0 con 0x7fa568109860 2024-05-27T07:22:45.312 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:22:45.320+0000 7fa567fff700 1 --2- 172.21.15.52:0/2819804616 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fa5681045e0 0x7fa568075ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-27T07:22:45.313 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:22:45.321+0000 7fa56cec2700 1 --2- 172.21.15.52:0/2819804616 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fa568109860 0x7fa568076530 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7fa55000e940 tx=0x7fa55000ec50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:22:45.313 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:22:45.321+0000 7fa5657fa700 1 -- 172.21.15.52:0/2819804616 <== mon.2 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa550004a30 con 0x7fa568109860 2024-05-27T07:22:45.313 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:22:45.321+0000 7fa56e925700 1 -- 172.21.15.52:0/2819804616 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa5681a94e0 con 0x7fa568109860 2024-05-27T07:22:45.313 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:22:45.321+0000 7fa5657fa700 1 -- 172.21.15.52:0/2819804616 <== mon.2 v2:172.21.15.105:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fa5500054d0 con 0x7fa568109860 2024-05-27T07:22:45.313 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:22:45.321+0000 7fa5657fa700 1 -- 172.21.15.52:0/2819804616 <== mon.2 v2:172.21.15.105:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa55000ccf0 con 0x7fa568109860 2024-05-27T07:22:45.313 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:22:45.321+0000 7fa56e925700 1 -- 172.21.15.52:0/2819804616 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa5681a99f0 con 0x7fa568109860 2024-05-27T07:22:45.313 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:22:45.322+0000 7fa56e925700 1 -- 172.21.15.52:0/2819804616 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa5680512b0 con 0x7fa568109860 2024-05-27T07:22:45.313 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:22:45.324+0000 7fa5657fa700 1 -- 172.21.15.52:0/2819804616 <== mon.2 v2:172.21.15.105:3300/0 4 ==== mgrmap(e 16) ==== 149652+0+0 (secure 0 0 0) 0x7fa550005000 con 0x7fa568109860 2024-05-27T07:22:45.314 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:22:45.324+0000 7fa5657fa700 1 --2- 172.21.15.52:0/2819804616 >> [v2:172.21.15.52:6800/3513075805,v1:172.21.15.52:6801/3513075805] conn(0x7fa5480b2900 0x7fa5480b4e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:22:45.314 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:22:45.325+0000 7fa567fff700 1 --2- 172.21.15.52:0/2819804616 >> [v2:172.21.15.52:6800/3513075805,v1:172.21.15.52:6801/3513075805] conn(0x7fa5480b2900 0x7fa5480b4e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:22:45.314 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:22:45.325+0000 7fa5657fa700 1 -- 172.21.15.52:0/2819804616 <== mon.2 v2:172.21.15.105:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 9092+0+0 (secure 0 0 0) 0x7fa5500e2880 con 0x7fa568109860 2024-05-27T07:22:45.315 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:22:45.325+0000 7fa567fff700 1 --2- 172.21.15.52:0/2819804616 >> [v2:172.21.15.52:6800/3513075805,v1:172.21.15.52:6801/3513075805] conn(0x7fa5480b2900 0x7fa5480b4e10 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7fa558009480 tx=0x7fa55800b040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:22:45.318 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:22:45.328+0000 7fa5657fa700 1 -- 172.21.15.52:0/2819804616 <== mon.2 v2:172.21.15.105:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191963 (secure 0 0 0) 0x7fa550054df0 con 0x7fa568109860 2024-05-27T07:22:45.576 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:22:45.586+0000 7fa56e925700 1 -- 172.21.15.52:0/2819804616 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fa56802e5e0 con 0x7fa568109860 2024-05-27T07:22:45.577 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:22:45.588+0000 7fa5657fa700 1 -- 172.21.15.52:0/2819804616 <== mon.2 v2:172.21.15.105:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v88) ==== 74+0+24654 (secure 0 0 0) 0x7fa5500052f0 con 0x7fa568109860 2024-05-27T07:22:45.578 INFO:teuthology.orchestra.run.smithi052.stdout: 2024-05-27T07:22:45.578 INFO:teuthology.orchestra.run.smithi052.stdout:{"epoch":88,"fsid":"73df2636-1bf8-11ef-bc9b-c7b262605968","created":"2024-05-27T07:13:34.738032+0000","modified":"2024-05-27T07:22:30.789020+0000","last_up_change":"2024-05-27T07:20:45.777597+0000","last_in_change":"2024-05-27T07:20:25.314823+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-05-27T07:16:57.644034+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-05-27T07:21:43.917748+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"79","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-05-27T07:21:47.341876+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"79","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-05-27T07:21:57.707611+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"86","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"347b8a89-9dbe-4848-83b0-fd0e9374f33d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":82,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.52:6802","nonce":1626508899},{"type":"v1","addr":"172.21.15.52:6803","nonce":1626508899}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.52:6804","nonce":1626508899},{"type":"v1","addr":"172.21.15.52:6805","nonce":1626508899}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.52:6808","nonce":1626508899},{"type":"v1","addr":"172.21.15.52:6809","nonce":1626508899}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.52:6806","nonce":1626508899},{"type":"v1","addr":"172.21.15.52:6807","nonce":1626508899}]},"public_addr":"172.21.15.52:6803/1626508899","cluster_addr":"172.21.15.52:6805/1626508899","heartbeat_back_addr":"172.21.15.52:6809/1626508899","heartbeat_front_addr":"172.21.15.52:6807/1626508899","state":["exists","up"]},{"osd":1,"uuid":"7a3623d4-e087-40cb-9336-d4a5c52203d4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":82,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6800","nonce":2900093195},{"type":"v1","addr":"172.21.15.105:6801","nonce":2900093195}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6802","nonce":2900093195},{"type":"v1","addr":"172.21.15.105:6803","nonce":2900093195}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6806","nonce":2900093195},{"type":"v1","addr":"172.21.15.105:6807","nonce":2900093195}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6804","nonce":2900093195},{"type":"v1","addr":"172.21.15.105:6805","nonce":2900093195}]},"public_addr":"172.21.15.105:6801/2900093195","cluster_addr":"172.21.15.105:6803/2900093195","heartbeat_back_addr":"172.21.15.105:6807/2900093195","heartbeat_front_addr":"172.21.15.105:6805/2900093195","state":["exists","up"]},{"osd":2,"uuid":"41a9df5b-882a-47da-bb26-7f4d26ccf6c1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6800","nonce":2251189314},{"type":"v1","addr":"172.21.15.173:6801","nonce":2251189314}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6802","nonce":2251189314},{"type":"v1","addr":"172.21.15.173:6803","nonce":2251189314}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6806","nonce":2251189314},{"type":"v1","addr":"172.21.15.173:6807","nonce":2251189314}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6804","nonce":2251189314},{"type":"v1","addr":"172.21.15.173:6805","nonce":2251189314}]},"public_addr":"172.21.15.173:6801/2251189314","cluster_addr":"172.21.15.173:6803/2251189314","heartbeat_back_addr":"172.21.15.173:6807/2251189314","heartbeat_front_addr":"172.21.15.173:6805/2251189314","state":["exists","up"]},{"osd":3,"uuid":"f1245439-42a5-46c5-a998-e0a78f9df3fb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":26,"up_thru":82,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.52:6810","nonce":57920794},{"type":"v1","addr":"172.21.15.52:6811","nonce":57920794}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.52:6812","nonce":57920794},{"type":"v1","addr":"172.21.15.52:6813","nonce":57920794}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.52:6816","nonce":57920794},{"type":"v1","addr":"172.21.15.52:6817","nonce":57920794}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.52:6814","nonce":57920794},{"type":"v1","addr":"172.21.15.52:6815","nonce":57920794}]},"public_addr":"172.21.15.52:6811/57920794","cluster_addr":"172.21.15.52:6813/57920794","heartbeat_back_addr":"172.21.15.52:6817/57920794","heartbeat_front_addr":"172.21.15.52:6815/57920794","state":["exists","up"]},{"osd":4,"uuid":"2592d1cc-1726-4359-83d8-a2e6314b54ec","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":32,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6808","nonce":3441134818},{"type":"v1","addr":"172.21.15.105:6809","nonce":3441134818}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6810","nonce":3441134818},{"type":"v1","addr":"172.21.15.105:6811","nonce":3441134818}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6814","nonce":3441134818},{"type":"v1","addr":"172.21.15.105:6815","nonce":3441134818}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6812","nonce":3441134818},{"type":"v1","addr":"172.21.15.105:6813","nonce":3441134818}]},"public_addr":"172.21.15.105:6809/3441134818","cluster_addr":"172.21.15.105:6811/3441134818","heartbeat_back_addr":"172.21.15.105:6815/3441134818","heartbeat_front_addr":"172.21.15.105:6813/3441134818","state":["exists","up"]},{"osd":5,"uuid":"27e98ee2-4a92-4972-b2a3-959d0ed90981","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":37,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6808","nonce":2989239729},{"type":"v1","addr":"172.21.15.173:6809","nonce":2989239729}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6810","nonce":2989239729},{"type":"v1","addr":"172.21.15.173:6811","nonce":2989239729}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6814","nonce":2989239729},{"type":"v1","addr":"172.21.15.173:6815","nonce":2989239729}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6812","nonce":2989239729},{"type":"v1","addr":"172.21.15.173:6813","nonce":2989239729}]},"public_addr":"172.21.15.173:6809/2989239729","cluster_addr":"172.21.15.173:6811/2989239729","heartbeat_back_addr":"172.21.15.173:6815/2989239729","heartbeat_front_addr":"172.21.15.173:6813/2989239729","state":["exists","up"]},{"osd":6,"uuid":"2b0ec0cb-f438-4b2c-8642-0778ed374e9a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":43,"up_thru":82,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.52:6818","nonce":4244423065},{"type":"v1","addr":"172.21.15.52:6819","nonce":4244423065}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.52:6820","nonce":4244423065},{"type":"v1","addr":"172.21.15.52:6821","nonce":4244423065}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.52:6824","nonce":4244423065},{"type":"v1","addr":"172.21.15.52:6825","nonce":4244423065}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.52:6822","nonce":4244423065},{"type":"v1","addr":"172.21.15.52:6823","nonce":4244423065}]},"public_addr":"172.21.15.52:6819/4244423065","cluster_addr":"172.21.15.52:6821/4244423065","heartbeat_back_addr":"172.21.15.52:6825/4244423065","heartbeat_front_addr":"172.21.15.52:6823/4244423065","state":["exists","up"]},{"osd":7,"uuid":"cb2a7a6b-8015-43ea-865a-be144e45c51a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":48,"up_thru":82,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6816","nonce":2063611552},{"type":"v1","addr":"172.21.15.105:6817","nonce":2063611552}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6818","nonce":2063611552},{"type":"v1","addr":"172.21.15.105:6819","nonce":2063611552}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6822","nonce":2063611552},{"type":"v1","addr":"172.21.15.105:6823","nonce":2063611552}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6820","nonce":2063611552},{"type":"v1","addr":"172.21.15.105:6821","nonce":2063611552}]},"public_addr":"172.21.15.105:6817/2063611552","cluster_addr":"172.21.15.105:6819/2063611552","heartbeat_back_addr":"172.21.15.105:6823/2063611552","heartbeat_front_addr":"172.21.15.105:6821/2063611552","state":["exists","up"]},{"osd":8,"uuid":"c2fc565d-a0b9-4396-a209-bd884cddcfe3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":54,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6816","nonce":4131399852},{"type":"v1","addr":"172.21.15.173:6817","nonce":4131399852}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6818","nonce":4131399852},{"type":"v1","addr":"172.21.15.173:6819","nonce":4131399852}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6822","nonce":4131399852},{"type":"v1","addr":"172.21.15.173:6823","nonce":4131399852}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6820","nonce":4131399852},{"type":"v1","addr":"172.21.15.173:6821","nonce":4131399852}]},"public_addr":"172.21.15.173:6817/4131399852","cluster_addr":"172.21.15.173:6819/4131399852","heartbeat_back_addr":"172.21.15.173:6823/4131399852","heartbeat_front_addr":"172.21.15.173:6821/4131399852","state":["exists","up"]},{"osd":9,"uuid":"78fa7f17-1dad-4262-be40-31fa82e75520","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":60,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.52:6826","nonce":479119778},{"type":"v1","addr":"172.21.15.52:6827","nonce":479119778}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.52:6828","nonce":479119778},{"type":"v1","addr":"172.21.15.52:6829","nonce":479119778}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.52:6832","nonce":479119778},{"type":"v1","addr":"172.21.15.52:6833","nonce":479119778}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.52:6830","nonce":479119778},{"type":"v1","addr":"172.21.15.52:6831","nonce":479119778}]},"public_addr":"172.21.15.52:6827/479119778","cluster_addr":"172.21.15.52:6829/479119778","heartbeat_back_addr":"172.21.15.52:6833/479119778","heartbeat_front_addr":"172.21.15.52:6831/479119778","state":["exists","up"]},{"osd":10,"uuid":"e6f8e753-e15c-45b5-8ee9-d50f384559f9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":65,"up_thru":76,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6824","nonce":4032386628},{"type":"v1","addr":"172.21.15.105:6825","nonce":4032386628}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6826","nonce":4032386628},{"type":"v1","addr":"172.21.15.105:6827","nonce":4032386628}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6830","nonce":4032386628},{"type":"v1","addr":"172.21.15.105:6831","nonce":4032386628}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6828","nonce":4032386628},{"type":"v1","addr":"172.21.15.105:6829","nonce":4032386628}]},"public_addr":"172.21.15.105:6825/4032386628","cluster_addr":"172.21.15.105:6827/4032386628","heartbeat_back_addr":"172.21.15.105:6831/4032386628","heartbeat_front_addr":"172.21.15.105:6829/4032386628","state":["exists","up"]},{"osd":11,"uuid":"164572bf-67ed-4e8e-b32d-d7a2e8da607b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":71,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6824","nonce":1816308401},{"type":"v1","addr":"172.21.15.173:6825","nonce":1816308401}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6826","nonce":1816308401},{"type":"v1","addr":"172.21.15.173:6827","nonce":1816308401}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6830","nonce":1816308401},{"type":"v1","addr":"172.21.15.173:6831","nonce":1816308401}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6828","nonce":1816308401},{"type":"v1","addr":"172.21.15.173:6829","nonce":1816308401}]},"public_addr":"172.21.15.173:6825/1816308401","cluster_addr":"172.21.15.173:6827/1816308401","heartbeat_back_addr":"172.21.15.173:6831/1816308401","heartbeat_front_addr":"172.21.15.173:6829/1816308401","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-27T07:16:06.279220+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-27T07:16:29.668167+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-27T07:16:53.903808+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-27T07:17:18.787372+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-27T07:17:43.480784+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-27T07:18:08.816030+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-27T07:18:34.076148+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-27T07:19:00.283468+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-27T07:19:25.403023+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-27T07:19:52.139173+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-27T07:20:17.530304+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-27T07:20:43.665632+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.f","mappings":[{"from":8,"to":11}]},{"pgid":"2.10","mappings":[{"from":2,"to":4}]},{"pgid":"3.0","mappings":[{"from":5,"to":9}]},{"pgid":"3.1b","mappings":[{"from":4,"to":8}]},{"pgid":"3.2c","mappings":[{"from":4,"to":9}]},{"pgid":"3.3c","mappings":[{"from":4,"to":9}]},{"pgid":"4.e","mappings":[{"from":9,"to":11}]},{"pgid":"4.12","mappings":[{"from":5,"to":10}]},{"pgid":"4.19","mappings":[{"from":9,"to":8}]},{"pgid":"4.1f","mappings":[{"from":9,"to":8}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.52:6801/1631683163":"2024-05-28T07:14:29.544633+0000","172.21.15.52:0/2776144659":"2024-05-28T07:14:29.544633+0000","172.21.15.52:6800/1631683163":"2024-05-28T07:14:29.544633+0000","172.21.15.52:0/3675529097":"2024-05-28T07:14:01.960550+0000","172.21.15.52:0/727808107":"2024-05-28T07:14:01.960550+0000","172.21.15.52:0/3839436833":"2024-05-28T07:14:29.544633+0000","172.21.15.52:6800/587662512":"2024-05-28T07:14:01.960550+0000","172.21.15.52:6801/587662512":"2024-05-28T07:14:01.960550+0000","172.21.15.52:0/853867465":"2024-05-28T07:14:01.960550+0000","172.21.15.52:0/3480918279":"2024-05-28T07:14:29.544633+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-05-27T07:22:45.580 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:22:45.590+0000 7fa56e925700 1 -- 172.21.15.52:0/2819804616 >> [v2:172.21.15.52:6800/3513075805,v1:172.21.15.52:6801/3513075805] conn(0x7fa5480b2900 msgr2=0x7fa5480b4e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:22:45.580 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:22:45.590+0000 7fa56e925700 1 --2- 172.21.15.52:0/2819804616 >> [v2:172.21.15.52:6800/3513075805,v1:172.21.15.52:6801/3513075805] conn(0x7fa5480b2900 0x7fa5480b4e10 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7fa558009480 tx=0x7fa55800b040 comp rx=0 tx=0).stop 2024-05-27T07:22:45.580 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:22:45.590+0000 7fa56e925700 1 -- 172.21.15.52:0/2819804616 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fa568109860 msgr2=0x7fa568076530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:22:45.580 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:22:45.590+0000 7fa56e925700 1 --2- 172.21.15.52:0/2819804616 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fa568109860 0x7fa568076530 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7fa55000e940 tx=0x7fa55000ec50 comp rx=0 tx=0).stop 2024-05-27T07:22:45.580 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:22:45.590+0000 7fa56e925700 1 -- 172.21.15.52:0/2819804616 shutdown_connections 2024-05-27T07:22:45.581 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:22:45.590+0000 7fa56e925700 1 --2- 172.21.15.52:0/2819804616 >> [v2:172.21.15.52:6800/3513075805,v1:172.21.15.52:6801/3513075805] conn(0x7fa5480b2900 0x7fa5480b4e10 unknown :-1 s=CLOSED pgs=158 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:22:45.581 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:22:45.590+0000 7fa56e925700 1 --2- 172.21.15.52:0/2819804616 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fa568109860 0x7fa568076530 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:22:45.581 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:22:45.590+0000 7fa56e925700 1 --2- 172.21.15.52:0/2819804616 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fa5681045e0 0x7fa568075ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:22:45.581 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:22:45.590+0000 7fa56e925700 1 --2- 172.21.15.52:0/2819804616 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fa568108f10 0x7fa568075ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:22:45.581 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:22:45.590+0000 7fa56e925700 1 -- 172.21.15.52:0/2819804616 >> 172.21.15.52:0/2819804616 conn(0x7fa5680fffd0 msgr2=0x7fa5681070c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:22:45.581 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:22:45.590+0000 7fa56e925700 1 -- 172.21.15.52:0/2819804616 shutdown_connections 2024-05-27T07:22:45.581 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:22:45.590+0000 7fa56e925700 1 -- 172.21.15.52:0/2819804616 wait complete. 2024-05-27T07:22:45.940 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:22:45 smithi105 ceph-mon[35977]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-27T07:22:45.941 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:22:45 smithi105 ceph-mon[35977]: from='client.? 172.21.15.52:0/2819804616' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-05-27T07:22:45.979 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:22:45 smithi052 ceph-mon[31955]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-27T07:22:45.979 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:22:45 smithi052 ceph-mon[31955]: from='client.? 172.21.15.52:0/2819804616' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-05-27T07:22:46.102 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:22:45 smithi173 ceph-mon[37000]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-27T07:22:46.102 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:22:45 smithi173 ceph-mon[37000]: from='client.? 172.21.15.52:0/2819804616' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-05-27T07:22:46.352 DEBUG:teuthology.orchestra.run.smithi052:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:38f9d900c83c63799fbdbe61acc9a11b0d3554a6 shell --fsid 73df2636-1bf8-11ef-bc9b-c7b262605968 -- ceph fs set cephfs standby_count_wanted 0 2024-05-27T07:22:46.739 INFO:teuthology.orchestra.run.smithi052.stderr:Inferring config /var/lib/ceph/73df2636-1bf8-11ef-bc9b-c7b262605968/mon.a/config 2024-05-27T07:22:47.102 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:22:46 smithi173 ceph-mon[37000]: pgmap v326: 129 pgs: 129 active+clean; 581 KiB data, 356 MiB used, 1.0 TiB / 1.0 TiB avail; 85 B/s wr, 0 op/s 2024-05-27T07:22:47.102 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:22:46 smithi173 ceph-mon[37000]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' 2024-05-27T07:22:47.102 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:22:46 smithi173 ceph-mon[37000]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' 2024-05-27T07:22:47.102 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:22:46 smithi173 ceph-mon[37000]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' 2024-05-27T07:22:47.102 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:22:46 smithi173 ceph-mon[37000]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' 2024-05-27T07:22:47.136 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:22:46 smithi052 ceph-mon[31955]: pgmap v326: 129 pgs: 129 active+clean; 581 KiB data, 356 MiB used, 1.0 TiB / 1.0 TiB avail; 85 B/s wr, 0 op/s 2024-05-27T07:22:47.136 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:22:46 smithi052 ceph-mon[31955]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' 2024-05-27T07:22:47.137 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:22:46 smithi052 ceph-mon[31955]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' 2024-05-27T07:22:47.137 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:22:46 smithi052 ceph-mon[31955]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' 2024-05-27T07:22:47.137 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:22:46 smithi052 ceph-mon[31955]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' 2024-05-27T07:22:47.190 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:22:46 smithi105 ceph-mon[35977]: pgmap v326: 129 pgs: 129 active+clean; 581 KiB data, 356 MiB used, 1.0 TiB / 1.0 TiB avail; 85 B/s wr, 0 op/s 2024-05-27T07:22:47.190 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:22:46 smithi105 ceph-mon[35977]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' 2024-05-27T07:22:47.190 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:22:46 smithi105 ceph-mon[35977]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' 2024-05-27T07:22:47.190 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:22:46 smithi105 ceph-mon[35977]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' 2024-05-27T07:22:47.190 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:22:46 smithi105 ceph-mon[35977]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' 2024-05-27T07:22:48.432 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:22:48.441+0000 7f5293721700 1 -- 172.21.15.52:0/1029472670 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f528c105e00 msgr2=0x7f528c10e350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:22:48.432 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:22:48.441+0000 7f5293721700 1 --2- 172.21.15.52:0/1029472670 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f528c105e00 0x7f528c10e350 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7f527c009990 tx=0x7f527c009ca0 comp rx=0 tx=0).stop 2024-05-27T07:22:48.432 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:22:48.441+0000 7f5293721700 1 -- 172.21.15.52:0/1029472670 shutdown_connections 2024-05-27T07:22:48.432 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:22:48.441+0000 7f5293721700 1 --2- 172.21.15.52:0/1029472670 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f528c10e8e0 0x7f528c110dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:22:48.432 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:22:48.441+0000 7f5293721700 1 --2- 172.21.15.52:0/1029472670 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f528c1054b0 0x7f528c1058c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:22:48.432 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:22:48.441+0000 7f5293721700 1 --2- 172.21.15.52:0/1029472670 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f528c105e00 0x7f528c10e350 unknown :-1 s=CLOSED pgs=217 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:22:48.432 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:22:48.441+0000 7f5293721700 1 -- 172.21.15.52:0/1029472670 >> 172.21.15.52:0/1029472670 conn(0x7f528c0fedb0 msgr2=0x7f528c1011d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:22:48.432 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:22:48.441+0000 7f5293721700 1 -- 172.21.15.52:0/1029472670 shutdown_connections 2024-05-27T07:22:48.433 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:22:48.442+0000 7f5293721700 1 -- 172.21.15.52:0/1029472670 wait complete. 2024-05-27T07:22:48.434 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:22:48.442+0000 7f5293721700 1 Processor -- start 2024-05-27T07:22:48.434 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:22:48.442+0000 7f5293721700 1 -- start start 2024-05-27T07:22:48.434 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:22:48.443+0000 7f5293721700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f528c1054b0 0x7f528c1031a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:22:48.434 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:22:48.443+0000 7f5293721700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f528c105e00 0x7f528c1036e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:22:48.434 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:22:48.443+0000 7f5293721700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f528c10e8e0 0x7f528c103c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:22:48.434 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:22:48.443+0000 7f5293721700 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x7f528c114440 con 0x7f528c105e00 2024-05-27T07:22:48.434 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:22:48.443+0000 7f5293721700 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f528c1142c0 con 0x7f528c1054b0 2024-05-27T07:22:48.434 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:22:48.443+0000 7f5293721700 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f528c1145c0 con 0x7f528c10e8e0 2024-05-27T07:22:48.435 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:22:48.443+0000 7f5290cbc700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f528c105e00 0x7f528c1036e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:22:48.435 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:22:48.443+0000 7f5290cbc700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f528c105e00 0x7f528c1036e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.52:60050/0 (socket says 172.21.15.52:60050) 2024-05-27T07:22:48.435 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:22:48.443+0000 7f5290cbc700 1 -- 172.21.15.52:0/4236339241 learned_addr learned my addr 172.21.15.52:0/4236339241 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-05-27T07:22:48.435 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:22:48.443+0000 7f52914bd700 1 --2- 172.21.15.52:0/4236339241 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f528c1054b0 0x7f528c1031a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:22:48.435 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:22:48.443+0000 7f5291cbe700 1 --2- 172.21.15.52:0/4236339241 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f528c10e8e0 0x7f528c103c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:22:48.435 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:22:48.443+0000 7f5290cbc700 1 -- 172.21.15.52:0/4236339241 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f528c10e8e0 msgr2=0x7f528c103c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:22:48.435 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:22:48.443+0000 7f5290cbc700 1 --2- 172.21.15.52:0/4236339241 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f528c10e8e0 0x7f528c103c20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:22:48.435 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:22:48.443+0000 7f5290cbc700 1 -- 172.21.15.52:0/4236339241 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f528c1054b0 msgr2=0x7f528c1031a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:22:48.435 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:22:48.443+0000 7f5290cbc700 1 --2- 172.21.15.52:0/4236339241 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f528c1054b0 0x7f528c1031a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:22:48.435 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:22:48.443+0000 7f5290cbc700 1 -- 172.21.15.52:0/4236339241 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f528c1a9130 con 0x7f528c105e00 2024-05-27T07:22:48.435 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:22:48.443+0000 7f5290cbc700 1 --2- 172.21.15.52:0/4236339241 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f528c105e00 0x7f528c1036e0 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7f527c00b5c0 tx=0x7f527c0058e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:22:48.435 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:22:48.444+0000 7f52827fc700 1 -- 172.21.15.52:0/4236339241 <== mon.0 v2:172.21.15.52:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f527c027070 con 0x7f528c105e00 2024-05-27T07:22:48.435 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:22:48.444+0000 7f5293721700 1 -- 172.21.15.52:0/4236339241 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f528c1a93c0 con 0x7f528c105e00 2024-05-27T07:22:48.436 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:22:48.444+0000 7f52827fc700 1 -- 172.21.15.52:0/4236339241 <== mon.0 v2:172.21.15.52:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f527c003fb0 con 0x7f528c105e00 2024-05-27T07:22:48.436 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:22:48.444+0000 7f52827fc700 1 -- 172.21.15.52:0/4236339241 <== mon.0 v2:172.21.15.52:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f527c022c80 con 0x7f528c105e00 2024-05-27T07:22:48.436 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:22:48.444+0000 7f5293721700 1 -- 172.21.15.52:0/4236339241 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f528c1a98f0 con 0x7f528c105e00 2024-05-27T07:22:48.436 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:22:48.445+0000 7f5293721700 1 -- 172.21.15.52:0/4236339241 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f528c0512b0 con 0x7f528c105e00 2024-05-27T07:22:48.436 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:22:48.447+0000 7f52827fc700 1 -- 172.21.15.52:0/4236339241 <== mon.0 v2:172.21.15.52:3300/0 4 ==== mgrmap(e 16) ==== 149652+0+0 (secure 0 0 0) 0x7f527c02a070 con 0x7f528c105e00 2024-05-27T07:22:48.437 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:22:48.447+0000 7f52827fc700 1 --2- 172.21.15.52:0/4236339241 >> [v2:172.21.15.52:6800/3513075805,v1:172.21.15.52:6801/3513075805] conn(0x7f52780b2880 0x7f52780b4d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:22:48.437 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:22:48.448+0000 7f52914bd700 1 --2- 172.21.15.52:0/4236339241 >> [v2:172.21.15.52:6800/3513075805,v1:172.21.15.52:6801/3513075805] conn(0x7f52780b2880 0x7f52780b4d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:22:48.438 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:22:48.448+0000 7f52827fc700 1 -- 172.21.15.52:0/4236339241 <== mon.0 v2:172.21.15.52:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 9092+0+0 (secure 0 0 0) 0x7f527c0e3c50 con 0x7f528c105e00 2024-05-27T07:22:48.438 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:22:48.448+0000 7f52914bd700 1 --2- 172.21.15.52:0/4236339241 >> [v2:172.21.15.52:6800/3513075805,v1:172.21.15.52:6801/3513075805] conn(0x7f52780b2880 0x7f52780b4d90 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7f5288009550 tx=0x7f52880091b0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:22:48.441 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:22:48.451+0000 7f52827fc700 1 -- 172.21.15.52:0/4236339241 <== mon.0 v2:172.21.15.52:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191963 (secure 0 0 0) 0x7f527c022e20 con 0x7f528c105e00 2024-05-27T07:22:48.738 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:22:48.747+0000 7f5293721700 1 -- 172.21.15.52:0/4236339241 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"} v 0) -- 0x7f528c02e5e0 con 0x7f528c105e00 2024-05-27T07:22:49.102 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:22:48 smithi173 ceph-mon[37000]: pgmap v327: 129 pgs: 129 active+clean; 581 KiB data, 356 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-27T07:22:49.102 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:22:48 smithi173 ceph-mon[37000]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' 2024-05-27T07:22:49.102 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:22:48 smithi173 ceph-mon[37000]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' 2024-05-27T07:22:49.102 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:22:48 smithi173 ceph-mon[37000]: from='client.? 172.21.15.52:0/4236339241' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]: dispatch 2024-05-27T07:22:49.191 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:22:48 smithi105 ceph-mon[35977]: pgmap v327: 129 pgs: 129 active+clean; 581 KiB data, 356 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-27T07:22:49.191 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:22:48 smithi105 ceph-mon[35977]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' 2024-05-27T07:22:49.191 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:22:48 smithi105 ceph-mon[35977]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' 2024-05-27T07:22:49.191 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:22:48 smithi105 ceph-mon[35977]: from='client.? 172.21.15.52:0/4236339241' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]: dispatch 2024-05-27T07:22:49.229 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:22:48 smithi052 ceph-mon[31955]: pgmap v327: 129 pgs: 129 active+clean; 581 KiB data, 356 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-27T07:22:49.229 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:22:48 smithi052 ceph-mon[31955]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' 2024-05-27T07:22:49.229 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:22:48 smithi052 ceph-mon[31955]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' 2024-05-27T07:22:49.229 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:22:48 smithi052 ceph-mon[31955]: from='client.? 172.21.15.52:0/4236339241' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]: dispatch 2024-05-27T07:22:49.726 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:22:49.736+0000 7f52827fc700 1 -- 172.21.15.52:0/4236339241 <== mon.0 v2:172.21.15.52:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]=0 v18) ==== 118+0+0 (secure 0 0 0) 0x7f527c013070 con 0x7f528c105e00 2024-05-27T07:22:49.729 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:22:49.738+0000 7f5293721700 1 -- 172.21.15.52:0/4236339241 >> [v2:172.21.15.52:6800/3513075805,v1:172.21.15.52:6801/3513075805] conn(0x7f52780b2880 msgr2=0x7f52780b4d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:22:49.729 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:22:49.738+0000 7f5293721700 1 --2- 172.21.15.52:0/4236339241 >> [v2:172.21.15.52:6800/3513075805,v1:172.21.15.52:6801/3513075805] conn(0x7f52780b2880 0x7f52780b4d90 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7f5288009550 tx=0x7f52880091b0 comp rx=0 tx=0).stop 2024-05-27T07:22:49.729 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:22:49.739+0000 7f5293721700 1 -- 172.21.15.52:0/4236339241 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f528c105e00 msgr2=0x7f528c1036e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:22:49.729 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:22:49.739+0000 7f5293721700 1 --2- 172.21.15.52:0/4236339241 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f528c105e00 0x7f528c1036e0 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7f527c00b5c0 tx=0x7f527c0058e0 comp rx=0 tx=0).stop 2024-05-27T07:22:49.729 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:22:49.739+0000 7f5293721700 1 -- 172.21.15.52:0/4236339241 shutdown_connections 2024-05-27T07:22:49.729 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:22:49.739+0000 7f5293721700 1 --2- 172.21.15.52:0/4236339241 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f528c10e8e0 0x7f528c103c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:22:49.729 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:22:49.739+0000 7f5293721700 1 --2- 172.21.15.52:0/4236339241 >> [v2:172.21.15.52:6800/3513075805,v1:172.21.15.52:6801/3513075805] conn(0x7f52780b2880 0x7f52780b4d90 unknown :-1 s=CLOSED pgs=159 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:22:49.729 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:22:49.739+0000 7f5293721700 1 --2- 172.21.15.52:0/4236339241 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f528c1054b0 0x7f528c1031a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:22:49.729 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:22:49.739+0000 7f5293721700 1 --2- 172.21.15.52:0/4236339241 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f528c105e00 0x7f528c1036e0 unknown :-1 s=CLOSED pgs=218 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:22:49.729 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:22:49.739+0000 7f5293721700 1 -- 172.21.15.52:0/4236339241 >> 172.21.15.52:0/4236339241 conn(0x7f528c0fedb0 msgr2=0x7f528c0ff440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:22:49.729 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:22:49.739+0000 7f5293721700 1 -- 172.21.15.52:0/4236339241 shutdown_connections 2024-05-27T07:22:49.730 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:22:49.739+0000 7f5293721700 1 -- 172.21.15.52:0/4236339241 wait complete. 2024-05-27T07:22:49.979 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:22:49 smithi052 ceph-mon[31955]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T07:22:49.980 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:22:49 smithi052 ceph-mon[31955]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-27T07:22:49.980 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:22:49 smithi052 ceph-mon[31955]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' 2024-05-27T07:22:49.980 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:22:49 smithi052 ceph-mon[31955]: from='client.? 172.21.15.52:0/4236339241' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]': finished 2024-05-27T07:22:49.980 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:22:49 smithi052 ceph-mon[31955]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-05-27T07:22:50.102 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:22:49 smithi173 ceph-mon[37000]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T07:22:50.102 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:22:49 smithi173 ceph-mon[37000]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-27T07:22:50.102 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:22:49 smithi173 ceph-mon[37000]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' 2024-05-27T07:22:50.103 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:22:49 smithi173 ceph-mon[37000]: from='client.? 172.21.15.52:0/4236339241' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]': finished 2024-05-27T07:22:50.103 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:22:49 smithi173 ceph-mon[37000]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-05-27T07:22:50.190 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:22:49 smithi105 ceph-mon[35977]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T07:22:50.190 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:22:49 smithi105 ceph-mon[35977]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-27T07:22:50.190 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:22:49 smithi105 ceph-mon[35977]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' 2024-05-27T07:22:50.190 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:22:49 smithi105 ceph-mon[35977]: from='client.? 172.21.15.52:0/4236339241' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]': finished 2024-05-27T07:22:50.190 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:22:49 smithi105 ceph-mon[35977]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-05-27T07:22:50.364 DEBUG:tasks.cephfs.filesystem:fs_config: {'ec_profile': ['m=2', 'k=2', 'crush-failure-domain=osd'], 'max_mds': 5, 'session_timeout': 300, 'standby_replay': True, 'subvols': {'create': 2}} 2024-05-27T07:22:50.365 DEBUG:teuthology.orchestra.run.smithi052:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:38f9d900c83c63799fbdbe61acc9a11b0d3554a6 shell --fsid 73df2636-1bf8-11ef-bc9b-c7b262605968 -- ceph fs set cephfs max_mds 5 2024-05-27T07:22:50.617 INFO:teuthology.orchestra.run.smithi052.stderr:Inferring config /var/lib/ceph/73df2636-1bf8-11ef-bc9b-c7b262605968/mon.a/config 2024-05-27T07:22:50.987 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:22:50 smithi052 ceph-mon[31955]: pgmap v328: 129 pgs: 129 active+clean; 581 KiB data, 356 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-27T07:22:51.102 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:22:50 smithi173 ceph-mon[37000]: pgmap v328: 129 pgs: 129 active+clean; 581 KiB data, 356 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-27T07:22:51.190 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:22:50 smithi105 ceph-mon[35977]: pgmap v328: 129 pgs: 129 active+clean; 581 KiB data, 356 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-27T07:22:52.425 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:22:52.434+0000 7f004f370700 1 -- 172.21.15.52:0/1832141429 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f00481054d0 msgr2=0x7f00481058e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:22:52.426 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:22:52.434+0000 7f004f370700 1 --2- 172.21.15.52:0/1832141429 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f00481054d0 0x7f00481058e0 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7f0044009990 tx=0x7f0044009ca0 comp rx=0 tx=0).stop 2024-05-27T07:22:52.426 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:22:52.434+0000 7f004f370700 1 -- 172.21.15.52:0/1832141429 shutdown_connections 2024-05-27T07:22:52.426 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:22:52.434+0000 7f004f370700 1 --2- 172.21.15.52:0/1832141429 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f004810e900 0x7f0048110dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:22:52.426 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:22:52.434+0000 7f004f370700 1 --2- 172.21.15.52:0/1832141429 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f00481054d0 0x7f00481058e0 unknown :-1 s=CLOSED pgs=219 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:22:52.426 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:22:52.434+0000 7f004f370700 1 --2- 172.21.15.52:0/1832141429 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f0048105e20 0x7f004810e370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:22:52.426 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:22:52.434+0000 7f004f370700 1 -- 172.21.15.52:0/1832141429 >> 172.21.15.52:0/1832141429 conn(0x7f00480fede0 msgr2=0x7f0048101200 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:22:52.427 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:22:52.434+0000 7f004f370700 1 -- 172.21.15.52:0/1832141429 shutdown_connections 2024-05-27T07:22:52.427 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:22:52.434+0000 7f004f370700 1 -- 172.21.15.52:0/1832141429 wait complete. 2024-05-27T07:22:52.429 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:22:52.435+0000 7f004f370700 1 Processor -- start 2024-05-27T07:22:52.429 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:22:52.435+0000 7f004f370700 1 -- start start 2024-05-27T07:22:52.430 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:22:52.435+0000 7f004f370700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f00481054d0 0x7f004819ae10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:22:52.430 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:22:52.435+0000 7f004f370700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f0048105e20 0x7f004819b3b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:22:52.430 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:22:52.435+0000 7f004f370700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f004810e900 0x7f004819f640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:22:52.430 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:22:52.435+0000 7f004f370700 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x7f0048114440 con 0x7f0048105e20 2024-05-27T07:22:52.430 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:22:52.436+0000 7f004f370700 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f00481142c0 con 0x7f004810e900 2024-05-27T07:22:52.430 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:22:52.436+0000 7f004f370700 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f00481145c0 con 0x7f00481054d0 2024-05-27T07:22:52.431 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:22:52.436+0000 7f004c90b700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f0048105e20 0x7f004819b3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:22:52.431 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:22:52.436+0000 7f004c90b700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f0048105e20 0x7f004819b3b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.52:37276/0 (socket says 172.21.15.52:37276) 2024-05-27T07:22:52.431 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:22:52.436+0000 7f004c90b700 1 -- 172.21.15.52:0/928429695 learned_addr learned my addr 172.21.15.52:0/928429695 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-05-27T07:22:52.431 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:22:52.436+0000 7f004d90d700 1 --2- 172.21.15.52:0/928429695 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f004810e900 0x7f004819f640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:22:52.431 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:22:52.436+0000 7f004d10c700 1 --2- 172.21.15.52:0/928429695 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f00481054d0 0x7f004819ae10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:22:52.432 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:22:52.436+0000 7f004c90b700 1 -- 172.21.15.52:0/928429695 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f00481054d0 msgr2=0x7f004819ae10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:22:52.432 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:22:52.436+0000 7f004c90b700 1 --2- 172.21.15.52:0/928429695 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f00481054d0 0x7f004819ae10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:22:52.432 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:22:52.436+0000 7f004c90b700 1 -- 172.21.15.52:0/928429695 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f004810e900 msgr2=0x7f004819f640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:22:52.432 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:22:52.436+0000 7f004c90b700 1 --2- 172.21.15.52:0/928429695 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f004810e900 0x7f004819f640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:22:52.432 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:22:52.436+0000 7f004c90b700 1 -- 172.21.15.52:0/928429695 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f004819fce0 con 0x7f0048105e20 2024-05-27T07:22:52.432 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:22:52.436+0000 7f004c90b700 1 --2- 172.21.15.52:0/928429695 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f0048105e20 0x7f004819b3b0 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7f003c00d950 tx=0x7f003c00dc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:22:52.432 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:22:52.436+0000 7f003a7fc700 1 -- 172.21.15.52:0/928429695 <== mon.0 v2:172.21.15.52:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f003c016070 con 0x7f0048105e20 2024-05-27T07:22:52.432 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:22:52.436+0000 7f003a7fc700 1 -- 172.21.15.52:0/928429695 <== mon.0 v2:172.21.15.52:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f003c00fd40 con 0x7f0048105e20 2024-05-27T07:22:52.432 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:22:52.437+0000 7f004f370700 1 -- 172.21.15.52:0/928429695 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f004819ffd0 con 0x7f0048105e20 2024-05-27T07:22:52.433 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:22:52.437+0000 7f003a7fc700 1 -- 172.21.15.52:0/928429695 <== mon.0 v2:172.21.15.52:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f003c00e3b0 con 0x7f0048105e20 2024-05-27T07:22:52.433 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:22:52.437+0000 7f004f370700 1 -- 172.21.15.52:0/928429695 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f00481a78f0 con 0x7f0048105e20 2024-05-27T07:22:52.433 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:22:52.438+0000 7f004f370700 1 -- 172.21.15.52:0/928429695 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f00480512b0 con 0x7f0048105e20 2024-05-27T07:22:52.434 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:22:52.440+0000 7f003a7fc700 1 -- 172.21.15.52:0/928429695 <== mon.0 v2:172.21.15.52:3300/0 4 ==== mgrmap(e 16) ==== 149652+0+0 (secure 0 0 0) 0x7f003c019070 con 0x7f0048105e20 2024-05-27T07:22:52.434 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:22:52.440+0000 7f003a7fc700 1 --2- 172.21.15.52:0/928429695 >> [v2:172.21.15.52:6800/3513075805,v1:172.21.15.52:6801/3513075805] conn(0x7f00340b2880 0x7f00340b4d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:22:52.434 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:22:52.440+0000 7f004d10c700 1 --2- 172.21.15.52:0/928429695 >> [v2:172.21.15.52:6800/3513075805,v1:172.21.15.52:6801/3513075805] conn(0x7f00340b2880 0x7f00340b4d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:22:52.434 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:22:52.440+0000 7f003a7fc700 1 -- 172.21.15.52:0/928429695 <== mon.0 v2:172.21.15.52:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 9092+0+0 (secure 0 0 0) 0x7f003c0e1fc0 con 0x7f0048105e20 2024-05-27T07:22:52.434 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:22:52.441+0000 7f004d10c700 1 --2- 172.21.15.52:0/928429695 >> [v2:172.21.15.52:6800/3513075805,v1:172.21.15.52:6801/3513075805] conn(0x7f00340b2880 0x7f00340b4d90 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7f0044006010 tx=0x7f0044005960 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:22:52.435 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:22:52.444+0000 7f003a7fc700 1 -- 172.21.15.52:0/928429695 <== mon.0 v2:172.21.15.52:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191963 (secure 0 0 0) 0x7f003c0a9980 con 0x7f0048105e20 2024-05-27T07:22:52.701 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:22:52.710+0000 7f004f370700 1 -- 172.21.15.52:0/928429695 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "5"} v 0) -- 0x7f004802e5e0 con 0x7f0048105e20 2024-05-27T07:22:52.830 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:22:52.839+0000 7f003a7fc700 1 -- 172.21.15.52:0/928429695 <== mon.0 v2:172.21.15.52:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "5"}]=0 v19) ==== 105+0+0 (secure 0 0 0) 0x7f003c0a9280 con 0x7f0048105e20 2024-05-27T07:22:52.834 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:22:52.842+0000 7f004f370700 1 -- 172.21.15.52:0/928429695 >> [v2:172.21.15.52:6800/3513075805,v1:172.21.15.52:6801/3513075805] conn(0x7f00340b2880 msgr2=0x7f00340b4d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:22:52.834 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:22:52.842+0000 7f004f370700 1 --2- 172.21.15.52:0/928429695 >> [v2:172.21.15.52:6800/3513075805,v1:172.21.15.52:6801/3513075805] conn(0x7f00340b2880 0x7f00340b4d90 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7f0044006010 tx=0x7f0044005960 comp rx=0 tx=0).stop 2024-05-27T07:22:52.834 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:22:52.842+0000 7f004f370700 1 -- 172.21.15.52:0/928429695 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f0048105e20 msgr2=0x7f004819b3b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:22:52.834 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:22:52.842+0000 7f004f370700 1 --2- 172.21.15.52:0/928429695 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f0048105e20 0x7f004819b3b0 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7f003c00d950 tx=0x7f003c00dc60 comp rx=0 tx=0).stop 2024-05-27T07:22:52.834 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:22:52.842+0000 7f004f370700 1 -- 172.21.15.52:0/928429695 shutdown_connections 2024-05-27T07:22:52.834 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:22:52.842+0000 7f004f370700 1 --2- 172.21.15.52:0/928429695 >> [v2:172.21.15.52:6800/3513075805,v1:172.21.15.52:6801/3513075805] conn(0x7f00340b2880 0x7f00340b4d90 unknown :-1 s=CLOSED pgs=160 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:22:52.834 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:22:52.842+0000 7f004f370700 1 --2- 172.21.15.52:0/928429695 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f004810e900 0x7f004819f640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:22:52.834 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:22:52.842+0000 7f004f370700 1 --2- 172.21.15.52:0/928429695 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f00481054d0 0x7f004819ae10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:22:52.834 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:22:52.842+0000 7f004f370700 1 --2- 172.21.15.52:0/928429695 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f0048105e20 0x7f004819b3b0 unknown :-1 s=CLOSED pgs=220 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:22:52.834 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:22:52.842+0000 7f004f370700 1 -- 172.21.15.52:0/928429695 >> 172.21.15.52:0/928429695 conn(0x7f00480fede0 msgr2=0x7f00480ff470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:22:52.835 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:22:52.842+0000 7f004f370700 1 -- 172.21.15.52:0/928429695 shutdown_connections 2024-05-27T07:22:52.835 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:22:52.843+0000 7f004f370700 1 -- 172.21.15.52:0/928429695 wait complete. 2024-05-27T07:22:52.979 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:22:52 smithi052 ceph-mon[31955]: pgmap v329: 129 pgs: 129 active+clean; 581 KiB data, 356 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-27T07:22:52.979 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:22:52 smithi052 ceph-mon[31955]: from='client.? 172.21.15.52:0/928429695' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "5"}]: dispatch 2024-05-27T07:22:53.102 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:22:52 smithi173 ceph-mon[37000]: pgmap v329: 129 pgs: 129 active+clean; 581 KiB data, 356 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-27T07:22:53.102 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:22:52 smithi173 ceph-mon[37000]: from='client.? 172.21.15.52:0/928429695' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "5"}]: dispatch 2024-05-27T07:22:53.190 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:22:52 smithi105 ceph-mon[35977]: pgmap v329: 129 pgs: 129 active+clean; 581 KiB data, 356 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-27T07:22:53.190 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:22:52 smithi105 ceph-mon[35977]: from='client.? 172.21.15.52:0/928429695' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "5"}]: dispatch 2024-05-27T07:22:53.526 DEBUG:teuthology.orchestra.run.smithi052:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:38f9d900c83c63799fbdbe61acc9a11b0d3554a6 shell --fsid 73df2636-1bf8-11ef-bc9b-c7b262605968 -- ceph fs set cephfs allow_standby_replay true 2024-05-27T07:22:53.791 INFO:teuthology.orchestra.run.smithi052.stderr:Inferring config /var/lib/ceph/73df2636-1bf8-11ef-bc9b-c7b262605968/mon.a/config 2024-05-27T07:22:54.102 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:22:53 smithi173 ceph-mon[37000]: Health check failed: 1 filesystem is online with fewer MDS than max_mds (MDS_UP_LESS_THAN_MAX) 2024-05-27T07:22:54.102 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:22:53 smithi173 ceph-mon[37000]: from='client.? 172.21.15.52:0/928429695' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "5"}]': finished 2024-05-27T07:22:54.102 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:22:53 smithi173 ceph-mon[37000]: daemon mds.h assigned to filesystem cephfs as rank 1 (now has 2 ranks) 2024-05-27T07:22:54.102 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:22:53 smithi173 ceph-mon[37000]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-05-27T07:22:54.102 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:22:53 smithi173 ceph-mon[37000]: fsmap cephfs:2 {0=b=up:active,1=h=up:creating} 10 up:standby 2024-05-27T07:22:54.102 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:22:53 smithi173 ceph-mon[37000]: daemon mds.h is now active in filesystem cephfs as rank 1 2024-05-27T07:22:54.173 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:22:53 smithi052 ceph-mon[31955]: Health check failed: 1 filesystem is online with fewer MDS than max_mds (MDS_UP_LESS_THAN_MAX) 2024-05-27T07:22:54.173 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:22:53 smithi052 ceph-mon[31955]: from='client.? 172.21.15.52:0/928429695' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "5"}]': finished 2024-05-27T07:22:54.173 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:22:53 smithi052 ceph-mon[31955]: daemon mds.h assigned to filesystem cephfs as rank 1 (now has 2 ranks) 2024-05-27T07:22:54.173 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:22:53 smithi052 ceph-mon[31955]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-05-27T07:22:54.173 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:22:53 smithi052 ceph-mon[31955]: fsmap cephfs:2 {0=b=up:active,1=h=up:creating} 10 up:standby 2024-05-27T07:22:54.173 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:22:53 smithi052 ceph-mon[31955]: daemon mds.h is now active in filesystem cephfs as rank 1 2024-05-27T07:22:54.190 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:22:53 smithi105 ceph-mon[35977]: Health check failed: 1 filesystem is online with fewer MDS than max_mds (MDS_UP_LESS_THAN_MAX) 2024-05-27T07:22:54.190 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:22:53 smithi105 ceph-mon[35977]: from='client.? 172.21.15.52:0/928429695' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "5"}]': finished 2024-05-27T07:22:54.190 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:22:53 smithi105 ceph-mon[35977]: daemon mds.h assigned to filesystem cephfs as rank 1 (now has 2 ranks) 2024-05-27T07:22:54.190 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:22:53 smithi105 ceph-mon[35977]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-05-27T07:22:54.190 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:22:53 smithi105 ceph-mon[35977]: fsmap cephfs:2 {0=b=up:active,1=h=up:creating} 10 up:standby 2024-05-27T07:22:54.190 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:22:53 smithi105 ceph-mon[35977]: daemon mds.h is now active in filesystem cephfs as rank 1 2024-05-27T07:22:54.919 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:22:54 smithi052 ceph-mon[31955]: pgmap v330: 129 pgs: 129 active+clean; 581 KiB data, 356 MiB used, 1.0 TiB / 1.0 TiB avail; 85 B/s wr, 0 op/s 2024-05-27T07:22:54.920 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:22:54 smithi052 ceph-mon[31955]: mds.? [v2:172.21.15.105:6836/191393149,v1:172.21.15.105:6837/191393149] up:active 2024-05-27T07:22:54.920 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:22:54 smithi052 ceph-mon[31955]: daemon mds.i assigned to filesystem cephfs as rank 2 (now has 3 ranks) 2024-05-27T07:22:54.920 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:22:54 smithi052 ceph-mon[31955]: fsmap cephfs:2 {0=b=up:active,1=h=up:active} 10 up:standby 2024-05-27T07:22:54.920 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:22:54 smithi052 ceph-mon[31955]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:creating} 9 up:standby 2024-05-27T07:22:54.920 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:22:54 smithi052 ceph-mon[31955]: daemon mds.i is now active in filesystem cephfs as rank 2 2024-05-27T07:22:55.102 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:22:54 smithi173 ceph-mon[37000]: pgmap v330: 129 pgs: 129 active+clean; 581 KiB data, 356 MiB used, 1.0 TiB / 1.0 TiB avail; 85 B/s wr, 0 op/s 2024-05-27T07:22:55.102 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:22:54 smithi173 ceph-mon[37000]: mds.? [v2:172.21.15.105:6836/191393149,v1:172.21.15.105:6837/191393149] up:active 2024-05-27T07:22:55.102 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:22:54 smithi173 ceph-mon[37000]: daemon mds.i assigned to filesystem cephfs as rank 2 (now has 3 ranks) 2024-05-27T07:22:55.102 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:22:54 smithi173 ceph-mon[37000]: fsmap cephfs:2 {0=b=up:active,1=h=up:active} 10 up:standby 2024-05-27T07:22:55.102 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:22:54 smithi173 ceph-mon[37000]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:creating} 9 up:standby 2024-05-27T07:22:55.102 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:22:54 smithi173 ceph-mon[37000]: daemon mds.i is now active in filesystem cephfs as rank 2 2024-05-27T07:22:55.190 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:22:54 smithi105 ceph-mon[35977]: pgmap v330: 129 pgs: 129 active+clean; 581 KiB data, 356 MiB used, 1.0 TiB / 1.0 TiB avail; 85 B/s wr, 0 op/s 2024-05-27T07:22:55.190 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:22:54 smithi105 ceph-mon[35977]: mds.? [v2:172.21.15.105:6836/191393149,v1:172.21.15.105:6837/191393149] up:active 2024-05-27T07:22:55.190 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:22:54 smithi105 ceph-mon[35977]: daemon mds.i assigned to filesystem cephfs as rank 2 (now has 3 ranks) 2024-05-27T07:22:55.190 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:22:54 smithi105 ceph-mon[35977]: fsmap cephfs:2 {0=b=up:active,1=h=up:active} 10 up:standby 2024-05-27T07:22:55.190 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:22:54 smithi105 ceph-mon[35977]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:creating} 9 up:standby 2024-05-27T07:22:55.190 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:22:54 smithi105 ceph-mon[35977]: daemon mds.i is now active in filesystem cephfs as rank 2 2024-05-27T07:22:55.477 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:22:55.486+0000 7fd8ff961700 1 -- 172.21.15.52:0/3671081281 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fd8f8108750 msgr2=0x7fd8f81100b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:22:55.478 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:22:55.486+0000 7fd8ff961700 1 --2- 172.21.15.52:0/3671081281 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fd8f8108750 0x7fd8f81100b0 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7fd8f000b0d0 tx=0x7fd8f000b3e0 comp rx=0 tx=0).stop 2024-05-27T07:22:55.478 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:22:55.486+0000 7fd8ff961700 1 -- 172.21.15.52:0/3671081281 shutdown_connections 2024-05-27T07:22:55.478 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:22:55.486+0000 7fd8ff961700 1 --2- 172.21.15.52:0/3671081281 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fd8f8108750 0x7fd8f81100b0 unknown :-1 s=CLOSED pgs=221 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:22:55.478 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:22:55.486+0000 7fd8ff961700 1 --2- 172.21.15.52:0/3671081281 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fd8f81034d0 0x7fd8f81038e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:22:55.478 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:22:55.486+0000 7fd8ff961700 1 --2- 172.21.15.52:0/3671081281 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fd8f8107e00 0x7fd8f8108210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:22:55.478 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:22:55.486+0000 7fd8ff961700 1 -- 172.21.15.52:0/3671081281 >> 172.21.15.52:0/3671081281 conn(0x7fd8f80fef20 msgr2=0x7fd8f8101340 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:22:55.478 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:22:55.486+0000 7fd8ff961700 1 -- 172.21.15.52:0/3671081281 shutdown_connections 2024-05-27T07:22:55.478 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:22:55.486+0000 7fd8ff961700 1 -- 172.21.15.52:0/3671081281 wait complete. 2024-05-27T07:22:55.480 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:22:55.487+0000 7fd8ff961700 1 Processor -- start 2024-05-27T07:22:55.480 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:22:55.487+0000 7fd8ff961700 1 -- start start 2024-05-27T07:22:55.480 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:22:55.488+0000 7fd8ff961700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fd8f81034d0 0x7fd8f819ade0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:22:55.480 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:22:55.488+0000 7fd8ff961700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fd8f8107e00 0x7fd8f819b380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:22:55.480 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:22:55.488+0000 7fd8ff961700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fd8f8108750 0x7fd8f819f610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:22:55.480 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:22:55.488+0000 7fd8ff961700 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x7fd8f8113380 con 0x7fd8f8107e00 2024-05-27T07:22:55.480 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:22:55.488+0000 7fd8ff961700 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7fd8f8113200 con 0x7fd8f81034d0 2024-05-27T07:22:55.480 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:22:55.488+0000 7fd8ff961700 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7fd8f8113500 con 0x7fd8f8108750 2024-05-27T07:22:55.480 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:22:55.488+0000 7fd8fcefc700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fd8f8107e00 0x7fd8f819b380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:22:55.480 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:22:55.488+0000 7fd8fcefc700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fd8f8107e00 0x7fd8f819b380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.52:37290/0 (socket says 172.21.15.52:37290) 2024-05-27T07:22:55.480 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:22:55.488+0000 7fd8fcefc700 1 -- 172.21.15.52:0/3961942100 learned_addr learned my addr 172.21.15.52:0/3961942100 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-05-27T07:22:55.481 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:22:55.488+0000 7fd8fd6fd700 1 --2- 172.21.15.52:0/3961942100 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fd8f81034d0 0x7fd8f819ade0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:22:55.481 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:22:55.488+0000 7fd8fdefe700 1 --2- 172.21.15.52:0/3961942100 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fd8f8108750 0x7fd8f819f610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:22:55.481 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:22:55.488+0000 7fd8fcefc700 1 -- 172.21.15.52:0/3961942100 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fd8f8108750 msgr2=0x7fd8f819f610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:22:55.481 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:22:55.488+0000 7fd8fcefc700 1 --2- 172.21.15.52:0/3961942100 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fd8f8108750 0x7fd8f819f610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:22:55.481 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:22:55.488+0000 7fd8fcefc700 1 -- 172.21.15.52:0/3961942100 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fd8f81034d0 msgr2=0x7fd8f819ade0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:22:55.481 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:22:55.488+0000 7fd8fcefc700 1 --2- 172.21.15.52:0/3961942100 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fd8f81034d0 0x7fd8f819ade0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:22:55.481 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:22:55.488+0000 7fd8fcefc700 1 -- 172.21.15.52:0/3961942100 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd8f819fcb0 con 0x7fd8f8107e00 2024-05-27T07:22:55.481 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:22:55.488+0000 7fd8fdefe700 1 --2- 172.21.15.52:0/3961942100 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fd8f8108750 0x7fd8f819f610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-27T07:22:55.481 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:22:55.488+0000 7fd8fcefc700 1 --2- 172.21.15.52:0/3961942100 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fd8f8107e00 0x7fd8f819b380 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7fd8e800d980 tx=0x7fd8e800dc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:22:55.481 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:22:55.488+0000 7fd8ee7fc700 1 -- 172.21.15.52:0/3961942100 <== mon.0 v2:172.21.15.52:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd8e8017070 con 0x7fd8f8107e00 2024-05-27T07:22:55.482 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:22:55.489+0000 7fd8ee7fc700 1 -- 172.21.15.52:0/3961942100 <== mon.0 v2:172.21.15.52:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fd8e8009de0 con 0x7fd8f8107e00 2024-05-27T07:22:55.482 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:22:55.489+0000 7fd8ff961700 1 -- 172.21.15.52:0/3961942100 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd8f819ffa0 con 0x7fd8f8107e00 2024-05-27T07:22:55.482 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:22:55.489+0000 7fd8ee7fc700 1 -- 172.21.15.52:0/3961942100 <== mon.0 v2:172.21.15.52:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd8e800f3c0 con 0x7fd8f8107e00 2024-05-27T07:22:55.482 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:22:55.489+0000 7fd8ff961700 1 -- 172.21.15.52:0/3961942100 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd8f81a78c0 con 0x7fd8f8107e00 2024-05-27T07:22:55.482 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:22:55.490+0000 7fd8ff961700 1 -- 172.21.15.52:0/3961942100 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd8f80512b0 con 0x7fd8f8107e00 2024-05-27T07:22:55.482 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:22:55.491+0000 7fd8ee7fc700 1 -- 172.21.15.52:0/3961942100 <== mon.0 v2:172.21.15.52:3300/0 4 ==== mgrmap(e 16) ==== 149652+0+0 (secure 0 0 0) 0x7fd8e8009930 con 0x7fd8f8107e00 2024-05-27T07:22:55.483 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:22:55.492+0000 7fd8ee7fc700 1 --2- 172.21.15.52:0/3961942100 >> [v2:172.21.15.52:6800/3513075805,v1:172.21.15.52:6801/3513075805] conn(0x7fd8e40b2880 0x7fd8e40b4d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:22:55.483 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:22:55.492+0000 7fd8fd6fd700 1 --2- 172.21.15.52:0/3961942100 >> [v2:172.21.15.52:6800/3513075805,v1:172.21.15.52:6801/3513075805] conn(0x7fd8e40b2880 0x7fd8e40b4d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:22:55.483 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:22:55.492+0000 7fd8ee7fc700 1 -- 172.21.15.52:0/3961942100 <== mon.0 v2:172.21.15.52:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 9092+0+0 (secure 0 0 0) 0x7fd8e80e1fb0 con 0x7fd8f8107e00 2024-05-27T07:22:55.484 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:22:55.493+0000 7fd8fd6fd700 1 --2- 172.21.15.52:0/3961942100 >> [v2:172.21.15.52:6800/3513075805,v1:172.21.15.52:6801/3513075805] conn(0x7fd8e40b2880 0x7fd8e40b4d90 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7fd8f4005950 tx=0x7fd8f400a300 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:22:55.486 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:22:55.496+0000 7fd8ee7fc700 1 -- 172.21.15.52:0/3961942100 <== mon.0 v2:172.21.15.52:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191963 (secure 0 0 0) 0x7fd8e80a9970 con 0x7fd8f8107e00 2024-05-27T07:22:55.754 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:22:55.763+0000 7fd8ff961700 1 -- 172.21.15.52:0/3961942100 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"} v 0) -- 0x7fd8f802e5e0 con 0x7fd8f8107e00 2024-05-27T07:22:55.863 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:22:55.872+0000 7fd8ee7fc700 1 -- 172.21.15.52:0/3961942100 <== mon.0 v2:172.21.15.52:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]=0 v25) ==== 121+0+0 (secure 0 0 0) 0x7fd8e80a9270 con 0x7fd8f8107e00 2024-05-27T07:22:55.866 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:22:55.875+0000 7fd8ff961700 1 -- 172.21.15.52:0/3961942100 >> [v2:172.21.15.52:6800/3513075805,v1:172.21.15.52:6801/3513075805] conn(0x7fd8e40b2880 msgr2=0x7fd8e40b4d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:22:55.866 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:22:55.875+0000 7fd8ff961700 1 --2- 172.21.15.52:0/3961942100 >> [v2:172.21.15.52:6800/3513075805,v1:172.21.15.52:6801/3513075805] conn(0x7fd8e40b2880 0x7fd8e40b4d90 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7fd8f4005950 tx=0x7fd8f400a300 comp rx=0 tx=0).stop 2024-05-27T07:22:55.866 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:22:55.875+0000 7fd8ff961700 1 -- 172.21.15.52:0/3961942100 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fd8f8107e00 msgr2=0x7fd8f819b380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:22:55.866 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:22:55.875+0000 7fd8ff961700 1 --2- 172.21.15.52:0/3961942100 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fd8f8107e00 0x7fd8f819b380 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7fd8e800d980 tx=0x7fd8e800dc90 comp rx=0 tx=0).stop 2024-05-27T07:22:55.866 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:22:55.875+0000 7fd8ff961700 1 -- 172.21.15.52:0/3961942100 shutdown_connections 2024-05-27T07:22:55.866 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:22:55.875+0000 7fd8ff961700 1 --2- 172.21.15.52:0/3961942100 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fd8f8108750 0x7fd8f819f610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:22:55.866 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:22:55.875+0000 7fd8ff961700 1 --2- 172.21.15.52:0/3961942100 >> [v2:172.21.15.52:6800/3513075805,v1:172.21.15.52:6801/3513075805] conn(0x7fd8e40b2880 0x7fd8e40b4d90 unknown :-1 s=CLOSED pgs=161 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:22:55.866 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:22:55.875+0000 7fd8ff961700 1 --2- 172.21.15.52:0/3961942100 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fd8f81034d0 0x7fd8f819ade0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:22:55.866 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:22:55.875+0000 7fd8ff961700 1 --2- 172.21.15.52:0/3961942100 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fd8f8107e00 0x7fd8f819b380 unknown :-1 s=CLOSED pgs=222 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:22:55.867 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:22:55.875+0000 7fd8ff961700 1 -- 172.21.15.52:0/3961942100 >> 172.21.15.52:0/3961942100 conn(0x7fd8f80fef20 msgr2=0x7fd8f806c3d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:22:55.867 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:22:55.875+0000 7fd8ff961700 1 -- 172.21.15.52:0/3961942100 shutdown_connections 2024-05-27T07:22:55.867 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:22:55.875+0000 7fd8ff961700 1 -- 172.21.15.52:0/3961942100 wait complete. 2024-05-27T07:22:56.190 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:22:55 smithi105 ceph-mon[35977]: mds.? [v2:172.21.15.173:6838/564778293,v1:172.21.15.173:6839/564778293] up:active 2024-05-27T07:22:56.190 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:22:55 smithi105 ceph-mon[35977]: daemon mds.e assigned to filesystem cephfs as rank 3 (now has 4 ranks) 2024-05-27T07:22:56.190 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:22:55 smithi105 ceph-mon[35977]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 9 up:standby 2024-05-27T07:22:56.190 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:22:55 smithi105 ceph-mon[35977]: fsmap cephfs:4 {0=b=up:active,1=h=up:active,2=i=up:active,3=e=up:creating} 8 up:standby 2024-05-27T07:22:56.190 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:22:55 smithi105 ceph-mon[35977]: daemon mds.e is now active in filesystem cephfs as rank 3 2024-05-27T07:22:56.190 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:22:55 smithi105 ceph-mon[35977]: from='client.? 172.21.15.52:0/3961942100' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]: dispatch 2024-05-27T07:22:56.226 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:22:55 smithi052 ceph-mon[31955]: mds.? [v2:172.21.15.173:6838/564778293,v1:172.21.15.173:6839/564778293] up:active 2024-05-27T07:22:56.227 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:22:55 smithi052 ceph-mon[31955]: daemon mds.e assigned to filesystem cephfs as rank 3 (now has 4 ranks) 2024-05-27T07:22:56.227 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:22:55 smithi052 ceph-mon[31955]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 9 up:standby 2024-05-27T07:22:56.227 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:22:55 smithi052 ceph-mon[31955]: fsmap cephfs:4 {0=b=up:active,1=h=up:active,2=i=up:active,3=e=up:creating} 8 up:standby 2024-05-27T07:22:56.227 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:22:55 smithi052 ceph-mon[31955]: daemon mds.e is now active in filesystem cephfs as rank 3 2024-05-27T07:22:56.227 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:22:55 smithi052 ceph-mon[31955]: from='client.? 172.21.15.52:0/3961942100' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]: dispatch 2024-05-27T07:22:56.352 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:22:55 smithi173 ceph-mon[37000]: mds.? [v2:172.21.15.173:6838/564778293,v1:172.21.15.173:6839/564778293] up:active 2024-05-27T07:22:56.352 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:22:55 smithi173 ceph-mon[37000]: daemon mds.e assigned to filesystem cephfs as rank 3 (now has 4 ranks) 2024-05-27T07:22:56.352 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:22:55 smithi173 ceph-mon[37000]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 9 up:standby 2024-05-27T07:22:56.352 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:22:55 smithi173 ceph-mon[37000]: fsmap cephfs:4 {0=b=up:active,1=h=up:active,2=i=up:active,3=e=up:creating} 8 up:standby 2024-05-27T07:22:56.352 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:22:55 smithi173 ceph-mon[37000]: daemon mds.e is now active in filesystem cephfs as rank 3 2024-05-27T07:22:56.352 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:22:55 smithi173 ceph-mon[37000]: from='client.? 172.21.15.52:0/3961942100' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]: dispatch 2024-05-27T07:22:56.526 DEBUG:teuthology.orchestra.run.smithi052:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:38f9d900c83c63799fbdbe61acc9a11b0d3554a6 shell --fsid 73df2636-1bf8-11ef-bc9b-c7b262605968 -- ceph fs set cephfs session_timeout 300 2024-05-27T07:22:56.782 INFO:teuthology.orchestra.run.smithi052.stderr:Inferring config /var/lib/ceph/73df2636-1bf8-11ef-bc9b-c7b262605968/mon.a/config 2024-05-27T07:22:57.141 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:22:56 smithi052 ceph-mon[31955]: pgmap v331: 129 pgs: 129 active+clean; 584 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail; 1.8 KiB/s wr, 5 op/s 2024-05-27T07:22:57.141 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:22:56 smithi052 ceph-mon[31955]: from='client.? 172.21.15.52:0/3961942100' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]': finished 2024-05-27T07:22:57.141 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:22:56 smithi052 ceph-mon[31955]: mds.? [v2:172.21.15.105:6834/3298498384,v1:172.21.15.105:6835/3298498384] up:active 2024-05-27T07:22:57.141 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:22:56 smithi052 ceph-mon[31955]: daemon mds.c assigned to filesystem cephfs as rank 4 (now has 5 ranks) 2024-05-27T07:22:57.141 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:22:56 smithi052 ceph-mon[31955]: Health check cleared: MDS_UP_LESS_THAN_MAX (was: 1 filesystem is online with fewer MDS than max_mds) 2024-05-27T07:22:57.141 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:22:56 smithi052 ceph-mon[31955]: Cluster is now healthy 2024-05-27T07:22:57.141 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:22:56 smithi052 ceph-mon[31955]: fsmap cephfs:4 {0=b=up:active,1=h=up:active,2=i=up:active,3=e=up:active} 8 up:standby 2024-05-27T07:22:57.141 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:22:56 smithi052 ceph-mon[31955]: fsmap cephfs:5 1 up:creating 7 up:standby 4 up:active 2024-05-27T07:22:57.141 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:22:56 smithi052 ceph-mon[31955]: daemon mds.c is now active in filesystem cephfs as rank 4 2024-05-27T07:22:57.190 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:22:56 smithi105 ceph-mon[35977]: pgmap v331: 129 pgs: 129 active+clean; 584 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail; 1.8 KiB/s wr, 5 op/s 2024-05-27T07:22:57.190 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:22:56 smithi105 ceph-mon[35977]: from='client.? 172.21.15.52:0/3961942100' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]': finished 2024-05-27T07:22:57.190 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:22:56 smithi105 ceph-mon[35977]: mds.? [v2:172.21.15.105:6834/3298498384,v1:172.21.15.105:6835/3298498384] up:active 2024-05-27T07:22:57.190 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:22:56 smithi105 ceph-mon[35977]: daemon mds.c assigned to filesystem cephfs as rank 4 (now has 5 ranks) 2024-05-27T07:22:57.190 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:22:56 smithi105 ceph-mon[35977]: Health check cleared: MDS_UP_LESS_THAN_MAX (was: 1 filesystem is online with fewer MDS than max_mds) 2024-05-27T07:22:57.190 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:22:56 smithi105 ceph-mon[35977]: Cluster is now healthy 2024-05-27T07:22:57.191 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:22:56 smithi105 ceph-mon[35977]: fsmap cephfs:4 {0=b=up:active,1=h=up:active,2=i=up:active,3=e=up:active} 8 up:standby 2024-05-27T07:22:57.191 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:22:56 smithi105 ceph-mon[35977]: fsmap cephfs:5 1 up:creating 7 up:standby 4 up:active 2024-05-27T07:22:57.191 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:22:56 smithi105 ceph-mon[35977]: daemon mds.c is now active in filesystem cephfs as rank 4 2024-05-27T07:22:57.352 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:22:56 smithi173 ceph-mon[37000]: pgmap v331: 129 pgs: 129 active+clean; 584 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail; 1.8 KiB/s wr, 5 op/s 2024-05-27T07:22:57.352 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:22:56 smithi173 ceph-mon[37000]: from='client.? 172.21.15.52:0/3961942100' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]': finished 2024-05-27T07:22:57.352 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:22:56 smithi173 ceph-mon[37000]: mds.? [v2:172.21.15.105:6834/3298498384,v1:172.21.15.105:6835/3298498384] up:active 2024-05-27T07:22:57.352 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:22:56 smithi173 ceph-mon[37000]: daemon mds.c assigned to filesystem cephfs as rank 4 (now has 5 ranks) 2024-05-27T07:22:57.352 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:22:56 smithi173 ceph-mon[37000]: Health check cleared: MDS_UP_LESS_THAN_MAX (was: 1 filesystem is online with fewer MDS than max_mds) 2024-05-27T07:22:57.352 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:22:56 smithi173 ceph-mon[37000]: Cluster is now healthy 2024-05-27T07:22:57.352 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:22:56 smithi173 ceph-mon[37000]: fsmap cephfs:4 {0=b=up:active,1=h=up:active,2=i=up:active,3=e=up:active} 8 up:standby 2024-05-27T07:22:57.352 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:22:56 smithi173 ceph-mon[37000]: fsmap cephfs:5 1 up:creating 7 up:standby 4 up:active 2024-05-27T07:22:57.353 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:22:56 smithi173 ceph-mon[37000]: daemon mds.c is now active in filesystem cephfs as rank 4 2024-05-27T07:22:58.124 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:22:57 smithi052 ceph-mon[31955]: mds.? [v2:172.21.15.173:6836/1396363827,v1:172.21.15.173:6837/1396363827] up:active 2024-05-27T07:22:58.124 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:22:57 smithi052 ceph-mon[31955]: fsmap cephfs:5 7 up:standby 5 up:active 2024-05-27T07:22:58.124 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:22:57 smithi052 ceph-mon[31955]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-05-27T07:22:58.190 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:22:57 smithi105 ceph-mon[35977]: mds.? [v2:172.21.15.173:6836/1396363827,v1:172.21.15.173:6837/1396363827] up:active 2024-05-27T07:22:58.190 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:22:57 smithi105 ceph-mon[35977]: fsmap cephfs:5 7 up:standby 5 up:active 2024-05-27T07:22:58.190 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:22:57 smithi105 ceph-mon[35977]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-05-27T07:22:58.352 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:22:57 smithi173 ceph-mon[37000]: mds.? [v2:172.21.15.173:6836/1396363827,v1:172.21.15.173:6837/1396363827] up:active 2024-05-27T07:22:58.352 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:22:57 smithi173 ceph-mon[37000]: fsmap cephfs:5 7 up:standby 5 up:active 2024-05-27T07:22:58.352 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:22:57 smithi173 ceph-mon[37000]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-05-27T07:22:58.530 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:22:58.538+0000 7f9d90789700 1 -- 172.21.15.52:0/2298268032 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f9d881034b0 msgr2=0x7f9d881038c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:22:58.530 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:22:58.538+0000 7f9d90789700 1 --2- 172.21.15.52:0/2298268032 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f9d881034b0 0x7f9d881038c0 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7f9d8400b0d0 tx=0x7f9d8400b3e0 comp rx=0 tx=0).stop 2024-05-27T07:22:58.530 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:22:58.539+0000 7f9d90789700 1 -- 172.21.15.52:0/2298268032 shutdown_connections 2024-05-27T07:22:58.530 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:22:58.539+0000 7f9d90789700 1 --2- 172.21.15.52:0/2298268032 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f9d88108730 0x7f9d88110090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:22:58.530 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:22:58.539+0000 7f9d90789700 1 --2- 172.21.15.52:0/2298268032 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f9d881034b0 0x7f9d881038c0 unknown :-1 s=CLOSED pgs=223 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:22:58.530 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:22:58.539+0000 7f9d90789700 1 --2- 172.21.15.52:0/2298268032 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f9d88107de0 0x7f9d881081f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:22:58.530 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:22:58.539+0000 7f9d90789700 1 -- 172.21.15.52:0/2298268032 >> 172.21.15.52:0/2298268032 conn(0x7f9d880fedb0 msgr2=0x7f9d881011d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:22:58.530 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:22:58.539+0000 7f9d90789700 1 -- 172.21.15.52:0/2298268032 shutdown_connections 2024-05-27T07:22:58.530 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:22:58.539+0000 7f9d90789700 1 -- 172.21.15.52:0/2298268032 wait complete. 2024-05-27T07:22:58.533 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:22:58.539+0000 7f9d90789700 1 Processor -- start 2024-05-27T07:22:58.533 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:22:58.539+0000 7f9d90789700 1 -- start start 2024-05-27T07:22:58.533 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:22:58.540+0000 7f9d90789700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f9d881034b0 0x7f9d8819bcb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:22:58.533 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:22:58.540+0000 7f9d90789700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f9d88107de0 0x7f9d8819c1f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:22:58.533 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:22:58.540+0000 7f9d90789700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f9d88108730 0x7f9d8819c770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:22:58.533 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:22:58.540+0000 7f9d90789700 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x7f9d881133d0 con 0x7f9d881034b0 2024-05-27T07:22:58.533 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:22:58.540+0000 7f9d90789700 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f9d88113250 con 0x7f9d88108730 2024-05-27T07:22:58.533 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:22:58.540+0000 7f9d90789700 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f9d88113550 con 0x7f9d88107de0 2024-05-27T07:22:58.533 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:22:58.540+0000 7f9d8e525700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f9d881034b0 0x7f9d8819bcb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:22:58.533 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:22:58.540+0000 7f9d8e525700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f9d881034b0 0x7f9d8819bcb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.52:37306/0 (socket says 172.21.15.52:37306) 2024-05-27T07:22:58.533 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:22:58.540+0000 7f9d8e525700 1 -- 172.21.15.52:0/1118125341 learned_addr learned my addr 172.21.15.52:0/1118125341 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-05-27T07:22:58.534 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:22:58.540+0000 7f9d8dd24700 1 --2- 172.21.15.52:0/1118125341 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f9d88107de0 0x7f9d8819c1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:22:58.534 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:22:58.540+0000 7f9d8ed26700 1 --2- 172.21.15.52:0/1118125341 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f9d88108730 0x7f9d8819c770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:22:58.534 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:22:58.540+0000 7f9d8e525700 1 -- 172.21.15.52:0/1118125341 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f9d88107de0 msgr2=0x7f9d8819c1f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:22:58.534 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:22:58.540+0000 7f9d8e525700 1 --2- 172.21.15.52:0/1118125341 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f9d88107de0 0x7f9d8819c1f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:22:58.534 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:22:58.540+0000 7f9d8e525700 1 -- 172.21.15.52:0/1118125341 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f9d88108730 msgr2=0x7f9d8819c770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:22:58.534 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:22:58.540+0000 7f9d8e525700 1 --2- 172.21.15.52:0/1118125341 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f9d88108730 0x7f9d8819c770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:22:58.534 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:22:58.540+0000 7f9d8e525700 1 -- 172.21.15.52:0/1118125341 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9d881a7020 con 0x7f9d881034b0 2024-05-27T07:22:58.534 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:22:58.540+0000 7f9d8ed26700 1 --2- 172.21.15.52:0/1118125341 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f9d88108730 0x7f9d8819c770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-27T07:22:58.534 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:22:58.541+0000 7f9d8e525700 1 --2- 172.21.15.52:0/1118125341 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f9d881034b0 0x7f9d8819bcb0 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7f9d84007a20 tx=0x7f9d84007a50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:22:58.534 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:22:58.541+0000 7f9d7f7fe700 1 -- 172.21.15.52:0/1118125341 <== mon.0 v2:172.21.15.52:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9d8401f070 con 0x7f9d881034b0 2024-05-27T07:22:58.534 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:22:58.541+0000 7f9d7f7fe700 1 -- 172.21.15.52:0/1118125341 <== mon.0 v2:172.21.15.52:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f9d84003c00 con 0x7f9d881034b0 2024-05-27T07:22:58.534 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:22:58.541+0000 7f9d7f7fe700 1 -- 172.21.15.52:0/1118125341 <== mon.0 v2:172.21.15.52:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9d8401b730 con 0x7f9d881034b0 2024-05-27T07:22:58.535 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:22:58.541+0000 7f9d90789700 1 -- 172.21.15.52:0/1118125341 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9d881a72b0 con 0x7f9d881034b0 2024-05-27T07:22:58.535 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:22:58.541+0000 7f9d90789700 1 -- 172.21.15.52:0/1118125341 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9d881a7710 con 0x7f9d881034b0 2024-05-27T07:22:58.535 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:22:58.542+0000 7f9d90789700 1 -- 172.21.15.52:0/1118125341 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9d880512b0 con 0x7f9d881034b0 2024-05-27T07:22:58.535 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:22:58.543+0000 7f9d7f7fe700 1 -- 172.21.15.52:0/1118125341 <== mon.0 v2:172.21.15.52:3300/0 4 ==== mgrmap(e 16) ==== 149652+0+0 (secure 0 0 0) 0x7f9d84022070 con 0x7f9d881034b0 2024-05-27T07:22:58.535 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:22:58.544+0000 7f9d7f7fe700 1 --2- 172.21.15.52:0/1118125341 >> [v2:172.21.15.52:6800/3513075805,v1:172.21.15.52:6801/3513075805] conn(0x7f9d740b2900 0x7f9d740b4e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:22:58.536 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:22:58.544+0000 7f9d8dd24700 1 --2- 172.21.15.52:0/1118125341 >> [v2:172.21.15.52:6800/3513075805,v1:172.21.15.52:6801/3513075805] conn(0x7f9d740b2900 0x7f9d740b4e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:22:58.536 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:22:58.544+0000 7f9d7f7fe700 1 -- 172.21.15.52:0/1118125341 <== mon.0 v2:172.21.15.52:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 9092+0+0 (secure 0 0 0) 0x7f9d8400f020 con 0x7f9d881034b0 2024-05-27T07:22:58.536 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:22:58.545+0000 7f9d8dd24700 1 --2- 172.21.15.52:0/1118125341 >> [v2:172.21.15.52:6800/3513075805,v1:172.21.15.52:6801/3513075805] conn(0x7f9d740b2900 0x7f9d740b4e10 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7f9d881069f0 tx=0x7f9d78009190 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:22:58.538 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:22:58.548+0000 7f9d7f7fe700 1 -- 172.21.15.52:0/1118125341 <== mon.0 v2:172.21.15.52:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191963 (secure 0 0 0) 0x7f9d84029020 con 0x7f9d881034b0 2024-05-27T07:22:58.811 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:22:58.820+0000 7f9d90789700 1 -- 172.21.15.52:0/1118125341 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"} v 0) -- 0x7f9d8802e5e0 con 0x7f9d881034b0 2024-05-27T07:22:58.877 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:22:58.886+0000 7f9d7f7fe700 1 -- 172.21.15.52:0/1118125341 <== mon.0 v2:172.21.15.52:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]=0 v29) ==== 115+0+0 (secure 0 0 0) 0x7f9d840ab610 con 0x7f9d881034b0 2024-05-27T07:22:58.880 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:22:58.889+0000 7f9d90789700 1 -- 172.21.15.52:0/1118125341 >> [v2:172.21.15.52:6800/3513075805,v1:172.21.15.52:6801/3513075805] conn(0x7f9d740b2900 msgr2=0x7f9d740b4e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:22:58.880 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:22:58.889+0000 7f9d90789700 1 --2- 172.21.15.52:0/1118125341 >> [v2:172.21.15.52:6800/3513075805,v1:172.21.15.52:6801/3513075805] conn(0x7f9d740b2900 0x7f9d740b4e10 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7f9d881069f0 tx=0x7f9d78009190 comp rx=0 tx=0).stop 2024-05-27T07:22:58.880 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:22:58.889+0000 7f9d90789700 1 -- 172.21.15.52:0/1118125341 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f9d881034b0 msgr2=0x7f9d8819bcb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:22:58.880 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:22:58.889+0000 7f9d90789700 1 --2- 172.21.15.52:0/1118125341 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f9d881034b0 0x7f9d8819bcb0 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7f9d84007a20 tx=0x7f9d84007a50 comp rx=0 tx=0).stop 2024-05-27T07:22:58.880 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:22:58.889+0000 7f9d90789700 1 -- 172.21.15.52:0/1118125341 shutdown_connections 2024-05-27T07:22:58.880 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:22:58.889+0000 7f9d90789700 1 --2- 172.21.15.52:0/1118125341 >> [v2:172.21.15.52:6800/3513075805,v1:172.21.15.52:6801/3513075805] conn(0x7f9d740b2900 0x7f9d740b4e10 unknown :-1 s=CLOSED pgs=162 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:22:58.880 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:22:58.889+0000 7f9d90789700 1 --2- 172.21.15.52:0/1118125341 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f9d88108730 0x7f9d8819c770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:22:58.880 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:22:58.889+0000 7f9d90789700 1 --2- 172.21.15.52:0/1118125341 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f9d881034b0 0x7f9d8819bcb0 unknown :-1 s=CLOSED pgs=224 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:22:58.881 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:22:58.889+0000 7f9d90789700 1 --2- 172.21.15.52:0/1118125341 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f9d88107de0 0x7f9d8819c1f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:22:58.881 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:22:58.889+0000 7f9d90789700 1 -- 172.21.15.52:0/1118125341 >> 172.21.15.52:0/1118125341 conn(0x7f9d880fedb0 msgr2=0x7f9d881011d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:22:58.881 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:22:58.889+0000 7f9d90789700 1 -- 172.21.15.52:0/1118125341 shutdown_connections 2024-05-27T07:22:58.881 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:22:58.889+0000 7f9d90789700 1 -- 172.21.15.52:0/1118125341 wait complete. 2024-05-27T07:22:59.190 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:22:58 smithi105 ceph-mon[35977]: pgmap v332: 129 pgs: 129 active+clean; 586 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail; 85 B/s rd, 2.3 KiB/s wr, 9 op/s 2024-05-27T07:22:59.190 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:22:58 smithi105 ceph-mon[35977]: from='client.? 172.21.15.52:0/1118125341' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]: dispatch 2024-05-27T07:22:59.205 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:22:58 smithi052 ceph-mon[31955]: pgmap v332: 129 pgs: 129 active+clean; 586 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail; 85 B/s rd, 2.3 KiB/s wr, 9 op/s 2024-05-27T07:22:59.205 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:22:58 smithi052 ceph-mon[31955]: from='client.? 172.21.15.52:0/1118125341' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]: dispatch 2024-05-27T07:22:59.352 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:22:58 smithi173 ceph-mon[37000]: pgmap v332: 129 pgs: 129 active+clean; 586 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail; 85 B/s rd, 2.3 KiB/s wr, 9 op/s 2024-05-27T07:22:59.352 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:22:58 smithi173 ceph-mon[37000]: from='client.? 172.21.15.52:0/1118125341' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]: dispatch 2024-05-27T07:22:59.540 DEBUG:tasks.cephfs.filesystem:Creating {'create': 2} subvols for filesystem 'cephfs' 2024-05-27T07:22:59.541 DEBUG:teuthology.orchestra.run.smithi052:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:38f9d900c83c63799fbdbe61acc9a11b0d3554a6 shell --fsid 73df2636-1bf8-11ef-bc9b-c7b262605968 -- ceph fs subvolumegroup create cephfs qa 2024-05-27T07:22:59.800 INFO:teuthology.orchestra.run.smithi052.stderr:Inferring config /var/lib/ceph/73df2636-1bf8-11ef-bc9b-c7b262605968/mon.a/config 2024-05-27T07:23:00.159 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:22:59 smithi052 ceph-mon[31955]: from='client.? 172.21.15.52:0/1118125341' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]': finished 2024-05-27T07:23:00.159 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:22:59 smithi052 ceph-mon[31955]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-05-27T07:23:00.190 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:22:59 smithi105 ceph-mon[35977]: from='client.? 172.21.15.52:0/1118125341' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]': finished 2024-05-27T07:23:00.190 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:22:59 smithi105 ceph-mon[35977]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-05-27T07:23:00.352 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:22:59 smithi173 ceph-mon[37000]: from='client.? 172.21.15.52:0/1118125341' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]': finished 2024-05-27T07:23:00.352 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:22:59 smithi173 ceph-mon[37000]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-05-27T07:23:01.183 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:23:00 smithi052 ceph-mon[31955]: pgmap v333: 129 pgs: 129 active+clean; 586 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail; 1.8 KiB/s rd, 3.2 KiB/s wr, 14 op/s 2024-05-27T07:23:01.190 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:23:00 smithi105 ceph-mon[35977]: pgmap v333: 129 pgs: 129 active+clean; 586 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail; 1.8 KiB/s rd, 3.2 KiB/s wr, 14 op/s 2024-05-27T07:23:01.355 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:23:00 smithi173 ceph-mon[37000]: pgmap v333: 129 pgs: 129 active+clean; 586 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail; 1.8 KiB/s rd, 3.2 KiB/s wr, 14 op/s 2024-05-27T07:23:01.395 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:23:01.404+0000 7fe029b54700 1 -- 172.21.15.52:0/3956738597 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fe024107e10 msgr2=0x7fe024108220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:23:01.395 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:23:01.404+0000 7fe029b54700 1 --2- 172.21.15.52:0/3956738597 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fe024107e10 0x7fe024108220 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7fe00c0099c0 tx=0x7fe00c009cd0 comp rx=0 tx=0).stop 2024-05-27T07:23:01.395 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:23:01.404+0000 7fe029b54700 1 -- 172.21.15.52:0/3956738597 shutdown_connections 2024-05-27T07:23:01.395 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:23:01.404+0000 7fe029b54700 1 --2- 172.21.15.52:0/3956738597 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fe024108760 0x7fe0241100c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:23:01.395 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:23:01.404+0000 7fe029b54700 1 --2- 172.21.15.52:0/3956738597 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fe024103530 0x7fe024103940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:23:01.396 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:23:01.404+0000 7fe029b54700 1 --2- 172.21.15.52:0/3956738597 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fe024107e10 0x7fe024108220 unknown :-1 s=CLOSED pgs=225 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:23:01.396 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:23:01.404+0000 7fe029b54700 1 -- 172.21.15.52:0/3956738597 >> 172.21.15.52:0/3956738597 conn(0x7fe024078e40 msgr2=0x7fe024079280 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:23:01.396 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:23:01.404+0000 7fe029b54700 1 -- 172.21.15.52:0/3956738597 shutdown_connections 2024-05-27T07:23:01.396 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:23:01.404+0000 7fe029b54700 1 -- 172.21.15.52:0/3956738597 wait complete. 2024-05-27T07:23:01.398 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:23:01.405+0000 7fe029b54700 1 Processor -- start 2024-05-27T07:23:01.398 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:23:01.405+0000 7fe029b54700 1 -- start start 2024-05-27T07:23:01.398 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:23:01.405+0000 7fe029b54700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fe024103530 0x7fe02419ae10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:23:01.398 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:23:01.405+0000 7fe029b54700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fe024107e10 0x7fe02419b3b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:23:01.398 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:23:01.405+0000 7fe029b54700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fe024108760 0x7fe02419f640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:23:01.398 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:23:01.405+0000 7fe029b54700 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x7fe0241144e0 con 0x7fe024107e10 2024-05-27T07:23:01.398 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:23:01.405+0000 7fe029b54700 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7fe024114360 con 0x7fe024103530 2024-05-27T07:23:01.398 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:23:01.405+0000 7fe029b54700 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7fe024114660 con 0x7fe024108760 2024-05-27T07:23:01.398 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:23:01.406+0000 7fe022ffd700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fe024107e10 0x7fe02419b3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:23:01.398 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:23:01.406+0000 7fe022ffd700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fe024107e10 0x7fe02419b3b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.52:39392/0 (socket says 172.21.15.52:39392) 2024-05-27T07:23:01.398 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:23:01.406+0000 7fe022ffd700 1 -- 172.21.15.52:0/1871865967 learned_addr learned my addr 172.21.15.52:0/1871865967 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-05-27T07:23:01.399 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:23:01.406+0000 7fe0237fe700 1 --2- 172.21.15.52:0/1871865967 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fe024103530 0x7fe02419ae10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:23:01.399 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:23:01.406+0000 7fe023fff700 1 --2- 172.21.15.52:0/1871865967 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fe024108760 0x7fe02419f640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:23:01.399 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:23:01.406+0000 7fe022ffd700 1 -- 172.21.15.52:0/1871865967 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fe024108760 msgr2=0x7fe02419f640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:23:01.399 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:23:01.406+0000 7fe022ffd700 1 --2- 172.21.15.52:0/1871865967 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fe024108760 0x7fe02419f640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:23:01.399 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:23:01.406+0000 7fe022ffd700 1 -- 172.21.15.52:0/1871865967 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fe024103530 msgr2=0x7fe02419ae10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:23:01.399 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:23:01.406+0000 7fe022ffd700 1 --2- 172.21.15.52:0/1871865967 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fe024103530 0x7fe02419ae10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:23:01.399 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:23:01.406+0000 7fe022ffd700 1 -- 172.21.15.52:0/1871865967 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe02419fce0 con 0x7fe024107e10 2024-05-27T07:23:01.399 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:23:01.406+0000 7fe0237fe700 1 --2- 172.21.15.52:0/1871865967 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fe024103530 0x7fe02419ae10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-27T07:23:01.399 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:23:01.406+0000 7fe022ffd700 1 --2- 172.21.15.52:0/1871865967 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fe024107e10 0x7fe02419b3b0 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7fe00c0098e0 tx=0x7fe00c005e80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:23:01.399 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:23:01.406+0000 7fe020ff9700 1 -- 172.21.15.52:0/1871865967 <== mon.0 v2:172.21.15.52:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe00c027070 con 0x7fe024107e10 2024-05-27T07:23:01.399 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:23:01.406+0000 7fe029b54700 1 -- 172.21.15.52:0/1871865967 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe02419ff70 con 0x7fe024107e10 2024-05-27T07:23:01.399 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:23:01.406+0000 7fe029b54700 1 -- 172.21.15.52:0/1871865967 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe0241a78f0 con 0x7fe024107e10 2024-05-27T07:23:01.400 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:23:01.406+0000 7fe020ff9700 1 -- 172.21.15.52:0/1871865967 <== mon.0 v2:172.21.15.52:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fe00c003fa0 con 0x7fe024107e10 2024-05-27T07:23:01.400 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:23:01.407+0000 7fe020ff9700 1 -- 172.21.15.52:0/1871865967 <== mon.0 v2:172.21.15.52:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe00c022d60 con 0x7fe024107e10 2024-05-27T07:23:01.400 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:23:01.407+0000 7fe029b54700 1 -- 172.21.15.52:0/1871865967 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe004005330 con 0x7fe024107e10 2024-05-27T07:23:01.400 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:23:01.409+0000 7fe020ff9700 1 -- 172.21.15.52:0/1871865967 <== mon.0 v2:172.21.15.52:3300/0 4 ==== mgrmap(e 16) ==== 149652+0+0 (secure 0 0 0) 0x7fe00c022470 con 0x7fe024107e10 2024-05-27T07:23:01.400 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:23:01.410+0000 7fe020ff9700 1 --2- 172.21.15.52:0/1871865967 >> [v2:172.21.15.52:6800/3513075805,v1:172.21.15.52:6801/3513075805] conn(0x7fe0100b2880 0x7fe0100b4d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:23:01.401 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:23:01.410+0000 7fe0237fe700 1 --2- 172.21.15.52:0/1871865967 >> [v2:172.21.15.52:6800/3513075805,v1:172.21.15.52:6801/3513075805] conn(0x7fe0100b2880 0x7fe0100b4d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:23:01.401 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:23:01.410+0000 7fe020ff9700 1 -- 172.21.15.52:0/1871865967 <== mon.0 v2:172.21.15.52:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 9092+0+0 (secure 0 0 0) 0x7fe00c0e3fc0 con 0x7fe024107e10 2024-05-27T07:23:01.401 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:23:01.410+0000 7fe0237fe700 1 --2- 172.21.15.52:0/1871865967 >> [v2:172.21.15.52:6800/3513075805,v1:172.21.15.52:6801/3513075805] conn(0x7fe0100b2880 0x7fe0100b4d90 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7fe014005950 tx=0x7fe0140058e0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:23:01.404 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:23:01.413+0000 7fe020ff9700 1 -- 172.21.15.52:0/1871865967 <== mon.0 v2:172.21.15.52:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191963 (secure 0 0 0) 0x7fe00c013070 con 0x7fe024107e10 2024-05-27T07:23:01.652 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:23:01.661+0000 7fe029b54700 1 -- 172.21.15.52:0/1871865967 --> [v2:172.21.15.52:6800/3513075805,v1:172.21.15.52:6801/3513075805] -- mgr_command(tid 0: {"prefix": "fs subvolumegroup create", "vol_name": "cephfs", "group_name": "qa", "target": ["mon-mgr", ""]}) -- 0x7fe004000bc0 con 0x7fe0100b2880 2024-05-27T07:23:01.683 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:23:01.692+0000 7fe020ff9700 1 -- 172.21.15.52:0/1871865967 <== mgr.14152 v2:172.21.15.52:6800/3513075805 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+0 (secure 0 0 0) 0x7fe004000bc0 con 0x7fe0100b2880 2024-05-27T07:23:01.686 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:23:01.695+0000 7fe029b54700 1 -- 172.21.15.52:0/1871865967 >> [v2:172.21.15.52:6800/3513075805,v1:172.21.15.52:6801/3513075805] conn(0x7fe0100b2880 msgr2=0x7fe0100b4d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:23:01.686 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:23:01.695+0000 7fe029b54700 1 --2- 172.21.15.52:0/1871865967 >> [v2:172.21.15.52:6800/3513075805,v1:172.21.15.52:6801/3513075805] conn(0x7fe0100b2880 0x7fe0100b4d90 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7fe014005950 tx=0x7fe0140058e0 comp rx=0 tx=0).stop 2024-05-27T07:23:01.686 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:23:01.695+0000 7fe029b54700 1 -- 172.21.15.52:0/1871865967 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fe024107e10 msgr2=0x7fe02419b3b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:23:01.686 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:23:01.695+0000 7fe029b54700 1 --2- 172.21.15.52:0/1871865967 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fe024107e10 0x7fe02419b3b0 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7fe00c0098e0 tx=0x7fe00c005e80 comp rx=0 tx=0).stop 2024-05-27T07:23:01.686 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:23:01.695+0000 7fe029b54700 1 -- 172.21.15.52:0/1871865967 shutdown_connections 2024-05-27T07:23:01.686 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:23:01.695+0000 7fe029b54700 1 --2- 172.21.15.52:0/1871865967 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fe024108760 0x7fe02419f640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:23:01.686 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:23:01.695+0000 7fe029b54700 1 --2- 172.21.15.52:0/1871865967 >> [v2:172.21.15.52:6800/3513075805,v1:172.21.15.52:6801/3513075805] conn(0x7fe0100b2880 0x7fe0100b4d90 unknown :-1 s=CLOSED pgs=163 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:23:01.687 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:23:01.695+0000 7fe029b54700 1 --2- 172.21.15.52:0/1871865967 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fe024103530 0x7fe02419ae10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:23:01.687 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:23:01.695+0000 7fe029b54700 1 --2- 172.21.15.52:0/1871865967 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fe024107e10 0x7fe02419b3b0 unknown :-1 s=CLOSED pgs=226 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:23:01.687 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:23:01.695+0000 7fe029b54700 1 -- 172.21.15.52:0/1871865967 >> 172.21.15.52:0/1871865967 conn(0x7fe024078e40 msgr2=0x7fe024101ad0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:23:01.687 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:23:01.695+0000 7fe029b54700 1 -- 172.21.15.52:0/1871865967 shutdown_connections 2024-05-27T07:23:01.687 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:23:01.695+0000 7fe029b54700 1 -- 172.21.15.52:0/1871865967 wait complete. 2024-05-27T07:23:01.979 INFO:journalctl@ceph.mgr.x.smithi052.stdout:May 27 07:23:01 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mgr-x[32184]: 2024-05-27T07:23:01.666+0000 7f8a35237700 -1 client.0 error registering admin socket command: (17) File exists 2024-05-27T07:23:01.979 INFO:journalctl@ceph.mgr.x.smithi052.stdout:May 27 07:23:01 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mgr-x[32184]: 2024-05-27T07:23:01.666+0000 7f8a35237700 -1 client.0 error registering admin socket command: (17) File exists 2024-05-27T07:23:01.979 INFO:journalctl@ceph.mgr.x.smithi052.stdout:May 27 07:23:01 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mgr-x[32184]: 2024-05-27T07:23:01.666+0000 7f8a35237700 -1 client.0 error registering admin socket command: (17) File exists 2024-05-27T07:23:01.980 INFO:journalctl@ceph.mgr.x.smithi052.stdout:May 27 07:23:01 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mgr-x[32184]: 2024-05-27T07:23:01.666+0000 7f8a35237700 -1 client.0 error registering admin socket command: (17) File exists 2024-05-27T07:23:01.980 INFO:journalctl@ceph.mgr.x.smithi052.stdout:May 27 07:23:01 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mgr-x[32184]: 2024-05-27T07:23:01.666+0000 7f8a35237700 -1 client.0 error registering admin socket command: (17) File exists 2024-05-27T07:23:02.370 DEBUG:teuthology.orchestra.run.smithi052:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:38f9d900c83c63799fbdbe61acc9a11b0d3554a6 shell --fsid 73df2636-1bf8-11ef-bc9b-c7b262605968 -- ceph fs subvolume create cephfs sv_0 --group_name qa 2024-05-27T07:23:02.623 INFO:teuthology.orchestra.run.smithi052.stderr:Inferring config /var/lib/ceph/73df2636-1bf8-11ef-bc9b-c7b262605968/mon.a/config 2024-05-27T07:23:03.018 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:23:02 smithi052 ceph-mon[31955]: pgmap v334: 129 pgs: 129 active+clean; 587 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 4.0 KiB/s wr, 17 op/s 2024-05-27T07:23:03.019 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:23:02 smithi052 ceph-mon[31955]: from='client.14805 -' entity='client.admin' cmd=[{"prefix": "fs subvolumegroup create", "vol_name": "cephfs", "group_name": "qa", "target": ["mon-mgr", ""]}]: dispatch 2024-05-27T07:23:03.019 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:23:02 smithi052 ceph-mon[31955]: mgrmap e17: x(active, since 8m), standbys: z, y 2024-05-27T07:23:03.190 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:23:02 smithi105 ceph-mon[35977]: pgmap v334: 129 pgs: 129 active+clean; 587 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 4.0 KiB/s wr, 17 op/s 2024-05-27T07:23:03.190 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:23:02 smithi105 ceph-mon[35977]: from='client.14805 -' entity='client.admin' cmd=[{"prefix": "fs subvolumegroup create", "vol_name": "cephfs", "group_name": "qa", "target": ["mon-mgr", ""]}]: dispatch 2024-05-27T07:23:03.190 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:23:02 smithi105 ceph-mon[35977]: mgrmap e17: x(active, since 8m), standbys: z, y 2024-05-27T07:23:03.352 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:23:02 smithi173 ceph-mon[37000]: pgmap v334: 129 pgs: 129 active+clean; 587 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 4.0 KiB/s wr, 17 op/s 2024-05-27T07:23:03.352 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:23:02 smithi173 ceph-mon[37000]: from='client.14805 -' entity='client.admin' cmd=[{"prefix": "fs subvolumegroup create", "vol_name": "cephfs", "group_name": "qa", "target": ["mon-mgr", ""]}]: dispatch 2024-05-27T07:23:03.352 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:23:02 smithi173 ceph-mon[37000]: mgrmap e17: x(active, since 8m), standbys: z, y 2024-05-27T07:23:04.283 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:23:04.293+0000 7fcfe0b63700 1 -- 172.21.15.52:0/3839351788 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fcfdc1034d0 msgr2=0x7fcfdc1038e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:23:04.283 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:23:04.293+0000 7fcfe0b63700 1 --2- 172.21.15.52:0/3839351788 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fcfdc1034d0 0x7fcfdc1038e0 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7fcfcc0099c0 tx=0x7fcfcc009cd0 comp rx=0 tx=0).stop 2024-05-27T07:23:04.283 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:23:04.293+0000 7fcfe0b63700 1 -- 172.21.15.52:0/3839351788 shutdown_connections 2024-05-27T07:23:04.283 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:23:04.293+0000 7fcfe0b63700 1 --2- 172.21.15.52:0/3839351788 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fcfdc108750 0x7fcfdc1100b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:23:04.283 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:23:04.293+0000 7fcfe0b63700 1 --2- 172.21.15.52:0/3839351788 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fcfdc1034d0 0x7fcfdc1038e0 unknown :-1 s=CLOSED pgs=229 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:23:04.283 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:23:04.293+0000 7fcfe0b63700 1 --2- 172.21.15.52:0/3839351788 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fcfdc107e00 0x7fcfdc108210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:23:04.284 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:23:04.293+0000 7fcfe0b63700 1 -- 172.21.15.52:0/3839351788 >> 172.21.15.52:0/3839351788 conn(0x7fcfdc0fef20 msgr2=0x7fcfdc101340 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:23:04.284 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:23:04.293+0000 7fcfe0b63700 1 -- 172.21.15.52:0/3839351788 shutdown_connections 2024-05-27T07:23:04.284 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:23:04.293+0000 7fcfe0b63700 1 -- 172.21.15.52:0/3839351788 wait complete. 2024-05-27T07:23:04.285 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:23:04.294+0000 7fcfe0b63700 1 Processor -- start 2024-05-27T07:23:04.286 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:23:04.294+0000 7fcfe0b63700 1 -- start start 2024-05-27T07:23:04.286 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:23:04.294+0000 7fcfe0b63700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fcfdc1034d0 0x7fcfdc19ade0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:23:04.286 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:23:04.294+0000 7fcfe0b63700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fcfdc107e00 0x7fcfdc19b380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:23:04.286 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:23:04.294+0000 7fcfe0b63700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fcfdc108750 0x7fcfdc19f610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:23:04.286 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:23:04.294+0000 7fcfe0b63700 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x7fcfdc113380 con 0x7fcfdc107e00 2024-05-27T07:23:04.286 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:23:04.294+0000 7fcfe0b63700 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7fcfdc113200 con 0x7fcfdc1034d0 2024-05-27T07:23:04.286 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:23:04.294+0000 7fcfe0b63700 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7fcfdc113500 con 0x7fcfdc108750 2024-05-27T07:23:04.286 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:23:04.294+0000 7fcfd1d9b700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fcfdc107e00 0x7fcfdc19b380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:23:04.286 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:23:04.294+0000 7fcfd1d9b700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fcfdc107e00 0x7fcfdc19b380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.52:39436/0 (socket says 172.21.15.52:39436) 2024-05-27T07:23:04.286 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:23:04.294+0000 7fcfd1d9b700 1 -- 172.21.15.52:0/1507816454 learned_addr learned my addr 172.21.15.52:0/1507816454 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-05-27T07:23:04.286 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:23:04.295+0000 7fcfda59c700 1 --2- 172.21.15.52:0/1507816454 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fcfdc1034d0 0x7fcfdc19ade0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:23:04.287 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:23:04.295+0000 7fcfdad9d700 1 --2- 172.21.15.52:0/1507816454 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fcfdc108750 0x7fcfdc19f610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:23:04.287 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:23:04.295+0000 7fcfd1d9b700 1 -- 172.21.15.52:0/1507816454 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fcfdc108750 msgr2=0x7fcfdc19f610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:23:04.287 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:23:04.295+0000 7fcfd1d9b700 1 --2- 172.21.15.52:0/1507816454 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fcfdc108750 0x7fcfdc19f610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:23:04.287 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:23:04.295+0000 7fcfd1d9b700 1 -- 172.21.15.52:0/1507816454 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fcfdc1034d0 msgr2=0x7fcfdc19ade0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:23:04.287 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:23:04.295+0000 7fcfd1d9b700 1 --2- 172.21.15.52:0/1507816454 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fcfdc1034d0 0x7fcfdc19ade0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:23:04.287 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:23:04.295+0000 7fcfd1d9b700 1 -- 172.21.15.52:0/1507816454 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcfdc19fcb0 con 0x7fcfdc107e00 2024-05-27T07:23:04.287 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:23:04.295+0000 7fcfda59c700 1 --2- 172.21.15.52:0/1507816454 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fcfdc1034d0 0x7fcfdc19ade0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-27T07:23:04.287 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:23:04.295+0000 7fcfdad9d700 1 --2- 172.21.15.52:0/1507816454 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fcfdc108750 0x7fcfdc19f610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-27T07:23:04.287 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:23:04.295+0000 7fcfd1d9b700 1 --2- 172.21.15.52:0/1507816454 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fcfdc107e00 0x7fcfdc19b380 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7fcfc400d980 tx=0x7fcfc400dd40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:23:04.287 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:23:04.295+0000 7fcfd3fff700 1 -- 172.21.15.52:0/1507816454 <== mon.0 v2:172.21.15.52:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcfc4017070 con 0x7fcfdc107e00 2024-05-27T07:23:04.287 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:23:04.295+0000 7fcfd3fff700 1 -- 172.21.15.52:0/1507816454 <== mon.0 v2:172.21.15.52:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fcfc4010460 con 0x7fcfdc107e00 2024-05-27T07:23:04.287 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:23:04.295+0000 7fcfd3fff700 1 -- 172.21.15.52:0/1507816454 <== mon.0 v2:172.21.15.52:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcfc400de60 con 0x7fcfdc107e00 2024-05-27T07:23:04.287 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:23:04.295+0000 7fcfe0b63700 1 -- 172.21.15.52:0/1507816454 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fcfdc19ffa0 con 0x7fcfdc107e00 2024-05-27T07:23:04.288 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:23:04.295+0000 7fcfe0b63700 1 -- 172.21.15.52:0/1507816454 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fcfdc1a78c0 con 0x7fcfdc107e00 2024-05-27T07:23:04.288 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:23:04.297+0000 7fcfe0b63700 1 -- 172.21.15.52:0/1507816454 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fcfdc0512b0 con 0x7fcfdc107e00 2024-05-27T07:23:04.288 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:23:04.298+0000 7fcfd3fff700 1 -- 172.21.15.52:0/1507816454 <== mon.0 v2:172.21.15.52:3300/0 4 ==== mgrmap(e 17) ==== 149703+0+0 (secure 0 0 0) 0x7fcfc4009b40 con 0x7fcfdc107e00 2024-05-27T07:23:04.288 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:23:04.299+0000 7fcfd3fff700 1 --2- 172.21.15.52:0/1507816454 >> [v2:172.21.15.52:6800/3513075805,v1:172.21.15.52:6801/3513075805] conn(0x7fcfc80b2910 0x7fcfc80b4e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:23:04.289 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:23:04.299+0000 7fcfda59c700 1 --2- 172.21.15.52:0/1507816454 >> [v2:172.21.15.52:6800/3513075805,v1:172.21.15.52:6801/3513075805] conn(0x7fcfc80b2910 0x7fcfc80b4e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:23:04.289 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:23:04.299+0000 7fcfd3fff700 1 -- 172.21.15.52:0/1507816454 <== mon.0 v2:172.21.15.52:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 9092+0+0 (secure 0 0 0) 0x7fcfc40e22f0 con 0x7fcfdc107e00 2024-05-27T07:23:04.289 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:23:04.299+0000 7fcfda59c700 1 --2- 172.21.15.52:0/1507816454 >> [v2:172.21.15.52:6800/3513075805,v1:172.21.15.52:6801/3513075805] conn(0x7fcfc80b2910 0x7fcfc80b4e20 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7fcfcc000c00 tx=0x7fcfcc01a040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:23:04.293 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:23:04.304+0000 7fcfd3fff700 1 -- 172.21.15.52:0/1507816454 <== mon.0 v2:172.21.15.52:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191963 (secure 0 0 0) 0x7fcfc40a9b50 con 0x7fcfdc107e00 2024-05-27T07:23:04.542 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:23:04.552+0000 7fcfe0b63700 1 -- 172.21.15.52:0/1507816454 --> [v2:172.21.15.52:6800/3513075805,v1:172.21.15.52:6801/3513075805] -- mgr_command(tid 0: {"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "sv_0", "group_name": "qa", "target": ["mon-mgr", ""]}) -- 0x7fcfdc19c140 con 0x7fcfc80b2910 2024-05-27T07:23:04.569 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:23:04.579+0000 7fcfd3fff700 1 -- 172.21.15.52:0/1507816454 <== mgr.14152 v2:172.21.15.52:6800/3513075805 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+0 (secure 0 0 0) 0x7fcfdc19c140 con 0x7fcfc80b2910 2024-05-27T07:23:04.571 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:23:04.581+0000 7fcfe0b63700 1 -- 172.21.15.52:0/1507816454 >> [v2:172.21.15.52:6800/3513075805,v1:172.21.15.52:6801/3513075805] conn(0x7fcfc80b2910 msgr2=0x7fcfc80b4e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:23:04.572 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:23:04.581+0000 7fcfe0b63700 1 --2- 172.21.15.52:0/1507816454 >> [v2:172.21.15.52:6800/3513075805,v1:172.21.15.52:6801/3513075805] conn(0x7fcfc80b2910 0x7fcfc80b4e20 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7fcfcc000c00 tx=0x7fcfcc01a040 comp rx=0 tx=0).stop 2024-05-27T07:23:04.572 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:23:04.581+0000 7fcfe0b63700 1 -- 172.21.15.52:0/1507816454 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fcfdc107e00 msgr2=0x7fcfdc19b380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:23:04.572 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:23:04.581+0000 7fcfe0b63700 1 --2- 172.21.15.52:0/1507816454 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fcfdc107e00 0x7fcfdc19b380 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7fcfc400d980 tx=0x7fcfc400dd40 comp rx=0 tx=0).stop 2024-05-27T07:23:04.572 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:23:04.581+0000 7fcfe0b63700 1 -- 172.21.15.52:0/1507816454 shutdown_connections 2024-05-27T07:23:04.572 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:23:04.581+0000 7fcfe0b63700 1 --2- 172.21.15.52:0/1507816454 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fcfdc108750 0x7fcfdc19f610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:23:04.572 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:23:04.581+0000 7fcfe0b63700 1 --2- 172.21.15.52:0/1507816454 >> [v2:172.21.15.52:6800/3513075805,v1:172.21.15.52:6801/3513075805] conn(0x7fcfc80b2910 0x7fcfc80b4e20 unknown :-1 s=CLOSED pgs=164 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:23:04.572 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:23:04.581+0000 7fcfe0b63700 1 --2- 172.21.15.52:0/1507816454 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fcfdc1034d0 0x7fcfdc19ade0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:23:04.572 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:23:04.581+0000 7fcfe0b63700 1 --2- 172.21.15.52:0/1507816454 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fcfdc107e00 0x7fcfdc19b380 unknown :-1 s=CLOSED pgs=230 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:23:04.572 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:23:04.581+0000 7fcfe0b63700 1 -- 172.21.15.52:0/1507816454 >> 172.21.15.52:0/1507816454 conn(0x7fcfdc0fef20 msgr2=0x7fcfdc06c3d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:23:04.572 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:23:04.582+0000 7fcfe0b63700 1 -- 172.21.15.52:0/1507816454 shutdown_connections 2024-05-27T07:23:04.572 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:23:04.582+0000 7fcfe0b63700 1 -- 172.21.15.52:0/1507816454 wait complete. 2024-05-27T07:23:05.183 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:23:04 smithi052 ceph-mon[31955]: pgmap v335: 129 pgs: 129 active+clean; 587 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 4.0 KiB/s wr, 19 op/s 2024-05-27T07:23:05.190 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:23:04 smithi105 ceph-mon[35977]: pgmap v335: 129 pgs: 129 active+clean; 587 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 4.0 KiB/s wr, 19 op/s 2024-05-27T07:23:05.213 DEBUG:teuthology.orchestra.run.smithi052:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:38f9d900c83c63799fbdbe61acc9a11b0d3554a6 shell --fsid 73df2636-1bf8-11ef-bc9b-c7b262605968 -- ceph fs subvolume getpath cephfs --group_name qa sv_0 2024-05-27T07:23:05.352 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:23:04 smithi173 ceph-mon[37000]: pgmap v335: 129 pgs: 129 active+clean; 587 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 4.0 KiB/s wr, 19 op/s 2024-05-27T07:23:05.465 INFO:teuthology.orchestra.run.smithi052.stderr:Inferring config /var/lib/ceph/73df2636-1bf8-11ef-bc9b-c7b262605968/mon.a/config 2024-05-27T07:23:06.190 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:23:05 smithi105 ceph-mon[35977]: from='client.14817 -' entity='client.admin' cmd=[{"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "sv_0", "group_name": "qa", "target": ["mon-mgr", ""]}]: dispatch 2024-05-27T07:23:06.229 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:23:05 smithi052 ceph-mon[31955]: from='client.14817 -' entity='client.admin' cmd=[{"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "sv_0", "group_name": "qa", "target": ["mon-mgr", ""]}]: dispatch 2024-05-27T07:23:06.352 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:23:05 smithi173 ceph-mon[37000]: from='client.14817 -' entity='client.admin' cmd=[{"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "sv_0", "group_name": "qa", "target": ["mon-mgr", ""]}]: dispatch 2024-05-27T07:23:07.182 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:23:07.192+0000 7ff57731b700 1 -- 172.21.15.52:0/283474526 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff5701089e0 msgr2=0x7ff570108df0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:23:07.183 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:23:07.192+0000 7ff57731b700 1 --2- 172.21.15.52:0/283474526 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff5701089e0 0x7ff570108df0 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7ff56c009990 tx=0x7ff56c009ca0 comp rx=0 tx=0).stop 2024-05-27T07:23:07.183 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:23:07.192+0000 7ff57731b700 1 -- 172.21.15.52:0/283474526 shutdown_connections 2024-05-27T07:23:07.183 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:23:07.192+0000 7ff57731b700 1 --2- 172.21.15.52:0/283474526 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7ff570103300 0x7ff57010f880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:23:07.183 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:23:07.192+0000 7ff57731b700 1 --2- 172.21.15.52:0/283474526 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff5701089e0 0x7ff570108df0 unknown :-1 s=CLOSED pgs=231 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:23:07.183 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:23:07.192+0000 7ff57731b700 1 --2- 172.21.15.52:0/283474526 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7ff5701029b0 0x7ff570102dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:23:07.183 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:23:07.192+0000 7ff57731b700 1 -- 172.21.15.52:0/283474526 >> 172.21.15.52:0/283474526 conn(0x7ff5700fe4d0 msgr2=0x7ff570100920 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:23:07.183 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:23:07.192+0000 7ff57731b700 1 -- 172.21.15.52:0/283474526 shutdown_connections 2024-05-27T07:23:07.183 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:23:07.192+0000 7ff57731b700 1 -- 172.21.15.52:0/283474526 wait complete. 2024-05-27T07:23:07.184 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:23:06 smithi052 ceph-mon[31955]: pgmap v336: 129 pgs: 129 active+clean; 608 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 6.7 KiB/s rd, 6.2 KiB/s wr, 22 op/s 2024-05-27T07:23:07.185 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:23:07.193+0000 7ff57731b700 1 Processor -- start 2024-05-27T07:23:07.185 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:23:07.193+0000 7ff57731b700 1 -- start start 2024-05-27T07:23:07.185 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:23:07.193+0000 7ff57731b700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7ff5701029b0 0x7ff57019f960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:23:07.185 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:23:07.193+0000 7ff57731b700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7ff570103300 0x7ff570198a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:23:07.185 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:23:07.193+0000 7ff57731b700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff5701089e0 0x7ff570199130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:23:07.186 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:23:07.193+0000 7ff57731b700 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x7ff5701122d0 con 0x7ff5701089e0 2024-05-27T07:23:07.186 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:23:07.193+0000 7ff57731b700 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7ff570112150 con 0x7ff5701029b0 2024-05-27T07:23:07.186 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:23:07.193+0000 7ff57731b700 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7ff570112450 con 0x7ff570103300 2024-05-27T07:23:07.186 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:23:07.193+0000 7ff5758b8700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff5701089e0 0x7ff570199130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:23:07.186 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:23:07.193+0000 7ff5758b8700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff5701089e0 0x7ff570199130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.52:39462/0 (socket says 172.21.15.52:39462) 2024-05-27T07:23:07.186 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:23:07.193+0000 7ff5758b8700 1 -- 172.21.15.52:0/3858403206 learned_addr learned my addr 172.21.15.52:0/3858403206 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-05-27T07:23:07.186 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:23:07.193+0000 7ff5750b7700 1 --2- 172.21.15.52:0/3858403206 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7ff5701029b0 0x7ff57019f960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:23:07.186 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:23:07.193+0000 7ff5748b6700 1 --2- 172.21.15.52:0/3858403206 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7ff570103300 0x7ff570198a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:23:07.186 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:23:07.194+0000 7ff5758b8700 1 -- 172.21.15.52:0/3858403206 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7ff570103300 msgr2=0x7ff570198a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:23:07.186 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:23:07.194+0000 7ff5758b8700 1 --2- 172.21.15.52:0/3858403206 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7ff570103300 0x7ff570198a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:23:07.186 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:23:07.194+0000 7ff5758b8700 1 -- 172.21.15.52:0/3858403206 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7ff5701029b0 msgr2=0x7ff57019f960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:23:07.186 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:23:07.194+0000 7ff5758b8700 1 --2- 172.21.15.52:0/3858403206 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7ff5701029b0 0x7ff57019f960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:23:07.186 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:23:07.194+0000 7ff5758b8700 1 -- 172.21.15.52:0/3858403206 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff57019d490 con 0x7ff5701089e0 2024-05-27T07:23:07.187 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:23:07.194+0000 7ff5758b8700 1 --2- 172.21.15.52:0/3858403206 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff5701089e0 0x7ff570199130 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7ff568007ee0 tx=0x7ff56800d3b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:23:07.187 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:23:07.194+0000 7ff5667fc700 1 -- 172.21.15.52:0/3858403206 <== mon.0 v2:172.21.15.52:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff568004bb0 con 0x7ff5701089e0 2024-05-27T07:23:07.187 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:23:07.194+0000 7ff57731b700 1 -- 172.21.15.52:0/3858403206 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff57019d780 con 0x7ff5701089e0 2024-05-27T07:23:07.187 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:23:07.194+0000 7ff5667fc700 1 -- 172.21.15.52:0/3858403206 <== mon.0 v2:172.21.15.52:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ff568004d50 con 0x7ff5701089e0 2024-05-27T07:23:07.187 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:23:07.194+0000 7ff5667fc700 1 -- 172.21.15.52:0/3858403206 <== mon.0 v2:172.21.15.52:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff568005920 con 0x7ff5701089e0 2024-05-27T07:23:07.187 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:23:07.194+0000 7ff57731b700 1 -- 172.21.15.52:0/3858403206 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff57019dc40 con 0x7ff5701089e0 2024-05-27T07:23:07.187 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:23:07.195+0000 7ff57731b700 1 -- 172.21.15.52:0/3858403206 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff5701040f0 con 0x7ff5701089e0 2024-05-27T07:23:07.188 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:23:07.198+0000 7ff5667fc700 1 -- 172.21.15.52:0/3858403206 <== mon.0 v2:172.21.15.52:3300/0 4 ==== mgrmap(e 17) ==== 149703+0+0 (secure 0 0 0) 0x7ff56801a020 con 0x7ff5701089e0 2024-05-27T07:23:07.189 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:23:07.199+0000 7ff5667fc700 1 --2- 172.21.15.52:0/3858403206 >> [v2:172.21.15.52:6800/3513075805,v1:172.21.15.52:6801/3513075805] conn(0x7ff55c0b2910 0x7ff55c0b4e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:23:07.189 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:23:07.199+0000 7ff5750b7700 1 --2- 172.21.15.52:0/3858403206 >> [v2:172.21.15.52:6800/3513075805,v1:172.21.15.52:6801/3513075805] conn(0x7ff55c0b2910 0x7ff55c0b4e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:23:07.190 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:23:07.199+0000 7ff5667fc700 1 -- 172.21.15.52:0/3858403206 <== mon.0 v2:172.21.15.52:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 9092+0+0 (secure 0 0 0) 0x7ff5680e23d0 con 0x7ff5701089e0 2024-05-27T07:23:07.190 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:23:07.200+0000 7ff5750b7700 1 --2- 172.21.15.52:0/3858403206 >> [v2:172.21.15.52:6800/3513075805,v1:172.21.15.52:6801/3513075805] conn(0x7ff55c0b2910 0x7ff55c0b4e20 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7ff56c006010 tx=0x7ff56c01a040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:23:07.190 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:23:06 smithi105 ceph-mon[35977]: pgmap v336: 129 pgs: 129 active+clean; 608 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 6.7 KiB/s rd, 6.2 KiB/s wr, 22 op/s 2024-05-27T07:23:07.192 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:23:07.203+0000 7ff5667fc700 1 -- 172.21.15.52:0/3858403206 <== mon.0 v2:172.21.15.52:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191963 (secure 0 0 0) 0x7ff568010070 con 0x7ff5701089e0 2024-05-27T07:23:07.352 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:23:06 smithi173 ceph-mon[37000]: pgmap v336: 129 pgs: 129 active+clean; 608 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 6.7 KiB/s rd, 6.2 KiB/s wr, 22 op/s 2024-05-27T07:23:07.447 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:23:07.457+0000 7ff57731b700 1 -- 172.21.15.52:0/3858403206 --> [v2:172.21.15.52:6800/3513075805,v1:172.21.15.52:6801/3513075805] -- mgr_command(tid 0: {"prefix": "fs subvolume getpath", "vol_name": "cephfs", "group_name": "qa", "sub_name": "sv_0", "target": ["mon-mgr", ""]}) -- 0x7ff570199fc0 con 0x7ff55c0b2910 2024-05-27T07:23:07.454 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:23:07.462+0000 7ff5667fc700 1 -- 172.21.15.52:0/3858403206 <== mgr.14152 v2:172.21.15.52:6800/3513075805 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+54 (secure 0 0 0) 0x7ff570199fc0 con 0x7ff55c0b2910 2024-05-27T07:23:07.454 INFO:teuthology.orchestra.run.smithi052.stdout:/volumes/qa/sv_0/7b08c89f-e16f-4253-9327-bc5f7758bb10 2024-05-27T07:23:07.455 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:23:07.464+0000 7ff57731b700 1 -- 172.21.15.52:0/3858403206 >> [v2:172.21.15.52:6800/3513075805,v1:172.21.15.52:6801/3513075805] conn(0x7ff55c0b2910 msgr2=0x7ff55c0b4e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:23:07.455 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:23:07.464+0000 7ff57731b700 1 --2- 172.21.15.52:0/3858403206 >> [v2:172.21.15.52:6800/3513075805,v1:172.21.15.52:6801/3513075805] conn(0x7ff55c0b2910 0x7ff55c0b4e20 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7ff56c006010 tx=0x7ff56c01a040 comp rx=0 tx=0).stop 2024-05-27T07:23:07.455 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:23:07.464+0000 7ff57731b700 1 -- 172.21.15.52:0/3858403206 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff5701089e0 msgr2=0x7ff570199130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:23:07.455 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:23:07.464+0000 7ff57731b700 1 --2- 172.21.15.52:0/3858403206 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff5701089e0 0x7ff570199130 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7ff568007ee0 tx=0x7ff56800d3b0 comp rx=0 tx=0).stop 2024-05-27T07:23:07.456 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:23:07.464+0000 7ff57731b700 1 -- 172.21.15.52:0/3858403206 shutdown_connections 2024-05-27T07:23:07.456 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:23:07.464+0000 7ff57731b700 1 --2- 172.21.15.52:0/3858403206 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff5701089e0 0x7ff570199130 unknown :-1 s=CLOSED pgs=232 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:23:07.456 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:23:07.464+0000 7ff57731b700 1 --2- 172.21.15.52:0/3858403206 >> [v2:172.21.15.52:6800/3513075805,v1:172.21.15.52:6801/3513075805] conn(0x7ff55c0b2910 0x7ff55c0b4e20 unknown :-1 s=CLOSED pgs=165 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:23:07.456 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:23:07.464+0000 7ff57731b700 1 --2- 172.21.15.52:0/3858403206 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7ff5701029b0 0x7ff57019f960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:23:07.456 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:23:07.464+0000 7ff57731b700 1 --2- 172.21.15.52:0/3858403206 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7ff570103300 0x7ff570198a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:23:07.456 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:23:07.464+0000 7ff57731b700 1 -- 172.21.15.52:0/3858403206 >> 172.21.15.52:0/3858403206 conn(0x7ff5700fe4d0 msgr2=0x7ff5701008e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:23:07.456 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:23:07.465+0000 7ff57731b700 1 -- 172.21.15.52:0/3858403206 shutdown_connections 2024-05-27T07:23:07.456 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:23:07.465+0000 7ff57731b700 1 -- 172.21.15.52:0/3858403206 wait complete. 2024-05-27T07:23:08.125 DEBUG:teuthology.orchestra.run.smithi052:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:38f9d900c83c63799fbdbe61acc9a11b0d3554a6 shell --fsid 73df2636-1bf8-11ef-bc9b-c7b262605968 -- ceph fs subvolume create cephfs sv_1 --group_name qa 2024-05-27T07:23:08.382 INFO:teuthology.orchestra.run.smithi052.stderr:Inferring config /var/lib/ceph/73df2636-1bf8-11ef-bc9b-c7b262605968/mon.a/config 2024-05-27T07:23:09.190 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:23:08 smithi105 ceph-mon[35977]: from='client.14823 -' entity='client.admin' cmd=[{"prefix": "fs subvolume getpath", "vol_name": "cephfs", "group_name": "qa", "sub_name": "sv_0", "target": ["mon-mgr", ""]}]: dispatch 2024-05-27T07:23:09.190 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:23:08 smithi105 ceph-mon[35977]: pgmap v337: 129 pgs: 129 active+clean; 608 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 7.2 KiB/s rd, 4.6 KiB/s wr, 19 op/s 2024-05-27T07:23:09.202 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:23:08 smithi052 ceph-mon[31955]: from='client.14823 -' entity='client.admin' cmd=[{"prefix": "fs subvolume getpath", "vol_name": "cephfs", "group_name": "qa", "sub_name": "sv_0", "target": ["mon-mgr", ""]}]: dispatch 2024-05-27T07:23:09.202 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:23:08 smithi052 ceph-mon[31955]: pgmap v337: 129 pgs: 129 active+clean; 608 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 7.2 KiB/s rd, 4.6 KiB/s wr, 19 op/s 2024-05-27T07:23:09.352 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:23:08 smithi173 ceph-mon[37000]: from='client.14823 -' entity='client.admin' cmd=[{"prefix": "fs subvolume getpath", "vol_name": "cephfs", "group_name": "qa", "sub_name": "sv_0", "target": ["mon-mgr", ""]}]: dispatch 2024-05-27T07:23:09.352 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:23:08 smithi173 ceph-mon[37000]: pgmap v337: 129 pgs: 129 active+clean; 608 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 7.2 KiB/s rd, 4.6 KiB/s wr, 19 op/s 2024-05-27T07:23:10.165 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:23:10.174+0000 7f991b377700 1 -- 172.21.15.52:0/482542111 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f9914107de0 msgr2=0x7f99141081f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:23:10.165 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:23:10.174+0000 7f991b377700 1 --2- 172.21.15.52:0/482542111 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f9914107de0 0x7f99141081f0 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7f99080099c0 tx=0x7f9908009cd0 comp rx=0 tx=0).stop 2024-05-27T07:23:10.165 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:23:10.174+0000 7f991b377700 1 -- 172.21.15.52:0/482542111 shutdown_connections 2024-05-27T07:23:10.165 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:23:10.174+0000 7f991b377700 1 --2- 172.21.15.52:0/482542111 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f9914108730 0x7f9914110090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:23:10.165 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:23:10.174+0000 7f991b377700 1 --2- 172.21.15.52:0/482542111 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f99141034b0 0x7f99141038c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:23:10.165 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:23:10.174+0000 7f991b377700 1 --2- 172.21.15.52:0/482542111 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f9914107de0 0x7f99141081f0 unknown :-1 s=CLOSED pgs=233 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:23:10.165 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:23:10.174+0000 7f991b377700 1 -- 172.21.15.52:0/482542111 >> 172.21.15.52:0/482542111 conn(0x7f99140fede0 msgr2=0x7f9914101200 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:23:10.165 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:23:10.175+0000 7f991b377700 1 -- 172.21.15.52:0/482542111 shutdown_connections 2024-05-27T07:23:10.165 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:23:10.175+0000 7f991b377700 1 -- 172.21.15.52:0/482542111 wait complete. 2024-05-27T07:23:10.167 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:23:10.175+0000 7f991b377700 1 Processor -- start 2024-05-27T07:23:10.167 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:23:10.175+0000 7f991b377700 1 -- start start 2024-05-27T07:23:10.167 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:23:10.176+0000 7f991b377700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f99141034b0 0x7f991419bc80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:23:10.167 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:23:10.176+0000 7f991b377700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f9914107de0 0x7f991419c1c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:23:10.167 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:23:10.176+0000 7f991b377700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f9914108730 0x7f991419c720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:23:10.167 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:23:10.176+0000 7f991b377700 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x7f9914113360 con 0x7f9914108730 2024-05-27T07:23:10.167 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:23:10.176+0000 7f991b377700 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f99141131e0 con 0x7f9914107de0 2024-05-27T07:23:10.168 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:23:10.176+0000 7f991b377700 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f99141134e0 con 0x7f99141034b0 2024-05-27T07:23:10.168 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:23:10.176+0000 7f9919914700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f9914108730 0x7f991419c720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:23:10.168 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:23:10.176+0000 7f9919914700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f9914108730 0x7f991419c720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.52:39484/0 (socket says 172.21.15.52:39484) 2024-05-27T07:23:10.168 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:23:10.176+0000 7f9919914700 1 -- 172.21.15.52:0/1023919947 learned_addr learned my addr 172.21.15.52:0/1023919947 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-05-27T07:23:10.168 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:23:10.176+0000 7f9918912700 1 --2- 172.21.15.52:0/1023919947 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f9914107de0 0x7f991419c1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:23:10.168 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:23:10.176+0000 7f9919113700 1 --2- 172.21.15.52:0/1023919947 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f99141034b0 0x7f991419bc80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:23:10.168 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:23:10.176+0000 7f9919914700 1 -- 172.21.15.52:0/1023919947 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f99141034b0 msgr2=0x7f991419bc80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:23:10.168 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:23:10.176+0000 7f9919914700 1 --2- 172.21.15.52:0/1023919947 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f99141034b0 0x7f991419bc80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:23:10.168 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:23:10.176+0000 7f9919914700 1 -- 172.21.15.52:0/1023919947 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f9914107de0 msgr2=0x7f991419c1c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:23:10.168 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:23:10.176+0000 7f9919914700 1 --2- 172.21.15.52:0/1023919947 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f9914107de0 0x7f991419c1c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:23:10.168 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:23:10.176+0000 7f9919914700 1 -- 172.21.15.52:0/1023919947 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f99141a6fd0 con 0x7f9914108730 2024-05-27T07:23:10.169 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:23:10.177+0000 7f9919914700 1 --2- 172.21.15.52:0/1023919947 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f9914108730 0x7f991419c720 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7f990400d980 tx=0x7f990400dc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:23:10.169 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:23:10.177+0000 7f99127fc700 1 -- 172.21.15.52:0/1023919947 <== mon.0 v2:172.21.15.52:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9904017070 con 0x7f9914108730 2024-05-27T07:23:10.169 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:23:10.177+0000 7f99127fc700 1 -- 172.21.15.52:0/1023919947 <== mon.0 v2:172.21.15.52:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f9904005020 con 0x7f9914108730 2024-05-27T07:23:10.169 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:23:10.177+0000 7f991b377700 1 -- 172.21.15.52:0/1023919947 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f99141a72c0 con 0x7f9914108730 2024-05-27T07:23:10.169 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:23:10.177+0000 7f99127fc700 1 -- 172.21.15.52:0/1023919947 <== mon.0 v2:172.21.15.52:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f990400fa10 con 0x7f9914108730 2024-05-27T07:23:10.169 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:23:10.177+0000 7f991b377700 1 -- 172.21.15.52:0/1023919947 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f99141a7780 con 0x7f9914108730 2024-05-27T07:23:10.169 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:23:10.178+0000 7f991b377700 1 -- 172.21.15.52:0/1023919947 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f99140512b0 con 0x7f9914108730 2024-05-27T07:23:10.170 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:23:10.180+0000 7f99127fc700 1 -- 172.21.15.52:0/1023919947 <== mon.0 v2:172.21.15.52:3300/0 4 ==== mgrmap(e 17) ==== 149703+0+0 (secure 0 0 0) 0x7f9904004a30 con 0x7f9914108730 2024-05-27T07:23:10.171 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:23:10.181+0000 7f99127fc700 1 --2- 172.21.15.52:0/1023919947 >> [v2:172.21.15.52:6800/3513075805,v1:172.21.15.52:6801/3513075805] conn(0x7f98fc0b2910 0x7f98fc0b4e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:23:10.171 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:23:10.181+0000 7f9919113700 1 --2- 172.21.15.52:0/1023919947 >> [v2:172.21.15.52:6800/3513075805,v1:172.21.15.52:6801/3513075805] conn(0x7f98fc0b2910 0x7f98fc0b4e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:23:10.171 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:23:10.181+0000 7f99127fc700 1 -- 172.21.15.52:0/1023919947 <== mon.0 v2:172.21.15.52:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 9092+0+0 (secure 0 0 0) 0x7f99040e2910 con 0x7f9914108730 2024-05-27T07:23:10.172 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:23:10.182+0000 7f9919113700 1 --2- 172.21.15.52:0/1023919947 >> [v2:172.21.15.52:6800/3513075805,v1:172.21.15.52:6801/3513075805] conn(0x7f98fc0b2910 0x7f98fc0b4e20 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7f990c003f90 tx=0x7f990c009180 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:23:10.175 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:23:10.185+0000 7f99127fc700 1 -- 172.21.15.52:0/1023919947 <== mon.0 v2:172.21.15.52:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191963 (secure 0 0 0) 0x7f9904020020 con 0x7f9914108730 2024-05-27T07:23:10.422 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:23:10.432+0000 7f991b377700 1 -- 172.21.15.52:0/1023919947 --> [v2:172.21.15.52:6800/3513075805,v1:172.21.15.52:6801/3513075805] -- mgr_command(tid 0: {"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "sv_1", "group_name": "qa", "target": ["mon-mgr", ""]}) -- 0x7f9914063bf0 con 0x7f98fc0b2910 2024-05-27T07:23:10.454 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:23:10.464+0000 7f99127fc700 1 -- 172.21.15.52:0/1023919947 <== mgr.14152 v2:172.21.15.52:6800/3513075805 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+0 (secure 0 0 0) 0x7f9914063bf0 con 0x7f98fc0b2910 2024-05-27T07:23:10.459 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:23:10.466+0000 7f991b377700 1 -- 172.21.15.52:0/1023919947 >> [v2:172.21.15.52:6800/3513075805,v1:172.21.15.52:6801/3513075805] conn(0x7f98fc0b2910 msgr2=0x7f98fc0b4e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:23:10.459 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:23:10.466+0000 7f991b377700 1 --2- 172.21.15.52:0/1023919947 >> [v2:172.21.15.52:6800/3513075805,v1:172.21.15.52:6801/3513075805] conn(0x7f98fc0b2910 0x7f98fc0b4e20 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7f990c003f90 tx=0x7f990c009180 comp rx=0 tx=0).stop 2024-05-27T07:23:10.459 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:23:10.466+0000 7f991b377700 1 -- 172.21.15.52:0/1023919947 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f9914108730 msgr2=0x7f991419c720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:23:10.460 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:23:10.466+0000 7f991b377700 1 --2- 172.21.15.52:0/1023919947 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f9914108730 0x7f991419c720 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7f990400d980 tx=0x7f990400dc90 comp rx=0 tx=0).stop 2024-05-27T07:23:10.460 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:23:10.466+0000 7f991b377700 1 -- 172.21.15.52:0/1023919947 shutdown_connections 2024-05-27T07:23:10.460 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:23:10.466+0000 7f991b377700 1 --2- 172.21.15.52:0/1023919947 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f9914108730 0x7f991419c720 unknown :-1 s=CLOSED pgs=234 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:23:10.460 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:23:10.466+0000 7f991b377700 1 --2- 172.21.15.52:0/1023919947 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f99141034b0 0x7f991419bc80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:23:10.460 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:23:10.466+0000 7f991b377700 1 --2- 172.21.15.52:0/1023919947 >> [v2:172.21.15.52:6800/3513075805,v1:172.21.15.52:6801/3513075805] conn(0x7f98fc0b2910 0x7f98fc0b4e20 unknown :-1 s=CLOSED pgs=166 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:23:10.461 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:23:10.466+0000 7f991b377700 1 --2- 172.21.15.52:0/1023919947 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f9914107de0 0x7f991419c1c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:23:10.461 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:23:10.466+0000 7f991b377700 1 -- 172.21.15.52:0/1023919947 >> 172.21.15.52:0/1023919947 conn(0x7f99140fede0 msgr2=0x7f9914109690 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:23:10.461 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:23:10.467+0000 7f991b377700 1 -- 172.21.15.52:0/1023919947 shutdown_connections 2024-05-27T07:23:10.461 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:23:10.467+0000 7f991b377700 1 -- 172.21.15.52:0/1023919947 wait complete. 2024-05-27T07:23:11.190 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:23:10 smithi105 ceph-mon[35977]: pgmap v338: 129 pgs: 129 active+clean; 612 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 9.9 KiB/s rd, 4.5 KiB/s wr, 17 op/s 2024-05-27T07:23:11.196 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:23:10 smithi052 ceph-mon[31955]: pgmap v338: 129 pgs: 129 active+clean; 612 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 9.9 KiB/s rd, 4.5 KiB/s wr, 17 op/s 2024-05-27T07:23:11.230 DEBUG:teuthology.orchestra.run.smithi052:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:38f9d900c83c63799fbdbe61acc9a11b0d3554a6 shell --fsid 73df2636-1bf8-11ef-bc9b-c7b262605968 -- ceph fs subvolume getpath cephfs --group_name qa sv_1 2024-05-27T07:23:11.352 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:23:10 smithi173 ceph-mon[37000]: pgmap v338: 129 pgs: 129 active+clean; 612 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 9.9 KiB/s rd, 4.5 KiB/s wr, 17 op/s 2024-05-27T07:23:11.486 INFO:teuthology.orchestra.run.smithi052.stderr:Inferring config /var/lib/ceph/73df2636-1bf8-11ef-bc9b-c7b262605968/mon.a/config 2024-05-27T07:23:12.190 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:23:11 smithi105 ceph-mon[35977]: from='client.14829 -' entity='client.admin' cmd=[{"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "sv_1", "group_name": "qa", "target": ["mon-mgr", ""]}]: dispatch 2024-05-27T07:23:12.229 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:23:11 smithi052 ceph-mon[31955]: from='client.14829 -' entity='client.admin' cmd=[{"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "sv_1", "group_name": "qa", "target": ["mon-mgr", ""]}]: dispatch 2024-05-27T07:23:12.253 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:23:11 smithi173 ceph-mon[37000]: from='client.14829 -' entity='client.admin' cmd=[{"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "sv_1", "group_name": "qa", "target": ["mon-mgr", ""]}]: dispatch 2024-05-27T07:23:13.039 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:23:12 smithi052 ceph-mon[31955]: pgmap v339: 129 pgs: 129 active+clean; 612 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 9.0 KiB/s rd, 3.7 KiB/s wr, 14 op/s 2024-05-27T07:23:13.190 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:23:12 smithi105 ceph-mon[35977]: pgmap v339: 129 pgs: 129 active+clean; 612 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 9.0 KiB/s rd, 3.7 KiB/s wr, 14 op/s 2024-05-27T07:23:13.295 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:23:13.305+0000 7f4cf7c4e700 1 -- 172.21.15.52:0/3094724983 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f4cf01034b0 msgr2=0x7f4cf01038c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:23:13.296 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:23:13.305+0000 7f4cf7c4e700 1 --2- 172.21.15.52:0/3094724983 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f4cf01034b0 0x7f4cf01038c0 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7f4cec00bb50 tx=0x7f4cec00be60 comp rx=0 tx=0).stop 2024-05-27T07:23:13.296 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:23:13.305+0000 7f4cf7c4e700 1 -- 172.21.15.52:0/3094724983 shutdown_connections 2024-05-27T07:23:13.296 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:23:13.305+0000 7f4cf7c4e700 1 --2- 172.21.15.52:0/3094724983 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f4cf0108730 0x7f4cf0110090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:23:13.296 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:23:13.305+0000 7f4cf7c4e700 1 --2- 172.21.15.52:0/3094724983 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f4cf01034b0 0x7f4cf01038c0 unknown :-1 s=CLOSED pgs=235 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:23:13.296 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:23:13.305+0000 7f4cf7c4e700 1 --2- 172.21.15.52:0/3094724983 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f4cf0107de0 0x7f4cf01081f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:23:13.296 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:23:13.305+0000 7f4cf7c4e700 1 -- 172.21.15.52:0/3094724983 >> 172.21.15.52:0/3094724983 conn(0x7f4cf00fede0 msgr2=0x7f4cf0101200 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:23:13.296 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:23:13.305+0000 7f4cf7c4e700 1 -- 172.21.15.52:0/3094724983 shutdown_connections 2024-05-27T07:23:13.296 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:23:13.305+0000 7f4cf7c4e700 1 -- 172.21.15.52:0/3094724983 wait complete. 2024-05-27T07:23:13.298 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:23:13.306+0000 7f4cf7c4e700 1 Processor -- start 2024-05-27T07:23:13.298 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:23:13.306+0000 7f4cf7c4e700 1 -- start start 2024-05-27T07:23:13.298 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:23:13.306+0000 7f4cf7c4e700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f4cf01034b0 0x7f4cf019d030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:23:13.298 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:23:13.306+0000 7f4cf7c4e700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f4cf0107de0 0x7f4cf019d5d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:23:13.298 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:23:13.306+0000 7f4cf7c4e700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f4cf0108730 0x7f4cf01a1860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:23:13.298 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:23:13.306+0000 7f4cf7c4e700 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x7f4cf0112310 con 0x7f4cf01034b0 2024-05-27T07:23:13.298 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:23:13.306+0000 7f4cf7c4e700 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f4cf0112190 con 0x7f4cf0108730 2024-05-27T07:23:13.298 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:23:13.306+0000 7f4cf7c4e700 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f4cf0112490 con 0x7f4cf0107de0 2024-05-27T07:23:13.298 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:23:13.307+0000 7f4cf59ea700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f4cf01034b0 0x7f4cf019d030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:23:13.298 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:23:13.307+0000 7f4cf59ea700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f4cf01034b0 0x7f4cf019d030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.52:34776/0 (socket says 172.21.15.52:34776) 2024-05-27T07:23:13.298 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:23:13.307+0000 7f4cf59ea700 1 -- 172.21.15.52:0/1230669589 learned_addr learned my addr 172.21.15.52:0/1230669589 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-05-27T07:23:13.299 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:23:13.307+0000 7f4cf51e9700 1 --2- 172.21.15.52:0/1230669589 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f4cf0107de0 0x7f4cf019d5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:23:13.299 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:23:13.307+0000 7f4cf59ea700 1 -- 172.21.15.52:0/1230669589 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f4cf0107de0 msgr2=0x7f4cf019d5d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:23:13.299 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:23:13.307+0000 7f4cf61eb700 1 --2- 172.21.15.52:0/1230669589 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f4cf0108730 0x7f4cf01a1860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:23:13.299 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:23:13.307+0000 7f4cf59ea700 1 --2- 172.21.15.52:0/1230669589 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f4cf0107de0 0x7f4cf019d5d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:23:13.299 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:23:13.307+0000 7f4cf59ea700 1 -- 172.21.15.52:0/1230669589 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f4cf0108730 msgr2=0x7f4cf01a1860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:23:13.299 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:23:13.307+0000 7f4cf59ea700 1 --2- 172.21.15.52:0/1230669589 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f4cf0108730 0x7f4cf01a1860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:23:13.299 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:23:13.307+0000 7f4cf59ea700 1 -- 172.21.15.52:0/1230669589 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4cf01a1f00 con 0x7f4cf01034b0 2024-05-27T07:23:13.299 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:23:13.307+0000 7f4cf59ea700 1 --2- 172.21.15.52:0/1230669589 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f4cf01034b0 0x7f4cf019d030 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7f4cec00d370 tx=0x7f4cec004660 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:23:13.299 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:23:13.307+0000 7f4ce6ffd700 1 -- 172.21.15.52:0/1230669589 <== mon.0 v2:172.21.15.52:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4cec023020 con 0x7f4cf01034b0 2024-05-27T07:23:13.299 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:23:13.307+0000 7f4ce6ffd700 1 -- 172.21.15.52:0/1230669589 <== mon.0 v2:172.21.15.52:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f4cec01f470 con 0x7f4cf01034b0 2024-05-27T07:23:13.299 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:23:13.307+0000 7f4cf7c4e700 1 -- 172.21.15.52:0/1230669589 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4cf01a2190 con 0x7f4cf01034b0 2024-05-27T07:23:13.299 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:23:13.307+0000 7f4ce6ffd700 1 -- 172.21.15.52:0/1230669589 <== mon.0 v2:172.21.15.52:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4cec014690 con 0x7f4cf01034b0 2024-05-27T07:23:13.299 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:23:13.307+0000 7f4cf7c4e700 1 -- 172.21.15.52:0/1230669589 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4cf01a2670 con 0x7f4cf01034b0 2024-05-27T07:23:13.300 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:23:13.309+0000 7f4cf7c4e700 1 -- 172.21.15.52:0/1230669589 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4cf00512b0 con 0x7f4cf01034b0 2024-05-27T07:23:13.300 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:23:13.310+0000 7f4ce6ffd700 1 -- 172.21.15.52:0/1230669589 <== mon.0 v2:172.21.15.52:3300/0 4 ==== mgrmap(e 17) ==== 149703+0+0 (secure 0 0 0) 0x7f4cec026020 con 0x7f4cf01034b0 2024-05-27T07:23:13.301 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:23:13.311+0000 7f4ce6ffd700 1 --2- 172.21.15.52:0/1230669589 >> [v2:172.21.15.52:6800/3513075805,v1:172.21.15.52:6801/3513075805] conn(0x7f4cdc0b2910 0x7f4cdc0b4e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:23:13.301 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:23:13.311+0000 7f4cf51e9700 1 --2- 172.21.15.52:0/1230669589 >> [v2:172.21.15.52:6800/3513075805,v1:172.21.15.52:6801/3513075805] conn(0x7f4cdc0b2910 0x7f4cdc0b4e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:23:13.301 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:23:13.311+0000 7f4ce6ffd700 1 -- 172.21.15.52:0/1230669589 <== mon.0 v2:172.21.15.52:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 9092+0+0 (secure 0 0 0) 0x7f4cec011020 con 0x7f4cf01034b0 2024-05-27T07:23:13.301 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:23:13.312+0000 7f4cf51e9700 1 --2- 172.21.15.52:0/1230669589 >> [v2:172.21.15.52:6800/3513075805,v1:172.21.15.52:6801/3513075805] conn(0x7f4cdc0b2910 0x7f4cdc0b4e20 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7f4cf0107d60 tx=0x7f4ce0009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:23:13.304 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:23:13.315+0000 7f4ce6ffd700 1 -- 172.21.15.52:0/1230669589 <== mon.0 v2:172.21.15.52:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191963 (secure 0 0 0) 0x7f4cec0ab140 con 0x7f4cf01034b0 2024-05-27T07:23:13.351 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:23:12 smithi173 ceph-mon[37000]: pgmap v339: 129 pgs: 129 active+clean; 612 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 9.0 KiB/s rd, 3.7 KiB/s wr, 14 op/s 2024-05-27T07:23:13.552 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:23:13.562+0000 7f4cf7c4e700 1 -- 172.21.15.52:0/1230669589 --> [v2:172.21.15.52:6800/3513075805,v1:172.21.15.52:6801/3513075805] -- mgr_command(tid 0: {"prefix": "fs subvolume getpath", "vol_name": "cephfs", "group_name": "qa", "sub_name": "sv_1", "target": ["mon-mgr", ""]}) -- 0x7f4cf0063bf0 con 0x7f4cdc0b2910 2024-05-27T07:23:13.557 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:23:13.567+0000 7f4ce6ffd700 1 -- 172.21.15.52:0/1230669589 <== mgr.14152 v2:172.21.15.52:6800/3513075805 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+54 (secure 0 0 0) 0x7f4cf0063bf0 con 0x7f4cdc0b2910 2024-05-27T07:23:13.557 INFO:teuthology.orchestra.run.smithi052.stdout:/volumes/qa/sv_1/95e9ac73-825f-48e3-998b-133d94d96559 2024-05-27T07:23:13.560 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:23:13.570+0000 7f4cf7c4e700 1 -- 172.21.15.52:0/1230669589 >> [v2:172.21.15.52:6800/3513075805,v1:172.21.15.52:6801/3513075805] conn(0x7f4cdc0b2910 msgr2=0x7f4cdc0b4e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:23:13.560 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:23:13.570+0000 7f4cf7c4e700 1 --2- 172.21.15.52:0/1230669589 >> [v2:172.21.15.52:6800/3513075805,v1:172.21.15.52:6801/3513075805] conn(0x7f4cdc0b2910 0x7f4cdc0b4e20 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7f4cf0107d60 tx=0x7f4ce0009260 comp rx=0 tx=0).stop 2024-05-27T07:23:13.560 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:23:13.570+0000 7f4cf7c4e700 1 -- 172.21.15.52:0/1230669589 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f4cf01034b0 msgr2=0x7f4cf019d030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:23:13.560 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:23:13.570+0000 7f4cf7c4e700 1 --2- 172.21.15.52:0/1230669589 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f4cf01034b0 0x7f4cf019d030 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7f4cec00d370 tx=0x7f4cec004660 comp rx=0 tx=0).stop 2024-05-27T07:23:13.560 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:23:13.570+0000 7f4cf7c4e700 1 -- 172.21.15.52:0/1230669589 shutdown_connections 2024-05-27T07:23:13.560 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:23:13.570+0000 7f4cf7c4e700 1 --2- 172.21.15.52:0/1230669589 >> [v2:172.21.15.52:6800/3513075805,v1:172.21.15.52:6801/3513075805] conn(0x7f4cdc0b2910 0x7f4cdc0b4e20 unknown :-1 s=CLOSED pgs=167 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:23:13.560 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:23:13.570+0000 7f4cf7c4e700 1 --2- 172.21.15.52:0/1230669589 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f4cf0108730 0x7f4cf01a1860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:23:13.560 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:23:13.570+0000 7f4cf7c4e700 1 --2- 172.21.15.52:0/1230669589 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f4cf01034b0 0x7f4cf019d030 unknown :-1 s=CLOSED pgs=236 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:23:13.560 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:23:13.570+0000 7f4cf7c4e700 1 --2- 172.21.15.52:0/1230669589 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f4cf0107de0 0x7f4cf019d5d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:23:13.560 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:23:13.570+0000 7f4cf7c4e700 1 -- 172.21.15.52:0/1230669589 >> 172.21.15.52:0/1230669589 conn(0x7f4cf00fede0 msgr2=0x7f4cf0101200 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:23:13.560 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:23:13.570+0000 7f4cf7c4e700 1 -- 172.21.15.52:0/1230669589 shutdown_connections 2024-05-27T07:23:13.561 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:23:13.570+0000 7f4cf7c4e700 1 -- 172.21.15.52:0/1230669589 wait complete. 2024-05-27T07:23:14.203 DEBUG:teuthology.orchestra.run.smithi052:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:38f9d900c83c63799fbdbe61acc9a11b0d3554a6 shell --fsid 73df2636-1bf8-11ef-bc9b-c7b262605968 -- ceph fs dump --format=json 2024-05-27T07:23:14.457 INFO:teuthology.orchestra.run.smithi052.stderr:Inferring config /var/lib/ceph/73df2636-1bf8-11ef-bc9b-c7b262605968/mon.a/config 2024-05-27T07:23:15.190 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:23:14 smithi105 ceph-mon[35977]: from='client.14835 -' entity='client.admin' cmd=[{"prefix": "fs subvolume getpath", "vol_name": "cephfs", "group_name": "qa", "sub_name": "sv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-05-27T07:23:15.190 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:23:14 smithi105 ceph-mon[35977]: pgmap v340: 129 pgs: 129 active+clean; 612 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 6.8 KiB/s rd, 3.0 KiB/s wr, 11 op/s 2024-05-27T07:23:15.229 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:23:14 smithi052 ceph-mon[31955]: from='client.14835 -' entity='client.admin' cmd=[{"prefix": "fs subvolume getpath", "vol_name": "cephfs", "group_name": "qa", "sub_name": "sv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-05-27T07:23:15.229 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:23:14 smithi052 ceph-mon[31955]: pgmap v340: 129 pgs: 129 active+clean; 612 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 6.8 KiB/s rd, 3.0 KiB/s wr, 11 op/s 2024-05-27T07:23:15.352 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:23:14 smithi173 ceph-mon[37000]: from='client.14835 -' entity='client.admin' cmd=[{"prefix": "fs subvolume getpath", "vol_name": "cephfs", "group_name": "qa", "sub_name": "sv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-05-27T07:23:15.352 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:23:14 smithi173 ceph-mon[37000]: pgmap v340: 129 pgs: 129 active+clean; 612 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 6.8 KiB/s rd, 3.0 KiB/s wr, 11 op/s 2024-05-27T07:23:16.261 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:23:16.270+0000 7f4562a4c700 1 -- 172.21.15.52:0/3288498754 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f455c1045c0 msgr2=0x7f455c1049d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:23:16.261 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:23:16.270+0000 7f4562a4c700 1 --2- 172.21.15.52:0/3288498754 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f455c1045c0 0x7f455c1049d0 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7f454c0099c0 tx=0x7f454c009cd0 comp rx=0 tx=0).stop 2024-05-27T07:23:16.261 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:23:16.270+0000 7f4562a4c700 1 -- 172.21.15.52:0/3288498754 shutdown_connections 2024-05-27T07:23:16.261 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:23:16.270+0000 7f4562a4c700 1 --2- 172.21.15.52:0/3288498754 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f455c109840 0x7f455c1111a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:23:16.261 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:23:16.270+0000 7f4562a4c700 1 --2- 172.21.15.52:0/3288498754 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f455c1045c0 0x7f455c1049d0 unknown :-1 s=CLOSED pgs=237 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:23:16.261 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:23:16.270+0000 7f4562a4c700 1 --2- 172.21.15.52:0/3288498754 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f455c108ef0 0x7f455c109300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:23:16.261 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:23:16.270+0000 7f4562a4c700 1 -- 172.21.15.52:0/3288498754 >> 172.21.15.52:0/3288498754 conn(0x7f455c0ffe90 msgr2=0x7f455c1022e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:23:16.261 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:23:16.270+0000 7f4562a4c700 1 -- 172.21.15.52:0/3288498754 shutdown_connections 2024-05-27T07:23:16.261 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:23:16.270+0000 7f4562a4c700 1 -- 172.21.15.52:0/3288498754 wait complete. 2024-05-27T07:23:16.263 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:23:16.271+0000 7f4562a4c700 1 Processor -- start 2024-05-27T07:23:16.263 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:23:16.271+0000 7f4562a4c700 1 -- start start 2024-05-27T07:23:16.263 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:23:16.271+0000 7f4562a4c700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f455c1045c0 0x7f455c19ada0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:23:16.263 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:23:16.272+0000 7f4562a4c700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f455c108ef0 0x7f455c19b340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:23:16.263 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:23:16.272+0000 7f4562a4c700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f455c109840 0x7f455c19f5d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:23:16.264 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:23:16.272+0000 7f4562a4c700 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x7f455c114490 con 0x7f455c108ef0 2024-05-27T07:23:16.264 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:23:16.272+0000 7f4562a4c700 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f455c114310 con 0x7f455c109840 2024-05-27T07:23:16.264 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:23:16.272+0000 7f4562a4c700 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f455c114610 con 0x7f455c1045c0 2024-05-27T07:23:16.264 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:23:16.272+0000 7f4553fff700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f455c108ef0 0x7f455c19b340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:23:16.264 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:23:16.272+0000 7f4553fff700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f455c108ef0 0x7f455c19b340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.52:34790/0 (socket says 172.21.15.52:34790) 2024-05-27T07:23:16.264 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:23:16.272+0000 7f4553fff700 1 -- 172.21.15.52:0/1141847180 learned_addr learned my addr 172.21.15.52:0/1141847180 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-05-27T07:23:16.264 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:23:16.272+0000 7f4560fe9700 1 --2- 172.21.15.52:0/1141847180 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f455c109840 0x7f455c19f5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:23:16.264 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:23:16.272+0000 7f455bfff700 1 --2- 172.21.15.52:0/1141847180 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f455c1045c0 0x7f455c19ada0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:23:16.264 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:23:16.272+0000 7f4553fff700 1 -- 172.21.15.52:0/1141847180 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f455c1045c0 msgr2=0x7f455c19ada0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:23:16.264 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:23:16.272+0000 7f4553fff700 1 --2- 172.21.15.52:0/1141847180 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f455c1045c0 0x7f455c19ada0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:23:16.264 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:23:16.272+0000 7f4553fff700 1 -- 172.21.15.52:0/1141847180 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f455c109840 msgr2=0x7f455c19f5d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:23:16.264 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:23:16.272+0000 7f4553fff700 1 --2- 172.21.15.52:0/1141847180 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f455c109840 0x7f455c19f5d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:23:16.264 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:23:16.272+0000 7f4553fff700 1 -- 172.21.15.52:0/1141847180 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f455c19fc70 con 0x7f455c108ef0 2024-05-27T07:23:16.265 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:23:16.272+0000 7f455bfff700 1 --2- 172.21.15.52:0/1141847180 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f455c1045c0 0x7f455c19ada0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-27T07:23:16.265 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:23:16.272+0000 7f4553fff700 1 --2- 172.21.15.52:0/1141847180 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f455c108ef0 0x7f455c19b340 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7f454400e9f0 tx=0x7f454400ed00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:23:16.265 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:23:16.272+0000 7f4559ffb700 1 -- 172.21.15.52:0/1141847180 <== mon.0 v2:172.21.15.52:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f454400cbd0 con 0x7f455c108ef0 2024-05-27T07:23:16.265 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:23:16.272+0000 7f4559ffb700 1 -- 172.21.15.52:0/1141847180 <== mon.0 v2:172.21.15.52:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f454400cd70 con 0x7f455c108ef0 2024-05-27T07:23:16.265 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:23:16.272+0000 7f4562a4c700 1 -- 172.21.15.52:0/1141847180 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f455c19ff60 con 0x7f455c108ef0 2024-05-27T07:23:16.265 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:23:16.272+0000 7f4559ffb700 1 -- 172.21.15.52:0/1141847180 <== mon.0 v2:172.21.15.52:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4544005650 con 0x7f455c108ef0 2024-05-27T07:23:16.265 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:23:16.273+0000 7f4562a4c700 1 -- 172.21.15.52:0/1141847180 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f455c1a7880 con 0x7f455c108ef0 2024-05-27T07:23:16.265 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:23:16.273+0000 7f4562a4c700 1 -- 172.21.15.52:0/1141847180 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f455c0512b0 con 0x7f455c108ef0 2024-05-27T07:23:16.266 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:23:16.275+0000 7f4559ffb700 1 -- 172.21.15.52:0/1141847180 <== mon.0 v2:172.21.15.52:3300/0 4 ==== mgrmap(e 17) ==== 149703+0+0 (secure 0 0 0) 0x7f4544004750 con 0x7f455c108ef0 2024-05-27T07:23:16.266 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:23:16.276+0000 7f4559ffb700 1 --2- 172.21.15.52:0/1141847180 >> [v2:172.21.15.52:6800/3513075805,v1:172.21.15.52:6801/3513075805] conn(0x7f45480b2910 0x7f45480b4e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:23:16.266 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:23:16.276+0000 7f455bfff700 1 --2- 172.21.15.52:0/1141847180 >> [v2:172.21.15.52:6800/3513075805,v1:172.21.15.52:6801/3513075805] conn(0x7f45480b2910 0x7f45480b4e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:23:16.266 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:23:16.276+0000 7f4559ffb700 1 -- 172.21.15.52:0/1141847180 <== mon.0 v2:172.21.15.52:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 9092+0+0 (secure 0 0 0) 0x7f45440e2c00 con 0x7f455c108ef0 2024-05-27T07:23:16.266 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:23:16.276+0000 7f455bfff700 1 --2- 172.21.15.52:0/1141847180 >> [v2:172.21.15.52:6800/3513075805,v1:172.21.15.52:6801/3513075805] conn(0x7f45480b2910 0x7f45480b4e20 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7f454c003760 tx=0x7f454c01a040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:23:16.269 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:23:16.279+0000 7f4559ffb700 1 -- 172.21.15.52:0/1141847180 <== mon.0 v2:172.21.15.52:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191963 (secure 0 0 0) 0x7f4544054df0 con 0x7f455c108ef0 2024-05-27T07:23:16.533 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:23:16.543+0000 7f4562a4c700 1 -- 172.21.15.52:0/1141847180 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f455c19c100 con 0x7f455c108ef0 2024-05-27T07:23:16.534 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:23:16.544+0000 7f4559ffb700 1 -- 172.21.15.52:0/1141847180 <== mon.0 v2:172.21.15.52:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 29 v29) ==== 94+0+11938 (secure 0 0 0) 0x7f45440c2ec0 con 0x7f455c108ef0 2024-05-27T07:23:16.534 INFO:teuthology.orchestra.run.smithi052.stdout: 2024-05-27T07:23:16.534 INFO:teuthology.orchestra.run.smithi052.stdout:{"epoch":29,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14556,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.52:6835/2584733778","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.52:6834","nonce":2584733778},{"type":"v1","addr":"172.21.15.52:6835","nonce":2584733778}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14568,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.52:6837/675445303","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.52:6836","nonce":675445303},{"type":"v1","addr":"172.21.15.52:6837","nonce":675445303}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":29,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-05-27T07:21:50.521783+0000","modified":"2024-05-27T07:22:58.822630+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24527,"mds_1":24497,"mds_2":24475,"mds_3":24473,"mds_4":24466},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14619":{"gid":14619,"name":"g","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.52:6839/4258549559","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.52:6838","nonce":4258549559},{"type":"v1","addr":"172.21.15.52:6839","nonce":4258549559}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_14631":{"gid":14631,"name":"a","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.52:6841/2878618188","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.52:6840","nonce":2878618188},{"type":"v1","addr":"172.21.15.52:6841","nonce":2878618188}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24433":{"gid":24433,"name":"l","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.173:6833/3013589220","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6832","nonce":3013589220},{"type":"v1","addr":"172.21.15.173:6833","nonce":3013589220}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24452":{"gid":24452,"name":"k","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.105:6833/2800678412","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6832","nonce":2800678412},{"type":"v1","addr":"172.21.15.105:6833","nonce":2800678412}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24460":{"gid":24460,"name":"f","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.173:6835/1619486437","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6834","nonce":1619486437},{"type":"v1","addr":"172.21.15.173:6835","nonce":1619486437}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24466":{"gid":24466,"name":"c","rank":4,"incarnation":26,"state":"up:active","state_seq":25,"addr":"172.21.15.173:6837/1396363827","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6836","nonce":1396363827},{"type":"v1","addr":"172.21.15.173:6837","nonce":1396363827}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24473":{"gid":24473,"name":"e","rank":3,"incarnation":24,"state":"up:active","state_seq":26,"addr":"172.21.15.105:6835/3298498384","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6834","nonce":3298498384},{"type":"v1","addr":"172.21.15.105:6835","nonce":3298498384}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24475":{"gid":24475,"name":"i","rank":2,"incarnation":22,"state":"up:active","state_seq":23,"addr":"172.21.15.173:6839/564778293","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6838","nonce":564778293},{"type":"v1","addr":"172.21.15.173:6839","nonce":564778293}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24497":{"gid":24497,"name":"h","rank":1,"incarnation":20,"state":"up:active","state_seq":24,"addr":"172.21.15.105:6837/191393149","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6836","nonce":191393149},{"type":"v1","addr":"172.21.15.105:6837","nonce":191393149}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24527":{"gid":24527,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":5,"addr":"172.21.15.105:6839/500348151","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6838","nonce":500348151},{"type":"v1","addr":"172.21.15.105:6839","nonce":500348151}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24527,"qdb_cluster":[24466,24473,24475,24497,24527]},"id":1}]} 2024-05-27T07:23:16.537 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:23:16.547+0000 7f4562a4c700 1 -- 172.21.15.52:0/1141847180 >> [v2:172.21.15.52:6800/3513075805,v1:172.21.15.52:6801/3513075805] conn(0x7f45480b2910 msgr2=0x7f45480b4e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:23:16.537 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:23:16.547+0000 7f4562a4c700 1 --2- 172.21.15.52:0/1141847180 >> [v2:172.21.15.52:6800/3513075805,v1:172.21.15.52:6801/3513075805] conn(0x7f45480b2910 0x7f45480b4e20 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7f454c003760 tx=0x7f454c01a040 comp rx=0 tx=0).stop 2024-05-27T07:23:16.537 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:23:16.547+0000 7f4562a4c700 1 -- 172.21.15.52:0/1141847180 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f455c108ef0 msgr2=0x7f455c19b340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:23:16.537 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:23:16.547+0000 7f4562a4c700 1 --2- 172.21.15.52:0/1141847180 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f455c108ef0 0x7f455c19b340 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7f454400e9f0 tx=0x7f454400ed00 comp rx=0 tx=0).stop 2024-05-27T07:23:16.537 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:23:16.547+0000 7f4562a4c700 1 -- 172.21.15.52:0/1141847180 shutdown_connections 2024-05-27T07:23:16.537 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:23:16.547+0000 7f4562a4c700 1 --2- 172.21.15.52:0/1141847180 >> [v2:172.21.15.52:6800/3513075805,v1:172.21.15.52:6801/3513075805] conn(0x7f45480b2910 0x7f45480b4e20 unknown :-1 s=CLOSED pgs=168 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:23:16.537 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:23:16.547+0000 7f4562a4c700 1 --2- 172.21.15.52:0/1141847180 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f455c109840 0x7f455c19f5d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:23:16.537 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:23:16.547+0000 7f4562a4c700 1 --2- 172.21.15.52:0/1141847180 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f455c1045c0 0x7f455c19ada0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:23:16.538 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:23:16.547+0000 7f4562a4c700 1 --2- 172.21.15.52:0/1141847180 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f455c108ef0 0x7f455c19b340 unknown :-1 s=CLOSED pgs=238 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:23:16.538 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:23:16.547+0000 7f4562a4c700 1 -- 172.21.15.52:0/1141847180 >> 172.21.15.52:0/1141847180 conn(0x7f455c0ffe90 msgr2=0x7f455c108160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:23:16.538 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:23:16.547+0000 7f4562a4c700 1 -- 172.21.15.52:0/1141847180 shutdown_connections 2024-05-27T07:23:16.538 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:23:16.547+0000 7f4562a4c700 1 -- 172.21.15.52:0/1141847180 wait complete. 2024-05-27T07:23:16.538 INFO:teuthology.orchestra.run.smithi052.stderr:dumped fsmap epoch 29 2024-05-27T07:23:17.181 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:23:16 smithi052 ceph-mon[31955]: pgmap v341: 129 pgs: 129 active+clean; 634 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 10 KiB/s rd, 5.3 KiB/s wr, 14 op/s 2024-05-27T07:23:17.181 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:23:16 smithi052 ceph-mon[31955]: from='client.? 172.21.15.52:0/1141847180' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-27T07:23:17.190 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:23:16 smithi105 ceph-mon[35977]: pgmap v341: 129 pgs: 129 active+clean; 634 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 10 KiB/s rd, 5.3 KiB/s wr, 14 op/s 2024-05-27T07:23:17.190 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:23:16 smithi105 ceph-mon[35977]: from='client.? 172.21.15.52:0/1141847180' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-27T07:23:17.212 DEBUG:teuthology.orchestra.run.smithi052:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:38f9d900c83c63799fbdbe61acc9a11b0d3554a6 shell --fsid 73df2636-1bf8-11ef-bc9b-c7b262605968 -- ceph osd dump --format=json 2024-05-27T07:23:17.352 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:23:16 smithi173 ceph-mon[37000]: pgmap v341: 129 pgs: 129 active+clean; 634 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 10 KiB/s rd, 5.3 KiB/s wr, 14 op/s 2024-05-27T07:23:17.352 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:23:16 smithi173 ceph-mon[37000]: from='client.? 172.21.15.52:0/1141847180' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-27T07:23:17.462 INFO:teuthology.orchestra.run.smithi052.stderr:Inferring config /var/lib/ceph/73df2636-1bf8-11ef-bc9b-c7b262605968/mon.a/config 2024-05-27T07:23:19.132 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:23:19.140+0000 7fda6438c700 1 -- 172.21.15.52:0/453527664 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fda5c1034d0 msgr2=0x7fda5c1038e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:23:19.132 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:23:19.140+0000 7fda6438c700 1 --2- 172.21.15.52:0/453527664 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fda5c1034d0 0x7fda5c1038e0 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7fda580099c0 tx=0x7fda58009cd0 comp rx=0 tx=0).stop 2024-05-27T07:23:19.133 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:23:19.140+0000 7fda6438c700 1 -- 172.21.15.52:0/453527664 shutdown_connections 2024-05-27T07:23:19.133 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:23:19.140+0000 7fda6438c700 1 --2- 172.21.15.52:0/453527664 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fda5c108750 0x7fda5c1100b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:23:19.133 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:23:19.140+0000 7fda6438c700 1 --2- 172.21.15.52:0/453527664 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fda5c1034d0 0x7fda5c1038e0 unknown :-1 s=CLOSED pgs=239 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:23:19.133 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:23:19.140+0000 7fda6438c700 1 --2- 172.21.15.52:0/453527664 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fda5c107e00 0x7fda5c108210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:23:19.133 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:23:19.140+0000 7fda6438c700 1 -- 172.21.15.52:0/453527664 >> 172.21.15.52:0/453527664 conn(0x7fda5c0fef20 msgr2=0x7fda5c101340 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:23:19.133 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:23:19.140+0000 7fda6438c700 1 -- 172.21.15.52:0/453527664 shutdown_connections 2024-05-27T07:23:19.133 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:23:19.140+0000 7fda6438c700 1 -- 172.21.15.52:0/453527664 wait complete. 2024-05-27T07:23:19.133 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:23:19.141+0000 7fda6438c700 1 Processor -- start 2024-05-27T07:23:19.133 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:23:19.141+0000 7fda6438c700 1 -- start start 2024-05-27T07:23:19.133 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:23:19.141+0000 7fda6438c700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fda5c1034d0 0x7fda5c19ae40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:23:19.133 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:23:19.141+0000 7fda6438c700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fda5c107e00 0x7fda5c19b3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:23:19.134 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:23:19.141+0000 7fda6438c700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fda5c108750 0x7fda5c19f670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:23:19.134 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:23:19.141+0000 7fda6438c700 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x7fda5c113380 con 0x7fda5c107e00 2024-05-27T07:23:19.134 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:23:19.141+0000 7fda6438c700 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7fda5c113200 con 0x7fda5c108750 2024-05-27T07:23:19.134 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:23:19.141+0000 7fda6438c700 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7fda5c113500 con 0x7fda5c1034d0 2024-05-27T07:23:19.134 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:23:19.141+0000 7fda61927700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fda5c107e00 0x7fda5c19b3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:23:19.134 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:23:19.141+0000 7fda61927700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fda5c107e00 0x7fda5c19b3e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.52:34800/0 (socket says 172.21.15.52:34800) 2024-05-27T07:23:19.134 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:23:19.141+0000 7fda61927700 1 -- 172.21.15.52:0/2512276974 learned_addr learned my addr 172.21.15.52:0/2512276974 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-05-27T07:23:19.134 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:23:19.141+0000 7fda62929700 1 --2- 172.21.15.52:0/2512276974 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fda5c108750 0x7fda5c19f670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:23:19.134 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:23:19.141+0000 7fda62128700 1 --2- 172.21.15.52:0/2512276974 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fda5c1034d0 0x7fda5c19ae40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:23:19.134 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:23:19.142+0000 7fda61927700 1 -- 172.21.15.52:0/2512276974 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fda5c1034d0 msgr2=0x7fda5c19ae40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:23:19.134 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:23:19.142+0000 7fda61927700 1 --2- 172.21.15.52:0/2512276974 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fda5c1034d0 0x7fda5c19ae40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:23:19.134 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:23:19.142+0000 7fda61927700 1 -- 172.21.15.52:0/2512276974 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fda5c108750 msgr2=0x7fda5c19f670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:23:19.135 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:23:19.142+0000 7fda61927700 1 --2- 172.21.15.52:0/2512276974 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fda5c108750 0x7fda5c19f670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:23:19.135 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:23:19.142+0000 7fda61927700 1 -- 172.21.15.52:0/2512276974 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fda5c19fd10 con 0x7fda5c107e00 2024-05-27T07:23:19.135 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:23:19.142+0000 7fda61927700 1 --2- 172.21.15.52:0/2512276974 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fda5c107e00 0x7fda5c19b3e0 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7fda4c00d950 tx=0x7fda4c00dc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:23:19.135 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:23:19.142+0000 7fda537fe700 1 -- 172.21.15.52:0/2512276974 <== mon.0 v2:172.21.15.52:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fda4c016070 con 0x7fda5c107e00 2024-05-27T07:23:19.135 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:23:19.142+0000 7fda537fe700 1 -- 172.21.15.52:0/2512276974 <== mon.0 v2:172.21.15.52:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fda4c004830 con 0x7fda5c107e00 2024-05-27T07:23:19.135 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:23:19.142+0000 7fda537fe700 1 -- 172.21.15.52:0/2512276974 <== mon.0 v2:172.21.15.52:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fda4c005520 con 0x7fda5c107e00 2024-05-27T07:23:19.135 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:23:19.142+0000 7fda6438c700 1 -- 172.21.15.52:0/2512276974 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fda5c1a0000 con 0x7fda5c107e00 2024-05-27T07:23:19.135 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:23:19.142+0000 7fda6438c700 1 -- 172.21.15.52:0/2512276974 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fda5c1a7920 con 0x7fda5c107e00 2024-05-27T07:23:19.136 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:23:19.143+0000 7fda6438c700 1 -- 172.21.15.52:0/2512276974 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fda5c0512b0 con 0x7fda5c107e00 2024-05-27T07:23:19.136 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:23:19.145+0000 7fda537fe700 1 -- 172.21.15.52:0/2512276974 <== mon.0 v2:172.21.15.52:3300/0 4 ==== mgrmap(e 17) ==== 149703+0+0 (secure 0 0 0) 0x7fda4c004220 con 0x7fda5c107e00 2024-05-27T07:23:19.136 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:23:19.145+0000 7fda537fe700 1 --2- 172.21.15.52:0/2512276974 >> [v2:172.21.15.52:6800/3513075805,v1:172.21.15.52:6801/3513075805] conn(0x7fda480b2990 0x7fda480b4ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:23:19.136 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:23:19.146+0000 7fda62128700 1 --2- 172.21.15.52:0/2512276974 >> [v2:172.21.15.52:6800/3513075805,v1:172.21.15.52:6801/3513075805] conn(0x7fda480b2990 0x7fda480b4ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:23:19.136 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:23:19.146+0000 7fda537fe700 1 -- 172.21.15.52:0/2512276974 <== mon.0 v2:172.21.15.52:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 9092+0+0 (secure 0 0 0) 0x7fda4c0e2890 con 0x7fda5c107e00 2024-05-27T07:23:19.137 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:23:19.146+0000 7fda62128700 1 --2- 172.21.15.52:0/2512276974 >> [v2:172.21.15.52:6800/3513075805,v1:172.21.15.52:6801/3513075805] conn(0x7fda480b2990 0x7fda480b4ea0 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7fda58009990 tx=0x7fda58000bc0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:23:19.139 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:23:19.149+0000 7fda537fe700 1 -- 172.21.15.52:0/2512276974 <== mon.0 v2:172.21.15.52:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191963 (secure 0 0 0) 0x7fda4c01f020 con 0x7fda5c107e00 2024-05-27T07:23:19.189 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:23:18 smithi105 ceph-mon[35977]: pgmap v342: 129 pgs: 129 active+clean; 634 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 8.1 KiB/s rd, 3.0 KiB/s wr, 11 op/s 2024-05-27T07:23:19.229 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:23:18 smithi052 ceph-mon[31955]: pgmap v342: 129 pgs: 129 active+clean; 634 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 8.1 KiB/s rd, 3.0 KiB/s wr, 11 op/s 2024-05-27T07:23:19.352 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:23:18 smithi173 ceph-mon[37000]: pgmap v342: 129 pgs: 129 active+clean; 634 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 8.1 KiB/s rd, 3.0 KiB/s wr, 11 op/s 2024-05-27T07:23:19.352 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:23:19.362+0000 7fda6438c700 1 -- 172.21.15.52:0/2512276974 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fda5c02e5e0 con 0x7fda5c107e00 2024-05-27T07:23:19.354 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:23:19.364+0000 7fda537fe700 1 -- 172.21.15.52:0/2512276974 <== mon.0 v2:172.21.15.52:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v88) ==== 74+0+24654 (secure 0 0 0) 0x7fda4c0aa0f0 con 0x7fda5c107e00 2024-05-27T07:23:19.355 INFO:teuthology.orchestra.run.smithi052.stdout: 2024-05-27T07:23:19.355 INFO:teuthology.orchestra.run.smithi052.stdout:{"epoch":88,"fsid":"73df2636-1bf8-11ef-bc9b-c7b262605968","created":"2024-05-27T07:13:34.738032+0000","modified":"2024-05-27T07:22:30.789020+0000","last_up_change":"2024-05-27T07:20:45.777597+0000","last_in_change":"2024-05-27T07:20:25.314823+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-05-27T07:16:57.644034+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-05-27T07:21:43.917748+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"79","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-05-27T07:21:47.341876+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"79","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-05-27T07:21:57.707611+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"86","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"347b8a89-9dbe-4848-83b0-fd0e9374f33d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":82,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.52:6802","nonce":1626508899},{"type":"v1","addr":"172.21.15.52:6803","nonce":1626508899}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.52:6804","nonce":1626508899},{"type":"v1","addr":"172.21.15.52:6805","nonce":1626508899}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.52:6808","nonce":1626508899},{"type":"v1","addr":"172.21.15.52:6809","nonce":1626508899}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.52:6806","nonce":1626508899},{"type":"v1","addr":"172.21.15.52:6807","nonce":1626508899}]},"public_addr":"172.21.15.52:6803/1626508899","cluster_addr":"172.21.15.52:6805/1626508899","heartbeat_back_addr":"172.21.15.52:6809/1626508899","heartbeat_front_addr":"172.21.15.52:6807/1626508899","state":["exists","up"]},{"osd":1,"uuid":"7a3623d4-e087-40cb-9336-d4a5c52203d4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":82,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6800","nonce":2900093195},{"type":"v1","addr":"172.21.15.105:6801","nonce":2900093195}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6802","nonce":2900093195},{"type":"v1","addr":"172.21.15.105:6803","nonce":2900093195}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6806","nonce":2900093195},{"type":"v1","addr":"172.21.15.105:6807","nonce":2900093195}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6804","nonce":2900093195},{"type":"v1","addr":"172.21.15.105:6805","nonce":2900093195}]},"public_addr":"172.21.15.105:6801/2900093195","cluster_addr":"172.21.15.105:6803/2900093195","heartbeat_back_addr":"172.21.15.105:6807/2900093195","heartbeat_front_addr":"172.21.15.105:6805/2900093195","state":["exists","up"]},{"osd":2,"uuid":"41a9df5b-882a-47da-bb26-7f4d26ccf6c1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6800","nonce":2251189314},{"type":"v1","addr":"172.21.15.173:6801","nonce":2251189314}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6802","nonce":2251189314},{"type":"v1","addr":"172.21.15.173:6803","nonce":2251189314}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6806","nonce":2251189314},{"type":"v1","addr":"172.21.15.173:6807","nonce":2251189314}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6804","nonce":2251189314},{"type":"v1","addr":"172.21.15.173:6805","nonce":2251189314}]},"public_addr":"172.21.15.173:6801/2251189314","cluster_addr":"172.21.15.173:6803/2251189314","heartbeat_back_addr":"172.21.15.173:6807/2251189314","heartbeat_front_addr":"172.21.15.173:6805/2251189314","state":["exists","up"]},{"osd":3,"uuid":"f1245439-42a5-46c5-a998-e0a78f9df3fb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":26,"up_thru":82,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.52:6810","nonce":57920794},{"type":"v1","addr":"172.21.15.52:6811","nonce":57920794}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.52:6812","nonce":57920794},{"type":"v1","addr":"172.21.15.52:6813","nonce":57920794}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.52:6816","nonce":57920794},{"type":"v1","addr":"172.21.15.52:6817","nonce":57920794}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.52:6814","nonce":57920794},{"type":"v1","addr":"172.21.15.52:6815","nonce":57920794}]},"public_addr":"172.21.15.52:6811/57920794","cluster_addr":"172.21.15.52:6813/57920794","heartbeat_back_addr":"172.21.15.52:6817/57920794","heartbeat_front_addr":"172.21.15.52:6815/57920794","state":["exists","up"]},{"osd":4,"uuid":"2592d1cc-1726-4359-83d8-a2e6314b54ec","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":32,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6808","nonce":3441134818},{"type":"v1","addr":"172.21.15.105:6809","nonce":3441134818}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6810","nonce":3441134818},{"type":"v1","addr":"172.21.15.105:6811","nonce":3441134818}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6814","nonce":3441134818},{"type":"v1","addr":"172.21.15.105:6815","nonce":3441134818}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6812","nonce":3441134818},{"type":"v1","addr":"172.21.15.105:6813","nonce":3441134818}]},"public_addr":"172.21.15.105:6809/3441134818","cluster_addr":"172.21.15.105:6811/3441134818","heartbeat_back_addr":"172.21.15.105:6815/3441134818","heartbeat_front_addr":"172.21.15.105:6813/3441134818","state":["exists","up"]},{"osd":5,"uuid":"27e98ee2-4a92-4972-b2a3-959d0ed90981","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":37,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6808","nonce":2989239729},{"type":"v1","addr":"172.21.15.173:6809","nonce":2989239729}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6810","nonce":2989239729},{"type":"v1","addr":"172.21.15.173:6811","nonce":2989239729}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6814","nonce":2989239729},{"type":"v1","addr":"172.21.15.173:6815","nonce":2989239729}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6812","nonce":2989239729},{"type":"v1","addr":"172.21.15.173:6813","nonce":2989239729}]},"public_addr":"172.21.15.173:6809/2989239729","cluster_addr":"172.21.15.173:6811/2989239729","heartbeat_back_addr":"172.21.15.173:6815/2989239729","heartbeat_front_addr":"172.21.15.173:6813/2989239729","state":["exists","up"]},{"osd":6,"uuid":"2b0ec0cb-f438-4b2c-8642-0778ed374e9a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":43,"up_thru":82,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.52:6818","nonce":4244423065},{"type":"v1","addr":"172.21.15.52:6819","nonce":4244423065}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.52:6820","nonce":4244423065},{"type":"v1","addr":"172.21.15.52:6821","nonce":4244423065}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.52:6824","nonce":4244423065},{"type":"v1","addr":"172.21.15.52:6825","nonce":4244423065}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.52:6822","nonce":4244423065},{"type":"v1","addr":"172.21.15.52:6823","nonce":4244423065}]},"public_addr":"172.21.15.52:6819/4244423065","cluster_addr":"172.21.15.52:6821/4244423065","heartbeat_back_addr":"172.21.15.52:6825/4244423065","heartbeat_front_addr":"172.21.15.52:6823/4244423065","state":["exists","up"]},{"osd":7,"uuid":"cb2a7a6b-8015-43ea-865a-be144e45c51a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":48,"up_thru":82,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6816","nonce":2063611552},{"type":"v1","addr":"172.21.15.105:6817","nonce":2063611552}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6818","nonce":2063611552},{"type":"v1","addr":"172.21.15.105:6819","nonce":2063611552}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6822","nonce":2063611552},{"type":"v1","addr":"172.21.15.105:6823","nonce":2063611552}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6820","nonce":2063611552},{"type":"v1","addr":"172.21.15.105:6821","nonce":2063611552}]},"public_addr":"172.21.15.105:6817/2063611552","cluster_addr":"172.21.15.105:6819/2063611552","heartbeat_back_addr":"172.21.15.105:6823/2063611552","heartbeat_front_addr":"172.21.15.105:6821/2063611552","state":["exists","up"]},{"osd":8,"uuid":"c2fc565d-a0b9-4396-a209-bd884cddcfe3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":54,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6816","nonce":4131399852},{"type":"v1","addr":"172.21.15.173:6817","nonce":4131399852}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6818","nonce":4131399852},{"type":"v1","addr":"172.21.15.173:6819","nonce":4131399852}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6822","nonce":4131399852},{"type":"v1","addr":"172.21.15.173:6823","nonce":4131399852}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6820","nonce":4131399852},{"type":"v1","addr":"172.21.15.173:6821","nonce":4131399852}]},"public_addr":"172.21.15.173:6817/4131399852","cluster_addr":"172.21.15.173:6819/4131399852","heartbeat_back_addr":"172.21.15.173:6823/4131399852","heartbeat_front_addr":"172.21.15.173:6821/4131399852","state":["exists","up"]},{"osd":9,"uuid":"78fa7f17-1dad-4262-be40-31fa82e75520","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":60,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.52:6826","nonce":479119778},{"type":"v1","addr":"172.21.15.52:6827","nonce":479119778}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.52:6828","nonce":479119778},{"type":"v1","addr":"172.21.15.52:6829","nonce":479119778}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.52:6832","nonce":479119778},{"type":"v1","addr":"172.21.15.52:6833","nonce":479119778}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.52:6830","nonce":479119778},{"type":"v1","addr":"172.21.15.52:6831","nonce":479119778}]},"public_addr":"172.21.15.52:6827/479119778","cluster_addr":"172.21.15.52:6829/479119778","heartbeat_back_addr":"172.21.15.52:6833/479119778","heartbeat_front_addr":"172.21.15.52:6831/479119778","state":["exists","up"]},{"osd":10,"uuid":"e6f8e753-e15c-45b5-8ee9-d50f384559f9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":65,"up_thru":76,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6824","nonce":4032386628},{"type":"v1","addr":"172.21.15.105:6825","nonce":4032386628}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6826","nonce":4032386628},{"type":"v1","addr":"172.21.15.105:6827","nonce":4032386628}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6830","nonce":4032386628},{"type":"v1","addr":"172.21.15.105:6831","nonce":4032386628}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6828","nonce":4032386628},{"type":"v1","addr":"172.21.15.105:6829","nonce":4032386628}]},"public_addr":"172.21.15.105:6825/4032386628","cluster_addr":"172.21.15.105:6827/4032386628","heartbeat_back_addr":"172.21.15.105:6831/4032386628","heartbeat_front_addr":"172.21.15.105:6829/4032386628","state":["exists","up"]},{"osd":11,"uuid":"164572bf-67ed-4e8e-b32d-d7a2e8da607b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":71,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6824","nonce":1816308401},{"type":"v1","addr":"172.21.15.173:6825","nonce":1816308401}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6826","nonce":1816308401},{"type":"v1","addr":"172.21.15.173:6827","nonce":1816308401}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6830","nonce":1816308401},{"type":"v1","addr":"172.21.15.173:6831","nonce":1816308401}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6828","nonce":1816308401},{"type":"v1","addr":"172.21.15.173:6829","nonce":1816308401}]},"public_addr":"172.21.15.173:6825/1816308401","cluster_addr":"172.21.15.173:6827/1816308401","heartbeat_back_addr":"172.21.15.173:6831/1816308401","heartbeat_front_addr":"172.21.15.173:6829/1816308401","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-27T07:16:06.279220+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-27T07:16:29.668167+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-27T07:16:53.903808+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-27T07:17:18.787372+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-27T07:17:43.480784+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-27T07:18:08.816030+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-27T07:18:34.076148+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-27T07:19:00.283468+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-27T07:19:25.403023+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-27T07:19:52.139173+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-27T07:20:17.530304+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-27T07:20:43.665632+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.f","mappings":[{"from":8,"to":11}]},{"pgid":"2.10","mappings":[{"from":2,"to":4}]},{"pgid":"3.0","mappings":[{"from":5,"to":9}]},{"pgid":"3.1b","mappings":[{"from":4,"to":8}]},{"pgid":"3.2c","mappings":[{"from":4,"to":9}]},{"pgid":"3.3c","mappings":[{"from":4,"to":9}]},{"pgid":"4.e","mappings":[{"from":9,"to":11}]},{"pgid":"4.12","mappings":[{"from":5,"to":10}]},{"pgid":"4.19","mappings":[{"from":9,"to":8}]},{"pgid":"4.1f","mappings":[{"from":9,"to":8}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.52:6801/1631683163":"2024-05-28T07:14:29.544633+0000","172.21.15.52:0/2776144659":"2024-05-28T07:14:29.544633+0000","172.21.15.52:6800/1631683163":"2024-05-28T07:14:29.544633+0000","172.21.15.52:0/3675529097":"2024-05-28T07:14:01.960550+0000","172.21.15.52:0/727808107":"2024-05-28T07:14:01.960550+0000","172.21.15.52:0/3839436833":"2024-05-28T07:14:29.544633+0000","172.21.15.52:6800/587662512":"2024-05-28T07:14:01.960550+0000","172.21.15.52:6801/587662512":"2024-05-28T07:14:01.960550+0000","172.21.15.52:0/853867465":"2024-05-28T07:14:01.960550+0000","172.21.15.52:0/3480918279":"2024-05-28T07:14:29.544633+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-05-27T07:23:19.357 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:23:19.367+0000 7fda6438c700 1 -- 172.21.15.52:0/2512276974 >> [v2:172.21.15.52:6800/3513075805,v1:172.21.15.52:6801/3513075805] conn(0x7fda480b2990 msgr2=0x7fda480b4ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:23:19.357 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:23:19.367+0000 7fda6438c700 1 --2- 172.21.15.52:0/2512276974 >> [v2:172.21.15.52:6800/3513075805,v1:172.21.15.52:6801/3513075805] conn(0x7fda480b2990 0x7fda480b4ea0 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7fda58009990 tx=0x7fda58000bc0 comp rx=0 tx=0).stop 2024-05-27T07:23:19.357 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:23:19.367+0000 7fda6438c700 1 -- 172.21.15.52:0/2512276974 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fda5c107e00 msgr2=0x7fda5c19b3e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:23:19.357 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:23:19.367+0000 7fda6438c700 1 --2- 172.21.15.52:0/2512276974 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fda5c107e00 0x7fda5c19b3e0 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7fda4c00d950 tx=0x7fda4c00dc60 comp rx=0 tx=0).stop 2024-05-27T07:23:19.357 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:23:19.367+0000 7fda6438c700 1 -- 172.21.15.52:0/2512276974 shutdown_connections 2024-05-27T07:23:19.357 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:23:19.367+0000 7fda6438c700 1 --2- 172.21.15.52:0/2512276974 >> [v2:172.21.15.52:6800/3513075805,v1:172.21.15.52:6801/3513075805] conn(0x7fda480b2990 0x7fda480b4ea0 unknown :-1 s=CLOSED pgs=169 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:23:19.357 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:23:19.367+0000 7fda6438c700 1 --2- 172.21.15.52:0/2512276974 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fda5c108750 0x7fda5c19f670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:23:19.357 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:23:19.367+0000 7fda6438c700 1 --2- 172.21.15.52:0/2512276974 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fda5c1034d0 0x7fda5c19ae40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:23:19.357 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:23:19.367+0000 7fda6438c700 1 --2- 172.21.15.52:0/2512276974 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fda5c107e00 0x7fda5c19b3e0 unknown :-1 s=CLOSED pgs=240 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:23:19.357 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:23:19.367+0000 7fda6438c700 1 -- 172.21.15.52:0/2512276974 >> 172.21.15.52:0/2512276974 conn(0x7fda5c0fef20 msgr2=0x7fda5c1063b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:23:19.358 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:23:19.367+0000 7fda6438c700 1 -- 172.21.15.52:0/2512276974 shutdown_connections 2024-05-27T07:23:19.358 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:23:19.367+0000 7fda6438c700 1 -- 172.21.15.52:0/2512276974 wait complete. 2024-05-27T07:23:20.116 INFO:tasks.cephadm.ceph_manager.ceph:waiting for clean 2024-05-27T07:23:20.116 DEBUG:teuthology.orchestra.run.smithi052:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:38f9d900c83c63799fbdbe61acc9a11b0d3554a6 shell --fsid 73df2636-1bf8-11ef-bc9b-c7b262605968 -- ceph pg dump --format=json 2024-05-27T07:23:20.190 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:23:19 smithi105 ceph-mon[35977]: from='client.? 172.21.15.52:0/2512276974' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-05-27T07:23:20.229 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:23:19 smithi052 ceph-mon[31955]: from='client.? 172.21.15.52:0/2512276974' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-05-27T07:23:20.352 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:23:19 smithi173 ceph-mon[37000]: from='client.? 172.21.15.52:0/2512276974' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-05-27T07:23:20.370 INFO:teuthology.orchestra.run.smithi052.stderr:Inferring config /var/lib/ceph/73df2636-1bf8-11ef-bc9b-c7b262605968/mon.a/config 2024-05-27T07:23:21.184 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:23:20 smithi052 ceph-mon[31955]: pgmap v343: 129 pgs: 129 active+clean; 639 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 9.3 KiB/s rd, 3.3 KiB/s wr, 12 op/s 2024-05-27T07:23:21.189 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:23:20 smithi105 ceph-mon[35977]: pgmap v343: 129 pgs: 129 active+clean; 639 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 9.3 KiB/s rd, 3.3 KiB/s wr, 12 op/s 2024-05-27T07:23:21.352 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:23:20 smithi173 ceph-mon[37000]: pgmap v343: 129 pgs: 129 active+clean; 639 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 9.3 KiB/s rd, 3.3 KiB/s wr, 12 op/s 2024-05-27T07:23:22.123 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:23:22.132+0000 7f79a4f3b700 1 -- 172.21.15.52:0/3478233857 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f79a01054f0 msgr2=0x7f79a0105900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:23:22.123 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:23:22.132+0000 7f79a4f3b700 1 --2- 172.21.15.52:0/3478233857 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f79a01054f0 0x7f79a0105900 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7f799000b0a0 tx=0x7f799000b3b0 comp rx=0 tx=0).stop 2024-05-27T07:23:22.123 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:23:22.132+0000 7f79a4f3b700 1 -- 172.21.15.52:0/3478233857 shutdown_connections 2024-05-27T07:23:22.123 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:23:22.132+0000 7f79a4f3b700 1 --2- 172.21.15.52:0/3478233857 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f79a010e920 0x7f79a0110dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:23:22.123 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:23:22.132+0000 7f79a4f3b700 1 --2- 172.21.15.52:0/3478233857 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f79a01054f0 0x7f79a0105900 unknown :-1 s=CLOSED pgs=241 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:23:22.123 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:23:22.132+0000 7f79a4f3b700 1 --2- 172.21.15.52:0/3478233857 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f79a0105e40 0x7f79a010e390 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:23:22.123 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:23:22.132+0000 7f79a4f3b700 1 -- 172.21.15.52:0/3478233857 >> 172.21.15.52:0/3478233857 conn(0x7f79a00fede0 msgr2=0x7f79a0101200 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:23:22.123 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:23:22.133+0000 7f79a4f3b700 1 -- 172.21.15.52:0/3478233857 shutdown_connections 2024-05-27T07:23:22.124 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:23:22.133+0000 7f79a4f3b700 1 -- 172.21.15.52:0/3478233857 wait complete. 2024-05-27T07:23:22.125 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:23:22.134+0000 7f79a4f3b700 1 Processor -- start 2024-05-27T07:23:22.125 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:23:22.134+0000 7f79a4f3b700 1 -- start start 2024-05-27T07:23:22.126 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:23:22.134+0000 7f79a4f3b700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f79a01054f0 0x7f79a01a2110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:23:22.126 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:23:22.134+0000 7f79a4f3b700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f79a0105e40 0x7f79a01a2650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:23:22.126 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:23:22.134+0000 7f79a4f3b700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f79a010e920 0x7f79a01ad190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:23:22.126 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:23:22.134+0000 7f79a4f3b700 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x7f79a0114430 con 0x7f79a010e920 2024-05-27T07:23:22.126 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:23:22.134+0000 7f79a4f3b700 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f79a01142b0 con 0x7f79a0105e40 2024-05-27T07:23:22.126 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:23:22.134+0000 7f79a4f3b700 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f79a01145b0 con 0x7f79a01054f0 2024-05-27T07:23:22.126 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:23:22.134+0000 7f799ed9d700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f79a010e920 0x7f79a01ad190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:23:22.126 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:23:22.134+0000 7f799ed9d700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f79a010e920 0x7f79a01ad190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.52:39722/0 (socket says 172.21.15.52:39722) 2024-05-27T07:23:22.126 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:23:22.134+0000 7f799ed9d700 1 -- 172.21.15.52:0/1930417881 learned_addr learned my addr 172.21.15.52:0/1930417881 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-05-27T07:23:22.126 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:23:22.134+0000 7f799dd9b700 1 --2- 172.21.15.52:0/1930417881 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f79a0105e40 0x7f79a01a2650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:23:22.127 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:23:22.134+0000 7f799e59c700 1 --2- 172.21.15.52:0/1930417881 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f79a01054f0 0x7f79a01a2110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:23:22.127 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:23:22.135+0000 7f799ed9d700 1 -- 172.21.15.52:0/1930417881 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f79a01054f0 msgr2=0x7f79a01a2110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:23:22.127 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:23:22.135+0000 7f799ed9d700 1 --2- 172.21.15.52:0/1930417881 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f79a01054f0 0x7f79a01a2110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:23:22.127 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:23:22.135+0000 7f799ed9d700 1 -- 172.21.15.52:0/1930417881 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f79a0105e40 msgr2=0x7f79a01a2650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:23:22.127 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:23:22.135+0000 7f799ed9d700 1 --2- 172.21.15.52:0/1930417881 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f79a0105e40 0x7f79a01a2650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:23:22.127 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:23:22.135+0000 7f799ed9d700 1 -- 172.21.15.52:0/1930417881 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f79a01ad6d0 con 0x7f79a010e920 2024-05-27T07:23:22.127 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:23:22.135+0000 7f799e59c700 1 --2- 172.21.15.52:0/1930417881 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f79a01054f0 0x7f79a01a2110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-27T07:23:22.127 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:23:22.135+0000 7f799ed9d700 1 --2- 172.21.15.52:0/1930417881 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f79a010e920 0x7f79a01ad190 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7f799800d980 tx=0x7f799800dc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:23:22.127 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:23:22.135+0000 7f798f7fe700 1 -- 172.21.15.52:0/1930417881 <== mon.0 v2:172.21.15.52:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7998017070 con 0x7f79a010e920 2024-05-27T07:23:22.127 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:23:22.135+0000 7f79a4f3b700 1 -- 172.21.15.52:0/1930417881 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f79a01ad8a0 con 0x7f79a010e920 2024-05-27T07:23:22.127 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:23:22.136+0000 7f79a4f3b700 1 -- 172.21.15.52:0/1930417881 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f79a01adcf0 con 0x7f79a010e920 2024-05-27T07:23:22.127 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:23:22.136+0000 7f798f7fe700 1 -- 172.21.15.52:0/1930417881 <== mon.0 v2:172.21.15.52:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f7998005020 con 0x7f79a010e920 2024-05-27T07:23:22.128 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:23:22.136+0000 7f798f7fe700 1 -- 172.21.15.52:0/1930417881 <== mon.0 v2:172.21.15.52:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7998009990 con 0x7f79a010e920 2024-05-27T07:23:22.128 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:23:22.137+0000 7f79a4f3b700 1 -- 172.21.15.52:0/1930417881 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7980005330 con 0x7f79a010e920 2024-05-27T07:23:22.128 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:23:22.139+0000 7f798f7fe700 1 -- 172.21.15.52:0/1930417881 <== mon.0 v2:172.21.15.52:3300/0 4 ==== mgrmap(e 17) ==== 149703+0+0 (secure 0 0 0) 0x7f7998004a30 con 0x7f79a010e920 2024-05-27T07:23:22.129 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:23:22.139+0000 7f798f7fe700 1 --2- 172.21.15.52:0/1930417881 >> [v2:172.21.15.52:6800/3513075805,v1:172.21.15.52:6801/3513075805] conn(0x7f79880b2990 0x7f79880b4ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:23:22.129 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:23:22.140+0000 7f799e59c700 1 --2- 172.21.15.52:0/1930417881 >> [v2:172.21.15.52:6800/3513075805,v1:172.21.15.52:6801/3513075805] conn(0x7f79880b2990 0x7f79880b4ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:23:22.129 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:23:22.140+0000 7f798f7fe700 1 -- 172.21.15.52:0/1930417881 <== mon.0 v2:172.21.15.52:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 9092+0+0 (secure 0 0 0) 0x7f79980e2430 con 0x7f79a010e920 2024-05-27T07:23:22.130 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:23:22.140+0000 7f799e59c700 1 --2- 172.21.15.52:0/1930417881 >> [v2:172.21.15.52:6800/3513075805,v1:172.21.15.52:6801/3513075805] conn(0x7f79880b2990 0x7f79880b4ea0 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7f79900062a0 tx=0x7f7990006230 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:23:22.134 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:23:22.144+0000 7f798f7fe700 1 -- 172.21.15.52:0/1930417881 <== mon.0 v2:172.21.15.52:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191963 (secure 0 0 0) 0x7f7998020020 con 0x7f79a010e920 2024-05-27T07:23:22.350 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:23:22.360+0000 7f79a4f3b700 1 -- 172.21.15.52:0/1930417881 --> [v2:172.21.15.52:6800/3513075805,v1:172.21.15.52:6801/3513075805] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f7980000bc0 con 0x7f79880b2990 2024-05-27T07:23:22.360 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:23:22.370+0000 7f798f7fe700 1 -- 172.21.15.52:0/1930417881 <== mgr.14152 v2:172.21.15.52:6800/3513075805 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+340640 (secure 0 0 0) 0x7f7980000bc0 con 0x7f79880b2990 2024-05-27T07:23:22.365 INFO:teuthology.orchestra.run.smithi052.stdout: 2024-05-27T07:23:22.368 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:23:22.375+0000 7f79a4f3b700 1 -- 172.21.15.52:0/1930417881 >> [v2:172.21.15.52:6800/3513075805,v1:172.21.15.52:6801/3513075805] conn(0x7f79880b2990 msgr2=0x7f79880b4ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:23:22.368 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:23:22.375+0000 7f79a4f3b700 1 --2- 172.21.15.52:0/1930417881 >> [v2:172.21.15.52:6800/3513075805,v1:172.21.15.52:6801/3513075805] conn(0x7f79880b2990 0x7f79880b4ea0 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7f79900062a0 tx=0x7f7990006230 comp rx=0 tx=0).stop 2024-05-27T07:23:22.368 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:23:22.375+0000 7f79a4f3b700 1 -- 172.21.15.52:0/1930417881 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f79a010e920 msgr2=0x7f79a01ad190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:23:22.368 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:23:22.375+0000 7f79a4f3b700 1 --2- 172.21.15.52:0/1930417881 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f79a010e920 0x7f79a01ad190 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7f799800d980 tx=0x7f799800dc90 comp rx=0 tx=0).stop 2024-05-27T07:23:22.369 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:23:22.375+0000 7f79a4f3b700 1 -- 172.21.15.52:0/1930417881 shutdown_connections 2024-05-27T07:23:22.369 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:23:22.375+0000 7f79a4f3b700 1 --2- 172.21.15.52:0/1930417881 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f79a010e920 0x7f79a01ad190 unknown :-1 s=CLOSED pgs=242 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:23:22.369 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:23:22.375+0000 7f79a4f3b700 1 --2- 172.21.15.52:0/1930417881 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f79a01054f0 0x7f79a01a2110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:23:22.369 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:23:22.375+0000 7f79a4f3b700 1 --2- 172.21.15.52:0/1930417881 >> [v2:172.21.15.52:6800/3513075805,v1:172.21.15.52:6801/3513075805] conn(0x7f79880b2990 0x7f79880b4ea0 unknown :-1 s=CLOSED pgs=170 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:23:22.369 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:23:22.375+0000 7f79a4f3b700 1 --2- 172.21.15.52:0/1930417881 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f79a0105e40 0x7f79a01a2650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:23:22.369 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:23:22.375+0000 7f79a4f3b700 1 -- 172.21.15.52:0/1930417881 >> 172.21.15.52:0/1930417881 conn(0x7f79a00fede0 msgr2=0x7f79a0103210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:23:22.369 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:23:22.375+0000 7f79a4f3b700 1 -- 172.21.15.52:0/1930417881 shutdown_connections 2024-05-27T07:23:22.369 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:23:22.375+0000 7f79a4f3b700 1 -- 172.21.15.52:0/1930417881 wait complete. 2024-05-27T07:23:22.369 INFO:teuthology.orchestra.run.smithi052.stderr:dumped all 2024-05-27T07:23:23.190 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:23:22 smithi105 ceph-mon[35977]: pgmap v344: 129 pgs: 129 active+clean; 639 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 7.3 KiB/s rd, 2.9 KiB/s wr, 12 op/s 2024-05-27T07:23:23.204 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:23:22 smithi052 ceph-mon[31955]: pgmap v344: 129 pgs: 129 active+clean; 639 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 7.3 KiB/s rd, 2.9 KiB/s wr, 12 op/s 2024-05-27T07:23:23.232 INFO:teuthology.orchestra.run.smithi052.stdout:{"pg_ready":true,"pg_map":{"version":344,"stamp":"2024-05-27T07:23:21.665058+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":653966,"num_objects":102,"num_object_clones":0,"num_object_copies":308,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":102,"num_whiteouts":0,"num_read":350,"num_read_kb":277,"num_write":371,"num_write_kb":2286,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":2951862,"num_keys_recovered":0,"num_objects_omap":66,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":305,"ondisk_log_size":305,"up":419,"acting":419,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":419,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":369368,"kb_used_data":11896,"kb_used_omap":18,"kb_used_meta":357357,"kb_avail":1124572456,"statfs":{"total":1151940427776,"available":1151562194944,"internally_reserved":0,"allocated":12181504,"data_stored":7931907,"data_compressed":425916,"data_compressed_allocated":577536,"data_compressed_original":2654208,"omap_allocated":19143,"internal_metadata":365933881},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":27070,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":123,"num_read_kb":88,"num_write":23,"num_write_kb":35,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003484"},"pg_stats":[{"pgid":"3.3f","version":"0'0","reported_seq":34,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-05-27T07:22:30.794972+0000","last_change":"2024-05-27T07:21:48.595870+0000","last_active":"2024-05-27T07:22:30.794972+0000","last_peered":"2024-05-27T07:22:30.794972+0000","last_clean":"2024-05-27T07:22:30.794972+0000","last_became_active":"2024-05-27T07:21:48.595020+0000","last_became_peered":"2024-05-27T07:21:48.595020+0000","last_unstale":"2024-05-27T07:22:30.794972+0000","last_undegraded":"2024-05-27T07:22:30.794972+0000","last_fullsized":"2024-05-27T07:22:30.794972+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:21:47.565641+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:21:47.565641+0000","last_clean_scrub_stamp":"2024-05-27T07:21:47.565641+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T16:43:08.920344+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.3e","version":"0'0","reported_seq":34,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-05-27T07:22:30.801613+0000","last_change":"2024-05-27T07:21:48.590237+0000","last_active":"2024-05-27T07:22:30.801613+0000","last_peered":"2024-05-27T07:22:30.801613+0000","last_clean":"2024-05-27T07:22:30.801613+0000","last_became_active":"2024-05-27T07:21:48.589876+0000","last_became_peered":"2024-05-27T07:21:48.589876+0000","last_unstale":"2024-05-27T07:22:30.801613+0000","last_undegraded":"2024-05-27T07:22:30.801613+0000","last_fullsized":"2024-05-27T07:22:30.801613+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:21:47.565641+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:21:47.565641+0000","last_clean_scrub_stamp":"2024-05-27T07:21:47.565641+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T16:23:08.881871+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,11,10],"acting":[0,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3d","version":"0'0","reported_seq":34,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-05-27T07:22:30.801204+0000","last_change":"2024-05-27T07:21:48.597744+0000","last_active":"2024-05-27T07:22:30.801204+0000","last_peered":"2024-05-27T07:22:30.801204+0000","last_clean":"2024-05-27T07:22:30.801204+0000","last_became_active":"2024-05-27T07:21:48.597410+0000","last_became_peered":"2024-05-27T07:21:48.597410+0000","last_unstale":"2024-05-27T07:22:30.801204+0000","last_undegraded":"2024-05-27T07:22:30.801204+0000","last_fullsized":"2024-05-27T07:22:30.801204+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:21:47.565641+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:21:47.565641+0000","last_clean_scrub_stamp":"2024-05-27T07:21:47.565641+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T19:06:49.643152+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"0'0","reported_seq":11,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-05-27T07:22:30.800824+0000","last_change":"2024-05-27T07:22:30.800824+0000","last_active":"2024-05-27T07:22:30.800824+0000","last_peered":"2024-05-27T07:22:30.800824+0000","last_clean":"2024-05-27T07:22:30.800824+0000","last_became_active":"2024-05-27T07:22:30.800415+0000","last_became_peered":"2024-05-27T07:22:30.800415+0000","last_unstale":"2024-05-27T07:22:30.800824+0000","last_undegraded":"2024-05-27T07:22:30.800824+0000","last_fullsized":"2024-05-27T07:22:30.800824+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:21:47.565641+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:21:47.565641+0000","last_clean_scrub_stamp":"2024-05-27T07:21:47.565641+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T11:06:10.451686+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,2],"acting":[9,8,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.3b","version":"0'0","reported_seq":34,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-05-27T07:22:30.799084+0000","last_change":"2024-05-27T07:21:48.583219+0000","last_active":"2024-05-27T07:22:30.799084+0000","last_peered":"2024-05-27T07:22:30.799084+0000","last_clean":"2024-05-27T07:22:30.799084+0000","last_became_active":"2024-05-27T07:21:48.582813+0000","last_became_peered":"2024-05-27T07:21:48.582813+0000","last_unstale":"2024-05-27T07:22:30.799084+0000","last_undegraded":"2024-05-27T07:22:30.799084+0000","last_fullsized":"2024-05-27T07:22:30.799084+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:21:47.565641+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:21:47.565641+0000","last_clean_scrub_stamp":"2024-05-27T07:21:47.565641+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T18:05:05.770965+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,3,6],"acting":[8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.3a","version":"0'0","reported_seq":34,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-05-27T07:22:31.416761+0000","last_change":"2024-05-27T07:21:48.592049+0000","last_active":"2024-05-27T07:22:31.416761+0000","last_peered":"2024-05-27T07:22:31.416761+0000","last_clean":"2024-05-27T07:22:31.416761+0000","last_became_active":"2024-05-27T07:21:48.591677+0000","last_became_peered":"2024-05-27T07:21:48.591677+0000","last_unstale":"2024-05-27T07:22:31.416761+0000","last_undegraded":"2024-05-27T07:22:31.416761+0000","last_fullsized":"2024-05-27T07:22:31.416761+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:21:47.565641+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:21:47.565641+0000","last_clean_scrub_stamp":"2024-05-27T07:21:47.565641+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T16:25:49.755893+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,4],"acting":[10,9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.39","version":"0'0","reported_seq":32,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-05-27T07:22:30.797598+0000","last_change":"2024-05-27T07:21:48.597876+0000","last_active":"2024-05-27T07:22:30.797598+0000","last_peered":"2024-05-27T07:22:30.797598+0000","last_clean":"2024-05-27T07:22:30.797598+0000","last_became_active":"2024-05-27T07:21:48.597566+0000","last_became_peered":"2024-05-27T07:21:48.597566+0000","last_unstale":"2024-05-27T07:22:30.797598+0000","last_undegraded":"2024-05-27T07:22:30.797598+0000","last_fullsized":"2024-05-27T07:22:30.797598+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:21:47.565641+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:21:47.565641+0000","last_clean_scrub_stamp":"2024-05-27T07:21:47.565641+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T09:30:43.438209+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,5],"acting":[7,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"0'0","reported_seq":34,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-05-27T07:22:30.855228+0000","last_change":"2024-05-27T07:21:48.593412+0000","last_active":"2024-05-27T07:22:30.855228+0000","last_peered":"2024-05-27T07:22:30.855228+0000","last_clean":"2024-05-27T07:22:30.855228+0000","last_became_active":"2024-05-27T07:21:48.593070+0000","last_became_peered":"2024-05-27T07:21:48.593070+0000","last_unstale":"2024-05-27T07:22:30.855228+0000","last_undegraded":"2024-05-27T07:22:30.855228+0000","last_fullsized":"2024-05-27T07:22:30.855228+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:21:47.565641+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:21:47.565641+0000","last_clean_scrub_stamp":"2024-05-27T07:21:47.565641+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T14:53:49.382796+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,10],"acting":[3,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"0'0","reported_seq":34,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-05-27T07:22:31.580336+0000","last_change":"2024-05-27T07:21:48.607244+0000","last_active":"2024-05-27T07:22:31.580336+0000","last_peered":"2024-05-27T07:22:31.580336+0000","last_clean":"2024-05-27T07:22:31.580336+0000","last_became_active":"2024-05-27T07:21:48.606238+0000","last_became_peered":"2024-05-27T07:21:48.606238+0000","last_unstale":"2024-05-27T07:22:31.580336+0000","last_undegraded":"2024-05-27T07:22:31.580336+0000","last_fullsized":"2024-05-27T07:22:31.580336+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:21:47.565641+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:21:47.565641+0000","last_clean_scrub_stamp":"2024-05-27T07:21:47.565641+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T12:36:33.142111+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,11],"acting":[1,5,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.36","version":"0'0","reported_seq":34,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-05-27T07:22:31.417314+0000","last_change":"2024-05-27T07:21:48.593275+0000","last_active":"2024-05-27T07:22:31.417314+0000","last_peered":"2024-05-27T07:22:31.417314+0000","last_clean":"2024-05-27T07:22:31.417314+0000","last_became_active":"2024-05-27T07:21:48.592890+0000","last_became_peered":"2024-05-27T07:21:48.592890+0000","last_unstale":"2024-05-27T07:22:31.417314+0000","last_undegraded":"2024-05-27T07:22:31.417314+0000","last_fullsized":"2024-05-27T07:22:31.417314+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:21:47.565641+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:21:47.565641+0000","last_clean_scrub_stamp":"2024-05-27T07:21:47.565641+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T13:47:20.827824+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,4],"acting":[10,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.35","version":"0'0","reported_seq":32,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-05-27T07:22:30.797332+0000","last_change":"2024-05-27T07:21:48.603424+0000","last_active":"2024-05-27T07:22:30.797332+0000","last_peered":"2024-05-27T07:22:30.797332+0000","last_clean":"2024-05-27T07:22:30.797332+0000","last_became_active":"2024-05-27T07:21:48.602962+0000","last_became_peered":"2024-05-27T07:21:48.602962+0000","last_unstale":"2024-05-27T07:22:30.797332+0000","last_undegraded":"2024-05-27T07:22:30.797332+0000","last_fullsized":"2024-05-27T07:22:30.797332+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:21:47.565641+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:21:47.565641+0000","last_clean_scrub_stamp":"2024-05-27T07:21:47.565641+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T19:19:42.063063+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,8,1],"acting":[7,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.34","version":"0'0","reported_seq":34,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-05-27T07:22:30.796472+0000","last_change":"2024-05-27T07:21:48.589323+0000","last_active":"2024-05-27T07:22:30.796472+0000","last_peered":"2024-05-27T07:22:30.796472+0000","last_clean":"2024-05-27T07:22:30.796472+0000","last_became_active":"2024-05-27T07:21:48.587435+0000","last_became_peered":"2024-05-27T07:21:48.587435+0000","last_unstale":"2024-05-27T07:22:30.796472+0000","last_undegraded":"2024-05-27T07:22:30.796472+0000","last_fullsized":"2024-05-27T07:22:30.796472+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:21:47.565641+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:21:47.565641+0000","last_clean_scrub_stamp":"2024-05-27T07:21:47.565641+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T13:34:44.340376+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,8],"acting":[4,10,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"0'0","reported_seq":34,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-05-27T07:22:31.626221+0000","last_change":"2024-05-27T07:21:48.593759+0000","last_active":"2024-05-27T07:22:31.626221+0000","last_peered":"2024-05-27T07:22:31.626221+0000","last_clean":"2024-05-27T07:22:31.626221+0000","last_became_active":"2024-05-27T07:21:48.593099+0000","last_became_peered":"2024-05-27T07:21:48.593099+0000","last_unstale":"2024-05-27T07:22:31.626221+0000","last_undegraded":"2024-05-27T07:22:31.626221+0000","last_fullsized":"2024-05-27T07:22:31.626221+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:21:47.565641+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:21:47.565641+0000","last_clean_scrub_stamp":"2024-05-27T07:21:47.565641+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T13:49:00.390792+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,5],"acting":[6,10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.32","version":"0'0","reported_seq":34,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-05-27T07:22:30.801091+0000","last_change":"2024-05-27T07:21:48.587486+0000","last_active":"2024-05-27T07:22:30.801091+0000","last_peered":"2024-05-27T07:22:30.801091+0000","last_clean":"2024-05-27T07:22:30.801091+0000","last_became_active":"2024-05-27T07:21:48.587159+0000","last_became_peered":"2024-05-27T07:21:48.587159+0000","last_unstale":"2024-05-27T07:22:30.801091+0000","last_undegraded":"2024-05-27T07:22:30.801091+0000","last_fullsized":"2024-05-27T07:22:30.801091+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:21:47.565641+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:21:47.565641+0000","last_clean_scrub_stamp":"2024-05-27T07:21:47.565641+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T15:34:11.908442+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,10],"acting":[0,8,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"0'0","reported_seq":34,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-05-27T07:22:30.801192+0000","last_change":"2024-05-27T07:21:48.596817+0000","last_active":"2024-05-27T07:22:30.801192+0000","last_peered":"2024-05-27T07:22:30.801192+0000","last_clean":"2024-05-27T07:22:30.801192+0000","last_became_active":"2024-05-27T07:21:48.596469+0000","last_became_peered":"2024-05-27T07:21:48.596469+0000","last_unstale":"2024-05-27T07:22:30.801192+0000","last_undegraded":"2024-05-27T07:22:30.801192+0000","last_fullsized":"2024-05-27T07:22:30.801192+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:21:47.565641+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:21:47.565641+0000","last_clean_scrub_stamp":"2024-05-27T07:21:47.565641+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T12:38:41.494060+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,2],"acting":[0,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.30","version":"0'0","reported_seq":34,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-05-27T07:22:31.579643+0000","last_change":"2024-05-27T07:21:48.604249+0000","last_active":"2024-05-27T07:22:31.579643+0000","last_peered":"2024-05-27T07:22:31.579643+0000","last_clean":"2024-05-27T07:22:31.579643+0000","last_became_active":"2024-05-27T07:21:48.603585+0000","last_became_peered":"2024-05-27T07:21:48.603585+0000","last_unstale":"2024-05-27T07:22:31.579643+0000","last_undegraded":"2024-05-27T07:22:31.579643+0000","last_fullsized":"2024-05-27T07:22:31.579643+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:21:47.565641+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:21:47.565641+0000","last_clean_scrub_stamp":"2024-05-27T07:21:47.565641+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T07:22:14.839824+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2f","version":"0'0","reported_seq":34,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-05-27T07:22:30.794834+0000","last_change":"2024-05-27T07:21:48.596035+0000","last_active":"2024-05-27T07:22:30.794834+0000","last_peered":"2024-05-27T07:22:30.794834+0000","last_clean":"2024-05-27T07:22:30.794834+0000","last_became_active":"2024-05-27T07:21:48.595509+0000","last_became_peered":"2024-05-27T07:21:48.595509+0000","last_unstale":"2024-05-27T07:22:30.794834+0000","last_undegraded":"2024-05-27T07:22:30.794834+0000","last_fullsized":"2024-05-27T07:22:30.794834+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:21:47.565641+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:21:47.565641+0000","last_clean_scrub_stamp":"2024-05-27T07:21:47.565641+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T13:23:03.734982+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.2e","version":"0'0","reported_seq":34,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-05-27T07:22:30.792717+0000","last_change":"2024-05-27T07:21:48.590002+0000","last_active":"2024-05-27T07:22:30.792717+0000","last_peered":"2024-05-27T07:22:30.792717+0000","last_clean":"2024-05-27T07:22:30.792717+0000","last_became_active":"2024-05-27T07:21:48.589608+0000","last_became_peered":"2024-05-27T07:21:48.589608+0000","last_unstale":"2024-05-27T07:22:30.792717+0000","last_undegraded":"2024-05-27T07:22:30.792717+0000","last_fullsized":"2024-05-27T07:22:30.792717+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:21:47.565641+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:21:47.565641+0000","last_clean_scrub_stamp":"2024-05-27T07:21:47.565641+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T09:33:39.174351+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,1,10],"acting":[9,1,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.2d","version":"0'0","reported_seq":34,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-05-27T07:22:30.801640+0000","last_change":"2024-05-27T07:21:48.594885+0000","last_active":"2024-05-27T07:22:30.801640+0000","last_peered":"2024-05-27T07:22:30.801640+0000","last_clean":"2024-05-27T07:22:30.801640+0000","last_became_active":"2024-05-27T07:21:48.594521+0000","last_became_peered":"2024-05-27T07:21:48.594521+0000","last_unstale":"2024-05-27T07:22:30.801640+0000","last_undegraded":"2024-05-27T07:22:30.801640+0000","last_fullsized":"2024-05-27T07:22:30.801640+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:21:47.565641+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:21:47.565641+0000","last_clean_scrub_stamp":"2024-05-27T07:21:47.565641+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T08:54:59.311938+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":31,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-05-27T07:22:30.857546+0000","last_change":"2024-05-27T07:22:30.857546+0000","last_active":"2024-05-27T07:22:30.857546+0000","last_peered":"2024-05-27T07:22:30.857546+0000","last_clean":"2024-05-27T07:22:30.857546+0000","last_became_active":"2024-05-27T07:22:30.856808+0000","last_became_peered":"2024-05-27T07:22:30.856808+0000","last_unstale":"2024-05-27T07:22:30.857546+0000","last_undegraded":"2024-05-27T07:22:30.857546+0000","last_fullsized":"2024-05-27T07:22:30.857546+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:21:58.698952+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:21:58.698952+0000","last_clean_scrub_stamp":"2024-05-27T07:21:58.698952+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T16:09:25.480864+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,8,0],"acting":[2,3,8,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1f","version":"88'13","reported_seq":78,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-05-27T07:23:18.926276+0000","last_change":"2024-05-27T07:21:45.566489+0000","last_active":"2024-05-27T07:23:18.926276+0000","last_peered":"2024-05-27T07:23:18.926276+0000","last_clean":"2024-05-27T07:23:18.926276+0000","last_became_active":"2024-05-27T07:21:45.566115+0000","last_became_peered":"2024-05-27T07:21:45.566115+0000","last_unstale":"2024-05-27T07:23:18.926276+0000","last_undegraded":"2024-05-27T07:23:18.926276+0000","last_fullsized":"2024-05-27T07:23:18.926276+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:21:44.543638+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:21:44.543638+0000","last_clean_scrub_stamp":"2024-05-27T07:21:44.543638+0000","objects_scrubbed":0,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T07:24:11.735014+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":25,"num_read_kb":25,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,8],"acting":[9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":34,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-05-27T07:22:30.855530+0000","last_change":"2024-05-27T07:21:48.588574+0000","last_active":"2024-05-27T07:22:30.855530+0000","last_peered":"2024-05-27T07:22:30.855530+0000","last_clean":"2024-05-27T07:22:30.855530+0000","last_became_active":"2024-05-27T07:21:48.588212+0000","last_became_peered":"2024-05-27T07:21:48.588212+0000","last_unstale":"2024-05-27T07:22:30.855530+0000","last_undegraded":"2024-05-27T07:22:30.855530+0000","last_fullsized":"2024-05-27T07:22:30.855530+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:21:47.565641+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:21:47.565641+0000","last_clean_scrub_stamp":"2024-05-27T07:21:47.565641+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T12:15:16.017635+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":24,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-05-27T07:22:30.795461+0000","last_change":"2024-05-27T07:21:59.734617+0000","last_active":"2024-05-27T07:22:30.795461+0000","last_peered":"2024-05-27T07:22:30.795461+0000","last_clean":"2024-05-27T07:22:30.795461+0000","last_became_active":"2024-05-27T07:21:59.734162+0000","last_became_peered":"2024-05-27T07:21:59.734162+0000","last_unstale":"2024-05-27T07:22:30.795461+0000","last_undegraded":"2024-05-27T07:22:30.795461+0000","last_fullsized":"2024-05-27T07:22:30.795461+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:21:58.698952+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:21:58.698952+0000","last_clean_scrub_stamp":"2024-05-27T07:21:58.698952+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T12:40:01.375520+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,9,3],"acting":[4,6,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1e","version":"88'2","reported_seq":41,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-05-27T07:22:30.855869+0000","last_change":"2024-05-27T07:21:45.790907+0000","last_active":"2024-05-27T07:22:30.855869+0000","last_peered":"2024-05-27T07:22:30.855869+0000","last_clean":"2024-05-27T07:22:30.855869+0000","last_became_active":"2024-05-27T07:21:45.790451+0000","last_became_peered":"2024-05-27T07:21:45.790451+0000","last_unstale":"2024-05-27T07:22:30.855869+0000","last_undegraded":"2024-05-27T07:22:30.855869+0000","last_fullsized":"2024-05-27T07:22:30.855869+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:21:44.543638+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:21:44.543638+0000","last_clean_scrub_stamp":"2024-05-27T07:21:44.543638+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T10:21:56.524930+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":34,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-05-27T07:22:30.801453+0000","last_change":"2024-05-27T07:21:48.595975+0000","last_active":"2024-05-27T07:22:30.801453+0000","last_peered":"2024-05-27T07:22:30.801453+0000","last_clean":"2024-05-27T07:22:30.801453+0000","last_became_active":"2024-05-27T07:21:48.595608+0000","last_became_peered":"2024-05-27T07:21:48.595608+0000","last_unstale":"2024-05-27T07:22:30.801453+0000","last_undegraded":"2024-05-27T07:22:30.801453+0000","last_fullsized":"2024-05-27T07:22:30.801453+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:21:47.565641+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:21:47.565641+0000","last_clean_scrub_stamp":"2024-05-27T07:21:47.565641+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T07:24:44.797973+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,9],"acting":[0,1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":24,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-05-27T07:22:30.795210+0000","last_change":"2024-05-27T07:21:59.732788+0000","last_active":"2024-05-27T07:22:30.795210+0000","last_peered":"2024-05-27T07:22:30.795210+0000","last_clean":"2024-05-27T07:22:30.795210+0000","last_became_active":"2024-05-27T07:21:59.732310+0000","last_became_peered":"2024-05-27T07:21:59.732310+0000","last_unstale":"2024-05-27T07:22:30.795210+0000","last_undegraded":"2024-05-27T07:22:30.795210+0000","last_fullsized":"2024-05-27T07:22:30.795210+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:21:58.698952+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:21:58.698952+0000","last_clean_scrub_stamp":"2024-05-27T07:21:58.698952+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T13:54:51.490154+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,9,5],"acting":[4,3,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1d","version":"88'12","reported_seq":53,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-05-27T07:22:56.890405+0000","last_change":"2024-05-27T07:21:45.569746+0000","last_active":"2024-05-27T07:22:56.890405+0000","last_peered":"2024-05-27T07:22:56.890405+0000","last_clean":"2024-05-27T07:22:56.890405+0000","last_became_active":"2024-05-27T07:21:45.569366+0000","last_became_peered":"2024-05-27T07:21:45.569366+0000","last_unstale":"2024-05-27T07:22:56.890405+0000","last_undegraded":"2024-05-27T07:22:56.890405+0000","last_fullsized":"2024-05-27T07:22:56.890405+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:21:44.543638+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:21:44.543638+0000","last_clean_scrub_stamp":"2024-05-27T07:21:44.543638+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T15:58:32.977754+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":16,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":7,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":34,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-05-27T07:22:31.579902+0000","last_change":"2024-05-27T07:21:48.606138+0000","last_active":"2024-05-27T07:22:31.579902+0000","last_peered":"2024-05-27T07:22:31.579902+0000","last_clean":"2024-05-27T07:22:31.579902+0000","last_became_active":"2024-05-27T07:21:48.605711+0000","last_became_peered":"2024-05-27T07:21:48.605711+0000","last_unstale":"2024-05-27T07:22:31.579902+0000","last_undegraded":"2024-05-27T07:22:31.579902+0000","last_fullsized":"2024-05-27T07:22:31.579902+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:21:47.565641+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:21:47.565641+0000","last_clean_scrub_stamp":"2024-05-27T07:21:47.565641+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T10:46:39.052387+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,6],"acting":[1,10,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":24,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-05-27T07:22:30.794971+0000","last_change":"2024-05-27T07:21:59.735475+0000","last_active":"2024-05-27T07:22:30.794971+0000","last_peered":"2024-05-27T07:22:30.794971+0000","last_clean":"2024-05-27T07:22:30.794971+0000","last_became_active":"2024-05-27T07:21:59.734986+0000","last_became_peered":"2024-05-27T07:21:59.734986+0000","last_unstale":"2024-05-27T07:22:30.794971+0000","last_undegraded":"2024-05-27T07:22:30.794971+0000","last_fullsized":"2024-05-27T07:22:30.794971+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:21:58.698952+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:21:58.698952+0000","last_clean_scrub_stamp":"2024-05-27T07:21:58.698952+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T14:48:28.198718+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5,3],"acting":[4,6,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1c","version":"88'5","reported_seq":46,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-05-27T07:22:55.890154+0000","last_change":"2024-05-27T07:21:45.572882+0000","last_active":"2024-05-27T07:22:55.890154+0000","last_peered":"2024-05-27T07:22:55.890154+0000","last_clean":"2024-05-27T07:22:55.890154+0000","last_became_active":"2024-05-27T07:21:45.572460+0000","last_became_peered":"2024-05-27T07:21:45.572460+0000","last_unstale":"2024-05-27T07:22:55.890154+0000","last_undegraded":"2024-05-27T07:22:55.890154+0000","last_fullsized":"2024-05-27T07:22:55.890154+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:21:44.543638+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:21:44.543638+0000","last_clean_scrub_stamp":"2024-05-27T07:21:44.543638+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T09:40:11.311923+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":7,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,1,5],"acting":[8,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":34,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-05-27T07:22:30.797360+0000","last_change":"2024-05-27T07:21:48.596217+0000","last_active":"2024-05-27T07:22:30.797360+0000","last_peered":"2024-05-27T07:22:30.797360+0000","last_clean":"2024-05-27T07:22:30.797360+0000","last_became_active":"2024-05-27T07:21:48.595832+0000","last_became_peered":"2024-05-27T07:21:48.595832+0000","last_unstale":"2024-05-27T07:22:30.797360+0000","last_undegraded":"2024-05-27T07:22:30.797360+0000","last_fullsized":"2024-05-27T07:22:30.797360+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:21:47.565641+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:21:47.565641+0000","last_clean_scrub_stamp":"2024-05-27T07:21:47.565641+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T10:08:05.304404+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":24,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-05-27T07:22:30.801397+0000","last_change":"2024-05-27T07:21:59.729617+0000","last_active":"2024-05-27T07:22:30.801397+0000","last_peered":"2024-05-27T07:22:30.801397+0000","last_clean":"2024-05-27T07:22:30.801397+0000","last_became_active":"2024-05-27T07:21:59.729003+0000","last_became_peered":"2024-05-27T07:21:59.729003+0000","last_unstale":"2024-05-27T07:22:30.801397+0000","last_undegraded":"2024-05-27T07:22:30.801397+0000","last_fullsized":"2024-05-27T07:22:30.801397+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:21:58.698952+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:21:58.698952+0000","last_clean_scrub_stamp":"2024-05-27T07:21:58.698952+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T14:55:04.844948+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7,11],"acting":[2,5,7,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1b","version":"88'6","reported_seq":72,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-05-27T07:23:18.927844+0000","last_change":"2024-05-27T07:21:45.566355+0000","last_active":"2024-05-27T07:23:18.927844+0000","last_peered":"2024-05-27T07:23:18.927844+0000","last_clean":"2024-05-27T07:23:18.927844+0000","last_became_active":"2024-05-27T07:21:45.564776+0000","last_became_peered":"2024-05-27T07:21:45.564776+0000","last_unstale":"2024-05-27T07:23:18.927844+0000","last_undegraded":"2024-05-27T07:23:18.927844+0000","last_fullsized":"2024-05-27T07:23:18.927844+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:21:44.543638+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:21:44.543638+0000","last_clean_scrub_stamp":"2024-05-27T07:21:44.543638+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T12:10:41.632630+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1013,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":27,"num_read_kb":3,"num_write":3,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":34,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-05-27T07:22:30.798264+0000","last_change":"2024-05-27T07:21:48.588837+0000","last_active":"2024-05-27T07:22:30.798264+0000","last_peered":"2024-05-27T07:22:30.798264+0000","last_clean":"2024-05-27T07:22:30.798264+0000","last_became_active":"2024-05-27T07:21:48.587149+0000","last_became_peered":"2024-05-27T07:21:48.587149+0000","last_unstale":"2024-05-27T07:22:30.798264+0000","last_undegraded":"2024-05-27T07:22:30.798264+0000","last_fullsized":"2024-05-27T07:22:30.798264+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:21:47.565641+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:21:47.565641+0000","last_clean_scrub_stamp":"2024-05-27T07:21:47.565641+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T09:20:05.124189+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,5],"acting":[8,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":24,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-05-27T07:22:30.801357+0000","last_change":"2024-05-27T07:21:59.730003+0000","last_active":"2024-05-27T07:22:30.801357+0000","last_peered":"2024-05-27T07:22:30.801357+0000","last_clean":"2024-05-27T07:22:30.801357+0000","last_became_active":"2024-05-27T07:21:59.729195+0000","last_became_peered":"2024-05-27T07:21:59.729195+0000","last_unstale":"2024-05-27T07:22:30.801357+0000","last_undegraded":"2024-05-27T07:22:30.801357+0000","last_fullsized":"2024-05-27T07:22:30.801357+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:21:58.698952+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:21:58.698952+0000","last_clean_scrub_stamp":"2024-05-27T07:21:58.698952+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T13:13:41.612300+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,9,5],"acting":[2,6,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1a","version":"88'7","reported_seq":47,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-05-27T07:22:54.881270+0000","last_change":"2024-05-27T07:21:45.567279+0000","last_active":"2024-05-27T07:22:54.881270+0000","last_peered":"2024-05-27T07:22:54.881270+0000","last_clean":"2024-05-27T07:22:54.881270+0000","last_became_active":"2024-05-27T07:21:45.566920+0000","last_became_peered":"2024-05-27T07:21:45.566920+0000","last_unstale":"2024-05-27T07:22:54.881270+0000","last_undegraded":"2024-05-27T07:22:54.881270+0000","last_fullsized":"2024-05-27T07:22:54.881270+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:21:44.543638+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:21:44.543638+0000","last_clean_scrub_stamp":"2024-05-27T07:21:44.543638+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T18:12:23.015535+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,8,6],"acting":[10,8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":40,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-05-27T07:22:30.857080+0000","last_change":"2024-05-27T07:22:30.857080+0000","last_active":"2024-05-27T07:22:30.857080+0000","last_peered":"2024-05-27T07:22:30.857080+0000","last_clean":"2024-05-27T07:22:30.857080+0000","last_became_active":"2024-05-27T07:22:30.856479+0000","last_became_peered":"2024-05-27T07:22:30.856479+0000","last_unstale":"2024-05-27T07:22:30.857080+0000","last_undegraded":"2024-05-27T07:22:30.857080+0000","last_fullsized":"2024-05-27T07:22:30.857080+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:21:47.565641+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:21:47.565641+0000","last_clean_scrub_stamp":"2024-05-27T07:21:47.565641+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T13:29:16.054448+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,8,3],"acting":[5,8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":31,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-05-27T07:22:30.805292+0000","last_change":"2024-05-27T07:22:30.805292+0000","last_active":"2024-05-27T07:22:30.805292+0000","last_peered":"2024-05-27T07:22:30.805292+0000","last_clean":"2024-05-27T07:22:30.805292+0000","last_became_active":"2024-05-27T07:22:30.804779+0000","last_became_peered":"2024-05-27T07:22:30.804779+0000","last_unstale":"2024-05-27T07:22:30.805292+0000","last_undegraded":"2024-05-27T07:22:30.805292+0000","last_fullsized":"2024-05-27T07:22:30.805292+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:21:58.698952+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:21:58.698952+0000","last_clean_scrub_stamp":"2024-05-27T07:21:58.698952+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T11:12:49.095931+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,8,0],"acting":[11,4,8,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.19","version":"88'3","reported_seq":42,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-05-27T07:22:53.872954+0000","last_change":"2024-05-27T07:21:45.561255+0000","last_active":"2024-05-27T07:22:53.872954+0000","last_peered":"2024-05-27T07:22:53.872954+0000","last_clean":"2024-05-27T07:22:53.872954+0000","last_became_active":"2024-05-27T07:21:45.560896+0000","last_became_peered":"2024-05-27T07:21:45.560896+0000","last_unstale":"2024-05-27T07:22:53.872954+0000","last_undegraded":"2024-05-27T07:22:53.872954+0000","last_fullsized":"2024-05-27T07:22:53.872954+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:21:44.543638+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:21:44.543638+0000","last_clean_scrub_stamp":"2024-05-27T07:21:44.543638+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T17:05:24.074628+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,9],"acting":[6,4,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":34,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-05-27T07:22:30.855361+0000","last_change":"2024-05-27T07:21:48.586045+0000","last_active":"2024-05-27T07:22:30.855361+0000","last_peered":"2024-05-27T07:22:30.855361+0000","last_clean":"2024-05-27T07:22:30.855361+0000","last_became_active":"2024-05-27T07:21:48.585681+0000","last_became_peered":"2024-05-27T07:21:48.585681+0000","last_unstale":"2024-05-27T07:22:30.855361+0000","last_undegraded":"2024-05-27T07:22:30.855361+0000","last_fullsized":"2024-05-27T07:22:30.855361+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:21:47.565641+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:21:47.565641+0000","last_clean_scrub_stamp":"2024-05-27T07:21:47.565641+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T09:56:36.408623+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,6],"acting":[3,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":24,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-05-27T07:22:30.801482+0000","last_change":"2024-05-27T07:21:59.739544+0000","last_active":"2024-05-27T07:22:30.801482+0000","last_peered":"2024-05-27T07:22:30.801482+0000","last_clean":"2024-05-27T07:22:30.801482+0000","last_became_active":"2024-05-27T07:21:59.739132+0000","last_became_peered":"2024-05-27T07:21:59.739132+0000","last_unstale":"2024-05-27T07:22:30.801482+0000","last_undegraded":"2024-05-27T07:22:30.801482+0000","last_fullsized":"2024-05-27T07:22:30.801482+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:21:58.698952+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:21:58.698952+0000","last_clean_scrub_stamp":"2024-05-27T07:21:58.698952+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T10:06:05.126579+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,6,1],"acting":[0,7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.18","version":"88'3","reported_seq":43,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-05-27T07:22:55.889742+0000","last_change":"2024-05-27T07:21:45.568171+0000","last_active":"2024-05-27T07:22:55.889742+0000","last_peered":"2024-05-27T07:22:55.889742+0000","last_clean":"2024-05-27T07:22:55.889742+0000","last_became_active":"2024-05-27T07:21:45.567627+0000","last_became_peered":"2024-05-27T07:21:45.567627+0000","last_unstale":"2024-05-27T07:22:55.889742+0000","last_undegraded":"2024-05-27T07:22:55.889742+0000","last_fullsized":"2024-05-27T07:22:55.889742+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:21:44.543638+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:21:44.543638+0000","last_clean_scrub_stamp":"2024-05-27T07:21:44.543638+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T16:43:56.407079+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":34,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-05-27T07:22:30.792156+0000","last_change":"2024-05-27T07:21:48.587452+0000","last_active":"2024-05-27T07:22:30.792156+0000","last_peered":"2024-05-27T07:22:30.792156+0000","last_clean":"2024-05-27T07:22:30.792156+0000","last_became_active":"2024-05-27T07:21:48.587027+0000","last_became_peered":"2024-05-27T07:21:48.587027+0000","last_unstale":"2024-05-27T07:22:30.792156+0000","last_undegraded":"2024-05-27T07:22:30.792156+0000","last_fullsized":"2024-05-27T07:22:30.792156+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:21:47.565641+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:21:47.565641+0000","last_clean_scrub_stamp":"2024-05-27T07:21:47.565641+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T13:51:44.890061+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,10],"acting":[9,3,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":24,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-05-27T07:22:30.855893+0000","last_change":"2024-05-27T07:21:59.738292+0000","last_active":"2024-05-27T07:22:30.855893+0000","last_peered":"2024-05-27T07:22:30.855893+0000","last_clean":"2024-05-27T07:22:30.855893+0000","last_became_active":"2024-05-27T07:21:59.737941+0000","last_became_peered":"2024-05-27T07:21:59.737941+0000","last_unstale":"2024-05-27T07:22:30.855893+0000","last_undegraded":"2024-05-27T07:22:30.855893+0000","last_fullsized":"2024-05-27T07:22:30.855893+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:21:58.698952+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:21:58.698952+0000","last_clean_scrub_stamp":"2024-05-27T07:21:58.698952+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T13:54:24.192917+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1,5],"acting":[3,7,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.17","version":"88'4","reported_seq":90,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-05-27T07:23:18.927012+0000","last_change":"2024-05-27T07:21:45.568338+0000","last_active":"2024-05-27T07:23:18.927012+0000","last_peered":"2024-05-27T07:23:18.927012+0000","last_clean":"2024-05-27T07:23:18.927012+0000","last_became_active":"2024-05-27T07:21:45.567856+0000","last_became_peered":"2024-05-27T07:21:45.567856+0000","last_unstale":"2024-05-27T07:23:18.927012+0000","last_undegraded":"2024-05-27T07:23:18.927012+0000","last_fullsized":"2024-05-27T07:23:18.927012+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:21:44.543638+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:21:44.543638+0000","last_clean_scrub_stamp":"2024-05-27T07:21:44.543638+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T15:02:04.188711+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":180,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":46,"num_read_kb":46,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,5,9],"acting":[10,5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":34,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-05-27T07:22:31.580247+0000","last_change":"2024-05-27T07:21:48.606062+0000","last_active":"2024-05-27T07:22:31.580247+0000","last_peered":"2024-05-27T07:22:31.580247+0000","last_clean":"2024-05-27T07:22:31.580247+0000","last_became_active":"2024-05-27T07:21:48.605423+0000","last_became_peered":"2024-05-27T07:21:48.605423+0000","last_unstale":"2024-05-27T07:22:31.580247+0000","last_undegraded":"2024-05-27T07:22:31.580247+0000","last_fullsized":"2024-05-27T07:22:31.580247+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:21:47.565641+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:21:47.565641+0000","last_clean_scrub_stamp":"2024-05-27T07:21:47.565641+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T14:39:06.921519+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,9],"acting":[1,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":24,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-05-27T07:22:30.792054+0000","last_change":"2024-05-27T07:21:59.735150+0000","last_active":"2024-05-27T07:22:30.792054+0000","last_peered":"2024-05-27T07:22:30.792054+0000","last_clean":"2024-05-27T07:22:30.792054+0000","last_became_active":"2024-05-27T07:21:59.734688+0000","last_became_peered":"2024-05-27T07:21:59.734688+0000","last_unstale":"2024-05-27T07:22:30.792054+0000","last_undegraded":"2024-05-27T07:22:30.792054+0000","last_fullsized":"2024-05-27T07:22:30.792054+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:21:58.698952+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:21:58.698952+0000","last_clean_scrub_stamp":"2024-05-27T07:21:58.698952+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T15:38:02.095947+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,0,2,4],"acting":[9,0,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.16","version":"88'4","reported_seq":41,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-05-27T07:22:55.889723+0000","last_change":"2024-05-27T07:21:45.559550+0000","last_active":"2024-05-27T07:22:55.889723+0000","last_peered":"2024-05-27T07:22:55.889723+0000","last_clean":"2024-05-27T07:22:55.889723+0000","last_became_active":"2024-05-27T07:21:45.559181+0000","last_became_peered":"2024-05-27T07:21:45.559181+0000","last_unstale":"2024-05-27T07:22:55.889723+0000","last_undegraded":"2024-05-27T07:22:55.889723+0000","last_fullsized":"2024-05-27T07:22:55.889723+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:21:44.543638+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:21:44.543638+0000","last_clean_scrub_stamp":"2024-05-27T07:21:44.543638+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T10:33:45.288437+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":34,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-05-27T07:22:30.801530+0000","last_change":"2024-05-27T07:21:48.593631+0000","last_active":"2024-05-27T07:22:30.801530+0000","last_peered":"2024-05-27T07:22:30.801530+0000","last_clean":"2024-05-27T07:22:30.801530+0000","last_became_active":"2024-05-27T07:21:48.592330+0000","last_became_peered":"2024-05-27T07:21:48.592330+0000","last_unstale":"2024-05-27T07:22:30.801530+0000","last_undegraded":"2024-05-27T07:22:30.801530+0000","last_fullsized":"2024-05-27T07:22:30.801530+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:21:47.565641+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:21:47.565641+0000","last_clean_scrub_stamp":"2024-05-27T07:21:47.565641+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T19:19:51.593307+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,5],"acting":[0,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":24,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-05-27T07:22:30.795471+0000","last_change":"2024-05-27T07:21:59.725158+0000","last_active":"2024-05-27T07:22:30.795471+0000","last_peered":"2024-05-27T07:22:30.795471+0000","last_clean":"2024-05-27T07:22:30.795471+0000","last_became_active":"2024-05-27T07:21:59.724651+0000","last_became_peered":"2024-05-27T07:21:59.724651+0000","last_unstale":"2024-05-27T07:22:30.795471+0000","last_undegraded":"2024-05-27T07:22:30.795471+0000","last_fullsized":"2024-05-27T07:22:30.795471+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:21:58.698952+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:21:58.698952+0000","last_clean_scrub_stamp":"2024-05-27T07:21:58.698952+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T12:10:45.747484+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2,9],"acting":[4,1,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.15","version":"88'5","reported_seq":45,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-05-27T07:22:54.880267+0000","last_change":"2024-05-27T07:21:45.791831+0000","last_active":"2024-05-27T07:22:54.880267+0000","last_peered":"2024-05-27T07:22:54.880267+0000","last_clean":"2024-05-27T07:22:54.880267+0000","last_became_active":"2024-05-27T07:21:45.791271+0000","last_became_peered":"2024-05-27T07:21:45.791271+0000","last_unstale":"2024-05-27T07:22:54.880267+0000","last_undegraded":"2024-05-27T07:22:54.880267+0000","last_fullsized":"2024-05-27T07:22:54.880267+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:21:44.543638+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:21:44.543638+0000","last_clean_scrub_stamp":"2024-05-27T07:21:44.543638+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T16:20:37.492337+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":7,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,10],"acting":[5,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":34,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-05-27T07:22:31.580163+0000","last_change":"2024-05-27T07:21:48.605605+0000","last_active":"2024-05-27T07:22:31.580163+0000","last_peered":"2024-05-27T07:22:31.580163+0000","last_clean":"2024-05-27T07:22:31.580163+0000","last_became_active":"2024-05-27T07:21:48.604655+0000","last_became_peered":"2024-05-27T07:21:48.604655+0000","last_unstale":"2024-05-27T07:22:31.580163+0000","last_undegraded":"2024-05-27T07:22:31.580163+0000","last_fullsized":"2024-05-27T07:22:31.580163+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:21:47.565641+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:21:47.565641+0000","last_clean_scrub_stamp":"2024-05-27T07:21:47.565641+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T14:33:53.661983+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,8,4],"acting":[1,8,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":31,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-05-27T07:22:31.418393+0000","last_change":"2024-05-27T07:22:31.418393+0000","last_active":"2024-05-27T07:22:31.418393+0000","last_peered":"2024-05-27T07:22:31.418393+0000","last_clean":"2024-05-27T07:22:31.418393+0000","last_became_active":"2024-05-27T07:22:31.418006+0000","last_became_peered":"2024-05-27T07:22:31.418006+0000","last_unstale":"2024-05-27T07:22:31.418393+0000","last_undegraded":"2024-05-27T07:22:31.418393+0000","last_fullsized":"2024-05-27T07:22:31.418393+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:21:58.698952+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:21:58.698952+0000","last_clean_scrub_stamp":"2024-05-27T07:21:58.698952+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T14:30:55.130461+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,11,10],"acting":[9,8,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.14","version":"88'32","reported_seq":126,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-05-27T07:23:18.933066+0000","last_change":"2024-05-27T07:21:45.564829+0000","last_active":"2024-05-27T07:23:18.933066+0000","last_peered":"2024-05-27T07:23:18.933066+0000","last_clean":"2024-05-27T07:23:18.933066+0000","last_became_active":"2024-05-27T07:21:45.564457+0000","last_became_peered":"2024-05-27T07:21:45.564457+0000","last_unstale":"2024-05-27T07:23:18.933066+0000","last_undegraded":"2024-05-27T07:23:18.933066+0000","last_fullsized":"2024-05-27T07:23:18.933066+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:21:44.543638+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:21:44.543638+0000","last_clean_scrub_stamp":"2024-05-27T07:21:44.543638+0000","objects_scrubbed":0,"log_size":32,"log_dups_size":0,"ondisk_log_size":32,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T19:12:44.546989+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":56160,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":54,"num_read_kb":82,"num_write":34,"num_write_kb":68,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,1,2],"acting":[10,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":34,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-05-27T07:22:30.798642+0000","last_change":"2024-05-27T07:21:48.591672+0000","last_active":"2024-05-27T07:22:30.798642+0000","last_peered":"2024-05-27T07:22:30.798642+0000","last_clean":"2024-05-27T07:22:30.798642+0000","last_became_active":"2024-05-27T07:21:48.590861+0000","last_became_peered":"2024-05-27T07:21:48.590861+0000","last_unstale":"2024-05-27T07:22:30.798642+0000","last_undegraded":"2024-05-27T07:22:30.798642+0000","last_fullsized":"2024-05-27T07:22:30.798642+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:21:47.565641+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:21:47.565641+0000","last_clean_scrub_stamp":"2024-05-27T07:21:47.565641+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T12:03:40.627764+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,7],"acting":[8,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":24,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-05-27T07:22:30.797886+0000","last_change":"2024-05-27T07:21:59.736446+0000","last_active":"2024-05-27T07:22:30.797886+0000","last_peered":"2024-05-27T07:22:30.797886+0000","last_clean":"2024-05-27T07:22:30.797886+0000","last_became_active":"2024-05-27T07:21:59.735854+0000","last_became_peered":"2024-05-27T07:21:59.735854+0000","last_unstale":"2024-05-27T07:22:30.797886+0000","last_undegraded":"2024-05-27T07:22:30.797886+0000","last_fullsized":"2024-05-27T07:22:30.797886+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:21:58.698952+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:21:58.698952+0000","last_clean_scrub_stamp":"2024-05-27T07:21:58.698952+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T08:02:14.640325+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,8,3,6],"acting":[5,8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.13","version":"88'9","reported_seq":50,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-05-27T07:22:56.890667+0000","last_change":"2024-05-27T07:21:45.562930+0000","last_active":"2024-05-27T07:22:56.890667+0000","last_peered":"2024-05-27T07:22:56.890667+0000","last_clean":"2024-05-27T07:22:56.890667+0000","last_became_active":"2024-05-27T07:21:45.562515+0000","last_became_peered":"2024-05-27T07:21:45.562515+0000","last_unstale":"2024-05-27T07:22:56.890667+0000","last_undegraded":"2024-05-27T07:22:56.890667+0000","last_fullsized":"2024-05-27T07:22:56.890667+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:21:44.543638+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:21:44.543638+0000","last_clean_scrub_stamp":"2024-05-27T07:21:44.543638+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T17:54:24.006219+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":68,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":13,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,10,3],"acting":[2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":34,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-05-27T07:22:31.626151+0000","last_change":"2024-05-27T07:21:48.591090+0000","last_active":"2024-05-27T07:22:31.626151+0000","last_peered":"2024-05-27T07:22:31.626151+0000","last_clean":"2024-05-27T07:22:31.626151+0000","last_became_active":"2024-05-27T07:21:48.590726+0000","last_became_peered":"2024-05-27T07:21:48.590726+0000","last_unstale":"2024-05-27T07:22:31.626151+0000","last_undegraded":"2024-05-27T07:22:31.626151+0000","last_fullsized":"2024-05-27T07:22:31.626151+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:21:47.565641+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:21:47.565641+0000","last_clean_scrub_stamp":"2024-05-27T07:21:47.565641+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T18:40:03.880407+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,3],"acting":[6,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.14","version":"0'0","reported_seq":24,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-05-27T07:22:30.793943+0000","last_change":"2024-05-27T07:21:59.738375+0000","last_active":"2024-05-27T07:22:30.793943+0000","last_peered":"2024-05-27T07:22:30.793943+0000","last_clean":"2024-05-27T07:22:30.793943+0000","last_became_active":"2024-05-27T07:21:59.737957+0000","last_became_peered":"2024-05-27T07:21:59.737957+0000","last_unstale":"2024-05-27T07:22:30.793943+0000","last_undegraded":"2024-05-27T07:22:30.793943+0000","last_fullsized":"2024-05-27T07:22:30.793943+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:21:58.698952+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:21:58.698952+0000","last_clean_scrub_stamp":"2024-05-27T07:21:58.698952+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T16:53:51.397143+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,9,7,6],"acting":[11,9,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.12","version":"88'5","reported_seq":45,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-05-27T07:22:55.888096+0000","last_change":"2024-05-27T07:21:45.566986+0000","last_active":"2024-05-27T07:22:55.888096+0000","last_peered":"2024-05-27T07:22:55.888096+0000","last_clean":"2024-05-27T07:22:55.888096+0000","last_became_active":"2024-05-27T07:21:45.566579+0000","last_became_peered":"2024-05-27T07:21:45.566579+0000","last_unstale":"2024-05-27T07:22:55.888096+0000","last_undegraded":"2024-05-27T07:22:55.888096+0000","last_fullsized":"2024-05-27T07:22:55.888096+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:21:44.543638+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:21:44.543638+0000","last_clean_scrub_stamp":"2024-05-27T07:21:44.543638+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T17:08:14.034740+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":5,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":32,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-05-27T07:22:30.796994+0000","last_change":"2024-05-27T07:21:48.596575+0000","last_active":"2024-05-27T07:22:30.796994+0000","last_peered":"2024-05-27T07:22:30.796994+0000","last_clean":"2024-05-27T07:22:30.796994+0000","last_became_active":"2024-05-27T07:21:48.596215+0000","last_became_peered":"2024-05-27T07:21:48.596215+0000","last_unstale":"2024-05-27T07:22:30.796994+0000","last_undegraded":"2024-05-27T07:22:30.796994+0000","last_fullsized":"2024-05-27T07:22:30.796994+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:21:47.565641+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:21:47.565641+0000","last_clean_scrub_stamp":"2024-05-27T07:21:47.565641+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T14:57:05.795269+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":24,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-05-27T07:22:30.792387+0000","last_change":"2024-05-27T07:21:59.737690+0000","last_active":"2024-05-27T07:22:30.792387+0000","last_peered":"2024-05-27T07:22:30.792387+0000","last_clean":"2024-05-27T07:22:30.792387+0000","last_became_active":"2024-05-27T07:21:59.735646+0000","last_became_peered":"2024-05-27T07:21:59.735646+0000","last_unstale":"2024-05-27T07:22:30.792387+0000","last_undegraded":"2024-05-27T07:22:30.792387+0000","last_fullsized":"2024-05-27T07:22:30.792387+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:21:58.698952+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:21:58.698952+0000","last_clean_scrub_stamp":"2024-05-27T07:21:58.698952+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T13:54:41.227626+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,0,3],"acting":[9,7,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.11","version":"88'6","reported_seq":73,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-05-27T07:23:18.927905+0000","last_change":"2024-05-27T07:21:45.564836+0000","last_active":"2024-05-27T07:23:18.927905+0000","last_peered":"2024-05-27T07:23:18.927905+0000","last_clean":"2024-05-27T07:23:18.927905+0000","last_became_active":"2024-05-27T07:21:45.564481+0000","last_became_peered":"2024-05-27T07:21:45.564481+0000","last_unstale":"2024-05-27T07:23:18.927905+0000","last_undegraded":"2024-05-27T07:23:18.927905+0000","last_fullsized":"2024-05-27T07:23:18.927905+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:21:44.543638+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:21:44.543638+0000","last_clean_scrub_stamp":"2024-05-27T07:21:44.543638+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T15:13:12.012769+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":979,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":28,"num_read_kb":4,"num_write":6,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,10],"acting":[4,7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":32,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-05-27T07:22:30.797769+0000","last_change":"2024-05-27T07:21:48.601331+0000","last_active":"2024-05-27T07:22:30.797769+0000","last_peered":"2024-05-27T07:22:30.797769+0000","last_clean":"2024-05-27T07:22:30.797769+0000","last_became_active":"2024-05-27T07:21:48.600917+0000","last_became_peered":"2024-05-27T07:21:48.600917+0000","last_unstale":"2024-05-27T07:22:30.797769+0000","last_undegraded":"2024-05-27T07:22:30.797769+0000","last_fullsized":"2024-05-27T07:22:30.797769+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:21:47.565641+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:21:47.565641+0000","last_clean_scrub_stamp":"2024-05-27T07:21:47.565641+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T17:26:42.129989+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":24,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-05-27T07:22:31.626720+0000","last_change":"2024-05-27T07:21:59.737662+0000","last_active":"2024-05-27T07:22:31.626720+0000","last_peered":"2024-05-27T07:22:31.626720+0000","last_clean":"2024-05-27T07:22:31.626720+0000","last_became_active":"2024-05-27T07:21:59.737229+0000","last_became_peered":"2024-05-27T07:21:59.737229+0000","last_unstale":"2024-05-27T07:22:31.626720+0000","last_undegraded":"2024-05-27T07:22:31.626720+0000","last_fullsized":"2024-05-27T07:22:31.626720+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:21:58.698952+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:21:58.698952+0000","last_clean_scrub_stamp":"2024-05-27T07:21:58.698952+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T15:32:31.147571+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,9,11],"acting":[6,3,9,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.10","version":"88'17","reported_seq":74,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-05-27T07:22:55.889266+0000","last_change":"2024-05-27T07:22:30.814815+0000","last_active":"2024-05-27T07:22:55.889266+0000","last_peered":"2024-05-27T07:22:55.889266+0000","last_clean":"2024-05-27T07:22:55.889266+0000","last_became_active":"2024-05-27T07:22:30.802828+0000","last_became_peered":"2024-05-27T07:22:30.802828+0000","last_unstale":"2024-05-27T07:22:55.889266+0000","last_undegraded":"2024-05-27T07:22:55.889266+0000","last_fullsized":"2024-05-27T07:22:55.889266+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:21:44.543638+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:21:44.543638+0000","last_clean_scrub_stamp":"2024-05-27T07:21:44.543638+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T11:38:07.363507+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,4],"acting":[8,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":34,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-05-27T07:22:30.797081+0000","last_change":"2024-05-27T07:21:48.596204+0000","last_active":"2024-05-27T07:22:30.797081+0000","last_peered":"2024-05-27T07:22:30.797081+0000","last_clean":"2024-05-27T07:22:30.797081+0000","last_became_active":"2024-05-27T07:21:48.595827+0000","last_became_peered":"2024-05-27T07:21:48.595827+0000","last_unstale":"2024-05-27T07:22:30.797081+0000","last_undegraded":"2024-05-27T07:22:30.797081+0000","last_fullsized":"2024-05-27T07:22:30.797081+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:21:47.565641+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:21:47.565641+0000","last_clean_scrub_stamp":"2024-05-27T07:21:47.565641+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T15:51:53.162554+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":24,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-05-27T07:22:30.796942+0000","last_change":"2024-05-27T07:21:59.735837+0000","last_active":"2024-05-27T07:22:30.796942+0000","last_peered":"2024-05-27T07:22:30.796942+0000","last_clean":"2024-05-27T07:22:30.796942+0000","last_became_active":"2024-05-27T07:21:59.735063+0000","last_became_peered":"2024-05-27T07:21:59.735063+0000","last_unstale":"2024-05-27T07:22:30.796942+0000","last_undegraded":"2024-05-27T07:22:30.796942+0000","last_fullsized":"2024-05-27T07:22:30.796942+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:21:58.698952+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:21:58.698952+0000","last_clean_scrub_stamp":"2024-05-27T07:21:58.698952+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T13:04:21.751043+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,3,1],"acting":[5,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.f","version":"88'2","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-05-27T07:22:54.869368+0000","last_change":"2024-05-27T07:22:30.809491+0000","last_active":"2024-05-27T07:22:54.869368+0000","last_peered":"2024-05-27T07:22:54.869368+0000","last_clean":"2024-05-27T07:22:54.869368+0000","last_became_active":"2024-05-27T07:22:30.802187+0000","last_became_peered":"2024-05-27T07:22:30.802187+0000","last_unstale":"2024-05-27T07:22:54.869368+0000","last_undegraded":"2024-05-27T07:22:54.869368+0000","last_fullsized":"2024-05-27T07:22:54.869368+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:21:44.543638+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:21:44.543638+0000","last_clean_scrub_stamp":"2024-05-27T07:21:44.543638+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T18:11:32.821962+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":22,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,4],"acting":[11,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":34,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-05-27T07:22:30.793572+0000","last_change":"2024-05-27T07:21:48.597362+0000","last_active":"2024-05-27T07:22:30.793572+0000","last_peered":"2024-05-27T07:22:30.793572+0000","last_clean":"2024-05-27T07:22:30.793572+0000","last_became_active":"2024-05-27T07:21:48.596920+0000","last_became_peered":"2024-05-27T07:21:48.596920+0000","last_unstale":"2024-05-27T07:22:30.793572+0000","last_undegraded":"2024-05-27T07:22:30.793572+0000","last_fullsized":"2024-05-27T07:22:30.793572+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:21:47.565641+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:21:47.565641+0000","last_clean_scrub_stamp":"2024-05-27T07:21:47.565641+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T16:50:21.021320+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,4],"acting":[11,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":24,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-05-27T07:22:30.798136+0000","last_change":"2024-05-27T07:21:59.732303+0000","last_active":"2024-05-27T07:22:30.798136+0000","last_peered":"2024-05-27T07:22:30.798136+0000","last_clean":"2024-05-27T07:22:30.798136+0000","last_became_active":"2024-05-27T07:21:59.731851+0000","last_became_peered":"2024-05-27T07:21:59.731851+0000","last_unstale":"2024-05-27T07:22:30.798136+0000","last_undegraded":"2024-05-27T07:22:30.798136+0000","last_fullsized":"2024-05-27T07:22:30.798136+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:21:58.698952+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:21:58.698952+0000","last_clean_scrub_stamp":"2024-05-27T07:21:58.698952+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T08:08:38.433818+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,10,11,9],"acting":[8,10,11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.4","version":"88'1","reported_seq":41,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-05-27T07:22:30.792697+0000","last_change":"2024-05-27T07:21:45.791370+0000","last_active":"2024-05-27T07:22:30.792697+0000","last_peered":"2024-05-27T07:22:30.792697+0000","last_clean":"2024-05-27T07:22:30.792697+0000","last_became_active":"2024-05-27T07:21:45.790987+0000","last_became_peered":"2024-05-27T07:21:45.790987+0000","last_unstale":"2024-05-27T07:22:30.792697+0000","last_undegraded":"2024-05-27T07:22:30.792697+0000","last_fullsized":"2024-05-27T07:22:30.792697+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:21:44.543638+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:21:44.543638+0000","last_clean_scrub_stamp":"2024-05-27T07:21:44.543638+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T14:20:18.570194+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2,0],"acting":[9,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":34,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-05-27T07:22:31.416954+0000","last_change":"2024-05-27T07:21:48.585984+0000","last_active":"2024-05-27T07:22:31.416954+0000","last_peered":"2024-05-27T07:22:31.416954+0000","last_clean":"2024-05-27T07:22:31.416954+0000","last_became_active":"2024-05-27T07:21:48.585606+0000","last_became_peered":"2024-05-27T07:21:48.585606+0000","last_unstale":"2024-05-27T07:22:31.416954+0000","last_undegraded":"2024-05-27T07:22:31.416954+0000","last_fullsized":"2024-05-27T07:22:31.416954+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:21:47.565641+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:21:47.565641+0000","last_clean_scrub_stamp":"2024-05-27T07:21:47.565641+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T17:56:17.659024+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,3,6],"acting":[10,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":24,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-05-27T07:22:31.626383+0000","last_change":"2024-05-27T07:21:59.734022+0000","last_active":"2024-05-27T07:22:31.626383+0000","last_peered":"2024-05-27T07:22:31.626383+0000","last_clean":"2024-05-27T07:22:31.626383+0000","last_became_active":"2024-05-27T07:21:59.733467+0000","last_became_peered":"2024-05-27T07:21:59.733467+0000","last_unstale":"2024-05-27T07:22:31.626383+0000","last_undegraded":"2024-05-27T07:22:31.626383+0000","last_fullsized":"2024-05-27T07:22:31.626383+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:21:58.698952+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:21:58.698952+0000","last_clean_scrub_stamp":"2024-05-27T07:21:58.698952+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T09:54:11.058495+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,1,8],"acting":[6,0,1,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.2","version":"88'5","reported_seq":64,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-05-27T07:23:15.922395+0000","last_change":"2024-05-27T07:21:45.563072+0000","last_active":"2024-05-27T07:23:15.922395+0000","last_peered":"2024-05-27T07:23:15.922395+0000","last_clean":"2024-05-27T07:23:15.922395+0000","last_became_active":"2024-05-27T07:21:45.562632+0000","last_became_peered":"2024-05-27T07:21:45.562632+0000","last_unstale":"2024-05-27T07:23:15.922395+0000","last_undegraded":"2024-05-27T07:23:15.922395+0000","last_fullsized":"2024-05-27T07:23:15.922395+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:21:44.543638+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:21:44.543638+0000","last_clean_scrub_stamp":"2024-05-27T07:21:44.543638+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T12:54:19.555467+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":588,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":22,"num_read_kb":1,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":34,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-05-27T07:22:31.416512+0000","last_change":"2024-05-27T07:21:48.594867+0000","last_active":"2024-05-27T07:22:31.416512+0000","last_peered":"2024-05-27T07:22:31.416512+0000","last_clean":"2024-05-27T07:22:31.416512+0000","last_became_active":"2024-05-27T07:21:48.594523+0000","last_became_peered":"2024-05-27T07:21:48.594523+0000","last_unstale":"2024-05-27T07:22:31.416512+0000","last_undegraded":"2024-05-27T07:22:31.416512+0000","last_fullsized":"2024-05-27T07:22:31.416512+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:21:47.565641+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:21:47.565641+0000","last_clean_scrub_stamp":"2024-05-27T07:21:47.565641+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T17:28:55.862014+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,4,0],"acting":[10,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":24,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-05-27T07:22:31.626438+0000","last_change":"2024-05-27T07:21:59.741936+0000","last_active":"2024-05-27T07:22:31.626438+0000","last_peered":"2024-05-27T07:22:31.626438+0000","last_clean":"2024-05-27T07:22:31.626438+0000","last_became_active":"2024-05-27T07:21:59.741458+0000","last_became_peered":"2024-05-27T07:21:59.741458+0000","last_unstale":"2024-05-27T07:22:31.626438+0000","last_undegraded":"2024-05-27T07:22:31.626438+0000","last_fullsized":"2024-05-27T07:22:31.626438+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:21:58.698952+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:21:58.698952+0000","last_clean_scrub_stamp":"2024-05-27T07:21:58.698952+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T15:51:28.560995+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,10,0],"acting":[6,5,10,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":11,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-05-27T07:22:30.803379+0000","last_change":"2024-05-27T07:22:30.803379+0000","last_active":"2024-05-27T07:22:30.803379+0000","last_peered":"2024-05-27T07:22:30.803379+0000","last_clean":"2024-05-27T07:22:30.803379+0000","last_became_active":"2024-05-27T07:22:30.803024+0000","last_became_peered":"2024-05-27T07:22:30.803024+0000","last_unstale":"2024-05-27T07:22:30.803379+0000","last_undegraded":"2024-05-27T07:22:30.803379+0000","last_fullsized":"2024-05-27T07:22:30.803379+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:21:47.565641+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:21:47.565641+0000","last_clean_scrub_stamp":"2024-05-27T07:21:47.565641+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T07:56:32.070610+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2,4],"acting":[9,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.1","version":"88'5","reported_seq":69,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-05-27T07:23:18.927282+0000","last_change":"2024-05-27T07:21:45.790849+0000","last_active":"2024-05-27T07:23:18.927282+0000","last_peered":"2024-05-27T07:23:18.927282+0000","last_clean":"2024-05-27T07:23:18.927282+0000","last_became_active":"2024-05-27T07:21:45.790435+0000","last_became_peered":"2024-05-27T07:21:45.790435+0000","last_unstale":"2024-05-27T07:23:18.927282+0000","last_undegraded":"2024-05-27T07:23:18.927282+0000","last_fullsized":"2024-05-27T07:23:18.927282+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:21:44.543638+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:21:44.543638+0000","last_clean_scrub_stamp":"2024-05-27T07:21:44.543638+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T09:34:51.479882+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":25,"num_read_kb":25,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,0],"acting":[6,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.5","version":"88'1","reported_seq":24,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-05-27T07:22:30.797393+0000","last_change":"2024-05-27T07:21:59.731338+0000","last_active":"2024-05-27T07:22:30.797393+0000","last_peered":"2024-05-27T07:22:30.797393+0000","last_clean":"2024-05-27T07:22:30.797393+0000","last_became_active":"2024-05-27T07:21:59.730847+0000","last_became_peered":"2024-05-27T07:21:59.730847+0000","last_unstale":"2024-05-27T07:22:30.797393+0000","last_undegraded":"2024-05-27T07:22:30.797393+0000","last_fullsized":"2024-05-27T07:22:30.797393+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:21:58.698952+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:21:58.698952+0000","last_clean_scrub_stamp":"2024-05-27T07:21:58.698952+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T18:18:50.901129+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,11,9],"acting":[7,5,11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":34,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-05-27T07:22:30.855389+0000","last_change":"2024-05-27T07:21:48.594274+0000","last_active":"2024-05-27T07:22:30.855389+0000","last_peered":"2024-05-27T07:22:30.855389+0000","last_clean":"2024-05-27T07:22:30.855389+0000","last_became_active":"2024-05-27T07:21:48.593940+0000","last_became_peered":"2024-05-27T07:21:48.593940+0000","last_unstale":"2024-05-27T07:22:30.855389+0000","last_undegraded":"2024-05-27T07:22:30.855389+0000","last_fullsized":"2024-05-27T07:22:30.855389+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:21:47.565641+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:21:47.565641+0000","last_clean_scrub_stamp":"2024-05-27T07:21:47.565641+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T13:25:46.531524+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"88'6","reported_seq":49,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-05-27T07:22:56.891759+0000","last_change":"2024-05-27T07:21:45.563365+0000","last_active":"2024-05-27T07:22:56.891759+0000","last_peered":"2024-05-27T07:22:56.891759+0000","last_clean":"2024-05-27T07:22:56.891759+0000","last_became_active":"2024-05-27T07:21:45.563012+0000","last_became_peered":"2024-05-27T07:21:45.563012+0000","last_unstale":"2024-05-27T07:22:56.891759+0000","last_undegraded":"2024-05-27T07:22:56.891759+0000","last_fullsized":"2024-05-27T07:22:56.891759+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:21:44.543638+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:21:44.543638+0000","last_clean_scrub_stamp":"2024-05-27T07:21:44.543638+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T12:58:51.079965+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":68,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":4,"num_read_kb":4,"num_write":8,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,9],"acting":[4,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.0","version":"20'92","reported_seq":112,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-05-27T07:22:30.796503+0000","last_change":"2024-05-27T07:20:21.611212+0000","last_active":"2024-05-27T07:22:30.796503+0000","last_peered":"2024-05-27T07:22:30.796503+0000","last_clean":"2024-05-27T07:22:30.796503+0000","last_became_active":"2024-05-27T07:20:21.605163+0000","last_became_peered":"2024-05-27T07:20:21.605163+0000","last_unstale":"2024-05-27T07:22:30.796503+0000","last_undegraded":"2024-05-27T07:22:30.796503+0000","last_fullsized":"2024-05-27T07:22:30.796503+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:16:57.668433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:16:57.668433+0000","last_clean_scrub_stamp":"2024-05-27T07:16:57.668433+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T12:10:59.074162+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":81,"num_read_kb":69,"num_write":123,"num_write_kb":2150,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":2951840,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":24,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-05-27T07:22:30.797236+0000","last_change":"2024-05-27T07:21:59.737210+0000","last_active":"2024-05-27T07:22:30.797236+0000","last_peered":"2024-05-27T07:22:30.797236+0000","last_clean":"2024-05-27T07:22:30.797236+0000","last_became_active":"2024-05-27T07:21:59.736799+0000","last_became_peered":"2024-05-27T07:21:59.736799+0000","last_unstale":"2024-05-27T07:22:30.797236+0000","last_undegraded":"2024-05-27T07:22:30.797236+0000","last_fullsized":"2024-05-27T07:22:30.797236+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:21:58.698952+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:21:58.698952+0000","last_clean_scrub_stamp":"2024-05-27T07:21:58.698952+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T13:03:17.153687+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,6,10],"acting":[5,9,6,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":34,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-05-27T07:22:31.625943+0000","last_change":"2024-05-27T07:21:48.591550+0000","last_active":"2024-05-27T07:22:31.625943+0000","last_peered":"2024-05-27T07:22:31.625943+0000","last_clean":"2024-05-27T07:22:31.625943+0000","last_became_active":"2024-05-27T07:21:48.591223+0000","last_became_peered":"2024-05-27T07:21:48.591223+0000","last_unstale":"2024-05-27T07:22:31.625943+0000","last_undegraded":"2024-05-27T07:22:31.625943+0000","last_fullsized":"2024-05-27T07:22:31.625943+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:21:47.565641+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:21:47.565641+0000","last_clean_scrub_stamp":"2024-05-27T07:21:47.565641+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T15:06:23.395645+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,2],"acting":[6,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.0","version":"88'4","reported_seq":41,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-05-27T07:22:53.872885+0000","last_change":"2024-05-27T07:21:45.569224+0000","last_active":"2024-05-27T07:22:53.872885+0000","last_peered":"2024-05-27T07:22:53.872885+0000","last_clean":"2024-05-27T07:22:53.872885+0000","last_became_active":"2024-05-27T07:21:45.568818+0000","last_became_peered":"2024-05-27T07:21:45.568818+0000","last_unstale":"2024-05-27T07:22:53.872885+0000","last_undegraded":"2024-05-27T07:22:53.872885+0000","last_fullsized":"2024-05-27T07:22:53.872885+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:21:44.543638+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:21:44.543638+0000","last_clean_scrub_stamp":"2024-05-27T07:21:44.543638+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T08:12:33.157842+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,8],"acting":[7,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":24,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-05-27T07:22:30.793255+0000","last_change":"2024-05-27T07:21:59.733733+0000","last_active":"2024-05-27T07:22:30.793255+0000","last_peered":"2024-05-27T07:22:30.793255+0000","last_clean":"2024-05-27T07:22:30.793255+0000","last_became_active":"2024-05-27T07:21:59.733112+0000","last_became_peered":"2024-05-27T07:21:59.733112+0000","last_unstale":"2024-05-27T07:22:30.793255+0000","last_undegraded":"2024-05-27T07:22:30.793255+0000","last_fullsized":"2024-05-27T07:22:30.793255+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:21:58.698952+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:21:58.698952+0000","last_clean_scrub_stamp":"2024-05-27T07:21:58.698952+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T16:01:30.783800+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,7,6],"acting":[11,5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.5","version":"88'4","reported_seq":45,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-05-27T07:22:55.890006+0000","last_change":"2024-05-27T07:21:45.792029+0000","last_active":"2024-05-27T07:22:55.890006+0000","last_peered":"2024-05-27T07:22:55.890006+0000","last_clean":"2024-05-27T07:22:55.890006+0000","last_became_active":"2024-05-27T07:21:45.791492+0000","last_became_peered":"2024-05-27T07:21:45.791492+0000","last_unstale":"2024-05-27T07:22:55.890006+0000","last_undegraded":"2024-05-27T07:22:55.890006+0000","last_fullsized":"2024-05-27T07:22:55.890006+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:21:44.543638+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:21:44.543638+0000","last_clean_scrub_stamp":"2024-05-27T07:21:44.543638+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T17:36:42.444243+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,9],"acting":[8,0,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":34,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-05-27T07:22:31.626503+0000","last_change":"2024-05-27T07:21:48.594769+0000","last_active":"2024-05-27T07:22:31.626503+0000","last_peered":"2024-05-27T07:22:31.626503+0000","last_clean":"2024-05-27T07:22:31.626503+0000","last_became_active":"2024-05-27T07:21:48.592979+0000","last_became_peered":"2024-05-27T07:21:48.592979+0000","last_unstale":"2024-05-27T07:22:31.626503+0000","last_undegraded":"2024-05-27T07:22:31.626503+0000","last_fullsized":"2024-05-27T07:22:31.626503+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:21:47.565641+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:21:47.565641+0000","last_clean_scrub_stamp":"2024-05-27T07:21:47.565641+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T10:06:48.268230+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,9],"acting":[6,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":24,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-05-27T07:22:30.855263+0000","last_change":"2024-05-27T07:21:59.736405+0000","last_active":"2024-05-27T07:22:30.855263+0000","last_peered":"2024-05-27T07:22:30.855263+0000","last_clean":"2024-05-27T07:22:30.855263+0000","last_became_active":"2024-05-27T07:21:59.735991+0000","last_became_peered":"2024-05-27T07:21:59.735991+0000","last_unstale":"2024-05-27T07:22:30.855263+0000","last_undegraded":"2024-05-27T07:22:30.855263+0000","last_fullsized":"2024-05-27T07:22:30.855263+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:21:58.698952+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:21:58.698952+0000","last_clean_scrub_stamp":"2024-05-27T07:21:58.698952+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T18:32:04.931725+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,10,0,7],"acting":[3,10,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"88'15","reported_seq":55,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-05-27T07:23:16.593341+0000","last_change":"2024-05-27T07:21:45.566513+0000","last_active":"2024-05-27T07:23:16.593341+0000","last_peered":"2024-05-27T07:23:16.593341+0000","last_clean":"2024-05-27T07:23:16.593341+0000","last_became_active":"2024-05-27T07:21:45.566107+0000","last_became_peered":"2024-05-27T07:21:45.566107+0000","last_unstale":"2024-05-27T07:23:16.593341+0000","last_undegraded":"2024-05-27T07:23:16.593341+0000","last_fullsized":"2024-05-27T07:23:16.593341+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:21:44.543638+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:21:44.543638+0000","last_clean_scrub_stamp":"2024-05-27T07:21:44.543638+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T14:11:34.851691+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":27,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,3],"acting":[9,8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":34,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-05-27T07:22:30.855851+0000","last_change":"2024-05-27T07:21:48.593642+0000","last_active":"2024-05-27T07:22:30.855851+0000","last_peered":"2024-05-27T07:22:30.855851+0000","last_clean":"2024-05-27T07:22:30.855851+0000","last_became_active":"2024-05-27T07:21:48.592885+0000","last_became_peered":"2024-05-27T07:21:48.592885+0000","last_unstale":"2024-05-27T07:22:30.855851+0000","last_undegraded":"2024-05-27T07:22:30.855851+0000","last_fullsized":"2024-05-27T07:22:30.855851+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:21:47.565641+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:21:47.565641+0000","last_clean_scrub_stamp":"2024-05-27T07:21:47.565641+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T11:51:27.186560+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,7],"acting":[3,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":24,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-05-27T07:22:30.800760+0000","last_change":"2024-05-27T07:21:59.725533+0000","last_active":"2024-05-27T07:22:30.800760+0000","last_peered":"2024-05-27T07:22:30.800760+0000","last_clean":"2024-05-27T07:22:30.800760+0000","last_became_active":"2024-05-27T07:21:59.724991+0000","last_became_peered":"2024-05-27T07:21:59.724991+0000","last_unstale":"2024-05-27T07:22:30.800760+0000","last_undegraded":"2024-05-27T07:22:30.800760+0000","last_fullsized":"2024-05-27T07:22:30.800760+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:21:58.698952+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:21:58.698952+0000","last_clean_scrub_stamp":"2024-05-27T07:21:58.698952+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T17:22:54.650553+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,4,7],"acting":[2,1,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"88'7","reported_seq":48,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-05-27T07:22:55.888243+0000","last_change":"2024-05-27T07:21:45.572701+0000","last_active":"2024-05-27T07:22:55.888243+0000","last_peered":"2024-05-27T07:22:55.888243+0000","last_clean":"2024-05-27T07:22:55.888243+0000","last_became_active":"2024-05-27T07:21:45.572176+0000","last_became_peered":"2024-05-27T07:21:45.572176+0000","last_unstale":"2024-05-27T07:22:55.888243+0000","last_undegraded":"2024-05-27T07:22:55.888243+0000","last_fullsized":"2024-05-27T07:22:55.888243+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:21:44.543638+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:21:44.543638+0000","last_clean_scrub_stamp":"2024-05-27T07:21:44.543638+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T16:11:20.901250+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,11,6],"acting":[4,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":34,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-05-27T07:22:30.801671+0000","last_change":"2024-05-27T07:21:48.581152+0000","last_active":"2024-05-27T07:22:30.801671+0000","last_peered":"2024-05-27T07:22:30.801671+0000","last_clean":"2024-05-27T07:22:30.801671+0000","last_became_active":"2024-05-27T07:21:48.580543+0000","last_became_peered":"2024-05-27T07:21:48.580543+0000","last_unstale":"2024-05-27T07:22:30.801671+0000","last_undegraded":"2024-05-27T07:22:30.801671+0000","last_fullsized":"2024-05-27T07:22:30.801671+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:21:47.565641+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:21:47.565641+0000","last_clean_scrub_stamp":"2024-05-27T07:21:47.565641+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T18:45:56.732828+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,4],"acting":[0,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":31,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-05-27T07:22:31.417863+0000","last_change":"2024-05-27T07:22:31.417863+0000","last_active":"2024-05-27T07:22:31.417863+0000","last_peered":"2024-05-27T07:22:31.417863+0000","last_clean":"2024-05-27T07:22:31.417863+0000","last_became_active":"2024-05-27T07:22:31.417298+0000","last_became_peered":"2024-05-27T07:22:31.417298+0000","last_unstale":"2024-05-27T07:22:31.417863+0000","last_undegraded":"2024-05-27T07:22:31.417863+0000","last_fullsized":"2024-05-27T07:22:31.417863+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:21:58.698952+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:21:58.698952+0000","last_clean_scrub_stamp":"2024-05-27T07:21:58.698952+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T12:26:13.857477+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,10,11],"acting":[4,5,10,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.8","version":"88'2","reported_seq":42,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-05-27T07:22:52.858389+0000","last_change":"2024-05-27T07:21:45.790914+0000","last_active":"2024-05-27T07:22:52.858389+0000","last_peered":"2024-05-27T07:22:52.858389+0000","last_clean":"2024-05-27T07:22:52.858389+0000","last_became_active":"2024-05-27T07:21:45.790518+0000","last_became_peered":"2024-05-27T07:21:45.790518+0000","last_unstale":"2024-05-27T07:22:52.858389+0000","last_undegraded":"2024-05-27T07:22:52.858389+0000","last_fullsized":"2024-05-27T07:22:52.858389+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:21:44.543638+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:21:44.543638+0000","last_clean_scrub_stamp":"2024-05-27T07:21:44.543638+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T08:50:51.651129+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":34,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-05-27T07:22:30.798819+0000","last_change":"2024-05-27T07:21:48.589605+0000","last_active":"2024-05-27T07:22:30.798819+0000","last_peered":"2024-05-27T07:22:30.798819+0000","last_clean":"2024-05-27T07:22:30.798819+0000","last_became_active":"2024-05-27T07:21:48.588294+0000","last_became_peered":"2024-05-27T07:21:48.588294+0000","last_unstale":"2024-05-27T07:22:30.798819+0000","last_undegraded":"2024-05-27T07:22:30.798819+0000","last_fullsized":"2024-05-27T07:22:30.798819+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:21:47.565641+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:21:47.565641+0000","last_clean_scrub_stamp":"2024-05-27T07:21:47.565641+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T09:02:29.618508+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,10],"acting":[8,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":24,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-05-27T07:22:30.792550+0000","last_change":"2024-05-27T07:21:59.738153+0000","last_active":"2024-05-27T07:22:30.792550+0000","last_peered":"2024-05-27T07:22:30.792550+0000","last_clean":"2024-05-27T07:22:30.792550+0000","last_became_active":"2024-05-27T07:21:59.737394+0000","last_became_peered":"2024-05-27T07:21:59.737394+0000","last_unstale":"2024-05-27T07:22:30.792550+0000","last_undegraded":"2024-05-27T07:22:30.792550+0000","last_fullsized":"2024-05-27T07:22:30.792550+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:21:58.698952+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:21:58.698952+0000","last_clean_scrub_stamp":"2024-05-27T07:21:58.698952+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T17:17:13.877688+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,2,5],"acting":[9,3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.9","version":"88'7","reported_seq":46,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-05-27T07:22:55.889211+0000","last_change":"2024-05-27T07:21:45.562638+0000","last_active":"2024-05-27T07:22:55.889211+0000","last_peered":"2024-05-27T07:22:55.889211+0000","last_clean":"2024-05-27T07:22:55.889211+0000","last_became_active":"2024-05-27T07:21:45.562242+0000","last_became_peered":"2024-05-27T07:21:45.562242+0000","last_unstale":"2024-05-27T07:22:55.889211+0000","last_undegraded":"2024-05-27T07:22:55.889211+0000","last_fullsized":"2024-05-27T07:22:55.889211+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:21:44.543638+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:21:44.543638+0000","last_clean_scrub_stamp":"2024-05-27T07:21:44.543638+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T10:40:15.847513+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,9,10],"acting":[3,9,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":34,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-05-27T07:22:30.793275+0000","last_change":"2024-05-27T07:21:48.598186+0000","last_active":"2024-05-27T07:22:30.793275+0000","last_peered":"2024-05-27T07:22:30.793275+0000","last_clean":"2024-05-27T07:22:30.793275+0000","last_became_active":"2024-05-27T07:21:48.597715+0000","last_became_peered":"2024-05-27T07:21:48.597715+0000","last_unstale":"2024-05-27T07:22:30.793275+0000","last_undegraded":"2024-05-27T07:22:30.793275+0000","last_fullsized":"2024-05-27T07:22:30.793275+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:21:47.565641+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:21:47.565641+0000","last_clean_scrub_stamp":"2024-05-27T07:21:47.565641+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T08:01:35.983687+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,2],"acting":[11,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.c","version":"88'1","reported_seq":25,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-05-27T07:22:30.795901+0000","last_change":"2024-05-27T07:21:59.727247+0000","last_active":"2024-05-27T07:22:30.795901+0000","last_peered":"2024-05-27T07:22:30.795901+0000","last_clean":"2024-05-27T07:22:30.795901+0000","last_became_active":"2024-05-27T07:21:59.726788+0000","last_became_peered":"2024-05-27T07:21:59.726788+0000","last_unstale":"2024-05-27T07:22:30.795901+0000","last_undegraded":"2024-05-27T07:22:30.795901+0000","last_fullsized":"2024-05-27T07:22:30.795901+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:21:58.698952+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:21:58.698952+0000","last_clean_scrub_stamp":"2024-05-27T07:21:58.698952+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T10:09:27.095188+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,9,7,8],"acting":[4,9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.a","version":"88'3","reported_seq":42,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-05-27T07:22:54.879698+0000","last_change":"2024-05-27T07:21:45.569285+0000","last_active":"2024-05-27T07:22:54.879698+0000","last_peered":"2024-05-27T07:22:54.879698+0000","last_clean":"2024-05-27T07:22:54.879698+0000","last_became_active":"2024-05-27T07:21:45.568835+0000","last_became_peered":"2024-05-27T07:21:45.568835+0000","last_unstale":"2024-05-27T07:22:54.879698+0000","last_undegraded":"2024-05-27T07:22:54.879698+0000","last_fullsized":"2024-05-27T07:22:54.879698+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:21:44.543638+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:21:44.543638+0000","last_clean_scrub_stamp":"2024-05-27T07:21:44.543638+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T10:03:23.353800+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,8],"acting":[2,3,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":34,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-05-27T07:22:30.855702+0000","last_change":"2024-05-27T07:21:48.583494+0000","last_active":"2024-05-27T07:22:30.855702+0000","last_peered":"2024-05-27T07:22:30.855702+0000","last_clean":"2024-05-27T07:22:30.855702+0000","last_became_active":"2024-05-27T07:21:48.583141+0000","last_became_peered":"2024-05-27T07:21:48.583141+0000","last_unstale":"2024-05-27T07:22:30.855702+0000","last_undegraded":"2024-05-27T07:22:30.855702+0000","last_fullsized":"2024-05-27T07:22:30.855702+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:21:47.565641+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:21:47.565641+0000","last_clean_scrub_stamp":"2024-05-27T07:21:47.565641+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T10:24:50.475195+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,10],"acting":[3,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":24,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-05-27T07:22:30.796714+0000","last_change":"2024-05-27T07:21:59.733966+0000","last_active":"2024-05-27T07:22:30.796714+0000","last_peered":"2024-05-27T07:22:30.796714+0000","last_clean":"2024-05-27T07:22:30.796714+0000","last_became_active":"2024-05-27T07:21:59.733411+0000","last_became_peered":"2024-05-27T07:21:59.733411+0000","last_unstale":"2024-05-27T07:22:30.796714+0000","last_undegraded":"2024-05-27T07:22:30.796714+0000","last_fullsized":"2024-05-27T07:22:30.796714+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:21:58.698952+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:21:58.698952+0000","last_clean_scrub_stamp":"2024-05-27T07:21:58.698952+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T15:08:04.918086+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,10,3],"acting":[4,2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.b","version":"88'4","reported_seq":45,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-05-27T07:22:52.858809+0000","last_change":"2024-05-27T07:21:45.570967+0000","last_active":"2024-05-27T07:22:52.858809+0000","last_peered":"2024-05-27T07:22:52.858809+0000","last_clean":"2024-05-27T07:22:52.858809+0000","last_became_active":"2024-05-27T07:21:45.570501+0000","last_became_peered":"2024-05-27T07:21:45.570501+0000","last_unstale":"2024-05-27T07:22:52.858809+0000","last_undegraded":"2024-05-27T07:22:52.858809+0000","last_fullsized":"2024-05-27T07:22:52.858809+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:21:44.543638+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:21:44.543638+0000","last_clean_scrub_stamp":"2024-05-27T07:21:44.543638+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T14:11:26.496194+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":32,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-05-27T07:22:30.796885+0000","last_change":"2024-05-27T07:21:48.594690+0000","last_active":"2024-05-27T07:22:30.796885+0000","last_peered":"2024-05-27T07:22:30.796885+0000","last_clean":"2024-05-27T07:22:30.796885+0000","last_became_active":"2024-05-27T07:21:48.594314+0000","last_became_peered":"2024-05-27T07:21:48.594314+0000","last_unstale":"2024-05-27T07:22:30.796885+0000","last_undegraded":"2024-05-27T07:22:30.796885+0000","last_fullsized":"2024-05-27T07:22:30.796885+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:21:47.565641+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:21:47.565641+0000","last_clean_scrub_stamp":"2024-05-27T07:21:47.565641+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T14:06:43.401844+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,6],"acting":[7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":24,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-05-27T07:22:31.579716+0000","last_change":"2024-05-27T07:21:59.732289+0000","last_active":"2024-05-27T07:22:31.579716+0000","last_peered":"2024-05-27T07:22:31.579716+0000","last_clean":"2024-05-27T07:22:31.579716+0000","last_became_active":"2024-05-27T07:21:59.731863+0000","last_became_peered":"2024-05-27T07:21:59.731863+0000","last_unstale":"2024-05-27T07:22:31.579716+0000","last_undegraded":"2024-05-27T07:22:31.579716+0000","last_fullsized":"2024-05-27T07:22:31.579716+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:21:58.698952+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:21:58.698952+0000","last_clean_scrub_stamp":"2024-05-27T07:21:58.698952+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T11:06:38.246484+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,6,7],"acting":[1,5,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.c","version":"88'5","reported_seq":66,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-05-27T07:23:14.921568+0000","last_change":"2024-05-27T07:21:45.791978+0000","last_active":"2024-05-27T07:23:14.921568+0000","last_peered":"2024-05-27T07:23:14.921568+0000","last_clean":"2024-05-27T07:23:14.921568+0000","last_became_active":"2024-05-27T07:21:45.791146+0000","last_became_peered":"2024-05-27T07:21:45.791146+0000","last_unstale":"2024-05-27T07:23:14.921568+0000","last_undegraded":"2024-05-27T07:23:14.921568+0000","last_fullsized":"2024-05-27T07:23:14.921568+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:21:44.543638+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:21:44.543638+0000","last_clean_scrub_stamp":"2024-05-27T07:21:44.543638+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T09:09:24.952876+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1170,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":22,"num_read_kb":2,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,11,0],"acting":[2,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":32,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-05-27T07:22:30.797609+0000","last_change":"2024-05-27T07:21:48.602786+0000","last_active":"2024-05-27T07:22:30.797609+0000","last_peered":"2024-05-27T07:22:30.797609+0000","last_clean":"2024-05-27T07:22:30.797609+0000","last_became_active":"2024-05-27T07:21:48.602476+0000","last_became_peered":"2024-05-27T07:21:48.602476+0000","last_unstale":"2024-05-27T07:22:30.797609+0000","last_undegraded":"2024-05-27T07:22:30.797609+0000","last_fullsized":"2024-05-27T07:22:30.797609+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:21:47.565641+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:21:47.565641+0000","last_clean_scrub_stamp":"2024-05-27T07:21:47.565641+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T13:22:58.583230+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,11],"acting":[7,1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":24,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-05-27T07:22:30.801096+0000","last_change":"2024-05-27T07:21:59.729595+0000","last_active":"2024-05-27T07:22:30.801096+0000","last_peered":"2024-05-27T07:22:30.801096+0000","last_clean":"2024-05-27T07:22:30.801096+0000","last_became_active":"2024-05-27T07:21:59.729147+0000","last_became_peered":"2024-05-27T07:21:59.729147+0000","last_unstale":"2024-05-27T07:22:30.801096+0000","last_undegraded":"2024-05-27T07:22:30.801096+0000","last_fullsized":"2024-05-27T07:22:30.801096+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:21:58.698952+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:21:58.698952+0000","last_clean_scrub_stamp":"2024-05-27T07:21:58.698952+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T13:31:28.579617+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,9,1,4],"acting":[0,9,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.d","version":"88'3","reported_seq":43,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-05-27T07:22:55.889855+0000","last_change":"2024-05-27T07:21:45.569287+0000","last_active":"2024-05-27T07:22:55.889855+0000","last_peered":"2024-05-27T07:22:55.889855+0000","last_clean":"2024-05-27T07:22:55.889855+0000","last_became_active":"2024-05-27T07:21:45.568935+0000","last_became_peered":"2024-05-27T07:21:45.568935+0000","last_unstale":"2024-05-27T07:22:55.889855+0000","last_undegraded":"2024-05-27T07:22:55.889855+0000","last_fullsized":"2024-05-27T07:22:55.889855+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:21:44.543638+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:21:44.543638+0000","last_clean_scrub_stamp":"2024-05-27T07:21:44.543638+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T07:27:14.674135+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,10,3],"acting":[5,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":32,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-05-27T07:22:30.796957+0000","last_change":"2024-05-27T07:21:48.597718+0000","last_active":"2024-05-27T07:22:30.796957+0000","last_peered":"2024-05-27T07:22:30.796957+0000","last_clean":"2024-05-27T07:22:30.796957+0000","last_became_active":"2024-05-27T07:21:48.597379+0000","last_became_peered":"2024-05-27T07:21:48.597379+0000","last_unstale":"2024-05-27T07:22:30.796957+0000","last_undegraded":"2024-05-27T07:22:30.796957+0000","last_fullsized":"2024-05-27T07:22:30.796957+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:21:47.565641+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:21:47.565641+0000","last_clean_scrub_stamp":"2024-05-27T07:21:47.565641+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T08:48:41.658364+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,5],"acting":[7,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":24,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-05-27T07:22:30.796397+0000","last_change":"2024-05-27T07:21:59.730191+0000","last_active":"2024-05-27T07:22:30.796397+0000","last_peered":"2024-05-27T07:22:30.796397+0000","last_clean":"2024-05-27T07:22:30.796397+0000","last_became_active":"2024-05-27T07:21:59.729787+0000","last_became_peered":"2024-05-27T07:21:59.729787+0000","last_unstale":"2024-05-27T07:22:30.796397+0000","last_undegraded":"2024-05-27T07:22:30.796397+0000","last_fullsized":"2024-05-27T07:22:30.796397+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:21:58.698952+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:21:58.698952+0000","last_clean_scrub_stamp":"2024-05-27T07:21:58.698952+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T13:22:38.961246+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7,1],"acting":[5,2,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.e","version":"88'5","reported_seq":44,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-05-27T07:22:55.887608+0000","last_change":"2024-05-27T07:21:45.559219+0000","last_active":"2024-05-27T07:22:55.887608+0000","last_peered":"2024-05-27T07:22:55.887608+0000","last_clean":"2024-05-27T07:22:55.887608+0000","last_became_active":"2024-05-27T07:21:45.558848+0000","last_became_peered":"2024-05-27T07:21:45.558848+0000","last_unstale":"2024-05-27T07:22:55.887608+0000","last_undegraded":"2024-05-27T07:22:55.887608+0000","last_fullsized":"2024-05-27T07:22:55.887608+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:21:44.543638+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:21:44.543638+0000","last_clean_scrub_stamp":"2024-05-27T07:21:44.543638+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T15:03:55.632162+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":32,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-05-27T07:22:30.797617+0000","last_change":"2024-05-27T07:21:48.596349+0000","last_active":"2024-05-27T07:22:30.797617+0000","last_peered":"2024-05-27T07:22:30.797617+0000","last_clean":"2024-05-27T07:22:30.797617+0000","last_became_active":"2024-05-27T07:21:48.595830+0000","last_became_peered":"2024-05-27T07:21:48.595830+0000","last_unstale":"2024-05-27T07:22:30.797617+0000","last_undegraded":"2024-05-27T07:22:30.797617+0000","last_fullsized":"2024-05-27T07:22:30.797617+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:21:47.565641+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:21:47.565641+0000","last_clean_scrub_stamp":"2024-05-27T07:21:47.565641+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T13:33:11.431988+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,8],"acting":[7,11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.20","version":"0'0","reported_seq":34,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-05-27T07:22:30.793403+0000","last_change":"2024-05-27T07:21:48.598504+0000","last_active":"2024-05-27T07:22:30.793403+0000","last_peered":"2024-05-27T07:22:30.793403+0000","last_clean":"2024-05-27T07:22:30.793403+0000","last_became_active":"2024-05-27T07:21:48.598103+0000","last_became_peered":"2024-05-27T07:21:48.598103+0000","last_unstale":"2024-05-27T07:22:30.793403+0000","last_undegraded":"2024-05-27T07:22:30.793403+0000","last_fullsized":"2024-05-27T07:22:30.793403+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:21:47.565641+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:21:47.565641+0000","last_clean_scrub_stamp":"2024-05-27T07:21:47.565641+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T10:34:02.743735+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,5],"acting":[11,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.21","version":"0'0","reported_seq":34,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-05-27T07:22:30.801101+0000","last_change":"2024-05-27T07:21:48.593379+0000","last_active":"2024-05-27T07:22:30.801101+0000","last_peered":"2024-05-27T07:22:30.801101+0000","last_clean":"2024-05-27T07:22:30.801101+0000","last_became_active":"2024-05-27T07:21:48.592995+0000","last_became_peered":"2024-05-27T07:21:48.592995+0000","last_unstale":"2024-05-27T07:22:30.801101+0000","last_undegraded":"2024-05-27T07:22:30.801101+0000","last_fullsized":"2024-05-27T07:22:30.801101+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:21:47.565641+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:21:47.565641+0000","last_clean_scrub_stamp":"2024-05-27T07:21:47.565641+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T19:01:56.288235+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,4],"acting":[2,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"0'0","reported_seq":34,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-05-27T07:22:30.800545+0000","last_change":"2024-05-27T07:21:48.592837+0000","last_active":"2024-05-27T07:22:30.800545+0000","last_peered":"2024-05-27T07:22:30.800545+0000","last_clean":"2024-05-27T07:22:30.800545+0000","last_became_active":"2024-05-27T07:21:48.592459+0000","last_became_peered":"2024-05-27T07:21:48.592459+0000","last_unstale":"2024-05-27T07:22:30.800545+0000","last_undegraded":"2024-05-27T07:22:30.800545+0000","last_fullsized":"2024-05-27T07:22:30.800545+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:21:47.565641+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:21:47.565641+0000","last_clean_scrub_stamp":"2024-05-27T07:21:47.565641+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T14:00:19.490348+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.23","version":"0'0","reported_seq":34,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-05-27T07:22:30.801437+0000","last_change":"2024-05-27T07:21:48.583139+0000","last_active":"2024-05-27T07:22:30.801437+0000","last_peered":"2024-05-27T07:22:30.801437+0000","last_clean":"2024-05-27T07:22:30.801437+0000","last_became_active":"2024-05-27T07:21:48.582822+0000","last_became_peered":"2024-05-27T07:21:48.582822+0000","last_unstale":"2024-05-27T07:22:30.801437+0000","last_undegraded":"2024-05-27T07:22:30.801437+0000","last_fullsized":"2024-05-27T07:22:30.801437+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:21:47.565641+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:21:47.565641+0000","last_clean_scrub_stamp":"2024-05-27T07:21:47.565641+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T16:30:03.454035+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,4],"acting":[0,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"0'0","reported_seq":34,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-05-27T07:22:31.626460+0000","last_change":"2024-05-27T07:21:48.588906+0000","last_active":"2024-05-27T07:22:31.626460+0000","last_peered":"2024-05-27T07:22:31.626460+0000","last_clean":"2024-05-27T07:22:31.626460+0000","last_became_active":"2024-05-27T07:21:48.588555+0000","last_became_peered":"2024-05-27T07:21:48.588555+0000","last_unstale":"2024-05-27T07:22:31.626460+0000","last_undegraded":"2024-05-27T07:22:31.626460+0000","last_fullsized":"2024-05-27T07:22:31.626460+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:21:47.565641+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:21:47.565641+0000","last_clean_scrub_stamp":"2024-05-27T07:21:47.565641+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T19:04:23.006852+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"0'0","reported_seq":34,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-05-27T07:22:30.793772+0000","last_change":"2024-05-27T07:21:48.590271+0000","last_active":"2024-05-27T07:22:30.793772+0000","last_peered":"2024-05-27T07:22:30.793772+0000","last_clean":"2024-05-27T07:22:30.793772+0000","last_became_active":"2024-05-27T07:21:48.589898+0000","last_became_peered":"2024-05-27T07:21:48.589898+0000","last_unstale":"2024-05-27T07:22:30.793772+0000","last_undegraded":"2024-05-27T07:22:30.793772+0000","last_fullsized":"2024-05-27T07:22:30.793772+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:21:47.565641+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:21:47.565641+0000","last_clean_scrub_stamp":"2024-05-27T07:21:47.565641+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T11:57:43.238211+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,5],"acting":[11,8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.26","version":"0'0","reported_seq":34,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-05-27T07:22:30.800884+0000","last_change":"2024-05-27T07:21:48.597321+0000","last_active":"2024-05-27T07:22:30.800884+0000","last_peered":"2024-05-27T07:22:30.800884+0000","last_clean":"2024-05-27T07:22:30.800884+0000","last_became_active":"2024-05-27T07:21:48.596953+0000","last_became_peered":"2024-05-27T07:21:48.596953+0000","last_unstale":"2024-05-27T07:22:30.800884+0000","last_undegraded":"2024-05-27T07:22:30.800884+0000","last_fullsized":"2024-05-27T07:22:30.800884+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:21:47.565641+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:21:47.565641+0000","last_clean_scrub_stamp":"2024-05-27T07:21:47.565641+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T16:21:17.645098+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,8,1],"acting":[2,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.27","version":"0'0","reported_seq":34,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-05-27T07:22:30.855978+0000","last_change":"2024-05-27T07:21:48.593717+0000","last_active":"2024-05-27T07:22:30.855978+0000","last_peered":"2024-05-27T07:22:30.855978+0000","last_clean":"2024-05-27T07:22:30.855978+0000","last_became_active":"2024-05-27T07:21:48.593356+0000","last_became_peered":"2024-05-27T07:21:48.593356+0000","last_unstale":"2024-05-27T07:22:30.855978+0000","last_undegraded":"2024-05-27T07:22:30.855978+0000","last_fullsized":"2024-05-27T07:22:30.855978+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:21:47.565641+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:21:47.565641+0000","last_clean_scrub_stamp":"2024-05-27T07:21:47.565641+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T17:06:27.341627+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"0'0","reported_seq":34,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-05-27T07:22:30.796577+0000","last_change":"2024-05-27T07:21:48.592360+0000","last_active":"2024-05-27T07:22:30.796577+0000","last_peered":"2024-05-27T07:22:30.796577+0000","last_clean":"2024-05-27T07:22:30.796577+0000","last_became_active":"2024-05-27T07:21:48.591650+0000","last_became_peered":"2024-05-27T07:21:48.591650+0000","last_unstale":"2024-05-27T07:22:30.796577+0000","last_undegraded":"2024-05-27T07:22:30.796577+0000","last_fullsized":"2024-05-27T07:22:30.796577+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:21:47.565641+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:21:47.565641+0000","last_clean_scrub_stamp":"2024-05-27T07:21:47.565641+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T17:32:19.408787+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,3],"acting":[5,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"0'0","reported_seq":34,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-05-27T07:22:30.856019+0000","last_change":"2024-05-27T07:21:48.587319+0000","last_active":"2024-05-27T07:22:30.856019+0000","last_peered":"2024-05-27T07:22:30.856019+0000","last_clean":"2024-05-27T07:22:30.856019+0000","last_became_active":"2024-05-27T07:21:48.586924+0000","last_became_peered":"2024-05-27T07:21:48.586924+0000","last_unstale":"2024-05-27T07:22:30.856019+0000","last_undegraded":"2024-05-27T07:22:30.856019+0000","last_fullsized":"2024-05-27T07:22:30.856019+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:21:47.565641+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:21:47.565641+0000","last_clean_scrub_stamp":"2024-05-27T07:21:47.565641+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T09:28:48.129037+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,11],"acting":[3,2,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2a","version":"0'0","reported_seq":34,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-05-27T07:22:31.580004+0000","last_change":"2024-05-27T07:21:48.604018+0000","last_active":"2024-05-27T07:22:31.580004+0000","last_peered":"2024-05-27T07:22:31.580004+0000","last_clean":"2024-05-27T07:22:31.580004+0000","last_became_active":"2024-05-27T07:21:48.603368+0000","last_became_peered":"2024-05-27T07:21:48.603368+0000","last_unstale":"2024-05-27T07:22:31.580004+0000","last_undegraded":"2024-05-27T07:22:31.580004+0000","last_fullsized":"2024-05-27T07:22:31.580004+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:21:47.565641+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:21:47.565641+0000","last_clean_scrub_stamp":"2024-05-27T07:21:47.565641+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T09:49:42.440480+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2b","version":"0'0","reported_seq":34,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-05-27T07:22:31.626316+0000","last_change":"2024-05-27T07:21:48.589434+0000","last_active":"2024-05-27T07:22:31.626316+0000","last_peered":"2024-05-27T07:22:31.626316+0000","last_clean":"2024-05-27T07:22:31.626316+0000","last_became_active":"2024-05-27T07:21:48.589102+0000","last_became_peered":"2024-05-27T07:21:48.589102+0000","last_unstale":"2024-05-27T07:22:31.626316+0000","last_undegraded":"2024-05-27T07:22:31.626316+0000","last_fullsized":"2024-05-27T07:22:31.626316+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:21:47.565641+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:21:47.565641+0000","last_clean_scrub_stamp":"2024-05-27T07:21:47.565641+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T18:28:52.955138+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2c","version":"0'0","reported_seq":40,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-05-27T07:22:30.802433+0000","last_change":"2024-05-27T07:22:30.802433+0000","last_active":"2024-05-27T07:22:30.802433+0000","last_peered":"2024-05-27T07:22:30.802433+0000","last_clean":"2024-05-27T07:22:30.802433+0000","last_became_active":"2024-05-27T07:22:30.801881+0000","last_became_peered":"2024-05-27T07:22:30.801881+0000","last_unstale":"2024-05-27T07:22:30.802433+0000","last_undegraded":"2024-05-27T07:22:30.802433+0000","last_fullsized":"2024-05-27T07:22:30.802433+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:21:47.565641+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:21:47.565641+0000","last_clean_scrub_stamp":"2024-05-27T07:21:47.565641+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T11:59:17.916052+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,9,8],"acting":[11,9,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":234,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":32768,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":128,"acting":128,"num_store_stats":12},{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":192,"acting":192,"num_store_stats":12},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":81,"num_read_kb":69,"num_write":123,"num_write_kb":2150,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":2951840,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":376832,"data_stored":4132576,"data_compressed":249874,"data_compressed_allocated":315392,"data_compressed_original":4096000,"omap_allocated":0,"internal_metadata":0},"log_size":92,"ondisk_log_size":92,"up":3,"acting":3,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":63364,"num_objects":98,"num_object_clones":0,"num_object_copies":294,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":98,"num_whiteouts":0,"num_read":269,"num_read_kb":208,"num_write":246,"num_write_kb":134,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":22,"num_keys_recovered":0,"num_objects_omap":66,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":552960,"data_stored":190092,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":211,"ondisk_log_size":211,"up":96,"acting":96,"num_store_stats":12}],"osd_stats":[{"osd":11,"up_from":71,"seq":304942678050,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32128,"kb_used_data":952,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713024,"statfs":{"total":95995035648,"available":95962136576,"internally_reserved":0,"allocated":974848,"data_stored":500494,"data_compressed":26453,"data_compressed_allocated":36864,"data_compressed_original":73728,"omap_allocated":1596,"internal_metadata":31914436},"hb_peers":[0,1,2,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":65,"seq":279172874278,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28104,"kb_used_data":1088,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717048,"statfs":{"total":95995035648,"available":95966257152,"internally_reserved":0,"allocated":1114112,"data_stored":1143839,"data_compressed":62613,"data_compressed_allocated":81920,"data_compressed_original":663552,"omap_allocated":1600,"internal_metadata":27654592},"hb_peers":[0,1,2,3,4,5,6,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":60,"seq":257698037804,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36312,"kb_used_data":976,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708840,"statfs":{"total":95995035648,"available":95957852160,"internally_reserved":0,"allocated":999424,"data_stored":504498,"data_compressed":26453,"data_compressed_allocated":36864,"data_compressed_original":73728,"omap_allocated":1600,"internal_metadata":36174272},"hb_peers":[0,1,2,3,4,5,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":54,"seq":231928234033,"num_pgs":31,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27980,"kb_used_data":964,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717172,"statfs":{"total":95995035648,"available":95966384128,"internally_reserved":0,"allocated":987136,"data_stored":500758,"data_compressed":26453,"data_compressed_allocated":36864,"data_compressed_original":73728,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":48,"seq":206158430261,"num_pgs":37,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32196,"kb_used_data":1020,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712956,"statfs":{"total":95995035648,"available":95962066944,"internally_reserved":0,"allocated":1044480,"data_stored":1097668,"data_compressed":62613,"data_compressed_allocated":81920,"data_compressed_original":663552,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":43,"seq":184683593787,"num_pgs":32,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27964,"kb_used_data":948,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717188,"statfs":{"total":95995035648,"available":95966400512,"internally_reserved":0,"allocated":970752,"data_stored":495906,"data_compressed":26453,"data_compressed_allocated":36864,"data_compressed_original":73728,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574932,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32184,"kb_used_data":1008,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712968,"statfs":{"total":95995035648,"available":95962079232,"internally_reserved":0,"allocated":1032192,"data_stored":553096,"data_compressed":26453,"data_compressed_allocated":36864,"data_compressed_original":73728,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738457,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27988,"kb_used_data":972,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717164,"statfs":{"total":95995035648,"available":95966375936,"internally_reserved":0,"allocated":995328,"data_stored":497362,"data_compressed":26453,"data_compressed_allocated":36864,"data_compressed_original":73728,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411407,"num_pgs":36,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32192,"kb_used_data":1016,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712960,"statfs":{"total":95995035648,"available":95962071040,"internally_reserved":0,"allocated":1040384,"data_stored":549330,"data_compressed":26453,"data_compressed_allocated":36864,"data_compressed_original":73728,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":26,"seq":111669149771,"num_pgs":40,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36296,"kb_used_data":960,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708856,"statfs":{"total":95995035648,"available":95957868544,"internally_reserved":0,"allocated":983040,"data_stored":495614,"data_compressed":26453,"data_compressed_allocated":36864,"data_compressed_original":73728,"omap_allocated":1593,"internal_metadata":36174279},"hb_peers":[0,1,2,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":32,"seq":137438953541,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27980,"kb_used_data":964,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717172,"statfs":{"total":95995035648,"available":95966384128,"internally_reserved":0,"allocated":987136,"data_stored":500483,"data_compressed":26453,"data_compressed_allocated":36864,"data_compressed_original":73728,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":37,"seq":158913790016,"num_pgs":44,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28044,"kb_used_data":1028,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717108,"statfs":{"total":95995035648,"available":95966318592,"internally_reserved":0,"allocated":1052672,"data_stored":1092859,"data_compressed":62613,"data_compressed_allocated":81920,"data_compressed_original":663552,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":34537,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":34537,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":36160,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":36160,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":36160,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":36160,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":36160,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":2178,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":77824,"data_stored":57912,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":90112,"data_stored":58411,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":430,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1203,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":3211,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":722,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":3924,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1478,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":1122,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":110592,"data_stored":58287,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1214,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":8192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":8192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-05-27T07:23:23.238 DEBUG:teuthology.orchestra.run.smithi052:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:38f9d900c83c63799fbdbe61acc9a11b0d3554a6 shell --fsid 73df2636-1bf8-11ef-bc9b-c7b262605968 -- ceph pg dump --format=json 2024-05-27T07:23:23.352 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:23:22 smithi173 ceph-mon[37000]: pgmap v344: 129 pgs: 129 active+clean; 639 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 7.3 KiB/s rd, 2.9 KiB/s wr, 12 op/s 2024-05-27T07:23:23.492 INFO:teuthology.orchestra.run.smithi052.stderr:Inferring config /var/lib/ceph/73df2636-1bf8-11ef-bc9b-c7b262605968/mon.a/config 2024-05-27T07:23:24.229 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:23:23 smithi052 ceph-mon[31955]: from='client.14853 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-05-27T07:23:24.352 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:23:23 smithi173 ceph-mon[37000]: from='client.14853 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-05-27T07:23:24.440 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:23:23 smithi105 ceph-mon[35977]: from='client.14853 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-05-27T07:23:25.069 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:23:24 smithi052 ceph-mon[31955]: pgmap v345: 129 pgs: 129 active+clean; 639 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 2.7 KiB/s wr, 11 op/s 2024-05-27T07:23:25.317 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:23:25.327+0000 7f7c3b78f700 1 -- 172.21.15.52:0/3740947996 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f7c34105610 msgr2=0x7f7c34102cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:23:25.318 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:23:25.327+0000 7f7c3b78f700 1 --2- 172.21.15.52:0/3740947996 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f7c34105610 0x7f7c34102cc0 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7f7c24009990 tx=0x7f7c24009ca0 comp rx=0 tx=0).stop 2024-05-27T07:23:25.318 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:23:25.327+0000 7f7c3b78f700 1 -- 172.21.15.52:0/3740947996 shutdown_connections 2024-05-27T07:23:25.318 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:23:25.327+0000 7f7c3b78f700 1 --2- 172.21.15.52:0/3740947996 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f7c34103350 0x7f7c34112fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:23:25.318 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:23:25.327+0000 7f7c3b78f700 1 --2- 172.21.15.52:0/3740947996 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f7c34105200 0x7f7c34102780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:23:25.318 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:23:25.327+0000 7f7c3b78f700 1 --2- 172.21.15.52:0/3740947996 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f7c34105610 0x7f7c34102cc0 unknown :-1 s=CLOSED pgs=243 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:23:25.318 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:23:25.327+0000 7f7c3b78f700 1 -- 172.21.15.52:0/3740947996 >> 172.21.15.52:0/3740947996 conn(0x7f7c340fe510 msgr2=0x7f7c34100960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:23:25.318 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:23:25.327+0000 7f7c3b78f700 1 -- 172.21.15.52:0/3740947996 shutdown_connections 2024-05-27T07:23:25.318 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:23:25.327+0000 7f7c3b78f700 1 -- 172.21.15.52:0/3740947996 wait complete. 2024-05-27T07:23:25.320 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:23:25.328+0000 7f7c3b78f700 1 Processor -- start 2024-05-27T07:23:25.320 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:23:25.328+0000 7f7c3b78f700 1 -- start start 2024-05-27T07:23:25.320 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:23:25.328+0000 7f7c3b78f700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f7c34103350 0x7f7c3419cfc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:23:25.320 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:23:25.328+0000 7f7c3b78f700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f7c34105200 0x7f7c3419d560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:23:25.320 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:23:25.329+0000 7f7c3b78f700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f7c34105610 0x7f7c341a17f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:23:25.320 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:23:25.329+0000 7f7c3b78f700 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x7f7c34116650 con 0x7f7c34103350 2024-05-27T07:23:25.320 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:23:25.329+0000 7f7c3b78f700 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f7c341164d0 con 0x7f7c34105610 2024-05-27T07:23:25.320 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:23:25.329+0000 7f7c3b78f700 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f7c341167d0 con 0x7f7c34105200 2024-05-27T07:23:25.320 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:23:25.329+0000 7f7c3952b700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f7c34103350 0x7f7c3419cfc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:23:25.321 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:23:25.329+0000 7f7c3952b700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f7c34103350 0x7f7c3419cfc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.52:39748/0 (socket says 172.21.15.52:39748) 2024-05-27T07:23:25.321 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:23:25.329+0000 7f7c3952b700 1 -- 172.21.15.52:0/2837326765 learned_addr learned my addr 172.21.15.52:0/2837326765 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-05-27T07:23:25.321 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:23:25.329+0000 7f7c38d2a700 1 --2- 172.21.15.52:0/2837326765 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f7c34105200 0x7f7c3419d560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:23:25.321 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:23:25.329+0000 7f7c39d2c700 1 --2- 172.21.15.52:0/2837326765 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f7c34105610 0x7f7c341a17f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:23:25.321 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:23:25.329+0000 7f7c3952b700 1 -- 172.21.15.52:0/2837326765 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f7c34105200 msgr2=0x7f7c3419d560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:23:25.321 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:23:25.329+0000 7f7c3952b700 1 --2- 172.21.15.52:0/2837326765 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f7c34105200 0x7f7c3419d560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:23:25.321 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:23:25.329+0000 7f7c3952b700 1 -- 172.21.15.52:0/2837326765 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f7c34105610 msgr2=0x7f7c341a17f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:23:25.321 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:23:25.329+0000 7f7c3952b700 1 --2- 172.21.15.52:0/2837326765 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f7c34105610 0x7f7c341a17f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:23:25.321 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:23:25.329+0000 7f7c3952b700 1 -- 172.21.15.52:0/2837326765 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7c341a1e90 con 0x7f7c34103350 2024-05-27T07:23:25.321 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:23:25.329+0000 7f7c3952b700 1 --2- 172.21.15.52:0/2837326765 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f7c34103350 0x7f7c3419cfc0 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7f7c3000d980 tx=0x7f7c3000dc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:23:25.321 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:23:25.329+0000 7f7c2a7fc700 1 -- 172.21.15.52:0/2837326765 <== mon.0 v2:172.21.15.52:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7c30017030 con 0x7f7c34103350 2024-05-27T07:23:25.321 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:23:25.330+0000 7f7c2a7fc700 1 -- 172.21.15.52:0/2837326765 <== mon.0 v2:172.21.15.52:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f7c300044d0 con 0x7f7c34103350 2024-05-27T07:23:25.321 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:23:25.330+0000 7f7c3b78f700 1 -- 172.21.15.52:0/2837326765 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7c341a2180 con 0x7f7c34103350 2024-05-27T07:23:25.322 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:23:25.330+0000 7f7c2a7fc700 1 -- 172.21.15.52:0/2837326765 <== mon.0 v2:172.21.15.52:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7c30003970 con 0x7f7c34103350 2024-05-27T07:23:25.322 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:23:25.330+0000 7f7c3b78f700 1 -- 172.21.15.52:0/2837326765 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7c341a9aa0 con 0x7f7c34103350 2024-05-27T07:23:25.322 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:23:25.331+0000 7f7c3b78f700 1 -- 172.21.15.52:0/2837326765 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7c340512b0 con 0x7f7c34103350 2024-05-27T07:23:25.322 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:23:25.333+0000 7f7c2a7fc700 1 -- 172.21.15.52:0/2837326765 <== mon.0 v2:172.21.15.52:3300/0 4 ==== mgrmap(e 17) ==== 149703+0+0 (secure 0 0 0) 0x7f7c3001a070 con 0x7f7c34103350 2024-05-27T07:23:25.323 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:23:25.333+0000 7f7c2a7fc700 1 --2- 172.21.15.52:0/2837326765 >> [v2:172.21.15.52:6800/3513075805,v1:172.21.15.52:6801/3513075805] conn(0x7f7c200b2990 0x7f7c200b4ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:23:25.323 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:23:25.334+0000 7f7c38d2a700 1 --2- 172.21.15.52:0/2837326765 >> [v2:172.21.15.52:6800/3513075805,v1:172.21.15.52:6801/3513075805] conn(0x7f7c200b2990 0x7f7c200b4ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:23:25.323 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:23:25.334+0000 7f7c2a7fc700 1 -- 172.21.15.52:0/2837326765 <== mon.0 v2:172.21.15.52:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 9092+0+0 (secure 0 0 0) 0x7f7c300e22c0 con 0x7f7c34103350 2024-05-27T07:23:25.324 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:23:25.334+0000 7f7c38d2a700 1 --2- 172.21.15.52:0/2837326765 >> [v2:172.21.15.52:6800/3513075805,v1:172.21.15.52:6801/3513075805] conn(0x7f7c200b2990 0x7f7c200b4ea0 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7f7c2400b5c0 tx=0x7f7c24005960 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:23:25.327 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:23:25.337+0000 7f7c2a7fc700 1 -- 172.21.15.52:0/2837326765 <== mon.0 v2:172.21.15.52:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191963 (secure 0 0 0) 0x7f7c300a9ba0 con 0x7f7c34103350 2024-05-27T07:23:25.352 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:23:24 smithi173 ceph-mon[37000]: pgmap v345: 129 pgs: 129 active+clean; 639 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 2.7 KiB/s wr, 11 op/s 2024-05-27T07:23:25.440 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:23:24 smithi105 ceph-mon[35977]: pgmap v345: 129 pgs: 129 active+clean; 639 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 2.7 KiB/s wr, 11 op/s 2024-05-27T07:23:25.542 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:23:25.552+0000 7f7c3b78f700 1 -- 172.21.15.52:0/2837326765 --> [v2:172.21.15.52:6800/3513075805,v1:172.21.15.52:6801/3513075805] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f7c341125b0 con 0x7f7c200b2990 2024-05-27T07:23:25.551 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:23:25.561+0000 7f7c2a7fc700 1 -- 172.21.15.52:0/2837326765 <== mgr.14152 v2:172.21.15.52:6800/3513075805 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+340640 (secure 0 0 0) 0x7f7c2407b2b0 con 0x7f7c200b2990 2024-05-27T07:23:25.557 INFO:teuthology.orchestra.run.smithi052.stdout: 2024-05-27T07:23:25.560 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:23:25.565+0000 7f7c3b78f700 1 -- 172.21.15.52:0/2837326765 >> [v2:172.21.15.52:6800/3513075805,v1:172.21.15.52:6801/3513075805] conn(0x7f7c200b2990 msgr2=0x7f7c200b4ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:23:25.560 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:23:25.565+0000 7f7c3b78f700 1 --2- 172.21.15.52:0/2837326765 >> [v2:172.21.15.52:6800/3513075805,v1:172.21.15.52:6801/3513075805] conn(0x7f7c200b2990 0x7f7c200b4ea0 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7f7c2400b5c0 tx=0x7f7c24005960 comp rx=0 tx=0).stop 2024-05-27T07:23:25.560 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:23:25.565+0000 7f7c3b78f700 1 -- 172.21.15.52:0/2837326765 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f7c34103350 msgr2=0x7f7c3419cfc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:23:25.560 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:23:25.565+0000 7f7c3b78f700 1 --2- 172.21.15.52:0/2837326765 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f7c34103350 0x7f7c3419cfc0 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7f7c3000d980 tx=0x7f7c3000dc90 comp rx=0 tx=0).stop 2024-05-27T07:23:25.560 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:23:25.565+0000 7f7c3b78f700 1 -- 172.21.15.52:0/2837326765 shutdown_connections 2024-05-27T07:23:25.560 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:23:25.565+0000 7f7c3b78f700 1 --2- 172.21.15.52:0/2837326765 >> [v2:172.21.15.52:6800/3513075805,v1:172.21.15.52:6801/3513075805] conn(0x7f7c200b2990 0x7f7c200b4ea0 unknown :-1 s=CLOSED pgs=171 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:23:25.560 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:23:25.565+0000 7f7c3b78f700 1 --2- 172.21.15.52:0/2837326765 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f7c34105610 0x7f7c341a17f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:23:25.560 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:23:25.565+0000 7f7c3b78f700 1 --2- 172.21.15.52:0/2837326765 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f7c34103350 0x7f7c3419cfc0 unknown :-1 s=CLOSED pgs=244 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:23:25.560 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:23:25.565+0000 7f7c3b78f700 1 --2- 172.21.15.52:0/2837326765 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f7c34105200 0x7f7c3419d560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:23:25.560 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:23:25.565+0000 7f7c3b78f700 1 -- 172.21.15.52:0/2837326765 >> 172.21.15.52:0/2837326765 conn(0x7f7c340fe510 msgr2=0x7f7c34111ea0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:23:25.560 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:23:25.566+0000 7f7c3b78f700 1 -- 172.21.15.52:0/2837326765 shutdown_connections 2024-05-27T07:23:25.561 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:23:25.566+0000 7f7c3b78f700 1 -- 172.21.15.52:0/2837326765 wait complete. 2024-05-27T07:23:25.561 INFO:teuthology.orchestra.run.smithi052.stderr:dumped all 2024-05-27T07:23:26.360 INFO:teuthology.orchestra.run.smithi052.stdout:{"pg_ready":true,"pg_map":{"version":345,"stamp":"2024-05-27T07:23:23.665629+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":653966,"num_objects":102,"num_object_clones":0,"num_object_copies":308,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":102,"num_whiteouts":0,"num_read":360,"num_read_kb":277,"num_write":371,"num_write_kb":2286,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":2951862,"num_keys_recovered":0,"num_objects_omap":66,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":305,"ondisk_log_size":305,"up":419,"acting":419,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":419,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":369380,"kb_used_data":11908,"kb_used_omap":18,"kb_used_meta":357357,"kb_avail":1124572444,"statfs":{"total":1151940427776,"available":1151562182656,"internally_reserved":0,"allocated":12193792,"data_stored":7942978,"data_compressed":425916,"data_compressed_allocated":577536,"data_compressed_original":2654208,"omap_allocated":19143,"internal_metadata":365933881},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":26953,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":113,"num_read_kb":78,"num_write":20,"num_write_kb":33,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003492"},"pg_stats":[{"pgid":"3.3f","version":"0'0","reported_seq":34,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-05-27T07:22:30.794972+0000","last_change":"2024-05-27T07:21:48.595870+0000","last_active":"2024-05-27T07:22:30.794972+0000","last_peered":"2024-05-27T07:22:30.794972+0000","last_clean":"2024-05-27T07:22:30.794972+0000","last_became_active":"2024-05-27T07:21:48.595020+0000","last_became_peered":"2024-05-27T07:21:48.595020+0000","last_unstale":"2024-05-27T07:22:30.794972+0000","last_undegraded":"2024-05-27T07:22:30.794972+0000","last_fullsized":"2024-05-27T07:22:30.794972+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:21:47.565641+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:21:47.565641+0000","last_clean_scrub_stamp":"2024-05-27T07:21:47.565641+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T16:43:08.920344+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.3e","version":"0'0","reported_seq":34,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-05-27T07:22:30.801613+0000","last_change":"2024-05-27T07:21:48.590237+0000","last_active":"2024-05-27T07:22:30.801613+0000","last_peered":"2024-05-27T07:22:30.801613+0000","last_clean":"2024-05-27T07:22:30.801613+0000","last_became_active":"2024-05-27T07:21:48.589876+0000","last_became_peered":"2024-05-27T07:21:48.589876+0000","last_unstale":"2024-05-27T07:22:30.801613+0000","last_undegraded":"2024-05-27T07:22:30.801613+0000","last_fullsized":"2024-05-27T07:22:30.801613+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:21:47.565641+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:21:47.565641+0000","last_clean_scrub_stamp":"2024-05-27T07:21:47.565641+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T16:23:08.881871+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,11,10],"acting":[0,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3d","version":"0'0","reported_seq":34,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-05-27T07:22:30.801204+0000","last_change":"2024-05-27T07:21:48.597744+0000","last_active":"2024-05-27T07:22:30.801204+0000","last_peered":"2024-05-27T07:22:30.801204+0000","last_clean":"2024-05-27T07:22:30.801204+0000","last_became_active":"2024-05-27T07:21:48.597410+0000","last_became_peered":"2024-05-27T07:21:48.597410+0000","last_unstale":"2024-05-27T07:22:30.801204+0000","last_undegraded":"2024-05-27T07:22:30.801204+0000","last_fullsized":"2024-05-27T07:22:30.801204+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:21:47.565641+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:21:47.565641+0000","last_clean_scrub_stamp":"2024-05-27T07:21:47.565641+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T19:06:49.643152+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"0'0","reported_seq":11,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-05-27T07:22:30.800824+0000","last_change":"2024-05-27T07:22:30.800824+0000","last_active":"2024-05-27T07:22:30.800824+0000","last_peered":"2024-05-27T07:22:30.800824+0000","last_clean":"2024-05-27T07:22:30.800824+0000","last_became_active":"2024-05-27T07:22:30.800415+0000","last_became_peered":"2024-05-27T07:22:30.800415+0000","last_unstale":"2024-05-27T07:22:30.800824+0000","last_undegraded":"2024-05-27T07:22:30.800824+0000","last_fullsized":"2024-05-27T07:22:30.800824+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:21:47.565641+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:21:47.565641+0000","last_clean_scrub_stamp":"2024-05-27T07:21:47.565641+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T11:06:10.451686+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,2],"acting":[9,8,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.3b","version":"0'0","reported_seq":34,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-05-27T07:22:30.799084+0000","last_change":"2024-05-27T07:21:48.583219+0000","last_active":"2024-05-27T07:22:30.799084+0000","last_peered":"2024-05-27T07:22:30.799084+0000","last_clean":"2024-05-27T07:22:30.799084+0000","last_became_active":"2024-05-27T07:21:48.582813+0000","last_became_peered":"2024-05-27T07:21:48.582813+0000","last_unstale":"2024-05-27T07:22:30.799084+0000","last_undegraded":"2024-05-27T07:22:30.799084+0000","last_fullsized":"2024-05-27T07:22:30.799084+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:21:47.565641+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:21:47.565641+0000","last_clean_scrub_stamp":"2024-05-27T07:21:47.565641+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T18:05:05.770965+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,3,6],"acting":[8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.3a","version":"0'0","reported_seq":34,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-05-27T07:22:31.416761+0000","last_change":"2024-05-27T07:21:48.592049+0000","last_active":"2024-05-27T07:22:31.416761+0000","last_peered":"2024-05-27T07:22:31.416761+0000","last_clean":"2024-05-27T07:22:31.416761+0000","last_became_active":"2024-05-27T07:21:48.591677+0000","last_became_peered":"2024-05-27T07:21:48.591677+0000","last_unstale":"2024-05-27T07:22:31.416761+0000","last_undegraded":"2024-05-27T07:22:31.416761+0000","last_fullsized":"2024-05-27T07:22:31.416761+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:21:47.565641+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:21:47.565641+0000","last_clean_scrub_stamp":"2024-05-27T07:21:47.565641+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T16:25:49.755893+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,4],"acting":[10,9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.39","version":"0'0","reported_seq":32,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-05-27T07:22:30.797598+0000","last_change":"2024-05-27T07:21:48.597876+0000","last_active":"2024-05-27T07:22:30.797598+0000","last_peered":"2024-05-27T07:22:30.797598+0000","last_clean":"2024-05-27T07:22:30.797598+0000","last_became_active":"2024-05-27T07:21:48.597566+0000","last_became_peered":"2024-05-27T07:21:48.597566+0000","last_unstale":"2024-05-27T07:22:30.797598+0000","last_undegraded":"2024-05-27T07:22:30.797598+0000","last_fullsized":"2024-05-27T07:22:30.797598+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:21:47.565641+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:21:47.565641+0000","last_clean_scrub_stamp":"2024-05-27T07:21:47.565641+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T09:30:43.438209+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,5],"acting":[7,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"0'0","reported_seq":34,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-05-27T07:22:30.855228+0000","last_change":"2024-05-27T07:21:48.593412+0000","last_active":"2024-05-27T07:22:30.855228+0000","last_peered":"2024-05-27T07:22:30.855228+0000","last_clean":"2024-05-27T07:22:30.855228+0000","last_became_active":"2024-05-27T07:21:48.593070+0000","last_became_peered":"2024-05-27T07:21:48.593070+0000","last_unstale":"2024-05-27T07:22:30.855228+0000","last_undegraded":"2024-05-27T07:22:30.855228+0000","last_fullsized":"2024-05-27T07:22:30.855228+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:21:47.565641+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:21:47.565641+0000","last_clean_scrub_stamp":"2024-05-27T07:21:47.565641+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T14:53:49.382796+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,10],"acting":[3,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"0'0","reported_seq":34,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-05-27T07:22:31.580336+0000","last_change":"2024-05-27T07:21:48.607244+0000","last_active":"2024-05-27T07:22:31.580336+0000","last_peered":"2024-05-27T07:22:31.580336+0000","last_clean":"2024-05-27T07:22:31.580336+0000","last_became_active":"2024-05-27T07:21:48.606238+0000","last_became_peered":"2024-05-27T07:21:48.606238+0000","last_unstale":"2024-05-27T07:22:31.580336+0000","last_undegraded":"2024-05-27T07:22:31.580336+0000","last_fullsized":"2024-05-27T07:22:31.580336+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:21:47.565641+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:21:47.565641+0000","last_clean_scrub_stamp":"2024-05-27T07:21:47.565641+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T12:36:33.142111+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,11],"acting":[1,5,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.36","version":"0'0","reported_seq":34,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-05-27T07:22:31.417314+0000","last_change":"2024-05-27T07:21:48.593275+0000","last_active":"2024-05-27T07:22:31.417314+0000","last_peered":"2024-05-27T07:22:31.417314+0000","last_clean":"2024-05-27T07:22:31.417314+0000","last_became_active":"2024-05-27T07:21:48.592890+0000","last_became_peered":"2024-05-27T07:21:48.592890+0000","last_unstale":"2024-05-27T07:22:31.417314+0000","last_undegraded":"2024-05-27T07:22:31.417314+0000","last_fullsized":"2024-05-27T07:22:31.417314+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:21:47.565641+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:21:47.565641+0000","last_clean_scrub_stamp":"2024-05-27T07:21:47.565641+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T13:47:20.827824+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,4],"acting":[10,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.35","version":"0'0","reported_seq":32,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-05-27T07:22:30.797332+0000","last_change":"2024-05-27T07:21:48.603424+0000","last_active":"2024-05-27T07:22:30.797332+0000","last_peered":"2024-05-27T07:22:30.797332+0000","last_clean":"2024-05-27T07:22:30.797332+0000","last_became_active":"2024-05-27T07:21:48.602962+0000","last_became_peered":"2024-05-27T07:21:48.602962+0000","last_unstale":"2024-05-27T07:22:30.797332+0000","last_undegraded":"2024-05-27T07:22:30.797332+0000","last_fullsized":"2024-05-27T07:22:30.797332+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:21:47.565641+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:21:47.565641+0000","last_clean_scrub_stamp":"2024-05-27T07:21:47.565641+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T19:19:42.063063+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,8,1],"acting":[7,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.34","version":"0'0","reported_seq":34,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-05-27T07:22:30.796472+0000","last_change":"2024-05-27T07:21:48.589323+0000","last_active":"2024-05-27T07:22:30.796472+0000","last_peered":"2024-05-27T07:22:30.796472+0000","last_clean":"2024-05-27T07:22:30.796472+0000","last_became_active":"2024-05-27T07:21:48.587435+0000","last_became_peered":"2024-05-27T07:21:48.587435+0000","last_unstale":"2024-05-27T07:22:30.796472+0000","last_undegraded":"2024-05-27T07:22:30.796472+0000","last_fullsized":"2024-05-27T07:22:30.796472+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:21:47.565641+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:21:47.565641+0000","last_clean_scrub_stamp":"2024-05-27T07:21:47.565641+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T13:34:44.340376+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,8],"acting":[4,10,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"0'0","reported_seq":34,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-05-27T07:22:31.626221+0000","last_change":"2024-05-27T07:21:48.593759+0000","last_active":"2024-05-27T07:22:31.626221+0000","last_peered":"2024-05-27T07:22:31.626221+0000","last_clean":"2024-05-27T07:22:31.626221+0000","last_became_active":"2024-05-27T07:21:48.593099+0000","last_became_peered":"2024-05-27T07:21:48.593099+0000","last_unstale":"2024-05-27T07:22:31.626221+0000","last_undegraded":"2024-05-27T07:22:31.626221+0000","last_fullsized":"2024-05-27T07:22:31.626221+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:21:47.565641+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:21:47.565641+0000","last_clean_scrub_stamp":"2024-05-27T07:21:47.565641+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T13:49:00.390792+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,5],"acting":[6,10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.32","version":"0'0","reported_seq":34,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-05-27T07:22:30.801091+0000","last_change":"2024-05-27T07:21:48.587486+0000","last_active":"2024-05-27T07:22:30.801091+0000","last_peered":"2024-05-27T07:22:30.801091+0000","last_clean":"2024-05-27T07:22:30.801091+0000","last_became_active":"2024-05-27T07:21:48.587159+0000","last_became_peered":"2024-05-27T07:21:48.587159+0000","last_unstale":"2024-05-27T07:22:30.801091+0000","last_undegraded":"2024-05-27T07:22:30.801091+0000","last_fullsized":"2024-05-27T07:22:30.801091+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:21:47.565641+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:21:47.565641+0000","last_clean_scrub_stamp":"2024-05-27T07:21:47.565641+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T15:34:11.908442+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,10],"acting":[0,8,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"0'0","reported_seq":34,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-05-27T07:22:30.801192+0000","last_change":"2024-05-27T07:21:48.596817+0000","last_active":"2024-05-27T07:22:30.801192+0000","last_peered":"2024-05-27T07:22:30.801192+0000","last_clean":"2024-05-27T07:22:30.801192+0000","last_became_active":"2024-05-27T07:21:48.596469+0000","last_became_peered":"2024-05-27T07:21:48.596469+0000","last_unstale":"2024-05-27T07:22:30.801192+0000","last_undegraded":"2024-05-27T07:22:30.801192+0000","last_fullsized":"2024-05-27T07:22:30.801192+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:21:47.565641+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:21:47.565641+0000","last_clean_scrub_stamp":"2024-05-27T07:21:47.565641+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T12:38:41.494060+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,2],"acting":[0,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.30","version":"0'0","reported_seq":34,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-05-27T07:22:31.579643+0000","last_change":"2024-05-27T07:21:48.604249+0000","last_active":"2024-05-27T07:22:31.579643+0000","last_peered":"2024-05-27T07:22:31.579643+0000","last_clean":"2024-05-27T07:22:31.579643+0000","last_became_active":"2024-05-27T07:21:48.603585+0000","last_became_peered":"2024-05-27T07:21:48.603585+0000","last_unstale":"2024-05-27T07:22:31.579643+0000","last_undegraded":"2024-05-27T07:22:31.579643+0000","last_fullsized":"2024-05-27T07:22:31.579643+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:21:47.565641+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:21:47.565641+0000","last_clean_scrub_stamp":"2024-05-27T07:21:47.565641+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T07:22:14.839824+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2f","version":"0'0","reported_seq":34,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-05-27T07:22:30.794834+0000","last_change":"2024-05-27T07:21:48.596035+0000","last_active":"2024-05-27T07:22:30.794834+0000","last_peered":"2024-05-27T07:22:30.794834+0000","last_clean":"2024-05-27T07:22:30.794834+0000","last_became_active":"2024-05-27T07:21:48.595509+0000","last_became_peered":"2024-05-27T07:21:48.595509+0000","last_unstale":"2024-05-27T07:22:30.794834+0000","last_undegraded":"2024-05-27T07:22:30.794834+0000","last_fullsized":"2024-05-27T07:22:30.794834+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:21:47.565641+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:21:47.565641+0000","last_clean_scrub_stamp":"2024-05-27T07:21:47.565641+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T13:23:03.734982+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.2e","version":"0'0","reported_seq":34,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-05-27T07:22:30.792717+0000","last_change":"2024-05-27T07:21:48.590002+0000","last_active":"2024-05-27T07:22:30.792717+0000","last_peered":"2024-05-27T07:22:30.792717+0000","last_clean":"2024-05-27T07:22:30.792717+0000","last_became_active":"2024-05-27T07:21:48.589608+0000","last_became_peered":"2024-05-27T07:21:48.589608+0000","last_unstale":"2024-05-27T07:22:30.792717+0000","last_undegraded":"2024-05-27T07:22:30.792717+0000","last_fullsized":"2024-05-27T07:22:30.792717+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:21:47.565641+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:21:47.565641+0000","last_clean_scrub_stamp":"2024-05-27T07:21:47.565641+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T09:33:39.174351+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,1,10],"acting":[9,1,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.2d","version":"0'0","reported_seq":34,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-05-27T07:22:30.801640+0000","last_change":"2024-05-27T07:21:48.594885+0000","last_active":"2024-05-27T07:22:30.801640+0000","last_peered":"2024-05-27T07:22:30.801640+0000","last_clean":"2024-05-27T07:22:30.801640+0000","last_became_active":"2024-05-27T07:21:48.594521+0000","last_became_peered":"2024-05-27T07:21:48.594521+0000","last_unstale":"2024-05-27T07:22:30.801640+0000","last_undegraded":"2024-05-27T07:22:30.801640+0000","last_fullsized":"2024-05-27T07:22:30.801640+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:21:47.565641+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:21:47.565641+0000","last_clean_scrub_stamp":"2024-05-27T07:21:47.565641+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T08:54:59.311938+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":31,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-05-27T07:22:30.857546+0000","last_change":"2024-05-27T07:22:30.857546+0000","last_active":"2024-05-27T07:22:30.857546+0000","last_peered":"2024-05-27T07:22:30.857546+0000","last_clean":"2024-05-27T07:22:30.857546+0000","last_became_active":"2024-05-27T07:22:30.856808+0000","last_became_peered":"2024-05-27T07:22:30.856808+0000","last_unstale":"2024-05-27T07:22:30.857546+0000","last_undegraded":"2024-05-27T07:22:30.857546+0000","last_fullsized":"2024-05-27T07:22:30.857546+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:21:58.698952+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:21:58.698952+0000","last_clean_scrub_stamp":"2024-05-27T07:21:58.698952+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T16:09:25.480864+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,8,0],"acting":[2,3,8,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1f","version":"88'13","reported_seq":78,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-05-27T07:23:18.926276+0000","last_change":"2024-05-27T07:21:45.566489+0000","last_active":"2024-05-27T07:23:18.926276+0000","last_peered":"2024-05-27T07:23:18.926276+0000","last_clean":"2024-05-27T07:23:18.926276+0000","last_became_active":"2024-05-27T07:21:45.566115+0000","last_became_peered":"2024-05-27T07:21:45.566115+0000","last_unstale":"2024-05-27T07:23:18.926276+0000","last_undegraded":"2024-05-27T07:23:18.926276+0000","last_fullsized":"2024-05-27T07:23:18.926276+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:21:44.543638+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:21:44.543638+0000","last_clean_scrub_stamp":"2024-05-27T07:21:44.543638+0000","objects_scrubbed":0,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T07:24:11.735014+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":25,"num_read_kb":25,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,8],"acting":[9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":34,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-05-27T07:22:30.855530+0000","last_change":"2024-05-27T07:21:48.588574+0000","last_active":"2024-05-27T07:22:30.855530+0000","last_peered":"2024-05-27T07:22:30.855530+0000","last_clean":"2024-05-27T07:22:30.855530+0000","last_became_active":"2024-05-27T07:21:48.588212+0000","last_became_peered":"2024-05-27T07:21:48.588212+0000","last_unstale":"2024-05-27T07:22:30.855530+0000","last_undegraded":"2024-05-27T07:22:30.855530+0000","last_fullsized":"2024-05-27T07:22:30.855530+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:21:47.565641+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:21:47.565641+0000","last_clean_scrub_stamp":"2024-05-27T07:21:47.565641+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T12:15:16.017635+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":24,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-05-27T07:22:30.795461+0000","last_change":"2024-05-27T07:21:59.734617+0000","last_active":"2024-05-27T07:22:30.795461+0000","last_peered":"2024-05-27T07:22:30.795461+0000","last_clean":"2024-05-27T07:22:30.795461+0000","last_became_active":"2024-05-27T07:21:59.734162+0000","last_became_peered":"2024-05-27T07:21:59.734162+0000","last_unstale":"2024-05-27T07:22:30.795461+0000","last_undegraded":"2024-05-27T07:22:30.795461+0000","last_fullsized":"2024-05-27T07:22:30.795461+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:21:58.698952+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:21:58.698952+0000","last_clean_scrub_stamp":"2024-05-27T07:21:58.698952+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T12:40:01.375520+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,9,3],"acting":[4,6,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1e","version":"88'2","reported_seq":41,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-05-27T07:22:30.855869+0000","last_change":"2024-05-27T07:21:45.790907+0000","last_active":"2024-05-27T07:22:30.855869+0000","last_peered":"2024-05-27T07:22:30.855869+0000","last_clean":"2024-05-27T07:22:30.855869+0000","last_became_active":"2024-05-27T07:21:45.790451+0000","last_became_peered":"2024-05-27T07:21:45.790451+0000","last_unstale":"2024-05-27T07:22:30.855869+0000","last_undegraded":"2024-05-27T07:22:30.855869+0000","last_fullsized":"2024-05-27T07:22:30.855869+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:21:44.543638+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:21:44.543638+0000","last_clean_scrub_stamp":"2024-05-27T07:21:44.543638+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T10:21:56.524930+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":34,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-05-27T07:22:30.801453+0000","last_change":"2024-05-27T07:21:48.595975+0000","last_active":"2024-05-27T07:22:30.801453+0000","last_peered":"2024-05-27T07:22:30.801453+0000","last_clean":"2024-05-27T07:22:30.801453+0000","last_became_active":"2024-05-27T07:21:48.595608+0000","last_became_peered":"2024-05-27T07:21:48.595608+0000","last_unstale":"2024-05-27T07:22:30.801453+0000","last_undegraded":"2024-05-27T07:22:30.801453+0000","last_fullsized":"2024-05-27T07:22:30.801453+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:21:47.565641+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:21:47.565641+0000","last_clean_scrub_stamp":"2024-05-27T07:21:47.565641+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T07:24:44.797973+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,9],"acting":[0,1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":24,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-05-27T07:22:30.795210+0000","last_change":"2024-05-27T07:21:59.732788+0000","last_active":"2024-05-27T07:22:30.795210+0000","last_peered":"2024-05-27T07:22:30.795210+0000","last_clean":"2024-05-27T07:22:30.795210+0000","last_became_active":"2024-05-27T07:21:59.732310+0000","last_became_peered":"2024-05-27T07:21:59.732310+0000","last_unstale":"2024-05-27T07:22:30.795210+0000","last_undegraded":"2024-05-27T07:22:30.795210+0000","last_fullsized":"2024-05-27T07:22:30.795210+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:21:58.698952+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:21:58.698952+0000","last_clean_scrub_stamp":"2024-05-27T07:21:58.698952+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T13:54:51.490154+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,9,5],"acting":[4,3,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1d","version":"88'12","reported_seq":53,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-05-27T07:22:56.890405+0000","last_change":"2024-05-27T07:21:45.569746+0000","last_active":"2024-05-27T07:22:56.890405+0000","last_peered":"2024-05-27T07:22:56.890405+0000","last_clean":"2024-05-27T07:22:56.890405+0000","last_became_active":"2024-05-27T07:21:45.569366+0000","last_became_peered":"2024-05-27T07:21:45.569366+0000","last_unstale":"2024-05-27T07:22:56.890405+0000","last_undegraded":"2024-05-27T07:22:56.890405+0000","last_fullsized":"2024-05-27T07:22:56.890405+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:21:44.543638+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:21:44.543638+0000","last_clean_scrub_stamp":"2024-05-27T07:21:44.543638+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T15:58:32.977754+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":16,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":7,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":34,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-05-27T07:22:31.579902+0000","last_change":"2024-05-27T07:21:48.606138+0000","last_active":"2024-05-27T07:22:31.579902+0000","last_peered":"2024-05-27T07:22:31.579902+0000","last_clean":"2024-05-27T07:22:31.579902+0000","last_became_active":"2024-05-27T07:21:48.605711+0000","last_became_peered":"2024-05-27T07:21:48.605711+0000","last_unstale":"2024-05-27T07:22:31.579902+0000","last_undegraded":"2024-05-27T07:22:31.579902+0000","last_fullsized":"2024-05-27T07:22:31.579902+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:21:47.565641+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:21:47.565641+0000","last_clean_scrub_stamp":"2024-05-27T07:21:47.565641+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T10:46:39.052387+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,6],"acting":[1,10,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":24,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-05-27T07:22:30.794971+0000","last_change":"2024-05-27T07:21:59.735475+0000","last_active":"2024-05-27T07:22:30.794971+0000","last_peered":"2024-05-27T07:22:30.794971+0000","last_clean":"2024-05-27T07:22:30.794971+0000","last_became_active":"2024-05-27T07:21:59.734986+0000","last_became_peered":"2024-05-27T07:21:59.734986+0000","last_unstale":"2024-05-27T07:22:30.794971+0000","last_undegraded":"2024-05-27T07:22:30.794971+0000","last_fullsized":"2024-05-27T07:22:30.794971+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:21:58.698952+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:21:58.698952+0000","last_clean_scrub_stamp":"2024-05-27T07:21:58.698952+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T14:48:28.198718+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5,3],"acting":[4,6,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1c","version":"88'5","reported_seq":46,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-05-27T07:22:55.890154+0000","last_change":"2024-05-27T07:21:45.572882+0000","last_active":"2024-05-27T07:22:55.890154+0000","last_peered":"2024-05-27T07:22:55.890154+0000","last_clean":"2024-05-27T07:22:55.890154+0000","last_became_active":"2024-05-27T07:21:45.572460+0000","last_became_peered":"2024-05-27T07:21:45.572460+0000","last_unstale":"2024-05-27T07:22:55.890154+0000","last_undegraded":"2024-05-27T07:22:55.890154+0000","last_fullsized":"2024-05-27T07:22:55.890154+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:21:44.543638+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:21:44.543638+0000","last_clean_scrub_stamp":"2024-05-27T07:21:44.543638+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T09:40:11.311923+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":7,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,1,5],"acting":[8,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":34,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-05-27T07:22:30.797360+0000","last_change":"2024-05-27T07:21:48.596217+0000","last_active":"2024-05-27T07:22:30.797360+0000","last_peered":"2024-05-27T07:22:30.797360+0000","last_clean":"2024-05-27T07:22:30.797360+0000","last_became_active":"2024-05-27T07:21:48.595832+0000","last_became_peered":"2024-05-27T07:21:48.595832+0000","last_unstale":"2024-05-27T07:22:30.797360+0000","last_undegraded":"2024-05-27T07:22:30.797360+0000","last_fullsized":"2024-05-27T07:22:30.797360+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:21:47.565641+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:21:47.565641+0000","last_clean_scrub_stamp":"2024-05-27T07:21:47.565641+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T10:08:05.304404+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":24,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-05-27T07:22:30.801397+0000","last_change":"2024-05-27T07:21:59.729617+0000","last_active":"2024-05-27T07:22:30.801397+0000","last_peered":"2024-05-27T07:22:30.801397+0000","last_clean":"2024-05-27T07:22:30.801397+0000","last_became_active":"2024-05-27T07:21:59.729003+0000","last_became_peered":"2024-05-27T07:21:59.729003+0000","last_unstale":"2024-05-27T07:22:30.801397+0000","last_undegraded":"2024-05-27T07:22:30.801397+0000","last_fullsized":"2024-05-27T07:22:30.801397+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:21:58.698952+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:21:58.698952+0000","last_clean_scrub_stamp":"2024-05-27T07:21:58.698952+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T14:55:04.844948+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7,11],"acting":[2,5,7,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1b","version":"88'6","reported_seq":72,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-05-27T07:23:18.927844+0000","last_change":"2024-05-27T07:21:45.566355+0000","last_active":"2024-05-27T07:23:18.927844+0000","last_peered":"2024-05-27T07:23:18.927844+0000","last_clean":"2024-05-27T07:23:18.927844+0000","last_became_active":"2024-05-27T07:21:45.564776+0000","last_became_peered":"2024-05-27T07:21:45.564776+0000","last_unstale":"2024-05-27T07:23:18.927844+0000","last_undegraded":"2024-05-27T07:23:18.927844+0000","last_fullsized":"2024-05-27T07:23:18.927844+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:21:44.543638+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:21:44.543638+0000","last_clean_scrub_stamp":"2024-05-27T07:21:44.543638+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T12:10:41.632630+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1013,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":27,"num_read_kb":3,"num_write":3,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":34,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-05-27T07:22:30.798264+0000","last_change":"2024-05-27T07:21:48.588837+0000","last_active":"2024-05-27T07:22:30.798264+0000","last_peered":"2024-05-27T07:22:30.798264+0000","last_clean":"2024-05-27T07:22:30.798264+0000","last_became_active":"2024-05-27T07:21:48.587149+0000","last_became_peered":"2024-05-27T07:21:48.587149+0000","last_unstale":"2024-05-27T07:22:30.798264+0000","last_undegraded":"2024-05-27T07:22:30.798264+0000","last_fullsized":"2024-05-27T07:22:30.798264+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:21:47.565641+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:21:47.565641+0000","last_clean_scrub_stamp":"2024-05-27T07:21:47.565641+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T09:20:05.124189+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,5],"acting":[8,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":24,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-05-27T07:22:30.801357+0000","last_change":"2024-05-27T07:21:59.730003+0000","last_active":"2024-05-27T07:22:30.801357+0000","last_peered":"2024-05-27T07:22:30.801357+0000","last_clean":"2024-05-27T07:22:30.801357+0000","last_became_active":"2024-05-27T07:21:59.729195+0000","last_became_peered":"2024-05-27T07:21:59.729195+0000","last_unstale":"2024-05-27T07:22:30.801357+0000","last_undegraded":"2024-05-27T07:22:30.801357+0000","last_fullsized":"2024-05-27T07:22:30.801357+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:21:58.698952+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:21:58.698952+0000","last_clean_scrub_stamp":"2024-05-27T07:21:58.698952+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T13:13:41.612300+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,9,5],"acting":[2,6,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1a","version":"88'7","reported_seq":47,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-05-27T07:22:54.881270+0000","last_change":"2024-05-27T07:21:45.567279+0000","last_active":"2024-05-27T07:22:54.881270+0000","last_peered":"2024-05-27T07:22:54.881270+0000","last_clean":"2024-05-27T07:22:54.881270+0000","last_became_active":"2024-05-27T07:21:45.566920+0000","last_became_peered":"2024-05-27T07:21:45.566920+0000","last_unstale":"2024-05-27T07:22:54.881270+0000","last_undegraded":"2024-05-27T07:22:54.881270+0000","last_fullsized":"2024-05-27T07:22:54.881270+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:21:44.543638+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:21:44.543638+0000","last_clean_scrub_stamp":"2024-05-27T07:21:44.543638+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T18:12:23.015535+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,8,6],"acting":[10,8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":40,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-05-27T07:22:30.857080+0000","last_change":"2024-05-27T07:22:30.857080+0000","last_active":"2024-05-27T07:22:30.857080+0000","last_peered":"2024-05-27T07:22:30.857080+0000","last_clean":"2024-05-27T07:22:30.857080+0000","last_became_active":"2024-05-27T07:22:30.856479+0000","last_became_peered":"2024-05-27T07:22:30.856479+0000","last_unstale":"2024-05-27T07:22:30.857080+0000","last_undegraded":"2024-05-27T07:22:30.857080+0000","last_fullsized":"2024-05-27T07:22:30.857080+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:21:47.565641+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:21:47.565641+0000","last_clean_scrub_stamp":"2024-05-27T07:21:47.565641+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T13:29:16.054448+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,8,3],"acting":[5,8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":31,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-05-27T07:22:30.805292+0000","last_change":"2024-05-27T07:22:30.805292+0000","last_active":"2024-05-27T07:22:30.805292+0000","last_peered":"2024-05-27T07:22:30.805292+0000","last_clean":"2024-05-27T07:22:30.805292+0000","last_became_active":"2024-05-27T07:22:30.804779+0000","last_became_peered":"2024-05-27T07:22:30.804779+0000","last_unstale":"2024-05-27T07:22:30.805292+0000","last_undegraded":"2024-05-27T07:22:30.805292+0000","last_fullsized":"2024-05-27T07:22:30.805292+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:21:58.698952+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:21:58.698952+0000","last_clean_scrub_stamp":"2024-05-27T07:21:58.698952+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T11:12:49.095931+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,8,0],"acting":[11,4,8,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.19","version":"88'3","reported_seq":42,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-05-27T07:22:53.872954+0000","last_change":"2024-05-27T07:21:45.561255+0000","last_active":"2024-05-27T07:22:53.872954+0000","last_peered":"2024-05-27T07:22:53.872954+0000","last_clean":"2024-05-27T07:22:53.872954+0000","last_became_active":"2024-05-27T07:21:45.560896+0000","last_became_peered":"2024-05-27T07:21:45.560896+0000","last_unstale":"2024-05-27T07:22:53.872954+0000","last_undegraded":"2024-05-27T07:22:53.872954+0000","last_fullsized":"2024-05-27T07:22:53.872954+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:21:44.543638+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:21:44.543638+0000","last_clean_scrub_stamp":"2024-05-27T07:21:44.543638+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T17:05:24.074628+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,9],"acting":[6,4,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":34,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-05-27T07:22:30.855361+0000","last_change":"2024-05-27T07:21:48.586045+0000","last_active":"2024-05-27T07:22:30.855361+0000","last_peered":"2024-05-27T07:22:30.855361+0000","last_clean":"2024-05-27T07:22:30.855361+0000","last_became_active":"2024-05-27T07:21:48.585681+0000","last_became_peered":"2024-05-27T07:21:48.585681+0000","last_unstale":"2024-05-27T07:22:30.855361+0000","last_undegraded":"2024-05-27T07:22:30.855361+0000","last_fullsized":"2024-05-27T07:22:30.855361+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:21:47.565641+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:21:47.565641+0000","last_clean_scrub_stamp":"2024-05-27T07:21:47.565641+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T09:56:36.408623+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,6],"acting":[3,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":24,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-05-27T07:22:30.801482+0000","last_change":"2024-05-27T07:21:59.739544+0000","last_active":"2024-05-27T07:22:30.801482+0000","last_peered":"2024-05-27T07:22:30.801482+0000","last_clean":"2024-05-27T07:22:30.801482+0000","last_became_active":"2024-05-27T07:21:59.739132+0000","last_became_peered":"2024-05-27T07:21:59.739132+0000","last_unstale":"2024-05-27T07:22:30.801482+0000","last_undegraded":"2024-05-27T07:22:30.801482+0000","last_fullsized":"2024-05-27T07:22:30.801482+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:21:58.698952+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:21:58.698952+0000","last_clean_scrub_stamp":"2024-05-27T07:21:58.698952+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T10:06:05.126579+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,6,1],"acting":[0,7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.18","version":"88'3","reported_seq":43,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-05-27T07:22:55.889742+0000","last_change":"2024-05-27T07:21:45.568171+0000","last_active":"2024-05-27T07:22:55.889742+0000","last_peered":"2024-05-27T07:22:55.889742+0000","last_clean":"2024-05-27T07:22:55.889742+0000","last_became_active":"2024-05-27T07:21:45.567627+0000","last_became_peered":"2024-05-27T07:21:45.567627+0000","last_unstale":"2024-05-27T07:22:55.889742+0000","last_undegraded":"2024-05-27T07:22:55.889742+0000","last_fullsized":"2024-05-27T07:22:55.889742+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:21:44.543638+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:21:44.543638+0000","last_clean_scrub_stamp":"2024-05-27T07:21:44.543638+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T16:43:56.407079+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":34,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-05-27T07:22:30.792156+0000","last_change":"2024-05-27T07:21:48.587452+0000","last_active":"2024-05-27T07:22:30.792156+0000","last_peered":"2024-05-27T07:22:30.792156+0000","last_clean":"2024-05-27T07:22:30.792156+0000","last_became_active":"2024-05-27T07:21:48.587027+0000","last_became_peered":"2024-05-27T07:21:48.587027+0000","last_unstale":"2024-05-27T07:22:30.792156+0000","last_undegraded":"2024-05-27T07:22:30.792156+0000","last_fullsized":"2024-05-27T07:22:30.792156+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:21:47.565641+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:21:47.565641+0000","last_clean_scrub_stamp":"2024-05-27T07:21:47.565641+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T13:51:44.890061+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,10],"acting":[9,3,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":24,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-05-27T07:22:30.855893+0000","last_change":"2024-05-27T07:21:59.738292+0000","last_active":"2024-05-27T07:22:30.855893+0000","last_peered":"2024-05-27T07:22:30.855893+0000","last_clean":"2024-05-27T07:22:30.855893+0000","last_became_active":"2024-05-27T07:21:59.737941+0000","last_became_peered":"2024-05-27T07:21:59.737941+0000","last_unstale":"2024-05-27T07:22:30.855893+0000","last_undegraded":"2024-05-27T07:22:30.855893+0000","last_fullsized":"2024-05-27T07:22:30.855893+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:21:58.698952+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:21:58.698952+0000","last_clean_scrub_stamp":"2024-05-27T07:21:58.698952+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T13:54:24.192917+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1,5],"acting":[3,7,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.17","version":"88'4","reported_seq":90,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-05-27T07:23:18.927012+0000","last_change":"2024-05-27T07:21:45.568338+0000","last_active":"2024-05-27T07:23:18.927012+0000","last_peered":"2024-05-27T07:23:18.927012+0000","last_clean":"2024-05-27T07:23:18.927012+0000","last_became_active":"2024-05-27T07:21:45.567856+0000","last_became_peered":"2024-05-27T07:21:45.567856+0000","last_unstale":"2024-05-27T07:23:18.927012+0000","last_undegraded":"2024-05-27T07:23:18.927012+0000","last_fullsized":"2024-05-27T07:23:18.927012+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:21:44.543638+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:21:44.543638+0000","last_clean_scrub_stamp":"2024-05-27T07:21:44.543638+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T15:02:04.188711+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":180,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":46,"num_read_kb":46,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,5,9],"acting":[10,5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":34,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-05-27T07:22:31.580247+0000","last_change":"2024-05-27T07:21:48.606062+0000","last_active":"2024-05-27T07:22:31.580247+0000","last_peered":"2024-05-27T07:22:31.580247+0000","last_clean":"2024-05-27T07:22:31.580247+0000","last_became_active":"2024-05-27T07:21:48.605423+0000","last_became_peered":"2024-05-27T07:21:48.605423+0000","last_unstale":"2024-05-27T07:22:31.580247+0000","last_undegraded":"2024-05-27T07:22:31.580247+0000","last_fullsized":"2024-05-27T07:22:31.580247+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:21:47.565641+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:21:47.565641+0000","last_clean_scrub_stamp":"2024-05-27T07:21:47.565641+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T14:39:06.921519+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,9],"acting":[1,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":24,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-05-27T07:22:30.792054+0000","last_change":"2024-05-27T07:21:59.735150+0000","last_active":"2024-05-27T07:22:30.792054+0000","last_peered":"2024-05-27T07:22:30.792054+0000","last_clean":"2024-05-27T07:22:30.792054+0000","last_became_active":"2024-05-27T07:21:59.734688+0000","last_became_peered":"2024-05-27T07:21:59.734688+0000","last_unstale":"2024-05-27T07:22:30.792054+0000","last_undegraded":"2024-05-27T07:22:30.792054+0000","last_fullsized":"2024-05-27T07:22:30.792054+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:21:58.698952+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:21:58.698952+0000","last_clean_scrub_stamp":"2024-05-27T07:21:58.698952+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T15:38:02.095947+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,0,2,4],"acting":[9,0,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.16","version":"88'4","reported_seq":41,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-05-27T07:22:55.889723+0000","last_change":"2024-05-27T07:21:45.559550+0000","last_active":"2024-05-27T07:22:55.889723+0000","last_peered":"2024-05-27T07:22:55.889723+0000","last_clean":"2024-05-27T07:22:55.889723+0000","last_became_active":"2024-05-27T07:21:45.559181+0000","last_became_peered":"2024-05-27T07:21:45.559181+0000","last_unstale":"2024-05-27T07:22:55.889723+0000","last_undegraded":"2024-05-27T07:22:55.889723+0000","last_fullsized":"2024-05-27T07:22:55.889723+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:21:44.543638+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:21:44.543638+0000","last_clean_scrub_stamp":"2024-05-27T07:21:44.543638+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T10:33:45.288437+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":34,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-05-27T07:22:30.801530+0000","last_change":"2024-05-27T07:21:48.593631+0000","last_active":"2024-05-27T07:22:30.801530+0000","last_peered":"2024-05-27T07:22:30.801530+0000","last_clean":"2024-05-27T07:22:30.801530+0000","last_became_active":"2024-05-27T07:21:48.592330+0000","last_became_peered":"2024-05-27T07:21:48.592330+0000","last_unstale":"2024-05-27T07:22:30.801530+0000","last_undegraded":"2024-05-27T07:22:30.801530+0000","last_fullsized":"2024-05-27T07:22:30.801530+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:21:47.565641+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:21:47.565641+0000","last_clean_scrub_stamp":"2024-05-27T07:21:47.565641+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T19:19:51.593307+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,5],"acting":[0,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":24,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-05-27T07:22:30.795471+0000","last_change":"2024-05-27T07:21:59.725158+0000","last_active":"2024-05-27T07:22:30.795471+0000","last_peered":"2024-05-27T07:22:30.795471+0000","last_clean":"2024-05-27T07:22:30.795471+0000","last_became_active":"2024-05-27T07:21:59.724651+0000","last_became_peered":"2024-05-27T07:21:59.724651+0000","last_unstale":"2024-05-27T07:22:30.795471+0000","last_undegraded":"2024-05-27T07:22:30.795471+0000","last_fullsized":"2024-05-27T07:22:30.795471+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:21:58.698952+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:21:58.698952+0000","last_clean_scrub_stamp":"2024-05-27T07:21:58.698952+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T12:10:45.747484+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2,9],"acting":[4,1,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.15","version":"88'5","reported_seq":45,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-05-27T07:22:54.880267+0000","last_change":"2024-05-27T07:21:45.791831+0000","last_active":"2024-05-27T07:22:54.880267+0000","last_peered":"2024-05-27T07:22:54.880267+0000","last_clean":"2024-05-27T07:22:54.880267+0000","last_became_active":"2024-05-27T07:21:45.791271+0000","last_became_peered":"2024-05-27T07:21:45.791271+0000","last_unstale":"2024-05-27T07:22:54.880267+0000","last_undegraded":"2024-05-27T07:22:54.880267+0000","last_fullsized":"2024-05-27T07:22:54.880267+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:21:44.543638+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:21:44.543638+0000","last_clean_scrub_stamp":"2024-05-27T07:21:44.543638+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T16:20:37.492337+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":7,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,10],"acting":[5,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":34,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-05-27T07:22:31.580163+0000","last_change":"2024-05-27T07:21:48.605605+0000","last_active":"2024-05-27T07:22:31.580163+0000","last_peered":"2024-05-27T07:22:31.580163+0000","last_clean":"2024-05-27T07:22:31.580163+0000","last_became_active":"2024-05-27T07:21:48.604655+0000","last_became_peered":"2024-05-27T07:21:48.604655+0000","last_unstale":"2024-05-27T07:22:31.580163+0000","last_undegraded":"2024-05-27T07:22:31.580163+0000","last_fullsized":"2024-05-27T07:22:31.580163+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:21:47.565641+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:21:47.565641+0000","last_clean_scrub_stamp":"2024-05-27T07:21:47.565641+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T14:33:53.661983+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,8,4],"acting":[1,8,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":31,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-05-27T07:22:31.418393+0000","last_change":"2024-05-27T07:22:31.418393+0000","last_active":"2024-05-27T07:22:31.418393+0000","last_peered":"2024-05-27T07:22:31.418393+0000","last_clean":"2024-05-27T07:22:31.418393+0000","last_became_active":"2024-05-27T07:22:31.418006+0000","last_became_peered":"2024-05-27T07:22:31.418006+0000","last_unstale":"2024-05-27T07:22:31.418393+0000","last_undegraded":"2024-05-27T07:22:31.418393+0000","last_fullsized":"2024-05-27T07:22:31.418393+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:21:58.698952+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:21:58.698952+0000","last_clean_scrub_stamp":"2024-05-27T07:21:58.698952+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T14:30:55.130461+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,11,10],"acting":[9,8,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.14","version":"88'32","reported_seq":126,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-05-27T07:23:18.933066+0000","last_change":"2024-05-27T07:21:45.564829+0000","last_active":"2024-05-27T07:23:18.933066+0000","last_peered":"2024-05-27T07:23:18.933066+0000","last_clean":"2024-05-27T07:23:18.933066+0000","last_became_active":"2024-05-27T07:21:45.564457+0000","last_became_peered":"2024-05-27T07:21:45.564457+0000","last_unstale":"2024-05-27T07:23:18.933066+0000","last_undegraded":"2024-05-27T07:23:18.933066+0000","last_fullsized":"2024-05-27T07:23:18.933066+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:21:44.543638+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:21:44.543638+0000","last_clean_scrub_stamp":"2024-05-27T07:21:44.543638+0000","objects_scrubbed":0,"log_size":32,"log_dups_size":0,"ondisk_log_size":32,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T19:12:44.546989+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":56160,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":54,"num_read_kb":82,"num_write":34,"num_write_kb":68,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,1,2],"acting":[10,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":34,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-05-27T07:22:30.798642+0000","last_change":"2024-05-27T07:21:48.591672+0000","last_active":"2024-05-27T07:22:30.798642+0000","last_peered":"2024-05-27T07:22:30.798642+0000","last_clean":"2024-05-27T07:22:30.798642+0000","last_became_active":"2024-05-27T07:21:48.590861+0000","last_became_peered":"2024-05-27T07:21:48.590861+0000","last_unstale":"2024-05-27T07:22:30.798642+0000","last_undegraded":"2024-05-27T07:22:30.798642+0000","last_fullsized":"2024-05-27T07:22:30.798642+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:21:47.565641+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:21:47.565641+0000","last_clean_scrub_stamp":"2024-05-27T07:21:47.565641+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T12:03:40.627764+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,7],"acting":[8,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":24,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-05-27T07:22:30.797886+0000","last_change":"2024-05-27T07:21:59.736446+0000","last_active":"2024-05-27T07:22:30.797886+0000","last_peered":"2024-05-27T07:22:30.797886+0000","last_clean":"2024-05-27T07:22:30.797886+0000","last_became_active":"2024-05-27T07:21:59.735854+0000","last_became_peered":"2024-05-27T07:21:59.735854+0000","last_unstale":"2024-05-27T07:22:30.797886+0000","last_undegraded":"2024-05-27T07:22:30.797886+0000","last_fullsized":"2024-05-27T07:22:30.797886+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:21:58.698952+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:21:58.698952+0000","last_clean_scrub_stamp":"2024-05-27T07:21:58.698952+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T08:02:14.640325+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,8,3,6],"acting":[5,8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.13","version":"88'9","reported_seq":50,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-05-27T07:22:56.890667+0000","last_change":"2024-05-27T07:21:45.562930+0000","last_active":"2024-05-27T07:22:56.890667+0000","last_peered":"2024-05-27T07:22:56.890667+0000","last_clean":"2024-05-27T07:22:56.890667+0000","last_became_active":"2024-05-27T07:21:45.562515+0000","last_became_peered":"2024-05-27T07:21:45.562515+0000","last_unstale":"2024-05-27T07:22:56.890667+0000","last_undegraded":"2024-05-27T07:22:56.890667+0000","last_fullsized":"2024-05-27T07:22:56.890667+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:21:44.543638+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:21:44.543638+0000","last_clean_scrub_stamp":"2024-05-27T07:21:44.543638+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T17:54:24.006219+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":68,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":13,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,10,3],"acting":[2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":34,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-05-27T07:22:31.626151+0000","last_change":"2024-05-27T07:21:48.591090+0000","last_active":"2024-05-27T07:22:31.626151+0000","last_peered":"2024-05-27T07:22:31.626151+0000","last_clean":"2024-05-27T07:22:31.626151+0000","last_became_active":"2024-05-27T07:21:48.590726+0000","last_became_peered":"2024-05-27T07:21:48.590726+0000","last_unstale":"2024-05-27T07:22:31.626151+0000","last_undegraded":"2024-05-27T07:22:31.626151+0000","last_fullsized":"2024-05-27T07:22:31.626151+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:21:47.565641+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:21:47.565641+0000","last_clean_scrub_stamp":"2024-05-27T07:21:47.565641+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T18:40:03.880407+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,3],"acting":[6,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.14","version":"0'0","reported_seq":24,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-05-27T07:22:30.793943+0000","last_change":"2024-05-27T07:21:59.738375+0000","last_active":"2024-05-27T07:22:30.793943+0000","last_peered":"2024-05-27T07:22:30.793943+0000","last_clean":"2024-05-27T07:22:30.793943+0000","last_became_active":"2024-05-27T07:21:59.737957+0000","last_became_peered":"2024-05-27T07:21:59.737957+0000","last_unstale":"2024-05-27T07:22:30.793943+0000","last_undegraded":"2024-05-27T07:22:30.793943+0000","last_fullsized":"2024-05-27T07:22:30.793943+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:21:58.698952+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:21:58.698952+0000","last_clean_scrub_stamp":"2024-05-27T07:21:58.698952+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T16:53:51.397143+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,9,7,6],"acting":[11,9,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.12","version":"88'5","reported_seq":45,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-05-27T07:22:55.888096+0000","last_change":"2024-05-27T07:21:45.566986+0000","last_active":"2024-05-27T07:22:55.888096+0000","last_peered":"2024-05-27T07:22:55.888096+0000","last_clean":"2024-05-27T07:22:55.888096+0000","last_became_active":"2024-05-27T07:21:45.566579+0000","last_became_peered":"2024-05-27T07:21:45.566579+0000","last_unstale":"2024-05-27T07:22:55.888096+0000","last_undegraded":"2024-05-27T07:22:55.888096+0000","last_fullsized":"2024-05-27T07:22:55.888096+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:21:44.543638+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:21:44.543638+0000","last_clean_scrub_stamp":"2024-05-27T07:21:44.543638+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T17:08:14.034740+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":5,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":32,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-05-27T07:22:30.796994+0000","last_change":"2024-05-27T07:21:48.596575+0000","last_active":"2024-05-27T07:22:30.796994+0000","last_peered":"2024-05-27T07:22:30.796994+0000","last_clean":"2024-05-27T07:22:30.796994+0000","last_became_active":"2024-05-27T07:21:48.596215+0000","last_became_peered":"2024-05-27T07:21:48.596215+0000","last_unstale":"2024-05-27T07:22:30.796994+0000","last_undegraded":"2024-05-27T07:22:30.796994+0000","last_fullsized":"2024-05-27T07:22:30.796994+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:21:47.565641+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:21:47.565641+0000","last_clean_scrub_stamp":"2024-05-27T07:21:47.565641+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T14:57:05.795269+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":24,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-05-27T07:22:30.792387+0000","last_change":"2024-05-27T07:21:59.737690+0000","last_active":"2024-05-27T07:22:30.792387+0000","last_peered":"2024-05-27T07:22:30.792387+0000","last_clean":"2024-05-27T07:22:30.792387+0000","last_became_active":"2024-05-27T07:21:59.735646+0000","last_became_peered":"2024-05-27T07:21:59.735646+0000","last_unstale":"2024-05-27T07:22:30.792387+0000","last_undegraded":"2024-05-27T07:22:30.792387+0000","last_fullsized":"2024-05-27T07:22:30.792387+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:21:58.698952+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:21:58.698952+0000","last_clean_scrub_stamp":"2024-05-27T07:21:58.698952+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T13:54:41.227626+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,0,3],"acting":[9,7,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.11","version":"88'6","reported_seq":73,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-05-27T07:23:18.927905+0000","last_change":"2024-05-27T07:21:45.564836+0000","last_active":"2024-05-27T07:23:18.927905+0000","last_peered":"2024-05-27T07:23:18.927905+0000","last_clean":"2024-05-27T07:23:18.927905+0000","last_became_active":"2024-05-27T07:21:45.564481+0000","last_became_peered":"2024-05-27T07:21:45.564481+0000","last_unstale":"2024-05-27T07:23:18.927905+0000","last_undegraded":"2024-05-27T07:23:18.927905+0000","last_fullsized":"2024-05-27T07:23:18.927905+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:21:44.543638+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:21:44.543638+0000","last_clean_scrub_stamp":"2024-05-27T07:21:44.543638+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T15:13:12.012769+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":979,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":28,"num_read_kb":4,"num_write":6,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,10],"acting":[4,7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":32,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-05-27T07:22:30.797769+0000","last_change":"2024-05-27T07:21:48.601331+0000","last_active":"2024-05-27T07:22:30.797769+0000","last_peered":"2024-05-27T07:22:30.797769+0000","last_clean":"2024-05-27T07:22:30.797769+0000","last_became_active":"2024-05-27T07:21:48.600917+0000","last_became_peered":"2024-05-27T07:21:48.600917+0000","last_unstale":"2024-05-27T07:22:30.797769+0000","last_undegraded":"2024-05-27T07:22:30.797769+0000","last_fullsized":"2024-05-27T07:22:30.797769+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:21:47.565641+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:21:47.565641+0000","last_clean_scrub_stamp":"2024-05-27T07:21:47.565641+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T17:26:42.129989+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":24,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-05-27T07:22:31.626720+0000","last_change":"2024-05-27T07:21:59.737662+0000","last_active":"2024-05-27T07:22:31.626720+0000","last_peered":"2024-05-27T07:22:31.626720+0000","last_clean":"2024-05-27T07:22:31.626720+0000","last_became_active":"2024-05-27T07:21:59.737229+0000","last_became_peered":"2024-05-27T07:21:59.737229+0000","last_unstale":"2024-05-27T07:22:31.626720+0000","last_undegraded":"2024-05-27T07:22:31.626720+0000","last_fullsized":"2024-05-27T07:22:31.626720+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:21:58.698952+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:21:58.698952+0000","last_clean_scrub_stamp":"2024-05-27T07:21:58.698952+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T15:32:31.147571+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,9,11],"acting":[6,3,9,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.10","version":"88'17","reported_seq":74,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-05-27T07:22:55.889266+0000","last_change":"2024-05-27T07:22:30.814815+0000","last_active":"2024-05-27T07:22:55.889266+0000","last_peered":"2024-05-27T07:22:55.889266+0000","last_clean":"2024-05-27T07:22:55.889266+0000","last_became_active":"2024-05-27T07:22:30.802828+0000","last_became_peered":"2024-05-27T07:22:30.802828+0000","last_unstale":"2024-05-27T07:22:55.889266+0000","last_undegraded":"2024-05-27T07:22:55.889266+0000","last_fullsized":"2024-05-27T07:22:55.889266+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:21:44.543638+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:21:44.543638+0000","last_clean_scrub_stamp":"2024-05-27T07:21:44.543638+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T11:38:07.363507+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,4],"acting":[8,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":34,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-05-27T07:22:30.797081+0000","last_change":"2024-05-27T07:21:48.596204+0000","last_active":"2024-05-27T07:22:30.797081+0000","last_peered":"2024-05-27T07:22:30.797081+0000","last_clean":"2024-05-27T07:22:30.797081+0000","last_became_active":"2024-05-27T07:21:48.595827+0000","last_became_peered":"2024-05-27T07:21:48.595827+0000","last_unstale":"2024-05-27T07:22:30.797081+0000","last_undegraded":"2024-05-27T07:22:30.797081+0000","last_fullsized":"2024-05-27T07:22:30.797081+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:21:47.565641+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:21:47.565641+0000","last_clean_scrub_stamp":"2024-05-27T07:21:47.565641+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T15:51:53.162554+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":24,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-05-27T07:22:30.796942+0000","last_change":"2024-05-27T07:21:59.735837+0000","last_active":"2024-05-27T07:22:30.796942+0000","last_peered":"2024-05-27T07:22:30.796942+0000","last_clean":"2024-05-27T07:22:30.796942+0000","last_became_active":"2024-05-27T07:21:59.735063+0000","last_became_peered":"2024-05-27T07:21:59.735063+0000","last_unstale":"2024-05-27T07:22:30.796942+0000","last_undegraded":"2024-05-27T07:22:30.796942+0000","last_fullsized":"2024-05-27T07:22:30.796942+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:21:58.698952+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:21:58.698952+0000","last_clean_scrub_stamp":"2024-05-27T07:21:58.698952+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T13:04:21.751043+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,3,1],"acting":[5,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.f","version":"88'2","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-05-27T07:22:54.869368+0000","last_change":"2024-05-27T07:22:30.809491+0000","last_active":"2024-05-27T07:22:54.869368+0000","last_peered":"2024-05-27T07:22:54.869368+0000","last_clean":"2024-05-27T07:22:54.869368+0000","last_became_active":"2024-05-27T07:22:30.802187+0000","last_became_peered":"2024-05-27T07:22:30.802187+0000","last_unstale":"2024-05-27T07:22:54.869368+0000","last_undegraded":"2024-05-27T07:22:54.869368+0000","last_fullsized":"2024-05-27T07:22:54.869368+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:21:44.543638+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:21:44.543638+0000","last_clean_scrub_stamp":"2024-05-27T07:21:44.543638+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T18:11:32.821962+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":22,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,4],"acting":[11,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":34,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-05-27T07:22:30.793572+0000","last_change":"2024-05-27T07:21:48.597362+0000","last_active":"2024-05-27T07:22:30.793572+0000","last_peered":"2024-05-27T07:22:30.793572+0000","last_clean":"2024-05-27T07:22:30.793572+0000","last_became_active":"2024-05-27T07:21:48.596920+0000","last_became_peered":"2024-05-27T07:21:48.596920+0000","last_unstale":"2024-05-27T07:22:30.793572+0000","last_undegraded":"2024-05-27T07:22:30.793572+0000","last_fullsized":"2024-05-27T07:22:30.793572+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:21:47.565641+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:21:47.565641+0000","last_clean_scrub_stamp":"2024-05-27T07:21:47.565641+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T16:50:21.021320+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,4],"acting":[11,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":24,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-05-27T07:22:30.798136+0000","last_change":"2024-05-27T07:21:59.732303+0000","last_active":"2024-05-27T07:22:30.798136+0000","last_peered":"2024-05-27T07:22:30.798136+0000","last_clean":"2024-05-27T07:22:30.798136+0000","last_became_active":"2024-05-27T07:21:59.731851+0000","last_became_peered":"2024-05-27T07:21:59.731851+0000","last_unstale":"2024-05-27T07:22:30.798136+0000","last_undegraded":"2024-05-27T07:22:30.798136+0000","last_fullsized":"2024-05-27T07:22:30.798136+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:21:58.698952+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:21:58.698952+0000","last_clean_scrub_stamp":"2024-05-27T07:21:58.698952+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T08:08:38.433818+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,10,11,9],"acting":[8,10,11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.4","version":"88'1","reported_seq":41,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-05-27T07:22:30.792697+0000","last_change":"2024-05-27T07:21:45.791370+0000","last_active":"2024-05-27T07:22:30.792697+0000","last_peered":"2024-05-27T07:22:30.792697+0000","last_clean":"2024-05-27T07:22:30.792697+0000","last_became_active":"2024-05-27T07:21:45.790987+0000","last_became_peered":"2024-05-27T07:21:45.790987+0000","last_unstale":"2024-05-27T07:22:30.792697+0000","last_undegraded":"2024-05-27T07:22:30.792697+0000","last_fullsized":"2024-05-27T07:22:30.792697+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:21:44.543638+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:21:44.543638+0000","last_clean_scrub_stamp":"2024-05-27T07:21:44.543638+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T14:20:18.570194+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2,0],"acting":[9,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":34,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-05-27T07:22:31.416954+0000","last_change":"2024-05-27T07:21:48.585984+0000","last_active":"2024-05-27T07:22:31.416954+0000","last_peered":"2024-05-27T07:22:31.416954+0000","last_clean":"2024-05-27T07:22:31.416954+0000","last_became_active":"2024-05-27T07:21:48.585606+0000","last_became_peered":"2024-05-27T07:21:48.585606+0000","last_unstale":"2024-05-27T07:22:31.416954+0000","last_undegraded":"2024-05-27T07:22:31.416954+0000","last_fullsized":"2024-05-27T07:22:31.416954+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:21:47.565641+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:21:47.565641+0000","last_clean_scrub_stamp":"2024-05-27T07:21:47.565641+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T17:56:17.659024+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,3,6],"acting":[10,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":24,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-05-27T07:22:31.626383+0000","last_change":"2024-05-27T07:21:59.734022+0000","last_active":"2024-05-27T07:22:31.626383+0000","last_peered":"2024-05-27T07:22:31.626383+0000","last_clean":"2024-05-27T07:22:31.626383+0000","last_became_active":"2024-05-27T07:21:59.733467+0000","last_became_peered":"2024-05-27T07:21:59.733467+0000","last_unstale":"2024-05-27T07:22:31.626383+0000","last_undegraded":"2024-05-27T07:22:31.626383+0000","last_fullsized":"2024-05-27T07:22:31.626383+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:21:58.698952+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:21:58.698952+0000","last_clean_scrub_stamp":"2024-05-27T07:21:58.698952+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T09:54:11.058495+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,1,8],"acting":[6,0,1,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.2","version":"88'5","reported_seq":69,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-05-27T07:23:20.929880+0000","last_change":"2024-05-27T07:21:45.563072+0000","last_active":"2024-05-27T07:23:20.929880+0000","last_peered":"2024-05-27T07:23:20.929880+0000","last_clean":"2024-05-27T07:23:20.929880+0000","last_became_active":"2024-05-27T07:21:45.562632+0000","last_became_peered":"2024-05-27T07:21:45.562632+0000","last_unstale":"2024-05-27T07:23:20.929880+0000","last_undegraded":"2024-05-27T07:23:20.929880+0000","last_fullsized":"2024-05-27T07:23:20.929880+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:21:44.543638+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:21:44.543638+0000","last_clean_scrub_stamp":"2024-05-27T07:21:44.543638+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T12:54:19.555467+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":588,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":27,"num_read_kb":1,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":34,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-05-27T07:22:31.416512+0000","last_change":"2024-05-27T07:21:48.594867+0000","last_active":"2024-05-27T07:22:31.416512+0000","last_peered":"2024-05-27T07:22:31.416512+0000","last_clean":"2024-05-27T07:22:31.416512+0000","last_became_active":"2024-05-27T07:21:48.594523+0000","last_became_peered":"2024-05-27T07:21:48.594523+0000","last_unstale":"2024-05-27T07:22:31.416512+0000","last_undegraded":"2024-05-27T07:22:31.416512+0000","last_fullsized":"2024-05-27T07:22:31.416512+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:21:47.565641+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:21:47.565641+0000","last_clean_scrub_stamp":"2024-05-27T07:21:47.565641+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T17:28:55.862014+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,4,0],"acting":[10,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":24,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-05-27T07:22:31.626438+0000","last_change":"2024-05-27T07:21:59.741936+0000","last_active":"2024-05-27T07:22:31.626438+0000","last_peered":"2024-05-27T07:22:31.626438+0000","last_clean":"2024-05-27T07:22:31.626438+0000","last_became_active":"2024-05-27T07:21:59.741458+0000","last_became_peered":"2024-05-27T07:21:59.741458+0000","last_unstale":"2024-05-27T07:22:31.626438+0000","last_undegraded":"2024-05-27T07:22:31.626438+0000","last_fullsized":"2024-05-27T07:22:31.626438+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:21:58.698952+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:21:58.698952+0000","last_clean_scrub_stamp":"2024-05-27T07:21:58.698952+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T15:51:28.560995+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,10,0],"acting":[6,5,10,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":11,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-05-27T07:22:30.803379+0000","last_change":"2024-05-27T07:22:30.803379+0000","last_active":"2024-05-27T07:22:30.803379+0000","last_peered":"2024-05-27T07:22:30.803379+0000","last_clean":"2024-05-27T07:22:30.803379+0000","last_became_active":"2024-05-27T07:22:30.803024+0000","last_became_peered":"2024-05-27T07:22:30.803024+0000","last_unstale":"2024-05-27T07:22:30.803379+0000","last_undegraded":"2024-05-27T07:22:30.803379+0000","last_fullsized":"2024-05-27T07:22:30.803379+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:21:47.565641+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:21:47.565641+0000","last_clean_scrub_stamp":"2024-05-27T07:21:47.565641+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T07:56:32.070610+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2,4],"acting":[9,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.1","version":"88'5","reported_seq":69,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-05-27T07:23:18.927282+0000","last_change":"2024-05-27T07:21:45.790849+0000","last_active":"2024-05-27T07:23:18.927282+0000","last_peered":"2024-05-27T07:23:18.927282+0000","last_clean":"2024-05-27T07:23:18.927282+0000","last_became_active":"2024-05-27T07:21:45.790435+0000","last_became_peered":"2024-05-27T07:21:45.790435+0000","last_unstale":"2024-05-27T07:23:18.927282+0000","last_undegraded":"2024-05-27T07:23:18.927282+0000","last_fullsized":"2024-05-27T07:23:18.927282+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:21:44.543638+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:21:44.543638+0000","last_clean_scrub_stamp":"2024-05-27T07:21:44.543638+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T09:34:51.479882+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":25,"num_read_kb":25,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,0],"acting":[6,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.5","version":"88'1","reported_seq":24,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-05-27T07:22:30.797393+0000","last_change":"2024-05-27T07:21:59.731338+0000","last_active":"2024-05-27T07:22:30.797393+0000","last_peered":"2024-05-27T07:22:30.797393+0000","last_clean":"2024-05-27T07:22:30.797393+0000","last_became_active":"2024-05-27T07:21:59.730847+0000","last_became_peered":"2024-05-27T07:21:59.730847+0000","last_unstale":"2024-05-27T07:22:30.797393+0000","last_undegraded":"2024-05-27T07:22:30.797393+0000","last_fullsized":"2024-05-27T07:22:30.797393+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:21:58.698952+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:21:58.698952+0000","last_clean_scrub_stamp":"2024-05-27T07:21:58.698952+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T18:18:50.901129+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,11,9],"acting":[7,5,11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":34,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-05-27T07:22:30.855389+0000","last_change":"2024-05-27T07:21:48.594274+0000","last_active":"2024-05-27T07:22:30.855389+0000","last_peered":"2024-05-27T07:22:30.855389+0000","last_clean":"2024-05-27T07:22:30.855389+0000","last_became_active":"2024-05-27T07:21:48.593940+0000","last_became_peered":"2024-05-27T07:21:48.593940+0000","last_unstale":"2024-05-27T07:22:30.855389+0000","last_undegraded":"2024-05-27T07:22:30.855389+0000","last_fullsized":"2024-05-27T07:22:30.855389+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:21:47.565641+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:21:47.565641+0000","last_clean_scrub_stamp":"2024-05-27T07:21:47.565641+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T13:25:46.531524+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"88'6","reported_seq":49,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-05-27T07:22:56.891759+0000","last_change":"2024-05-27T07:21:45.563365+0000","last_active":"2024-05-27T07:22:56.891759+0000","last_peered":"2024-05-27T07:22:56.891759+0000","last_clean":"2024-05-27T07:22:56.891759+0000","last_became_active":"2024-05-27T07:21:45.563012+0000","last_became_peered":"2024-05-27T07:21:45.563012+0000","last_unstale":"2024-05-27T07:22:56.891759+0000","last_undegraded":"2024-05-27T07:22:56.891759+0000","last_fullsized":"2024-05-27T07:22:56.891759+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:21:44.543638+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:21:44.543638+0000","last_clean_scrub_stamp":"2024-05-27T07:21:44.543638+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T12:58:51.079965+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":68,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":4,"num_read_kb":4,"num_write":8,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,9],"acting":[4,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.0","version":"20'92","reported_seq":112,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-05-27T07:22:30.796503+0000","last_change":"2024-05-27T07:20:21.611212+0000","last_active":"2024-05-27T07:22:30.796503+0000","last_peered":"2024-05-27T07:22:30.796503+0000","last_clean":"2024-05-27T07:22:30.796503+0000","last_became_active":"2024-05-27T07:20:21.605163+0000","last_became_peered":"2024-05-27T07:20:21.605163+0000","last_unstale":"2024-05-27T07:22:30.796503+0000","last_undegraded":"2024-05-27T07:22:30.796503+0000","last_fullsized":"2024-05-27T07:22:30.796503+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:16:57.668433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:16:57.668433+0000","last_clean_scrub_stamp":"2024-05-27T07:16:57.668433+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T12:10:59.074162+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":81,"num_read_kb":69,"num_write":123,"num_write_kb":2150,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":2951840,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":24,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-05-27T07:22:30.797236+0000","last_change":"2024-05-27T07:21:59.737210+0000","last_active":"2024-05-27T07:22:30.797236+0000","last_peered":"2024-05-27T07:22:30.797236+0000","last_clean":"2024-05-27T07:22:30.797236+0000","last_became_active":"2024-05-27T07:21:59.736799+0000","last_became_peered":"2024-05-27T07:21:59.736799+0000","last_unstale":"2024-05-27T07:22:30.797236+0000","last_undegraded":"2024-05-27T07:22:30.797236+0000","last_fullsized":"2024-05-27T07:22:30.797236+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:21:58.698952+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:21:58.698952+0000","last_clean_scrub_stamp":"2024-05-27T07:21:58.698952+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T13:03:17.153687+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,6,10],"acting":[5,9,6,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":34,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-05-27T07:22:31.625943+0000","last_change":"2024-05-27T07:21:48.591550+0000","last_active":"2024-05-27T07:22:31.625943+0000","last_peered":"2024-05-27T07:22:31.625943+0000","last_clean":"2024-05-27T07:22:31.625943+0000","last_became_active":"2024-05-27T07:21:48.591223+0000","last_became_peered":"2024-05-27T07:21:48.591223+0000","last_unstale":"2024-05-27T07:22:31.625943+0000","last_undegraded":"2024-05-27T07:22:31.625943+0000","last_fullsized":"2024-05-27T07:22:31.625943+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:21:47.565641+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:21:47.565641+0000","last_clean_scrub_stamp":"2024-05-27T07:21:47.565641+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T15:06:23.395645+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,2],"acting":[6,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.0","version":"88'4","reported_seq":41,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-05-27T07:22:53.872885+0000","last_change":"2024-05-27T07:21:45.569224+0000","last_active":"2024-05-27T07:22:53.872885+0000","last_peered":"2024-05-27T07:22:53.872885+0000","last_clean":"2024-05-27T07:22:53.872885+0000","last_became_active":"2024-05-27T07:21:45.568818+0000","last_became_peered":"2024-05-27T07:21:45.568818+0000","last_unstale":"2024-05-27T07:22:53.872885+0000","last_undegraded":"2024-05-27T07:22:53.872885+0000","last_fullsized":"2024-05-27T07:22:53.872885+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:21:44.543638+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:21:44.543638+0000","last_clean_scrub_stamp":"2024-05-27T07:21:44.543638+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T08:12:33.157842+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,8],"acting":[7,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":24,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-05-27T07:22:30.793255+0000","last_change":"2024-05-27T07:21:59.733733+0000","last_active":"2024-05-27T07:22:30.793255+0000","last_peered":"2024-05-27T07:22:30.793255+0000","last_clean":"2024-05-27T07:22:30.793255+0000","last_became_active":"2024-05-27T07:21:59.733112+0000","last_became_peered":"2024-05-27T07:21:59.733112+0000","last_unstale":"2024-05-27T07:22:30.793255+0000","last_undegraded":"2024-05-27T07:22:30.793255+0000","last_fullsized":"2024-05-27T07:22:30.793255+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:21:58.698952+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:21:58.698952+0000","last_clean_scrub_stamp":"2024-05-27T07:21:58.698952+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T16:01:30.783800+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,7,6],"acting":[11,5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.5","version":"88'4","reported_seq":45,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-05-27T07:22:55.890006+0000","last_change":"2024-05-27T07:21:45.792029+0000","last_active":"2024-05-27T07:22:55.890006+0000","last_peered":"2024-05-27T07:22:55.890006+0000","last_clean":"2024-05-27T07:22:55.890006+0000","last_became_active":"2024-05-27T07:21:45.791492+0000","last_became_peered":"2024-05-27T07:21:45.791492+0000","last_unstale":"2024-05-27T07:22:55.890006+0000","last_undegraded":"2024-05-27T07:22:55.890006+0000","last_fullsized":"2024-05-27T07:22:55.890006+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:21:44.543638+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:21:44.543638+0000","last_clean_scrub_stamp":"2024-05-27T07:21:44.543638+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T17:36:42.444243+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,9],"acting":[8,0,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":34,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-05-27T07:22:31.626503+0000","last_change":"2024-05-27T07:21:48.594769+0000","last_active":"2024-05-27T07:22:31.626503+0000","last_peered":"2024-05-27T07:22:31.626503+0000","last_clean":"2024-05-27T07:22:31.626503+0000","last_became_active":"2024-05-27T07:21:48.592979+0000","last_became_peered":"2024-05-27T07:21:48.592979+0000","last_unstale":"2024-05-27T07:22:31.626503+0000","last_undegraded":"2024-05-27T07:22:31.626503+0000","last_fullsized":"2024-05-27T07:22:31.626503+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:21:47.565641+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:21:47.565641+0000","last_clean_scrub_stamp":"2024-05-27T07:21:47.565641+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T10:06:48.268230+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,9],"acting":[6,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":24,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-05-27T07:22:30.855263+0000","last_change":"2024-05-27T07:21:59.736405+0000","last_active":"2024-05-27T07:22:30.855263+0000","last_peered":"2024-05-27T07:22:30.855263+0000","last_clean":"2024-05-27T07:22:30.855263+0000","last_became_active":"2024-05-27T07:21:59.735991+0000","last_became_peered":"2024-05-27T07:21:59.735991+0000","last_unstale":"2024-05-27T07:22:30.855263+0000","last_undegraded":"2024-05-27T07:22:30.855263+0000","last_fullsized":"2024-05-27T07:22:30.855263+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:21:58.698952+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:21:58.698952+0000","last_clean_scrub_stamp":"2024-05-27T07:21:58.698952+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T18:32:04.931725+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,10,0,7],"acting":[3,10,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"88'15","reported_seq":55,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-05-27T07:23:16.593341+0000","last_change":"2024-05-27T07:21:45.566513+0000","last_active":"2024-05-27T07:23:16.593341+0000","last_peered":"2024-05-27T07:23:16.593341+0000","last_clean":"2024-05-27T07:23:16.593341+0000","last_became_active":"2024-05-27T07:21:45.566107+0000","last_became_peered":"2024-05-27T07:21:45.566107+0000","last_unstale":"2024-05-27T07:23:16.593341+0000","last_undegraded":"2024-05-27T07:23:16.593341+0000","last_fullsized":"2024-05-27T07:23:16.593341+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:21:44.543638+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:21:44.543638+0000","last_clean_scrub_stamp":"2024-05-27T07:21:44.543638+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T14:11:34.851691+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":27,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,3],"acting":[9,8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":34,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-05-27T07:22:30.855851+0000","last_change":"2024-05-27T07:21:48.593642+0000","last_active":"2024-05-27T07:22:30.855851+0000","last_peered":"2024-05-27T07:22:30.855851+0000","last_clean":"2024-05-27T07:22:30.855851+0000","last_became_active":"2024-05-27T07:21:48.592885+0000","last_became_peered":"2024-05-27T07:21:48.592885+0000","last_unstale":"2024-05-27T07:22:30.855851+0000","last_undegraded":"2024-05-27T07:22:30.855851+0000","last_fullsized":"2024-05-27T07:22:30.855851+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:21:47.565641+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:21:47.565641+0000","last_clean_scrub_stamp":"2024-05-27T07:21:47.565641+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T11:51:27.186560+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,7],"acting":[3,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":24,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-05-27T07:22:30.800760+0000","last_change":"2024-05-27T07:21:59.725533+0000","last_active":"2024-05-27T07:22:30.800760+0000","last_peered":"2024-05-27T07:22:30.800760+0000","last_clean":"2024-05-27T07:22:30.800760+0000","last_became_active":"2024-05-27T07:21:59.724991+0000","last_became_peered":"2024-05-27T07:21:59.724991+0000","last_unstale":"2024-05-27T07:22:30.800760+0000","last_undegraded":"2024-05-27T07:22:30.800760+0000","last_fullsized":"2024-05-27T07:22:30.800760+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:21:58.698952+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:21:58.698952+0000","last_clean_scrub_stamp":"2024-05-27T07:21:58.698952+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T17:22:54.650553+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,4,7],"acting":[2,1,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"88'7","reported_seq":48,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-05-27T07:22:55.888243+0000","last_change":"2024-05-27T07:21:45.572701+0000","last_active":"2024-05-27T07:22:55.888243+0000","last_peered":"2024-05-27T07:22:55.888243+0000","last_clean":"2024-05-27T07:22:55.888243+0000","last_became_active":"2024-05-27T07:21:45.572176+0000","last_became_peered":"2024-05-27T07:21:45.572176+0000","last_unstale":"2024-05-27T07:22:55.888243+0000","last_undegraded":"2024-05-27T07:22:55.888243+0000","last_fullsized":"2024-05-27T07:22:55.888243+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:21:44.543638+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:21:44.543638+0000","last_clean_scrub_stamp":"2024-05-27T07:21:44.543638+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T16:11:20.901250+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,11,6],"acting":[4,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":34,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-05-27T07:22:30.801671+0000","last_change":"2024-05-27T07:21:48.581152+0000","last_active":"2024-05-27T07:22:30.801671+0000","last_peered":"2024-05-27T07:22:30.801671+0000","last_clean":"2024-05-27T07:22:30.801671+0000","last_became_active":"2024-05-27T07:21:48.580543+0000","last_became_peered":"2024-05-27T07:21:48.580543+0000","last_unstale":"2024-05-27T07:22:30.801671+0000","last_undegraded":"2024-05-27T07:22:30.801671+0000","last_fullsized":"2024-05-27T07:22:30.801671+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:21:47.565641+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:21:47.565641+0000","last_clean_scrub_stamp":"2024-05-27T07:21:47.565641+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T18:45:56.732828+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,4],"acting":[0,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":31,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-05-27T07:22:31.417863+0000","last_change":"2024-05-27T07:22:31.417863+0000","last_active":"2024-05-27T07:22:31.417863+0000","last_peered":"2024-05-27T07:22:31.417863+0000","last_clean":"2024-05-27T07:22:31.417863+0000","last_became_active":"2024-05-27T07:22:31.417298+0000","last_became_peered":"2024-05-27T07:22:31.417298+0000","last_unstale":"2024-05-27T07:22:31.417863+0000","last_undegraded":"2024-05-27T07:22:31.417863+0000","last_fullsized":"2024-05-27T07:22:31.417863+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:21:58.698952+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:21:58.698952+0000","last_clean_scrub_stamp":"2024-05-27T07:21:58.698952+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T12:26:13.857477+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,10,11],"acting":[4,5,10,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.8","version":"88'2","reported_seq":42,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-05-27T07:22:52.858389+0000","last_change":"2024-05-27T07:21:45.790914+0000","last_active":"2024-05-27T07:22:52.858389+0000","last_peered":"2024-05-27T07:22:52.858389+0000","last_clean":"2024-05-27T07:22:52.858389+0000","last_became_active":"2024-05-27T07:21:45.790518+0000","last_became_peered":"2024-05-27T07:21:45.790518+0000","last_unstale":"2024-05-27T07:22:52.858389+0000","last_undegraded":"2024-05-27T07:22:52.858389+0000","last_fullsized":"2024-05-27T07:22:52.858389+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:21:44.543638+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:21:44.543638+0000","last_clean_scrub_stamp":"2024-05-27T07:21:44.543638+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T08:50:51.651129+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":34,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-05-27T07:22:30.798819+0000","last_change":"2024-05-27T07:21:48.589605+0000","last_active":"2024-05-27T07:22:30.798819+0000","last_peered":"2024-05-27T07:22:30.798819+0000","last_clean":"2024-05-27T07:22:30.798819+0000","last_became_active":"2024-05-27T07:21:48.588294+0000","last_became_peered":"2024-05-27T07:21:48.588294+0000","last_unstale":"2024-05-27T07:22:30.798819+0000","last_undegraded":"2024-05-27T07:22:30.798819+0000","last_fullsized":"2024-05-27T07:22:30.798819+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:21:47.565641+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:21:47.565641+0000","last_clean_scrub_stamp":"2024-05-27T07:21:47.565641+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T09:02:29.618508+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,10],"acting":[8,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":24,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-05-27T07:22:30.792550+0000","last_change":"2024-05-27T07:21:59.738153+0000","last_active":"2024-05-27T07:22:30.792550+0000","last_peered":"2024-05-27T07:22:30.792550+0000","last_clean":"2024-05-27T07:22:30.792550+0000","last_became_active":"2024-05-27T07:21:59.737394+0000","last_became_peered":"2024-05-27T07:21:59.737394+0000","last_unstale":"2024-05-27T07:22:30.792550+0000","last_undegraded":"2024-05-27T07:22:30.792550+0000","last_fullsized":"2024-05-27T07:22:30.792550+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:21:58.698952+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:21:58.698952+0000","last_clean_scrub_stamp":"2024-05-27T07:21:58.698952+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T17:17:13.877688+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,2,5],"acting":[9,3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.9","version":"88'7","reported_seq":46,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-05-27T07:22:55.889211+0000","last_change":"2024-05-27T07:21:45.562638+0000","last_active":"2024-05-27T07:22:55.889211+0000","last_peered":"2024-05-27T07:22:55.889211+0000","last_clean":"2024-05-27T07:22:55.889211+0000","last_became_active":"2024-05-27T07:21:45.562242+0000","last_became_peered":"2024-05-27T07:21:45.562242+0000","last_unstale":"2024-05-27T07:22:55.889211+0000","last_undegraded":"2024-05-27T07:22:55.889211+0000","last_fullsized":"2024-05-27T07:22:55.889211+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:21:44.543638+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:21:44.543638+0000","last_clean_scrub_stamp":"2024-05-27T07:21:44.543638+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T10:40:15.847513+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,9,10],"acting":[3,9,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":34,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-05-27T07:22:30.793275+0000","last_change":"2024-05-27T07:21:48.598186+0000","last_active":"2024-05-27T07:22:30.793275+0000","last_peered":"2024-05-27T07:22:30.793275+0000","last_clean":"2024-05-27T07:22:30.793275+0000","last_became_active":"2024-05-27T07:21:48.597715+0000","last_became_peered":"2024-05-27T07:21:48.597715+0000","last_unstale":"2024-05-27T07:22:30.793275+0000","last_undegraded":"2024-05-27T07:22:30.793275+0000","last_fullsized":"2024-05-27T07:22:30.793275+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:21:47.565641+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:21:47.565641+0000","last_clean_scrub_stamp":"2024-05-27T07:21:47.565641+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T08:01:35.983687+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,2],"acting":[11,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.c","version":"88'1","reported_seq":25,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-05-27T07:22:30.795901+0000","last_change":"2024-05-27T07:21:59.727247+0000","last_active":"2024-05-27T07:22:30.795901+0000","last_peered":"2024-05-27T07:22:30.795901+0000","last_clean":"2024-05-27T07:22:30.795901+0000","last_became_active":"2024-05-27T07:21:59.726788+0000","last_became_peered":"2024-05-27T07:21:59.726788+0000","last_unstale":"2024-05-27T07:22:30.795901+0000","last_undegraded":"2024-05-27T07:22:30.795901+0000","last_fullsized":"2024-05-27T07:22:30.795901+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:21:58.698952+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:21:58.698952+0000","last_clean_scrub_stamp":"2024-05-27T07:21:58.698952+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T10:09:27.095188+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,9,7,8],"acting":[4,9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.a","version":"88'3","reported_seq":42,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-05-27T07:22:54.879698+0000","last_change":"2024-05-27T07:21:45.569285+0000","last_active":"2024-05-27T07:22:54.879698+0000","last_peered":"2024-05-27T07:22:54.879698+0000","last_clean":"2024-05-27T07:22:54.879698+0000","last_became_active":"2024-05-27T07:21:45.568835+0000","last_became_peered":"2024-05-27T07:21:45.568835+0000","last_unstale":"2024-05-27T07:22:54.879698+0000","last_undegraded":"2024-05-27T07:22:54.879698+0000","last_fullsized":"2024-05-27T07:22:54.879698+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:21:44.543638+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:21:44.543638+0000","last_clean_scrub_stamp":"2024-05-27T07:21:44.543638+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T10:03:23.353800+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,8],"acting":[2,3,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":34,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-05-27T07:22:30.855702+0000","last_change":"2024-05-27T07:21:48.583494+0000","last_active":"2024-05-27T07:22:30.855702+0000","last_peered":"2024-05-27T07:22:30.855702+0000","last_clean":"2024-05-27T07:22:30.855702+0000","last_became_active":"2024-05-27T07:21:48.583141+0000","last_became_peered":"2024-05-27T07:21:48.583141+0000","last_unstale":"2024-05-27T07:22:30.855702+0000","last_undegraded":"2024-05-27T07:22:30.855702+0000","last_fullsized":"2024-05-27T07:22:30.855702+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:21:47.565641+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:21:47.565641+0000","last_clean_scrub_stamp":"2024-05-27T07:21:47.565641+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T10:24:50.475195+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,10],"acting":[3,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":24,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-05-27T07:22:30.796714+0000","last_change":"2024-05-27T07:21:59.733966+0000","last_active":"2024-05-27T07:22:30.796714+0000","last_peered":"2024-05-27T07:22:30.796714+0000","last_clean":"2024-05-27T07:22:30.796714+0000","last_became_active":"2024-05-27T07:21:59.733411+0000","last_became_peered":"2024-05-27T07:21:59.733411+0000","last_unstale":"2024-05-27T07:22:30.796714+0000","last_undegraded":"2024-05-27T07:22:30.796714+0000","last_fullsized":"2024-05-27T07:22:30.796714+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:21:58.698952+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:21:58.698952+0000","last_clean_scrub_stamp":"2024-05-27T07:21:58.698952+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T15:08:04.918086+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,10,3],"acting":[4,2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.b","version":"88'4","reported_seq":45,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-05-27T07:22:52.858809+0000","last_change":"2024-05-27T07:21:45.570967+0000","last_active":"2024-05-27T07:22:52.858809+0000","last_peered":"2024-05-27T07:22:52.858809+0000","last_clean":"2024-05-27T07:22:52.858809+0000","last_became_active":"2024-05-27T07:21:45.570501+0000","last_became_peered":"2024-05-27T07:21:45.570501+0000","last_unstale":"2024-05-27T07:22:52.858809+0000","last_undegraded":"2024-05-27T07:22:52.858809+0000","last_fullsized":"2024-05-27T07:22:52.858809+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:21:44.543638+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:21:44.543638+0000","last_clean_scrub_stamp":"2024-05-27T07:21:44.543638+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T14:11:26.496194+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":32,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-05-27T07:22:30.796885+0000","last_change":"2024-05-27T07:21:48.594690+0000","last_active":"2024-05-27T07:22:30.796885+0000","last_peered":"2024-05-27T07:22:30.796885+0000","last_clean":"2024-05-27T07:22:30.796885+0000","last_became_active":"2024-05-27T07:21:48.594314+0000","last_became_peered":"2024-05-27T07:21:48.594314+0000","last_unstale":"2024-05-27T07:22:30.796885+0000","last_undegraded":"2024-05-27T07:22:30.796885+0000","last_fullsized":"2024-05-27T07:22:30.796885+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:21:47.565641+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:21:47.565641+0000","last_clean_scrub_stamp":"2024-05-27T07:21:47.565641+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T14:06:43.401844+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,6],"acting":[7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":24,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-05-27T07:22:31.579716+0000","last_change":"2024-05-27T07:21:59.732289+0000","last_active":"2024-05-27T07:22:31.579716+0000","last_peered":"2024-05-27T07:22:31.579716+0000","last_clean":"2024-05-27T07:22:31.579716+0000","last_became_active":"2024-05-27T07:21:59.731863+0000","last_became_peered":"2024-05-27T07:21:59.731863+0000","last_unstale":"2024-05-27T07:22:31.579716+0000","last_undegraded":"2024-05-27T07:22:31.579716+0000","last_fullsized":"2024-05-27T07:22:31.579716+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:21:58.698952+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:21:58.698952+0000","last_clean_scrub_stamp":"2024-05-27T07:21:58.698952+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T11:06:38.246484+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,6,7],"acting":[1,5,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.c","version":"88'5","reported_seq":71,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-05-27T07:23:19.929278+0000","last_change":"2024-05-27T07:21:45.791978+0000","last_active":"2024-05-27T07:23:19.929278+0000","last_peered":"2024-05-27T07:23:19.929278+0000","last_clean":"2024-05-27T07:23:19.929278+0000","last_became_active":"2024-05-27T07:21:45.791146+0000","last_became_peered":"2024-05-27T07:21:45.791146+0000","last_unstale":"2024-05-27T07:23:19.929278+0000","last_undegraded":"2024-05-27T07:23:19.929278+0000","last_fullsized":"2024-05-27T07:23:19.929278+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:21:44.543638+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:21:44.543638+0000","last_clean_scrub_stamp":"2024-05-27T07:21:44.543638+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T09:09:24.952876+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1170,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":27,"num_read_kb":2,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,11,0],"acting":[2,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":32,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-05-27T07:22:30.797609+0000","last_change":"2024-05-27T07:21:48.602786+0000","last_active":"2024-05-27T07:22:30.797609+0000","last_peered":"2024-05-27T07:22:30.797609+0000","last_clean":"2024-05-27T07:22:30.797609+0000","last_became_active":"2024-05-27T07:21:48.602476+0000","last_became_peered":"2024-05-27T07:21:48.602476+0000","last_unstale":"2024-05-27T07:22:30.797609+0000","last_undegraded":"2024-05-27T07:22:30.797609+0000","last_fullsized":"2024-05-27T07:22:30.797609+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:21:47.565641+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:21:47.565641+0000","last_clean_scrub_stamp":"2024-05-27T07:21:47.565641+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T13:22:58.583230+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,11],"acting":[7,1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":24,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-05-27T07:22:30.801096+0000","last_change":"2024-05-27T07:21:59.729595+0000","last_active":"2024-05-27T07:22:30.801096+0000","last_peered":"2024-05-27T07:22:30.801096+0000","last_clean":"2024-05-27T07:22:30.801096+0000","last_became_active":"2024-05-27T07:21:59.729147+0000","last_became_peered":"2024-05-27T07:21:59.729147+0000","last_unstale":"2024-05-27T07:22:30.801096+0000","last_undegraded":"2024-05-27T07:22:30.801096+0000","last_fullsized":"2024-05-27T07:22:30.801096+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:21:58.698952+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:21:58.698952+0000","last_clean_scrub_stamp":"2024-05-27T07:21:58.698952+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T13:31:28.579617+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,9,1,4],"acting":[0,9,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.d","version":"88'3","reported_seq":43,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-05-27T07:22:55.889855+0000","last_change":"2024-05-27T07:21:45.569287+0000","last_active":"2024-05-27T07:22:55.889855+0000","last_peered":"2024-05-27T07:22:55.889855+0000","last_clean":"2024-05-27T07:22:55.889855+0000","last_became_active":"2024-05-27T07:21:45.568935+0000","last_became_peered":"2024-05-27T07:21:45.568935+0000","last_unstale":"2024-05-27T07:22:55.889855+0000","last_undegraded":"2024-05-27T07:22:55.889855+0000","last_fullsized":"2024-05-27T07:22:55.889855+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:21:44.543638+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:21:44.543638+0000","last_clean_scrub_stamp":"2024-05-27T07:21:44.543638+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T07:27:14.674135+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,10,3],"acting":[5,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":32,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-05-27T07:22:30.796957+0000","last_change":"2024-05-27T07:21:48.597718+0000","last_active":"2024-05-27T07:22:30.796957+0000","last_peered":"2024-05-27T07:22:30.796957+0000","last_clean":"2024-05-27T07:22:30.796957+0000","last_became_active":"2024-05-27T07:21:48.597379+0000","last_became_peered":"2024-05-27T07:21:48.597379+0000","last_unstale":"2024-05-27T07:22:30.796957+0000","last_undegraded":"2024-05-27T07:22:30.796957+0000","last_fullsized":"2024-05-27T07:22:30.796957+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:21:47.565641+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:21:47.565641+0000","last_clean_scrub_stamp":"2024-05-27T07:21:47.565641+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T08:48:41.658364+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,5],"acting":[7,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":24,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-05-27T07:22:30.796397+0000","last_change":"2024-05-27T07:21:59.730191+0000","last_active":"2024-05-27T07:22:30.796397+0000","last_peered":"2024-05-27T07:22:30.796397+0000","last_clean":"2024-05-27T07:22:30.796397+0000","last_became_active":"2024-05-27T07:21:59.729787+0000","last_became_peered":"2024-05-27T07:21:59.729787+0000","last_unstale":"2024-05-27T07:22:30.796397+0000","last_undegraded":"2024-05-27T07:22:30.796397+0000","last_fullsized":"2024-05-27T07:22:30.796397+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:21:58.698952+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:21:58.698952+0000","last_clean_scrub_stamp":"2024-05-27T07:21:58.698952+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T13:22:38.961246+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7,1],"acting":[5,2,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.e","version":"88'5","reported_seq":44,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-05-27T07:22:55.887608+0000","last_change":"2024-05-27T07:21:45.559219+0000","last_active":"2024-05-27T07:22:55.887608+0000","last_peered":"2024-05-27T07:22:55.887608+0000","last_clean":"2024-05-27T07:22:55.887608+0000","last_became_active":"2024-05-27T07:21:45.558848+0000","last_became_peered":"2024-05-27T07:21:45.558848+0000","last_unstale":"2024-05-27T07:22:55.887608+0000","last_undegraded":"2024-05-27T07:22:55.887608+0000","last_fullsized":"2024-05-27T07:22:55.887608+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:21:44.543638+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:21:44.543638+0000","last_clean_scrub_stamp":"2024-05-27T07:21:44.543638+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T15:03:55.632162+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":32,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-05-27T07:22:30.797617+0000","last_change":"2024-05-27T07:21:48.596349+0000","last_active":"2024-05-27T07:22:30.797617+0000","last_peered":"2024-05-27T07:22:30.797617+0000","last_clean":"2024-05-27T07:22:30.797617+0000","last_became_active":"2024-05-27T07:21:48.595830+0000","last_became_peered":"2024-05-27T07:21:48.595830+0000","last_unstale":"2024-05-27T07:22:30.797617+0000","last_undegraded":"2024-05-27T07:22:30.797617+0000","last_fullsized":"2024-05-27T07:22:30.797617+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:21:47.565641+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:21:47.565641+0000","last_clean_scrub_stamp":"2024-05-27T07:21:47.565641+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T13:33:11.431988+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,8],"acting":[7,11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.20","version":"0'0","reported_seq":34,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-05-27T07:22:30.793403+0000","last_change":"2024-05-27T07:21:48.598504+0000","last_active":"2024-05-27T07:22:30.793403+0000","last_peered":"2024-05-27T07:22:30.793403+0000","last_clean":"2024-05-27T07:22:30.793403+0000","last_became_active":"2024-05-27T07:21:48.598103+0000","last_became_peered":"2024-05-27T07:21:48.598103+0000","last_unstale":"2024-05-27T07:22:30.793403+0000","last_undegraded":"2024-05-27T07:22:30.793403+0000","last_fullsized":"2024-05-27T07:22:30.793403+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:21:47.565641+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:21:47.565641+0000","last_clean_scrub_stamp":"2024-05-27T07:21:47.565641+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T10:34:02.743735+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,5],"acting":[11,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.21","version":"0'0","reported_seq":34,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-05-27T07:22:30.801101+0000","last_change":"2024-05-27T07:21:48.593379+0000","last_active":"2024-05-27T07:22:30.801101+0000","last_peered":"2024-05-27T07:22:30.801101+0000","last_clean":"2024-05-27T07:22:30.801101+0000","last_became_active":"2024-05-27T07:21:48.592995+0000","last_became_peered":"2024-05-27T07:21:48.592995+0000","last_unstale":"2024-05-27T07:22:30.801101+0000","last_undegraded":"2024-05-27T07:22:30.801101+0000","last_fullsized":"2024-05-27T07:22:30.801101+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:21:47.565641+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:21:47.565641+0000","last_clean_scrub_stamp":"2024-05-27T07:21:47.565641+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T19:01:56.288235+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,4],"acting":[2,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"0'0","reported_seq":34,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-05-27T07:22:30.800545+0000","last_change":"2024-05-27T07:21:48.592837+0000","last_active":"2024-05-27T07:22:30.800545+0000","last_peered":"2024-05-27T07:22:30.800545+0000","last_clean":"2024-05-27T07:22:30.800545+0000","last_became_active":"2024-05-27T07:21:48.592459+0000","last_became_peered":"2024-05-27T07:21:48.592459+0000","last_unstale":"2024-05-27T07:22:30.800545+0000","last_undegraded":"2024-05-27T07:22:30.800545+0000","last_fullsized":"2024-05-27T07:22:30.800545+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:21:47.565641+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:21:47.565641+0000","last_clean_scrub_stamp":"2024-05-27T07:21:47.565641+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T14:00:19.490348+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.23","version":"0'0","reported_seq":34,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-05-27T07:22:30.801437+0000","last_change":"2024-05-27T07:21:48.583139+0000","last_active":"2024-05-27T07:22:30.801437+0000","last_peered":"2024-05-27T07:22:30.801437+0000","last_clean":"2024-05-27T07:22:30.801437+0000","last_became_active":"2024-05-27T07:21:48.582822+0000","last_became_peered":"2024-05-27T07:21:48.582822+0000","last_unstale":"2024-05-27T07:22:30.801437+0000","last_undegraded":"2024-05-27T07:22:30.801437+0000","last_fullsized":"2024-05-27T07:22:30.801437+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:21:47.565641+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:21:47.565641+0000","last_clean_scrub_stamp":"2024-05-27T07:21:47.565641+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T16:30:03.454035+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,4],"acting":[0,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"0'0","reported_seq":34,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-05-27T07:22:31.626460+0000","last_change":"2024-05-27T07:21:48.588906+0000","last_active":"2024-05-27T07:22:31.626460+0000","last_peered":"2024-05-27T07:22:31.626460+0000","last_clean":"2024-05-27T07:22:31.626460+0000","last_became_active":"2024-05-27T07:21:48.588555+0000","last_became_peered":"2024-05-27T07:21:48.588555+0000","last_unstale":"2024-05-27T07:22:31.626460+0000","last_undegraded":"2024-05-27T07:22:31.626460+0000","last_fullsized":"2024-05-27T07:22:31.626460+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:21:47.565641+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:21:47.565641+0000","last_clean_scrub_stamp":"2024-05-27T07:21:47.565641+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T19:04:23.006852+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"0'0","reported_seq":34,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-05-27T07:22:30.793772+0000","last_change":"2024-05-27T07:21:48.590271+0000","last_active":"2024-05-27T07:22:30.793772+0000","last_peered":"2024-05-27T07:22:30.793772+0000","last_clean":"2024-05-27T07:22:30.793772+0000","last_became_active":"2024-05-27T07:21:48.589898+0000","last_became_peered":"2024-05-27T07:21:48.589898+0000","last_unstale":"2024-05-27T07:22:30.793772+0000","last_undegraded":"2024-05-27T07:22:30.793772+0000","last_fullsized":"2024-05-27T07:22:30.793772+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:21:47.565641+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:21:47.565641+0000","last_clean_scrub_stamp":"2024-05-27T07:21:47.565641+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T11:57:43.238211+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,5],"acting":[11,8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.26","version":"0'0","reported_seq":34,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-05-27T07:22:30.800884+0000","last_change":"2024-05-27T07:21:48.597321+0000","last_active":"2024-05-27T07:22:30.800884+0000","last_peered":"2024-05-27T07:22:30.800884+0000","last_clean":"2024-05-27T07:22:30.800884+0000","last_became_active":"2024-05-27T07:21:48.596953+0000","last_became_peered":"2024-05-27T07:21:48.596953+0000","last_unstale":"2024-05-27T07:22:30.800884+0000","last_undegraded":"2024-05-27T07:22:30.800884+0000","last_fullsized":"2024-05-27T07:22:30.800884+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:21:47.565641+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:21:47.565641+0000","last_clean_scrub_stamp":"2024-05-27T07:21:47.565641+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T16:21:17.645098+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,8,1],"acting":[2,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.27","version":"0'0","reported_seq":34,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-05-27T07:22:30.855978+0000","last_change":"2024-05-27T07:21:48.593717+0000","last_active":"2024-05-27T07:22:30.855978+0000","last_peered":"2024-05-27T07:22:30.855978+0000","last_clean":"2024-05-27T07:22:30.855978+0000","last_became_active":"2024-05-27T07:21:48.593356+0000","last_became_peered":"2024-05-27T07:21:48.593356+0000","last_unstale":"2024-05-27T07:22:30.855978+0000","last_undegraded":"2024-05-27T07:22:30.855978+0000","last_fullsized":"2024-05-27T07:22:30.855978+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:21:47.565641+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:21:47.565641+0000","last_clean_scrub_stamp":"2024-05-27T07:21:47.565641+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T17:06:27.341627+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"0'0","reported_seq":34,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-05-27T07:22:30.796577+0000","last_change":"2024-05-27T07:21:48.592360+0000","last_active":"2024-05-27T07:22:30.796577+0000","last_peered":"2024-05-27T07:22:30.796577+0000","last_clean":"2024-05-27T07:22:30.796577+0000","last_became_active":"2024-05-27T07:21:48.591650+0000","last_became_peered":"2024-05-27T07:21:48.591650+0000","last_unstale":"2024-05-27T07:22:30.796577+0000","last_undegraded":"2024-05-27T07:22:30.796577+0000","last_fullsized":"2024-05-27T07:22:30.796577+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:21:47.565641+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:21:47.565641+0000","last_clean_scrub_stamp":"2024-05-27T07:21:47.565641+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T17:32:19.408787+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,3],"acting":[5,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"0'0","reported_seq":34,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-05-27T07:22:30.856019+0000","last_change":"2024-05-27T07:21:48.587319+0000","last_active":"2024-05-27T07:22:30.856019+0000","last_peered":"2024-05-27T07:22:30.856019+0000","last_clean":"2024-05-27T07:22:30.856019+0000","last_became_active":"2024-05-27T07:21:48.586924+0000","last_became_peered":"2024-05-27T07:21:48.586924+0000","last_unstale":"2024-05-27T07:22:30.856019+0000","last_undegraded":"2024-05-27T07:22:30.856019+0000","last_fullsized":"2024-05-27T07:22:30.856019+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:21:47.565641+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:21:47.565641+0000","last_clean_scrub_stamp":"2024-05-27T07:21:47.565641+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T09:28:48.129037+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,11],"acting":[3,2,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2a","version":"0'0","reported_seq":34,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-05-27T07:22:31.580004+0000","last_change":"2024-05-27T07:21:48.604018+0000","last_active":"2024-05-27T07:22:31.580004+0000","last_peered":"2024-05-27T07:22:31.580004+0000","last_clean":"2024-05-27T07:22:31.580004+0000","last_became_active":"2024-05-27T07:21:48.603368+0000","last_became_peered":"2024-05-27T07:21:48.603368+0000","last_unstale":"2024-05-27T07:22:31.580004+0000","last_undegraded":"2024-05-27T07:22:31.580004+0000","last_fullsized":"2024-05-27T07:22:31.580004+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:21:47.565641+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:21:47.565641+0000","last_clean_scrub_stamp":"2024-05-27T07:21:47.565641+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T09:49:42.440480+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2b","version":"0'0","reported_seq":34,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-05-27T07:22:31.626316+0000","last_change":"2024-05-27T07:21:48.589434+0000","last_active":"2024-05-27T07:22:31.626316+0000","last_peered":"2024-05-27T07:22:31.626316+0000","last_clean":"2024-05-27T07:22:31.626316+0000","last_became_active":"2024-05-27T07:21:48.589102+0000","last_became_peered":"2024-05-27T07:21:48.589102+0000","last_unstale":"2024-05-27T07:22:31.626316+0000","last_undegraded":"2024-05-27T07:22:31.626316+0000","last_fullsized":"2024-05-27T07:22:31.626316+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:21:47.565641+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:21:47.565641+0000","last_clean_scrub_stamp":"2024-05-27T07:21:47.565641+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T18:28:52.955138+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2c","version":"0'0","reported_seq":40,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-05-27T07:22:30.802433+0000","last_change":"2024-05-27T07:22:30.802433+0000","last_active":"2024-05-27T07:22:30.802433+0000","last_peered":"2024-05-27T07:22:30.802433+0000","last_clean":"2024-05-27T07:22:30.802433+0000","last_became_active":"2024-05-27T07:22:30.801881+0000","last_became_peered":"2024-05-27T07:22:30.801881+0000","last_unstale":"2024-05-27T07:22:30.802433+0000","last_undegraded":"2024-05-27T07:22:30.802433+0000","last_fullsized":"2024-05-27T07:22:30.802433+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:21:47.565641+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:21:47.565641+0000","last_clean_scrub_stamp":"2024-05-27T07:21:47.565641+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T11:59:17.916052+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,9,8],"acting":[11,9,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":234,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":32768,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":128,"acting":128,"num_store_stats":12},{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":192,"acting":192,"num_store_stats":12},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":81,"num_read_kb":69,"num_write":123,"num_write_kb":2150,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":2951840,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":376832,"data_stored":4132576,"data_compressed":249874,"data_compressed_allocated":315392,"data_compressed_original":4096000,"omap_allocated":0,"internal_metadata":0},"log_size":92,"ondisk_log_size":92,"up":3,"acting":3,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":63364,"num_objects":98,"num_object_clones":0,"num_object_copies":294,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":98,"num_whiteouts":0,"num_read":279,"num_read_kb":208,"num_write":246,"num_write_kb":134,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":22,"num_keys_recovered":0,"num_objects_omap":66,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":569344,"data_stored":203704,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":211,"ondisk_log_size":211,"up":96,"acting":96,"num_store_stats":12}],"osd_stats":[{"osd":11,"up_from":71,"seq":304942678050,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32128,"kb_used_data":952,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713024,"statfs":{"total":95995035648,"available":95962136576,"internally_reserved":0,"allocated":974848,"data_stored":500494,"data_compressed":26453,"data_compressed_allocated":36864,"data_compressed_original":73728,"omap_allocated":1596,"internal_metadata":31914436},"hb_peers":[0,1,2,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":65,"seq":279172874278,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28104,"kb_used_data":1088,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717048,"statfs":{"total":95995035648,"available":95966257152,"internally_reserved":0,"allocated":1114112,"data_stored":1143839,"data_compressed":62613,"data_compressed_allocated":81920,"data_compressed_original":663552,"omap_allocated":1600,"internal_metadata":27654592},"hb_peers":[0,1,2,3,4,5,6,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":60,"seq":257698037804,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36312,"kb_used_data":976,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708840,"statfs":{"total":95995035648,"available":95957852160,"internally_reserved":0,"allocated":999424,"data_stored":504498,"data_compressed":26453,"data_compressed_allocated":36864,"data_compressed_original":73728,"omap_allocated":1600,"internal_metadata":36174272},"hb_peers":[0,1,2,3,4,5,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":54,"seq":231928234034,"num_pgs":31,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27980,"kb_used_data":964,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717172,"statfs":{"total":95995035648,"available":95966384128,"internally_reserved":0,"allocated":987136,"data_stored":500758,"data_compressed":26453,"data_compressed_allocated":36864,"data_compressed_original":73728,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":48,"seq":206158430262,"num_pgs":37,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32196,"kb_used_data":1020,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712956,"statfs":{"total":95995035648,"available":95962066944,"internally_reserved":0,"allocated":1044480,"data_stored":1097668,"data_compressed":62613,"data_compressed_allocated":81920,"data_compressed_original":663552,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":43,"seq":184683593787,"num_pgs":32,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27964,"kb_used_data":948,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717188,"statfs":{"total":95995035648,"available":95966400512,"internally_reserved":0,"allocated":970752,"data_stored":495906,"data_compressed":26453,"data_compressed_allocated":36864,"data_compressed_original":73728,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574933,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32192,"kb_used_data":1016,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712960,"statfs":{"total":95995035648,"available":95962071040,"internally_reserved":0,"allocated":1040384,"data_stored":559902,"data_compressed":26453,"data_compressed_allocated":36864,"data_compressed_original":73728,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738457,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27988,"kb_used_data":972,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717164,"statfs":{"total":95995035648,"available":95966375936,"internally_reserved":0,"allocated":995328,"data_stored":497362,"data_compressed":26453,"data_compressed_allocated":36864,"data_compressed_original":73728,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411408,"num_pgs":36,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32196,"kb_used_data":1020,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712956,"statfs":{"total":95995035648,"available":95962066944,"internally_reserved":0,"allocated":1044480,"data_stored":553595,"data_compressed":26453,"data_compressed_allocated":36864,"data_compressed_original":73728,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":26,"seq":111669149771,"num_pgs":40,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36296,"kb_used_data":960,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708856,"statfs":{"total":95995035648,"available":95957868544,"internally_reserved":0,"allocated":983040,"data_stored":495614,"data_compressed":26453,"data_compressed_allocated":36864,"data_compressed_original":73728,"omap_allocated":1593,"internal_metadata":36174279},"hb_peers":[0,1,2,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":32,"seq":137438953541,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27980,"kb_used_data":964,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717172,"statfs":{"total":95995035648,"available":95966384128,"internally_reserved":0,"allocated":987136,"data_stored":500483,"data_compressed":26453,"data_compressed_allocated":36864,"data_compressed_original":73728,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":37,"seq":158913790016,"num_pgs":44,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28044,"kb_used_data":1028,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717108,"statfs":{"total":95995035648,"available":95966318592,"internally_reserved":0,"allocated":1052672,"data_stored":1092859,"data_compressed":62613,"data_compressed_allocated":81920,"data_compressed_original":663552,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":34537,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":34537,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":36160,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":36160,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":36160,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":36160,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":36160,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":2178,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":86016,"data_stored":64718,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":98304,"data_stored":65217,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":430,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1203,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":3211,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":722,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":3924,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1478,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":1122,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":110592,"data_stored":58287,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1214,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":8192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":8192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-05-27T07:23:26.367 INFO:tasks.cephadm.ceph_manager.ceph:clean! 2024-05-27T07:23:26.367 INFO:tasks.cephadm:Setting up client nodes... 2024-05-27T07:23:26.368 DEBUG:teuthology.orchestra.run.smithi052:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:38f9d900c83c63799fbdbe61acc9a11b0d3554a6 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 73df2636-1bf8-11ef-bc9b-c7b262605968 -- ceph auth get-or-create client.0 mon 'allow *' osd 'allow *' mds 'allow *' mgr 'allow *' 2024-05-27T07:23:26.628 INFO:teuthology.orchestra.run.smithi052.stderr:Inferring config /var/lib/ceph/73df2636-1bf8-11ef-bc9b-c7b262605968/mon.a/config 2024-05-27T07:23:27.004 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:23:26 smithi052 ceph-mon[31955]: from='client.14859 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-05-27T07:23:27.004 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:23:26 smithi052 ceph-mon[31955]: pgmap v346: 129 pgs: 129 active+clean; 645 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 8.7 KiB/s rd, 3.6 KiB/s wr, 13 op/s 2024-05-27T07:23:27.352 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:23:26 smithi173 ceph-mon[37000]: from='client.14859 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-05-27T07:23:27.352 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:23:26 smithi173 ceph-mon[37000]: pgmap v346: 129 pgs: 129 active+clean; 645 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 8.7 KiB/s rd, 3.6 KiB/s wr, 13 op/s 2024-05-27T07:23:27.441 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:23:26 smithi105 ceph-mon[35977]: from='client.14859 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-05-27T07:23:27.441 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:23:26 smithi105 ceph-mon[35977]: pgmap v346: 129 pgs: 129 active+clean; 645 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 8.7 KiB/s rd, 3.6 KiB/s wr, 13 op/s 2024-05-27T07:23:28.436 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:23:28.446+0000 7f60011e2700 1 -- 172.21.15.52:0/654831948 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f5ffc1059a0 msgr2=0x7f5ffc105db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:23:28.436 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:23:28.446+0000 7f60011e2700 1 --2- 172.21.15.52:0/654831948 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f5ffc1059a0 0x7f5ffc105db0 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7f5fec00b0a0 tx=0x7f5fec00b3b0 comp rx=0 tx=0).stop 2024-05-27T07:23:28.436 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:23:28.446+0000 7f60011e2700 1 -- 172.21.15.52:0/654831948 shutdown_connections 2024-05-27T07:23:28.436 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:23:28.446+0000 7f60011e2700 1 --2- 172.21.15.52:0/654831948 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f5ffc1049d0 0x7f5ffc10c6e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:23:28.437 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:23:28.446+0000 7f60011e2700 1 --2- 172.21.15.52:0/654831948 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f5ffc1059a0 0x7f5ffc105db0 unknown :-1 s=CLOSED pgs=245 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:23:28.437 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:23:28.446+0000 7f60011e2700 1 --2- 172.21.15.52:0/654831948 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f5ffc103f50 0x7f5ffc104360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:23:28.437 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:23:28.446+0000 7f60011e2700 1 -- 172.21.15.52:0/654831948 >> 172.21.15.52:0/654831948 conn(0x7f5ffc0ff9c0 msgr2=0x7f5ffc101e50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:23:28.437 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:23:28.446+0000 7f60011e2700 1 -- 172.21.15.52:0/654831948 shutdown_connections 2024-05-27T07:23:28.437 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:23:28.446+0000 7f60011e2700 1 -- 172.21.15.52:0/654831948 wait complete. 2024-05-27T07:23:28.439 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:23:28.447+0000 7f60011e2700 1 Processor -- start 2024-05-27T07:23:28.439 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:23:28.447+0000 7f60011e2700 1 -- start start 2024-05-27T07:23:28.439 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:23:28.447+0000 7f60011e2700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f5ffc103f50 0x7f5ffc19cc20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:23:28.439 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:23:28.447+0000 7f60011e2700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f5ffc1049d0 0x7f5ffc19d160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:23:28.439 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:23:28.447+0000 7f60011e2700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f5ffc1059a0 0x7f5ffc1a4100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:23:28.439 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:23:28.447+0000 7f60011e2700 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x7f5ffc10fec0 con 0x7f5ffc103f50 2024-05-27T07:23:28.439 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:23:28.447+0000 7f60011e2700 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f5ffc10fd40 con 0x7f5ffc1059a0 2024-05-27T07:23:28.440 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:23:28.447+0000 7f60011e2700 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f5ffc110040 con 0x7f5ffc1049d0 2024-05-27T07:23:28.440 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:23:28.447+0000 7f5ffad9d700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f5ffc103f50 0x7f5ffc19cc20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:23:28.440 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:23:28.447+0000 7f5ffad9d700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f5ffc103f50 0x7f5ffc19cc20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.52:39776/0 (socket says 172.21.15.52:39776) 2024-05-27T07:23:28.440 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:23:28.447+0000 7f5ffad9d700 1 -- 172.21.15.52:0/2687991003 learned_addr learned my addr 172.21.15.52:0/2687991003 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-05-27T07:23:28.440 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:23:28.448+0000 7f5ffa59c700 1 --2- 172.21.15.52:0/2687991003 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f5ffc1049d0 0x7f5ffc19d160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:23:28.440 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:23:28.448+0000 7f5ffb59e700 1 --2- 172.21.15.52:0/2687991003 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f5ffc1059a0 0x7f5ffc1a4100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:23:28.440 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:23:28.448+0000 7f5ffad9d700 1 -- 172.21.15.52:0/2687991003 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f5ffc1049d0 msgr2=0x7f5ffc19d160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:23:28.440 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:23:28.448+0000 7f5ffad9d700 1 --2- 172.21.15.52:0/2687991003 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f5ffc1049d0 0x7f5ffc19d160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:23:28.440 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:23:28.448+0000 7f5ffad9d700 1 -- 172.21.15.52:0/2687991003 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f5ffc1059a0 msgr2=0x7f5ffc1a4100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:23:28.440 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:23:28.448+0000 7f5ffad9d700 1 --2- 172.21.15.52:0/2687991003 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f5ffc1059a0 0x7f5ffc1a4100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:23:28.440 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:23:28.448+0000 7f5ffad9d700 1 -- 172.21.15.52:0/2687991003 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5ffc1a4780 con 0x7f5ffc103f50 2024-05-27T07:23:28.440 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:23:28.448+0000 7f5ffad9d700 1 --2- 172.21.15.52:0/2687991003 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f5ffc103f50 0x7f5ffc19cc20 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7f5fec000f80 tx=0x7f5fec004e40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:23:28.440 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:23:28.448+0000 7f5ff3fff700 1 -- 172.21.15.52:0/2687991003 <== mon.0 v2:172.21.15.52:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5fec021070 con 0x7f5ffc103f50 2024-05-27T07:23:28.441 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:23:28.448+0000 7f5ff3fff700 1 -- 172.21.15.52:0/2687991003 <== mon.0 v2:172.21.15.52:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f5fec007b80 con 0x7f5ffc103f50 2024-05-27T07:23:28.441 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:23:28.448+0000 7f60011e2700 1 -- 172.21.15.52:0/2687991003 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5ffc1a4a10 con 0x7f5ffc103f50 2024-05-27T07:23:28.441 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:23:28.448+0000 7f5ff3fff700 1 -- 172.21.15.52:0/2687991003 <== mon.0 v2:172.21.15.52:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5fec005040 con 0x7f5ffc103f50 2024-05-27T07:23:28.441 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:23:28.449+0000 7f60011e2700 1 -- 172.21.15.52:0/2687991003 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5ffc1a4fa0 con 0x7f5ffc103f50 2024-05-27T07:23:28.441 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:23:28.450+0000 7f60011e2700 1 -- 172.21.15.52:0/2687991003 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5ffc0512b0 con 0x7f5ffc103f50 2024-05-27T07:23:28.441 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:23:28.451+0000 7f5ff3fff700 1 -- 172.21.15.52:0/2687991003 <== mon.0 v2:172.21.15.52:3300/0 4 ==== mgrmap(e 17) ==== 149703+0+0 (secure 0 0 0) 0x7f5fec024070 con 0x7f5ffc103f50 2024-05-27T07:23:28.442 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:23:28.452+0000 7f5ff3fff700 1 --2- 172.21.15.52:0/2687991003 >> [v2:172.21.15.52:6800/3513075805,v1:172.21.15.52:6801/3513075805] conn(0x7f5fe80b2990 0x7f5fe80b4ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:23:28.442 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:23:28.452+0000 7f5ffa59c700 1 --2- 172.21.15.52:0/2687991003 >> [v2:172.21.15.52:6800/3513075805,v1:172.21.15.52:6801/3513075805] conn(0x7f5fe80b2990 0x7f5fe80b4ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:23:28.442 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:23:28.452+0000 7f5ff3fff700 1 -- 172.21.15.52:0/2687991003 <== mon.0 v2:172.21.15.52:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 9092+0+0 (secure 0 0 0) 0x7f5fec00f020 con 0x7f5ffc103f50 2024-05-27T07:23:28.442 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:23:28.453+0000 7f5ffa59c700 1 --2- 172.21.15.52:0/2687991003 >> [v2:172.21.15.52:6800/3513075805,v1:172.21.15.52:6801/3513075805] conn(0x7f5fe80b2990 0x7f5fe80b4ea0 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7f5ffc19dcd0 tx=0x7f5fe400b410 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:23:28.446 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:23:28.457+0000 7f5ff3fff700 1 -- 172.21.15.52:0/2687991003 <== mon.0 v2:172.21.15.52:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191963 (secure 0 0 0) 0x7f5fec0e5e20 con 0x7f5ffc103f50 2024-05-27T07:23:28.753 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:23:28.763+0000 7f60011e2700 1 -- 172.21.15.52:0/2687991003 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_command({"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]} v 0) -- 0x7f5ffc1a5630 con 0x7f5ffc103f50 2024-05-27T07:23:28.756 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:23:28.766+0000 7f5ff3fff700 1 -- 172.21.15.52:0/2687991003 <== mon.0 v2:172.21.15.52:3300/0 7 ==== mon_command_ack([{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]=0 v32) ==== 170+0+59 (secure 0 0 0) 0x7f5fec02b020 con 0x7f5ffc103f50 2024-05-27T07:23:28.756 INFO:teuthology.orchestra.run.smithi052.stdout:[client.0] 2024-05-27T07:23:28.756 INFO:teuthology.orchestra.run.smithi052.stdout: key = AQDwNFRmY7qYLRAAD0QbOomPV4w8/QHpTJOCKg== 2024-05-27T07:23:28.759 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:23:28.769+0000 7f60011e2700 1 -- 172.21.15.52:0/2687991003 >> [v2:172.21.15.52:6800/3513075805,v1:172.21.15.52:6801/3513075805] conn(0x7f5fe80b2990 msgr2=0x7f5fe80b4ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:23:28.759 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:23:28.769+0000 7f60011e2700 1 --2- 172.21.15.52:0/2687991003 >> [v2:172.21.15.52:6800/3513075805,v1:172.21.15.52:6801/3513075805] conn(0x7f5fe80b2990 0x7f5fe80b4ea0 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7f5ffc19dcd0 tx=0x7f5fe400b410 comp rx=0 tx=0).stop 2024-05-27T07:23:28.759 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:23:28.769+0000 7f60011e2700 1 -- 172.21.15.52:0/2687991003 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f5ffc103f50 msgr2=0x7f5ffc19cc20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:23:28.759 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:23:28.769+0000 7f60011e2700 1 --2- 172.21.15.52:0/2687991003 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f5ffc103f50 0x7f5ffc19cc20 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7f5fec000f80 tx=0x7f5fec004e40 comp rx=0 tx=0).stop 2024-05-27T07:23:28.759 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:23:28.769+0000 7f60011e2700 1 -- 172.21.15.52:0/2687991003 shutdown_connections 2024-05-27T07:23:28.759 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:23:28.769+0000 7f60011e2700 1 --2- 172.21.15.52:0/2687991003 >> [v2:172.21.15.52:6800/3513075805,v1:172.21.15.52:6801/3513075805] conn(0x7f5fe80b2990 0x7f5fe80b4ea0 unknown :-1 s=CLOSED pgs=172 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:23:28.759 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:23:28.769+0000 7f60011e2700 1 --2- 172.21.15.52:0/2687991003 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f5ffc1059a0 0x7f5ffc1a4100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:23:28.759 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:23:28.769+0000 7f60011e2700 1 --2- 172.21.15.52:0/2687991003 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f5ffc103f50 0x7f5ffc19cc20 unknown :-1 s=CLOSED pgs=246 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:23:28.759 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:23:28.769+0000 7f60011e2700 1 --2- 172.21.15.52:0/2687991003 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f5ffc1049d0 0x7f5ffc19d160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:23:28.760 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:23:28.769+0000 7f60011e2700 1 -- 172.21.15.52:0/2687991003 >> 172.21.15.52:0/2687991003 conn(0x7f5ffc0ff9c0 msgr2=0x7f5ffc076ad0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:23:28.760 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:23:28.769+0000 7f60011e2700 1 -- 172.21.15.52:0/2687991003 shutdown_connections 2024-05-27T07:23:28.760 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:23:28.769+0000 7f60011e2700 1 -- 172.21.15.52:0/2687991003 wait complete. 2024-05-27T07:23:29.102 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:23:28 smithi052 ceph-mon[31955]: pgmap v347: 129 pgs: 129 active+clean; 645 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 5.6 KiB/s rd, 1.2 KiB/s wr, 10 op/s 2024-05-27T07:23:29.103 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:23:28 smithi052 ceph-mon[31955]: from='client.? 172.21.15.52:0/2687991003' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-05-27T07:23:29.103 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:23:28 smithi052 ceph-mon[31955]: from='client.? 172.21.15.52:0/2687991003' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-05-27T07:23:29.352 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:23:28 smithi173 ceph-mon[37000]: pgmap v347: 129 pgs: 129 active+clean; 645 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 5.6 KiB/s rd, 1.2 KiB/s wr, 10 op/s 2024-05-27T07:23:29.352 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:23:28 smithi173 ceph-mon[37000]: from='client.? 172.21.15.52:0/2687991003' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-05-27T07:23:29.352 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:23:28 smithi173 ceph-mon[37000]: from='client.? 172.21.15.52:0/2687991003' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-05-27T07:23:29.399 DEBUG:teuthology.orchestra.run.smithi052:> set -ex 2024-05-27T07:23:29.399 DEBUG:teuthology.orchestra.run.smithi052:> sudo dd of=/etc/ceph/ceph.client.0.keyring 2024-05-27T07:23:29.399 DEBUG:teuthology.orchestra.run.smithi052:> sudo chmod 0644 /etc/ceph/ceph.client.0.keyring 2024-05-27T07:23:29.439 INFO:tasks.ceph:Waiting until ceph daemons up and pgs clean... 2024-05-27T07:23:29.440 INFO:tasks.cephadm.ceph_manager.ceph:waiting for mgr available 2024-05-27T07:23:29.440 DEBUG:teuthology.orchestra.run.smithi052:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:38f9d900c83c63799fbdbe61acc9a11b0d3554a6 shell --fsid 73df2636-1bf8-11ef-bc9b-c7b262605968 -- ceph mgr dump --format=json 2024-05-27T07:23:29.440 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:23:28 smithi105 ceph-mon[35977]: pgmap v347: 129 pgs: 129 active+clean; 645 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 5.6 KiB/s rd, 1.2 KiB/s wr, 10 op/s 2024-05-27T07:23:29.440 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:23:28 smithi105 ceph-mon[35977]: from='client.? 172.21.15.52:0/2687991003' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-05-27T07:23:29.441 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:23:28 smithi105 ceph-mon[35977]: from='client.? 172.21.15.52:0/2687991003' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-05-27T07:23:29.733 INFO:teuthology.orchestra.run.smithi052.stderr:Inferring config /var/lib/ceph/73df2636-1bf8-11ef-bc9b-c7b262605968/mon.a/config 2024-05-27T07:23:30.174 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:23:29 smithi052 ceph-mon[31955]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.8", "id": [2, 11]}]: dispatch 2024-05-27T07:23:30.352 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:23:29 smithi173 ceph-mon[37000]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.8", "id": [2, 11]}]: dispatch 2024-05-27T07:23:30.440 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:23:29 smithi105 ceph-mon[35977]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.8", "id": [2, 11]}]: dispatch 2024-05-27T07:23:31.229 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:23:30 smithi052 ceph-mon[31955]: pgmap v348: 129 pgs: 129 active+clean; 645 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 6.4 KiB/s rd, 1.2 KiB/s wr, 10 op/s 2024-05-27T07:23:31.229 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:23:30 smithi052 ceph-mon[31955]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.8", "id": [2, 11]}]': finished 2024-05-27T07:23:31.230 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:23:30 smithi052 ceph-mon[31955]: osdmap e89: 12 total, 12 up, 12 in 2024-05-27T07:23:31.352 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:23:30 smithi173 ceph-mon[37000]: pgmap v348: 129 pgs: 129 active+clean; 645 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 6.4 KiB/s rd, 1.2 KiB/s wr, 10 op/s 2024-05-27T07:23:31.352 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:23:30 smithi173 ceph-mon[37000]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.8", "id": [2, 11]}]': finished 2024-05-27T07:23:31.352 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:23:30 smithi173 ceph-mon[37000]: osdmap e89: 12 total, 12 up, 12 in 2024-05-27T07:23:31.440 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:23:30 smithi105 ceph-mon[35977]: pgmap v348: 129 pgs: 129 active+clean; 645 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 6.4 KiB/s rd, 1.2 KiB/s wr, 10 op/s 2024-05-27T07:23:31.440 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:23:30 smithi105 ceph-mon[35977]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.8", "id": [2, 11]}]': finished 2024-05-27T07:23:31.440 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:23:30 smithi105 ceph-mon[35977]: osdmap e89: 12 total, 12 up, 12 in 2024-05-27T07:23:31.501 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:23:31.510+0000 7f84302ee700 1 -- 172.21.15.52:0/1727788183 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f8428108730 msgr2=0x7f8428110090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:23:31.501 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:23:31.510+0000 7f84302ee700 1 --2- 172.21.15.52:0/1727788183 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f8428108730 0x7f8428110090 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7f84200099c0 tx=0x7f8420009cd0 comp rx=0 tx=0).stop 2024-05-27T07:23:31.501 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:23:31.511+0000 7f84302ee700 1 -- 172.21.15.52:0/1727788183 shutdown_connections 2024-05-27T07:23:31.501 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:23:31.511+0000 7f84302ee700 1 --2- 172.21.15.52:0/1727788183 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f8428108730 0x7f8428110090 unknown :-1 s=CLOSED pgs=247 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:23:31.501 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:23:31.511+0000 7f84302ee700 1 --2- 172.21.15.52:0/1727788183 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f84281034b0 0x7f84281038c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:23:31.501 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:23:31.511+0000 7f84302ee700 1 --2- 172.21.15.52:0/1727788183 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f8428107de0 0x7f84281081f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:23:31.501 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:23:31.511+0000 7f84302ee700 1 -- 172.21.15.52:0/1727788183 >> 172.21.15.52:0/1727788183 conn(0x7f84280fedb0 msgr2=0x7f84281011d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:23:31.501 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:23:31.511+0000 7f84302ee700 1 -- 172.21.15.52:0/1727788183 shutdown_connections 2024-05-27T07:23:31.501 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:23:31.511+0000 7f84302ee700 1 -- 172.21.15.52:0/1727788183 wait complete. 2024-05-27T07:23:31.503 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:23:31.511+0000 7f84302ee700 1 Processor -- start 2024-05-27T07:23:31.503 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:23:31.512+0000 7f84302ee700 1 -- start start 2024-05-27T07:23:31.503 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:23:31.512+0000 7f84302ee700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f84281034b0 0x7f842819ad60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:23:31.503 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:23:31.512+0000 7f84302ee700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f8428107de0 0x7f842819b300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:23:31.504 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:23:31.512+0000 7f84302ee700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f8428108730 0x7f842819f590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:23:31.504 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:23:31.512+0000 7f84302ee700 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x7f8428113360 con 0x7f84281034b0 2024-05-27T07:23:31.504 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:23:31.512+0000 7f84302ee700 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f84281131e0 con 0x7f8428107de0 2024-05-27T07:23:31.504 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:23:31.512+0000 7f84302ee700 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f84281134e0 con 0x7f8428108730 2024-05-27T07:23:31.504 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:23:31.512+0000 7f842e08a700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f84281034b0 0x7f842819ad60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:23:31.504 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:23:31.512+0000 7f842e08a700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f84281034b0 0x7f842819ad60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.52:56964/0 (socket says 172.21.15.52:56964) 2024-05-27T07:23:31.504 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:23:31.512+0000 7f842e08a700 1 -- 172.21.15.52:0/786851321 learned_addr learned my addr 172.21.15.52:0/786851321 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-05-27T07:23:31.504 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:23:31.512+0000 7f842d889700 1 --2- 172.21.15.52:0/786851321 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f8428107de0 0x7f842819b300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:23:31.504 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:23:31.512+0000 7f842e88b700 1 --2- 172.21.15.52:0/786851321 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f8428108730 0x7f842819f590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:23:31.504 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:23:31.512+0000 7f842e08a700 1 -- 172.21.15.52:0/786851321 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f8428108730 msgr2=0x7f842819f590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:23:31.504 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:23:31.512+0000 7f842e08a700 1 --2- 172.21.15.52:0/786851321 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f8428108730 0x7f842819f590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:23:31.504 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:23:31.512+0000 7f842e08a700 1 -- 172.21.15.52:0/786851321 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f8428107de0 msgr2=0x7f842819b300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:23:31.505 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:23:31.512+0000 7f842e08a700 1 --2- 172.21.15.52:0/786851321 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f8428107de0 0x7f842819b300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:23:31.505 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:23:31.512+0000 7f842e08a700 1 -- 172.21.15.52:0/786851321 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f842819fb80 con 0x7f84281034b0 2024-05-27T07:23:31.505 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:23:31.512+0000 7f842e88b700 1 --2- 172.21.15.52:0/786851321 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f8428108730 0x7f842819f590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-27T07:23:31.505 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:23:31.513+0000 7f842e08a700 1 --2- 172.21.15.52:0/786851321 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f84281034b0 0x7f842819ad60 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7f84180060b0 tx=0x7f841800d3b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:23:31.505 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:23:31.513+0000 7f84277fe700 1 -- 172.21.15.52:0/786851321 <== mon.0 v2:172.21.15.52:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f841801a070 con 0x7f84281034b0 2024-05-27T07:23:31.505 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:23:31.513+0000 7f84302ee700 1 -- 172.21.15.52:0/786851321 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f842819fe70 con 0x7f84281034b0 2024-05-27T07:23:31.505 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:23:31.513+0000 7f84277fe700 1 -- 172.21.15.52:0/786851321 <== mon.0 v2:172.21.15.52:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f841800ddd0 con 0x7f84281034b0 2024-05-27T07:23:31.505 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:23:31.513+0000 7f84277fe700 1 -- 172.21.15.52:0/786851321 <== mon.0 v2:172.21.15.52:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8418014720 con 0x7f84281034b0 2024-05-27T07:23:31.505 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:23:31.513+0000 7f84302ee700 1 -- 172.21.15.52:0/786851321 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f84281a03b0 con 0x7f84281034b0 2024-05-27T07:23:31.505 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:23:31.514+0000 7f84302ee700 1 -- 172.21.15.52:0/786851321 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f84280512b0 con 0x7f84281034b0 2024-05-27T07:23:31.506 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:23:31.515+0000 7f84277fe700 1 -- 172.21.15.52:0/786851321 <== mon.0 v2:172.21.15.52:3300/0 4 ==== mgrmap(e 17) ==== 149703+0+0 (secure 0 0 0) 0x7f841801d070 con 0x7f84281034b0 2024-05-27T07:23:31.506 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:23:31.516+0000 7f84277fe700 1 --2- 172.21.15.52:0/786851321 >> [v2:172.21.15.52:6800/3513075805,v1:172.21.15.52:6801/3513075805] conn(0x7f84100b2910 0x7f84100b4e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:23:31.506 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:23:31.516+0000 7f84277fe700 1 -- 172.21.15.52:0/786851321 <== mon.0 v2:172.21.15.52:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9121+0+0 (secure 0 0 0) 0x7f84180e2470 con 0x7f84281034b0 2024-05-27T07:23:31.506 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:23:31.516+0000 7f842d889700 1 --2- 172.21.15.52:0/786851321 >> [v2:172.21.15.52:6800/3513075805,v1:172.21.15.52:6801/3513075805] conn(0x7f84100b2910 0x7f84100b4e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:23:31.507 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:23:31.517+0000 7f842d889700 1 --2- 172.21.15.52:0/786851321 >> [v2:172.21.15.52:6800/3513075805,v1:172.21.15.52:6801/3513075805] conn(0x7f84100b2910 0x7f84100b4e20 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7f8428109880 tx=0x7f841c00a380 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:23:31.510 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:23:31.520+0000 7f84277fe700 1 -- 172.21.15.52:0/786851321 <== mon.0 v2:172.21.15.52:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191963 (secure 0 0 0) 0x7f8418010070 con 0x7f84281034b0 2024-05-27T07:23:31.786 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:23:31.797+0000 7f84302ee700 1 -- 172.21.15.52:0/786851321 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_command({"prefix": "mgr dump", "format": "json"} v 0) -- 0x7f842819c0f0 con 0x7f84281034b0 2024-05-27T07:23:31.792 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:23:31.802+0000 7f84277fe700 1 -- 172.21.15.52:0/786851321 <== mon.0 v2:172.21.15.52:3300/0 7 ==== mon_command_ack([{"prefix": "mgr dump", "format": "json"}]=0 v17) ==== 74+0+288646 (secure 0 0 0) 0x7f842819c0f0 con 0x7f84281034b0 2024-05-27T07:23:31.795 INFO:teuthology.orchestra.run.smithi052.stdout: 2024-05-27T07:23:31.797 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:23:31.807+0000 7f84302ee700 1 -- 172.21.15.52:0/786851321 >> [v2:172.21.15.52:6800/3513075805,v1:172.21.15.52:6801/3513075805] conn(0x7f84100b2910 msgr2=0x7f84100b4e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:23:31.797 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:23:31.807+0000 7f84302ee700 1 --2- 172.21.15.52:0/786851321 >> [v2:172.21.15.52:6800/3513075805,v1:172.21.15.52:6801/3513075805] conn(0x7f84100b2910 0x7f84100b4e20 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7f8428109880 tx=0x7f841c00a380 comp rx=0 tx=0).stop 2024-05-27T07:23:31.798 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:23:31.807+0000 7f84302ee700 1 -- 172.21.15.52:0/786851321 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f84281034b0 msgr2=0x7f842819ad60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:23:31.798 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:23:31.807+0000 7f84302ee700 1 --2- 172.21.15.52:0/786851321 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f84281034b0 0x7f842819ad60 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7f84180060b0 tx=0x7f841800d3b0 comp rx=0 tx=0).stop 2024-05-27T07:23:31.798 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:23:31.807+0000 7f84302ee700 1 -- 172.21.15.52:0/786851321 shutdown_connections 2024-05-27T07:23:31.798 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:23:31.807+0000 7f84302ee700 1 --2- 172.21.15.52:0/786851321 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f8428108730 0x7f842819f590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:23:31.798 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:23:31.807+0000 7f84302ee700 1 --2- 172.21.15.52:0/786851321 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f84281034b0 0x7f842819ad60 unknown :-1 s=CLOSED pgs=248 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:23:31.798 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:23:31.807+0000 7f84302ee700 1 --2- 172.21.15.52:0/786851321 >> [v2:172.21.15.52:6800/3513075805,v1:172.21.15.52:6801/3513075805] conn(0x7f84100b2910 0x7f84100b4e20 unknown :-1 s=CLOSED pgs=173 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:23:31.798 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:23:31.807+0000 7f84302ee700 1 --2- 172.21.15.52:0/786851321 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f8428107de0 0x7f842819b300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:23:31.798 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:23:31.807+0000 7f84302ee700 1 -- 172.21.15.52:0/786851321 >> 172.21.15.52:0/786851321 conn(0x7f84280fedb0 msgr2=0x7f842806b8a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:23:31.798 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:23:31.807+0000 7f84302ee700 1 -- 172.21.15.52:0/786851321 shutdown_connections 2024-05-27T07:23:31.798 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:23:31.807+0000 7f84302ee700 1 -- 172.21.15.52:0/786851321 wait complete. 2024-05-27T07:23:32.130 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:23:31 smithi052 ceph-mon[31955]: osdmap e90: 12 total, 12 up, 12 in 2024-05-27T07:23:32.130 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:23:31 smithi052 ceph-mon[31955]: from='client.? 172.21.15.52:0/786851321' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json"}]: dispatch 2024-05-27T07:23:32.352 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:23:31 smithi173 ceph-mon[37000]: osdmap e90: 12 total, 12 up, 12 in 2024-05-27T07:23:32.352 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:23:31 smithi173 ceph-mon[37000]: from='client.? 172.21.15.52:0/786851321' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json"}]: dispatch 2024-05-27T07:23:32.440 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:23:31 smithi105 ceph-mon[35977]: osdmap e90: 12 total, 12 up, 12 in 2024-05-27T07:23:32.440 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:23:31 smithi105 ceph-mon[35977]: from='client.? 172.21.15.52:0/786851321' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json"}]: dispatch 2024-05-27T07:23:32.574 INFO:teuthology.orchestra.run.smithi052.stdout:{"epoch":17,"flags":0,"active_gid":14152,"active_name":"x","active_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.52:6800","nonce":3513075805},{"type":"v1","addr":"172.21.15.52:6801","nonce":3513075805}]},"active_addr":"172.21.15.52:6801/3513075805","active_change":"2024-05-27T07:14:29.544802+0000","active_mgr_features":4540701547738038271,"available":true,"standbys":[{"gid":24118,"name":"z","mgr_features":4540701547738038271,"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","read","upmap","upmap-read"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cephadm_log_destination":{"name":"cephadm_log_destination","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":["file","file,syslog","syslog"],"desc":"Destination for cephadm command's persistent logging","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/grafana:9.4.12","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:3.0.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:1.2.5","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:3.0.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_samba":{"name":"container_image_samba","type":"str","level":"advanced","flags":0,"default_value":"quay.io/samba.org/samba-server:devbuilds-centos-amd64","min":"","max":"","enum_allowed":[],"desc":"Samba/SMB container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"grafana_dashboards_path":{"name":"grafana_dashboards_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/grafana/dashboards/ceph-dashboard/","min":"","max":"","enum_allowed":[],"desc":"location of dashboards to include in grafana deployments","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"hw_monitoring":{"name":"hw_monitoring","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Deploy hw monitoring daemon on every host.","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"oob_default_addr":{"name":"oob_default_addr","type":"str","level":"advanced","flags":0,"default_value":"169.254.1.1","min":"","max":"","enum_allowed":[],"desc":"Default address for RedFish API (oob management).","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"MANAGED_BY_CLUSTERS":{"name":"MANAGED_BY_CLUSTERS","type":"str","level":"advanced","flags":0,"default_value":"[]","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"MULTICLUSTER_CONFIG":{"name":"MULTICLUSTER_CONFIG","type":"str","level":"advanced","flags":0,"default_value":"{}","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"UNSAFE_TLS_v1_2":{"name":"UNSAFE_TLS_v1_2","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"k8sevents","can_run":true,"error_string":"","module_options":{"ceph_event_retention_days":{"name":"ceph_event_retention_days","type":"int","level":"advanced","flags":0,"default_value":"7","min":"","max":"","enum_allowed":[],"desc":"Days to hold ceph event information within local cache","long_desc":"","tags":[],"see_also":[]},"config_check_secs":{"name":"config_check_secs","type":"int","level":"advanced","flags":0,"default_value":"10","min":"10","max":"","enum_allowed":[],"desc":"interval (secs) to check for cluster configuration changes","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"smb","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"update_orchestration":{"name":"update_orchestration","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically update orchestration when smb resources are changed","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"periodic_async_work":{"name":"periodic_async_work","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Periodically check for async work","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_no_wait":{"name":"snapshot_clone_no_wait","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Reject subvolume clone request when cloner threads are busy","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}]},{"gid":24125,"name":"y","mgr_features":4540701547738038271,"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","read","upmap","upmap-read"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cephadm_log_destination":{"name":"cephadm_log_destination","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":["file","file,syslog","syslog"],"desc":"Destination for cephadm command's persistent logging","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/grafana:9.4.12","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:3.0.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:1.2.5","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:3.0.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_samba":{"name":"container_image_samba","type":"str","level":"advanced","flags":0,"default_value":"quay.io/samba.org/samba-server:devbuilds-centos-amd64","min":"","max":"","enum_allowed":[],"desc":"Samba/SMB container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"grafana_dashboards_path":{"name":"grafana_dashboards_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/grafana/dashboards/ceph-dashboard/","min":"","max":"","enum_allowed":[],"desc":"location of dashboards to include in grafana deployments","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"hw_monitoring":{"name":"hw_monitoring","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Deploy hw monitoring daemon on every host.","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"oob_default_addr":{"name":"oob_default_addr","type":"str","level":"advanced","flags":0,"default_value":"169.254.1.1","min":"","max":"","enum_allowed":[],"desc":"Default address for RedFish API (oob management).","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"MANAGED_BY_CLUSTERS":{"name":"MANAGED_BY_CLUSTERS","type":"str","level":"advanced","flags":0,"default_value":"[]","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"MULTICLUSTER_CONFIG":{"name":"MULTICLUSTER_CONFIG","type":"str","level":"advanced","flags":0,"default_value":"{}","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"UNSAFE_TLS_v1_2":{"name":"UNSAFE_TLS_v1_2","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"k8sevents","can_run":true,"error_string":"","module_options":{"ceph_event_retention_days":{"name":"ceph_event_retention_days","type":"int","level":"advanced","flags":0,"default_value":"7","min":"","max":"","enum_allowed":[],"desc":"Days to hold ceph event information within local cache","long_desc":"","tags":[],"see_also":[]},"config_check_secs":{"name":"config_check_secs","type":"int","level":"advanced","flags":0,"default_value":"10","min":"10","max":"","enum_allowed":[],"desc":"interval (secs) to check for cluster configuration changes","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"smb","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"update_orchestration":{"name":"update_orchestration","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically update orchestration when smb resources are changed","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"periodic_async_work":{"name":"periodic_async_work","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Periodically check for async work","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_no_wait":{"name":"snapshot_clone_no_wait","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Reject subvolume clone request when cloner threads are busy","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}]}],"modules":["cephadm","dashboard","iostat","nfs","restful"],"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","read","upmap","upmap-read"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cephadm_log_destination":{"name":"cephadm_log_destination","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":["file","file,syslog","syslog"],"desc":"Destination for cephadm command's persistent logging","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/grafana:9.4.12","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:3.0.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:1.2.5","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:3.0.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_samba":{"name":"container_image_samba","type":"str","level":"advanced","flags":0,"default_value":"quay.io/samba.org/samba-server:devbuilds-centos-amd64","min":"","max":"","enum_allowed":[],"desc":"Samba/SMB container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"grafana_dashboards_path":{"name":"grafana_dashboards_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/grafana/dashboards/ceph-dashboard/","min":"","max":"","enum_allowed":[],"desc":"location of dashboards to include in grafana deployments","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"hw_monitoring":{"name":"hw_monitoring","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Deploy hw monitoring daemon on every host.","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"oob_default_addr":{"name":"oob_default_addr","type":"str","level":"advanced","flags":0,"default_value":"169.254.1.1","min":"","max":"","enum_allowed":[],"desc":"Default address for RedFish API (oob management).","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"MANAGED_BY_CLUSTERS":{"name":"MANAGED_BY_CLUSTERS","type":"str","level":"advanced","flags":0,"default_value":"[]","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"MULTICLUSTER_CONFIG":{"name":"MULTICLUSTER_CONFIG","type":"str","level":"advanced","flags":0,"default_value":"{}","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"UNSAFE_TLS_v1_2":{"name":"UNSAFE_TLS_v1_2","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"k8sevents","can_run":true,"error_string":"","module_options":{"ceph_event_retention_days":{"name":"ceph_event_retention_days","type":"int","level":"advanced","flags":0,"default_value":"7","min":"","max":"","enum_allowed":[],"desc":"Days to hold ceph event information within local cache","long_desc":"","tags":[],"see_also":[]},"config_check_secs":{"name":"config_check_secs","type":"int","level":"advanced","flags":0,"default_value":"10","min":"10","max":"","enum_allowed":[],"desc":"interval (secs) to check for cluster configuration changes","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"smb","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"update_orchestration":{"name":"update_orchestration","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically update orchestration when smb resources are changed","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"periodic_async_work":{"name":"periodic_async_work","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Periodically check for async work","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_no_wait":{"name":"snapshot_clone_no_wait","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Reject subvolume clone request when cloner threads are busy","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}],"services":{"dashboard":"https://172.21.15.52:8443/"},"always_on_modules":{"octopus":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"pacific":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"quincy":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"reef":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"squid":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"]},"last_failure_osd_epoch":3,"active_clients":[{"name":"devicehealth","addrvec":[{"type":"v2","addr":"172.21.15.52:0","nonce":3577886285}]},{"name":"libcephsqlite","addrvec":[{"type":"v2","addr":"172.21.15.52:0","nonce":2723769517}]},{"name":"rbd_support","addrvec":[{"type":"v2","addr":"172.21.15.52:0","nonce":262308791}]},{"name":"volumes","addrvec":[{"type":"v2","addr":"172.21.15.52:0","nonce":3430810618}]},{"name":"volumes","addrvec":[{"type":"v2","addr":"172.21.15.52:0","nonce":1000566071}]}]} 2024-05-27T07:23:32.580 INFO:tasks.cephadm.ceph_manager.ceph:mgr available! 2024-05-27T07:23:32.580 INFO:tasks.cephadm.ceph_manager.ceph:waiting for all up 2024-05-27T07:23:32.580 DEBUG:teuthology.orchestra.run.smithi052:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:38f9d900c83c63799fbdbe61acc9a11b0d3554a6 shell --fsid 73df2636-1bf8-11ef-bc9b-c7b262605968 -- ceph osd dump --format=json 2024-05-27T07:23:32.835 INFO:teuthology.orchestra.run.smithi052.stderr:Inferring config /var/lib/ceph/73df2636-1bf8-11ef-bc9b-c7b262605968/mon.a/config 2024-05-27T07:23:33.229 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:23:32 smithi052 ceph-mon[31955]: pgmap v351: 129 pgs: 129 active+clean; 645 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 7.1 KiB/s rd, 1.2 KiB/s wr, 11 op/s 2024-05-27T07:23:33.352 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:23:32 smithi173 ceph-mon[37000]: pgmap v351: 129 pgs: 129 active+clean; 645 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 7.1 KiB/s rd, 1.2 KiB/s wr, 11 op/s 2024-05-27T07:23:33.440 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:23:32 smithi105 ceph-mon[35977]: pgmap v351: 129 pgs: 129 active+clean; 645 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 7.1 KiB/s rd, 1.2 KiB/s wr, 11 op/s 2024-05-27T07:23:34.658 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:23:34.667+0000 7fa684350700 1 -- 172.21.15.52:0/45889611 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fa67c109850 msgr2=0x7fa67c1111b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:23:34.658 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:23:34.667+0000 7fa684350700 1 --2- 172.21.15.52:0/45889611 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fa67c109850 0x7fa67c1111b0 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7fa6740099c0 tx=0x7fa674009cd0 comp rx=0 tx=0).stop 2024-05-27T07:23:34.658 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:23:34.667+0000 7fa684350700 1 -- 172.21.15.52:0/45889611 shutdown_connections 2024-05-27T07:23:34.658 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:23:34.667+0000 7fa684350700 1 --2- 172.21.15.52:0/45889611 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fa67c109850 0x7fa67c1111b0 unknown :-1 s=CLOSED pgs=249 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:23:34.658 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:23:34.667+0000 7fa684350700 1 --2- 172.21.15.52:0/45889611 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fa67c1045d0 0x7fa67c1049e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:23:34.658 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:23:34.667+0000 7fa684350700 1 --2- 172.21.15.52:0/45889611 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fa67c108f00 0x7fa67c109310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:23:34.658 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:23:34.667+0000 7fa684350700 1 -- 172.21.15.52:0/45889611 >> 172.21.15.52:0/45889611 conn(0x7fa67c0ffe60 msgr2=0x7fa67c1022b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:23:34.659 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:23:34.668+0000 7fa684350700 1 -- 172.21.15.52:0/45889611 shutdown_connections 2024-05-27T07:23:34.659 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:23:34.668+0000 7fa684350700 1 -- 172.21.15.52:0/45889611 wait complete. 2024-05-27T07:23:34.660 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:23:34.668+0000 7fa684350700 1 Processor -- start 2024-05-27T07:23:34.660 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:23:34.668+0000 7fa684350700 1 -- start start 2024-05-27T07:23:34.661 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:23:34.669+0000 7fa684350700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fa67c1045d0 0x7fa67c19adf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:23:34.661 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:23:34.669+0000 7fa684350700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fa67c108f00 0x7fa67c19b390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:23:34.661 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:23:34.669+0000 7fa684350700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fa67c109850 0x7fa67c19f5d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:23:34.661 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:23:34.669+0000 7fa684350700 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x7fa67c114480 con 0x7fa67c1045d0 2024-05-27T07:23:34.661 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:23:34.669+0000 7fa684350700 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7fa67c114300 con 0x7fa67c108f00 2024-05-27T07:23:34.661 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:23:34.669+0000 7fa684350700 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7fa67c114600 con 0x7fa67c109850 2024-05-27T07:23:34.661 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:23:34.669+0000 7fa6820ec700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fa67c1045d0 0x7fa67c19adf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:23:34.661 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:23:34.669+0000 7fa6820ec700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fa67c1045d0 0x7fa67c19adf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.52:56980/0 (socket says 172.21.15.52:56980) 2024-05-27T07:23:34.661 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:23:34.669+0000 7fa6820ec700 1 -- 172.21.15.52:0/1179944392 learned_addr learned my addr 172.21.15.52:0/1179944392 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-05-27T07:23:34.661 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:23:34.669+0000 7fa6818eb700 1 --2- 172.21.15.52:0/1179944392 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fa67c108f00 0x7fa67c19b390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:23:34.661 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:23:34.669+0000 7fa6828ed700 1 --2- 172.21.15.52:0/1179944392 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fa67c109850 0x7fa67c19f5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:23:34.661 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:23:34.669+0000 7fa6820ec700 1 -- 172.21.15.52:0/1179944392 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fa67c109850 msgr2=0x7fa67c19f5d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:23:34.662 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:23:34.669+0000 7fa6820ec700 1 --2- 172.21.15.52:0/1179944392 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fa67c109850 0x7fa67c19f5d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:23:34.662 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:23:34.669+0000 7fa6820ec700 1 -- 172.21.15.52:0/1179944392 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fa67c108f00 msgr2=0x7fa67c19b390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:23:34.662 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:23:34.669+0000 7fa6820ec700 1 --2- 172.21.15.52:0/1179944392 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fa67c108f00 0x7fa67c19b390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:23:34.662 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:23:34.669+0000 7fa6820ec700 1 -- 172.21.15.52:0/1179944392 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa67c19fc70 con 0x7fa67c1045d0 2024-05-27T07:23:34.662 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:23:34.669+0000 7fa6820ec700 1 --2- 172.21.15.52:0/1179944392 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fa67c1045d0 0x7fa67c19adf0 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7fa66c007f10 tx=0x7fa66c00d3b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:23:34.662 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:23:34.670+0000 7fa67b7fe700 1 -- 172.21.15.52:0/1179944392 <== mon.0 v2:172.21.15.52:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa66c004db0 con 0x7fa67c1045d0 2024-05-27T07:23:34.662 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:23:34.670+0000 7fa684350700 1 -- 172.21.15.52:0/1179944392 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa67c19ff60 con 0x7fa67c1045d0 2024-05-27T07:23:34.662 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:23:34.670+0000 7fa67b7fe700 1 -- 172.21.15.52:0/1179944392 <== mon.0 v2:172.21.15.52:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fa66c003a00 con 0x7fa67c1045d0 2024-05-27T07:23:34.662 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:23:34.670+0000 7fa67b7fe700 1 -- 172.21.15.52:0/1179944392 <== mon.0 v2:172.21.15.52:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa66c0045e0 con 0x7fa67c1045d0 2024-05-27T07:23:34.662 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:23:34.670+0000 7fa684350700 1 -- 172.21.15.52:0/1179944392 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa67c1a7880 con 0x7fa67c1045d0 2024-05-27T07:23:34.662 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:23:34.671+0000 7fa684350700 1 -- 172.21.15.52:0/1179944392 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa67c0512b0 con 0x7fa67c1045d0 2024-05-27T07:23:34.663 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:23:34.673+0000 7fa67b7fe700 1 -- 172.21.15.52:0/1179944392 <== mon.0 v2:172.21.15.52:3300/0 4 ==== mgrmap(e 17) ==== 149703+0+0 (secure 0 0 0) 0x7fa66c01c020 con 0x7fa67c1045d0 2024-05-27T07:23:34.664 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:23:34.674+0000 7fa67b7fe700 1 --2- 172.21.15.52:0/1179944392 >> [v2:172.21.15.52:6800/3513075805,v1:172.21.15.52:6801/3513075805] conn(0x7fa6640b2910 0x7fa6640b4e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:23:34.664 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:23:34.675+0000 7fa6818eb700 1 --2- 172.21.15.52:0/1179944392 >> [v2:172.21.15.52:6800/3513075805,v1:172.21.15.52:6801/3513075805] conn(0x7fa6640b2910 0x7fa6640b4e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:23:34.664 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:23:34.675+0000 7fa67b7fe700 1 -- 172.21.15.52:0/1179944392 <== mon.0 v2:172.21.15.52:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9121+0+0 (secure 0 0 0) 0x7fa66c0e2a30 con 0x7fa67c1045d0 2024-05-27T07:23:34.665 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:23:34.675+0000 7fa6818eb700 1 --2- 172.21.15.52:0/1179944392 >> [v2:172.21.15.52:6800/3513075805,v1:172.21.15.52:6801/3513075805] conn(0x7fa6640b2910 0x7fa6640b4e20 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7fa67c1087e0 tx=0x7fa67000b410 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:23:34.667 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:23:34.678+0000 7fa67b7fe700 1 -- 172.21.15.52:0/1179944392 <== mon.0 v2:172.21.15.52:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191963 (secure 0 0 0) 0x7fa66c023030 con 0x7fa67c1045d0 2024-05-27T07:23:34.890 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:23:34.900+0000 7fa684350700 1 -- 172.21.15.52:0/1179944392 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fa67c19c100 con 0x7fa67c1045d0 2024-05-27T07:23:34.891 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:23:34.901+0000 7fa67b7fe700 1 -- 172.21.15.52:0/1179944392 <== mon.0 v2:172.21.15.52:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v90) ==== 74+0+24701 (secure 0 0 0) 0x7fa66c010070 con 0x7fa67c1045d0 2024-05-27T07:23:34.892 INFO:teuthology.orchestra.run.smithi052.stdout: 2024-05-27T07:23:34.892 INFO:teuthology.orchestra.run.smithi052.stdout:{"epoch":90,"fsid":"73df2636-1bf8-11ef-bc9b-c7b262605968","created":"2024-05-27T07:13:34.738032+0000","modified":"2024-05-27T07:23:30.969990+0000","last_up_change":"2024-05-27T07:20:45.777597+0000","last_in_change":"2024-05-27T07:20:25.314823+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-05-27T07:16:57.644034+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-05-27T07:21:43.917748+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"79","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-05-27T07:21:47.341876+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"79","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-05-27T07:21:57.707611+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"86","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"347b8a89-9dbe-4848-83b0-fd0e9374f33d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":82,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.52:6802","nonce":1626508899},{"type":"v1","addr":"172.21.15.52:6803","nonce":1626508899}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.52:6804","nonce":1626508899},{"type":"v1","addr":"172.21.15.52:6805","nonce":1626508899}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.52:6808","nonce":1626508899},{"type":"v1","addr":"172.21.15.52:6809","nonce":1626508899}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.52:6806","nonce":1626508899},{"type":"v1","addr":"172.21.15.52:6807","nonce":1626508899}]},"public_addr":"172.21.15.52:6803/1626508899","cluster_addr":"172.21.15.52:6805/1626508899","heartbeat_back_addr":"172.21.15.52:6809/1626508899","heartbeat_front_addr":"172.21.15.52:6807/1626508899","state":["exists","up"]},{"osd":1,"uuid":"7a3623d4-e087-40cb-9336-d4a5c52203d4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6800","nonce":2900093195},{"type":"v1","addr":"172.21.15.105:6801","nonce":2900093195}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6802","nonce":2900093195},{"type":"v1","addr":"172.21.15.105:6803","nonce":2900093195}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6806","nonce":2900093195},{"type":"v1","addr":"172.21.15.105:6807","nonce":2900093195}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6804","nonce":2900093195},{"type":"v1","addr":"172.21.15.105:6805","nonce":2900093195}]},"public_addr":"172.21.15.105:6801/2900093195","cluster_addr":"172.21.15.105:6803/2900093195","heartbeat_back_addr":"172.21.15.105:6807/2900093195","heartbeat_front_addr":"172.21.15.105:6805/2900093195","state":["exists","up"]},{"osd":2,"uuid":"41a9df5b-882a-47da-bb26-7f4d26ccf6c1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6800","nonce":2251189314},{"type":"v1","addr":"172.21.15.173:6801","nonce":2251189314}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6802","nonce":2251189314},{"type":"v1","addr":"172.21.15.173:6803","nonce":2251189314}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6806","nonce":2251189314},{"type":"v1","addr":"172.21.15.173:6807","nonce":2251189314}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6804","nonce":2251189314},{"type":"v1","addr":"172.21.15.173:6805","nonce":2251189314}]},"public_addr":"172.21.15.173:6801/2251189314","cluster_addr":"172.21.15.173:6803/2251189314","heartbeat_back_addr":"172.21.15.173:6807/2251189314","heartbeat_front_addr":"172.21.15.173:6805/2251189314","state":["exists","up"]},{"osd":3,"uuid":"f1245439-42a5-46c5-a998-e0a78f9df3fb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":26,"up_thru":82,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.52:6810","nonce":57920794},{"type":"v1","addr":"172.21.15.52:6811","nonce":57920794}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.52:6812","nonce":57920794},{"type":"v1","addr":"172.21.15.52:6813","nonce":57920794}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.52:6816","nonce":57920794},{"type":"v1","addr":"172.21.15.52:6817","nonce":57920794}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.52:6814","nonce":57920794},{"type":"v1","addr":"172.21.15.52:6815","nonce":57920794}]},"public_addr":"172.21.15.52:6811/57920794","cluster_addr":"172.21.15.52:6813/57920794","heartbeat_back_addr":"172.21.15.52:6817/57920794","heartbeat_front_addr":"172.21.15.52:6815/57920794","state":["exists","up"]},{"osd":4,"uuid":"2592d1cc-1726-4359-83d8-a2e6314b54ec","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":32,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6808","nonce":3441134818},{"type":"v1","addr":"172.21.15.105:6809","nonce":3441134818}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6810","nonce":3441134818},{"type":"v1","addr":"172.21.15.105:6811","nonce":3441134818}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6814","nonce":3441134818},{"type":"v1","addr":"172.21.15.105:6815","nonce":3441134818}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6812","nonce":3441134818},{"type":"v1","addr":"172.21.15.105:6813","nonce":3441134818}]},"public_addr":"172.21.15.105:6809/3441134818","cluster_addr":"172.21.15.105:6811/3441134818","heartbeat_back_addr":"172.21.15.105:6815/3441134818","heartbeat_front_addr":"172.21.15.105:6813/3441134818","state":["exists","up"]},{"osd":5,"uuid":"27e98ee2-4a92-4972-b2a3-959d0ed90981","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":37,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6808","nonce":2989239729},{"type":"v1","addr":"172.21.15.173:6809","nonce":2989239729}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6810","nonce":2989239729},{"type":"v1","addr":"172.21.15.173:6811","nonce":2989239729}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6814","nonce":2989239729},{"type":"v1","addr":"172.21.15.173:6815","nonce":2989239729}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6812","nonce":2989239729},{"type":"v1","addr":"172.21.15.173:6813","nonce":2989239729}]},"public_addr":"172.21.15.173:6809/2989239729","cluster_addr":"172.21.15.173:6811/2989239729","heartbeat_back_addr":"172.21.15.173:6815/2989239729","heartbeat_front_addr":"172.21.15.173:6813/2989239729","state":["exists","up"]},{"osd":6,"uuid":"2b0ec0cb-f438-4b2c-8642-0778ed374e9a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":43,"up_thru":82,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.52:6818","nonce":4244423065},{"type":"v1","addr":"172.21.15.52:6819","nonce":4244423065}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.52:6820","nonce":4244423065},{"type":"v1","addr":"172.21.15.52:6821","nonce":4244423065}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.52:6824","nonce":4244423065},{"type":"v1","addr":"172.21.15.52:6825","nonce":4244423065}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.52:6822","nonce":4244423065},{"type":"v1","addr":"172.21.15.52:6823","nonce":4244423065}]},"public_addr":"172.21.15.52:6819/4244423065","cluster_addr":"172.21.15.52:6821/4244423065","heartbeat_back_addr":"172.21.15.52:6825/4244423065","heartbeat_front_addr":"172.21.15.52:6823/4244423065","state":["exists","up"]},{"osd":7,"uuid":"cb2a7a6b-8015-43ea-865a-be144e45c51a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":48,"up_thru":82,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6816","nonce":2063611552},{"type":"v1","addr":"172.21.15.105:6817","nonce":2063611552}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6818","nonce":2063611552},{"type":"v1","addr":"172.21.15.105:6819","nonce":2063611552}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6822","nonce":2063611552},{"type":"v1","addr":"172.21.15.105:6823","nonce":2063611552}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6820","nonce":2063611552},{"type":"v1","addr":"172.21.15.105:6821","nonce":2063611552}]},"public_addr":"172.21.15.105:6817/2063611552","cluster_addr":"172.21.15.105:6819/2063611552","heartbeat_back_addr":"172.21.15.105:6823/2063611552","heartbeat_front_addr":"172.21.15.105:6821/2063611552","state":["exists","up"]},{"osd":8,"uuid":"c2fc565d-a0b9-4396-a209-bd884cddcfe3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":54,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6816","nonce":4131399852},{"type":"v1","addr":"172.21.15.173:6817","nonce":4131399852}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6818","nonce":4131399852},{"type":"v1","addr":"172.21.15.173:6819","nonce":4131399852}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6822","nonce":4131399852},{"type":"v1","addr":"172.21.15.173:6823","nonce":4131399852}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6820","nonce":4131399852},{"type":"v1","addr":"172.21.15.173:6821","nonce":4131399852}]},"public_addr":"172.21.15.173:6817/4131399852","cluster_addr":"172.21.15.173:6819/4131399852","heartbeat_back_addr":"172.21.15.173:6823/4131399852","heartbeat_front_addr":"172.21.15.173:6821/4131399852","state":["exists","up"]},{"osd":9,"uuid":"78fa7f17-1dad-4262-be40-31fa82e75520","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":60,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.52:6826","nonce":479119778},{"type":"v1","addr":"172.21.15.52:6827","nonce":479119778}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.52:6828","nonce":479119778},{"type":"v1","addr":"172.21.15.52:6829","nonce":479119778}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.52:6832","nonce":479119778},{"type":"v1","addr":"172.21.15.52:6833","nonce":479119778}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.52:6830","nonce":479119778},{"type":"v1","addr":"172.21.15.52:6831","nonce":479119778}]},"public_addr":"172.21.15.52:6827/479119778","cluster_addr":"172.21.15.52:6829/479119778","heartbeat_back_addr":"172.21.15.52:6833/479119778","heartbeat_front_addr":"172.21.15.52:6831/479119778","state":["exists","up"]},{"osd":10,"uuid":"e6f8e753-e15c-45b5-8ee9-d50f384559f9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":65,"up_thru":76,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6824","nonce":4032386628},{"type":"v1","addr":"172.21.15.105:6825","nonce":4032386628}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6826","nonce":4032386628},{"type":"v1","addr":"172.21.15.105:6827","nonce":4032386628}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6830","nonce":4032386628},{"type":"v1","addr":"172.21.15.105:6831","nonce":4032386628}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6828","nonce":4032386628},{"type":"v1","addr":"172.21.15.105:6829","nonce":4032386628}]},"public_addr":"172.21.15.105:6825/4032386628","cluster_addr":"172.21.15.105:6827/4032386628","heartbeat_back_addr":"172.21.15.105:6831/4032386628","heartbeat_front_addr":"172.21.15.105:6829/4032386628","state":["exists","up"]},{"osd":11,"uuid":"164572bf-67ed-4e8e-b32d-d7a2e8da607b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":71,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6824","nonce":1816308401},{"type":"v1","addr":"172.21.15.173:6825","nonce":1816308401}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6826","nonce":1816308401},{"type":"v1","addr":"172.21.15.173:6827","nonce":1816308401}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6830","nonce":1816308401},{"type":"v1","addr":"172.21.15.173:6831","nonce":1816308401}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6828","nonce":1816308401},{"type":"v1","addr":"172.21.15.173:6829","nonce":1816308401}]},"public_addr":"172.21.15.173:6825/1816308401","cluster_addr":"172.21.15.173:6827/1816308401","heartbeat_back_addr":"172.21.15.173:6831/1816308401","heartbeat_front_addr":"172.21.15.173:6829/1816308401","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-27T07:16:06.279220+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-27T07:16:29.668167+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-27T07:16:53.903808+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-27T07:17:18.787372+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-27T07:17:43.480784+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-27T07:18:08.816030+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-27T07:18:34.076148+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-27T07:19:00.283468+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-27T07:19:25.403023+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-27T07:19:52.139173+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-27T07:20:17.530304+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-27T07:20:43.665632+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.8","mappings":[{"from":2,"to":11}]},{"pgid":"2.f","mappings":[{"from":8,"to":11}]},{"pgid":"2.10","mappings":[{"from":2,"to":4}]},{"pgid":"3.0","mappings":[{"from":5,"to":9}]},{"pgid":"3.1b","mappings":[{"from":4,"to":8}]},{"pgid":"3.2c","mappings":[{"from":4,"to":9}]},{"pgid":"3.3c","mappings":[{"from":4,"to":9}]},{"pgid":"4.e","mappings":[{"from":9,"to":11}]},{"pgid":"4.12","mappings":[{"from":5,"to":10}]},{"pgid":"4.19","mappings":[{"from":9,"to":8}]},{"pgid":"4.1f","mappings":[{"from":9,"to":8}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.52:6801/1631683163":"2024-05-28T07:14:29.544633+0000","172.21.15.52:0/2776144659":"2024-05-28T07:14:29.544633+0000","172.21.15.52:6800/1631683163":"2024-05-28T07:14:29.544633+0000","172.21.15.52:0/3675529097":"2024-05-28T07:14:01.960550+0000","172.21.15.52:0/727808107":"2024-05-28T07:14:01.960550+0000","172.21.15.52:0/3839436833":"2024-05-28T07:14:29.544633+0000","172.21.15.52:6800/587662512":"2024-05-28T07:14:01.960550+0000","172.21.15.52:6801/587662512":"2024-05-28T07:14:01.960550+0000","172.21.15.52:0/853867465":"2024-05-28T07:14:01.960550+0000","172.21.15.52:0/3480918279":"2024-05-28T07:14:29.544633+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-05-27T07:23:34.894 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:23:34.904+0000 7fa684350700 1 -- 172.21.15.52:0/1179944392 >> [v2:172.21.15.52:6800/3513075805,v1:172.21.15.52:6801/3513075805] conn(0x7fa6640b2910 msgr2=0x7fa6640b4e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:23:34.894 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:23:34.904+0000 7fa684350700 1 --2- 172.21.15.52:0/1179944392 >> [v2:172.21.15.52:6800/3513075805,v1:172.21.15.52:6801/3513075805] conn(0x7fa6640b2910 0x7fa6640b4e20 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7fa67c1087e0 tx=0x7fa67000b410 comp rx=0 tx=0).stop 2024-05-27T07:23:34.894 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:23:34.904+0000 7fa684350700 1 -- 172.21.15.52:0/1179944392 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fa67c1045d0 msgr2=0x7fa67c19adf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:23:34.894 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:23:34.904+0000 7fa684350700 1 --2- 172.21.15.52:0/1179944392 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fa67c1045d0 0x7fa67c19adf0 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7fa66c007f10 tx=0x7fa66c00d3b0 comp rx=0 tx=0).stop 2024-05-27T07:23:34.894 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:23:34.904+0000 7fa684350700 1 -- 172.21.15.52:0/1179944392 shutdown_connections 2024-05-27T07:23:34.895 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:23:34.904+0000 7fa684350700 1 --2- 172.21.15.52:0/1179944392 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fa67c109850 0x7fa67c19f5d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:23:34.895 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:23:34.904+0000 7fa684350700 1 --2- 172.21.15.52:0/1179944392 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fa67c1045d0 0x7fa67c19adf0 unknown :-1 s=CLOSED pgs=250 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:23:34.895 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:23:34.904+0000 7fa684350700 1 --2- 172.21.15.52:0/1179944392 >> [v2:172.21.15.52:6800/3513075805,v1:172.21.15.52:6801/3513075805] conn(0x7fa6640b2910 0x7fa6640b4e20 unknown :-1 s=CLOSED pgs=174 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:23:34.895 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:23:34.904+0000 7fa684350700 1 --2- 172.21.15.52:0/1179944392 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fa67c108f00 0x7fa67c19b390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:23:34.895 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:23:34.904+0000 7fa684350700 1 -- 172.21.15.52:0/1179944392 >> 172.21.15.52:0/1179944392 conn(0x7fa67c0ffe60 msgr2=0x7fa67c107030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:23:34.895 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:23:34.904+0000 7fa684350700 1 -- 172.21.15.52:0/1179944392 shutdown_connections 2024-05-27T07:23:34.895 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:23:34.904+0000 7fa684350700 1 -- 172.21.15.52:0/1179944392 wait complete. 2024-05-27T07:23:35.229 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:23:34 smithi052 ceph-mon[31955]: pgmap v352: 129 pgs: 129 active+clean; 645 KiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s; 0 B/s, 0 objects/s recovering 2024-05-27T07:23:35.229 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:23:34 smithi052 ceph-mon[31955]: from='client.? 172.21.15.52:0/1179944392' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-05-27T07:23:35.352 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:23:34 smithi173 ceph-mon[37000]: pgmap v352: 129 pgs: 129 active+clean; 645 KiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s; 0 B/s, 0 objects/s recovering 2024-05-27T07:23:35.352 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:23:34 smithi173 ceph-mon[37000]: from='client.? 172.21.15.52:0/1179944392' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-05-27T07:23:35.440 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:23:34 smithi105 ceph-mon[35977]: pgmap v352: 129 pgs: 129 active+clean; 645 KiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s; 0 B/s, 0 objects/s recovering 2024-05-27T07:23:35.440 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:23:34 smithi105 ceph-mon[35977]: from='client.? 172.21.15.52:0/1179944392' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-05-27T07:23:35.758 INFO:tasks.cephadm.ceph_manager.ceph:all up! 2024-05-27T07:23:35.758 DEBUG:teuthology.orchestra.run.smithi052:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:38f9d900c83c63799fbdbe61acc9a11b0d3554a6 shell --fsid 73df2636-1bf8-11ef-bc9b-c7b262605968 -- ceph osd dump --format=json 2024-05-27T07:23:36.017 INFO:teuthology.orchestra.run.smithi052.stderr:Inferring config /var/lib/ceph/73df2636-1bf8-11ef-bc9b-c7b262605968/mon.a/config 2024-05-27T07:23:37.086 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:23:36 smithi052 ceph-mon[31955]: pgmap v353: 129 pgs: 129 active+clean; 647 KiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail; 5.9 KiB/s rd, 255 B/s wr, 10 op/s; 0 B/s, 0 objects/s recovering 2024-05-27T07:23:37.352 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:23:36 smithi173 ceph-mon[37000]: pgmap v353: 129 pgs: 129 active+clean; 647 KiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail; 5.9 KiB/s rd, 255 B/s wr, 10 op/s; 0 B/s, 0 objects/s recovering 2024-05-27T07:23:37.440 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:23:36 smithi105 ceph-mon[35977]: pgmap v353: 129 pgs: 129 active+clean; 647 KiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail; 5.9 KiB/s rd, 255 B/s wr, 10 op/s; 0 B/s, 0 objects/s recovering 2024-05-27T07:23:37.810 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:23:37.811+0000 7f23f19e4700 1 -- 172.21.15.52:0/2486960324 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f23ec109840 msgr2=0x7f23ec1111a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:23:37.811 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:23:37.811+0000 7f23f19e4700 1 --2- 172.21.15.52:0/2486960324 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f23ec109840 0x7f23ec1111a0 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7f23e400b0d0 tx=0x7f23e400b3e0 comp rx=0 tx=0).stop 2024-05-27T07:23:37.811 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:23:37.811+0000 7f23f19e4700 1 -- 172.21.15.52:0/2486960324 shutdown_connections 2024-05-27T07:23:37.811 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:23:37.811+0000 7f23f19e4700 1 --2- 172.21.15.52:0/2486960324 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f23ec109840 0x7f23ec1111a0 unknown :-1 s=CLOSED pgs=251 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:23:37.811 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:23:37.811+0000 7f23f19e4700 1 --2- 172.21.15.52:0/2486960324 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f23ec102ad0 0x7f23ec102ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:23:37.811 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:23:37.811+0000 7f23f19e4700 1 --2- 172.21.15.52:0/2486960324 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f23ec108ef0 0x7f23ec109300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:23:37.811 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:23:37.811+0000 7f23f19e4700 1 -- 172.21.15.52:0/2486960324 >> 172.21.15.52:0/2486960324 conn(0x7f23ec0fe530 msgr2=0x7f23ec100980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:23:37.811 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:23:37.811+0000 7f23f19e4700 1 -- 172.21.15.52:0/2486960324 shutdown_connections 2024-05-27T07:23:37.811 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:23:37.811+0000 7f23f19e4700 1 -- 172.21.15.52:0/2486960324 wait complete. 2024-05-27T07:23:37.811 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:23:37.812+0000 7f23f19e4700 1 Processor -- start 2024-05-27T07:23:37.811 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:23:37.812+0000 7f23f19e4700 1 -- start start 2024-05-27T07:23:37.811 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:23:37.812+0000 7f23f19e4700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f23ec102ad0 0x7f23ec19ae00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:23:37.812 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:23:37.812+0000 7f23f19e4700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f23ec108ef0 0x7f23ec19b3a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:23:37.812 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:23:37.812+0000 7f23f19e4700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f23ec109840 0x7f23ec19f5e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:23:37.812 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:23:37.812+0000 7f23f19e4700 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x7f23ec114470 con 0x7f23ec102ad0 2024-05-27T07:23:37.812 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:23:37.812+0000 7f23f19e4700 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f23ec1142f0 con 0x7f23ec108ef0 2024-05-27T07:23:37.812 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:23:37.812+0000 7f23f19e4700 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f23ec1145f0 con 0x7f23ec109840 2024-05-27T07:23:37.812 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:23:37.812+0000 7f23eaffd700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f23ec102ad0 0x7f23ec19ae00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:23:37.812 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:23:37.812+0000 7f23eaffd700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f23ec102ad0 0x7f23ec19ae00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.52:57002/0 (socket says 172.21.15.52:57002) 2024-05-27T07:23:37.812 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:23:37.813+0000 7f23eaffd700 1 -- 172.21.15.52:0/3367693299 learned_addr learned my addr 172.21.15.52:0/3367693299 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-05-27T07:23:37.812 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:23:37.813+0000 7f23eb7fe700 1 --2- 172.21.15.52:0/3367693299 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f23ec109840 0x7f23ec19f5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:23:37.812 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:23:37.813+0000 7f23ea7fc700 1 --2- 172.21.15.52:0/3367693299 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f23ec108ef0 0x7f23ec19b3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:23:37.812 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:23:37.813+0000 7f23eaffd700 1 -- 172.21.15.52:0/3367693299 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f23ec109840 msgr2=0x7f23ec19f5e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:23:37.812 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:23:37.813+0000 7f23eaffd700 1 --2- 172.21.15.52:0/3367693299 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f23ec109840 0x7f23ec19f5e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:23:37.812 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:23:37.813+0000 7f23eaffd700 1 -- 172.21.15.52:0/3367693299 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f23ec108ef0 msgr2=0x7f23ec19b3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:23:37.813 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:23:37.813+0000 7f23eaffd700 1 --2- 172.21.15.52:0/3367693299 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f23ec108ef0 0x7f23ec19b3a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:23:37.813 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:23:37.813+0000 7f23eaffd700 1 -- 172.21.15.52:0/3367693299 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f23ec19fbd0 con 0x7f23ec102ad0 2024-05-27T07:23:37.813 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:23:37.813+0000 7f23eaffd700 1 --2- 172.21.15.52:0/3367693299 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f23ec102ad0 0x7f23ec19ae00 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7f23dc00c9b0 tx=0x7f23dc00ccc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:23:37.813 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:23:37.813+0000 7f23f09e2700 1 -- 172.21.15.52:0/3367693299 <== mon.0 v2:172.21.15.52:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f23dc007b00 con 0x7f23ec102ad0 2024-05-27T07:23:37.813 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:23:37.813+0000 7f23f09e2700 1 -- 172.21.15.52:0/3367693299 <== mon.0 v2:172.21.15.52:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f23dc00ce50 con 0x7f23ec102ad0 2024-05-27T07:23:37.813 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:23:37.813+0000 7f23f19e4700 1 -- 172.21.15.52:0/3367693299 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f23ec19fec0 con 0x7f23ec102ad0 2024-05-27T07:23:37.813 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:23:37.813+0000 7f23f09e2700 1 -- 172.21.15.52:0/3367693299 <== mon.0 v2:172.21.15.52:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f23dc018660 con 0x7f23ec102ad0 2024-05-27T07:23:37.813 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:23:37.813+0000 7f23f19e4700 1 -- 172.21.15.52:0/3367693299 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f23ec1a0400 con 0x7f23ec102ad0 2024-05-27T07:23:37.813 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:23:37.814+0000 7f23f19e4700 1 -- 172.21.15.52:0/3367693299 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f23ec0512b0 con 0x7f23ec102ad0 2024-05-27T07:23:37.813 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:23:37.816+0000 7f23f09e2700 1 -- 172.21.15.52:0/3367693299 <== mon.0 v2:172.21.15.52:3300/0 4 ==== mgrmap(e 17) ==== 149703+0+0 (secure 0 0 0) 0x7f23dc00f450 con 0x7f23ec102ad0 2024-05-27T07:23:37.813 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:23:37.817+0000 7f23f09e2700 1 --2- 172.21.15.52:0/3367693299 >> [v2:172.21.15.52:6800/3513075805,v1:172.21.15.52:6801/3513075805] conn(0x7f23d40b2910 0x7f23d40b4e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:23:37.813 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:23:37.817+0000 7f23ea7fc700 1 --2- 172.21.15.52:0/3367693299 >> [v2:172.21.15.52:6800/3513075805,v1:172.21.15.52:6801/3513075805] conn(0x7f23d40b2910 0x7f23d40b4e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:23:37.814 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:23:37.817+0000 7f23f09e2700 1 -- 172.21.15.52:0/3367693299 <== mon.0 v2:172.21.15.52:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9121+0+0 (secure 0 0 0) 0x7f23dc0e2920 con 0x7f23ec102ad0 2024-05-27T07:23:37.814 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:23:37.818+0000 7f23ea7fc700 1 --2- 172.21.15.52:0/3367693299 >> [v2:172.21.15.52:6800/3513075805,v1:172.21.15.52:6801/3513075805] conn(0x7f23d40b2910 0x7f23d40b4e20 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7f23ec1087c0 tx=0x7f23e00073d0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:23:37.814 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:23:37.821+0000 7f23f09e2700 1 -- 172.21.15.52:0/3367693299 <== mon.0 v2:172.21.15.52:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191963 (secure 0 0 0) 0x7f23dc014030 con 0x7f23ec102ad0 2024-05-27T07:23:38.031 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:23:38.041+0000 7f23f19e4700 1 -- 172.21.15.52:0/3367693299 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f23ec19c140 con 0x7f23ec102ad0 2024-05-27T07:23:38.033 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:23:38.043+0000 7f23f09e2700 1 -- 172.21.15.52:0/3367693299 <== mon.0 v2:172.21.15.52:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v90) ==== 74+0+24701 (secure 0 0 0) 0x7f23ec19c140 con 0x7f23ec102ad0 2024-05-27T07:23:38.033 INFO:teuthology.orchestra.run.smithi052.stdout: 2024-05-27T07:23:38.034 INFO:teuthology.orchestra.run.smithi052.stdout:{"epoch":90,"fsid":"73df2636-1bf8-11ef-bc9b-c7b262605968","created":"2024-05-27T07:13:34.738032+0000","modified":"2024-05-27T07:23:30.969990+0000","last_up_change":"2024-05-27T07:20:45.777597+0000","last_in_change":"2024-05-27T07:20:25.314823+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-05-27T07:16:57.644034+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-05-27T07:21:43.917748+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"79","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-05-27T07:21:47.341876+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"79","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-05-27T07:21:57.707611+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"86","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"347b8a89-9dbe-4848-83b0-fd0e9374f33d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":82,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.52:6802","nonce":1626508899},{"type":"v1","addr":"172.21.15.52:6803","nonce":1626508899}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.52:6804","nonce":1626508899},{"type":"v1","addr":"172.21.15.52:6805","nonce":1626508899}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.52:6808","nonce":1626508899},{"type":"v1","addr":"172.21.15.52:6809","nonce":1626508899}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.52:6806","nonce":1626508899},{"type":"v1","addr":"172.21.15.52:6807","nonce":1626508899}]},"public_addr":"172.21.15.52:6803/1626508899","cluster_addr":"172.21.15.52:6805/1626508899","heartbeat_back_addr":"172.21.15.52:6809/1626508899","heartbeat_front_addr":"172.21.15.52:6807/1626508899","state":["exists","up"]},{"osd":1,"uuid":"7a3623d4-e087-40cb-9336-d4a5c52203d4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6800","nonce":2900093195},{"type":"v1","addr":"172.21.15.105:6801","nonce":2900093195}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6802","nonce":2900093195},{"type":"v1","addr":"172.21.15.105:6803","nonce":2900093195}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6806","nonce":2900093195},{"type":"v1","addr":"172.21.15.105:6807","nonce":2900093195}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6804","nonce":2900093195},{"type":"v1","addr":"172.21.15.105:6805","nonce":2900093195}]},"public_addr":"172.21.15.105:6801/2900093195","cluster_addr":"172.21.15.105:6803/2900093195","heartbeat_back_addr":"172.21.15.105:6807/2900093195","heartbeat_front_addr":"172.21.15.105:6805/2900093195","state":["exists","up"]},{"osd":2,"uuid":"41a9df5b-882a-47da-bb26-7f4d26ccf6c1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6800","nonce":2251189314},{"type":"v1","addr":"172.21.15.173:6801","nonce":2251189314}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6802","nonce":2251189314},{"type":"v1","addr":"172.21.15.173:6803","nonce":2251189314}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6806","nonce":2251189314},{"type":"v1","addr":"172.21.15.173:6807","nonce":2251189314}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6804","nonce":2251189314},{"type":"v1","addr":"172.21.15.173:6805","nonce":2251189314}]},"public_addr":"172.21.15.173:6801/2251189314","cluster_addr":"172.21.15.173:6803/2251189314","heartbeat_back_addr":"172.21.15.173:6807/2251189314","heartbeat_front_addr":"172.21.15.173:6805/2251189314","state":["exists","up"]},{"osd":3,"uuid":"f1245439-42a5-46c5-a998-e0a78f9df3fb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":26,"up_thru":82,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.52:6810","nonce":57920794},{"type":"v1","addr":"172.21.15.52:6811","nonce":57920794}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.52:6812","nonce":57920794},{"type":"v1","addr":"172.21.15.52:6813","nonce":57920794}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.52:6816","nonce":57920794},{"type":"v1","addr":"172.21.15.52:6817","nonce":57920794}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.52:6814","nonce":57920794},{"type":"v1","addr":"172.21.15.52:6815","nonce":57920794}]},"public_addr":"172.21.15.52:6811/57920794","cluster_addr":"172.21.15.52:6813/57920794","heartbeat_back_addr":"172.21.15.52:6817/57920794","heartbeat_front_addr":"172.21.15.52:6815/57920794","state":["exists","up"]},{"osd":4,"uuid":"2592d1cc-1726-4359-83d8-a2e6314b54ec","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":32,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6808","nonce":3441134818},{"type":"v1","addr":"172.21.15.105:6809","nonce":3441134818}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6810","nonce":3441134818},{"type":"v1","addr":"172.21.15.105:6811","nonce":3441134818}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6814","nonce":3441134818},{"type":"v1","addr":"172.21.15.105:6815","nonce":3441134818}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6812","nonce":3441134818},{"type":"v1","addr":"172.21.15.105:6813","nonce":3441134818}]},"public_addr":"172.21.15.105:6809/3441134818","cluster_addr":"172.21.15.105:6811/3441134818","heartbeat_back_addr":"172.21.15.105:6815/3441134818","heartbeat_front_addr":"172.21.15.105:6813/3441134818","state":["exists","up"]},{"osd":5,"uuid":"27e98ee2-4a92-4972-b2a3-959d0ed90981","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":37,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6808","nonce":2989239729},{"type":"v1","addr":"172.21.15.173:6809","nonce":2989239729}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6810","nonce":2989239729},{"type":"v1","addr":"172.21.15.173:6811","nonce":2989239729}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6814","nonce":2989239729},{"type":"v1","addr":"172.21.15.173:6815","nonce":2989239729}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6812","nonce":2989239729},{"type":"v1","addr":"172.21.15.173:6813","nonce":2989239729}]},"public_addr":"172.21.15.173:6809/2989239729","cluster_addr":"172.21.15.173:6811/2989239729","heartbeat_back_addr":"172.21.15.173:6815/2989239729","heartbeat_front_addr":"172.21.15.173:6813/2989239729","state":["exists","up"]},{"osd":6,"uuid":"2b0ec0cb-f438-4b2c-8642-0778ed374e9a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":43,"up_thru":82,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.52:6818","nonce":4244423065},{"type":"v1","addr":"172.21.15.52:6819","nonce":4244423065}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.52:6820","nonce":4244423065},{"type":"v1","addr":"172.21.15.52:6821","nonce":4244423065}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.52:6824","nonce":4244423065},{"type":"v1","addr":"172.21.15.52:6825","nonce":4244423065}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.52:6822","nonce":4244423065},{"type":"v1","addr":"172.21.15.52:6823","nonce":4244423065}]},"public_addr":"172.21.15.52:6819/4244423065","cluster_addr":"172.21.15.52:6821/4244423065","heartbeat_back_addr":"172.21.15.52:6825/4244423065","heartbeat_front_addr":"172.21.15.52:6823/4244423065","state":["exists","up"]},{"osd":7,"uuid":"cb2a7a6b-8015-43ea-865a-be144e45c51a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":48,"up_thru":82,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6816","nonce":2063611552},{"type":"v1","addr":"172.21.15.105:6817","nonce":2063611552}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6818","nonce":2063611552},{"type":"v1","addr":"172.21.15.105:6819","nonce":2063611552}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6822","nonce":2063611552},{"type":"v1","addr":"172.21.15.105:6823","nonce":2063611552}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6820","nonce":2063611552},{"type":"v1","addr":"172.21.15.105:6821","nonce":2063611552}]},"public_addr":"172.21.15.105:6817/2063611552","cluster_addr":"172.21.15.105:6819/2063611552","heartbeat_back_addr":"172.21.15.105:6823/2063611552","heartbeat_front_addr":"172.21.15.105:6821/2063611552","state":["exists","up"]},{"osd":8,"uuid":"c2fc565d-a0b9-4396-a209-bd884cddcfe3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":54,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6816","nonce":4131399852},{"type":"v1","addr":"172.21.15.173:6817","nonce":4131399852}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6818","nonce":4131399852},{"type":"v1","addr":"172.21.15.173:6819","nonce":4131399852}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6822","nonce":4131399852},{"type":"v1","addr":"172.21.15.173:6823","nonce":4131399852}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6820","nonce":4131399852},{"type":"v1","addr":"172.21.15.173:6821","nonce":4131399852}]},"public_addr":"172.21.15.173:6817/4131399852","cluster_addr":"172.21.15.173:6819/4131399852","heartbeat_back_addr":"172.21.15.173:6823/4131399852","heartbeat_front_addr":"172.21.15.173:6821/4131399852","state":["exists","up"]},{"osd":9,"uuid":"78fa7f17-1dad-4262-be40-31fa82e75520","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":60,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.52:6826","nonce":479119778},{"type":"v1","addr":"172.21.15.52:6827","nonce":479119778}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.52:6828","nonce":479119778},{"type":"v1","addr":"172.21.15.52:6829","nonce":479119778}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.52:6832","nonce":479119778},{"type":"v1","addr":"172.21.15.52:6833","nonce":479119778}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.52:6830","nonce":479119778},{"type":"v1","addr":"172.21.15.52:6831","nonce":479119778}]},"public_addr":"172.21.15.52:6827/479119778","cluster_addr":"172.21.15.52:6829/479119778","heartbeat_back_addr":"172.21.15.52:6833/479119778","heartbeat_front_addr":"172.21.15.52:6831/479119778","state":["exists","up"]},{"osd":10,"uuid":"e6f8e753-e15c-45b5-8ee9-d50f384559f9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":65,"up_thru":76,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6824","nonce":4032386628},{"type":"v1","addr":"172.21.15.105:6825","nonce":4032386628}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6826","nonce":4032386628},{"type":"v1","addr":"172.21.15.105:6827","nonce":4032386628}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6830","nonce":4032386628},{"type":"v1","addr":"172.21.15.105:6831","nonce":4032386628}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6828","nonce":4032386628},{"type":"v1","addr":"172.21.15.105:6829","nonce":4032386628}]},"public_addr":"172.21.15.105:6825/4032386628","cluster_addr":"172.21.15.105:6827/4032386628","heartbeat_back_addr":"172.21.15.105:6831/4032386628","heartbeat_front_addr":"172.21.15.105:6829/4032386628","state":["exists","up"]},{"osd":11,"uuid":"164572bf-67ed-4e8e-b32d-d7a2e8da607b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":71,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6824","nonce":1816308401},{"type":"v1","addr":"172.21.15.173:6825","nonce":1816308401}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6826","nonce":1816308401},{"type":"v1","addr":"172.21.15.173:6827","nonce":1816308401}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6830","nonce":1816308401},{"type":"v1","addr":"172.21.15.173:6831","nonce":1816308401}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6828","nonce":1816308401},{"type":"v1","addr":"172.21.15.173:6829","nonce":1816308401}]},"public_addr":"172.21.15.173:6825/1816308401","cluster_addr":"172.21.15.173:6827/1816308401","heartbeat_back_addr":"172.21.15.173:6831/1816308401","heartbeat_front_addr":"172.21.15.173:6829/1816308401","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-27T07:16:06.279220+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-27T07:16:29.668167+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-27T07:16:53.903808+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-27T07:17:18.787372+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-27T07:17:43.480784+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-27T07:18:08.816030+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-27T07:18:34.076148+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-27T07:19:00.283468+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-27T07:19:25.403023+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-27T07:19:52.139173+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-27T07:20:17.530304+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-27T07:20:43.665632+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.8","mappings":[{"from":2,"to":11}]},{"pgid":"2.f","mappings":[{"from":8,"to":11}]},{"pgid":"2.10","mappings":[{"from":2,"to":4}]},{"pgid":"3.0","mappings":[{"from":5,"to":9}]},{"pgid":"3.1b","mappings":[{"from":4,"to":8}]},{"pgid":"3.2c","mappings":[{"from":4,"to":9}]},{"pgid":"3.3c","mappings":[{"from":4,"to":9}]},{"pgid":"4.e","mappings":[{"from":9,"to":11}]},{"pgid":"4.12","mappings":[{"from":5,"to":10}]},{"pgid":"4.19","mappings":[{"from":9,"to":8}]},{"pgid":"4.1f","mappings":[{"from":9,"to":8}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.52:6801/1631683163":"2024-05-28T07:14:29.544633+0000","172.21.15.52:0/2776144659":"2024-05-28T07:14:29.544633+0000","172.21.15.52:6800/1631683163":"2024-05-28T07:14:29.544633+0000","172.21.15.52:0/3675529097":"2024-05-28T07:14:01.960550+0000","172.21.15.52:0/727808107":"2024-05-28T07:14:01.960550+0000","172.21.15.52:0/3839436833":"2024-05-28T07:14:29.544633+0000","172.21.15.52:6800/587662512":"2024-05-28T07:14:01.960550+0000","172.21.15.52:6801/587662512":"2024-05-28T07:14:01.960550+0000","172.21.15.52:0/853867465":"2024-05-28T07:14:01.960550+0000","172.21.15.52:0/3480918279":"2024-05-28T07:14:29.544633+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-05-27T07:23:38.035 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:23:38.045+0000 7f23f19e4700 1 -- 172.21.15.52:0/3367693299 >> [v2:172.21.15.52:6800/3513075805,v1:172.21.15.52:6801/3513075805] conn(0x7f23d40b2910 msgr2=0x7f23d40b4e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:23:38.035 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:23:38.045+0000 7f23f19e4700 1 --2- 172.21.15.52:0/3367693299 >> [v2:172.21.15.52:6800/3513075805,v1:172.21.15.52:6801/3513075805] conn(0x7f23d40b2910 0x7f23d40b4e20 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7f23ec1087c0 tx=0x7f23e00073d0 comp rx=0 tx=0).stop 2024-05-27T07:23:38.035 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:23:38.045+0000 7f23f19e4700 1 -- 172.21.15.52:0/3367693299 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f23ec102ad0 msgr2=0x7f23ec19ae00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:23:38.036 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:23:38.045+0000 7f23f19e4700 1 --2- 172.21.15.52:0/3367693299 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f23ec102ad0 0x7f23ec19ae00 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7f23dc00c9b0 tx=0x7f23dc00ccc0 comp rx=0 tx=0).stop 2024-05-27T07:23:38.036 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:23:38.045+0000 7f23f19e4700 1 -- 172.21.15.52:0/3367693299 shutdown_connections 2024-05-27T07:23:38.036 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:23:38.045+0000 7f23f19e4700 1 --2- 172.21.15.52:0/3367693299 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f23ec109840 0x7f23ec19f5e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:23:38.036 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:23:38.045+0000 7f23f19e4700 1 --2- 172.21.15.52:0/3367693299 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f23ec102ad0 0x7f23ec19ae00 unknown :-1 s=CLOSED pgs=252 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:23:38.036 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:23:38.045+0000 7f23f19e4700 1 --2- 172.21.15.52:0/3367693299 >> [v2:172.21.15.52:6800/3513075805,v1:172.21.15.52:6801/3513075805] conn(0x7f23d40b2910 0x7f23d40b4e20 unknown :-1 s=CLOSED pgs=175 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:23:38.036 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:23:38.045+0000 7f23f19e4700 1 --2- 172.21.15.52:0/3367693299 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f23ec108ef0 0x7f23ec19b3a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:23:38.036 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:23:38.045+0000 7f23f19e4700 1 -- 172.21.15.52:0/3367693299 >> 172.21.15.52:0/3367693299 conn(0x7f23ec0fe530 msgr2=0x7f23ec107080 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:23:38.036 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:23:38.046+0000 7f23f19e4700 1 -- 172.21.15.52:0/3367693299 shutdown_connections 2024-05-27T07:23:38.036 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:23:38.046+0000 7f23f19e4700 1 -- 172.21.15.52:0/3367693299 wait complete. 2024-05-27T07:23:38.686 DEBUG:teuthology.orchestra.run.smithi052:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:38f9d900c83c63799fbdbe61acc9a11b0d3554a6 shell --fsid 73df2636-1bf8-11ef-bc9b-c7b262605968 -- ceph tell osd.0 flush_pg_stats 2024-05-27T07:23:38.686 DEBUG:teuthology.orchestra.run.smithi052:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:38f9d900c83c63799fbdbe61acc9a11b0d3554a6 shell --fsid 73df2636-1bf8-11ef-bc9b-c7b262605968 -- ceph tell osd.1 flush_pg_stats 2024-05-27T07:23:38.687 DEBUG:teuthology.orchestra.run.smithi052:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:38f9d900c83c63799fbdbe61acc9a11b0d3554a6 shell --fsid 73df2636-1bf8-11ef-bc9b-c7b262605968 -- ceph tell osd.2 flush_pg_stats 2024-05-27T07:23:38.687 DEBUG:teuthology.orchestra.run.smithi052:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:38f9d900c83c63799fbdbe61acc9a11b0d3554a6 shell --fsid 73df2636-1bf8-11ef-bc9b-c7b262605968 -- ceph tell osd.3 flush_pg_stats 2024-05-27T07:23:38.688 DEBUG:teuthology.orchestra.run.smithi052:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:38f9d900c83c63799fbdbe61acc9a11b0d3554a6 shell --fsid 73df2636-1bf8-11ef-bc9b-c7b262605968 -- ceph tell osd.4 flush_pg_stats 2024-05-27T07:23:38.688 DEBUG:teuthology.orchestra.run.smithi052:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:38f9d900c83c63799fbdbe61acc9a11b0d3554a6 shell --fsid 73df2636-1bf8-11ef-bc9b-c7b262605968 -- ceph tell osd.5 flush_pg_stats 2024-05-27T07:23:38.688 DEBUG:teuthology.orchestra.run.smithi052:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:38f9d900c83c63799fbdbe61acc9a11b0d3554a6 shell --fsid 73df2636-1bf8-11ef-bc9b-c7b262605968 -- ceph tell osd.6 flush_pg_stats 2024-05-27T07:23:38.689 DEBUG:teuthology.orchestra.run.smithi052:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:38f9d900c83c63799fbdbe61acc9a11b0d3554a6 shell --fsid 73df2636-1bf8-11ef-bc9b-c7b262605968 -- ceph tell osd.7 flush_pg_stats 2024-05-27T07:23:38.689 DEBUG:teuthology.orchestra.run.smithi052:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:38f9d900c83c63799fbdbe61acc9a11b0d3554a6 shell --fsid 73df2636-1bf8-11ef-bc9b-c7b262605968 -- ceph tell osd.8 flush_pg_stats 2024-05-27T07:23:38.690 DEBUG:teuthology.orchestra.run.smithi052:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:38f9d900c83c63799fbdbe61acc9a11b0d3554a6 shell --fsid 73df2636-1bf8-11ef-bc9b-c7b262605968 -- ceph tell osd.9 flush_pg_stats 2024-05-27T07:23:38.690 DEBUG:teuthology.orchestra.run.smithi052:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:38f9d900c83c63799fbdbe61acc9a11b0d3554a6 shell --fsid 73df2636-1bf8-11ef-bc9b-c7b262605968 -- ceph tell osd.10 flush_pg_stats 2024-05-27T07:23:38.691 DEBUG:teuthology.orchestra.run.smithi052:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:38f9d900c83c63799fbdbe61acc9a11b0d3554a6 shell --fsid 73df2636-1bf8-11ef-bc9b-c7b262605968 -- ceph tell osd.11 flush_pg_stats 2024-05-27T07:23:39.352 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:23:38 smithi173 ceph-mon[37000]: pgmap v354: 129 pgs: 129 active+clean; 647 KiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 255 B/s wr, 10 op/s; 0 B/s, 0 objects/s recovering 2024-05-27T07:23:39.352 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:23:38 smithi173 ceph-mon[37000]: from='client.? 172.21.15.52:0/3367693299' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-05-27T07:23:39.355 INFO:teuthology.orchestra.run.smithi052.stderr:Inferring config /var/lib/ceph/73df2636-1bf8-11ef-bc9b-c7b262605968/mon.a/config 2024-05-27T07:23:39.440 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:23:38 smithi105 ceph-mon[35977]: pgmap v354: 129 pgs: 129 active+clean; 647 KiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 255 B/s wr, 10 op/s; 0 B/s, 0 objects/s recovering 2024-05-27T07:23:39.440 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:23:38 smithi105 ceph-mon[35977]: from='client.? 172.21.15.52:0/3367693299' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-05-27T07:23:39.441 INFO:teuthology.orchestra.run.smithi052.stderr:Inferring config /var/lib/ceph/73df2636-1bf8-11ef-bc9b-c7b262605968/mon.a/config 2024-05-27T07:23:39.479 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:23:38 smithi052 ceph-mon[31955]: pgmap v354: 129 pgs: 129 active+clean; 647 KiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 255 B/s wr, 10 op/s; 0 B/s, 0 objects/s recovering 2024-05-27T07:23:39.479 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:23:38 smithi052 ceph-mon[31955]: from='client.? 172.21.15.52:0/3367693299' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-05-27T07:23:40.756 INFO:teuthology.orchestra.run.smithi052.stderr:Inferring config /var/lib/ceph/73df2636-1bf8-11ef-bc9b-c7b262605968/mon.a/config 2024-05-27T07:23:40.756 INFO:teuthology.orchestra.run.smithi052.stderr:Inferring config /var/lib/ceph/73df2636-1bf8-11ef-bc9b-c7b262605968/mon.a/config 2024-05-27T07:23:40.758 INFO:teuthology.orchestra.run.smithi052.stderr:Inferring config /var/lib/ceph/73df2636-1bf8-11ef-bc9b-c7b262605968/mon.a/config 2024-05-27T07:23:40.765 INFO:teuthology.orchestra.run.smithi052.stderr:Inferring config /var/lib/ceph/73df2636-1bf8-11ef-bc9b-c7b262605968/mon.a/config 2024-05-27T07:23:40.765 INFO:teuthology.orchestra.run.smithi052.stderr:Inferring config /var/lib/ceph/73df2636-1bf8-11ef-bc9b-c7b262605968/mon.a/config 2024-05-27T07:23:40.766 INFO:teuthology.orchestra.run.smithi052.stderr:Inferring config /var/lib/ceph/73df2636-1bf8-11ef-bc9b-c7b262605968/mon.a/config 2024-05-27T07:23:40.766 INFO:teuthology.orchestra.run.smithi052.stderr:Inferring config /var/lib/ceph/73df2636-1bf8-11ef-bc9b-c7b262605968/mon.a/config 2024-05-27T07:23:40.767 INFO:teuthology.orchestra.run.smithi052.stderr:Inferring config /var/lib/ceph/73df2636-1bf8-11ef-bc9b-c7b262605968/mon.a/config 2024-05-27T07:23:40.770 INFO:teuthology.orchestra.run.smithi052.stderr:Inferring config /var/lib/ceph/73df2636-1bf8-11ef-bc9b-c7b262605968/mon.a/config 2024-05-27T07:23:40.798 INFO:teuthology.orchestra.run.smithi052.stderr:Inferring config /var/lib/ceph/73df2636-1bf8-11ef-bc9b-c7b262605968/mon.a/config 2024-05-27T07:23:41.182 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:23:40 smithi052 ceph-mon[31955]: pgmap v355: 129 pgs: 129 active+clean; 647 KiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 211 B/s wr, 10 op/s; 0 B/s, 0 objects/s recovering 2024-05-27T07:23:41.351 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:23:40 smithi173 ceph-mon[37000]: pgmap v355: 129 pgs: 129 active+clean; 647 KiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 211 B/s wr, 10 op/s; 0 B/s, 0 objects/s recovering 2024-05-27T07:23:41.440 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:23:40 smithi105 ceph-mon[35977]: pgmap v355: 129 pgs: 129 active+clean; 647 KiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 211 B/s wr, 10 op/s; 0 B/s, 0 objects/s recovering 2024-05-27T07:23:43.060 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:23:43 smithi052 ceph-mon[31955]: pgmap v356: 129 pgs: 129 active+clean; 647 KiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 191 B/s wr, 9 op/s; 0 B/s, 0 objects/s recovering 2024-05-27T07:23:43.352 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:23:43 smithi173 ceph-mon[37000]: pgmap v356: 129 pgs: 129 active+clean; 647 KiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 191 B/s wr, 9 op/s; 0 B/s, 0 objects/s recovering 2024-05-27T07:23:43.440 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:23:43 smithi105 ceph-mon[35977]: pgmap v356: 129 pgs: 129 active+clean; 647 KiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 191 B/s wr, 9 op/s; 0 B/s, 0 objects/s recovering 2024-05-27T07:23:45.312 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:23:45 smithi052 ceph-mon[31955]: pgmap v357: 129 pgs: 129 active+clean; 647 KiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 170 B/s wr, 9 op/s; 0 B/s, 0 objects/s recovering 2024-05-27T07:23:45.352 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:23:45 smithi173 ceph-mon[37000]: pgmap v357: 129 pgs: 129 active+clean; 647 KiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 170 B/s wr, 9 op/s; 0 B/s, 0 objects/s recovering 2024-05-27T07:23:45.440 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:23:45 smithi105 ceph-mon[35977]: pgmap v357: 129 pgs: 129 active+clean; 647 KiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 170 B/s wr, 9 op/s; 0 B/s, 0 objects/s recovering 2024-05-27T07:23:47.229 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:23:47 smithi052 ceph-mon[31955]: pgmap v358: 129 pgs: 129 active+clean; 647 KiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail; 6.0 KiB/s rd, 170 B/s wr, 11 op/s 2024-05-27T07:23:47.352 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:23:47 smithi173 ceph-mon[37000]: pgmap v358: 129 pgs: 129 active+clean; 647 KiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail; 6.0 KiB/s rd, 170 B/s wr, 11 op/s 2024-05-27T07:23:47.440 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:23:47 smithi105 ceph-mon[35977]: pgmap v358: 129 pgs: 129 active+clean; 647 KiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail; 6.0 KiB/s rd, 170 B/s wr, 11 op/s 2024-05-27T07:23:49.352 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:23:49 smithi173 ceph-mon[37000]: pgmap v359: 129 pgs: 129 active+clean; 647 KiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 9 op/s 2024-05-27T07:23:49.391 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:23:49 smithi052 ceph-mon[31955]: pgmap v359: 129 pgs: 129 active+clean; 647 KiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 9 op/s 2024-05-27T07:23:49.440 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:23:49 smithi105 ceph-mon[35977]: pgmap v359: 129 pgs: 129 active+clean; 647 KiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 9 op/s 2024-05-27T07:23:49.501 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:23:46.999+0000 7fcb679ec700 1 -- 172.21.15.52:0/2488418763 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fcb601034d0 msgr2=0x7fcb601038e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:23:49.501 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:23:46.999+0000 7fcb679ec700 1 --2- 172.21.15.52:0/2488418763 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fcb601034d0 0x7fcb601038e0 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7fcb5c0099c0 tx=0x7fcb5c009cd0 comp rx=0 tx=0).stop 2024-05-27T07:23:49.501 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:23:46.999+0000 7fcb679ec700 1 -- 172.21.15.52:0/2488418763 shutdown_connections 2024-05-27T07:23:49.501 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:23:46.999+0000 7fcb679ec700 1 --2- 172.21.15.52:0/2488418763 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fcb60108750 0x7fcb601100b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:23:49.501 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:23:46.999+0000 7fcb679ec700 1 --2- 172.21.15.52:0/2488418763 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fcb601034d0 0x7fcb601038e0 unknown :-1 s=CLOSED pgs=253 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:23:49.501 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:23:46.999+0000 7fcb679ec700 1 --2- 172.21.15.52:0/2488418763 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fcb60107e00 0x7fcb60108210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:23:49.501 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:23:46.999+0000 7fcb679ec700 1 -- 172.21.15.52:0/2488418763 >> 172.21.15.52:0/2488418763 conn(0x7fcb600fef20 msgr2=0x7fcb60101340 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:23:49.502 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:23:46.999+0000 7fcb679ec700 1 -- 172.21.15.52:0/2488418763 shutdown_connections 2024-05-27T07:23:49.502 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:23:46.999+0000 7fcb679ec700 1 -- 172.21.15.52:0/2488418763 wait complete. 2024-05-27T07:23:49.502 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:23:47.000+0000 7fcb679ec700 1 Processor -- start 2024-05-27T07:23:49.502 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:23:47.000+0000 7fcb679ec700 1 -- start start 2024-05-27T07:23:49.502 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:23:47.000+0000 7fcb679ec700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fcb601034d0 0x7fcb6019ade0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:23:49.502 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:23:47.000+0000 7fcb679ec700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fcb60107e00 0x7fcb6019b380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:23:49.502 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:23:47.000+0000 7fcb679ec700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fcb60108750 0x7fcb6019f610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:23:49.502 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:23:47.000+0000 7fcb679ec700 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x7fcb601133a0 con 0x7fcb60108750 2024-05-27T07:23:49.502 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:23:47.000+0000 7fcb679ec700 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7fcb60113220 con 0x7fcb60107e00 2024-05-27T07:23:49.503 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:23:47.000+0000 7fcb679ec700 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7fcb60113520 con 0x7fcb601034d0 2024-05-27T07:23:49.503 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:23:47.001+0000 7fcb65f89700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fcb60108750 0x7fcb6019f610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:23:49.503 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:23:47.001+0000 7fcb65f89700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fcb60108750 0x7fcb6019f610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.52:40800/0 (socket says 172.21.15.52:40800) 2024-05-27T07:23:49.503 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:23:47.001+0000 7fcb65f89700 1 -- 172.21.15.52:0/3423747181 learned_addr learned my addr 172.21.15.52:0/3423747181 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-05-27T07:23:49.503 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:23:47.001+0000 7fcb64f87700 1 --2- 172.21.15.52:0/3423747181 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fcb60107e00 0x7fcb6019b380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:23:49.503 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:23:47.001+0000 7fcb65788700 1 --2- 172.21.15.52:0/3423747181 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fcb601034d0 0x7fcb6019ade0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:23:49.503 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:23:47.001+0000 7fcb65f89700 1 -- 172.21.15.52:0/3423747181 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fcb601034d0 msgr2=0x7fcb6019ade0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:23:49.503 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:23:47.001+0000 7fcb65f89700 1 --2- 172.21.15.52:0/3423747181 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fcb601034d0 0x7fcb6019ade0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:23:49.503 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:23:47.001+0000 7fcb65f89700 1 -- 172.21.15.52:0/3423747181 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fcb60107e00 msgr2=0x7fcb6019b380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:23:49.503 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:23:47.001+0000 7fcb65f89700 1 --2- 172.21.15.52:0/3423747181 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fcb60107e00 0x7fcb6019b380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:23:49.503 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:23:47.001+0000 7fcb65f89700 1 -- 172.21.15.52:0/3423747181 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcb6019fcb0 con 0x7fcb60108750 2024-05-27T07:23:49.504 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:23:47.001+0000 7fcb65f89700 1 --2- 172.21.15.52:0/3423747181 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fcb60108750 0x7fcb6019f610 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7fcb58007ee0 tx=0x7fcb5800d3b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:23:49.504 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:23:47.001+0000 7fcb567fc700 1 -- 172.21.15.52:0/3423747181 <== mon.0 v2:172.21.15.52:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcb58004bb0 con 0x7fcb60108750 2024-05-27T07:23:49.504 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:23:47.001+0000 7fcb567fc700 1 -- 172.21.15.52:0/3423747181 <== mon.0 v2:172.21.15.52:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fcb58004d50 con 0x7fcb60108750 2024-05-27T07:23:49.504 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:23:47.001+0000 7fcb679ec700 1 -- 172.21.15.52:0/3423747181 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fcb6019ffa0 con 0x7fcb60108750 2024-05-27T07:23:49.504 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:23:47.001+0000 7fcb567fc700 1 -- 172.21.15.52:0/3423747181 <== mon.0 v2:172.21.15.52:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcb58005920 con 0x7fcb60108750 2024-05-27T07:23:49.504 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:23:47.001+0000 7fcb679ec700 1 -- 172.21.15.52:0/3423747181 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fcb601a78c0 con 0x7fcb60108750 2024-05-27T07:23:49.504 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:23:47.002+0000 7fcb679ec700 1 -- 172.21.15.52:0/3423747181 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7fcb60063f40 con 0x7fcb60108750 2024-05-27T07:23:49.504 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:23:47.004+0000 7fcb567fc700 1 -- 172.21.15.52:0/3423747181 <== mon.0 v2:172.21.15.52:3300/0 4 ==== mgrmap(e 17) ==== 149703+0+0 (secure 0 0 0) 0x7fcb5801a020 con 0x7fcb60108750 2024-05-27T07:23:49.504 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:23:47.005+0000 7fcb567fc700 1 --2- 172.21.15.52:0/3423747181 >> [v2:172.21.15.52:6800/3513075805,v1:172.21.15.52:6801/3513075805] conn(0x7fcb4c0b2910 0x7fcb4c0b4e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:23:49.504 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:23:47.005+0000 7fcb65788700 1 --2- 172.21.15.52:0/3423747181 >> [v2:172.21.15.52:6800/3513075805,v1:172.21.15.52:6801/3513075805] conn(0x7fcb4c0b2910 0x7fcb4c0b4e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:23:49.505 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:23:47.005+0000 7fcb567fc700 1 -- 172.21.15.52:0/3423747181 <== mon.0 v2:172.21.15.52:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9121+0+0 (secure 0 0 0) 0x7fcb580e2360 con 0x7fcb60108750 2024-05-27T07:23:49.505 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:23:47.005+0000 7fcb567fc700 1 --2- 172.21.15.52:0/3423747181 >> [v2:172.21.15.52:6826/479119778,v1:172.21.15.52:6827/479119778] conn(0x7fcb4c0bdb40 0x7fcb4c0c0030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:23:49.505 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:23:47.005+0000 7fcb567fc700 1 -- 172.21.15.52:0/3423747181 --> [v2:172.21.15.52:6826/479119778,v1:172.21.15.52:6827/479119778] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fcb4c0c0700 con 0x7fcb4c0bdb40 2024-05-27T07:23:49.505 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:23:47.005+0000 7fcb567fc700 1 -- 172.21.15.52:0/3423747181 <== mon.0 v2:172.21.15.52:3300/0 6 ==== mon_get_version_reply(handle=1 version=90) ==== 24+0+0 (secure 0 0 0) 0x7fcb580e2840 con 0x7fcb60108750 2024-05-27T07:23:49.505 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:23:47.006+0000 7fcb64f87700 1 --2- 172.21.15.52:0/3423747181 >> [v2:172.21.15.52:6826/479119778,v1:172.21.15.52:6827/479119778] conn(0x7fcb4c0bdb40 0x7fcb4c0c0030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:23:49.505 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:23:47.006+0000 7fcb65788700 1 --2- 172.21.15.52:0/3423747181 >> [v2:172.21.15.52:6800/3513075805,v1:172.21.15.52:6801/3513075805] conn(0x7fcb4c0b2910 0x7fcb4c0b4e20 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7fcb5c009990 tx=0x7fcb5c0036e0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:23:49.505 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:23:47.006+0000 7fcb64f87700 1 --2- 172.21.15.52:0/3423747181 >> [v2:172.21.15.52:6826/479119778,v1:172.21.15.52:6827/479119778] conn(0x7fcb4c0bdb40 0x7fcb4c0c0030 crc :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.9 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:23:49.506 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:23:47.007+0000 7fcb567fc700 1 -- 172.21.15.52:0/3423747181 <== osd.9 v2:172.21.15.52:6826/479119778 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7fcb4c0c0700 con 0x7fcb4c0bdb40 2024-05-27T07:23:49.506 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:23:47.024+0000 7fcb679ec700 1 -- 172.21.15.52:0/3423747181 --> [v2:172.21.15.52:6826/479119778,v1:172.21.15.52:6827/479119778] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7fcb600643c0 con 0x7fcb4c0bdb40 2024-05-27T07:23:49.506 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:23:47.024+0000 7fcb567fc700 1 -- 172.21.15.52:0/3423747181 <== osd.9 v2:172.21.15.52:6826/479119778 2 ==== command_reply(tid 2: 0 ) ==== 8+0+12 (crc 0 0 0) 0x7fcb600643c0 con 0x7fcb4c0bdb40 2024-05-27T07:23:49.506 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:23:47.025+0000 7fcb679ec700 1 -- 172.21.15.52:0/3423747181 >> [v2:172.21.15.52:6826/479119778,v1:172.21.15.52:6827/479119778] conn(0x7fcb4c0bdb40 msgr2=0x7fcb4c0c0030 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:23:49.506 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:23:47.025+0000 7fcb679ec700 1 --2- 172.21.15.52:0/3423747181 >> [v2:172.21.15.52:6826/479119778,v1:172.21.15.52:6827/479119778] conn(0x7fcb4c0bdb40 0x7fcb4c0c0030 crc :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:23:49.506 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:23:47.025+0000 7fcb679ec700 1 -- 172.21.15.52:0/3423747181 >> [v2:172.21.15.52:6800/3513075805,v1:172.21.15.52:6801/3513075805] conn(0x7fcb4c0b2910 msgr2=0x7fcb4c0b4e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:23:49.506 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:23:47.025+0000 7fcb679ec700 1 --2- 172.21.15.52:0/3423747181 >> [v2:172.21.15.52:6800/3513075805,v1:172.21.15.52:6801/3513075805] conn(0x7fcb4c0b2910 0x7fcb4c0b4e20 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7fcb5c009990 tx=0x7fcb5c0036e0 comp rx=0 tx=0).stop 2024-05-27T07:23:49.506 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:23:47.025+0000 7fcb679ec700 1 -- 172.21.15.52:0/3423747181 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fcb60108750 msgr2=0x7fcb6019f610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:23:49.506 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:23:47.025+0000 7fcb679ec700 1 --2- 172.21.15.52:0/3423747181 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fcb60108750 0x7fcb6019f610 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7fcb58007ee0 tx=0x7fcb5800d3b0 comp rx=0 tx=0).stop 2024-05-27T07:23:49.506 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:23:47.025+0000 7fcb65f89700 1 -- 172.21.15.52:0/3423747181 reap_dead start 2024-05-27T07:23:49.506 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:23:47.025+0000 7fcb679ec700 1 -- 172.21.15.52:0/3423747181 shutdown_connections 2024-05-27T07:23:49.507 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:23:47.025+0000 7fcb679ec700 1 -- 172.21.15.52:0/3423747181 >> 172.21.15.52:0/3423747181 conn(0x7fcb600fef20 msgr2=0x7fcb6006c3d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:23:49.507 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:23:47.025+0000 7fcb679ec700 1 -- 172.21.15.52:0/3423747181 shutdown_connections 2024-05-27T07:23:49.507 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:23:47.025+0000 7fcb679ec700 1 -- 172.21.15.52:0/3423747181 wait complete. 2024-05-27T07:23:50.124 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:23:50 smithi052 ceph-mon[31955]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-27T07:23:50.351 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:23:50 smithi105 ceph-mon[35977]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-27T07:23:50.351 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:23:50 smithi173 ceph-mon[37000]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-27T07:23:51.200 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:23:51 smithi052 ceph-mon[31955]: pgmap v360: 129 pgs: 129 active+clean; 647 KiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail; 5.6 KiB/s rd, 10 op/s 2024-05-27T07:23:51.230 INFO:teuthology.orchestra.run.smithi052.stdout:257698037810 2024-05-27T07:23:51.231 DEBUG:teuthology.orchestra.run.smithi052:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:38f9d900c83c63799fbdbe61acc9a11b0d3554a6 shell --fsid 73df2636-1bf8-11ef-bc9b-c7b262605968 -- ceph osd last-stat-seq osd.9 2024-05-27T07:23:51.351 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:23:51 smithi173 ceph-mon[37000]: pgmap v360: 129 pgs: 129 active+clean; 647 KiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail; 5.6 KiB/s rd, 10 op/s 2024-05-27T07:23:51.440 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:23:51 smithi105 ceph-mon[35977]: pgmap v360: 129 pgs: 129 active+clean; 647 KiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail; 5.6 KiB/s rd, 10 op/s 2024-05-27T07:23:52.479 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:23:52 smithi052 ceph-mon[31955]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' 2024-05-27T07:23:52.516 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:23:52 smithi052 ceph-mon[31955]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' 2024-05-27T07:23:52.517 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:23:52 smithi052 ceph-mon[31955]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' 2024-05-27T07:23:52.517 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:23:52 smithi052 ceph-mon[31955]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' 2024-05-27T07:23:52.517 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:23:52 smithi052 ceph-mon[31955]: pgmap v361: 129 pgs: 129 active+clean; 647 KiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-05-27T07:23:52.602 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:23:52 smithi173 ceph-mon[37000]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' 2024-05-27T07:23:52.602 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:23:52 smithi173 ceph-mon[37000]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' 2024-05-27T07:23:52.602 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:23:52 smithi173 ceph-mon[37000]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' 2024-05-27T07:23:52.602 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:23:52 smithi173 ceph-mon[37000]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' 2024-05-27T07:23:52.602 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:23:52 smithi173 ceph-mon[37000]: pgmap v361: 129 pgs: 129 active+clean; 647 KiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-05-27T07:23:52.690 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:23:52 smithi105 ceph-mon[35977]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' 2024-05-27T07:23:52.690 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:23:52 smithi105 ceph-mon[35977]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' 2024-05-27T07:23:52.690 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:23:52 smithi105 ceph-mon[35977]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' 2024-05-27T07:23:52.690 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:23:52 smithi105 ceph-mon[35977]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' 2024-05-27T07:23:52.690 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:23:52 smithi105 ceph-mon[35977]: pgmap v361: 129 pgs: 129 active+clean; 647 KiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-05-27T07:23:52.741 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:23:49.155+0000 7f577683a700 1 -- 172.21.15.52:0/4041620903 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f5770109880 msgr2=0x7f57701111e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:23:52.741 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:23:49.155+0000 7f577683a700 1 --2- 172.21.15.52:0/4041620903 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f5770109880 0x7f57701111e0 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7f576800b0d0 tx=0x7f576800b3e0 comp rx=0 tx=0).stop 2024-05-27T07:23:52.741 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:23:49.155+0000 7f577683a700 1 -- 172.21.15.52:0/4041620903 shutdown_connections 2024-05-27T07:23:52.741 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:23:49.155+0000 7f577683a700 1 --2- 172.21.15.52:0/4041620903 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f5770109880 0x7f57701111e0 unknown :-1 s=CLOSED pgs=255 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:23:52.741 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:23:49.155+0000 7f577683a700 1 --2- 172.21.15.52:0/4041620903 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f5770104600 0x7f5770104a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:23:52.742 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:23:49.155+0000 7f577683a700 1 --2- 172.21.15.52:0/4041620903 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f5770108f30 0x7f5770109340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:23:52.742 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:23:49.155+0000 7f577683a700 1 -- 172.21.15.52:0/4041620903 >> 172.21.15.52:0/4041620903 conn(0x7f577007af60 msgr2=0x7f577007b3a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:23:52.742 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:23:49.155+0000 7f577683a700 1 -- 172.21.15.52:0/4041620903 shutdown_connections 2024-05-27T07:23:52.742 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:23:49.155+0000 7f577683a700 1 -- 172.21.15.52:0/4041620903 wait complete. 2024-05-27T07:23:52.742 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:23:49.155+0000 7f577683a700 1 Processor -- start 2024-05-27T07:23:52.742 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:23:49.155+0000 7f577683a700 1 -- start start 2024-05-27T07:23:52.742 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:23:49.156+0000 7f577683a700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f5770104600 0x7f577019ada0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:23:52.742 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:23:49.156+0000 7f577683a700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f5770108f30 0x7f577019b340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:23:52.742 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:23:49.156+0000 7f577683a700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f5770109880 0x7f577019f5d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:23:52.742 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:23:49.156+0000 7f577683a700 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x7f57701144b0 con 0x7f5770104600 2024-05-27T07:23:52.743 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:23:49.156+0000 7f577683a700 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f5770114330 con 0x7f5770108f30 2024-05-27T07:23:52.743 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:23:49.156+0000 7f577683a700 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f5770114630 con 0x7f5770109880 2024-05-27T07:23:52.743 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:23:49.156+0000 7f576ffff700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f5770104600 0x7f577019ada0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:23:52.743 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:23:49.156+0000 7f576ffff700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f5770104600 0x7f577019ada0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.52:40832/0 (socket says 172.21.15.52:40832) 2024-05-27T07:23:52.743 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:23:49.156+0000 7f576ffff700 1 -- 172.21.15.52:0/3069218184 learned_addr learned my addr 172.21.15.52:0/3069218184 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-05-27T07:23:52.743 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:23:49.156+0000 7f5767fff700 1 --2- 172.21.15.52:0/3069218184 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f5770108f30 0x7f577019b340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:23:52.743 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:23:49.156+0000 7f5774dd7700 1 --2- 172.21.15.52:0/3069218184 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f5770109880 0x7f577019f5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:23:52.743 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:23:49.156+0000 7f576ffff700 1 -- 172.21.15.52:0/3069218184 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f5770109880 msgr2=0x7f577019f5d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:23:52.744 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:23:49.156+0000 7f576ffff700 1 --2- 172.21.15.52:0/3069218184 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f5770109880 0x7f577019f5d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:23:52.744 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:23:49.156+0000 7f576ffff700 1 -- 172.21.15.52:0/3069218184 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f5770108f30 msgr2=0x7f577019b340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:23:52.744 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:23:49.156+0000 7f576ffff700 1 --2- 172.21.15.52:0/3069218184 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f5770108f30 0x7f577019b340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:23:52.744 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:23:49.156+0000 7f576ffff700 1 -- 172.21.15.52:0/3069218184 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f577019fc70 con 0x7f5770104600 2024-05-27T07:23:52.744 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:23:49.157+0000 7f576ffff700 1 --2- 172.21.15.52:0/3069218184 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f5770104600 0x7f577019ada0 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7f576000c920 tx=0x7f576000cc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:23:52.744 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:23:49.157+0000 7f576dffb700 1 -- 172.21.15.52:0/3069218184 <== mon.0 v2:172.21.15.52:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5760007b00 con 0x7f5770104600 2024-05-27T07:23:52.744 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:23:49.157+0000 7f576dffb700 1 -- 172.21.15.52:0/3069218184 <== mon.0 v2:172.21.15.52:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f5760004ce0 con 0x7f5770104600 2024-05-27T07:23:52.744 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:23:49.157+0000 7f576dffb700 1 -- 172.21.15.52:0/3069218184 <== mon.0 v2:172.21.15.52:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5760005550 con 0x7f5770104600 2024-05-27T07:23:52.744 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:23:49.157+0000 7f577683a700 1 -- 172.21.15.52:0/3069218184 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f577019ff60 con 0x7f5770104600 2024-05-27T07:23:52.744 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:23:49.157+0000 7f577683a700 1 -- 172.21.15.52:0/3069218184 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f57701a7880 con 0x7f5770104600 2024-05-27T07:23:52.744 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:23:49.158+0000 7f577683a700 1 -- 172.21.15.52:0/3069218184 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f5770101fc0 con 0x7f5770104600 2024-05-27T07:23:52.744 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:23:49.160+0000 7f576dffb700 1 -- 172.21.15.52:0/3069218184 <== mon.0 v2:172.21.15.52:3300/0 4 ==== mgrmap(e 17) ==== 149703+0+0 (secure 0 0 0) 0x7f5760007ca0 con 0x7f5770104600 2024-05-27T07:23:52.744 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:23:49.160+0000 7f576dffb700 1 --2- 172.21.15.52:0/3069218184 >> [v2:172.21.15.52:6800/3513075805,v1:172.21.15.52:6801/3513075805] conn(0x7f575c0b2910 0x7f575c0b4e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:23:52.745 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:23:49.161+0000 7f5767fff700 1 --2- 172.21.15.52:0/3069218184 >> [v2:172.21.15.52:6800/3513075805,v1:172.21.15.52:6801/3513075805] conn(0x7f575c0b2910 0x7f575c0b4e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:23:52.745 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:23:49.161+0000 7f576dffb700 1 -- 172.21.15.52:0/3069218184 <== mon.0 v2:172.21.15.52:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9121+0+0 (secure 0 0 0) 0x7f57600e2810 con 0x7f5770104600 2024-05-27T07:23:52.745 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:23:49.161+0000 7f576dffb700 1 --2- 172.21.15.52:0/3069218184 >> [v2:172.21.15.52:6810/57920794,v1:172.21.15.52:6811/57920794] conn(0x7f575c0bdb40 0x7f575c0c0030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:23:52.745 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:23:49.161+0000 7f576dffb700 1 -- 172.21.15.52:0/3069218184 --> [v2:172.21.15.52:6810/57920794,v1:172.21.15.52:6811/57920794] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f5760012ec0 con 0x7f575c0bdb40 2024-05-27T07:23:52.745 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:23:49.161+0000 7f576dffb700 1 -- 172.21.15.52:0/3069218184 <== mon.0 v2:172.21.15.52:3300/0 6 ==== mon_get_version_reply(handle=1 version=90) ==== 24+0+0 (secure 0 0 0) 0x7f57600e2c60 con 0x7f5770104600 2024-05-27T07:23:52.745 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:23:49.161+0000 7f5767fff700 1 --2- 172.21.15.52:0/3069218184 >> [v2:172.21.15.52:6800/3513075805,v1:172.21.15.52:6801/3513075805] conn(0x7f575c0b2910 0x7f575c0b4e20 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7f5770103c30 tx=0x7f575800b410 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:23:52.745 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:23:49.161+0000 7f5774dd7700 1 --2- 172.21.15.52:0/3069218184 >> [v2:172.21.15.52:6810/57920794,v1:172.21.15.52:6811/57920794] conn(0x7f575c0bdb40 0x7f575c0c0030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:23:52.745 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:23:49.161+0000 7f5774dd7700 1 --2- 172.21.15.52:0/3069218184 >> [v2:172.21.15.52:6810/57920794,v1:172.21.15.52:6811/57920794] conn(0x7f575c0bdb40 0x7f575c0c0030 crc :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:23:52.745 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:23:49.162+0000 7f576dffb700 1 -- 172.21.15.52:0/3069218184 <== osd.3 v2:172.21.15.52:6810/57920794 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f5760012ec0 con 0x7f575c0bdb40 2024-05-27T07:23:52.745 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:23:49.179+0000 7f577683a700 1 -- 172.21.15.52:0/3069218184 --> [v2:172.21.15.52:6810/57920794,v1:172.21.15.52:6811/57920794] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f5770105ab0 con 0x7f575c0bdb40 2024-05-27T07:23:52.745 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:23:49.179+0000 7f576dffb700 1 -- 172.21.15.52:0/3069218184 <== osd.3 v2:172.21.15.52:6810/57920794 2 ==== command_reply(tid 2: 0 ) ==== 8+0+12 (crc 0 0 0) 0x7f5770105ab0 con 0x7f575c0bdb40 2024-05-27T07:23:52.746 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:23:49.180+0000 7f577683a700 1 -- 172.21.15.52:0/3069218184 >> [v2:172.21.15.52:6810/57920794,v1:172.21.15.52:6811/57920794] conn(0x7f575c0bdb40 msgr2=0x7f575c0c0030 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:23:52.746 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:23:49.180+0000 7f577683a700 1 --2- 172.21.15.52:0/3069218184 >> [v2:172.21.15.52:6810/57920794,v1:172.21.15.52:6811/57920794] conn(0x7f575c0bdb40 0x7f575c0c0030 crc :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:23:52.746 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:23:49.180+0000 7f577683a700 1 -- 172.21.15.52:0/3069218184 >> [v2:172.21.15.52:6800/3513075805,v1:172.21.15.52:6801/3513075805] conn(0x7f575c0b2910 msgr2=0x7f575c0b4e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:23:52.746 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:23:49.180+0000 7f577683a700 1 --2- 172.21.15.52:0/3069218184 >> [v2:172.21.15.52:6800/3513075805,v1:172.21.15.52:6801/3513075805] conn(0x7f575c0b2910 0x7f575c0b4e20 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7f5770103c30 tx=0x7f575800b410 comp rx=0 tx=0).stop 2024-05-27T07:23:52.746 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:23:49.180+0000 7f577683a700 1 -- 172.21.15.52:0/3069218184 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f5770104600 msgr2=0x7f577019ada0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:23:52.746 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:23:49.180+0000 7f577683a700 1 --2- 172.21.15.52:0/3069218184 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f5770104600 0x7f577019ada0 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7f576000c920 tx=0x7f576000cc30 comp rx=0 tx=0).stop 2024-05-27T07:23:52.746 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:23:49.180+0000 7f5774dd7700 1 -- 172.21.15.52:0/3069218184 reap_dead start 2024-05-27T07:23:52.746 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:23:49.180+0000 7f577683a700 1 -- 172.21.15.52:0/3069218184 shutdown_connections 2024-05-27T07:23:52.747 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:23:49.180+0000 7f577683a700 1 -- 172.21.15.52:0/3069218184 >> 172.21.15.52:0/3069218184 conn(0x7f577007af60 msgr2=0x7f5770101560 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:23:52.747 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:23:49.180+0000 7f577683a700 1 -- 172.21.15.52:0/3069218184 shutdown_connections 2024-05-27T07:23:52.747 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:23:49.180+0000 7f577683a700 1 -- 172.21.15.52:0/3069218184 wait complete. 2024-05-27T07:23:54.216 INFO:teuthology.orchestra.run.smithi052.stdout:111669149777 2024-05-27T07:23:54.217 DEBUG:teuthology.orchestra.run.smithi052:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:38f9d900c83c63799fbdbe61acc9a11b0d3554a6 shell --fsid 73df2636-1bf8-11ef-bc9b-c7b262605968 -- ceph osd last-stat-seq osd.3 2024-05-27T07:23:54.372 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:23:54.382+0000 7ff190fda700 1 -- 172.21.15.52:0/3440069331 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff18c1031c0 msgr2=0x7ff18c111160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:23:54.372 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:23:54.382+0000 7ff190fda700 1 --2- 172.21.15.52:0/3440069331 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff18c1031c0 0x7ff18c111160 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7ff18400b0a0 tx=0x7ff18400b3b0 comp rx=0 tx=0).stop 2024-05-27T07:23:54.372 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:23:54.382+0000 7ff190fda700 1 -- 172.21.15.52:0/3440069331 shutdown_connections 2024-05-27T07:23:54.372 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:23:54.382+0000 7ff190fda700 1 --2- 172.21.15.52:0/3440069331 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff18c1031c0 0x7ff18c111160 unknown :-1 s=CLOSED pgs=257 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:23:54.372 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:23:54.382+0000 7ff190fda700 1 --2- 172.21.15.52:0/3440069331 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7ff18c1092f0 0x7ff18c109700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:23:54.373 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:23:54.382+0000 7ff190fda700 1 --2- 172.21.15.52:0/3440069331 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7ff18c102870 0x7ff18c102c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:23:54.373 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:23:54.382+0000 7ff190fda700 1 -- 172.21.15.52:0/3440069331 >> 172.21.15.52:0/3440069331 conn(0x7ff18c0fe6e0 msgr2=0x7ff18c100b00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:23:54.373 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:23:54.382+0000 7ff190fda700 1 -- 172.21.15.52:0/3440069331 shutdown_connections 2024-05-27T07:23:54.373 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:23:54.382+0000 7ff190fda700 1 -- 172.21.15.52:0/3440069331 wait complete. 2024-05-27T07:23:54.374 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:23:54.383+0000 7ff190fda700 1 Processor -- start 2024-05-27T07:23:54.375 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:23:54.383+0000 7ff190fda700 1 -- start start 2024-05-27T07:23:54.375 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:23:54.384+0000 7ff190fda700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7ff18c102870 0x7ff18c19af00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:23:54.375 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:23:54.384+0000 7ff190fda700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff18c1031c0 0x7ff18c19b4a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:23:54.375 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:23:54.384+0000 7ff190fda700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7ff18c1092f0 0x7ff18c19f730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:23:54.375 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:23:54.384+0000 7ff190fda700 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x7ff18c114590 con 0x7ff18c1031c0 2024-05-27T07:23:54.375 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:23:54.384+0000 7ff190fda700 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7ff18c114410 con 0x7ff18c102870 2024-05-27T07:23:54.375 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:23:54.384+0000 7ff190fda700 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7ff18c114710 con 0x7ff18c1092f0 2024-05-27T07:23:54.375 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:23:54.384+0000 7ff189d9b700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff18c1031c0 0x7ff18c19b4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:23:54.375 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:23:54.384+0000 7ff189d9b700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff18c1031c0 0x7ff18c19b4a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.52:34846/0 (socket says 172.21.15.52:34846) 2024-05-27T07:23:54.375 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:23:54.384+0000 7ff189d9b700 1 -- 172.21.15.52:0/4057514821 learned_addr learned my addr 172.21.15.52:0/4057514821 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-05-27T07:23:54.375 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:23:54.384+0000 7ff18ad9d700 1 --2- 172.21.15.52:0/4057514821 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7ff18c1092f0 0x7ff18c19f730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:23:54.376 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:23:54.384+0000 7ff18a59c700 1 --2- 172.21.15.52:0/4057514821 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7ff18c102870 0x7ff18c19af00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:23:54.376 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:23:54.384+0000 7ff18ad9d700 1 -- 172.21.15.52:0/4057514821 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7ff18c102870 msgr2=0x7ff18c19af00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:23:54.376 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:23:54.384+0000 7ff18ad9d700 1 --2- 172.21.15.52:0/4057514821 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7ff18c102870 0x7ff18c19af00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:23:54.376 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:23:54.384+0000 7ff18ad9d700 1 -- 172.21.15.52:0/4057514821 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff18c1031c0 msgr2=0x7ff18c19b4a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:23:54.376 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:23:54.384+0000 7ff18ad9d700 1 --2- 172.21.15.52:0/4057514821 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff18c1031c0 0x7ff18c19b4a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:23:54.376 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:23:54.384+0000 7ff18ad9d700 1 -- 172.21.15.52:0/4057514821 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff18c19fdd0 con 0x7ff18c1092f0 2024-05-27T07:23:54.376 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:23:54.384+0000 7ff189d9b700 1 --2- 172.21.15.52:0/4057514821 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff18c1031c0 0x7ff18c19b4a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-27T07:23:54.376 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:23:54.384+0000 7ff18ad9d700 1 --2- 172.21.15.52:0/4057514821 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7ff18c1092f0 0x7ff18c19f730 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7ff18400d010 tx=0x7ff184004d00 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:23:54.376 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:23:54.385+0000 7ff17b7fe700 1 -- 172.21.15.52:0/4057514821 <== mon.1 v2:172.21.15.173:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff184021070 con 0x7ff18c1092f0 2024-05-27T07:23:54.376 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:23:54.385+0000 7ff190fda700 1 -- 172.21.15.52:0/4057514821 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff18c1a0060 con 0x7ff18c1092f0 2024-05-27T07:23:54.376 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:23:54.385+0000 7ff190fda700 1 -- 172.21.15.52:0/4057514821 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff18c103980 con 0x7ff18c1092f0 2024-05-27T07:23:54.376 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:23:54.385+0000 7ff17b7fe700 1 -- 172.21.15.52:0/4057514821 <== mon.1 v2:172.21.15.173:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ff184007e30 con 0x7ff18c1092f0 2024-05-27T07:23:54.376 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:23:54.385+0000 7ff17b7fe700 1 -- 172.21.15.52:0/4057514821 <== mon.1 v2:172.21.15.173:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff1840045a0 con 0x7ff18c1092f0 2024-05-27T07:23:54.377 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:23:54.386+0000 7ff190fda700 1 -- 172.21.15.52:0/4057514821 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7ff16c001030 con 0x7ff18c1092f0 2024-05-27T07:23:54.379 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:23:54.389+0000 7ff17b7fe700 1 -- 172.21.15.52:0/4057514821 <== mon.1 v2:172.21.15.173:3300/0 4 ==== mgrmap(e 17) ==== 149703+0+0 (secure 0 0 0) 0x7ff184004740 con 0x7ff18c1092f0 2024-05-27T07:23:54.379 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:23:54.390+0000 7ff17b7fe700 1 --2- 172.21.15.52:0/4057514821 >> [v2:172.21.15.52:6800/3513075805,v1:172.21.15.52:6801/3513075805] conn(0x7ff1740b2990 0x7ff1740b4ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:23:54.380 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:23:54.390+0000 7ff18a59c700 1 --2- 172.21.15.52:0/4057514821 >> [v2:172.21.15.52:6800/3513075805,v1:172.21.15.52:6801/3513075805] conn(0x7ff1740b2990 0x7ff1740b4ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:23:54.380 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:23:54.390+0000 7ff17b7fe700 1 -- 172.21.15.52:0/4057514821 <== mon.1 v2:172.21.15.173:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9121+0+0 (secure 0 0 0) 0x7ff18400f020 con 0x7ff18c1092f0 2024-05-27T07:23:54.380 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:23:54.390+0000 7ff17b7fe700 1 --2- 172.21.15.52:0/4057514821 >> [v2:172.21.15.173:6800/2251189314,v1:172.21.15.173:6801/2251189314] conn(0x7ff1740bdbc0 0x7ff1740c00b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:23:54.380 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:23:54.390+0000 7ff17b7fe700 1 -- 172.21.15.52:0/4057514821 --> [v2:172.21.15.173:6800/2251189314,v1:172.21.15.173:6801/2251189314] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7ff184022ec0 con 0x7ff1740bdbc0 2024-05-27T07:23:54.380 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:23:54.390+0000 7ff17b7fe700 1 -- 172.21.15.52:0/4057514821 <== mon.1 v2:172.21.15.173:3300/0 6 ==== mon_get_version_reply(handle=1 version=90) ==== 24+0+0 (secure 0 0 0) 0x7ff1840e4150 con 0x7ff18c1092f0 2024-05-27T07:23:54.380 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:23:54.390+0000 7ff18a59c700 1 --2- 172.21.15.52:0/4057514821 >> [v2:172.21.15.52:6800/3513075805,v1:172.21.15.52:6801/3513075805] conn(0x7ff1740b2990 0x7ff1740b4ea0 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7ff18c076270 tx=0x7ff17c009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:23:54.381 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:23:54.390+0000 7ff189d9b700 1 --2- 172.21.15.52:0/4057514821 >> [v2:172.21.15.173:6800/2251189314,v1:172.21.15.173:6801/2251189314] conn(0x7ff1740bdbc0 0x7ff1740c00b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:23:54.381 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:23:54.391+0000 7ff189d9b700 1 --2- 172.21.15.52:0/4057514821 >> [v2:172.21.15.173:6800/2251189314,v1:172.21.15.173:6801/2251189314] conn(0x7ff1740bdbc0 0x7ff1740c00b0 crc :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:23:54.381 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:23:54.392+0000 7ff17b7fe700 1 -- 172.21.15.52:0/4057514821 <== osd.2 v2:172.21.15.173:6800/2251189314 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7ff184022ec0 con 0x7ff1740bdbc0 2024-05-27T07:23:54.399 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:23:54.409+0000 7ff190fda700 1 -- 172.21.15.52:0/4057514821 --> [v2:172.21.15.173:6800/2251189314,v1:172.21.15.173:6801/2251189314] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7ff16c002e60 con 0x7ff1740bdbc0 2024-05-27T07:23:54.400 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:23:54.410+0000 7ff17b7fe700 1 -- 172.21.15.52:0/4057514821 <== osd.2 v2:172.21.15.173:6800/2251189314 2 ==== command_reply(tid 2: 0 ) ==== 8+0+11 (crc 0 0 0) 0x7ff16c002e60 con 0x7ff1740bdbc0 2024-05-27T07:23:54.400 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:23:54.410+0000 7ff190fda700 1 -- 172.21.15.52:0/4057514821 >> [v2:172.21.15.173:6800/2251189314,v1:172.21.15.173:6801/2251189314] conn(0x7ff1740bdbc0 msgr2=0x7ff1740c00b0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:23:54.401 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:23:54.410+0000 7ff190fda700 1 --2- 172.21.15.52:0/4057514821 >> [v2:172.21.15.173:6800/2251189314,v1:172.21.15.173:6801/2251189314] conn(0x7ff1740bdbc0 0x7ff1740c00b0 crc :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:23:54.401 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:23:54.411+0000 7ff190fda700 1 -- 172.21.15.52:0/4057514821 >> [v2:172.21.15.52:6800/3513075805,v1:172.21.15.52:6801/3513075805] conn(0x7ff1740b2990 msgr2=0x7ff1740b4ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:23:54.401 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:23:54.411+0000 7ff190fda700 1 --2- 172.21.15.52:0/4057514821 >> [v2:172.21.15.52:6800/3513075805,v1:172.21.15.52:6801/3513075805] conn(0x7ff1740b2990 0x7ff1740b4ea0 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7ff18c076270 tx=0x7ff17c009260 comp rx=0 tx=0).stop 2024-05-27T07:23:54.401 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:23:54.411+0000 7ff190fda700 1 -- 172.21.15.52:0/4057514821 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7ff18c1092f0 msgr2=0x7ff18c19f730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:23:54.401 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:23:54.411+0000 7ff190fda700 1 --2- 172.21.15.52:0/4057514821 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7ff18c1092f0 0x7ff18c19f730 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7ff18400d010 tx=0x7ff184004d00 comp rx=0 tx=0).stop 2024-05-27T07:23:54.401 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:23:54.411+0000 7ff18ad9d700 1 -- 172.21.15.52:0/4057514821 reap_dead start 2024-05-27T07:23:54.401 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:23:54.411+0000 7ff190fda700 1 -- 172.21.15.52:0/4057514821 shutdown_connections 2024-05-27T07:23:54.401 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:23:54.411+0000 7ff190fda700 1 -- 172.21.15.52:0/4057514821 >> 172.21.15.52:0/4057514821 conn(0x7ff18c0fe6e0 msgr2=0x7ff18c108690 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:23:54.401 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:23:54.411+0000 7ff190fda700 1 -- 172.21.15.52:0/4057514821 shutdown_connections 2024-05-27T07:23:54.401 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:23:54.411+0000 7ff190fda700 1 -- 172.21.15.52:0/4057514821 wait complete. 2024-05-27T07:23:54.817 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:23:54 smithi052 ceph-mon[31955]: pgmap v362: 129 pgs: 129 active+clean; 647 KiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 9 op/s 2024-05-27T07:23:55.102 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:23:54 smithi173 ceph-mon[37000]: pgmap v362: 129 pgs: 129 active+clean; 647 KiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 9 op/s 2024-05-27T07:23:55.190 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:23:54 smithi105 ceph-mon[35977]: pgmap v362: 129 pgs: 129 active+clean; 647 KiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 9 op/s 2024-05-27T07:23:55.235 INFO:teuthology.orchestra.run.smithi052.stdout:77309411415 2024-05-27T07:23:55.235 DEBUG:teuthology.orchestra.run.smithi052:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:38f9d900c83c63799fbdbe61acc9a11b0d3554a6 shell --fsid 73df2636-1bf8-11ef-bc9b-c7b262605968 -- ceph osd last-stat-seq osd.2 2024-05-27T07:23:55.454 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:23:55.464+0000 7fce60977700 1 -- 172.21.15.52:0/3352411063 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fce5c1045d0 msgr2=0x7fce5c1049e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:23:55.455 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:23:55.464+0000 7fce60977700 1 --2- 172.21.15.52:0/3352411063 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fce5c1045d0 0x7fce5c1049e0 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7fce4c0099c0 tx=0x7fce4c009cd0 comp rx=0 tx=0).stop 2024-05-27T07:23:55.455 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:23:55.464+0000 7fce60977700 1 -- 172.21.15.52:0/3352411063 shutdown_connections 2024-05-27T07:23:55.455 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:23:55.464+0000 7fce60977700 1 --2- 172.21.15.52:0/3352411063 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fce5c109850 0x7fce5c1111b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:23:55.455 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:23:55.464+0000 7fce60977700 1 --2- 172.21.15.52:0/3352411063 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fce5c1045d0 0x7fce5c1049e0 unknown :-1 s=CLOSED pgs=260 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:23:55.455 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:23:55.464+0000 7fce60977700 1 --2- 172.21.15.52:0/3352411063 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fce5c108f00 0x7fce5c109310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:23:55.455 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:23:55.464+0000 7fce60977700 1 -- 172.21.15.52:0/3352411063 >> 172.21.15.52:0/3352411063 conn(0x7fce5c0ffe60 msgr2=0x7fce5c1022b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:23:55.455 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:23:55.464+0000 7fce60977700 1 -- 172.21.15.52:0/3352411063 shutdown_connections 2024-05-27T07:23:55.455 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:23:55.464+0000 7fce60977700 1 -- 172.21.15.52:0/3352411063 wait complete. 2024-05-27T07:23:55.456 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:23:55.465+0000 7fce60977700 1 Processor -- start 2024-05-27T07:23:55.456 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:23:55.465+0000 7fce60977700 1 -- start start 2024-05-27T07:23:55.456 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:23:55.466+0000 7fce60977700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fce5c1045d0 0x7fce5c075ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:23:55.456 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:23:55.466+0000 7fce60977700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fce5c108f00 0x7fce5c075ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:23:55.456 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:23:55.466+0000 7fce60977700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fce5c109850 0x7fce5c076530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:23:55.456 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:23:55.466+0000 7fce60977700 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x7fce5c114480 con 0x7fce5c109850 2024-05-27T07:23:55.456 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:23:55.466+0000 7fce60977700 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7fce5c114300 con 0x7fce5c108f00 2024-05-27T07:23:55.456 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:23:55.466+0000 7fce60977700 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7fce5c114600 con 0x7fce5c1045d0 2024-05-27T07:23:55.457 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:23:55.466+0000 7fce5ad9d700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fce5c109850 0x7fce5c076530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:23:55.457 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:23:55.466+0000 7fce5a59c700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fce5c1045d0 0x7fce5c075ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:23:55.457 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:23:55.466+0000 7fce5ad9d700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fce5c109850 0x7fce5c076530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.52:34882/0 (socket says 172.21.15.52:34882) 2024-05-27T07:23:55.457 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:23:55.466+0000 7fce5ad9d700 1 -- 172.21.15.52:0/1122532988 learned_addr learned my addr 172.21.15.52:0/1122532988 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-05-27T07:23:55.457 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:23:55.466+0000 7fce59d9b700 1 --2- 172.21.15.52:0/1122532988 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fce5c108f00 0x7fce5c075ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:23:55.458 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:23:55.467+0000 7fce5ad9d700 1 -- 172.21.15.52:0/1122532988 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fce5c1045d0 msgr2=0x7fce5c075ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:23:55.458 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:23:55.467+0000 7fce5ad9d700 1 --2- 172.21.15.52:0/1122532988 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fce5c1045d0 0x7fce5c075ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:23:55.458 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:23:55.467+0000 7fce5ad9d700 1 -- 172.21.15.52:0/1122532988 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fce5c108f00 msgr2=0x7fce5c075ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:23:55.458 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:23:55.467+0000 7fce5ad9d700 1 --2- 172.21.15.52:0/1122532988 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fce5c108f00 0x7fce5c075ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:23:55.458 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:23:55.467+0000 7fce5ad9d700 1 -- 172.21.15.52:0/1122532988 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fce5c1a9120 con 0x7fce5c109850 2024-05-27T07:23:55.458 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:23:55.467+0000 7fce5ad9d700 1 --2- 172.21.15.52:0/1122532988 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fce5c109850 0x7fce5c076530 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7fce54007ee0 tx=0x7fce5400d3b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:23:55.458 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:23:55.467+0000 7fce4b7fe700 1 -- 172.21.15.52:0/1122532988 <== mon.0 v2:172.21.15.52:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fce54004bb0 con 0x7fce5c109850 2024-05-27T07:23:55.459 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:23:55.467+0000 7fce4b7fe700 1 -- 172.21.15.52:0/1122532988 <== mon.0 v2:172.21.15.52:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fce54004d50 con 0x7fce5c109850 2024-05-27T07:23:55.459 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:23:55.467+0000 7fce60977700 1 -- 172.21.15.52:0/1122532988 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fce5c1a9410 con 0x7fce5c109850 2024-05-27T07:23:55.459 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:23:55.468+0000 7fce4b7fe700 1 -- 172.21.15.52:0/1122532988 <== mon.0 v2:172.21.15.52:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fce54005920 con 0x7fce5c109850 2024-05-27T07:23:55.459 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:23:55.468+0000 7fce60977700 1 -- 172.21.15.52:0/1122532988 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fce5c1a9950 con 0x7fce5c109850 2024-05-27T07:23:55.460 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:23:55.470+0000 7fce60977700 1 -- 172.21.15.52:0/1122532988 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7fce5c063ec0 con 0x7fce5c109850 2024-05-27T07:23:55.461 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:23:55.471+0000 7fce4b7fe700 1 -- 172.21.15.52:0/1122532988 <== mon.0 v2:172.21.15.52:3300/0 4 ==== mgrmap(e 17) ==== 149703+0+0 (secure 0 0 0) 0x7fce5401a020 con 0x7fce5c109850 2024-05-27T07:23:55.463 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:23:55.473+0000 7fce4b7fe700 1 --2- 172.21.15.52:0/1122532988 >> [v2:172.21.15.52:6800/3513075805,v1:172.21.15.52:6801/3513075805] conn(0x7fce440b2930 0x7fce440b4e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:23:55.463 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:23:55.473+0000 7fce5a59c700 1 --2- 172.21.15.52:0/1122532988 >> [v2:172.21.15.52:6800/3513075805,v1:172.21.15.52:6801/3513075805] conn(0x7fce440b2930 0x7fce440b4e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:23:55.463 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:23:55.473+0000 7fce4b7fe700 1 -- 172.21.15.52:0/1122532988 <== mon.0 v2:172.21.15.52:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9121+0+0 (secure 0 0 0) 0x7fce540e1550 con 0x7fce5c109850 2024-05-27T07:23:55.463 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:23:55.473+0000 7fce4b7fe700 1 --2- 172.21.15.52:0/1122532988 >> [v2:172.21.15.52:6802/1626508899,v1:172.21.15.52:6803/1626508899] conn(0x7fce440bdb60 0x7fce440c0050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:23:55.463 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:23:55.473+0000 7fce4b7fe700 1 -- 172.21.15.52:0/1122532988 --> [v2:172.21.15.52:6802/1626508899,v1:172.21.15.52:6803/1626508899] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fce54004d50 con 0x7fce440bdb60 2024-05-27T07:23:55.463 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:23:55.473+0000 7fce4b7fe700 1 -- 172.21.15.52:0/1122532988 <== mon.0 v2:172.21.15.52:3300/0 6 ==== mon_get_version_reply(handle=1 version=90) ==== 24+0+0 (secure 0 0 0) 0x7fce540e8050 con 0x7fce5c109850 2024-05-27T07:23:55.463 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:23:55.473+0000 7fce59d9b700 1 --2- 172.21.15.52:0/1122532988 >> [v2:172.21.15.52:6802/1626508899,v1:172.21.15.52:6803/1626508899] conn(0x7fce440bdb60 0x7fce440c0050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:23:55.464 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:23:55.473+0000 7fce5a59c700 1 --2- 172.21.15.52:0/1122532988 >> [v2:172.21.15.52:6800/3513075805,v1:172.21.15.52:6801/3513075805] conn(0x7fce440b2930 0x7fce440b4e40 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7fce4c009990 tx=0x7fce4c0036e0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:23:55.464 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:23:55.474+0000 7fce59d9b700 1 --2- 172.21.15.52:0/1122532988 >> [v2:172.21.15.52:6802/1626508899,v1:172.21.15.52:6803/1626508899] conn(0x7fce440bdb60 0x7fce440c0050 crc :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:23:55.464 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:23:55.474+0000 7fce4b7fe700 1 -- 172.21.15.52:0/1122532988 <== osd.0 v2:172.21.15.52:6802/1626508899 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7fce54004d50 con 0x7fce440bdb60 2024-05-27T07:23:55.482 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:23:55.492+0000 7fce60977700 1 -- 172.21.15.52:0/1122532988 --> [v2:172.21.15.52:6802/1626508899,v1:172.21.15.52:6803/1626508899] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7fce5c0642f0 con 0x7fce440bdb60 2024-05-27T07:23:55.482 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:23:55.492+0000 7fce4b7fe700 1 -- 172.21.15.52:0/1122532988 <== osd.0 v2:172.21.15.52:6802/1626508899 2 ==== command_reply(tid 2: 0 ) ==== 8+0+11 (crc 0 0 0) 0x7fce5c0642f0 con 0x7fce440bdb60 2024-05-27T07:23:55.483 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:23:55.492+0000 7fce60977700 1 -- 172.21.15.52:0/1122532988 >> [v2:172.21.15.52:6802/1626508899,v1:172.21.15.52:6803/1626508899] conn(0x7fce440bdb60 msgr2=0x7fce440c0050 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:23:55.483 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:23:55.492+0000 7fce60977700 1 --2- 172.21.15.52:0/1122532988 >> [v2:172.21.15.52:6802/1626508899,v1:172.21.15.52:6803/1626508899] conn(0x7fce440bdb60 0x7fce440c0050 crc :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:23:55.483 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:23:55.492+0000 7fce60977700 1 -- 172.21.15.52:0/1122532988 >> [v2:172.21.15.52:6800/3513075805,v1:172.21.15.52:6801/3513075805] conn(0x7fce440b2930 msgr2=0x7fce440b4e40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:23:55.483 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:23:55.492+0000 7fce60977700 1 --2- 172.21.15.52:0/1122532988 >> [v2:172.21.15.52:6800/3513075805,v1:172.21.15.52:6801/3513075805] conn(0x7fce440b2930 0x7fce440b4e40 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7fce4c009990 tx=0x7fce4c0036e0 comp rx=0 tx=0).stop 2024-05-27T07:23:55.483 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:23:55.492+0000 7fce60977700 1 -- 172.21.15.52:0/1122532988 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fce5c109850 msgr2=0x7fce5c076530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:23:55.483 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:23:55.492+0000 7fce60977700 1 --2- 172.21.15.52:0/1122532988 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fce5c109850 0x7fce5c076530 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7fce54007ee0 tx=0x7fce5400d3b0 comp rx=0 tx=0).stop 2024-05-27T07:23:55.483 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:23:55.492+0000 7fce5ad9d700 1 -- 172.21.15.52:0/1122532988 reap_dead start 2024-05-27T07:23:55.483 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:23:55.492+0000 7fce60977700 1 -- 172.21.15.52:0/1122532988 shutdown_connections 2024-05-27T07:23:55.483 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:23:55.492+0000 7fce60977700 1 -- 172.21.15.52:0/1122532988 >> 172.21.15.52:0/1122532988 conn(0x7fce5c0ffe60 msgr2=0x7fce5c102d50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:23:55.483 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:23:55.493+0000 7fce60977700 1 -- 172.21.15.52:0/1122532988 shutdown_connections 2024-05-27T07:23:55.484 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:23:55.493+0000 7fce60977700 1 -- 172.21.15.52:0/1122532988 wait complete. 2024-05-27T07:23:55.652 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:23:54.555+0000 7f167577e700 1 -- 172.21.15.52:0/4265783533 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f1670108750 msgr2=0x7f16701100b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:23:55.652 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:23:54.555+0000 7f167577e700 1 --2- 172.21.15.52:0/4265783533 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f1670108750 0x7f16701100b0 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7f166800b0d0 tx=0x7f166800b3e0 comp rx=0 tx=0).stop 2024-05-27T07:23:55.652 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:23:54.555+0000 7f167577e700 1 -- 172.21.15.52:0/4265783533 shutdown_connections 2024-05-27T07:23:55.652 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:23:54.555+0000 7f167577e700 1 --2- 172.21.15.52:0/4265783533 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f1670108750 0x7f16701100b0 unknown :-1 s=CLOSED pgs=258 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:23:55.652 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:23:54.555+0000 7f167577e700 1 --2- 172.21.15.52:0/4265783533 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f16701034d0 0x7f16701038e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:23:55.652 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:23:54.555+0000 7f167577e700 1 --2- 172.21.15.52:0/4265783533 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f1670107e00 0x7f1670108210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:23:55.652 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:23:54.555+0000 7f167577e700 1 -- 172.21.15.52:0/4265783533 >> 172.21.15.52:0/4265783533 conn(0x7f16700fee40 msgr2=0x7f1670101260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:23:55.652 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:23:54.555+0000 7f167577e700 1 -- 172.21.15.52:0/4265783533 shutdown_connections 2024-05-27T07:23:55.653 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:23:54.555+0000 7f167577e700 1 -- 172.21.15.52:0/4265783533 wait complete. 2024-05-27T07:23:55.653 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:23:54.556+0000 7f167577e700 1 Processor -- start 2024-05-27T07:23:55.653 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:23:54.556+0000 7f167577e700 1 -- start start 2024-05-27T07:23:55.653 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:23:54.556+0000 7f167577e700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f16701034d0 0x7f167019bbe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:23:55.653 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:23:54.556+0000 7f167577e700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f1670107e00 0x7f167019c120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:23:55.653 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:23:54.556+0000 7f167577e700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f1670108750 0x7f167019c680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:23:55.653 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:23:54.556+0000 7f167577e700 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x7f1670113380 con 0x7f16701034d0 2024-05-27T07:23:55.653 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:23:54.556+0000 7f167577e700 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f1670113200 con 0x7f1670107e00 2024-05-27T07:23:55.653 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:23:54.556+0000 7f167577e700 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f1670113500 con 0x7f1670108750 2024-05-27T07:23:55.653 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:23:54.556+0000 7f166effd700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f16701034d0 0x7f167019bbe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:23:55.653 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:23:54.556+0000 7f166effd700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f16701034d0 0x7f167019bbe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.52:34864/0 (socket says 172.21.15.52:34864) 2024-05-27T07:23:55.653 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:23:54.556+0000 7f166effd700 1 -- 172.21.15.52:0/1288310219 learned_addr learned my addr 172.21.15.52:0/1288310219 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-05-27T07:23:55.654 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:23:54.556+0000 7f1667fff700 1 --2- 172.21.15.52:0/1288310219 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f1670107e00 0x7f167019c120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:23:55.654 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:23:54.556+0000 7f166f7fe700 1 --2- 172.21.15.52:0/1288310219 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f1670108750 0x7f167019c680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:23:55.654 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:23:54.556+0000 7f166effd700 1 -- 172.21.15.52:0/1288310219 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f1670108750 msgr2=0x7f167019c680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:23:55.654 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:23:54.556+0000 7f166effd700 1 --2- 172.21.15.52:0/1288310219 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f1670108750 0x7f167019c680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:23:55.654 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:23:54.556+0000 7f166effd700 1 -- 172.21.15.52:0/1288310219 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f1670107e00 msgr2=0x7f167019c120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:23:55.655 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:23:54.556+0000 7f166effd700 1 --2- 172.21.15.52:0/1288310219 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f1670107e00 0x7f167019c120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:23:55.655 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:23:54.556+0000 7f166effd700 1 -- 172.21.15.52:0/1288310219 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f16701a6f30 con 0x7f16701034d0 2024-05-27T07:23:55.655 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:23:54.557+0000 7f166f7fe700 1 --2- 172.21.15.52:0/1288310219 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f1670108750 0x7f167019c680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-27T07:23:55.655 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:23:54.557+0000 7f166effd700 1 --2- 172.21.15.52:0/1288310219 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f16701034d0 0x7f167019bbe0 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7f166000d950 tx=0x7f166000dc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:23:55.655 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:23:54.557+0000 7f166cff9700 1 -- 172.21.15.52:0/1288310219 <== mon.0 v2:172.21.15.52:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1660016070 con 0x7f16701034d0 2024-05-27T07:23:55.655 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:23:54.557+0000 7f167577e700 1 -- 172.21.15.52:0/1288310219 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f16701a7220 con 0x7f16701034d0 2024-05-27T07:23:55.655 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:23:54.557+0000 7f166cff9700 1 -- 172.21.15.52:0/1288310219 <== mon.0 v2:172.21.15.52:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f1660009df0 con 0x7f16701034d0 2024-05-27T07:23:55.655 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:23:54.557+0000 7f166cff9700 1 -- 172.21.15.52:0/1288310219 <== mon.0 v2:172.21.15.52:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f166000f3c0 con 0x7f16701034d0 2024-05-27T07:23:55.655 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:23:54.557+0000 7f167577e700 1 -- 172.21.15.52:0/1288310219 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f16701a7760 con 0x7f16701034d0 2024-05-27T07:23:55.655 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:23:54.559+0000 7f167577e700 1 -- 172.21.15.52:0/1288310219 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f1670063ec0 con 0x7f16701034d0 2024-05-27T07:23:55.655 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:23:54.561+0000 7f166cff9700 1 -- 172.21.15.52:0/1288310219 <== mon.0 v2:172.21.15.52:3300/0 4 ==== mgrmap(e 17) ==== 149703+0+0 (secure 0 0 0) 0x7f1660019070 con 0x7f16701034d0 2024-05-27T07:23:55.655 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:23:54.561+0000 7f166cff9700 1 --2- 172.21.15.52:0/1288310219 >> [v2:172.21.15.52:6800/3513075805,v1:172.21.15.52:6801/3513075805] conn(0x7f165c0b2910 0x7f165c0b4e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:23:55.655 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:23:54.562+0000 7f1667fff700 1 --2- 172.21.15.52:0/1288310219 >> [v2:172.21.15.52:6800/3513075805,v1:172.21.15.52:6801/3513075805] conn(0x7f165c0b2910 0x7f165c0b4e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:23:55.656 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:23:54.562+0000 7f166cff9700 1 -- 172.21.15.52:0/1288310219 <== mon.0 v2:172.21.15.52:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9121+0+0 (secure 0 0 0) 0x7f16600e2340 con 0x7f16701034d0 2024-05-27T07:23:55.656 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:23:54.562+0000 7f166cff9700 1 --2- 172.21.15.52:0/1288310219 >> [v2:172.21.15.173:6808/2989239729,v1:172.21.15.173:6809/2989239729] conn(0x7f165c0bdb30 0x7f165c0c0020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:23:55.656 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:23:54.562+0000 7f166cff9700 1 -- 172.21.15.52:0/1288310219 --> [v2:172.21.15.173:6808/2989239729,v1:172.21.15.173:6809/2989239729] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f165c0c06f0 con 0x7f165c0bdb30 2024-05-27T07:23:55.656 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:23:54.562+0000 7f166cff9700 1 -- 172.21.15.52:0/1288310219 <== mon.0 v2:172.21.15.52:3300/0 6 ==== mon_get_version_reply(handle=1 version=90) ==== 24+0+0 (secure 0 0 0) 0x7f16600e9050 con 0x7f16701034d0 2024-05-27T07:23:55.656 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:23:54.562+0000 7f166f7fe700 1 --2- 172.21.15.52:0/1288310219 >> [v2:172.21.15.173:6808/2989239729,v1:172.21.15.173:6809/2989239729] conn(0x7f165c0bdb30 0x7f165c0c0020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:23:55.656 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:23:54.562+0000 7f1667fff700 1 --2- 172.21.15.52:0/1288310219 >> [v2:172.21.15.52:6800/3513075805,v1:172.21.15.52:6801/3513075805] conn(0x7f165c0b2910 0x7f165c0b4e20 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7f1670106760 tx=0x7f165800b410 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:23:55.656 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:23:54.563+0000 7f166f7fe700 1 --2- 172.21.15.52:0/1288310219 >> [v2:172.21.15.173:6808/2989239729,v1:172.21.15.173:6809/2989239729] conn(0x7f165c0bdb30 0x7f165c0c0020 crc :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:23:55.656 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:23:54.563+0000 7f166cff9700 1 -- 172.21.15.52:0/1288310219 <== osd.5 v2:172.21.15.173:6808/2989239729 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f165c0c06f0 con 0x7f165c0bdb30 2024-05-27T07:23:55.656 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:23:54.580+0000 7f167577e700 1 -- 172.21.15.52:0/1288310219 --> [v2:172.21.15.173:6808/2989239729,v1:172.21.15.173:6809/2989239729] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f1670064340 con 0x7f165c0bdb30 2024-05-27T07:23:55.656 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:23:54.581+0000 7f166cff9700 1 -- 172.21.15.52:0/1288310219 <== osd.5 v2:172.21.15.173:6808/2989239729 2 ==== command_reply(tid 2: 0 ) ==== 8+0+12 (crc 0 0 0) 0x7f1670064340 con 0x7f165c0bdb30 2024-05-27T07:23:55.656 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:23:54.581+0000 7f167577e700 1 -- 172.21.15.52:0/1288310219 >> [v2:172.21.15.173:6808/2989239729,v1:172.21.15.173:6809/2989239729] conn(0x7f165c0bdb30 msgr2=0x7f165c0c0020 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:23:55.656 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:23:54.581+0000 7f167577e700 1 --2- 172.21.15.52:0/1288310219 >> [v2:172.21.15.173:6808/2989239729,v1:172.21.15.173:6809/2989239729] conn(0x7f165c0bdb30 0x7f165c0c0020 crc :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:23:55.657 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:23:54.581+0000 7f167577e700 1 -- 172.21.15.52:0/1288310219 >> [v2:172.21.15.52:6800/3513075805,v1:172.21.15.52:6801/3513075805] conn(0x7f165c0b2910 msgr2=0x7f165c0b4e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:23:55.657 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:23:54.581+0000 7f167577e700 1 --2- 172.21.15.52:0/1288310219 >> [v2:172.21.15.52:6800/3513075805,v1:172.21.15.52:6801/3513075805] conn(0x7f165c0b2910 0x7f165c0b4e20 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7f1670106760 tx=0x7f165800b410 comp rx=0 tx=0).stop 2024-05-27T07:23:55.657 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:23:54.581+0000 7f167577e700 1 -- 172.21.15.52:0/1288310219 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f16701034d0 msgr2=0x7f167019bbe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:23:55.657 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:23:54.581+0000 7f167577e700 1 --2- 172.21.15.52:0/1288310219 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f16701034d0 0x7f167019bbe0 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7f166000d950 tx=0x7f166000dc60 comp rx=0 tx=0).stop 2024-05-27T07:23:55.657 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:23:54.581+0000 7f166f7fe700 1 -- 172.21.15.52:0/1288310219 reap_dead start 2024-05-27T07:23:55.657 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:23:54.581+0000 7f167577e700 1 -- 172.21.15.52:0/1288310219 shutdown_connections 2024-05-27T07:23:55.657 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:23:54.581+0000 7f167577e700 1 -- 172.21.15.52:0/1288310219 >> 172.21.15.52:0/1288310219 conn(0x7f16700fee40 msgr2=0x7f16701074b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:23:55.657 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:23:54.581+0000 7f167577e700 1 -- 172.21.15.52:0/1288310219 shutdown_connections 2024-05-27T07:23:55.657 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:23:54.581+0000 7f167577e700 1 -- 172.21.15.52:0/1288310219 wait complete. 2024-05-27T07:23:56.504 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:23:55.999+0000 7f7b80934700 1 -- 172.21.15.52:0/2281451302 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f7b7c108770 msgr2=0x7f7b7c1100d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:23:56.504 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:23:55.999+0000 7f7b80934700 1 --2- 172.21.15.52:0/2281451302 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f7b7c108770 0x7f7b7c1100d0 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7f7b7400b0d0 tx=0x7f7b7400b3e0 comp rx=0 tx=0).stop 2024-05-27T07:23:56.504 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:23:55.999+0000 7f7b80934700 1 -- 172.21.15.52:0/2281451302 shutdown_connections 2024-05-27T07:23:56.504 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:23:55.999+0000 7f7b80934700 1 --2- 172.21.15.52:0/2281451302 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f7b7c108770 0x7f7b7c1100d0 unknown :-1 s=CLOSED pgs=262 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:23:56.504 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:23:55.999+0000 7f7b80934700 1 --2- 172.21.15.52:0/2281451302 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f7b7c103540 0x7f7b7c103950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:23:56.504 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:23:55.999+0000 7f7b80934700 1 --2- 172.21.15.52:0/2281451302 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f7b7c107e20 0x7f7b7c108230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:23:56.504 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:23:55.999+0000 7f7b80934700 1 -- 172.21.15.52:0/2281451302 >> 172.21.15.52:0/2281451302 conn(0x7f7b7c078df0 msgr2=0x7f7b7c079230 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:23:56.504 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:23:55.999+0000 7f7b80934700 1 -- 172.21.15.52:0/2281451302 shutdown_connections 2024-05-27T07:23:56.504 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:23:55.999+0000 7f7b80934700 1 -- 172.21.15.52:0/2281451302 wait complete. 2024-05-27T07:23:56.504 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:23:56.000+0000 7f7b80934700 1 Processor -- start 2024-05-27T07:23:56.505 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:23:56.000+0000 7f7b80934700 1 -- start start 2024-05-27T07:23:56.505 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:23:56.001+0000 7f7b80934700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f7b7c103540 0x7f7b7c19bce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:23:56.505 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:23:56.001+0000 7f7b80934700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f7b7c107e20 0x7f7b7c19c220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:23:56.505 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:23:56.001+0000 7f7b80934700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f7b7c108770 0x7f7b7c19c780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:23:56.505 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:23:56.001+0000 7f7b80934700 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x7f7b7c1133a0 con 0x7f7b7c108770 2024-05-27T07:23:56.505 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:23:56.001+0000 7f7b80934700 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f7b7c113220 con 0x7f7b7c103540 2024-05-27T07:23:56.505 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:23:56.001+0000 7f7b80934700 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f7b7c113520 con 0x7f7b7c107e20 2024-05-27T07:23:56.505 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:23:56.001+0000 7f7b7ad9d700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f7b7c108770 0x7f7b7c19c780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:23:56.505 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:23:56.001+0000 7f7b7a59c700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f7b7c103540 0x7f7b7c19bce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:23:56.505 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:23:56.001+0000 7f7b7ad9d700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f7b7c108770 0x7f7b7c19c780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.52:34900/0 (socket says 172.21.15.52:34900) 2024-05-27T07:23:56.505 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:23:56.001+0000 7f7b7ad9d700 1 -- 172.21.15.52:0/996605959 learned_addr learned my addr 172.21.15.52:0/996605959 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-05-27T07:23:56.505 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:23:56.001+0000 7f7b79d9b700 1 --2- 172.21.15.52:0/996605959 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f7b7c107e20 0x7f7b7c19c220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:23:56.505 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:23:56.001+0000 7f7b7ad9d700 1 -- 172.21.15.52:0/996605959 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f7b7c107e20 msgr2=0x7f7b7c19c220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:23:56.506 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:23:56.001+0000 7f7b7ad9d700 1 --2- 172.21.15.52:0/996605959 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f7b7c107e20 0x7f7b7c19c220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:23:56.506 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:23:56.001+0000 7f7b7ad9d700 1 -- 172.21.15.52:0/996605959 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f7b7c103540 msgr2=0x7f7b7c19bce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:23:56.506 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:23:56.001+0000 7f7b7ad9d700 1 --2- 172.21.15.52:0/996605959 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f7b7c103540 0x7f7b7c19bce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:23:56.506 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:23:56.001+0000 7f7b7ad9d700 1 -- 172.21.15.52:0/996605959 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7b7c1a6fa0 con 0x7f7b7c108770 2024-05-27T07:23:56.506 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:23:56.001+0000 7f7b7ad9d700 1 --2- 172.21.15.52:0/996605959 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f7b7c108770 0x7f7b7c19c780 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7f7b740095c0 tx=0x7f7b74007a20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:23:56.506 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:23:56.001+0000 7f7b737fe700 1 -- 172.21.15.52:0/996605959 <== mon.0 v2:172.21.15.52:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7b7401f070 con 0x7f7b7c108770 2024-05-27T07:23:56.506 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:23:56.002+0000 7f7b737fe700 1 -- 172.21.15.52:0/996605959 <== mon.0 v2:172.21.15.52:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f7b74007e20 con 0x7f7b7c108770 2024-05-27T07:23:56.506 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:23:56.002+0000 7f7b80934700 1 -- 172.21.15.52:0/996605959 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7b7c1a7230 con 0x7f7b7c108770 2024-05-27T07:23:56.507 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:23:56.002+0000 7f7b737fe700 1 -- 172.21.15.52:0/996605959 <== mon.0 v2:172.21.15.52:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7b7401b4a0 con 0x7f7b7c108770 2024-05-27T07:23:56.507 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:23:56.002+0000 7f7b80934700 1 -- 172.21.15.52:0/996605959 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7b7c1a76e0 con 0x7f7b7c108770 2024-05-27T07:23:56.507 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:23:56.002+0000 7f7b80934700 1 -- 172.21.15.52:0/996605959 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f7b7c063ec0 con 0x7f7b7c108770 2024-05-27T07:23:56.507 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:23:56.004+0000 7f7b737fe700 1 -- 172.21.15.52:0/996605959 <== mon.0 v2:172.21.15.52:3300/0 4 ==== mgrmap(e 17) ==== 149703+0+0 (secure 0 0 0) 0x7f7b74022070 con 0x7f7b7c108770 2024-05-27T07:23:56.507 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:23:56.005+0000 7f7b737fe700 1 --2- 172.21.15.52:0/996605959 >> [v2:172.21.15.52:6800/3513075805,v1:172.21.15.52:6801/3513075805] conn(0x7f7b680b2990 0x7f7b680b4ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:23:56.507 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:23:56.005+0000 7f7b7a59c700 1 --2- 172.21.15.52:0/996605959 >> [v2:172.21.15.52:6800/3513075805,v1:172.21.15.52:6801/3513075805] conn(0x7f7b680b2990 0x7f7b680b4ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:23:56.507 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:23:56.005+0000 7f7b737fe700 1 -- 172.21.15.52:0/996605959 <== mon.0 v2:172.21.15.52:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9121+0+0 (secure 0 0 0) 0x7f7b7400f020 con 0x7f7b7c108770 2024-05-27T07:23:56.507 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:23:56.005+0000 7f7b737fe700 1 --2- 172.21.15.52:0/996605959 >> [v2:172.21.15.173:6824/1816308401,v1:172.21.15.173:6825/1816308401] conn(0x7f7b680bdbc0 0x7f7b680c00b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:23:56.507 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:23:56.005+0000 7f7b737fe700 1 -- 172.21.15.52:0/996605959 --> [v2:172.21.15.173:6824/1816308401,v1:172.21.15.173:6825/1816308401] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f7b680c0780 con 0x7f7b680bdbc0 2024-05-27T07:23:56.507 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:23:56.005+0000 7f7b737fe700 1 -- 172.21.15.52:0/996605959 <== mon.0 v2:172.21.15.52:3300/0 6 ==== mon_get_version_reply(handle=1 version=90) ==== 24+0+0 (secure 0 0 0) 0x7f7b740e3e70 con 0x7f7b7c108770 2024-05-27T07:23:56.507 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:23:56.005+0000 7f7b7a59c700 1 --2- 172.21.15.52:0/996605959 >> [v2:172.21.15.52:6800/3513075805,v1:172.21.15.52:6801/3513075805] conn(0x7f7b680b2990 0x7f7b680b4ea0 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7f7b6c005950 tx=0x7f7b6c00b410 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:23:56.508 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:23:56.006+0000 7f7b79d9b700 1 --2- 172.21.15.52:0/996605959 >> [v2:172.21.15.173:6824/1816308401,v1:172.21.15.173:6825/1816308401] conn(0x7f7b680bdbc0 0x7f7b680c00b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:23:56.508 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:23:56.006+0000 7f7b79d9b700 1 --2- 172.21.15.52:0/996605959 >> [v2:172.21.15.173:6824/1816308401,v1:172.21.15.173:6825/1816308401] conn(0x7f7b680bdbc0 0x7f7b680c00b0 crc :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.11 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:23:56.508 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:23:56.007+0000 7f7b737fe700 1 -- 172.21.15.52:0/996605959 <== osd.11 v2:172.21.15.173:6824/1816308401 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f7b680c0780 con 0x7f7b680bdbc0 2024-05-27T07:23:56.508 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:23:56.024+0000 7f7b80934700 1 -- 172.21.15.52:0/996605959 --> [v2:172.21.15.173:6824/1816308401,v1:172.21.15.173:6825/1816308401] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f7b7c0642c0 con 0x7f7b680bdbc0 2024-05-27T07:23:56.508 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:23:56.025+0000 7f7b737fe700 1 -- 172.21.15.52:0/996605959 <== osd.11 v2:172.21.15.173:6824/1816308401 2 ==== command_reply(tid 2: 0 ) ==== 8+0+12 (crc 0 0 0) 0x7f7b7c0642c0 con 0x7f7b680bdbc0 2024-05-27T07:23:56.508 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:23:56.025+0000 7f7b80934700 1 -- 172.21.15.52:0/996605959 >> [v2:172.21.15.173:6824/1816308401,v1:172.21.15.173:6825/1816308401] conn(0x7f7b680bdbc0 msgr2=0x7f7b680c00b0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:23:56.508 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:23:56.025+0000 7f7b80934700 1 --2- 172.21.15.52:0/996605959 >> [v2:172.21.15.173:6824/1816308401,v1:172.21.15.173:6825/1816308401] conn(0x7f7b680bdbc0 0x7f7b680c00b0 crc :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:23:56.508 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:23:56.025+0000 7f7b80934700 1 -- 172.21.15.52:0/996605959 >> [v2:172.21.15.52:6800/3513075805,v1:172.21.15.52:6801/3513075805] conn(0x7f7b680b2990 msgr2=0x7f7b680b4ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:23:56.508 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:23:56.025+0000 7f7b80934700 1 --2- 172.21.15.52:0/996605959 >> [v2:172.21.15.52:6800/3513075805,v1:172.21.15.52:6801/3513075805] conn(0x7f7b680b2990 0x7f7b680b4ea0 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7f7b6c005950 tx=0x7f7b6c00b410 comp rx=0 tx=0).stop 2024-05-27T07:23:56.508 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:23:56.025+0000 7f7b80934700 1 -- 172.21.15.52:0/996605959 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f7b7c108770 msgr2=0x7f7b7c19c780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:23:56.508 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:23:56.025+0000 7f7b80934700 1 --2- 172.21.15.52:0/996605959 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f7b7c108770 0x7f7b7c19c780 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7f7b740095c0 tx=0x7f7b74007a20 comp rx=0 tx=0).stop 2024-05-27T07:23:56.508 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:23:56.025+0000 7f7b7ad9d700 1 -- 172.21.15.52:0/996605959 reap_dead start 2024-05-27T07:23:56.509 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:23:56.025+0000 7f7b80934700 1 -- 172.21.15.52:0/996605959 shutdown_connections 2024-05-27T07:23:56.509 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:23:56.025+0000 7f7b80934700 1 -- 172.21.15.52:0/996605959 >> 172.21.15.52:0/996605959 conn(0x7f7b7c078df0 msgr2=0x7f7b7c100f40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:23:56.509 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:23:56.025+0000 7f7b80934700 1 -- 172.21.15.52:0/996605959 shutdown_connections 2024-05-27T07:23:56.509 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:23:56.025+0000 7f7b80934700 1 -- 172.21.15.52:0/996605959 wait complete. 2024-05-27T07:23:56.979 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:23:56 smithi052 ceph-mon[31955]: pgmap v363: 129 pgs: 129 active+clean; 647 KiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 11 op/s 2024-05-27T07:23:57.102 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:23:56 smithi173 ceph-mon[37000]: pgmap v363: 129 pgs: 129 active+clean; 647 KiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 11 op/s 2024-05-27T07:23:57.149 INFO:teuthology.orchestra.run.smithi052.stdout:304942678057 2024-05-27T07:23:57.149 DEBUG:teuthology.orchestra.run.smithi052:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:38f9d900c83c63799fbdbe61acc9a11b0d3554a6 shell --fsid 73df2636-1bf8-11ef-bc9b-c7b262605968 -- ceph osd last-stat-seq osd.11 2024-05-27T07:23:57.190 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:23:56 smithi105 ceph-mon[35977]: pgmap v363: 129 pgs: 129 active+clean; 647 KiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 11 op/s 2024-05-27T07:23:57.803 INFO:teuthology.orchestra.run.smithi052.stdout:34359738465 2024-05-27T07:23:57.803 DEBUG:teuthology.orchestra.run.smithi052:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:38f9d900c83c63799fbdbe61acc9a11b0d3554a6 shell --fsid 73df2636-1bf8-11ef-bc9b-c7b262605968 -- ceph osd last-stat-seq osd.0 2024-05-27T07:23:57.985 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:23:57.995+0000 7f5b4a213700 1 -- 172.21.15.52:0/926925474 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f5b441045d0 msgr2=0x7f5b441049e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:23:57.985 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:23:57.995+0000 7f5b4a213700 1 --2- 172.21.15.52:0/926925474 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f5b441045d0 0x7f5b441049e0 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7f5b340099c0 tx=0x7f5b34009cd0 comp rx=0 tx=0).stop 2024-05-27T07:23:57.985 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:23:57.995+0000 7f5b4a213700 1 -- 172.21.15.52:0/926925474 shutdown_connections 2024-05-27T07:23:57.985 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:23:57.995+0000 7f5b4a213700 1 --2- 172.21.15.52:0/926925474 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f5b44109850 0x7f5b441111b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:23:57.985 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:23:57.995+0000 7f5b4a213700 1 --2- 172.21.15.52:0/926925474 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f5b441045d0 0x7f5b441049e0 unknown :-1 s=CLOSED pgs=264 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:23:57.985 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:23:57.995+0000 7f5b4a213700 1 --2- 172.21.15.52:0/926925474 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f5b44108f00 0x7f5b44109310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:23:57.985 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:23:57.995+0000 7f5b4a213700 1 -- 172.21.15.52:0/926925474 >> 172.21.15.52:0/926925474 conn(0x7f5b440ffe60 msgr2=0x7f5b441022b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:23:57.985 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:23:57.995+0000 7f5b4a213700 1 -- 172.21.15.52:0/926925474 shutdown_connections 2024-05-27T07:23:57.985 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:23:57.995+0000 7f5b4a213700 1 -- 172.21.15.52:0/926925474 wait complete. 2024-05-27T07:23:57.987 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:23:57.996+0000 7f5b4a213700 1 Processor -- start 2024-05-27T07:23:57.987 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:23:57.996+0000 7f5b4a213700 1 -- start start 2024-05-27T07:23:57.987 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:23:57.996+0000 7f5b4a213700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f5b441045d0 0x7f5b44075ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:23:57.987 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:23:57.996+0000 7f5b4a213700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f5b44108f00 0x7f5b44075ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:23:57.987 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:23:57.996+0000 7f5b4a213700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f5b44109850 0x7f5b44076530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:23:57.988 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:23:57.996+0000 7f5b4a213700 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x7f5b44114480 con 0x7f5b441045d0 2024-05-27T07:23:57.988 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:23:57.996+0000 7f5b4a213700 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f5b44114300 con 0x7f5b44109850 2024-05-27T07:23:57.988 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:23:57.996+0000 7f5b4a213700 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f5b44114600 con 0x7f5b44108f00 2024-05-27T07:23:57.988 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:23:57.997+0000 7f5b437fe700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f5b441045d0 0x7f5b44075ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:23:57.988 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:23:57.997+0000 7f5b42ffd700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f5b44108f00 0x7f5b44075ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:23:57.988 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:23:57.997+0000 7f5b437fe700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f5b441045d0 0x7f5b44075ab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.52:34922/0 (socket says 172.21.15.52:34922) 2024-05-27T07:23:57.988 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:23:57.997+0000 7f5b437fe700 1 -- 172.21.15.52:0/3275204654 learned_addr learned my addr 172.21.15.52:0/3275204654 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-05-27T07:23:57.988 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:23:57.997+0000 7f5b43fff700 1 --2- 172.21.15.52:0/3275204654 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f5b44109850 0x7f5b44076530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:23:57.988 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:23:57.997+0000 7f5b437fe700 1 -- 172.21.15.52:0/3275204654 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f5b44108f00 msgr2=0x7f5b44075ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:23:57.988 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:23:57.997+0000 7f5b437fe700 1 --2- 172.21.15.52:0/3275204654 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f5b44108f00 0x7f5b44075ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:23:57.988 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:23:57.997+0000 7f5b437fe700 1 -- 172.21.15.52:0/3275204654 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f5b44109850 msgr2=0x7f5b44076530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:23:57.988 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:23:57.997+0000 7f5b437fe700 1 --2- 172.21.15.52:0/3275204654 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f5b44109850 0x7f5b44076530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:23:57.989 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:23:57.997+0000 7f5b437fe700 1 -- 172.21.15.52:0/3275204654 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5b441a9100 con 0x7f5b441045d0 2024-05-27T07:23:57.989 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:23:57.997+0000 7f5b437fe700 1 --2- 172.21.15.52:0/3275204654 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f5b441045d0 0x7f5b44075ab0 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0x7f5b340098e0 tx=0x7f5b340057c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:23:57.989 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:23:57.997+0000 7f5b40ff9700 1 -- 172.21.15.52:0/3275204654 <== mon.0 v2:172.21.15.52:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5b34027070 con 0x7f5b441045d0 2024-05-27T07:23:57.989 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:23:57.998+0000 7f5b4a213700 1 -- 172.21.15.52:0/3275204654 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5b441a9390 con 0x7f5b441045d0 2024-05-27T07:23:57.989 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:23:57.998+0000 7f5b40ff9700 1 -- 172.21.15.52:0/3275204654 <== mon.0 v2:172.21.15.52:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f5b34003f20 con 0x7f5b441045d0 2024-05-27T07:23:57.989 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:23:57.998+0000 7f5b40ff9700 1 -- 172.21.15.52:0/3275204654 <== mon.0 v2:172.21.15.52:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5b34022ca0 con 0x7f5b441045d0 2024-05-27T07:23:57.989 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:23:57.998+0000 7f5b4a213700 1 -- 172.21.15.52:0/3275204654 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5b441a9840 con 0x7f5b441045d0 2024-05-27T07:23:57.989 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:23:57.999+0000 7f5b4a213700 1 -- 172.21.15.52:0/3275204654 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f5b44063ec0 con 0x7f5b441045d0 2024-05-27T07:23:57.991 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:23:58.001+0000 7f5b40ff9700 1 -- 172.21.15.52:0/3275204654 <== mon.0 v2:172.21.15.52:3300/0 4 ==== mgrmap(e 17) ==== 149703+0+0 (secure 0 0 0) 0x7f5b3402a070 con 0x7f5b441045d0 2024-05-27T07:23:57.992 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:23:58.002+0000 7f5b40ff9700 1 --2- 172.21.15.52:0/3275204654 >> [v2:172.21.15.52:6800/3513075805,v1:172.21.15.52:6801/3513075805] conn(0x7f5b2c0b2910 0x7f5b2c0b4e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:23:57.993 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:23:58.003+0000 7f5b40ff9700 1 -- 172.21.15.52:0/3275204654 <== mon.0 v2:172.21.15.52:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9121+0+0 (secure 0 0 0) 0x7f5b340e4210 con 0x7f5b441045d0 2024-05-27T07:23:57.993 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:23:58.003+0000 7f5b42ffd700 1 --2- 172.21.15.52:0/3275204654 >> [v2:172.21.15.52:6800/3513075805,v1:172.21.15.52:6801/3513075805] conn(0x7f5b2c0b2910 0x7f5b2c0b4e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:23:57.993 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:23:58.003+0000 7f5b40ff9700 1 --2- 172.21.15.52:0/3275204654 >> [v2:172.21.15.105:6808/3441134818,v1:172.21.15.105:6809/3441134818] conn(0x7f5b2c0bda10 0x7f5b2c0bff00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:23:57.993 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:23:58.003+0000 7f5b40ff9700 1 -- 172.21.15.52:0/3275204654 --> [v2:172.21.15.105:6808/3441134818,v1:172.21.15.105:6809/3441134818] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f5b3401de50 con 0x7f5b2c0bda10 2024-05-27T07:23:57.993 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:23:58.003+0000 7f5b40ff9700 1 -- 172.21.15.52:0/3275204654 <== mon.0 v2:172.21.15.52:3300/0 6 ==== mon_get_version_reply(handle=1 version=90) ==== 24+0+0 (secure 0 0 0) 0x7f5b340e45c0 con 0x7f5b441045d0 2024-05-27T07:23:57.993 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:23:58.003+0000 7f5b42ffd700 1 --2- 172.21.15.52:0/3275204654 >> [v2:172.21.15.52:6800/3513075805,v1:172.21.15.52:6801/3513075805] conn(0x7f5b2c0b2910 0x7f5b2c0b4e20 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7f5b441035a0 tx=0x7f5b3800a380 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:23:57.993 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:23:58.003+0000 7f5b43fff700 1 --2- 172.21.15.52:0/3275204654 >> [v2:172.21.15.105:6808/3441134818,v1:172.21.15.105:6809/3441134818] conn(0x7f5b2c0bda10 0x7f5b2c0bff00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:23:57.994 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:23:58.004+0000 7f5b43fff700 1 --2- 172.21.15.52:0/3275204654 >> [v2:172.21.15.105:6808/3441134818,v1:172.21.15.105:6809/3441134818] conn(0x7f5b2c0bda10 0x7f5b2c0bff00 crc :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:23:57.994 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:23:58.004+0000 7f5b40ff9700 1 -- 172.21.15.52:0/3275204654 <== osd.4 v2:172.21.15.105:6808/3441134818 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f5b3401de50 con 0x7f5b2c0bda10 2024-05-27T07:23:58.011 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:23:58.021+0000 7f5b4a213700 1 -- 172.21.15.52:0/3275204654 --> [v2:172.21.15.105:6808/3441134818,v1:172.21.15.105:6809/3441134818] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f5b44064340 con 0x7f5b2c0bda10 2024-05-27T07:23:58.011 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:23:58.022+0000 7f5b40ff9700 1 -- 172.21.15.52:0/3275204654 <== osd.4 v2:172.21.15.105:6808/3441134818 2 ==== command_reply(tid 2: 0 ) ==== 8+0+12 (crc 0 0 0) 0x7f5b44064340 con 0x7f5b2c0bda10 2024-05-27T07:23:58.012 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:23:58.022+0000 7f5b4a213700 1 -- 172.21.15.52:0/3275204654 >> [v2:172.21.15.105:6808/3441134818,v1:172.21.15.105:6809/3441134818] conn(0x7f5b2c0bda10 msgr2=0x7f5b2c0bff00 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:23:58.012 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:23:58.022+0000 7f5b4a213700 1 --2- 172.21.15.52:0/3275204654 >> [v2:172.21.15.105:6808/3441134818,v1:172.21.15.105:6809/3441134818] conn(0x7f5b2c0bda10 0x7f5b2c0bff00 crc :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:23:58.012 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:23:58.022+0000 7f5b4a213700 1 -- 172.21.15.52:0/3275204654 >> [v2:172.21.15.52:6800/3513075805,v1:172.21.15.52:6801/3513075805] conn(0x7f5b2c0b2910 msgr2=0x7f5b2c0b4e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:23:58.012 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:23:58.022+0000 7f5b4a213700 1 --2- 172.21.15.52:0/3275204654 >> [v2:172.21.15.52:6800/3513075805,v1:172.21.15.52:6801/3513075805] conn(0x7f5b2c0b2910 0x7f5b2c0b4e20 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7f5b441035a0 tx=0x7f5b3800a380 comp rx=0 tx=0).stop 2024-05-27T07:23:58.013 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:23:58.022+0000 7f5b4a213700 1 -- 172.21.15.52:0/3275204654 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f5b441045d0 msgr2=0x7f5b44075ab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:23:58.013 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:23:58.022+0000 7f5b4a213700 1 --2- 172.21.15.52:0/3275204654 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f5b441045d0 0x7f5b44075ab0 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0x7f5b340098e0 tx=0x7f5b340057c0 comp rx=0 tx=0).stop 2024-05-27T07:23:58.013 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:23:58.022+0000 7f5b43fff700 1 -- 172.21.15.52:0/3275204654 reap_dead start 2024-05-27T07:23:58.013 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:23:58.022+0000 7f5b4a213700 1 -- 172.21.15.52:0/3275204654 shutdown_connections 2024-05-27T07:23:58.013 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:23:58.022+0000 7f5b4a213700 1 -- 172.21.15.52:0/3275204654 >> 172.21.15.52:0/3275204654 conn(0x7f5b440ffe60 msgr2=0x7f5b44100650 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:23:58.013 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:23:58.022+0000 7f5b4a213700 1 -- 172.21.15.52:0/3275204654 shutdown_connections 2024-05-27T07:23:58.013 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:23:58.022+0000 7f5b4a213700 1 -- 172.21.15.52:0/3275204654 wait complete. 2024-05-27T07:23:58.154 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:23:58.162+0000 7f76a1bc7700 1 -- 172.21.15.52:0/1959029946 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f769c1034d0 msgr2=0x7f769c1038e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:23:58.154 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:23:58.162+0000 7f76a1bc7700 1 --2- 172.21.15.52:0/1959029946 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f769c1034d0 0x7f769c1038e0 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x7f768c0099c0 tx=0x7f768c009cd0 comp rx=0 tx=0).stop 2024-05-27T07:23:58.154 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:23:58.162+0000 7f76a1bc7700 1 -- 172.21.15.52:0/1959029946 shutdown_connections 2024-05-27T07:23:58.154 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:23:58.162+0000 7f76a1bc7700 1 --2- 172.21.15.52:0/1959029946 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f769c108750 0x7f769c1100b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:23:58.155 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:23:58.162+0000 7f76a1bc7700 1 --2- 172.21.15.52:0/1959029946 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f769c1034d0 0x7f769c1038e0 unknown :-1 s=CLOSED pgs=266 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:23:58.155 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:23:58.162+0000 7f76a1bc7700 1 --2- 172.21.15.52:0/1959029946 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f769c107e00 0x7f769c108210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:23:58.155 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:23:58.162+0000 7f76a1bc7700 1 -- 172.21.15.52:0/1959029946 >> 172.21.15.52:0/1959029946 conn(0x7f769c0fef20 msgr2=0x7f769c101340 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:23:58.155 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:23:58.162+0000 7f76a1bc7700 1 -- 172.21.15.52:0/1959029946 shutdown_connections 2024-05-27T07:23:58.155 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:23:58.162+0000 7f76a1bc7700 1 -- 172.21.15.52:0/1959029946 wait complete. 2024-05-27T07:23:58.155 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:23:58.163+0000 7f76a1bc7700 1 Processor -- start 2024-05-27T07:23:58.155 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:23:58.163+0000 7f76a1bc7700 1 -- start start 2024-05-27T07:23:58.155 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:23:58.164+0000 7f76a1bc7700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f769c1034d0 0x7f769c113b40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:23:58.155 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:23:58.164+0000 7f76a1bc7700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f769c107e00 0x7f769c114080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:23:58.155 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:23:58.164+0000 7f76a1bc7700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f769c108750 0x7f769c1145c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:23:58.155 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:23:58.164+0000 7f76a1bc7700 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x7f769c113380 con 0x7f769c108750 2024-05-27T07:23:58.155 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:23:58.164+0000 7f76a1bc7700 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f769c113200 con 0x7f769c1034d0 2024-05-27T07:23:58.156 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:23:58.164+0000 7f76a1bc7700 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f769c113500 con 0x7f769c107e00 2024-05-27T07:23:58.156 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:23:58.164+0000 7f769bfff700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f769c108750 0x7f769c1145c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:23:58.156 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:23:58.164+0000 7f769bfff700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f769c108750 0x7f769c1145c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.52:34928/0 (socket says 172.21.15.52:34928) 2024-05-27T07:23:58.156 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:23:58.164+0000 7f769bfff700 1 -- 172.21.15.52:0/2058522077 learned_addr learned my addr 172.21.15.52:0/2058522077 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-05-27T07:23:58.156 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:23:58.164+0000 7f769b7fe700 1 --2- 172.21.15.52:0/2058522077 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f769c1034d0 0x7f769c113b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:23:58.156 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:23:58.164+0000 7f769affd700 1 --2- 172.21.15.52:0/2058522077 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f769c107e00 0x7f769c114080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:23:58.156 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:23:58.164+0000 7f769bfff700 1 -- 172.21.15.52:0/2058522077 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f769c107e00 msgr2=0x7f769c114080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:23:58.156 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:23:58.164+0000 7f769bfff700 1 --2- 172.21.15.52:0/2058522077 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f769c107e00 0x7f769c114080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:23:58.156 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:23:58.164+0000 7f769bfff700 1 -- 172.21.15.52:0/2058522077 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f769c1034d0 msgr2=0x7f769c113b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:23:58.156 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:23:58.164+0000 7f769bfff700 1 --2- 172.21.15.52:0/2058522077 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f769c1034d0 0x7f769c113b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:23:58.157 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:23:58.164+0000 7f769bfff700 1 -- 172.21.15.52:0/2058522077 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f769c1a9160 con 0x7f769c108750 2024-05-27T07:23:58.157 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:23:58.165+0000 7f769bfff700 1 --2- 172.21.15.52:0/2058522077 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f769c108750 0x7f769c1145c0 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 crypto rx=0x7f7694007ee0 tx=0x7f769400d3b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:23:58.157 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:23:58.165+0000 7f7698ff9700 1 -- 172.21.15.52:0/2058522077 <== mon.0 v2:172.21.15.52:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7694004bb0 con 0x7f769c108750 2024-05-27T07:23:58.158 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:23:58.165+0000 7f76a1bc7700 1 -- 172.21.15.52:0/2058522077 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f769c1a9450 con 0x7f769c108750 2024-05-27T07:23:58.158 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:23:58.165+0000 7f7698ff9700 1 -- 172.21.15.52:0/2058522077 <== mon.0 v2:172.21.15.52:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f7694004d50 con 0x7f769c108750 2024-05-27T07:23:58.158 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:23:58.165+0000 7f7698ff9700 1 -- 172.21.15.52:0/2058522077 <== mon.0 v2:172.21.15.52:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7694005920 con 0x7f769c108750 2024-05-27T07:23:58.158 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:23:58.165+0000 7f76a1bc7700 1 -- 172.21.15.52:0/2058522077 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f769c1a9940 con 0x7f769c108750 2024-05-27T07:23:58.158 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:23:58.166+0000 7f76a1bc7700 1 -- 172.21.15.52:0/2058522077 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f769c107880 con 0x7f769c108750 2024-05-27T07:23:58.158 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:23:58.167+0000 7f7698ff9700 1 -- 172.21.15.52:0/2058522077 <== mon.0 v2:172.21.15.52:3300/0 4 ==== mgrmap(e 17) ==== 149703+0+0 (secure 0 0 0) 0x7f769401a020 con 0x7f769c108750 2024-05-27T07:23:58.159 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:23:58.168+0000 7f7698ff9700 1 --2- 172.21.15.52:0/2058522077 >> [v2:172.21.15.52:6800/3513075805,v1:172.21.15.52:6801/3513075805] conn(0x7f76880b2910 0x7f76880b4e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:23:58.159 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:23:58.168+0000 7f769b7fe700 1 --2- 172.21.15.52:0/2058522077 >> [v2:172.21.15.52:6800/3513075805,v1:172.21.15.52:6801/3513075805] conn(0x7f76880b2910 0x7f76880b4e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:23:58.159 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:23:58.168+0000 7f7698ff9700 1 -- 172.21.15.52:0/2058522077 <== mon.0 v2:172.21.15.52:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9121+0+0 (secure 0 0 0) 0x7f76940e27a0 con 0x7f769c108750 2024-05-27T07:23:58.159 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:23:58.169+0000 7f7698ff9700 1 --2- 172.21.15.52:0/2058522077 >> [v2:172.21.15.105:6800/2900093195,v1:172.21.15.105:6801/2900093195] conn(0x7f76880bdb40 0x7f76880c0030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:23:58.159 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:23:58.169+0000 7f7698ff9700 1 -- 172.21.15.52:0/2058522077 --> [v2:172.21.15.105:6800/2900093195,v1:172.21.15.105:6801/2900093195] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f7694019e70 con 0x7f76880bdb40 2024-05-27T07:23:58.159 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:23:58.169+0000 7f7698ff9700 1 -- 172.21.15.52:0/2058522077 <== mon.0 v2:172.21.15.52:3300/0 6 ==== mon_get_version_reply(handle=1 version=90) ==== 24+0+0 (secure 0 0 0) 0x7f76940e2bf0 con 0x7f769c108750 2024-05-27T07:23:58.159 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:23:58.169+0000 7f769b7fe700 1 --2- 172.21.15.52:0/2058522077 >> [v2:172.21.15.52:6800/3513075805,v1:172.21.15.52:6801/3513075805] conn(0x7f76880b2910 0x7f76880b4e20 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7f769c114d80 tx=0x7f768c01a040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:23:58.160 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:23:58.169+0000 7f769affd700 1 --2- 172.21.15.52:0/2058522077 >> [v2:172.21.15.105:6800/2900093195,v1:172.21.15.105:6801/2900093195] conn(0x7f76880bdb40 0x7f76880c0030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:23:58.160 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:23:58.169+0000 7f769affd700 1 --2- 172.21.15.52:0/2058522077 >> [v2:172.21.15.105:6800/2900093195,v1:172.21.15.105:6801/2900093195] conn(0x7f76880bdb40 0x7f76880c0030 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:23:58.160 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:23:58.170+0000 7f7698ff9700 1 -- 172.21.15.52:0/2058522077 <== osd.1 v2:172.21.15.105:6800/2900093195 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f7694019e70 con 0x7f76880bdb40 2024-05-27T07:23:58.177 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:23:58.187+0000 7f76a1bc7700 1 -- 172.21.15.52:0/2058522077 --> [v2:172.21.15.105:6800/2900093195,v1:172.21.15.105:6801/2900093195] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f769c104980 con 0x7f76880bdb40 2024-05-27T07:23:58.178 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:23:58.188+0000 7f7698ff9700 1 -- 172.21.15.52:0/2058522077 <== osd.1 v2:172.21.15.105:6800/2900093195 2 ==== command_reply(tid 2: 0 ) ==== 8+0+11 (crc 0 0 0) 0x7f769c104980 con 0x7f76880bdb40 2024-05-27T07:23:58.179 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:23:58.189+0000 7f76a1bc7700 1 -- 172.21.15.52:0/2058522077 >> [v2:172.21.15.105:6800/2900093195,v1:172.21.15.105:6801/2900093195] conn(0x7f76880bdb40 msgr2=0x7f76880c0030 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:23:58.179 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:23:58.189+0000 7f76a1bc7700 1 --2- 172.21.15.52:0/2058522077 >> [v2:172.21.15.105:6800/2900093195,v1:172.21.15.105:6801/2900093195] conn(0x7f76880bdb40 0x7f76880c0030 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:23:58.179 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:23:58.189+0000 7f76a1bc7700 1 -- 172.21.15.52:0/2058522077 >> [v2:172.21.15.52:6800/3513075805,v1:172.21.15.52:6801/3513075805] conn(0x7f76880b2910 msgr2=0x7f76880b4e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:23:58.179 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:23:58.189+0000 7f76a1bc7700 1 --2- 172.21.15.52:0/2058522077 >> [v2:172.21.15.52:6800/3513075805,v1:172.21.15.52:6801/3513075805] conn(0x7f76880b2910 0x7f76880b4e20 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7f769c114d80 tx=0x7f768c01a040 comp rx=0 tx=0).stop 2024-05-27T07:23:58.179 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:23:58.189+0000 7f76a1bc7700 1 -- 172.21.15.52:0/2058522077 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f769c108750 msgr2=0x7f769c1145c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:23:58.179 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:23:58.189+0000 7f76a1bc7700 1 --2- 172.21.15.52:0/2058522077 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f769c108750 0x7f769c1145c0 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 crypto rx=0x7f7694007ee0 tx=0x7f769400d3b0 comp rx=0 tx=0).stop 2024-05-27T07:23:58.179 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:23:58.189+0000 7f769bfff700 1 -- 172.21.15.52:0/2058522077 reap_dead start 2024-05-27T07:23:58.179 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:23:58.189+0000 7f76a1bc7700 1 -- 172.21.15.52:0/2058522077 shutdown_connections 2024-05-27T07:23:58.179 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:23:58.189+0000 7f76a1bc7700 1 -- 172.21.15.52:0/2058522077 >> 172.21.15.52:0/2058522077 conn(0x7f769c0fef20 msgr2=0x7f769c101340 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:23:58.180 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:23:58.189+0000 7f76a1bc7700 1 -- 172.21.15.52:0/2058522077 shutdown_connections 2024-05-27T07:23:58.180 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:23:58.189+0000 7f76a1bc7700 1 -- 172.21.15.52:0/2058522077 wait complete. 2024-05-27T07:23:58.979 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:23:58 smithi052 ceph-mon[31955]: pgmap v364: 129 pgs: 129 active+clean; 647 KiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-05-27T07:23:59.095 INFO:teuthology.orchestra.run.smithi052.stdout:55834574941 2024-05-27T07:23:59.095 DEBUG:teuthology.orchestra.run.smithi052:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:38f9d900c83c63799fbdbe61acc9a11b0d3554a6 shell --fsid 73df2636-1bf8-11ef-bc9b-c7b262605968 -- ceph osd last-stat-seq osd.1 2024-05-27T07:23:59.101 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:23:58 smithi173 ceph-mon[37000]: pgmap v364: 129 pgs: 129 active+clean; 647 KiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-05-27T07:23:59.190 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:23:58 smithi105 ceph-mon[35977]: pgmap v364: 129 pgs: 129 active+clean; 647 KiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-05-27T07:23:59.298 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:23:59.306+0000 7fdc17243700 1 -- 172.21.15.52:0/2624942419 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fdc10108730 msgr2=0x7fdc10110090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:23:59.299 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:23:59.306+0000 7fdc17243700 1 --2- 172.21.15.52:0/2624942419 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fdc10108730 0x7fdc10110090 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 crypto rx=0x7fdc0800b0d0 tx=0x7fdc0800b3e0 comp rx=0 tx=0).stop 2024-05-27T07:23:59.299 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:23:59.306+0000 7fdc17243700 1 -- 172.21.15.52:0/2624942419 shutdown_connections 2024-05-27T07:23:59.299 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:23:59.306+0000 7fdc17243700 1 --2- 172.21.15.52:0/2624942419 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fdc10108730 0x7fdc10110090 unknown :-1 s=CLOSED pgs=268 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:23:59.299 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:23:59.306+0000 7fdc17243700 1 --2- 172.21.15.52:0/2624942419 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fdc101034b0 0x7fdc101038c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:23:59.299 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:23:59.306+0000 7fdc17243700 1 --2- 172.21.15.52:0/2624942419 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fdc10107de0 0x7fdc101081f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:23:59.299 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:23:59.306+0000 7fdc17243700 1 -- 172.21.15.52:0/2624942419 >> 172.21.15.52:0/2624942419 conn(0x7fdc100fedb0 msgr2=0x7fdc101011d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:23:59.299 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:23:59.306+0000 7fdc17243700 1 -- 172.21.15.52:0/2624942419 shutdown_connections 2024-05-27T07:23:59.299 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:23:59.306+0000 7fdc17243700 1 -- 172.21.15.52:0/2624942419 wait complete. 2024-05-27T07:23:59.299 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:23:59.307+0000 7fdc17243700 1 Processor -- start 2024-05-27T07:23:59.300 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:23:59.307+0000 7fdc17243700 1 -- start start 2024-05-27T07:23:59.300 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:23:59.308+0000 7fdc17243700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fdc101034b0 0x7fdc1019ae60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:23:59.300 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:23:59.308+0000 7fdc17243700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fdc10107de0 0x7fdc1019b400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:23:59.300 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:23:59.308+0000 7fdc17243700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fdc10108730 0x7fdc1019f690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:23:59.300 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:23:59.308+0000 7fdc17243700 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x7fdc101144f0 con 0x7fdc10108730 2024-05-27T07:23:59.300 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:23:59.308+0000 7fdc17243700 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7fdc10114370 con 0x7fdc101034b0 2024-05-27T07:23:59.300 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:23:59.308+0000 7fdc17243700 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7fdc10114670 con 0x7fdc10107de0 2024-05-27T07:23:59.300 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:23:59.308+0000 7fdc14fdf700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fdc101034b0 0x7fdc1019ae60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:23:59.300 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:23:59.308+0000 7fdc157e0700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fdc10108730 0x7fdc1019f690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:23:59.300 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:23:59.308+0000 7fdc14fdf700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fdc101034b0 0x7fdc1019ae60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.52:39706/0 (socket says 172.21.15.52:39706) 2024-05-27T07:23:59.300 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:23:59.308+0000 7fdc157e0700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fdc10108730 0x7fdc1019f690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.52:34936/0 (socket says 172.21.15.52:34936) 2024-05-27T07:23:59.300 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:23:59.308+0000 7fdc14fdf700 1 -- 172.21.15.52:0/2672785650 learned_addr learned my addr 172.21.15.52:0/2672785650 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-05-27T07:23:59.300 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:23:59.308+0000 7fdc0ffff700 1 --2- 172.21.15.52:0/2672785650 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fdc10107de0 0x7fdc1019b400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:23:59.301 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:23:59.308+0000 7fdc14fdf700 1 -- 172.21.15.52:0/2672785650 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fdc10107de0 msgr2=0x7fdc1019b400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:23:59.301 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:23:59.308+0000 7fdc14fdf700 1 --2- 172.21.15.52:0/2672785650 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fdc10107de0 0x7fdc1019b400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:23:59.301 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:23:59.308+0000 7fdc14fdf700 1 -- 172.21.15.52:0/2672785650 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fdc10108730 msgr2=0x7fdc1019f690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:23:59.301 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:23:59.308+0000 7fdc14fdf700 1 --2- 172.21.15.52:0/2672785650 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fdc10108730 0x7fdc1019f690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:23:59.301 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:23:59.308+0000 7fdc14fdf700 1 -- 172.21.15.52:0/2672785650 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdc1019fd30 con 0x7fdc101034b0 2024-05-27T07:23:59.301 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:23:59.308+0000 7fdc14fdf700 1 --2- 172.21.15.52:0/2672785650 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fdc101034b0 0x7fdc1019ae60 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7fdc0400d950 tx=0x7fdc0400dc60 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:23:59.301 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:23:59.309+0000 7fdc0dffb700 1 -- 172.21.15.52:0/2672785650 <== mon.2 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdc04016070 con 0x7fdc101034b0 2024-05-27T07:23:59.301 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:23:59.309+0000 7fdc17243700 1 -- 172.21.15.52:0/2672785650 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fdc101a0020 con 0x7fdc101034b0 2024-05-27T07:23:59.301 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:23:59.309+0000 7fdc0dffb700 1 -- 172.21.15.52:0/2672785650 <== mon.2 v2:172.21.15.105:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fdc0400fd40 con 0x7fdc101034b0 2024-05-27T07:23:59.301 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:23:59.309+0000 7fdc0dffb700 1 -- 172.21.15.52:0/2672785650 <== mon.2 v2:172.21.15.105:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdc0400e3b0 con 0x7fdc101034b0 2024-05-27T07:23:59.301 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:23:59.309+0000 7fdc17243700 1 -- 172.21.15.52:0/2672785650 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fdc101a7940 con 0x7fdc101034b0 2024-05-27T07:23:59.302 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:23:59.310+0000 7fdc17243700 1 -- 172.21.15.52:0/2672785650 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7fdc10063ec0 con 0x7fdc101034b0 2024-05-27T07:23:59.302 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:23:59.312+0000 7fdc0dffb700 1 -- 172.21.15.52:0/2672785650 <== mon.2 v2:172.21.15.105:3300/0 4 ==== mgrmap(e 17) ==== 149703+0+0 (secure 0 0 0) 0x7fdc04019070 con 0x7fdc101034b0 2024-05-27T07:23:59.303 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:23:59.313+0000 7fdc0dffb700 1 --2- 172.21.15.52:0/2672785650 >> [v2:172.21.15.52:6800/3513075805,v1:172.21.15.52:6801/3513075805] conn(0x7fdc000b2910 0x7fdc000b4e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:23:59.303 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:23:59.313+0000 7fdc0ffff700 1 --2- 172.21.15.52:0/2672785650 >> [v2:172.21.15.52:6800/3513075805,v1:172.21.15.52:6801/3513075805] conn(0x7fdc000b2910 0x7fdc000b4e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:23:59.303 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:23:59.313+0000 7fdc0dffb700 1 -- 172.21.15.52:0/2672785650 <== mon.2 v2:172.21.15.105:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9121+0+0 (secure 0 0 0) 0x7fdc040e2630 con 0x7fdc101034b0 2024-05-27T07:23:59.304 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:23:59.313+0000 7fdc0dffb700 1 --2- 172.21.15.52:0/2672785650 >> [v2:172.21.15.173:6816/4131399852,v1:172.21.15.173:6817/4131399852] conn(0x7fdc000bdb40 0x7fdc000c0030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:23:59.304 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:23:59.313+0000 7fdc0dffb700 1 -- 172.21.15.52:0/2672785650 --> [v2:172.21.15.173:6816/4131399852,v1:172.21.15.173:6817/4131399852] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fdc000c0700 con 0x7fdc000bdb40 2024-05-27T07:23:59.304 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:23:59.313+0000 7fdc0dffb700 1 -- 172.21.15.52:0/2672785650 <== mon.2 v2:172.21.15.105:3300/0 6 ==== mon_get_version_reply(handle=1 version=90) ==== 24+0+0 (secure 0 0 0) 0x7fdc040e2a80 con 0x7fdc101034b0 2024-05-27T07:23:59.304 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:23:59.313+0000 7fdc0ffff700 1 --2- 172.21.15.52:0/2672785650 >> [v2:172.21.15.52:6800/3513075805,v1:172.21.15.52:6801/3513075805] conn(0x7fdc000b2910 0x7fdc000b4e20 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7fdc10102720 tx=0x7fdbfc009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:23:59.304 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:23:59.313+0000 7fdc157e0700 1 --2- 172.21.15.52:0/2672785650 >> [v2:172.21.15.173:6816/4131399852,v1:172.21.15.173:6817/4131399852] conn(0x7fdc000bdb40 0x7fdc000c0030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:23:59.304 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:23:59.314+0000 7fdc157e0700 1 --2- 172.21.15.52:0/2672785650 >> [v2:172.21.15.173:6816/4131399852,v1:172.21.15.173:6817/4131399852] conn(0x7fdc000bdb40 0x7fdc000c0030 crc :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.8 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:23:59.305 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:23:59.315+0000 7fdc0dffb700 1 -- 172.21.15.52:0/2672785650 <== osd.8 v2:172.21.15.173:6816/4131399852 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7fdc000c0700 con 0x7fdc000bdb40 2024-05-27T07:23:59.322 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:23:59.332+0000 7fdc17243700 1 -- 172.21.15.52:0/2672785650 --> [v2:172.21.15.173:6816/4131399852,v1:172.21.15.173:6817/4131399852] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7fdc100642f0 con 0x7fdc000bdb40 2024-05-27T07:23:59.323 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:23:59.333+0000 7fdc0dffb700 1 -- 172.21.15.52:0/2672785650 <== osd.8 v2:172.21.15.173:6816/4131399852 2 ==== command_reply(tid 2: 0 ) ==== 8+0+12 (crc 0 0 0) 0x7fdc100642f0 con 0x7fdc000bdb40 2024-05-27T07:23:59.324 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:23:59.333+0000 7fdc17243700 1 -- 172.21.15.52:0/2672785650 >> [v2:172.21.15.173:6816/4131399852,v1:172.21.15.173:6817/4131399852] conn(0x7fdc000bdb40 msgr2=0x7fdc000c0030 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:23:59.324 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:23:59.333+0000 7fdc17243700 1 --2- 172.21.15.52:0/2672785650 >> [v2:172.21.15.173:6816/4131399852,v1:172.21.15.173:6817/4131399852] conn(0x7fdc000bdb40 0x7fdc000c0030 crc :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:23:59.324 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:23:59.333+0000 7fdc17243700 1 -- 172.21.15.52:0/2672785650 >> [v2:172.21.15.52:6800/3513075805,v1:172.21.15.52:6801/3513075805] conn(0x7fdc000b2910 msgr2=0x7fdc000b4e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:23:59.324 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:23:59.333+0000 7fdc17243700 1 --2- 172.21.15.52:0/2672785650 >> [v2:172.21.15.52:6800/3513075805,v1:172.21.15.52:6801/3513075805] conn(0x7fdc000b2910 0x7fdc000b4e20 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7fdc10102720 tx=0x7fdbfc009260 comp rx=0 tx=0).stop 2024-05-27T07:23:59.324 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:23:59.333+0000 7fdc17243700 1 -- 172.21.15.52:0/2672785650 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fdc101034b0 msgr2=0x7fdc1019ae60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:23:59.324 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:23:59.333+0000 7fdc17243700 1 --2- 172.21.15.52:0/2672785650 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fdc101034b0 0x7fdc1019ae60 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7fdc0400d950 tx=0x7fdc0400dc60 comp rx=0 tx=0).stop 2024-05-27T07:23:59.324 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:23:59.333+0000 7fdc157e0700 1 -- 172.21.15.52:0/2672785650 reap_dead start 2024-05-27T07:23:59.324 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:23:59.333+0000 7fdc17243700 1 -- 172.21.15.52:0/2672785650 shutdown_connections 2024-05-27T07:23:59.324 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:23:59.333+0000 7fdc17243700 1 -- 172.21.15.52:0/2672785650 >> 172.21.15.52:0/2672785650 conn(0x7fdc100fedb0 msgr2=0x7fdc10107550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:23:59.324 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:23:59.334+0000 7fdc17243700 1 -- 172.21.15.52:0/2672785650 shutdown_connections 2024-05-27T07:23:59.324 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:23:59.334+0000 7fdc17243700 1 -- 172.21.15.52:0/2672785650 wait complete. 2024-05-27T07:24:00.614 INFO:teuthology.orchestra.run.smithi052.stdout:231928234042 2024-05-27T07:24:00.615 DEBUG:teuthology.orchestra.run.smithi052:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:38f9d900c83c63799fbdbe61acc9a11b0d3554a6 shell --fsid 73df2636-1bf8-11ef-bc9b-c7b262605968 -- ceph osd last-stat-seq osd.8 2024-05-27T07:24:00.974 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:24:00 smithi052 ceph-mon[31955]: pgmap v365: 129 pgs: 129 active+clean; 647 KiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 10 op/s 2024-05-27T07:24:01.102 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:24:00 smithi173 ceph-mon[37000]: pgmap v365: 129 pgs: 129 active+clean; 647 KiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 10 op/s 2024-05-27T07:24:01.190 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:24:00 smithi105 ceph-mon[35977]: pgmap v365: 129 pgs: 129 active+clean; 647 KiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 10 op/s 2024-05-27T07:24:01.464 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:01.474+0000 7fd07adac700 1 -- 172.21.15.52:0/3065240129 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fd074107de0 msgr2=0x7fd0741081f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:24:01.465 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:01.474+0000 7fd07adac700 1 --2- 172.21.15.52:0/3065240129 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fd074107de0 0x7fd0741081f0 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 crypto rx=0x7fd05c0099c0 tx=0x7fd05c009cd0 comp rx=0 tx=0).stop 2024-05-27T07:24:01.465 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:01.474+0000 7fd07adac700 1 -- 172.21.15.52:0/3065240129 shutdown_connections 2024-05-27T07:24:01.465 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:01.474+0000 7fd07adac700 1 --2- 172.21.15.52:0/3065240129 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fd074108730 0x7fd074110090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:24:01.465 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:01.474+0000 7fd07adac700 1 --2- 172.21.15.52:0/3065240129 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fd0741034b0 0x7fd0741038c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:24:01.465 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:01.474+0000 7fd07adac700 1 --2- 172.21.15.52:0/3065240129 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fd074107de0 0x7fd0741081f0 unknown :-1 s=CLOSED pgs=269 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:24:01.465 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:01.474+0000 7fd07adac700 1 -- 172.21.15.52:0/3065240129 >> 172.21.15.52:0/3065240129 conn(0x7fd0740fedb0 msgr2=0x7fd0741011d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:24:01.465 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:01.474+0000 7fd07adac700 1 -- 172.21.15.52:0/3065240129 shutdown_connections 2024-05-27T07:24:01.465 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:01.474+0000 7fd07adac700 1 -- 172.21.15.52:0/3065240129 wait complete. 2024-05-27T07:24:01.467 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:01.475+0000 7fd07adac700 1 Processor -- start 2024-05-27T07:24:01.467 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:01.475+0000 7fd07adac700 1 -- start start 2024-05-27T07:24:01.467 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:01.475+0000 7fd07adac700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fd0741034b0 0x7fd07419ae00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:24:01.467 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:01.475+0000 7fd07adac700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fd074107de0 0x7fd07419b3a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:24:01.468 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:01.475+0000 7fd07adac700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fd074108730 0x7fd07419f630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:24:01.468 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:01.475+0000 7fd07adac700 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x7fd074113360 con 0x7fd074108730 2024-05-27T07:24:01.468 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:01.475+0000 7fd07adac700 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7fd0741131e0 con 0x7fd0741034b0 2024-05-27T07:24:01.468 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:01.475+0000 7fd07adac700 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7fd0741134e0 con 0x7fd074107de0 2024-05-27T07:24:01.468 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:01.476+0000 7fd079349700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fd074108730 0x7fd07419f630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:24:01.468 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:01.476+0000 7fd079349700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fd074108730 0x7fd07419f630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.52:55582/0 (socket says 172.21.15.52:55582) 2024-05-27T07:24:01.468 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:01.476+0000 7fd079349700 1 -- 172.21.15.52:0/4174945255 learned_addr learned my addr 172.21.15.52:0/4174945255 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-05-27T07:24:01.468 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:01.476+0000 7fd073fff700 1 --2- 172.21.15.52:0/4174945255 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fd074107de0 0x7fd07419b3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:24:01.468 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:01.476+0000 7fd078b48700 1 --2- 172.21.15.52:0/4174945255 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fd0741034b0 0x7fd07419ae00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:24:01.468 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:01.476+0000 7fd079349700 1 -- 172.21.15.52:0/4174945255 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fd074107de0 msgr2=0x7fd07419b3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:24:01.468 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:01.476+0000 7fd079349700 1 --2- 172.21.15.52:0/4174945255 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fd074107de0 0x7fd07419b3a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:24:01.468 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:01.476+0000 7fd079349700 1 -- 172.21.15.52:0/4174945255 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fd0741034b0 msgr2=0x7fd07419ae00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:24:01.469 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:01.476+0000 7fd079349700 1 --2- 172.21.15.52:0/4174945255 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fd0741034b0 0x7fd07419ae00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:24:01.469 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:01.476+0000 7fd079349700 1 -- 172.21.15.52:0/4174945255 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd07419fcd0 con 0x7fd074108730 2024-05-27T07:24:01.469 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:01.476+0000 7fd078b48700 1 --2- 172.21.15.52:0/4174945255 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fd0741034b0 0x7fd07419ae00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-27T07:24:01.469 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:01.476+0000 7fd079349700 1 --2- 172.21.15.52:0/4174945255 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fd074108730 0x7fd07419f630 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 crypto rx=0x7fd06c00d980 tx=0x7fd06c00dc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:24:01.469 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:01.476+0000 7fd071ffb700 1 -- 172.21.15.52:0/4174945255 <== mon.0 v2:172.21.15.52:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd06c017070 con 0x7fd074108730 2024-05-27T07:24:01.469 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:01.476+0000 7fd071ffb700 1 -- 172.21.15.52:0/4174945255 <== mon.0 v2:172.21.15.52:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fd06c005020 con 0x7fd074108730 2024-05-27T07:24:01.469 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:01.476+0000 7fd07adac700 1 -- 172.21.15.52:0/4174945255 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd07419ffc0 con 0x7fd074108730 2024-05-27T07:24:01.469 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:01.476+0000 7fd071ffb700 1 -- 172.21.15.52:0/4174945255 <== mon.0 v2:172.21.15.52:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd06c0099b0 con 0x7fd074108730 2024-05-27T07:24:01.469 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:01.477+0000 7fd07adac700 1 -- 172.21.15.52:0/4174945255 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd0741a78e0 con 0x7fd074108730 2024-05-27T07:24:01.469 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:01.477+0000 7fd07adac700 1 -- 172.21.15.52:0/4174945255 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7fd074063f00 con 0x7fd074108730 2024-05-27T07:24:01.470 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:01.479+0000 7fd071ffb700 1 -- 172.21.15.52:0/4174945255 <== mon.0 v2:172.21.15.52:3300/0 4 ==== mgrmap(e 17) ==== 149703+0+0 (secure 0 0 0) 0x7fd06c009b50 con 0x7fd074108730 2024-05-27T07:24:01.470 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:01.480+0000 7fd071ffb700 1 --2- 172.21.15.52:0/4174945255 >> [v2:172.21.15.52:6800/3513075805,v1:172.21.15.52:6801/3513075805] conn(0x7fd0600b2990 0x7fd0600b4ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:24:01.470 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:01.480+0000 7fd078b48700 1 --2- 172.21.15.52:0/4174945255 >> [v2:172.21.15.52:6800/3513075805,v1:172.21.15.52:6801/3513075805] conn(0x7fd0600b2990 0x7fd0600b4ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:24:01.470 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:01.480+0000 7fd071ffb700 1 -- 172.21.15.52:0/4174945255 <== mon.0 v2:172.21.15.52:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9121+0+0 (secure 0 0 0) 0x7fd06c0e2ae0 con 0x7fd074108730 2024-05-27T07:24:01.470 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:01.480+0000 7fd071ffb700 1 --2- 172.21.15.52:0/4174945255 >> [v2:172.21.15.52:6818/4244423065,v1:172.21.15.52:6819/4244423065] conn(0x7fd0600bdbc0 0x7fd0600c00b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:24:01.471 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:01.480+0000 7fd071ffb700 1 -- 172.21.15.52:0/4174945255 --> [v2:172.21.15.52:6818/4244423065,v1:172.21.15.52:6819/4244423065] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fd06c01ae70 con 0x7fd0600bdbc0 2024-05-27T07:24:01.471 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:01.480+0000 7fd071ffb700 1 -- 172.21.15.52:0/4174945255 <== mon.0 v2:172.21.15.52:3300/0 6 ==== mon_get_version_reply(handle=1 version=90) ==== 24+0+0 (secure 0 0 0) 0x7fd06c0e2e90 con 0x7fd074108730 2024-05-27T07:24:01.471 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:01.480+0000 7fd073fff700 1 --2- 172.21.15.52:0/4174945255 >> [v2:172.21.15.52:6818/4244423065,v1:172.21.15.52:6819/4244423065] conn(0x7fd0600bdbc0 0x7fd0600c00b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:24:01.471 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:01.481+0000 7fd078b48700 1 --2- 172.21.15.52:0/4174945255 >> [v2:172.21.15.52:6800/3513075805,v1:172.21.15.52:6801/3513075805] conn(0x7fd0600b2990 0x7fd0600b4ea0 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7fd064009dd0 tx=0x7fd064009040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:24:01.471 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:01.481+0000 7fd073fff700 1 --2- 172.21.15.52:0/4174945255 >> [v2:172.21.15.52:6818/4244423065,v1:172.21.15.52:6819/4244423065] conn(0x7fd0600bdbc0 0x7fd0600c00b0 crc :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:24:01.471 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:01.482+0000 7fd071ffb700 1 -- 172.21.15.52:0/4174945255 <== osd.6 v2:172.21.15.52:6818/4244423065 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7fd06c01ae70 con 0x7fd0600bdbc0 2024-05-27T07:24:01.488 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:01.498+0000 7fd07adac700 1 -- 172.21.15.52:0/4174945255 --> [v2:172.21.15.52:6818/4244423065,v1:172.21.15.52:6819/4244423065] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7fd074064380 con 0x7fd0600bdbc0 2024-05-27T07:24:01.488 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:01.499+0000 7fd071ffb700 1 -- 172.21.15.52:0/4174945255 <== osd.6 v2:172.21.15.52:6818/4244423065 2 ==== command_reply(tid 2: 0 ) ==== 8+0+12 (crc 0 0 0) 0x7fd074064380 con 0x7fd0600bdbc0 2024-05-27T07:24:01.489 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:01.499+0000 7fd07adac700 1 -- 172.21.15.52:0/4174945255 >> [v2:172.21.15.52:6818/4244423065,v1:172.21.15.52:6819/4244423065] conn(0x7fd0600bdbc0 msgr2=0x7fd0600c00b0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:24:01.489 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:01.499+0000 7fd07adac700 1 --2- 172.21.15.52:0/4174945255 >> [v2:172.21.15.52:6818/4244423065,v1:172.21.15.52:6819/4244423065] conn(0x7fd0600bdbc0 0x7fd0600c00b0 crc :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:24:01.489 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:01.499+0000 7fd07adac700 1 -- 172.21.15.52:0/4174945255 >> [v2:172.21.15.52:6800/3513075805,v1:172.21.15.52:6801/3513075805] conn(0x7fd0600b2990 msgr2=0x7fd0600b4ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:24:01.489 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:01.499+0000 7fd07adac700 1 --2- 172.21.15.52:0/4174945255 >> [v2:172.21.15.52:6800/3513075805,v1:172.21.15.52:6801/3513075805] conn(0x7fd0600b2990 0x7fd0600b4ea0 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7fd064009dd0 tx=0x7fd064009040 comp rx=0 tx=0).stop 2024-05-27T07:24:01.489 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:01.499+0000 7fd07adac700 1 -- 172.21.15.52:0/4174945255 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fd074108730 msgr2=0x7fd07419f630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:24:01.490 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:01.499+0000 7fd07adac700 1 --2- 172.21.15.52:0/4174945255 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fd074108730 0x7fd07419f630 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 crypto rx=0x7fd06c00d980 tx=0x7fd06c00dc90 comp rx=0 tx=0).stop 2024-05-27T07:24:01.490 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:01.499+0000 7fd079349700 1 -- 172.21.15.52:0/4174945255 reap_dead start 2024-05-27T07:24:01.490 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:01.499+0000 7fd07adac700 1 -- 172.21.15.52:0/4174945255 shutdown_connections 2024-05-27T07:24:01.490 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:01.499+0000 7fd07adac700 1 -- 172.21.15.52:0/4174945255 >> 172.21.15.52:0/4174945255 conn(0x7fd0740fedb0 msgr2=0x7fd07406b940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:24:01.490 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:01.499+0000 7fd07adac700 1 -- 172.21.15.52:0/4174945255 shutdown_connections 2024-05-27T07:24:01.490 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:01.499+0000 7fd07adac700 1 -- 172.21.15.52:0/4174945255 wait complete. 2024-05-27T07:24:01.680 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:01.690+0000 7f0789c7d700 1 -- 172.21.15.52:0/152990956 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f0784107de0 msgr2=0x7f07841081f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:24:01.681 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:01.690+0000 7f0789c7d700 1 --2- 172.21.15.52:0/152990956 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f0784107de0 0x7f07841081f0 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 crypto rx=0x7f07780099c0 tx=0x7f0778009cd0 comp rx=0 tx=0).stop 2024-05-27T07:24:01.681 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:01.690+0000 7f0789c7d700 1 -- 172.21.15.52:0/152990956 shutdown_connections 2024-05-27T07:24:01.681 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:01.690+0000 7f0789c7d700 1 --2- 172.21.15.52:0/152990956 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f0784108730 0x7f0784110090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:24:01.681 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:01.690+0000 7f0789c7d700 1 --2- 172.21.15.52:0/152990956 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f07841034b0 0x7f07841038c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:24:01.681 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:01.690+0000 7f0789c7d700 1 --2- 172.21.15.52:0/152990956 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f0784107de0 0x7f07841081f0 unknown :-1 s=CLOSED pgs=271 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:24:01.681 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:01.690+0000 7f0789c7d700 1 -- 172.21.15.52:0/152990956 >> 172.21.15.52:0/152990956 conn(0x7f07840fedb0 msgr2=0x7f07841011d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:24:01.681 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:01.691+0000 7f0789c7d700 1 -- 172.21.15.52:0/152990956 shutdown_connections 2024-05-27T07:24:01.681 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:01.691+0000 7f0789c7d700 1 -- 172.21.15.52:0/152990956 wait complete. 2024-05-27T07:24:01.683 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:01.691+0000 7f0789c7d700 1 Processor -- start 2024-05-27T07:24:01.683 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:01.691+0000 7f0789c7d700 1 -- start start 2024-05-27T07:24:01.683 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:01.692+0000 7f0789c7d700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f07841034b0 0x7f078419ad40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:24:01.683 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:01.692+0000 7f0789c7d700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f0784107de0 0x7f078419b2e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:24:01.684 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:01.692+0000 7f0789c7d700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f0784108730 0x7f078419f570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:24:01.684 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:01.692+0000 7f0789c7d700 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x7f0784113340 con 0x7f0784108730 2024-05-27T07:24:01.684 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:01.692+0000 7f0789c7d700 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f07841131c0 con 0x7f07841034b0 2024-05-27T07:24:01.684 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:01.692+0000 7f0789c7d700 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f07841134c0 con 0x7f0784107de0 2024-05-27T07:24:01.684 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:01.692+0000 7f0783fff700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f0784108730 0x7f078419f570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:24:01.684 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:01.692+0000 7f0783fff700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f0784108730 0x7f078419f570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.52:55604/0 (socket says 172.21.15.52:55604) 2024-05-27T07:24:01.684 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:01.692+0000 7f0783fff700 1 -- 172.21.15.52:0/2875182481 learned_addr learned my addr 172.21.15.52:0/2875182481 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-05-27T07:24:01.684 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:01.692+0000 7f0782ffd700 1 --2- 172.21.15.52:0/2875182481 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f0784107de0 0x7f078419b2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:24:01.684 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:01.692+0000 7f07837fe700 1 --2- 172.21.15.52:0/2875182481 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f07841034b0 0x7f078419ad40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:24:01.684 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:01.692+0000 7f0783fff700 1 -- 172.21.15.52:0/2875182481 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f0784107de0 msgr2=0x7f078419b2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:24:01.684 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:01.692+0000 7f0783fff700 1 --2- 172.21.15.52:0/2875182481 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f0784107de0 0x7f078419b2e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:24:01.684 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:01.692+0000 7f0783fff700 1 -- 172.21.15.52:0/2875182481 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f07841034b0 msgr2=0x7f078419ad40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:24:01.685 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:01.692+0000 7f0783fff700 1 --2- 172.21.15.52:0/2875182481 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f07841034b0 0x7f078419ad40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:24:01.685 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:01.692+0000 7f0783fff700 1 -- 172.21.15.52:0/2875182481 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f078419fb60 con 0x7f0784108730 2024-05-27T07:24:01.685 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:01.693+0000 7f0783fff700 1 --2- 172.21.15.52:0/2875182481 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f0784108730 0x7f078419f570 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 crypto rx=0x7f077c007d70 tx=0x7f077c00a340 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:24:01.685 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:01.693+0000 7f0780ff9700 1 -- 172.21.15.52:0/2875182481 <== mon.0 v2:172.21.15.52:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f077c016020 con 0x7f0784108730 2024-05-27T07:24:01.685 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:01.693+0000 7f0780ff9700 1 -- 172.21.15.52:0/2875182481 <== mon.0 v2:172.21.15.52:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f077c00ac90 con 0x7f0784108730 2024-05-27T07:24:01.685 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:01.693+0000 7f0789c7d700 1 -- 172.21.15.52:0/2875182481 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f078419fe50 con 0x7f0784108730 2024-05-27T07:24:01.685 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:01.693+0000 7f0780ff9700 1 -- 172.21.15.52:0/2875182481 <== mon.0 v2:172.21.15.52:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f077c0115e0 con 0x7f0784108730 2024-05-27T07:24:01.685 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:01.693+0000 7f0789c7d700 1 -- 172.21.15.52:0/2875182481 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f07841a0360 con 0x7f0784108730 2024-05-27T07:24:01.685 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:01.694+0000 7f0789c7d700 1 -- 172.21.15.52:0/2875182481 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f0784063f00 con 0x7f0784108730 2024-05-27T07:24:01.685 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:01.696+0000 7f0780ff9700 1 -- 172.21.15.52:0/2875182481 <== mon.0 v2:172.21.15.52:3300/0 4 ==== mgrmap(e 17) ==== 149703+0+0 (secure 0 0 0) 0x7f077c019020 con 0x7f0784108730 2024-05-27T07:24:01.686 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:01.696+0000 7f0780ff9700 1 --2- 172.21.15.52:0/2875182481 >> [v2:172.21.15.52:6800/3513075805,v1:172.21.15.52:6801/3513075805] conn(0x7f076c0b2990 0x7f076c0b4ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:24:01.686 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:01.696+0000 7f07837fe700 1 --2- 172.21.15.52:0/2875182481 >> [v2:172.21.15.52:6800/3513075805,v1:172.21.15.52:6801/3513075805] conn(0x7f076c0b2990 0x7f076c0b4ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:24:01.686 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:01.697+0000 7f0780ff9700 1 -- 172.21.15.52:0/2875182481 <== mon.0 v2:172.21.15.52:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9121+0+0 (secure 0 0 0) 0x7f077c0e2980 con 0x7f0784108730 2024-05-27T07:24:01.687 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:01.697+0000 7f0780ff9700 1 --2- 172.21.15.52:0/2875182481 >> [v2:172.21.15.105:6816/2063611552,v1:172.21.15.105:6817/2063611552] conn(0x7f076c0bdbc0 0x7f076c0c00b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:24:01.687 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:01.697+0000 7f0780ff9700 1 -- 172.21.15.52:0/2875182481 --> [v2:172.21.15.105:6816/2063611552,v1:172.21.15.105:6817/2063611552] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f077c024e50 con 0x7f076c0bdbc0 2024-05-27T07:24:01.687 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:01.697+0000 7f0780ff9700 1 -- 172.21.15.52:0/2875182481 <== mon.0 v2:172.21.15.52:3300/0 6 ==== mon_get_version_reply(handle=1 version=90) ==== 24+0+0 (secure 0 0 0) 0x7f077c0e2dd0 con 0x7f0784108730 2024-05-27T07:24:01.687 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:01.697+0000 7f07837fe700 1 --2- 172.21.15.52:0/2875182481 >> [v2:172.21.15.52:6800/3513075805,v1:172.21.15.52:6801/3513075805] conn(0x7f076c0b2990 0x7f076c0b4ea0 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7f077400ac50 tx=0x7f077400a380 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:24:01.687 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:01.697+0000 7f0782ffd700 1 --2- 172.21.15.52:0/2875182481 >> [v2:172.21.15.105:6816/2063611552,v1:172.21.15.105:6817/2063611552] conn(0x7f076c0bdbc0 0x7f076c0c00b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:24:01.687 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:01.698+0000 7f0782ffd700 1 --2- 172.21.15.52:0/2875182481 >> [v2:172.21.15.105:6816/2063611552,v1:172.21.15.105:6817/2063611552] conn(0x7f076c0bdbc0 0x7f076c0c00b0 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:24:01.688 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:01.698+0000 7f0780ff9700 1 -- 172.21.15.52:0/2875182481 <== osd.7 v2:172.21.15.105:6816/2063611552 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f077c024e50 con 0x7f076c0bdbc0 2024-05-27T07:24:01.705 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:01.715+0000 7f0789c7d700 1 -- 172.21.15.52:0/2875182481 --> [v2:172.21.15.105:6816/2063611552,v1:172.21.15.105:6817/2063611552] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f0784064330 con 0x7f076c0bdbc0 2024-05-27T07:24:01.706 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:01.716+0000 7f0780ff9700 1 -- 172.21.15.52:0/2875182481 <== osd.7 v2:172.21.15.105:6816/2063611552 2 ==== command_reply(tid 2: 0 ) ==== 8+0+12 (crc 0 0 0) 0x7f0784064330 con 0x7f076c0bdbc0 2024-05-27T07:24:01.707 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:01.716+0000 7f0789c7d700 1 -- 172.21.15.52:0/2875182481 >> [v2:172.21.15.105:6816/2063611552,v1:172.21.15.105:6817/2063611552] conn(0x7f076c0bdbc0 msgr2=0x7f076c0c00b0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:24:01.707 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:01.716+0000 7f0789c7d700 1 --2- 172.21.15.52:0/2875182481 >> [v2:172.21.15.105:6816/2063611552,v1:172.21.15.105:6817/2063611552] conn(0x7f076c0bdbc0 0x7f076c0c00b0 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:24:01.707 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:01.716+0000 7f0789c7d700 1 -- 172.21.15.52:0/2875182481 >> [v2:172.21.15.52:6800/3513075805,v1:172.21.15.52:6801/3513075805] conn(0x7f076c0b2990 msgr2=0x7f076c0b4ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:24:01.707 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:01.716+0000 7f0789c7d700 1 --2- 172.21.15.52:0/2875182481 >> [v2:172.21.15.52:6800/3513075805,v1:172.21.15.52:6801/3513075805] conn(0x7f076c0b2990 0x7f076c0b4ea0 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7f077400ac50 tx=0x7f077400a380 comp rx=0 tx=0).stop 2024-05-27T07:24:01.707 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:01.716+0000 7f0789c7d700 1 -- 172.21.15.52:0/2875182481 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f0784108730 msgr2=0x7f078419f570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:24:01.707 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:01.716+0000 7f0789c7d700 1 --2- 172.21.15.52:0/2875182481 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f0784108730 0x7f078419f570 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 crypto rx=0x7f077c007d70 tx=0x7f077c00a340 comp rx=0 tx=0).stop 2024-05-27T07:24:01.707 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:01.716+0000 7f0783fff700 1 -- 172.21.15.52:0/2875182481 reap_dead start 2024-05-27T07:24:01.707 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:01.716+0000 7f0789c7d700 1 -- 172.21.15.52:0/2875182481 shutdown_connections 2024-05-27T07:24:01.707 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:01.716+0000 7f0789c7d700 1 -- 172.21.15.52:0/2875182481 >> 172.21.15.52:0/2875182481 conn(0x7f07840fedb0 msgr2=0x7f078406b880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:24:01.707 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:01.717+0000 7f0789c7d700 1 -- 172.21.15.52:0/2875182481 shutdown_connections 2024-05-27T07:24:01.707 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:01.717+0000 7f0789c7d700 1 -- 172.21.15.52:0/2875182481 wait complete. 2024-05-27T07:24:02.301 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:02.308+0000 7f2e688ba700 1 -- 172.21.15.52:0/1586905352 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f2e641045e0 msgr2=0x7f2e641049f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:24:02.301 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:02.308+0000 7f2e688ba700 1 --2- 172.21.15.52:0/1586905352 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f2e641045e0 0x7f2e641049f0 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 crypto rx=0x7f2e540099c0 tx=0x7f2e54009cd0 comp rx=0 tx=0).stop 2024-05-27T07:24:02.301 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:02.309+0000 7f2e688ba700 1 -- 172.21.15.52:0/1586905352 shutdown_connections 2024-05-27T07:24:02.301 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:02.309+0000 7f2e688ba700 1 --2- 172.21.15.52:0/1586905352 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f2e64109860 0x7f2e641111c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:24:02.301 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:02.309+0000 7f2e688ba700 1 --2- 172.21.15.52:0/1586905352 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f2e641045e0 0x7f2e641049f0 unknown :-1 s=CLOSED pgs=273 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:24:02.301 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:02.309+0000 7f2e688ba700 1 --2- 172.21.15.52:0/1586905352 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f2e64108f10 0x7f2e64109320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:24:02.301 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:02.309+0000 7f2e688ba700 1 -- 172.21.15.52:0/1586905352 >> 172.21.15.52:0/1586905352 conn(0x7f2e64100030 msgr2=0x7f2e64102450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:24:02.301 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:02.309+0000 7f2e688ba700 1 -- 172.21.15.52:0/1586905352 shutdown_connections 2024-05-27T07:24:02.301 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:02.309+0000 7f2e688ba700 1 -- 172.21.15.52:0/1586905352 wait complete. 2024-05-27T07:24:02.301 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:02.310+0000 7f2e688ba700 1 Processor -- start 2024-05-27T07:24:02.302 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:02.310+0000 7f2e688ba700 1 -- start start 2024-05-27T07:24:02.302 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:02.310+0000 7f2e688ba700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f2e641045e0 0x7f2e6419ae40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:24:02.302 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:02.310+0000 7f2e688ba700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f2e64108f10 0x7f2e6419b3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:24:02.302 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:02.310+0000 7f2e688ba700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f2e64109860 0x7f2e6419f670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:24:02.302 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:02.310+0000 7f2e688ba700 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x7f2e64114490 con 0x7f2e64109860 2024-05-27T07:24:02.302 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:02.310+0000 7f2e688ba700 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f2e64114310 con 0x7f2e641045e0 2024-05-27T07:24:02.302 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:02.310+0000 7f2e688ba700 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f2e64114610 con 0x7f2e64108f10 2024-05-27T07:24:02.302 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:02.310+0000 7f2e62d9d700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f2e64109860 0x7f2e6419f670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:24:02.302 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:02.310+0000 7f2e62d9d700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f2e64109860 0x7f2e6419f670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.52:55624/0 (socket says 172.21.15.52:55624) 2024-05-27T07:24:02.302 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:02.310+0000 7f2e62d9d700 1 -- 172.21.15.52:0/552656397 learned_addr learned my addr 172.21.15.52:0/552656397 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-05-27T07:24:02.302 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:02.310+0000 7f2e6259c700 1 --2- 172.21.15.52:0/552656397 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f2e641045e0 0x7f2e6419ae40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:24:02.302 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:02.311+0000 7f2e59d9b700 1 --2- 172.21.15.52:0/552656397 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f2e64108f10 0x7f2e6419b3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:24:02.303 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:02.311+0000 7f2e6259c700 1 -- 172.21.15.52:0/552656397 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f2e64108f10 msgr2=0x7f2e6419b3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:24:02.303 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:02.311+0000 7f2e6259c700 1 --2- 172.21.15.52:0/552656397 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f2e64108f10 0x7f2e6419b3e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:24:02.303 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:02.311+0000 7f2e6259c700 1 -- 172.21.15.52:0/552656397 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f2e64109860 msgr2=0x7f2e6419f670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:24:02.303 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:02.311+0000 7f2e6259c700 1 --2- 172.21.15.52:0/552656397 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f2e64109860 0x7f2e6419f670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:24:02.303 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:02.311+0000 7f2e6259c700 1 -- 172.21.15.52:0/552656397 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2e6419fd10 con 0x7f2e641045e0 2024-05-27T07:24:02.303 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:02.311+0000 7f2e62d9d700 1 --2- 172.21.15.52:0/552656397 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f2e64109860 0x7f2e6419f670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-27T07:24:02.303 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:02.311+0000 7f2e6259c700 1 --2- 172.21.15.52:0/552656397 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f2e641045e0 0x7f2e6419ae40 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f2e5400b580 tx=0x7f2e540055f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:24:02.304 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:02.311+0000 7f2e5bfff700 1 -- 172.21.15.52:0/552656397 <== mon.2 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2e54026070 con 0x7f2e641045e0 2024-05-27T07:24:02.304 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:02.311+0000 7f2e688ba700 1 -- 172.21.15.52:0/552656397 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2e6419ffa0 con 0x7f2e641045e0 2024-05-27T07:24:02.304 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:02.311+0000 7f2e5bfff700 1 -- 172.21.15.52:0/552656397 <== mon.2 v2:172.21.15.105:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f2e5400f460 con 0x7f2e641045e0 2024-05-27T07:24:02.304 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:02.311+0000 7f2e5bfff700 1 -- 172.21.15.52:0/552656397 <== mon.2 v2:172.21.15.105:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2e54009e60 con 0x7f2e641045e0 2024-05-27T07:24:02.304 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:02.311+0000 7f2e688ba700 1 -- 172.21.15.52:0/552656397 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2e641a7920 con 0x7f2e641045e0 2024-05-27T07:24:02.304 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:02.312+0000 7f2e688ba700 1 -- 172.21.15.52:0/552656397 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f2e641a04d0 con 0x7f2e641045e0 2024-05-27T07:24:02.304 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:02.314+0000 7f2e5bfff700 1 -- 172.21.15.52:0/552656397 <== mon.2 v2:172.21.15.105:3300/0 4 ==== mgrmap(e 17) ==== 149703+0+0 (secure 0 0 0) 0x7f2e54029070 con 0x7f2e641045e0 2024-05-27T07:24:02.305 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:02.315+0000 7f2e5bfff700 1 --2- 172.21.15.52:0/552656397 >> [v2:172.21.15.52:6800/3513075805,v1:172.21.15.52:6801/3513075805] conn(0x7f2e500b2910 0x7f2e500b4e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:24:02.305 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:02.315+0000 7f2e5bfff700 1 -- 172.21.15.52:0/552656397 <== mon.2 v2:172.21.15.105:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9121+0+0 (secure 0 0 0) 0x7f2e540e37d0 con 0x7f2e641045e0 2024-05-27T07:24:02.306 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:02.315+0000 7f2e59d9b700 1 --2- 172.21.15.52:0/552656397 >> [v2:172.21.15.52:6800/3513075805,v1:172.21.15.52:6801/3513075805] conn(0x7f2e500b2910 0x7f2e500b4e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:24:02.306 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:02.315+0000 7f2e5bfff700 1 --2- 172.21.15.52:0/552656397 >> [v2:172.21.15.105:6824/4032386628,v1:172.21.15.105:6825/4032386628] conn(0x7f2e500bdb40 0x7f2e500c0030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:24:02.306 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:02.315+0000 7f2e5bfff700 1 -- 172.21.15.52:0/552656397 --> [v2:172.21.15.105:6824/4032386628,v1:172.21.15.105:6825/4032386628] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f2e500c0700 con 0x7f2e500bdb40 2024-05-27T07:24:02.306 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:02.315+0000 7f2e5bfff700 1 -- 172.21.15.52:0/552656397 <== mon.2 v2:172.21.15.105:3300/0 6 ==== mon_get_version_reply(handle=1 version=90) ==== 24+0+0 (secure 0 0 0) 0x7f2e540e3c10 con 0x7f2e641045e0 2024-05-27T07:24:02.306 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:02.316+0000 7f2e59d9b700 1 --2- 172.21.15.52:0/552656397 >> [v2:172.21.15.52:6800/3513075805,v1:172.21.15.52:6801/3513075805] conn(0x7f2e500b2910 0x7f2e500b4e20 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7f2e64107850 tx=0x7f2e4c009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:24:02.306 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:02.316+0000 7f2e62d9d700 1 --2- 172.21.15.52:0/552656397 >> [v2:172.21.15.105:6824/4032386628,v1:172.21.15.105:6825/4032386628] conn(0x7f2e500bdb40 0x7f2e500c0030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:24:02.306 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:02.316+0000 7f2e62d9d700 1 --2- 172.21.15.52:0/552656397 >> [v2:172.21.15.105:6824/4032386628,v1:172.21.15.105:6825/4032386628] conn(0x7f2e500bdb40 0x7f2e500c0030 crc :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.10 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:24:02.307 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:02.317+0000 7f2e5bfff700 1 -- 172.21.15.52:0/552656397 <== osd.10 v2:172.21.15.105:6824/4032386628 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f2e500c0700 con 0x7f2e500bdb40 2024-05-27T07:24:02.324 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:02.334+0000 7f2e688ba700 1 -- 172.21.15.52:0/552656397 --> [v2:172.21.15.105:6824/4032386628,v1:172.21.15.105:6825/4032386628] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f2e64064190 con 0x7f2e500bdb40 2024-05-27T07:24:02.324 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:02.335+0000 7f2e5bfff700 1 -- 172.21.15.52:0/552656397 <== osd.10 v2:172.21.15.105:6824/4032386628 2 ==== command_reply(tid 2: 0 ) ==== 8+0+12 (crc 0 0 0) 0x7f2e64064190 con 0x7f2e500bdb40 2024-05-27T07:24:02.325 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:02.335+0000 7f2e688ba700 1 -- 172.21.15.52:0/552656397 >> [v2:172.21.15.105:6824/4032386628,v1:172.21.15.105:6825/4032386628] conn(0x7f2e500bdb40 msgr2=0x7f2e500c0030 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:24:02.325 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:02.335+0000 7f2e688ba700 1 --2- 172.21.15.52:0/552656397 >> [v2:172.21.15.105:6824/4032386628,v1:172.21.15.105:6825/4032386628] conn(0x7f2e500bdb40 0x7f2e500c0030 crc :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:24:02.325 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:02.335+0000 7f2e688ba700 1 -- 172.21.15.52:0/552656397 >> [v2:172.21.15.52:6800/3513075805,v1:172.21.15.52:6801/3513075805] conn(0x7f2e500b2910 msgr2=0x7f2e500b4e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:24:02.326 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:02.335+0000 7f2e688ba700 1 --2- 172.21.15.52:0/552656397 >> [v2:172.21.15.52:6800/3513075805,v1:172.21.15.52:6801/3513075805] conn(0x7f2e500b2910 0x7f2e500b4e20 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7f2e64107850 tx=0x7f2e4c009260 comp rx=0 tx=0).stop 2024-05-27T07:24:02.326 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:02.335+0000 7f2e688ba700 1 -- 172.21.15.52:0/552656397 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f2e641045e0 msgr2=0x7f2e6419ae40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:24:02.326 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:02.335+0000 7f2e688ba700 1 --2- 172.21.15.52:0/552656397 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f2e641045e0 0x7f2e6419ae40 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f2e5400b580 tx=0x7f2e540055f0 comp rx=0 tx=0).stop 2024-05-27T07:24:02.326 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:02.335+0000 7f2e62d9d700 1 -- 172.21.15.52:0/552656397 reap_dead start 2024-05-27T07:24:02.326 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:02.335+0000 7f2e688ba700 1 -- 172.21.15.52:0/552656397 shutdown_connections 2024-05-27T07:24:02.326 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:02.336+0000 7f2e688ba700 1 -- 172.21.15.52:0/552656397 >> 172.21.15.52:0/552656397 conn(0x7f2e64100030 msgr2=0x7f2e6410a7c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:24:02.326 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:02.336+0000 7f2e688ba700 1 -- 172.21.15.52:0/552656397 shutdown_connections 2024-05-27T07:24:02.326 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:02.336+0000 7f2e688ba700 1 -- 172.21.15.52:0/552656397 wait complete. 2024-05-27T07:24:02.741 INFO:teuthology.orchestra.run.smithi052.stdout:206158430270 2024-05-27T07:24:02.742 DEBUG:teuthology.orchestra.run.smithi052:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:38f9d900c83c63799fbdbe61acc9a11b0d3554a6 shell --fsid 73df2636-1bf8-11ef-bc9b-c7b262605968 -- ceph osd last-stat-seq osd.7 2024-05-27T07:24:02.979 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:24:02 smithi052 ceph-mon[31955]: pgmap v366: 129 pgs: 129 active+clean; 647 KiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 9 op/s 2024-05-27T07:24:03.102 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:24:02 smithi173 ceph-mon[37000]: pgmap v366: 129 pgs: 129 active+clean; 647 KiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 9 op/s 2024-05-27T07:24:03.189 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:24:02 smithi105 ceph-mon[35977]: pgmap v366: 129 pgs: 129 active+clean; 647 KiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 9 op/s 2024-05-27T07:24:03.205 INFO:teuthology.orchestra.run.smithi052.stdout:137438953549 2024-05-27T07:24:03.205 DEBUG:teuthology.orchestra.run.smithi052:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:38f9d900c83c63799fbdbe61acc9a11b0d3554a6 shell --fsid 73df2636-1bf8-11ef-bc9b-c7b262605968 -- ceph osd last-stat-seq osd.4 2024-05-27T07:24:03.818 INFO:teuthology.orchestra.run.smithi052.stdout:184683593796 2024-05-27T07:24:03.819 DEBUG:teuthology.orchestra.run.smithi052:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:38f9d900c83c63799fbdbe61acc9a11b0d3554a6 shell --fsid 73df2636-1bf8-11ef-bc9b-c7b262605968 -- ceph osd last-stat-seq osd.6 2024-05-27T07:24:04.284 INFO:teuthology.orchestra.run.smithi052.stdout:158913790023 2024-05-27T07:24:04.284 DEBUG:teuthology.orchestra.run.smithi052:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:38f9d900c83c63799fbdbe61acc9a11b0d3554a6 shell --fsid 73df2636-1bf8-11ef-bc9b-c7b262605968 -- ceph osd last-stat-seq osd.5 2024-05-27T07:24:04.488 INFO:teuthology.orchestra.run.smithi052.stderr:Inferring config /var/lib/ceph/73df2636-1bf8-11ef-bc9b-c7b262605968/mon.a/config 2024-05-27T07:24:04.489 INFO:teuthology.orchestra.run.smithi052.stderr:Inferring config /var/lib/ceph/73df2636-1bf8-11ef-bc9b-c7b262605968/mon.a/config 2024-05-27T07:24:04.858 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:24:04 smithi052 ceph-mon[31955]: pgmap v367: 129 pgs: 129 active+clean; 647 KiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 10 op/s 2024-05-27T07:24:04.887 INFO:teuthology.orchestra.run.smithi052.stdout:279172874287 2024-05-27T07:24:04.887 DEBUG:teuthology.orchestra.run.smithi052:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:38f9d900c83c63799fbdbe61acc9a11b0d3554a6 shell --fsid 73df2636-1bf8-11ef-bc9b-c7b262605968 -- ceph osd last-stat-seq osd.10 2024-05-27T07:24:04.893 INFO:teuthology.orchestra.run.smithi052.stderr:Inferring config /var/lib/ceph/73df2636-1bf8-11ef-bc9b-c7b262605968/mon.a/config 2024-05-27T07:24:04.938 INFO:teuthology.orchestra.run.smithi052.stderr:Inferring config /var/lib/ceph/73df2636-1bf8-11ef-bc9b-c7b262605968/mon.a/config 2024-05-27T07:24:04.939 INFO:teuthology.orchestra.run.smithi052.stderr:Inferring config /var/lib/ceph/73df2636-1bf8-11ef-bc9b-c7b262605968/mon.a/config 2024-05-27T07:24:05.101 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:24:04 smithi173 ceph-mon[37000]: pgmap v367: 129 pgs: 129 active+clean; 647 KiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 10 op/s 2024-05-27T07:24:05.190 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:24:04 smithi105 ceph-mon[35977]: pgmap v367: 129 pgs: 129 active+clean; 647 KiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 10 op/s 2024-05-27T07:24:05.404 INFO:teuthology.orchestra.run.smithi052.stderr:Inferring config /var/lib/ceph/73df2636-1bf8-11ef-bc9b-c7b262605968/mon.a/config 2024-05-27T07:24:06.964 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:24:06 smithi052 ceph-mon[31955]: pgmap v368: 129 pgs: 129 active+clean; 647 KiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail; 5.9 KiB/s rd, 11 op/s 2024-05-27T07:24:07.102 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:24:06 smithi173 ceph-mon[37000]: pgmap v368: 129 pgs: 129 active+clean; 647 KiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail; 5.9 KiB/s rd, 11 op/s 2024-05-27T07:24:07.190 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:24:06 smithi105 ceph-mon[35977]: pgmap v368: 129 pgs: 129 active+clean; 647 KiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail; 5.9 KiB/s rd, 11 op/s 2024-05-27T07:24:08.979 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:24:08 smithi052 ceph-mon[31955]: pgmap v369: 129 pgs: 129 active+clean; 647 KiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-05-27T07:24:09.102 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:24:08 smithi173 ceph-mon[37000]: pgmap v369: 129 pgs: 129 active+clean; 647 KiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-05-27T07:24:09.190 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:24:08 smithi105 ceph-mon[35977]: pgmap v369: 129 pgs: 129 active+clean; 647 KiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-05-27T07:24:09.335 INFO:teuthology.orchestra.run.smithi052.stderr:Inferring config /var/lib/ceph/73df2636-1bf8-11ef-bc9b-c7b262605968/mon.a/config 2024-05-27T07:24:09.336 INFO:teuthology.orchestra.run.smithi052.stderr:Inferring config /var/lib/ceph/73df2636-1bf8-11ef-bc9b-c7b262605968/mon.a/config 2024-05-27T07:24:10.427 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:10.434+0000 7f3104632700 1 -- 172.21.15.52:0/3244858867 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f30fc105580 msgr2=0x7f30fc105990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:24:10.427 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:10.434+0000 7f3104632700 1 --2- 172.21.15.52:0/3244858867 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f30fc105580 0x7f30fc105990 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 crypto rx=0x7f30f800b0a0 tx=0x7f30f800b3b0 comp rx=0 tx=0).stop 2024-05-27T07:24:10.427 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:10.434+0000 7f3104632700 1 -- 172.21.15.52:0/3244858867 shutdown_connections 2024-05-27T07:24:10.427 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:10.434+0000 7f3104632700 1 --2- 172.21.15.52:0/3244858867 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f30fc10e960 0x7f30fc110e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:24:10.427 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:10.434+0000 7f3104632700 1 --2- 172.21.15.52:0/3244858867 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f30fc105580 0x7f30fc105990 unknown :-1 s=CLOSED pgs=274 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:24:10.427 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:10.434+0000 7f3104632700 1 --2- 172.21.15.52:0/3244858867 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f30fc105ed0 0x7f30fc10e420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:24:10.427 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:10.434+0000 7f3104632700 1 -- 172.21.15.52:0/3244858867 >> 172.21.15.52:0/3244858867 conn(0x7f30fc078df0 msgr2=0x7f30fc079230 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:24:10.427 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:10.434+0000 7f3104632700 1 -- 172.21.15.52:0/3244858867 shutdown_connections 2024-05-27T07:24:10.427 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:10.435+0000 7f3104632700 1 -- 172.21.15.52:0/3244858867 wait complete. 2024-05-27T07:24:10.428 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:10.435+0000 7f3104632700 1 Processor -- start 2024-05-27T07:24:10.428 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:10.435+0000 7f3104632700 1 -- start start 2024-05-27T07:24:10.428 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:10.435+0000 7f3104632700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f30fc105580 0x7f30fc19ae00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:24:10.428 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:10.435+0000 7f3104632700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f30fc105ed0 0x7f30fc19b3a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:24:10.428 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:10.435+0000 7f3104632700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f30fc10e960 0x7f30fc19f630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:24:10.428 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:10.436+0000 7f3104632700 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x7f30fc114470 con 0x7f30fc105580 2024-05-27T07:24:10.428 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:10.436+0000 7f3104632700 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f30fc1142f0 con 0x7f30fc10e960 2024-05-27T07:24:10.428 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:10.436+0000 7f3104632700 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f30fc1145f0 con 0x7f30fc105ed0 2024-05-27T07:24:10.428 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:10.436+0000 7f31023ce700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f30fc105580 0x7f30fc19ae00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:24:10.428 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:10.436+0000 7f31023ce700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f30fc105580 0x7f30fc19ae00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.52:55636/0 (socket says 172.21.15.52:55636) 2024-05-27T07:24:10.428 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:10.436+0000 7f31023ce700 1 -- 172.21.15.52:0/4232184723 learned_addr learned my addr 172.21.15.52:0/4232184723 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-05-27T07:24:10.428 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:10.436+0000 7f3101bcd700 1 --2- 172.21.15.52:0/4232184723 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f30fc105ed0 0x7f30fc19b3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:24:10.429 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:10.436+0000 7f3102bcf700 1 --2- 172.21.15.52:0/4232184723 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f30fc10e960 0x7f30fc19f630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:24:10.429 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:10.436+0000 7f31023ce700 1 -- 172.21.15.52:0/4232184723 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f30fc105ed0 msgr2=0x7f30fc19b3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:24:10.429 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:10.436+0000 7f31023ce700 1 --2- 172.21.15.52:0/4232184723 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f30fc105ed0 0x7f30fc19b3a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:24:10.429 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:10.436+0000 7f31023ce700 1 -- 172.21.15.52:0/4232184723 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f30fc10e960 msgr2=0x7f30fc19f630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:24:10.429 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:10.436+0000 7f31023ce700 1 --2- 172.21.15.52:0/4232184723 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f30fc10e960 0x7f30fc19f630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:24:10.429 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:10.436+0000 7f31023ce700 1 -- 172.21.15.52:0/4232184723 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f30fc19fc20 con 0x7f30fc105580 2024-05-27T07:24:10.429 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:10.436+0000 7f31023ce700 1 --2- 172.21.15.52:0/4232184723 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f30fc105580 0x7f30fc19ae00 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 crypto rx=0x7f30f8006070 tx=0x7f30f8004ef0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:24:10.429 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:10.436+0000 7f30f37fe700 1 -- 172.21.15.52:0/4232184723 <== mon.0 v2:172.21.15.52:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f30f8021070 con 0x7f30fc105580 2024-05-27T07:24:10.429 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:10.436+0000 7f3104632700 1 -- 172.21.15.52:0/4232184723 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f30fc19feb0 con 0x7f30fc105580 2024-05-27T07:24:10.429 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:10.436+0000 7f30f37fe700 1 -- 172.21.15.52:0/4232184723 <== mon.0 v2:172.21.15.52:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f30f8007df0 con 0x7f30fc105580 2024-05-27T07:24:10.430 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:10.436+0000 7f30f37fe700 1 -- 172.21.15.52:0/4232184723 <== mon.0 v2:172.21.15.52:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f30f801d450 con 0x7f30fc105580 2024-05-27T07:24:10.431 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:10.437+0000 7f3104632700 1 -- 172.21.15.52:0/4232184723 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f30fc1a0440 con 0x7f30fc105580 2024-05-27T07:24:10.431 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:10.437+0000 7f3104632700 1 -- 172.21.15.52:0/4232184723 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f30fc0512b0 con 0x7f30fc105580 2024-05-27T07:24:10.431 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:10.439+0000 7f30f37fe700 1 -- 172.21.15.52:0/4232184723 <== mon.0 v2:172.21.15.52:3300/0 4 ==== mgrmap(e 17) ==== 149703+0+0 (secure 0 0 0) 0x7f30f80040b0 con 0x7f30fc105580 2024-05-27T07:24:10.431 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:10.440+0000 7f30f37fe700 1 --2- 172.21.15.52:0/4232184723 >> [v2:172.21.15.52:6800/3513075805,v1:172.21.15.52:6801/3513075805] conn(0x7f30e80b2910 0x7f30e80b4e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:24:10.431 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:10.440+0000 7f3101bcd700 1 --2- 172.21.15.52:0/4232184723 >> [v2:172.21.15.52:6800/3513075805,v1:172.21.15.52:6801/3513075805] conn(0x7f30e80b2910 0x7f30e80b4e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:24:10.431 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:10.440+0000 7f30f37fe700 1 -- 172.21.15.52:0/4232184723 <== mon.0 v2:172.21.15.52:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9121+0+0 (secure 0 0 0) 0x7f30f800f020 con 0x7f30fc105580 2024-05-27T07:24:10.431 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:10.441+0000 7f3101bcd700 1 --2- 172.21.15.52:0/4232184723 >> [v2:172.21.15.52:6800/3513075805,v1:172.21.15.52:6801/3513075805] conn(0x7f30e80b2910 0x7f30e80b4e20 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7f30ec005fd0 tx=0x7f30ec00a560 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:24:10.434 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:10.444+0000 7f30f37fe700 1 -- 172.21.15.52:0/4232184723 <== mon.0 v2:172.21.15.52:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191963 (secure 0 0 0) 0x7f30f802a020 con 0x7f30fc105580 2024-05-27T07:24:10.669 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:10.679+0000 7f3104632700 1 -- 172.21.15.52:0/4232184723 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 4} v 0) -- 0x7f30fc19c160 con 0x7f30fc105580 2024-05-27T07:24:10.669 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:10.679+0000 7f30f37fe700 1 -- 172.21.15.52:0/4232184723 <== mon.0 v2:172.21.15.52:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 4}]=0 v0) ==== 74+0+13 (secure 0 0 0) 0x7f30f80abb40 con 0x7f30fc105580 2024-05-27T07:24:10.670 INFO:teuthology.orchestra.run.smithi052.stdout:137438953551 2024-05-27T07:24:10.673 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:10.682+0000 7f3104632700 1 -- 172.21.15.52:0/4232184723 >> [v2:172.21.15.52:6800/3513075805,v1:172.21.15.52:6801/3513075805] conn(0x7f30e80b2910 msgr2=0x7f30e80b4e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:24:10.673 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:10.682+0000 7f3104632700 1 --2- 172.21.15.52:0/4232184723 >> [v2:172.21.15.52:6800/3513075805,v1:172.21.15.52:6801/3513075805] conn(0x7f30e80b2910 0x7f30e80b4e20 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7f30ec005fd0 tx=0x7f30ec00a560 comp rx=0 tx=0).stop 2024-05-27T07:24:10.673 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:10.682+0000 7f3104632700 1 -- 172.21.15.52:0/4232184723 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f30fc105580 msgr2=0x7f30fc19ae00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:24:10.673 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:10.682+0000 7f3104632700 1 --2- 172.21.15.52:0/4232184723 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f30fc105580 0x7f30fc19ae00 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 crypto rx=0x7f30f8006070 tx=0x7f30f8004ef0 comp rx=0 tx=0).stop 2024-05-27T07:24:10.673 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:10.682+0000 7f3104632700 1 -- 172.21.15.52:0/4232184723 shutdown_connections 2024-05-27T07:24:10.673 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:10.682+0000 7f3104632700 1 --2- 172.21.15.52:0/4232184723 >> [v2:172.21.15.52:6800/3513075805,v1:172.21.15.52:6801/3513075805] conn(0x7f30e80b2910 0x7f30e80b4e20 unknown :-1 s=CLOSED pgs=188 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:24:10.673 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:10.682+0000 7f3104632700 1 --2- 172.21.15.52:0/4232184723 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f30fc10e960 0x7f30fc19f630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:24:10.673 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:10.683+0000 7f3104632700 1 --2- 172.21.15.52:0/4232184723 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f30fc105580 0x7f30fc19ae00 unknown :-1 s=CLOSED pgs=275 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:24:10.673 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:10.683+0000 7f3104632700 1 --2- 172.21.15.52:0/4232184723 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f30fc105ed0 0x7f30fc19b3a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:24:10.673 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:10.683+0000 7f3104632700 1 -- 172.21.15.52:0/4232184723 >> 172.21.15.52:0/4232184723 conn(0x7f30fc078df0 msgr2=0x7f30fc1000b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:24:10.673 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:10.683+0000 7f3104632700 1 -- 172.21.15.52:0/4232184723 shutdown_connections 2024-05-27T07:24:10.673 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:10.683+0000 7f3104632700 1 -- 172.21.15.52:0/4232184723 wait complete. 2024-05-27T07:24:10.863 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:24:10 smithi052 ceph-mon[31955]: pgmap v370: 129 pgs: 129 active+clean; 647 KiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 9 op/s 2024-05-27T07:24:10.864 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:24:10 smithi052 ceph-mon[31955]: from='client.? 172.21.15.52:0/4232184723' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 4}]: dispatch 2024-05-27T07:24:11.062 INFO:teuthology.orchestra.run.smithi052.stderr:Inferring config /var/lib/ceph/73df2636-1bf8-11ef-bc9b-c7b262605968/mon.a/config 2024-05-27T07:24:11.062 INFO:teuthology.orchestra.run.smithi052.stderr:Inferring config /var/lib/ceph/73df2636-1bf8-11ef-bc9b-c7b262605968/mon.a/config 2024-05-27T07:24:11.102 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:24:10 smithi173 ceph-mon[37000]: pgmap v370: 129 pgs: 129 active+clean; 647 KiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 9 op/s 2024-05-27T07:24:11.102 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:24:10 smithi173 ceph-mon[37000]: from='client.? 172.21.15.52:0/4232184723' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 4}]: dispatch 2024-05-27T07:24:11.190 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:24:10 smithi105 ceph-mon[35977]: pgmap v370: 129 pgs: 129 active+clean; 647 KiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 9 op/s 2024-05-27T07:24:11.190 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:24:10 smithi105 ceph-mon[35977]: from='client.? 172.21.15.52:0/4232184723' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 4}]: dispatch 2024-05-27T07:24:11.359 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:11.369+0000 7f8c781ec700 1 -- 172.21.15.52:0/1849529767 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f8c70108730 msgr2=0x7f8c70110090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:24:11.359 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:11.369+0000 7f8c781ec700 1 --2- 172.21.15.52:0/1849529767 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f8c70108730 0x7f8c70110090 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 crypto rx=0x7f8c680099c0 tx=0x7f8c68009cd0 comp rx=0 tx=0).stop 2024-05-27T07:24:11.359 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:11.369+0000 7f8c781ec700 1 -- 172.21.15.52:0/1849529767 shutdown_connections 2024-05-27T07:24:11.359 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:11.369+0000 7f8c781ec700 1 --2- 172.21.15.52:0/1849529767 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f8c70108730 0x7f8c70110090 unknown :-1 s=CLOSED pgs=276 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:24:11.359 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:11.369+0000 7f8c781ec700 1 --2- 172.21.15.52:0/1849529767 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f8c701034b0 0x7f8c701038c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:24:11.359 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:11.369+0000 7f8c781ec700 1 --2- 172.21.15.52:0/1849529767 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f8c70107de0 0x7f8c701081f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:24:11.359 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:11.369+0000 7f8c781ec700 1 -- 172.21.15.52:0/1849529767 >> 172.21.15.52:0/1849529767 conn(0x7f8c700fedb0 msgr2=0x7f8c701011d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:24:11.359 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:11.369+0000 7f8c781ec700 1 -- 172.21.15.52:0/1849529767 shutdown_connections 2024-05-27T07:24:11.359 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:11.369+0000 7f8c781ec700 1 -- 172.21.15.52:0/1849529767 wait complete. 2024-05-27T07:24:11.361 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:11.369+0000 7f8c781ec700 1 Processor -- start 2024-05-27T07:24:11.361 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:11.370+0000 7f8c781ec700 1 -- start start 2024-05-27T07:24:11.361 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:11.370+0000 7f8c781ec700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f8c701034b0 0x7f8c7019bc80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:24:11.361 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:11.370+0000 7f8c781ec700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f8c70107de0 0x7f8c7019c1c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:24:11.362 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:11.370+0000 7f8c781ec700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f8c70108730 0x7f8c7019c740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:24:11.362 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:11.370+0000 7f8c781ec700 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x7f8c70113380 con 0x7f8c70107de0 2024-05-27T07:24:11.362 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:11.370+0000 7f8c781ec700 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f8c70113200 con 0x7f8c70108730 2024-05-27T07:24:11.362 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:11.370+0000 7f8c781ec700 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f8c70113500 con 0x7f8c701034b0 2024-05-27T07:24:11.362 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:11.370+0000 7f8c75787700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f8c70107de0 0x7f8c7019c1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:24:11.362 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:11.370+0000 7f8c75787700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f8c70107de0 0x7f8c7019c1c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.52:60882/0 (socket says 172.21.15.52:60882) 2024-05-27T07:24:11.362 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:11.370+0000 7f8c75787700 1 -- 172.21.15.52:0/4181765950 learned_addr learned my addr 172.21.15.52:0/4181765950 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-05-27T07:24:11.362 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:11.370+0000 7f8c75f88700 1 --2- 172.21.15.52:0/4181765950 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f8c701034b0 0x7f8c7019bc80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:24:11.362 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:11.370+0000 7f8c76789700 1 --2- 172.21.15.52:0/4181765950 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f8c70108730 0x7f8c7019c740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:24:11.362 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:11.370+0000 7f8c75787700 1 -- 172.21.15.52:0/4181765950 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f8c701034b0 msgr2=0x7f8c7019bc80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:24:11.362 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:11.370+0000 7f8c75787700 1 --2- 172.21.15.52:0/4181765950 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f8c701034b0 0x7f8c7019bc80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:24:11.362 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:11.370+0000 7f8c75787700 1 -- 172.21.15.52:0/4181765950 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f8c70108730 msgr2=0x7f8c7019c740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:24:11.363 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:11.370+0000 7f8c75787700 1 --2- 172.21.15.52:0/4181765950 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f8c70108730 0x7f8c7019c740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:24:11.363 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:11.370+0000 7f8c75787700 1 -- 172.21.15.52:0/4181765950 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8c701a6ff0 con 0x7f8c70107de0 2024-05-27T07:24:11.363 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:11.371+0000 7f8c75787700 1 --2- 172.21.15.52:0/4181765950 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f8c70107de0 0x7f8c7019c1c0 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 crypto rx=0x7f8c6000d950 tx=0x7f8c6000dc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:24:11.363 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:11.371+0000 7f8c66ffd700 1 -- 172.21.15.52:0/4181765950 <== mon.0 v2:172.21.15.52:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8c60017070 con 0x7f8c70107de0 2024-05-27T07:24:11.363 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:11.371+0000 7f8c66ffd700 1 -- 172.21.15.52:0/4181765950 <== mon.0 v2:172.21.15.52:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f8c60005020 con 0x7f8c70107de0 2024-05-27T07:24:11.363 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:11.371+0000 7f8c781ec700 1 -- 172.21.15.52:0/4181765950 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8c701a72e0 con 0x7f8c70107de0 2024-05-27T07:24:11.363 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:11.371+0000 7f8c66ffd700 1 -- 172.21.15.52:0/4181765950 <== mon.0 v2:172.21.15.52:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8c6000fa10 con 0x7f8c70107de0 2024-05-27T07:24:11.363 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:11.371+0000 7f8c781ec700 1 -- 172.21.15.52:0/4181765950 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8c701a7820 con 0x7f8c70107de0 2024-05-27T07:24:11.364 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:11.372+0000 7f8c781ec700 1 -- 172.21.15.52:0/4181765950 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8c700512b0 con 0x7f8c70107de0 2024-05-27T07:24:11.364 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:11.374+0000 7f8c66ffd700 1 -- 172.21.15.52:0/4181765950 <== mon.0 v2:172.21.15.52:3300/0 4 ==== mgrmap(e 17) ==== 149703+0+0 (secure 0 0 0) 0x7f8c60004a30 con 0x7f8c70107de0 2024-05-27T07:24:11.364 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:11.374+0000 7f8c66ffd700 1 --2- 172.21.15.52:0/4181765950 >> [v2:172.21.15.52:6800/3513075805,v1:172.21.15.52:6801/3513075805] conn(0x7f8c5c0b2990 0x7f8c5c0b4ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:24:11.365 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:11.374+0000 7f8c75f88700 1 --2- 172.21.15.52:0/4181765950 >> [v2:172.21.15.52:6800/3513075805,v1:172.21.15.52:6801/3513075805] conn(0x7f8c5c0b2990 0x7f8c5c0b4ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:24:11.365 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:11.375+0000 7f8c66ffd700 1 -- 172.21.15.52:0/4181765950 <== mon.0 v2:172.21.15.52:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9121+0+0 (secure 0 0 0) 0x7f8c600e22a0 con 0x7f8c70107de0 2024-05-27T07:24:11.365 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:11.375+0000 7f8c75f88700 1 --2- 172.21.15.52:0/4181765950 >> [v2:172.21.15.52:6800/3513075805,v1:172.21.15.52:6801/3513075805] conn(0x7f8c5c0b2990 0x7f8c5c0b4ea0 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7f8c6c009550 tx=0x7f8c6c0091b0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:24:11.368 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:11.378+0000 7f8c66ffd700 1 -- 172.21.15.52:0/4181765950 <== mon.0 v2:172.21.15.52:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191963 (secure 0 0 0) 0x7f8c600a9b60 con 0x7f8c70107de0 2024-05-27T07:24:11.590 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:11.600+0000 7f8c781ec700 1 -- 172.21.15.52:0/4181765950 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 1} v 0) -- 0x7f8c7002e5e0 con 0x7f8c70107de0 2024-05-27T07:24:11.590 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:11.600+0000 7f8c66ffd700 1 -- 172.21.15.52:0/4181765950 <== mon.0 v2:172.21.15.52:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 1}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7f8c600a9460 con 0x7f8c70107de0 2024-05-27T07:24:11.590 INFO:teuthology.orchestra.run.smithi052.stdout:55834574943 2024-05-27T07:24:11.593 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:11.603+0000 7f8c781ec700 1 -- 172.21.15.52:0/4181765950 >> [v2:172.21.15.52:6800/3513075805,v1:172.21.15.52:6801/3513075805] conn(0x7f8c5c0b2990 msgr2=0x7f8c5c0b4ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:24:11.593 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:11.603+0000 7f8c781ec700 1 --2- 172.21.15.52:0/4181765950 >> [v2:172.21.15.52:6800/3513075805,v1:172.21.15.52:6801/3513075805] conn(0x7f8c5c0b2990 0x7f8c5c0b4ea0 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7f8c6c009550 tx=0x7f8c6c0091b0 comp rx=0 tx=0).stop 2024-05-27T07:24:11.593 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:11.603+0000 7f8c781ec700 1 -- 172.21.15.52:0/4181765950 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f8c70107de0 msgr2=0x7f8c7019c1c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:24:11.593 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:11.603+0000 7f8c781ec700 1 --2- 172.21.15.52:0/4181765950 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f8c70107de0 0x7f8c7019c1c0 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 crypto rx=0x7f8c6000d950 tx=0x7f8c6000dc60 comp rx=0 tx=0).stop 2024-05-27T07:24:11.593 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:11.603+0000 7f8c781ec700 1 -- 172.21.15.52:0/4181765950 shutdown_connections 2024-05-27T07:24:11.593 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:11.603+0000 7f8c781ec700 1 --2- 172.21.15.52:0/4181765950 >> [v2:172.21.15.52:6800/3513075805,v1:172.21.15.52:6801/3513075805] conn(0x7f8c5c0b2990 0x7f8c5c0b4ea0 unknown :-1 s=CLOSED pgs=189 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:24:11.593 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:11.603+0000 7f8c781ec700 1 --2- 172.21.15.52:0/4181765950 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f8c70108730 0x7f8c7019c740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:24:11.593 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:11.603+0000 7f8c781ec700 1 --2- 172.21.15.52:0/4181765950 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f8c701034b0 0x7f8c7019bc80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:24:11.593 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:11.603+0000 7f8c781ec700 1 --2- 172.21.15.52:0/4181765950 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f8c70107de0 0x7f8c7019c1c0 unknown :-1 s=CLOSED pgs=277 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:24:11.593 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:11.603+0000 7f8c781ec700 1 -- 172.21.15.52:0/4181765950 >> 172.21.15.52:0/4181765950 conn(0x7f8c700fedb0 msgr2=0x7f8c70107550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:24:11.594 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:11.603+0000 7f8c781ec700 1 -- 172.21.15.52:0/4181765950 shutdown_connections 2024-05-27T07:24:11.594 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:11.603+0000 7f8c781ec700 1 -- 172.21.15.52:0/4181765950 wait complete. 2024-05-27T07:24:11.788 INFO:tasks.cephadm.ceph_manager.ceph:need seq 137438953549 got 137438953551 for osd.4 2024-05-27T07:24:11.789 DEBUG:teuthology.parallel:result is None 2024-05-27T07:24:12.102 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:24:11 smithi173 ceph-mon[37000]: from='client.? 172.21.15.52:0/4181765950' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 1}]: dispatch 2024-05-27T07:24:12.190 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:24:11 smithi105 ceph-mon[35977]: from='client.? 172.21.15.52:0/4181765950' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 1}]: dispatch 2024-05-27T07:24:12.229 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:24:11 smithi052 ceph-mon[31955]: from='client.? 172.21.15.52:0/4181765950' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 1}]: dispatch 2024-05-27T07:24:12.979 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:24:12 smithi052 ceph-mon[31955]: pgmap v371: 129 pgs: 129 active+clean; 647 KiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-05-27T07:24:13.003 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:24:12 smithi173 ceph-mon[37000]: pgmap v371: 129 pgs: 129 active+clean; 647 KiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-05-27T07:24:13.098 INFO:teuthology.orchestra.run.smithi052.stderr:Inferring config /var/lib/ceph/73df2636-1bf8-11ef-bc9b-c7b262605968/mon.a/config 2024-05-27T07:24:13.099 INFO:teuthology.orchestra.run.smithi052.stderr:Inferring config /var/lib/ceph/73df2636-1bf8-11ef-bc9b-c7b262605968/mon.a/config 2024-05-27T07:24:13.190 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:24:12 smithi105 ceph-mon[35977]: pgmap v371: 129 pgs: 129 active+clean; 647 KiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-05-27T07:24:14.870 INFO:tasks.cephadm.ceph_manager.ceph:need seq 55834574941 got 55834574943 for osd.1 2024-05-27T07:24:14.871 DEBUG:teuthology.parallel:result is None 2024-05-27T07:24:14.979 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:24:14 smithi052 ceph-mon[31955]: pgmap v372: 129 pgs: 129 active+clean; 647 KiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 9 op/s 2024-05-27T07:24:15.062 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:15.072+0000 7f485c145700 1 -- 172.21.15.52:0/1495083388 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f4854103380 msgr2=0x7f4854113000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:24:15.063 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:15.072+0000 7f485c145700 1 --2- 172.21.15.52:0/1495083388 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f4854103380 0x7f4854113000 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 crypto rx=0x7f484c00b0a0 tx=0x7f484c00b3b0 comp rx=0 tx=0).stop 2024-05-27T07:24:15.063 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:15.072+0000 7f485c145700 1 -- 172.21.15.52:0/1495083388 shutdown_connections 2024-05-27T07:24:15.063 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:15.072+0000 7f485c145700 1 --2- 172.21.15.52:0/1495083388 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f4854103380 0x7f4854113000 unknown :-1 s=CLOSED pgs=278 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:24:15.063 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:15.072+0000 7f485c145700 1 --2- 172.21.15.52:0/1495083388 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f4854105210 0x7f4854105620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:24:15.063 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:15.072+0000 7f485c145700 1 --2- 172.21.15.52:0/1495083388 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f4854102900 0x7f4854102d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:24:15.063 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:15.072+0000 7f485c145700 1 -- 172.21.15.52:0/1495083388 >> 172.21.15.52:0/1495083388 conn(0x7f48540fe6e0 msgr2=0x7f4854100b00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:24:15.063 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:15.072+0000 7f485c145700 1 -- 172.21.15.52:0/1495083388 shutdown_connections 2024-05-27T07:24:15.063 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:15.073+0000 7f485c145700 1 -- 172.21.15.52:0/1495083388 wait complete. 2024-05-27T07:24:15.064 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:15.073+0000 7f485c145700 1 Processor -- start 2024-05-27T07:24:15.064 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:15.073+0000 7f485c145700 1 -- start start 2024-05-27T07:24:15.064 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:15.073+0000 7f485c145700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f4854102900 0x7f485419d100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:24:15.064 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:15.074+0000 7f485c145700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f4854103380 0x7f485419d6a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:24:15.064 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:15.074+0000 7f485c145700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f4854105210 0x7f48541a1930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:24:15.064 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:15.074+0000 7f485c145700 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x7f4854116710 con 0x7f4854103380 2024-05-27T07:24:15.064 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:15.074+0000 7f485c145700 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f4854116590 con 0x7f4854105210 2024-05-27T07:24:15.064 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:15.074+0000 7f485c145700 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f4854116890 con 0x7f4854102900 2024-05-27T07:24:15.065 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:15.074+0000 7f48596e0700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f4854103380 0x7f485419d6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:24:15.065 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:15.074+0000 7f48596e0700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f4854103380 0x7f485419d6a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.52:60890/0 (socket says 172.21.15.52:60890) 2024-05-27T07:24:15.065 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:15.074+0000 7f48596e0700 1 -- 172.21.15.52:0/1212361776 learned_addr learned my addr 172.21.15.52:0/1212361776 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-05-27T07:24:15.065 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:15.074+0000 7f4859ee1700 1 --2- 172.21.15.52:0/1212361776 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f4854102900 0x7f485419d100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:24:15.065 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:15.074+0000 7f48596e0700 1 -- 172.21.15.52:0/1212361776 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f4854102900 msgr2=0x7f485419d100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:24:15.065 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:15.074+0000 7f48596e0700 1 --2- 172.21.15.52:0/1212361776 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f4854102900 0x7f485419d100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:24:15.065 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:15.074+0000 7f48596e0700 1 -- 172.21.15.52:0/1212361776 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f4854105210 msgr2=0x7f48541a1930 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-05-27T07:24:15.065 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:15.074+0000 7f48596e0700 1 --2- 172.21.15.52:0/1212361776 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f4854105210 0x7f48541a1930 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:24:15.065 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:15.074+0000 7f48596e0700 1 -- 172.21.15.52:0/1212361776 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f48541a1fd0 con 0x7f4854103380 2024-05-27T07:24:15.066 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:15.075+0000 7f48596e0700 1 --2- 172.21.15.52:0/1212361776 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f4854103380 0x7f485419d6a0 secure :-1 s=READY pgs=279 cs=0 l=1 rev1=1 crypto rx=0x7f484400e940 tx=0x7f484400ed00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:24:15.066 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:15.075+0000 7f484affd700 1 -- 172.21.15.52:0/1212361776 <== mon.0 v2:172.21.15.52:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f484400cbd0 con 0x7f4854103380 2024-05-27T07:24:15.066 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:15.075+0000 7f485c145700 1 -- 172.21.15.52:0/1212361776 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f48541a22c0 con 0x7f4854103380 2024-05-27T07:24:15.066 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:15.075+0000 7f485c145700 1 -- 172.21.15.52:0/1212361776 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f48541a9be0 con 0x7f4854103380 2024-05-27T07:24:15.066 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:15.075+0000 7f484affd700 1 -- 172.21.15.52:0/1212361776 <== mon.0 v2:172.21.15.52:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f48440044d0 con 0x7f4854103380 2024-05-27T07:24:15.066 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:15.075+0000 7f484affd700 1 -- 172.21.15.52:0/1212361776 <== mon.0 v2:172.21.15.52:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4844003df0 con 0x7f4854103380 2024-05-27T07:24:15.067 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:15.076+0000 7f485c145700 1 -- 172.21.15.52:0/1212361776 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f48540512b0 con 0x7f4854103380 2024-05-27T07:24:15.069 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:15.079+0000 7f484affd700 1 -- 172.21.15.52:0/1212361776 <== mon.0 v2:172.21.15.52:3300/0 4 ==== mgrmap(e 17) ==== 149703+0+0 (secure 0 0 0) 0x7f4844004000 con 0x7f4854103380 2024-05-27T07:24:15.070 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:15.080+0000 7f484affd700 1 --2- 172.21.15.52:0/1212361776 >> [v2:172.21.15.52:6800/3513075805,v1:172.21.15.52:6801/3513075805] conn(0x7f48400b2990 0x7f48400b4ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:24:15.070 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:15.081+0000 7f4859ee1700 1 --2- 172.21.15.52:0/1212361776 >> [v2:172.21.15.52:6800/3513075805,v1:172.21.15.52:6801/3513075805] conn(0x7f48400b2990 0x7f48400b4ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:24:15.071 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:15.081+0000 7f484affd700 1 -- 172.21.15.52:0/1212361776 <== mon.0 v2:172.21.15.52:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9121+0+0 (secure 0 0 0) 0x7f48440e2a50 con 0x7f4854103380 2024-05-27T07:24:15.071 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:15.081+0000 7f4859ee1700 1 --2- 172.21.15.52:0/1212361776 >> [v2:172.21.15.52:6800/3513075805,v1:172.21.15.52:6801/3513075805] conn(0x7f48400b2990 0x7f48400b4ea0 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7f485406b100 tx=0x7f485000d4b0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:24:15.075 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:15.085+0000 7f484affd700 1 -- 172.21.15.52:0/1212361776 <== mon.0 v2:172.21.15.52:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191963 (secure 0 0 0) 0x7f48440042f0 con 0x7f4854103380 2024-05-27T07:24:15.102 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:24:14 smithi173 ceph-mon[37000]: pgmap v372: 129 pgs: 129 active+clean; 647 KiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 9 op/s 2024-05-27T07:24:15.189 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:24:14 smithi105 ceph-mon[35977]: pgmap v372: 129 pgs: 129 active+clean; 647 KiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 9 op/s 2024-05-27T07:24:15.296 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:15.306+0000 7f485c145700 1 -- 172.21.15.52:0/1212361776 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 9} v 0) -- 0x7f48541a9e10 con 0x7f4854103380 2024-05-27T07:24:15.297 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:15.307+0000 7f484affd700 1 -- 172.21.15.52:0/1212361776 <== mon.0 v2:172.21.15.52:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 9}]=0 v0) ==== 74+0+13 (secure 0 0 0) 0x7f48440aa320 con 0x7f4854103380 2024-05-27T07:24:15.297 INFO:teuthology.orchestra.run.smithi052.stdout:257698037815 2024-05-27T07:24:15.300 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:15.310+0000 7f485c145700 1 -- 172.21.15.52:0/1212361776 >> [v2:172.21.15.52:6800/3513075805,v1:172.21.15.52:6801/3513075805] conn(0x7f48400b2990 msgr2=0x7f48400b4ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:24:15.300 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:15.310+0000 7f485c145700 1 --2- 172.21.15.52:0/1212361776 >> [v2:172.21.15.52:6800/3513075805,v1:172.21.15.52:6801/3513075805] conn(0x7f48400b2990 0x7f48400b4ea0 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7f485406b100 tx=0x7f485000d4b0 comp rx=0 tx=0).stop 2024-05-27T07:24:15.300 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:15.310+0000 7f485c145700 1 -- 172.21.15.52:0/1212361776 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f4854103380 msgr2=0x7f485419d6a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:24:15.300 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:15.310+0000 7f485c145700 1 --2- 172.21.15.52:0/1212361776 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f4854103380 0x7f485419d6a0 secure :-1 s=READY pgs=279 cs=0 l=1 rev1=1 crypto rx=0x7f484400e940 tx=0x7f484400ed00 comp rx=0 tx=0).stop 2024-05-27T07:24:15.300 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:15.310+0000 7f485c145700 1 -- 172.21.15.52:0/1212361776 shutdown_connections 2024-05-27T07:24:15.300 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:15.310+0000 7f485c145700 1 --2- 172.21.15.52:0/1212361776 >> [v2:172.21.15.52:6800/3513075805,v1:172.21.15.52:6801/3513075805] conn(0x7f48400b2990 0x7f48400b4ea0 unknown :-1 s=CLOSED pgs=190 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:24:15.300 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:15.310+0000 7f485c145700 1 --2- 172.21.15.52:0/1212361776 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f4854105210 0x7f48541a1930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:24:15.301 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:15.310+0000 7f485c145700 1 --2- 172.21.15.52:0/1212361776 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f4854102900 0x7f485419d100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:24:15.301 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:15.310+0000 7f485c145700 1 --2- 172.21.15.52:0/1212361776 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f4854103380 0x7f485419d6a0 unknown :-1 s=CLOSED pgs=279 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:24:15.301 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:15.310+0000 7f485c145700 1 -- 172.21.15.52:0/1212361776 >> 172.21.15.52:0/1212361776 conn(0x7f48540fe6e0 msgr2=0x7f485410d050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:24:15.301 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:15.310+0000 7f485c145700 1 -- 172.21.15.52:0/1212361776 shutdown_connections 2024-05-27T07:24:15.301 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:15.310+0000 7f485c145700 1 -- 172.21.15.52:0/1212361776 wait complete. 2024-05-27T07:24:15.540 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:15.550+0000 7f018e1e7700 1 -- 172.21.15.52:0/3251430009 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f0188110c00 msgr2=0x7f0188113020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:24:15.541 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:15.550+0000 7f018e1e7700 1 --2- 172.21.15.52:0/3251430009 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f0188110c00 0x7f0188113020 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 crypto rx=0x7f0180009990 tx=0x7f0180009ca0 comp rx=0 tx=0).stop 2024-05-27T07:24:15.541 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:15.550+0000 7f018e1e7700 1 -- 172.21.15.52:0/3251430009 shutdown_connections 2024-05-27T07:24:15.541 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:15.550+0000 7f018e1e7700 1 --2- 172.21.15.52:0/3251430009 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f0188110c00 0x7f0188113020 unknown :-1 s=CLOSED pgs=280 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:24:15.541 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:15.550+0000 7f018e1e7700 1 --2- 172.21.15.52:0/3251430009 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f0188075ad0 0x7f0188075ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:24:15.541 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:15.550+0000 7f018e1e7700 1 --2- 172.21.15.52:0/3251430009 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f0188076420 0x7f01881106c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:24:15.541 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:15.550+0000 7f018e1e7700 1 -- 172.21.15.52:0/3251430009 >> 172.21.15.52:0/3251430009 conn(0x7f01880ffe60 msgr2=0x7f01881022b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:24:15.541 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:15.550+0000 7f018e1e7700 1 -- 172.21.15.52:0/3251430009 shutdown_connections 2024-05-27T07:24:15.541 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:15.550+0000 7f018e1e7700 1 -- 172.21.15.52:0/3251430009 wait complete. 2024-05-27T07:24:15.541 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:15.551+0000 7f018e1e7700 1 Processor -- start 2024-05-27T07:24:15.543 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:15.551+0000 7f018e1e7700 1 -- start start 2024-05-27T07:24:15.543 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:15.551+0000 7f018e1e7700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f0188075ad0 0x7f018819cff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:24:15.543 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:15.551+0000 7f018e1e7700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f0188076420 0x7f018819d590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:24:15.543 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:15.551+0000 7f018e1e7700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f0188110c00 0x7f01881a1820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:24:15.543 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:15.552+0000 7f018e1e7700 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x7f01881166e0 con 0x7f0188110c00 2024-05-27T07:24:15.543 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:15.552+0000 7f018e1e7700 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f0188116560 con 0x7f0188075ad0 2024-05-27T07:24:15.543 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:15.552+0000 7f018e1e7700 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f0188116860 con 0x7f0188076420 2024-05-27T07:24:15.543 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:15.552+0000 7f0187fff700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f0188110c00 0x7f01881a1820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:24:15.543 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:15.552+0000 7f01877fe700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f0188075ad0 0x7f018819cff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:24:15.543 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:15.552+0000 7f0186ffd700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f0188076420 0x7f018819d590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:24:15.544 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:15.552+0000 7f0187fff700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f0188110c00 0x7f01881a1820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.52:60906/0 (socket says 172.21.15.52:60906) 2024-05-27T07:24:15.544 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:15.552+0000 7f0187fff700 1 -- 172.21.15.52:0/2123843217 learned_addr learned my addr 172.21.15.52:0/2123843217 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-05-27T07:24:15.544 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:15.552+0000 7f0187fff700 1 -- 172.21.15.52:0/2123843217 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f0188076420 msgr2=0x7f018819d590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:24:15.544 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:15.552+0000 7f0187fff700 1 --2- 172.21.15.52:0/2123843217 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f0188076420 0x7f018819d590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:24:15.544 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:15.552+0000 7f0187fff700 1 -- 172.21.15.52:0/2123843217 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f0188075ad0 msgr2=0x7f018819cff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:24:15.544 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:15.552+0000 7f0187fff700 1 --2- 172.21.15.52:0/2123843217 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f0188075ad0 0x7f018819cff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:24:15.544 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:15.552+0000 7f0187fff700 1 -- 172.21.15.52:0/2123843217 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f01881a1ec0 con 0x7f0188110c00 2024-05-27T07:24:15.544 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:15.552+0000 7f0187fff700 1 --2- 172.21.15.52:0/2123843217 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f0188110c00 0x7f01881a1820 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 crypto rx=0x7f01800047b0 tx=0x7f0180005ef0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:24:15.544 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:15.552+0000 7f0184ff9700 1 -- 172.21.15.52:0/2123843217 <== mon.0 v2:172.21.15.52:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0180004830 con 0x7f0188110c00 2024-05-27T07:24:15.544 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:15.553+0000 7f0184ff9700 1 -- 172.21.15.52:0/2123843217 <== mon.0 v2:172.21.15.52:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f0180005020 con 0x7f0188110c00 2024-05-27T07:24:15.544 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:15.553+0000 7f018e1e7700 1 -- 172.21.15.52:0/2123843217 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f01881a2150 con 0x7f0188110c00 2024-05-27T07:24:15.545 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:15.553+0000 7f0184ff9700 1 -- 172.21.15.52:0/2123843217 <== mon.0 v2:172.21.15.52:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0180003cd0 con 0x7f0188110c00 2024-05-27T07:24:15.545 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:15.553+0000 7f018e1e7700 1 -- 172.21.15.52:0/2123843217 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f01881a25d0 con 0x7f0188110c00 2024-05-27T07:24:15.545 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:15.554+0000 7f018e1e7700 1 -- 172.21.15.52:0/2123843217 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0168005360 con 0x7f0188110c00 2024-05-27T07:24:15.545 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:15.556+0000 7f0184ff9700 1 -- 172.21.15.52:0/2123843217 <== mon.0 v2:172.21.15.52:3300/0 4 ==== mgrmap(e 17) ==== 149703+0+0 (secure 0 0 0) 0x7f01800049d0 con 0x7f0188110c00 2024-05-27T07:24:15.546 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:15.556+0000 7f0184ff9700 1 --2- 172.21.15.52:0/2123843217 >> [v2:172.21.15.52:6800/3513075805,v1:172.21.15.52:6801/3513075805] conn(0x7f01740b2910 0x7f01740b4e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:24:15.546 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:15.556+0000 7f01877fe700 1 --2- 172.21.15.52:0/2123843217 >> [v2:172.21.15.52:6800/3513075805,v1:172.21.15.52:6801/3513075805] conn(0x7f01740b2910 0x7f01740b4e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:24:15.546 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:15.557+0000 7f0184ff9700 1 -- 172.21.15.52:0/2123843217 <== mon.0 v2:172.21.15.52:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9121+0+0 (secure 0 0 0) 0x7f01800e3020 con 0x7f0188110c00 2024-05-27T07:24:15.547 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:15.557+0000 7f01877fe700 1 --2- 172.21.15.52:0/2123843217 >> [v2:172.21.15.52:6800/3513075805,v1:172.21.15.52:6801/3513075805] conn(0x7f01740b2910 0x7f01740b4e20 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7f0178009550 tx=0x7f01780091b0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:24:15.554 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:15.562+0000 7f0184ff9700 1 -- 172.21.15.52:0/2123843217 <== mon.0 v2:172.21.15.52:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191963 (secure 0 0 0) 0x7f0180013070 con 0x7f0188110c00 2024-05-27T07:24:15.771 INFO:teuthology.orchestra.run.smithi052.stdout:111669149782 2024-05-27T07:24:15.771 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:15.780+0000 7f018e1e7700 1 -- 172.21.15.52:0/2123843217 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 3} v 0) -- 0x7f0168005920 con 0x7f0188110c00 2024-05-27T07:24:15.771 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:15.781+0000 7f0184ff9700 1 -- 172.21.15.52:0/2123843217 <== mon.0 v2:172.21.15.52:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 3}]=0 v0) ==== 74+0+13 (secure 0 0 0) 0x7f01800aa8f0 con 0x7f0188110c00 2024-05-27T07:24:15.773 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:15.783+0000 7f018e1e7700 1 -- 172.21.15.52:0/2123843217 >> [v2:172.21.15.52:6800/3513075805,v1:172.21.15.52:6801/3513075805] conn(0x7f01740b2910 msgr2=0x7f01740b4e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:24:15.773 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:15.783+0000 7f018e1e7700 1 --2- 172.21.15.52:0/2123843217 >> [v2:172.21.15.52:6800/3513075805,v1:172.21.15.52:6801/3513075805] conn(0x7f01740b2910 0x7f01740b4e20 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7f0178009550 tx=0x7f01780091b0 comp rx=0 tx=0).stop 2024-05-27T07:24:15.773 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:15.783+0000 7f018e1e7700 1 -- 172.21.15.52:0/2123843217 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f0188110c00 msgr2=0x7f01881a1820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:24:15.773 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:15.783+0000 7f018e1e7700 1 --2- 172.21.15.52:0/2123843217 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f0188110c00 0x7f01881a1820 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 crypto rx=0x7f01800047b0 tx=0x7f0180005ef0 comp rx=0 tx=0).stop 2024-05-27T07:24:15.773 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:15.783+0000 7f018e1e7700 1 -- 172.21.15.52:0/2123843217 shutdown_connections 2024-05-27T07:24:15.773 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:15.783+0000 7f018e1e7700 1 --2- 172.21.15.52:0/2123843217 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f0188110c00 0x7f01881a1820 unknown :-1 s=CLOSED pgs=281 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:24:15.773 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:15.783+0000 7f018e1e7700 1 --2- 172.21.15.52:0/2123843217 >> [v2:172.21.15.52:6800/3513075805,v1:172.21.15.52:6801/3513075805] conn(0x7f01740b2910 0x7f01740b4e20 unknown :-1 s=CLOSED pgs=191 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:24:15.773 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:15.783+0000 7f018e1e7700 1 --2- 172.21.15.52:0/2123843217 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f0188075ad0 0x7f018819cff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:24:15.774 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:15.783+0000 7f018e1e7700 1 --2- 172.21.15.52:0/2123843217 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f0188076420 0x7f018819d590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:24:15.774 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:15.783+0000 7f018e1e7700 1 -- 172.21.15.52:0/2123843217 >> 172.21.15.52:0/2123843217 conn(0x7f01880ffe60 msgr2=0x7f01881003f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:24:15.774 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:15.783+0000 7f018e1e7700 1 -- 172.21.15.52:0/2123843217 shutdown_connections 2024-05-27T07:24:15.774 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:15.783+0000 7f018e1e7700 1 -- 172.21.15.52:0/2123843217 wait complete. 2024-05-27T07:24:15.818 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:24:15 smithi052 ceph-mon[31955]: from='client.? 172.21.15.52:0/1212361776' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 9}]: dispatch 2024-05-27T07:24:16.102 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:24:15 smithi173 ceph-mon[37000]: from='client.? 172.21.15.52:0/1212361776' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 9}]: dispatch 2024-05-27T07:24:16.190 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:24:15 smithi105 ceph-mon[35977]: from='client.? 172.21.15.52:0/1212361776' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 9}]: dispatch 2024-05-27T07:24:16.433 INFO:tasks.cephadm.ceph_manager.ceph:need seq 111669149777 got 111669149782 for osd.3 2024-05-27T07:24:16.433 DEBUG:teuthology.parallel:result is None 2024-05-27T07:24:16.979 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:24:16 smithi052 ceph-mon[31955]: pgmap v373: 129 pgs: 129 active+clean; 647 KiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 10 op/s 2024-05-27T07:24:16.979 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:24:16 smithi052 ceph-mon[31955]: from='client.? 172.21.15.52:0/2123843217' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 3}]: dispatch 2024-05-27T07:24:17.102 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:24:16 smithi173 ceph-mon[37000]: pgmap v373: 129 pgs: 129 active+clean; 647 KiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 10 op/s 2024-05-27T07:24:17.102 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:24:16 smithi173 ceph-mon[37000]: from='client.? 172.21.15.52:0/2123843217' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 3}]: dispatch 2024-05-27T07:24:17.190 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:24:16 smithi105 ceph-mon[35977]: pgmap v373: 129 pgs: 129 active+clean; 647 KiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 10 op/s 2024-05-27T07:24:17.190 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:24:16 smithi105 ceph-mon[35977]: from='client.? 172.21.15.52:0/2123843217' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 3}]: dispatch 2024-05-27T07:24:19.036 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:24:18 smithi052 ceph-mon[31955]: pgmap v374: 129 pgs: 129 active+clean; 647 KiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 9 op/s 2024-05-27T07:24:19.063 INFO:tasks.cephadm.ceph_manager.ceph:need seq 257698037810 got 257698037815 for osd.9 2024-05-27T07:24:19.063 DEBUG:teuthology.parallel:result is None 2024-05-27T07:24:19.101 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:24:18 smithi173 ceph-mon[37000]: pgmap v374: 129 pgs: 129 active+clean; 647 KiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 9 op/s 2024-05-27T07:24:19.190 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:24:18 smithi105 ceph-mon[35977]: pgmap v374: 129 pgs: 129 active+clean; 647 KiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 9 op/s 2024-05-27T07:24:20.503 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:20.511+0000 7f40854f2700 1 -- 172.21.15.52:0/950953121 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f4080109840 msgr2=0x7f40801111a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:24:20.503 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:20.511+0000 7f40854f2700 1 --2- 172.21.15.52:0/950953121 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f4080109840 0x7f40801111a0 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 crypto rx=0x7f407800b0d0 tx=0x7f407800b3e0 comp rx=0 tx=0).stop 2024-05-27T07:24:20.503 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:20.511+0000 7f40854f2700 1 -- 172.21.15.52:0/950953121 shutdown_connections 2024-05-27T07:24:20.503 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:20.511+0000 7f40854f2700 1 --2- 172.21.15.52:0/950953121 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f4080109840 0x7f40801111a0 unknown :-1 s=CLOSED pgs=282 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:24:20.504 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:20.511+0000 7f40854f2700 1 --2- 172.21.15.52:0/950953121 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f40801045c0 0x7f40801049d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:24:20.504 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:20.511+0000 7f40854f2700 1 --2- 172.21.15.52:0/950953121 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f4080108ef0 0x7f4080109300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:24:20.504 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:20.511+0000 7f40854f2700 1 -- 172.21.15.52:0/950953121 >> 172.21.15.52:0/950953121 conn(0x7f40800ffe90 msgr2=0x7f40801022e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:24:20.504 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:20.511+0000 7f40854f2700 1 -- 172.21.15.52:0/950953121 shutdown_connections 2024-05-27T07:24:20.504 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:20.511+0000 7f40854f2700 1 -- 172.21.15.52:0/950953121 wait complete. 2024-05-27T07:24:20.504 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:20.512+0000 7f40854f2700 1 Processor -- start 2024-05-27T07:24:20.504 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:20.512+0000 7f40854f2700 1 -- start start 2024-05-27T07:24:20.506 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:20.513+0000 7f40854f2700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f40801045c0 0x7f408019ada0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:24:20.506 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:20.513+0000 7f40854f2700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f4080108ef0 0x7f408019b340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:24:20.506 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:20.513+0000 7f40854f2700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f4080109840 0x7f408019f580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:24:20.506 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:20.513+0000 7f40854f2700 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x7f4080114470 con 0x7f4080109840 2024-05-27T07:24:20.506 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:20.513+0000 7f40854f2700 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f40801142f0 con 0x7f4080108ef0 2024-05-27T07:24:20.506 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:20.513+0000 7f40854f2700 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f40801145f0 con 0x7f40801045c0 2024-05-27T07:24:20.506 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:20.513+0000 7f407f7fe700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f4080109840 0x7f408019f580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:24:20.506 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:20.513+0000 7f407f7fe700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f4080109840 0x7f408019f580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.52:60922/0 (socket says 172.21.15.52:60922) 2024-05-27T07:24:20.506 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:20.513+0000 7f407f7fe700 1 -- 172.21.15.52:0/1877688362 learned_addr learned my addr 172.21.15.52:0/1877688362 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-05-27T07:24:20.506 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:20.513+0000 7f407effd700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f40801045c0 0x7f408019ada0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:24:20.506 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:20.513+0000 7f407e7fc700 1 --2- 172.21.15.52:0/1877688362 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f4080108ef0 0x7f408019b340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:24:20.507 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:20.513+0000 7f407effd700 1 -- 172.21.15.52:0/1877688362 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f4080108ef0 msgr2=0x7f408019b340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:24:20.507 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:20.513+0000 7f407effd700 1 --2- 172.21.15.52:0/1877688362 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f4080108ef0 0x7f408019b340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:24:20.507 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:20.513+0000 7f407effd700 1 -- 172.21.15.52:0/1877688362 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f4080109840 msgr2=0x7f408019f580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:24:20.507 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:20.513+0000 7f407effd700 1 --2- 172.21.15.52:0/1877688362 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f4080109840 0x7f408019f580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:24:20.507 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:20.513+0000 7f407effd700 1 -- 172.21.15.52:0/1877688362 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f408019fb70 con 0x7f40801045c0 2024-05-27T07:24:20.507 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:20.513+0000 7f407f7fe700 1 --2- 172.21.15.52:0/1877688362 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f4080109840 0x7f408019f580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-27T07:24:20.507 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:20.514+0000 7f407effd700 1 --2- 172.21.15.52:0/1877688362 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f40801045c0 0x7f408019ada0 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f407000c970 tx=0x7f407000cc80 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:24:20.507 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:20.514+0000 7f4067fff700 1 -- 172.21.15.52:0/1877688362 <== mon.1 v2:172.21.15.173:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4070014070 con 0x7f40801045c0 2024-05-27T07:24:20.507 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:20.514+0000 7f40854f2700 1 -- 172.21.15.52:0/1877688362 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f408019fe60 con 0x7f40801045c0 2024-05-27T07:24:20.507 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:20.514+0000 7f4067fff700 1 -- 172.21.15.52:0/1877688362 <== mon.1 v2:172.21.15.173:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f4070004440 con 0x7f40801045c0 2024-05-27T07:24:20.507 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:20.514+0000 7f40854f2700 1 -- 172.21.15.52:0/1877688362 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f40801a7830 con 0x7f40801045c0 2024-05-27T07:24:20.507 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:20.514+0000 7f4067fff700 1 -- 172.21.15.52:0/1877688362 <== mon.1 v2:172.21.15.173:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4070007a30 con 0x7f40801045c0 2024-05-27T07:24:20.508 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:20.516+0000 7f40854f2700 1 -- 172.21.15.52:0/1877688362 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f406c005330 con 0x7f40801045c0 2024-05-27T07:24:20.508 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:20.517+0000 7f4067fff700 1 -- 172.21.15.52:0/1877688362 <== mon.1 v2:172.21.15.173:3300/0 4 ==== mgrmap(e 17) ==== 149703+0+0 (secure 0 0 0) 0x7f4070017070 con 0x7f40801045c0 2024-05-27T07:24:20.509 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:20.518+0000 7f4067fff700 1 --2- 172.21.15.52:0/1877688362 >> [v2:172.21.15.52:6800/3513075805,v1:172.21.15.52:6801/3513075805] conn(0x7f40680b2910 0x7f40680b4e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:24:20.509 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:20.518+0000 7f407e7fc700 1 --2- 172.21.15.52:0/1877688362 >> [v2:172.21.15.52:6800/3513075805,v1:172.21.15.52:6801/3513075805] conn(0x7f40680b2910 0x7f40680b4e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:24:20.509 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:20.518+0000 7f4067fff700 1 -- 172.21.15.52:0/1877688362 <== mon.1 v2:172.21.15.173:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9121+0+0 (secure 0 0 0) 0x7f40700e2270 con 0x7f40801045c0 2024-05-27T07:24:20.509 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:20.519+0000 7f407e7fc700 1 --2- 172.21.15.52:0/1877688362 >> [v2:172.21.15.52:6800/3513075805,v1:172.21.15.52:6801/3513075805] conn(0x7f40680b2910 0x7f40680b4e20 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7f4080103430 tx=0x7f407400b410 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:24:20.513 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:20.522+0000 7f4067fff700 1 -- 172.21.15.52:0/1877688362 <== mon.1 v2:172.21.15.173:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191963 (secure 0 0 0) 0x7f407001e030 con 0x7f40801045c0 2024-05-27T07:24:20.737 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:20.746+0000 7f40854f2700 1 -- 172.21.15.52:0/1877688362 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 0} v 0) -- 0x7f406c0058f0 con 0x7f40801045c0 2024-05-27T07:24:20.737 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:20.746+0000 7f4067fff700 1 -- 172.21.15.52:0/1877688362 <== mon.1 v2:172.21.15.173:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 0}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7f40700a9a50 con 0x7f40801045c0 2024-05-27T07:24:20.738 INFO:teuthology.orchestra.run.smithi052.stdout:34359738470 2024-05-27T07:24:20.740 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:20.749+0000 7f40854f2700 1 -- 172.21.15.52:0/1877688362 >> [v2:172.21.15.52:6800/3513075805,v1:172.21.15.52:6801/3513075805] conn(0x7f40680b2910 msgr2=0x7f40680b4e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:24:20.740 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:20.749+0000 7f40854f2700 1 --2- 172.21.15.52:0/1877688362 >> [v2:172.21.15.52:6800/3513075805,v1:172.21.15.52:6801/3513075805] conn(0x7f40680b2910 0x7f40680b4e20 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7f4080103430 tx=0x7f407400b410 comp rx=0 tx=0).stop 2024-05-27T07:24:20.740 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:20.749+0000 7f40854f2700 1 -- 172.21.15.52:0/1877688362 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f40801045c0 msgr2=0x7f408019ada0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:24:20.740 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:20.749+0000 7f40854f2700 1 --2- 172.21.15.52:0/1877688362 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f40801045c0 0x7f408019ada0 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f407000c970 tx=0x7f407000cc80 comp rx=0 tx=0).stop 2024-05-27T07:24:20.740 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:20.749+0000 7f40854f2700 1 -- 172.21.15.52:0/1877688362 shutdown_connections 2024-05-27T07:24:20.740 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:20.749+0000 7f40854f2700 1 --2- 172.21.15.52:0/1877688362 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f4080109840 0x7f408019f580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:24:20.740 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:20.749+0000 7f40854f2700 1 --2- 172.21.15.52:0/1877688362 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f40801045c0 0x7f408019ada0 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:24:20.740 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:20.749+0000 7f40854f2700 1 --2- 172.21.15.52:0/1877688362 >> [v2:172.21.15.52:6800/3513075805,v1:172.21.15.52:6801/3513075805] conn(0x7f40680b2910 0x7f40680b4e20 unknown :-1 s=CLOSED pgs=192 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:24:20.740 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:20.749+0000 7f40854f2700 1 --2- 172.21.15.52:0/1877688362 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f4080108ef0 0x7f408019b340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:24:20.740 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:20.749+0000 7f40854f2700 1 -- 172.21.15.52:0/1877688362 >> 172.21.15.52:0/1877688362 conn(0x7f40800ffe90 msgr2=0x7f4080108630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:24:20.740 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:20.749+0000 7f40854f2700 1 -- 172.21.15.52:0/1877688362 shutdown_connections 2024-05-27T07:24:20.741 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:20.749+0000 7f40854f2700 1 -- 172.21.15.52:0/1877688362 wait complete. 2024-05-27T07:24:21.102 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:24:20 smithi173 ceph-mon[37000]: pgmap v375: 129 pgs: 129 active+clean; 647 KiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 9 op/s 2024-05-27T07:24:21.102 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:24:20 smithi173 ceph-mon[37000]: from='client.? 172.21.15.52:0/1877688362' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 0}]: dispatch 2024-05-27T07:24:21.130 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:24:20 smithi052 ceph-mon[31955]: pgmap v375: 129 pgs: 129 active+clean; 647 KiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 9 op/s 2024-05-27T07:24:21.131 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:24:20 smithi052 ceph-mon[31955]: from='client.? 172.21.15.52:0/1877688362' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 0}]: dispatch 2024-05-27T07:24:21.190 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:24:20 smithi105 ceph-mon[35977]: pgmap v375: 129 pgs: 129 active+clean; 647 KiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 9 op/s 2024-05-27T07:24:21.190 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:24:20 smithi105 ceph-mon[35977]: from='client.? 172.21.15.52:0/1877688362' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 0}]: dispatch 2024-05-27T07:24:22.920 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:24:22 smithi052 ceph-mon[31955]: pgmap v376: 129 pgs: 129 active+clean; 647 KiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-05-27T07:24:22.921 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:24:22 smithi052 ceph-mon[31955]: from='client.? 172.21.15.52:0/399148352' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 6}]: dispatch 2024-05-27T07:24:23.102 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:24:22 smithi173 ceph-mon[37000]: pgmap v376: 129 pgs: 129 active+clean; 647 KiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-05-27T07:24:23.102 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:24:22 smithi173 ceph-mon[37000]: from='client.? 172.21.15.52:0/399148352' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 6}]: dispatch 2024-05-27T07:24:23.190 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:24:22 smithi105 ceph-mon[35977]: pgmap v376: 129 pgs: 129 active+clean; 647 KiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-05-27T07:24:23.190 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:24:22 smithi105 ceph-mon[35977]: from='client.? 172.21.15.52:0/399148352' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 6}]: dispatch 2024-05-27T07:24:23.249 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:21.936+0000 7f6e75cb4700 1 -- 172.21.15.52:0/906974953 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f6e70103930 msgr2=0x7f6e70103d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:24:23.250 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:21.936+0000 7f6e75cb4700 1 --2- 172.21.15.52:0/906974953 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f6e70103930 0x7f6e70103d40 secure :-1 s=READY pgs=283 cs=0 l=1 rev1=1 crypto rx=0x7f6e64009990 tx=0x7f6e64009ca0 comp rx=0 tx=0).stop 2024-05-27T07:24:23.250 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:21.936+0000 7f6e75cb4700 1 -- 172.21.15.52:0/906974953 shutdown_connections 2024-05-27T07:24:23.250 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:21.936+0000 7f6e75cb4700 1 --2- 172.21.15.52:0/906974953 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f6e70104280 0x7f6e701100f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:24:23.250 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:21.936+0000 7f6e75cb4700 1 --2- 172.21.15.52:0/906974953 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f6e70109350 0x7f6e70109760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:24:23.250 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:21.936+0000 7f6e75cb4700 1 --2- 172.21.15.52:0/906974953 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f6e70103930 0x7f6e70103d40 unknown :-1 s=CLOSED pgs=283 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:24:23.250 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:21.937+0000 7f6e75cb4700 1 -- 172.21.15.52:0/906974953 >> 172.21.15.52:0/906974953 conn(0x7f6e700fe4d0 msgr2=0x7f6e70100920 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:24:23.250 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:21.937+0000 7f6e75cb4700 1 -- 172.21.15.52:0/906974953 shutdown_connections 2024-05-27T07:24:23.250 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:21.937+0000 7f6e75cb4700 1 -- 172.21.15.52:0/906974953 wait complete. 2024-05-27T07:24:23.250 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:21.937+0000 7f6e75cb4700 1 Processor -- start 2024-05-27T07:24:23.250 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:21.937+0000 7f6e75cb4700 1 -- start start 2024-05-27T07:24:23.251 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:21.938+0000 7f6e75cb4700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f6e70103930 0x7f6e700fe9b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:24:23.251 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:21.938+0000 7f6e75cb4700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f6e70104280 0x7f6e700feef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:24:23.251 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:21.938+0000 7f6e75cb4700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f6e70109350 0x7f6e700ff430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:24:23.251 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:21.938+0000 7f6e75cb4700 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x7f6e70113480 con 0x7f6e70104280 2024-05-27T07:24:23.251 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:21.938+0000 7f6e75cb4700 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f6e70113300 con 0x7f6e70103930 2024-05-27T07:24:23.251 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:21.938+0000 7f6e75cb4700 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f6e70113600 con 0x7f6e70109350 2024-05-27T07:24:23.251 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:21.938+0000 7f6e6effd700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f6e70104280 0x7f6e700feef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:24:23.251 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:21.938+0000 7f6e6effd700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f6e70104280 0x7f6e700feef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.52:39964/0 (socket says 172.21.15.52:39964) 2024-05-27T07:24:23.251 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:21.938+0000 7f6e6effd700 1 -- 172.21.15.52:0/399148352 learned_addr learned my addr 172.21.15.52:0/399148352 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-05-27T07:24:23.251 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:21.938+0000 7f6e6f7fe700 1 --2- 172.21.15.52:0/399148352 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f6e70103930 0x7f6e700fe9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:24:23.251 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:21.938+0000 7f6e6ffff700 1 --2- 172.21.15.52:0/399148352 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f6e70109350 0x7f6e700ff430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:24:23.251 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:21.938+0000 7f6e6effd700 1 -- 172.21.15.52:0/399148352 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f6e70109350 msgr2=0x7f6e700ff430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:24:23.251 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:21.938+0000 7f6e6effd700 1 --2- 172.21.15.52:0/399148352 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f6e70109350 0x7f6e700ff430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:24:23.252 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:21.938+0000 7f6e6effd700 1 -- 172.21.15.52:0/399148352 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f6e70103930 msgr2=0x7f6e700fe9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:24:23.252 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:21.938+0000 7f6e6effd700 1 --2- 172.21.15.52:0/399148352 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f6e70103930 0x7f6e700fe9b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:24:23.252 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:21.938+0000 7f6e6effd700 1 -- 172.21.15.52:0/399148352 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6e701a9290 con 0x7f6e70104280 2024-05-27T07:24:23.252 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:21.939+0000 7f6e6effd700 1 --2- 172.21.15.52:0/399148352 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f6e70104280 0x7f6e700feef0 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 crypto rx=0x7f6e640057c0 tx=0x7f6e64005ef0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:24:23.252 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:21.939+0000 7f6e6cff9700 1 -- 172.21.15.52:0/399148352 <== mon.0 v2:172.21.15.52:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6e64027070 con 0x7f6e70104280 2024-05-27T07:24:23.252 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:21.939+0000 7f6e75cb4700 1 -- 172.21.15.52:0/399148352 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6e701a9520 con 0x7f6e70104280 2024-05-27T07:24:23.252 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:21.939+0000 7f6e6cff9700 1 -- 172.21.15.52:0/399148352 <== mon.0 v2:172.21.15.52:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f6e6400fb20 con 0x7f6e70104280 2024-05-27T07:24:23.252 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:21.939+0000 7f6e6cff9700 1 -- 172.21.15.52:0/399148352 <== mon.0 v2:172.21.15.52:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6e64022bb0 con 0x7f6e70104280 2024-05-27T07:24:23.252 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:21.939+0000 7f6e75cb4700 1 -- 172.21.15.52:0/399148352 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6e701a9a50 con 0x7f6e70104280 2024-05-27T07:24:23.252 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:21.940+0000 7f6e75cb4700 1 -- 172.21.15.52:0/399148352 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6e700512b0 con 0x7f6e70104280 2024-05-27T07:24:23.252 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:21.942+0000 7f6e6cff9700 1 -- 172.21.15.52:0/399148352 <== mon.0 v2:172.21.15.52:3300/0 4 ==== mgrmap(e 17) ==== 149703+0+0 (secure 0 0 0) 0x7f6e6402a070 con 0x7f6e70104280 2024-05-27T07:24:23.252 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:21.943+0000 7f6e6cff9700 1 --2- 172.21.15.52:0/399148352 >> [v2:172.21.15.52:6800/3513075805,v1:172.21.15.52:6801/3513075805] conn(0x7f6e580b2910 0x7f6e580b4e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:24:23.253 INFO:teuthology.orchestra.run.smithi052.stdout:184683593800 2024-05-27T07:24:23.253 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:21.943+0000 7f6e6f7fe700 1 --2- 172.21.15.52:0/399148352 >> [v2:172.21.15.52:6800/3513075805,v1:172.21.15.52:6801/3513075805] conn(0x7f6e580b2910 0x7f6e580b4e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:24:23.253 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:21.943+0000 7f6e6cff9700 1 -- 172.21.15.52:0/399148352 <== mon.0 v2:172.21.15.52:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9121+0+0 (secure 0 0 0) 0x7f6e640e3860 con 0x7f6e70104280 2024-05-27T07:24:23.253 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:21.943+0000 7f6e6f7fe700 1 --2- 172.21.15.52:0/399148352 >> [v2:172.21.15.52:6800/3513075805,v1:172.21.15.52:6801/3513075805] conn(0x7f6e580b2910 0x7f6e580b4e20 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7f6e7006bd20 tx=0x7f6e6000a380 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:24:23.253 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:21.947+0000 7f6e6cff9700 1 -- 172.21.15.52:0/399148352 <== mon.0 v2:172.21.15.52:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191963 (secure 0 0 0) 0x7f6e64013070 con 0x7f6e70104280 2024-05-27T07:24:23.253 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:22.167+0000 7f6e75cb4700 1 -- 172.21.15.52:0/399148352 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 6} v 0) -- 0x7f6e7002e5e0 con 0x7f6e70104280 2024-05-27T07:24:23.253 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:22.168+0000 7f6e6cff9700 1 -- 172.21.15.52:0/399148352 <== mon.0 v2:172.21.15.52:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 6}]=0 v0) ==== 74+0+13 (secure 0 0 0) 0x7f6e640ab130 con 0x7f6e70104280 2024-05-27T07:24:23.253 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:22.170+0000 7f6e75cb4700 1 -- 172.21.15.52:0/399148352 >> [v2:172.21.15.52:6800/3513075805,v1:172.21.15.52:6801/3513075805] conn(0x7f6e580b2910 msgr2=0x7f6e580b4e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:24:23.254 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:22.170+0000 7f6e75cb4700 1 --2- 172.21.15.52:0/399148352 >> [v2:172.21.15.52:6800/3513075805,v1:172.21.15.52:6801/3513075805] conn(0x7f6e580b2910 0x7f6e580b4e20 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7f6e7006bd20 tx=0x7f6e6000a380 comp rx=0 tx=0).stop 2024-05-27T07:24:23.254 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:22.170+0000 7f6e75cb4700 1 -- 172.21.15.52:0/399148352 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f6e70104280 msgr2=0x7f6e700feef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:24:23.254 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:22.170+0000 7f6e75cb4700 1 --2- 172.21.15.52:0/399148352 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f6e70104280 0x7f6e700feef0 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 crypto rx=0x7f6e640057c0 tx=0x7f6e64005ef0 comp rx=0 tx=0).stop 2024-05-27T07:24:23.254 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:22.170+0000 7f6e75cb4700 1 -- 172.21.15.52:0/399148352 shutdown_connections 2024-05-27T07:24:23.254 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:22.170+0000 7f6e75cb4700 1 --2- 172.21.15.52:0/399148352 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f6e70109350 0x7f6e700ff430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:24:23.254 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:22.170+0000 7f6e75cb4700 1 --2- 172.21.15.52:0/399148352 >> [v2:172.21.15.52:6800/3513075805,v1:172.21.15.52:6801/3513075805] conn(0x7f6e580b2910 0x7f6e580b4e20 unknown :-1 s=CLOSED pgs=193 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:24:23.254 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:22.170+0000 7f6e75cb4700 1 --2- 172.21.15.52:0/399148352 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f6e70103930 0x7f6e700fe9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:24:23.254 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:22.170+0000 7f6e75cb4700 1 --2- 172.21.15.52:0/399148352 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f6e70104280 0x7f6e700feef0 unknown :-1 s=CLOSED pgs=284 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:24:23.254 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:22.170+0000 7f6e75cb4700 1 -- 172.21.15.52:0/399148352 >> 172.21.15.52:0/399148352 conn(0x7f6e700fe4d0 msgr2=0x7f6e70102c80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:24:23.254 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:22.171+0000 7f6e75cb4700 1 -- 172.21.15.52:0/399148352 shutdown_connections 2024-05-27T07:24:23.254 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:22.171+0000 7f6e75cb4700 1 -- 172.21.15.52:0/399148352 wait complete. 2024-05-27T07:24:23.836 INFO:tasks.cephadm.ceph_manager.ceph:need seq 184683593796 got 184683593800 for osd.6 2024-05-27T07:24:23.836 DEBUG:teuthology.parallel:result is None 2024-05-27T07:24:24.238 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:24.245+0000 7f08f882b700 1 -- 172.21.15.52:0/532575557 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f08f4108730 msgr2=0x7f08f4110090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:24:24.238 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:24.245+0000 7f08f882b700 1 --2- 172.21.15.52:0/532575557 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f08f4108730 0x7f08f4110090 secure :-1 s=READY pgs=285 cs=0 l=1 rev1=1 crypto rx=0x7f08ec00b0d0 tx=0x7f08ec00b3e0 comp rx=0 tx=0).stop 2024-05-27T07:24:24.239 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:24.245+0000 7f08f882b700 1 -- 172.21.15.52:0/532575557 shutdown_connections 2024-05-27T07:24:24.239 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:24.245+0000 7f08f882b700 1 --2- 172.21.15.52:0/532575557 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f08f4108730 0x7f08f4110090 unknown :-1 s=CLOSED pgs=285 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:24:24.239 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:24.245+0000 7f08f882b700 1 --2- 172.21.15.52:0/532575557 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f08f41034b0 0x7f08f41038c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:24:24.239 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:24.245+0000 7f08f882b700 1 --2- 172.21.15.52:0/532575557 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f08f4107de0 0x7f08f41081f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:24:24.239 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:24.245+0000 7f08f882b700 1 -- 172.21.15.52:0/532575557 >> 172.21.15.52:0/532575557 conn(0x7f08f40fedb0 msgr2=0x7f08f41011d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:24:24.239 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:24.245+0000 7f08f882b700 1 -- 172.21.15.52:0/532575557 shutdown_connections 2024-05-27T07:24:24.239 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:24.245+0000 7f08f882b700 1 -- 172.21.15.52:0/532575557 wait complete. 2024-05-27T07:24:24.239 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:24.246+0000 7f08f882b700 1 Processor -- start 2024-05-27T07:24:24.239 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:24.246+0000 7f08f882b700 1 -- start start 2024-05-27T07:24:24.239 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:24.246+0000 7f08f882b700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f08f41034b0 0x7f08f419ae60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:24:24.239 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:24.246+0000 7f08f882b700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f08f4107de0 0x7f08f419b400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:24:24.239 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:24.246+0000 7f08f882b700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f08f4108730 0x7f08f419f690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:24:24.240 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:24.246+0000 7f08f882b700 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x7f08f4113360 con 0x7f08f4108730 2024-05-27T07:24:24.240 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:24.246+0000 7f08f882b700 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f08f41131e0 con 0x7f08f4107de0 2024-05-27T07:24:24.240 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:24.246+0000 7f08f882b700 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f08f41134e0 con 0x7f08f41034b0 2024-05-27T07:24:24.240 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:24.246+0000 7f08f2d9d700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f08f4108730 0x7f08f419f690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:24:24.240 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:24.246+0000 7f08f2d9d700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f08f4108730 0x7f08f419f690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.52:39984/0 (socket says 172.21.15.52:39984) 2024-05-27T07:24:24.240 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:24.246+0000 7f08f2d9d700 1 -- 172.21.15.52:0/1683908647 learned_addr learned my addr 172.21.15.52:0/1683908647 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-05-27T07:24:24.240 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:24.247+0000 7f08f259c700 1 --2- 172.21.15.52:0/1683908647 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f08f41034b0 0x7f08f419ae60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:24:24.241 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:24.247+0000 7f08f1d9b700 1 --2- 172.21.15.52:0/1683908647 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f08f4107de0 0x7f08f419b400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:24:24.241 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:24.247+0000 7f08f2d9d700 1 -- 172.21.15.52:0/1683908647 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f08f41034b0 msgr2=0x7f08f419ae60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:24:24.241 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:24.247+0000 7f08f2d9d700 1 --2- 172.21.15.52:0/1683908647 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f08f41034b0 0x7f08f419ae60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:24:24.241 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:24.247+0000 7f08f2d9d700 1 -- 172.21.15.52:0/1683908647 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f08f4107de0 msgr2=0x7f08f419b400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:24:24.241 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:24.247+0000 7f08f2d9d700 1 --2- 172.21.15.52:0/1683908647 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f08f4107de0 0x7f08f419b400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:24:24.241 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:24.247+0000 7f08f2d9d700 1 -- 172.21.15.52:0/1683908647 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f08f419fd30 con 0x7f08f4108730 2024-05-27T07:24:24.241 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:24.247+0000 7f08f1d9b700 1 --2- 172.21.15.52:0/1683908647 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f08f4107de0 0x7f08f419b400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-27T07:24:24.241 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:24.247+0000 7f08f2d9d700 1 --2- 172.21.15.52:0/1683908647 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f08f4108730 0x7f08f419f690 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 crypto rx=0x7f08ec009850 tx=0x7f08ec007a20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:24:24.241 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:24.247+0000 7f08eb7fe700 1 -- 172.21.15.52:0/1683908647 <== mon.0 v2:172.21.15.52:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f08ec01f070 con 0x7f08f4108730 2024-05-27T07:24:24.241 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:24.247+0000 7f08f882b700 1 -- 172.21.15.52:0/1683908647 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f08f419ffc0 con 0x7f08f4108730 2024-05-27T07:24:24.241 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:24.247+0000 7f08eb7fe700 1 -- 172.21.15.52:0/1683908647 <== mon.0 v2:172.21.15.52:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f08ec004c50 con 0x7f08f4108730 2024-05-27T07:24:24.241 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:24.247+0000 7f08eb7fe700 1 -- 172.21.15.52:0/1683908647 <== mon.0 v2:172.21.15.52:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f08ec01b7b0 con 0x7f08f4108730 2024-05-27T07:24:24.241 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:24.247+0000 7f08f882b700 1 -- 172.21.15.52:0/1683908647 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f08f41a04a0 con 0x7f08f4108730 2024-05-27T07:24:24.242 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:24.249+0000 7f08f882b700 1 -- 172.21.15.52:0/1683908647 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f08f40512b0 con 0x7f08f4108730 2024-05-27T07:24:24.242 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:24.251+0000 7f08eb7fe700 1 -- 172.21.15.52:0/1683908647 <== mon.0 v2:172.21.15.52:3300/0 4 ==== mgrmap(e 17) ==== 149703+0+0 (secure 0 0 0) 0x7f08ec022070 con 0x7f08f4108730 2024-05-27T07:24:24.243 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:24.252+0000 7f08eb7fe700 1 --2- 172.21.15.52:0/1683908647 >> [v2:172.21.15.52:6800/3513075805,v1:172.21.15.52:6801/3513075805] conn(0x7f08e00b2990 0x7f08e00b4ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:24:24.243 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:24.252+0000 7f08f259c700 1 --2- 172.21.15.52:0/1683908647 >> [v2:172.21.15.52:6800/3513075805,v1:172.21.15.52:6801/3513075805] conn(0x7f08e00b2990 0x7f08e00b4ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:24:24.243 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:24.252+0000 7f08eb7fe700 1 -- 172.21.15.52:0/1683908647 <== mon.0 v2:172.21.15.52:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9121+0+0 (secure 0 0 0) 0x7f08ec0e3760 con 0x7f08f4108730 2024-05-27T07:24:24.244 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:24.253+0000 7f08f259c700 1 --2- 172.21.15.52:0/1683908647 >> [v2:172.21.15.52:6800/3513075805,v1:172.21.15.52:6801/3513075805] conn(0x7f08e00b2990 0x7f08e00b4ea0 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7f08e4009c60 tx=0x7f08e4009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:24:24.245 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:24.255+0000 7f08eb7fe700 1 -- 172.21.15.52:0/1683908647 <== mon.0 v2:172.21.15.52:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191963 (secure 0 0 0) 0x7f08ec0ab050 con 0x7f08f4108730 2024-05-27T07:24:24.463 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:24.472+0000 7f08f882b700 1 -- 172.21.15.52:0/1683908647 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 7} v 0) -- 0x7f08f402e5e0 con 0x7f08f4108730 2024-05-27T07:24:24.464 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:24.472+0000 7f08eb7fe700 1 -- 172.21.15.52:0/1683908647 <== mon.0 v2:172.21.15.52:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 7}]=0 v0) ==== 74+0+13 (secure 0 0 0) 0x7f08ec009cb0 con 0x7f08f4108730 2024-05-27T07:24:24.464 INFO:teuthology.orchestra.run.smithi052.stdout:206158430275 2024-05-27T07:24:24.466 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:24.475+0000 7f08f882b700 1 -- 172.21.15.52:0/1683908647 >> [v2:172.21.15.52:6800/3513075805,v1:172.21.15.52:6801/3513075805] conn(0x7f08e00b2990 msgr2=0x7f08e00b4ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:24:24.466 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:24.475+0000 7f08f882b700 1 --2- 172.21.15.52:0/1683908647 >> [v2:172.21.15.52:6800/3513075805,v1:172.21.15.52:6801/3513075805] conn(0x7f08e00b2990 0x7f08e00b4ea0 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7f08e4009c60 tx=0x7f08e4009260 comp rx=0 tx=0).stop 2024-05-27T07:24:24.466 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:24.475+0000 7f08f882b700 1 -- 172.21.15.52:0/1683908647 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f08f4108730 msgr2=0x7f08f419f690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:24:24.466 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:24.475+0000 7f08f882b700 1 --2- 172.21.15.52:0/1683908647 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f08f4108730 0x7f08f419f690 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 crypto rx=0x7f08ec009850 tx=0x7f08ec007a20 comp rx=0 tx=0).stop 2024-05-27T07:24:24.466 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:24.475+0000 7f08f882b700 1 -- 172.21.15.52:0/1683908647 shutdown_connections 2024-05-27T07:24:24.466 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:24.475+0000 7f08f882b700 1 --2- 172.21.15.52:0/1683908647 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f08f4108730 0x7f08f419f690 unknown :-1 s=CLOSED pgs=286 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:24:24.467 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:24.475+0000 7f08f882b700 1 --2- 172.21.15.52:0/1683908647 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f08f41034b0 0x7f08f419ae60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:24:24.467 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:24.475+0000 7f08f882b700 1 --2- 172.21.15.52:0/1683908647 >> [v2:172.21.15.52:6800/3513075805,v1:172.21.15.52:6801/3513075805] conn(0x7f08e00b2990 0x7f08e00b4ea0 unknown :-1 s=CLOSED pgs=194 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:24:24.467 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:24.475+0000 7f08f882b700 1 --2- 172.21.15.52:0/1683908647 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f08f4107de0 0x7f08f419b400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:24:24.467 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:24.475+0000 7f08f882b700 1 -- 172.21.15.52:0/1683908647 >> 172.21.15.52:0/1683908647 conn(0x7f08f40fedb0 msgr2=0x7f08f406b9a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:24:24.467 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:24.475+0000 7f08f882b700 1 -- 172.21.15.52:0/1683908647 shutdown_connections 2024-05-27T07:24:24.467 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:24.475+0000 7f08f882b700 1 -- 172.21.15.52:0/1683908647 wait complete. 2024-05-27T07:24:25.102 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:24:24 smithi173 ceph-mon[37000]: pgmap v377: 129 pgs: 129 active+clean; 647 KiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 9 op/s 2024-05-27T07:24:25.102 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:24:24 smithi173 ceph-mon[37000]: from='client.? 172.21.15.52:0/1683908647' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 7}]: dispatch 2024-05-27T07:24:25.132 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:24:24 smithi052 ceph-mon[31955]: pgmap v377: 129 pgs: 129 active+clean; 647 KiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 9 op/s 2024-05-27T07:24:25.132 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:24:24 smithi052 ceph-mon[31955]: from='client.? 172.21.15.52:0/1683908647' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 7}]: dispatch 2024-05-27T07:24:25.190 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:24:24 smithi105 ceph-mon[35977]: pgmap v377: 129 pgs: 129 active+clean; 647 KiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 9 op/s 2024-05-27T07:24:25.190 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:24:24 smithi105 ceph-mon[35977]: from='client.? 172.21.15.52:0/1683908647' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 7}]: dispatch 2024-05-27T07:24:25.462 INFO:tasks.cephadm.ceph_manager.ceph:need seq 206158430270 got 206158430275 for osd.7 2024-05-27T07:24:25.462 DEBUG:teuthology.parallel:result is None 2024-05-27T07:24:26.101 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:24:25 smithi173 ceph-mon[37000]: from='client.? 172.21.15.52:0/2915794634' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 10}]: dispatch 2024-05-27T07:24:26.190 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:24:25 smithi105 ceph-mon[35977]: from='client.? 172.21.15.52:0/2915794634' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 10}]: dispatch 2024-05-27T07:24:26.229 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:24:25 smithi052 ceph-mon[31955]: from='client.? 172.21.15.52:0/2915794634' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 10}]: dispatch 2024-05-27T07:24:26.496 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:26.505+0000 7f2f2b01c700 1 -- 172.21.15.52:0/49197132 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f2f241034b0 msgr2=0x7f2f241038c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:24:26.496 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:26.505+0000 7f2f2b01c700 1 --2- 172.21.15.52:0/49197132 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f2f241034b0 0x7f2f241038c0 secure :-1 s=READY pgs=288 cs=0 l=1 rev1=1 crypto rx=0x7f2f180099c0 tx=0x7f2f18009cd0 comp rx=0 tx=0).stop 2024-05-27T07:24:26.497 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:26.505+0000 7f2f2b01c700 1 -- 172.21.15.52:0/49197132 shutdown_connections 2024-05-27T07:24:26.497 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:26.505+0000 7f2f2b01c700 1 --2- 172.21.15.52:0/49197132 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f2f24108730 0x7f2f24110090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:24:26.497 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:26.505+0000 7f2f2b01c700 1 --2- 172.21.15.52:0/49197132 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f2f241034b0 0x7f2f241038c0 unknown :-1 s=CLOSED pgs=288 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:24:26.497 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:26.505+0000 7f2f2b01c700 1 --2- 172.21.15.52:0/49197132 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f2f24107de0 0x7f2f241081f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:24:26.497 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:26.505+0000 7f2f2b01c700 1 -- 172.21.15.52:0/49197132 >> 172.21.15.52:0/49197132 conn(0x7f2f240fedb0 msgr2=0x7f2f241011d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:24:26.497 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:26.505+0000 7f2f2b01c700 1 -- 172.21.15.52:0/49197132 shutdown_connections 2024-05-27T07:24:26.497 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:26.505+0000 7f2f2b01c700 1 -- 172.21.15.52:0/49197132 wait complete. 2024-05-27T07:24:26.497 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:26.506+0000 7f2f2b01c700 1 Processor -- start 2024-05-27T07:24:26.497 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:26.506+0000 7f2f2b01c700 1 -- start start 2024-05-27T07:24:26.499 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:26.506+0000 7f2f2b01c700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f2f241034b0 0x7f2f2419bc40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:24:26.499 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:26.506+0000 7f2f2b01c700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f2f24107de0 0x7f2f2419c180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:24:26.499 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:26.506+0000 7f2f2b01c700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f2f24108730 0x7f2f2419c700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:24:26.499 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:26.506+0000 7f2f2b01c700 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x7f2f24113380 con 0x7f2f24108730 2024-05-27T07:24:26.499 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:26.506+0000 7f2f2b01c700 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f2f24113200 con 0x7f2f24107de0 2024-05-27T07:24:26.499 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:26.506+0000 7f2f2b01c700 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f2f24113500 con 0x7f2f241034b0 2024-05-27T07:24:26.499 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:26.506+0000 7f2f295b9700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f2f24108730 0x7f2f2419c700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:24:26.499 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:26.506+0000 7f2f295b9700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f2f24108730 0x7f2f2419c700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.52:40006/0 (socket says 172.21.15.52:40006) 2024-05-27T07:24:26.500 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:26.506+0000 7f2f295b9700 1 -- 172.21.15.52:0/2715967528 learned_addr learned my addr 172.21.15.52:0/2715967528 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-05-27T07:24:26.500 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:26.507+0000 7f2f23fff700 1 --2- 172.21.15.52:0/2715967528 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f2f24107de0 0x7f2f2419c180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:24:26.500 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:26.507+0000 7f2f28db8700 1 --2- 172.21.15.52:0/2715967528 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f2f241034b0 0x7f2f2419bc40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:24:26.500 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:26.507+0000 7f2f295b9700 1 -- 172.21.15.52:0/2715967528 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f2f241034b0 msgr2=0x7f2f2419bc40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:24:26.500 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:26.507+0000 7f2f295b9700 1 --2- 172.21.15.52:0/2715967528 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f2f241034b0 0x7f2f2419bc40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:24:26.500 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:26.507+0000 7f2f295b9700 1 -- 172.21.15.52:0/2715967528 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f2f24107de0 msgr2=0x7f2f2419c180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:24:26.500 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:26.507+0000 7f2f295b9700 1 --2- 172.21.15.52:0/2715967528 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f2f24107de0 0x7f2f2419c180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:24:26.500 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:26.507+0000 7f2f295b9700 1 -- 172.21.15.52:0/2715967528 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2f241a6fb0 con 0x7f2f24108730 2024-05-27T07:24:26.500 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:26.507+0000 7f2f23fff700 1 --2- 172.21.15.52:0/2715967528 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f2f24107de0 0x7f2f2419c180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-27T07:24:26.500 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:26.507+0000 7f2f295b9700 1 --2- 172.21.15.52:0/2715967528 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f2f24108730 0x7f2f2419c700 secure :-1 s=READY pgs=289 cs=0 l=1 rev1=1 crypto rx=0x7f2f1c007ee0 tx=0x7f2f1c00d3b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:24:26.500 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:26.507+0000 7f2f21ffb700 1 -- 172.21.15.52:0/2715967528 <== mon.0 v2:172.21.15.52:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2f1c00dbd0 con 0x7f2f24108730 2024-05-27T07:24:26.500 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:26.507+0000 7f2f21ffb700 1 -- 172.21.15.52:0/2715967528 <== mon.0 v2:172.21.15.52:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f2f1c00dd70 con 0x7f2f24108730 2024-05-27T07:24:26.500 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:26.507+0000 7f2f21ffb700 1 -- 172.21.15.52:0/2715967528 <== mon.0 v2:172.21.15.52:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2f1c005990 con 0x7f2f24108730 2024-05-27T07:24:26.501 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:26.507+0000 7f2f2b01c700 1 -- 172.21.15.52:0/2715967528 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2f241a72a0 con 0x7f2f24108730 2024-05-27T07:24:26.501 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:26.507+0000 7f2f2b01c700 1 -- 172.21.15.52:0/2715967528 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2f241a7790 con 0x7f2f24108730 2024-05-27T07:24:26.501 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:26.508+0000 7f2f2b01c700 1 -- 172.21.15.52:0/2715967528 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2f240512b0 con 0x7f2f24108730 2024-05-27T07:24:26.501 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:26.511+0000 7f2f21ffb700 1 -- 172.21.15.52:0/2715967528 <== mon.0 v2:172.21.15.52:3300/0 4 ==== mgrmap(e 17) ==== 149703+0+0 (secure 0 0 0) 0x7f2f1c01a020 con 0x7f2f24108730 2024-05-27T07:24:26.502 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:26.512+0000 7f2f21ffb700 1 --2- 172.21.15.52:0/2715967528 >> [v2:172.21.15.52:6800/3513075805,v1:172.21.15.52:6801/3513075805] conn(0x7f2f140b2910 0x7f2f140b4e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:24:26.502 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:26.512+0000 7f2f28db8700 1 --2- 172.21.15.52:0/2715967528 >> [v2:172.21.15.52:6800/3513075805,v1:172.21.15.52:6801/3513075805] conn(0x7f2f140b2910 0x7f2f140b4e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:24:26.502 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:26.512+0000 7f2f21ffb700 1 -- 172.21.15.52:0/2715967528 <== mon.0 v2:172.21.15.52:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9121+0+0 (secure 0 0 0) 0x7f2f1c0e25e0 con 0x7f2f24108730 2024-05-27T07:24:26.503 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:26.512+0000 7f2f28db8700 1 --2- 172.21.15.52:0/2715967528 >> [v2:172.21.15.52:6800/3513075805,v1:172.21.15.52:6801/3513075805] conn(0x7f2f140b2910 0x7f2f140b4e20 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7f2f18003800 tx=0x7f2f1801a040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:24:26.506 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:26.515+0000 7f2f21ffb700 1 -- 172.21.15.52:0/2715967528 <== mon.0 v2:172.21.15.52:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191963 (secure 0 0 0) 0x7f2f1c010070 con 0x7f2f24108730 2024-05-27T07:24:26.711 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:26.720+0000 7f69814f2700 1 -- 172.21.15.52:0/3408106840 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f697c105dc0 msgr2=0x7f697c10e310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:24:26.711 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:26.720+0000 7f69814f2700 1 --2- 172.21.15.52:0/3408106840 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f697c105dc0 0x7f697c10e310 secure :-1 s=READY pgs=290 cs=0 l=1 rev1=1 crypto rx=0x7f6964009990 tx=0x7f6964009ca0 comp rx=0 tx=0).stop 2024-05-27T07:24:26.711 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:26.720+0000 7f69814f2700 1 -- 172.21.15.52:0/3408106840 shutdown_connections 2024-05-27T07:24:26.712 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:26.720+0000 7f69814f2700 1 --2- 172.21.15.52:0/3408106840 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f697c10e8e0 0x7f697c110dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:24:26.712 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:26.720+0000 7f69814f2700 1 --2- 172.21.15.52:0/3408106840 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f697c105470 0x7f697c105880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:24:26.712 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:26.720+0000 7f69814f2700 1 --2- 172.21.15.52:0/3408106840 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f697c105dc0 0x7f697c10e310 unknown :-1 s=CLOSED pgs=290 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:24:26.712 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:26.720+0000 7f69814f2700 1 -- 172.21.15.52:0/3408106840 >> 172.21.15.52:0/3408106840 conn(0x7f697c0fef20 msgr2=0x7f697c101340 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:24:26.712 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:26.720+0000 7f69814f2700 1 -- 172.21.15.52:0/3408106840 shutdown_connections 2024-05-27T07:24:26.712 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:26.720+0000 7f69814f2700 1 -- 172.21.15.52:0/3408106840 wait complete. 2024-05-27T07:24:26.715 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:26.721+0000 7f69814f2700 1 Processor -- start 2024-05-27T07:24:26.715 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:26.721+0000 7f69814f2700 1 -- start start 2024-05-27T07:24:26.715 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:26.721+0000 7f69814f2700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f697c105470 0x7f697c103160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:24:26.715 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:26.721+0000 7f69814f2700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f697c105dc0 0x7f697c1036a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:24:26.715 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:26.721+0000 7f69814f2700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f697c10e8e0 0x7f697c103be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:24:26.715 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:26.722+0000 7f69814f2700 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x7f697c114440 con 0x7f697c105dc0 2024-05-27T07:24:26.715 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:26.722+0000 7f69814f2700 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f697c1142c0 con 0x7f697c10e8e0 2024-05-27T07:24:26.715 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:26.722+0000 7f69814f2700 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f697c1145c0 con 0x7f697c105470 2024-05-27T07:24:26.715 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:26.722+0000 7f69727fc700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f697c105dc0 0x7f697c1036a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:24:26.716 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:26.722+0000 7f69727fc700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f697c105dc0 0x7f697c1036a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.52:40012/0 (socket says 172.21.15.52:40012) 2024-05-27T07:24:26.716 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:26.722+0000 7f69727fc700 1 -- 172.21.15.52:0/1864990731 learned_addr learned my addr 172.21.15.52:0/1864990731 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-05-27T07:24:26.716 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:26.722+0000 7f697b7fe700 1 --2- 172.21.15.52:0/1864990731 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f697c10e8e0 0x7f697c103be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:24:26.716 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:26.722+0000 7f697affd700 1 --2- 172.21.15.52:0/1864990731 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f697c105470 0x7f697c103160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:24:26.716 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:26.722+0000 7f69727fc700 1 -- 172.21.15.52:0/1864990731 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f697c105470 msgr2=0x7f697c103160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:24:26.716 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:26.722+0000 7f69727fc700 1 --2- 172.21.15.52:0/1864990731 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f697c105470 0x7f697c103160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:24:26.716 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:26.722+0000 7f69727fc700 1 -- 172.21.15.52:0/1864990731 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f697c10e8e0 msgr2=0x7f697c103be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:24:26.716 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:26.722+0000 7f69727fc700 1 --2- 172.21.15.52:0/1864990731 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f697c10e8e0 0x7f697c103be0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:24:26.716 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:26.722+0000 7f69727fc700 1 -- 172.21.15.52:0/1864990731 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f697c1a9190 con 0x7f697c105dc0 2024-05-27T07:24:26.717 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:26.722+0000 7f697b7fe700 1 --2- 172.21.15.52:0/1864990731 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f697c10e8e0 0x7f697c103be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-27T07:24:26.717 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:26.722+0000 7f69727fc700 1 --2- 172.21.15.52:0/1864990731 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f697c105dc0 0x7f697c1036a0 secure :-1 s=READY pgs=291 cs=0 l=1 rev1=1 crypto rx=0x7f69640057c0 tx=0x7f6964009580 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:24:26.717 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:26.722+0000 7f6978ff9700 1 -- 172.21.15.52:0/1864990731 <== mon.0 v2:172.21.15.52:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6964026070 con 0x7f697c105dc0 2024-05-27T07:24:26.717 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:26.723+0000 7f69814f2700 1 -- 172.21.15.52:0/1864990731 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f697c1a9420 con 0x7f697c105dc0 2024-05-27T07:24:26.717 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:26.723+0000 7f6978ff9700 1 -- 172.21.15.52:0/1864990731 <== mon.0 v2:172.21.15.52:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f6964021410 con 0x7f697c105dc0 2024-05-27T07:24:26.717 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:26.723+0000 7f6978ff9700 1 -- 172.21.15.52:0/1864990731 <== mon.0 v2:172.21.15.52:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6964017650 con 0x7f697c105dc0 2024-05-27T07:24:26.717 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:26.723+0000 7f69814f2700 1 -- 172.21.15.52:0/1864990731 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f697c1a9950 con 0x7f697c105dc0 2024-05-27T07:24:26.717 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:26.724+0000 7f69814f2700 1 -- 172.21.15.52:0/1864990731 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f697c0512b0 con 0x7f697c105dc0 2024-05-27T07:24:26.718 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:26.727+0000 7f6978ff9700 1 -- 172.21.15.52:0/1864990731 <== mon.0 v2:172.21.15.52:3300/0 4 ==== mgrmap(e 17) ==== 149703+0+0 (secure 0 0 0) 0x7f6964029070 con 0x7f697c105dc0 2024-05-27T07:24:26.718 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:26.727+0000 7f6978ff9700 1 --2- 172.21.15.52:0/1864990731 >> [v2:172.21.15.52:6800/3513075805,v1:172.21.15.52:6801/3513075805] conn(0x7f69680b2910 0x7f69680b4e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:24:26.718 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:26.727+0000 7f697affd700 1 --2- 172.21.15.52:0/1864990731 >> [v2:172.21.15.52:6800/3513075805,v1:172.21.15.52:6801/3513075805] conn(0x7f69680b2910 0x7f69680b4e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:24:26.718 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:26.727+0000 7f6978ff9700 1 -- 172.21.15.52:0/1864990731 <== mon.0 v2:172.21.15.52:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9121+0+0 (secure 0 0 0) 0x7f69640e36b0 con 0x7f697c105dc0 2024-05-27T07:24:26.719 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:26.728+0000 7f697affd700 1 --2- 172.21.15.52:0/1864990731 >> [v2:172.21.15.52:6800/3513075805,v1:172.21.15.52:6801/3513075805] conn(0x7f69680b2910 0x7f69680b4e20 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7f696c005950 tx=0x7f696c00b410 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:24:26.721 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:26.731+0000 7f6978ff9700 1 -- 172.21.15.52:0/1864990731 <== mon.0 v2:172.21.15.52:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191963 (secure 0 0 0) 0x7f6964013070 con 0x7f697c105dc0 2024-05-27T07:24:26.729 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:26.738+0000 7f2f2b01c700 1 -- 172.21.15.52:0/2715967528 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 2} v 0) -- 0x7f2f2402e5e0 con 0x7f2f24108730 2024-05-27T07:24:26.729 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:26.738+0000 7f2f21ffb700 1 -- 172.21.15.52:0/2715967528 <== mon.0 v2:172.21.15.52:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 2}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7f2f1c0a9dc0 con 0x7f2f24108730 2024-05-27T07:24:26.730 INFO:teuthology.orchestra.run.smithi052.stdout:77309411421 2024-05-27T07:24:26.732 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:26.741+0000 7f2f2b01c700 1 -- 172.21.15.52:0/2715967528 >> [v2:172.21.15.52:6800/3513075805,v1:172.21.15.52:6801/3513075805] conn(0x7f2f140b2910 msgr2=0x7f2f140b4e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:24:26.733 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:26.741+0000 7f2f2b01c700 1 --2- 172.21.15.52:0/2715967528 >> [v2:172.21.15.52:6800/3513075805,v1:172.21.15.52:6801/3513075805] conn(0x7f2f140b2910 0x7f2f140b4e20 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7f2f18003800 tx=0x7f2f1801a040 comp rx=0 tx=0).stop 2024-05-27T07:24:26.733 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:26.741+0000 7f2f2b01c700 1 -- 172.21.15.52:0/2715967528 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f2f24108730 msgr2=0x7f2f2419c700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:24:26.733 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:26.741+0000 7f2f2b01c700 1 --2- 172.21.15.52:0/2715967528 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f2f24108730 0x7f2f2419c700 secure :-1 s=READY pgs=289 cs=0 l=1 rev1=1 crypto rx=0x7f2f1c007ee0 tx=0x7f2f1c00d3b0 comp rx=0 tx=0).stop 2024-05-27T07:24:26.733 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:26.741+0000 7f2f2b01c700 1 -- 172.21.15.52:0/2715967528 shutdown_connections 2024-05-27T07:24:26.733 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:26.741+0000 7f2f2b01c700 1 --2- 172.21.15.52:0/2715967528 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f2f24108730 0x7f2f2419c700 unknown :-1 s=CLOSED pgs=289 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:24:26.733 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:26.741+0000 7f2f2b01c700 1 --2- 172.21.15.52:0/2715967528 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f2f241034b0 0x7f2f2419bc40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:24:26.734 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:26.741+0000 7f2f2b01c700 1 --2- 172.21.15.52:0/2715967528 >> [v2:172.21.15.52:6800/3513075805,v1:172.21.15.52:6801/3513075805] conn(0x7f2f140b2910 0x7f2f140b4e20 unknown :-1 s=CLOSED pgs=196 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:24:26.734 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:26.741+0000 7f2f2b01c700 1 --2- 172.21.15.52:0/2715967528 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f2f24107de0 0x7f2f2419c180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:24:26.734 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:26.741+0000 7f2f2b01c700 1 -- 172.21.15.52:0/2715967528 >> 172.21.15.52:0/2715967528 conn(0x7f2f240fedb0 msgr2=0x7f2f241075c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:24:26.734 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:26.741+0000 7f2f2b01c700 1 -- 172.21.15.52:0/2715967528 shutdown_connections 2024-05-27T07:24:26.734 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:26.741+0000 7f2f2b01c700 1 -- 172.21.15.52:0/2715967528 wait complete. 2024-05-27T07:24:26.951 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:26.960+0000 7f69814f2700 1 -- 172.21.15.52:0/1864990731 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 8} v 0) -- 0x7f697c02e5e0 con 0x7f697c105dc0 2024-05-27T07:24:26.951 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:26.960+0000 7f6978ff9700 1 -- 172.21.15.52:0/1864990731 <== mon.0 v2:172.21.15.52:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 8}]=0 v0) ==== 74+0+13 (secure 0 0 0) 0x7f69640aafb0 con 0x7f697c105dc0 2024-05-27T07:24:26.951 INFO:teuthology.orchestra.run.smithi052.stdout:231928234047 2024-05-27T07:24:26.953 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:26.962+0000 7f69814f2700 1 -- 172.21.15.52:0/1864990731 >> [v2:172.21.15.52:6800/3513075805,v1:172.21.15.52:6801/3513075805] conn(0x7f69680b2910 msgr2=0x7f69680b4e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:24:26.953 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:26.962+0000 7f69814f2700 1 --2- 172.21.15.52:0/1864990731 >> [v2:172.21.15.52:6800/3513075805,v1:172.21.15.52:6801/3513075805] conn(0x7f69680b2910 0x7f69680b4e20 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7f696c005950 tx=0x7f696c00b410 comp rx=0 tx=0).stop 2024-05-27T07:24:26.953 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:26.962+0000 7f69814f2700 1 -- 172.21.15.52:0/1864990731 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f697c105dc0 msgr2=0x7f697c1036a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:24:26.954 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:26.962+0000 7f69814f2700 1 --2- 172.21.15.52:0/1864990731 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f697c105dc0 0x7f697c1036a0 secure :-1 s=READY pgs=291 cs=0 l=1 rev1=1 crypto rx=0x7f69640057c0 tx=0x7f6964009580 comp rx=0 tx=0).stop 2024-05-27T07:24:26.954 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:26.962+0000 7f69814f2700 1 -- 172.21.15.52:0/1864990731 shutdown_connections 2024-05-27T07:24:26.954 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:26.962+0000 7f69814f2700 1 --2- 172.21.15.52:0/1864990731 >> [v2:172.21.15.52:6800/3513075805,v1:172.21.15.52:6801/3513075805] conn(0x7f69680b2910 0x7f69680b4e20 unknown :-1 s=CLOSED pgs=197 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:24:26.954 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:26.962+0000 7f69814f2700 1 --2- 172.21.15.52:0/1864990731 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f697c10e8e0 0x7f697c103be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:24:26.954 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:26.962+0000 7f69814f2700 1 --2- 172.21.15.52:0/1864990731 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f697c105470 0x7f697c103160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:24:26.954 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:26.962+0000 7f69814f2700 1 --2- 172.21.15.52:0/1864990731 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f697c105dc0 0x7f697c1036a0 unknown :-1 s=CLOSED pgs=291 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:24:26.954 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:26.962+0000 7f69814f2700 1 -- 172.21.15.52:0/1864990731 >> 172.21.15.52:0/1864990731 conn(0x7f697c0fef20 msgr2=0x7f697c100c90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:24:26.954 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:26.962+0000 7f69814f2700 1 -- 172.21.15.52:0/1864990731 shutdown_connections 2024-05-27T07:24:26.954 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:26.963+0000 7f69814f2700 1 -- 172.21.15.52:0/1864990731 wait complete. 2024-05-27T07:24:27.102 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:24:26 smithi173 ceph-mon[37000]: pgmap v378: 129 pgs: 129 active+clean; 647 KiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 11 op/s 2024-05-27T07:24:27.102 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:24:26 smithi173 ceph-mon[37000]: from='client.? 172.21.15.52:0/2715967528' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 2}]: dispatch 2024-05-27T07:24:27.124 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:24:26 smithi052 ceph-mon[31955]: pgmap v378: 129 pgs: 129 active+clean; 647 KiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 11 op/s 2024-05-27T07:24:27.124 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:24:26 smithi052 ceph-mon[31955]: from='client.? 172.21.15.52:0/2715967528' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 2}]: dispatch 2024-05-27T07:24:27.151 INFO:tasks.cephadm.ceph_manager.ceph:need seq 34359738465 got 34359738470 for osd.0 2024-05-27T07:24:27.151 DEBUG:teuthology.parallel:result is None 2024-05-27T07:24:27.190 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:24:26 smithi105 ceph-mon[35977]: pgmap v378: 129 pgs: 129 active+clean; 647 KiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 11 op/s 2024-05-27T07:24:27.190 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:24:26 smithi105 ceph-mon[35977]: from='client.? 172.21.15.52:0/2715967528' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 2}]: dispatch 2024-05-27T07:24:27.815 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:27.824+0000 7fc1462e1700 1 -- 172.21.15.52:0/3931094096 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fc140107de0 msgr2=0x7fc1401081f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:24:27.815 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:27.824+0000 7fc1462e1700 1 --2- 172.21.15.52:0/3931094096 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fc140107de0 0x7fc1401081f0 secure :-1 s=READY pgs=292 cs=0 l=1 rev1=1 crypto rx=0x7fc1340099c0 tx=0x7fc134009cd0 comp rx=0 tx=0).stop 2024-05-27T07:24:27.816 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:27.824+0000 7fc1462e1700 1 -- 172.21.15.52:0/3931094096 shutdown_connections 2024-05-27T07:24:27.816 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:27.824+0000 7fc1462e1700 1 --2- 172.21.15.52:0/3931094096 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fc140108730 0x7fc140110090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:24:27.816 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:27.824+0000 7fc1462e1700 1 --2- 172.21.15.52:0/3931094096 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fc1401034b0 0x7fc1401038c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:24:27.816 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:27.824+0000 7fc1462e1700 1 --2- 172.21.15.52:0/3931094096 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fc140107de0 0x7fc1401081f0 unknown :-1 s=CLOSED pgs=292 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:24:27.816 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:27.824+0000 7fc1462e1700 1 -- 172.21.15.52:0/3931094096 >> 172.21.15.52:0/3931094096 conn(0x7fc1400fede0 msgr2=0x7fc140101200 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:24:27.816 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:27.824+0000 7fc1462e1700 1 -- 172.21.15.52:0/3931094096 shutdown_connections 2024-05-27T07:24:27.816 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:27.824+0000 7fc1462e1700 1 -- 172.21.15.52:0/3931094096 wait complete. 2024-05-27T07:24:27.816 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:27.825+0000 7fc1462e1700 1 Processor -- start 2024-05-27T07:24:27.816 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:27.825+0000 7fc1462e1700 1 -- start start 2024-05-27T07:24:27.818 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:27.825+0000 7fc1462e1700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fc1401034b0 0x7fc14019bce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:24:27.818 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:27.825+0000 7fc1462e1700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fc140107de0 0x7fc14019c220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:24:27.818 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:27.825+0000 7fc1462e1700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fc140108730 0x7fc14019c780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:24:27.818 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:27.825+0000 7fc1462e1700 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x7fc140113340 con 0x7fc1401034b0 2024-05-27T07:24:27.819 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:27.825+0000 7fc1462e1700 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7fc1401131c0 con 0x7fc140107de0 2024-05-27T07:24:27.819 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:27.825+0000 7fc1462e1700 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7fc1401134c0 con 0x7fc140108730 2024-05-27T07:24:27.819 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:27.825+0000 7fc13ffff700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fc1401034b0 0x7fc14019bce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:24:27.819 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:27.825+0000 7fc13ffff700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fc1401034b0 0x7fc14019bce0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.52:40028/0 (socket says 172.21.15.52:40028) 2024-05-27T07:24:27.819 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:27.825+0000 7fc13ffff700 1 -- 172.21.15.52:0/3549775543 learned_addr learned my addr 172.21.15.52:0/3549775543 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-05-27T07:24:27.819 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:27.826+0000 7fc13f7fe700 1 --2- 172.21.15.52:0/3549775543 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fc140107de0 0x7fc14019c220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:24:27.819 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:27.826+0000 7fc14487e700 1 --2- 172.21.15.52:0/3549775543 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fc140108730 0x7fc14019c780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:24:27.819 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:27.826+0000 7fc13ffff700 1 -- 172.21.15.52:0/3549775543 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fc140108730 msgr2=0x7fc14019c780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:24:27.819 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:27.826+0000 7fc13ffff700 1 --2- 172.21.15.52:0/3549775543 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fc140108730 0x7fc14019c780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:24:27.819 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:27.826+0000 7fc13ffff700 1 -- 172.21.15.52:0/3549775543 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fc140107de0 msgr2=0x7fc14019c220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:24:27.819 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:27.826+0000 7fc13ffff700 1 --2- 172.21.15.52:0/3549775543 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fc140107de0 0x7fc14019c220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:24:27.819 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:27.826+0000 7fc13ffff700 1 -- 172.21.15.52:0/3549775543 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc1401a6fa0 con 0x7fc1401034b0 2024-05-27T07:24:27.820 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:27.826+0000 7fc13ffff700 1 --2- 172.21.15.52:0/3549775543 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fc1401034b0 0x7fc14019bce0 secure :-1 s=READY pgs=293 cs=0 l=1 rev1=1 crypto rx=0x7fc13000c9b0 tx=0x7fc13000ccc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:24:27.820 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:27.826+0000 7fc13d7fa700 1 -- 172.21.15.52:0/3549775543 <== mon.0 v2:172.21.15.52:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc130007b00 con 0x7fc1401034b0 2024-05-27T07:24:27.820 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:27.826+0000 7fc13d7fa700 1 -- 172.21.15.52:0/3549775543 <== mon.0 v2:172.21.15.52:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fc130007ca0 con 0x7fc1401034b0 2024-05-27T07:24:27.820 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:27.826+0000 7fc1462e1700 1 -- 172.21.15.52:0/3549775543 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc1401a7290 con 0x7fc1401034b0 2024-05-27T07:24:27.820 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:27.826+0000 7fc13d7fa700 1 -- 172.21.15.52:0/3549775543 <== mon.0 v2:172.21.15.52:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc130018660 con 0x7fc1401034b0 2024-05-27T07:24:27.820 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:27.826+0000 7fc1462e1700 1 -- 172.21.15.52:0/3549775543 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc1401a77f0 con 0x7fc1401034b0 2024-05-27T07:24:27.820 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:27.827+0000 7fc1462e1700 1 -- 172.21.15.52:0/3549775543 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc1400512b0 con 0x7fc1401034b0 2024-05-27T07:24:27.821 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:27.829+0000 7fc13d7fa700 1 -- 172.21.15.52:0/3549775543 <== mon.0 v2:172.21.15.52:3300/0 4 ==== mgrmap(e 17) ==== 149703+0+0 (secure 0 0 0) 0x7fc13000f450 con 0x7fc1401034b0 2024-05-27T07:24:27.821 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:27.830+0000 7fc13d7fa700 1 --2- 172.21.15.52:0/3549775543 >> [v2:172.21.15.52:6800/3513075805,v1:172.21.15.52:6801/3513075805] conn(0x7fc1280b2990 0x7fc1280b4ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:24:27.821 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:27.830+0000 7fc13d7fa700 1 -- 172.21.15.52:0/3549775543 <== mon.0 v2:172.21.15.52:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9121+0+0 (secure 0 0 0) 0x7fc1300e28f0 con 0x7fc1401034b0 2024-05-27T07:24:27.821 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:27.830+0000 7fc13f7fe700 1 --2- 172.21.15.52:0/3549775543 >> [v2:172.21.15.52:6800/3513075805,v1:172.21.15.52:6801/3513075805] conn(0x7fc1280b2990 0x7fc1280b4ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:24:27.821 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:27.830+0000 7fc13f7fe700 1 --2- 172.21.15.52:0/3549775543 >> [v2:172.21.15.52:6800/3513075805,v1:172.21.15.52:6801/3513075805] conn(0x7fc1280b2990 0x7fc1280b4ea0 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7fc134006010 tx=0x7fc134000bc0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:24:27.824 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:27.833+0000 7fc13d7fa700 1 -- 172.21.15.52:0/3549775543 <== mon.0 v2:172.21.15.52:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191963 (secure 0 0 0) 0x7fc130014030 con 0x7fc1401034b0 2024-05-27T07:24:27.979 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:24:27 smithi052 ceph-mon[31955]: from='client.? 172.21.15.52:0/1864990731' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 8}]: dispatch 2024-05-27T07:24:28.044 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:28.053+0000 7fc1462e1700 1 -- 172.21.15.52:0/3549775543 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 11} v 0) -- 0x7fc140075fd0 con 0x7fc1401034b0 2024-05-27T07:24:28.044 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:28.053+0000 7fc13d7fa700 1 -- 172.21.15.52:0/3549775543 <== mon.0 v2:172.21.15.52:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 11}]=0 v0) ==== 75+0+13 (secure 0 0 0) 0x7fc1300aa240 con 0x7fc1401034b0 2024-05-27T07:24:28.044 INFO:teuthology.orchestra.run.smithi052.stdout:304942678064 2024-05-27T07:24:28.046 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:28.055+0000 7fc1462e1700 1 -- 172.21.15.52:0/3549775543 >> [v2:172.21.15.52:6800/3513075805,v1:172.21.15.52:6801/3513075805] conn(0x7fc1280b2990 msgr2=0x7fc1280b4ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:24:28.046 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:28.055+0000 7fc1462e1700 1 --2- 172.21.15.52:0/3549775543 >> [v2:172.21.15.52:6800/3513075805,v1:172.21.15.52:6801/3513075805] conn(0x7fc1280b2990 0x7fc1280b4ea0 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7fc134006010 tx=0x7fc134000bc0 comp rx=0 tx=0).stop 2024-05-27T07:24:28.046 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:28.055+0000 7fc1462e1700 1 -- 172.21.15.52:0/3549775543 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fc1401034b0 msgr2=0x7fc14019bce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:24:28.046 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:28.055+0000 7fc1462e1700 1 --2- 172.21.15.52:0/3549775543 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fc1401034b0 0x7fc14019bce0 secure :-1 s=READY pgs=293 cs=0 l=1 rev1=1 crypto rx=0x7fc13000c9b0 tx=0x7fc13000ccc0 comp rx=0 tx=0).stop 2024-05-27T07:24:28.047 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:28.055+0000 7fc1462e1700 1 -- 172.21.15.52:0/3549775543 shutdown_connections 2024-05-27T07:24:28.047 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:28.055+0000 7fc1462e1700 1 --2- 172.21.15.52:0/3549775543 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fc140108730 0x7fc14019c780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:24:28.047 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:28.055+0000 7fc1462e1700 1 --2- 172.21.15.52:0/3549775543 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fc1401034b0 0x7fc14019bce0 unknown :-1 s=CLOSED pgs=293 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:24:28.047 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:28.055+0000 7fc1462e1700 1 --2- 172.21.15.52:0/3549775543 >> [v2:172.21.15.52:6800/3513075805,v1:172.21.15.52:6801/3513075805] conn(0x7fc1280b2990 0x7fc1280b4ea0 unknown :-1 s=CLOSED pgs=198 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:24:28.047 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:28.055+0000 7fc1462e1700 1 --2- 172.21.15.52:0/3549775543 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fc140107de0 0x7fc14019c220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:24:28.047 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:28.055+0000 7fc1462e1700 1 -- 172.21.15.52:0/3549775543 >> 172.21.15.52:0/3549775543 conn(0x7fc1400fede0 msgr2=0x7fc140101200 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:24:28.047 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:28.055+0000 7fc1462e1700 1 -- 172.21.15.52:0/3549775543 shutdown_connections 2024-05-27T07:24:28.047 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:28.055+0000 7fc1462e1700 1 -- 172.21.15.52:0/3549775543 wait complete. 2024-05-27T07:24:28.102 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:24:27 smithi173 ceph-mon[37000]: from='client.? 172.21.15.52:0/1864990731' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 8}]: dispatch 2024-05-27T07:24:28.108 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:24.619+0000 7ff7523b9700 1 -- 172.21.15.52:0/745880158 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff74c1034b0 msgr2=0x7ff74c1038c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:24:28.108 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:24.619+0000 7ff7523b9700 1 --2- 172.21.15.52:0/745880158 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff74c1034b0 0x7ff74c1038c0 secure :-1 s=READY pgs=287 cs=0 l=1 rev1=1 crypto rx=0x7ff73c0099c0 tx=0x7ff73c009cd0 comp rx=0 tx=0).stop 2024-05-27T07:24:28.108 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:24.619+0000 7ff7523b9700 1 -- 172.21.15.52:0/745880158 shutdown_connections 2024-05-27T07:24:28.108 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:24.619+0000 7ff7523b9700 1 --2- 172.21.15.52:0/745880158 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7ff74c108730 0x7ff74c110090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:24:28.108 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:24.619+0000 7ff7523b9700 1 --2- 172.21.15.52:0/745880158 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff74c1034b0 0x7ff74c1038c0 unknown :-1 s=CLOSED pgs=287 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:24:28.108 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:24.619+0000 7ff7523b9700 1 --2- 172.21.15.52:0/745880158 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7ff74c107de0 0x7ff74c1081f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:24:28.108 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:24.619+0000 7ff7523b9700 1 -- 172.21.15.52:0/745880158 >> 172.21.15.52:0/745880158 conn(0x7ff74c0fede0 msgr2=0x7ff74c101200 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:24:28.108 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:24.619+0000 7ff7523b9700 1 -- 172.21.15.52:0/745880158 shutdown_connections 2024-05-27T07:24:28.109 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:24.619+0000 7ff7523b9700 1 -- 172.21.15.52:0/745880158 wait complete. 2024-05-27T07:24:28.109 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:24.620+0000 7ff7523b9700 1 Processor -- start 2024-05-27T07:24:28.109 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:24.620+0000 7ff7523b9700 1 -- start start 2024-05-27T07:24:28.109 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:24.620+0000 7ff7523b9700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7ff74c1034b0 0x7ff74c0ff2c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:24:28.109 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:24.620+0000 7ff7523b9700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff74c107de0 0x7ff74c0ff800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:24:28.109 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:24.620+0000 7ff7523b9700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7ff74c108730 0x7ff74c0ffd40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:24:28.109 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:24.620+0000 7ff7523b9700 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x7ff74c113360 con 0x7ff74c107de0 2024-05-27T07:24:28.109 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:24.620+0000 7ff7523b9700 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7ff74c1131e0 con 0x7ff74c1034b0 2024-05-27T07:24:28.109 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:24.620+0000 7ff7523b9700 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7ff74c1134e0 con 0x7ff74c108730 2024-05-27T07:24:28.109 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:24.620+0000 7ff74b7fe700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff74c107de0 0x7ff74c0ff800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:24:28.109 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:24.620+0000 7ff74b7fe700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff74c107de0 0x7ff74c0ff800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.52:40002/0 (socket says 172.21.15.52:40002) 2024-05-27T07:24:28.109 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:24.621+0000 7ff74b7fe700 1 -- 172.21.15.52:0/2915794634 learned_addr learned my addr 172.21.15.52:0/2915794634 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-05-27T07:24:28.109 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:24.621+0000 7ff750956700 1 --2- 172.21.15.52:0/2915794634 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7ff74c108730 0x7ff74c0ffd40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:24:28.110 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:24.621+0000 7ff74bfff700 1 --2- 172.21.15.52:0/2915794634 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7ff74c1034b0 0x7ff74c0ff2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:24:28.110 INFO:teuthology.orchestra.run.smithi052.stdout:279172874291 2024-05-27T07:24:28.110 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:24.621+0000 7ff750956700 1 -- 172.21.15.52:0/2915794634 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7ff74c1034b0 msgr2=0x7ff74c0ff2c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:24:28.110 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:24.621+0000 7ff750956700 1 --2- 172.21.15.52:0/2915794634 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7ff74c1034b0 0x7ff74c0ff2c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:24:28.110 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:24.621+0000 7ff750956700 1 -- 172.21.15.52:0/2915794634 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff74c107de0 msgr2=0x7ff74c0ff800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:24:28.111 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:24.621+0000 7ff750956700 1 --2- 172.21.15.52:0/2915794634 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff74c107de0 0x7ff74c0ff800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:24:28.111 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:24.621+0000 7ff750956700 1 -- 172.21.15.52:0/2915794634 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff74c1a91d0 con 0x7ff74c108730 2024-05-27T07:24:28.111 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:24.621+0000 7ff750956700 1 --2- 172.21.15.52:0/2915794634 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7ff74c108730 0x7ff74c0ffd40 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7ff74400ab40 tx=0x7ff74400ae50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:24:28.111 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:24.621+0000 7ff7497fa700 1 -- 172.21.15.52:0/2915794634 <== mon.1 v2:172.21.15.173:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff744017020 con 0x7ff74c108730 2024-05-27T07:24:28.111 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:24.621+0000 7ff7523b9700 1 -- 172.21.15.52:0/2915794634 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff74c1a94c0 con 0x7ff74c108730 2024-05-27T07:24:28.111 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:24.621+0000 7ff7497fa700 1 -- 172.21.15.52:0/2915794634 <== mon.1 v2:172.21.15.173:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ff744004bb0 con 0x7ff74c108730 2024-05-27T07:24:28.111 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:24.621+0000 7ff7497fa700 1 -- 172.21.15.52:0/2915794634 <== mon.1 v2:172.21.15.173:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff744005280 con 0x7ff74c108730 2024-05-27T07:24:28.111 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:24.621+0000 7ff7523b9700 1 -- 172.21.15.52:0/2915794634 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff74c1a9a00 con 0x7ff74c108730 2024-05-27T07:24:28.111 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:24.622+0000 7ff7523b9700 1 -- 172.21.15.52:0/2915794634 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff74c0512b0 con 0x7ff74c108730 2024-05-27T07:24:28.111 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:24.624+0000 7ff7497fa700 1 -- 172.21.15.52:0/2915794634 <== mon.1 v2:172.21.15.173:3300/0 4 ==== mgrmap(e 17) ==== 149703+0+0 (secure 0 0 0) 0x7ff74400c580 con 0x7ff74c108730 2024-05-27T07:24:28.111 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:24.625+0000 7ff7497fa700 1 --2- 172.21.15.52:0/2915794634 >> [v2:172.21.15.52:6800/3513075805,v1:172.21.15.52:6801/3513075805] conn(0x7ff7380b2910 0x7ff7380b4e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:24:28.111 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:24.625+0000 7ff74bfff700 1 --2- 172.21.15.52:0/2915794634 >> [v2:172.21.15.52:6800/3513075805,v1:172.21.15.52:6801/3513075805] conn(0x7ff7380b2910 0x7ff7380b4e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:24:28.111 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:24.625+0000 7ff7497fa700 1 -- 172.21.15.52:0/2915794634 <== mon.1 v2:172.21.15.173:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9121+0+0 (secure 0 0 0) 0x7ff7440e21d0 con 0x7ff74c108730 2024-05-27T07:24:28.112 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:24.626+0000 7ff74bfff700 1 --2- 172.21.15.52:0/2915794634 >> [v2:172.21.15.52:6800/3513075805,v1:172.21.15.52:6801/3513075805] conn(0x7ff7380b2910 0x7ff7380b4e20 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7ff73c009990 tx=0x7ff73c000bc0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:24:28.112 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:24.629+0000 7ff7497fa700 1 -- 172.21.15.52:0/2915794634 <== mon.1 v2:172.21.15.173:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191963 (secure 0 0 0) 0x7ff744021020 con 0x7ff74c108730 2024-05-27T07:24:28.112 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:24.861+0000 7ff7523b9700 1 -- 172.21.15.52:0/2915794634 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 10} v 0) -- 0x7ff74c1a9d20 con 0x7ff74c108730 2024-05-27T07:24:28.112 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:24.861+0000 7ff7497fa700 1 -- 172.21.15.52:0/2915794634 <== mon.1 v2:172.21.15.173:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 10}]=0 v0) ==== 75+0+13 (secure 0 0 0) 0x7ff7440a9aa0 con 0x7ff74c108730 2024-05-27T07:24:28.112 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:24.864+0000 7ff7523b9700 1 -- 172.21.15.52:0/2915794634 >> [v2:172.21.15.52:6800/3513075805,v1:172.21.15.52:6801/3513075805] conn(0x7ff7380b2910 msgr2=0x7ff7380b4e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:24:28.112 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:24.864+0000 7ff7523b9700 1 --2- 172.21.15.52:0/2915794634 >> [v2:172.21.15.52:6800/3513075805,v1:172.21.15.52:6801/3513075805] conn(0x7ff7380b2910 0x7ff7380b4e20 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7ff73c009990 tx=0x7ff73c000bc0 comp rx=0 tx=0).stop 2024-05-27T07:24:28.112 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:24.864+0000 7ff7523b9700 1 -- 172.21.15.52:0/2915794634 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7ff74c108730 msgr2=0x7ff74c0ffd40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:24:28.112 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:24.864+0000 7ff7523b9700 1 --2- 172.21.15.52:0/2915794634 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7ff74c108730 0x7ff74c0ffd40 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7ff74400ab40 tx=0x7ff74400ae50 comp rx=0 tx=0).stop 2024-05-27T07:24:28.112 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:24.864+0000 7ff7523b9700 1 -- 172.21.15.52:0/2915794634 shutdown_connections 2024-05-27T07:24:28.112 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:24.864+0000 7ff7523b9700 1 --2- 172.21.15.52:0/2915794634 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7ff74c108730 0x7ff74c0ffd40 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:24:28.112 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:24.864+0000 7ff7523b9700 1 --2- 172.21.15.52:0/2915794634 >> [v2:172.21.15.52:6800/3513075805,v1:172.21.15.52:6801/3513075805] conn(0x7ff7380b2910 0x7ff7380b4e20 unknown :-1 s=CLOSED pgs=195 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:24:28.112 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:24.864+0000 7ff7523b9700 1 --2- 172.21.15.52:0/2915794634 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7ff74c1034b0 0x7ff74c0ff2c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:24:28.113 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:24.864+0000 7ff7523b9700 1 --2- 172.21.15.52:0/2915794634 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff74c107de0 0x7ff74c0ff800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:24:28.113 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:24.864+0000 7ff7523b9700 1 -- 172.21.15.52:0/2915794634 >> 172.21.15.52:0/2915794634 conn(0x7ff74c0fede0 msgr2=0x7ff74c109690 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:24:28.113 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:24.864+0000 7ff7523b9700 1 -- 172.21.15.52:0/2915794634 shutdown_connections 2024-05-27T07:24:28.113 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:24.864+0000 7ff7523b9700 1 -- 172.21.15.52:0/2915794634 wait complete. 2024-05-27T07:24:28.190 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:24:27 smithi105 ceph-mon[35977]: from='client.? 172.21.15.52:0/1864990731' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 8}]: dispatch 2024-05-27T07:24:28.810 INFO:tasks.cephadm.ceph_manager.ceph:need seq 279172874287 got 279172874291 for osd.10 2024-05-27T07:24:28.810 DEBUG:teuthology.parallel:result is None 2024-05-27T07:24:29.101 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:24:28 smithi173 ceph-mon[37000]: pgmap v379: 129 pgs: 129 active+clean; 647 KiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-05-27T07:24:29.102 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:24:28 smithi173 ceph-mon[37000]: from='client.? 172.21.15.52:0/3549775543' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 11}]: dispatch 2024-05-27T07:24:29.126 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:24:28 smithi052 ceph-mon[31955]: pgmap v379: 129 pgs: 129 active+clean; 647 KiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-05-27T07:24:29.126 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:24:28 smithi052 ceph-mon[31955]: from='client.? 172.21.15.52:0/3549775543' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 11}]: dispatch 2024-05-27T07:24:29.190 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:24:28 smithi105 ceph-mon[35977]: pgmap v379: 129 pgs: 129 active+clean; 647 KiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-05-27T07:24:29.190 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:24:28 smithi105 ceph-mon[35977]: from='client.? 172.21.15.52:0/3549775543' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 11}]: dispatch 2024-05-27T07:24:29.437 INFO:tasks.cephadm.ceph_manager.ceph:need seq 304942678057 got 304942678064 for osd.11 2024-05-27T07:24:29.437 DEBUG:teuthology.parallel:result is None 2024-05-27T07:24:29.869 INFO:tasks.cephadm.ceph_manager.ceph:need seq 231928234042 got 231928234047 for osd.8 2024-05-27T07:24:29.869 DEBUG:teuthology.parallel:result is None 2024-05-27T07:24:30.103 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:30.112+0000 7feddd748700 1 -- 172.21.15.52:0/3721575708 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fedd8107de0 msgr2=0x7fedd81081f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:24:30.103 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:30.112+0000 7feddd748700 1 --2- 172.21.15.52:0/3721575708 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fedd8107de0 0x7fedd81081f0 secure :-1 s=READY pgs=294 cs=0 l=1 rev1=1 crypto rx=0x7fedcc0099c0 tx=0x7fedcc009cd0 comp rx=0 tx=0).stop 2024-05-27T07:24:30.104 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:30.112+0000 7feddd748700 1 -- 172.21.15.52:0/3721575708 shutdown_connections 2024-05-27T07:24:30.104 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:30.112+0000 7feddd748700 1 --2- 172.21.15.52:0/3721575708 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fedd8108730 0x7fedd8110090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:24:30.104 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:30.112+0000 7feddd748700 1 --2- 172.21.15.52:0/3721575708 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fedd81034b0 0x7fedd81038c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:24:30.104 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:30.112+0000 7feddd748700 1 --2- 172.21.15.52:0/3721575708 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fedd8107de0 0x7fedd81081f0 unknown :-1 s=CLOSED pgs=294 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:24:30.104 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:30.112+0000 7feddd748700 1 -- 172.21.15.52:0/3721575708 >> 172.21.15.52:0/3721575708 conn(0x7fedd80fedb0 msgr2=0x7fedd81011d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:24:30.104 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:30.112+0000 7feddd748700 1 -- 172.21.15.52:0/3721575708 shutdown_connections 2024-05-27T07:24:30.104 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:30.112+0000 7feddd748700 1 -- 172.21.15.52:0/3721575708 wait complete. 2024-05-27T07:24:30.105 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:30.113+0000 7feddd748700 1 Processor -- start 2024-05-27T07:24:30.105 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:30.114+0000 7feddd748700 1 -- start start 2024-05-27T07:24:30.106 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:30.114+0000 7feddd748700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fedd81034b0 0x7fedd8199db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:24:30.106 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:30.114+0000 7feddd748700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fedd8107de0 0x7fedd819a350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:24:30.106 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:30.114+0000 7feddd748700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fedd8108730 0x7fedd819e5e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:24:30.106 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:30.114+0000 7feddd748700 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x7fedd8113340 con 0x7fedd8107de0 2024-05-27T07:24:30.106 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:30.114+0000 7feddd748700 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7fedd81131c0 con 0x7fedd81034b0 2024-05-27T07:24:30.106 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:30.114+0000 7feddd748700 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7fedd81134c0 con 0x7fedd8108730 2024-05-27T07:24:30.106 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:30.114+0000 7fedd67fc700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fedd8107de0 0x7fedd819a350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:24:30.106 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:30.114+0000 7fedd67fc700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fedd8107de0 0x7fedd819a350 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.52:40042/0 (socket says 172.21.15.52:40042) 2024-05-27T07:24:30.106 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:30.114+0000 7fedd67fc700 1 -- 172.21.15.52:0/1395290848 learned_addr learned my addr 172.21.15.52:0/1395290848 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-05-27T07:24:30.106 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:30.114+0000 7fedd6ffd700 1 --2- 172.21.15.52:0/1395290848 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fedd81034b0 0x7fedd8199db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:24:30.106 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:30.114+0000 7fedd77fe700 1 --2- 172.21.15.52:0/1395290848 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fedd8108730 0x7fedd819e5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:24:30.106 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:30.115+0000 7fedd67fc700 1 -- 172.21.15.52:0/1395290848 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fedd8108730 msgr2=0x7fedd819e5e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:24:30.107 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:30.115+0000 7fedd67fc700 1 --2- 172.21.15.52:0/1395290848 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fedd8108730 0x7fedd819e5e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:24:30.107 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:30.115+0000 7fedd67fc700 1 -- 172.21.15.52:0/1395290848 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fedd81034b0 msgr2=0x7fedd8199db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:24:30.107 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:30.115+0000 7fedd67fc700 1 --2- 172.21.15.52:0/1395290848 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fedd81034b0 0x7fedd8199db0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:24:30.107 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:30.115+0000 7fedd67fc700 1 -- 172.21.15.52:0/1395290848 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fedd819ebd0 con 0x7fedd8107de0 2024-05-27T07:24:30.107 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:30.115+0000 7fedd67fc700 1 --2- 172.21.15.52:0/1395290848 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fedd8107de0 0x7fedd819a350 secure :-1 s=READY pgs=295 cs=0 l=1 rev1=1 crypto rx=0x7fedcc006010 tx=0x7fedcc0047b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:24:30.107 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:30.115+0000 7fedbffff700 1 -- 172.21.15.52:0/1395290848 <== mon.0 v2:172.21.15.52:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fedcc029070 con 0x7fedd8107de0 2024-05-27T07:24:30.108 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:30.115+0000 7fedbffff700 1 -- 172.21.15.52:0/1395290848 <== mon.0 v2:172.21.15.52:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fedcc019cc0 con 0x7fedd8107de0 2024-05-27T07:24:30.108 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:30.115+0000 7feddd748700 1 -- 172.21.15.52:0/1395290848 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fedd819ee60 con 0x7fedd8107de0 2024-05-27T07:24:30.108 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:30.115+0000 7feddd748700 1 -- 172.21.15.52:0/1395290848 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fedd819f340 con 0x7fedd8107de0 2024-05-27T07:24:30.108 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:30.115+0000 7fedbffff700 1 -- 172.21.15.52:0/1395290848 <== mon.0 v2:172.21.15.52:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fedcc0041c0 con 0x7fedd8107de0 2024-05-27T07:24:30.108 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:30.116+0000 7feddd748700 1 -- 172.21.15.52:0/1395290848 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fedd80512b0 con 0x7fedd8107de0 2024-05-27T07:24:30.109 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:30.118+0000 7fedbffff700 1 -- 172.21.15.52:0/1395290848 <== mon.0 v2:172.21.15.52:3300/0 4 ==== mgrmap(e 18) ==== 149652+0+0 (secure 0 0 0) 0x7fedcc02c070 con 0x7fedd8107de0 2024-05-27T07:24:30.110 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:30.119+0000 7fedbffff700 1 --2- 172.21.15.52:0/1395290848 >> [v2:172.21.15.52:6800/3513075805,v1:172.21.15.52:6801/3513075805] conn(0x7fedc00b2880 0x7fedc00b4d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:24:30.110 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:30.119+0000 7fedd6ffd700 1 --2- 172.21.15.52:0/1395290848 >> [v2:172.21.15.52:6800/3513075805,v1:172.21.15.52:6801/3513075805] conn(0x7fedc00b2880 0x7fedc00b4d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:24:30.110 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:30.119+0000 7fedbffff700 1 -- 172.21.15.52:0/1395290848 <== mon.0 v2:172.21.15.52:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9121+0+0 (secure 0 0 0) 0x7fedcc0e55c0 con 0x7fedd8107de0 2024-05-27T07:24:30.110 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:30.120+0000 7fedd6ffd700 1 --2- 172.21.15.52:0/1395290848 >> [v2:172.21.15.52:6800/3513075805,v1:172.21.15.52:6801/3513075805] conn(0x7fedc00b2880 0x7fedc00b4d90 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7fedc8005950 tx=0x7fedc800b410 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:24:30.114 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:30.123+0000 7fedbffff700 1 -- 172.21.15.52:0/1395290848 <== mon.0 v2:172.21.15.52:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191963 (secure 0 0 0) 0x7fedcc061df0 con 0x7fedd8107de0 2024-05-27T07:24:30.330 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:30.340+0000 7feddd748700 1 -- 172.21.15.52:0/1395290848 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 5} v 0) -- 0x7fedd810b1c0 con 0x7fedd8107de0 2024-05-27T07:24:30.331 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:30.341+0000 7fedbffff700 1 -- 172.21.15.52:0/1395290848 <== mon.0 v2:172.21.15.52:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 5}]=0 v0) ==== 74+0+13 (secure 0 0 0) 0x7fedcc013070 con 0x7fedd8107de0 2024-05-27T07:24:30.331 INFO:teuthology.orchestra.run.smithi052.stdout:158913790030 2024-05-27T07:24:30.333 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:30.343+0000 7feddd748700 1 -- 172.21.15.52:0/1395290848 >> [v2:172.21.15.52:6800/3513075805,v1:172.21.15.52:6801/3513075805] conn(0x7fedc00b2880 msgr2=0x7fedc00b4d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:24:30.333 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:30.343+0000 7feddd748700 1 --2- 172.21.15.52:0/1395290848 >> [v2:172.21.15.52:6800/3513075805,v1:172.21.15.52:6801/3513075805] conn(0x7fedc00b2880 0x7fedc00b4d90 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7fedc8005950 tx=0x7fedc800b410 comp rx=0 tx=0).stop 2024-05-27T07:24:30.333 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:30.343+0000 7feddd748700 1 -- 172.21.15.52:0/1395290848 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fedd8107de0 msgr2=0x7fedd819a350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:24:30.333 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:30.343+0000 7feddd748700 1 --2- 172.21.15.52:0/1395290848 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fedd8107de0 0x7fedd819a350 secure :-1 s=READY pgs=295 cs=0 l=1 rev1=1 crypto rx=0x7fedcc006010 tx=0x7fedcc0047b0 comp rx=0 tx=0).stop 2024-05-27T07:24:30.333 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:30.343+0000 7feddd748700 1 -- 172.21.15.52:0/1395290848 shutdown_connections 2024-05-27T07:24:30.333 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:30.343+0000 7feddd748700 1 --2- 172.21.15.52:0/1395290848 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fedd8108730 0x7fedd819e5e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:24:30.334 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:30.343+0000 7feddd748700 1 --2- 172.21.15.52:0/1395290848 >> [v2:172.21.15.52:6800/3513075805,v1:172.21.15.52:6801/3513075805] conn(0x7fedc00b2880 0x7fedc00b4d90 unknown :-1 s=CLOSED pgs=199 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:24:30.334 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:30.343+0000 7feddd748700 1 --2- 172.21.15.52:0/1395290848 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fedd81034b0 0x7fedd8199db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:24:30.334 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:30.343+0000 7feddd748700 1 --2- 172.21.15.52:0/1395290848 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fedd8107de0 0x7fedd819a350 unknown :-1 s=CLOSED pgs=295 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:24:30.334 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:30.343+0000 7feddd748700 1 -- 172.21.15.52:0/1395290848 >> 172.21.15.52:0/1395290848 conn(0x7fedd80fedb0 msgr2=0x7fedd81011d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:24:30.334 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:30.343+0000 7feddd748700 1 -- 172.21.15.52:0/1395290848 shutdown_connections 2024-05-27T07:24:30.334 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:30.343+0000 7feddd748700 1 -- 172.21.15.52:0/1395290848 wait complete. 2024-05-27T07:24:30.916 INFO:tasks.cephadm.ceph_manager.ceph:need seq 77309411415 got 77309411421 for osd.2 2024-05-27T07:24:30.916 DEBUG:teuthology.parallel:result is None 2024-05-27T07:24:31.102 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:24:30 smithi173 ceph-mon[37000]: pgmap v380: 129 pgs: 129 active+clean; 647 KiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 255 B/s wr, 10 op/s 2024-05-27T07:24:31.102 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:24:30 smithi173 ceph-mon[37000]: mgrmap e18: x(active, since 10m), standbys: z, y 2024-05-27T07:24:31.102 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:24:30 smithi173 ceph-mon[37000]: from='client.? 172.21.15.52:0/1395290848' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 5}]: dispatch 2024-05-27T07:24:31.190 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:24:30 smithi105 ceph-mon[35977]: pgmap v380: 129 pgs: 129 active+clean; 647 KiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 255 B/s wr, 10 op/s 2024-05-27T07:24:31.190 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:24:30 smithi105 ceph-mon[35977]: mgrmap e18: x(active, since 10m), standbys: z, y 2024-05-27T07:24:31.190 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:24:30 smithi105 ceph-mon[35977]: from='client.? 172.21.15.52:0/1395290848' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 5}]: dispatch 2024-05-27T07:24:31.229 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:24:30 smithi052 ceph-mon[31955]: pgmap v380: 129 pgs: 129 active+clean; 647 KiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 255 B/s wr, 10 op/s 2024-05-27T07:24:31.229 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:24:30 smithi052 ceph-mon[31955]: mgrmap e18: x(active, since 10m), standbys: z, y 2024-05-27T07:24:31.229 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:24:30 smithi052 ceph-mon[31955]: from='client.? 172.21.15.52:0/1395290848' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 5}]: dispatch 2024-05-27T07:24:31.632 INFO:tasks.cephadm.ceph_manager.ceph:need seq 158913790023 got 158913790030 for osd.5 2024-05-27T07:24:31.632 DEBUG:teuthology.parallel:result is None 2024-05-27T07:24:31.632 INFO:tasks.cephadm.ceph_manager.ceph:waiting for clean 2024-05-27T07:24:31.633 DEBUG:teuthology.orchestra.run.smithi052:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:38f9d900c83c63799fbdbe61acc9a11b0d3554a6 shell --fsid 73df2636-1bf8-11ef-bc9b-c7b262605968 -- ceph pg dump --format=json 2024-05-27T07:24:31.900 INFO:teuthology.orchestra.run.smithi052.stderr:Inferring config /var/lib/ceph/73df2636-1bf8-11ef-bc9b-c7b262605968/mon.a/config 2024-05-27T07:24:32.942 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:24:32 smithi105 ceph-mon[35977]: pgmap v381: 129 pgs: 129 active+clean; 647 KiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 511 B/s wr, 10 op/s 2024-05-27T07:24:32.942 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:24:32 smithi105 ceph-mon[35977]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' 2024-05-27T07:24:32.942 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:24:32 smithi105 ceph-mon[35977]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' 2024-05-27T07:24:32.942 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:24:32 smithi105 ceph-mon[35977]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T07:24:32.942 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:24:32 smithi105 ceph-mon[35977]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-27T07:24:32.942 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:24:32 smithi105 ceph-mon[35977]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' 2024-05-27T07:24:32.979 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:24:32 smithi052 ceph-mon[31955]: pgmap v381: 129 pgs: 129 active+clean; 647 KiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 511 B/s wr, 10 op/s 2024-05-27T07:24:32.979 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:24:32 smithi052 ceph-mon[31955]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' 2024-05-27T07:24:32.979 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:24:32 smithi052 ceph-mon[31955]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' 2024-05-27T07:24:32.980 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:24:32 smithi052 ceph-mon[31955]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T07:24:32.980 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:24:32 smithi052 ceph-mon[31955]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-27T07:24:32.980 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:24:32 smithi052 ceph-mon[31955]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' 2024-05-27T07:24:33.102 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:24:32 smithi173 ceph-mon[37000]: pgmap v381: 129 pgs: 129 active+clean; 647 KiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 511 B/s wr, 10 op/s 2024-05-27T07:24:33.102 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:24:32 smithi173 ceph-mon[37000]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' 2024-05-27T07:24:33.102 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:24:32 smithi173 ceph-mon[37000]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' 2024-05-27T07:24:33.102 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:24:32 smithi173 ceph-mon[37000]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T07:24:33.102 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:24:32 smithi173 ceph-mon[37000]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-27T07:24:33.102 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:24:32 smithi173 ceph-mon[37000]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' 2024-05-27T07:24:33.639 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:33.581+0000 7f9b71c3a700 1 -- 172.21.15.52:0/2758351628 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f9b6c108010 msgr2=0x7f9b6c108420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:24:33.639 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:33.581+0000 7f9b71c3a700 1 --2- 172.21.15.52:0/2758351628 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f9b6c108010 0x7f9b6c108420 secure :-1 s=READY pgs=296 cs=0 l=1 rev1=1 crypto rx=0x7f9b540099c0 tx=0x7f9b54009cd0 comp rx=0 tx=0).stop 2024-05-27T07:24:33.639 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:33.581+0000 7f9b71c3a700 1 -- 172.21.15.52:0/2758351628 shutdown_connections 2024-05-27T07:24:33.639 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:33.581+0000 7f9b71c3a700 1 --2- 172.21.15.52:0/2758351628 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f9b6c108960 0x7f9b6c1102c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:24:33.639 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:33.581+0000 7f9b71c3a700 1 --2- 172.21.15.52:0/2758351628 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f9b6c103c00 0x7f9b6c104010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:24:33.639 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:33.581+0000 7f9b71c3a700 1 --2- 172.21.15.52:0/2758351628 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f9b6c108010 0x7f9b6c108420 unknown :-1 s=CLOSED pgs=296 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:24:33.639 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:33.581+0000 7f9b71c3a700 1 -- 172.21.15.52:0/2758351628 >> 172.21.15.52:0/2758351628 conn(0x7f9b6c0fe4d0 msgr2=0x7f9b6c100920 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:24:33.639 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:33.581+0000 7f9b71c3a700 1 -- 172.21.15.52:0/2758351628 shutdown_connections 2024-05-27T07:24:33.639 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:33.581+0000 7f9b71c3a700 1 -- 172.21.15.52:0/2758351628 wait complete. 2024-05-27T07:24:33.640 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:33.582+0000 7f9b71c3a700 1 Processor -- start 2024-05-27T07:24:33.640 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:33.582+0000 7f9b71c3a700 1 -- start start 2024-05-27T07:24:33.640 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:33.582+0000 7f9b71c3a700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f9b6c103c00 0x7f9b6c199f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:24:33.640 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:33.582+0000 7f9b71c3a700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f9b6c108010 0x7f9b6c19a4c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:24:33.640 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:33.583+0000 7f9b71c3a700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f9b6c108960 0x7f9b6c19e750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:24:33.640 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:33.583+0000 7f9b71c3a700 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x7f9b6c113570 con 0x7f9b6c108010 2024-05-27T07:24:33.640 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:33.583+0000 7f9b71c3a700 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f9b6c1133f0 con 0x7f9b6c103c00 2024-05-27T07:24:33.640 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:33.583+0000 7f9b71c3a700 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f9b6c1136f0 con 0x7f9b6c108960 2024-05-27T07:24:33.640 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:33.583+0000 7f9b62ffd700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f9b6c108010 0x7f9b6c19a4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:24:33.640 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:33.583+0000 7f9b62ffd700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f9b6c108010 0x7f9b6c19a4c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.52:49928/0 (socket says 172.21.15.52:49928) 2024-05-27T07:24:33.640 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:33.583+0000 7f9b62ffd700 1 -- 172.21.15.52:0/195257365 learned_addr learned my addr 172.21.15.52:0/195257365 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-05-27T07:24:33.640 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:33.583+0000 7f9b6bfff700 1 --2- 172.21.15.52:0/195257365 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f9b6c108960 0x7f9b6c19e750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:24:33.640 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:33.583+0000 7f9b6b7fe700 1 --2- 172.21.15.52:0/195257365 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f9b6c103c00 0x7f9b6c199f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:24:33.640 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:33.583+0000 7f9b62ffd700 1 -- 172.21.15.52:0/195257365 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f9b6c108960 msgr2=0x7f9b6c19e750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:24:33.641 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:33.583+0000 7f9b62ffd700 1 --2- 172.21.15.52:0/195257365 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f9b6c108960 0x7f9b6c19e750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:24:33.641 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:33.583+0000 7f9b62ffd700 1 -- 172.21.15.52:0/195257365 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f9b6c103c00 msgr2=0x7f9b6c199f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:24:33.641 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:33.583+0000 7f9b62ffd700 1 --2- 172.21.15.52:0/195257365 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f9b6c103c00 0x7f9b6c199f20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:24:33.641 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:33.583+0000 7f9b62ffd700 1 -- 172.21.15.52:0/195257365 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9b6c19edf0 con 0x7f9b6c108010 2024-05-27T07:24:33.641 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:33.583+0000 7f9b62ffd700 1 --2- 172.21.15.52:0/195257365 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f9b6c108010 0x7f9b6c19a4c0 secure :-1 s=READY pgs=297 cs=0 l=1 rev1=1 crypto rx=0x7f9b54009f50 tx=0x7f9b540037c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:24:33.641 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:33.583+0000 7f9b697fa700 1 -- 172.21.15.52:0/195257365 <== mon.0 v2:172.21.15.52:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9b54027070 con 0x7f9b6c108010 2024-05-27T07:24:33.641 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:33.583+0000 7f9b71c3a700 1 -- 172.21.15.52:0/195257365 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9b6c19f080 con 0x7f9b6c108010 2024-05-27T07:24:33.641 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:33.583+0000 7f9b697fa700 1 -- 172.21.15.52:0/195257365 <== mon.0 v2:172.21.15.52:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f9b54004090 con 0x7f9b6c108010 2024-05-27T07:24:33.641 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:33.583+0000 7f9b697fa700 1 -- 172.21.15.52:0/195257365 <== mon.0 v2:172.21.15.52:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9b54022e60 con 0x7f9b6c108010 2024-05-27T07:24:33.641 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:33.583+0000 7f9b71c3a700 1 -- 172.21.15.52:0/195257365 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9b6c19f560 con 0x7f9b6c108010 2024-05-27T07:24:33.641 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:33.584+0000 7f9b71c3a700 1 -- 172.21.15.52:0/195257365 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9b6c0512b0 con 0x7f9b6c108010 2024-05-27T07:24:33.641 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:33.587+0000 7f9b697fa700 1 -- 172.21.15.52:0/195257365 <== mon.0 v2:172.21.15.52:3300/0 4 ==== mgrmap(e 18) ==== 149652+0+0 (secure 0 0 0) 0x7f9b5402a070 con 0x7f9b6c108010 2024-05-27T07:24:33.641 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:33.588+0000 7f9b697fa700 1 --2- 172.21.15.52:0/195257365 >> [v2:172.21.15.52:6800/3513075805,v1:172.21.15.52:6801/3513075805] conn(0x7f9b580b2880 0x7f9b580b4d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:24:33.641 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:33.588+0000 7f9b697fa700 1 -- 172.21.15.52:0/195257365 <== mon.0 v2:172.21.15.52:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9121+0+0 (secure 0 0 0) 0x7f9b540e3810 con 0x7f9b6c108010 2024-05-27T07:24:33.642 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:33.588+0000 7f9b6b7fe700 1 --2- 172.21.15.52:0/195257365 >> [v2:172.21.15.52:6800/3513075805,v1:172.21.15.52:6801/3513075805] conn(0x7f9b580b2880 0x7f9b580b4d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:24:33.642 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:33.588+0000 7f9b6b7fe700 1 --2- 172.21.15.52:0/195257365 >> [v2:172.21.15.52:6800/3513075805,v1:172.21.15.52:6801/3513075805] conn(0x7f9b580b2880 0x7f9b580b4d90 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7f9b5c005fd0 tx=0x7f9b5c00a560 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:24:33.642 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:33.591+0000 7f9b697fa700 1 -- 172.21.15.52:0/195257365 <== mon.0 v2:172.21.15.52:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191963 (secure 0 0 0) 0x7f9b54031080 con 0x7f9b6c108010 2024-05-27T07:24:33.796 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:33.806+0000 7f9b71c3a700 1 -- 172.21.15.52:0/195257365 --> [v2:172.21.15.52:6800/3513075805,v1:172.21.15.52:6801/3513075805] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f9b6c19b280 con 0x7f9b580b2880 2024-05-27T07:24:33.805 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:33.815+0000 7f9b697fa700 1 -- 172.21.15.52:0/195257365 <== mgr.14152 v2:172.21.15.52:6800/3513075805 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+340899 (secure 0 0 0) 0x7f9b6c19b280 con 0x7f9b580b2880 2024-05-27T07:24:33.810 INFO:teuthology.orchestra.run.smithi052.stdout: 2024-05-27T07:24:33.813 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:33.819+0000 7f9b71c3a700 1 -- 172.21.15.52:0/195257365 >> [v2:172.21.15.52:6800/3513075805,v1:172.21.15.52:6801/3513075805] conn(0x7f9b580b2880 msgr2=0x7f9b580b4d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:24:33.813 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:33.819+0000 7f9b71c3a700 1 --2- 172.21.15.52:0/195257365 >> [v2:172.21.15.52:6800/3513075805,v1:172.21.15.52:6801/3513075805] conn(0x7f9b580b2880 0x7f9b580b4d90 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7f9b5c005fd0 tx=0x7f9b5c00a560 comp rx=0 tx=0).stop 2024-05-27T07:24:33.813 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:33.819+0000 7f9b71c3a700 1 -- 172.21.15.52:0/195257365 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f9b6c108010 msgr2=0x7f9b6c19a4c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:24:33.813 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:33.819+0000 7f9b71c3a700 1 --2- 172.21.15.52:0/195257365 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f9b6c108010 0x7f9b6c19a4c0 secure :-1 s=READY pgs=297 cs=0 l=1 rev1=1 crypto rx=0x7f9b54009f50 tx=0x7f9b540037c0 comp rx=0 tx=0).stop 2024-05-27T07:24:33.813 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:33.819+0000 7f9b71c3a700 1 -- 172.21.15.52:0/195257365 shutdown_connections 2024-05-27T07:24:33.813 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:33.819+0000 7f9b71c3a700 1 --2- 172.21.15.52:0/195257365 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f9b6c108960 0x7f9b6c19e750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:24:33.813 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:33.819+0000 7f9b71c3a700 1 --2- 172.21.15.52:0/195257365 >> [v2:172.21.15.52:6800/3513075805,v1:172.21.15.52:6801/3513075805] conn(0x7f9b580b2880 0x7f9b580b4d90 unknown :-1 s=CLOSED pgs=200 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:24:33.813 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:33.819+0000 7f9b71c3a700 1 --2- 172.21.15.52:0/195257365 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f9b6c103c00 0x7f9b6c199f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:24:33.813 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:33.819+0000 7f9b71c3a700 1 --2- 172.21.15.52:0/195257365 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f9b6c108010 0x7f9b6c19a4c0 unknown :-1 s=CLOSED pgs=297 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:24:33.813 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:33.819+0000 7f9b71c3a700 1 -- 172.21.15.52:0/195257365 >> 172.21.15.52:0/195257365 conn(0x7f9b6c0fe4d0 msgr2=0x7f9b6c1077b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:24:33.813 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:33.819+0000 7f9b71c3a700 1 -- 172.21.15.52:0/195257365 shutdown_connections 2024-05-27T07:24:33.813 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:33.819+0000 7f9b71c3a700 1 -- 172.21.15.52:0/195257365 wait complete. 2024-05-27T07:24:33.813 INFO:teuthology.orchestra.run.smithi052.stderr:dumped all 2024-05-27T07:24:34.469 INFO:teuthology.orchestra.run.smithi052.stdout:{"pg_ready":true,"pg_map":{"version":382,"stamp":"2024-05-27T07:24:33.688034+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":662879,"num_objects":102,"num_object_clones":0,"num_object_copies":308,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":102,"num_whiteouts":0,"num_read":1063,"num_read_kb":638,"num_write":382,"num_write_kb":2304,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":2951862,"num_keys_recovered":0,"num_objects_omap":66,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":801,"ondisk_log_size":801,"up":419,"acting":419,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":419,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":378012,"kb_used_data":12212,"kb_used_omap":18,"kb_used_meta":365677,"kb_avail":1124563812,"statfs":{"total":1151940427776,"available":1151553343488,"internally_reserved":0,"allocated":12505088,"data_stored":8186897,"data_compressed":497532,"data_compressed_allocated":675840,"data_compressed_original":2850816,"omap_allocated":19143,"internal_metadata":374453561},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":19,"apply_latency_ms":19,"commit_latency_ns":19000000,"apply_latency_ns":19000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":455,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":112,"num_read_kb":52,"num_write":6,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003771"},"pg_stats":[{"pgid":"3.3f","version":"90'8","reported_seq":47,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:24:29.683427+0000","last_change":"2024-05-27T07:21:48.595870+0000","last_active":"2024-05-27T07:24:29.683427+0000","last_peered":"2024-05-27T07:24:29.683427+0000","last_clean":"2024-05-27T07:24:29.683427+0000","last_became_active":"2024-05-27T07:21:48.595020+0000","last_became_peered":"2024-05-27T07:21:48.595020+0000","last_unstale":"2024-05-27T07:24:29.683427+0000","last_undegraded":"2024-05-27T07:24:29.683427+0000","last_fullsized":"2024-05-27T07:24:29.683427+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:21:47.565641+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:21:47.565641+0000","last_clean_scrub_stamp":"2024-05-27T07:21:47.565641+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T16:43:08.920344+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.3e","version":"90'4","reported_seq":43,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:24:29.672501+0000","last_change":"2024-05-27T07:21:48.590237+0000","last_active":"2024-05-27T07:24:29.672501+0000","last_peered":"2024-05-27T07:24:29.672501+0000","last_clean":"2024-05-27T07:24:29.672501+0000","last_became_active":"2024-05-27T07:21:48.589876+0000","last_became_peered":"2024-05-27T07:21:48.589876+0000","last_unstale":"2024-05-27T07:24:29.672501+0000","last_undegraded":"2024-05-27T07:24:29.672501+0000","last_fullsized":"2024-05-27T07:24:29.672501+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:21:47.565641+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:21:47.565641+0000","last_clean_scrub_stamp":"2024-05-27T07:21:47.565641+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T16:23:08.881871+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,11,10],"acting":[0,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3d","version":"90'7","reported_seq":46,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:24:29.683006+0000","last_change":"2024-05-27T07:21:48.597744+0000","last_active":"2024-05-27T07:24:29.683006+0000","last_peered":"2024-05-27T07:24:29.683006+0000","last_clean":"2024-05-27T07:24:29.683006+0000","last_became_active":"2024-05-27T07:21:48.597410+0000","last_became_peered":"2024-05-27T07:21:48.597410+0000","last_unstale":"2024-05-27T07:24:29.683006+0000","last_undegraded":"2024-05-27T07:24:29.683006+0000","last_fullsized":"2024-05-27T07:24:29.683006+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:21:47.565641+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:21:47.565641+0000","last_clean_scrub_stamp":"2024-05-27T07:21:47.565641+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T19:06:49.643152+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"90'8","reported_seq":24,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:24:29.674553+0000","last_change":"2024-05-27T07:22:30.800824+0000","last_active":"2024-05-27T07:24:29.674553+0000","last_peered":"2024-05-27T07:24:29.674553+0000","last_clean":"2024-05-27T07:24:29.674553+0000","last_became_active":"2024-05-27T07:22:30.800415+0000","last_became_peered":"2024-05-27T07:22:30.800415+0000","last_unstale":"2024-05-27T07:24:29.674553+0000","last_undegraded":"2024-05-27T07:24:29.674553+0000","last_fullsized":"2024-05-27T07:24:29.674553+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:21:47.565641+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:21:47.565641+0000","last_clean_scrub_stamp":"2024-05-27T07:21:47.565641+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T11:06:10.451686+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,2],"acting":[9,8,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.3b","version":"90'4","reported_seq":43,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:24:29.652869+0000","last_change":"2024-05-27T07:21:48.583219+0000","last_active":"2024-05-27T07:24:29.652869+0000","last_peered":"2024-05-27T07:24:29.652869+0000","last_clean":"2024-05-27T07:24:29.652869+0000","last_became_active":"2024-05-27T07:21:48.582813+0000","last_became_peered":"2024-05-27T07:21:48.582813+0000","last_unstale":"2024-05-27T07:24:29.652869+0000","last_undegraded":"2024-05-27T07:24:29.652869+0000","last_fullsized":"2024-05-27T07:24:29.652869+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:21:47.565641+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:21:47.565641+0000","last_clean_scrub_stamp":"2024-05-27T07:21:47.565641+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T18:05:05.770965+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,3,6],"acting":[8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.3a","version":"90'2","reported_seq":40,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:24:19.638248+0000","last_change":"2024-05-27T07:21:48.592049+0000","last_active":"2024-05-27T07:24:19.638248+0000","last_peered":"2024-05-27T07:24:19.638248+0000","last_clean":"2024-05-27T07:24:19.638248+0000","last_became_active":"2024-05-27T07:21:48.591677+0000","last_became_peered":"2024-05-27T07:21:48.591677+0000","last_unstale":"2024-05-27T07:24:19.638248+0000","last_undegraded":"2024-05-27T07:24:19.638248+0000","last_fullsized":"2024-05-27T07:24:19.638248+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:21:47.565641+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:21:47.565641+0000","last_clean_scrub_stamp":"2024-05-27T07:21:47.565641+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T16:25:49.755893+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,4],"acting":[10,9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.39","version":"90'5","reported_seq":42,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:24:29.672343+0000","last_change":"2024-05-27T07:21:48.597876+0000","last_active":"2024-05-27T07:24:29.672343+0000","last_peered":"2024-05-27T07:24:29.672343+0000","last_clean":"2024-05-27T07:24:29.672343+0000","last_became_active":"2024-05-27T07:21:48.597566+0000","last_became_peered":"2024-05-27T07:21:48.597566+0000","last_unstale":"2024-05-27T07:24:29.672343+0000","last_undegraded":"2024-05-27T07:24:29.672343+0000","last_fullsized":"2024-05-27T07:24:29.672343+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:21:47.565641+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:21:47.565641+0000","last_clean_scrub_stamp":"2024-05-27T07:21:47.565641+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T09:30:43.438209+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,5],"acting":[7,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"90'9","reported_seq":48,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:24:29.675504+0000","last_change":"2024-05-27T07:21:48.593412+0000","last_active":"2024-05-27T07:24:29.675504+0000","last_peered":"2024-05-27T07:24:29.675504+0000","last_clean":"2024-05-27T07:24:29.675504+0000","last_became_active":"2024-05-27T07:21:48.593070+0000","last_became_peered":"2024-05-27T07:21:48.593070+0000","last_unstale":"2024-05-27T07:24:29.675504+0000","last_undegraded":"2024-05-27T07:24:29.675504+0000","last_fullsized":"2024-05-27T07:24:29.675504+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:21:47.565641+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:21:47.565641+0000","last_clean_scrub_stamp":"2024-05-27T07:21:47.565641+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T14:53:49.382796+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,10],"acting":[3,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"90'11","reported_seq":50,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:24:29.684456+0000","last_change":"2024-05-27T07:21:48.607244+0000","last_active":"2024-05-27T07:24:29.684456+0000","last_peered":"2024-05-27T07:24:29.684456+0000","last_clean":"2024-05-27T07:24:29.684456+0000","last_became_active":"2024-05-27T07:21:48.606238+0000","last_became_peered":"2024-05-27T07:21:48.606238+0000","last_unstale":"2024-05-27T07:24:29.684456+0000","last_undegraded":"2024-05-27T07:24:29.684456+0000","last_fullsized":"2024-05-27T07:24:29.684456+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:21:47.565641+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:21:47.565641+0000","last_clean_scrub_stamp":"2024-05-27T07:21:47.565641+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T12:36:33.142111+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,11],"acting":[1,5,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.36","version":"90'11","reported_seq":40,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:24:14.637717+0000","last_change":"2024-05-27T07:21:48.593275+0000","last_active":"2024-05-27T07:24:14.637717+0000","last_peered":"2024-05-27T07:24:14.637717+0000","last_clean":"2024-05-27T07:24:14.637717+0000","last_became_active":"2024-05-27T07:21:48.592890+0000","last_became_peered":"2024-05-27T07:21:48.592890+0000","last_unstale":"2024-05-27T07:24:14.637717+0000","last_undegraded":"2024-05-27T07:24:14.637717+0000","last_fullsized":"2024-05-27T07:24:14.637717+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:21:47.565641+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:21:47.565641+0000","last_clean_scrub_stamp":"2024-05-27T07:21:47.565641+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T13:47:20.827824+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,4],"acting":[10,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.35","version":"90'6","reported_seq":43,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:24:29.685097+0000","last_change":"2024-05-27T07:21:48.603424+0000","last_active":"2024-05-27T07:24:29.685097+0000","last_peered":"2024-05-27T07:24:29.685097+0000","last_clean":"2024-05-27T07:24:29.685097+0000","last_became_active":"2024-05-27T07:21:48.602962+0000","last_became_peered":"2024-05-27T07:21:48.602962+0000","last_unstale":"2024-05-27T07:24:29.685097+0000","last_undegraded":"2024-05-27T07:24:29.685097+0000","last_fullsized":"2024-05-27T07:24:29.685097+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:21:47.565641+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:21:47.565641+0000","last_clean_scrub_stamp":"2024-05-27T07:21:47.565641+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T19:19:42.063063+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,8,1],"acting":[7,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.34","version":"90'7","reported_seq":46,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:24:29.672689+0000","last_change":"2024-05-27T07:21:48.589323+0000","last_active":"2024-05-27T07:24:29.672689+0000","last_peered":"2024-05-27T07:24:29.672689+0000","last_clean":"2024-05-27T07:24:29.672689+0000","last_became_active":"2024-05-27T07:21:48.587435+0000","last_became_peered":"2024-05-27T07:21:48.587435+0000","last_unstale":"2024-05-27T07:24:29.672689+0000","last_undegraded":"2024-05-27T07:24:29.672689+0000","last_fullsized":"2024-05-27T07:24:29.672689+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:21:47.565641+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:21:47.565641+0000","last_clean_scrub_stamp":"2024-05-27T07:21:47.565641+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T13:34:44.340376+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,8],"acting":[4,10,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"90'10","reported_seq":49,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:24:29.660510+0000","last_change":"2024-05-27T07:21:48.593759+0000","last_active":"2024-05-27T07:24:29.660510+0000","last_peered":"2024-05-27T07:24:29.660510+0000","last_clean":"2024-05-27T07:24:29.660510+0000","last_became_active":"2024-05-27T07:21:48.593099+0000","last_became_peered":"2024-05-27T07:21:48.593099+0000","last_unstale":"2024-05-27T07:24:29.660510+0000","last_undegraded":"2024-05-27T07:24:29.660510+0000","last_fullsized":"2024-05-27T07:24:29.660510+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:21:47.565641+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:21:47.565641+0000","last_clean_scrub_stamp":"2024-05-27T07:21:47.565641+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T13:49:00.390792+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,5],"acting":[6,10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.32","version":"90'7","reported_seq":46,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:24:29.675244+0000","last_change":"2024-05-27T07:21:48.587486+0000","last_active":"2024-05-27T07:24:29.675244+0000","last_peered":"2024-05-27T07:24:29.675244+0000","last_clean":"2024-05-27T07:24:29.675244+0000","last_became_active":"2024-05-27T07:21:48.587159+0000","last_became_peered":"2024-05-27T07:21:48.587159+0000","last_unstale":"2024-05-27T07:24:29.675244+0000","last_undegraded":"2024-05-27T07:24:29.675244+0000","last_fullsized":"2024-05-27T07:24:29.675244+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:21:47.565641+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:21:47.565641+0000","last_clean_scrub_stamp":"2024-05-27T07:21:47.565641+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T15:34:11.908442+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,10],"acting":[0,8,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"90'7","reported_seq":46,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:24:29.682260+0000","last_change":"2024-05-27T07:21:48.596817+0000","last_active":"2024-05-27T07:24:29.682260+0000","last_peered":"2024-05-27T07:24:29.682260+0000","last_clean":"2024-05-27T07:24:29.682260+0000","last_became_active":"2024-05-27T07:21:48.596469+0000","last_became_peered":"2024-05-27T07:21:48.596469+0000","last_unstale":"2024-05-27T07:24:29.682260+0000","last_undegraded":"2024-05-27T07:24:29.682260+0000","last_fullsized":"2024-05-27T07:24:29.682260+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:21:47.565641+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:21:47.565641+0000","last_clean_scrub_stamp":"2024-05-27T07:21:47.565641+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T12:38:41.494060+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,2],"acting":[0,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.30","version":"90'9","reported_seq":48,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:24:29.683528+0000","last_change":"2024-05-27T07:21:48.604249+0000","last_active":"2024-05-27T07:24:29.683528+0000","last_peered":"2024-05-27T07:24:29.683528+0000","last_clean":"2024-05-27T07:24:29.683528+0000","last_became_active":"2024-05-27T07:21:48.603585+0000","last_became_peered":"2024-05-27T07:21:48.603585+0000","last_unstale":"2024-05-27T07:24:29.683528+0000","last_undegraded":"2024-05-27T07:24:29.683528+0000","last_fullsized":"2024-05-27T07:24:29.683528+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:21:47.565641+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:21:47.565641+0000","last_clean_scrub_stamp":"2024-05-27T07:21:47.565641+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T07:22:14.839824+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2f","version":"90'8","reported_seq":47,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:24:29.683480+0000","last_change":"2024-05-27T07:21:48.596035+0000","last_active":"2024-05-27T07:24:29.683480+0000","last_peered":"2024-05-27T07:24:29.683480+0000","last_clean":"2024-05-27T07:24:29.683480+0000","last_became_active":"2024-05-27T07:21:48.595509+0000","last_became_peered":"2024-05-27T07:21:48.595509+0000","last_unstale":"2024-05-27T07:24:29.683480+0000","last_undegraded":"2024-05-27T07:24:29.683480+0000","last_fullsized":"2024-05-27T07:24:29.683480+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:21:47.565641+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:21:47.565641+0000","last_clean_scrub_stamp":"2024-05-27T07:21:47.565641+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T13:23:03.734982+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.2e","version":"90'10","reported_seq":49,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:24:29.682582+0000","last_change":"2024-05-27T07:21:48.590002+0000","last_active":"2024-05-27T07:24:29.682582+0000","last_peered":"2024-05-27T07:24:29.682582+0000","last_clean":"2024-05-27T07:24:29.682582+0000","last_became_active":"2024-05-27T07:21:48.589608+0000","last_became_peered":"2024-05-27T07:21:48.589608+0000","last_unstale":"2024-05-27T07:24:29.682582+0000","last_undegraded":"2024-05-27T07:24:29.682582+0000","last_fullsized":"2024-05-27T07:24:29.682582+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:21:47.565641+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:21:47.565641+0000","last_clean_scrub_stamp":"2024-05-27T07:21:47.565641+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T09:33:39.174351+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,1,10],"acting":[9,1,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.2d","version":"90'7","reported_seq":46,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:24:29.671511+0000","last_change":"2024-05-27T07:21:48.594885+0000","last_active":"2024-05-27T07:24:29.671511+0000","last_peered":"2024-05-27T07:24:29.671511+0000","last_clean":"2024-05-27T07:24:29.671511+0000","last_became_active":"2024-05-27T07:21:48.594521+0000","last_became_peered":"2024-05-27T07:21:48.594521+0000","last_unstale":"2024-05-27T07:24:29.671511+0000","last_undegraded":"2024-05-27T07:24:29.671511+0000","last_fullsized":"2024-05-27T07:24:29.671511+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:21:47.565641+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:21:47.565641+0000","last_clean_scrub_stamp":"2024-05-27T07:21:47.565641+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T08:54:59.311938+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":36,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:23:30.982372+0000","last_change":"2024-05-27T07:22:30.857546+0000","last_active":"2024-05-27T07:23:30.982372+0000","last_peered":"2024-05-27T07:23:30.982372+0000","last_clean":"2024-05-27T07:23:30.982372+0000","last_became_active":"2024-05-27T07:22:30.856808+0000","last_became_peered":"2024-05-27T07:22:30.856808+0000","last_unstale":"2024-05-27T07:23:30.982372+0000","last_undegraded":"2024-05-27T07:23:30.982372+0000","last_fullsized":"2024-05-27T07:23:30.982372+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:21:58.698952+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:21:58.698952+0000","last_clean_scrub_stamp":"2024-05-27T07:21:58.698952+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T16:09:25.480864+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,8,0],"acting":[2,3,8,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1f","version":"88'13","reported_seq":153,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:24:29.037967+0000","last_change":"2024-05-27T07:21:45.566489+0000","last_active":"2024-05-27T07:24:29.037967+0000","last_peered":"2024-05-27T07:24:29.037967+0000","last_clean":"2024-05-27T07:24:29.037967+0000","last_became_active":"2024-05-27T07:21:45.566115+0000","last_became_peered":"2024-05-27T07:21:45.566115+0000","last_unstale":"2024-05-27T07:24:29.037967+0000","last_undegraded":"2024-05-27T07:24:29.037967+0000","last_fullsized":"2024-05-27T07:24:29.037967+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:21:44.543638+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:21:44.543638+0000","last_clean_scrub_stamp":"2024-05-27T07:21:44.543638+0000","objects_scrubbed":0,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T07:24:11.735014+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":95,"num_read_kb":95,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,8],"acting":[9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.1e","version":"90'5","reported_seq":44,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:24:29.673749+0000","last_change":"2024-05-27T07:21:48.588574+0000","last_active":"2024-05-27T07:24:29.673749+0000","last_peered":"2024-05-27T07:24:29.673749+0000","last_clean":"2024-05-27T07:24:29.673749+0000","last_became_active":"2024-05-27T07:21:48.588212+0000","last_became_peered":"2024-05-27T07:21:48.588212+0000","last_unstale":"2024-05-27T07:24:29.673749+0000","last_undegraded":"2024-05-27T07:24:29.673749+0000","last_fullsized":"2024-05-27T07:24:29.673749+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:21:47.565641+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:21:47.565641+0000","last_clean_scrub_stamp":"2024-05-27T07:21:47.565641+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T12:15:16.017635+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":29,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:23:31.757134+0000","last_change":"2024-05-27T07:21:59.734617+0000","last_active":"2024-05-27T07:23:31.757134+0000","last_peered":"2024-05-27T07:23:31.757134+0000","last_clean":"2024-05-27T07:23:31.757134+0000","last_became_active":"2024-05-27T07:21:59.734162+0000","last_became_peered":"2024-05-27T07:21:59.734162+0000","last_unstale":"2024-05-27T07:23:31.757134+0000","last_undegraded":"2024-05-27T07:23:31.757134+0000","last_fullsized":"2024-05-27T07:23:31.757134+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:21:58.698952+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:21:58.698952+0000","last_clean_scrub_stamp":"2024-05-27T07:21:58.698952+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T12:40:01.375520+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,9,3],"acting":[4,6,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1e","version":"88'2","reported_seq":46,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:23:31.103539+0000","last_change":"2024-05-27T07:21:45.790907+0000","last_active":"2024-05-27T07:23:31.103539+0000","last_peered":"2024-05-27T07:23:31.103539+0000","last_clean":"2024-05-27T07:23:31.103539+0000","last_became_active":"2024-05-27T07:21:45.790451+0000","last_became_peered":"2024-05-27T07:21:45.790451+0000","last_unstale":"2024-05-27T07:23:31.103539+0000","last_undegraded":"2024-05-27T07:23:31.103539+0000","last_fullsized":"2024-05-27T07:23:31.103539+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:21:44.543638+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:21:44.543638+0000","last_clean_scrub_stamp":"2024-05-27T07:21:44.543638+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T10:21:56.524930+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"90'6","reported_seq":45,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:24:29.682781+0000","last_change":"2024-05-27T07:21:48.595975+0000","last_active":"2024-05-27T07:24:29.682781+0000","last_peered":"2024-05-27T07:24:29.682781+0000","last_clean":"2024-05-27T07:24:29.682781+0000","last_became_active":"2024-05-27T07:21:48.595608+0000","last_became_peered":"2024-05-27T07:21:48.595608+0000","last_unstale":"2024-05-27T07:24:29.682781+0000","last_undegraded":"2024-05-27T07:24:29.682781+0000","last_fullsized":"2024-05-27T07:24:29.682781+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:21:47.565641+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:21:47.565641+0000","last_clean_scrub_stamp":"2024-05-27T07:21:47.565641+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T07:24:44.797973+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,9],"acting":[0,1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":29,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:23:31.757236+0000","last_change":"2024-05-27T07:21:59.732788+0000","last_active":"2024-05-27T07:23:31.757236+0000","last_peered":"2024-05-27T07:23:31.757236+0000","last_clean":"2024-05-27T07:23:31.757236+0000","last_became_active":"2024-05-27T07:21:59.732310+0000","last_became_peered":"2024-05-27T07:21:59.732310+0000","last_unstale":"2024-05-27T07:23:31.757236+0000","last_undegraded":"2024-05-27T07:23:31.757236+0000","last_fullsized":"2024-05-27T07:23:31.757236+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:21:58.698952+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:21:58.698952+0000","last_clean_scrub_stamp":"2024-05-27T07:21:58.698952+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T13:54:51.490154+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,9,5],"acting":[4,3,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1d","version":"88'12","reported_seq":58,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:23:31.061913+0000","last_change":"2024-05-27T07:21:45.569746+0000","last_active":"2024-05-27T07:23:31.061913+0000","last_peered":"2024-05-27T07:23:31.061913+0000","last_clean":"2024-05-27T07:23:31.061913+0000","last_became_active":"2024-05-27T07:21:45.569366+0000","last_became_peered":"2024-05-27T07:21:45.569366+0000","last_unstale":"2024-05-27T07:23:31.061913+0000","last_undegraded":"2024-05-27T07:23:31.061913+0000","last_fullsized":"2024-05-27T07:23:31.061913+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:21:44.543638+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:21:44.543638+0000","last_clean_scrub_stamp":"2024-05-27T07:21:44.543638+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T15:58:32.977754+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":16,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":7,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1c","version":"90'9","reported_seq":48,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:24:29.686621+0000","last_change":"2024-05-27T07:21:48.606138+0000","last_active":"2024-05-27T07:24:29.686621+0000","last_peered":"2024-05-27T07:24:29.686621+0000","last_clean":"2024-05-27T07:24:29.686621+0000","last_became_active":"2024-05-27T07:21:48.605711+0000","last_became_peered":"2024-05-27T07:21:48.605711+0000","last_unstale":"2024-05-27T07:24:29.686621+0000","last_undegraded":"2024-05-27T07:24:29.686621+0000","last_fullsized":"2024-05-27T07:24:29.686621+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:21:47.565641+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:21:47.565641+0000","last_clean_scrub_stamp":"2024-05-27T07:21:47.565641+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T10:46:39.052387+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,6],"acting":[1,10,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":29,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:23:31.757067+0000","last_change":"2024-05-27T07:21:59.735475+0000","last_active":"2024-05-27T07:23:31.757067+0000","last_peered":"2024-05-27T07:23:31.757067+0000","last_clean":"2024-05-27T07:23:31.757067+0000","last_became_active":"2024-05-27T07:21:59.734986+0000","last_became_peered":"2024-05-27T07:21:59.734986+0000","last_unstale":"2024-05-27T07:23:31.757067+0000","last_undegraded":"2024-05-27T07:23:31.757067+0000","last_fullsized":"2024-05-27T07:23:31.757067+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:21:58.698952+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:21:58.698952+0000","last_clean_scrub_stamp":"2024-05-27T07:21:58.698952+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T14:48:28.198718+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5,3],"acting":[4,6,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1c","version":"88'5","reported_seq":51,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:23:31.061705+0000","last_change":"2024-05-27T07:21:45.572882+0000","last_active":"2024-05-27T07:23:31.061705+0000","last_peered":"2024-05-27T07:23:31.061705+0000","last_clean":"2024-05-27T07:23:31.061705+0000","last_became_active":"2024-05-27T07:21:45.572460+0000","last_became_peered":"2024-05-27T07:21:45.572460+0000","last_unstale":"2024-05-27T07:23:31.061705+0000","last_undegraded":"2024-05-27T07:23:31.061705+0000","last_fullsized":"2024-05-27T07:23:31.061705+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:21:44.543638+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:21:44.543638+0000","last_clean_scrub_stamp":"2024-05-27T07:21:44.543638+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T09:40:11.311923+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":7,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,1,5],"acting":[8,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1d","version":"90'15","reported_seq":54,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:24:29.676331+0000","last_change":"2024-05-27T07:21:48.596217+0000","last_active":"2024-05-27T07:24:29.676331+0000","last_peered":"2024-05-27T07:24:29.676331+0000","last_clean":"2024-05-27T07:24:29.676331+0000","last_became_active":"2024-05-27T07:21:48.595832+0000","last_became_peered":"2024-05-27T07:21:48.595832+0000","last_unstale":"2024-05-27T07:24:29.676331+0000","last_undegraded":"2024-05-27T07:24:29.676331+0000","last_fullsized":"2024-05-27T07:24:29.676331+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:21:47.565641+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:21:47.565641+0000","last_clean_scrub_stamp":"2024-05-27T07:21:47.565641+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T10:08:05.304404+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":29,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:23:30.982988+0000","last_change":"2024-05-27T07:21:59.729617+0000","last_active":"2024-05-27T07:23:30.982988+0000","last_peered":"2024-05-27T07:23:30.982988+0000","last_clean":"2024-05-27T07:23:30.982988+0000","last_became_active":"2024-05-27T07:21:59.729003+0000","last_became_peered":"2024-05-27T07:21:59.729003+0000","last_unstale":"2024-05-27T07:23:30.982988+0000","last_undegraded":"2024-05-27T07:23:30.982988+0000","last_fullsized":"2024-05-27T07:23:30.982988+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:21:58.698952+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:21:58.698952+0000","last_clean_scrub_stamp":"2024-05-27T07:21:58.698952+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T14:55:04.844948+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7,11],"acting":[2,5,7,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1b","version":"90'7","reported_seq":149,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:24:30.031198+0000","last_change":"2024-05-27T07:21:45.566355+0000","last_active":"2024-05-27T07:24:30.031198+0000","last_peered":"2024-05-27T07:24:30.031198+0000","last_clean":"2024-05-27T07:24:30.031198+0000","last_became_active":"2024-05-27T07:21:45.564776+0000","last_became_peered":"2024-05-27T07:21:45.564776+0000","last_unstale":"2024-05-27T07:24:30.031198+0000","last_undegraded":"2024-05-27T07:24:30.031198+0000","last_fullsized":"2024-05-27T07:24:30.031198+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:21:44.543638+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:21:44.543638+0000","last_clean_scrub_stamp":"2024-05-27T07:21:44.543638+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T12:10:41.632630+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1154,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":98,"num_read_kb":4,"num_write":4,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1a","version":"90'7","reported_seq":46,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:24:29.659499+0000","last_change":"2024-05-27T07:21:48.588837+0000","last_active":"2024-05-27T07:24:29.659499+0000","last_peered":"2024-05-27T07:24:29.659499+0000","last_clean":"2024-05-27T07:24:29.659499+0000","last_became_active":"2024-05-27T07:21:48.587149+0000","last_became_peered":"2024-05-27T07:21:48.587149+0000","last_unstale":"2024-05-27T07:24:29.659499+0000","last_undegraded":"2024-05-27T07:24:29.659499+0000","last_fullsized":"2024-05-27T07:24:29.659499+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:21:47.565641+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:21:47.565641+0000","last_clean_scrub_stamp":"2024-05-27T07:21:47.565641+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T09:20:05.124189+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,5],"acting":[8,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":29,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:23:30.982753+0000","last_change":"2024-05-27T07:21:59.730003+0000","last_active":"2024-05-27T07:23:30.982753+0000","last_peered":"2024-05-27T07:23:30.982753+0000","last_clean":"2024-05-27T07:23:30.982753+0000","last_became_active":"2024-05-27T07:21:59.729195+0000","last_became_peered":"2024-05-27T07:21:59.729195+0000","last_unstale":"2024-05-27T07:23:30.982753+0000","last_undegraded":"2024-05-27T07:23:30.982753+0000","last_fullsized":"2024-05-27T07:23:30.982753+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:21:58.698952+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:21:58.698952+0000","last_clean_scrub_stamp":"2024-05-27T07:21:58.698952+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T13:13:41.612300+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,9,5],"acting":[2,6,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1a","version":"88'7","reported_seq":52,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:23:31.528908+0000","last_change":"2024-05-27T07:21:45.567279+0000","last_active":"2024-05-27T07:23:31.528908+0000","last_peered":"2024-05-27T07:23:31.528908+0000","last_clean":"2024-05-27T07:23:31.528908+0000","last_became_active":"2024-05-27T07:21:45.566920+0000","last_became_peered":"2024-05-27T07:21:45.566920+0000","last_unstale":"2024-05-27T07:23:31.528908+0000","last_undegraded":"2024-05-27T07:23:31.528908+0000","last_fullsized":"2024-05-27T07:23:31.528908+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:21:44.543638+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:21:44.543638+0000","last_clean_scrub_stamp":"2024-05-27T07:21:44.543638+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T18:12:23.015535+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,8,6],"acting":[10,8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.1b","version":"90'8","reported_seq":53,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:24:29.669757+0000","last_change":"2024-05-27T07:22:30.857080+0000","last_active":"2024-05-27T07:24:29.669757+0000","last_peered":"2024-05-27T07:24:29.669757+0000","last_clean":"2024-05-27T07:24:29.669757+0000","last_became_active":"2024-05-27T07:22:30.856479+0000","last_became_peered":"2024-05-27T07:22:30.856479+0000","last_unstale":"2024-05-27T07:24:29.669757+0000","last_undegraded":"2024-05-27T07:24:29.669757+0000","last_fullsized":"2024-05-27T07:24:29.669757+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:21:47.565641+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:21:47.565641+0000","last_clean_scrub_stamp":"2024-05-27T07:21:47.565641+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T13:29:16.054448+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,8,3],"acting":[5,8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":36,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:23:31.622401+0000","last_change":"2024-05-27T07:22:30.805292+0000","last_active":"2024-05-27T07:23:31.622401+0000","last_peered":"2024-05-27T07:23:31.622401+0000","last_clean":"2024-05-27T07:23:31.622401+0000","last_became_active":"2024-05-27T07:22:30.804779+0000","last_became_peered":"2024-05-27T07:22:30.804779+0000","last_unstale":"2024-05-27T07:23:31.622401+0000","last_undegraded":"2024-05-27T07:23:31.622401+0000","last_fullsized":"2024-05-27T07:23:31.622401+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:21:58.698952+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:21:58.698952+0000","last_clean_scrub_stamp":"2024-05-27T07:21:58.698952+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T11:12:49.095931+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,8,0],"acting":[11,4,8,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.19","version":"88'3","reported_seq":47,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:23:30.973995+0000","last_change":"2024-05-27T07:21:45.561255+0000","last_active":"2024-05-27T07:23:30.973995+0000","last_peered":"2024-05-27T07:23:30.973995+0000","last_clean":"2024-05-27T07:23:30.973995+0000","last_became_active":"2024-05-27T07:21:45.560896+0000","last_became_peered":"2024-05-27T07:21:45.560896+0000","last_unstale":"2024-05-27T07:23:30.973995+0000","last_undegraded":"2024-05-27T07:23:30.973995+0000","last_fullsized":"2024-05-27T07:23:30.973995+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:21:44.543638+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:21:44.543638+0000","last_clean_scrub_stamp":"2024-05-27T07:21:44.543638+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T17:05:24.074628+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,9],"acting":[6,4,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"90'8","reported_seq":47,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:24:29.679127+0000","last_change":"2024-05-27T07:21:48.586045+0000","last_active":"2024-05-27T07:24:29.679127+0000","last_peered":"2024-05-27T07:24:29.679127+0000","last_clean":"2024-05-27T07:24:29.679127+0000","last_became_active":"2024-05-27T07:21:48.585681+0000","last_became_peered":"2024-05-27T07:21:48.585681+0000","last_unstale":"2024-05-27T07:24:29.679127+0000","last_undegraded":"2024-05-27T07:24:29.679127+0000","last_fullsized":"2024-05-27T07:24:29.679127+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:21:47.565641+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:21:47.565641+0000","last_clean_scrub_stamp":"2024-05-27T07:21:47.565641+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T09:56:36.408623+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,6],"acting":[3,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":29,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:23:30.981574+0000","last_change":"2024-05-27T07:21:59.739544+0000","last_active":"2024-05-27T07:23:30.981574+0000","last_peered":"2024-05-27T07:23:30.981574+0000","last_clean":"2024-05-27T07:23:30.981574+0000","last_became_active":"2024-05-27T07:21:59.739132+0000","last_became_peered":"2024-05-27T07:21:59.739132+0000","last_unstale":"2024-05-27T07:23:30.981574+0000","last_undegraded":"2024-05-27T07:23:30.981574+0000","last_fullsized":"2024-05-27T07:23:30.981574+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:21:58.698952+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:21:58.698952+0000","last_clean_scrub_stamp":"2024-05-27T07:21:58.698952+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T10:06:05.126579+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,6,1],"acting":[0,7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.18","version":"88'3","reported_seq":48,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:23:30.977006+0000","last_change":"2024-05-27T07:21:45.568171+0000","last_active":"2024-05-27T07:23:30.977006+0000","last_peered":"2024-05-27T07:23:30.977006+0000","last_clean":"2024-05-27T07:23:30.977006+0000","last_became_active":"2024-05-27T07:21:45.567627+0000","last_became_peered":"2024-05-27T07:21:45.567627+0000","last_unstale":"2024-05-27T07:23:30.977006+0000","last_undegraded":"2024-05-27T07:23:30.977006+0000","last_fullsized":"2024-05-27T07:23:30.977006+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:21:44.543638+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:21:44.543638+0000","last_clean_scrub_stamp":"2024-05-27T07:21:44.543638+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T16:43:56.407079+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"90'11","reported_seq":50,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:24:29.679091+0000","last_change":"2024-05-27T07:21:48.587452+0000","last_active":"2024-05-27T07:24:29.679091+0000","last_peered":"2024-05-27T07:24:29.679091+0000","last_clean":"2024-05-27T07:24:29.679091+0000","last_became_active":"2024-05-27T07:21:48.587027+0000","last_became_peered":"2024-05-27T07:21:48.587027+0000","last_unstale":"2024-05-27T07:24:29.679091+0000","last_undegraded":"2024-05-27T07:24:29.679091+0000","last_fullsized":"2024-05-27T07:24:29.679091+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:21:47.565641+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:21:47.565641+0000","last_clean_scrub_stamp":"2024-05-27T07:21:47.565641+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T13:51:44.890061+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,10],"acting":[9,3,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":29,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:23:31.102968+0000","last_change":"2024-05-27T07:21:59.738292+0000","last_active":"2024-05-27T07:23:31.102968+0000","last_peered":"2024-05-27T07:23:31.102968+0000","last_clean":"2024-05-27T07:23:31.102968+0000","last_became_active":"2024-05-27T07:21:59.737941+0000","last_became_peered":"2024-05-27T07:21:59.737941+0000","last_unstale":"2024-05-27T07:23:31.102968+0000","last_undegraded":"2024-05-27T07:23:31.102968+0000","last_fullsized":"2024-05-27T07:23:31.102968+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:21:58.698952+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:21:58.698952+0000","last_clean_scrub_stamp":"2024-05-27T07:21:58.698952+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T13:54:24.192917+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1,5],"acting":[3,7,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.17","version":"90'5","reported_seq":236,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:24:29.029184+0000","last_change":"2024-05-27T07:21:45.568338+0000","last_active":"2024-05-27T07:24:29.029184+0000","last_peered":"2024-05-27T07:24:29.029184+0000","last_clean":"2024-05-27T07:24:29.029184+0000","last_became_active":"2024-05-27T07:21:45.567856+0000","last_became_peered":"2024-05-27T07:21:45.567856+0000","last_unstale":"2024-05-27T07:24:29.029184+0000","last_undegraded":"2024-05-27T07:24:29.029184+0000","last_fullsized":"2024-05-27T07:24:29.029184+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:21:44.543638+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:21:44.543638+0000","last_clean_scrub_stamp":"2024-05-27T07:21:44.543638+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T15:02:04.188711+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":180,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":186,"num_read_kb":186,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,5,9],"acting":[10,5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.16","version":"90'10","reported_seq":49,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:24:29.684905+0000","last_change":"2024-05-27T07:21:48.606062+0000","last_active":"2024-05-27T07:24:29.684905+0000","last_peered":"2024-05-27T07:24:29.684905+0000","last_clean":"2024-05-27T07:24:29.684905+0000","last_became_active":"2024-05-27T07:21:48.605423+0000","last_became_peered":"2024-05-27T07:21:48.605423+0000","last_unstale":"2024-05-27T07:24:29.684905+0000","last_undegraded":"2024-05-27T07:24:29.684905+0000","last_fullsized":"2024-05-27T07:24:29.684905+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:21:47.565641+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:21:47.565641+0000","last_clean_scrub_stamp":"2024-05-27T07:21:47.565641+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T14:39:06.921519+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,9],"acting":[1,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":29,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:23:31.652581+0000","last_change":"2024-05-27T07:21:59.735150+0000","last_active":"2024-05-27T07:23:31.652581+0000","last_peered":"2024-05-27T07:23:31.652581+0000","last_clean":"2024-05-27T07:23:31.652581+0000","last_became_active":"2024-05-27T07:21:59.734688+0000","last_became_peered":"2024-05-27T07:21:59.734688+0000","last_unstale":"2024-05-27T07:23:31.652581+0000","last_undegraded":"2024-05-27T07:23:31.652581+0000","last_fullsized":"2024-05-27T07:23:31.652581+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:21:58.698952+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:21:58.698952+0000","last_clean_scrub_stamp":"2024-05-27T07:21:58.698952+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T15:38:02.095947+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,0,2,4],"acting":[9,0,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.16","version":"88'4","reported_seq":46,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:23:30.975440+0000","last_change":"2024-05-27T07:21:45.559550+0000","last_active":"2024-05-27T07:23:30.975440+0000","last_peered":"2024-05-27T07:23:30.975440+0000","last_clean":"2024-05-27T07:23:30.975440+0000","last_became_active":"2024-05-27T07:21:45.559181+0000","last_became_peered":"2024-05-27T07:21:45.559181+0000","last_unstale":"2024-05-27T07:23:30.975440+0000","last_undegraded":"2024-05-27T07:23:30.975440+0000","last_fullsized":"2024-05-27T07:23:30.975440+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:21:44.543638+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:21:44.543638+0000","last_clean_scrub_stamp":"2024-05-27T07:21:44.543638+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T10:33:45.288437+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.17","version":"90'9","reported_seq":48,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:24:29.681347+0000","last_change":"2024-05-27T07:21:48.593631+0000","last_active":"2024-05-27T07:24:29.681347+0000","last_peered":"2024-05-27T07:24:29.681347+0000","last_clean":"2024-05-27T07:24:29.681347+0000","last_became_active":"2024-05-27T07:21:48.592330+0000","last_became_peered":"2024-05-27T07:21:48.592330+0000","last_unstale":"2024-05-27T07:24:29.681347+0000","last_undegraded":"2024-05-27T07:24:29.681347+0000","last_fullsized":"2024-05-27T07:24:29.681347+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:21:47.565641+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:21:47.565641+0000","last_clean_scrub_stamp":"2024-05-27T07:21:47.565641+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T19:19:51.593307+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,5],"acting":[0,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":29,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:23:31.757433+0000","last_change":"2024-05-27T07:21:59.725158+0000","last_active":"2024-05-27T07:23:31.757433+0000","last_peered":"2024-05-27T07:23:31.757433+0000","last_clean":"2024-05-27T07:23:31.757433+0000","last_became_active":"2024-05-27T07:21:59.724651+0000","last_became_peered":"2024-05-27T07:21:59.724651+0000","last_unstale":"2024-05-27T07:23:31.757433+0000","last_undegraded":"2024-05-27T07:23:31.757433+0000","last_fullsized":"2024-05-27T07:23:31.757433+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:21:58.698952+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:21:58.698952+0000","last_clean_scrub_stamp":"2024-05-27T07:21:58.698952+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T12:10:45.747484+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2,9],"acting":[4,1,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.15","version":"88'5","reported_seq":50,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:23:30.977408+0000","last_change":"2024-05-27T07:21:45.791831+0000","last_active":"2024-05-27T07:23:30.977408+0000","last_peered":"2024-05-27T07:23:30.977408+0000","last_clean":"2024-05-27T07:23:30.977408+0000","last_became_active":"2024-05-27T07:21:45.791271+0000","last_became_peered":"2024-05-27T07:21:45.791271+0000","last_unstale":"2024-05-27T07:23:30.977408+0000","last_undegraded":"2024-05-27T07:23:30.977408+0000","last_fullsized":"2024-05-27T07:23:30.977408+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:21:44.543638+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:21:44.543638+0000","last_clean_scrub_stamp":"2024-05-27T07:21:44.543638+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T16:20:37.492337+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":7,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,10],"acting":[5,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.14","version":"90'11","reported_seq":50,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:24:29.687145+0000","last_change":"2024-05-27T07:21:48.605605+0000","last_active":"2024-05-27T07:24:29.687145+0000","last_peered":"2024-05-27T07:24:29.687145+0000","last_clean":"2024-05-27T07:24:29.687145+0000","last_became_active":"2024-05-27T07:21:48.604655+0000","last_became_peered":"2024-05-27T07:21:48.604655+0000","last_unstale":"2024-05-27T07:24:29.687145+0000","last_undegraded":"2024-05-27T07:24:29.687145+0000","last_fullsized":"2024-05-27T07:24:29.687145+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:21:47.565641+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:21:47.565641+0000","last_clean_scrub_stamp":"2024-05-27T07:21:47.565641+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T14:33:53.661983+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,8,4],"acting":[1,8,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":36,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:23:31.652808+0000","last_change":"2024-05-27T07:22:31.418393+0000","last_active":"2024-05-27T07:23:31.652808+0000","last_peered":"2024-05-27T07:23:31.652808+0000","last_clean":"2024-05-27T07:23:31.652808+0000","last_became_active":"2024-05-27T07:22:31.418006+0000","last_became_peered":"2024-05-27T07:22:31.418006+0000","last_unstale":"2024-05-27T07:23:31.652808+0000","last_undegraded":"2024-05-27T07:23:31.652808+0000","last_fullsized":"2024-05-27T07:23:31.652808+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:21:58.698952+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:21:58.698952+0000","last_clean_scrub_stamp":"2024-05-27T07:21:58.698952+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T14:30:55.130461+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,11,10],"acting":[9,8,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.14","version":"90'39","reported_seq":280,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:24:29.624651+0000","last_change":"2024-05-27T07:21:45.564829+0000","last_active":"2024-05-27T07:24:29.624651+0000","last_peered":"2024-05-27T07:24:29.624651+0000","last_clean":"2024-05-27T07:24:29.624651+0000","last_became_active":"2024-05-27T07:21:45.564457+0000","last_became_peered":"2024-05-27T07:21:45.564457+0000","last_unstale":"2024-05-27T07:24:29.624651+0000","last_undegraded":"2024-05-27T07:24:29.624651+0000","last_fullsized":"2024-05-27T07:24:29.624651+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:21:44.543638+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:21:44.543638+0000","last_clean_scrub_stamp":"2024-05-27T07:21:44.543638+0000","objects_scrubbed":0,"log_size":39,"log_dups_size":0,"ondisk_log_size":39,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T19:12:44.546989+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":64791,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":196,"num_read_kb":161,"num_write":41,"num_write_kb":82,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,1,2],"acting":[10,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.15","version":"90'6","reported_seq":45,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:24:29.669019+0000","last_change":"2024-05-27T07:21:48.591672+0000","last_active":"2024-05-27T07:24:29.669019+0000","last_peered":"2024-05-27T07:24:29.669019+0000","last_clean":"2024-05-27T07:24:29.669019+0000","last_became_active":"2024-05-27T07:21:48.590861+0000","last_became_peered":"2024-05-27T07:21:48.590861+0000","last_unstale":"2024-05-27T07:24:29.669019+0000","last_undegraded":"2024-05-27T07:24:29.669019+0000","last_fullsized":"2024-05-27T07:24:29.669019+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:21:47.565641+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:21:47.565641+0000","last_clean_scrub_stamp":"2024-05-27T07:21:47.565641+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T12:03:40.627764+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,7],"acting":[8,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":29,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:23:30.977741+0000","last_change":"2024-05-27T07:21:59.736446+0000","last_active":"2024-05-27T07:23:30.977741+0000","last_peered":"2024-05-27T07:23:30.977741+0000","last_clean":"2024-05-27T07:23:30.977741+0000","last_became_active":"2024-05-27T07:21:59.735854+0000","last_became_peered":"2024-05-27T07:21:59.735854+0000","last_unstale":"2024-05-27T07:23:30.977741+0000","last_undegraded":"2024-05-27T07:23:30.977741+0000","last_fullsized":"2024-05-27T07:23:30.977741+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:21:58.698952+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:21:58.698952+0000","last_clean_scrub_stamp":"2024-05-27T07:21:58.698952+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T08:02:14.640325+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,8,3,6],"acting":[5,8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.13","version":"88'9","reported_seq":55,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:23:30.982597+0000","last_change":"2024-05-27T07:21:45.562930+0000","last_active":"2024-05-27T07:23:30.982597+0000","last_peered":"2024-05-27T07:23:30.982597+0000","last_clean":"2024-05-27T07:23:30.982597+0000","last_became_active":"2024-05-27T07:21:45.562515+0000","last_became_peered":"2024-05-27T07:21:45.562515+0000","last_unstale":"2024-05-27T07:23:30.982597+0000","last_undegraded":"2024-05-27T07:23:30.982597+0000","last_fullsized":"2024-05-27T07:23:30.982597+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:21:44.543638+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:21:44.543638+0000","last_clean_scrub_stamp":"2024-05-27T07:21:44.543638+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T17:54:24.006219+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":68,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":13,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,10,3],"acting":[2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.12","version":"90'9","reported_seq":48,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:24:29.676179+0000","last_change":"2024-05-27T07:21:48.591090+0000","last_active":"2024-05-27T07:24:29.676179+0000","last_peered":"2024-05-27T07:24:29.676179+0000","last_clean":"2024-05-27T07:24:29.676179+0000","last_became_active":"2024-05-27T07:21:48.590726+0000","last_became_peered":"2024-05-27T07:21:48.590726+0000","last_unstale":"2024-05-27T07:24:29.676179+0000","last_undegraded":"2024-05-27T07:24:29.676179+0000","last_fullsized":"2024-05-27T07:24:29.676179+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:21:47.565641+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:21:47.565641+0000","last_clean_scrub_stamp":"2024-05-27T07:21:47.565641+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T18:40:03.880407+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,3],"acting":[6,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.14","version":"0'0","reported_seq":29,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:23:31.621855+0000","last_change":"2024-05-27T07:21:59.738375+0000","last_active":"2024-05-27T07:23:31.621855+0000","last_peered":"2024-05-27T07:23:31.621855+0000","last_clean":"2024-05-27T07:23:31.621855+0000","last_became_active":"2024-05-27T07:21:59.737957+0000","last_became_peered":"2024-05-27T07:21:59.737957+0000","last_unstale":"2024-05-27T07:23:31.621855+0000","last_undegraded":"2024-05-27T07:23:31.621855+0000","last_fullsized":"2024-05-27T07:23:31.621855+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:21:58.698952+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:21:58.698952+0000","last_clean_scrub_stamp":"2024-05-27T07:21:58.698952+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T16:53:51.397143+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,9,7,6],"acting":[11,9,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.12","version":"88'5","reported_seq":50,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:23:30.977272+0000","last_change":"2024-05-27T07:21:45.566986+0000","last_active":"2024-05-27T07:23:30.977272+0000","last_peered":"2024-05-27T07:23:30.977272+0000","last_clean":"2024-05-27T07:23:30.977272+0000","last_became_active":"2024-05-27T07:21:45.566579+0000","last_became_peered":"2024-05-27T07:21:45.566579+0000","last_unstale":"2024-05-27T07:23:30.977272+0000","last_undegraded":"2024-05-27T07:23:30.977272+0000","last_fullsized":"2024-05-27T07:23:30.977272+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:21:44.543638+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:21:44.543638+0000","last_clean_scrub_stamp":"2024-05-27T07:21:44.543638+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T17:08:14.034740+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":5,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"90'15","reported_seq":52,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:24:29.677606+0000","last_change":"2024-05-27T07:21:48.596575+0000","last_active":"2024-05-27T07:24:29.677606+0000","last_peered":"2024-05-27T07:24:29.677606+0000","last_clean":"2024-05-27T07:24:29.677606+0000","last_became_active":"2024-05-27T07:21:48.596215+0000","last_became_peered":"2024-05-27T07:21:48.596215+0000","last_unstale":"2024-05-27T07:24:29.677606+0000","last_undegraded":"2024-05-27T07:24:29.677606+0000","last_fullsized":"2024-05-27T07:24:29.677606+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:21:47.565641+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:21:47.565641+0000","last_clean_scrub_stamp":"2024-05-27T07:21:47.565641+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T14:57:05.795269+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":29,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:23:31.653172+0000","last_change":"2024-05-27T07:21:59.737690+0000","last_active":"2024-05-27T07:23:31.653172+0000","last_peered":"2024-05-27T07:23:31.653172+0000","last_clean":"2024-05-27T07:23:31.653172+0000","last_became_active":"2024-05-27T07:21:59.735646+0000","last_became_peered":"2024-05-27T07:21:59.735646+0000","last_unstale":"2024-05-27T07:23:31.653172+0000","last_undegraded":"2024-05-27T07:23:31.653172+0000","last_fullsized":"2024-05-27T07:23:31.653172+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:21:58.698952+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:21:58.698952+0000","last_clean_scrub_stamp":"2024-05-27T07:21:58.698952+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T13:54:41.227626+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,0,3],"acting":[9,7,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.11","version":"90'7","reported_seq":150,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:24:30.031261+0000","last_change":"2024-05-27T07:21:45.564836+0000","last_active":"2024-05-27T07:24:30.031261+0000","last_peered":"2024-05-27T07:24:30.031261+0000","last_clean":"2024-05-27T07:24:30.031261+0000","last_became_active":"2024-05-27T07:21:45.564481+0000","last_became_peered":"2024-05-27T07:21:45.564481+0000","last_unstale":"2024-05-27T07:24:30.031261+0000","last_undegraded":"2024-05-27T07:24:30.031261+0000","last_fullsized":"2024-05-27T07:24:30.031261+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:21:44.543638+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:21:44.543638+0000","last_clean_scrub_stamp":"2024-05-27T07:21:44.543638+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T15:13:12.012769+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1120,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":99,"num_read_kb":5,"num_write":7,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,10],"acting":[4,7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.10","version":"90'6","reported_seq":43,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:24:29.672831+0000","last_change":"2024-05-27T07:21:48.601331+0000","last_active":"2024-05-27T07:24:29.672831+0000","last_peered":"2024-05-27T07:24:29.672831+0000","last_clean":"2024-05-27T07:24:29.672831+0000","last_became_active":"2024-05-27T07:21:48.600917+0000","last_became_peered":"2024-05-27T07:21:48.600917+0000","last_unstale":"2024-05-27T07:24:29.672831+0000","last_undegraded":"2024-05-27T07:24:29.672831+0000","last_fullsized":"2024-05-27T07:24:29.672831+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:21:47.565641+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:21:47.565641+0000","last_clean_scrub_stamp":"2024-05-27T07:21:47.565641+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T17:26:42.129989+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":29,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:23:30.974225+0000","last_change":"2024-05-27T07:21:59.737662+0000","last_active":"2024-05-27T07:23:30.974225+0000","last_peered":"2024-05-27T07:23:30.974225+0000","last_clean":"2024-05-27T07:23:30.974225+0000","last_became_active":"2024-05-27T07:21:59.737229+0000","last_became_peered":"2024-05-27T07:21:59.737229+0000","last_unstale":"2024-05-27T07:23:30.974225+0000","last_undegraded":"2024-05-27T07:23:30.974225+0000","last_fullsized":"2024-05-27T07:23:30.974225+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:21:58.698952+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:21:58.698952+0000","last_clean_scrub_stamp":"2024-05-27T07:21:58.698952+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T15:32:31.147571+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,9,11],"acting":[6,3,9,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.10","version":"88'17","reported_seq":79,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:23:31.061322+0000","last_change":"2024-05-27T07:22:30.814815+0000","last_active":"2024-05-27T07:23:31.061322+0000","last_peered":"2024-05-27T07:23:31.061322+0000","last_clean":"2024-05-27T07:23:31.061322+0000","last_became_active":"2024-05-27T07:22:30.802828+0000","last_became_peered":"2024-05-27T07:22:30.802828+0000","last_unstale":"2024-05-27T07:23:31.061322+0000","last_undegraded":"2024-05-27T07:23:31.061322+0000","last_fullsized":"2024-05-27T07:23:31.061322+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:21:44.543638+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:21:44.543638+0000","last_clean_scrub_stamp":"2024-05-27T07:21:44.543638+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T11:38:07.363507+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,4],"acting":[8,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.11","version":"90'6","reported_seq":45,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:24:29.660613+0000","last_change":"2024-05-27T07:21:48.596204+0000","last_active":"2024-05-27T07:24:29.660613+0000","last_peered":"2024-05-27T07:24:29.660613+0000","last_clean":"2024-05-27T07:24:29.660613+0000","last_became_active":"2024-05-27T07:21:48.595827+0000","last_became_peered":"2024-05-27T07:21:48.595827+0000","last_unstale":"2024-05-27T07:24:29.660613+0000","last_undegraded":"2024-05-27T07:24:29.660613+0000","last_fullsized":"2024-05-27T07:24:29.660613+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:21:47.565641+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:21:47.565641+0000","last_clean_scrub_stamp":"2024-05-27T07:21:47.565641+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T15:51:53.162554+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":29,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:23:30.977011+0000","last_change":"2024-05-27T07:21:59.735837+0000","last_active":"2024-05-27T07:23:30.977011+0000","last_peered":"2024-05-27T07:23:30.977011+0000","last_clean":"2024-05-27T07:23:30.977011+0000","last_became_active":"2024-05-27T07:21:59.735063+0000","last_became_peered":"2024-05-27T07:21:59.735063+0000","last_unstale":"2024-05-27T07:23:30.977011+0000","last_undegraded":"2024-05-27T07:23:30.977011+0000","last_fullsized":"2024-05-27T07:23:30.977011+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:21:58.698952+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:21:58.698952+0000","last_clean_scrub_stamp":"2024-05-27T07:21:58.698952+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T13:04:21.751043+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,3,1],"acting":[5,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.f","version":"88'2","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:23:31.622220+0000","last_change":"2024-05-27T07:22:30.809491+0000","last_active":"2024-05-27T07:23:31.622220+0000","last_peered":"2024-05-27T07:23:31.622220+0000","last_clean":"2024-05-27T07:23:31.622220+0000","last_became_active":"2024-05-27T07:22:30.802187+0000","last_became_peered":"2024-05-27T07:22:30.802187+0000","last_unstale":"2024-05-27T07:23:31.622220+0000","last_undegraded":"2024-05-27T07:23:31.622220+0000","last_fullsized":"2024-05-27T07:23:31.622220+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:21:44.543638+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:21:44.543638+0000","last_clean_scrub_stamp":"2024-05-27T07:21:44.543638+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T18:11:32.821962+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":22,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,4],"acting":[11,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.e","version":"90'5","reported_seq":44,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:24:29.671137+0000","last_change":"2024-05-27T07:21:48.597362+0000","last_active":"2024-05-27T07:24:29.671137+0000","last_peered":"2024-05-27T07:24:29.671137+0000","last_clean":"2024-05-27T07:24:29.671137+0000","last_became_active":"2024-05-27T07:21:48.596920+0000","last_became_peered":"2024-05-27T07:21:48.596920+0000","last_unstale":"2024-05-27T07:24:29.671137+0000","last_undegraded":"2024-05-27T07:24:29.671137+0000","last_fullsized":"2024-05-27T07:24:29.671137+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:21:47.565641+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:21:47.565641+0000","last_clean_scrub_stamp":"2024-05-27T07:21:47.565641+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T16:50:21.021320+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,4],"acting":[11,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":29,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:23:31.061286+0000","last_change":"2024-05-27T07:21:59.732303+0000","last_active":"2024-05-27T07:23:31.061286+0000","last_peered":"2024-05-27T07:23:31.061286+0000","last_clean":"2024-05-27T07:23:31.061286+0000","last_became_active":"2024-05-27T07:21:59.731851+0000","last_became_peered":"2024-05-27T07:21:59.731851+0000","last_unstale":"2024-05-27T07:23:31.061286+0000","last_undegraded":"2024-05-27T07:23:31.061286+0000","last_fullsized":"2024-05-27T07:23:31.061286+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:21:58.698952+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:21:58.698952+0000","last_clean_scrub_stamp":"2024-05-27T07:21:58.698952+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T08:08:38.433818+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,10,11,9],"acting":[8,10,11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.4","version":"88'1","reported_seq":46,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:23:31.653172+0000","last_change":"2024-05-27T07:21:45.791370+0000","last_active":"2024-05-27T07:23:31.653172+0000","last_peered":"2024-05-27T07:23:31.653172+0000","last_clean":"2024-05-27T07:23:31.653172+0000","last_became_active":"2024-05-27T07:21:45.790987+0000","last_became_peered":"2024-05-27T07:21:45.790987+0000","last_unstale":"2024-05-27T07:23:31.653172+0000","last_undegraded":"2024-05-27T07:23:31.653172+0000","last_fullsized":"2024-05-27T07:23:31.653172+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:21:44.543638+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:21:44.543638+0000","last_clean_scrub_stamp":"2024-05-27T07:21:44.543638+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T14:20:18.570194+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2,0],"acting":[9,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":39,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:23:31.528919+0000","last_change":"2024-05-27T07:21:48.585984+0000","last_active":"2024-05-27T07:23:31.528919+0000","last_peered":"2024-05-27T07:23:31.528919+0000","last_clean":"2024-05-27T07:23:31.528919+0000","last_became_active":"2024-05-27T07:21:48.585606+0000","last_became_peered":"2024-05-27T07:21:48.585606+0000","last_unstale":"2024-05-27T07:23:31.528919+0000","last_undegraded":"2024-05-27T07:23:31.528919+0000","last_fullsized":"2024-05-27T07:23:31.528919+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:21:47.565641+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:21:47.565641+0000","last_clean_scrub_stamp":"2024-05-27T07:21:47.565641+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T17:56:17.659024+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,3,6],"acting":[10,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":29,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:23:30.973955+0000","last_change":"2024-05-27T07:21:59.734022+0000","last_active":"2024-05-27T07:23:30.973955+0000","last_peered":"2024-05-27T07:23:30.973955+0000","last_clean":"2024-05-27T07:23:30.973955+0000","last_became_active":"2024-05-27T07:21:59.733467+0000","last_became_peered":"2024-05-27T07:21:59.733467+0000","last_unstale":"2024-05-27T07:23:30.973955+0000","last_undegraded":"2024-05-27T07:23:30.973955+0000","last_fullsized":"2024-05-27T07:23:30.973955+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:21:58.698952+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:21:58.698952+0000","last_clean_scrub_stamp":"2024-05-27T07:21:58.698952+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T09:54:11.058495+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,1,8],"acting":[6,0,1,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.2","version":"88'5","reported_seq":143,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:24:30.040121+0000","last_change":"2024-05-27T07:21:45.563072+0000","last_active":"2024-05-27T07:24:30.040121+0000","last_peered":"2024-05-27T07:24:30.040121+0000","last_clean":"2024-05-27T07:24:30.040121+0000","last_became_active":"2024-05-27T07:21:45.562632+0000","last_became_peered":"2024-05-27T07:21:45.562632+0000","last_unstale":"2024-05-27T07:24:30.040121+0000","last_undegraded":"2024-05-27T07:24:30.040121+0000","last_fullsized":"2024-05-27T07:24:30.040121+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:21:44.543638+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:21:44.543638+0000","last_clean_scrub_stamp":"2024-05-27T07:21:44.543638+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T12:54:19.555467+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":588,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":96,"num_read_kb":1,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3","version":"90'5","reported_seq":40,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:23:54.635291+0000","last_change":"2024-05-27T07:21:48.594867+0000","last_active":"2024-05-27T07:23:54.635291+0000","last_peered":"2024-05-27T07:23:54.635291+0000","last_clean":"2024-05-27T07:23:54.635291+0000","last_became_active":"2024-05-27T07:21:48.594523+0000","last_became_peered":"2024-05-27T07:21:48.594523+0000","last_unstale":"2024-05-27T07:23:54.635291+0000","last_undegraded":"2024-05-27T07:23:54.635291+0000","last_fullsized":"2024-05-27T07:23:54.635291+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:21:47.565641+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:21:47.565641+0000","last_clean_scrub_stamp":"2024-05-27T07:21:47.565641+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T17:28:55.862014+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,4,0],"acting":[10,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":29,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:23:30.974221+0000","last_change":"2024-05-27T07:21:59.741936+0000","last_active":"2024-05-27T07:23:30.974221+0000","last_peered":"2024-05-27T07:23:30.974221+0000","last_clean":"2024-05-27T07:23:30.974221+0000","last_became_active":"2024-05-27T07:21:59.741458+0000","last_became_peered":"2024-05-27T07:21:59.741458+0000","last_unstale":"2024-05-27T07:23:30.974221+0000","last_undegraded":"2024-05-27T07:23:30.974221+0000","last_fullsized":"2024-05-27T07:23:30.974221+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:21:58.698952+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:21:58.698952+0000","last_clean_scrub_stamp":"2024-05-27T07:21:58.698952+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T15:51:28.560995+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,10,0],"acting":[6,5,10,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.0","version":"90'8","reported_seq":24,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:24:29.671317+0000","last_change":"2024-05-27T07:22:30.803379+0000","last_active":"2024-05-27T07:24:29.671317+0000","last_peered":"2024-05-27T07:24:29.671317+0000","last_clean":"2024-05-27T07:24:29.671317+0000","last_became_active":"2024-05-27T07:22:30.803024+0000","last_became_peered":"2024-05-27T07:22:30.803024+0000","last_unstale":"2024-05-27T07:24:29.671317+0000","last_undegraded":"2024-05-27T07:24:29.671317+0000","last_fullsized":"2024-05-27T07:24:29.671317+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:21:47.565641+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:21:47.565641+0000","last_clean_scrub_stamp":"2024-05-27T07:21:47.565641+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T07:56:32.070610+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2,4],"acting":[9,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.1","version":"90'6","reported_seq":145,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:24:29.624436+0000","last_change":"2024-05-27T07:21:45.790849+0000","last_active":"2024-05-27T07:24:29.624436+0000","last_peered":"2024-05-27T07:24:29.624436+0000","last_clean":"2024-05-27T07:24:29.624436+0000","last_became_active":"2024-05-27T07:21:45.790435+0000","last_became_peered":"2024-05-27T07:21:45.790435+0000","last_unstale":"2024-05-27T07:24:29.624436+0000","last_undegraded":"2024-05-27T07:24:29.624436+0000","last_fullsized":"2024-05-27T07:24:29.624436+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:21:44.543638+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:21:44.543638+0000","last_clean_scrub_stamp":"2024-05-27T07:21:44.543638+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T09:34:51.479882+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":95,"num_read_kb":95,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,0],"acting":[6,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.5","version":"88'1","reported_seq":29,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:23:30.975407+0000","last_change":"2024-05-27T07:21:59.731338+0000","last_active":"2024-05-27T07:23:30.975407+0000","last_peered":"2024-05-27T07:23:30.975407+0000","last_clean":"2024-05-27T07:23:30.975407+0000","last_became_active":"2024-05-27T07:21:59.730847+0000","last_became_peered":"2024-05-27T07:21:59.730847+0000","last_unstale":"2024-05-27T07:23:30.975407+0000","last_undegraded":"2024-05-27T07:23:30.975407+0000","last_fullsized":"2024-05-27T07:23:30.975407+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:21:58.698952+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:21:58.698952+0000","last_clean_scrub_stamp":"2024-05-27T07:21:58.698952+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T18:18:50.901129+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,11,9],"acting":[7,5,11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2","version":"90'8","reported_seq":47,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:24:29.673758+0000","last_change":"2024-05-27T07:21:48.594274+0000","last_active":"2024-05-27T07:24:29.673758+0000","last_peered":"2024-05-27T07:24:29.673758+0000","last_clean":"2024-05-27T07:24:29.673758+0000","last_became_active":"2024-05-27T07:21:48.593940+0000","last_became_peered":"2024-05-27T07:21:48.593940+0000","last_unstale":"2024-05-27T07:24:29.673758+0000","last_undegraded":"2024-05-27T07:24:29.673758+0000","last_fullsized":"2024-05-27T07:24:29.673758+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:21:47.565641+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:21:47.565641+0000","last_clean_scrub_stamp":"2024-05-27T07:21:47.565641+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T13:25:46.531524+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"88'6","reported_seq":54,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:23:31.757237+0000","last_change":"2024-05-27T07:21:45.563365+0000","last_active":"2024-05-27T07:23:31.757237+0000","last_peered":"2024-05-27T07:23:31.757237+0000","last_clean":"2024-05-27T07:23:31.757237+0000","last_became_active":"2024-05-27T07:21:45.563012+0000","last_became_peered":"2024-05-27T07:21:45.563012+0000","last_unstale":"2024-05-27T07:23:31.757237+0000","last_undegraded":"2024-05-27T07:23:31.757237+0000","last_fullsized":"2024-05-27T07:23:31.757237+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:21:44.543638+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:21:44.543638+0000","last_clean_scrub_stamp":"2024-05-27T07:21:44.543638+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T12:58:51.079965+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":68,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":4,"num_read_kb":4,"num_write":8,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,9],"acting":[4,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.0","version":"20'92","reported_seq":116,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:23:30.974809+0000","last_change":"2024-05-27T07:20:21.611212+0000","last_active":"2024-05-27T07:23:30.974809+0000","last_peered":"2024-05-27T07:23:30.974809+0000","last_clean":"2024-05-27T07:23:30.974809+0000","last_became_active":"2024-05-27T07:20:21.605163+0000","last_became_peered":"2024-05-27T07:20:21.605163+0000","last_unstale":"2024-05-27T07:23:30.974809+0000","last_undegraded":"2024-05-27T07:23:30.974809+0000","last_fullsized":"2024-05-27T07:23:30.974809+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:16:57.668433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:16:57.668433+0000","last_clean_scrub_stamp":"2024-05-27T07:16:57.668433+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T12:10:59.074162+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":81,"num_read_kb":69,"num_write":123,"num_write_kb":2150,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":2951840,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":29,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:23:30.977689+0000","last_change":"2024-05-27T07:21:59.737210+0000","last_active":"2024-05-27T07:23:30.977689+0000","last_peered":"2024-05-27T07:23:30.977689+0000","last_clean":"2024-05-27T07:23:30.977689+0000","last_became_active":"2024-05-27T07:21:59.736799+0000","last_became_peered":"2024-05-27T07:21:59.736799+0000","last_unstale":"2024-05-27T07:23:30.977689+0000","last_undegraded":"2024-05-27T07:23:30.977689+0000","last_fullsized":"2024-05-27T07:23:30.977689+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:21:58.698952+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:21:58.698952+0000","last_clean_scrub_stamp":"2024-05-27T07:21:58.698952+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T13:03:17.153687+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,6,10],"acting":[5,9,6,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1","version":"90'5","reported_seq":44,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:24:29.658895+0000","last_change":"2024-05-27T07:21:48.591550+0000","last_active":"2024-05-27T07:24:29.658895+0000","last_peered":"2024-05-27T07:24:29.658895+0000","last_clean":"2024-05-27T07:24:29.658895+0000","last_became_active":"2024-05-27T07:21:48.591223+0000","last_became_peered":"2024-05-27T07:21:48.591223+0000","last_unstale":"2024-05-27T07:24:29.658895+0000","last_undegraded":"2024-05-27T07:24:29.658895+0000","last_fullsized":"2024-05-27T07:24:29.658895+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:21:47.565641+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:21:47.565641+0000","last_clean_scrub_stamp":"2024-05-27T07:21:47.565641+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T15:06:23.395645+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,2],"acting":[6,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.0","version":"88'4","reported_seq":46,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:23:30.974992+0000","last_change":"2024-05-27T07:21:45.569224+0000","last_active":"2024-05-27T07:23:30.974992+0000","last_peered":"2024-05-27T07:23:30.974992+0000","last_clean":"2024-05-27T07:23:30.974992+0000","last_became_active":"2024-05-27T07:21:45.568818+0000","last_became_peered":"2024-05-27T07:21:45.568818+0000","last_unstale":"2024-05-27T07:23:30.974992+0000","last_undegraded":"2024-05-27T07:23:30.974992+0000","last_fullsized":"2024-05-27T07:23:30.974992+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:21:44.543638+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:21:44.543638+0000","last_clean_scrub_stamp":"2024-05-27T07:21:44.543638+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T08:12:33.157842+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,8],"acting":[7,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":29,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:23:31.621828+0000","last_change":"2024-05-27T07:21:59.733733+0000","last_active":"2024-05-27T07:23:31.621828+0000","last_peered":"2024-05-27T07:23:31.621828+0000","last_clean":"2024-05-27T07:23:31.621828+0000","last_became_active":"2024-05-27T07:21:59.733112+0000","last_became_peered":"2024-05-27T07:21:59.733112+0000","last_unstale":"2024-05-27T07:23:31.621828+0000","last_undegraded":"2024-05-27T07:23:31.621828+0000","last_fullsized":"2024-05-27T07:23:31.621828+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:21:58.698952+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:21:58.698952+0000","last_clean_scrub_stamp":"2024-05-27T07:21:58.698952+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T16:01:30.783800+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,7,6],"acting":[11,5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.5","version":"88'4","reported_seq":50,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:23:31.061825+0000","last_change":"2024-05-27T07:21:45.792029+0000","last_active":"2024-05-27T07:23:31.061825+0000","last_peered":"2024-05-27T07:23:31.061825+0000","last_clean":"2024-05-27T07:23:31.061825+0000","last_became_active":"2024-05-27T07:21:45.791492+0000","last_became_peered":"2024-05-27T07:21:45.791492+0000","last_unstale":"2024-05-27T07:23:31.061825+0000","last_undegraded":"2024-05-27T07:23:31.061825+0000","last_fullsized":"2024-05-27T07:23:31.061825+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:21:44.543638+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:21:44.543638+0000","last_clean_scrub_stamp":"2024-05-27T07:21:44.543638+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T17:36:42.444243+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,9],"acting":[8,0,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.4","version":"90'9","reported_seq":48,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:24:29.666461+0000","last_change":"2024-05-27T07:21:48.594769+0000","last_active":"2024-05-27T07:24:29.666461+0000","last_peered":"2024-05-27T07:24:29.666461+0000","last_clean":"2024-05-27T07:24:29.666461+0000","last_became_active":"2024-05-27T07:21:48.592979+0000","last_became_peered":"2024-05-27T07:21:48.592979+0000","last_unstale":"2024-05-27T07:24:29.666461+0000","last_undegraded":"2024-05-27T07:24:29.666461+0000","last_fullsized":"2024-05-27T07:24:29.666461+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:21:47.565641+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:21:47.565641+0000","last_clean_scrub_stamp":"2024-05-27T07:21:47.565641+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T10:06:48.268230+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,9],"acting":[6,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":29,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:23:31.102883+0000","last_change":"2024-05-27T07:21:59.736405+0000","last_active":"2024-05-27T07:23:31.102883+0000","last_peered":"2024-05-27T07:23:31.102883+0000","last_clean":"2024-05-27T07:23:31.102883+0000","last_became_active":"2024-05-27T07:21:59.735991+0000","last_became_peered":"2024-05-27T07:21:59.735991+0000","last_unstale":"2024-05-27T07:23:31.102883+0000","last_undegraded":"2024-05-27T07:23:31.102883+0000","last_fullsized":"2024-05-27T07:23:31.102883+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:21:58.698952+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:21:58.698952+0000","last_clean_scrub_stamp":"2024-05-27T07:21:58.698952+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T18:32:04.931725+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,10,0,7],"acting":[3,10,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"88'15","reported_seq":60,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:23:31.653304+0000","last_change":"2024-05-27T07:21:45.566513+0000","last_active":"2024-05-27T07:23:31.653304+0000","last_peered":"2024-05-27T07:23:31.653304+0000","last_clean":"2024-05-27T07:23:31.653304+0000","last_became_active":"2024-05-27T07:21:45.566107+0000","last_became_peered":"2024-05-27T07:21:45.566107+0000","last_unstale":"2024-05-27T07:23:31.653304+0000","last_undegraded":"2024-05-27T07:23:31.653304+0000","last_fullsized":"2024-05-27T07:23:31.653304+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:21:44.543638+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:21:44.543638+0000","last_clean_scrub_stamp":"2024-05-27T07:21:44.543638+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T14:11:34.851691+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":27,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,3],"acting":[9,8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.7","version":"90'5","reported_seq":44,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:24:29.678400+0000","last_change":"2024-05-27T07:21:48.593642+0000","last_active":"2024-05-27T07:24:29.678400+0000","last_peered":"2024-05-27T07:24:29.678400+0000","last_clean":"2024-05-27T07:24:29.678400+0000","last_became_active":"2024-05-27T07:21:48.592885+0000","last_became_peered":"2024-05-27T07:21:48.592885+0000","last_unstale":"2024-05-27T07:24:29.678400+0000","last_undegraded":"2024-05-27T07:24:29.678400+0000","last_fullsized":"2024-05-27T07:24:29.678400+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:21:47.565641+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:21:47.565641+0000","last_clean_scrub_stamp":"2024-05-27T07:21:47.565641+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T11:51:27.186560+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,7],"acting":[3,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":29,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:23:30.982161+0000","last_change":"2024-05-27T07:21:59.725533+0000","last_active":"2024-05-27T07:23:30.982161+0000","last_peered":"2024-05-27T07:23:30.982161+0000","last_clean":"2024-05-27T07:23:30.982161+0000","last_became_active":"2024-05-27T07:21:59.724991+0000","last_became_peered":"2024-05-27T07:21:59.724991+0000","last_unstale":"2024-05-27T07:23:30.982161+0000","last_undegraded":"2024-05-27T07:23:30.982161+0000","last_fullsized":"2024-05-27T07:23:30.982161+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:21:58.698952+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:21:58.698952+0000","last_clean_scrub_stamp":"2024-05-27T07:21:58.698952+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T17:22:54.650553+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,4,7],"acting":[2,1,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"88'7","reported_seq":53,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:23:31.757701+0000","last_change":"2024-05-27T07:21:45.572701+0000","last_active":"2024-05-27T07:23:31.757701+0000","last_peered":"2024-05-27T07:23:31.757701+0000","last_clean":"2024-05-27T07:23:31.757701+0000","last_became_active":"2024-05-27T07:21:45.572176+0000","last_became_peered":"2024-05-27T07:21:45.572176+0000","last_unstale":"2024-05-27T07:23:31.757701+0000","last_undegraded":"2024-05-27T07:23:31.757701+0000","last_fullsized":"2024-05-27T07:23:31.757701+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:21:44.543638+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:21:44.543638+0000","last_clean_scrub_stamp":"2024-05-27T07:21:44.543638+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T16:11:20.901250+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,11,6],"acting":[4,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"90'7","reported_seq":46,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:24:29.681965+0000","last_change":"2024-05-27T07:21:48.581152+0000","last_active":"2024-05-27T07:24:29.681965+0000","last_peered":"2024-05-27T07:24:29.681965+0000","last_clean":"2024-05-27T07:24:29.681965+0000","last_became_active":"2024-05-27T07:21:48.580543+0000","last_became_peered":"2024-05-27T07:21:48.580543+0000","last_unstale":"2024-05-27T07:24:29.681965+0000","last_undegraded":"2024-05-27T07:24:29.681965+0000","last_fullsized":"2024-05-27T07:24:29.681965+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:21:47.565641+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:21:47.565641+0000","last_clean_scrub_stamp":"2024-05-27T07:21:47.565641+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T18:45:56.732828+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,4],"acting":[0,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":36,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:23:31.757562+0000","last_change":"2024-05-27T07:22:31.417863+0000","last_active":"2024-05-27T07:23:31.757562+0000","last_peered":"2024-05-27T07:23:31.757562+0000","last_clean":"2024-05-27T07:23:31.757562+0000","last_became_active":"2024-05-27T07:22:31.417298+0000","last_became_peered":"2024-05-27T07:22:31.417298+0000","last_unstale":"2024-05-27T07:23:31.757562+0000","last_undegraded":"2024-05-27T07:23:31.757562+0000","last_fullsized":"2024-05-27T07:23:31.757562+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:21:58.698952+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:21:58.698952+0000","last_clean_scrub_stamp":"2024-05-27T07:21:58.698952+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T12:26:13.857477+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,10,11],"acting":[4,5,10,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.8","version":"88'2","reported_seq":57,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:23:31.626909+0000","last_change":"2024-05-27T07:23:31.626374+0000","last_active":"2024-05-27T07:23:31.626909+0000","last_peered":"2024-05-27T07:23:31.626909+0000","last_clean":"2024-05-27T07:23:31.626909+0000","last_became_active":"2024-05-27T07:23:31.623214+0000","last_became_peered":"2024-05-27T07:23:31.623214+0000","last_unstale":"2024-05-27T07:23:31.626909+0000","last_undegraded":"2024-05-27T07:23:31.626909+0000","last_fullsized":"2024-05-27T07:23:31.626909+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:21:44.543638+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:21:44.543638+0000","last_clean_scrub_stamp":"2024-05-27T07:21:44.543638+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T10:42:47.710138+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,11,0],"acting":[1,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"90'2","reported_seq":41,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:24:29.634809+0000","last_change":"2024-05-27T07:21:48.589605+0000","last_active":"2024-05-27T07:24:29.634809+0000","last_peered":"2024-05-27T07:24:29.634809+0000","last_clean":"2024-05-27T07:24:29.634809+0000","last_became_active":"2024-05-27T07:21:48.588294+0000","last_became_peered":"2024-05-27T07:21:48.588294+0000","last_unstale":"2024-05-27T07:24:29.634809+0000","last_undegraded":"2024-05-27T07:24:29.634809+0000","last_fullsized":"2024-05-27T07:24:29.634809+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:21:47.565641+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:21:47.565641+0000","last_clean_scrub_stamp":"2024-05-27T07:21:47.565641+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T09:02:29.618508+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,10],"acting":[8,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":29,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:23:31.653164+0000","last_change":"2024-05-27T07:21:59.738153+0000","last_active":"2024-05-27T07:23:31.653164+0000","last_peered":"2024-05-27T07:23:31.653164+0000","last_clean":"2024-05-27T07:23:31.653164+0000","last_became_active":"2024-05-27T07:21:59.737394+0000","last_became_peered":"2024-05-27T07:21:59.737394+0000","last_unstale":"2024-05-27T07:23:31.653164+0000","last_undegraded":"2024-05-27T07:23:31.653164+0000","last_fullsized":"2024-05-27T07:23:31.653164+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:21:58.698952+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:21:58.698952+0000","last_clean_scrub_stamp":"2024-05-27T07:21:58.698952+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T17:17:13.877688+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,2,5],"acting":[9,3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.9","version":"88'7","reported_seq":51,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:23:31.102972+0000","last_change":"2024-05-27T07:21:45.562638+0000","last_active":"2024-05-27T07:23:31.102972+0000","last_peered":"2024-05-27T07:23:31.102972+0000","last_clean":"2024-05-27T07:23:31.102972+0000","last_became_active":"2024-05-27T07:21:45.562242+0000","last_became_peered":"2024-05-27T07:21:45.562242+0000","last_unstale":"2024-05-27T07:23:31.102972+0000","last_undegraded":"2024-05-27T07:23:31.102972+0000","last_fullsized":"2024-05-27T07:23:31.102972+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:21:44.543638+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:21:44.543638+0000","last_clean_scrub_stamp":"2024-05-27T07:21:44.543638+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T10:40:15.847513+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,9,10],"acting":[3,9,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.8","version":"90'9","reported_seq":48,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:24:29.683130+0000","last_change":"2024-05-27T07:21:48.598186+0000","last_active":"2024-05-27T07:24:29.683130+0000","last_peered":"2024-05-27T07:24:29.683130+0000","last_clean":"2024-05-27T07:24:29.683130+0000","last_became_active":"2024-05-27T07:21:48.597715+0000","last_became_peered":"2024-05-27T07:21:48.597715+0000","last_unstale":"2024-05-27T07:24:29.683130+0000","last_undegraded":"2024-05-27T07:24:29.683130+0000","last_fullsized":"2024-05-27T07:24:29.683130+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:21:47.565641+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:21:47.565641+0000","last_clean_scrub_stamp":"2024-05-27T07:21:47.565641+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T08:01:35.983687+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,2],"acting":[11,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.c","version":"88'1","reported_seq":30,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:23:31.757394+0000","last_change":"2024-05-27T07:21:59.727247+0000","last_active":"2024-05-27T07:23:31.757394+0000","last_peered":"2024-05-27T07:23:31.757394+0000","last_clean":"2024-05-27T07:23:31.757394+0000","last_became_active":"2024-05-27T07:21:59.726788+0000","last_became_peered":"2024-05-27T07:21:59.726788+0000","last_unstale":"2024-05-27T07:23:31.757394+0000","last_undegraded":"2024-05-27T07:23:31.757394+0000","last_fullsized":"2024-05-27T07:23:31.757394+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:21:58.698952+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:21:58.698952+0000","last_clean_scrub_stamp":"2024-05-27T07:21:58.698952+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T10:09:27.095188+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,9,7,8],"acting":[4,9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.a","version":"88'3","reported_seq":47,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:23:30.982476+0000","last_change":"2024-05-27T07:21:45.569285+0000","last_active":"2024-05-27T07:23:30.982476+0000","last_peered":"2024-05-27T07:23:30.982476+0000","last_clean":"2024-05-27T07:23:30.982476+0000","last_became_active":"2024-05-27T07:21:45.568835+0000","last_became_peered":"2024-05-27T07:21:45.568835+0000","last_unstale":"2024-05-27T07:23:30.982476+0000","last_undegraded":"2024-05-27T07:23:30.982476+0000","last_fullsized":"2024-05-27T07:23:30.982476+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:21:44.543638+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:21:44.543638+0000","last_clean_scrub_stamp":"2024-05-27T07:21:44.543638+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T10:03:23.353800+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,8],"acting":[2,3,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.b","version":"90'8","reported_seq":47,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:24:29.679025+0000","last_change":"2024-05-27T07:21:48.583494+0000","last_active":"2024-05-27T07:24:29.679025+0000","last_peered":"2024-05-27T07:24:29.679025+0000","last_clean":"2024-05-27T07:24:29.679025+0000","last_became_active":"2024-05-27T07:21:48.583141+0000","last_became_peered":"2024-05-27T07:21:48.583141+0000","last_unstale":"2024-05-27T07:24:29.679025+0000","last_undegraded":"2024-05-27T07:24:29.679025+0000","last_fullsized":"2024-05-27T07:24:29.679025+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:21:47.565641+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:21:47.565641+0000","last_clean_scrub_stamp":"2024-05-27T07:21:47.565641+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T10:24:50.475195+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,10],"acting":[3,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":29,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:23:31.757429+0000","last_change":"2024-05-27T07:21:59.733966+0000","last_active":"2024-05-27T07:23:31.757429+0000","last_peered":"2024-05-27T07:23:31.757429+0000","last_clean":"2024-05-27T07:23:31.757429+0000","last_became_active":"2024-05-27T07:21:59.733411+0000","last_became_peered":"2024-05-27T07:21:59.733411+0000","last_unstale":"2024-05-27T07:23:31.757429+0000","last_undegraded":"2024-05-27T07:23:31.757429+0000","last_fullsized":"2024-05-27T07:23:31.757429+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:21:58.698952+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:21:58.698952+0000","last_clean_scrub_stamp":"2024-05-27T07:21:58.698952+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T15:08:04.918086+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,10,3],"acting":[4,2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.b","version":"88'4","reported_seq":50,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:23:31.061461+0000","last_change":"2024-05-27T07:21:45.570967+0000","last_active":"2024-05-27T07:23:31.061461+0000","last_peered":"2024-05-27T07:23:31.061461+0000","last_clean":"2024-05-27T07:23:31.061461+0000","last_became_active":"2024-05-27T07:21:45.570501+0000","last_became_peered":"2024-05-27T07:21:45.570501+0000","last_unstale":"2024-05-27T07:23:31.061461+0000","last_undegraded":"2024-05-27T07:23:31.061461+0000","last_fullsized":"2024-05-27T07:23:31.061461+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:21:44.543638+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:21:44.543638+0000","last_clean_scrub_stamp":"2024-05-27T07:21:44.543638+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T14:11:26.496194+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.a","version":"90'5","reported_seq":42,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:24:29.682141+0000","last_change":"2024-05-27T07:21:48.594690+0000","last_active":"2024-05-27T07:24:29.682141+0000","last_peered":"2024-05-27T07:24:29.682141+0000","last_clean":"2024-05-27T07:24:29.682141+0000","last_became_active":"2024-05-27T07:21:48.594314+0000","last_became_peered":"2024-05-27T07:21:48.594314+0000","last_unstale":"2024-05-27T07:24:29.682141+0000","last_undegraded":"2024-05-27T07:24:29.682141+0000","last_fullsized":"2024-05-27T07:24:29.682141+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:21:47.565641+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:21:47.565641+0000","last_clean_scrub_stamp":"2024-05-27T07:21:47.565641+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T14:06:43.401844+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,6],"acting":[7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":29,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:23:30.978984+0000","last_change":"2024-05-27T07:21:59.732289+0000","last_active":"2024-05-27T07:23:30.978984+0000","last_peered":"2024-05-27T07:23:30.978984+0000","last_clean":"2024-05-27T07:23:30.978984+0000","last_became_active":"2024-05-27T07:21:59.731863+0000","last_became_peered":"2024-05-27T07:21:59.731863+0000","last_unstale":"2024-05-27T07:23:30.978984+0000","last_undegraded":"2024-05-27T07:23:30.978984+0000","last_fullsized":"2024-05-27T07:23:30.978984+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:21:58.698952+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:21:58.698952+0000","last_clean_scrub_stamp":"2024-05-27T07:21:58.698952+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T11:06:38.246484+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,6,7],"acting":[1,5,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.c","version":"88'5","reported_seq":146,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:24:30.031254+0000","last_change":"2024-05-27T07:21:45.791978+0000","last_active":"2024-05-27T07:24:30.031254+0000","last_peered":"2024-05-27T07:24:30.031254+0000","last_clean":"2024-05-27T07:24:30.031254+0000","last_became_active":"2024-05-27T07:21:45.791146+0000","last_became_peered":"2024-05-27T07:21:45.791146+0000","last_unstale":"2024-05-27T07:24:30.031254+0000","last_undegraded":"2024-05-27T07:24:30.031254+0000","last_fullsized":"2024-05-27T07:24:30.031254+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:21:44.543638+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:21:44.543638+0000","last_clean_scrub_stamp":"2024-05-27T07:21:44.543638+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T09:09:24.952876+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1170,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":97,"num_read_kb":2,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,11,0],"acting":[2,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"90'7","reported_seq":44,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:24:29.684312+0000","last_change":"2024-05-27T07:21:48.602786+0000","last_active":"2024-05-27T07:24:29.684312+0000","last_peered":"2024-05-27T07:24:29.684312+0000","last_clean":"2024-05-27T07:24:29.684312+0000","last_became_active":"2024-05-27T07:21:48.602476+0000","last_became_peered":"2024-05-27T07:21:48.602476+0000","last_unstale":"2024-05-27T07:24:29.684312+0000","last_undegraded":"2024-05-27T07:24:29.684312+0000","last_fullsized":"2024-05-27T07:24:29.684312+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:21:47.565641+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:21:47.565641+0000","last_clean_scrub_stamp":"2024-05-27T07:21:47.565641+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T13:22:58.583230+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,11],"acting":[7,1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":29,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:23:30.981918+0000","last_change":"2024-05-27T07:21:59.729595+0000","last_active":"2024-05-27T07:23:30.981918+0000","last_peered":"2024-05-27T07:23:30.981918+0000","last_clean":"2024-05-27T07:23:30.981918+0000","last_became_active":"2024-05-27T07:21:59.729147+0000","last_became_peered":"2024-05-27T07:21:59.729147+0000","last_unstale":"2024-05-27T07:23:30.981918+0000","last_undegraded":"2024-05-27T07:23:30.981918+0000","last_fullsized":"2024-05-27T07:23:30.981918+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:21:58.698952+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:21:58.698952+0000","last_clean_scrub_stamp":"2024-05-27T07:21:58.698952+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T13:31:28.579617+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,9,1,4],"acting":[0,9,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.d","version":"88'3","reported_seq":48,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:23:30.977593+0000","last_change":"2024-05-27T07:21:45.569287+0000","last_active":"2024-05-27T07:23:30.977593+0000","last_peered":"2024-05-27T07:23:30.977593+0000","last_clean":"2024-05-27T07:23:30.977593+0000","last_became_active":"2024-05-27T07:21:45.568935+0000","last_became_peered":"2024-05-27T07:21:45.568935+0000","last_unstale":"2024-05-27T07:23:30.977593+0000","last_undegraded":"2024-05-27T07:23:30.977593+0000","last_fullsized":"2024-05-27T07:23:30.977593+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:21:44.543638+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:21:44.543638+0000","last_clean_scrub_stamp":"2024-05-27T07:21:44.543638+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T07:27:14.674135+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,10,3],"acting":[5,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.c","version":"90'6","reported_seq":43,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:24:29.671545+0000","last_change":"2024-05-27T07:21:48.597718+0000","last_active":"2024-05-27T07:24:29.671545+0000","last_peered":"2024-05-27T07:24:29.671545+0000","last_clean":"2024-05-27T07:24:29.671545+0000","last_became_active":"2024-05-27T07:21:48.597379+0000","last_became_peered":"2024-05-27T07:21:48.597379+0000","last_unstale":"2024-05-27T07:24:29.671545+0000","last_undegraded":"2024-05-27T07:24:29.671545+0000","last_fullsized":"2024-05-27T07:24:29.671545+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:21:47.565641+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:21:47.565641+0000","last_clean_scrub_stamp":"2024-05-27T07:21:47.565641+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T08:48:41.658364+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,5],"acting":[7,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":29,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:23:30.976864+0000","last_change":"2024-05-27T07:21:59.730191+0000","last_active":"2024-05-27T07:23:30.976864+0000","last_peered":"2024-05-27T07:23:30.976864+0000","last_clean":"2024-05-27T07:23:30.976864+0000","last_became_active":"2024-05-27T07:21:59.729787+0000","last_became_peered":"2024-05-27T07:21:59.729787+0000","last_unstale":"2024-05-27T07:23:30.976864+0000","last_undegraded":"2024-05-27T07:23:30.976864+0000","last_fullsized":"2024-05-27T07:23:30.976864+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:21:58.698952+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:21:58.698952+0000","last_clean_scrub_stamp":"2024-05-27T07:21:58.698952+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T13:22:38.961246+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7,1],"acting":[5,2,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.e","version":"88'5","reported_seq":49,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:23:30.974354+0000","last_change":"2024-05-27T07:21:45.559219+0000","last_active":"2024-05-27T07:23:30.974354+0000","last_peered":"2024-05-27T07:23:30.974354+0000","last_clean":"2024-05-27T07:23:30.974354+0000","last_became_active":"2024-05-27T07:21:45.558848+0000","last_became_peered":"2024-05-27T07:21:45.558848+0000","last_unstale":"2024-05-27T07:23:30.974354+0000","last_undegraded":"2024-05-27T07:23:30.974354+0000","last_fullsized":"2024-05-27T07:23:30.974354+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:21:44.543638+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:21:44.543638+0000","last_clean_scrub_stamp":"2024-05-27T07:21:44.543638+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T15:03:55.632162+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.f","version":"90'9","reported_seq":46,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:24:29.672917+0000","last_change":"2024-05-27T07:21:48.596349+0000","last_active":"2024-05-27T07:24:29.672917+0000","last_peered":"2024-05-27T07:24:29.672917+0000","last_clean":"2024-05-27T07:24:29.672917+0000","last_became_active":"2024-05-27T07:21:48.595830+0000","last_became_peered":"2024-05-27T07:21:48.595830+0000","last_unstale":"2024-05-27T07:24:29.672917+0000","last_undegraded":"2024-05-27T07:24:29.672917+0000","last_fullsized":"2024-05-27T07:24:29.672917+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:21:47.565641+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:21:47.565641+0000","last_clean_scrub_stamp":"2024-05-27T07:21:47.565641+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T13:33:11.431988+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,8],"acting":[7,11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.20","version":"90'9","reported_seq":48,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:24:29.674515+0000","last_change":"2024-05-27T07:21:48.598504+0000","last_active":"2024-05-27T07:24:29.674515+0000","last_peered":"2024-05-27T07:24:29.674515+0000","last_clean":"2024-05-27T07:24:29.674515+0000","last_became_active":"2024-05-27T07:21:48.598103+0000","last_became_peered":"2024-05-27T07:21:48.598103+0000","last_unstale":"2024-05-27T07:24:29.674515+0000","last_undegraded":"2024-05-27T07:24:29.674515+0000","last_fullsized":"2024-05-27T07:24:29.674515+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:21:47.565641+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:21:47.565641+0000","last_clean_scrub_stamp":"2024-05-27T07:21:47.565641+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T10:34:02.743735+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,5],"acting":[11,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.21","version":"90'12","reported_seq":51,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:24:29.669423+0000","last_change":"2024-05-27T07:21:48.593379+0000","last_active":"2024-05-27T07:24:29.669423+0000","last_peered":"2024-05-27T07:24:29.669423+0000","last_clean":"2024-05-27T07:24:29.669423+0000","last_became_active":"2024-05-27T07:21:48.592995+0000","last_became_peered":"2024-05-27T07:21:48.592995+0000","last_unstale":"2024-05-27T07:24:29.669423+0000","last_undegraded":"2024-05-27T07:24:29.669423+0000","last_fullsized":"2024-05-27T07:24:29.669423+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:21:47.565641+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:21:47.565641+0000","last_clean_scrub_stamp":"2024-05-27T07:21:47.565641+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T19:01:56.288235+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,4],"acting":[2,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"90'8","reported_seq":47,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:24:29.666667+0000","last_change":"2024-05-27T07:21:48.592837+0000","last_active":"2024-05-27T07:24:29.666667+0000","last_peered":"2024-05-27T07:24:29.666667+0000","last_clean":"2024-05-27T07:24:29.666667+0000","last_became_active":"2024-05-27T07:21:48.592459+0000","last_became_peered":"2024-05-27T07:21:48.592459+0000","last_unstale":"2024-05-27T07:24:29.666667+0000","last_undegraded":"2024-05-27T07:24:29.666667+0000","last_fullsized":"2024-05-27T07:24:29.666667+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:21:47.565641+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:21:47.565641+0000","last_clean_scrub_stamp":"2024-05-27T07:21:47.565641+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T14:00:19.490348+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.23","version":"90'7","reported_seq":46,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:24:29.681980+0000","last_change":"2024-05-27T07:21:48.583139+0000","last_active":"2024-05-27T07:24:29.681980+0000","last_peered":"2024-05-27T07:24:29.681980+0000","last_clean":"2024-05-27T07:24:29.681980+0000","last_became_active":"2024-05-27T07:21:48.582822+0000","last_became_peered":"2024-05-27T07:21:48.582822+0000","last_unstale":"2024-05-27T07:24:29.681980+0000","last_undegraded":"2024-05-27T07:24:29.681980+0000","last_fullsized":"2024-05-27T07:24:29.681980+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:21:47.565641+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:21:47.565641+0000","last_clean_scrub_stamp":"2024-05-27T07:21:47.565641+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T16:30:03.454035+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,4],"acting":[0,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"90'4","reported_seq":43,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:24:29.668048+0000","last_change":"2024-05-27T07:21:48.588906+0000","last_active":"2024-05-27T07:24:29.668048+0000","last_peered":"2024-05-27T07:24:29.668048+0000","last_clean":"2024-05-27T07:24:29.668048+0000","last_became_active":"2024-05-27T07:21:48.588555+0000","last_became_peered":"2024-05-27T07:21:48.588555+0000","last_unstale":"2024-05-27T07:24:29.668048+0000","last_undegraded":"2024-05-27T07:24:29.668048+0000","last_fullsized":"2024-05-27T07:24:29.668048+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:21:47.565641+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:21:47.565641+0000","last_clean_scrub_stamp":"2024-05-27T07:21:47.565641+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T19:04:23.006852+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"90'6","reported_seq":45,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:24:29.671980+0000","last_change":"2024-05-27T07:21:48.590271+0000","last_active":"2024-05-27T07:24:29.671980+0000","last_peered":"2024-05-27T07:24:29.671980+0000","last_clean":"2024-05-27T07:24:29.671980+0000","last_became_active":"2024-05-27T07:21:48.589898+0000","last_became_peered":"2024-05-27T07:21:48.589898+0000","last_unstale":"2024-05-27T07:24:29.671980+0000","last_undegraded":"2024-05-27T07:24:29.671980+0000","last_fullsized":"2024-05-27T07:24:29.671980+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:21:47.565641+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:21:47.565641+0000","last_clean_scrub_stamp":"2024-05-27T07:21:47.565641+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T11:57:43.238211+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,5],"acting":[11,8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.26","version":"90'15","reported_seq":54,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:24:29.672568+0000","last_change":"2024-05-27T07:21:48.597321+0000","last_active":"2024-05-27T07:24:29.672568+0000","last_peered":"2024-05-27T07:24:29.672568+0000","last_clean":"2024-05-27T07:24:29.672568+0000","last_became_active":"2024-05-27T07:21:48.596953+0000","last_became_peered":"2024-05-27T07:21:48.596953+0000","last_unstale":"2024-05-27T07:24:29.672568+0000","last_undegraded":"2024-05-27T07:24:29.672568+0000","last_fullsized":"2024-05-27T07:24:29.672568+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:21:47.565641+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:21:47.565641+0000","last_clean_scrub_stamp":"2024-05-27T07:21:47.565641+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T16:21:17.645098+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,8,1],"acting":[2,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.27","version":"90'2","reported_seq":41,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:24:29.673709+0000","last_change":"2024-05-27T07:21:48.593717+0000","last_active":"2024-05-27T07:24:29.673709+0000","last_peered":"2024-05-27T07:24:29.673709+0000","last_clean":"2024-05-27T07:24:29.673709+0000","last_became_active":"2024-05-27T07:21:48.593356+0000","last_became_peered":"2024-05-27T07:21:48.593356+0000","last_unstale":"2024-05-27T07:24:29.673709+0000","last_undegraded":"2024-05-27T07:24:29.673709+0000","last_fullsized":"2024-05-27T07:24:29.673709+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:21:47.565641+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:21:47.565641+0000","last_clean_scrub_stamp":"2024-05-27T07:21:47.565641+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T17:06:27.341627+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"90'6","reported_seq":45,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:24:29.667694+0000","last_change":"2024-05-27T07:21:48.592360+0000","last_active":"2024-05-27T07:24:29.667694+0000","last_peered":"2024-05-27T07:24:29.667694+0000","last_clean":"2024-05-27T07:24:29.667694+0000","last_became_active":"2024-05-27T07:21:48.591650+0000","last_became_peered":"2024-05-27T07:21:48.591650+0000","last_unstale":"2024-05-27T07:24:29.667694+0000","last_undegraded":"2024-05-27T07:24:29.667694+0000","last_fullsized":"2024-05-27T07:24:29.667694+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:21:47.565641+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:21:47.565641+0000","last_clean_scrub_stamp":"2024-05-27T07:21:47.565641+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T17:32:19.408787+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,3],"acting":[5,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"90'8","reported_seq":47,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:24:29.674125+0000","last_change":"2024-05-27T07:21:48.587319+0000","last_active":"2024-05-27T07:24:29.674125+0000","last_peered":"2024-05-27T07:24:29.674125+0000","last_clean":"2024-05-27T07:24:29.674125+0000","last_became_active":"2024-05-27T07:21:48.586924+0000","last_became_peered":"2024-05-27T07:21:48.586924+0000","last_unstale":"2024-05-27T07:24:29.674125+0000","last_undegraded":"2024-05-27T07:24:29.674125+0000","last_fullsized":"2024-05-27T07:24:29.674125+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:21:47.565641+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:21:47.565641+0000","last_clean_scrub_stamp":"2024-05-27T07:21:47.565641+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T09:28:48.129037+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,11],"acting":[3,2,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2a","version":"90'10","reported_seq":49,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:24:29.684848+0000","last_change":"2024-05-27T07:21:48.604018+0000","last_active":"2024-05-27T07:24:29.684848+0000","last_peered":"2024-05-27T07:24:29.684848+0000","last_clean":"2024-05-27T07:24:29.684848+0000","last_became_active":"2024-05-27T07:21:48.603368+0000","last_became_peered":"2024-05-27T07:21:48.603368+0000","last_unstale":"2024-05-27T07:24:29.684848+0000","last_undegraded":"2024-05-27T07:24:29.684848+0000","last_fullsized":"2024-05-27T07:24:29.684848+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:21:47.565641+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:21:47.565641+0000","last_clean_scrub_stamp":"2024-05-27T07:21:47.565641+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T09:49:42.440480+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2b","version":"90'11","reported_seq":50,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:24:29.678674+0000","last_change":"2024-05-27T07:21:48.589434+0000","last_active":"2024-05-27T07:24:29.678674+0000","last_peered":"2024-05-27T07:24:29.678674+0000","last_clean":"2024-05-27T07:24:29.678674+0000","last_became_active":"2024-05-27T07:21:48.589102+0000","last_became_peered":"2024-05-27T07:21:48.589102+0000","last_unstale":"2024-05-27T07:24:29.678674+0000","last_undegraded":"2024-05-27T07:24:29.678674+0000","last_fullsized":"2024-05-27T07:24:29.678674+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:21:47.565641+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:21:47.565641+0000","last_clean_scrub_stamp":"2024-05-27T07:21:47.565641+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T18:28:52.955138+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2c","version":"90'8","reported_seq":53,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:24:29.672868+0000","last_change":"2024-05-27T07:22:30.802433+0000","last_active":"2024-05-27T07:24:29.672868+0000","last_peered":"2024-05-27T07:24:29.672868+0000","last_clean":"2024-05-27T07:24:29.672868+0000","last_became_active":"2024-05-27T07:22:30.801881+0000","last_became_peered":"2024-05-27T07:22:30.801881+0000","last_unstale":"2024-05-27T07:24:29.672868+0000","last_undegraded":"2024-05-27T07:24:29.672868+0000","last_fullsized":"2024-05-27T07:24:29.672868+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:21:47.565641+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:21:47.565641+0000","last_clean_scrub_stamp":"2024-05-27T07:21:47.565641+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T11:59:17.916052+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,9,8],"acting":[11,9,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":234,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":32768,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":128,"acting":128,"num_store_stats":12},{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":485,"ondisk_log_size":485,"up":192,"acting":192,"num_store_stats":12},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":81,"num_read_kb":69,"num_write":123,"num_write_kb":2150,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":2951840,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":376832,"data_stored":4132576,"data_compressed":249874,"data_compressed_allocated":315392,"data_compressed_original":4096000,"omap_allocated":0,"internal_metadata":0},"log_size":92,"ondisk_log_size":92,"up":3,"acting":3,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":72277,"num_objects":98,"num_object_clones":0,"num_object_copies":294,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":98,"num_whiteouts":0,"num_read":982,"num_read_kb":569,"num_write":257,"num_write_kb":152,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":22,"num_keys_recovered":0,"num_objects_omap":66,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":577536,"data_stored":216983,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":222,"ondisk_log_size":222,"up":96,"acting":96,"num_store_stats":12}],"osd_stats":[{"osd":11,"up_from":71,"seq":304942678065,"num_pgs":29,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32152,"kb_used_data":976,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713000,"statfs":{"total":95995035648,"available":95962112000,"internally_reserved":0,"allocated":999424,"data_stored":519173,"data_compressed":32421,"data_compressed_allocated":45056,"data_compressed_original":90112,"omap_allocated":1596,"internal_metadata":31914436},"hb_peers":[0,1,2,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":10,"up_from":65,"seq":279172874293,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28144,"kb_used_data":1120,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717008,"statfs":{"total":95995035648,"available":95966216192,"internally_reserved":0,"allocated":1146880,"data_stored":1170976,"data_compressed":68581,"data_compressed_allocated":90112,"data_compressed_original":679936,"omap_allocated":1600,"internal_metadata":27654592},"hb_peers":[0,1,2,3,4,5,6,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":60,"seq":257698037819,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36336,"kb_used_data":1000,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708816,"statfs":{"total":95995035648,"available":95957827584,"internally_reserved":0,"allocated":1024000,"data_stored":523177,"data_compressed":32421,"data_compressed_allocated":45056,"data_compressed_original":90112,"omap_allocated":1600,"internal_metadata":36174272},"hb_peers":[0,1,2,3,4,5,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":8,"up_from":54,"seq":231928234049,"num_pgs":31,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28004,"kb_used_data":988,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717148,"statfs":{"total":95995035648,"available":95966359552,"internally_reserved":0,"allocated":1011712,"data_stored":519437,"data_compressed":32421,"data_compressed_allocated":45056,"data_compressed_original":90112,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":48,"seq":206158430277,"num_pgs":37,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32220,"kb_used_data":1044,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712932,"statfs":{"total":95995035648,"available":95962042368,"internally_reserved":0,"allocated":1069056,"data_stored":1116629,"data_compressed":68581,"data_compressed_allocated":90112,"data_compressed_original":679936,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":6,"up_from":43,"seq":184683593802,"num_pgs":32,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27988,"kb_used_data":972,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717164,"statfs":{"total":95995035648,"available":95966375936,"internally_reserved":0,"allocated":995328,"data_stored":514585,"data_compressed":32421,"data_compressed_allocated":45056,"data_compressed_original":90112,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574948,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36376,"kb_used_data":1040,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708776,"statfs":{"total":95995035648,"available":95957786624,"internally_reserved":0,"allocated":1064960,"data_stored":580482,"data_compressed":32421,"data_compressed_allocated":45056,"data_compressed_original":90112,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738473,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28012,"kb_used_data":996,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717140,"statfs":{"total":95995035648,"available":95966351360,"internally_reserved":0,"allocated":1019904,"data_stored":516041,"data_compressed":32421,"data_compressed_allocated":45056,"data_compressed_original":90112,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411423,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36388,"kb_used_data":1052,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708764,"statfs":{"total":95995035648,"available":95957774336,"internally_reserved":0,"allocated":1077248,"data_stored":581122,"data_compressed":32421,"data_compressed_allocated":45056,"data_compressed_original":90112,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,1,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":3,"up_from":26,"seq":111669149786,"num_pgs":40,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36320,"kb_used_data":984,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708832,"statfs":{"total":95995035648,"available":95957843968,"internally_reserved":0,"allocated":1007616,"data_stored":514293,"data_compressed":32421,"data_compressed_allocated":45056,"data_compressed_original":90112,"omap_allocated":1593,"internal_metadata":36174279},"hb_peers":[0,1,2,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":4,"up_from":32,"seq":137438953556,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28004,"kb_used_data":988,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717148,"statfs":{"total":95995035648,"available":95966359552,"internally_reserved":0,"allocated":1011712,"data_stored":519303,"data_compressed":32421,"data_compressed_allocated":45056,"data_compressed_original":90112,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":5,"up_from":37,"seq":158913790031,"num_pgs":44,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28068,"kb_used_data":1052,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717084,"statfs":{"total":95995035648,"available":95966294016,"internally_reserved":0,"allocated":1077248,"data_stored":1111679,"data_compressed":68581,"data_compressed_allocated":90112,"data_compressed_original":679936,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":34537,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":34537,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":36160,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":36160,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":36160,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":36160,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":36160,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":2178,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":86016,"data_stored":66619,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":98304,"data_stored":67259,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":430,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1344,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":3352,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":722,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":4206,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1478,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":1122,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":118784,"data_stored":67059,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1214,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":8192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":8192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-05-27T07:24:34.475 DEBUG:teuthology.orchestra.run.smithi052:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:38f9d900c83c63799fbdbe61acc9a11b0d3554a6 shell --fsid 73df2636-1bf8-11ef-bc9b-c7b262605968 -- ceph pg dump --format=json 2024-05-27T07:24:34.729 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:24:34 smithi052 ceph-mon[31955]: pgmap v382: 129 pgs: 129 active+clean; 647 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 511 B/s wr, 9 op/s 2024-05-27T07:24:34.729 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:24:34 smithi052 ceph-mon[31955]: from='client.15033 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-05-27T07:24:34.735 INFO:teuthology.orchestra.run.smithi052.stderr:Inferring config /var/lib/ceph/73df2636-1bf8-11ef-bc9b-c7b262605968/mon.a/config 2024-05-27T07:24:34.940 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:24:34 smithi105 ceph-mon[35977]: pgmap v382: 129 pgs: 129 active+clean; 647 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 511 B/s wr, 9 op/s 2024-05-27T07:24:34.940 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:24:34 smithi105 ceph-mon[35977]: from='client.15033 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-05-27T07:24:35.102 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:24:34 smithi173 ceph-mon[37000]: pgmap v382: 129 pgs: 129 active+clean; 647 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 511 B/s wr, 9 op/s 2024-05-27T07:24:35.102 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:24:34 smithi173 ceph-mon[37000]: from='client.15033 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-05-27T07:24:36.307 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:36.316+0000 7f53d6874700 1 -- 172.21.15.52:0/1029684124 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f53d0103a50 msgr2=0x7f53d01106e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:24:36.307 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:36.316+0000 7f53d6874700 1 --2- 172.21.15.52:0/1029684124 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f53d0103a50 0x7f53d01106e0 secure :-1 s=READY pgs=298 cs=0 l=1 rev1=1 crypto rx=0x7f53b8009990 tx=0x7f53b8009ca0 comp rx=0 tx=0).stop 2024-05-27T07:24:36.307 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:36.317+0000 7f53d6874700 1 -- 172.21.15.52:0/1029684124 shutdown_connections 2024-05-27T07:24:36.307 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:36.317+0000 7f53d6874700 1 --2- 172.21.15.52:0/1029684124 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f53d0110c20 0x7f53d0113040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:24:36.307 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:36.317+0000 7f53d6874700 1 --2- 172.21.15.52:0/1029684124 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f53d0103070 0x7f53d0103480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:24:36.307 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:36.317+0000 7f53d6874700 1 --2- 172.21.15.52:0/1029684124 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f53d0103a50 0x7f53d01106e0 unknown :-1 s=CLOSED pgs=298 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:24:36.307 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:36.317+0000 7f53d6874700 1 -- 172.21.15.52:0/1029684124 >> 172.21.15.52:0/1029684124 conn(0x7f53d00fedb0 msgr2=0x7f53d01011d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:24:36.307 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:36.317+0000 7f53d6874700 1 -- 172.21.15.52:0/1029684124 shutdown_connections 2024-05-27T07:24:36.307 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:36.317+0000 7f53d6874700 1 -- 172.21.15.52:0/1029684124 wait complete. 2024-05-27T07:24:36.309 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:36.317+0000 7f53d6874700 1 Processor -- start 2024-05-27T07:24:36.309 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:36.317+0000 7f53d6874700 1 -- start start 2024-05-27T07:24:36.309 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:36.318+0000 7f53d6874700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f53d0103070 0x7f53d019d050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:24:36.309 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:36.318+0000 7f53d6874700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f53d0103a50 0x7f53d019d5f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:24:36.309 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:36.318+0000 7f53d6874700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f53d0110c20 0x7f53d01a1880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:24:36.309 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:36.318+0000 7f53d6874700 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x7f53d01166a0 con 0x7f53d0103a50 2024-05-27T07:24:36.310 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:36.318+0000 7f53d6874700 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f53d0116520 con 0x7f53d0110c20 2024-05-27T07:24:36.310 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:36.318+0000 7f53d6874700 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f53d0116820 con 0x7f53d0103070 2024-05-27T07:24:36.310 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:36.318+0000 7f53c7fff700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f53d0103a50 0x7f53d019d5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:24:36.310 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:36.318+0000 7f53c7fff700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f53d0103a50 0x7f53d019d5f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.52:49950/0 (socket says 172.21.15.52:49950) 2024-05-27T07:24:36.310 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:36.318+0000 7f53c7fff700 1 -- 172.21.15.52:0/3737692311 learned_addr learned my addr 172.21.15.52:0/3737692311 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-05-27T07:24:36.310 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:36.318+0000 7f53cffff700 1 --2- 172.21.15.52:0/3737692311 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f53d0103070 0x7f53d019d050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:24:36.310 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:36.318+0000 7f53d4e11700 1 --2- 172.21.15.52:0/3737692311 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f53d0110c20 0x7f53d01a1880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:24:36.310 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:36.318+0000 7f53c7fff700 1 -- 172.21.15.52:0/3737692311 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f53d0103070 msgr2=0x7f53d019d050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:24:36.310 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:36.318+0000 7f53c7fff700 1 --2- 172.21.15.52:0/3737692311 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f53d0103070 0x7f53d019d050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:24:36.310 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:36.318+0000 7f53c7fff700 1 -- 172.21.15.52:0/3737692311 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f53d0110c20 msgr2=0x7f53d01a1880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:24:36.310 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:36.318+0000 7f53c7fff700 1 --2- 172.21.15.52:0/3737692311 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f53d0110c20 0x7f53d01a1880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:24:36.310 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:36.318+0000 7f53c7fff700 1 -- 172.21.15.52:0/3737692311 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f53d01a1e70 con 0x7f53d0103a50 2024-05-27T07:24:36.311 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:36.319+0000 7f53c7fff700 1 --2- 172.21.15.52:0/3737692311 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f53d0103a50 0x7f53d019d5f0 secure :-1 s=READY pgs=299 cs=0 l=1 rev1=1 crypto rx=0x7f53b8000c00 tx=0x7f53b80037a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:24:36.311 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:36.319+0000 7f53cdffb700 1 -- 172.21.15.52:0/3737692311 <== mon.0 v2:172.21.15.52:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f53b8004830 con 0x7f53d0103a50 2024-05-27T07:24:36.311 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:36.319+0000 7f53cdffb700 1 -- 172.21.15.52:0/3737692311 <== mon.0 v2:172.21.15.52:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f53b8003b80 con 0x7f53d0103a50 2024-05-27T07:24:36.311 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:36.319+0000 7f53cdffb700 1 -- 172.21.15.52:0/3737692311 <== mon.0 v2:172.21.15.52:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f53b80053a0 con 0x7f53d0103a50 2024-05-27T07:24:36.311 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:36.319+0000 7f53d6874700 1 -- 172.21.15.52:0/3737692311 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f53d01a2100 con 0x7f53d0103a50 2024-05-27T07:24:36.311 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:36.319+0000 7f53d6874700 1 -- 172.21.15.52:0/3737692311 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f53d01a2530 con 0x7f53d0103a50 2024-05-27T07:24:36.311 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:36.320+0000 7f53d6874700 1 -- 172.21.15.52:0/3737692311 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f53d00512b0 con 0x7f53d0103a50 2024-05-27T07:24:36.312 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:36.322+0000 7f53cdffb700 1 -- 172.21.15.52:0/3737692311 <== mon.0 v2:172.21.15.52:3300/0 4 ==== mgrmap(e 18) ==== 149652+0+0 (secure 0 0 0) 0x7f53b8005580 con 0x7f53d0103a50 2024-05-27T07:24:36.312 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:36.322+0000 7f53cdffb700 1 --2- 172.21.15.52:0/3737692311 >> [v2:172.21.15.52:6800/3513075805,v1:172.21.15.52:6801/3513075805] conn(0x7f53bc0b2900 0x7f53bc0b4e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:24:36.312 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:36.323+0000 7f53cffff700 1 --2- 172.21.15.52:0/3737692311 >> [v2:172.21.15.52:6800/3513075805,v1:172.21.15.52:6801/3513075805] conn(0x7f53bc0b2900 0x7f53bc0b4e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:24:36.312 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:36.323+0000 7f53cdffb700 1 -- 172.21.15.52:0/3737692311 <== mon.0 v2:172.21.15.52:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9121+0+0 (secure 0 0 0) 0x7f53b80e3200 con 0x7f53d0103a50 2024-05-27T07:24:36.313 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:36.323+0000 7f53cffff700 1 --2- 172.21.15.52:0/3737692311 >> [v2:172.21.15.52:6800/3513075805,v1:172.21.15.52:6801/3513075805] conn(0x7f53bc0b2900 0x7f53bc0b4e10 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7f53c0005ea0 tx=0x7f53c0005e30 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:24:36.317 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:36.328+0000 7f53cdffb700 1 -- 172.21.15.52:0/3737692311 <== mon.0 v2:172.21.15.52:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191963 (secure 0 0 0) 0x7f53b8013070 con 0x7f53d0103a50 2024-05-27T07:24:36.529 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:36.539+0000 7f53d6874700 1 -- 172.21.15.52:0/3737692311 --> [v2:172.21.15.52:6800/3513075805,v1:172.21.15.52:6801/3513075805] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f53d0063bf0 con 0x7f53bc0b2900 2024-05-27T07:24:36.538 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:36.548+0000 7f53cdffb700 1 -- 172.21.15.52:0/3737692311 <== mgr.14152 v2:172.21.15.52:6800/3513075805 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+340881 (secure 0 0 0) 0x7f53d0063bf0 con 0x7f53bc0b2900 2024-05-27T07:24:36.543 INFO:teuthology.orchestra.run.smithi052.stdout: 2024-05-27T07:24:36.545 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:36.553+0000 7f53d6874700 1 -- 172.21.15.52:0/3737692311 >> [v2:172.21.15.52:6800/3513075805,v1:172.21.15.52:6801/3513075805] conn(0x7f53bc0b2900 msgr2=0x7f53bc0b4e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:24:36.545 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:36.553+0000 7f53d6874700 1 --2- 172.21.15.52:0/3737692311 >> [v2:172.21.15.52:6800/3513075805,v1:172.21.15.52:6801/3513075805] conn(0x7f53bc0b2900 0x7f53bc0b4e10 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7f53c0005ea0 tx=0x7f53c0005e30 comp rx=0 tx=0).stop 2024-05-27T07:24:36.545 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:36.553+0000 7f53d6874700 1 -- 172.21.15.52:0/3737692311 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f53d0103a50 msgr2=0x7f53d019d5f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:24:36.545 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:36.553+0000 7f53d6874700 1 --2- 172.21.15.52:0/3737692311 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f53d0103a50 0x7f53d019d5f0 secure :-1 s=READY pgs=299 cs=0 l=1 rev1=1 crypto rx=0x7f53b8000c00 tx=0x7f53b80037a0 comp rx=0 tx=0).stop 2024-05-27T07:24:36.546 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:36.553+0000 7f53d6874700 1 -- 172.21.15.52:0/3737692311 shutdown_connections 2024-05-27T07:24:36.546 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:36.553+0000 7f53d6874700 1 --2- 172.21.15.52:0/3737692311 >> [v2:172.21.15.52:6800/3513075805,v1:172.21.15.52:6801/3513075805] conn(0x7f53bc0b2900 0x7f53bc0b4e10 unknown :-1 s=CLOSED pgs=201 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:24:36.546 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:36.553+0000 7f53d6874700 1 --2- 172.21.15.52:0/3737692311 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f53d0110c20 0x7f53d01a1880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:24:36.546 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:36.553+0000 7f53d6874700 1 --2- 172.21.15.52:0/3737692311 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f53d0103070 0x7f53d019d050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:24:36.546 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:36.553+0000 7f53d6874700 1 --2- 172.21.15.52:0/3737692311 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f53d0103a50 0x7f53d019d5f0 unknown :-1 s=CLOSED pgs=299 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:24:36.546 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:36.553+0000 7f53d6874700 1 -- 172.21.15.52:0/3737692311 >> 172.21.15.52:0/3737692311 conn(0x7f53d00fedb0 msgr2=0x7f53d0112880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:24:36.546 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:36.554+0000 7f53d6874700 1 -- 172.21.15.52:0/3737692311 shutdown_connections 2024-05-27T07:24:36.546 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:36.554+0000 7f53d6874700 1 -- 172.21.15.52:0/3737692311 wait complete. 2024-05-27T07:24:36.546 INFO:teuthology.orchestra.run.smithi052.stderr:dumped all 2024-05-27T07:24:36.940 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:24:36 smithi105 ceph-mon[35977]: pgmap v383: 129 pgs: 129 active+clean; 647 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 6.1 KiB/s rd, 597 B/s wr, 12 op/s 2024-05-27T07:24:36.979 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:24:36 smithi052 ceph-mon[31955]: pgmap v383: 129 pgs: 129 active+clean; 647 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 6.1 KiB/s rd, 597 B/s wr, 12 op/s 2024-05-27T07:24:37.101 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:24:36 smithi173 ceph-mon[37000]: pgmap v383: 129 pgs: 129 active+clean; 647 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 6.1 KiB/s rd, 597 B/s wr, 12 op/s 2024-05-27T07:24:37.289 INFO:teuthology.orchestra.run.smithi052.stdout:{"pg_ready":true,"pg_map":{"version":383,"stamp":"2024-05-27T07:24:35.688597+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":662955,"num_objects":102,"num_object_clones":0,"num_object_copies":308,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":102,"num_whiteouts":0,"num_read":1099,"num_read_kb":659,"num_write":385,"num_write_kb":2305,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":2951862,"num_keys_recovered":0,"num_objects_omap":66,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":818,"ondisk_log_size":818,"up":419,"acting":419,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":419,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":378004,"kb_used_data":12212,"kb_used_omap":18,"kb_used_meta":365677,"kb_avail":1124563820,"statfs":{"total":1151940427776,"available":1151553351680,"internally_reserved":0,"allocated":12505088,"data_stored":8187287,"data_compressed":497532,"data_compressed_allocated":675840,"data_compressed_original":2850816,"omap_allocated":19143,"internal_metadata":374453561},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":13,"apply_latency_ms":13,"commit_latency_ns":13000000,"apply_latency_ns":13000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":531,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":138,"num_read_kb":73,"num_write":9,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003712"},"pg_stats":[{"pgid":"3.3f","version":"90'8","reported_seq":47,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:24:29.683427+0000","last_change":"2024-05-27T07:21:48.595870+0000","last_active":"2024-05-27T07:24:29.683427+0000","last_peered":"2024-05-27T07:24:29.683427+0000","last_clean":"2024-05-27T07:24:29.683427+0000","last_became_active":"2024-05-27T07:21:48.595020+0000","last_became_peered":"2024-05-27T07:21:48.595020+0000","last_unstale":"2024-05-27T07:24:29.683427+0000","last_undegraded":"2024-05-27T07:24:29.683427+0000","last_fullsized":"2024-05-27T07:24:29.683427+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:21:47.565641+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:21:47.565641+0000","last_clean_scrub_stamp":"2024-05-27T07:21:47.565641+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T16:43:08.920344+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.3e","version":"90'4","reported_seq":43,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:24:29.672501+0000","last_change":"2024-05-27T07:21:48.590237+0000","last_active":"2024-05-27T07:24:29.672501+0000","last_peered":"2024-05-27T07:24:29.672501+0000","last_clean":"2024-05-27T07:24:29.672501+0000","last_became_active":"2024-05-27T07:21:48.589876+0000","last_became_peered":"2024-05-27T07:21:48.589876+0000","last_unstale":"2024-05-27T07:24:29.672501+0000","last_undegraded":"2024-05-27T07:24:29.672501+0000","last_fullsized":"2024-05-27T07:24:29.672501+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:21:47.565641+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:21:47.565641+0000","last_clean_scrub_stamp":"2024-05-27T07:21:47.565641+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T16:23:08.881871+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,11,10],"acting":[0,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3d","version":"90'7","reported_seq":46,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:24:29.683006+0000","last_change":"2024-05-27T07:21:48.597744+0000","last_active":"2024-05-27T07:24:29.683006+0000","last_peered":"2024-05-27T07:24:29.683006+0000","last_clean":"2024-05-27T07:24:29.683006+0000","last_became_active":"2024-05-27T07:21:48.597410+0000","last_became_peered":"2024-05-27T07:21:48.597410+0000","last_unstale":"2024-05-27T07:24:29.683006+0000","last_undegraded":"2024-05-27T07:24:29.683006+0000","last_fullsized":"2024-05-27T07:24:29.683006+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:21:47.565641+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:21:47.565641+0000","last_clean_scrub_stamp":"2024-05-27T07:21:47.565641+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T19:06:49.643152+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"90'8","reported_seq":24,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:24:29.674553+0000","last_change":"2024-05-27T07:22:30.800824+0000","last_active":"2024-05-27T07:24:29.674553+0000","last_peered":"2024-05-27T07:24:29.674553+0000","last_clean":"2024-05-27T07:24:29.674553+0000","last_became_active":"2024-05-27T07:22:30.800415+0000","last_became_peered":"2024-05-27T07:22:30.800415+0000","last_unstale":"2024-05-27T07:24:29.674553+0000","last_undegraded":"2024-05-27T07:24:29.674553+0000","last_fullsized":"2024-05-27T07:24:29.674553+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:21:47.565641+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:21:47.565641+0000","last_clean_scrub_stamp":"2024-05-27T07:21:47.565641+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T11:06:10.451686+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,2],"acting":[9,8,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.3b","version":"90'4","reported_seq":43,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:24:29.652869+0000","last_change":"2024-05-27T07:21:48.583219+0000","last_active":"2024-05-27T07:24:29.652869+0000","last_peered":"2024-05-27T07:24:29.652869+0000","last_clean":"2024-05-27T07:24:29.652869+0000","last_became_active":"2024-05-27T07:21:48.582813+0000","last_became_peered":"2024-05-27T07:21:48.582813+0000","last_unstale":"2024-05-27T07:24:29.652869+0000","last_undegraded":"2024-05-27T07:24:29.652869+0000","last_fullsized":"2024-05-27T07:24:29.652869+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:21:47.565641+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:21:47.565641+0000","last_clean_scrub_stamp":"2024-05-27T07:21:47.565641+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T18:05:05.770965+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,3,6],"acting":[8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.3a","version":"90'6","reported_seq":45,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:24:29.671507+0000","last_change":"2024-05-27T07:21:48.592049+0000","last_active":"2024-05-27T07:24:29.671507+0000","last_peered":"2024-05-27T07:24:29.671507+0000","last_clean":"2024-05-27T07:24:29.671507+0000","last_became_active":"2024-05-27T07:21:48.591677+0000","last_became_peered":"2024-05-27T07:21:48.591677+0000","last_unstale":"2024-05-27T07:24:29.671507+0000","last_undegraded":"2024-05-27T07:24:29.671507+0000","last_fullsized":"2024-05-27T07:24:29.671507+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:21:47.565641+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:21:47.565641+0000","last_clean_scrub_stamp":"2024-05-27T07:21:47.565641+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T16:25:49.755893+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,4],"acting":[10,9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.39","version":"90'5","reported_seq":42,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:24:29.672343+0000","last_change":"2024-05-27T07:21:48.597876+0000","last_active":"2024-05-27T07:24:29.672343+0000","last_peered":"2024-05-27T07:24:29.672343+0000","last_clean":"2024-05-27T07:24:29.672343+0000","last_became_active":"2024-05-27T07:21:48.597566+0000","last_became_peered":"2024-05-27T07:21:48.597566+0000","last_unstale":"2024-05-27T07:24:29.672343+0000","last_undegraded":"2024-05-27T07:24:29.672343+0000","last_fullsized":"2024-05-27T07:24:29.672343+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:21:47.565641+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:21:47.565641+0000","last_clean_scrub_stamp":"2024-05-27T07:21:47.565641+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T09:30:43.438209+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,5],"acting":[7,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"90'9","reported_seq":48,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:24:29.675504+0000","last_change":"2024-05-27T07:21:48.593412+0000","last_active":"2024-05-27T07:24:29.675504+0000","last_peered":"2024-05-27T07:24:29.675504+0000","last_clean":"2024-05-27T07:24:29.675504+0000","last_became_active":"2024-05-27T07:21:48.593070+0000","last_became_peered":"2024-05-27T07:21:48.593070+0000","last_unstale":"2024-05-27T07:24:29.675504+0000","last_undegraded":"2024-05-27T07:24:29.675504+0000","last_fullsized":"2024-05-27T07:24:29.675504+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:21:47.565641+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:21:47.565641+0000","last_clean_scrub_stamp":"2024-05-27T07:21:47.565641+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T14:53:49.382796+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,10],"acting":[3,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"90'11","reported_seq":50,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:24:29.684456+0000","last_change":"2024-05-27T07:21:48.607244+0000","last_active":"2024-05-27T07:24:29.684456+0000","last_peered":"2024-05-27T07:24:29.684456+0000","last_clean":"2024-05-27T07:24:29.684456+0000","last_became_active":"2024-05-27T07:21:48.606238+0000","last_became_peered":"2024-05-27T07:21:48.606238+0000","last_unstale":"2024-05-27T07:24:29.684456+0000","last_undegraded":"2024-05-27T07:24:29.684456+0000","last_fullsized":"2024-05-27T07:24:29.684456+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:21:47.565641+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:21:47.565641+0000","last_clean_scrub_stamp":"2024-05-27T07:21:47.565641+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T12:36:33.142111+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,11],"acting":[1,5,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.36","version":"90'16","reported_seq":55,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:24:29.670091+0000","last_change":"2024-05-27T07:21:48.593275+0000","last_active":"2024-05-27T07:24:29.670091+0000","last_peered":"2024-05-27T07:24:29.670091+0000","last_clean":"2024-05-27T07:24:29.670091+0000","last_became_active":"2024-05-27T07:21:48.592890+0000","last_became_peered":"2024-05-27T07:21:48.592890+0000","last_unstale":"2024-05-27T07:24:29.670091+0000","last_undegraded":"2024-05-27T07:24:29.670091+0000","last_fullsized":"2024-05-27T07:24:29.670091+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:21:47.565641+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:21:47.565641+0000","last_clean_scrub_stamp":"2024-05-27T07:21:47.565641+0000","objects_scrubbed":0,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T13:47:20.827824+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,4],"acting":[10,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.35","version":"90'6","reported_seq":43,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:24:29.685097+0000","last_change":"2024-05-27T07:21:48.603424+0000","last_active":"2024-05-27T07:24:29.685097+0000","last_peered":"2024-05-27T07:24:29.685097+0000","last_clean":"2024-05-27T07:24:29.685097+0000","last_became_active":"2024-05-27T07:21:48.602962+0000","last_became_peered":"2024-05-27T07:21:48.602962+0000","last_unstale":"2024-05-27T07:24:29.685097+0000","last_undegraded":"2024-05-27T07:24:29.685097+0000","last_fullsized":"2024-05-27T07:24:29.685097+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:21:47.565641+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:21:47.565641+0000","last_clean_scrub_stamp":"2024-05-27T07:21:47.565641+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T19:19:42.063063+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,8,1],"acting":[7,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.34","version":"90'7","reported_seq":46,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:24:29.672689+0000","last_change":"2024-05-27T07:21:48.589323+0000","last_active":"2024-05-27T07:24:29.672689+0000","last_peered":"2024-05-27T07:24:29.672689+0000","last_clean":"2024-05-27T07:24:29.672689+0000","last_became_active":"2024-05-27T07:21:48.587435+0000","last_became_peered":"2024-05-27T07:21:48.587435+0000","last_unstale":"2024-05-27T07:24:29.672689+0000","last_undegraded":"2024-05-27T07:24:29.672689+0000","last_fullsized":"2024-05-27T07:24:29.672689+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:21:47.565641+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:21:47.565641+0000","last_clean_scrub_stamp":"2024-05-27T07:21:47.565641+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T13:34:44.340376+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,8],"acting":[4,10,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"90'10","reported_seq":49,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:24:29.660510+0000","last_change":"2024-05-27T07:21:48.593759+0000","last_active":"2024-05-27T07:24:29.660510+0000","last_peered":"2024-05-27T07:24:29.660510+0000","last_clean":"2024-05-27T07:24:29.660510+0000","last_became_active":"2024-05-27T07:21:48.593099+0000","last_became_peered":"2024-05-27T07:21:48.593099+0000","last_unstale":"2024-05-27T07:24:29.660510+0000","last_undegraded":"2024-05-27T07:24:29.660510+0000","last_fullsized":"2024-05-27T07:24:29.660510+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:21:47.565641+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:21:47.565641+0000","last_clean_scrub_stamp":"2024-05-27T07:21:47.565641+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T13:49:00.390792+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,5],"acting":[6,10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.32","version":"90'7","reported_seq":46,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:24:29.675244+0000","last_change":"2024-05-27T07:21:48.587486+0000","last_active":"2024-05-27T07:24:29.675244+0000","last_peered":"2024-05-27T07:24:29.675244+0000","last_clean":"2024-05-27T07:24:29.675244+0000","last_became_active":"2024-05-27T07:21:48.587159+0000","last_became_peered":"2024-05-27T07:21:48.587159+0000","last_unstale":"2024-05-27T07:24:29.675244+0000","last_undegraded":"2024-05-27T07:24:29.675244+0000","last_fullsized":"2024-05-27T07:24:29.675244+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:21:47.565641+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:21:47.565641+0000","last_clean_scrub_stamp":"2024-05-27T07:21:47.565641+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T15:34:11.908442+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,10],"acting":[0,8,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"90'7","reported_seq":46,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:24:29.682260+0000","last_change":"2024-05-27T07:21:48.596817+0000","last_active":"2024-05-27T07:24:29.682260+0000","last_peered":"2024-05-27T07:24:29.682260+0000","last_clean":"2024-05-27T07:24:29.682260+0000","last_became_active":"2024-05-27T07:21:48.596469+0000","last_became_peered":"2024-05-27T07:21:48.596469+0000","last_unstale":"2024-05-27T07:24:29.682260+0000","last_undegraded":"2024-05-27T07:24:29.682260+0000","last_fullsized":"2024-05-27T07:24:29.682260+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:21:47.565641+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:21:47.565641+0000","last_clean_scrub_stamp":"2024-05-27T07:21:47.565641+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T12:38:41.494060+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,2],"acting":[0,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.30","version":"90'9","reported_seq":48,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:24:29.683528+0000","last_change":"2024-05-27T07:21:48.604249+0000","last_active":"2024-05-27T07:24:29.683528+0000","last_peered":"2024-05-27T07:24:29.683528+0000","last_clean":"2024-05-27T07:24:29.683528+0000","last_became_active":"2024-05-27T07:21:48.603585+0000","last_became_peered":"2024-05-27T07:21:48.603585+0000","last_unstale":"2024-05-27T07:24:29.683528+0000","last_undegraded":"2024-05-27T07:24:29.683528+0000","last_fullsized":"2024-05-27T07:24:29.683528+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:21:47.565641+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:21:47.565641+0000","last_clean_scrub_stamp":"2024-05-27T07:21:47.565641+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T07:22:14.839824+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2f","version":"90'8","reported_seq":47,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:24:29.683480+0000","last_change":"2024-05-27T07:21:48.596035+0000","last_active":"2024-05-27T07:24:29.683480+0000","last_peered":"2024-05-27T07:24:29.683480+0000","last_clean":"2024-05-27T07:24:29.683480+0000","last_became_active":"2024-05-27T07:21:48.595509+0000","last_became_peered":"2024-05-27T07:21:48.595509+0000","last_unstale":"2024-05-27T07:24:29.683480+0000","last_undegraded":"2024-05-27T07:24:29.683480+0000","last_fullsized":"2024-05-27T07:24:29.683480+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:21:47.565641+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:21:47.565641+0000","last_clean_scrub_stamp":"2024-05-27T07:21:47.565641+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T13:23:03.734982+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.2e","version":"90'10","reported_seq":49,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:24:29.682582+0000","last_change":"2024-05-27T07:21:48.590002+0000","last_active":"2024-05-27T07:24:29.682582+0000","last_peered":"2024-05-27T07:24:29.682582+0000","last_clean":"2024-05-27T07:24:29.682582+0000","last_became_active":"2024-05-27T07:21:48.589608+0000","last_became_peered":"2024-05-27T07:21:48.589608+0000","last_unstale":"2024-05-27T07:24:29.682582+0000","last_undegraded":"2024-05-27T07:24:29.682582+0000","last_fullsized":"2024-05-27T07:24:29.682582+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:21:47.565641+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:21:47.565641+0000","last_clean_scrub_stamp":"2024-05-27T07:21:47.565641+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T09:33:39.174351+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,1,10],"acting":[9,1,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.2d","version":"90'7","reported_seq":46,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:24:29.671511+0000","last_change":"2024-05-27T07:21:48.594885+0000","last_active":"2024-05-27T07:24:29.671511+0000","last_peered":"2024-05-27T07:24:29.671511+0000","last_clean":"2024-05-27T07:24:29.671511+0000","last_became_active":"2024-05-27T07:21:48.594521+0000","last_became_peered":"2024-05-27T07:21:48.594521+0000","last_unstale":"2024-05-27T07:24:29.671511+0000","last_undegraded":"2024-05-27T07:24:29.671511+0000","last_fullsized":"2024-05-27T07:24:29.671511+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:21:47.565641+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:21:47.565641+0000","last_clean_scrub_stamp":"2024-05-27T07:21:47.565641+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T08:54:59.311938+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":36,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:23:30.982372+0000","last_change":"2024-05-27T07:22:30.857546+0000","last_active":"2024-05-27T07:23:30.982372+0000","last_peered":"2024-05-27T07:23:30.982372+0000","last_clean":"2024-05-27T07:23:30.982372+0000","last_became_active":"2024-05-27T07:22:30.856808+0000","last_became_peered":"2024-05-27T07:22:30.856808+0000","last_unstale":"2024-05-27T07:23:30.982372+0000","last_undegraded":"2024-05-27T07:23:30.982372+0000","last_fullsized":"2024-05-27T07:23:30.982372+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:21:58.698952+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:21:58.698952+0000","last_clean_scrub_stamp":"2024-05-27T07:21:58.698952+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T16:09:25.480864+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,8,0],"acting":[2,3,8,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1f","version":"88'13","reported_seq":158,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:24:34.044501+0000","last_change":"2024-05-27T07:21:45.566489+0000","last_active":"2024-05-27T07:24:34.044501+0000","last_peered":"2024-05-27T07:24:34.044501+0000","last_clean":"2024-05-27T07:24:34.044501+0000","last_became_active":"2024-05-27T07:21:45.566115+0000","last_became_peered":"2024-05-27T07:21:45.566115+0000","last_unstale":"2024-05-27T07:24:34.044501+0000","last_undegraded":"2024-05-27T07:24:34.044501+0000","last_fullsized":"2024-05-27T07:24:34.044501+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:21:44.543638+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:21:44.543638+0000","last_clean_scrub_stamp":"2024-05-27T07:21:44.543638+0000","objects_scrubbed":0,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T07:24:11.735014+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":100,"num_read_kb":100,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,8],"acting":[9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.1e","version":"90'5","reported_seq":44,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:24:29.673749+0000","last_change":"2024-05-27T07:21:48.588574+0000","last_active":"2024-05-27T07:24:29.673749+0000","last_peered":"2024-05-27T07:24:29.673749+0000","last_clean":"2024-05-27T07:24:29.673749+0000","last_became_active":"2024-05-27T07:21:48.588212+0000","last_became_peered":"2024-05-27T07:21:48.588212+0000","last_unstale":"2024-05-27T07:24:29.673749+0000","last_undegraded":"2024-05-27T07:24:29.673749+0000","last_fullsized":"2024-05-27T07:24:29.673749+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:21:47.565641+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:21:47.565641+0000","last_clean_scrub_stamp":"2024-05-27T07:21:47.565641+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T12:15:16.017635+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":29,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:23:31.757134+0000","last_change":"2024-05-27T07:21:59.734617+0000","last_active":"2024-05-27T07:23:31.757134+0000","last_peered":"2024-05-27T07:23:31.757134+0000","last_clean":"2024-05-27T07:23:31.757134+0000","last_became_active":"2024-05-27T07:21:59.734162+0000","last_became_peered":"2024-05-27T07:21:59.734162+0000","last_unstale":"2024-05-27T07:23:31.757134+0000","last_undegraded":"2024-05-27T07:23:31.757134+0000","last_fullsized":"2024-05-27T07:23:31.757134+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:21:58.698952+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:21:58.698952+0000","last_clean_scrub_stamp":"2024-05-27T07:21:58.698952+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T12:40:01.375520+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,9,3],"acting":[4,6,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1e","version":"88'2","reported_seq":46,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:23:31.103539+0000","last_change":"2024-05-27T07:21:45.790907+0000","last_active":"2024-05-27T07:23:31.103539+0000","last_peered":"2024-05-27T07:23:31.103539+0000","last_clean":"2024-05-27T07:23:31.103539+0000","last_became_active":"2024-05-27T07:21:45.790451+0000","last_became_peered":"2024-05-27T07:21:45.790451+0000","last_unstale":"2024-05-27T07:23:31.103539+0000","last_undegraded":"2024-05-27T07:23:31.103539+0000","last_fullsized":"2024-05-27T07:23:31.103539+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:21:44.543638+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:21:44.543638+0000","last_clean_scrub_stamp":"2024-05-27T07:21:44.543638+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T10:21:56.524930+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"90'6","reported_seq":45,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:24:29.682781+0000","last_change":"2024-05-27T07:21:48.595975+0000","last_active":"2024-05-27T07:24:29.682781+0000","last_peered":"2024-05-27T07:24:29.682781+0000","last_clean":"2024-05-27T07:24:29.682781+0000","last_became_active":"2024-05-27T07:21:48.595608+0000","last_became_peered":"2024-05-27T07:21:48.595608+0000","last_unstale":"2024-05-27T07:24:29.682781+0000","last_undegraded":"2024-05-27T07:24:29.682781+0000","last_fullsized":"2024-05-27T07:24:29.682781+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:21:47.565641+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:21:47.565641+0000","last_clean_scrub_stamp":"2024-05-27T07:21:47.565641+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T07:24:44.797973+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,9],"acting":[0,1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":29,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:23:31.757236+0000","last_change":"2024-05-27T07:21:59.732788+0000","last_active":"2024-05-27T07:23:31.757236+0000","last_peered":"2024-05-27T07:23:31.757236+0000","last_clean":"2024-05-27T07:23:31.757236+0000","last_became_active":"2024-05-27T07:21:59.732310+0000","last_became_peered":"2024-05-27T07:21:59.732310+0000","last_unstale":"2024-05-27T07:23:31.757236+0000","last_undegraded":"2024-05-27T07:23:31.757236+0000","last_fullsized":"2024-05-27T07:23:31.757236+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:21:58.698952+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:21:58.698952+0000","last_clean_scrub_stamp":"2024-05-27T07:21:58.698952+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T13:54:51.490154+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,9,5],"acting":[4,3,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1d","version":"88'12","reported_seq":58,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:23:31.061913+0000","last_change":"2024-05-27T07:21:45.569746+0000","last_active":"2024-05-27T07:23:31.061913+0000","last_peered":"2024-05-27T07:23:31.061913+0000","last_clean":"2024-05-27T07:23:31.061913+0000","last_became_active":"2024-05-27T07:21:45.569366+0000","last_became_peered":"2024-05-27T07:21:45.569366+0000","last_unstale":"2024-05-27T07:23:31.061913+0000","last_undegraded":"2024-05-27T07:23:31.061913+0000","last_fullsized":"2024-05-27T07:23:31.061913+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:21:44.543638+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:21:44.543638+0000","last_clean_scrub_stamp":"2024-05-27T07:21:44.543638+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T15:58:32.977754+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":16,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":7,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1c","version":"90'9","reported_seq":48,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:24:29.686621+0000","last_change":"2024-05-27T07:21:48.606138+0000","last_active":"2024-05-27T07:24:29.686621+0000","last_peered":"2024-05-27T07:24:29.686621+0000","last_clean":"2024-05-27T07:24:29.686621+0000","last_became_active":"2024-05-27T07:21:48.605711+0000","last_became_peered":"2024-05-27T07:21:48.605711+0000","last_unstale":"2024-05-27T07:24:29.686621+0000","last_undegraded":"2024-05-27T07:24:29.686621+0000","last_fullsized":"2024-05-27T07:24:29.686621+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:21:47.565641+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:21:47.565641+0000","last_clean_scrub_stamp":"2024-05-27T07:21:47.565641+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T10:46:39.052387+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,6],"acting":[1,10,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":29,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:23:31.757067+0000","last_change":"2024-05-27T07:21:59.735475+0000","last_active":"2024-05-27T07:23:31.757067+0000","last_peered":"2024-05-27T07:23:31.757067+0000","last_clean":"2024-05-27T07:23:31.757067+0000","last_became_active":"2024-05-27T07:21:59.734986+0000","last_became_peered":"2024-05-27T07:21:59.734986+0000","last_unstale":"2024-05-27T07:23:31.757067+0000","last_undegraded":"2024-05-27T07:23:31.757067+0000","last_fullsized":"2024-05-27T07:23:31.757067+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:21:58.698952+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:21:58.698952+0000","last_clean_scrub_stamp":"2024-05-27T07:21:58.698952+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T14:48:28.198718+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5,3],"acting":[4,6,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1c","version":"88'5","reported_seq":51,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:23:31.061705+0000","last_change":"2024-05-27T07:21:45.572882+0000","last_active":"2024-05-27T07:23:31.061705+0000","last_peered":"2024-05-27T07:23:31.061705+0000","last_clean":"2024-05-27T07:23:31.061705+0000","last_became_active":"2024-05-27T07:21:45.572460+0000","last_became_peered":"2024-05-27T07:21:45.572460+0000","last_unstale":"2024-05-27T07:23:31.061705+0000","last_undegraded":"2024-05-27T07:23:31.061705+0000","last_fullsized":"2024-05-27T07:23:31.061705+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:21:44.543638+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:21:44.543638+0000","last_clean_scrub_stamp":"2024-05-27T07:21:44.543638+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T09:40:11.311923+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":7,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,1,5],"acting":[8,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1d","version":"90'15","reported_seq":54,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:24:29.676331+0000","last_change":"2024-05-27T07:21:48.596217+0000","last_active":"2024-05-27T07:24:29.676331+0000","last_peered":"2024-05-27T07:24:29.676331+0000","last_clean":"2024-05-27T07:24:29.676331+0000","last_became_active":"2024-05-27T07:21:48.595832+0000","last_became_peered":"2024-05-27T07:21:48.595832+0000","last_unstale":"2024-05-27T07:24:29.676331+0000","last_undegraded":"2024-05-27T07:24:29.676331+0000","last_fullsized":"2024-05-27T07:24:29.676331+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:21:47.565641+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:21:47.565641+0000","last_clean_scrub_stamp":"2024-05-27T07:21:47.565641+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T10:08:05.304404+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":29,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:23:30.982988+0000","last_change":"2024-05-27T07:21:59.729617+0000","last_active":"2024-05-27T07:23:30.982988+0000","last_peered":"2024-05-27T07:23:30.982988+0000","last_clean":"2024-05-27T07:23:30.982988+0000","last_became_active":"2024-05-27T07:21:59.729003+0000","last_became_peered":"2024-05-27T07:21:59.729003+0000","last_unstale":"2024-05-27T07:23:30.982988+0000","last_undegraded":"2024-05-27T07:23:30.982988+0000","last_fullsized":"2024-05-27T07:23:30.982988+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:21:58.698952+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:21:58.698952+0000","last_clean_scrub_stamp":"2024-05-27T07:21:58.698952+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T14:55:04.844948+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7,11],"acting":[2,5,7,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1b","version":"90'7","reported_seq":154,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:24:34.037902+0000","last_change":"2024-05-27T07:21:45.566355+0000","last_active":"2024-05-27T07:24:34.037902+0000","last_peered":"2024-05-27T07:24:34.037902+0000","last_clean":"2024-05-27T07:24:34.037902+0000","last_became_active":"2024-05-27T07:21:45.564776+0000","last_became_peered":"2024-05-27T07:21:45.564776+0000","last_unstale":"2024-05-27T07:24:34.037902+0000","last_undegraded":"2024-05-27T07:24:34.037902+0000","last_fullsized":"2024-05-27T07:24:34.037902+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:21:44.543638+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:21:44.543638+0000","last_clean_scrub_stamp":"2024-05-27T07:21:44.543638+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T12:10:41.632630+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1154,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":103,"num_read_kb":4,"num_write":4,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1a","version":"90'7","reported_seq":46,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:24:29.659499+0000","last_change":"2024-05-27T07:21:48.588837+0000","last_active":"2024-05-27T07:24:29.659499+0000","last_peered":"2024-05-27T07:24:29.659499+0000","last_clean":"2024-05-27T07:24:29.659499+0000","last_became_active":"2024-05-27T07:21:48.587149+0000","last_became_peered":"2024-05-27T07:21:48.587149+0000","last_unstale":"2024-05-27T07:24:29.659499+0000","last_undegraded":"2024-05-27T07:24:29.659499+0000","last_fullsized":"2024-05-27T07:24:29.659499+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:21:47.565641+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:21:47.565641+0000","last_clean_scrub_stamp":"2024-05-27T07:21:47.565641+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T09:20:05.124189+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,5],"acting":[8,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":29,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:23:30.982753+0000","last_change":"2024-05-27T07:21:59.730003+0000","last_active":"2024-05-27T07:23:30.982753+0000","last_peered":"2024-05-27T07:23:30.982753+0000","last_clean":"2024-05-27T07:23:30.982753+0000","last_became_active":"2024-05-27T07:21:59.729195+0000","last_became_peered":"2024-05-27T07:21:59.729195+0000","last_unstale":"2024-05-27T07:23:30.982753+0000","last_undegraded":"2024-05-27T07:23:30.982753+0000","last_fullsized":"2024-05-27T07:23:30.982753+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:21:58.698952+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:21:58.698952+0000","last_clean_scrub_stamp":"2024-05-27T07:21:58.698952+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T13:13:41.612300+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,9,5],"acting":[2,6,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1a","version":"88'7","reported_seq":52,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:23:31.528908+0000","last_change":"2024-05-27T07:21:45.567279+0000","last_active":"2024-05-27T07:23:31.528908+0000","last_peered":"2024-05-27T07:23:31.528908+0000","last_clean":"2024-05-27T07:23:31.528908+0000","last_became_active":"2024-05-27T07:21:45.566920+0000","last_became_peered":"2024-05-27T07:21:45.566920+0000","last_unstale":"2024-05-27T07:23:31.528908+0000","last_undegraded":"2024-05-27T07:23:31.528908+0000","last_fullsized":"2024-05-27T07:23:31.528908+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:21:44.543638+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:21:44.543638+0000","last_clean_scrub_stamp":"2024-05-27T07:21:44.543638+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T18:12:23.015535+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,8,6],"acting":[10,8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.1b","version":"90'8","reported_seq":53,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:24:29.669757+0000","last_change":"2024-05-27T07:22:30.857080+0000","last_active":"2024-05-27T07:24:29.669757+0000","last_peered":"2024-05-27T07:24:29.669757+0000","last_clean":"2024-05-27T07:24:29.669757+0000","last_became_active":"2024-05-27T07:22:30.856479+0000","last_became_peered":"2024-05-27T07:22:30.856479+0000","last_unstale":"2024-05-27T07:24:29.669757+0000","last_undegraded":"2024-05-27T07:24:29.669757+0000","last_fullsized":"2024-05-27T07:24:29.669757+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:21:47.565641+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:21:47.565641+0000","last_clean_scrub_stamp":"2024-05-27T07:21:47.565641+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T13:29:16.054448+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,8,3],"acting":[5,8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":36,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:23:31.622401+0000","last_change":"2024-05-27T07:22:30.805292+0000","last_active":"2024-05-27T07:23:31.622401+0000","last_peered":"2024-05-27T07:23:31.622401+0000","last_clean":"2024-05-27T07:23:31.622401+0000","last_became_active":"2024-05-27T07:22:30.804779+0000","last_became_peered":"2024-05-27T07:22:30.804779+0000","last_unstale":"2024-05-27T07:23:31.622401+0000","last_undegraded":"2024-05-27T07:23:31.622401+0000","last_fullsized":"2024-05-27T07:23:31.622401+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:21:58.698952+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:21:58.698952+0000","last_clean_scrub_stamp":"2024-05-27T07:21:58.698952+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T11:12:49.095931+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,8,0],"acting":[11,4,8,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.19","version":"88'3","reported_seq":47,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:23:30.973995+0000","last_change":"2024-05-27T07:21:45.561255+0000","last_active":"2024-05-27T07:23:30.973995+0000","last_peered":"2024-05-27T07:23:30.973995+0000","last_clean":"2024-05-27T07:23:30.973995+0000","last_became_active":"2024-05-27T07:21:45.560896+0000","last_became_peered":"2024-05-27T07:21:45.560896+0000","last_unstale":"2024-05-27T07:23:30.973995+0000","last_undegraded":"2024-05-27T07:23:30.973995+0000","last_fullsized":"2024-05-27T07:23:30.973995+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:21:44.543638+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:21:44.543638+0000","last_clean_scrub_stamp":"2024-05-27T07:21:44.543638+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T17:05:24.074628+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,9],"acting":[6,4,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"90'8","reported_seq":47,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:24:29.679127+0000","last_change":"2024-05-27T07:21:48.586045+0000","last_active":"2024-05-27T07:24:29.679127+0000","last_peered":"2024-05-27T07:24:29.679127+0000","last_clean":"2024-05-27T07:24:29.679127+0000","last_became_active":"2024-05-27T07:21:48.585681+0000","last_became_peered":"2024-05-27T07:21:48.585681+0000","last_unstale":"2024-05-27T07:24:29.679127+0000","last_undegraded":"2024-05-27T07:24:29.679127+0000","last_fullsized":"2024-05-27T07:24:29.679127+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:21:47.565641+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:21:47.565641+0000","last_clean_scrub_stamp":"2024-05-27T07:21:47.565641+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T09:56:36.408623+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,6],"acting":[3,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":29,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:23:30.981574+0000","last_change":"2024-05-27T07:21:59.739544+0000","last_active":"2024-05-27T07:23:30.981574+0000","last_peered":"2024-05-27T07:23:30.981574+0000","last_clean":"2024-05-27T07:23:30.981574+0000","last_became_active":"2024-05-27T07:21:59.739132+0000","last_became_peered":"2024-05-27T07:21:59.739132+0000","last_unstale":"2024-05-27T07:23:30.981574+0000","last_undegraded":"2024-05-27T07:23:30.981574+0000","last_fullsized":"2024-05-27T07:23:30.981574+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:21:58.698952+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:21:58.698952+0000","last_clean_scrub_stamp":"2024-05-27T07:21:58.698952+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T10:06:05.126579+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,6,1],"acting":[0,7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.18","version":"88'3","reported_seq":48,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:23:30.977006+0000","last_change":"2024-05-27T07:21:45.568171+0000","last_active":"2024-05-27T07:23:30.977006+0000","last_peered":"2024-05-27T07:23:30.977006+0000","last_clean":"2024-05-27T07:23:30.977006+0000","last_became_active":"2024-05-27T07:21:45.567627+0000","last_became_peered":"2024-05-27T07:21:45.567627+0000","last_unstale":"2024-05-27T07:23:30.977006+0000","last_undegraded":"2024-05-27T07:23:30.977006+0000","last_fullsized":"2024-05-27T07:23:30.977006+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:21:44.543638+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:21:44.543638+0000","last_clean_scrub_stamp":"2024-05-27T07:21:44.543638+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T16:43:56.407079+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"90'11","reported_seq":50,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:24:29.679091+0000","last_change":"2024-05-27T07:21:48.587452+0000","last_active":"2024-05-27T07:24:29.679091+0000","last_peered":"2024-05-27T07:24:29.679091+0000","last_clean":"2024-05-27T07:24:29.679091+0000","last_became_active":"2024-05-27T07:21:48.587027+0000","last_became_peered":"2024-05-27T07:21:48.587027+0000","last_unstale":"2024-05-27T07:24:29.679091+0000","last_undegraded":"2024-05-27T07:24:29.679091+0000","last_fullsized":"2024-05-27T07:24:29.679091+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:21:47.565641+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:21:47.565641+0000","last_clean_scrub_stamp":"2024-05-27T07:21:47.565641+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T13:51:44.890061+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,10],"acting":[9,3,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":29,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:23:31.102968+0000","last_change":"2024-05-27T07:21:59.738292+0000","last_active":"2024-05-27T07:23:31.102968+0000","last_peered":"2024-05-27T07:23:31.102968+0000","last_clean":"2024-05-27T07:23:31.102968+0000","last_became_active":"2024-05-27T07:21:59.737941+0000","last_became_peered":"2024-05-27T07:21:59.737941+0000","last_unstale":"2024-05-27T07:23:31.102968+0000","last_undegraded":"2024-05-27T07:23:31.102968+0000","last_fullsized":"2024-05-27T07:23:31.102968+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:21:58.698952+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:21:58.698952+0000","last_clean_scrub_stamp":"2024-05-27T07:21:58.698952+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T13:54:24.192917+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1,5],"acting":[3,7,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.17","version":"90'5","reported_seq":246,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:24:34.037153+0000","last_change":"2024-05-27T07:21:45.568338+0000","last_active":"2024-05-27T07:24:34.037153+0000","last_peered":"2024-05-27T07:24:34.037153+0000","last_clean":"2024-05-27T07:24:34.037153+0000","last_became_active":"2024-05-27T07:21:45.567856+0000","last_became_peered":"2024-05-27T07:21:45.567856+0000","last_unstale":"2024-05-27T07:24:34.037153+0000","last_undegraded":"2024-05-27T07:24:34.037153+0000","last_fullsized":"2024-05-27T07:24:34.037153+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:21:44.543638+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:21:44.543638+0000","last_clean_scrub_stamp":"2024-05-27T07:21:44.543638+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T15:02:04.188711+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":180,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":196,"num_read_kb":196,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,5,9],"acting":[10,5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.16","version":"90'10","reported_seq":49,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:24:29.684905+0000","last_change":"2024-05-27T07:21:48.606062+0000","last_active":"2024-05-27T07:24:29.684905+0000","last_peered":"2024-05-27T07:24:29.684905+0000","last_clean":"2024-05-27T07:24:29.684905+0000","last_became_active":"2024-05-27T07:21:48.605423+0000","last_became_peered":"2024-05-27T07:21:48.605423+0000","last_unstale":"2024-05-27T07:24:29.684905+0000","last_undegraded":"2024-05-27T07:24:29.684905+0000","last_fullsized":"2024-05-27T07:24:29.684905+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:21:47.565641+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:21:47.565641+0000","last_clean_scrub_stamp":"2024-05-27T07:21:47.565641+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T14:39:06.921519+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,9],"acting":[1,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":29,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:23:31.652581+0000","last_change":"2024-05-27T07:21:59.735150+0000","last_active":"2024-05-27T07:23:31.652581+0000","last_peered":"2024-05-27T07:23:31.652581+0000","last_clean":"2024-05-27T07:23:31.652581+0000","last_became_active":"2024-05-27T07:21:59.734688+0000","last_became_peered":"2024-05-27T07:21:59.734688+0000","last_unstale":"2024-05-27T07:23:31.652581+0000","last_undegraded":"2024-05-27T07:23:31.652581+0000","last_fullsized":"2024-05-27T07:23:31.652581+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:21:58.698952+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:21:58.698952+0000","last_clean_scrub_stamp":"2024-05-27T07:21:58.698952+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T15:38:02.095947+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,0,2,4],"acting":[9,0,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.16","version":"88'4","reported_seq":46,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:23:30.975440+0000","last_change":"2024-05-27T07:21:45.559550+0000","last_active":"2024-05-27T07:23:30.975440+0000","last_peered":"2024-05-27T07:23:30.975440+0000","last_clean":"2024-05-27T07:23:30.975440+0000","last_became_active":"2024-05-27T07:21:45.559181+0000","last_became_peered":"2024-05-27T07:21:45.559181+0000","last_unstale":"2024-05-27T07:23:30.975440+0000","last_undegraded":"2024-05-27T07:23:30.975440+0000","last_fullsized":"2024-05-27T07:23:30.975440+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:21:44.543638+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:21:44.543638+0000","last_clean_scrub_stamp":"2024-05-27T07:21:44.543638+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T10:33:45.288437+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.17","version":"90'9","reported_seq":48,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:24:29.681347+0000","last_change":"2024-05-27T07:21:48.593631+0000","last_active":"2024-05-27T07:24:29.681347+0000","last_peered":"2024-05-27T07:24:29.681347+0000","last_clean":"2024-05-27T07:24:29.681347+0000","last_became_active":"2024-05-27T07:21:48.592330+0000","last_became_peered":"2024-05-27T07:21:48.592330+0000","last_unstale":"2024-05-27T07:24:29.681347+0000","last_undegraded":"2024-05-27T07:24:29.681347+0000","last_fullsized":"2024-05-27T07:24:29.681347+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:21:47.565641+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:21:47.565641+0000","last_clean_scrub_stamp":"2024-05-27T07:21:47.565641+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T19:19:51.593307+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,5],"acting":[0,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":29,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:23:31.757433+0000","last_change":"2024-05-27T07:21:59.725158+0000","last_active":"2024-05-27T07:23:31.757433+0000","last_peered":"2024-05-27T07:23:31.757433+0000","last_clean":"2024-05-27T07:23:31.757433+0000","last_became_active":"2024-05-27T07:21:59.724651+0000","last_became_peered":"2024-05-27T07:21:59.724651+0000","last_unstale":"2024-05-27T07:23:31.757433+0000","last_undegraded":"2024-05-27T07:23:31.757433+0000","last_fullsized":"2024-05-27T07:23:31.757433+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:21:58.698952+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:21:58.698952+0000","last_clean_scrub_stamp":"2024-05-27T07:21:58.698952+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T12:10:45.747484+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2,9],"acting":[4,1,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.15","version":"88'5","reported_seq":50,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:23:30.977408+0000","last_change":"2024-05-27T07:21:45.791831+0000","last_active":"2024-05-27T07:23:30.977408+0000","last_peered":"2024-05-27T07:23:30.977408+0000","last_clean":"2024-05-27T07:23:30.977408+0000","last_became_active":"2024-05-27T07:21:45.791271+0000","last_became_peered":"2024-05-27T07:21:45.791271+0000","last_unstale":"2024-05-27T07:23:30.977408+0000","last_undegraded":"2024-05-27T07:23:30.977408+0000","last_fullsized":"2024-05-27T07:23:30.977408+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:21:44.543638+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:21:44.543638+0000","last_clean_scrub_stamp":"2024-05-27T07:21:44.543638+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T16:20:37.492337+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":7,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,10],"acting":[5,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.14","version":"90'11","reported_seq":50,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:24:29.687145+0000","last_change":"2024-05-27T07:21:48.605605+0000","last_active":"2024-05-27T07:24:29.687145+0000","last_peered":"2024-05-27T07:24:29.687145+0000","last_clean":"2024-05-27T07:24:29.687145+0000","last_became_active":"2024-05-27T07:21:48.604655+0000","last_became_peered":"2024-05-27T07:21:48.604655+0000","last_unstale":"2024-05-27T07:24:29.687145+0000","last_undegraded":"2024-05-27T07:24:29.687145+0000","last_fullsized":"2024-05-27T07:24:29.687145+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:21:47.565641+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:21:47.565641+0000","last_clean_scrub_stamp":"2024-05-27T07:21:47.565641+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T14:33:53.661983+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,8,4],"acting":[1,8,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":36,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:23:31.652808+0000","last_change":"2024-05-27T07:22:31.418393+0000","last_active":"2024-05-27T07:23:31.652808+0000","last_peered":"2024-05-27T07:23:31.652808+0000","last_clean":"2024-05-27T07:23:31.652808+0000","last_became_active":"2024-05-27T07:22:31.418006+0000","last_became_peered":"2024-05-27T07:22:31.418006+0000","last_unstale":"2024-05-27T07:23:31.652808+0000","last_undegraded":"2024-05-27T07:23:31.652808+0000","last_fullsized":"2024-05-27T07:23:31.652808+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:21:58.698952+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:21:58.698952+0000","last_clean_scrub_stamp":"2024-05-27T07:21:58.698952+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T14:30:55.130461+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,11,10],"acting":[9,8,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.14","version":"90'40","reported_seq":292,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:24:34.051406+0000","last_change":"2024-05-27T07:21:45.564829+0000","last_active":"2024-05-27T07:24:34.051406+0000","last_peered":"2024-05-27T07:24:34.051406+0000","last_clean":"2024-05-27T07:24:34.051406+0000","last_became_active":"2024-05-27T07:21:45.564457+0000","last_became_peered":"2024-05-27T07:21:45.564457+0000","last_unstale":"2024-05-27T07:24:34.051406+0000","last_undegraded":"2024-05-27T07:24:34.051406+0000","last_fullsized":"2024-05-27T07:24:34.051406+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:21:44.543638+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:21:44.543638+0000","last_clean_scrub_stamp":"2024-05-27T07:21:44.543638+0000","objects_scrubbed":0,"log_size":40,"log_dups_size":0,"ondisk_log_size":40,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T19:12:44.546989+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":64867,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":207,"num_read_kb":167,"num_write":42,"num_write_kb":83,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,1,2],"acting":[10,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.15","version":"90'6","reported_seq":45,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:24:29.669019+0000","last_change":"2024-05-27T07:21:48.591672+0000","last_active":"2024-05-27T07:24:29.669019+0000","last_peered":"2024-05-27T07:24:29.669019+0000","last_clean":"2024-05-27T07:24:29.669019+0000","last_became_active":"2024-05-27T07:21:48.590861+0000","last_became_peered":"2024-05-27T07:21:48.590861+0000","last_unstale":"2024-05-27T07:24:29.669019+0000","last_undegraded":"2024-05-27T07:24:29.669019+0000","last_fullsized":"2024-05-27T07:24:29.669019+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:21:47.565641+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:21:47.565641+0000","last_clean_scrub_stamp":"2024-05-27T07:21:47.565641+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T12:03:40.627764+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,7],"acting":[8,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":29,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:23:30.977741+0000","last_change":"2024-05-27T07:21:59.736446+0000","last_active":"2024-05-27T07:23:30.977741+0000","last_peered":"2024-05-27T07:23:30.977741+0000","last_clean":"2024-05-27T07:23:30.977741+0000","last_became_active":"2024-05-27T07:21:59.735854+0000","last_became_peered":"2024-05-27T07:21:59.735854+0000","last_unstale":"2024-05-27T07:23:30.977741+0000","last_undegraded":"2024-05-27T07:23:30.977741+0000","last_fullsized":"2024-05-27T07:23:30.977741+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:21:58.698952+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:21:58.698952+0000","last_clean_scrub_stamp":"2024-05-27T07:21:58.698952+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T08:02:14.640325+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,8,3,6],"acting":[5,8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.13","version":"88'9","reported_seq":55,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:23:30.982597+0000","last_change":"2024-05-27T07:21:45.562930+0000","last_active":"2024-05-27T07:23:30.982597+0000","last_peered":"2024-05-27T07:23:30.982597+0000","last_clean":"2024-05-27T07:23:30.982597+0000","last_became_active":"2024-05-27T07:21:45.562515+0000","last_became_peered":"2024-05-27T07:21:45.562515+0000","last_unstale":"2024-05-27T07:23:30.982597+0000","last_undegraded":"2024-05-27T07:23:30.982597+0000","last_fullsized":"2024-05-27T07:23:30.982597+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:21:44.543638+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:21:44.543638+0000","last_clean_scrub_stamp":"2024-05-27T07:21:44.543638+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T17:54:24.006219+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":68,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":13,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,10,3],"acting":[2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.12","version":"90'9","reported_seq":48,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:24:29.676179+0000","last_change":"2024-05-27T07:21:48.591090+0000","last_active":"2024-05-27T07:24:29.676179+0000","last_peered":"2024-05-27T07:24:29.676179+0000","last_clean":"2024-05-27T07:24:29.676179+0000","last_became_active":"2024-05-27T07:21:48.590726+0000","last_became_peered":"2024-05-27T07:21:48.590726+0000","last_unstale":"2024-05-27T07:24:29.676179+0000","last_undegraded":"2024-05-27T07:24:29.676179+0000","last_fullsized":"2024-05-27T07:24:29.676179+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:21:47.565641+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:21:47.565641+0000","last_clean_scrub_stamp":"2024-05-27T07:21:47.565641+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T18:40:03.880407+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,3],"acting":[6,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.14","version":"0'0","reported_seq":29,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:23:31.621855+0000","last_change":"2024-05-27T07:21:59.738375+0000","last_active":"2024-05-27T07:23:31.621855+0000","last_peered":"2024-05-27T07:23:31.621855+0000","last_clean":"2024-05-27T07:23:31.621855+0000","last_became_active":"2024-05-27T07:21:59.737957+0000","last_became_peered":"2024-05-27T07:21:59.737957+0000","last_unstale":"2024-05-27T07:23:31.621855+0000","last_undegraded":"2024-05-27T07:23:31.621855+0000","last_fullsized":"2024-05-27T07:23:31.621855+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:21:58.698952+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:21:58.698952+0000","last_clean_scrub_stamp":"2024-05-27T07:21:58.698952+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T16:53:51.397143+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,9,7,6],"acting":[11,9,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.12","version":"88'5","reported_seq":50,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:23:30.977272+0000","last_change":"2024-05-27T07:21:45.566986+0000","last_active":"2024-05-27T07:23:30.977272+0000","last_peered":"2024-05-27T07:23:30.977272+0000","last_clean":"2024-05-27T07:23:30.977272+0000","last_became_active":"2024-05-27T07:21:45.566579+0000","last_became_peered":"2024-05-27T07:21:45.566579+0000","last_unstale":"2024-05-27T07:23:30.977272+0000","last_undegraded":"2024-05-27T07:23:30.977272+0000","last_fullsized":"2024-05-27T07:23:30.977272+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:21:44.543638+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:21:44.543638+0000","last_clean_scrub_stamp":"2024-05-27T07:21:44.543638+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T17:08:14.034740+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":5,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"90'15","reported_seq":52,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:24:29.677606+0000","last_change":"2024-05-27T07:21:48.596575+0000","last_active":"2024-05-27T07:24:29.677606+0000","last_peered":"2024-05-27T07:24:29.677606+0000","last_clean":"2024-05-27T07:24:29.677606+0000","last_became_active":"2024-05-27T07:21:48.596215+0000","last_became_peered":"2024-05-27T07:21:48.596215+0000","last_unstale":"2024-05-27T07:24:29.677606+0000","last_undegraded":"2024-05-27T07:24:29.677606+0000","last_fullsized":"2024-05-27T07:24:29.677606+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:21:47.565641+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:21:47.565641+0000","last_clean_scrub_stamp":"2024-05-27T07:21:47.565641+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T14:57:05.795269+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":29,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:23:31.653172+0000","last_change":"2024-05-27T07:21:59.737690+0000","last_active":"2024-05-27T07:23:31.653172+0000","last_peered":"2024-05-27T07:23:31.653172+0000","last_clean":"2024-05-27T07:23:31.653172+0000","last_became_active":"2024-05-27T07:21:59.735646+0000","last_became_peered":"2024-05-27T07:21:59.735646+0000","last_unstale":"2024-05-27T07:23:31.653172+0000","last_undegraded":"2024-05-27T07:23:31.653172+0000","last_fullsized":"2024-05-27T07:23:31.653172+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:21:58.698952+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:21:58.698952+0000","last_clean_scrub_stamp":"2024-05-27T07:21:58.698952+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T13:54:41.227626+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,0,3],"acting":[9,7,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.11","version":"90'7","reported_seq":155,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:24:34.037845+0000","last_change":"2024-05-27T07:21:45.564836+0000","last_active":"2024-05-27T07:24:34.037845+0000","last_peered":"2024-05-27T07:24:34.037845+0000","last_clean":"2024-05-27T07:24:34.037845+0000","last_became_active":"2024-05-27T07:21:45.564481+0000","last_became_peered":"2024-05-27T07:21:45.564481+0000","last_unstale":"2024-05-27T07:24:34.037845+0000","last_undegraded":"2024-05-27T07:24:34.037845+0000","last_fullsized":"2024-05-27T07:24:34.037845+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:21:44.543638+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:21:44.543638+0000","last_clean_scrub_stamp":"2024-05-27T07:21:44.543638+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T15:13:12.012769+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1120,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":104,"num_read_kb":5,"num_write":7,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,10],"acting":[4,7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.10","version":"90'6","reported_seq":43,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:24:29.672831+0000","last_change":"2024-05-27T07:21:48.601331+0000","last_active":"2024-05-27T07:24:29.672831+0000","last_peered":"2024-05-27T07:24:29.672831+0000","last_clean":"2024-05-27T07:24:29.672831+0000","last_became_active":"2024-05-27T07:21:48.600917+0000","last_became_peered":"2024-05-27T07:21:48.600917+0000","last_unstale":"2024-05-27T07:24:29.672831+0000","last_undegraded":"2024-05-27T07:24:29.672831+0000","last_fullsized":"2024-05-27T07:24:29.672831+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:21:47.565641+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:21:47.565641+0000","last_clean_scrub_stamp":"2024-05-27T07:21:47.565641+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T17:26:42.129989+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":29,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:23:30.974225+0000","last_change":"2024-05-27T07:21:59.737662+0000","last_active":"2024-05-27T07:23:30.974225+0000","last_peered":"2024-05-27T07:23:30.974225+0000","last_clean":"2024-05-27T07:23:30.974225+0000","last_became_active":"2024-05-27T07:21:59.737229+0000","last_became_peered":"2024-05-27T07:21:59.737229+0000","last_unstale":"2024-05-27T07:23:30.974225+0000","last_undegraded":"2024-05-27T07:23:30.974225+0000","last_fullsized":"2024-05-27T07:23:30.974225+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:21:58.698952+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:21:58.698952+0000","last_clean_scrub_stamp":"2024-05-27T07:21:58.698952+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T15:32:31.147571+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,9,11],"acting":[6,3,9,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.10","version":"88'17","reported_seq":79,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:23:31.061322+0000","last_change":"2024-05-27T07:22:30.814815+0000","last_active":"2024-05-27T07:23:31.061322+0000","last_peered":"2024-05-27T07:23:31.061322+0000","last_clean":"2024-05-27T07:23:31.061322+0000","last_became_active":"2024-05-27T07:22:30.802828+0000","last_became_peered":"2024-05-27T07:22:30.802828+0000","last_unstale":"2024-05-27T07:23:31.061322+0000","last_undegraded":"2024-05-27T07:23:31.061322+0000","last_fullsized":"2024-05-27T07:23:31.061322+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:21:44.543638+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:21:44.543638+0000","last_clean_scrub_stamp":"2024-05-27T07:21:44.543638+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T11:38:07.363507+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,4],"acting":[8,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.11","version":"90'6","reported_seq":45,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:24:29.660613+0000","last_change":"2024-05-27T07:21:48.596204+0000","last_active":"2024-05-27T07:24:29.660613+0000","last_peered":"2024-05-27T07:24:29.660613+0000","last_clean":"2024-05-27T07:24:29.660613+0000","last_became_active":"2024-05-27T07:21:48.595827+0000","last_became_peered":"2024-05-27T07:21:48.595827+0000","last_unstale":"2024-05-27T07:24:29.660613+0000","last_undegraded":"2024-05-27T07:24:29.660613+0000","last_fullsized":"2024-05-27T07:24:29.660613+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:21:47.565641+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:21:47.565641+0000","last_clean_scrub_stamp":"2024-05-27T07:21:47.565641+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T15:51:53.162554+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":29,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:23:30.977011+0000","last_change":"2024-05-27T07:21:59.735837+0000","last_active":"2024-05-27T07:23:30.977011+0000","last_peered":"2024-05-27T07:23:30.977011+0000","last_clean":"2024-05-27T07:23:30.977011+0000","last_became_active":"2024-05-27T07:21:59.735063+0000","last_became_peered":"2024-05-27T07:21:59.735063+0000","last_unstale":"2024-05-27T07:23:30.977011+0000","last_undegraded":"2024-05-27T07:23:30.977011+0000","last_fullsized":"2024-05-27T07:23:30.977011+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:21:58.698952+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:21:58.698952+0000","last_clean_scrub_stamp":"2024-05-27T07:21:58.698952+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T13:04:21.751043+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,3,1],"acting":[5,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.f","version":"88'2","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:23:31.622220+0000","last_change":"2024-05-27T07:22:30.809491+0000","last_active":"2024-05-27T07:23:31.622220+0000","last_peered":"2024-05-27T07:23:31.622220+0000","last_clean":"2024-05-27T07:23:31.622220+0000","last_became_active":"2024-05-27T07:22:30.802187+0000","last_became_peered":"2024-05-27T07:22:30.802187+0000","last_unstale":"2024-05-27T07:23:31.622220+0000","last_undegraded":"2024-05-27T07:23:31.622220+0000","last_fullsized":"2024-05-27T07:23:31.622220+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:21:44.543638+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:21:44.543638+0000","last_clean_scrub_stamp":"2024-05-27T07:21:44.543638+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T18:11:32.821962+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":22,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,4],"acting":[11,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.e","version":"90'5","reported_seq":44,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:24:29.671137+0000","last_change":"2024-05-27T07:21:48.597362+0000","last_active":"2024-05-27T07:24:29.671137+0000","last_peered":"2024-05-27T07:24:29.671137+0000","last_clean":"2024-05-27T07:24:29.671137+0000","last_became_active":"2024-05-27T07:21:48.596920+0000","last_became_peered":"2024-05-27T07:21:48.596920+0000","last_unstale":"2024-05-27T07:24:29.671137+0000","last_undegraded":"2024-05-27T07:24:29.671137+0000","last_fullsized":"2024-05-27T07:24:29.671137+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:21:47.565641+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:21:47.565641+0000","last_clean_scrub_stamp":"2024-05-27T07:21:47.565641+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T16:50:21.021320+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,4],"acting":[11,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":29,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:23:31.061286+0000","last_change":"2024-05-27T07:21:59.732303+0000","last_active":"2024-05-27T07:23:31.061286+0000","last_peered":"2024-05-27T07:23:31.061286+0000","last_clean":"2024-05-27T07:23:31.061286+0000","last_became_active":"2024-05-27T07:21:59.731851+0000","last_became_peered":"2024-05-27T07:21:59.731851+0000","last_unstale":"2024-05-27T07:23:31.061286+0000","last_undegraded":"2024-05-27T07:23:31.061286+0000","last_fullsized":"2024-05-27T07:23:31.061286+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:21:58.698952+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:21:58.698952+0000","last_clean_scrub_stamp":"2024-05-27T07:21:58.698952+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T08:08:38.433818+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,10,11,9],"acting":[8,10,11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.4","version":"88'1","reported_seq":46,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:23:31.653172+0000","last_change":"2024-05-27T07:21:45.791370+0000","last_active":"2024-05-27T07:23:31.653172+0000","last_peered":"2024-05-27T07:23:31.653172+0000","last_clean":"2024-05-27T07:23:31.653172+0000","last_became_active":"2024-05-27T07:21:45.790987+0000","last_became_peered":"2024-05-27T07:21:45.790987+0000","last_unstale":"2024-05-27T07:23:31.653172+0000","last_undegraded":"2024-05-27T07:23:31.653172+0000","last_fullsized":"2024-05-27T07:23:31.653172+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:21:44.543638+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:21:44.543638+0000","last_clean_scrub_stamp":"2024-05-27T07:21:44.543638+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T14:20:18.570194+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2,0],"acting":[9,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.5","version":"90'4","reported_seq":43,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:24:29.671184+0000","last_change":"2024-05-27T07:21:48.585984+0000","last_active":"2024-05-27T07:24:29.671184+0000","last_peered":"2024-05-27T07:24:29.671184+0000","last_clean":"2024-05-27T07:24:29.671184+0000","last_became_active":"2024-05-27T07:21:48.585606+0000","last_became_peered":"2024-05-27T07:21:48.585606+0000","last_unstale":"2024-05-27T07:24:29.671184+0000","last_undegraded":"2024-05-27T07:24:29.671184+0000","last_fullsized":"2024-05-27T07:24:29.671184+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:21:47.565641+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:21:47.565641+0000","last_clean_scrub_stamp":"2024-05-27T07:21:47.565641+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T17:56:17.659024+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,3,6],"acting":[10,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":29,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:23:30.973955+0000","last_change":"2024-05-27T07:21:59.734022+0000","last_active":"2024-05-27T07:23:30.973955+0000","last_peered":"2024-05-27T07:23:30.973955+0000","last_clean":"2024-05-27T07:23:30.973955+0000","last_became_active":"2024-05-27T07:21:59.733467+0000","last_became_peered":"2024-05-27T07:21:59.733467+0000","last_unstale":"2024-05-27T07:23:30.973955+0000","last_undegraded":"2024-05-27T07:23:30.973955+0000","last_fullsized":"2024-05-27T07:23:30.973955+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:21:58.698952+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:21:58.698952+0000","last_clean_scrub_stamp":"2024-05-27T07:21:58.698952+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T09:54:11.058495+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,1,8],"acting":[6,0,1,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.2","version":"88'5","reported_seq":143,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:24:30.040121+0000","last_change":"2024-05-27T07:21:45.563072+0000","last_active":"2024-05-27T07:24:30.040121+0000","last_peered":"2024-05-27T07:24:30.040121+0000","last_clean":"2024-05-27T07:24:30.040121+0000","last_became_active":"2024-05-27T07:21:45.562632+0000","last_became_peered":"2024-05-27T07:21:45.562632+0000","last_unstale":"2024-05-27T07:24:30.040121+0000","last_undegraded":"2024-05-27T07:24:30.040121+0000","last_fullsized":"2024-05-27T07:24:30.040121+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:21:44.543638+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:21:44.543638+0000","last_clean_scrub_stamp":"2024-05-27T07:21:44.543638+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T12:54:19.555467+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":588,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":96,"num_read_kb":1,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3","version":"90'7","reported_seq":46,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:24:29.672679+0000","last_change":"2024-05-27T07:21:48.594867+0000","last_active":"2024-05-27T07:24:29.672679+0000","last_peered":"2024-05-27T07:24:29.672679+0000","last_clean":"2024-05-27T07:24:29.672679+0000","last_became_active":"2024-05-27T07:21:48.594523+0000","last_became_peered":"2024-05-27T07:21:48.594523+0000","last_unstale":"2024-05-27T07:24:29.672679+0000","last_undegraded":"2024-05-27T07:24:29.672679+0000","last_fullsized":"2024-05-27T07:24:29.672679+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:21:47.565641+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:21:47.565641+0000","last_clean_scrub_stamp":"2024-05-27T07:21:47.565641+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T17:28:55.862014+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,4,0],"acting":[10,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":29,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:23:30.974221+0000","last_change":"2024-05-27T07:21:59.741936+0000","last_active":"2024-05-27T07:23:30.974221+0000","last_peered":"2024-05-27T07:23:30.974221+0000","last_clean":"2024-05-27T07:23:30.974221+0000","last_became_active":"2024-05-27T07:21:59.741458+0000","last_became_peered":"2024-05-27T07:21:59.741458+0000","last_unstale":"2024-05-27T07:23:30.974221+0000","last_undegraded":"2024-05-27T07:23:30.974221+0000","last_fullsized":"2024-05-27T07:23:30.974221+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:21:58.698952+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:21:58.698952+0000","last_clean_scrub_stamp":"2024-05-27T07:21:58.698952+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T15:51:28.560995+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,10,0],"acting":[6,5,10,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.0","version":"90'8","reported_seq":24,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:24:29.671317+0000","last_change":"2024-05-27T07:22:30.803379+0000","last_active":"2024-05-27T07:24:29.671317+0000","last_peered":"2024-05-27T07:24:29.671317+0000","last_clean":"2024-05-27T07:24:29.671317+0000","last_became_active":"2024-05-27T07:22:30.803024+0000","last_became_peered":"2024-05-27T07:22:30.803024+0000","last_unstale":"2024-05-27T07:24:29.671317+0000","last_undegraded":"2024-05-27T07:24:29.671317+0000","last_fullsized":"2024-05-27T07:24:29.671317+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:21:47.565641+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:21:47.565641+0000","last_clean_scrub_stamp":"2024-05-27T07:21:47.565641+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T07:56:32.070610+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2,4],"acting":[9,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.1","version":"90'6","reported_seq":145,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:24:29.624436+0000","last_change":"2024-05-27T07:21:45.790849+0000","last_active":"2024-05-27T07:24:29.624436+0000","last_peered":"2024-05-27T07:24:29.624436+0000","last_clean":"2024-05-27T07:24:29.624436+0000","last_became_active":"2024-05-27T07:21:45.790435+0000","last_became_peered":"2024-05-27T07:21:45.790435+0000","last_unstale":"2024-05-27T07:24:29.624436+0000","last_undegraded":"2024-05-27T07:24:29.624436+0000","last_fullsized":"2024-05-27T07:24:29.624436+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:21:44.543638+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:21:44.543638+0000","last_clean_scrub_stamp":"2024-05-27T07:21:44.543638+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T09:34:51.479882+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":95,"num_read_kb":95,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,0],"acting":[6,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.5","version":"88'1","reported_seq":29,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:23:30.975407+0000","last_change":"2024-05-27T07:21:59.731338+0000","last_active":"2024-05-27T07:23:30.975407+0000","last_peered":"2024-05-27T07:23:30.975407+0000","last_clean":"2024-05-27T07:23:30.975407+0000","last_became_active":"2024-05-27T07:21:59.730847+0000","last_became_peered":"2024-05-27T07:21:59.730847+0000","last_unstale":"2024-05-27T07:23:30.975407+0000","last_undegraded":"2024-05-27T07:23:30.975407+0000","last_fullsized":"2024-05-27T07:23:30.975407+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:21:58.698952+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:21:58.698952+0000","last_clean_scrub_stamp":"2024-05-27T07:21:58.698952+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T18:18:50.901129+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,11,9],"acting":[7,5,11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2","version":"90'8","reported_seq":47,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:24:29.673758+0000","last_change":"2024-05-27T07:21:48.594274+0000","last_active":"2024-05-27T07:24:29.673758+0000","last_peered":"2024-05-27T07:24:29.673758+0000","last_clean":"2024-05-27T07:24:29.673758+0000","last_became_active":"2024-05-27T07:21:48.593940+0000","last_became_peered":"2024-05-27T07:21:48.593940+0000","last_unstale":"2024-05-27T07:24:29.673758+0000","last_undegraded":"2024-05-27T07:24:29.673758+0000","last_fullsized":"2024-05-27T07:24:29.673758+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:21:47.565641+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:21:47.565641+0000","last_clean_scrub_stamp":"2024-05-27T07:21:47.565641+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T13:25:46.531524+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"88'6","reported_seq":54,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:23:31.757237+0000","last_change":"2024-05-27T07:21:45.563365+0000","last_active":"2024-05-27T07:23:31.757237+0000","last_peered":"2024-05-27T07:23:31.757237+0000","last_clean":"2024-05-27T07:23:31.757237+0000","last_became_active":"2024-05-27T07:21:45.563012+0000","last_became_peered":"2024-05-27T07:21:45.563012+0000","last_unstale":"2024-05-27T07:23:31.757237+0000","last_undegraded":"2024-05-27T07:23:31.757237+0000","last_fullsized":"2024-05-27T07:23:31.757237+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:21:44.543638+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:21:44.543638+0000","last_clean_scrub_stamp":"2024-05-27T07:21:44.543638+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T12:58:51.079965+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":68,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":4,"num_read_kb":4,"num_write":8,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,9],"acting":[4,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.0","version":"20'92","reported_seq":116,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:23:30.974809+0000","last_change":"2024-05-27T07:20:21.611212+0000","last_active":"2024-05-27T07:23:30.974809+0000","last_peered":"2024-05-27T07:23:30.974809+0000","last_clean":"2024-05-27T07:23:30.974809+0000","last_became_active":"2024-05-27T07:20:21.605163+0000","last_became_peered":"2024-05-27T07:20:21.605163+0000","last_unstale":"2024-05-27T07:23:30.974809+0000","last_undegraded":"2024-05-27T07:23:30.974809+0000","last_fullsized":"2024-05-27T07:23:30.974809+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:16:57.668433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:16:57.668433+0000","last_clean_scrub_stamp":"2024-05-27T07:16:57.668433+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T12:10:59.074162+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":81,"num_read_kb":69,"num_write":123,"num_write_kb":2150,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":2951840,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":29,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:23:30.977689+0000","last_change":"2024-05-27T07:21:59.737210+0000","last_active":"2024-05-27T07:23:30.977689+0000","last_peered":"2024-05-27T07:23:30.977689+0000","last_clean":"2024-05-27T07:23:30.977689+0000","last_became_active":"2024-05-27T07:21:59.736799+0000","last_became_peered":"2024-05-27T07:21:59.736799+0000","last_unstale":"2024-05-27T07:23:30.977689+0000","last_undegraded":"2024-05-27T07:23:30.977689+0000","last_fullsized":"2024-05-27T07:23:30.977689+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:21:58.698952+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:21:58.698952+0000","last_clean_scrub_stamp":"2024-05-27T07:21:58.698952+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T13:03:17.153687+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,6,10],"acting":[5,9,6,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1","version":"90'5","reported_seq":44,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:24:29.658895+0000","last_change":"2024-05-27T07:21:48.591550+0000","last_active":"2024-05-27T07:24:29.658895+0000","last_peered":"2024-05-27T07:24:29.658895+0000","last_clean":"2024-05-27T07:24:29.658895+0000","last_became_active":"2024-05-27T07:21:48.591223+0000","last_became_peered":"2024-05-27T07:21:48.591223+0000","last_unstale":"2024-05-27T07:24:29.658895+0000","last_undegraded":"2024-05-27T07:24:29.658895+0000","last_fullsized":"2024-05-27T07:24:29.658895+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:21:47.565641+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:21:47.565641+0000","last_clean_scrub_stamp":"2024-05-27T07:21:47.565641+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T15:06:23.395645+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,2],"acting":[6,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.0","version":"88'4","reported_seq":46,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:23:30.974992+0000","last_change":"2024-05-27T07:21:45.569224+0000","last_active":"2024-05-27T07:23:30.974992+0000","last_peered":"2024-05-27T07:23:30.974992+0000","last_clean":"2024-05-27T07:23:30.974992+0000","last_became_active":"2024-05-27T07:21:45.568818+0000","last_became_peered":"2024-05-27T07:21:45.568818+0000","last_unstale":"2024-05-27T07:23:30.974992+0000","last_undegraded":"2024-05-27T07:23:30.974992+0000","last_fullsized":"2024-05-27T07:23:30.974992+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:21:44.543638+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:21:44.543638+0000","last_clean_scrub_stamp":"2024-05-27T07:21:44.543638+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T08:12:33.157842+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,8],"acting":[7,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":29,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:23:31.621828+0000","last_change":"2024-05-27T07:21:59.733733+0000","last_active":"2024-05-27T07:23:31.621828+0000","last_peered":"2024-05-27T07:23:31.621828+0000","last_clean":"2024-05-27T07:23:31.621828+0000","last_became_active":"2024-05-27T07:21:59.733112+0000","last_became_peered":"2024-05-27T07:21:59.733112+0000","last_unstale":"2024-05-27T07:23:31.621828+0000","last_undegraded":"2024-05-27T07:23:31.621828+0000","last_fullsized":"2024-05-27T07:23:31.621828+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:21:58.698952+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:21:58.698952+0000","last_clean_scrub_stamp":"2024-05-27T07:21:58.698952+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T16:01:30.783800+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,7,6],"acting":[11,5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.5","version":"88'4","reported_seq":50,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:23:31.061825+0000","last_change":"2024-05-27T07:21:45.792029+0000","last_active":"2024-05-27T07:23:31.061825+0000","last_peered":"2024-05-27T07:23:31.061825+0000","last_clean":"2024-05-27T07:23:31.061825+0000","last_became_active":"2024-05-27T07:21:45.791492+0000","last_became_peered":"2024-05-27T07:21:45.791492+0000","last_unstale":"2024-05-27T07:23:31.061825+0000","last_undegraded":"2024-05-27T07:23:31.061825+0000","last_fullsized":"2024-05-27T07:23:31.061825+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:21:44.543638+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:21:44.543638+0000","last_clean_scrub_stamp":"2024-05-27T07:21:44.543638+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T17:36:42.444243+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,9],"acting":[8,0,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.4","version":"90'9","reported_seq":48,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:24:29.666461+0000","last_change":"2024-05-27T07:21:48.594769+0000","last_active":"2024-05-27T07:24:29.666461+0000","last_peered":"2024-05-27T07:24:29.666461+0000","last_clean":"2024-05-27T07:24:29.666461+0000","last_became_active":"2024-05-27T07:21:48.592979+0000","last_became_peered":"2024-05-27T07:21:48.592979+0000","last_unstale":"2024-05-27T07:24:29.666461+0000","last_undegraded":"2024-05-27T07:24:29.666461+0000","last_fullsized":"2024-05-27T07:24:29.666461+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:21:47.565641+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:21:47.565641+0000","last_clean_scrub_stamp":"2024-05-27T07:21:47.565641+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T10:06:48.268230+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,9],"acting":[6,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":29,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:23:31.102883+0000","last_change":"2024-05-27T07:21:59.736405+0000","last_active":"2024-05-27T07:23:31.102883+0000","last_peered":"2024-05-27T07:23:31.102883+0000","last_clean":"2024-05-27T07:23:31.102883+0000","last_became_active":"2024-05-27T07:21:59.735991+0000","last_became_peered":"2024-05-27T07:21:59.735991+0000","last_unstale":"2024-05-27T07:23:31.102883+0000","last_undegraded":"2024-05-27T07:23:31.102883+0000","last_fullsized":"2024-05-27T07:23:31.102883+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:21:58.698952+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:21:58.698952+0000","last_clean_scrub_stamp":"2024-05-27T07:21:58.698952+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T18:32:04.931725+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,10,0,7],"acting":[3,10,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"90'16","reported_seq":61,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:23:55.051249+0000","last_change":"2024-05-27T07:21:45.566513+0000","last_active":"2024-05-27T07:23:55.051249+0000","last_peered":"2024-05-27T07:23:55.051249+0000","last_clean":"2024-05-27T07:23:55.051249+0000","last_became_active":"2024-05-27T07:21:45.566107+0000","last_became_peered":"2024-05-27T07:21:45.566107+0000","last_unstale":"2024-05-27T07:23:55.051249+0000","last_undegraded":"2024-05-27T07:23:55.051249+0000","last_fullsized":"2024-05-27T07:23:55.051249+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:21:44.543638+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:21:44.543638+0000","last_clean_scrub_stamp":"2024-05-27T07:21:44.543638+0000","objects_scrubbed":0,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T14:11:34.851691+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":29,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,3],"acting":[9,8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.7","version":"90'5","reported_seq":44,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:24:29.678400+0000","last_change":"2024-05-27T07:21:48.593642+0000","last_active":"2024-05-27T07:24:29.678400+0000","last_peered":"2024-05-27T07:24:29.678400+0000","last_clean":"2024-05-27T07:24:29.678400+0000","last_became_active":"2024-05-27T07:21:48.592885+0000","last_became_peered":"2024-05-27T07:21:48.592885+0000","last_unstale":"2024-05-27T07:24:29.678400+0000","last_undegraded":"2024-05-27T07:24:29.678400+0000","last_fullsized":"2024-05-27T07:24:29.678400+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:21:47.565641+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:21:47.565641+0000","last_clean_scrub_stamp":"2024-05-27T07:21:47.565641+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T11:51:27.186560+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,7],"acting":[3,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":29,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:23:30.982161+0000","last_change":"2024-05-27T07:21:59.725533+0000","last_active":"2024-05-27T07:23:30.982161+0000","last_peered":"2024-05-27T07:23:30.982161+0000","last_clean":"2024-05-27T07:23:30.982161+0000","last_became_active":"2024-05-27T07:21:59.724991+0000","last_became_peered":"2024-05-27T07:21:59.724991+0000","last_unstale":"2024-05-27T07:23:30.982161+0000","last_undegraded":"2024-05-27T07:23:30.982161+0000","last_fullsized":"2024-05-27T07:23:30.982161+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:21:58.698952+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:21:58.698952+0000","last_clean_scrub_stamp":"2024-05-27T07:21:58.698952+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T17:22:54.650553+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,4,7],"acting":[2,1,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"88'7","reported_seq":53,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:23:31.757701+0000","last_change":"2024-05-27T07:21:45.572701+0000","last_active":"2024-05-27T07:23:31.757701+0000","last_peered":"2024-05-27T07:23:31.757701+0000","last_clean":"2024-05-27T07:23:31.757701+0000","last_became_active":"2024-05-27T07:21:45.572176+0000","last_became_peered":"2024-05-27T07:21:45.572176+0000","last_unstale":"2024-05-27T07:23:31.757701+0000","last_undegraded":"2024-05-27T07:23:31.757701+0000","last_fullsized":"2024-05-27T07:23:31.757701+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:21:44.543638+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:21:44.543638+0000","last_clean_scrub_stamp":"2024-05-27T07:21:44.543638+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T16:11:20.901250+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,11,6],"acting":[4,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"90'7","reported_seq":46,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:24:29.681965+0000","last_change":"2024-05-27T07:21:48.581152+0000","last_active":"2024-05-27T07:24:29.681965+0000","last_peered":"2024-05-27T07:24:29.681965+0000","last_clean":"2024-05-27T07:24:29.681965+0000","last_became_active":"2024-05-27T07:21:48.580543+0000","last_became_peered":"2024-05-27T07:21:48.580543+0000","last_unstale":"2024-05-27T07:24:29.681965+0000","last_undegraded":"2024-05-27T07:24:29.681965+0000","last_fullsized":"2024-05-27T07:24:29.681965+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:21:47.565641+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:21:47.565641+0000","last_clean_scrub_stamp":"2024-05-27T07:21:47.565641+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T18:45:56.732828+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,4],"acting":[0,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":36,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:23:31.757562+0000","last_change":"2024-05-27T07:22:31.417863+0000","last_active":"2024-05-27T07:23:31.757562+0000","last_peered":"2024-05-27T07:23:31.757562+0000","last_clean":"2024-05-27T07:23:31.757562+0000","last_became_active":"2024-05-27T07:22:31.417298+0000","last_became_peered":"2024-05-27T07:22:31.417298+0000","last_unstale":"2024-05-27T07:23:31.757562+0000","last_undegraded":"2024-05-27T07:23:31.757562+0000","last_fullsized":"2024-05-27T07:23:31.757562+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:21:58.698952+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:21:58.698952+0000","last_clean_scrub_stamp":"2024-05-27T07:21:58.698952+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T12:26:13.857477+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,10,11],"acting":[4,5,10,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.8","version":"88'2","reported_seq":57,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:23:31.626909+0000","last_change":"2024-05-27T07:23:31.626374+0000","last_active":"2024-05-27T07:23:31.626909+0000","last_peered":"2024-05-27T07:23:31.626909+0000","last_clean":"2024-05-27T07:23:31.626909+0000","last_became_active":"2024-05-27T07:23:31.623214+0000","last_became_peered":"2024-05-27T07:23:31.623214+0000","last_unstale":"2024-05-27T07:23:31.626909+0000","last_undegraded":"2024-05-27T07:23:31.626909+0000","last_fullsized":"2024-05-27T07:23:31.626909+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:21:44.543638+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:21:44.543638+0000","last_clean_scrub_stamp":"2024-05-27T07:21:44.543638+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T10:42:47.710138+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,11,0],"acting":[1,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"90'2","reported_seq":41,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:24:29.634809+0000","last_change":"2024-05-27T07:21:48.589605+0000","last_active":"2024-05-27T07:24:29.634809+0000","last_peered":"2024-05-27T07:24:29.634809+0000","last_clean":"2024-05-27T07:24:29.634809+0000","last_became_active":"2024-05-27T07:21:48.588294+0000","last_became_peered":"2024-05-27T07:21:48.588294+0000","last_unstale":"2024-05-27T07:24:29.634809+0000","last_undegraded":"2024-05-27T07:24:29.634809+0000","last_fullsized":"2024-05-27T07:24:29.634809+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:21:47.565641+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:21:47.565641+0000","last_clean_scrub_stamp":"2024-05-27T07:21:47.565641+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T09:02:29.618508+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,10],"acting":[8,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":29,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:23:31.653164+0000","last_change":"2024-05-27T07:21:59.738153+0000","last_active":"2024-05-27T07:23:31.653164+0000","last_peered":"2024-05-27T07:23:31.653164+0000","last_clean":"2024-05-27T07:23:31.653164+0000","last_became_active":"2024-05-27T07:21:59.737394+0000","last_became_peered":"2024-05-27T07:21:59.737394+0000","last_unstale":"2024-05-27T07:23:31.653164+0000","last_undegraded":"2024-05-27T07:23:31.653164+0000","last_fullsized":"2024-05-27T07:23:31.653164+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:21:58.698952+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:21:58.698952+0000","last_clean_scrub_stamp":"2024-05-27T07:21:58.698952+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T17:17:13.877688+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,2,5],"acting":[9,3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.9","version":"88'7","reported_seq":51,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:23:31.102972+0000","last_change":"2024-05-27T07:21:45.562638+0000","last_active":"2024-05-27T07:23:31.102972+0000","last_peered":"2024-05-27T07:23:31.102972+0000","last_clean":"2024-05-27T07:23:31.102972+0000","last_became_active":"2024-05-27T07:21:45.562242+0000","last_became_peered":"2024-05-27T07:21:45.562242+0000","last_unstale":"2024-05-27T07:23:31.102972+0000","last_undegraded":"2024-05-27T07:23:31.102972+0000","last_fullsized":"2024-05-27T07:23:31.102972+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:21:44.543638+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:21:44.543638+0000","last_clean_scrub_stamp":"2024-05-27T07:21:44.543638+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T10:40:15.847513+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,9,10],"acting":[3,9,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.8","version":"90'9","reported_seq":48,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:24:29.683130+0000","last_change":"2024-05-27T07:21:48.598186+0000","last_active":"2024-05-27T07:24:29.683130+0000","last_peered":"2024-05-27T07:24:29.683130+0000","last_clean":"2024-05-27T07:24:29.683130+0000","last_became_active":"2024-05-27T07:21:48.597715+0000","last_became_peered":"2024-05-27T07:21:48.597715+0000","last_unstale":"2024-05-27T07:24:29.683130+0000","last_undegraded":"2024-05-27T07:24:29.683130+0000","last_fullsized":"2024-05-27T07:24:29.683130+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:21:47.565641+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:21:47.565641+0000","last_clean_scrub_stamp":"2024-05-27T07:21:47.565641+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T08:01:35.983687+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,2],"acting":[11,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.c","version":"88'1","reported_seq":30,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:23:31.757394+0000","last_change":"2024-05-27T07:21:59.727247+0000","last_active":"2024-05-27T07:23:31.757394+0000","last_peered":"2024-05-27T07:23:31.757394+0000","last_clean":"2024-05-27T07:23:31.757394+0000","last_became_active":"2024-05-27T07:21:59.726788+0000","last_became_peered":"2024-05-27T07:21:59.726788+0000","last_unstale":"2024-05-27T07:23:31.757394+0000","last_undegraded":"2024-05-27T07:23:31.757394+0000","last_fullsized":"2024-05-27T07:23:31.757394+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:21:58.698952+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:21:58.698952+0000","last_clean_scrub_stamp":"2024-05-27T07:21:58.698952+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T10:09:27.095188+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,9,7,8],"acting":[4,9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.a","version":"88'3","reported_seq":47,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:23:30.982476+0000","last_change":"2024-05-27T07:21:45.569285+0000","last_active":"2024-05-27T07:23:30.982476+0000","last_peered":"2024-05-27T07:23:30.982476+0000","last_clean":"2024-05-27T07:23:30.982476+0000","last_became_active":"2024-05-27T07:21:45.568835+0000","last_became_peered":"2024-05-27T07:21:45.568835+0000","last_unstale":"2024-05-27T07:23:30.982476+0000","last_undegraded":"2024-05-27T07:23:30.982476+0000","last_fullsized":"2024-05-27T07:23:30.982476+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:21:44.543638+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:21:44.543638+0000","last_clean_scrub_stamp":"2024-05-27T07:21:44.543638+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T10:03:23.353800+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,8],"acting":[2,3,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.b","version":"90'8","reported_seq":47,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:24:29.679025+0000","last_change":"2024-05-27T07:21:48.583494+0000","last_active":"2024-05-27T07:24:29.679025+0000","last_peered":"2024-05-27T07:24:29.679025+0000","last_clean":"2024-05-27T07:24:29.679025+0000","last_became_active":"2024-05-27T07:21:48.583141+0000","last_became_peered":"2024-05-27T07:21:48.583141+0000","last_unstale":"2024-05-27T07:24:29.679025+0000","last_undegraded":"2024-05-27T07:24:29.679025+0000","last_fullsized":"2024-05-27T07:24:29.679025+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:21:47.565641+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:21:47.565641+0000","last_clean_scrub_stamp":"2024-05-27T07:21:47.565641+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T10:24:50.475195+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,10],"acting":[3,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":29,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:23:31.757429+0000","last_change":"2024-05-27T07:21:59.733966+0000","last_active":"2024-05-27T07:23:31.757429+0000","last_peered":"2024-05-27T07:23:31.757429+0000","last_clean":"2024-05-27T07:23:31.757429+0000","last_became_active":"2024-05-27T07:21:59.733411+0000","last_became_peered":"2024-05-27T07:21:59.733411+0000","last_unstale":"2024-05-27T07:23:31.757429+0000","last_undegraded":"2024-05-27T07:23:31.757429+0000","last_fullsized":"2024-05-27T07:23:31.757429+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:21:58.698952+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:21:58.698952+0000","last_clean_scrub_stamp":"2024-05-27T07:21:58.698952+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T15:08:04.918086+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,10,3],"acting":[4,2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.b","version":"88'4","reported_seq":50,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:23:31.061461+0000","last_change":"2024-05-27T07:21:45.570967+0000","last_active":"2024-05-27T07:23:31.061461+0000","last_peered":"2024-05-27T07:23:31.061461+0000","last_clean":"2024-05-27T07:23:31.061461+0000","last_became_active":"2024-05-27T07:21:45.570501+0000","last_became_peered":"2024-05-27T07:21:45.570501+0000","last_unstale":"2024-05-27T07:23:31.061461+0000","last_undegraded":"2024-05-27T07:23:31.061461+0000","last_fullsized":"2024-05-27T07:23:31.061461+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:21:44.543638+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:21:44.543638+0000","last_clean_scrub_stamp":"2024-05-27T07:21:44.543638+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T14:11:26.496194+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.a","version":"90'5","reported_seq":42,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:24:29.682141+0000","last_change":"2024-05-27T07:21:48.594690+0000","last_active":"2024-05-27T07:24:29.682141+0000","last_peered":"2024-05-27T07:24:29.682141+0000","last_clean":"2024-05-27T07:24:29.682141+0000","last_became_active":"2024-05-27T07:21:48.594314+0000","last_became_peered":"2024-05-27T07:21:48.594314+0000","last_unstale":"2024-05-27T07:24:29.682141+0000","last_undegraded":"2024-05-27T07:24:29.682141+0000","last_fullsized":"2024-05-27T07:24:29.682141+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:21:47.565641+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:21:47.565641+0000","last_clean_scrub_stamp":"2024-05-27T07:21:47.565641+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T14:06:43.401844+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,6],"acting":[7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":29,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:23:30.978984+0000","last_change":"2024-05-27T07:21:59.732289+0000","last_active":"2024-05-27T07:23:30.978984+0000","last_peered":"2024-05-27T07:23:30.978984+0000","last_clean":"2024-05-27T07:23:30.978984+0000","last_became_active":"2024-05-27T07:21:59.731863+0000","last_became_peered":"2024-05-27T07:21:59.731863+0000","last_unstale":"2024-05-27T07:23:30.978984+0000","last_undegraded":"2024-05-27T07:23:30.978984+0000","last_fullsized":"2024-05-27T07:23:30.978984+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:21:58.698952+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:21:58.698952+0000","last_clean_scrub_stamp":"2024-05-27T07:21:58.698952+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T11:06:38.246484+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,6,7],"acting":[1,5,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.c","version":"88'5","reported_seq":146,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:24:30.031254+0000","last_change":"2024-05-27T07:21:45.791978+0000","last_active":"2024-05-27T07:24:30.031254+0000","last_peered":"2024-05-27T07:24:30.031254+0000","last_clean":"2024-05-27T07:24:30.031254+0000","last_became_active":"2024-05-27T07:21:45.791146+0000","last_became_peered":"2024-05-27T07:21:45.791146+0000","last_unstale":"2024-05-27T07:24:30.031254+0000","last_undegraded":"2024-05-27T07:24:30.031254+0000","last_fullsized":"2024-05-27T07:24:30.031254+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:21:44.543638+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:21:44.543638+0000","last_clean_scrub_stamp":"2024-05-27T07:21:44.543638+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T09:09:24.952876+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1170,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":97,"num_read_kb":2,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,11,0],"acting":[2,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"90'7","reported_seq":44,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:24:29.684312+0000","last_change":"2024-05-27T07:21:48.602786+0000","last_active":"2024-05-27T07:24:29.684312+0000","last_peered":"2024-05-27T07:24:29.684312+0000","last_clean":"2024-05-27T07:24:29.684312+0000","last_became_active":"2024-05-27T07:21:48.602476+0000","last_became_peered":"2024-05-27T07:21:48.602476+0000","last_unstale":"2024-05-27T07:24:29.684312+0000","last_undegraded":"2024-05-27T07:24:29.684312+0000","last_fullsized":"2024-05-27T07:24:29.684312+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:21:47.565641+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:21:47.565641+0000","last_clean_scrub_stamp":"2024-05-27T07:21:47.565641+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T13:22:58.583230+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,11],"acting":[7,1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":29,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:23:30.981918+0000","last_change":"2024-05-27T07:21:59.729595+0000","last_active":"2024-05-27T07:23:30.981918+0000","last_peered":"2024-05-27T07:23:30.981918+0000","last_clean":"2024-05-27T07:23:30.981918+0000","last_became_active":"2024-05-27T07:21:59.729147+0000","last_became_peered":"2024-05-27T07:21:59.729147+0000","last_unstale":"2024-05-27T07:23:30.981918+0000","last_undegraded":"2024-05-27T07:23:30.981918+0000","last_fullsized":"2024-05-27T07:23:30.981918+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:21:58.698952+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:21:58.698952+0000","last_clean_scrub_stamp":"2024-05-27T07:21:58.698952+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T13:31:28.579617+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,9,1,4],"acting":[0,9,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.d","version":"88'3","reported_seq":48,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:23:30.977593+0000","last_change":"2024-05-27T07:21:45.569287+0000","last_active":"2024-05-27T07:23:30.977593+0000","last_peered":"2024-05-27T07:23:30.977593+0000","last_clean":"2024-05-27T07:23:30.977593+0000","last_became_active":"2024-05-27T07:21:45.568935+0000","last_became_peered":"2024-05-27T07:21:45.568935+0000","last_unstale":"2024-05-27T07:23:30.977593+0000","last_undegraded":"2024-05-27T07:23:30.977593+0000","last_fullsized":"2024-05-27T07:23:30.977593+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:21:44.543638+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:21:44.543638+0000","last_clean_scrub_stamp":"2024-05-27T07:21:44.543638+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T07:27:14.674135+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,10,3],"acting":[5,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.c","version":"90'6","reported_seq":43,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:24:29.671545+0000","last_change":"2024-05-27T07:21:48.597718+0000","last_active":"2024-05-27T07:24:29.671545+0000","last_peered":"2024-05-27T07:24:29.671545+0000","last_clean":"2024-05-27T07:24:29.671545+0000","last_became_active":"2024-05-27T07:21:48.597379+0000","last_became_peered":"2024-05-27T07:21:48.597379+0000","last_unstale":"2024-05-27T07:24:29.671545+0000","last_undegraded":"2024-05-27T07:24:29.671545+0000","last_fullsized":"2024-05-27T07:24:29.671545+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:21:47.565641+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:21:47.565641+0000","last_clean_scrub_stamp":"2024-05-27T07:21:47.565641+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T08:48:41.658364+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,5],"acting":[7,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":29,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:23:30.976864+0000","last_change":"2024-05-27T07:21:59.730191+0000","last_active":"2024-05-27T07:23:30.976864+0000","last_peered":"2024-05-27T07:23:30.976864+0000","last_clean":"2024-05-27T07:23:30.976864+0000","last_became_active":"2024-05-27T07:21:59.729787+0000","last_became_peered":"2024-05-27T07:21:59.729787+0000","last_unstale":"2024-05-27T07:23:30.976864+0000","last_undegraded":"2024-05-27T07:23:30.976864+0000","last_fullsized":"2024-05-27T07:23:30.976864+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:21:58.698952+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:21:58.698952+0000","last_clean_scrub_stamp":"2024-05-27T07:21:58.698952+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T13:22:38.961246+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7,1],"acting":[5,2,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.e","version":"88'5","reported_seq":49,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:23:30.974354+0000","last_change":"2024-05-27T07:21:45.559219+0000","last_active":"2024-05-27T07:23:30.974354+0000","last_peered":"2024-05-27T07:23:30.974354+0000","last_clean":"2024-05-27T07:23:30.974354+0000","last_became_active":"2024-05-27T07:21:45.558848+0000","last_became_peered":"2024-05-27T07:21:45.558848+0000","last_unstale":"2024-05-27T07:23:30.974354+0000","last_undegraded":"2024-05-27T07:23:30.974354+0000","last_fullsized":"2024-05-27T07:23:30.974354+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:21:44.543638+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:21:44.543638+0000","last_clean_scrub_stamp":"2024-05-27T07:21:44.543638+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T15:03:55.632162+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.f","version":"90'9","reported_seq":46,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:24:29.672917+0000","last_change":"2024-05-27T07:21:48.596349+0000","last_active":"2024-05-27T07:24:29.672917+0000","last_peered":"2024-05-27T07:24:29.672917+0000","last_clean":"2024-05-27T07:24:29.672917+0000","last_became_active":"2024-05-27T07:21:48.595830+0000","last_became_peered":"2024-05-27T07:21:48.595830+0000","last_unstale":"2024-05-27T07:24:29.672917+0000","last_undegraded":"2024-05-27T07:24:29.672917+0000","last_fullsized":"2024-05-27T07:24:29.672917+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:21:47.565641+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:21:47.565641+0000","last_clean_scrub_stamp":"2024-05-27T07:21:47.565641+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T13:33:11.431988+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,8],"acting":[7,11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.20","version":"90'9","reported_seq":48,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:24:29.674515+0000","last_change":"2024-05-27T07:21:48.598504+0000","last_active":"2024-05-27T07:24:29.674515+0000","last_peered":"2024-05-27T07:24:29.674515+0000","last_clean":"2024-05-27T07:24:29.674515+0000","last_became_active":"2024-05-27T07:21:48.598103+0000","last_became_peered":"2024-05-27T07:21:48.598103+0000","last_unstale":"2024-05-27T07:24:29.674515+0000","last_undegraded":"2024-05-27T07:24:29.674515+0000","last_fullsized":"2024-05-27T07:24:29.674515+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:21:47.565641+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:21:47.565641+0000","last_clean_scrub_stamp":"2024-05-27T07:21:47.565641+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T10:34:02.743735+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,5],"acting":[11,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.21","version":"90'12","reported_seq":51,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:24:29.669423+0000","last_change":"2024-05-27T07:21:48.593379+0000","last_active":"2024-05-27T07:24:29.669423+0000","last_peered":"2024-05-27T07:24:29.669423+0000","last_clean":"2024-05-27T07:24:29.669423+0000","last_became_active":"2024-05-27T07:21:48.592995+0000","last_became_peered":"2024-05-27T07:21:48.592995+0000","last_unstale":"2024-05-27T07:24:29.669423+0000","last_undegraded":"2024-05-27T07:24:29.669423+0000","last_fullsized":"2024-05-27T07:24:29.669423+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:21:47.565641+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:21:47.565641+0000","last_clean_scrub_stamp":"2024-05-27T07:21:47.565641+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T19:01:56.288235+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,4],"acting":[2,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"90'8","reported_seq":47,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:24:29.666667+0000","last_change":"2024-05-27T07:21:48.592837+0000","last_active":"2024-05-27T07:24:29.666667+0000","last_peered":"2024-05-27T07:24:29.666667+0000","last_clean":"2024-05-27T07:24:29.666667+0000","last_became_active":"2024-05-27T07:21:48.592459+0000","last_became_peered":"2024-05-27T07:21:48.592459+0000","last_unstale":"2024-05-27T07:24:29.666667+0000","last_undegraded":"2024-05-27T07:24:29.666667+0000","last_fullsized":"2024-05-27T07:24:29.666667+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:21:47.565641+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:21:47.565641+0000","last_clean_scrub_stamp":"2024-05-27T07:21:47.565641+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T14:00:19.490348+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.23","version":"90'7","reported_seq":46,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:24:29.681980+0000","last_change":"2024-05-27T07:21:48.583139+0000","last_active":"2024-05-27T07:24:29.681980+0000","last_peered":"2024-05-27T07:24:29.681980+0000","last_clean":"2024-05-27T07:24:29.681980+0000","last_became_active":"2024-05-27T07:21:48.582822+0000","last_became_peered":"2024-05-27T07:21:48.582822+0000","last_unstale":"2024-05-27T07:24:29.681980+0000","last_undegraded":"2024-05-27T07:24:29.681980+0000","last_fullsized":"2024-05-27T07:24:29.681980+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:21:47.565641+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:21:47.565641+0000","last_clean_scrub_stamp":"2024-05-27T07:21:47.565641+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T16:30:03.454035+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,4],"acting":[0,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"90'4","reported_seq":43,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:24:29.668048+0000","last_change":"2024-05-27T07:21:48.588906+0000","last_active":"2024-05-27T07:24:29.668048+0000","last_peered":"2024-05-27T07:24:29.668048+0000","last_clean":"2024-05-27T07:24:29.668048+0000","last_became_active":"2024-05-27T07:21:48.588555+0000","last_became_peered":"2024-05-27T07:21:48.588555+0000","last_unstale":"2024-05-27T07:24:29.668048+0000","last_undegraded":"2024-05-27T07:24:29.668048+0000","last_fullsized":"2024-05-27T07:24:29.668048+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:21:47.565641+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:21:47.565641+0000","last_clean_scrub_stamp":"2024-05-27T07:21:47.565641+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T19:04:23.006852+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"90'6","reported_seq":45,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:24:29.671980+0000","last_change":"2024-05-27T07:21:48.590271+0000","last_active":"2024-05-27T07:24:29.671980+0000","last_peered":"2024-05-27T07:24:29.671980+0000","last_clean":"2024-05-27T07:24:29.671980+0000","last_became_active":"2024-05-27T07:21:48.589898+0000","last_became_peered":"2024-05-27T07:21:48.589898+0000","last_unstale":"2024-05-27T07:24:29.671980+0000","last_undegraded":"2024-05-27T07:24:29.671980+0000","last_fullsized":"2024-05-27T07:24:29.671980+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:21:47.565641+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:21:47.565641+0000","last_clean_scrub_stamp":"2024-05-27T07:21:47.565641+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T11:57:43.238211+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,5],"acting":[11,8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.26","version":"90'15","reported_seq":54,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:24:29.672568+0000","last_change":"2024-05-27T07:21:48.597321+0000","last_active":"2024-05-27T07:24:29.672568+0000","last_peered":"2024-05-27T07:24:29.672568+0000","last_clean":"2024-05-27T07:24:29.672568+0000","last_became_active":"2024-05-27T07:21:48.596953+0000","last_became_peered":"2024-05-27T07:21:48.596953+0000","last_unstale":"2024-05-27T07:24:29.672568+0000","last_undegraded":"2024-05-27T07:24:29.672568+0000","last_fullsized":"2024-05-27T07:24:29.672568+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:21:47.565641+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:21:47.565641+0000","last_clean_scrub_stamp":"2024-05-27T07:21:47.565641+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T16:21:17.645098+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,8,1],"acting":[2,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.27","version":"90'2","reported_seq":41,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:24:29.673709+0000","last_change":"2024-05-27T07:21:48.593717+0000","last_active":"2024-05-27T07:24:29.673709+0000","last_peered":"2024-05-27T07:24:29.673709+0000","last_clean":"2024-05-27T07:24:29.673709+0000","last_became_active":"2024-05-27T07:21:48.593356+0000","last_became_peered":"2024-05-27T07:21:48.593356+0000","last_unstale":"2024-05-27T07:24:29.673709+0000","last_undegraded":"2024-05-27T07:24:29.673709+0000","last_fullsized":"2024-05-27T07:24:29.673709+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:21:47.565641+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:21:47.565641+0000","last_clean_scrub_stamp":"2024-05-27T07:21:47.565641+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T17:06:27.341627+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"90'6","reported_seq":45,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:24:29.667694+0000","last_change":"2024-05-27T07:21:48.592360+0000","last_active":"2024-05-27T07:24:29.667694+0000","last_peered":"2024-05-27T07:24:29.667694+0000","last_clean":"2024-05-27T07:24:29.667694+0000","last_became_active":"2024-05-27T07:21:48.591650+0000","last_became_peered":"2024-05-27T07:21:48.591650+0000","last_unstale":"2024-05-27T07:24:29.667694+0000","last_undegraded":"2024-05-27T07:24:29.667694+0000","last_fullsized":"2024-05-27T07:24:29.667694+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:21:47.565641+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:21:47.565641+0000","last_clean_scrub_stamp":"2024-05-27T07:21:47.565641+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T17:32:19.408787+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,3],"acting":[5,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"90'8","reported_seq":47,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:24:29.674125+0000","last_change":"2024-05-27T07:21:48.587319+0000","last_active":"2024-05-27T07:24:29.674125+0000","last_peered":"2024-05-27T07:24:29.674125+0000","last_clean":"2024-05-27T07:24:29.674125+0000","last_became_active":"2024-05-27T07:21:48.586924+0000","last_became_peered":"2024-05-27T07:21:48.586924+0000","last_unstale":"2024-05-27T07:24:29.674125+0000","last_undegraded":"2024-05-27T07:24:29.674125+0000","last_fullsized":"2024-05-27T07:24:29.674125+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:21:47.565641+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:21:47.565641+0000","last_clean_scrub_stamp":"2024-05-27T07:21:47.565641+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T09:28:48.129037+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,11],"acting":[3,2,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2a","version":"90'10","reported_seq":49,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:24:29.684848+0000","last_change":"2024-05-27T07:21:48.604018+0000","last_active":"2024-05-27T07:24:29.684848+0000","last_peered":"2024-05-27T07:24:29.684848+0000","last_clean":"2024-05-27T07:24:29.684848+0000","last_became_active":"2024-05-27T07:21:48.603368+0000","last_became_peered":"2024-05-27T07:21:48.603368+0000","last_unstale":"2024-05-27T07:24:29.684848+0000","last_undegraded":"2024-05-27T07:24:29.684848+0000","last_fullsized":"2024-05-27T07:24:29.684848+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:21:47.565641+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:21:47.565641+0000","last_clean_scrub_stamp":"2024-05-27T07:21:47.565641+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T09:49:42.440480+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2b","version":"90'11","reported_seq":50,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:24:29.678674+0000","last_change":"2024-05-27T07:21:48.589434+0000","last_active":"2024-05-27T07:24:29.678674+0000","last_peered":"2024-05-27T07:24:29.678674+0000","last_clean":"2024-05-27T07:24:29.678674+0000","last_became_active":"2024-05-27T07:21:48.589102+0000","last_became_peered":"2024-05-27T07:21:48.589102+0000","last_unstale":"2024-05-27T07:24:29.678674+0000","last_undegraded":"2024-05-27T07:24:29.678674+0000","last_fullsized":"2024-05-27T07:24:29.678674+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:21:47.565641+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:21:47.565641+0000","last_clean_scrub_stamp":"2024-05-27T07:21:47.565641+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T18:28:52.955138+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2c","version":"90'8","reported_seq":53,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:24:29.672868+0000","last_change":"2024-05-27T07:22:30.802433+0000","last_active":"2024-05-27T07:24:29.672868+0000","last_peered":"2024-05-27T07:24:29.672868+0000","last_clean":"2024-05-27T07:24:29.672868+0000","last_became_active":"2024-05-27T07:22:30.801881+0000","last_became_peered":"2024-05-27T07:22:30.801881+0000","last_unstale":"2024-05-27T07:24:29.672868+0000","last_undegraded":"2024-05-27T07:24:29.672868+0000","last_fullsized":"2024-05-27T07:24:29.672868+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:21:47.565641+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:21:47.565641+0000","last_clean_scrub_stamp":"2024-05-27T07:21:47.565641+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T11:59:17.916052+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,9,8],"acting":[11,9,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":234,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":32768,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":128,"acting":128,"num_store_stats":12},{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":500,"ondisk_log_size":500,"up":192,"acting":192,"num_store_stats":12},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":81,"num_read_kb":69,"num_write":123,"num_write_kb":2150,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":2951840,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":376832,"data_stored":4132576,"data_compressed":249874,"data_compressed_allocated":315392,"data_compressed_original":4096000,"omap_allocated":0,"internal_metadata":0},"log_size":92,"ondisk_log_size":92,"up":3,"acting":3,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":72353,"num_objects":98,"num_object_clones":0,"num_object_copies":294,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":98,"num_whiteouts":0,"num_read":1018,"num_read_kb":590,"num_write":260,"num_write_kb":153,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":22,"num_keys_recovered":0,"num_objects_omap":66,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":577536,"data_stored":217059,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":224,"ondisk_log_size":224,"up":96,"acting":96,"num_store_stats":12}],"osd_stats":[{"osd":11,"up_from":71,"seq":304942678065,"num_pgs":29,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32152,"kb_used_data":976,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713000,"statfs":{"total":95995035648,"available":95962112000,"internally_reserved":0,"allocated":999424,"data_stored":519173,"data_compressed":32421,"data_compressed_allocated":45056,"data_compressed_original":90112,"omap_allocated":1596,"internal_metadata":31914436},"hb_peers":[0,1,2,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":10,"up_from":65,"seq":279172874294,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28136,"kb_used_data":1120,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717016,"statfs":{"total":95995035648,"available":95966224384,"internally_reserved":0,"allocated":1146880,"data_stored":1171366,"data_compressed":68581,"data_compressed_allocated":90112,"data_compressed_original":679936,"omap_allocated":1600,"internal_metadata":27654592},"hb_peers":[0,1,2,3,4,5,6,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":9,"up_from":60,"seq":257698037820,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36336,"kb_used_data":1000,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708816,"statfs":{"total":95995035648,"available":95957827584,"internally_reserved":0,"allocated":1024000,"data_stored":523177,"data_compressed":32421,"data_compressed_allocated":45056,"data_compressed_original":90112,"omap_allocated":1600,"internal_metadata":36174272},"hb_peers":[0,1,2,3,4,5,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":54,"seq":231928234049,"num_pgs":31,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28004,"kb_used_data":988,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717148,"statfs":{"total":95995035648,"available":95966359552,"internally_reserved":0,"allocated":1011712,"data_stored":519437,"data_compressed":32421,"data_compressed_allocated":45056,"data_compressed_original":90112,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":48,"seq":206158430277,"num_pgs":37,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32220,"kb_used_data":1044,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712932,"statfs":{"total":95995035648,"available":95962042368,"internally_reserved":0,"allocated":1069056,"data_stored":1116629,"data_compressed":68581,"data_compressed_allocated":90112,"data_compressed_original":679936,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":6,"up_from":43,"seq":184683593802,"num_pgs":32,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27988,"kb_used_data":972,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717164,"statfs":{"total":95995035648,"available":95966375936,"internally_reserved":0,"allocated":995328,"data_stored":514585,"data_compressed":32421,"data_compressed_allocated":45056,"data_compressed_original":90112,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574948,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36376,"kb_used_data":1040,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708776,"statfs":{"total":95995035648,"available":95957786624,"internally_reserved":0,"allocated":1064960,"data_stored":580482,"data_compressed":32421,"data_compressed_allocated":45056,"data_compressed_original":90112,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738473,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28012,"kb_used_data":996,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717140,"statfs":{"total":95995035648,"available":95966351360,"internally_reserved":0,"allocated":1019904,"data_stored":516041,"data_compressed":32421,"data_compressed_allocated":45056,"data_compressed_original":90112,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411423,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36388,"kb_used_data":1052,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708764,"statfs":{"total":95995035648,"available":95957774336,"internally_reserved":0,"allocated":1077248,"data_stored":581122,"data_compressed":32421,"data_compressed_allocated":45056,"data_compressed_original":90112,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,1,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":3,"up_from":26,"seq":111669149786,"num_pgs":40,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36320,"kb_used_data":984,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708832,"statfs":{"total":95995035648,"available":95957843968,"internally_reserved":0,"allocated":1007616,"data_stored":514293,"data_compressed":32421,"data_compressed_allocated":45056,"data_compressed_original":90112,"omap_allocated":1593,"internal_metadata":36174279},"hb_peers":[0,1,2,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":4,"up_from":32,"seq":137438953557,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28004,"kb_used_data":988,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717148,"statfs":{"total":95995035648,"available":95966359552,"internally_reserved":0,"allocated":1011712,"data_stored":519303,"data_compressed":32421,"data_compressed_allocated":45056,"data_compressed_original":90112,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":37,"seq":158913790032,"num_pgs":44,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28068,"kb_used_data":1052,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717084,"statfs":{"total":95995035648,"available":95966294016,"internally_reserved":0,"allocated":1077248,"data_stored":1111679,"data_compressed":68581,"data_compressed_allocated":90112,"data_compressed_original":679936,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":34537,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":34537,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":36160,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":36160,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":36160,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":36160,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":36160,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":2178,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":86016,"data_stored":66619,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":98304,"data_stored":67259,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":430,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1344,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":3352,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":722,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":4206,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1478,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":1122,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":118784,"data_stored":67135,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1214,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":8192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":8192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-05-27T07:24:37.294 INFO:tasks.cephadm.ceph_manager.ceph:clean! 2024-05-27T07:24:37.294 INFO:tasks.ceph:Waiting until ceph cluster ceph is healthy... 2024-05-27T07:24:37.295 INFO:tasks.cephadm.ceph_manager.ceph:wait_until_healthy 2024-05-27T07:24:37.295 DEBUG:teuthology.orchestra.run.smithi052:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:38f9d900c83c63799fbdbe61acc9a11b0d3554a6 shell --fsid 73df2636-1bf8-11ef-bc9b-c7b262605968 -- ceph health --format=json 2024-05-27T07:24:37.579 INFO:teuthology.orchestra.run.smithi052.stderr:Inferring config /var/lib/ceph/73df2636-1bf8-11ef-bc9b-c7b262605968/mon.a/config 2024-05-27T07:24:37.974 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:24:37 smithi052 ceph-mon[31955]: from='client.15039 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-05-27T07:24:38.102 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:24:37 smithi173 ceph-mon[37000]: from='client.15039 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-05-27T07:24:38.190 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:24:37 smithi105 ceph-mon[35977]: from='client.15039 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-05-27T07:24:38.979 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:24:38 smithi052 ceph-mon[31955]: pgmap v384: 129 pgs: 129 active+clean; 647 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 4.8 KiB/s rd, 597 B/s wr, 10 op/s 2024-05-27T07:24:39.102 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:24:38 smithi173 ceph-mon[37000]: pgmap v384: 129 pgs: 129 active+clean; 647 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 4.8 KiB/s rd, 597 B/s wr, 10 op/s 2024-05-27T07:24:39.190 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:24:38 smithi105 ceph-mon[35977]: pgmap v384: 129 pgs: 129 active+clean; 647 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 4.8 KiB/s rd, 597 B/s wr, 10 op/s 2024-05-27T07:24:39.417 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:39.427+0000 7f8d2d1a4700 1 -- 172.21.15.52:0/911788345 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f8d28109850 msgr2=0x7f8d281111b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:24:39.417 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:39.427+0000 7f8d2d1a4700 1 --2- 172.21.15.52:0/911788345 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f8d28109850 0x7f8d281111b0 secure :-1 s=READY pgs=300 cs=0 l=1 rev1=1 crypto rx=0x7f8d2000b0d0 tx=0x7f8d2000b3e0 comp rx=0 tx=0).stop 2024-05-27T07:24:39.418 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:39.427+0000 7f8d2d1a4700 1 -- 172.21.15.52:0/911788345 shutdown_connections 2024-05-27T07:24:39.418 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:39.427+0000 7f8d2d1a4700 1 --2- 172.21.15.52:0/911788345 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f8d28109850 0x7f8d281111b0 unknown :-1 s=CLOSED pgs=300 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:24:39.418 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:39.427+0000 7f8d2d1a4700 1 --2- 172.21.15.52:0/911788345 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f8d281045d0 0x7f8d281049e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:24:39.418 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:39.427+0000 7f8d2d1a4700 1 --2- 172.21.15.52:0/911788345 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f8d28108f00 0x7f8d28109310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:24:39.418 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:39.427+0000 7f8d2d1a4700 1 -- 172.21.15.52:0/911788345 >> 172.21.15.52:0/911788345 conn(0x7f8d280ffee0 msgr2=0x7f8d28102330 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:24:39.418 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:39.427+0000 7f8d2d1a4700 1 -- 172.21.15.52:0/911788345 shutdown_connections 2024-05-27T07:24:39.418 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:39.427+0000 7f8d2d1a4700 1 -- 172.21.15.52:0/911788345 wait complete. 2024-05-27T07:24:39.420 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:39.428+0000 7f8d2d1a4700 1 Processor -- start 2024-05-27T07:24:39.420 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:39.429+0000 7f8d2d1a4700 1 -- start start 2024-05-27T07:24:39.420 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:39.429+0000 7f8d2d1a4700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f8d281045d0 0x7f8d2819adb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:24:39.420 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:39.429+0000 7f8d2d1a4700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f8d28108f00 0x7f8d2819b350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:24:39.420 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:39.429+0000 7f8d2d1a4700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f8d28109850 0x7f8d2819f5e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:24:39.420 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:39.429+0000 7f8d2d1a4700 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x7f8d28114480 con 0x7f8d28109850 2024-05-27T07:24:39.420 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:39.429+0000 7f8d2d1a4700 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f8d28114300 con 0x7f8d281045d0 2024-05-27T07:24:39.420 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:39.429+0000 7f8d2d1a4700 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f8d28114600 con 0x7f8d28108f00 2024-05-27T07:24:39.420 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:39.429+0000 7f8d2759e700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f8d28109850 0x7f8d2819f5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:24:39.420 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:39.429+0000 7f8d2759e700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f8d28109850 0x7f8d2819f5e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.52:49962/0 (socket says 172.21.15.52:49962) 2024-05-27T07:24:39.420 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:39.429+0000 7f8d2759e700 1 -- 172.21.15.52:0/1902117694 learned_addr learned my addr 172.21.15.52:0/1902117694 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-05-27T07:24:39.421 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:39.429+0000 7f8d2659c700 1 --2- 172.21.15.52:0/1902117694 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f8d28108f00 0x7f8d2819b350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:24:39.421 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:39.429+0000 7f8d26d9d700 1 --2- 172.21.15.52:0/1902117694 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f8d281045d0 0x7f8d2819adb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:24:39.421 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:39.430+0000 7f8d2759e700 1 -- 172.21.15.52:0/1902117694 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f8d28108f00 msgr2=0x7f8d2819b350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:24:39.421 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:39.430+0000 7f8d2759e700 1 --2- 172.21.15.52:0/1902117694 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f8d28108f00 0x7f8d2819b350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:24:39.421 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:39.430+0000 7f8d2759e700 1 -- 172.21.15.52:0/1902117694 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f8d281045d0 msgr2=0x7f8d2819adb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:24:39.421 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:39.430+0000 7f8d2759e700 1 --2- 172.21.15.52:0/1902117694 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f8d281045d0 0x7f8d2819adb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:24:39.421 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:39.430+0000 7f8d2759e700 1 -- 172.21.15.52:0/1902117694 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8d2819fc80 con 0x7f8d28109850 2024-05-27T07:24:39.421 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:39.430+0000 7f8d2759e700 1 --2- 172.21.15.52:0/1902117694 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f8d28109850 0x7f8d2819f5e0 secure :-1 s=READY pgs=301 cs=0 l=1 rev1=1 crypto rx=0x7f8d200095c0 tx=0x7f8d20007a20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:24:39.421 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:39.430+0000 7f8d1ffff700 1 -- 172.21.15.52:0/1902117694 <== mon.0 v2:172.21.15.52:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8d2001f070 con 0x7f8d28109850 2024-05-27T07:24:39.421 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:39.430+0000 7f8d2d1a4700 1 -- 172.21.15.52:0/1902117694 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8d2819ff10 con 0x7f8d28109850 2024-05-27T07:24:39.421 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:39.430+0000 7f8d1ffff700 1 -- 172.21.15.52:0/1902117694 <== mon.0 v2:172.21.15.52:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f8d20007e20 con 0x7f8d28109850 2024-05-27T07:24:39.421 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:39.430+0000 7f8d1ffff700 1 -- 172.21.15.52:0/1902117694 <== mon.0 v2:172.21.15.52:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8d2001b4a0 con 0x7f8d28109850 2024-05-27T07:24:39.422 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:39.430+0000 7f8d2d1a4700 1 -- 172.21.15.52:0/1902117694 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8d281a03f0 con 0x7f8d28109850 2024-05-27T07:24:39.422 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:39.431+0000 7f8d2d1a4700 1 -- 172.21.15.52:0/1902117694 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8d280512b0 con 0x7f8d28109850 2024-05-27T07:24:39.422 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:39.433+0000 7f8d1ffff700 1 -- 172.21.15.52:0/1902117694 <== mon.0 v2:172.21.15.52:3300/0 4 ==== mgrmap(e 18) ==== 149652+0+0 (secure 0 0 0) 0x7f8d20022070 con 0x7f8d28109850 2024-05-27T07:24:39.423 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:39.434+0000 7f8d1ffff700 1 --2- 172.21.15.52:0/1902117694 >> [v2:172.21.15.52:6800/3513075805,v1:172.21.15.52:6801/3513075805] conn(0x7f8d140b2880 0x7f8d140b4d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:24:39.423 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:39.434+0000 7f8d26d9d700 1 --2- 172.21.15.52:0/1902117694 >> [v2:172.21.15.52:6800/3513075805,v1:172.21.15.52:6801/3513075805] conn(0x7f8d140b2880 0x7f8d140b4d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:24:39.424 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:39.434+0000 7f8d1ffff700 1 -- 172.21.15.52:0/1902117694 <== mon.0 v2:172.21.15.52:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9121+0+0 (secure 0 0 0) 0x7f8d2000f020 con 0x7f8d28109850 2024-05-27T07:24:39.424 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:39.434+0000 7f8d26d9d700 1 --2- 172.21.15.52:0/1902117694 >> [v2:172.21.15.52:6800/3513075805,v1:172.21.15.52:6801/3513075805] conn(0x7f8d140b2880 0x7f8d140b4d90 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7f8d10005950 tx=0x7f8d1000b410 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:24:39.427 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:39.437+0000 7f8d1ffff700 1 -- 172.21.15.52:0/1902117694 <== mon.0 v2:172.21.15.52:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191963 (secure 0 0 0) 0x7f8d20029020 con 0x7f8d28109850 2024-05-27T07:24:39.694 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:39.704+0000 7f8d2d1a4700 1 -- 172.21.15.52:0/1902117694 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) -- 0x7f8d2802e5e0 con 0x7f8d28109850 2024-05-27T07:24:39.694 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:39.704+0000 7f8d1ffff700 1 -- 172.21.15.52:0/1902117694 <== mon.0 v2:172.21.15.52:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) ==== 72+0+46 (secure 0 0 0) 0x7f8d200ab590 con 0x7f8d28109850 2024-05-27T07:24:39.694 INFO:teuthology.orchestra.run.smithi052.stdout: 2024-05-27T07:24:39.694 INFO:teuthology.orchestra.run.smithi052.stdout:{"status":"HEALTH_OK","checks":{},"mutes":[]} 2024-05-27T07:24:39.697 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:39.707+0000 7f8d2d1a4700 1 -- 172.21.15.52:0/1902117694 >> [v2:172.21.15.52:6800/3513075805,v1:172.21.15.52:6801/3513075805] conn(0x7f8d140b2880 msgr2=0x7f8d140b4d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:24:39.697 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:39.707+0000 7f8d2d1a4700 1 --2- 172.21.15.52:0/1902117694 >> [v2:172.21.15.52:6800/3513075805,v1:172.21.15.52:6801/3513075805] conn(0x7f8d140b2880 0x7f8d140b4d90 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7f8d10005950 tx=0x7f8d1000b410 comp rx=0 tx=0).stop 2024-05-27T07:24:39.697 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:39.707+0000 7f8d2d1a4700 1 -- 172.21.15.52:0/1902117694 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f8d28109850 msgr2=0x7f8d2819f5e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:24:39.697 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:39.707+0000 7f8d2d1a4700 1 --2- 172.21.15.52:0/1902117694 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f8d28109850 0x7f8d2819f5e0 secure :-1 s=READY pgs=301 cs=0 l=1 rev1=1 crypto rx=0x7f8d200095c0 tx=0x7f8d20007a20 comp rx=0 tx=0).stop 2024-05-27T07:24:39.697 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:39.707+0000 7f8d2d1a4700 1 -- 172.21.15.52:0/1902117694 shutdown_connections 2024-05-27T07:24:39.697 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:39.707+0000 7f8d2d1a4700 1 --2- 172.21.15.52:0/1902117694 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f8d28109850 0x7f8d2819f5e0 unknown :-1 s=CLOSED pgs=301 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:24:39.697 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:39.707+0000 7f8d2d1a4700 1 --2- 172.21.15.52:0/1902117694 >> [v2:172.21.15.52:6800/3513075805,v1:172.21.15.52:6801/3513075805] conn(0x7f8d140b2880 0x7f8d140b4d90 unknown :-1 s=CLOSED pgs=202 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:24:39.697 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:39.707+0000 7f8d2d1a4700 1 --2- 172.21.15.52:0/1902117694 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f8d281045d0 0x7f8d2819adb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:24:39.697 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:39.707+0000 7f8d2d1a4700 1 --2- 172.21.15.52:0/1902117694 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f8d28108f00 0x7f8d2819b350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:24:39.697 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:39.707+0000 7f8d2d1a4700 1 -- 172.21.15.52:0/1902117694 >> 172.21.15.52:0/1902117694 conn(0x7f8d280ffee0 msgr2=0x7f8d28108670 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:24:39.698 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:39.707+0000 7f8d2d1a4700 1 -- 172.21.15.52:0/1902117694 shutdown_connections 2024-05-27T07:24:39.698 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:39.707+0000 7f8d2d1a4700 1 -- 172.21.15.52:0/1902117694 wait complete. 2024-05-27T07:24:39.979 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:24:39 smithi052 ceph-mon[31955]: from='client.? 172.21.15.52:0/1902117694' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-05-27T07:24:40.102 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:24:39 smithi173 ceph-mon[37000]: from='client.? 172.21.15.52:0/1902117694' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-05-27T07:24:40.189 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:24:39 smithi105 ceph-mon[35977]: from='client.? 172.21.15.52:0/1902117694' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-05-27T07:24:40.555 INFO:tasks.cephadm.ceph_manager.ceph:wait_until_healthy done 2024-05-27T07:24:40.556 DEBUG:teuthology.orchestra.run.smithi052:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:38f9d900c83c63799fbdbe61acc9a11b0d3554a6 shell --fsid 73df2636-1bf8-11ef-bc9b-c7b262605968 -- ceph fs dump --format=json 2024-05-27T07:24:40.814 INFO:teuthology.orchestra.run.smithi052.stderr:Inferring config /var/lib/ceph/73df2636-1bf8-11ef-bc9b-c7b262605968/mon.a/config 2024-05-27T07:24:40.936 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:24:40 smithi052 ceph-mon[31955]: pgmap v385: 129 pgs: 129 active+clean; 647 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 5.7 KiB/s rd, 597 B/s wr, 10 op/s 2024-05-27T07:24:41.101 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:24:40 smithi173 ceph-mon[37000]: pgmap v385: 129 pgs: 129 active+clean; 647 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 5.7 KiB/s rd, 597 B/s wr, 10 op/s 2024-05-27T07:24:41.190 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:24:40 smithi105 ceph-mon[35977]: pgmap v385: 129 pgs: 129 active+clean; 647 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 5.7 KiB/s rd, 597 B/s wr, 10 op/s 2024-05-27T07:24:42.612 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:42.621+0000 7fa64ff5d700 1 -- 172.21.15.52:0/393452907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fa648108730 msgr2=0x7fa648110090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:24:42.612 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:42.621+0000 7fa64ff5d700 1 --2- 172.21.15.52:0/393452907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fa648108730 0x7fa648110090 secure :-1 s=READY pgs=302 cs=0 l=1 rev1=1 crypto rx=0x7fa6400099c0 tx=0x7fa640009cd0 comp rx=0 tx=0).stop 2024-05-27T07:24:42.612 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:42.621+0000 7fa64ff5d700 1 -- 172.21.15.52:0/393452907 shutdown_connections 2024-05-27T07:24:42.612 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:42.621+0000 7fa64ff5d700 1 --2- 172.21.15.52:0/393452907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fa648108730 0x7fa648110090 unknown :-1 s=CLOSED pgs=302 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:24:42.612 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:42.621+0000 7fa64ff5d700 1 --2- 172.21.15.52:0/393452907 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fa6481034b0 0x7fa6481038c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:24:42.613 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:42.621+0000 7fa64ff5d700 1 --2- 172.21.15.52:0/393452907 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fa648107de0 0x7fa6481081f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:24:42.613 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:42.621+0000 7fa64ff5d700 1 -- 172.21.15.52:0/393452907 >> 172.21.15.52:0/393452907 conn(0x7fa6480fede0 msgr2=0x7fa648101200 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:24:42.613 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:42.622+0000 7fa64ff5d700 1 -- 172.21.15.52:0/393452907 shutdown_connections 2024-05-27T07:24:42.613 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:42.622+0000 7fa64ff5d700 1 -- 172.21.15.52:0/393452907 wait complete. 2024-05-27T07:24:42.613 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:42.622+0000 7fa64ff5d700 1 Processor -- start 2024-05-27T07:24:42.613 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:42.622+0000 7fa64ff5d700 1 -- start start 2024-05-27T07:24:42.614 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:42.623+0000 7fa64ff5d700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fa6481034b0 0x7fa64819bbe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:24:42.614 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:42.623+0000 7fa64ff5d700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fa648107de0 0x7fa64819c120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:24:42.614 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:42.623+0000 7fa64ff5d700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fa648108730 0x7fa64819c6a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:24:42.614 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:42.623+0000 7fa64ff5d700 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x7fa648113380 con 0x7fa648107de0 2024-05-27T07:24:42.614 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:42.623+0000 7fa64ff5d700 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7fa648113200 con 0x7fa648108730 2024-05-27T07:24:42.614 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:42.623+0000 7fa64ff5d700 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7fa648113500 con 0x7fa6481034b0 2024-05-27T07:24:42.614 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:42.623+0000 7fa64d4f8700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fa648107de0 0x7fa64819c120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:24:42.615 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:42.623+0000 7fa64d4f8700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fa648107de0 0x7fa64819c120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.52:41478/0 (socket says 172.21.15.52:41478) 2024-05-27T07:24:42.615 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:42.623+0000 7fa64d4f8700 1 -- 172.21.15.52:0/1684230279 learned_addr learned my addr 172.21.15.52:0/1684230279 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-05-27T07:24:42.615 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:42.623+0000 7fa64e4fa700 1 --2- 172.21.15.52:0/1684230279 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fa648108730 0x7fa64819c6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:24:42.615 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:42.623+0000 7fa64dcf9700 1 --2- 172.21.15.52:0/1684230279 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fa6481034b0 0x7fa64819bbe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:24:42.615 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:42.623+0000 7fa64d4f8700 1 -- 172.21.15.52:0/1684230279 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fa6481034b0 msgr2=0x7fa64819bbe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:24:42.615 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:42.623+0000 7fa64d4f8700 1 --2- 172.21.15.52:0/1684230279 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fa6481034b0 0x7fa64819bbe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:24:42.615 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:42.623+0000 7fa64d4f8700 1 -- 172.21.15.52:0/1684230279 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fa648108730 msgr2=0x7fa64819c6a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:24:42.615 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:42.623+0000 7fa64d4f8700 1 --2- 172.21.15.52:0/1684230279 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fa648108730 0x7fa64819c6a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:24:42.615 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:42.623+0000 7fa64d4f8700 1 -- 172.21.15.52:0/1684230279 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa6481a6f50 con 0x7fa648107de0 2024-05-27T07:24:42.615 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:42.623+0000 7fa64d4f8700 1 --2- 172.21.15.52:0/1684230279 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fa648107de0 0x7fa64819c120 secure :-1 s=READY pgs=303 cs=0 l=1 rev1=1 crypto rx=0x7fa63800d980 tx=0x7fa63800dc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:24:42.615 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:42.624+0000 7fa63effd700 1 -- 172.21.15.52:0/1684230279 <== mon.0 v2:172.21.15.52:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa638017070 con 0x7fa648107de0 2024-05-27T07:24:42.615 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:42.624+0000 7fa63effd700 1 -- 172.21.15.52:0/1684230279 <== mon.0 v2:172.21.15.52:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fa638009de0 con 0x7fa648107de0 2024-05-27T07:24:42.616 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:42.624+0000 7fa64ff5d700 1 -- 172.21.15.52:0/1684230279 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa6481a7240 con 0x7fa648107de0 2024-05-27T07:24:42.616 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:42.624+0000 7fa63effd700 1 -- 172.21.15.52:0/1684230279 <== mon.0 v2:172.21.15.52:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa63800f3c0 con 0x7fa648107de0 2024-05-27T07:24:42.616 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:42.624+0000 7fa64ff5d700 1 -- 172.21.15.52:0/1684230279 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa6481a7780 con 0x7fa648107de0 2024-05-27T07:24:42.616 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:42.625+0000 7fa64ff5d700 1 -- 172.21.15.52:0/1684230279 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa6480512b0 con 0x7fa648107de0 2024-05-27T07:24:42.616 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:42.627+0000 7fa63effd700 1 -- 172.21.15.52:0/1684230279 <== mon.0 v2:172.21.15.52:3300/0 4 ==== mgrmap(e 18) ==== 149652+0+0 (secure 0 0 0) 0x7fa638009930 con 0x7fa648107de0 2024-05-27T07:24:42.617 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:42.627+0000 7fa63effd700 1 --2- 172.21.15.52:0/1684230279 >> [v2:172.21.15.52:6800/3513075805,v1:172.21.15.52:6801/3513075805] conn(0x7fa6340b2880 0x7fa6340b4d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:24:42.617 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:42.627+0000 7fa64dcf9700 1 --2- 172.21.15.52:0/1684230279 >> [v2:172.21.15.52:6800/3513075805,v1:172.21.15.52:6801/3513075805] conn(0x7fa6340b2880 0x7fa6340b4d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:24:42.617 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:42.627+0000 7fa63effd700 1 -- 172.21.15.52:0/1684230279 <== mon.0 v2:172.21.15.52:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9121+0+0 (secure 0 0 0) 0x7fa6380e1fd0 con 0x7fa648107de0 2024-05-27T07:24:42.618 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:42.628+0000 7fa64dcf9700 1 --2- 172.21.15.52:0/1684230279 >> [v2:172.21.15.52:6800/3513075805,v1:172.21.15.52:6801/3513075805] conn(0x7fa6340b2880 0x7fa6340b4d90 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7fa644009d10 tx=0x7fa644009040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:24:42.620 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:42.631+0000 7fa63effd700 1 -- 172.21.15.52:0/1684230279 <== mon.0 v2:172.21.15.52:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191963 (secure 0 0 0) 0x7fa6380a9970 con 0x7fa648107de0 2024-05-27T07:24:42.880 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:42.889+0000 7fa64ff5d700 1 -- 172.21.15.52:0/1684230279 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fa64802e5e0 con 0x7fa648107de0 2024-05-27T07:24:42.880 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:24:42 smithi052 ceph-mon[31955]: pgmap v386: 129 pgs: 129 active+clean; 647 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 341 B/s wr, 10 op/s 2024-05-27T07:24:42.880 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:42.890+0000 7fa63effd700 1 -- 172.21.15.52:0/1684230279 <== mon.0 v2:172.21.15.52:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 29 v29) ==== 94+0+11938 (secure 0 0 0) 0x7fa6380a9270 con 0x7fa648107de0 2024-05-27T07:24:42.881 INFO:teuthology.orchestra.run.smithi052.stdout: 2024-05-27T07:24:42.881 INFO:teuthology.orchestra.run.smithi052.stdout:{"epoch":29,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14556,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.52:6835/2584733778","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.52:6834","nonce":2584733778},{"type":"v1","addr":"172.21.15.52:6835","nonce":2584733778}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14568,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.52:6837/675445303","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.52:6836","nonce":675445303},{"type":"v1","addr":"172.21.15.52:6837","nonce":675445303}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":29,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-05-27T07:21:50.521783+0000","modified":"2024-05-27T07:22:58.822630+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24527,"mds_1":24497,"mds_2":24475,"mds_3":24473,"mds_4":24466},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14619":{"gid":14619,"name":"g","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.52:6839/4258549559","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.52:6838","nonce":4258549559},{"type":"v1","addr":"172.21.15.52:6839","nonce":4258549559}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_14631":{"gid":14631,"name":"a","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.52:6841/2878618188","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.52:6840","nonce":2878618188},{"type":"v1","addr":"172.21.15.52:6841","nonce":2878618188}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24433":{"gid":24433,"name":"l","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.173:6833/3013589220","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6832","nonce":3013589220},{"type":"v1","addr":"172.21.15.173:6833","nonce":3013589220}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24452":{"gid":24452,"name":"k","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.105:6833/2800678412","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6832","nonce":2800678412},{"type":"v1","addr":"172.21.15.105:6833","nonce":2800678412}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24460":{"gid":24460,"name":"f","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.173:6835/1619486437","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6834","nonce":1619486437},{"type":"v1","addr":"172.21.15.173:6835","nonce":1619486437}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24466":{"gid":24466,"name":"c","rank":4,"incarnation":26,"state":"up:active","state_seq":25,"addr":"172.21.15.173:6837/1396363827","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6836","nonce":1396363827},{"type":"v1","addr":"172.21.15.173:6837","nonce":1396363827}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24473":{"gid":24473,"name":"e","rank":3,"incarnation":24,"state":"up:active","state_seq":26,"addr":"172.21.15.105:6835/3298498384","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6834","nonce":3298498384},{"type":"v1","addr":"172.21.15.105:6835","nonce":3298498384}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24475":{"gid":24475,"name":"i","rank":2,"incarnation":22,"state":"up:active","state_seq":23,"addr":"172.21.15.173:6839/564778293","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6838","nonce":564778293},{"type":"v1","addr":"172.21.15.173:6839","nonce":564778293}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24497":{"gid":24497,"name":"h","rank":1,"incarnation":20,"state":"up:active","state_seq":24,"addr":"172.21.15.105:6837/191393149","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6836","nonce":191393149},{"type":"v1","addr":"172.21.15.105:6837","nonce":191393149}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24527":{"gid":24527,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":5,"addr":"172.21.15.105:6839/500348151","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6838","nonce":500348151},{"type":"v1","addr":"172.21.15.105:6839","nonce":500348151}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24527,"qdb_cluster":[24466,24473,24475,24497,24527]},"id":1}]} 2024-05-27T07:24:42.883 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:42.893+0000 7fa64ff5d700 1 -- 172.21.15.52:0/1684230279 >> [v2:172.21.15.52:6800/3513075805,v1:172.21.15.52:6801/3513075805] conn(0x7fa6340b2880 msgr2=0x7fa6340b4d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:24:42.883 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:42.893+0000 7fa64ff5d700 1 --2- 172.21.15.52:0/1684230279 >> [v2:172.21.15.52:6800/3513075805,v1:172.21.15.52:6801/3513075805] conn(0x7fa6340b2880 0x7fa6340b4d90 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7fa644009d10 tx=0x7fa644009040 comp rx=0 tx=0).stop 2024-05-27T07:24:42.883 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:42.893+0000 7fa64ff5d700 1 -- 172.21.15.52:0/1684230279 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fa648107de0 msgr2=0x7fa64819c120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:24:42.883 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:42.893+0000 7fa64ff5d700 1 --2- 172.21.15.52:0/1684230279 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fa648107de0 0x7fa64819c120 secure :-1 s=READY pgs=303 cs=0 l=1 rev1=1 crypto rx=0x7fa63800d980 tx=0x7fa63800dc90 comp rx=0 tx=0).stop 2024-05-27T07:24:42.883 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:42.893+0000 7fa64ff5d700 1 -- 172.21.15.52:0/1684230279 shutdown_connections 2024-05-27T07:24:42.883 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:42.893+0000 7fa64ff5d700 1 --2- 172.21.15.52:0/1684230279 >> [v2:172.21.15.52:6800/3513075805,v1:172.21.15.52:6801/3513075805] conn(0x7fa6340b2880 0x7fa6340b4d90 unknown :-1 s=CLOSED pgs=203 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:24:42.883 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:42.893+0000 7fa64ff5d700 1 --2- 172.21.15.52:0/1684230279 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fa648108730 0x7fa64819c6a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:24:42.883 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:42.893+0000 7fa64ff5d700 1 --2- 172.21.15.52:0/1684230279 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fa6481034b0 0x7fa64819bbe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:24:42.884 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:42.893+0000 7fa64ff5d700 1 --2- 172.21.15.52:0/1684230279 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fa648107de0 0x7fa64819c120 unknown :-1 s=CLOSED pgs=303 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:24:42.884 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:42.893+0000 7fa64ff5d700 1 -- 172.21.15.52:0/1684230279 >> 172.21.15.52:0/1684230279 conn(0x7fa6480fede0 msgr2=0x7fa648107520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:24:42.884 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:42.893+0000 7fa64ff5d700 1 -- 172.21.15.52:0/1684230279 shutdown_connections 2024-05-27T07:24:42.884 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:42.893+0000 7fa64ff5d700 1 -- 172.21.15.52:0/1684230279 wait complete. 2024-05-27T07:24:42.884 INFO:teuthology.orchestra.run.smithi052.stderr:dumped fsmap epoch 29 2024-05-27T07:24:43.102 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:24:42 smithi173 ceph-mon[37000]: pgmap v386: 129 pgs: 129 active+clean; 647 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 341 B/s wr, 10 op/s 2024-05-27T07:24:43.190 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:24:42 smithi105 ceph-mon[35977]: pgmap v386: 129 pgs: 129 active+clean; 647 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 341 B/s wr, 10 op/s 2024-05-27T07:24:43.580 DEBUG:teuthology.orchestra.run.smithi052:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:38f9d900c83c63799fbdbe61acc9a11b0d3554a6 shell --fsid 73df2636-1bf8-11ef-bc9b-c7b262605968 -- ceph fs dump --format=json 2024-05-27T07:24:43.833 INFO:teuthology.orchestra.run.smithi052.stderr:Inferring config /var/lib/ceph/73df2636-1bf8-11ef-bc9b-c7b262605968/mon.a/config 2024-05-27T07:24:43.956 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:24:43 smithi052 ceph-mon[31955]: from='client.? 172.21.15.52:0/1684230279' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-27T07:24:44.004 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:24:43 smithi173 ceph-mon[37000]: from='client.? 172.21.15.52:0/1684230279' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-27T07:24:44.190 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:24:43 smithi105 ceph-mon[35977]: from='client.? 172.21.15.52:0/1684230279' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-27T07:24:44.979 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:24:44 smithi052 ceph-mon[31955]: pgmap v387: 129 pgs: 129 active+clean; 647 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 85 B/s wr, 9 op/s 2024-05-27T07:24:45.091 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:24:44 smithi105 ceph-mon[35977]: pgmap v387: 129 pgs: 129 active+clean; 647 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 85 B/s wr, 9 op/s 2024-05-27T07:24:45.101 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:24:44 smithi173 ceph-mon[37000]: pgmap v387: 129 pgs: 129 active+clean; 647 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 85 B/s wr, 9 op/s 2024-05-27T07:24:45.679 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:45.689+0000 7fa51dbf6700 1 -- 172.21.15.52:0/1044867669 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fa518108ef0 msgr2=0x7fa518109300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:24:45.680 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:45.689+0000 7fa51dbf6700 1 --2- 172.21.15.52:0/1044867669 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fa518108ef0 0x7fa518109300 secure :-1 s=READY pgs=304 cs=0 l=1 rev1=1 crypto rx=0x7fa50c0099c0 tx=0x7fa50c009cd0 comp rx=0 tx=0).stop 2024-05-27T07:24:45.680 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:45.689+0000 7fa51dbf6700 1 -- 172.21.15.52:0/1044867669 shutdown_connections 2024-05-27T07:24:45.680 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:45.689+0000 7fa51dbf6700 1 --2- 172.21.15.52:0/1044867669 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fa518109840 0x7fa5181111a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:24:45.680 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:45.689+0000 7fa51dbf6700 1 --2- 172.21.15.52:0/1044867669 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fa5181045c0 0x7fa5181049d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:24:45.680 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:45.689+0000 7fa51dbf6700 1 --2- 172.21.15.52:0/1044867669 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fa518108ef0 0x7fa518109300 unknown :-1 s=CLOSED pgs=304 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:24:45.680 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:45.689+0000 7fa51dbf6700 1 -- 172.21.15.52:0/1044867669 >> 172.21.15.52:0/1044867669 conn(0x7fa5180ffe90 msgr2=0x7fa5181022e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:24:45.680 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:45.689+0000 7fa51dbf6700 1 -- 172.21.15.52:0/1044867669 shutdown_connections 2024-05-27T07:24:45.680 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:45.690+0000 7fa51dbf6700 1 -- 172.21.15.52:0/1044867669 wait complete. 2024-05-27T07:24:45.682 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:45.690+0000 7fa51dbf6700 1 Processor -- start 2024-05-27T07:24:45.682 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:45.691+0000 7fa51dbf6700 1 -- start start 2024-05-27T07:24:45.682 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:45.691+0000 7fa51dbf6700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fa5181045c0 0x7fa51819adf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:24:45.682 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:45.691+0000 7fa51dbf6700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fa518108ef0 0x7fa51819b390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:24:45.682 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:45.691+0000 7fa51dbf6700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fa518109840 0x7fa51819f620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:24:45.682 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:45.691+0000 7fa51dbf6700 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x7fa5181144a0 con 0x7fa518109840 2024-05-27T07:24:45.682 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:45.691+0000 7fa51dbf6700 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7fa518114320 con 0x7fa5181045c0 2024-05-27T07:24:45.683 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:45.691+0000 7fa51dbf6700 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7fa518114620 con 0x7fa518108ef0 2024-05-27T07:24:45.683 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:45.691+0000 7fa517fff700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fa518109840 0x7fa51819f620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:24:45.683 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:45.691+0000 7fa517fff700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fa518109840 0x7fa51819f620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.52:41486/0 (socket says 172.21.15.52:41486) 2024-05-27T07:24:45.683 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:45.691+0000 7fa517fff700 1 -- 172.21.15.52:0/2474516205 learned_addr learned my addr 172.21.15.52:0/2474516205 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-05-27T07:24:45.683 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:45.691+0000 7fa516ffd700 1 --2- 172.21.15.52:0/2474516205 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fa518108ef0 0x7fa51819b390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:24:45.683 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:45.691+0000 7fa5177fe700 1 --2- 172.21.15.52:0/2474516205 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fa5181045c0 0x7fa51819adf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:24:45.683 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:45.691+0000 7fa517fff700 1 -- 172.21.15.52:0/2474516205 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fa518108ef0 msgr2=0x7fa51819b390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:24:45.683 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:45.691+0000 7fa517fff700 1 --2- 172.21.15.52:0/2474516205 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fa518108ef0 0x7fa51819b390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:24:45.683 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:45.691+0000 7fa517fff700 1 -- 172.21.15.52:0/2474516205 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fa5181045c0 msgr2=0x7fa51819adf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:24:45.683 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:45.691+0000 7fa517fff700 1 --2- 172.21.15.52:0/2474516205 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fa5181045c0 0x7fa51819adf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:24:45.683 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:45.691+0000 7fa517fff700 1 -- 172.21.15.52:0/2474516205 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa51819fc10 con 0x7fa518109840 2024-05-27T07:24:45.683 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:45.691+0000 7fa516ffd700 1 --2- 172.21.15.52:0/2474516205 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fa518108ef0 0x7fa51819b390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-27T07:24:45.684 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:45.691+0000 7fa5177fe700 1 --2- 172.21.15.52:0/2474516205 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fa5181045c0 0x7fa51819adf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-27T07:24:45.684 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:45.692+0000 7fa517fff700 1 --2- 172.21.15.52:0/2474516205 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fa518109840 0x7fa51819f620 secure :-1 s=READY pgs=305 cs=0 l=1 rev1=1 crypto rx=0x7fa510007ee0 tx=0x7fa51000d3b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:24:45.684 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:45.692+0000 7fa514ff9700 1 -- 172.21.15.52:0/2474516205 <== mon.0 v2:172.21.15.52:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa51000dbd0 con 0x7fa518109840 2024-05-27T07:24:45.684 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:45.692+0000 7fa514ff9700 1 -- 172.21.15.52:0/2474516205 <== mon.0 v2:172.21.15.52:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fa51000dd70 con 0x7fa518109840 2024-05-27T07:24:45.684 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:45.692+0000 7fa51dbf6700 1 -- 172.21.15.52:0/2474516205 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa51819ff00 con 0x7fa518109840 2024-05-27T07:24:45.684 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:45.692+0000 7fa514ff9700 1 -- 172.21.15.52:0/2474516205 <== mon.0 v2:172.21.15.52:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa510005990 con 0x7fa518109840 2024-05-27T07:24:45.684 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:45.692+0000 7fa51dbf6700 1 -- 172.21.15.52:0/2474516205 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa5181a03c0 con 0x7fa518109840 2024-05-27T07:24:45.684 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:45.693+0000 7fa51dbf6700 1 -- 172.21.15.52:0/2474516205 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa5180512b0 con 0x7fa518109840 2024-05-27T07:24:45.685 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:45.695+0000 7fa514ff9700 1 -- 172.21.15.52:0/2474516205 <== mon.0 v2:172.21.15.52:3300/0 4 ==== mgrmap(e 18) ==== 149652+0+0 (secure 0 0 0) 0x7fa51001a020 con 0x7fa518109840 2024-05-27T07:24:45.685 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:45.695+0000 7fa514ff9700 1 --2- 172.21.15.52:0/2474516205 >> [v2:172.21.15.52:6800/3513075805,v1:172.21.15.52:6801/3513075805] conn(0x7fa5000b2880 0x7fa5000b4d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:24:45.686 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:45.696+0000 7fa5177fe700 1 --2- 172.21.15.52:0/2474516205 >> [v2:172.21.15.52:6800/3513075805,v1:172.21.15.52:6801/3513075805] conn(0x7fa5000b2880 0x7fa5000b4d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:24:45.686 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:45.696+0000 7fa514ff9700 1 -- 172.21.15.52:0/2474516205 <== mon.0 v2:172.21.15.52:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9121+0+0 (secure 0 0 0) 0x7fa5100e2370 con 0x7fa518109840 2024-05-27T07:24:45.686 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:45.696+0000 7fa5177fe700 1 --2- 172.21.15.52:0/2474516205 >> [v2:172.21.15.52:6800/3513075805,v1:172.21.15.52:6801/3513075805] conn(0x7fa5000b2880 0x7fa5000b4d90 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7fa508005fd0 tx=0x7fa508005e30 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:24:45.689 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:45.699+0000 7fa514ff9700 1 -- 172.21.15.52:0/2474516205 <== mon.0 v2:172.21.15.52:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191963 (secure 0 0 0) 0x7fa510010070 con 0x7fa518109840 2024-05-27T07:24:45.951 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:45.961+0000 7fa51dbf6700 1 -- 172.21.15.52:0/2474516205 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fa51802e5e0 con 0x7fa518109840 2024-05-27T07:24:45.951 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:45.961+0000 7fa514ff9700 1 -- 172.21.15.52:0/2474516205 <== mon.0 v2:172.21.15.52:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 29 v29) ==== 94+0+11938 (secure 0 0 0) 0x7fa5100a9db0 con 0x7fa518109840 2024-05-27T07:24:45.952 INFO:teuthology.orchestra.run.smithi052.stdout: 2024-05-27T07:24:45.952 INFO:teuthology.orchestra.run.smithi052.stdout:{"epoch":29,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14556,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.52:6835/2584733778","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.52:6834","nonce":2584733778},{"type":"v1","addr":"172.21.15.52:6835","nonce":2584733778}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14568,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.52:6837/675445303","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.52:6836","nonce":675445303},{"type":"v1","addr":"172.21.15.52:6837","nonce":675445303}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":29,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-05-27T07:21:50.521783+0000","modified":"2024-05-27T07:22:58.822630+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24527,"mds_1":24497,"mds_2":24475,"mds_3":24473,"mds_4":24466},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14619":{"gid":14619,"name":"g","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.52:6839/4258549559","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.52:6838","nonce":4258549559},{"type":"v1","addr":"172.21.15.52:6839","nonce":4258549559}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_14631":{"gid":14631,"name":"a","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.52:6841/2878618188","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.52:6840","nonce":2878618188},{"type":"v1","addr":"172.21.15.52:6841","nonce":2878618188}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24433":{"gid":24433,"name":"l","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.173:6833/3013589220","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6832","nonce":3013589220},{"type":"v1","addr":"172.21.15.173:6833","nonce":3013589220}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24452":{"gid":24452,"name":"k","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.105:6833/2800678412","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6832","nonce":2800678412},{"type":"v1","addr":"172.21.15.105:6833","nonce":2800678412}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24460":{"gid":24460,"name":"f","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.173:6835/1619486437","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6834","nonce":1619486437},{"type":"v1","addr":"172.21.15.173:6835","nonce":1619486437}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24466":{"gid":24466,"name":"c","rank":4,"incarnation":26,"state":"up:active","state_seq":25,"addr":"172.21.15.173:6837/1396363827","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6836","nonce":1396363827},{"type":"v1","addr":"172.21.15.173:6837","nonce":1396363827}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24473":{"gid":24473,"name":"e","rank":3,"incarnation":24,"state":"up:active","state_seq":26,"addr":"172.21.15.105:6835/3298498384","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6834","nonce":3298498384},{"type":"v1","addr":"172.21.15.105:6835","nonce":3298498384}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24475":{"gid":24475,"name":"i","rank":2,"incarnation":22,"state":"up:active","state_seq":23,"addr":"172.21.15.173:6839/564778293","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6838","nonce":564778293},{"type":"v1","addr":"172.21.15.173:6839","nonce":564778293}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24497":{"gid":24497,"name":"h","rank":1,"incarnation":20,"state":"up:active","state_seq":24,"addr":"172.21.15.105:6837/191393149","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6836","nonce":191393149},{"type":"v1","addr":"172.21.15.105:6837","nonce":191393149}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24527":{"gid":24527,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":5,"addr":"172.21.15.105:6839/500348151","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6838","nonce":500348151},{"type":"v1","addr":"172.21.15.105:6839","nonce":500348151}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24527,"qdb_cluster":[24466,24473,24475,24497,24527]},"id":1}]} 2024-05-27T07:24:45.954 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:45.964+0000 7fa51dbf6700 1 -- 172.21.15.52:0/2474516205 >> [v2:172.21.15.52:6800/3513075805,v1:172.21.15.52:6801/3513075805] conn(0x7fa5000b2880 msgr2=0x7fa5000b4d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:24:45.954 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:45.964+0000 7fa51dbf6700 1 --2- 172.21.15.52:0/2474516205 >> [v2:172.21.15.52:6800/3513075805,v1:172.21.15.52:6801/3513075805] conn(0x7fa5000b2880 0x7fa5000b4d90 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7fa508005fd0 tx=0x7fa508005e30 comp rx=0 tx=0).stop 2024-05-27T07:24:45.954 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:45.964+0000 7fa51dbf6700 1 -- 172.21.15.52:0/2474516205 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fa518109840 msgr2=0x7fa51819f620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:24:45.954 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:45.964+0000 7fa51dbf6700 1 --2- 172.21.15.52:0/2474516205 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fa518109840 0x7fa51819f620 secure :-1 s=READY pgs=305 cs=0 l=1 rev1=1 crypto rx=0x7fa510007ee0 tx=0x7fa51000d3b0 comp rx=0 tx=0).stop 2024-05-27T07:24:45.955 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:45.964+0000 7fa51dbf6700 1 -- 172.21.15.52:0/2474516205 shutdown_connections 2024-05-27T07:24:45.955 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:45.964+0000 7fa51dbf6700 1 --2- 172.21.15.52:0/2474516205 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fa518109840 0x7fa51819f620 unknown :-1 s=CLOSED pgs=305 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:24:45.955 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:45.964+0000 7fa51dbf6700 1 --2- 172.21.15.52:0/2474516205 >> [v2:172.21.15.52:6800/3513075805,v1:172.21.15.52:6801/3513075805] conn(0x7fa5000b2880 0x7fa5000b4d90 unknown :-1 s=CLOSED pgs=204 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:24:45.955 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:45.964+0000 7fa51dbf6700 1 --2- 172.21.15.52:0/2474516205 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fa5181045c0 0x7fa51819adf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:24:45.955 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:45.964+0000 7fa51dbf6700 1 --2- 172.21.15.52:0/2474516205 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fa518108ef0 0x7fa51819b390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:24:45.955 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:45.964+0000 7fa51dbf6700 1 -- 172.21.15.52:0/2474516205 >> 172.21.15.52:0/2474516205 conn(0x7fa5180ffe90 msgr2=0x7fa51810a7a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:24:45.955 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:45.964+0000 7fa51dbf6700 1 -- 172.21.15.52:0/2474516205 shutdown_connections 2024-05-27T07:24:45.955 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:45.964+0000 7fa51dbf6700 1 -- 172.21.15.52:0/2474516205 wait complete. 2024-05-27T07:24:45.956 INFO:teuthology.orchestra.run.smithi052.stderr:dumped fsmap epoch 29 2024-05-27T07:24:46.691 DEBUG:teuthology.orchestra.run.smithi052:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:38f9d900c83c63799fbdbe61acc9a11b0d3554a6 shell --fsid 73df2636-1bf8-11ef-bc9b-c7b262605968 -- ceph osd dump --format=json 2024-05-27T07:24:46.947 INFO:teuthology.orchestra.run.smithi052.stderr:Inferring config /var/lib/ceph/73df2636-1bf8-11ef-bc9b-c7b262605968/mon.a/config 2024-05-27T07:24:46.979 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:24:46 smithi052 ceph-mon[31955]: pgmap v388: 129 pgs: 129 active+clean; 647 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 5.9 KiB/s rd, 85 B/s wr, 11 op/s 2024-05-27T07:24:46.979 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:24:46 smithi052 ceph-mon[31955]: from='client.? 172.21.15.52:0/2474516205' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-27T07:24:47.102 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:24:46 smithi173 ceph-mon[37000]: pgmap v388: 129 pgs: 129 active+clean; 647 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 5.9 KiB/s rd, 85 B/s wr, 11 op/s 2024-05-27T07:24:47.102 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:24:46 smithi173 ceph-mon[37000]: from='client.? 172.21.15.52:0/2474516205' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-27T07:24:47.189 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:24:46 smithi105 ceph-mon[35977]: pgmap v388: 129 pgs: 129 active+clean; 647 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 5.9 KiB/s rd, 85 B/s wr, 11 op/s 2024-05-27T07:24:47.190 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:24:46 smithi105 ceph-mon[35977]: from='client.? 172.21.15.52:0/2474516205' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-27T07:24:48.566 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:48.571+0000 7fb6cbf17700 1 -- 172.21.15.52:0/3714130514 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fb6c4107de0 msgr2=0x7fb6c41081f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:24:48.566 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:48.571+0000 7fb6cbf17700 1 --2- 172.21.15.52:0/3714130514 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fb6c4107de0 0x7fb6c41081f0 secure :-1 s=READY pgs=306 cs=0 l=1 rev1=1 crypto rx=0x7fb6b40099c0 tx=0x7fb6b4009cd0 comp rx=0 tx=0).stop 2024-05-27T07:24:48.566 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:48.571+0000 7fb6cbf17700 1 -- 172.21.15.52:0/3714130514 shutdown_connections 2024-05-27T07:24:48.567 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:48.571+0000 7fb6cbf17700 1 --2- 172.21.15.52:0/3714130514 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fb6c4108730 0x7fb6c4110090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:24:48.567 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:48.571+0000 7fb6cbf17700 1 --2- 172.21.15.52:0/3714130514 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fb6c41034b0 0x7fb6c41038c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:24:48.567 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:48.571+0000 7fb6cbf17700 1 --2- 172.21.15.52:0/3714130514 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fb6c4107de0 0x7fb6c41081f0 unknown :-1 s=CLOSED pgs=306 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:24:48.567 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:48.571+0000 7fb6cbf17700 1 -- 172.21.15.52:0/3714130514 >> 172.21.15.52:0/3714130514 conn(0x7fb6c40fede0 msgr2=0x7fb6c4101200 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:24:48.567 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:48.571+0000 7fb6cbf17700 1 -- 172.21.15.52:0/3714130514 shutdown_connections 2024-05-27T07:24:48.567 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:48.571+0000 7fb6cbf17700 1 -- 172.21.15.52:0/3714130514 wait complete. 2024-05-27T07:24:48.567 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:48.572+0000 7fb6cbf17700 1 Processor -- start 2024-05-27T07:24:48.567 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:48.572+0000 7fb6cbf17700 1 -- start start 2024-05-27T07:24:48.567 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:48.572+0000 7fb6cbf17700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fb6c41034b0 0x7fb6c4199d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:24:48.567 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:48.572+0000 7fb6cbf17700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fb6c4107de0 0x7fb6c419a2e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:24:48.567 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:48.572+0000 7fb6cbf17700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fb6c4108730 0x7fb6c419e570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:24:48.567 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:48.572+0000 7fb6cbf17700 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x7fb6c4113390 con 0x7fb6c4108730 2024-05-27T07:24:48.567 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:48.572+0000 7fb6cbf17700 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7fb6c4113210 con 0x7fb6c41034b0 2024-05-27T07:24:48.568 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:48.572+0000 7fb6cbf17700 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7fb6c4113510 con 0x7fb6c4107de0 2024-05-27T07:24:48.568 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:48.573+0000 7fb6ca4b4700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fb6c4108730 0x7fb6c419e570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:24:48.568 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:48.573+0000 7fb6c94b2700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fb6c4107de0 0x7fb6c419a2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:24:48.568 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:48.573+0000 7fb6ca4b4700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fb6c4108730 0x7fb6c419e570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.52:41514/0 (socket says 172.21.15.52:41514) 2024-05-27T07:24:48.568 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:48.573+0000 7fb6ca4b4700 1 -- 172.21.15.52:0/3565964882 learned_addr learned my addr 172.21.15.52:0/3565964882 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-05-27T07:24:48.568 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:48.573+0000 7fb6c9cb3700 1 --2- 172.21.15.52:0/3565964882 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fb6c41034b0 0x7fb6c4199d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:24:48.568 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:48.573+0000 7fb6ca4b4700 1 -- 172.21.15.52:0/3565964882 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fb6c4107de0 msgr2=0x7fb6c419a2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:24:48.568 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:48.573+0000 7fb6ca4b4700 1 --2- 172.21.15.52:0/3565964882 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fb6c4107de0 0x7fb6c419a2e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:24:48.568 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:48.573+0000 7fb6ca4b4700 1 -- 172.21.15.52:0/3565964882 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fb6c41034b0 msgr2=0x7fb6c4199d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:24:48.568 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:48.573+0000 7fb6ca4b4700 1 --2- 172.21.15.52:0/3565964882 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fb6c41034b0 0x7fb6c4199d40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:24:48.568 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:48.573+0000 7fb6ca4b4700 1 -- 172.21.15.52:0/3565964882 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb6c419ec10 con 0x7fb6c4108730 2024-05-27T07:24:48.568 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:48.573+0000 7fb6ca4b4700 1 --2- 172.21.15.52:0/3565964882 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fb6c4108730 0x7fb6c419e570 secure :-1 s=READY pgs=307 cs=0 l=1 rev1=1 crypto rx=0x7fb6bc007ee0 tx=0x7fb6bc00d3b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:24:48.568 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:48.573+0000 7fb6baffd700 1 -- 172.21.15.52:0/3565964882 <== mon.0 v2:172.21.15.52:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb6bc004bb0 con 0x7fb6c4108730 2024-05-27T07:24:48.569 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:48.573+0000 7fb6baffd700 1 -- 172.21.15.52:0/3565964882 <== mon.0 v2:172.21.15.52:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fb6bc004d50 con 0x7fb6c4108730 2024-05-27T07:24:48.569 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:48.573+0000 7fb6cbf17700 1 -- 172.21.15.52:0/3565964882 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb6c419ef00 con 0x7fb6c4108730 2024-05-27T07:24:48.569 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:48.573+0000 7fb6baffd700 1 -- 172.21.15.52:0/3565964882 <== mon.0 v2:172.21.15.52:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb6bc005920 con 0x7fb6c4108730 2024-05-27T07:24:48.569 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:48.573+0000 7fb6cbf17700 1 -- 172.21.15.52:0/3565964882 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb6c41a6820 con 0x7fb6c4108730 2024-05-27T07:24:48.569 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:48.574+0000 7fb6cbf17700 1 -- 172.21.15.52:0/3565964882 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb6c4104960 con 0x7fb6c4108730 2024-05-27T07:24:48.569 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:48.576+0000 7fb6baffd700 1 -- 172.21.15.52:0/3565964882 <== mon.0 v2:172.21.15.52:3300/0 4 ==== mgrmap(e 18) ==== 149652+0+0 (secure 0 0 0) 0x7fb6bc01a020 con 0x7fb6c4108730 2024-05-27T07:24:48.569 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:48.577+0000 7fb6baffd700 1 --2- 172.21.15.52:0/3565964882 >> [v2:172.21.15.52:6800/3513075805,v1:172.21.15.52:6801/3513075805] conn(0x7fb6b00b2880 0x7fb6b00b4d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:24:48.570 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:48.577+0000 7fb6c9cb3700 1 --2- 172.21.15.52:0/3565964882 >> [v2:172.21.15.52:6800/3513075805,v1:172.21.15.52:6801/3513075805] conn(0x7fb6b00b2880 0x7fb6b00b4d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:24:48.570 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:48.577+0000 7fb6baffd700 1 -- 172.21.15.52:0/3565964882 <== mon.0 v2:172.21.15.52:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9121+0+0 (secure 0 0 0) 0x7fb6bc0e21a0 con 0x7fb6c4108730 2024-05-27T07:24:48.570 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:48.578+0000 7fb6c9cb3700 1 --2- 172.21.15.52:0/3565964882 >> [v2:172.21.15.52:6800/3513075805,v1:172.21.15.52:6801/3513075805] conn(0x7fb6b00b2880 0x7fb6b00b4d90 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7fb6c0009c60 tx=0x7fb6c0009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:24:48.570 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:48.580+0000 7fb6baffd700 1 -- 172.21.15.52:0/3565964882 <== mon.0 v2:172.21.15.52:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191963 (secure 0 0 0) 0x7fb6bc010070 con 0x7fb6c4108730 2024-05-27T07:24:48.785 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:48.795+0000 7fb6cbf17700 1 -- 172.21.15.52:0/3565964882 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fb6c402e5e0 con 0x7fb6c4108730 2024-05-27T07:24:48.787 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:48.797+0000 7fb6baffd700 1 -- 172.21.15.52:0/3565964882 <== mon.0 v2:172.21.15.52:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v90) ==== 74+0+24701 (secure 0 0 0) 0x7fb6bc0a9b40 con 0x7fb6c4108730 2024-05-27T07:24:48.787 INFO:teuthology.orchestra.run.smithi052.stdout: 2024-05-27T07:24:48.787 INFO:teuthology.orchestra.run.smithi052.stdout:{"epoch":90,"fsid":"73df2636-1bf8-11ef-bc9b-c7b262605968","created":"2024-05-27T07:13:34.738032+0000","modified":"2024-05-27T07:23:30.969990+0000","last_up_change":"2024-05-27T07:20:45.777597+0000","last_in_change":"2024-05-27T07:20:25.314823+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-05-27T07:16:57.644034+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-05-27T07:21:43.917748+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"79","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-05-27T07:21:47.341876+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"79","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-05-27T07:21:57.707611+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"86","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"347b8a89-9dbe-4848-83b0-fd0e9374f33d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":82,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.52:6802","nonce":1626508899},{"type":"v1","addr":"172.21.15.52:6803","nonce":1626508899}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.52:6804","nonce":1626508899},{"type":"v1","addr":"172.21.15.52:6805","nonce":1626508899}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.52:6808","nonce":1626508899},{"type":"v1","addr":"172.21.15.52:6809","nonce":1626508899}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.52:6806","nonce":1626508899},{"type":"v1","addr":"172.21.15.52:6807","nonce":1626508899}]},"public_addr":"172.21.15.52:6803/1626508899","cluster_addr":"172.21.15.52:6805/1626508899","heartbeat_back_addr":"172.21.15.52:6809/1626508899","heartbeat_front_addr":"172.21.15.52:6807/1626508899","state":["exists","up"]},{"osd":1,"uuid":"7a3623d4-e087-40cb-9336-d4a5c52203d4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6800","nonce":2900093195},{"type":"v1","addr":"172.21.15.105:6801","nonce":2900093195}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6802","nonce":2900093195},{"type":"v1","addr":"172.21.15.105:6803","nonce":2900093195}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6806","nonce":2900093195},{"type":"v1","addr":"172.21.15.105:6807","nonce":2900093195}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6804","nonce":2900093195},{"type":"v1","addr":"172.21.15.105:6805","nonce":2900093195}]},"public_addr":"172.21.15.105:6801/2900093195","cluster_addr":"172.21.15.105:6803/2900093195","heartbeat_back_addr":"172.21.15.105:6807/2900093195","heartbeat_front_addr":"172.21.15.105:6805/2900093195","state":["exists","up"]},{"osd":2,"uuid":"41a9df5b-882a-47da-bb26-7f4d26ccf6c1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6800","nonce":2251189314},{"type":"v1","addr":"172.21.15.173:6801","nonce":2251189314}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6802","nonce":2251189314},{"type":"v1","addr":"172.21.15.173:6803","nonce":2251189314}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6806","nonce":2251189314},{"type":"v1","addr":"172.21.15.173:6807","nonce":2251189314}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6804","nonce":2251189314},{"type":"v1","addr":"172.21.15.173:6805","nonce":2251189314}]},"public_addr":"172.21.15.173:6801/2251189314","cluster_addr":"172.21.15.173:6803/2251189314","heartbeat_back_addr":"172.21.15.173:6807/2251189314","heartbeat_front_addr":"172.21.15.173:6805/2251189314","state":["exists","up"]},{"osd":3,"uuid":"f1245439-42a5-46c5-a998-e0a78f9df3fb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":26,"up_thru":82,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.52:6810","nonce":57920794},{"type":"v1","addr":"172.21.15.52:6811","nonce":57920794}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.52:6812","nonce":57920794},{"type":"v1","addr":"172.21.15.52:6813","nonce":57920794}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.52:6816","nonce":57920794},{"type":"v1","addr":"172.21.15.52:6817","nonce":57920794}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.52:6814","nonce":57920794},{"type":"v1","addr":"172.21.15.52:6815","nonce":57920794}]},"public_addr":"172.21.15.52:6811/57920794","cluster_addr":"172.21.15.52:6813/57920794","heartbeat_back_addr":"172.21.15.52:6817/57920794","heartbeat_front_addr":"172.21.15.52:6815/57920794","state":["exists","up"]},{"osd":4,"uuid":"2592d1cc-1726-4359-83d8-a2e6314b54ec","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":32,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6808","nonce":3441134818},{"type":"v1","addr":"172.21.15.105:6809","nonce":3441134818}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6810","nonce":3441134818},{"type":"v1","addr":"172.21.15.105:6811","nonce":3441134818}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6814","nonce":3441134818},{"type":"v1","addr":"172.21.15.105:6815","nonce":3441134818}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6812","nonce":3441134818},{"type":"v1","addr":"172.21.15.105:6813","nonce":3441134818}]},"public_addr":"172.21.15.105:6809/3441134818","cluster_addr":"172.21.15.105:6811/3441134818","heartbeat_back_addr":"172.21.15.105:6815/3441134818","heartbeat_front_addr":"172.21.15.105:6813/3441134818","state":["exists","up"]},{"osd":5,"uuid":"27e98ee2-4a92-4972-b2a3-959d0ed90981","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":37,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6808","nonce":2989239729},{"type":"v1","addr":"172.21.15.173:6809","nonce":2989239729}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6810","nonce":2989239729},{"type":"v1","addr":"172.21.15.173:6811","nonce":2989239729}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6814","nonce":2989239729},{"type":"v1","addr":"172.21.15.173:6815","nonce":2989239729}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6812","nonce":2989239729},{"type":"v1","addr":"172.21.15.173:6813","nonce":2989239729}]},"public_addr":"172.21.15.173:6809/2989239729","cluster_addr":"172.21.15.173:6811/2989239729","heartbeat_back_addr":"172.21.15.173:6815/2989239729","heartbeat_front_addr":"172.21.15.173:6813/2989239729","state":["exists","up"]},{"osd":6,"uuid":"2b0ec0cb-f438-4b2c-8642-0778ed374e9a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":43,"up_thru":82,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.52:6818","nonce":4244423065},{"type":"v1","addr":"172.21.15.52:6819","nonce":4244423065}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.52:6820","nonce":4244423065},{"type":"v1","addr":"172.21.15.52:6821","nonce":4244423065}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.52:6824","nonce":4244423065},{"type":"v1","addr":"172.21.15.52:6825","nonce":4244423065}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.52:6822","nonce":4244423065},{"type":"v1","addr":"172.21.15.52:6823","nonce":4244423065}]},"public_addr":"172.21.15.52:6819/4244423065","cluster_addr":"172.21.15.52:6821/4244423065","heartbeat_back_addr":"172.21.15.52:6825/4244423065","heartbeat_front_addr":"172.21.15.52:6823/4244423065","state":["exists","up"]},{"osd":7,"uuid":"cb2a7a6b-8015-43ea-865a-be144e45c51a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":48,"up_thru":82,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6816","nonce":2063611552},{"type":"v1","addr":"172.21.15.105:6817","nonce":2063611552}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6818","nonce":2063611552},{"type":"v1","addr":"172.21.15.105:6819","nonce":2063611552}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6822","nonce":2063611552},{"type":"v1","addr":"172.21.15.105:6823","nonce":2063611552}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6820","nonce":2063611552},{"type":"v1","addr":"172.21.15.105:6821","nonce":2063611552}]},"public_addr":"172.21.15.105:6817/2063611552","cluster_addr":"172.21.15.105:6819/2063611552","heartbeat_back_addr":"172.21.15.105:6823/2063611552","heartbeat_front_addr":"172.21.15.105:6821/2063611552","state":["exists","up"]},{"osd":8,"uuid":"c2fc565d-a0b9-4396-a209-bd884cddcfe3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":54,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6816","nonce":4131399852},{"type":"v1","addr":"172.21.15.173:6817","nonce":4131399852}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6818","nonce":4131399852},{"type":"v1","addr":"172.21.15.173:6819","nonce":4131399852}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6822","nonce":4131399852},{"type":"v1","addr":"172.21.15.173:6823","nonce":4131399852}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6820","nonce":4131399852},{"type":"v1","addr":"172.21.15.173:6821","nonce":4131399852}]},"public_addr":"172.21.15.173:6817/4131399852","cluster_addr":"172.21.15.173:6819/4131399852","heartbeat_back_addr":"172.21.15.173:6823/4131399852","heartbeat_front_addr":"172.21.15.173:6821/4131399852","state":["exists","up"]},{"osd":9,"uuid":"78fa7f17-1dad-4262-be40-31fa82e75520","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":60,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.52:6826","nonce":479119778},{"type":"v1","addr":"172.21.15.52:6827","nonce":479119778}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.52:6828","nonce":479119778},{"type":"v1","addr":"172.21.15.52:6829","nonce":479119778}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.52:6832","nonce":479119778},{"type":"v1","addr":"172.21.15.52:6833","nonce":479119778}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.52:6830","nonce":479119778},{"type":"v1","addr":"172.21.15.52:6831","nonce":479119778}]},"public_addr":"172.21.15.52:6827/479119778","cluster_addr":"172.21.15.52:6829/479119778","heartbeat_back_addr":"172.21.15.52:6833/479119778","heartbeat_front_addr":"172.21.15.52:6831/479119778","state":["exists","up"]},{"osd":10,"uuid":"e6f8e753-e15c-45b5-8ee9-d50f384559f9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":65,"up_thru":76,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6824","nonce":4032386628},{"type":"v1","addr":"172.21.15.105:6825","nonce":4032386628}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6826","nonce":4032386628},{"type":"v1","addr":"172.21.15.105:6827","nonce":4032386628}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6830","nonce":4032386628},{"type":"v1","addr":"172.21.15.105:6831","nonce":4032386628}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6828","nonce":4032386628},{"type":"v1","addr":"172.21.15.105:6829","nonce":4032386628}]},"public_addr":"172.21.15.105:6825/4032386628","cluster_addr":"172.21.15.105:6827/4032386628","heartbeat_back_addr":"172.21.15.105:6831/4032386628","heartbeat_front_addr":"172.21.15.105:6829/4032386628","state":["exists","up"]},{"osd":11,"uuid":"164572bf-67ed-4e8e-b32d-d7a2e8da607b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":71,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6824","nonce":1816308401},{"type":"v1","addr":"172.21.15.173:6825","nonce":1816308401}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6826","nonce":1816308401},{"type":"v1","addr":"172.21.15.173:6827","nonce":1816308401}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6830","nonce":1816308401},{"type":"v1","addr":"172.21.15.173:6831","nonce":1816308401}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6828","nonce":1816308401},{"type":"v1","addr":"172.21.15.173:6829","nonce":1816308401}]},"public_addr":"172.21.15.173:6825/1816308401","cluster_addr":"172.21.15.173:6827/1816308401","heartbeat_back_addr":"172.21.15.173:6831/1816308401","heartbeat_front_addr":"172.21.15.173:6829/1816308401","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-27T07:16:06.279220+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-27T07:16:29.668167+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-27T07:16:53.903808+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-27T07:17:18.787372+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-27T07:17:43.480784+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-27T07:18:08.816030+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-27T07:18:34.076148+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-27T07:19:00.283468+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-27T07:19:25.403023+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-27T07:19:52.139173+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-27T07:20:17.530304+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-27T07:20:43.665632+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.8","mappings":[{"from":2,"to":11}]},{"pgid":"2.f","mappings":[{"from":8,"to":11}]},{"pgid":"2.10","mappings":[{"from":2,"to":4}]},{"pgid":"3.0","mappings":[{"from":5,"to":9}]},{"pgid":"3.1b","mappings":[{"from":4,"to":8}]},{"pgid":"3.2c","mappings":[{"from":4,"to":9}]},{"pgid":"3.3c","mappings":[{"from":4,"to":9}]},{"pgid":"4.e","mappings":[{"from":9,"to":11}]},{"pgid":"4.12","mappings":[{"from":5,"to":10}]},{"pgid":"4.19","mappings":[{"from":9,"to":8}]},{"pgid":"4.1f","mappings":[{"from":9,"to":8}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.52:6801/1631683163":"2024-05-28T07:14:29.544633+0000","172.21.15.52:0/2776144659":"2024-05-28T07:14:29.544633+0000","172.21.15.52:6800/1631683163":"2024-05-28T07:14:29.544633+0000","172.21.15.52:0/3675529097":"2024-05-28T07:14:01.960550+0000","172.21.15.52:0/727808107":"2024-05-28T07:14:01.960550+0000","172.21.15.52:0/3839436833":"2024-05-28T07:14:29.544633+0000","172.21.15.52:6800/587662512":"2024-05-28T07:14:01.960550+0000","172.21.15.52:6801/587662512":"2024-05-28T07:14:01.960550+0000","172.21.15.52:0/853867465":"2024-05-28T07:14:01.960550+0000","172.21.15.52:0/3480918279":"2024-05-28T07:14:29.544633+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-05-27T07:24:48.790 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:48.800+0000 7fb6cbf17700 1 -- 172.21.15.52:0/3565964882 >> [v2:172.21.15.52:6800/3513075805,v1:172.21.15.52:6801/3513075805] conn(0x7fb6b00b2880 msgr2=0x7fb6b00b4d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:24:48.790 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:48.800+0000 7fb6cbf17700 1 --2- 172.21.15.52:0/3565964882 >> [v2:172.21.15.52:6800/3513075805,v1:172.21.15.52:6801/3513075805] conn(0x7fb6b00b2880 0x7fb6b00b4d90 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7fb6c0009c60 tx=0x7fb6c0009260 comp rx=0 tx=0).stop 2024-05-27T07:24:48.790 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:48.800+0000 7fb6cbf17700 1 -- 172.21.15.52:0/3565964882 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fb6c4108730 msgr2=0x7fb6c419e570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:24:48.790 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:48.800+0000 7fb6cbf17700 1 --2- 172.21.15.52:0/3565964882 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fb6c4108730 0x7fb6c419e570 secure :-1 s=READY pgs=307 cs=0 l=1 rev1=1 crypto rx=0x7fb6bc007ee0 tx=0x7fb6bc00d3b0 comp rx=0 tx=0).stop 2024-05-27T07:24:48.790 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:48.800+0000 7fb6cbf17700 1 -- 172.21.15.52:0/3565964882 shutdown_connections 2024-05-27T07:24:48.790 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:48.800+0000 7fb6cbf17700 1 --2- 172.21.15.52:0/3565964882 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fb6c4108730 0x7fb6c419e570 unknown :-1 s=CLOSED pgs=307 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:24:48.790 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:48.800+0000 7fb6cbf17700 1 --2- 172.21.15.52:0/3565964882 >> [v2:172.21.15.52:6800/3513075805,v1:172.21.15.52:6801/3513075805] conn(0x7fb6b00b2880 0x7fb6b00b4d90 unknown :-1 s=CLOSED pgs=205 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:24:48.790 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:48.800+0000 7fb6cbf17700 1 --2- 172.21.15.52:0/3565964882 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fb6c41034b0 0x7fb6c4199d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:24:48.791 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:48.800+0000 7fb6cbf17700 1 --2- 172.21.15.52:0/3565964882 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fb6c4107de0 0x7fb6c419a2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:24:48.791 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:48.800+0000 7fb6cbf17700 1 -- 172.21.15.52:0/3565964882 >> 172.21.15.52:0/3565964882 conn(0x7fb6c40fede0 msgr2=0x7fb6c4109690 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:24:48.791 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:48.800+0000 7fb6cbf17700 1 -- 172.21.15.52:0/3565964882 shutdown_connections 2024-05-27T07:24:48.791 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:48.800+0000 7fb6cbf17700 1 -- 172.21.15.52:0/3565964882 wait complete. 2024-05-27T07:24:48.979 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:24:48 smithi052 ceph-mon[31955]: pgmap v389: 129 pgs: 129 active+clean; 647 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-05-27T07:24:49.102 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:24:48 smithi173 ceph-mon[37000]: pgmap v389: 129 pgs: 129 active+clean; 647 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-05-27T07:24:49.190 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:24:48 smithi105 ceph-mon[35977]: pgmap v389: 129 pgs: 129 active+clean; 647 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-05-27T07:24:49.563 DEBUG:teuthology.orchestra.run.smithi052:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:38f9d900c83c63799fbdbe61acc9a11b0d3554a6 shell --fsid 73df2636-1bf8-11ef-bc9b-c7b262605968 -- ceph fs dump --format=json 2024-05-27T07:24:49.822 INFO:teuthology.orchestra.run.smithi052.stderr:Inferring config /var/lib/ceph/73df2636-1bf8-11ef-bc9b-c7b262605968/mon.a/config 2024-05-27T07:24:49.939 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:24:49 smithi052 ceph-mon[31955]: from='client.? 172.21.15.52:0/3565964882' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-05-27T07:24:50.101 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:24:49 smithi173 ceph-mon[37000]: from='client.? 172.21.15.52:0/3565964882' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-05-27T07:24:50.189 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:24:49 smithi105 ceph-mon[35977]: from='client.? 172.21.15.52:0/3565964882' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-05-27T07:24:50.979 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:24:50 smithi052 ceph-mon[31955]: pgmap v390: 129 pgs: 129 active+clean; 647 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 9 op/s 2024-05-27T07:24:51.101 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:24:50 smithi173 ceph-mon[37000]: pgmap v390: 129 pgs: 129 active+clean; 647 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 9 op/s 2024-05-27T07:24:51.190 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:24:50 smithi105 ceph-mon[35977]: pgmap v390: 129 pgs: 129 active+clean; 647 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 9 op/s 2024-05-27T07:24:51.529 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:51.539+0000 7fadec777700 1 -- 172.21.15.52:0/2700841235 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fade41034b0 msgr2=0x7fade41038c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:24:51.529 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:51.539+0000 7fadec777700 1 --2- 172.21.15.52:0/2700841235 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fade41034b0 0x7fade41038c0 secure :-1 s=READY pgs=308 cs=0 l=1 rev1=1 crypto rx=0x7fade000b0d0 tx=0x7fade000b3e0 comp rx=0 tx=0).stop 2024-05-27T07:24:51.529 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:51.539+0000 7fadec777700 1 -- 172.21.15.52:0/2700841235 shutdown_connections 2024-05-27T07:24:51.529 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:51.539+0000 7fadec777700 1 --2- 172.21.15.52:0/2700841235 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fade4108730 0x7fade4110090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:24:51.529 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:51.539+0000 7fadec777700 1 --2- 172.21.15.52:0/2700841235 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fade41034b0 0x7fade41038c0 unknown :-1 s=CLOSED pgs=308 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:24:51.529 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:51.539+0000 7fadec777700 1 --2- 172.21.15.52:0/2700841235 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fade4107de0 0x7fade41081f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:24:51.529 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:51.539+0000 7fadec777700 1 -- 172.21.15.52:0/2700841235 >> 172.21.15.52:0/2700841235 conn(0x7fade40fedb0 msgr2=0x7fade41011d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:24:51.529 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:51.539+0000 7fadec777700 1 -- 172.21.15.52:0/2700841235 shutdown_connections 2024-05-27T07:24:51.530 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:51.539+0000 7fadec777700 1 -- 172.21.15.52:0/2700841235 wait complete. 2024-05-27T07:24:51.531 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:51.539+0000 7fadec777700 1 Processor -- start 2024-05-27T07:24:51.531 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:51.540+0000 7fadec777700 1 -- start start 2024-05-27T07:24:51.531 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:51.540+0000 7fadec777700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fade41034b0 0x7fade419adc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:24:51.531 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:51.540+0000 7fadec777700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fade4107de0 0x7fade419b360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:24:51.531 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:51.540+0000 7fadec777700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fade4108730 0x7fade419f5f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:24:51.532 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:51.540+0000 7fadec777700 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x7fade41144b0 con 0x7fade41034b0 2024-05-27T07:24:51.532 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:51.540+0000 7fadec777700 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7fade4114330 con 0x7fade4108730 2024-05-27T07:24:51.532 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:51.540+0000 7fadec777700 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7fade4114630 con 0x7fade4107de0 2024-05-27T07:24:51.532 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:51.540+0000 7fadea513700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fade41034b0 0x7fade419adc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:24:51.532 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:51.540+0000 7fadea513700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fade41034b0 0x7fade419adc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.52:32980/0 (socket says 172.21.15.52:32980) 2024-05-27T07:24:51.532 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:51.540+0000 7fadea513700 1 -- 172.21.15.52:0/1568630857 learned_addr learned my addr 172.21.15.52:0/1568630857 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-05-27T07:24:51.532 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:51.540+0000 7fade9d12700 1 --2- 172.21.15.52:0/1568630857 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fade4107de0 0x7fade419b360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:24:51.532 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:51.540+0000 7fadead14700 1 --2- 172.21.15.52:0/1568630857 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fade4108730 0x7fade419f5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:24:51.532 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:51.540+0000 7fadea513700 1 -- 172.21.15.52:0/1568630857 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fade4107de0 msgr2=0x7fade419b360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:24:51.532 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:51.540+0000 7fadea513700 1 --2- 172.21.15.52:0/1568630857 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fade4107de0 0x7fade419b360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:24:51.532 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:51.540+0000 7fadea513700 1 -- 172.21.15.52:0/1568630857 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fade4108730 msgr2=0x7fade419f5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:24:51.532 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:51.540+0000 7fadea513700 1 --2- 172.21.15.52:0/1568630857 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fade4108730 0x7fade419f5f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:24:51.532 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:51.540+0000 7fadea513700 1 -- 172.21.15.52:0/1568630857 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fade419fc90 con 0x7fade41034b0 2024-05-27T07:24:51.533 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:51.541+0000 7fadea513700 1 --2- 172.21.15.52:0/1568630857 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fade41034b0 0x7fade419adc0 secure :-1 s=READY pgs=309 cs=0 l=1 rev1=1 crypto rx=0x7fade000d010 tx=0x7fade0007a20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:24:51.533 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:51.541+0000 7faddb7fe700 1 -- 172.21.15.52:0/1568630857 <== mon.0 v2:172.21.15.52:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fade001f070 con 0x7fade41034b0 2024-05-27T07:24:51.533 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:51.541+0000 7faddb7fe700 1 -- 172.21.15.52:0/1568630857 <== mon.0 v2:172.21.15.52:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fade0007e50 con 0x7fade41034b0 2024-05-27T07:24:51.533 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:51.541+0000 7fadec777700 1 -- 172.21.15.52:0/1568630857 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fade419ff20 con 0x7fade41034b0 2024-05-27T07:24:51.533 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:51.541+0000 7faddb7fe700 1 -- 172.21.15.52:0/1568630857 <== mon.0 v2:172.21.15.52:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fade001b4a0 con 0x7fade41034b0 2024-05-27T07:24:51.533 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:51.541+0000 7fadec777700 1 -- 172.21.15.52:0/1568630857 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fade41109a0 con 0x7fade41034b0 2024-05-27T07:24:51.533 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:51.542+0000 7fadec777700 1 -- 172.21.15.52:0/1568630857 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fade4104960 con 0x7fade41034b0 2024-05-27T07:24:51.533 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:51.543+0000 7faddb7fe700 1 -- 172.21.15.52:0/1568630857 <== mon.0 v2:172.21.15.52:3300/0 4 ==== mgrmap(e 18) ==== 149652+0+0 (secure 0 0 0) 0x7fade0022070 con 0x7fade41034b0 2024-05-27T07:24:51.534 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:51.544+0000 7faddb7fe700 1 --2- 172.21.15.52:0/1568630857 >> [v2:172.21.15.52:6800/3513075805,v1:172.21.15.52:6801/3513075805] conn(0x7fadd00b2900 0x7fadd00b4e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:24:51.534 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:51.544+0000 7fade9d12700 1 --2- 172.21.15.52:0/1568630857 >> [v2:172.21.15.52:6800/3513075805,v1:172.21.15.52:6801/3513075805] conn(0x7fadd00b2900 0x7fadd00b4e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:24:51.534 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:51.544+0000 7faddb7fe700 1 -- 172.21.15.52:0/1568630857 <== mon.0 v2:172.21.15.52:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9121+0+0 (secure 0 0 0) 0x7fade000f020 con 0x7fade41034b0 2024-05-27T07:24:51.534 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:51.545+0000 7fade9d12700 1 --2- 172.21.15.52:0/1568630857 >> [v2:172.21.15.52:6800/3513075805,v1:172.21.15.52:6801/3513075805] conn(0x7fadd00b2900 0x7fadd00b4e10 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7fade4107db0 tx=0x7fadd400b410 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:24:51.538 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:51.548+0000 7faddb7fe700 1 -- 172.21.15.52:0/1568630857 <== mon.0 v2:172.21.15.52:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191963 (secure 0 0 0) 0x7fade0029020 con 0x7fade41034b0 2024-05-27T07:24:51.805 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:51.815+0000 7fadec777700 1 -- 172.21.15.52:0/1568630857 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fade419c120 con 0x7fade41034b0 2024-05-27T07:24:51.806 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:51.816+0000 7faddb7fe700 1 -- 172.21.15.52:0/1568630857 <== mon.0 v2:172.21.15.52:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 29 v29) ==== 94+0+11938 (secure 0 0 0) 0x7fade00ab4e0 con 0x7fade41034b0 2024-05-27T07:24:51.806 INFO:teuthology.orchestra.run.smithi052.stdout: 2024-05-27T07:24:51.806 INFO:teuthology.orchestra.run.smithi052.stdout:{"epoch":29,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14556,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.52:6835/2584733778","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.52:6834","nonce":2584733778},{"type":"v1","addr":"172.21.15.52:6835","nonce":2584733778}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14568,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.52:6837/675445303","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.52:6836","nonce":675445303},{"type":"v1","addr":"172.21.15.52:6837","nonce":675445303}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":29,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-05-27T07:21:50.521783+0000","modified":"2024-05-27T07:22:58.822630+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24527,"mds_1":24497,"mds_2":24475,"mds_3":24473,"mds_4":24466},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14619":{"gid":14619,"name":"g","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.52:6839/4258549559","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.52:6838","nonce":4258549559},{"type":"v1","addr":"172.21.15.52:6839","nonce":4258549559}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_14631":{"gid":14631,"name":"a","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.52:6841/2878618188","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.52:6840","nonce":2878618188},{"type":"v1","addr":"172.21.15.52:6841","nonce":2878618188}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24433":{"gid":24433,"name":"l","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.173:6833/3013589220","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6832","nonce":3013589220},{"type":"v1","addr":"172.21.15.173:6833","nonce":3013589220}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24452":{"gid":24452,"name":"k","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.105:6833/2800678412","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6832","nonce":2800678412},{"type":"v1","addr":"172.21.15.105:6833","nonce":2800678412}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24460":{"gid":24460,"name":"f","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.173:6835/1619486437","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6834","nonce":1619486437},{"type":"v1","addr":"172.21.15.173:6835","nonce":1619486437}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24466":{"gid":24466,"name":"c","rank":4,"incarnation":26,"state":"up:active","state_seq":25,"addr":"172.21.15.173:6837/1396363827","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6836","nonce":1396363827},{"type":"v1","addr":"172.21.15.173:6837","nonce":1396363827}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24473":{"gid":24473,"name":"e","rank":3,"incarnation":24,"state":"up:active","state_seq":26,"addr":"172.21.15.105:6835/3298498384","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6834","nonce":3298498384},{"type":"v1","addr":"172.21.15.105:6835","nonce":3298498384}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24475":{"gid":24475,"name":"i","rank":2,"incarnation":22,"state":"up:active","state_seq":23,"addr":"172.21.15.173:6839/564778293","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6838","nonce":564778293},{"type":"v1","addr":"172.21.15.173:6839","nonce":564778293}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24497":{"gid":24497,"name":"h","rank":1,"incarnation":20,"state":"up:active","state_seq":24,"addr":"172.21.15.105:6837/191393149","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6836","nonce":191393149},{"type":"v1","addr":"172.21.15.105:6837","nonce":191393149}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24527":{"gid":24527,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":5,"addr":"172.21.15.105:6839/500348151","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6838","nonce":500348151},{"type":"v1","addr":"172.21.15.105:6839","nonce":500348151}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24527,"qdb_cluster":[24466,24473,24475,24497,24527]},"id":1}]} 2024-05-27T07:24:51.808 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:51.819+0000 7fadec777700 1 -- 172.21.15.52:0/1568630857 >> [v2:172.21.15.52:6800/3513075805,v1:172.21.15.52:6801/3513075805] conn(0x7fadd00b2900 msgr2=0x7fadd00b4e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:24:51.809 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:51.819+0000 7fadec777700 1 --2- 172.21.15.52:0/1568630857 >> [v2:172.21.15.52:6800/3513075805,v1:172.21.15.52:6801/3513075805] conn(0x7fadd00b2900 0x7fadd00b4e10 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7fade4107db0 tx=0x7fadd400b410 comp rx=0 tx=0).stop 2024-05-27T07:24:51.809 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:51.819+0000 7fadec777700 1 -- 172.21.15.52:0/1568630857 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fade41034b0 msgr2=0x7fade419adc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:24:51.809 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:51.819+0000 7fadec777700 1 --2- 172.21.15.52:0/1568630857 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fade41034b0 0x7fade419adc0 secure :-1 s=READY pgs=309 cs=0 l=1 rev1=1 crypto rx=0x7fade000d010 tx=0x7fade0007a20 comp rx=0 tx=0).stop 2024-05-27T07:24:51.809 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:51.819+0000 7fadec777700 1 -- 172.21.15.52:0/1568630857 shutdown_connections 2024-05-27T07:24:51.809 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:51.819+0000 7fadec777700 1 --2- 172.21.15.52:0/1568630857 >> [v2:172.21.15.52:6800/3513075805,v1:172.21.15.52:6801/3513075805] conn(0x7fadd00b2900 0x7fadd00b4e10 unknown :-1 s=CLOSED pgs=206 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:24:51.809 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:51.819+0000 7fadec777700 1 --2- 172.21.15.52:0/1568630857 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fade4108730 0x7fade419f5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:24:51.809 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:51.819+0000 7fadec777700 1 --2- 172.21.15.52:0/1568630857 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fade41034b0 0x7fade419adc0 unknown :-1 s=CLOSED pgs=309 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:24:51.809 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:51.819+0000 7fadec777700 1 --2- 172.21.15.52:0/1568630857 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fade4107de0 0x7fade419b360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:24:51.809 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:51.819+0000 7fadec777700 1 -- 172.21.15.52:0/1568630857 >> 172.21.15.52:0/1568630857 conn(0x7fade40fedb0 msgr2=0x7fade4101ca0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:24:51.809 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:51.819+0000 7fadec777700 1 -- 172.21.15.52:0/1568630857 shutdown_connections 2024-05-27T07:24:51.809 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:51.819+0000 7fadec777700 1 -- 172.21.15.52:0/1568630857 wait complete. 2024-05-27T07:24:51.810 INFO:teuthology.orchestra.run.smithi052.stderr:dumped fsmap epoch 29 2024-05-27T07:24:52.620 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 29, 'flags': 50, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': True, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 32, 'explicitly_allowed_features': 32, 'created': '2024-05-27T07:21:50.521783+0000', 'modified': '2024-05-27T07:22:58.822630+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 24527, 'mds_1': 24497, 'mds_2': 24475, 'mds_3': 24473, 'mds_4': 24466}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_14619': {'gid': 14619, 'name': 'g', 'rank': 4, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 1, 'addr': '172.21.15.52:6839/4258549559', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.52:6838', 'nonce': 4258549559}, {'type': 'v1', 'addr': '172.21.15.52:6839', 'nonce': 4258549559}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_14631': {'gid': 14631, 'name': 'a', 'rank': 3, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 1, 'addr': '172.21.15.52:6841/2878618188', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.52:6840', 'nonce': 2878618188}, {'type': 'v1', 'addr': '172.21.15.52:6841', 'nonce': 2878618188}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24433': {'gid': 24433, 'name': 'l', 'rank': 2, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 1, 'addr': '172.21.15.173:6833/3013589220', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.173:6832', 'nonce': 3013589220}, {'type': 'v1', 'addr': '172.21.15.173:6833', 'nonce': 3013589220}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24452': {'gid': 24452, 'name': 'k', 'rank': 1, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 1, 'addr': '172.21.15.105:6833/2800678412', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.105:6832', 'nonce': 2800678412}, {'type': 'v1', 'addr': '172.21.15.105:6833', 'nonce': 2800678412}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24460': {'gid': 24460, 'name': 'f', 'rank': 0, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 1, 'addr': '172.21.15.173:6835/1619486437', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.173:6834', 'nonce': 1619486437}, {'type': 'v1', 'addr': '172.21.15.173:6835', 'nonce': 1619486437}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24466': {'gid': 24466, 'name': 'c', 'rank': 4, 'incarnation': 26, 'state': 'up:active', 'state_seq': 25, 'addr': '172.21.15.173:6837/1396363827', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.173:6836', 'nonce': 1396363827}, {'type': 'v1', 'addr': '172.21.15.173:6837', 'nonce': 1396363827}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24473': {'gid': 24473, 'name': 'e', 'rank': 3, 'incarnation': 24, 'state': 'up:active', 'state_seq': 26, 'addr': '172.21.15.105:6835/3298498384', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.105:6834', 'nonce': 3298498384}, {'type': 'v1', 'addr': '172.21.15.105:6835', 'nonce': 3298498384}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24475': {'gid': 24475, 'name': 'i', 'rank': 2, 'incarnation': 22, 'state': 'up:active', 'state_seq': 23, 'addr': '172.21.15.173:6839/564778293', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.173:6838', 'nonce': 564778293}, {'type': 'v1', 'addr': '172.21.15.173:6839', 'nonce': 564778293}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24497': {'gid': 24497, 'name': 'h', 'rank': 1, 'incarnation': 20, 'state': 'up:active', 'state_seq': 24, 'addr': '172.21.15.105:6837/191393149', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.105:6836', 'nonce': 191393149}, {'type': 'v1', 'addr': '172.21.15.105:6837', 'nonce': 191393149}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24527': {'gid': 24527, 'name': 'b', 'rank': 0, 'incarnation': 14, 'state': 'up:active', 'state_seq': 5, 'addr': '172.21.15.105:6839/500348151', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.105:6838', 'nonce': 500348151}, {'type': 'v1', 'addr': '172.21.15.105:6839', 'nonce': 500348151}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 24527, 'qdb_cluster': [24466, 24473, 24475, 24497, 24527]} 2024-05-27T07:24:52.620 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 5/5 2024-05-27T07:24:52.621 DEBUG:teuthology.orchestra.run.smithi052:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:38f9d900c83c63799fbdbe61acc9a11b0d3554a6 shell --fsid 73df2636-1bf8-11ef-bc9b-c7b262605968 -- ceph tell mds.c status 2024-05-27T07:24:52.878 INFO:teuthology.orchestra.run.smithi052.stderr:Inferring config /var/lib/ceph/73df2636-1bf8-11ef-bc9b-c7b262605968/mon.a/config 2024-05-27T07:24:52.979 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:24:52 smithi052 ceph-mon[31955]: pgmap v391: 129 pgs: 129 active+clean; 647 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-05-27T07:24:52.979 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:24:52 smithi052 ceph-mon[31955]: from='client.? 172.21.15.52:0/1568630857' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-27T07:24:53.102 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:24:52 smithi173 ceph-mon[37000]: pgmap v391: 129 pgs: 129 active+clean; 647 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-05-27T07:24:53.102 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:24:52 smithi173 ceph-mon[37000]: from='client.? 172.21.15.52:0/1568630857' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-27T07:24:53.190 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:24:52 smithi105 ceph-mon[35977]: pgmap v391: 129 pgs: 129 active+clean; 647 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-05-27T07:24:53.190 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:24:52 smithi105 ceph-mon[35977]: from='client.? 172.21.15.52:0/1568630857' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-27T07:24:54.520 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:54.530+0000 7fc38123a700 1 -- 172.21.15.52:0/3248697972 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fc37c1034b0 msgr2=0x7fc37c1038c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:24:54.521 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:54.530+0000 7fc38123a700 1 --2- 172.21.15.52:0/3248697972 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fc37c1034b0 0x7fc37c1038c0 secure :-1 s=READY pgs=310 cs=0 l=1 rev1=1 crypto rx=0x7fc36c00b0d0 tx=0x7fc36c00b3e0 comp rx=0 tx=0).stop 2024-05-27T07:24:54.521 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:54.530+0000 7fc38123a700 1 -- 172.21.15.52:0/3248697972 shutdown_connections 2024-05-27T07:24:54.521 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:54.530+0000 7fc38123a700 1 --2- 172.21.15.52:0/3248697972 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fc37c108730 0x7fc37c110090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:24:54.521 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:54.530+0000 7fc38123a700 1 --2- 172.21.15.52:0/3248697972 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fc37c1034b0 0x7fc37c1038c0 unknown :-1 s=CLOSED pgs=310 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:24:54.521 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:54.530+0000 7fc38123a700 1 --2- 172.21.15.52:0/3248697972 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fc37c107de0 0x7fc37c1081f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:24:54.521 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:54.530+0000 7fc38123a700 1 -- 172.21.15.52:0/3248697972 >> 172.21.15.52:0/3248697972 conn(0x7fc37c0fedb0 msgr2=0x7fc37c1011d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:24:54.521 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:54.530+0000 7fc38123a700 1 -- 172.21.15.52:0/3248697972 shutdown_connections 2024-05-27T07:24:54.521 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:54.531+0000 7fc38123a700 1 -- 172.21.15.52:0/3248697972 wait complete. 2024-05-27T07:24:54.523 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:54.531+0000 7fc38123a700 1 Processor -- start 2024-05-27T07:24:54.523 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:54.532+0000 7fc38123a700 1 -- start start 2024-05-27T07:24:54.523 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:54.532+0000 7fc38123a700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fc37c1034b0 0x7fc37c19f8c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:24:54.523 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:54.532+0000 7fc38123a700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fc37c107de0 0x7fc37c19fe20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:24:54.523 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:54.532+0000 7fc38123a700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fc37c108730 0x7fc37c199b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:24:54.523 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:54.532+0000 7fc38123a700 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x7fc37c113360 con 0x7fc37c1034b0 2024-05-27T07:24:54.523 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:54.532+0000 7fc38123a700 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7fc37c1131e0 con 0x7fc37c107de0 2024-05-27T07:24:54.523 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:54.532+0000 7fc38123a700 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7fc37c1134e0 con 0x7fc37c108730 2024-05-27T07:24:54.523 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:54.532+0000 7fc37ad9d700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fc37c1034b0 0x7fc37c19f8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:24:54.523 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:54.532+0000 7fc37ad9d700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fc37c1034b0 0x7fc37c19f8c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.52:32996/0 (socket says 172.21.15.52:32996) 2024-05-27T07:24:54.524 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:54.532+0000 7fc37ad9d700 1 -- 172.21.15.52:0/3346152711 learned_addr learned my addr 172.21.15.52:0/3346152711 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-05-27T07:24:54.524 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:54.532+0000 7fc37a59c700 1 --2- 172.21.15.52:0/3346152711 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fc37c107de0 0x7fc37c19fe20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:24:54.524 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:54.532+0000 7fc37b59e700 1 --2- 172.21.15.52:0/3346152711 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fc37c108730 0x7fc37c199b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:24:54.524 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:54.532+0000 7fc37ad9d700 1 -- 172.21.15.52:0/3346152711 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fc37c108730 msgr2=0x7fc37c199b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:24:54.524 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:54.532+0000 7fc37ad9d700 1 --2- 172.21.15.52:0/3346152711 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fc37c108730 0x7fc37c199b00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:24:54.524 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:54.532+0000 7fc37ad9d700 1 -- 172.21.15.52:0/3346152711 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fc37c107de0 msgr2=0x7fc37c19fe20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:24:54.524 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:54.532+0000 7fc37ad9d700 1 --2- 172.21.15.52:0/3346152711 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fc37c107de0 0x7fc37c19fe20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:24:54.524 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:54.532+0000 7fc37ad9d700 1 -- 172.21.15.52:0/3346152711 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc37c19a170 con 0x7fc37c1034b0 2024-05-27T07:24:54.524 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:54.532+0000 7fc37b59e700 1 --2- 172.21.15.52:0/3346152711 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fc37c108730 0x7fc37c199b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-27T07:24:54.524 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:54.533+0000 7fc37ad9d700 1 --2- 172.21.15.52:0/3346152711 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fc37c1034b0 0x7fc37c19f8c0 secure :-1 s=READY pgs=311 cs=0 l=1 rev1=1 crypto rx=0x7fc36c007a20 tx=0x7fc36c007a50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:24:54.524 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:54.533+0000 7fc373fff700 1 -- 172.21.15.52:0/3346152711 <== mon.0 v2:172.21.15.52:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc36c01f070 con 0x7fc37c1034b0 2024-05-27T07:24:54.524 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:54.533+0000 7fc38123a700 1 -- 172.21.15.52:0/3346152711 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc37c19a400 con 0x7fc37c1034b0 2024-05-27T07:24:54.525 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:54.533+0000 7fc373fff700 1 -- 172.21.15.52:0/3346152711 <== mon.0 v2:172.21.15.52:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fc36c003c00 con 0x7fc37c1034b0 2024-05-27T07:24:54.525 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:54.533+0000 7fc373fff700 1 -- 172.21.15.52:0/3346152711 <== mon.0 v2:172.21.15.52:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc36c01b730 con 0x7fc37c1034b0 2024-05-27T07:24:54.525 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:54.533+0000 7fc38123a700 1 -- 172.21.15.52:0/3346152711 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc37c19a930 con 0x7fc37c1034b0 2024-05-27T07:24:54.525 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:54.536+0000 7fc373fff700 1 -- 172.21.15.52:0/3346152711 <== mon.0 v2:172.21.15.52:3300/0 4 ==== mgrmap(e 18) ==== 149652+0+0 (secure 0 0 0) 0x7fc36c022070 con 0x7fc37c1034b0 2024-05-27T07:24:54.526 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:54.536+0000 7fc373fff700 1 --2- 172.21.15.52:0/3346152711 >> [v2:172.21.15.52:6800/3513075805,v1:172.21.15.52:6801/3513075805] conn(0x7fc3680b2900 0x7fc3680b4e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:24:54.526 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:54.537+0000 7fc373fff700 1 -- 172.21.15.52:0/3346152711 <== mon.0 v2:172.21.15.52:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9121+0+0 (secure 0 0 0) 0x7fc36c00f020 con 0x7fc37c1034b0 2024-05-27T07:24:54.527 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:54.537+0000 7fc37a59c700 1 --2- 172.21.15.52:0/3346152711 >> [v2:172.21.15.52:6800/3513075805,v1:172.21.15.52:6801/3513075805] conn(0x7fc3680b2900 0x7fc3680b4e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:24:54.527 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:54.537+0000 7fc38fc4fb80 1 Processor -- start 2024-05-27T07:24:54.527 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:54.537+0000 7fc38fc4fb80 1 -- start start 2024-05-27T07:24:54.528 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:54.537+0000 7fc37a59c700 1 --2- 172.21.15.52:0/3346152711 >> [v2:172.21.15.52:6800/3513075805,v1:172.21.15.52:6801/3513075805] conn(0x7fc3680b2900 0x7fc3680b4e10 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7fc37c1023c0 tx=0x7fc36400a380 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:24:54.528 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:54.537+0000 7fc38fc4fb80 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5633eab00df0 0x5633eac84240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:24:54.528 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:54.537+0000 7fc38fc4fb80 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x5633eac84780 0x5633eac88a20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:24:54.528 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:54.537+0000 7fc38fc4fb80 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5633eac88f60 0x5633eac89370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:24:54.528 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:54.537+0000 7fc38fc4fb80 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x5633eabadd20 con 0x5633eac84780 2024-05-27T07:24:54.529 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:54.537+0000 7fc38fc4fb80 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x5633eab9d8c0 con 0x5633eab00df0 2024-05-27T07:24:54.529 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:54.537+0000 7fc38fc4fb80 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x5633eaba1900 con 0x5633eac88f60 2024-05-27T07:24:54.529 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:54.538+0000 7fc37a59c700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x5633eac84780 0x5633eac88a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:24:54.529 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:54.538+0000 7fc37a59c700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x5633eac84780 0x5633eac88a20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.52:33004/0 (socket says 172.21.15.52:33004) 2024-05-27T07:24:54.529 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:54.538+0000 7fc37a59c700 1 -- 172.21.15.52:0/252109876 learned_addr learned my addr 172.21.15.52:0/252109876 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-05-27T07:24:54.529 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:54.538+0000 7fc37b59e700 1 --2- 172.21.15.52:0/252109876 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5633eac88f60 0x5633eac89370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:24:54.529 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:54.538+0000 7fc37ad9d700 1 --2- 172.21.15.52:0/252109876 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5633eab00df0 0x5633eac84240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:24:54.529 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:54.538+0000 7fc37a59c700 1 -- 172.21.15.52:0/252109876 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5633eac88f60 msgr2=0x5633eac89370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:24:54.529 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:54.538+0000 7fc37a59c700 1 --2- 172.21.15.52:0/252109876 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5633eac88f60 0x5633eac89370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:24:54.529 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:54.538+0000 7fc37a59c700 1 -- 172.21.15.52:0/252109876 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5633eab00df0 msgr2=0x5633eac84240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:24:54.529 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:54.538+0000 7fc37a59c700 1 --2- 172.21.15.52:0/252109876 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5633eab00df0 0x5633eac84240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:24:54.529 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:54.538+0000 7fc37a59c700 1 -- 172.21.15.52:0/252109876 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5633eab46f60 con 0x5633eac84780 2024-05-27T07:24:54.530 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:54.538+0000 7fc37b59e700 1 --2- 172.21.15.52:0/252109876 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5633eac88f60 0x5633eac89370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-27T07:24:54.530 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:54.538+0000 7fc37ad9d700 1 --2- 172.21.15.52:0/252109876 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5633eab00df0 0x5633eac84240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-27T07:24:54.530 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:54.538+0000 7fc37a59c700 1 --2- 172.21.15.52:0/252109876 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x5633eac84780 0x5633eac88a20 secure :-1 s=READY pgs=312 cs=0 l=1 rev1=1 crypto rx=0x7fc36400fe70 tx=0x7fc364012000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e4fd0a6aad7d3f4e server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:24:54.530 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:54.538+0000 7fc3717fa700 1 -- 172.21.15.52:0/252109876 <== mon.0 v2:172.21.15.52:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc364016070 con 0x5633eac84780 2024-05-27T07:24:54.530 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:54.538+0000 7fc3717fa700 1 -- 172.21.15.52:0/252109876 <== mon.0 v2:172.21.15.52:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fc364014070 con 0x5633eac84780 2024-05-27T07:24:54.530 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:54.538+0000 7fc3717fa700 1 -- 172.21.15.52:0/252109876 <== mon.0 v2:172.21.15.52:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc36401d400 con 0x5633eac84780 2024-05-27T07:24:54.533 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:54.539+0000 7fc38fc4fb80 1 -- 172.21.15.52:0/252109876 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x5633eac84780 msgr2=0x5633eac88a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:24:54.533 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:54.539+0000 7fc38fc4fb80 1 --2- 172.21.15.52:0/252109876 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x5633eac84780 0x5633eac88a20 secure :-1 s=READY pgs=312 cs=0 l=1 rev1=1 crypto rx=0x7fc36400fe70 tx=0x7fc364012000 comp rx=0 tx=0).stop 2024-05-27T07:24:54.533 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:54.539+0000 7fc38fc4fb80 1 -- 172.21.15.52:0/252109876 shutdown_connections 2024-05-27T07:24:54.533 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:54.539+0000 7fc38fc4fb80 1 --2- 172.21.15.52:0/252109876 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5633eac88f60 0x5633eac89370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:24:54.533 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:54.539+0000 7fc38fc4fb80 1 --2- 172.21.15.52:0/252109876 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5633eab00df0 0x5633eac84240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:24:54.533 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:54.539+0000 7fc38fc4fb80 1 --2- 172.21.15.52:0/252109876 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x5633eac84780 0x5633eac88a20 unknown :-1 s=CLOSED pgs=312 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:24:54.534 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:54.539+0000 7fc38fc4fb80 1 -- 172.21.15.52:0/252109876 >> 172.21.15.52:0/252109876 conn(0x5633eab03720 msgr2=0x5633eac7ad40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:24:54.534 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:54.539+0000 7fc38fc4fb80 1 -- 172.21.15.52:0/252109876 shutdown_connections 2024-05-27T07:24:54.534 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:54.539+0000 7fc38fc4fb80 1 -- 172.21.15.52:0/252109876 wait complete. 2024-05-27T07:24:54.534 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:54.540+0000 7fc38fc4fb80 1 -- start start 2024-05-27T07:24:54.534 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:54.540+0000 7fc38fc4fb80 1 Processor -- start 2024-05-27T07:24:54.534 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:54.540+0000 7fc38fc4fb80 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5633eab00df0 0x5633eac8dd30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:24:54.534 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:54.540+0000 7fc38fc4fb80 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x5633eac84780 0x5633eac8e270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:24:54.534 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:54.540+0000 7fc38fc4fb80 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5633eac88f60 0x5633eac8e7b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:24:54.534 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:54.540+0000 7fc38fc4fb80 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x5633eaaa3f50 con 0x5633eac84780 2024-05-27T07:24:54.534 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:54.540+0000 7fc38fc4fb80 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x5633eab05d50 con 0x5633eab00df0 2024-05-27T07:24:54.534 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:54.540+0000 7fc38fc4fb80 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x5633eac90f70 con 0x5633eac88f60 2024-05-27T07:24:54.534 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:54.540+0000 7fc38fc4fb80 10 client.0 _ll_register_callbacks cb 0x5633eab34560 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T07:24:54.534 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:54.540+0000 7fc37a59c700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x5633eac84780 0x5633eac8e270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:24:54.535 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:54.540+0000 7fc37a59c700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x5633eac84780 0x5633eac8e270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.52:33008/0 (socket says 172.21.15.52:33008) 2024-05-27T07:24:54.535 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:54.540+0000 7fc37a59c700 1 -- 172.21.15.52:0/3280231148 learned_addr learned my addr 172.21.15.52:0/3280231148 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-05-27T07:24:54.535 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:54.541+0000 7fc37ad9d700 1 --2- 172.21.15.52:0/3280231148 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5633eab00df0 0x5633eac8dd30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:24:54.535 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:54.541+0000 7fc37b59e700 1 --2- 172.21.15.52:0/3280231148 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5633eac88f60 0x5633eac8e7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:24:54.535 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:54.541+0000 7fc37a59c700 1 -- 172.21.15.52:0/3280231148 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5633eac88f60 msgr2=0x5633eac8e7b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:24:54.535 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:54.541+0000 7fc37a59c700 1 --2- 172.21.15.52:0/3280231148 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5633eac88f60 0x5633eac8e7b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:24:54.535 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:54.541+0000 7fc37a59c700 1 -- 172.21.15.52:0/3280231148 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5633eab00df0 msgr2=0x5633eac8dd30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:24:54.535 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:54.541+0000 7fc37a59c700 1 --2- 172.21.15.52:0/3280231148 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5633eab00df0 0x5633eac8dd30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:24:54.535 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:54.541+0000 7fc37a59c700 1 -- 172.21.15.52:0/3280231148 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5633eaaed880 con 0x5633eac84780 2024-05-27T07:24:54.535 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:54.541+0000 7fc37b59e700 1 --2- 172.21.15.52:0/3280231148 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5633eac88f60 0x5633eac8e7b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-27T07:24:54.535 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:54.541+0000 7fc37a59c700 1 --2- 172.21.15.52:0/3280231148 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x5633eac84780 0x5633eac8e270 secure :-1 s=READY pgs=313 cs=0 l=1 rev1=1 crypto rx=0x7fc364005810 tx=0x7fc364010760 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ea120cb1e5ba83b0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:24:54.535 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:54.541+0000 7fc3627fc700 10 client.0 ms_handle_connect on v2:172.21.15.52:3300/0 2024-05-27T07:24:54.535 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:54.541+0000 7fc3627fc700 1 -- 172.21.15.52:0/3280231148 <== mon.0 v2:172.21.15.52:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc364005030 con 0x5633eac84780 2024-05-27T07:24:54.536 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:54.541+0000 7fc3627fc700 1 -- 172.21.15.52:0/3280231148 <== mon.0 v2:172.21.15.52:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9121+0+0 (secure 0 0 0) 0x7fc3640051d0 con 0x5633eac84780 2024-05-27T07:24:54.536 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:54.541+0000 7fc38fc4fb80 1 -- 172.21.15.52:0/3280231148 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fc364014070 con 0x5633eac84780 2024-05-27T07:24:54.536 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:54.541+0000 7fc3627fc700 1 -- 172.21.15.52:0/3280231148 <== mon.0 v2:172.21.15.52:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7fc3640053d0 con 0x5633eac84780 2024-05-27T07:24:54.536 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:54.541+0000 7fc38fc4fb80 10 client.15081 fetch_fsmap learned FSMap version 29 2024-05-27T07:24:54.536 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:54.541+0000 7fc38fc4fb80 1 -- 172.21.15.52:0/3280231148 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x5633eab57710 con 0x5633eac84780 2024-05-27T07:24:54.536 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:54.542+0000 7fc3627fc700 1 -- 172.21.15.52:0/3280231148 <== mon.0 v2:172.21.15.52:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fc364014070 con 0x5633eac84780 2024-05-27T07:24:54.536 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:54.542+0000 7fc3627fc700 1 -- 172.21.15.52:0/3280231148 <== mon.0 v2:172.21.15.52:3300/0 5 ==== fsmap(e 29) ==== 8362+0+0 (secure 0 0 0) 0x7fc364004a10 con 0x5633eac84780 2024-05-27T07:24:54.536 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:54.542+0000 7fc38fc4fb80 10 client.15081 fetch_fsmap finished waiting for FSMap version 29 2024-05-27T07:24:54.536 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:54.542+0000 7fc38fc4fb80 10 client.15081 resolve_mds: resolved name 'c' to daemon mds.c 2024-05-27T07:24:54.536 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:54.542+0000 7fc38fc4fb80 20 client.15081 populate_metadata read hostname 'smithi052' 2024-05-27T07:24:54.536 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:54.542+0000 7fc38fc4fb80 1 --2- 172.21.15.52:0/3280231148 >> [v2:172.21.15.173:6836/1396363827,v1:172.21.15.173:6837/1396363827] conn(0x5633eac9dc60 0x5633eac9f0d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:24:54.536 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:54.542+0000 7fc38fc4fb80 4 client.15081 mds_command: new command op to 24466 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-27T07:24:54.536 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:54.542+0000 7fc38fc4fb80 1 -- 172.21.15.52:0/3280231148 --> [v2:172.21.15.173:6836/1396363827,v1:172.21.15.173:6837/1396363827] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5633eabafc70 con 0x5633eac9dc60 2024-05-27T07:24:54.537 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:54.542+0000 7fc37ad9d700 1 --2- 172.21.15.52:0/3280231148 >> [v2:172.21.15.173:6836/1396363827,v1:172.21.15.173:6837/1396363827] conn(0x5633eac9dc60 0x5633eac9f0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:24:54.537 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:54.543+0000 7fc37ad9d700 1 --2- 172.21.15.52:0/3280231148 >> [v2:172.21.15.173:6836/1396363827,v1:172.21.15.173:6837/1396363827] conn(0x5633eac9dc60 0x5633eac9f0d0 crc :-1 s=READY pgs=15 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=18d618d03c9638e6 server_cookie=47c6c4d9f631c53d in_seq=0 out_seq=0 2024-05-27T07:24:54.537 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:54.543+0000 7fc3627fc700 10 client.15081 ms_handle_connect on v2:172.21.15.173:6836/1396363827 2024-05-27T07:24:54.538 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:54.544+0000 7fc3627fc700 1 -- 172.21.15.52:0/3280231148 <== mds.4 v2:172.21.15.173:6836/1396363827 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24041 (crc 0 0 0) 0x5633eabafc70 con 0x5633eac9dc60 2024-05-27T07:24:54.538 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:54.544+0000 7fc3627fc700 10 client.15081 handle_command_reply: tid=0 2024-05-27T07:24:54.538 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:54.544+0000 7fc38fc4fb80 1 client.15081 shutdown 2024-05-27T07:24:54.538 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:54.544+0000 7fc38fc4fb80 1 -- 172.21.15.52:0/3280231148 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x5633eac84780 msgr2=0x5633eac8e270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:24:54.538 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:54.544+0000 7fc38fc4fb80 1 --2- 172.21.15.52:0/3280231148 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x5633eac84780 0x5633eac8e270 secure :-1 s=READY pgs=313 cs=0 l=1 rev1=1 crypto rx=0x7fc364005810 tx=0x7fc364010760 comp rx=0 tx=0).stop 2024-05-27T07:24:54.539 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:54.544+0000 7fc38fc4fb80 1 -- 172.21.15.52:0/3280231148 shutdown_connections 2024-05-27T07:24:54.539 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:54.544+0000 7fc38fc4fb80 1 --2- 172.21.15.52:0/3280231148 >> [v2:172.21.15.173:6836/1396363827,v1:172.21.15.173:6837/1396363827] conn(0x5633eac9dc60 0x5633eac9f0d0 crc :-1 s=READY pgs=15 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:24:54.539 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:54.544+0000 7fc3627fc700 0 client.15081 ms_handle_reset on v2:172.21.15.173:6836/1396363827 2024-05-27T07:24:54.539 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:54.544+0000 7fc38fc4fb80 1 --2- 172.21.15.52:0/3280231148 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5633eac88f60 0x5633eac8e7b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:24:54.539 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:54.544+0000 7fc38fc4fb80 1 --2- 172.21.15.52:0/3280231148 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5633eab00df0 0x5633eac8dd30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:24:54.539 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:54.544+0000 7fc38fc4fb80 1 --2- 172.21.15.52:0/3280231148 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x5633eac84780 0x5633eac8e270 unknown :-1 s=CLOSED pgs=313 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:24:54.539 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:54.544+0000 7fc38fc4fb80 1 -- 172.21.15.52:0/3280231148 >> 172.21.15.52:0/3280231148 conn(0x5633eab03720 msgr2=0x5633eac7eb80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:24:54.539 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:54.544+0000 7fc38fc4fb80 1 -- 172.21.15.52:0/3280231148 shutdown_connections 2024-05-27T07:24:54.539 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:54.544+0000 7fc38fc4fb80 1 -- 172.21.15.52:0/3280231148 wait complete. 2024-05-27T07:24:54.539 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:54.544+0000 7fc38fc4fb80 20 client.15081 trim_cache size 0 max 16384 2024-05-27T07:24:54.555 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:54.566+0000 7fc38fc4fb80 1 Processor -- start 2024-05-27T07:24:54.556 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:54.566+0000 7fc38fc4fb80 1 -- start start 2024-05-27T07:24:54.557 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:54.566+0000 7fc38fc4fb80 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x5633eab00df0 0x5633eac9ea00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:24:54.557 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:54.566+0000 7fc38fc4fb80 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5633eac84780 0x5633eac9d590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:24:54.557 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:54.566+0000 7fc38fc4fb80 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5633eac88f60 0x5633eac8e630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:24:54.557 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:54.567+0000 7fc38fc4fb80 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x5633eac9d410 con 0x5633eab00df0 2024-05-27T07:24:54.557 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:54.567+0000 7fc38fc4fb80 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x5633eac9dae0 con 0x5633eac88f60 2024-05-27T07:24:54.557 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:54.567+0000 7fc38fc4fb80 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x5633eac9e1b0 con 0x5633eac84780 2024-05-27T07:24:54.558 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:54.567+0000 7fc37ad9d700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x5633eab00df0 0x5633eac9ea00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:24:54.558 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:54.567+0000 7fc37ad9d700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x5633eab00df0 0x5633eac9ea00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.52:33020/0 (socket says 172.21.15.52:33020) 2024-05-27T07:24:54.558 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:54.567+0000 7fc37ad9d700 1 -- 172.21.15.52:0/3869335683 learned_addr learned my addr 172.21.15.52:0/3869335683 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-05-27T07:24:54.558 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:54.567+0000 7fc37b59e700 1 --2- 172.21.15.52:0/3869335683 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5633eac88f60 0x5633eac8e630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:24:54.558 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:54.567+0000 7fc37a59c700 1 --2- 172.21.15.52:0/3869335683 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5633eac84780 0x5633eac9d590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:24:54.558 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:54.567+0000 7fc37ad9d700 1 -- 172.21.15.52:0/3869335683 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5633eac84780 msgr2=0x5633eac9d590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:24:54.558 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:54.567+0000 7fc37ad9d700 1 --2- 172.21.15.52:0/3869335683 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5633eac84780 0x5633eac9d590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:24:54.558 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:54.567+0000 7fc37ad9d700 1 -- 172.21.15.52:0/3869335683 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5633eac88f60 msgr2=0x5633eac8e630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:24:54.558 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:54.567+0000 7fc37ad9d700 1 --2- 172.21.15.52:0/3869335683 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5633eac88f60 0x5633eac8e630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:24:54.558 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:54.567+0000 7fc37ad9d700 1 -- 172.21.15.52:0/3869335683 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5633eac76a50 con 0x5633eab00df0 2024-05-27T07:24:54.558 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:54.567+0000 7fc37a59c700 1 --2- 172.21.15.52:0/3869335683 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5633eac84780 0x5633eac9d590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-27T07:24:54.558 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:54.567+0000 7fc37b59e700 1 --2- 172.21.15.52:0/3869335683 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5633eac88f60 0x5633eac8e630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-27T07:24:54.559 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:54.567+0000 7fc37ad9d700 1 --2- 172.21.15.52:0/3869335683 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x5633eab00df0 0x5633eac9ea00 secure :-1 s=READY pgs=314 cs=0 l=1 rev1=1 crypto rx=0x7fc36c0e0fb0 tx=0x7fc36c0099e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a51d41fe333cc3fa server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:24:54.559 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:54.567+0000 7fc370ff9700 1 -- 172.21.15.52:0/3869335683 <== mon.0 v2:172.21.15.52:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc36c0c4070 con 0x5633eab00df0 2024-05-27T07:24:54.559 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:54.567+0000 7fc370ff9700 1 -- 172.21.15.52:0/3869335683 <== mon.0 v2:172.21.15.52:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fc36c003db0 con 0x5633eab00df0 2024-05-27T07:24:54.559 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:54.567+0000 7fc370ff9700 1 -- 172.21.15.52:0/3869335683 <== mon.0 v2:172.21.15.52:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc36c0ab020 con 0x5633eab00df0 2024-05-27T07:24:54.561 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:54.569+0000 7fc38fc4fb80 1 -- 172.21.15.52:0/3869335683 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x5633eab00df0 msgr2=0x5633eac9ea00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:24:54.561 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:54.569+0000 7fc38fc4fb80 1 --2- 172.21.15.52:0/3869335683 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x5633eab00df0 0x5633eac9ea00 secure :-1 s=READY pgs=314 cs=0 l=1 rev1=1 crypto rx=0x7fc36c0e0fb0 tx=0x7fc36c0099e0 comp rx=0 tx=0).stop 2024-05-27T07:24:54.561 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:54.569+0000 7fc38fc4fb80 1 -- 172.21.15.52:0/3869335683 shutdown_connections 2024-05-27T07:24:54.561 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:54.569+0000 7fc38fc4fb80 1 --2- 172.21.15.52:0/3869335683 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5633eac88f60 0x5633eac8e630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:24:54.561 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:54.569+0000 7fc38fc4fb80 1 --2- 172.21.15.52:0/3869335683 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x5633eab00df0 0x5633eac9ea00 unknown :-1 s=CLOSED pgs=314 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:24:54.561 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:54.569+0000 7fc38fc4fb80 1 --2- 172.21.15.52:0/3869335683 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5633eac84780 0x5633eac9d590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:24:54.561 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:54.569+0000 7fc38fc4fb80 1 -- 172.21.15.52:0/3869335683 >> 172.21.15.52:0/3869335683 conn(0x5633eab03720 msgr2=0x5633eac7c120 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:24:54.561 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:54.569+0000 7fc38fc4fb80 1 -- 172.21.15.52:0/3869335683 shutdown_connections 2024-05-27T07:24:54.561 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:54.569+0000 7fc38fc4fb80 1 -- 172.21.15.52:0/3869335683 wait complete. 2024-05-27T07:24:54.561 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:54.569+0000 7fc38fc4fb80 1 -- start start 2024-05-27T07:24:54.562 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:54.570+0000 7fc38fc4fb80 1 Processor -- start 2024-05-27T07:24:54.562 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:54.570+0000 7fc38fc4fb80 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5633eab00df0 0x5633eaca5660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:24:54.562 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:54.570+0000 7fc38fc4fb80 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x5633eac84780 0x5633eaca5ba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:24:54.562 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:54.570+0000 7fc38fc4fb80 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5633eac88f60 0x5633ead073f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:24:54.562 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:54.570+0000 7fc38fc4fb80 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x5633eac9ef50 con 0x5633eac84780 2024-05-27T07:24:54.562 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:54.570+0000 7fc38fc4fb80 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x5633eac9e880 con 0x5633eac88f60 2024-05-27T07:24:54.562 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:54.570+0000 7fc38fc4fb80 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x5633eac9f620 con 0x5633eab00df0 2024-05-27T07:24:54.562 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:54.570+0000 7fc38fc4fb80 10 client.0 _ll_register_callbacks cb 0x5633eab34560 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T07:24:54.562 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:54.570+0000 7fc37a59c700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x5633eac84780 0x5633eaca5ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:24:54.562 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:54.570+0000 7fc37a59c700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x5633eac84780 0x5633eaca5ba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.52:33026/0 (socket says 172.21.15.52:33026) 2024-05-27T07:24:54.562 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:54.570+0000 7fc37a59c700 1 -- 172.21.15.52:0/2934944619 learned_addr learned my addr 172.21.15.52:0/2934944619 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-05-27T07:24:54.562 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:54.570+0000 7fc37b59e700 1 --2- 172.21.15.52:0/2934944619 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5633eac88f60 0x5633ead073f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:24:54.562 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:54.570+0000 7fc37ad9d700 1 --2- 172.21.15.52:0/2934944619 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5633eab00df0 0x5633eaca5660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:24:54.563 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:54.570+0000 7fc37a59c700 1 -- 172.21.15.52:0/2934944619 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5633eab00df0 msgr2=0x5633eaca5660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:24:54.563 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:54.570+0000 7fc37a59c700 1 --2- 172.21.15.52:0/2934944619 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5633eab00df0 0x5633eaca5660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:24:54.563 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:54.571+0000 7fc37a59c700 1 -- 172.21.15.52:0/2934944619 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5633eac88f60 msgr2=0x5633ead073f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:24:54.563 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:54.571+0000 7fc37a59c700 1 --2- 172.21.15.52:0/2934944619 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5633eac88f60 0x5633ead073f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:24:54.563 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:54.571+0000 7fc37a59c700 1 -- 172.21.15.52:0/2934944619 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5633eaca6450 con 0x5633eac84780 2024-05-27T07:24:54.563 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:54.571+0000 7fc37a59c700 1 --2- 172.21.15.52:0/2934944619 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x5633eac84780 0x5633eaca5ba0 secure :-1 s=READY pgs=315 cs=0 l=1 rev1=1 crypto rx=0x7fc364004f20 tx=0x7fc36400ff40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ce7bae0af9d35478 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:24:54.563 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:54.571+0000 7fc3627fc700 10 client.0 ms_handle_connect on v2:172.21.15.52:3300/0 2024-05-27T07:24:54.563 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:54.571+0000 7fc3627fc700 1 -- 172.21.15.52:0/2934944619 <== mon.0 v2:172.21.15.52:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc364016070 con 0x5633eac84780 2024-05-27T07:24:54.563 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:54.571+0000 7fc3627fc700 1 -- 172.21.15.52:0/2934944619 <== mon.0 v2:172.21.15.52:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9121+0+0 (secure 0 0 0) 0x7fc3640308f0 con 0x5633eac84780 2024-05-27T07:24:54.563 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:54.571+0000 7fc38fc4fb80 1 -- 172.21.15.52:0/2934944619 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fc36c003db0 con 0x5633eac84780 2024-05-27T07:24:54.563 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:54.571+0000 7fc3627fc700 1 -- 172.21.15.52:0/2934944619 <== mon.0 v2:172.21.15.52:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7fc364030b30 con 0x5633eac84780 2024-05-27T07:24:54.563 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:54.571+0000 7fc38fc4fb80 10 client.15087 fetch_fsmap learned FSMap version 29 2024-05-27T07:24:54.563 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:54.571+0000 7fc38fc4fb80 1 -- 172.21.15.52:0/2934944619 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x5633ead08970 con 0x5633eac84780 2024-05-27T07:24:54.565 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:54.571+0000 7fc3627fc700 1 -- 172.21.15.52:0/2934944619 <== mon.0 v2:172.21.15.52:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fc36c003db0 con 0x5633eac84780 2024-05-27T07:24:54.565 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:54.572+0000 7fc3627fc700 1 -- 172.21.15.52:0/2934944619 <== mon.0 v2:172.21.15.52:3300/0 5 ==== fsmap(e 29) ==== 8362+0+0 (secure 0 0 0) 0x7fc36403d030 con 0x5633eac84780 2024-05-27T07:24:54.565 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:54.572+0000 7fc38fc4fb80 10 client.15087 fetch_fsmap finished waiting for FSMap version 29 2024-05-27T07:24:54.565 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:54.572+0000 7fc38fc4fb80 10 client.15087 resolve_mds: resolved name 'c' to daemon mds.c 2024-05-27T07:24:54.565 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:54.572+0000 7fc38fc4fb80 20 client.15087 populate_metadata read hostname 'smithi052' 2024-05-27T07:24:54.565 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:54.572+0000 7fc38fc4fb80 1 --2- 172.21.15.52:0/2934944619 >> [v2:172.21.15.173:6836/1396363827,v1:172.21.15.173:6837/1396363827] conn(0x5633eac9dc60 0x5633ead0bfe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:24:54.566 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:54.572+0000 7fc38fc4fb80 4 client.15087 mds_command: new command op to 24466 tid=0 multi_id=0 [{"prefix": "status"}] 2024-05-27T07:24:54.566 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:54.572+0000 7fc38fc4fb80 1 -- 172.21.15.52:0/2934944619 --> [v2:172.21.15.173:6836/1396363827,v1:172.21.15.173:6837/1396363827] -- command(tid 0: {"prefix": "status"}) -- 0x5633eaba9450 con 0x5633eac9dc60 2024-05-27T07:24:54.566 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:54.572+0000 7fc37ad9d700 1 --2- 172.21.15.52:0/2934944619 >> [v2:172.21.15.173:6836/1396363827,v1:172.21.15.173:6837/1396363827] conn(0x5633eac9dc60 0x5633ead0bfe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:24:54.566 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:54.572+0000 7fc3627fc700 10 client.15087 ms_handle_connect on v2:172.21.15.173:6836/1396363827 2024-05-27T07:24:54.566 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:54.572+0000 7fc37ad9d700 1 --2- 172.21.15.52:0/2934944619 >> [v2:172.21.15.173:6836/1396363827,v1:172.21.15.173:6837/1396363827] conn(0x5633eac9dc60 0x5633ead0bfe0 crc :-1 s=READY pgs=16 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=85649ffc0cf68737 server_cookie=e02b54aef8e18b4 in_seq=0 out_seq=0 2024-05-27T07:24:54.566 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:54.573+0000 7fc3627fc700 1 -- 172.21.15.52:0/2934944619 <== mds.4 v2:172.21.15.173:6836/1396363827 1 ==== command_reply(tid 0: 0 ) ==== 8+0+327 (crc 0 0 0) 0x5633eaba9450 con 0x5633eac9dc60 2024-05-27T07:24:54.566 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:54.573+0000 7fc3627fc700 10 client.15087 handle_command_reply: tid=0 2024-05-27T07:24:54.566 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:54.573+0000 7fc38fc4fb80 1 client.15087 shutdown 2024-05-27T07:24:54.566 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:54.573+0000 7fc38fc4fb80 1 -- 172.21.15.52:0/2934944619 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x5633eac84780 msgr2=0x5633eaca5ba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:24:54.566 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:54.573+0000 7fc38fc4fb80 1 --2- 172.21.15.52:0/2934944619 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x5633eac84780 0x5633eaca5ba0 secure :-1 s=READY pgs=315 cs=0 l=1 rev1=1 crypto rx=0x7fc364004f20 tx=0x7fc36400ff40 comp rx=0 tx=0).stop 2024-05-27T07:24:54.566 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:54.573+0000 7fc38fc4fb80 1 -- 172.21.15.52:0/2934944619 shutdown_connections 2024-05-27T07:24:54.566 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:54.573+0000 7fc38fc4fb80 1 --2- 172.21.15.52:0/2934944619 >> [v2:172.21.15.173:6836/1396363827,v1:172.21.15.173:6837/1396363827] conn(0x5633eac9dc60 0x5633ead0bfe0 crc :-1 s=READY pgs=16 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:24:54.567 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:54.573+0000 7fc38fc4fb80 1 --2- 172.21.15.52:0/2934944619 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5633eac88f60 0x5633ead073f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:24:54.567 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:54.573+0000 7fc38fc4fb80 1 --2- 172.21.15.52:0/2934944619 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5633eab00df0 0x5633eaca5660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:24:54.567 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:54.573+0000 7fc38fc4fb80 1 --2- 172.21.15.52:0/2934944619 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x5633eac84780 0x5633eaca5ba0 unknown :-1 s=CLOSED pgs=315 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:24:54.567 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:54.573+0000 7fc38fc4fb80 1 -- 172.21.15.52:0/2934944619 >> 172.21.15.52:0/2934944619 conn(0x5633eab03720 msgr2=0x5633eac7c0c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:24:54.567 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:54.573+0000 7fc3627fc700 0 client.15087 ms_handle_reset on v2:172.21.15.173:6836/1396363827 2024-05-27T07:24:54.567 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:54.573+0000 7fc38fc4fb80 1 -- 172.21.15.52:0/2934944619 shutdown_connections 2024-05-27T07:24:54.567 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:54.573+0000 7fc38fc4fb80 1 -- 172.21.15.52:0/2934944619 wait complete. 2024-05-27T07:24:54.567 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:54.573+0000 7fc38fc4fb80 20 client.15087 trim_cache size 0 max 16384 2024-05-27T07:24:54.567 INFO:teuthology.orchestra.run.smithi052.stdout:{ 2024-05-27T07:24:54.567 INFO:teuthology.orchestra.run.smithi052.stdout: "cluster_fsid": "73df2636-1bf8-11ef-bc9b-c7b262605968", 2024-05-27T07:24:54.567 INFO:teuthology.orchestra.run.smithi052.stdout: "whoami": 4, 2024-05-27T07:24:54.568 INFO:teuthology.orchestra.run.smithi052.stdout: "id": 24466, 2024-05-27T07:24:54.568 INFO:teuthology.orchestra.run.smithi052.stdout: "want_state": "up:active", 2024-05-27T07:24:54.568 INFO:teuthology.orchestra.run.smithi052.stdout: "state": "up:active", 2024-05-27T07:24:54.568 INFO:teuthology.orchestra.run.smithi052.stdout: "fs_name": "cephfs", 2024-05-27T07:24:54.568 INFO:teuthology.orchestra.run.smithi052.stdout: "rank_uptime": 118.694197911, 2024-05-27T07:24:54.568 INFO:teuthology.orchestra.run.smithi052.stdout: "mdsmap_epoch": 29, 2024-05-27T07:24:54.568 INFO:teuthology.orchestra.run.smithi052.stdout: "osdmap_epoch": 90, 2024-05-27T07:24:54.568 INFO:teuthology.orchestra.run.smithi052.stdout: "osdmap_epoch_barrier": 88, 2024-05-27T07:24:54.568 INFO:teuthology.orchestra.run.smithi052.stdout: "uptime": 213.84540272000001 2024-05-27T07:24:54.568 INFO:teuthology.orchestra.run.smithi052.stdout:} 2024-05-27T07:24:54.569 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:54.574+0000 7fc38123a700 1 -- 172.21.15.52:0/3346152711 >> [v2:172.21.15.52:6800/3513075805,v1:172.21.15.52:6801/3513075805] conn(0x7fc3680b2900 msgr2=0x7fc3680b4e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:24:54.569 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:54.574+0000 7fc38123a700 1 --2- 172.21.15.52:0/3346152711 >> [v2:172.21.15.52:6800/3513075805,v1:172.21.15.52:6801/3513075805] conn(0x7fc3680b2900 0x7fc3680b4e10 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7fc37c1023c0 tx=0x7fc36400a380 comp rx=0 tx=0).stop 2024-05-27T07:24:54.569 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:54.574+0000 7fc38123a700 1 -- 172.21.15.52:0/3346152711 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fc37c1034b0 msgr2=0x7fc37c19f8c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:24:54.569 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:54.574+0000 7fc38123a700 1 --2- 172.21.15.52:0/3346152711 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fc37c1034b0 0x7fc37c19f8c0 secure :-1 s=READY pgs=311 cs=0 l=1 rev1=1 crypto rx=0x7fc36c007a20 tx=0x7fc36c007a50 comp rx=0 tx=0).stop 2024-05-27T07:24:54.569 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:54.574+0000 7fc38123a700 1 -- 172.21.15.52:0/3346152711 shutdown_connections 2024-05-27T07:24:54.569 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:54.574+0000 7fc38123a700 1 --2- 172.21.15.52:0/3346152711 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fc37c108730 0x7fc37c199b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:24:54.569 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:54.574+0000 7fc38123a700 1 --2- 172.21.15.52:0/3346152711 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fc37c1034b0 0x7fc37c19f8c0 unknown :-1 s=CLOSED pgs=311 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:24:54.569 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:54.574+0000 7fc38123a700 1 --2- 172.21.15.52:0/3346152711 >> [v2:172.21.15.52:6800/3513075805,v1:172.21.15.52:6801/3513075805] conn(0x7fc3680b2900 0x7fc3680b4e10 unknown :-1 s=CLOSED pgs=207 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:24:54.570 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:54.574+0000 7fc38123a700 1 --2- 172.21.15.52:0/3346152711 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fc37c107de0 0x7fc37c19fe20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:24:54.570 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:54.574+0000 7fc38123a700 1 -- 172.21.15.52:0/3346152711 >> 172.21.15.52:0/3346152711 conn(0x7fc37c0fedb0 msgr2=0x7fc37c109690 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:24:54.570 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:54.574+0000 7fc38123a700 1 -- 172.21.15.52:0/3346152711 shutdown_connections 2024-05-27T07:24:54.570 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:54.574+0000 7fc38123a700 1 -- 172.21.15.52:0/3346152711 wait complete. 2024-05-27T07:24:54.979 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:24:54 smithi052 ceph-mon[31955]: pgmap v392: 129 pgs: 129 active+clean; 647 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 9 op/s 2024-05-27T07:24:55.102 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:24:54 smithi173 ceph-mon[37000]: pgmap v392: 129 pgs: 129 active+clean; 647 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 9 op/s 2024-05-27T07:24:55.190 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:24:54 smithi105 ceph-mon[35977]: pgmap v392: 129 pgs: 129 active+clean; 647 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 9 op/s 2024-05-27T07:24:55.352 DEBUG:teuthology.orchestra.run.smithi052:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:38f9d900c83c63799fbdbe61acc9a11b0d3554a6 shell --fsid 73df2636-1bf8-11ef-bc9b-c7b262605968 -- ceph tell mds.e status 2024-05-27T07:24:55.611 INFO:teuthology.orchestra.run.smithi052.stderr:Inferring config /var/lib/ceph/73df2636-1bf8-11ef-bc9b-c7b262605968/mon.a/config 2024-05-27T07:24:56.788 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:24:56 smithi052 ceph-mon[31955]: pgmap v393: 129 pgs: 129 active+clean; 647 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 11 op/s 2024-05-27T07:24:57.102 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:24:56 smithi173 ceph-mon[37000]: pgmap v393: 129 pgs: 129 active+clean; 647 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 11 op/s 2024-05-27T07:24:57.190 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:24:56 smithi105 ceph-mon[35977]: pgmap v393: 129 pgs: 129 active+clean; 647 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 11 op/s 2024-05-27T07:24:57.318 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:57.328+0000 7f2be8f65700 1 -- 172.21.15.52:0/2150347235 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f2be4107de0 msgr2=0x7f2be41081f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:24:57.318 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:57.328+0000 7f2be8f65700 1 --2- 172.21.15.52:0/2150347235 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f2be4107de0 0x7f2be41081f0 secure :-1 s=READY pgs=316 cs=0 l=1 rev1=1 crypto rx=0x7f2bcc0099c0 tx=0x7f2bcc009cd0 comp rx=0 tx=0).stop 2024-05-27T07:24:57.319 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:57.328+0000 7f2be8f65700 1 -- 172.21.15.52:0/2150347235 shutdown_connections 2024-05-27T07:24:57.319 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:57.328+0000 7f2be8f65700 1 --2- 172.21.15.52:0/2150347235 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f2be4108730 0x7f2be4110090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:24:57.319 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:57.328+0000 7f2be8f65700 1 --2- 172.21.15.52:0/2150347235 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f2be41034b0 0x7f2be41038c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:24:57.319 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:57.328+0000 7f2be8f65700 1 --2- 172.21.15.52:0/2150347235 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f2be4107de0 0x7f2be41081f0 unknown :-1 s=CLOSED pgs=316 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:24:57.319 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:57.328+0000 7f2be8f65700 1 -- 172.21.15.52:0/2150347235 >> 172.21.15.52:0/2150347235 conn(0x7f2be40fede0 msgr2=0x7f2be4101200 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:24:57.319 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:57.328+0000 7f2be8f65700 1 -- 172.21.15.52:0/2150347235 shutdown_connections 2024-05-27T07:24:57.319 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:57.328+0000 7f2be8f65700 1 -- 172.21.15.52:0/2150347235 wait complete. 2024-05-27T07:24:57.321 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:57.329+0000 7f2be8f65700 1 Processor -- start 2024-05-27T07:24:57.321 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:57.329+0000 7f2be8f65700 1 -- start start 2024-05-27T07:24:57.321 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:57.330+0000 7f2be8f65700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f2be41034b0 0x7f2be40ff2c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:24:57.321 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:57.330+0000 7f2be8f65700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f2be4107de0 0x7f2be40ff800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:24:57.322 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:57.330+0000 7f2be8f65700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f2be4108730 0x7f2be40ffd40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:24:57.322 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:57.330+0000 7f2be8f65700 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x7f2be4113340 con 0x7f2be41034b0 2024-05-27T07:24:57.322 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:57.330+0000 7f2be8f65700 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f2be41131c0 con 0x7f2be4108730 2024-05-27T07:24:57.322 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:57.330+0000 7f2be8f65700 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f2be41134c0 con 0x7f2be4107de0 2024-05-27T07:24:57.322 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:57.330+0000 7f2be259c700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f2be41034b0 0x7f2be40ff2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:24:57.322 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:57.330+0000 7f2be259c700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f2be41034b0 0x7f2be40ff2c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.52:33046/0 (socket says 172.21.15.52:33046) 2024-05-27T07:24:57.322 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:57.330+0000 7f2be259c700 1 -- 172.21.15.52:0/3816766947 learned_addr learned my addr 172.21.15.52:0/3816766947 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-05-27T07:24:57.322 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:57.330+0000 7f2bd9d9b700 1 --2- 172.21.15.52:0/3816766947 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f2be4107de0 0x7f2be40ff800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:24:57.322 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:57.330+0000 7f2be2d9d700 1 --2- 172.21.15.52:0/3816766947 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f2be4108730 0x7f2be40ffd40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:24:57.323 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:57.330+0000 7f2be259c700 1 -- 172.21.15.52:0/3816766947 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f2be4107de0 msgr2=0x7f2be40ff800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:24:57.323 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:57.330+0000 7f2be259c700 1 --2- 172.21.15.52:0/3816766947 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f2be4107de0 0x7f2be40ff800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:24:57.323 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:57.330+0000 7f2be259c700 1 -- 172.21.15.52:0/3816766947 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f2be4108730 msgr2=0x7f2be40ffd40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:24:57.323 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:57.330+0000 7f2be259c700 1 --2- 172.21.15.52:0/3816766947 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f2be4108730 0x7f2be40ffd40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:24:57.323 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:57.330+0000 7f2be259c700 1 -- 172.21.15.52:0/3816766947 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2be41a91a0 con 0x7f2be41034b0 2024-05-27T07:24:57.323 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:57.330+0000 7f2be2d9d700 1 --2- 172.21.15.52:0/3816766947 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f2be4108730 0x7f2be40ffd40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-27T07:24:57.323 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:57.330+0000 7f2be259c700 1 --2- 172.21.15.52:0/3816766947 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f2be41034b0 0x7f2be40ff2c0 secure :-1 s=READY pgs=317 cs=0 l=1 rev1=1 crypto rx=0x7f2bd400d950 tx=0x7f2bd400dc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:24:57.323 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:57.331+0000 7f2bdbfff700 1 -- 172.21.15.52:0/3816766947 <== mon.0 v2:172.21.15.52:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2bd4016070 con 0x7f2be41034b0 2024-05-27T07:24:57.323 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:57.331+0000 7f2be8f65700 1 -- 172.21.15.52:0/3816766947 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2be41a9490 con 0x7f2be41034b0 2024-05-27T07:24:57.323 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:57.331+0000 7f2bdbfff700 1 -- 172.21.15.52:0/3816766947 <== mon.0 v2:172.21.15.52:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f2bd4009df0 con 0x7f2be41034b0 2024-05-27T07:24:57.323 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:57.331+0000 7f2bdbfff700 1 -- 172.21.15.52:0/3816766947 <== mon.0 v2:172.21.15.52:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2bd400f3c0 con 0x7f2be41034b0 2024-05-27T07:24:57.323 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:57.331+0000 7f2be8f65700 1 -- 172.21.15.52:0/3816766947 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2be41a99f0 con 0x7f2be41034b0 2024-05-27T07:24:57.324 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:57.334+0000 7f2bdbfff700 1 -- 172.21.15.52:0/3816766947 <== mon.0 v2:172.21.15.52:3300/0 4 ==== mgrmap(e 18) ==== 149652+0+0 (secure 0 0 0) 0x7f2bd4019070 con 0x7f2be41034b0 2024-05-27T07:24:57.325 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:57.335+0000 7f2bdbfff700 1 --2- 172.21.15.52:0/3816766947 >> [v2:172.21.15.52:6800/3513075805,v1:172.21.15.52:6801/3513075805] conn(0x7f2bd00b2880 0x7f2bd00b4d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:24:57.325 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:57.335+0000 7f2bd9d9b700 1 --2- 172.21.15.52:0/3816766947 >> [v2:172.21.15.52:6800/3513075805,v1:172.21.15.52:6801/3513075805] conn(0x7f2bd00b2880 0x7f2bd00b4d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:24:57.325 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:57.335+0000 7f2bdbfff700 1 -- 172.21.15.52:0/3816766947 <== mon.0 v2:172.21.15.52:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9121+0+0 (secure 0 0 0) 0x7f2bd40e21b0 con 0x7f2be41034b0 2024-05-27T07:24:57.325 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:57.335+0000 7f2bf797ab80 1 Processor -- start 2024-05-27T07:24:57.327 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:57.336+0000 7f2bf797ab80 1 -- start start 2024-05-27T07:24:57.327 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:57.336+0000 7f2bd9d9b700 1 --2- 172.21.15.52:0/3816766947 >> [v2:172.21.15.52:6800/3513075805,v1:172.21.15.52:6801/3513075805] conn(0x7f2bd00b2880 0x7f2bd00b4d90 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7f2bcc006010 tx=0x7f2bcc000bc0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:24:57.327 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:57.336+0000 7f2bf797ab80 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5570b8059df0 0x5570b81deae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:24:57.327 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:57.336+0000 7f2bf797ab80 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5570b81df020 0x5570b81d8b90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:24:57.327 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:57.336+0000 7f2bf797ab80 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x5570b81d90d0 0x5570b81d94e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:24:57.327 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:57.336+0000 7f2bf797ab80 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x5570b8106d20 con 0x5570b81d90d0 2024-05-27T07:24:57.327 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:57.336+0000 7f2bf797ab80 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x5570b80f68c0 con 0x5570b81df020 2024-05-27T07:24:57.327 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:57.336+0000 7f2bf797ab80 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x5570b80fa900 con 0x5570b8059df0 2024-05-27T07:24:57.327 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:57.336+0000 7f2be2d9d700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x5570b81d90d0 0x5570b81d94e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:24:57.327 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:57.336+0000 7f2be2d9d700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x5570b81d90d0 0x5570b81d94e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.52:33062/0 (socket says 172.21.15.52:33062) 2024-05-27T07:24:57.328 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:57.336+0000 7f2be2d9d700 1 -- 172.21.15.52:0/1907725822 learned_addr learned my addr 172.21.15.52:0/1907725822 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-05-27T07:24:57.328 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:57.336+0000 7f2bd9d9b700 1 --2- 172.21.15.52:0/1907725822 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5570b81df020 0x5570b81d8b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:24:57.328 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:57.336+0000 7f2be259c700 1 --2- 172.21.15.52:0/1907725822 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5570b8059df0 0x5570b81deae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:24:57.328 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:57.336+0000 7f2be2d9d700 1 -- 172.21.15.52:0/1907725822 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5570b8059df0 msgr2=0x5570b81deae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:24:57.328 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:57.336+0000 7f2be2d9d700 1 --2- 172.21.15.52:0/1907725822 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5570b8059df0 0x5570b81deae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:24:57.328 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:57.336+0000 7f2be2d9d700 1 -- 172.21.15.52:0/1907725822 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5570b81df020 msgr2=0x5570b81d8b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:24:57.328 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:57.336+0000 7f2be2d9d700 1 --2- 172.21.15.52:0/1907725822 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5570b81df020 0x5570b81d8b90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:24:57.328 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:57.336+0000 7f2be2d9d700 1 -- 172.21.15.52:0/1907725822 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5570b809ff60 con 0x5570b81d90d0 2024-05-27T07:24:57.328 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:57.337+0000 7f2be2d9d700 1 --2- 172.21.15.52:0/1907725822 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x5570b81d90d0 0x5570b81d94e0 secure :-1 s=READY pgs=318 cs=0 l=1 rev1=1 crypto rx=0x7f2bdc009dd0 tx=0x7f2bdc0077e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f109bc9ba857e0d4 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:24:57.328 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:57.337+0000 7f2bd8d99700 1 -- 172.21.15.52:0/1907725822 <== mon.0 v2:172.21.15.52:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2bdc014610 con 0x5570b81d90d0 2024-05-27T07:24:57.329 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:57.337+0000 7f2bd8d99700 1 -- 172.21.15.52:0/1907725822 <== mon.0 v2:172.21.15.52:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f2bdc019070 con 0x5570b81d90d0 2024-05-27T07:24:57.329 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:57.337+0000 7f2bd8d99700 1 -- 172.21.15.52:0/1907725822 <== mon.0 v2:172.21.15.52:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2bdc013970 con 0x5570b81d90d0 2024-05-27T07:24:57.333 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:57.338+0000 7f2bf797ab80 1 -- 172.21.15.52:0/1907725822 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x5570b81d90d0 msgr2=0x5570b81d94e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:24:57.333 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:57.338+0000 7f2bf797ab80 1 --2- 172.21.15.52:0/1907725822 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x5570b81d90d0 0x5570b81d94e0 secure :-1 s=READY pgs=318 cs=0 l=1 rev1=1 crypto rx=0x7f2bdc009dd0 tx=0x7f2bdc0077e0 comp rx=0 tx=0).stop 2024-05-27T07:24:57.333 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:57.338+0000 7f2bf797ab80 1 -- 172.21.15.52:0/1907725822 shutdown_connections 2024-05-27T07:24:57.333 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:57.338+0000 7f2bf797ab80 1 --2- 172.21.15.52:0/1907725822 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x5570b81d90d0 0x5570b81d94e0 unknown :-1 s=CLOSED pgs=318 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:24:57.333 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:57.338+0000 7f2bf797ab80 1 --2- 172.21.15.52:0/1907725822 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5570b8059df0 0x5570b81deae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:24:57.333 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:57.338+0000 7f2bf797ab80 1 --2- 172.21.15.52:0/1907725822 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5570b81df020 0x5570b81d8b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:24:57.333 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:57.338+0000 7f2bf797ab80 1 -- 172.21.15.52:0/1907725822 >> 172.21.15.52:0/1907725822 conn(0x5570b805c720 msgr2=0x5570b81d3d40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:24:57.333 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:57.338+0000 7f2bf797ab80 1 -- 172.21.15.52:0/1907725822 shutdown_connections 2024-05-27T07:24:57.333 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:57.338+0000 7f2bf797ab80 1 -- 172.21.15.52:0/1907725822 wait complete. 2024-05-27T07:24:57.334 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:57.338+0000 7f2bf797ab80 1 -- start start 2024-05-27T07:24:57.334 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:57.338+0000 7f2bf797ab80 1 Processor -- start 2024-05-27T07:24:57.334 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:57.339+0000 7f2bf797ab80 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5570b8059df0 0x5570b81e2810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:24:57.334 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:57.339+0000 7f2bf797ab80 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x5570b81d90d0 0x5570b81e2d50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:24:57.334 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:57.339+0000 7f2bf797ab80 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5570b81df020 0x5570b81e3290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:24:57.334 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:57.339+0000 7f2bf797ab80 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x5570b7ffcf50 con 0x5570b81d90d0 2024-05-27T07:24:57.334 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:57.339+0000 7f2bf797ab80 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x5570b805ed50 con 0x5570b8059df0 2024-05-27T07:24:57.334 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:57.339+0000 7f2bf797ab80 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x5570b81e5b10 con 0x5570b81df020 2024-05-27T07:24:57.334 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:57.339+0000 7f2bf797ab80 10 client.0 _ll_register_callbacks cb 0x5570b808d560 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T07:24:57.334 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:57.339+0000 7f2bd9d9b700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x5570b81d90d0 0x5570b81e2d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:24:57.335 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:57.339+0000 7f2bd9d9b700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x5570b81d90d0 0x5570b81e2d50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.52:33066/0 (socket says 172.21.15.52:33066) 2024-05-27T07:24:57.335 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:57.339+0000 7f2bd9d9b700 1 -- 172.21.15.52:0/2320218793 learned_addr learned my addr 172.21.15.52:0/2320218793 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-05-27T07:24:57.335 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:57.339+0000 7f2be2d9d700 1 --2- 172.21.15.52:0/2320218793 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5570b81df020 0x5570b81e3290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:24:57.335 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:57.339+0000 7f2be259c700 1 --2- 172.21.15.52:0/2320218793 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5570b8059df0 0x5570b81e2810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:24:57.335 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:57.339+0000 7f2be2d9d700 1 -- 172.21.15.52:0/2320218793 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5570b8059df0 msgr2=0x5570b81e2810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:24:57.335 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:57.339+0000 7f2be2d9d700 1 --2- 172.21.15.52:0/2320218793 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5570b8059df0 0x5570b81e2810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:24:57.335 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:57.339+0000 7f2be2d9d700 1 -- 172.21.15.52:0/2320218793 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x5570b81d90d0 msgr2=0x5570b81e2d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:24:57.335 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:57.339+0000 7f2be2d9d700 1 --2- 172.21.15.52:0/2320218793 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x5570b81d90d0 0x5570b81e2d50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:24:57.335 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:57.339+0000 7f2be2d9d700 1 -- 172.21.15.52:0/2320218793 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5570b8046880 con 0x5570b81df020 2024-05-27T07:24:57.335 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:57.339+0000 7f2bd9d9b700 1 --2- 172.21.15.52:0/2320218793 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x5570b81d90d0 0x5570b81e2d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-27T07:24:57.335 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:57.340+0000 7f2be2d9d700 1 --2- 172.21.15.52:0/2320218793 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5570b81df020 0x5570b81e3290 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f2bdc000f80 tx=0x7f2bdc013900 comp rx=0 tx=0).ready entity=mon.1 client_cookie=34f1a9b677b42bdc server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:24:57.335 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:57.340+0000 7f2bc9ffb700 10 client.0 ms_handle_connect on v2:172.21.15.173:3300/0 2024-05-27T07:24:57.335 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:57.340+0000 7f2bc9ffb700 1 -- 172.21.15.52:0/2320218793 <== mon.1 v2:172.21.15.173:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2bdc003a00 con 0x5570b81df020 2024-05-27T07:24:57.336 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:57.340+0000 7f2bf797ab80 1 -- 172.21.15.52:0/2320218793 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f2bdc019070 con 0x5570b81df020 2024-05-27T07:24:57.336 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:57.340+0000 7f2bc9ffb700 1 -- 172.21.15.52:0/2320218793 <== mon.1 v2:172.21.15.173:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9121+0+0 (secure 0 0 0) 0x7f2bdc003ba0 con 0x5570b81df020 2024-05-27T07:24:57.336 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:57.340+0000 7f2bc9ffb700 1 -- 172.21.15.52:0/2320218793 <== mon.1 v2:172.21.15.173:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f2bdc02e050 con 0x5570b81df020 2024-05-27T07:24:57.336 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:57.340+0000 7f2bf797ab80 10 client.24862 fetch_fsmap learned FSMap version 29 2024-05-27T07:24:57.336 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:57.340+0000 7f2bf797ab80 1 -- 172.21.15.52:0/2320218793 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x5570b80b0710 con 0x5570b81df020 2024-05-27T07:24:57.336 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:57.340+0000 7f2bc9ffb700 1 -- 172.21.15.52:0/2320218793 <== mon.1 v2:172.21.15.173:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f2bdc019070 con 0x5570b81df020 2024-05-27T07:24:57.336 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:57.340+0000 7f2bc9ffb700 1 -- 172.21.15.52:0/2320218793 <== mon.1 v2:172.21.15.173:3300/0 5 ==== fsmap(e 29) ==== 8362+0+0 (secure 0 0 0) 0x7f2bdc004670 con 0x5570b81df020 2024-05-27T07:24:57.336 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:57.341+0000 7f2bf797ab80 10 client.24862 fetch_fsmap finished waiting for FSMap version 29 2024-05-27T07:24:57.336 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:57.341+0000 7f2bf797ab80 10 client.24862 resolve_mds: resolved name 'e' to daemon mds.e 2024-05-27T07:24:57.336 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:57.341+0000 7f2bf797ab80 20 client.24862 populate_metadata read hostname 'smithi052' 2024-05-27T07:24:57.336 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:57.341+0000 7f2bf797ab80 1 --2- 172.21.15.52:0/2320218793 >> [v2:172.21.15.105:6834/3298498384,v1:172.21.15.105:6835/3298498384] conn(0x5570b81f2640 0x5570b81f3ab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:24:57.336 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:57.341+0000 7f2bf797ab80 4 client.24862 mds_command: new command op to 24473 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-27T07:24:57.337 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:57.341+0000 7f2bf797ab80 1 -- 172.21.15.52:0/2320218793 --> [v2:172.21.15.105:6834/3298498384,v1:172.21.15.105:6835/3298498384] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5570b8108c70 con 0x5570b81f2640 2024-05-27T07:24:57.337 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:57.341+0000 7f2be259c700 1 --2- 172.21.15.52:0/2320218793 >> [v2:172.21.15.105:6834/3298498384,v1:172.21.15.105:6835/3298498384] conn(0x5570b81f2640 0x5570b81f3ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:24:57.337 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:57.341+0000 7f2be259c700 1 --2- 172.21.15.52:0/2320218793 >> [v2:172.21.15.105:6834/3298498384,v1:172.21.15.105:6835/3298498384] conn(0x5570b81f2640 0x5570b81f3ab0 crc :-1 s=READY pgs=14 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=ccce3ba53c260f43 server_cookie=6b4453b57897c93d in_seq=0 out_seq=0 2024-05-27T07:24:57.337 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:57.341+0000 7f2bc9ffb700 10 client.24862 ms_handle_connect on v2:172.21.15.105:6834/3298498384 2024-05-27T07:24:57.337 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:57.342+0000 7f2bc9ffb700 1 -- 172.21.15.52:0/2320218793 <== mds.3 v2:172.21.15.105:6834/3298498384 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24041 (crc 0 0 0) 0x5570b8108c70 con 0x5570b81f2640 2024-05-27T07:24:57.337 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:57.342+0000 7f2bc9ffb700 10 client.24862 handle_command_reply: tid=0 2024-05-27T07:24:57.337 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:57.342+0000 7f2bf797ab80 1 client.24862 shutdown 2024-05-27T07:24:57.337 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:57.342+0000 7f2bf797ab80 1 -- 172.21.15.52:0/2320218793 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5570b81df020 msgr2=0x5570b81e3290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:24:57.337 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:57.342+0000 7f2bf797ab80 1 --2- 172.21.15.52:0/2320218793 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5570b81df020 0x5570b81e3290 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f2bdc000f80 tx=0x7f2bdc013900 comp rx=0 tx=0).stop 2024-05-27T07:24:57.337 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:57.342+0000 7f2bf797ab80 1 -- 172.21.15.52:0/2320218793 shutdown_connections 2024-05-27T07:24:57.337 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:57.342+0000 7f2bf797ab80 1 --2- 172.21.15.52:0/2320218793 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5570b81df020 0x5570b81e3290 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:24:57.337 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:57.343+0000 7f2bf797ab80 1 --2- 172.21.15.52:0/2320218793 >> [v2:172.21.15.105:6834/3298498384,v1:172.21.15.105:6835/3298498384] conn(0x5570b81f2640 0x5570b81f3ab0 crc :-1 s=READY pgs=14 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:24:57.337 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:57.343+0000 7f2bf797ab80 1 --2- 172.21.15.52:0/2320218793 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5570b8059df0 0x5570b81e2810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:24:57.338 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:57.343+0000 7f2bf797ab80 1 --2- 172.21.15.52:0/2320218793 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x5570b81d90d0 0x5570b81e2d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:24:57.338 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:57.343+0000 7f2bc9ffb700 0 client.24862 ms_handle_reset on v2:172.21.15.105:6834/3298498384 2024-05-27T07:24:57.338 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:57.343+0000 7f2bf797ab80 1 -- 172.21.15.52:0/2320218793 >> 172.21.15.52:0/2320218793 conn(0x5570b805c720 msgr2=0x5570b81d7b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:24:57.338 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:57.343+0000 7f2bf797ab80 1 -- 172.21.15.52:0/2320218793 shutdown_connections 2024-05-27T07:24:57.338 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:57.343+0000 7f2bf797ab80 1 -- 172.21.15.52:0/2320218793 wait complete. 2024-05-27T07:24:57.338 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:57.343+0000 7f2bf797ab80 20 client.24862 trim_cache size 0 max 16384 2024-05-27T07:24:57.355 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:57.364+0000 7f2bf797ab80 1 Processor -- start 2024-05-27T07:24:57.355 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:57.364+0000 7f2bf797ab80 1 -- start start 2024-05-27T07:24:57.355 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:57.364+0000 7f2bf797ab80 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5570b8059df0 0x5570b81d8b90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:24:57.355 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:57.364+0000 7f2bf797ab80 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x5570b81d90d0 0x5570b81de2d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:24:57.355 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:57.364+0000 7f2bf797ab80 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5570b81df020 0x5570b81eedb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:24:57.355 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:57.364+0000 7f2bf797ab80 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x5570b81f1df0 con 0x5570b81d90d0 2024-05-27T07:24:57.355 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:57.364+0000 7f2bf797ab80 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x5570b81f24c0 con 0x5570b81df020 2024-05-27T07:24:57.355 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:57.364+0000 7f2bf797ab80 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x5570b81f2b90 con 0x5570b8059df0 2024-05-27T07:24:57.355 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:57.364+0000 7f2bd9d9b700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x5570b81d90d0 0x5570b81de2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:24:57.355 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:57.364+0000 7f2bd9d9b700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x5570b81d90d0 0x5570b81de2d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.52:33078/0 (socket says 172.21.15.52:33078) 2024-05-27T07:24:57.356 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:57.364+0000 7f2bd9d9b700 1 -- 172.21.15.52:0/673571721 learned_addr learned my addr 172.21.15.52:0/673571721 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-05-27T07:24:57.356 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:57.365+0000 7f2be259c700 1 --2- 172.21.15.52:0/673571721 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5570b8059df0 0x5570b81d8b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:24:57.356 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:57.365+0000 7f2be2d9d700 1 --2- 172.21.15.52:0/673571721 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5570b81df020 0x5570b81eedb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:24:57.356 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:57.365+0000 7f2bd9d9b700 1 -- 172.21.15.52:0/673571721 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5570b8059df0 msgr2=0x5570b81d8b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:24:57.356 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:57.365+0000 7f2bd9d9b700 1 --2- 172.21.15.52:0/673571721 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5570b8059df0 0x5570b81d8b90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:24:57.356 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:57.365+0000 7f2bd9d9b700 1 -- 172.21.15.52:0/673571721 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5570b81df020 msgr2=0x5570b81eedb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:24:57.356 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:57.365+0000 7f2bd9d9b700 1 --2- 172.21.15.52:0/673571721 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5570b81df020 0x5570b81eedb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:24:57.356 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:57.365+0000 7f2bd9d9b700 1 -- 172.21.15.52:0/673571721 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5570b81f3080 con 0x5570b81d90d0 2024-05-27T07:24:57.356 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:57.365+0000 7f2bd9d9b700 1 --2- 172.21.15.52:0/673571721 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x5570b81d90d0 0x5570b81de2d0 secure :-1 s=READY pgs=319 cs=0 l=1 rev1=1 crypto rx=0x7f2bcc009f90 tx=0x7f2bcc01e000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=58870ca33282b32b server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:24:57.356 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:57.365+0000 7f2bcbfff700 1 -- 172.21.15.52:0/673571721 <== mon.0 v2:172.21.15.52:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2bcc00bd70 con 0x5570b81d90d0 2024-05-27T07:24:57.356 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:57.365+0000 7f2bcbfff700 1 -- 172.21.15.52:0/673571721 <== mon.0 v2:172.21.15.52:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f2bcc00f910 con 0x5570b81d90d0 2024-05-27T07:24:57.357 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:57.365+0000 7f2bcbfff700 1 -- 172.21.15.52:0/673571721 <== mon.0 v2:172.21.15.52:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2bcc023660 con 0x5570b81d90d0 2024-05-27T07:24:57.360 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:57.366+0000 7f2bf797ab80 1 -- 172.21.15.52:0/673571721 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x5570b81d90d0 msgr2=0x5570b81de2d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:24:57.360 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:57.366+0000 7f2bf797ab80 1 --2- 172.21.15.52:0/673571721 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x5570b81d90d0 0x5570b81de2d0 secure :-1 s=READY pgs=319 cs=0 l=1 rev1=1 crypto rx=0x7f2bcc009f90 tx=0x7f2bcc01e000 comp rx=0 tx=0).stop 2024-05-27T07:24:57.360 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:57.366+0000 7f2bf797ab80 1 -- 172.21.15.52:0/673571721 shutdown_connections 2024-05-27T07:24:57.360 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:57.366+0000 7f2bf797ab80 1 --2- 172.21.15.52:0/673571721 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5570b81df020 0x5570b81eedb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:24:57.360 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:57.366+0000 7f2bf797ab80 1 --2- 172.21.15.52:0/673571721 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5570b8059df0 0x5570b81d8b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:24:57.360 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:57.366+0000 7f2bf797ab80 1 --2- 172.21.15.52:0/673571721 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x5570b81d90d0 0x5570b81de2d0 unknown :-1 s=CLOSED pgs=319 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:24:57.360 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:57.366+0000 7f2bf797ab80 1 -- 172.21.15.52:0/673571721 >> 172.21.15.52:0/673571721 conn(0x5570b805c720 msgr2=0x5570b81dc440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:24:57.360 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:57.366+0000 7f2bf797ab80 1 -- 172.21.15.52:0/673571721 shutdown_connections 2024-05-27T07:24:57.360 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:57.366+0000 7f2bf797ab80 1 -- 172.21.15.52:0/673571721 wait complete. 2024-05-27T07:24:57.361 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:57.366+0000 7f2bf797ab80 1 -- start start 2024-05-27T07:24:57.361 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:57.366+0000 7f2bf797ab80 1 Processor -- start 2024-05-27T07:24:57.361 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:57.367+0000 7f2bf797ab80 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5570b8059df0 0x5570b81f8700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:24:57.361 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:57.367+0000 7f2bf797ab80 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5570b81d90d0 0x5570b81f8c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:24:57.361 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:57.367+0000 7f2bf797ab80 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x5570b81df020 0x5570b81fcbf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:24:57.361 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:57.367+0000 7f2bf797ab80 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x5570b81f3930 con 0x5570b81df020 2024-05-27T07:24:57.361 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:57.367+0000 7f2bf797ab80 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x5570b81f3260 con 0x5570b81d90d0 2024-05-27T07:24:57.361 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:57.367+0000 7f2bf797ab80 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x5570b81f4000 con 0x5570b8059df0 2024-05-27T07:24:57.361 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:57.367+0000 7f2bf797ab80 10 client.0 _ll_register_callbacks cb 0x5570b808d560 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T07:24:57.361 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:57.367+0000 7f2be2d9d700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x5570b81df020 0x5570b81fcbf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:24:57.361 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:57.367+0000 7f2be2d9d700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x5570b81df020 0x5570b81fcbf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.52:33080/0 (socket says 172.21.15.52:33080) 2024-05-27T07:24:57.361 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:57.367+0000 7f2be2d9d700 1 -- 172.21.15.52:0/3723814162 learned_addr learned my addr 172.21.15.52:0/3723814162 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-05-27T07:24:57.362 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:57.367+0000 7f2be259c700 1 --2- 172.21.15.52:0/3723814162 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5570b8059df0 0x5570b81f8700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:24:57.362 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:57.367+0000 7f2bd9d9b700 1 --2- 172.21.15.52:0/3723814162 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5570b81d90d0 0x5570b81f8c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:24:57.362 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:57.367+0000 7f2be2d9d700 1 -- 172.21.15.52:0/3723814162 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5570b8059df0 msgr2=0x5570b81f8700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:24:57.362 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:57.367+0000 7f2be2d9d700 1 --2- 172.21.15.52:0/3723814162 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5570b8059df0 0x5570b81f8700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:24:57.362 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:57.367+0000 7f2be2d9d700 1 -- 172.21.15.52:0/3723814162 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5570b81d90d0 msgr2=0x5570b81f8c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:24:57.362 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:57.367+0000 7f2be2d9d700 1 --2- 172.21.15.52:0/3723814162 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5570b81d90d0 0x5570b81f8c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:24:57.362 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:57.367+0000 7f2be2d9d700 1 -- 172.21.15.52:0/3723814162 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5570b81fd1b0 con 0x5570b81df020 2024-05-27T07:24:57.362 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:57.368+0000 7f2be2d9d700 1 --2- 172.21.15.52:0/3723814162 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x5570b81df020 0x5570b81fcbf0 secure :-1 s=READY pgs=320 cs=0 l=1 rev1=1 crypto rx=0x7f2bdc0138c0 tx=0x7f2bdc013970 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f5d4948bd77a62c4 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:24:57.362 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:57.368+0000 7f2bc9ffb700 10 client.0 ms_handle_connect on v2:172.21.15.52:3300/0 2024-05-27T07:24:57.362 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:57.368+0000 7f2bc9ffb700 1 -- 172.21.15.52:0/3723814162 <== mon.0 v2:172.21.15.52:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2bdc037060 con 0x5570b81df020 2024-05-27T07:24:57.362 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:57.368+0000 7f2bf797ab80 1 -- 172.21.15.52:0/3723814162 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f2bcc00f910 con 0x5570b81df020 2024-05-27T07:24:57.363 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:57.368+0000 7f2bc9ffb700 1 -- 172.21.15.52:0/3723814162 <== mon.0 v2:172.21.15.52:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9121+0+0 (secure 0 0 0) 0x7f2bdc014ac0 con 0x5570b81df020 2024-05-27T07:24:57.363 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:57.368+0000 7f2bc9ffb700 1 -- 172.21.15.52:0/3723814162 <== mon.0 v2:172.21.15.52:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f2bdc03e050 con 0x5570b81df020 2024-05-27T07:24:57.363 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:57.368+0000 7f2bf797ab80 10 client.15105 fetch_fsmap learned FSMap version 29 2024-05-27T07:24:57.363 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:57.368+0000 7f2bf797ab80 1 -- 172.21.15.52:0/3723814162 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x5570b825d7a0 con 0x5570b81df020 2024-05-27T07:24:57.363 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:57.368+0000 7f2bc9ffb700 1 -- 172.21.15.52:0/3723814162 <== mon.0 v2:172.21.15.52:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f2bcc00f910 con 0x5570b81df020 2024-05-27T07:24:57.363 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:57.368+0000 7f2bc9ffb700 1 -- 172.21.15.52:0/3723814162 <== mon.0 v2:172.21.15.52:3300/0 5 ==== fsmap(e 29) ==== 8362+0+0 (secure 0 0 0) 0x7f2bdc032430 con 0x5570b81df020 2024-05-27T07:24:57.363 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:57.368+0000 7f2bf797ab80 10 client.15105 fetch_fsmap finished waiting for FSMap version 29 2024-05-27T07:24:57.363 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:57.368+0000 7f2bf797ab80 10 client.15105 resolve_mds: resolved name 'e' to daemon mds.e 2024-05-27T07:24:57.363 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:57.368+0000 7f2bf797ab80 20 client.15105 populate_metadata read hostname 'smithi052' 2024-05-27T07:24:57.363 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:57.368+0000 7f2bf797ab80 1 --2- 172.21.15.52:0/3723814162 >> [v2:172.21.15.105:6834/3298498384,v1:172.21.15.105:6835/3298498384] conn(0x5570b825fb60 0x5570b8260fd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:24:57.363 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:57.369+0000 7f2bf797ab80 4 client.15105 mds_command: new command op to 24473 tid=0 multi_id=0 [{"prefix": "status"}] 2024-05-27T07:24:57.363 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:57.369+0000 7f2bf797ab80 1 -- 172.21.15.52:0/3723814162 --> [v2:172.21.15.105:6834/3298498384,v1:172.21.15.105:6835/3298498384] -- command(tid 0: {"prefix": "status"}) -- 0x5570b8102450 con 0x5570b825fb60 2024-05-27T07:24:57.363 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:57.369+0000 7f2be259c700 1 --2- 172.21.15.52:0/3723814162 >> [v2:172.21.15.105:6834/3298498384,v1:172.21.15.105:6835/3298498384] conn(0x5570b825fb60 0x5570b8260fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:24:57.363 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:57.369+0000 7f2be259c700 1 --2- 172.21.15.52:0/3723814162 >> [v2:172.21.15.105:6834/3298498384,v1:172.21.15.105:6835/3298498384] conn(0x5570b825fb60 0x5570b8260fd0 crc :-1 s=READY pgs=15 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=4707775fda3faf8f server_cookie=3ebfc24a43db0cb3 in_seq=0 out_seq=0 2024-05-27T07:24:57.364 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:57.369+0000 7f2bc9ffb700 10 client.15105 ms_handle_connect on v2:172.21.15.105:6834/3298498384 2024-05-27T07:24:57.364 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:57.369+0000 7f2bc9ffb700 1 -- 172.21.15.52:0/3723814162 <== mds.3 v2:172.21.15.105:6834/3298498384 1 ==== command_reply(tid 0: 0 ) ==== 8+0+322 (crc 0 0 0) 0x5570b8102450 con 0x5570b825fb60 2024-05-27T07:24:57.364 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:57.369+0000 7f2bc9ffb700 10 client.15105 handle_command_reply: tid=0 2024-05-27T07:24:57.364 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:57.370+0000 7f2bf797ab80 1 client.15105 shutdown 2024-05-27T07:24:57.364 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:57.370+0000 7f2bf797ab80 1 -- 172.21.15.52:0/3723814162 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x5570b81df020 msgr2=0x5570b81fcbf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:24:57.364 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:57.370+0000 7f2bf797ab80 1 --2- 172.21.15.52:0/3723814162 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x5570b81df020 0x5570b81fcbf0 secure :-1 s=READY pgs=320 cs=0 l=1 rev1=1 crypto rx=0x7f2bdc0138c0 tx=0x7f2bdc013970 comp rx=0 tx=0).stop 2024-05-27T07:24:57.364 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:57.370+0000 7f2bf797ab80 1 -- 172.21.15.52:0/3723814162 shutdown_connections 2024-05-27T07:24:57.364 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:57.370+0000 7f2bf797ab80 1 --2- 172.21.15.52:0/3723814162 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x5570b81df020 0x5570b81fcbf0 unknown :-1 s=CLOSED pgs=320 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:24:57.364 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:57.370+0000 7f2bf797ab80 1 --2- 172.21.15.52:0/3723814162 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5570b8059df0 0x5570b81f8700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:24:57.364 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:57.370+0000 7f2bf797ab80 1 --2- 172.21.15.52:0/3723814162 >> [v2:172.21.15.105:6834/3298498384,v1:172.21.15.105:6835/3298498384] conn(0x5570b825fb60 0x5570b8260fd0 crc :-1 s=READY pgs=15 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:24:57.364 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:57.370+0000 7f2bf797ab80 1 --2- 172.21.15.52:0/3723814162 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5570b81d90d0 0x5570b81f8c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:24:57.364 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:57.370+0000 7f2bc9ffb700 0 client.15105 ms_handle_reset on v2:172.21.15.105:6834/3298498384 2024-05-27T07:24:57.364 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:57.370+0000 7f2bf797ab80 1 -- 172.21.15.52:0/3723814162 >> 172.21.15.52:0/3723814162 conn(0x5570b805c720 msgr2=0x5570b81de030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:24:57.365 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:57.370+0000 7f2bf797ab80 1 -- 172.21.15.52:0/3723814162 shutdown_connections 2024-05-27T07:24:57.365 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:57.370+0000 7f2bf797ab80 1 -- 172.21.15.52:0/3723814162 wait complete. 2024-05-27T07:24:57.365 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:57.370+0000 7f2bf797ab80 20 client.15105 trim_cache size 0 max 16384 2024-05-27T07:24:57.365 INFO:teuthology.orchestra.run.smithi052.stdout:{ 2024-05-27T07:24:57.365 INFO:teuthology.orchestra.run.smithi052.stdout: "cluster_fsid": "73df2636-1bf8-11ef-bc9b-c7b262605968", 2024-05-27T07:24:57.365 INFO:teuthology.orchestra.run.smithi052.stdout: "whoami": 3, 2024-05-27T07:24:57.365 INFO:teuthology.orchestra.run.smithi052.stdout: "id": 24473, 2024-05-27T07:24:57.366 INFO:teuthology.orchestra.run.smithi052.stdout: "want_state": "up:active", 2024-05-27T07:24:57.366 INFO:teuthology.orchestra.run.smithi052.stdout: "state": "up:active", 2024-05-27T07:24:57.366 INFO:teuthology.orchestra.run.smithi052.stdout: "fs_name": "cephfs", 2024-05-27T07:24:57.366 INFO:teuthology.orchestra.run.smithi052.stdout: "rank_uptime": 122.499726143, 2024-05-27T07:24:57.366 INFO:teuthology.orchestra.run.smithi052.stdout: "mdsmap_epoch": 29, 2024-05-27T07:24:57.366 INFO:teuthology.orchestra.run.smithi052.stdout: "osdmap_epoch": 90, 2024-05-27T07:24:57.366 INFO:teuthology.orchestra.run.smithi052.stdout: "osdmap_epoch_barrier": 88, 2024-05-27T07:24:57.366 INFO:teuthology.orchestra.run.smithi052.stdout: "uptime": 222.238512171 2024-05-27T07:24:57.366 INFO:teuthology.orchestra.run.smithi052.stdout:} 2024-05-27T07:24:57.367 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:57.370+0000 7f2be8f65700 1 -- 172.21.15.52:0/3816766947 >> [v2:172.21.15.52:6800/3513075805,v1:172.21.15.52:6801/3513075805] conn(0x7f2bd00b2880 msgr2=0x7f2bd00b4d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:24:57.367 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:57.370+0000 7f2be8f65700 1 --2- 172.21.15.52:0/3816766947 >> [v2:172.21.15.52:6800/3513075805,v1:172.21.15.52:6801/3513075805] conn(0x7f2bd00b2880 0x7f2bd00b4d90 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7f2bcc006010 tx=0x7f2bcc000bc0 comp rx=0 tx=0).stop 2024-05-27T07:24:57.367 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:57.370+0000 7f2be8f65700 1 -- 172.21.15.52:0/3816766947 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f2be41034b0 msgr2=0x7f2be40ff2c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:24:57.367 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:57.370+0000 7f2be8f65700 1 --2- 172.21.15.52:0/3816766947 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f2be41034b0 0x7f2be40ff2c0 secure :-1 s=READY pgs=317 cs=0 l=1 rev1=1 crypto rx=0x7f2bd400d950 tx=0x7f2bd400dc60 comp rx=0 tx=0).stop 2024-05-27T07:24:57.367 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:57.371+0000 7f2be8f65700 1 -- 172.21.15.52:0/3816766947 shutdown_connections 2024-05-27T07:24:57.367 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:57.371+0000 7f2be8f65700 1 --2- 172.21.15.52:0/3816766947 >> [v2:172.21.15.52:6800/3513075805,v1:172.21.15.52:6801/3513075805] conn(0x7f2bd00b2880 0x7f2bd00b4d90 unknown :-1 s=CLOSED pgs=208 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:24:57.367 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:57.371+0000 7f2be8f65700 1 --2- 172.21.15.52:0/3816766947 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f2be4108730 0x7f2be40ffd40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:24:57.367 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:57.371+0000 7f2be8f65700 1 --2- 172.21.15.52:0/3816766947 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f2be41034b0 0x7f2be40ff2c0 unknown :-1 s=CLOSED pgs=317 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:24:57.367 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:57.371+0000 7f2be8f65700 1 --2- 172.21.15.52:0/3816766947 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f2be4107de0 0x7f2be40ff800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:24:57.367 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:57.371+0000 7f2be8f65700 1 -- 172.21.15.52:0/3816766947 >> 172.21.15.52:0/3816766947 conn(0x7f2be40fede0 msgr2=0x7f2be4075cc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:24:57.368 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:57.371+0000 7f2be8f65700 1 -- 172.21.15.52:0/3816766947 shutdown_connections 2024-05-27T07:24:57.368 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:57.371+0000 7f2be8f65700 1 -- 172.21.15.52:0/3816766947 wait complete. 2024-05-27T07:24:58.061 DEBUG:teuthology.orchestra.run.smithi052:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:38f9d900c83c63799fbdbe61acc9a11b0d3554a6 shell --fsid 73df2636-1bf8-11ef-bc9b-c7b262605968 -- ceph tell mds.i status 2024-05-27T07:24:58.315 INFO:teuthology.orchestra.run.smithi052.stderr:Inferring config /var/lib/ceph/73df2636-1bf8-11ef-bc9b-c7b262605968/mon.a/config 2024-05-27T07:24:58.979 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:24:58 smithi052 ceph-mon[31955]: pgmap v394: 129 pgs: 129 active+clean; 647 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-05-27T07:24:59.101 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:24:58 smithi173 ceph-mon[37000]: pgmap v394: 129 pgs: 129 active+clean; 647 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-05-27T07:24:59.190 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:24:58 smithi105 ceph-mon[35977]: pgmap v394: 129 pgs: 129 active+clean; 647 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-05-27T07:24:59.978 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:59.988+0000 7f2f6da25700 1 -- 172.21.15.52:0/3658199339 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f2f681054e0 msgr2=0x7f2f681058f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:24:59.979 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:59.988+0000 7f2f6da25700 1 --2- 172.21.15.52:0/3658199339 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f2f681054e0 0x7f2f681058f0 secure :-1 s=READY pgs=321 cs=0 l=1 rev1=1 crypto rx=0x7f2f58009990 tx=0x7f2f58009ca0 comp rx=0 tx=0).stop 2024-05-27T07:24:59.979 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:59.988+0000 7f2f6da25700 1 -- 172.21.15.52:0/3658199339 shutdown_connections 2024-05-27T07:24:59.979 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:59.988+0000 7f2f6da25700 1 --2- 172.21.15.52:0/3658199339 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f2f6810e910 0x7f2f68110df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:24:59.979 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:59.988+0000 7f2f6da25700 1 --2- 172.21.15.52:0/3658199339 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f2f681054e0 0x7f2f681058f0 unknown :-1 s=CLOSED pgs=321 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:24:59.979 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:59.988+0000 7f2f6da25700 1 --2- 172.21.15.52:0/3658199339 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f2f68105e30 0x7f2f6810e380 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:24:59.979 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:59.988+0000 7f2f6da25700 1 -- 172.21.15.52:0/3658199339 >> 172.21.15.52:0/3658199339 conn(0x7f2f68078df0 msgr2=0x7f2f68079230 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:24:59.979 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:59.988+0000 7f2f6da25700 1 -- 172.21.15.52:0/3658199339 shutdown_connections 2024-05-27T07:24:59.979 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:59.988+0000 7f2f6da25700 1 -- 172.21.15.52:0/3658199339 wait complete. 2024-05-27T07:24:59.979 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:59.989+0000 7f2f6da25700 1 Processor -- start 2024-05-27T07:24:59.979 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:59.989+0000 7f2f6da25700 1 -- start start 2024-05-27T07:24:59.981 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:59.989+0000 7f2f6da25700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f2f681054e0 0x7f2f6819ae70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:24:59.981 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:59.989+0000 7f2f6da25700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f2f68105e30 0x7f2f6819b410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:24:59.981 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:59.989+0000 7f2f6da25700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f2f6810e910 0x7f2f6819f650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:24:59.981 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:59.989+0000 7f2f6da25700 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x7f2f681144e0 con 0x7f2f6810e910 2024-05-27T07:24:59.981 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:59.989+0000 7f2f6da25700 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f2f68114360 con 0x7f2f681054e0 2024-05-27T07:24:59.981 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:59.989+0000 7f2f6da25700 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f2f68114660 con 0x7f2f68105e30 2024-05-27T07:24:59.981 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:59.989+0000 7f2f677fe700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f2f6810e910 0x7f2f6819f650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:24:59.981 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:59.990+0000 7f2f677fe700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f2f6810e910 0x7f2f6819f650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.52:33100/0 (socket says 172.21.15.52:33100) 2024-05-27T07:24:59.981 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:59.990+0000 7f2f677fe700 1 -- 172.21.15.52:0/4004354248 learned_addr learned my addr 172.21.15.52:0/4004354248 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-05-27T07:24:59.982 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:59.990+0000 7f2f667fc700 1 --2- 172.21.15.52:0/4004354248 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f2f68105e30 0x7f2f6819b410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:24:59.982 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:59.990+0000 7f2f66ffd700 1 --2- 172.21.15.52:0/4004354248 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f2f681054e0 0x7f2f6819ae70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:24:59.982 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:59.990+0000 7f2f677fe700 1 -- 172.21.15.52:0/4004354248 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f2f68105e30 msgr2=0x7f2f6819b410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:24:59.982 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:59.990+0000 7f2f677fe700 1 --2- 172.21.15.52:0/4004354248 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f2f68105e30 0x7f2f6819b410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:24:59.982 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:59.990+0000 7f2f677fe700 1 -- 172.21.15.52:0/4004354248 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f2f681054e0 msgr2=0x7f2f6819ae70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:24:59.982 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:59.990+0000 7f2f677fe700 1 --2- 172.21.15.52:0/4004354248 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f2f681054e0 0x7f2f6819ae70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:24:59.982 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:59.990+0000 7f2f677fe700 1 -- 172.21.15.52:0/4004354248 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2f6819fc40 con 0x7f2f6810e910 2024-05-27T07:24:59.982 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:59.990+0000 7f2f677fe700 1 --2- 172.21.15.52:0/4004354248 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f2f6810e910 0x7f2f6819f650 secure :-1 s=READY pgs=322 cs=0 l=1 rev1=1 crypto rx=0x7f2f6000b970 tx=0x7f2f6000bc80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:24:59.982 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:59.990+0000 7f2f6ca23700 1 -- 172.21.15.52:0/4004354248 <== mon.0 v2:172.21.15.52:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2f6000d860 con 0x7f2f6810e910 2024-05-27T07:24:59.982 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:59.990+0000 7f2f6ca23700 1 -- 172.21.15.52:0/4004354248 <== mon.0 v2:172.21.15.52:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f2f6000e430 con 0x7f2f6810e910 2024-05-27T07:24:59.982 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:59.990+0000 7f2f6da25700 1 -- 172.21.15.52:0/4004354248 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2f6819ff30 con 0x7f2f6810e910 2024-05-27T07:24:59.982 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:59.990+0000 7f2f6ca23700 1 -- 172.21.15.52:0/4004354248 <== mon.0 v2:172.21.15.52:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2f60017660 con 0x7f2f6810e910 2024-05-27T07:24:59.983 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:59.990+0000 7f2f6da25700 1 -- 172.21.15.52:0/4004354248 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2f681a0470 con 0x7f2f6810e910 2024-05-27T07:24:59.983 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:59.993+0000 7f2f6ca23700 1 -- 172.21.15.52:0/4004354248 <== mon.0 v2:172.21.15.52:3300/0 4 ==== mgrmap(e 18) ==== 149652+0+0 (secure 0 0 0) 0x7f2f600203f0 con 0x7f2f6810e910 2024-05-27T07:24:59.983 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:59.994+0000 7f2f6ca23700 1 --2- 172.21.15.52:0/4004354248 >> [v2:172.21.15.52:6800/3513075805,v1:172.21.15.52:6801/3513075805] conn(0x7f2f500b2900 0x7f2f500b4e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:24:59.984 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:59.994+0000 7f2f66ffd700 1 --2- 172.21.15.52:0/4004354248 >> [v2:172.21.15.52:6800/3513075805,v1:172.21.15.52:6801/3513075805] conn(0x7f2f500b2900 0x7f2f500b4e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:24:59.984 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:59.994+0000 7f2f6ca23700 1 -- 172.21.15.52:0/4004354248 <== mon.0 v2:172.21.15.52:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9121+0+0 (secure 0 0 0) 0x7f2f600e2180 con 0x7f2f6810e910 2024-05-27T07:24:59.984 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:59.994+0000 7f2f66ffd700 1 --2- 172.21.15.52:0/4004354248 >> [v2:172.21.15.52:6800/3513075805,v1:172.21.15.52:6801/3513075805] conn(0x7f2f500b2900 0x7f2f500b4e10 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7f2f5800b5c0 tx=0x7f2f58005960 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:24:59.984 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:59.994+0000 7f2f7c43ab80 1 Processor -- start 2024-05-27T07:24:59.984 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:59.994+0000 7f2f7c43ab80 1 -- start start 2024-05-27T07:24:59.986 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:59.995+0000 7f2f7c43ab80 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x559768ebedf0 0x559769043ae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:24:59.986 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:59.995+0000 7f2f7c43ab80 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x559769044020 0x55976903db90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:24:59.986 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:59.995+0000 7f2f7c43ab80 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55976903e0d0 0x55976903e4e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:24:59.986 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:59.995+0000 7f2f7c43ab80 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x559768f6bd20 con 0x55976903e0d0 2024-05-27T07:24:59.986 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:59.995+0000 7f2f7c43ab80 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x559768f5b8c0 con 0x559768ebedf0 2024-05-27T07:24:59.986 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:59.995+0000 7f2f7c43ab80 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x559768f5f900 con 0x559769044020 2024-05-27T07:24:59.986 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:59.995+0000 7f2f677fe700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55976903e0d0 0x55976903e4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:24:59.986 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:59.995+0000 7f2f677fe700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55976903e0d0 0x55976903e4e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.52:33106/0 (socket says 172.21.15.52:33106) 2024-05-27T07:24:59.986 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:59.995+0000 7f2f667fc700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x559769044020 0x55976903db90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:24:59.986 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:59.995+0000 7f2f677fe700 1 -- 172.21.15.52:0/3111368511 learned_addr learned my addr 172.21.15.52:0/3111368511 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-05-27T07:24:59.986 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:59.995+0000 7f2f66ffd700 1 --2- 172.21.15.52:0/3111368511 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x559768ebedf0 0x559769043ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:24:59.987 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:59.995+0000 7f2f677fe700 1 -- 172.21.15.52:0/3111368511 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x559769044020 msgr2=0x55976903db90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:24:59.987 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:59.995+0000 7f2f677fe700 1 --2- 172.21.15.52:0/3111368511 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x559769044020 0x55976903db90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:24:59.987 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:59.995+0000 7f2f677fe700 1 -- 172.21.15.52:0/3111368511 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x559768ebedf0 msgr2=0x559769043ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:24:59.987 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:59.995+0000 7f2f677fe700 1 --2- 172.21.15.52:0/3111368511 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x559768ebedf0 0x559769043ae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:24:59.987 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:59.995+0000 7f2f677fe700 1 -- 172.21.15.52:0/3111368511 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x559768f04f60 con 0x55976903e0d0 2024-05-27T07:24:59.987 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:59.995+0000 7f2f66ffd700 1 --2- 172.21.15.52:0/3111368511 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x559768ebedf0 0x559769043ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-27T07:24:59.987 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:59.996+0000 7f2f677fe700 1 --2- 172.21.15.52:0/3111368511 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55976903e0d0 0x55976903e4e0 secure :-1 s=READY pgs=323 cs=0 l=1 rev1=1 crypto rx=0x7f2f600a5440 tx=0x7f2f6000ecb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bfe5efc2222b6cc server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:24:59.987 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:59.996+0000 7f2f4dffb700 1 -- 172.21.15.52:0/3111368511 <== mon.0 v2:172.21.15.52:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2f600ad770 con 0x55976903e0d0 2024-05-27T07:24:59.987 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:59.996+0000 7f2f4dffb700 1 -- 172.21.15.52:0/3111368511 <== mon.0 v2:172.21.15.52:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f2f600b2220 con 0x55976903e0d0 2024-05-27T07:24:59.987 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:59.996+0000 7f2f4dffb700 1 -- 172.21.15.52:0/3111368511 <== mon.0 v2:172.21.15.52:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2f600b1d90 con 0x55976903e0d0 2024-05-27T07:24:59.989 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:59.997+0000 7f2f7c43ab80 1 -- 172.21.15.52:0/3111368511 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55976903e0d0 msgr2=0x55976903e4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:24:59.989 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:59.997+0000 7f2f7c43ab80 1 --2- 172.21.15.52:0/3111368511 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55976903e0d0 0x55976903e4e0 secure :-1 s=READY pgs=323 cs=0 l=1 rev1=1 crypto rx=0x7f2f600a5440 tx=0x7f2f6000ecb0 comp rx=0 tx=0).stop 2024-05-27T07:24:59.990 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:59.997+0000 7f2f7c43ab80 1 -- 172.21.15.52:0/3111368511 shutdown_connections 2024-05-27T07:24:59.990 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:59.997+0000 7f2f7c43ab80 1 --2- 172.21.15.52:0/3111368511 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55976903e0d0 0x55976903e4e0 unknown :-1 s=CLOSED pgs=323 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:24:59.990 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:59.997+0000 7f2f7c43ab80 1 --2- 172.21.15.52:0/3111368511 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x559768ebedf0 0x559769043ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:24:59.990 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:59.997+0000 7f2f7c43ab80 1 --2- 172.21.15.52:0/3111368511 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x559769044020 0x55976903db90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:24:59.990 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:59.997+0000 7f2f7c43ab80 1 -- 172.21.15.52:0/3111368511 >> 172.21.15.52:0/3111368511 conn(0x559768ec1720 msgr2=0x559769038d40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:24:59.990 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:59.997+0000 7f2f7c43ab80 1 -- 172.21.15.52:0/3111368511 shutdown_connections 2024-05-27T07:24:59.990 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:59.997+0000 7f2f7c43ab80 1 -- 172.21.15.52:0/3111368511 wait complete. 2024-05-27T07:24:59.990 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:59.997+0000 7f2f7c43ab80 1 -- start start 2024-05-27T07:24:59.990 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:59.997+0000 7f2f7c43ab80 1 Processor -- start 2024-05-27T07:24:59.990 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:59.998+0000 7f2f7c43ab80 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x559768ebedf0 0x5597690477c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:24:59.990 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:59.998+0000 7f2f7c43ab80 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55976903e0d0 0x559769047d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:24:59.990 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:59.998+0000 7f2f7c43ab80 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x559769044020 0x559769048240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:24:59.991 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:59.998+0000 7f2f7c43ab80 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x559768e61f50 con 0x559768ebedf0 2024-05-27T07:24:59.991 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:59.998+0000 7f2f7c43ab80 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x559768ec3d50 con 0x559769044020 2024-05-27T07:24:59.991 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:59.998+0000 7f2f7c43ab80 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x55976904ab10 con 0x55976903e0d0 2024-05-27T07:24:59.991 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:59.998+0000 7f2f7c43ab80 10 client.0 _ll_register_callbacks cb 0x559768ef2560 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T07:24:59.991 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:59.998+0000 7f2f66ffd700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x559768ebedf0 0x5597690477c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:24:59.991 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:59.998+0000 7f2f66ffd700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x559768ebedf0 0x5597690477c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.52:33108/0 (socket says 172.21.15.52:33108) 2024-05-27T07:24:59.991 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:59.998+0000 7f2f66ffd700 1 -- 172.21.15.52:0/1906011905 learned_addr learned my addr 172.21.15.52:0/1906011905 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-05-27T07:24:59.991 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:59.998+0000 7f2f667fc700 1 --2- 172.21.15.52:0/1906011905 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55976903e0d0 0x559769047d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:24:59.991 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:59.998+0000 7f2f677fe700 1 --2- 172.21.15.52:0/1906011905 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x559769044020 0x559769048240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:24:59.991 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:59.998+0000 7f2f66ffd700 1 -- 172.21.15.52:0/1906011905 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55976903e0d0 msgr2=0x559769047d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:24:59.991 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:59.998+0000 7f2f66ffd700 1 --2- 172.21.15.52:0/1906011905 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55976903e0d0 0x559769047d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:24:59.991 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:59.998+0000 7f2f66ffd700 1 -- 172.21.15.52:0/1906011905 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x559769044020 msgr2=0x559769048240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:24:59.992 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:59.998+0000 7f2f66ffd700 1 --2- 172.21.15.52:0/1906011905 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x559769044020 0x559769048240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:24:59.992 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:59.998+0000 7f2f66ffd700 1 -- 172.21.15.52:0/1906011905 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({osdmap=0}) -- 0x559768eab880 con 0x559768ebedf0 2024-05-27T07:24:59.992 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:59.999+0000 7f2f66ffd700 1 --2- 172.21.15.52:0/1906011905 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x559768ebedf0 0x5597690477c0 secure :-1 s=READY pgs=324 cs=0 l=1 rev1=1 crypto rx=0x7f2f58009990 tx=0x7f2f5801a050 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d901515867053c2a server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:24:59.992 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:59.999+0000 7f2f46ffd700 10 client.0 ms_handle_connect on v2:172.21.15.52:3300/0 2024-05-27T07:24:59.992 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:59.999+0000 7f2f46ffd700 1 -- 172.21.15.52:0/1906011905 <== mon.0 v2:172.21.15.52:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2f5800f660 con 0x559768ebedf0 2024-05-27T07:24:59.992 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:59.999+0000 7f2f46ffd700 1 -- 172.21.15.52:0/1906011905 <== mon.0 v2:172.21.15.52:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9121+0+0 (secure 0 0 0) 0x7f2f5800f800 con 0x559768ebedf0 2024-05-27T07:24:59.992 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:59.999+0000 7f2f7c43ab80 1 -- 172.21.15.52:0/1906011905 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f2f600b2220 con 0x559768ebedf0 2024-05-27T07:24:59.992 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:59.999+0000 7f2f46ffd700 1 -- 172.21.15.52:0/1906011905 <== mon.0 v2:172.21.15.52:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f2f5803d050 con 0x559768ebedf0 2024-05-27T07:24:59.992 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:59.999+0000 7f2f7c43ab80 10 client.15117 fetch_fsmap learned FSMap version 29 2024-05-27T07:24:59.992 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:59.999+0000 7f2f7c43ab80 1 -- 172.21.15.52:0/1906011905 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x559768f15710 con 0x559768ebedf0 2024-05-27T07:24:59.992 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:59.999+0000 7f2f46ffd700 1 -- 172.21.15.52:0/1906011905 <== mon.0 v2:172.21.15.52:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f2f600b2220 con 0x559768ebedf0 2024-05-27T07:24:59.992 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:59.999+0000 7f2f46ffd700 1 -- 172.21.15.52:0/1906011905 <== mon.0 v2:172.21.15.52:3300/0 5 ==== fsmap(e 29) ==== 8362+0+0 (secure 0 0 0) 0x7f2f58022070 con 0x559768ebedf0 2024-05-27T07:24:59.992 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:59.999+0000 7f2f7c43ab80 10 client.15117 fetch_fsmap finished waiting for FSMap version 29 2024-05-27T07:24:59.993 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:24:59.999+0000 7f2f7c43ab80 10 client.15117 resolve_mds: resolved name 'i' to daemon mds.i 2024-05-27T07:24:59.993 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:00.000+0000 7f2f7c43ab80 20 client.15117 populate_metadata read hostname 'smithi052' 2024-05-27T07:24:59.993 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:00.000+0000 7f2f7c43ab80 1 --2- 172.21.15.52:0/1906011905 >> [v2:172.21.15.173:6838/564778293,v1:172.21.15.173:6839/564778293] conn(0x559769057b50 0x559769058fc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:24:59.993 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:00.000+0000 7f2f7c43ab80 4 client.15117 mds_command: new command op to 24475 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-27T07:24:59.993 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:00.000+0000 7f2f7c43ab80 1 -- 172.21.15.52:0/1906011905 --> [v2:172.21.15.173:6838/564778293,v1:172.21.15.173:6839/564778293] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x559768f6dc70 con 0x559769057b50 2024-05-27T07:24:59.993 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:00.000+0000 7f2f667fc700 1 --2- 172.21.15.52:0/1906011905 >> [v2:172.21.15.173:6838/564778293,v1:172.21.15.173:6839/564778293] conn(0x559769057b50 0x559769058fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:24:59.995 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:00.000+0000 7f2f667fc700 1 --2- 172.21.15.52:0/1906011905 >> [v2:172.21.15.173:6838/564778293,v1:172.21.15.173:6839/564778293] conn(0x559769057b50 0x559769058fc0 crc :-1 s=READY pgs=16 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=1f61c3171917cd23 server_cookie=6265cf160a67cb89 in_seq=0 out_seq=0 2024-05-27T07:24:59.995 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:00.000+0000 7f2f46ffd700 10 client.15117 ms_handle_connect on v2:172.21.15.173:6838/564778293 2024-05-27T07:24:59.995 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:00.001+0000 7f2f46ffd700 1 -- 172.21.15.52:0/1906011905 <== mds.2 v2:172.21.15.173:6838/564778293 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24041 (crc 0 0 0) 0x559768f6dc70 con 0x559769057b50 2024-05-27T07:24:59.995 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:00.001+0000 7f2f46ffd700 10 client.15117 handle_command_reply: tid=0 2024-05-27T07:24:59.995 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:00.001+0000 7f2f7c43ab80 1 client.15117 shutdown 2024-05-27T07:24:59.995 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:00.002+0000 7f2f7c43ab80 1 -- 172.21.15.52:0/1906011905 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x559768ebedf0 msgr2=0x5597690477c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:24:59.995 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:00.002+0000 7f2f7c43ab80 1 --2- 172.21.15.52:0/1906011905 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x559768ebedf0 0x5597690477c0 secure :-1 s=READY pgs=324 cs=0 l=1 rev1=1 crypto rx=0x7f2f58009990 tx=0x7f2f5801a050 comp rx=0 tx=0).stop 2024-05-27T07:24:59.995 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:00.002+0000 7f2f7c43ab80 1 -- 172.21.15.52:0/1906011905 shutdown_connections 2024-05-27T07:24:59.995 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:00.002+0000 7f2f7c43ab80 1 --2- 172.21.15.52:0/1906011905 >> [v2:172.21.15.173:6838/564778293,v1:172.21.15.173:6839/564778293] conn(0x559769057b50 0x559769058fc0 crc :-1 s=READY pgs=16 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:24:59.996 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:00.002+0000 7f2f7c43ab80 1 --2- 172.21.15.52:0/1906011905 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x559769044020 0x559769048240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:24:59.996 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:00.002+0000 7f2f7c43ab80 1 --2- 172.21.15.52:0/1906011905 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x559768ebedf0 0x5597690477c0 unknown :-1 s=CLOSED pgs=324 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:24:59.996 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:00.002+0000 7f2f7c43ab80 1 --2- 172.21.15.52:0/1906011905 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55976903e0d0 0x559769047d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:24:59.996 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:00.002+0000 7f2f7c43ab80 1 -- 172.21.15.52:0/1906011905 >> 172.21.15.52:0/1906011905 conn(0x559768ec1720 msgr2=0x55976903cb80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:24:59.996 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:00.002+0000 7f2f46ffd700 0 client.15117 ms_handle_reset on v2:172.21.15.173:6838/564778293 2024-05-27T07:24:59.996 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:00.002+0000 7f2f7c43ab80 1 -- 172.21.15.52:0/1906011905 shutdown_connections 2024-05-27T07:24:59.996 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:00.002+0000 7f2f7c43ab80 1 -- 172.21.15.52:0/1906011905 wait complete. 2024-05-27T07:24:59.996 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:00.002+0000 7f2f7c43ab80 20 client.15117 trim_cache size 0 max 16384 2024-05-27T07:25:00.017 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:00.026+0000 7f2f7c43ab80 1 Processor -- start 2024-05-27T07:25:00.017 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:00.026+0000 7f2f7c43ab80 1 -- start start 2024-05-27T07:25:00.017 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:00.026+0000 7f2f7c43ab80 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x559768ebedf0 0x55976903db90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:25:00.017 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:00.026+0000 7f2f7c43ab80 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55976903e0d0 0x559769057480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:25:00.017 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:00.026+0000 7f2f7c43ab80 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x559769044020 0x559769053de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:25:00.017 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:00.026+0000 7f2f7c43ab80 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x559769057300 con 0x559769044020 2024-05-27T07:25:00.017 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:00.026+0000 7f2f7c43ab80 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x5597690579d0 con 0x559768ebedf0 2024-05-27T07:25:00.018 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:00.026+0000 7f2f7c43ab80 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x5597690580a0 con 0x55976903e0d0 2024-05-27T07:25:00.018 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:00.026+0000 7f2f677fe700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x559769044020 0x559769053de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:25:00.018 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:00.026+0000 7f2f677fe700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x559769044020 0x559769053de0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.52:33120/0 (socket says 172.21.15.52:33120) 2024-05-27T07:25:00.018 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:00.026+0000 7f2f677fe700 1 -- 172.21.15.52:0/539048960 learned_addr learned my addr 172.21.15.52:0/539048960 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-05-27T07:25:00.018 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:00.026+0000 7f2f66ffd700 1 --2- 172.21.15.52:0/539048960 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x559768ebedf0 0x55976903db90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:25:00.018 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:00.026+0000 7f2f667fc700 1 --2- 172.21.15.52:0/539048960 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55976903e0d0 0x559769057480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:25:00.018 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:00.026+0000 7f2f677fe700 1 -- 172.21.15.52:0/539048960 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55976903e0d0 msgr2=0x559769057480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:25:00.018 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:00.026+0000 7f2f677fe700 1 --2- 172.21.15.52:0/539048960 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55976903e0d0 0x559769057480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:25:00.018 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:00.026+0000 7f2f677fe700 1 -- 172.21.15.52:0/539048960 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x559768ebedf0 msgr2=0x55976903db90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:25:00.018 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:00.026+0000 7f2f677fe700 1 --2- 172.21.15.52:0/539048960 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x559768ebedf0 0x55976903db90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:25:00.018 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:00.026+0000 7f2f677fe700 1 -- 172.21.15.52:0/539048960 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x559769034a50 con 0x559769044020 2024-05-27T07:25:00.018 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:00.027+0000 7f2f667fc700 1 --2- 172.21.15.52:0/539048960 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55976903e0d0 0x559769057480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-27T07:25:00.019 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:00.027+0000 7f2f677fe700 1 --2- 172.21.15.52:0/539048960 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x559769044020 0x559769053de0 secure :-1 s=READY pgs=325 cs=0 l=1 rev1=1 crypto rx=0x7f2f600a90c0 tx=0x7f2f600a9000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=aa616ea39bca61b8 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:25:00.019 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:00.027+0000 7f2f4d7fa700 1 -- 172.21.15.52:0/539048960 <== mon.0 v2:172.21.15.52:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2f600ad770 con 0x559769044020 2024-05-27T07:25:00.019 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:00.027+0000 7f2f4d7fa700 1 -- 172.21.15.52:0/539048960 <== mon.0 v2:172.21.15.52:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f2f6000e8e0 con 0x559769044020 2024-05-27T07:25:00.019 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:00.027+0000 7f2f4d7fa700 1 -- 172.21.15.52:0/539048960 <== mon.0 v2:172.21.15.52:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2f600b1ff0 con 0x559769044020 2024-05-27T07:25:00.023 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:00.028+0000 7f2f7c43ab80 1 -- 172.21.15.52:0/539048960 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x559769044020 msgr2=0x559769053de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:25:00.023 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:00.028+0000 7f2f7c43ab80 1 --2- 172.21.15.52:0/539048960 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x559769044020 0x559769053de0 secure :-1 s=READY pgs=325 cs=0 l=1 rev1=1 crypto rx=0x7f2f600a90c0 tx=0x7f2f600a9000 comp rx=0 tx=0).stop 2024-05-27T07:25:00.023 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:00.028+0000 7f2f7c43ab80 1 -- 172.21.15.52:0/539048960 shutdown_connections 2024-05-27T07:25:00.023 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:00.028+0000 7f2f7c43ab80 1 --2- 172.21.15.52:0/539048960 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x559769044020 0x559769053de0 unknown :-1 s=CLOSED pgs=325 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:25:00.023 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:00.028+0000 7f2f7c43ab80 1 --2- 172.21.15.52:0/539048960 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x559768ebedf0 0x55976903db90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:25:00.023 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:00.028+0000 7f2f7c43ab80 1 --2- 172.21.15.52:0/539048960 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55976903e0d0 0x559769057480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:25:00.023 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:00.028+0000 7f2f7c43ab80 1 -- 172.21.15.52:0/539048960 >> 172.21.15.52:0/539048960 conn(0x559768ec1720 msgr2=0x559769042720 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:25:00.023 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:00.028+0000 7f2f7c43ab80 1 -- 172.21.15.52:0/539048960 shutdown_connections 2024-05-27T07:25:00.023 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:00.028+0000 7f2f7c43ab80 1 -- 172.21.15.52:0/539048960 wait complete. 2024-05-27T07:25:00.023 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:00.028+0000 7f2f7c43ab80 1 -- start start 2024-05-27T07:25:00.024 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:00.028+0000 7f2f7c43ab80 1 Processor -- start 2024-05-27T07:25:00.024 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:00.029+0000 7f2f7c43ab80 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x559768ebedf0 0x55976903cb80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:25:00.024 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:00.029+0000 7f2f7c43ab80 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55976903e0d0 0x55976903d0c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:25:00.024 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:00.029+0000 7f2f7c43ab80 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x559769044020 0x55976903d600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:25:00.024 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:00.029+0000 7f2f7c43ab80 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x559769058e40 con 0x559768ebedf0 2024-05-27T07:25:00.024 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:00.029+0000 7f2f7c43ab80 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x559769058770 con 0x55976903e0d0 2024-05-27T07:25:00.024 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:00.029+0000 7f2f7c43ab80 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x559769059510 con 0x559769044020 2024-05-27T07:25:00.024 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:00.029+0000 7f2f66ffd700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x559768ebedf0 0x55976903cb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:25:00.024 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:00.029+0000 7f2f7c43ab80 10 client.0 _ll_register_callbacks cb 0x559768ef2560 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T07:25:00.024 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:00.029+0000 7f2f66ffd700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x559768ebedf0 0x55976903cb80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.52:33124/0 (socket says 172.21.15.52:33124) 2024-05-27T07:25:00.024 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:00.029+0000 7f2f66ffd700 1 -- 172.21.15.52:0/3363100373 learned_addr learned my addr 172.21.15.52:0/3363100373 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-05-27T07:25:00.024 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:00.029+0000 7f2f667fc700 1 --2- 172.21.15.52:0/3363100373 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55976903e0d0 0x55976903d0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:25:00.025 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:00.029+0000 7f2f677fe700 1 --2- 172.21.15.52:0/3363100373 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x559769044020 0x55976903d600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:25:00.025 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:00.029+0000 7f2f66ffd700 1 -- 172.21.15.52:0/3363100373 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x559769044020 msgr2=0x55976903d600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:25:00.025 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:00.029+0000 7f2f66ffd700 1 --2- 172.21.15.52:0/3363100373 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x559769044020 0x55976903d600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:25:00.025 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:00.029+0000 7f2f66ffd700 1 -- 172.21.15.52:0/3363100373 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55976903e0d0 msgr2=0x55976903d0c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:25:00.025 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:00.029+0000 7f2f66ffd700 1 --2- 172.21.15.52:0/3363100373 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55976903e0d0 0x55976903d0c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:25:00.025 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:00.029+0000 7f2f66ffd700 1 -- 172.21.15.52:0/3363100373 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({osdmap=0}) -- 0x559769061a00 con 0x559768ebedf0 2024-05-27T07:25:00.025 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:00.029+0000 7f2f667fc700 1 --2- 172.21.15.52:0/3363100373 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55976903e0d0 0x55976903d0c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-27T07:25:00.025 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:00.029+0000 7f2f66ffd700 1 --2- 172.21.15.52:0/3363100373 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x559768ebedf0 0x55976903cb80 secure :-1 s=READY pgs=326 cs=0 l=1 rev1=1 crypto rx=0x7f2f58023ba0 tx=0x7f2f5800bfc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=81722140d203ee77 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:25:00.025 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:00.029+0000 7f2f46ffd700 10 client.0 ms_handle_connect on v2:172.21.15.52:3300/0 2024-05-27T07:25:00.025 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:00.030+0000 7f2f46ffd700 1 -- 172.21.15.52:0/3363100373 <== mon.0 v2:172.21.15.52:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2f58047020 con 0x559768ebedf0 2024-05-27T07:25:00.025 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:00.030+0000 7f2f7c43ab80 1 -- 172.21.15.52:0/3363100373 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f2f6000e8e0 con 0x559768ebedf0 2024-05-27T07:25:00.025 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:00.030+0000 7f2f46ffd700 1 -- 172.21.15.52:0/3363100373 <== mon.0 v2:172.21.15.52:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9121+0+0 (secure 0 0 0) 0x7f2f5800f660 con 0x559768ebedf0 2024-05-27T07:25:00.025 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:00.030+0000 7f2f46ffd700 1 -- 172.21.15.52:0/3363100373 <== mon.0 v2:172.21.15.52:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f2f58052050 con 0x559768ebedf0 2024-05-27T07:25:00.026 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:00.030+0000 7f2f7c43ab80 10 client.15123 fetch_fsmap learned FSMap version 29 2024-05-27T07:25:00.026 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:00.030+0000 7f2f7c43ab80 1 -- 172.21.15.52:0/3363100373 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x559769062c60 con 0x559768ebedf0 2024-05-27T07:25:00.026 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:00.030+0000 7f2f46ffd700 1 -- 172.21.15.52:0/3363100373 <== mon.0 v2:172.21.15.52:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f2f6000e8e0 con 0x559768ebedf0 2024-05-27T07:25:00.026 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:00.030+0000 7f2f46ffd700 1 -- 172.21.15.52:0/3363100373 <== mon.0 v2:172.21.15.52:3300/0 5 ==== fsmap(e 29) ==== 8362+0+0 (secure 0 0 0) 0x7f2f58022040 con 0x559768ebedf0 2024-05-27T07:25:00.026 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:00.030+0000 7f2f7c43ab80 10 client.15123 fetch_fsmap finished waiting for FSMap version 29 2024-05-27T07:25:00.026 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:00.030+0000 7f2f7c43ab80 10 client.15123 resolve_mds: resolved name 'i' to daemon mds.i 2024-05-27T07:25:00.026 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:00.030+0000 7f2f7c43ab80 20 client.15123 populate_metadata read hostname 'smithi052' 2024-05-27T07:25:00.026 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:00.030+0000 7f2f7c43ab80 1 --2- 172.21.15.52:0/3363100373 >> [v2:172.21.15.173:6838/564778293,v1:172.21.15.173:6839/564778293] conn(0x5597690c50f0 0x5597690c6560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:25:00.026 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:00.030+0000 7f2f7c43ab80 4 client.15123 mds_command: new command op to 24475 tid=0 multi_id=0 [{"prefix": "status"}] 2024-05-27T07:25:00.026 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:00.030+0000 7f2f7c43ab80 1 -- 172.21.15.52:0/3363100373 --> [v2:172.21.15.173:6838/564778293,v1:172.21.15.173:6839/564778293] -- command(tid 0: {"prefix": "status"}) -- 0x559768f67450 con 0x5597690c50f0 2024-05-27T07:25:00.026 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:00.031+0000 7f2f667fc700 1 --2- 172.21.15.52:0/3363100373 >> [v2:172.21.15.173:6838/564778293,v1:172.21.15.173:6839/564778293] conn(0x5597690c50f0 0x5597690c6560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:25:00.026 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:00.031+0000 7f2f46ffd700 10 client.15123 ms_handle_connect on v2:172.21.15.173:6838/564778293 2024-05-27T07:25:00.027 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:00.031+0000 7f2f667fc700 1 --2- 172.21.15.52:0/3363100373 >> [v2:172.21.15.173:6838/564778293,v1:172.21.15.173:6839/564778293] conn(0x5597690c50f0 0x5597690c6560 crc :-1 s=READY pgs=17 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=16d0d9cef7d43be4 server_cookie=944e0f805081a050 in_seq=0 out_seq=0 2024-05-27T07:25:00.027 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:00.031+0000 7f2f46ffd700 1 -- 172.21.15.52:0/3363100373 <== mds.2 v2:172.21.15.173:6838/564778293 1 ==== command_reply(tid 0: 0 ) ==== 8+0+327 (crc 0 0 0) 0x559768f67450 con 0x5597690c50f0 2024-05-27T07:25:00.027 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:00.031+0000 7f2f46ffd700 10 client.15123 handle_command_reply: tid=0 2024-05-27T07:25:00.027 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:00.031+0000 7f2f7c43ab80 1 client.15123 shutdown 2024-05-27T07:25:00.027 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:00.032+0000 7f2f7c43ab80 1 -- 172.21.15.52:0/3363100373 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x559768ebedf0 msgr2=0x55976903cb80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:25:00.027 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:00.032+0000 7f2f7c43ab80 1 --2- 172.21.15.52:0/3363100373 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x559768ebedf0 0x55976903cb80 secure :-1 s=READY pgs=326 cs=0 l=1 rev1=1 crypto rx=0x7f2f58023ba0 tx=0x7f2f5800bfc0 comp rx=0 tx=0).stop 2024-05-27T07:25:00.027 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:00.032+0000 7f2f7c43ab80 1 -- 172.21.15.52:0/3363100373 shutdown_connections 2024-05-27T07:25:00.027 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:00.032+0000 7f2f7c43ab80 1 --2- 172.21.15.52:0/3363100373 >> [v2:172.21.15.173:6838/564778293,v1:172.21.15.173:6839/564778293] conn(0x5597690c50f0 0x5597690c6560 crc :-1 s=READY pgs=17 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:25:00.027 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:00.032+0000 7f2f7c43ab80 1 --2- 172.21.15.52:0/3363100373 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x559769044020 0x55976903d600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:25:00.027 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:00.032+0000 7f2f7c43ab80 1 --2- 172.21.15.52:0/3363100373 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x559768ebedf0 0x55976903cb80 unknown :-1 s=CLOSED pgs=326 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:25:00.027 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:00.032+0000 7f2f7c43ab80 1 --2- 172.21.15.52:0/3363100373 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55976903e0d0 0x55976903d0c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:25:00.027 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:00.032+0000 7f2f7c43ab80 1 -- 172.21.15.52:0/3363100373 >> 172.21.15.52:0/3363100373 conn(0x559768ec1720 msgr2=0x5597690460d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:25:00.027 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:00.032+0000 7f2f46ffd700 0 client.15123 ms_handle_reset on v2:172.21.15.173:6838/564778293 2024-05-27T07:25:00.027 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:00.032+0000 7f2f7c43ab80 1 -- 172.21.15.52:0/3363100373 shutdown_connections 2024-05-27T07:25:00.028 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:00.032+0000 7f2f7c43ab80 1 -- 172.21.15.52:0/3363100373 wait complete. 2024-05-27T07:25:00.028 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:00.032+0000 7f2f7c43ab80 20 client.15123 trim_cache size 0 max 16384 2024-05-27T07:25:00.028 INFO:teuthology.orchestra.run.smithi052.stdout:{ 2024-05-27T07:25:00.028 INFO:teuthology.orchestra.run.smithi052.stdout: "cluster_fsid": "73df2636-1bf8-11ef-bc9b-c7b262605968", 2024-05-27T07:25:00.028 INFO:teuthology.orchestra.run.smithi052.stdout: "whoami": 2, 2024-05-27T07:25:00.028 INFO:teuthology.orchestra.run.smithi052.stdout: "id": 24475, 2024-05-27T07:25:00.028 INFO:teuthology.orchestra.run.smithi052.stdout: "want_state": "up:active", 2024-05-27T07:25:00.028 INFO:teuthology.orchestra.run.smithi052.stdout: "state": "up:active", 2024-05-27T07:25:00.028 INFO:teuthology.orchestra.run.smithi052.stdout: "fs_name": "cephfs", 2024-05-27T07:25:00.028 INFO:teuthology.orchestra.run.smithi052.stdout: "rank_uptime": 126.168512577, 2024-05-27T07:25:00.029 INFO:teuthology.orchestra.run.smithi052.stdout: "mdsmap_epoch": 29, 2024-05-27T07:25:00.029 INFO:teuthology.orchestra.run.smithi052.stdout: "osdmap_epoch": 90, 2024-05-27T07:25:00.029 INFO:teuthology.orchestra.run.smithi052.stdout: "osdmap_epoch_barrier": 88, 2024-05-27T07:25:00.029 INFO:teuthology.orchestra.run.smithi052.stdout: "uptime": 213.79436069600001 2024-05-27T07:25:00.029 INFO:teuthology.orchestra.run.smithi052.stdout:} 2024-05-27T07:25:00.030 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:00.032+0000 7f2f6da25700 1 -- 172.21.15.52:0/4004354248 >> [v2:172.21.15.52:6800/3513075805,v1:172.21.15.52:6801/3513075805] conn(0x7f2f500b2900 msgr2=0x7f2f500b4e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:25:00.030 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:00.032+0000 7f2f6da25700 1 --2- 172.21.15.52:0/4004354248 >> [v2:172.21.15.52:6800/3513075805,v1:172.21.15.52:6801/3513075805] conn(0x7f2f500b2900 0x7f2f500b4e10 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7f2f5800b5c0 tx=0x7f2f58005960 comp rx=0 tx=0).stop 2024-05-27T07:25:00.030 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:00.032+0000 7f2f6da25700 1 -- 172.21.15.52:0/4004354248 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f2f6810e910 msgr2=0x7f2f6819f650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:25:00.030 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:00.033+0000 7f2f6da25700 1 --2- 172.21.15.52:0/4004354248 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f2f6810e910 0x7f2f6819f650 secure :-1 s=READY pgs=322 cs=0 l=1 rev1=1 crypto rx=0x7f2f6000b970 tx=0x7f2f6000bc80 comp rx=0 tx=0).stop 2024-05-27T07:25:00.030 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:00.033+0000 7f2f6da25700 1 -- 172.21.15.52:0/4004354248 shutdown_connections 2024-05-27T07:25:00.030 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:00.033+0000 7f2f6da25700 1 --2- 172.21.15.52:0/4004354248 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f2f6810e910 0x7f2f6819f650 unknown :-1 s=CLOSED pgs=322 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:25:00.030 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:00.033+0000 7f2f6da25700 1 --2- 172.21.15.52:0/4004354248 >> [v2:172.21.15.52:6800/3513075805,v1:172.21.15.52:6801/3513075805] conn(0x7f2f500b2900 0x7f2f500b4e10 unknown :-1 s=CLOSED pgs=209 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:25:00.030 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:00.033+0000 7f2f6da25700 1 --2- 172.21.15.52:0/4004354248 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f2f681054e0 0x7f2f6819ae70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:25:00.030 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:00.033+0000 7f2f6da25700 1 --2- 172.21.15.52:0/4004354248 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f2f68105e30 0x7f2f6819b410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:25:00.030 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:00.033+0000 7f2f6da25700 1 -- 172.21.15.52:0/4004354248 >> 172.21.15.52:0/4004354248 conn(0x7f2f68078df0 msgr2=0x7f2f681000b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:25:00.031 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:00.033+0000 7f2f6da25700 1 -- 172.21.15.52:0/4004354248 shutdown_connections 2024-05-27T07:25:00.031 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:00.033+0000 7f2f6da25700 1 -- 172.21.15.52:0/4004354248 wait complete. 2024-05-27T07:25:00.672 DEBUG:teuthology.orchestra.run.smithi052:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:38f9d900c83c63799fbdbe61acc9a11b0d3554a6 shell --fsid 73df2636-1bf8-11ef-bc9b-c7b262605968 -- ceph tell mds.h status 2024-05-27T07:25:00.928 INFO:teuthology.orchestra.run.smithi052.stderr:Inferring config /var/lib/ceph/73df2636-1bf8-11ef-bc9b-c7b262605968/mon.a/config 2024-05-27T07:25:00.979 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:25:00 smithi052 ceph-mon[31955]: pgmap v395: 129 pgs: 129 active+clean; 647 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 9 op/s 2024-05-27T07:25:01.102 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:25:00 smithi173 ceph-mon[37000]: pgmap v395: 129 pgs: 129 active+clean; 647 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 9 op/s 2024-05-27T07:25:01.189 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:25:00 smithi105 ceph-mon[35977]: pgmap v395: 129 pgs: 129 active+clean; 647 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 9 op/s 2024-05-27T07:25:02.633 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:02.641+0000 7f248ffcf700 1 -- 172.21.15.52:0/168352953 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f24881034d0 msgr2=0x7f24881038e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:25:02.633 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:02.641+0000 7f248ffcf700 1 --2- 172.21.15.52:0/168352953 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f24881034d0 0x7f24881038e0 secure :-1 s=READY pgs=327 cs=0 l=1 rev1=1 crypto rx=0x7f24840099c0 tx=0x7f2484009cd0 comp rx=0 tx=0).stop 2024-05-27T07:25:02.633 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:02.641+0000 7f248ffcf700 1 -- 172.21.15.52:0/168352953 shutdown_connections 2024-05-27T07:25:02.633 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:02.641+0000 7f248ffcf700 1 --2- 172.21.15.52:0/168352953 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f2488108750 0x7f24881100b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:25:02.633 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:02.641+0000 7f248ffcf700 1 --2- 172.21.15.52:0/168352953 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f24881034d0 0x7f24881038e0 unknown :-1 s=CLOSED pgs=327 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:25:02.633 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:02.641+0000 7f248ffcf700 1 --2- 172.21.15.52:0/168352953 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f2488107e00 0x7f2488108210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:25:02.634 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:02.641+0000 7f248ffcf700 1 -- 172.21.15.52:0/168352953 >> 172.21.15.52:0/168352953 conn(0x7f24880fef20 msgr2=0x7f2488101340 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:25:02.634 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:02.641+0000 7f248ffcf700 1 -- 172.21.15.52:0/168352953 shutdown_connections 2024-05-27T07:25:02.634 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:02.642+0000 7f248ffcf700 1 -- 172.21.15.52:0/168352953 wait complete. 2024-05-27T07:25:02.634 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:02.642+0000 7f248ffcf700 1 Processor -- start 2024-05-27T07:25:02.634 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:02.643+0000 7f248ffcf700 1 -- start start 2024-05-27T07:25:02.634 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:02.643+0000 7f248ffcf700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f24881034d0 0x7f24880ff400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:25:02.634 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:02.643+0000 7f248ffcf700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f2488107e00 0x7f24880ff940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:25:02.634 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:02.643+0000 7f248ffcf700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f2488108750 0x7f24880ffe80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:25:02.634 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:02.643+0000 7f248ffcf700 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x7f2488113380 con 0x7f2488107e00 2024-05-27T07:25:02.635 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:02.643+0000 7f248ffcf700 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f2488113200 con 0x7f2488108750 2024-05-27T07:25:02.635 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:02.643+0000 7f248ffcf700 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f2488113500 con 0x7f24881034d0 2024-05-27T07:25:02.635 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:02.643+0000 7f248d56a700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f2488107e00 0x7f24880ff940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:25:02.635 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:02.643+0000 7f248d56a700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f2488107e00 0x7f24880ff940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.52:45128/0 (socket says 172.21.15.52:45128) 2024-05-27T07:25:02.635 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:02.643+0000 7f248d56a700 1 -- 172.21.15.52:0/1681135330 learned_addr learned my addr 172.21.15.52:0/1681135330 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-05-27T07:25:02.635 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:02.643+0000 7f248dd6b700 1 --2- 172.21.15.52:0/1681135330 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f24881034d0 0x7f24880ff400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:25:02.635 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:02.643+0000 7f248e56c700 1 --2- 172.21.15.52:0/1681135330 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f2488108750 0x7f24880ffe80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:25:02.635 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:02.643+0000 7f248d56a700 1 -- 172.21.15.52:0/1681135330 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f24881034d0 msgr2=0x7f24880ff400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:25:02.635 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:02.643+0000 7f248d56a700 1 --2- 172.21.15.52:0/1681135330 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f24881034d0 0x7f24880ff400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:25:02.635 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:02.643+0000 7f248d56a700 1 -- 172.21.15.52:0/1681135330 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f2488108750 msgr2=0x7f24880ffe80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:25:02.635 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:02.643+0000 7f248d56a700 1 --2- 172.21.15.52:0/1681135330 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f2488108750 0x7f24880ffe80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:25:02.635 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:02.643+0000 7f248d56a700 1 -- 172.21.15.52:0/1681135330 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f24881a9240 con 0x7f2488107e00 2024-05-27T07:25:02.636 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:02.644+0000 7f248d56a700 1 --2- 172.21.15.52:0/1681135330 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f2488107e00 0x7f24880ff940 secure :-1 s=READY pgs=328 cs=0 l=1 rev1=1 crypto rx=0x7f247c00d980 tx=0x7f247c00dc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:25:02.636 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:02.644+0000 7f247affd700 1 -- 172.21.15.52:0/1681135330 <== mon.0 v2:172.21.15.52:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f247c017070 con 0x7f2488107e00 2024-05-27T07:25:02.636 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:02.644+0000 7f248ffcf700 1 -- 172.21.15.52:0/1681135330 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f24881a9530 con 0x7f2488107e00 2024-05-27T07:25:02.637 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:02.644+0000 7f247affd700 1 -- 172.21.15.52:0/1681135330 <== mon.0 v2:172.21.15.52:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f247c009de0 con 0x7f2488107e00 2024-05-27T07:25:02.637 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:02.644+0000 7f247affd700 1 -- 172.21.15.52:0/1681135330 <== mon.0 v2:172.21.15.52:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f247c00f3c0 con 0x7f2488107e00 2024-05-27T07:25:02.637 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:02.644+0000 7f248ffcf700 1 -- 172.21.15.52:0/1681135330 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f24881a9a70 con 0x7f2488107e00 2024-05-27T07:25:02.637 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:02.647+0000 7f247affd700 1 -- 172.21.15.52:0/1681135330 <== mon.0 v2:172.21.15.52:3300/0 4 ==== mgrmap(e 18) ==== 149652+0+0 (secure 0 0 0) 0x7f247c009930 con 0x7f2488107e00 2024-05-27T07:25:02.639 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:02.647+0000 7f247affd700 1 --2- 172.21.15.52:0/1681135330 >> [v2:172.21.15.52:6800/3513075805,v1:172.21.15.52:6801/3513075805] conn(0x7f24740b2880 0x7f24740b4d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:25:02.639 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:02.648+0000 7f247affd700 1 -- 172.21.15.52:0/1681135330 <== mon.0 v2:172.21.15.52:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9121+0+0 (secure 0 0 0) 0x7f247c0e2430 con 0x7f2488107e00 2024-05-27T07:25:02.639 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:02.648+0000 7f248dd6b700 1 --2- 172.21.15.52:0/1681135330 >> [v2:172.21.15.52:6800/3513075805,v1:172.21.15.52:6801/3513075805] conn(0x7f24740b2880 0x7f24740b4d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:25:02.639 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:02.648+0000 7f249e9e4b80 1 Processor -- start 2024-05-27T07:25:02.639 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:02.648+0000 7f249e9e4b80 1 -- start start 2024-05-27T07:25:02.640 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:02.648+0000 7f248dd6b700 1 --2- 172.21.15.52:0/1681135330 >> [v2:172.21.15.52:6800/3513075805,v1:172.21.15.52:6801/3513075805] conn(0x7f24740b2880 0x7f24740b4d90 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7f2484009990 tx=0x7f24840036e0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:25:02.640 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:02.648+0000 7f249e9e4b80 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x563a55042df0 0x563a551c7ae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:25:02.640 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:02.648+0000 7f249e9e4b80 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x563a551c8020 0x563a551c1b90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:25:02.640 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:02.648+0000 7f249e9e4b80 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x563a551c20d0 0x563a551c24e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:25:02.640 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:02.648+0000 7f249e9e4b80 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x563a550efd20 con 0x563a551c8020 2024-05-27T07:25:02.640 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:02.648+0000 7f249e9e4b80 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x563a550df8c0 con 0x563a55042df0 2024-05-27T07:25:02.640 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:02.648+0000 7f249e9e4b80 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x563a550e3900 con 0x563a551c20d0 2024-05-27T07:25:02.640 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:02.648+0000 7f248d56a700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x563a551c8020 0x563a551c1b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:25:02.640 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:02.648+0000 7f248d56a700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x563a551c8020 0x563a551c1b90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.52:45144/0 (socket says 172.21.15.52:45144) 2024-05-27T07:25:02.640 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:02.648+0000 7f248d56a700 1 -- 172.21.15.52:0/2624366688 learned_addr learned my addr 172.21.15.52:0/2624366688 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-05-27T07:25:02.640 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:02.649+0000 7f248e56c700 1 --2- 172.21.15.52:0/2624366688 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x563a551c20d0 0x563a551c24e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:25:02.641 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:02.649+0000 7f248dd6b700 1 --2- 172.21.15.52:0/2624366688 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x563a55042df0 0x563a551c7ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:25:02.641 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:02.649+0000 7f248d56a700 1 -- 172.21.15.52:0/2624366688 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x563a551c20d0 msgr2=0x563a551c24e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:25:02.641 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:02.649+0000 7f248d56a700 1 --2- 172.21.15.52:0/2624366688 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x563a551c20d0 0x563a551c24e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:25:02.641 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:02.649+0000 7f248d56a700 1 -- 172.21.15.52:0/2624366688 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x563a55042df0 msgr2=0x563a551c7ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:25:02.641 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:02.649+0000 7f248d56a700 1 --2- 172.21.15.52:0/2624366688 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x563a55042df0 0x563a551c7ae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:25:02.641 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:02.649+0000 7f248d56a700 1 -- 172.21.15.52:0/2624366688 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x563a55088f60 con 0x563a551c8020 2024-05-27T07:25:02.641 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:02.649+0000 7f248d56a700 1 --2- 172.21.15.52:0/2624366688 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x563a551c8020 0x563a551c1b90 secure :-1 s=READY pgs=329 cs=0 l=1 rev1=1 crypto rx=0x7f247c0a93d0 tx=0x7f247c0a9150 comp rx=0 tx=0).ready entity=mon.0 client_cookie=30a76e3f3dd11210 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:25:02.641 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:02.649+0000 7f2473fff700 1 -- 172.21.15.52:0/2624366688 <== mon.0 v2:172.21.15.52:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f247c020070 con 0x563a551c8020 2024-05-27T07:25:02.641 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:02.649+0000 7f2473fff700 1 -- 172.21.15.52:0/2624366688 <== mon.0 v2:172.21.15.52:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f247c0a96d0 con 0x563a551c8020 2024-05-27T07:25:02.641 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:02.649+0000 7f2473fff700 1 -- 172.21.15.52:0/2624366688 <== mon.0 v2:172.21.15.52:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f247c0106a0 con 0x563a551c8020 2024-05-27T07:25:02.645 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:02.650+0000 7f249e9e4b80 1 -- 172.21.15.52:0/2624366688 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x563a551c8020 msgr2=0x563a551c1b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:25:02.645 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:02.650+0000 7f249e9e4b80 1 --2- 172.21.15.52:0/2624366688 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x563a551c8020 0x563a551c1b90 secure :-1 s=READY pgs=329 cs=0 l=1 rev1=1 crypto rx=0x7f247c0a93d0 tx=0x7f247c0a9150 comp rx=0 tx=0).stop 2024-05-27T07:25:02.645 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:02.650+0000 7f249e9e4b80 1 -- 172.21.15.52:0/2624366688 shutdown_connections 2024-05-27T07:25:02.645 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:02.650+0000 7f249e9e4b80 1 --2- 172.21.15.52:0/2624366688 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x563a551c20d0 0x563a551c24e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:25:02.645 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:02.650+0000 7f249e9e4b80 1 --2- 172.21.15.52:0/2624366688 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x563a55042df0 0x563a551c7ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:25:02.645 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:02.650+0000 7f249e9e4b80 1 --2- 172.21.15.52:0/2624366688 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x563a551c8020 0x563a551c1b90 unknown :-1 s=CLOSED pgs=329 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:25:02.645 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:02.650+0000 7f249e9e4b80 1 -- 172.21.15.52:0/2624366688 >> 172.21.15.52:0/2624366688 conn(0x563a55045720 msgr2=0x563a551bcd40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:25:02.645 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:02.650+0000 7f249e9e4b80 1 -- 172.21.15.52:0/2624366688 shutdown_connections 2024-05-27T07:25:02.645 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:02.650+0000 7f249e9e4b80 1 -- 172.21.15.52:0/2624366688 wait complete. 2024-05-27T07:25:02.646 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:02.651+0000 7f249e9e4b80 1 -- start start 2024-05-27T07:25:02.646 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:02.651+0000 7f249e9e4b80 1 Processor -- start 2024-05-27T07:25:02.646 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:02.651+0000 7f249e9e4b80 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x563a55042df0 0x563a551cb900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:25:02.646 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:02.651+0000 7f249e9e4b80 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x563a551c20d0 0x563a551cbe40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:25:02.646 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:02.651+0000 7f249e9e4b80 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x563a551c8020 0x563a551cc380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:25:02.646 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:02.651+0000 7f249e9e4b80 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x563a54fe5f50 con 0x563a551c8020 2024-05-27T07:25:02.646 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:02.651+0000 7f249e9e4b80 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x563a55047d50 con 0x563a551c20d0 2024-05-27T07:25:02.646 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:02.651+0000 7f249e9e4b80 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x563a551ceab0 con 0x563a55042df0 2024-05-27T07:25:02.646 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:02.651+0000 7f249e9e4b80 10 client.0 _ll_register_callbacks cb 0x563a55076560 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T07:25:02.646 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:02.651+0000 7f248e56c700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x563a551c8020 0x563a551cc380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:25:02.646 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:02.651+0000 7f248e56c700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x563a551c8020 0x563a551cc380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.52:45146/0 (socket says 172.21.15.52:45146) 2024-05-27T07:25:02.647 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:02.651+0000 7f248e56c700 1 -- 172.21.15.52:0/1668355085 learned_addr learned my addr 172.21.15.52:0/1668355085 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-05-27T07:25:02.647 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:02.652+0000 7f248dd6b700 1 --2- 172.21.15.52:0/1668355085 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x563a55042df0 0x563a551cb900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:25:02.647 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:02.652+0000 7f248d56a700 1 --2- 172.21.15.52:0/1668355085 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x563a551c20d0 0x563a551cbe40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:25:02.647 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:02.652+0000 7f248e56c700 1 -- 172.21.15.52:0/1668355085 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x563a55042df0 msgr2=0x563a551cb900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:25:02.647 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:02.652+0000 7f248e56c700 1 --2- 172.21.15.52:0/1668355085 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x563a55042df0 0x563a551cb900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:25:02.647 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:02.652+0000 7f248e56c700 1 -- 172.21.15.52:0/1668355085 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x563a551c20d0 msgr2=0x563a551cbe40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:25:02.647 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:02.652+0000 7f248e56c700 1 --2- 172.21.15.52:0/1668355085 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x563a551c20d0 0x563a551cbe40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:25:02.647 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:02.652+0000 7f248e56c700 1 -- 172.21.15.52:0/1668355085 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({osdmap=0}) -- 0x563a5502f880 con 0x563a551c8020 2024-05-27T07:25:02.647 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:02.652+0000 7f248e56c700 1 --2- 172.21.15.52:0/1668355085 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x563a551c8020 0x563a551cc380 secure :-1 s=READY pgs=330 cs=0 l=1 rev1=1 crypto rx=0x7f2480008ed0 tx=0x7f24800114a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4e75ac22651ac08d server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:25:02.647 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:02.652+0000 7f24717fa700 10 client.0 ms_handle_connect on v2:172.21.15.52:3300/0 2024-05-27T07:25:02.647 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:02.652+0000 7f24717fa700 1 -- 172.21.15.52:0/1668355085 <== mon.0 v2:172.21.15.52:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2480011e10 con 0x563a551c8020 2024-05-27T07:25:02.647 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:02.652+0000 7f24717fa700 1 -- 172.21.15.52:0/1668355085 <== mon.0 v2:172.21.15.52:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9121+0+0 (secure 0 0 0) 0x7f2480016440 con 0x563a551c8020 2024-05-27T07:25:02.647 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:02.652+0000 7f249e9e4b80 1 -- 172.21.15.52:0/1668355085 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f247c0a96d0 con 0x563a551c8020 2024-05-27T07:25:02.648 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:02.652+0000 7f24717fa700 1 -- 172.21.15.52:0/1668355085 <== mon.0 v2:172.21.15.52:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f2480024050 con 0x563a551c8020 2024-05-27T07:25:02.648 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:02.652+0000 7f249e9e4b80 10 client.15135 fetch_fsmap learned FSMap version 29 2024-05-27T07:25:02.648 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:02.653+0000 7f249e9e4b80 1 -- 172.21.15.52:0/1668355085 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x563a55099710 con 0x563a551c8020 2024-05-27T07:25:02.648 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:02.653+0000 7f24717fa700 1 -- 172.21.15.52:0/1668355085 <== mon.0 v2:172.21.15.52:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f247c0a96d0 con 0x563a551c8020 2024-05-27T07:25:02.648 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:02.653+0000 7f24717fa700 1 -- 172.21.15.52:0/1668355085 <== mon.0 v2:172.21.15.52:3300/0 5 ==== fsmap(e 29) ==== 8362+0+0 (secure 0 0 0) 0x7f248001f4d0 con 0x563a551c8020 2024-05-27T07:25:02.648 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:02.653+0000 7f249e9e4b80 10 client.15135 fetch_fsmap finished waiting for FSMap version 29 2024-05-27T07:25:02.648 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:02.653+0000 7f249e9e4b80 10 client.15135 resolve_mds: resolved name 'h' to daemon mds.h 2024-05-27T07:25:02.648 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:02.653+0000 7f249e9e4b80 20 client.15135 populate_metadata read hostname 'smithi052' 2024-05-27T07:25:02.648 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:02.653+0000 7f249e9e4b80 1 --2- 172.21.15.52:0/1668355085 >> [v2:172.21.15.105:6836/191393149,v1:172.21.15.105:6837/191393149] conn(0x563a551dbba0 0x563a551dd010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:25:02.648 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:02.653+0000 7f249e9e4b80 4 client.15135 mds_command: new command op to 24497 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-27T07:25:02.648 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:02.653+0000 7f249e9e4b80 1 -- 172.21.15.52:0/1668355085 --> [v2:172.21.15.105:6836/191393149,v1:172.21.15.105:6837/191393149] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x563a550f1c70 con 0x563a551dbba0 2024-05-27T07:25:02.648 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:02.653+0000 7f248dd6b700 1 --2- 172.21.15.52:0/1668355085 >> [v2:172.21.15.105:6836/191393149,v1:172.21.15.105:6837/191393149] conn(0x563a551dbba0 0x563a551dd010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:25:02.648 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:02.654+0000 7f24717fa700 10 client.15135 ms_handle_connect on v2:172.21.15.105:6836/191393149 2024-05-27T07:25:02.649 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:02.654+0000 7f248dd6b700 1 --2- 172.21.15.52:0/1668355085 >> [v2:172.21.15.105:6836/191393149,v1:172.21.15.105:6837/191393149] conn(0x563a551dbba0 0x563a551dd010 crc :-1 s=READY pgs=16 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=9fa44a58378b3661 server_cookie=a35ff8cd56118707 in_seq=0 out_seq=0 2024-05-27T07:25:02.649 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:02.655+0000 7f24717fa700 1 -- 172.21.15.52:0/1668355085 <== mds.1 v2:172.21.15.105:6836/191393149 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24041 (crc 0 0 0) 0x563a550f1c70 con 0x563a551dbba0 2024-05-27T07:25:02.649 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:02.655+0000 7f24717fa700 10 client.15135 handle_command_reply: tid=0 2024-05-27T07:25:02.649 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:02.655+0000 7f249e9e4b80 1 client.15135 shutdown 2024-05-27T07:25:02.649 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:02.655+0000 7f249e9e4b80 1 -- 172.21.15.52:0/1668355085 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x563a551c8020 msgr2=0x563a551cc380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:25:02.649 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:02.655+0000 7f249e9e4b80 1 --2- 172.21.15.52:0/1668355085 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x563a551c8020 0x563a551cc380 secure :-1 s=READY pgs=330 cs=0 l=1 rev1=1 crypto rx=0x7f2480008ed0 tx=0x7f24800114a0 comp rx=0 tx=0).stop 2024-05-27T07:25:02.649 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:02.655+0000 7f249e9e4b80 1 -- 172.21.15.52:0/1668355085 shutdown_connections 2024-05-27T07:25:02.649 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:02.655+0000 7f249e9e4b80 1 --2- 172.21.15.52:0/1668355085 >> [v2:172.21.15.105:6836/191393149,v1:172.21.15.105:6837/191393149] conn(0x563a551dbba0 0x563a551dd010 crc :-1 s=READY pgs=16 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:25:02.649 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:02.655+0000 7f249e9e4b80 1 --2- 172.21.15.52:0/1668355085 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x563a551c8020 0x563a551cc380 unknown :-1 s=CLOSED pgs=330 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:25:02.649 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:02.655+0000 7f24717fa700 0 client.15135 ms_handle_reset on v2:172.21.15.105:6836/191393149 2024-05-27T07:25:02.649 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:02.655+0000 7f249e9e4b80 1 --2- 172.21.15.52:0/1668355085 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x563a55042df0 0x563a551cb900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:25:02.649 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:02.655+0000 7f249e9e4b80 1 --2- 172.21.15.52:0/1668355085 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x563a551c20d0 0x563a551cbe40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:25:02.649 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:02.655+0000 7f249e9e4b80 1 -- 172.21.15.52:0/1668355085 >> 172.21.15.52:0/1668355085 conn(0x563a55045720 msgr2=0x563a551c0b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:25:02.650 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:02.655+0000 7f249e9e4b80 1 -- 172.21.15.52:0/1668355085 shutdown_connections 2024-05-27T07:25:02.650 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:02.655+0000 7f249e9e4b80 1 -- 172.21.15.52:0/1668355085 wait complete. 2024-05-27T07:25:02.650 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:02.655+0000 7f249e9e4b80 20 client.15135 trim_cache size 0 max 16384 2024-05-27T07:25:02.668 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:02.677+0000 7f249e9e4b80 1 Processor -- start 2024-05-27T07:25:02.668 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:02.677+0000 7f249e9e4b80 1 -- start start 2024-05-27T07:25:02.668 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:02.677+0000 7f249e9e4b80 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x563a55042df0 0x563a551bd810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:25:02.668 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:02.677+0000 7f249e9e4b80 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x563a551c20d0 0x563a551c0b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:25:02.668 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:02.677+0000 7f249e9e4b80 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x563a551c8020 0x563a551bca30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:25:02.668 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:02.677+0000 7f249e9e4b80 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x563a551db350 con 0x563a551c8020 2024-05-27T07:25:02.668 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:02.677+0000 7f249e9e4b80 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x563a551dba20 con 0x563a551c20d0 2024-05-27T07:25:02.668 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:02.677+0000 7f249e9e4b80 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x563a551dc0f0 con 0x563a55042df0 2024-05-27T07:25:02.668 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:02.677+0000 7f248e56c700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x563a551c8020 0x563a551bca30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:25:02.669 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:02.677+0000 7f248e56c700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x563a551c8020 0x563a551bca30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.52:45160/0 (socket says 172.21.15.52:45160) 2024-05-27T07:25:02.669 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:02.677+0000 7f248e56c700 1 -- 172.21.15.52:0/3138314005 learned_addr learned my addr 172.21.15.52:0/3138314005 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-05-27T07:25:02.669 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:02.677+0000 7f248dd6b700 1 --2- 172.21.15.52:0/3138314005 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x563a55042df0 0x563a551bd810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:25:02.669 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:02.677+0000 7f248d56a700 1 --2- 172.21.15.52:0/3138314005 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x563a551c20d0 0x563a551c0b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:25:02.669 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:02.677+0000 7f248e56c700 1 -- 172.21.15.52:0/3138314005 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x563a55042df0 msgr2=0x563a551bd810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:25:02.669 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:02.677+0000 7f248e56c700 1 --2- 172.21.15.52:0/3138314005 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x563a55042df0 0x563a551bd810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:25:02.669 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:02.677+0000 7f248e56c700 1 -- 172.21.15.52:0/3138314005 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x563a551c20d0 msgr2=0x563a551c0b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:25:02.669 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:02.678+0000 7f248e56c700 1 --2- 172.21.15.52:0/3138314005 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x563a551c20d0 0x563a551c0b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:25:02.669 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:02.678+0000 7f248e56c700 1 -- 172.21.15.52:0/3138314005 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x563a551c72a0 con 0x563a551c8020 2024-05-27T07:25:02.669 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:02.678+0000 7f248d56a700 1 --2- 172.21.15.52:0/3138314005 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x563a551c20d0 0x563a551c0b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-27T07:25:02.669 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:02.678+0000 7f248e56c700 1 --2- 172.21.15.52:0/3138314005 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x563a551c8020 0x563a551bca30 secure :-1 s=READY pgs=331 cs=0 l=1 rev1=1 crypto rx=0x7f2480008bf0 tx=0x7f2480007da0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=384d486ad2666cfb server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:25:02.669 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:02.678+0000 7f24737fe700 1 -- 172.21.15.52:0/3138314005 <== mon.0 v2:172.21.15.52:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f248001fd20 con 0x563a551c8020 2024-05-27T07:25:02.669 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:02.678+0000 7f24737fe700 1 -- 172.21.15.52:0/3138314005 <== mon.0 v2:172.21.15.52:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f248001b070 con 0x563a551c8020 2024-05-27T07:25:02.670 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:02.678+0000 7f24737fe700 1 -- 172.21.15.52:0/3138314005 <== mon.0 v2:172.21.15.52:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2480004e30 con 0x563a551c8020 2024-05-27T07:25:02.673 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:02.679+0000 7f249e9e4b80 1 -- 172.21.15.52:0/3138314005 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x563a551c8020 msgr2=0x563a551bca30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:25:02.673 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:02.679+0000 7f249e9e4b80 1 --2- 172.21.15.52:0/3138314005 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x563a551c8020 0x563a551bca30 secure :-1 s=READY pgs=331 cs=0 l=1 rev1=1 crypto rx=0x7f2480008bf0 tx=0x7f2480007da0 comp rx=0 tx=0).stop 2024-05-27T07:25:02.673 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:02.679+0000 7f249e9e4b80 1 -- 172.21.15.52:0/3138314005 shutdown_connections 2024-05-27T07:25:02.673 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:02.679+0000 7f249e9e4b80 1 --2- 172.21.15.52:0/3138314005 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x563a551c8020 0x563a551bca30 unknown :-1 s=CLOSED pgs=331 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:25:02.673 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:02.679+0000 7f249e9e4b80 1 --2- 172.21.15.52:0/3138314005 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x563a55042df0 0x563a551bd810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:25:02.673 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:02.679+0000 7f249e9e4b80 1 --2- 172.21.15.52:0/3138314005 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x563a551c20d0 0x563a551c0b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:25:02.673 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:02.679+0000 7f249e9e4b80 1 -- 172.21.15.52:0/3138314005 >> 172.21.15.52:0/3138314005 conn(0x563a55045720 msgr2=0x563a551cc200 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:25:02.674 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:02.679+0000 7f249e9e4b80 1 -- 172.21.15.52:0/3138314005 shutdown_connections 2024-05-27T07:25:02.674 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:02.679+0000 7f249e9e4b80 1 -- 172.21.15.52:0/3138314005 wait complete. 2024-05-27T07:25:02.674 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:02.679+0000 7f249e9e4b80 1 -- start start 2024-05-27T07:25:02.674 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:02.679+0000 7f249e9e4b80 1 Processor -- start 2024-05-27T07:25:02.674 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:02.680+0000 7f249e9e4b80 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x563a55042df0 0x563a551e6a60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:25:02.674 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:02.680+0000 7f249e9e4b80 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x563a551c20d0 0x563a551e2a20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:25:02.674 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:02.680+0000 7f249e9e4b80 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x563a551c8020 0x563a551e2f60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:25:02.674 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:02.680+0000 7f249e9e4b80 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x563a551dce90 con 0x563a551c8020 2024-05-27T07:25:02.674 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:02.680+0000 7f249e9e4b80 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x563a551dc7c0 con 0x563a55042df0 2024-05-27T07:25:02.674 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:02.680+0000 7f249e9e4b80 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x563a551dd560 con 0x563a551c20d0 2024-05-27T07:25:02.674 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:02.680+0000 7f249e9e4b80 10 client.0 _ll_register_callbacks cb 0x563a55076560 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T07:25:02.675 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:02.680+0000 7f248e56c700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x563a551c8020 0x563a551e2f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:25:02.675 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:02.680+0000 7f248d56a700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x563a551c20d0 0x563a551e2a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:25:02.675 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:02.680+0000 7f248dd6b700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x563a55042df0 0x563a551e6a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:25:02.675 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:02.680+0000 7f248e56c700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x563a551c8020 0x563a551e2f60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.52:45162/0 (socket says 172.21.15.52:45162) 2024-05-27T07:25:02.675 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:02.680+0000 7f248e56c700 1 -- 172.21.15.52:0/732628858 learned_addr learned my addr 172.21.15.52:0/732628858 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-05-27T07:25:02.675 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:02.680+0000 7f248e56c700 1 -- 172.21.15.52:0/732628858 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x563a551c20d0 msgr2=0x563a551e2a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:25:02.676 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:02.681+0000 7f248e56c700 1 --2- 172.21.15.52:0/732628858 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x563a551c20d0 0x563a551e2a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:25:02.676 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:02.681+0000 7f248e56c700 1 -- 172.21.15.52:0/732628858 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x563a55042df0 msgr2=0x563a551e6a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:25:02.676 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:02.681+0000 7f248e56c700 1 --2- 172.21.15.52:0/732628858 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x563a55042df0 0x563a551e6a60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:25:02.676 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:02.681+0000 7f248e56c700 1 -- 172.21.15.52:0/732628858 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({osdmap=0}) -- 0x563a551e34a0 con 0x563a551c8020 2024-05-27T07:25:02.676 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:02.681+0000 7f248e56c700 1 --2- 172.21.15.52:0/732628858 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x563a551c8020 0x563a551e2f60 secure :-1 s=READY pgs=332 cs=0 l=1 rev1=1 crypto rx=0x7f2480002230 tx=0x7f2480005e90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1ae87811bca44875 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:25:02.676 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:02.681+0000 7f24717fa700 10 client.0 ms_handle_connect on v2:172.21.15.52:3300/0 2024-05-27T07:25:02.676 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:02.681+0000 7f24717fa700 1 -- 172.21.15.52:0/732628858 <== mon.0 v2:172.21.15.52:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f248001f3b0 con 0x563a551c8020 2024-05-27T07:25:02.676 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:02.681+0000 7f24717fa700 1 -- 172.21.15.52:0/732628858 <== mon.0 v2:172.21.15.52:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9121+0+0 (secure 0 0 0) 0x7f248001f550 con 0x563a551c8020 2024-05-27T07:25:02.676 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:02.681+0000 7f249e9e4b80 1 -- 172.21.15.52:0/732628858 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f248001b070 con 0x563a551c8020 2024-05-27T07:25:02.676 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:02.681+0000 7f24717fa700 1 -- 172.21.15.52:0/732628858 <== mon.0 v2:172.21.15.52:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f2480004a90 con 0x563a551c8020 2024-05-27T07:25:02.677 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:02.681+0000 7f249e9e4b80 10 client.15141 fetch_fsmap learned FSMap version 29 2024-05-27T07:25:02.677 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:02.681+0000 7f249e9e4b80 1 -- 172.21.15.52:0/732628858 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x563a55246f50 con 0x563a551c8020 2024-05-27T07:25:02.677 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:02.681+0000 7f24717fa700 1 -- 172.21.15.52:0/732628858 <== mon.0 v2:172.21.15.52:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f248001b070 con 0x563a551c8020 2024-05-27T07:25:02.677 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:02.682+0000 7f24717fa700 1 -- 172.21.15.52:0/732628858 <== mon.0 v2:172.21.15.52:3300/0 5 ==== fsmap(e 29) ==== 8362+0+0 (secure 0 0 0) 0x7f2480033c30 con 0x563a551c8020 2024-05-27T07:25:02.677 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:02.682+0000 7f249e9e4b80 10 client.15141 fetch_fsmap finished waiting for FSMap version 29 2024-05-27T07:25:02.677 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:02.682+0000 7f249e9e4b80 10 client.15141 resolve_mds: resolved name 'h' to daemon mds.h 2024-05-27T07:25:02.677 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:02.682+0000 7f249e9e4b80 20 client.15141 populate_metadata read hostname 'smithi052' 2024-05-27T07:25:02.677 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:02.682+0000 7f249e9e4b80 1 --2- 172.21.15.52:0/732628858 >> [v2:172.21.15.105:6836/191393149,v1:172.21.15.105:6837/191393149] conn(0x563a552491a0 0x563a5524a610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:25:02.677 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:02.682+0000 7f249e9e4b80 4 client.15141 mds_command: new command op to 24497 tid=0 multi_id=0 [{"prefix": "status"}] 2024-05-27T07:25:02.677 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:02.682+0000 7f249e9e4b80 1 -- 172.21.15.52:0/732628858 --> [v2:172.21.15.105:6836/191393149,v1:172.21.15.105:6837/191393149] -- command(tid 0: {"prefix": "status"}) -- 0x563a550eb450 con 0x563a552491a0 2024-05-27T07:25:02.677 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:02.682+0000 7f248dd6b700 1 --2- 172.21.15.52:0/732628858 >> [v2:172.21.15.105:6836/191393149,v1:172.21.15.105:6837/191393149] conn(0x563a552491a0 0x563a5524a610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:25:02.677 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:02.682+0000 7f24717fa700 10 client.15141 ms_handle_connect on v2:172.21.15.105:6836/191393149 2024-05-27T07:25:02.678 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:02.682+0000 7f248dd6b700 1 --2- 172.21.15.52:0/732628858 >> [v2:172.21.15.105:6836/191393149,v1:172.21.15.105:6837/191393149] conn(0x563a552491a0 0x563a5524a610 crc :-1 s=READY pgs=17 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=10c8ecb4ab351c8b server_cookie=f05fbf92fab96231 in_seq=0 out_seq=0 2024-05-27T07:25:02.678 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:02.682+0000 7f24717fa700 1 -- 172.21.15.52:0/732628858 <== mds.1 v2:172.21.15.105:6836/191393149 1 ==== command_reply(tid 0: 0 ) ==== 8+0+327 (crc 0 0 0) 0x563a550eb450 con 0x563a552491a0 2024-05-27T07:25:02.678 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:02.682+0000 7f24717fa700 10 client.15141 handle_command_reply: tid=0 2024-05-27T07:25:02.678 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:02.682+0000 7f249e9e4b80 1 client.15141 shutdown 2024-05-27T07:25:02.678 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:02.683+0000 7f249e9e4b80 1 -- 172.21.15.52:0/732628858 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x563a551c8020 msgr2=0x563a551e2f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:25:02.678 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:02.683+0000 7f249e9e4b80 1 --2- 172.21.15.52:0/732628858 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x563a551c8020 0x563a551e2f60 secure :-1 s=READY pgs=332 cs=0 l=1 rev1=1 crypto rx=0x7f2480002230 tx=0x7f2480005e90 comp rx=0 tx=0).stop 2024-05-27T07:25:02.678 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:02.683+0000 7f249e9e4b80 1 -- 172.21.15.52:0/732628858 shutdown_connections 2024-05-27T07:25:02.678 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:02.683+0000 7f249e9e4b80 1 --2- 172.21.15.52:0/732628858 >> [v2:172.21.15.105:6836/191393149,v1:172.21.15.105:6837/191393149] conn(0x563a552491a0 0x563a5524a610 crc :-1 s=READY pgs=17 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:25:02.678 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:02.683+0000 7f249e9e4b80 1 --2- 172.21.15.52:0/732628858 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x563a551c8020 0x563a551e2f60 unknown :-1 s=CLOSED pgs=332 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:25:02.678 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:02.683+0000 7f249e9e4b80 1 --2- 172.21.15.52:0/732628858 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x563a55042df0 0x563a551e6a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:25:02.678 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:02.683+0000 7f24717fa700 0 client.15141 ms_handle_reset on v2:172.21.15.105:6836/191393149 2024-05-27T07:25:02.678 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:02.683+0000 7f249e9e4b80 1 --2- 172.21.15.52:0/732628858 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x563a551c20d0 0x563a551e2a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:25:02.679 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:02.683+0000 7f249e9e4b80 1 -- 172.21.15.52:0/732628858 >> 172.21.15.52:0/732628858 conn(0x563a55045720 msgr2=0x563a551cc200 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:25:02.679 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:02.683+0000 7f249e9e4b80 1 -- 172.21.15.52:0/732628858 shutdown_connections 2024-05-27T07:25:02.679 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:02.683+0000 7f249e9e4b80 1 -- 172.21.15.52:0/732628858 wait complete. 2024-05-27T07:25:02.679 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:02.683+0000 7f249e9e4b80 20 client.15141 trim_cache size 0 max 16384 2024-05-27T07:25:02.679 INFO:teuthology.orchestra.run.smithi052.stdout:{ 2024-05-27T07:25:02.679 INFO:teuthology.orchestra.run.smithi052.stdout: "cluster_fsid": "73df2636-1bf8-11ef-bc9b-c7b262605968", 2024-05-27T07:25:02.679 INFO:teuthology.orchestra.run.smithi052.stdout: "whoami": 1, 2024-05-27T07:25:02.679 INFO:teuthology.orchestra.run.smithi052.stdout: "id": 24497, 2024-05-27T07:25:02.679 INFO:teuthology.orchestra.run.smithi052.stdout: "want_state": "up:active", 2024-05-27T07:25:02.679 INFO:teuthology.orchestra.run.smithi052.stdout: "state": "up:active", 2024-05-27T07:25:02.680 INFO:teuthology.orchestra.run.smithi052.stdout: "fs_name": "cephfs", 2024-05-27T07:25:02.680 INFO:teuthology.orchestra.run.smithi052.stdout: "rank_uptime": 129.83541498899999, 2024-05-27T07:25:02.680 INFO:teuthology.orchestra.run.smithi052.stdout: "mdsmap_epoch": 29, 2024-05-27T07:25:02.680 INFO:teuthology.orchestra.run.smithi052.stdout: "osdmap_epoch": 90, 2024-05-27T07:25:02.680 INFO:teuthology.orchestra.run.smithi052.stdout: "osdmap_epoch_barrier": 88, 2024-05-27T07:25:02.680 INFO:teuthology.orchestra.run.smithi052.stdout: "uptime": 219.094682409 2024-05-27T07:25:02.680 INFO:teuthology.orchestra.run.smithi052.stdout:} 2024-05-27T07:25:02.681 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:02.683+0000 7f248ffcf700 1 -- 172.21.15.52:0/1681135330 >> [v2:172.21.15.52:6800/3513075805,v1:172.21.15.52:6801/3513075805] conn(0x7f24740b2880 msgr2=0x7f24740b4d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:25:02.681 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:02.683+0000 7f248ffcf700 1 --2- 172.21.15.52:0/1681135330 >> [v2:172.21.15.52:6800/3513075805,v1:172.21.15.52:6801/3513075805] conn(0x7f24740b2880 0x7f24740b4d90 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7f2484009990 tx=0x7f24840036e0 comp rx=0 tx=0).stop 2024-05-27T07:25:02.681 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:02.683+0000 7f248ffcf700 1 -- 172.21.15.52:0/1681135330 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f2488107e00 msgr2=0x7f24880ff940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:25:02.681 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:02.683+0000 7f248ffcf700 1 --2- 172.21.15.52:0/1681135330 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f2488107e00 0x7f24880ff940 secure :-1 s=READY pgs=328 cs=0 l=1 rev1=1 crypto rx=0x7f247c00d980 tx=0x7f247c00dc90 comp rx=0 tx=0).stop 2024-05-27T07:25:02.681 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:02.684+0000 7f248ffcf700 1 -- 172.21.15.52:0/1681135330 shutdown_connections 2024-05-27T07:25:02.681 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:02.684+0000 7f248ffcf700 1 --2- 172.21.15.52:0/1681135330 >> [v2:172.21.15.52:6800/3513075805,v1:172.21.15.52:6801/3513075805] conn(0x7f24740b2880 0x7f24740b4d90 unknown :-1 s=CLOSED pgs=210 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:25:02.681 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:02.684+0000 7f248ffcf700 1 --2- 172.21.15.52:0/1681135330 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f2488108750 0x7f24880ffe80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:25:02.681 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:02.684+0000 7f248ffcf700 1 --2- 172.21.15.52:0/1681135330 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f24881034d0 0x7f24880ff400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:25:02.681 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:02.684+0000 7f248ffcf700 1 --2- 172.21.15.52:0/1681135330 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f2488107e00 0x7f24880ff940 unknown :-1 s=CLOSED pgs=328 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:25:02.681 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:02.684+0000 7f248ffcf700 1 -- 172.21.15.52:0/1681135330 >> 172.21.15.52:0/1681135330 conn(0x7f24880fef20 msgr2=0x7f248806c470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:25:02.681 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:02.684+0000 7f248ffcf700 1 -- 172.21.15.52:0/1681135330 shutdown_connections 2024-05-27T07:25:02.682 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:02.684+0000 7f248ffcf700 1 -- 172.21.15.52:0/1681135330 wait complete. 2024-05-27T07:25:02.979 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:25:02 smithi052 ceph-mon[31955]: pgmap v396: 129 pgs: 129 active+clean; 647 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 9 op/s 2024-05-27T07:25:03.101 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:25:02 smithi173 ceph-mon[37000]: pgmap v396: 129 pgs: 129 active+clean; 647 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 9 op/s 2024-05-27T07:25:03.190 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:25:02 smithi105 ceph-mon[35977]: pgmap v396: 129 pgs: 129 active+clean; 647 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 9 op/s 2024-05-27T07:25:03.415 DEBUG:teuthology.orchestra.run.smithi052:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:38f9d900c83c63799fbdbe61acc9a11b0d3554a6 shell --fsid 73df2636-1bf8-11ef-bc9b-c7b262605968 -- ceph tell mds.b status 2024-05-27T07:25:03.665 INFO:teuthology.orchestra.run.smithi052.stderr:Inferring config /var/lib/ceph/73df2636-1bf8-11ef-bc9b-c7b262605968/mon.a/config 2024-05-27T07:25:04.778 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:25:04 smithi052 ceph-mon[31955]: pgmap v397: 129 pgs: 129 active+clean; 647 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 9 op/s 2024-05-27T07:25:05.102 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:25:04 smithi173 ceph-mon[37000]: pgmap v397: 129 pgs: 129 active+clean; 647 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 9 op/s 2024-05-27T07:25:05.190 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:25:04 smithi105 ceph-mon[35977]: pgmap v397: 129 pgs: 129 active+clean; 647 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 9 op/s 2024-05-27T07:25:05.352 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:05.362+0000 7f5ff0053700 1 -- 172.21.15.52:0/1690989878 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f5fe8107de0 msgr2=0x7f5fe81081f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:25:05.352 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:05.362+0000 7f5ff0053700 1 --2- 172.21.15.52:0/1690989878 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f5fe8107de0 0x7f5fe81081f0 secure :-1 s=READY pgs=333 cs=0 l=1 rev1=1 crypto rx=0x7f5fd80099c0 tx=0x7f5fd8009cd0 comp rx=0 tx=0).stop 2024-05-27T07:25:05.352 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:05.362+0000 7f5ff0053700 1 -- 172.21.15.52:0/1690989878 shutdown_connections 2024-05-27T07:25:05.353 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:05.362+0000 7f5ff0053700 1 --2- 172.21.15.52:0/1690989878 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f5fe8108730 0x7f5fe8110090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:25:05.353 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:05.362+0000 7f5ff0053700 1 --2- 172.21.15.52:0/1690989878 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f5fe81034b0 0x7f5fe81038c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:25:05.353 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:05.362+0000 7f5ff0053700 1 --2- 172.21.15.52:0/1690989878 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f5fe8107de0 0x7f5fe81081f0 unknown :-1 s=CLOSED pgs=333 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:25:05.353 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:05.362+0000 7f5ff0053700 1 -- 172.21.15.52:0/1690989878 >> 172.21.15.52:0/1690989878 conn(0x7f5fe80fede0 msgr2=0x7f5fe8101200 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:25:05.353 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:05.362+0000 7f5ff0053700 1 -- 172.21.15.52:0/1690989878 shutdown_connections 2024-05-27T07:25:05.353 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:05.362+0000 7f5ff0053700 1 -- 172.21.15.52:0/1690989878 wait complete. 2024-05-27T07:25:05.355 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:05.363+0000 7f5ff0053700 1 Processor -- start 2024-05-27T07:25:05.355 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:05.363+0000 7f5ff0053700 1 -- start start 2024-05-27T07:25:05.355 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:05.363+0000 7f5ff0053700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f5fe81034b0 0x7f5fe80ff2c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:25:05.355 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:05.363+0000 7f5ff0053700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f5fe8107de0 0x7f5fe80ff800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:25:05.355 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:05.363+0000 7f5ff0053700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f5fe8108730 0x7f5fe80ffd40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:25:05.355 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:05.363+0000 7f5ff0053700 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x7f5fe81133b0 con 0x7f5fe81034b0 2024-05-27T07:25:05.355 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:05.363+0000 7f5ff0053700 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f5fe8113230 con 0x7f5fe8107de0 2024-05-27T07:25:05.355 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:05.363+0000 7f5ff0053700 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f5fe8113530 con 0x7f5fe8108730 2024-05-27T07:25:05.356 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:05.363+0000 7f5feddef700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f5fe81034b0 0x7f5fe80ff2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:25:05.356 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:05.363+0000 7f5feddef700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f5fe81034b0 0x7f5fe80ff2c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.52:45182/0 (socket says 172.21.15.52:45182) 2024-05-27T07:25:05.356 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:05.364+0000 7f5feddef700 1 -- 172.21.15.52:0/2278690597 learned_addr learned my addr 172.21.15.52:0/2278690597 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-05-27T07:25:05.356 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:05.364+0000 7f5fee5f0700 1 --2- 172.21.15.52:0/2278690597 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f5fe8108730 0x7f5fe80ffd40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:25:05.356 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:05.364+0000 7f5fed5ee700 1 --2- 172.21.15.52:0/2278690597 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f5fe8107de0 0x7f5fe80ff800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:25:05.356 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:05.364+0000 7f5feddef700 1 -- 172.21.15.52:0/2278690597 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f5fe8108730 msgr2=0x7f5fe80ffd40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:25:05.356 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:05.364+0000 7f5feddef700 1 --2- 172.21.15.52:0/2278690597 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f5fe8108730 0x7f5fe80ffd40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:25:05.356 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:05.364+0000 7f5feddef700 1 -- 172.21.15.52:0/2278690597 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f5fe8107de0 msgr2=0x7f5fe80ff800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:25:05.356 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:05.364+0000 7f5feddef700 1 --2- 172.21.15.52:0/2278690597 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f5fe8107de0 0x7f5fe80ff800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:25:05.356 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:05.364+0000 7f5feddef700 1 -- 172.21.15.52:0/2278690597 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5fe81a9230 con 0x7f5fe81034b0 2024-05-27T07:25:05.356 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:05.364+0000 7f5feddef700 1 --2- 172.21.15.52:0/2278690597 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f5fe81034b0 0x7f5fe80ff2c0 secure :-1 s=READY pgs=334 cs=0 l=1 rev1=1 crypto rx=0x7f5fe400c920 tx=0x7f5fe400cce0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:25:05.356 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:05.364+0000 7f5fdeffd700 1 -- 172.21.15.52:0/2278690597 <== mon.0 v2:172.21.15.52:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5fe4007b00 con 0x7f5fe81034b0 2024-05-27T07:25:05.356 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:05.364+0000 7f5ff0053700 1 -- 172.21.15.52:0/2278690597 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5fe81a9520 con 0x7f5fe81034b0 2024-05-27T07:25:05.357 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:05.364+0000 7f5fdeffd700 1 -- 172.21.15.52:0/2278690597 <== mon.0 v2:172.21.15.52:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f5fe4004ce0 con 0x7f5fe81034b0 2024-05-27T07:25:05.357 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:05.364+0000 7f5fdeffd700 1 -- 172.21.15.52:0/2278690597 <== mon.0 v2:172.21.15.52:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5fe4005650 con 0x7f5fe81034b0 2024-05-27T07:25:05.357 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:05.364+0000 7f5ff0053700 1 -- 172.21.15.52:0/2278690597 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5fe81a9a80 con 0x7f5fe81034b0 2024-05-27T07:25:05.357 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:05.367+0000 7f5fdeffd700 1 -- 172.21.15.52:0/2278690597 <== mon.0 v2:172.21.15.52:3300/0 4 ==== mgrmap(e 18) ==== 149652+0+0 (secure 0 0 0) 0x7f5fe4004750 con 0x7f5fe81034b0 2024-05-27T07:25:05.358 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:05.368+0000 7f5fdeffd700 1 --2- 172.21.15.52:0/2278690597 >> [v2:172.21.15.52:6800/3513075805,v1:172.21.15.52:6801/3513075805] conn(0x7f5fd40b2900 0x7f5fd40b4e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:25:05.358 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:05.368+0000 7f5fed5ee700 1 --2- 172.21.15.52:0/2278690597 >> [v2:172.21.15.52:6800/3513075805,v1:172.21.15.52:6801/3513075805] conn(0x7f5fd40b2900 0x7f5fd40b4e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:25:05.358 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:05.368+0000 7f5fdeffd700 1 -- 172.21.15.52:0/2278690597 <== mon.0 v2:172.21.15.52:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9121+0+0 (secure 0 0 0) 0x7f5fe40e21b0 con 0x7f5fe81034b0 2024-05-27T07:25:05.358 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:05.368+0000 7f5ffea68b80 1 Processor -- start 2024-05-27T07:25:05.360 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:05.369+0000 7f5ffea68b80 1 -- start start 2024-05-27T07:25:05.360 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:05.369+0000 7f5fed5ee700 1 --2- 172.21.15.52:0/2278690597 >> [v2:172.21.15.52:6800/3513075805,v1:172.21.15.52:6801/3513075805] conn(0x7f5fd40b2900 0x7f5fd40b4e10 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7f5fd8000c00 tx=0x7f5fd801a040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:25:05.360 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:05.369+0000 7f5ffea68b80 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x5599299c9df0 0x559929b4d6c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:25:05.360 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:05.369+0000 7f5ffea68b80 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x559929b4dc00 0x559929b479f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:25:05.360 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:05.369+0000 7f5ffea68b80 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x559929b47f30 0x559929b48340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:25:05.360 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:05.369+0000 7f5ffea68b80 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x559929a76c40 con 0x5599299c9df0 2024-05-27T07:25:05.360 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:05.369+0000 7f5ffea68b80 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x559929a667e0 con 0x559929b4dc00 2024-05-27T07:25:05.360 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:05.369+0000 7f5ffea68b80 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x559929a6a820 con 0x559929b47f30 2024-05-27T07:25:05.360 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:05.369+0000 7f5feddef700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x5599299c9df0 0x559929b4d6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:25:05.360 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:05.369+0000 7f5feddef700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x5599299c9df0 0x559929b4d6c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.52:45186/0 (socket says 172.21.15.52:45186) 2024-05-27T07:25:05.361 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:05.369+0000 7f5feddef700 1 -- 172.21.15.52:0/2787400245 learned_addr learned my addr 172.21.15.52:0/2787400245 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-05-27T07:25:05.361 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:05.369+0000 7f5fed5ee700 1 --2- 172.21.15.52:0/2787400245 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x559929b4dc00 0x559929b479f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:25:05.361 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:05.369+0000 7f5fee5f0700 1 --2- 172.21.15.52:0/2787400245 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x559929b47f30 0x559929b48340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:25:05.361 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:05.369+0000 7f5feddef700 1 -- 172.21.15.52:0/2787400245 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x559929b47f30 msgr2=0x559929b48340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:25:05.361 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:05.369+0000 7f5feddef700 1 --2- 172.21.15.52:0/2787400245 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x559929b47f30 0x559929b48340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:25:05.361 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:05.369+0000 7f5feddef700 1 -- 172.21.15.52:0/2787400245 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x559929b4dc00 msgr2=0x559929b479f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:25:05.361 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:05.369+0000 7f5feddef700 1 --2- 172.21.15.52:0/2787400245 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x559929b4dc00 0x559929b479f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:25:05.361 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:05.369+0000 7f5feddef700 1 -- 172.21.15.52:0/2787400245 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5599299b6880 con 0x5599299c9df0 2024-05-27T07:25:05.361 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:05.369+0000 7f5fed5ee700 1 --2- 172.21.15.52:0/2787400245 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x559929b4dc00 0x559929b479f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-27T07:25:05.361 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:05.369+0000 7f5fee5f0700 1 --2- 172.21.15.52:0/2787400245 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x559929b47f30 0x559929b48340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-27T07:25:05.361 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:05.370+0000 7f5feddef700 1 --2- 172.21.15.52:0/2787400245 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x5599299c9df0 0x559929b4d6c0 secure :-1 s=READY pgs=335 cs=0 l=1 rev1=1 crypto rx=0x7f5fe40a6660 tx=0x7f5fe4007ca0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=13f24e98c2435957 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:25:05.361 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:05.370+0000 7f5fd3fff700 1 -- 172.21.15.52:0/2787400245 <== mon.0 v2:172.21.15.52:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5fe401d040 con 0x5599299c9df0 2024-05-27T07:25:05.362 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:05.370+0000 7f5fd3fff700 1 -- 172.21.15.52:0/2787400245 <== mon.0 v2:172.21.15.52:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f5fe40ad7a0 con 0x5599299c9df0 2024-05-27T07:25:05.362 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:05.370+0000 7f5fd3fff700 1 -- 172.21.15.52:0/2787400245 <== mon.0 v2:172.21.15.52:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5fe40a9650 con 0x5599299c9df0 2024-05-27T07:25:05.365 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:05.371+0000 7f5ffea68b80 1 -- 172.21.15.52:0/2787400245 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x5599299c9df0 msgr2=0x559929b4d6c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:25:05.365 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:05.371+0000 7f5ffea68b80 1 --2- 172.21.15.52:0/2787400245 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x5599299c9df0 0x559929b4d6c0 secure :-1 s=READY pgs=335 cs=0 l=1 rev1=1 crypto rx=0x7f5fe40a6660 tx=0x7f5fe4007ca0 comp rx=0 tx=0).stop 2024-05-27T07:25:05.366 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:05.371+0000 7f5ffea68b80 1 -- 172.21.15.52:0/2787400245 shutdown_connections 2024-05-27T07:25:05.366 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:05.371+0000 7f5ffea68b80 1 --2- 172.21.15.52:0/2787400245 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x559929b47f30 0x559929b48340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:25:05.366 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:05.371+0000 7f5ffea68b80 1 --2- 172.21.15.52:0/2787400245 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x5599299c9df0 0x559929b4d6c0 unknown :-1 s=CLOSED pgs=335 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:25:05.366 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:05.371+0000 7f5ffea68b80 1 --2- 172.21.15.52:0/2787400245 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x559929b4dc00 0x559929b479f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:25:05.366 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:05.371+0000 7f5ffea68b80 1 -- 172.21.15.52:0/2787400245 >> 172.21.15.52:0/2787400245 conn(0x559929a24aa0 msgr2=0x559929b425f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:25:05.366 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:05.371+0000 7f5ffea68b80 1 -- 172.21.15.52:0/2787400245 shutdown_connections 2024-05-27T07:25:05.366 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:05.371+0000 7f5ffea68b80 1 -- 172.21.15.52:0/2787400245 wait complete. 2024-05-27T07:25:05.366 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:05.371+0000 7f5ffea68b80 1 -- start start 2024-05-27T07:25:05.366 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:05.371+0000 7f5ffea68b80 1 Processor -- start 2024-05-27T07:25:05.366 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:05.372+0000 7f5ffea68b80 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5599299c9df0 0x559929b51330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:25:05.366 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:05.372+0000 7f5ffea68b80 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x559929b47f30 0x559929b51870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:25:05.367 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:05.372+0000 7f5ffea68b80 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x559929b4dc00 0x559929b51db0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:25:05.367 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:05.372+0000 7f5ffea68b80 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x55992996cf50 con 0x559929b47f30 2024-05-27T07:25:05.367 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:05.372+0000 7f5ffea68b80 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x559929a08770 con 0x5599299c9df0 2024-05-27T07:25:05.367 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:05.372+0000 7f5ffea68b80 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x559929b54690 con 0x559929b4dc00 2024-05-27T07:25:05.367 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:05.372+0000 7f5ffea68b80 10 client.0 _ll_register_callbacks cb 0x559929a018d0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T07:25:05.367 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:05.372+0000 7f5fed5ee700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x559929b47f30 0x559929b51870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:25:05.367 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:05.372+0000 7f5fed5ee700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x559929b47f30 0x559929b51870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.52:45200/0 (socket says 172.21.15.52:45200) 2024-05-27T07:25:05.367 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:05.372+0000 7f5fed5ee700 1 -- 172.21.15.52:0/3077080553 learned_addr learned my addr 172.21.15.52:0/3077080553 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-05-27T07:25:05.367 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:05.372+0000 7f5fee5f0700 1 --2- 172.21.15.52:0/3077080553 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x559929b4dc00 0x559929b51db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:25:05.367 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:05.372+0000 7f5feddef700 1 --2- 172.21.15.52:0/3077080553 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5599299c9df0 0x559929b51330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:25:05.367 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:05.372+0000 7f5fed5ee700 1 -- 172.21.15.52:0/3077080553 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x559929b4dc00 msgr2=0x559929b51db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:25:05.367 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:05.372+0000 7f5fed5ee700 1 --2- 172.21.15.52:0/3077080553 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x559929b4dc00 0x559929b51db0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:25:05.367 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:05.372+0000 7f5fed5ee700 1 -- 172.21.15.52:0/3077080553 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5599299c9df0 msgr2=0x559929b51330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:25:05.368 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:05.372+0000 7f5fed5ee700 1 --2- 172.21.15.52:0/3077080553 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5599299c9df0 0x559929b51330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:25:05.368 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:05.373+0000 7f5fed5ee700 1 -- 172.21.15.52:0/3077080553 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({osdmap=0}) -- 0x559929a05640 con 0x559929b47f30 2024-05-27T07:25:05.368 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:05.373+0000 7f5feddef700 1 --2- 172.21.15.52:0/3077080553 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5599299c9df0 0x559929b51330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-27T07:25:05.368 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:05.373+0000 7f5fed5ee700 1 --2- 172.21.15.52:0/3077080553 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x559929b47f30 0x559929b51870 secure :-1 s=READY pgs=336 cs=0 l=1 rev1=1 crypto rx=0x7f5fd80099c0 tx=0x7f5fd8009cd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7be065efa30be0e server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:25:05.368 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:05.373+0000 7f5fd17fa700 10 client.0 ms_handle_connect on v2:172.21.15.52:3300/0 2024-05-27T07:25:05.368 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:05.373+0000 7f5fd17fa700 1 -- 172.21.15.52:0/3077080553 <== mon.0 v2:172.21.15.52:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5fd8032060 con 0x559929b47f30 2024-05-27T07:25:05.368 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:05.373+0000 7f5fd17fa700 1 -- 172.21.15.52:0/3077080553 <== mon.0 v2:172.21.15.52:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9121+0+0 (secure 0 0 0) 0x7f5fd8022810 con 0x559929b47f30 2024-05-27T07:25:05.368 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:05.373+0000 7f5ffea68b80 1 -- 172.21.15.52:0/3077080553 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f5fe40ad7a0 con 0x559929b47f30 2024-05-27T07:25:05.368 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:05.373+0000 7f5fd17fa700 1 -- 172.21.15.52:0/3077080553 <== mon.0 v2:172.21.15.52:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f5fd8039050 con 0x559929b47f30 2024-05-27T07:25:05.368 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:05.373+0000 7f5ffea68b80 10 client.15153 fetch_fsmap learned FSMap version 29 2024-05-27T07:25:05.368 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:05.373+0000 7f5ffea68b80 1 -- 172.21.15.52:0/3077080553 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x559929b5f070 con 0x559929b47f30 2024-05-27T07:25:05.368 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:05.373+0000 7f5fd17fa700 1 -- 172.21.15.52:0/3077080553 <== mon.0 v2:172.21.15.52:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f5fe40ad7a0 con 0x559929b47f30 2024-05-27T07:25:05.369 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:05.374+0000 7f5fd17fa700 1 -- 172.21.15.52:0/3077080553 <== mon.0 v2:172.21.15.52:3300/0 5 ==== fsmap(e 29) ==== 8362+0+0 (secure 0 0 0) 0x7f5fd802dc50 con 0x559929b47f30 2024-05-27T07:25:05.369 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:05.374+0000 7f5ffea68b80 10 client.15153 fetch_fsmap finished waiting for FSMap version 29 2024-05-27T07:25:05.369 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:05.374+0000 7f5ffea68b80 10 client.15153 resolve_mds: resolved name 'b' to daemon mds.b 2024-05-27T07:25:05.369 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:05.374+0000 7f5ffea68b80 20 client.15153 populate_metadata read hostname 'smithi052' 2024-05-27T07:25:05.369 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:05.374+0000 7f5ffea68b80 1 --2- 172.21.15.52:0/3077080553 >> [v2:172.21.15.105:6838/500348151,v1:172.21.15.105:6839/500348151] conn(0x559929b61640 0x559929b62ab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:25:05.369 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:05.374+0000 7f5ffea68b80 4 client.15153 mds_command: new command op to 24527 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-27T07:25:05.369 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:05.374+0000 7f5ffea68b80 1 -- 172.21.15.52:0/3077080553 --> [v2:172.21.15.105:6838/500348151,v1:172.21.15.105:6839/500348151] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x559929b139a0 con 0x559929b61640 2024-05-27T07:25:05.369 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:05.374+0000 7f5feddef700 1 --2- 172.21.15.52:0/3077080553 >> [v2:172.21.15.105:6838/500348151,v1:172.21.15.105:6839/500348151] conn(0x559929b61640 0x559929b62ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:25:05.369 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:05.374+0000 7f5fd17fa700 10 client.15153 ms_handle_connect on v2:172.21.15.105:6838/500348151 2024-05-27T07:25:05.369 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:05.374+0000 7f5feddef700 1 --2- 172.21.15.52:0/3077080553 >> [v2:172.21.15.105:6838/500348151,v1:172.21.15.105:6839/500348151] conn(0x559929b61640 0x559929b62ab0 crc :-1 s=READY pgs=28 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=333d9d8dd76fece8 server_cookie=111415ec37066d63 in_seq=0 out_seq=0 2024-05-27T07:25:05.369 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:05.375+0000 7f5fd17fa700 1 -- 172.21.15.52:0/3077080553 <== mds.0 v2:172.21.15.105:6838/500348151 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24041 (crc 0 0 0) 0x559929b139a0 con 0x559929b61640 2024-05-27T07:25:05.369 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:05.375+0000 7f5fd17fa700 10 client.15153 handle_command_reply: tid=0 2024-05-27T07:25:05.369 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:05.375+0000 7f5ffea68b80 1 client.15153 shutdown 2024-05-27T07:25:05.370 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:05.375+0000 7f5ffea68b80 1 -- 172.21.15.52:0/3077080553 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x559929b47f30 msgr2=0x559929b51870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:25:05.370 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:05.375+0000 7f5ffea68b80 1 --2- 172.21.15.52:0/3077080553 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x559929b47f30 0x559929b51870 secure :-1 s=READY pgs=336 cs=0 l=1 rev1=1 crypto rx=0x7f5fd80099c0 tx=0x7f5fd8009cd0 comp rx=0 tx=0).stop 2024-05-27T07:25:05.370 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:05.376+0000 7f5ffea68b80 1 -- 172.21.15.52:0/3077080553 shutdown_connections 2024-05-27T07:25:05.370 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:05.376+0000 7f5ffea68b80 1 --2- 172.21.15.52:0/3077080553 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x559929b4dc00 0x559929b51db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:25:05.370 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:05.376+0000 7f5ffea68b80 1 --2- 172.21.15.52:0/3077080553 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5599299c9df0 0x559929b51330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:25:05.370 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:05.376+0000 7f5ffea68b80 1 --2- 172.21.15.52:0/3077080553 >> [v2:172.21.15.105:6838/500348151,v1:172.21.15.105:6839/500348151] conn(0x559929b61640 0x559929b62ab0 crc :-1 s=READY pgs=28 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:25:05.370 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:05.376+0000 7f5ffea68b80 1 --2- 172.21.15.52:0/3077080553 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x559929b47f30 0x559929b51870 unknown :-1 s=CLOSED pgs=336 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:25:05.370 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:05.376+0000 7f5fd17fa700 0 client.15153 ms_handle_reset on v2:172.21.15.105:6838/500348151 2024-05-27T07:25:05.370 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:05.376+0000 7f5ffea68b80 1 -- 172.21.15.52:0/3077080553 >> 172.21.15.52:0/3077080553 conn(0x559929a24aa0 msgr2=0x559929b469e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:25:05.370 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:05.376+0000 7f5ffea68b80 1 -- 172.21.15.52:0/3077080553 shutdown_connections 2024-05-27T07:25:05.370 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:05.376+0000 7f5ffea68b80 1 -- 172.21.15.52:0/3077080553 wait complete. 2024-05-27T07:25:05.370 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:05.376+0000 7f5ffea68b80 20 client.15153 trim_cache size 0 max 16384 2024-05-27T07:25:05.388 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:05.397+0000 7f5ffea68b80 1 Processor -- start 2024-05-27T07:25:05.388 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:05.397+0000 7f5ffea68b80 1 -- start start 2024-05-27T07:25:05.388 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:05.398+0000 7f5ffea68b80 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5599299c9df0 0x559929b479f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:25:05.388 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:05.398+0000 7f5ffea68b80 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x559929b47f30 0x559929b5d990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:25:05.388 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:05.398+0000 7f5ffea68b80 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x559929b4dc00 0x559929b4afc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:25:05.389 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:05.398+0000 7f5ffea68b80 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x559929b60df0 con 0x559929b47f30 2024-05-27T07:25:05.389 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:05.398+0000 7f5ffea68b80 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x559929b614c0 con 0x5599299c9df0 2024-05-27T07:25:05.389 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:05.398+0000 7f5ffea68b80 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x559929b61b90 con 0x559929b4dc00 2024-05-27T07:25:05.389 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:05.398+0000 7f5fed5ee700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x559929b47f30 0x559929b5d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:25:05.389 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:05.398+0000 7f5fed5ee700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x559929b47f30 0x559929b5d990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.52:45214/0 (socket says 172.21.15.52:45214) 2024-05-27T07:25:05.389 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:05.398+0000 7f5fed5ee700 1 -- 172.21.15.52:0/2412296526 learned_addr learned my addr 172.21.15.52:0/2412296526 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-05-27T07:25:05.389 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:05.398+0000 7f5fee5f0700 1 --2- 172.21.15.52:0/2412296526 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x559929b4dc00 0x559929b4afc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:25:05.389 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:05.398+0000 7f5feddef700 1 --2- 172.21.15.52:0/2412296526 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5599299c9df0 0x559929b479f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:25:05.389 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:05.398+0000 7f5fed5ee700 1 -- 172.21.15.52:0/2412296526 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x559929b4dc00 msgr2=0x559929b4afc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:25:05.389 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:05.398+0000 7f5fed5ee700 1 --2- 172.21.15.52:0/2412296526 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x559929b4dc00 0x559929b4afc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:25:05.389 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:05.398+0000 7f5fed5ee700 1 -- 172.21.15.52:0/2412296526 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5599299c9df0 msgr2=0x559929b479f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:25:05.389 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:05.398+0000 7f5fed5ee700 1 --2- 172.21.15.52:0/2412296526 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5599299c9df0 0x559929b479f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:25:05.390 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:05.398+0000 7f5fed5ee700 1 -- 172.21.15.52:0/2412296526 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x559929b4f8a0 con 0x559929b47f30 2024-05-27T07:25:05.390 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:05.398+0000 7f5fee5f0700 1 --2- 172.21.15.52:0/2412296526 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x559929b4dc00 0x559929b4afc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-27T07:25:05.390 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:05.398+0000 7f5feddef700 1 --2- 172.21.15.52:0/2412296526 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5599299c9df0 0x559929b479f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-27T07:25:05.390 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:05.398+0000 7f5fed5ee700 1 --2- 172.21.15.52:0/2412296526 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x559929b47f30 0x559929b5d990 secure :-1 s=READY pgs=337 cs=0 l=1 rev1=1 crypto rx=0x7f5fd80184f0 tx=0x7f5fd8009310 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b7d1491f3aed0b59 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:25:05.390 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:05.398+0000 7f5fd37fe700 1 -- 172.21.15.52:0/2412296526 <== mon.0 v2:172.21.15.52:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5fd8041020 con 0x559929b47f30 2024-05-27T07:25:05.390 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:05.399+0000 7f5fd37fe700 1 -- 172.21.15.52:0/2412296526 <== mon.0 v2:172.21.15.52:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f5fd8018c10 con 0x559929b47f30 2024-05-27T07:25:05.390 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:05.399+0000 7f5fd37fe700 1 -- 172.21.15.52:0/2412296526 <== mon.0 v2:172.21.15.52:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5fd8032040 con 0x559929b47f30 2024-05-27T07:25:05.394 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:05.399+0000 7f5ffea68b80 1 -- 172.21.15.52:0/2412296526 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x559929b47f30 msgr2=0x559929b5d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:25:05.394 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:05.399+0000 7f5ffea68b80 1 --2- 172.21.15.52:0/2412296526 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x559929b47f30 0x559929b5d990 secure :-1 s=READY pgs=337 cs=0 l=1 rev1=1 crypto rx=0x7f5fd80184f0 tx=0x7f5fd8009310 comp rx=0 tx=0).stop 2024-05-27T07:25:05.394 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:05.399+0000 7f5ffea68b80 1 -- 172.21.15.52:0/2412296526 shutdown_connections 2024-05-27T07:25:05.394 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:05.399+0000 7f5ffea68b80 1 --2- 172.21.15.52:0/2412296526 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x559929b4dc00 0x559929b4afc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:25:05.394 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:05.399+0000 7f5ffea68b80 1 --2- 172.21.15.52:0/2412296526 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5599299c9df0 0x559929b479f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:25:05.394 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:05.399+0000 7f5ffea68b80 1 --2- 172.21.15.52:0/2412296526 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x559929b47f30 0x559929b5d990 unknown :-1 s=CLOSED pgs=337 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:25:05.395 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:05.399+0000 7f5ffea68b80 1 -- 172.21.15.52:0/2412296526 >> 172.21.15.52:0/2412296526 conn(0x559929a24aa0 msgr2=0x559929b4fc20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:25:05.395 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:05.399+0000 7f5ffea68b80 1 -- 172.21.15.52:0/2412296526 shutdown_connections 2024-05-27T07:25:05.395 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:05.399+0000 7f5ffea68b80 1 -- 172.21.15.52:0/2412296526 wait complete. 2024-05-27T07:25:05.395 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:05.400+0000 7f5ffea68b80 1 -- start start 2024-05-27T07:25:05.395 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:05.400+0000 7f5ffea68b80 1 Processor -- start 2024-05-27T07:25:05.395 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:05.400+0000 7f5ffea68b80 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5599299c9df0 0x559929b6bf30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:25:05.395 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:05.400+0000 7f5ffea68b80 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x559929b47f30 0x559929b6c470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:25:05.395 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:05.400+0000 7f5ffea68b80 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x559929b4dc00 0x559929b66ee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:25:05.395 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:05.400+0000 7f5ffea68b80 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x559929b62930 con 0x559929b47f30 2024-05-27T07:25:05.395 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:05.400+0000 7f5ffea68b80 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x559929b62260 con 0x559929b4dc00 2024-05-27T07:25:05.395 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:05.400+0000 7f5ffea68b80 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x559929b63000 con 0x5599299c9df0 2024-05-27T07:25:05.396 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:05.400+0000 7f5ffea68b80 10 client.0 _ll_register_callbacks cb 0x559929b51220 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T07:25:05.396 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:05.401+0000 7f5fed5ee700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x559929b47f30 0x559929b6c470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:25:05.396 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:05.401+0000 7f5fed5ee700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x559929b47f30 0x559929b6c470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.52:45222/0 (socket says 172.21.15.52:45222) 2024-05-27T07:25:05.396 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:05.401+0000 7f5fed5ee700 1 -- 172.21.15.52:0/530382739 learned_addr learned my addr 172.21.15.52:0/530382739 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-05-27T07:25:05.396 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:05.401+0000 7f5feddef700 1 --2- 172.21.15.52:0/530382739 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5599299c9df0 0x559929b6bf30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:25:05.396 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:05.401+0000 7f5fee5f0700 1 --2- 172.21.15.52:0/530382739 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x559929b4dc00 0x559929b66ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:25:05.396 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:05.401+0000 7f5fed5ee700 1 -- 172.21.15.52:0/530382739 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5599299c9df0 msgr2=0x559929b6bf30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:25:05.396 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:05.401+0000 7f5fed5ee700 1 --2- 172.21.15.52:0/530382739 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5599299c9df0 0x559929b6bf30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:25:05.396 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:05.401+0000 7f5fed5ee700 1 -- 172.21.15.52:0/530382739 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x559929b4dc00 msgr2=0x559929b66ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:25:05.396 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:05.401+0000 7f5fed5ee700 1 --2- 172.21.15.52:0/530382739 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x559929b4dc00 0x559929b66ee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:25:05.396 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:05.401+0000 7f5fed5ee700 1 -- 172.21.15.52:0/530382739 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({osdmap=0}) -- 0x559929b67420 con 0x559929b47f30 2024-05-27T07:25:05.396 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:05.401+0000 7f5fed5ee700 1 --2- 172.21.15.52:0/530382739 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x559929b47f30 0x559929b6c470 secure :-1 s=READY pgs=338 cs=0 l=1 rev1=1 crypto rx=0x7f5fd8018540 tx=0x7f5fd8005d50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=14ddfebaa1856877 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:25:05.397 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:05.401+0000 7f5fd17fa700 10 client.0 ms_handle_connect on v2:172.21.15.52:3300/0 2024-05-27T07:25:05.397 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:05.401+0000 7f5fd17fa700 1 -- 172.21.15.52:0/530382739 <== mon.0 v2:172.21.15.52:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5fd8041030 con 0x559929b47f30 2024-05-27T07:25:05.397 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:05.401+0000 7f5fd17fa700 1 -- 172.21.15.52:0/530382739 <== mon.0 v2:172.21.15.52:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9121+0+0 (secure 0 0 0) 0x7f5fd8022a20 con 0x559929b47f30 2024-05-27T07:25:05.397 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:05.401+0000 7f5ffea68b80 1 -- 172.21.15.52:0/530382739 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f5fd8018c10 con 0x559929b47f30 2024-05-27T07:25:05.397 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:05.402+0000 7f5fd17fa700 1 -- 172.21.15.52:0/530382739 <== mon.0 v2:172.21.15.52:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f5fd8055050 con 0x559929b47f30 2024-05-27T07:25:05.397 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:05.402+0000 7f5ffea68b80 10 client.15159 fetch_fsmap learned FSMap version 29 2024-05-27T07:25:05.397 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:05.402+0000 7f5ffea68b80 1 -- 172.21.15.52:0/530382739 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x559929bcd430 con 0x559929b47f30 2024-05-27T07:25:05.397 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:05.402+0000 7f5fd17fa700 1 -- 172.21.15.52:0/530382739 <== mon.0 v2:172.21.15.52:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f5fd8018c10 con 0x559929b47f30 2024-05-27T07:25:05.397 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:05.402+0000 7f5fd17fa700 1 -- 172.21.15.52:0/530382739 <== mon.0 v2:172.21.15.52:3300/0 5 ==== fsmap(e 29) ==== 8362+0+0 (secure 0 0 0) 0x7f5fd803d4c0 con 0x559929b47f30 2024-05-27T07:25:05.397 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:05.402+0000 7f5ffea68b80 10 client.15159 fetch_fsmap finished waiting for FSMap version 29 2024-05-27T07:25:05.397 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:05.402+0000 7f5ffea68b80 10 client.15159 resolve_mds: resolved name 'b' to daemon mds.b 2024-05-27T07:25:05.397 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:05.402+0000 7f5ffea68b80 20 client.15159 populate_metadata read hostname 'smithi052' 2024-05-27T07:25:05.397 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:05.402+0000 7f5ffea68b80 1 --2- 172.21.15.52:0/530382739 >> [v2:172.21.15.105:6838/500348151,v1:172.21.15.105:6839/500348151] conn(0x559929bcf4b0 0x559929bd0920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:25:05.398 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:05.402+0000 7f5ffea68b80 4 client.15159 mds_command: new command op to 24527 tid=0 multi_id=0 [{"prefix": "status"}] 2024-05-27T07:25:05.398 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:05.402+0000 7f5ffea68b80 1 -- 172.21.15.52:0/530382739 --> [v2:172.21.15.105:6838/500348151,v1:172.21.15.105:6839/500348151] -- command(tid 0: {"prefix": "status"}) -- 0x559929a78b90 con 0x559929bcf4b0 2024-05-27T07:25:05.398 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:05.402+0000 7f5feddef700 1 --2- 172.21.15.52:0/530382739 >> [v2:172.21.15.105:6838/500348151,v1:172.21.15.105:6839/500348151] conn(0x559929bcf4b0 0x559929bd0920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:25:05.398 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:05.403+0000 7f5fd17fa700 10 client.15159 ms_handle_connect on v2:172.21.15.105:6838/500348151 2024-05-27T07:25:05.398 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:05.403+0000 7f5feddef700 1 --2- 172.21.15.52:0/530382739 >> [v2:172.21.15.105:6838/500348151,v1:172.21.15.105:6839/500348151] conn(0x559929bcf4b0 0x559929bd0920 crc :-1 s=READY pgs=29 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=60e116a4afa864ab server_cookie=953cff665a55abc1 in_seq=0 out_seq=0 2024-05-27T07:25:05.398 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:05.403+0000 7f5fd17fa700 1 -- 172.21.15.52:0/530382739 <== mds.0 v2:172.21.15.105:6838/500348151 1 ==== command_reply(tid 0: 0 ) ==== 8+0+327 (crc 0 0 0) 0x559929a78b90 con 0x559929bcf4b0 2024-05-27T07:25:05.398 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:05.403+0000 7f5fd17fa700 10 client.15159 handle_command_reply: tid=0 2024-05-27T07:25:05.398 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:05.403+0000 7f5ffea68b80 1 client.15159 shutdown 2024-05-27T07:25:05.398 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:05.403+0000 7f5ffea68b80 1 -- 172.21.15.52:0/530382739 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x559929b47f30 msgr2=0x559929b6c470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:25:05.398 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:05.403+0000 7f5ffea68b80 1 --2- 172.21.15.52:0/530382739 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x559929b47f30 0x559929b6c470 secure :-1 s=READY pgs=338 cs=0 l=1 rev1=1 crypto rx=0x7f5fd8018540 tx=0x7f5fd8005d50 comp rx=0 tx=0).stop 2024-05-27T07:25:05.398 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:05.404+0000 7f5ffea68b80 1 -- 172.21.15.52:0/530382739 shutdown_connections 2024-05-27T07:25:05.399 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:05.404+0000 7f5ffea68b80 1 --2- 172.21.15.52:0/530382739 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x559929b4dc00 0x559929b66ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:25:05.399 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:05.404+0000 7f5ffea68b80 1 --2- 172.21.15.52:0/530382739 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5599299c9df0 0x559929b6bf30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:25:05.399 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:05.404+0000 7f5ffea68b80 1 --2- 172.21.15.52:0/530382739 >> [v2:172.21.15.105:6838/500348151,v1:172.21.15.105:6839/500348151] conn(0x559929bcf4b0 0x559929bd0920 crc :-1 s=READY pgs=29 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:25:05.399 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:05.404+0000 7f5ffea68b80 1 --2- 172.21.15.52:0/530382739 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x559929b47f30 0x559929b6c470 unknown :-1 s=CLOSED pgs=338 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:25:05.399 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:05.404+0000 7f5ffea68b80 1 -- 172.21.15.52:0/530382739 >> 172.21.15.52:0/530382739 conn(0x559929a24aa0 msgr2=0x559929b4afc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:25:05.399 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:05.404+0000 7f5fd17fa700 0 client.15159 ms_handle_reset on v2:172.21.15.105:6838/500348151 2024-05-27T07:25:05.399 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:05.404+0000 7f5ffea68b80 1 -- 172.21.15.52:0/530382739 shutdown_connections 2024-05-27T07:25:05.399 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:05.404+0000 7f5ffea68b80 1 -- 172.21.15.52:0/530382739 wait complete. 2024-05-27T07:25:05.399 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:05.404+0000 7f5ffea68b80 20 client.15159 trim_cache size 0 max 16384 2024-05-27T07:25:05.399 INFO:teuthology.orchestra.run.smithi052.stdout:{ 2024-05-27T07:25:05.400 INFO:teuthology.orchestra.run.smithi052.stdout: "cluster_fsid": "73df2636-1bf8-11ef-bc9b-c7b262605968", 2024-05-27T07:25:05.400 INFO:teuthology.orchestra.run.smithi052.stdout: "whoami": 0, 2024-05-27T07:25:05.400 INFO:teuthology.orchestra.run.smithi052.stdout: "id": 24527, 2024-05-27T07:25:05.400 INFO:teuthology.orchestra.run.smithi052.stdout: "want_state": "up:active", 2024-05-27T07:25:05.400 INFO:teuthology.orchestra.run.smithi052.stdout: "state": "up:active", 2024-05-27T07:25:05.400 INFO:teuthology.orchestra.run.smithi052.stdout: "fs_name": "cephfs", 2024-05-27T07:25:05.400 INFO:teuthology.orchestra.run.smithi052.stdout: "rank_uptime": 194.82912569300001, 2024-05-27T07:25:05.400 INFO:teuthology.orchestra.run.smithi052.stdout: "mdsmap_epoch": 29, 2024-05-27T07:25:05.400 INFO:teuthology.orchestra.run.smithi052.stdout: "osdmap_epoch": 90, 2024-05-27T07:25:05.400 INFO:teuthology.orchestra.run.smithi052.stdout: "osdmap_epoch_barrier": 88, 2024-05-27T07:25:05.400 INFO:teuthology.orchestra.run.smithi052.stdout: "uptime": 208.883318495 2024-05-27T07:25:05.400 INFO:teuthology.orchestra.run.smithi052.stdout:} 2024-05-27T07:25:05.401 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:05.404+0000 7f5ff0053700 1 -- 172.21.15.52:0/2278690597 >> [v2:172.21.15.52:6800/3513075805,v1:172.21.15.52:6801/3513075805] conn(0x7f5fd40b2900 msgr2=0x7f5fd40b4e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:25:05.401 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:05.404+0000 7f5ff0053700 1 --2- 172.21.15.52:0/2278690597 >> [v2:172.21.15.52:6800/3513075805,v1:172.21.15.52:6801/3513075805] conn(0x7f5fd40b2900 0x7f5fd40b4e10 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7f5fd8000c00 tx=0x7f5fd801a040 comp rx=0 tx=0).stop 2024-05-27T07:25:05.401 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:05.404+0000 7f5ff0053700 1 -- 172.21.15.52:0/2278690597 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f5fe81034b0 msgr2=0x7f5fe80ff2c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:25:05.401 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:05.404+0000 7f5ff0053700 1 --2- 172.21.15.52:0/2278690597 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f5fe81034b0 0x7f5fe80ff2c0 secure :-1 s=READY pgs=334 cs=0 l=1 rev1=1 crypto rx=0x7f5fe400c920 tx=0x7f5fe400cce0 comp rx=0 tx=0).stop 2024-05-27T07:25:05.402 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:05.404+0000 7f5ff0053700 1 -- 172.21.15.52:0/2278690597 shutdown_connections 2024-05-27T07:25:05.402 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:05.404+0000 7f5ff0053700 1 --2- 172.21.15.52:0/2278690597 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f5fe8108730 0x7f5fe80ffd40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:25:05.402 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:05.404+0000 7f5ff0053700 1 --2- 172.21.15.52:0/2278690597 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f5fe81034b0 0x7f5fe80ff2c0 unknown :-1 s=CLOSED pgs=334 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:25:05.402 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:05.404+0000 7f5ff0053700 1 --2- 172.21.15.52:0/2278690597 >> [v2:172.21.15.52:6800/3513075805,v1:172.21.15.52:6801/3513075805] conn(0x7f5fd40b2900 0x7f5fd40b4e10 unknown :-1 s=CLOSED pgs=211 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:25:05.402 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:05.404+0000 7f5ff0053700 1 --2- 172.21.15.52:0/2278690597 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f5fe8107de0 0x7f5fe80ff800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:25:05.402 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:05.404+0000 7f5ff0053700 1 -- 172.21.15.52:0/2278690597 >> 172.21.15.52:0/2278690597 conn(0x7f5fe80fede0 msgr2=0x7f5fe8075d20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:25:05.402 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:05.404+0000 7f5ff0053700 1 -- 172.21.15.52:0/2278690597 shutdown_connections 2024-05-27T07:25:05.402 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:05.405+0000 7f5ff0053700 1 -- 172.21.15.52:0/2278690597 wait complete. 2024-05-27T07:25:06.168 INFO:tasks.cephadm:Setup complete, yielding 2024-05-27T07:25:06.169 INFO:teuthology.run_tasks:Running task cephadm.shell... 2024-05-27T07:25:06.181 INFO:tasks.cephadm:Running commands on role mon.a host ubuntu@smithi052.front.sepia.ceph.com 2024-05-27T07:25:06.182 DEBUG:teuthology.orchestra.run.smithi052:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:38f9d900c83c63799fbdbe61acc9a11b0d3554a6 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 73df2636-1bf8-11ef-bc9b-c7b262605968 -- bash -c 'ceph orch status' 2024-05-27T07:25:06.432 INFO:teuthology.orchestra.run.smithi052.stderr:Inferring config /var/lib/ceph/73df2636-1bf8-11ef-bc9b-c7b262605968/mon.a/config 2024-05-27T07:25:06.825 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:25:06 smithi052 ceph-mon[31955]: pgmap v398: 129 pgs: 129 active+clean; 647 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 5.7 KiB/s rd, 11 op/s 2024-05-27T07:25:07.101 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:25:06 smithi173 ceph-mon[37000]: pgmap v398: 129 pgs: 129 active+clean; 647 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 5.7 KiB/s rd, 11 op/s 2024-05-27T07:25:07.190 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:25:06 smithi105 ceph-mon[35977]: pgmap v398: 129 pgs: 129 active+clean; 647 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 5.7 KiB/s rd, 11 op/s 2024-05-27T07:25:08.019 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:08.029+0000 7f53d0f90700 1 -- 172.21.15.52:0/1178523886 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f53cc1037c0 msgr2=0x7f53cc103bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:25:08.020 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:08.030+0000 7f53d0f90700 1 --2- 172.21.15.52:0/1178523886 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f53cc1037c0 0x7f53cc103bd0 secure :-1 s=READY pgs=339 cs=0 l=1 rev1=1 crypto rx=0x7f53bc009990 tx=0x7f53bc009ca0 comp rx=0 tx=0).stop 2024-05-27T07:25:08.020 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:08.030+0000 7f53d0f90700 1 -- 172.21.15.52:0/1178523886 shutdown_connections 2024-05-27T07:25:08.020 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:08.030+0000 7f53d0f90700 1 --2- 172.21.15.52:0/1178523886 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f53cc104d00 0x7f53cc10b790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:25:08.020 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:08.030+0000 7f53d0f90700 1 --2- 172.21.15.52:0/1178523886 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f53cc1037c0 0x7f53cc103bd0 unknown :-1 s=CLOSED pgs=339 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:25:08.020 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:08.030+0000 7f53d0f90700 1 --2- 172.21.15.52:0/1178523886 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f53cc1041a0 0x7f53cc104670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:25:08.020 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:08.030+0000 7f53d0f90700 1 -- 172.21.15.52:0/1178523886 >> 172.21.15.52:0/1178523886 conn(0x7f53cc0fe010 msgr2=0x7f53cc1004c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:25:08.020 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:08.030+0000 7f53d0f90700 1 -- 172.21.15.52:0/1178523886 shutdown_connections 2024-05-27T07:25:08.020 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:08.030+0000 7f53d0f90700 1 -- 172.21.15.52:0/1178523886 wait complete. 2024-05-27T07:25:08.022 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:08.031+0000 7f53d0f90700 1 Processor -- start 2024-05-27T07:25:08.022 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:08.031+0000 7f53d0f90700 1 -- start start 2024-05-27T07:25:08.022 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:08.031+0000 7f53d0f90700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f53cc1037c0 0x7f53cc19a990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:25:08.022 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:08.031+0000 7f53d0f90700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f53cc1041a0 0x7f53cc19aed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:25:08.022 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:08.031+0000 7f53d0f90700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f53cc104d00 0x7f53cc1a1e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:25:08.022 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:08.031+0000 7f53d0f90700 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x7f53cc10ed10 con 0x7f53cc1037c0 2024-05-27T07:25:08.022 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:08.031+0000 7f53d0f90700 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f53cc10eb90 con 0x7f53cc1041a0 2024-05-27T07:25:08.023 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:08.031+0000 7f53d0f90700 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f53cc10ee90 con 0x7f53cc104d00 2024-05-27T07:25:08.023 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:08.031+0000 7f53ca59c700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f53cc1037c0 0x7f53cc19a990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:25:08.023 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:08.031+0000 7f53ca59c700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f53cc1037c0 0x7f53cc19a990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.52:45238/0 (socket says 172.21.15.52:45238) 2024-05-27T07:25:08.023 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:08.031+0000 7f53ca59c700 1 -- 172.21.15.52:0/133016593 learned_addr learned my addr 172.21.15.52:0/133016593 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-05-27T07:25:08.023 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:08.031+0000 7f53c9d9b700 1 --2- 172.21.15.52:0/133016593 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f53cc1041a0 0x7f53cc19aed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:25:08.023 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:08.031+0000 7f53cad9d700 1 --2- 172.21.15.52:0/133016593 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f53cc104d00 0x7f53cc1a1e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:25:08.023 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:08.031+0000 7f53ca59c700 1 -- 172.21.15.52:0/133016593 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f53cc104d00 msgr2=0x7f53cc1a1e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:25:08.023 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:08.031+0000 7f53ca59c700 1 --2- 172.21.15.52:0/133016593 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f53cc104d00 0x7f53cc1a1e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:25:08.023 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:08.031+0000 7f53ca59c700 1 -- 172.21.15.52:0/133016593 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f53cc1041a0 msgr2=0x7f53cc19aed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:25:08.023 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:08.031+0000 7f53ca59c700 1 --2- 172.21.15.52:0/133016593 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f53cc1041a0 0x7f53cc19aed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:25:08.023 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:08.031+0000 7f53ca59c700 1 -- 172.21.15.52:0/133016593 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f53cc1a24f0 con 0x7f53cc1037c0 2024-05-27T07:25:08.023 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:08.031+0000 7f53cad9d700 1 --2- 172.21.15.52:0/133016593 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f53cc104d00 0x7f53cc1a1e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-27T07:25:08.023 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:08.032+0000 7f53ca59c700 1 --2- 172.21.15.52:0/133016593 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f53cc1037c0 0x7f53cc19a990 secure :-1 s=READY pgs=340 cs=0 l=1 rev1=1 crypto rx=0x7f53bc003800 tx=0x7f53bc0047b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:25:08.024 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:08.032+0000 7f53bb7fe700 1 -- 172.21.15.52:0/133016593 <== mon.0 v2:172.21.15.52:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f53bc027070 con 0x7f53cc1037c0 2024-05-27T07:25:08.024 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:08.032+0000 7f53d0f90700 1 -- 172.21.15.52:0/133016593 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f53cc1a2780 con 0x7f53cc1037c0 2024-05-27T07:25:08.024 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:08.032+0000 7f53bb7fe700 1 -- 172.21.15.52:0/133016593 <== mon.0 v2:172.21.15.52:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f53bc022470 con 0x7f53cc1037c0 2024-05-27T07:25:08.024 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:08.032+0000 7f53bb7fe700 1 -- 172.21.15.52:0/133016593 <== mon.0 v2:172.21.15.52:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f53bc00f6e0 con 0x7f53cc1037c0 2024-05-27T07:25:08.024 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:08.032+0000 7f53d0f90700 1 -- 172.21.15.52:0/133016593 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f53cc1a2cb0 con 0x7f53cc1037c0 2024-05-27T07:25:08.024 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:08.033+0000 7f53d0f90700 1 -- 172.21.15.52:0/133016593 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f53cc0512b0 con 0x7f53cc1037c0 2024-05-27T07:25:08.024 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:08.034+0000 7f53bb7fe700 1 -- 172.21.15.52:0/133016593 <== mon.0 v2:172.21.15.52:3300/0 4 ==== mgrmap(e 18) ==== 149652+0+0 (secure 0 0 0) 0x7f53bc02a070 con 0x7f53cc1037c0 2024-05-27T07:25:08.025 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:08.035+0000 7f53bb7fe700 1 --2- 172.21.15.52:0/133016593 >> [v2:172.21.15.52:6800/3513075805,v1:172.21.15.52:6801/3513075805] conn(0x7f53b40b2880 0x7f53b40b4d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:25:08.025 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:08.035+0000 7f53c9d9b700 1 --2- 172.21.15.52:0/133016593 >> [v2:172.21.15.52:6800/3513075805,v1:172.21.15.52:6801/3513075805] conn(0x7f53b40b2880 0x7f53b40b4d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:25:08.025 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:08.035+0000 7f53bb7fe700 1 -- 172.21.15.52:0/133016593 <== mon.0 v2:172.21.15.52:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9121+0+0 (secure 0 0 0) 0x7f53bc0e3710 con 0x7f53cc1037c0 2024-05-27T07:25:08.025 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:08.036+0000 7f53c9d9b700 1 --2- 172.21.15.52:0/133016593 >> [v2:172.21.15.52:6800/3513075805,v1:172.21.15.52:6801/3513075805] conn(0x7f53b40b2880 0x7f53b40b4d90 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7f53cc19ba40 tx=0x7f53c0009190 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:25:08.030 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:08.040+0000 7f53bb7fe700 1 -- 172.21.15.52:0/133016593 <== mon.0 v2:172.21.15.52:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191963 (secure 0 0 0) 0x7f53bc013070 con 0x7f53cc1037c0 2024-05-27T07:25:08.272 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:08.282+0000 7f53d0f90700 1 -- 172.21.15.52:0/133016593 --> [v2:172.21.15.52:6800/3513075805,v1:172.21.15.52:6801/3513075805] -- mgr_command(tid 0: {"prefix": "orch status", "target": ["mon-mgr", ""]}) -- 0x7f53cc1a3090 con 0x7f53b40b2880 2024-05-27T07:25:08.272 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:08.283+0000 7f53bb7fe700 1 -- 172.21.15.52:0/133016593 <== mgr.14152 v2:172.21.15.52:6800/3513075805 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+43 (secure 0 0 0) 0x7f53cc1a3090 con 0x7f53b40b2880 2024-05-27T07:25:08.273 INFO:teuthology.orchestra.run.smithi052.stdout:Backend: cephadm 2024-05-27T07:25:08.273 INFO:teuthology.orchestra.run.smithi052.stdout:Available: Yes 2024-05-27T07:25:08.273 INFO:teuthology.orchestra.run.smithi052.stdout:Paused: No 2024-05-27T07:25:08.275 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:08.285+0000 7f53d0f90700 1 -- 172.21.15.52:0/133016593 >> [v2:172.21.15.52:6800/3513075805,v1:172.21.15.52:6801/3513075805] conn(0x7f53b40b2880 msgr2=0x7f53b40b4d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:25:08.275 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:08.285+0000 7f53d0f90700 1 --2- 172.21.15.52:0/133016593 >> [v2:172.21.15.52:6800/3513075805,v1:172.21.15.52:6801/3513075805] conn(0x7f53b40b2880 0x7f53b40b4d90 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7f53cc19ba40 tx=0x7f53c0009190 comp rx=0 tx=0).stop 2024-05-27T07:25:08.275 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:08.285+0000 7f53d0f90700 1 -- 172.21.15.52:0/133016593 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f53cc1037c0 msgr2=0x7f53cc19a990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:25:08.275 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:08.285+0000 7f53d0f90700 1 --2- 172.21.15.52:0/133016593 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f53cc1037c0 0x7f53cc19a990 secure :-1 s=READY pgs=340 cs=0 l=1 rev1=1 crypto rx=0x7f53bc003800 tx=0x7f53bc0047b0 comp rx=0 tx=0).stop 2024-05-27T07:25:08.275 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:08.285+0000 7f53d0f90700 1 -- 172.21.15.52:0/133016593 shutdown_connections 2024-05-27T07:25:08.276 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:08.285+0000 7f53d0f90700 1 --2- 172.21.15.52:0/133016593 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f53cc104d00 0x7f53cc1a1e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:25:08.276 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:08.285+0000 7f53d0f90700 1 --2- 172.21.15.52:0/133016593 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f53cc1037c0 0x7f53cc19a990 unknown :-1 s=CLOSED pgs=340 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:25:08.276 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:08.285+0000 7f53d0f90700 1 --2- 172.21.15.52:0/133016593 >> [v2:172.21.15.52:6800/3513075805,v1:172.21.15.52:6801/3513075805] conn(0x7f53b40b2880 0x7f53b40b4d90 unknown :-1 s=CLOSED pgs=212 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:25:08.276 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:08.285+0000 7f53d0f90700 1 --2- 172.21.15.52:0/133016593 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f53cc1041a0 0x7f53cc19aed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:25:08.276 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:08.285+0000 7f53d0f90700 1 -- 172.21.15.52:0/133016593 >> 172.21.15.52:0/133016593 conn(0x7f53cc0fe010 msgr2=0x7f53cc0ff9b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:25:08.276 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:08.285+0000 7f53d0f90700 1 -- 172.21.15.52:0/133016593 shutdown_connections 2024-05-27T07:25:08.276 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:08.285+0000 7f53d0f90700 1 -- 172.21.15.52:0/133016593 wait complete. 2024-05-27T07:25:08.873 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:25:08 smithi052 ceph-mon[31955]: pgmap v399: 129 pgs: 129 active+clean; 647 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 9 op/s 2024-05-27T07:25:08.904 DEBUG:teuthology.orchestra.run.smithi052:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:38f9d900c83c63799fbdbe61acc9a11b0d3554a6 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 73df2636-1bf8-11ef-bc9b-c7b262605968 -- bash -c 'ceph orch ps' 2024-05-27T07:25:09.102 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:25:08 smithi173 ceph-mon[37000]: pgmap v399: 129 pgs: 129 active+clean; 647 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 9 op/s 2024-05-27T07:25:09.156 INFO:teuthology.orchestra.run.smithi052.stderr:Inferring config /var/lib/ceph/73df2636-1bf8-11ef-bc9b-c7b262605968/mon.a/config 2024-05-27T07:25:09.190 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:25:08 smithi105 ceph-mon[35977]: pgmap v399: 129 pgs: 129 active+clean; 647 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 9 op/s 2024-05-27T07:25:09.979 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:25:09 smithi052 ceph-mon[31955]: from='client.15165 -' entity='client.admin' cmd=[{"prefix": "orch status", "target": ["mon-mgr", ""]}]: dispatch 2024-05-27T07:25:10.102 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:25:09 smithi173 ceph-mon[37000]: from='client.15165 -' entity='client.admin' cmd=[{"prefix": "orch status", "target": ["mon-mgr", ""]}]: dispatch 2024-05-27T07:25:10.190 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:25:09 smithi105 ceph-mon[35977]: from='client.15165 -' entity='client.admin' cmd=[{"prefix": "orch status", "target": ["mon-mgr", ""]}]: dispatch 2024-05-27T07:25:10.826 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:10.834+0000 7fe1f7503700 1 -- 172.21.15.52:0/2496817345 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fe1f0103f50 msgr2=0x7fe1f0104360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:25:10.826 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:10.834+0000 7fe1f7503700 1 --2- 172.21.15.52:0/2496817345 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fe1f0103f50 0x7fe1f0104360 secure :-1 s=READY pgs=341 cs=0 l=1 rev1=1 crypto rx=0x7fe1e0009990 tx=0x7fe1e0009ca0 comp rx=0 tx=0).stop 2024-05-27T07:25:10.826 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:10.834+0000 7fe1f7503700 1 -- 172.21.15.52:0/2496817345 shutdown_connections 2024-05-27T07:25:10.826 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:10.834+0000 7fe1f7503700 1 --2- 172.21.15.52:0/2496817345 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fe1f01049d0 0x7fe1f010c6e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:25:10.826 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:10.834+0000 7fe1f7503700 1 --2- 172.21.15.52:0/2496817345 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fe1f01059a0 0x7fe1f0105db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:25:10.826 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:10.834+0000 7fe1f7503700 1 --2- 172.21.15.52:0/2496817345 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fe1f0103f50 0x7fe1f0104360 unknown :-1 s=CLOSED pgs=341 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:25:10.826 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:10.834+0000 7fe1f7503700 1 -- 172.21.15.52:0/2496817345 >> 172.21.15.52:0/2496817345 conn(0x7fe1f00ff9c0 msgr2=0x7fe1f0101e50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:25:10.826 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:10.834+0000 7fe1f7503700 1 -- 172.21.15.52:0/2496817345 shutdown_connections 2024-05-27T07:25:10.826 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:10.834+0000 7fe1f7503700 1 -- 172.21.15.52:0/2496817345 wait complete. 2024-05-27T07:25:10.826 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:10.835+0000 7fe1f7503700 1 Processor -- start 2024-05-27T07:25:10.827 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:10.835+0000 7fe1f7503700 1 -- start start 2024-05-27T07:25:10.827 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:10.835+0000 7fe1f7503700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fe1f0103f50 0x7fe1f019aa50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:25:10.827 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:10.835+0000 7fe1f7503700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fe1f01049d0 0x7fe1f019af90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:25:10.827 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:10.835+0000 7fe1f7503700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fe1f01059a0 0x7fe1f01a1f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:25:10.827 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:10.835+0000 7fe1f7503700 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x7fe1f010ff30 con 0x7fe1f0103f50 2024-05-27T07:25:10.827 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:10.835+0000 7fe1f7503700 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7fe1f010fdb0 con 0x7fe1f01049d0 2024-05-27T07:25:10.827 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:10.835+0000 7fe1f7503700 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7fe1f01100b0 con 0x7fe1f01059a0 2024-05-27T07:25:10.827 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:10.835+0000 7fe1f529f700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fe1f0103f50 0x7fe1f019aa50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:25:10.827 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:10.835+0000 7fe1f529f700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fe1f0103f50 0x7fe1f019aa50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.52:35044/0 (socket says 172.21.15.52:35044) 2024-05-27T07:25:10.827 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:10.835+0000 7fe1f529f700 1 -- 172.21.15.52:0/3977994260 learned_addr learned my addr 172.21.15.52:0/3977994260 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-05-27T07:25:10.827 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:10.835+0000 7fe1f4a9e700 1 --2- 172.21.15.52:0/3977994260 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fe1f01049d0 0x7fe1f019af90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:25:10.827 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:10.835+0000 7fe1f529f700 1 -- 172.21.15.52:0/3977994260 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fe1f01059a0 msgr2=0x7fe1f01a1f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:25:10.827 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:10.835+0000 7fe1f529f700 1 --2- 172.21.15.52:0/3977994260 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fe1f01059a0 0x7fe1f01a1f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:25:10.827 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:10.835+0000 7fe1f529f700 1 -- 172.21.15.52:0/3977994260 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fe1f01049d0 msgr2=0x7fe1f019af90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:25:10.828 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:10.835+0000 7fe1f529f700 1 --2- 172.21.15.52:0/3977994260 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fe1f01049d0 0x7fe1f019af90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:25:10.828 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:10.835+0000 7fe1f529f700 1 -- 172.21.15.52:0/3977994260 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe1f01a25b0 con 0x7fe1f0103f50 2024-05-27T07:25:10.828 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:10.836+0000 7fe1f529f700 1 --2- 172.21.15.52:0/3977994260 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fe1f0103f50 0x7fe1f019aa50 secure :-1 s=READY pgs=342 cs=0 l=1 rev1=1 crypto rx=0x7fe1ec00d950 tx=0x7fe1ec00dd10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:25:10.828 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:10.836+0000 7fe1e67fc700 1 -- 172.21.15.52:0/3977994260 <== mon.0 v2:172.21.15.52:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe1ec017070 con 0x7fe1f0103f50 2024-05-27T07:25:10.828 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:10.836+0000 7fe1e67fc700 1 -- 172.21.15.52:0/3977994260 <== mon.0 v2:172.21.15.52:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fe1ec00de30 con 0x7fe1f0103f50 2024-05-27T07:25:10.828 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:10.836+0000 7fe1f7503700 1 -- 172.21.15.52:0/3977994260 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe1f01a28a0 con 0x7fe1f0103f50 2024-05-27T07:25:10.828 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:10.836+0000 7fe1e67fc700 1 -- 172.21.15.52:0/3977994260 <== mon.0 v2:172.21.15.52:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe1ec003b10 con 0x7fe1f0103f50 2024-05-27T07:25:10.828 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:10.836+0000 7fe1f7503700 1 -- 172.21.15.52:0/3977994260 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe1f01a2e30 con 0x7fe1f0103f50 2024-05-27T07:25:10.828 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:10.837+0000 7fe1f7503700 1 -- 172.21.15.52:0/3977994260 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe1f00512b0 con 0x7fe1f0103f50 2024-05-27T07:25:10.829 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:10.839+0000 7fe1e67fc700 1 -- 172.21.15.52:0/3977994260 <== mon.0 v2:172.21.15.52:3300/0 4 ==== mgrmap(e 18) ==== 149652+0+0 (secure 0 0 0) 0x7fe1ec003cb0 con 0x7fe1f0103f50 2024-05-27T07:25:10.829 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:10.839+0000 7fe1e67fc700 1 --2- 172.21.15.52:0/3977994260 >> [v2:172.21.15.52:6800/3513075805,v1:172.21.15.52:6801/3513075805] conn(0x7fe1dc0b2900 0x7fe1dc0b4e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:25:10.829 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:10.839+0000 7fe1f4a9e700 1 --2- 172.21.15.52:0/3977994260 >> [v2:172.21.15.52:6800/3513075805,v1:172.21.15.52:6801/3513075805] conn(0x7fe1dc0b2900 0x7fe1dc0b4e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:25:10.830 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:10.839+0000 7fe1e67fc700 1 -- 172.21.15.52:0/3977994260 <== mon.0 v2:172.21.15.52:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9121+0+0 (secure 0 0 0) 0x7fe1ec0e2940 con 0x7fe1f0103f50 2024-05-27T07:25:10.830 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:10.840+0000 7fe1f4a9e700 1 --2- 172.21.15.52:0/3977994260 >> [v2:172.21.15.52:6800/3513075805,v1:172.21.15.52:6801/3513075805] conn(0x7fe1dc0b2900 0x7fe1dc0b4e10 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7fe1e00038a0 tx=0x7fe1e001a040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:25:10.832 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:10.843+0000 7fe1e67fc700 1 -- 172.21.15.52:0/3977994260 <== mon.0 v2:172.21.15.52:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191963 (secure 0 0 0) 0x7fe1ec020030 con 0x7fe1f0103f50 2024-05-27T07:25:10.979 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:25:10 smithi052 ceph-mon[31955]: pgmap v400: 129 pgs: 129 active+clean; 647 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 5.3 KiB/s rd, 9 op/s 2024-05-27T07:25:11.060 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:11.070+0000 7fe1f7503700 1 -- 172.21.15.52:0/3977994260 --> [v2:172.21.15.52:6800/3513075805,v1:172.21.15.52:6801/3513075805] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) -- 0x7fe1f0063bf0 con 0x7fe1dc0b2900 2024-05-27T07:25:11.074 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:11.084+0000 7fe1e67fc700 1 -- 172.21.15.52:0/3977994260 <== mgr.14152 v2:172.21.15.52:6800/3513075805 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+4123 (secure 0 0 0) 0x7fe1f0063bf0 con 0x7fe1dc0b2900 2024-05-27T07:25:11.074 INFO:teuthology.orchestra.run.smithi052.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-05-27T07:25:11.074 INFO:teuthology.orchestra.run.smithi052.stdout:mds.a smithi052 running (3m) 39s ago 3m 85.3M - 19.0.0-3872-g38f9d900 e36ea4fc8a58 eda21eb53157 2024-05-27T07:25:11.074 INFO:teuthology.orchestra.run.smithi052.stdout:mds.b smithi105 running (3m) 79s ago 3m 96.6M - 19.0.0-3872-g38f9d900 e36ea4fc8a58 94f6b4e4fe56 2024-05-27T07:25:11.074 INFO:teuthology.orchestra.run.smithi052.stdout:mds.c smithi173 running (3m) 79s ago 3m 84.6M - 19.0.0-3872-g38f9d900 e36ea4fc8a58 2c67bd7fd298 2024-05-27T07:25:11.074 INFO:teuthology.orchestra.run.smithi052.stdout:mds.d smithi052 running (3m) 39s ago 3m 81.2M - 19.0.0-3872-g38f9d900 e36ea4fc8a58 4e3bdcf3433d 2024-05-27T07:25:11.075 INFO:teuthology.orchestra.run.smithi052.stdout:mds.e smithi105 running (3m) 79s ago 3m 85.5M - 19.0.0-3872-g38f9d900 e36ea4fc8a58 04d00bde09b4 2024-05-27T07:25:11.075 INFO:teuthology.orchestra.run.smithi052.stdout:mds.f smithi173 running (3m) 79s ago 3m 86.0M - 19.0.0-3872-g38f9d900 e36ea4fc8a58 e053aecb9864 2024-05-27T07:25:11.075 INFO:teuthology.orchestra.run.smithi052.stdout:mds.g smithi052 running (3m) 39s ago 3m 85.4M - 19.0.0-3872-g38f9d900 e36ea4fc8a58 e7a40afa0af4 2024-05-27T07:25:11.075 INFO:teuthology.orchestra.run.smithi052.stdout:mds.h smithi105 running (3m) 79s ago 3m 85.6M - 19.0.0-3872-g38f9d900 e36ea4fc8a58 15a481459439 2024-05-27T07:25:11.075 INFO:teuthology.orchestra.run.smithi052.stdout:mds.i smithi173 running (3m) 79s ago 3m 85.6M - 19.0.0-3872-g38f9d900 e36ea4fc8a58 6c1bde6811ef 2024-05-27T07:25:11.075 INFO:teuthology.orchestra.run.smithi052.stdout:mds.j smithi052 running (4m) 39s ago 4m 83.5M - 19.0.0-3872-g38f9d900 e36ea4fc8a58 107b05d83164 2024-05-27T07:25:11.075 INFO:teuthology.orchestra.run.smithi052.stdout:mds.k smithi105 running (4m) 79s ago 4m 84.6M - 19.0.0-3872-g38f9d900 e36ea4fc8a58 5d00409929d1 2024-05-27T07:25:11.075 INFO:teuthology.orchestra.run.smithi052.stdout:mds.l smithi173 running (4m) 79s ago 4m 87.2M - 19.0.0-3872-g38f9d900 e36ea4fc8a58 4752e4a243d5 2024-05-27T07:25:11.075 INFO:teuthology.orchestra.run.smithi052.stdout:mgr.x smithi052 *:9283,8765 running (11m) 39s ago 11m 599M - 19.0.0-3872-g38f9d900 e36ea4fc8a58 706e40386396 2024-05-27T07:25:11.075 INFO:teuthology.orchestra.run.smithi052.stdout:mgr.y smithi105 *:8443,8765 running (9m) 79s ago 9m 534M - 19.0.0-3872-g38f9d900 e36ea4fc8a58 09be672cee56 2024-05-27T07:25:11.075 INFO:teuthology.orchestra.run.smithi052.stdout:mgr.z smithi173 *:8443,8765 running (9m) 79s ago 9m 534M - 19.0.0-3872-g38f9d900 e36ea4fc8a58 db6f11578c16 2024-05-27T07:25:11.075 INFO:teuthology.orchestra.run.smithi052.stdout:mon.a smithi052 running (11m) 39s ago 11m 118M 2048M 19.0.0-3872-g38f9d900 e36ea4fc8a58 2dc34a76e3c2 2024-05-27T07:25:11.076 INFO:teuthology.orchestra.run.smithi052.stdout:mon.b smithi105 running (9m) 79s ago 9m 109M 2048M 19.0.0-3872-g38f9d900 e36ea4fc8a58 ac140bd84408 2024-05-27T07:25:11.076 INFO:teuthology.orchestra.run.smithi052.stdout:mon.c smithi173 running (9m) 79s ago 9m 107M 2048M 19.0.0-3872-g38f9d900 e36ea4fc8a58 de9961ba1521 2024-05-27T07:25:11.076 INFO:teuthology.orchestra.run.smithi052.stdout:osd.0 smithi052 running (9m) 39s ago 9m 112M 4273M 19.0.0-3872-g38f9d900 e36ea4fc8a58 c5b7c1548202 2024-05-27T07:25:11.076 INFO:teuthology.orchestra.run.smithi052.stdout:osd.1 smithi105 running (8m) 79s ago 8m 127M 4272M 19.0.0-3872-g38f9d900 e36ea4fc8a58 1f9c3cf032b5 2024-05-27T07:25:11.076 INFO:teuthology.orchestra.run.smithi052.stdout:osd.2 smithi173 running (8m) 79s ago 8m 126M 4272M 19.0.0-3872-g38f9d900 e36ea4fc8a58 d7fbbe1e5b24 2024-05-27T07:25:11.076 INFO:teuthology.orchestra.run.smithi052.stdout:osd.3 smithi052 running (7m) 39s ago 7m 125M 4273M 19.0.0-3872-g38f9d900 e36ea4fc8a58 04dfdd33a347 2024-05-27T07:25:11.076 INFO:teuthology.orchestra.run.smithi052.stdout:osd.4 smithi105 running (7m) 79s ago 7m 111M 4272M 19.0.0-3872-g38f9d900 e36ea4fc8a58 69b940abbf36 2024-05-27T07:25:11.076 INFO:teuthology.orchestra.run.smithi052.stdout:osd.5 smithi173 running (7m) 79s ago 7m 117M 4272M 19.0.0-3872-g38f9d900 e36ea4fc8a58 1e1290f6e23a 2024-05-27T07:25:11.076 INFO:teuthology.orchestra.run.smithi052.stdout:osd.6 smithi052 running (6m) 39s ago 6m 110M 4273M 19.0.0-3872-g38f9d900 e36ea4fc8a58 23fe920dff3b 2024-05-27T07:25:11.076 INFO:teuthology.orchestra.run.smithi052.stdout:osd.7 smithi105 running (6m) 79s ago 6m 124M 4272M 19.0.0-3872-g38f9d900 e36ea4fc8a58 ae6d7314f1dc 2024-05-27T07:25:11.076 INFO:teuthology.orchestra.run.smithi052.stdout:osd.8 smithi173 running (5m) 79s ago 5m 111M 4272M 19.0.0-3872-g38f9d900 e36ea4fc8a58 a2a560a11b93 2024-05-27T07:25:11.076 INFO:teuthology.orchestra.run.smithi052.stdout:osd.9 smithi052 running (5m) 39s ago 5m 120M 4273M 19.0.0-3872-g38f9d900 e36ea4fc8a58 ac0d99cb0d4c 2024-05-27T07:25:11.077 INFO:teuthology.orchestra.run.smithi052.stdout:osd.10 smithi105 running (4m) 79s ago 4m 112M 4272M 19.0.0-3872-g38f9d900 e36ea4fc8a58 6f53fe5b4826 2024-05-27T07:25:11.077 INFO:teuthology.orchestra.run.smithi052.stdout:osd.11 smithi173 running (4m) 79s ago 4m 123M 4272M 19.0.0-3872-g38f9d900 e36ea4fc8a58 559955021bb4 2024-05-27T07:25:11.077 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:11.086+0000 7fe1f7503700 1 -- 172.21.15.52:0/3977994260 >> [v2:172.21.15.52:6800/3513075805,v1:172.21.15.52:6801/3513075805] conn(0x7fe1dc0b2900 msgr2=0x7fe1dc0b4e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:25:11.078 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:11.086+0000 7fe1f7503700 1 --2- 172.21.15.52:0/3977994260 >> [v2:172.21.15.52:6800/3513075805,v1:172.21.15.52:6801/3513075805] conn(0x7fe1dc0b2900 0x7fe1dc0b4e10 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7fe1e00038a0 tx=0x7fe1e001a040 comp rx=0 tx=0).stop 2024-05-27T07:25:11.078 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:11.086+0000 7fe1f7503700 1 -- 172.21.15.52:0/3977994260 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fe1f0103f50 msgr2=0x7fe1f019aa50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:25:11.078 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:11.086+0000 7fe1f7503700 1 --2- 172.21.15.52:0/3977994260 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fe1f0103f50 0x7fe1f019aa50 secure :-1 s=READY pgs=342 cs=0 l=1 rev1=1 crypto rx=0x7fe1ec00d950 tx=0x7fe1ec00dd10 comp rx=0 tx=0).stop 2024-05-27T07:25:11.078 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:11.086+0000 7fe1f7503700 1 -- 172.21.15.52:0/3977994260 shutdown_connections 2024-05-27T07:25:11.078 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:11.086+0000 7fe1f7503700 1 --2- 172.21.15.52:0/3977994260 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fe1f01059a0 0x7fe1f01a1f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:25:11.078 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:11.086+0000 7fe1f7503700 1 --2- 172.21.15.52:0/3977994260 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fe1f0103f50 0x7fe1f019aa50 unknown :-1 s=CLOSED pgs=342 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:25:11.078 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:11.086+0000 7fe1f7503700 1 --2- 172.21.15.52:0/3977994260 >> [v2:172.21.15.52:6800/3513075805,v1:172.21.15.52:6801/3513075805] conn(0x7fe1dc0b2900 0x7fe1dc0b4e10 unknown :-1 s=CLOSED pgs=213 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:25:11.078 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:11.086+0000 7fe1f7503700 1 --2- 172.21.15.52:0/3977994260 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fe1f01049d0 0x7fe1f019af90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:25:11.078 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:11.086+0000 7fe1f7503700 1 -- 172.21.15.52:0/3977994260 >> 172.21.15.52:0/3977994260 conn(0x7fe1f00ff9c0 msgr2=0x7fe1f01086c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:25:11.079 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:11.087+0000 7fe1f7503700 1 -- 172.21.15.52:0/3977994260 shutdown_connections 2024-05-27T07:25:11.079 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:11.087+0000 7fe1f7503700 1 -- 172.21.15.52:0/3977994260 wait complete. 2024-05-27T07:25:11.101 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:25:10 smithi173 ceph-mon[37000]: pgmap v400: 129 pgs: 129 active+clean; 647 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 5.3 KiB/s rd, 9 op/s 2024-05-27T07:25:11.189 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:25:10 smithi105 ceph-mon[35977]: pgmap v400: 129 pgs: 129 active+clean; 647 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 5.3 KiB/s rd, 9 op/s 2024-05-27T07:25:11.752 DEBUG:teuthology.orchestra.run.smithi052:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:38f9d900c83c63799fbdbe61acc9a11b0d3554a6 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 73df2636-1bf8-11ef-bc9b-c7b262605968 -- bash -c 'ceph orch ls' 2024-05-27T07:25:11.979 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:25:11 smithi052 ceph-mon[31955]: from='client.15171 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-05-27T07:25:12.009 INFO:teuthology.orchestra.run.smithi052.stderr:Inferring config /var/lib/ceph/73df2636-1bf8-11ef-bc9b-c7b262605968/mon.a/config 2024-05-27T07:25:12.101 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:25:11 smithi173 ceph-mon[37000]: from='client.15171 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-05-27T07:25:12.190 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:25:11 smithi105 ceph-mon[35977]: from='client.15171 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-05-27T07:25:13.101 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:25:12 smithi173 ceph-mon[37000]: pgmap v401: 129 pgs: 129 active+clean; 647 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 9 op/s 2024-05-27T07:25:13.109 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:25:12 smithi052 ceph-mon[31955]: pgmap v401: 129 pgs: 129 active+clean; 647 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 9 op/s 2024-05-27T07:25:13.189 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:25:12 smithi105 ceph-mon[35977]: pgmap v401: 129 pgs: 129 active+clean; 647 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 9 op/s 2024-05-27T07:25:13.681 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:13.691+0000 7f2399e1e700 1 -- 172.21.15.52:0/2249703225 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f2394103220 msgr2=0x7f2394103630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:25:13.682 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:13.691+0000 7f2399e1e700 1 --2- 172.21.15.52:0/2249703225 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f2394103220 0x7f2394103630 secure :-1 s=READY pgs=343 cs=0 l=1 rev1=1 crypto rx=0x7f2384009990 tx=0x7f2384009ca0 comp rx=0 tx=0).stop 2024-05-27T07:25:13.682 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:13.691+0000 7f2399e1e700 1 -- 172.21.15.52:0/2249703225 shutdown_connections 2024-05-27T07:25:13.682 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:13.691+0000 7f2399e1e700 1 --2- 172.21.15.52:0/2249703225 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f2394104cf0 0x7f239410b5a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:25:13.682 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:13.691+0000 7f2399e1e700 1 --2- 172.21.15.52:0/2249703225 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f2394103220 0x7f2394103630 unknown :-1 s=CLOSED pgs=343 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:25:13.682 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:13.691+0000 7f2399e1e700 1 --2- 172.21.15.52:0/2249703225 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f2394104350 0x7f2394104760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:25:13.682 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:13.691+0000 7f2399e1e700 1 -- 172.21.15.52:0/2249703225 >> 172.21.15.52:0/2249703225 conn(0x7f23940fe860 msgr2=0x7f2394100d10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:25:13.682 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:13.691+0000 7f2399e1e700 1 -- 172.21.15.52:0/2249703225 shutdown_connections 2024-05-27T07:25:13.682 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:13.691+0000 7f2399e1e700 1 -- 172.21.15.52:0/2249703225 wait complete. 2024-05-27T07:25:13.682 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:13.692+0000 7f2399e1e700 1 Processor -- start 2024-05-27T07:25:13.682 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:13.692+0000 7f2399e1e700 1 -- start start 2024-05-27T07:25:13.684 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:13.692+0000 7f2399e1e700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f2394103220 0x7f239419a950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:25:13.684 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:13.692+0000 7f2399e1e700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f2394104350 0x7f239419ae90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:25:13.684 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:13.692+0000 7f2399e1e700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f2394104cf0 0x7f23941a1e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:25:13.685 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:13.692+0000 7f2399e1e700 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x7f239410ec70 con 0x7f2394103220 2024-05-27T07:25:13.685 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:13.692+0000 7f2399e1e700 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f239410eaf0 con 0x7f2394104cf0 2024-05-27T07:25:13.685 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:13.692+0000 7f2399e1e700 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f239410edf0 con 0x7f2394104350 2024-05-27T07:25:13.685 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:13.692+0000 7f23937fe700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f2394103220 0x7f239419a950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:25:13.685 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:13.693+0000 7f23937fe700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f2394103220 0x7f239419a950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.52:35064/0 (socket says 172.21.15.52:35064) 2024-05-27T07:25:13.685 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:13.693+0000 7f23937fe700 1 -- 172.21.15.52:0/1938134843 learned_addr learned my addr 172.21.15.52:0/1938134843 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-05-27T07:25:13.685 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:13.693+0000 7f2392ffd700 1 --2- 172.21.15.52:0/1938134843 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f2394104350 0x7f239419ae90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:25:13.685 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:13.693+0000 7f2393fff700 1 --2- 172.21.15.52:0/1938134843 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f2394104cf0 0x7f23941a1e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:25:13.685 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:13.693+0000 7f23937fe700 1 -- 172.21.15.52:0/1938134843 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f2394104350 msgr2=0x7f239419ae90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:25:13.685 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:13.693+0000 7f23937fe700 1 --2- 172.21.15.52:0/1938134843 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f2394104350 0x7f239419ae90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:25:13.685 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:13.693+0000 7f23937fe700 1 -- 172.21.15.52:0/1938134843 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f2394104cf0 msgr2=0x7f23941a1e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:25:13.685 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:13.693+0000 7f23937fe700 1 --2- 172.21.15.52:0/1938134843 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f2394104cf0 0x7f23941a1e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:25:13.685 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:13.693+0000 7f23937fe700 1 -- 172.21.15.52:0/1938134843 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f23941a24b0 con 0x7f2394103220 2024-05-27T07:25:13.685 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:13.693+0000 7f23937fe700 1 --2- 172.21.15.52:0/1938134843 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f2394103220 0x7f239419a950 secure :-1 s=READY pgs=344 cs=0 l=1 rev1=1 crypto rx=0x7f2384006010 tx=0x7f23840057c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:25:13.686 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:13.693+0000 7f2390ff9700 1 -- 172.21.15.52:0/1938134843 <== mon.0 v2:172.21.15.52:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2384026070 con 0x7f2394103220 2024-05-27T07:25:13.686 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:13.693+0000 7f2390ff9700 1 -- 172.21.15.52:0/1938134843 <== mon.0 v2:172.21.15.52:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f2384021410 con 0x7f2394103220 2024-05-27T07:25:13.686 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:13.693+0000 7f2399e1e700 1 -- 172.21.15.52:0/1938134843 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f23941a2740 con 0x7f2394103220 2024-05-27T07:25:13.686 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:13.693+0000 7f2390ff9700 1 -- 172.21.15.52:0/1938134843 <== mon.0 v2:172.21.15.52:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2384017670 con 0x7f2394103220 2024-05-27T07:25:13.686 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:13.693+0000 7f2399e1e700 1 -- 172.21.15.52:0/1938134843 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f23941a2b90 con 0x7f2394103220 2024-05-27T07:25:13.686 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:13.694+0000 7f2399e1e700 1 -- 172.21.15.52:0/1938134843 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f23940512b0 con 0x7f2394103220 2024-05-27T07:25:13.687 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:13.696+0000 7f2390ff9700 1 -- 172.21.15.52:0/1938134843 <== mon.0 v2:172.21.15.52:3300/0 4 ==== mgrmap(e 18) ==== 149652+0+0 (secure 0 0 0) 0x7f2384029070 con 0x7f2394103220 2024-05-27T07:25:13.687 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:13.697+0000 7f2390ff9700 1 --2- 172.21.15.52:0/1938134843 >> [v2:172.21.15.52:6800/3513075805,v1:172.21.15.52:6801/3513075805] conn(0x7f237c0b2880 0x7f237c0b4d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:25:13.687 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:13.697+0000 7f2392ffd700 1 --2- 172.21.15.52:0/1938134843 >> [v2:172.21.15.52:6800/3513075805,v1:172.21.15.52:6801/3513075805] conn(0x7f237c0b2880 0x7f237c0b4d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:25:13.687 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:13.697+0000 7f2390ff9700 1 -- 172.21.15.52:0/1938134843 <== mon.0 v2:172.21.15.52:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9121+0+0 (secure 0 0 0) 0x7f23840e3630 con 0x7f2394103220 2024-05-27T07:25:13.687 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:13.697+0000 7f2392ffd700 1 --2- 172.21.15.52:0/1938134843 >> [v2:172.21.15.52:6800/3513075805,v1:172.21.15.52:6801/3513075805] conn(0x7f237c0b2880 0x7f237c0b4d90 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7f239419ba00 tx=0x7f2388009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:25:13.690 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:13.700+0000 7f2390ff9700 1 -- 172.21.15.52:0/1938134843 <== mon.0 v2:172.21.15.52:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191963 (secure 0 0 0) 0x7f2384013070 con 0x7f2394103220 2024-05-27T07:25:13.914 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:13.924+0000 7f2399e1e700 1 -- 172.21.15.52:0/1938134843 --> [v2:172.21.15.52:6800/3513075805,v1:172.21.15.52:6801/3513075805] -- mgr_command(tid 0: {"prefix": "orch ls", "target": ["mon-mgr", ""]}) -- 0x7f23941a2eb0 con 0x7f237c0b2880 2024-05-27T07:25:13.920 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:13.930+0000 7f2390ff9700 1 -- 172.21.15.52:0/1938134843 <== mgr.14152 v2:172.21.15.52:6800/3513075805 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+980 (secure 0 0 0) 0x7f23941a2eb0 con 0x7f237c0b2880 2024-05-27T07:25:13.920 INFO:teuthology.orchestra.run.smithi052.stdout:NAME PORTS RUNNING REFRESHED AGE PLACEMENT 2024-05-27T07:25:13.920 INFO:teuthology.orchestra.run.smithi052.stdout:mds.all 12/12 82s ago 4m smithi052=a;smithi052=d;smithi052=g;smithi052=j;smithi105=b;smithi105=e;smithi105=h;smithi105=k;smithi173=c;smithi173=f;smithi173=i;smithi173=l;count:12 2024-05-27T07:25:13.920 INFO:teuthology.orchestra.run.smithi052.stdout:mgr 3/3 82s ago 9m smithi052=x;smithi105=y;smithi173=z;count:3 2024-05-27T07:25:13.920 INFO:teuthology.orchestra.run.smithi052.stdout:mon 3/3 82s ago 10m smithi052:172.21.15.52=a;smithi105:172.21.15.105=b;smithi173:172.21.15.173=c;count:3 2024-05-27T07:25:13.921 INFO:teuthology.orchestra.run.smithi052.stdout:osd 12 82s ago - 2024-05-27T07:25:13.923 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:13.933+0000 7f2399e1e700 1 -- 172.21.15.52:0/1938134843 >> [v2:172.21.15.52:6800/3513075805,v1:172.21.15.52:6801/3513075805] conn(0x7f237c0b2880 msgr2=0x7f237c0b4d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:25:13.923 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:13.933+0000 7f2399e1e700 1 --2- 172.21.15.52:0/1938134843 >> [v2:172.21.15.52:6800/3513075805,v1:172.21.15.52:6801/3513075805] conn(0x7f237c0b2880 0x7f237c0b4d90 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7f239419ba00 tx=0x7f2388009260 comp rx=0 tx=0).stop 2024-05-27T07:25:13.923 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:13.933+0000 7f2399e1e700 1 -- 172.21.15.52:0/1938134843 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f2394103220 msgr2=0x7f239419a950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:25:13.923 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:13.933+0000 7f2399e1e700 1 --2- 172.21.15.52:0/1938134843 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f2394103220 0x7f239419a950 secure :-1 s=READY pgs=344 cs=0 l=1 rev1=1 crypto rx=0x7f2384006010 tx=0x7f23840057c0 comp rx=0 tx=0).stop 2024-05-27T07:25:13.923 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:13.933+0000 7f2399e1e700 1 -- 172.21.15.52:0/1938134843 shutdown_connections 2024-05-27T07:25:13.923 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:13.933+0000 7f2399e1e700 1 --2- 172.21.15.52:0/1938134843 >> [v2:172.21.15.52:6800/3513075805,v1:172.21.15.52:6801/3513075805] conn(0x7f237c0b2880 0x7f237c0b4d90 unknown :-1 s=CLOSED pgs=214 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:25:13.924 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:13.933+0000 7f2399e1e700 1 --2- 172.21.15.52:0/1938134843 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f2394104cf0 0x7f23941a1e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:25:13.924 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:13.933+0000 7f2399e1e700 1 --2- 172.21.15.52:0/1938134843 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f2394103220 0x7f239419a950 unknown :-1 s=CLOSED pgs=344 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:25:13.924 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:13.933+0000 7f2399e1e700 1 --2- 172.21.15.52:0/1938134843 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f2394104350 0x7f239419ae90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:25:13.924 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:13.933+0000 7f2399e1e700 1 -- 172.21.15.52:0/1938134843 >> 172.21.15.52:0/1938134843 conn(0x7f23940fe860 msgr2=0x7f2394100240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:25:13.924 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:13.933+0000 7f2399e1e700 1 -- 172.21.15.52:0/1938134843 shutdown_connections 2024-05-27T07:25:13.924 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:13.933+0000 7f2399e1e700 1 -- 172.21.15.52:0/1938134843 wait complete. 2024-05-27T07:25:14.719 DEBUG:teuthology.orchestra.run.smithi052:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:38f9d900c83c63799fbdbe61acc9a11b0d3554a6 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 73df2636-1bf8-11ef-bc9b-c7b262605968 -- bash -c 'ceph orch host ls' 2024-05-27T07:25:14.975 INFO:teuthology.orchestra.run.smithi052.stderr:Inferring config /var/lib/ceph/73df2636-1bf8-11ef-bc9b-c7b262605968/mon.a/config 2024-05-27T07:25:14.979 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:25:14 smithi052 ceph-mon[31955]: pgmap v402: 129 pgs: 129 active+clean; 647 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 9 op/s 2024-05-27T07:25:14.979 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:25:14 smithi052 ceph-mon[31955]: from='client.15177 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""]}]: dispatch 2024-05-27T07:25:15.101 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:25:14 smithi173 ceph-mon[37000]: pgmap v402: 129 pgs: 129 active+clean; 647 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 9 op/s 2024-05-27T07:25:15.102 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:25:14 smithi173 ceph-mon[37000]: from='client.15177 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""]}]: dispatch 2024-05-27T07:25:15.190 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:25:14 smithi105 ceph-mon[35977]: pgmap v402: 129 pgs: 129 active+clean; 647 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 9 op/s 2024-05-27T07:25:15.190 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:25:14 smithi105 ceph-mon[35977]: from='client.15177 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""]}]: dispatch 2024-05-27T07:25:16.634 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:16.644+0000 7f7e8909e700 1 -- 172.21.15.52:0/2873130602 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f7e84103f50 msgr2=0x7f7e84104360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:25:16.634 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:16.644+0000 7f7e8909e700 1 --2- 172.21.15.52:0/2873130602 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f7e84103f50 0x7f7e84104360 secure :-1 s=READY pgs=345 cs=0 l=1 rev1=1 crypto rx=0x7f7e78009990 tx=0x7f7e78009ca0 comp rx=0 tx=0).stop 2024-05-27T07:25:16.635 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:16.644+0000 7f7e8909e700 1 -- 172.21.15.52:0/2873130602 shutdown_connections 2024-05-27T07:25:16.635 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:16.644+0000 7f7e8909e700 1 --2- 172.21.15.52:0/2873130602 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f7e841049d0 0x7f7e8410c6e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:25:16.635 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:16.644+0000 7f7e8909e700 1 --2- 172.21.15.52:0/2873130602 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f7e841059a0 0x7f7e84105db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:25:16.635 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:16.644+0000 7f7e8909e700 1 --2- 172.21.15.52:0/2873130602 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f7e84103f50 0x7f7e84104360 unknown :-1 s=CLOSED pgs=345 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:25:16.635 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:16.644+0000 7f7e8909e700 1 -- 172.21.15.52:0/2873130602 >> 172.21.15.52:0/2873130602 conn(0x7f7e840ff9c0 msgr2=0x7f7e84101e50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:25:16.635 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:16.644+0000 7f7e8909e700 1 -- 172.21.15.52:0/2873130602 shutdown_connections 2024-05-27T07:25:16.635 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:16.644+0000 7f7e8909e700 1 -- 172.21.15.52:0/2873130602 wait complete. 2024-05-27T07:25:16.635 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:16.644+0000 7f7e8909e700 1 Processor -- start 2024-05-27T07:25:16.635 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:16.645+0000 7f7e8909e700 1 -- start start 2024-05-27T07:25:16.637 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:16.645+0000 7f7e8909e700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f7e84103f50 0x7f7e8419cc10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:25:16.637 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:16.645+0000 7f7e8909e700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f7e841049d0 0x7f7e8419d150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:25:16.637 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:16.645+0000 7f7e8909e700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f7e841059a0 0x7f7e841a40f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:25:16.637 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:16.645+0000 7f7e8909e700 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x7f7e8410fee0 con 0x7f7e84103f50 2024-05-27T07:25:16.637 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:16.645+0000 7f7e8909e700 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f7e8410fd60 con 0x7f7e841049d0 2024-05-27T07:25:16.637 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:16.645+0000 7f7e8909e700 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f7e84110060 con 0x7f7e841059a0 2024-05-27T07:25:16.637 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:16.645+0000 7f7e82d9d700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f7e84103f50 0x7f7e8419cc10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:25:16.637 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:16.645+0000 7f7e82d9d700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f7e84103f50 0x7f7e8419cc10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.52:35078/0 (socket says 172.21.15.52:35078) 2024-05-27T07:25:16.638 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:16.645+0000 7f7e82d9d700 1 -- 172.21.15.52:0/4125232846 learned_addr learned my addr 172.21.15.52:0/4125232846 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-05-27T07:25:16.638 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:16.645+0000 7f7e8359e700 1 --2- 172.21.15.52:0/4125232846 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f7e841059a0 0x7f7e841a40f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:25:16.638 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:16.645+0000 7f7e8259c700 1 --2- 172.21.15.52:0/4125232846 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f7e841049d0 0x7f7e8419d150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:25:16.638 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:16.646+0000 7f7e82d9d700 1 -- 172.21.15.52:0/4125232846 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f7e841059a0 msgr2=0x7f7e841a40f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:25:16.638 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:16.646+0000 7f7e82d9d700 1 --2- 172.21.15.52:0/4125232846 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f7e841059a0 0x7f7e841a40f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:25:16.638 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:16.646+0000 7f7e82d9d700 1 -- 172.21.15.52:0/4125232846 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f7e841049d0 msgr2=0x7f7e8419d150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:25:16.638 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:16.646+0000 7f7e82d9d700 1 --2- 172.21.15.52:0/4125232846 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f7e841049d0 0x7f7e8419d150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:25:16.638 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:16.646+0000 7f7e82d9d700 1 -- 172.21.15.52:0/4125232846 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7e841a4770 con 0x7f7e84103f50 2024-05-27T07:25:16.638 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:16.646+0000 7f7e8259c700 1 --2- 172.21.15.52:0/4125232846 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f7e841049d0 0x7f7e8419d150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-27T07:25:16.638 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:16.646+0000 7f7e82d9d700 1 --2- 172.21.15.52:0/4125232846 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f7e84103f50 0x7f7e8419cc10 secure :-1 s=READY pgs=346 cs=0 l=1 rev1=1 crypto rx=0x7f7e7400d950 tx=0x7f7e7400dc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:25:16.638 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:16.646+0000 7f7e6bfff700 1 -- 172.21.15.52:0/4125232846 <== mon.0 v2:172.21.15.52:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7e74017070 con 0x7f7e84103f50 2024-05-27T07:25:16.638 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:16.646+0000 7f7e8909e700 1 -- 172.21.15.52:0/4125232846 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7e841a4a60 con 0x7f7e84103f50 2024-05-27T07:25:16.639 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:16.646+0000 7f7e6bfff700 1 -- 172.21.15.52:0/4125232846 <== mon.0 v2:172.21.15.52:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f7e74009730 con 0x7f7e84103f50 2024-05-27T07:25:16.639 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:16.646+0000 7f7e6bfff700 1 -- 172.21.15.52:0/4125232846 <== mon.0 v2:172.21.15.52:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7e74003a00 con 0x7f7e84103f50 2024-05-27T07:25:16.639 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:16.646+0000 7f7e8909e700 1 -- 172.21.15.52:0/4125232846 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7e841a4ff0 con 0x7f7e84103f50 2024-05-27T07:25:16.639 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:16.647+0000 7f7e8909e700 1 -- 172.21.15.52:0/4125232846 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7e840512b0 con 0x7f7e84103f50 2024-05-27T07:25:16.640 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:16.650+0000 7f7e6bfff700 1 -- 172.21.15.52:0/4125232846 <== mon.0 v2:172.21.15.52:3300/0 4 ==== mgrmap(e 18) ==== 149652+0+0 (secure 0 0 0) 0x7f7e74003ba0 con 0x7f7e84103f50 2024-05-27T07:25:16.640 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:16.651+0000 7f7e6bfff700 1 --2- 172.21.15.52:0/4125232846 >> [v2:172.21.15.52:6800/3513075805,v1:172.21.15.52:6801/3513075805] conn(0x7f7e6c0b2880 0x7f7e6c0b4d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:25:16.640 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:16.651+0000 7f7e8259c700 1 --2- 172.21.15.52:0/4125232846 >> [v2:172.21.15.52:6800/3513075805,v1:172.21.15.52:6801/3513075805] conn(0x7f7e6c0b2880 0x7f7e6c0b4d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:25:16.640 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:16.651+0000 7f7e6bfff700 1 -- 172.21.15.52:0/4125232846 <== mon.0 v2:172.21.15.52:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9121+0+0 (secure 0 0 0) 0x7f7e740e2920 con 0x7f7e84103f50 2024-05-27T07:25:16.641 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:16.651+0000 7f7e8259c700 1 --2- 172.21.15.52:0/4125232846 >> [v2:172.21.15.52:6800/3513075805,v1:172.21.15.52:6801/3513075805] conn(0x7f7e6c0b2880 0x7f7e6c0b4d90 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7f7e780038a0 tx=0x7f7e7801a040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:25:16.644 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:16.654+0000 7f7e6bfff700 1 -- 172.21.15.52:0/4125232846 <== mon.0 v2:172.21.15.52:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191963 (secure 0 0 0) 0x7f7e74020030 con 0x7f7e84103f50 2024-05-27T07:25:16.869 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:16.879+0000 7f7e8909e700 1 -- 172.21.15.52:0/4125232846 --> [v2:172.21.15.52:6800/3513075805,v1:172.21.15.52:6801/3513075805] -- mgr_command(tid 0: {"prefix": "orch host ls", "target": ["mon-mgr", ""]}) -- 0x7f7e84078210 con 0x7f7e6c0b2880 2024-05-27T07:25:16.870 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:16.880+0000 7f7e6bfff700 1 -- 172.21.15.52:0/4125232846 <== mgr.14152 v2:172.21.15.52:6800/3513075805 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+191 (secure 0 0 0) 0x7f7e84078210 con 0x7f7e6c0b2880 2024-05-27T07:25:16.870 INFO:teuthology.orchestra.run.smithi052.stdout:HOST ADDR LABELS STATUS 2024-05-27T07:25:16.870 INFO:teuthology.orchestra.run.smithi052.stdout:smithi052 172.21.15.52 2024-05-27T07:25:16.870 INFO:teuthology.orchestra.run.smithi052.stdout:smithi105 172.21.15.105 2024-05-27T07:25:16.870 INFO:teuthology.orchestra.run.smithi052.stdout:smithi173 172.21.15.173 2024-05-27T07:25:16.870 INFO:teuthology.orchestra.run.smithi052.stdout:3 hosts in cluster 2024-05-27T07:25:16.872 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:16.882+0000 7f7e8909e700 1 -- 172.21.15.52:0/4125232846 >> [v2:172.21.15.52:6800/3513075805,v1:172.21.15.52:6801/3513075805] conn(0x7f7e6c0b2880 msgr2=0x7f7e6c0b4d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:25:16.873 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:16.882+0000 7f7e8909e700 1 --2- 172.21.15.52:0/4125232846 >> [v2:172.21.15.52:6800/3513075805,v1:172.21.15.52:6801/3513075805] conn(0x7f7e6c0b2880 0x7f7e6c0b4d90 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7f7e780038a0 tx=0x7f7e7801a040 comp rx=0 tx=0).stop 2024-05-27T07:25:16.873 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:16.883+0000 7f7e8909e700 1 -- 172.21.15.52:0/4125232846 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f7e84103f50 msgr2=0x7f7e8419cc10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:25:16.873 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:16.883+0000 7f7e8909e700 1 --2- 172.21.15.52:0/4125232846 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f7e84103f50 0x7f7e8419cc10 secure :-1 s=READY pgs=346 cs=0 l=1 rev1=1 crypto rx=0x7f7e7400d950 tx=0x7f7e7400dc60 comp rx=0 tx=0).stop 2024-05-27T07:25:16.873 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:16.883+0000 7f7e8909e700 1 -- 172.21.15.52:0/4125232846 shutdown_connections 2024-05-27T07:25:16.873 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:16.883+0000 7f7e8909e700 1 --2- 172.21.15.52:0/4125232846 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f7e841059a0 0x7f7e841a40f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:25:16.873 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:16.883+0000 7f7e8909e700 1 --2- 172.21.15.52:0/4125232846 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f7e84103f50 0x7f7e8419cc10 unknown :-1 s=CLOSED pgs=346 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:25:16.873 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:16.883+0000 7f7e8909e700 1 --2- 172.21.15.52:0/4125232846 >> [v2:172.21.15.52:6800/3513075805,v1:172.21.15.52:6801/3513075805] conn(0x7f7e6c0b2880 0x7f7e6c0b4d90 unknown :-1 s=CLOSED pgs=215 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:25:16.873 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:16.883+0000 7f7e8909e700 1 --2- 172.21.15.52:0/4125232846 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f7e841049d0 0x7f7e8419d150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:25:16.873 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:16.883+0000 7f7e8909e700 1 -- 172.21.15.52:0/4125232846 >> 172.21.15.52:0/4125232846 conn(0x7f7e840ff9c0 msgr2=0x7f7e84076af0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:25:16.873 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:16.883+0000 7f7e8909e700 1 -- 172.21.15.52:0/4125232846 shutdown_connections 2024-05-27T07:25:16.873 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:16.883+0000 7f7e8909e700 1 -- 172.21.15.52:0/4125232846 wait complete. 2024-05-27T07:25:17.102 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:25:16 smithi173 ceph-mon[37000]: pgmap v403: 129 pgs: 129 active+clean; 647 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 11 op/s 2024-05-27T07:25:17.185 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:25:16 smithi052 ceph-mon[31955]: pgmap v403: 129 pgs: 129 active+clean; 647 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 11 op/s 2024-05-27T07:25:17.189 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:25:16 smithi105 ceph-mon[35977]: pgmap v403: 129 pgs: 129 active+clean; 647 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 11 op/s 2024-05-27T07:25:17.519 DEBUG:teuthology.orchestra.run.smithi052:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:38f9d900c83c63799fbdbe61acc9a11b0d3554a6 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 73df2636-1bf8-11ef-bc9b-c7b262605968 -- bash -c 'ceph orch device ls' 2024-05-27T07:25:17.774 INFO:teuthology.orchestra.run.smithi052.stderr:Inferring config /var/lib/ceph/73df2636-1bf8-11ef-bc9b-c7b262605968/mon.a/config 2024-05-27T07:25:18.046 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:25:17 smithi052 ceph-mon[31955]: from='client.15183 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""]}]: dispatch 2024-05-27T07:25:18.101 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:25:17 smithi173 ceph-mon[37000]: from='client.15183 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""]}]: dispatch 2024-05-27T07:25:18.190 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:25:17 smithi105 ceph-mon[35977]: from='client.15183 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""]}]: dispatch 2024-05-27T07:25:19.083 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:25:18 smithi052 ceph-mon[31955]: pgmap v404: 129 pgs: 129 active+clean; 647 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-05-27T07:25:19.102 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:25:18 smithi173 ceph-mon[37000]: pgmap v404: 129 pgs: 129 active+clean; 647 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-05-27T07:25:19.190 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:25:18 smithi105 ceph-mon[35977]: pgmap v404: 129 pgs: 129 active+clean; 647 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-05-27T07:25:19.509 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:19.518+0000 7fa2c214e700 1 -- 172.21.15.52:0/1085557482 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fa2bc104370 msgr2=0x7fa2bc104780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:25:19.509 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:19.518+0000 7fa2c214e700 1 --2- 172.21.15.52:0/1085557482 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fa2bc104370 0x7fa2bc104780 secure :-1 s=READY pgs=347 cs=0 l=1 rev1=1 crypto rx=0x7fa2b0009990 tx=0x7fa2b0009ca0 comp rx=0 tx=0).stop 2024-05-27T07:25:19.509 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:19.518+0000 7fa2c214e700 1 -- 172.21.15.52:0/1085557482 shutdown_connections 2024-05-27T07:25:19.509 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:19.518+0000 7fa2c214e700 1 --2- 172.21.15.52:0/1085557482 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fa2bc104d10 0x7fa2bc10b5c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:25:19.509 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:19.518+0000 7fa2c214e700 1 --2- 172.21.15.52:0/1085557482 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fa2bc103240 0x7fa2bc103650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:25:19.509 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:19.518+0000 7fa2c214e700 1 --2- 172.21.15.52:0/1085557482 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fa2bc104370 0x7fa2bc104780 unknown :-1 s=CLOSED pgs=347 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:25:19.509 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:19.518+0000 7fa2c214e700 1 -- 172.21.15.52:0/1085557482 >> 172.21.15.52:0/1085557482 conn(0x7fa2bc0fe880 msgr2=0x7fa2bc100d30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:25:19.509 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:19.518+0000 7fa2c214e700 1 -- 172.21.15.52:0/1085557482 shutdown_connections 2024-05-27T07:25:19.509 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:19.518+0000 7fa2c214e700 1 -- 172.21.15.52:0/1085557482 wait complete. 2024-05-27T07:25:19.510 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:19.519+0000 7fa2c214e700 1 Processor -- start 2024-05-27T07:25:19.510 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:19.519+0000 7fa2c214e700 1 -- start start 2024-05-27T07:25:19.511 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:19.519+0000 7fa2c214e700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fa2bc103240 0x7fa2bc0764a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:25:19.511 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:19.519+0000 7fa2c214e700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fa2bc104370 0x7fa2bc19f670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:25:19.512 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:19.519+0000 7fa2c214e700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fa2bc104d10 0x7fa2bc1a1c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:25:19.512 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:19.519+0000 7fa2c214e700 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x7fa2bc10eca0 con 0x7fa2bc104d10 2024-05-27T07:25:19.512 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:19.519+0000 7fa2c214e700 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7fa2bc10eb20 con 0x7fa2bc104370 2024-05-27T07:25:19.512 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:19.519+0000 7fa2c214e700 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7fa2bc10ee20 con 0x7fa2bc103240 2024-05-27T07:25:19.512 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:19.519+0000 7fa2bbfff700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fa2bc104d10 0x7fa2bc1a1c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:25:19.512 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:19.519+0000 7fa2bbfff700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fa2bc104d10 0x7fa2bc1a1c40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.52:35092/0 (socket says 172.21.15.52:35092) 2024-05-27T07:25:19.512 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:19.519+0000 7fa2bbfff700 1 -- 172.21.15.52:0/1366187099 learned_addr learned my addr 172.21.15.52:0/1366187099 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-05-27T07:25:19.512 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:19.520+0000 7fa2baffd700 1 --2- 172.21.15.52:0/1366187099 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fa2bc104370 0x7fa2bc19f670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:25:19.512 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:19.520+0000 7fa2bb7fe700 1 --2- 172.21.15.52:0/1366187099 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fa2bc103240 0x7fa2bc0764a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:25:19.512 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:19.520+0000 7fa2bbfff700 1 -- 172.21.15.52:0/1366187099 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fa2bc103240 msgr2=0x7fa2bc0764a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:25:19.512 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:19.520+0000 7fa2bbfff700 1 --2- 172.21.15.52:0/1366187099 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fa2bc103240 0x7fa2bc0764a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:25:19.513 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:19.520+0000 7fa2bbfff700 1 -- 172.21.15.52:0/1366187099 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fa2bc104370 msgr2=0x7fa2bc19f670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:25:19.513 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:19.520+0000 7fa2bbfff700 1 --2- 172.21.15.52:0/1366187099 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fa2bc104370 0x7fa2bc19f670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:25:19.513 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:19.520+0000 7fa2bbfff700 1 -- 172.21.15.52:0/1366187099 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa2bc1a2470 con 0x7fa2bc104d10 2024-05-27T07:25:19.513 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:19.520+0000 7fa2bbfff700 1 --2- 172.21.15.52:0/1366187099 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fa2bc104d10 0x7fa2bc1a1c40 secure :-1 s=READY pgs=348 cs=0 l=1 rev1=1 crypto rx=0x7fa2b400ada0 tx=0x7fa2b400c330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:25:19.513 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:19.520+0000 7fa2b8ff9700 1 -- 172.21.15.52:0/1366187099 <== mon.0 v2:172.21.15.52:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa2b4017020 con 0x7fa2bc104d10 2024-05-27T07:25:19.513 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:19.520+0000 7fa2c214e700 1 -- 172.21.15.52:0/1366187099 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa2bc1a2760 con 0x7fa2bc104d10 2024-05-27T07:25:19.513 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:19.520+0000 7fa2b8ff9700 1 -- 172.21.15.52:0/1366187099 <== mon.0 v2:172.21.15.52:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fa2b400cd80 con 0x7fa2bc104d10 2024-05-27T07:25:19.513 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:19.520+0000 7fa2b8ff9700 1 -- 172.21.15.52:0/1366187099 <== mon.0 v2:172.21.15.52:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa2b4005260 con 0x7fa2bc104d10 2024-05-27T07:25:19.513 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:19.520+0000 7fa2c214e700 1 -- 172.21.15.52:0/1366187099 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa2bc1a2c50 con 0x7fa2bc104d10 2024-05-27T07:25:19.513 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:19.522+0000 7fa2c214e700 1 -- 172.21.15.52:0/1366187099 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa2bc0512b0 con 0x7fa2bc104d10 2024-05-27T07:25:19.514 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:19.523+0000 7fa2b8ff9700 1 -- 172.21.15.52:0/1366187099 <== mon.0 v2:172.21.15.52:3300/0 4 ==== mgrmap(e 18) ==== 149652+0+0 (secure 0 0 0) 0x7fa2b400a090 con 0x7fa2bc104d10 2024-05-27T07:25:19.514 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:19.524+0000 7fa2b8ff9700 1 --2- 172.21.15.52:0/1366187099 >> [v2:172.21.15.52:6800/3513075805,v1:172.21.15.52:6801/3513075805] conn(0x7fa2a40b2880 0x7fa2a40b4d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:25:19.514 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:19.524+0000 7fa2bb7fe700 1 --2- 172.21.15.52:0/1366187099 >> [v2:172.21.15.52:6800/3513075805,v1:172.21.15.52:6801/3513075805] conn(0x7fa2a40b2880 0x7fa2a40b4d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:25:19.514 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:19.524+0000 7fa2b8ff9700 1 -- 172.21.15.52:0/1366187099 <== mon.0 v2:172.21.15.52:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9121+0+0 (secure 0 0 0) 0x7fa2b40e2270 con 0x7fa2bc104d10 2024-05-27T07:25:19.515 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:19.525+0000 7fa2bb7fe700 1 --2- 172.21.15.52:0/1366187099 >> [v2:172.21.15.52:6800/3513075805,v1:172.21.15.52:6801/3513075805] conn(0x7fa2a40b2880 0x7fa2a40b4d90 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7fa2bc104270 tx=0x7fa2ac007400 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:25:19.517 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:19.527+0000 7fa2b8ff9700 1 -- 172.21.15.52:0/1366187099 <== mon.0 v2:172.21.15.52:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191963 (secure 0 0 0) 0x7fa2b4021020 con 0x7fa2bc104d10 2024-05-27T07:25:19.744 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:19.755+0000 7fa2c214e700 1 -- 172.21.15.52:0/1366187099 --> [v2:172.21.15.52:6800/3513075805,v1:172.21.15.52:6801/3513075805] -- mgr_command(tid 0: {"prefix": "orch device ls", "target": ["mon-mgr", ""]}) -- 0x7fa2bc063bf0 con 0x7fa2a40b2880 2024-05-27T07:25:19.746 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:19.756+0000 7fa2b8ff9700 1 -- 172.21.15.52:0/1366187099 <== mgr.14152 v2:172.21.15.52:6800/3513075805 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+532 (secure 0 0 0) 0x7fa2bc063bf0 con 0x7fa2a40b2880 2024-05-27T07:25:19.746 INFO:teuthology.orchestra.run.smithi052.stdout:HOST PATH TYPE DEVICE ID SIZE AVAILABLE REFRESHED REJECT REASONS 2024-05-27T07:25:19.746 INFO:teuthology.orchestra.run.smithi052.stdout:smithi052 /dev/nvme0n1 ssd INTEL_SSDPED1D480GA_PHMB7513010N480DGN 447G No 5m ago Has a FileSystem, LVM detected 2024-05-27T07:25:19.746 INFO:teuthology.orchestra.run.smithi052.stdout:smithi105 /dev/nvme0n1 ssd INTEL_SSDPEDMD400G4_PHFT620400TR400BGN 372G No 4m ago Has a FileSystem, LVM detected 2024-05-27T07:25:19.746 INFO:teuthology.orchestra.run.smithi052.stdout:smithi173 /dev/nvme0n1 ssd INTEL_SSDPEDMD400G4_PHFT6204019S400BGN 372G No 4m ago Has a FileSystem, LVM detected 2024-05-27T07:25:19.748 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:19.759+0000 7fa2c214e700 1 -- 172.21.15.52:0/1366187099 >> [v2:172.21.15.52:6800/3513075805,v1:172.21.15.52:6801/3513075805] conn(0x7fa2a40b2880 msgr2=0x7fa2a40b4d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:25:19.749 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:19.759+0000 7fa2c214e700 1 --2- 172.21.15.52:0/1366187099 >> [v2:172.21.15.52:6800/3513075805,v1:172.21.15.52:6801/3513075805] conn(0x7fa2a40b2880 0x7fa2a40b4d90 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7fa2bc104270 tx=0x7fa2ac007400 comp rx=0 tx=0).stop 2024-05-27T07:25:19.749 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:19.759+0000 7fa2c214e700 1 -- 172.21.15.52:0/1366187099 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fa2bc104d10 msgr2=0x7fa2bc1a1c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:25:19.749 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:19.759+0000 7fa2c214e700 1 --2- 172.21.15.52:0/1366187099 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fa2bc104d10 0x7fa2bc1a1c40 secure :-1 s=READY pgs=348 cs=0 l=1 rev1=1 crypto rx=0x7fa2b400ada0 tx=0x7fa2b400c330 comp rx=0 tx=0).stop 2024-05-27T07:25:19.749 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:19.759+0000 7fa2c214e700 1 -- 172.21.15.52:0/1366187099 shutdown_connections 2024-05-27T07:25:19.749 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:19.759+0000 7fa2c214e700 1 --2- 172.21.15.52:0/1366187099 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fa2bc104d10 0x7fa2bc1a1c40 unknown :-1 s=CLOSED pgs=348 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:25:19.749 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:19.759+0000 7fa2c214e700 1 --2- 172.21.15.52:0/1366187099 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fa2bc103240 0x7fa2bc0764a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:25:19.749 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:19.759+0000 7fa2c214e700 1 --2- 172.21.15.52:0/1366187099 >> [v2:172.21.15.52:6800/3513075805,v1:172.21.15.52:6801/3513075805] conn(0x7fa2a40b2880 0x7fa2a40b4d90 unknown :-1 s=CLOSED pgs=216 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:25:19.749 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:19.759+0000 7fa2c214e700 1 --2- 172.21.15.52:0/1366187099 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fa2bc104370 0x7fa2bc19f670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:25:19.749 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:19.759+0000 7fa2c214e700 1 -- 172.21.15.52:0/1366187099 >> 172.21.15.52:0/1366187099 conn(0x7fa2bc0fe880 msgr2=0x7fa2bc1075a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:25:19.749 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:19.759+0000 7fa2c214e700 1 -- 172.21.15.52:0/1366187099 shutdown_connections 2024-05-27T07:25:19.749 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:19.759+0000 7fa2c214e700 1 -- 172.21.15.52:0/1366187099 wait complete. 2024-05-27T07:25:20.411 INFO:teuthology.run_tasks:Running task cephadm.shell... 2024-05-27T07:25:20.421 INFO:tasks.cephadm:Running commands on role mon.a host ubuntu@smithi052.front.sepia.ceph.com 2024-05-27T07:25:20.422 DEBUG:teuthology.orchestra.run.smithi052:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:38f9d900c83c63799fbdbe61acc9a11b0d3554a6 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 73df2636-1bf8-11ef-bc9b-c7b262605968 -- bash -c 'ceph fs dump' 2024-05-27T07:25:20.678 INFO:teuthology.orchestra.run.smithi052.stderr:Inferring config /var/lib/ceph/73df2636-1bf8-11ef-bc9b-c7b262605968/mon.a/config 2024-05-27T07:25:21.100 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:25:20 smithi052 ceph-mon[31955]: pgmap v405: 129 pgs: 129 active+clean; 647 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 9 op/s 2024-05-27T07:25:21.100 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:25:20 smithi052 ceph-mon[31955]: from='client.15189 -' entity='client.admin' cmd=[{"prefix": "orch device ls", "target": ["mon-mgr", ""]}]: dispatch 2024-05-27T07:25:21.101 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:25:20 smithi173 ceph-mon[37000]: pgmap v405: 129 pgs: 129 active+clean; 647 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 9 op/s 2024-05-27T07:25:21.101 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:25:20 smithi173 ceph-mon[37000]: from='client.15189 -' entity='client.admin' cmd=[{"prefix": "orch device ls", "target": ["mon-mgr", ""]}]: dispatch 2024-05-27T07:25:21.190 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:25:20 smithi105 ceph-mon[35977]: pgmap v405: 129 pgs: 129 active+clean; 647 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 9 op/s 2024-05-27T07:25:21.190 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:25:20 smithi105 ceph-mon[35977]: from='client.15189 -' entity='client.admin' cmd=[{"prefix": "orch device ls", "target": ["mon-mgr", ""]}]: dispatch 2024-05-27T07:25:22.320 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:22.330+0000 7f2dc633f700 1 -- 172.21.15.52:0/1326692400 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f2dc0104350 msgr2=0x7f2dc0104760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:25:22.320 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:22.330+0000 7f2dc633f700 1 --2- 172.21.15.52:0/1326692400 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f2dc0104350 0x7f2dc0104760 secure :-1 s=READY pgs=349 cs=0 l=1 rev1=1 crypto rx=0x7f2db0009990 tx=0x7f2db0009ca0 comp rx=0 tx=0).stop 2024-05-27T07:25:22.320 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:22.330+0000 7f2dc633f700 1 -- 172.21.15.52:0/1326692400 shutdown_connections 2024-05-27T07:25:22.320 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:22.330+0000 7f2dc633f700 1 --2- 172.21.15.52:0/1326692400 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f2dc0105e20 0x7f2dc010c6d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:25:22.321 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:22.330+0000 7f2dc633f700 1 --2- 172.21.15.52:0/1326692400 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f2dc0104350 0x7f2dc0104760 unknown :-1 s=CLOSED pgs=349 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:25:22.321 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:22.330+0000 7f2dc633f700 1 --2- 172.21.15.52:0/1326692400 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f2dc0105480 0x7f2dc0105890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:25:22.321 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:22.330+0000 7f2dc633f700 1 -- 172.21.15.52:0/1326692400 >> 172.21.15.52:0/1326692400 conn(0x7f2dc00ff9b0 msgr2=0x7f2dc0101e40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:25:22.321 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:22.330+0000 7f2dc633f700 1 -- 172.21.15.52:0/1326692400 shutdown_connections 2024-05-27T07:25:22.321 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:22.330+0000 7f2dc633f700 1 -- 172.21.15.52:0/1326692400 wait complete. 2024-05-27T07:25:22.322 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:22.331+0000 7f2dc633f700 1 Processor -- start 2024-05-27T07:25:22.322 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:22.331+0000 7f2dc633f700 1 -- start start 2024-05-27T07:25:22.323 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:22.331+0000 7f2dc633f700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f2dc0104350 0x7f2dc019a9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:25:22.323 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:22.331+0000 7f2dc633f700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f2dc0105480 0x7f2dc019aee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:25:22.323 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:22.331+0000 7f2dc633f700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f2dc0105e20 0x7f2dc01a1e80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:25:22.323 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:22.331+0000 7f2dc633f700 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x7f2dc010fd90 con 0x7f2dc0105e20 2024-05-27T07:25:22.323 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:22.331+0000 7f2dc633f700 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f2dc010fc10 con 0x7f2dc0104350 2024-05-27T07:25:22.323 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:22.331+0000 7f2dc633f700 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f2dc010ff10 con 0x7f2dc0105480 2024-05-27T07:25:22.323 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:22.331+0000 7f2dc48dc700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f2dc0105e20 0x7f2dc01a1e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:25:22.323 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:22.331+0000 7f2dc48dc700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f2dc0105e20 0x7f2dc01a1e80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.52:48028/0 (socket says 172.21.15.52:48028) 2024-05-27T07:25:22.323 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:22.331+0000 7f2dc48dc700 1 -- 172.21.15.52:0/1765381821 learned_addr learned my addr 172.21.15.52:0/1765381821 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-05-27T07:25:22.323 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:22.331+0000 7f2db7fff700 1 --2- 172.21.15.52:0/1765381821 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f2dc0105480 0x7f2dc019aee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:25:22.323 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:22.331+0000 7f2dbffff700 1 --2- 172.21.15.52:0/1765381821 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f2dc0104350 0x7f2dc019a9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:25:22.323 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:22.332+0000 7f2dc48dc700 1 -- 172.21.15.52:0/1765381821 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f2dc0105480 msgr2=0x7f2dc019aee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:25:22.324 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:22.332+0000 7f2dc48dc700 1 --2- 172.21.15.52:0/1765381821 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f2dc0105480 0x7f2dc019aee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:25:22.324 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:22.332+0000 7f2dc48dc700 1 -- 172.21.15.52:0/1765381821 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f2dc0104350 msgr2=0x7f2dc019a9a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:25:22.324 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:22.332+0000 7f2dc48dc700 1 --2- 172.21.15.52:0/1765381821 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f2dc0104350 0x7f2dc019a9a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:25:22.324 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:22.332+0000 7f2dc48dc700 1 -- 172.21.15.52:0/1765381821 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2dc01a2470 con 0x7f2dc0105e20 2024-05-27T07:25:22.324 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:22.332+0000 7f2dbffff700 1 --2- 172.21.15.52:0/1765381821 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f2dc0104350 0x7f2dc019a9a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-27T07:25:22.324 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:22.332+0000 7f2dc48dc700 1 --2- 172.21.15.52:0/1765381821 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f2dc0105e20 0x7f2dc01a1e80 secure :-1 s=READY pgs=350 cs=0 l=1 rev1=1 crypto rx=0x7f2db8007d70 tx=0x7f2db800a340 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:25:22.324 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:22.332+0000 7f2dbdffb700 1 -- 172.21.15.52:0/1765381821 <== mon.0 v2:172.21.15.52:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2db8016020 con 0x7f2dc0105e20 2024-05-27T07:25:22.324 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:22.332+0000 7f2dc633f700 1 -- 172.21.15.52:0/1765381821 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2dc01a2760 con 0x7f2dc0105e20 2024-05-27T07:25:22.324 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:22.332+0000 7f2dbdffb700 1 -- 172.21.15.52:0/1765381821 <== mon.0 v2:172.21.15.52:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f2db800ad40 con 0x7f2dc0105e20 2024-05-27T07:25:22.324 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:22.332+0000 7f2dbdffb700 1 -- 172.21.15.52:0/1765381821 <== mon.0 v2:172.21.15.52:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2db80115e0 con 0x7f2dc0105e20 2024-05-27T07:25:22.324 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:22.332+0000 7f2dc633f700 1 -- 172.21.15.52:0/1765381821 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2dc01a2ca0 con 0x7f2dc0105e20 2024-05-27T07:25:22.325 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:22.333+0000 7f2dc633f700 1 -- 172.21.15.52:0/1765381821 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2dc00512b0 con 0x7f2dc0105e20 2024-05-27T07:25:22.325 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:22.335+0000 7f2dbdffb700 1 -- 172.21.15.52:0/1765381821 <== mon.0 v2:172.21.15.52:3300/0 4 ==== mgrmap(e 18) ==== 149652+0+0 (secure 0 0 0) 0x7f2db8019020 con 0x7f2dc0105e20 2024-05-27T07:25:22.325 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:22.336+0000 7f2dbdffb700 1 --2- 172.21.15.52:0/1765381821 >> [v2:172.21.15.52:6800/3513075805,v1:172.21.15.52:6801/3513075805] conn(0x7f2dac0b2880 0x7f2dac0b4d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:25:22.325 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:22.336+0000 7f2dbffff700 1 --2- 172.21.15.52:0/1765381821 >> [v2:172.21.15.52:6800/3513075805,v1:172.21.15.52:6801/3513075805] conn(0x7f2dac0b2880 0x7f2dac0b4d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:25:22.325 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:22.336+0000 7f2dbdffb700 1 -- 172.21.15.52:0/1765381821 <== mon.0 v2:172.21.15.52:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9121+0+0 (secure 0 0 0) 0x7f2db80e2590 con 0x7f2dc0105e20 2024-05-27T07:25:22.326 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:22.336+0000 7f2dbffff700 1 --2- 172.21.15.52:0/1765381821 >> [v2:172.21.15.52:6800/3513075805,v1:172.21.15.52:6801/3513075805] conn(0x7f2dac0b2880 0x7f2dac0b4d90 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7f2db00038a0 tx=0x7f2db001a040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:25:22.330 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:22.340+0000 7f2dbdffb700 1 -- 172.21.15.52:0/1765381821 <== mon.0 v2:172.21.15.52:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191963 (secure 0 0 0) 0x7f2db8020020 con 0x7f2dc0105e20 2024-05-27T07:25:22.597 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:22.607+0000 7f2dc633f700 1 -- 172.21.15.52:0/1765381821 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_command({"prefix": "fs dump"} v 0) -- 0x7f2dc00643c0 con 0x7f2dc0105e20 2024-05-27T07:25:22.597 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:22.607+0000 7f2dbdffb700 1 -- 172.21.15.52:0/1765381821 <== mon.0 v2:172.21.15.52:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump"}]=0 dumped fsmap epoch 29 v29) ==== 76+0+3096 (secure 0 0 0) 0x7f2db80a9ed0 con 0x7f2dc0105e20 2024-05-27T07:25:22.597 INFO:teuthology.orchestra.run.smithi052.stdout:e29 2024-05-27T07:25:22.597 INFO:teuthology.orchestra.run.smithi052.stdout:enable_multiple, ever_enabled_multiple: 1,1 2024-05-27T07:25:22.598 INFO:teuthology.orchestra.run.smithi052.stdout:default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2,11=minor log segments,12=quiesce subvolumes} 2024-05-27T07:25:22.598 INFO:teuthology.orchestra.run.smithi052.stdout:legacy client fscid: 1 2024-05-27T07:25:22.598 INFO:teuthology.orchestra.run.smithi052.stdout: 2024-05-27T07:25:22.598 INFO:teuthology.orchestra.run.smithi052.stdout:Filesystem 'cephfs' (1) 2024-05-27T07:25:22.598 INFO:teuthology.orchestra.run.smithi052.stdout:fs_name cephfs 2024-05-27T07:25:22.598 INFO:teuthology.orchestra.run.smithi052.stdout:epoch 29 2024-05-27T07:25:22.598 INFO:teuthology.orchestra.run.smithi052.stdout:flags 32 joinable allow_snaps allow_multimds_snaps allow_standby_replay 2024-05-27T07:25:22.598 INFO:teuthology.orchestra.run.smithi052.stdout:created 2024-05-27T07:21:50.521783+0000 2024-05-27T07:25:22.598 INFO:teuthology.orchestra.run.smithi052.stdout:modified 2024-05-27T07:22:58.822630+0000 2024-05-27T07:25:22.598 INFO:teuthology.orchestra.run.smithi052.stdout:tableserver 0 2024-05-27T07:25:22.598 INFO:teuthology.orchestra.run.smithi052.stdout:root 0 2024-05-27T07:25:22.598 INFO:teuthology.orchestra.run.smithi052.stdout:session_timeout 300 2024-05-27T07:25:22.598 INFO:teuthology.orchestra.run.smithi052.stdout:session_autoclose 300 2024-05-27T07:25:22.599 INFO:teuthology.orchestra.run.smithi052.stdout:max_file_size 1099511627776 2024-05-27T07:25:22.599 INFO:teuthology.orchestra.run.smithi052.stdout:max_xattr_size 65536 2024-05-27T07:25:22.599 INFO:teuthology.orchestra.run.smithi052.stdout:required_client_features {} 2024-05-27T07:25:22.599 INFO:teuthology.orchestra.run.smithi052.stdout:last_failure 0 2024-05-27T07:25:22.599 INFO:teuthology.orchestra.run.smithi052.stdout:last_failure_osd_epoch 0 2024-05-27T07:25:22.599 INFO:teuthology.orchestra.run.smithi052.stdout:compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2,11=minor log segments,12=quiesce subvolumes} 2024-05-27T07:25:22.599 INFO:teuthology.orchestra.run.smithi052.stdout:max_mds 5 2024-05-27T07:25:22.599 INFO:teuthology.orchestra.run.smithi052.stdout:in 0,1,2,3,4 2024-05-27T07:25:22.599 INFO:teuthology.orchestra.run.smithi052.stdout:up {0=24527,1=24497,2=24475,3=24473,4=24466} 2024-05-27T07:25:22.599 INFO:teuthology.orchestra.run.smithi052.stdout:failed 2024-05-27T07:25:22.599 INFO:teuthology.orchestra.run.smithi052.stdout:damaged 2024-05-27T07:25:22.599 INFO:teuthology.orchestra.run.smithi052.stdout:stopped 2024-05-27T07:25:22.599 INFO:teuthology.orchestra.run.smithi052.stdout:data_pools [3,4] 2024-05-27T07:25:22.599 INFO:teuthology.orchestra.run.smithi052.stdout:metadata_pool 2 2024-05-27T07:25:22.600 INFO:teuthology.orchestra.run.smithi052.stdout:inline_data disabled 2024-05-27T07:25:22.600 INFO:teuthology.orchestra.run.smithi052.stdout:balancer 2024-05-27T07:25:22.600 INFO:teuthology.orchestra.run.smithi052.stdout:bal_rank_mask -1 2024-05-27T07:25:22.600 INFO:teuthology.orchestra.run.smithi052.stdout:standby_count_wanted 0 2024-05-27T07:25:22.600 INFO:teuthology.orchestra.run.smithi052.stdout:qdb_cluster leader: 24527 members: 24466,24473,24475,24497,24527 2024-05-27T07:25:22.600 INFO:teuthology.orchestra.run.smithi052.stdout:[mds.b{0:24527} state up:active seq 5 addr [v2:172.21.15.105:6838/500348151,v1:172.21.15.105:6839/500348151] compat {c=[1],r=[1],i=[1fff]}] 2024-05-27T07:25:22.600 INFO:teuthology.orchestra.run.smithi052.stdout:[mds.f{0:24460} state up:standby-replay seq 1 addr [v2:172.21.15.173:6834/1619486437,v1:172.21.15.173:6835/1619486437] compat {c=[1],r=[1],i=[1fff]}] 2024-05-27T07:25:22.600 INFO:teuthology.orchestra.run.smithi052.stdout:[mds.h{1:24497} state up:active seq 24 addr [v2:172.21.15.105:6836/191393149,v1:172.21.15.105:6837/191393149] compat {c=[1],r=[1],i=[1fff]}] 2024-05-27T07:25:22.600 INFO:teuthology.orchestra.run.smithi052.stdout:[mds.k{1:24452} state up:standby-replay seq 1 addr [v2:172.21.15.105:6832/2800678412,v1:172.21.15.105:6833/2800678412] compat {c=[1],r=[1],i=[1fff]}] 2024-05-27T07:25:22.600 INFO:teuthology.orchestra.run.smithi052.stdout:[mds.i{2:24475} state up:active seq 23 addr [v2:172.21.15.173:6838/564778293,v1:172.21.15.173:6839/564778293] compat {c=[1],r=[1],i=[1fff]}] 2024-05-27T07:25:22.600 INFO:teuthology.orchestra.run.smithi052.stdout:[mds.l{2:24433} state up:standby-replay seq 1 addr [v2:172.21.15.173:6832/3013589220,v1:172.21.15.173:6833/3013589220] compat {c=[1],r=[1],i=[1fff]}] 2024-05-27T07:25:22.600 INFO:teuthology.orchestra.run.smithi052.stdout:[mds.e{3:24473} state up:active seq 26 addr [v2:172.21.15.105:6834/3298498384,v1:172.21.15.105:6835/3298498384] compat {c=[1],r=[1],i=[1fff]}] 2024-05-27T07:25:22.600 INFO:teuthology.orchestra.run.smithi052.stdout:[mds.a{3:14631} state up:standby-replay seq 1 addr [v2:172.21.15.52:6840/2878618188,v1:172.21.15.52:6841/2878618188] compat {c=[1],r=[1],i=[1fff]}] 2024-05-27T07:25:22.600 INFO:teuthology.orchestra.run.smithi052.stdout:[mds.c{4:24466} state up:active seq 25 addr [v2:172.21.15.173:6836/1396363827,v1:172.21.15.173:6837/1396363827] compat {c=[1],r=[1],i=[1fff]}] 2024-05-27T07:25:22.601 INFO:teuthology.orchestra.run.smithi052.stdout:[mds.g{4:14619} state up:standby-replay seq 1 addr [v2:172.21.15.52:6838/4258549559,v1:172.21.15.52:6839/4258549559] compat {c=[1],r=[1],i=[1fff]}] 2024-05-27T07:25:22.601 INFO:teuthology.orchestra.run.smithi052.stdout: 2024-05-27T07:25:22.601 INFO:teuthology.orchestra.run.smithi052.stdout: 2024-05-27T07:25:22.601 INFO:teuthology.orchestra.run.smithi052.stdout:Standby daemons: 2024-05-27T07:25:22.601 INFO:teuthology.orchestra.run.smithi052.stdout: 2024-05-27T07:25:22.601 INFO:teuthology.orchestra.run.smithi052.stdout:[mds.j{-1:14556} state up:standby seq 1 addr [v2:172.21.15.52:6834/2584733778,v1:172.21.15.52:6835/2584733778] compat {c=[1],r=[1],i=[1fff]}] 2024-05-27T07:25:22.601 INFO:teuthology.orchestra.run.smithi052.stdout:[mds.d{-1:14568} state up:standby seq 1 addr [v2:172.21.15.52:6836/675445303,v1:172.21.15.52:6837/675445303] compat {c=[1],r=[1],i=[1fff]}] 2024-05-27T07:25:22.602 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:22.610+0000 7f2dc633f700 1 -- 172.21.15.52:0/1765381821 >> [v2:172.21.15.52:6800/3513075805,v1:172.21.15.52:6801/3513075805] conn(0x7f2dac0b2880 msgr2=0x7f2dac0b4d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:25:22.602 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:22.610+0000 7f2dc633f700 1 --2- 172.21.15.52:0/1765381821 >> [v2:172.21.15.52:6800/3513075805,v1:172.21.15.52:6801/3513075805] conn(0x7f2dac0b2880 0x7f2dac0b4d90 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7f2db00038a0 tx=0x7f2db001a040 comp rx=0 tx=0).stop 2024-05-27T07:25:22.602 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:22.610+0000 7f2dc633f700 1 -- 172.21.15.52:0/1765381821 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f2dc0105e20 msgr2=0x7f2dc01a1e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:25:22.602 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:22.610+0000 7f2dc633f700 1 --2- 172.21.15.52:0/1765381821 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f2dc0105e20 0x7f2dc01a1e80 secure :-1 s=READY pgs=350 cs=0 l=1 rev1=1 crypto rx=0x7f2db8007d70 tx=0x7f2db800a340 comp rx=0 tx=0).stop 2024-05-27T07:25:22.602 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:22.610+0000 7f2dc633f700 1 -- 172.21.15.52:0/1765381821 shutdown_connections 2024-05-27T07:25:22.602 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:22.610+0000 7f2dc633f700 1 --2- 172.21.15.52:0/1765381821 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f2dc0105e20 0x7f2dc01a1e80 unknown :-1 s=CLOSED pgs=350 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:25:22.602 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:22.610+0000 7f2dc633f700 1 --2- 172.21.15.52:0/1765381821 >> [v2:172.21.15.52:6800/3513075805,v1:172.21.15.52:6801/3513075805] conn(0x7f2dac0b2880 0x7f2dac0b4d90 unknown :-1 s=CLOSED pgs=217 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:25:22.602 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:22.610+0000 7f2dc633f700 1 --2- 172.21.15.52:0/1765381821 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f2dc0104350 0x7f2dc019a9a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:25:22.602 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:22.610+0000 7f2dc633f700 1 --2- 172.21.15.52:0/1765381821 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f2dc0105480 0x7f2dc019aee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:25:22.603 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:22.610+0000 7f2dc633f700 1 -- 172.21.15.52:0/1765381821 >> 172.21.15.52:0/1765381821 conn(0x7f2dc00ff9b0 msgr2=0x7f2dc0108720 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:25:22.603 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:22.610+0000 7f2dc633f700 1 -- 172.21.15.52:0/1765381821 shutdown_connections 2024-05-27T07:25:22.603 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:22.610+0000 7f2dc633f700 1 -- 172.21.15.52:0/1765381821 wait complete. 2024-05-27T07:25:22.603 INFO:teuthology.orchestra.run.smithi052.stderr:dumped fsmap epoch 29 2024-05-27T07:25:22.963 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:25:22 smithi052 ceph-mon[31955]: pgmap v406: 129 pgs: 129 active+clean; 647 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-05-27T07:25:22.963 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:25:22 smithi052 ceph-mon[31955]: from='client.? 172.21.15.52:0/1765381821' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-05-27T07:25:23.101 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:25:22 smithi173 ceph-mon[37000]: pgmap v406: 129 pgs: 129 active+clean; 647 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-05-27T07:25:23.102 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:25:22 smithi173 ceph-mon[37000]: from='client.? 172.21.15.52:0/1765381821' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-05-27T07:25:23.190 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:25:22 smithi105 ceph-mon[35977]: pgmap v406: 129 pgs: 129 active+clean; 647 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-05-27T07:25:23.190 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:25:22 smithi105 ceph-mon[35977]: from='client.? 172.21.15.52:0/1765381821' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-05-27T07:25:23.263 DEBUG:teuthology.orchestra.run.smithi052:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:38f9d900c83c63799fbdbe61acc9a11b0d3554a6 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 73df2636-1bf8-11ef-bc9b-c7b262605968 -- bash -c 'ceph osd dump' 2024-05-27T07:25:23.515 INFO:teuthology.orchestra.run.smithi052.stderr:Inferring config /var/lib/ceph/73df2636-1bf8-11ef-bc9b-c7b262605968/mon.a/config 2024-05-27T07:25:24.949 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:25:24 smithi052 ceph-mon[31955]: pgmap v407: 129 pgs: 129 active+clean; 647 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 9 op/s 2024-05-27T07:25:25.101 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:25:24 smithi173 ceph-mon[37000]: pgmap v407: 129 pgs: 129 active+clean; 647 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 9 op/s 2024-05-27T07:25:25.189 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:25:24 smithi105 ceph-mon[35977]: pgmap v407: 129 pgs: 129 active+clean; 647 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 9 op/s 2024-05-27T07:25:25.190 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:25.200+0000 7f5fb7388700 1 -- 172.21.15.52:0/3123091897 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f5fb006c280 msgr2=0x7f5fb0106ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:25:25.190 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:25.200+0000 7f5fb7388700 1 --2- 172.21.15.52:0/3123091897 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f5fb006c280 0x7f5fb0106ee0 secure :-1 s=READY pgs=351 cs=0 l=1 rev1=1 crypto rx=0x7f5fa000b0a0 tx=0x7f5fa000b3b0 comp rx=0 tx=0).stop 2024-05-27T07:25:25.191 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:25.200+0000 7f5fb7388700 1 -- 172.21.15.52:0/3123091897 shutdown_connections 2024-05-27T07:25:25.191 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:25.200+0000 7f5fb7388700 1 --2- 172.21.15.52:0/3123091897 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f5fb0109fe0 0x7f5fb010c4f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:25:25.191 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:25.200+0000 7f5fb7388700 1 --2- 172.21.15.52:0/3123091897 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f5fb006c280 0x7f5fb0106ee0 unknown :-1 s=CLOSED pgs=351 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:25:25.191 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:25.200+0000 7f5fb7388700 1 --2- 172.21.15.52:0/3123091897 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f5fb01074b0 0x7f5fb0109950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:25:25.191 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:25.200+0000 7f5fb7388700 1 -- 172.21.15.52:0/3123091897 >> 172.21.15.52:0/3123091897 conn(0x7f5fb00fe870 msgr2=0x7f5fb0100d20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:25:25.191 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:25.200+0000 7f5fb7388700 1 -- 172.21.15.52:0/3123091897 shutdown_connections 2024-05-27T07:25:25.191 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:25.200+0000 7f5fb7388700 1 -- 172.21.15.52:0/3123091897 wait complete. 2024-05-27T07:25:25.193 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:25.201+0000 7f5fb7388700 1 Processor -- start 2024-05-27T07:25:25.193 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:25.201+0000 7f5fb7388700 1 -- start start 2024-05-27T07:25:25.193 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:25.201+0000 7f5fb7388700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f5fb006c280 0x7f5fb0104730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:25:25.193 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:25.201+0000 7f5fb7388700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f5fb01074b0 0x7f5fb0102de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:25:25.193 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:25.201+0000 7f5fb7388700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f5fb0109fe0 0x7f5fb0103320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:25:25.193 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:25.201+0000 7f5fb7388700 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x7f5fb010fdc0 con 0x7f5fb006c280 2024-05-27T07:25:25.193 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:25.201+0000 7f5fb7388700 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f5fb010fc40 con 0x7f5fb0109fe0 2024-05-27T07:25:25.193 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:25.201+0000 7f5fb7388700 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f5fb010ff40 con 0x7f5fb01074b0 2024-05-27T07:25:25.194 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:25.202+0000 7f5fb5124700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f5fb006c280 0x7f5fb0104730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:25:25.194 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:25.202+0000 7f5fb5124700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f5fb006c280 0x7f5fb0104730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.52:48048/0 (socket says 172.21.15.52:48048) 2024-05-27T07:25:25.194 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:25.202+0000 7f5fb5124700 1 -- 172.21.15.52:0/3888162089 learned_addr learned my addr 172.21.15.52:0/3888162089 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-05-27T07:25:25.194 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:25.202+0000 7f5fb5925700 1 --2- 172.21.15.52:0/3888162089 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f5fb0109fe0 0x7f5fb0103320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:25:25.194 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:25.202+0000 7f5fb4923700 1 --2- 172.21.15.52:0/3888162089 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f5fb01074b0 0x7f5fb0102de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:25:25.194 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:25.202+0000 7f5fb5124700 1 -- 172.21.15.52:0/3888162089 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f5fb01074b0 msgr2=0x7f5fb0102de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:25:25.194 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:25.202+0000 7f5fb5124700 1 --2- 172.21.15.52:0/3888162089 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f5fb01074b0 0x7f5fb0102de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:25:25.194 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:25.202+0000 7f5fb5124700 1 -- 172.21.15.52:0/3888162089 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f5fb0109fe0 msgr2=0x7f5fb0103320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:25:25.194 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:25.202+0000 7f5fb5124700 1 --2- 172.21.15.52:0/3888162089 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f5fb0109fe0 0x7f5fb0103320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:25:25.194 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:25.202+0000 7f5fb5124700 1 -- 172.21.15.52:0/3888162089 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5fb0103a80 con 0x7f5fb006c280 2024-05-27T07:25:25.194 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:25.202+0000 7f5fb5925700 1 --2- 172.21.15.52:0/3888162089 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f5fb0109fe0 0x7f5fb0103320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-27T07:25:25.194 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:25.202+0000 7f5fb5124700 1 --2- 172.21.15.52:0/3888162089 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f5fb006c280 0x7f5fb0104730 secure :-1 s=READY pgs=352 cs=0 l=1 rev1=1 crypto rx=0x7f5fa00062a0 tx=0x7f5fa0004ef0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:25:25.195 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:25.202+0000 7f5fae7fc700 1 -- 172.21.15.52:0/3888162089 <== mon.0 v2:172.21.15.52:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5fa0021070 con 0x7f5fb006c280 2024-05-27T07:25:25.195 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:25.202+0000 7f5fb7388700 1 -- 172.21.15.52:0/3888162089 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5fb01a4a00 con 0x7f5fb006c280 2024-05-27T07:25:25.195 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:25.202+0000 7f5fae7fc700 1 -- 172.21.15.52:0/3888162089 <== mon.0 v2:172.21.15.52:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f5fa0004020 con 0x7f5fb006c280 2024-05-27T07:25:25.195 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:25.202+0000 7f5fae7fc700 1 -- 172.21.15.52:0/3888162089 <== mon.0 v2:172.21.15.52:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5fa001d6b0 con 0x7f5fb006c280 2024-05-27T07:25:25.195 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:25.202+0000 7f5fb7388700 1 -- 172.21.15.52:0/3888162089 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5fb01a4e60 con 0x7f5fb006c280 2024-05-27T07:25:25.195 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:25.203+0000 7f5fb7388700 1 -- 172.21.15.52:0/3888162089 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5fb00512b0 con 0x7f5fb006c280 2024-05-27T07:25:25.195 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:25.205+0000 7f5fae7fc700 1 -- 172.21.15.52:0/3888162089 <== mon.0 v2:172.21.15.52:3300/0 4 ==== mgrmap(e 18) ==== 149652+0+0 (secure 0 0 0) 0x7f5fa0003a00 con 0x7f5fb006c280 2024-05-27T07:25:25.196 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:25.206+0000 7f5fae7fc700 1 --2- 172.21.15.52:0/3888162089 >> [v2:172.21.15.52:6800/3513075805,v1:172.21.15.52:6801/3513075805] conn(0x7f5f980b2880 0x7f5f980b4d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:25:25.196 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:25.206+0000 7f5fb4923700 1 --2- 172.21.15.52:0/3888162089 >> [v2:172.21.15.52:6800/3513075805,v1:172.21.15.52:6801/3513075805] conn(0x7f5f980b2880 0x7f5f980b4d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:25:25.196 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:25.206+0000 7f5fae7fc700 1 -- 172.21.15.52:0/3888162089 <== mon.0 v2:172.21.15.52:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9121+0+0 (secure 0 0 0) 0x7f5fa000f020 con 0x7f5fb006c280 2024-05-27T07:25:25.196 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:25.207+0000 7f5fb4923700 1 --2- 172.21.15.52:0/3888162089 >> [v2:172.21.15.52:6800/3513075805,v1:172.21.15.52:6801/3513075805] conn(0x7f5f980b2880 0x7f5f980b4d90 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7f5fa4005950 tx=0x7f5fa40058e0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:25:25.200 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:25.210+0000 7f5fae7fc700 1 -- 172.21.15.52:0/3888162089 <== mon.0 v2:172.21.15.52:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191963 (secure 0 0 0) 0x7f5fa002a020 con 0x7f5fb006c280 2024-05-27T07:25:25.418 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:25.428+0000 7f5fb7388700 1 -- 172.21.15.52:0/3888162089 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_command({"prefix": "osd dump"} v 0) -- 0x7f5fb002e990 con 0x7f5fb006c280 2024-05-27T07:25:25.419 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:25.429+0000 7f5fae7fc700 1 -- 172.21.15.52:0/3888162089 <== mon.0 v2:172.21.15.52:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump"}]=0 v90) ==== 56+0+5627 (secure 0 0 0) 0x7f5fa00abba0 con 0x7f5fb006c280 2024-05-27T07:25:25.419 INFO:teuthology.orchestra.run.smithi052.stdout:epoch 90 2024-05-27T07:25:25.419 INFO:teuthology.orchestra.run.smithi052.stdout:fsid 73df2636-1bf8-11ef-bc9b-c7b262605968 2024-05-27T07:25:25.419 INFO:teuthology.orchestra.run.smithi052.stdout:created 2024-05-27T07:13:34.738032+0000 2024-05-27T07:25:25.419 INFO:teuthology.orchestra.run.smithi052.stdout:modified 2024-05-27T07:23:30.969990+0000 2024-05-27T07:25:25.419 INFO:teuthology.orchestra.run.smithi052.stdout:flags sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit 2024-05-27T07:25:25.419 INFO:teuthology.orchestra.run.smithi052.stdout:crush_version 27 2024-05-27T07:25:25.419 INFO:teuthology.orchestra.run.smithi052.stdout:full_ratio 0.95 2024-05-27T07:25:25.420 INFO:teuthology.orchestra.run.smithi052.stdout:backfillfull_ratio 0.9 2024-05-27T07:25:25.420 INFO:teuthology.orchestra.run.smithi052.stdout:nearfull_ratio 0.85 2024-05-27T07:25:25.420 INFO:teuthology.orchestra.run.smithi052.stdout:require_min_compat_client luminous 2024-05-27T07:25:25.420 INFO:teuthology.orchestra.run.smithi052.stdout:min_compat_client luminous 2024-05-27T07:25:25.420 INFO:teuthology.orchestra.run.smithi052.stdout:require_osd_release squid 2024-05-27T07:25:25.420 INFO:teuthology.orchestra.run.smithi052.stdout:stretch_mode_enabled false 2024-05-27T07:25:25.420 INFO:teuthology.orchestra.run.smithi052.stdout:pool 1 '.mgr' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 1 pgp_num 1 autoscale_mode off last_change 21 flags hashpspool stripe_width 0 pg_num_max 32 pg_num_min 1 application mgr read_balance_score 12.00 2024-05-27T07:25:25.420 INFO:teuthology.orchestra.run.smithi052.stdout:pool 2 'cephfs_metadata' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 32 pgp_num 32 autoscale_mode off last_change 79 flags hashpspool stripe_width 0 pg_autoscale_bias 4 pg_num_min 16 recovery_priority 5 application cephfs read_balance_score 1.88 2024-05-27T07:25:25.420 INFO:teuthology.orchestra.run.smithi052.stdout:pool 3 'cephfs_data' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 64 pgp_num 64 autoscale_mode off last_change 79 flags hashpspool stripe_width 0 pg_num_min 64 target_size_ratio 0.05 application cephfs read_balance_score 1.50 2024-05-27T07:25:25.420 INFO:teuthology.orchestra.run.smithi052.stdout:pool 4 'cephfs_data_ec' erasure profile cephfs_data_ec size 4 min_size 3 crush_rule 1 object_hash rjenkins pg_num 32 pgp_num 32 autoscale_mode off last_change 86 flags hashpspool,ec_overwrites stripe_width 8192 pg_num_min 64 target_size_ratio 0.9 application cephfs 2024-05-27T07:25:25.420 INFO:teuthology.orchestra.run.smithi052.stdout:max_osd 12 2024-05-27T07:25:25.420 INFO:teuthology.orchestra.run.smithi052.stdout:osd.0 up in weight 1 up_from 8 up_thru 82 down_at 0 last_clean_interval [0,0) [v2:172.21.15.52:6802/1626508899,v1:172.21.15.52:6803/1626508899] [v2:172.21.15.52:6804/1626508899,v1:172.21.15.52:6805/1626508899] exists,up 347b8a89-9dbe-4848-83b0-fd0e9374f33d 2024-05-27T07:25:25.420 INFO:teuthology.orchestra.run.smithi052.stdout:osd.1 up in weight 1 up_from 13 up_thru 89 down_at 0 last_clean_interval [0,0) [v2:172.21.15.105:6800/2900093195,v1:172.21.15.105:6801/2900093195] [v2:172.21.15.105:6802/2900093195,v1:172.21.15.105:6803/2900093195] exists,up 7a3623d4-e087-40cb-9336-d4a5c52203d4 2024-05-27T07:25:25.421 INFO:teuthology.orchestra.run.smithi052.stdout:osd.2 up in weight 1 up_from 18 up_thru 87 down_at 0 last_clean_interval [0,0) [v2:172.21.15.173:6800/2251189314,v1:172.21.15.173:6801/2251189314] [v2:172.21.15.173:6802/2251189314,v1:172.21.15.173:6803/2251189314] exists,up 41a9df5b-882a-47da-bb26-7f4d26ccf6c1 2024-05-27T07:25:25.421 INFO:teuthology.orchestra.run.smithi052.stdout:osd.3 up in weight 1 up_from 26 up_thru 82 down_at 0 last_clean_interval [0,0) [v2:172.21.15.52:6810/57920794,v1:172.21.15.52:6811/57920794] [v2:172.21.15.52:6812/57920794,v1:172.21.15.52:6813/57920794] exists,up f1245439-42a5-46c5-a998-e0a78f9df3fb 2024-05-27T07:25:25.421 INFO:teuthology.orchestra.run.smithi052.stdout:osd.4 up in weight 1 up_from 32 up_thru 87 down_at 0 last_clean_interval [0,0) [v2:172.21.15.105:6808/3441134818,v1:172.21.15.105:6809/3441134818] [v2:172.21.15.105:6810/3441134818,v1:172.21.15.105:6811/3441134818] exists,up 2592d1cc-1726-4359-83d8-a2e6314b54ec 2024-05-27T07:25:25.421 INFO:teuthology.orchestra.run.smithi052.stdout:osd.5 up in weight 1 up_from 37 up_thru 87 down_at 0 last_clean_interval [0,0) [v2:172.21.15.173:6808/2989239729,v1:172.21.15.173:6809/2989239729] [v2:172.21.15.173:6810/2989239729,v1:172.21.15.173:6811/2989239729] exists,up 27e98ee2-4a92-4972-b2a3-959d0ed90981 2024-05-27T07:25:25.421 INFO:teuthology.orchestra.run.smithi052.stdout:osd.6 up in weight 1 up_from 43 up_thru 82 down_at 0 last_clean_interval [0,0) [v2:172.21.15.52:6818/4244423065,v1:172.21.15.52:6819/4244423065] [v2:172.21.15.52:6820/4244423065,v1:172.21.15.52:6821/4244423065] exists,up 2b0ec0cb-f438-4b2c-8642-0778ed374e9a 2024-05-27T07:25:25.421 INFO:teuthology.orchestra.run.smithi052.stdout:osd.7 up in weight 1 up_from 48 up_thru 82 down_at 0 last_clean_interval [0,0) [v2:172.21.15.105:6816/2063611552,v1:172.21.15.105:6817/2063611552] [v2:172.21.15.105:6818/2063611552,v1:172.21.15.105:6819/2063611552] exists,up cb2a7a6b-8015-43ea-865a-be144e45c51a 2024-05-27T07:25:25.421 INFO:teuthology.orchestra.run.smithi052.stdout:osd.8 up in weight 1 up_from 54 up_thru 87 down_at 0 last_clean_interval [0,0) [v2:172.21.15.173:6816/4131399852,v1:172.21.15.173:6817/4131399852] [v2:172.21.15.173:6818/4131399852,v1:172.21.15.173:6819/4131399852] exists,up c2fc565d-a0b9-4396-a209-bd884cddcfe3 2024-05-27T07:25:25.421 INFO:teuthology.orchestra.run.smithi052.stdout:osd.9 up in weight 1 up_from 60 up_thru 87 down_at 0 last_clean_interval [0,0) [v2:172.21.15.52:6826/479119778,v1:172.21.15.52:6827/479119778] [v2:172.21.15.52:6828/479119778,v1:172.21.15.52:6829/479119778] exists,up 78fa7f17-1dad-4262-be40-31fa82e75520 2024-05-27T07:25:25.421 INFO:teuthology.orchestra.run.smithi052.stdout:osd.10 up in weight 1 up_from 65 up_thru 76 down_at 0 last_clean_interval [0,0) [v2:172.21.15.105:6824/4032386628,v1:172.21.15.105:6825/4032386628] [v2:172.21.15.105:6826/4032386628,v1:172.21.15.105:6827/4032386628] exists,up e6f8e753-e15c-45b5-8ee9-d50f384559f9 2024-05-27T07:25:25.421 INFO:teuthology.orchestra.run.smithi052.stdout:osd.11 up in weight 1 up_from 71 up_thru 87 down_at 0 last_clean_interval [0,0) [v2:172.21.15.173:6824/1816308401,v1:172.21.15.173:6825/1816308401] [v2:172.21.15.173:6826/1816308401,v1:172.21.15.173:6827/1816308401] exists,up 164572bf-67ed-4e8e-b32d-d7a2e8da607b 2024-05-27T07:25:25.421 INFO:teuthology.orchestra.run.smithi052.stdout:pg_upmap_items 2.8 [2,11] 2024-05-27T07:25:25.421 INFO:teuthology.orchestra.run.smithi052.stdout:pg_upmap_items 2.f [8,11] 2024-05-27T07:25:25.422 INFO:teuthology.orchestra.run.smithi052.stdout:pg_upmap_items 2.10 [2,4] 2024-05-27T07:25:25.422 INFO:teuthology.orchestra.run.smithi052.stdout:pg_upmap_items 3.0 [5,9] 2024-05-27T07:25:25.422 INFO:teuthology.orchestra.run.smithi052.stdout:pg_upmap_items 3.1b [4,8] 2024-05-27T07:25:25.422 INFO:teuthology.orchestra.run.smithi052.stdout:pg_upmap_items 3.2c [4,9] 2024-05-27T07:25:25.422 INFO:teuthology.orchestra.run.smithi052.stdout:pg_upmap_items 3.3c [4,9] 2024-05-27T07:25:25.422 INFO:teuthology.orchestra.run.smithi052.stdout:pg_upmap_items 4.e [9,11] 2024-05-27T07:25:25.422 INFO:teuthology.orchestra.run.smithi052.stdout:pg_upmap_items 4.12 [5,10] 2024-05-27T07:25:25.422 INFO:teuthology.orchestra.run.smithi052.stdout:pg_upmap_items 4.19 [9,8] 2024-05-27T07:25:25.422 INFO:teuthology.orchestra.run.smithi052.stdout:pg_upmap_items 4.1f [9,8] 2024-05-27T07:25:25.422 INFO:teuthology.orchestra.run.smithi052.stdout:blocklist 172.21.15.52:6801/1631683163 expires 2024-05-28T07:14:29.544633+0000 2024-05-27T07:25:25.422 INFO:teuthology.orchestra.run.smithi052.stdout:blocklist 172.21.15.52:0/2776144659 expires 2024-05-28T07:14:29.544633+0000 2024-05-27T07:25:25.422 INFO:teuthology.orchestra.run.smithi052.stdout:blocklist 172.21.15.52:6800/1631683163 expires 2024-05-28T07:14:29.544633+0000 2024-05-27T07:25:25.423 INFO:teuthology.orchestra.run.smithi052.stdout:blocklist 172.21.15.52:0/3675529097 expires 2024-05-28T07:14:01.960550+0000 2024-05-27T07:25:25.423 INFO:teuthology.orchestra.run.smithi052.stdout:blocklist 172.21.15.52:0/727808107 expires 2024-05-28T07:14:01.960550+0000 2024-05-27T07:25:25.423 INFO:teuthology.orchestra.run.smithi052.stdout:blocklist 172.21.15.52:0/3839436833 expires 2024-05-28T07:14:29.544633+0000 2024-05-27T07:25:25.423 INFO:teuthology.orchestra.run.smithi052.stdout:blocklist 172.21.15.52:6800/587662512 expires 2024-05-28T07:14:01.960550+0000 2024-05-27T07:25:25.423 INFO:teuthology.orchestra.run.smithi052.stdout:blocklist 172.21.15.52:6801/587662512 expires 2024-05-28T07:14:01.960550+0000 2024-05-27T07:25:25.423 INFO:teuthology.orchestra.run.smithi052.stdout:blocklist 172.21.15.52:0/853867465 expires 2024-05-28T07:14:01.960550+0000 2024-05-27T07:25:25.423 INFO:teuthology.orchestra.run.smithi052.stdout:blocklist 172.21.15.52:0/3480918279 expires 2024-05-28T07:14:29.544633+0000 2024-05-27T07:25:25.424 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:25.432+0000 7f5fb7388700 1 -- 172.21.15.52:0/3888162089 >> [v2:172.21.15.52:6800/3513075805,v1:172.21.15.52:6801/3513075805] conn(0x7f5f980b2880 msgr2=0x7f5f980b4d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:25:25.424 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:25.432+0000 7f5fb7388700 1 --2- 172.21.15.52:0/3888162089 >> [v2:172.21.15.52:6800/3513075805,v1:172.21.15.52:6801/3513075805] conn(0x7f5f980b2880 0x7f5f980b4d90 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7f5fa4005950 tx=0x7f5fa40058e0 comp rx=0 tx=0).stop 2024-05-27T07:25:25.424 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:25.432+0000 7f5fb7388700 1 -- 172.21.15.52:0/3888162089 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f5fb006c280 msgr2=0x7f5fb0104730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:25:25.424 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:25.432+0000 7f5fb7388700 1 --2- 172.21.15.52:0/3888162089 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f5fb006c280 0x7f5fb0104730 secure :-1 s=READY pgs=352 cs=0 l=1 rev1=1 crypto rx=0x7f5fa00062a0 tx=0x7f5fa0004ef0 comp rx=0 tx=0).stop 2024-05-27T07:25:25.424 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:25.432+0000 7f5fb7388700 1 -- 172.21.15.52:0/3888162089 shutdown_connections 2024-05-27T07:25:25.424 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:25.432+0000 7f5fb7388700 1 --2- 172.21.15.52:0/3888162089 >> [v2:172.21.15.52:6800/3513075805,v1:172.21.15.52:6801/3513075805] conn(0x7f5f980b2880 0x7f5f980b4d90 unknown :-1 s=CLOSED pgs=218 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:25:25.424 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:25.432+0000 7f5fb7388700 1 --2- 172.21.15.52:0/3888162089 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f5fb0109fe0 0x7f5fb0103320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:25:25.424 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:25.432+0000 7f5fb7388700 1 --2- 172.21.15.52:0/3888162089 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f5fb006c280 0x7f5fb0104730 unknown :-1 s=CLOSED pgs=352 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:25:25.424 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:25.432+0000 7f5fb7388700 1 --2- 172.21.15.52:0/3888162089 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f5fb01074b0 0x7f5fb0102de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:25:25.424 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:25.432+0000 7f5fb7388700 1 -- 172.21.15.52:0/3888162089 >> 172.21.15.52:0/3888162089 conn(0x7f5fb00fe870 msgr2=0x7f5fb0106a40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:25:25.425 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:25.432+0000 7f5fb7388700 1 -- 172.21.15.52:0/3888162089 shutdown_connections 2024-05-27T07:25:25.425 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:25.432+0000 7f5fb7388700 1 -- 172.21.15.52:0/3888162089 wait complete. 2024-05-27T07:25:26.082 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:25:25 smithi052 ceph-mon[31955]: from='client.? 172.21.15.52:0/3888162089' entity='client.admin' cmd=[{"prefix": "osd dump"}]: dispatch 2024-05-27T07:25:26.101 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:25:25 smithi173 ceph-mon[37000]: from='client.? 172.21.15.52:0/3888162089' entity='client.admin' cmd=[{"prefix": "osd dump"}]: dispatch 2024-05-27T07:25:26.111 INFO:teuthology.run_tasks:Running task fs.ready... 2024-05-27T07:25:26.123 DEBUG:teuthology.orchestra.run.smithi052:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:38f9d900c83c63799fbdbe61acc9a11b0d3554a6 shell --fsid 73df2636-1bf8-11ef-bc9b-c7b262605968 -- ceph fs dump --format=json 2024-05-27T07:25:26.189 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:25:25 smithi105 ceph-mon[35977]: from='client.? 172.21.15.52:0/3888162089' entity='client.admin' cmd=[{"prefix": "osd dump"}]: dispatch 2024-05-27T07:25:26.382 INFO:teuthology.orchestra.run.smithi052.stderr:Inferring config /var/lib/ceph/73df2636-1bf8-11ef-bc9b-c7b262605968/mon.a/config 2024-05-27T07:25:27.102 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:25:26 smithi173 ceph-mon[37000]: pgmap v408: 129 pgs: 129 active+clean; 647 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 11 op/s 2024-05-27T07:25:27.189 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:25:26 smithi105 ceph-mon[35977]: pgmap v408: 129 pgs: 129 active+clean; 647 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 11 op/s 2024-05-27T07:25:27.229 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:25:26 smithi052 ceph-mon[31955]: pgmap v408: 129 pgs: 129 active+clean; 647 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 11 op/s 2024-05-27T07:25:27.972 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:27.982+0000 7fd040929700 1 -- 172.21.15.52:0/880603289 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fd03c1092f0 msgr2=0x7fd03c109700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:25:27.972 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:27.982+0000 7fd040929700 1 --2- 172.21.15.52:0/880603289 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fd03c1092f0 0x7fd03c109700 secure :-1 s=READY pgs=353 cs=0 l=1 rev1=1 crypto rx=0x7fd02c009990 tx=0x7fd02c009ca0 comp rx=0 tx=0).stop 2024-05-27T07:25:27.972 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:27.982+0000 7fd040929700 1 -- 172.21.15.52:0/880603289 shutdown_connections 2024-05-27T07:25:27.972 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:27.982+0000 7fd040929700 1 --2- 172.21.15.52:0/880603289 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fd03c1030d0 0x7fd03c111160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:25:27.972 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:27.982+0000 7fd040929700 1 --2- 172.21.15.52:0/880603289 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fd03c1092f0 0x7fd03c109700 unknown :-1 s=CLOSED pgs=353 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:25:27.973 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:27.982+0000 7fd040929700 1 --2- 172.21.15.52:0/880603289 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fd03c102780 0x7fd03c102b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:25:27.973 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:27.982+0000 7fd040929700 1 -- 172.21.15.52:0/880603289 >> 172.21.15.52:0/880603289 conn(0x7fd03c0fe4d0 msgr2=0x7fd03c100920 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:25:27.973 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:27.982+0000 7fd040929700 1 -- 172.21.15.52:0/880603289 shutdown_connections 2024-05-27T07:25:27.973 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:27.982+0000 7fd040929700 1 -- 172.21.15.52:0/880603289 wait complete. 2024-05-27T07:25:27.974 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:27.983+0000 7fd040929700 1 Processor -- start 2024-05-27T07:25:27.974 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:27.983+0000 7fd040929700 1 -- start start 2024-05-27T07:25:27.975 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:27.983+0000 7fd040929700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fd03c102780 0x7fd03c19af30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:25:27.975 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:27.983+0000 7fd040929700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fd03c1030d0 0x7fd03c19b4d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:25:27.975 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:27.983+0000 7fd040929700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fd03c1092f0 0x7fd03c19f710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:25:27.975 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:27.983+0000 7fd040929700 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x7fd03c114580 con 0x7fd03c102780 2024-05-27T07:25:27.975 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:27.983+0000 7fd040929700 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7fd03c114400 con 0x7fd03c1030d0 2024-05-27T07:25:27.975 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:27.983+0000 7fd040929700 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7fd03c114700 con 0x7fd03c1092f0 2024-05-27T07:25:27.975 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:27.983+0000 7fd03a59c700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fd03c102780 0x7fd03c19af30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:25:27.975 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:27.983+0000 7fd03a59c700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fd03c102780 0x7fd03c19af30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.52:48064/0 (socket says 172.21.15.52:48064) 2024-05-27T07:25:27.975 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:27.983+0000 7fd03a59c700 1 -- 172.21.15.52:0/2909406000 learned_addr learned my addr 172.21.15.52:0/2909406000 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-05-27T07:25:27.975 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:27.984+0000 7fd03ad9d700 1 --2- 172.21.15.52:0/2909406000 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fd03c1092f0 0x7fd03c19f710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:25:27.975 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:27.984+0000 7fd039d9b700 1 --2- 172.21.15.52:0/2909406000 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fd03c1030d0 0x7fd03c19b4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:25:27.975 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:27.984+0000 7fd03a59c700 1 -- 172.21.15.52:0/2909406000 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fd03c1092f0 msgr2=0x7fd03c19f710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:25:27.976 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:27.984+0000 7fd03a59c700 1 --2- 172.21.15.52:0/2909406000 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fd03c1092f0 0x7fd03c19f710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:25:27.976 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:27.984+0000 7fd03a59c700 1 -- 172.21.15.52:0/2909406000 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fd03c1030d0 msgr2=0x7fd03c19b4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:25:27.976 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:27.984+0000 7fd03a59c700 1 --2- 172.21.15.52:0/2909406000 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fd03c1030d0 0x7fd03c19b4d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:25:27.976 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:27.984+0000 7fd03a59c700 1 -- 172.21.15.52:0/2909406000 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd03c19fd00 con 0x7fd03c102780 2024-05-27T07:25:27.976 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:27.984+0000 7fd03a59c700 1 --2- 172.21.15.52:0/2909406000 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fd03c102780 0x7fd03c19af30 secure :-1 s=READY pgs=354 cs=0 l=1 rev1=1 crypto rx=0x7fd02c006010 tx=0x7fd02c0047b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:25:27.976 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:27.984+0000 7fd02b7fe700 1 -- 172.21.15.52:0/2909406000 <== mon.0 v2:172.21.15.52:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd02c027070 con 0x7fd03c102780 2024-05-27T07:25:27.976 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:27.984+0000 7fd02b7fe700 1 -- 172.21.15.52:0/2909406000 <== mon.0 v2:172.21.15.52:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fd02c004020 con 0x7fd03c102780 2024-05-27T07:25:27.976 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:27.984+0000 7fd02b7fe700 1 -- 172.21.15.52:0/2909406000 <== mon.0 v2:172.21.15.52:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd02c022db0 con 0x7fd03c102780 2024-05-27T07:25:27.976 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:27.984+0000 7fd040929700 1 -- 172.21.15.52:0/2909406000 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd03c19ff90 con 0x7fd03c102780 2024-05-27T07:25:27.976 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:27.985+0000 7fd040929700 1 -- 172.21.15.52:0/2909406000 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd03c1a0470 con 0x7fd03c102780 2024-05-27T07:25:27.977 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:27.985+0000 7fd040929700 1 -- 172.21.15.52:0/2909406000 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd03c10c890 con 0x7fd03c102780 2024-05-27T07:25:27.977 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:27.988+0000 7fd02b7fe700 1 -- 172.21.15.52:0/2909406000 <== mon.0 v2:172.21.15.52:3300/0 4 ==== mgrmap(e 18) ==== 149652+0+0 (secure 0 0 0) 0x7fd02c02a070 con 0x7fd03c102780 2024-05-27T07:25:27.978 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:27.988+0000 7fd02b7fe700 1 --2- 172.21.15.52:0/2909406000 >> [v2:172.21.15.52:6800/3513075805,v1:172.21.15.52:6801/3513075805] conn(0x7fd0240b2880 0x7fd0240b4d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:25:27.978 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:27.988+0000 7fd039d9b700 1 --2- 172.21.15.52:0/2909406000 >> [v2:172.21.15.52:6800/3513075805,v1:172.21.15.52:6801/3513075805] conn(0x7fd0240b2880 0x7fd0240b4d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:25:27.978 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:27.988+0000 7fd02b7fe700 1 -- 172.21.15.52:0/2909406000 <== mon.0 v2:172.21.15.52:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9121+0+0 (secure 0 0 0) 0x7fd02c0e3b10 con 0x7fd03c102780 2024-05-27T07:25:27.978 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:27.989+0000 7fd039d9b700 1 --2- 172.21.15.52:0/2909406000 >> [v2:172.21.15.52:6800/3513075805,v1:172.21.15.52:6801/3513075805] conn(0x7fd0240b2880 0x7fd0240b4d90 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7fd030005ea0 tx=0x7fd030005e30 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:25:27.982 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:27.992+0000 7fd02b7fe700 1 -- 172.21.15.52:0/2909406000 <== mon.0 v2:172.21.15.52:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191963 (secure 0 0 0) 0x7fd02c031020 con 0x7fd03c102780 2024-05-27T07:25:28.245 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:28.255+0000 7fd040929700 1 -- 172.21.15.52:0/2909406000 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fd03c0512b0 con 0x7fd03c102780 2024-05-27T07:25:28.245 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:28.256+0000 7fd02b7fe700 1 -- 172.21.15.52:0/2909406000 <== mon.0 v2:172.21.15.52:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 29 v29) ==== 94+0+11938 (secure 0 0 0) 0x7fd02c013070 con 0x7fd03c102780 2024-05-27T07:25:28.246 INFO:teuthology.orchestra.run.smithi052.stdout: 2024-05-27T07:25:28.246 INFO:teuthology.orchestra.run.smithi052.stdout:{"epoch":29,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14556,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.52:6835/2584733778","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.52:6834","nonce":2584733778},{"type":"v1","addr":"172.21.15.52:6835","nonce":2584733778}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14568,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.52:6837/675445303","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.52:6836","nonce":675445303},{"type":"v1","addr":"172.21.15.52:6837","nonce":675445303}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":29,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-05-27T07:21:50.521783+0000","modified":"2024-05-27T07:22:58.822630+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24527,"mds_1":24497,"mds_2":24475,"mds_3":24473,"mds_4":24466},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14619":{"gid":14619,"name":"g","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.52:6839/4258549559","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.52:6838","nonce":4258549559},{"type":"v1","addr":"172.21.15.52:6839","nonce":4258549559}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_14631":{"gid":14631,"name":"a","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.52:6841/2878618188","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.52:6840","nonce":2878618188},{"type":"v1","addr":"172.21.15.52:6841","nonce":2878618188}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24433":{"gid":24433,"name":"l","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.173:6833/3013589220","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6832","nonce":3013589220},{"type":"v1","addr":"172.21.15.173:6833","nonce":3013589220}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24452":{"gid":24452,"name":"k","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.105:6833/2800678412","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6832","nonce":2800678412},{"type":"v1","addr":"172.21.15.105:6833","nonce":2800678412}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24460":{"gid":24460,"name":"f","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.173:6835/1619486437","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6834","nonce":1619486437},{"type":"v1","addr":"172.21.15.173:6835","nonce":1619486437}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24466":{"gid":24466,"name":"c","rank":4,"incarnation":26,"state":"up:active","state_seq":25,"addr":"172.21.15.173:6837/1396363827","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6836","nonce":1396363827},{"type":"v1","addr":"172.21.15.173:6837","nonce":1396363827}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24473":{"gid":24473,"name":"e","rank":3,"incarnation":24,"state":"up:active","state_seq":26,"addr":"172.21.15.105:6835/3298498384","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6834","nonce":3298498384},{"type":"v1","addr":"172.21.15.105:6835","nonce":3298498384}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24475":{"gid":24475,"name":"i","rank":2,"incarnation":22,"state":"up:active","state_seq":23,"addr":"172.21.15.173:6839/564778293","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6838","nonce":564778293},{"type":"v1","addr":"172.21.15.173:6839","nonce":564778293}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24497":{"gid":24497,"name":"h","rank":1,"incarnation":20,"state":"up:active","state_seq":24,"addr":"172.21.15.105:6837/191393149","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6836","nonce":191393149},{"type":"v1","addr":"172.21.15.105:6837","nonce":191393149}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24527":{"gid":24527,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":5,"addr":"172.21.15.105:6839/500348151","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6838","nonce":500348151},{"type":"v1","addr":"172.21.15.105:6839","nonce":500348151}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24527,"qdb_cluster":[24466,24473,24475,24497,24527]},"id":1}]} 2024-05-27T07:25:28.248 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:28.258+0000 7fd040929700 1 -- 172.21.15.52:0/2909406000 >> [v2:172.21.15.52:6800/3513075805,v1:172.21.15.52:6801/3513075805] conn(0x7fd0240b2880 msgr2=0x7fd0240b4d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:25:28.248 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:28.258+0000 7fd040929700 1 --2- 172.21.15.52:0/2909406000 >> [v2:172.21.15.52:6800/3513075805,v1:172.21.15.52:6801/3513075805] conn(0x7fd0240b2880 0x7fd0240b4d90 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7fd030005ea0 tx=0x7fd030005e30 comp rx=0 tx=0).stop 2024-05-27T07:25:28.248 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:28.258+0000 7fd040929700 1 -- 172.21.15.52:0/2909406000 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fd03c102780 msgr2=0x7fd03c19af30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:25:28.248 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:28.258+0000 7fd040929700 1 --2- 172.21.15.52:0/2909406000 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fd03c102780 0x7fd03c19af30 secure :-1 s=READY pgs=354 cs=0 l=1 rev1=1 crypto rx=0x7fd02c006010 tx=0x7fd02c0047b0 comp rx=0 tx=0).stop 2024-05-27T07:25:28.248 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:28.258+0000 7fd040929700 1 -- 172.21.15.52:0/2909406000 shutdown_connections 2024-05-27T07:25:28.249 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:28.258+0000 7fd040929700 1 --2- 172.21.15.52:0/2909406000 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fd03c1092f0 0x7fd03c19f710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:25:28.249 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:28.258+0000 7fd040929700 1 --2- 172.21.15.52:0/2909406000 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fd03c102780 0x7fd03c19af30 unknown :-1 s=CLOSED pgs=354 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:25:28.249 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:28.258+0000 7fd040929700 1 --2- 172.21.15.52:0/2909406000 >> [v2:172.21.15.52:6800/3513075805,v1:172.21.15.52:6801/3513075805] conn(0x7fd0240b2880 0x7fd0240b4d90 unknown :-1 s=CLOSED pgs=219 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:25:28.249 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:28.258+0000 7fd040929700 1 --2- 172.21.15.52:0/2909406000 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fd03c1030d0 0x7fd03c19b4d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:25:28.249 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:28.258+0000 7fd040929700 1 -- 172.21.15.52:0/2909406000 >> 172.21.15.52:0/2909406000 conn(0x7fd03c0fe4d0 msgr2=0x7fd03c107080 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:25:28.249 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:28.258+0000 7fd040929700 1 -- 172.21.15.52:0/2909406000 shutdown_connections 2024-05-27T07:25:28.249 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:28.258+0000 7fd040929700 1 -- 172.21.15.52:0/2909406000 wait complete. 2024-05-27T07:25:28.249 INFO:teuthology.orchestra.run.smithi052.stderr:dumped fsmap epoch 29 2024-05-27T07:25:28.861 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:25:28 smithi052 ceph-mon[31955]: pgmap v409: 129 pgs: 129 active+clean; 647 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-05-27T07:25:28.861 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:25:28 smithi052 ceph-mon[31955]: from='client.? 172.21.15.52:0/2909406000' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-27T07:25:28.894 DEBUG:teuthology.orchestra.run.smithi052:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:38f9d900c83c63799fbdbe61acc9a11b0d3554a6 shell --fsid 73df2636-1bf8-11ef-bc9b-c7b262605968 -- ceph fs dump --format=json 2024-05-27T07:25:29.102 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:25:28 smithi173 ceph-mon[37000]: pgmap v409: 129 pgs: 129 active+clean; 647 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-05-27T07:25:29.102 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:25:28 smithi173 ceph-mon[37000]: from='client.? 172.21.15.52:0/2909406000' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-27T07:25:29.150 INFO:teuthology.orchestra.run.smithi052.stderr:Inferring config /var/lib/ceph/73df2636-1bf8-11ef-bc9b-c7b262605968/mon.a/config 2024-05-27T07:25:29.189 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:25:28 smithi105 ceph-mon[35977]: pgmap v409: 129 pgs: 129 active+clean; 647 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-05-27T07:25:29.190 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:25:28 smithi105 ceph-mon[35977]: from='client.? 172.21.15.52:0/2909406000' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-27T07:25:30.783 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:30.792+0000 7fed0f0c1700 1 -- 172.21.15.52:0/1593334456 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fed081034b0 msgr2=0x7fed081038c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:25:30.783 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:30.792+0000 7fed0f0c1700 1 --2- 172.21.15.52:0/1593334456 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fed081034b0 0x7fed081038c0 secure :-1 s=READY pgs=355 cs=0 l=1 rev1=1 crypto rx=0x7fecf80099c0 tx=0x7fecf8009cd0 comp rx=0 tx=0).stop 2024-05-27T07:25:30.783 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:30.792+0000 7fed0f0c1700 1 -- 172.21.15.52:0/1593334456 shutdown_connections 2024-05-27T07:25:30.783 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:30.792+0000 7fed0f0c1700 1 --2- 172.21.15.52:0/1593334456 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fed08108730 0x7fed08110090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:25:30.783 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:30.792+0000 7fed0f0c1700 1 --2- 172.21.15.52:0/1593334456 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fed081034b0 0x7fed081038c0 unknown :-1 s=CLOSED pgs=355 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:25:30.783 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:30.792+0000 7fed0f0c1700 1 --2- 172.21.15.52:0/1593334456 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fed08107de0 0x7fed081081f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:25:30.783 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:30.792+0000 7fed0f0c1700 1 -- 172.21.15.52:0/1593334456 >> 172.21.15.52:0/1593334456 conn(0x7fed080fede0 msgr2=0x7fed08101200 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:25:30.784 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:30.793+0000 7fed0f0c1700 1 -- 172.21.15.52:0/1593334456 shutdown_connections 2024-05-27T07:25:30.784 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:30.793+0000 7fed0f0c1700 1 -- 172.21.15.52:0/1593334456 wait complete. 2024-05-27T07:25:30.785 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:30.793+0000 7fed0f0c1700 1 Processor -- start 2024-05-27T07:25:30.786 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:30.793+0000 7fed0f0c1700 1 -- start start 2024-05-27T07:25:30.786 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:30.794+0000 7fed0f0c1700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fed081034b0 0x7fed0819ae20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:25:30.786 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:30.794+0000 7fed0f0c1700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fed08107de0 0x7fed0819b3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:25:30.786 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:30.794+0000 7fed0f0c1700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fed08108730 0x7fed0819f650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:25:30.786 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:30.794+0000 7fed0f0c1700 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x7fed08113360 con 0x7fed081034b0 2024-05-27T07:25:30.786 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:30.794+0000 7fed0f0c1700 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7fed081131e0 con 0x7fed08108730 2024-05-27T07:25:30.786 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:30.794+0000 7fed0f0c1700 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7fed081134e0 con 0x7fed08107de0 2024-05-27T07:25:30.786 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:30.794+0000 7fed0ce5d700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fed081034b0 0x7fed0819ae20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:25:30.786 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:30.794+0000 7fed0ce5d700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fed081034b0 0x7fed0819ae20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.52:36664/0 (socket says 172.21.15.52:36664) 2024-05-27T07:25:30.786 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:30.794+0000 7fed0ce5d700 1 -- 172.21.15.52:0/3653064549 learned_addr learned my addr 172.21.15.52:0/3653064549 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-05-27T07:25:30.786 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:30.794+0000 7fed07fff700 1 --2- 172.21.15.52:0/3653064549 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fed08107de0 0x7fed0819b3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:25:30.787 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:30.794+0000 7fed0d65e700 1 --2- 172.21.15.52:0/3653064549 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fed08108730 0x7fed0819f650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:25:30.787 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:30.794+0000 7fed0ce5d700 1 -- 172.21.15.52:0/3653064549 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fed08107de0 msgr2=0x7fed0819b3c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:25:30.787 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:30.794+0000 7fed0ce5d700 1 --2- 172.21.15.52:0/3653064549 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fed08107de0 0x7fed0819b3c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:25:30.787 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:30.794+0000 7fed0ce5d700 1 -- 172.21.15.52:0/3653064549 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fed08108730 msgr2=0x7fed0819f650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:25:30.787 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:30.794+0000 7fed0ce5d700 1 --2- 172.21.15.52:0/3653064549 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fed08108730 0x7fed0819f650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:25:30.787 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:30.794+0000 7fed0ce5d700 1 -- 172.21.15.52:0/3653064549 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fed0819fc40 con 0x7fed081034b0 2024-05-27T07:25:30.787 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:30.794+0000 7fed07fff700 1 --2- 172.21.15.52:0/3653064549 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fed08107de0 0x7fed0819b3c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-27T07:25:30.787 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:30.794+0000 7fed0d65e700 1 --2- 172.21.15.52:0/3653064549 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fed08108730 0x7fed0819f650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-27T07:25:30.787 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:30.795+0000 7fed0ce5d700 1 --2- 172.21.15.52:0/3653064549 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fed081034b0 0x7fed0819ae20 secure :-1 s=READY pgs=356 cs=0 l=1 rev1=1 crypto rx=0x7fecf800b580 tx=0x7fecf80047b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:25:30.787 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:30.795+0000 7fed05ffb700 1 -- 172.21.15.52:0/3653064549 <== mon.0 v2:172.21.15.52:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fecf8027070 con 0x7fed081034b0 2024-05-27T07:25:30.787 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:30.795+0000 7fed0f0c1700 1 -- 172.21.15.52:0/3653064549 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fed0819fed0 con 0x7fed081034b0 2024-05-27T07:25:30.787 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:30.795+0000 7fed05ffb700 1 -- 172.21.15.52:0/3653064549 <== mon.0 v2:172.21.15.52:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fecf8004020 con 0x7fed081034b0 2024-05-27T07:25:30.788 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:30.795+0000 7fed05ffb700 1 -- 172.21.15.52:0/3653064549 <== mon.0 v2:172.21.15.52:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fecf8022e30 con 0x7fed081034b0 2024-05-27T07:25:30.788 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:30.795+0000 7fed0f0c1700 1 -- 172.21.15.52:0/3653064549 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fed081a03b0 con 0x7fed081034b0 2024-05-27T07:25:30.788 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:30.796+0000 7fed0f0c1700 1 -- 172.21.15.52:0/3653064549 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fed080512b0 con 0x7fed081034b0 2024-05-27T07:25:30.788 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:30.798+0000 7fed05ffb700 1 -- 172.21.15.52:0/3653064549 <== mon.0 v2:172.21.15.52:3300/0 4 ==== mgrmap(e 18) ==== 149652+0+0 (secure 0 0 0) 0x7fecf8022590 con 0x7fed081034b0 2024-05-27T07:25:30.788 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:30.798+0000 7fed05ffb700 1 --2- 172.21.15.52:0/3653064549 >> [v2:172.21.15.52:6800/3513075805,v1:172.21.15.52:6801/3513075805] conn(0x7fecf00b2880 0x7fecf00b4d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:25:30.789 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:30.799+0000 7fed05ffb700 1 -- 172.21.15.52:0/3653064549 <== mon.0 v2:172.21.15.52:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9121+0+0 (secure 0 0 0) 0x7fecf80e3670 con 0x7fed081034b0 2024-05-27T07:25:30.789 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:30.799+0000 7fed07fff700 1 --2- 172.21.15.52:0/3653064549 >> [v2:172.21.15.52:6800/3513075805,v1:172.21.15.52:6801/3513075805] conn(0x7fecf00b2880 0x7fecf00b4d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:25:30.789 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:30.799+0000 7fed07fff700 1 --2- 172.21.15.52:0/3653064549 >> [v2:172.21.15.52:6800/3513075805,v1:172.21.15.52:6801/3513075805] conn(0x7fecf00b2880 0x7fecf00b4d90 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7fecfc005950 tx=0x7fecfc0058e0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:25:30.791 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:30.802+0000 7fed05ffb700 1 -- 172.21.15.52:0/3653064549 <== mon.0 v2:172.21.15.52:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191963 (secure 0 0 0) 0x7fecf8030080 con 0x7fed081034b0 2024-05-27T07:25:31.057 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:31.066+0000 7fed0f0c1700 1 -- 172.21.15.52:0/3653064549 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fed0802e5e0 con 0x7fed081034b0 2024-05-27T07:25:31.058 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:25:30 smithi052 ceph-mon[31955]: pgmap v410: 129 pgs: 129 active+clean; 647 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 9 op/s 2024-05-27T07:25:31.058 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:31.067+0000 7fed05ffb700 1 -- 172.21.15.52:0/3653064549 <== mon.0 v2:172.21.15.52:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 29 v29) ==== 94+0+11938 (secure 0 0 0) 0x7fecf8013070 con 0x7fed081034b0 2024-05-27T07:25:31.058 INFO:teuthology.orchestra.run.smithi052.stdout: 2024-05-27T07:25:31.058 INFO:teuthology.orchestra.run.smithi052.stdout:{"epoch":29,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14556,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.52:6835/2584733778","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.52:6834","nonce":2584733778},{"type":"v1","addr":"172.21.15.52:6835","nonce":2584733778}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14568,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.52:6837/675445303","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.52:6836","nonce":675445303},{"type":"v1","addr":"172.21.15.52:6837","nonce":675445303}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":29,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-05-27T07:21:50.521783+0000","modified":"2024-05-27T07:22:58.822630+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24527,"mds_1":24497,"mds_2":24475,"mds_3":24473,"mds_4":24466},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14619":{"gid":14619,"name":"g","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.52:6839/4258549559","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.52:6838","nonce":4258549559},{"type":"v1","addr":"172.21.15.52:6839","nonce":4258549559}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_14631":{"gid":14631,"name":"a","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.52:6841/2878618188","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.52:6840","nonce":2878618188},{"type":"v1","addr":"172.21.15.52:6841","nonce":2878618188}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24433":{"gid":24433,"name":"l","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.173:6833/3013589220","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6832","nonce":3013589220},{"type":"v1","addr":"172.21.15.173:6833","nonce":3013589220}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24452":{"gid":24452,"name":"k","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.105:6833/2800678412","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6832","nonce":2800678412},{"type":"v1","addr":"172.21.15.105:6833","nonce":2800678412}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24460":{"gid":24460,"name":"f","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.173:6835/1619486437","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6834","nonce":1619486437},{"type":"v1","addr":"172.21.15.173:6835","nonce":1619486437}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24466":{"gid":24466,"name":"c","rank":4,"incarnation":26,"state":"up:active","state_seq":25,"addr":"172.21.15.173:6837/1396363827","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6836","nonce":1396363827},{"type":"v1","addr":"172.21.15.173:6837","nonce":1396363827}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24473":{"gid":24473,"name":"e","rank":3,"incarnation":24,"state":"up:active","state_seq":26,"addr":"172.21.15.105:6835/3298498384","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6834","nonce":3298498384},{"type":"v1","addr":"172.21.15.105:6835","nonce":3298498384}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24475":{"gid":24475,"name":"i","rank":2,"incarnation":22,"state":"up:active","state_seq":23,"addr":"172.21.15.173:6839/564778293","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6838","nonce":564778293},{"type":"v1","addr":"172.21.15.173:6839","nonce":564778293}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24497":{"gid":24497,"name":"h","rank":1,"incarnation":20,"state":"up:active","state_seq":24,"addr":"172.21.15.105:6837/191393149","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6836","nonce":191393149},{"type":"v1","addr":"172.21.15.105:6837","nonce":191393149}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24527":{"gid":24527,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":5,"addr":"172.21.15.105:6839/500348151","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6838","nonce":500348151},{"type":"v1","addr":"172.21.15.105:6839","nonce":500348151}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24527,"qdb_cluster":[24466,24473,24475,24497,24527]},"id":1}]} 2024-05-27T07:25:31.060 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:31.070+0000 7fed0f0c1700 1 -- 172.21.15.52:0/3653064549 >> [v2:172.21.15.52:6800/3513075805,v1:172.21.15.52:6801/3513075805] conn(0x7fecf00b2880 msgr2=0x7fecf00b4d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:25:31.060 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:31.070+0000 7fed0f0c1700 1 --2- 172.21.15.52:0/3653064549 >> [v2:172.21.15.52:6800/3513075805,v1:172.21.15.52:6801/3513075805] conn(0x7fecf00b2880 0x7fecf00b4d90 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7fecfc005950 tx=0x7fecfc0058e0 comp rx=0 tx=0).stop 2024-05-27T07:25:31.060 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:31.070+0000 7fed0f0c1700 1 -- 172.21.15.52:0/3653064549 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fed081034b0 msgr2=0x7fed0819ae20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:25:31.060 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:31.070+0000 7fed0f0c1700 1 --2- 172.21.15.52:0/3653064549 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fed081034b0 0x7fed0819ae20 secure :-1 s=READY pgs=356 cs=0 l=1 rev1=1 crypto rx=0x7fecf800b580 tx=0x7fecf80047b0 comp rx=0 tx=0).stop 2024-05-27T07:25:31.060 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:31.070+0000 7fed0f0c1700 1 -- 172.21.15.52:0/3653064549 shutdown_connections 2024-05-27T07:25:31.060 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:31.070+0000 7fed0f0c1700 1 --2- 172.21.15.52:0/3653064549 >> [v2:172.21.15.52:6800/3513075805,v1:172.21.15.52:6801/3513075805] conn(0x7fecf00b2880 0x7fecf00b4d90 unknown :-1 s=CLOSED pgs=220 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:25:31.060 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:31.070+0000 7fed0f0c1700 1 --2- 172.21.15.52:0/3653064549 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fed08108730 0x7fed0819f650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:25:31.060 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:31.070+0000 7fed0f0c1700 1 --2- 172.21.15.52:0/3653064549 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fed081034b0 0x7fed0819ae20 unknown :-1 s=CLOSED pgs=356 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:25:31.060 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:31.070+0000 7fed0f0c1700 1 --2- 172.21.15.52:0/3653064549 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fed08107de0 0x7fed0819b3c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:25:31.060 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:31.070+0000 7fed0f0c1700 1 -- 172.21.15.52:0/3653064549 >> 172.21.15.52:0/3653064549 conn(0x7fed080fede0 msgr2=0x7fed08076130 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:25:31.061 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:31.070+0000 7fed0f0c1700 1 -- 172.21.15.52:0/3653064549 shutdown_connections 2024-05-27T07:25:31.061 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:31.070+0000 7fed0f0c1700 1 -- 172.21.15.52:0/3653064549 wait complete. 2024-05-27T07:25:31.061 INFO:teuthology.orchestra.run.smithi052.stderr:dumped fsmap epoch 29 2024-05-27T07:25:31.101 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:25:30 smithi173 ceph-mon[37000]: pgmap v410: 129 pgs: 129 active+clean; 647 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 9 op/s 2024-05-27T07:25:31.189 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:25:30 smithi105 ceph-mon[35977]: pgmap v410: 129 pgs: 129 active+clean; 647 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 9 op/s 2024-05-27T07:25:31.718 DEBUG:teuthology.orchestra.run.smithi052:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:38f9d900c83c63799fbdbe61acc9a11b0d3554a6 shell --fsid 73df2636-1bf8-11ef-bc9b-c7b262605968 -- ceph osd dump --format=json 2024-05-27T07:25:31.972 INFO:teuthology.orchestra.run.smithi052.stderr:Inferring config /var/lib/ceph/73df2636-1bf8-11ef-bc9b-c7b262605968/mon.a/config 2024-05-27T07:25:31.979 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:25:31 smithi052 ceph-mon[31955]: from='client.? 172.21.15.52:0/3653064549' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-27T07:25:32.101 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:25:31 smithi173 ceph-mon[37000]: from='client.? 172.21.15.52:0/3653064549' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-27T07:25:32.190 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:25:31 smithi105 ceph-mon[35977]: from='client.? 172.21.15.52:0/3653064549' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-27T07:25:33.097 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:25:32 smithi052 ceph-mon[31955]: pgmap v411: 129 pgs: 129 active+clean; 647 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-05-27T07:25:33.097 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:25:32 smithi052 ceph-mon[31955]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-27T07:25:33.101 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:25:32 smithi173 ceph-mon[37000]: pgmap v411: 129 pgs: 129 active+clean; 647 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-05-27T07:25:33.101 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:25:32 smithi173 ceph-mon[37000]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-27T07:25:33.189 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:25:32 smithi105 ceph-mon[35977]: pgmap v411: 129 pgs: 129 active+clean; 647 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-05-27T07:25:33.190 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:25:32 smithi105 ceph-mon[35977]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-27T07:25:33.663 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:33.672+0000 7ff669adc700 1 -- 172.21.15.52:0/1047867816 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff664109860 msgr2=0x7ff6641111c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:25:33.663 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:33.672+0000 7ff669adc700 1 --2- 172.21.15.52:0/1047867816 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff664109860 0x7ff6641111c0 secure :-1 s=READY pgs=357 cs=0 l=1 rev1=1 crypto rx=0x7ff65c00b0d0 tx=0x7ff65c00b3e0 comp rx=0 tx=0).stop 2024-05-27T07:25:33.663 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:33.672+0000 7ff669adc700 1 -- 172.21.15.52:0/1047867816 shutdown_connections 2024-05-27T07:25:33.663 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:33.672+0000 7ff669adc700 1 --2- 172.21.15.52:0/1047867816 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff664109860 0x7ff6641111c0 unknown :-1 s=CLOSED pgs=357 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:25:33.663 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:33.672+0000 7ff669adc700 1 --2- 172.21.15.52:0/1047867816 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7ff6641045e0 0x7ff6641049f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:25:33.663 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:33.672+0000 7ff669adc700 1 --2- 172.21.15.52:0/1047867816 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7ff664108f10 0x7ff664109320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:25:33.663 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:33.672+0000 7ff669adc700 1 -- 172.21.15.52:0/1047867816 >> 172.21.15.52:0/1047867816 conn(0x7ff6640fffd0 msgr2=0x7ff6641023f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:25:33.663 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:33.672+0000 7ff669adc700 1 -- 172.21.15.52:0/1047867816 shutdown_connections 2024-05-27T07:25:33.663 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:33.672+0000 7ff669adc700 1 -- 172.21.15.52:0/1047867816 wait complete. 2024-05-27T07:25:33.663 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:33.673+0000 7ff669adc700 1 Processor -- start 2024-05-27T07:25:33.664 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:33.673+0000 7ff669adc700 1 -- start start 2024-05-27T07:25:33.665 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:33.674+0000 7ff669adc700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7ff6641045e0 0x7ff664075ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:25:33.665 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:33.674+0000 7ff669adc700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff664108f10 0x7ff664075ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:25:33.665 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:33.674+0000 7ff669adc700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7ff664109860 0x7ff664076530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:25:33.665 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:33.674+0000 7ff669adc700 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x7ff6641144b0 con 0x7ff664108f10 2024-05-27T07:25:33.665 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:33.674+0000 7ff669adc700 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7ff664114330 con 0x7ff6641045e0 2024-05-27T07:25:33.665 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:33.674+0000 7ff669adc700 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7ff664114630 con 0x7ff664109860 2024-05-27T07:25:33.665 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:33.674+0000 7ff6637fe700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7ff6641045e0 0x7ff664075ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:25:33.665 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:33.674+0000 7ff6637fe700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7ff6641045e0 0x7ff664075ab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.52:43528/0 (socket says 172.21.15.52:43528) 2024-05-27T07:25:33.665 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:33.674+0000 7ff662ffd700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff664108f10 0x7ff664075ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:25:33.665 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:33.674+0000 7ff6637fe700 1 -- 172.21.15.52:0/2465335471 learned_addr learned my addr 172.21.15.52:0/2465335471 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-05-27T07:25:33.665 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:33.674+0000 7ff663fff700 1 --2- 172.21.15.52:0/2465335471 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7ff664109860 0x7ff664076530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:25:33.665 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:33.674+0000 7ff6637fe700 1 -- 172.21.15.52:0/2465335471 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7ff664109860 msgr2=0x7ff664076530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:25:33.666 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:33.674+0000 7ff6637fe700 1 --2- 172.21.15.52:0/2465335471 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7ff664109860 0x7ff664076530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:25:33.666 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:33.674+0000 7ff6637fe700 1 -- 172.21.15.52:0/2465335471 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff664108f10 msgr2=0x7ff664075ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:25:33.666 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:33.674+0000 7ff6637fe700 1 --2- 172.21.15.52:0/2465335471 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff664108f10 0x7ff664075ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:25:33.666 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:33.674+0000 7ff6637fe700 1 -- 172.21.15.52:0/2465335471 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff6641a9160 con 0x7ff6641045e0 2024-05-27T07:25:33.666 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:33.675+0000 7ff6637fe700 1 --2- 172.21.15.52:0/2465335471 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7ff6641045e0 0x7ff664075ab0 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7ff65400d980 tx=0x7ff65400dc90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:25:33.666 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:33.675+0000 7ff660ff9700 1 -- 172.21.15.52:0/2465335471 <== mon.2 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff654017070 con 0x7ff6641045e0 2024-05-27T07:25:33.666 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:33.675+0000 7ff660ff9700 1 -- 172.21.15.52:0/2465335471 <== mon.2 v2:172.21.15.105:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ff654005020 con 0x7ff6641045e0 2024-05-27T07:25:33.666 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:33.675+0000 7ff660ff9700 1 -- 172.21.15.52:0/2465335471 <== mon.2 v2:172.21.15.105:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff65400fa10 con 0x7ff6641045e0 2024-05-27T07:25:33.667 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:33.675+0000 7ff669adc700 1 -- 172.21.15.52:0/2465335471 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff6641a9450 con 0x7ff6641045e0 2024-05-27T07:25:33.667 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:33.675+0000 7ff669adc700 1 -- 172.21.15.52:0/2465335471 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff6641a9940 con 0x7ff6641045e0 2024-05-27T07:25:33.667 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:33.676+0000 7ff669adc700 1 -- 172.21.15.52:0/2465335471 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff6640512b0 con 0x7ff6641045e0 2024-05-27T07:25:33.668 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:33.679+0000 7ff660ff9700 1 -- 172.21.15.52:0/2465335471 <== mon.2 v2:172.21.15.105:3300/0 4 ==== mgrmap(e 18) ==== 149652+0+0 (secure 0 0 0) 0x7ff654004a30 con 0x7ff6641045e0 2024-05-27T07:25:33.669 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:33.680+0000 7ff660ff9700 1 --2- 172.21.15.52:0/2465335471 >> [v2:172.21.15.52:6800/3513075805,v1:172.21.15.52:6801/3513075805] conn(0x7ff64c0b2900 0x7ff64c0b4e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:25:33.669 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:33.680+0000 7ff660ff9700 1 -- 172.21.15.52:0/2465335471 <== mon.2 v2:172.21.15.105:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9121+0+0 (secure 0 0 0) 0x7ff6540e2900 con 0x7ff6641045e0 2024-05-27T07:25:33.670 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:33.680+0000 7ff662ffd700 1 --2- 172.21.15.52:0/2465335471 >> [v2:172.21.15.52:6800/3513075805,v1:172.21.15.52:6801/3513075805] conn(0x7ff64c0b2900 0x7ff64c0b4e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:25:33.670 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:33.680+0000 7ff662ffd700 1 --2- 172.21.15.52:0/2465335471 >> [v2:172.21.15.52:6800/3513075805,v1:172.21.15.52:6801/3513075805] conn(0x7ff64c0b2900 0x7ff64c0b4e10 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7ff664108920 tx=0x7ff658009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:25:33.672 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:33.682+0000 7ff660ff9700 1 -- 172.21.15.52:0/2465335471 <== mon.2 v2:172.21.15.105:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191963 (secure 0 0 0) 0x7ff654020020 con 0x7ff6641045e0 2024-05-27T07:25:33.932 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:33.942+0000 7ff669adc700 1 -- 172.21.15.52:0/2465335471 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7ff664076fe0 con 0x7ff6641045e0 2024-05-27T07:25:33.934 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:33.944+0000 7ff660ff9700 1 -- 172.21.15.52:0/2465335471 <== mon.2 v2:172.21.15.105:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v90) ==== 74+0+24701 (secure 0 0 0) 0x7ff6540aa2c0 con 0x7ff6641045e0 2024-05-27T07:25:33.934 INFO:teuthology.orchestra.run.smithi052.stdout: 2024-05-27T07:25:33.935 INFO:teuthology.orchestra.run.smithi052.stdout:{"epoch":90,"fsid":"73df2636-1bf8-11ef-bc9b-c7b262605968","created":"2024-05-27T07:13:34.738032+0000","modified":"2024-05-27T07:23:30.969990+0000","last_up_change":"2024-05-27T07:20:45.777597+0000","last_in_change":"2024-05-27T07:20:25.314823+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-05-27T07:16:57.644034+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-05-27T07:21:43.917748+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"79","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-05-27T07:21:47.341876+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"79","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-05-27T07:21:57.707611+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"86","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"347b8a89-9dbe-4848-83b0-fd0e9374f33d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":82,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.52:6802","nonce":1626508899},{"type":"v1","addr":"172.21.15.52:6803","nonce":1626508899}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.52:6804","nonce":1626508899},{"type":"v1","addr":"172.21.15.52:6805","nonce":1626508899}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.52:6808","nonce":1626508899},{"type":"v1","addr":"172.21.15.52:6809","nonce":1626508899}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.52:6806","nonce":1626508899},{"type":"v1","addr":"172.21.15.52:6807","nonce":1626508899}]},"public_addr":"172.21.15.52:6803/1626508899","cluster_addr":"172.21.15.52:6805/1626508899","heartbeat_back_addr":"172.21.15.52:6809/1626508899","heartbeat_front_addr":"172.21.15.52:6807/1626508899","state":["exists","up"]},{"osd":1,"uuid":"7a3623d4-e087-40cb-9336-d4a5c52203d4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6800","nonce":2900093195},{"type":"v1","addr":"172.21.15.105:6801","nonce":2900093195}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6802","nonce":2900093195},{"type":"v1","addr":"172.21.15.105:6803","nonce":2900093195}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6806","nonce":2900093195},{"type":"v1","addr":"172.21.15.105:6807","nonce":2900093195}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6804","nonce":2900093195},{"type":"v1","addr":"172.21.15.105:6805","nonce":2900093195}]},"public_addr":"172.21.15.105:6801/2900093195","cluster_addr":"172.21.15.105:6803/2900093195","heartbeat_back_addr":"172.21.15.105:6807/2900093195","heartbeat_front_addr":"172.21.15.105:6805/2900093195","state":["exists","up"]},{"osd":2,"uuid":"41a9df5b-882a-47da-bb26-7f4d26ccf6c1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6800","nonce":2251189314},{"type":"v1","addr":"172.21.15.173:6801","nonce":2251189314}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6802","nonce":2251189314},{"type":"v1","addr":"172.21.15.173:6803","nonce":2251189314}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6806","nonce":2251189314},{"type":"v1","addr":"172.21.15.173:6807","nonce":2251189314}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6804","nonce":2251189314},{"type":"v1","addr":"172.21.15.173:6805","nonce":2251189314}]},"public_addr":"172.21.15.173:6801/2251189314","cluster_addr":"172.21.15.173:6803/2251189314","heartbeat_back_addr":"172.21.15.173:6807/2251189314","heartbeat_front_addr":"172.21.15.173:6805/2251189314","state":["exists","up"]},{"osd":3,"uuid":"f1245439-42a5-46c5-a998-e0a78f9df3fb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":26,"up_thru":82,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.52:6810","nonce":57920794},{"type":"v1","addr":"172.21.15.52:6811","nonce":57920794}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.52:6812","nonce":57920794},{"type":"v1","addr":"172.21.15.52:6813","nonce":57920794}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.52:6816","nonce":57920794},{"type":"v1","addr":"172.21.15.52:6817","nonce":57920794}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.52:6814","nonce":57920794},{"type":"v1","addr":"172.21.15.52:6815","nonce":57920794}]},"public_addr":"172.21.15.52:6811/57920794","cluster_addr":"172.21.15.52:6813/57920794","heartbeat_back_addr":"172.21.15.52:6817/57920794","heartbeat_front_addr":"172.21.15.52:6815/57920794","state":["exists","up"]},{"osd":4,"uuid":"2592d1cc-1726-4359-83d8-a2e6314b54ec","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":32,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6808","nonce":3441134818},{"type":"v1","addr":"172.21.15.105:6809","nonce":3441134818}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6810","nonce":3441134818},{"type":"v1","addr":"172.21.15.105:6811","nonce":3441134818}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6814","nonce":3441134818},{"type":"v1","addr":"172.21.15.105:6815","nonce":3441134818}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6812","nonce":3441134818},{"type":"v1","addr":"172.21.15.105:6813","nonce":3441134818}]},"public_addr":"172.21.15.105:6809/3441134818","cluster_addr":"172.21.15.105:6811/3441134818","heartbeat_back_addr":"172.21.15.105:6815/3441134818","heartbeat_front_addr":"172.21.15.105:6813/3441134818","state":["exists","up"]},{"osd":5,"uuid":"27e98ee2-4a92-4972-b2a3-959d0ed90981","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":37,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6808","nonce":2989239729},{"type":"v1","addr":"172.21.15.173:6809","nonce":2989239729}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6810","nonce":2989239729},{"type":"v1","addr":"172.21.15.173:6811","nonce":2989239729}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6814","nonce":2989239729},{"type":"v1","addr":"172.21.15.173:6815","nonce":2989239729}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6812","nonce":2989239729},{"type":"v1","addr":"172.21.15.173:6813","nonce":2989239729}]},"public_addr":"172.21.15.173:6809/2989239729","cluster_addr":"172.21.15.173:6811/2989239729","heartbeat_back_addr":"172.21.15.173:6815/2989239729","heartbeat_front_addr":"172.21.15.173:6813/2989239729","state":["exists","up"]},{"osd":6,"uuid":"2b0ec0cb-f438-4b2c-8642-0778ed374e9a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":43,"up_thru":82,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.52:6818","nonce":4244423065},{"type":"v1","addr":"172.21.15.52:6819","nonce":4244423065}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.52:6820","nonce":4244423065},{"type":"v1","addr":"172.21.15.52:6821","nonce":4244423065}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.52:6824","nonce":4244423065},{"type":"v1","addr":"172.21.15.52:6825","nonce":4244423065}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.52:6822","nonce":4244423065},{"type":"v1","addr":"172.21.15.52:6823","nonce":4244423065}]},"public_addr":"172.21.15.52:6819/4244423065","cluster_addr":"172.21.15.52:6821/4244423065","heartbeat_back_addr":"172.21.15.52:6825/4244423065","heartbeat_front_addr":"172.21.15.52:6823/4244423065","state":["exists","up"]},{"osd":7,"uuid":"cb2a7a6b-8015-43ea-865a-be144e45c51a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":48,"up_thru":82,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6816","nonce":2063611552},{"type":"v1","addr":"172.21.15.105:6817","nonce":2063611552}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6818","nonce":2063611552},{"type":"v1","addr":"172.21.15.105:6819","nonce":2063611552}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6822","nonce":2063611552},{"type":"v1","addr":"172.21.15.105:6823","nonce":2063611552}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6820","nonce":2063611552},{"type":"v1","addr":"172.21.15.105:6821","nonce":2063611552}]},"public_addr":"172.21.15.105:6817/2063611552","cluster_addr":"172.21.15.105:6819/2063611552","heartbeat_back_addr":"172.21.15.105:6823/2063611552","heartbeat_front_addr":"172.21.15.105:6821/2063611552","state":["exists","up"]},{"osd":8,"uuid":"c2fc565d-a0b9-4396-a209-bd884cddcfe3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":54,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6816","nonce":4131399852},{"type":"v1","addr":"172.21.15.173:6817","nonce":4131399852}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6818","nonce":4131399852},{"type":"v1","addr":"172.21.15.173:6819","nonce":4131399852}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6822","nonce":4131399852},{"type":"v1","addr":"172.21.15.173:6823","nonce":4131399852}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6820","nonce":4131399852},{"type":"v1","addr":"172.21.15.173:6821","nonce":4131399852}]},"public_addr":"172.21.15.173:6817/4131399852","cluster_addr":"172.21.15.173:6819/4131399852","heartbeat_back_addr":"172.21.15.173:6823/4131399852","heartbeat_front_addr":"172.21.15.173:6821/4131399852","state":["exists","up"]},{"osd":9,"uuid":"78fa7f17-1dad-4262-be40-31fa82e75520","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":60,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.52:6826","nonce":479119778},{"type":"v1","addr":"172.21.15.52:6827","nonce":479119778}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.52:6828","nonce":479119778},{"type":"v1","addr":"172.21.15.52:6829","nonce":479119778}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.52:6832","nonce":479119778},{"type":"v1","addr":"172.21.15.52:6833","nonce":479119778}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.52:6830","nonce":479119778},{"type":"v1","addr":"172.21.15.52:6831","nonce":479119778}]},"public_addr":"172.21.15.52:6827/479119778","cluster_addr":"172.21.15.52:6829/479119778","heartbeat_back_addr":"172.21.15.52:6833/479119778","heartbeat_front_addr":"172.21.15.52:6831/479119778","state":["exists","up"]},{"osd":10,"uuid":"e6f8e753-e15c-45b5-8ee9-d50f384559f9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":65,"up_thru":76,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6824","nonce":4032386628},{"type":"v1","addr":"172.21.15.105:6825","nonce":4032386628}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6826","nonce":4032386628},{"type":"v1","addr":"172.21.15.105:6827","nonce":4032386628}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6830","nonce":4032386628},{"type":"v1","addr":"172.21.15.105:6831","nonce":4032386628}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6828","nonce":4032386628},{"type":"v1","addr":"172.21.15.105:6829","nonce":4032386628}]},"public_addr":"172.21.15.105:6825/4032386628","cluster_addr":"172.21.15.105:6827/4032386628","heartbeat_back_addr":"172.21.15.105:6831/4032386628","heartbeat_front_addr":"172.21.15.105:6829/4032386628","state":["exists","up"]},{"osd":11,"uuid":"164572bf-67ed-4e8e-b32d-d7a2e8da607b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":71,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6824","nonce":1816308401},{"type":"v1","addr":"172.21.15.173:6825","nonce":1816308401}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6826","nonce":1816308401},{"type":"v1","addr":"172.21.15.173:6827","nonce":1816308401}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6830","nonce":1816308401},{"type":"v1","addr":"172.21.15.173:6831","nonce":1816308401}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6828","nonce":1816308401},{"type":"v1","addr":"172.21.15.173:6829","nonce":1816308401}]},"public_addr":"172.21.15.173:6825/1816308401","cluster_addr":"172.21.15.173:6827/1816308401","heartbeat_back_addr":"172.21.15.173:6831/1816308401","heartbeat_front_addr":"172.21.15.173:6829/1816308401","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-27T07:16:06.279220+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-27T07:16:29.668167+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-27T07:16:53.903808+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-27T07:17:18.787372+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-27T07:17:43.480784+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-27T07:18:08.816030+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-27T07:18:34.076148+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-27T07:19:00.283468+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-27T07:19:25.403023+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-27T07:19:52.139173+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-27T07:20:17.530304+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-27T07:20:43.665632+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.8","mappings":[{"from":2,"to":11}]},{"pgid":"2.f","mappings":[{"from":8,"to":11}]},{"pgid":"2.10","mappings":[{"from":2,"to":4}]},{"pgid":"3.0","mappings":[{"from":5,"to":9}]},{"pgid":"3.1b","mappings":[{"from":4,"to":8}]},{"pgid":"3.2c","mappings":[{"from":4,"to":9}]},{"pgid":"3.3c","mappings":[{"from":4,"to":9}]},{"pgid":"4.e","mappings":[{"from":9,"to":11}]},{"pgid":"4.12","mappings":[{"from":5,"to":10}]},{"pgid":"4.19","mappings":[{"from":9,"to":8}]},{"pgid":"4.1f","mappings":[{"from":9,"to":8}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.52:6801/1631683163":"2024-05-28T07:14:29.544633+0000","172.21.15.52:0/2776144659":"2024-05-28T07:14:29.544633+0000","172.21.15.52:6800/1631683163":"2024-05-28T07:14:29.544633+0000","172.21.15.52:0/3675529097":"2024-05-28T07:14:01.960550+0000","172.21.15.52:0/727808107":"2024-05-28T07:14:01.960550+0000","172.21.15.52:0/3839436833":"2024-05-28T07:14:29.544633+0000","172.21.15.52:6800/587662512":"2024-05-28T07:14:01.960550+0000","172.21.15.52:6801/587662512":"2024-05-28T07:14:01.960550+0000","172.21.15.52:0/853867465":"2024-05-28T07:14:01.960550+0000","172.21.15.52:0/3480918279":"2024-05-28T07:14:29.544633+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-05-27T07:25:33.936 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:33.946+0000 7ff669adc700 1 -- 172.21.15.52:0/2465335471 >> [v2:172.21.15.52:6800/3513075805,v1:172.21.15.52:6801/3513075805] conn(0x7ff64c0b2900 msgr2=0x7ff64c0b4e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:25:33.936 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:33.946+0000 7ff669adc700 1 --2- 172.21.15.52:0/2465335471 >> [v2:172.21.15.52:6800/3513075805,v1:172.21.15.52:6801/3513075805] conn(0x7ff64c0b2900 0x7ff64c0b4e10 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7ff664108920 tx=0x7ff658009260 comp rx=0 tx=0).stop 2024-05-27T07:25:33.936 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:33.946+0000 7ff669adc700 1 -- 172.21.15.52:0/2465335471 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7ff6641045e0 msgr2=0x7ff664075ab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:25:33.937 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:33.946+0000 7ff669adc700 1 --2- 172.21.15.52:0/2465335471 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7ff6641045e0 0x7ff664075ab0 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7ff65400d980 tx=0x7ff65400dc90 comp rx=0 tx=0).stop 2024-05-27T07:25:33.937 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:33.946+0000 7ff669adc700 1 -- 172.21.15.52:0/2465335471 shutdown_connections 2024-05-27T07:25:33.937 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:33.946+0000 7ff669adc700 1 --2- 172.21.15.52:0/2465335471 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7ff664109860 0x7ff664076530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:25:33.937 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:33.946+0000 7ff669adc700 1 --2- 172.21.15.52:0/2465335471 >> [v2:172.21.15.52:6800/3513075805,v1:172.21.15.52:6801/3513075805] conn(0x7ff64c0b2900 0x7ff64c0b4e10 unknown :-1 s=CLOSED pgs=221 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:25:33.937 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:33.946+0000 7ff669adc700 1 --2- 172.21.15.52:0/2465335471 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7ff6641045e0 0x7ff664075ab0 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:25:33.937 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:33.946+0000 7ff669adc700 1 --2- 172.21.15.52:0/2465335471 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff664108f10 0x7ff664075ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:25:33.937 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:33.946+0000 7ff669adc700 1 -- 172.21.15.52:0/2465335471 >> 172.21.15.52:0/2465335471 conn(0x7ff6640fffd0 msgr2=0x7ff664106ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:25:33.937 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:33.947+0000 7ff669adc700 1 -- 172.21.15.52:0/2465335471 shutdown_connections 2024-05-27T07:25:33.937 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:33.947+0000 7ff669adc700 1 -- 172.21.15.52:0/2465335471 wait complete. 2024-05-27T07:25:34.229 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:25:34 smithi052 ceph-mon[31955]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' 2024-05-27T07:25:34.230 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:25:34 smithi052 ceph-mon[31955]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' 2024-05-27T07:25:34.230 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:25:34 smithi052 ceph-mon[31955]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' 2024-05-27T07:25:34.230 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:25:34 smithi052 ceph-mon[31955]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' 2024-05-27T07:25:34.230 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:25:34 smithi052 ceph-mon[31955]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' 2024-05-27T07:25:34.230 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:25:34 smithi052 ceph-mon[31955]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' 2024-05-27T07:25:34.230 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:25:34 smithi052 ceph-mon[31955]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T07:25:34.230 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:25:34 smithi052 ceph-mon[31955]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-27T07:25:34.230 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:25:34 smithi052 ceph-mon[31955]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' 2024-05-27T07:25:34.230 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:25:34 smithi052 ceph-mon[31955]: from='client.? 172.21.15.52:0/2465335471' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-05-27T07:25:34.351 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:25:34 smithi173 ceph-mon[37000]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' 2024-05-27T07:25:34.352 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:25:34 smithi173 ceph-mon[37000]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' 2024-05-27T07:25:34.352 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:25:34 smithi173 ceph-mon[37000]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' 2024-05-27T07:25:34.352 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:25:34 smithi173 ceph-mon[37000]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' 2024-05-27T07:25:34.352 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:25:34 smithi173 ceph-mon[37000]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' 2024-05-27T07:25:34.352 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:25:34 smithi173 ceph-mon[37000]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' 2024-05-27T07:25:34.352 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:25:34 smithi173 ceph-mon[37000]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T07:25:34.352 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:25:34 smithi173 ceph-mon[37000]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-27T07:25:34.352 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:25:34 smithi173 ceph-mon[37000]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' 2024-05-27T07:25:34.352 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:25:34 smithi173 ceph-mon[37000]: from='client.? 172.21.15.52:0/2465335471' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-05-27T07:25:34.440 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:25:34 smithi105 ceph-mon[35977]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' 2024-05-27T07:25:34.440 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:25:34 smithi105 ceph-mon[35977]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' 2024-05-27T07:25:34.440 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:25:34 smithi105 ceph-mon[35977]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' 2024-05-27T07:25:34.440 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:25:34 smithi105 ceph-mon[35977]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' 2024-05-27T07:25:34.440 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:25:34 smithi105 ceph-mon[35977]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' 2024-05-27T07:25:34.440 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:25:34 smithi105 ceph-mon[35977]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' 2024-05-27T07:25:34.440 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:25:34 smithi105 ceph-mon[35977]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T07:25:34.440 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:25:34 smithi105 ceph-mon[35977]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-27T07:25:34.440 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:25:34 smithi105 ceph-mon[35977]: from='mgr.14152 172.21.15.52:0/1451096593' entity='mgr.x' 2024-05-27T07:25:34.440 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:25:34 smithi105 ceph-mon[35977]: from='client.? 172.21.15.52:0/2465335471' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-05-27T07:25:34.646 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 29, 'flags': 50, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': True, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 32, 'explicitly_allowed_features': 32, 'created': '2024-05-27T07:21:50.521783+0000', 'modified': '2024-05-27T07:22:58.822630+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 24527, 'mds_1': 24497, 'mds_2': 24475, 'mds_3': 24473, 'mds_4': 24466}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_14619': {'gid': 14619, 'name': 'g', 'rank': 4, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 1, 'addr': '172.21.15.52:6839/4258549559', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.52:6838', 'nonce': 4258549559}, {'type': 'v1', 'addr': '172.21.15.52:6839', 'nonce': 4258549559}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_14631': {'gid': 14631, 'name': 'a', 'rank': 3, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 1, 'addr': '172.21.15.52:6841/2878618188', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.52:6840', 'nonce': 2878618188}, {'type': 'v1', 'addr': '172.21.15.52:6841', 'nonce': 2878618188}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24433': {'gid': 24433, 'name': 'l', 'rank': 2, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 1, 'addr': '172.21.15.173:6833/3013589220', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.173:6832', 'nonce': 3013589220}, {'type': 'v1', 'addr': '172.21.15.173:6833', 'nonce': 3013589220}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24452': {'gid': 24452, 'name': 'k', 'rank': 1, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 1, 'addr': '172.21.15.105:6833/2800678412', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.105:6832', 'nonce': 2800678412}, {'type': 'v1', 'addr': '172.21.15.105:6833', 'nonce': 2800678412}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24460': {'gid': 24460, 'name': 'f', 'rank': 0, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 1, 'addr': '172.21.15.173:6835/1619486437', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.173:6834', 'nonce': 1619486437}, {'type': 'v1', 'addr': '172.21.15.173:6835', 'nonce': 1619486437}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24466': {'gid': 24466, 'name': 'c', 'rank': 4, 'incarnation': 26, 'state': 'up:active', 'state_seq': 25, 'addr': '172.21.15.173:6837/1396363827', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.173:6836', 'nonce': 1396363827}, {'type': 'v1', 'addr': '172.21.15.173:6837', 'nonce': 1396363827}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24473': {'gid': 24473, 'name': 'e', 'rank': 3, 'incarnation': 24, 'state': 'up:active', 'state_seq': 26, 'addr': '172.21.15.105:6835/3298498384', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.105:6834', 'nonce': 3298498384}, {'type': 'v1', 'addr': '172.21.15.105:6835', 'nonce': 3298498384}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24475': {'gid': 24475, 'name': 'i', 'rank': 2, 'incarnation': 22, 'state': 'up:active', 'state_seq': 23, 'addr': '172.21.15.173:6839/564778293', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.173:6838', 'nonce': 564778293}, {'type': 'v1', 'addr': '172.21.15.173:6839', 'nonce': 564778293}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24497': {'gid': 24497, 'name': 'h', 'rank': 1, 'incarnation': 20, 'state': 'up:active', 'state_seq': 24, 'addr': '172.21.15.105:6837/191393149', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.105:6836', 'nonce': 191393149}, {'type': 'v1', 'addr': '172.21.15.105:6837', 'nonce': 191393149}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24527': {'gid': 24527, 'name': 'b', 'rank': 0, 'incarnation': 14, 'state': 'up:active', 'state_seq': 5, 'addr': '172.21.15.105:6839/500348151', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.105:6838', 'nonce': 500348151}, {'type': 'v1', 'addr': '172.21.15.105:6839', 'nonce': 500348151}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 24527, 'qdb_cluster': [24466, 24473, 24475, 24497, 24527]} 2024-05-27T07:25:34.647 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 5/5 2024-05-27T07:25:34.647 DEBUG:teuthology.orchestra.run.smithi052:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:38f9d900c83c63799fbdbe61acc9a11b0d3554a6 shell --fsid 73df2636-1bf8-11ef-bc9b-c7b262605968 -- ceph tell mds.c status 2024-05-27T07:25:34.901 INFO:teuthology.orchestra.run.smithi052.stderr:Inferring config /var/lib/ceph/73df2636-1bf8-11ef-bc9b-c7b262605968/mon.a/config 2024-05-27T07:25:35.277 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:25:35 smithi052 ceph-mon[31955]: pgmap v412: 129 pgs: 129 active+clean; 647 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 9 op/s 2024-05-27T07:25:35.351 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:25:35 smithi173 ceph-mon[37000]: pgmap v412: 129 pgs: 129 active+clean; 647 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 9 op/s 2024-05-27T07:25:35.440 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:25:35 smithi105 ceph-mon[35977]: pgmap v412: 129 pgs: 129 active+clean; 647 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 9 op/s 2024-05-27T07:25:36.687 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:36.602+0000 7f57b32d3700 1 -- 172.21.15.52:0/4140038951 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f57ac108ef0 msgr2=0x7f57ac109300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:25:36.687 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:36.602+0000 7f57b32d3700 1 --2- 172.21.15.52:0/4140038951 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f57ac108ef0 0x7f57ac109300 secure :-1 s=READY pgs=358 cs=0 l=1 rev1=1 crypto rx=0x7f57a00099c0 tx=0x7f57a0009cd0 comp rx=0 tx=0).stop 2024-05-27T07:25:36.687 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:36.602+0000 7f57b32d3700 1 -- 172.21.15.52:0/4140038951 shutdown_connections 2024-05-27T07:25:36.687 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:36.602+0000 7f57b32d3700 1 --2- 172.21.15.52:0/4140038951 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f57ac109840 0x7f57ac1111a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:25:36.687 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:36.602+0000 7f57b32d3700 1 --2- 172.21.15.52:0/4140038951 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f57ac1045c0 0x7f57ac1049d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:25:36.687 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:36.602+0000 7f57b32d3700 1 --2- 172.21.15.52:0/4140038951 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f57ac108ef0 0x7f57ac109300 unknown :-1 s=CLOSED pgs=358 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:25:36.687 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:36.602+0000 7f57b32d3700 1 -- 172.21.15.52:0/4140038951 >> 172.21.15.52:0/4140038951 conn(0x7f57ac0ffe90 msgr2=0x7f57ac1022e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:25:36.688 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:36.602+0000 7f57b32d3700 1 -- 172.21.15.52:0/4140038951 shutdown_connections 2024-05-27T07:25:36.688 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:36.602+0000 7f57b32d3700 1 -- 172.21.15.52:0/4140038951 wait complete. 2024-05-27T07:25:36.688 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:36.602+0000 7f57b32d3700 1 Processor -- start 2024-05-27T07:25:36.688 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:36.602+0000 7f57b32d3700 1 -- start start 2024-05-27T07:25:36.688 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:36.603+0000 7f57b32d3700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f57ac1045c0 0x7f57ac19aea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:25:36.688 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:36.603+0000 7f57b32d3700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f57ac108ef0 0x7f57ac19b440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:25:36.688 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:36.603+0000 7f57b32d3700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f57ac109840 0x7f57ac19f6d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:25:36.688 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:36.603+0000 7f57b32d3700 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x7f57ac114450 con 0x7f57ac1045c0 2024-05-27T07:25:36.688 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:36.603+0000 7f57b32d3700 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f57ac1142d0 con 0x7f57ac109840 2024-05-27T07:25:36.688 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:36.603+0000 7f57b32d3700 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f57ac1145d0 con 0x7f57ac108ef0 2024-05-27T07:25:36.688 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:36.603+0000 7f57b106f700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f57ac1045c0 0x7f57ac19aea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:25:36.688 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:36.603+0000 7f57b106f700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f57ac1045c0 0x7f57ac19aea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.52:36702/0 (socket says 172.21.15.52:36702) 2024-05-27T07:25:36.689 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:36.603+0000 7f57b106f700 1 -- 172.21.15.52:0/913753868 learned_addr learned my addr 172.21.15.52:0/913753868 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-05-27T07:25:36.689 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:36.603+0000 7f57b086e700 1 --2- 172.21.15.52:0/913753868 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f57ac108ef0 0x7f57ac19b440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:25:36.689 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:36.603+0000 7f57b1870700 1 --2- 172.21.15.52:0/913753868 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f57ac109840 0x7f57ac19f6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:25:36.689 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:36.603+0000 7f57b106f700 1 -- 172.21.15.52:0/913753868 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f57ac108ef0 msgr2=0x7f57ac19b440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:25:36.689 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:36.603+0000 7f57b106f700 1 --2- 172.21.15.52:0/913753868 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f57ac108ef0 0x7f57ac19b440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:25:36.689 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:36.603+0000 7f57b106f700 1 -- 172.21.15.52:0/913753868 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f57ac109840 msgr2=0x7f57ac19f6d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:25:36.689 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:36.603+0000 7f57b106f700 1 --2- 172.21.15.52:0/913753868 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f57ac109840 0x7f57ac19f6d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:25:36.689 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:36.603+0000 7f57b106f700 1 -- 172.21.15.52:0/913753868 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f57ac19fd70 con 0x7f57ac1045c0 2024-05-27T07:25:36.689 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:36.603+0000 7f57b106f700 1 --2- 172.21.15.52:0/913753868 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f57ac1045c0 0x7f57ac19aea0 secure :-1 s=READY pgs=359 cs=0 l=1 rev1=1 crypto rx=0x7f579c0060b0 tx=0x7f579c007f90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:25:36.689 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:36.604+0000 7f57aa7fc700 1 -- 172.21.15.52:0/913753868 <== mon.0 v2:172.21.15.52:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f579c01a070 con 0x7f57ac1045c0 2024-05-27T07:25:36.689 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:36.604+0000 7f57b32d3700 1 -- 172.21.15.52:0/913753868 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f57ac1a0060 con 0x7f57ac1045c0 2024-05-27T07:25:36.689 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:36.604+0000 7f57aa7fc700 1 -- 172.21.15.52:0/913753868 <== mon.0 v2:172.21.15.52:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f579c00de20 con 0x7f57ac1045c0 2024-05-27T07:25:36.689 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:36.604+0000 7f57aa7fc700 1 -- 172.21.15.52:0/913753868 <== mon.0 v2:172.21.15.52:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f579c014720 con 0x7f57ac1045c0 2024-05-27T07:25:36.690 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:36.604+0000 7f57b32d3700 1 -- 172.21.15.52:0/913753868 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f57ac1a7980 con 0x7f57ac1045c0 2024-05-27T07:25:36.690 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:36.606+0000 7f57aa7fc700 1 -- 172.21.15.52:0/913753868 <== mon.0 v2:172.21.15.52:3300/0 4 ==== mgrmap(e 18) ==== 149652+0+0 (secure 0 0 0) 0x7f579c01d070 con 0x7f57ac1045c0 2024-05-27T07:25:36.690 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:36.607+0000 7f57aa7fc700 1 --2- 172.21.15.52:0/913753868 >> [v2:172.21.15.52:6800/3513075805,v1:172.21.15.52:6801/3513075805] conn(0x7f57940b2880 0x7f57940b4d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:25:36.690 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:36.607+0000 7f57b086e700 1 --2- 172.21.15.52:0/913753868 >> [v2:172.21.15.52:6800/3513075805,v1:172.21.15.52:6801/3513075805] conn(0x7f57940b2880 0x7f57940b4d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:25:36.690 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:36.607+0000 7f57aa7fc700 1 -- 172.21.15.52:0/913753868 <== mon.0 v2:172.21.15.52:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9121+0+0 (secure 0 0 0) 0x7f579c0e1eb0 con 0x7f57ac1045c0 2024-05-27T07:25:36.690 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:36.607+0000 7f57c1ce8b80 1 Processor -- start 2024-05-27T07:25:36.690 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:36.608+0000 7f57c1ce8b80 1 -- start start 2024-05-27T07:25:36.690 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:36.608+0000 7f57c1ce8b80 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x5636cf2a1df0 0x5636cf4256c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:25:36.690 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:36.608+0000 7f57c1ce8b80 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5636cf425c00 0x5636cf41f9f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:25:36.690 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:36.608+0000 7f57c1ce8b80 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5636cf41ff30 0x5636cf420340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:25:36.690 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:36.608+0000 7f57b086e700 1 --2- 172.21.15.52:0/913753868 >> [v2:172.21.15.52:6800/3513075805,v1:172.21.15.52:6801/3513075805] conn(0x7f57940b2880 0x7f57940b4d90 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7f57a0000c00 tx=0x7f57a001a040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:25:36.690 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:36.608+0000 7f57c1ce8b80 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x5636cf34ec40 con 0x5636cf2a1df0 2024-05-27T07:25:36.691 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:36.608+0000 7f57c1ce8b80 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x5636cf33e7e0 con 0x5636cf425c00 2024-05-27T07:25:36.691 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:36.608+0000 7f57c1ce8b80 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x5636cf342820 con 0x5636cf41ff30 2024-05-27T07:25:36.691 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:36.608+0000 7f57b106f700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x5636cf2a1df0 0x5636cf4256c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:25:36.691 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:36.608+0000 7f57b106f700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x5636cf2a1df0 0x5636cf4256c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.52:36708/0 (socket says 172.21.15.52:36708) 2024-05-27T07:25:36.691 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:36.608+0000 7f57b106f700 1 -- 172.21.15.52:0/2147664340 learned_addr learned my addr 172.21.15.52:0/2147664340 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-05-27T07:25:36.691 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:36.608+0000 7f57b086e700 1 --2- 172.21.15.52:0/2147664340 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5636cf425c00 0x5636cf41f9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:25:36.691 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:36.608+0000 7f57b1870700 1 --2- 172.21.15.52:0/2147664340 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5636cf41ff30 0x5636cf420340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:25:36.691 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:36.608+0000 7f57b106f700 1 -- 172.21.15.52:0/2147664340 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5636cf41ff30 msgr2=0x5636cf420340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:25:36.691 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:36.608+0000 7f57b106f700 1 --2- 172.21.15.52:0/2147664340 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5636cf41ff30 0x5636cf420340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:25:36.691 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:36.608+0000 7f57b106f700 1 -- 172.21.15.52:0/2147664340 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5636cf425c00 msgr2=0x5636cf41f9f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:25:36.691 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:36.608+0000 7f57b106f700 1 --2- 172.21.15.52:0/2147664340 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5636cf425c00 0x5636cf41f9f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:25:36.692 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:36.608+0000 7f57b106f700 1 -- 172.21.15.52:0/2147664340 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5636cf28e880 con 0x5636cf2a1df0 2024-05-27T07:25:36.692 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:36.609+0000 7f57b106f700 1 --2- 172.21.15.52:0/2147664340 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x5636cf2a1df0 0x5636cf4256c0 secure :-1 s=READY pgs=360 cs=0 l=1 rev1=1 crypto rx=0x7f579c000f80 tx=0x7f579c0a8ce0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=44c70f9bbe886298 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:25:36.692 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:36.609+0000 7f57937fe700 1 -- 172.21.15.52:0/2147664340 <== mon.0 v2:172.21.15.52:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f579c0ad500 con 0x5636cf2a1df0 2024-05-27T07:25:36.692 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:36.609+0000 7f57937fe700 1 -- 172.21.15.52:0/2147664340 <== mon.0 v2:172.21.15.52:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f579c0b1fb0 con 0x5636cf2a1df0 2024-05-27T07:25:36.692 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:36.609+0000 7f57937fe700 1 -- 172.21.15.52:0/2147664340 <== mon.0 v2:172.21.15.52:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f579c0b1b20 con 0x5636cf2a1df0 2024-05-27T07:25:36.692 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:36.610+0000 7f57c1ce8b80 1 -- 172.21.15.52:0/2147664340 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x5636cf2a1df0 msgr2=0x5636cf4256c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:25:36.692 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:36.610+0000 7f57c1ce8b80 1 --2- 172.21.15.52:0/2147664340 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x5636cf2a1df0 0x5636cf4256c0 secure :-1 s=READY pgs=360 cs=0 l=1 rev1=1 crypto rx=0x7f579c000f80 tx=0x7f579c0a8ce0 comp rx=0 tx=0).stop 2024-05-27T07:25:36.692 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:36.610+0000 7f57c1ce8b80 1 -- 172.21.15.52:0/2147664340 shutdown_connections 2024-05-27T07:25:36.692 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:36.610+0000 7f57c1ce8b80 1 --2- 172.21.15.52:0/2147664340 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5636cf41ff30 0x5636cf420340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:25:36.692 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:36.610+0000 7f57c1ce8b80 1 --2- 172.21.15.52:0/2147664340 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x5636cf2a1df0 0x5636cf4256c0 unknown :-1 s=CLOSED pgs=360 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:25:36.692 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:36.610+0000 7f57c1ce8b80 1 --2- 172.21.15.52:0/2147664340 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5636cf425c00 0x5636cf41f9f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:25:36.692 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:36.610+0000 7f57c1ce8b80 1 -- 172.21.15.52:0/2147664340 >> 172.21.15.52:0/2147664340 conn(0x5636cf2fcaa0 msgr2=0x5636cf41a5f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:25:36.692 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:36.610+0000 7f57c1ce8b80 1 -- 172.21.15.52:0/2147664340 shutdown_connections 2024-05-27T07:25:36.693 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:36.610+0000 7f57c1ce8b80 1 -- 172.21.15.52:0/2147664340 wait complete. 2024-05-27T07:25:36.693 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:36.610+0000 7f57c1ce8b80 1 -- start start 2024-05-27T07:25:36.693 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:36.610+0000 7f57c1ce8b80 1 Processor -- start 2024-05-27T07:25:36.693 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:36.611+0000 7f57c1ce8b80 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x5636cf2a1df0 0x5636cf429250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:25:36.693 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:36.611+0000 7f57c1ce8b80 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5636cf41ff30 0x5636cf429790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:25:36.693 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:36.611+0000 7f57c1ce8b80 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5636cf425c00 0x5636cf429cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:25:36.693 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:36.611+0000 7f57c1ce8b80 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x5636cf244f50 con 0x5636cf2a1df0 2024-05-27T07:25:36.693 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:36.611+0000 7f57c1ce8b80 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x5636cf2e0770 con 0x5636cf41ff30 2024-05-27T07:25:36.693 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:36.611+0000 7f57c1ce8b80 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x5636cf430b00 con 0x5636cf425c00 2024-05-27T07:25:36.693 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:36.611+0000 7f57c1ce8b80 10 client.0 _ll_register_callbacks cb 0x5636cf2d98d0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T07:25:36.693 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:36.611+0000 7f57b106f700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x5636cf2a1df0 0x5636cf429250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:25:36.694 INFO:teuthology.orchestra.run.smithi052.stdout:{ 2024-05-27T07:25:36.694 INFO:teuthology.orchestra.run.smithi052.stdout: "cluster_fsid": "73df2636-1bf8-11ef-bc9b-c7b262605968", 2024-05-27T07:25:36.694 INFO:teuthology.orchestra.run.smithi052.stdout: "whoami": 4, 2024-05-27T07:25:36.694 INFO:teuthology.orchestra.run.smithi052.stdout: "id": 24466, 2024-05-27T07:25:36.694 INFO:teuthology.orchestra.run.smithi052.stdout: "want_state": "up:active", 2024-05-27T07:25:36.694 INFO:teuthology.orchestra.run.smithi052.stdout: "state": "up:active", 2024-05-27T07:25:36.694 INFO:teuthology.orchestra.run.smithi052.stdout: "fs_name": "cephfs", 2024-05-27T07:25:36.694 INFO:teuthology.orchestra.run.smithi052.stdout: "rank_uptime": 160.763881682, 2024-05-27T07:25:36.694 INFO:teuthology.orchestra.run.smithi052.stdout: "mdsmap_epoch": 29, 2024-05-27T07:25:36.694 INFO:teuthology.orchestra.run.smithi052.stdout: "osdmap_epoch": 90, 2024-05-27T07:25:36.695 INFO:teuthology.orchestra.run.smithi052.stdout: "osdmap_epoch_barrier": 88, 2024-05-27T07:25:36.695 INFO:teuthology.orchestra.run.smithi052.stdout: "uptime": 255.91508623600001 2024-05-27T07:25:36.695 INFO:teuthology.orchestra.run.smithi052.stdout:} 2024-05-27T07:25:36.695 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:36.611+0000 7f57b106f700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x5636cf2a1df0 0x5636cf429250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.52:36724/0 (socket says 172.21.15.52:36724) 2024-05-27T07:25:36.695 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:36.611+0000 7f57b106f700 1 -- 172.21.15.52:0/30083834 learned_addr learned my addr 172.21.15.52:0/30083834 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-05-27T07:25:36.695 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:36.611+0000 7f57b086e700 1 --2- 172.21.15.52:0/30083834 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5636cf41ff30 0x5636cf429790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:25:36.695 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:36.611+0000 7f57b106f700 1 -- 172.21.15.52:0/30083834 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5636cf425c00 msgr2=0x5636cf429cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:25:36.695 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:36.611+0000 7f57b106f700 1 --2- 172.21.15.52:0/30083834 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5636cf425c00 0x5636cf429cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:25:36.696 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:36.611+0000 7f57b106f700 1 -- 172.21.15.52:0/30083834 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5636cf41ff30 msgr2=0x5636cf429790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:25:36.696 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:36.611+0000 7f57b106f700 1 --2- 172.21.15.52:0/30083834 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5636cf41ff30 0x5636cf429790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:25:36.696 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:36.611+0000 7f57b106f700 1 -- 172.21.15.52:0/30083834 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5636cf2dd640 con 0x5636cf2a1df0 2024-05-27T07:25:36.696 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:36.611+0000 7f57b086e700 1 --2- 172.21.15.52:0/30083834 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5636cf41ff30 0x5636cf429790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-27T07:25:36.696 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:36.611+0000 7f57b106f700 1 --2- 172.21.15.52:0/30083834 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x5636cf2a1df0 0x5636cf429250 secure :-1 s=READY pgs=361 cs=0 l=1 rev1=1 crypto rx=0x7f579c009510 tx=0x7f579c015c70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e5908c52d39a84f5 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:25:36.696 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:36.611+0000 7f5790ff9700 10 client.0 ms_handle_connect on v2:172.21.15.52:3300/0 2024-05-27T07:25:36.696 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:36.612+0000 7f5790ff9700 1 -- 172.21.15.52:0/30083834 <== mon.0 v2:172.21.15.52:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f579c0b1d20 con 0x5636cf2a1df0 2024-05-27T07:25:36.696 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:36.612+0000 7f5790ff9700 1 -- 172.21.15.52:0/30083834 <== mon.0 v2:172.21.15.52:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9121+0+0 (secure 0 0 0) 0x7f579c0a83b0 con 0x5636cf2a1df0 2024-05-27T07:25:36.696 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:36.612+0000 7f57c1ce8b80 1 -- 172.21.15.52:0/30083834 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f579c0b1fb0 con 0x5636cf2a1df0 2024-05-27T07:25:36.696 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:36.612+0000 7f5790ff9700 1 -- 172.21.15.52:0/30083834 <== mon.0 v2:172.21.15.52:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f579c08a050 con 0x5636cf2a1df0 2024-05-27T07:25:36.696 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:36.612+0000 7f57c1ce8b80 10 client.15231 fetch_fsmap learned FSMap version 29 2024-05-27T07:25:36.696 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:36.612+0000 7f57c1ce8b80 1 -- 172.21.15.52:0/30083834 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x5636cf43b520 con 0x5636cf2a1df0 2024-05-27T07:25:36.697 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:36.612+0000 7f5790ff9700 1 -- 172.21.15.52:0/30083834 <== mon.0 v2:172.21.15.52:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f579c0b1fb0 con 0x5636cf2a1df0 2024-05-27T07:25:36.697 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:36.612+0000 7f5790ff9700 1 -- 172.21.15.52:0/30083834 <== mon.0 v2:172.21.15.52:3300/0 5 ==== fsmap(e 29) ==== 8362+0+0 (secure 0 0 0) 0x7f579c004ad0 con 0x5636cf2a1df0 2024-05-27T07:25:36.697 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:36.612+0000 7f57c1ce8b80 10 client.15231 fetch_fsmap finished waiting for FSMap version 29 2024-05-27T07:25:36.697 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:36.612+0000 7f57c1ce8b80 10 client.15231 resolve_mds: resolved name 'c' to daemon mds.c 2024-05-27T07:25:36.697 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:36.612+0000 7f57c1ce8b80 20 client.15231 populate_metadata read hostname 'smithi052' 2024-05-27T07:25:36.697 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:36.612+0000 7f57c1ce8b80 1 --2- 172.21.15.52:0/30083834 >> [v2:172.21.15.173:6836/1396363827,v1:172.21.15.173:6837/1396363827] conn(0x5636cf43d620 0x5636cf43ea90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:25:36.697 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:36.612+0000 7f57c1ce8b80 4 client.15231 mds_command: new command op to 24466 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-27T07:25:36.697 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:36.612+0000 7f57c1ce8b80 1 -- 172.21.15.52:0/30083834 --> [v2:172.21.15.173:6836/1396363827,v1:172.21.15.173:6837/1396363827] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5636cf3eb9a0 con 0x5636cf43d620 2024-05-27T07:25:36.697 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:36.613+0000 7f57b086e700 1 --2- 172.21.15.52:0/30083834 >> [v2:172.21.15.173:6836/1396363827,v1:172.21.15.173:6837/1396363827] conn(0x5636cf43d620 0x5636cf43ea90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:25:36.697 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:36.613+0000 7f57b086e700 1 --2- 172.21.15.52:0/30083834 >> [v2:172.21.15.173:6836/1396363827,v1:172.21.15.173:6837/1396363827] conn(0x5636cf43d620 0x5636cf43ea90 crc :-1 s=READY pgs=17 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=8a0f76f052578a88 server_cookie=65a909bfe2db8060 in_seq=0 out_seq=0 2024-05-27T07:25:36.697 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:36.613+0000 7f5790ff9700 10 client.15231 ms_handle_connect on v2:172.21.15.173:6836/1396363827 2024-05-27T07:25:36.697 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:36.614+0000 7f5790ff9700 1 -- 172.21.15.52:0/30083834 <== mds.4 v2:172.21.15.173:6836/1396363827 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24041 (crc 0 0 0) 0x5636cf3eb9a0 con 0x5636cf43d620 2024-05-27T07:25:36.697 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:36.614+0000 7f5790ff9700 10 client.15231 handle_command_reply: tid=0 2024-05-27T07:25:36.698 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:36.614+0000 7f57c1ce8b80 1 client.15231 shutdown 2024-05-27T07:25:36.698 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:36.614+0000 7f57c1ce8b80 1 -- 172.21.15.52:0/30083834 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x5636cf2a1df0 msgr2=0x5636cf429250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:25:36.698 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:36.614+0000 7f57c1ce8b80 1 --2- 172.21.15.52:0/30083834 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x5636cf2a1df0 0x5636cf429250 secure :-1 s=READY pgs=361 cs=0 l=1 rev1=1 crypto rx=0x7f579c009510 tx=0x7f579c015c70 comp rx=0 tx=0).stop 2024-05-27T07:25:36.698 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:36.615+0000 7f57c1ce8b80 1 -- 172.21.15.52:0/30083834 shutdown_connections 2024-05-27T07:25:36.698 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:36.615+0000 7f57c1ce8b80 1 --2- 172.21.15.52:0/30083834 >> [v2:172.21.15.173:6836/1396363827,v1:172.21.15.173:6837/1396363827] conn(0x5636cf43d620 0x5636cf43ea90 crc :-1 s=READY pgs=17 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:25:36.698 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:36.615+0000 7f57c1ce8b80 1 --2- 172.21.15.52:0/30083834 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5636cf425c00 0x5636cf429cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:25:36.698 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:36.615+0000 7f57c1ce8b80 1 --2- 172.21.15.52:0/30083834 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x5636cf2a1df0 0x5636cf429250 unknown :-1 s=CLOSED pgs=361 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:25:36.698 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:36.615+0000 7f57c1ce8b80 1 --2- 172.21.15.52:0/30083834 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5636cf41ff30 0x5636cf429790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:25:36.698 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:36.615+0000 7f57c1ce8b80 1 -- 172.21.15.52:0/30083834 >> 172.21.15.52:0/30083834 conn(0x5636cf2fcaa0 msgr2=0x5636cf41e9e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:25:36.698 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:36.615+0000 7f5790ff9700 0 client.15231 ms_handle_reset on v2:172.21.15.173:6836/1396363827 2024-05-27T07:25:36.698 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:36.615+0000 7f57c1ce8b80 1 -- 172.21.15.52:0/30083834 shutdown_connections 2024-05-27T07:25:36.698 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:36.615+0000 7f57c1ce8b80 1 -- 172.21.15.52:0/30083834 wait complete. 2024-05-27T07:25:36.699 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:36.615+0000 7f57c1ce8b80 20 client.15231 trim_cache size 0 max 16384 2024-05-27T07:25:36.699 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:36.637+0000 7f57c1ce8b80 1 Processor -- start 2024-05-27T07:25:36.699 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:36.637+0000 7f57c1ce8b80 1 -- start start 2024-05-27T07:25:36.699 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:36.637+0000 7f57c1ce8b80 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5636cf2a1df0 0x5636cf41f9f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:25:36.699 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:36.637+0000 7f57c1ce8b80 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x5636cf41ff30 0x5636cf439e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:25:36.699 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:36.637+0000 7f57c1ce8b80 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5636cf425c00 0x5636cf41c010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:25:36.699 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:36.637+0000 7f57c1ce8b80 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x5636cf43cdd0 con 0x5636cf41ff30 2024-05-27T07:25:36.699 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:36.637+0000 7f57c1ce8b80 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x5636cf43d4a0 con 0x5636cf2a1df0 2024-05-27T07:25:36.699 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:36.637+0000 7f57c1ce8b80 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x5636cf43db70 con 0x5636cf425c00 2024-05-27T07:25:36.699 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:36.637+0000 7f57b086e700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x5636cf41ff30 0x5636cf439e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:25:36.699 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:36.637+0000 7f57b086e700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x5636cf41ff30 0x5636cf439e00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.52:36740/0 (socket says 172.21.15.52:36740) 2024-05-27T07:25:36.699 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:36.637+0000 7f57b086e700 1 -- 172.21.15.52:0/2849338380 learned_addr learned my addr 172.21.15.52:0/2849338380 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-05-27T07:25:36.699 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:36.637+0000 7f57b1870700 1 --2- 172.21.15.52:0/2849338380 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5636cf425c00 0x5636cf41c010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:25:36.700 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:36.637+0000 7f57b106f700 1 --2- 172.21.15.52:0/2849338380 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5636cf2a1df0 0x5636cf41f9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:25:36.700 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:36.637+0000 7f57b086e700 1 -- 172.21.15.52:0/2849338380 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5636cf425c00 msgr2=0x5636cf41c010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:25:36.700 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:36.637+0000 7f57b086e700 1 --2- 172.21.15.52:0/2849338380 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5636cf425c00 0x5636cf41c010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:25:36.700 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:36.637+0000 7f57b086e700 1 -- 172.21.15.52:0/2849338380 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5636cf2a1df0 msgr2=0x5636cf41f9f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:25:36.700 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:36.637+0000 7f57b086e700 1 --2- 172.21.15.52:0/2849338380 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5636cf2a1df0 0x5636cf41f9f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:25:36.700 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:36.637+0000 7f57b086e700 1 -- 172.21.15.52:0/2849338380 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5636cf43cbe0 con 0x5636cf41ff30 2024-05-27T07:25:36.700 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:36.637+0000 7f57b106f700 1 --2- 172.21.15.52:0/2849338380 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5636cf2a1df0 0x5636cf41f9f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-27T07:25:36.700 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:36.638+0000 7f57b086e700 1 --2- 172.21.15.52:0/2849338380 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x5636cf41ff30 0x5636cf439e00 secure :-1 s=READY pgs=362 cs=0 l=1 rev1=1 crypto rx=0x7f57a00059d0 tx=0x7f57a001d000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=991183723555ff27 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:25:36.700 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:36.638+0000 7f5792ffd700 1 -- 172.21.15.52:0/2849338380 <== mon.0 v2:172.21.15.52:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f57a000fda0 con 0x5636cf41ff30 2024-05-27T07:25:36.700 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:36.638+0000 7f5792ffd700 1 -- 172.21.15.52:0/2849338380 <== mon.0 v2:172.21.15.52:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f57a00228b0 con 0x5636cf41ff30 2024-05-27T07:25:36.700 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:36.638+0000 7f5792ffd700 1 -- 172.21.15.52:0/2849338380 <== mon.0 v2:172.21.15.52:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f57a0018710 con 0x5636cf41ff30 2024-05-27T07:25:36.700 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:36.639+0000 7f57c1ce8b80 1 -- 172.21.15.52:0/2849338380 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x5636cf41ff30 msgr2=0x5636cf439e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:25:36.700 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:36.639+0000 7f57c1ce8b80 1 --2- 172.21.15.52:0/2849338380 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x5636cf41ff30 0x5636cf439e00 secure :-1 s=READY pgs=362 cs=0 l=1 rev1=1 crypto rx=0x7f57a00059d0 tx=0x7f57a001d000 comp rx=0 tx=0).stop 2024-05-27T07:25:36.701 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:36.639+0000 7f57c1ce8b80 1 -- 172.21.15.52:0/2849338380 shutdown_connections 2024-05-27T07:25:36.701 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:36.639+0000 7f57c1ce8b80 1 --2- 172.21.15.52:0/2849338380 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5636cf425c00 0x5636cf41c010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:25:36.701 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:36.639+0000 7f57c1ce8b80 1 --2- 172.21.15.52:0/2849338380 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5636cf2a1df0 0x5636cf41f9f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:25:36.701 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:36.639+0000 7f57c1ce8b80 1 --2- 172.21.15.52:0/2849338380 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x5636cf41ff30 0x5636cf439e00 unknown :-1 s=CLOSED pgs=362 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:25:36.701 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:36.639+0000 7f57c1ce8b80 1 -- 172.21.15.52:0/2849338380 >> 172.21.15.52:0/2849338380 conn(0x5636cf2fcaa0 msgr2=0x5636cf41a4f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:25:36.701 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:36.639+0000 7f57c1ce8b80 1 -- 172.21.15.52:0/2849338380 shutdown_connections 2024-05-27T07:25:36.701 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:36.639+0000 7f57c1ce8b80 1 -- 172.21.15.52:0/2849338380 wait complete. 2024-05-27T07:25:36.701 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:36.639+0000 7f57c1ce8b80 1 -- start start 2024-05-27T07:25:36.701 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:36.639+0000 7f57c1ce8b80 1 Processor -- start 2024-05-27T07:25:36.701 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:36.640+0000 7f57c1ce8b80 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5636cf2a1df0 0x5636cf4485a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:25:36.701 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:36.640+0000 7f57c1ce8b80 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5636cf41ff30 0x5636cf4471e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:25:36.701 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:36.640+0000 7f57c1ce8b80 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x5636cf425c00 0x5636cf447720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:25:36.701 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:36.640+0000 7f57c1ce8b80 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x5636cf43e910 con 0x5636cf425c00 2024-05-27T07:25:36.701 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:36.640+0000 7f57c1ce8b80 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x5636cf43e240 con 0x5636cf2a1df0 2024-05-27T07:25:36.702 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:36.640+0000 7f57c1ce8b80 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x5636cf43efe0 con 0x5636cf41ff30 2024-05-27T07:25:36.702 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:36.640+0000 7f57c1ce8b80 10 client.0 _ll_register_callbacks cb 0x5636cf2d98d0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T07:25:36.702 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:36.640+0000 7f57b1870700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x5636cf425c00 0x5636cf447720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:25:36.702 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:36.640+0000 7f57b1870700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x5636cf425c00 0x5636cf447720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.52:36748/0 (socket says 172.21.15.52:36748) 2024-05-27T07:25:36.702 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:36.640+0000 7f57b1870700 1 -- 172.21.15.52:0/3425902470 learned_addr learned my addr 172.21.15.52:0/3425902470 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-05-27T07:25:36.702 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:36.640+0000 7f57b086e700 1 --2- 172.21.15.52:0/3425902470 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5636cf41ff30 0x5636cf4471e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:25:36.702 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:36.640+0000 7f57b106f700 1 --2- 172.21.15.52:0/3425902470 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5636cf2a1df0 0x5636cf4485a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:25:36.702 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:36.640+0000 7f57b1870700 1 -- 172.21.15.52:0/3425902470 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5636cf41ff30 msgr2=0x5636cf4471e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:25:36.702 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:36.640+0000 7f57b1870700 1 --2- 172.21.15.52:0/3425902470 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5636cf41ff30 0x5636cf4471e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:25:36.702 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:36.640+0000 7f57b1870700 1 -- 172.21.15.52:0/3425902470 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5636cf2a1df0 msgr2=0x5636cf4485a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:25:36.703 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:36.640+0000 7f57b1870700 1 --2- 172.21.15.52:0/3425902470 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5636cf2a1df0 0x5636cf4485a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:25:36.703 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:36.640+0000 7f57b1870700 1 -- 172.21.15.52:0/3425902470 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5636cf447e70 con 0x5636cf425c00 2024-05-27T07:25:36.703 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:36.640+0000 7f57b1870700 1 --2- 172.21.15.52:0/3425902470 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x5636cf425c00 0x5636cf447720 secure :-1 s=READY pgs=363 cs=0 l=1 rev1=1 crypto rx=0x7f57a400e000 tx=0x7f57a4012730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=89e2bfcae3b2aaa9 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:25:36.703 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:36.640+0000 7f5790ff9700 10 client.0 ms_handle_connect on v2:172.21.15.52:3300/0 2024-05-27T07:25:36.703 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:36.641+0000 7f5790ff9700 1 -- 172.21.15.52:0/3425902470 <== mon.0 v2:172.21.15.52:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f57a4005260 con 0x5636cf425c00 2024-05-27T07:25:36.703 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:36.641+0000 7f5790ff9700 1 -- 172.21.15.52:0/3425902470 <== mon.0 v2:172.21.15.52:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9121+0+0 (secure 0 0 0) 0x7f57a4005400 con 0x5636cf425c00 2024-05-27T07:25:36.703 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:36.641+0000 7f57c1ce8b80 1 -- 172.21.15.52:0/3425902470 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f57a00228b0 con 0x5636cf425c00 2024-05-27T07:25:36.703 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:36.641+0000 7f5790ff9700 1 -- 172.21.15.52:0/3425902470 <== mon.0 v2:172.21.15.52:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f57a4025050 con 0x5636cf425c00 2024-05-27T07:25:36.703 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:36.641+0000 7f57c1ce8b80 10 client.15237 fetch_fsmap learned FSMap version 29 2024-05-27T07:25:36.703 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:36.641+0000 7f57c1ce8b80 1 -- 172.21.15.52:0/3425902470 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x5636cf4a8db0 con 0x5636cf425c00 2024-05-27T07:25:36.703 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:36.641+0000 7f5790ff9700 1 -- 172.21.15.52:0/3425902470 <== mon.0 v2:172.21.15.52:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f57a00228b0 con 0x5636cf425c00 2024-05-27T07:25:36.703 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:36.641+0000 7f5790ff9700 1 -- 172.21.15.52:0/3425902470 <== mon.0 v2:172.21.15.52:3300/0 5 ==== fsmap(e 29) ==== 8362+0+0 (secure 0 0 0) 0x7f57a401c860 con 0x5636cf425c00 2024-05-27T07:25:36.703 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:36.641+0000 7f57c1ce8b80 10 client.15237 fetch_fsmap finished waiting for FSMap version 29 2024-05-27T07:25:36.704 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:36.641+0000 7f57c1ce8b80 10 client.15237 resolve_mds: resolved name 'c' to daemon mds.c 2024-05-27T07:25:36.704 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:36.641+0000 7f57c1ce8b80 20 client.15237 populate_metadata read hostname 'smithi052' 2024-05-27T07:25:36.704 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:36.641+0000 7f57c1ce8b80 1 --2- 172.21.15.52:0/3425902470 >> [v2:172.21.15.173:6836/1396363827,v1:172.21.15.173:6837/1396363827] conn(0x5636cf4ab150 0x5636cf4ac5c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:25:36.704 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:36.641+0000 7f57c1ce8b80 4 client.15237 mds_command: new command op to 24466 tid=0 multi_id=0 [{"prefix": "status"}] 2024-05-27T07:25:36.704 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:36.641+0000 7f57c1ce8b80 1 -- 172.21.15.52:0/3425902470 --> [v2:172.21.15.173:6836/1396363827,v1:172.21.15.173:6837/1396363827] -- command(tid 0: {"prefix": "status"}) -- 0x5636cf350b90 con 0x5636cf4ab150 2024-05-27T07:25:36.704 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:36.642+0000 7f57b106f700 1 --2- 172.21.15.52:0/3425902470 >> [v2:172.21.15.173:6836/1396363827,v1:172.21.15.173:6837/1396363827] conn(0x5636cf4ab150 0x5636cf4ac5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:25:36.704 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:36.642+0000 7f5790ff9700 10 client.15237 ms_handle_connect on v2:172.21.15.173:6836/1396363827 2024-05-27T07:25:36.704 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:36.642+0000 7f57b106f700 1 --2- 172.21.15.52:0/3425902470 >> [v2:172.21.15.173:6836/1396363827,v1:172.21.15.173:6837/1396363827] conn(0x5636cf4ab150 0x5636cf4ac5c0 crc :-1 s=READY pgs=18 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=8cc47e96bd95f6f0 server_cookie=455645b5309d130a in_seq=0 out_seq=0 2024-05-27T07:25:36.704 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:36.642+0000 7f5790ff9700 1 -- 172.21.15.52:0/3425902470 <== mds.4 v2:172.21.15.173:6836/1396363827 1 ==== command_reply(tid 0: 0 ) ==== 8+0+327 (crc 0 0 0) 0x5636cf350b90 con 0x5636cf4ab150 2024-05-27T07:25:36.704 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:36.642+0000 7f5790ff9700 10 client.15237 handle_command_reply: tid=0 2024-05-27T07:25:36.704 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:36.642+0000 7f57c1ce8b80 1 client.15237 shutdown 2024-05-27T07:25:36.704 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:36.643+0000 7f57c1ce8b80 1 -- 172.21.15.52:0/3425902470 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x5636cf425c00 msgr2=0x5636cf447720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:25:36.704 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:36.643+0000 7f57c1ce8b80 1 --2- 172.21.15.52:0/3425902470 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x5636cf425c00 0x5636cf447720 secure :-1 s=READY pgs=363 cs=0 l=1 rev1=1 crypto rx=0x7f57a400e000 tx=0x7f57a4012730 comp rx=0 tx=0).stop 2024-05-27T07:25:36.705 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:36.643+0000 7f57c1ce8b80 1 -- 172.21.15.52:0/3425902470 shutdown_connections 2024-05-27T07:25:36.705 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:36.643+0000 7f57c1ce8b80 1 --2- 172.21.15.52:0/3425902470 >> [v2:172.21.15.173:6836/1396363827,v1:172.21.15.173:6837/1396363827] conn(0x5636cf4ab150 0x5636cf4ac5c0 crc :-1 s=READY pgs=18 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:25:36.705 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:36.643+0000 7f57c1ce8b80 1 --2- 172.21.15.52:0/3425902470 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x5636cf425c00 0x5636cf447720 unknown :-1 s=CLOSED pgs=363 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:25:36.705 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:36.643+0000 7f5790ff9700 0 client.15237 ms_handle_reset on v2:172.21.15.173:6836/1396363827 2024-05-27T07:25:36.705 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:36.643+0000 7f57c1ce8b80 1 --2- 172.21.15.52:0/3425902470 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5636cf2a1df0 0x5636cf4485a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:25:36.705 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:36.643+0000 7f57c1ce8b80 1 --2- 172.21.15.52:0/3425902470 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5636cf41ff30 0x5636cf4471e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:25:36.705 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:36.643+0000 7f57c1ce8b80 1 -- 172.21.15.52:0/3425902470 >> 172.21.15.52:0/3425902470 conn(0x5636cf2fcaa0 msgr2=0x5636cf4146a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:25:36.705 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:36.643+0000 7f57c1ce8b80 1 -- 172.21.15.52:0/3425902470 shutdown_connections 2024-05-27T07:25:36.705 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:36.643+0000 7f57c1ce8b80 1 -- 172.21.15.52:0/3425902470 wait complete. 2024-05-27T07:25:36.705 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:36.643+0000 7f57c1ce8b80 20 client.15237 trim_cache size 0 max 16384 2024-05-27T07:25:36.705 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:36.644+0000 7f57b32d3700 1 -- 172.21.15.52:0/913753868 >> [v2:172.21.15.52:6800/3513075805,v1:172.21.15.52:6801/3513075805] conn(0x7f57940b2880 msgr2=0x7f57940b4d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:25:36.705 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:36.644+0000 7f57b32d3700 1 --2- 172.21.15.52:0/913753868 >> [v2:172.21.15.52:6800/3513075805,v1:172.21.15.52:6801/3513075805] conn(0x7f57940b2880 0x7f57940b4d90 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7f57a0000c00 tx=0x7f57a001a040 comp rx=0 tx=0).stop 2024-05-27T07:25:36.705 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:36.644+0000 7f57b32d3700 1 -- 172.21.15.52:0/913753868 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f57ac1045c0 msgr2=0x7f57ac19aea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:25:36.706 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:36.644+0000 7f57b32d3700 1 --2- 172.21.15.52:0/913753868 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f57ac1045c0 0x7f57ac19aea0 secure :-1 s=READY pgs=359 cs=0 l=1 rev1=1 crypto rx=0x7f579c0060b0 tx=0x7f579c007f90 comp rx=0 tx=0).stop 2024-05-27T07:25:36.706 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:36.644+0000 7f57b32d3700 1 -- 172.21.15.52:0/913753868 shutdown_connections 2024-05-27T07:25:36.706 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:36.644+0000 7f57b32d3700 1 --2- 172.21.15.52:0/913753868 >> [v2:172.21.15.52:6800/3513075805,v1:172.21.15.52:6801/3513075805] conn(0x7f57940b2880 0x7f57940b4d90 unknown :-1 s=CLOSED pgs=222 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:25:36.706 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:36.644+0000 7f57b32d3700 1 --2- 172.21.15.52:0/913753868 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f57ac109840 0x7f57ac19f6d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:25:36.706 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:36.644+0000 7f57b32d3700 1 --2- 172.21.15.52:0/913753868 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f57ac1045c0 0x7f57ac19aea0 unknown :-1 s=CLOSED pgs=359 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:25:36.706 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:36.644+0000 7f57b32d3700 1 --2- 172.21.15.52:0/913753868 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f57ac108ef0 0x7f57ac19b440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:25:36.706 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:36.644+0000 7f57b32d3700 1 -- 172.21.15.52:0/913753868 >> 172.21.15.52:0/913753868 conn(0x7f57ac0ffe90 msgr2=0x7f57ac10a7a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:25:36.706 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:36.644+0000 7f57b32d3700 1 -- 172.21.15.52:0/913753868 shutdown_connections 2024-05-27T07:25:36.706 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:36.644+0000 7f57b32d3700 1 -- 172.21.15.52:0/913753868 wait complete. 2024-05-27T07:25:37.351 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:25:37 smithi052 ceph-mon[31955]: pgmap v413: 129 pgs: 129 active+clean; 647 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 11 op/s 2024-05-27T07:25:37.351 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:25:37 smithi173 ceph-mon[37000]: pgmap v413: 129 pgs: 129 active+clean; 647 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 11 op/s 2024-05-27T07:25:37.380 DEBUG:teuthology.orchestra.run.smithi052:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:38f9d900c83c63799fbdbe61acc9a11b0d3554a6 shell --fsid 73df2636-1bf8-11ef-bc9b-c7b262605968 -- ceph tell mds.e status 2024-05-27T07:25:37.440 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:25:37 smithi105 ceph-mon[35977]: pgmap v413: 129 pgs: 129 active+clean; 647 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 11 op/s 2024-05-27T07:25:37.632 INFO:teuthology.orchestra.run.smithi052.stderr:Inferring config /var/lib/ceph/73df2636-1bf8-11ef-bc9b-c7b262605968/mon.a/config 2024-05-27T07:25:39.315 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:39.325+0000 7f92b5b2f700 1 -- 172.21.15.52:0/2798302723 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f92b0108730 msgr2=0x7f92b0110090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:25:39.315 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:39.325+0000 7f92b5b2f700 1 --2- 172.21.15.52:0/2798302723 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f92b0108730 0x7f92b0110090 secure :-1 s=READY pgs=364 cs=0 l=1 rev1=1 crypto rx=0x7f92a800b0d0 tx=0x7f92a800b3e0 comp rx=0 tx=0).stop 2024-05-27T07:25:39.315 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:39.325+0000 7f92b5b2f700 1 -- 172.21.15.52:0/2798302723 shutdown_connections 2024-05-27T07:25:39.315 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:39.325+0000 7f92b5b2f700 1 --2- 172.21.15.52:0/2798302723 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f92b0108730 0x7f92b0110090 unknown :-1 s=CLOSED pgs=364 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:25:39.315 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:39.325+0000 7f92b5b2f700 1 --2- 172.21.15.52:0/2798302723 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f92b01034b0 0x7f92b01038c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:25:39.315 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:39.325+0000 7f92b5b2f700 1 --2- 172.21.15.52:0/2798302723 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f92b0107de0 0x7f92b01081f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:25:39.315 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:39.325+0000 7f92b5b2f700 1 -- 172.21.15.52:0/2798302723 >> 172.21.15.52:0/2798302723 conn(0x7f92b00fede0 msgr2=0x7f92b0101200 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:25:39.315 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:39.325+0000 7f92b5b2f700 1 -- 172.21.15.52:0/2798302723 shutdown_connections 2024-05-27T07:25:39.316 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:39.325+0000 7f92b5b2f700 1 -- 172.21.15.52:0/2798302723 wait complete. 2024-05-27T07:25:39.317 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:39.326+0000 7f92b5b2f700 1 Processor -- start 2024-05-27T07:25:39.317 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:39.326+0000 7f92b5b2f700 1 -- start start 2024-05-27T07:25:39.317 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:39.326+0000 7f92b5b2f700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f92b01034b0 0x7f92b019bc90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:25:39.317 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:39.326+0000 7f92b5b2f700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f92b0107de0 0x7f92b019c1d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:25:39.317 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:39.326+0000 7f92b5b2f700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f92b0108730 0x7f92b019c750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:25:39.318 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:39.326+0000 7f92b5b2f700 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x7f92b01133d0 con 0x7f92b0107de0 2024-05-27T07:25:39.318 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:39.326+0000 7f92b5b2f700 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f92b0113250 con 0x7f92b01034b0 2024-05-27T07:25:39.318 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:39.326+0000 7f92b5b2f700 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f92b0113550 con 0x7f92b0108730 2024-05-27T07:25:39.318 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:39.326+0000 7f92aeffd700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f92b0107de0 0x7f92b019c1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:25:39.318 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:39.326+0000 7f92aeffd700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f92b0107de0 0x7f92b019c1d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.52:36772/0 (socket says 172.21.15.52:36772) 2024-05-27T07:25:39.318 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:39.326+0000 7f92aeffd700 1 -- 172.21.15.52:0/3364701528 learned_addr learned my addr 172.21.15.52:0/3364701528 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-05-27T07:25:39.318 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:39.326+0000 7f92affff700 1 --2- 172.21.15.52:0/3364701528 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f92b0108730 0x7f92b019c750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:25:39.318 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:39.326+0000 7f92af7fe700 1 --2- 172.21.15.52:0/3364701528 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f92b01034b0 0x7f92b019bc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:25:39.318 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:39.327+0000 7f92aeffd700 1 -- 172.21.15.52:0/3364701528 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f92b0108730 msgr2=0x7f92b019c750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:25:39.318 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:39.327+0000 7f92aeffd700 1 --2- 172.21.15.52:0/3364701528 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f92b0108730 0x7f92b019c750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:25:39.318 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:39.327+0000 7f92aeffd700 1 -- 172.21.15.52:0/3364701528 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f92b01034b0 msgr2=0x7f92b019bc90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:25:39.318 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:39.327+0000 7f92aeffd700 1 --2- 172.21.15.52:0/3364701528 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f92b01034b0 0x7f92b019bc90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:25:39.319 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:39.327+0000 7f92aeffd700 1 -- 172.21.15.52:0/3364701528 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f92b01a7000 con 0x7f92b0107de0 2024-05-27T07:25:39.319 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:39.327+0000 7f92affff700 1 --2- 172.21.15.52:0/3364701528 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f92b0108730 0x7f92b019c750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-27T07:25:39.319 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:39.327+0000 7f92aeffd700 1 --2- 172.21.15.52:0/3364701528 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f92b0107de0 0x7f92b019c1d0 secure :-1 s=READY pgs=365 cs=0 l=1 rev1=1 crypto rx=0x7f92a400b500 tx=0x7f92a400b8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:25:39.319 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:39.327+0000 7f92acff9700 1 -- 172.21.15.52:0/3364701528 <== mon.0 v2:172.21.15.52:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f92a4015020 con 0x7f92b0107de0 2024-05-27T07:25:39.319 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:39.327+0000 7f92b5b2f700 1 -- 172.21.15.52:0/3364701528 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f92b01a72f0 con 0x7f92b0107de0 2024-05-27T07:25:39.319 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:39.327+0000 7f92acff9700 1 -- 172.21.15.52:0/3364701528 <== mon.0 v2:172.21.15.52:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f92a4011ce0 con 0x7f92b0107de0 2024-05-27T07:25:39.319 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:39.327+0000 7f92acff9700 1 -- 172.21.15.52:0/3364701528 <== mon.0 v2:172.21.15.52:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f92a4010d40 con 0x7f92b0107de0 2024-05-27T07:25:39.319 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:39.327+0000 7f92b5b2f700 1 -- 172.21.15.52:0/3364701528 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f92b01a7880 con 0x7f92b0107de0 2024-05-27T07:25:39.319 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:39.330+0000 7f92acff9700 1 -- 172.21.15.52:0/3364701528 <== mon.0 v2:172.21.15.52:3300/0 4 ==== mgrmap(e 18) ==== 149652+0+0 (secure 0 0 0) 0x7f92a4018020 con 0x7f92b0107de0 2024-05-27T07:25:39.320 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:39.331+0000 7f92acff9700 1 --2- 172.21.15.52:0/3364701528 >> [v2:172.21.15.52:6800/3513075805,v1:172.21.15.52:6801/3513075805] conn(0x7f92980b2880 0x7f92980b4d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:25:39.320 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:39.331+0000 7f92af7fe700 1 --2- 172.21.15.52:0/3364701528 >> [v2:172.21.15.52:6800/3513075805,v1:172.21.15.52:6801/3513075805] conn(0x7f92980b2880 0x7f92980b4d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:25:39.321 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:39.331+0000 7f92acff9700 1 -- 172.21.15.52:0/3364701528 <== mon.0 v2:172.21.15.52:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9121+0+0 (secure 0 0 0) 0x7f92a40e1e40 con 0x7f92b0107de0 2024-05-27T07:25:39.321 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:39.331+0000 7f92c4544b80 1 Processor -- start 2024-05-27T07:25:39.321 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:39.331+0000 7f92c4544b80 1 -- start start 2024-05-27T07:25:39.322 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:39.331+0000 7f92c4544b80 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x560fee0b0df0 0x560fee238b50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:25:39.322 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:39.331+0000 7f92af7fe700 1 --2- 172.21.15.52:0/3364701528 >> [v2:172.21.15.52:6800/3513075805,v1:172.21.15.52:6801/3513075805] conn(0x7f92980b2880 0x7f92980b4d90 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7f92a0005fd0 tx=0x7f92a000a560 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:25:39.322 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:39.331+0000 7f92c4544b80 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x560fee239090 0x560fee232e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:25:39.322 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:39.331+0000 7f92c4544b80 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x560fee233340 0x560fee233750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:25:39.322 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:39.331+0000 7f92c4544b80 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x560fee15dc40 con 0x560fee0b0df0 2024-05-27T07:25:39.323 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:39.331+0000 7f92c4544b80 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x560fee14d7e0 con 0x560fee239090 2024-05-27T07:25:39.323 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:39.331+0000 7f92c4544b80 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x560fee151820 con 0x560fee233340 2024-05-27T07:25:39.323 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:39.331+0000 7f92af7fe700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x560fee0b0df0 0x560fee238b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:25:39.323 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:39.331+0000 7f92af7fe700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x560fee0b0df0 0x560fee238b50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.52:36780/0 (socket says 172.21.15.52:36780) 2024-05-27T07:25:39.323 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:39.331+0000 7f92af7fe700 1 -- 172.21.15.52:0/1857835878 learned_addr learned my addr 172.21.15.52:0/1857835878 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-05-27T07:25:39.323 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:39.332+0000 7f92aeffd700 1 --2- 172.21.15.52:0/1857835878 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x560fee239090 0x560fee232e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:25:39.323 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:39.332+0000 7f92affff700 1 --2- 172.21.15.52:0/1857835878 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x560fee233340 0x560fee233750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:25:39.323 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:39.332+0000 7f92af7fe700 1 -- 172.21.15.52:0/1857835878 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x560fee233340 msgr2=0x560fee233750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:25:39.323 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:39.332+0000 7f92af7fe700 1 --2- 172.21.15.52:0/1857835878 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x560fee233340 0x560fee233750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:25:39.323 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:39.332+0000 7f92af7fe700 1 -- 172.21.15.52:0/1857835878 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x560fee239090 msgr2=0x560fee232e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:25:39.323 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:39.332+0000 7f92af7fe700 1 --2- 172.21.15.52:0/1857835878 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x560fee239090 0x560fee232e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:25:39.323 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:39.332+0000 7f92af7fe700 1 -- 172.21.15.52:0/1857835878 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x560fee09d880 con 0x560fee0b0df0 2024-05-27T07:25:39.323 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:39.332+0000 7f92aeffd700 1 --2- 172.21.15.52:0/1857835878 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x560fee239090 0x560fee232e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-27T07:25:39.324 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:39.332+0000 7f92af7fe700 1 --2- 172.21.15.52:0/1857835878 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x560fee0b0df0 0x560fee238b50 secure :-1 s=READY pgs=366 cs=0 l=1 rev1=1 crypto rx=0x7f92a000ec80 tx=0x7f92a0011000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2b3aad7485b970d1 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:25:39.324 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:39.332+0000 7f9295ffb700 1 -- 172.21.15.52:0/1857835878 <== mon.0 v2:172.21.15.52:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f92a0021070 con 0x560fee0b0df0 2024-05-27T07:25:39.324 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:39.332+0000 7f9295ffb700 1 -- 172.21.15.52:0/1857835878 <== mon.0 v2:172.21.15.52:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f92a0010cd0 con 0x560fee0b0df0 2024-05-27T07:25:39.324 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:39.332+0000 7f9295ffb700 1 -- 172.21.15.52:0/1857835878 <== mon.0 v2:172.21.15.52:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f92a000f620 con 0x560fee0b0df0 2024-05-27T07:25:39.327 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:39.333+0000 7f92c4544b80 1 -- 172.21.15.52:0/1857835878 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x560fee0b0df0 msgr2=0x560fee238b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:25:39.327 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:39.333+0000 7f92c4544b80 1 --2- 172.21.15.52:0/1857835878 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x560fee0b0df0 0x560fee238b50 secure :-1 s=READY pgs=366 cs=0 l=1 rev1=1 crypto rx=0x7f92a000ec80 tx=0x7f92a0011000 comp rx=0 tx=0).stop 2024-05-27T07:25:39.327 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:39.333+0000 7f92c4544b80 1 -- 172.21.15.52:0/1857835878 shutdown_connections 2024-05-27T07:25:39.327 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:39.333+0000 7f92c4544b80 1 --2- 172.21.15.52:0/1857835878 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x560fee233340 0x560fee233750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:25:39.327 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:39.333+0000 7f92c4544b80 1 --2- 172.21.15.52:0/1857835878 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x560fee0b0df0 0x560fee238b50 unknown :-1 s=CLOSED pgs=366 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:25:39.327 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:39.333+0000 7f92c4544b80 1 --2- 172.21.15.52:0/1857835878 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x560fee239090 0x560fee232e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:25:39.327 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:39.333+0000 7f92c4544b80 1 -- 172.21.15.52:0/1857835878 >> 172.21.15.52:0/1857835878 conn(0x560fee10baa0 msgr2=0x560fee2295f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:25:39.327 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:39.333+0000 7f92c4544b80 1 -- 172.21.15.52:0/1857835878 shutdown_connections 2024-05-27T07:25:39.327 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:39.333+0000 7f92c4544b80 1 -- 172.21.15.52:0/1857835878 wait complete. 2024-05-27T07:25:39.327 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:39.334+0000 7f92c4544b80 1 -- start start 2024-05-27T07:25:39.327 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:39.334+0000 7f92c4544b80 1 Processor -- start 2024-05-27T07:25:39.327 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:39.334+0000 7f92c4544b80 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x560fee0b0df0 0x560fee23c6c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:25:39.328 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:39.334+0000 7f92c4544b80 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x560fee233340 0x560fee23cc00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:25:39.328 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:39.334+0000 7f92c4544b80 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x560fee239090 0x560fee23d140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:25:39.328 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:39.334+0000 7f92c4544b80 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x560fee053f50 con 0x560fee239090 2024-05-27T07:25:39.328 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:39.334+0000 7f92c4544b80 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x560fee0ef770 con 0x560fee0b0df0 2024-05-27T07:25:39.328 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:39.334+0000 7f92c4544b80 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x560fee23fac0 con 0x560fee233340 2024-05-27T07:25:39.328 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:39.334+0000 7f92c4544b80 10 client.0 _ll_register_callbacks cb 0x560fee0e88d0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T07:25:39.328 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:39.334+0000 7f92affff700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x560fee239090 0x560fee23d140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:25:39.328 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:39.334+0000 7f92affff700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x560fee239090 0x560fee23d140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.52:36794/0 (socket says 172.21.15.52:36794) 2024-05-27T07:25:39.328 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:39.334+0000 7f92affff700 1 -- 172.21.15.52:0/1439827229 learned_addr learned my addr 172.21.15.52:0/1439827229 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-05-27T07:25:39.328 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:39.335+0000 7f92af7fe700 1 --2- 172.21.15.52:0/1439827229 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x560fee0b0df0 0x560fee23c6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:25:39.328 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:39.335+0000 7f92aeffd700 1 --2- 172.21.15.52:0/1439827229 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x560fee233340 0x560fee23cc00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:25:39.328 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:39.335+0000 7f92affff700 1 -- 172.21.15.52:0/1439827229 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x560fee233340 msgr2=0x560fee23cc00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:25:39.328 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:39.335+0000 7f92affff700 1 --2- 172.21.15.52:0/1439827229 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x560fee233340 0x560fee23cc00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:25:39.329 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:39.335+0000 7f92affff700 1 -- 172.21.15.52:0/1439827229 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x560fee0b0df0 msgr2=0x560fee23c6c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:25:39.329 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:39.335+0000 7f92affff700 1 --2- 172.21.15.52:0/1439827229 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x560fee0b0df0 0x560fee23c6c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:25:39.329 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:39.335+0000 7f92affff700 1 -- 172.21.15.52:0/1439827229 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({osdmap=0}) -- 0x560fee0ec640 con 0x560fee239090 2024-05-27T07:25:39.329 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:39.335+0000 7f92af7fe700 1 --2- 172.21.15.52:0/1439827229 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x560fee0b0df0 0x560fee23c6c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-27T07:25:39.329 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:39.335+0000 7f92affff700 1 --2- 172.21.15.52:0/1439827229 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x560fee239090 0x560fee23d140 secure :-1 s=READY pgs=367 cs=0 l=1 rev1=1 crypto rx=0x7f92a8015000 tx=0x7f92a8004db0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=675183ed14141bbb server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:25:39.329 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:39.335+0000 7f928effd700 10 client.0 ms_handle_connect on v2:172.21.15.52:3300/0 2024-05-27T07:25:39.329 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:39.335+0000 7f928effd700 1 -- 172.21.15.52:0/1439827229 <== mon.0 v2:172.21.15.52:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f92a8021070 con 0x560fee239090 2024-05-27T07:25:39.329 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:39.335+0000 7f928effd700 1 -- 172.21.15.52:0/1439827229 <== mon.0 v2:172.21.15.52:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9121+0+0 (secure 0 0 0) 0x7f92a80079d0 con 0x560fee239090 2024-05-27T07:25:39.329 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:39.335+0000 7f92c4544b80 1 -- 172.21.15.52:0/1439827229 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f92a0010cd0 con 0x560fee239090 2024-05-27T07:25:39.329 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:39.335+0000 7f928effd700 1 -- 172.21.15.52:0/1439827229 <== mon.0 v2:172.21.15.52:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f92a802e050 con 0x560fee239090 2024-05-27T07:25:39.329 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:39.335+0000 7f92c4544b80 10 client.15249 fetch_fsmap learned FSMap version 29 2024-05-27T07:25:39.329 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:39.335+0000 7f92c4544b80 1 -- 172.21.15.52:0/1439827229 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x560fee24a420 con 0x560fee239090 2024-05-27T07:25:39.329 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:39.336+0000 7f928effd700 1 -- 172.21.15.52:0/1439827229 <== mon.0 v2:172.21.15.52:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f92a0010cd0 con 0x560fee239090 2024-05-27T07:25:39.329 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:39.336+0000 7f928effd700 1 -- 172.21.15.52:0/1439827229 <== mon.0 v2:172.21.15.52:3300/0 5 ==== fsmap(e 29) ==== 8362+0+0 (secure 0 0 0) 0x7f92a8011070 con 0x560fee239090 2024-05-27T07:25:39.330 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:39.336+0000 7f92c4544b80 10 client.15249 fetch_fsmap finished waiting for FSMap version 29 2024-05-27T07:25:39.330 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:39.336+0000 7f92c4544b80 10 client.15249 resolve_mds: resolved name 'e' to daemon mds.e 2024-05-27T07:25:39.330 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:39.336+0000 7f92c4544b80 20 client.15249 populate_metadata read hostname 'smithi052' 2024-05-27T07:25:39.330 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:39.336+0000 7f92c4544b80 1 --2- 172.21.15.52:0/1439827229 >> [v2:172.21.15.105:6834/3298498384,v1:172.21.15.105:6835/3298498384] conn(0x560fee24ca50 0x560fee24dec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:25:39.330 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:39.336+0000 7f92c4544b80 4 client.15249 mds_command: new command op to 24473 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-27T07:25:39.330 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:39.336+0000 7f92c4544b80 1 -- 172.21.15.52:0/1439827229 --> [v2:172.21.15.105:6834/3298498384,v1:172.21.15.105:6835/3298498384] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x560fee1fa9a0 con 0x560fee24ca50 2024-05-27T07:25:39.330 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:39.336+0000 7f92af7fe700 1 --2- 172.21.15.52:0/1439827229 >> [v2:172.21.15.105:6834/3298498384,v1:172.21.15.105:6835/3298498384] conn(0x560fee24ca50 0x560fee24dec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:25:39.330 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:39.337+0000 7f928effd700 10 client.15249 ms_handle_connect on v2:172.21.15.105:6834/3298498384 2024-05-27T07:25:39.330 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:39.337+0000 7f92af7fe700 1 --2- 172.21.15.52:0/1439827229 >> [v2:172.21.15.105:6834/3298498384,v1:172.21.15.105:6835/3298498384] conn(0x560fee24ca50 0x560fee24dec0 crc :-1 s=READY pgs=16 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=80f084f1d990521c server_cookie=ea78c8951591fcb7 in_seq=0 out_seq=0 2024-05-27T07:25:39.331 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:39.338+0000 7f928effd700 1 -- 172.21.15.52:0/1439827229 <== mds.3 v2:172.21.15.105:6834/3298498384 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24041 (crc 0 0 0) 0x560fee1fa9a0 con 0x560fee24ca50 2024-05-27T07:25:39.331 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:39.338+0000 7f928effd700 10 client.15249 handle_command_reply: tid=0 2024-05-27T07:25:39.331 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:39.338+0000 7f92c4544b80 1 client.15249 shutdown 2024-05-27T07:25:39.331 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:39.338+0000 7f92c4544b80 1 -- 172.21.15.52:0/1439827229 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x560fee239090 msgr2=0x560fee23d140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:25:39.331 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:39.338+0000 7f92c4544b80 1 --2- 172.21.15.52:0/1439827229 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x560fee239090 0x560fee23d140 secure :-1 s=READY pgs=367 cs=0 l=1 rev1=1 crypto rx=0x7f92a8015000 tx=0x7f92a8004db0 comp rx=0 tx=0).stop 2024-05-27T07:25:39.331 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:39.338+0000 7f92c4544b80 1 -- 172.21.15.52:0/1439827229 shutdown_connections 2024-05-27T07:25:39.332 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:39.338+0000 7f92c4544b80 1 --2- 172.21.15.52:0/1439827229 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x560fee239090 0x560fee23d140 unknown :-1 s=CLOSED pgs=367 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:25:39.332 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:39.338+0000 7f92c4544b80 1 --2- 172.21.15.52:0/1439827229 >> [v2:172.21.15.105:6834/3298498384,v1:172.21.15.105:6835/3298498384] conn(0x560fee24ca50 0x560fee24dec0 crc :-1 s=READY pgs=16 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:25:39.332 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:39.338+0000 7f92c4544b80 1 --2- 172.21.15.52:0/1439827229 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x560fee0b0df0 0x560fee23c6c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:25:39.332 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:39.338+0000 7f92c4544b80 1 --2- 172.21.15.52:0/1439827229 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x560fee233340 0x560fee23cc00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:25:39.332 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:39.338+0000 7f92c4544b80 1 -- 172.21.15.52:0/1439827229 >> 172.21.15.52:0/1439827229 conn(0x560fee10baa0 msgr2=0x560fee22d9e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:25:39.332 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:39.338+0000 7f928effd700 0 client.15249 ms_handle_reset on v2:172.21.15.105:6834/3298498384 2024-05-27T07:25:39.332 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:39.338+0000 7f92c4544b80 1 -- 172.21.15.52:0/1439827229 shutdown_connections 2024-05-27T07:25:39.332 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:39.338+0000 7f92c4544b80 1 -- 172.21.15.52:0/1439827229 wait complete. 2024-05-27T07:25:39.332 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:39.338+0000 7f92c4544b80 20 client.15249 trim_cache size 0 max 16384 2024-05-27T07:25:39.349 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:39.359+0000 7f92c4544b80 1 Processor -- start 2024-05-27T07:25:39.349 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:39.359+0000 7f92c4544b80 1 -- start start 2024-05-27T07:25:39.350 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:25:39 smithi052 ceph-mon[31955]: pgmap v414: 129 pgs: 129 active+clean; 647 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-05-27T07:25:39.351 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:39.360+0000 7f92c4544b80 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x560fee0b0df0 0x560fee248d20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:25:39.351 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:39.360+0000 7f92c4544b80 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x560fee233340 0x560fee23cf80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:25:39.351 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:39.360+0000 7f92c4544b80 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x560fee239090 0x560fee2377c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:25:39.351 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:39.360+0000 7f92c4544b80 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x560fee24c200 con 0x560fee233340 2024-05-27T07:25:39.351 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:39.360+0000 7f92c4544b80 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x560fee24c8d0 con 0x560fee239090 2024-05-27T07:25:39.351 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:39.360+0000 7f92c4544b80 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x560fee24cfa0 con 0x560fee0b0df0 2024-05-27T07:25:39.351 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:39.360+0000 7f92aeffd700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x560fee233340 0x560fee23cf80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:25:39.351 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:39.360+0000 7f92aeffd700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x560fee233340 0x560fee23cf80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.52:36798/0 (socket says 172.21.15.52:36798) 2024-05-27T07:25:39.352 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:39.360+0000 7f92aeffd700 1 -- 172.21.15.52:0/3736825468 learned_addr learned my addr 172.21.15.52:0/3736825468 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-05-27T07:25:39.352 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:39.360+0000 7f92af7fe700 1 --2- 172.21.15.52:0/3736825468 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x560fee0b0df0 0x560fee248d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:25:39.352 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:39.360+0000 7f92affff700 1 --2- 172.21.15.52:0/3736825468 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x560fee239090 0x560fee2377c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:25:39.352 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:39.360+0000 7f92aeffd700 1 -- 172.21.15.52:0/3736825468 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x560fee0b0df0 msgr2=0x560fee248d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:25:39.352 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:39.360+0000 7f92aeffd700 1 --2- 172.21.15.52:0/3736825468 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x560fee0b0df0 0x560fee248d20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:25:39.352 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:39.360+0000 7f92aeffd700 1 -- 172.21.15.52:0/3736825468 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x560fee239090 msgr2=0x560fee2377c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:25:39.352 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:39.360+0000 7f92aeffd700 1 --2- 172.21.15.52:0/3736825468 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x560fee239090 0x560fee2377c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:25:39.352 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:39.360+0000 7f92aeffd700 1 -- 172.21.15.52:0/3736825468 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x560fee22ae40 con 0x560fee233340 2024-05-27T07:25:39.352 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:39.361+0000 7f92aeffd700 1 --2- 172.21.15.52:0/3736825468 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x560fee233340 0x560fee23cf80 secure :-1 s=READY pgs=368 cs=0 l=1 rev1=1 crypto rx=0x7f92a40a8d80 tx=0x7f92a4004860 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d7abd16143d2d866 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:25:39.352 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:39.361+0000 7f92957fa700 1 -- 172.21.15.52:0/3736825468 <== mon.0 v2:172.21.15.52:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f92a40c3070 con 0x560fee233340 2024-05-27T07:25:39.352 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:39.361+0000 7f92957fa700 1 -- 172.21.15.52:0/3736825468 <== mon.0 v2:172.21.15.52:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f92a40a9080 con 0x560fee233340 2024-05-27T07:25:39.352 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:39.361+0000 7f92957fa700 1 -- 172.21.15.52:0/3736825468 <== mon.0 v2:172.21.15.52:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f92a40b1cb0 con 0x560fee233340 2024-05-27T07:25:39.353 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:25:39 smithi173 ceph-mon[37000]: pgmap v414: 129 pgs: 129 active+clean; 647 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-05-27T07:25:39.356 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:39.362+0000 7f92c4544b80 1 -- 172.21.15.52:0/3736825468 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x560fee233340 msgr2=0x560fee23cf80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:25:39.356 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:39.362+0000 7f92c4544b80 1 --2- 172.21.15.52:0/3736825468 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x560fee233340 0x560fee23cf80 secure :-1 s=READY pgs=368 cs=0 l=1 rev1=1 crypto rx=0x7f92a40a8d80 tx=0x7f92a4004860 comp rx=0 tx=0).stop 2024-05-27T07:25:39.356 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:39.362+0000 7f92c4544b80 1 -- 172.21.15.52:0/3736825468 shutdown_connections 2024-05-27T07:25:39.356 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:39.362+0000 7f92c4544b80 1 --2- 172.21.15.52:0/3736825468 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x560fee239090 0x560fee2377c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:25:39.356 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:39.362+0000 7f92c4544b80 1 --2- 172.21.15.52:0/3736825468 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x560fee0b0df0 0x560fee248d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:25:39.356 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:39.362+0000 7f92c4544b80 1 --2- 172.21.15.52:0/3736825468 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x560fee233340 0x560fee23cf80 unknown :-1 s=CLOSED pgs=368 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:25:39.356 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:39.362+0000 7f92c4544b80 1 -- 172.21.15.52:0/3736825468 >> 172.21.15.52:0/3736825468 conn(0x560fee10baa0 msgr2=0x560fee229170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:25:39.356 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:39.362+0000 7f92c4544b80 1 -- 172.21.15.52:0/3736825468 shutdown_connections 2024-05-27T07:25:39.356 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:39.362+0000 7f92c4544b80 1 -- 172.21.15.52:0/3736825468 wait complete. 2024-05-27T07:25:39.357 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:39.362+0000 7f92c4544b80 1 -- start start 2024-05-27T07:25:39.357 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:39.362+0000 7f92c4544b80 1 Processor -- start 2024-05-27T07:25:39.357 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:39.363+0000 7f92c4544b80 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x560fee0b0df0 0x560fee2326c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:25:39.357 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:39.363+0000 7f92c4544b80 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x560fee233340 0x560fee257330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:25:39.357 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:39.363+0000 7f92c4544b80 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x560fee239090 0x560fee257870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:25:39.357 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:39.363+0000 7f92c4544b80 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x560fee24dd40 con 0x560fee239090 2024-05-27T07:25:39.357 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:39.363+0000 7f92c4544b80 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x560fee24d670 con 0x560fee233340 2024-05-27T07:25:39.357 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:39.363+0000 7f92c4544b80 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x560fee24e410 con 0x560fee0b0df0 2024-05-27T07:25:39.357 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:39.363+0000 7f92c4544b80 10 client.0 _ll_register_callbacks cb 0x560fee0e88d0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T07:25:39.357 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:39.363+0000 7f92affff700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x560fee239090 0x560fee257870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:25:39.357 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:39.363+0000 7f92affff700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x560fee239090 0x560fee257870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.52:36804/0 (socket says 172.21.15.52:36804) 2024-05-27T07:25:39.357 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:39.363+0000 7f92affff700 1 -- 172.21.15.52:0/431888803 learned_addr learned my addr 172.21.15.52:0/431888803 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-05-27T07:25:39.358 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:39.363+0000 7f92af7fe700 1 --2- 172.21.15.52:0/431888803 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x560fee0b0df0 0x560fee2326c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:25:39.358 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:39.363+0000 7f92aeffd700 1 --2- 172.21.15.52:0/431888803 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x560fee233340 0x560fee257330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:25:39.358 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:39.363+0000 7f92affff700 1 -- 172.21.15.52:0/431888803 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x560fee0b0df0 msgr2=0x560fee2326c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:25:39.358 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:39.363+0000 7f92affff700 1 --2- 172.21.15.52:0/431888803 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x560fee0b0df0 0x560fee2326c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:25:39.358 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:39.363+0000 7f92affff700 1 -- 172.21.15.52:0/431888803 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x560fee233340 msgr2=0x560fee257330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:25:39.358 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:39.363+0000 7f92affff700 1 --2- 172.21.15.52:0/431888803 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x560fee233340 0x560fee257330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:25:39.358 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:39.363+0000 7f92affff700 1 -- 172.21.15.52:0/431888803 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({osdmap=0}) -- 0x560fee22fa50 con 0x560fee239090 2024-05-27T07:25:39.358 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:39.364+0000 7f92affff700 1 --2- 172.21.15.52:0/431888803 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x560fee239090 0x560fee257870 secure :-1 s=READY pgs=369 cs=0 l=1 rev1=1 crypto rx=0x7f92a800b0d0 tx=0x7f92a8005f10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=42456360a9ea65f8 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:25:39.358 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:39.364+0000 7f928effd700 10 client.0 ms_handle_connect on v2:172.21.15.52:3300/0 2024-05-27T07:25:39.358 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:39.364+0000 7f928effd700 1 -- 172.21.15.52:0/431888803 <== mon.0 v2:172.21.15.52:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f92a8021070 con 0x560fee239090 2024-05-27T07:25:39.358 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:39.364+0000 7f928effd700 1 -- 172.21.15.52:0/431888803 <== mon.0 v2:172.21.15.52:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9121+0+0 (secure 0 0 0) 0x7f92a80079e0 con 0x560fee239090 2024-05-27T07:25:39.359 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:39.364+0000 7f92c4544b80 1 -- 172.21.15.52:0/431888803 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f92a40a9080 con 0x560fee239090 2024-05-27T07:25:39.359 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:39.364+0000 7f928effd700 1 -- 172.21.15.52:0/431888803 <== mon.0 v2:172.21.15.52:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f92a8007be0 con 0x560fee239090 2024-05-27T07:25:39.359 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:39.364+0000 7f92c4544b80 10 client.15255 fetch_fsmap learned FSMap version 29 2024-05-27T07:25:39.359 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:39.364+0000 7f92c4544b80 1 -- 172.21.15.52:0/431888803 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x560fee2bcae0 con 0x560fee239090 2024-05-27T07:25:39.359 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:39.364+0000 7f928effd700 1 -- 172.21.15.52:0/431888803 <== mon.0 v2:172.21.15.52:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f92a40a9080 con 0x560fee239090 2024-05-27T07:25:39.359 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:39.365+0000 7f928effd700 1 -- 172.21.15.52:0/431888803 <== mon.0 v2:172.21.15.52:3300/0 5 ==== fsmap(e 29) ==== 8362+0+0 (secure 0 0 0) 0x7f92a8011070 con 0x560fee239090 2024-05-27T07:25:39.359 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:39.365+0000 7f92c4544b80 10 client.15255 fetch_fsmap finished waiting for FSMap version 29 2024-05-27T07:25:39.359 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:39.365+0000 7f92c4544b80 10 client.15255 resolve_mds: resolved name 'e' to daemon mds.e 2024-05-27T07:25:39.359 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:39.365+0000 7f92c4544b80 20 client.15255 populate_metadata read hostname 'smithi052' 2024-05-27T07:25:39.359 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:39.365+0000 7f92c4544b80 1 --2- 172.21.15.52:0/431888803 >> [v2:172.21.15.105:6834/3298498384,v1:172.21.15.105:6835/3298498384] conn(0x560fee2bef60 0x560fee2c03d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:25:39.359 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:39.365+0000 7f92c4544b80 4 client.15255 mds_command: new command op to 24473 tid=0 multi_id=0 [{"prefix": "status"}] 2024-05-27T07:25:39.359 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:39.365+0000 7f92c4544b80 1 -- 172.21.15.52:0/431888803 --> [v2:172.21.15.105:6834/3298498384,v1:172.21.15.105:6835/3298498384] -- command(tid 0: {"prefix": "status"}) -- 0x560fee15fb90 con 0x560fee2bef60 2024-05-27T07:25:39.359 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:39.365+0000 7f92af7fe700 1 --2- 172.21.15.52:0/431888803 >> [v2:172.21.15.105:6834/3298498384,v1:172.21.15.105:6835/3298498384] conn(0x560fee2bef60 0x560fee2c03d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:25:39.360 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:39.365+0000 7f92af7fe700 1 --2- 172.21.15.52:0/431888803 >> [v2:172.21.15.105:6834/3298498384,v1:172.21.15.105:6835/3298498384] conn(0x560fee2bef60 0x560fee2c03d0 crc :-1 s=READY pgs=17 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=61413f68805079b6 server_cookie=6ec664b70ae47bc5 in_seq=0 out_seq=0 2024-05-27T07:25:39.360 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:39.365+0000 7f928effd700 10 client.15255 ms_handle_connect on v2:172.21.15.105:6834/3298498384 2024-05-27T07:25:39.360 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:39.366+0000 7f928effd700 1 -- 172.21.15.52:0/431888803 <== mds.3 v2:172.21.15.105:6834/3298498384 1 ==== command_reply(tid 0: 0 ) ==== 8+0+327 (crc 0 0 0) 0x560fee15fb90 con 0x560fee2bef60 2024-05-27T07:25:39.360 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:39.366+0000 7f928effd700 10 client.15255 handle_command_reply: tid=0 2024-05-27T07:25:39.360 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:39.366+0000 7f92c4544b80 1 client.15255 shutdown 2024-05-27T07:25:39.360 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:39.366+0000 7f92c4544b80 1 -- 172.21.15.52:0/431888803 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x560fee239090 msgr2=0x560fee257870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:25:39.360 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:39.366+0000 7f92c4544b80 1 --2- 172.21.15.52:0/431888803 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x560fee239090 0x560fee257870 secure :-1 s=READY pgs=369 cs=0 l=1 rev1=1 crypto rx=0x7f92a800b0d0 tx=0x7f92a8005f10 comp rx=0 tx=0).stop 2024-05-27T07:25:39.360 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:39.366+0000 7f92c4544b80 1 -- 172.21.15.52:0/431888803 shutdown_connections 2024-05-27T07:25:39.360 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:39.366+0000 7f92c4544b80 1 --2- 172.21.15.52:0/431888803 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x560fee239090 0x560fee257870 unknown :-1 s=CLOSED pgs=369 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:25:39.360 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:39.366+0000 7f92c4544b80 1 --2- 172.21.15.52:0/431888803 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x560fee0b0df0 0x560fee2326c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:25:39.360 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:39.366+0000 7f92c4544b80 1 --2- 172.21.15.52:0/431888803 >> [v2:172.21.15.105:6834/3298498384,v1:172.21.15.105:6835/3298498384] conn(0x560fee2bef60 0x560fee2c03d0 crc :-1 s=READY pgs=17 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:25:39.360 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:39.366+0000 7f92c4544b80 1 --2- 172.21.15.52:0/431888803 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x560fee233340 0x560fee257330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:25:39.360 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:39.366+0000 7f928effd700 0 client.15255 ms_handle_reset on v2:172.21.15.105:6834/3298498384 2024-05-27T07:25:39.360 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:39.366+0000 7f92c4544b80 1 -- 172.21.15.52:0/431888803 >> 172.21.15.52:0/431888803 conn(0x560fee10baa0 msgr2=0x560fee2377c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:25:39.361 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:39.366+0000 7f92c4544b80 1 -- 172.21.15.52:0/431888803 shutdown_connections 2024-05-27T07:25:39.361 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:39.366+0000 7f92c4544b80 1 -- 172.21.15.52:0/431888803 wait complete. 2024-05-27T07:25:39.361 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:39.366+0000 7f92c4544b80 20 client.15255 trim_cache size 0 max 16384 2024-05-27T07:25:39.361 INFO:teuthology.orchestra.run.smithi052.stdout:{ 2024-05-27T07:25:39.361 INFO:teuthology.orchestra.run.smithi052.stdout: "cluster_fsid": "73df2636-1bf8-11ef-bc9b-c7b262605968", 2024-05-27T07:25:39.361 INFO:teuthology.orchestra.run.smithi052.stdout: "whoami": 3, 2024-05-27T07:25:39.361 INFO:teuthology.orchestra.run.smithi052.stdout: "id": 24473, 2024-05-27T07:25:39.361 INFO:teuthology.orchestra.run.smithi052.stdout: "want_state": "up:active", 2024-05-27T07:25:39.361 INFO:teuthology.orchestra.run.smithi052.stdout: "state": "up:active", 2024-05-27T07:25:39.361 INFO:teuthology.orchestra.run.smithi052.stdout: "fs_name": "cephfs", 2024-05-27T07:25:39.362 INFO:teuthology.orchestra.run.smithi052.stdout: "rank_uptime": 164.49577701000001, 2024-05-27T07:25:39.362 INFO:teuthology.orchestra.run.smithi052.stdout: "mdsmap_epoch": 29, 2024-05-27T07:25:39.362 INFO:teuthology.orchestra.run.smithi052.stdout: "osdmap_epoch": 90, 2024-05-27T07:25:39.362 INFO:teuthology.orchestra.run.smithi052.stdout: "osdmap_epoch_barrier": 88, 2024-05-27T07:25:39.362 INFO:teuthology.orchestra.run.smithi052.stdout: "uptime": 264.234563299 2024-05-27T07:25:39.362 INFO:teuthology.orchestra.run.smithi052.stdout:} 2024-05-27T07:25:39.363 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:39.367+0000 7f92b5b2f700 1 -- 172.21.15.52:0/3364701528 >> [v2:172.21.15.52:6800/3513075805,v1:172.21.15.52:6801/3513075805] conn(0x7f92980b2880 msgr2=0x7f92980b4d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:25:39.363 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:39.367+0000 7f92b5b2f700 1 --2- 172.21.15.52:0/3364701528 >> [v2:172.21.15.52:6800/3513075805,v1:172.21.15.52:6801/3513075805] conn(0x7f92980b2880 0x7f92980b4d90 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7f92a0005fd0 tx=0x7f92a000a560 comp rx=0 tx=0).stop 2024-05-27T07:25:39.363 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:39.367+0000 7f92b5b2f700 1 -- 172.21.15.52:0/3364701528 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f92b0107de0 msgr2=0x7f92b019c1d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:25:39.363 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:39.367+0000 7f92b5b2f700 1 --2- 172.21.15.52:0/3364701528 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f92b0107de0 0x7f92b019c1d0 secure :-1 s=READY pgs=365 cs=0 l=1 rev1=1 crypto rx=0x7f92a400b500 tx=0x7f92a400b8c0 comp rx=0 tx=0).stop 2024-05-27T07:25:39.363 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:39.367+0000 7f92b5b2f700 1 -- 172.21.15.52:0/3364701528 shutdown_connections 2024-05-27T07:25:39.363 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:39.367+0000 7f92b5b2f700 1 --2- 172.21.15.52:0/3364701528 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f92b0108730 0x7f92b019c750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:25:39.363 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:39.367+0000 7f92b5b2f700 1 --2- 172.21.15.52:0/3364701528 >> [v2:172.21.15.52:6800/3513075805,v1:172.21.15.52:6801/3513075805] conn(0x7f92980b2880 0x7f92980b4d90 unknown :-1 s=CLOSED pgs=223 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:25:39.363 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:39.367+0000 7f92b5b2f700 1 --2- 172.21.15.52:0/3364701528 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f92b01034b0 0x7f92b019bc90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:25:39.363 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:39.367+0000 7f92b5b2f700 1 --2- 172.21.15.52:0/3364701528 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f92b0107de0 0x7f92b019c1d0 unknown :-1 s=CLOSED pgs=365 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:25:39.363 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:39.367+0000 7f92b5b2f700 1 -- 172.21.15.52:0/3364701528 >> 172.21.15.52:0/3364701528 conn(0x7f92b00fede0 msgr2=0x7f92b0107550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:25:39.364 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:39.367+0000 7f92b5b2f700 1 -- 172.21.15.52:0/3364701528 shutdown_connections 2024-05-27T07:25:39.364 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:39.367+0000 7f92b5b2f700 1 -- 172.21.15.52:0/3364701528 wait complete. 2024-05-27T07:25:39.440 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:25:39 smithi105 ceph-mon[35977]: pgmap v414: 129 pgs: 129 active+clean; 647 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-05-27T07:25:40.068 DEBUG:teuthology.orchestra.run.smithi052:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:38f9d900c83c63799fbdbe61acc9a11b0d3554a6 shell --fsid 73df2636-1bf8-11ef-bc9b-c7b262605968 -- ceph tell mds.i status 2024-05-27T07:25:40.321 INFO:teuthology.orchestra.run.smithi052.stderr:Inferring config /var/lib/ceph/73df2636-1bf8-11ef-bc9b-c7b262605968/mon.a/config 2024-05-27T07:25:41.352 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:25:41 smithi173 ceph-mon[37000]: pgmap v415: 129 pgs: 129 active+clean; 647 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 9 op/s 2024-05-27T07:25:41.429 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:25:41 smithi052 ceph-mon[31955]: pgmap v415: 129 pgs: 129 active+clean; 647 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 9 op/s 2024-05-27T07:25:41.439 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:25:41 smithi105 ceph-mon[35977]: pgmap v415: 129 pgs: 129 active+clean; 647 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 9 op/s 2024-05-27T07:25:41.977 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:41.986+0000 7f0123db1700 1 -- 172.21.15.52:0/3506454521 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f011c107e00 msgr2=0x7f011c108210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:25:41.977 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:41.986+0000 7f0123db1700 1 --2- 172.21.15.52:0/3506454521 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f011c107e00 0x7f011c108210 secure :-1 s=READY pgs=370 cs=0 l=1 rev1=1 crypto rx=0x7f01100099c0 tx=0x7f0110009cd0 comp rx=0 tx=0).stop 2024-05-27T07:25:41.977 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:41.986+0000 7f0123db1700 1 -- 172.21.15.52:0/3506454521 shutdown_connections 2024-05-27T07:25:41.977 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:41.986+0000 7f0123db1700 1 --2- 172.21.15.52:0/3506454521 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f011c108750 0x7f011c1100b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:25:41.977 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:41.986+0000 7f0123db1700 1 --2- 172.21.15.52:0/3506454521 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f011c1034d0 0x7f011c1038e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:25:41.977 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:41.986+0000 7f0123db1700 1 --2- 172.21.15.52:0/3506454521 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f011c107e00 0x7f011c108210 unknown :-1 s=CLOSED pgs=370 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:25:41.978 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:41.986+0000 7f0123db1700 1 -- 172.21.15.52:0/3506454521 >> 172.21.15.52:0/3506454521 conn(0x7f011c0fef20 msgr2=0x7f011c101340 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:25:41.978 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:41.986+0000 7f0123db1700 1 -- 172.21.15.52:0/3506454521 shutdown_connections 2024-05-27T07:25:41.978 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:41.986+0000 7f0123db1700 1 -- 172.21.15.52:0/3506454521 wait complete. 2024-05-27T07:25:41.980 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:41.987+0000 7f0123db1700 1 Processor -- start 2024-05-27T07:25:41.980 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:41.987+0000 7f0123db1700 1 -- start start 2024-05-27T07:25:41.980 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:41.987+0000 7f0123db1700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f011c1034d0 0x7f011c19bca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:25:41.980 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:41.987+0000 7f0123db1700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f011c107e00 0x7f011c19c1e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:25:41.980 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:41.987+0000 7f0123db1700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f011c108750 0x7f011c19c760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:25:41.980 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:41.987+0000 7f0123db1700 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x7f011c113360 con 0x7f011c108750 2024-05-27T07:25:41.980 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:41.987+0000 7f0123db1700 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f011c1131e0 con 0x7f011c107e00 2024-05-27T07:25:41.980 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:41.987+0000 7f0123db1700 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f011c1134e0 con 0x7f011c1034d0 2024-05-27T07:25:41.980 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:41.987+0000 7f012234e700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f011c108750 0x7f011c19c760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:25:41.980 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:41.987+0000 7f012234e700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f011c108750 0x7f011c19c760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.52:48116/0 (socket says 172.21.15.52:48116) 2024-05-27T07:25:41.980 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:41.987+0000 7f012234e700 1 -- 172.21.15.52:0/27382024 learned_addr learned my addr 172.21.15.52:0/27382024 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-05-27T07:25:41.980 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:41.987+0000 7f012134c700 1 --2- 172.21.15.52:0/27382024 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f011c107e00 0x7f011c19c1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:25:41.981 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:41.987+0000 7f0121b4d700 1 --2- 172.21.15.52:0/27382024 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f011c1034d0 0x7f011c19bca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:25:41.981 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:41.988+0000 7f012234e700 1 -- 172.21.15.52:0/27382024 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f011c1034d0 msgr2=0x7f011c19bca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:25:41.981 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:41.988+0000 7f012234e700 1 --2- 172.21.15.52:0/27382024 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f011c1034d0 0x7f011c19bca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:25:41.981 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:41.988+0000 7f012234e700 1 -- 172.21.15.52:0/27382024 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f011c107e00 msgr2=0x7f011c19c1e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:25:41.981 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:41.988+0000 7f012234e700 1 --2- 172.21.15.52:0/27382024 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f011c107e00 0x7f011c19c1e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:25:41.981 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:41.988+0000 7f012234e700 1 -- 172.21.15.52:0/27382024 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f011c1a6f80 con 0x7f011c108750 2024-05-27T07:25:41.981 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:41.988+0000 7f0121b4d700 1 --2- 172.21.15.52:0/27382024 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f011c1034d0 0x7f011c19bca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-27T07:25:41.981 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:41.988+0000 7f012234e700 1 --2- 172.21.15.52:0/27382024 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f011c108750 0x7f011c19c760 secure :-1 s=READY pgs=371 cs=0 l=1 rev1=1 crypto rx=0x7f0114007d70 tx=0x7f011400a340 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:25:41.981 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:41.988+0000 7f010effd700 1 -- 172.21.15.52:0/27382024 <== mon.0 v2:172.21.15.52:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0114016020 con 0x7f011c108750 2024-05-27T07:25:41.981 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:41.988+0000 7f0123db1700 1 -- 172.21.15.52:0/27382024 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f011c1a7270 con 0x7f011c108750 2024-05-27T07:25:41.981 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:41.988+0000 7f010effd700 1 -- 172.21.15.52:0/27382024 <== mon.0 v2:172.21.15.52:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f011400ad40 con 0x7f011c108750 2024-05-27T07:25:41.981 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:41.988+0000 7f010effd700 1 -- 172.21.15.52:0/27382024 <== mon.0 v2:172.21.15.52:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f01140115e0 con 0x7f011c108750 2024-05-27T07:25:41.981 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:41.988+0000 7f0123db1700 1 -- 172.21.15.52:0/27382024 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f011c1a7730 con 0x7f011c108750 2024-05-27T07:25:41.982 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:41.991+0000 7f010effd700 1 -- 172.21.15.52:0/27382024 <== mon.0 v2:172.21.15.52:3300/0 4 ==== mgrmap(e 18) ==== 149652+0+0 (secure 0 0 0) 0x7f0114019020 con 0x7f011c108750 2024-05-27T07:25:41.982 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:41.992+0000 7f010effd700 1 --2- 172.21.15.52:0/27382024 >> [v2:172.21.15.52:6800/3513075805,v1:172.21.15.52:6801/3513075805] conn(0x7f01080b2880 0x7f01080b4d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:25:41.984 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:41.992+0000 7f0121b4d700 1 --2- 172.21.15.52:0/27382024 >> [v2:172.21.15.52:6800/3513075805,v1:172.21.15.52:6801/3513075805] conn(0x7f01080b2880 0x7f01080b4d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:25:41.984 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:41.992+0000 7f010effd700 1 -- 172.21.15.52:0/27382024 <== mon.0 v2:172.21.15.52:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9121+0+0 (secure 0 0 0) 0x7f01140e2560 con 0x7f011c108750 2024-05-27T07:25:41.984 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:41.992+0000 7f01327c6b80 1 Processor -- start 2024-05-27T07:25:41.984 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:41.992+0000 7f01327c6b80 1 -- start start 2024-05-27T07:25:41.984 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:41.992+0000 7f0121b4d700 1 --2- 172.21.15.52:0/27382024 >> [v2:172.21.15.52:6800/3513075805,v1:172.21.15.52:6801/3513075805] conn(0x7f01080b2880 0x7f01080b4d90 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7f0118005950 tx=0x7f01180058e0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:25:41.984 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:41.993+0000 7f01327c6b80 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x560f54009df0 0x560f5418d6c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:25:41.985 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:41.993+0000 7f01327c6b80 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x560f5418dc00 0x560f541879f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:25:41.985 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:41.993+0000 7f01327c6b80 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x560f54187f30 0x560f54188340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:25:41.985 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:41.993+0000 7f01327c6b80 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x560f540b6c40 con 0x560f5418dc00 2024-05-27T07:25:41.985 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:41.993+0000 7f01327c6b80 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x560f540a67e0 con 0x560f54009df0 2024-05-27T07:25:41.985 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:41.993+0000 7f01327c6b80 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x560f540aa820 con 0x560f54187f30 2024-05-27T07:25:41.985 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:41.993+0000 7f012134c700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x560f5418dc00 0x560f541879f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:25:41.985 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:41.993+0000 7f012134c700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x560f5418dc00 0x560f541879f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.52:48128/0 (socket says 172.21.15.52:48128) 2024-05-27T07:25:41.985 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:41.993+0000 7f012134c700 1 -- 172.21.15.52:0/1574992979 learned_addr learned my addr 172.21.15.52:0/1574992979 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-05-27T07:25:41.985 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:41.993+0000 7f0121b4d700 1 --2- 172.21.15.52:0/1574992979 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x560f54009df0 0x560f5418d6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:25:41.985 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:41.993+0000 7f012234e700 1 --2- 172.21.15.52:0/1574992979 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x560f54187f30 0x560f54188340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:25:41.985 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:41.993+0000 7f012134c700 1 -- 172.21.15.52:0/1574992979 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x560f54187f30 msgr2=0x560f54188340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:25:41.985 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:41.993+0000 7f012134c700 1 --2- 172.21.15.52:0/1574992979 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x560f54187f30 0x560f54188340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:25:41.985 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:41.993+0000 7f012134c700 1 -- 172.21.15.52:0/1574992979 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x560f54009df0 msgr2=0x560f5418d6c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:25:41.987 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:41.993+0000 7f012134c700 1 --2- 172.21.15.52:0/1574992979 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x560f54009df0 0x560f5418d6c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:25:41.987 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:41.993+0000 7f012134c700 1 -- 172.21.15.52:0/1574992979 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x560f53ff6880 con 0x560f5418dc00 2024-05-27T07:25:41.987 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:41.993+0000 7f012134c700 1 --2- 172.21.15.52:0/1574992979 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x560f5418dc00 0x560f541879f0 secure :-1 s=READY pgs=372 cs=0 l=1 rev1=1 crypto rx=0x7f011c076aa0 tx=0x7f01100037b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=47280eae111897f server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:25:41.987 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:41.993+0000 7f0107fff700 1 -- 172.21.15.52:0/1574992979 <== mon.0 v2:172.21.15.52:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0110003ba0 con 0x560f5418dc00 2024-05-27T07:25:41.988 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:41.994+0000 7f0107fff700 1 -- 172.21.15.52:0/1574992979 <== mon.0 v2:172.21.15.52:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f0110003d40 con 0x560f5418dc00 2024-05-27T07:25:41.988 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:41.994+0000 7f0107fff700 1 -- 172.21.15.52:0/1574992979 <== mon.0 v2:172.21.15.52:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f011000bd00 con 0x560f5418dc00 2024-05-27T07:25:41.992 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:41.994+0000 7f01327c6b80 1 -- 172.21.15.52:0/1574992979 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x560f5418dc00 msgr2=0x560f541879f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:25:41.992 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:41.994+0000 7f01327c6b80 1 --2- 172.21.15.52:0/1574992979 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x560f5418dc00 0x560f541879f0 secure :-1 s=READY pgs=372 cs=0 l=1 rev1=1 crypto rx=0x7f011c076aa0 tx=0x7f01100037b0 comp rx=0 tx=0).stop 2024-05-27T07:25:41.992 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:41.994+0000 7f01327c6b80 1 -- 172.21.15.52:0/1574992979 shutdown_connections 2024-05-27T07:25:41.992 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:41.995+0000 7f01327c6b80 1 --2- 172.21.15.52:0/1574992979 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x560f54187f30 0x560f54188340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:25:41.992 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:41.995+0000 7f01327c6b80 1 --2- 172.21.15.52:0/1574992979 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x560f54009df0 0x560f5418d6c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:25:41.992 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:41.995+0000 7f01327c6b80 1 --2- 172.21.15.52:0/1574992979 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x560f5418dc00 0x560f541879f0 unknown :-1 s=CLOSED pgs=372 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:25:41.992 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:41.995+0000 7f01327c6b80 1 -- 172.21.15.52:0/1574992979 >> 172.21.15.52:0/1574992979 conn(0x560f54064aa0 msgr2=0x560f541825f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:25:41.992 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:41.995+0000 7f01327c6b80 1 -- 172.21.15.52:0/1574992979 shutdown_connections 2024-05-27T07:25:41.992 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:41.995+0000 7f01327c6b80 1 -- 172.21.15.52:0/1574992979 wait complete. 2024-05-27T07:25:41.993 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:41.995+0000 7f01327c6b80 1 -- start start 2024-05-27T07:25:41.993 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:41.995+0000 7f01327c6b80 1 Processor -- start 2024-05-27T07:25:41.993 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:41.996+0000 7f01327c6b80 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x560f54009df0 0x560f54191360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:25:41.993 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:41.996+0000 7f01327c6b80 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x560f54187f30 0x560f541918a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:25:41.993 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:41.996+0000 7f01327c6b80 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x560f54191de0 0x560f5419d960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:25:41.993 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:41.996+0000 7f01327c6b80 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x560f53facf50 con 0x560f54191de0 2024-05-27T07:25:41.993 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:41.996+0000 7f01327c6b80 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x560f54048770 con 0x560f54009df0 2024-05-27T07:25:41.993 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:41.996+0000 7f012234e700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x560f54191de0 0x560f5419d960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:25:41.993 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:41.996+0000 7f012234e700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x560f54191de0 0x560f5419d960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.52:48130/0 (socket says 172.21.15.52:48130) 2024-05-27T07:25:41.993 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:41.996+0000 7f012234e700 1 -- 172.21.15.52:0/643940536 learned_addr learned my addr 172.21.15.52:0/643940536 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-05-27T07:25:41.993 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:41.996+0000 7f01327c6b80 1 -- 172.21.15.52:0/643940536 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x560f54194630 con 0x560f54187f30 2024-05-27T07:25:41.993 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:41.996+0000 7f0121b4d700 1 --2- 172.21.15.52:0/643940536 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x560f54009df0 0x560f54191360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:25:41.994 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:41.996+0000 7f01327c6b80 10 client.0 _ll_register_callbacks cb 0x560f540418d0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T07:25:41.994 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:41.996+0000 7f012134c700 1 --2- 172.21.15.52:0/643940536 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x560f54187f30 0x560f541918a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:25:41.994 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:41.996+0000 7f012234e700 1 -- 172.21.15.52:0/643940536 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x560f54187f30 msgr2=0x560f541918a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:25:41.994 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:41.996+0000 7f012234e700 1 --2- 172.21.15.52:0/643940536 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x560f54187f30 0x560f541918a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:25:41.994 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:41.996+0000 7f012234e700 1 -- 172.21.15.52:0/643940536 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x560f54009df0 msgr2=0x560f54191360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:25:41.994 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:41.996+0000 7f012234e700 1 --2- 172.21.15.52:0/643940536 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x560f54009df0 0x560f54191360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:25:41.994 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:41.996+0000 7f012234e700 1 -- 172.21.15.52:0/643940536 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({osdmap=0}) -- 0x560f54045640 con 0x560f54191de0 2024-05-27T07:25:41.994 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:41.996+0000 7f012134c700 1 --2- 172.21.15.52:0/643940536 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x560f54187f30 0x560f541918a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-27T07:25:41.994 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:41.996+0000 7f0121b4d700 1 --2- 172.21.15.52:0/643940536 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x560f54009df0 0x560f54191360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-27T07:25:41.994 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:41.997+0000 7f012234e700 1 --2- 172.21.15.52:0/643940536 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x560f54191de0 0x560f5419d960 secure :-1 s=READY pgs=373 cs=0 l=1 rev1=1 crypto rx=0x7f011400a740 tx=0x7f0114007ec0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b8c8f6304014c641 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:25:41.994 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:41.997+0000 7f01057fa700 10 client.0 ms_handle_connect on v2:172.21.15.52:3300/0 2024-05-27T07:25:41.994 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:41.997+0000 7f01057fa700 1 -- 172.21.15.52:0/643940536 <== mon.0 v2:172.21.15.52:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f01140adb50 con 0x560f54191de0 2024-05-27T07:25:41.995 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:41.997+0000 7f01057fa700 1 -- 172.21.15.52:0/643940536 <== mon.0 v2:172.21.15.52:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9121+0+0 (secure 0 0 0) 0x7f01140e2240 con 0x560f54191de0 2024-05-27T07:25:41.995 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:41.997+0000 7f01327c6b80 1 -- 172.21.15.52:0/643940536 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f0110003d40 con 0x560f54191de0 2024-05-27T07:25:41.995 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:41.997+0000 7f01057fa700 1 -- 172.21.15.52:0/643940536 <== mon.0 v2:172.21.15.52:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f01140b2600 con 0x560f54191de0 2024-05-27T07:25:41.995 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:41.997+0000 7f01327c6b80 10 client.15267 fetch_fsmap learned FSMap version 29 2024-05-27T07:25:41.995 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:41.997+0000 7f01327c6b80 1 -- 172.21.15.52:0/643940536 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x560f541a39e0 con 0x560f54191de0 2024-05-27T07:25:41.995 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:41.997+0000 7f01057fa700 1 -- 172.21.15.52:0/643940536 <== mon.0 v2:172.21.15.52:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f0110003d40 con 0x560f54191de0 2024-05-27T07:25:41.995 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:41.997+0000 7f01057fa700 1 -- 172.21.15.52:0/643940536 <== mon.0 v2:172.21.15.52:3300/0 5 ==== fsmap(e 29) ==== 8362+0+0 (secure 0 0 0) 0x7f0114004ce0 con 0x560f54191de0 2024-05-27T07:25:41.995 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:41.998+0000 7f01327c6b80 10 client.15267 fetch_fsmap finished waiting for FSMap version 29 2024-05-27T07:25:41.995 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:41.998+0000 7f01327c6b80 10 client.15267 resolve_mds: resolved name 'i' to daemon mds.i 2024-05-27T07:25:41.995 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:41.998+0000 7f01327c6b80 20 client.15267 populate_metadata read hostname 'smithi052' 2024-05-27T07:25:41.995 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:41.998+0000 7f01327c6b80 1 --2- 172.21.15.52:0/643940536 >> [v2:172.21.15.173:6838/564778293,v1:172.21.15.173:6839/564778293] conn(0x560f541a6040 0x560f541a74b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:25:41.995 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:41.998+0000 7f01327c6b80 4 client.15267 mds_command: new command op to 24475 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-27T07:25:41.995 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:41.998+0000 7f01327c6b80 1 -- 172.21.15.52:0/643940536 --> [v2:172.21.15.173:6838/564778293,v1:172.21.15.173:6839/564778293] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x560f541539a0 con 0x560f541a6040 2024-05-27T07:25:41.996 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:41.998+0000 7f0121b4d700 1 --2- 172.21.15.52:0/643940536 >> [v2:172.21.15.173:6838/564778293,v1:172.21.15.173:6839/564778293] conn(0x560f541a6040 0x560f541a74b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:25:41.996 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:41.998+0000 7f01057fa700 10 client.15267 ms_handle_connect on v2:172.21.15.173:6838/564778293 2024-05-27T07:25:41.996 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:41.998+0000 7f0121b4d700 1 --2- 172.21.15.52:0/643940536 >> [v2:172.21.15.173:6838/564778293,v1:172.21.15.173:6839/564778293] conn(0x560f541a6040 0x560f541a74b0 crc :-1 s=READY pgs=18 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=1aa420cbb03a3bf3 server_cookie=7003326303b6f4c2 in_seq=0 out_seq=0 2024-05-27T07:25:41.996 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:41.999+0000 7f01057fa700 1 -- 172.21.15.52:0/643940536 <== mds.2 v2:172.21.15.173:6838/564778293 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24041 (crc 0 0 0) 0x560f541539a0 con 0x560f541a6040 2024-05-27T07:25:41.996 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:41.999+0000 7f01057fa700 10 client.15267 handle_command_reply: tid=0 2024-05-27T07:25:41.996 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:41.999+0000 7f01327c6b80 1 client.15267 shutdown 2024-05-27T07:25:41.996 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:41.999+0000 7f01327c6b80 1 -- 172.21.15.52:0/643940536 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x560f54191de0 msgr2=0x560f5419d960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:25:41.996 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:41.999+0000 7f01327c6b80 1 --2- 172.21.15.52:0/643940536 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x560f54191de0 0x560f5419d960 secure :-1 s=READY pgs=373 cs=0 l=1 rev1=1 crypto rx=0x7f011400a740 tx=0x7f0114007ec0 comp rx=0 tx=0).stop 2024-05-27T07:25:41.996 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:42.000+0000 7f01327c6b80 1 -- 172.21.15.52:0/643940536 shutdown_connections 2024-05-27T07:25:41.997 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:42.000+0000 7f01327c6b80 1 --2- 172.21.15.52:0/643940536 >> [v2:172.21.15.173:6838/564778293,v1:172.21.15.173:6839/564778293] conn(0x560f541a6040 0x560f541a74b0 crc :-1 s=READY pgs=18 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:25:41.997 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:42.000+0000 7f01327c6b80 1 --2- 172.21.15.52:0/643940536 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x560f54191de0 0x560f5419d960 unknown :-1 s=CLOSED pgs=373 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:25:41.997 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:42.000+0000 7f01327c6b80 1 --2- 172.21.15.52:0/643940536 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x560f54009df0 0x560f54191360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:25:41.997 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:42.000+0000 7f01327c6b80 1 --2- 172.21.15.52:0/643940536 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x560f54187f30 0x560f541918a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:25:41.997 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:42.000+0000 7f01057fa700 0 client.15267 ms_handle_reset on v2:172.21.15.173:6838/564778293 2024-05-27T07:25:41.997 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:42.000+0000 7f01327c6b80 1 -- 172.21.15.52:0/643940536 >> 172.21.15.52:0/643940536 conn(0x560f54064aa0 msgr2=0x560f541869e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:25:41.997 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:42.000+0000 7f01327c6b80 1 -- 172.21.15.52:0/643940536 shutdown_connections 2024-05-27T07:25:41.997 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:42.000+0000 7f01327c6b80 1 -- 172.21.15.52:0/643940536 wait complete. 2024-05-27T07:25:41.997 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:42.000+0000 7f01327c6b80 20 client.15267 trim_cache size 0 max 16384 2024-05-27T07:25:42.013 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:42.021+0000 7f01327c6b80 1 Processor -- start 2024-05-27T07:25:42.013 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:42.021+0000 7f01327c6b80 1 -- start start 2024-05-27T07:25:42.013 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:42.021+0000 7f01327c6b80 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x560f54009df0 0x560f541879f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:25:42.013 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:42.021+0000 7f01327c6b80 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x560f54187f30 0x560f5418cab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:25:42.014 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:42.021+0000 7f01327c6b80 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x560f54191de0 0x560f5419d8d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:25:42.014 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:42.021+0000 7f01327c6b80 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x560f541a57f0 con 0x560f54187f30 2024-05-27T07:25:42.014 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:42.021+0000 7f01327c6b80 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x560f541a5ec0 con 0x560f54009df0 2024-05-27T07:25:42.014 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:42.021+0000 7f01327c6b80 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x560f541a6590 con 0x560f54191de0 2024-05-27T07:25:42.014 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:42.021+0000 7f012134c700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x560f54187f30 0x560f5418cab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:25:42.014 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:42.021+0000 7f012134c700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x560f54187f30 0x560f5418cab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.52:48144/0 (socket says 172.21.15.52:48144) 2024-05-27T07:25:42.014 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:42.021+0000 7f012134c700 1 -- 172.21.15.52:0/3007971294 learned_addr learned my addr 172.21.15.52:0/3007971294 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-05-27T07:25:42.014 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:42.022+0000 7f0121b4d700 1 --2- 172.21.15.52:0/3007971294 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x560f54009df0 0x560f541879f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:25:42.014 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:42.022+0000 7f012234e700 1 --2- 172.21.15.52:0/3007971294 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x560f54191de0 0x560f5419d8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:25:42.014 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:42.022+0000 7f012134c700 1 -- 172.21.15.52:0/3007971294 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x560f54191de0 msgr2=0x560f5419d8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:25:42.014 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:42.022+0000 7f012134c700 1 --2- 172.21.15.52:0/3007971294 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x560f54191de0 0x560f5419d8d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:25:42.014 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:42.022+0000 7f012134c700 1 -- 172.21.15.52:0/3007971294 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x560f54009df0 msgr2=0x560f541879f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:25:42.015 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:42.022+0000 7f012134c700 1 --2- 172.21.15.52:0/3007971294 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x560f54009df0 0x560f541879f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:25:42.015 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:42.022+0000 7f012134c700 1 -- 172.21.15.52:0/3007971294 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x560f54193ea0 con 0x560f54187f30 2024-05-27T07:25:42.015 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:42.022+0000 7f012134c700 1 --2- 172.21.15.52:0/3007971294 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x560f54187f30 0x560f5418cab0 secure :-1 s=READY pgs=374 cs=0 l=1 rev1=1 crypto rx=0x7f0110009750 tx=0x7f0110009cd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=37600ff9a9f2485e server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:25:42.015 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:42.022+0000 7f01077fe700 1 -- 172.21.15.52:0/3007971294 <== mon.0 v2:172.21.15.52:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0110030030 con 0x560f54187f30 2024-05-27T07:25:42.015 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:42.022+0000 7f01077fe700 1 -- 172.21.15.52:0/3007971294 <== mon.0 v2:172.21.15.52:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f0110003e50 con 0x560f54187f30 2024-05-27T07:25:42.015 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:42.022+0000 7f01077fe700 1 -- 172.21.15.52:0/3007971294 <== mon.0 v2:172.21.15.52:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f011000b9b0 con 0x560f54187f30 2024-05-27T07:25:42.019 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:42.023+0000 7f01327c6b80 1 -- 172.21.15.52:0/3007971294 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x560f54187f30 msgr2=0x560f5418cab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:25:42.019 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:42.023+0000 7f01327c6b80 1 --2- 172.21.15.52:0/3007971294 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x560f54187f30 0x560f5418cab0 secure :-1 s=READY pgs=374 cs=0 l=1 rev1=1 crypto rx=0x7f0110009750 tx=0x7f0110009cd0 comp rx=0 tx=0).stop 2024-05-27T07:25:42.019 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:42.023+0000 7f01327c6b80 1 -- 172.21.15.52:0/3007971294 shutdown_connections 2024-05-27T07:25:42.019 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:42.023+0000 7f01327c6b80 1 --2- 172.21.15.52:0/3007971294 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x560f54191de0 0x560f5419d8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:25:42.019 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:42.023+0000 7f01327c6b80 1 --2- 172.21.15.52:0/3007971294 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x560f54009df0 0x560f541879f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:25:42.019 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:42.023+0000 7f01327c6b80 1 --2- 172.21.15.52:0/3007971294 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x560f54187f30 0x560f5418cab0 unknown :-1 s=CLOSED pgs=374 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:25:42.019 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:42.023+0000 7f01327c6b80 1 -- 172.21.15.52:0/3007971294 >> 172.21.15.52:0/3007971294 conn(0x560f54064aa0 msgr2=0x560f5418c3a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:25:42.020 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:42.023+0000 7f01327c6b80 1 -- 172.21.15.52:0/3007971294 shutdown_connections 2024-05-27T07:25:42.020 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:42.023+0000 7f01327c6b80 1 -- 172.21.15.52:0/3007971294 wait complete. 2024-05-27T07:25:42.020 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:42.024+0000 7f01327c6b80 1 -- start start 2024-05-27T07:25:42.020 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:42.024+0000 7f01327c6b80 1 Processor -- start 2024-05-27T07:25:42.020 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:42.024+0000 7f01327c6b80 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x560f54009df0 0x560f541b0930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:25:42.020 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:42.024+0000 7f01327c6b80 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x560f54187f30 0x560f541b0e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:25:42.020 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:42.024+0000 7f01327c6b80 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x560f54191de0 0x560f541ab8e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:25:42.021 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:42.024+0000 7f01327c6b80 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x560f541a7330 con 0x560f54187f30 2024-05-27T07:25:42.021 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:42.024+0000 7f01327c6b80 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x560f541a6c60 con 0x560f54191de0 2024-05-27T07:25:42.021 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:42.024+0000 7f01327c6b80 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x560f541a7a00 con 0x560f54009df0 2024-05-27T07:25:42.021 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:42.024+0000 7f01327c6b80 10 client.0 _ll_register_callbacks cb 0x560f540418d0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T07:25:42.021 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:42.024+0000 7f012134c700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x560f54187f30 0x560f541b0e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:25:42.021 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:42.024+0000 7f012134c700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x560f54187f30 0x560f541b0e70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.52:48152/0 (socket says 172.21.15.52:48152) 2024-05-27T07:25:42.021 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:42.024+0000 7f012134c700 1 -- 172.21.15.52:0/3723874709 learned_addr learned my addr 172.21.15.52:0/3723874709 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-05-27T07:25:42.021 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:42.025+0000 7f0121b4d700 1 --2- 172.21.15.52:0/3723874709 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x560f54009df0 0x560f541b0930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:25:42.021 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:42.025+0000 7f012234e700 1 --2- 172.21.15.52:0/3723874709 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x560f54191de0 0x560f541ab8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:25:42.021 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:42.025+0000 7f012134c700 1 -- 172.21.15.52:0/3723874709 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x560f54009df0 msgr2=0x560f541b0930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:25:42.021 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:42.025+0000 7f012134c700 1 --2- 172.21.15.52:0/3723874709 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x560f54009df0 0x560f541b0930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:25:42.021 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:42.025+0000 7f012134c700 1 -- 172.21.15.52:0/3723874709 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x560f54191de0 msgr2=0x560f541ab8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:25:42.022 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:42.025+0000 7f012134c700 1 --2- 172.21.15.52:0/3723874709 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x560f54191de0 0x560f541ab8e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:25:42.022 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:42.025+0000 7f012134c700 1 -- 172.21.15.52:0/3723874709 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({osdmap=0}) -- 0x560f54187ca0 con 0x560f54187f30 2024-05-27T07:25:42.022 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:42.025+0000 7f012134c700 1 --2- 172.21.15.52:0/3723874709 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x560f54187f30 0x560f541b0e70 secure :-1 s=READY pgs=375 cs=0 l=1 rev1=1 crypto rx=0x7f0110009720 tx=0x7f0110003d00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5b2fe4b3dde2a2d server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:25:42.022 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:42.025+0000 7f01057fa700 10 client.0 ms_handle_connect on v2:172.21.15.52:3300/0 2024-05-27T07:25:42.022 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:42.025+0000 7f01057fa700 1 -- 172.21.15.52:0/3723874709 <== mon.0 v2:172.21.15.52:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f011000b990 con 0x560f54187f30 2024-05-27T07:25:42.022 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:42.025+0000 7f01057fa700 1 -- 172.21.15.52:0/3723874709 <== mon.0 v2:172.21.15.52:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9121+0+0 (secure 0 0 0) 0x7f011000bb30 con 0x560f54187f30 2024-05-27T07:25:42.022 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:42.025+0000 7f01327c6b80 1 -- 172.21.15.52:0/3723874709 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f0110003e50 con 0x560f54187f30 2024-05-27T07:25:42.022 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:42.025+0000 7f01057fa700 1 -- 172.21.15.52:0/3723874709 <== mon.0 v2:172.21.15.52:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f0110045050 con 0x560f54187f30 2024-05-27T07:25:42.022 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:42.025+0000 7f01327c6b80 10 client.15273 fetch_fsmap learned FSMap version 29 2024-05-27T07:25:42.022 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:42.025+0000 7f01327c6b80 1 -- 172.21.15.52:0/3723874709 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x560f54211cd0 con 0x560f54187f30 2024-05-27T07:25:42.022 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:42.026+0000 7f01057fa700 1 -- 172.21.15.52:0/3723874709 <== mon.0 v2:172.21.15.52:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f0110003e50 con 0x560f54187f30 2024-05-27T07:25:42.022 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:42.026+0000 7f01057fa700 1 -- 172.21.15.52:0/3723874709 <== mon.0 v2:172.21.15.52:3300/0 5 ==== fsmap(e 29) ==== 8362+0+0 (secure 0 0 0) 0x7f011002c430 con 0x560f54187f30 2024-05-27T07:25:42.022 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:42.026+0000 7f01327c6b80 10 client.15273 fetch_fsmap finished waiting for FSMap version 29 2024-05-27T07:25:42.023 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:42.026+0000 7f01327c6b80 10 client.15273 resolve_mds: resolved name 'i' to daemon mds.i 2024-05-27T07:25:42.023 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:42.026+0000 7f01327c6b80 20 client.15273 populate_metadata read hostname 'smithi052' 2024-05-27T07:25:42.023 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:42.026+0000 7f01327c6b80 1 --2- 172.21.15.52:0/3723874709 >> [v2:172.21.15.173:6838/564778293,v1:172.21.15.173:6839/564778293] conn(0x560f542139f0 0x560f54214e60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:25:42.023 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:42.026+0000 7f01327c6b80 4 client.15273 mds_command: new command op to 24475 tid=0 multi_id=0 [{"prefix": "status"}] 2024-05-27T07:25:42.023 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:42.026+0000 7f01327c6b80 1 -- 172.21.15.52:0/3723874709 --> [v2:172.21.15.173:6838/564778293,v1:172.21.15.173:6839/564778293] -- command(tid 0: {"prefix": "status"}) -- 0x560f540b8b90 con 0x560f542139f0 2024-05-27T07:25:42.023 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:42.026+0000 7f0121b4d700 1 --2- 172.21.15.52:0/3723874709 >> [v2:172.21.15.173:6838/564778293,v1:172.21.15.173:6839/564778293] conn(0x560f542139f0 0x560f54214e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:25:42.023 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:42.027+0000 7f01057fa700 10 client.15273 ms_handle_connect on v2:172.21.15.173:6838/564778293 2024-05-27T07:25:42.023 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:42.027+0000 7f0121b4d700 1 --2- 172.21.15.52:0/3723874709 >> [v2:172.21.15.173:6838/564778293,v1:172.21.15.173:6839/564778293] conn(0x560f542139f0 0x560f54214e60 crc :-1 s=READY pgs=19 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=8220e1c7f10cab30 server_cookie=81c0da9cb19f9760 in_seq=0 out_seq=0 2024-05-27T07:25:42.023 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:42.027+0000 7f01057fa700 1 -- 172.21.15.52:0/3723874709 <== mds.2 v2:172.21.15.173:6838/564778293 1 ==== command_reply(tid 0: 0 ) ==== 8+0+322 (crc 0 0 0) 0x560f540b8b90 con 0x560f542139f0 2024-05-27T07:25:42.023 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:42.027+0000 7f01057fa700 10 client.15273 handle_command_reply: tid=0 2024-05-27T07:25:42.023 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:42.027+0000 7f01327c6b80 1 client.15273 shutdown 2024-05-27T07:25:42.023 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:42.027+0000 7f01327c6b80 1 -- 172.21.15.52:0/3723874709 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x560f54187f30 msgr2=0x560f541b0e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:25:42.023 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:42.027+0000 7f01327c6b80 1 --2- 172.21.15.52:0/3723874709 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x560f54187f30 0x560f541b0e70 secure :-1 s=READY pgs=375 cs=0 l=1 rev1=1 crypto rx=0x7f0110009720 tx=0x7f0110003d00 comp rx=0 tx=0).stop 2024-05-27T07:25:42.024 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:42.027+0000 7f01327c6b80 1 -- 172.21.15.52:0/3723874709 shutdown_connections 2024-05-27T07:25:42.024 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:42.027+0000 7f01327c6b80 1 --2- 172.21.15.52:0/3723874709 >> [v2:172.21.15.173:6838/564778293,v1:172.21.15.173:6839/564778293] conn(0x560f542139f0 0x560f54214e60 crc :-1 s=READY pgs=19 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:25:42.024 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:42.027+0000 7f01327c6b80 1 --2- 172.21.15.52:0/3723874709 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x560f54191de0 0x560f541ab8e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:25:42.024 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:42.027+0000 7f01057fa700 0 client.15273 ms_handle_reset on v2:172.21.15.173:6838/564778293 2024-05-27T07:25:42.024 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:42.027+0000 7f01327c6b80 1 --2- 172.21.15.52:0/3723874709 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x560f54009df0 0x560f541b0930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:25:42.024 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:42.027+0000 7f01327c6b80 1 --2- 172.21.15.52:0/3723874709 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x560f54187f30 0x560f541b0e70 unknown :-1 s=CLOSED pgs=375 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:25:42.024 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:42.027+0000 7f01327c6b80 1 -- 172.21.15.52:0/3723874709 >> 172.21.15.52:0/3723874709 conn(0x560f54064aa0 msgr2=0x560f5418cb30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:25:42.024 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:42.027+0000 7f01327c6b80 1 -- 172.21.15.52:0/3723874709 shutdown_connections 2024-05-27T07:25:42.024 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:42.027+0000 7f01327c6b80 1 -- 172.21.15.52:0/3723874709 wait complete. 2024-05-27T07:25:42.024 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:42.027+0000 7f01327c6b80 20 client.15273 trim_cache size 0 max 16384 2024-05-27T07:25:42.025 INFO:teuthology.orchestra.run.smithi052.stdout:{ 2024-05-27T07:25:42.025 INFO:teuthology.orchestra.run.smithi052.stdout: "cluster_fsid": "73df2636-1bf8-11ef-bc9b-c7b262605968", 2024-05-27T07:25:42.025 INFO:teuthology.orchestra.run.smithi052.stdout: "whoami": 2, 2024-05-27T07:25:42.025 INFO:teuthology.orchestra.run.smithi052.stdout: "id": 24475, 2024-05-27T07:25:42.025 INFO:teuthology.orchestra.run.smithi052.stdout: "want_state": "up:active", 2024-05-27T07:25:42.025 INFO:teuthology.orchestra.run.smithi052.stdout: "state": "up:active", 2024-05-27T07:25:42.025 INFO:teuthology.orchestra.run.smithi052.stdout: "fs_name": "cephfs", 2024-05-27T07:25:42.025 INFO:teuthology.orchestra.run.smithi052.stdout: "rank_uptime": 168.164952787, 2024-05-27T07:25:42.025 INFO:teuthology.orchestra.run.smithi052.stdout: "mdsmap_epoch": 29, 2024-05-27T07:25:42.025 INFO:teuthology.orchestra.run.smithi052.stdout: "osdmap_epoch": 90, 2024-05-27T07:25:42.026 INFO:teuthology.orchestra.run.smithi052.stdout: "osdmap_epoch_barrier": 88, 2024-05-27T07:25:42.026 INFO:teuthology.orchestra.run.smithi052.stdout: "uptime": 255.790800983 2024-05-27T07:25:42.026 INFO:teuthology.orchestra.run.smithi052.stdout:} 2024-05-27T07:25:42.026 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:42.028+0000 7f0123db1700 1 -- 172.21.15.52:0/27382024 >> [v2:172.21.15.52:6800/3513075805,v1:172.21.15.52:6801/3513075805] conn(0x7f01080b2880 msgr2=0x7f01080b4d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:25:42.026 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:42.028+0000 7f0123db1700 1 --2- 172.21.15.52:0/27382024 >> [v2:172.21.15.52:6800/3513075805,v1:172.21.15.52:6801/3513075805] conn(0x7f01080b2880 0x7f01080b4d90 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7f0118005950 tx=0x7f01180058e0 comp rx=0 tx=0).stop 2024-05-27T07:25:42.027 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:42.028+0000 7f0123db1700 1 -- 172.21.15.52:0/27382024 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f011c108750 msgr2=0x7f011c19c760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:25:42.027 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:42.028+0000 7f0123db1700 1 --2- 172.21.15.52:0/27382024 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f011c108750 0x7f011c19c760 secure :-1 s=READY pgs=371 cs=0 l=1 rev1=1 crypto rx=0x7f0114007d70 tx=0x7f011400a340 comp rx=0 tx=0).stop 2024-05-27T07:25:42.027 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:42.028+0000 7f0123db1700 1 -- 172.21.15.52:0/27382024 shutdown_connections 2024-05-27T07:25:42.027 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:42.028+0000 7f0123db1700 1 --2- 172.21.15.52:0/27382024 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f011c108750 0x7f011c19c760 unknown :-1 s=CLOSED pgs=371 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:25:42.027 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:42.028+0000 7f0123db1700 1 --2- 172.21.15.52:0/27382024 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f011c1034d0 0x7f011c19bca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:25:42.027 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:42.028+0000 7f0123db1700 1 --2- 172.21.15.52:0/27382024 >> [v2:172.21.15.52:6800/3513075805,v1:172.21.15.52:6801/3513075805] conn(0x7f01080b2880 0x7f01080b4d90 unknown :-1 s=CLOSED pgs=224 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:25:42.027 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:42.028+0000 7f0123db1700 1 --2- 172.21.15.52:0/27382024 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f011c107e00 0x7f011c19c1e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:25:42.027 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:42.028+0000 7f0123db1700 1 -- 172.21.15.52:0/27382024 >> 172.21.15.52:0/27382024 conn(0x7f011c0fef20 msgr2=0x7f011c1096b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:25:42.027 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:42.028+0000 7f0123db1700 1 -- 172.21.15.52:0/27382024 shutdown_connections 2024-05-27T07:25:42.027 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:42.028+0000 7f0123db1700 1 -- 172.21.15.52:0/27382024 wait complete. 2024-05-27T07:25:42.651 DEBUG:teuthology.orchestra.run.smithi052:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:38f9d900c83c63799fbdbe61acc9a11b0d3554a6 shell --fsid 73df2636-1bf8-11ef-bc9b-c7b262605968 -- ceph tell mds.h status 2024-05-27T07:25:42.901 INFO:teuthology.orchestra.run.smithi052.stderr:Inferring config /var/lib/ceph/73df2636-1bf8-11ef-bc9b-c7b262605968/mon.a/config 2024-05-27T07:25:43.273 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:25:43 smithi052 ceph-mon[31955]: pgmap v416: 129 pgs: 129 active+clean; 647 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-05-27T07:25:43.351 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:25:43 smithi173 ceph-mon[37000]: pgmap v416: 129 pgs: 129 active+clean; 647 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-05-27T07:25:43.440 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:25:43 smithi105 ceph-mon[35977]: pgmap v416: 129 pgs: 129 active+clean; 647 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-05-27T07:25:44.670 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:44.679+0000 7f261dfd0700 1 -- 172.21.15.52:0/1052791871 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f26181034d0 msgr2=0x7f26181038e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:25:44.671 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:44.679+0000 7f261dfd0700 1 --2- 172.21.15.52:0/1052791871 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f26181034d0 0x7f26181038e0 secure :-1 s=READY pgs=376 cs=0 l=1 rev1=1 crypto rx=0x7f26080099c0 tx=0x7f2608009cd0 comp rx=0 tx=0).stop 2024-05-27T07:25:44.671 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:44.679+0000 7f261dfd0700 1 -- 172.21.15.52:0/1052791871 shutdown_connections 2024-05-27T07:25:44.671 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:44.679+0000 7f261dfd0700 1 --2- 172.21.15.52:0/1052791871 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f2618108750 0x7f26181100b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:25:44.671 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:44.679+0000 7f261dfd0700 1 --2- 172.21.15.52:0/1052791871 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f26181034d0 0x7f26181038e0 unknown :-1 s=CLOSED pgs=376 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:25:44.671 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:44.679+0000 7f261dfd0700 1 --2- 172.21.15.52:0/1052791871 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f2618107e00 0x7f2618108210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:25:44.671 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:44.679+0000 7f261dfd0700 1 -- 172.21.15.52:0/1052791871 >> 172.21.15.52:0/1052791871 conn(0x7f26180fef20 msgr2=0x7f2618101340 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:25:44.671 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:44.679+0000 7f261dfd0700 1 -- 172.21.15.52:0/1052791871 shutdown_connections 2024-05-27T07:25:44.671 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:44.679+0000 7f261dfd0700 1 -- 172.21.15.52:0/1052791871 wait complete. 2024-05-27T07:25:44.673 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:44.680+0000 7f261dfd0700 1 Processor -- start 2024-05-27T07:25:44.673 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:44.680+0000 7f261dfd0700 1 -- start start 2024-05-27T07:25:44.673 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:44.681+0000 7f261dfd0700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f26181034d0 0x7f261819aee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:25:44.673 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:44.681+0000 7f261dfd0700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f2618107e00 0x7f261819b480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:25:44.673 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:44.681+0000 7f261dfd0700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f2618108750 0x7f261819f710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:25:44.673 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:44.681+0000 7f261dfd0700 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x7f26181144d0 con 0x7f26181034d0 2024-05-27T07:25:44.673 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:44.681+0000 7f261dfd0700 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f2618114350 con 0x7f2618107e00 2024-05-27T07:25:44.673 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:44.681+0000 7f261dfd0700 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f2618114650 con 0x7f2618108750 2024-05-27T07:25:44.673 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:44.681+0000 7f26177fe700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f26181034d0 0x7f261819aee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:25:44.673 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:44.681+0000 7f26177fe700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f26181034d0 0x7f261819aee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.52:48174/0 (socket says 172.21.15.52:48174) 2024-05-27T07:25:44.674 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:44.681+0000 7f26177fe700 1 -- 172.21.15.52:0/1292848734 learned_addr learned my addr 172.21.15.52:0/1292848734 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-05-27T07:25:44.674 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:44.681+0000 7f260effd700 1 --2- 172.21.15.52:0/1292848734 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f2618107e00 0x7f261819b480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:25:44.674 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:44.681+0000 7f2617fff700 1 --2- 172.21.15.52:0/1292848734 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f2618108750 0x7f261819f710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:25:44.674 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:44.681+0000 7f26177fe700 1 -- 172.21.15.52:0/1292848734 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f2618108750 msgr2=0x7f261819f710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:25:44.674 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:44.681+0000 7f26177fe700 1 --2- 172.21.15.52:0/1292848734 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f2618108750 0x7f261819f710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:25:44.674 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:44.681+0000 7f26177fe700 1 -- 172.21.15.52:0/1292848734 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f2618107e00 msgr2=0x7f261819b480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:25:44.674 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:44.681+0000 7f26177fe700 1 --2- 172.21.15.52:0/1292848734 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f2618107e00 0x7f261819b480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:25:44.674 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:44.681+0000 7f26177fe700 1 -- 172.21.15.52:0/1292848734 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f261819fdb0 con 0x7f26181034d0 2024-05-27T07:25:44.674 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:44.682+0000 7f26177fe700 1 --2- 172.21.15.52:0/1292848734 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f26181034d0 0x7f261819aee0 secure :-1 s=READY pgs=377 cs=0 l=1 rev1=1 crypto rx=0x7f260800b580 tx=0x7f26080047b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:25:44.674 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:44.682+0000 7f26157fa700 1 -- 172.21.15.52:0/1292848734 <== mon.0 v2:172.21.15.52:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2608027070 con 0x7f26181034d0 2024-05-27T07:25:44.674 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:44.682+0000 7f261dfd0700 1 -- 172.21.15.52:0/1292848734 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f26181a0040 con 0x7f26181034d0 2024-05-27T07:25:44.674 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:44.682+0000 7f26157fa700 1 -- 172.21.15.52:0/1292848734 <== mon.0 v2:172.21.15.52:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f2608003f20 con 0x7f26181034d0 2024-05-27T07:25:44.675 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:44.682+0000 7f26157fa700 1 -- 172.21.15.52:0/1292848734 <== mon.0 v2:172.21.15.52:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2608022d70 con 0x7f26181034d0 2024-05-27T07:25:44.675 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:44.682+0000 7f261dfd0700 1 -- 172.21.15.52:0/1292848734 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f26181a79c0 con 0x7f26181034d0 2024-05-27T07:25:44.675 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:44.685+0000 7f26157fa700 1 -- 172.21.15.52:0/1292848734 <== mon.0 v2:172.21.15.52:3300/0 4 ==== mgrmap(e 18) ==== 149652+0+0 (secure 0 0 0) 0x7f260800f460 con 0x7f26181034d0 2024-05-27T07:25:44.676 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:44.686+0000 7f262c9e5b80 1 Processor -- start 2024-05-27T07:25:44.677 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:44.686+0000 7f26157fa700 1 --2- 172.21.15.52:0/1292848734 >> [v2:172.21.15.52:6800/3513075805,v1:172.21.15.52:6801/3513075805] conn(0x7f26040b2880 0x7f26040b4d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:25:44.677 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:44.686+0000 7f262c9e5b80 1 -- start start 2024-05-27T07:25:44.677 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:44.686+0000 7f2617fff700 1 --2- 172.21.15.52:0/1292848734 >> [v2:172.21.15.52:6800/3513075805,v1:172.21.15.52:6801/3513075805] conn(0x7f26040b2880 0x7f26040b4d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:25:44.677 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:44.686+0000 7f26157fa700 1 -- 172.21.15.52:0/1292848734 <== mon.0 v2:172.21.15.52:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9121+0+0 (secure 0 0 0) 0x7f26080e32a0 con 0x7f26181034d0 2024-05-27T07:25:44.677 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:44.686+0000 7f262c9e5b80 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55b6656c9df0 0x55b66584d6c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:25:44.677 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:44.686+0000 7f262c9e5b80 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55b66584dc00 0x55b6658479f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:25:44.677 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:44.686+0000 7f262c9e5b80 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55b665847f30 0x55b665848340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:25:44.677 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:44.686+0000 7f262c9e5b80 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x55b665776c40 con 0x55b665847f30 2024-05-27T07:25:44.677 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:44.686+0000 7f262c9e5b80 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x55b6657667e0 con 0x55b6656c9df0 2024-05-27T07:25:44.677 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:44.686+0000 7f262c9e5b80 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x55b66576a820 con 0x55b66584dc00 2024-05-27T07:25:44.679 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:44.686+0000 7f2617fff700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55b665847f30 0x55b665848340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:25:44.679 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:44.686+0000 7f2617fff700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55b665847f30 0x55b665848340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.52:48180/0 (socket says 172.21.15.52:48180) 2024-05-27T07:25:44.679 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:44.686+0000 7f2617fff700 1 -- 172.21.15.52:0/2002623206 learned_addr learned my addr 172.21.15.52:0/2002623206 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-05-27T07:25:44.679 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:44.686+0000 7f26177fe700 1 --2- 172.21.15.52:0/2002623206 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55b6656c9df0 0x55b66584d6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:25:44.679 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:44.686+0000 7f260effd700 1 --2- 172.21.15.52:0/2002623206 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55b66584dc00 0x55b6658479f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:25:44.679 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:44.686+0000 7f2617fff700 1 --2- 172.21.15.52:0/1292848734 >> [v2:172.21.15.52:6800/3513075805,v1:172.21.15.52:6801/3513075805] conn(0x7f26040b2880 0x7f26040b4d90 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7f2610009c60 tx=0x7f26100077e0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:25:44.680 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:44.687+0000 7f2617fff700 1 -- 172.21.15.52:0/2002623206 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55b66584dc00 msgr2=0x55b6658479f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:25:44.680 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:44.687+0000 7f2617fff700 1 --2- 172.21.15.52:0/2002623206 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55b66584dc00 0x55b6658479f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:25:44.680 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:44.687+0000 7f2617fff700 1 -- 172.21.15.52:0/2002623206 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55b6656c9df0 msgr2=0x55b66584d6c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:25:44.680 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:44.687+0000 7f2617fff700 1 --2- 172.21.15.52:0/2002623206 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55b6656c9df0 0x55b66584d6c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:25:44.680 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:44.687+0000 7f2617fff700 1 -- 172.21.15.52:0/2002623206 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55b6656b6880 con 0x55b665847f30 2024-05-27T07:25:44.680 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:44.687+0000 7f2617fff700 1 --2- 172.21.15.52:0/2002623206 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55b665847f30 0x55b665848340 secure :-1 s=READY pgs=378 cs=0 l=1 rev1=1 crypto rx=0x7f2610019510 tx=0x7f261001a000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=211ebc144d22c90 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:25:44.680 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:44.687+0000 7f260dffb700 1 -- 172.21.15.52:0/2002623206 <== mon.0 v2:172.21.15.52:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2610026070 con 0x55b665847f30 2024-05-27T07:25:44.680 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:44.687+0000 7f260dffb700 1 -- 172.21.15.52:0/2002623206 <== mon.0 v2:172.21.15.52:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f261001d040 con 0x55b665847f30 2024-05-27T07:25:44.680 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:44.687+0000 7f260dffb700 1 -- 172.21.15.52:0/2002623206 <== mon.0 v2:172.21.15.52:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2610018680 con 0x55b665847f30 2024-05-27T07:25:44.680 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:44.688+0000 7f262c9e5b80 1 -- 172.21.15.52:0/2002623206 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55b665847f30 msgr2=0x55b665848340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:25:44.680 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:44.688+0000 7f262c9e5b80 1 --2- 172.21.15.52:0/2002623206 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55b665847f30 0x55b665848340 secure :-1 s=READY pgs=378 cs=0 l=1 rev1=1 crypto rx=0x7f2610019510 tx=0x7f261001a000 comp rx=0 tx=0).stop 2024-05-27T07:25:44.680 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:44.688+0000 7f262c9e5b80 1 -- 172.21.15.52:0/2002623206 shutdown_connections 2024-05-27T07:25:44.680 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:44.688+0000 7f262c9e5b80 1 --2- 172.21.15.52:0/2002623206 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55b665847f30 0x55b665848340 unknown :-1 s=CLOSED pgs=378 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:25:44.681 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:44.688+0000 7f262c9e5b80 1 --2- 172.21.15.52:0/2002623206 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55b6656c9df0 0x55b66584d6c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:25:44.681 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:44.688+0000 7f262c9e5b80 1 --2- 172.21.15.52:0/2002623206 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55b66584dc00 0x55b6658479f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:25:44.681 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:44.688+0000 7f262c9e5b80 1 -- 172.21.15.52:0/2002623206 >> 172.21.15.52:0/2002623206 conn(0x55b665724aa0 msgr2=0x55b6658425f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:25:44.681 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:44.688+0000 7f262c9e5b80 1 -- 172.21.15.52:0/2002623206 shutdown_connections 2024-05-27T07:25:44.681 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:44.688+0000 7f262c9e5b80 1 -- 172.21.15.52:0/2002623206 wait complete. 2024-05-27T07:25:44.681 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:44.689+0000 7f262c9e5b80 1 -- start start 2024-05-27T07:25:44.684 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:44.689+0000 7f262c9e5b80 1 Processor -- start 2024-05-27T07:25:44.684 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:44.689+0000 7f262c9e5b80 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55b6656c9df0 0x55b665851360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:25:44.684 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:44.689+0000 7f262c9e5b80 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55b665847f30 0x55b6658518a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:25:44.684 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:44.689+0000 7f262c9e5b80 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55b66584dc00 0x55b665851de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:25:44.684 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:44.689+0000 7f262c9e5b80 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x55b66566cf50 con 0x55b6656c9df0 2024-05-27T07:25:44.684 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:44.689+0000 7f262c9e5b80 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x55b665708770 con 0x55b66584dc00 2024-05-27T07:25:44.684 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:44.689+0000 7f262c9e5b80 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x55b6658546f0 con 0x55b665847f30 2024-05-27T07:25:44.685 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:44.689+0000 7f262c9e5b80 10 client.0 _ll_register_callbacks cb 0x55b6657018d0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T07:25:44.685 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:44.689+0000 7f26177fe700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55b6656c9df0 0x55b665851360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:25:44.685 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:44.689+0000 7f26177fe700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55b6656c9df0 0x55b665851360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.52:48186/0 (socket says 172.21.15.52:48186) 2024-05-27T07:25:44.685 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:44.689+0000 7f26177fe700 1 -- 172.21.15.52:0/3206669525 learned_addr learned my addr 172.21.15.52:0/3206669525 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-05-27T07:25:44.685 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:44.690+0000 7f260effd700 1 --2- 172.21.15.52:0/3206669525 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55b665847f30 0x55b6658518a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:25:44.685 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:44.690+0000 7f2617fff700 1 --2- 172.21.15.52:0/3206669525 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55b66584dc00 0x55b665851de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:25:44.685 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:44.690+0000 7f26177fe700 1 -- 172.21.15.52:0/3206669525 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55b665847f30 msgr2=0x55b6658518a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:25:44.685 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:44.690+0000 7f26177fe700 1 --2- 172.21.15.52:0/3206669525 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55b665847f30 0x55b6658518a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:25:44.685 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:44.690+0000 7f26177fe700 1 -- 172.21.15.52:0/3206669525 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55b66584dc00 msgr2=0x55b665851de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:25:44.686 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:44.690+0000 7f26177fe700 1 --2- 172.21.15.52:0/3206669525 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55b66584dc00 0x55b665851de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:25:44.686 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:44.690+0000 7f26177fe700 1 -- 172.21.15.52:0/3206669525 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55b665705640 con 0x55b6656c9df0 2024-05-27T07:25:44.686 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:44.690+0000 7f2617fff700 1 --2- 172.21.15.52:0/3206669525 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55b66584dc00 0x55b665851de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-27T07:25:44.686 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:44.690+0000 7f26177fe700 1 --2- 172.21.15.52:0/3206669525 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55b6656c9df0 0x55b665851360 secure :-1 s=READY pgs=379 cs=0 l=1 rev1=1 crypto rx=0x7f2608000c00 tx=0x7f26080227e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=17f9f9a7d9037ccf server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:25:44.686 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:44.690+0000 7f25f6ffd700 10 client.0 ms_handle_connect on v2:172.21.15.52:3300/0 2024-05-27T07:25:44.686 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:44.690+0000 7f25f6ffd700 1 -- 172.21.15.52:0/3206669525 <== mon.0 v2:172.21.15.52:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f26080aa570 con 0x55b6656c9df0 2024-05-27T07:25:44.686 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:44.690+0000 7f25f6ffd700 1 -- 172.21.15.52:0/3206669525 <== mon.0 v2:172.21.15.52:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9121+0+0 (secure 0 0 0) 0x7f2608092070 con 0x55b6656c9df0 2024-05-27T07:25:44.686 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:44.690+0000 7f262c9e5b80 1 -- 172.21.15.52:0/3206669525 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f261001d040 con 0x55b6656c9df0 2024-05-27T07:25:44.686 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:44.690+0000 7f25f6ffd700 1 -- 172.21.15.52:0/3206669525 <== mon.0 v2:172.21.15.52:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f26080ae920 con 0x55b6656c9df0 2024-05-27T07:25:44.686 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:44.690+0000 7f262c9e5b80 10 client.15285 fetch_fsmap learned FSMap version 29 2024-05-27T07:25:44.686 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:44.690+0000 7f262c9e5b80 1 -- 172.21.15.52:0/3206669525 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x55b66585f1c0 con 0x55b6656c9df0 2024-05-27T07:25:44.686 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:44.690+0000 7f25f6ffd700 1 -- 172.21.15.52:0/3206669525 <== mon.0 v2:172.21.15.52:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f261001d040 con 0x55b6656c9df0 2024-05-27T07:25:44.687 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:44.691+0000 7f25f6ffd700 1 -- 172.21.15.52:0/3206669525 <== mon.0 v2:172.21.15.52:3300/0 5 ==== fsmap(e 29) ==== 8362+0+0 (secure 0 0 0) 0x7f26080a97d0 con 0x55b6656c9df0 2024-05-27T07:25:44.687 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:44.691+0000 7f262c9e5b80 10 client.15285 fetch_fsmap finished waiting for FSMap version 29 2024-05-27T07:25:44.687 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:44.691+0000 7f262c9e5b80 10 client.15285 resolve_mds: resolved name 'h' to daemon mds.h 2024-05-27T07:25:44.687 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:44.691+0000 7f262c9e5b80 20 client.15285 populate_metadata read hostname 'smithi052' 2024-05-27T07:25:44.687 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:44.691+0000 7f262c9e5b80 1 --2- 172.21.15.52:0/3206669525 >> [v2:172.21.15.105:6836/191393149,v1:172.21.15.105:6837/191393149] conn(0x55b665861870 0x55b665862ce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:25:44.687 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:44.691+0000 7f262c9e5b80 4 client.15285 mds_command: new command op to 24497 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-27T07:25:44.687 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:44.691+0000 7f262c9e5b80 1 -- 172.21.15.52:0/3206669525 --> [v2:172.21.15.105:6836/191393149,v1:172.21.15.105:6837/191393149] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55b6658139a0 con 0x55b665861870 2024-05-27T07:25:44.687 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:44.691+0000 7f260effd700 1 --2- 172.21.15.52:0/3206669525 >> [v2:172.21.15.105:6836/191393149,v1:172.21.15.105:6837/191393149] conn(0x55b665861870 0x55b665862ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:25:44.687 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:44.692+0000 7f25f6ffd700 10 client.15285 ms_handle_connect on v2:172.21.15.105:6836/191393149 2024-05-27T07:25:44.687 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:44.692+0000 7f260effd700 1 --2- 172.21.15.52:0/3206669525 >> [v2:172.21.15.105:6836/191393149,v1:172.21.15.105:6837/191393149] conn(0x55b665861870 0x55b665862ce0 crc :-1 s=READY pgs=18 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=b6a0e753a0e56128 server_cookie=31881b72f7647e59 in_seq=0 out_seq=0 2024-05-27T07:25:44.687 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:44.692+0000 7f25f6ffd700 1 -- 172.21.15.52:0/3206669525 <== mds.1 v2:172.21.15.105:6836/191393149 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24041 (crc 0 0 0) 0x55b6658139a0 con 0x55b665861870 2024-05-27T07:25:44.687 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:44.692+0000 7f25f6ffd700 10 client.15285 handle_command_reply: tid=0 2024-05-27T07:25:44.687 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:44.692+0000 7f262c9e5b80 1 client.15285 shutdown 2024-05-27T07:25:44.688 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:44.693+0000 7f262c9e5b80 1 -- 172.21.15.52:0/3206669525 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55b6656c9df0 msgr2=0x55b665851360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:25:44.688 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:44.693+0000 7f262c9e5b80 1 --2- 172.21.15.52:0/3206669525 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55b6656c9df0 0x55b665851360 secure :-1 s=READY pgs=379 cs=0 l=1 rev1=1 crypto rx=0x7f2608000c00 tx=0x7f26080227e0 comp rx=0 tx=0).stop 2024-05-27T07:25:44.688 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:44.693+0000 7f262c9e5b80 1 -- 172.21.15.52:0/3206669525 shutdown_connections 2024-05-27T07:25:44.688 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:44.693+0000 7f262c9e5b80 1 --2- 172.21.15.52:0/3206669525 >> [v2:172.21.15.105:6836/191393149,v1:172.21.15.105:6837/191393149] conn(0x55b665861870 0x55b665862ce0 crc :-1 s=READY pgs=18 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:25:44.688 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:44.693+0000 7f262c9e5b80 1 --2- 172.21.15.52:0/3206669525 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55b66584dc00 0x55b665851de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:25:44.688 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:44.693+0000 7f262c9e5b80 1 --2- 172.21.15.52:0/3206669525 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55b6656c9df0 0x55b665851360 unknown :-1 s=CLOSED pgs=379 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:25:44.688 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:44.693+0000 7f262c9e5b80 1 --2- 172.21.15.52:0/3206669525 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55b665847f30 0x55b6658518a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:25:44.688 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:44.693+0000 7f262c9e5b80 1 -- 172.21.15.52:0/3206669525 >> 172.21.15.52:0/3206669525 conn(0x55b665724aa0 msgr2=0x55b6658469e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:25:44.688 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:44.693+0000 7f25f6ffd700 0 client.15285 ms_handle_reset on v2:172.21.15.105:6836/191393149 2024-05-27T07:25:44.688 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:44.693+0000 7f262c9e5b80 1 -- 172.21.15.52:0/3206669525 shutdown_connections 2024-05-27T07:25:44.688 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:44.693+0000 7f262c9e5b80 1 -- 172.21.15.52:0/3206669525 wait complete. 2024-05-27T07:25:44.688 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:44.693+0000 7f262c9e5b80 20 client.15285 trim_cache size 0 max 16384 2024-05-27T07:25:44.706 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:44.714+0000 7f262c9e5b80 1 Processor -- start 2024-05-27T07:25:44.706 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:44.714+0000 7f262c9e5b80 1 -- start start 2024-05-27T07:25:44.706 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:44.715+0000 7f262c9e5b80 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55b6656c9df0 0x55b6658479f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:25:44.707 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:44.715+0000 7f262c9e5b80 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55b665847f30 0x55b665848340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:25:44.707 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:44.715+0000 7f262c9e5b80 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55b66584dc00 0x55b66585d990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:25:44.707 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:44.715+0000 7f262c9e5b80 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x55b665861020 con 0x55b665847f30 2024-05-27T07:25:44.707 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:44.715+0000 7f262c9e5b80 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x55b6658616f0 con 0x55b6656c9df0 2024-05-27T07:25:44.707 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:44.715+0000 7f262c9e5b80 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x55b665861dc0 con 0x55b66584dc00 2024-05-27T07:25:44.707 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:44.715+0000 7f260effd700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55b665847f30 0x55b665848340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:25:44.707 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:44.715+0000 7f260effd700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55b665847f30 0x55b665848340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.52:48188/0 (socket says 172.21.15.52:48188) 2024-05-27T07:25:44.707 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:44.715+0000 7f260effd700 1 -- 172.21.15.52:0/2789496679 learned_addr learned my addr 172.21.15.52:0/2789496679 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-05-27T07:25:44.707 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:44.715+0000 7f26177fe700 1 --2- 172.21.15.52:0/2789496679 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55b6656c9df0 0x55b6658479f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:25:44.707 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:44.715+0000 7f2617fff700 1 --2- 172.21.15.52:0/2789496679 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55b66584dc00 0x55b66585d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:25:44.708 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:44.715+0000 7f260effd700 1 -- 172.21.15.52:0/2789496679 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55b66584dc00 msgr2=0x55b66585d990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:25:44.708 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:44.715+0000 7f260effd700 1 --2- 172.21.15.52:0/2789496679 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55b66584dc00 0x55b66585d990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:25:44.708 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:44.715+0000 7f260effd700 1 -- 172.21.15.52:0/2789496679 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55b6656c9df0 msgr2=0x55b6658479f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:25:44.708 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:44.715+0000 7f260effd700 1 --2- 172.21.15.52:0/2789496679 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55b6656c9df0 0x55b6658479f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:25:44.708 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:44.715+0000 7f260effd700 1 -- 172.21.15.52:0/2789496679 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55b6658489a0 con 0x55b665847f30 2024-05-27T07:25:44.708 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:44.715+0000 7f2617fff700 1 --2- 172.21.15.52:0/2789496679 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55b66584dc00 0x55b66585d990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-27T07:25:44.708 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:44.715+0000 7f260effd700 1 --2- 172.21.15.52:0/2789496679 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55b665847f30 0x55b665848340 secure :-1 s=READY pgs=380 cs=0 l=1 rev1=1 crypto rx=0x7f260001dfd0 tx=0x7f260000bba0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d8bf39a301c880ef server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:25:44.708 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:44.716+0000 7f260d7fa700 1 -- 172.21.15.52:0/2789496679 <== mon.0 v2:172.21.15.52:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2600015070 con 0x55b665847f30 2024-05-27T07:25:44.708 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:44.716+0000 7f260d7fa700 1 -- 172.21.15.52:0/2789496679 <== mon.0 v2:172.21.15.52:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f2600005020 con 0x55b665847f30 2024-05-27T07:25:44.708 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:44.716+0000 7f260d7fa700 1 -- 172.21.15.52:0/2789496679 <== mon.0 v2:172.21.15.52:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2600003c60 con 0x55b665847f30 2024-05-27T07:25:44.712 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:44.716+0000 7f262c9e5b80 1 -- 172.21.15.52:0/2789496679 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55b665847f30 msgr2=0x55b665848340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:25:44.712 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:44.716+0000 7f262c9e5b80 1 --2- 172.21.15.52:0/2789496679 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55b665847f30 0x55b665848340 secure :-1 s=READY pgs=380 cs=0 l=1 rev1=1 crypto rx=0x7f260001dfd0 tx=0x7f260000bba0 comp rx=0 tx=0).stop 2024-05-27T07:25:44.712 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:44.716+0000 7f262c9e5b80 1 -- 172.21.15.52:0/2789496679 shutdown_connections 2024-05-27T07:25:44.712 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:44.716+0000 7f262c9e5b80 1 --2- 172.21.15.52:0/2789496679 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55b66584dc00 0x55b66585d990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:25:44.712 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:44.716+0000 7f262c9e5b80 1 --2- 172.21.15.52:0/2789496679 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55b6656c9df0 0x55b6658479f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:25:44.712 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:44.716+0000 7f262c9e5b80 1 --2- 172.21.15.52:0/2789496679 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55b665847f30 0x55b665848340 unknown :-1 s=CLOSED pgs=380 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:25:44.712 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:44.716+0000 7f262c9e5b80 1 -- 172.21.15.52:0/2789496679 >> 172.21.15.52:0/2789496679 conn(0x55b665724aa0 msgr2=0x55b665860510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:25:44.712 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:44.717+0000 7f262c9e5b80 1 -- 172.21.15.52:0/2789496679 shutdown_connections 2024-05-27T07:25:44.712 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:44.717+0000 7f262c9e5b80 1 -- 172.21.15.52:0/2789496679 wait complete. 2024-05-27T07:25:44.712 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:44.717+0000 7f262c9e5b80 1 -- start start 2024-05-27T07:25:44.712 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:44.717+0000 7f262c9e5b80 1 Processor -- start 2024-05-27T07:25:44.713 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:44.717+0000 7f262c9e5b80 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55b6656c9df0 0x55b66584ee00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:25:44.713 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:44.717+0000 7f262c9e5b80 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55b665847f30 0x55b6658469e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:25:44.713 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:44.717+0000 7f262c9e5b80 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55b66584dc00 0x55b665846f20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:25:44.713 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:44.717+0000 7f262c9e5b80 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x55b665862b60 con 0x55b665847f30 2024-05-27T07:25:44.713 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:44.717+0000 7f262c9e5b80 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x55b665862490 con 0x55b6656c9df0 2024-05-27T07:25:44.713 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:44.717+0000 7f262c9e5b80 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x55b665863230 con 0x55b66584dc00 2024-05-27T07:25:44.713 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:44.718+0000 7f262c9e5b80 10 client.0 _ll_register_callbacks cb 0x55b6657018d0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T07:25:44.713 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:44.718+0000 7f260effd700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55b665847f30 0x55b6658469e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:25:44.713 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:44.718+0000 7f260effd700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55b665847f30 0x55b6658469e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.52:48204/0 (socket says 172.21.15.52:48204) 2024-05-27T07:25:44.713 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:44.718+0000 7f260effd700 1 -- 172.21.15.52:0/1306814504 learned_addr learned my addr 172.21.15.52:0/1306814504 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-05-27T07:25:44.713 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:44.718+0000 7f26177fe700 1 --2- 172.21.15.52:0/1306814504 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55b6656c9df0 0x55b66584ee00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:25:44.714 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:44.718+0000 7f2617fff700 1 --2- 172.21.15.52:0/1306814504 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55b66584dc00 0x55b665846f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:25:44.714 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:44.718+0000 7f260effd700 1 -- 172.21.15.52:0/1306814504 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55b66584dc00 msgr2=0x55b665846f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:25:44.714 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:44.718+0000 7f260effd700 1 --2- 172.21.15.52:0/1306814504 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55b66584dc00 0x55b665846f20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:25:44.714 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:44.718+0000 7f260effd700 1 -- 172.21.15.52:0/1306814504 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55b6656c9df0 msgr2=0x55b66584ee00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:25:44.714 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:44.718+0000 7f260effd700 1 --2- 172.21.15.52:0/1306814504 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55b6656c9df0 0x55b66584ee00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:25:44.714 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:44.718+0000 7f260effd700 1 -- 172.21.15.52:0/1306814504 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55b6658474b0 con 0x55b665847f30 2024-05-27T07:25:44.714 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:44.718+0000 7f2617fff700 1 --2- 172.21.15.52:0/1306814504 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55b66584dc00 0x55b665846f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-27T07:25:44.714 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:44.718+0000 7f260effd700 1 --2- 172.21.15.52:0/1306814504 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55b665847f30 0x55b6658469e0 secure :-1 s=READY pgs=381 cs=0 l=1 rev1=1 crypto rx=0x7f2600003970 tx=0x7f26000039a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=95e38b47440e3e76 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:25:44.714 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:44.718+0000 7f25f6ffd700 10 client.0 ms_handle_connect on v2:172.21.15.52:3300/0 2024-05-27T07:25:44.714 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:44.718+0000 7f25f6ffd700 1 -- 172.21.15.52:0/1306814504 <== mon.0 v2:172.21.15.52:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f260002c030 con 0x55b665847f30 2024-05-27T07:25:44.714 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:44.718+0000 7f25f6ffd700 1 -- 172.21.15.52:0/1306814504 <== mon.0 v2:172.21.15.52:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9121+0+0 (secure 0 0 0) 0x7f26000053e0 con 0x55b665847f30 2024-05-27T07:25:44.715 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:44.718+0000 7f262c9e5b80 1 -- 172.21.15.52:0/1306814504 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f2600005020 con 0x55b665847f30 2024-05-27T07:25:44.715 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:44.719+0000 7f25f6ffd700 1 -- 172.21.15.52:0/1306814504 <== mon.0 v2:172.21.15.52:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f2600036050 con 0x55b665847f30 2024-05-27T07:25:44.715 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:44.719+0000 7f262c9e5b80 10 client.15291 fetch_fsmap learned FSMap version 29 2024-05-27T07:25:44.715 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:44.719+0000 7f262c9e5b80 1 -- 172.21.15.52:0/1306814504 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x55b66586c3d0 con 0x55b665847f30 2024-05-27T07:25:44.715 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:44.719+0000 7f25f6ffd700 1 -- 172.21.15.52:0/1306814504 <== mon.0 v2:172.21.15.52:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f2600005020 con 0x55b665847f30 2024-05-27T07:25:44.715 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:44.719+0000 7f25f6ffd700 1 -- 172.21.15.52:0/1306814504 <== mon.0 v2:172.21.15.52:3300/0 5 ==== fsmap(e 29) ==== 8362+0+0 (secure 0 0 0) 0x7f260001d5b0 con 0x55b665847f30 2024-05-27T07:25:44.715 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:44.719+0000 7f262c9e5b80 10 client.15291 fetch_fsmap finished waiting for FSMap version 29 2024-05-27T07:25:44.715 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:44.719+0000 7f262c9e5b80 10 client.15291 resolve_mds: resolved name 'h' to daemon mds.h 2024-05-27T07:25:44.715 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:44.719+0000 7f262c9e5b80 20 client.15291 populate_metadata read hostname 'smithi052' 2024-05-27T07:25:44.715 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:44.719+0000 7f262c9e5b80 1 --2- 172.21.15.52:0/1306814504 >> [v2:172.21.15.105:6836/191393149,v1:172.21.15.105:6837/191393149] conn(0x55b6658ce7f0 0x55b6658cfc60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:25:44.715 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:44.719+0000 7f262c9e5b80 4 client.15291 mds_command: new command op to 24497 tid=0 multi_id=0 [{"prefix": "status"}] 2024-05-27T07:25:44.715 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:44.719+0000 7f262c9e5b80 1 -- 172.21.15.52:0/1306814504 --> [v2:172.21.15.105:6836/191393149,v1:172.21.15.105:6837/191393149] -- command(tid 0: {"prefix": "status"}) -- 0x55b665778b90 con 0x55b6658ce7f0 2024-05-27T07:25:44.715 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:44.719+0000 7f26177fe700 1 --2- 172.21.15.52:0/1306814504 >> [v2:172.21.15.105:6836/191393149,v1:172.21.15.105:6837/191393149] conn(0x55b6658ce7f0 0x55b6658cfc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:25:44.715 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:44.720+0000 7f25f6ffd700 10 client.15291 ms_handle_connect on v2:172.21.15.105:6836/191393149 2024-05-27T07:25:44.716 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:44.720+0000 7f26177fe700 1 --2- 172.21.15.52:0/1306814504 >> [v2:172.21.15.105:6836/191393149,v1:172.21.15.105:6837/191393149] conn(0x55b6658ce7f0 0x55b6658cfc60 crc :-1 s=READY pgs=19 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=77f1d8dbede306df server_cookie=f42a5d5dc209e3c3 in_seq=0 out_seq=0 2024-05-27T07:25:44.716 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:44.720+0000 7f25f6ffd700 1 -- 172.21.15.52:0/1306814504 <== mds.1 v2:172.21.15.105:6836/191393149 1 ==== command_reply(tid 0: 0 ) ==== 8+0+332 (crc 0 0 0) 0x55b665778b90 con 0x55b6658ce7f0 2024-05-27T07:25:44.716 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:44.720+0000 7f25f6ffd700 10 client.15291 handle_command_reply: tid=0 2024-05-27T07:25:44.716 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:44.720+0000 7f262c9e5b80 1 client.15291 shutdown 2024-05-27T07:25:44.716 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:44.720+0000 7f262c9e5b80 1 -- 172.21.15.52:0/1306814504 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55b665847f30 msgr2=0x55b6658469e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:25:44.716 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:44.720+0000 7f262c9e5b80 1 --2- 172.21.15.52:0/1306814504 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55b665847f30 0x55b6658469e0 secure :-1 s=READY pgs=381 cs=0 l=1 rev1=1 crypto rx=0x7f2600003970 tx=0x7f26000039a0 comp rx=0 tx=0).stop 2024-05-27T07:25:44.716 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:44.720+0000 7f262c9e5b80 1 -- 172.21.15.52:0/1306814504 shutdown_connections 2024-05-27T07:25:44.716 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:44.720+0000 7f262c9e5b80 1 --2- 172.21.15.52:0/1306814504 >> [v2:172.21.15.105:6836/191393149,v1:172.21.15.105:6837/191393149] conn(0x55b6658ce7f0 0x55b6658cfc60 crc :-1 s=READY pgs=19 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:25:44.716 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:44.720+0000 7f262c9e5b80 1 --2- 172.21.15.52:0/1306814504 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55b66584dc00 0x55b665846f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:25:44.716 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:44.720+0000 7f262c9e5b80 1 --2- 172.21.15.52:0/1306814504 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55b6656c9df0 0x55b66584ee00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:25:44.716 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:44.720+0000 7f262c9e5b80 1 --2- 172.21.15.52:0/1306814504 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55b665847f30 0x55b6658469e0 unknown :-1 s=CLOSED pgs=381 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:25:44.716 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:44.720+0000 7f262c9e5b80 1 -- 172.21.15.52:0/1306814504 >> 172.21.15.52:0/1306814504 conn(0x55b665724aa0 msgr2=0x55b665860510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:25:44.716 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:44.720+0000 7f25f6ffd700 0 client.15291 ms_handle_reset on v2:172.21.15.105:6836/191393149 2024-05-27T07:25:44.717 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:44.721+0000 7f262c9e5b80 1 -- 172.21.15.52:0/1306814504 shutdown_connections 2024-05-27T07:25:44.717 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:44.721+0000 7f262c9e5b80 1 -- 172.21.15.52:0/1306814504 wait complete. 2024-05-27T07:25:44.717 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:44.721+0000 7f262c9e5b80 20 client.15291 trim_cache size 0 max 16384 2024-05-27T07:25:44.717 INFO:teuthology.orchestra.run.smithi052.stdout:{ 2024-05-27T07:25:44.717 INFO:teuthology.orchestra.run.smithi052.stdout: "cluster_fsid": "73df2636-1bf8-11ef-bc9b-c7b262605968", 2024-05-27T07:25:44.717 INFO:teuthology.orchestra.run.smithi052.stdout: "whoami": 1, 2024-05-27T07:25:44.717 INFO:teuthology.orchestra.run.smithi052.stdout: "id": 24497, 2024-05-27T07:25:44.717 INFO:teuthology.orchestra.run.smithi052.stdout: "want_state": "up:active", 2024-05-27T07:25:44.717 INFO:teuthology.orchestra.run.smithi052.stdout: "state": "up:active", 2024-05-27T07:25:44.717 INFO:teuthology.orchestra.run.smithi052.stdout: "fs_name": "cephfs", 2024-05-27T07:25:44.718 INFO:teuthology.orchestra.run.smithi052.stdout: "rank_uptime": 171.87404809200001, 2024-05-27T07:25:44.718 INFO:teuthology.orchestra.run.smithi052.stdout: "mdsmap_epoch": 29, 2024-05-27T07:25:44.718 INFO:teuthology.orchestra.run.smithi052.stdout: "osdmap_epoch": 90, 2024-05-27T07:25:44.718 INFO:teuthology.orchestra.run.smithi052.stdout: "osdmap_epoch_barrier": 88, 2024-05-27T07:25:44.718 INFO:teuthology.orchestra.run.smithi052.stdout: "uptime": 261.13331528800001 2024-05-27T07:25:44.718 INFO:teuthology.orchestra.run.smithi052.stdout:} 2024-05-27T07:25:44.718 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:44.721+0000 7f261dfd0700 1 -- 172.21.15.52:0/1292848734 >> [v2:172.21.15.52:6800/3513075805,v1:172.21.15.52:6801/3513075805] conn(0x7f26040b2880 msgr2=0x7f26040b4d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:25:44.719 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:44.721+0000 7f261dfd0700 1 --2- 172.21.15.52:0/1292848734 >> [v2:172.21.15.52:6800/3513075805,v1:172.21.15.52:6801/3513075805] conn(0x7f26040b2880 0x7f26040b4d90 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7f2610009c60 tx=0x7f26100077e0 comp rx=0 tx=0).stop 2024-05-27T07:25:44.719 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:44.721+0000 7f261dfd0700 1 -- 172.21.15.52:0/1292848734 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f26181034d0 msgr2=0x7f261819aee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:25:44.719 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:44.721+0000 7f261dfd0700 1 --2- 172.21.15.52:0/1292848734 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f26181034d0 0x7f261819aee0 secure :-1 s=READY pgs=377 cs=0 l=1 rev1=1 crypto rx=0x7f260800b580 tx=0x7f26080047b0 comp rx=0 tx=0).stop 2024-05-27T07:25:44.719 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:44.722+0000 7f261dfd0700 1 -- 172.21.15.52:0/1292848734 shutdown_connections 2024-05-27T07:25:44.719 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:44.722+0000 7f261dfd0700 1 --2- 172.21.15.52:0/1292848734 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f2618108750 0x7f261819f710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:25:44.719 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:44.722+0000 7f261dfd0700 1 --2- 172.21.15.52:0/1292848734 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f26181034d0 0x7f261819aee0 unknown :-1 s=CLOSED pgs=377 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:25:44.719 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:44.722+0000 7f261dfd0700 1 --2- 172.21.15.52:0/1292848734 >> [v2:172.21.15.52:6800/3513075805,v1:172.21.15.52:6801/3513075805] conn(0x7f26040b2880 0x7f26040b4d90 unknown :-1 s=CLOSED pgs=225 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:25:44.719 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:44.722+0000 7f261dfd0700 1 --2- 172.21.15.52:0/1292848734 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f2618107e00 0x7f261819b480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:25:44.719 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:44.722+0000 7f261dfd0700 1 -- 172.21.15.52:0/1292848734 >> 172.21.15.52:0/1292848734 conn(0x7f26180fef20 msgr2=0x7f2618101eb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:25:44.719 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:44.722+0000 7f261dfd0700 1 -- 172.21.15.52:0/1292848734 shutdown_connections 2024-05-27T07:25:44.719 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:44.722+0000 7f261dfd0700 1 -- 172.21.15.52:0/1292848734 wait complete. 2024-05-27T07:25:45.351 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:25:45 smithi173 ceph-mon[37000]: pgmap v417: 129 pgs: 129 active+clean; 647 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 9 op/s 2024-05-27T07:25:45.439 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:25:45 smithi105 ceph-mon[35977]: pgmap v417: 129 pgs: 129 active+clean; 647 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 9 op/s 2024-05-27T07:25:45.480 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:25:45 smithi052 ceph-mon[31955]: pgmap v417: 129 pgs: 129 active+clean; 647 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 9 op/s 2024-05-27T07:25:45.523 DEBUG:teuthology.orchestra.run.smithi052:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:38f9d900c83c63799fbdbe61acc9a11b0d3554a6 shell --fsid 73df2636-1bf8-11ef-bc9b-c7b262605968 -- ceph tell mds.b status 2024-05-27T07:25:45.778 INFO:teuthology.orchestra.run.smithi052.stderr:Inferring config /var/lib/ceph/73df2636-1bf8-11ef-bc9b-c7b262605968/mon.a/config 2024-05-27T07:25:47.166 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:25:47 smithi052 ceph-mon[31955]: pgmap v418: 129 pgs: 129 active+clean; 647 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 11 op/s 2024-05-27T07:25:47.352 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:25:47 smithi173 ceph-mon[37000]: pgmap v418: 129 pgs: 129 active+clean; 647 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 11 op/s 2024-05-27T07:25:47.440 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:25:47 smithi105 ceph-mon[35977]: pgmap v418: 129 pgs: 129 active+clean; 647 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 11 op/s 2024-05-27T07:25:47.443 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:47.449+0000 7f02b1a00700 1 -- 172.21.15.52:0/2733337698 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f02ac1096b0 msgr2=0x7f02ac111010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:25:47.443 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:47.449+0000 7f02b1a00700 1 --2- 172.21.15.52:0/2733337698 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f02ac1096b0 0x7f02ac111010 secure :-1 s=READY pgs=382 cs=0 l=1 rev1=1 crypto rx=0x7f02a40099c0 tx=0x7f02a4009cd0 comp rx=0 tx=0).stop 2024-05-27T07:25:47.443 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:47.449+0000 7f02b1a00700 1 -- 172.21.15.52:0/2733337698 shutdown_connections 2024-05-27T07:25:47.443 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:47.449+0000 7f02b1a00700 1 --2- 172.21.15.52:0/2733337698 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f02ac1096b0 0x7f02ac111010 unknown :-1 s=CLOSED pgs=382 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:25:47.443 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:47.449+0000 7f02b1a00700 1 --2- 172.21.15.52:0/2733337698 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f02ac104430 0x7f02ac104840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:25:47.444 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:47.449+0000 7f02b1a00700 1 --2- 172.21.15.52:0/2733337698 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f02ac108d60 0x7f02ac109170 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:25:47.444 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:47.449+0000 7f02b1a00700 1 -- 172.21.15.52:0/2733337698 >> 172.21.15.52:0/2733337698 conn(0x7f02ac079cc0 msgr2=0x7f02ac07a120 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:25:47.444 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:47.449+0000 7f02b1a00700 1 -- 172.21.15.52:0/2733337698 shutdown_connections 2024-05-27T07:25:47.444 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:47.449+0000 7f02b1a00700 1 -- 172.21.15.52:0/2733337698 wait complete. 2024-05-27T07:25:47.444 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:47.450+0000 7f02b1a00700 1 Processor -- start 2024-05-27T07:25:47.444 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:47.450+0000 7f02b1a00700 1 -- start start 2024-05-27T07:25:47.445 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:47.450+0000 7f02b1a00700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f02ac104430 0x7f02ac1a1fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:25:47.445 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:47.450+0000 7f02b1a00700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f02ac108d60 0x7f02ac1a2500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:25:47.445 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:47.450+0000 7f02b1a00700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f02ac1096b0 0x7f02ac1ad040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:25:47.445 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:47.450+0000 7f02b1a00700 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x7f02ac1142e0 con 0x7f02ac104430 2024-05-27T07:25:47.445 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:47.450+0000 7f02b1a00700 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f02ac114160 con 0x7f02ac108d60 2024-05-27T07:25:47.445 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:47.450+0000 7f02b1a00700 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f02ac114460 con 0x7f02ac1096b0 2024-05-27T07:25:47.446 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:47.450+0000 7f02aaffd700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f02ac104430 0x7f02ac1a1fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:25:47.446 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:47.450+0000 7f02aaffd700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f02ac104430 0x7f02ac1a1fc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.52:48224/0 (socket says 172.21.15.52:48224) 2024-05-27T07:25:47.446 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:47.450+0000 7f02aaffd700 1 -- 172.21.15.52:0/3647081071 learned_addr learned my addr 172.21.15.52:0/3647081071 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-05-27T07:25:47.446 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:47.451+0000 7f02aa7fc700 1 --2- 172.21.15.52:0/3647081071 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f02ac108d60 0x7f02ac1a2500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:25:47.446 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:47.451+0000 7f02ab7fe700 1 --2- 172.21.15.52:0/3647081071 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f02ac1096b0 0x7f02ac1ad040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:25:47.446 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:47.451+0000 7f02aaffd700 1 -- 172.21.15.52:0/3647081071 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f02ac1096b0 msgr2=0x7f02ac1ad040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:25:47.447 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:47.451+0000 7f02aaffd700 1 --2- 172.21.15.52:0/3647081071 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f02ac1096b0 0x7f02ac1ad040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:25:47.447 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:47.451+0000 7f02aaffd700 1 -- 172.21.15.52:0/3647081071 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f02ac108d60 msgr2=0x7f02ac1a2500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:25:47.447 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:47.451+0000 7f02aaffd700 1 --2- 172.21.15.52:0/3647081071 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f02ac108d60 0x7f02ac1a2500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:25:47.447 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:47.451+0000 7f02aaffd700 1 -- 172.21.15.52:0/3647081071 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f02ac1a2dc0 con 0x7f02ac104430 2024-05-27T07:25:47.447 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:47.451+0000 7f02aaffd700 1 --2- 172.21.15.52:0/3647081071 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f02ac104430 0x7f02ac1a1fc0 secure :-1 s=READY pgs=383 cs=0 l=1 rev1=1 crypto rx=0x7f029c0060b0 tx=0x7f029c00d3b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:25:47.447 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:47.451+0000 7f02b09fe700 1 -- 172.21.15.52:0/3647081071 <== mon.0 v2:172.21.15.52:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f029c01a070 con 0x7f02ac104430 2024-05-27T07:25:47.448 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:47.451+0000 7f02b1a00700 1 -- 172.21.15.52:0/3647081071 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f02ac1ad6a0 con 0x7f02ac104430 2024-05-27T07:25:47.448 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:47.451+0000 7f02b09fe700 1 -- 172.21.15.52:0/3647081071 <== mon.0 v2:172.21.15.52:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f029c00dd20 con 0x7f02ac104430 2024-05-27T07:25:47.448 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:47.451+0000 7f02b09fe700 1 -- 172.21.15.52:0/3647081071 <== mon.0 v2:172.21.15.52:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f029c014600 con 0x7f02ac104430 2024-05-27T07:25:47.448 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:47.451+0000 7f02b1a00700 1 -- 172.21.15.52:0/3647081071 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f02ac1adc00 con 0x7f02ac104430 2024-05-27T07:25:47.457 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:47.454+0000 7f02b09fe700 1 -- 172.21.15.52:0/3647081071 <== mon.0 v2:172.21.15.52:3300/0 4 ==== mgrmap(e 18) ==== 149652+0+0 (secure 0 0 0) 0x7f029c01d070 con 0x7f02ac104430 2024-05-27T07:25:47.458 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:47.455+0000 7f02b09fe700 1 --2- 172.21.15.52:0/3647081071 >> [v2:172.21.15.52:6800/3513075805,v1:172.21.15.52:6801/3513075805] conn(0x7f02940b2900 0x7f02940b4e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:25:47.458 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:47.455+0000 7f02b09fe700 1 -- 172.21.15.52:0/3647081071 <== mon.0 v2:172.21.15.52:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9121+0+0 (secure 0 0 0) 0x7f029c0e1d70 con 0x7f02ac104430 2024-05-27T07:25:47.458 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:47.455+0000 7f02aa7fc700 1 --2- 172.21.15.52:0/3647081071 >> [v2:172.21.15.52:6800/3513075805,v1:172.21.15.52:6801/3513075805] conn(0x7f02940b2900 0x7f02940b4e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:25:47.458 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:47.455+0000 7f02c0415b80 1 Processor -- start 2024-05-27T07:25:47.458 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:47.455+0000 7f02c0415b80 1 -- start start 2024-05-27T07:25:47.459 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:47.456+0000 7f02c0415b80 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x560d42533df0 0x560d426b1fc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:25:47.459 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:47.456+0000 7f02aa7fc700 1 --2- 172.21.15.52:0/3647081071 >> [v2:172.21.15.52:6800/3513075805,v1:172.21.15.52:6801/3513075805] conn(0x7f02940b2900 0x7f02940b4e10 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7f02ac10c130 tx=0x7f02a0009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:25:47.459 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:47.456+0000 7f02c0415b80 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x560d426b2500 0x560d426b2910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:25:47.459 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:47.456+0000 7f02c0415b80 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x560d426bcd90 0x560d426bf1b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:25:47.459 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:47.456+0000 7f02c0415b80 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x560d425e0d20 con 0x560d42533df0 2024-05-27T07:25:47.459 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:47.456+0000 7f02c0415b80 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x560d425d08c0 con 0x560d426b2500 2024-05-27T07:25:47.459 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:47.456+0000 7f02c0415b80 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x560d425d4900 con 0x560d426bcd90 2024-05-27T07:25:47.460 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:47.456+0000 7f02aaffd700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x560d42533df0 0x560d426b1fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:25:47.460 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:47.456+0000 7f02aaffd700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x560d42533df0 0x560d426b1fc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.52:48228/0 (socket says 172.21.15.52:48228) 2024-05-27T07:25:47.460 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:47.456+0000 7f02aaffd700 1 -- 172.21.15.52:0/281392857 learned_addr learned my addr 172.21.15.52:0/281392857 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-05-27T07:25:47.460 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:47.456+0000 7f02aa7fc700 1 --2- 172.21.15.52:0/281392857 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x560d426b2500 0x560d426b2910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:25:47.460 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:47.456+0000 7f02ab7fe700 1 --2- 172.21.15.52:0/281392857 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x560d426bcd90 0x560d426bf1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:25:47.460 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:47.456+0000 7f02aaffd700 1 -- 172.21.15.52:0/281392857 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x560d426bcd90 msgr2=0x560d426bf1b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:25:47.461 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:47.456+0000 7f02aaffd700 1 --2- 172.21.15.52:0/281392857 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x560d426bcd90 0x560d426bf1b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:25:47.461 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:47.456+0000 7f02aaffd700 1 -- 172.21.15.52:0/281392857 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x560d426b2500 msgr2=0x560d426b2910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:25:47.461 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:47.456+0000 7f02aaffd700 1 --2- 172.21.15.52:0/281392857 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x560d426b2500 0x560d426b2910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:25:47.461 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:47.456+0000 7f02aaffd700 1 -- 172.21.15.52:0/281392857 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x560d42579f60 con 0x560d42533df0 2024-05-27T07:25:47.461 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:47.456+0000 7f02aa7fc700 1 --2- 172.21.15.52:0/281392857 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x560d426b2500 0x560d426b2910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-27T07:25:47.461 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:47.457+0000 7f02aaffd700 1 --2- 172.21.15.52:0/281392857 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x560d42533df0 0x560d426b1fc0 secure :-1 s=READY pgs=384 cs=0 l=1 rev1=1 crypto rx=0x7f029c0a5030 tx=0x7f029c0a8820 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c5775941272f6ad4 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:25:47.461 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:47.457+0000 7f0291ffb700 1 -- 172.21.15.52:0/281392857 <== mon.0 v2:172.21.15.52:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f029c0ad360 con 0x560d42533df0 2024-05-27T07:25:47.462 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:47.457+0000 7f0291ffb700 1 -- 172.21.15.52:0/281392857 <== mon.0 v2:172.21.15.52:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f029c0b1e10 con 0x560d42533df0 2024-05-27T07:25:47.462 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:47.457+0000 7f0291ffb700 1 -- 172.21.15.52:0/281392857 <== mon.0 v2:172.21.15.52:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f029c0a4920 con 0x560d42533df0 2024-05-27T07:25:47.462 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:47.458+0000 7f02c0415b80 1 -- 172.21.15.52:0/281392857 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x560d42533df0 msgr2=0x560d426b1fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:25:47.462 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:47.458+0000 7f02c0415b80 1 --2- 172.21.15.52:0/281392857 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x560d42533df0 0x560d426b1fc0 secure :-1 s=READY pgs=384 cs=0 l=1 rev1=1 crypto rx=0x7f029c0a5030 tx=0x7f029c0a8820 comp rx=0 tx=0).stop 2024-05-27T07:25:47.462 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:47.458+0000 7f02c0415b80 1 -- 172.21.15.52:0/281392857 shutdown_connections 2024-05-27T07:25:47.462 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:47.458+0000 7f02c0415b80 1 --2- 172.21.15.52:0/281392857 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x560d426bcd90 0x560d426bf1b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:25:47.463 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:47.458+0000 7f02c0415b80 1 --2- 172.21.15.52:0/281392857 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x560d42533df0 0x560d426b1fc0 unknown :-1 s=CLOSED pgs=384 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:25:47.463 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:47.458+0000 7f02c0415b80 1 --2- 172.21.15.52:0/281392857 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x560d426b2500 0x560d426b2910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:25:47.463 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:47.458+0000 7f02c0415b80 1 -- 172.21.15.52:0/281392857 >> 172.21.15.52:0/281392857 conn(0x560d42536720 msgr2=0x560d426adff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:25:47.463 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:47.458+0000 7f02c0415b80 1 -- 172.21.15.52:0/281392857 shutdown_connections 2024-05-27T07:25:47.463 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:47.458+0000 7f02c0415b80 1 -- 172.21.15.52:0/281392857 wait complete. 2024-05-27T07:25:47.464 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:47.458+0000 7f02c0415b80 1 -- start start 2024-05-27T07:25:47.464 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:47.459+0000 7f02c0415b80 1 Processor -- start 2024-05-27T07:25:47.464 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:47.459+0000 7f02c0415b80 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x560d42533df0 0x560d426b9d40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:25:47.464 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:47.459+0000 7f02c0415b80 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x560d426b2500 0x560d426ba280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:25:47.464 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:47.459+0000 7f02c0415b80 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x560d426bcd90 0x560d426ba7c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:25:47.464 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:47.459+0000 7f02c0415b80 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x560d424d6f50 con 0x560d42533df0 2024-05-27T07:25:47.464 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:47.459+0000 7f02c0415b80 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x560d42538d50 con 0x560d426b2500 2024-05-27T07:25:47.465 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:47.459+0000 7f02c0415b80 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x560d426bfc50 con 0x560d426bcd90 2024-05-27T07:25:47.465 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:47.459+0000 7f02c0415b80 10 client.0 _ll_register_callbacks cb 0x560d42567560 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T07:25:47.465 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:47.459+0000 7f02aaffd700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x560d42533df0 0x560d426b9d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:25:47.465 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:47.459+0000 7f02aaffd700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x560d42533df0 0x560d426b9d40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.52:48244/0 (socket says 172.21.15.52:48244) 2024-05-27T07:25:47.465 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:47.459+0000 7f02aaffd700 1 -- 172.21.15.52:0/1237848264 learned_addr learned my addr 172.21.15.52:0/1237848264 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-05-27T07:25:47.465 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:47.459+0000 7f02ab7fe700 1 --2- 172.21.15.52:0/1237848264 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x560d426bcd90 0x560d426ba7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:25:47.466 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:47.459+0000 7f02aaffd700 1 -- 172.21.15.52:0/1237848264 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x560d426bcd90 msgr2=0x560d426ba7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:25:47.466 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:47.459+0000 7f02aa7fc700 1 --2- 172.21.15.52:0/1237848264 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x560d426b2500 0x560d426ba280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:25:47.466 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:47.459+0000 7f02aaffd700 1 --2- 172.21.15.52:0/1237848264 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x560d426bcd90 0x560d426ba7c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:25:47.466 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:47.459+0000 7f02aaffd700 1 -- 172.21.15.52:0/1237848264 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x560d426b2500 msgr2=0x560d426ba280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:25:47.466 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:47.459+0000 7f02aaffd700 1 --2- 172.21.15.52:0/1237848264 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x560d426b2500 0x560d426ba280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:25:47.466 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:47.459+0000 7f02aaffd700 1 -- 172.21.15.52:0/1237848264 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({osdmap=0}) -- 0x560d42520880 con 0x560d42533df0 2024-05-27T07:25:47.466 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:47.460+0000 7f02aaffd700 1 --2- 172.21.15.52:0/1237848264 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x560d42533df0 0x560d426b9d40 secure :-1 s=READY pgs=385 cs=0 l=1 rev1=1 crypto rx=0x7f029c0a8cb0 tx=0x7f029c015b70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d224094d2f307f4c server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:25:47.467 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:47.460+0000 7f028affd700 10 client.0 ms_handle_connect on v2:172.21.15.52:3300/0 2024-05-27T07:25:47.467 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:47.460+0000 7f028affd700 1 -- 172.21.15.52:0/1237848264 <== mon.0 v2:172.21.15.52:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f029c0c7020 con 0x560d42533df0 2024-05-27T07:25:47.467 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:47.460+0000 7f028affd700 1 -- 172.21.15.52:0/1237848264 <== mon.0 v2:172.21.15.52:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9121+0+0 (secure 0 0 0) 0x7f029c0b1b80 con 0x560d42533df0 2024-05-27T07:25:47.467 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:47.460+0000 7f02c0415b80 1 -- 172.21.15.52:0/1237848264 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f029c0b1e10 con 0x560d42533df0 2024-05-27T07:25:47.467 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:47.460+0000 7f028affd700 1 -- 172.21.15.52:0/1237848264 <== mon.0 v2:172.21.15.52:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f029c0a8210 con 0x560d42533df0 2024-05-27T07:25:47.467 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:47.460+0000 7f02c0415b80 10 client.15303 fetch_fsmap learned FSMap version 29 2024-05-27T07:25:47.468 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:47.460+0000 7f02c0415b80 1 -- 172.21.15.52:0/1237848264 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x560d4258a710 con 0x560d42533df0 2024-05-27T07:25:47.468 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:47.460+0000 7f028affd700 1 -- 172.21.15.52:0/1237848264 <== mon.0 v2:172.21.15.52:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f029c0b1e10 con 0x560d42533df0 2024-05-27T07:25:47.468 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:47.460+0000 7f028affd700 1 -- 172.21.15.52:0/1237848264 <== mon.0 v2:172.21.15.52:3300/0 5 ==== fsmap(e 29) ==== 8362+0+0 (secure 0 0 0) 0x7f029c004ad0 con 0x560d42533df0 2024-05-27T07:25:47.468 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:47.460+0000 7f02c0415b80 10 client.15303 fetch_fsmap finished waiting for FSMap version 29 2024-05-27T07:25:47.468 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:47.460+0000 7f02c0415b80 10 client.15303 resolve_mds: resolved name 'b' to daemon mds.b 2024-05-27T07:25:47.468 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:47.460+0000 7f02c0415b80 20 client.15303 populate_metadata read hostname 'smithi052' 2024-05-27T07:25:47.468 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:47.460+0000 7f02c0415b80 1 --2- 172.21.15.52:0/1237848264 >> [v2:172.21.15.105:6838/500348151,v1:172.21.15.105:6839/500348151] conn(0x560d426cc680 0x560d426cdaf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:25:47.469 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:47.460+0000 7f02c0415b80 4 client.15303 mds_command: new command op to 24527 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-27T07:25:47.469 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:47.460+0000 7f02c0415b80 1 -- 172.21.15.52:0/1237848264 --> [v2:172.21.15.105:6838/500348151,v1:172.21.15.105:6839/500348151] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x560d425e2c70 con 0x560d426cc680 2024-05-27T07:25:47.469 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:47.461+0000 7f02aa7fc700 1 --2- 172.21.15.52:0/1237848264 >> [v2:172.21.15.105:6838/500348151,v1:172.21.15.105:6839/500348151] conn(0x560d426cc680 0x560d426cdaf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:25:47.469 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:47.461+0000 7f02aa7fc700 1 --2- 172.21.15.52:0/1237848264 >> [v2:172.21.15.105:6838/500348151,v1:172.21.15.105:6839/500348151] conn(0x560d426cc680 0x560d426cdaf0 crc :-1 s=READY pgs=30 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=557695c462ec3690 server_cookie=1d99f266558b688 in_seq=0 out_seq=0 2024-05-27T07:25:47.469 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:47.461+0000 7f028affd700 10 client.15303 ms_handle_connect on v2:172.21.15.105:6838/500348151 2024-05-27T07:25:47.469 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:47.462+0000 7f028affd700 1 -- 172.21.15.52:0/1237848264 <== mds.0 v2:172.21.15.105:6838/500348151 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24041 (crc 0 0 0) 0x560d425e2c70 con 0x560d426cc680 2024-05-27T07:25:47.469 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:47.462+0000 7f028affd700 10 client.15303 handle_command_reply: tid=0 2024-05-27T07:25:47.470 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:47.462+0000 7f02c0415b80 1 client.15303 shutdown 2024-05-27T07:25:47.470 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:47.462+0000 7f02c0415b80 1 -- 172.21.15.52:0/1237848264 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x560d42533df0 msgr2=0x560d426b9d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:25:47.470 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:47.462+0000 7f02c0415b80 1 --2- 172.21.15.52:0/1237848264 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x560d42533df0 0x560d426b9d40 secure :-1 s=READY pgs=385 cs=0 l=1 rev1=1 crypto rx=0x7f029c0a8cb0 tx=0x7f029c015b70 comp rx=0 tx=0).stop 2024-05-27T07:25:47.470 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:47.462+0000 7f02c0415b80 1 -- 172.21.15.52:0/1237848264 shutdown_connections 2024-05-27T07:25:47.470 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:47.462+0000 7f02c0415b80 1 --2- 172.21.15.52:0/1237848264 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x560d426bcd90 0x560d426ba7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:25:47.470 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:47.462+0000 7f02c0415b80 1 --2- 172.21.15.52:0/1237848264 >> [v2:172.21.15.105:6838/500348151,v1:172.21.15.105:6839/500348151] conn(0x560d426cc680 0x560d426cdaf0 crc :-1 s=READY pgs=30 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:25:47.471 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:47.462+0000 7f02c0415b80 1 --2- 172.21.15.52:0/1237848264 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x560d42533df0 0x560d426b9d40 unknown :-1 s=CLOSED pgs=385 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:25:47.471 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:47.462+0000 7f02c0415b80 1 --2- 172.21.15.52:0/1237848264 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x560d426b2500 0x560d426ba280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:25:47.471 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:47.462+0000 7f02c0415b80 1 -- 172.21.15.52:0/1237848264 >> 172.21.15.52:0/1237848264 conn(0x560d42536720 msgr2=0x560d426ac550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:25:47.471 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:47.462+0000 7f028affd700 0 client.15303 ms_handle_reset on v2:172.21.15.105:6838/500348151 2024-05-27T07:25:47.471 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:47.462+0000 7f02c0415b80 1 -- 172.21.15.52:0/1237848264 shutdown_connections 2024-05-27T07:25:47.471 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:47.462+0000 7f02c0415b80 1 -- 172.21.15.52:0/1237848264 wait complete. 2024-05-27T07:25:47.471 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:47.463+0000 7f02c0415b80 20 client.15303 trim_cache size 0 max 16384 2024-05-27T07:25:47.483 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:47.484+0000 7f02c0415b80 1 Processor -- start 2024-05-27T07:25:47.483 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:47.484+0000 7f02c0415b80 1 -- start start 2024-05-27T07:25:47.483 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:47.484+0000 7f02c0415b80 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x560d42533df0 0x560d426cb8a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:25:47.483 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:47.484+0000 7f02c0415b80 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x560d426b2500 0x560d426c8f20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:25:47.484 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:47.484+0000 7f02c0415b80 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x560d426bcd90 0x560d426b8d40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:25:47.484 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:47.484+0000 7f02c0415b80 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x560d426cbe30 con 0x560d426bcd90 2024-05-27T07:25:47.484 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:47.484+0000 7f02c0415b80 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x560d426cc500 con 0x560d426b2500 2024-05-27T07:25:47.484 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:47.484+0000 7f02c0415b80 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x560d426ccbd0 con 0x560d42533df0 2024-05-27T07:25:47.484 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:47.484+0000 7f02ab7fe700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x560d426bcd90 0x560d426b8d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:25:47.484 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:47.484+0000 7f02ab7fe700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x560d426bcd90 0x560d426b8d40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.52:48258/0 (socket says 172.21.15.52:48258) 2024-05-27T07:25:47.485 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:47.484+0000 7f02ab7fe700 1 -- 172.21.15.52:0/2226133115 learned_addr learned my addr 172.21.15.52:0/2226133115 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-05-27T07:25:47.485 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:47.485+0000 7f02aaffd700 1 --2- 172.21.15.52:0/2226133115 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x560d42533df0 0x560d426cb8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:25:47.485 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:47.485+0000 7f02aa7fc700 1 --2- 172.21.15.52:0/2226133115 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x560d426b2500 0x560d426c8f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:25:47.485 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:47.485+0000 7f02ab7fe700 1 -- 172.21.15.52:0/2226133115 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x560d42533df0 msgr2=0x560d426cb8a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:25:47.485 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:47.485+0000 7f02ab7fe700 1 --2- 172.21.15.52:0/2226133115 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x560d42533df0 0x560d426cb8a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:25:47.485 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:47.485+0000 7f02ab7fe700 1 -- 172.21.15.52:0/2226133115 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x560d426b2500 msgr2=0x560d426c8f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:25:47.485 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:47.485+0000 7f02ab7fe700 1 --2- 172.21.15.52:0/2226133115 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x560d426b2500 0x560d426c8f20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:25:47.486 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:47.485+0000 7f02ab7fe700 1 -- 172.21.15.52:0/2226133115 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x560d426bc5e0 con 0x560d426bcd90 2024-05-27T07:25:47.486 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:47.485+0000 7f02aaffd700 1 --2- 172.21.15.52:0/2226133115 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x560d42533df0 0x560d426cb8a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-27T07:25:47.486 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:47.485+0000 7f02ab7fe700 1 --2- 172.21.15.52:0/2226133115 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x560d426bcd90 0x560d426b8d40 secure :-1 s=READY pgs=386 cs=0 l=1 rev1=1 crypto rx=0x7f02a4015000 tx=0x7f02a4005040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=887ae17bd4e9b7bc server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:25:47.486 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:47.485+0000 7f02917fa700 1 -- 172.21.15.52:0/2226133115 <== mon.0 v2:172.21.15.52:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f02a4005330 con 0x560d426bcd90 2024-05-27T07:25:47.486 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:47.485+0000 7f02917fa700 1 -- 172.21.15.52:0/2226133115 <== mon.0 v2:172.21.15.52:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f02a40058e0 con 0x560d426bcd90 2024-05-27T07:25:47.486 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:47.485+0000 7f02917fa700 1 -- 172.21.15.52:0/2226133115 <== mon.0 v2:172.21.15.52:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f02a400bc30 con 0x560d426bcd90 2024-05-27T07:25:47.487 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:47.486+0000 7f02c0415b80 1 -- 172.21.15.52:0/2226133115 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x560d426bcd90 msgr2=0x560d426b8d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:25:47.487 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:47.486+0000 7f02c0415b80 1 --2- 172.21.15.52:0/2226133115 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x560d426bcd90 0x560d426b8d40 secure :-1 s=READY pgs=386 cs=0 l=1 rev1=1 crypto rx=0x7f02a4015000 tx=0x7f02a4005040 comp rx=0 tx=0).stop 2024-05-27T07:25:47.487 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:47.486+0000 7f02c0415b80 1 -- 172.21.15.52:0/2226133115 shutdown_connections 2024-05-27T07:25:47.487 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:47.486+0000 7f02c0415b80 1 --2- 172.21.15.52:0/2226133115 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x560d426bcd90 0x560d426b8d40 unknown :-1 s=CLOSED pgs=386 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:25:47.487 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:47.486+0000 7f02c0415b80 1 --2- 172.21.15.52:0/2226133115 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x560d42533df0 0x560d426cb8a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:25:47.487 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:47.486+0000 7f02c0415b80 1 --2- 172.21.15.52:0/2226133115 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x560d426b2500 0x560d426c8f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:25:47.488 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:47.486+0000 7f02c0415b80 1 -- 172.21.15.52:0/2226133115 >> 172.21.15.52:0/2226133115 conn(0x560d42536720 msgr2=0x560d426b8630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:25:47.488 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:47.486+0000 7f02c0415b80 1 -- 172.21.15.52:0/2226133115 shutdown_connections 2024-05-27T07:25:47.488 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:47.486+0000 7f02c0415b80 1 -- 172.21.15.52:0/2226133115 wait complete. 2024-05-27T07:25:47.488 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:47.486+0000 7f02c0415b80 1 -- start start 2024-05-27T07:25:47.488 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:47.487+0000 7f02c0415b80 1 Processor -- start 2024-05-27T07:25:47.488 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:47.487+0000 7f02c0415b80 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x560d42533df0 0x560d426d14f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:25:47.488 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:47.487+0000 7f02c0415b80 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x560d426b2500 0x560d426d1a30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:25:47.489 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:47.487+0000 7f02c0415b80 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x560d426bcd90 0x560d426d1f70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:25:47.489 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:47.487+0000 7f02c0415b80 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x560d426cd970 con 0x560d42533df0 2024-05-27T07:25:47.489 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:47.487+0000 7f02c0415b80 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x560d426cd2a0 con 0x560d426b2500 2024-05-27T07:25:47.489 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:47.487+0000 7f02c0415b80 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x560d426ce040 con 0x560d426bcd90 2024-05-27T07:25:47.489 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:47.487+0000 7f02c0415b80 10 client.0 _ll_register_callbacks cb 0x560d42567560 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T07:25:47.489 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:47.487+0000 7f02aaffd700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x560d42533df0 0x560d426d14f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:25:47.490 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:47.487+0000 7f02aaffd700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x560d42533df0 0x560d426d14f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.52:48272/0 (socket says 172.21.15.52:48272) 2024-05-27T07:25:47.490 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:47.487+0000 7f02aaffd700 1 -- 172.21.15.52:0/589544660 learned_addr learned my addr 172.21.15.52:0/589544660 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-05-27T07:25:47.490 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:47.487+0000 7f02aa7fc700 1 --2- 172.21.15.52:0/589544660 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x560d426b2500 0x560d426d1a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:25:47.490 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:47.487+0000 7f02ab7fe700 1 --2- 172.21.15.52:0/589544660 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x560d426bcd90 0x560d426d1f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:25:47.490 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:47.487+0000 7f02aaffd700 1 -- 172.21.15.52:0/589544660 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x560d426bcd90 msgr2=0x560d426d1f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:25:47.490 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:47.487+0000 7f02aaffd700 1 --2- 172.21.15.52:0/589544660 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x560d426bcd90 0x560d426d1f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:25:47.491 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:47.488+0000 7f02aaffd700 1 -- 172.21.15.52:0/589544660 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x560d426b2500 msgr2=0x560d426d1a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:25:47.491 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:47.488+0000 7f02aaffd700 1 --2- 172.21.15.52:0/589544660 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x560d426b2500 0x560d426d1a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:25:47.491 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:47.488+0000 7f02aaffd700 1 -- 172.21.15.52:0/589544660 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({osdmap=0}) -- 0x560d426d65b0 con 0x560d42533df0 2024-05-27T07:25:47.491 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:47.488+0000 7f02ab7fe700 1 --2- 172.21.15.52:0/589544660 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x560d426bcd90 0x560d426d1f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-27T07:25:47.491 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:47.488+0000 7f02aaffd700 1 --2- 172.21.15.52:0/589544660 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x560d42533df0 0x560d426d14f0 secure :-1 s=READY pgs=387 cs=0 l=1 rev1=1 crypto rx=0x7f029c004e70 tx=0x7f029c0a87d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f5175c78cd167384 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:25:47.491 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:47.488+0000 7f028affd700 10 client.0 ms_handle_connect on v2:172.21.15.52:3300/0 2024-05-27T07:25:47.491 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:47.488+0000 7f028affd700 1 -- 172.21.15.52:0/589544660 <== mon.0 v2:172.21.15.52:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f029c0c7070 con 0x560d42533df0 2024-05-27T07:25:47.492 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:47.488+0000 7f028affd700 1 -- 172.21.15.52:0/589544660 <== mon.0 v2:172.21.15.52:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9121+0+0 (secure 0 0 0) 0x7f029c015400 con 0x560d42533df0 2024-05-27T07:25:47.492 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:47.488+0000 7f02c0415b80 1 -- 172.21.15.52:0/589544660 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f02a40058e0 con 0x560d42533df0 2024-05-27T07:25:47.492 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:47.488+0000 7f028affd700 1 -- 172.21.15.52:0/589544660 <== mon.0 v2:172.21.15.52:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f029c005e60 con 0x560d42533df0 2024-05-27T07:25:47.492 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:47.488+0000 7f02c0415b80 10 client.15309 fetch_fsmap learned FSMap version 29 2024-05-27T07:25:47.492 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:47.488+0000 7f02c0415b80 1 -- 172.21.15.52:0/589544660 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x560d42737f60 con 0x560d42533df0 2024-05-27T07:25:47.492 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:47.488+0000 7f028affd700 1 -- 172.21.15.52:0/589544660 <== mon.0 v2:172.21.15.52:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f02a40058e0 con 0x560d42533df0 2024-05-27T07:25:47.493 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:47.488+0000 7f028affd700 1 -- 172.21.15.52:0/589544660 <== mon.0 v2:172.21.15.52:3300/0 5 ==== fsmap(e 29) ==== 8362+0+0 (secure 0 0 0) 0x7f029c0b1980 con 0x560d42533df0 2024-05-27T07:25:47.493 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:47.488+0000 7f02c0415b80 10 client.15309 fetch_fsmap finished waiting for FSMap version 29 2024-05-27T07:25:47.493 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:47.489+0000 7f02c0415b80 10 client.15309 resolve_mds: resolved name 'b' to daemon mds.b 2024-05-27T07:25:47.493 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:47.489+0000 7f02c0415b80 20 client.15309 populate_metadata read hostname 'smithi052' 2024-05-27T07:25:47.493 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:47.489+0000 7f02c0415b80 1 --2- 172.21.15.52:0/589544660 >> [v2:172.21.15.105:6838/500348151,v1:172.21.15.105:6839/500348151] conn(0x560d4273a170 0x560d4273b5e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:25:47.493 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:47.489+0000 7f02c0415b80 4 client.15309 mds_command: new command op to 24527 tid=0 multi_id=0 [{"prefix": "status"}] 2024-05-27T07:25:47.494 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:47.489+0000 7f02c0415b80 1 -- 172.21.15.52:0/589544660 --> [v2:172.21.15.105:6838/500348151,v1:172.21.15.105:6839/500348151] -- command(tid 0: {"prefix": "status"}) -- 0x560d425dc450 con 0x560d4273a170 2024-05-27T07:25:47.494 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:47.489+0000 7f02aa7fc700 1 --2- 172.21.15.52:0/589544660 >> [v2:172.21.15.105:6838/500348151,v1:172.21.15.105:6839/500348151] conn(0x560d4273a170 0x560d4273b5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:25:47.494 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:47.489+0000 7f028affd700 10 client.15309 ms_handle_connect on v2:172.21.15.105:6838/500348151 2024-05-27T07:25:47.494 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:47.489+0000 7f02aa7fc700 1 --2- 172.21.15.52:0/589544660 >> [v2:172.21.15.105:6838/500348151,v1:172.21.15.105:6839/500348151] conn(0x560d4273a170 0x560d4273b5e0 crc :-1 s=READY pgs=31 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=9104594824f4071b server_cookie=32a22b8e1f2af848 in_seq=0 out_seq=0 2024-05-27T07:25:47.494 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:47.489+0000 7f028affd700 1 -- 172.21.15.52:0/589544660 <== mds.0 v2:172.21.15.105:6838/500348151 1 ==== command_reply(tid 0: 0 ) ==== 8+0+327 (crc 0 0 0) 0x560d425dc450 con 0x560d4273a170 2024-05-27T07:25:47.494 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:47.489+0000 7f028affd700 10 client.15309 handle_command_reply: tid=0 2024-05-27T07:25:47.494 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:47.489+0000 7f02c0415b80 1 client.15309 shutdown 2024-05-27T07:25:47.495 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:47.490+0000 7f02c0415b80 1 -- 172.21.15.52:0/589544660 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x560d42533df0 msgr2=0x560d426d14f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:25:47.495 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:47.490+0000 7f02c0415b80 1 --2- 172.21.15.52:0/589544660 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x560d42533df0 0x560d426d14f0 secure :-1 s=READY pgs=387 cs=0 l=1 rev1=1 crypto rx=0x7f029c004e70 tx=0x7f029c0a87d0 comp rx=0 tx=0).stop 2024-05-27T07:25:47.495 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:47.490+0000 7f02c0415b80 1 -- 172.21.15.52:0/589544660 shutdown_connections 2024-05-27T07:25:47.495 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:47.490+0000 7f02c0415b80 1 --2- 172.21.15.52:0/589544660 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x560d426bcd90 0x560d426d1f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:25:47.495 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:47.490+0000 7f02c0415b80 1 --2- 172.21.15.52:0/589544660 >> [v2:172.21.15.105:6838/500348151,v1:172.21.15.105:6839/500348151] conn(0x560d4273a170 0x560d4273b5e0 crc :-1 s=READY pgs=31 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:25:47.495 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:47.490+0000 7f02c0415b80 1 --2- 172.21.15.52:0/589544660 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x560d42533df0 0x560d426d14f0 unknown :-1 s=CLOSED pgs=387 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:25:47.496 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:47.490+0000 7f02c0415b80 1 --2- 172.21.15.52:0/589544660 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x560d426b2500 0x560d426d1a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:25:47.496 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:47.490+0000 7f02c0415b80 1 -- 172.21.15.52:0/589544660 >> 172.21.15.52:0/589544660 conn(0x560d42536720 msgr2=0x560d426b9d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:25:47.496 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:47.490+0000 7f028affd700 0 client.15309 ms_handle_reset on v2:172.21.15.105:6838/500348151 2024-05-27T07:25:47.496 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:47.490+0000 7f02c0415b80 1 -- 172.21.15.52:0/589544660 shutdown_connections 2024-05-27T07:25:47.496 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:47.490+0000 7f02c0415b80 1 -- 172.21.15.52:0/589544660 wait complete. 2024-05-27T07:25:47.496 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:47.490+0000 7f02c0415b80 20 client.15309 trim_cache size 0 max 16384 2024-05-27T07:25:47.497 INFO:teuthology.orchestra.run.smithi052.stdout:{ 2024-05-27T07:25:47.497 INFO:teuthology.orchestra.run.smithi052.stdout: "cluster_fsid": "73df2636-1bf8-11ef-bc9b-c7b262605968", 2024-05-27T07:25:47.497 INFO:teuthology.orchestra.run.smithi052.stdout: "whoami": 0, 2024-05-27T07:25:47.497 INFO:teuthology.orchestra.run.smithi052.stdout: "id": 24527, 2024-05-27T07:25:47.498 INFO:teuthology.orchestra.run.smithi052.stdout: "want_state": "up:active", 2024-05-27T07:25:47.498 INFO:teuthology.orchestra.run.smithi052.stdout: "state": "up:active", 2024-05-27T07:25:47.498 INFO:teuthology.orchestra.run.smithi052.stdout: "fs_name": "cephfs", 2024-05-27T07:25:47.498 INFO:teuthology.orchestra.run.smithi052.stdout: "rank_uptime": 236.916350728, 2024-05-27T07:25:47.498 INFO:teuthology.orchestra.run.smithi052.stdout: "mdsmap_epoch": 29, 2024-05-27T07:25:47.498 INFO:teuthology.orchestra.run.smithi052.stdout: "osdmap_epoch": 90, 2024-05-27T07:25:47.499 INFO:teuthology.orchestra.run.smithi052.stdout: "osdmap_epoch_barrier": 88, 2024-05-27T07:25:47.499 INFO:teuthology.orchestra.run.smithi052.stdout: "uptime": 250.97052829899999 2024-05-27T07:25:47.499 INFO:teuthology.orchestra.run.smithi052.stdout:} 2024-05-27T07:25:47.500 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:47.491+0000 7f02b1a00700 1 -- 172.21.15.52:0/3647081071 >> [v2:172.21.15.52:6800/3513075805,v1:172.21.15.52:6801/3513075805] conn(0x7f02940b2900 msgr2=0x7f02940b4e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:25:47.500 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:47.491+0000 7f02b1a00700 1 --2- 172.21.15.52:0/3647081071 >> [v2:172.21.15.52:6800/3513075805,v1:172.21.15.52:6801/3513075805] conn(0x7f02940b2900 0x7f02940b4e10 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7f02ac10c130 tx=0x7f02a0009260 comp rx=0 tx=0).stop 2024-05-27T07:25:47.500 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:47.491+0000 7f02b1a00700 1 -- 172.21.15.52:0/3647081071 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f02ac104430 msgr2=0x7f02ac1a1fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:25:47.500 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:47.491+0000 7f02b1a00700 1 --2- 172.21.15.52:0/3647081071 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f02ac104430 0x7f02ac1a1fc0 secure :-1 s=READY pgs=383 cs=0 l=1 rev1=1 crypto rx=0x7f029c0060b0 tx=0x7f029c00d3b0 comp rx=0 tx=0).stop 2024-05-27T07:25:47.500 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:47.491+0000 7f02b1a00700 1 -- 172.21.15.52:0/3647081071 shutdown_connections 2024-05-27T07:25:47.501 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:47.491+0000 7f02b1a00700 1 --2- 172.21.15.52:0/3647081071 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f02ac1096b0 0x7f02ac1ad040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:25:47.501 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:47.491+0000 7f02b1a00700 1 --2- 172.21.15.52:0/3647081071 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f02ac104430 0x7f02ac1a1fc0 unknown :-1 s=CLOSED pgs=383 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:25:47.501 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:47.491+0000 7f02b1a00700 1 --2- 172.21.15.52:0/3647081071 >> [v2:172.21.15.52:6800/3513075805,v1:172.21.15.52:6801/3513075805] conn(0x7f02940b2900 0x7f02940b4e10 unknown :-1 s=CLOSED pgs=226 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:25:47.501 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:47.491+0000 7f02b1a00700 1 --2- 172.21.15.52:0/3647081071 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f02ac108d60 0x7f02ac1a2500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:25:47.501 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:47.491+0000 7f02b1a00700 1 -- 172.21.15.52:0/3647081071 >> 172.21.15.52:0/3647081071 conn(0x7f02ac079cc0 msgr2=0x7f02ac1013a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:25:47.501 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:47.491+0000 7f02b1a00700 1 -- 172.21.15.52:0/3647081071 shutdown_connections 2024-05-27T07:25:47.501 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:47.491+0000 7f02b1a00700 1 -- 172.21.15.52:0/3647081071 wait complete. 2024-05-27T07:25:48.270 INFO:teuthology.run_tasks:Running task sequential... 2024-05-27T07:25:48.280 INFO:teuthology.task.sequential:In sequential, running task sequential... 2024-05-27T07:25:48.280 INFO:teuthology.task.sequential:In sequential, running task print... 2024-05-27T07:25:48.280 INFO:teuthology.task.print:Enabling mgr modules 2024-05-27T07:25:48.280 INFO:teuthology.task.sequential:In sequential, running task exec... 2024-05-27T07:25:48.281 INFO:teuthology.task.exec:Executing custom commands... 2024-05-27T07:25:48.281 INFO:teuthology.task.exec:Running commands on role mon.a host ubuntu@smithi052.front.sepia.ceph.com 2024-05-27T07:25:48.282 DEBUG:teuthology.orchestra.run.smithi052:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'ceph mgr module enable snap_schedule' 2024-05-27T07:25:48.406 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:48.413+0000 7f5089e61640 1 -- 172.21.15.52:0/200013149 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f50841043e0 msgr2=0x7f5084106800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:25:48.406 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:48.413+0000 7f5089e61640 1 --2- 172.21.15.52:0/200013149 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f50841043e0 0x7f5084106800 secure :-1 s=READY pgs=388 cs=0 l=1 rev1=1 crypto rx=0x7f5074009900 tx=0x7f507402a410 comp rx=0 tx=0).stop 2024-05-27T07:25:48.406 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:48.413+0000 7f5089e61640 1 -- 172.21.15.52:0/200013149 shutdown_connections 2024-05-27T07:25:48.406 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:48.413+0000 7f5089e61640 1 --2- 172.21.15.52:0/200013149 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f50841098e0 0x7f508410bdf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:25:48.406 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:48.413+0000 7f5089e61640 1 --2- 172.21.15.52:0/200013149 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f5084106dd0 0x7f5084109250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:25:48.407 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:48.413+0000 7f5089e61640 1 --2- 172.21.15.52:0/200013149 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f50841043e0 0x7f5084106800 unknown :-1 s=CLOSED pgs=388 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:25:48.407 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:48.413+0000 7f5089e61640 1 -- 172.21.15.52:0/200013149 >> 172.21.15.52:0/200013149 conn(0x7f50840fdd40 msgr2=0x7f50841001f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:25:48.407 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:48.413+0000 7f5089e61640 1 -- 172.21.15.52:0/200013149 shutdown_connections 2024-05-27T07:25:48.407 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:48.413+0000 7f5089e61640 1 -- 172.21.15.52:0/200013149 wait complete. 2024-05-27T07:25:48.407 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:48.414+0000 7f5089e61640 1 Processor -- start 2024-05-27T07:25:48.407 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:48.414+0000 7f5089e61640 1 -- start start 2024-05-27T07:25:48.407 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:48.414+0000 7f5089e61640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f50841043e0 0x7f508419a380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:25:48.407 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:48.414+0000 7f5089e61640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f5084106dd0 0x7f508419a8c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:25:48.407 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:48.414+0000 7f5089e61640 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f50841098e0 0x7f50841a1850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:25:48.407 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:48.414+0000 7f5089e61640 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x7f508410f750 con 0x7f50841098e0 2024-05-27T07:25:48.407 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:48.414+0000 7f5089e61640 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f508410f5d0 con 0x7f50841043e0 2024-05-27T07:25:48.407 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:48.414+0000 7f5089e61640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f508410f8d0 con 0x7f5084106dd0 2024-05-27T07:25:48.408 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:48.414+0000 7f5083fff640 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f50841098e0 0x7f50841a1850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:25:48.408 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:48.414+0000 7f5083fff640 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f50841098e0 0x7f50841a1850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.52:48284/0 (socket says 172.21.15.52:48284) 2024-05-27T07:25:48.408 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:48.414+0000 7f5083fff640 1 -- 172.21.15.52:0/820720034 learned_addr learned my addr 172.21.15.52:0/820720034 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-05-27T07:25:48.408 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:48.414+0000 7f507bfff640 1 --2- 172.21.15.52:0/820720034 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f5084106dd0 0x7f508419a8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:25:48.408 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:48.414+0000 7f50837fe640 1 --2- 172.21.15.52:0/820720034 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f50841043e0 0x7f508419a380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:25:48.408 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:48.415+0000 7f5083fff640 1 -- 172.21.15.52:0/820720034 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f5084106dd0 msgr2=0x7f508419a8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:25:48.408 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:48.415+0000 7f5083fff640 1 --2- 172.21.15.52:0/820720034 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f5084106dd0 0x7f508419a8c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:25:48.408 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:48.415+0000 7f5083fff640 1 -- 172.21.15.52:0/820720034 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f50841043e0 msgr2=0x7f508419a380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:25:48.408 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:48.415+0000 7f5083fff640 1 --2- 172.21.15.52:0/820720034 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f50841043e0 0x7f508419a380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:25:48.408 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:48.415+0000 7f5083fff640 1 -- 172.21.15.52:0/820720034 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f50841a1ec0 con 0x7f50841098e0 2024-05-27T07:25:48.408 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:48.415+0000 7f507bfff640 1 --2- 172.21.15.52:0/820720034 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f5084106dd0 0x7f508419a8c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-27T07:25:48.408 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:48.415+0000 7f5083fff640 1 --2- 172.21.15.52:0/820720034 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f50841098e0 0x7f50841a1850 secure :-1 s=READY pgs=389 cs=0 l=1 rev1=1 crypto rx=0x7f507c00eaa0 tx=0x7f507c00ef60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:25:48.408 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:48.415+0000 7f50817fa640 1 -- 172.21.15.52:0/820720034 <== mon.0 v2:172.21.15.52:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f507c00cdb0 con 0x7f50841098e0 2024-05-27T07:25:48.408 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:48.415+0000 7f50817fa640 1 -- 172.21.15.52:0/820720034 <== mon.0 v2:172.21.15.52:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f507c005020 con 0x7f50841098e0 2024-05-27T07:25:48.409 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:48.415+0000 7f5089e61640 1 -- 172.21.15.52:0/820720034 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f50841a21b0 con 0x7f50841098e0 2024-05-27T07:25:48.409 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:48.415+0000 7f50817fa640 1 -- 172.21.15.52:0/820720034 <== mon.0 v2:172.21.15.52:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f507c0116d0 con 0x7f50841098e0 2024-05-27T07:25:48.409 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:48.415+0000 7f5089e61640 1 -- 172.21.15.52:0/820720034 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f50841a26b0 con 0x7f50841098e0 2024-05-27T07:25:48.409 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:48.416+0000 7f5089e61640 1 -- 172.21.15.52:0/820720034 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f50840ff860 con 0x7f50841098e0 2024-05-27T07:25:48.410 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:48.418+0000 7f50817fa640 1 -- 172.21.15.52:0/820720034 <== mon.0 v2:172.21.15.52:3300/0 4 ==== mgrmap(e 18) ==== 149652+0+0 (secure 0 0 0) 0x7f507c0051d0 con 0x7f50841098e0 2024-05-27T07:25:48.410 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:48.418+0000 7f50817fa640 1 --2- 172.21.15.52:0/820720034 >> [v2:172.21.15.52:6800/3513075805,v1:172.21.15.52:6801/3513075805] conn(0x7f50700b27e0 0x7f50700b4cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:25:48.410 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:48.418+0000 7f50837fe640 1 --2- 172.21.15.52:0/820720034 >> [v2:172.21.15.52:6800/3513075805,v1:172.21.15.52:6801/3513075805] conn(0x7f50700b27e0 0x7f50700b4cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:25:48.410 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:48.419+0000 7f50817fa640 1 -- 172.21.15.52:0/820720034 <== mon.0 v2:172.21.15.52:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9121+0+0 (secure 0 0 0) 0x7f507c0e2bd0 con 0x7f50841098e0 2024-05-27T07:25:48.410 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:48.419+0000 7f50837fe640 1 --2- 172.21.15.52:0/820720034 >> [v2:172.21.15.52:6800/3513075805,v1:172.21.15.52:6801/3513075805] conn(0x7f50700b27e0 0x7f50700b4cf0 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7f5074000c00 tx=0x7f5074031400 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:25:48.412 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:48.422+0000 7f50817fa640 1 -- 172.21.15.52:0/820720034 <== mon.0 v2:172.21.15.52:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191963 (secure 0 0 0) 0x7f507c016030 con 0x7f50841098e0 2024-05-27T07:25:48.659 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:48.669+0000 7f5089e61640 1 -- 172.21.15.52:0/820720034 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_command({"prefix": "mgr module enable", "module": "snap_schedule"} v 0) -- 0x7f50840638c0 con 0x7f50841098e0 2024-05-27T07:25:49.107 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:49.116+0000 7f50817fa640 1 -- 172.21.15.52:0/820720034 <== mon.0 v2:172.21.15.52:3300/0 7 ==== mon_command_ack([{"prefix": "mgr module enable", "module": "snap_schedule"}]=0 v19) ==== 92+0+0 (secure 0 0 0) 0x7f507c0aa570 con 0x7f50841098e0 2024-05-27T07:25:49.110 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:49.119+0000 7f5089e61640 1 -- 172.21.15.52:0/820720034 >> [v2:172.21.15.52:6800/3513075805,v1:172.21.15.52:6801/3513075805] conn(0x7f50700b27e0 msgr2=0x7f50700b4cf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:25:49.110 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:49.119+0000 7f5089e61640 1 --2- 172.21.15.52:0/820720034 >> [v2:172.21.15.52:6800/3513075805,v1:172.21.15.52:6801/3513075805] conn(0x7f50700b27e0 0x7f50700b4cf0 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7f5074000c00 tx=0x7f5074031400 comp rx=0 tx=0).stop 2024-05-27T07:25:49.110 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:49.119+0000 7f5089e61640 1 -- 172.21.15.52:0/820720034 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f50841098e0 msgr2=0x7f50841a1850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:25:49.110 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:49.119+0000 7f5089e61640 1 --2- 172.21.15.52:0/820720034 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f50841098e0 0x7f50841a1850 secure :-1 s=READY pgs=389 cs=0 l=1 rev1=1 crypto rx=0x7f507c00eaa0 tx=0x7f507c00ef60 comp rx=0 tx=0).stop 2024-05-27T07:25:49.111 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:49.119+0000 7f5089e61640 1 -- 172.21.15.52:0/820720034 shutdown_connections 2024-05-27T07:25:49.111 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:49.119+0000 7f5089e61640 1 --2- 172.21.15.52:0/820720034 >> [v2:172.21.15.52:6800/3513075805,v1:172.21.15.52:6801/3513075805] conn(0x7f50700b27e0 0x7f50700b4cf0 unknown :-1 s=CLOSED pgs=227 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:25:49.111 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:49.119+0000 7f5089e61640 1 --2- 172.21.15.52:0/820720034 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f50841098e0 0x7f50841a1850 unknown :-1 s=CLOSED pgs=389 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:25:49.111 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:49.119+0000 7f5089e61640 1 --2- 172.21.15.52:0/820720034 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f5084106dd0 0x7f508419a8c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:25:49.111 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:49.119+0000 7f5089e61640 1 --2- 172.21.15.52:0/820720034 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f50841043e0 0x7f508419a380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:25:49.111 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:49.119+0000 7f5089e61640 1 -- 172.21.15.52:0/820720034 >> 172.21.15.52:0/820720034 conn(0x7f50840fdd40 msgr2=0x7f5084105010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:25:49.111 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:49.119+0000 7f5089e61640 1 -- 172.21.15.52:0/820720034 shutdown_connections 2024-05-27T07:25:49.111 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:49.119+0000 7f5089e61640 1 -- 172.21.15.52:0/820720034 wait complete. 2024-05-27T07:25:49.129 INFO:teuthology.run_tasks:Running task kclient... 2024-05-27T07:25:49.150 INFO:tasks.kclient:Mounting kernel clients... 2024-05-27T07:25:49.151 INFO:tasks.kclient:config is {'client.0': {}} 2024-05-27T07:25:49.151 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-27T07:25:49.151 DEBUG:teuthology.orchestra.run.smithi052:> ip netns list 2024-05-27T07:25:49.171 INFO:teuthology.orchestra.run.smithi052.stdout:ceph-ns--home-ubuntu-cephtest-mnt.admin (id: 0) 2024-05-27T07:25:49.171 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-27T07:25:49.172 DEBUG:teuthology.orchestra.run.smithi052:> sudo ip netns delete ceph-ns--home-ubuntu-cephtest-mnt.admin 2024-05-27T07:25:49.239 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-27T07:25:49.240 DEBUG:teuthology.orchestra.run.smithi052:> sudo ip link delete ceph-brx 2024-05-27T07:25:49.352 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:25:49 smithi173 ceph-mon[37000]: pgmap v419: 129 pgs: 129 active+clean; 647 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-05-27T07:25:49.352 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:25:49 smithi173 ceph-mon[37000]: from='client.? 172.21.15.52:0/820720034' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "snap_schedule"}]: dispatch 2024-05-27T07:25:49.353 INFO:journalctl@ceph.mgr.z.smithi173.stdout:May 27 07:25:49 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mgr-z[38891]: ignoring --setuser ceph since I am not root 2024-05-27T07:25:49.353 INFO:journalctl@ceph.mgr.z.smithi173.stdout:May 27 07:25:49 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mgr-z[38891]: ignoring --setgroup ceph since I am not root 2024-05-27T07:25:49.353 INFO:journalctl@ceph.mgr.z.smithi173.stdout:May 27 07:25:49 smithi173 ceph-mgr[38946]: -- 172.21.15.173:0/3507063430 <== mon.2 v2:172.21.15.105:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x564c1893a5a0 con 0x564c1d6c4480 2024-05-27T07:25:49.354 INFO:journalctl@ceph.mgr.z.smithi173.stdout:May 27 07:25:49 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mgr-z[38891]: 2024-05-27T07:25:49.324+0000 7f41c267f200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-05-27T07:25:49.356 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:25:49 smithi052 ceph-mon[31955]: pgmap v419: 129 pgs: 129 active+clean; 647 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-05-27T07:25:49.356 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:25:49 smithi052 ceph-mon[31955]: from='client.? 172.21.15.52:0/820720034' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "snap_schedule"}]: dispatch 2024-05-27T07:25:49.357 INFO:journalctl@ceph.mgr.x.smithi052.stdout:May 27 07:25:49 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mgr-x[32184]: ignoring --setuser ceph since I am not root 2024-05-27T07:25:49.357 INFO:journalctl@ceph.mgr.x.smithi052.stdout:May 27 07:25:49 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mgr-x[32184]: ignoring --setgroup ceph since I am not root 2024-05-27T07:25:49.357 INFO:journalctl@ceph.mgr.x.smithi052.stdout:May 27 07:25:49 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mgr-x[32184]: 2024-05-27T07:25:49.349+0000 7fd16c7b8200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-05-27T07:25:49.358 INFO:tasks.kclient:client.0 config is {'mntopts': ['ms_mode=legacy', 'wsync'], 'syntax': 'v1', 'mount_subvol_num': 1} 2024-05-27T07:25:49.359 INFO:tasks.cephfs.mount:cephfs_mntpt = None 2024-05-27T07:25:49.359 INFO:tasks.cephfs.mount:hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2024-05-27T07:25:49.359 DEBUG:teuthology.orchestra.run.smithi052:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:38f9d900c83c63799fbdbe61acc9a11b0d3554a6 shell --fsid 73df2636-1bf8-11ef-bc9b-c7b262605968 -- ceph fs ls 2024-05-27T07:25:49.440 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:25:49 smithi105 ceph-mon[35977]: pgmap v419: 129 pgs: 129 active+clean; 647 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-05-27T07:25:49.440 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:25:49 smithi105 ceph-mon[35977]: from='client.? 172.21.15.52:0/820720034' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "snap_schedule"}]: dispatch 2024-05-27T07:25:49.440 INFO:journalctl@ceph.mgr.y.smithi105.stdout:May 27 07:25:49 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mgr-y[37393]: ignoring --setuser ceph since I am not root 2024-05-27T07:25:49.440 INFO:journalctl@ceph.mgr.y.smithi105.stdout:May 27 07:25:49 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mgr-y[37393]: ignoring --setgroup ceph since I am not root 2024-05-27T07:25:49.440 INFO:journalctl@ceph.mgr.y.smithi105.stdout:May 27 07:25:49 smithi105 ceph-mgr[37397]: -- 172.21.15.105:0/4119786686 <== mon.2 v2:172.21.15.105:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55ccf16925a0 con 0x55ccf641c000 2024-05-27T07:25:49.440 INFO:journalctl@ceph.mgr.y.smithi105.stdout:May 27 07:25:49 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mgr-y[37393]: 2024-05-27T07:25:49.318+0000 7f1202724200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-05-27T07:25:49.622 INFO:teuthology.orchestra.run.smithi052.stderr:Inferring config /var/lib/ceph/73df2636-1bf8-11ef-bc9b-c7b262605968/mon.a/config 2024-05-27T07:25:49.670 INFO:journalctl@ceph.mgr.x.smithi052.stdout:May 27 07:25:49 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mgr-x[32184]: 2024-05-27T07:25:49.554+0000 7fd16c7b8200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-05-27T07:25:49.671 INFO:journalctl@ceph.mgr.x.smithi052.stdout:May 27 07:25:49 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mgr-x[32184]: 2024-05-27T07:25:49.680+0000 7fd16c7b8200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-05-27T07:25:49.852 INFO:journalctl@ceph.mgr.z.smithi173.stdout:May 27 07:25:49 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mgr-z[38891]: 2024-05-27T07:25:49.512+0000 7f41c267f200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-05-27T07:25:49.852 INFO:journalctl@ceph.mgr.z.smithi173.stdout:May 27 07:25:49 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mgr-z[38891]: 2024-05-27T07:25:49.614+0000 7f41c267f200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-05-27T07:25:49.904 INFO:journalctl@ceph.mgr.y.smithi105.stdout:May 27 07:25:49 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mgr-y[37393]: 2024-05-27T07:25:49.504+0000 7f1202724200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-05-27T07:25:49.905 INFO:journalctl@ceph.mgr.y.smithi105.stdout:May 27 07:25:49 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mgr-y[37393]: 2024-05-27T07:25:49.606+0000 7f1202724200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-05-27T07:25:50.189 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:25:50 smithi105 ceph-mon[35977]: from='client.? 172.21.15.52:0/820720034' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "snap_schedule"}]': finished 2024-05-27T07:25:50.190 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:25:50 smithi105 ceph-mon[35977]: mgrmap e19: x(active, since 11m), standbys: z, y 2024-05-27T07:25:50.190 INFO:journalctl@ceph.mgr.y.smithi105.stdout:May 27 07:25:49 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mgr-y[37393]: 2024-05-27T07:25:49.914+0000 7f1202724200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-05-27T07:25:50.190 INFO:journalctl@ceph.mgr.y.smithi105.stdout:May 27 07:25:50 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mgr-y[37393]: 2024-05-27T07:25:50.002+0000 7f1202724200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-05-27T07:25:50.352 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:25:50 smithi173 ceph-mon[37000]: from='client.? 172.21.15.52:0/820720034' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "snap_schedule"}]': finished 2024-05-27T07:25:50.352 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:25:50 smithi173 ceph-mon[37000]: mgrmap e19: x(active, since 11m), standbys: z, y 2024-05-27T07:25:50.352 INFO:journalctl@ceph.mgr.z.smithi173.stdout:May 27 07:25:49 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mgr-z[38891]: 2024-05-27T07:25:49.921+0000 7f41c267f200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-05-27T07:25:50.352 INFO:journalctl@ceph.mgr.z.smithi173.stdout:May 27 07:25:50 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mgr-z[38891]: 2024-05-27T07:25:50.008+0000 7f41c267f200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-05-27T07:25:50.463 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:25:50 smithi052 ceph-mon[31955]: from='client.? 172.21.15.52:0/820720034' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "snap_schedule"}]': finished 2024-05-27T07:25:50.463 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:25:50 smithi052 ceph-mon[31955]: mgrmap e19: x(active, since 11m), standbys: z, y 2024-05-27T07:25:50.463 INFO:journalctl@ceph.mgr.x.smithi052.stdout:May 27 07:25:50 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mgr-x[32184]: 2024-05-27T07:25:50.012+0000 7fd16c7b8200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-05-27T07:25:50.463 INFO:journalctl@ceph.mgr.x.smithi052.stdout:May 27 07:25:50 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mgr-x[32184]: 2024-05-27T07:25:50.111+0000 7fd16c7b8200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-05-27T07:25:51.436 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:51.445+0000 7fc9201bd700 1 -- 172.21.15.52:0/1081737123 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fc9181034b0 msgr2=0x7fc9181038c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:25:51.436 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:51.445+0000 7fc9201bd700 1 --2- 172.21.15.52:0/1081737123 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fc9181034b0 0x7fc9181038c0 secure :-1 s=READY pgs=392 cs=0 l=1 rev1=1 crypto rx=0x7fc9140099c0 tx=0x7fc914009cd0 comp rx=0 tx=0).stop 2024-05-27T07:25:51.436 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:51.445+0000 7fc9201bd700 1 -- 172.21.15.52:0/1081737123 shutdown_connections 2024-05-27T07:25:51.436 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:51.445+0000 7fc9201bd700 1 --2- 172.21.15.52:0/1081737123 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fc918108730 0x7fc918110090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:25:51.437 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:51.445+0000 7fc9201bd700 1 --2- 172.21.15.52:0/1081737123 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fc9181034b0 0x7fc9181038c0 unknown :-1 s=CLOSED pgs=392 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:25:51.437 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:51.445+0000 7fc9201bd700 1 --2- 172.21.15.52:0/1081737123 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fc918107de0 0x7fc9181081f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:25:51.437 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:51.445+0000 7fc9201bd700 1 -- 172.21.15.52:0/1081737123 >> 172.21.15.52:0/1081737123 conn(0x7fc9180fedb0 msgr2=0x7fc9181011d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:25:51.438 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:51.446+0000 7fc9201bd700 1 -- 172.21.15.52:0/1081737123 shutdown_connections 2024-05-27T07:25:51.438 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:51.446+0000 7fc9201bd700 1 -- 172.21.15.52:0/1081737123 wait complete. 2024-05-27T07:25:51.439 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:51.447+0000 7fc9201bd700 1 Processor -- start 2024-05-27T07:25:51.439 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:51.447+0000 7fc9201bd700 1 -- start start 2024-05-27T07:25:51.439 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:51.447+0000 7fc9201bd700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fc9181034b0 0x7fc9180ff290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:25:51.439 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:51.447+0000 7fc9201bd700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fc918107de0 0x7fc9180ff7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:25:51.439 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:51.447+0000 7fc9201bd700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fc918108730 0x7fc9180ffd10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:25:51.440 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:51.447+0000 7fc9201bd700 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x7fc918113360 con 0x7fc9181034b0 2024-05-27T07:25:51.440 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:51.447+0000 7fc9201bd700 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7fc9181131e0 con 0x7fc918108730 2024-05-27T07:25:51.440 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:51.447+0000 7fc9201bd700 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7fc9181134e0 con 0x7fc918107de0 2024-05-27T07:25:51.440 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:51.447+0000 7fc91df59700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fc9181034b0 0x7fc9180ff290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:25:51.440 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:51.447+0000 7fc91df59700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fc9181034b0 0x7fc9180ff290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.52:59208/0 (socket says 172.21.15.52:59208) 2024-05-27T07:25:51.440 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:51.447+0000 7fc91df59700 1 -- 172.21.15.52:0/44374434 learned_addr learned my addr 172.21.15.52:0/44374434 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-05-27T07:25:51.440 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:51.447+0000 7fc91d758700 1 --2- 172.21.15.52:0/44374434 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fc918107de0 0x7fc9180ff7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:25:51.440 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:51.447+0000 7fc91e75a700 1 --2- 172.21.15.52:0/44374434 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fc918108730 0x7fc9180ffd10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:25:51.440 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:51.448+0000 7fc91df59700 1 -- 172.21.15.52:0/44374434 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fc918107de0 msgr2=0x7fc9180ff7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:25:51.440 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:51.448+0000 7fc91df59700 1 --2- 172.21.15.52:0/44374434 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fc918107de0 0x7fc9180ff7d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:25:51.441 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:51.448+0000 7fc91df59700 1 -- 172.21.15.52:0/44374434 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fc918108730 msgr2=0x7fc9180ffd10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:25:51.441 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:51.448+0000 7fc91df59700 1 --2- 172.21.15.52:0/44374434 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fc918108730 0x7fc9180ffd10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:25:51.441 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:51.448+0000 7fc91df59700 1 -- 172.21.15.52:0/44374434 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc9181a9180 con 0x7fc9181034b0 2024-05-27T07:25:51.441 INFO:journalctl@ceph.mgr.x.smithi052.stdout:May 27 07:25:51 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mgr-x[32184]: 2024-05-27T07:25:51.415+0000 7fd16c7b8200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-05-27T07:25:51.442 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:51.448+0000 7fc91df59700 1 --2- 172.21.15.52:0/44374434 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fc9181034b0 0x7fc9180ff290 secure :-1 s=READY pgs=393 cs=0 l=1 rev1=1 crypto rx=0x7fc9140098e0 tx=0x7fc9140057c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:25:51.442 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:51.448+0000 7fc90effd700 1 -- 172.21.15.52:0/44374434 <== mon.0 v2:172.21.15.52:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc914027070 con 0x7fc9181034b0 2024-05-27T07:25:51.442 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:51.448+0000 7fc90effd700 1 -- 172.21.15.52:0/44374434 <== mon.0 v2:172.21.15.52:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fc914003f20 con 0x7fc9181034b0 2024-05-27T07:25:51.442 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:51.448+0000 7fc90effd700 1 -- 172.21.15.52:0/44374434 <== mon.0 v2:172.21.15.52:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc914022ca0 con 0x7fc9181034b0 2024-05-27T07:25:51.442 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:51.448+0000 7fc9201bd700 1 -- 172.21.15.52:0/44374434 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc9181a9410 con 0x7fc9181034b0 2024-05-27T07:25:51.443 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:51.449+0000 7fc9201bd700 1 -- 172.21.15.52:0/44374434 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc9181a9860 con 0x7fc9181034b0 2024-05-27T07:25:51.443 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:51.450+0000 7fc9201bd700 1 -- 172.21.15.52:0/44374434 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc9180512b0 con 0x7fc9181034b0 2024-05-27T07:25:51.443 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:51.451+0000 7fc90effd700 1 -- 172.21.15.52:0/44374434 <== mon.0 v2:172.21.15.52:3300/0 4 ==== mgrmap(e 19) ==== 149669+0+0 (secure 0 0 0) 0x7fc91402a070 con 0x7fc9181034b0 2024-05-27T07:25:51.443 INFO:journalctl@ceph.mgr.y.smithi105.stdout:May 27 07:25:51 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mgr-y[37393]: 2024-05-27T07:25:51.188+0000 7f1202724200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-05-27T07:25:51.443 INFO:journalctl@ceph.mgr.y.smithi105.stdout:May 27 07:25:51 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mgr-y[37393]: 2024-05-27T07:25:51.358+0000 7f1202724200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-05-27T07:25:51.444 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:51.452+0000 7fc90effd700 1 --2- 172.21.15.52:0/44374434 >> [v2:172.21.15.52:6800/3513075805,v1:172.21.15.52:6801/3513075805] conn(0x7fc9040b28d0 0x7fc9040b4de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:25:51.444 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:51.452+0000 7fc91d758700 1 -- 172.21.15.52:0/44374434 >> [v2:172.21.15.52:6800/3513075805,v1:172.21.15.52:6801/3513075805] conn(0x7fc9040b28d0 msgr2=0x7fc9040b4de0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.52:6800/3513075805 2024-05-27T07:25:51.444 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:51.452+0000 7fc90effd700 1 -- 172.21.15.52:0/44374434 <== mon.0 v2:172.21.15.52:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9121+0+0 (secure 0 0 0) 0x7fc9140e34c0 con 0x7fc9181034b0 2024-05-27T07:25:51.444 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:51.452+0000 7fc91d758700 1 --2- 172.21.15.52:0/44374434 >> [v2:172.21.15.52:6800/3513075805,v1:172.21.15.52:6801/3513075805] conn(0x7fc9040b28d0 0x7fc9040b4de0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-05-27T07:25:51.448 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:51.457+0000 7fc90effd700 1 -- 172.21.15.52:0/44374434 <== mon.0 v2:172.21.15.52:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191963 (secure 0 0 0) 0x7fc914013070 con 0x7fc9181034b0 2024-05-27T07:25:51.528 INFO:journalctl@ceph.mgr.z.smithi173.stdout:May 27 07:25:51 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mgr-z[38891]: 2024-05-27T07:25:51.195+0000 7f41c267f200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-05-27T07:25:51.528 INFO:journalctl@ceph.mgr.z.smithi173.stdout:May 27 07:25:51 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mgr-z[38891]: 2024-05-27T07:25:51.367+0000 7f41c267f200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-05-27T07:25:51.528 INFO:journalctl@ceph.mgr.z.smithi173.stdout:May 27 07:25:51 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mgr-z[38891]: 2024-05-27T07:25:51.538+0000 7f41c267f200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-05-27T07:25:51.643 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:51.653+0000 7fc91d758700 1 -- 172.21.15.52:0/44374434 >> [v2:172.21.15.52:6800/3513075805,v1:172.21.15.52:6801/3513075805] conn(0x7fc9040b28d0 msgr2=0x7fc9040b4de0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.52:6800/3513075805 2024-05-27T07:25:51.644 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:51.653+0000 7fc91d758700 1 --2- 172.21.15.52:0/44374434 >> [v2:172.21.15.52:6800/3513075805,v1:172.21.15.52:6801/3513075805] conn(0x7fc9040b28d0 0x7fc9040b4de0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-05-27T07:25:51.710 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:51.719+0000 7fc9201bd700 1 -- 172.21.15.52:0/44374434 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_command({"prefix": "fs ls"} v 0) -- 0x7fc91802e7f0 con 0x7fc9181034b0 2024-05-27T07:25:51.710 INFO:journalctl@ceph.mgr.x.smithi052.stdout:May 27 07:25:51 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mgr-x[32184]: 2024-05-27T07:25:51.605+0000 7fd16c7b8200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-05-27T07:25:51.710 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:51.719+0000 7fc90effd700 1 -- 172.21.15.52:0/44374434 <== mon.0 v2:172.21.15.52:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v29) ==== 53+0+88 (secure 0 0 0) 0x7fc9140aae90 con 0x7fc9181034b0 2024-05-27T07:25:51.711 INFO:teuthology.orchestra.run.smithi052.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data cephfs_data_ec ] 2024-05-27T07:25:51.712 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:51.722+0000 7fc9201bd700 1 -- 172.21.15.52:0/44374434 >> [v2:172.21.15.52:6800/3513075805,v1:172.21.15.52:6801/3513075805] conn(0x7fc9040b28d0 msgr2=0x7fc9040b4de0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-05-27T07:25:51.712 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:51.722+0000 7fc9201bd700 1 --2- 172.21.15.52:0/44374434 >> [v2:172.21.15.52:6800/3513075805,v1:172.21.15.52:6801/3513075805] conn(0x7fc9040b28d0 0x7fc9040b4de0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:25:51.713 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:51.722+0000 7fc9201bd700 1 -- 172.21.15.52:0/44374434 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fc9181034b0 msgr2=0x7fc9180ff290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:25:51.713 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:51.722+0000 7fc9201bd700 1 --2- 172.21.15.52:0/44374434 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fc9181034b0 0x7fc9180ff290 secure :-1 s=READY pgs=393 cs=0 l=1 rev1=1 crypto rx=0x7fc9140098e0 tx=0x7fc9140057c0 comp rx=0 tx=0).stop 2024-05-27T07:25:51.713 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:51.722+0000 7fc9201bd700 1 -- 172.21.15.52:0/44374434 shutdown_connections 2024-05-27T07:25:51.713 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:51.722+0000 7fc9201bd700 1 --2- 172.21.15.52:0/44374434 >> [v2:172.21.15.52:6800/3513075805,v1:172.21.15.52:6801/3513075805] conn(0x7fc9040b28d0 0x7fc9040b4de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:25:51.713 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:51.722+0000 7fc9201bd700 1 --2- 172.21.15.52:0/44374434 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fc918108730 0x7fc9180ffd10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:25:51.713 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:51.722+0000 7fc9201bd700 1 --2- 172.21.15.52:0/44374434 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fc9181034b0 0x7fc9180ff290 unknown :-1 s=CLOSED pgs=393 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:25:51.713 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:51.722+0000 7fc9201bd700 1 --2- 172.21.15.52:0/44374434 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fc918107de0 0x7fc9180ff7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:25:51.713 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:51.722+0000 7fc9201bd700 1 -- 172.21.15.52:0/44374434 >> 172.21.15.52:0/44374434 conn(0x7fc9180fedb0 msgr2=0x7fc91806b9e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:25:51.713 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:51.722+0000 7fc9201bd700 1 -- 172.21.15.52:0/44374434 shutdown_connections 2024-05-27T07:25:51.713 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:51.722+0000 7fc9201bd700 1 -- 172.21.15.52:0/44374434 wait complete. 2024-05-27T07:25:51.815 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:25:51 smithi105 ceph-mon[35977]: from='client.? 172.21.15.52:0/44374434' entity='client.admin' cmd=[{"prefix": "fs ls"}]: dispatch 2024-05-27T07:25:51.815 INFO:journalctl@ceph.mgr.y.smithi105.stdout:May 27 07:25:51 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mgr-y[37393]: 2024-05-27T07:25:51.529+0000 7f1202724200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-05-27T07:25:51.821 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:25:51 smithi173 ceph-mon[37000]: from='client.? 172.21.15.52:0/44374434' entity='client.admin' cmd=[{"prefix": "fs ls"}]: dispatch 2024-05-27T07:25:51.979 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:25:51 smithi052 ceph-mon[31955]: from='client.? 172.21.15.52:0/44374434' entity='client.admin' cmd=[{"prefix": "fs ls"}]: dispatch 2024-05-27T07:25:51.979 INFO:journalctl@ceph.mgr.x.smithi052.stdout:May 27 07:25:51 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mgr-x[32184]: 2024-05-27T07:25:51.795+0000 7fd16c7b8200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-05-27T07:25:52.101 INFO:journalctl@ceph.mgr.z.smithi173.stdout:May 27 07:25:51 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mgr-z[38891]: 2024-05-27T07:25:51.831+0000 7f41c267f200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-05-27T07:25:52.102 INFO:journalctl@ceph.mgr.z.smithi173.stdout:May 27 07:25:51 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mgr-z[38891]: 2024-05-27T07:25:51.924+0000 7f41c267f200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-05-27T07:25:52.110 INFO:journalctl@ceph.mgr.y.smithi105.stdout:May 27 07:25:51 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mgr-y[37393]: 2024-05-27T07:25:51.824+0000 7f1202724200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-05-27T07:25:52.110 INFO:journalctl@ceph.mgr.y.smithi105.stdout:May 27 07:25:51 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mgr-y[37393]: 2024-05-27T07:25:51.916+0000 7f1202724200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-05-27T07:25:52.400 INFO:journalctl@ceph.mgr.x.smithi052.stdout:May 27 07:25:52 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mgr-x[32184]: 2024-05-27T07:25:52.109+0000 7fd16c7b8200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-05-27T07:25:52.400 INFO:journalctl@ceph.mgr.x.smithi052.stdout:May 27 07:25:52 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mgr-x[32184]: 2024-05-27T07:25:52.209+0000 7fd16c7b8200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-05-27T07:25:52.428 INFO:tasks.cephfs.mount:Mounting default Ceph FS; just confirmed its presence on cluster 2024-05-27T07:25:52.428 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-05-27T07:25:52.429 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi052.front.sepia.ceph.com 2024-05-27T07:25:52.429 INFO:tasks.cephfs.mount:self.client.name = client.0 2024-05-27T07:25:52.429 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2024-05-27T07:25:52.429 INFO:tasks.cephfs.mount:self.cephfs_name = None 2024-05-27T07:25:52.429 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2024-05-27T07:25:52.429 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-05-27T07:25:52.429 INFO:tasks.cephfs.mount:Setting the 'None' netns for '/home/ubuntu/cephtest/mnt.0' 2024-05-27T07:25:52.429 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-27T07:25:52.429 DEBUG:teuthology.orchestra.run.smithi052:> ip addr 2024-05-27T07:25:52.439 INFO:journalctl@ceph.mgr.y.smithi105.stdout:May 27 07:25:52 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mgr-y[37393]: 2024-05-27T07:25:52.119+0000 7f1202724200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-05-27T07:25:52.439 INFO:journalctl@ceph.mgr.y.smithi105.stdout:May 27 07:25:52 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mgr-y[37393]: 2024-05-27T07:25:52.316+0000 7f1202724200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-05-27T07:25:52.446 INFO:teuthology.orchestra.run.smithi052.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-05-27T07:25:52.447 INFO:teuthology.orchestra.run.smithi052.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-05-27T07:25:52.447 INFO:teuthology.orchestra.run.smithi052.stdout: inet 127.0.0.1/8 scope host lo 2024-05-27T07:25:52.447 INFO:teuthology.orchestra.run.smithi052.stdout: valid_lft forever preferred_lft forever 2024-05-27T07:25:52.447 INFO:teuthology.orchestra.run.smithi052.stdout: inet6 ::1/128 scope host 2024-05-27T07:25:52.447 INFO:teuthology.orchestra.run.smithi052.stdout: valid_lft forever preferred_lft forever 2024-05-27T07:25:52.447 INFO:teuthology.orchestra.run.smithi052.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-05-27T07:25:52.447 INFO:teuthology.orchestra.run.smithi052.stdout: link/ether 0c:c4:7a:6b:fd:d2 brd ff:ff:ff:ff:ff:ff 2024-05-27T07:25:52.447 INFO:teuthology.orchestra.run.smithi052.stdout: altname enp7s0f0 2024-05-27T07:25:52.447 INFO:teuthology.orchestra.run.smithi052.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-05-27T07:25:52.447 INFO:teuthology.orchestra.run.smithi052.stdout: link/ether 0c:c4:7a:6b:fd:d3 brd ff:ff:ff:ff:ff:ff 2024-05-27T07:25:52.447 INFO:teuthology.orchestra.run.smithi052.stdout: altname enp7s0f1 2024-05-27T07:25:52.447 INFO:teuthology.orchestra.run.smithi052.stdout:4: ens1f0: mtu 1500 qdisc mq state UP group default qlen 1000 2024-05-27T07:25:52.447 INFO:teuthology.orchestra.run.smithi052.stdout: link/ether 0c:c4:7a:bd:15:de brd ff:ff:ff:ff:ff:ff 2024-05-27T07:25:52.448 INFO:teuthology.orchestra.run.smithi052.stdout: altname enp1s0f0 2024-05-27T07:25:52.448 INFO:teuthology.orchestra.run.smithi052.stdout: inet 172.21.15.52/20 brd 172.21.15.255 scope global dynamic noprefixroute ens1f0 2024-05-27T07:25:52.448 INFO:teuthology.orchestra.run.smithi052.stdout: valid_lft 41964sec preferred_lft 41964sec 2024-05-27T07:25:52.448 INFO:teuthology.orchestra.run.smithi052.stdout: inet6 fe80::ec4:7aff:febd:15de/64 scope link noprefixroute 2024-05-27T07:25:52.448 INFO:teuthology.orchestra.run.smithi052.stdout: valid_lft forever preferred_lft forever 2024-05-27T07:25:52.448 INFO:teuthology.orchestra.run.smithi052.stdout:5: ens1f1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-05-27T07:25:52.448 INFO:teuthology.orchestra.run.smithi052.stdout: link/ether 0c:c4:7a:bd:15:df brd ff:ff:ff:ff:ff:ff 2024-05-27T07:25:52.448 INFO:teuthology.orchestra.run.smithi052.stdout: altname enp1s0f1 2024-05-27T07:25:52.448 INFO:tasks.cephfs.mount:Setuping the 'ceph-brx' with 192.168.255.254/16 2024-05-27T07:25:52.449 DEBUG:teuthology.orchestra.run.smithi052:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-05-27T07:25:52.449 DEBUG:teuthology.orchestra.run.smithi052:> set -e 2024-05-27T07:25:52.449 DEBUG:teuthology.orchestra.run.smithi052:> sudo ip link add name ceph-brx type bridge 2024-05-27T07:25:52.449 DEBUG:teuthology.orchestra.run.smithi052:> sudo ip addr flush dev ceph-brx 2024-05-27T07:25:52.449 DEBUG:teuthology.orchestra.run.smithi052:> sudo ip link set ceph-brx up 2024-05-27T07:25:52.449 DEBUG:teuthology.orchestra.run.smithi052:> sudo ip addr add 192.168.255.254/16 brd 192.168.255.255 dev ceph-brx 2024-05-27T07:25:52.449 DEBUG:teuthology.orchestra.run.smithi052:> ') 2024-05-27T07:25:52.538 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:52 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-05-27T07:25:52.588 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:52 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-05-27T07:25:52.595 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-27T07:25:52.595 DEBUG:teuthology.orchestra.run.smithi052:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-05-27T07:25:52.601 INFO:journalctl@ceph.mgr.z.smithi173.stdout:May 27 07:25:52 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mgr-z[38891]: 2024-05-27T07:25:52.130+0000 7f41c267f200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-05-27T07:25:52.601 INFO:journalctl@ceph.mgr.z.smithi173.stdout:May 27 07:25:52 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mgr-z[38891]: 2024-05-27T07:25:52.325+0000 7f41c267f200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-05-27T07:25:52.623 INFO:teuthology.orchestra.run.smithi052.stdout:1 2024-05-27T07:25:52.625 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-27T07:25:52.625 DEBUG:teuthology.orchestra.run.smithi052:> route 2024-05-27T07:25:52.683 INFO:teuthology.orchestra.run.smithi052.stdout:Kernel IP routing table 2024-05-27T07:25:52.684 INFO:teuthology.orchestra.run.smithi052.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-05-27T07:25:52.684 INFO:teuthology.orchestra.run.smithi052.stdout:default _gateway 0.0.0.0 UG 100 0 0 ens1f0 2024-05-27T07:25:52.684 INFO:teuthology.orchestra.run.smithi052.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 ens1f0 2024-05-27T07:25:52.684 INFO:teuthology.orchestra.run.smithi052.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-05-27T07:25:52.684 DEBUG:teuthology.orchestra.run.smithi052:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-05-27T07:25:52.684 DEBUG:teuthology.orchestra.run.smithi052:> set -e 2024-05-27T07:25:52.684 DEBUG:teuthology.orchestra.run.smithi052:> sudo iptables -A FORWARD -o ens1f0 -i ceph-brx -j ACCEPT 2024-05-27T07:25:52.684 DEBUG:teuthology.orchestra.run.smithi052:> sudo iptables -A FORWARD -i ens1f0 -o ceph-brx -j ACCEPT 2024-05-27T07:25:52.685 DEBUG:teuthology.orchestra.run.smithi052:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o ens1f0 -j MASQUERADE 2024-05-27T07:25:52.685 DEBUG:teuthology.orchestra.run.smithi052:> ') 2024-05-27T07:25:52.729 INFO:journalctl@ceph.mgr.x.smithi052.stdout:May 27 07:25:52 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mgr-x[32184]: 2024-05-27T07:25:52.427+0000 7fd16c7b8200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-05-27T07:25:52.729 INFO:journalctl@ceph.mgr.x.smithi052.stdout:May 27 07:25:52 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mgr-x[32184]: 2024-05-27T07:25:52.646+0000 7fd16c7b8200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-05-27T07:25:52.776 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:52 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-05-27T07:25:52.811 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:52 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-05-27T07:25:52.818 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-27T07:25:52.818 DEBUG:teuthology.orchestra.run.smithi052:> ip netns list 2024-05-27T07:25:52.874 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-27T07:25:52.874 DEBUG:teuthology.orchestra.run.smithi052:> ip netns list-id 2024-05-27T07:25:52.930 DEBUG:teuthology.orchestra.run.smithi052:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-05-27T07:25:52.930 DEBUG:teuthology.orchestra.run.smithi052:> set -e 2024-05-27T07:25:52.930 DEBUG:teuthology.orchestra.run.smithi052:> sudo ip netns add ceph-ns--home-ubuntu-cephtest-mnt.0 2024-05-27T07:25:52.930 DEBUG:teuthology.orchestra.run.smithi052:> sudo ip netns set ceph-ns--home-ubuntu-cephtest-mnt.0 0 2024-05-27T07:25:52.930 DEBUG:teuthology.orchestra.run.smithi052:> ') 2024-05-27T07:25:53.022 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:53 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-05-27T07:25:53.047 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:53 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-05-27T07:25:53.056 INFO:tasks.cephfs.mount:Setuping the netns 'ceph-ns--home-ubuntu-cephtest-mnt.0' with 192.168.0.1/16 2024-05-27T07:25:53.056 DEBUG:teuthology.orchestra.run.smithi052:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-05-27T07:25:53.057 DEBUG:teuthology.orchestra.run.smithi052:> set -e 2024-05-27T07:25:53.057 DEBUG:teuthology.orchestra.run.smithi052:> sudo ip link add veth0 netns ceph-ns--home-ubuntu-cephtest-mnt.0 type veth peer name brx.0 2024-05-27T07:25:53.057 DEBUG:teuthology.orchestra.run.smithi052:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip addr add 192.168.0.1/16 brd 192.168.255.255 dev veth0 2024-05-27T07:25:53.057 DEBUG:teuthology.orchestra.run.smithi052:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip link set veth0 up 2024-05-27T07:25:53.057 DEBUG:teuthology.orchestra.run.smithi052:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip link set lo up 2024-05-27T07:25:53.057 DEBUG:teuthology.orchestra.run.smithi052:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip route add default via 192.168.255.254 2024-05-27T07:25:53.057 DEBUG:teuthology.orchestra.run.smithi052:> ') 2024-05-27T07:25:53.108 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:53 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-05-27T07:25:53.174 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:53 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-05-27T07:25:53.181 DEBUG:teuthology.orchestra.run.smithi052:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-05-27T07:25:53.181 DEBUG:teuthology.orchestra.run.smithi052:> set -e 2024-05-27T07:25:53.181 DEBUG:teuthology.orchestra.run.smithi052:> sudo ip link set brx.0 up 2024-05-27T07:25:53.182 DEBUG:teuthology.orchestra.run.smithi052:> sudo ip link set dev brx.0 master ceph-brx 2024-05-27T07:25:53.182 DEBUG:teuthology.orchestra.run.smithi052:> ') 2024-05-27T07:25:53.189 INFO:journalctl@ceph.mgr.y.smithi105.stdout:May 27 07:25:52 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mgr-y[37393]: 2024-05-27T07:25:52.916+0000 7f1202724200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-05-27T07:25:53.189 INFO:journalctl@ceph.mgr.y.smithi105.stdout:May 27 07:25:53 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mgr-y[37393]: 2024-05-27T07:25:53.139+0000 7f1202724200 -1 mgr[py] Module smb has missing NOTIFY_TYPES member 2024-05-27T07:25:53.233 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:53 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-05-27T07:25:53.258 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:53 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-05-27T07:25:53.265 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-05-27T07:25:53.265 DEBUG:teuthology.orchestra.run.smithi052:> mkdir -p -v /home/ubuntu/cephtest/mnt.0 2024-05-27T07:25:53.280 INFO:teuthology.orchestra.run.smithi052.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.0' 2024-05-27T07:25:53.280 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-05-27T07:25:53.281 DEBUG:teuthology.orchestra.run.smithi052:> chmod 0000 /home/ubuntu/cephtest/mnt.0 2024-05-27T07:25:53.306 INFO:journalctl@ceph.mgr.x.smithi052.stdout:May 27 07:25:53 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mgr-x[32184]: 2024-05-27T07:25:53.315+0000 7fd16c7b8200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-05-27T07:25:53.321 WARNING:tasks.cephfs.mount:You cannot specify both: cephfs_mntpt and mount_subvol_num 2024-05-27T07:25:53.321 INFO:tasks.cephfs.mount:Mounting subvol 1 for now 2024-05-27T07:25:53.321 INFO:tasks.cephfs.kernel_mount:mounting using device: :/volumes/qa/sv_1/95e9ac73-825f-48e3-998b-133d94d96559 2024-05-27T07:25:53.322 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-27T07:25:53.322 DEBUG:teuthology.orchestra.run.smithi052:> sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage /bin/mount -t ceph :/volumes/qa/sv_1/95e9ac73-825f-48e3-998b-133d94d96559 /home/ubuntu/cephtest/mnt.0 -v -o norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs,ms_mode=legacy,wsync 2024-05-27T07:25:53.351 INFO:journalctl@ceph.mgr.z.smithi173.stdout:May 27 07:25:52 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mgr-z[38891]: 2024-05-27T07:25:52.930+0000 7f41c267f200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-05-27T07:25:53.352 INFO:journalctl@ceph.mgr.z.smithi173.stdout:May 27 07:25:53 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mgr-z[38891]: 2024-05-27T07:25:53.152+0000 7f41c267f200 -1 mgr[py] Module smb has missing NOTIFY_TYPES member 2024-05-27T07:25:53.418 INFO:teuthology.orchestra.run.smithi052.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs,ms_mode=legacy,wsync 2024-05-27T07:25:53.418 INFO:teuthology.orchestra.run.smithi052.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs,ms_mode=legacy,wsync". 2024-05-27T07:25:53.418 INFO:teuthology.orchestra.run.smithi052.stdout:invalid new device string format 2024-05-27T07:25:53.553 INFO:teuthology.orchestra.run.smithi052.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs,ms_mode=legacy,wsync 2024-05-27T07:25:53.553 INFO:teuthology.orchestra.run.smithi052.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs,ms_mode=legacy,wsync". 2024-05-27T07:25:53.553 INFO:teuthology.orchestra.run.smithi052.stdout:invalid new device string format 2024-05-27T07:25:53.553 INFO:teuthology.orchestra.run.smithi052.stdout:mount.ceph: resolved to: "172.21.15.52:6789,172.21.15.105:6789,172.21.15.173:6789" 2024-05-27T07:25:53.553 INFO:teuthology.orchestra.run.smithi052.stdout:mount.ceph: trying mount with old device syntax: 172.21.15.52:6789,172.21.15.105:6789,172.21.15.173:6789:/volumes/qa/sv_1/95e9ac73-825f-48e3-998b-133d94d96559 2024-05-27T07:25:53.553 INFO:teuthology.orchestra.run.smithi052.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs,ms_mode=legacy,wsync,key=0,fsid=73df2636-1bf8-11ef-bc9b-c7b262605968" will pass to kernel 2024-05-27T07:25:53.554 INFO:teuthology.orchestra.run.smithi052.stdout:mount: /home/ubuntu/cephtest/mnt.0 does not contain SELinux labels. 2024-05-27T07:25:53.554 INFO:teuthology.orchestra.run.smithi052.stdout: You just mounted a file system that supports labels which does not 2024-05-27T07:25:53.554 INFO:teuthology.orchestra.run.smithi052.stdout: contain labels, onto an SELinux box. It is likely that confined 2024-05-27T07:25:53.554 INFO:teuthology.orchestra.run.smithi052.stdout: applications will generate AVC messages and not be allowed access to 2024-05-27T07:25:53.554 INFO:teuthology.orchestra.run.smithi052.stdout: this file system. For more details see restorecon(8) and mount(8). 2024-05-27T07:25:53.556 INFO:tasks.cephfs.kernel_mount:mount command passed 2024-05-27T07:25:53.556 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-27T07:25:53.556 DEBUG:teuthology.orchestra.run.smithi052:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.0 2024-05-27T07:25:53.580 INFO:journalctl@ceph.mgr.x.smithi052.stdout:May 27 07:25:53 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mgr-x[32184]: 2024-05-27T07:25:53.562+0000 7fd16c7b8200 -1 mgr[py] Module smb has missing NOTIFY_TYPES member 2024-05-27T07:25:53.585 DEBUG:teuthology.orchestra.run.smithi052:> (cd /home/ubuntu/cephtest/mnt.0 && exec stdin-killer --timeout=900 -- bash -c 'getfattr --only-values -n ceph.client_id .') 2024-05-27T07:25:53.677 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:53 stdin-killer INFO: expiration expected; waiting 900 seconds for command to complete 2024-05-27T07:25:53.679 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:53 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-05-27T07:25:53.685 INFO:teuthology.orchestra.run.smithi052.stdout:client25060 2024-05-27T07:25:53.686 DEBUG:teuthology.orchestra.run.smithi052:> set -ex 2024-05-27T07:25:53.686 DEBUG:teuthology.orchestra.run.smithi052:> dd if=/proc/self/mounts of=/dev/stdout 2024-05-27T07:25:53.703 DEBUG:teuthology.orchestra.run.smithi052:> (cd /home/ubuntu/cephtest/mnt.0 && exec stdin-killer --timeout=900 -- bash -c 'getfattr --only-values -n ceph.client_id .') 2024-05-27T07:25:53.795 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:53 stdin-killer INFO: expiration expected; waiting 900 seconds for command to complete 2024-05-27T07:25:53.797 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:53 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-05-27T07:25:53.803 INFO:teuthology.orchestra.run.smithi052.stdout:client25060 2024-05-27T07:25:53.804 DEBUG:teuthology.orchestra.run.smithi052:> (cd /home/ubuntu/cephtest/mnt.0 && exec stdin-killer --timeout=900 -- bash -c 'getfattr --only-values -n ceph.client_id .') 2024-05-27T07:25:53.851 INFO:journalctl@ceph.mgr.z.smithi173.stdout:May 27 07:25:53 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mgr-z[38891]: 2024-05-27T07:25:53.584+0000 7f41c267f200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-05-27T07:25:53.852 INFO:journalctl@ceph.mgr.z.smithi173.stdout:May 27 07:25:53 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mgr-z[38891]: 2024-05-27T07:25:53.682+0000 7f41c267f200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-05-27T07:25:53.855 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:53 stdin-killer INFO: expiration expected; waiting 900 seconds for command to complete 2024-05-27T07:25:53.857 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:53 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-05-27T07:25:53.863 INFO:teuthology.orchestra.run.smithi052.stdout:client25060 2024-05-27T07:25:53.863 DEBUG:teuthology.orchestra.run.smithi052:> (cd /home/ubuntu/cephtest/mnt.0 && exec stdin-killer --timeout=300 -- bash -c 'sudo dd if=/sys/kernel/debug/ceph/73df2636-1bf8-11ef-bc9b-c7b262605968.client25060/status') 2024-05-27T07:25:53.914 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:53 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-05-27T07:25:53.925 INFO:teuthology.orchestra.run.smithi052.stdout:instance: client.25060 (3)172.21.15.52:0/2820888226 2024-05-27T07:25:53.925 INFO:teuthology.orchestra.run.smithi052.stdout:blocklisted: false 2024-05-27T07:25:53.926 INFO:teuthology.orchestra.run.smithi052.stderr:0+1 records in 2024-05-27T07:25:53.926 INFO:teuthology.orchestra.run.smithi052.stderr:0+1 records out 2024-05-27T07:25:53.926 INFO:teuthology.orchestra.run.smithi052.stderr:71 bytes copied, 2.549e-05 s, 2.8 MB/s 2024-05-27T07:25:53.926 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:53 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-05-27T07:25:53.933 INFO:teuthology.run_tasks:Running task exec... 2024-05-27T07:25:53.943 INFO:teuthology.task.exec:Executing custom commands... 2024-05-27T07:25:53.944 INFO:teuthology.task.exec:Running commands on role mon.a host ubuntu@smithi052.front.sepia.ceph.com 2024-05-27T07:25:53.944 DEBUG:teuthology.orchestra.run.smithi052:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'ceph fs set cephfs balance_automate false' 2024-05-27T07:25:53.944 INFO:journalctl@ceph.mgr.y.smithi105.stdout:May 27 07:25:53 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mgr-y[37393]: 2024-05-27T07:25:53.563+0000 7f1202724200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-05-27T07:25:53.945 INFO:journalctl@ceph.mgr.y.smithi105.stdout:May 27 07:25:53 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mgr-y[37393]: 2024-05-27T07:25:53.663+0000 7f1202724200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-05-27T07:25:54.062 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:54.071+0000 7f4d313cf640 1 -- 172.21.15.52:0/184796027 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f4d2c103c60 msgr2=0x7f4d2c104960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:25:54.062 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:54.071+0000 7f4d313cf640 1 --2- 172.21.15.52:0/184796027 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f4d2c103c60 0x7f4d2c104960 secure :-1 s=READY pgs=394 cs=0 l=1 rev1=1 crypto rx=0x7f4d1c009900 tx=0x7f4d1c02a410 comp rx=0 tx=0).stop 2024-05-27T07:25:54.062 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:54.071+0000 7f4d313cf640 1 -- 172.21.15.52:0/184796027 shutdown_connections 2024-05-27T07:25:54.063 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:54.071+0000 7f4d313cf640 1 --2- 172.21.15.52:0/184796027 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f4d2c109ba0 0x7f4d2c10bfc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:25:54.063 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:54.071+0000 7f4d313cf640 1 --2- 172.21.15.52:0/184796027 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f4d2c104ea0 0x7f4d2c1052b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:25:54.063 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:54.071+0000 7f4d313cf640 1 --2- 172.21.15.52:0/184796027 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f4d2c103c60 0x7f4d2c104960 unknown :-1 s=CLOSED pgs=394 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:25:54.063 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:54.071+0000 7f4d313cf640 1 -- 172.21.15.52:0/184796027 >> 172.21.15.52:0/184796027 conn(0x7f4d2c0fee30 msgr2=0x7f4d2c101300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:25:54.063 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:54.071+0000 7f4d313cf640 1 -- 172.21.15.52:0/184796027 shutdown_connections 2024-05-27T07:25:54.063 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:54.071+0000 7f4d313cf640 1 -- 172.21.15.52:0/184796027 wait complete. 2024-05-27T07:25:54.063 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:54.071+0000 7f4d313cf640 1 Processor -- start 2024-05-27T07:25:54.063 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:54.072+0000 7f4d313cf640 1 -- start start 2024-05-27T07:25:54.065 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:54.072+0000 7f4d313cf640 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f4d2c103c60 0x7f4d2c19a380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:25:54.065 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:54.072+0000 7f4d313cf640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f4d2c104ea0 0x7f4d2c19a8c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:25:54.065 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:54.072+0000 7f4d313cf640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f4d2c109ba0 0x7f4d2c1a1850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:25:54.065 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:54.072+0000 7f4d313cf640 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x7f4d2c10f740 con 0x7f4d2c103c60 2024-05-27T07:25:54.065 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:54.072+0000 7f4d313cf640 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f4d2c10f5c0 con 0x7f4d2c109ba0 2024-05-27T07:25:54.065 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:54.072+0000 7f4d313cf640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f4d2c10f8c0 con 0x7f4d2c104ea0 2024-05-27T07:25:54.065 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:54.072+0000 7f4d2affd640 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f4d2c103c60 0x7f4d2c19a380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:25:54.065 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:54.072+0000 7f4d2affd640 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f4d2c103c60 0x7f4d2c19a380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.52:59234/0 (socket says 172.21.15.52:59234) 2024-05-27T07:25:54.065 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:54.072+0000 7f4d2affd640 1 -- 172.21.15.52:0/2121853702 learned_addr learned my addr 172.21.15.52:0/2121853702 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-05-27T07:25:54.066 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:54.072+0000 7f4d2a7fc640 1 --2- 172.21.15.52:0/2121853702 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f4d2c104ea0 0x7f4d2c19a8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:25:54.066 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:54.072+0000 7f4d2b7fe640 1 --2- 172.21.15.52:0/2121853702 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f4d2c109ba0 0x7f4d2c1a1850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:25:54.066 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:54.072+0000 7f4d2affd640 1 -- 172.21.15.52:0/2121853702 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f4d2c104ea0 msgr2=0x7f4d2c19a8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:25:54.066 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:54.072+0000 7f4d2affd640 1 --2- 172.21.15.52:0/2121853702 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f4d2c104ea0 0x7f4d2c19a8c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:25:54.066 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:54.072+0000 7f4d2affd640 1 -- 172.21.15.52:0/2121853702 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f4d2c109ba0 msgr2=0x7f4d2c1a1850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:25:54.066 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:54.072+0000 7f4d2affd640 1 --2- 172.21.15.52:0/2121853702 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f4d2c109ba0 0x7f4d2c1a1850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:25:54.066 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:54.072+0000 7f4d2affd640 1 -- 172.21.15.52:0/2121853702 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4d2c1a1ec0 con 0x7f4d2c103c60 2024-05-27T07:25:54.066 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:54.072+0000 7f4d2a7fc640 1 --2- 172.21.15.52:0/2121853702 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f4d2c104ea0 0x7f4d2c19a8c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-27T07:25:54.066 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:54.072+0000 7f4d2b7fe640 1 --2- 172.21.15.52:0/2121853702 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f4d2c109ba0 0x7f4d2c1a1850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-27T07:25:54.066 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:54.073+0000 7f4d2affd640 1 --2- 172.21.15.52:0/2121853702 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f4d2c103c60 0x7f4d2c19a380 secure :-1 s=READY pgs=395 cs=0 l=1 rev1=1 crypto rx=0x7f4d1c006010 tx=0x7f4d1c004270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:25:54.066 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:54.073+0000 7f4d13fff640 1 -- 172.21.15.52:0/2121853702 <== mon.0 v2:172.21.15.52:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4d1c03f030 con 0x7f4d2c103c60 2024-05-27T07:25:54.067 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:54.073+0000 7f4d313cf640 1 -- 172.21.15.52:0/2121853702 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4d2c1a2150 con 0x7f4d2c103c60 2024-05-27T07:25:54.067 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:54.073+0000 7f4d13fff640 1 -- 172.21.15.52:0/2121853702 <== mon.0 v2:172.21.15.52:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f4d1c031910 con 0x7f4d2c103c60 2024-05-27T07:25:54.067 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:54.073+0000 7f4d13fff640 1 -- 172.21.15.52:0/2121853702 <== mon.0 v2:172.21.15.52:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4d1c035070 con 0x7f4d2c103c60 2024-05-27T07:25:54.067 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:54.073+0000 7f4d313cf640 1 -- 172.21.15.52:0/2121853702 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4d2c1a2690 con 0x7f4d2c103c60 2024-05-27T07:25:54.067 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:54.074+0000 7f4d313cf640 1 -- 172.21.15.52:0/2121853702 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4d2c1089b0 con 0x7f4d2c103c60 2024-05-27T07:25:54.068 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:54.076+0000 7f4d13fff640 1 -- 172.21.15.52:0/2121853702 <== mon.0 v2:172.21.15.52:3300/0 4 ==== mgrmap(e 19) ==== 149669+0+0 (secure 0 0 0) 0x7f4d1c042030 con 0x7f4d2c103c60 2024-05-27T07:25:54.068 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:54.077+0000 7f4d13fff640 1 --2- 172.21.15.52:0/2121853702 >> [v2:172.21.15.52:6800/3513075805,v1:172.21.15.52:6801/3513075805] conn(0x7f4d140b2830 0x7f4d140b4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:25:54.068 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:54.077+0000 7f4d2a7fc640 1 -- 172.21.15.52:0/2121853702 >> [v2:172.21.15.52:6800/3513075805,v1:172.21.15.52:6801/3513075805] conn(0x7f4d140b2830 msgr2=0x7f4d140b4d40 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.52:6800/3513075805 2024-05-27T07:25:54.068 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:54.077+0000 7f4d13fff640 1 -- 172.21.15.52:0/2121853702 <== mon.0 v2:172.21.15.52:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9121+0+0 (secure 0 0 0) 0x7f4d1c102950 con 0x7f4d2c103c60 2024-05-27T07:25:54.068 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:54.077+0000 7f4d2a7fc640 1 --2- 172.21.15.52:0/2121853702 >> [v2:172.21.15.52:6800/3513075805,v1:172.21.15.52:6801/3513075805] conn(0x7f4d140b2830 0x7f4d140b4d40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-05-27T07:25:54.072 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:54.082+0000 7f4d13fff640 1 -- 172.21.15.52:0/2121853702 <== mon.0 v2:172.21.15.52:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191963 (secure 0 0 0) 0x7f4d1c0ca290 con 0x7f4d2c103c60 2024-05-27T07:25:54.229 INFO:journalctl@ceph.mgr.x.smithi052.stdout:May 27 07:25:54 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mgr-x[32184]: 2024-05-27T07:25:54.039+0000 7fd16c7b8200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-05-27T07:25:54.229 INFO:journalctl@ceph.mgr.x.smithi052.stdout:May 27 07:25:54 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mgr-x[32184]: 2024-05-27T07:25:54.149+0000 7fd16c7b8200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-05-27T07:25:54.268 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:54.278+0000 7f4d2a7fc640 1 -- 172.21.15.52:0/2121853702 >> [v2:172.21.15.52:6800/3513075805,v1:172.21.15.52:6801/3513075805] conn(0x7f4d140b2830 msgr2=0x7f4d140b4d40 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.52:6800/3513075805 2024-05-27T07:25:54.268 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:54.278+0000 7f4d2a7fc640 1 --2- 172.21.15.52:0/2121853702 >> [v2:172.21.15.52:6800/3513075805,v1:172.21.15.52:6801/3513075805] conn(0x7f4d140b2830 0x7f4d140b4d40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-05-27T07:25:54.312 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:54.321+0000 7f4d313cf640 1 -- 172.21.15.52:0/2121853702 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "balance_automate", "val": "false"} v 0) -- 0x7f4d2c02dc20 con 0x7f4d2c103c60 2024-05-27T07:25:54.351 INFO:journalctl@ceph.mgr.z.smithi173.stdout:May 27 07:25:54 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mgr-z[38891]: 2024-05-27T07:25:54.058+0000 7f41c267f200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-05-27T07:25:54.364 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:54.373+0000 7f4d13fff640 1 -- 172.21.15.52:0/2121853702 <== mon.0 v2:172.21.15.52:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "balance_automate", "val": "false"}]=0 v30) ==== 118+0+0 (secure 0 0 0) 0x7f4d1c0c9b90 con 0x7f4d2c103c60 2024-05-27T07:25:54.367 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:54.376+0000 7f4d313cf640 1 -- 172.21.15.52:0/2121853702 >> [v2:172.21.15.52:6800/3513075805,v1:172.21.15.52:6801/3513075805] conn(0x7f4d140b2830 msgr2=0x7f4d140b4d40 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-05-27T07:25:54.367 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:54.376+0000 7f4d313cf640 1 --2- 172.21.15.52:0/2121853702 >> [v2:172.21.15.52:6800/3513075805,v1:172.21.15.52:6801/3513075805] conn(0x7f4d140b2830 0x7f4d140b4d40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:25:54.367 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:54.376+0000 7f4d313cf640 1 -- 172.21.15.52:0/2121853702 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f4d2c103c60 msgr2=0x7f4d2c19a380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:25:54.367 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:54.376+0000 7f4d313cf640 1 --2- 172.21.15.52:0/2121853702 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f4d2c103c60 0x7f4d2c19a380 secure :-1 s=READY pgs=395 cs=0 l=1 rev1=1 crypto rx=0x7f4d1c006010 tx=0x7f4d1c004270 comp rx=0 tx=0).stop 2024-05-27T07:25:54.367 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:54.376+0000 7f4d313cf640 1 -- 172.21.15.52:0/2121853702 shutdown_connections 2024-05-27T07:25:54.368 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:54.376+0000 7f4d313cf640 1 --2- 172.21.15.52:0/2121853702 >> [v2:172.21.15.52:6800/3513075805,v1:172.21.15.52:6801/3513075805] conn(0x7f4d140b2830 0x7f4d140b4d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:25:54.368 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:54.376+0000 7f4d313cf640 1 --2- 172.21.15.52:0/2121853702 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f4d2c109ba0 0x7f4d2c1a1850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:25:54.368 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:54.376+0000 7f4d313cf640 1 --2- 172.21.15.52:0/2121853702 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f4d2c104ea0 0x7f4d2c19a8c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:25:54.368 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:54.376+0000 7f4d313cf640 1 --2- 172.21.15.52:0/2121853702 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f4d2c103c60 0x7f4d2c19a380 unknown :-1 s=CLOSED pgs=395 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:25:54.368 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:54.376+0000 7f4d313cf640 1 -- 172.21.15.52:0/2121853702 >> 172.21.15.52:0/2121853702 conn(0x7f4d2c0fee30 msgr2=0x7f4d2c105b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:25:54.368 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:54.376+0000 7f4d313cf640 1 -- 172.21.15.52:0/2121853702 shutdown_connections 2024-05-27T07:25:54.368 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:54.376+0000 7f4d313cf640 1 -- 172.21.15.52:0/2121853702 wait complete. 2024-05-27T07:25:54.369 INFO:journalctl@ceph.mgr.y.smithi105.stdout:May 27 07:25:54 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mgr-y[37393]: 2024-05-27T07:25:54.036+0000 7f1202724200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-05-27T07:25:54.380 DEBUG:teuthology.orchestra.run.smithi052:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'ceph fs subvolumegroup pin cephfs qa random 0.10' 2024-05-27T07:25:54.499 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:54.508+0000 7f25db186640 1 -- 172.21.15.52:0/3771031302 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f25d4108a80 msgr2=0x7f25d410aea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:25:54.499 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:54.508+0000 7f25db186640 1 --2- 172.21.15.52:0/3771031302 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f25d4108a80 0x7f25d410aea0 secure :-1 s=READY pgs=396 cs=0 l=1 rev1=1 crypto rx=0x7f25cc009ca0 tx=0x7f25cc02a8c0 comp rx=0 tx=0).stop 2024-05-27T07:25:54.499 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:54.508+0000 7f25db186640 1 -- 172.21.15.52:0/3771031302 shutdown_connections 2024-05-27T07:25:54.499 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:54.508+0000 7f25db186640 1 --2- 172.21.15.52:0/3771031302 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f25d4108a80 0x7f25d410aea0 unknown :-1 s=CLOSED pgs=396 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:25:54.499 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:54.508+0000 7f25db186640 1 --2- 172.21.15.52:0/3771031302 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f25d4103d80 0x7f25d4104190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:25:54.500 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:54.508+0000 7f25db186640 1 --2- 172.21.15.52:0/3771031302 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f25d4102b40 0x7f25d4103840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:25:54.500 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:54.508+0000 7f25db186640 1 -- 172.21.15.52:0/3771031302 >> 172.21.15.52:0/3771031302 conn(0x7f25d40fdd30 msgr2=0x7f25d41001e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:25:54.500 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:54.508+0000 7f25db186640 1 -- 172.21.15.52:0/3771031302 shutdown_connections 2024-05-27T07:25:54.500 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:54.508+0000 7f25db186640 1 -- 172.21.15.52:0/3771031302 wait complete. 2024-05-27T07:25:54.500 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:54.508+0000 7f25db186640 1 Processor -- start 2024-05-27T07:25:54.500 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:54.509+0000 7f25db186640 1 -- start start 2024-05-27T07:25:54.502 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:54.509+0000 7f25db186640 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f25d4102b40 0x7f25d419c560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:25:54.502 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:54.509+0000 7f25db186640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f25d4103d80 0x7f25d419caa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:25:54.502 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:54.509+0000 7f25db186640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f25d4108a80 0x7f25d41a3a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:25:54.503 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:54.509+0000 7f25db186640 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x7f25d410e670 con 0x7f25d4102b40 2024-05-27T07:25:54.503 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:54.509+0000 7f25db186640 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f25d410e4f0 con 0x7f25d4103d80 2024-05-27T07:25:54.503 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:54.509+0000 7f25db186640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f25d410e7f0 con 0x7f25d4108a80 2024-05-27T07:25:54.503 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:54.509+0000 7f25d8f22640 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f25d4102b40 0x7f25d419c560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:25:54.503 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:54.509+0000 7f25d8f22640 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f25d4102b40 0x7f25d419c560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.52:59238/0 (socket says 172.21.15.52:59238) 2024-05-27T07:25:54.503 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:54.509+0000 7f25d8f22640 1 -- 172.21.15.52:0/3625918091 learned_addr learned my addr 172.21.15.52:0/3625918091 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-05-27T07:25:54.503 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:54.509+0000 7f25d3fff640 1 --2- 172.21.15.52:0/3625918091 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f25d4103d80 0x7f25d419caa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:25:54.503 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:54.509+0000 7f25d9723640 1 --2- 172.21.15.52:0/3625918091 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f25d4108a80 0x7f25d41a3a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:25:54.503 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:54.509+0000 7f25d3fff640 1 -- 172.21.15.52:0/3625918091 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f25d4108a80 msgr2=0x7f25d41a3a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:25:54.503 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:54.509+0000 7f25d3fff640 1 --2- 172.21.15.52:0/3625918091 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f25d4108a80 0x7f25d41a3a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:25:54.503 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:54.509+0000 7f25d3fff640 1 -- 172.21.15.52:0/3625918091 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f25d4102b40 msgr2=0x7f25d419c560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:25:54.503 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:54.509+0000 7f25d3fff640 1 --2- 172.21.15.52:0/3625918091 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f25d4102b40 0x7f25d419c560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:25:54.503 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:54.509+0000 7f25d3fff640 1 -- 172.21.15.52:0/3625918091 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f25d41a40a0 con 0x7f25d4103d80 2024-05-27T07:25:54.504 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:54.509+0000 7f25d8f22640 1 --2- 172.21.15.52:0/3625918091 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f25d4102b40 0x7f25d419c560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-27T07:25:54.504 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:54.510+0000 7f25d3fff640 1 --2- 172.21.15.52:0/3625918091 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f25d4103d80 0x7f25d419caa0 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f25c000e9f0 tx=0x7f25c000eeb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:25:54.504 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:54.510+0000 7f25d1ffb640 1 -- 172.21.15.52:0/3625918091 <== mon.2 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f25c000ccc0 con 0x7f25d4103d80 2024-05-27T07:25:54.504 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:54.510+0000 7f25db186640 1 -- 172.21.15.52:0/3625918091 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f25d41a4390 con 0x7f25d4103d80 2024-05-27T07:25:54.504 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:54.510+0000 7f25d1ffb640 1 -- 172.21.15.52:0/3625918091 <== mon.2 v2:172.21.15.105:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f25c0005020 con 0x7f25d4103d80 2024-05-27T07:25:54.504 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:54.510+0000 7f25d1ffb640 1 -- 172.21.15.52:0/3625918091 <== mon.2 v2:172.21.15.105:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f25c0004a00 con 0x7f25d4103d80 2024-05-27T07:25:54.504 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:54.510+0000 7f25db186640 1 -- 172.21.15.52:0/3625918091 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f25d41a48e0 con 0x7f25d4103d80 2024-05-27T07:25:54.504 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:54.511+0000 7f25db186640 1 -- 172.21.15.52:0/3625918091 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f25d406a230 con 0x7f25d4103d80 2024-05-27T07:25:54.505 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:54.512+0000 7f25d1ffb640 1 -- 172.21.15.52:0/3625918091 <== mon.2 v2:172.21.15.105:3300/0 4 ==== mgrmap(e 19) ==== 149669+0+0 (secure 0 0 0) 0x7f25c0004ba0 con 0x7f25d4103d80 2024-05-27T07:25:54.505 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:54.513+0000 7f25d1ffb640 1 --2- 172.21.15.52:0/3625918091 >> [v2:172.21.15.52:6800/3513075805,v1:172.21.15.52:6801/3513075805] conn(0x7f25c40b2830 0x7f25c40b4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:25:54.505 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:54.513+0000 7f25d8f22640 1 -- 172.21.15.52:0/3625918091 >> [v2:172.21.15.52:6800/3513075805,v1:172.21.15.52:6801/3513075805] conn(0x7f25c40b2830 msgr2=0x7f25c40b4d40 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.52:6800/3513075805 2024-05-27T07:25:54.505 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:54.513+0000 7f25d1ffb640 1 -- 172.21.15.52:0/3625918091 <== mon.2 v2:172.21.15.105:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9121+0+0 (secure 0 0 0) 0x7f25c00e25b0 con 0x7f25d4103d80 2024-05-27T07:25:54.505 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:54.513+0000 7f25d8f22640 1 --2- 172.21.15.52:0/3625918091 >> [v2:172.21.15.52:6800/3513075805,v1:172.21.15.52:6801/3513075805] conn(0x7f25c40b2830 0x7f25c40b4d40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-05-27T07:25:54.508 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:54.518+0000 7f25d1ffb640 1 -- 172.21.15.52:0/3625918091 <== mon.2 v2:172.21.15.105:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191963 (secure 0 0 0) 0x7f25c00a9ef0 con 0x7f25d4103d80 2024-05-27T07:25:54.689 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:25:54 smithi105 ceph-mon[35977]: from='client.? 172.21.15.52:0/2121853702' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "balance_automate", "val": "false"}]: dispatch 2024-05-27T07:25:54.690 INFO:journalctl@ceph.mgr.y.smithi105.stdout:May 27 07:25:54 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mgr-y[37393]: 2024-05-27T07:25:54.395+0000 7f1202724200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-05-27T07:25:54.690 INFO:journalctl@ceph.mgr.y.smithi105.stdout:May 27 07:25:54 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mgr-y[37393]: 2024-05-27T07:25:54.488+0000 7f1202724200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-05-27T07:25:54.704 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:54.714+0000 7f25d8f22640 1 -- 172.21.15.52:0/3625918091 >> [v2:172.21.15.52:6800/3513075805,v1:172.21.15.52:6801/3513075805] conn(0x7f25c40b2830 msgr2=0x7f25c40b4d40 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.52:6800/3513075805 2024-05-27T07:25:54.704 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:54.714+0000 7f25d8f22640 1 --2- 172.21.15.52:0/3625918091 >> [v2:172.21.15.52:6800/3513075805,v1:172.21.15.52:6801/3513075805] conn(0x7f25c40b2830 0x7f25c40b4d40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-05-27T07:25:54.729 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:25:54 smithi052 ceph-mon[31955]: from='client.? 172.21.15.52:0/2121853702' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "balance_automate", "val": "false"}]: dispatch 2024-05-27T07:25:54.729 INFO:journalctl@ceph.mgr.x.smithi052.stdout:May 27 07:25:54 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mgr-x[32184]: 2024-05-27T07:25:54.552+0000 7fd16c7b8200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-05-27T07:25:54.733 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:54.743+0000 7f25db186640 1 -- 172.21.15.52:0/3625918091 --> [v2:172.21.15.52:6800/3513075805,v1:172.21.15.52:6801/3513075805] -- mgr_command(tid 0: {"prefix": "fs subvolumegroup pin", "vol_name": "cephfs", "group_name": "qa", "pin_type": "random", "pin_setting": "0.10", "target": ["mon-mgr", ""]}) -- 0x7f25d4063100 con 0x7f25c40b2830 2024-05-27T07:25:54.746 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:25:54 smithi173 ceph-mon[37000]: from='client.? 172.21.15.52:0/2121853702' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "balance_automate", "val": "false"}]: dispatch 2024-05-27T07:25:54.746 INFO:journalctl@ceph.mgr.z.smithi173.stdout:May 27 07:25:54 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mgr-z[38891]: 2024-05-27T07:25:54.416+0000 7f41c267f200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-05-27T07:25:54.746 INFO:journalctl@ceph.mgr.z.smithi173.stdout:May 27 07:25:54 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mgr-z[38891]: 2024-05-27T07:25:54.510+0000 7f41c267f200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-05-27T07:25:55.101 INFO:journalctl@ceph.mgr.z.smithi173.stdout:May 27 07:25:54 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mgr-z[38891]: 2024-05-27T07:25:54.755+0000 7f41c267f200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-05-27T07:25:55.102 INFO:journalctl@ceph.mgr.z.smithi173.stdout:May 27 07:25:54 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mgr-z[38891]: 2024-05-27T07:25:54.889+0000 7f41c267f200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-05-27T07:25:55.102 INFO:journalctl@ceph.mgr.z.smithi173.stdout:May 27 07:25:54 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mgr-z[38891]: 2024-05-27T07:25:54.986+0000 7f41c267f200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-05-27T07:25:55.105 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:55.115+0000 7f25d8f22640 1 -- 172.21.15.52:0/3625918091 >> [v2:172.21.15.52:6800/3513075805,v1:172.21.15.52:6801/3513075805] conn(0x7f25c40b2830 msgr2=0x7f25c40b4d40 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.52:6800/3513075805 2024-05-27T07:25:55.105 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:55.115+0000 7f25d8f22640 1 --2- 172.21.15.52:0/3625918091 >> [v2:172.21.15.52:6800/3513075805,v1:172.21.15.52:6801/3513075805] conn(0x7f25c40b2830 0x7f25c40b4d40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-05-27T07:25:55.190 INFO:journalctl@ceph.mgr.y.smithi105.stdout:May 27 07:25:54 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mgr-y[37393]: 2024-05-27T07:25:54.737+0000 7f1202724200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-05-27T07:25:55.190 INFO:journalctl@ceph.mgr.y.smithi105.stdout:May 27 07:25:54 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mgr-y[37393]: 2024-05-27T07:25:54.869+0000 7f1202724200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-05-27T07:25:55.190 INFO:journalctl@ceph.mgr.y.smithi105.stdout:May 27 07:25:54 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mgr-y[37393]: 2024-05-27T07:25:54.967+0000 7f1202724200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-05-27T07:25:55.229 INFO:journalctl@ceph.mgr.x.smithi052.stdout:May 27 07:25:54 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mgr-x[32184]: 2024-05-27T07:25:54.941+0000 7fd16c7b8200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-05-27T07:25:55.229 INFO:journalctl@ceph.mgr.x.smithi052.stdout:May 27 07:25:55 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mgr-x[32184]: 2024-05-27T07:25:55.043+0000 7fd16c7b8200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-05-27T07:25:55.553 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:25:55 smithi052 ceph-mon[31955]: from='client.? 172.21.15.52:0/2121853702' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "balance_automate", "val": "false"}]': finished 2024-05-27T07:25:55.553 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:25:55 smithi052 ceph-mon[31955]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-05-27T07:25:55.553 INFO:journalctl@ceph.mgr.x.smithi052.stdout:May 27 07:25:55 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mgr-x[32184]: 2024-05-27T07:25:55.310+0000 7fd16c7b8200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-05-27T07:25:55.554 INFO:journalctl@ceph.mgr.x.smithi052.stdout:May 27 07:25:55 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mgr-x[32184]: 2024-05-27T07:25:55.457+0000 7fd16c7b8200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-05-27T07:25:55.601 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:25:55 smithi173 ceph-mon[37000]: from='client.? 172.21.15.52:0/2121853702' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "balance_automate", "val": "false"}]': finished 2024-05-27T07:25:55.602 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:25:55 smithi173 ceph-mon[37000]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-05-27T07:25:55.602 INFO:journalctl@ceph.mgr.z.smithi173.stdout:May 27 07:25:55 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mgr-z[38891]: 2024-05-27T07:25:55.316+0000 7f41c267f200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-05-27T07:25:55.602 INFO:journalctl@ceph.mgr.z.smithi173.stdout:May 27 07:25:55 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mgr-z[38891]: 2024-05-27T07:25:55.411+0000 7f41c267f200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-05-27T07:25:55.689 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:25:55 smithi105 ceph-mon[35977]: from='client.? 172.21.15.52:0/2121853702' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "balance_automate", "val": "false"}]': finished 2024-05-27T07:25:55.690 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:25:55 smithi105 ceph-mon[35977]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-05-27T07:25:55.690 INFO:journalctl@ceph.mgr.y.smithi105.stdout:May 27 07:25:55 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mgr-y[37393]: 2024-05-27T07:25:55.293+0000 7f1202724200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-05-27T07:25:55.690 INFO:journalctl@ceph.mgr.y.smithi105.stdout:May 27 07:25:55 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mgr-y[37393]: 2024-05-27T07:25:55.387+0000 7f1202724200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-05-27T07:25:55.898 INFO:journalctl@ceph.mgr.x.smithi052.stdout:May 27 07:25:55 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mgr-x[32184]: 2024-05-27T07:25:55.563+0000 7fd16c7b8200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-05-27T07:25:55.905 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:55.916+0000 7f25d8f22640 1 -- 172.21.15.52:0/3625918091 >> [v2:172.21.15.52:6800/3513075805,v1:172.21.15.52:6801/3513075805] conn(0x7f25c40b2830 msgr2=0x7f25c40b4d40 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.52:6800/3513075805 2024-05-27T07:25:55.905 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:55.916+0000 7f25d8f22640 1 --2- 172.21.15.52:0/3625918091 >> [v2:172.21.15.52:6800/3513075805,v1:172.21.15.52:6801/3513075805] conn(0x7f25c40b2830 0x7f25c40b4d40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-05-27T07:25:56.229 INFO:journalctl@ceph.mgr.x.smithi052.stdout:May 27 07:25:55 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mgr-x[32184]: 2024-05-27T07:25:55.908+0000 7fd16c7b8200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-05-27T07:25:56.229 INFO:journalctl@ceph.mgr.x.smithi052.stdout:May 27 07:25:56 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mgr-x[32184]: 2024-05-27T07:25:56.010+0000 7fd16c7b8200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-05-27T07:25:56.352 INFO:journalctl@ceph.mgr.z.smithi173.stdout:May 27 07:25:56 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mgr-z[38891]: 2024-05-27T07:25:56.017+0000 7f41c267f200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-05-27T07:25:56.352 INFO:journalctl@ceph.mgr.z.smithi173.stdout:May 27 07:25:56 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mgr-z[38891]: 2024-05-27T07:25:56.235+0000 7f41c267f200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-05-27T07:25:56.371 INFO:journalctl@ceph.mgr.y.smithi105.stdout:May 27 07:25:55 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mgr-y[37393]: 2024-05-27T07:25:55.991+0000 7f1202724200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-05-27T07:25:56.371 INFO:journalctl@ceph.mgr.y.smithi105.stdout:May 27 07:25:56 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mgr-y[37393]: 2024-05-27T07:25:56.208+0000 7f1202724200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-05-27T07:25:56.385 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:56.395+0000 7f25d1ffb640 1 -- 172.21.15.52:0/3625918091 <== mon.2 v2:172.21.15.105:3300/0 7 ==== mgrmap(e 20) ==== 149669+0+0 (secure 0 0 0) 0x7f25c00a5a80 con 0x7f25d4103d80 2024-05-27T07:25:56.515 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:25:56 smithi052 ceph-mon[31955]: Standby manager daemon y restarted 2024-05-27T07:25:56.516 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:25:56 smithi052 ceph-mon[31955]: Standby manager daemon y started 2024-05-27T07:25:56.516 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:25:56 smithi052 ceph-mon[31955]: from='mgr.? 172.21.15.105:0/4114059748' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/crt"}]: dispatch 2024-05-27T07:25:56.516 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:25:56 smithi052 ceph-mon[31955]: from='mgr.? 172.21.15.105:0/4114059748' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-05-27T07:25:56.516 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:25:56 smithi052 ceph-mon[31955]: from='mgr.? 172.21.15.105:0/4114059748' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/key"}]: dispatch 2024-05-27T07:25:56.516 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:25:56 smithi052 ceph-mon[31955]: from='mgr.? 172.21.15.105:0/4114059748' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-05-27T07:25:56.516 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:25:56 smithi052 ceph-mon[31955]: Standby manager daemon z restarted 2024-05-27T07:25:56.516 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:25:56 smithi052 ceph-mon[31955]: Standby manager daemon z started 2024-05-27T07:25:56.516 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:25:56 smithi052 ceph-mon[31955]: from='mgr.? 172.21.15.173:0/1919599564' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/crt"}]: dispatch 2024-05-27T07:25:56.516 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:25:56 smithi052 ceph-mon[31955]: from='mgr.? 172.21.15.173:0/1919599564' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-05-27T07:25:56.516 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:25:56 smithi052 ceph-mon[31955]: from='mgr.? 172.21.15.173:0/1919599564' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/key"}]: dispatch 2024-05-27T07:25:56.516 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:25:56 smithi052 ceph-mon[31955]: from='mgr.? 172.21.15.173:0/1919599564' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-05-27T07:25:56.690 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:25:56 smithi105 ceph-mon[35977]: Standby manager daemon y restarted 2024-05-27T07:25:56.690 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:25:56 smithi105 ceph-mon[35977]: Standby manager daemon y started 2024-05-27T07:25:56.690 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:25:56 smithi105 ceph-mon[35977]: from='mgr.? 172.21.15.105:0/4114059748' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/crt"}]: dispatch 2024-05-27T07:25:56.690 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:25:56 smithi105 ceph-mon[35977]: from='mgr.? 172.21.15.105:0/4114059748' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-05-27T07:25:56.690 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:25:56 smithi105 ceph-mon[35977]: from='mgr.? 172.21.15.105:0/4114059748' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/key"}]: dispatch 2024-05-27T07:25:56.690 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:25:56 smithi105 ceph-mon[35977]: from='mgr.? 172.21.15.105:0/4114059748' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-05-27T07:25:56.690 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:25:56 smithi105 ceph-mon[35977]: Standby manager daemon z restarted 2024-05-27T07:25:56.690 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:25:56 smithi105 ceph-mon[35977]: Standby manager daemon z started 2024-05-27T07:25:56.690 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:25:56 smithi105 ceph-mon[35977]: from='mgr.? 172.21.15.173:0/1919599564' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/crt"}]: dispatch 2024-05-27T07:25:56.691 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:25:56 smithi105 ceph-mon[35977]: from='mgr.? 172.21.15.173:0/1919599564' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-05-27T07:25:56.691 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:25:56 smithi105 ceph-mon[35977]: from='mgr.? 172.21.15.173:0/1919599564' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/key"}]: dispatch 2024-05-27T07:25:56.691 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:25:56 smithi105 ceph-mon[35977]: from='mgr.? 172.21.15.173:0/1919599564' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-05-27T07:25:56.852 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:25:56 smithi173 ceph-mon[37000]: Standby manager daemon y restarted 2024-05-27T07:25:56.852 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:25:56 smithi173 ceph-mon[37000]: Standby manager daemon y started 2024-05-27T07:25:56.852 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:25:56 smithi173 ceph-mon[37000]: from='mgr.? 172.21.15.105:0/4114059748' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/crt"}]: dispatch 2024-05-27T07:25:56.852 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:25:56 smithi173 ceph-mon[37000]: from='mgr.? 172.21.15.105:0/4114059748' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-05-27T07:25:56.852 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:25:56 smithi173 ceph-mon[37000]: from='mgr.? 172.21.15.105:0/4114059748' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/key"}]: dispatch 2024-05-27T07:25:56.852 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:25:56 smithi173 ceph-mon[37000]: from='mgr.? 172.21.15.105:0/4114059748' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-05-27T07:25:56.852 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:25:56 smithi173 ceph-mon[37000]: Standby manager daemon z restarted 2024-05-27T07:25:56.852 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:25:56 smithi173 ceph-mon[37000]: Standby manager daemon z started 2024-05-27T07:25:56.852 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:25:56 smithi173 ceph-mon[37000]: from='mgr.? 172.21.15.173:0/1919599564' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/crt"}]: dispatch 2024-05-27T07:25:56.853 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:25:56 smithi173 ceph-mon[37000]: from='mgr.? 172.21.15.173:0/1919599564' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-05-27T07:25:56.853 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:25:56 smithi173 ceph-mon[37000]: from='mgr.? 172.21.15.173:0/1919599564' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/key"}]: dispatch 2024-05-27T07:25:56.853 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:25:56 smithi173 ceph-mon[37000]: from='mgr.? 172.21.15.173:0/1919599564' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-05-27T07:25:56.913 INFO:journalctl@ceph.mgr.x.smithi052.stdout:May 27 07:25:56 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mgr-x[32184]: 2024-05-27T07:25:56.680+0000 7fd16c7b8200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-05-27T07:25:56.931 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:56.941+0000 7f25d1ffb640 1 -- 172.21.15.52:0/3625918091 <== mon.2 v2:172.21.15.105:3300/0 8 ==== mgrmap(e 21) ==== 149337+0+0 (secure 0 0 0) 0x7f25c00dfd50 con 0x7f25d4103d80 2024-05-27T07:25:56.931 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:56.942+0000 7f25d1ffb640 1 -- 172.21.15.52:0/3625918091 >> [v2:172.21.15.52:6800/3513075805,v1:172.21.15.52:6801/3513075805] conn(0x7f25c40b2830 msgr2=0x7f25c40b4d40 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-05-27T07:25:56.931 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:56.942+0000 7f25d1ffb640 1 --2- 172.21.15.52:0/3625918091 >> [v2:172.21.15.52:6800/3513075805,v1:172.21.15.52:6801/3513075805] conn(0x7f25c40b2830 0x7f25c40b4d40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:25:57.207 INFO:journalctl@ceph.mgr.x.smithi052.stdout:May 27 07:25:56 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mgr-x[32184]: 2024-05-27T07:25:56.922+0000 7fd16c7b8200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-05-27T07:25:57.479 INFO:journalctl@ceph.mgr.x.smithi052.stdout:May 27 07:25:57 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mgr-x[32184]: 2024-05-27T07:25:57.217+0000 7fd0cd5b1700 -1 client.0 error registering admin socket command: (17) File exists 2024-05-27T07:25:57.480 INFO:journalctl@ceph.mgr.x.smithi052.stdout:May 27 07:25:57 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mgr-x[32184]: 2024-05-27T07:25:57.217+0000 7fd0cd5b1700 -1 client.0 error registering admin socket command: (17) File exists 2024-05-27T07:25:57.480 INFO:journalctl@ceph.mgr.x.smithi052.stdout:May 27 07:25:57 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mgr-x[32184]: 2024-05-27T07:25:57.217+0000 7fd0cd5b1700 -1 client.0 error registering admin socket command: (17) File exists 2024-05-27T07:25:57.480 INFO:journalctl@ceph.mgr.x.smithi052.stdout:May 27 07:25:57 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mgr-x[32184]: 2024-05-27T07:25:57.217+0000 7fd0cd5b1700 -1 client.0 error registering admin socket command: (17) File exists 2024-05-27T07:25:57.480 INFO:journalctl@ceph.mgr.x.smithi052.stdout:May 27 07:25:57 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mgr-x[32184]: 2024-05-27T07:25:57.217+0000 7fd0cd5b1700 -1 client.0 error registering admin socket command: (17) File exists 2024-05-27T07:25:57.480 INFO:journalctl@ceph.mgr.x.smithi052.stdout:May 27 07:25:57 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mgr-x[32184]: 2024-05-27T07:25:57.218+0000 7fd0cadac700 -1 client.0 error registering admin socket command: (17) File exists 2024-05-27T07:25:57.480 INFO:journalctl@ceph.mgr.x.smithi052.stdout:May 27 07:25:57 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mgr-x[32184]: 2024-05-27T07:25:57.218+0000 7fd0cadac700 -1 client.0 error registering admin socket command: (17) File exists 2024-05-27T07:25:57.480 INFO:journalctl@ceph.mgr.x.smithi052.stdout:May 27 07:25:57 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mgr-x[32184]: 2024-05-27T07:25:57.218+0000 7fd0cadac700 -1 client.0 error registering admin socket command: (17) File exists 2024-05-27T07:25:57.480 INFO:journalctl@ceph.mgr.x.smithi052.stdout:May 27 07:25:57 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mgr-x[32184]: 2024-05-27T07:25:57.218+0000 7fd0cadac700 -1 client.0 error registering admin socket command: (17) File exists 2024-05-27T07:25:57.480 INFO:journalctl@ceph.mgr.x.smithi052.stdout:May 27 07:25:57 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mgr-x[32184]: 2024-05-27T07:25:57.218+0000 7fd0cadac700 -1 client.0 error registering admin socket command: (17) File exists 2024-05-27T07:25:57.480 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:25:57 smithi052 ceph-mon[31955]: mgrmap e20: x(active, since 11m), standbys: z, y 2024-05-27T07:25:57.481 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:25:57 smithi052 ceph-mon[31955]: Active manager daemon x restarted 2024-05-27T07:25:57.481 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:25:57 smithi052 ceph-mon[31955]: Activating manager daemon x 2024-05-27T07:25:57.481 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:25:57 smithi052 ceph-mon[31955]: osdmap e91: 12 total, 12 up, 12 in 2024-05-27T07:25:57.481 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:25:57 smithi052 ceph-mon[31955]: mgrmap e21: x(active, starting, since 0.0147974s), standbys: z, y 2024-05-27T07:25:57.481 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:25:57 smithi052 ceph-mon[31955]: from='mgr.15333 172.21.15.52:0/416389082' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-05-27T07:25:57.481 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:25:57 smithi052 ceph-mon[31955]: from='mgr.15333 172.21.15.52:0/416389082' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-05-27T07:25:57.481 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:25:57 smithi052 ceph-mon[31955]: from='mgr.15333 172.21.15.52:0/416389082' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-05-27T07:25:57.481 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:25:57 smithi052 ceph-mon[31955]: from='mgr.15333 172.21.15.52:0/416389082' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "j"}]: dispatch 2024-05-27T07:25:57.481 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:25:57 smithi052 ceph-mon[31955]: from='mgr.15333 172.21.15.52:0/416389082' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "d"}]: dispatch 2024-05-27T07:25:57.481 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:25:57 smithi052 ceph-mon[31955]: from='mgr.15333 172.21.15.52:0/416389082' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2024-05-27T07:25:57.481 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:25:57 smithi052 ceph-mon[31955]: from='mgr.15333 172.21.15.52:0/416389082' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "a"}]: dispatch 2024-05-27T07:25:57.482 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:25:57 smithi052 ceph-mon[31955]: from='mgr.15333 172.21.15.52:0/416389082' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "l"}]: dispatch 2024-05-27T07:25:57.482 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:25:57 smithi052 ceph-mon[31955]: from='mgr.15333 172.21.15.52:0/416389082' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "k"}]: dispatch 2024-05-27T07:25:57.482 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:25:57 smithi052 ceph-mon[31955]: from='mgr.15333 172.21.15.52:0/416389082' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2024-05-27T07:25:57.482 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:25:57 smithi052 ceph-mon[31955]: from='mgr.15333 172.21.15.52:0/416389082' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2024-05-27T07:25:57.482 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:25:57 smithi052 ceph-mon[31955]: from='mgr.15333 172.21.15.52:0/416389082' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "e"}]: dispatch 2024-05-27T07:25:57.482 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:25:57 smithi052 ceph-mon[31955]: from='mgr.15333 172.21.15.52:0/416389082' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "i"}]: dispatch 2024-05-27T07:25:57.482 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:25:57 smithi052 ceph-mon[31955]: from='mgr.15333 172.21.15.52:0/416389082' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2024-05-27T07:25:57.482 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:25:57 smithi052 ceph-mon[31955]: from='mgr.15333 172.21.15.52:0/416389082' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "b"}]: dispatch 2024-05-27T07:25:57.482 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:25:57 smithi052 ceph-mon[31955]: from='mgr.15333 172.21.15.52:0/416389082' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-05-27T07:25:57.482 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:25:57 smithi052 ceph-mon[31955]: from='mgr.15333 172.21.15.52:0/416389082' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "z", "id": "z"}]: dispatch 2024-05-27T07:25:57.482 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:25:57 smithi052 ceph-mon[31955]: from='mgr.15333 172.21.15.52:0/416389082' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-05-27T07:25:57.482 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:25:57 smithi052 ceph-mon[31955]: from='mgr.15333 172.21.15.52:0/416389082' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-05-27T07:25:57.482 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:25:57 smithi052 ceph-mon[31955]: from='mgr.15333 172.21.15.52:0/416389082' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-05-27T07:25:57.483 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:25:57 smithi052 ceph-mon[31955]: from='mgr.15333 172.21.15.52:0/416389082' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-05-27T07:25:57.483 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:25:57 smithi052 ceph-mon[31955]: from='mgr.15333 172.21.15.52:0/416389082' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-05-27T07:25:57.483 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:25:57 smithi052 ceph-mon[31955]: from='mgr.15333 172.21.15.52:0/416389082' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-05-27T07:25:57.483 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:25:57 smithi052 ceph-mon[31955]: from='mgr.15333 172.21.15.52:0/416389082' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-05-27T07:25:57.483 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:25:57 smithi052 ceph-mon[31955]: from='mgr.15333 172.21.15.52:0/416389082' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-05-27T07:25:57.483 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:25:57 smithi052 ceph-mon[31955]: from='mgr.15333 172.21.15.52:0/416389082' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-05-27T07:25:57.483 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:25:57 smithi052 ceph-mon[31955]: from='mgr.15333 172.21.15.52:0/416389082' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-05-27T07:25:57.483 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:25:57 smithi052 ceph-mon[31955]: from='mgr.15333 172.21.15.52:0/416389082' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-05-27T07:25:57.483 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:25:57 smithi052 ceph-mon[31955]: from='mgr.15333 172.21.15.52:0/416389082' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-05-27T07:25:57.483 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:25:57 smithi052 ceph-mon[31955]: from='mgr.15333 172.21.15.52:0/416389082' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-05-27T07:25:57.483 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:25:57 smithi052 ceph-mon[31955]: from='mgr.15333 172.21.15.52:0/416389082' entity='mgr.x' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-05-27T07:25:57.483 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:25:57 smithi052 ceph-mon[31955]: from='mgr.15333 172.21.15.52:0/416389082' entity='mgr.x' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-05-27T07:25:57.483 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:25:57 smithi052 ceph-mon[31955]: from='mgr.15333 172.21.15.52:0/416389082' entity='mgr.x' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-05-27T07:25:57.483 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:25:57 smithi052 ceph-mon[31955]: Manager daemon x is now available 2024-05-27T07:25:57.484 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:25:57 smithi052 ceph-mon[31955]: from='mgr.15333 172.21.15.52:0/416389082' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-27T07:25:57.484 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:25:57 smithi052 ceph-mon[31955]: from='mgr.15333 172.21.15.52:0/416389082' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2024-05-27T07:25:57.484 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:25:57 smithi052 ceph-mon[31955]: from='mgr.15333 172.21.15.52:0/416389082' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2024-05-27T07:25:57.690 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:25:57 smithi105 ceph-mon[35977]: mgrmap e20: x(active, since 11m), standbys: z, y 2024-05-27T07:25:57.690 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:25:57 smithi105 ceph-mon[35977]: Active manager daemon x restarted 2024-05-27T07:25:57.690 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:25:57 smithi105 ceph-mon[35977]: Activating manager daemon x 2024-05-27T07:25:57.690 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:25:57 smithi105 ceph-mon[35977]: osdmap e91: 12 total, 12 up, 12 in 2024-05-27T07:25:57.690 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:25:57 smithi105 ceph-mon[35977]: mgrmap e21: x(active, starting, since 0.0147974s), standbys: z, y 2024-05-27T07:25:57.690 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:25:57 smithi105 ceph-mon[35977]: from='mgr.15333 172.21.15.52:0/416389082' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-05-27T07:25:57.691 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:25:57 smithi105 ceph-mon[35977]: from='mgr.15333 172.21.15.52:0/416389082' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-05-27T07:25:57.691 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:25:57 smithi105 ceph-mon[35977]: from='mgr.15333 172.21.15.52:0/416389082' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-05-27T07:25:57.691 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:25:57 smithi105 ceph-mon[35977]: from='mgr.15333 172.21.15.52:0/416389082' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "j"}]: dispatch 2024-05-27T07:25:57.691 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:25:57 smithi105 ceph-mon[35977]: from='mgr.15333 172.21.15.52:0/416389082' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "d"}]: dispatch 2024-05-27T07:25:57.691 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:25:57 smithi105 ceph-mon[35977]: from='mgr.15333 172.21.15.52:0/416389082' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2024-05-27T07:25:57.691 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:25:57 smithi105 ceph-mon[35977]: from='mgr.15333 172.21.15.52:0/416389082' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "a"}]: dispatch 2024-05-27T07:25:57.691 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:25:57 smithi105 ceph-mon[35977]: from='mgr.15333 172.21.15.52:0/416389082' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "l"}]: dispatch 2024-05-27T07:25:57.691 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:25:57 smithi105 ceph-mon[35977]: from='mgr.15333 172.21.15.52:0/416389082' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "k"}]: dispatch 2024-05-27T07:25:57.691 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:25:57 smithi105 ceph-mon[35977]: from='mgr.15333 172.21.15.52:0/416389082' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2024-05-27T07:25:57.691 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:25:57 smithi105 ceph-mon[35977]: from='mgr.15333 172.21.15.52:0/416389082' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2024-05-27T07:25:57.691 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:25:57 smithi105 ceph-mon[35977]: from='mgr.15333 172.21.15.52:0/416389082' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "e"}]: dispatch 2024-05-27T07:25:57.691 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:25:57 smithi105 ceph-mon[35977]: from='mgr.15333 172.21.15.52:0/416389082' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "i"}]: dispatch 2024-05-27T07:25:57.691 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:25:57 smithi105 ceph-mon[35977]: from='mgr.15333 172.21.15.52:0/416389082' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2024-05-27T07:25:57.692 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:25:57 smithi105 ceph-mon[35977]: from='mgr.15333 172.21.15.52:0/416389082' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "b"}]: dispatch 2024-05-27T07:25:57.692 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:25:57 smithi105 ceph-mon[35977]: from='mgr.15333 172.21.15.52:0/416389082' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-05-27T07:25:57.692 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:25:57 smithi105 ceph-mon[35977]: from='mgr.15333 172.21.15.52:0/416389082' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "z", "id": "z"}]: dispatch 2024-05-27T07:25:57.692 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:25:57 smithi105 ceph-mon[35977]: from='mgr.15333 172.21.15.52:0/416389082' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-05-27T07:25:57.692 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:25:57 smithi105 ceph-mon[35977]: from='mgr.15333 172.21.15.52:0/416389082' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-05-27T07:25:57.692 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:25:57 smithi105 ceph-mon[35977]: from='mgr.15333 172.21.15.52:0/416389082' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-05-27T07:25:57.692 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:25:57 smithi105 ceph-mon[35977]: from='mgr.15333 172.21.15.52:0/416389082' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-05-27T07:25:57.692 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:25:57 smithi105 ceph-mon[35977]: from='mgr.15333 172.21.15.52:0/416389082' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-05-27T07:25:57.692 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:25:57 smithi105 ceph-mon[35977]: from='mgr.15333 172.21.15.52:0/416389082' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-05-27T07:25:57.692 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:25:57 smithi105 ceph-mon[35977]: from='mgr.15333 172.21.15.52:0/416389082' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-05-27T07:25:57.692 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:25:57 smithi105 ceph-mon[35977]: from='mgr.15333 172.21.15.52:0/416389082' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-05-27T07:25:57.692 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:25:57 smithi105 ceph-mon[35977]: from='mgr.15333 172.21.15.52:0/416389082' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-05-27T07:25:57.692 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:25:57 smithi105 ceph-mon[35977]: from='mgr.15333 172.21.15.52:0/416389082' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-05-27T07:25:57.693 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:25:57 smithi105 ceph-mon[35977]: from='mgr.15333 172.21.15.52:0/416389082' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-05-27T07:25:57.693 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:25:57 smithi105 ceph-mon[35977]: from='mgr.15333 172.21.15.52:0/416389082' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-05-27T07:25:57.693 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:25:57 smithi105 ceph-mon[35977]: from='mgr.15333 172.21.15.52:0/416389082' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-05-27T07:25:57.693 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:25:57 smithi105 ceph-mon[35977]: from='mgr.15333 172.21.15.52:0/416389082' entity='mgr.x' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-05-27T07:25:57.693 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:25:57 smithi105 ceph-mon[35977]: from='mgr.15333 172.21.15.52:0/416389082' entity='mgr.x' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-05-27T07:25:57.693 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:25:57 smithi105 ceph-mon[35977]: from='mgr.15333 172.21.15.52:0/416389082' entity='mgr.x' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-05-27T07:25:57.693 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:25:57 smithi105 ceph-mon[35977]: Manager daemon x is now available 2024-05-27T07:25:57.693 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:25:57 smithi105 ceph-mon[35977]: from='mgr.15333 172.21.15.52:0/416389082' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-27T07:25:57.693 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:25:57 smithi105 ceph-mon[35977]: from='mgr.15333 172.21.15.52:0/416389082' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2024-05-27T07:25:57.693 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:25:57 smithi105 ceph-mon[35977]: from='mgr.15333 172.21.15.52:0/416389082' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2024-05-27T07:25:57.852 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:25:57 smithi173 ceph-mon[37000]: mgrmap e20: x(active, since 11m), standbys: z, y 2024-05-27T07:25:57.852 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:25:57 smithi173 ceph-mon[37000]: Active manager daemon x restarted 2024-05-27T07:25:57.852 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:25:57 smithi173 ceph-mon[37000]: Activating manager daemon x 2024-05-27T07:25:57.852 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:25:57 smithi173 ceph-mon[37000]: osdmap e91: 12 total, 12 up, 12 in 2024-05-27T07:25:57.852 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:25:57 smithi173 ceph-mon[37000]: mgrmap e21: x(active, starting, since 0.0147974s), standbys: z, y 2024-05-27T07:25:57.853 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:25:57 smithi173 ceph-mon[37000]: from='mgr.15333 172.21.15.52:0/416389082' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-05-27T07:25:57.853 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:25:57 smithi173 ceph-mon[37000]: from='mgr.15333 172.21.15.52:0/416389082' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-05-27T07:25:57.853 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:25:57 smithi173 ceph-mon[37000]: from='mgr.15333 172.21.15.52:0/416389082' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-05-27T07:25:57.853 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:25:57 smithi173 ceph-mon[37000]: from='mgr.15333 172.21.15.52:0/416389082' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "j"}]: dispatch 2024-05-27T07:25:57.853 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:25:57 smithi173 ceph-mon[37000]: from='mgr.15333 172.21.15.52:0/416389082' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "d"}]: dispatch 2024-05-27T07:25:57.853 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:25:57 smithi173 ceph-mon[37000]: from='mgr.15333 172.21.15.52:0/416389082' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2024-05-27T07:25:57.853 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:25:57 smithi173 ceph-mon[37000]: from='mgr.15333 172.21.15.52:0/416389082' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "a"}]: dispatch 2024-05-27T07:25:57.853 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:25:57 smithi173 ceph-mon[37000]: from='mgr.15333 172.21.15.52:0/416389082' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "l"}]: dispatch 2024-05-27T07:25:57.854 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:25:57 smithi173 ceph-mon[37000]: from='mgr.15333 172.21.15.52:0/416389082' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "k"}]: dispatch 2024-05-27T07:25:57.854 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:25:57 smithi173 ceph-mon[37000]: from='mgr.15333 172.21.15.52:0/416389082' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2024-05-27T07:25:57.854 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:25:57 smithi173 ceph-mon[37000]: from='mgr.15333 172.21.15.52:0/416389082' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2024-05-27T07:25:57.854 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:25:57 smithi173 ceph-mon[37000]: from='mgr.15333 172.21.15.52:0/416389082' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "e"}]: dispatch 2024-05-27T07:25:57.854 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:25:57 smithi173 ceph-mon[37000]: from='mgr.15333 172.21.15.52:0/416389082' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "i"}]: dispatch 2024-05-27T07:25:57.854 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:25:57 smithi173 ceph-mon[37000]: from='mgr.15333 172.21.15.52:0/416389082' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2024-05-27T07:25:57.854 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:25:57 smithi173 ceph-mon[37000]: from='mgr.15333 172.21.15.52:0/416389082' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "b"}]: dispatch 2024-05-27T07:25:57.854 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:25:57 smithi173 ceph-mon[37000]: from='mgr.15333 172.21.15.52:0/416389082' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-05-27T07:25:57.854 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:25:57 smithi173 ceph-mon[37000]: from='mgr.15333 172.21.15.52:0/416389082' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "z", "id": "z"}]: dispatch 2024-05-27T07:25:57.854 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:25:57 smithi173 ceph-mon[37000]: from='mgr.15333 172.21.15.52:0/416389082' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-05-27T07:25:57.855 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:25:57 smithi173 ceph-mon[37000]: from='mgr.15333 172.21.15.52:0/416389082' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-05-27T07:25:57.855 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:25:57 smithi173 ceph-mon[37000]: from='mgr.15333 172.21.15.52:0/416389082' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-05-27T07:25:57.855 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:25:57 smithi173 ceph-mon[37000]: from='mgr.15333 172.21.15.52:0/416389082' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-05-27T07:25:57.855 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:25:57 smithi173 ceph-mon[37000]: from='mgr.15333 172.21.15.52:0/416389082' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-05-27T07:25:57.855 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:25:57 smithi173 ceph-mon[37000]: from='mgr.15333 172.21.15.52:0/416389082' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-05-27T07:25:57.855 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:25:57 smithi173 ceph-mon[37000]: from='mgr.15333 172.21.15.52:0/416389082' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-05-27T07:25:57.855 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:25:57 smithi173 ceph-mon[37000]: from='mgr.15333 172.21.15.52:0/416389082' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-05-27T07:25:57.855 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:25:57 smithi173 ceph-mon[37000]: from='mgr.15333 172.21.15.52:0/416389082' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-05-27T07:25:57.855 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:25:57 smithi173 ceph-mon[37000]: from='mgr.15333 172.21.15.52:0/416389082' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-05-27T07:25:57.855 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:25:57 smithi173 ceph-mon[37000]: from='mgr.15333 172.21.15.52:0/416389082' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-05-27T07:25:57.855 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:25:57 smithi173 ceph-mon[37000]: from='mgr.15333 172.21.15.52:0/416389082' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-05-27T07:25:57.855 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:25:57 smithi173 ceph-mon[37000]: from='mgr.15333 172.21.15.52:0/416389082' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-05-27T07:25:57.855 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:25:57 smithi173 ceph-mon[37000]: from='mgr.15333 172.21.15.52:0/416389082' entity='mgr.x' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-05-27T07:25:57.856 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:25:57 smithi173 ceph-mon[37000]: from='mgr.15333 172.21.15.52:0/416389082' entity='mgr.x' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-05-27T07:25:57.856 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:25:57 smithi173 ceph-mon[37000]: from='mgr.15333 172.21.15.52:0/416389082' entity='mgr.x' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-05-27T07:25:57.856 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:25:57 smithi173 ceph-mon[37000]: Manager daemon x is now available 2024-05-27T07:25:57.856 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:25:57 smithi173 ceph-mon[37000]: from='mgr.15333 172.21.15.52:0/416389082' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-27T07:25:57.856 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:25:57 smithi173 ceph-mon[37000]: from='mgr.15333 172.21.15.52:0/416389082' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2024-05-27T07:25:57.856 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:25:57 smithi173 ceph-mon[37000]: from='mgr.15333 172.21.15.52:0/416389082' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2024-05-27T07:25:57.935 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:57.943+0000 7f25d1ffb640 1 -- 172.21.15.52:0/3625918091 <== mon.2 v2:172.21.15.105:3300/0 9 ==== mgrmap(e 22) ==== 149464+0+0 (secure 0 0 0) 0x7f25c0065bf0 con 0x7f25d4103d80 2024-05-27T07:25:57.935 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:57.944+0000 7f25d1ffb640 1 --2- 172.21.15.52:0/3625918091 >> [v2:172.21.15.52:6800/1822318384,v1:172.21.15.52:6801/1822318384] conn(0x7f25c4036430 0x7f25c4036840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:25:57.935 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:57.944+0000 7f25d1ffb640 1 -- 172.21.15.52:0/3625918091 --> [v2:172.21.15.52:6800/1822318384,v1:172.21.15.52:6801/1822318384] -- mgr_command(tid 0: {"prefix": "fs subvolumegroup pin", "vol_name": "cephfs", "group_name": "qa", "pin_type": "random", "pin_setting": "0.10", "target": ["mon-mgr", ""]}) -- 0x7f25d4063100 con 0x7f25c4036430 2024-05-27T07:25:57.938 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:57.948+0000 7f25d8f22640 1 --2- 172.21.15.52:0/3625918091 >> [v2:172.21.15.52:6800/1822318384,v1:172.21.15.52:6801/1822318384] conn(0x7f25c4036430 0x7f25c4036840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:25:57.942 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:57.952+0000 7f25d8f22640 1 --2- 172.21.15.52:0/3625918091 >> [v2:172.21.15.52:6800/1822318384,v1:172.21.15.52:6801/1822318384] conn(0x7f25c4036430 0x7f25c4036840 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7f25c800a010 tx=0x7f25c8005f90 comp rx=0 tx=0).ready entity=mgr.15333 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:25:57.962 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:57.972+0000 7f25d1ffb640 1 -- 172.21.15.52:0/3625918091 <== mgr.15333 v2:172.21.15.52:6800/1822318384 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+3 (secure 0 0 0) 0x7f25d4063100 con 0x7f25c4036430 2024-05-27T07:25:57.962 INFO:teuthology.orchestra.run.smithi052.stdout:{} 2024-05-27T07:25:57.965 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:57.975+0000 7f25db186640 1 -- 172.21.15.52:0/3625918091 >> [v2:172.21.15.52:6800/1822318384,v1:172.21.15.52:6801/1822318384] conn(0x7f25c4036430 msgr2=0x7f25c4036840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:25:57.965 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:57.975+0000 7f25db186640 1 --2- 172.21.15.52:0/3625918091 >> [v2:172.21.15.52:6800/1822318384,v1:172.21.15.52:6801/1822318384] conn(0x7f25c4036430 0x7f25c4036840 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7f25c800a010 tx=0x7f25c8005f90 comp rx=0 tx=0).stop 2024-05-27T07:25:57.965 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:57.975+0000 7f25db186640 1 -- 172.21.15.52:0/3625918091 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f25d4103d80 msgr2=0x7f25d419caa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:25:57.965 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:57.975+0000 7f25db186640 1 --2- 172.21.15.52:0/3625918091 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f25d4103d80 0x7f25d419caa0 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f25c000e9f0 tx=0x7f25c000eeb0 comp rx=0 tx=0).stop 2024-05-27T07:25:57.965 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:57.975+0000 7f25d9723640 1 -- 172.21.15.52:0/3625918091 reap_dead start 2024-05-27T07:25:57.965 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:57.975+0000 7f25db186640 1 -- 172.21.15.52:0/3625918091 shutdown_connections 2024-05-27T07:25:57.965 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:57.975+0000 7f25db186640 1 -- 172.21.15.52:0/3625918091 >> 172.21.15.52:0/3625918091 conn(0x7f25d40fdd30 msgr2=0x7f25d4100150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:25:57.965 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:57.975+0000 7f25db186640 1 -- 172.21.15.52:0/3625918091 shutdown_connections 2024-05-27T07:25:57.965 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:57.975+0000 7f25db186640 1 -- 172.21.15.52:0/3625918091 wait complete. 2024-05-27T07:25:57.976 INFO:teuthology.run_tasks:Running task check-counter... 2024-05-27T07:25:57.988 INFO:teuthology.run_tasks:Running task fwd_scrub... 2024-05-27T07:25:57.998 DEBUG:teuthology.orchestra.run.smithi052:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-05-27T07:25:58.121 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:58.131+0000 7f703cfce640 1 -- 172.21.15.52:0/3306552316 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f7038108a40 msgr2=0x7f703810ae60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:25:58.121 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:58.131+0000 7f703cfce640 1 --2- 172.21.15.52:0/3306552316 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f7038108a40 0x7f703810ae60 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f7030009900 tx=0x7f703002a410 comp rx=0 tx=0).stop 2024-05-27T07:25:58.121 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:58.131+0000 7f703cfce640 1 -- 172.21.15.52:0/3306552316 shutdown_connections 2024-05-27T07:25:58.121 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:58.131+0000 7f703cfce640 1 --2- 172.21.15.52:0/3306552316 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f7038108a40 0x7f703810ae60 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:25:58.121 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:58.131+0000 7f703cfce640 1 --2- 172.21.15.52:0/3306552316 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f7038103d40 0x7f7038104150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:25:58.122 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:58.131+0000 7f703cfce640 1 --2- 172.21.15.52:0/3306552316 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f7038102b00 0x7f7038103800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:25:58.122 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:58.131+0000 7f703cfce640 1 -- 172.21.15.52:0/3306552316 >> 172.21.15.52:0/3306552316 conn(0x7f70380fdcf0 msgr2=0x7f70381001a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:25:58.122 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:58.131+0000 7f703cfce640 1 -- 172.21.15.52:0/3306552316 shutdown_connections 2024-05-27T07:25:58.122 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:58.131+0000 7f703cfce640 1 -- 172.21.15.52:0/3306552316 wait complete. 2024-05-27T07:25:58.124 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:58.132+0000 7f703cfce640 1 Processor -- start 2024-05-27T07:25:58.124 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:58.132+0000 7f703cfce640 1 -- start start 2024-05-27T07:25:58.124 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:58.132+0000 7f703cfce640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f7038102b00 0x7f703819a360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:25:58.124 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:58.132+0000 7f703cfce640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f7038103d40 0x7f703819a8a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:25:58.124 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:58.132+0000 7f703cfce640 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f7038108a40 0x7f70381a1830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:25:58.124 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:58.132+0000 7f703cfce640 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x7f703810e630 con 0x7f7038108a40 2024-05-27T07:25:58.124 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:58.132+0000 7f703cfce640 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f703810e4b0 con 0x7f7038102b00 2024-05-27T07:25:58.124 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:58.132+0000 7f703cfce640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f703810e7b0 con 0x7f7038103d40 2024-05-27T07:25:58.124 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:58.132+0000 7f7036d9d640 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f7038108a40 0x7f70381a1830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:25:58.125 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:58.132+0000 7f703659c640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f7038102b00 0x7f703819a360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:25:58.125 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:58.132+0000 7f7036d9d640 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f7038108a40 0x7f70381a1830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.52:59374/0 (socket says 172.21.15.52:59374) 2024-05-27T07:25:58.125 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:58.132+0000 7f7036d9d640 1 -- 172.21.15.52:0/2235682404 learned_addr learned my addr 172.21.15.52:0/2235682404 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-05-27T07:25:58.125 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:58.133+0000 7f7035d9b640 1 --2- 172.21.15.52:0/2235682404 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f7038103d40 0x7f703819a8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:25:58.125 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:58.133+0000 7f7036d9d640 1 -- 172.21.15.52:0/2235682404 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f7038103d40 msgr2=0x7f703819a8a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:25:58.125 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:58.133+0000 7f7036d9d640 1 --2- 172.21.15.52:0/2235682404 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f7038103d40 0x7f703819a8a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:25:58.125 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:58.133+0000 7f7036d9d640 1 -- 172.21.15.52:0/2235682404 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f7038102b00 msgr2=0x7f703819a360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:25:58.125 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:58.133+0000 7f7036d9d640 1 --2- 172.21.15.52:0/2235682404 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f7038102b00 0x7f703819a360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:25:58.125 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:58.133+0000 7f7036d9d640 1 -- 172.21.15.52:0/2235682404 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f70381a1ea0 con 0x7f7038108a40 2024-05-27T07:25:58.125 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:58.133+0000 7f703659c640 1 --2- 172.21.15.52:0/2235682404 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f7038102b00 0x7f703819a360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-27T07:25:58.125 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:58.133+0000 7f7036d9d640 1 --2- 172.21.15.52:0/2235682404 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f7038108a40 0x7f70381a1830 secure :-1 s=READY pgs=406 cs=0 l=1 rev1=1 crypto rx=0x7f7030009230 tx=0x7f70300042f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:25:58.125 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:58.133+0000 7f70277fe640 1 -- 172.21.15.52:0/2235682404 <== mon.0 v2:172.21.15.52:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f703003f030 con 0x7f7038108a40 2024-05-27T07:25:58.126 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:58.133+0000 7f70277fe640 1 -- 172.21.15.52:0/2235682404 <== mon.0 v2:172.21.15.52:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f7030031910 con 0x7f7038108a40 2024-05-27T07:25:58.126 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:58.133+0000 7f70277fe640 1 -- 172.21.15.52:0/2235682404 <== mon.0 v2:172.21.15.52:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7030035070 con 0x7f7038108a40 2024-05-27T07:25:58.126 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:58.133+0000 7f703cfce640 1 -- 172.21.15.52:0/2235682404 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f70381a2130 con 0x7f7038108a40 2024-05-27T07:25:58.126 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:58.133+0000 7f703cfce640 1 -- 172.21.15.52:0/2235682404 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f70381a2670 con 0x7f7038108a40 2024-05-27T07:25:58.126 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:58.134+0000 7f703cfce640 1 -- 172.21.15.52:0/2235682404 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f70381a28d0 con 0x7f7038108a40 2024-05-27T07:25:58.127 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:58.136+0000 7f70277fe640 1 -- 172.21.15.52:0/2235682404 <== mon.0 v2:172.21.15.52:3300/0 4 ==== mgrmap(e 22) ==== 149464+0+0 (secure 0 0 0) 0x7f7030042030 con 0x7f7038108a40 2024-05-27T07:25:58.127 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:58.136+0000 7f70277fe640 1 --2- 172.21.15.52:0/2235682404 >> [v2:172.21.15.52:6800/1822318384,v1:172.21.15.52:6801/1822318384] conn(0x7f70200b2660 0x7f70200b4b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:25:58.127 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:58.137+0000 7f703659c640 1 --2- 172.21.15.52:0/2235682404 >> [v2:172.21.15.52:6800/1822318384,v1:172.21.15.52:6801/1822318384] conn(0x7f70200b2660 0x7f70200b4b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:25:58.127 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:58.137+0000 7f70277fe640 1 -- 172.21.15.52:0/2235682404 <== mon.0 v2:172.21.15.52:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9379+0+0 (secure 0 0 0) 0x7f7030102ab0 con 0x7f7038108a40 2024-05-27T07:25:58.127 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:58.137+0000 7f703659c640 1 --2- 172.21.15.52:0/2235682404 >> [v2:172.21.15.52:6800/1822318384,v1:172.21.15.52:6801/1822318384] conn(0x7f70200b2660 0x7f70200b4b70 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f7028005fd0 tx=0x7f702800b390 comp rx=0 tx=0).ready entity=mgr.15333 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:25:58.131 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:58.141+0000 7f70277fe640 1 -- 172.21.15.52:0/2235682404 <== mon.0 v2:172.21.15.52:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191963 (secure 0 0 0) 0x7f70300ca640 con 0x7f7038108a40 2024-05-27T07:25:58.229 INFO:journalctl@ceph.mgr.x.smithi052.stdout:May 27 07:25:57 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mgr-x[32184]: 2024-05-27T07:25:57.960+0000 7fd0c7da6700 -1 client.0 error registering admin socket command: (17) File exists 2024-05-27T07:25:58.229 INFO:journalctl@ceph.mgr.x.smithi052.stdout:May 27 07:25:57 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mgr-x[32184]: 2024-05-27T07:25:57.960+0000 7fd0c7da6700 -1 client.0 error registering admin socket command: (17) File exists 2024-05-27T07:25:58.229 INFO:journalctl@ceph.mgr.x.smithi052.stdout:May 27 07:25:57 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mgr-x[32184]: 2024-05-27T07:25:57.960+0000 7fd0c7da6700 -1 client.0 error registering admin socket command: (17) File exists 2024-05-27T07:25:58.229 INFO:journalctl@ceph.mgr.x.smithi052.stdout:May 27 07:25:57 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mgr-x[32184]: 2024-05-27T07:25:57.960+0000 7fd0c7da6700 -1 client.0 error registering admin socket command: (17) File exists 2024-05-27T07:25:58.229 INFO:journalctl@ceph.mgr.x.smithi052.stdout:May 27 07:25:57 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mgr-x[32184]: 2024-05-27T07:25:57.960+0000 7fd0c7da6700 -1 client.0 error registering admin socket command: (17) File exists 2024-05-27T07:25:58.324 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:58.334+0000 7f703cfce640 1 -- 172.21.15.52:0/2235682404 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f70381a2aa0 con 0x7f7038108a40 2024-05-27T07:25:58.326 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:58.336+0000 7f70277fe640 1 -- 172.21.15.52:0/2235682404 <== mon.0 v2:172.21.15.52:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v91) ==== 74+0+25078 (secure 0 0 0) 0x7f70300c9f40 con 0x7f7038108a40 2024-05-27T07:25:58.326 INFO:teuthology.orchestra.run.smithi052.stdout: 2024-05-27T07:25:58.326 INFO:teuthology.orchestra.run.smithi052.stdout:{"epoch":91,"fsid":"73df2636-1bf8-11ef-bc9b-c7b262605968","created":"2024-05-27T07:13:34.738032+0000","modified":"2024-05-27T07:25:56.925109+0000","last_up_change":"2024-05-27T07:20:45.777597+0000","last_in_change":"2024-05-27T07:20:25.314823+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-05-27T07:16:57.644034+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-05-27T07:21:43.917748+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"79","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-05-27T07:21:47.341876+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"79","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-05-27T07:21:57.707611+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"86","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"347b8a89-9dbe-4848-83b0-fd0e9374f33d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":82,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.52:6802","nonce":1626508899},{"type":"v1","addr":"172.21.15.52:6803","nonce":1626508899}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.52:6804","nonce":1626508899},{"type":"v1","addr":"172.21.15.52:6805","nonce":1626508899}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.52:6808","nonce":1626508899},{"type":"v1","addr":"172.21.15.52:6809","nonce":1626508899}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.52:6806","nonce":1626508899},{"type":"v1","addr":"172.21.15.52:6807","nonce":1626508899}]},"public_addr":"172.21.15.52:6803/1626508899","cluster_addr":"172.21.15.52:6805/1626508899","heartbeat_back_addr":"172.21.15.52:6809/1626508899","heartbeat_front_addr":"172.21.15.52:6807/1626508899","state":["exists","up"]},{"osd":1,"uuid":"7a3623d4-e087-40cb-9336-d4a5c52203d4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6800","nonce":2900093195},{"type":"v1","addr":"172.21.15.105:6801","nonce":2900093195}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6802","nonce":2900093195},{"type":"v1","addr":"172.21.15.105:6803","nonce":2900093195}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6806","nonce":2900093195},{"type":"v1","addr":"172.21.15.105:6807","nonce":2900093195}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6804","nonce":2900093195},{"type":"v1","addr":"172.21.15.105:6805","nonce":2900093195}]},"public_addr":"172.21.15.105:6801/2900093195","cluster_addr":"172.21.15.105:6803/2900093195","heartbeat_back_addr":"172.21.15.105:6807/2900093195","heartbeat_front_addr":"172.21.15.105:6805/2900093195","state":["exists","up"]},{"osd":2,"uuid":"41a9df5b-882a-47da-bb26-7f4d26ccf6c1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6800","nonce":2251189314},{"type":"v1","addr":"172.21.15.173:6801","nonce":2251189314}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6802","nonce":2251189314},{"type":"v1","addr":"172.21.15.173:6803","nonce":2251189314}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6806","nonce":2251189314},{"type":"v1","addr":"172.21.15.173:6807","nonce":2251189314}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6804","nonce":2251189314},{"type":"v1","addr":"172.21.15.173:6805","nonce":2251189314}]},"public_addr":"172.21.15.173:6801/2251189314","cluster_addr":"172.21.15.173:6803/2251189314","heartbeat_back_addr":"172.21.15.173:6807/2251189314","heartbeat_front_addr":"172.21.15.173:6805/2251189314","state":["exists","up"]},{"osd":3,"uuid":"f1245439-42a5-46c5-a998-e0a78f9df3fb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":26,"up_thru":82,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.52:6810","nonce":57920794},{"type":"v1","addr":"172.21.15.52:6811","nonce":57920794}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.52:6812","nonce":57920794},{"type":"v1","addr":"172.21.15.52:6813","nonce":57920794}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.52:6816","nonce":57920794},{"type":"v1","addr":"172.21.15.52:6817","nonce":57920794}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.52:6814","nonce":57920794},{"type":"v1","addr":"172.21.15.52:6815","nonce":57920794}]},"public_addr":"172.21.15.52:6811/57920794","cluster_addr":"172.21.15.52:6813/57920794","heartbeat_back_addr":"172.21.15.52:6817/57920794","heartbeat_front_addr":"172.21.15.52:6815/57920794","state":["exists","up"]},{"osd":4,"uuid":"2592d1cc-1726-4359-83d8-a2e6314b54ec","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":32,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6808","nonce":3441134818},{"type":"v1","addr":"172.21.15.105:6809","nonce":3441134818}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6810","nonce":3441134818},{"type":"v1","addr":"172.21.15.105:6811","nonce":3441134818}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6814","nonce":3441134818},{"type":"v1","addr":"172.21.15.105:6815","nonce":3441134818}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6812","nonce":3441134818},{"type":"v1","addr":"172.21.15.105:6813","nonce":3441134818}]},"public_addr":"172.21.15.105:6809/3441134818","cluster_addr":"172.21.15.105:6811/3441134818","heartbeat_back_addr":"172.21.15.105:6815/3441134818","heartbeat_front_addr":"172.21.15.105:6813/3441134818","state":["exists","up"]},{"osd":5,"uuid":"27e98ee2-4a92-4972-b2a3-959d0ed90981","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":37,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6808","nonce":2989239729},{"type":"v1","addr":"172.21.15.173:6809","nonce":2989239729}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6810","nonce":2989239729},{"type":"v1","addr":"172.21.15.173:6811","nonce":2989239729}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6814","nonce":2989239729},{"type":"v1","addr":"172.21.15.173:6815","nonce":2989239729}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6812","nonce":2989239729},{"type":"v1","addr":"172.21.15.173:6813","nonce":2989239729}]},"public_addr":"172.21.15.173:6809/2989239729","cluster_addr":"172.21.15.173:6811/2989239729","heartbeat_back_addr":"172.21.15.173:6815/2989239729","heartbeat_front_addr":"172.21.15.173:6813/2989239729","state":["exists","up"]},{"osd":6,"uuid":"2b0ec0cb-f438-4b2c-8642-0778ed374e9a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":43,"up_thru":82,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.52:6818","nonce":4244423065},{"type":"v1","addr":"172.21.15.52:6819","nonce":4244423065}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.52:6820","nonce":4244423065},{"type":"v1","addr":"172.21.15.52:6821","nonce":4244423065}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.52:6824","nonce":4244423065},{"type":"v1","addr":"172.21.15.52:6825","nonce":4244423065}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.52:6822","nonce":4244423065},{"type":"v1","addr":"172.21.15.52:6823","nonce":4244423065}]},"public_addr":"172.21.15.52:6819/4244423065","cluster_addr":"172.21.15.52:6821/4244423065","heartbeat_back_addr":"172.21.15.52:6825/4244423065","heartbeat_front_addr":"172.21.15.52:6823/4244423065","state":["exists","up"]},{"osd":7,"uuid":"cb2a7a6b-8015-43ea-865a-be144e45c51a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":48,"up_thru":82,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6816","nonce":2063611552},{"type":"v1","addr":"172.21.15.105:6817","nonce":2063611552}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6818","nonce":2063611552},{"type":"v1","addr":"172.21.15.105:6819","nonce":2063611552}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6822","nonce":2063611552},{"type":"v1","addr":"172.21.15.105:6823","nonce":2063611552}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6820","nonce":2063611552},{"type":"v1","addr":"172.21.15.105:6821","nonce":2063611552}]},"public_addr":"172.21.15.105:6817/2063611552","cluster_addr":"172.21.15.105:6819/2063611552","heartbeat_back_addr":"172.21.15.105:6823/2063611552","heartbeat_front_addr":"172.21.15.105:6821/2063611552","state":["exists","up"]},{"osd":8,"uuid":"c2fc565d-a0b9-4396-a209-bd884cddcfe3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":54,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6816","nonce":4131399852},{"type":"v1","addr":"172.21.15.173:6817","nonce":4131399852}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6818","nonce":4131399852},{"type":"v1","addr":"172.21.15.173:6819","nonce":4131399852}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6822","nonce":4131399852},{"type":"v1","addr":"172.21.15.173:6823","nonce":4131399852}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6820","nonce":4131399852},{"type":"v1","addr":"172.21.15.173:6821","nonce":4131399852}]},"public_addr":"172.21.15.173:6817/4131399852","cluster_addr":"172.21.15.173:6819/4131399852","heartbeat_back_addr":"172.21.15.173:6823/4131399852","heartbeat_front_addr":"172.21.15.173:6821/4131399852","state":["exists","up"]},{"osd":9,"uuid":"78fa7f17-1dad-4262-be40-31fa82e75520","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":60,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.52:6826","nonce":479119778},{"type":"v1","addr":"172.21.15.52:6827","nonce":479119778}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.52:6828","nonce":479119778},{"type":"v1","addr":"172.21.15.52:6829","nonce":479119778}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.52:6832","nonce":479119778},{"type":"v1","addr":"172.21.15.52:6833","nonce":479119778}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.52:6830","nonce":479119778},{"type":"v1","addr":"172.21.15.52:6831","nonce":479119778}]},"public_addr":"172.21.15.52:6827/479119778","cluster_addr":"172.21.15.52:6829/479119778","heartbeat_back_addr":"172.21.15.52:6833/479119778","heartbeat_front_addr":"172.21.15.52:6831/479119778","state":["exists","up"]},{"osd":10,"uuid":"e6f8e753-e15c-45b5-8ee9-d50f384559f9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":65,"up_thru":76,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6824","nonce":4032386628},{"type":"v1","addr":"172.21.15.105:6825","nonce":4032386628}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6826","nonce":4032386628},{"type":"v1","addr":"172.21.15.105:6827","nonce":4032386628}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6830","nonce":4032386628},{"type":"v1","addr":"172.21.15.105:6831","nonce":4032386628}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6828","nonce":4032386628},{"type":"v1","addr":"172.21.15.105:6829","nonce":4032386628}]},"public_addr":"172.21.15.105:6825/4032386628","cluster_addr":"172.21.15.105:6827/4032386628","heartbeat_back_addr":"172.21.15.105:6831/4032386628","heartbeat_front_addr":"172.21.15.105:6829/4032386628","state":["exists","up"]},{"osd":11,"uuid":"164572bf-67ed-4e8e-b32d-d7a2e8da607b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":71,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6824","nonce":1816308401},{"type":"v1","addr":"172.21.15.173:6825","nonce":1816308401}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6826","nonce":1816308401},{"type":"v1","addr":"172.21.15.173:6827","nonce":1816308401}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6830","nonce":1816308401},{"type":"v1","addr":"172.21.15.173:6831","nonce":1816308401}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6828","nonce":1816308401},{"type":"v1","addr":"172.21.15.173:6829","nonce":1816308401}]},"public_addr":"172.21.15.173:6825/1816308401","cluster_addr":"172.21.15.173:6827/1816308401","heartbeat_back_addr":"172.21.15.173:6831/1816308401","heartbeat_front_addr":"172.21.15.173:6829/1816308401","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-27T07:16:06.279220+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-27T07:16:29.668167+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-27T07:16:53.903808+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-27T07:17:18.787372+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-27T07:17:43.480784+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-27T07:18:08.816030+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-27T07:18:34.076148+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-27T07:19:00.283468+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-27T07:19:25.403023+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-27T07:19:52.139173+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-27T07:20:17.530304+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-27T07:20:43.665632+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.8","mappings":[{"from":2,"to":11}]},{"pgid":"2.f","mappings":[{"from":8,"to":11}]},{"pgid":"2.10","mappings":[{"from":2,"to":4}]},{"pgid":"3.0","mappings":[{"from":5,"to":9}]},{"pgid":"3.1b","mappings":[{"from":4,"to":8}]},{"pgid":"3.2c","mappings":[{"from":4,"to":9}]},{"pgid":"3.3c","mappings":[{"from":4,"to":9}]},{"pgid":"4.e","mappings":[{"from":9,"to":11}]},{"pgid":"4.12","mappings":[{"from":5,"to":10}]},{"pgid":"4.19","mappings":[{"from":9,"to":8}]},{"pgid":"4.1f","mappings":[{"from":9,"to":8}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.52:0/3430810618":"2024-05-28T07:25:56.925056+0000","172.21.15.52:0/2723769517":"2024-05-28T07:25:56.925056+0000","172.21.15.52:0/262308791":"2024-05-28T07:25:56.925056+0000","172.21.15.52:6801/1631683163":"2024-05-28T07:14:29.544633+0000","172.21.15.52:6801/3513075805":"2024-05-28T07:25:56.925056+0000","172.21.15.52:0/2776144659":"2024-05-28T07:14:29.544633+0000","172.21.15.52:0/3577886285":"2024-05-28T07:25:56.925056+0000","172.21.15.52:6800/1631683163":"2024-05-28T07:14:29.544633+0000","172.21.15.52:0/3675529097":"2024-05-28T07:14:01.960550+0000","172.21.15.52:0/727808107":"2024-05-28T07:14:01.960550+0000","172.21.15.52:0/3839436833":"2024-05-28T07:14:29.544633+0000","172.21.15.52:6800/587662512":"2024-05-28T07:14:01.960550+0000","172.21.15.52:6800/3513075805":"2024-05-28T07:25:56.925056+0000","172.21.15.52:6801/587662512":"2024-05-28T07:14:01.960550+0000","172.21.15.52:0/853867465":"2024-05-28T07:14:01.960550+0000","172.21.15.52:0/3480918279":"2024-05-28T07:14:29.544633+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-05-27T07:25:58.328 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:58.338+0000 7f703cfce640 1 -- 172.21.15.52:0/2235682404 >> [v2:172.21.15.52:6800/1822318384,v1:172.21.15.52:6801/1822318384] conn(0x7f70200b2660 msgr2=0x7f70200b4b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:25:58.328 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:58.338+0000 7f703cfce640 1 --2- 172.21.15.52:0/2235682404 >> [v2:172.21.15.52:6800/1822318384,v1:172.21.15.52:6801/1822318384] conn(0x7f70200b2660 0x7f70200b4b70 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f7028005fd0 tx=0x7f702800b390 comp rx=0 tx=0).stop 2024-05-27T07:25:58.328 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:58.338+0000 7f703cfce640 1 -- 172.21.15.52:0/2235682404 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f7038108a40 msgr2=0x7f70381a1830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:25:58.328 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:58.338+0000 7f703cfce640 1 --2- 172.21.15.52:0/2235682404 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f7038108a40 0x7f70381a1830 secure :-1 s=READY pgs=406 cs=0 l=1 rev1=1 crypto rx=0x7f7030009230 tx=0x7f70300042f0 comp rx=0 tx=0).stop 2024-05-27T07:25:58.328 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:58.338+0000 7f703cfce640 1 -- 172.21.15.52:0/2235682404 shutdown_connections 2024-05-27T07:25:58.329 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:58.338+0000 7f703cfce640 1 --2- 172.21.15.52:0/2235682404 >> [v2:172.21.15.52:6800/1822318384,v1:172.21.15.52:6801/1822318384] conn(0x7f70200b2660 0x7f70200b4b70 unknown :-1 s=CLOSED pgs=36 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:25:58.329 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:58.338+0000 7f703cfce640 1 --2- 172.21.15.52:0/2235682404 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f7038108a40 0x7f70381a1830 unknown :-1 s=CLOSED pgs=406 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:25:58.329 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:58.338+0000 7f703cfce640 1 --2- 172.21.15.52:0/2235682404 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f7038103d40 0x7f703819a8a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:25:58.329 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:58.338+0000 7f703cfce640 1 --2- 172.21.15.52:0/2235682404 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f7038102b00 0x7f703819a360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:25:58.329 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:58.338+0000 7f703cfce640 1 -- 172.21.15.52:0/2235682404 >> 172.21.15.52:0/2235682404 conn(0x7f70380fdcf0 msgr2=0x7f7038109fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:25:58.329 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:58.338+0000 7f703cfce640 1 -- 172.21.15.52:0/2235682404 shutdown_connections 2024-05-27T07:25:58.329 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:58.338+0000 7f703cfce640 1 -- 172.21.15.52:0/2235682404 wait complete. 2024-05-27T07:25:58.340 INFO:tasks.fwd_scrub.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-05-27T07:16:57.644034+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-05-27T07:21:43.917748+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '79', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.8799999952316284, 'score_stable': 1.8799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.8799999952316284, 'raw_score_stable': 1.8799999952316284, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-05-27T07:21:47.341876+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '79', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2024-05-27T07:21:57.707611+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '86', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2024-05-27T07:25:58.341 DEBUG:teuthology.orchestra.run.smithi052:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-05-27T07:25:58.463 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:58.473+0000 7faf09405640 1 -- 172.21.15.52:0/1071185279 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7faf04103d40 msgr2=0x7faf04104150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:25:58.463 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:58.473+0000 7faf09405640 1 --2- 172.21.15.52:0/1071185279 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7faf04103d40 0x7faf04104150 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7faef8009900 tx=0x7faef802a410 comp rx=0 tx=0).stop 2024-05-27T07:25:58.463 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:58.473+0000 7faf09405640 1 -- 172.21.15.52:0/1071185279 shutdown_connections 2024-05-27T07:25:58.463 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:58.473+0000 7faf09405640 1 --2- 172.21.15.52:0/1071185279 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7faf04108a40 0x7faf0410ae60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:25:58.463 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:58.473+0000 7faf09405640 1 --2- 172.21.15.52:0/1071185279 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7faf04103d40 0x7faf04104150 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:25:58.464 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:58.473+0000 7faf09405640 1 --2- 172.21.15.52:0/1071185279 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7faf04102b00 0x7faf04103800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:25:58.464 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:58.473+0000 7faf09405640 1 -- 172.21.15.52:0/1071185279 >> 172.21.15.52:0/1071185279 conn(0x7faf040fdcf0 msgr2=0x7faf041001a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:25:58.464 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:58.473+0000 7faf09405640 1 -- 172.21.15.52:0/1071185279 shutdown_connections 2024-05-27T07:25:58.464 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:58.473+0000 7faf09405640 1 -- 172.21.15.52:0/1071185279 wait complete. 2024-05-27T07:25:58.466 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:58.474+0000 7faf09405640 1 Processor -- start 2024-05-27T07:25:58.466 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:58.474+0000 7faf09405640 1 -- start start 2024-05-27T07:25:58.466 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:58.474+0000 7faf09405640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7faf04102b00 0x7faf0406aa40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:25:58.466 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:58.474+0000 7faf09405640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7faf04103d40 0x7faf0419f3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:25:58.466 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:58.474+0000 7faf09405640 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7faf04108a40 0x7faf041a1950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:25:58.466 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:58.474+0000 7faf09405640 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x7faf0410e600 con 0x7faf04108a40 2024-05-27T07:25:58.466 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:58.474+0000 7faf09405640 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7faf0410e480 con 0x7faf04103d40 2024-05-27T07:25:58.467 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:58.474+0000 7faf09405640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7faf0410e780 con 0x7faf04102b00 2024-05-27T07:25:58.467 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:58.475+0000 7faf037fe640 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7faf04108a40 0x7faf041a1950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:25:58.467 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:58.475+0000 7faf037fe640 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7faf04108a40 0x7faf041a1950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.52:59386/0 (socket says 172.21.15.52:59386) 2024-05-27T07:25:58.467 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:58.475+0000 7faf037fe640 1 -- 172.21.15.52:0/244311144 learned_addr learned my addr 172.21.15.52:0/244311144 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-05-27T07:25:58.467 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:58.475+0000 7faf027fc640 1 --2- 172.21.15.52:0/244311144 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7faf04103d40 0x7faf0419f3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:25:58.467 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:58.475+0000 7faf02ffd640 1 --2- 172.21.15.52:0/244311144 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7faf04102b00 0x7faf0406aa40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:25:58.467 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:58.475+0000 7faf037fe640 1 -- 172.21.15.52:0/244311144 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7faf04102b00 msgr2=0x7faf0406aa40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:25:58.467 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:58.475+0000 7faf037fe640 1 --2- 172.21.15.52:0/244311144 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7faf04102b00 0x7faf0406aa40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:25:58.467 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:58.475+0000 7faf037fe640 1 -- 172.21.15.52:0/244311144 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7faf04103d40 msgr2=0x7faf0419f3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:25:58.467 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:58.475+0000 7faf037fe640 1 --2- 172.21.15.52:0/244311144 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7faf04103d40 0x7faf0419f3e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:25:58.467 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:58.475+0000 7faf037fe640 1 -- 172.21.15.52:0/244311144 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7faf041a2170 con 0x7faf04108a40 2024-05-27T07:25:58.467 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:58.475+0000 7faf02ffd640 1 --2- 172.21.15.52:0/244311144 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7faf04102b00 0x7faf0406aa40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-27T07:25:58.467 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:58.475+0000 7faf037fe640 1 --2- 172.21.15.52:0/244311144 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7faf04108a40 0x7faf041a1950 secure :-1 s=READY pgs=407 cs=0 l=1 rev1=1 crypto rx=0x7faefc00efd0 tx=0x7faefc00c6a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:25:58.468 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:58.475+0000 7faeebfff640 1 -- 172.21.15.52:0/244311144 <== mon.0 v2:172.21.15.52:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faefc003b70 con 0x7faf04108a40 2024-05-27T07:25:58.468 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:58.475+0000 7faf09405640 1 -- 172.21.15.52:0/244311144 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7faf041a2460 con 0x7faf04108a40 2024-05-27T07:25:58.468 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:58.475+0000 7faeebfff640 1 -- 172.21.15.52:0/244311144 <== mon.0 v2:172.21.15.52:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7faefc003d10 con 0x7faf04108a40 2024-05-27T07:25:58.468 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:58.475+0000 7faeebfff640 1 -- 172.21.15.52:0/244311144 <== mon.0 v2:172.21.15.52:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faefc0108a0 con 0x7faf04108a40 2024-05-27T07:25:58.468 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:58.475+0000 7faf09405640 1 -- 172.21.15.52:0/244311144 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7faf041a29b0 con 0x7faf04108a40 2024-05-27T07:25:58.468 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:58.476+0000 7faf09405640 1 -- 172.21.15.52:0/244311144 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7faf04107850 con 0x7faf04108a40 2024-05-27T07:25:58.469 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:58.478+0000 7faeebfff640 1 -- 172.21.15.52:0/244311144 <== mon.0 v2:172.21.15.52:3300/0 4 ==== mgrmap(e 22) ==== 149464+0+0 (secure 0 0 0) 0x7faefc004b40 con 0x7faf04108a40 2024-05-27T07:25:58.469 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:58.479+0000 7faeebfff640 1 --2- 172.21.15.52:0/244311144 >> [v2:172.21.15.52:6800/1822318384,v1:172.21.15.52:6801/1822318384] conn(0x7faeec0b25e0 0x7faeec0b4af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:25:58.469 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:58.479+0000 7faeebfff640 1 -- 172.21.15.52:0/244311144 <== mon.0 v2:172.21.15.52:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9379+0+0 (secure 0 0 0) 0x7faefc0e2b70 con 0x7faf04108a40 2024-05-27T07:25:58.469 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:58.479+0000 7faf02ffd640 1 --2- 172.21.15.52:0/244311144 >> [v2:172.21.15.52:6800/1822318384,v1:172.21.15.52:6801/1822318384] conn(0x7faeec0b25e0 0x7faeec0b4af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:25:58.469 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:58.479+0000 7faf02ffd640 1 --2- 172.21.15.52:0/244311144 >> [v2:172.21.15.52:6800/1822318384,v1:172.21.15.52:6801/1822318384] conn(0x7faeec0b25e0 0x7faeec0b4af0 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7faf04103700 tx=0x7faef4005f90 comp rx=0 tx=0).ready entity=mgr.15333 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:25:58.472 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:58.483+0000 7faeebfff640 1 -- 172.21.15.52:0/244311144 <== mon.0 v2:172.21.15.52:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191963 (secure 0 0 0) 0x7faefc0aa700 con 0x7faf04108a40 2024-05-27T07:25:58.667 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:58.677+0000 7faf09405640 1 -- 172.21.15.52:0/244311144 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7faf0402dd00 con 0x7faf04108a40 2024-05-27T07:25:58.668 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:58.678+0000 7faeebfff640 1 -- 172.21.15.52:0/244311144 <== mon.0 v2:172.21.15.52:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v91) ==== 93+0+10 (secure 0 0 0) 0x7faefc0aa000 con 0x7faf04108a40 2024-05-27T07:25:58.668 INFO:teuthology.orchestra.run.smithi052.stdout:pg_num: 1 2024-05-27T07:25:58.671 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:58.680+0000 7faf09405640 1 -- 172.21.15.52:0/244311144 >> [v2:172.21.15.52:6800/1822318384,v1:172.21.15.52:6801/1822318384] conn(0x7faeec0b25e0 msgr2=0x7faeec0b4af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:25:58.671 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:58.680+0000 7faf09405640 1 --2- 172.21.15.52:0/244311144 >> [v2:172.21.15.52:6800/1822318384,v1:172.21.15.52:6801/1822318384] conn(0x7faeec0b25e0 0x7faeec0b4af0 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7faf04103700 tx=0x7faef4005f90 comp rx=0 tx=0).stop 2024-05-27T07:25:58.671 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:58.680+0000 7faf09405640 1 -- 172.21.15.52:0/244311144 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7faf04108a40 msgr2=0x7faf041a1950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:25:58.671 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:58.680+0000 7faf09405640 1 --2- 172.21.15.52:0/244311144 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7faf04108a40 0x7faf041a1950 secure :-1 s=READY pgs=407 cs=0 l=1 rev1=1 crypto rx=0x7faefc00efd0 tx=0x7faefc00c6a0 comp rx=0 tx=0).stop 2024-05-27T07:25:58.671 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:58.680+0000 7faf09405640 1 -- 172.21.15.52:0/244311144 shutdown_connections 2024-05-27T07:25:58.671 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:58.680+0000 7faf09405640 1 --2- 172.21.15.52:0/244311144 >> [v2:172.21.15.52:6800/1822318384,v1:172.21.15.52:6801/1822318384] conn(0x7faeec0b25e0 0x7faeec0b4af0 unknown :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:25:58.671 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:58.680+0000 7faf09405640 1 --2- 172.21.15.52:0/244311144 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7faf04108a40 0x7faf041a1950 unknown :-1 s=CLOSED pgs=407 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:25:58.671 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:58.680+0000 7faf09405640 1 --2- 172.21.15.52:0/244311144 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7faf04103d40 0x7faf0419f3e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:25:58.671 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:58.680+0000 7faf09405640 1 --2- 172.21.15.52:0/244311144 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7faf04102b00 0x7faf0406aa40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:25:58.671 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:58.680+0000 7faf09405640 1 -- 172.21.15.52:0/244311144 >> 172.21.15.52:0/244311144 conn(0x7faf040fdcf0 msgr2=0x7faf04104a20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:25:58.671 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:58.680+0000 7faf09405640 1 -- 172.21.15.52:0/244311144 shutdown_connections 2024-05-27T07:25:58.672 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:58.680+0000 7faf09405640 1 -- 172.21.15.52:0/244311144 wait complete. 2024-05-27T07:25:58.681 DEBUG:teuthology.orchestra.run.smithi052:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-05-27T07:25:58.807 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:58.815+0000 7fb04195e640 1 -- 172.21.15.52:0/372333722 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fb03c103d40 msgr2=0x7fb03c104150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:25:58.807 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:58.815+0000 7fb04195e640 1 --2- 172.21.15.52:0/372333722 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fb03c103d40 0x7fb03c104150 secure :-1 s=READY pgs=408 cs=0 l=1 rev1=1 crypto rx=0x7fb030009900 tx=0x7fb03002a410 comp rx=0 tx=0).stop 2024-05-27T07:25:58.807 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:58.815+0000 7fb04195e640 1 -- 172.21.15.52:0/372333722 shutdown_connections 2024-05-27T07:25:58.808 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:58.815+0000 7fb04195e640 1 --2- 172.21.15.52:0/372333722 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fb03c108a40 0x7fb03c10ae60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:25:58.808 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:58.815+0000 7fb04195e640 1 --2- 172.21.15.52:0/372333722 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fb03c103d40 0x7fb03c104150 unknown :-1 s=CLOSED pgs=408 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:25:58.808 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:58.815+0000 7fb04195e640 1 --2- 172.21.15.52:0/372333722 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fb03c102b00 0x7fb03c103800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:25:58.808 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:58.815+0000 7fb04195e640 1 -- 172.21.15.52:0/372333722 >> 172.21.15.52:0/372333722 conn(0x7fb03c0fdcf0 msgr2=0x7fb03c1001a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:25:58.808 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:58.815+0000 7fb04195e640 1 -- 172.21.15.52:0/372333722 shutdown_connections 2024-05-27T07:25:58.808 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:58.815+0000 7fb04195e640 1 -- 172.21.15.52:0/372333722 wait complete. 2024-05-27T07:25:58.808 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:58.815+0000 7fb04195e640 1 Processor -- start 2024-05-27T07:25:58.808 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:58.816+0000 7fb04195e640 1 -- start start 2024-05-27T07:25:58.808 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:58.816+0000 7fb04195e640 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fb03c102b00 0x7fb03c06a9b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:25:58.808 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:58.816+0000 7fb04195e640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fb03c103d40 0x7fb03c19f470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:25:58.808 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:58.816+0000 7fb04195e640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fb03c108a40 0x7fb03c1a19c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:25:58.808 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:58.816+0000 7fb04195e640 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x7fb03c10e600 con 0x7fb03c102b00 2024-05-27T07:25:58.809 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:58.816+0000 7fb04195e640 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7fb03c10e480 con 0x7fb03c108a40 2024-05-27T07:25:58.809 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:58.816+0000 7fb04195e640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7fb03c10e780 con 0x7fb03c103d40 2024-05-27T07:25:58.809 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:58.816+0000 7fb03affd640 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fb03c102b00 0x7fb03c06a9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:25:58.809 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:58.816+0000 7fb03affd640 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fb03c102b00 0x7fb03c06a9b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.52:59398/0 (socket says 172.21.15.52:59398) 2024-05-27T07:25:58.809 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:58.816+0000 7fb03affd640 1 -- 172.21.15.52:0/2620532663 learned_addr learned my addr 172.21.15.52:0/2620532663 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-05-27T07:25:58.809 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:58.816+0000 7fb03b7fe640 1 --2- 172.21.15.52:0/2620532663 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fb03c108a40 0x7fb03c1a19c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:25:58.809 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:58.816+0000 7fb03a7fc640 1 --2- 172.21.15.52:0/2620532663 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fb03c103d40 0x7fb03c19f470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:25:58.809 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:58.816+0000 7fb03affd640 1 -- 172.21.15.52:0/2620532663 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fb03c103d40 msgr2=0x7fb03c19f470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:25:58.809 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:58.816+0000 7fb03affd640 1 --2- 172.21.15.52:0/2620532663 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fb03c103d40 0x7fb03c19f470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:25:58.809 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:58.816+0000 7fb03affd640 1 -- 172.21.15.52:0/2620532663 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fb03c108a40 msgr2=0x7fb03c1a19c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:25:58.809 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:58.816+0000 7fb03affd640 1 --2- 172.21.15.52:0/2620532663 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fb03c108a40 0x7fb03c1a19c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:25:58.809 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:58.817+0000 7fb03affd640 1 -- 172.21.15.52:0/2620532663 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb03c1a2190 con 0x7fb03c102b00 2024-05-27T07:25:58.809 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:58.817+0000 7fb03b7fe640 1 --2- 172.21.15.52:0/2620532663 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fb03c108a40 0x7fb03c1a19c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-27T07:25:58.810 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:58.817+0000 7fb03affd640 1 --2- 172.21.15.52:0/2620532663 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fb03c102b00 0x7fb03c06a9b0 secure :-1 s=READY pgs=409 cs=0 l=1 rev1=1 crypto rx=0x7fb02c00e9f0 tx=0x7fb02c00eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:25:58.810 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:58.817+0000 7fb04095c640 1 -- 172.21.15.52:0/2620532663 <== mon.0 v2:172.21.15.52:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb02c00ccc0 con 0x7fb03c102b00 2024-05-27T07:25:58.810 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:58.817+0000 7fb04095c640 1 -- 172.21.15.52:0/2620532663 <== mon.0 v2:172.21.15.52:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fb02c0035f0 con 0x7fb03c102b00 2024-05-27T07:25:58.810 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:58.817+0000 7fb04095c640 1 -- 172.21.15.52:0/2620532663 <== mon.0 v2:172.21.15.52:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb02c00ce60 con 0x7fb03c102b00 2024-05-27T07:25:58.810 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:58.817+0000 7fb04195e640 1 -- 172.21.15.52:0/2620532663 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb03c1a2480 con 0x7fb03c102b00 2024-05-27T07:25:58.810 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:58.817+0000 7fb04195e640 1 -- 172.21.15.52:0/2620532663 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb03c1a2980 con 0x7fb03c102b00 2024-05-27T07:25:58.811 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:58.818+0000 7fb04195e640 1 -- 172.21.15.52:0/2620532663 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb03c107850 con 0x7fb03c102b00 2024-05-27T07:25:58.811 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:58.820+0000 7fb04095c640 1 -- 172.21.15.52:0/2620532663 <== mon.0 v2:172.21.15.52:3300/0 4 ==== mgrmap(e 22) ==== 149464+0+0 (secure 0 0 0) 0x7fb02c003aa0 con 0x7fb03c102b00 2024-05-27T07:25:58.811 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:58.821+0000 7fb04095c640 1 --2- 172.21.15.52:0/2620532663 >> [v2:172.21.15.52:6800/1822318384,v1:172.21.15.52:6801/1822318384] conn(0x7fb0240b25e0 0x7fb0240b4af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:25:58.811 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:58.821+0000 7fb03a7fc640 1 --2- 172.21.15.52:0/2620532663 >> [v2:172.21.15.52:6800/1822318384,v1:172.21.15.52:6801/1822318384] conn(0x7fb0240b25e0 0x7fb0240b4af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:25:58.811 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:58.821+0000 7fb04095c640 1 -- 172.21.15.52:0/2620532663 <== mon.0 v2:172.21.15.52:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9379+0+0 (secure 0 0 0) 0x7fb02c0e2640 con 0x7fb03c102b00 2024-05-27T07:25:58.812 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:58.822+0000 7fb03a7fc640 1 --2- 172.21.15.52:0/2620532663 >> [v2:172.21.15.52:6800/1822318384,v1:172.21.15.52:6801/1822318384] conn(0x7fb0240b25e0 0x7fb0240b4af0 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7fb030009230 tx=0x7fb030031400 comp rx=0 tx=0).ready entity=mgr.15333 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:25:58.814 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:58.824+0000 7fb04095c640 1 -- 172.21.15.52:0/2620532663 <== mon.0 v2:172.21.15.52:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191963 (secure 0 0 0) 0x7fb02c0aa1d0 con 0x7fb03c102b00 2024-05-27T07:25:58.947 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:58.958+0000 7fb04095c640 1 -- 172.21.15.52:0/2620532663 <== mon.0 v2:172.21.15.52:3300/0 7 ==== mgrmap(e 23) ==== 149823+0+0 (secure 0 0 0) 0x7fb02c0a5d60 con 0x7fb03c102b00 2024-05-27T07:25:59.033 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:59.044+0000 7fb04195e640 1 -- 172.21.15.52:0/2620532663 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7fb03c02de90 con 0x7fb03c102b00 2024-05-27T07:25:59.034 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:59.044+0000 7fb04095c640 1 -- 172.21.15.52:0/2620532663 <== mon.0 v2:172.21.15.52:3300/0 8 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v91) ==== 104+0+11 (secure 0 0 0) 0x7fb02c0a1b30 con 0x7fb03c102b00 2024-05-27T07:25:59.034 INFO:teuthology.orchestra.run.smithi052.stdout:pg_num: 32 2024-05-27T07:25:59.037 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:59.047+0000 7fb04195e640 1 -- 172.21.15.52:0/2620532663 >> [v2:172.21.15.52:6800/1822318384,v1:172.21.15.52:6801/1822318384] conn(0x7fb0240b25e0 msgr2=0x7fb0240b4af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:25:59.037 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:59.047+0000 7fb04195e640 1 --2- 172.21.15.52:0/2620532663 >> [v2:172.21.15.52:6800/1822318384,v1:172.21.15.52:6801/1822318384] conn(0x7fb0240b25e0 0x7fb0240b4af0 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7fb030009230 tx=0x7fb030031400 comp rx=0 tx=0).stop 2024-05-27T07:25:59.037 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:59.047+0000 7fb04195e640 1 -- 172.21.15.52:0/2620532663 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fb03c102b00 msgr2=0x7fb03c06a9b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:25:59.038 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:59.047+0000 7fb04195e640 1 --2- 172.21.15.52:0/2620532663 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fb03c102b00 0x7fb03c06a9b0 secure :-1 s=READY pgs=409 cs=0 l=1 rev1=1 crypto rx=0x7fb02c00e9f0 tx=0x7fb02c00eeb0 comp rx=0 tx=0).stop 2024-05-27T07:25:59.038 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:59.047+0000 7fb04195e640 1 -- 172.21.15.52:0/2620532663 shutdown_connections 2024-05-27T07:25:59.038 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:59.047+0000 7fb04195e640 1 --2- 172.21.15.52:0/2620532663 >> [v2:172.21.15.52:6800/1822318384,v1:172.21.15.52:6801/1822318384] conn(0x7fb0240b25e0 0x7fb0240b4af0 unknown :-1 s=CLOSED pgs=38 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:25:59.038 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:59.047+0000 7fb04195e640 1 --2- 172.21.15.52:0/2620532663 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fb03c108a40 0x7fb03c1a19c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:25:59.038 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:59.047+0000 7fb04195e640 1 --2- 172.21.15.52:0/2620532663 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fb03c103d40 0x7fb03c19f470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:25:59.038 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:59.047+0000 7fb04195e640 1 --2- 172.21.15.52:0/2620532663 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fb03c102b00 0x7fb03c06a9b0 unknown :-1 s=CLOSED pgs=409 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:25:59.038 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:59.047+0000 7fb04195e640 1 -- 172.21.15.52:0/2620532663 >> 172.21.15.52:0/2620532663 conn(0x7fb03c0fdcf0 msgr2=0x7fb03c104a20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:25:59.038 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:59.048+0000 7fb04195e640 1 -- 172.21.15.52:0/2620532663 shutdown_connections 2024-05-27T07:25:59.038 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:59.048+0000 7fb04195e640 1 -- 172.21.15.52:0/2620532663 wait complete. 2024-05-27T07:25:59.052 DEBUG:teuthology.orchestra.run.smithi052:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-05-27T07:25:59.188 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:59.196+0000 7fa42b630640 1 -- 172.21.15.52:0/510186494 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fa424103db0 msgr2=0x7fa4241041c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:25:59.189 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:59.197+0000 7fa42b630640 1 --2- 172.21.15.52:0/510186494 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fa424103db0 0x7fa4241041c0 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7fa414009860 tx=0x7fa41402a340 comp rx=0 tx=0).stop 2024-05-27T07:25:59.189 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:59.197+0000 7fa42b630640 1 -- 172.21.15.52:0/510186494 shutdown_connections 2024-05-27T07:25:59.189 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:59.197+0000 7fa42b630640 1 --2- 172.21.15.52:0/510186494 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fa424108ab0 0x7fa42410aed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:25:59.189 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:59.197+0000 7fa42b630640 1 --2- 172.21.15.52:0/510186494 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fa424103db0 0x7fa4241041c0 unknown :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:25:59.189 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:59.197+0000 7fa42b630640 1 --2- 172.21.15.52:0/510186494 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fa424102b70 0x7fa424103870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:25:59.189 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:59.197+0000 7fa42b630640 1 -- 172.21.15.52:0/510186494 >> 172.21.15.52:0/510186494 conn(0x7fa4240fdd60 msgr2=0x7fa424100210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:25:59.189 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:59.197+0000 7fa42b630640 1 -- 172.21.15.52:0/510186494 shutdown_connections 2024-05-27T07:25:59.189 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:59.197+0000 7fa42b630640 1 -- 172.21.15.52:0/510186494 wait complete. 2024-05-27T07:25:59.189 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:59.197+0000 7fa42b630640 1 Processor -- start 2024-05-27T07:25:59.189 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:59.197+0000 7fa42b630640 1 -- start start 2024-05-27T07:25:59.190 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:59.198+0000 7fa42b630640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fa424102b70 0x7fa424198120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:25:59.190 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:59.198+0000 7fa42b630640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fa424103db0 0x7fa424198660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:25:59.190 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:59.198+0000 7fa42b630640 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fa424108ab0 0x7fa42419f5f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:25:59.190 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:59.198+0000 7fa42b630640 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x7fa42410d530 con 0x7fa424108ab0 2024-05-27T07:25:59.190 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:59.198+0000 7fa42b630640 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7fa42410d3b0 con 0x7fa424102b70 2024-05-27T07:25:59.190 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:59.198+0000 7fa42b630640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7fa42410d6b0 con 0x7fa424103db0 2024-05-27T07:25:59.190 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:59.198+0000 7fa4293cc640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fa424102b70 0x7fa424198120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:25:59.190 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:59.198+0000 7fa429bcd640 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fa424108ab0 0x7fa42419f5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:25:59.190 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:59.198+0000 7fa4293cc640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fa424102b70 0x7fa424198120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.52:33386/0 (socket says 172.21.15.52:33386) 2024-05-27T07:25:59.190 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:59.198+0000 7fa4293cc640 1 -- 172.21.15.52:0/3580314251 learned_addr learned my addr 172.21.15.52:0/3580314251 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-05-27T07:25:59.190 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:59.198+0000 7fa428bcb640 1 --2- 172.21.15.52:0/3580314251 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fa424103db0 0x7fa424198660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:25:59.191 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:59.198+0000 7fa4293cc640 1 -- 172.21.15.52:0/3580314251 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fa424103db0 msgr2=0x7fa424198660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:25:59.191 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:59.198+0000 7fa4293cc640 1 --2- 172.21.15.52:0/3580314251 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fa424103db0 0x7fa424198660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:25:59.191 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:59.198+0000 7fa4293cc640 1 -- 172.21.15.52:0/3580314251 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fa424108ab0 msgr2=0x7fa42419f5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:25:59.191 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:59.198+0000 7fa4293cc640 1 --2- 172.21.15.52:0/3580314251 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fa424108ab0 0x7fa42419f5f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:25:59.191 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:59.198+0000 7fa4293cc640 1 -- 172.21.15.52:0/3580314251 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa42419fc60 con 0x7fa424102b70 2024-05-27T07:25:59.191 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:59.198+0000 7fa428bcb640 1 --2- 172.21.15.52:0/3580314251 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fa424103db0 0x7fa424198660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-27T07:25:59.191 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:59.199+0000 7fa4293cc640 1 --2- 172.21.15.52:0/3580314251 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fa424102b70 0x7fa424198120 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7fa420009990 tx=0x7fa420009e50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:25:59.191 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:59.199+0000 7fa41a7fc640 1 -- 172.21.15.52:0/3580314251 <== mon.2 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa420007a50 con 0x7fa424102b70 2024-05-27T07:25:59.191 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:59.199+0000 7fa41a7fc640 1 -- 172.21.15.52:0/3580314251 <== mon.2 v2:172.21.15.105:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fa420007bf0 con 0x7fa424102b70 2024-05-27T07:25:59.191 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:59.199+0000 7fa42b630640 1 -- 172.21.15.52:0/3580314251 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa42419ff50 con 0x7fa424102b70 2024-05-27T07:25:59.191 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:59.199+0000 7fa41a7fc640 1 -- 172.21.15.52:0/3580314251 <== mon.2 v2:172.21.15.105:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa420003d60 con 0x7fa424102b70 2024-05-27T07:25:59.191 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:59.199+0000 7fa42b630640 1 -- 172.21.15.52:0/3580314251 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa4241a0410 con 0x7fa424102b70 2024-05-27T07:25:59.192 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:25:58 smithi105 ceph-mon[35977]: [27/May/2024:07:25:57] ENGINE Bus STARTING 2024-05-27T07:25:59.192 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:25:58 smithi105 ceph-mon[35977]: [27/May/2024:07:25:57] ENGINE Serving on http://172.21.15.52:8765 2024-05-27T07:25:59.192 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:25:58 smithi105 ceph-mon[35977]: [27/May/2024:07:25:57] ENGINE Serving on https://172.21.15.52:7150 2024-05-27T07:25:59.192 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:25:58 smithi105 ceph-mon[35977]: [27/May/2024:07:25:57] ENGINE Bus STARTED 2024-05-27T07:25:59.193 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:25:58 smithi105 ceph-mon[35977]: mgrmap e22: x(active, since 1.02211s), standbys: z, y 2024-05-27T07:25:59.193 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:25:58 smithi105 ceph-mon[35977]: from='client.25100 -' entity='client.admin' cmd=[{"prefix": "fs subvolumegroup pin", "vol_name": "cephfs", "group_name": "qa", "pin_type": "random", "pin_setting": "0.10", "target": ["mon-mgr", ""]}]: dispatch 2024-05-27T07:25:59.193 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:25:58 smithi105 ceph-mon[35977]: pgmap v3: 129 pgs: 129 active+clean; 1.1 MiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-27T07:25:59.193 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:25:58 smithi105 ceph-mon[35977]: from='client.? 172.21.15.52:0/2235682404' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-05-27T07:25:59.193 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:25:58 smithi105 ceph-mon[35977]: from='client.? 172.21.15.52:0/244311144' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-05-27T07:25:59.193 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:59.200+0000 7fa42b630640 1 -- 172.21.15.52:0/3580314251 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa4241078c0 con 0x7fa424102b70 2024-05-27T07:25:59.194 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:59.202+0000 7fa41a7fc640 1 -- 172.21.15.52:0/3580314251 <== mon.2 v2:172.21.15.105:3300/0 4 ==== mgrmap(e 23) ==== 149823+0+0 (secure 0 0 0) 0x7fa42001e030 con 0x7fa424102b70 2024-05-27T07:25:59.194 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:59.203+0000 7fa41a7fc640 1 --2- 172.21.15.52:0/3580314251 >> [v2:172.21.15.52:6800/1822318384,v1:172.21.15.52:6801/1822318384] conn(0x7fa4100b2a70 0x7fa4100b4f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:25:59.194 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:59.203+0000 7fa41a7fc640 1 -- 172.21.15.52:0/3580314251 <== mon.2 v2:172.21.15.105:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9379+0+0 (secure 0 0 0) 0x7fa4200e4080 con 0x7fa424102b70 2024-05-27T07:25:59.194 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:59.203+0000 7fa428bcb640 1 --2- 172.21.15.52:0/3580314251 >> [v2:172.21.15.52:6800/1822318384,v1:172.21.15.52:6801/1822318384] conn(0x7fa4100b2a70 0x7fa4100b4f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:25:59.194 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:59.204+0000 7fa428bcb640 1 --2- 172.21.15.52:0/3580314251 >> [v2:172.21.15.52:6800/1822318384,v1:172.21.15.52:6801/1822318384] conn(0x7fa4100b2a70 0x7fa4100b4f80 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7fa41402a930 tx=0x7fa414030450 comp rx=0 tx=0).ready entity=mgr.15333 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:25:59.195 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:59.206+0000 7fa41a7fc640 1 -- 172.21.15.52:0/3580314251 <== mon.2 v2:172.21.15.105:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191963 (secure 0 0 0) 0x7fa420025060 con 0x7fa424102b70 2024-05-27T07:25:59.229 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:25:58 smithi052 ceph-mon[31955]: [27/May/2024:07:25:57] ENGINE Bus STARTING 2024-05-27T07:25:59.229 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:25:58 smithi052 ceph-mon[31955]: [27/May/2024:07:25:57] ENGINE Serving on http://172.21.15.52:8765 2024-05-27T07:25:59.229 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:25:58 smithi052 ceph-mon[31955]: [27/May/2024:07:25:57] ENGINE Serving on https://172.21.15.52:7150 2024-05-27T07:25:59.229 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:25:58 smithi052 ceph-mon[31955]: [27/May/2024:07:25:57] ENGINE Bus STARTED 2024-05-27T07:25:59.230 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:25:58 smithi052 ceph-mon[31955]: mgrmap e22: x(active, since 1.02211s), standbys: z, y 2024-05-27T07:25:59.230 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:25:58 smithi052 ceph-mon[31955]: from='client.25100 -' entity='client.admin' cmd=[{"prefix": "fs subvolumegroup pin", "vol_name": "cephfs", "group_name": "qa", "pin_type": "random", "pin_setting": "0.10", "target": ["mon-mgr", ""]}]: dispatch 2024-05-27T07:25:59.230 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:25:58 smithi052 ceph-mon[31955]: pgmap v3: 129 pgs: 129 active+clean; 1.1 MiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-27T07:25:59.230 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:25:58 smithi052 ceph-mon[31955]: from='client.? 172.21.15.52:0/2235682404' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-05-27T07:25:59.230 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:25:58 smithi052 ceph-mon[31955]: from='client.? 172.21.15.52:0/244311144' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-05-27T07:25:59.352 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:25:58 smithi173 ceph-mon[37000]: [27/May/2024:07:25:57] ENGINE Bus STARTING 2024-05-27T07:25:59.352 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:25:58 smithi173 ceph-mon[37000]: [27/May/2024:07:25:57] ENGINE Serving on http://172.21.15.52:8765 2024-05-27T07:25:59.352 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:25:58 smithi173 ceph-mon[37000]: [27/May/2024:07:25:57] ENGINE Serving on https://172.21.15.52:7150 2024-05-27T07:25:59.352 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:25:58 smithi173 ceph-mon[37000]: [27/May/2024:07:25:57] ENGINE Bus STARTED 2024-05-27T07:25:59.352 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:25:58 smithi173 ceph-mon[37000]: mgrmap e22: x(active, since 1.02211s), standbys: z, y 2024-05-27T07:25:59.352 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:25:58 smithi173 ceph-mon[37000]: from='client.25100 -' entity='client.admin' cmd=[{"prefix": "fs subvolumegroup pin", "vol_name": "cephfs", "group_name": "qa", "pin_type": "random", "pin_setting": "0.10", "target": ["mon-mgr", ""]}]: dispatch 2024-05-27T07:25:59.352 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:25:58 smithi173 ceph-mon[37000]: pgmap v3: 129 pgs: 129 active+clean; 1.1 MiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-27T07:25:59.352 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:25:58 smithi173 ceph-mon[37000]: from='client.? 172.21.15.52:0/2235682404' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-05-27T07:25:59.352 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:25:58 smithi173 ceph-mon[37000]: from='client.? 172.21.15.52:0/244311144' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-05-27T07:25:59.393 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:59.404+0000 7fa42b630640 1 -- 172.21.15.52:0/3580314251 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7fa42402de70 con 0x7fa424102b70 2024-05-27T07:25:59.394 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:59.405+0000 7fa41a7fc640 1 -- 172.21.15.52:0/3580314251 <== mon.2 v2:172.21.15.105:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v91) ==== 100+0+11 (secure 0 0 0) 0x7fa420014070 con 0x7fa424102b70 2024-05-27T07:25:59.394 INFO:teuthology.orchestra.run.smithi052.stdout:pg_num: 64 2024-05-27T07:25:59.397 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:59.407+0000 7fa42b630640 1 -- 172.21.15.52:0/3580314251 >> [v2:172.21.15.52:6800/1822318384,v1:172.21.15.52:6801/1822318384] conn(0x7fa4100b2a70 msgr2=0x7fa4100b4f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:25:59.397 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:59.407+0000 7fa42b630640 1 --2- 172.21.15.52:0/3580314251 >> [v2:172.21.15.52:6800/1822318384,v1:172.21.15.52:6801/1822318384] conn(0x7fa4100b2a70 0x7fa4100b4f80 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7fa41402a930 tx=0x7fa414030450 comp rx=0 tx=0).stop 2024-05-27T07:25:59.397 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:59.407+0000 7fa42b630640 1 -- 172.21.15.52:0/3580314251 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fa424102b70 msgr2=0x7fa424198120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:25:59.397 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:59.407+0000 7fa42b630640 1 --2- 172.21.15.52:0/3580314251 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fa424102b70 0x7fa424198120 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7fa420009990 tx=0x7fa420009e50 comp rx=0 tx=0).stop 2024-05-27T07:25:59.397 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:59.407+0000 7fa42b630640 1 -- 172.21.15.52:0/3580314251 shutdown_connections 2024-05-27T07:25:59.397 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:59.407+0000 7fa42b630640 1 --2- 172.21.15.52:0/3580314251 >> [v2:172.21.15.52:6800/1822318384,v1:172.21.15.52:6801/1822318384] conn(0x7fa4100b2a70 0x7fa4100b4f80 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:25:59.397 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:59.407+0000 7fa42b630640 1 --2- 172.21.15.52:0/3580314251 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fa424108ab0 0x7fa42419f5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:25:59.397 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:59.407+0000 7fa42b630640 1 --2- 172.21.15.52:0/3580314251 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fa424103db0 0x7fa424198660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:25:59.397 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:59.407+0000 7fa42b630640 1 --2- 172.21.15.52:0/3580314251 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fa424102b70 0x7fa424198120 unknown :-1 s=CLOSED pgs=68 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:25:59.397 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:59.407+0000 7fa42b630640 1 -- 172.21.15.52:0/3580314251 >> 172.21.15.52:0/3580314251 conn(0x7fa4240fdd60 msgr2=0x7fa424104a90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:25:59.397 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:59.407+0000 7fa42b630640 1 -- 172.21.15.52:0/3580314251 shutdown_connections 2024-05-27T07:25:59.398 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:59.407+0000 7fa42b630640 1 -- 172.21.15.52:0/3580314251 wait complete. 2024-05-27T07:25:59.408 DEBUG:teuthology.orchestra.run.smithi052:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2024-05-27T07:25:59.528 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:59.538+0000 7f9f3ccaa640 1 -- 172.21.15.52:0/1112951333 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f9f38102b10 msgr2=0x7f9f38103810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:25:59.528 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:59.538+0000 7f9f3ccaa640 1 --2- 172.21.15.52:0/1112951333 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f9f38102b10 0x7f9f38103810 secure :-1 s=READY pgs=412 cs=0 l=1 rev1=1 crypto rx=0x7f9f28009900 tx=0x7f9f2802a410 comp rx=0 tx=0).stop 2024-05-27T07:25:59.528 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:59.538+0000 7f9f3ccaa640 1 -- 172.21.15.52:0/1112951333 shutdown_connections 2024-05-27T07:25:59.528 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:59.538+0000 7f9f3ccaa640 1 --2- 172.21.15.52:0/1112951333 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f9f38108a50 0x7f9f3810ae70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:25:59.528 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:59.538+0000 7f9f3ccaa640 1 --2- 172.21.15.52:0/1112951333 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f9f38103d50 0x7f9f38104160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:25:59.528 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:59.538+0000 7f9f3ccaa640 1 --2- 172.21.15.52:0/1112951333 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f9f38102b10 0x7f9f38103810 unknown :-1 s=CLOSED pgs=412 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:25:59.528 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:59.538+0000 7f9f3ccaa640 1 -- 172.21.15.52:0/1112951333 >> 172.21.15.52:0/1112951333 conn(0x7f9f380fdd20 msgr2=0x7f9f381001b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:25:59.528 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:59.538+0000 7f9f3ccaa640 1 -- 172.21.15.52:0/1112951333 shutdown_connections 2024-05-27T07:25:59.529 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:59.538+0000 7f9f3ccaa640 1 -- 172.21.15.52:0/1112951333 wait complete. 2024-05-27T07:25:59.531 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:59.539+0000 7f9f3ccaa640 1 Processor -- start 2024-05-27T07:25:59.531 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:59.539+0000 7f9f3ccaa640 1 -- start start 2024-05-27T07:25:59.531 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:59.539+0000 7f9f3ccaa640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f9f38102b10 0x7f9f3819a380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:25:59.531 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:59.539+0000 7f9f3ccaa640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f9f38103d50 0x7f9f3819a8c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:25:59.531 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:59.539+0000 7f9f3ccaa640 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f9f38108a50 0x7f9f381a1850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:25:59.531 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:59.539+0000 7f9f3ccaa640 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x7f9f3810f740 con 0x7f9f38108a50 2024-05-27T07:25:59.531 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:59.539+0000 7f9f3ccaa640 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f9f3810f5c0 con 0x7f9f38102b10 2024-05-27T07:25:59.532 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:59.539+0000 7f9f3ccaa640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f9f3810f8c0 con 0x7f9f38103d50 2024-05-27T07:25:59.532 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:59.539+0000 7f9f36d9d640 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f9f38108a50 0x7f9f381a1850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:25:59.532 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:59.539+0000 7f9f36d9d640 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f9f38108a50 0x7f9f381a1850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.52:59462/0 (socket says 172.21.15.52:59462) 2024-05-27T07:25:59.532 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:59.539+0000 7f9f36d9d640 1 -- 172.21.15.52:0/2630188342 learned_addr learned my addr 172.21.15.52:0/2630188342 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-05-27T07:25:59.532 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:59.539+0000 7f9f3659c640 1 --2- 172.21.15.52:0/2630188342 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f9f38102b10 0x7f9f3819a380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:25:59.532 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:59.539+0000 7f9f2ffff640 1 --2- 172.21.15.52:0/2630188342 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f9f38103d50 0x7f9f3819a8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:25:59.532 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:59.539+0000 7f9f36d9d640 1 -- 172.21.15.52:0/2630188342 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f9f38103d50 msgr2=0x7f9f3819a8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:25:59.532 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:59.539+0000 7f9f36d9d640 1 --2- 172.21.15.52:0/2630188342 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f9f38103d50 0x7f9f3819a8c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:25:59.532 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:59.539+0000 7f9f36d9d640 1 -- 172.21.15.52:0/2630188342 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f9f38102b10 msgr2=0x7f9f3819a380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:25:59.532 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:59.539+0000 7f9f36d9d640 1 --2- 172.21.15.52:0/2630188342 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f9f38102b10 0x7f9f3819a380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:25:59.532 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:59.540+0000 7f9f36d9d640 1 -- 172.21.15.52:0/2630188342 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9f381a1ec0 con 0x7f9f38108a50 2024-05-27T07:25:59.532 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:59.540+0000 7f9f3659c640 1 --2- 172.21.15.52:0/2630188342 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f9f38102b10 0x7f9f3819a380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-27T07:25:59.533 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:59.540+0000 7f9f36d9d640 1 --2- 172.21.15.52:0/2630188342 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f9f38108a50 0x7f9f381a1850 secure :-1 s=READY pgs=413 cs=0 l=1 rev1=1 crypto rx=0x7f9f3000eaa0 tx=0x7f9f3000ef60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:25:59.533 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:59.540+0000 7f9f2f7fe640 1 -- 172.21.15.52:0/2630188342 <== mon.0 v2:172.21.15.52:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9f3000cdb0 con 0x7f9f38108a50 2024-05-27T07:25:59.533 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:59.540+0000 7f9f2f7fe640 1 -- 172.21.15.52:0/2630188342 <== mon.0 v2:172.21.15.52:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f9f30005020 con 0x7f9f38108a50 2024-05-27T07:25:59.533 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:59.540+0000 7f9f2f7fe640 1 -- 172.21.15.52:0/2630188342 <== mon.0 v2:172.21.15.52:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9f300116d0 con 0x7f9f38108a50 2024-05-27T07:25:59.533 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:59.540+0000 7f9f3ccaa640 1 -- 172.21.15.52:0/2630188342 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9f381a21b0 con 0x7f9f38108a50 2024-05-27T07:25:59.533 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:59.540+0000 7f9f3ccaa640 1 -- 172.21.15.52:0/2630188342 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9f381a26b0 con 0x7f9f38108a50 2024-05-27T07:25:59.533 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:59.541+0000 7f9f3ccaa640 1 -- 172.21.15.52:0/2630188342 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9f38107860 con 0x7f9f38108a50 2024-05-27T07:25:59.533 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:59.544+0000 7f9f2f7fe640 1 -- 172.21.15.52:0/2630188342 <== mon.0 v2:172.21.15.52:3300/0 4 ==== mgrmap(e 23) ==== 149823+0+0 (secure 0 0 0) 0x7f9f300051d0 con 0x7f9f38108a50 2024-05-27T07:25:59.534 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:59.545+0000 7f9f2f7fe640 1 --2- 172.21.15.52:0/2630188342 >> [v2:172.21.15.52:6800/1822318384,v1:172.21.15.52:6801/1822318384] conn(0x7f9f240b2a70 0x7f9f240b4f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:25:59.534 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:59.545+0000 7f9f2f7fe640 1 -- 172.21.15.52:0/2630188342 <== mon.0 v2:172.21.15.52:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9379+0+0 (secure 0 0 0) 0x7f9f300e2fb0 con 0x7f9f38108a50 2024-05-27T07:25:59.535 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:59.545+0000 7f9f3659c640 1 --2- 172.21.15.52:0/2630188342 >> [v2:172.21.15.52:6800/1822318384,v1:172.21.15.52:6801/1822318384] conn(0x7f9f240b2a70 0x7f9f240b4f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:25:59.535 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:59.545+0000 7f9f3659c640 1 --2- 172.21.15.52:0/2630188342 >> [v2:172.21.15.52:6800/1822318384,v1:172.21.15.52:6801/1822318384] conn(0x7f9f240b2a70 0x7f9f240b4f80 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f9f28005ab0 tx=0x7f9f28031400 comp rx=0 tx=0).ready entity=mgr.15333 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:25:59.537 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:59.547+0000 7f9f2f7fe640 1 -- 172.21.15.52:0/2630188342 <== mon.0 v2:172.21.15.52:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191963 (secure 0 0 0) 0x7f9f30016030 con 0x7f9f38108a50 2024-05-27T07:25:59.736 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:59.746+0000 7f9f3ccaa640 1 -- 172.21.15.52:0/2630188342 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) -- 0x7f9f3802ddd0 con 0x7f9f38108a50 2024-05-27T07:25:59.736 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:59.747+0000 7f9f2f7fe640 1 -- 172.21.15.52:0/2630188342 <== mon.0 v2:172.21.15.52:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v91) ==== 103+0+11 (secure 0 0 0) 0x7f9f300a9e90 con 0x7f9f38108a50 2024-05-27T07:25:59.736 INFO:teuthology.orchestra.run.smithi052.stdout:pg_num: 32 2024-05-27T07:25:59.739 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:59.749+0000 7f9f3ccaa640 1 -- 172.21.15.52:0/2630188342 >> [v2:172.21.15.52:6800/1822318384,v1:172.21.15.52:6801/1822318384] conn(0x7f9f240b2a70 msgr2=0x7f9f240b4f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:25:59.740 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:59.749+0000 7f9f3ccaa640 1 --2- 172.21.15.52:0/2630188342 >> [v2:172.21.15.52:6800/1822318384,v1:172.21.15.52:6801/1822318384] conn(0x7f9f240b2a70 0x7f9f240b4f80 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f9f28005ab0 tx=0x7f9f28031400 comp rx=0 tx=0).stop 2024-05-27T07:25:59.740 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:59.749+0000 7f9f3ccaa640 1 -- 172.21.15.52:0/2630188342 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f9f38108a50 msgr2=0x7f9f381a1850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:25:59.740 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:59.749+0000 7f9f3ccaa640 1 --2- 172.21.15.52:0/2630188342 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f9f38108a50 0x7f9f381a1850 secure :-1 s=READY pgs=413 cs=0 l=1 rev1=1 crypto rx=0x7f9f3000eaa0 tx=0x7f9f3000ef60 comp rx=0 tx=0).stop 2024-05-27T07:25:59.740 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:59.749+0000 7f9f3ccaa640 1 -- 172.21.15.52:0/2630188342 shutdown_connections 2024-05-27T07:25:59.740 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:59.749+0000 7f9f3ccaa640 1 --2- 172.21.15.52:0/2630188342 >> [v2:172.21.15.52:6800/1822318384,v1:172.21.15.52:6801/1822318384] conn(0x7f9f240b2a70 0x7f9f240b4f80 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:25:59.740 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:59.750+0000 7f9f3ccaa640 1 --2- 172.21.15.52:0/2630188342 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f9f38108a50 0x7f9f381a1850 unknown :-1 s=CLOSED pgs=413 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:25:59.740 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:59.750+0000 7f9f3ccaa640 1 --2- 172.21.15.52:0/2630188342 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f9f38103d50 0x7f9f3819a8c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:25:59.740 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:59.750+0000 7f9f3ccaa640 1 --2- 172.21.15.52:0/2630188342 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f9f38102b10 0x7f9f3819a380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:25:59.740 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:59.750+0000 7f9f3ccaa640 1 -- 172.21.15.52:0/2630188342 >> 172.21.15.52:0/2630188342 conn(0x7f9f380fdd20 msgr2=0x7f9f38104a30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:25:59.740 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:59.750+0000 7f9f3ccaa640 1 -- 172.21.15.52:0/2630188342 shutdown_connections 2024-05-27T07:25:59.740 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:25:59.750+0000 7f9f3ccaa640 1 -- 172.21.15.52:0/2630188342 wait complete. 2024-05-27T07:25:59.751 INFO:tasks.fwd_scrub:Wait for all MDSs to reach steady state... 2024-05-27T07:25:59.751 DEBUG:teuthology.orchestra.run.smithi052:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:38f9d900c83c63799fbdbe61acc9a11b0d3554a6 shell --fsid 73df2636-1bf8-11ef-bc9b-c7b262605968 -- ceph fs dump --format=json 2024-05-27T07:26:00.013 INFO:teuthology.orchestra.run.smithi052.stderr:Inferring config /var/lib/ceph/73df2636-1bf8-11ef-bc9b-c7b262605968/mon.a/config 2024-05-27T07:26:00.136 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:25:59 smithi052 ceph-mon[31955]: pgmap v4: 129 pgs: 129 active+clean; 1.1 MiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-27T07:26:00.136 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:25:59 smithi052 ceph-mon[31955]: mgrmap e23: x(active, since 2s), standbys: z, y 2024-05-27T07:26:00.136 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:25:59 smithi052 ceph-mon[31955]: from='mgr.15333 172.21.15.52:0/416389082' entity='mgr.x' 2024-05-27T07:26:00.136 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:25:59 smithi052 ceph-mon[31955]: from='mgr.15333 172.21.15.52:0/416389082' entity='mgr.x' 2024-05-27T07:26:00.137 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:25:59 smithi052 ceph-mon[31955]: from='client.? 172.21.15.52:0/2620532663' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-05-27T07:26:00.137 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:25:59 smithi052 ceph-mon[31955]: from='mgr.15333 172.21.15.52:0/416389082' entity='mgr.x' 2024-05-27T07:26:00.137 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:25:59 smithi052 ceph-mon[31955]: from='mgr.15333 172.21.15.52:0/416389082' entity='mgr.x' 2024-05-27T07:26:00.137 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:25:59 smithi052 ceph-mon[31955]: from='mgr.15333 172.21.15.52:0/416389082' entity='mgr.x' 2024-05-27T07:26:00.137 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:25:59 smithi052 ceph-mon[31955]: from='mgr.15333 172.21.15.52:0/416389082' entity='mgr.x' 2024-05-27T07:26:00.137 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:25:59 smithi052 ceph-mon[31955]: from='client.? 172.21.15.52:0/3580314251' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-05-27T07:26:00.137 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:25:59 smithi052 ceph-mon[31955]: from='client.? 172.21.15.52:0/2630188342' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-05-27T07:26:00.190 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:25:59 smithi105 ceph-mon[35977]: pgmap v4: 129 pgs: 129 active+clean; 1.1 MiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-27T07:26:00.190 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:25:59 smithi105 ceph-mon[35977]: mgrmap e23: x(active, since 2s), standbys: z, y 2024-05-27T07:26:00.190 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:25:59 smithi105 ceph-mon[35977]: from='mgr.15333 172.21.15.52:0/416389082' entity='mgr.x' 2024-05-27T07:26:00.190 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:25:59 smithi105 ceph-mon[35977]: from='mgr.15333 172.21.15.52:0/416389082' entity='mgr.x' 2024-05-27T07:26:00.190 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:25:59 smithi105 ceph-mon[35977]: from='client.? 172.21.15.52:0/2620532663' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-05-27T07:26:00.190 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:25:59 smithi105 ceph-mon[35977]: from='mgr.15333 172.21.15.52:0/416389082' entity='mgr.x' 2024-05-27T07:26:00.191 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:25:59 smithi105 ceph-mon[35977]: from='mgr.15333 172.21.15.52:0/416389082' entity='mgr.x' 2024-05-27T07:26:00.191 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:25:59 smithi105 ceph-mon[35977]: from='mgr.15333 172.21.15.52:0/416389082' entity='mgr.x' 2024-05-27T07:26:00.191 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:25:59 smithi105 ceph-mon[35977]: from='mgr.15333 172.21.15.52:0/416389082' entity='mgr.x' 2024-05-27T07:26:00.191 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:25:59 smithi105 ceph-mon[35977]: from='client.? 172.21.15.52:0/3580314251' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-05-27T07:26:00.191 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:25:59 smithi105 ceph-mon[35977]: from='client.? 172.21.15.52:0/2630188342' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-05-27T07:26:00.352 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:25:59 smithi173 ceph-mon[37000]: pgmap v4: 129 pgs: 129 active+clean; 1.1 MiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-27T07:26:00.352 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:25:59 smithi173 ceph-mon[37000]: mgrmap e23: x(active, since 2s), standbys: z, y 2024-05-27T07:26:00.352 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:25:59 smithi173 ceph-mon[37000]: from='mgr.15333 172.21.15.52:0/416389082' entity='mgr.x' 2024-05-27T07:26:00.352 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:25:59 smithi173 ceph-mon[37000]: from='mgr.15333 172.21.15.52:0/416389082' entity='mgr.x' 2024-05-27T07:26:00.352 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:25:59 smithi173 ceph-mon[37000]: from='client.? 172.21.15.52:0/2620532663' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-05-27T07:26:00.352 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:25:59 smithi173 ceph-mon[37000]: from='mgr.15333 172.21.15.52:0/416389082' entity='mgr.x' 2024-05-27T07:26:00.352 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:25:59 smithi173 ceph-mon[37000]: from='mgr.15333 172.21.15.52:0/416389082' entity='mgr.x' 2024-05-27T07:26:00.352 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:25:59 smithi173 ceph-mon[37000]: from='mgr.15333 172.21.15.52:0/416389082' entity='mgr.x' 2024-05-27T07:26:00.353 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:25:59 smithi173 ceph-mon[37000]: from='mgr.15333 172.21.15.52:0/416389082' entity='mgr.x' 2024-05-27T07:26:00.353 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:25:59 smithi173 ceph-mon[37000]: from='client.? 172.21.15.52:0/3580314251' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-05-27T07:26:00.353 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:25:59 smithi173 ceph-mon[37000]: from='client.? 172.21.15.52:0/2630188342' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-05-27T07:26:01.229 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:26:01 smithi052 ceph-mon[31955]: from='mgr.15333 172.21.15.52:0/416389082' entity='mgr.x' 2024-05-27T07:26:01.229 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:26:01 smithi052 ceph-mon[31955]: from='mgr.15333 172.21.15.52:0/416389082' entity='mgr.x' 2024-05-27T07:26:01.229 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:26:01 smithi052 ceph-mon[31955]: from='mgr.15333 172.21.15.52:0/416389082' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-05-27T07:26:01.229 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:26:01 smithi052 ceph-mon[31955]: from='mgr.15333 172.21.15.52:0/416389082' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-05-27T07:26:01.230 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:26:01 smithi052 ceph-mon[31955]: from='mgr.15333 172.21.15.52:0/416389082' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-05-27T07:26:01.230 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:26:01 smithi052 ceph-mon[31955]: from='mgr.15333 172.21.15.52:0/416389082' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-05-27T07:26:01.230 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:26:01 smithi052 ceph-mon[31955]: Adjusting osd_memory_target on smithi173 to 176.2M 2024-05-27T07:26:01.230 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:26:01 smithi052 ceph-mon[31955]: Unable to set osd_memory_target on smithi173 to 184791040: error parsing value: Value '184791040' is below minimum 939524096 2024-05-27T07:26:01.230 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:26:01 smithi052 ceph-mon[31955]: from='mgr.15333 172.21.15.52:0/416389082' entity='mgr.x' 2024-05-27T07:26:01.230 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:26:01 smithi052 ceph-mon[31955]: from='mgr.15333 172.21.15.52:0/416389082' entity='mgr.x' 2024-05-27T07:26:01.230 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:26:01 smithi052 ceph-mon[31955]: from='mgr.15333 172.21.15.52:0/416389082' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-05-27T07:26:01.230 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:26:01 smithi052 ceph-mon[31955]: from='mgr.15333 172.21.15.52:0/416389082' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-05-27T07:26:01.230 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:26:01 smithi052 ceph-mon[31955]: from='mgr.15333 172.21.15.52:0/416389082' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-05-27T07:26:01.230 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:26:01 smithi052 ceph-mon[31955]: from='mgr.15333 172.21.15.52:0/416389082' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-05-27T07:26:01.230 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:26:01 smithi052 ceph-mon[31955]: Adjusting osd_memory_target on smithi105 to 176.2M 2024-05-27T07:26:01.231 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:26:01 smithi052 ceph-mon[31955]: Unable to set osd_memory_target on smithi105 to 184784588: error parsing value: Value '184784588' is below minimum 939524096 2024-05-27T07:26:01.231 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:26:01 smithi052 ceph-mon[31955]: from='mgr.15333 172.21.15.52:0/416389082' entity='mgr.x' 2024-05-27T07:26:01.231 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:26:01 smithi052 ceph-mon[31955]: from='mgr.15333 172.21.15.52:0/416389082' entity='mgr.x' 2024-05-27T07:26:01.231 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:26:01 smithi052 ceph-mon[31955]: from='mgr.15333 172.21.15.52:0/416389082' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-05-27T07:26:01.231 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:26:01 smithi052 ceph-mon[31955]: from='mgr.15333 172.21.15.52:0/416389082' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-05-27T07:26:01.231 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:26:01 smithi052 ceph-mon[31955]: from='mgr.15333 172.21.15.52:0/416389082' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-05-27T07:26:01.231 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:26:01 smithi052 ceph-mon[31955]: from='mgr.15333 172.21.15.52:0/416389082' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-05-27T07:26:01.231 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:26:01 smithi052 ceph-mon[31955]: from='mgr.15333 172.21.15.52:0/416389082' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T07:26:01.231 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:26:01 smithi052 ceph-mon[31955]: from='mgr.15333 172.21.15.52:0/416389082' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-27T07:26:01.352 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:26:01 smithi173 ceph-mon[37000]: from='mgr.15333 172.21.15.52:0/416389082' entity='mgr.x' 2024-05-27T07:26:01.352 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:26:01 smithi173 ceph-mon[37000]: from='mgr.15333 172.21.15.52:0/416389082' entity='mgr.x' 2024-05-27T07:26:01.352 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:26:01 smithi173 ceph-mon[37000]: from='mgr.15333 172.21.15.52:0/416389082' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-05-27T07:26:01.352 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:26:01 smithi173 ceph-mon[37000]: from='mgr.15333 172.21.15.52:0/416389082' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-05-27T07:26:01.352 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:26:01 smithi173 ceph-mon[37000]: from='mgr.15333 172.21.15.52:0/416389082' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-05-27T07:26:01.352 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:26:01 smithi173 ceph-mon[37000]: from='mgr.15333 172.21.15.52:0/416389082' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-05-27T07:26:01.353 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:26:01 smithi173 ceph-mon[37000]: Adjusting osd_memory_target on smithi173 to 176.2M 2024-05-27T07:26:01.353 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:26:01 smithi173 ceph-mon[37000]: Unable to set osd_memory_target on smithi173 to 184791040: error parsing value: Value '184791040' is below minimum 939524096 2024-05-27T07:26:01.353 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:26:01 smithi173 ceph-mon[37000]: from='mgr.15333 172.21.15.52:0/416389082' entity='mgr.x' 2024-05-27T07:26:01.353 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:26:01 smithi173 ceph-mon[37000]: from='mgr.15333 172.21.15.52:0/416389082' entity='mgr.x' 2024-05-27T07:26:01.353 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:26:01 smithi173 ceph-mon[37000]: from='mgr.15333 172.21.15.52:0/416389082' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-05-27T07:26:01.353 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:26:01 smithi173 ceph-mon[37000]: from='mgr.15333 172.21.15.52:0/416389082' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-05-27T07:26:01.353 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:26:01 smithi173 ceph-mon[37000]: from='mgr.15333 172.21.15.52:0/416389082' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-05-27T07:26:01.353 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:26:01 smithi173 ceph-mon[37000]: from='mgr.15333 172.21.15.52:0/416389082' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-05-27T07:26:01.353 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:26:01 smithi173 ceph-mon[37000]: Adjusting osd_memory_target on smithi105 to 176.2M 2024-05-27T07:26:01.354 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:26:01 smithi173 ceph-mon[37000]: Unable to set osd_memory_target on smithi105 to 184784588: error parsing value: Value '184784588' is below minimum 939524096 2024-05-27T07:26:01.354 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:26:01 smithi173 ceph-mon[37000]: from='mgr.15333 172.21.15.52:0/416389082' entity='mgr.x' 2024-05-27T07:26:01.354 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:26:01 smithi173 ceph-mon[37000]: from='mgr.15333 172.21.15.52:0/416389082' entity='mgr.x' 2024-05-27T07:26:01.354 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:26:01 smithi173 ceph-mon[37000]: from='mgr.15333 172.21.15.52:0/416389082' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-05-27T07:26:01.354 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:26:01 smithi173 ceph-mon[37000]: from='mgr.15333 172.21.15.52:0/416389082' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-05-27T07:26:01.354 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:26:01 smithi173 ceph-mon[37000]: from='mgr.15333 172.21.15.52:0/416389082' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-05-27T07:26:01.354 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:26:01 smithi173 ceph-mon[37000]: from='mgr.15333 172.21.15.52:0/416389082' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-05-27T07:26:01.354 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:26:01 smithi173 ceph-mon[37000]: from='mgr.15333 172.21.15.52:0/416389082' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T07:26:01.354 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:26:01 smithi173 ceph-mon[37000]: from='mgr.15333 172.21.15.52:0/416389082' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-27T07:26:01.440 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:26:01 smithi105 ceph-mon[35977]: from='mgr.15333 172.21.15.52:0/416389082' entity='mgr.x' 2024-05-27T07:26:01.440 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:26:01 smithi105 ceph-mon[35977]: from='mgr.15333 172.21.15.52:0/416389082' entity='mgr.x' 2024-05-27T07:26:01.440 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:26:01 smithi105 ceph-mon[35977]: from='mgr.15333 172.21.15.52:0/416389082' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-05-27T07:26:01.440 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:26:01 smithi105 ceph-mon[35977]: from='mgr.15333 172.21.15.52:0/416389082' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-05-27T07:26:01.440 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:26:01 smithi105 ceph-mon[35977]: from='mgr.15333 172.21.15.52:0/416389082' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-05-27T07:26:01.440 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:26:01 smithi105 ceph-mon[35977]: from='mgr.15333 172.21.15.52:0/416389082' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-05-27T07:26:01.440 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:26:01 smithi105 ceph-mon[35977]: Adjusting osd_memory_target on smithi173 to 176.2M 2024-05-27T07:26:01.440 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:26:01 smithi105 ceph-mon[35977]: Unable to set osd_memory_target on smithi173 to 184791040: error parsing value: Value '184791040' is below minimum 939524096 2024-05-27T07:26:01.440 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:26:01 smithi105 ceph-mon[35977]: from='mgr.15333 172.21.15.52:0/416389082' entity='mgr.x' 2024-05-27T07:26:01.440 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:26:01 smithi105 ceph-mon[35977]: from='mgr.15333 172.21.15.52:0/416389082' entity='mgr.x' 2024-05-27T07:26:01.441 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:26:01 smithi105 ceph-mon[35977]: from='mgr.15333 172.21.15.52:0/416389082' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-05-27T07:26:01.441 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:26:01 smithi105 ceph-mon[35977]: from='mgr.15333 172.21.15.52:0/416389082' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-05-27T07:26:01.441 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:26:01 smithi105 ceph-mon[35977]: from='mgr.15333 172.21.15.52:0/416389082' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-05-27T07:26:01.441 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:26:01 smithi105 ceph-mon[35977]: from='mgr.15333 172.21.15.52:0/416389082' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-05-27T07:26:01.441 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:26:01 smithi105 ceph-mon[35977]: Adjusting osd_memory_target on smithi105 to 176.2M 2024-05-27T07:26:01.441 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:26:01 smithi105 ceph-mon[35977]: Unable to set osd_memory_target on smithi105 to 184784588: error parsing value: Value '184784588' is below minimum 939524096 2024-05-27T07:26:01.441 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:26:01 smithi105 ceph-mon[35977]: from='mgr.15333 172.21.15.52:0/416389082' entity='mgr.x' 2024-05-27T07:26:01.441 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:26:01 smithi105 ceph-mon[35977]: from='mgr.15333 172.21.15.52:0/416389082' entity='mgr.x' 2024-05-27T07:26:01.441 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:26:01 smithi105 ceph-mon[35977]: from='mgr.15333 172.21.15.52:0/416389082' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-05-27T07:26:01.441 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:26:01 smithi105 ceph-mon[35977]: from='mgr.15333 172.21.15.52:0/416389082' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-05-27T07:26:01.441 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:26:01 smithi105 ceph-mon[35977]: from='mgr.15333 172.21.15.52:0/416389082' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-05-27T07:26:01.441 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:26:01 smithi105 ceph-mon[35977]: from='mgr.15333 172.21.15.52:0/416389082' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-05-27T07:26:01.441 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:26:01 smithi105 ceph-mon[35977]: from='mgr.15333 172.21.15.52:0/416389082' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T07:26:01.441 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:26:01 smithi105 ceph-mon[35977]: from='mgr.15333 172.21.15.52:0/416389082' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-27T07:26:01.843 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:01.853+0000 7fbc72246700 1 -- 172.21.15.52:0/3552768669 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fbc6c107de0 msgr2=0x7fbc6c1081f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:26:01.843 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:01.853+0000 7fbc72246700 1 --2- 172.21.15.52:0/3552768669 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fbc6c107de0 0x7fbc6c1081f0 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7fbc60009920 tx=0x7fbc60009c30 comp rx=0 tx=0).stop 2024-05-27T07:26:01.843 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:01.853+0000 7fbc72246700 1 -- 172.21.15.52:0/3552768669 shutdown_connections 2024-05-27T07:26:01.843 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:01.853+0000 7fbc72246700 1 --2- 172.21.15.52:0/3552768669 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fbc6c108730 0x7fbc6c110090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:26:01.844 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:01.853+0000 7fbc72246700 1 --2- 172.21.15.52:0/3552768669 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fbc6c1034b0 0x7fbc6c1038c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:26:01.844 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:01.853+0000 7fbc72246700 1 --2- 172.21.15.52:0/3552768669 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fbc6c107de0 0x7fbc6c1081f0 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:26:01.844 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:01.853+0000 7fbc72246700 1 -- 172.21.15.52:0/3552768669 >> 172.21.15.52:0/3552768669 conn(0x7fbc6c0fedb0 msgr2=0x7fbc6c1011d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:26:01.844 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:01.853+0000 7fbc72246700 1 -- 172.21.15.52:0/3552768669 shutdown_connections 2024-05-27T07:26:01.844 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:01.853+0000 7fbc72246700 1 -- 172.21.15.52:0/3552768669 wait complete. 2024-05-27T07:26:01.845 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:01.854+0000 7fbc72246700 1 Processor -- start 2024-05-27T07:26:01.846 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:01.854+0000 7fbc72246700 1 -- start start 2024-05-27T07:26:01.846 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:01.854+0000 7fbc72246700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fbc6c1034b0 0x7fbc6c19d0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:26:01.846 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:01.854+0000 7fbc72246700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fbc6c107de0 0x7fbc6c19d690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:26:01.846 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:01.855+0000 7fbc72246700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fbc6c108730 0x7fbc6c1a1920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:26:01.846 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:01.855+0000 7fbc72246700 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x7fbc6c116680 con 0x7fbc6c107de0 2024-05-27T07:26:01.846 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:01.855+0000 7fbc72246700 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7fbc6c116500 con 0x7fbc6c108730 2024-05-27T07:26:01.846 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:01.855+0000 7fbc72246700 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7fbc6c116800 con 0x7fbc6c1034b0 2024-05-27T07:26:01.846 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:01.855+0000 7fbc6affd700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fbc6c107de0 0x7fbc6c19d690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:26:01.846 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:01.855+0000 7fbc6b7fe700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fbc6c1034b0 0x7fbc6c19d0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:26:01.846 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:01.855+0000 7fbc6affd700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fbc6c107de0 0x7fbc6c19d690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.52:58390/0 (socket says 172.21.15.52:58390) 2024-05-27T07:26:01.846 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:01.855+0000 7fbc6affd700 1 -- 172.21.15.52:0/3248081509 learned_addr learned my addr 172.21.15.52:0/3248081509 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-05-27T07:26:01.847 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:01.855+0000 7fbc6bfff700 1 --2- 172.21.15.52:0/3248081509 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fbc6c108730 0x7fbc6c1a1920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:26:01.847 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:01.855+0000 7fbc6bfff700 1 -- 172.21.15.52:0/3248081509 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fbc6c1034b0 msgr2=0x7fbc6c19d0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:26:01.847 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:01.855+0000 7fbc6bfff700 1 --2- 172.21.15.52:0/3248081509 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fbc6c1034b0 0x7fbc6c19d0f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:26:01.847 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:01.855+0000 7fbc6bfff700 1 -- 172.21.15.52:0/3248081509 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fbc6c107de0 msgr2=0x7fbc6c19d690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:26:01.847 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:01.855+0000 7fbc6bfff700 1 --2- 172.21.15.52:0/3248081509 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fbc6c107de0 0x7fbc6c19d690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:26:01.847 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:01.855+0000 7fbc6bfff700 1 -- 172.21.15.52:0/3248081509 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbc6c1a1fc0 con 0x7fbc6c108730 2024-05-27T07:26:01.847 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:01.855+0000 7fbc6affd700 1 --2- 172.21.15.52:0/3248081509 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fbc6c107de0 0x7fbc6c19d690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-27T07:26:01.847 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:01.855+0000 7fbc6b7fe700 1 --2- 172.21.15.52:0/3248081509 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fbc6c1034b0 0x7fbc6c19d0f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-27T07:26:01.847 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:01.855+0000 7fbc6bfff700 1 --2- 172.21.15.52:0/3248081509 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fbc6c108730 0x7fbc6c1a1920 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7fbc64009cd0 tx=0x7fbc6400c620 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:26:01.847 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:01.856+0000 7fbc68ff9700 1 -- 172.21.15.52:0/3248081509 <== mon.2 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbc6400cd10 con 0x7fbc6c108730 2024-05-27T07:26:01.847 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:01.856+0000 7fbc72246700 1 -- 172.21.15.52:0/3248081509 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbc6c1a2250 con 0x7fbc6c108730 2024-05-27T07:26:01.847 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:01.856+0000 7fbc68ff9700 1 -- 172.21.15.52:0/3248081509 <== mon.2 v2:172.21.15.105:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fbc64003a00 con 0x7fbc6c108730 2024-05-27T07:26:01.847 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:01.856+0000 7fbc68ff9700 1 -- 172.21.15.52:0/3248081509 <== mon.2 v2:172.21.15.105:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbc64004520 con 0x7fbc6c108730 2024-05-27T07:26:01.848 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:01.856+0000 7fbc72246700 1 -- 172.21.15.52:0/3248081509 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbc6c1a2740 con 0x7fbc6c108730 2024-05-27T07:26:01.848 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:01.857+0000 7fbc72246700 1 -- 172.21.15.52:0/3248081509 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fbc6c0512b0 con 0x7fbc6c108730 2024-05-27T07:26:01.848 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:01.858+0000 7fbc68ff9700 1 -- 172.21.15.52:0/3248081509 <== mon.2 v2:172.21.15.105:3300/0 4 ==== mgrmap(e 24) ==== 149879+0+0 (secure 0 0 0) 0x7fbc6401a020 con 0x7fbc6c108730 2024-05-27T07:26:01.849 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:01.859+0000 7fbc68ff9700 1 --2- 172.21.15.52:0/3248081509 >> [v2:172.21.15.52:6800/1822318384,v1:172.21.15.52:6801/1822318384] conn(0x7fbc540b2af0 0x7fbc540b5000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:26:01.849 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:01.859+0000 7fbc6b7fe700 1 --2- 172.21.15.52:0/3248081509 >> [v2:172.21.15.52:6800/1822318384,v1:172.21.15.52:6801/1822318384] conn(0x7fbc540b2af0 0x7fbc540b5000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:26:01.849 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:01.859+0000 7fbc68ff9700 1 -- 172.21.15.52:0/3248081509 <== mon.2 v2:172.21.15.105:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9379+0+0 (secure 0 0 0) 0x7fbc640e2940 con 0x7fbc6c108730 2024-05-27T07:26:01.849 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:01.859+0000 7fbc6b7fe700 1 --2- 172.21.15.52:0/3248081509 >> [v2:172.21.15.52:6800/1822318384,v1:172.21.15.52:6801/1822318384] conn(0x7fbc540b2af0 0x7fbc540b5000 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7fbc5c011490 tx=0x7fbc5c009260 comp rx=0 tx=0).ready entity=mgr.15333 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:26:01.852 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:01.862+0000 7fbc68ff9700 1 -- 172.21.15.52:0/3248081509 <== mon.2 v2:172.21.15.105:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191963 (secure 0 0 0) 0x7fbc64010040 con 0x7fbc6c108730 2024-05-27T07:26:02.147 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:02.157+0000 7fbc72246700 1 -- 172.21.15.52:0/3248081509 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fbc6c02e5e0 con 0x7fbc6c108730 2024-05-27T07:26:02.148 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:26:02 smithi052 ceph-mon[31955]: Adjusting osd_memory_target on smithi052 to 177.2M 2024-05-27T07:26:02.148 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:26:02 smithi052 ceph-mon[31955]: Unable to set osd_memory_target on smithi052 to 185879142: error parsing value: Value '185879142' is below minimum 939524096 2024-05-27T07:26:02.148 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:26:02 smithi052 ceph-mon[31955]: Updating smithi052:/etc/ceph/ceph.conf 2024-05-27T07:26:02.148 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:26:02 smithi052 ceph-mon[31955]: Updating smithi105:/etc/ceph/ceph.conf 2024-05-27T07:26:02.148 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:26:02 smithi052 ceph-mon[31955]: Updating smithi173:/etc/ceph/ceph.conf 2024-05-27T07:26:02.148 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:26:02 smithi052 ceph-mon[31955]: pgmap v5: 129 pgs: 129 active+clean; 1.1 MiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-27T07:26:02.148 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:26:02 smithi052 ceph-mon[31955]: Updating smithi105:/var/lib/ceph/73df2636-1bf8-11ef-bc9b-c7b262605968/config/ceph.conf 2024-05-27T07:26:02.148 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:26:02 smithi052 ceph-mon[31955]: Updating smithi052:/var/lib/ceph/73df2636-1bf8-11ef-bc9b-c7b262605968/config/ceph.conf 2024-05-27T07:26:02.148 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:26:02 smithi052 ceph-mon[31955]: Updating smithi173:/var/lib/ceph/73df2636-1bf8-11ef-bc9b-c7b262605968/config/ceph.conf 2024-05-27T07:26:02.148 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:26:02 smithi052 ceph-mon[31955]: mgrmap e24: x(active, since 4s), standbys: z, y 2024-05-27T07:26:02.148 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:26:02 smithi052 ceph-mon[31955]: from='mgr.15333 172.21.15.52:0/416389082' entity='mgr.x' 2024-05-27T07:26:02.148 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:26:02 smithi052 ceph-mon[31955]: from='mgr.15333 172.21.15.52:0/416389082' entity='mgr.x' 2024-05-27T07:26:02.149 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:26:02 smithi052 ceph-mon[31955]: from='mgr.15333 172.21.15.52:0/416389082' entity='mgr.x' 2024-05-27T07:26:02.149 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:26:02 smithi052 ceph-mon[31955]: from='mgr.15333 172.21.15.52:0/416389082' entity='mgr.x' 2024-05-27T07:26:02.149 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:26:02 smithi052 ceph-mon[31955]: from='mgr.15333 172.21.15.52:0/416389082' entity='mgr.x' 2024-05-27T07:26:02.149 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:26:02 smithi052 ceph-mon[31955]: from='mgr.15333 172.21.15.52:0/416389082' entity='mgr.x' 2024-05-27T07:26:02.149 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:26:02 smithi052 ceph-mon[31955]: from='mgr.15333 172.21.15.52:0/416389082' entity='mgr.x' 2024-05-27T07:26:02.149 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:02.158+0000 7fbc68ff9700 1 -- 172.21.15.52:0/3248081509 <== mon.2 v2:172.21.15.105:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 30 v30) ==== 94+0+11938 (secure 0 0 0) 0x7fbc640a9d70 con 0x7fbc6c108730 2024-05-27T07:26:02.149 INFO:teuthology.orchestra.run.smithi052.stdout: 2024-05-27T07:26:02.150 INFO:teuthology.orchestra.run.smithi052.stdout:{"epoch":30,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14556,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.52:6835/2584733778","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.52:6834","nonce":2584733778},{"type":"v1","addr":"172.21.15.52:6835","nonce":2584733778}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14568,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.52:6837/675445303","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.52:6836","nonce":675445303},{"type":"v1","addr":"172.21.15.52:6837","nonce":675445303}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":30,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-05-27T07:21:50.521783+0000","modified":"2024-05-27T07:25:54.323921+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24527,"mds_1":24497,"mds_2":24475,"mds_3":24473,"mds_4":24466},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14619":{"gid":14619,"name":"g","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.52:6839/4258549559","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.52:6838","nonce":4258549559},{"type":"v1","addr":"172.21.15.52:6839","nonce":4258549559}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_14631":{"gid":14631,"name":"a","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.52:6841/2878618188","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.52:6840","nonce":2878618188},{"type":"v1","addr":"172.21.15.52:6841","nonce":2878618188}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24433":{"gid":24433,"name":"l","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.173:6833/3013589220","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6832","nonce":3013589220},{"type":"v1","addr":"172.21.15.173:6833","nonce":3013589220}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24452":{"gid":24452,"name":"k","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.105:6833/2800678412","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6832","nonce":2800678412},{"type":"v1","addr":"172.21.15.105:6833","nonce":2800678412}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24460":{"gid":24460,"name":"f","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.173:6835/1619486437","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6834","nonce":1619486437},{"type":"v1","addr":"172.21.15.173:6835","nonce":1619486437}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24466":{"gid":24466,"name":"c","rank":4,"incarnation":26,"state":"up:active","state_seq":25,"addr":"172.21.15.173:6837/1396363827","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6836","nonce":1396363827},{"type":"v1","addr":"172.21.15.173:6837","nonce":1396363827}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24473":{"gid":24473,"name":"e","rank":3,"incarnation":24,"state":"up:active","state_seq":26,"addr":"172.21.15.105:6835/3298498384","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6834","nonce":3298498384},{"type":"v1","addr":"172.21.15.105:6835","nonce":3298498384}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24475":{"gid":24475,"name":"i","rank":2,"incarnation":22,"state":"up:active","state_seq":23,"addr":"172.21.15.173:6839/564778293","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6838","nonce":564778293},{"type":"v1","addr":"172.21.15.173:6839","nonce":564778293}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24497":{"gid":24497,"name":"h","rank":1,"incarnation":20,"state":"up:active","state_seq":24,"addr":"172.21.15.105:6837/191393149","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6836","nonce":191393149},{"type":"v1","addr":"172.21.15.105:6837","nonce":191393149}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24527":{"gid":24527,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":5,"addr":"172.21.15.105:6839/500348151","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6838","nonce":500348151},{"type":"v1","addr":"172.21.15.105:6839","nonce":500348151}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24527,"qdb_cluster":[24527,24497,24475,24466,24473]},"id":1}]} 2024-05-27T07:26:02.150 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:02.160+0000 7fbc72246700 1 -- 172.21.15.52:0/3248081509 >> [v2:172.21.15.52:6800/1822318384,v1:172.21.15.52:6801/1822318384] conn(0x7fbc540b2af0 msgr2=0x7fbc540b5000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:26:02.150 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:02.160+0000 7fbc72246700 1 --2- 172.21.15.52:0/3248081509 >> [v2:172.21.15.52:6800/1822318384,v1:172.21.15.52:6801/1822318384] conn(0x7fbc540b2af0 0x7fbc540b5000 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7fbc5c011490 tx=0x7fbc5c009260 comp rx=0 tx=0).stop 2024-05-27T07:26:02.151 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:02.160+0000 7fbc72246700 1 -- 172.21.15.52:0/3248081509 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fbc6c108730 msgr2=0x7fbc6c1a1920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:26:02.151 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:02.160+0000 7fbc72246700 1 --2- 172.21.15.52:0/3248081509 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fbc6c108730 0x7fbc6c1a1920 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7fbc64009cd0 tx=0x7fbc6400c620 comp rx=0 tx=0).stop 2024-05-27T07:26:02.151 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:02.161+0000 7fbc72246700 1 -- 172.21.15.52:0/3248081509 shutdown_connections 2024-05-27T07:26:02.151 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:02.161+0000 7fbc72246700 1 --2- 172.21.15.52:0/3248081509 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fbc6c108730 0x7fbc6c1a1920 unknown :-1 s=CLOSED pgs=69 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:26:02.151 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:02.161+0000 7fbc72246700 1 --2- 172.21.15.52:0/3248081509 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fbc6c1034b0 0x7fbc6c19d0f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:26:02.151 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:02.161+0000 7fbc72246700 1 --2- 172.21.15.52:0/3248081509 >> [v2:172.21.15.52:6800/1822318384,v1:172.21.15.52:6801/1822318384] conn(0x7fbc540b2af0 0x7fbc540b5000 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:26:02.151 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:02.161+0000 7fbc72246700 1 --2- 172.21.15.52:0/3248081509 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fbc6c107de0 0x7fbc6c19d690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:26:02.151 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:02.161+0000 7fbc72246700 1 -- 172.21.15.52:0/3248081509 >> 172.21.15.52:0/3248081509 conn(0x7fbc6c0fedb0 msgr2=0x7fbc6c1011d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:26:02.151 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:02.161+0000 7fbc72246700 1 -- 172.21.15.52:0/3248081509 shutdown_connections 2024-05-27T07:26:02.151 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:02.161+0000 7fbc72246700 1 -- 172.21.15.52:0/3248081509 wait complete. 2024-05-27T07:26:02.152 INFO:teuthology.orchestra.run.smithi052.stderr:dumped fsmap epoch 30 2024-05-27T07:26:02.352 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:26:02 smithi173 ceph-mon[37000]: Adjusting osd_memory_target on smithi052 to 177.2M 2024-05-27T07:26:02.352 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:26:02 smithi173 ceph-mon[37000]: Unable to set osd_memory_target on smithi052 to 185879142: error parsing value: Value '185879142' is below minimum 939524096 2024-05-27T07:26:02.352 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:26:02 smithi173 ceph-mon[37000]: Updating smithi052:/etc/ceph/ceph.conf 2024-05-27T07:26:02.352 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:26:02 smithi173 ceph-mon[37000]: Updating smithi105:/etc/ceph/ceph.conf 2024-05-27T07:26:02.352 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:26:02 smithi173 ceph-mon[37000]: Updating smithi173:/etc/ceph/ceph.conf 2024-05-27T07:26:02.352 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:26:02 smithi173 ceph-mon[37000]: pgmap v5: 129 pgs: 129 active+clean; 1.1 MiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-27T07:26:02.352 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:26:02 smithi173 ceph-mon[37000]: Updating smithi105:/var/lib/ceph/73df2636-1bf8-11ef-bc9b-c7b262605968/config/ceph.conf 2024-05-27T07:26:02.352 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:26:02 smithi173 ceph-mon[37000]: Updating smithi052:/var/lib/ceph/73df2636-1bf8-11ef-bc9b-c7b262605968/config/ceph.conf 2024-05-27T07:26:02.352 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:26:02 smithi173 ceph-mon[37000]: Updating smithi173:/var/lib/ceph/73df2636-1bf8-11ef-bc9b-c7b262605968/config/ceph.conf 2024-05-27T07:26:02.352 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:26:02 smithi173 ceph-mon[37000]: mgrmap e24: x(active, since 4s), standbys: z, y 2024-05-27T07:26:02.352 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:26:02 smithi173 ceph-mon[37000]: from='mgr.15333 172.21.15.52:0/416389082' entity='mgr.x' 2024-05-27T07:26:02.352 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:26:02 smithi173 ceph-mon[37000]: from='mgr.15333 172.21.15.52:0/416389082' entity='mgr.x' 2024-05-27T07:26:02.353 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:26:02 smithi173 ceph-mon[37000]: from='mgr.15333 172.21.15.52:0/416389082' entity='mgr.x' 2024-05-27T07:26:02.353 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:26:02 smithi173 ceph-mon[37000]: from='mgr.15333 172.21.15.52:0/416389082' entity='mgr.x' 2024-05-27T07:26:02.353 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:26:02 smithi173 ceph-mon[37000]: from='mgr.15333 172.21.15.52:0/416389082' entity='mgr.x' 2024-05-27T07:26:02.353 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:26:02 smithi173 ceph-mon[37000]: from='mgr.15333 172.21.15.52:0/416389082' entity='mgr.x' 2024-05-27T07:26:02.353 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:26:02 smithi173 ceph-mon[37000]: from='mgr.15333 172.21.15.52:0/416389082' entity='mgr.x' 2024-05-27T07:26:02.440 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:26:02 smithi105 ceph-mon[35977]: Adjusting osd_memory_target on smithi052 to 177.2M 2024-05-27T07:26:02.440 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:26:02 smithi105 ceph-mon[35977]: Unable to set osd_memory_target on smithi052 to 185879142: error parsing value: Value '185879142' is below minimum 939524096 2024-05-27T07:26:02.440 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:26:02 smithi105 ceph-mon[35977]: Updating smithi052:/etc/ceph/ceph.conf 2024-05-27T07:26:02.440 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:26:02 smithi105 ceph-mon[35977]: Updating smithi105:/etc/ceph/ceph.conf 2024-05-27T07:26:02.440 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:26:02 smithi105 ceph-mon[35977]: Updating smithi173:/etc/ceph/ceph.conf 2024-05-27T07:26:02.440 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:26:02 smithi105 ceph-mon[35977]: pgmap v5: 129 pgs: 129 active+clean; 1.1 MiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-27T07:26:02.440 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:26:02 smithi105 ceph-mon[35977]: Updating smithi105:/var/lib/ceph/73df2636-1bf8-11ef-bc9b-c7b262605968/config/ceph.conf 2024-05-27T07:26:02.440 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:26:02 smithi105 ceph-mon[35977]: Updating smithi052:/var/lib/ceph/73df2636-1bf8-11ef-bc9b-c7b262605968/config/ceph.conf 2024-05-27T07:26:02.440 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:26:02 smithi105 ceph-mon[35977]: Updating smithi173:/var/lib/ceph/73df2636-1bf8-11ef-bc9b-c7b262605968/config/ceph.conf 2024-05-27T07:26:02.440 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:26:02 smithi105 ceph-mon[35977]: mgrmap e24: x(active, since 4s), standbys: z, y 2024-05-27T07:26:02.441 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:26:02 smithi105 ceph-mon[35977]: from='mgr.15333 172.21.15.52:0/416389082' entity='mgr.x' 2024-05-27T07:26:02.441 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:26:02 smithi105 ceph-mon[35977]: from='mgr.15333 172.21.15.52:0/416389082' entity='mgr.x' 2024-05-27T07:26:02.441 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:26:02 smithi105 ceph-mon[35977]: from='mgr.15333 172.21.15.52:0/416389082' entity='mgr.x' 2024-05-27T07:26:02.441 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:26:02 smithi105 ceph-mon[35977]: from='mgr.15333 172.21.15.52:0/416389082' entity='mgr.x' 2024-05-27T07:26:02.441 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:26:02 smithi105 ceph-mon[35977]: from='mgr.15333 172.21.15.52:0/416389082' entity='mgr.x' 2024-05-27T07:26:02.441 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:26:02 smithi105 ceph-mon[35977]: from='mgr.15333 172.21.15.52:0/416389082' entity='mgr.x' 2024-05-27T07:26:02.441 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:26:02 smithi105 ceph-mon[35977]: from='mgr.15333 172.21.15.52:0/416389082' entity='mgr.x' 2024-05-27T07:26:02.957 INFO:tasks.fwd_scrub:Ready to start scrub thrashing 2024-05-27T07:26:02.957 INFO:tasks.fwd_scrub.ceph_manager:waiting for clean 2024-05-27T07:26:02.957 DEBUG:teuthology.orchestra.run.smithi052:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-05-27T07:26:03.078 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:03.087+0000 7faf5f00d640 1 -- 172.21.15.52:0/2935555281 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7faf58108fd0 msgr2=0x7faf58102c60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:26:03.078 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:03.087+0000 7faf5f00d640 1 --2- 172.21.15.52:0/2935555281 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7faf58108fd0 0x7faf58102c60 secure :-1 s=READY pgs=414 cs=0 l=1 rev1=1 crypto rx=0x7faf44009900 tx=0x7faf4402a410 comp rx=0 tx=0).stop 2024-05-27T07:26:03.078 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:03.088+0000 7faf5f00d640 1 -- 172.21.15.52:0/2935555281 shutdown_connections 2024-05-27T07:26:03.078 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:03.088+0000 7faf5f00d640 1 --2- 172.21.15.52:0/2935555281 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7faf581032d0 0x7faf5810f650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:26:03.078 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:03.088+0000 7faf5f00d640 1 --2- 172.21.15.52:0/2935555281 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7faf58108fd0 0x7faf58102c60 unknown :-1 s=CLOSED pgs=414 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:26:03.079 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:03.088+0000 7faf5f00d640 1 --2- 172.21.15.52:0/2935555281 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7faf58108b30 0x7faf58102720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:26:03.079 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:03.088+0000 7faf5f00d640 1 -- 172.21.15.52:0/2935555281 >> 172.21.15.52:0/2935555281 conn(0x7faf58078160 msgr2=0x7faf580785a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:26:03.079 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:03.088+0000 7faf5f00d640 1 -- 172.21.15.52:0/2935555281 shutdown_connections 2024-05-27T07:26:03.079 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:03.088+0000 7faf5f00d640 1 -- 172.21.15.52:0/2935555281 wait complete. 2024-05-27T07:26:03.079 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:03.088+0000 7faf5f00d640 1 Processor -- start 2024-05-27T07:26:03.079 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:03.088+0000 7faf5f00d640 1 -- start start 2024-05-27T07:26:03.081 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:03.089+0000 7faf5f00d640 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7faf581032d0 0x7faf5819b720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:26:03.081 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:03.089+0000 7faf5f00d640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7faf58108b30 0x7faf5819bc60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:26:03.081 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:03.089+0000 7faf5f00d640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7faf58108fd0 0x7faf5819c1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:26:03.081 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:03.089+0000 7faf5f00d640 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x7faf58112d20 con 0x7faf581032d0 2024-05-27T07:26:03.081 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:03.089+0000 7faf5f00d640 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7faf58112ba0 con 0x7faf58108b30 2024-05-27T07:26:03.081 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:03.089+0000 7faf5f00d640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7faf58112ea0 con 0x7faf58108fd0 2024-05-27T07:26:03.081 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:03.089+0000 7faf5cda9640 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7faf581032d0 0x7faf5819b720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:26:03.082 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:03.089+0000 7faf5cda9640 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7faf581032d0 0x7faf5819b720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.52:58412/0 (socket says 172.21.15.52:58412) 2024-05-27T07:26:03.082 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:03.089+0000 7faf5cda9640 1 -- 172.21.15.52:0/3553055372 learned_addr learned my addr 172.21.15.52:0/3553055372 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-05-27T07:26:03.082 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:03.089+0000 7faf5d5aa640 1 --2- 172.21.15.52:0/3553055372 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7faf58108fd0 0x7faf5819c1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:26:03.082 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:03.089+0000 7faf57fff640 1 --2- 172.21.15.52:0/3553055372 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7faf58108b30 0x7faf5819bc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:26:03.082 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:03.089+0000 7faf5cda9640 1 -- 172.21.15.52:0/3553055372 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7faf58108fd0 msgr2=0x7faf5819c1a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:26:03.082 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:03.089+0000 7faf5cda9640 1 --2- 172.21.15.52:0/3553055372 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7faf58108fd0 0x7faf5819c1a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:26:03.082 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:03.089+0000 7faf5cda9640 1 -- 172.21.15.52:0/3553055372 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7faf58108b30 msgr2=0x7faf5819bc60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:26:03.082 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:03.089+0000 7faf5cda9640 1 --2- 172.21.15.52:0/3553055372 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7faf58108b30 0x7faf5819bc60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:26:03.083 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:03.089+0000 7faf5cda9640 1 -- 172.21.15.52:0/3553055372 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7faf581a6a30 con 0x7faf581032d0 2024-05-27T07:26:03.083 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:03.089+0000 7faf57fff640 1 --2- 172.21.15.52:0/3553055372 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7faf58108b30 0x7faf5819bc60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-27T07:26:03.083 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:03.089+0000 7faf5d5aa640 1 --2- 172.21.15.52:0/3553055372 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7faf58108fd0 0x7faf5819c1a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-27T07:26:03.083 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:03.089+0000 7faf5cda9640 1 --2- 172.21.15.52:0/3553055372 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7faf581032d0 0x7faf5819b720 secure :-1 s=READY pgs=415 cs=0 l=1 rev1=1 crypto rx=0x7faf4c00c970 tx=0x7faf4c00ce30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:26:03.083 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:03.089+0000 7faf55ffb640 1 -- 172.21.15.52:0/3553055372 <== mon.0 v2:172.21.15.52:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faf4c007bb0 con 0x7faf581032d0 2024-05-27T07:26:03.083 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:03.090+0000 7faf55ffb640 1 -- 172.21.15.52:0/3553055372 <== mon.0 v2:172.21.15.52:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7faf4c007d50 con 0x7faf581032d0 2024-05-27T07:26:03.083 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:03.090+0000 7faf5f00d640 1 -- 172.21.15.52:0/3553055372 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7faf581a6d20 con 0x7faf581032d0 2024-05-27T07:26:03.083 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:03.090+0000 7faf55ffb640 1 -- 172.21.15.52:0/3553055372 <== mon.0 v2:172.21.15.52:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faf4c018730 con 0x7faf581032d0 2024-05-27T07:26:03.083 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:03.090+0000 7faf5f00d640 1 -- 172.21.15.52:0/3553055372 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7faf581a7190 con 0x7faf581032d0 2024-05-27T07:26:03.083 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:03.091+0000 7faf5f00d640 1 -- 172.21.15.52:0/3553055372 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7faf5806a2b0 con 0x7faf581032d0 2024-05-27T07:26:03.084 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:03.092+0000 7faf55ffb640 1 -- 172.21.15.52:0/3553055372 <== mon.0 v2:172.21.15.52:3300/0 4 ==== mgrmap(e 24) ==== 149879+0+0 (secure 0 0 0) 0x7faf4c00f990 con 0x7faf581032d0 2024-05-27T07:26:03.084 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:03.093+0000 7faf55ffb640 1 --2- 172.21.15.52:0/3553055372 >> [v2:172.21.15.52:6800/1822318384,v1:172.21.15.52:6801/1822318384] conn(0x7faf480b2a80 0x7faf480b4f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:26:03.084 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:03.093+0000 7faf57fff640 1 --2- 172.21.15.52:0/3553055372 >> [v2:172.21.15.52:6800/1822318384,v1:172.21.15.52:6801/1822318384] conn(0x7faf480b2a80 0x7faf480b4f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:26:03.084 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:03.093+0000 7faf55ffb640 1 -- 172.21.15.52:0/3553055372 <== mon.0 v2:172.21.15.52:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9379+0+0 (secure 0 0 0) 0x7faf4c0e2c80 con 0x7faf581032d0 2024-05-27T07:26:03.085 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:03.094+0000 7faf57fff640 1 --2- 172.21.15.52:0/3553055372 >> [v2:172.21.15.52:6800/1822318384,v1:172.21.15.52:6801/1822318384] conn(0x7faf480b2a80 0x7faf480b4f90 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7faf44009230 tx=0x7faf44000bc0 comp rx=0 tx=0).ready entity=mgr.15333 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:26:03.087 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:03.098+0000 7faf55ffb640 1 -- 172.21.15.52:0/3553055372 <== mon.0 v2:172.21.15.52:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191963 (secure 0 0 0) 0x7faf4c014030 con 0x7faf581032d0 2024-05-27T07:26:03.229 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:26:03 smithi052 ceph-mon[31955]: Updating smithi052:/etc/ceph/ceph.client.admin.keyring 2024-05-27T07:26:03.229 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:26:03 smithi052 ceph-mon[31955]: Updating smithi173:/etc/ceph/ceph.client.admin.keyring 2024-05-27T07:26:03.229 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:26:03 smithi052 ceph-mon[31955]: Updating smithi105:/etc/ceph/ceph.client.admin.keyring 2024-05-27T07:26:03.229 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:26:03 smithi052 ceph-mon[31955]: Updating smithi052:/var/lib/ceph/73df2636-1bf8-11ef-bc9b-c7b262605968/config/ceph.client.admin.keyring 2024-05-27T07:26:03.229 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:26:03 smithi052 ceph-mon[31955]: Updating smithi173:/var/lib/ceph/73df2636-1bf8-11ef-bc9b-c7b262605968/config/ceph.client.admin.keyring 2024-05-27T07:26:03.230 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:26:03 smithi052 ceph-mon[31955]: Updating smithi105:/var/lib/ceph/73df2636-1bf8-11ef-bc9b-c7b262605968/config/ceph.client.admin.keyring 2024-05-27T07:26:03.230 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:26:03 smithi052 ceph-mon[31955]: from='client.? 172.21.15.52:0/3248081509' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-27T07:26:03.230 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:26:03 smithi052 ceph-mon[31955]: from='mgr.15333 172.21.15.52:0/416389082' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-27T07:26:03.231 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:26:03 smithi052 ceph-mon[31955]: from='mgr.15333 172.21.15.52:0/416389082' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T07:26:03.231 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:26:03 smithi052 ceph-mon[31955]: from='mgr.15333 172.21.15.52:0/416389082' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-27T07:26:03.231 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:26:03 smithi052 ceph-mon[31955]: from='mgr.15333 172.21.15.52:0/416389082' entity='mgr.x' 2024-05-27T07:26:03.278 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:03.288+0000 7faf5f00d640 1 -- 172.21.15.52:0/3553055372 --> [v2:172.21.15.52:6800/1822318384,v1:172.21.15.52:6801/1822318384] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7faf58101a30 con 0x7faf480b2a80 2024-05-27T07:26:03.288 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:03.299+0000 7faf55ffb640 1 -- 172.21.15.52:0/3553055372 <== mgr.15333 v2:172.21.15.52:6800/1822318384 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+339710 (secure 0 0 0) 0x7faf58101a30 con 0x7faf480b2a80 2024-05-27T07:26:03.293 INFO:teuthology.orchestra.run.smithi052.stdout: 2024-05-27T07:26:03.295 INFO:teuthology.orchestra.run.smithi052.stderr:dumped all 2024-05-27T07:26:03.296 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:03.303+0000 7faf5f00d640 1 -- 172.21.15.52:0/3553055372 >> [v2:172.21.15.52:6800/1822318384,v1:172.21.15.52:6801/1822318384] conn(0x7faf480b2a80 msgr2=0x7faf480b4f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:26:03.297 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:03.303+0000 7faf5f00d640 1 --2- 172.21.15.52:0/3553055372 >> [v2:172.21.15.52:6800/1822318384,v1:172.21.15.52:6801/1822318384] conn(0x7faf480b2a80 0x7faf480b4f90 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7faf44009230 tx=0x7faf44000bc0 comp rx=0 tx=0).stop 2024-05-27T07:26:03.297 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:03.303+0000 7faf5f00d640 1 -- 172.21.15.52:0/3553055372 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7faf581032d0 msgr2=0x7faf5819b720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:26:03.297 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:03.303+0000 7faf5f00d640 1 --2- 172.21.15.52:0/3553055372 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7faf581032d0 0x7faf5819b720 secure :-1 s=READY pgs=415 cs=0 l=1 rev1=1 crypto rx=0x7faf4c00c970 tx=0x7faf4c00ce30 comp rx=0 tx=0).stop 2024-05-27T07:26:03.297 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:03.303+0000 7faf5f00d640 1 -- 172.21.15.52:0/3553055372 shutdown_connections 2024-05-27T07:26:03.297 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:03.303+0000 7faf5f00d640 1 --2- 172.21.15.52:0/3553055372 >> [v2:172.21.15.52:6800/1822318384,v1:172.21.15.52:6801/1822318384] conn(0x7faf480b2a80 0x7faf480b4f90 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:26:03.297 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:03.303+0000 7faf5f00d640 1 --2- 172.21.15.52:0/3553055372 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7faf58108fd0 0x7faf5819c1a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:26:03.297 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:03.303+0000 7faf5f00d640 1 --2- 172.21.15.52:0/3553055372 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7faf58108b30 0x7faf5819bc60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:26:03.297 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:03.303+0000 7faf5f00d640 1 --2- 172.21.15.52:0/3553055372 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7faf581032d0 0x7faf5819b720 unknown :-1 s=CLOSED pgs=415 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:26:03.297 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:03.303+0000 7faf5f00d640 1 -- 172.21.15.52:0/3553055372 >> 172.21.15.52:0/3553055372 conn(0x7faf58078160 msgr2=0x7faf58101320 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:26:03.297 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:03.303+0000 7faf5f00d640 1 -- 172.21.15.52:0/3553055372 shutdown_connections 2024-05-27T07:26:03.297 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:03.303+0000 7faf5f00d640 1 -- 172.21.15.52:0/3553055372 wait complete. 2024-05-27T07:26:03.304 INFO:teuthology.orchestra.run.smithi052.stdout:{"pg_ready":true,"pg_map":{"version":6,"stamp":"2024-05-27T07:26:02.949326+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":1195407,"num_objects":104,"num_object_clones":0,"num_object_copies":314,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":104,"num_whiteouts":0,"num_read":2035,"num_read_kb":1168,"num_write":511,"num_write_kb":3631,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":2951862,"num_keys_recovered":0,"num_objects_omap":66,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":905,"ondisk_log_size":905,"up":419,"acting":419,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":419,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":378332,"kb_used_data":12492,"kb_used_omap":18,"kb_used_meta":365677,"kb_avail":1124563492,"statfs":{"total":1151940427776,"available":1151553015808,"internally_reserved":0,"allocated":12791808,"data_stored":9769914,"data_compressed":602946,"data_compressed_allocated":815104,"data_compressed_original":4374528,"omap_allocated":19143,"internal_metadata":374453561},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":5,"num_read_kb":5,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.988934"},"pg_stats":[{"pgid":"3.3f","version":"90'8","reported_seq":49,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-27T07:25:57.024226+0000","last_change":"2024-05-27T07:21:48.595870+0000","last_active":"2024-05-27T07:25:57.024226+0000","last_peered":"2024-05-27T07:25:57.024226+0000","last_clean":"2024-05-27T07:25:57.024226+0000","last_became_active":"2024-05-27T07:21:48.595020+0000","last_became_peered":"2024-05-27T07:21:48.595020+0000","last_unstale":"2024-05-27T07:25:57.024226+0000","last_undegraded":"2024-05-27T07:25:57.024226+0000","last_fullsized":"2024-05-27T07:25:57.024226+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:21:47.565641+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:21:47.565641+0000","last_clean_scrub_stamp":"2024-05-27T07:21:47.565641+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T16:43:08.920344+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.3e","version":"90'4","reported_seq":46,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-27T07:25:57.026092+0000","last_change":"2024-05-27T07:21:48.590237+0000","last_active":"2024-05-27T07:25:57.026092+0000","last_peered":"2024-05-27T07:25:57.026092+0000","last_clean":"2024-05-27T07:25:57.026092+0000","last_became_active":"2024-05-27T07:21:48.589876+0000","last_became_peered":"2024-05-27T07:21:48.589876+0000","last_unstale":"2024-05-27T07:25:57.026092+0000","last_undegraded":"2024-05-27T07:25:57.026092+0000","last_fullsized":"2024-05-27T07:25:57.026092+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:21:47.565641+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:21:47.565641+0000","last_clean_scrub_stamp":"2024-05-27T07:21:47.565641+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T16:23:08.881871+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,11,10],"acting":[0,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3d","version":"90'7","reported_seq":49,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-27T07:25:57.025547+0000","last_change":"2024-05-27T07:21:48.597744+0000","last_active":"2024-05-27T07:25:57.025547+0000","last_peered":"2024-05-27T07:25:57.025547+0000","last_clean":"2024-05-27T07:25:57.025547+0000","last_became_active":"2024-05-27T07:21:48.597410+0000","last_became_peered":"2024-05-27T07:21:48.597410+0000","last_unstale":"2024-05-27T07:25:57.025547+0000","last_undegraded":"2024-05-27T07:25:57.025547+0000","last_fullsized":"2024-05-27T07:25:57.025547+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:21:47.565641+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:21:47.565641+0000","last_clean_scrub_stamp":"2024-05-27T07:21:47.565641+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T19:06:49.643152+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"90'8","reported_seq":26,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-27T07:25:56.931367+0000","last_change":"2024-05-27T07:22:30.800824+0000","last_active":"2024-05-27T07:25:56.931367+0000","last_peered":"2024-05-27T07:25:56.931367+0000","last_clean":"2024-05-27T07:25:56.931367+0000","last_became_active":"2024-05-27T07:22:30.800415+0000","last_became_peered":"2024-05-27T07:22:30.800415+0000","last_unstale":"2024-05-27T07:25:56.931367+0000","last_undegraded":"2024-05-27T07:25:56.931367+0000","last_fullsized":"2024-05-27T07:25:56.931367+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:21:47.565641+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:21:47.565641+0000","last_clean_scrub_stamp":"2024-05-27T07:21:47.565641+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T11:06:10.451686+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,2],"acting":[9,8,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.3b","version":"90'4","reported_seq":45,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-27T07:25:57.023025+0000","last_change":"2024-05-27T07:21:48.583219+0000","last_active":"2024-05-27T07:25:57.023025+0000","last_peered":"2024-05-27T07:25:57.023025+0000","last_clean":"2024-05-27T07:25:57.023025+0000","last_became_active":"2024-05-27T07:21:48.582813+0000","last_became_peered":"2024-05-27T07:21:48.582813+0000","last_unstale":"2024-05-27T07:25:57.023025+0000","last_undegraded":"2024-05-27T07:25:57.023025+0000","last_fullsized":"2024-05-27T07:25:57.023025+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:21:47.565641+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:21:47.565641+0000","last_clean_scrub_stamp":"2024-05-27T07:21:47.565641+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T18:05:05.770965+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,3,6],"acting":[8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.3a","version":"90'6","reported_seq":47,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-27T07:25:57.023067+0000","last_change":"2024-05-27T07:21:48.592049+0000","last_active":"2024-05-27T07:25:57.023067+0000","last_peered":"2024-05-27T07:25:57.023067+0000","last_clean":"2024-05-27T07:25:57.023067+0000","last_became_active":"2024-05-27T07:21:48.591677+0000","last_became_peered":"2024-05-27T07:21:48.591677+0000","last_unstale":"2024-05-27T07:25:57.023067+0000","last_undegraded":"2024-05-27T07:25:57.023067+0000","last_fullsized":"2024-05-27T07:25:57.023067+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:21:47.565641+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:21:47.565641+0000","last_clean_scrub_stamp":"2024-05-27T07:21:47.565641+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T16:25:49.755893+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,4],"acting":[10,9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.39","version":"90'5","reported_seq":44,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-27T07:25:57.021734+0000","last_change":"2024-05-27T07:21:48.597876+0000","last_active":"2024-05-27T07:25:57.021734+0000","last_peered":"2024-05-27T07:25:57.021734+0000","last_clean":"2024-05-27T07:25:57.021734+0000","last_became_active":"2024-05-27T07:21:48.597566+0000","last_became_peered":"2024-05-27T07:21:48.597566+0000","last_unstale":"2024-05-27T07:25:57.021734+0000","last_undegraded":"2024-05-27T07:25:57.021734+0000","last_fullsized":"2024-05-27T07:25:57.021734+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:21:47.565641+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:21:47.565641+0000","last_clean_scrub_stamp":"2024-05-27T07:21:47.565641+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T09:30:43.438209+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,5],"acting":[7,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"90'9","reported_seq":50,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-27T07:25:57.021645+0000","last_change":"2024-05-27T07:21:48.593412+0000","last_active":"2024-05-27T07:25:57.021645+0000","last_peered":"2024-05-27T07:25:57.021645+0000","last_clean":"2024-05-27T07:25:57.021645+0000","last_became_active":"2024-05-27T07:21:48.593070+0000","last_became_peered":"2024-05-27T07:21:48.593070+0000","last_unstale":"2024-05-27T07:25:57.021645+0000","last_undegraded":"2024-05-27T07:25:57.021645+0000","last_fullsized":"2024-05-27T07:25:57.021645+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:21:47.565641+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:21:47.565641+0000","last_clean_scrub_stamp":"2024-05-27T07:21:47.565641+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T14:53:49.382796+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,10],"acting":[3,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"90'11","reported_seq":52,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-27T07:25:56.935451+0000","last_change":"2024-05-27T07:21:48.607244+0000","last_active":"2024-05-27T07:25:56.935451+0000","last_peered":"2024-05-27T07:25:56.935451+0000","last_clean":"2024-05-27T07:25:56.935451+0000","last_became_active":"2024-05-27T07:21:48.606238+0000","last_became_peered":"2024-05-27T07:21:48.606238+0000","last_unstale":"2024-05-27T07:25:56.935451+0000","last_undegraded":"2024-05-27T07:25:56.935451+0000","last_fullsized":"2024-05-27T07:25:56.935451+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:21:47.565641+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:21:47.565641+0000","last_clean_scrub_stamp":"2024-05-27T07:21:47.565641+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T12:36:33.142111+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,11],"acting":[1,5,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.36","version":"90'16","reported_seq":57,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-27T07:25:57.022630+0000","last_change":"2024-05-27T07:21:48.593275+0000","last_active":"2024-05-27T07:25:57.022630+0000","last_peered":"2024-05-27T07:25:57.022630+0000","last_clean":"2024-05-27T07:25:57.022630+0000","last_became_active":"2024-05-27T07:21:48.592890+0000","last_became_peered":"2024-05-27T07:21:48.592890+0000","last_unstale":"2024-05-27T07:25:57.022630+0000","last_undegraded":"2024-05-27T07:25:57.022630+0000","last_fullsized":"2024-05-27T07:25:57.022630+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:21:47.565641+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:21:47.565641+0000","last_clean_scrub_stamp":"2024-05-27T07:21:47.565641+0000","objects_scrubbed":0,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T13:47:20.827824+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,4],"acting":[10,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.35","version":"90'6","reported_seq":45,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-27T07:25:57.021854+0000","last_change":"2024-05-27T07:21:48.603424+0000","last_active":"2024-05-27T07:25:57.021854+0000","last_peered":"2024-05-27T07:25:57.021854+0000","last_clean":"2024-05-27T07:25:57.021854+0000","last_became_active":"2024-05-27T07:21:48.602962+0000","last_became_peered":"2024-05-27T07:21:48.602962+0000","last_unstale":"2024-05-27T07:25:57.021854+0000","last_undegraded":"2024-05-27T07:25:57.021854+0000","last_fullsized":"2024-05-27T07:25:57.021854+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:21:47.565641+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:21:47.565641+0000","last_clean_scrub_stamp":"2024-05-27T07:21:47.565641+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T19:19:42.063063+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,8,1],"acting":[7,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.34","version":"90'7","reported_seq":48,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-27T07:25:57.024017+0000","last_change":"2024-05-27T07:21:48.589323+0000","last_active":"2024-05-27T07:25:57.024017+0000","last_peered":"2024-05-27T07:25:57.024017+0000","last_clean":"2024-05-27T07:25:57.024017+0000","last_became_active":"2024-05-27T07:21:48.587435+0000","last_became_peered":"2024-05-27T07:21:48.587435+0000","last_unstale":"2024-05-27T07:25:57.024017+0000","last_undegraded":"2024-05-27T07:25:57.024017+0000","last_fullsized":"2024-05-27T07:25:57.024017+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:21:47.565641+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:21:47.565641+0000","last_clean_scrub_stamp":"2024-05-27T07:21:47.565641+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T13:34:44.340376+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,8],"acting":[4,10,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"90'10","reported_seq":51,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-27T07:25:57.024520+0000","last_change":"2024-05-27T07:21:48.593759+0000","last_active":"2024-05-27T07:25:57.024520+0000","last_peered":"2024-05-27T07:25:57.024520+0000","last_clean":"2024-05-27T07:25:57.024520+0000","last_became_active":"2024-05-27T07:21:48.593099+0000","last_became_peered":"2024-05-27T07:21:48.593099+0000","last_unstale":"2024-05-27T07:25:57.024520+0000","last_undegraded":"2024-05-27T07:25:57.024520+0000","last_fullsized":"2024-05-27T07:25:57.024520+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:21:47.565641+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:21:47.565641+0000","last_clean_scrub_stamp":"2024-05-27T07:21:47.565641+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T13:49:00.390792+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,5],"acting":[6,10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.32","version":"90'7","reported_seq":48,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-27T07:25:57.024016+0000","last_change":"2024-05-27T07:21:48.587486+0000","last_active":"2024-05-27T07:25:57.024016+0000","last_peered":"2024-05-27T07:25:57.024016+0000","last_clean":"2024-05-27T07:25:57.024016+0000","last_became_active":"2024-05-27T07:21:48.587159+0000","last_became_peered":"2024-05-27T07:21:48.587159+0000","last_unstale":"2024-05-27T07:25:57.024016+0000","last_undegraded":"2024-05-27T07:25:57.024016+0000","last_fullsized":"2024-05-27T07:25:57.024016+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:21:47.565641+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:21:47.565641+0000","last_clean_scrub_stamp":"2024-05-27T07:21:47.565641+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T15:34:11.908442+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,10],"acting":[0,8,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"90'7","reported_seq":48,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-27T07:25:57.024296+0000","last_change":"2024-05-27T07:21:48.596817+0000","last_active":"2024-05-27T07:25:57.024296+0000","last_peered":"2024-05-27T07:25:57.024296+0000","last_clean":"2024-05-27T07:25:57.024296+0000","last_became_active":"2024-05-27T07:21:48.596469+0000","last_became_peered":"2024-05-27T07:21:48.596469+0000","last_unstale":"2024-05-27T07:25:57.024296+0000","last_undegraded":"2024-05-27T07:25:57.024296+0000","last_fullsized":"2024-05-27T07:25:57.024296+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:21:47.565641+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:21:47.565641+0000","last_clean_scrub_stamp":"2024-05-27T07:21:47.565641+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T12:38:41.494060+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,2],"acting":[0,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.30","version":"90'9","reported_seq":50,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-27T07:25:56.934824+0000","last_change":"2024-05-27T07:21:48.604249+0000","last_active":"2024-05-27T07:25:56.934824+0000","last_peered":"2024-05-27T07:25:56.934824+0000","last_clean":"2024-05-27T07:25:56.934824+0000","last_became_active":"2024-05-27T07:21:48.603585+0000","last_became_peered":"2024-05-27T07:21:48.603585+0000","last_unstale":"2024-05-27T07:25:56.934824+0000","last_undegraded":"2024-05-27T07:25:56.934824+0000","last_fullsized":"2024-05-27T07:25:56.934824+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:21:47.565641+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:21:47.565641+0000","last_clean_scrub_stamp":"2024-05-27T07:21:47.565641+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T07:22:14.839824+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2f","version":"90'8","reported_seq":49,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-27T07:25:57.023941+0000","last_change":"2024-05-27T07:21:48.596035+0000","last_active":"2024-05-27T07:25:57.023941+0000","last_peered":"2024-05-27T07:25:57.023941+0000","last_clean":"2024-05-27T07:25:57.023941+0000","last_became_active":"2024-05-27T07:21:48.595509+0000","last_became_peered":"2024-05-27T07:21:48.595509+0000","last_unstale":"2024-05-27T07:25:57.023941+0000","last_undegraded":"2024-05-27T07:25:57.023941+0000","last_fullsized":"2024-05-27T07:25:57.023941+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:21:47.565641+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:21:47.565641+0000","last_clean_scrub_stamp":"2024-05-27T07:21:47.565641+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T13:23:03.734982+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.2e","version":"90'10","reported_seq":51,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-27T07:25:56.931790+0000","last_change":"2024-05-27T07:21:48.590002+0000","last_active":"2024-05-27T07:25:56.931790+0000","last_peered":"2024-05-27T07:25:56.931790+0000","last_clean":"2024-05-27T07:25:56.931790+0000","last_became_active":"2024-05-27T07:21:48.589608+0000","last_became_peered":"2024-05-27T07:21:48.589608+0000","last_unstale":"2024-05-27T07:25:56.931790+0000","last_undegraded":"2024-05-27T07:25:56.931790+0000","last_fullsized":"2024-05-27T07:25:56.931790+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:21:47.565641+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:21:47.565641+0000","last_clean_scrub_stamp":"2024-05-27T07:21:47.565641+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T09:33:39.174351+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,1,10],"acting":[9,1,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.2d","version":"90'7","reported_seq":48,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-27T07:25:56.935673+0000","last_change":"2024-05-27T07:21:48.594885+0000","last_active":"2024-05-27T07:25:56.935673+0000","last_peered":"2024-05-27T07:25:56.935673+0000","last_clean":"2024-05-27T07:25:56.935673+0000","last_became_active":"2024-05-27T07:21:48.594521+0000","last_became_peered":"2024-05-27T07:21:48.594521+0000","last_unstale":"2024-05-27T07:25:56.935673+0000","last_undegraded":"2024-05-27T07:25:56.935673+0000","last_fullsized":"2024-05-27T07:25:56.935673+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:21:47.565641+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:21:47.565641+0000","last_clean_scrub_stamp":"2024-05-27T07:21:47.565641+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T08:54:59.311938+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":38,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-27T07:25:56.935012+0000","last_change":"2024-05-27T07:22:30.857546+0000","last_active":"2024-05-27T07:25:56.935012+0000","last_peered":"2024-05-27T07:25:56.935012+0000","last_clean":"2024-05-27T07:25:56.935012+0000","last_became_active":"2024-05-27T07:22:30.856808+0000","last_became_peered":"2024-05-27T07:22:30.856808+0000","last_unstale":"2024-05-27T07:25:56.935012+0000","last_undegraded":"2024-05-27T07:25:56.935012+0000","last_fullsized":"2024-05-27T07:25:56.935012+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:21:58.698952+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:21:58.698952+0000","last_clean_scrub_stamp":"2024-05-27T07:21:58.698952+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T16:09:25.480864+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,8,0],"acting":[2,3,8,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1f","version":"91'14","reported_seq":244,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-27T07:25:57.227211+0000","last_change":"2024-05-27T07:21:45.566489+0000","last_active":"2024-05-27T07:25:57.227211+0000","last_peered":"2024-05-27T07:25:57.227211+0000","last_clean":"2024-05-27T07:25:57.227211+0000","last_became_active":"2024-05-27T07:21:45.566115+0000","last_became_peered":"2024-05-27T07:21:45.566115+0000","last_unstale":"2024-05-27T07:25:57.227211+0000","last_undegraded":"2024-05-27T07:25:57.227211+0000","last_fullsized":"2024-05-27T07:25:57.227211+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:21:44.543638+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:21:44.543638+0000","last_clean_scrub_stamp":"2024-05-27T07:21:44.543638+0000","objects_scrubbed":0,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T07:24:11.735014+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":182,"num_read_kb":182,"num_write":15,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,8],"acting":[9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.1e","version":"90'5","reported_seq":47,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-27T07:25:57.022556+0000","last_change":"2024-05-27T07:21:48.588574+0000","last_active":"2024-05-27T07:25:57.022556+0000","last_peered":"2024-05-27T07:25:57.022556+0000","last_clean":"2024-05-27T07:25:57.022556+0000","last_became_active":"2024-05-27T07:21:48.588212+0000","last_became_peered":"2024-05-27T07:21:48.588212+0000","last_unstale":"2024-05-27T07:25:57.022556+0000","last_undegraded":"2024-05-27T07:25:57.022556+0000","last_fullsized":"2024-05-27T07:25:57.022556+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:21:47.565641+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:21:47.565641+0000","last_clean_scrub_stamp":"2024-05-27T07:21:47.565641+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T12:15:16.017635+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":31,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-27T07:25:57.022914+0000","last_change":"2024-05-27T07:21:59.734617+0000","last_active":"2024-05-27T07:25:57.022914+0000","last_peered":"2024-05-27T07:25:57.022914+0000","last_clean":"2024-05-27T07:25:57.022914+0000","last_became_active":"2024-05-27T07:21:59.734162+0000","last_became_peered":"2024-05-27T07:21:59.734162+0000","last_unstale":"2024-05-27T07:25:57.022914+0000","last_undegraded":"2024-05-27T07:25:57.022914+0000","last_fullsized":"2024-05-27T07:25:57.022914+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:21:58.698952+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:21:58.698952+0000","last_clean_scrub_stamp":"2024-05-27T07:21:58.698952+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T12:40:01.375520+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,9,3],"acting":[4,6,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1e","version":"88'2","reported_seq":49,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-27T07:25:57.022733+0000","last_change":"2024-05-27T07:21:45.790907+0000","last_active":"2024-05-27T07:25:57.022733+0000","last_peered":"2024-05-27T07:25:57.022733+0000","last_clean":"2024-05-27T07:25:57.022733+0000","last_became_active":"2024-05-27T07:21:45.790451+0000","last_became_peered":"2024-05-27T07:21:45.790451+0000","last_unstale":"2024-05-27T07:25:57.022733+0000","last_undegraded":"2024-05-27T07:25:57.022733+0000","last_fullsized":"2024-05-27T07:25:57.022733+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:21:44.543638+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:21:44.543638+0000","last_clean_scrub_stamp":"2024-05-27T07:21:44.543638+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T10:21:56.524930+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"90'6","reported_seq":48,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-27T07:25:57.026415+0000","last_change":"2024-05-27T07:21:48.595975+0000","last_active":"2024-05-27T07:25:57.026415+0000","last_peered":"2024-05-27T07:25:57.026415+0000","last_clean":"2024-05-27T07:25:57.026415+0000","last_became_active":"2024-05-27T07:21:48.595608+0000","last_became_peered":"2024-05-27T07:21:48.595608+0000","last_unstale":"2024-05-27T07:25:57.026415+0000","last_undegraded":"2024-05-27T07:25:57.026415+0000","last_fullsized":"2024-05-27T07:25:57.026415+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:21:47.565641+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:21:47.565641+0000","last_clean_scrub_stamp":"2024-05-27T07:21:47.565641+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T07:24:44.797973+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,9],"acting":[0,1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":31,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-27T07:25:57.023806+0000","last_change":"2024-05-27T07:21:59.732788+0000","last_active":"2024-05-27T07:25:57.023806+0000","last_peered":"2024-05-27T07:25:57.023806+0000","last_clean":"2024-05-27T07:25:57.023806+0000","last_became_active":"2024-05-27T07:21:59.732310+0000","last_became_peered":"2024-05-27T07:21:59.732310+0000","last_unstale":"2024-05-27T07:25:57.023806+0000","last_undegraded":"2024-05-27T07:25:57.023806+0000","last_fullsized":"2024-05-27T07:25:57.023806+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:21:58.698952+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:21:58.698952+0000","last_clean_scrub_stamp":"2024-05-27T07:21:58.698952+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T13:54:51.490154+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,9,5],"acting":[4,3,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1d","version":"91'13","reported_seq":61,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-27T07:25:57.023929+0000","last_change":"2024-05-27T07:21:45.569746+0000","last_active":"2024-05-27T07:25:57.023929+0000","last_peered":"2024-05-27T07:25:57.023929+0000","last_clean":"2024-05-27T07:25:57.023929+0000","last_became_active":"2024-05-27T07:21:45.569366+0000","last_became_peered":"2024-05-27T07:21:45.569366+0000","last_unstale":"2024-05-27T07:25:57.023929+0000","last_undegraded":"2024-05-27T07:25:57.023929+0000","last_fullsized":"2024-05-27T07:25:57.023929+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:21:44.543638+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:21:44.543638+0000","last_clean_scrub_stamp":"2024-05-27T07:21:44.543638+0000","objects_scrubbed":0,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T15:58:32.977754+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":18,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":7,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1c","version":"90'9","reported_seq":50,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-27T07:25:56.935051+0000","last_change":"2024-05-27T07:21:48.606138+0000","last_active":"2024-05-27T07:25:56.935051+0000","last_peered":"2024-05-27T07:25:56.935051+0000","last_clean":"2024-05-27T07:25:56.935051+0000","last_became_active":"2024-05-27T07:21:48.605711+0000","last_became_peered":"2024-05-27T07:21:48.605711+0000","last_unstale":"2024-05-27T07:25:56.935051+0000","last_undegraded":"2024-05-27T07:25:56.935051+0000","last_fullsized":"2024-05-27T07:25:56.935051+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:21:47.565641+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:21:47.565641+0000","last_clean_scrub_stamp":"2024-05-27T07:21:47.565641+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T10:46:39.052387+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,6],"acting":[1,10,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":31,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-27T07:25:57.023535+0000","last_change":"2024-05-27T07:21:59.735475+0000","last_active":"2024-05-27T07:25:57.023535+0000","last_peered":"2024-05-27T07:25:57.023535+0000","last_clean":"2024-05-27T07:25:57.023535+0000","last_became_active":"2024-05-27T07:21:59.734986+0000","last_became_peered":"2024-05-27T07:21:59.734986+0000","last_unstale":"2024-05-27T07:25:57.023535+0000","last_undegraded":"2024-05-27T07:25:57.023535+0000","last_fullsized":"2024-05-27T07:25:57.023535+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:21:58.698952+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:21:58.698952+0000","last_clean_scrub_stamp":"2024-05-27T07:21:58.698952+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T14:48:28.198718+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5,3],"acting":[4,6,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1c","version":"88'5","reported_seq":53,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-27T07:25:57.023547+0000","last_change":"2024-05-27T07:21:45.572882+0000","last_active":"2024-05-27T07:25:57.023547+0000","last_peered":"2024-05-27T07:25:57.023547+0000","last_clean":"2024-05-27T07:25:57.023547+0000","last_became_active":"2024-05-27T07:21:45.572460+0000","last_became_peered":"2024-05-27T07:21:45.572460+0000","last_unstale":"2024-05-27T07:25:57.023547+0000","last_undegraded":"2024-05-27T07:25:57.023547+0000","last_fullsized":"2024-05-27T07:25:57.023547+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:21:44.543638+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:21:44.543638+0000","last_clean_scrub_stamp":"2024-05-27T07:21:44.543638+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T09:40:11.311923+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":7,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,1,5],"acting":[8,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1d","version":"90'15","reported_seq":56,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-27T07:25:57.023149+0000","last_change":"2024-05-27T07:21:48.596217+0000","last_active":"2024-05-27T07:25:57.023149+0000","last_peered":"2024-05-27T07:25:57.023149+0000","last_clean":"2024-05-27T07:25:57.023149+0000","last_became_active":"2024-05-27T07:21:48.595832+0000","last_became_peered":"2024-05-27T07:21:48.595832+0000","last_unstale":"2024-05-27T07:25:57.023149+0000","last_undegraded":"2024-05-27T07:25:57.023149+0000","last_fullsized":"2024-05-27T07:25:57.023149+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:21:47.565641+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:21:47.565641+0000","last_clean_scrub_stamp":"2024-05-27T07:21:47.565641+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T10:08:05.304404+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":31,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-27T07:25:56.935413+0000","last_change":"2024-05-27T07:21:59.729617+0000","last_active":"2024-05-27T07:25:56.935413+0000","last_peered":"2024-05-27T07:25:56.935413+0000","last_clean":"2024-05-27T07:25:56.935413+0000","last_became_active":"2024-05-27T07:21:59.729003+0000","last_became_peered":"2024-05-27T07:21:59.729003+0000","last_unstale":"2024-05-27T07:25:56.935413+0000","last_undegraded":"2024-05-27T07:25:56.935413+0000","last_fullsized":"2024-05-27T07:25:56.935413+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:21:58.698952+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:21:58.698952+0000","last_clean_scrub_stamp":"2024-05-27T07:21:58.698952+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T14:55:04.844948+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7,11],"acting":[2,5,7,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1b","version":"91'8","reported_seq":239,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-27T07:25:57.161701+0000","last_change":"2024-05-27T07:21:45.566355+0000","last_active":"2024-05-27T07:25:57.161701+0000","last_peered":"2024-05-27T07:25:57.161701+0000","last_clean":"2024-05-27T07:25:57.161701+0000","last_became_active":"2024-05-27T07:21:45.564776+0000","last_became_peered":"2024-05-27T07:21:45.564776+0000","last_unstale":"2024-05-27T07:25:57.161701+0000","last_undegraded":"2024-05-27T07:25:57.161701+0000","last_fullsized":"2024-05-27T07:25:57.161701+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:21:44.543638+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:21:44.543638+0000","last_clean_scrub_stamp":"2024-05-27T07:21:44.543638+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T12:10:41.632630+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1545,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":185,"num_read_kb":4,"num_write":5,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1a","version":"90'7","reported_seq":48,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-27T07:25:57.023028+0000","last_change":"2024-05-27T07:21:48.588837+0000","last_active":"2024-05-27T07:25:57.023028+0000","last_peered":"2024-05-27T07:25:57.023028+0000","last_clean":"2024-05-27T07:25:57.023028+0000","last_became_active":"2024-05-27T07:21:48.587149+0000","last_became_peered":"2024-05-27T07:21:48.587149+0000","last_unstale":"2024-05-27T07:25:57.023028+0000","last_undegraded":"2024-05-27T07:25:57.023028+0000","last_fullsized":"2024-05-27T07:25:57.023028+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:21:47.565641+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:21:47.565641+0000","last_clean_scrub_stamp":"2024-05-27T07:21:47.565641+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T09:20:05.124189+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,5],"acting":[8,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":31,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-27T07:25:56.935491+0000","last_change":"2024-05-27T07:21:59.730003+0000","last_active":"2024-05-27T07:25:56.935491+0000","last_peered":"2024-05-27T07:25:56.935491+0000","last_clean":"2024-05-27T07:25:56.935491+0000","last_became_active":"2024-05-27T07:21:59.729195+0000","last_became_peered":"2024-05-27T07:21:59.729195+0000","last_unstale":"2024-05-27T07:25:56.935491+0000","last_undegraded":"2024-05-27T07:25:56.935491+0000","last_fullsized":"2024-05-27T07:25:56.935491+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:21:58.698952+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:21:58.698952+0000","last_clean_scrub_stamp":"2024-05-27T07:21:58.698952+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T13:13:41.612300+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,9,5],"acting":[2,6,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1a","version":"88'7","reported_seq":54,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-27T07:25:57.022914+0000","last_change":"2024-05-27T07:21:45.567279+0000","last_active":"2024-05-27T07:25:57.022914+0000","last_peered":"2024-05-27T07:25:57.022914+0000","last_clean":"2024-05-27T07:25:57.022914+0000","last_became_active":"2024-05-27T07:21:45.566920+0000","last_became_peered":"2024-05-27T07:21:45.566920+0000","last_unstale":"2024-05-27T07:25:57.022914+0000","last_undegraded":"2024-05-27T07:25:57.022914+0000","last_fullsized":"2024-05-27T07:25:57.022914+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:21:44.543638+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:21:44.543638+0000","last_clean_scrub_stamp":"2024-05-27T07:21:44.543638+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T18:12:23.015535+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,8,6],"acting":[10,8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.1b","version":"90'8","reported_seq":55,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-27T07:25:57.023087+0000","last_change":"2024-05-27T07:22:30.857080+0000","last_active":"2024-05-27T07:25:57.023087+0000","last_peered":"2024-05-27T07:25:57.023087+0000","last_clean":"2024-05-27T07:25:57.023087+0000","last_became_active":"2024-05-27T07:22:30.856479+0000","last_became_peered":"2024-05-27T07:22:30.856479+0000","last_unstale":"2024-05-27T07:25:57.023087+0000","last_undegraded":"2024-05-27T07:25:57.023087+0000","last_fullsized":"2024-05-27T07:25:57.023087+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:21:47.565641+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:21:47.565641+0000","last_clean_scrub_stamp":"2024-05-27T07:21:47.565641+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T13:29:16.054448+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,8,3],"acting":[5,8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":38,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-27T07:25:57.023817+0000","last_change":"2024-05-27T07:22:30.805292+0000","last_active":"2024-05-27T07:25:57.023817+0000","last_peered":"2024-05-27T07:25:57.023817+0000","last_clean":"2024-05-27T07:25:57.023817+0000","last_became_active":"2024-05-27T07:22:30.804779+0000","last_became_peered":"2024-05-27T07:22:30.804779+0000","last_unstale":"2024-05-27T07:25:57.023817+0000","last_undegraded":"2024-05-27T07:25:57.023817+0000","last_fullsized":"2024-05-27T07:25:57.023817+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:21:58.698952+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:21:58.698952+0000","last_clean_scrub_stamp":"2024-05-27T07:21:58.698952+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T11:12:49.095931+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,8,0],"acting":[11,4,8,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.19","version":"88'3","reported_seq":49,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-27T07:25:57.024211+0000","last_change":"2024-05-27T07:21:45.561255+0000","last_active":"2024-05-27T07:25:57.024211+0000","last_peered":"2024-05-27T07:25:57.024211+0000","last_clean":"2024-05-27T07:25:57.024211+0000","last_became_active":"2024-05-27T07:21:45.560896+0000","last_became_peered":"2024-05-27T07:21:45.560896+0000","last_unstale":"2024-05-27T07:25:57.024211+0000","last_undegraded":"2024-05-27T07:25:57.024211+0000","last_fullsized":"2024-05-27T07:25:57.024211+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:21:44.543638+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:21:44.543638+0000","last_clean_scrub_stamp":"2024-05-27T07:21:44.543638+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T17:05:24.074628+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,9],"acting":[6,4,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"90'8","reported_seq":49,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-27T07:25:57.021953+0000","last_change":"2024-05-27T07:21:48.586045+0000","last_active":"2024-05-27T07:25:57.021953+0000","last_peered":"2024-05-27T07:25:57.021953+0000","last_clean":"2024-05-27T07:25:57.021953+0000","last_became_active":"2024-05-27T07:21:48.585681+0000","last_became_peered":"2024-05-27T07:21:48.585681+0000","last_unstale":"2024-05-27T07:25:57.021953+0000","last_undegraded":"2024-05-27T07:25:57.021953+0000","last_fullsized":"2024-05-27T07:25:57.021953+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:21:47.565641+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:21:47.565641+0000","last_clean_scrub_stamp":"2024-05-27T07:21:47.565641+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T09:56:36.408623+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,6],"acting":[3,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":31,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-27T07:25:57.025932+0000","last_change":"2024-05-27T07:21:59.739544+0000","last_active":"2024-05-27T07:25:57.025932+0000","last_peered":"2024-05-27T07:25:57.025932+0000","last_clean":"2024-05-27T07:25:57.025932+0000","last_became_active":"2024-05-27T07:21:59.739132+0000","last_became_peered":"2024-05-27T07:21:59.739132+0000","last_unstale":"2024-05-27T07:25:57.025932+0000","last_undegraded":"2024-05-27T07:25:57.025932+0000","last_fullsized":"2024-05-27T07:25:57.025932+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:21:58.698952+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:21:58.698952+0000","last_clean_scrub_stamp":"2024-05-27T07:21:58.698952+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T10:06:05.126579+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,6,1],"acting":[0,7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.18","version":"88'3","reported_seq":50,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-27T07:25:57.022592+0000","last_change":"2024-05-27T07:21:45.568171+0000","last_active":"2024-05-27T07:25:57.022592+0000","last_peered":"2024-05-27T07:25:57.022592+0000","last_clean":"2024-05-27T07:25:57.022592+0000","last_became_active":"2024-05-27T07:21:45.567627+0000","last_became_peered":"2024-05-27T07:21:45.567627+0000","last_unstale":"2024-05-27T07:25:57.022592+0000","last_undegraded":"2024-05-27T07:25:57.022592+0000","last_fullsized":"2024-05-27T07:25:57.022592+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:21:44.543638+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:21:44.543638+0000","last_clean_scrub_stamp":"2024-05-27T07:21:44.543638+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T16:43:56.407079+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"90'11","reported_seq":52,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-27T07:25:56.930989+0000","last_change":"2024-05-27T07:21:48.587452+0000","last_active":"2024-05-27T07:25:56.930989+0000","last_peered":"2024-05-27T07:25:56.930989+0000","last_clean":"2024-05-27T07:25:56.930989+0000","last_became_active":"2024-05-27T07:21:48.587027+0000","last_became_peered":"2024-05-27T07:21:48.587027+0000","last_unstale":"2024-05-27T07:25:56.930989+0000","last_undegraded":"2024-05-27T07:25:56.930989+0000","last_fullsized":"2024-05-27T07:25:56.930989+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:21:47.565641+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:21:47.565641+0000","last_clean_scrub_stamp":"2024-05-27T07:21:47.565641+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T13:51:44.890061+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,10],"acting":[9,3,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":31,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-27T07:25:57.021908+0000","last_change":"2024-05-27T07:21:59.738292+0000","last_active":"2024-05-27T07:25:57.021908+0000","last_peered":"2024-05-27T07:25:57.021908+0000","last_clean":"2024-05-27T07:25:57.021908+0000","last_became_active":"2024-05-27T07:21:59.737941+0000","last_became_peered":"2024-05-27T07:21:59.737941+0000","last_unstale":"2024-05-27T07:25:57.021908+0000","last_undegraded":"2024-05-27T07:25:57.021908+0000","last_fullsized":"2024-05-27T07:25:57.021908+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:21:58.698952+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:21:58.698952+0000","last_clean_scrub_stamp":"2024-05-27T07:21:58.698952+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T13:54:24.192917+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1,5],"acting":[3,7,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.17","version":"91'7","reported_seq":416,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-27T07:25:57.227899+0000","last_change":"2024-05-27T07:21:45.568338+0000","last_active":"2024-05-27T07:25:57.227899+0000","last_peered":"2024-05-27T07:25:57.227899+0000","last_clean":"2024-05-27T07:25:57.227899+0000","last_became_active":"2024-05-27T07:21:45.567856+0000","last_became_peered":"2024-05-27T07:21:45.567856+0000","last_unstale":"2024-05-27T07:25:57.227899+0000","last_undegraded":"2024-05-27T07:25:57.227899+0000","last_fullsized":"2024-05-27T07:25:57.227899+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:21:44.543638+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:21:44.543638+0000","last_clean_scrub_stamp":"2024-05-27T07:21:44.543638+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T15:02:04.188711+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":180,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":362,"num_read_kb":362,"num_write":8,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,5,9],"acting":[10,5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.16","version":"90'10","reported_seq":51,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-27T07:25:56.935595+0000","last_change":"2024-05-27T07:21:48.606062+0000","last_active":"2024-05-27T07:25:56.935595+0000","last_peered":"2024-05-27T07:25:56.935595+0000","last_clean":"2024-05-27T07:25:56.935595+0000","last_became_active":"2024-05-27T07:21:48.605423+0000","last_became_peered":"2024-05-27T07:21:48.605423+0000","last_unstale":"2024-05-27T07:25:56.935595+0000","last_undegraded":"2024-05-27T07:25:56.935595+0000","last_fullsized":"2024-05-27T07:25:56.935595+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:21:47.565641+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:21:47.565641+0000","last_clean_scrub_stamp":"2024-05-27T07:21:47.565641+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T14:39:06.921519+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,9],"acting":[1,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":31,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-27T07:25:56.930933+0000","last_change":"2024-05-27T07:21:59.735150+0000","last_active":"2024-05-27T07:25:56.930933+0000","last_peered":"2024-05-27T07:25:56.930933+0000","last_clean":"2024-05-27T07:25:56.930933+0000","last_became_active":"2024-05-27T07:21:59.734688+0000","last_became_peered":"2024-05-27T07:21:59.734688+0000","last_unstale":"2024-05-27T07:25:56.930933+0000","last_undegraded":"2024-05-27T07:25:56.930933+0000","last_fullsized":"2024-05-27T07:25:56.930933+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:21:58.698952+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:21:58.698952+0000","last_clean_scrub_stamp":"2024-05-27T07:21:58.698952+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T15:38:02.095947+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,0,2,4],"acting":[9,0,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.16","version":"88'4","reported_seq":48,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-27T07:25:57.022536+0000","last_change":"2024-05-27T07:21:45.559550+0000","last_active":"2024-05-27T07:25:57.022536+0000","last_peered":"2024-05-27T07:25:57.022536+0000","last_clean":"2024-05-27T07:25:57.022536+0000","last_became_active":"2024-05-27T07:21:45.559181+0000","last_became_peered":"2024-05-27T07:21:45.559181+0000","last_unstale":"2024-05-27T07:25:57.022536+0000","last_undegraded":"2024-05-27T07:25:57.022536+0000","last_fullsized":"2024-05-27T07:25:57.022536+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:21:44.543638+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:21:44.543638+0000","last_clean_scrub_stamp":"2024-05-27T07:21:44.543638+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T10:33:45.288437+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.17","version":"90'9","reported_seq":50,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-27T07:25:57.026187+0000","last_change":"2024-05-27T07:21:48.593631+0000","last_active":"2024-05-27T07:25:57.026187+0000","last_peered":"2024-05-27T07:25:57.026187+0000","last_clean":"2024-05-27T07:25:57.026187+0000","last_became_active":"2024-05-27T07:21:48.592330+0000","last_became_peered":"2024-05-27T07:21:48.592330+0000","last_unstale":"2024-05-27T07:25:57.026187+0000","last_undegraded":"2024-05-27T07:25:57.026187+0000","last_fullsized":"2024-05-27T07:25:57.026187+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:21:47.565641+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:21:47.565641+0000","last_clean_scrub_stamp":"2024-05-27T07:21:47.565641+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T19:19:51.593307+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,5],"acting":[0,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":31,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-27T07:25:57.023937+0000","last_change":"2024-05-27T07:21:59.725158+0000","last_active":"2024-05-27T07:25:57.023937+0000","last_peered":"2024-05-27T07:25:57.023937+0000","last_clean":"2024-05-27T07:25:57.023937+0000","last_became_active":"2024-05-27T07:21:59.724651+0000","last_became_peered":"2024-05-27T07:21:59.724651+0000","last_unstale":"2024-05-27T07:25:57.023937+0000","last_undegraded":"2024-05-27T07:25:57.023937+0000","last_fullsized":"2024-05-27T07:25:57.023937+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:21:58.698952+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:21:58.698952+0000","last_clean_scrub_stamp":"2024-05-27T07:21:58.698952+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T12:10:45.747484+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2,9],"acting":[4,1,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.15","version":"88'5","reported_seq":52,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-27T07:25:57.022957+0000","last_change":"2024-05-27T07:21:45.791831+0000","last_active":"2024-05-27T07:25:57.022957+0000","last_peered":"2024-05-27T07:25:57.022957+0000","last_clean":"2024-05-27T07:25:57.022957+0000","last_became_active":"2024-05-27T07:21:45.791271+0000","last_became_peered":"2024-05-27T07:21:45.791271+0000","last_unstale":"2024-05-27T07:25:57.022957+0000","last_undegraded":"2024-05-27T07:25:57.022957+0000","last_fullsized":"2024-05-27T07:25:57.022957+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:21:44.543638+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:21:44.543638+0000","last_clean_scrub_stamp":"2024-05-27T07:21:44.543638+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T16:20:37.492337+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":7,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,10],"acting":[5,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.14","version":"90'11","reported_seq":52,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-27T07:25:56.935320+0000","last_change":"2024-05-27T07:21:48.605605+0000","last_active":"2024-05-27T07:25:56.935320+0000","last_peered":"2024-05-27T07:25:56.935320+0000","last_clean":"2024-05-27T07:25:56.935320+0000","last_became_active":"2024-05-27T07:21:48.604655+0000","last_became_peered":"2024-05-27T07:21:48.604655+0000","last_unstale":"2024-05-27T07:25:56.935320+0000","last_undegraded":"2024-05-27T07:25:56.935320+0000","last_fullsized":"2024-05-27T07:25:56.935320+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:21:47.565641+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:21:47.565641+0000","last_clean_scrub_stamp":"2024-05-27T07:21:47.565641+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T14:33:53.661983+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,8,4],"acting":[1,8,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":38,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-27T07:25:56.931232+0000","last_change":"2024-05-27T07:22:31.418393+0000","last_active":"2024-05-27T07:25:56.931232+0000","last_peered":"2024-05-27T07:25:56.931232+0000","last_clean":"2024-05-27T07:25:56.931232+0000","last_became_active":"2024-05-27T07:22:31.418006+0000","last_became_peered":"2024-05-27T07:22:31.418006+0000","last_unstale":"2024-05-27T07:25:56.931232+0000","last_undegraded":"2024-05-27T07:25:56.931232+0000","last_fullsized":"2024-05-27T07:25:56.931232+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:21:58.698952+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:21:58.698952+0000","last_clean_scrub_stamp":"2024-05-27T07:21:58.698952+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T14:30:55.130461+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,11,10],"acting":[9,8,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.14","version":"91'44","reported_seq":466,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-27T07:25:57.181678+0000","last_change":"2024-05-27T07:21:45.564829+0000","last_active":"2024-05-27T07:25:57.181678+0000","last_peered":"2024-05-27T07:25:57.181678+0000","last_clean":"2024-05-27T07:25:57.181678+0000","last_became_active":"2024-05-27T07:21:45.564457+0000","last_became_peered":"2024-05-27T07:21:45.564457+0000","last_unstale":"2024-05-27T07:25:57.181678+0000","last_undegraded":"2024-05-27T07:25:57.181678+0000","last_fullsized":"2024-05-27T07:25:57.181678+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:21:44.543638+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:21:44.543638+0000","last_clean_scrub_stamp":"2024-05-27T07:21:44.543638+0000","objects_scrubbed":0,"log_size":44,"log_dups_size":0,"ondisk_log_size":44,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T19:12:44.546989+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70939,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":375,"num_read_kb":257,"num_write":46,"num_write_kb":92,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,1,2],"acting":[10,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.15","version":"90'6","reported_seq":47,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-27T07:25:57.023640+0000","last_change":"2024-05-27T07:21:48.591672+0000","last_active":"2024-05-27T07:25:57.023640+0000","last_peered":"2024-05-27T07:25:57.023640+0000","last_clean":"2024-05-27T07:25:57.023640+0000","last_became_active":"2024-05-27T07:21:48.590861+0000","last_became_peered":"2024-05-27T07:21:48.590861+0000","last_unstale":"2024-05-27T07:25:57.023640+0000","last_undegraded":"2024-05-27T07:25:57.023640+0000","last_fullsized":"2024-05-27T07:25:57.023640+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:21:47.565641+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:21:47.565641+0000","last_clean_scrub_stamp":"2024-05-27T07:21:47.565641+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T12:03:40.627764+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,7],"acting":[8,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":31,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-27T07:25:57.023508+0000","last_change":"2024-05-27T07:21:59.736446+0000","last_active":"2024-05-27T07:25:57.023508+0000","last_peered":"2024-05-27T07:25:57.023508+0000","last_clean":"2024-05-27T07:25:57.023508+0000","last_became_active":"2024-05-27T07:21:59.735854+0000","last_became_peered":"2024-05-27T07:21:59.735854+0000","last_unstale":"2024-05-27T07:25:57.023508+0000","last_undegraded":"2024-05-27T07:25:57.023508+0000","last_fullsized":"2024-05-27T07:25:57.023508+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:21:58.698952+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:21:58.698952+0000","last_clean_scrub_stamp":"2024-05-27T07:21:58.698952+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T08:02:14.640325+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,8,3,6],"acting":[5,8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.13","version":"88'9","reported_seq":57,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-27T07:25:56.935277+0000","last_change":"2024-05-27T07:21:45.562930+0000","last_active":"2024-05-27T07:25:56.935277+0000","last_peered":"2024-05-27T07:25:56.935277+0000","last_clean":"2024-05-27T07:25:56.935277+0000","last_became_active":"2024-05-27T07:21:45.562515+0000","last_became_peered":"2024-05-27T07:21:45.562515+0000","last_unstale":"2024-05-27T07:25:56.935277+0000","last_undegraded":"2024-05-27T07:25:56.935277+0000","last_fullsized":"2024-05-27T07:25:56.935277+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:21:44.543638+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:21:44.543638+0000","last_clean_scrub_stamp":"2024-05-27T07:21:44.543638+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T17:54:24.006219+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":68,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":13,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,10,3],"acting":[2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.12","version":"90'9","reported_seq":50,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-27T07:25:57.024207+0000","last_change":"2024-05-27T07:21:48.591090+0000","last_active":"2024-05-27T07:25:57.024207+0000","last_peered":"2024-05-27T07:25:57.024207+0000","last_clean":"2024-05-27T07:25:57.024207+0000","last_became_active":"2024-05-27T07:21:48.590726+0000","last_became_peered":"2024-05-27T07:21:48.590726+0000","last_unstale":"2024-05-27T07:25:57.024207+0000","last_undegraded":"2024-05-27T07:25:57.024207+0000","last_fullsized":"2024-05-27T07:25:57.024207+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:21:47.565641+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:21:47.565641+0000","last_clean_scrub_stamp":"2024-05-27T07:21:47.565641+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T18:40:03.880407+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,3],"acting":[6,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.14","version":"0'0","reported_seq":31,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-27T07:25:57.023543+0000","last_change":"2024-05-27T07:21:59.738375+0000","last_active":"2024-05-27T07:25:57.023543+0000","last_peered":"2024-05-27T07:25:57.023543+0000","last_clean":"2024-05-27T07:25:57.023543+0000","last_became_active":"2024-05-27T07:21:59.737957+0000","last_became_peered":"2024-05-27T07:21:59.737957+0000","last_unstale":"2024-05-27T07:25:57.023543+0000","last_undegraded":"2024-05-27T07:25:57.023543+0000","last_fullsized":"2024-05-27T07:25:57.023543+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:21:58.698952+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:21:58.698952+0000","last_clean_scrub_stamp":"2024-05-27T07:21:58.698952+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T16:53:51.397143+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,9,7,6],"acting":[11,9,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.12","version":"88'5","reported_seq":52,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-27T07:25:57.023142+0000","last_change":"2024-05-27T07:21:45.566986+0000","last_active":"2024-05-27T07:25:57.023142+0000","last_peered":"2024-05-27T07:25:57.023142+0000","last_clean":"2024-05-27T07:25:57.023142+0000","last_became_active":"2024-05-27T07:21:45.566579+0000","last_became_peered":"2024-05-27T07:21:45.566579+0000","last_unstale":"2024-05-27T07:25:57.023142+0000","last_undegraded":"2024-05-27T07:25:57.023142+0000","last_fullsized":"2024-05-27T07:25:57.023142+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:21:44.543638+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:21:44.543638+0000","last_clean_scrub_stamp":"2024-05-27T07:21:44.543638+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T17:08:14.034740+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":5,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"90'15","reported_seq":54,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-27T07:25:57.021914+0000","last_change":"2024-05-27T07:21:48.596575+0000","last_active":"2024-05-27T07:25:57.021914+0000","last_peered":"2024-05-27T07:25:57.021914+0000","last_clean":"2024-05-27T07:25:57.021914+0000","last_became_active":"2024-05-27T07:21:48.596215+0000","last_became_peered":"2024-05-27T07:21:48.596215+0000","last_unstale":"2024-05-27T07:25:57.021914+0000","last_undegraded":"2024-05-27T07:25:57.021914+0000","last_fullsized":"2024-05-27T07:25:57.021914+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:21:47.565641+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:21:47.565641+0000","last_clean_scrub_stamp":"2024-05-27T07:21:47.565641+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T14:57:05.795269+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":31,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-27T07:25:56.931967+0000","last_change":"2024-05-27T07:21:59.737690+0000","last_active":"2024-05-27T07:25:56.931967+0000","last_peered":"2024-05-27T07:25:56.931967+0000","last_clean":"2024-05-27T07:25:56.931967+0000","last_became_active":"2024-05-27T07:21:59.735646+0000","last_became_peered":"2024-05-27T07:21:59.735646+0000","last_unstale":"2024-05-27T07:25:56.931967+0000","last_undegraded":"2024-05-27T07:25:56.931967+0000","last_fullsized":"2024-05-27T07:25:56.931967+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:21:58.698952+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:21:58.698952+0000","last_clean_scrub_stamp":"2024-05-27T07:21:58.698952+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T13:54:41.227626+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,0,3],"acting":[9,7,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.11","version":"91'8","reported_seq":240,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-27T07:25:57.161569+0000","last_change":"2024-05-27T07:21:45.564836+0000","last_active":"2024-05-27T07:25:57.161569+0000","last_peered":"2024-05-27T07:25:57.161569+0000","last_clean":"2024-05-27T07:25:57.161569+0000","last_became_active":"2024-05-27T07:21:45.564481+0000","last_became_peered":"2024-05-27T07:21:45.564481+0000","last_unstale":"2024-05-27T07:25:57.161569+0000","last_undegraded":"2024-05-27T07:25:57.161569+0000","last_fullsized":"2024-05-27T07:25:57.161569+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:21:44.543638+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:21:44.543638+0000","last_clean_scrub_stamp":"2024-05-27T07:21:44.543638+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T15:13:12.012769+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1511,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":186,"num_read_kb":5,"num_write":8,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,10],"acting":[4,7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.10","version":"90'6","reported_seq":45,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-27T07:25:57.022004+0000","last_change":"2024-05-27T07:21:48.601331+0000","last_active":"2024-05-27T07:25:57.022004+0000","last_peered":"2024-05-27T07:25:57.022004+0000","last_clean":"2024-05-27T07:25:57.022004+0000","last_became_active":"2024-05-27T07:21:48.600917+0000","last_became_peered":"2024-05-27T07:21:48.600917+0000","last_unstale":"2024-05-27T07:25:57.022004+0000","last_undegraded":"2024-05-27T07:25:57.022004+0000","last_fullsized":"2024-05-27T07:25:57.022004+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:21:47.565641+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:21:47.565641+0000","last_clean_scrub_stamp":"2024-05-27T07:21:47.565641+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T17:26:42.129989+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":31,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-27T07:25:57.024036+0000","last_change":"2024-05-27T07:21:59.737662+0000","last_active":"2024-05-27T07:25:57.024036+0000","last_peered":"2024-05-27T07:25:57.024036+0000","last_clean":"2024-05-27T07:25:57.024036+0000","last_became_active":"2024-05-27T07:21:59.737229+0000","last_became_peered":"2024-05-27T07:21:59.737229+0000","last_unstale":"2024-05-27T07:25:57.024036+0000","last_undegraded":"2024-05-27T07:25:57.024036+0000","last_fullsized":"2024-05-27T07:25:57.024036+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:21:58.698952+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:21:58.698952+0000","last_clean_scrub_stamp":"2024-05-27T07:21:58.698952+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T15:32:31.147571+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,9,11],"acting":[6,3,9,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.10","version":"88'17","reported_seq":81,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-27T07:25:57.022607+0000","last_change":"2024-05-27T07:22:30.814815+0000","last_active":"2024-05-27T07:25:57.022607+0000","last_peered":"2024-05-27T07:25:57.022607+0000","last_clean":"2024-05-27T07:25:57.022607+0000","last_became_active":"2024-05-27T07:22:30.802828+0000","last_became_peered":"2024-05-27T07:22:30.802828+0000","last_unstale":"2024-05-27T07:25:57.022607+0000","last_undegraded":"2024-05-27T07:25:57.022607+0000","last_fullsized":"2024-05-27T07:25:57.022607+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:21:44.543638+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:21:44.543638+0000","last_clean_scrub_stamp":"2024-05-27T07:21:44.543638+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T11:38:07.363507+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,4],"acting":[8,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.11","version":"90'6","reported_seq":47,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-27T07:25:57.022823+0000","last_change":"2024-05-27T07:21:48.596204+0000","last_active":"2024-05-27T07:25:57.022823+0000","last_peered":"2024-05-27T07:25:57.022823+0000","last_clean":"2024-05-27T07:25:57.022823+0000","last_became_active":"2024-05-27T07:21:48.595827+0000","last_became_peered":"2024-05-27T07:21:48.595827+0000","last_unstale":"2024-05-27T07:25:57.022823+0000","last_undegraded":"2024-05-27T07:25:57.022823+0000","last_fullsized":"2024-05-27T07:25:57.022823+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:21:47.565641+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:21:47.565641+0000","last_clean_scrub_stamp":"2024-05-27T07:21:47.565641+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T15:51:53.162554+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":31,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-27T07:25:57.022694+0000","last_change":"2024-05-27T07:21:59.735837+0000","last_active":"2024-05-27T07:25:57.022694+0000","last_peered":"2024-05-27T07:25:57.022694+0000","last_clean":"2024-05-27T07:25:57.022694+0000","last_became_active":"2024-05-27T07:21:59.735063+0000","last_became_peered":"2024-05-27T07:21:59.735063+0000","last_unstale":"2024-05-27T07:25:57.022694+0000","last_undegraded":"2024-05-27T07:25:57.022694+0000","last_fullsized":"2024-05-27T07:25:57.022694+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:21:58.698952+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:21:58.698952+0000","last_clean_scrub_stamp":"2024-05-27T07:21:58.698952+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T13:04:21.751043+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,3,1],"acting":[5,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.f","version":"88'2","reported_seq":24,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-27T07:25:57.023668+0000","last_change":"2024-05-27T07:22:30.809491+0000","last_active":"2024-05-27T07:25:57.023668+0000","last_peered":"2024-05-27T07:25:57.023668+0000","last_clean":"2024-05-27T07:25:57.023668+0000","last_became_active":"2024-05-27T07:22:30.802187+0000","last_became_peered":"2024-05-27T07:22:30.802187+0000","last_unstale":"2024-05-27T07:25:57.023668+0000","last_undegraded":"2024-05-27T07:25:57.023668+0000","last_fullsized":"2024-05-27T07:25:57.023668+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:21:44.543638+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:21:44.543638+0000","last_clean_scrub_stamp":"2024-05-27T07:21:44.543638+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T18:11:32.821962+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":22,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,4],"acting":[11,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.e","version":"90'5","reported_seq":46,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-27T07:25:57.024123+0000","last_change":"2024-05-27T07:21:48.597362+0000","last_active":"2024-05-27T07:25:57.024123+0000","last_peered":"2024-05-27T07:25:57.024123+0000","last_clean":"2024-05-27T07:25:57.024123+0000","last_became_active":"2024-05-27T07:21:48.596920+0000","last_became_peered":"2024-05-27T07:21:48.596920+0000","last_unstale":"2024-05-27T07:25:57.024123+0000","last_undegraded":"2024-05-27T07:25:57.024123+0000","last_fullsized":"2024-05-27T07:25:57.024123+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:21:47.565641+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:21:47.565641+0000","last_clean_scrub_stamp":"2024-05-27T07:21:47.565641+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T16:50:21.021320+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,4],"acting":[11,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":31,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-27T07:25:57.022899+0000","last_change":"2024-05-27T07:21:59.732303+0000","last_active":"2024-05-27T07:25:57.022899+0000","last_peered":"2024-05-27T07:25:57.022899+0000","last_clean":"2024-05-27T07:25:57.022899+0000","last_became_active":"2024-05-27T07:21:59.731851+0000","last_became_peered":"2024-05-27T07:21:59.731851+0000","last_unstale":"2024-05-27T07:25:57.022899+0000","last_undegraded":"2024-05-27T07:25:57.022899+0000","last_fullsized":"2024-05-27T07:25:57.022899+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:21:58.698952+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:21:58.698952+0000","last_clean_scrub_stamp":"2024-05-27T07:21:58.698952+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T08:08:38.433818+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,10,11,9],"acting":[8,10,11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.4","version":"88'1","reported_seq":49,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-27T07:25:56.931533+0000","last_change":"2024-05-27T07:21:45.791370+0000","last_active":"2024-05-27T07:25:56.931533+0000","last_peered":"2024-05-27T07:25:56.931533+0000","last_clean":"2024-05-27T07:25:56.931533+0000","last_became_active":"2024-05-27T07:21:45.790987+0000","last_became_peered":"2024-05-27T07:21:45.790987+0000","last_unstale":"2024-05-27T07:25:56.931533+0000","last_undegraded":"2024-05-27T07:25:56.931533+0000","last_fullsized":"2024-05-27T07:25:56.931533+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:21:44.543638+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:21:44.543638+0000","last_clean_scrub_stamp":"2024-05-27T07:21:44.543638+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T14:20:18.570194+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2,0],"acting":[9,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.5","version":"90'4","reported_seq":45,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-27T07:25:57.023525+0000","last_change":"2024-05-27T07:21:48.585984+0000","last_active":"2024-05-27T07:25:57.023525+0000","last_peered":"2024-05-27T07:25:57.023525+0000","last_clean":"2024-05-27T07:25:57.023525+0000","last_became_active":"2024-05-27T07:21:48.585606+0000","last_became_peered":"2024-05-27T07:21:48.585606+0000","last_unstale":"2024-05-27T07:25:57.023525+0000","last_undegraded":"2024-05-27T07:25:57.023525+0000","last_fullsized":"2024-05-27T07:25:57.023525+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:21:47.565641+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:21:47.565641+0000","last_clean_scrub_stamp":"2024-05-27T07:21:47.565641+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T17:56:17.659024+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,3,6],"acting":[10,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":31,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-27T07:25:57.024779+0000","last_change":"2024-05-27T07:21:59.734022+0000","last_active":"2024-05-27T07:25:57.024779+0000","last_peered":"2024-05-27T07:25:57.024779+0000","last_clean":"2024-05-27T07:25:57.024779+0000","last_became_active":"2024-05-27T07:21:59.733467+0000","last_became_peered":"2024-05-27T07:21:59.733467+0000","last_unstale":"2024-05-27T07:25:57.024779+0000","last_undegraded":"2024-05-27T07:25:57.024779+0000","last_fullsized":"2024-05-27T07:25:57.024779+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:21:58.698952+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:21:58.698952+0000","last_clean_scrub_stamp":"2024-05-27T07:21:58.698952+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T09:54:11.058495+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,1,8],"acting":[6,0,1,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.2","version":"91'6","reported_seq":233,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-27T07:25:57.227437+0000","last_change":"2024-05-27T07:21:45.563072+0000","last_active":"2024-05-27T07:25:57.227437+0000","last_peered":"2024-05-27T07:25:57.227437+0000","last_clean":"2024-05-27T07:25:57.227437+0000","last_became_active":"2024-05-27T07:21:45.562632+0000","last_became_peered":"2024-05-27T07:21:45.562632+0000","last_unstale":"2024-05-27T07:25:57.227437+0000","last_undegraded":"2024-05-27T07:25:57.227437+0000","last_fullsized":"2024-05-27T07:25:57.227437+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:21:44.543638+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:21:44.543638+0000","last_clean_scrub_stamp":"2024-05-27T07:21:44.543638+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T12:54:19.555467+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":979,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":182,"num_read_kb":1,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3","version":"90'7","reported_seq":49,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-27T07:25:57.023358+0000","last_change":"2024-05-27T07:21:48.594867+0000","last_active":"2024-05-27T07:25:57.023358+0000","last_peered":"2024-05-27T07:25:57.023358+0000","last_clean":"2024-05-27T07:25:57.023358+0000","last_became_active":"2024-05-27T07:21:48.594523+0000","last_became_peered":"2024-05-27T07:21:48.594523+0000","last_unstale":"2024-05-27T07:25:57.023358+0000","last_undegraded":"2024-05-27T07:25:57.023358+0000","last_fullsized":"2024-05-27T07:25:57.023358+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:21:47.565641+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:21:47.565641+0000","last_clean_scrub_stamp":"2024-05-27T07:21:47.565641+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T17:28:55.862014+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,4,0],"acting":[10,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":31,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-27T07:25:57.024044+0000","last_change":"2024-05-27T07:21:59.741936+0000","last_active":"2024-05-27T07:25:57.024044+0000","last_peered":"2024-05-27T07:25:57.024044+0000","last_clean":"2024-05-27T07:25:57.024044+0000","last_became_active":"2024-05-27T07:21:59.741458+0000","last_became_peered":"2024-05-27T07:21:59.741458+0000","last_unstale":"2024-05-27T07:25:57.024044+0000","last_undegraded":"2024-05-27T07:25:57.024044+0000","last_fullsized":"2024-05-27T07:25:57.024044+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:21:58.698952+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:21:58.698952+0000","last_clean_scrub_stamp":"2024-05-27T07:21:58.698952+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T15:51:28.560995+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,10,0],"acting":[6,5,10,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.0","version":"90'8","reported_seq":26,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-27T07:25:56.930929+0000","last_change":"2024-05-27T07:22:30.803379+0000","last_active":"2024-05-27T07:25:56.930929+0000","last_peered":"2024-05-27T07:25:56.930929+0000","last_clean":"2024-05-27T07:25:56.930929+0000","last_became_active":"2024-05-27T07:22:30.803024+0000","last_became_peered":"2024-05-27T07:22:30.803024+0000","last_unstale":"2024-05-27T07:25:56.930929+0000","last_undegraded":"2024-05-27T07:25:56.930929+0000","last_fullsized":"2024-05-27T07:25:56.930929+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:21:47.565641+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:21:47.565641+0000","last_clean_scrub_stamp":"2024-05-27T07:21:47.565641+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T07:56:32.070610+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2,4],"acting":[9,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.1","version":"91'7","reported_seq":241,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-27T07:26:01.166673+0000","last_change":"2024-05-27T07:21:45.790849+0000","last_active":"2024-05-27T07:26:01.166673+0000","last_peered":"2024-05-27T07:26:01.166673+0000","last_clean":"2024-05-27T07:26:01.166673+0000","last_became_active":"2024-05-27T07:21:45.790435+0000","last_became_peered":"2024-05-27T07:21:45.790435+0000","last_unstale":"2024-05-27T07:26:01.166673+0000","last_undegraded":"2024-05-27T07:26:01.166673+0000","last_fullsized":"2024-05-27T07:26:01.166673+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:21:44.543638+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:21:44.543638+0000","last_clean_scrub_stamp":"2024-05-27T07:21:44.543638+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T09:34:51.479882+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":187,"num_read_kb":187,"num_write":7,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,0],"acting":[6,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.5","version":"88'1","reported_seq":31,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-27T07:25:57.021734+0000","last_change":"2024-05-27T07:21:59.731338+0000","last_active":"2024-05-27T07:25:57.021734+0000","last_peered":"2024-05-27T07:25:57.021734+0000","last_clean":"2024-05-27T07:25:57.021734+0000","last_became_active":"2024-05-27T07:21:59.730847+0000","last_became_peered":"2024-05-27T07:21:59.730847+0000","last_unstale":"2024-05-27T07:25:57.021734+0000","last_undegraded":"2024-05-27T07:25:57.021734+0000","last_fullsized":"2024-05-27T07:25:57.021734+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:21:58.698952+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:21:58.698952+0000","last_clean_scrub_stamp":"2024-05-27T07:21:58.698952+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T18:18:50.901129+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,11,9],"acting":[7,5,11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2","version":"90'8","reported_seq":49,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-27T07:25:57.022192+0000","last_change":"2024-05-27T07:21:48.594274+0000","last_active":"2024-05-27T07:25:57.022192+0000","last_peered":"2024-05-27T07:25:57.022192+0000","last_clean":"2024-05-27T07:25:57.022192+0000","last_became_active":"2024-05-27T07:21:48.593940+0000","last_became_peered":"2024-05-27T07:21:48.593940+0000","last_unstale":"2024-05-27T07:25:57.022192+0000","last_undegraded":"2024-05-27T07:25:57.022192+0000","last_fullsized":"2024-05-27T07:25:57.022192+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:21:47.565641+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:21:47.565641+0000","last_clean_scrub_stamp":"2024-05-27T07:21:47.565641+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T13:25:46.531524+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"88'6","reported_seq":57,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-27T07:25:57.023871+0000","last_change":"2024-05-27T07:21:45.563365+0000","last_active":"2024-05-27T07:25:57.023871+0000","last_peered":"2024-05-27T07:25:57.023871+0000","last_clean":"2024-05-27T07:25:57.023871+0000","last_became_active":"2024-05-27T07:21:45.563012+0000","last_became_peered":"2024-05-27T07:21:45.563012+0000","last_unstale":"2024-05-27T07:25:57.023871+0000","last_undegraded":"2024-05-27T07:25:57.023871+0000","last_fullsized":"2024-05-27T07:25:57.023871+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:21:44.543638+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:21:44.543638+0000","last_clean_scrub_stamp":"2024-05-27T07:21:44.543638+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T12:58:51.079965+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":68,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":4,"num_read_kb":4,"num_write":8,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,9],"acting":[4,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.0","version":"20'92","reported_seq":119,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-27T07:25:57.021608+0000","last_change":"2024-05-27T07:20:21.611212+0000","last_active":"2024-05-27T07:25:57.021608+0000","last_peered":"2024-05-27T07:25:57.021608+0000","last_clean":"2024-05-27T07:25:57.021608+0000","last_became_active":"2024-05-27T07:20:21.605163+0000","last_became_peered":"2024-05-27T07:20:21.605163+0000","last_unstale":"2024-05-27T07:25:57.021608+0000","last_undegraded":"2024-05-27T07:25:57.021608+0000","last_fullsized":"2024-05-27T07:25:57.021608+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:16:57.668433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:16:57.668433+0000","last_clean_scrub_stamp":"2024-05-27T07:16:57.668433+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T12:10:59.074162+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":81,"num_read_kb":69,"num_write":123,"num_write_kb":2150,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":2951840,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":31,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-27T07:25:57.023734+0000","last_change":"2024-05-27T07:21:59.737210+0000","last_active":"2024-05-27T07:25:57.023734+0000","last_peered":"2024-05-27T07:25:57.023734+0000","last_clean":"2024-05-27T07:25:57.023734+0000","last_became_active":"2024-05-27T07:21:59.736799+0000","last_became_peered":"2024-05-27T07:21:59.736799+0000","last_unstale":"2024-05-27T07:25:57.023734+0000","last_undegraded":"2024-05-27T07:25:57.023734+0000","last_fullsized":"2024-05-27T07:25:57.023734+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:21:58.698952+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:21:58.698952+0000","last_clean_scrub_stamp":"2024-05-27T07:21:58.698952+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T13:03:17.153687+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,6,10],"acting":[5,9,6,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1","version":"90'5","reported_seq":47,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-27T07:25:57.023957+0000","last_change":"2024-05-27T07:21:48.591550+0000","last_active":"2024-05-27T07:25:57.023957+0000","last_peered":"2024-05-27T07:25:57.023957+0000","last_clean":"2024-05-27T07:25:57.023957+0000","last_became_active":"2024-05-27T07:21:48.591223+0000","last_became_peered":"2024-05-27T07:21:48.591223+0000","last_unstale":"2024-05-27T07:25:57.023957+0000","last_undegraded":"2024-05-27T07:25:57.023957+0000","last_fullsized":"2024-05-27T07:25:57.023957+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:21:47.565641+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:21:47.565641+0000","last_clean_scrub_stamp":"2024-05-27T07:21:47.565641+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T15:06:23.395645+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,2],"acting":[6,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.0","version":"88'4","reported_seq":49,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-27T07:25:57.021833+0000","last_change":"2024-05-27T07:21:45.569224+0000","last_active":"2024-05-27T07:25:57.021833+0000","last_peered":"2024-05-27T07:25:57.021833+0000","last_clean":"2024-05-27T07:25:57.021833+0000","last_became_active":"2024-05-27T07:21:45.568818+0000","last_became_peered":"2024-05-27T07:21:45.568818+0000","last_unstale":"2024-05-27T07:25:57.021833+0000","last_undegraded":"2024-05-27T07:25:57.021833+0000","last_fullsized":"2024-05-27T07:25:57.021833+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:21:44.543638+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:21:44.543638+0000","last_clean_scrub_stamp":"2024-05-27T07:21:44.543638+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T08:12:33.157842+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,8],"acting":[7,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":31,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-27T07:25:57.023468+0000","last_change":"2024-05-27T07:21:59.733733+0000","last_active":"2024-05-27T07:25:57.023468+0000","last_peered":"2024-05-27T07:25:57.023468+0000","last_clean":"2024-05-27T07:25:57.023468+0000","last_became_active":"2024-05-27T07:21:59.733112+0000","last_became_peered":"2024-05-27T07:21:59.733112+0000","last_unstale":"2024-05-27T07:25:57.023468+0000","last_undegraded":"2024-05-27T07:25:57.023468+0000","last_fullsized":"2024-05-27T07:25:57.023468+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:21:58.698952+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:21:58.698952+0000","last_clean_scrub_stamp":"2024-05-27T07:21:58.698952+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T16:01:30.783800+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,7,6],"acting":[11,5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.5","version":"88'4","reported_seq":52,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-27T07:25:57.024039+0000","last_change":"2024-05-27T07:21:45.792029+0000","last_active":"2024-05-27T07:25:57.024039+0000","last_peered":"2024-05-27T07:25:57.024039+0000","last_clean":"2024-05-27T07:25:57.024039+0000","last_became_active":"2024-05-27T07:21:45.791492+0000","last_became_peered":"2024-05-27T07:21:45.791492+0000","last_unstale":"2024-05-27T07:25:57.024039+0000","last_undegraded":"2024-05-27T07:25:57.024039+0000","last_fullsized":"2024-05-27T07:25:57.024039+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:21:44.543638+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:21:44.543638+0000","last_clean_scrub_stamp":"2024-05-27T07:21:44.543638+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T17:36:42.444243+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,9],"acting":[8,0,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.4","version":"90'9","reported_seq":51,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-27T07:25:57.025118+0000","last_change":"2024-05-27T07:21:48.594769+0000","last_active":"2024-05-27T07:25:57.025118+0000","last_peered":"2024-05-27T07:25:57.025118+0000","last_clean":"2024-05-27T07:25:57.025118+0000","last_became_active":"2024-05-27T07:21:48.592979+0000","last_became_peered":"2024-05-27T07:21:48.592979+0000","last_unstale":"2024-05-27T07:25:57.025118+0000","last_undegraded":"2024-05-27T07:25:57.025118+0000","last_fullsized":"2024-05-27T07:25:57.025118+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:21:47.565641+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:21:47.565641+0000","last_clean_scrub_stamp":"2024-05-27T07:21:47.565641+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T10:06:48.268230+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,9],"acting":[6,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":31,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-27T07:25:57.021843+0000","last_change":"2024-05-27T07:21:59.736405+0000","last_active":"2024-05-27T07:25:57.021843+0000","last_peered":"2024-05-27T07:25:57.021843+0000","last_clean":"2024-05-27T07:25:57.021843+0000","last_became_active":"2024-05-27T07:21:59.735991+0000","last_became_peered":"2024-05-27T07:21:59.735991+0000","last_unstale":"2024-05-27T07:25:57.021843+0000","last_undegraded":"2024-05-27T07:25:57.021843+0000","last_fullsized":"2024-05-27T07:25:57.021843+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:21:58.698952+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:21:58.698952+0000","last_clean_scrub_stamp":"2024-05-27T07:21:58.698952+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T18:32:04.931725+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,10,0,7],"acting":[3,10,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"91'51","reported_seq":121,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-27T07:25:57.176466+0000","last_change":"2024-05-27T07:21:45.566513+0000","last_active":"2024-05-27T07:25:57.176466+0000","last_peered":"2024-05-27T07:25:57.176466+0000","last_clean":"2024-05-27T07:25:57.176466+0000","last_became_active":"2024-05-27T07:21:45.566107+0000","last_became_peered":"2024-05-27T07:21:45.566107+0000","last_unstale":"2024-05-27T07:25:57.176466+0000","last_undegraded":"2024-05-27T07:25:57.176466+0000","last_fullsized":"2024-05-27T07:25:57.176466+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:21:44.543638+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:21:44.543638+0000","last_clean_scrub_stamp":"2024-05-27T07:21:44.543638+0000","objects_scrubbed":0,"log_size":51,"log_dups_size":0,"ondisk_log_size":51,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T14:11:34.851691+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":393216,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":82,"num_read_kb":70,"num_write":91,"num_write_kb":838,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,3],"acting":[9,8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.7","version":"90'5","reported_seq":46,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-27T07:25:57.022710+0000","last_change":"2024-05-27T07:21:48.593642+0000","last_active":"2024-05-27T07:25:57.022710+0000","last_peered":"2024-05-27T07:25:57.022710+0000","last_clean":"2024-05-27T07:25:57.022710+0000","last_became_active":"2024-05-27T07:21:48.592885+0000","last_became_peered":"2024-05-27T07:21:48.592885+0000","last_unstale":"2024-05-27T07:25:57.022710+0000","last_undegraded":"2024-05-27T07:25:57.022710+0000","last_fullsized":"2024-05-27T07:25:57.022710+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:21:47.565641+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:21:47.565641+0000","last_clean_scrub_stamp":"2024-05-27T07:21:47.565641+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T11:51:27.186560+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,7],"acting":[3,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":31,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-27T07:25:56.935000+0000","last_change":"2024-05-27T07:21:59.725533+0000","last_active":"2024-05-27T07:25:56.935000+0000","last_peered":"2024-05-27T07:25:56.935000+0000","last_clean":"2024-05-27T07:25:56.935000+0000","last_became_active":"2024-05-27T07:21:59.724991+0000","last_became_peered":"2024-05-27T07:21:59.724991+0000","last_unstale":"2024-05-27T07:25:56.935000+0000","last_undegraded":"2024-05-27T07:25:56.935000+0000","last_fullsized":"2024-05-27T07:25:56.935000+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:21:58.698952+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:21:58.698952+0000","last_clean_scrub_stamp":"2024-05-27T07:21:58.698952+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T17:22:54.650553+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,4,7],"acting":[2,1,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"88'7","reported_seq":55,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-27T07:25:57.024242+0000","last_change":"2024-05-27T07:21:45.572701+0000","last_active":"2024-05-27T07:25:57.024242+0000","last_peered":"2024-05-27T07:25:57.024242+0000","last_clean":"2024-05-27T07:25:57.024242+0000","last_became_active":"2024-05-27T07:21:45.572176+0000","last_became_peered":"2024-05-27T07:21:45.572176+0000","last_unstale":"2024-05-27T07:25:57.024242+0000","last_undegraded":"2024-05-27T07:25:57.024242+0000","last_fullsized":"2024-05-27T07:25:57.024242+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:21:44.543638+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:21:44.543638+0000","last_clean_scrub_stamp":"2024-05-27T07:21:44.543638+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T16:11:20.901250+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,11,6],"acting":[4,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"90'7","reported_seq":49,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-27T07:25:57.026174+0000","last_change":"2024-05-27T07:21:48.581152+0000","last_active":"2024-05-27T07:25:57.026174+0000","last_peered":"2024-05-27T07:25:57.026174+0000","last_clean":"2024-05-27T07:25:57.026174+0000","last_became_active":"2024-05-27T07:21:48.580543+0000","last_became_peered":"2024-05-27T07:21:48.580543+0000","last_unstale":"2024-05-27T07:25:57.026174+0000","last_undegraded":"2024-05-27T07:25:57.026174+0000","last_fullsized":"2024-05-27T07:25:57.026174+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:21:47.565641+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:21:47.565641+0000","last_clean_scrub_stamp":"2024-05-27T07:21:47.565641+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T18:45:56.732828+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,4],"acting":[0,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":38,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-27T07:25:57.024155+0000","last_change":"2024-05-27T07:22:31.417863+0000","last_active":"2024-05-27T07:25:57.024155+0000","last_peered":"2024-05-27T07:25:57.024155+0000","last_clean":"2024-05-27T07:25:57.024155+0000","last_became_active":"2024-05-27T07:22:31.417298+0000","last_became_peered":"2024-05-27T07:22:31.417298+0000","last_unstale":"2024-05-27T07:25:57.024155+0000","last_undegraded":"2024-05-27T07:25:57.024155+0000","last_fullsized":"2024-05-27T07:25:57.024155+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:21:58.698952+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:21:58.698952+0000","last_clean_scrub_stamp":"2024-05-27T07:21:58.698952+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T12:26:13.857477+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,10,11],"acting":[4,5,10,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.8","version":"88'2","reported_seq":59,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-27T07:25:56.934912+0000","last_change":"2024-05-27T07:23:31.626374+0000","last_active":"2024-05-27T07:25:56.934912+0000","last_peered":"2024-05-27T07:25:56.934912+0000","last_clean":"2024-05-27T07:25:56.934912+0000","last_became_active":"2024-05-27T07:23:31.623214+0000","last_became_peered":"2024-05-27T07:23:31.623214+0000","last_unstale":"2024-05-27T07:25:56.934912+0000","last_undegraded":"2024-05-27T07:25:56.934912+0000","last_fullsized":"2024-05-27T07:25:56.934912+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:21:44.543638+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:21:44.543638+0000","last_clean_scrub_stamp":"2024-05-27T07:21:44.543638+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T10:42:47.710138+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,11,0],"acting":[1,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"90'2","reported_seq":43,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-27T07:25:57.022726+0000","last_change":"2024-05-27T07:21:48.589605+0000","last_active":"2024-05-27T07:25:57.022726+0000","last_peered":"2024-05-27T07:25:57.022726+0000","last_clean":"2024-05-27T07:25:57.022726+0000","last_became_active":"2024-05-27T07:21:48.588294+0000","last_became_peered":"2024-05-27T07:21:48.588294+0000","last_unstale":"2024-05-27T07:25:57.022726+0000","last_undegraded":"2024-05-27T07:25:57.022726+0000","last_fullsized":"2024-05-27T07:25:57.022726+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:21:47.565641+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:21:47.565641+0000","last_clean_scrub_stamp":"2024-05-27T07:21:47.565641+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T09:02:29.618508+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,10],"acting":[8,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":31,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-27T07:25:56.931975+0000","last_change":"2024-05-27T07:21:59.738153+0000","last_active":"2024-05-27T07:25:56.931975+0000","last_peered":"2024-05-27T07:25:56.931975+0000","last_clean":"2024-05-27T07:25:56.931975+0000","last_became_active":"2024-05-27T07:21:59.737394+0000","last_became_peered":"2024-05-27T07:21:59.737394+0000","last_unstale":"2024-05-27T07:25:56.931975+0000","last_undegraded":"2024-05-27T07:25:56.931975+0000","last_fullsized":"2024-05-27T07:25:56.931975+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:21:58.698952+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:21:58.698952+0000","last_clean_scrub_stamp":"2024-05-27T07:21:58.698952+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T17:17:13.877688+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,2,5],"acting":[9,3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.9","version":"88'7","reported_seq":53,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-27T07:25:57.021941+0000","last_change":"2024-05-27T07:21:45.562638+0000","last_active":"2024-05-27T07:25:57.021941+0000","last_peered":"2024-05-27T07:25:57.021941+0000","last_clean":"2024-05-27T07:25:57.021941+0000","last_became_active":"2024-05-27T07:21:45.562242+0000","last_became_peered":"2024-05-27T07:21:45.562242+0000","last_unstale":"2024-05-27T07:25:57.021941+0000","last_undegraded":"2024-05-27T07:25:57.021941+0000","last_fullsized":"2024-05-27T07:25:57.021941+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:21:44.543638+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:21:44.543638+0000","last_clean_scrub_stamp":"2024-05-27T07:21:44.543638+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T10:40:15.847513+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,9,10],"acting":[3,9,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.8","version":"90'9","reported_seq":50,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-27T07:25:57.022448+0000","last_change":"2024-05-27T07:21:48.598186+0000","last_active":"2024-05-27T07:25:57.022448+0000","last_peered":"2024-05-27T07:25:57.022448+0000","last_clean":"2024-05-27T07:25:57.022448+0000","last_became_active":"2024-05-27T07:21:48.597715+0000","last_became_peered":"2024-05-27T07:21:48.597715+0000","last_unstale":"2024-05-27T07:25:57.022448+0000","last_undegraded":"2024-05-27T07:25:57.022448+0000","last_fullsized":"2024-05-27T07:25:57.022448+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:21:47.565641+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:21:47.565641+0000","last_clean_scrub_stamp":"2024-05-27T07:21:47.565641+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T08:01:35.983687+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,2],"acting":[11,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.c","version":"88'1","reported_seq":32,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-27T07:25:57.023735+0000","last_change":"2024-05-27T07:21:59.727247+0000","last_active":"2024-05-27T07:25:57.023735+0000","last_peered":"2024-05-27T07:25:57.023735+0000","last_clean":"2024-05-27T07:25:57.023735+0000","last_became_active":"2024-05-27T07:21:59.726788+0000","last_became_peered":"2024-05-27T07:21:59.726788+0000","last_unstale":"2024-05-27T07:25:57.023735+0000","last_undegraded":"2024-05-27T07:25:57.023735+0000","last_fullsized":"2024-05-27T07:25:57.023735+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:21:58.698952+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:21:58.698952+0000","last_clean_scrub_stamp":"2024-05-27T07:21:58.698952+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T10:09:27.095188+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,9,7,8],"acting":[4,9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.a","version":"91'4","reported_seq":50,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-27T07:25:56.935299+0000","last_change":"2024-05-27T07:21:45.569285+0000","last_active":"2024-05-27T07:25:56.935299+0000","last_peered":"2024-05-27T07:25:56.935299+0000","last_clean":"2024-05-27T07:25:56.935299+0000","last_became_active":"2024-05-27T07:21:45.568835+0000","last_became_peered":"2024-05-27T07:21:45.568835+0000","last_unstale":"2024-05-27T07:25:56.935299+0000","last_undegraded":"2024-05-27T07:25:56.935299+0000","last_fullsized":"2024-05-27T07:25:56.935299+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:21:44.543638+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:21:44.543638+0000","last_clean_scrub_stamp":"2024-05-27T07:21:44.543638+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T10:03:23.353800+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,8],"acting":[2,3,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.b","version":"90'8","reported_seq":49,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-27T07:25:57.022876+0000","last_change":"2024-05-27T07:21:48.583494+0000","last_active":"2024-05-27T07:25:57.022876+0000","last_peered":"2024-05-27T07:25:57.022876+0000","last_clean":"2024-05-27T07:25:57.022876+0000","last_became_active":"2024-05-27T07:21:48.583141+0000","last_became_peered":"2024-05-27T07:21:48.583141+0000","last_unstale":"2024-05-27T07:25:57.022876+0000","last_undegraded":"2024-05-27T07:25:57.022876+0000","last_fullsized":"2024-05-27T07:25:57.022876+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:21:47.565641+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:21:47.565641+0000","last_clean_scrub_stamp":"2024-05-27T07:21:47.565641+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T10:24:50.475195+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,10],"acting":[3,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":31,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-27T07:25:57.022646+0000","last_change":"2024-05-27T07:21:59.733966+0000","last_active":"2024-05-27T07:25:57.022646+0000","last_peered":"2024-05-27T07:25:57.022646+0000","last_clean":"2024-05-27T07:25:57.022646+0000","last_became_active":"2024-05-27T07:21:59.733411+0000","last_became_peered":"2024-05-27T07:21:59.733411+0000","last_unstale":"2024-05-27T07:25:57.022646+0000","last_undegraded":"2024-05-27T07:25:57.022646+0000","last_fullsized":"2024-05-27T07:25:57.022646+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:21:58.698952+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:21:58.698952+0000","last_clean_scrub_stamp":"2024-05-27T07:21:58.698952+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T15:08:04.918086+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,10,3],"acting":[4,2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.b","version":"88'4","reported_seq":52,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-27T07:25:57.022886+0000","last_change":"2024-05-27T07:21:45.570967+0000","last_active":"2024-05-27T07:25:57.022886+0000","last_peered":"2024-05-27T07:25:57.022886+0000","last_clean":"2024-05-27T07:25:57.022886+0000","last_became_active":"2024-05-27T07:21:45.570501+0000","last_became_peered":"2024-05-27T07:21:45.570501+0000","last_unstale":"2024-05-27T07:25:57.022886+0000","last_undegraded":"2024-05-27T07:25:57.022886+0000","last_fullsized":"2024-05-27T07:25:57.022886+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:21:44.543638+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:21:44.543638+0000","last_clean_scrub_stamp":"2024-05-27T07:21:44.543638+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T14:11:26.496194+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.a","version":"90'5","reported_seq":44,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-27T07:25:57.022260+0000","last_change":"2024-05-27T07:21:48.594690+0000","last_active":"2024-05-27T07:25:57.022260+0000","last_peered":"2024-05-27T07:25:57.022260+0000","last_clean":"2024-05-27T07:25:57.022260+0000","last_became_active":"2024-05-27T07:21:48.594314+0000","last_became_peered":"2024-05-27T07:21:48.594314+0000","last_unstale":"2024-05-27T07:25:57.022260+0000","last_undegraded":"2024-05-27T07:25:57.022260+0000","last_fullsized":"2024-05-27T07:25:57.022260+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:21:47.565641+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:21:47.565641+0000","last_clean_scrub_stamp":"2024-05-27T07:21:47.565641+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T14:06:43.401844+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,6],"acting":[7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":31,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-27T07:25:56.934946+0000","last_change":"2024-05-27T07:21:59.732289+0000","last_active":"2024-05-27T07:25:56.934946+0000","last_peered":"2024-05-27T07:25:56.934946+0000","last_clean":"2024-05-27T07:25:56.934946+0000","last_became_active":"2024-05-27T07:21:59.731863+0000","last_became_peered":"2024-05-27T07:21:59.731863+0000","last_unstale":"2024-05-27T07:25:56.934946+0000","last_undegraded":"2024-05-27T07:25:56.934946+0000","last_fullsized":"2024-05-27T07:25:56.934946+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:21:58.698952+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:21:58.698952+0000","last_clean_scrub_stamp":"2024-05-27T07:21:58.698952+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T11:06:38.246484+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,6,7],"acting":[1,5,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.c","version":"91'6","reported_seq":235,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-27T07:25:57.161598+0000","last_change":"2024-05-27T07:21:45.791978+0000","last_active":"2024-05-27T07:25:57.161598+0000","last_peered":"2024-05-27T07:25:57.161598+0000","last_clean":"2024-05-27T07:25:57.161598+0000","last_became_active":"2024-05-27T07:21:45.791146+0000","last_became_peered":"2024-05-27T07:21:45.791146+0000","last_unstale":"2024-05-27T07:25:57.161598+0000","last_undegraded":"2024-05-27T07:25:57.161598+0000","last_fullsized":"2024-05-27T07:25:57.161598+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:21:44.543638+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:21:44.543638+0000","last_clean_scrub_stamp":"2024-05-27T07:21:44.543638+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T09:09:24.952876+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1561,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":183,"num_read_kb":2,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,11,0],"acting":[2,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"90'7","reported_seq":46,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-27T07:25:57.022770+0000","last_change":"2024-05-27T07:21:48.602786+0000","last_active":"2024-05-27T07:25:57.022770+0000","last_peered":"2024-05-27T07:25:57.022770+0000","last_clean":"2024-05-27T07:25:57.022770+0000","last_became_active":"2024-05-27T07:21:48.602476+0000","last_became_peered":"2024-05-27T07:21:48.602476+0000","last_unstale":"2024-05-27T07:25:57.022770+0000","last_undegraded":"2024-05-27T07:25:57.022770+0000","last_fullsized":"2024-05-27T07:25:57.022770+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:21:47.565641+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:21:47.565641+0000","last_clean_scrub_stamp":"2024-05-27T07:21:47.565641+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T13:22:58.583230+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,11],"acting":[7,1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":31,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-27T07:25:57.023937+0000","last_change":"2024-05-27T07:21:59.729595+0000","last_active":"2024-05-27T07:25:57.023937+0000","last_peered":"2024-05-27T07:25:57.023937+0000","last_clean":"2024-05-27T07:25:57.023937+0000","last_became_active":"2024-05-27T07:21:59.729147+0000","last_became_peered":"2024-05-27T07:21:59.729147+0000","last_unstale":"2024-05-27T07:25:57.023937+0000","last_undegraded":"2024-05-27T07:25:57.023937+0000","last_fullsized":"2024-05-27T07:25:57.023937+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:21:58.698952+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:21:58.698952+0000","last_clean_scrub_stamp":"2024-05-27T07:21:58.698952+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T13:31:28.579617+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,9,1,4],"acting":[0,9,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.d","version":"91'41","reported_seq":95,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-27T07:25:57.174009+0000","last_change":"2024-05-27T07:21:45.569287+0000","last_active":"2024-05-27T07:25:57.174009+0000","last_peered":"2024-05-27T07:25:57.174009+0000","last_clean":"2024-05-27T07:25:57.174009+0000","last_became_active":"2024-05-27T07:21:45.568935+0000","last_became_peered":"2024-05-27T07:21:45.568935+0000","last_unstale":"2024-05-27T07:25:57.174009+0000","last_undegraded":"2024-05-27T07:25:57.174009+0000","last_fullsized":"2024-05-27T07:25:57.174009+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:21:44.543638+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:21:44.543638+0000","last_clean_scrub_stamp":"2024-05-27T07:21:44.543638+0000","objects_scrubbed":0,"log_size":41,"log_dups_size":0,"ondisk_log_size":41,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T07:27:14.674135+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":131670,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":11,"num_read_kb":10,"num_write":49,"num_write_kb":475,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,10,3],"acting":[5,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.c","version":"90'6","reported_seq":45,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-27T07:25:57.021865+0000","last_change":"2024-05-27T07:21:48.597718+0000","last_active":"2024-05-27T07:25:57.021865+0000","last_peered":"2024-05-27T07:25:57.021865+0000","last_clean":"2024-05-27T07:25:57.021865+0000","last_became_active":"2024-05-27T07:21:48.597379+0000","last_became_peered":"2024-05-27T07:21:48.597379+0000","last_unstale":"2024-05-27T07:25:57.021865+0000","last_undegraded":"2024-05-27T07:25:57.021865+0000","last_fullsized":"2024-05-27T07:25:57.021865+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:21:47.565641+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:21:47.565641+0000","last_clean_scrub_stamp":"2024-05-27T07:21:47.565641+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T08:48:41.658364+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,5],"acting":[7,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":31,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-27T07:25:57.022372+0000","last_change":"2024-05-27T07:21:59.730191+0000","last_active":"2024-05-27T07:25:57.022372+0000","last_peered":"2024-05-27T07:25:57.022372+0000","last_clean":"2024-05-27T07:25:57.022372+0000","last_became_active":"2024-05-27T07:21:59.729787+0000","last_became_peered":"2024-05-27T07:21:59.729787+0000","last_unstale":"2024-05-27T07:25:57.022372+0000","last_undegraded":"2024-05-27T07:25:57.022372+0000","last_fullsized":"2024-05-27T07:25:57.022372+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:21:58.698952+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:21:58.698952+0000","last_clean_scrub_stamp":"2024-05-27T07:21:58.698952+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T13:22:38.961246+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7,1],"acting":[5,2,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.e","version":"88'5","reported_seq":51,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-27T07:25:57.024170+0000","last_change":"2024-05-27T07:21:45.559219+0000","last_active":"2024-05-27T07:25:57.024170+0000","last_peered":"2024-05-27T07:25:57.024170+0000","last_clean":"2024-05-27T07:25:57.024170+0000","last_became_active":"2024-05-27T07:21:45.558848+0000","last_became_peered":"2024-05-27T07:21:45.558848+0000","last_unstale":"2024-05-27T07:25:57.024170+0000","last_undegraded":"2024-05-27T07:25:57.024170+0000","last_fullsized":"2024-05-27T07:25:57.024170+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:21:44.543638+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:21:44.543638+0000","last_clean_scrub_stamp":"2024-05-27T07:21:44.543638+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T15:03:55.632162+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.f","version":"90'9","reported_seq":48,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-27T07:25:57.022988+0000","last_change":"2024-05-27T07:21:48.596349+0000","last_active":"2024-05-27T07:25:57.022988+0000","last_peered":"2024-05-27T07:25:57.022988+0000","last_clean":"2024-05-27T07:25:57.022988+0000","last_became_active":"2024-05-27T07:21:48.595830+0000","last_became_peered":"2024-05-27T07:21:48.595830+0000","last_unstale":"2024-05-27T07:25:57.022988+0000","last_undegraded":"2024-05-27T07:25:57.022988+0000","last_fullsized":"2024-05-27T07:25:57.022988+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:21:47.565641+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:21:47.565641+0000","last_clean_scrub_stamp":"2024-05-27T07:21:47.565641+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T13:33:11.431988+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,8],"acting":[7,11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.20","version":"90'9","reported_seq":50,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-27T07:25:57.022558+0000","last_change":"2024-05-27T07:21:48.598504+0000","last_active":"2024-05-27T07:25:57.022558+0000","last_peered":"2024-05-27T07:25:57.022558+0000","last_clean":"2024-05-27T07:25:57.022558+0000","last_became_active":"2024-05-27T07:21:48.598103+0000","last_became_peered":"2024-05-27T07:21:48.598103+0000","last_unstale":"2024-05-27T07:25:57.022558+0000","last_undegraded":"2024-05-27T07:25:57.022558+0000","last_fullsized":"2024-05-27T07:25:57.022558+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:21:47.565641+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:21:47.565641+0000","last_clean_scrub_stamp":"2024-05-27T07:21:47.565641+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T10:34:02.743735+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,5],"acting":[11,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.21","version":"90'12","reported_seq":53,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-27T07:25:56.935301+0000","last_change":"2024-05-27T07:21:48.593379+0000","last_active":"2024-05-27T07:25:56.935301+0000","last_peered":"2024-05-27T07:25:56.935301+0000","last_clean":"2024-05-27T07:25:56.935301+0000","last_became_active":"2024-05-27T07:21:48.592995+0000","last_became_peered":"2024-05-27T07:21:48.592995+0000","last_unstale":"2024-05-27T07:25:56.935301+0000","last_undegraded":"2024-05-27T07:25:56.935301+0000","last_fullsized":"2024-05-27T07:25:56.935301+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:21:47.565641+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:21:47.565641+0000","last_clean_scrub_stamp":"2024-05-27T07:21:47.565641+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T19:01:56.288235+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,4],"acting":[2,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"90'8","reported_seq":49,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-27T07:25:56.935066+0000","last_change":"2024-05-27T07:21:48.592837+0000","last_active":"2024-05-27T07:25:56.935066+0000","last_peered":"2024-05-27T07:25:56.935066+0000","last_clean":"2024-05-27T07:25:56.935066+0000","last_became_active":"2024-05-27T07:21:48.592459+0000","last_became_peered":"2024-05-27T07:21:48.592459+0000","last_unstale":"2024-05-27T07:25:56.935066+0000","last_undegraded":"2024-05-27T07:25:56.935066+0000","last_fullsized":"2024-05-27T07:25:56.935066+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:21:47.565641+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:21:47.565641+0000","last_clean_scrub_stamp":"2024-05-27T07:21:47.565641+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T14:00:19.490348+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.23","version":"90'7","reported_seq":49,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-27T07:25:57.025975+0000","last_change":"2024-05-27T07:21:48.583139+0000","last_active":"2024-05-27T07:25:57.025975+0000","last_peered":"2024-05-27T07:25:57.025975+0000","last_clean":"2024-05-27T07:25:57.025975+0000","last_became_active":"2024-05-27T07:21:48.582822+0000","last_became_peered":"2024-05-27T07:21:48.582822+0000","last_unstale":"2024-05-27T07:25:57.025975+0000","last_undegraded":"2024-05-27T07:25:57.025975+0000","last_fullsized":"2024-05-27T07:25:57.025975+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:21:47.565641+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:21:47.565641+0000","last_clean_scrub_stamp":"2024-05-27T07:21:47.565641+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T16:30:03.454035+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,4],"acting":[0,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"90'4","reported_seq":45,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-27T07:25:57.024904+0000","last_change":"2024-05-27T07:21:48.588906+0000","last_active":"2024-05-27T07:25:57.024904+0000","last_peered":"2024-05-27T07:25:57.024904+0000","last_clean":"2024-05-27T07:25:57.024904+0000","last_became_active":"2024-05-27T07:21:48.588555+0000","last_became_peered":"2024-05-27T07:21:48.588555+0000","last_unstale":"2024-05-27T07:25:57.024904+0000","last_undegraded":"2024-05-27T07:25:57.024904+0000","last_fullsized":"2024-05-27T07:25:57.024904+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:21:47.565641+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:21:47.565641+0000","last_clean_scrub_stamp":"2024-05-27T07:21:47.565641+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T19:04:23.006852+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"90'6","reported_seq":47,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-27T07:25:57.023930+0000","last_change":"2024-05-27T07:21:48.590271+0000","last_active":"2024-05-27T07:25:57.023930+0000","last_peered":"2024-05-27T07:25:57.023930+0000","last_clean":"2024-05-27T07:25:57.023930+0000","last_became_active":"2024-05-27T07:21:48.589898+0000","last_became_peered":"2024-05-27T07:21:48.589898+0000","last_unstale":"2024-05-27T07:25:57.023930+0000","last_undegraded":"2024-05-27T07:25:57.023930+0000","last_fullsized":"2024-05-27T07:25:57.023930+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:21:47.565641+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:21:47.565641+0000","last_clean_scrub_stamp":"2024-05-27T07:21:47.565641+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T11:57:43.238211+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,5],"acting":[11,8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.26","version":"90'15","reported_seq":56,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-27T07:25:56.935533+0000","last_change":"2024-05-27T07:21:48.597321+0000","last_active":"2024-05-27T07:25:56.935533+0000","last_peered":"2024-05-27T07:25:56.935533+0000","last_clean":"2024-05-27T07:25:56.935533+0000","last_became_active":"2024-05-27T07:21:48.596953+0000","last_became_peered":"2024-05-27T07:21:48.596953+0000","last_unstale":"2024-05-27T07:25:56.935533+0000","last_undegraded":"2024-05-27T07:25:56.935533+0000","last_fullsized":"2024-05-27T07:25:56.935533+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:21:47.565641+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:21:47.565641+0000","last_clean_scrub_stamp":"2024-05-27T07:21:47.565641+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T16:21:17.645098+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,8,1],"acting":[2,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.27","version":"90'2","reported_seq":43,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-27T07:25:57.022779+0000","last_change":"2024-05-27T07:21:48.593717+0000","last_active":"2024-05-27T07:25:57.022779+0000","last_peered":"2024-05-27T07:25:57.022779+0000","last_clean":"2024-05-27T07:25:57.022779+0000","last_became_active":"2024-05-27T07:21:48.593356+0000","last_became_peered":"2024-05-27T07:21:48.593356+0000","last_unstale":"2024-05-27T07:25:57.022779+0000","last_undegraded":"2024-05-27T07:25:57.022779+0000","last_fullsized":"2024-05-27T07:25:57.022779+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:21:47.565641+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:21:47.565641+0000","last_clean_scrub_stamp":"2024-05-27T07:21:47.565641+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T17:06:27.341627+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"90'6","reported_seq":47,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-27T07:25:57.022723+0000","last_change":"2024-05-27T07:21:48.592360+0000","last_active":"2024-05-27T07:25:57.022723+0000","last_peered":"2024-05-27T07:25:57.022723+0000","last_clean":"2024-05-27T07:25:57.022723+0000","last_became_active":"2024-05-27T07:21:48.591650+0000","last_became_peered":"2024-05-27T07:21:48.591650+0000","last_unstale":"2024-05-27T07:25:57.022723+0000","last_undegraded":"2024-05-27T07:25:57.022723+0000","last_fullsized":"2024-05-27T07:25:57.022723+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:21:47.565641+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:21:47.565641+0000","last_clean_scrub_stamp":"2024-05-27T07:21:47.565641+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T17:32:19.408787+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,3],"acting":[5,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"90'8","reported_seq":49,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-27T07:25:57.022029+0000","last_change":"2024-05-27T07:21:48.587319+0000","last_active":"2024-05-27T07:25:57.022029+0000","last_peered":"2024-05-27T07:25:57.022029+0000","last_clean":"2024-05-27T07:25:57.022029+0000","last_became_active":"2024-05-27T07:21:48.586924+0000","last_became_peered":"2024-05-27T07:21:48.586924+0000","last_unstale":"2024-05-27T07:25:57.022029+0000","last_undegraded":"2024-05-27T07:25:57.022029+0000","last_fullsized":"2024-05-27T07:25:57.022029+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:21:47.565641+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:21:47.565641+0000","last_clean_scrub_stamp":"2024-05-27T07:21:47.565641+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T09:28:48.129037+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,11],"acting":[3,2,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2a","version":"90'10","reported_seq":51,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-27T07:25:56.935355+0000","last_change":"2024-05-27T07:21:48.604018+0000","last_active":"2024-05-27T07:25:56.935355+0000","last_peered":"2024-05-27T07:25:56.935355+0000","last_clean":"2024-05-27T07:25:56.935355+0000","last_became_active":"2024-05-27T07:21:48.603368+0000","last_became_peered":"2024-05-27T07:21:48.603368+0000","last_unstale":"2024-05-27T07:25:56.935355+0000","last_undegraded":"2024-05-27T07:25:56.935355+0000","last_fullsized":"2024-05-27T07:25:56.935355+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:21:47.565641+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:21:47.565641+0000","last_clean_scrub_stamp":"2024-05-27T07:21:47.565641+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T09:49:42.440480+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2b","version":"90'11","reported_seq":52,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-27T07:25:57.024693+0000","last_change":"2024-05-27T07:21:48.589434+0000","last_active":"2024-05-27T07:25:57.024693+0000","last_peered":"2024-05-27T07:25:57.024693+0000","last_clean":"2024-05-27T07:25:57.024693+0000","last_became_active":"2024-05-27T07:21:48.589102+0000","last_became_peered":"2024-05-27T07:21:48.589102+0000","last_unstale":"2024-05-27T07:25:57.024693+0000","last_undegraded":"2024-05-27T07:25:57.024693+0000","last_fullsized":"2024-05-27T07:25:57.024693+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:21:47.565641+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:21:47.565641+0000","last_clean_scrub_stamp":"2024-05-27T07:21:47.565641+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T18:28:52.955138+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2c","version":"90'8","reported_seq":55,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-27T07:25:57.023752+0000","last_change":"2024-05-27T07:22:30.802433+0000","last_active":"2024-05-27T07:25:57.023752+0000","last_peered":"2024-05-27T07:25:57.023752+0000","last_clean":"2024-05-27T07:25:57.023752+0000","last_became_active":"2024-05-27T07:22:30.801881+0000","last_became_peered":"2024-05-27T07:22:30.801881+0000","last_unstale":"2024-05-27T07:25:57.023752+0000","last_undegraded":"2024-05-27T07:25:57.023752+0000","last_fullsized":"2024-05-27T07:25:57.023752+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:21:47.565641+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:21:47.565641+0000","last_clean_scrub_stamp":"2024-05-27T07:21:47.565641+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T11:59:17.916052+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,9,8],"acting":[11,9,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":234,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":32768,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":128,"acting":128,"num_store_stats":12},{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":500,"ondisk_log_size":500,"up":192,"acting":192,"num_store_stats":12},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":81,"num_read_kb":69,"num_write":123,"num_write_kb":2150,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":2951840,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":147456,"data_stored":1771104,"data_compressed":108480,"data_compressed_allocated":135168,"data_compressed_original":1769472,"omap_allocated":0,"internal_metadata":0},"log_size":92,"ondisk_log_size":92,"up":3,"acting":3,"num_store_stats":3},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":604805,"num_objects":100,"num_object_clones":0,"num_object_copies":300,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":100,"num_whiteouts":0,"num_read":1954,"num_read_kb":1099,"num_write":386,"num_write_kb":1479,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":22,"num_keys_recovered":0,"num_objects_omap":66,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":737280,"data_stored":1814415,"data_compressed":75906,"data_compressed_allocated":98304,"data_compressed_original":1548288,"omap_allocated":0,"internal_metadata":0},"log_size":311,"ondisk_log_size":311,"up":96,"acting":96,"num_store_stats":12}],"osd_stats":[{"osd":11,"up_from":71,"seq":304942678081,"num_pgs":29,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32168,"kb_used_data":988,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712984,"statfs":{"total":95995035648,"available":95962095616,"internally_reserved":0,"allocated":1011712,"data_stored":529010,"data_compressed":35393,"data_compressed_allocated":49152,"data_compressed_original":98304,"omap_allocated":1596,"internal_metadata":31914436},"hb_peers":[0,1,2,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":65,"seq":279172874309,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28160,"kb_used_data":1140,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716992,"statfs":{"total":95995035648,"available":95966199808,"internally_reserved":0,"allocated":1167360,"data_stored":1186884,"data_compressed":71553,"data_compressed_allocated":94208,"data_compressed_original":688128,"omap_allocated":1600,"internal_metadata":27654592},"hb_peers":[0,1,2,3,4,5,6,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":60,"seq":257698037835,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36372,"kb_used_data":1036,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708780,"statfs":{"total":95995035648,"available":95957790720,"internally_reserved":0,"allocated":1060864,"data_stored":926230,"data_compressed":54539,"data_compressed_allocated":73728,"data_compressed_original":491520,"omap_allocated":1600,"internal_metadata":36174272},"hb_peers":[0,1,2,3,4,5,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":54,"seq":231928234065,"num_pgs":31,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28044,"kb_used_data":1024,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717108,"statfs":{"total":95995035648,"available":95966318592,"internally_reserved":0,"allocated":1048576,"data_stored":922490,"data_compressed":54539,"data_compressed_allocated":73728,"data_compressed_original":491520,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":48,"seq":206158430293,"num_pgs":37,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32236,"kb_used_data":1056,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712916,"statfs":{"total":95995035648,"available":95962025984,"internally_reserved":0,"allocated":1081344,"data_stored":1127639,"data_compressed":71553,"data_compressed_allocated":94208,"data_compressed_original":688128,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":43,"seq":184683593819,"num_pgs":32,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28004,"kb_used_data":984,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717148,"statfs":{"total":95995035648,"available":95966359552,"internally_reserved":0,"allocated":1007616,"data_stored":524813,"data_compressed":35393,"data_compressed_allocated":49152,"data_compressed_original":98304,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574964,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36400,"kb_used_data":1060,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708752,"statfs":{"total":95995035648,"available":95957762048,"internally_reserved":0,"allocated":1085440,"data_stored":596782,"data_compressed":35393,"data_compressed_allocated":49152,"data_compressed_original":98304,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738490,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28028,"kb_used_data":1008,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717124,"statfs":{"total":95995035648,"available":95966334976,"internally_reserved":0,"allocated":1032192,"data_stored":526269,"data_compressed":35393,"data_compressed_allocated":49152,"data_compressed_original":98304,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411439,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36412,"kb_used_data":1072,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708740,"statfs":{"total":95995035648,"available":95957749760,"internally_reserved":0,"allocated":1097728,"data_stored":597813,"data_compressed":35393,"data_compressed_allocated":49152,"data_compressed_original":98304,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,1,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":26,"seq":111669149802,"num_pgs":40,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36376,"kb_used_data":1040,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708776,"statfs":{"total":95995035648,"available":95957786624,"internally_reserved":0,"allocated":1064960,"data_stored":1048946,"data_compressed":60695,"data_compressed_allocated":81920,"data_compressed_original":614400,"omap_allocated":1593,"internal_metadata":36174279},"hb_peers":[0,1,2,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":32,"seq":137438953572,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28020,"kb_used_data":1000,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717132,"statfs":{"total":95995035648,"available":95966343168,"internally_reserved":0,"allocated":1024000,"data_stored":529531,"data_compressed":35393,"data_compressed_allocated":49152,"data_compressed_original":98304,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":37,"seq":158913790047,"num_pgs":44,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28112,"kb_used_data":1084,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717040,"statfs":{"total":95995035648,"available":95966248960,"internally_reserved":0,"allocated":1110016,"data_stored":1253507,"data_compressed":77709,"data_compressed_allocated":102400,"data_compressed_original":811008,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":36160,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":36160,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":36160,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":2569,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":94208,"data_stored":73082,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":106496,"data_stored":74113,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":525246,"data_compressed":25302,"data_compressed_allocated":32768,"data_compressed_original":516096,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1735,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":135343,"data_compressed":6156,"data_compressed_allocated":8192,"data_compressed_original":122880,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1113,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":5379,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":394694,"data_compressed":19146,"data_compressed_allocated":24576,"data_compressed_original":393216,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":394338,"data_compressed":19146,"data_compressed_allocated":24576,"data_compressed_original":393216,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":147456,"data_stored":205198,"data_compressed":6156,"data_compressed_allocated":8192,"data_compressed_original":122880,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1605,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":8192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":8192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-05-27T07:26:03.310 DEBUG:teuthology.orchestra.run.smithi052:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-05-27T07:26:03.352 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:26:03 smithi173 ceph-mon[37000]: Updating smithi052:/etc/ceph/ceph.client.admin.keyring 2024-05-27T07:26:03.352 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:26:03 smithi173 ceph-mon[37000]: Updating smithi173:/etc/ceph/ceph.client.admin.keyring 2024-05-27T07:26:03.352 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:26:03 smithi173 ceph-mon[37000]: Updating smithi105:/etc/ceph/ceph.client.admin.keyring 2024-05-27T07:26:03.352 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:26:03 smithi173 ceph-mon[37000]: Updating smithi052:/var/lib/ceph/73df2636-1bf8-11ef-bc9b-c7b262605968/config/ceph.client.admin.keyring 2024-05-27T07:26:03.352 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:26:03 smithi173 ceph-mon[37000]: Updating smithi173:/var/lib/ceph/73df2636-1bf8-11ef-bc9b-c7b262605968/config/ceph.client.admin.keyring 2024-05-27T07:26:03.352 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:26:03 smithi173 ceph-mon[37000]: Updating smithi105:/var/lib/ceph/73df2636-1bf8-11ef-bc9b-c7b262605968/config/ceph.client.admin.keyring 2024-05-27T07:26:03.352 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:26:03 smithi173 ceph-mon[37000]: from='client.? 172.21.15.52:0/3248081509' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-27T07:26:03.352 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:26:03 smithi173 ceph-mon[37000]: from='mgr.15333 172.21.15.52:0/416389082' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-27T07:26:03.352 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:26:03 smithi173 ceph-mon[37000]: from='mgr.15333 172.21.15.52:0/416389082' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T07:26:03.352 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:26:03 smithi173 ceph-mon[37000]: from='mgr.15333 172.21.15.52:0/416389082' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-27T07:26:03.352 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:26:03 smithi173 ceph-mon[37000]: from='mgr.15333 172.21.15.52:0/416389082' entity='mgr.x' 2024-05-27T07:26:03.440 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:26:03 smithi105 ceph-mon[35977]: Updating smithi052:/etc/ceph/ceph.client.admin.keyring 2024-05-27T07:26:03.440 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:26:03 smithi105 ceph-mon[35977]: Updating smithi173:/etc/ceph/ceph.client.admin.keyring 2024-05-27T07:26:03.440 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:26:03 smithi105 ceph-mon[35977]: Updating smithi105:/etc/ceph/ceph.client.admin.keyring 2024-05-27T07:26:03.440 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:26:03 smithi105 ceph-mon[35977]: Updating smithi052:/var/lib/ceph/73df2636-1bf8-11ef-bc9b-c7b262605968/config/ceph.client.admin.keyring 2024-05-27T07:26:03.440 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:26:03 smithi105 ceph-mon[35977]: Updating smithi173:/var/lib/ceph/73df2636-1bf8-11ef-bc9b-c7b262605968/config/ceph.client.admin.keyring 2024-05-27T07:26:03.440 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:26:03 smithi105 ceph-mon[35977]: Updating smithi105:/var/lib/ceph/73df2636-1bf8-11ef-bc9b-c7b262605968/config/ceph.client.admin.keyring 2024-05-27T07:26:03.440 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:26:03 smithi105 ceph-mon[35977]: from='client.? 172.21.15.52:0/3248081509' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-27T07:26:03.441 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:26:03 smithi105 ceph-mon[35977]: from='mgr.15333 172.21.15.52:0/416389082' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-27T07:26:03.441 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:26:03 smithi105 ceph-mon[35977]: from='mgr.15333 172.21.15.52:0/416389082' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T07:26:03.441 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:26:03 smithi105 ceph-mon[35977]: from='mgr.15333 172.21.15.52:0/416389082' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-27T07:26:03.441 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:26:03 smithi105 ceph-mon[35977]: from='mgr.15333 172.21.15.52:0/416389082' entity='mgr.x' 2024-05-27T07:26:03.459 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:03.469+0000 7fd9fedf2640 1 -- 172.21.15.52:0/3433300782 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fd9f8103530 msgr2=0x7fd9f8103940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:26:03.460 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:03.469+0000 7fd9fedf2640 1 --2- 172.21.15.52:0/3433300782 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fd9f8103530 0x7fd9f8103940 secure :-1 s=READY pgs=416 cs=0 l=1 rev1=1 crypto rx=0x7fd9e0009900 tx=0x7fd9e002a410 comp rx=0 tx=0).stop 2024-05-27T07:26:03.460 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:03.469+0000 7fd9fedf2640 1 -- 172.21.15.52:0/3433300782 shutdown_connections 2024-05-27T07:26:03.460 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:03.469+0000 7fd9fedf2640 1 --2- 172.21.15.52:0/3433300782 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fd9f8107e10 0x7fd9f810f640 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:26:03.460 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:03.469+0000 7fd9fedf2640 1 --2- 172.21.15.52:0/3433300782 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fd9f8107390 0x7fd9f81077a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:26:03.460 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:03.469+0000 7fd9fedf2640 1 --2- 172.21.15.52:0/3433300782 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fd9f8103530 0x7fd9f8103940 unknown :-1 s=CLOSED pgs=416 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:26:03.460 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:03.470+0000 7fd9fedf2640 1 -- 172.21.15.52:0/3433300782 >> 172.21.15.52:0/3433300782 conn(0x7fd9f80fd970 msgr2=0x7fd9f80ffdc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:26:03.460 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:03.470+0000 7fd9fedf2640 1 -- 172.21.15.52:0/3433300782 shutdown_connections 2024-05-27T07:26:03.460 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:03.470+0000 7fd9fedf2640 1 -- 172.21.15.52:0/3433300782 wait complete. 2024-05-27T07:26:03.462 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:03.470+0000 7fd9fedf2640 1 Processor -- start 2024-05-27T07:26:03.462 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:03.470+0000 7fd9fedf2640 1 -- start start 2024-05-27T07:26:03.462 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:03.471+0000 7fd9fedf2640 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fd9f8103530 0x7fd9f819a7e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:26:03.463 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:03.471+0000 7fd9fedf2640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fd9f8107390 0x7fd9f819ad40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:26:03.463 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:03.471+0000 7fd9fedf2640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fd9f8107e10 0x7fd9f819f000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:26:03.463 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:03.471+0000 7fd9fedf2640 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x7fd9f8112d10 con 0x7fd9f8103530 2024-05-27T07:26:03.463 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:03.471+0000 7fd9fedf2640 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7fd9f8112b90 con 0x7fd9f8107e10 2024-05-27T07:26:03.463 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:03.471+0000 7fd9fedf2640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7fd9f8112e90 con 0x7fd9f8107390 2024-05-27T07:26:03.463 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:03.471+0000 7fd9fcb8e640 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fd9f8103530 0x7fd9f819a7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:26:03.463 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:03.471+0000 7fd9fcb8e640 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fd9f8103530 0x7fd9f819a7e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.52:58430/0 (socket says 172.21.15.52:58430) 2024-05-27T07:26:03.463 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:03.471+0000 7fd9fcb8e640 1 -- 172.21.15.52:0/4149831447 learned_addr learned my addr 172.21.15.52:0/4149831447 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-05-27T07:26:03.463 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:03.471+0000 7fd9fd38f640 1 --2- 172.21.15.52:0/4149831447 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fd9f8107e10 0x7fd9f819f000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:26:03.463 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:03.471+0000 7fd9f7fff640 1 --2- 172.21.15.52:0/4149831447 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fd9f8107390 0x7fd9f819ad40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:26:03.463 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:03.471+0000 7fd9fcb8e640 1 -- 172.21.15.52:0/4149831447 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fd9f8107390 msgr2=0x7fd9f819ad40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:26:03.463 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:03.471+0000 7fd9fcb8e640 1 --2- 172.21.15.52:0/4149831447 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fd9f8107390 0x7fd9f819ad40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:26:03.463 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:03.471+0000 7fd9fcb8e640 1 -- 172.21.15.52:0/4149831447 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fd9f8107e10 msgr2=0x7fd9f819f000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:26:03.464 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:03.471+0000 7fd9fcb8e640 1 --2- 172.21.15.52:0/4149831447 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fd9f8107e10 0x7fd9f819f000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:26:03.464 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:03.471+0000 7fd9fcb8e640 1 -- 172.21.15.52:0/4149831447 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd9f819f640 con 0x7fd9f8103530 2024-05-27T07:26:03.464 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:03.471+0000 7fd9fcb8e640 1 --2- 172.21.15.52:0/4149831447 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fd9f8103530 0x7fd9f819a7e0 secure :-1 s=READY pgs=417 cs=0 l=1 rev1=1 crypto rx=0x7fd9e0006010 tx=0x7fd9e0004060 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:26:03.464 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:03.471+0000 7fd9f5ffb640 1 -- 172.21.15.52:0/4149831447 <== mon.0 v2:172.21.15.52:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd9e003f030 con 0x7fd9f8103530 2024-05-27T07:26:03.464 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:03.471+0000 7fd9fedf2640 1 -- 172.21.15.52:0/4149831447 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd9f819f8d0 con 0x7fd9f8103530 2024-05-27T07:26:03.464 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:03.471+0000 7fd9f5ffb640 1 -- 172.21.15.52:0/4149831447 <== mon.0 v2:172.21.15.52:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fd9e0003aa0 con 0x7fd9f8103530 2024-05-27T07:26:03.464 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:03.472+0000 7fd9f5ffb640 1 -- 172.21.15.52:0/4149831447 <== mon.0 v2:172.21.15.52:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd9e0035070 con 0x7fd9f8103530 2024-05-27T07:26:03.464 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:03.472+0000 7fd9fedf2640 1 -- 172.21.15.52:0/4149831447 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd9f819fe10 con 0x7fd9f8103530 2024-05-27T07:26:03.464 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:03.473+0000 7fd9fedf2640 1 -- 172.21.15.52:0/4149831447 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd9f806a2b0 con 0x7fd9f8103530 2024-05-27T07:26:03.465 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:03.474+0000 7fd9f5ffb640 1 -- 172.21.15.52:0/4149831447 <== mon.0 v2:172.21.15.52:3300/0 4 ==== mgrmap(e 24) ==== 149879+0+0 (secure 0 0 0) 0x7fd9e0042030 con 0x7fd9f8103530 2024-05-27T07:26:03.465 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:03.475+0000 7fd9f5ffb640 1 --2- 172.21.15.52:0/4149831447 >> [v2:172.21.15.52:6800/1822318384,v1:172.21.15.52:6801/1822318384] conn(0x7fd9e40b2a80 0x7fd9e40b4f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:26:03.465 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:03.475+0000 7fd9f5ffb640 1 -- 172.21.15.52:0/4149831447 <== mon.0 v2:172.21.15.52:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9379+0+0 (secure 0 0 0) 0x7fd9e0103470 con 0x7fd9f8103530 2024-05-27T07:26:03.465 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:03.475+0000 7fd9f7fff640 1 --2- 172.21.15.52:0/4149831447 >> [v2:172.21.15.52:6800/1822318384,v1:172.21.15.52:6801/1822318384] conn(0x7fd9e40b2a80 0x7fd9e40b4f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:26:03.465 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:03.475+0000 7fd9f7fff640 1 --2- 172.21.15.52:0/4149831447 >> [v2:172.21.15.52:6800/1822318384,v1:172.21.15.52:6801/1822318384] conn(0x7fd9e40b2a80 0x7fd9e40b4f90 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7fd9f8101ef0 tx=0x7fd9e8009870 comp rx=0 tx=0).ready entity=mgr.15333 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:26:03.468 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:03.478+0000 7fd9f5ffb640 1 -- 172.21.15.52:0/4149831447 <== mon.0 v2:172.21.15.52:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191963 (secure 0 0 0) 0x7fd9e00ca970 con 0x7fd9f8103530 2024-05-27T07:26:03.655 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:03.665+0000 7fd9fedf2640 1 -- 172.21.15.52:0/4149831447 --> [v2:172.21.15.52:6800/1822318384,v1:172.21.15.52:6801/1822318384] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7fd9f80fff90 con 0x7fd9e40b2a80 2024-05-27T07:26:03.664 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:03.675+0000 7fd9f5ffb640 1 -- 172.21.15.52:0/4149831447 <== mgr.15333 v2:172.21.15.52:6800/1822318384 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+339710 (secure 0 0 0) 0x7fd9f80fff90 con 0x7fd9e40b2a80 2024-05-27T07:26:03.668 INFO:teuthology.orchestra.run.smithi052.stdout: 2024-05-27T07:26:03.670 INFO:teuthology.orchestra.run.smithi052.stderr:dumped all 2024-05-27T07:26:03.672 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:03.678+0000 7fd9fedf2640 1 -- 172.21.15.52:0/4149831447 >> [v2:172.21.15.52:6800/1822318384,v1:172.21.15.52:6801/1822318384] conn(0x7fd9e40b2a80 msgr2=0x7fd9e40b4f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:26:03.672 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:03.678+0000 7fd9fedf2640 1 --2- 172.21.15.52:0/4149831447 >> [v2:172.21.15.52:6800/1822318384,v1:172.21.15.52:6801/1822318384] conn(0x7fd9e40b2a80 0x7fd9e40b4f90 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7fd9f8101ef0 tx=0x7fd9e8009870 comp rx=0 tx=0).stop 2024-05-27T07:26:03.672 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:03.678+0000 7fd9fedf2640 1 -- 172.21.15.52:0/4149831447 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fd9f8103530 msgr2=0x7fd9f819a7e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:26:03.672 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:03.679+0000 7fd9fedf2640 1 --2- 172.21.15.52:0/4149831447 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fd9f8103530 0x7fd9f819a7e0 secure :-1 s=READY pgs=417 cs=0 l=1 rev1=1 crypto rx=0x7fd9e0006010 tx=0x7fd9e0004060 comp rx=0 tx=0).stop 2024-05-27T07:26:03.672 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:03.679+0000 7fd9fedf2640 1 -- 172.21.15.52:0/4149831447 shutdown_connections 2024-05-27T07:26:03.672 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:03.679+0000 7fd9fedf2640 1 --2- 172.21.15.52:0/4149831447 >> [v2:172.21.15.52:6800/1822318384,v1:172.21.15.52:6801/1822318384] conn(0x7fd9e40b2a80 0x7fd9e40b4f90 unknown :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:26:03.672 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:03.679+0000 7fd9fedf2640 1 --2- 172.21.15.52:0/4149831447 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fd9f8107e10 0x7fd9f819f000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:26:03.672 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:03.679+0000 7fd9fedf2640 1 --2- 172.21.15.52:0/4149831447 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fd9f8107390 0x7fd9f819ad40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:26:03.673 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:03.679+0000 7fd9fedf2640 1 --2- 172.21.15.52:0/4149831447 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fd9f8103530 0x7fd9f819a7e0 unknown :-1 s=CLOSED pgs=417 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:26:03.673 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:03.679+0000 7fd9fedf2640 1 -- 172.21.15.52:0/4149831447 >> 172.21.15.52:0/4149831447 conn(0x7fd9f80fd970 msgr2=0x7fd9f8102330 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:26:03.673 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:03.679+0000 7fd9fedf2640 1 -- 172.21.15.52:0/4149831447 shutdown_connections 2024-05-27T07:26:03.673 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:03.679+0000 7fd9fedf2640 1 -- 172.21.15.52:0/4149831447 wait complete. 2024-05-27T07:26:03.680 INFO:teuthology.orchestra.run.smithi052.stdout:{"pg_ready":true,"pg_map":{"version":6,"stamp":"2024-05-27T07:26:02.949326+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":1195407,"num_objects":104,"num_object_clones":0,"num_object_copies":314,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":104,"num_whiteouts":0,"num_read":2035,"num_read_kb":1168,"num_write":511,"num_write_kb":3631,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":2951862,"num_keys_recovered":0,"num_objects_omap":66,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":905,"ondisk_log_size":905,"up":419,"acting":419,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":419,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":378332,"kb_used_data":12492,"kb_used_omap":18,"kb_used_meta":365677,"kb_avail":1124563492,"statfs":{"total":1151940427776,"available":1151553015808,"internally_reserved":0,"allocated":12791808,"data_stored":9769914,"data_compressed":602946,"data_compressed_allocated":815104,"data_compressed_original":4374528,"omap_allocated":19143,"internal_metadata":374453561},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":5,"num_read_kb":5,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.988934"},"pg_stats":[{"pgid":"3.3f","version":"90'8","reported_seq":49,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-27T07:25:57.024226+0000","last_change":"2024-05-27T07:21:48.595870+0000","last_active":"2024-05-27T07:25:57.024226+0000","last_peered":"2024-05-27T07:25:57.024226+0000","last_clean":"2024-05-27T07:25:57.024226+0000","last_became_active":"2024-05-27T07:21:48.595020+0000","last_became_peered":"2024-05-27T07:21:48.595020+0000","last_unstale":"2024-05-27T07:25:57.024226+0000","last_undegraded":"2024-05-27T07:25:57.024226+0000","last_fullsized":"2024-05-27T07:25:57.024226+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:21:47.565641+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:21:47.565641+0000","last_clean_scrub_stamp":"2024-05-27T07:21:47.565641+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T16:43:08.920344+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.3e","version":"90'4","reported_seq":46,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-27T07:25:57.026092+0000","last_change":"2024-05-27T07:21:48.590237+0000","last_active":"2024-05-27T07:25:57.026092+0000","last_peered":"2024-05-27T07:25:57.026092+0000","last_clean":"2024-05-27T07:25:57.026092+0000","last_became_active":"2024-05-27T07:21:48.589876+0000","last_became_peered":"2024-05-27T07:21:48.589876+0000","last_unstale":"2024-05-27T07:25:57.026092+0000","last_undegraded":"2024-05-27T07:25:57.026092+0000","last_fullsized":"2024-05-27T07:25:57.026092+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:21:47.565641+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:21:47.565641+0000","last_clean_scrub_stamp":"2024-05-27T07:21:47.565641+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T16:23:08.881871+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,11,10],"acting":[0,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3d","version":"90'7","reported_seq":49,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-27T07:25:57.025547+0000","last_change":"2024-05-27T07:21:48.597744+0000","last_active":"2024-05-27T07:25:57.025547+0000","last_peered":"2024-05-27T07:25:57.025547+0000","last_clean":"2024-05-27T07:25:57.025547+0000","last_became_active":"2024-05-27T07:21:48.597410+0000","last_became_peered":"2024-05-27T07:21:48.597410+0000","last_unstale":"2024-05-27T07:25:57.025547+0000","last_undegraded":"2024-05-27T07:25:57.025547+0000","last_fullsized":"2024-05-27T07:25:57.025547+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:21:47.565641+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:21:47.565641+0000","last_clean_scrub_stamp":"2024-05-27T07:21:47.565641+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T19:06:49.643152+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"90'8","reported_seq":26,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-27T07:25:56.931367+0000","last_change":"2024-05-27T07:22:30.800824+0000","last_active":"2024-05-27T07:25:56.931367+0000","last_peered":"2024-05-27T07:25:56.931367+0000","last_clean":"2024-05-27T07:25:56.931367+0000","last_became_active":"2024-05-27T07:22:30.800415+0000","last_became_peered":"2024-05-27T07:22:30.800415+0000","last_unstale":"2024-05-27T07:25:56.931367+0000","last_undegraded":"2024-05-27T07:25:56.931367+0000","last_fullsized":"2024-05-27T07:25:56.931367+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:21:47.565641+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:21:47.565641+0000","last_clean_scrub_stamp":"2024-05-27T07:21:47.565641+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T11:06:10.451686+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,2],"acting":[9,8,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.3b","version":"90'4","reported_seq":45,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-27T07:25:57.023025+0000","last_change":"2024-05-27T07:21:48.583219+0000","last_active":"2024-05-27T07:25:57.023025+0000","last_peered":"2024-05-27T07:25:57.023025+0000","last_clean":"2024-05-27T07:25:57.023025+0000","last_became_active":"2024-05-27T07:21:48.582813+0000","last_became_peered":"2024-05-27T07:21:48.582813+0000","last_unstale":"2024-05-27T07:25:57.023025+0000","last_undegraded":"2024-05-27T07:25:57.023025+0000","last_fullsized":"2024-05-27T07:25:57.023025+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:21:47.565641+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:21:47.565641+0000","last_clean_scrub_stamp":"2024-05-27T07:21:47.565641+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T18:05:05.770965+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,3,6],"acting":[8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.3a","version":"90'6","reported_seq":47,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-27T07:25:57.023067+0000","last_change":"2024-05-27T07:21:48.592049+0000","last_active":"2024-05-27T07:25:57.023067+0000","last_peered":"2024-05-27T07:25:57.023067+0000","last_clean":"2024-05-27T07:25:57.023067+0000","last_became_active":"2024-05-27T07:21:48.591677+0000","last_became_peered":"2024-05-27T07:21:48.591677+0000","last_unstale":"2024-05-27T07:25:57.023067+0000","last_undegraded":"2024-05-27T07:25:57.023067+0000","last_fullsized":"2024-05-27T07:25:57.023067+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:21:47.565641+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:21:47.565641+0000","last_clean_scrub_stamp":"2024-05-27T07:21:47.565641+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T16:25:49.755893+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,4],"acting":[10,9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.39","version":"90'5","reported_seq":44,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-27T07:25:57.021734+0000","last_change":"2024-05-27T07:21:48.597876+0000","last_active":"2024-05-27T07:25:57.021734+0000","last_peered":"2024-05-27T07:25:57.021734+0000","last_clean":"2024-05-27T07:25:57.021734+0000","last_became_active":"2024-05-27T07:21:48.597566+0000","last_became_peered":"2024-05-27T07:21:48.597566+0000","last_unstale":"2024-05-27T07:25:57.021734+0000","last_undegraded":"2024-05-27T07:25:57.021734+0000","last_fullsized":"2024-05-27T07:25:57.021734+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:21:47.565641+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:21:47.565641+0000","last_clean_scrub_stamp":"2024-05-27T07:21:47.565641+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T09:30:43.438209+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,5],"acting":[7,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"90'9","reported_seq":50,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-27T07:25:57.021645+0000","last_change":"2024-05-27T07:21:48.593412+0000","last_active":"2024-05-27T07:25:57.021645+0000","last_peered":"2024-05-27T07:25:57.021645+0000","last_clean":"2024-05-27T07:25:57.021645+0000","last_became_active":"2024-05-27T07:21:48.593070+0000","last_became_peered":"2024-05-27T07:21:48.593070+0000","last_unstale":"2024-05-27T07:25:57.021645+0000","last_undegraded":"2024-05-27T07:25:57.021645+0000","last_fullsized":"2024-05-27T07:25:57.021645+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:21:47.565641+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:21:47.565641+0000","last_clean_scrub_stamp":"2024-05-27T07:21:47.565641+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T14:53:49.382796+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,10],"acting":[3,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"90'11","reported_seq":52,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-27T07:25:56.935451+0000","last_change":"2024-05-27T07:21:48.607244+0000","last_active":"2024-05-27T07:25:56.935451+0000","last_peered":"2024-05-27T07:25:56.935451+0000","last_clean":"2024-05-27T07:25:56.935451+0000","last_became_active":"2024-05-27T07:21:48.606238+0000","last_became_peered":"2024-05-27T07:21:48.606238+0000","last_unstale":"2024-05-27T07:25:56.935451+0000","last_undegraded":"2024-05-27T07:25:56.935451+0000","last_fullsized":"2024-05-27T07:25:56.935451+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:21:47.565641+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:21:47.565641+0000","last_clean_scrub_stamp":"2024-05-27T07:21:47.565641+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T12:36:33.142111+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,11],"acting":[1,5,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.36","version":"90'16","reported_seq":57,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-27T07:25:57.022630+0000","last_change":"2024-05-27T07:21:48.593275+0000","last_active":"2024-05-27T07:25:57.022630+0000","last_peered":"2024-05-27T07:25:57.022630+0000","last_clean":"2024-05-27T07:25:57.022630+0000","last_became_active":"2024-05-27T07:21:48.592890+0000","last_became_peered":"2024-05-27T07:21:48.592890+0000","last_unstale":"2024-05-27T07:25:57.022630+0000","last_undegraded":"2024-05-27T07:25:57.022630+0000","last_fullsized":"2024-05-27T07:25:57.022630+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:21:47.565641+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:21:47.565641+0000","last_clean_scrub_stamp":"2024-05-27T07:21:47.565641+0000","objects_scrubbed":0,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T13:47:20.827824+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,4],"acting":[10,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.35","version":"90'6","reported_seq":45,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-27T07:25:57.021854+0000","last_change":"2024-05-27T07:21:48.603424+0000","last_active":"2024-05-27T07:25:57.021854+0000","last_peered":"2024-05-27T07:25:57.021854+0000","last_clean":"2024-05-27T07:25:57.021854+0000","last_became_active":"2024-05-27T07:21:48.602962+0000","last_became_peered":"2024-05-27T07:21:48.602962+0000","last_unstale":"2024-05-27T07:25:57.021854+0000","last_undegraded":"2024-05-27T07:25:57.021854+0000","last_fullsized":"2024-05-27T07:25:57.021854+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:21:47.565641+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:21:47.565641+0000","last_clean_scrub_stamp":"2024-05-27T07:21:47.565641+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T19:19:42.063063+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,8,1],"acting":[7,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.34","version":"90'7","reported_seq":48,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-27T07:25:57.024017+0000","last_change":"2024-05-27T07:21:48.589323+0000","last_active":"2024-05-27T07:25:57.024017+0000","last_peered":"2024-05-27T07:25:57.024017+0000","last_clean":"2024-05-27T07:25:57.024017+0000","last_became_active":"2024-05-27T07:21:48.587435+0000","last_became_peered":"2024-05-27T07:21:48.587435+0000","last_unstale":"2024-05-27T07:25:57.024017+0000","last_undegraded":"2024-05-27T07:25:57.024017+0000","last_fullsized":"2024-05-27T07:25:57.024017+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:21:47.565641+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:21:47.565641+0000","last_clean_scrub_stamp":"2024-05-27T07:21:47.565641+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T13:34:44.340376+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,8],"acting":[4,10,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"90'10","reported_seq":51,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-27T07:25:57.024520+0000","last_change":"2024-05-27T07:21:48.593759+0000","last_active":"2024-05-27T07:25:57.024520+0000","last_peered":"2024-05-27T07:25:57.024520+0000","last_clean":"2024-05-27T07:25:57.024520+0000","last_became_active":"2024-05-27T07:21:48.593099+0000","last_became_peered":"2024-05-27T07:21:48.593099+0000","last_unstale":"2024-05-27T07:25:57.024520+0000","last_undegraded":"2024-05-27T07:25:57.024520+0000","last_fullsized":"2024-05-27T07:25:57.024520+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:21:47.565641+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:21:47.565641+0000","last_clean_scrub_stamp":"2024-05-27T07:21:47.565641+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T13:49:00.390792+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,5],"acting":[6,10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.32","version":"90'7","reported_seq":48,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-27T07:25:57.024016+0000","last_change":"2024-05-27T07:21:48.587486+0000","last_active":"2024-05-27T07:25:57.024016+0000","last_peered":"2024-05-27T07:25:57.024016+0000","last_clean":"2024-05-27T07:25:57.024016+0000","last_became_active":"2024-05-27T07:21:48.587159+0000","last_became_peered":"2024-05-27T07:21:48.587159+0000","last_unstale":"2024-05-27T07:25:57.024016+0000","last_undegraded":"2024-05-27T07:25:57.024016+0000","last_fullsized":"2024-05-27T07:25:57.024016+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:21:47.565641+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:21:47.565641+0000","last_clean_scrub_stamp":"2024-05-27T07:21:47.565641+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T15:34:11.908442+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,10],"acting":[0,8,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"90'7","reported_seq":48,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-27T07:25:57.024296+0000","last_change":"2024-05-27T07:21:48.596817+0000","last_active":"2024-05-27T07:25:57.024296+0000","last_peered":"2024-05-27T07:25:57.024296+0000","last_clean":"2024-05-27T07:25:57.024296+0000","last_became_active":"2024-05-27T07:21:48.596469+0000","last_became_peered":"2024-05-27T07:21:48.596469+0000","last_unstale":"2024-05-27T07:25:57.024296+0000","last_undegraded":"2024-05-27T07:25:57.024296+0000","last_fullsized":"2024-05-27T07:25:57.024296+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:21:47.565641+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:21:47.565641+0000","last_clean_scrub_stamp":"2024-05-27T07:21:47.565641+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T12:38:41.494060+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,2],"acting":[0,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.30","version":"90'9","reported_seq":50,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-27T07:25:56.934824+0000","last_change":"2024-05-27T07:21:48.604249+0000","last_active":"2024-05-27T07:25:56.934824+0000","last_peered":"2024-05-27T07:25:56.934824+0000","last_clean":"2024-05-27T07:25:56.934824+0000","last_became_active":"2024-05-27T07:21:48.603585+0000","last_became_peered":"2024-05-27T07:21:48.603585+0000","last_unstale":"2024-05-27T07:25:56.934824+0000","last_undegraded":"2024-05-27T07:25:56.934824+0000","last_fullsized":"2024-05-27T07:25:56.934824+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:21:47.565641+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:21:47.565641+0000","last_clean_scrub_stamp":"2024-05-27T07:21:47.565641+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T07:22:14.839824+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2f","version":"90'8","reported_seq":49,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-27T07:25:57.023941+0000","last_change":"2024-05-27T07:21:48.596035+0000","last_active":"2024-05-27T07:25:57.023941+0000","last_peered":"2024-05-27T07:25:57.023941+0000","last_clean":"2024-05-27T07:25:57.023941+0000","last_became_active":"2024-05-27T07:21:48.595509+0000","last_became_peered":"2024-05-27T07:21:48.595509+0000","last_unstale":"2024-05-27T07:25:57.023941+0000","last_undegraded":"2024-05-27T07:25:57.023941+0000","last_fullsized":"2024-05-27T07:25:57.023941+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:21:47.565641+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:21:47.565641+0000","last_clean_scrub_stamp":"2024-05-27T07:21:47.565641+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T13:23:03.734982+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.2e","version":"90'10","reported_seq":51,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-27T07:25:56.931790+0000","last_change":"2024-05-27T07:21:48.590002+0000","last_active":"2024-05-27T07:25:56.931790+0000","last_peered":"2024-05-27T07:25:56.931790+0000","last_clean":"2024-05-27T07:25:56.931790+0000","last_became_active":"2024-05-27T07:21:48.589608+0000","last_became_peered":"2024-05-27T07:21:48.589608+0000","last_unstale":"2024-05-27T07:25:56.931790+0000","last_undegraded":"2024-05-27T07:25:56.931790+0000","last_fullsized":"2024-05-27T07:25:56.931790+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:21:47.565641+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:21:47.565641+0000","last_clean_scrub_stamp":"2024-05-27T07:21:47.565641+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T09:33:39.174351+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,1,10],"acting":[9,1,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.2d","version":"90'7","reported_seq":48,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-27T07:25:56.935673+0000","last_change":"2024-05-27T07:21:48.594885+0000","last_active":"2024-05-27T07:25:56.935673+0000","last_peered":"2024-05-27T07:25:56.935673+0000","last_clean":"2024-05-27T07:25:56.935673+0000","last_became_active":"2024-05-27T07:21:48.594521+0000","last_became_peered":"2024-05-27T07:21:48.594521+0000","last_unstale":"2024-05-27T07:25:56.935673+0000","last_undegraded":"2024-05-27T07:25:56.935673+0000","last_fullsized":"2024-05-27T07:25:56.935673+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:21:47.565641+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:21:47.565641+0000","last_clean_scrub_stamp":"2024-05-27T07:21:47.565641+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T08:54:59.311938+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":38,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-27T07:25:56.935012+0000","last_change":"2024-05-27T07:22:30.857546+0000","last_active":"2024-05-27T07:25:56.935012+0000","last_peered":"2024-05-27T07:25:56.935012+0000","last_clean":"2024-05-27T07:25:56.935012+0000","last_became_active":"2024-05-27T07:22:30.856808+0000","last_became_peered":"2024-05-27T07:22:30.856808+0000","last_unstale":"2024-05-27T07:25:56.935012+0000","last_undegraded":"2024-05-27T07:25:56.935012+0000","last_fullsized":"2024-05-27T07:25:56.935012+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:21:58.698952+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:21:58.698952+0000","last_clean_scrub_stamp":"2024-05-27T07:21:58.698952+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T16:09:25.480864+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,8,0],"acting":[2,3,8,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1f","version":"91'14","reported_seq":244,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-27T07:25:57.227211+0000","last_change":"2024-05-27T07:21:45.566489+0000","last_active":"2024-05-27T07:25:57.227211+0000","last_peered":"2024-05-27T07:25:57.227211+0000","last_clean":"2024-05-27T07:25:57.227211+0000","last_became_active":"2024-05-27T07:21:45.566115+0000","last_became_peered":"2024-05-27T07:21:45.566115+0000","last_unstale":"2024-05-27T07:25:57.227211+0000","last_undegraded":"2024-05-27T07:25:57.227211+0000","last_fullsized":"2024-05-27T07:25:57.227211+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:21:44.543638+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:21:44.543638+0000","last_clean_scrub_stamp":"2024-05-27T07:21:44.543638+0000","objects_scrubbed":0,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T07:24:11.735014+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":182,"num_read_kb":182,"num_write":15,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,8],"acting":[9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.1e","version":"90'5","reported_seq":47,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-27T07:25:57.022556+0000","last_change":"2024-05-27T07:21:48.588574+0000","last_active":"2024-05-27T07:25:57.022556+0000","last_peered":"2024-05-27T07:25:57.022556+0000","last_clean":"2024-05-27T07:25:57.022556+0000","last_became_active":"2024-05-27T07:21:48.588212+0000","last_became_peered":"2024-05-27T07:21:48.588212+0000","last_unstale":"2024-05-27T07:25:57.022556+0000","last_undegraded":"2024-05-27T07:25:57.022556+0000","last_fullsized":"2024-05-27T07:25:57.022556+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:21:47.565641+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:21:47.565641+0000","last_clean_scrub_stamp":"2024-05-27T07:21:47.565641+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T12:15:16.017635+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":31,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-27T07:25:57.022914+0000","last_change":"2024-05-27T07:21:59.734617+0000","last_active":"2024-05-27T07:25:57.022914+0000","last_peered":"2024-05-27T07:25:57.022914+0000","last_clean":"2024-05-27T07:25:57.022914+0000","last_became_active":"2024-05-27T07:21:59.734162+0000","last_became_peered":"2024-05-27T07:21:59.734162+0000","last_unstale":"2024-05-27T07:25:57.022914+0000","last_undegraded":"2024-05-27T07:25:57.022914+0000","last_fullsized":"2024-05-27T07:25:57.022914+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:21:58.698952+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:21:58.698952+0000","last_clean_scrub_stamp":"2024-05-27T07:21:58.698952+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T12:40:01.375520+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,9,3],"acting":[4,6,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1e","version":"88'2","reported_seq":49,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-27T07:25:57.022733+0000","last_change":"2024-05-27T07:21:45.790907+0000","last_active":"2024-05-27T07:25:57.022733+0000","last_peered":"2024-05-27T07:25:57.022733+0000","last_clean":"2024-05-27T07:25:57.022733+0000","last_became_active":"2024-05-27T07:21:45.790451+0000","last_became_peered":"2024-05-27T07:21:45.790451+0000","last_unstale":"2024-05-27T07:25:57.022733+0000","last_undegraded":"2024-05-27T07:25:57.022733+0000","last_fullsized":"2024-05-27T07:25:57.022733+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:21:44.543638+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:21:44.543638+0000","last_clean_scrub_stamp":"2024-05-27T07:21:44.543638+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T10:21:56.524930+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"90'6","reported_seq":48,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-27T07:25:57.026415+0000","last_change":"2024-05-27T07:21:48.595975+0000","last_active":"2024-05-27T07:25:57.026415+0000","last_peered":"2024-05-27T07:25:57.026415+0000","last_clean":"2024-05-27T07:25:57.026415+0000","last_became_active":"2024-05-27T07:21:48.595608+0000","last_became_peered":"2024-05-27T07:21:48.595608+0000","last_unstale":"2024-05-27T07:25:57.026415+0000","last_undegraded":"2024-05-27T07:25:57.026415+0000","last_fullsized":"2024-05-27T07:25:57.026415+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:21:47.565641+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:21:47.565641+0000","last_clean_scrub_stamp":"2024-05-27T07:21:47.565641+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T07:24:44.797973+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,9],"acting":[0,1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":31,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-27T07:25:57.023806+0000","last_change":"2024-05-27T07:21:59.732788+0000","last_active":"2024-05-27T07:25:57.023806+0000","last_peered":"2024-05-27T07:25:57.023806+0000","last_clean":"2024-05-27T07:25:57.023806+0000","last_became_active":"2024-05-27T07:21:59.732310+0000","last_became_peered":"2024-05-27T07:21:59.732310+0000","last_unstale":"2024-05-27T07:25:57.023806+0000","last_undegraded":"2024-05-27T07:25:57.023806+0000","last_fullsized":"2024-05-27T07:25:57.023806+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:21:58.698952+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:21:58.698952+0000","last_clean_scrub_stamp":"2024-05-27T07:21:58.698952+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T13:54:51.490154+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,9,5],"acting":[4,3,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1d","version":"91'13","reported_seq":61,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-27T07:25:57.023929+0000","last_change":"2024-05-27T07:21:45.569746+0000","last_active":"2024-05-27T07:25:57.023929+0000","last_peered":"2024-05-27T07:25:57.023929+0000","last_clean":"2024-05-27T07:25:57.023929+0000","last_became_active":"2024-05-27T07:21:45.569366+0000","last_became_peered":"2024-05-27T07:21:45.569366+0000","last_unstale":"2024-05-27T07:25:57.023929+0000","last_undegraded":"2024-05-27T07:25:57.023929+0000","last_fullsized":"2024-05-27T07:25:57.023929+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:21:44.543638+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:21:44.543638+0000","last_clean_scrub_stamp":"2024-05-27T07:21:44.543638+0000","objects_scrubbed":0,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T15:58:32.977754+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":18,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":7,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1c","version":"90'9","reported_seq":50,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-27T07:25:56.935051+0000","last_change":"2024-05-27T07:21:48.606138+0000","last_active":"2024-05-27T07:25:56.935051+0000","last_peered":"2024-05-27T07:25:56.935051+0000","last_clean":"2024-05-27T07:25:56.935051+0000","last_became_active":"2024-05-27T07:21:48.605711+0000","last_became_peered":"2024-05-27T07:21:48.605711+0000","last_unstale":"2024-05-27T07:25:56.935051+0000","last_undegraded":"2024-05-27T07:25:56.935051+0000","last_fullsized":"2024-05-27T07:25:56.935051+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:21:47.565641+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:21:47.565641+0000","last_clean_scrub_stamp":"2024-05-27T07:21:47.565641+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T10:46:39.052387+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,6],"acting":[1,10,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":31,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-27T07:25:57.023535+0000","last_change":"2024-05-27T07:21:59.735475+0000","last_active":"2024-05-27T07:25:57.023535+0000","last_peered":"2024-05-27T07:25:57.023535+0000","last_clean":"2024-05-27T07:25:57.023535+0000","last_became_active":"2024-05-27T07:21:59.734986+0000","last_became_peered":"2024-05-27T07:21:59.734986+0000","last_unstale":"2024-05-27T07:25:57.023535+0000","last_undegraded":"2024-05-27T07:25:57.023535+0000","last_fullsized":"2024-05-27T07:25:57.023535+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:21:58.698952+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:21:58.698952+0000","last_clean_scrub_stamp":"2024-05-27T07:21:58.698952+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T14:48:28.198718+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5,3],"acting":[4,6,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1c","version":"88'5","reported_seq":53,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-27T07:25:57.023547+0000","last_change":"2024-05-27T07:21:45.572882+0000","last_active":"2024-05-27T07:25:57.023547+0000","last_peered":"2024-05-27T07:25:57.023547+0000","last_clean":"2024-05-27T07:25:57.023547+0000","last_became_active":"2024-05-27T07:21:45.572460+0000","last_became_peered":"2024-05-27T07:21:45.572460+0000","last_unstale":"2024-05-27T07:25:57.023547+0000","last_undegraded":"2024-05-27T07:25:57.023547+0000","last_fullsized":"2024-05-27T07:25:57.023547+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:21:44.543638+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:21:44.543638+0000","last_clean_scrub_stamp":"2024-05-27T07:21:44.543638+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T09:40:11.311923+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":7,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,1,5],"acting":[8,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1d","version":"90'15","reported_seq":56,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-27T07:25:57.023149+0000","last_change":"2024-05-27T07:21:48.596217+0000","last_active":"2024-05-27T07:25:57.023149+0000","last_peered":"2024-05-27T07:25:57.023149+0000","last_clean":"2024-05-27T07:25:57.023149+0000","last_became_active":"2024-05-27T07:21:48.595832+0000","last_became_peered":"2024-05-27T07:21:48.595832+0000","last_unstale":"2024-05-27T07:25:57.023149+0000","last_undegraded":"2024-05-27T07:25:57.023149+0000","last_fullsized":"2024-05-27T07:25:57.023149+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:21:47.565641+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:21:47.565641+0000","last_clean_scrub_stamp":"2024-05-27T07:21:47.565641+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T10:08:05.304404+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":31,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-27T07:25:56.935413+0000","last_change":"2024-05-27T07:21:59.729617+0000","last_active":"2024-05-27T07:25:56.935413+0000","last_peered":"2024-05-27T07:25:56.935413+0000","last_clean":"2024-05-27T07:25:56.935413+0000","last_became_active":"2024-05-27T07:21:59.729003+0000","last_became_peered":"2024-05-27T07:21:59.729003+0000","last_unstale":"2024-05-27T07:25:56.935413+0000","last_undegraded":"2024-05-27T07:25:56.935413+0000","last_fullsized":"2024-05-27T07:25:56.935413+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:21:58.698952+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:21:58.698952+0000","last_clean_scrub_stamp":"2024-05-27T07:21:58.698952+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T14:55:04.844948+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7,11],"acting":[2,5,7,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1b","version":"91'8","reported_seq":239,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-27T07:25:57.161701+0000","last_change":"2024-05-27T07:21:45.566355+0000","last_active":"2024-05-27T07:25:57.161701+0000","last_peered":"2024-05-27T07:25:57.161701+0000","last_clean":"2024-05-27T07:25:57.161701+0000","last_became_active":"2024-05-27T07:21:45.564776+0000","last_became_peered":"2024-05-27T07:21:45.564776+0000","last_unstale":"2024-05-27T07:25:57.161701+0000","last_undegraded":"2024-05-27T07:25:57.161701+0000","last_fullsized":"2024-05-27T07:25:57.161701+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:21:44.543638+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:21:44.543638+0000","last_clean_scrub_stamp":"2024-05-27T07:21:44.543638+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T12:10:41.632630+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1545,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":185,"num_read_kb":4,"num_write":5,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1a","version":"90'7","reported_seq":48,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-27T07:25:57.023028+0000","last_change":"2024-05-27T07:21:48.588837+0000","last_active":"2024-05-27T07:25:57.023028+0000","last_peered":"2024-05-27T07:25:57.023028+0000","last_clean":"2024-05-27T07:25:57.023028+0000","last_became_active":"2024-05-27T07:21:48.587149+0000","last_became_peered":"2024-05-27T07:21:48.587149+0000","last_unstale":"2024-05-27T07:25:57.023028+0000","last_undegraded":"2024-05-27T07:25:57.023028+0000","last_fullsized":"2024-05-27T07:25:57.023028+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:21:47.565641+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:21:47.565641+0000","last_clean_scrub_stamp":"2024-05-27T07:21:47.565641+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T09:20:05.124189+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,5],"acting":[8,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":31,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-27T07:25:56.935491+0000","last_change":"2024-05-27T07:21:59.730003+0000","last_active":"2024-05-27T07:25:56.935491+0000","last_peered":"2024-05-27T07:25:56.935491+0000","last_clean":"2024-05-27T07:25:56.935491+0000","last_became_active":"2024-05-27T07:21:59.729195+0000","last_became_peered":"2024-05-27T07:21:59.729195+0000","last_unstale":"2024-05-27T07:25:56.935491+0000","last_undegraded":"2024-05-27T07:25:56.935491+0000","last_fullsized":"2024-05-27T07:25:56.935491+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:21:58.698952+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:21:58.698952+0000","last_clean_scrub_stamp":"2024-05-27T07:21:58.698952+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T13:13:41.612300+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,9,5],"acting":[2,6,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1a","version":"88'7","reported_seq":54,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-27T07:25:57.022914+0000","last_change":"2024-05-27T07:21:45.567279+0000","last_active":"2024-05-27T07:25:57.022914+0000","last_peered":"2024-05-27T07:25:57.022914+0000","last_clean":"2024-05-27T07:25:57.022914+0000","last_became_active":"2024-05-27T07:21:45.566920+0000","last_became_peered":"2024-05-27T07:21:45.566920+0000","last_unstale":"2024-05-27T07:25:57.022914+0000","last_undegraded":"2024-05-27T07:25:57.022914+0000","last_fullsized":"2024-05-27T07:25:57.022914+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:21:44.543638+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:21:44.543638+0000","last_clean_scrub_stamp":"2024-05-27T07:21:44.543638+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T18:12:23.015535+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,8,6],"acting":[10,8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.1b","version":"90'8","reported_seq":55,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-27T07:25:57.023087+0000","last_change":"2024-05-27T07:22:30.857080+0000","last_active":"2024-05-27T07:25:57.023087+0000","last_peered":"2024-05-27T07:25:57.023087+0000","last_clean":"2024-05-27T07:25:57.023087+0000","last_became_active":"2024-05-27T07:22:30.856479+0000","last_became_peered":"2024-05-27T07:22:30.856479+0000","last_unstale":"2024-05-27T07:25:57.023087+0000","last_undegraded":"2024-05-27T07:25:57.023087+0000","last_fullsized":"2024-05-27T07:25:57.023087+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:21:47.565641+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:21:47.565641+0000","last_clean_scrub_stamp":"2024-05-27T07:21:47.565641+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T13:29:16.054448+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,8,3],"acting":[5,8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":38,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-27T07:25:57.023817+0000","last_change":"2024-05-27T07:22:30.805292+0000","last_active":"2024-05-27T07:25:57.023817+0000","last_peered":"2024-05-27T07:25:57.023817+0000","last_clean":"2024-05-27T07:25:57.023817+0000","last_became_active":"2024-05-27T07:22:30.804779+0000","last_became_peered":"2024-05-27T07:22:30.804779+0000","last_unstale":"2024-05-27T07:25:57.023817+0000","last_undegraded":"2024-05-27T07:25:57.023817+0000","last_fullsized":"2024-05-27T07:25:57.023817+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:21:58.698952+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:21:58.698952+0000","last_clean_scrub_stamp":"2024-05-27T07:21:58.698952+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T11:12:49.095931+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,8,0],"acting":[11,4,8,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.19","version":"88'3","reported_seq":49,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-27T07:25:57.024211+0000","last_change":"2024-05-27T07:21:45.561255+0000","last_active":"2024-05-27T07:25:57.024211+0000","last_peered":"2024-05-27T07:25:57.024211+0000","last_clean":"2024-05-27T07:25:57.024211+0000","last_became_active":"2024-05-27T07:21:45.560896+0000","last_became_peered":"2024-05-27T07:21:45.560896+0000","last_unstale":"2024-05-27T07:25:57.024211+0000","last_undegraded":"2024-05-27T07:25:57.024211+0000","last_fullsized":"2024-05-27T07:25:57.024211+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:21:44.543638+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:21:44.543638+0000","last_clean_scrub_stamp":"2024-05-27T07:21:44.543638+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T17:05:24.074628+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,9],"acting":[6,4,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"90'8","reported_seq":49,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-27T07:25:57.021953+0000","last_change":"2024-05-27T07:21:48.586045+0000","last_active":"2024-05-27T07:25:57.021953+0000","last_peered":"2024-05-27T07:25:57.021953+0000","last_clean":"2024-05-27T07:25:57.021953+0000","last_became_active":"2024-05-27T07:21:48.585681+0000","last_became_peered":"2024-05-27T07:21:48.585681+0000","last_unstale":"2024-05-27T07:25:57.021953+0000","last_undegraded":"2024-05-27T07:25:57.021953+0000","last_fullsized":"2024-05-27T07:25:57.021953+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:21:47.565641+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:21:47.565641+0000","last_clean_scrub_stamp":"2024-05-27T07:21:47.565641+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T09:56:36.408623+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,6],"acting":[3,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":31,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-27T07:25:57.025932+0000","last_change":"2024-05-27T07:21:59.739544+0000","last_active":"2024-05-27T07:25:57.025932+0000","last_peered":"2024-05-27T07:25:57.025932+0000","last_clean":"2024-05-27T07:25:57.025932+0000","last_became_active":"2024-05-27T07:21:59.739132+0000","last_became_peered":"2024-05-27T07:21:59.739132+0000","last_unstale":"2024-05-27T07:25:57.025932+0000","last_undegraded":"2024-05-27T07:25:57.025932+0000","last_fullsized":"2024-05-27T07:25:57.025932+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:21:58.698952+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:21:58.698952+0000","last_clean_scrub_stamp":"2024-05-27T07:21:58.698952+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T10:06:05.126579+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,6,1],"acting":[0,7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.18","version":"88'3","reported_seq":50,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-27T07:25:57.022592+0000","last_change":"2024-05-27T07:21:45.568171+0000","last_active":"2024-05-27T07:25:57.022592+0000","last_peered":"2024-05-27T07:25:57.022592+0000","last_clean":"2024-05-27T07:25:57.022592+0000","last_became_active":"2024-05-27T07:21:45.567627+0000","last_became_peered":"2024-05-27T07:21:45.567627+0000","last_unstale":"2024-05-27T07:25:57.022592+0000","last_undegraded":"2024-05-27T07:25:57.022592+0000","last_fullsized":"2024-05-27T07:25:57.022592+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:21:44.543638+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:21:44.543638+0000","last_clean_scrub_stamp":"2024-05-27T07:21:44.543638+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T16:43:56.407079+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"90'11","reported_seq":52,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-27T07:25:56.930989+0000","last_change":"2024-05-27T07:21:48.587452+0000","last_active":"2024-05-27T07:25:56.930989+0000","last_peered":"2024-05-27T07:25:56.930989+0000","last_clean":"2024-05-27T07:25:56.930989+0000","last_became_active":"2024-05-27T07:21:48.587027+0000","last_became_peered":"2024-05-27T07:21:48.587027+0000","last_unstale":"2024-05-27T07:25:56.930989+0000","last_undegraded":"2024-05-27T07:25:56.930989+0000","last_fullsized":"2024-05-27T07:25:56.930989+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:21:47.565641+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:21:47.565641+0000","last_clean_scrub_stamp":"2024-05-27T07:21:47.565641+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T13:51:44.890061+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,10],"acting":[9,3,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":31,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-27T07:25:57.021908+0000","last_change":"2024-05-27T07:21:59.738292+0000","last_active":"2024-05-27T07:25:57.021908+0000","last_peered":"2024-05-27T07:25:57.021908+0000","last_clean":"2024-05-27T07:25:57.021908+0000","last_became_active":"2024-05-27T07:21:59.737941+0000","last_became_peered":"2024-05-27T07:21:59.737941+0000","last_unstale":"2024-05-27T07:25:57.021908+0000","last_undegraded":"2024-05-27T07:25:57.021908+0000","last_fullsized":"2024-05-27T07:25:57.021908+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:21:58.698952+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:21:58.698952+0000","last_clean_scrub_stamp":"2024-05-27T07:21:58.698952+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T13:54:24.192917+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1,5],"acting":[3,7,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.17","version":"91'7","reported_seq":416,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-27T07:25:57.227899+0000","last_change":"2024-05-27T07:21:45.568338+0000","last_active":"2024-05-27T07:25:57.227899+0000","last_peered":"2024-05-27T07:25:57.227899+0000","last_clean":"2024-05-27T07:25:57.227899+0000","last_became_active":"2024-05-27T07:21:45.567856+0000","last_became_peered":"2024-05-27T07:21:45.567856+0000","last_unstale":"2024-05-27T07:25:57.227899+0000","last_undegraded":"2024-05-27T07:25:57.227899+0000","last_fullsized":"2024-05-27T07:25:57.227899+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:21:44.543638+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:21:44.543638+0000","last_clean_scrub_stamp":"2024-05-27T07:21:44.543638+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T15:02:04.188711+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":180,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":362,"num_read_kb":362,"num_write":8,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,5,9],"acting":[10,5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.16","version":"90'10","reported_seq":51,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-27T07:25:56.935595+0000","last_change":"2024-05-27T07:21:48.606062+0000","last_active":"2024-05-27T07:25:56.935595+0000","last_peered":"2024-05-27T07:25:56.935595+0000","last_clean":"2024-05-27T07:25:56.935595+0000","last_became_active":"2024-05-27T07:21:48.605423+0000","last_became_peered":"2024-05-27T07:21:48.605423+0000","last_unstale":"2024-05-27T07:25:56.935595+0000","last_undegraded":"2024-05-27T07:25:56.935595+0000","last_fullsized":"2024-05-27T07:25:56.935595+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:21:47.565641+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:21:47.565641+0000","last_clean_scrub_stamp":"2024-05-27T07:21:47.565641+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T14:39:06.921519+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,9],"acting":[1,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":31,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-27T07:25:56.930933+0000","last_change":"2024-05-27T07:21:59.735150+0000","last_active":"2024-05-27T07:25:56.930933+0000","last_peered":"2024-05-27T07:25:56.930933+0000","last_clean":"2024-05-27T07:25:56.930933+0000","last_became_active":"2024-05-27T07:21:59.734688+0000","last_became_peered":"2024-05-27T07:21:59.734688+0000","last_unstale":"2024-05-27T07:25:56.930933+0000","last_undegraded":"2024-05-27T07:25:56.930933+0000","last_fullsized":"2024-05-27T07:25:56.930933+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:21:58.698952+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:21:58.698952+0000","last_clean_scrub_stamp":"2024-05-27T07:21:58.698952+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T15:38:02.095947+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,0,2,4],"acting":[9,0,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.16","version":"88'4","reported_seq":48,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-27T07:25:57.022536+0000","last_change":"2024-05-27T07:21:45.559550+0000","last_active":"2024-05-27T07:25:57.022536+0000","last_peered":"2024-05-27T07:25:57.022536+0000","last_clean":"2024-05-27T07:25:57.022536+0000","last_became_active":"2024-05-27T07:21:45.559181+0000","last_became_peered":"2024-05-27T07:21:45.559181+0000","last_unstale":"2024-05-27T07:25:57.022536+0000","last_undegraded":"2024-05-27T07:25:57.022536+0000","last_fullsized":"2024-05-27T07:25:57.022536+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:21:44.543638+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:21:44.543638+0000","last_clean_scrub_stamp":"2024-05-27T07:21:44.543638+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T10:33:45.288437+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.17","version":"90'9","reported_seq":50,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-27T07:25:57.026187+0000","last_change":"2024-05-27T07:21:48.593631+0000","last_active":"2024-05-27T07:25:57.026187+0000","last_peered":"2024-05-27T07:25:57.026187+0000","last_clean":"2024-05-27T07:25:57.026187+0000","last_became_active":"2024-05-27T07:21:48.592330+0000","last_became_peered":"2024-05-27T07:21:48.592330+0000","last_unstale":"2024-05-27T07:25:57.026187+0000","last_undegraded":"2024-05-27T07:25:57.026187+0000","last_fullsized":"2024-05-27T07:25:57.026187+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:21:47.565641+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:21:47.565641+0000","last_clean_scrub_stamp":"2024-05-27T07:21:47.565641+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T19:19:51.593307+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,5],"acting":[0,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":31,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-27T07:25:57.023937+0000","last_change":"2024-05-27T07:21:59.725158+0000","last_active":"2024-05-27T07:25:57.023937+0000","last_peered":"2024-05-27T07:25:57.023937+0000","last_clean":"2024-05-27T07:25:57.023937+0000","last_became_active":"2024-05-27T07:21:59.724651+0000","last_became_peered":"2024-05-27T07:21:59.724651+0000","last_unstale":"2024-05-27T07:25:57.023937+0000","last_undegraded":"2024-05-27T07:25:57.023937+0000","last_fullsized":"2024-05-27T07:25:57.023937+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:21:58.698952+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:21:58.698952+0000","last_clean_scrub_stamp":"2024-05-27T07:21:58.698952+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T12:10:45.747484+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2,9],"acting":[4,1,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.15","version":"88'5","reported_seq":52,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-27T07:25:57.022957+0000","last_change":"2024-05-27T07:21:45.791831+0000","last_active":"2024-05-27T07:25:57.022957+0000","last_peered":"2024-05-27T07:25:57.022957+0000","last_clean":"2024-05-27T07:25:57.022957+0000","last_became_active":"2024-05-27T07:21:45.791271+0000","last_became_peered":"2024-05-27T07:21:45.791271+0000","last_unstale":"2024-05-27T07:25:57.022957+0000","last_undegraded":"2024-05-27T07:25:57.022957+0000","last_fullsized":"2024-05-27T07:25:57.022957+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:21:44.543638+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:21:44.543638+0000","last_clean_scrub_stamp":"2024-05-27T07:21:44.543638+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T16:20:37.492337+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":7,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,10],"acting":[5,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.14","version":"90'11","reported_seq":52,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-27T07:25:56.935320+0000","last_change":"2024-05-27T07:21:48.605605+0000","last_active":"2024-05-27T07:25:56.935320+0000","last_peered":"2024-05-27T07:25:56.935320+0000","last_clean":"2024-05-27T07:25:56.935320+0000","last_became_active":"2024-05-27T07:21:48.604655+0000","last_became_peered":"2024-05-27T07:21:48.604655+0000","last_unstale":"2024-05-27T07:25:56.935320+0000","last_undegraded":"2024-05-27T07:25:56.935320+0000","last_fullsized":"2024-05-27T07:25:56.935320+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:21:47.565641+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:21:47.565641+0000","last_clean_scrub_stamp":"2024-05-27T07:21:47.565641+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T14:33:53.661983+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,8,4],"acting":[1,8,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":38,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-27T07:25:56.931232+0000","last_change":"2024-05-27T07:22:31.418393+0000","last_active":"2024-05-27T07:25:56.931232+0000","last_peered":"2024-05-27T07:25:56.931232+0000","last_clean":"2024-05-27T07:25:56.931232+0000","last_became_active":"2024-05-27T07:22:31.418006+0000","last_became_peered":"2024-05-27T07:22:31.418006+0000","last_unstale":"2024-05-27T07:25:56.931232+0000","last_undegraded":"2024-05-27T07:25:56.931232+0000","last_fullsized":"2024-05-27T07:25:56.931232+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:21:58.698952+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:21:58.698952+0000","last_clean_scrub_stamp":"2024-05-27T07:21:58.698952+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T14:30:55.130461+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,11,10],"acting":[9,8,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.14","version":"91'44","reported_seq":466,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-27T07:25:57.181678+0000","last_change":"2024-05-27T07:21:45.564829+0000","last_active":"2024-05-27T07:25:57.181678+0000","last_peered":"2024-05-27T07:25:57.181678+0000","last_clean":"2024-05-27T07:25:57.181678+0000","last_became_active":"2024-05-27T07:21:45.564457+0000","last_became_peered":"2024-05-27T07:21:45.564457+0000","last_unstale":"2024-05-27T07:25:57.181678+0000","last_undegraded":"2024-05-27T07:25:57.181678+0000","last_fullsized":"2024-05-27T07:25:57.181678+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:21:44.543638+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:21:44.543638+0000","last_clean_scrub_stamp":"2024-05-27T07:21:44.543638+0000","objects_scrubbed":0,"log_size":44,"log_dups_size":0,"ondisk_log_size":44,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T19:12:44.546989+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70939,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":375,"num_read_kb":257,"num_write":46,"num_write_kb":92,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,1,2],"acting":[10,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.15","version":"90'6","reported_seq":47,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-27T07:25:57.023640+0000","last_change":"2024-05-27T07:21:48.591672+0000","last_active":"2024-05-27T07:25:57.023640+0000","last_peered":"2024-05-27T07:25:57.023640+0000","last_clean":"2024-05-27T07:25:57.023640+0000","last_became_active":"2024-05-27T07:21:48.590861+0000","last_became_peered":"2024-05-27T07:21:48.590861+0000","last_unstale":"2024-05-27T07:25:57.023640+0000","last_undegraded":"2024-05-27T07:25:57.023640+0000","last_fullsized":"2024-05-27T07:25:57.023640+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:21:47.565641+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:21:47.565641+0000","last_clean_scrub_stamp":"2024-05-27T07:21:47.565641+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T12:03:40.627764+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,7],"acting":[8,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":31,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-27T07:25:57.023508+0000","last_change":"2024-05-27T07:21:59.736446+0000","last_active":"2024-05-27T07:25:57.023508+0000","last_peered":"2024-05-27T07:25:57.023508+0000","last_clean":"2024-05-27T07:25:57.023508+0000","last_became_active":"2024-05-27T07:21:59.735854+0000","last_became_peered":"2024-05-27T07:21:59.735854+0000","last_unstale":"2024-05-27T07:25:57.023508+0000","last_undegraded":"2024-05-27T07:25:57.023508+0000","last_fullsized":"2024-05-27T07:25:57.023508+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:21:58.698952+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:21:58.698952+0000","last_clean_scrub_stamp":"2024-05-27T07:21:58.698952+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T08:02:14.640325+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,8,3,6],"acting":[5,8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.13","version":"88'9","reported_seq":57,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-27T07:25:56.935277+0000","last_change":"2024-05-27T07:21:45.562930+0000","last_active":"2024-05-27T07:25:56.935277+0000","last_peered":"2024-05-27T07:25:56.935277+0000","last_clean":"2024-05-27T07:25:56.935277+0000","last_became_active":"2024-05-27T07:21:45.562515+0000","last_became_peered":"2024-05-27T07:21:45.562515+0000","last_unstale":"2024-05-27T07:25:56.935277+0000","last_undegraded":"2024-05-27T07:25:56.935277+0000","last_fullsized":"2024-05-27T07:25:56.935277+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:21:44.543638+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:21:44.543638+0000","last_clean_scrub_stamp":"2024-05-27T07:21:44.543638+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T17:54:24.006219+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":68,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":13,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,10,3],"acting":[2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.12","version":"90'9","reported_seq":50,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-27T07:25:57.024207+0000","last_change":"2024-05-27T07:21:48.591090+0000","last_active":"2024-05-27T07:25:57.024207+0000","last_peered":"2024-05-27T07:25:57.024207+0000","last_clean":"2024-05-27T07:25:57.024207+0000","last_became_active":"2024-05-27T07:21:48.590726+0000","last_became_peered":"2024-05-27T07:21:48.590726+0000","last_unstale":"2024-05-27T07:25:57.024207+0000","last_undegraded":"2024-05-27T07:25:57.024207+0000","last_fullsized":"2024-05-27T07:25:57.024207+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:21:47.565641+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:21:47.565641+0000","last_clean_scrub_stamp":"2024-05-27T07:21:47.565641+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T18:40:03.880407+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,3],"acting":[6,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.14","version":"0'0","reported_seq":31,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-27T07:25:57.023543+0000","last_change":"2024-05-27T07:21:59.738375+0000","last_active":"2024-05-27T07:25:57.023543+0000","last_peered":"2024-05-27T07:25:57.023543+0000","last_clean":"2024-05-27T07:25:57.023543+0000","last_became_active":"2024-05-27T07:21:59.737957+0000","last_became_peered":"2024-05-27T07:21:59.737957+0000","last_unstale":"2024-05-27T07:25:57.023543+0000","last_undegraded":"2024-05-27T07:25:57.023543+0000","last_fullsized":"2024-05-27T07:25:57.023543+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:21:58.698952+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:21:58.698952+0000","last_clean_scrub_stamp":"2024-05-27T07:21:58.698952+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T16:53:51.397143+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,9,7,6],"acting":[11,9,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.12","version":"88'5","reported_seq":52,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-27T07:25:57.023142+0000","last_change":"2024-05-27T07:21:45.566986+0000","last_active":"2024-05-27T07:25:57.023142+0000","last_peered":"2024-05-27T07:25:57.023142+0000","last_clean":"2024-05-27T07:25:57.023142+0000","last_became_active":"2024-05-27T07:21:45.566579+0000","last_became_peered":"2024-05-27T07:21:45.566579+0000","last_unstale":"2024-05-27T07:25:57.023142+0000","last_undegraded":"2024-05-27T07:25:57.023142+0000","last_fullsized":"2024-05-27T07:25:57.023142+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:21:44.543638+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:21:44.543638+0000","last_clean_scrub_stamp":"2024-05-27T07:21:44.543638+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T17:08:14.034740+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":5,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"90'15","reported_seq":54,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-27T07:25:57.021914+0000","last_change":"2024-05-27T07:21:48.596575+0000","last_active":"2024-05-27T07:25:57.021914+0000","last_peered":"2024-05-27T07:25:57.021914+0000","last_clean":"2024-05-27T07:25:57.021914+0000","last_became_active":"2024-05-27T07:21:48.596215+0000","last_became_peered":"2024-05-27T07:21:48.596215+0000","last_unstale":"2024-05-27T07:25:57.021914+0000","last_undegraded":"2024-05-27T07:25:57.021914+0000","last_fullsized":"2024-05-27T07:25:57.021914+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:21:47.565641+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:21:47.565641+0000","last_clean_scrub_stamp":"2024-05-27T07:21:47.565641+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T14:57:05.795269+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":31,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-27T07:25:56.931967+0000","last_change":"2024-05-27T07:21:59.737690+0000","last_active":"2024-05-27T07:25:56.931967+0000","last_peered":"2024-05-27T07:25:56.931967+0000","last_clean":"2024-05-27T07:25:56.931967+0000","last_became_active":"2024-05-27T07:21:59.735646+0000","last_became_peered":"2024-05-27T07:21:59.735646+0000","last_unstale":"2024-05-27T07:25:56.931967+0000","last_undegraded":"2024-05-27T07:25:56.931967+0000","last_fullsized":"2024-05-27T07:25:56.931967+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:21:58.698952+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:21:58.698952+0000","last_clean_scrub_stamp":"2024-05-27T07:21:58.698952+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T13:54:41.227626+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,0,3],"acting":[9,7,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.11","version":"91'8","reported_seq":240,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-27T07:25:57.161569+0000","last_change":"2024-05-27T07:21:45.564836+0000","last_active":"2024-05-27T07:25:57.161569+0000","last_peered":"2024-05-27T07:25:57.161569+0000","last_clean":"2024-05-27T07:25:57.161569+0000","last_became_active":"2024-05-27T07:21:45.564481+0000","last_became_peered":"2024-05-27T07:21:45.564481+0000","last_unstale":"2024-05-27T07:25:57.161569+0000","last_undegraded":"2024-05-27T07:25:57.161569+0000","last_fullsized":"2024-05-27T07:25:57.161569+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:21:44.543638+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:21:44.543638+0000","last_clean_scrub_stamp":"2024-05-27T07:21:44.543638+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T15:13:12.012769+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1511,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":186,"num_read_kb":5,"num_write":8,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,10],"acting":[4,7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.10","version":"90'6","reported_seq":45,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-27T07:25:57.022004+0000","last_change":"2024-05-27T07:21:48.601331+0000","last_active":"2024-05-27T07:25:57.022004+0000","last_peered":"2024-05-27T07:25:57.022004+0000","last_clean":"2024-05-27T07:25:57.022004+0000","last_became_active":"2024-05-27T07:21:48.600917+0000","last_became_peered":"2024-05-27T07:21:48.600917+0000","last_unstale":"2024-05-27T07:25:57.022004+0000","last_undegraded":"2024-05-27T07:25:57.022004+0000","last_fullsized":"2024-05-27T07:25:57.022004+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:21:47.565641+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:21:47.565641+0000","last_clean_scrub_stamp":"2024-05-27T07:21:47.565641+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T17:26:42.129989+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":31,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-27T07:25:57.024036+0000","last_change":"2024-05-27T07:21:59.737662+0000","last_active":"2024-05-27T07:25:57.024036+0000","last_peered":"2024-05-27T07:25:57.024036+0000","last_clean":"2024-05-27T07:25:57.024036+0000","last_became_active":"2024-05-27T07:21:59.737229+0000","last_became_peered":"2024-05-27T07:21:59.737229+0000","last_unstale":"2024-05-27T07:25:57.024036+0000","last_undegraded":"2024-05-27T07:25:57.024036+0000","last_fullsized":"2024-05-27T07:25:57.024036+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:21:58.698952+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:21:58.698952+0000","last_clean_scrub_stamp":"2024-05-27T07:21:58.698952+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T15:32:31.147571+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,9,11],"acting":[6,3,9,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.10","version":"88'17","reported_seq":81,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-27T07:25:57.022607+0000","last_change":"2024-05-27T07:22:30.814815+0000","last_active":"2024-05-27T07:25:57.022607+0000","last_peered":"2024-05-27T07:25:57.022607+0000","last_clean":"2024-05-27T07:25:57.022607+0000","last_became_active":"2024-05-27T07:22:30.802828+0000","last_became_peered":"2024-05-27T07:22:30.802828+0000","last_unstale":"2024-05-27T07:25:57.022607+0000","last_undegraded":"2024-05-27T07:25:57.022607+0000","last_fullsized":"2024-05-27T07:25:57.022607+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:21:44.543638+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:21:44.543638+0000","last_clean_scrub_stamp":"2024-05-27T07:21:44.543638+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T11:38:07.363507+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,4],"acting":[8,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.11","version":"90'6","reported_seq":47,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-27T07:25:57.022823+0000","last_change":"2024-05-27T07:21:48.596204+0000","last_active":"2024-05-27T07:25:57.022823+0000","last_peered":"2024-05-27T07:25:57.022823+0000","last_clean":"2024-05-27T07:25:57.022823+0000","last_became_active":"2024-05-27T07:21:48.595827+0000","last_became_peered":"2024-05-27T07:21:48.595827+0000","last_unstale":"2024-05-27T07:25:57.022823+0000","last_undegraded":"2024-05-27T07:25:57.022823+0000","last_fullsized":"2024-05-27T07:25:57.022823+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:21:47.565641+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:21:47.565641+0000","last_clean_scrub_stamp":"2024-05-27T07:21:47.565641+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T15:51:53.162554+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":31,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-27T07:25:57.022694+0000","last_change":"2024-05-27T07:21:59.735837+0000","last_active":"2024-05-27T07:25:57.022694+0000","last_peered":"2024-05-27T07:25:57.022694+0000","last_clean":"2024-05-27T07:25:57.022694+0000","last_became_active":"2024-05-27T07:21:59.735063+0000","last_became_peered":"2024-05-27T07:21:59.735063+0000","last_unstale":"2024-05-27T07:25:57.022694+0000","last_undegraded":"2024-05-27T07:25:57.022694+0000","last_fullsized":"2024-05-27T07:25:57.022694+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:21:58.698952+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:21:58.698952+0000","last_clean_scrub_stamp":"2024-05-27T07:21:58.698952+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T13:04:21.751043+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,3,1],"acting":[5,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.f","version":"88'2","reported_seq":24,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-27T07:25:57.023668+0000","last_change":"2024-05-27T07:22:30.809491+0000","last_active":"2024-05-27T07:25:57.023668+0000","last_peered":"2024-05-27T07:25:57.023668+0000","last_clean":"2024-05-27T07:25:57.023668+0000","last_became_active":"2024-05-27T07:22:30.802187+0000","last_became_peered":"2024-05-27T07:22:30.802187+0000","last_unstale":"2024-05-27T07:25:57.023668+0000","last_undegraded":"2024-05-27T07:25:57.023668+0000","last_fullsized":"2024-05-27T07:25:57.023668+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:21:44.543638+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:21:44.543638+0000","last_clean_scrub_stamp":"2024-05-27T07:21:44.543638+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T18:11:32.821962+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":22,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,4],"acting":[11,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.e","version":"90'5","reported_seq":46,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-27T07:25:57.024123+0000","last_change":"2024-05-27T07:21:48.597362+0000","last_active":"2024-05-27T07:25:57.024123+0000","last_peered":"2024-05-27T07:25:57.024123+0000","last_clean":"2024-05-27T07:25:57.024123+0000","last_became_active":"2024-05-27T07:21:48.596920+0000","last_became_peered":"2024-05-27T07:21:48.596920+0000","last_unstale":"2024-05-27T07:25:57.024123+0000","last_undegraded":"2024-05-27T07:25:57.024123+0000","last_fullsized":"2024-05-27T07:25:57.024123+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:21:47.565641+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:21:47.565641+0000","last_clean_scrub_stamp":"2024-05-27T07:21:47.565641+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T16:50:21.021320+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,4],"acting":[11,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":31,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-27T07:25:57.022899+0000","last_change":"2024-05-27T07:21:59.732303+0000","last_active":"2024-05-27T07:25:57.022899+0000","last_peered":"2024-05-27T07:25:57.022899+0000","last_clean":"2024-05-27T07:25:57.022899+0000","last_became_active":"2024-05-27T07:21:59.731851+0000","last_became_peered":"2024-05-27T07:21:59.731851+0000","last_unstale":"2024-05-27T07:25:57.022899+0000","last_undegraded":"2024-05-27T07:25:57.022899+0000","last_fullsized":"2024-05-27T07:25:57.022899+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:21:58.698952+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:21:58.698952+0000","last_clean_scrub_stamp":"2024-05-27T07:21:58.698952+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T08:08:38.433818+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,10,11,9],"acting":[8,10,11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.4","version":"88'1","reported_seq":49,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-27T07:25:56.931533+0000","last_change":"2024-05-27T07:21:45.791370+0000","last_active":"2024-05-27T07:25:56.931533+0000","last_peered":"2024-05-27T07:25:56.931533+0000","last_clean":"2024-05-27T07:25:56.931533+0000","last_became_active":"2024-05-27T07:21:45.790987+0000","last_became_peered":"2024-05-27T07:21:45.790987+0000","last_unstale":"2024-05-27T07:25:56.931533+0000","last_undegraded":"2024-05-27T07:25:56.931533+0000","last_fullsized":"2024-05-27T07:25:56.931533+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:21:44.543638+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:21:44.543638+0000","last_clean_scrub_stamp":"2024-05-27T07:21:44.543638+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T14:20:18.570194+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2,0],"acting":[9,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.5","version":"90'4","reported_seq":45,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-27T07:25:57.023525+0000","last_change":"2024-05-27T07:21:48.585984+0000","last_active":"2024-05-27T07:25:57.023525+0000","last_peered":"2024-05-27T07:25:57.023525+0000","last_clean":"2024-05-27T07:25:57.023525+0000","last_became_active":"2024-05-27T07:21:48.585606+0000","last_became_peered":"2024-05-27T07:21:48.585606+0000","last_unstale":"2024-05-27T07:25:57.023525+0000","last_undegraded":"2024-05-27T07:25:57.023525+0000","last_fullsized":"2024-05-27T07:25:57.023525+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:21:47.565641+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:21:47.565641+0000","last_clean_scrub_stamp":"2024-05-27T07:21:47.565641+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T17:56:17.659024+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,3,6],"acting":[10,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":31,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-27T07:25:57.024779+0000","last_change":"2024-05-27T07:21:59.734022+0000","last_active":"2024-05-27T07:25:57.024779+0000","last_peered":"2024-05-27T07:25:57.024779+0000","last_clean":"2024-05-27T07:25:57.024779+0000","last_became_active":"2024-05-27T07:21:59.733467+0000","last_became_peered":"2024-05-27T07:21:59.733467+0000","last_unstale":"2024-05-27T07:25:57.024779+0000","last_undegraded":"2024-05-27T07:25:57.024779+0000","last_fullsized":"2024-05-27T07:25:57.024779+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:21:58.698952+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:21:58.698952+0000","last_clean_scrub_stamp":"2024-05-27T07:21:58.698952+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T09:54:11.058495+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,1,8],"acting":[6,0,1,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.2","version":"91'6","reported_seq":233,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-27T07:25:57.227437+0000","last_change":"2024-05-27T07:21:45.563072+0000","last_active":"2024-05-27T07:25:57.227437+0000","last_peered":"2024-05-27T07:25:57.227437+0000","last_clean":"2024-05-27T07:25:57.227437+0000","last_became_active":"2024-05-27T07:21:45.562632+0000","last_became_peered":"2024-05-27T07:21:45.562632+0000","last_unstale":"2024-05-27T07:25:57.227437+0000","last_undegraded":"2024-05-27T07:25:57.227437+0000","last_fullsized":"2024-05-27T07:25:57.227437+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:21:44.543638+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:21:44.543638+0000","last_clean_scrub_stamp":"2024-05-27T07:21:44.543638+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T12:54:19.555467+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":979,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":182,"num_read_kb":1,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3","version":"90'7","reported_seq":49,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-27T07:25:57.023358+0000","last_change":"2024-05-27T07:21:48.594867+0000","last_active":"2024-05-27T07:25:57.023358+0000","last_peered":"2024-05-27T07:25:57.023358+0000","last_clean":"2024-05-27T07:25:57.023358+0000","last_became_active":"2024-05-27T07:21:48.594523+0000","last_became_peered":"2024-05-27T07:21:48.594523+0000","last_unstale":"2024-05-27T07:25:57.023358+0000","last_undegraded":"2024-05-27T07:25:57.023358+0000","last_fullsized":"2024-05-27T07:25:57.023358+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:21:47.565641+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:21:47.565641+0000","last_clean_scrub_stamp":"2024-05-27T07:21:47.565641+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T17:28:55.862014+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,4,0],"acting":[10,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":31,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-27T07:25:57.024044+0000","last_change":"2024-05-27T07:21:59.741936+0000","last_active":"2024-05-27T07:25:57.024044+0000","last_peered":"2024-05-27T07:25:57.024044+0000","last_clean":"2024-05-27T07:25:57.024044+0000","last_became_active":"2024-05-27T07:21:59.741458+0000","last_became_peered":"2024-05-27T07:21:59.741458+0000","last_unstale":"2024-05-27T07:25:57.024044+0000","last_undegraded":"2024-05-27T07:25:57.024044+0000","last_fullsized":"2024-05-27T07:25:57.024044+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:21:58.698952+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:21:58.698952+0000","last_clean_scrub_stamp":"2024-05-27T07:21:58.698952+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T15:51:28.560995+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,10,0],"acting":[6,5,10,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.0","version":"90'8","reported_seq":26,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-27T07:25:56.930929+0000","last_change":"2024-05-27T07:22:30.803379+0000","last_active":"2024-05-27T07:25:56.930929+0000","last_peered":"2024-05-27T07:25:56.930929+0000","last_clean":"2024-05-27T07:25:56.930929+0000","last_became_active":"2024-05-27T07:22:30.803024+0000","last_became_peered":"2024-05-27T07:22:30.803024+0000","last_unstale":"2024-05-27T07:25:56.930929+0000","last_undegraded":"2024-05-27T07:25:56.930929+0000","last_fullsized":"2024-05-27T07:25:56.930929+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:21:47.565641+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:21:47.565641+0000","last_clean_scrub_stamp":"2024-05-27T07:21:47.565641+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T07:56:32.070610+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2,4],"acting":[9,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.1","version":"91'7","reported_seq":241,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-27T07:26:01.166673+0000","last_change":"2024-05-27T07:21:45.790849+0000","last_active":"2024-05-27T07:26:01.166673+0000","last_peered":"2024-05-27T07:26:01.166673+0000","last_clean":"2024-05-27T07:26:01.166673+0000","last_became_active":"2024-05-27T07:21:45.790435+0000","last_became_peered":"2024-05-27T07:21:45.790435+0000","last_unstale":"2024-05-27T07:26:01.166673+0000","last_undegraded":"2024-05-27T07:26:01.166673+0000","last_fullsized":"2024-05-27T07:26:01.166673+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:21:44.543638+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:21:44.543638+0000","last_clean_scrub_stamp":"2024-05-27T07:21:44.543638+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T09:34:51.479882+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":187,"num_read_kb":187,"num_write":7,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,0],"acting":[6,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.5","version":"88'1","reported_seq":31,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-27T07:25:57.021734+0000","last_change":"2024-05-27T07:21:59.731338+0000","last_active":"2024-05-27T07:25:57.021734+0000","last_peered":"2024-05-27T07:25:57.021734+0000","last_clean":"2024-05-27T07:25:57.021734+0000","last_became_active":"2024-05-27T07:21:59.730847+0000","last_became_peered":"2024-05-27T07:21:59.730847+0000","last_unstale":"2024-05-27T07:25:57.021734+0000","last_undegraded":"2024-05-27T07:25:57.021734+0000","last_fullsized":"2024-05-27T07:25:57.021734+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:21:58.698952+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:21:58.698952+0000","last_clean_scrub_stamp":"2024-05-27T07:21:58.698952+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T18:18:50.901129+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,11,9],"acting":[7,5,11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2","version":"90'8","reported_seq":49,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-27T07:25:57.022192+0000","last_change":"2024-05-27T07:21:48.594274+0000","last_active":"2024-05-27T07:25:57.022192+0000","last_peered":"2024-05-27T07:25:57.022192+0000","last_clean":"2024-05-27T07:25:57.022192+0000","last_became_active":"2024-05-27T07:21:48.593940+0000","last_became_peered":"2024-05-27T07:21:48.593940+0000","last_unstale":"2024-05-27T07:25:57.022192+0000","last_undegraded":"2024-05-27T07:25:57.022192+0000","last_fullsized":"2024-05-27T07:25:57.022192+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:21:47.565641+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:21:47.565641+0000","last_clean_scrub_stamp":"2024-05-27T07:21:47.565641+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T13:25:46.531524+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"88'6","reported_seq":57,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-27T07:25:57.023871+0000","last_change":"2024-05-27T07:21:45.563365+0000","last_active":"2024-05-27T07:25:57.023871+0000","last_peered":"2024-05-27T07:25:57.023871+0000","last_clean":"2024-05-27T07:25:57.023871+0000","last_became_active":"2024-05-27T07:21:45.563012+0000","last_became_peered":"2024-05-27T07:21:45.563012+0000","last_unstale":"2024-05-27T07:25:57.023871+0000","last_undegraded":"2024-05-27T07:25:57.023871+0000","last_fullsized":"2024-05-27T07:25:57.023871+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:21:44.543638+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:21:44.543638+0000","last_clean_scrub_stamp":"2024-05-27T07:21:44.543638+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T12:58:51.079965+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":68,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":4,"num_read_kb":4,"num_write":8,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,9],"acting":[4,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.0","version":"20'92","reported_seq":119,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-27T07:25:57.021608+0000","last_change":"2024-05-27T07:20:21.611212+0000","last_active":"2024-05-27T07:25:57.021608+0000","last_peered":"2024-05-27T07:25:57.021608+0000","last_clean":"2024-05-27T07:25:57.021608+0000","last_became_active":"2024-05-27T07:20:21.605163+0000","last_became_peered":"2024-05-27T07:20:21.605163+0000","last_unstale":"2024-05-27T07:25:57.021608+0000","last_undegraded":"2024-05-27T07:25:57.021608+0000","last_fullsized":"2024-05-27T07:25:57.021608+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:16:57.668433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:16:57.668433+0000","last_clean_scrub_stamp":"2024-05-27T07:16:57.668433+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T12:10:59.074162+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":81,"num_read_kb":69,"num_write":123,"num_write_kb":2150,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":2951840,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":31,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-27T07:25:57.023734+0000","last_change":"2024-05-27T07:21:59.737210+0000","last_active":"2024-05-27T07:25:57.023734+0000","last_peered":"2024-05-27T07:25:57.023734+0000","last_clean":"2024-05-27T07:25:57.023734+0000","last_became_active":"2024-05-27T07:21:59.736799+0000","last_became_peered":"2024-05-27T07:21:59.736799+0000","last_unstale":"2024-05-27T07:25:57.023734+0000","last_undegraded":"2024-05-27T07:25:57.023734+0000","last_fullsized":"2024-05-27T07:25:57.023734+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:21:58.698952+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:21:58.698952+0000","last_clean_scrub_stamp":"2024-05-27T07:21:58.698952+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T13:03:17.153687+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,6,10],"acting":[5,9,6,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1","version":"90'5","reported_seq":47,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-27T07:25:57.023957+0000","last_change":"2024-05-27T07:21:48.591550+0000","last_active":"2024-05-27T07:25:57.023957+0000","last_peered":"2024-05-27T07:25:57.023957+0000","last_clean":"2024-05-27T07:25:57.023957+0000","last_became_active":"2024-05-27T07:21:48.591223+0000","last_became_peered":"2024-05-27T07:21:48.591223+0000","last_unstale":"2024-05-27T07:25:57.023957+0000","last_undegraded":"2024-05-27T07:25:57.023957+0000","last_fullsized":"2024-05-27T07:25:57.023957+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:21:47.565641+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:21:47.565641+0000","last_clean_scrub_stamp":"2024-05-27T07:21:47.565641+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T15:06:23.395645+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,2],"acting":[6,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.0","version":"88'4","reported_seq":49,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-27T07:25:57.021833+0000","last_change":"2024-05-27T07:21:45.569224+0000","last_active":"2024-05-27T07:25:57.021833+0000","last_peered":"2024-05-27T07:25:57.021833+0000","last_clean":"2024-05-27T07:25:57.021833+0000","last_became_active":"2024-05-27T07:21:45.568818+0000","last_became_peered":"2024-05-27T07:21:45.568818+0000","last_unstale":"2024-05-27T07:25:57.021833+0000","last_undegraded":"2024-05-27T07:25:57.021833+0000","last_fullsized":"2024-05-27T07:25:57.021833+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:21:44.543638+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:21:44.543638+0000","last_clean_scrub_stamp":"2024-05-27T07:21:44.543638+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T08:12:33.157842+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,8],"acting":[7,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":31,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-27T07:25:57.023468+0000","last_change":"2024-05-27T07:21:59.733733+0000","last_active":"2024-05-27T07:25:57.023468+0000","last_peered":"2024-05-27T07:25:57.023468+0000","last_clean":"2024-05-27T07:25:57.023468+0000","last_became_active":"2024-05-27T07:21:59.733112+0000","last_became_peered":"2024-05-27T07:21:59.733112+0000","last_unstale":"2024-05-27T07:25:57.023468+0000","last_undegraded":"2024-05-27T07:25:57.023468+0000","last_fullsized":"2024-05-27T07:25:57.023468+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:21:58.698952+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:21:58.698952+0000","last_clean_scrub_stamp":"2024-05-27T07:21:58.698952+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T16:01:30.783800+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,7,6],"acting":[11,5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.5","version":"88'4","reported_seq":52,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-27T07:25:57.024039+0000","last_change":"2024-05-27T07:21:45.792029+0000","last_active":"2024-05-27T07:25:57.024039+0000","last_peered":"2024-05-27T07:25:57.024039+0000","last_clean":"2024-05-27T07:25:57.024039+0000","last_became_active":"2024-05-27T07:21:45.791492+0000","last_became_peered":"2024-05-27T07:21:45.791492+0000","last_unstale":"2024-05-27T07:25:57.024039+0000","last_undegraded":"2024-05-27T07:25:57.024039+0000","last_fullsized":"2024-05-27T07:25:57.024039+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:21:44.543638+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:21:44.543638+0000","last_clean_scrub_stamp":"2024-05-27T07:21:44.543638+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T17:36:42.444243+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,9],"acting":[8,0,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.4","version":"90'9","reported_seq":51,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-27T07:25:57.025118+0000","last_change":"2024-05-27T07:21:48.594769+0000","last_active":"2024-05-27T07:25:57.025118+0000","last_peered":"2024-05-27T07:25:57.025118+0000","last_clean":"2024-05-27T07:25:57.025118+0000","last_became_active":"2024-05-27T07:21:48.592979+0000","last_became_peered":"2024-05-27T07:21:48.592979+0000","last_unstale":"2024-05-27T07:25:57.025118+0000","last_undegraded":"2024-05-27T07:25:57.025118+0000","last_fullsized":"2024-05-27T07:25:57.025118+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:21:47.565641+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:21:47.565641+0000","last_clean_scrub_stamp":"2024-05-27T07:21:47.565641+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T10:06:48.268230+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,9],"acting":[6,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":31,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-27T07:25:57.021843+0000","last_change":"2024-05-27T07:21:59.736405+0000","last_active":"2024-05-27T07:25:57.021843+0000","last_peered":"2024-05-27T07:25:57.021843+0000","last_clean":"2024-05-27T07:25:57.021843+0000","last_became_active":"2024-05-27T07:21:59.735991+0000","last_became_peered":"2024-05-27T07:21:59.735991+0000","last_unstale":"2024-05-27T07:25:57.021843+0000","last_undegraded":"2024-05-27T07:25:57.021843+0000","last_fullsized":"2024-05-27T07:25:57.021843+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:21:58.698952+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:21:58.698952+0000","last_clean_scrub_stamp":"2024-05-27T07:21:58.698952+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T18:32:04.931725+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,10,0,7],"acting":[3,10,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"91'51","reported_seq":121,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-27T07:25:57.176466+0000","last_change":"2024-05-27T07:21:45.566513+0000","last_active":"2024-05-27T07:25:57.176466+0000","last_peered":"2024-05-27T07:25:57.176466+0000","last_clean":"2024-05-27T07:25:57.176466+0000","last_became_active":"2024-05-27T07:21:45.566107+0000","last_became_peered":"2024-05-27T07:21:45.566107+0000","last_unstale":"2024-05-27T07:25:57.176466+0000","last_undegraded":"2024-05-27T07:25:57.176466+0000","last_fullsized":"2024-05-27T07:25:57.176466+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:21:44.543638+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:21:44.543638+0000","last_clean_scrub_stamp":"2024-05-27T07:21:44.543638+0000","objects_scrubbed":0,"log_size":51,"log_dups_size":0,"ondisk_log_size":51,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T14:11:34.851691+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":393216,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":82,"num_read_kb":70,"num_write":91,"num_write_kb":838,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,3],"acting":[9,8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.7","version":"90'5","reported_seq":46,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-27T07:25:57.022710+0000","last_change":"2024-05-27T07:21:48.593642+0000","last_active":"2024-05-27T07:25:57.022710+0000","last_peered":"2024-05-27T07:25:57.022710+0000","last_clean":"2024-05-27T07:25:57.022710+0000","last_became_active":"2024-05-27T07:21:48.592885+0000","last_became_peered":"2024-05-27T07:21:48.592885+0000","last_unstale":"2024-05-27T07:25:57.022710+0000","last_undegraded":"2024-05-27T07:25:57.022710+0000","last_fullsized":"2024-05-27T07:25:57.022710+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:21:47.565641+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:21:47.565641+0000","last_clean_scrub_stamp":"2024-05-27T07:21:47.565641+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T11:51:27.186560+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,7],"acting":[3,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":31,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-27T07:25:56.935000+0000","last_change":"2024-05-27T07:21:59.725533+0000","last_active":"2024-05-27T07:25:56.935000+0000","last_peered":"2024-05-27T07:25:56.935000+0000","last_clean":"2024-05-27T07:25:56.935000+0000","last_became_active":"2024-05-27T07:21:59.724991+0000","last_became_peered":"2024-05-27T07:21:59.724991+0000","last_unstale":"2024-05-27T07:25:56.935000+0000","last_undegraded":"2024-05-27T07:25:56.935000+0000","last_fullsized":"2024-05-27T07:25:56.935000+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:21:58.698952+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:21:58.698952+0000","last_clean_scrub_stamp":"2024-05-27T07:21:58.698952+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T17:22:54.650553+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,4,7],"acting":[2,1,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"88'7","reported_seq":55,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-27T07:25:57.024242+0000","last_change":"2024-05-27T07:21:45.572701+0000","last_active":"2024-05-27T07:25:57.024242+0000","last_peered":"2024-05-27T07:25:57.024242+0000","last_clean":"2024-05-27T07:25:57.024242+0000","last_became_active":"2024-05-27T07:21:45.572176+0000","last_became_peered":"2024-05-27T07:21:45.572176+0000","last_unstale":"2024-05-27T07:25:57.024242+0000","last_undegraded":"2024-05-27T07:25:57.024242+0000","last_fullsized":"2024-05-27T07:25:57.024242+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:21:44.543638+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:21:44.543638+0000","last_clean_scrub_stamp":"2024-05-27T07:21:44.543638+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T16:11:20.901250+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,11,6],"acting":[4,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"90'7","reported_seq":49,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-27T07:25:57.026174+0000","last_change":"2024-05-27T07:21:48.581152+0000","last_active":"2024-05-27T07:25:57.026174+0000","last_peered":"2024-05-27T07:25:57.026174+0000","last_clean":"2024-05-27T07:25:57.026174+0000","last_became_active":"2024-05-27T07:21:48.580543+0000","last_became_peered":"2024-05-27T07:21:48.580543+0000","last_unstale":"2024-05-27T07:25:57.026174+0000","last_undegraded":"2024-05-27T07:25:57.026174+0000","last_fullsized":"2024-05-27T07:25:57.026174+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:21:47.565641+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:21:47.565641+0000","last_clean_scrub_stamp":"2024-05-27T07:21:47.565641+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T18:45:56.732828+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,4],"acting":[0,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":38,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-27T07:25:57.024155+0000","last_change":"2024-05-27T07:22:31.417863+0000","last_active":"2024-05-27T07:25:57.024155+0000","last_peered":"2024-05-27T07:25:57.024155+0000","last_clean":"2024-05-27T07:25:57.024155+0000","last_became_active":"2024-05-27T07:22:31.417298+0000","last_became_peered":"2024-05-27T07:22:31.417298+0000","last_unstale":"2024-05-27T07:25:57.024155+0000","last_undegraded":"2024-05-27T07:25:57.024155+0000","last_fullsized":"2024-05-27T07:25:57.024155+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:21:58.698952+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:21:58.698952+0000","last_clean_scrub_stamp":"2024-05-27T07:21:58.698952+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T12:26:13.857477+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,10,11],"acting":[4,5,10,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.8","version":"88'2","reported_seq":59,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-27T07:25:56.934912+0000","last_change":"2024-05-27T07:23:31.626374+0000","last_active":"2024-05-27T07:25:56.934912+0000","last_peered":"2024-05-27T07:25:56.934912+0000","last_clean":"2024-05-27T07:25:56.934912+0000","last_became_active":"2024-05-27T07:23:31.623214+0000","last_became_peered":"2024-05-27T07:23:31.623214+0000","last_unstale":"2024-05-27T07:25:56.934912+0000","last_undegraded":"2024-05-27T07:25:56.934912+0000","last_fullsized":"2024-05-27T07:25:56.934912+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:21:44.543638+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:21:44.543638+0000","last_clean_scrub_stamp":"2024-05-27T07:21:44.543638+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T10:42:47.710138+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,11,0],"acting":[1,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"90'2","reported_seq":43,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-27T07:25:57.022726+0000","last_change":"2024-05-27T07:21:48.589605+0000","last_active":"2024-05-27T07:25:57.022726+0000","last_peered":"2024-05-27T07:25:57.022726+0000","last_clean":"2024-05-27T07:25:57.022726+0000","last_became_active":"2024-05-27T07:21:48.588294+0000","last_became_peered":"2024-05-27T07:21:48.588294+0000","last_unstale":"2024-05-27T07:25:57.022726+0000","last_undegraded":"2024-05-27T07:25:57.022726+0000","last_fullsized":"2024-05-27T07:25:57.022726+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:21:47.565641+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:21:47.565641+0000","last_clean_scrub_stamp":"2024-05-27T07:21:47.565641+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T09:02:29.618508+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,10],"acting":[8,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":31,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-27T07:25:56.931975+0000","last_change":"2024-05-27T07:21:59.738153+0000","last_active":"2024-05-27T07:25:56.931975+0000","last_peered":"2024-05-27T07:25:56.931975+0000","last_clean":"2024-05-27T07:25:56.931975+0000","last_became_active":"2024-05-27T07:21:59.737394+0000","last_became_peered":"2024-05-27T07:21:59.737394+0000","last_unstale":"2024-05-27T07:25:56.931975+0000","last_undegraded":"2024-05-27T07:25:56.931975+0000","last_fullsized":"2024-05-27T07:25:56.931975+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:21:58.698952+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:21:58.698952+0000","last_clean_scrub_stamp":"2024-05-27T07:21:58.698952+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T17:17:13.877688+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,2,5],"acting":[9,3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.9","version":"88'7","reported_seq":53,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-27T07:25:57.021941+0000","last_change":"2024-05-27T07:21:45.562638+0000","last_active":"2024-05-27T07:25:57.021941+0000","last_peered":"2024-05-27T07:25:57.021941+0000","last_clean":"2024-05-27T07:25:57.021941+0000","last_became_active":"2024-05-27T07:21:45.562242+0000","last_became_peered":"2024-05-27T07:21:45.562242+0000","last_unstale":"2024-05-27T07:25:57.021941+0000","last_undegraded":"2024-05-27T07:25:57.021941+0000","last_fullsized":"2024-05-27T07:25:57.021941+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:21:44.543638+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:21:44.543638+0000","last_clean_scrub_stamp":"2024-05-27T07:21:44.543638+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T10:40:15.847513+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,9,10],"acting":[3,9,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.8","version":"90'9","reported_seq":50,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-27T07:25:57.022448+0000","last_change":"2024-05-27T07:21:48.598186+0000","last_active":"2024-05-27T07:25:57.022448+0000","last_peered":"2024-05-27T07:25:57.022448+0000","last_clean":"2024-05-27T07:25:57.022448+0000","last_became_active":"2024-05-27T07:21:48.597715+0000","last_became_peered":"2024-05-27T07:21:48.597715+0000","last_unstale":"2024-05-27T07:25:57.022448+0000","last_undegraded":"2024-05-27T07:25:57.022448+0000","last_fullsized":"2024-05-27T07:25:57.022448+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:21:47.565641+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:21:47.565641+0000","last_clean_scrub_stamp":"2024-05-27T07:21:47.565641+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T08:01:35.983687+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,2],"acting":[11,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.c","version":"88'1","reported_seq":32,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-27T07:25:57.023735+0000","last_change":"2024-05-27T07:21:59.727247+0000","last_active":"2024-05-27T07:25:57.023735+0000","last_peered":"2024-05-27T07:25:57.023735+0000","last_clean":"2024-05-27T07:25:57.023735+0000","last_became_active":"2024-05-27T07:21:59.726788+0000","last_became_peered":"2024-05-27T07:21:59.726788+0000","last_unstale":"2024-05-27T07:25:57.023735+0000","last_undegraded":"2024-05-27T07:25:57.023735+0000","last_fullsized":"2024-05-27T07:25:57.023735+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:21:58.698952+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:21:58.698952+0000","last_clean_scrub_stamp":"2024-05-27T07:21:58.698952+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T10:09:27.095188+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,9,7,8],"acting":[4,9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.a","version":"91'4","reported_seq":50,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-27T07:25:56.935299+0000","last_change":"2024-05-27T07:21:45.569285+0000","last_active":"2024-05-27T07:25:56.935299+0000","last_peered":"2024-05-27T07:25:56.935299+0000","last_clean":"2024-05-27T07:25:56.935299+0000","last_became_active":"2024-05-27T07:21:45.568835+0000","last_became_peered":"2024-05-27T07:21:45.568835+0000","last_unstale":"2024-05-27T07:25:56.935299+0000","last_undegraded":"2024-05-27T07:25:56.935299+0000","last_fullsized":"2024-05-27T07:25:56.935299+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:21:44.543638+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:21:44.543638+0000","last_clean_scrub_stamp":"2024-05-27T07:21:44.543638+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T10:03:23.353800+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,8],"acting":[2,3,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.b","version":"90'8","reported_seq":49,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-27T07:25:57.022876+0000","last_change":"2024-05-27T07:21:48.583494+0000","last_active":"2024-05-27T07:25:57.022876+0000","last_peered":"2024-05-27T07:25:57.022876+0000","last_clean":"2024-05-27T07:25:57.022876+0000","last_became_active":"2024-05-27T07:21:48.583141+0000","last_became_peered":"2024-05-27T07:21:48.583141+0000","last_unstale":"2024-05-27T07:25:57.022876+0000","last_undegraded":"2024-05-27T07:25:57.022876+0000","last_fullsized":"2024-05-27T07:25:57.022876+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:21:47.565641+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:21:47.565641+0000","last_clean_scrub_stamp":"2024-05-27T07:21:47.565641+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T10:24:50.475195+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,10],"acting":[3,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":31,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-27T07:25:57.022646+0000","last_change":"2024-05-27T07:21:59.733966+0000","last_active":"2024-05-27T07:25:57.022646+0000","last_peered":"2024-05-27T07:25:57.022646+0000","last_clean":"2024-05-27T07:25:57.022646+0000","last_became_active":"2024-05-27T07:21:59.733411+0000","last_became_peered":"2024-05-27T07:21:59.733411+0000","last_unstale":"2024-05-27T07:25:57.022646+0000","last_undegraded":"2024-05-27T07:25:57.022646+0000","last_fullsized":"2024-05-27T07:25:57.022646+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:21:58.698952+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:21:58.698952+0000","last_clean_scrub_stamp":"2024-05-27T07:21:58.698952+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T15:08:04.918086+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,10,3],"acting":[4,2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.b","version":"88'4","reported_seq":52,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-27T07:25:57.022886+0000","last_change":"2024-05-27T07:21:45.570967+0000","last_active":"2024-05-27T07:25:57.022886+0000","last_peered":"2024-05-27T07:25:57.022886+0000","last_clean":"2024-05-27T07:25:57.022886+0000","last_became_active":"2024-05-27T07:21:45.570501+0000","last_became_peered":"2024-05-27T07:21:45.570501+0000","last_unstale":"2024-05-27T07:25:57.022886+0000","last_undegraded":"2024-05-27T07:25:57.022886+0000","last_fullsized":"2024-05-27T07:25:57.022886+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:21:44.543638+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:21:44.543638+0000","last_clean_scrub_stamp":"2024-05-27T07:21:44.543638+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T14:11:26.496194+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.a","version":"90'5","reported_seq":44,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-27T07:25:57.022260+0000","last_change":"2024-05-27T07:21:48.594690+0000","last_active":"2024-05-27T07:25:57.022260+0000","last_peered":"2024-05-27T07:25:57.022260+0000","last_clean":"2024-05-27T07:25:57.022260+0000","last_became_active":"2024-05-27T07:21:48.594314+0000","last_became_peered":"2024-05-27T07:21:48.594314+0000","last_unstale":"2024-05-27T07:25:57.022260+0000","last_undegraded":"2024-05-27T07:25:57.022260+0000","last_fullsized":"2024-05-27T07:25:57.022260+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:21:47.565641+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:21:47.565641+0000","last_clean_scrub_stamp":"2024-05-27T07:21:47.565641+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T14:06:43.401844+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,6],"acting":[7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":31,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-27T07:25:56.934946+0000","last_change":"2024-05-27T07:21:59.732289+0000","last_active":"2024-05-27T07:25:56.934946+0000","last_peered":"2024-05-27T07:25:56.934946+0000","last_clean":"2024-05-27T07:25:56.934946+0000","last_became_active":"2024-05-27T07:21:59.731863+0000","last_became_peered":"2024-05-27T07:21:59.731863+0000","last_unstale":"2024-05-27T07:25:56.934946+0000","last_undegraded":"2024-05-27T07:25:56.934946+0000","last_fullsized":"2024-05-27T07:25:56.934946+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:21:58.698952+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:21:58.698952+0000","last_clean_scrub_stamp":"2024-05-27T07:21:58.698952+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T11:06:38.246484+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,6,7],"acting":[1,5,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.c","version":"91'6","reported_seq":235,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-27T07:25:57.161598+0000","last_change":"2024-05-27T07:21:45.791978+0000","last_active":"2024-05-27T07:25:57.161598+0000","last_peered":"2024-05-27T07:25:57.161598+0000","last_clean":"2024-05-27T07:25:57.161598+0000","last_became_active":"2024-05-27T07:21:45.791146+0000","last_became_peered":"2024-05-27T07:21:45.791146+0000","last_unstale":"2024-05-27T07:25:57.161598+0000","last_undegraded":"2024-05-27T07:25:57.161598+0000","last_fullsized":"2024-05-27T07:25:57.161598+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:21:44.543638+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:21:44.543638+0000","last_clean_scrub_stamp":"2024-05-27T07:21:44.543638+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T09:09:24.952876+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1561,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":183,"num_read_kb":2,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,11,0],"acting":[2,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"90'7","reported_seq":46,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-27T07:25:57.022770+0000","last_change":"2024-05-27T07:21:48.602786+0000","last_active":"2024-05-27T07:25:57.022770+0000","last_peered":"2024-05-27T07:25:57.022770+0000","last_clean":"2024-05-27T07:25:57.022770+0000","last_became_active":"2024-05-27T07:21:48.602476+0000","last_became_peered":"2024-05-27T07:21:48.602476+0000","last_unstale":"2024-05-27T07:25:57.022770+0000","last_undegraded":"2024-05-27T07:25:57.022770+0000","last_fullsized":"2024-05-27T07:25:57.022770+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:21:47.565641+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:21:47.565641+0000","last_clean_scrub_stamp":"2024-05-27T07:21:47.565641+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T13:22:58.583230+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,11],"acting":[7,1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":31,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-27T07:25:57.023937+0000","last_change":"2024-05-27T07:21:59.729595+0000","last_active":"2024-05-27T07:25:57.023937+0000","last_peered":"2024-05-27T07:25:57.023937+0000","last_clean":"2024-05-27T07:25:57.023937+0000","last_became_active":"2024-05-27T07:21:59.729147+0000","last_became_peered":"2024-05-27T07:21:59.729147+0000","last_unstale":"2024-05-27T07:25:57.023937+0000","last_undegraded":"2024-05-27T07:25:57.023937+0000","last_fullsized":"2024-05-27T07:25:57.023937+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:21:58.698952+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:21:58.698952+0000","last_clean_scrub_stamp":"2024-05-27T07:21:58.698952+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T13:31:28.579617+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,9,1,4],"acting":[0,9,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.d","version":"91'41","reported_seq":95,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-27T07:25:57.174009+0000","last_change":"2024-05-27T07:21:45.569287+0000","last_active":"2024-05-27T07:25:57.174009+0000","last_peered":"2024-05-27T07:25:57.174009+0000","last_clean":"2024-05-27T07:25:57.174009+0000","last_became_active":"2024-05-27T07:21:45.568935+0000","last_became_peered":"2024-05-27T07:21:45.568935+0000","last_unstale":"2024-05-27T07:25:57.174009+0000","last_undegraded":"2024-05-27T07:25:57.174009+0000","last_fullsized":"2024-05-27T07:25:57.174009+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:21:44.543638+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:21:44.543638+0000","last_clean_scrub_stamp":"2024-05-27T07:21:44.543638+0000","objects_scrubbed":0,"log_size":41,"log_dups_size":0,"ondisk_log_size":41,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T07:27:14.674135+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":131670,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":11,"num_read_kb":10,"num_write":49,"num_write_kb":475,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,10,3],"acting":[5,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.c","version":"90'6","reported_seq":45,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-27T07:25:57.021865+0000","last_change":"2024-05-27T07:21:48.597718+0000","last_active":"2024-05-27T07:25:57.021865+0000","last_peered":"2024-05-27T07:25:57.021865+0000","last_clean":"2024-05-27T07:25:57.021865+0000","last_became_active":"2024-05-27T07:21:48.597379+0000","last_became_peered":"2024-05-27T07:21:48.597379+0000","last_unstale":"2024-05-27T07:25:57.021865+0000","last_undegraded":"2024-05-27T07:25:57.021865+0000","last_fullsized":"2024-05-27T07:25:57.021865+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:21:47.565641+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:21:47.565641+0000","last_clean_scrub_stamp":"2024-05-27T07:21:47.565641+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T08:48:41.658364+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,5],"acting":[7,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":31,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-27T07:25:57.022372+0000","last_change":"2024-05-27T07:21:59.730191+0000","last_active":"2024-05-27T07:25:57.022372+0000","last_peered":"2024-05-27T07:25:57.022372+0000","last_clean":"2024-05-27T07:25:57.022372+0000","last_became_active":"2024-05-27T07:21:59.729787+0000","last_became_peered":"2024-05-27T07:21:59.729787+0000","last_unstale":"2024-05-27T07:25:57.022372+0000","last_undegraded":"2024-05-27T07:25:57.022372+0000","last_fullsized":"2024-05-27T07:25:57.022372+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:21:58.698952+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:21:58.698952+0000","last_clean_scrub_stamp":"2024-05-27T07:21:58.698952+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T13:22:38.961246+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7,1],"acting":[5,2,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.e","version":"88'5","reported_seq":51,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-27T07:25:57.024170+0000","last_change":"2024-05-27T07:21:45.559219+0000","last_active":"2024-05-27T07:25:57.024170+0000","last_peered":"2024-05-27T07:25:57.024170+0000","last_clean":"2024-05-27T07:25:57.024170+0000","last_became_active":"2024-05-27T07:21:45.558848+0000","last_became_peered":"2024-05-27T07:21:45.558848+0000","last_unstale":"2024-05-27T07:25:57.024170+0000","last_undegraded":"2024-05-27T07:25:57.024170+0000","last_fullsized":"2024-05-27T07:25:57.024170+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:21:44.543638+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:21:44.543638+0000","last_clean_scrub_stamp":"2024-05-27T07:21:44.543638+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T15:03:55.632162+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.f","version":"90'9","reported_seq":48,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-27T07:25:57.022988+0000","last_change":"2024-05-27T07:21:48.596349+0000","last_active":"2024-05-27T07:25:57.022988+0000","last_peered":"2024-05-27T07:25:57.022988+0000","last_clean":"2024-05-27T07:25:57.022988+0000","last_became_active":"2024-05-27T07:21:48.595830+0000","last_became_peered":"2024-05-27T07:21:48.595830+0000","last_unstale":"2024-05-27T07:25:57.022988+0000","last_undegraded":"2024-05-27T07:25:57.022988+0000","last_fullsized":"2024-05-27T07:25:57.022988+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:21:47.565641+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:21:47.565641+0000","last_clean_scrub_stamp":"2024-05-27T07:21:47.565641+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T13:33:11.431988+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,8],"acting":[7,11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.20","version":"90'9","reported_seq":50,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-27T07:25:57.022558+0000","last_change":"2024-05-27T07:21:48.598504+0000","last_active":"2024-05-27T07:25:57.022558+0000","last_peered":"2024-05-27T07:25:57.022558+0000","last_clean":"2024-05-27T07:25:57.022558+0000","last_became_active":"2024-05-27T07:21:48.598103+0000","last_became_peered":"2024-05-27T07:21:48.598103+0000","last_unstale":"2024-05-27T07:25:57.022558+0000","last_undegraded":"2024-05-27T07:25:57.022558+0000","last_fullsized":"2024-05-27T07:25:57.022558+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:21:47.565641+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:21:47.565641+0000","last_clean_scrub_stamp":"2024-05-27T07:21:47.565641+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T10:34:02.743735+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,5],"acting":[11,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.21","version":"90'12","reported_seq":53,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-27T07:25:56.935301+0000","last_change":"2024-05-27T07:21:48.593379+0000","last_active":"2024-05-27T07:25:56.935301+0000","last_peered":"2024-05-27T07:25:56.935301+0000","last_clean":"2024-05-27T07:25:56.935301+0000","last_became_active":"2024-05-27T07:21:48.592995+0000","last_became_peered":"2024-05-27T07:21:48.592995+0000","last_unstale":"2024-05-27T07:25:56.935301+0000","last_undegraded":"2024-05-27T07:25:56.935301+0000","last_fullsized":"2024-05-27T07:25:56.935301+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:21:47.565641+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:21:47.565641+0000","last_clean_scrub_stamp":"2024-05-27T07:21:47.565641+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T19:01:56.288235+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,4],"acting":[2,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"90'8","reported_seq":49,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-27T07:25:56.935066+0000","last_change":"2024-05-27T07:21:48.592837+0000","last_active":"2024-05-27T07:25:56.935066+0000","last_peered":"2024-05-27T07:25:56.935066+0000","last_clean":"2024-05-27T07:25:56.935066+0000","last_became_active":"2024-05-27T07:21:48.592459+0000","last_became_peered":"2024-05-27T07:21:48.592459+0000","last_unstale":"2024-05-27T07:25:56.935066+0000","last_undegraded":"2024-05-27T07:25:56.935066+0000","last_fullsized":"2024-05-27T07:25:56.935066+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:21:47.565641+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:21:47.565641+0000","last_clean_scrub_stamp":"2024-05-27T07:21:47.565641+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T14:00:19.490348+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.23","version":"90'7","reported_seq":49,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-27T07:25:57.025975+0000","last_change":"2024-05-27T07:21:48.583139+0000","last_active":"2024-05-27T07:25:57.025975+0000","last_peered":"2024-05-27T07:25:57.025975+0000","last_clean":"2024-05-27T07:25:57.025975+0000","last_became_active":"2024-05-27T07:21:48.582822+0000","last_became_peered":"2024-05-27T07:21:48.582822+0000","last_unstale":"2024-05-27T07:25:57.025975+0000","last_undegraded":"2024-05-27T07:25:57.025975+0000","last_fullsized":"2024-05-27T07:25:57.025975+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:21:47.565641+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:21:47.565641+0000","last_clean_scrub_stamp":"2024-05-27T07:21:47.565641+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T16:30:03.454035+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,4],"acting":[0,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"90'4","reported_seq":45,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-27T07:25:57.024904+0000","last_change":"2024-05-27T07:21:48.588906+0000","last_active":"2024-05-27T07:25:57.024904+0000","last_peered":"2024-05-27T07:25:57.024904+0000","last_clean":"2024-05-27T07:25:57.024904+0000","last_became_active":"2024-05-27T07:21:48.588555+0000","last_became_peered":"2024-05-27T07:21:48.588555+0000","last_unstale":"2024-05-27T07:25:57.024904+0000","last_undegraded":"2024-05-27T07:25:57.024904+0000","last_fullsized":"2024-05-27T07:25:57.024904+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:21:47.565641+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:21:47.565641+0000","last_clean_scrub_stamp":"2024-05-27T07:21:47.565641+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T19:04:23.006852+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"90'6","reported_seq":47,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-27T07:25:57.023930+0000","last_change":"2024-05-27T07:21:48.590271+0000","last_active":"2024-05-27T07:25:57.023930+0000","last_peered":"2024-05-27T07:25:57.023930+0000","last_clean":"2024-05-27T07:25:57.023930+0000","last_became_active":"2024-05-27T07:21:48.589898+0000","last_became_peered":"2024-05-27T07:21:48.589898+0000","last_unstale":"2024-05-27T07:25:57.023930+0000","last_undegraded":"2024-05-27T07:25:57.023930+0000","last_fullsized":"2024-05-27T07:25:57.023930+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:21:47.565641+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:21:47.565641+0000","last_clean_scrub_stamp":"2024-05-27T07:21:47.565641+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T11:57:43.238211+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,5],"acting":[11,8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.26","version":"90'15","reported_seq":56,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-27T07:25:56.935533+0000","last_change":"2024-05-27T07:21:48.597321+0000","last_active":"2024-05-27T07:25:56.935533+0000","last_peered":"2024-05-27T07:25:56.935533+0000","last_clean":"2024-05-27T07:25:56.935533+0000","last_became_active":"2024-05-27T07:21:48.596953+0000","last_became_peered":"2024-05-27T07:21:48.596953+0000","last_unstale":"2024-05-27T07:25:56.935533+0000","last_undegraded":"2024-05-27T07:25:56.935533+0000","last_fullsized":"2024-05-27T07:25:56.935533+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:21:47.565641+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:21:47.565641+0000","last_clean_scrub_stamp":"2024-05-27T07:21:47.565641+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T16:21:17.645098+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,8,1],"acting":[2,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.27","version":"90'2","reported_seq":43,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-27T07:25:57.022779+0000","last_change":"2024-05-27T07:21:48.593717+0000","last_active":"2024-05-27T07:25:57.022779+0000","last_peered":"2024-05-27T07:25:57.022779+0000","last_clean":"2024-05-27T07:25:57.022779+0000","last_became_active":"2024-05-27T07:21:48.593356+0000","last_became_peered":"2024-05-27T07:21:48.593356+0000","last_unstale":"2024-05-27T07:25:57.022779+0000","last_undegraded":"2024-05-27T07:25:57.022779+0000","last_fullsized":"2024-05-27T07:25:57.022779+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:21:47.565641+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:21:47.565641+0000","last_clean_scrub_stamp":"2024-05-27T07:21:47.565641+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T17:06:27.341627+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"90'6","reported_seq":47,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-27T07:25:57.022723+0000","last_change":"2024-05-27T07:21:48.592360+0000","last_active":"2024-05-27T07:25:57.022723+0000","last_peered":"2024-05-27T07:25:57.022723+0000","last_clean":"2024-05-27T07:25:57.022723+0000","last_became_active":"2024-05-27T07:21:48.591650+0000","last_became_peered":"2024-05-27T07:21:48.591650+0000","last_unstale":"2024-05-27T07:25:57.022723+0000","last_undegraded":"2024-05-27T07:25:57.022723+0000","last_fullsized":"2024-05-27T07:25:57.022723+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:21:47.565641+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:21:47.565641+0000","last_clean_scrub_stamp":"2024-05-27T07:21:47.565641+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T17:32:19.408787+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,3],"acting":[5,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"90'8","reported_seq":49,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-27T07:25:57.022029+0000","last_change":"2024-05-27T07:21:48.587319+0000","last_active":"2024-05-27T07:25:57.022029+0000","last_peered":"2024-05-27T07:25:57.022029+0000","last_clean":"2024-05-27T07:25:57.022029+0000","last_became_active":"2024-05-27T07:21:48.586924+0000","last_became_peered":"2024-05-27T07:21:48.586924+0000","last_unstale":"2024-05-27T07:25:57.022029+0000","last_undegraded":"2024-05-27T07:25:57.022029+0000","last_fullsized":"2024-05-27T07:25:57.022029+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:21:47.565641+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:21:47.565641+0000","last_clean_scrub_stamp":"2024-05-27T07:21:47.565641+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T09:28:48.129037+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,11],"acting":[3,2,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2a","version":"90'10","reported_seq":51,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-27T07:25:56.935355+0000","last_change":"2024-05-27T07:21:48.604018+0000","last_active":"2024-05-27T07:25:56.935355+0000","last_peered":"2024-05-27T07:25:56.935355+0000","last_clean":"2024-05-27T07:25:56.935355+0000","last_became_active":"2024-05-27T07:21:48.603368+0000","last_became_peered":"2024-05-27T07:21:48.603368+0000","last_unstale":"2024-05-27T07:25:56.935355+0000","last_undegraded":"2024-05-27T07:25:56.935355+0000","last_fullsized":"2024-05-27T07:25:56.935355+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:21:47.565641+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:21:47.565641+0000","last_clean_scrub_stamp":"2024-05-27T07:21:47.565641+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T09:49:42.440480+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2b","version":"90'11","reported_seq":52,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-27T07:25:57.024693+0000","last_change":"2024-05-27T07:21:48.589434+0000","last_active":"2024-05-27T07:25:57.024693+0000","last_peered":"2024-05-27T07:25:57.024693+0000","last_clean":"2024-05-27T07:25:57.024693+0000","last_became_active":"2024-05-27T07:21:48.589102+0000","last_became_peered":"2024-05-27T07:21:48.589102+0000","last_unstale":"2024-05-27T07:25:57.024693+0000","last_undegraded":"2024-05-27T07:25:57.024693+0000","last_fullsized":"2024-05-27T07:25:57.024693+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:21:47.565641+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:21:47.565641+0000","last_clean_scrub_stamp":"2024-05-27T07:21:47.565641+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T18:28:52.955138+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2c","version":"90'8","reported_seq":55,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-27T07:25:57.023752+0000","last_change":"2024-05-27T07:22:30.802433+0000","last_active":"2024-05-27T07:25:57.023752+0000","last_peered":"2024-05-27T07:25:57.023752+0000","last_clean":"2024-05-27T07:25:57.023752+0000","last_became_active":"2024-05-27T07:22:30.801881+0000","last_became_peered":"2024-05-27T07:22:30.801881+0000","last_unstale":"2024-05-27T07:25:57.023752+0000","last_undegraded":"2024-05-27T07:25:57.023752+0000","last_fullsized":"2024-05-27T07:25:57.023752+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:21:47.565641+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:21:47.565641+0000","last_clean_scrub_stamp":"2024-05-27T07:21:47.565641+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T11:59:17.916052+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,9,8],"acting":[11,9,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":234,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":32768,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":128,"acting":128,"num_store_stats":12},{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":500,"ondisk_log_size":500,"up":192,"acting":192,"num_store_stats":12},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":81,"num_read_kb":69,"num_write":123,"num_write_kb":2150,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":2951840,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":147456,"data_stored":1771104,"data_compressed":108480,"data_compressed_allocated":135168,"data_compressed_original":1769472,"omap_allocated":0,"internal_metadata":0},"log_size":92,"ondisk_log_size":92,"up":3,"acting":3,"num_store_stats":3},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":604805,"num_objects":100,"num_object_clones":0,"num_object_copies":300,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":100,"num_whiteouts":0,"num_read":1954,"num_read_kb":1099,"num_write":386,"num_write_kb":1479,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":22,"num_keys_recovered":0,"num_objects_omap":66,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":737280,"data_stored":1814415,"data_compressed":75906,"data_compressed_allocated":98304,"data_compressed_original":1548288,"omap_allocated":0,"internal_metadata":0},"log_size":311,"ondisk_log_size":311,"up":96,"acting":96,"num_store_stats":12}],"osd_stats":[{"osd":11,"up_from":71,"seq":304942678081,"num_pgs":29,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32168,"kb_used_data":988,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712984,"statfs":{"total":95995035648,"available":95962095616,"internally_reserved":0,"allocated":1011712,"data_stored":529010,"data_compressed":35393,"data_compressed_allocated":49152,"data_compressed_original":98304,"omap_allocated":1596,"internal_metadata":31914436},"hb_peers":[0,1,2,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":65,"seq":279172874309,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28160,"kb_used_data":1140,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716992,"statfs":{"total":95995035648,"available":95966199808,"internally_reserved":0,"allocated":1167360,"data_stored":1186884,"data_compressed":71553,"data_compressed_allocated":94208,"data_compressed_original":688128,"omap_allocated":1600,"internal_metadata":27654592},"hb_peers":[0,1,2,3,4,5,6,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":60,"seq":257698037835,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36372,"kb_used_data":1036,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708780,"statfs":{"total":95995035648,"available":95957790720,"internally_reserved":0,"allocated":1060864,"data_stored":926230,"data_compressed":54539,"data_compressed_allocated":73728,"data_compressed_original":491520,"omap_allocated":1600,"internal_metadata":36174272},"hb_peers":[0,1,2,3,4,5,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":54,"seq":231928234065,"num_pgs":31,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28044,"kb_used_data":1024,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717108,"statfs":{"total":95995035648,"available":95966318592,"internally_reserved":0,"allocated":1048576,"data_stored":922490,"data_compressed":54539,"data_compressed_allocated":73728,"data_compressed_original":491520,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":48,"seq":206158430293,"num_pgs":37,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32236,"kb_used_data":1056,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712916,"statfs":{"total":95995035648,"available":95962025984,"internally_reserved":0,"allocated":1081344,"data_stored":1127639,"data_compressed":71553,"data_compressed_allocated":94208,"data_compressed_original":688128,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":43,"seq":184683593819,"num_pgs":32,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28004,"kb_used_data":984,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717148,"statfs":{"total":95995035648,"available":95966359552,"internally_reserved":0,"allocated":1007616,"data_stored":524813,"data_compressed":35393,"data_compressed_allocated":49152,"data_compressed_original":98304,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574964,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36400,"kb_used_data":1060,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708752,"statfs":{"total":95995035648,"available":95957762048,"internally_reserved":0,"allocated":1085440,"data_stored":596782,"data_compressed":35393,"data_compressed_allocated":49152,"data_compressed_original":98304,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738490,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28028,"kb_used_data":1008,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717124,"statfs":{"total":95995035648,"available":95966334976,"internally_reserved":0,"allocated":1032192,"data_stored":526269,"data_compressed":35393,"data_compressed_allocated":49152,"data_compressed_original":98304,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411439,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36412,"kb_used_data":1072,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708740,"statfs":{"total":95995035648,"available":95957749760,"internally_reserved":0,"allocated":1097728,"data_stored":597813,"data_compressed":35393,"data_compressed_allocated":49152,"data_compressed_original":98304,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,1,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":26,"seq":111669149802,"num_pgs":40,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36376,"kb_used_data":1040,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708776,"statfs":{"total":95995035648,"available":95957786624,"internally_reserved":0,"allocated":1064960,"data_stored":1048946,"data_compressed":60695,"data_compressed_allocated":81920,"data_compressed_original":614400,"omap_allocated":1593,"internal_metadata":36174279},"hb_peers":[0,1,2,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":32,"seq":137438953572,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28020,"kb_used_data":1000,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717132,"statfs":{"total":95995035648,"available":95966343168,"internally_reserved":0,"allocated":1024000,"data_stored":529531,"data_compressed":35393,"data_compressed_allocated":49152,"data_compressed_original":98304,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":37,"seq":158913790047,"num_pgs":44,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28112,"kb_used_data":1084,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717040,"statfs":{"total":95995035648,"available":95966248960,"internally_reserved":0,"allocated":1110016,"data_stored":1253507,"data_compressed":77709,"data_compressed_allocated":102400,"data_compressed_original":811008,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":36160,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":36160,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":36160,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":2569,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":94208,"data_stored":73082,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":106496,"data_stored":74113,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":525246,"data_compressed":25302,"data_compressed_allocated":32768,"data_compressed_original":516096,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1735,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":135343,"data_compressed":6156,"data_compressed_allocated":8192,"data_compressed_original":122880,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1113,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":5379,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":394694,"data_compressed":19146,"data_compressed_allocated":24576,"data_compressed_original":393216,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":394338,"data_compressed":19146,"data_compressed_allocated":24576,"data_compressed_original":393216,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":147456,"data_stored":205198,"data_compressed":6156,"data_compressed_allocated":8192,"data_compressed_original":122880,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1605,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":8192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":8192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-05-27T07:26:03.686 INFO:tasks.fwd_scrub.ceph_manager:clean! 2024-05-27T07:26:03.686 DEBUG:teuthology.orchestra.run.smithi052:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-05-27T07:26:03.839 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:03.846+0000 7f4d84061640 1 -- 172.21.15.52:0/226273576 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f4d7c1075e0 msgr2=0x7f4d7c1079f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:26:03.839 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:03.846+0000 7f4d84061640 1 --2- 172.21.15.52:0/226273576 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f4d7c1075e0 0x7f4d7c1079f0 secure :-1 s=READY pgs=418 cs=0 l=1 rev1=1 crypto rx=0x7f4d70009900 tx=0x7f4d7002a410 comp rx=0 tx=0).stop 2024-05-27T07:26:03.839 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:03.846+0000 7f4d84061640 1 -- 172.21.15.52:0/226273576 shutdown_connections 2024-05-27T07:26:03.840 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:03.846+0000 7f4d84061640 1 --2- 172.21.15.52:0/226273576 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f4d7c108010 0x7f4d7c108480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:26:03.840 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:03.846+0000 7f4d84061640 1 --2- 172.21.15.52:0/226273576 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f4d7c1075e0 0x7f4d7c1079f0 unknown :-1 s=CLOSED pgs=418 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:26:03.840 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:03.846+0000 7f4d84061640 1 --2- 172.21.15.52:0/226273576 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f4d7c102b90 0x7f4d7c102fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:26:03.840 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:03.846+0000 7f4d84061640 1 -- 172.21.15.52:0/226273576 >> 172.21.15.52:0/226273576 conn(0x7f4d7c0782a0 msgr2=0x7f4d7c0786e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:26:03.840 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:03.846+0000 7f4d84061640 1 -- 172.21.15.52:0/226273576 shutdown_connections 2024-05-27T07:26:03.840 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:03.847+0000 7f4d84061640 1 -- 172.21.15.52:0/226273576 wait complete. 2024-05-27T07:26:03.841 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:03.847+0000 7f4d84061640 1 Processor -- start 2024-05-27T07:26:03.841 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:03.847+0000 7f4d84061640 1 -- start start 2024-05-27T07:26:03.841 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:03.847+0000 7f4d84061640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f4d7c102b90 0x7f4d7c1a0fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:26:03.841 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:03.847+0000 7f4d84061640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f4d7c1075e0 0x7f4d7c1a1510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:26:03.841 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:03.847+0000 7f4d84061640 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f4d7c108010 0x7f4d7c1a57d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:26:03.841 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:03.847+0000 7f4d84061640 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x7f4d7c11a430 con 0x7f4d7c108010 2024-05-27T07:26:03.841 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:03.847+0000 7f4d84061640 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f4d7c11a2b0 con 0x7f4d7c102b90 2024-05-27T07:26:03.841 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:03.847+0000 7f4d84061640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f4d7c11a5b0 con 0x7f4d7c1075e0 2024-05-27T07:26:03.841 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:03.848+0000 7f4d825fe640 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f4d7c108010 0x7f4d7c1a57d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:26:03.842 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:03.848+0000 7f4d825fe640 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f4d7c108010 0x7f4d7c1a57d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.52:58442/0 (socket says 172.21.15.52:58442) 2024-05-27T07:26:03.842 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:03.848+0000 7f4d825fe640 1 -- 172.21.15.52:0/3836461713 learned_addr learned my addr 172.21.15.52:0/3836461713 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-05-27T07:26:03.842 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:03.848+0000 7f4d815fc640 1 --2- 172.21.15.52:0/3836461713 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f4d7c1075e0 0x7f4d7c1a1510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:26:03.842 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:03.848+0000 7f4d81dfd640 1 --2- 172.21.15.52:0/3836461713 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f4d7c102b90 0x7f4d7c1a0fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:26:03.842 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:03.848+0000 7f4d825fe640 1 -- 172.21.15.52:0/3836461713 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f4d7c1075e0 msgr2=0x7f4d7c1a1510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:26:03.842 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:03.848+0000 7f4d825fe640 1 --2- 172.21.15.52:0/3836461713 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f4d7c1075e0 0x7f4d7c1a1510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:26:03.842 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:03.848+0000 7f4d825fe640 1 -- 172.21.15.52:0/3836461713 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f4d7c102b90 msgr2=0x7f4d7c1a0fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:26:03.842 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:03.848+0000 7f4d825fe640 1 --2- 172.21.15.52:0/3836461713 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f4d7c102b90 0x7f4d7c1a0fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:26:03.842 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:03.848+0000 7f4d825fe640 1 -- 172.21.15.52:0/3836461713 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4d7c1a5e10 con 0x7f4d7c108010 2024-05-27T07:26:03.842 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:03.848+0000 7f4d825fe640 1 --2- 172.21.15.52:0/3836461713 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f4d7c108010 0x7f4d7c1a57d0 secure :-1 s=READY pgs=419 cs=0 l=1 rev1=1 crypto rx=0x7f4d7400d950 tx=0x7f4d7400de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:26:03.842 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:03.848+0000 7f4d7affd640 1 -- 172.21.15.52:0/3836461713 <== mon.0 v2:172.21.15.52:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4d74017070 con 0x7f4d7c108010 2024-05-27T07:26:03.842 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:03.848+0000 7f4d7affd640 1 -- 172.21.15.52:0/3836461713 <== mon.0 v2:172.21.15.52:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f4d74009c30 con 0x7f4d7c108010 2024-05-27T07:26:03.843 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:03.848+0000 7f4d84061640 1 -- 172.21.15.52:0/3836461713 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4d7c1a6100 con 0x7f4d7c108010 2024-05-27T07:26:03.843 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:03.848+0000 7f4d7affd640 1 -- 172.21.15.52:0/3836461713 <== mon.0 v2:172.21.15.52:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4d74003c10 con 0x7f4d7c108010 2024-05-27T07:26:03.843 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:03.848+0000 7f4d84061640 1 -- 172.21.15.52:0/3836461713 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4d7c1a65d0 con 0x7f4d7c108010 2024-05-27T07:26:03.844 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:03.850+0000 7f4d84061640 1 -- 172.21.15.52:0/3836461713 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4d7c06a2b0 con 0x7f4d7c108010 2024-05-27T07:26:03.844 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:03.851+0000 7f4d7affd640 1 -- 172.21.15.52:0/3836461713 <== mon.0 v2:172.21.15.52:3300/0 4 ==== mgrmap(e 24) ==== 149879+0+0 (secure 0 0 0) 0x7f4d74003dd0 con 0x7f4d7c108010 2024-05-27T07:26:03.844 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:03.852+0000 7f4d7affd640 1 --2- 172.21.15.52:0/3836461713 >> [v2:172.21.15.52:6800/1822318384,v1:172.21.15.52:6801/1822318384] conn(0x7f4d640b2a80 0x7f4d640b4f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:26:03.844 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:03.852+0000 7f4d81dfd640 1 --2- 172.21.15.52:0/3836461713 >> [v2:172.21.15.52:6800/1822318384,v1:172.21.15.52:6801/1822318384] conn(0x7f4d640b2a80 0x7f4d640b4f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:26:03.844 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:03.852+0000 7f4d7affd640 1 -- 172.21.15.52:0/3836461713 <== mon.0 v2:172.21.15.52:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9379+0+0 (secure 0 0 0) 0x7f4d740e2420 con 0x7f4d7c108010 2024-05-27T07:26:03.844 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:03.852+0000 7f4d81dfd640 1 --2- 172.21.15.52:0/3836461713 >> [v2:172.21.15.52:6800/1822318384,v1:172.21.15.52:6801/1822318384] conn(0x7f4d640b2a80 0x7f4d640b4f90 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f4d7c06c650 tx=0x7f4d6c00a590 comp rx=0 tx=0).ready entity=mgr.15333 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:26:03.846 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:03.856+0000 7f4d7affd640 1 -- 172.21.15.52:0/3836461713 <== mon.0 v2:172.21.15.52:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191963 (secure 0 0 0) 0x7f4d74009de0 con 0x7f4d7c108010 2024-05-27T07:26:04.035 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:04.045+0000 7f4d84061640 1 -- 172.21.15.52:0/3836461713 --> [v2:172.21.15.52:6800/1822318384,v1:172.21.15.52:6801/1822318384] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f4d7c106ca0 con 0x7f4d640b2a80 2024-05-27T07:26:04.045 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:04.055+0000 7f4d7affd640 1 -- 172.21.15.52:0/3836461713 <== mgr.15333 v2:172.21.15.52:6800/1822318384 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+339710 (secure 0 0 0) 0x7f4d7c106ca0 con 0x7f4d640b2a80 2024-05-27T07:26:04.048 INFO:teuthology.orchestra.run.smithi052.stdout: 2024-05-27T07:26:04.050 INFO:teuthology.orchestra.run.smithi052.stderr:dumped all 2024-05-27T07:26:04.052 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:04.059+0000 7f4d84061640 1 -- 172.21.15.52:0/3836461713 >> [v2:172.21.15.52:6800/1822318384,v1:172.21.15.52:6801/1822318384] conn(0x7f4d640b2a80 msgr2=0x7f4d640b4f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:26:04.052 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:04.059+0000 7f4d84061640 1 --2- 172.21.15.52:0/3836461713 >> [v2:172.21.15.52:6800/1822318384,v1:172.21.15.52:6801/1822318384] conn(0x7f4d640b2a80 0x7f4d640b4f90 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f4d7c06c650 tx=0x7f4d6c00a590 comp rx=0 tx=0).stop 2024-05-27T07:26:04.052 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:04.059+0000 7f4d84061640 1 -- 172.21.15.52:0/3836461713 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f4d7c108010 msgr2=0x7f4d7c1a57d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:26:04.052 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:04.059+0000 7f4d84061640 1 --2- 172.21.15.52:0/3836461713 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f4d7c108010 0x7f4d7c1a57d0 secure :-1 s=READY pgs=419 cs=0 l=1 rev1=1 crypto rx=0x7f4d7400d950 tx=0x7f4d7400de10 comp rx=0 tx=0).stop 2024-05-27T07:26:04.052 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:04.059+0000 7f4d84061640 1 -- 172.21.15.52:0/3836461713 shutdown_connections 2024-05-27T07:26:04.052 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:04.059+0000 7f4d84061640 1 --2- 172.21.15.52:0/3836461713 >> [v2:172.21.15.52:6800/1822318384,v1:172.21.15.52:6801/1822318384] conn(0x7f4d640b2a80 0x7f4d640b4f90 unknown :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:26:04.052 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:04.059+0000 7f4d84061640 1 --2- 172.21.15.52:0/3836461713 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f4d7c108010 0x7f4d7c1a57d0 unknown :-1 s=CLOSED pgs=419 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:26:04.052 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:04.059+0000 7f4d84061640 1 --2- 172.21.15.52:0/3836461713 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f4d7c1075e0 0x7f4d7c1a1510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:26:04.053 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:04.059+0000 7f4d84061640 1 --2- 172.21.15.52:0/3836461713 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f4d7c102b90 0x7f4d7c1a0fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:26:04.053 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:04.059+0000 7f4d84061640 1 -- 172.21.15.52:0/3836461713 >> 172.21.15.52:0/3836461713 conn(0x7f4d7c0782a0 msgr2=0x7f4d7c105580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:26:04.053 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:04.059+0000 7f4d84061640 1 -- 172.21.15.52:0/3836461713 shutdown_connections 2024-05-27T07:26:04.053 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:04.059+0000 7f4d84061640 1 -- 172.21.15.52:0/3836461713 wait complete. 2024-05-27T07:26:04.060 INFO:teuthology.orchestra.run.smithi052.stdout:{"pg_ready":true,"pg_map":{"version":6,"stamp":"2024-05-27T07:26:02.949326+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":1195407,"num_objects":104,"num_object_clones":0,"num_object_copies":314,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":104,"num_whiteouts":0,"num_read":2035,"num_read_kb":1168,"num_write":511,"num_write_kb":3631,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":2951862,"num_keys_recovered":0,"num_objects_omap":66,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":905,"ondisk_log_size":905,"up":419,"acting":419,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":419,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":378332,"kb_used_data":12492,"kb_used_omap":18,"kb_used_meta":365677,"kb_avail":1124563492,"statfs":{"total":1151940427776,"available":1151553015808,"internally_reserved":0,"allocated":12791808,"data_stored":9769914,"data_compressed":602946,"data_compressed_allocated":815104,"data_compressed_original":4374528,"omap_allocated":19143,"internal_metadata":374453561},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":5,"num_read_kb":5,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.988934"},"pg_stats":[{"pgid":"3.3f","version":"90'8","reported_seq":49,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-27T07:25:57.024226+0000","last_change":"2024-05-27T07:21:48.595870+0000","last_active":"2024-05-27T07:25:57.024226+0000","last_peered":"2024-05-27T07:25:57.024226+0000","last_clean":"2024-05-27T07:25:57.024226+0000","last_became_active":"2024-05-27T07:21:48.595020+0000","last_became_peered":"2024-05-27T07:21:48.595020+0000","last_unstale":"2024-05-27T07:25:57.024226+0000","last_undegraded":"2024-05-27T07:25:57.024226+0000","last_fullsized":"2024-05-27T07:25:57.024226+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:21:47.565641+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:21:47.565641+0000","last_clean_scrub_stamp":"2024-05-27T07:21:47.565641+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T16:43:08.920344+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.3e","version":"90'4","reported_seq":46,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-27T07:25:57.026092+0000","last_change":"2024-05-27T07:21:48.590237+0000","last_active":"2024-05-27T07:25:57.026092+0000","last_peered":"2024-05-27T07:25:57.026092+0000","last_clean":"2024-05-27T07:25:57.026092+0000","last_became_active":"2024-05-27T07:21:48.589876+0000","last_became_peered":"2024-05-27T07:21:48.589876+0000","last_unstale":"2024-05-27T07:25:57.026092+0000","last_undegraded":"2024-05-27T07:25:57.026092+0000","last_fullsized":"2024-05-27T07:25:57.026092+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:21:47.565641+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:21:47.565641+0000","last_clean_scrub_stamp":"2024-05-27T07:21:47.565641+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T16:23:08.881871+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,11,10],"acting":[0,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3d","version":"90'7","reported_seq":49,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-27T07:25:57.025547+0000","last_change":"2024-05-27T07:21:48.597744+0000","last_active":"2024-05-27T07:25:57.025547+0000","last_peered":"2024-05-27T07:25:57.025547+0000","last_clean":"2024-05-27T07:25:57.025547+0000","last_became_active":"2024-05-27T07:21:48.597410+0000","last_became_peered":"2024-05-27T07:21:48.597410+0000","last_unstale":"2024-05-27T07:25:57.025547+0000","last_undegraded":"2024-05-27T07:25:57.025547+0000","last_fullsized":"2024-05-27T07:25:57.025547+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:21:47.565641+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:21:47.565641+0000","last_clean_scrub_stamp":"2024-05-27T07:21:47.565641+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T19:06:49.643152+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"90'8","reported_seq":26,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-27T07:25:56.931367+0000","last_change":"2024-05-27T07:22:30.800824+0000","last_active":"2024-05-27T07:25:56.931367+0000","last_peered":"2024-05-27T07:25:56.931367+0000","last_clean":"2024-05-27T07:25:56.931367+0000","last_became_active":"2024-05-27T07:22:30.800415+0000","last_became_peered":"2024-05-27T07:22:30.800415+0000","last_unstale":"2024-05-27T07:25:56.931367+0000","last_undegraded":"2024-05-27T07:25:56.931367+0000","last_fullsized":"2024-05-27T07:25:56.931367+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:21:47.565641+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:21:47.565641+0000","last_clean_scrub_stamp":"2024-05-27T07:21:47.565641+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T11:06:10.451686+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,2],"acting":[9,8,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.3b","version":"90'4","reported_seq":45,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-27T07:25:57.023025+0000","last_change":"2024-05-27T07:21:48.583219+0000","last_active":"2024-05-27T07:25:57.023025+0000","last_peered":"2024-05-27T07:25:57.023025+0000","last_clean":"2024-05-27T07:25:57.023025+0000","last_became_active":"2024-05-27T07:21:48.582813+0000","last_became_peered":"2024-05-27T07:21:48.582813+0000","last_unstale":"2024-05-27T07:25:57.023025+0000","last_undegraded":"2024-05-27T07:25:57.023025+0000","last_fullsized":"2024-05-27T07:25:57.023025+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:21:47.565641+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:21:47.565641+0000","last_clean_scrub_stamp":"2024-05-27T07:21:47.565641+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T18:05:05.770965+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,3,6],"acting":[8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.3a","version":"90'6","reported_seq":47,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-27T07:25:57.023067+0000","last_change":"2024-05-27T07:21:48.592049+0000","last_active":"2024-05-27T07:25:57.023067+0000","last_peered":"2024-05-27T07:25:57.023067+0000","last_clean":"2024-05-27T07:25:57.023067+0000","last_became_active":"2024-05-27T07:21:48.591677+0000","last_became_peered":"2024-05-27T07:21:48.591677+0000","last_unstale":"2024-05-27T07:25:57.023067+0000","last_undegraded":"2024-05-27T07:25:57.023067+0000","last_fullsized":"2024-05-27T07:25:57.023067+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:21:47.565641+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:21:47.565641+0000","last_clean_scrub_stamp":"2024-05-27T07:21:47.565641+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T16:25:49.755893+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,4],"acting":[10,9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.39","version":"90'5","reported_seq":44,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-27T07:25:57.021734+0000","last_change":"2024-05-27T07:21:48.597876+0000","last_active":"2024-05-27T07:25:57.021734+0000","last_peered":"2024-05-27T07:25:57.021734+0000","last_clean":"2024-05-27T07:25:57.021734+0000","last_became_active":"2024-05-27T07:21:48.597566+0000","last_became_peered":"2024-05-27T07:21:48.597566+0000","last_unstale":"2024-05-27T07:25:57.021734+0000","last_undegraded":"2024-05-27T07:25:57.021734+0000","last_fullsized":"2024-05-27T07:25:57.021734+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:21:47.565641+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:21:47.565641+0000","last_clean_scrub_stamp":"2024-05-27T07:21:47.565641+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T09:30:43.438209+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,5],"acting":[7,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"90'9","reported_seq":50,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-27T07:25:57.021645+0000","last_change":"2024-05-27T07:21:48.593412+0000","last_active":"2024-05-27T07:25:57.021645+0000","last_peered":"2024-05-27T07:25:57.021645+0000","last_clean":"2024-05-27T07:25:57.021645+0000","last_became_active":"2024-05-27T07:21:48.593070+0000","last_became_peered":"2024-05-27T07:21:48.593070+0000","last_unstale":"2024-05-27T07:25:57.021645+0000","last_undegraded":"2024-05-27T07:25:57.021645+0000","last_fullsized":"2024-05-27T07:25:57.021645+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:21:47.565641+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:21:47.565641+0000","last_clean_scrub_stamp":"2024-05-27T07:21:47.565641+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T14:53:49.382796+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,10],"acting":[3,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"90'11","reported_seq":52,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-27T07:25:56.935451+0000","last_change":"2024-05-27T07:21:48.607244+0000","last_active":"2024-05-27T07:25:56.935451+0000","last_peered":"2024-05-27T07:25:56.935451+0000","last_clean":"2024-05-27T07:25:56.935451+0000","last_became_active":"2024-05-27T07:21:48.606238+0000","last_became_peered":"2024-05-27T07:21:48.606238+0000","last_unstale":"2024-05-27T07:25:56.935451+0000","last_undegraded":"2024-05-27T07:25:56.935451+0000","last_fullsized":"2024-05-27T07:25:56.935451+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:21:47.565641+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:21:47.565641+0000","last_clean_scrub_stamp":"2024-05-27T07:21:47.565641+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T12:36:33.142111+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,11],"acting":[1,5,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.36","version":"90'16","reported_seq":57,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-27T07:25:57.022630+0000","last_change":"2024-05-27T07:21:48.593275+0000","last_active":"2024-05-27T07:25:57.022630+0000","last_peered":"2024-05-27T07:25:57.022630+0000","last_clean":"2024-05-27T07:25:57.022630+0000","last_became_active":"2024-05-27T07:21:48.592890+0000","last_became_peered":"2024-05-27T07:21:48.592890+0000","last_unstale":"2024-05-27T07:25:57.022630+0000","last_undegraded":"2024-05-27T07:25:57.022630+0000","last_fullsized":"2024-05-27T07:25:57.022630+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:21:47.565641+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:21:47.565641+0000","last_clean_scrub_stamp":"2024-05-27T07:21:47.565641+0000","objects_scrubbed":0,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T13:47:20.827824+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,4],"acting":[10,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.35","version":"90'6","reported_seq":45,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-27T07:25:57.021854+0000","last_change":"2024-05-27T07:21:48.603424+0000","last_active":"2024-05-27T07:25:57.021854+0000","last_peered":"2024-05-27T07:25:57.021854+0000","last_clean":"2024-05-27T07:25:57.021854+0000","last_became_active":"2024-05-27T07:21:48.602962+0000","last_became_peered":"2024-05-27T07:21:48.602962+0000","last_unstale":"2024-05-27T07:25:57.021854+0000","last_undegraded":"2024-05-27T07:25:57.021854+0000","last_fullsized":"2024-05-27T07:25:57.021854+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:21:47.565641+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:21:47.565641+0000","last_clean_scrub_stamp":"2024-05-27T07:21:47.565641+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T19:19:42.063063+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,8,1],"acting":[7,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.34","version":"90'7","reported_seq":48,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-27T07:25:57.024017+0000","last_change":"2024-05-27T07:21:48.589323+0000","last_active":"2024-05-27T07:25:57.024017+0000","last_peered":"2024-05-27T07:25:57.024017+0000","last_clean":"2024-05-27T07:25:57.024017+0000","last_became_active":"2024-05-27T07:21:48.587435+0000","last_became_peered":"2024-05-27T07:21:48.587435+0000","last_unstale":"2024-05-27T07:25:57.024017+0000","last_undegraded":"2024-05-27T07:25:57.024017+0000","last_fullsized":"2024-05-27T07:25:57.024017+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:21:47.565641+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:21:47.565641+0000","last_clean_scrub_stamp":"2024-05-27T07:21:47.565641+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T13:34:44.340376+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,8],"acting":[4,10,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"90'10","reported_seq":51,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-27T07:25:57.024520+0000","last_change":"2024-05-27T07:21:48.593759+0000","last_active":"2024-05-27T07:25:57.024520+0000","last_peered":"2024-05-27T07:25:57.024520+0000","last_clean":"2024-05-27T07:25:57.024520+0000","last_became_active":"2024-05-27T07:21:48.593099+0000","last_became_peered":"2024-05-27T07:21:48.593099+0000","last_unstale":"2024-05-27T07:25:57.024520+0000","last_undegraded":"2024-05-27T07:25:57.024520+0000","last_fullsized":"2024-05-27T07:25:57.024520+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:21:47.565641+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:21:47.565641+0000","last_clean_scrub_stamp":"2024-05-27T07:21:47.565641+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T13:49:00.390792+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,5],"acting":[6,10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.32","version":"90'7","reported_seq":48,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-27T07:25:57.024016+0000","last_change":"2024-05-27T07:21:48.587486+0000","last_active":"2024-05-27T07:25:57.024016+0000","last_peered":"2024-05-27T07:25:57.024016+0000","last_clean":"2024-05-27T07:25:57.024016+0000","last_became_active":"2024-05-27T07:21:48.587159+0000","last_became_peered":"2024-05-27T07:21:48.587159+0000","last_unstale":"2024-05-27T07:25:57.024016+0000","last_undegraded":"2024-05-27T07:25:57.024016+0000","last_fullsized":"2024-05-27T07:25:57.024016+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:21:47.565641+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:21:47.565641+0000","last_clean_scrub_stamp":"2024-05-27T07:21:47.565641+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T15:34:11.908442+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,10],"acting":[0,8,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"90'7","reported_seq":48,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-27T07:25:57.024296+0000","last_change":"2024-05-27T07:21:48.596817+0000","last_active":"2024-05-27T07:25:57.024296+0000","last_peered":"2024-05-27T07:25:57.024296+0000","last_clean":"2024-05-27T07:25:57.024296+0000","last_became_active":"2024-05-27T07:21:48.596469+0000","last_became_peered":"2024-05-27T07:21:48.596469+0000","last_unstale":"2024-05-27T07:25:57.024296+0000","last_undegraded":"2024-05-27T07:25:57.024296+0000","last_fullsized":"2024-05-27T07:25:57.024296+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:21:47.565641+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:21:47.565641+0000","last_clean_scrub_stamp":"2024-05-27T07:21:47.565641+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T12:38:41.494060+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,2],"acting":[0,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.30","version":"90'9","reported_seq":50,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-27T07:25:56.934824+0000","last_change":"2024-05-27T07:21:48.604249+0000","last_active":"2024-05-27T07:25:56.934824+0000","last_peered":"2024-05-27T07:25:56.934824+0000","last_clean":"2024-05-27T07:25:56.934824+0000","last_became_active":"2024-05-27T07:21:48.603585+0000","last_became_peered":"2024-05-27T07:21:48.603585+0000","last_unstale":"2024-05-27T07:25:56.934824+0000","last_undegraded":"2024-05-27T07:25:56.934824+0000","last_fullsized":"2024-05-27T07:25:56.934824+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:21:47.565641+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:21:47.565641+0000","last_clean_scrub_stamp":"2024-05-27T07:21:47.565641+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T07:22:14.839824+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2f","version":"90'8","reported_seq":49,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-27T07:25:57.023941+0000","last_change":"2024-05-27T07:21:48.596035+0000","last_active":"2024-05-27T07:25:57.023941+0000","last_peered":"2024-05-27T07:25:57.023941+0000","last_clean":"2024-05-27T07:25:57.023941+0000","last_became_active":"2024-05-27T07:21:48.595509+0000","last_became_peered":"2024-05-27T07:21:48.595509+0000","last_unstale":"2024-05-27T07:25:57.023941+0000","last_undegraded":"2024-05-27T07:25:57.023941+0000","last_fullsized":"2024-05-27T07:25:57.023941+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:21:47.565641+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:21:47.565641+0000","last_clean_scrub_stamp":"2024-05-27T07:21:47.565641+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T13:23:03.734982+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.2e","version":"90'10","reported_seq":51,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-27T07:25:56.931790+0000","last_change":"2024-05-27T07:21:48.590002+0000","last_active":"2024-05-27T07:25:56.931790+0000","last_peered":"2024-05-27T07:25:56.931790+0000","last_clean":"2024-05-27T07:25:56.931790+0000","last_became_active":"2024-05-27T07:21:48.589608+0000","last_became_peered":"2024-05-27T07:21:48.589608+0000","last_unstale":"2024-05-27T07:25:56.931790+0000","last_undegraded":"2024-05-27T07:25:56.931790+0000","last_fullsized":"2024-05-27T07:25:56.931790+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:21:47.565641+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:21:47.565641+0000","last_clean_scrub_stamp":"2024-05-27T07:21:47.565641+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T09:33:39.174351+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,1,10],"acting":[9,1,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.2d","version":"90'7","reported_seq":48,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-27T07:25:56.935673+0000","last_change":"2024-05-27T07:21:48.594885+0000","last_active":"2024-05-27T07:25:56.935673+0000","last_peered":"2024-05-27T07:25:56.935673+0000","last_clean":"2024-05-27T07:25:56.935673+0000","last_became_active":"2024-05-27T07:21:48.594521+0000","last_became_peered":"2024-05-27T07:21:48.594521+0000","last_unstale":"2024-05-27T07:25:56.935673+0000","last_undegraded":"2024-05-27T07:25:56.935673+0000","last_fullsized":"2024-05-27T07:25:56.935673+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:21:47.565641+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:21:47.565641+0000","last_clean_scrub_stamp":"2024-05-27T07:21:47.565641+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T08:54:59.311938+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":38,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-27T07:25:56.935012+0000","last_change":"2024-05-27T07:22:30.857546+0000","last_active":"2024-05-27T07:25:56.935012+0000","last_peered":"2024-05-27T07:25:56.935012+0000","last_clean":"2024-05-27T07:25:56.935012+0000","last_became_active":"2024-05-27T07:22:30.856808+0000","last_became_peered":"2024-05-27T07:22:30.856808+0000","last_unstale":"2024-05-27T07:25:56.935012+0000","last_undegraded":"2024-05-27T07:25:56.935012+0000","last_fullsized":"2024-05-27T07:25:56.935012+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:21:58.698952+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:21:58.698952+0000","last_clean_scrub_stamp":"2024-05-27T07:21:58.698952+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T16:09:25.480864+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,8,0],"acting":[2,3,8,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1f","version":"91'14","reported_seq":244,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-27T07:25:57.227211+0000","last_change":"2024-05-27T07:21:45.566489+0000","last_active":"2024-05-27T07:25:57.227211+0000","last_peered":"2024-05-27T07:25:57.227211+0000","last_clean":"2024-05-27T07:25:57.227211+0000","last_became_active":"2024-05-27T07:21:45.566115+0000","last_became_peered":"2024-05-27T07:21:45.566115+0000","last_unstale":"2024-05-27T07:25:57.227211+0000","last_undegraded":"2024-05-27T07:25:57.227211+0000","last_fullsized":"2024-05-27T07:25:57.227211+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:21:44.543638+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:21:44.543638+0000","last_clean_scrub_stamp":"2024-05-27T07:21:44.543638+0000","objects_scrubbed":0,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T07:24:11.735014+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":182,"num_read_kb":182,"num_write":15,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,8],"acting":[9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.1e","version":"90'5","reported_seq":47,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-27T07:25:57.022556+0000","last_change":"2024-05-27T07:21:48.588574+0000","last_active":"2024-05-27T07:25:57.022556+0000","last_peered":"2024-05-27T07:25:57.022556+0000","last_clean":"2024-05-27T07:25:57.022556+0000","last_became_active":"2024-05-27T07:21:48.588212+0000","last_became_peered":"2024-05-27T07:21:48.588212+0000","last_unstale":"2024-05-27T07:25:57.022556+0000","last_undegraded":"2024-05-27T07:25:57.022556+0000","last_fullsized":"2024-05-27T07:25:57.022556+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:21:47.565641+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:21:47.565641+0000","last_clean_scrub_stamp":"2024-05-27T07:21:47.565641+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T12:15:16.017635+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":31,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-27T07:25:57.022914+0000","last_change":"2024-05-27T07:21:59.734617+0000","last_active":"2024-05-27T07:25:57.022914+0000","last_peered":"2024-05-27T07:25:57.022914+0000","last_clean":"2024-05-27T07:25:57.022914+0000","last_became_active":"2024-05-27T07:21:59.734162+0000","last_became_peered":"2024-05-27T07:21:59.734162+0000","last_unstale":"2024-05-27T07:25:57.022914+0000","last_undegraded":"2024-05-27T07:25:57.022914+0000","last_fullsized":"2024-05-27T07:25:57.022914+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:21:58.698952+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:21:58.698952+0000","last_clean_scrub_stamp":"2024-05-27T07:21:58.698952+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T12:40:01.375520+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,9,3],"acting":[4,6,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1e","version":"88'2","reported_seq":49,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-27T07:25:57.022733+0000","last_change":"2024-05-27T07:21:45.790907+0000","last_active":"2024-05-27T07:25:57.022733+0000","last_peered":"2024-05-27T07:25:57.022733+0000","last_clean":"2024-05-27T07:25:57.022733+0000","last_became_active":"2024-05-27T07:21:45.790451+0000","last_became_peered":"2024-05-27T07:21:45.790451+0000","last_unstale":"2024-05-27T07:25:57.022733+0000","last_undegraded":"2024-05-27T07:25:57.022733+0000","last_fullsized":"2024-05-27T07:25:57.022733+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:21:44.543638+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:21:44.543638+0000","last_clean_scrub_stamp":"2024-05-27T07:21:44.543638+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T10:21:56.524930+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"90'6","reported_seq":48,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-27T07:25:57.026415+0000","last_change":"2024-05-27T07:21:48.595975+0000","last_active":"2024-05-27T07:25:57.026415+0000","last_peered":"2024-05-27T07:25:57.026415+0000","last_clean":"2024-05-27T07:25:57.026415+0000","last_became_active":"2024-05-27T07:21:48.595608+0000","last_became_peered":"2024-05-27T07:21:48.595608+0000","last_unstale":"2024-05-27T07:25:57.026415+0000","last_undegraded":"2024-05-27T07:25:57.026415+0000","last_fullsized":"2024-05-27T07:25:57.026415+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:21:47.565641+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:21:47.565641+0000","last_clean_scrub_stamp":"2024-05-27T07:21:47.565641+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T07:24:44.797973+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,9],"acting":[0,1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":31,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-27T07:25:57.023806+0000","last_change":"2024-05-27T07:21:59.732788+0000","last_active":"2024-05-27T07:25:57.023806+0000","last_peered":"2024-05-27T07:25:57.023806+0000","last_clean":"2024-05-27T07:25:57.023806+0000","last_became_active":"2024-05-27T07:21:59.732310+0000","last_became_peered":"2024-05-27T07:21:59.732310+0000","last_unstale":"2024-05-27T07:25:57.023806+0000","last_undegraded":"2024-05-27T07:25:57.023806+0000","last_fullsized":"2024-05-27T07:25:57.023806+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:21:58.698952+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:21:58.698952+0000","last_clean_scrub_stamp":"2024-05-27T07:21:58.698952+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T13:54:51.490154+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,9,5],"acting":[4,3,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1d","version":"91'13","reported_seq":61,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-27T07:25:57.023929+0000","last_change":"2024-05-27T07:21:45.569746+0000","last_active":"2024-05-27T07:25:57.023929+0000","last_peered":"2024-05-27T07:25:57.023929+0000","last_clean":"2024-05-27T07:25:57.023929+0000","last_became_active":"2024-05-27T07:21:45.569366+0000","last_became_peered":"2024-05-27T07:21:45.569366+0000","last_unstale":"2024-05-27T07:25:57.023929+0000","last_undegraded":"2024-05-27T07:25:57.023929+0000","last_fullsized":"2024-05-27T07:25:57.023929+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:21:44.543638+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:21:44.543638+0000","last_clean_scrub_stamp":"2024-05-27T07:21:44.543638+0000","objects_scrubbed":0,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T15:58:32.977754+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":18,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":7,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1c","version":"90'9","reported_seq":50,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-27T07:25:56.935051+0000","last_change":"2024-05-27T07:21:48.606138+0000","last_active":"2024-05-27T07:25:56.935051+0000","last_peered":"2024-05-27T07:25:56.935051+0000","last_clean":"2024-05-27T07:25:56.935051+0000","last_became_active":"2024-05-27T07:21:48.605711+0000","last_became_peered":"2024-05-27T07:21:48.605711+0000","last_unstale":"2024-05-27T07:25:56.935051+0000","last_undegraded":"2024-05-27T07:25:56.935051+0000","last_fullsized":"2024-05-27T07:25:56.935051+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:21:47.565641+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:21:47.565641+0000","last_clean_scrub_stamp":"2024-05-27T07:21:47.565641+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T10:46:39.052387+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,6],"acting":[1,10,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":31,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-27T07:25:57.023535+0000","last_change":"2024-05-27T07:21:59.735475+0000","last_active":"2024-05-27T07:25:57.023535+0000","last_peered":"2024-05-27T07:25:57.023535+0000","last_clean":"2024-05-27T07:25:57.023535+0000","last_became_active":"2024-05-27T07:21:59.734986+0000","last_became_peered":"2024-05-27T07:21:59.734986+0000","last_unstale":"2024-05-27T07:25:57.023535+0000","last_undegraded":"2024-05-27T07:25:57.023535+0000","last_fullsized":"2024-05-27T07:25:57.023535+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:21:58.698952+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:21:58.698952+0000","last_clean_scrub_stamp":"2024-05-27T07:21:58.698952+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T14:48:28.198718+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5,3],"acting":[4,6,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1c","version":"88'5","reported_seq":53,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-27T07:25:57.023547+0000","last_change":"2024-05-27T07:21:45.572882+0000","last_active":"2024-05-27T07:25:57.023547+0000","last_peered":"2024-05-27T07:25:57.023547+0000","last_clean":"2024-05-27T07:25:57.023547+0000","last_became_active":"2024-05-27T07:21:45.572460+0000","last_became_peered":"2024-05-27T07:21:45.572460+0000","last_unstale":"2024-05-27T07:25:57.023547+0000","last_undegraded":"2024-05-27T07:25:57.023547+0000","last_fullsized":"2024-05-27T07:25:57.023547+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:21:44.543638+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:21:44.543638+0000","last_clean_scrub_stamp":"2024-05-27T07:21:44.543638+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T09:40:11.311923+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":7,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,1,5],"acting":[8,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1d","version":"90'15","reported_seq":56,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-27T07:25:57.023149+0000","last_change":"2024-05-27T07:21:48.596217+0000","last_active":"2024-05-27T07:25:57.023149+0000","last_peered":"2024-05-27T07:25:57.023149+0000","last_clean":"2024-05-27T07:25:57.023149+0000","last_became_active":"2024-05-27T07:21:48.595832+0000","last_became_peered":"2024-05-27T07:21:48.595832+0000","last_unstale":"2024-05-27T07:25:57.023149+0000","last_undegraded":"2024-05-27T07:25:57.023149+0000","last_fullsized":"2024-05-27T07:25:57.023149+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:21:47.565641+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:21:47.565641+0000","last_clean_scrub_stamp":"2024-05-27T07:21:47.565641+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T10:08:05.304404+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":31,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-27T07:25:56.935413+0000","last_change":"2024-05-27T07:21:59.729617+0000","last_active":"2024-05-27T07:25:56.935413+0000","last_peered":"2024-05-27T07:25:56.935413+0000","last_clean":"2024-05-27T07:25:56.935413+0000","last_became_active":"2024-05-27T07:21:59.729003+0000","last_became_peered":"2024-05-27T07:21:59.729003+0000","last_unstale":"2024-05-27T07:25:56.935413+0000","last_undegraded":"2024-05-27T07:25:56.935413+0000","last_fullsized":"2024-05-27T07:25:56.935413+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:21:58.698952+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:21:58.698952+0000","last_clean_scrub_stamp":"2024-05-27T07:21:58.698952+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T14:55:04.844948+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7,11],"acting":[2,5,7,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1b","version":"91'8","reported_seq":239,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-27T07:25:57.161701+0000","last_change":"2024-05-27T07:21:45.566355+0000","last_active":"2024-05-27T07:25:57.161701+0000","last_peered":"2024-05-27T07:25:57.161701+0000","last_clean":"2024-05-27T07:25:57.161701+0000","last_became_active":"2024-05-27T07:21:45.564776+0000","last_became_peered":"2024-05-27T07:21:45.564776+0000","last_unstale":"2024-05-27T07:25:57.161701+0000","last_undegraded":"2024-05-27T07:25:57.161701+0000","last_fullsized":"2024-05-27T07:25:57.161701+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:21:44.543638+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:21:44.543638+0000","last_clean_scrub_stamp":"2024-05-27T07:21:44.543638+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T12:10:41.632630+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1545,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":185,"num_read_kb":4,"num_write":5,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1a","version":"90'7","reported_seq":48,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-27T07:25:57.023028+0000","last_change":"2024-05-27T07:21:48.588837+0000","last_active":"2024-05-27T07:25:57.023028+0000","last_peered":"2024-05-27T07:25:57.023028+0000","last_clean":"2024-05-27T07:25:57.023028+0000","last_became_active":"2024-05-27T07:21:48.587149+0000","last_became_peered":"2024-05-27T07:21:48.587149+0000","last_unstale":"2024-05-27T07:25:57.023028+0000","last_undegraded":"2024-05-27T07:25:57.023028+0000","last_fullsized":"2024-05-27T07:25:57.023028+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:21:47.565641+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:21:47.565641+0000","last_clean_scrub_stamp":"2024-05-27T07:21:47.565641+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T09:20:05.124189+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,5],"acting":[8,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":31,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-27T07:25:56.935491+0000","last_change":"2024-05-27T07:21:59.730003+0000","last_active":"2024-05-27T07:25:56.935491+0000","last_peered":"2024-05-27T07:25:56.935491+0000","last_clean":"2024-05-27T07:25:56.935491+0000","last_became_active":"2024-05-27T07:21:59.729195+0000","last_became_peered":"2024-05-27T07:21:59.729195+0000","last_unstale":"2024-05-27T07:25:56.935491+0000","last_undegraded":"2024-05-27T07:25:56.935491+0000","last_fullsized":"2024-05-27T07:25:56.935491+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:21:58.698952+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:21:58.698952+0000","last_clean_scrub_stamp":"2024-05-27T07:21:58.698952+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T13:13:41.612300+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,9,5],"acting":[2,6,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1a","version":"88'7","reported_seq":54,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-27T07:25:57.022914+0000","last_change":"2024-05-27T07:21:45.567279+0000","last_active":"2024-05-27T07:25:57.022914+0000","last_peered":"2024-05-27T07:25:57.022914+0000","last_clean":"2024-05-27T07:25:57.022914+0000","last_became_active":"2024-05-27T07:21:45.566920+0000","last_became_peered":"2024-05-27T07:21:45.566920+0000","last_unstale":"2024-05-27T07:25:57.022914+0000","last_undegraded":"2024-05-27T07:25:57.022914+0000","last_fullsized":"2024-05-27T07:25:57.022914+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:21:44.543638+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:21:44.543638+0000","last_clean_scrub_stamp":"2024-05-27T07:21:44.543638+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T18:12:23.015535+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,8,6],"acting":[10,8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.1b","version":"90'8","reported_seq":55,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-27T07:25:57.023087+0000","last_change":"2024-05-27T07:22:30.857080+0000","last_active":"2024-05-27T07:25:57.023087+0000","last_peered":"2024-05-27T07:25:57.023087+0000","last_clean":"2024-05-27T07:25:57.023087+0000","last_became_active":"2024-05-27T07:22:30.856479+0000","last_became_peered":"2024-05-27T07:22:30.856479+0000","last_unstale":"2024-05-27T07:25:57.023087+0000","last_undegraded":"2024-05-27T07:25:57.023087+0000","last_fullsized":"2024-05-27T07:25:57.023087+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:21:47.565641+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:21:47.565641+0000","last_clean_scrub_stamp":"2024-05-27T07:21:47.565641+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T13:29:16.054448+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,8,3],"acting":[5,8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":38,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-27T07:25:57.023817+0000","last_change":"2024-05-27T07:22:30.805292+0000","last_active":"2024-05-27T07:25:57.023817+0000","last_peered":"2024-05-27T07:25:57.023817+0000","last_clean":"2024-05-27T07:25:57.023817+0000","last_became_active":"2024-05-27T07:22:30.804779+0000","last_became_peered":"2024-05-27T07:22:30.804779+0000","last_unstale":"2024-05-27T07:25:57.023817+0000","last_undegraded":"2024-05-27T07:25:57.023817+0000","last_fullsized":"2024-05-27T07:25:57.023817+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:21:58.698952+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:21:58.698952+0000","last_clean_scrub_stamp":"2024-05-27T07:21:58.698952+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T11:12:49.095931+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,8,0],"acting":[11,4,8,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.19","version":"88'3","reported_seq":49,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-27T07:25:57.024211+0000","last_change":"2024-05-27T07:21:45.561255+0000","last_active":"2024-05-27T07:25:57.024211+0000","last_peered":"2024-05-27T07:25:57.024211+0000","last_clean":"2024-05-27T07:25:57.024211+0000","last_became_active":"2024-05-27T07:21:45.560896+0000","last_became_peered":"2024-05-27T07:21:45.560896+0000","last_unstale":"2024-05-27T07:25:57.024211+0000","last_undegraded":"2024-05-27T07:25:57.024211+0000","last_fullsized":"2024-05-27T07:25:57.024211+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:21:44.543638+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:21:44.543638+0000","last_clean_scrub_stamp":"2024-05-27T07:21:44.543638+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T17:05:24.074628+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,9],"acting":[6,4,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"90'8","reported_seq":49,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-27T07:25:57.021953+0000","last_change":"2024-05-27T07:21:48.586045+0000","last_active":"2024-05-27T07:25:57.021953+0000","last_peered":"2024-05-27T07:25:57.021953+0000","last_clean":"2024-05-27T07:25:57.021953+0000","last_became_active":"2024-05-27T07:21:48.585681+0000","last_became_peered":"2024-05-27T07:21:48.585681+0000","last_unstale":"2024-05-27T07:25:57.021953+0000","last_undegraded":"2024-05-27T07:25:57.021953+0000","last_fullsized":"2024-05-27T07:25:57.021953+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:21:47.565641+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:21:47.565641+0000","last_clean_scrub_stamp":"2024-05-27T07:21:47.565641+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T09:56:36.408623+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,6],"acting":[3,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":31,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-27T07:25:57.025932+0000","last_change":"2024-05-27T07:21:59.739544+0000","last_active":"2024-05-27T07:25:57.025932+0000","last_peered":"2024-05-27T07:25:57.025932+0000","last_clean":"2024-05-27T07:25:57.025932+0000","last_became_active":"2024-05-27T07:21:59.739132+0000","last_became_peered":"2024-05-27T07:21:59.739132+0000","last_unstale":"2024-05-27T07:25:57.025932+0000","last_undegraded":"2024-05-27T07:25:57.025932+0000","last_fullsized":"2024-05-27T07:25:57.025932+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:21:58.698952+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:21:58.698952+0000","last_clean_scrub_stamp":"2024-05-27T07:21:58.698952+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T10:06:05.126579+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,6,1],"acting":[0,7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.18","version":"88'3","reported_seq":50,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-27T07:25:57.022592+0000","last_change":"2024-05-27T07:21:45.568171+0000","last_active":"2024-05-27T07:25:57.022592+0000","last_peered":"2024-05-27T07:25:57.022592+0000","last_clean":"2024-05-27T07:25:57.022592+0000","last_became_active":"2024-05-27T07:21:45.567627+0000","last_became_peered":"2024-05-27T07:21:45.567627+0000","last_unstale":"2024-05-27T07:25:57.022592+0000","last_undegraded":"2024-05-27T07:25:57.022592+0000","last_fullsized":"2024-05-27T07:25:57.022592+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:21:44.543638+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:21:44.543638+0000","last_clean_scrub_stamp":"2024-05-27T07:21:44.543638+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T16:43:56.407079+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"90'11","reported_seq":52,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-27T07:25:56.930989+0000","last_change":"2024-05-27T07:21:48.587452+0000","last_active":"2024-05-27T07:25:56.930989+0000","last_peered":"2024-05-27T07:25:56.930989+0000","last_clean":"2024-05-27T07:25:56.930989+0000","last_became_active":"2024-05-27T07:21:48.587027+0000","last_became_peered":"2024-05-27T07:21:48.587027+0000","last_unstale":"2024-05-27T07:25:56.930989+0000","last_undegraded":"2024-05-27T07:25:56.930989+0000","last_fullsized":"2024-05-27T07:25:56.930989+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:21:47.565641+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:21:47.565641+0000","last_clean_scrub_stamp":"2024-05-27T07:21:47.565641+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T13:51:44.890061+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,10],"acting":[9,3,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":31,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-27T07:25:57.021908+0000","last_change":"2024-05-27T07:21:59.738292+0000","last_active":"2024-05-27T07:25:57.021908+0000","last_peered":"2024-05-27T07:25:57.021908+0000","last_clean":"2024-05-27T07:25:57.021908+0000","last_became_active":"2024-05-27T07:21:59.737941+0000","last_became_peered":"2024-05-27T07:21:59.737941+0000","last_unstale":"2024-05-27T07:25:57.021908+0000","last_undegraded":"2024-05-27T07:25:57.021908+0000","last_fullsized":"2024-05-27T07:25:57.021908+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:21:58.698952+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:21:58.698952+0000","last_clean_scrub_stamp":"2024-05-27T07:21:58.698952+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T13:54:24.192917+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1,5],"acting":[3,7,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.17","version":"91'7","reported_seq":416,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-27T07:25:57.227899+0000","last_change":"2024-05-27T07:21:45.568338+0000","last_active":"2024-05-27T07:25:57.227899+0000","last_peered":"2024-05-27T07:25:57.227899+0000","last_clean":"2024-05-27T07:25:57.227899+0000","last_became_active":"2024-05-27T07:21:45.567856+0000","last_became_peered":"2024-05-27T07:21:45.567856+0000","last_unstale":"2024-05-27T07:25:57.227899+0000","last_undegraded":"2024-05-27T07:25:57.227899+0000","last_fullsized":"2024-05-27T07:25:57.227899+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:21:44.543638+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:21:44.543638+0000","last_clean_scrub_stamp":"2024-05-27T07:21:44.543638+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T15:02:04.188711+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":180,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":362,"num_read_kb":362,"num_write":8,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,5,9],"acting":[10,5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.16","version":"90'10","reported_seq":51,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-27T07:25:56.935595+0000","last_change":"2024-05-27T07:21:48.606062+0000","last_active":"2024-05-27T07:25:56.935595+0000","last_peered":"2024-05-27T07:25:56.935595+0000","last_clean":"2024-05-27T07:25:56.935595+0000","last_became_active":"2024-05-27T07:21:48.605423+0000","last_became_peered":"2024-05-27T07:21:48.605423+0000","last_unstale":"2024-05-27T07:25:56.935595+0000","last_undegraded":"2024-05-27T07:25:56.935595+0000","last_fullsized":"2024-05-27T07:25:56.935595+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:21:47.565641+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:21:47.565641+0000","last_clean_scrub_stamp":"2024-05-27T07:21:47.565641+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T14:39:06.921519+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,9],"acting":[1,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":31,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-27T07:25:56.930933+0000","last_change":"2024-05-27T07:21:59.735150+0000","last_active":"2024-05-27T07:25:56.930933+0000","last_peered":"2024-05-27T07:25:56.930933+0000","last_clean":"2024-05-27T07:25:56.930933+0000","last_became_active":"2024-05-27T07:21:59.734688+0000","last_became_peered":"2024-05-27T07:21:59.734688+0000","last_unstale":"2024-05-27T07:25:56.930933+0000","last_undegraded":"2024-05-27T07:25:56.930933+0000","last_fullsized":"2024-05-27T07:25:56.930933+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:21:58.698952+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:21:58.698952+0000","last_clean_scrub_stamp":"2024-05-27T07:21:58.698952+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T15:38:02.095947+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,0,2,4],"acting":[9,0,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.16","version":"88'4","reported_seq":48,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-27T07:25:57.022536+0000","last_change":"2024-05-27T07:21:45.559550+0000","last_active":"2024-05-27T07:25:57.022536+0000","last_peered":"2024-05-27T07:25:57.022536+0000","last_clean":"2024-05-27T07:25:57.022536+0000","last_became_active":"2024-05-27T07:21:45.559181+0000","last_became_peered":"2024-05-27T07:21:45.559181+0000","last_unstale":"2024-05-27T07:25:57.022536+0000","last_undegraded":"2024-05-27T07:25:57.022536+0000","last_fullsized":"2024-05-27T07:25:57.022536+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:21:44.543638+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:21:44.543638+0000","last_clean_scrub_stamp":"2024-05-27T07:21:44.543638+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T10:33:45.288437+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.17","version":"90'9","reported_seq":50,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-27T07:25:57.026187+0000","last_change":"2024-05-27T07:21:48.593631+0000","last_active":"2024-05-27T07:25:57.026187+0000","last_peered":"2024-05-27T07:25:57.026187+0000","last_clean":"2024-05-27T07:25:57.026187+0000","last_became_active":"2024-05-27T07:21:48.592330+0000","last_became_peered":"2024-05-27T07:21:48.592330+0000","last_unstale":"2024-05-27T07:25:57.026187+0000","last_undegraded":"2024-05-27T07:25:57.026187+0000","last_fullsized":"2024-05-27T07:25:57.026187+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:21:47.565641+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:21:47.565641+0000","last_clean_scrub_stamp":"2024-05-27T07:21:47.565641+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T19:19:51.593307+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,5],"acting":[0,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":31,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-27T07:25:57.023937+0000","last_change":"2024-05-27T07:21:59.725158+0000","last_active":"2024-05-27T07:25:57.023937+0000","last_peered":"2024-05-27T07:25:57.023937+0000","last_clean":"2024-05-27T07:25:57.023937+0000","last_became_active":"2024-05-27T07:21:59.724651+0000","last_became_peered":"2024-05-27T07:21:59.724651+0000","last_unstale":"2024-05-27T07:25:57.023937+0000","last_undegraded":"2024-05-27T07:25:57.023937+0000","last_fullsized":"2024-05-27T07:25:57.023937+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:21:58.698952+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:21:58.698952+0000","last_clean_scrub_stamp":"2024-05-27T07:21:58.698952+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T12:10:45.747484+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2,9],"acting":[4,1,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.15","version":"88'5","reported_seq":52,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-27T07:25:57.022957+0000","last_change":"2024-05-27T07:21:45.791831+0000","last_active":"2024-05-27T07:25:57.022957+0000","last_peered":"2024-05-27T07:25:57.022957+0000","last_clean":"2024-05-27T07:25:57.022957+0000","last_became_active":"2024-05-27T07:21:45.791271+0000","last_became_peered":"2024-05-27T07:21:45.791271+0000","last_unstale":"2024-05-27T07:25:57.022957+0000","last_undegraded":"2024-05-27T07:25:57.022957+0000","last_fullsized":"2024-05-27T07:25:57.022957+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:21:44.543638+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:21:44.543638+0000","last_clean_scrub_stamp":"2024-05-27T07:21:44.543638+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T16:20:37.492337+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":7,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,10],"acting":[5,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.14","version":"90'11","reported_seq":52,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-27T07:25:56.935320+0000","last_change":"2024-05-27T07:21:48.605605+0000","last_active":"2024-05-27T07:25:56.935320+0000","last_peered":"2024-05-27T07:25:56.935320+0000","last_clean":"2024-05-27T07:25:56.935320+0000","last_became_active":"2024-05-27T07:21:48.604655+0000","last_became_peered":"2024-05-27T07:21:48.604655+0000","last_unstale":"2024-05-27T07:25:56.935320+0000","last_undegraded":"2024-05-27T07:25:56.935320+0000","last_fullsized":"2024-05-27T07:25:56.935320+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:21:47.565641+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:21:47.565641+0000","last_clean_scrub_stamp":"2024-05-27T07:21:47.565641+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T14:33:53.661983+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,8,4],"acting":[1,8,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":38,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-27T07:25:56.931232+0000","last_change":"2024-05-27T07:22:31.418393+0000","last_active":"2024-05-27T07:25:56.931232+0000","last_peered":"2024-05-27T07:25:56.931232+0000","last_clean":"2024-05-27T07:25:56.931232+0000","last_became_active":"2024-05-27T07:22:31.418006+0000","last_became_peered":"2024-05-27T07:22:31.418006+0000","last_unstale":"2024-05-27T07:25:56.931232+0000","last_undegraded":"2024-05-27T07:25:56.931232+0000","last_fullsized":"2024-05-27T07:25:56.931232+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:21:58.698952+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:21:58.698952+0000","last_clean_scrub_stamp":"2024-05-27T07:21:58.698952+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T14:30:55.130461+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,11,10],"acting":[9,8,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.14","version":"91'44","reported_seq":466,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-27T07:25:57.181678+0000","last_change":"2024-05-27T07:21:45.564829+0000","last_active":"2024-05-27T07:25:57.181678+0000","last_peered":"2024-05-27T07:25:57.181678+0000","last_clean":"2024-05-27T07:25:57.181678+0000","last_became_active":"2024-05-27T07:21:45.564457+0000","last_became_peered":"2024-05-27T07:21:45.564457+0000","last_unstale":"2024-05-27T07:25:57.181678+0000","last_undegraded":"2024-05-27T07:25:57.181678+0000","last_fullsized":"2024-05-27T07:25:57.181678+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:21:44.543638+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:21:44.543638+0000","last_clean_scrub_stamp":"2024-05-27T07:21:44.543638+0000","objects_scrubbed":0,"log_size":44,"log_dups_size":0,"ondisk_log_size":44,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T19:12:44.546989+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70939,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":375,"num_read_kb":257,"num_write":46,"num_write_kb":92,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,1,2],"acting":[10,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.15","version":"90'6","reported_seq":47,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-27T07:25:57.023640+0000","last_change":"2024-05-27T07:21:48.591672+0000","last_active":"2024-05-27T07:25:57.023640+0000","last_peered":"2024-05-27T07:25:57.023640+0000","last_clean":"2024-05-27T07:25:57.023640+0000","last_became_active":"2024-05-27T07:21:48.590861+0000","last_became_peered":"2024-05-27T07:21:48.590861+0000","last_unstale":"2024-05-27T07:25:57.023640+0000","last_undegraded":"2024-05-27T07:25:57.023640+0000","last_fullsized":"2024-05-27T07:25:57.023640+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:21:47.565641+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:21:47.565641+0000","last_clean_scrub_stamp":"2024-05-27T07:21:47.565641+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T12:03:40.627764+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,7],"acting":[8,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":31,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-27T07:25:57.023508+0000","last_change":"2024-05-27T07:21:59.736446+0000","last_active":"2024-05-27T07:25:57.023508+0000","last_peered":"2024-05-27T07:25:57.023508+0000","last_clean":"2024-05-27T07:25:57.023508+0000","last_became_active":"2024-05-27T07:21:59.735854+0000","last_became_peered":"2024-05-27T07:21:59.735854+0000","last_unstale":"2024-05-27T07:25:57.023508+0000","last_undegraded":"2024-05-27T07:25:57.023508+0000","last_fullsized":"2024-05-27T07:25:57.023508+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:21:58.698952+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:21:58.698952+0000","last_clean_scrub_stamp":"2024-05-27T07:21:58.698952+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T08:02:14.640325+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,8,3,6],"acting":[5,8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.13","version":"88'9","reported_seq":57,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-27T07:25:56.935277+0000","last_change":"2024-05-27T07:21:45.562930+0000","last_active":"2024-05-27T07:25:56.935277+0000","last_peered":"2024-05-27T07:25:56.935277+0000","last_clean":"2024-05-27T07:25:56.935277+0000","last_became_active":"2024-05-27T07:21:45.562515+0000","last_became_peered":"2024-05-27T07:21:45.562515+0000","last_unstale":"2024-05-27T07:25:56.935277+0000","last_undegraded":"2024-05-27T07:25:56.935277+0000","last_fullsized":"2024-05-27T07:25:56.935277+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:21:44.543638+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:21:44.543638+0000","last_clean_scrub_stamp":"2024-05-27T07:21:44.543638+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T17:54:24.006219+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":68,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":13,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,10,3],"acting":[2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.12","version":"90'9","reported_seq":50,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-27T07:25:57.024207+0000","last_change":"2024-05-27T07:21:48.591090+0000","last_active":"2024-05-27T07:25:57.024207+0000","last_peered":"2024-05-27T07:25:57.024207+0000","last_clean":"2024-05-27T07:25:57.024207+0000","last_became_active":"2024-05-27T07:21:48.590726+0000","last_became_peered":"2024-05-27T07:21:48.590726+0000","last_unstale":"2024-05-27T07:25:57.024207+0000","last_undegraded":"2024-05-27T07:25:57.024207+0000","last_fullsized":"2024-05-27T07:25:57.024207+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:21:47.565641+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:21:47.565641+0000","last_clean_scrub_stamp":"2024-05-27T07:21:47.565641+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T18:40:03.880407+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,3],"acting":[6,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.14","version":"0'0","reported_seq":31,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-27T07:25:57.023543+0000","last_change":"2024-05-27T07:21:59.738375+0000","last_active":"2024-05-27T07:25:57.023543+0000","last_peered":"2024-05-27T07:25:57.023543+0000","last_clean":"2024-05-27T07:25:57.023543+0000","last_became_active":"2024-05-27T07:21:59.737957+0000","last_became_peered":"2024-05-27T07:21:59.737957+0000","last_unstale":"2024-05-27T07:25:57.023543+0000","last_undegraded":"2024-05-27T07:25:57.023543+0000","last_fullsized":"2024-05-27T07:25:57.023543+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:21:58.698952+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:21:58.698952+0000","last_clean_scrub_stamp":"2024-05-27T07:21:58.698952+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T16:53:51.397143+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,9,7,6],"acting":[11,9,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.12","version":"88'5","reported_seq":52,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-27T07:25:57.023142+0000","last_change":"2024-05-27T07:21:45.566986+0000","last_active":"2024-05-27T07:25:57.023142+0000","last_peered":"2024-05-27T07:25:57.023142+0000","last_clean":"2024-05-27T07:25:57.023142+0000","last_became_active":"2024-05-27T07:21:45.566579+0000","last_became_peered":"2024-05-27T07:21:45.566579+0000","last_unstale":"2024-05-27T07:25:57.023142+0000","last_undegraded":"2024-05-27T07:25:57.023142+0000","last_fullsized":"2024-05-27T07:25:57.023142+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:21:44.543638+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:21:44.543638+0000","last_clean_scrub_stamp":"2024-05-27T07:21:44.543638+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T17:08:14.034740+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":5,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"90'15","reported_seq":54,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-27T07:25:57.021914+0000","last_change":"2024-05-27T07:21:48.596575+0000","last_active":"2024-05-27T07:25:57.021914+0000","last_peered":"2024-05-27T07:25:57.021914+0000","last_clean":"2024-05-27T07:25:57.021914+0000","last_became_active":"2024-05-27T07:21:48.596215+0000","last_became_peered":"2024-05-27T07:21:48.596215+0000","last_unstale":"2024-05-27T07:25:57.021914+0000","last_undegraded":"2024-05-27T07:25:57.021914+0000","last_fullsized":"2024-05-27T07:25:57.021914+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:21:47.565641+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:21:47.565641+0000","last_clean_scrub_stamp":"2024-05-27T07:21:47.565641+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T14:57:05.795269+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":31,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-27T07:25:56.931967+0000","last_change":"2024-05-27T07:21:59.737690+0000","last_active":"2024-05-27T07:25:56.931967+0000","last_peered":"2024-05-27T07:25:56.931967+0000","last_clean":"2024-05-27T07:25:56.931967+0000","last_became_active":"2024-05-27T07:21:59.735646+0000","last_became_peered":"2024-05-27T07:21:59.735646+0000","last_unstale":"2024-05-27T07:25:56.931967+0000","last_undegraded":"2024-05-27T07:25:56.931967+0000","last_fullsized":"2024-05-27T07:25:56.931967+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:21:58.698952+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:21:58.698952+0000","last_clean_scrub_stamp":"2024-05-27T07:21:58.698952+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T13:54:41.227626+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,0,3],"acting":[9,7,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.11","version":"91'8","reported_seq":240,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-27T07:25:57.161569+0000","last_change":"2024-05-27T07:21:45.564836+0000","last_active":"2024-05-27T07:25:57.161569+0000","last_peered":"2024-05-27T07:25:57.161569+0000","last_clean":"2024-05-27T07:25:57.161569+0000","last_became_active":"2024-05-27T07:21:45.564481+0000","last_became_peered":"2024-05-27T07:21:45.564481+0000","last_unstale":"2024-05-27T07:25:57.161569+0000","last_undegraded":"2024-05-27T07:25:57.161569+0000","last_fullsized":"2024-05-27T07:25:57.161569+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:21:44.543638+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:21:44.543638+0000","last_clean_scrub_stamp":"2024-05-27T07:21:44.543638+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T15:13:12.012769+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1511,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":186,"num_read_kb":5,"num_write":8,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,10],"acting":[4,7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.10","version":"90'6","reported_seq":45,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-27T07:25:57.022004+0000","last_change":"2024-05-27T07:21:48.601331+0000","last_active":"2024-05-27T07:25:57.022004+0000","last_peered":"2024-05-27T07:25:57.022004+0000","last_clean":"2024-05-27T07:25:57.022004+0000","last_became_active":"2024-05-27T07:21:48.600917+0000","last_became_peered":"2024-05-27T07:21:48.600917+0000","last_unstale":"2024-05-27T07:25:57.022004+0000","last_undegraded":"2024-05-27T07:25:57.022004+0000","last_fullsized":"2024-05-27T07:25:57.022004+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:21:47.565641+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:21:47.565641+0000","last_clean_scrub_stamp":"2024-05-27T07:21:47.565641+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T17:26:42.129989+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":31,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-27T07:25:57.024036+0000","last_change":"2024-05-27T07:21:59.737662+0000","last_active":"2024-05-27T07:25:57.024036+0000","last_peered":"2024-05-27T07:25:57.024036+0000","last_clean":"2024-05-27T07:25:57.024036+0000","last_became_active":"2024-05-27T07:21:59.737229+0000","last_became_peered":"2024-05-27T07:21:59.737229+0000","last_unstale":"2024-05-27T07:25:57.024036+0000","last_undegraded":"2024-05-27T07:25:57.024036+0000","last_fullsized":"2024-05-27T07:25:57.024036+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:21:58.698952+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:21:58.698952+0000","last_clean_scrub_stamp":"2024-05-27T07:21:58.698952+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T15:32:31.147571+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,9,11],"acting":[6,3,9,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.10","version":"88'17","reported_seq":81,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-27T07:25:57.022607+0000","last_change":"2024-05-27T07:22:30.814815+0000","last_active":"2024-05-27T07:25:57.022607+0000","last_peered":"2024-05-27T07:25:57.022607+0000","last_clean":"2024-05-27T07:25:57.022607+0000","last_became_active":"2024-05-27T07:22:30.802828+0000","last_became_peered":"2024-05-27T07:22:30.802828+0000","last_unstale":"2024-05-27T07:25:57.022607+0000","last_undegraded":"2024-05-27T07:25:57.022607+0000","last_fullsized":"2024-05-27T07:25:57.022607+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:21:44.543638+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:21:44.543638+0000","last_clean_scrub_stamp":"2024-05-27T07:21:44.543638+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T11:38:07.363507+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,4],"acting":[8,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.11","version":"90'6","reported_seq":47,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-27T07:25:57.022823+0000","last_change":"2024-05-27T07:21:48.596204+0000","last_active":"2024-05-27T07:25:57.022823+0000","last_peered":"2024-05-27T07:25:57.022823+0000","last_clean":"2024-05-27T07:25:57.022823+0000","last_became_active":"2024-05-27T07:21:48.595827+0000","last_became_peered":"2024-05-27T07:21:48.595827+0000","last_unstale":"2024-05-27T07:25:57.022823+0000","last_undegraded":"2024-05-27T07:25:57.022823+0000","last_fullsized":"2024-05-27T07:25:57.022823+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:21:47.565641+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:21:47.565641+0000","last_clean_scrub_stamp":"2024-05-27T07:21:47.565641+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T15:51:53.162554+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":31,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-27T07:25:57.022694+0000","last_change":"2024-05-27T07:21:59.735837+0000","last_active":"2024-05-27T07:25:57.022694+0000","last_peered":"2024-05-27T07:25:57.022694+0000","last_clean":"2024-05-27T07:25:57.022694+0000","last_became_active":"2024-05-27T07:21:59.735063+0000","last_became_peered":"2024-05-27T07:21:59.735063+0000","last_unstale":"2024-05-27T07:25:57.022694+0000","last_undegraded":"2024-05-27T07:25:57.022694+0000","last_fullsized":"2024-05-27T07:25:57.022694+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:21:58.698952+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:21:58.698952+0000","last_clean_scrub_stamp":"2024-05-27T07:21:58.698952+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T13:04:21.751043+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,3,1],"acting":[5,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.f","version":"88'2","reported_seq":24,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-27T07:25:57.023668+0000","last_change":"2024-05-27T07:22:30.809491+0000","last_active":"2024-05-27T07:25:57.023668+0000","last_peered":"2024-05-27T07:25:57.023668+0000","last_clean":"2024-05-27T07:25:57.023668+0000","last_became_active":"2024-05-27T07:22:30.802187+0000","last_became_peered":"2024-05-27T07:22:30.802187+0000","last_unstale":"2024-05-27T07:25:57.023668+0000","last_undegraded":"2024-05-27T07:25:57.023668+0000","last_fullsized":"2024-05-27T07:25:57.023668+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:21:44.543638+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:21:44.543638+0000","last_clean_scrub_stamp":"2024-05-27T07:21:44.543638+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T18:11:32.821962+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":22,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,4],"acting":[11,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.e","version":"90'5","reported_seq":46,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-27T07:25:57.024123+0000","last_change":"2024-05-27T07:21:48.597362+0000","last_active":"2024-05-27T07:25:57.024123+0000","last_peered":"2024-05-27T07:25:57.024123+0000","last_clean":"2024-05-27T07:25:57.024123+0000","last_became_active":"2024-05-27T07:21:48.596920+0000","last_became_peered":"2024-05-27T07:21:48.596920+0000","last_unstale":"2024-05-27T07:25:57.024123+0000","last_undegraded":"2024-05-27T07:25:57.024123+0000","last_fullsized":"2024-05-27T07:25:57.024123+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:21:47.565641+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:21:47.565641+0000","last_clean_scrub_stamp":"2024-05-27T07:21:47.565641+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T16:50:21.021320+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,4],"acting":[11,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":31,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-27T07:25:57.022899+0000","last_change":"2024-05-27T07:21:59.732303+0000","last_active":"2024-05-27T07:25:57.022899+0000","last_peered":"2024-05-27T07:25:57.022899+0000","last_clean":"2024-05-27T07:25:57.022899+0000","last_became_active":"2024-05-27T07:21:59.731851+0000","last_became_peered":"2024-05-27T07:21:59.731851+0000","last_unstale":"2024-05-27T07:25:57.022899+0000","last_undegraded":"2024-05-27T07:25:57.022899+0000","last_fullsized":"2024-05-27T07:25:57.022899+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:21:58.698952+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:21:58.698952+0000","last_clean_scrub_stamp":"2024-05-27T07:21:58.698952+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T08:08:38.433818+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,10,11,9],"acting":[8,10,11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.4","version":"88'1","reported_seq":49,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-27T07:25:56.931533+0000","last_change":"2024-05-27T07:21:45.791370+0000","last_active":"2024-05-27T07:25:56.931533+0000","last_peered":"2024-05-27T07:25:56.931533+0000","last_clean":"2024-05-27T07:25:56.931533+0000","last_became_active":"2024-05-27T07:21:45.790987+0000","last_became_peered":"2024-05-27T07:21:45.790987+0000","last_unstale":"2024-05-27T07:25:56.931533+0000","last_undegraded":"2024-05-27T07:25:56.931533+0000","last_fullsized":"2024-05-27T07:25:56.931533+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:21:44.543638+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:21:44.543638+0000","last_clean_scrub_stamp":"2024-05-27T07:21:44.543638+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T14:20:18.570194+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2,0],"acting":[9,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.5","version":"90'4","reported_seq":45,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-27T07:25:57.023525+0000","last_change":"2024-05-27T07:21:48.585984+0000","last_active":"2024-05-27T07:25:57.023525+0000","last_peered":"2024-05-27T07:25:57.023525+0000","last_clean":"2024-05-27T07:25:57.023525+0000","last_became_active":"2024-05-27T07:21:48.585606+0000","last_became_peered":"2024-05-27T07:21:48.585606+0000","last_unstale":"2024-05-27T07:25:57.023525+0000","last_undegraded":"2024-05-27T07:25:57.023525+0000","last_fullsized":"2024-05-27T07:25:57.023525+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:21:47.565641+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:21:47.565641+0000","last_clean_scrub_stamp":"2024-05-27T07:21:47.565641+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T17:56:17.659024+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,3,6],"acting":[10,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":31,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-27T07:25:57.024779+0000","last_change":"2024-05-27T07:21:59.734022+0000","last_active":"2024-05-27T07:25:57.024779+0000","last_peered":"2024-05-27T07:25:57.024779+0000","last_clean":"2024-05-27T07:25:57.024779+0000","last_became_active":"2024-05-27T07:21:59.733467+0000","last_became_peered":"2024-05-27T07:21:59.733467+0000","last_unstale":"2024-05-27T07:25:57.024779+0000","last_undegraded":"2024-05-27T07:25:57.024779+0000","last_fullsized":"2024-05-27T07:25:57.024779+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:21:58.698952+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:21:58.698952+0000","last_clean_scrub_stamp":"2024-05-27T07:21:58.698952+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T09:54:11.058495+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,1,8],"acting":[6,0,1,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.2","version":"91'6","reported_seq":233,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-27T07:25:57.227437+0000","last_change":"2024-05-27T07:21:45.563072+0000","last_active":"2024-05-27T07:25:57.227437+0000","last_peered":"2024-05-27T07:25:57.227437+0000","last_clean":"2024-05-27T07:25:57.227437+0000","last_became_active":"2024-05-27T07:21:45.562632+0000","last_became_peered":"2024-05-27T07:21:45.562632+0000","last_unstale":"2024-05-27T07:25:57.227437+0000","last_undegraded":"2024-05-27T07:25:57.227437+0000","last_fullsized":"2024-05-27T07:25:57.227437+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:21:44.543638+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:21:44.543638+0000","last_clean_scrub_stamp":"2024-05-27T07:21:44.543638+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T12:54:19.555467+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":979,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":182,"num_read_kb":1,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3","version":"90'7","reported_seq":49,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-27T07:25:57.023358+0000","last_change":"2024-05-27T07:21:48.594867+0000","last_active":"2024-05-27T07:25:57.023358+0000","last_peered":"2024-05-27T07:25:57.023358+0000","last_clean":"2024-05-27T07:25:57.023358+0000","last_became_active":"2024-05-27T07:21:48.594523+0000","last_became_peered":"2024-05-27T07:21:48.594523+0000","last_unstale":"2024-05-27T07:25:57.023358+0000","last_undegraded":"2024-05-27T07:25:57.023358+0000","last_fullsized":"2024-05-27T07:25:57.023358+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:21:47.565641+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:21:47.565641+0000","last_clean_scrub_stamp":"2024-05-27T07:21:47.565641+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T17:28:55.862014+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,4,0],"acting":[10,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":31,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-27T07:25:57.024044+0000","last_change":"2024-05-27T07:21:59.741936+0000","last_active":"2024-05-27T07:25:57.024044+0000","last_peered":"2024-05-27T07:25:57.024044+0000","last_clean":"2024-05-27T07:25:57.024044+0000","last_became_active":"2024-05-27T07:21:59.741458+0000","last_became_peered":"2024-05-27T07:21:59.741458+0000","last_unstale":"2024-05-27T07:25:57.024044+0000","last_undegraded":"2024-05-27T07:25:57.024044+0000","last_fullsized":"2024-05-27T07:25:57.024044+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:21:58.698952+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:21:58.698952+0000","last_clean_scrub_stamp":"2024-05-27T07:21:58.698952+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T15:51:28.560995+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,10,0],"acting":[6,5,10,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.0","version":"90'8","reported_seq":26,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-27T07:25:56.930929+0000","last_change":"2024-05-27T07:22:30.803379+0000","last_active":"2024-05-27T07:25:56.930929+0000","last_peered":"2024-05-27T07:25:56.930929+0000","last_clean":"2024-05-27T07:25:56.930929+0000","last_became_active":"2024-05-27T07:22:30.803024+0000","last_became_peered":"2024-05-27T07:22:30.803024+0000","last_unstale":"2024-05-27T07:25:56.930929+0000","last_undegraded":"2024-05-27T07:25:56.930929+0000","last_fullsized":"2024-05-27T07:25:56.930929+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:21:47.565641+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:21:47.565641+0000","last_clean_scrub_stamp":"2024-05-27T07:21:47.565641+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T07:56:32.070610+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2,4],"acting":[9,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.1","version":"91'7","reported_seq":241,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-27T07:26:01.166673+0000","last_change":"2024-05-27T07:21:45.790849+0000","last_active":"2024-05-27T07:26:01.166673+0000","last_peered":"2024-05-27T07:26:01.166673+0000","last_clean":"2024-05-27T07:26:01.166673+0000","last_became_active":"2024-05-27T07:21:45.790435+0000","last_became_peered":"2024-05-27T07:21:45.790435+0000","last_unstale":"2024-05-27T07:26:01.166673+0000","last_undegraded":"2024-05-27T07:26:01.166673+0000","last_fullsized":"2024-05-27T07:26:01.166673+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:21:44.543638+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:21:44.543638+0000","last_clean_scrub_stamp":"2024-05-27T07:21:44.543638+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T09:34:51.479882+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":187,"num_read_kb":187,"num_write":7,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,0],"acting":[6,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.5","version":"88'1","reported_seq":31,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-27T07:25:57.021734+0000","last_change":"2024-05-27T07:21:59.731338+0000","last_active":"2024-05-27T07:25:57.021734+0000","last_peered":"2024-05-27T07:25:57.021734+0000","last_clean":"2024-05-27T07:25:57.021734+0000","last_became_active":"2024-05-27T07:21:59.730847+0000","last_became_peered":"2024-05-27T07:21:59.730847+0000","last_unstale":"2024-05-27T07:25:57.021734+0000","last_undegraded":"2024-05-27T07:25:57.021734+0000","last_fullsized":"2024-05-27T07:25:57.021734+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:21:58.698952+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:21:58.698952+0000","last_clean_scrub_stamp":"2024-05-27T07:21:58.698952+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T18:18:50.901129+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,11,9],"acting":[7,5,11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2","version":"90'8","reported_seq":49,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-27T07:25:57.022192+0000","last_change":"2024-05-27T07:21:48.594274+0000","last_active":"2024-05-27T07:25:57.022192+0000","last_peered":"2024-05-27T07:25:57.022192+0000","last_clean":"2024-05-27T07:25:57.022192+0000","last_became_active":"2024-05-27T07:21:48.593940+0000","last_became_peered":"2024-05-27T07:21:48.593940+0000","last_unstale":"2024-05-27T07:25:57.022192+0000","last_undegraded":"2024-05-27T07:25:57.022192+0000","last_fullsized":"2024-05-27T07:25:57.022192+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:21:47.565641+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:21:47.565641+0000","last_clean_scrub_stamp":"2024-05-27T07:21:47.565641+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T13:25:46.531524+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"88'6","reported_seq":57,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-27T07:25:57.023871+0000","last_change":"2024-05-27T07:21:45.563365+0000","last_active":"2024-05-27T07:25:57.023871+0000","last_peered":"2024-05-27T07:25:57.023871+0000","last_clean":"2024-05-27T07:25:57.023871+0000","last_became_active":"2024-05-27T07:21:45.563012+0000","last_became_peered":"2024-05-27T07:21:45.563012+0000","last_unstale":"2024-05-27T07:25:57.023871+0000","last_undegraded":"2024-05-27T07:25:57.023871+0000","last_fullsized":"2024-05-27T07:25:57.023871+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:21:44.543638+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:21:44.543638+0000","last_clean_scrub_stamp":"2024-05-27T07:21:44.543638+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T12:58:51.079965+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":68,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":4,"num_read_kb":4,"num_write":8,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,9],"acting":[4,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.0","version":"20'92","reported_seq":119,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-27T07:25:57.021608+0000","last_change":"2024-05-27T07:20:21.611212+0000","last_active":"2024-05-27T07:25:57.021608+0000","last_peered":"2024-05-27T07:25:57.021608+0000","last_clean":"2024-05-27T07:25:57.021608+0000","last_became_active":"2024-05-27T07:20:21.605163+0000","last_became_peered":"2024-05-27T07:20:21.605163+0000","last_unstale":"2024-05-27T07:25:57.021608+0000","last_undegraded":"2024-05-27T07:25:57.021608+0000","last_fullsized":"2024-05-27T07:25:57.021608+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:16:57.668433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:16:57.668433+0000","last_clean_scrub_stamp":"2024-05-27T07:16:57.668433+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T12:10:59.074162+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":81,"num_read_kb":69,"num_write":123,"num_write_kb":2150,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":2951840,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":31,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-27T07:25:57.023734+0000","last_change":"2024-05-27T07:21:59.737210+0000","last_active":"2024-05-27T07:25:57.023734+0000","last_peered":"2024-05-27T07:25:57.023734+0000","last_clean":"2024-05-27T07:25:57.023734+0000","last_became_active":"2024-05-27T07:21:59.736799+0000","last_became_peered":"2024-05-27T07:21:59.736799+0000","last_unstale":"2024-05-27T07:25:57.023734+0000","last_undegraded":"2024-05-27T07:25:57.023734+0000","last_fullsized":"2024-05-27T07:25:57.023734+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:21:58.698952+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:21:58.698952+0000","last_clean_scrub_stamp":"2024-05-27T07:21:58.698952+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T13:03:17.153687+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,6,10],"acting":[5,9,6,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1","version":"90'5","reported_seq":47,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-27T07:25:57.023957+0000","last_change":"2024-05-27T07:21:48.591550+0000","last_active":"2024-05-27T07:25:57.023957+0000","last_peered":"2024-05-27T07:25:57.023957+0000","last_clean":"2024-05-27T07:25:57.023957+0000","last_became_active":"2024-05-27T07:21:48.591223+0000","last_became_peered":"2024-05-27T07:21:48.591223+0000","last_unstale":"2024-05-27T07:25:57.023957+0000","last_undegraded":"2024-05-27T07:25:57.023957+0000","last_fullsized":"2024-05-27T07:25:57.023957+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:21:47.565641+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:21:47.565641+0000","last_clean_scrub_stamp":"2024-05-27T07:21:47.565641+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T15:06:23.395645+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,2],"acting":[6,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.0","version":"88'4","reported_seq":49,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-27T07:25:57.021833+0000","last_change":"2024-05-27T07:21:45.569224+0000","last_active":"2024-05-27T07:25:57.021833+0000","last_peered":"2024-05-27T07:25:57.021833+0000","last_clean":"2024-05-27T07:25:57.021833+0000","last_became_active":"2024-05-27T07:21:45.568818+0000","last_became_peered":"2024-05-27T07:21:45.568818+0000","last_unstale":"2024-05-27T07:25:57.021833+0000","last_undegraded":"2024-05-27T07:25:57.021833+0000","last_fullsized":"2024-05-27T07:25:57.021833+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:21:44.543638+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:21:44.543638+0000","last_clean_scrub_stamp":"2024-05-27T07:21:44.543638+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T08:12:33.157842+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,8],"acting":[7,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":31,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-27T07:25:57.023468+0000","last_change":"2024-05-27T07:21:59.733733+0000","last_active":"2024-05-27T07:25:57.023468+0000","last_peered":"2024-05-27T07:25:57.023468+0000","last_clean":"2024-05-27T07:25:57.023468+0000","last_became_active":"2024-05-27T07:21:59.733112+0000","last_became_peered":"2024-05-27T07:21:59.733112+0000","last_unstale":"2024-05-27T07:25:57.023468+0000","last_undegraded":"2024-05-27T07:25:57.023468+0000","last_fullsized":"2024-05-27T07:25:57.023468+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:21:58.698952+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:21:58.698952+0000","last_clean_scrub_stamp":"2024-05-27T07:21:58.698952+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T16:01:30.783800+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,7,6],"acting":[11,5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.5","version":"88'4","reported_seq":52,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-27T07:25:57.024039+0000","last_change":"2024-05-27T07:21:45.792029+0000","last_active":"2024-05-27T07:25:57.024039+0000","last_peered":"2024-05-27T07:25:57.024039+0000","last_clean":"2024-05-27T07:25:57.024039+0000","last_became_active":"2024-05-27T07:21:45.791492+0000","last_became_peered":"2024-05-27T07:21:45.791492+0000","last_unstale":"2024-05-27T07:25:57.024039+0000","last_undegraded":"2024-05-27T07:25:57.024039+0000","last_fullsized":"2024-05-27T07:25:57.024039+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:21:44.543638+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:21:44.543638+0000","last_clean_scrub_stamp":"2024-05-27T07:21:44.543638+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T17:36:42.444243+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,9],"acting":[8,0,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.4","version":"90'9","reported_seq":51,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-27T07:25:57.025118+0000","last_change":"2024-05-27T07:21:48.594769+0000","last_active":"2024-05-27T07:25:57.025118+0000","last_peered":"2024-05-27T07:25:57.025118+0000","last_clean":"2024-05-27T07:25:57.025118+0000","last_became_active":"2024-05-27T07:21:48.592979+0000","last_became_peered":"2024-05-27T07:21:48.592979+0000","last_unstale":"2024-05-27T07:25:57.025118+0000","last_undegraded":"2024-05-27T07:25:57.025118+0000","last_fullsized":"2024-05-27T07:25:57.025118+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:21:47.565641+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:21:47.565641+0000","last_clean_scrub_stamp":"2024-05-27T07:21:47.565641+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T10:06:48.268230+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,9],"acting":[6,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":31,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-27T07:25:57.021843+0000","last_change":"2024-05-27T07:21:59.736405+0000","last_active":"2024-05-27T07:25:57.021843+0000","last_peered":"2024-05-27T07:25:57.021843+0000","last_clean":"2024-05-27T07:25:57.021843+0000","last_became_active":"2024-05-27T07:21:59.735991+0000","last_became_peered":"2024-05-27T07:21:59.735991+0000","last_unstale":"2024-05-27T07:25:57.021843+0000","last_undegraded":"2024-05-27T07:25:57.021843+0000","last_fullsized":"2024-05-27T07:25:57.021843+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:21:58.698952+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:21:58.698952+0000","last_clean_scrub_stamp":"2024-05-27T07:21:58.698952+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T18:32:04.931725+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,10,0,7],"acting":[3,10,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"91'51","reported_seq":121,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-27T07:25:57.176466+0000","last_change":"2024-05-27T07:21:45.566513+0000","last_active":"2024-05-27T07:25:57.176466+0000","last_peered":"2024-05-27T07:25:57.176466+0000","last_clean":"2024-05-27T07:25:57.176466+0000","last_became_active":"2024-05-27T07:21:45.566107+0000","last_became_peered":"2024-05-27T07:21:45.566107+0000","last_unstale":"2024-05-27T07:25:57.176466+0000","last_undegraded":"2024-05-27T07:25:57.176466+0000","last_fullsized":"2024-05-27T07:25:57.176466+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:21:44.543638+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:21:44.543638+0000","last_clean_scrub_stamp":"2024-05-27T07:21:44.543638+0000","objects_scrubbed":0,"log_size":51,"log_dups_size":0,"ondisk_log_size":51,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T14:11:34.851691+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":393216,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":82,"num_read_kb":70,"num_write":91,"num_write_kb":838,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,3],"acting":[9,8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.7","version":"90'5","reported_seq":46,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-27T07:25:57.022710+0000","last_change":"2024-05-27T07:21:48.593642+0000","last_active":"2024-05-27T07:25:57.022710+0000","last_peered":"2024-05-27T07:25:57.022710+0000","last_clean":"2024-05-27T07:25:57.022710+0000","last_became_active":"2024-05-27T07:21:48.592885+0000","last_became_peered":"2024-05-27T07:21:48.592885+0000","last_unstale":"2024-05-27T07:25:57.022710+0000","last_undegraded":"2024-05-27T07:25:57.022710+0000","last_fullsized":"2024-05-27T07:25:57.022710+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:21:47.565641+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:21:47.565641+0000","last_clean_scrub_stamp":"2024-05-27T07:21:47.565641+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T11:51:27.186560+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,7],"acting":[3,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":31,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-27T07:25:56.935000+0000","last_change":"2024-05-27T07:21:59.725533+0000","last_active":"2024-05-27T07:25:56.935000+0000","last_peered":"2024-05-27T07:25:56.935000+0000","last_clean":"2024-05-27T07:25:56.935000+0000","last_became_active":"2024-05-27T07:21:59.724991+0000","last_became_peered":"2024-05-27T07:21:59.724991+0000","last_unstale":"2024-05-27T07:25:56.935000+0000","last_undegraded":"2024-05-27T07:25:56.935000+0000","last_fullsized":"2024-05-27T07:25:56.935000+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:21:58.698952+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:21:58.698952+0000","last_clean_scrub_stamp":"2024-05-27T07:21:58.698952+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T17:22:54.650553+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,4,7],"acting":[2,1,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"88'7","reported_seq":55,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-27T07:25:57.024242+0000","last_change":"2024-05-27T07:21:45.572701+0000","last_active":"2024-05-27T07:25:57.024242+0000","last_peered":"2024-05-27T07:25:57.024242+0000","last_clean":"2024-05-27T07:25:57.024242+0000","last_became_active":"2024-05-27T07:21:45.572176+0000","last_became_peered":"2024-05-27T07:21:45.572176+0000","last_unstale":"2024-05-27T07:25:57.024242+0000","last_undegraded":"2024-05-27T07:25:57.024242+0000","last_fullsized":"2024-05-27T07:25:57.024242+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:21:44.543638+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:21:44.543638+0000","last_clean_scrub_stamp":"2024-05-27T07:21:44.543638+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T16:11:20.901250+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,11,6],"acting":[4,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"90'7","reported_seq":49,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-27T07:25:57.026174+0000","last_change":"2024-05-27T07:21:48.581152+0000","last_active":"2024-05-27T07:25:57.026174+0000","last_peered":"2024-05-27T07:25:57.026174+0000","last_clean":"2024-05-27T07:25:57.026174+0000","last_became_active":"2024-05-27T07:21:48.580543+0000","last_became_peered":"2024-05-27T07:21:48.580543+0000","last_unstale":"2024-05-27T07:25:57.026174+0000","last_undegraded":"2024-05-27T07:25:57.026174+0000","last_fullsized":"2024-05-27T07:25:57.026174+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:21:47.565641+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:21:47.565641+0000","last_clean_scrub_stamp":"2024-05-27T07:21:47.565641+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T18:45:56.732828+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,4],"acting":[0,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":38,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-27T07:25:57.024155+0000","last_change":"2024-05-27T07:22:31.417863+0000","last_active":"2024-05-27T07:25:57.024155+0000","last_peered":"2024-05-27T07:25:57.024155+0000","last_clean":"2024-05-27T07:25:57.024155+0000","last_became_active":"2024-05-27T07:22:31.417298+0000","last_became_peered":"2024-05-27T07:22:31.417298+0000","last_unstale":"2024-05-27T07:25:57.024155+0000","last_undegraded":"2024-05-27T07:25:57.024155+0000","last_fullsized":"2024-05-27T07:25:57.024155+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:21:58.698952+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:21:58.698952+0000","last_clean_scrub_stamp":"2024-05-27T07:21:58.698952+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T12:26:13.857477+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,10,11],"acting":[4,5,10,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.8","version":"88'2","reported_seq":59,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-27T07:25:56.934912+0000","last_change":"2024-05-27T07:23:31.626374+0000","last_active":"2024-05-27T07:25:56.934912+0000","last_peered":"2024-05-27T07:25:56.934912+0000","last_clean":"2024-05-27T07:25:56.934912+0000","last_became_active":"2024-05-27T07:23:31.623214+0000","last_became_peered":"2024-05-27T07:23:31.623214+0000","last_unstale":"2024-05-27T07:25:56.934912+0000","last_undegraded":"2024-05-27T07:25:56.934912+0000","last_fullsized":"2024-05-27T07:25:56.934912+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:21:44.543638+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:21:44.543638+0000","last_clean_scrub_stamp":"2024-05-27T07:21:44.543638+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T10:42:47.710138+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,11,0],"acting":[1,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"90'2","reported_seq":43,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-27T07:25:57.022726+0000","last_change":"2024-05-27T07:21:48.589605+0000","last_active":"2024-05-27T07:25:57.022726+0000","last_peered":"2024-05-27T07:25:57.022726+0000","last_clean":"2024-05-27T07:25:57.022726+0000","last_became_active":"2024-05-27T07:21:48.588294+0000","last_became_peered":"2024-05-27T07:21:48.588294+0000","last_unstale":"2024-05-27T07:25:57.022726+0000","last_undegraded":"2024-05-27T07:25:57.022726+0000","last_fullsized":"2024-05-27T07:25:57.022726+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:21:47.565641+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:21:47.565641+0000","last_clean_scrub_stamp":"2024-05-27T07:21:47.565641+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T09:02:29.618508+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,10],"acting":[8,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":31,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-27T07:25:56.931975+0000","last_change":"2024-05-27T07:21:59.738153+0000","last_active":"2024-05-27T07:25:56.931975+0000","last_peered":"2024-05-27T07:25:56.931975+0000","last_clean":"2024-05-27T07:25:56.931975+0000","last_became_active":"2024-05-27T07:21:59.737394+0000","last_became_peered":"2024-05-27T07:21:59.737394+0000","last_unstale":"2024-05-27T07:25:56.931975+0000","last_undegraded":"2024-05-27T07:25:56.931975+0000","last_fullsized":"2024-05-27T07:25:56.931975+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:21:58.698952+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:21:58.698952+0000","last_clean_scrub_stamp":"2024-05-27T07:21:58.698952+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T17:17:13.877688+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,2,5],"acting":[9,3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.9","version":"88'7","reported_seq":53,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-27T07:25:57.021941+0000","last_change":"2024-05-27T07:21:45.562638+0000","last_active":"2024-05-27T07:25:57.021941+0000","last_peered":"2024-05-27T07:25:57.021941+0000","last_clean":"2024-05-27T07:25:57.021941+0000","last_became_active":"2024-05-27T07:21:45.562242+0000","last_became_peered":"2024-05-27T07:21:45.562242+0000","last_unstale":"2024-05-27T07:25:57.021941+0000","last_undegraded":"2024-05-27T07:25:57.021941+0000","last_fullsized":"2024-05-27T07:25:57.021941+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:21:44.543638+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:21:44.543638+0000","last_clean_scrub_stamp":"2024-05-27T07:21:44.543638+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T10:40:15.847513+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,9,10],"acting":[3,9,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.8","version":"90'9","reported_seq":50,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-27T07:25:57.022448+0000","last_change":"2024-05-27T07:21:48.598186+0000","last_active":"2024-05-27T07:25:57.022448+0000","last_peered":"2024-05-27T07:25:57.022448+0000","last_clean":"2024-05-27T07:25:57.022448+0000","last_became_active":"2024-05-27T07:21:48.597715+0000","last_became_peered":"2024-05-27T07:21:48.597715+0000","last_unstale":"2024-05-27T07:25:57.022448+0000","last_undegraded":"2024-05-27T07:25:57.022448+0000","last_fullsized":"2024-05-27T07:25:57.022448+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:21:47.565641+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:21:47.565641+0000","last_clean_scrub_stamp":"2024-05-27T07:21:47.565641+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T08:01:35.983687+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,2],"acting":[11,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.c","version":"88'1","reported_seq":32,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-27T07:25:57.023735+0000","last_change":"2024-05-27T07:21:59.727247+0000","last_active":"2024-05-27T07:25:57.023735+0000","last_peered":"2024-05-27T07:25:57.023735+0000","last_clean":"2024-05-27T07:25:57.023735+0000","last_became_active":"2024-05-27T07:21:59.726788+0000","last_became_peered":"2024-05-27T07:21:59.726788+0000","last_unstale":"2024-05-27T07:25:57.023735+0000","last_undegraded":"2024-05-27T07:25:57.023735+0000","last_fullsized":"2024-05-27T07:25:57.023735+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:21:58.698952+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:21:58.698952+0000","last_clean_scrub_stamp":"2024-05-27T07:21:58.698952+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T10:09:27.095188+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,9,7,8],"acting":[4,9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.a","version":"91'4","reported_seq":50,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-27T07:25:56.935299+0000","last_change":"2024-05-27T07:21:45.569285+0000","last_active":"2024-05-27T07:25:56.935299+0000","last_peered":"2024-05-27T07:25:56.935299+0000","last_clean":"2024-05-27T07:25:56.935299+0000","last_became_active":"2024-05-27T07:21:45.568835+0000","last_became_peered":"2024-05-27T07:21:45.568835+0000","last_unstale":"2024-05-27T07:25:56.935299+0000","last_undegraded":"2024-05-27T07:25:56.935299+0000","last_fullsized":"2024-05-27T07:25:56.935299+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:21:44.543638+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:21:44.543638+0000","last_clean_scrub_stamp":"2024-05-27T07:21:44.543638+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T10:03:23.353800+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,8],"acting":[2,3,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.b","version":"90'8","reported_seq":49,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-27T07:25:57.022876+0000","last_change":"2024-05-27T07:21:48.583494+0000","last_active":"2024-05-27T07:25:57.022876+0000","last_peered":"2024-05-27T07:25:57.022876+0000","last_clean":"2024-05-27T07:25:57.022876+0000","last_became_active":"2024-05-27T07:21:48.583141+0000","last_became_peered":"2024-05-27T07:21:48.583141+0000","last_unstale":"2024-05-27T07:25:57.022876+0000","last_undegraded":"2024-05-27T07:25:57.022876+0000","last_fullsized":"2024-05-27T07:25:57.022876+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:21:47.565641+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:21:47.565641+0000","last_clean_scrub_stamp":"2024-05-27T07:21:47.565641+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T10:24:50.475195+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,10],"acting":[3,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":31,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-27T07:25:57.022646+0000","last_change":"2024-05-27T07:21:59.733966+0000","last_active":"2024-05-27T07:25:57.022646+0000","last_peered":"2024-05-27T07:25:57.022646+0000","last_clean":"2024-05-27T07:25:57.022646+0000","last_became_active":"2024-05-27T07:21:59.733411+0000","last_became_peered":"2024-05-27T07:21:59.733411+0000","last_unstale":"2024-05-27T07:25:57.022646+0000","last_undegraded":"2024-05-27T07:25:57.022646+0000","last_fullsized":"2024-05-27T07:25:57.022646+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:21:58.698952+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:21:58.698952+0000","last_clean_scrub_stamp":"2024-05-27T07:21:58.698952+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T15:08:04.918086+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,10,3],"acting":[4,2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.b","version":"88'4","reported_seq":52,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-27T07:25:57.022886+0000","last_change":"2024-05-27T07:21:45.570967+0000","last_active":"2024-05-27T07:25:57.022886+0000","last_peered":"2024-05-27T07:25:57.022886+0000","last_clean":"2024-05-27T07:25:57.022886+0000","last_became_active":"2024-05-27T07:21:45.570501+0000","last_became_peered":"2024-05-27T07:21:45.570501+0000","last_unstale":"2024-05-27T07:25:57.022886+0000","last_undegraded":"2024-05-27T07:25:57.022886+0000","last_fullsized":"2024-05-27T07:25:57.022886+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:21:44.543638+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:21:44.543638+0000","last_clean_scrub_stamp":"2024-05-27T07:21:44.543638+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T14:11:26.496194+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.a","version":"90'5","reported_seq":44,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-27T07:25:57.022260+0000","last_change":"2024-05-27T07:21:48.594690+0000","last_active":"2024-05-27T07:25:57.022260+0000","last_peered":"2024-05-27T07:25:57.022260+0000","last_clean":"2024-05-27T07:25:57.022260+0000","last_became_active":"2024-05-27T07:21:48.594314+0000","last_became_peered":"2024-05-27T07:21:48.594314+0000","last_unstale":"2024-05-27T07:25:57.022260+0000","last_undegraded":"2024-05-27T07:25:57.022260+0000","last_fullsized":"2024-05-27T07:25:57.022260+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:21:47.565641+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:21:47.565641+0000","last_clean_scrub_stamp":"2024-05-27T07:21:47.565641+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T14:06:43.401844+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,6],"acting":[7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":31,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-27T07:25:56.934946+0000","last_change":"2024-05-27T07:21:59.732289+0000","last_active":"2024-05-27T07:25:56.934946+0000","last_peered":"2024-05-27T07:25:56.934946+0000","last_clean":"2024-05-27T07:25:56.934946+0000","last_became_active":"2024-05-27T07:21:59.731863+0000","last_became_peered":"2024-05-27T07:21:59.731863+0000","last_unstale":"2024-05-27T07:25:56.934946+0000","last_undegraded":"2024-05-27T07:25:56.934946+0000","last_fullsized":"2024-05-27T07:25:56.934946+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:21:58.698952+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:21:58.698952+0000","last_clean_scrub_stamp":"2024-05-27T07:21:58.698952+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T11:06:38.246484+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,6,7],"acting":[1,5,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.c","version":"91'6","reported_seq":235,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-27T07:25:57.161598+0000","last_change":"2024-05-27T07:21:45.791978+0000","last_active":"2024-05-27T07:25:57.161598+0000","last_peered":"2024-05-27T07:25:57.161598+0000","last_clean":"2024-05-27T07:25:57.161598+0000","last_became_active":"2024-05-27T07:21:45.791146+0000","last_became_peered":"2024-05-27T07:21:45.791146+0000","last_unstale":"2024-05-27T07:25:57.161598+0000","last_undegraded":"2024-05-27T07:25:57.161598+0000","last_fullsized":"2024-05-27T07:25:57.161598+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:21:44.543638+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:21:44.543638+0000","last_clean_scrub_stamp":"2024-05-27T07:21:44.543638+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T09:09:24.952876+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1561,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":183,"num_read_kb":2,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,11,0],"acting":[2,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"90'7","reported_seq":46,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-27T07:25:57.022770+0000","last_change":"2024-05-27T07:21:48.602786+0000","last_active":"2024-05-27T07:25:57.022770+0000","last_peered":"2024-05-27T07:25:57.022770+0000","last_clean":"2024-05-27T07:25:57.022770+0000","last_became_active":"2024-05-27T07:21:48.602476+0000","last_became_peered":"2024-05-27T07:21:48.602476+0000","last_unstale":"2024-05-27T07:25:57.022770+0000","last_undegraded":"2024-05-27T07:25:57.022770+0000","last_fullsized":"2024-05-27T07:25:57.022770+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:21:47.565641+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:21:47.565641+0000","last_clean_scrub_stamp":"2024-05-27T07:21:47.565641+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T13:22:58.583230+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,11],"acting":[7,1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":31,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-27T07:25:57.023937+0000","last_change":"2024-05-27T07:21:59.729595+0000","last_active":"2024-05-27T07:25:57.023937+0000","last_peered":"2024-05-27T07:25:57.023937+0000","last_clean":"2024-05-27T07:25:57.023937+0000","last_became_active":"2024-05-27T07:21:59.729147+0000","last_became_peered":"2024-05-27T07:21:59.729147+0000","last_unstale":"2024-05-27T07:25:57.023937+0000","last_undegraded":"2024-05-27T07:25:57.023937+0000","last_fullsized":"2024-05-27T07:25:57.023937+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:21:58.698952+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:21:58.698952+0000","last_clean_scrub_stamp":"2024-05-27T07:21:58.698952+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T13:31:28.579617+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,9,1,4],"acting":[0,9,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.d","version":"91'41","reported_seq":95,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-27T07:25:57.174009+0000","last_change":"2024-05-27T07:21:45.569287+0000","last_active":"2024-05-27T07:25:57.174009+0000","last_peered":"2024-05-27T07:25:57.174009+0000","last_clean":"2024-05-27T07:25:57.174009+0000","last_became_active":"2024-05-27T07:21:45.568935+0000","last_became_peered":"2024-05-27T07:21:45.568935+0000","last_unstale":"2024-05-27T07:25:57.174009+0000","last_undegraded":"2024-05-27T07:25:57.174009+0000","last_fullsized":"2024-05-27T07:25:57.174009+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:21:44.543638+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:21:44.543638+0000","last_clean_scrub_stamp":"2024-05-27T07:21:44.543638+0000","objects_scrubbed":0,"log_size":41,"log_dups_size":0,"ondisk_log_size":41,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T07:27:14.674135+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":131670,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":11,"num_read_kb":10,"num_write":49,"num_write_kb":475,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,10,3],"acting":[5,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.c","version":"90'6","reported_seq":45,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-27T07:25:57.021865+0000","last_change":"2024-05-27T07:21:48.597718+0000","last_active":"2024-05-27T07:25:57.021865+0000","last_peered":"2024-05-27T07:25:57.021865+0000","last_clean":"2024-05-27T07:25:57.021865+0000","last_became_active":"2024-05-27T07:21:48.597379+0000","last_became_peered":"2024-05-27T07:21:48.597379+0000","last_unstale":"2024-05-27T07:25:57.021865+0000","last_undegraded":"2024-05-27T07:25:57.021865+0000","last_fullsized":"2024-05-27T07:25:57.021865+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:21:47.565641+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:21:47.565641+0000","last_clean_scrub_stamp":"2024-05-27T07:21:47.565641+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T08:48:41.658364+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,5],"acting":[7,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":31,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-27T07:25:57.022372+0000","last_change":"2024-05-27T07:21:59.730191+0000","last_active":"2024-05-27T07:25:57.022372+0000","last_peered":"2024-05-27T07:25:57.022372+0000","last_clean":"2024-05-27T07:25:57.022372+0000","last_became_active":"2024-05-27T07:21:59.729787+0000","last_became_peered":"2024-05-27T07:21:59.729787+0000","last_unstale":"2024-05-27T07:25:57.022372+0000","last_undegraded":"2024-05-27T07:25:57.022372+0000","last_fullsized":"2024-05-27T07:25:57.022372+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:21:58.698952+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:21:58.698952+0000","last_clean_scrub_stamp":"2024-05-27T07:21:58.698952+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T13:22:38.961246+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7,1],"acting":[5,2,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.e","version":"88'5","reported_seq":51,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-27T07:25:57.024170+0000","last_change":"2024-05-27T07:21:45.559219+0000","last_active":"2024-05-27T07:25:57.024170+0000","last_peered":"2024-05-27T07:25:57.024170+0000","last_clean":"2024-05-27T07:25:57.024170+0000","last_became_active":"2024-05-27T07:21:45.558848+0000","last_became_peered":"2024-05-27T07:21:45.558848+0000","last_unstale":"2024-05-27T07:25:57.024170+0000","last_undegraded":"2024-05-27T07:25:57.024170+0000","last_fullsized":"2024-05-27T07:25:57.024170+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:21:44.543638+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:21:44.543638+0000","last_clean_scrub_stamp":"2024-05-27T07:21:44.543638+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T15:03:55.632162+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.f","version":"90'9","reported_seq":48,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-27T07:25:57.022988+0000","last_change":"2024-05-27T07:21:48.596349+0000","last_active":"2024-05-27T07:25:57.022988+0000","last_peered":"2024-05-27T07:25:57.022988+0000","last_clean":"2024-05-27T07:25:57.022988+0000","last_became_active":"2024-05-27T07:21:48.595830+0000","last_became_peered":"2024-05-27T07:21:48.595830+0000","last_unstale":"2024-05-27T07:25:57.022988+0000","last_undegraded":"2024-05-27T07:25:57.022988+0000","last_fullsized":"2024-05-27T07:25:57.022988+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:21:47.565641+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:21:47.565641+0000","last_clean_scrub_stamp":"2024-05-27T07:21:47.565641+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T13:33:11.431988+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,8],"acting":[7,11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.20","version":"90'9","reported_seq":50,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-27T07:25:57.022558+0000","last_change":"2024-05-27T07:21:48.598504+0000","last_active":"2024-05-27T07:25:57.022558+0000","last_peered":"2024-05-27T07:25:57.022558+0000","last_clean":"2024-05-27T07:25:57.022558+0000","last_became_active":"2024-05-27T07:21:48.598103+0000","last_became_peered":"2024-05-27T07:21:48.598103+0000","last_unstale":"2024-05-27T07:25:57.022558+0000","last_undegraded":"2024-05-27T07:25:57.022558+0000","last_fullsized":"2024-05-27T07:25:57.022558+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:21:47.565641+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:21:47.565641+0000","last_clean_scrub_stamp":"2024-05-27T07:21:47.565641+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T10:34:02.743735+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,5],"acting":[11,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.21","version":"90'12","reported_seq":53,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-27T07:25:56.935301+0000","last_change":"2024-05-27T07:21:48.593379+0000","last_active":"2024-05-27T07:25:56.935301+0000","last_peered":"2024-05-27T07:25:56.935301+0000","last_clean":"2024-05-27T07:25:56.935301+0000","last_became_active":"2024-05-27T07:21:48.592995+0000","last_became_peered":"2024-05-27T07:21:48.592995+0000","last_unstale":"2024-05-27T07:25:56.935301+0000","last_undegraded":"2024-05-27T07:25:56.935301+0000","last_fullsized":"2024-05-27T07:25:56.935301+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:21:47.565641+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:21:47.565641+0000","last_clean_scrub_stamp":"2024-05-27T07:21:47.565641+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T19:01:56.288235+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,4],"acting":[2,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"90'8","reported_seq":49,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-27T07:25:56.935066+0000","last_change":"2024-05-27T07:21:48.592837+0000","last_active":"2024-05-27T07:25:56.935066+0000","last_peered":"2024-05-27T07:25:56.935066+0000","last_clean":"2024-05-27T07:25:56.935066+0000","last_became_active":"2024-05-27T07:21:48.592459+0000","last_became_peered":"2024-05-27T07:21:48.592459+0000","last_unstale":"2024-05-27T07:25:56.935066+0000","last_undegraded":"2024-05-27T07:25:56.935066+0000","last_fullsized":"2024-05-27T07:25:56.935066+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:21:47.565641+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:21:47.565641+0000","last_clean_scrub_stamp":"2024-05-27T07:21:47.565641+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T14:00:19.490348+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.23","version":"90'7","reported_seq":49,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-27T07:25:57.025975+0000","last_change":"2024-05-27T07:21:48.583139+0000","last_active":"2024-05-27T07:25:57.025975+0000","last_peered":"2024-05-27T07:25:57.025975+0000","last_clean":"2024-05-27T07:25:57.025975+0000","last_became_active":"2024-05-27T07:21:48.582822+0000","last_became_peered":"2024-05-27T07:21:48.582822+0000","last_unstale":"2024-05-27T07:25:57.025975+0000","last_undegraded":"2024-05-27T07:25:57.025975+0000","last_fullsized":"2024-05-27T07:25:57.025975+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:21:47.565641+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:21:47.565641+0000","last_clean_scrub_stamp":"2024-05-27T07:21:47.565641+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T16:30:03.454035+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,4],"acting":[0,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"90'4","reported_seq":45,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-27T07:25:57.024904+0000","last_change":"2024-05-27T07:21:48.588906+0000","last_active":"2024-05-27T07:25:57.024904+0000","last_peered":"2024-05-27T07:25:57.024904+0000","last_clean":"2024-05-27T07:25:57.024904+0000","last_became_active":"2024-05-27T07:21:48.588555+0000","last_became_peered":"2024-05-27T07:21:48.588555+0000","last_unstale":"2024-05-27T07:25:57.024904+0000","last_undegraded":"2024-05-27T07:25:57.024904+0000","last_fullsized":"2024-05-27T07:25:57.024904+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:21:47.565641+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:21:47.565641+0000","last_clean_scrub_stamp":"2024-05-27T07:21:47.565641+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T19:04:23.006852+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"90'6","reported_seq":47,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-27T07:25:57.023930+0000","last_change":"2024-05-27T07:21:48.590271+0000","last_active":"2024-05-27T07:25:57.023930+0000","last_peered":"2024-05-27T07:25:57.023930+0000","last_clean":"2024-05-27T07:25:57.023930+0000","last_became_active":"2024-05-27T07:21:48.589898+0000","last_became_peered":"2024-05-27T07:21:48.589898+0000","last_unstale":"2024-05-27T07:25:57.023930+0000","last_undegraded":"2024-05-27T07:25:57.023930+0000","last_fullsized":"2024-05-27T07:25:57.023930+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:21:47.565641+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:21:47.565641+0000","last_clean_scrub_stamp":"2024-05-27T07:21:47.565641+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T11:57:43.238211+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,5],"acting":[11,8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.26","version":"90'15","reported_seq":56,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-27T07:25:56.935533+0000","last_change":"2024-05-27T07:21:48.597321+0000","last_active":"2024-05-27T07:25:56.935533+0000","last_peered":"2024-05-27T07:25:56.935533+0000","last_clean":"2024-05-27T07:25:56.935533+0000","last_became_active":"2024-05-27T07:21:48.596953+0000","last_became_peered":"2024-05-27T07:21:48.596953+0000","last_unstale":"2024-05-27T07:25:56.935533+0000","last_undegraded":"2024-05-27T07:25:56.935533+0000","last_fullsized":"2024-05-27T07:25:56.935533+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:21:47.565641+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:21:47.565641+0000","last_clean_scrub_stamp":"2024-05-27T07:21:47.565641+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T16:21:17.645098+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,8,1],"acting":[2,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.27","version":"90'2","reported_seq":43,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-27T07:25:57.022779+0000","last_change":"2024-05-27T07:21:48.593717+0000","last_active":"2024-05-27T07:25:57.022779+0000","last_peered":"2024-05-27T07:25:57.022779+0000","last_clean":"2024-05-27T07:25:57.022779+0000","last_became_active":"2024-05-27T07:21:48.593356+0000","last_became_peered":"2024-05-27T07:21:48.593356+0000","last_unstale":"2024-05-27T07:25:57.022779+0000","last_undegraded":"2024-05-27T07:25:57.022779+0000","last_fullsized":"2024-05-27T07:25:57.022779+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:21:47.565641+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:21:47.565641+0000","last_clean_scrub_stamp":"2024-05-27T07:21:47.565641+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T17:06:27.341627+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"90'6","reported_seq":47,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-27T07:25:57.022723+0000","last_change":"2024-05-27T07:21:48.592360+0000","last_active":"2024-05-27T07:25:57.022723+0000","last_peered":"2024-05-27T07:25:57.022723+0000","last_clean":"2024-05-27T07:25:57.022723+0000","last_became_active":"2024-05-27T07:21:48.591650+0000","last_became_peered":"2024-05-27T07:21:48.591650+0000","last_unstale":"2024-05-27T07:25:57.022723+0000","last_undegraded":"2024-05-27T07:25:57.022723+0000","last_fullsized":"2024-05-27T07:25:57.022723+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:21:47.565641+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:21:47.565641+0000","last_clean_scrub_stamp":"2024-05-27T07:21:47.565641+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T17:32:19.408787+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,3],"acting":[5,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"90'8","reported_seq":49,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-27T07:25:57.022029+0000","last_change":"2024-05-27T07:21:48.587319+0000","last_active":"2024-05-27T07:25:57.022029+0000","last_peered":"2024-05-27T07:25:57.022029+0000","last_clean":"2024-05-27T07:25:57.022029+0000","last_became_active":"2024-05-27T07:21:48.586924+0000","last_became_peered":"2024-05-27T07:21:48.586924+0000","last_unstale":"2024-05-27T07:25:57.022029+0000","last_undegraded":"2024-05-27T07:25:57.022029+0000","last_fullsized":"2024-05-27T07:25:57.022029+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:21:47.565641+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:21:47.565641+0000","last_clean_scrub_stamp":"2024-05-27T07:21:47.565641+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T09:28:48.129037+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,11],"acting":[3,2,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2a","version":"90'10","reported_seq":51,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-27T07:25:56.935355+0000","last_change":"2024-05-27T07:21:48.604018+0000","last_active":"2024-05-27T07:25:56.935355+0000","last_peered":"2024-05-27T07:25:56.935355+0000","last_clean":"2024-05-27T07:25:56.935355+0000","last_became_active":"2024-05-27T07:21:48.603368+0000","last_became_peered":"2024-05-27T07:21:48.603368+0000","last_unstale":"2024-05-27T07:25:56.935355+0000","last_undegraded":"2024-05-27T07:25:56.935355+0000","last_fullsized":"2024-05-27T07:25:56.935355+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:21:47.565641+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:21:47.565641+0000","last_clean_scrub_stamp":"2024-05-27T07:21:47.565641+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T09:49:42.440480+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2b","version":"90'11","reported_seq":52,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-27T07:25:57.024693+0000","last_change":"2024-05-27T07:21:48.589434+0000","last_active":"2024-05-27T07:25:57.024693+0000","last_peered":"2024-05-27T07:25:57.024693+0000","last_clean":"2024-05-27T07:25:57.024693+0000","last_became_active":"2024-05-27T07:21:48.589102+0000","last_became_peered":"2024-05-27T07:21:48.589102+0000","last_unstale":"2024-05-27T07:25:57.024693+0000","last_undegraded":"2024-05-27T07:25:57.024693+0000","last_fullsized":"2024-05-27T07:25:57.024693+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:21:47.565641+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:21:47.565641+0000","last_clean_scrub_stamp":"2024-05-27T07:21:47.565641+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T18:28:52.955138+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2c","version":"90'8","reported_seq":55,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-27T07:25:57.023752+0000","last_change":"2024-05-27T07:22:30.802433+0000","last_active":"2024-05-27T07:25:57.023752+0000","last_peered":"2024-05-27T07:25:57.023752+0000","last_clean":"2024-05-27T07:25:57.023752+0000","last_became_active":"2024-05-27T07:22:30.801881+0000","last_became_peered":"2024-05-27T07:22:30.801881+0000","last_unstale":"2024-05-27T07:25:57.023752+0000","last_undegraded":"2024-05-27T07:25:57.023752+0000","last_fullsized":"2024-05-27T07:25:57.023752+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:21:47.565641+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:21:47.565641+0000","last_clean_scrub_stamp":"2024-05-27T07:21:47.565641+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T11:59:17.916052+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,9,8],"acting":[11,9,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":234,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":32768,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":128,"acting":128,"num_store_stats":12},{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":500,"ondisk_log_size":500,"up":192,"acting":192,"num_store_stats":12},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":81,"num_read_kb":69,"num_write":123,"num_write_kb":2150,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":2951840,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":147456,"data_stored":1771104,"data_compressed":108480,"data_compressed_allocated":135168,"data_compressed_original":1769472,"omap_allocated":0,"internal_metadata":0},"log_size":92,"ondisk_log_size":92,"up":3,"acting":3,"num_store_stats":3},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":604805,"num_objects":100,"num_object_clones":0,"num_object_copies":300,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":100,"num_whiteouts":0,"num_read":1954,"num_read_kb":1099,"num_write":386,"num_write_kb":1479,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":22,"num_keys_recovered":0,"num_objects_omap":66,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":737280,"data_stored":1814415,"data_compressed":75906,"data_compressed_allocated":98304,"data_compressed_original":1548288,"omap_allocated":0,"internal_metadata":0},"log_size":311,"ondisk_log_size":311,"up":96,"acting":96,"num_store_stats":12}],"osd_stats":[{"osd":11,"up_from":71,"seq":304942678081,"num_pgs":29,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32168,"kb_used_data":988,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712984,"statfs":{"total":95995035648,"available":95962095616,"internally_reserved":0,"allocated":1011712,"data_stored":529010,"data_compressed":35393,"data_compressed_allocated":49152,"data_compressed_original":98304,"omap_allocated":1596,"internal_metadata":31914436},"hb_peers":[0,1,2,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":65,"seq":279172874309,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28160,"kb_used_data":1140,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716992,"statfs":{"total":95995035648,"available":95966199808,"internally_reserved":0,"allocated":1167360,"data_stored":1186884,"data_compressed":71553,"data_compressed_allocated":94208,"data_compressed_original":688128,"omap_allocated":1600,"internal_metadata":27654592},"hb_peers":[0,1,2,3,4,5,6,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":60,"seq":257698037835,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36372,"kb_used_data":1036,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708780,"statfs":{"total":95995035648,"available":95957790720,"internally_reserved":0,"allocated":1060864,"data_stored":926230,"data_compressed":54539,"data_compressed_allocated":73728,"data_compressed_original":491520,"omap_allocated":1600,"internal_metadata":36174272},"hb_peers":[0,1,2,3,4,5,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":54,"seq":231928234065,"num_pgs":31,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28044,"kb_used_data":1024,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717108,"statfs":{"total":95995035648,"available":95966318592,"internally_reserved":0,"allocated":1048576,"data_stored":922490,"data_compressed":54539,"data_compressed_allocated":73728,"data_compressed_original":491520,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":48,"seq":206158430293,"num_pgs":37,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32236,"kb_used_data":1056,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712916,"statfs":{"total":95995035648,"available":95962025984,"internally_reserved":0,"allocated":1081344,"data_stored":1127639,"data_compressed":71553,"data_compressed_allocated":94208,"data_compressed_original":688128,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":43,"seq":184683593819,"num_pgs":32,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28004,"kb_used_data":984,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717148,"statfs":{"total":95995035648,"available":95966359552,"internally_reserved":0,"allocated":1007616,"data_stored":524813,"data_compressed":35393,"data_compressed_allocated":49152,"data_compressed_original":98304,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574964,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36400,"kb_used_data":1060,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708752,"statfs":{"total":95995035648,"available":95957762048,"internally_reserved":0,"allocated":1085440,"data_stored":596782,"data_compressed":35393,"data_compressed_allocated":49152,"data_compressed_original":98304,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738490,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28028,"kb_used_data":1008,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717124,"statfs":{"total":95995035648,"available":95966334976,"internally_reserved":0,"allocated":1032192,"data_stored":526269,"data_compressed":35393,"data_compressed_allocated":49152,"data_compressed_original":98304,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411439,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36412,"kb_used_data":1072,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708740,"statfs":{"total":95995035648,"available":95957749760,"internally_reserved":0,"allocated":1097728,"data_stored":597813,"data_compressed":35393,"data_compressed_allocated":49152,"data_compressed_original":98304,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,1,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":26,"seq":111669149802,"num_pgs":40,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36376,"kb_used_data":1040,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708776,"statfs":{"total":95995035648,"available":95957786624,"internally_reserved":0,"allocated":1064960,"data_stored":1048946,"data_compressed":60695,"data_compressed_allocated":81920,"data_compressed_original":614400,"omap_allocated":1593,"internal_metadata":36174279},"hb_peers":[0,1,2,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":32,"seq":137438953572,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28020,"kb_used_data":1000,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717132,"statfs":{"total":95995035648,"available":95966343168,"internally_reserved":0,"allocated":1024000,"data_stored":529531,"data_compressed":35393,"data_compressed_allocated":49152,"data_compressed_original":98304,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":37,"seq":158913790047,"num_pgs":44,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28112,"kb_used_data":1084,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717040,"statfs":{"total":95995035648,"available":95966248960,"internally_reserved":0,"allocated":1110016,"data_stored":1253507,"data_compressed":77709,"data_compressed_allocated":102400,"data_compressed_original":811008,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":36160,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":36160,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":36160,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":2569,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":94208,"data_stored":73082,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":106496,"data_stored":74113,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":525246,"data_compressed":25302,"data_compressed_allocated":32768,"data_compressed_original":516096,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1735,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":135343,"data_compressed":6156,"data_compressed_allocated":8192,"data_compressed_original":122880,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1113,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":5379,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":394694,"data_compressed":19146,"data_compressed_allocated":24576,"data_compressed_original":393216,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":394338,"data_compressed":19146,"data_compressed_allocated":24576,"data_compressed_original":393216,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":147456,"data_stored":205198,"data_compressed":6156,"data_compressed_allocated":8192,"data_compressed_original":122880,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1605,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":8192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":8192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-05-27T07:26:04.065 DEBUG:teuthology.orchestra.run.smithi052:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:38f9d900c83c63799fbdbe61acc9a11b0d3554a6 shell --fsid 73df2636-1bf8-11ef-bc9b-c7b262605968 -- ceph fs dump --format=json 2024-05-27T07:26:04.351 INFO:teuthology.orchestra.run.smithi052.stderr:Inferring config /var/lib/ceph/73df2636-1bf8-11ef-bc9b-c7b262605968/mon.a/config 2024-05-27T07:26:04.440 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:26:04 smithi105 ceph-mon[35977]: pgmap v6: 129 pgs: 129 active+clean; 1.1 MiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 1.0 KiB/s rd, 1 op/s 2024-05-27T07:26:04.474 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:26:04 smithi052 ceph-mon[31955]: pgmap v6: 129 pgs: 129 active+clean; 1.1 MiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 1.0 KiB/s rd, 1 op/s 2024-05-27T07:26:04.602 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:26:04 smithi173 ceph-mon[37000]: pgmap v6: 129 pgs: 129 active+clean; 1.1 MiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 1.0 KiB/s rd, 1 op/s 2024-05-27T07:26:05.439 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:26:05 smithi105 ceph-mon[35977]: from='client.15441 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-05-27T07:26:05.440 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:26:05 smithi105 ceph-mon[35977]: from='client.15447 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-05-27T07:26:05.440 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:26:05 smithi105 ceph-mon[35977]: from='client.15453 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-05-27T07:26:05.479 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:26:05 smithi052 ceph-mon[31955]: from='client.15441 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-05-27T07:26:05.479 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:26:05 smithi052 ceph-mon[31955]: from='client.15447 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-05-27T07:26:05.479 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:26:05 smithi052 ceph-mon[31955]: from='client.15453 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-05-27T07:26:05.601 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:26:05 smithi173 ceph-mon[37000]: from='client.15441 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-05-27T07:26:05.602 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:26:05 smithi173 ceph-mon[37000]: from='client.15447 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-05-27T07:26:05.602 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:26:05 smithi173 ceph-mon[37000]: from='client.15453 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-05-27T07:26:06.183 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:06.192+0000 7fbc8a70a700 1 -- 172.21.15.52:0/2185423732 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fbc841034b0 msgr2=0x7fbc841038c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:26:06.183 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:06.192+0000 7fbc8a70a700 1 --2- 172.21.15.52:0/2185423732 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fbc841034b0 0x7fbc841038c0 secure :-1 s=READY pgs=420 cs=0 l=1 rev1=1 crypto rx=0x7fbc740099c0 tx=0x7fbc74009cd0 comp rx=0 tx=0).stop 2024-05-27T07:26:06.183 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:06.192+0000 7fbc8a70a700 1 -- 172.21.15.52:0/2185423732 shutdown_connections 2024-05-27T07:26:06.183 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:06.192+0000 7fbc8a70a700 1 --2- 172.21.15.52:0/2185423732 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fbc84108730 0x7fbc84110090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:26:06.183 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:06.192+0000 7fbc8a70a700 1 --2- 172.21.15.52:0/2185423732 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fbc841034b0 0x7fbc841038c0 unknown :-1 s=CLOSED pgs=420 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:26:06.183 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:06.192+0000 7fbc8a70a700 1 --2- 172.21.15.52:0/2185423732 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fbc84107de0 0x7fbc841081f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:26:06.183 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:06.192+0000 7fbc8a70a700 1 -- 172.21.15.52:0/2185423732 >> 172.21.15.52:0/2185423732 conn(0x7fbc840fede0 msgr2=0x7fbc84101200 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:26:06.184 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:06.193+0000 7fbc8a70a700 1 -- 172.21.15.52:0/2185423732 shutdown_connections 2024-05-27T07:26:06.184 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:06.193+0000 7fbc8a70a700 1 -- 172.21.15.52:0/2185423732 wait complete. 2024-05-27T07:26:06.184 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:06.193+0000 7fbc8a70a700 1 Processor -- start 2024-05-27T07:26:06.184 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:06.193+0000 7fbc8a70a700 1 -- start start 2024-05-27T07:26:06.185 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:06.194+0000 7fbc8a70a700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fbc841034b0 0x7fbc840ff2c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:26:06.186 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:06.194+0000 7fbc8a70a700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fbc84107de0 0x7fbc840ff800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:26:06.186 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:06.194+0000 7fbc8a70a700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fbc84108730 0x7fbc840ffd40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:26:06.186 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:06.194+0000 7fbc8a70a700 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x7fbc841133b0 con 0x7fbc841034b0 2024-05-27T07:26:06.186 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:06.194+0000 7fbc8a70a700 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7fbc84113230 con 0x7fbc84107de0 2024-05-27T07:26:06.186 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:06.194+0000 7fbc8a70a700 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7fbc84113530 con 0x7fbc84108730 2024-05-27T07:26:06.186 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:06.194+0000 7fbc83fff700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fbc841034b0 0x7fbc840ff2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:26:06.186 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:06.194+0000 7fbc83fff700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fbc841034b0 0x7fbc840ff2c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.52:58460/0 (socket says 172.21.15.52:58460) 2024-05-27T07:26:06.186 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:06.194+0000 7fbc83fff700 1 -- 172.21.15.52:0/3997099925 learned_addr learned my addr 172.21.15.52:0/3997099925 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-05-27T07:26:06.186 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:06.194+0000 7fbc7bfff700 1 --2- 172.21.15.52:0/3997099925 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fbc84107de0 0x7fbc840ff800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:26:06.186 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:06.194+0000 7fbc88ca7700 1 --2- 172.21.15.52:0/3997099925 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fbc84108730 0x7fbc840ffd40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:26:06.186 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:06.194+0000 7fbc83fff700 1 -- 172.21.15.52:0/3997099925 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fbc84108730 msgr2=0x7fbc840ffd40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:26:06.186 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:06.194+0000 7fbc83fff700 1 --2- 172.21.15.52:0/3997099925 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fbc84108730 0x7fbc840ffd40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:26:06.187 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:06.194+0000 7fbc83fff700 1 -- 172.21.15.52:0/3997099925 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fbc84107de0 msgr2=0x7fbc840ff800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:26:06.187 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:06.194+0000 7fbc83fff700 1 --2- 172.21.15.52:0/3997099925 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fbc84107de0 0x7fbc840ff800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:26:06.187 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:06.194+0000 7fbc83fff700 1 -- 172.21.15.52:0/3997099925 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbc841a91c0 con 0x7fbc841034b0 2024-05-27T07:26:06.187 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:06.194+0000 7fbc83fff700 1 --2- 172.21.15.52:0/3997099925 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fbc841034b0 0x7fbc840ff2c0 secure :-1 s=READY pgs=421 cs=0 l=1 rev1=1 crypto rx=0x7fbc7400b580 tx=0x7fbc740057c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:26:06.187 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:06.194+0000 7fbc81ffb700 1 -- 172.21.15.52:0/3997099925 <== mon.0 v2:172.21.15.52:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbc74026070 con 0x7fbc841034b0 2024-05-27T07:26:06.187 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:06.195+0000 7fbc81ffb700 1 -- 172.21.15.52:0/3997099925 <== mon.0 v2:172.21.15.52:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fbc74003f20 con 0x7fbc841034b0 2024-05-27T07:26:06.187 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:06.195+0000 7fbc8a70a700 1 -- 172.21.15.52:0/3997099925 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbc841a9450 con 0x7fbc841034b0 2024-05-27T07:26:06.187 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:06.195+0000 7fbc81ffb700 1 -- 172.21.15.52:0/3997099925 <== mon.0 v2:172.21.15.52:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbc7400fd00 con 0x7fbc841034b0 2024-05-27T07:26:06.187 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:06.195+0000 7fbc8a70a700 1 -- 172.21.15.52:0/3997099925 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbc841a98a0 con 0x7fbc841034b0 2024-05-27T07:26:06.187 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:06.196+0000 7fbc8a70a700 1 -- 172.21.15.52:0/3997099925 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fbc840512b0 con 0x7fbc841034b0 2024-05-27T07:26:06.188 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:06.197+0000 7fbc81ffb700 1 -- 172.21.15.52:0/3997099925 <== mon.0 v2:172.21.15.52:3300/0 4 ==== mgrmap(e 24) ==== 149879+0+0 (secure 0 0 0) 0x7fbc74029070 con 0x7fbc841034b0 2024-05-27T07:26:06.188 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:06.198+0000 7fbc81ffb700 1 --2- 172.21.15.52:0/3997099925 >> [v2:172.21.15.52:6800/1822318384,v1:172.21.15.52:6801/1822318384] conn(0x7fbc700b2b10 0x7fbc700b5020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:26:06.188 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:06.198+0000 7fbc7bfff700 1 --2- 172.21.15.52:0/3997099925 >> [v2:172.21.15.52:6800/1822318384,v1:172.21.15.52:6801/1822318384] conn(0x7fbc700b2b10 0x7fbc700b5020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:26:06.188 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:06.198+0000 7fbc81ffb700 1 -- 172.21.15.52:0/3997099925 <== mon.0 v2:172.21.15.52:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9379+0+0 (secure 0 0 0) 0x7fbc740e36d0 con 0x7fbc841034b0 2024-05-27T07:26:06.188 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:06.198+0000 7fbc7bfff700 1 --2- 172.21.15.52:0/3997099925 >> [v2:172.21.15.52:6800/1822318384,v1:172.21.15.52:6801/1822318384] conn(0x7fbc700b2b10 0x7fbc700b5020 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7fbc84107ca0 tx=0x7fbc6c00a300 comp rx=0 tx=0).ready entity=mgr.15333 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:26:06.192 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:06.203+0000 7fbc81ffb700 1 -- 172.21.15.52:0/3997099925 <== mon.0 v2:172.21.15.52:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191963 (secure 0 0 0) 0x7fbc74013070 con 0x7fbc841034b0 2024-05-27T07:26:06.440 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:26:06 smithi105 ceph-mon[35977]: pgmap v7: 129 pgs: 129 active+clean; 1.1 MiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 30 KiB/s rd, 1.0 KiB/s wr, 19 op/s 2024-05-27T07:26:06.451 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:06.462+0000 7fbc8a70a700 1 -- 172.21.15.52:0/3997099925 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fbc8402e7a0 con 0x7fbc841034b0 2024-05-27T07:26:06.452 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:26:06 smithi052 ceph-mon[31955]: pgmap v7: 129 pgs: 129 active+clean; 1.1 MiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 30 KiB/s rd, 1.0 KiB/s wr, 19 op/s 2024-05-27T07:26:06.452 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:06.463+0000 7fbc81ffb700 1 -- 172.21.15.52:0/3997099925 <== mon.0 v2:172.21.15.52:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 30 v30) ==== 94+0+11938 (secure 0 0 0) 0x7fbc740aab30 con 0x7fbc841034b0 2024-05-27T07:26:06.453 INFO:teuthology.orchestra.run.smithi052.stdout: 2024-05-27T07:26:06.453 INFO:teuthology.orchestra.run.smithi052.stdout:{"epoch":30,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14556,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.52:6835/2584733778","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.52:6834","nonce":2584733778},{"type":"v1","addr":"172.21.15.52:6835","nonce":2584733778}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14568,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.52:6837/675445303","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.52:6836","nonce":675445303},{"type":"v1","addr":"172.21.15.52:6837","nonce":675445303}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":30,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-05-27T07:21:50.521783+0000","modified":"2024-05-27T07:25:54.323921+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24527,"mds_1":24497,"mds_2":24475,"mds_3":24473,"mds_4":24466},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14619":{"gid":14619,"name":"g","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.52:6839/4258549559","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.52:6838","nonce":4258549559},{"type":"v1","addr":"172.21.15.52:6839","nonce":4258549559}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_14631":{"gid":14631,"name":"a","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.52:6841/2878618188","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.52:6840","nonce":2878618188},{"type":"v1","addr":"172.21.15.52:6841","nonce":2878618188}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24433":{"gid":24433,"name":"l","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.173:6833/3013589220","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6832","nonce":3013589220},{"type":"v1","addr":"172.21.15.173:6833","nonce":3013589220}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24452":{"gid":24452,"name":"k","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.105:6833/2800678412","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6832","nonce":2800678412},{"type":"v1","addr":"172.21.15.105:6833","nonce":2800678412}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24460":{"gid":24460,"name":"f","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.173:6835/1619486437","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6834","nonce":1619486437},{"type":"v1","addr":"172.21.15.173:6835","nonce":1619486437}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24466":{"gid":24466,"name":"c","rank":4,"incarnation":26,"state":"up:active","state_seq":25,"addr":"172.21.15.173:6837/1396363827","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6836","nonce":1396363827},{"type":"v1","addr":"172.21.15.173:6837","nonce":1396363827}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24473":{"gid":24473,"name":"e","rank":3,"incarnation":24,"state":"up:active","state_seq":26,"addr":"172.21.15.105:6835/3298498384","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6834","nonce":3298498384},{"type":"v1","addr":"172.21.15.105:6835","nonce":3298498384}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24475":{"gid":24475,"name":"i","rank":2,"incarnation":22,"state":"up:active","state_seq":23,"addr":"172.21.15.173:6839/564778293","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6838","nonce":564778293},{"type":"v1","addr":"172.21.15.173:6839","nonce":564778293}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24497":{"gid":24497,"name":"h","rank":1,"incarnation":20,"state":"up:active","state_seq":24,"addr":"172.21.15.105:6837/191393149","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6836","nonce":191393149},{"type":"v1","addr":"172.21.15.105:6837","nonce":191393149}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24527":{"gid":24527,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":5,"addr":"172.21.15.105:6839/500348151","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6838","nonce":500348151},{"type":"v1","addr":"172.21.15.105:6839","nonce":500348151}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24527,"qdb_cluster":[24527,24497,24475,24466,24473]},"id":1}]} 2024-05-27T07:26:06.455 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:06.465+0000 7fbc8a70a700 1 -- 172.21.15.52:0/3997099925 >> [v2:172.21.15.52:6800/1822318384,v1:172.21.15.52:6801/1822318384] conn(0x7fbc700b2b10 msgr2=0x7fbc700b5020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:26:06.455 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:06.465+0000 7fbc8a70a700 1 --2- 172.21.15.52:0/3997099925 >> [v2:172.21.15.52:6800/1822318384,v1:172.21.15.52:6801/1822318384] conn(0x7fbc700b2b10 0x7fbc700b5020 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7fbc84107ca0 tx=0x7fbc6c00a300 comp rx=0 tx=0).stop 2024-05-27T07:26:06.455 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:06.465+0000 7fbc8a70a700 1 -- 172.21.15.52:0/3997099925 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fbc841034b0 msgr2=0x7fbc840ff2c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:26:06.455 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:06.465+0000 7fbc8a70a700 1 --2- 172.21.15.52:0/3997099925 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fbc841034b0 0x7fbc840ff2c0 secure :-1 s=READY pgs=421 cs=0 l=1 rev1=1 crypto rx=0x7fbc7400b580 tx=0x7fbc740057c0 comp rx=0 tx=0).stop 2024-05-27T07:26:06.455 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:06.465+0000 7fbc8a70a700 1 -- 172.21.15.52:0/3997099925 shutdown_connections 2024-05-27T07:26:06.455 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:06.465+0000 7fbc8a70a700 1 --2- 172.21.15.52:0/3997099925 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fbc84108730 0x7fbc840ffd40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:26:06.455 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:06.465+0000 7fbc8a70a700 1 --2- 172.21.15.52:0/3997099925 >> [v2:172.21.15.52:6800/1822318384,v1:172.21.15.52:6801/1822318384] conn(0x7fbc700b2b10 0x7fbc700b5020 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:26:06.456 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:06.465+0000 7fbc8a70a700 1 --2- 172.21.15.52:0/3997099925 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fbc841034b0 0x7fbc840ff2c0 unknown :-1 s=CLOSED pgs=421 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:26:06.456 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:06.465+0000 7fbc8a70a700 1 --2- 172.21.15.52:0/3997099925 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fbc84107de0 0x7fbc840ff800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:26:06.456 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:06.465+0000 7fbc8a70a700 1 -- 172.21.15.52:0/3997099925 >> 172.21.15.52:0/3997099925 conn(0x7fbc840fede0 msgr2=0x7fbc84075e00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:26:06.456 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:06.465+0000 7fbc8a70a700 1 -- 172.21.15.52:0/3997099925 shutdown_connections 2024-05-27T07:26:06.456 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:06.465+0000 7fbc8a70a700 1 -- 172.21.15.52:0/3997099925 wait complete. 2024-05-27T07:26:06.456 INFO:teuthology.orchestra.run.smithi052.stderr:dumped fsmap epoch 30 2024-05-27T07:26:06.602 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:26:06 smithi173 ceph-mon[37000]: pgmap v7: 129 pgs: 129 active+clean; 1.1 MiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 30 KiB/s rd, 1.0 KiB/s wr, 19 op/s 2024-05-27T07:26:07.171 DEBUG:teuthology.orchestra.run.smithi052:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:38f9d900c83c63799fbdbe61acc9a11b0d3554a6 shell --fsid 73df2636-1bf8-11ef-bc9b-c7b262605968 -- ceph osd dump --format=json 2024-05-27T07:26:07.429 INFO:teuthology.orchestra.run.smithi052.stderr:Inferring config /var/lib/ceph/73df2636-1bf8-11ef-bc9b-c7b262605968/mon.a/config 2024-05-27T07:26:07.439 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:26:07 smithi105 ceph-mon[35977]: from='client.? 172.21.15.52:0/3997099925' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-27T07:26:07.479 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:26:07 smithi052 ceph-mon[31955]: from='client.? 172.21.15.52:0/3997099925' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-27T07:26:07.601 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:26:07 smithi173 ceph-mon[37000]: from='client.? 172.21.15.52:0/3997099925' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-27T07:26:08.440 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:26:08 smithi105 ceph-mon[35977]: pgmap v8: 129 pgs: 129 active+clean; 1.1 MiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 24 KiB/s rd, 797 B/s wr, 15 op/s 2024-05-27T07:26:08.479 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:26:08 smithi052 ceph-mon[31955]: pgmap v8: 129 pgs: 129 active+clean; 1.1 MiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 24 KiB/s rd, 797 B/s wr, 15 op/s 2024-05-27T07:26:08.601 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:26:08 smithi173 ceph-mon[37000]: pgmap v8: 129 pgs: 129 active+clean; 1.1 MiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 24 KiB/s rd, 797 B/s wr, 15 op/s 2024-05-27T07:26:09.229 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:09.239+0000 7fa805d8e700 1 -- 172.21.15.52:0/2978166062 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fa800108730 msgr2=0x7fa800110090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:26:09.229 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:09.239+0000 7fa805d8e700 1 --2- 172.21.15.52:0/2978166062 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fa800108730 0x7fa800110090 secure :-1 s=READY pgs=422 cs=0 l=1 rev1=1 crypto rx=0x7fa7f800b0d0 tx=0x7fa7f800b3e0 comp rx=0 tx=0).stop 2024-05-27T07:26:09.229 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:09.239+0000 7fa805d8e700 1 -- 172.21.15.52:0/2978166062 shutdown_connections 2024-05-27T07:26:09.229 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:09.239+0000 7fa805d8e700 1 --2- 172.21.15.52:0/2978166062 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fa800108730 0x7fa800110090 unknown :-1 s=CLOSED pgs=422 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:26:09.229 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:09.239+0000 7fa805d8e700 1 --2- 172.21.15.52:0/2978166062 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fa8001034b0 0x7fa8001038c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:26:09.229 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:09.239+0000 7fa805d8e700 1 --2- 172.21.15.52:0/2978166062 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fa800107de0 0x7fa8001081f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:26:09.229 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:09.239+0000 7fa805d8e700 1 -- 172.21.15.52:0/2978166062 >> 172.21.15.52:0/2978166062 conn(0x7fa8000fede0 msgr2=0x7fa800101200 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:26:09.230 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:09.239+0000 7fa805d8e700 1 -- 172.21.15.52:0/2978166062 shutdown_connections 2024-05-27T07:26:09.230 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:09.239+0000 7fa805d8e700 1 -- 172.21.15.52:0/2978166062 wait complete. 2024-05-27T07:26:09.232 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:09.240+0000 7fa805d8e700 1 Processor -- start 2024-05-27T07:26:09.232 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:09.240+0000 7fa805d8e700 1 -- start start 2024-05-27T07:26:09.232 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:09.240+0000 7fa805d8e700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fa8001034b0 0x7fa80019bd50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:26:09.232 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:09.240+0000 7fa805d8e700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fa800107de0 0x7fa80019c290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:26:09.232 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:09.240+0000 7fa805d8e700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fa800108730 0x7fa80019c810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:26:09.232 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:09.240+0000 7fa805d8e700 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x7fa800113450 con 0x7fa800108730 2024-05-27T07:26:09.232 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:09.240+0000 7fa805d8e700 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7fa8001132d0 con 0x7fa8001034b0 2024-05-27T07:26:09.232 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:09.240+0000 7fa805d8e700 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7fa8001135d0 con 0x7fa800107de0 2024-05-27T07:26:09.232 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:09.240+0000 7fa7fffff700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fa800108730 0x7fa80019c810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:26:09.233 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:09.240+0000 7fa7fffff700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fa800108730 0x7fa80019c810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.52:58468/0 (socket says 172.21.15.52:58468) 2024-05-27T07:26:09.233 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:09.240+0000 7fa7fffff700 1 -- 172.21.15.52:0/1204754086 learned_addr learned my addr 172.21.15.52:0/1204754086 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-05-27T07:26:09.233 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:09.241+0000 7fa7ff7fe700 1 --2- 172.21.15.52:0/1204754086 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fa8001034b0 0x7fa80019bd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:26:09.233 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:09.241+0000 7fa7feffd700 1 --2- 172.21.15.52:0/1204754086 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fa800107de0 0x7fa80019c290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:26:09.233 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:09.241+0000 7fa7fffff700 1 -- 172.21.15.52:0/1204754086 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fa800107de0 msgr2=0x7fa80019c290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:26:09.233 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:09.241+0000 7fa7fffff700 1 --2- 172.21.15.52:0/1204754086 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fa800107de0 0x7fa80019c290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:26:09.233 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:09.241+0000 7fa7fffff700 1 -- 172.21.15.52:0/1204754086 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fa8001034b0 msgr2=0x7fa80019bd50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:26:09.233 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:09.241+0000 7fa7fffff700 1 --2- 172.21.15.52:0/1204754086 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fa8001034b0 0x7fa80019bd50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:26:09.233 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:09.241+0000 7fa7fffff700 1 -- 172.21.15.52:0/1204754086 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa8001a70c0 con 0x7fa800108730 2024-05-27T07:26:09.233 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:09.241+0000 7fa7fffff700 1 --2- 172.21.15.52:0/1204754086 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fa800108730 0x7fa80019c810 secure :-1 s=READY pgs=423 cs=0 l=1 rev1=1 crypto rx=0x7fa7f8009fd0 tx=0x7fa7f8007ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:26:09.233 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:09.241+0000 7fa7fcff9700 1 -- 172.21.15.52:0/1204754086 <== mon.0 v2:172.21.15.52:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa7f801f070 con 0x7fa800108730 2024-05-27T07:26:09.234 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:09.241+0000 7fa805d8e700 1 -- 172.21.15.52:0/1204754086 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa8001a7350 con 0x7fa800108730 2024-05-27T07:26:09.234 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:09.241+0000 7fa7fcff9700 1 -- 172.21.15.52:0/1204754086 <== mon.0 v2:172.21.15.52:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fa7f8003c00 con 0x7fa800108730 2024-05-27T07:26:09.234 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:09.241+0000 7fa7fcff9700 1 -- 172.21.15.52:0/1204754086 <== mon.0 v2:172.21.15.52:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa7f801b8e0 con 0x7fa800108730 2024-05-27T07:26:09.234 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:09.241+0000 7fa805d8e700 1 -- 172.21.15.52:0/1204754086 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa8001a77a0 con 0x7fa800108730 2024-05-27T07:26:09.235 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:09.243+0000 7fa805d8e700 1 -- 172.21.15.52:0/1204754086 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa8000512b0 con 0x7fa800108730 2024-05-27T07:26:09.235 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:09.244+0000 7fa7fcff9700 1 -- 172.21.15.52:0/1204754086 <== mon.0 v2:172.21.15.52:3300/0 4 ==== mgrmap(e 24) ==== 149879+0+0 (secure 0 0 0) 0x7fa7f8022070 con 0x7fa800108730 2024-05-27T07:26:09.235 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:09.245+0000 7fa7fcff9700 1 --2- 172.21.15.52:0/1204754086 >> [v2:172.21.15.52:6800/1822318384,v1:172.21.15.52:6801/1822318384] conn(0x7fa7e80b2b10 0x7fa7e80b5020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:26:09.235 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:09.245+0000 7fa7ff7fe700 1 --2- 172.21.15.52:0/1204754086 >> [v2:172.21.15.52:6800/1822318384,v1:172.21.15.52:6801/1822318384] conn(0x7fa7e80b2b10 0x7fa7e80b5020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:26:09.235 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:09.245+0000 7fa7fcff9700 1 -- 172.21.15.52:0/1204754086 <== mon.0 v2:172.21.15.52:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9379+0+0 (secure 0 0 0) 0x7fa7f800f020 con 0x7fa800108730 2024-05-27T07:26:09.235 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:09.245+0000 7fa7ff7fe700 1 --2- 172.21.15.52:0/1204754086 >> [v2:172.21.15.52:6800/1822318384,v1:172.21.15.52:6801/1822318384] conn(0x7fa7e80b2b10 0x7fa7e80b5020 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7fa7f0006fd0 tx=0x7fa7f0008040 comp rx=0 tx=0).ready entity=mgr.15333 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:26:09.239 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:09.249+0000 7fa7fcff9700 1 -- 172.21.15.52:0/1204754086 <== mon.0 v2:172.21.15.52:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191963 (secure 0 0 0) 0x7fa7f8029020 con 0x7fa800108730 2024-05-27T07:26:09.461 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:09.472+0000 7fa805d8e700 1 -- 172.21.15.52:0/1204754086 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fa80002e5e0 con 0x7fa800108730 2024-05-27T07:26:09.463 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:09.473+0000 7fa7fcff9700 1 -- 172.21.15.52:0/1204754086 <== mon.0 v2:172.21.15.52:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v91) ==== 74+0+25078 (secure 0 0 0) 0x7fa7f80ab4c0 con 0x7fa800108730 2024-05-27T07:26:09.463 INFO:teuthology.orchestra.run.smithi052.stdout: 2024-05-27T07:26:09.464 INFO:teuthology.orchestra.run.smithi052.stdout:{"epoch":91,"fsid":"73df2636-1bf8-11ef-bc9b-c7b262605968","created":"2024-05-27T07:13:34.738032+0000","modified":"2024-05-27T07:25:56.925109+0000","last_up_change":"2024-05-27T07:20:45.777597+0000","last_in_change":"2024-05-27T07:20:25.314823+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-05-27T07:16:57.644034+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-05-27T07:21:43.917748+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"79","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-05-27T07:21:47.341876+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"79","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-05-27T07:21:57.707611+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"86","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"347b8a89-9dbe-4848-83b0-fd0e9374f33d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":82,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.52:6802","nonce":1626508899},{"type":"v1","addr":"172.21.15.52:6803","nonce":1626508899}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.52:6804","nonce":1626508899},{"type":"v1","addr":"172.21.15.52:6805","nonce":1626508899}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.52:6808","nonce":1626508899},{"type":"v1","addr":"172.21.15.52:6809","nonce":1626508899}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.52:6806","nonce":1626508899},{"type":"v1","addr":"172.21.15.52:6807","nonce":1626508899}]},"public_addr":"172.21.15.52:6803/1626508899","cluster_addr":"172.21.15.52:6805/1626508899","heartbeat_back_addr":"172.21.15.52:6809/1626508899","heartbeat_front_addr":"172.21.15.52:6807/1626508899","state":["exists","up"]},{"osd":1,"uuid":"7a3623d4-e087-40cb-9336-d4a5c52203d4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6800","nonce":2900093195},{"type":"v1","addr":"172.21.15.105:6801","nonce":2900093195}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6802","nonce":2900093195},{"type":"v1","addr":"172.21.15.105:6803","nonce":2900093195}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6806","nonce":2900093195},{"type":"v1","addr":"172.21.15.105:6807","nonce":2900093195}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6804","nonce":2900093195},{"type":"v1","addr":"172.21.15.105:6805","nonce":2900093195}]},"public_addr":"172.21.15.105:6801/2900093195","cluster_addr":"172.21.15.105:6803/2900093195","heartbeat_back_addr":"172.21.15.105:6807/2900093195","heartbeat_front_addr":"172.21.15.105:6805/2900093195","state":["exists","up"]},{"osd":2,"uuid":"41a9df5b-882a-47da-bb26-7f4d26ccf6c1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6800","nonce":2251189314},{"type":"v1","addr":"172.21.15.173:6801","nonce":2251189314}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6802","nonce":2251189314},{"type":"v1","addr":"172.21.15.173:6803","nonce":2251189314}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6806","nonce":2251189314},{"type":"v1","addr":"172.21.15.173:6807","nonce":2251189314}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6804","nonce":2251189314},{"type":"v1","addr":"172.21.15.173:6805","nonce":2251189314}]},"public_addr":"172.21.15.173:6801/2251189314","cluster_addr":"172.21.15.173:6803/2251189314","heartbeat_back_addr":"172.21.15.173:6807/2251189314","heartbeat_front_addr":"172.21.15.173:6805/2251189314","state":["exists","up"]},{"osd":3,"uuid":"f1245439-42a5-46c5-a998-e0a78f9df3fb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":26,"up_thru":82,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.52:6810","nonce":57920794},{"type":"v1","addr":"172.21.15.52:6811","nonce":57920794}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.52:6812","nonce":57920794},{"type":"v1","addr":"172.21.15.52:6813","nonce":57920794}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.52:6816","nonce":57920794},{"type":"v1","addr":"172.21.15.52:6817","nonce":57920794}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.52:6814","nonce":57920794},{"type":"v1","addr":"172.21.15.52:6815","nonce":57920794}]},"public_addr":"172.21.15.52:6811/57920794","cluster_addr":"172.21.15.52:6813/57920794","heartbeat_back_addr":"172.21.15.52:6817/57920794","heartbeat_front_addr":"172.21.15.52:6815/57920794","state":["exists","up"]},{"osd":4,"uuid":"2592d1cc-1726-4359-83d8-a2e6314b54ec","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":32,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6808","nonce":3441134818},{"type":"v1","addr":"172.21.15.105:6809","nonce":3441134818}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6810","nonce":3441134818},{"type":"v1","addr":"172.21.15.105:6811","nonce":3441134818}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6814","nonce":3441134818},{"type":"v1","addr":"172.21.15.105:6815","nonce":3441134818}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6812","nonce":3441134818},{"type":"v1","addr":"172.21.15.105:6813","nonce":3441134818}]},"public_addr":"172.21.15.105:6809/3441134818","cluster_addr":"172.21.15.105:6811/3441134818","heartbeat_back_addr":"172.21.15.105:6815/3441134818","heartbeat_front_addr":"172.21.15.105:6813/3441134818","state":["exists","up"]},{"osd":5,"uuid":"27e98ee2-4a92-4972-b2a3-959d0ed90981","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":37,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6808","nonce":2989239729},{"type":"v1","addr":"172.21.15.173:6809","nonce":2989239729}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6810","nonce":2989239729},{"type":"v1","addr":"172.21.15.173:6811","nonce":2989239729}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6814","nonce":2989239729},{"type":"v1","addr":"172.21.15.173:6815","nonce":2989239729}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6812","nonce":2989239729},{"type":"v1","addr":"172.21.15.173:6813","nonce":2989239729}]},"public_addr":"172.21.15.173:6809/2989239729","cluster_addr":"172.21.15.173:6811/2989239729","heartbeat_back_addr":"172.21.15.173:6815/2989239729","heartbeat_front_addr":"172.21.15.173:6813/2989239729","state":["exists","up"]},{"osd":6,"uuid":"2b0ec0cb-f438-4b2c-8642-0778ed374e9a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":43,"up_thru":82,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.52:6818","nonce":4244423065},{"type":"v1","addr":"172.21.15.52:6819","nonce":4244423065}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.52:6820","nonce":4244423065},{"type":"v1","addr":"172.21.15.52:6821","nonce":4244423065}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.52:6824","nonce":4244423065},{"type":"v1","addr":"172.21.15.52:6825","nonce":4244423065}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.52:6822","nonce":4244423065},{"type":"v1","addr":"172.21.15.52:6823","nonce":4244423065}]},"public_addr":"172.21.15.52:6819/4244423065","cluster_addr":"172.21.15.52:6821/4244423065","heartbeat_back_addr":"172.21.15.52:6825/4244423065","heartbeat_front_addr":"172.21.15.52:6823/4244423065","state":["exists","up"]},{"osd":7,"uuid":"cb2a7a6b-8015-43ea-865a-be144e45c51a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":48,"up_thru":82,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6816","nonce":2063611552},{"type":"v1","addr":"172.21.15.105:6817","nonce":2063611552}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6818","nonce":2063611552},{"type":"v1","addr":"172.21.15.105:6819","nonce":2063611552}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6822","nonce":2063611552},{"type":"v1","addr":"172.21.15.105:6823","nonce":2063611552}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6820","nonce":2063611552},{"type":"v1","addr":"172.21.15.105:6821","nonce":2063611552}]},"public_addr":"172.21.15.105:6817/2063611552","cluster_addr":"172.21.15.105:6819/2063611552","heartbeat_back_addr":"172.21.15.105:6823/2063611552","heartbeat_front_addr":"172.21.15.105:6821/2063611552","state":["exists","up"]},{"osd":8,"uuid":"c2fc565d-a0b9-4396-a209-bd884cddcfe3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":54,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6816","nonce":4131399852},{"type":"v1","addr":"172.21.15.173:6817","nonce":4131399852}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6818","nonce":4131399852},{"type":"v1","addr":"172.21.15.173:6819","nonce":4131399852}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6822","nonce":4131399852},{"type":"v1","addr":"172.21.15.173:6823","nonce":4131399852}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6820","nonce":4131399852},{"type":"v1","addr":"172.21.15.173:6821","nonce":4131399852}]},"public_addr":"172.21.15.173:6817/4131399852","cluster_addr":"172.21.15.173:6819/4131399852","heartbeat_back_addr":"172.21.15.173:6823/4131399852","heartbeat_front_addr":"172.21.15.173:6821/4131399852","state":["exists","up"]},{"osd":9,"uuid":"78fa7f17-1dad-4262-be40-31fa82e75520","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":60,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.52:6826","nonce":479119778},{"type":"v1","addr":"172.21.15.52:6827","nonce":479119778}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.52:6828","nonce":479119778},{"type":"v1","addr":"172.21.15.52:6829","nonce":479119778}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.52:6832","nonce":479119778},{"type":"v1","addr":"172.21.15.52:6833","nonce":479119778}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.52:6830","nonce":479119778},{"type":"v1","addr":"172.21.15.52:6831","nonce":479119778}]},"public_addr":"172.21.15.52:6827/479119778","cluster_addr":"172.21.15.52:6829/479119778","heartbeat_back_addr":"172.21.15.52:6833/479119778","heartbeat_front_addr":"172.21.15.52:6831/479119778","state":["exists","up"]},{"osd":10,"uuid":"e6f8e753-e15c-45b5-8ee9-d50f384559f9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":65,"up_thru":76,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6824","nonce":4032386628},{"type":"v1","addr":"172.21.15.105:6825","nonce":4032386628}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6826","nonce":4032386628},{"type":"v1","addr":"172.21.15.105:6827","nonce":4032386628}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6830","nonce":4032386628},{"type":"v1","addr":"172.21.15.105:6831","nonce":4032386628}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6828","nonce":4032386628},{"type":"v1","addr":"172.21.15.105:6829","nonce":4032386628}]},"public_addr":"172.21.15.105:6825/4032386628","cluster_addr":"172.21.15.105:6827/4032386628","heartbeat_back_addr":"172.21.15.105:6831/4032386628","heartbeat_front_addr":"172.21.15.105:6829/4032386628","state":["exists","up"]},{"osd":11,"uuid":"164572bf-67ed-4e8e-b32d-d7a2e8da607b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":71,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6824","nonce":1816308401},{"type":"v1","addr":"172.21.15.173:6825","nonce":1816308401}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6826","nonce":1816308401},{"type":"v1","addr":"172.21.15.173:6827","nonce":1816308401}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6830","nonce":1816308401},{"type":"v1","addr":"172.21.15.173:6831","nonce":1816308401}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6828","nonce":1816308401},{"type":"v1","addr":"172.21.15.173:6829","nonce":1816308401}]},"public_addr":"172.21.15.173:6825/1816308401","cluster_addr":"172.21.15.173:6827/1816308401","heartbeat_back_addr":"172.21.15.173:6831/1816308401","heartbeat_front_addr":"172.21.15.173:6829/1816308401","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-27T07:16:06.279220+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-27T07:16:29.668167+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-27T07:16:53.903808+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-27T07:17:18.787372+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-27T07:17:43.480784+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-27T07:18:08.816030+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-27T07:18:34.076148+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-27T07:19:00.283468+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-27T07:19:25.403023+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-27T07:19:52.139173+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-27T07:20:17.530304+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-27T07:20:43.665632+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.8","mappings":[{"from":2,"to":11}]},{"pgid":"2.f","mappings":[{"from":8,"to":11}]},{"pgid":"2.10","mappings":[{"from":2,"to":4}]},{"pgid":"3.0","mappings":[{"from":5,"to":9}]},{"pgid":"3.1b","mappings":[{"from":4,"to":8}]},{"pgid":"3.2c","mappings":[{"from":4,"to":9}]},{"pgid":"3.3c","mappings":[{"from":4,"to":9}]},{"pgid":"4.e","mappings":[{"from":9,"to":11}]},{"pgid":"4.12","mappings":[{"from":5,"to":10}]},{"pgid":"4.19","mappings":[{"from":9,"to":8}]},{"pgid":"4.1f","mappings":[{"from":9,"to":8}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.52:0/3430810618":"2024-05-28T07:25:56.925056+0000","172.21.15.52:0/2723769517":"2024-05-28T07:25:56.925056+0000","172.21.15.52:0/262308791":"2024-05-28T07:25:56.925056+0000","172.21.15.52:6801/1631683163":"2024-05-28T07:14:29.544633+0000","172.21.15.52:6801/3513075805":"2024-05-28T07:25:56.925056+0000","172.21.15.52:0/2776144659":"2024-05-28T07:14:29.544633+0000","172.21.15.52:0/3577886285":"2024-05-28T07:25:56.925056+0000","172.21.15.52:6800/1631683163":"2024-05-28T07:14:29.544633+0000","172.21.15.52:0/3675529097":"2024-05-28T07:14:01.960550+0000","172.21.15.52:0/727808107":"2024-05-28T07:14:01.960550+0000","172.21.15.52:0/3839436833":"2024-05-28T07:14:29.544633+0000","172.21.15.52:6800/587662512":"2024-05-28T07:14:01.960550+0000","172.21.15.52:6800/3513075805":"2024-05-28T07:25:56.925056+0000","172.21.15.52:6801/587662512":"2024-05-28T07:14:01.960550+0000","172.21.15.52:0/853867465":"2024-05-28T07:14:01.960550+0000","172.21.15.52:0/3480918279":"2024-05-28T07:14:29.544633+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-05-27T07:26:09.466 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:09.476+0000 7fa805d8e700 1 -- 172.21.15.52:0/1204754086 >> [v2:172.21.15.52:6800/1822318384,v1:172.21.15.52:6801/1822318384] conn(0x7fa7e80b2b10 msgr2=0x7fa7e80b5020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:26:09.466 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:09.476+0000 7fa805d8e700 1 --2- 172.21.15.52:0/1204754086 >> [v2:172.21.15.52:6800/1822318384,v1:172.21.15.52:6801/1822318384] conn(0x7fa7e80b2b10 0x7fa7e80b5020 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7fa7f0006fd0 tx=0x7fa7f0008040 comp rx=0 tx=0).stop 2024-05-27T07:26:09.466 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:09.476+0000 7fa805d8e700 1 -- 172.21.15.52:0/1204754086 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fa800108730 msgr2=0x7fa80019c810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:26:09.466 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:09.476+0000 7fa805d8e700 1 --2- 172.21.15.52:0/1204754086 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fa800108730 0x7fa80019c810 secure :-1 s=READY pgs=423 cs=0 l=1 rev1=1 crypto rx=0x7fa7f8009fd0 tx=0x7fa7f8007ad0 comp rx=0 tx=0).stop 2024-05-27T07:26:09.466 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:09.476+0000 7fa805d8e700 1 -- 172.21.15.52:0/1204754086 shutdown_connections 2024-05-27T07:26:09.466 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:09.476+0000 7fa805d8e700 1 --2- 172.21.15.52:0/1204754086 >> [v2:172.21.15.52:6800/1822318384,v1:172.21.15.52:6801/1822318384] conn(0x7fa7e80b2b10 0x7fa7e80b5020 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:26:09.466 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:09.476+0000 7fa805d8e700 1 --2- 172.21.15.52:0/1204754086 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fa800108730 0x7fa80019c810 unknown :-1 s=CLOSED pgs=423 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:26:09.466 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:09.476+0000 7fa805d8e700 1 --2- 172.21.15.52:0/1204754086 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fa8001034b0 0x7fa80019bd50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:26:09.467 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:09.476+0000 7fa805d8e700 1 --2- 172.21.15.52:0/1204754086 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fa800107de0 0x7fa80019c290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:26:09.467 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:09.476+0000 7fa805d8e700 1 -- 172.21.15.52:0/1204754086 >> 172.21.15.52:0/1204754086 conn(0x7fa8000fede0 msgr2=0x7fa800109690 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:26:09.467 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:09.476+0000 7fa805d8e700 1 -- 172.21.15.52:0/1204754086 shutdown_connections 2024-05-27T07:26:09.467 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:09.476+0000 7fa805d8e700 1 -- 172.21.15.52:0/1204754086 wait complete. 2024-05-27T07:26:10.200 DEBUG:tasks.fwd_scrub:Yielding 2024-05-27T07:26:10.200 INFO:teuthology.run_tasks:Running task exec... 2024-05-27T07:26:10.210 INFO:teuthology.task.exec:Executing custom commands... 2024-05-27T07:26:10.211 INFO:teuthology.task.exec:Running commands on role mon.a host ubuntu@smithi052.front.sepia.ceph.com 2024-05-27T07:26:10.211 DEBUG:teuthology.orchestra.run.smithi052:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'ceph config set mgr mgr/snap_schedule/log_level debug' 2024-05-27T07:26:10.211 INFO:tasks.fwd_scrub.fs.[cephfs]:start scrubbing fs: cephfs 2024-05-27T07:26:10.211 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-05-27T07:26:10.211 DEBUG:teuthology.orchestra.run.smithi052:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:38f9d900c83c63799fbdbe61acc9a11b0d3554a6 shell --fsid 73df2636-1bf8-11ef-bc9b-c7b262605968 -- ceph tell mds.1:0 scrub start / force,recursive 2024-05-27T07:26:10.338 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:10.348+0000 7f3c1e301640 1 -- 172.21.15.52:0/2584661573 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f3c1810b870 msgr2=0x7f3c1810dc90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:26:10.338 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:10.348+0000 7f3c1e301640 1 --2- 172.21.15.52:0/2584661573 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f3c1810b870 0x7f3c1810dc90 secure :-1 s=READY pgs=424 cs=0 l=1 rev1=1 crypto rx=0x7f3c0c009900 tx=0x7f3c0c02a510 comp rx=0 tx=0).stop 2024-05-27T07:26:10.338 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:10.348+0000 7f3c1e301640 1 -- 172.21.15.52:0/2584661573 shutdown_connections 2024-05-27T07:26:10.338 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:10.348+0000 7f3c1e301640 1 --2- 172.21.15.52:0/2584661573 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f3c1810e220 0x7f3c18110710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:26:10.338 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:10.348+0000 7f3c1e301640 1 --2- 172.21.15.52:0/2584661573 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f3c1810b870 0x7f3c1810dc90 unknown :-1 s=CLOSED pgs=424 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:26:10.338 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:10.348+0000 7f3c1e301640 1 --2- 172.21.15.52:0/2584661573 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f3c18104e10 0x7f3c18105220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:26:10.339 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:10.348+0000 7f3c1e301640 1 -- 172.21.15.52:0/2584661573 >> 172.21.15.52:0/2584661573 conn(0x7f3c180781a0 msgr2=0x7f3c18078600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:26:10.339 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:10.348+0000 7f3c1e301640 1 -- 172.21.15.52:0/2584661573 shutdown_connections 2024-05-27T07:26:10.339 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:10.348+0000 7f3c1e301640 1 -- 172.21.15.52:0/2584661573 wait complete. 2024-05-27T07:26:10.341 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:10.349+0000 7f3c1e301640 1 Processor -- start 2024-05-27T07:26:10.341 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:10.349+0000 7f3c1e301640 1 -- start start 2024-05-27T07:26:10.341 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:10.349+0000 7f3c1e301640 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f3c18104e10 0x7f3c181026d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:26:10.341 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:10.349+0000 7f3c1e301640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f3c1810b870 0x7f3c18102c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:26:10.341 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:10.349+0000 7f3c1e301640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f3c1810e220 0x7f3c18103150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:26:10.341 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:10.349+0000 7f3c1e301640 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x7f3c18113e80 con 0x7f3c18104e10 2024-05-27T07:26:10.341 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:10.349+0000 7f3c1e301640 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f3c18113d00 con 0x7f3c1810e220 2024-05-27T07:26:10.341 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:10.349+0000 7f3c1e301640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f3c18114000 con 0x7f3c1810b870 2024-05-27T07:26:10.342 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:10.349+0000 7f3c17fff640 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f3c18104e10 0x7f3c181026d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:26:10.342 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:10.349+0000 7f3c17fff640 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f3c18104e10 0x7f3c181026d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.52:58486/0 (socket says 172.21.15.52:58486) 2024-05-27T07:26:10.342 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:10.349+0000 7f3c17fff640 1 -- 172.21.15.52:0/2931251683 learned_addr learned my addr 172.21.15.52:0/2931251683 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-05-27T07:26:10.342 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:10.350+0000 7f3c177fe640 1 --2- 172.21.15.52:0/2931251683 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f3c1810b870 0x7f3c18102c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:26:10.342 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:10.350+0000 7f3c1c89e640 1 --2- 172.21.15.52:0/2931251683 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f3c1810e220 0x7f3c18103150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:26:10.342 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:10.350+0000 7f3c17fff640 1 -- 172.21.15.52:0/2931251683 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f3c1810b870 msgr2=0x7f3c18102c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:26:10.342 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:10.350+0000 7f3c17fff640 1 --2- 172.21.15.52:0/2931251683 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f3c1810b870 0x7f3c18102c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:26:10.342 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:10.350+0000 7f3c17fff640 1 -- 172.21.15.52:0/2931251683 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f3c1810e220 msgr2=0x7f3c18103150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:26:10.342 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:10.350+0000 7f3c17fff640 1 --2- 172.21.15.52:0/2931251683 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f3c1810e220 0x7f3c18103150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:26:10.342 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:10.350+0000 7f3c17fff640 1 -- 172.21.15.52:0/2931251683 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3c181a8c00 con 0x7f3c18104e10 2024-05-27T07:26:10.342 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:10.350+0000 7f3c1c89e640 1 --2- 172.21.15.52:0/2931251683 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f3c1810e220 0x7f3c18103150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-27T07:26:10.342 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:10.350+0000 7f3c17fff640 1 --2- 172.21.15.52:0/2931251683 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f3c18104e10 0x7f3c181026d0 secure :-1 s=READY pgs=425 cs=0 l=1 rev1=1 crypto rx=0x7f3c0800e9f0 tx=0x7f3c0800eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:26:10.343 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:10.350+0000 7f3c157fa640 1 -- 172.21.15.52:0/2931251683 <== mon.0 v2:172.21.15.52:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3c0800cd80 con 0x7f3c18104e10 2024-05-27T07:26:10.343 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:10.350+0000 7f3c157fa640 1 -- 172.21.15.52:0/2931251683 <== mon.0 v2:172.21.15.52:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f3c080108e0 con 0x7f3c18104e10 2024-05-27T07:26:10.343 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:10.350+0000 7f3c1e301640 1 -- 172.21.15.52:0/2931251683 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3c181a8ef0 con 0x7f3c18104e10 2024-05-27T07:26:10.343 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:10.350+0000 7f3c157fa640 1 -- 172.21.15.52:0/2931251683 <== mon.0 v2:172.21.15.52:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3c08018980 con 0x7f3c18104e10 2024-05-27T07:26:10.343 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:10.350+0000 7f3c1e301640 1 -- 172.21.15.52:0/2931251683 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3c181a9300 con 0x7f3c18104e10 2024-05-27T07:26:10.343 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:10.351+0000 7f3c1e301640 1 -- 172.21.15.52:0/2931251683 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3c18050860 con 0x7f3c18104e10 2024-05-27T07:26:10.343 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:10.354+0000 7f3c157fa640 1 -- 172.21.15.52:0/2931251683 <== mon.0 v2:172.21.15.52:3300/0 4 ==== mgrmap(e 24) ==== 149879+0+0 (secure 0 0 0) 0x7f3c08010430 con 0x7f3c18104e10 2024-05-27T07:26:10.344 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:10.354+0000 7f3c157fa640 1 --2- 172.21.15.52:0/2931251683 >> [v2:172.21.15.52:6800/1822318384,v1:172.21.15.52:6801/1822318384] conn(0x7f3c000b2a70 0x7f3c000b4f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:26:10.344 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:10.355+0000 7f3c177fe640 1 --2- 172.21.15.52:0/2931251683 >> [v2:172.21.15.52:6800/1822318384,v1:172.21.15.52:6801/1822318384] conn(0x7f3c000b2a70 0x7f3c000b4f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:26:10.344 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:10.355+0000 7f3c157fa640 1 -- 172.21.15.52:0/2931251683 <== mon.0 v2:172.21.15.52:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9379+0+0 (secure 0 0 0) 0x7f3c080e3530 con 0x7f3c18104e10 2024-05-27T07:26:10.345 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:10.355+0000 7f3c177fe640 1 --2- 172.21.15.52:0/2931251683 >> [v2:172.21.15.52:6800/1822318384,v1:172.21.15.52:6801/1822318384] conn(0x7f3c000b2a70 0x7f3c000b4f80 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f3c0c031440 tx=0x7f3c0c005d50 comp rx=0 tx=0).ready entity=mgr.15333 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:26:10.348 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:10.358+0000 7f3c157fa640 1 -- 172.21.15.52:0/2931251683 <== mon.0 v2:172.21.15.52:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191963 (secure 0 0 0) 0x7f3c080aa960 con 0x7f3c18104e10 2024-05-27T07:26:10.439 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:26:10 smithi105 ceph-mon[35977]: pgmap v9: 129 pgs: 129 active+clean; 1.1 MiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 22 KiB/s rd, 652 B/s wr, 17 op/s 2024-05-27T07:26:10.440 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:26:10 smithi105 ceph-mon[35977]: from='client.? 172.21.15.52:0/1204754086' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-05-27T07:26:10.471 INFO:teuthology.orchestra.run.smithi052.stderr:Inferring config /var/lib/ceph/73df2636-1bf8-11ef-bc9b-c7b262605968/mon.a/config 2024-05-27T07:26:10.479 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:26:10 smithi052 ceph-mon[31955]: pgmap v9: 129 pgs: 129 active+clean; 1.1 MiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 22 KiB/s rd, 652 B/s wr, 17 op/s 2024-05-27T07:26:10.479 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:26:10 smithi052 ceph-mon[31955]: from='client.? 172.21.15.52:0/1204754086' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-05-27T07:26:10.548 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:10.558+0000 7f3c1e301640 1 -- 172.21.15.52:0/2931251683 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_command([{prefix=config set, name=mgr/snap_schedule/log_level}] v 0) -- 0x7f3c181a96e0 con 0x7f3c18104e10 2024-05-27T07:26:10.552 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:10.562+0000 7f3c157fa640 1 -- 172.21.15.52:0/2931251683 <== mon.0 v2:172.21.15.52:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/snap_schedule/log_level}]=0 v29)=0 v29) ==== 129+0+0 (secure 0 0 0) 0x7f3c080aa260 con 0x7f3c18104e10 2024-05-27T07:26:10.555 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:10.565+0000 7f3c1e301640 1 -- 172.21.15.52:0/2931251683 >> [v2:172.21.15.52:6800/1822318384,v1:172.21.15.52:6801/1822318384] conn(0x7f3c000b2a70 msgr2=0x7f3c000b4f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:26:10.555 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:10.565+0000 7f3c1e301640 1 --2- 172.21.15.52:0/2931251683 >> [v2:172.21.15.52:6800/1822318384,v1:172.21.15.52:6801/1822318384] conn(0x7f3c000b2a70 0x7f3c000b4f80 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f3c0c031440 tx=0x7f3c0c005d50 comp rx=0 tx=0).stop 2024-05-27T07:26:10.555 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:10.565+0000 7f3c1e301640 1 -- 172.21.15.52:0/2931251683 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f3c18104e10 msgr2=0x7f3c181026d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:26:10.555 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:10.565+0000 7f3c1e301640 1 --2- 172.21.15.52:0/2931251683 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f3c18104e10 0x7f3c181026d0 secure :-1 s=READY pgs=425 cs=0 l=1 rev1=1 crypto rx=0x7f3c0800e9f0 tx=0x7f3c0800eeb0 comp rx=0 tx=0).stop 2024-05-27T07:26:10.555 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:10.565+0000 7f3c1e301640 1 -- 172.21.15.52:0/2931251683 shutdown_connections 2024-05-27T07:26:10.555 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:10.565+0000 7f3c1e301640 1 --2- 172.21.15.52:0/2931251683 >> [v2:172.21.15.52:6800/1822318384,v1:172.21.15.52:6801/1822318384] conn(0x7f3c000b2a70 0x7f3c000b4f80 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:26:10.555 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:10.565+0000 7f3c1e301640 1 --2- 172.21.15.52:0/2931251683 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f3c1810e220 0x7f3c18103150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:26:10.555 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:10.565+0000 7f3c1e301640 1 --2- 172.21.15.52:0/2931251683 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f3c1810b870 0x7f3c18102c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:26:10.555 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:10.565+0000 7f3c1e301640 1 --2- 172.21.15.52:0/2931251683 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f3c18104e10 0x7f3c181026d0 unknown :-1 s=CLOSED pgs=425 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:26:10.556 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:10.565+0000 7f3c1e301640 1 -- 172.21.15.52:0/2931251683 >> 172.21.15.52:0/2931251683 conn(0x7f3c180781a0 msgr2=0x7f3c181012d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:26:10.556 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:10.565+0000 7f3c1e301640 1 -- 172.21.15.52:0/2931251683 shutdown_connections 2024-05-27T07:26:10.556 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:10.565+0000 7f3c1e301640 1 -- 172.21.15.52:0/2931251683 wait complete. 2024-05-27T07:26:10.568 DEBUG:teuthology.orchestra.run.smithi052:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'ceph config set mgr mgr/snap_schedule/allow_m_granularity true' 2024-05-27T07:26:10.602 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:26:10 smithi173 ceph-mon[37000]: pgmap v9: 129 pgs: 129 active+clean; 1.1 MiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 22 KiB/s rd, 652 B/s wr, 17 op/s 2024-05-27T07:26:10.602 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:26:10 smithi173 ceph-mon[37000]: from='client.? 172.21.15.52:0/1204754086' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-05-27T07:26:10.688 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:10.698+0000 7f73b8de3640 1 -- 172.21.15.52:0/2267871771 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f73b410e200 msgr2=0x7f73b41106f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:26:10.688 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:10.698+0000 7f73b8de3640 1 --2- 172.21.15.52:0/2267871771 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f73b410e200 0x7f73b41106f0 secure :-1 s=READY pgs=426 cs=0 l=1 rev1=1 crypto rx=0x7f73ac009900 tx=0x7f73ac02a510 comp rx=0 tx=0).stop 2024-05-27T07:26:10.688 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:10.698+0000 7f73b8de3640 1 -- 172.21.15.52:0/2267871771 shutdown_connections 2024-05-27T07:26:10.688 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:10.698+0000 7f73b8de3640 1 --2- 172.21.15.52:0/2267871771 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f73b410e200 0x7f73b41106f0 unknown :-1 s=CLOSED pgs=426 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:26:10.688 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:10.698+0000 7f73b8de3640 1 --2- 172.21.15.52:0/2267871771 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f73b410b850 0x7f73b410dc70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:26:10.688 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:10.698+0000 7f73b8de3640 1 --2- 172.21.15.52:0/2267871771 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f73b4104df0 0x7f73b4105200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:26:10.688 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:10.698+0000 7f73b8de3640 1 -- 172.21.15.52:0/2267871771 >> 172.21.15.52:0/2267871771 conn(0x7f73b40782d0 msgr2=0x7f73b4078710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:26:10.688 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:10.698+0000 7f73b8de3640 1 -- 172.21.15.52:0/2267871771 shutdown_connections 2024-05-27T07:26:10.689 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:10.698+0000 7f73b8de3640 1 -- 172.21.15.52:0/2267871771 wait complete. 2024-05-27T07:26:10.691 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:10.699+0000 7f73b8de3640 1 Processor -- start 2024-05-27T07:26:10.691 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:10.699+0000 7f73b8de3640 1 -- start start 2024-05-27T07:26:10.691 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:10.699+0000 7f73b8de3640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f73b4104df0 0x7f73b41a0500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:26:10.691 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:10.699+0000 7f73b8de3640 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f73b410b850 0x7f73b41a0a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:26:10.691 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:10.699+0000 7f73b8de3640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f73b410e200 0x7f73b419a680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:26:10.691 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:10.699+0000 7f73b8de3640 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x7f73b4113e60 con 0x7f73b410b850 2024-05-27T07:26:10.691 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:10.699+0000 7f73b8de3640 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f73b4113ce0 con 0x7f73b4104df0 2024-05-27T07:26:10.691 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:10.699+0000 7f73b8de3640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f73b4113fe0 con 0x7f73b410e200 2024-05-27T07:26:10.691 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:10.699+0000 7f73b1d9b640 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f73b410b850 0x7f73b41a0a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:26:10.691 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:10.699+0000 7f73b1d9b640 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f73b410b850 0x7f73b41a0a40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.52:49696/0 (socket says 172.21.15.52:49696) 2024-05-27T07:26:10.691 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:10.699+0000 7f73b1d9b640 1 -- 172.21.15.52:0/4201894600 learned_addr learned my addr 172.21.15.52:0/4201894600 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-05-27T07:26:10.692 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:10.699+0000 7f73b259c640 1 --2- 172.21.15.52:0/4201894600 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f73b4104df0 0x7f73b41a0500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:26:10.692 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:10.700+0000 7f73b2d9d640 1 --2- 172.21.15.52:0/4201894600 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f73b410e200 0x7f73b419a680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:26:10.692 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:10.700+0000 7f73b1d9b640 1 -- 172.21.15.52:0/4201894600 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f73b410e200 msgr2=0x7f73b419a680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:26:10.692 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:10.700+0000 7f73b1d9b640 1 --2- 172.21.15.52:0/4201894600 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f73b410e200 0x7f73b419a680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:26:10.692 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:10.700+0000 7f73b1d9b640 1 -- 172.21.15.52:0/4201894600 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f73b4104df0 msgr2=0x7f73b41a0500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:26:10.692 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:10.700+0000 7f73b1d9b640 1 --2- 172.21.15.52:0/4201894600 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f73b4104df0 0x7f73b41a0500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:26:10.692 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:10.700+0000 7f73b1d9b640 1 -- 172.21.15.52:0/4201894600 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f73b419adb0 con 0x7f73b410b850 2024-05-27T07:26:10.692 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:10.700+0000 7f73b1d9b640 1 --2- 172.21.15.52:0/4201894600 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f73b410b850 0x7f73b41a0a40 secure :-1 s=READY pgs=427 cs=0 l=1 rev1=1 crypto rx=0x7f73a800ebc0 tx=0x7f73a800c6a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:26:10.692 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:10.700+0000 7f73a37fe640 1 -- 172.21.15.52:0/4201894600 <== mon.0 v2:172.21.15.52:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f73a8010630 con 0x7f73b410b850 2024-05-27T07:26:10.692 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:10.700+0000 7f73a37fe640 1 -- 172.21.15.52:0/4201894600 <== mon.0 v2:172.21.15.52:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f73a8010c80 con 0x7f73b410b850 2024-05-27T07:26:10.692 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:10.700+0000 7f73b8de3640 1 -- 172.21.15.52:0/4201894600 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f73b419b040 con 0x7f73b410b850 2024-05-27T07:26:10.692 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:10.700+0000 7f73a37fe640 1 -- 172.21.15.52:0/4201894600 <== mon.0 v2:172.21.15.52:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f73a8018b60 con 0x7f73b410b850 2024-05-27T07:26:10.692 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:10.700+0000 7f73b8de3640 1 -- 172.21.15.52:0/4201894600 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f73b41a72a0 con 0x7f73b410b850 2024-05-27T07:26:10.692 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:10.701+0000 7f73b8de3640 1 -- 172.21.15.52:0/4201894600 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f73b406ad60 con 0x7f73b410b850 2024-05-27T07:26:10.693 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:10.704+0000 7f73a37fe640 1 -- 172.21.15.52:0/4201894600 <== mon.0 v2:172.21.15.52:3300/0 4 ==== mgrmap(e 24) ==== 149879+0+0 (secure 0 0 0) 0x7f73a8018d00 con 0x7f73b410b850 2024-05-27T07:26:10.694 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:10.704+0000 7f73a37fe640 1 --2- 172.21.15.52:0/4201894600 >> [v2:172.21.15.52:6800/1822318384,v1:172.21.15.52:6801/1822318384] conn(0x7f739c0b2a70 0x7f739c0b4f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:26:10.694 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:10.704+0000 7f73b259c640 1 --2- 172.21.15.52:0/4201894600 >> [v2:172.21.15.52:6800/1822318384,v1:172.21.15.52:6801/1822318384] conn(0x7f739c0b2a70 0x7f739c0b4f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:26:10.694 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:10.704+0000 7f73a37fe640 1 -- 172.21.15.52:0/4201894600 <== mon.0 v2:172.21.15.52:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9379+0+0 (secure 0 0 0) 0x7f73a80e3730 con 0x7f73b410b850 2024-05-27T07:26:10.694 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:10.705+0000 7f73b259c640 1 --2- 172.21.15.52:0/4201894600 >> [v2:172.21.15.52:6800/1822318384,v1:172.21.15.52:6801/1822318384] conn(0x7f739c0b2a70 0x7f739c0b4f80 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f73a4005fd0 tx=0x7f73a400b390 comp rx=0 tx=0).ready entity=mgr.15333 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:26:10.698 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:10.708+0000 7f73a37fe640 1 -- 172.21.15.52:0/4201894600 <== mon.0 v2:172.21.15.52:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191963 (secure 0 0 0) 0x7f73a80aab60 con 0x7f73b410b850 2024-05-27T07:26:10.898 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:10.908+0000 7f73b8de3640 1 -- 172.21.15.52:0/4201894600 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_command([{prefix=config set, name=mgr/snap_schedule/allow_m_granularity}] v 0) -- 0x7f73b41a7470 con 0x7f73b410b850 2024-05-27T07:26:10.902 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:10.912+0000 7f73a37fe640 1 -- 172.21.15.52:0/4201894600 <== mon.0 v2:172.21.15.52:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/snap_schedule/allow_m_granularity}]=0 v30)=0 v30) ==== 138+0+0 (secure 0 0 0) 0x7f73a80aa460 con 0x7f73b410b850 2024-05-27T07:26:10.905 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:10.914+0000 7f73b8de3640 1 -- 172.21.15.52:0/4201894600 >> [v2:172.21.15.52:6800/1822318384,v1:172.21.15.52:6801/1822318384] conn(0x7f739c0b2a70 msgr2=0x7f739c0b4f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:26:10.905 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:10.914+0000 7f73b8de3640 1 --2- 172.21.15.52:0/4201894600 >> [v2:172.21.15.52:6800/1822318384,v1:172.21.15.52:6801/1822318384] conn(0x7f739c0b2a70 0x7f739c0b4f80 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f73a4005fd0 tx=0x7f73a400b390 comp rx=0 tx=0).stop 2024-05-27T07:26:10.905 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:10.914+0000 7f73b8de3640 1 -- 172.21.15.52:0/4201894600 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f73b410b850 msgr2=0x7f73b41a0a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:26:10.905 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:10.914+0000 7f73b8de3640 1 --2- 172.21.15.52:0/4201894600 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f73b410b850 0x7f73b41a0a40 secure :-1 s=READY pgs=427 cs=0 l=1 rev1=1 crypto rx=0x7f73a800ebc0 tx=0x7f73a800c6a0 comp rx=0 tx=0).stop 2024-05-27T07:26:10.905 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:10.914+0000 7f73b8de3640 1 -- 172.21.15.52:0/4201894600 shutdown_connections 2024-05-27T07:26:10.905 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:10.914+0000 7f73b8de3640 1 --2- 172.21.15.52:0/4201894600 >> [v2:172.21.15.52:6800/1822318384,v1:172.21.15.52:6801/1822318384] conn(0x7f739c0b2a70 0x7f739c0b4f80 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:26:10.905 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:10.914+0000 7f73b8de3640 1 --2- 172.21.15.52:0/4201894600 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f73b410e200 0x7f73b419a680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:26:10.905 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:10.914+0000 7f73b8de3640 1 --2- 172.21.15.52:0/4201894600 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f73b410b850 0x7f73b41a0a40 unknown :-1 s=CLOSED pgs=427 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:26:10.905 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:10.914+0000 7f73b8de3640 1 --2- 172.21.15.52:0/4201894600 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f73b4104df0 0x7f73b41a0500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:26:10.905 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:10.914+0000 7f73b8de3640 1 -- 172.21.15.52:0/4201894600 >> 172.21.15.52:0/4201894600 conn(0x7f73b40782d0 msgr2=0x7f73b41012b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:26:10.905 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:10.915+0000 7f73b8de3640 1 -- 172.21.15.52:0/4201894600 shutdown_connections 2024-05-27T07:26:10.906 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:10.915+0000 7f73b8de3640 1 -- 172.21.15.52:0/4201894600 wait complete. 2024-05-27T07:26:10.917 DEBUG:teuthology.orchestra.run.smithi052:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'ceph config set mgr mgr/snap_schedule/dump_on_update true' 2024-05-27T07:26:11.044 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:11.052+0000 7f94f82b6640 1 -- 172.21.15.52:0/2764628577 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f94f0108490 msgr2=0x7f94f01088a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:26:11.045 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:11.052+0000 7f94f82b6640 1 --2- 172.21.15.52:0/2764628577 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f94f0108490 0x7f94f01088a0 secure :-1 s=READY pgs=428 cs=0 l=1 rev1=1 crypto rx=0x7f94e0009900 tx=0x7f94e002a410 comp rx=0 tx=0).stop 2024-05-27T07:26:11.045 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:11.052+0000 7f94f82b6640 1 -- 172.21.15.52:0/2764628577 shutdown_connections 2024-05-27T07:26:11.045 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:11.052+0000 7f94f82b6640 1 --2- 172.21.15.52:0/2764628577 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f94f0108f30 0x7f94f01106d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:26:11.045 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:11.052+0000 7f94f82b6640 1 --2- 172.21.15.52:0/2764628577 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f94f0108490 0x7f94f01088a0 unknown :-1 s=CLOSED pgs=428 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:26:11.045 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:11.052+0000 7f94f82b6640 1 --2- 172.21.15.52:0/2764628577 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f94f0102410 0x7f94f0102820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:26:11.045 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:11.052+0000 7f94f82b6640 1 -- 172.21.15.52:0/2764628577 >> 172.21.15.52:0/2764628577 conn(0x7f94f00fd9a0 msgr2=0x7f94f00ffdf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:26:11.045 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:11.052+0000 7f94f82b6640 1 -- 172.21.15.52:0/2764628577 shutdown_connections 2024-05-27T07:26:11.045 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:11.052+0000 7f94f82b6640 1 -- 172.21.15.52:0/2764628577 wait complete. 2024-05-27T07:26:11.045 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:11.053+0000 7f94f82b6640 1 Processor -- start 2024-05-27T07:26:11.045 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:11.053+0000 7f94f82b6640 1 -- start start 2024-05-27T07:26:11.045 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:11.053+0000 7f94f82b6640 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f94f0102410 0x7f94f01a0580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:26:11.045 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:11.053+0000 7f94f82b6640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f94f0108490 0x7f94f01a0ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:26:11.046 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:11.053+0000 7f94f82b6640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f94f0108f30 0x7f94f019a700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:26:11.046 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:11.053+0000 7f94f82b6640 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x7f94f0113e40 con 0x7f94f0102410 2024-05-27T07:26:11.046 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:11.053+0000 7f94f82b6640 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f94f0113cc0 con 0x7f94f0108f30 2024-05-27T07:26:11.046 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:11.053+0000 7f94f82b6640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f94f0113fc0 con 0x7f94f0108490 2024-05-27T07:26:11.046 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:11.053+0000 7f94f6052640 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f94f0102410 0x7f94f01a0580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:26:11.046 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:11.053+0000 7f94f6052640 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f94f0102410 0x7f94f01a0580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.52:49714/0 (socket says 172.21.15.52:49714) 2024-05-27T07:26:11.046 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:11.053+0000 7f94f6052640 1 -- 172.21.15.52:0/2876772090 learned_addr learned my addr 172.21.15.52:0/2876772090 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-05-27T07:26:11.046 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:11.053+0000 7f94f6853640 1 --2- 172.21.15.52:0/2876772090 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f94f0108f30 0x7f94f019a700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:26:11.046 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:11.053+0000 7f94f6052640 1 -- 172.21.15.52:0/2876772090 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f94f0108490 msgr2=0x7f94f01a0ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:26:11.046 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:11.053+0000 7f94f6052640 1 --2- 172.21.15.52:0/2876772090 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f94f0108490 0x7f94f01a0ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:26:11.046 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:11.053+0000 7f94f6052640 1 -- 172.21.15.52:0/2876772090 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f94f0108f30 msgr2=0x7f94f019a700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:26:11.046 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:11.053+0000 7f94f6052640 1 --2- 172.21.15.52:0/2876772090 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f94f0108f30 0x7f94f019a700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:26:11.047 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:11.053+0000 7f94f6052640 1 -- 172.21.15.52:0/2876772090 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f94f019ae30 con 0x7f94f0102410 2024-05-27T07:26:11.047 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:11.053+0000 7f94f6853640 1 --2- 172.21.15.52:0/2876772090 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f94f0108f30 0x7f94f019a700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-27T07:26:11.047 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:11.054+0000 7f94f6052640 1 --2- 172.21.15.52:0/2876772090 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f94f0102410 0x7f94f01a0580 secure :-1 s=READY pgs=429 cs=0 l=1 rev1=1 crypto rx=0x7f94ec00e9c0 tx=0x7f94ec00ee80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:26:11.047 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:11.054+0000 7f94e77fe640 1 -- 172.21.15.52:0/2876772090 <== mon.0 v2:172.21.15.52:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f94ec00cd80 con 0x7f94f0102410 2024-05-27T07:26:11.047 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:11.054+0000 7f94e77fe640 1 -- 172.21.15.52:0/2876772090 <== mon.0 v2:172.21.15.52:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f94ec0035f0 con 0x7f94f0102410 2024-05-27T07:26:11.047 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:11.054+0000 7f94f82b6640 1 -- 172.21.15.52:0/2876772090 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f94f019b0c0 con 0x7f94f0102410 2024-05-27T07:26:11.047 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:11.054+0000 7f94e77fe640 1 -- 172.21.15.52:0/2876772090 <== mon.0 v2:172.21.15.52:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f94ec010690 con 0x7f94f0102410 2024-05-27T07:26:11.047 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:11.054+0000 7f94f82b6640 1 -- 172.21.15.52:0/2876772090 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f94f01a7320 con 0x7f94f0102410 2024-05-27T07:26:11.048 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:11.055+0000 7f94f82b6640 1 -- 172.21.15.52:0/2876772090 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f94f0105070 con 0x7f94f0102410 2024-05-27T07:26:11.048 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:11.057+0000 7f94e77fe640 1 -- 172.21.15.52:0/2876772090 <== mon.0 v2:172.21.15.52:3300/0 4 ==== mgrmap(e 24) ==== 149879+0+0 (secure 0 0 0) 0x7f94ec0037a0 con 0x7f94f0102410 2024-05-27T07:26:11.048 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:11.057+0000 7f94e77fe640 1 --2- 172.21.15.52:0/2876772090 >> [v2:172.21.15.52:6800/1822318384,v1:172.21.15.52:6801/1822318384] conn(0x7f94dc0b2a70 0x7f94dc0b4f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:26:11.048 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:11.058+0000 7f94f5851640 1 --2- 172.21.15.52:0/2876772090 >> [v2:172.21.15.52:6800/1822318384,v1:172.21.15.52:6801/1822318384] conn(0x7f94dc0b2a70 0x7f94dc0b4f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:26:11.048 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:11.058+0000 7f94e77fe640 1 -- 172.21.15.52:0/2876772090 <== mon.0 v2:172.21.15.52:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9379+0+0 (secure 0 0 0) 0x7f94ec0e31f0 con 0x7f94f0102410 2024-05-27T07:26:11.048 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:11.058+0000 7f94f5851640 1 --2- 172.21.15.52:0/2876772090 >> [v2:172.21.15.52:6800/1822318384,v1:172.21.15.52:6801/1822318384] conn(0x7f94dc0b2a70 0x7f94dc0b4f80 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f94e002a930 tx=0x7f94e0031400 comp rx=0 tx=0).ready entity=mgr.15333 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:26:11.050 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:11.061+0000 7f94e77fe640 1 -- 172.21.15.52:0/2876772090 <== mon.0 v2:172.21.15.52:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191963 (secure 0 0 0) 0x7f94ec0aa620 con 0x7f94f0102410 2024-05-27T07:26:11.245 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:11.255+0000 7f94f82b6640 1 -- 172.21.15.52:0/2876772090 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_command([{prefix=config set, name=mgr/snap_schedule/dump_on_update}] v 0) -- 0x7f94f019bc10 con 0x7f94f0102410 2024-05-27T07:26:11.249 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:11.260+0000 7f94e77fe640 1 -- 172.21.15.52:0/2876772090 <== mon.0 v2:172.21.15.52:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/snap_schedule/dump_on_update}]=0 v31)=0 v31) ==== 133+0+0 (secure 0 0 0) 0x7f94ec0a9f20 con 0x7f94f0102410 2024-05-27T07:26:11.252 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:11.262+0000 7f94f82b6640 1 -- 172.21.15.52:0/2876772090 >> [v2:172.21.15.52:6800/1822318384,v1:172.21.15.52:6801/1822318384] conn(0x7f94dc0b2a70 msgr2=0x7f94dc0b4f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:26:11.252 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:11.262+0000 7f94f82b6640 1 --2- 172.21.15.52:0/2876772090 >> [v2:172.21.15.52:6800/1822318384,v1:172.21.15.52:6801/1822318384] conn(0x7f94dc0b2a70 0x7f94dc0b4f80 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f94e002a930 tx=0x7f94e0031400 comp rx=0 tx=0).stop 2024-05-27T07:26:11.252 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:11.262+0000 7f94f82b6640 1 -- 172.21.15.52:0/2876772090 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f94f0102410 msgr2=0x7f94f01a0580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:26:11.252 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:11.262+0000 7f94f82b6640 1 --2- 172.21.15.52:0/2876772090 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f94f0102410 0x7f94f01a0580 secure :-1 s=READY pgs=429 cs=0 l=1 rev1=1 crypto rx=0x7f94ec00e9c0 tx=0x7f94ec00ee80 comp rx=0 tx=0).stop 2024-05-27T07:26:11.253 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:11.262+0000 7f94f82b6640 1 -- 172.21.15.52:0/2876772090 shutdown_connections 2024-05-27T07:26:11.253 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:11.262+0000 7f94f82b6640 1 --2- 172.21.15.52:0/2876772090 >> [v2:172.21.15.52:6800/1822318384,v1:172.21.15.52:6801/1822318384] conn(0x7f94dc0b2a70 0x7f94dc0b4f80 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:26:11.253 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:11.262+0000 7f94f82b6640 1 --2- 172.21.15.52:0/2876772090 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f94f0108f30 0x7f94f019a700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:26:11.253 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:11.262+0000 7f94f82b6640 1 --2- 172.21.15.52:0/2876772090 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f94f0108490 0x7f94f01a0ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:26:11.253 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:11.262+0000 7f94f82b6640 1 --2- 172.21.15.52:0/2876772090 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f94f0102410 0x7f94f01a0580 unknown :-1 s=CLOSED pgs=429 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:26:11.253 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:11.262+0000 7f94f82b6640 1 -- 172.21.15.52:0/2876772090 >> 172.21.15.52:0/2876772090 conn(0x7f94f00fd9a0 msgr2=0x7f94f01053d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:26:11.253 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:11.262+0000 7f94f82b6640 1 -- 172.21.15.52:0/2876772090 shutdown_connections 2024-05-27T07:26:11.253 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:11.262+0000 7f94f82b6640 1 -- 172.21.15.52:0/2876772090 wait complete. 2024-05-27T07:26:11.265 DEBUG:teuthology.orchestra.run.smithi052:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'ceph fs snap-schedule add --fs=cephfs --path=/ --snap_schedule=1m' 2024-05-27T07:26:11.387 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:11.395+0000 7f7ecbc13640 1 -- 172.21.15.52:0/626618669 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f7ec4103310 msgr2=0x7f7ec410f6e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:26:11.387 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:11.395+0000 7f7ecbc13640 1 --2- 172.21.15.52:0/626618669 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f7ec4103310 0x7f7ec410f6e0 secure :-1 s=READY pgs=430 cs=0 l=1 rev1=1 crypto rx=0x7f7ebc009ca0 tx=0x7f7ebc02a8c0 comp rx=0 tx=0).stop 2024-05-27T07:26:11.387 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:11.395+0000 7f7ecbc13640 1 -- 172.21.15.52:0/626618669 shutdown_connections 2024-05-27T07:26:11.387 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:11.395+0000 7f7ecbc13640 1 --2- 172.21.15.52:0/626618669 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f7ec4103310 0x7f7ec410f6e0 unknown :-1 s=CLOSED pgs=430 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:26:11.387 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:11.395+0000 7f7ecbc13640 1 --2- 172.21.15.52:0/626618669 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f7ec4109060 0x7f7ec4102ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:26:11.387 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:11.395+0000 7f7ecbc13640 1 --2- 172.21.15.52:0/626618669 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f7ec4108bc0 0x7f7ec4102760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:26:11.388 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:11.395+0000 7f7ecbc13640 1 -- 172.21.15.52:0/626618669 >> 172.21.15.52:0/626618669 conn(0x7f7ec40782c0 msgr2=0x7f7ec4078700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:26:11.388 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:11.395+0000 7f7ecbc13640 1 -- 172.21.15.52:0/626618669 shutdown_connections 2024-05-27T07:26:11.388 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:11.395+0000 7f7ecbc13640 1 -- 172.21.15.52:0/626618669 wait complete. 2024-05-27T07:26:11.388 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:11.396+0000 7f7ecbc13640 1 Processor -- start 2024-05-27T07:26:11.388 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:11.396+0000 7f7ecbc13640 1 -- start start 2024-05-27T07:26:11.388 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:11.396+0000 7f7ecbc13640 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f7ec4103310 0x7f7ec41a0540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:26:11.388 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:11.396+0000 7f7ecbc13640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f7ec4108bc0 0x7f7ec41a0a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:26:11.388 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:11.396+0000 7f7ecbc13640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f7ec4109060 0x7f7ec419a6c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:26:11.388 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:11.396+0000 7f7ecbc13640 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x7f7ec4112dd0 con 0x7f7ec4103310 2024-05-27T07:26:11.388 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:11.396+0000 7f7ecbc13640 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f7ec4112c50 con 0x7f7ec4109060 2024-05-27T07:26:11.388 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:11.396+0000 7f7ecbc13640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f7ec4112f50 con 0x7f7ec4108bc0 2024-05-27T07:26:11.388 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:11.396+0000 7f7ec99af640 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f7ec4103310 0x7f7ec41a0540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:26:11.388 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:11.396+0000 7f7ec99af640 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f7ec4103310 0x7f7ec41a0540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.52:49738/0 (socket says 172.21.15.52:49738) 2024-05-27T07:26:11.389 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:11.396+0000 7f7ec99af640 1 -- 172.21.15.52:0/2038151065 learned_addr learned my addr 172.21.15.52:0/2038151065 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-05-27T07:26:11.389 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:11.396+0000 7f7ec91ae640 1 --2- 172.21.15.52:0/2038151065 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f7ec4108bc0 0x7f7ec41a0a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:26:11.389 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:11.396+0000 7f7eca1b0640 1 --2- 172.21.15.52:0/2038151065 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f7ec4109060 0x7f7ec419a6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:26:11.389 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:11.396+0000 7f7ec99af640 1 -- 172.21.15.52:0/2038151065 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f7ec4108bc0 msgr2=0x7f7ec41a0a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:26:11.389 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:11.396+0000 7f7ec99af640 1 --2- 172.21.15.52:0/2038151065 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f7ec4108bc0 0x7f7ec41a0a80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:26:11.389 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:11.396+0000 7f7ec99af640 1 -- 172.21.15.52:0/2038151065 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f7ec4109060 msgr2=0x7f7ec419a6c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:26:11.389 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:11.396+0000 7f7ec99af640 1 --2- 172.21.15.52:0/2038151065 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f7ec4109060 0x7f7ec419a6c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:26:11.389 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:11.396+0000 7f7ec99af640 1 -- 172.21.15.52:0/2038151065 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7ec419adf0 con 0x7f7ec4103310 2024-05-27T07:26:11.389 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:11.397+0000 7f7ec99af640 1 --2- 172.21.15.52:0/2038151065 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f7ec4103310 0x7f7ec41a0540 secure :-1 s=READY pgs=431 cs=0 l=1 rev1=1 crypto rx=0x7f7ec000c970 tx=0x7f7ec000ce30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:26:11.389 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:11.397+0000 7f7ebaffd640 1 -- 172.21.15.52:0/2038151065 <== mon.0 v2:172.21.15.52:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7ec0007bb0 con 0x7f7ec4103310 2024-05-27T07:26:11.389 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:11.397+0000 7f7ecbc13640 1 -- 172.21.15.52:0/2038151065 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7ec419b080 con 0x7f7ec4103310 2024-05-27T07:26:11.389 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:11.397+0000 7f7ebaffd640 1 -- 172.21.15.52:0/2038151065 <== mon.0 v2:172.21.15.52:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f7ec0007d50 con 0x7f7ec4103310 2024-05-27T07:26:11.389 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:11.397+0000 7f7ebaffd640 1 -- 172.21.15.52:0/2038151065 <== mon.0 v2:172.21.15.52:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7ec0018730 con 0x7f7ec4103310 2024-05-27T07:26:11.390 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:11.397+0000 7f7ecbc13640 1 -- 172.21.15.52:0/2038151065 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7ec41a72e0 con 0x7f7ec4103310 2024-05-27T07:26:11.390 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:11.398+0000 7f7ecbc13640 1 -- 172.21.15.52:0/2038151065 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7ec4103ea0 con 0x7f7ec4103310 2024-05-27T07:26:11.390 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:11.400+0000 7f7ebaffd640 1 -- 172.21.15.52:0/2038151065 <== mon.0 v2:172.21.15.52:3300/0 4 ==== mgrmap(e 24) ==== 149879+0+0 (secure 0 0 0) 0x7f7ec000f990 con 0x7f7ec4103310 2024-05-27T07:26:11.391 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:11.401+0000 7f7ebaffd640 1 --2- 172.21.15.52:0/2038151065 >> [v2:172.21.15.52:6800/1822318384,v1:172.21.15.52:6801/1822318384] conn(0x7f7eb00b2a70 0x7f7eb00b4f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:26:11.391 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:11.401+0000 7f7ec91ae640 1 --2- 172.21.15.52:0/2038151065 >> [v2:172.21.15.52:6800/1822318384,v1:172.21.15.52:6801/1822318384] conn(0x7f7eb00b2a70 0x7f7eb00b4f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:26:11.391 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:11.401+0000 7f7ebaffd640 1 -- 172.21.15.52:0/2038151065 <== mon.0 v2:172.21.15.52:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9379+0+0 (secure 0 0 0) 0x7f7ec00e3080 con 0x7f7ec4103310 2024-05-27T07:26:11.391 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:11.402+0000 7f7ec91ae640 1 --2- 172.21.15.52:0/2038151065 >> [v2:172.21.15.52:6800/1822318384,v1:172.21.15.52:6801/1822318384] conn(0x7f7eb00b2a70 0x7f7eb00b4f80 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f7ec419be60 tx=0x7f7eb400b390 comp rx=0 tx=0).ready entity=mgr.15333 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:26:11.394 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:11.404+0000 7f7ebaffd640 1 -- 172.21.15.52:0/2038151065 <== mon.0 v2:172.21.15.52:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191963 (secure 0 0 0) 0x7f7ec0014030 con 0x7f7ec4103310 2024-05-27T07:26:11.616 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:11.627+0000 7f7ecbc13640 1 -- 172.21.15.52:0/2038151065 --> [v2:172.21.15.52:6800/1822318384,v1:172.21.15.52:6801/1822318384] -- mgr_command(tid 0: {"prefix": "fs snap-schedule add", "fs": "cephfs", "path": "/", "snap_schedule": "1m", "target": ["mon-mgr", ""]}) -- 0x7f7ec4063100 con 0x7f7eb00b2a70 2024-05-27T07:26:11.686 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:11.696+0000 7f7ebaffd640 1 -- 172.21.15.52:0/2038151065 <== mgr.15333 v2:172.21.15.52:6800/1822318384 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+24 (secure 0 0 0) 0x7f7ec4063100 con 0x7f7eb00b2a70 2024-05-27T07:26:11.686 INFO:teuthology.orchestra.run.smithi052.stdout:Schedule set for path / 2024-05-27T07:26:11.688 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:11.699+0000 7f7ecbc13640 1 -- 172.21.15.52:0/2038151065 >> [v2:172.21.15.52:6800/1822318384,v1:172.21.15.52:6801/1822318384] conn(0x7f7eb00b2a70 msgr2=0x7f7eb00b4f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:26:11.689 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:11.699+0000 7f7ecbc13640 1 --2- 172.21.15.52:0/2038151065 >> [v2:172.21.15.52:6800/1822318384,v1:172.21.15.52:6801/1822318384] conn(0x7f7eb00b2a70 0x7f7eb00b4f80 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f7ec419be60 tx=0x7f7eb400b390 comp rx=0 tx=0).stop 2024-05-27T07:26:11.689 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:11.699+0000 7f7ecbc13640 1 -- 172.21.15.52:0/2038151065 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f7ec4103310 msgr2=0x7f7ec41a0540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:26:11.689 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:11.699+0000 7f7ecbc13640 1 --2- 172.21.15.52:0/2038151065 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f7ec4103310 0x7f7ec41a0540 secure :-1 s=READY pgs=431 cs=0 l=1 rev1=1 crypto rx=0x7f7ec000c970 tx=0x7f7ec000ce30 comp rx=0 tx=0).stop 2024-05-27T07:26:11.689 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:11.699+0000 7f7ecbc13640 1 -- 172.21.15.52:0/2038151065 shutdown_connections 2024-05-27T07:26:11.689 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:11.699+0000 7f7ecbc13640 1 --2- 172.21.15.52:0/2038151065 >> [v2:172.21.15.52:6800/1822318384,v1:172.21.15.52:6801/1822318384] conn(0x7f7eb00b2a70 0x7f7eb00b4f80 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:26:11.689 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:11.699+0000 7f7ecbc13640 1 --2- 172.21.15.52:0/2038151065 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f7ec4109060 0x7f7ec419a6c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:26:11.689 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:11.699+0000 7f7ecbc13640 1 --2- 172.21.15.52:0/2038151065 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f7ec4108bc0 0x7f7ec41a0a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:26:11.689 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:11.699+0000 7f7ecbc13640 1 --2- 172.21.15.52:0/2038151065 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f7ec4103310 0x7f7ec41a0540 unknown :-1 s=CLOSED pgs=431 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:26:11.689 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:11.699+0000 7f7ecbc13640 1 -- 172.21.15.52:0/2038151065 >> 172.21.15.52:0/2038151065 conn(0x7f7ec40782c0 msgr2=0x7f7ec4075370 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:26:11.689 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:11.699+0000 7f7ecbc13640 1 -- 172.21.15.52:0/2038151065 shutdown_connections 2024-05-27T07:26:11.689 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:11.699+0000 7f7ecbc13640 1 -- 172.21.15.52:0/2038151065 wait complete. 2024-05-27T07:26:11.699 DEBUG:teuthology.orchestra.run.smithi052:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'ceph fs snap-schedule retention add --fs=cephfs --path=/ --retention-spec-or-period=6m3h' 2024-05-27T07:26:11.828 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:11.836+0000 7fa727ea8640 1 -- 172.21.15.52:0/834071975 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fa720109030 msgr2=0x7fa720102c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:26:11.828 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:11.836+0000 7fa727ea8640 1 --2- 172.21.15.52:0/834071975 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fa720109030 0x7fa720102c70 secure :-1 s=READY pgs=432 cs=0 l=1 rev1=1 crypto rx=0x7fa710009900 tx=0x7fa71002a410 comp rx=0 tx=0).stop 2024-05-27T07:26:11.828 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:11.836+0000 7fa727ea8640 1 -- 172.21.15.52:0/834071975 shutdown_connections 2024-05-27T07:26:11.828 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:11.836+0000 7fa727ea8640 1 --2- 172.21.15.52:0/834071975 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fa7201032e0 0x7fa72010f6b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:26:11.828 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:11.836+0000 7fa727ea8640 1 --2- 172.21.15.52:0/834071975 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fa720109030 0x7fa720102c70 unknown :-1 s=CLOSED pgs=432 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:26:11.829 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:11.836+0000 7fa727ea8640 1 --2- 172.21.15.52:0/834071975 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fa720108b90 0x7fa720102730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:26:11.829 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:11.836+0000 7fa727ea8640 1 -- 172.21.15.52:0/834071975 >> 172.21.15.52:0/834071975 conn(0x7fa7200782d0 msgr2=0x7fa720078710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:26:11.829 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:11.836+0000 7fa727ea8640 1 -- 172.21.15.52:0/834071975 shutdown_connections 2024-05-27T07:26:11.829 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:11.836+0000 7fa727ea8640 1 -- 172.21.15.52:0/834071975 wait complete. 2024-05-27T07:26:11.829 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:11.837+0000 7fa727ea8640 1 Processor -- start 2024-05-27T07:26:11.829 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:11.837+0000 7fa727ea8640 1 -- start start 2024-05-27T07:26:11.829 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:11.837+0000 7fa727ea8640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fa7201032e0 0x7fa7201a0560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:26:11.829 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:11.837+0000 7fa727ea8640 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fa720108b90 0x7fa7201a0aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:26:11.829 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:11.837+0000 7fa727ea8640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fa720109030 0x7fa72019a6e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:26:11.829 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:11.837+0000 7fa727ea8640 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x7fa720112dd0 con 0x7fa720108b90 2024-05-27T07:26:11.829 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:11.837+0000 7fa727ea8640 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7fa720112c50 con 0x7fa720109030 2024-05-27T07:26:11.829 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:11.837+0000 7fa727ea8640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7fa720112f50 con 0x7fa7201032e0 2024-05-27T07:26:11.830 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:11.837+0000 7fa725443640 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fa720108b90 0x7fa7201a0aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:26:11.830 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:11.837+0000 7fa725443640 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fa720108b90 0x7fa7201a0aa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.52:49758/0 (socket says 172.21.15.52:49758) 2024-05-27T07:26:11.830 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:11.837+0000 7fa725443640 1 -- 172.21.15.52:0/2297040574 learned_addr learned my addr 172.21.15.52:0/2297040574 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-05-27T07:26:11.830 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:11.838+0000 7fa725c44640 1 --2- 172.21.15.52:0/2297040574 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fa7201032e0 0x7fa7201a0560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:26:11.830 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:11.838+0000 7fa726445640 1 --2- 172.21.15.52:0/2297040574 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fa720109030 0x7fa72019a6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:26:11.830 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:11.838+0000 7fa725443640 1 -- 172.21.15.52:0/2297040574 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fa7201032e0 msgr2=0x7fa7201a0560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:26:11.830 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:11.838+0000 7fa725443640 1 --2- 172.21.15.52:0/2297040574 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fa7201032e0 0x7fa7201a0560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:26:11.830 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:11.838+0000 7fa725443640 1 -- 172.21.15.52:0/2297040574 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fa720109030 msgr2=0x7fa72019a6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:26:11.830 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:11.838+0000 7fa725443640 1 --2- 172.21.15.52:0/2297040574 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fa720109030 0x7fa72019a6e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:26:11.830 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:11.838+0000 7fa725443640 1 -- 172.21.15.52:0/2297040574 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa72019ae10 con 0x7fa720108b90 2024-05-27T07:26:11.830 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:11.838+0000 7fa725c44640 1 --2- 172.21.15.52:0/2297040574 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fa7201032e0 0x7fa7201a0560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-27T07:26:11.830 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:11.838+0000 7fa726445640 1 --2- 172.21.15.52:0/2297040574 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fa720109030 0x7fa72019a6e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-27T07:26:11.830 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:11.838+0000 7fa725443640 1 --2- 172.21.15.52:0/2297040574 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fa720108b90 0x7fa7201a0aa0 secure :-1 s=READY pgs=433 cs=0 l=1 rev1=1 crypto rx=0x7fa710005ab0 tx=0x7fa710003cd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:26:11.831 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:11.838+0000 7fa716ffd640 1 -- 172.21.15.52:0/2297040574 <== mon.0 v2:172.21.15.52:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa71003f070 con 0x7fa720108b90 2024-05-27T07:26:11.831 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:11.838+0000 7fa727ea8640 1 -- 172.21.15.52:0/2297040574 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa72019b040 con 0x7fa720108b90 2024-05-27T07:26:11.831 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:11.838+0000 7fa716ffd640 1 -- 172.21.15.52:0/2297040574 <== mon.0 v2:172.21.15.52:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fa710031d60 con 0x7fa720108b90 2024-05-27T07:26:11.831 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:11.838+0000 7fa716ffd640 1 -- 172.21.15.52:0/2297040574 <== mon.0 v2:172.21.15.52:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa710035030 con 0x7fa720108b90 2024-05-27T07:26:11.831 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:11.838+0000 7fa727ea8640 1 -- 172.21.15.52:0/2297040574 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa72019b530 con 0x7fa720108b90 2024-05-27T07:26:11.832 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:11.839+0000 7fa727ea8640 1 -- 172.21.15.52:0/2297040574 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa720103e70 con 0x7fa720108b90 2024-05-27T07:26:11.832 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:11.841+0000 7fa716ffd640 1 -- 172.21.15.52:0/2297040574 <== mon.0 v2:172.21.15.52:3300/0 4 ==== mgrmap(e 24) ==== 149879+0+0 (secure 0 0 0) 0x7fa710042070 con 0x7fa720108b90 2024-05-27T07:26:11.832 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:11.842+0000 7fa716ffd640 1 --2- 172.21.15.52:0/2297040574 >> [v2:172.21.15.52:6800/1822318384,v1:172.21.15.52:6801/1822318384] conn(0x7fa70c0b2a70 0x7fa70c0b4f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:26:11.832 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:11.842+0000 7fa725c44640 1 --2- 172.21.15.52:0/2297040574 >> [v2:172.21.15.52:6800/1822318384,v1:172.21.15.52:6801/1822318384] conn(0x7fa70c0b2a70 0x7fa70c0b4f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:26:11.832 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:11.842+0000 7fa716ffd640 1 -- 172.21.15.52:0/2297040574 <== mon.0 v2:172.21.15.52:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9379+0+0 (secure 0 0 0) 0x7fa710103120 con 0x7fa720108b90 2024-05-27T07:26:11.832 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:11.842+0000 7fa725c44640 1 --2- 172.21.15.52:0/2297040574 >> [v2:172.21.15.52:6800/1822318384,v1:172.21.15.52:6801/1822318384] conn(0x7fa70c0b2a70 0x7fa70c0b4f80 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7fa720103870 tx=0x7fa71c0094b0 comp rx=0 tx=0).ready entity=mgr.15333 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:26:11.836 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:11.846+0000 7fa716ffd640 1 -- 172.21.15.52:0/2297040574 <== mon.0 v2:172.21.15.52:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191963 (secure 0 0 0) 0x7fa7100ca5c0 con 0x7fa720108b90 2024-05-27T07:26:11.852 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:26:11 smithi173 ceph-mon[37000]: from='client.? 172.21.15.52:0/2931251683' entity='client.admin' 2024-05-27T07:26:11.852 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:26:11 smithi173 ceph-mon[37000]: from='mgr.15333 172.21.15.52:0/416389082' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-27T07:26:11.852 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:26:11 smithi173 ceph-mon[37000]: from='mgr.15333 172.21.15.52:0/416389082' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T07:26:11.852 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:26:11 smithi173 ceph-mon[37000]: from='mgr.15333 172.21.15.52:0/416389082' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-27T07:26:11.852 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:26:11 smithi173 ceph-mon[37000]: from='mgr.15333 172.21.15.52:0/416389082' entity='mgr.x' 2024-05-27T07:26:11.852 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:26:11 smithi173 ceph-mon[37000]: from='client.? 172.21.15.52:0/4201894600' entity='client.admin' 2024-05-27T07:26:11.852 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:26:11 smithi173 ceph-mon[37000]: pgmap v10: 129 pgs: 129 active+clean; 1.1 MiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 597 B/s wr, 15 op/s 2024-05-27T07:26:11.852 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:26:11 smithi173 ceph-mon[37000]: from='mgr.15333 172.21.15.52:0/416389082' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-27T07:26:11.852 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:26:11 smithi173 ceph-mon[37000]: from='mgr.15333 172.21.15.52:0/416389082' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T07:26:11.852 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:26:11 smithi173 ceph-mon[37000]: from='mgr.15333 172.21.15.52:0/416389082' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-27T07:26:11.852 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:26:11 smithi173 ceph-mon[37000]: from='mgr.15333 172.21.15.52:0/416389082' entity='mgr.x' 2024-05-27T07:26:11.853 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:26:11 smithi173 ceph-mon[37000]: from='client.? 172.21.15.52:0/2876772090' entity='client.admin' 2024-05-27T07:26:11.853 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:26:11 smithi173 ceph-mon[37000]: from='mgr.15333 172.21.15.52:0/416389082' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-27T07:26:11.853 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:26:11 smithi173 ceph-mon[37000]: from='mgr.15333 172.21.15.52:0/416389082' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T07:26:11.853 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:26:11 smithi173 ceph-mon[37000]: from='mgr.15333 172.21.15.52:0/416389082' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-27T07:26:11.853 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:26:11 smithi173 ceph-mon[37000]: from='mgr.15333 172.21.15.52:0/416389082' entity='mgr.x' 2024-05-27T07:26:11.895 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:26:11 smithi052 ceph-mon[31955]: from='client.? 172.21.15.52:0/2931251683' entity='client.admin' 2024-05-27T07:26:11.895 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:26:11 smithi052 ceph-mon[31955]: from='mgr.15333 172.21.15.52:0/416389082' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-27T07:26:11.895 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:26:11 smithi052 ceph-mon[31955]: from='mgr.15333 172.21.15.52:0/416389082' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T07:26:11.895 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:26:11 smithi052 ceph-mon[31955]: from='mgr.15333 172.21.15.52:0/416389082' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-27T07:26:11.896 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:26:11 smithi052 ceph-mon[31955]: from='mgr.15333 172.21.15.52:0/416389082' entity='mgr.x' 2024-05-27T07:26:11.896 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:26:11 smithi052 ceph-mon[31955]: from='client.? 172.21.15.52:0/4201894600' entity='client.admin' 2024-05-27T07:26:11.896 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:26:11 smithi052 ceph-mon[31955]: pgmap v10: 129 pgs: 129 active+clean; 1.1 MiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 597 B/s wr, 15 op/s 2024-05-27T07:26:11.896 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:26:11 smithi052 ceph-mon[31955]: from='mgr.15333 172.21.15.52:0/416389082' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-27T07:26:11.896 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:26:11 smithi052 ceph-mon[31955]: from='mgr.15333 172.21.15.52:0/416389082' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T07:26:11.896 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:26:11 smithi052 ceph-mon[31955]: from='mgr.15333 172.21.15.52:0/416389082' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-27T07:26:11.896 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:26:11 smithi052 ceph-mon[31955]: from='mgr.15333 172.21.15.52:0/416389082' entity='mgr.x' 2024-05-27T07:26:11.896 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:26:11 smithi052 ceph-mon[31955]: from='client.? 172.21.15.52:0/2876772090' entity='client.admin' 2024-05-27T07:26:11.896 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:26:11 smithi052 ceph-mon[31955]: from='mgr.15333 172.21.15.52:0/416389082' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-27T07:26:11.896 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:26:11 smithi052 ceph-mon[31955]: from='mgr.15333 172.21.15.52:0/416389082' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T07:26:11.896 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:26:11 smithi052 ceph-mon[31955]: from='mgr.15333 172.21.15.52:0/416389082' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-27T07:26:11.896 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:26:11 smithi052 ceph-mon[31955]: from='mgr.15333 172.21.15.52:0/416389082' entity='mgr.x' 2024-05-27T07:26:11.940 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:26:11 smithi105 ceph-mon[35977]: from='client.? 172.21.15.52:0/2931251683' entity='client.admin' 2024-05-27T07:26:11.940 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:26:11 smithi105 ceph-mon[35977]: from='mgr.15333 172.21.15.52:0/416389082' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-27T07:26:11.940 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:26:11 smithi105 ceph-mon[35977]: from='mgr.15333 172.21.15.52:0/416389082' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T07:26:11.940 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:26:11 smithi105 ceph-mon[35977]: from='mgr.15333 172.21.15.52:0/416389082' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-27T07:26:11.940 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:26:11 smithi105 ceph-mon[35977]: from='mgr.15333 172.21.15.52:0/416389082' entity='mgr.x' 2024-05-27T07:26:11.940 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:26:11 smithi105 ceph-mon[35977]: from='client.? 172.21.15.52:0/4201894600' entity='client.admin' 2024-05-27T07:26:11.941 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:26:11 smithi105 ceph-mon[35977]: pgmap v10: 129 pgs: 129 active+clean; 1.1 MiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 597 B/s wr, 15 op/s 2024-05-27T07:26:11.941 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:26:11 smithi105 ceph-mon[35977]: from='mgr.15333 172.21.15.52:0/416389082' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-27T07:26:11.941 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:26:11 smithi105 ceph-mon[35977]: from='mgr.15333 172.21.15.52:0/416389082' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T07:26:11.941 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:26:11 smithi105 ceph-mon[35977]: from='mgr.15333 172.21.15.52:0/416389082' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-27T07:26:11.941 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:26:11 smithi105 ceph-mon[35977]: from='mgr.15333 172.21.15.52:0/416389082' entity='mgr.x' 2024-05-27T07:26:11.941 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:26:11 smithi105 ceph-mon[35977]: from='client.? 172.21.15.52:0/2876772090' entity='client.admin' 2024-05-27T07:26:11.941 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:26:11 smithi105 ceph-mon[35977]: from='mgr.15333 172.21.15.52:0/416389082' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-27T07:26:11.941 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:26:11 smithi105 ceph-mon[35977]: from='mgr.15333 172.21.15.52:0/416389082' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T07:26:11.941 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:26:11 smithi105 ceph-mon[35977]: from='mgr.15333 172.21.15.52:0/416389082' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-27T07:26:11.941 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:26:11 smithi105 ceph-mon[35977]: from='mgr.15333 172.21.15.52:0/416389082' entity='mgr.x' 2024-05-27T07:26:12.057 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:12.067+0000 7fa727ea8640 1 -- 172.21.15.52:0/2297040574 --> [v2:172.21.15.52:6800/1822318384,v1:172.21.15.52:6801/1822318384] -- mgr_command(tid 0: {"prefix": "fs snap-schedule retention add", "fs": "cephfs", "path": "/", "retention_spec_or_period": "6m3h", "target": ["mon-mgr", ""]}) -- 0x7fa7201062e0 con 0x7fa70c0b2a70 2024-05-27T07:26:12.123 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:12.133+0000 7fa716ffd640 1 -- 172.21.15.52:0/2297040574 <== mgr.15333 v2:172.21.15.52:6800/1822318384 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+26 (secure 0 0 0) 0x7fa7201062e0 con 0x7fa70c0b2a70 2024-05-27T07:26:12.124 INFO:teuthology.orchestra.run.smithi052.stdout:Retention added to path / 2024-05-27T07:26:12.125 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:12.135+0000 7fa727ea8640 1 -- 172.21.15.52:0/2297040574 >> [v2:172.21.15.52:6800/1822318384,v1:172.21.15.52:6801/1822318384] conn(0x7fa70c0b2a70 msgr2=0x7fa70c0b4f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:26:12.125 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:12.135+0000 7fa727ea8640 1 --2- 172.21.15.52:0/2297040574 >> [v2:172.21.15.52:6800/1822318384,v1:172.21.15.52:6801/1822318384] conn(0x7fa70c0b2a70 0x7fa70c0b4f80 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7fa720103870 tx=0x7fa71c0094b0 comp rx=0 tx=0).stop 2024-05-27T07:26:12.126 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:12.135+0000 7fa727ea8640 1 -- 172.21.15.52:0/2297040574 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fa720108b90 msgr2=0x7fa7201a0aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:26:12.126 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:12.135+0000 7fa727ea8640 1 --2- 172.21.15.52:0/2297040574 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fa720108b90 0x7fa7201a0aa0 secure :-1 s=READY pgs=433 cs=0 l=1 rev1=1 crypto rx=0x7fa710005ab0 tx=0x7fa710003cd0 comp rx=0 tx=0).stop 2024-05-27T07:26:12.126 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:12.135+0000 7fa727ea8640 1 -- 172.21.15.52:0/2297040574 shutdown_connections 2024-05-27T07:26:12.126 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:12.135+0000 7fa727ea8640 1 --2- 172.21.15.52:0/2297040574 >> [v2:172.21.15.52:6800/1822318384,v1:172.21.15.52:6801/1822318384] conn(0x7fa70c0b2a70 0x7fa70c0b4f80 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:26:12.126 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:12.136+0000 7fa727ea8640 1 --2- 172.21.15.52:0/2297040574 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fa720109030 0x7fa72019a6e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:26:12.126 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:12.136+0000 7fa727ea8640 1 --2- 172.21.15.52:0/2297040574 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fa720108b90 0x7fa7201a0aa0 unknown :-1 s=CLOSED pgs=433 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:26:12.126 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:12.136+0000 7fa727ea8640 1 --2- 172.21.15.52:0/2297040574 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fa7201032e0 0x7fa7201a0560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:26:12.126 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:12.136+0000 7fa727ea8640 1 -- 172.21.15.52:0/2297040574 >> 172.21.15.52:0/2297040574 conn(0x7fa7200782d0 msgr2=0x7fa7201014b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:26:12.126 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:12.136+0000 7fa727ea8640 1 -- 172.21.15.52:0/2297040574 shutdown_connections 2024-05-27T07:26:12.126 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:12.136+0000 7fa727ea8640 1 -- 172.21.15.52:0/2297040574 wait complete. 2024-05-27T07:26:12.137 DEBUG:teuthology.orchestra.run.smithi052:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'ceph fs snap-schedule status --fs=cephfs --path=/' 2024-05-27T07:26:12.281 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:12.291+0000 7f89e4842640 1 -- 172.21.15.52:0/3960387002 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f89e0104d90 msgr2=0x7f89e01051a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:26:12.281 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:12.291+0000 7f89e4842640 1 --2- 172.21.15.52:0/3960387002 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f89e0104d90 0x7f89e01051a0 secure :-1 s=READY pgs=434 cs=0 l=1 rev1=1 crypto rx=0x7f89d0009900 tx=0x7f89d002a410 comp rx=0 tx=0).stop 2024-05-27T07:26:12.282 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:12.291+0000 7f89e4842640 1 -- 172.21.15.52:0/3960387002 shutdown_connections 2024-05-27T07:26:12.282 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:12.291+0000 7f89e4842640 1 --2- 172.21.15.52:0/3960387002 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f89e010e1e0 0x7f89e01106d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:26:12.282 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:12.291+0000 7f89e4842640 1 --2- 172.21.15.52:0/3960387002 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f89e010b7f0 0x7f89e010dc10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:26:12.282 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:12.291+0000 7f89e4842640 1 --2- 172.21.15.52:0/3960387002 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f89e0104d90 0x7f89e01051a0 unknown :-1 s=CLOSED pgs=434 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:26:12.282 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:12.291+0000 7f89e4842640 1 -- 172.21.15.52:0/3960387002 >> 172.21.15.52:0/3960387002 conn(0x7f89e00782d0 msgr2=0x7f89e0078710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:26:12.282 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:12.291+0000 7f89e4842640 1 -- 172.21.15.52:0/3960387002 shutdown_connections 2024-05-27T07:26:12.282 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:12.291+0000 7f89e4842640 1 -- 172.21.15.52:0/3960387002 wait complete. 2024-05-27T07:26:12.284 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:12.292+0000 7f89e4842640 1 Processor -- start 2024-05-27T07:26:12.284 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:12.292+0000 7f89e4842640 1 -- start start 2024-05-27T07:26:12.284 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:12.293+0000 7f89e4842640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f89e0104d90 0x7f89e01a0520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:26:12.284 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:12.293+0000 7f89e4842640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f89e010b7f0 0x7f89e01a0a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:26:12.284 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:12.293+0000 7f89e4842640 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f89e010e1e0 0x7f89e019a6a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:26:12.284 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:12.293+0000 7f89e4842640 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x7f89e0113e40 con 0x7f89e010e1e0 2024-05-27T07:26:12.285 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:12.293+0000 7f89e4842640 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f89e0113cc0 con 0x7f89e010b7f0 2024-05-27T07:26:12.285 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:12.293+0000 7f89e4842640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f89e0113fc0 con 0x7f89e0104d90 2024-05-27T07:26:12.285 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:12.293+0000 7f89ded9d640 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f89e010e1e0 0x7f89e019a6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:26:12.285 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:12.293+0000 7f89ded9d640 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f89e010e1e0 0x7f89e019a6a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.52:49774/0 (socket says 172.21.15.52:49774) 2024-05-27T07:26:12.285 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:12.293+0000 7f89ded9d640 1 -- 172.21.15.52:0/803146963 learned_addr learned my addr 172.21.15.52:0/803146963 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-05-27T07:26:12.285 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:12.293+0000 7f89de59c640 1 --2- 172.21.15.52:0/803146963 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f89e0104d90 0x7f89e01a0520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:26:12.285 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:12.293+0000 7f89d5d9b640 1 --2- 172.21.15.52:0/803146963 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f89e010b7f0 0x7f89e01a0a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:26:12.285 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:12.293+0000 7f89ded9d640 1 -- 172.21.15.52:0/803146963 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f89e0104d90 msgr2=0x7f89e01a0520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:26:12.285 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:12.293+0000 7f89ded9d640 1 --2- 172.21.15.52:0/803146963 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f89e0104d90 0x7f89e01a0520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:26:12.285 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:12.293+0000 7f89ded9d640 1 -- 172.21.15.52:0/803146963 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f89e010b7f0 msgr2=0x7f89e01a0a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:26:12.285 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:12.293+0000 7f89ded9d640 1 --2- 172.21.15.52:0/803146963 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f89e010b7f0 0x7f89e01a0a60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:26:12.285 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:12.293+0000 7f89ded9d640 1 -- 172.21.15.52:0/803146963 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f89e019add0 con 0x7f89e010e1e0 2024-05-27T07:26:12.286 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:12.294+0000 7f89ded9d640 1 --2- 172.21.15.52:0/803146963 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f89e010e1e0 0x7f89e019a6a0 secure :-1 s=READY pgs=435 cs=0 l=1 rev1=1 crypto rx=0x7f89d8009fd0 tx=0x7f89d800b260 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:26:12.286 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:12.294+0000 7f89d7fff640 1 -- 172.21.15.52:0/803146963 <== mon.0 v2:172.21.15.52:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f89d8017070 con 0x7f89e010e1e0 2024-05-27T07:26:12.286 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:12.294+0000 7f89e4842640 1 -- 172.21.15.52:0/803146963 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f89e019b060 con 0x7f89e010e1e0 2024-05-27T07:26:12.286 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:12.294+0000 7f89d7fff640 1 -- 172.21.15.52:0/803146963 <== mon.0 v2:172.21.15.52:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f89d8003e20 con 0x7f89e010e1e0 2024-05-27T07:26:12.286 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:12.294+0000 7f89d7fff640 1 -- 172.21.15.52:0/803146963 <== mon.0 v2:172.21.15.52:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f89d8010460 con 0x7f89e010e1e0 2024-05-27T07:26:12.286 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:12.294+0000 7f89e4842640 1 -- 172.21.15.52:0/803146963 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f89e01a72c0 con 0x7f89e010e1e0 2024-05-27T07:26:12.287 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:12.295+0000 7f89e4842640 1 -- 172.21.15.52:0/803146963 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f89e006adc0 con 0x7f89e010e1e0 2024-05-27T07:26:12.287 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:12.296+0000 7f89d7fff640 1 -- 172.21.15.52:0/803146963 <== mon.0 v2:172.21.15.52:3300/0 4 ==== mgrmap(e 24) ==== 149879+0+0 (secure 0 0 0) 0x7f89d8003970 con 0x7f89e010e1e0 2024-05-27T07:26:12.287 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:12.297+0000 7f89d7fff640 1 --2- 172.21.15.52:0/803146963 >> [v2:172.21.15.52:6800/1822318384,v1:172.21.15.52:6801/1822318384] conn(0x7f89cc0b2a70 0x7f89cc0b4f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:26:12.288 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:12.297+0000 7f89de59c640 1 --2- 172.21.15.52:0/803146963 >> [v2:172.21.15.52:6800/1822318384,v1:172.21.15.52:6801/1822318384] conn(0x7f89cc0b2a70 0x7f89cc0b4f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:26:12.288 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:12.297+0000 7f89d7fff640 1 -- 172.21.15.52:0/803146963 <== mon.0 v2:172.21.15.52:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9379+0+0 (secure 0 0 0) 0x7f89d80e2910 con 0x7f89e010e1e0 2024-05-27T07:26:12.288 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:12.298+0000 7f89de59c640 1 --2- 172.21.15.52:0/803146963 >> [v2:172.21.15.52:6800/1822318384,v1:172.21.15.52:6801/1822318384] conn(0x7f89cc0b2a70 0x7f89cc0b4f80 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f89d0031440 tx=0x7f89d0009560 comp rx=0 tx=0).ready entity=mgr.15333 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:26:12.291 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:12.302+0000 7f89d7fff640 1 -- 172.21.15.52:0/803146963 <== mon.0 v2:172.21.15.52:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191963 (secure 0 0 0) 0x7f89d80a9690 con 0x7f89e010e1e0 2024-05-27T07:26:12.309 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:12.319+0000 7f1cf46fc700 1 -- 172.21.15.52:0/2115000809 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f1cec108730 msgr2=0x7f1cec110090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:26:12.310 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:12.319+0000 7f1cf46fc700 1 --2- 172.21.15.52:0/2115000809 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f1cec108730 0x7f1cec110090 secure :-1 s=READY pgs=436 cs=0 l=1 rev1=1 crypto rx=0x7f1ce400b0d0 tx=0x7f1ce400b3e0 comp rx=0 tx=0).stop 2024-05-27T07:26:12.310 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:12.319+0000 7f1cf46fc700 1 -- 172.21.15.52:0/2115000809 shutdown_connections 2024-05-27T07:26:12.310 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:12.319+0000 7f1cf46fc700 1 --2- 172.21.15.52:0/2115000809 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f1cec108730 0x7f1cec110090 unknown :-1 s=CLOSED pgs=436 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:26:12.310 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:12.319+0000 7f1cf46fc700 1 --2- 172.21.15.52:0/2115000809 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f1cec1034b0 0x7f1cec1038c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:26:12.310 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:12.319+0000 7f1cf46fc700 1 --2- 172.21.15.52:0/2115000809 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f1cec107de0 0x7f1cec1081f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:26:12.310 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:12.319+0000 7f1cf46fc700 1 -- 172.21.15.52:0/2115000809 >> 172.21.15.52:0/2115000809 conn(0x7f1cec0fedb0 msgr2=0x7f1cec1011d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:26:12.310 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:12.320+0000 7f1cf46fc700 1 -- 172.21.15.52:0/2115000809 shutdown_connections 2024-05-27T07:26:12.310 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:12.320+0000 7f1cf46fc700 1 -- 172.21.15.52:0/2115000809 wait complete. 2024-05-27T07:26:12.312 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:12.320+0000 7f1cf46fc700 1 Processor -- start 2024-05-27T07:26:12.312 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:12.320+0000 7f1cf46fc700 1 -- start start 2024-05-27T07:26:12.312 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:12.321+0000 7f1cf46fc700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f1cec1034b0 0x7f1cec19ade0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:26:12.312 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:12.321+0000 7f1cf46fc700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f1cec107de0 0x7f1cec19b380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:26:12.312 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:12.321+0000 7f1cf46fc700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f1cec108730 0x7f1cec19f610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:26:12.312 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:12.321+0000 7f1cf46fc700 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x7f1cec113380 con 0x7f1cec108730 2024-05-27T07:26:12.312 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:12.321+0000 7f1cf46fc700 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f1cec113200 con 0x7f1cec107de0 2024-05-27T07:26:12.312 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:12.321+0000 7f1cf46fc700 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f1cec113500 con 0x7f1cec1034b0 2024-05-27T07:26:12.313 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:12.321+0000 7f1cf2c99700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f1cec108730 0x7f1cec19f610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:26:12.313 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:12.321+0000 7f1cf1c97700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f1cec107de0 0x7f1cec19b380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:26:12.313 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:12.321+0000 7f1cf2c99700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f1cec108730 0x7f1cec19f610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.52:49806/0 (socket says 172.21.15.52:49806) 2024-05-27T07:26:12.313 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:12.321+0000 7f1cf2c99700 1 -- 172.21.15.52:0/988669189 learned_addr learned my addr 172.21.15.52:0/988669189 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-05-27T07:26:12.313 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:12.321+0000 7f1cf2498700 1 --2- 172.21.15.52:0/988669189 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f1cec1034b0 0x7f1cec19ade0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:26:12.313 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:12.321+0000 7f1cf1c97700 1 -- 172.21.15.52:0/988669189 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f1cec1034b0 msgr2=0x7f1cec19ade0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:26:12.313 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:12.321+0000 7f1cf1c97700 1 --2- 172.21.15.52:0/988669189 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f1cec1034b0 0x7f1cec19ade0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:26:12.313 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:12.321+0000 7f1cf1c97700 1 -- 172.21.15.52:0/988669189 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f1cec108730 msgr2=0x7f1cec19f610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:26:12.313 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:12.321+0000 7f1cf1c97700 1 --2- 172.21.15.52:0/988669189 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f1cec108730 0x7f1cec19f610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:26:12.313 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:12.321+0000 7f1cf1c97700 1 -- 172.21.15.52:0/988669189 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1cec19fcb0 con 0x7f1cec107de0 2024-05-27T07:26:12.313 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:12.321+0000 7f1cf2498700 1 --2- 172.21.15.52:0/988669189 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f1cec1034b0 0x7f1cec19ade0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-27T07:26:12.313 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:12.321+0000 7f1cf2c99700 1 --2- 172.21.15.52:0/988669189 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f1cec108730 0x7f1cec19f610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-27T07:26:12.314 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:12.322+0000 7f1cf1c97700 1 --2- 172.21.15.52:0/988669189 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f1cec107de0 0x7f1cec19b380 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f1cdc00e9c0 tx=0x7f1cdc00ecd0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:26:12.314 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:12.322+0000 7f1ce37fe700 1 -- 172.21.15.52:0/988669189 <== mon.2 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1cdc00cbd0 con 0x7f1cec107de0 2024-05-27T07:26:12.314 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:12.322+0000 7f1cf46fc700 1 -- 172.21.15.52:0/988669189 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1cec19ffa0 con 0x7f1cec107de0 2024-05-27T07:26:12.314 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:12.322+0000 7f1ce37fe700 1 -- 172.21.15.52:0/988669189 <== mon.2 v2:172.21.15.105:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f1cdc0044d0 con 0x7f1cec107de0 2024-05-27T07:26:12.314 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:12.322+0000 7f1ce37fe700 1 -- 172.21.15.52:0/988669189 <== mon.2 v2:172.21.15.105:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1cdc005020 con 0x7f1cec107de0 2024-05-27T07:26:12.314 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:12.322+0000 7f1cf46fc700 1 -- 172.21.15.52:0/988669189 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1cec0ff1c0 con 0x7f1cec107de0 2024-05-27T07:26:12.314 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:12.325+0000 7f1ce37fe700 1 -- 172.21.15.52:0/988669189 <== mon.2 v2:172.21.15.105:3300/0 4 ==== mgrmap(e 24) ==== 149879+0+0 (secure 0 0 0) 0x7f1cdc0051c0 con 0x7f1cec107de0 2024-05-27T07:26:12.315 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:12.326+0000 7f1d03111b80 1 Processor -- start 2024-05-27T07:26:12.316 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:12.326+0000 7f1d03111b80 1 -- start start 2024-05-27T07:26:12.316 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:12.326+0000 7f1ce37fe700 1 --2- 172.21.15.52:0/988669189 >> [v2:172.21.15.52:6800/1822318384,v1:172.21.15.52:6801/1822318384] conn(0x7f1cd80b6f20 0x7f1cd80b9430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:26:12.316 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:12.326+0000 7f1cf2c99700 1 --2- 172.21.15.52:0/988669189 >> [v2:172.21.15.52:6800/1822318384,v1:172.21.15.52:6801/1822318384] conn(0x7f1cd80b6f20 0x7f1cd80b9430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:26:12.316 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:12.326+0000 7f1ce37fe700 1 -- 172.21.15.52:0/988669189 <== mon.2 v2:172.21.15.105:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9379+0+0 (secure 0 0 0) 0x7f1cdc0e2a70 con 0x7f1cec107de0 2024-05-27T07:26:12.316 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:12.326+0000 7f1d03111b80 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55dfa7074ec0 0x55dfa71f9bb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:26:12.316 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:12.326+0000 7f1d03111b80 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55dfa71fa0f0 0x55dfa71f3c60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:26:12.316 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:12.326+0000 7f1d03111b80 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55dfa71f41a0 0x55dfa71f45b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:26:12.316 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:12.326+0000 7f1d03111b80 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x55dfa7121df0 con 0x55dfa71f41a0 2024-05-27T07:26:12.317 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:12.326+0000 7f1d03111b80 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x55dfa7111990 con 0x55dfa7074ec0 2024-05-27T07:26:12.317 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:12.326+0000 7f1d03111b80 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x55dfa71159d0 con 0x55dfa71fa0f0 2024-05-27T07:26:12.318 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:12.327+0000 7f1cf1c97700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55dfa71fa0f0 0x55dfa71f3c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:26:12.319 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:12.327+0000 7f1cf1c97700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55dfa71fa0f0 0x55dfa71f3c60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.173:3300/0 says I am v2:172.21.15.52:58906/0 (socket says 172.21.15.52:58906) 2024-05-27T07:26:12.319 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:12.327+0000 7f1cf1c97700 1 -- 172.21.15.52:0/2895738587 learned_addr learned my addr 172.21.15.52:0/2895738587 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-05-27T07:26:12.319 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:12.327+0000 7f1cf2498700 1 --2- 172.21.15.52:0/2895738587 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55dfa7074ec0 0x55dfa71f9bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:26:12.319 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:12.327+0000 7f1cf2c99700 1 --2- 172.21.15.52:0/2895738587 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55dfa71f41a0 0x55dfa71f45b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:26:12.319 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:12.327+0000 7f1cf1c97700 1 -- 172.21.15.52:0/2895738587 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55dfa7074ec0 msgr2=0x55dfa71f9bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:26:12.319 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:12.327+0000 7f1cf1c97700 1 --2- 172.21.15.52:0/2895738587 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55dfa7074ec0 0x55dfa71f9bb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:26:12.319 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:12.327+0000 7f1cf2c99700 1 --2- 172.21.15.52:0/988669189 >> [v2:172.21.15.52:6800/1822318384,v1:172.21.15.52:6801/1822318384] conn(0x7f1cd80b6f20 0x7f1cd80b9430 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f1ce400bfd0 tx=0x7f1ce4009ca0 comp rx=0 tx=0).ready entity=mgr.15333 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:26:12.319 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:12.327+0000 7f1cf1c97700 1 -- 172.21.15.52:0/2895738587 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55dfa71f41a0 msgr2=0x55dfa71f45b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:26:12.319 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:12.327+0000 7f1cf1c97700 1 --2- 172.21.15.52:0/2895738587 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55dfa71f41a0 0x55dfa71f45b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:26:12.319 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:12.327+0000 7f1cf1c97700 1 -- 172.21.15.52:0/2895738587 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55dfa70bb030 con 0x55dfa71fa0f0 2024-05-27T07:26:12.319 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:12.327+0000 7f1cf1c97700 1 --2- 172.21.15.52:0/2895738587 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55dfa71fa0f0 0x55dfa71f3c60 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f1cdc0a58f0 tx=0x7f1cdc0a9260 comp rx=0 tx=0).ready entity=mon.1 client_cookie=6e68a4cef6974a55 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:26:12.319 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:12.327+0000 7f1ce0ff9700 1 -- 172.21.15.52:0/2895738587 <== mon.1 v2:172.21.15.173:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1cdc004750 con 0x55dfa71fa0f0 2024-05-27T07:26:12.320 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:12.327+0000 7f1ce0ff9700 1 -- 172.21.15.52:0/2895738587 <== mon.1 v2:172.21.15.173:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f1cdc0a9380 con 0x55dfa71fa0f0 2024-05-27T07:26:12.320 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:12.327+0000 7f1ce0ff9700 1 -- 172.21.15.52:0/2895738587 <== mon.1 v2:172.21.15.173:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1cdc0a9cd0 con 0x55dfa71fa0f0 2024-05-27T07:26:12.320 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:12.328+0000 7f1d03111b80 1 -- 172.21.15.52:0/2895738587 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55dfa71fa0f0 msgr2=0x55dfa71f3c60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:26:12.320 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:12.328+0000 7f1d03111b80 1 --2- 172.21.15.52:0/2895738587 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55dfa71fa0f0 0x55dfa71f3c60 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f1cdc0a58f0 tx=0x7f1cdc0a9260 comp rx=0 tx=0).stop 2024-05-27T07:26:12.320 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:12.328+0000 7f1d03111b80 1 -- 172.21.15.52:0/2895738587 shutdown_connections 2024-05-27T07:26:12.320 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:12.328+0000 7f1d03111b80 1 --2- 172.21.15.52:0/2895738587 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55dfa71f41a0 0x55dfa71f45b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:26:12.320 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:12.328+0000 7f1d03111b80 1 --2- 172.21.15.52:0/2895738587 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55dfa7074ec0 0x55dfa71f9bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:26:12.320 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:12.328+0000 7f1d03111b80 1 --2- 172.21.15.52:0/2895738587 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55dfa71fa0f0 0x55dfa71f3c60 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:26:12.320 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:12.328+0000 7f1d03111b80 1 -- 172.21.15.52:0/2895738587 >> 172.21.15.52:0/2895738587 conn(0x55dfa70777f0 msgr2=0x55dfa71eee10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:26:12.320 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:12.329+0000 7f1d03111b80 1 -- 172.21.15.52:0/2895738587 shutdown_connections 2024-05-27T07:26:12.320 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:12.329+0000 7f1d03111b80 1 -- 172.21.15.52:0/2895738587 wait complete. 2024-05-27T07:26:12.320 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:12.329+0000 7f1d03111b80 1 -- start start 2024-05-27T07:26:12.320 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:12.329+0000 7f1d03111b80 1 Processor -- start 2024-05-27T07:26:12.324 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:12.329+0000 7f1d03111b80 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55dfa7074ec0 0x55dfa71fd8e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:26:12.324 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:12.329+0000 7f1d03111b80 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55dfa71f41a0 0x55dfa71fde20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:26:12.324 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:12.329+0000 7f1d03111b80 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55dfa71fa0f0 0x55dfa71fe360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:26:12.324 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:12.329+0000 7f1d03111b80 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x55dfa7018020 con 0x55dfa71fa0f0 2024-05-27T07:26:12.324 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:12.329+0000 7f1d03111b80 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x55dfa7079e20 con 0x55dfa71f41a0 2024-05-27T07:26:12.324 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:12.329+0000 7f1d03111b80 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x55dfa7200be0 con 0x55dfa7074ec0 2024-05-27T07:26:12.324 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:12.330+0000 7f1d03111b80 10 client.0 _ll_register_callbacks cb 0x55dfa70a8630 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T07:26:12.324 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:12.330+0000 7f1cf2498700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55dfa7074ec0 0x55dfa71fd8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:26:12.324 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:12.330+0000 7f1cf2c99700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55dfa71fa0f0 0x55dfa71fe360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:26:12.325 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:12.330+0000 7f1cf2498700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55dfa7074ec0 0x55dfa71fd8e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.173:3300/0 says I am v2:172.21.15.52:58922/0 (socket says 172.21.15.52:58922) 2024-05-27T07:26:12.325 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:12.330+0000 7f1cf2498700 1 -- 172.21.15.52:0/1098469273 learned_addr learned my addr 172.21.15.52:0/1098469273 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-05-27T07:26:12.325 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:12.330+0000 7f1cf1c97700 1 --2- 172.21.15.52:0/1098469273 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55dfa71f41a0 0x55dfa71fde20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:26:12.325 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:12.330+0000 7f1cf2498700 1 -- 172.21.15.52:0/1098469273 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55dfa71f41a0 msgr2=0x55dfa71fde20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:26:12.325 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:12.330+0000 7f1cf2498700 1 --2- 172.21.15.52:0/1098469273 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55dfa71f41a0 0x55dfa71fde20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:26:12.325 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:12.330+0000 7f1cf2498700 1 -- 172.21.15.52:0/1098469273 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55dfa71fa0f0 msgr2=0x55dfa71fe360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:26:12.325 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:12.330+0000 7f1cf2498700 1 --2- 172.21.15.52:0/1098469273 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55dfa71fa0f0 0x55dfa71fe360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:26:12.325 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:12.330+0000 7f1cf2498700 1 -- 172.21.15.52:0/1098469273 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55dfa7061950 con 0x55dfa7074ec0 2024-05-27T07:26:12.325 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:12.330+0000 7f1cf2498700 1 --2- 172.21.15.52:0/1098469273 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55dfa7074ec0 0x55dfa71fd8e0 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f1ce8012880 tx=0x7f1ce8012b90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=f98f5f16011d907b server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:26:12.325 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:12.330+0000 7f1cd5ffb700 10 client.0 ms_handle_connect on v2:172.21.15.173:3300/0 2024-05-27T07:26:12.325 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:12.330+0000 7f1cd5ffb700 1 -- 172.21.15.52:0/1098469273 <== mon.1 v2:172.21.15.173:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1ce8011810 con 0x55dfa7074ec0 2024-05-27T07:26:12.325 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:12.331+0000 7f1d03111b80 1 -- 172.21.15.52:0/1098469273 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f1cdc0a9380 con 0x55dfa7074ec0 2024-05-27T07:26:12.326 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:12.331+0000 7f1cd5ffb700 1 -- 172.21.15.52:0/1098469273 <== mon.1 v2:172.21.15.173:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 9379+0+0 (secure 0 0 0) 0x7f1ce80119b0 con 0x55dfa7074ec0 2024-05-27T07:26:12.326 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:12.331+0000 7f1cd5ffb700 1 -- 172.21.15.52:0/1098469273 <== mon.1 v2:172.21.15.173:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7f1ce8011bb0 con 0x55dfa7074ec0 2024-05-27T07:26:12.326 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:12.331+0000 7f1d03111b80 10 client.25210 fetch_fsmap learned FSMap version 30 2024-05-27T07:26:12.326 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:12.331+0000 7f1d03111b80 1 -- 172.21.15.52:0/1098469273 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x55dfa70cb7e0 con 0x55dfa7074ec0 2024-05-27T07:26:12.326 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:12.331+0000 7f1cd5ffb700 1 -- 172.21.15.52:0/1098469273 <== mon.1 v2:172.21.15.173:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f1cdc0a9380 con 0x55dfa7074ec0 2024-05-27T07:26:12.326 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:12.332+0000 7f1cd5ffb700 1 -- 172.21.15.52:0/1098469273 <== mon.1 v2:172.21.15.173:3300/0 5 ==== fsmap(e 30) ==== 8362+0+0 (secure 0 0 0) 0x7f1ce801c420 con 0x55dfa7074ec0 2024-05-27T07:26:12.326 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:12.332+0000 7f1d03111b80 10 client.25210 fetch_fsmap finished waiting for FSMap version 30 2024-05-27T07:26:12.326 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:12.332+0000 7f1d03111b80 10 client.25210 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-05-27T07:26:12.326 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:12.332+0000 7f1d03111b80 20 client.25210 populate_metadata read hostname 'smithi052' 2024-05-27T07:26:12.326 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:12.332+0000 7f1d03111b80 1 --2- 172.21.15.52:0/1098469273 >> [v2:172.21.15.105:6838/500348151,v1:172.21.15.105:6839/500348151] conn(0x55dfa720db30 0x55dfa720ff90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:26:12.326 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:12.332+0000 7f1d03111b80 4 client.25210 mds_command: new command op to 24527 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-27T07:26:12.326 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:12.332+0000 7f1d03111b80 1 -- 172.21.15.52:0/1098469273 --> [v2:172.21.15.105:6838/500348151,v1:172.21.15.105:6839/500348151] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55dfa7123d40 con 0x55dfa720db30 2024-05-27T07:26:12.327 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:12.332+0000 7f1cf1c97700 1 --2- 172.21.15.52:0/1098469273 >> [v2:172.21.15.105:6838/500348151,v1:172.21.15.105:6839/500348151] conn(0x55dfa720db30 0x55dfa720ff90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:26:12.327 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:12.333+0000 7f1cf1c97700 1 --2- 172.21.15.52:0/1098469273 >> [v2:172.21.15.105:6838/500348151,v1:172.21.15.105:6839/500348151] conn(0x55dfa720db30 0x55dfa720ff90 crc :-1 s=READY pgs=42 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=3d30f856bf3d1037 server_cookie=86ad6d792d28e5aa in_seq=0 out_seq=0 2024-05-27T07:26:12.327 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:12.333+0000 7f1cd5ffb700 10 client.25210 ms_handle_connect on v2:172.21.15.105:6838/500348151 2024-05-27T07:26:12.327 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:12.333+0000 7f1cd5ffb700 1 -- 172.21.15.52:0/1098469273 <== mds.0 v2:172.21.15.105:6838/500348151 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24041 (crc 0 0 0) 0x55dfa7123d40 con 0x55dfa720db30 2024-05-27T07:26:12.327 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:12.333+0000 7f1cd5ffb700 10 client.25210 handle_command_reply: tid=0 2024-05-27T07:26:12.327 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:12.333+0000 7f1d03111b80 1 client.25210 shutdown 2024-05-27T07:26:12.327 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:12.334+0000 7f1d03111b80 1 -- 172.21.15.52:0/1098469273 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55dfa7074ec0 msgr2=0x55dfa71fd8e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:26:12.327 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:12.334+0000 7f1d03111b80 1 --2- 172.21.15.52:0/1098469273 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55dfa7074ec0 0x55dfa71fd8e0 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f1ce8012880 tx=0x7f1ce8012b90 comp rx=0 tx=0).stop 2024-05-27T07:26:12.327 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:12.334+0000 7f1d03111b80 1 -- 172.21.15.52:0/1098469273 shutdown_connections 2024-05-27T07:26:12.327 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:12.334+0000 7f1d03111b80 1 --2- 172.21.15.52:0/1098469273 >> [v2:172.21.15.105:6838/500348151,v1:172.21.15.105:6839/500348151] conn(0x55dfa720db30 0x55dfa720ff90 crc :-1 s=READY pgs=42 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:26:12.327 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:12.334+0000 7f1d03111b80 1 --2- 172.21.15.52:0/1098469273 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55dfa71fa0f0 0x55dfa71fe360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:26:12.327 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:12.334+0000 7f1d03111b80 1 --2- 172.21.15.52:0/1098469273 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55dfa7074ec0 0x55dfa71fd8e0 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:26:12.327 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:12.334+0000 7f1d03111b80 1 --2- 172.21.15.52:0/1098469273 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55dfa71f41a0 0x55dfa71fde20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:26:12.328 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:12.334+0000 7f1d03111b80 1 -- 172.21.15.52:0/1098469273 >> 172.21.15.52:0/1098469273 conn(0x55dfa70777f0 msgr2=0x55dfa71f2c50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:26:12.328 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:12.334+0000 7f1cd5ffb700 0 client.25210 ms_handle_reset on v2:172.21.15.105:6838/500348151 2024-05-27T07:26:12.328 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:12.334+0000 7f1d03111b80 1 -- 172.21.15.52:0/1098469273 shutdown_connections 2024-05-27T07:26:12.328 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:12.334+0000 7f1d03111b80 1 -- 172.21.15.52:0/1098469273 wait complete. 2024-05-27T07:26:12.328 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:12.334+0000 7f1d03111b80 20 client.25210 trim_cache size 0 max 16384 2024-05-27T07:26:12.345 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:12.355+0000 7f1d03111b80 1 Processor -- start 2024-05-27T07:26:12.345 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:12.355+0000 7f1d03111b80 1 -- start start 2024-05-27T07:26:12.346 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:12.355+0000 7f1d03111b80 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55dfa7074ec0 0x55dfa71fc840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:26:12.346 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:12.355+0000 7f1d03111b80 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55dfa71f41a0 0x55dfa71ffd80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:26:12.346 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:12.355+0000 7f1d03111b80 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55dfa71fa0f0 0x55dfa71fe1a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:26:12.347 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:12.355+0000 7f1d03111b80 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x7f1cec1ae2d0 con 0x55dfa71f41a0 2024-05-27T07:26:12.347 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:12.355+0000 7f1d03111b80 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f1cec1afe10 con 0x55dfa7074ec0 2024-05-27T07:26:12.347 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:12.355+0000 7f1d03111b80 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f1cec1ae9a0 con 0x55dfa71fa0f0 2024-05-27T07:26:12.347 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:12.356+0000 7f1cf1c97700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55dfa71f41a0 0x55dfa71ffd80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:26:12.347 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:12.356+0000 7f1cf1c97700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55dfa71f41a0 0x55dfa71ffd80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.52:49848/0 (socket says 172.21.15.52:49848) 2024-05-27T07:26:12.347 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:12.356+0000 7f1cf1c97700 1 -- 172.21.15.52:0/838497221 learned_addr learned my addr 172.21.15.52:0/838497221 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-05-27T07:26:12.347 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:12.356+0000 7f1cf2498700 1 --2- 172.21.15.52:0/838497221 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55dfa7074ec0 0x55dfa71fc840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:26:12.347 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:12.356+0000 7f1cf2c99700 1 --2- 172.21.15.52:0/838497221 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55dfa71fa0f0 0x55dfa71fe1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:26:12.347 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:12.356+0000 7f1cf1c97700 1 -- 172.21.15.52:0/838497221 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55dfa71fa0f0 msgr2=0x55dfa71fe1a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:26:12.347 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:12.356+0000 7f1cf1c97700 1 --2- 172.21.15.52:0/838497221 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55dfa71fa0f0 0x55dfa71fe1a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:26:12.347 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:12.356+0000 7f1cf1c97700 1 -- 172.21.15.52:0/838497221 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55dfa7074ec0 msgr2=0x55dfa71fc840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:26:12.347 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:12.356+0000 7f1cf1c97700 1 --2- 172.21.15.52:0/838497221 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55dfa7074ec0 0x55dfa71fc840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:26:12.347 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:12.356+0000 7f1cf1c97700 1 -- 172.21.15.52:0/838497221 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55dfa71fbe50 con 0x55dfa71f41a0 2024-05-27T07:26:12.348 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:12.356+0000 7f1cf2c99700 1 --2- 172.21.15.52:0/838497221 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55dfa71fa0f0 0x55dfa71fe1a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-27T07:26:12.348 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:12.356+0000 7f1cf2498700 1 --2- 172.21.15.52:0/838497221 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55dfa7074ec0 0x55dfa71fc840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-27T07:26:12.348 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:12.356+0000 7f1cf1c97700 1 --2- 172.21.15.52:0/838497221 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55dfa71f41a0 0x55dfa71ffd80 secure :-1 s=READY pgs=437 cs=0 l=1 rev1=1 crypto rx=0x7f1cdc0a9800 tx=0x7f1cdc0cc040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8132f1f1ea8ad460 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:26:12.348 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:12.356+0000 7f1cd7fff700 1 -- 172.21.15.52:0/838497221 <== mon.0 v2:172.21.15.52:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1cdc0c2030 con 0x55dfa71f41a0 2024-05-27T07:26:12.348 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:12.357+0000 7f1cd7fff700 1 -- 172.21.15.52:0/838497221 <== mon.0 v2:172.21.15.52:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f1cdc0a9380 con 0x55dfa71f41a0 2024-05-27T07:26:12.348 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:12.357+0000 7f1cd7fff700 1 -- 172.21.15.52:0/838497221 <== mon.0 v2:172.21.15.52:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1cdc0a9870 con 0x55dfa71f41a0 2024-05-27T07:26:12.351 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:12.357+0000 7f1d03111b80 1 -- 172.21.15.52:0/838497221 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55dfa71f41a0 msgr2=0x55dfa71ffd80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:26:12.351 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:12.357+0000 7f1d03111b80 1 --2- 172.21.15.52:0/838497221 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55dfa71f41a0 0x55dfa71ffd80 secure :-1 s=READY pgs=437 cs=0 l=1 rev1=1 crypto rx=0x7f1cdc0a9800 tx=0x7f1cdc0cc040 comp rx=0 tx=0).stop 2024-05-27T07:26:12.351 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:12.357+0000 7f1d03111b80 1 -- 172.21.15.52:0/838497221 shutdown_connections 2024-05-27T07:26:12.351 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:12.357+0000 7f1d03111b80 1 --2- 172.21.15.52:0/838497221 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55dfa71fa0f0 0x55dfa71fe1a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:26:12.351 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:12.357+0000 7f1d03111b80 1 --2- 172.21.15.52:0/838497221 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55dfa7074ec0 0x55dfa71fc840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:26:12.351 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:12.357+0000 7f1d03111b80 1 --2- 172.21.15.52:0/838497221 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55dfa71f41a0 0x55dfa71ffd80 unknown :-1 s=CLOSED pgs=437 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:26:12.351 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:12.357+0000 7f1d03111b80 1 -- 172.21.15.52:0/838497221 >> 172.21.15.52:0/838497221 conn(0x55dfa70777f0 msgr2=0x55dfa71e8570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:26:12.351 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:12.358+0000 7f1d03111b80 1 -- 172.21.15.52:0/838497221 shutdown_connections 2024-05-27T07:26:12.351 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:12.358+0000 7f1d03111b80 1 -- 172.21.15.52:0/838497221 wait complete. 2024-05-27T07:26:12.352 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:12.358+0000 7f1d03111b80 1 -- start start 2024-05-27T07:26:12.352 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:12.358+0000 7f1d03111b80 1 Processor -- start 2024-05-27T07:26:12.352 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:12.358+0000 7f1d03111b80 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55dfa7074ec0 0x55dfa7215460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:26:12.352 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:12.358+0000 7f1d03111b80 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55dfa71f41a0 0x55dfa72159a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:26:12.352 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:12.358+0000 7f1d03111b80 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55dfa71fa0f0 0x55dfa7215ee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:26:12.352 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:12.358+0000 7f1d03111b80 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x7f1cec1af740 con 0x55dfa71f41a0 2024-05-27T07:26:12.352 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:12.358+0000 7f1d03111b80 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f1cec1af070 con 0x55dfa71fa0f0 2024-05-27T07:26:12.352 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:12.358+0000 7f1d03111b80 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f1cec1ac290 con 0x55dfa7074ec0 2024-05-27T07:26:12.352 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:12.358+0000 7f1d03111b80 10 client.0 _ll_register_callbacks cb 0x55dfa70a8630 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T07:26:12.352 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:12.358+0000 7f1cf1c97700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55dfa71f41a0 0x55dfa72159a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:26:12.352 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:12.358+0000 7f1cf1c97700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55dfa71f41a0 0x55dfa72159a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.52:49856/0 (socket says 172.21.15.52:49856) 2024-05-27T07:26:12.352 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:12.359+0000 7f1cf1c97700 1 -- 172.21.15.52:0/70645949 learned_addr learned my addr 172.21.15.52:0/70645949 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-05-27T07:26:12.353 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:12.359+0000 7f1cf2c99700 1 --2- 172.21.15.52:0/70645949 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55dfa71fa0f0 0x55dfa7215ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:26:12.353 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:12.359+0000 7f1cf2498700 1 --2- 172.21.15.52:0/70645949 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55dfa7074ec0 0x55dfa7215460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:26:12.353 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:12.359+0000 7f1cf1c97700 1 -- 172.21.15.52:0/70645949 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55dfa7074ec0 msgr2=0x55dfa7215460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:26:12.353 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:12.359+0000 7f1cf1c97700 1 --2- 172.21.15.52:0/70645949 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55dfa7074ec0 0x55dfa7215460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:26:12.353 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:12.359+0000 7f1cf1c97700 1 -- 172.21.15.52:0/70645949 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55dfa71fa0f0 msgr2=0x55dfa7215ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:26:12.353 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:12.359+0000 7f1cf1c97700 1 --2- 172.21.15.52:0/70645949 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55dfa71fa0f0 0x55dfa7215ee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:26:12.353 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:12.359+0000 7f1cf1c97700 1 -- 172.21.15.52:0/70645949 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55dfa72777b0 con 0x55dfa71f41a0 2024-05-27T07:26:12.353 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:12.359+0000 7f1cf2498700 1 --2- 172.21.15.52:0/70645949 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55dfa7074ec0 0x55dfa7215460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-27T07:26:12.353 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:12.359+0000 7f1cf1c97700 1 --2- 172.21.15.52:0/70645949 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55dfa71f41a0 0x55dfa72159a0 secure :-1 s=READY pgs=438 cs=0 l=1 rev1=1 crypto rx=0x7f1cdc0b1fd0 tx=0x7f1cdc0dfe20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e5dfb9866e9f324f server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:26:12.353 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:12.359+0000 7f1cd5ffb700 10 client.0 ms_handle_connect on v2:172.21.15.52:3300/0 2024-05-27T07:26:12.353 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:12.359+0000 7f1cd5ffb700 1 -- 172.21.15.52:0/70645949 <== mon.0 v2:172.21.15.52:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1cdc0c2070 con 0x55dfa71f41a0 2024-05-27T07:26:12.353 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:12.359+0000 7f1cd5ffb700 1 -- 172.21.15.52:0/70645949 <== mon.0 v2:172.21.15.52:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 9379+0+0 (secure 0 0 0) 0x7f1cdc0a9ad0 con 0x55dfa71f41a0 2024-05-27T07:26:12.353 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:12.359+0000 7f1d03111b80 1 -- 172.21.15.52:0/70645949 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f1cdc0a9380 con 0x55dfa71f41a0 2024-05-27T07:26:12.354 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:12.359+0000 7f1cd5ffb700 1 -- 172.21.15.52:0/70645949 <== mon.0 v2:172.21.15.52:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7f1cdc06f280 con 0x55dfa71f41a0 2024-05-27T07:26:12.354 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:12.360+0000 7f1d03111b80 10 client.15519 fetch_fsmap learned FSMap version 30 2024-05-27T07:26:12.354 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:12.360+0000 7f1d03111b80 1 -- 172.21.15.52:0/70645949 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x55dfa7278960 con 0x55dfa71f41a0 2024-05-27T07:26:12.354 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:12.360+0000 7f1cd5ffb700 1 -- 172.21.15.52:0/70645949 <== mon.0 v2:172.21.15.52:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f1cdc0a9380 con 0x55dfa71f41a0 2024-05-27T07:26:12.354 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:12.360+0000 7f1cd5ffb700 1 -- 172.21.15.52:0/70645949 <== mon.0 v2:172.21.15.52:3300/0 5 ==== fsmap(e 30) ==== 8362+0+0 (secure 0 0 0) 0x7f1cdc06b070 con 0x55dfa71f41a0 2024-05-27T07:26:12.354 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:12.360+0000 7f1d03111b80 10 client.15519 fetch_fsmap finished waiting for FSMap version 30 2024-05-27T07:26:12.354 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:12.360+0000 7f1d03111b80 10 client.15519 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-05-27T07:26:12.354 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:12.360+0000 7f1d03111b80 20 client.15519 populate_metadata read hostname 'smithi052' 2024-05-27T07:26:12.354 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:12.360+0000 7f1d03111b80 1 --2- 172.21.15.52:0/70645949 >> [v2:172.21.15.105:6838/500348151,v1:172.21.15.105:6839/500348151] conn(0x55dfa7278be0 0x55dfa727b000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:26:12.354 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:12.360+0000 7f1d03111b80 4 client.15519 mds_command: new command op to 24527 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-05-27T07:26:12.354 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:12.360+0000 7f1d03111b80 1 -- 172.21.15.52:0/70645949 --> [v2:172.21.15.105:6838/500348151,v1:172.21.15.105:6839/500348151] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) -- 0x55dfa711d520 con 0x55dfa7278be0 2024-05-27T07:26:12.354 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:12.360+0000 7f1cf2498700 1 --2- 172.21.15.52:0/70645949 >> [v2:172.21.15.105:6838/500348151,v1:172.21.15.105:6839/500348151] conn(0x55dfa7278be0 0x55dfa727b000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:26:12.354 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:12.361+0000 7f1cd5ffb700 10 client.15519 ms_handle_connect on v2:172.21.15.105:6838/500348151 2024-05-27T07:26:12.355 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:12.361+0000 7f1cf2498700 1 --2- 172.21.15.52:0/70645949 >> [v2:172.21.15.105:6838/500348151,v1:172.21.15.105:6839/500348151] conn(0x55dfa7278be0 0x55dfa727b000 crc :-1 s=READY pgs=43 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=27a83b9e0d23c1a6 server_cookie=6bea8a5936993faa in_seq=0 out_seq=0 2024-05-27T07:26:12.356 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:12.362+0000 7f1cd5ffb700 1 -- 172.21.15.52:0/70645949 <== mds.0 v2:172.21.15.105:6838/500348151 1 ==== command_reply(tid 0: 0 ) ==== 8+0+110 (crc 0 0 0) 0x55dfa711d520 con 0x55dfa7278be0 2024-05-27T07:26:12.356 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:12.362+0000 7f1cd5ffb700 10 client.15519 handle_command_reply: tid=0 2024-05-27T07:26:12.356 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:12.362+0000 7f1d03111b80 1 client.15519 shutdown 2024-05-27T07:26:12.356 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:12.362+0000 7f1d03111b80 1 -- 172.21.15.52:0/70645949 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55dfa71f41a0 msgr2=0x55dfa72159a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:26:12.356 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:12.362+0000 7f1d03111b80 1 --2- 172.21.15.52:0/70645949 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55dfa71f41a0 0x55dfa72159a0 secure :-1 s=READY pgs=438 cs=0 l=1 rev1=1 crypto rx=0x7f1cdc0b1fd0 tx=0x7f1cdc0dfe20 comp rx=0 tx=0).stop 2024-05-27T07:26:12.356 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:12.362+0000 7f1d03111b80 1 -- 172.21.15.52:0/70645949 shutdown_connections 2024-05-27T07:26:12.356 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:12.362+0000 7f1d03111b80 1 --2- 172.21.15.52:0/70645949 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55dfa71fa0f0 0x55dfa7215ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:26:12.356 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:12.362+0000 7f1d03111b80 1 --2- 172.21.15.52:0/70645949 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55dfa7074ec0 0x55dfa7215460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:26:12.356 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:12.362+0000 7f1d03111b80 1 --2- 172.21.15.52:0/70645949 >> [v2:172.21.15.105:6838/500348151,v1:172.21.15.105:6839/500348151] conn(0x55dfa7278be0 0x55dfa727b000 crc :-1 s=READY pgs=43 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:26:12.356 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:12.363+0000 7f1d03111b80 1 --2- 172.21.15.52:0/70645949 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55dfa71f41a0 0x55dfa72159a0 unknown :-1 s=CLOSED pgs=438 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:26:12.356 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:12.363+0000 7f1cd5ffb700 0 client.15519 ms_handle_reset on v2:172.21.15.105:6838/500348151 2024-05-27T07:26:12.356 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:12.363+0000 7f1d03111b80 1 -- 172.21.15.52:0/70645949 >> 172.21.15.52:0/70645949 conn(0x55dfa70777f0 msgr2=0x55dfa71f7ca0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:26:12.356 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:12.363+0000 7f1d03111b80 1 -- 172.21.15.52:0/70645949 shutdown_connections 2024-05-27T07:26:12.357 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:12.363+0000 7f1d03111b80 1 -- 172.21.15.52:0/70645949 wait complete. 2024-05-27T07:26:12.357 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:12.363+0000 7f1d03111b80 20 client.15519 trim_cache size 0 max 16384 2024-05-27T07:26:12.357 INFO:teuthology.orchestra.run.smithi052.stdout:{ 2024-05-27T07:26:12.357 INFO:teuthology.orchestra.run.smithi052.stdout: "return_code": 0, 2024-05-27T07:26:12.357 INFO:teuthology.orchestra.run.smithi052.stdout: "scrub_tag": "ae7a37c0-a7de-4a0a-aade-1c8cbfbe54a3", 2024-05-27T07:26:12.357 INFO:teuthology.orchestra.run.smithi052.stdout: "mode": "asynchronous" 2024-05-27T07:26:12.357 INFO:teuthology.orchestra.run.smithi052.stdout:} 2024-05-27T07:26:12.358 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:12.363+0000 7f1cf46fc700 1 -- 172.21.15.52:0/988669189 >> [v2:172.21.15.52:6800/1822318384,v1:172.21.15.52:6801/1822318384] conn(0x7f1cd80b6f20 msgr2=0x7f1cd80b9430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:26:12.358 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:12.363+0000 7f1cf46fc700 1 --2- 172.21.15.52:0/988669189 >> [v2:172.21.15.52:6800/1822318384,v1:172.21.15.52:6801/1822318384] conn(0x7f1cd80b6f20 0x7f1cd80b9430 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f1ce400bfd0 tx=0x7f1ce4009ca0 comp rx=0 tx=0).stop 2024-05-27T07:26:12.358 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:12.363+0000 7f1cf46fc700 1 -- 172.21.15.52:0/988669189 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f1cec107de0 msgr2=0x7f1cec19b380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:26:12.358 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:12.363+0000 7f1cf46fc700 1 --2- 172.21.15.52:0/988669189 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f1cec107de0 0x7f1cec19b380 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f1cdc00e9c0 tx=0x7f1cdc00ecd0 comp rx=0 tx=0).stop 2024-05-27T07:26:12.358 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:12.363+0000 7f1cf46fc700 1 -- 172.21.15.52:0/988669189 shutdown_connections 2024-05-27T07:26:12.358 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:12.363+0000 7f1cf46fc700 1 --2- 172.21.15.52:0/988669189 >> [v2:172.21.15.52:6800/1822318384,v1:172.21.15.52:6801/1822318384] conn(0x7f1cd80b6f20 0x7f1cd80b9430 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:26:12.358 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:12.363+0000 7f1cf46fc700 1 --2- 172.21.15.52:0/988669189 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f1cec108730 0x7f1cec19f610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:26:12.358 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:12.363+0000 7f1cf46fc700 1 --2- 172.21.15.52:0/988669189 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f1cec1034b0 0x7f1cec19ade0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:26:12.358 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:12.363+0000 7f1cf46fc700 1 --2- 172.21.15.52:0/988669189 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f1cec107de0 0x7f1cec19b380 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:26:12.359 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:12.363+0000 7f1cf46fc700 1 -- 172.21.15.52:0/988669189 >> 172.21.15.52:0/988669189 conn(0x7f1cec0fedb0 msgr2=0x7f1cec06b940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:26:12.359 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:12.364+0000 7f1cf46fc700 1 -- 172.21.15.52:0/988669189 shutdown_connections 2024-05-27T07:26:12.359 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:12.364+0000 7f1cf46fc700 1 -- 172.21.15.52:0/988669189 wait complete. 2024-05-27T07:26:12.513 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:12.523+0000 7f89e4842640 1 -- 172.21.15.52:0/803146963 --> [v2:172.21.15.52:6800/1822318384,v1:172.21.15.52:6801/1822318384] -- mgr_command(tid 0: {"prefix": "fs snap-schedule status", "fs": "cephfs", "path": "/", "target": ["mon-mgr", ""]}) -- 0x7f89e01019c0 con 0x7f89cc0b2a70 2024-05-27T07:26:12.521 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:12.532+0000 7f89d7fff640 1 -- 172.21.15.52:0/803146963 <== mgr.15333 v2:172.21.15.52:6800/1822318384 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+298 (secure 0 0 0) 0x7f89e01019c0 con 0x7f89cc0b2a70 2024-05-27T07:26:12.521 INFO:teuthology.orchestra.run.smithi052.stdout:{"fs": "cephfs", "subvol": null, "group": null, "path": "/", "rel_path": "/", "schedule": "1m", "retention": {"m": 6, "h": 3}, "start": "2024-05-27T00:00:00", "created": "2024-05-27T07:26:11", "first": null, "last": null, "last_pruned": null, "created_count": 0, "pruned_count": 0, "active": true} 2024-05-27T07:26:12.523 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:12.534+0000 7f89e4842640 1 -- 172.21.15.52:0/803146963 >> [v2:172.21.15.52:6800/1822318384,v1:172.21.15.52:6801/1822318384] conn(0x7f89cc0b2a70 msgr2=0x7f89cc0b4f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:26:12.524 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:12.534+0000 7f89e4842640 1 --2- 172.21.15.52:0/803146963 >> [v2:172.21.15.52:6800/1822318384,v1:172.21.15.52:6801/1822318384] conn(0x7f89cc0b2a70 0x7f89cc0b4f80 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f89d0031440 tx=0x7f89d0009560 comp rx=0 tx=0).stop 2024-05-27T07:26:12.524 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:12.534+0000 7f89e4842640 1 -- 172.21.15.52:0/803146963 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f89e010e1e0 msgr2=0x7f89e019a6a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:26:12.524 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:12.534+0000 7f89e4842640 1 --2- 172.21.15.52:0/803146963 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f89e010e1e0 0x7f89e019a6a0 secure :-1 s=READY pgs=435 cs=0 l=1 rev1=1 crypto rx=0x7f89d8009fd0 tx=0x7f89d800b260 comp rx=0 tx=0).stop 2024-05-27T07:26:12.524 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:12.534+0000 7f89e4842640 1 -- 172.21.15.52:0/803146963 shutdown_connections 2024-05-27T07:26:12.524 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:12.534+0000 7f89e4842640 1 --2- 172.21.15.52:0/803146963 >> [v2:172.21.15.52:6800/1822318384,v1:172.21.15.52:6801/1822318384] conn(0x7f89cc0b2a70 0x7f89cc0b4f80 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:26:12.524 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:12.534+0000 7f89e4842640 1 --2- 172.21.15.52:0/803146963 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f89e010e1e0 0x7f89e019a6a0 unknown :-1 s=CLOSED pgs=435 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:26:12.524 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:12.534+0000 7f89e4842640 1 --2- 172.21.15.52:0/803146963 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f89e010b7f0 0x7f89e01a0a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:26:12.524 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:12.534+0000 7f89e4842640 1 --2- 172.21.15.52:0/803146963 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f89e0104d90 0x7f89e01a0520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:26:12.524 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:12.534+0000 7f89e4842640 1 -- 172.21.15.52:0/803146963 >> 172.21.15.52:0/803146963 conn(0x7f89e00782d0 msgr2=0x7f89e01012b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:26:12.524 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:12.534+0000 7f89e4842640 1 -- 172.21.15.52:0/803146963 shutdown_connections 2024-05-27T07:26:12.524 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:12.534+0000 7f89e4842640 1 -- 172.21.15.52:0/803146963 wait complete. 2024-05-27T07:26:12.535 DEBUG:teuthology.orchestra.run.smithi052:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'ceph fs snap-schedule list --fs=cephfs --path=/ --recursive=true' 2024-05-27T07:26:12.660 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:12.670+0000 7ffab338d640 1 -- 172.21.15.52:0/2909701166 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ffaac1075a0 msgr2=0x7ffaac10ed40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:26:12.660 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:12.670+0000 7ffab338d640 1 --2- 172.21.15.52:0/2909701166 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ffaac1075a0 0x7ffaac10ed40 secure :-1 s=READY pgs=439 cs=0 l=1 rev1=1 crypto rx=0x7ffa9c009ca0 tx=0x7ffa9c02a8c0 comp rx=0 tx=0).stop 2024-05-27T07:26:12.660 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:12.670+0000 7ffab338d640 1 -- 172.21.15.52:0/2909701166 shutdown_connections 2024-05-27T07:26:12.660 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:12.670+0000 7ffab338d640 1 --2- 172.21.15.52:0/2909701166 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ffaac1075a0 0x7ffaac10ed40 unknown :-1 s=CLOSED pgs=439 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:26:12.661 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:12.670+0000 7ffab338d640 1 --2- 172.21.15.52:0/2909701166 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7ffaac106ae0 0x7ffaac106f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:26:12.661 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:12.670+0000 7ffab338d640 1 --2- 172.21.15.52:0/2909701166 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7ffaac102550 0x7ffaac102960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:26:12.661 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:12.670+0000 7ffab338d640 1 -- 172.21.15.52:0/2909701166 >> 172.21.15.52:0/2909701166 conn(0x7ffaac0fda10 msgr2=0x7ffaac0ffe60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:26:12.661 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:12.670+0000 7ffab338d640 1 -- 172.21.15.52:0/2909701166 shutdown_connections 2024-05-27T07:26:12.661 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:12.670+0000 7ffab338d640 1 -- 172.21.15.52:0/2909701166 wait complete. 2024-05-27T07:26:12.661 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:12.671+0000 7ffab338d640 1 Processor -- start 2024-05-27T07:26:12.661 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:12.671+0000 7ffab338d640 1 -- start start 2024-05-27T07:26:12.663 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:12.671+0000 7ffab338d640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7ffaac102550 0x7ffaac1a0530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:26:12.663 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:12.671+0000 7ffab338d640 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ffaac106ae0 0x7ffaac1a0a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:26:12.663 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:12.671+0000 7ffab338d640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7ffaac1075a0 0x7ffaac19a6b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:26:12.663 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:12.671+0000 7ffab338d640 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x7ffaac111c30 con 0x7ffaac106ae0 2024-05-27T07:26:12.663 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:12.671+0000 7ffab338d640 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7ffaac111ab0 con 0x7ffaac1075a0 2024-05-27T07:26:12.663 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:12.671+0000 7ffab338d640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7ffaac111db0 con 0x7ffaac102550 2024-05-27T07:26:12.663 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:12.671+0000 7ffab0928640 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ffaac106ae0 0x7ffaac1a0a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:26:12.663 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:12.671+0000 7ffab0928640 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ffaac106ae0 0x7ffaac1a0a70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.52:49882/0 (socket says 172.21.15.52:49882) 2024-05-27T07:26:12.663 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:12.671+0000 7ffab0928640 1 -- 172.21.15.52:0/1106991784 learned_addr learned my addr 172.21.15.52:0/1106991784 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-05-27T07:26:12.664 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:12.671+0000 7ffab192a640 1 --2- 172.21.15.52:0/1106991784 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7ffaac1075a0 0x7ffaac19a6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:26:12.664 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:12.671+0000 7ffab1129640 1 --2- 172.21.15.52:0/1106991784 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7ffaac102550 0x7ffaac1a0530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:26:12.664 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:12.671+0000 7ffab0928640 1 -- 172.21.15.52:0/1106991784 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7ffaac102550 msgr2=0x7ffaac1a0530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:26:12.664 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:12.671+0000 7ffab0928640 1 --2- 172.21.15.52:0/1106991784 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7ffaac102550 0x7ffaac1a0530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:26:12.664 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:12.671+0000 7ffab0928640 1 -- 172.21.15.52:0/1106991784 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7ffaac1075a0 msgr2=0x7ffaac19a6b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:26:12.664 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:12.671+0000 7ffab0928640 1 --2- 172.21.15.52:0/1106991784 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7ffaac1075a0 0x7ffaac19a6b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:26:12.664 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:12.671+0000 7ffab0928640 1 -- 172.21.15.52:0/1106991784 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ffaac19acf0 con 0x7ffaac106ae0 2024-05-27T07:26:12.664 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:12.672+0000 7ffab0928640 1 --2- 172.21.15.52:0/1106991784 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ffaac106ae0 0x7ffaac1a0a70 secure :-1 s=READY pgs=440 cs=0 l=1 rev1=1 crypto rx=0x7ffaa000dbc0 tx=0x7ffaa000a250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:26:12.664 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:12.672+0000 7ffaaa7fc640 1 -- 172.21.15.52:0/1106991784 <== mon.0 v2:172.21.15.52:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffaa001b070 con 0x7ffaac106ae0 2024-05-27T07:26:12.664 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:12.672+0000 7ffab338d640 1 -- 172.21.15.52:0/1106991784 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ffaac19afe0 con 0x7ffaac106ae0 2024-05-27T07:26:12.664 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:12.672+0000 7ffaaa7fc640 1 -- 172.21.15.52:0/1106991784 <== mon.0 v2:172.21.15.52:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ffaa000e900 con 0x7ffaac106ae0 2024-05-27T07:26:12.664 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:12.672+0000 7ffaaa7fc640 1 -- 172.21.15.52:0/1106991784 <== mon.0 v2:172.21.15.52:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffaa00168b0 con 0x7ffaac106ae0 2024-05-27T07:26:12.664 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:12.672+0000 7ffab338d640 1 -- 172.21.15.52:0/1106991784 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ffaac1a72d0 con 0x7ffaac106ae0 2024-05-27T07:26:12.665 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:12.673+0000 7ffab338d640 1 -- 172.21.15.52:0/1106991784 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ffaac1035d0 con 0x7ffaac106ae0 2024-05-27T07:26:12.665 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:12.675+0000 7ffaaa7fc640 1 -- 172.21.15.52:0/1106991784 <== mon.0 v2:172.21.15.52:3300/0 4 ==== mgrmap(e 24) ==== 149879+0+0 (secure 0 0 0) 0x7ffaa001e070 con 0x7ffaac106ae0 2024-05-27T07:26:12.665 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:12.675+0000 7ffaaa7fc640 1 --2- 172.21.15.52:0/1106991784 >> [v2:172.21.15.52:6800/1822318384,v1:172.21.15.52:6801/1822318384] conn(0x7ffa940b2a70 0x7ffa940b4f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:26:12.665 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:12.675+0000 7ffaaa7fc640 1 -- 172.21.15.52:0/1106991784 <== mon.0 v2:172.21.15.52:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9379+0+0 (secure 0 0 0) 0x7ffaa00e2770 con 0x7ffaac106ae0 2024-05-27T07:26:12.666 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:12.675+0000 7ffab1129640 1 --2- 172.21.15.52:0/1106991784 >> [v2:172.21.15.52:6800/1822318384,v1:172.21.15.52:6801/1822318384] conn(0x7ffa940b2a70 0x7ffa940b4f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:26:12.666 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:12.676+0000 7ffab1129640 1 --2- 172.21.15.52:0/1106991784 >> [v2:172.21.15.52:6800/1822318384,v1:172.21.15.52:6801/1822318384] conn(0x7ffa940b2a70 0x7ffa940b4f80 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7ffaa400abe0 tx=0x7ffaa400a2d0 comp rx=0 tx=0).ready entity=mgr.15333 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:26:12.668 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:12.679+0000 7ffaaa7fc640 1 -- 172.21.15.52:0/1106991784 <== mon.0 v2:172.21.15.52:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191963 (secure 0 0 0) 0x7ffaa00a9660 con 0x7ffaac106ae0 2024-05-27T07:26:12.884 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:12.894+0000 7ffab338d640 1 -- 172.21.15.52:0/1106991784 --> [v2:172.21.15.52:6800/1822318384,v1:172.21.15.52:6801/1822318384] -- mgr_command(tid 0: {"prefix": "fs snap-schedule list", "fs": "cephfs", "path": "/", "recursive": true, "target": ["mon-mgr", ""]}) -- 0x7ffaac063100 con 0x7ffa940b2a70 2024-05-27T07:26:12.892 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:12.902+0000 7ffaaa7fc640 1 -- 172.21.15.52:0/1106991784 <== mgr.15333 v2:172.21.15.52:6800/1822318384 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+10 (secure 0 0 0) 0x7ffaac063100 con 0x7ffa940b2a70 2024-05-27T07:26:12.892 INFO:teuthology.orchestra.run.smithi052.stdout:/ 1m 6m3h 2024-05-27T07:26:12.894 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:12.904+0000 7ffab338d640 1 -- 172.21.15.52:0/1106991784 >> [v2:172.21.15.52:6800/1822318384,v1:172.21.15.52:6801/1822318384] conn(0x7ffa940b2a70 msgr2=0x7ffa940b4f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:26:12.894 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:12.904+0000 7ffab338d640 1 --2- 172.21.15.52:0/1106991784 >> [v2:172.21.15.52:6800/1822318384,v1:172.21.15.52:6801/1822318384] conn(0x7ffa940b2a70 0x7ffa940b4f80 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7ffaa400abe0 tx=0x7ffaa400a2d0 comp rx=0 tx=0).stop 2024-05-27T07:26:12.894 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:12.904+0000 7ffab338d640 1 -- 172.21.15.52:0/1106991784 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ffaac106ae0 msgr2=0x7ffaac1a0a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:26:12.894 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:12.904+0000 7ffab338d640 1 --2- 172.21.15.52:0/1106991784 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ffaac106ae0 0x7ffaac1a0a70 secure :-1 s=READY pgs=440 cs=0 l=1 rev1=1 crypto rx=0x7ffaa000dbc0 tx=0x7ffaa000a250 comp rx=0 tx=0).stop 2024-05-27T07:26:12.894 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:12.904+0000 7ffab338d640 1 -- 172.21.15.52:0/1106991784 shutdown_connections 2024-05-27T07:26:12.894 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:12.904+0000 7ffab338d640 1 --2- 172.21.15.52:0/1106991784 >> [v2:172.21.15.52:6800/1822318384,v1:172.21.15.52:6801/1822318384] conn(0x7ffa940b2a70 0x7ffa940b4f80 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:26:12.894 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:12.904+0000 7ffab338d640 1 --2- 172.21.15.52:0/1106991784 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7ffaac1075a0 0x7ffaac19a6b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:26:12.894 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:12.904+0000 7ffab338d640 1 --2- 172.21.15.52:0/1106991784 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ffaac106ae0 0x7ffaac1a0a70 unknown :-1 s=CLOSED pgs=440 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:26:12.894 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:12.904+0000 7ffab338d640 1 --2- 172.21.15.52:0/1106991784 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7ffaac102550 0x7ffaac1a0530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:26:12.895 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:12.904+0000 7ffab338d640 1 -- 172.21.15.52:0/1106991784 >> 172.21.15.52:0/1106991784 conn(0x7ffaac0fda10 msgr2=0x7ffaac1004d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:26:12.895 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:12.904+0000 7ffab338d640 1 -- 172.21.15.52:0/1106991784 shutdown_connections 2024-05-27T07:26:12.895 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:12.904+0000 7ffab338d640 1 -- 172.21.15.52:0/1106991784 wait complete. 2024-05-27T07:26:12.906 DEBUG:teuthology.orchestra.run.smithi052:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'date +%s > START_TIME' 2024-05-27T07:26:12.934 INFO:teuthology.run_tasks:Running task full_sequential_finally... 2024-05-27T07:26:12.944 INFO:teuthology.run_tasks:Running task background_exec... 2024-05-27T07:26:12.963 INFO:teuthology.task.background_exec:Running background command on role mon.a host ubuntu@smithi052.front.sepia.ceph.com 2024-05-27T07:26:12.963 DEBUG:teuthology.orchestra.run.smithi052:> sudo TESTDIR=/home/ubuntu/cephtest daemon-helper kill --kill-group bash -c 'while sleep 13; do ceph tell mds.cephfs:0 flush journal; done' 2024-05-27T07:26:12.978 INFO:teuthology.run_tasks:Running task workunit... 2024-05-27T07:26:12.992 INFO:tasks.workunit:Pulling workunits from ref 38f9d900c83c63799fbdbe61acc9a11b0d3554a6 2024-05-27T07:26:12.992 INFO:tasks.workunit:Making a separate scratch dir for every client... 2024-05-27T07:26:12.992 INFO:tasks.workunit:timeout=3h 2024-05-27T07:26:12.992 INFO:tasks.workunit:cleanup=True 2024-05-27T07:26:12.993 DEBUG:teuthology.orchestra.run.smithi052:> stat -- /home/ubuntu/cephtest/mnt.0 2024-05-27T07:26:13.009 INFO:teuthology.orchestra.run.smithi052.stdout: File: /home/ubuntu/cephtest/mnt.0 2024-05-27T07:26:13.009 INFO:teuthology.orchestra.run.smithi052.stdout: Size: 0 Blocks: 0 IO Block: 65536 directory 2024-05-27T07:26:13.009 INFO:teuthology.orchestra.run.smithi052.stdout:Device: 77h/119d Inode: 1099511628283 Links: 2 2024-05-27T07:26:13.010 INFO:teuthology.orchestra.run.smithi052.stdout:Access: (1777/drwxrwxrwt) Uid: ( 0/ root) Gid: ( 0/ root) 2024-05-27T07:26:13.010 INFO:teuthology.orchestra.run.smithi052.stdout:Context: system_u:object_r:unlabeled_t:s0 2024-05-27T07:26:13.010 INFO:teuthology.orchestra.run.smithi052.stdout:Access: 2024-05-27 07:23:10.438877753 +0000 2024-05-27T07:26:13.010 INFO:teuthology.orchestra.run.smithi052.stdout:Modify: 2024-05-27 07:23:10.438877753 +0000 2024-05-27T07:26:13.010 INFO:teuthology.orchestra.run.smithi052.stdout:Change: 2024-05-27 07:25:53.592855566 +0000 2024-05-27T07:26:13.010 INFO:teuthology.orchestra.run.smithi052.stdout: Birth: 2024-05-27 07:23:10.438877753 +0000 2024-05-27T07:26:13.010 INFO:tasks.workunit:Did not need to create dir /home/ubuntu/cephtest/mnt.0 2024-05-27T07:26:13.010 DEBUG:teuthology.orchestra.run.smithi052:> cd -- /home/ubuntu/cephtest/mnt.0 && sudo install -d -m 0755 --owner=ubuntu -- client.0 2024-05-27T07:26:13.085 DEBUG:teuthology.orchestra.run.smithi052:> rm -rf /home/ubuntu/cephtest/clone.client.0 && git clone https://git.ceph.com/ceph-ci.git /home/ubuntu/cephtest/clone.client.0 && cd /home/ubuntu/cephtest/clone.client.0 && git checkout 38f9d900c83c63799fbdbe61acc9a11b0d3554a6 2024-05-27T07:26:13.143 INFO:tasks.workunit.client.0.smithi052.stderr:Cloning into '/home/ubuntu/cephtest/clone.client.0'... 2024-05-27T07:26:13.162 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:26:12 smithi052 ceph-mon[31955]: from='client.15489 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule add", "fs": "cephfs", "path": "/", "snap_schedule": "1m", "target": ["mon-mgr", ""]}]: dispatch 2024-05-27T07:26:13.163 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:26:12 smithi052 ceph-mon[31955]: from='client.15495 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule retention add", "fs": "cephfs", "path": "/", "retention_spec_or_period": "6m3h", "target": ["mon-mgr", ""]}]: dispatch 2024-05-27T07:26:13.163 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:26:12 smithi052 ceph-mon[31955]: scrub queued for path: / 2024-05-27T07:26:13.163 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:26:12 smithi052 ceph-mon[31955]: scrub summary: idle+waiting paths [/] 2024-05-27T07:26:13.163 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:26:12 smithi052 ceph-mon[31955]: scrub summary: active paths [/] 2024-05-27T07:26:13.163 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:26:12 smithi052 ceph-mon[31955]: scrub summary: idle+waiting paths [/] 2024-05-27T07:26:13.164 DEBUG:teuthology.orchestra.run.smithi052:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:38f9d900c83c63799fbdbe61acc9a11b0d3554a6 shell --fsid 73df2636-1bf8-11ef-bc9b-c7b262605968 -- ceph tell mds.1:0 scrub status 2024-05-27T07:26:13.190 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:26:12 smithi105 ceph-mon[35977]: from='client.15489 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule add", "fs": "cephfs", "path": "/", "snap_schedule": "1m", "target": ["mon-mgr", ""]}]: dispatch 2024-05-27T07:26:13.190 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:26:12 smithi105 ceph-mon[35977]: from='client.15495 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule retention add", "fs": "cephfs", "path": "/", "retention_spec_or_period": "6m3h", "target": ["mon-mgr", ""]}]: dispatch 2024-05-27T07:26:13.190 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:26:12 smithi105 ceph-mon[35977]: scrub queued for path: / 2024-05-27T07:26:13.190 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:26:12 smithi105 ceph-mon[35977]: scrub summary: idle+waiting paths [/] 2024-05-27T07:26:13.190 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:26:12 smithi105 ceph-mon[35977]: scrub summary: active paths [/] 2024-05-27T07:26:13.190 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:26:12 smithi105 ceph-mon[35977]: scrub summary: idle+waiting paths [/] 2024-05-27T07:26:13.351 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:26:12 smithi173 ceph-mon[37000]: from='client.15489 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule add", "fs": "cephfs", "path": "/", "snap_schedule": "1m", "target": ["mon-mgr", ""]}]: dispatch 2024-05-27T07:26:13.352 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:26:12 smithi173 ceph-mon[37000]: from='client.15495 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule retention add", "fs": "cephfs", "path": "/", "retention_spec_or_period": "6m3h", "target": ["mon-mgr", ""]}]: dispatch 2024-05-27T07:26:13.352 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:26:12 smithi173 ceph-mon[37000]: scrub queued for path: / 2024-05-27T07:26:13.352 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:26:12 smithi173 ceph-mon[37000]: scrub summary: idle+waiting paths [/] 2024-05-27T07:26:13.352 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:26:12 smithi173 ceph-mon[37000]: scrub summary: active paths [/] 2024-05-27T07:26:13.352 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:26:12 smithi173 ceph-mon[37000]: scrub summary: idle+waiting paths [/] 2024-05-27T07:26:13.487 INFO:teuthology.orchestra.run.smithi052.stderr:Inferring config /var/lib/ceph/73df2636-1bf8-11ef-bc9b-c7b262605968/mon.a/config 2024-05-27T07:26:14.190 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:26:13 smithi105 ceph-mon[35977]: from='client.15501 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule status", "fs": "cephfs", "path": "/", "target": ["mon-mgr", ""]}]: dispatch 2024-05-27T07:26:14.190 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:26:13 smithi105 ceph-mon[35977]: from='client.15525 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule list", "fs": "cephfs", "path": "/", "recursive": true, "target": ["mon-mgr", ""]}]: dispatch 2024-05-27T07:26:14.190 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:26:13 smithi105 ceph-mon[35977]: pgmap v11: 129 pgs: 129 active+clean; 1.1 MiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 597 B/s wr, 16 op/s 2024-05-27T07:26:14.229 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:26:13 smithi052 ceph-mon[31955]: from='client.15501 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule status", "fs": "cephfs", "path": "/", "target": ["mon-mgr", ""]}]: dispatch 2024-05-27T07:26:14.229 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:26:13 smithi052 ceph-mon[31955]: from='client.15525 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule list", "fs": "cephfs", "path": "/", "recursive": true, "target": ["mon-mgr", ""]}]: dispatch 2024-05-27T07:26:14.229 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:26:13 smithi052 ceph-mon[31955]: pgmap v11: 129 pgs: 129 active+clean; 1.1 MiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 597 B/s wr, 16 op/s 2024-05-27T07:26:14.352 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:26:13 smithi173 ceph-mon[37000]: from='client.15501 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule status", "fs": "cephfs", "path": "/", "target": ["mon-mgr", ""]}]: dispatch 2024-05-27T07:26:14.352 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:26:13 smithi173 ceph-mon[37000]: from='client.15525 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule list", "fs": "cephfs", "path": "/", "recursive": true, "target": ["mon-mgr", ""]}]: dispatch 2024-05-27T07:26:14.352 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:26:13 smithi173 ceph-mon[37000]: pgmap v11: 129 pgs: 129 active+clean; 1.1 MiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 597 B/s wr, 16 op/s 2024-05-27T07:26:15.155 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:15.165+0000 7fc8e7d1d700 1 -- 172.21.15.52:0/3244183319 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fc8e0109860 msgr2=0x7fc8e01111c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:26:15.155 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:15.165+0000 7fc8e7d1d700 1 --2- 172.21.15.52:0/3244183319 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fc8e0109860 0x7fc8e01111c0 secure :-1 s=READY pgs=441 cs=0 l=1 rev1=1 crypto rx=0x7fc8d800b0d0 tx=0x7fc8d800b3e0 comp rx=0 tx=0).stop 2024-05-27T07:26:15.155 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:15.165+0000 7fc8e7d1d700 1 -- 172.21.15.52:0/3244183319 shutdown_connections 2024-05-27T07:26:15.155 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:15.165+0000 7fc8e7d1d700 1 --2- 172.21.15.52:0/3244183319 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fc8e0109860 0x7fc8e01111c0 unknown :-1 s=CLOSED pgs=441 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:26:15.155 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:15.165+0000 7fc8e7d1d700 1 --2- 172.21.15.52:0/3244183319 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fc8e01045e0 0x7fc8e01049f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:26:15.155 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:15.165+0000 7fc8e7d1d700 1 --2- 172.21.15.52:0/3244183319 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fc8e0108f10 0x7fc8e0109320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:26:15.156 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:15.165+0000 7fc8e7d1d700 1 -- 172.21.15.52:0/3244183319 >> 172.21.15.52:0/3244183319 conn(0x7fc8e00ffeb0 msgr2=0x7fc8e0102300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:26:15.156 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:15.165+0000 7fc8e7d1d700 1 -- 172.21.15.52:0/3244183319 shutdown_connections 2024-05-27T07:26:15.156 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:15.165+0000 7fc8e7d1d700 1 -- 172.21.15.52:0/3244183319 wait complete. 2024-05-27T07:26:15.157 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:15.166+0000 7fc8e7d1d700 1 Processor -- start 2024-05-27T07:26:15.157 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:15.166+0000 7fc8e7d1d700 1 -- start start 2024-05-27T07:26:15.157 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:15.166+0000 7fc8e7d1d700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fc8e01045e0 0x7fc8e019ade0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:26:15.157 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:15.166+0000 7fc8e7d1d700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fc8e0108f10 0x7fc8e019b380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:26:15.158 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:15.166+0000 7fc8e7d1d700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fc8e0109860 0x7fc8e019f610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:26:15.158 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:15.166+0000 7fc8e7d1d700 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x7fc8e0114490 con 0x7fc8e0109860 2024-05-27T07:26:15.158 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:15.166+0000 7fc8e7d1d700 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7fc8e0114310 con 0x7fc8e0108f10 2024-05-27T07:26:15.158 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:15.166+0000 7fc8e7d1d700 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7fc8e0114610 con 0x7fc8e01045e0 2024-05-27T07:26:15.158 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:15.166+0000 7fc8e62ba700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fc8e0109860 0x7fc8e019f610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:26:15.158 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:15.166+0000 7fc8e62ba700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fc8e0109860 0x7fc8e019f610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.52:49896/0 (socket says 172.21.15.52:49896) 2024-05-27T07:26:15.158 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:15.166+0000 7fc8e62ba700 1 -- 172.21.15.52:0/1505576564 learned_addr learned my addr 172.21.15.52:0/1505576564 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-05-27T07:26:15.158 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:15.166+0000 7fc8e52b8700 1 --2- 172.21.15.52:0/1505576564 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fc8e0108f10 0x7fc8e019b380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:26:15.158 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:15.167+0000 7fc8e5ab9700 1 --2- 172.21.15.52:0/1505576564 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fc8e01045e0 0x7fc8e019ade0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:26:15.158 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:15.167+0000 7fc8e62ba700 1 -- 172.21.15.52:0/1505576564 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fc8e01045e0 msgr2=0x7fc8e019ade0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:26:15.158 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:15.167+0000 7fc8e62ba700 1 --2- 172.21.15.52:0/1505576564 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fc8e01045e0 0x7fc8e019ade0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:26:15.158 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:15.167+0000 7fc8e62ba700 1 -- 172.21.15.52:0/1505576564 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fc8e0108f10 msgr2=0x7fc8e019b380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:26:15.159 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:15.167+0000 7fc8e62ba700 1 --2- 172.21.15.52:0/1505576564 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fc8e0108f10 0x7fc8e019b380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:26:15.159 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:15.167+0000 7fc8e62ba700 1 -- 172.21.15.52:0/1505576564 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc8e019fc00 con 0x7fc8e0109860 2024-05-27T07:26:15.159 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:15.167+0000 7fc8e52b8700 1 --2- 172.21.15.52:0/1505576564 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fc8e0108f10 0x7fc8e019b380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-27T07:26:15.159 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:15.167+0000 7fc8e62ba700 1 --2- 172.21.15.52:0/1505576564 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fc8e0109860 0x7fc8e019f610 secure :-1 s=READY pgs=442 cs=0 l=1 rev1=1 crypto rx=0x7fc8d8009850 tx=0x7fc8d8007a20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:26:15.159 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:15.167+0000 7fc8d6ffd700 1 -- 172.21.15.52:0/1505576564 <== mon.0 v2:172.21.15.52:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc8d801f070 con 0x7fc8e0109860 2024-05-27T07:26:15.159 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:15.167+0000 7fc8e7d1d700 1 -- 172.21.15.52:0/1505576564 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc8e019fe90 con 0x7fc8e0109860 2024-05-27T07:26:15.159 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:15.167+0000 7fc8d6ffd700 1 -- 172.21.15.52:0/1505576564 <== mon.0 v2:172.21.15.52:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fc8d8004c50 con 0x7fc8e0109860 2024-05-27T07:26:15.159 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:15.167+0000 7fc8d6ffd700 1 -- 172.21.15.52:0/1505576564 <== mon.0 v2:172.21.15.52:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc8d801b7b0 con 0x7fc8e0109860 2024-05-27T07:26:15.159 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:15.167+0000 7fc8e7d1d700 1 -- 172.21.15.52:0/1505576564 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc8e01a03c0 con 0x7fc8e0109860 2024-05-27T07:26:15.160 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:15.170+0000 7fc8d6ffd700 1 -- 172.21.15.52:0/1505576564 <== mon.0 v2:172.21.15.52:3300/0 4 ==== mgrmap(e 24) ==== 149879+0+0 (secure 0 0 0) 0x7fc8d8022070 con 0x7fc8e0109860 2024-05-27T07:26:15.161 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:15.171+0000 7fc8d6ffd700 1 --2- 172.21.15.52:0/1505576564 >> [v2:172.21.15.52:6800/1822318384,v1:172.21.15.52:6801/1822318384] conn(0x7fc8cc0b2b10 0x7fc8cc0b5020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:26:15.161 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:15.171+0000 7fc8e5ab9700 1 --2- 172.21.15.52:0/1505576564 >> [v2:172.21.15.52:6800/1822318384,v1:172.21.15.52:6801/1822318384] conn(0x7fc8cc0b2b10 0x7fc8cc0b5020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:26:15.161 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:15.171+0000 7fc8d6ffd700 1 -- 172.21.15.52:0/1505576564 <== mon.0 v2:172.21.15.52:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9379+0+0 (secure 0 0 0) 0x7fc8d800f020 con 0x7fc8e0109860 2024-05-27T07:26:15.161 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:15.171+0000 7fc8f6732b80 1 Processor -- start 2024-05-27T07:26:15.163 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:15.171+0000 7fc8f6732b80 1 -- start start 2024-05-27T07:26:15.163 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:15.172+0000 7fc8e5ab9700 1 --2- 172.21.15.52:0/1505576564 >> [v2:172.21.15.52:6800/1822318384,v1:172.21.15.52:6801/1822318384] conn(0x7fc8cc0b2b10 0x7fc8cc0b5020 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7fc8dc005ea0 tx=0x7fc8dc005e30 comp rx=0 tx=0).ready entity=mgr.15333 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:26:15.163 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:15.172+0000 7fc8f6732b80 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55b6f6197e40 0x55b6f631cb30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:26:15.163 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:15.172+0000 7fc8f6732b80 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55b6f631d070 0x55b6f6316be0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:26:15.163 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:15.172+0000 7fc8f6732b80 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55b6f6317120 0x55b6f6317530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:26:15.163 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:15.172+0000 7fc8f6732b80 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x55b6f6244d70 con 0x55b6f631d070 2024-05-27T07:26:15.163 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:15.172+0000 7fc8f6732b80 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x55b6f6234910 con 0x55b6f6197e40 2024-05-27T07:26:15.163 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:15.172+0000 7fc8f6732b80 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x55b6f6238950 con 0x55b6f6317120 2024-05-27T07:26:15.163 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:15.172+0000 7fc8e52b8700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55b6f631d070 0x55b6f6316be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:26:15.163 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:15.172+0000 7fc8e52b8700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55b6f631d070 0x55b6f6316be0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.52:49908/0 (socket says 172.21.15.52:49908) 2024-05-27T07:26:15.163 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:15.172+0000 7fc8e52b8700 1 -- 172.21.15.52:0/4229717705 learned_addr learned my addr 172.21.15.52:0/4229717705 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-05-27T07:26:15.163 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:15.172+0000 7fc8e62ba700 1 --2- 172.21.15.52:0/4229717705 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55b6f6317120 0x55b6f6317530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:26:15.164 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:15.172+0000 7fc8e5ab9700 1 --2- 172.21.15.52:0/4229717705 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55b6f6197e40 0x55b6f631cb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:26:15.164 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:15.172+0000 7fc8e52b8700 1 -- 172.21.15.52:0/4229717705 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55b6f6317120 msgr2=0x55b6f6317530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:26:15.164 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:15.172+0000 7fc8e52b8700 1 --2- 172.21.15.52:0/4229717705 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55b6f6317120 0x55b6f6317530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:26:15.164 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:15.172+0000 7fc8e52b8700 1 -- 172.21.15.52:0/4229717705 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55b6f6197e40 msgr2=0x55b6f631cb30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:26:15.164 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:15.172+0000 7fc8e52b8700 1 --2- 172.21.15.52:0/4229717705 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55b6f6197e40 0x55b6f631cb30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:26:15.164 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:15.172+0000 7fc8e52b8700 1 -- 172.21.15.52:0/4229717705 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55b6f61ddfb0 con 0x55b6f631d070 2024-05-27T07:26:15.164 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:15.172+0000 7fc8e5ab9700 1 --2- 172.21.15.52:0/4229717705 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55b6f6197e40 0x55b6f631cb30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-27T07:26:15.164 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:15.172+0000 7fc8e62ba700 1 --2- 172.21.15.52:0/4229717705 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55b6f6317120 0x55b6f6317530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-27T07:26:15.164 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:15.172+0000 7fc8e52b8700 1 --2- 172.21.15.52:0/4229717705 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55b6f631d070 0x55b6f6316be0 secure :-1 s=READY pgs=443 cs=0 l=1 rev1=1 crypto rx=0x7fc8d00114f0 tx=0x7fc8d0011800 comp rx=0 tx=0).ready entity=mon.0 client_cookie=edd2b8b593720b0c server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:26:15.164 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:15.173+0000 7fc8cbfff700 1 -- 172.21.15.52:0/4229717705 <== mon.0 v2:172.21.15.52:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc8d0004a30 con 0x55b6f631d070 2024-05-27T07:26:15.164 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:15.173+0000 7fc8cbfff700 1 -- 172.21.15.52:0/4229717705 <== mon.0 v2:172.21.15.52:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fc8d00054d0 con 0x55b6f631d070 2024-05-27T07:26:15.164 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:15.173+0000 7fc8cbfff700 1 -- 172.21.15.52:0/4229717705 <== mon.0 v2:172.21.15.52:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc8d000e970 con 0x55b6f631d070 2024-05-27T07:26:15.169 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:15.174+0000 7fc8f6732b80 1 -- 172.21.15.52:0/4229717705 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55b6f631d070 msgr2=0x55b6f6316be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:26:15.169 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:15.174+0000 7fc8f6732b80 1 --2- 172.21.15.52:0/4229717705 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55b6f631d070 0x55b6f6316be0 secure :-1 s=READY pgs=443 cs=0 l=1 rev1=1 crypto rx=0x7fc8d00114f0 tx=0x7fc8d0011800 comp rx=0 tx=0).stop 2024-05-27T07:26:15.169 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:15.174+0000 7fc8f6732b80 1 -- 172.21.15.52:0/4229717705 shutdown_connections 2024-05-27T07:26:15.169 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:15.174+0000 7fc8f6732b80 1 --2- 172.21.15.52:0/4229717705 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55b6f6317120 0x55b6f6317530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:26:15.169 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:15.174+0000 7fc8f6732b80 1 --2- 172.21.15.52:0/4229717705 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55b6f6197e40 0x55b6f631cb30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:26:15.169 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:15.174+0000 7fc8f6732b80 1 --2- 172.21.15.52:0/4229717705 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55b6f631d070 0x55b6f6316be0 unknown :-1 s=CLOSED pgs=443 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:26:15.169 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:15.174+0000 7fc8f6732b80 1 -- 172.21.15.52:0/4229717705 >> 172.21.15.52:0/4229717705 conn(0x55b6f619a770 msgr2=0x55b6f6311d90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:26:15.169 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:15.174+0000 7fc8f6732b80 1 -- 172.21.15.52:0/4229717705 shutdown_connections 2024-05-27T07:26:15.169 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:15.174+0000 7fc8f6732b80 1 -- 172.21.15.52:0/4229717705 wait complete. 2024-05-27T07:26:15.169 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:15.174+0000 7fc8f6732b80 1 -- start start 2024-05-27T07:26:15.169 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:15.174+0000 7fc8f6732b80 1 Processor -- start 2024-05-27T07:26:15.169 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:15.175+0000 7fc8f6732b80 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55b6f6197e40 0x55b6f6320890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:26:15.170 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:15.175+0000 7fc8f6732b80 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55b6f6317120 0x55b6f6320dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:26:15.170 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:15.175+0000 7fc8f6732b80 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55b6f631d070 0x55b6f6321310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:26:15.170 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:15.175+0000 7fc8f6732b80 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x55b6f613afa0 con 0x55b6f631d070 2024-05-27T07:26:15.170 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:15.175+0000 7fc8f6732b80 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x55b6f619cda0 con 0x55b6f6197e40 2024-05-27T07:26:15.170 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:15.175+0000 7fc8f6732b80 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x55b6f6323b60 con 0x55b6f6317120 2024-05-27T07:26:15.170 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:15.175+0000 7fc8f6732b80 10 client.0 _ll_register_callbacks cb 0x55b6f61cb5b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T07:26:15.170 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:15.175+0000 7fc8e62ba700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55b6f631d070 0x55b6f6321310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:26:15.170 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:15.175+0000 7fc8e62ba700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55b6f631d070 0x55b6f6321310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.52:49918/0 (socket says 172.21.15.52:49918) 2024-05-27T07:26:15.170 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:15.175+0000 7fc8e62ba700 1 -- 172.21.15.52:0/433608600 learned_addr learned my addr 172.21.15.52:0/433608600 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-05-27T07:26:15.170 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:15.175+0000 7fc8e5ab9700 1 --2- 172.21.15.52:0/433608600 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55b6f6197e40 0x55b6f6320890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:26:15.170 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:15.175+0000 7fc8e52b8700 1 --2- 172.21.15.52:0/433608600 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55b6f6317120 0x55b6f6320dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:26:15.170 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:15.175+0000 7fc8e62ba700 1 -- 172.21.15.52:0/433608600 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55b6f6317120 msgr2=0x55b6f6320dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:26:15.171 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:15.175+0000 7fc8e62ba700 1 --2- 172.21.15.52:0/433608600 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55b6f6317120 0x55b6f6320dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:26:15.171 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:15.175+0000 7fc8e62ba700 1 -- 172.21.15.52:0/433608600 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55b6f6197e40 msgr2=0x55b6f6320890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:26:15.171 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:15.175+0000 7fc8e62ba700 1 --2- 172.21.15.52:0/433608600 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55b6f6197e40 0x55b6f6320890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:26:15.171 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:15.175+0000 7fc8e62ba700 1 -- 172.21.15.52:0/433608600 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55b6f61848d0 con 0x55b6f631d070 2024-05-27T07:26:15.171 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:15.176+0000 7fc8e62ba700 1 --2- 172.21.15.52:0/433608600 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55b6f631d070 0x55b6f6321310 secure :-1 s=READY pgs=444 cs=0 l=1 rev1=1 crypto rx=0x7fc8d8009fd0 tx=0x7fc8d80058c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=50613af57d480483 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:26:15.171 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:15.176+0000 7fc8c97fa700 10 client.0 ms_handle_connect on v2:172.21.15.52:3300/0 2024-05-27T07:26:15.171 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:15.176+0000 7fc8c97fa700 1 -- 172.21.15.52:0/433608600 <== mon.0 v2:172.21.15.52:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc8d80aa650 con 0x55b6f631d070 2024-05-27T07:26:15.171 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:15.176+0000 7fc8c97fa700 1 -- 172.21.15.52:0/433608600 <== mon.0 v2:172.21.15.52:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 9379+0+0 (secure 0 0 0) 0x7fc8d8022e20 con 0x55b6f631d070 2024-05-27T07:26:15.171 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:15.176+0000 7fc8f6732b80 1 -- 172.21.15.52:0/433608600 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fc8d00054d0 con 0x55b6f631d070 2024-05-27T07:26:15.171 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:15.176+0000 7fc8c97fa700 1 -- 172.21.15.52:0/433608600 <== mon.0 v2:172.21.15.52:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7fc8d80ab240 con 0x55b6f631d070 2024-05-27T07:26:15.171 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:15.176+0000 7fc8f6732b80 10 client.15537 fetch_fsmap learned FSMap version 30 2024-05-27T07:26:15.171 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:15.176+0000 7fc8f6732b80 1 -- 172.21.15.52:0/433608600 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x55b6f61ee760 con 0x55b6f631d070 2024-05-27T07:26:15.171 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:15.176+0000 7fc8c97fa700 1 -- 172.21.15.52:0/433608600 <== mon.0 v2:172.21.15.52:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fc8d00054d0 con 0x55b6f631d070 2024-05-27T07:26:15.172 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:15.176+0000 7fc8c97fa700 1 -- 172.21.15.52:0/433608600 <== mon.0 v2:172.21.15.52:3300/0 5 ==== fsmap(e 30) ==== 8362+0+0 (secure 0 0 0) 0x7fc8d8011070 con 0x55b6f631d070 2024-05-27T07:26:15.172 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:15.176+0000 7fc8f6732b80 10 client.15537 fetch_fsmap finished waiting for FSMap version 30 2024-05-27T07:26:15.172 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:15.176+0000 7fc8f6732b80 10 client.15537 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-05-27T07:26:15.172 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:15.176+0000 7fc8f6732b80 20 client.15537 populate_metadata read hostname 'smithi052' 2024-05-27T07:26:15.172 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:15.176+0000 7fc8f6732b80 1 --2- 172.21.15.52:0/433608600 >> [v2:172.21.15.105:6838/500348151,v1:172.21.15.105:6839/500348151] conn(0x55b6f6330a00 0x55b6f6332e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:26:15.172 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:15.176+0000 7fc8f6732b80 4 client.15537 mds_command: new command op to 24527 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-27T07:26:15.172 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:15.176+0000 7fc8f6732b80 1 -- 172.21.15.52:0/433608600 --> [v2:172.21.15.105:6838/500348151,v1:172.21.15.105:6839/500348151] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55b6f6246cc0 con 0x55b6f6330a00 2024-05-27T07:26:15.172 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:15.177+0000 7fc8e5ab9700 1 --2- 172.21.15.52:0/433608600 >> [v2:172.21.15.105:6838/500348151,v1:172.21.15.105:6839/500348151] conn(0x55b6f6330a00 0x55b6f6332e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:26:15.172 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:15.177+0000 7fc8e5ab9700 1 --2- 172.21.15.52:0/433608600 >> [v2:172.21.15.105:6838/500348151,v1:172.21.15.105:6839/500348151] conn(0x55b6f6330a00 0x55b6f6332e80 crc :-1 s=READY pgs=44 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=27a2b0deb589a7c5 server_cookie=d7b8045a056ebdc7 in_seq=0 out_seq=0 2024-05-27T07:26:15.172 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:15.177+0000 7fc8c97fa700 10 client.15537 ms_handle_connect on v2:172.21.15.105:6838/500348151 2024-05-27T07:26:15.172 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:15.178+0000 7fc8c97fa700 1 -- 172.21.15.52:0/433608600 <== mds.0 v2:172.21.15.105:6838/500348151 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24041 (crc 0 0 0) 0x55b6f6246cc0 con 0x55b6f6330a00 2024-05-27T07:26:15.172 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:15.178+0000 7fc8c97fa700 10 client.15537 handle_command_reply: tid=0 2024-05-27T07:26:15.172 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:15.178+0000 7fc8f6732b80 1 client.15537 shutdown 2024-05-27T07:26:15.173 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:15.178+0000 7fc8f6732b80 1 -- 172.21.15.52:0/433608600 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55b6f631d070 msgr2=0x55b6f6321310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:26:15.173 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:15.178+0000 7fc8f6732b80 1 --2- 172.21.15.52:0/433608600 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55b6f631d070 0x55b6f6321310 secure :-1 s=READY pgs=444 cs=0 l=1 rev1=1 crypto rx=0x7fc8d8009fd0 tx=0x7fc8d80058c0 comp rx=0 tx=0).stop 2024-05-27T07:26:15.173 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:15.178+0000 7fc8f6732b80 1 -- 172.21.15.52:0/433608600 shutdown_connections 2024-05-27T07:26:15.173 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:15.178+0000 7fc8f6732b80 1 --2- 172.21.15.52:0/433608600 >> [v2:172.21.15.105:6838/500348151,v1:172.21.15.105:6839/500348151] conn(0x55b6f6330a00 0x55b6f6332e80 crc :-1 s=READY pgs=44 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:26:15.173 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:15.178+0000 7fc8f6732b80 1 --2- 172.21.15.52:0/433608600 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55b6f631d070 0x55b6f6321310 unknown :-1 s=CLOSED pgs=444 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:26:15.173 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:15.178+0000 7fc8f6732b80 1 --2- 172.21.15.52:0/433608600 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55b6f6197e40 0x55b6f6320890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:26:15.173 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:15.178+0000 7fc8f6732b80 1 --2- 172.21.15.52:0/433608600 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55b6f6317120 0x55b6f6320dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:26:15.173 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:15.178+0000 7fc8c97fa700 0 client.15537 ms_handle_reset on v2:172.21.15.105:6838/500348151 2024-05-27T07:26:15.173 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:15.178+0000 7fc8f6732b80 1 -- 172.21.15.52:0/433608600 >> 172.21.15.52:0/433608600 conn(0x55b6f619a770 msgr2=0x55b6f6315bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:26:15.173 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:15.179+0000 7fc8f6732b80 1 -- 172.21.15.52:0/433608600 shutdown_connections 2024-05-27T07:26:15.173 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:15.179+0000 7fc8f6732b80 1 -- 172.21.15.52:0/433608600 wait complete. 2024-05-27T07:26:15.173 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:15.179+0000 7fc8f6732b80 20 client.15537 trim_cache size 0 max 16384 2024-05-27T07:26:15.190 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:15.200+0000 7fc8f6732b80 1 Processor -- start 2024-05-27T07:26:15.191 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:15.200+0000 7fc8f6732b80 1 -- start start 2024-05-27T07:26:15.191 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:15.200+0000 7fc8f6732b80 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55b6f6197e40 0x55b6f6316be0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:26:15.191 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:15.200+0000 7fc8f6732b80 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55b6f6317120 0x55b6f630bc00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:26:15.191 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:15.200+0000 7fc8f6732b80 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55b6f631d070 0x55b6f6320e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:26:15.191 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:15.200+0000 7fc8f6732b80 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x7fc8e01b0330 con 0x55b6f6197e40 2024-05-27T07:26:15.191 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:15.200+0000 7fc8f6732b80 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7fc8e01b1e70 con 0x55b6f6317120 2024-05-27T07:26:15.191 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:15.200+0000 7fc8f6732b80 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7fc8e01b0a00 con 0x55b6f631d070 2024-05-27T07:26:15.191 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:15.200+0000 7fc8e5ab9700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55b6f6197e40 0x55b6f6316be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:26:15.191 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:15.200+0000 7fc8e5ab9700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55b6f6197e40 0x55b6f6316be0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.52:49922/0 (socket says 172.21.15.52:49922) 2024-05-27T07:26:15.191 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:15.200+0000 7fc8e5ab9700 1 -- 172.21.15.52:0/807250602 learned_addr learned my addr 172.21.15.52:0/807250602 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-05-27T07:26:15.191 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:15.200+0000 7fc8e52b8700 1 --2- 172.21.15.52:0/807250602 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55b6f6317120 0x55b6f630bc00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:26:15.191 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:15.200+0000 7fc8e62ba700 1 --2- 172.21.15.52:0/807250602 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55b6f631d070 0x55b6f6320e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:26:15.192 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:15.200+0000 7fc8e5ab9700 1 -- 172.21.15.52:0/807250602 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55b6f631d070 msgr2=0x55b6f6320e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:26:15.192 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:15.200+0000 7fc8e5ab9700 1 --2- 172.21.15.52:0/807250602 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55b6f631d070 0x55b6f6320e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:26:15.192 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:15.200+0000 7fc8e5ab9700 1 -- 172.21.15.52:0/807250602 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55b6f6317120 msgr2=0x55b6f630bc00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:26:15.192 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:15.200+0000 7fc8e5ab9700 1 --2- 172.21.15.52:0/807250602 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55b6f6317120 0x55b6f630bc00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:26:15.192 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:15.200+0000 7fc8e5ab9700 1 -- 172.21.15.52:0/807250602 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55b6f631ee00 con 0x55b6f6197e40 2024-05-27T07:26:15.192 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:15.201+0000 7fc8e52b8700 1 --2- 172.21.15.52:0/807250602 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55b6f6317120 0x55b6f630bc00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-27T07:26:15.192 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:15.201+0000 7fc8e5ab9700 1 --2- 172.21.15.52:0/807250602 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55b6f6197e40 0x55b6f6316be0 secure :-1 s=READY pgs=445 cs=0 l=1 rev1=1 crypto rx=0x55b6f63335e0 tx=0x7fc8dc00af90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b379143a565c8db2 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:26:15.192 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:15.201+0000 7fc8cb7fe700 1 -- 172.21.15.52:0/807250602 <== mon.0 v2:172.21.15.52:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc8dc02d070 con 0x55b6f6197e40 2024-05-27T07:26:15.192 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:15.201+0000 7fc8cb7fe700 1 -- 172.21.15.52:0/807250602 <== mon.0 v2:172.21.15.52:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fc8dc0272d0 con 0x55b6f6197e40 2024-05-27T07:26:15.192 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:15.201+0000 7fc8cb7fe700 1 -- 172.21.15.52:0/807250602 <== mon.0 v2:172.21.15.52:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc8dc01e670 con 0x55b6f6197e40 2024-05-27T07:26:15.196 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:15.202+0000 7fc8f6732b80 1 -- 172.21.15.52:0/807250602 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55b6f6197e40 msgr2=0x55b6f6316be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:26:15.196 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:15.202+0000 7fc8f6732b80 1 --2- 172.21.15.52:0/807250602 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55b6f6197e40 0x55b6f6316be0 secure :-1 s=READY pgs=445 cs=0 l=1 rev1=1 crypto rx=0x55b6f63335e0 tx=0x7fc8dc00af90 comp rx=0 tx=0).stop 2024-05-27T07:26:15.196 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:15.202+0000 7fc8f6732b80 1 -- 172.21.15.52:0/807250602 shutdown_connections 2024-05-27T07:26:15.196 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:15.202+0000 7fc8f6732b80 1 --2- 172.21.15.52:0/807250602 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55b6f631d070 0x55b6f6320e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:26:15.196 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:15.202+0000 7fc8f6732b80 1 --2- 172.21.15.52:0/807250602 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55b6f6197e40 0x55b6f6316be0 unknown :-1 s=CLOSED pgs=445 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:26:15.196 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:15.202+0000 7fc8f6732b80 1 --2- 172.21.15.52:0/807250602 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55b6f6317120 0x55b6f630bc00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:26:15.196 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:15.202+0000 7fc8f6732b80 1 -- 172.21.15.52:0/807250602 >> 172.21.15.52:0/807250602 conn(0x55b6f619a770 msgr2=0x55b6f630b4f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:26:15.196 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:15.202+0000 7fc8f6732b80 1 -- 172.21.15.52:0/807250602 shutdown_connections 2024-05-27T07:26:15.196 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:15.202+0000 7fc8f6732b80 1 -- 172.21.15.52:0/807250602 wait complete. 2024-05-27T07:26:15.196 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:15.202+0000 7fc8f6732b80 1 -- start start 2024-05-27T07:26:15.196 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:15.202+0000 7fc8f6732b80 1 Processor -- start 2024-05-27T07:26:15.196 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:15.203+0000 7fc8f6732b80 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55b6f6197e40 0x55b6f63292c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:26:15.197 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:15.203+0000 7fc8f6732b80 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55b6f6317120 0x55b6f6325dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:26:15.197 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:15.203+0000 7fc8f6732b80 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55b6f631d070 0x55b6f6326300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:26:15.197 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:15.203+0000 7fc8f6732b80 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x7fc8e01b17a0 con 0x55b6f631d070 2024-05-27T07:26:15.197 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:15.203+0000 7fc8f6732b80 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7fc8e01b10d0 con 0x55b6f6317120 2024-05-27T07:26:15.197 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:15.203+0000 7fc8f6732b80 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7fc8e01ae2d0 con 0x55b6f6197e40 2024-05-27T07:26:15.197 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:15.203+0000 7fc8f6732b80 10 client.0 _ll_register_callbacks cb 0x55b6f61cb5b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T07:26:15.197 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:15.203+0000 7fc8e62ba700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55b6f631d070 0x55b6f6326300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:26:15.197 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:15.203+0000 7fc8e62ba700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55b6f631d070 0x55b6f6326300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.52:49928/0 (socket says 172.21.15.52:49928) 2024-05-27T07:26:15.197 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:15.203+0000 7fc8e62ba700 1 -- 172.21.15.52:0/3001065169 learned_addr learned my addr 172.21.15.52:0/3001065169 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-05-27T07:26:15.197 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:15.203+0000 7fc8e52b8700 1 --2- 172.21.15.52:0/3001065169 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55b6f6317120 0x55b6f6325dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:26:15.197 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:15.203+0000 7fc8e5ab9700 1 --2- 172.21.15.52:0/3001065169 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55b6f6197e40 0x55b6f63292c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:26:15.197 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:15.203+0000 7fc8e62ba700 1 -- 172.21.15.52:0/3001065169 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55b6f6197e40 msgr2=0x55b6f63292c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:26:15.197 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:15.203+0000 7fc8e62ba700 1 --2- 172.21.15.52:0/3001065169 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55b6f6197e40 0x55b6f63292c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:26:15.198 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:15.203+0000 7fc8e62ba700 1 -- 172.21.15.52:0/3001065169 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55b6f6317120 msgr2=0x55b6f6325dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:26:15.198 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:15.203+0000 7fc8e62ba700 1 --2- 172.21.15.52:0/3001065169 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55b6f6317120 0x55b6f6325dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:26:15.198 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:15.203+0000 7fc8e62ba700 1 -- 172.21.15.52:0/3001065169 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55b6f6326840 con 0x55b6f631d070 2024-05-27T07:26:15.198 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:15.204+0000 7fc8e62ba700 1 --2- 172.21.15.52:0/3001065169 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55b6f631d070 0x55b6f6326300 secure :-1 s=READY pgs=446 cs=0 l=1 rev1=1 crypto rx=0x7fc8d800bfd0 tx=0x7fc8d8005e10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1d9367337a6b8f60 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:26:15.198 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:15.204+0000 7fc8c97fa700 10 client.0 ms_handle_connect on v2:172.21.15.52:3300/0 2024-05-27T07:26:15.198 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:15.204+0000 7fc8c97fa700 1 -- 172.21.15.52:0/3001065169 <== mon.0 v2:172.21.15.52:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc8d806f060 con 0x55b6f631d070 2024-05-27T07:26:15.198 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:15.204+0000 7fc8c97fa700 1 -- 172.21.15.52:0/3001065169 <== mon.0 v2:172.21.15.52:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 9379+0+0 (secure 0 0 0) 0x7fc8d80a7130 con 0x55b6f631d070 2024-05-27T07:26:15.198 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:15.204+0000 7fc8f6732b80 1 -- 172.21.15.52:0/3001065169 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fc8dc0272d0 con 0x55b6f631d070 2024-05-27T07:26:15.198 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:15.204+0000 7fc8c97fa700 1 -- 172.21.15.52:0/3001065169 <== mon.0 v2:172.21.15.52:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7fc8d80aa650 con 0x55b6f631d070 2024-05-27T07:26:15.198 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:15.204+0000 7fc8f6732b80 10 client.15543 fetch_fsmap learned FSMap version 30 2024-05-27T07:26:15.198 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:15.204+0000 7fc8f6732b80 1 -- 172.21.15.52:0/3001065169 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x55b6f639c170 con 0x55b6f631d070 2024-05-27T07:26:15.198 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:15.204+0000 7fc8c97fa700 1 -- 172.21.15.52:0/3001065169 <== mon.0 v2:172.21.15.52:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fc8dc0272d0 con 0x55b6f631d070 2024-05-27T07:26:15.198 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:15.204+0000 7fc8c97fa700 1 -- 172.21.15.52:0/3001065169 <== mon.0 v2:172.21.15.52:3300/0 5 ==== fsmap(e 30) ==== 8362+0+0 (secure 0 0 0) 0x7fc8d8011040 con 0x55b6f631d070 2024-05-27T07:26:15.199 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:15.204+0000 7fc8f6732b80 10 client.15543 fetch_fsmap finished waiting for FSMap version 30 2024-05-27T07:26:15.199 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:15.204+0000 7fc8f6732b80 10 client.15543 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-05-27T07:26:15.199 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:15.204+0000 7fc8f6732b80 20 client.15543 populate_metadata read hostname 'smithi052' 2024-05-27T07:26:15.199 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:15.204+0000 7fc8f6732b80 1 --2- 172.21.15.52:0/3001065169 >> [v2:172.21.15.105:6838/500348151,v1:172.21.15.105:6839/500348151] conn(0x55b6f639c690 0x55b6f639eab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:26:15.199 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:15.205+0000 7fc8f6732b80 4 client.15543 mds_command: new command op to 24527 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-05-27T07:26:15.199 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:15.205+0000 7fc8f6732b80 1 -- 172.21.15.52:0/3001065169 --> [v2:172.21.15.105:6838/500348151,v1:172.21.15.105:6839/500348151] -- command(tid 0: {"prefix": "scrub status"}) -- 0x55b6f62404a0 con 0x55b6f639c690 2024-05-27T07:26:15.199 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:15.205+0000 7fc8e5ab9700 1 --2- 172.21.15.52:0/3001065169 >> [v2:172.21.15.105:6838/500348151,v1:172.21.15.105:6839/500348151] conn(0x55b6f639c690 0x55b6f639eab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:26:15.199 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:15.205+0000 7fc8c97fa700 10 client.15543 ms_handle_connect on v2:172.21.15.105:6838/500348151 2024-05-27T07:26:15.199 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:15.205+0000 7fc8e5ab9700 1 --2- 172.21.15.52:0/3001065169 >> [v2:172.21.15.105:6838/500348151,v1:172.21.15.105:6839/500348151] conn(0x55b6f639c690 0x55b6f639eab0 crc :-1 s=READY pgs=45 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=89822b155bae4a1 server_cookie=1baead46632987ed in_seq=0 out_seq=0 2024-05-27T07:26:15.199 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:15.206+0000 7fc8c97fa700 1 -- 172.21.15.52:0/3001065169 <== mds.0 v2:172.21.15.105:6838/500348151 1 ==== command_reply(tid 0: 0 ) ==== 8+0+259 (crc 0 0 0) 0x55b6f62404a0 con 0x55b6f639c690 2024-05-27T07:26:15.199 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:15.206+0000 7fc8c97fa700 10 client.15543 handle_command_reply: tid=0 2024-05-27T07:26:15.199 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:15.206+0000 7fc8f6732b80 1 client.15543 shutdown 2024-05-27T07:26:15.200 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:15.206+0000 7fc8f6732b80 1 -- 172.21.15.52:0/3001065169 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55b6f631d070 msgr2=0x55b6f6326300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:26:15.200 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:15.206+0000 7fc8f6732b80 1 --2- 172.21.15.52:0/3001065169 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55b6f631d070 0x55b6f6326300 secure :-1 s=READY pgs=446 cs=0 l=1 rev1=1 crypto rx=0x7fc8d800bfd0 tx=0x7fc8d8005e10 comp rx=0 tx=0).stop 2024-05-27T07:26:15.200 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:15.206+0000 7fc8f6732b80 1 -- 172.21.15.52:0/3001065169 shutdown_connections 2024-05-27T07:26:15.200 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:15.206+0000 7fc8f6732b80 1 --2- 172.21.15.52:0/3001065169 >> [v2:172.21.15.105:6838/500348151,v1:172.21.15.105:6839/500348151] conn(0x55b6f639c690 0x55b6f639eab0 crc :-1 s=READY pgs=45 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:26:15.200 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:15.206+0000 7fc8f6732b80 1 --2- 172.21.15.52:0/3001065169 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55b6f631d070 0x55b6f6326300 unknown :-1 s=CLOSED pgs=446 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:26:15.200 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:15.206+0000 7fc8f6732b80 1 --2- 172.21.15.52:0/3001065169 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55b6f6197e40 0x55b6f63292c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:26:15.200 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:15.206+0000 7fc8f6732b80 1 --2- 172.21.15.52:0/3001065169 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55b6f6317120 0x55b6f6325dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:26:15.200 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:15.206+0000 7fc8c97fa700 0 client.15543 ms_handle_reset on v2:172.21.15.105:6838/500348151 2024-05-27T07:26:15.200 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:15.206+0000 7fc8f6732b80 1 -- 172.21.15.52:0/3001065169 >> 172.21.15.52:0/3001065169 conn(0x55b6f619a770 msgr2=0x55b6f631ac20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:26:15.200 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:15.206+0000 7fc8f6732b80 1 -- 172.21.15.52:0/3001065169 shutdown_connections 2024-05-27T07:26:15.200 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:15.206+0000 7fc8f6732b80 1 -- 172.21.15.52:0/3001065169 wait complete. 2024-05-27T07:26:15.200 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:15.206+0000 7fc8f6732b80 20 client.15543 trim_cache size 0 max 16384 2024-05-27T07:26:15.201 INFO:teuthology.orchestra.run.smithi052.stdout:{ 2024-05-27T07:26:15.201 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "IDLE (waiting for more scrubs)", 2024-05-27T07:26:15.201 INFO:teuthology.orchestra.run.smithi052.stdout: "scrubs": { 2024-05-27T07:26:15.201 INFO:teuthology.orchestra.run.smithi052.stdout: "ae7a37c0-a7de-4a0a-aade-1c8cbfbe54a3": { 2024-05-27T07:26:15.201 INFO:teuthology.orchestra.run.smithi052.stdout: "path": "/", 2024-05-27T07:26:15.201 INFO:teuthology.orchestra.run.smithi052.stdout: "tag": "ae7a37c0-a7de-4a0a-aade-1c8cbfbe54a3", 2024-05-27T07:26:15.201 INFO:teuthology.orchestra.run.smithi052.stdout: "options": "recursive,force" 2024-05-27T07:26:15.201 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-05-27T07:26:15.201 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-05-27T07:26:15.202 INFO:teuthology.orchestra.run.smithi052.stdout:} 2024-05-27T07:26:15.202 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:15.207+0000 7fc8e7d1d700 1 -- 172.21.15.52:0/1505576564 >> [v2:172.21.15.52:6800/1822318384,v1:172.21.15.52:6801/1822318384] conn(0x7fc8cc0b2b10 msgr2=0x7fc8cc0b5020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:26:15.202 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:15.207+0000 7fc8e7d1d700 1 --2- 172.21.15.52:0/1505576564 >> [v2:172.21.15.52:6800/1822318384,v1:172.21.15.52:6801/1822318384] conn(0x7fc8cc0b2b10 0x7fc8cc0b5020 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7fc8dc005ea0 tx=0x7fc8dc005e30 comp rx=0 tx=0).stop 2024-05-27T07:26:15.202 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:15.207+0000 7fc8e7d1d700 1 -- 172.21.15.52:0/1505576564 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fc8e0109860 msgr2=0x7fc8e019f610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:26:15.202 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:15.207+0000 7fc8e7d1d700 1 --2- 172.21.15.52:0/1505576564 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fc8e0109860 0x7fc8e019f610 secure :-1 s=READY pgs=442 cs=0 l=1 rev1=1 crypto rx=0x7fc8d8009850 tx=0x7fc8d8007a20 comp rx=0 tx=0).stop 2024-05-27T07:26:15.203 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:15.207+0000 7fc8e7d1d700 1 -- 172.21.15.52:0/1505576564 shutdown_connections 2024-05-27T07:26:15.203 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:15.207+0000 7fc8e7d1d700 1 --2- 172.21.15.52:0/1505576564 >> [v2:172.21.15.52:6800/1822318384,v1:172.21.15.52:6801/1822318384] conn(0x7fc8cc0b2b10 0x7fc8cc0b5020 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:26:15.203 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:15.207+0000 7fc8e7d1d700 1 --2- 172.21.15.52:0/1505576564 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fc8e0109860 0x7fc8e019f610 unknown :-1 s=CLOSED pgs=442 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:26:15.203 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:15.207+0000 7fc8e7d1d700 1 --2- 172.21.15.52:0/1505576564 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fc8e01045e0 0x7fc8e019ade0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:26:15.203 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:15.207+0000 7fc8e7d1d700 1 --2- 172.21.15.52:0/1505576564 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fc8e0108f10 0x7fc8e019b380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:26:15.203 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:15.207+0000 7fc8e7d1d700 1 -- 172.21.15.52:0/1505576564 >> 172.21.15.52:0/1505576564 conn(0x7fc8e00ffeb0 msgr2=0x7fc8e01086b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:26:15.203 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:15.207+0000 7fc8e7d1d700 1 -- 172.21.15.52:0/1505576564 shutdown_connections 2024-05-27T07:26:15.203 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:15.207+0000 7fc8e7d1d700 1 -- 172.21.15.52:0/1505576564 wait complete. 2024-05-27T07:26:16.056 INFO:tasks.cephfs.filesystem:scrub status for tag:ae7a37c0-a7de-4a0a-aade-1c8cbfbe54a3 - {'path': '/', 'tag': 'ae7a37c0-a7de-4a0a-aade-1c8cbfbe54a3', 'options': 'recursive,force'} 2024-05-27T07:26:16.352 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:26:16 smithi173 ceph-mon[37000]: pgmap v12: 129 pgs: 129 active+clean; 1.3 MiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 41 KiB/s rd, 77 KiB/s wr, 51 op/s 2024-05-27T07:26:16.439 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:26:16 smithi105 ceph-mon[35977]: pgmap v12: 129 pgs: 129 active+clean; 1.3 MiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 41 KiB/s rd, 77 KiB/s wr, 51 op/s 2024-05-27T07:26:16.479 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:26:16 smithi052 ceph-mon[31955]: pgmap v12: 129 pgs: 129 active+clean; 1.3 MiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 41 KiB/s rd, 77 KiB/s wr, 51 op/s 2024-05-27T07:26:18.352 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:26:18 smithi173 ceph-mon[37000]: pgmap v13: 129 pgs: 129 active+clean; 1.3 MiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 23 KiB/s rd, 76 KiB/s wr, 40 op/s 2024-05-27T07:26:18.440 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:26:18 smithi105 ceph-mon[35977]: pgmap v13: 129 pgs: 129 active+clean; 1.3 MiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 23 KiB/s rd, 76 KiB/s wr, 40 op/s 2024-05-27T07:26:18.479 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:26:18 smithi052 ceph-mon[31955]: pgmap v13: 129 pgs: 129 active+clean; 1.3 MiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 23 KiB/s rd, 76 KiB/s wr, 40 op/s 2024-05-27T07:26:20.351 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:26:20 smithi173 ceph-mon[37000]: pgmap v14: 129 pgs: 129 active+clean; 1.3 MiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 26 KiB/s rd, 77 KiB/s wr, 44 op/s 2024-05-27T07:26:20.439 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:26:20 smithi105 ceph-mon[35977]: pgmap v14: 129 pgs: 129 active+clean; 1.3 MiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 26 KiB/s rd, 77 KiB/s wr, 44 op/s 2024-05-27T07:26:20.479 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:26:20 smithi052 ceph-mon[31955]: pgmap v14: 129 pgs: 129 active+clean; 1.3 MiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 26 KiB/s rd, 77 KiB/s wr, 44 op/s 2024-05-27T07:26:22.351 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:26:22 smithi173 ceph-mon[37000]: pgmap v15: 129 pgs: 129 active+clean; 1.3 MiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 24 KiB/s rd, 77 KiB/s wr, 40 op/s 2024-05-27T07:26:22.439 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:26:22 smithi105 ceph-mon[35977]: pgmap v15: 129 pgs: 129 active+clean; 1.3 MiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 24 KiB/s rd, 77 KiB/s wr, 40 op/s 2024-05-27T07:26:22.479 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:26:22 smithi052 ceph-mon[31955]: pgmap v15: 129 pgs: 129 active+clean; 1.3 MiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 24 KiB/s rd, 77 KiB/s wr, 40 op/s 2024-05-27T07:26:24.352 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:26:24 smithi173 ceph-mon[37000]: pgmap v16: 129 pgs: 129 active+clean; 1.3 MiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 24 KiB/s rd, 77 KiB/s wr, 41 op/s 2024-05-27T07:26:24.439 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:26:24 smithi105 ceph-mon[35977]: pgmap v16: 129 pgs: 129 active+clean; 1.3 MiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 24 KiB/s rd, 77 KiB/s wr, 41 op/s 2024-05-27T07:26:24.479 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:26:24 smithi052 ceph-mon[31955]: pgmap v16: 129 pgs: 129 active+clean; 1.3 MiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 24 KiB/s rd, 77 KiB/s wr, 41 op/s 2024-05-27T07:26:26.133 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:26:26.143+0000 7fe06664c640 1 -- 172.21.15.52:0/531900982 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fe060109020 msgr2=0x7fe060102c60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:26:26.134 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:26:26.143+0000 7fe06664c640 1 --2- 172.21.15.52:0/531900982 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fe060109020 0x7fe060102c60 secure :-1 s=READY pgs=447 cs=0 l=1 rev1=1 crypto rx=0x7fe054009900 tx=0x7fe05402a410 comp rx=0 tx=0).stop 2024-05-27T07:26:26.134 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:26:26.143+0000 7fe06664c640 1 -- 172.21.15.52:0/531900982 shutdown_connections 2024-05-27T07:26:26.134 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:26:26.143+0000 7fe06664c640 1 --2- 172.21.15.52:0/531900982 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fe0601032d0 0x7fe06010f6a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:26:26.134 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:26:26.143+0000 7fe06664c640 1 --2- 172.21.15.52:0/531900982 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fe060109020 0x7fe060102c60 unknown :-1 s=CLOSED pgs=447 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:26:26.134 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:26:26.143+0000 7fe06664c640 1 --2- 172.21.15.52:0/531900982 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fe060108b80 0x7fe060102720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:26:26.134 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:26:26.143+0000 7fe06664c640 1 -- 172.21.15.52:0/531900982 >> 172.21.15.52:0/531900982 conn(0x7fe0600fe380 msgr2=0x7fe0601007a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:26:26.134 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:26:26.143+0000 7fe06664c640 1 -- 172.21.15.52:0/531900982 shutdown_connections 2024-05-27T07:26:26.134 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:26:26.143+0000 7fe06664c640 1 -- 172.21.15.52:0/531900982 wait complete. 2024-05-27T07:26:26.134 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:26:26.144+0000 7fe06664c640 1 Processor -- start 2024-05-27T07:26:26.134 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:26:26.144+0000 7fe06664c640 1 -- start start 2024-05-27T07:26:26.136 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:26:26.144+0000 7fe06664c640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fe0601032d0 0x7fe060198750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:26:26.136 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:26:26.144+0000 7fe06664c640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fe060108b80 0x7fe060198cb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:26:26.136 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:26:26.144+0000 7fe06664c640 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fe060109020 0x7fe06019cf70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:26:26.136 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:26:26.144+0000 7fe06664c640 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x7fe060111c70 con 0x7fe060109020 2024-05-27T07:26:26.137 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:26:26.144+0000 7fe06664c640 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7fe060111af0 con 0x7fe060108b80 2024-05-27T07:26:26.137 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:26:26.144+0000 7fe06664c640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7fe060111df0 con 0x7fe0601032d0 2024-05-27T07:26:26.137 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:26:26.145+0000 7fe064be9640 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fe060109020 0x7fe06019cf70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:26:26.137 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:26:26.145+0000 7fe064be9640 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fe060109020 0x7fe06019cf70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.52:56480/0 (socket says 172.21.15.52:56480) 2024-05-27T07:26:26.137 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:26:26.145+0000 7fe064be9640 1 -- 172.21.15.52:0/743743678 learned_addr learned my addr 172.21.15.52:0/743743678 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-05-27T07:26:26.137 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:26:26.145+0000 7fe05ffff640 1 --2- 172.21.15.52:0/743743678 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fe0601032d0 0x7fe060198750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:26:26.137 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:26:26.145+0000 7fe05f7fe640 1 --2- 172.21.15.52:0/743743678 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fe060108b80 0x7fe060198cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:26:26.137 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:26:26.145+0000 7fe064be9640 1 -- 172.21.15.52:0/743743678 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fe0601032d0 msgr2=0x7fe060198750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:26:26.137 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:26:26.145+0000 7fe064be9640 1 --2- 172.21.15.52:0/743743678 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fe0601032d0 0x7fe060198750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:26:26.137 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:26:26.145+0000 7fe064be9640 1 -- 172.21.15.52:0/743743678 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fe060108b80 msgr2=0x7fe060198cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:26:26.137 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:26:26.145+0000 7fe064be9640 1 --2- 172.21.15.52:0/743743678 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fe060108b80 0x7fe060198cb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:26:26.137 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:26:26.145+0000 7fe064be9640 1 -- 172.21.15.52:0/743743678 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe06019d5b0 con 0x7fe060109020 2024-05-27T07:26:26.137 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:26:26.145+0000 7fe064be9640 1 --2- 172.21.15.52:0/743743678 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fe060109020 0x7fe06019cf70 secure :-1 s=READY pgs=448 cs=0 l=1 rev1=1 crypto rx=0x7fe05800bfb0 tx=0x7fe0580114a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:26:26.137 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:26:26.145+0000 7fe05d7fa640 1 -- 172.21.15.52:0/743743678 <== mon.0 v2:172.21.15.52:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe058003970 con 0x7fe060109020 2024-05-27T07:26:26.138 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:26:26.145+0000 7fe05d7fa640 1 -- 172.21.15.52:0/743743678 <== mon.0 v2:172.21.15.52:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fe058004020 con 0x7fe060109020 2024-05-27T07:26:26.138 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:26:26.145+0000 7fe06664c640 1 -- 172.21.15.52:0/743743678 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe06019d8a0 con 0x7fe060109020 2024-05-27T07:26:26.138 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:26:26.145+0000 7fe05d7fa640 1 -- 172.21.15.52:0/743743678 <== mon.0 v2:172.21.15.52:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe0580196f0 con 0x7fe060109020 2024-05-27T07:26:26.138 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:26:26.145+0000 7fe06664c640 1 -- 172.21.15.52:0/743743678 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe06019dd70 con 0x7fe060109020 2024-05-27T07:26:26.140 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:26:26.148+0000 7fe06d5dd740 1 Processor -- start 2024-05-27T07:26:26.140 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:26:26.148+0000 7fe05d7fa640 1 -- 172.21.15.52:0/743743678 <== mon.0 v2:172.21.15.52:3300/0 4 ==== mgrmap(e 24) ==== 149879+0+0 (secure 0 0 0) 0x7fe0580044d0 con 0x7fe060109020 2024-05-27T07:26:26.140 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:26:26.149+0000 7fe06d5dd740 1 -- start start 2024-05-27T07:26:26.140 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:26:26.149+0000 7fe06d5dd740 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x56218059e1b0 0x56218093f3d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:26:26.140 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:26:26.149+0000 7fe06d5dd740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5621805752f0 0x56218093f910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:26:26.140 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:26:26.149+0000 7fe06d5dd740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x56218057d4d0 0x56218093fe50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:26:26.140 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:26:26.149+0000 7fe06d5dd740 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x5621808b1c30 con 0x56218059e1b0 2024-05-27T07:26:26.140 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:26:26.149+0000 7fe06d5dd740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x5621808a17d0 con 0x56218057d4d0 2024-05-27T07:26:26.141 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:26:26.149+0000 7fe06d5dd740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x5621808a5810 con 0x5621805752f0 2024-05-27T07:26:26.141 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:26:26.149+0000 7fe05f7fe640 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x56218059e1b0 0x56218093f3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:26:26.141 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:26:26.149+0000 7fe05f7fe640 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x56218059e1b0 0x56218093f3d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.52:56494/0 (socket says 172.21.15.52:56494) 2024-05-27T07:26:26.141 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:26:26.149+0000 7fe05f7fe640 1 -- 172.21.15.52:0/1573028728 learned_addr learned my addr 172.21.15.52:0/1573028728 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-05-27T07:26:26.141 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:26:26.149+0000 7fe05ffff640 1 --2- 172.21.15.52:0/1573028728 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5621805752f0 0x56218093f910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:26:26.141 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:26:26.149+0000 7fe05f7fe640 1 --2- 172.21.15.52:0/1573028728 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x56218057d4d0 0x56218093fe50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:26:26.141 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:26:26.150+0000 7fe05f7fe640 1 -- 172.21.15.52:0/1573028728 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5621805752f0 msgr2=0x56218093f910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:26:26.141 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:26:26.150+0000 7fe05f7fe640 1 --2- 172.21.15.52:0/1573028728 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5621805752f0 0x56218093f910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:26:26.141 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:26:26.150+0000 7fe05f7fe640 1 -- 172.21.15.52:0/1573028728 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x56218057d4d0 msgr2=0x56218093fe50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:26:26.141 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:26:26.150+0000 7fe05f7fe640 1 --2- 172.21.15.52:0/1573028728 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x56218057d4d0 0x56218093fe50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:26:26.141 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:26:26.150+0000 7fe05d7fa640 1 --2- 172.21.15.52:0/743743678 >> [v2:172.21.15.52:6800/1822318384,v1:172.21.15.52:6801/1822318384] conn(0x7fe0480b2a70 0x7fe0480b4f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:26:26.141 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:26:26.150+0000 7fe05f7fe640 1 -- 172.21.15.52:0/1573028728 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5621805b6210 con 0x56218059e1b0 2024-05-27T07:26:26.141 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:26:26.150+0000 7fe064be9640 1 --2- 172.21.15.52:0/743743678 >> [v2:172.21.15.52:6800/1822318384,v1:172.21.15.52:6801/1822318384] conn(0x7fe0480b2a70 0x7fe0480b4f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:26:26.142 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:26:26.150+0000 7fe05f7fe640 1 --2- 172.21.15.52:0/1573028728 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x56218059e1b0 0x56218093f3d0 secure :-1 s=READY pgs=449 cs=0 l=1 rev1=1 crypto rx=0x7fe054005940 tx=0x7fe0540058b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3c409374761ffe47 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:26:26.142 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:26:26.150+0000 7fe05d7fa640 1 -- 172.21.15.52:0/743743678 <== mon.0 v2:172.21.15.52:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9379+0+0 (secure 0 0 0) 0x7fe0580e69b0 con 0x7fe060109020 2024-05-27T07:26:26.142 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:26:26.150+0000 7fe0467fc640 1 -- 172.21.15.52:0/1573028728 <== mon.0 v2:172.21.15.52:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe054035070 con 0x56218059e1b0 2024-05-27T07:26:26.142 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:26:26.150+0000 7fe0467fc640 1 -- 172.21.15.52:0/1573028728 <== mon.0 v2:172.21.15.52:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fe054044070 con 0x56218059e1b0 2024-05-27T07:26:26.142 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:26:26.150+0000 7fe0467fc640 1 -- 172.21.15.52:0/1573028728 <== mon.0 v2:172.21.15.52:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe054005240 con 0x56218059e1b0 2024-05-27T07:26:26.146 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:26:26.151+0000 7fe06d5dd740 1 -- 172.21.15.52:0/1573028728 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x56218059e1b0 msgr2=0x56218093f3d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:26:26.147 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:26:26.151+0000 7fe06d5dd740 1 --2- 172.21.15.52:0/1573028728 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x56218059e1b0 0x56218093f3d0 secure :-1 s=READY pgs=449 cs=0 l=1 rev1=1 crypto rx=0x7fe054005940 tx=0x7fe0540058b0 comp rx=0 tx=0).stop 2024-05-27T07:26:26.147 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:26:26.151+0000 7fe064be9640 1 --2- 172.21.15.52:0/743743678 >> [v2:172.21.15.52:6800/1822318384,v1:172.21.15.52:6801/1822318384] conn(0x7fe0480b2a70 0x7fe0480b4f80 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7fe0580e3c80 tx=0x7fe05807b2b0 comp rx=0 tx=0).ready entity=mgr.15333 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:26:26.147 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:26:26.151+0000 7fe06d5dd740 1 -- 172.21.15.52:0/1573028728 shutdown_connections 2024-05-27T07:26:26.147 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:26:26.151+0000 7fe06d5dd740 1 --2- 172.21.15.52:0/1573028728 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x56218057d4d0 0x56218093fe50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:26:26.147 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:26:26.151+0000 7fe06d5dd740 1 --2- 172.21.15.52:0/1573028728 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5621805752f0 0x56218093f910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:26:26.147 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:26:26.151+0000 7fe06d5dd740 1 --2- 172.21.15.52:0/1573028728 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x56218059e1b0 0x56218093f3d0 unknown :-1 s=CLOSED pgs=449 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:26:26.147 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:26:26.151+0000 7fe06d5dd740 1 -- 172.21.15.52:0/1573028728 >> 172.21.15.52:0/1573028728 conn(0x562180592300 msgr2=0x56218093bb80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:26:26.147 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:26:26.151+0000 7fe06d5dd740 1 -- 172.21.15.52:0/1573028728 shutdown_connections 2024-05-27T07:26:26.147 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:26:26.151+0000 7fe06d5dd740 1 -- 172.21.15.52:0/1573028728 wait complete. 2024-05-27T07:26:26.147 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:26:26.151+0000 7fe06d5dd740 1 -- start start 2024-05-27T07:26:26.147 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:26:26.152+0000 7fe06d5dd740 1 Processor -- start 2024-05-27T07:26:26.147 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:26:26.152+0000 7fe06d5dd740 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x5621805752f0 0x56218094b410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:26:26.148 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:26:26.152+0000 7fe06d5dd740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x56218057d4d0 0x56218094b970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:26:26.148 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:26:26.152+0000 7fe06d5dd740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x56218059e1b0 0x562180949920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:26:26.148 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:26:26.152+0000 7fe06d5dd740 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x562180587790 con 0x5621805752f0 2024-05-27T07:26:26.148 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:26:26.152+0000 7fe06d5dd740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x56218085d890 con 0x56218059e1b0 2024-05-27T07:26:26.148 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:26:26.152+0000 7fe06d5dd740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x56218094c4a0 con 0x56218057d4d0 2024-05-27T07:26:26.148 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:26:26.152+0000 7fe06d5dd740 10 client.0 _ll_register_callbacks cb 0x5621805b5a00 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T07:26:26.148 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:26:26.152+0000 7fe05f7fe640 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x5621805752f0 0x56218094b410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:26:26.148 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:26:26.152+0000 7fe05f7fe640 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x5621805752f0 0x56218094b410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.52:56502/0 (socket says 172.21.15.52:56502) 2024-05-27T07:26:26.148 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:26:26.152+0000 7fe05f7fe640 1 -- 172.21.15.52:0/663253229 learned_addr learned my addr 172.21.15.52:0/663253229 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-05-27T07:26:26.148 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:26:26.152+0000 7fe05ffff640 1 --2- 172.21.15.52:0/663253229 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x56218057d4d0 0x56218094b970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:26:26.148 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:26:26.153+0000 7fe05f7fe640 1 --2- 172.21.15.52:0/663253229 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x56218059e1b0 0x562180949920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:26:26.148 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:26:26.153+0000 7fe05f7fe640 1 -- 172.21.15.52:0/663253229 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x56218057d4d0 msgr2=0x56218094b970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:26:26.148 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:26:26.153+0000 7fe05f7fe640 1 --2- 172.21.15.52:0/663253229 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x56218057d4d0 0x56218094b970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:26:26.149 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:26:26.153+0000 7fe05f7fe640 1 -- 172.21.15.52:0/663253229 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x56218059e1b0 msgr2=0x562180949920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:26:26.149 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:26:26.153+0000 7fe05f7fe640 1 --2- 172.21.15.52:0/663253229 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x56218059e1b0 0x562180949920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:26:26.149 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:26:26.153+0000 7fe05f7fe640 1 -- 172.21.15.52:0/663253229 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({osdmap=0}) -- 0x562180682620 con 0x5621805752f0 2024-05-27T07:26:26.149 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:26:26.153+0000 7fe05f7fe640 1 --2- 172.21.15.52:0/663253229 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x5621805752f0 0x56218094b410 secure :-1 s=READY pgs=450 cs=0 l=1 rev1=1 crypto rx=0x7fe054005a80 tx=0x7fe05402a410 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5744236197ce7ecb server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:26:26.149 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:26:26.153+0000 7fe03f7fe640 10 client.0 ms_handle_connect on v2:172.21.15.52:3300/0 2024-05-27T07:26:26.149 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:26:26.153+0000 7fe03f7fe640 1 -- 172.21.15.52:0/663253229 <== mon.0 v2:172.21.15.52:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe054052060 con 0x5621805752f0 2024-05-27T07:26:26.149 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:26:26.153+0000 7fe03f7fe640 1 -- 172.21.15.52:0/663253229 <== mon.0 v2:172.21.15.52:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 9379+0+0 (secure 0 0 0) 0x7fe054005270 con 0x5621805752f0 2024-05-27T07:26:26.149 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:26:26.153+0000 7fe06d5dd740 1 -- 172.21.15.52:0/663253229 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fe054044070 con 0x5621805752f0 2024-05-27T07:26:26.149 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:26:26.153+0000 7fe03f7fe640 1 -- 172.21.15.52:0/663253229 <== mon.0 v2:172.21.15.52:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7fe054035040 con 0x5621805752f0 2024-05-27T07:26:26.149 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:26:26.153+0000 7fe06d5dd740 10 client.15555 fetch_fsmap learned FSMap version 30 2024-05-27T07:26:26.149 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:26:26.153+0000 7fe06d5dd740 1 -- 172.21.15.52:0/663253229 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x562180958850 con 0x5621805752f0 2024-05-27T07:26:26.149 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:26:26.153+0000 7fe03f7fe640 1 -- 172.21.15.52:0/663253229 <== mon.0 v2:172.21.15.52:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fe054044070 con 0x5621805752f0 2024-05-27T07:26:26.149 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:26:26.154+0000 7fe03f7fe640 1 -- 172.21.15.52:0/663253229 <== mon.0 v2:172.21.15.52:3300/0 5 ==== fsmap(e 30) ==== 8362+0+0 (secure 0 0 0) 0x7fe0540035f0 con 0x5621805752f0 2024-05-27T07:26:26.150 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:26:26.154+0000 7fe06d5dd740 10 client.15555 fetch_fsmap finished waiting for FSMap version 30 2024-05-27T07:26:26.150 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:26:26.154+0000 7fe06d5dd740 10 client.15555 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-05-27T07:26:26.150 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:26:26.154+0000 7fe06d5dd740 20 client.15555 populate_metadata read hostname 'smithi052' 2024-05-27T07:26:26.150 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:26:26.154+0000 7fe06d5dd740 1 --2- 172.21.15.52:0/663253229 >> [v2:172.21.15.105:6838/500348151,v1:172.21.15.105:6839/500348151] conn(0x56218055cd80 0x56218095bbb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:26:26.150 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:26:26.154+0000 7fe06d5dd740 4 client.15555 mds_command: new command op to 24527 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-27T07:26:26.150 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:26:26.154+0000 7fe06d5dd740 1 -- 172.21.15.52:0/663253229 --> [v2:172.21.15.105:6838/500348151,v1:172.21.15.105:6839/500348151] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5621808b3b80 con 0x56218055cd80 2024-05-27T07:26:26.150 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:26:26.154+0000 7fe05ffff640 1 --2- 172.21.15.52:0/663253229 >> [v2:172.21.15.105:6838/500348151,v1:172.21.15.105:6839/500348151] conn(0x56218055cd80 0x56218095bbb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:26:26.150 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:26:26.155+0000 7fe03f7fe640 10 client.15555 ms_handle_connect on v2:172.21.15.105:6838/500348151 2024-05-27T07:26:26.150 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:26:26.155+0000 7fe05ffff640 1 --2- 172.21.15.52:0/663253229 >> [v2:172.21.15.105:6838/500348151,v1:172.21.15.105:6839/500348151] conn(0x56218055cd80 0x56218095bbb0 crc :-1 s=READY pgs=46 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=30fa509d3f9c9b91 server_cookie=e628503aa8b8dea1 in_seq=0 out_seq=0 2024-05-27T07:26:26.150 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:26:26.156+0000 7fe03f7fe640 1 -- 172.21.15.52:0/663253229 <== mds.0 v2:172.21.15.105:6838/500348151 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24041 (crc 0 0 0) 0x5621808b3b80 con 0x56218055cd80 2024-05-27T07:26:26.150 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:26:26.156+0000 7fe03f7fe640 10 client.15555 handle_command_reply: tid=0 2024-05-27T07:26:26.150 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:26:26.156+0000 7fe06d5dd740 1 client.15555 shutdown 2024-05-27T07:26:26.151 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:26:26.156+0000 7fe06d5dd740 1 -- 172.21.15.52:0/663253229 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x5621805752f0 msgr2=0x56218094b410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:26:26.151 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:26:26.156+0000 7fe06d5dd740 1 --2- 172.21.15.52:0/663253229 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x5621805752f0 0x56218094b410 secure :-1 s=READY pgs=450 cs=0 l=1 rev1=1 crypto rx=0x7fe054005a80 tx=0x7fe05402a410 comp rx=0 tx=0).stop 2024-05-27T07:26:26.151 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:26:26.156+0000 7fe06d5dd740 1 -- 172.21.15.52:0/663253229 shutdown_connections 2024-05-27T07:26:26.151 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:26:26.156+0000 7fe06d5dd740 1 --2- 172.21.15.52:0/663253229 >> [v2:172.21.15.105:6838/500348151,v1:172.21.15.105:6839/500348151] conn(0x56218055cd80 0x56218095bbb0 crc :-1 s=READY pgs=46 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:26:26.151 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:26:26.156+0000 7fe06d5dd740 1 --2- 172.21.15.52:0/663253229 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x56218059e1b0 0x562180949920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:26:26.151 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:26:26.156+0000 7fe06d5dd740 1 --2- 172.21.15.52:0/663253229 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x56218057d4d0 0x56218094b970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:26:26.151 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:26:26.156+0000 7fe03f7fe640 0 client.15555 ms_handle_reset on v2:172.21.15.105:6838/500348151 2024-05-27T07:26:26.151 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:26:26.156+0000 7fe06d5dd740 1 --2- 172.21.15.52:0/663253229 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x5621805752f0 0x56218094b410 unknown :-1 s=CLOSED pgs=450 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:26:26.151 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:26:26.156+0000 7fe06d5dd740 1 -- 172.21.15.52:0/663253229 >> 172.21.15.52:0/663253229 conn(0x562180592300 msgr2=0x56218093f1a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:26:26.151 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:26:26.156+0000 7fe06d5dd740 1 -- 172.21.15.52:0/663253229 shutdown_connections 2024-05-27T07:26:26.151 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:26:26.156+0000 7fe06d5dd740 1 -- 172.21.15.52:0/663253229 wait complete. 2024-05-27T07:26:26.151 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:26:26.156+0000 7fe06d5dd740 20 client.15555 trim_cache size 0 max 16384 2024-05-27T07:26:26.164 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:26:26.172+0000 7fe06d5dd740 1 Processor -- start 2024-05-27T07:26:26.164 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:26:26.172+0000 7fe06d5dd740 1 -- start start 2024-05-27T07:26:26.164 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:26:26.172+0000 7fe06d5dd740 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x5621805752f0 0x56218093f870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:26:26.164 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:26:26.172+0000 7fe06d5dd740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x56218057d4d0 0x56218095ebe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:26:26.164 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:26:26.172+0000 7fe06d5dd740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x56218059e1b0 0x56218095f120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:26:26.164 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:26:26.172+0000 7fe06d5dd740 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x7fe0601aef30 con 0x5621805752f0 2024-05-27T07:26:26.164 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:26:26.172+0000 7fe06d5dd740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7fe0601b0a70 con 0x56218057d4d0 2024-05-27T07:26:26.164 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:26:26.172+0000 7fe06d5dd740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7fe0601af600 con 0x56218059e1b0 2024-05-27T07:26:26.164 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:26:26.172+0000 7fe05f7fe640 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x5621805752f0 0x56218093f870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:26:26.164 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:26:26.172+0000 7fe05f7fe640 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x5621805752f0 0x56218093f870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.52:56514/0 (socket says 172.21.15.52:56514) 2024-05-27T07:26:26.165 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:26:26.172+0000 7fe05f7fe640 1 -- 172.21.15.52:0/2233030459 learned_addr learned my addr 172.21.15.52:0/2233030459 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-05-27T07:26:26.165 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:26:26.173+0000 7fe05ffff640 1 --2- 172.21.15.52:0/2233030459 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x56218057d4d0 0x56218095ebe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:26:26.165 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:26:26.173+0000 7fe05f7fe640 1 -- 172.21.15.52:0/2233030459 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x56218059e1b0 msgr2=0x56218095f120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:26:26.165 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:26:26.173+0000 7fe05f7fe640 1 --2- 172.21.15.52:0/2233030459 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x56218059e1b0 0x56218095f120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:26:26.165 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:26:26.173+0000 7fe05f7fe640 1 -- 172.21.15.52:0/2233030459 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x56218057d4d0 msgr2=0x56218095ebe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:26:26.165 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:26:26.173+0000 7fe05f7fe640 1 --2- 172.21.15.52:0/2233030459 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x56218057d4d0 0x56218095ebe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:26:26.165 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:26:26.173+0000 7fe05f7fe640 1 -- 172.21.15.52:0/2233030459 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x562180940120 con 0x5621805752f0 2024-05-27T07:26:26.165 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:26:26.173+0000 7fe05ffff640 1 --2- 172.21.15.52:0/2233030459 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x56218057d4d0 0x56218095ebe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-27T07:26:26.165 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:26:26.173+0000 7fe05f7fe640 1 --2- 172.21.15.52:0/2233030459 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x5621805752f0 0x56218093f870 secure :-1 s=READY pgs=451 cs=0 l=1 rev1=1 crypto rx=0x7fe05402a930 tx=0x7fe054000bc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fccef4cc7b1b3e11 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:26:26.165 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:26:26.173+0000 7fe045ffb640 1 -- 172.21.15.52:0/2233030459 <== mon.0 v2:172.21.15.52:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe054052040 con 0x5621805752f0 2024-05-27T07:26:26.165 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:26:26.173+0000 7fe045ffb640 1 -- 172.21.15.52:0/2233030459 <== mon.0 v2:172.21.15.52:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fe05404e070 con 0x5621805752f0 2024-05-27T07:26:26.165 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:26:26.173+0000 7fe045ffb640 1 -- 172.21.15.52:0/2233030459 <== mon.0 v2:172.21.15.52:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe0540046c0 con 0x5621805752f0 2024-05-27T07:26:26.165 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:26:26.174+0000 7fe06d5dd740 1 -- 172.21.15.52:0/2233030459 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x5621805752f0 msgr2=0x56218093f870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:26:26.166 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:26:26.174+0000 7fe06d5dd740 1 --2- 172.21.15.52:0/2233030459 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x5621805752f0 0x56218093f870 secure :-1 s=READY pgs=451 cs=0 l=1 rev1=1 crypto rx=0x7fe05402a930 tx=0x7fe054000bc0 comp rx=0 tx=0).stop 2024-05-27T07:26:26.166 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:26:26.174+0000 7fe06d5dd740 1 -- 172.21.15.52:0/2233030459 shutdown_connections 2024-05-27T07:26:26.166 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:26:26.174+0000 7fe06d5dd740 1 --2- 172.21.15.52:0/2233030459 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x56218059e1b0 0x56218095f120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:26:26.166 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:26:26.174+0000 7fe06d5dd740 1 --2- 172.21.15.52:0/2233030459 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x56218057d4d0 0x56218095ebe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:26:26.166 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:26:26.174+0000 7fe06d5dd740 1 --2- 172.21.15.52:0/2233030459 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x5621805752f0 0x56218093f870 unknown :-1 s=CLOSED pgs=451 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:26:26.166 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:26:26.174+0000 7fe06d5dd740 1 -- 172.21.15.52:0/2233030459 >> 172.21.15.52:0/2233030459 conn(0x562180592300 msgr2=0x56218093f160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:26:26.166 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:26:26.174+0000 7fe06d5dd740 1 -- 172.21.15.52:0/2233030459 shutdown_connections 2024-05-27T07:26:26.166 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:26:26.174+0000 7fe06d5dd740 1 -- 172.21.15.52:0/2233030459 wait complete. 2024-05-27T07:26:26.166 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:26:26.174+0000 7fe06d5dd740 1 -- start start 2024-05-27T07:26:26.169 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:26:26.175+0000 7fe06d5dd740 1 Processor -- start 2024-05-27T07:26:26.169 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:26:26.175+0000 7fe06d5dd740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5621805752f0 0x56218095d950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:26:26.169 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:26:26.175+0000 7fe06d5dd740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x56218057d4d0 0x56218095de90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:26:26.169 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:26:26.175+0000 7fe06d5dd740 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x56218059e1b0 0x56218095e3d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:26:26.169 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:26:26.175+0000 7fe06d5dd740 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x7fe0601b03a0 con 0x56218059e1b0 2024-05-27T07:26:26.169 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:26:26.175+0000 7fe06d5dd740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7fe0601afcd0 con 0x56218057d4d0 2024-05-27T07:26:26.169 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:26:26.175+0000 7fe06d5dd740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7fe0601acf10 con 0x5621805752f0 2024-05-27T07:26:26.170 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:26:26.175+0000 7fe06d5dd740 10 client.0 _ll_register_callbacks cb 0x7fe05000b550 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T07:26:26.170 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:26:26.175+0000 7fe05f7fe640 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x56218059e1b0 0x56218095e3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:26:26.170 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:26:26.175+0000 7fe05f7fe640 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x56218059e1b0 0x56218095e3d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.52:56524/0 (socket says 172.21.15.52:56524) 2024-05-27T07:26:26.170 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:26:26.175+0000 7fe05f7fe640 1 -- 172.21.15.52:0/3947544389 learned_addr learned my addr 172.21.15.52:0/3947544389 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-05-27T07:26:26.170 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:26:26.175+0000 7fe05ffff640 1 --2- 172.21.15.52:0/3947544389 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x56218057d4d0 0x56218095de90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:26:26.170 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:26:26.175+0000 7fe05f7fe640 1 --2- 172.21.15.52:0/3947544389 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5621805752f0 0x56218095d950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:26:26.170 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:26:26.175+0000 7fe05f7fe640 1 -- 172.21.15.52:0/3947544389 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5621805752f0 msgr2=0x56218095d950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:26:26.170 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:26:26.175+0000 7fe05f7fe640 1 --2- 172.21.15.52:0/3947544389 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5621805752f0 0x56218095d950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:26:26.170 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:26:26.175+0000 7fe05f7fe640 1 -- 172.21.15.52:0/3947544389 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x56218057d4d0 msgr2=0x56218095de90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:26:26.170 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:26:26.175+0000 7fe05f7fe640 1 --2- 172.21.15.52:0/3947544389 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x56218057d4d0 0x56218095de90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:26:26.170 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:26:26.176+0000 7fe05f7fe640 1 -- 172.21.15.52:0/3947544389 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5621809c2000 con 0x56218059e1b0 2024-05-27T07:26:26.170 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:26:26.176+0000 7fe05f7fe640 1 --2- 172.21.15.52:0/3947544389 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x56218059e1b0 0x56218095e3d0 secure :-1 s=READY pgs=452 cs=0 l=1 rev1=1 crypto rx=0x7fe054003f30 tx=0x7fe05402a410 comp rx=0 tx=0).ready entity=mon.0 client_cookie=936bb4d3baad240b server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:26:26.171 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:26:26.176+0000 7fe03f7fe640 10 client.0 ms_handle_connect on v2:172.21.15.52:3300/0 2024-05-27T07:26:26.171 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:26:26.176+0000 7fe03f7fe640 1 -- 172.21.15.52:0/3947544389 <== mon.0 v2:172.21.15.52:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe054062040 con 0x56218059e1b0 2024-05-27T07:26:26.171 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:26:26.176+0000 7fe03f7fe640 1 -- 172.21.15.52:0/3947544389 <== mon.0 v2:172.21.15.52:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 9379+0+0 (secure 0 0 0) 0x7fe05403d450 con 0x56218059e1b0 2024-05-27T07:26:26.171 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:26:26.176+0000 7fe06d5dd740 1 -- 172.21.15.52:0/3947544389 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fe05404e070 con 0x56218059e1b0 2024-05-27T07:26:26.171 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:26:26.176+0000 7fe03f7fe640 1 -- 172.21.15.52:0/3947544389 <== mon.0 v2:172.21.15.52:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7fe0540037f0 con 0x56218059e1b0 2024-05-27T07:26:26.171 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:26:26.176+0000 7fe06d5dd740 10 client.15561 fetch_fsmap learned FSMap version 30 2024-05-27T07:26:26.171 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:26:26.176+0000 7fe06d5dd740 1 -- 172.21.15.52:0/3947544389 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x5621809c3120 con 0x56218059e1b0 2024-05-27T07:26:26.171 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:26:26.176+0000 7fe03f7fe640 1 -- 172.21.15.52:0/3947544389 <== mon.0 v2:172.21.15.52:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fe05404e070 con 0x56218059e1b0 2024-05-27T07:26:26.171 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:26:26.176+0000 7fe03f7fe640 1 -- 172.21.15.52:0/3947544389 <== mon.0 v2:172.21.15.52:3300/0 5 ==== fsmap(e 30) ==== 8362+0+0 (secure 0 0 0) 0x7fe05403a6b0 con 0x56218059e1b0 2024-05-27T07:26:26.171 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:26:26.176+0000 7fe06d5dd740 10 client.15561 fetch_fsmap finished waiting for FSMap version 30 2024-05-27T07:26:26.171 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:26:26.176+0000 7fe06d5dd740 10 client.15561 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-05-27T07:26:26.171 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:26:26.176+0000 7fe06d5dd740 20 client.15561 populate_metadata read hostname 'smithi052' 2024-05-27T07:26:26.171 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:26:26.176+0000 7fe06d5dd740 1 --2- 172.21.15.52:0/3947544389 >> [v2:172.21.15.105:6838/500348151,v1:172.21.15.105:6839/500348151] conn(0x5621805321c0 0x5621809c57d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:26:26.172 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:26:26.176+0000 7fe06d5dd740 4 client.15561 mds_command: new command op to 24527 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-05-27T07:26:26.172 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:26:26.177+0000 7fe06d5dd740 1 -- 172.21.15.52:0/3947544389 --> [v2:172.21.15.105:6838/500348151,v1:172.21.15.105:6839/500348151] -- command(tid 0: {"prefix": "flush journal"}) -- 0x5621808ad360 con 0x5621805321c0 2024-05-27T07:26:26.172 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:26:26.177+0000 7fe05ffff640 1 --2- 172.21.15.52:0/3947544389 >> [v2:172.21.15.105:6838/500348151,v1:172.21.15.105:6839/500348151] conn(0x5621805321c0 0x5621809c57d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:26:26.172 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:26:26.177+0000 7fe03f7fe640 10 client.15561 ms_handle_connect on v2:172.21.15.105:6838/500348151 2024-05-27T07:26:26.172 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:26:26.177+0000 7fe05ffff640 1 --2- 172.21.15.52:0/3947544389 >> [v2:172.21.15.105:6838/500348151,v1:172.21.15.105:6839/500348151] conn(0x5621805321c0 0x5621809c57d0 crc :-1 s=READY pgs=47 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=f0a655e22e0b424f server_cookie=39bf86deb361d7ff in_seq=0 out_seq=0 2024-05-27T07:26:26.181 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:26:26.190+0000 7fe03f7fe640 1 -- 172.21.15.52:0/3947544389 <== mds.0 v2:172.21.15.105:6838/500348151 1 ==== command_reply(tid 0: 0 ) ==== 8+0+25 (crc 0 0 0) 0x5621808ad360 con 0x5621805321c0 2024-05-27T07:26:26.181 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:26:26.190+0000 7fe03f7fe640 10 client.15561 handle_command_reply: tid=0 2024-05-27T07:26:26.181 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:26:26.190+0000 7fe06d5dd740 1 client.15561 shutdown 2024-05-27T07:26:26.181 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:26:26.191+0000 7fe06d5dd740 1 -- 172.21.15.52:0/3947544389 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x56218059e1b0 msgr2=0x56218095e3d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:26:26.181 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:26:26.191+0000 7fe06d5dd740 1 --2- 172.21.15.52:0/3947544389 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x56218059e1b0 0x56218095e3d0 secure :-1 s=READY pgs=452 cs=0 l=1 rev1=1 crypto rx=0x7fe054003f30 tx=0x7fe05402a410 comp rx=0 tx=0).stop 2024-05-27T07:26:26.181 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:26:26.191+0000 7fe06d5dd740 1 -- 172.21.15.52:0/3947544389 shutdown_connections 2024-05-27T07:26:26.181 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:26:26.191+0000 7fe06d5dd740 1 --2- 172.21.15.52:0/3947544389 >> [v2:172.21.15.105:6838/500348151,v1:172.21.15.105:6839/500348151] conn(0x5621805321c0 0x5621809c57d0 crc :-1 s=READY pgs=47 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:26:26.181 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:26:26.191+0000 7fe06d5dd740 1 --2- 172.21.15.52:0/3947544389 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x56218059e1b0 0x56218095e3d0 unknown :-1 s=CLOSED pgs=452 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:26:26.181 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:26:26.191+0000 7fe03f7fe640 0 client.15561 ms_handle_reset on v2:172.21.15.105:6838/500348151 2024-05-27T07:26:26.181 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:26:26.191+0000 7fe06d5dd740 1 --2- 172.21.15.52:0/3947544389 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x56218057d4d0 0x56218095de90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:26:26.181 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:26:26.191+0000 7fe06d5dd740 1 --2- 172.21.15.52:0/3947544389 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5621805752f0 0x56218095d950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:26:26.182 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:26:26.191+0000 7fe06d5dd740 1 -- 172.21.15.52:0/3947544389 >> 172.21.15.52:0/3947544389 conn(0x562180592300 msgr2=0x562180942e90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:26:26.182 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:26:26.191+0000 7fe06d5dd740 1 -- 172.21.15.52:0/3947544389 shutdown_connections 2024-05-27T07:26:26.182 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:26:26.191+0000 7fe06d5dd740 1 -- 172.21.15.52:0/3947544389 wait complete. 2024-05-27T07:26:26.182 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:26:26.191+0000 7fe06d5dd740 20 client.15561 trim_cache size 0 max 16384 2024-05-27T07:26:26.182 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stdout:{ 2024-05-27T07:26:26.182 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stdout: "return_code": 0 2024-05-27T07:26:26.182 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stdout:} 2024-05-27T07:26:26.183 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:26:26.192+0000 7fe06664c640 1 -- 172.21.15.52:0/743743678 >> [v2:172.21.15.52:6800/1822318384,v1:172.21.15.52:6801/1822318384] conn(0x7fe0480b2a70 msgr2=0x7fe0480b4f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:26:26.183 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:26:26.192+0000 7fe06664c640 1 --2- 172.21.15.52:0/743743678 >> [v2:172.21.15.52:6800/1822318384,v1:172.21.15.52:6801/1822318384] conn(0x7fe0480b2a70 0x7fe0480b4f80 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7fe0580e3c80 tx=0x7fe05807b2b0 comp rx=0 tx=0).stop 2024-05-27T07:26:26.183 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:26:26.192+0000 7fe06664c640 1 -- 172.21.15.52:0/743743678 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fe060109020 msgr2=0x7fe06019cf70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:26:26.184 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:26:26.192+0000 7fe06664c640 1 --2- 172.21.15.52:0/743743678 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fe060109020 0x7fe06019cf70 secure :-1 s=READY pgs=448 cs=0 l=1 rev1=1 crypto rx=0x7fe05800bfb0 tx=0x7fe0580114a0 comp rx=0 tx=0).stop 2024-05-27T07:26:26.184 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:26:26.192+0000 7fe06664c640 1 -- 172.21.15.52:0/743743678 shutdown_connections 2024-05-27T07:26:26.184 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:26:26.192+0000 7fe06664c640 1 --2- 172.21.15.52:0/743743678 >> [v2:172.21.15.52:6800/1822318384,v1:172.21.15.52:6801/1822318384] conn(0x7fe0480b2a70 0x7fe0480b4f80 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:26:26.184 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:26:26.192+0000 7fe06664c640 1 --2- 172.21.15.52:0/743743678 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fe060109020 0x7fe06019cf70 unknown :-1 s=CLOSED pgs=448 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:26:26.184 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:26:26.192+0000 7fe06664c640 1 --2- 172.21.15.52:0/743743678 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fe060108b80 0x7fe060198cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:26:26.184 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:26:26.192+0000 7fe06664c640 1 --2- 172.21.15.52:0/743743678 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fe0601032d0 0x7fe060198750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:26:26.184 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:26:26.192+0000 7fe06664c640 1 -- 172.21.15.52:0/743743678 >> 172.21.15.52:0/743743678 conn(0x7fe0600fe380 msgr2=0x7fe06010e730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:26:26.184 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:26:26.192+0000 7fe06664c640 1 -- 172.21.15.52:0/743743678 shutdown_connections 2024-05-27T07:26:26.184 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:26:26.192+0000 7fe06664c640 1 -- 172.21.15.52:0/743743678 wait complete. 2024-05-27T07:26:26.282 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:26:26 smithi052 ceph-mon[31955]: pgmap v17: 129 pgs: 129 active+clean; 1.3 MiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 26 KiB/s rd, 77 KiB/s wr, 44 op/s 2024-05-27T07:26:26.351 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:26:26 smithi173 ceph-mon[37000]: pgmap v17: 129 pgs: 129 active+clean; 1.3 MiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 26 KiB/s rd, 77 KiB/s wr, 44 op/s 2024-05-27T07:26:26.439 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:26:26 smithi105 ceph-mon[35977]: pgmap v17: 129 pgs: 129 active+clean; 1.3 MiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 26 KiB/s rd, 77 KiB/s wr, 44 op/s 2024-05-27T07:26:27.352 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:26:27 smithi173 ceph-mon[37000]: scrub summary: idle 2024-05-27T07:26:27.439 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:26:27 smithi105 ceph-mon[35977]: scrub summary: idle 2024-05-27T07:26:27.479 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:26:27 smithi052 ceph-mon[31955]: scrub summary: idle 2024-05-27T07:26:28.351 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:26:28 smithi173 ceph-mon[37000]: pgmap v18: 129 pgs: 129 active+clean; 1.3 MiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 853 B/s wr, 8 op/s 2024-05-27T07:26:28.439 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:26:28 smithi105 ceph-mon[35977]: pgmap v18: 129 pgs: 129 active+clean; 1.3 MiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 853 B/s wr, 8 op/s 2024-05-27T07:26:28.479 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:26:28 smithi052 ceph-mon[31955]: pgmap v18: 129 pgs: 129 active+clean; 1.3 MiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 853 B/s wr, 8 op/s 2024-05-27T07:26:30.351 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:26:30 smithi173 ceph-mon[37000]: pgmap v19: 129 pgs: 129 active+clean; 1.3 MiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 6.9 KiB/s rd, 2.3 KiB/s wr, 17 op/s 2024-05-27T07:26:30.439 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:26:30 smithi105 ceph-mon[35977]: pgmap v19: 129 pgs: 129 active+clean; 1.3 MiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 6.9 KiB/s rd, 2.3 KiB/s wr, 17 op/s 2024-05-27T07:26:30.479 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:26:30 smithi052 ceph-mon[31955]: pgmap v19: 129 pgs: 129 active+clean; 1.3 MiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 6.9 KiB/s rd, 2.3 KiB/s wr, 17 op/s 2024-05-27T07:26:32.352 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:26:32 smithi173 ceph-mon[37000]: pgmap v20: 129 pgs: 129 active+clean; 1.3 MiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 1.5 KiB/s wr, 12 op/s 2024-05-27T07:26:32.439 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:26:32 smithi105 ceph-mon[35977]: pgmap v20: 129 pgs: 129 active+clean; 1.3 MiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 1.5 KiB/s wr, 12 op/s 2024-05-27T07:26:32.479 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:26:32 smithi052 ceph-mon[31955]: pgmap v20: 129 pgs: 129 active+clean; 1.3 MiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 1.5 KiB/s wr, 12 op/s 2024-05-27T07:26:34.352 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:26:34 smithi173 ceph-mon[37000]: pgmap v21: 129 pgs: 129 active+clean; 1.3 MiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 5.9 KiB/s rd, 1.5 KiB/s wr, 16 op/s 2024-05-27T07:26:34.440 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:26:34 smithi105 ceph-mon[35977]: pgmap v21: 129 pgs: 129 active+clean; 1.3 MiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 5.9 KiB/s rd, 1.5 KiB/s wr, 16 op/s 2024-05-27T07:26:34.479 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:26:34 smithi052 ceph-mon[31955]: pgmap v21: 129 pgs: 129 active+clean; 1.3 MiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 5.9 KiB/s rd, 1.5 KiB/s wr, 16 op/s 2024-05-27T07:26:36.351 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:26:36 smithi173 ceph-mon[37000]: pgmap v22: 129 pgs: 129 active+clean; 1.3 MiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 5.9 KiB/s rd, 1.5 KiB/s wr, 16 op/s 2024-05-27T07:26:36.439 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:26:36 smithi105 ceph-mon[35977]: pgmap v22: 129 pgs: 129 active+clean; 1.3 MiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 5.9 KiB/s rd, 1.5 KiB/s wr, 16 op/s 2024-05-27T07:26:36.479 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:26:36 smithi052 ceph-mon[31955]: pgmap v22: 129 pgs: 129 active+clean; 1.3 MiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 5.9 KiB/s rd, 1.5 KiB/s wr, 16 op/s 2024-05-27T07:26:38.352 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:26:38 smithi173 ceph-mon[37000]: pgmap v23: 129 pgs: 129 active+clean; 1.3 MiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 1.5 KiB/s wr, 12 op/s 2024-05-27T07:26:38.439 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:26:38 smithi105 ceph-mon[35977]: pgmap v23: 129 pgs: 129 active+clean; 1.3 MiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 1.5 KiB/s wr, 12 op/s 2024-05-27T07:26:38.479 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:26:38 smithi052 ceph-mon[31955]: pgmap v23: 129 pgs: 129 active+clean; 1.3 MiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 1.5 KiB/s wr, 12 op/s 2024-05-27T07:26:39.297 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:26:39.307+0000 7fa25a11f640 1 -- 172.21.15.52:0/1085476176 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fa2541043d0 msgr2=0x7fa2541107a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:26:39.298 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:26:39.307+0000 7fa25a11f640 1 --2- 172.21.15.52:0/1085476176 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fa2541043d0 0x7fa2541107a0 secure :-1 s=READY pgs=453 cs=0 l=1 rev1=1 crypto rx=0x7fa24c009ca0 tx=0x7fa24c02a8c0 comp rx=0 tx=0).stop 2024-05-27T07:26:39.298 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:26:39.307+0000 7fa25a11f640 1 -- 172.21.15.52:0/1085476176 shutdown_connections 2024-05-27T07:26:39.298 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:26:39.307+0000 7fa25a11f640 1 --2- 172.21.15.52:0/1085476176 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fa2541043d0 0x7fa2541107a0 unknown :-1 s=CLOSED pgs=453 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:26:39.298 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:26:39.307+0000 7fa25a11f640 1 --2- 172.21.15.52:0/1085476176 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fa25410a120 0x7fa254103d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:26:39.298 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:26:39.307+0000 7fa25a11f640 1 --2- 172.21.15.52:0/1085476176 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fa254109c80 0x7fa254103820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:26:39.298 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:26:39.307+0000 7fa25a11f640 1 -- 172.21.15.52:0/1085476176 >> 172.21.15.52:0/1085476176 conn(0x7fa254079290 msgr2=0x7fa2540796d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:26:39.298 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:26:39.307+0000 7fa25a11f640 1 -- 172.21.15.52:0/1085476176 shutdown_connections 2024-05-27T07:26:39.298 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:26:39.307+0000 7fa25a11f640 1 -- 172.21.15.52:0/1085476176 wait complete. 2024-05-27T07:26:39.299 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:26:39.308+0000 7fa25a11f640 1 Processor -- start 2024-05-27T07:26:39.300 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:26:39.308+0000 7fa25a11f640 1 -- start start 2024-05-27T07:26:39.301 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:26:39.309+0000 7fa25a11f640 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fa2541043d0 0x7fa2541a04e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:26:39.301 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:26:39.309+0000 7fa25a11f640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fa254109c80 0x7fa2541a0a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:26:39.301 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:26:39.309+0000 7fa25a11f640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fa25410a120 0x7fa25419a6b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:26:39.301 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:26:39.309+0000 7fa25a11f640 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x7fa254113e90 con 0x7fa2541043d0 2024-05-27T07:26:39.301 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:26:39.309+0000 7fa25a11f640 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7fa254113d10 con 0x7fa254109c80 2024-05-27T07:26:39.301 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:26:39.309+0000 7fa25a11f640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7fa254114010 con 0x7fa25410a120 2024-05-27T07:26:39.301 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:26:39.309+0000 7fa2537fe640 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fa2541043d0 0x7fa2541a04e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:26:39.301 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:26:39.309+0000 7fa2537fe640 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fa2541043d0 0x7fa2541a04e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.52:48762/0 (socket says 172.21.15.52:48762) 2024-05-27T07:26:39.301 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:26:39.309+0000 7fa2537fe640 1 -- 172.21.15.52:0/4034355998 learned_addr learned my addr 172.21.15.52:0/4034355998 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-05-27T07:26:39.302 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:26:39.309+0000 7fa253fff640 1 --2- 172.21.15.52:0/4034355998 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fa25410a120 0x7fa25419a6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:26:39.302 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:26:39.309+0000 7fa252ffd640 1 --2- 172.21.15.52:0/4034355998 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fa254109c80 0x7fa2541a0a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:26:39.302 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:26:39.309+0000 7fa2537fe640 1 -- 172.21.15.52:0/4034355998 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fa25410a120 msgr2=0x7fa25419a6b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:26:39.302 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:26:39.309+0000 7fa2537fe640 1 --2- 172.21.15.52:0/4034355998 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fa25410a120 0x7fa25419a6b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:26:39.302 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:26:39.309+0000 7fa2537fe640 1 -- 172.21.15.52:0/4034355998 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fa254109c80 msgr2=0x7fa2541a0a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:26:39.302 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:26:39.309+0000 7fa2537fe640 1 --2- 172.21.15.52:0/4034355998 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fa254109c80 0x7fa2541a0a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:26:39.302 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:26:39.309+0000 7fa2537fe640 1 -- 172.21.15.52:0/4034355998 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa25419ade0 con 0x7fa2541043d0 2024-05-27T07:26:39.302 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:26:39.310+0000 7fa2537fe640 1 --2- 172.21.15.52:0/4034355998 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fa2541043d0 0x7fa2541a04e0 secure :-1 s=READY pgs=454 cs=0 l=1 rev1=1 crypto rx=0x7fa244005fd0 tx=0x7fa24400dde0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:26:39.302 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:26:39.310+0000 7fa250ff9640 1 -- 172.21.15.52:0/4034355998 <== mon.0 v2:172.21.15.52:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa244017070 con 0x7fa2541043d0 2024-05-27T07:26:39.302 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:26:39.310+0000 7fa25a11f640 1 -- 172.21.15.52:0/4034355998 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa25419b070 con 0x7fa2541043d0 2024-05-27T07:26:39.303 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:26:39.310+0000 7fa250ff9640 1 -- 172.21.15.52:0/4034355998 <== mon.0 v2:172.21.15.52:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fa244009c30 con 0x7fa2541043d0 2024-05-27T07:26:39.303 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:26:39.310+0000 7fa250ff9640 1 -- 172.21.15.52:0/4034355998 <== mon.0 v2:172.21.15.52:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa244003d30 con 0x7fa2541043d0 2024-05-27T07:26:39.303 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:26:39.310+0000 7fa25a11f640 1 -- 172.21.15.52:0/4034355998 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa2541a7280 con 0x7fa2541043d0 2024-05-27T07:26:39.310 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:26:39.312+0000 7fa260fd0740 1 Processor -- start 2024-05-27T07:26:39.310 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:26:39.312+0000 7fa260fd0740 1 -- start start 2024-05-27T07:26:39.310 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:26:39.313+0000 7fa250ff9640 1 -- 172.21.15.52:0/4034355998 <== mon.0 v2:172.21.15.52:3300/0 4 ==== mgrmap(e 24) ==== 149879+0+0 (secure 0 0 0) 0x7fa244003ed0 con 0x7fa2541043d0 2024-05-27T07:26:39.310 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:26:39.313+0000 7fa260fd0740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55f1523d61b0 0x55f1527773d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:26:39.310 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:26:39.313+0000 7fa260fd0740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55f1523ad2f0 0x55f152777910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:26:39.311 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:26:39.313+0000 7fa260fd0740 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55f1523b54d0 0x55f152777e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:26:39.311 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:26:39.313+0000 7fa260fd0740 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x55f1526e9c30 con 0x55f1523b54d0 2024-05-27T07:26:39.311 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:26:39.313+0000 7fa260fd0740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x55f1526d97d0 con 0x55f1523d61b0 2024-05-27T07:26:39.311 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:26:39.313+0000 7fa260fd0740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x55f1526dd810 con 0x55f1523ad2f0 2024-05-27T07:26:39.311 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:26:39.313+0000 7fa252ffd640 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55f1523b54d0 0x55f152777e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:26:39.311 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:26:39.313+0000 7fa252ffd640 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55f1523b54d0 0x55f152777e50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.52:48772/0 (socket says 172.21.15.52:48772) 2024-05-27T07:26:39.311 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:26:39.313+0000 7fa252ffd640 1 -- 172.21.15.52:0/351875779 learned_addr learned my addr 172.21.15.52:0/351875779 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-05-27T07:26:39.311 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:26:39.313+0000 7fa2537fe640 1 --2- 172.21.15.52:0/351875779 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55f1523ad2f0 0x55f152777910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:26:39.311 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:26:39.313+0000 7fa253fff640 1 --2- 172.21.15.52:0/351875779 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55f1523d61b0 0x55f1527773d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:26:39.311 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:26:39.313+0000 7fa252ffd640 1 -- 172.21.15.52:0/351875779 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55f1523ad2f0 msgr2=0x55f152777910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:26:39.311 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:26:39.313+0000 7fa252ffd640 1 --2- 172.21.15.52:0/351875779 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55f1523ad2f0 0x55f152777910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:26:39.311 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:26:39.313+0000 7fa252ffd640 1 -- 172.21.15.52:0/351875779 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55f1523d61b0 msgr2=0x55f1527773d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:26:39.311 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:26:39.313+0000 7fa252ffd640 1 --2- 172.21.15.52:0/351875779 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55f1523d61b0 0x55f1527773d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:26:39.312 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:26:39.313+0000 7fa252ffd640 1 -- 172.21.15.52:0/351875779 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55f1523ee210 con 0x55f1523b54d0 2024-05-27T07:26:39.312 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:26:39.314+0000 7fa252ffd640 1 --2- 172.21.15.52:0/351875779 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55f1523b54d0 0x55f152777e50 secure :-1 s=READY pgs=455 cs=0 l=1 rev1=1 crypto rx=0x7fa24800bb80 tx=0x7fa2480107a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=81cb4e988dff1243 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:26:39.312 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:26:39.314+0000 7fa250ff9640 1 --2- 172.21.15.52:0/4034355998 >> [v2:172.21.15.52:6800/1822318384,v1:172.21.15.52:6801/1822318384] conn(0x7fa23c0b2a70 0x7fa23c0b4f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:26:39.312 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:26:39.314+0000 7fa239ffb640 1 -- 172.21.15.52:0/351875779 <== mon.0 v2:172.21.15.52:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa248005020 con 0x55f1523b54d0 2024-05-27T07:26:39.312 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:26:39.314+0000 7fa239ffb640 1 -- 172.21.15.52:0/351875779 <== mon.0 v2:172.21.15.52:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fa2480047a0 con 0x55f1523b54d0 2024-05-27T07:26:39.312 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:26:39.314+0000 7fa239ffb640 1 -- 172.21.15.52:0/351875779 <== mon.0 v2:172.21.15.52:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa248011680 con 0x55f1523b54d0 2024-05-27T07:26:39.312 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:26:39.314+0000 7fa253fff640 1 --2- 172.21.15.52:0/4034355998 >> [v2:172.21.15.52:6800/1822318384,v1:172.21.15.52:6801/1822318384] conn(0x7fa23c0b2a70 0x7fa23c0b4f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:26:39.312 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:26:39.314+0000 7fa250ff9640 1 -- 172.21.15.52:0/4034355998 <== mon.0 v2:172.21.15.52:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9379+0+0 (secure 0 0 0) 0x7fa2440e2800 con 0x7fa2541043d0 2024-05-27T07:26:39.312 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:26:39.315+0000 7fa260fd0740 1 -- 172.21.15.52:0/351875779 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55f1523b54d0 msgr2=0x55f152777e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:26:39.312 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:26:39.315+0000 7fa253fff640 1 --2- 172.21.15.52:0/4034355998 >> [v2:172.21.15.52:6800/1822318384,v1:172.21.15.52:6801/1822318384] conn(0x7fa23c0b2a70 0x7fa23c0b4f80 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x55f152774540 tx=0x7fa24c02f390 comp rx=0 tx=0).ready entity=mgr.15333 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:26:39.312 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:26:39.315+0000 7fa260fd0740 1 --2- 172.21.15.52:0/351875779 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55f1523b54d0 0x55f152777e50 secure :-1 s=READY pgs=455 cs=0 l=1 rev1=1 crypto rx=0x7fa24800bb80 tx=0x7fa2480107a0 comp rx=0 tx=0).stop 2024-05-27T07:26:39.312 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:26:39.315+0000 7fa260fd0740 1 -- 172.21.15.52:0/351875779 shutdown_connections 2024-05-27T07:26:39.312 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:26:39.315+0000 7fa260fd0740 1 --2- 172.21.15.52:0/351875779 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55f1523b54d0 0x55f152777e50 unknown :-1 s=CLOSED pgs=455 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:26:39.313 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:26:39.315+0000 7fa260fd0740 1 --2- 172.21.15.52:0/351875779 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55f1523ad2f0 0x55f152777910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:26:39.313 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:26:39.315+0000 7fa260fd0740 1 --2- 172.21.15.52:0/351875779 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55f1523d61b0 0x55f1527773d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:26:39.313 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:26:39.315+0000 7fa260fd0740 1 -- 172.21.15.52:0/351875779 >> 172.21.15.52:0/351875779 conn(0x55f1523ca300 msgr2=0x55f152773b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:26:39.313 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:26:39.315+0000 7fa260fd0740 1 -- 172.21.15.52:0/351875779 shutdown_connections 2024-05-27T07:26:39.313 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:26:39.315+0000 7fa260fd0740 1 -- 172.21.15.52:0/351875779 wait complete. 2024-05-27T07:26:39.313 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:26:39.315+0000 7fa260fd0740 1 -- start start 2024-05-27T07:26:39.313 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:26:39.316+0000 7fa260fd0740 1 Processor -- start 2024-05-27T07:26:39.313 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:26:39.316+0000 7fa260fd0740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55f1523ad2f0 0x55f1527834b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:26:39.313 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:26:39.316+0000 7fa260fd0740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55f1523b54d0 0x55f1527839f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:26:39.313 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:26:39.316+0000 7fa260fd0740 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55f1523d61b0 0x55f152781910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:26:39.313 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:26:39.316+0000 7fa260fd0740 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x55f1523bf790 con 0x55f1523d61b0 2024-05-27T07:26:39.313 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:26:39.316+0000 7fa260fd0740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x55f152695890 con 0x55f1523ad2f0 2024-05-27T07:26:39.314 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:26:39.316+0000 7fa260fd0740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x55f152784500 con 0x55f1523b54d0 2024-05-27T07:26:39.314 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:26:39.316+0000 7fa260fd0740 10 client.0 _ll_register_callbacks cb 0x55f1523eda00 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T07:26:39.314 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:26:39.316+0000 7fa253fff640 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55f1523d61b0 0x55f152781910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:26:39.314 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:26:39.316+0000 7fa253fff640 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55f1523d61b0 0x55f152781910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.52:48784/0 (socket says 172.21.15.52:48784) 2024-05-27T07:26:39.314 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:26:39.316+0000 7fa253fff640 1 -- 172.21.15.52:0/3882371446 learned_addr learned my addr 172.21.15.52:0/3882371446 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-05-27T07:26:39.314 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:26:39.316+0000 7fa2537fe640 1 --2- 172.21.15.52:0/3882371446 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55f1523ad2f0 0x55f1527834b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:26:39.314 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:26:39.316+0000 7fa252ffd640 1 --2- 172.21.15.52:0/3882371446 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55f1523b54d0 0x55f1527839f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:26:39.314 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:26:39.317+0000 7fa253fff640 1 -- 172.21.15.52:0/3882371446 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55f1523b54d0 msgr2=0x55f1527839f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:26:39.314 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:26:39.317+0000 7fa253fff640 1 --2- 172.21.15.52:0/3882371446 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55f1523b54d0 0x55f1527839f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:26:39.314 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:26:39.317+0000 7fa253fff640 1 -- 172.21.15.52:0/3882371446 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55f1523ad2f0 msgr2=0x55f1527834b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:26:39.314 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:26:39.317+0000 7fa253fff640 1 --2- 172.21.15.52:0/3882371446 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55f1523ad2f0 0x55f1527834b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:26:39.314 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:26:39.317+0000 7fa253fff640 1 -- 172.21.15.52:0/3882371446 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55f1524ba620 con 0x55f1523d61b0 2024-05-27T07:26:39.315 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:26:39.317+0000 7fa253fff640 1 --2- 172.21.15.52:0/3882371446 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55f1523d61b0 0x55f152781910 secure :-1 s=READY pgs=456 cs=0 l=1 rev1=1 crypto rx=0x7fa24c031540 tx=0x7fa24c02a8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=be5f2b4abaacb525 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:26:39.315 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:26:39.317+0000 7fa232ffd640 10 client.0 ms_handle_connect on v2:172.21.15.52:3300/0 2024-05-27T07:26:39.315 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:26:39.317+0000 7fa232ffd640 1 -- 172.21.15.52:0/3882371446 <== mon.0 v2:172.21.15.52:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa24c003e00 con 0x55f1523d61b0 2024-05-27T07:26:39.315 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:26:39.317+0000 7fa232ffd640 1 -- 172.21.15.52:0/3882371446 <== mon.0 v2:172.21.15.52:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 9379+0+0 (secure 0 0 0) 0x7fa24c004040 con 0x55f1523d61b0 2024-05-27T07:26:39.315 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:26:39.317+0000 7fa260fd0740 1 -- 172.21.15.52:0/3882371446 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fa2480047a0 con 0x55f1523d61b0 2024-05-27T07:26:39.315 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:26:39.317+0000 7fa232ffd640 1 -- 172.21.15.52:0/3882371446 <== mon.0 v2:172.21.15.52:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7fa24c004240 con 0x55f1523d61b0 2024-05-27T07:26:39.315 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:26:39.317+0000 7fa260fd0740 10 client.15573 fetch_fsmap learned FSMap version 30 2024-05-27T07:26:39.315 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:26:39.317+0000 7fa260fd0740 1 -- 172.21.15.52:0/3882371446 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x55f152790800 con 0x55f1523d61b0 2024-05-27T07:26:39.315 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:26:39.317+0000 7fa232ffd640 1 -- 172.21.15.52:0/3882371446 <== mon.0 v2:172.21.15.52:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fa2480047a0 con 0x55f1523d61b0 2024-05-27T07:26:39.315 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:26:39.318+0000 7fa232ffd640 1 -- 172.21.15.52:0/3882371446 <== mon.0 v2:172.21.15.52:3300/0 5 ==== fsmap(e 30) ==== 8362+0+0 (secure 0 0 0) 0x7fa24c051070 con 0x55f1523d61b0 2024-05-27T07:26:39.315 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:26:39.318+0000 7fa260fd0740 10 client.15573 fetch_fsmap finished waiting for FSMap version 30 2024-05-27T07:26:39.315 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:26:39.318+0000 7fa260fd0740 10 client.15573 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-05-27T07:26:39.315 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:26:39.318+0000 7fa260fd0740 20 client.15573 populate_metadata read hostname 'smithi052' 2024-05-27T07:26:39.316 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:26:39.318+0000 7fa260fd0740 1 --2- 172.21.15.52:0/3882371446 >> [v2:172.21.15.105:6838/500348151,v1:172.21.15.105:6839/500348151] conn(0x55f152394d80 0x55f152793de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:26:39.316 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:26:39.318+0000 7fa260fd0740 4 client.15573 mds_command: new command op to 24527 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-27T07:26:39.316 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:26:39.318+0000 7fa260fd0740 1 -- 172.21.15.52:0/3882371446 --> [v2:172.21.15.105:6838/500348151,v1:172.21.15.105:6839/500348151] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55f1526ebb80 con 0x55f152394d80 2024-05-27T07:26:39.316 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:26:39.318+0000 7fa2537fe640 1 --2- 172.21.15.52:0/3882371446 >> [v2:172.21.15.105:6838/500348151,v1:172.21.15.105:6839/500348151] conn(0x55f152394d80 0x55f152793de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:26:39.316 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:26:39.318+0000 7fa2537fe640 1 --2- 172.21.15.52:0/3882371446 >> [v2:172.21.15.105:6838/500348151,v1:172.21.15.105:6839/500348151] conn(0x55f152394d80 0x55f152793de0 crc :-1 s=READY pgs=48 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=7e00af5f45f25999 server_cookie=37ba56d4f64cd508 in_seq=0 out_seq=0 2024-05-27T07:26:39.316 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:26:39.318+0000 7fa232ffd640 10 client.15573 ms_handle_connect on v2:172.21.15.105:6838/500348151 2024-05-27T07:26:39.316 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:26:39.319+0000 7fa232ffd640 1 -- 172.21.15.52:0/3882371446 <== mds.0 v2:172.21.15.105:6838/500348151 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24041 (crc 0 0 0) 0x55f1526ebb80 con 0x55f152394d80 2024-05-27T07:26:39.316 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:26:39.319+0000 7fa232ffd640 10 client.15573 handle_command_reply: tid=0 2024-05-27T07:26:39.316 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:26:39.319+0000 7fa260fd0740 1 client.15573 shutdown 2024-05-27T07:26:39.316 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:26:39.320+0000 7fa260fd0740 1 -- 172.21.15.52:0/3882371446 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55f1523d61b0 msgr2=0x55f152781910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:26:39.316 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:26:39.320+0000 7fa260fd0740 1 --2- 172.21.15.52:0/3882371446 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55f1523d61b0 0x55f152781910 secure :-1 s=READY pgs=456 cs=0 l=1 rev1=1 crypto rx=0x7fa24c031540 tx=0x7fa24c02a8c0 comp rx=0 tx=0).stop 2024-05-27T07:26:39.316 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:26:39.320+0000 7fa260fd0740 1 -- 172.21.15.52:0/3882371446 shutdown_connections 2024-05-27T07:26:39.316 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:26:39.320+0000 7fa260fd0740 1 --2- 172.21.15.52:0/3882371446 >> [v2:172.21.15.105:6838/500348151,v1:172.21.15.105:6839/500348151] conn(0x55f152394d80 0x55f152793de0 crc :-1 s=READY pgs=48 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:26:39.316 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:26:39.320+0000 7fa260fd0740 1 --2- 172.21.15.52:0/3882371446 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55f1523d61b0 0x55f152781910 unknown :-1 s=CLOSED pgs=456 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:26:39.317 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:26:39.320+0000 7fa260fd0740 1 --2- 172.21.15.52:0/3882371446 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55f1523b54d0 0x55f1527839f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:26:39.317 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:26:39.320+0000 7fa232ffd640 0 client.15573 ms_handle_reset on v2:172.21.15.105:6838/500348151 2024-05-27T07:26:39.317 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:26:39.320+0000 7fa260fd0740 1 --2- 172.21.15.52:0/3882371446 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55f1523ad2f0 0x55f1527834b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:26:39.317 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:26:39.320+0000 7fa260fd0740 1 -- 172.21.15.52:0/3882371446 >> 172.21.15.52:0/3882371446 conn(0x55f1523ca300 msgr2=0x55f1527790f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:26:39.317 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:26:39.320+0000 7fa260fd0740 1 -- 172.21.15.52:0/3882371446 shutdown_connections 2024-05-27T07:26:39.317 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:26:39.320+0000 7fa260fd0740 1 -- 172.21.15.52:0/3882371446 wait complete. 2024-05-27T07:26:39.317 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:26:39.320+0000 7fa260fd0740 20 client.15573 trim_cache size 0 max 16384 2024-05-27T07:26:39.326 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:26:39.335+0000 7fa260fd0740 1 Processor -- start 2024-05-27T07:26:39.326 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:26:39.335+0000 7fa260fd0740 1 -- start start 2024-05-27T07:26:39.326 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:26:39.336+0000 7fa260fd0740 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55f1523ad2f0 0x55f152779a40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:26:39.326 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:26:39.336+0000 7fa260fd0740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55f1523b54d0 0x55f152778260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:26:39.326 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:26:39.336+0000 7fa260fd0740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55f1523d61b0 0x55f1527818b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:26:39.326 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:26:39.336+0000 7fa260fd0740 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x7fa2541b10d0 con 0x55f1523ad2f0 2024-05-27T07:26:39.327 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:26:39.336+0000 7fa260fd0740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7fa2541b2c10 con 0x55f1523d61b0 2024-05-27T07:26:39.327 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:26:39.336+0000 7fa260fd0740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7fa2541b17a0 con 0x55f1523b54d0 2024-05-27T07:26:39.327 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:26:39.336+0000 7fa2537fe640 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55f1523ad2f0 0x55f152779a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:26:39.327 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:26:39.336+0000 7fa2537fe640 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55f1523ad2f0 0x55f152779a40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.52:48798/0 (socket says 172.21.15.52:48798) 2024-05-27T07:26:39.327 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:26:39.336+0000 7fa2537fe640 1 -- 172.21.15.52:0/147770678 learned_addr learned my addr 172.21.15.52:0/147770678 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-05-27T07:26:39.327 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:26:39.336+0000 7fa252ffd640 1 --2- 172.21.15.52:0/147770678 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55f1523b54d0 0x55f152778260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:26:39.327 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:26:39.336+0000 7fa253fff640 1 --2- 172.21.15.52:0/147770678 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55f1523d61b0 0x55f1527818b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:26:39.327 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:26:39.336+0000 7fa2537fe640 1 -- 172.21.15.52:0/147770678 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55f1523b54d0 msgr2=0x55f152778260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:26:39.327 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:26:39.336+0000 7fa2537fe640 1 --2- 172.21.15.52:0/147770678 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55f1523b54d0 0x55f152778260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:26:39.327 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:26:39.336+0000 7fa2537fe640 1 -- 172.21.15.52:0/147770678 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55f1523d61b0 msgr2=0x55f1527818b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:26:39.328 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:26:39.336+0000 7fa2537fe640 1 --2- 172.21.15.52:0/147770678 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55f1523d61b0 0x55f1527818b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:26:39.328 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:26:39.336+0000 7fa2537fe640 1 -- 172.21.15.52:0/147770678 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55f152783280 con 0x55f1523ad2f0 2024-05-27T07:26:39.328 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:26:39.336+0000 7fa252ffd640 1 --2- 172.21.15.52:0/147770678 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55f1523b54d0 0x55f152778260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-27T07:26:39.328 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:26:39.336+0000 7fa2537fe640 1 --2- 172.21.15.52:0/147770678 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55f1523ad2f0 0x55f152779a40 secure :-1 s=READY pgs=457 cs=0 l=1 rev1=1 crypto rx=0x7fa2440a55b0 tx=0x7fa2440e4040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=75c55e508a5235d1 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:26:39.328 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:26:39.336+0000 7fa2397fa640 1 -- 172.21.15.52:0/147770678 <== mon.0 v2:172.21.15.52:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa2440e5e60 con 0x55f1523ad2f0 2024-05-27T07:26:39.328 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:26:39.336+0000 7fa2397fa640 1 -- 172.21.15.52:0/147770678 <== mon.0 v2:172.21.15.52:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fa2440a9c30 con 0x55f1523ad2f0 2024-05-27T07:26:39.328 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:26:39.336+0000 7fa2397fa640 1 -- 172.21.15.52:0/147770678 <== mon.0 v2:172.21.15.52:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa2440a57c0 con 0x55f1523ad2f0 2024-05-27T07:26:39.335 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:26:39.337+0000 7fa260fd0740 1 -- 172.21.15.52:0/147770678 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55f1523ad2f0 msgr2=0x55f152779a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:26:39.335 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:26:39.337+0000 7fa260fd0740 1 --2- 172.21.15.52:0/147770678 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55f1523ad2f0 0x55f152779a40 secure :-1 s=READY pgs=457 cs=0 l=1 rev1=1 crypto rx=0x7fa2440a55b0 tx=0x7fa2440e4040 comp rx=0 tx=0).stop 2024-05-27T07:26:39.335 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:26:39.337+0000 7fa260fd0740 1 -- 172.21.15.52:0/147770678 shutdown_connections 2024-05-27T07:26:39.335 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:26:39.337+0000 7fa260fd0740 1 --2- 172.21.15.52:0/147770678 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55f1523d61b0 0x55f1527818b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:26:39.335 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:26:39.337+0000 7fa260fd0740 1 --2- 172.21.15.52:0/147770678 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55f1523b54d0 0x55f152778260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:26:39.336 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:26:39.337+0000 7fa260fd0740 1 --2- 172.21.15.52:0/147770678 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55f1523ad2f0 0x55f152779a40 unknown :-1 s=CLOSED pgs=457 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:26:39.336 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:26:39.337+0000 7fa260fd0740 1 -- 172.21.15.52:0/147770678 >> 172.21.15.52:0/147770678 conn(0x55f1523ca300 msgr2=0x55f1527811a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:26:39.336 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:26:39.337+0000 7fa260fd0740 1 -- 172.21.15.52:0/147770678 shutdown_connections 2024-05-27T07:26:39.336 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:26:39.337+0000 7fa260fd0740 1 -- 172.21.15.52:0/147770678 wait complete. 2024-05-27T07:26:39.336 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:26:39.338+0000 7fa260fd0740 1 -- start start 2024-05-27T07:26:39.336 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:26:39.338+0000 7fa260fd0740 1 Processor -- start 2024-05-27T07:26:39.336 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:26:39.338+0000 7fa260fd0740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55f1523ad2f0 0x55f152784790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:26:39.336 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:26:39.338+0000 7fa260fd0740 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55f1523b54d0 0x55f152784cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:26:39.337 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:26:39.338+0000 7fa260fd0740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55f1523d61b0 0x55f152785210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:26:39.337 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:26:39.338+0000 7fa260fd0740 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x7fa2541b2540 con 0x55f1523b54d0 2024-05-27T07:26:39.337 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:26:39.338+0000 7fa260fd0740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7fa2541b1e70 con 0x55f1523d61b0 2024-05-27T07:26:39.337 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:26:39.338+0000 7fa260fd0740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7fa2541af0b0 con 0x55f1523ad2f0 2024-05-27T07:26:39.337 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:26:39.338+0000 7fa260fd0740 10 client.0 _ll_register_callbacks cb 0x7fa24c031450 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T07:26:39.337 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:26:39.338+0000 7fa252ffd640 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55f1523b54d0 0x55f152784cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:26:39.337 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:26:39.338+0000 7fa252ffd640 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55f1523b54d0 0x55f152784cd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.52:48808/0 (socket says 172.21.15.52:48808) 2024-05-27T07:26:39.337 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:26:39.338+0000 7fa252ffd640 1 -- 172.21.15.52:0/923078439 learned_addr learned my addr 172.21.15.52:0/923078439 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-05-27T07:26:39.338 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:26:39.338+0000 7fa2537fe640 1 --2- 172.21.15.52:0/923078439 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55f1523ad2f0 0x55f152784790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:26:39.338 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:26:39.339+0000 7fa253fff640 1 --2- 172.21.15.52:0/923078439 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55f1523d61b0 0x55f152785210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:26:39.338 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:26:39.339+0000 7fa252ffd640 1 -- 172.21.15.52:0/923078439 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55f1523ad2f0 msgr2=0x55f152784790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:26:39.338 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:26:39.339+0000 7fa252ffd640 1 --2- 172.21.15.52:0/923078439 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55f1523ad2f0 0x55f152784790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:26:39.338 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:26:39.339+0000 7fa252ffd640 1 -- 172.21.15.52:0/923078439 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55f1523d61b0 msgr2=0x55f152785210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:26:39.338 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:26:39.339+0000 7fa252ffd640 1 --2- 172.21.15.52:0/923078439 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55f1523d61b0 0x55f152785210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:26:39.338 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:26:39.339+0000 7fa252ffd640 1 -- 172.21.15.52:0/923078439 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55f1527fa660 con 0x55f1523b54d0 2024-05-27T07:26:39.339 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:26:39.339+0000 7fa253fff640 1 --2- 172.21.15.52:0/923078439 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55f1523d61b0 0x55f152785210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-27T07:26:39.339 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:26:39.339+0000 7fa252ffd640 1 --2- 172.21.15.52:0/923078439 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55f1523b54d0 0x55f152784cd0 secure :-1 s=READY pgs=458 cs=0 l=1 rev1=1 crypto rx=0x7fa24800bb80 tx=0x7fa24800b8a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=56920a624188c75f server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:26:39.339 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:26:39.339+0000 7fa232ffd640 10 client.0 ms_handle_connect on v2:172.21.15.52:3300/0 2024-05-27T07:26:39.339 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:26:39.339+0000 7fa232ffd640 1 -- 172.21.15.52:0/923078439 <== mon.0 v2:172.21.15.52:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa24802d060 con 0x55f1523b54d0 2024-05-27T07:26:39.339 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:26:39.339+0000 7fa232ffd640 1 -- 172.21.15.52:0/923078439 <== mon.0 v2:172.21.15.52:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 9379+0+0 (secure 0 0 0) 0x7fa248005230 con 0x55f1523b54d0 2024-05-27T07:26:39.339 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:26:39.339+0000 7fa260fd0740 1 -- 172.21.15.52:0/923078439 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fa2440a9c30 con 0x55f1523b54d0 2024-05-27T07:26:39.339 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:26:39.339+0000 7fa232ffd640 1 -- 172.21.15.52:0/923078439 <== mon.0 v2:172.21.15.52:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7fa248034050 con 0x55f1523b54d0 2024-05-27T07:26:39.339 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:26:39.339+0000 7fa260fd0740 10 client.15579 fetch_fsmap learned FSMap version 30 2024-05-27T07:26:39.340 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:26:39.339+0000 7fa260fd0740 1 -- 172.21.15.52:0/923078439 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x55f1527fb8b0 con 0x55f1523b54d0 2024-05-27T07:26:39.340 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:26:39.339+0000 7fa232ffd640 1 -- 172.21.15.52:0/923078439 <== mon.0 v2:172.21.15.52:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fa2440a9c30 con 0x55f1523b54d0 2024-05-27T07:26:39.340 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:26:39.340+0000 7fa232ffd640 1 -- 172.21.15.52:0/923078439 <== mon.0 v2:172.21.15.52:3300/0 5 ==== fsmap(e 30) ==== 8362+0+0 (secure 0 0 0) 0x7fa248011440 con 0x55f1523b54d0 2024-05-27T07:26:39.340 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:26:39.340+0000 7fa260fd0740 10 client.15579 fetch_fsmap finished waiting for FSMap version 30 2024-05-27T07:26:39.340 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:26:39.340+0000 7fa260fd0740 10 client.15579 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-05-27T07:26:39.340 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:26:39.340+0000 7fa260fd0740 20 client.15579 populate_metadata read hostname 'smithi052' 2024-05-27T07:26:39.340 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:26:39.340+0000 7fa260fd0740 1 --2- 172.21.15.52:0/923078439 >> [v2:172.21.15.105:6838/500348151,v1:172.21.15.105:6839/500348151] conn(0x55f15236a1c0 0x55f1527fdd10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:26:39.340 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:26:39.340+0000 7fa260fd0740 4 client.15579 mds_command: new command op to 24527 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-05-27T07:26:39.341 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:26:39.340+0000 7fa260fd0740 1 -- 172.21.15.52:0/923078439 --> [v2:172.21.15.105:6838/500348151,v1:172.21.15.105:6839/500348151] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55f1526e5360 con 0x55f15236a1c0 2024-05-27T07:26:39.341 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:26:39.340+0000 7fa2537fe640 1 --2- 172.21.15.52:0/923078439 >> [v2:172.21.15.105:6838/500348151,v1:172.21.15.105:6839/500348151] conn(0x55f15236a1c0 0x55f1527fdd10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:26:39.341 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:26:39.340+0000 7fa2537fe640 1 --2- 172.21.15.52:0/923078439 >> [v2:172.21.15.105:6838/500348151,v1:172.21.15.105:6839/500348151] conn(0x55f15236a1c0 0x55f1527fdd10 crc :-1 s=READY pgs=49 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=dc56d8ffa2292c81 server_cookie=ec611c28857ea1cf in_seq=0 out_seq=0 2024-05-27T07:26:39.341 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:26:39.340+0000 7fa232ffd640 10 client.15579 ms_handle_connect on v2:172.21.15.105:6838/500348151 2024-05-27T07:26:39.341 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:26:39.344+0000 7fa232ffd640 1 -- 172.21.15.52:0/923078439 <== mds.0 v2:172.21.15.105:6838/500348151 1 ==== command_reply(tid 0: 0 ) ==== 8+0+25 (crc 0 0 0) 0x55f1526e5360 con 0x55f15236a1c0 2024-05-27T07:26:39.341 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:26:39.344+0000 7fa232ffd640 10 client.15579 handle_command_reply: tid=0 2024-05-27T07:26:39.341 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:26:39.344+0000 7fa260fd0740 1 client.15579 shutdown 2024-05-27T07:26:39.341 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:26:39.344+0000 7fa260fd0740 1 -- 172.21.15.52:0/923078439 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55f1523b54d0 msgr2=0x55f152784cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:26:39.342 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:26:39.344+0000 7fa260fd0740 1 --2- 172.21.15.52:0/923078439 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55f1523b54d0 0x55f152784cd0 secure :-1 s=READY pgs=458 cs=0 l=1 rev1=1 crypto rx=0x7fa24800bb80 tx=0x7fa24800b8a0 comp rx=0 tx=0).stop 2024-05-27T07:26:39.342 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:26:39.344+0000 7fa260fd0740 1 -- 172.21.15.52:0/923078439 shutdown_connections 2024-05-27T07:26:39.342 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:26:39.344+0000 7fa260fd0740 1 --2- 172.21.15.52:0/923078439 >> [v2:172.21.15.105:6838/500348151,v1:172.21.15.105:6839/500348151] conn(0x55f15236a1c0 0x55f1527fdd10 crc :-1 s=READY pgs=49 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:26:39.342 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:26:39.344+0000 7fa260fd0740 1 --2- 172.21.15.52:0/923078439 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55f1523d61b0 0x55f152785210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:26:39.342 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:26:39.344+0000 7fa260fd0740 1 --2- 172.21.15.52:0/923078439 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55f1523b54d0 0x55f152784cd0 unknown :-1 s=CLOSED pgs=458 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:26:39.342 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:26:39.344+0000 7fa232ffd640 0 client.15579 ms_handle_reset on v2:172.21.15.105:6838/500348151 2024-05-27T07:26:39.342 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:26:39.344+0000 7fa260fd0740 1 --2- 172.21.15.52:0/923078439 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55f1523ad2f0 0x55f152784790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:26:39.343 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:26:39.344+0000 7fa260fd0740 1 -- 172.21.15.52:0/923078439 >> 172.21.15.52:0/923078439 conn(0x55f1523ca300 msgr2=0x55f1527797a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:26:39.343 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:26:39.344+0000 7fa260fd0740 1 -- 172.21.15.52:0/923078439 shutdown_connections 2024-05-27T07:26:39.343 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:26:39.344+0000 7fa260fd0740 1 -- 172.21.15.52:0/923078439 wait complete. 2024-05-27T07:26:39.343 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:26:39.345+0000 7fa260fd0740 20 client.15579 trim_cache size 0 max 16384 2024-05-27T07:26:39.343 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stdout:{ 2024-05-27T07:26:39.343 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stdout: "return_code": 0 2024-05-27T07:26:39.344 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stdout:} 2024-05-27T07:26:39.345 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:26:39.345+0000 7fa25a11f640 1 -- 172.21.15.52:0/4034355998 >> [v2:172.21.15.52:6800/1822318384,v1:172.21.15.52:6801/1822318384] conn(0x7fa23c0b2a70 msgr2=0x7fa23c0b4f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:26:39.345 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:26:39.345+0000 7fa25a11f640 1 --2- 172.21.15.52:0/4034355998 >> [v2:172.21.15.52:6800/1822318384,v1:172.21.15.52:6801/1822318384] conn(0x7fa23c0b2a70 0x7fa23c0b4f80 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x55f152774540 tx=0x7fa24c02f390 comp rx=0 tx=0).stop 2024-05-27T07:26:39.345 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:26:39.345+0000 7fa25a11f640 1 -- 172.21.15.52:0/4034355998 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fa2541043d0 msgr2=0x7fa2541a04e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:26:39.345 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:26:39.345+0000 7fa25a11f640 1 --2- 172.21.15.52:0/4034355998 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fa2541043d0 0x7fa2541a04e0 secure :-1 s=READY pgs=454 cs=0 l=1 rev1=1 crypto rx=0x7fa244005fd0 tx=0x7fa24400dde0 comp rx=0 tx=0).stop 2024-05-27T07:26:39.345 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:26:39.345+0000 7fa25a11f640 1 -- 172.21.15.52:0/4034355998 shutdown_connections 2024-05-27T07:26:39.345 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:26:39.345+0000 7fa25a11f640 1 --2- 172.21.15.52:0/4034355998 >> [v2:172.21.15.52:6800/1822318384,v1:172.21.15.52:6801/1822318384] conn(0x7fa23c0b2a70 0x7fa23c0b4f80 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:26:39.345 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:26:39.345+0000 7fa25a11f640 1 --2- 172.21.15.52:0/4034355998 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fa25410a120 0x7fa25419a6b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:26:39.345 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:26:39.345+0000 7fa25a11f640 1 --2- 172.21.15.52:0/4034355998 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fa254109c80 0x7fa2541a0a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:26:39.345 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:26:39.345+0000 7fa25a11f640 1 --2- 172.21.15.52:0/4034355998 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fa2541043d0 0x7fa2541a04e0 unknown :-1 s=CLOSED pgs=454 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:26:39.345 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:26:39.345+0000 7fa25a11f640 1 -- 172.21.15.52:0/4034355998 >> 172.21.15.52:0/4034355998 conn(0x7fa254079290 msgr2=0x7fa254102030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:26:39.346 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:26:39.345+0000 7fa25a11f640 1 -- 172.21.15.52:0/4034355998 shutdown_connections 2024-05-27T07:26:39.346 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:26:39.345+0000 7fa25a11f640 1 -- 172.21.15.52:0/4034355998 wait complete. 2024-05-27T07:26:40.352 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:26:40 smithi173 ceph-mon[37000]: pgmap v24: 129 pgs: 129 active+clean; 1.3 MiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 1.5 KiB/s wr, 17 op/s 2024-05-27T07:26:40.439 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:26:40 smithi105 ceph-mon[35977]: pgmap v24: 129 pgs: 129 active+clean; 1.3 MiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 1.5 KiB/s wr, 17 op/s 2024-05-27T07:26:40.479 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:26:40 smithi052 ceph-mon[31955]: pgmap v24: 129 pgs: 129 active+clean; 1.3 MiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 1.5 KiB/s wr, 17 op/s 2024-05-27T07:26:42.351 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:26:42 smithi173 ceph-mon[37000]: pgmap v25: 129 pgs: 129 active+clean; 1.3 MiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-05-27T07:26:42.439 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:26:42 smithi105 ceph-mon[35977]: pgmap v25: 129 pgs: 129 active+clean; 1.3 MiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-05-27T07:26:42.479 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:26:42 smithi052 ceph-mon[31955]: pgmap v25: 129 pgs: 129 active+clean; 1.3 MiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-05-27T07:26:44.352 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:26:44 smithi173 ceph-mon[37000]: pgmap v26: 129 pgs: 129 active+clean; 1.3 MiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 255 B/s wr, 12 op/s 2024-05-27T07:26:44.439 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:26:44 smithi105 ceph-mon[35977]: pgmap v26: 129 pgs: 129 active+clean; 1.3 MiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 255 B/s wr, 12 op/s 2024-05-27T07:26:44.479 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:26:44 smithi052 ceph-mon[31955]: pgmap v26: 129 pgs: 129 active+clean; 1.3 MiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 255 B/s wr, 12 op/s 2024-05-27T07:26:46.058 DEBUG:teuthology.orchestra.run.smithi052:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:38f9d900c83c63799fbdbe61acc9a11b0d3554a6 shell --fsid 73df2636-1bf8-11ef-bc9b-c7b262605968 -- ceph tell mds.1:0 scrub status 2024-05-27T07:26:46.325 INFO:teuthology.orchestra.run.smithi052.stderr:Inferring config /var/lib/ceph/73df2636-1bf8-11ef-bc9b-c7b262605968/mon.a/config 2024-05-27T07:26:46.351 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:26:46 smithi173 ceph-mon[37000]: pgmap v27: 129 pgs: 129 active+clean; 1.3 MiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 255 B/s wr, 9 op/s 2024-05-27T07:26:46.439 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:26:46 smithi105 ceph-mon[35977]: pgmap v27: 129 pgs: 129 active+clean; 1.3 MiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 255 B/s wr, 9 op/s 2024-05-27T07:26:46.448 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:26:46 smithi052 ceph-mon[31955]: pgmap v27: 129 pgs: 129 active+clean; 1.3 MiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 255 B/s wr, 9 op/s 2024-05-27T07:26:48.279 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:26:48 smithi052 ceph-mon[31955]: pgmap v28: 129 pgs: 129 active+clean; 1.3 MiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 255 B/s wr, 8 op/s 2024-05-27T07:26:48.308 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:48.318+0000 7fdf53742700 1 -- 172.21.15.52:0/2179089325 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fdf4c107de0 msgr2=0x7fdf4c1081f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:26:48.308 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:48.318+0000 7fdf53742700 1 --2- 172.21.15.52:0/2179089325 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fdf4c107de0 0x7fdf4c1081f0 secure :-1 s=READY pgs=459 cs=0 l=1 rev1=1 crypto rx=0x7fdf3c0099c0 tx=0x7fdf3c009cd0 comp rx=0 tx=0).stop 2024-05-27T07:26:48.309 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:48.318+0000 7fdf53742700 1 -- 172.21.15.52:0/2179089325 shutdown_connections 2024-05-27T07:26:48.309 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:48.318+0000 7fdf53742700 1 --2- 172.21.15.52:0/2179089325 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fdf4c108730 0x7fdf4c110090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:26:48.309 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:48.318+0000 7fdf53742700 1 --2- 172.21.15.52:0/2179089325 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fdf4c1034b0 0x7fdf4c1038c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:26:48.309 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:48.318+0000 7fdf53742700 1 --2- 172.21.15.52:0/2179089325 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fdf4c107de0 0x7fdf4c1081f0 unknown :-1 s=CLOSED pgs=459 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:26:48.309 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:48.318+0000 7fdf53742700 1 -- 172.21.15.52:0/2179089325 >> 172.21.15.52:0/2179089325 conn(0x7fdf4c0fedb0 msgr2=0x7fdf4c1011d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:26:48.309 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:48.319+0000 7fdf53742700 1 -- 172.21.15.52:0/2179089325 shutdown_connections 2024-05-27T07:26:48.309 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:48.319+0000 7fdf53742700 1 -- 172.21.15.52:0/2179089325 wait complete. 2024-05-27T07:26:48.311 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:48.319+0000 7fdf53742700 1 Processor -- start 2024-05-27T07:26:48.311 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:48.319+0000 7fdf53742700 1 -- start start 2024-05-27T07:26:48.311 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:48.320+0000 7fdf53742700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fdf4c1034b0 0x7fdf4c19ae00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:26:48.311 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:48.320+0000 7fdf53742700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fdf4c107de0 0x7fdf4c19b3a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:26:48.311 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:48.320+0000 7fdf53742700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fdf4c108730 0x7fdf4c19f630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:26:48.311 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:48.320+0000 7fdf53742700 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x7fdf4c113340 con 0x7fdf4c107de0 2024-05-27T07:26:48.312 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:48.320+0000 7fdf53742700 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7fdf4c1131c0 con 0x7fdf4c108730 2024-05-27T07:26:48.312 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:48.320+0000 7fdf53742700 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7fdf4c1134c0 con 0x7fdf4c1034b0 2024-05-27T07:26:48.312 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:48.320+0000 7fdf50cdd700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fdf4c107de0 0x7fdf4c19b3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:26:48.312 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:48.320+0000 7fdf50cdd700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fdf4c107de0 0x7fdf4c19b3a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.52:52794/0 (socket says 172.21.15.52:52794) 2024-05-27T07:26:48.312 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:48.320+0000 7fdf50cdd700 1 -- 172.21.15.52:0/518760136 learned_addr learned my addr 172.21.15.52:0/518760136 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-05-27T07:26:48.312 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:48.320+0000 7fdf514de700 1 --2- 172.21.15.52:0/518760136 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fdf4c1034b0 0x7fdf4c19ae00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:26:48.312 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:48.320+0000 7fdf51cdf700 1 --2- 172.21.15.52:0/518760136 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fdf4c108730 0x7fdf4c19f630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:26:48.312 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:48.320+0000 7fdf50cdd700 1 -- 172.21.15.52:0/518760136 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fdf4c1034b0 msgr2=0x7fdf4c19ae00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:26:48.312 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:48.320+0000 7fdf50cdd700 1 --2- 172.21.15.52:0/518760136 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fdf4c1034b0 0x7fdf4c19ae00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:26:48.312 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:48.320+0000 7fdf50cdd700 1 -- 172.21.15.52:0/518760136 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fdf4c108730 msgr2=0x7fdf4c19f630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:26:48.312 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:48.320+0000 7fdf50cdd700 1 --2- 172.21.15.52:0/518760136 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fdf4c108730 0x7fdf4c19f630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:26:48.312 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:48.320+0000 7fdf50cdd700 1 -- 172.21.15.52:0/518760136 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdf4c19fc20 con 0x7fdf4c107de0 2024-05-27T07:26:48.313 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:48.320+0000 7fdf50cdd700 1 --2- 172.21.15.52:0/518760136 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fdf4c107de0 0x7fdf4c19b3a0 secure :-1 s=READY pgs=460 cs=0 l=1 rev1=1 crypto rx=0x7fdf3c006010 tx=0x7fdf3c0047b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:26:48.313 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:48.321+0000 7fdf427fc700 1 -- 172.21.15.52:0/518760136 <== mon.0 v2:172.21.15.52:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdf3c027070 con 0x7fdf4c107de0 2024-05-27T07:26:48.313 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:48.321+0000 7fdf427fc700 1 -- 172.21.15.52:0/518760136 <== mon.0 v2:172.21.15.52:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fdf3c003f20 con 0x7fdf4c107de0 2024-05-27T07:26:48.313 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:48.321+0000 7fdf427fc700 1 -- 172.21.15.52:0/518760136 <== mon.0 v2:172.21.15.52:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdf3c022cd0 con 0x7fdf4c107de0 2024-05-27T07:26:48.313 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:48.321+0000 7fdf53742700 1 -- 172.21.15.52:0/518760136 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fdf4c19feb0 con 0x7fdf4c107de0 2024-05-27T07:26:48.313 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:48.321+0000 7fdf53742700 1 -- 172.21.15.52:0/518760136 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fdf4c1a0340 con 0x7fdf4c107de0 2024-05-27T07:26:48.313 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:48.324+0000 7fdf427fc700 1 -- 172.21.15.52:0/518760136 <== mon.0 v2:172.21.15.52:3300/0 4 ==== mgrmap(e 24) ==== 149879+0+0 (secure 0 0 0) 0x7fdf3c02a070 con 0x7fdf4c107de0 2024-05-27T07:26:48.316 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:48.325+0000 7fdf427fc700 1 --2- 172.21.15.52:0/518760136 >> [v2:172.21.15.52:6800/1822318384,v1:172.21.15.52:6801/1822318384] conn(0x7fdf380b2b90 0x7fdf380b50a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:26:48.316 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:48.325+0000 7fdf62157b80 1 Processor -- start 2024-05-27T07:26:48.316 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:48.325+0000 7fdf62157b80 1 -- start start 2024-05-27T07:26:48.316 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:48.325+0000 7fdf427fc700 1 -- 172.21.15.52:0/518760136 <== mon.0 v2:172.21.15.52:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9379+0+0 (secure 0 0 0) 0x7fdf3c0e3a70 con 0x7fdf4c107de0 2024-05-27T07:26:48.316 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:48.325+0000 7fdf514de700 1 --2- 172.21.15.52:0/518760136 >> [v2:172.21.15.52:6800/1822318384,v1:172.21.15.52:6801/1822318384] conn(0x7fdf380b2b90 0x7fdf380b50a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:26:48.317 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:48.325+0000 7fdf62157b80 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x562187dfbe40 0x562187f80b30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:26:48.317 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:48.325+0000 7fdf62157b80 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x562187f81070 0x562187f7abe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:26:48.317 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:48.325+0000 7fdf62157b80 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x562187f7b120 0x562187f7b530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:26:48.317 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:48.325+0000 7fdf62157b80 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x562187ea8d70 con 0x562187dfbe40 2024-05-27T07:26:48.317 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:48.325+0000 7fdf62157b80 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x562187e98910 con 0x562187f81070 2024-05-27T07:26:48.317 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:48.325+0000 7fdf62157b80 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x562187e9c950 con 0x562187f7b120 2024-05-27T07:26:48.317 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:48.325+0000 7fdf514de700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x562187dfbe40 0x562187f80b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:26:48.317 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:48.325+0000 7fdf514de700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x562187dfbe40 0x562187f80b30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.52:52810/0 (socket says 172.21.15.52:52810) 2024-05-27T07:26:48.317 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:48.325+0000 7fdf50cdd700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x562187f81070 0x562187f7abe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:26:48.317 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:48.325+0000 7fdf514de700 1 -- 172.21.15.52:0/603390565 learned_addr learned my addr 172.21.15.52:0/603390565 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-05-27T07:26:48.317 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:48.326+0000 7fdf51cdf700 1 --2- 172.21.15.52:0/603390565 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x562187f7b120 0x562187f7b530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:26:48.317 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:48.326+0000 7fdf514de700 1 --2- 172.21.15.52:0/518760136 >> [v2:172.21.15.52:6800/1822318384,v1:172.21.15.52:6801/1822318384] conn(0x7fdf380b2b90 0x7fdf380b50a0 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7fdf4800aca0 tx=0x7fdf4800f6a0 comp rx=0 tx=0).ready entity=mgr.15333 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:26:48.318 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:48.326+0000 7fdf514de700 1 -- 172.21.15.52:0/603390565 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x562187f7b120 msgr2=0x562187f7b530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:26:48.318 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:48.326+0000 7fdf514de700 1 --2- 172.21.15.52:0/603390565 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x562187f7b120 0x562187f7b530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:26:48.318 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:48.326+0000 7fdf514de700 1 -- 172.21.15.52:0/603390565 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x562187f81070 msgr2=0x562187f7abe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:26:48.318 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:48.326+0000 7fdf514de700 1 --2- 172.21.15.52:0/603390565 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x562187f81070 0x562187f7abe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:26:48.318 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:48.326+0000 7fdf514de700 1 -- 172.21.15.52:0/603390565 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x562187e41fb0 con 0x562187dfbe40 2024-05-27T07:26:48.318 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:48.326+0000 7fdf50cdd700 1 --2- 172.21.15.52:0/603390565 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x562187f81070 0x562187f7abe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-27T07:26:48.318 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:48.326+0000 7fdf514de700 1 --2- 172.21.15.52:0/603390565 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x562187dfbe40 0x562187f80b30 secure :-1 s=READY pgs=461 cs=0 l=1 rev1=1 crypto rx=0x7fdf4800fef0 tx=0x7fdf4801a000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c5a9753259f7d1b1 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:26:48.318 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:48.326+0000 7fdf377fe700 1 -- 172.21.15.52:0/603390565 <== mon.0 v2:172.21.15.52:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdf48018070 con 0x562187dfbe40 2024-05-27T07:26:48.318 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:48.326+0000 7fdf377fe700 1 -- 172.21.15.52:0/603390565 <== mon.0 v2:172.21.15.52:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fdf480044d0 con 0x562187dfbe40 2024-05-27T07:26:48.318 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:48.326+0000 7fdf377fe700 1 -- 172.21.15.52:0/603390565 <== mon.0 v2:172.21.15.52:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdf48003dd0 con 0x562187dfbe40 2024-05-27T07:26:48.322 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:48.327+0000 7fdf62157b80 1 -- 172.21.15.52:0/603390565 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x562187dfbe40 msgr2=0x562187f80b30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:26:48.322 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:48.327+0000 7fdf62157b80 1 --2- 172.21.15.52:0/603390565 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x562187dfbe40 0x562187f80b30 secure :-1 s=READY pgs=461 cs=0 l=1 rev1=1 crypto rx=0x7fdf4800fef0 tx=0x7fdf4801a000 comp rx=0 tx=0).stop 2024-05-27T07:26:48.322 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:48.327+0000 7fdf62157b80 1 -- 172.21.15.52:0/603390565 shutdown_connections 2024-05-27T07:26:48.322 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:48.327+0000 7fdf62157b80 1 --2- 172.21.15.52:0/603390565 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x562187f7b120 0x562187f7b530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:26:48.322 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:48.327+0000 7fdf62157b80 1 --2- 172.21.15.52:0/603390565 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x562187dfbe40 0x562187f80b30 unknown :-1 s=CLOSED pgs=461 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:26:48.322 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:48.327+0000 7fdf62157b80 1 --2- 172.21.15.52:0/603390565 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x562187f81070 0x562187f7abe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:26:48.322 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:48.327+0000 7fdf62157b80 1 -- 172.21.15.52:0/603390565 >> 172.21.15.52:0/603390565 conn(0x562187dfe770 msgr2=0x562187f75d90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:26:48.323 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:48.327+0000 7fdf62157b80 1 -- 172.21.15.52:0/603390565 shutdown_connections 2024-05-27T07:26:48.323 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:48.327+0000 7fdf62157b80 1 -- 172.21.15.52:0/603390565 wait complete. 2024-05-27T07:26:48.323 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:48.328+0000 7fdf62157b80 1 -- start start 2024-05-27T07:26:48.323 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:48.328+0000 7fdf62157b80 1 Processor -- start 2024-05-27T07:26:48.323 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:48.328+0000 7fdf62157b80 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x562187dfbe40 0x562187f848a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:26:48.323 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:48.328+0000 7fdf62157b80 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x562187f7b120 0x562187f84de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:26:48.323 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:48.328+0000 7fdf62157b80 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x562187f81070 0x562187f85320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:26:48.323 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:48.328+0000 7fdf62157b80 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x562187d9efa0 con 0x562187dfbe40 2024-05-27T07:26:48.323 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:48.328+0000 7fdf62157b80 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x562187e00da0 con 0x562187f7b120 2024-05-27T07:26:48.323 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:48.328+0000 7fdf62157b80 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x562187f87b60 con 0x562187f81070 2024-05-27T07:26:48.323 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:48.328+0000 7fdf62157b80 10 client.0 _ll_register_callbacks cb 0x562187e2f5b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T07:26:48.323 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:48.328+0000 7fdf514de700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x562187dfbe40 0x562187f848a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:26:48.323 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:48.328+0000 7fdf514de700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x562187dfbe40 0x562187f848a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.52:52826/0 (socket says 172.21.15.52:52826) 2024-05-27T07:26:48.324 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:48.328+0000 7fdf514de700 1 -- 172.21.15.52:0/1546649763 learned_addr learned my addr 172.21.15.52:0/1546649763 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-05-27T07:26:48.324 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:48.329+0000 7fdf50cdd700 1 --2- 172.21.15.52:0/1546649763 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x562187f7b120 0x562187f84de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:26:48.324 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:48.329+0000 7fdf51cdf700 1 --2- 172.21.15.52:0/1546649763 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x562187f81070 0x562187f85320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:26:48.324 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:48.329+0000 7fdf514de700 1 -- 172.21.15.52:0/1546649763 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x562187f81070 msgr2=0x562187f85320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:26:48.324 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:48.329+0000 7fdf514de700 1 --2- 172.21.15.52:0/1546649763 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x562187f81070 0x562187f85320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:26:48.324 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:48.329+0000 7fdf514de700 1 -- 172.21.15.52:0/1546649763 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x562187f7b120 msgr2=0x562187f84de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:26:48.324 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:48.329+0000 7fdf514de700 1 --2- 172.21.15.52:0/1546649763 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x562187f7b120 0x562187f84de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:26:48.324 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:48.329+0000 7fdf514de700 1 -- 172.21.15.52:0/1546649763 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({osdmap=0}) -- 0x562187de88d0 con 0x562187dfbe40 2024-05-27T07:26:48.324 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:48.329+0000 7fdf51cdf700 1 --2- 172.21.15.52:0/1546649763 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x562187f81070 0x562187f85320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-27T07:26:48.324 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:48.329+0000 7fdf514de700 1 --2- 172.21.15.52:0/1546649763 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x562187dfbe40 0x562187f848a0 secure :-1 s=READY pgs=462 cs=0 l=1 rev1=1 crypto rx=0x7fdf480046a0 tx=0x7fdf48012f20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ada3c6e880233802 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:26:48.324 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:48.329+0000 7fdf34ff9700 10 client.0 ms_handle_connect on v2:172.21.15.52:3300/0 2024-05-27T07:26:48.324 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:48.329+0000 7fdf34ff9700 1 -- 172.21.15.52:0/1546649763 <== mon.0 v2:172.21.15.52:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdf48018030 con 0x562187dfbe40 2024-05-27T07:26:48.324 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:48.329+0000 7fdf34ff9700 1 -- 172.21.15.52:0/1546649763 <== mon.0 v2:172.21.15.52:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 9379+0+0 (secure 0 0 0) 0x7fdf48005460 con 0x562187dfbe40 2024-05-27T07:26:48.325 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:48.329+0000 7fdf62157b80 1 -- 172.21.15.52:0/1546649763 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fdf480044d0 con 0x562187dfbe40 2024-05-27T07:26:48.325 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:48.329+0000 7fdf34ff9700 1 -- 172.21.15.52:0/1546649763 <== mon.0 v2:172.21.15.52:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7fdf48005660 con 0x562187dfbe40 2024-05-27T07:26:48.325 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:48.329+0000 7fdf62157b80 10 client.15591 fetch_fsmap learned FSMap version 30 2024-05-27T07:26:48.325 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:48.329+0000 7fdf62157b80 1 -- 172.21.15.52:0/1546649763 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x562187e52760 con 0x562187dfbe40 2024-05-27T07:26:48.325 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:48.330+0000 7fdf34ff9700 1 -- 172.21.15.52:0/1546649763 <== mon.0 v2:172.21.15.52:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fdf480044d0 con 0x562187dfbe40 2024-05-27T07:26:48.325 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:48.330+0000 7fdf34ff9700 1 -- 172.21.15.52:0/1546649763 <== mon.0 v2:172.21.15.52:3300/0 5 ==== fsmap(e 30) ==== 8362+0+0 (secure 0 0 0) 0x7fdf48003ef0 con 0x562187dfbe40 2024-05-27T07:26:48.325 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:48.330+0000 7fdf62157b80 10 client.15591 fetch_fsmap finished waiting for FSMap version 30 2024-05-27T07:26:48.325 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:48.330+0000 7fdf62157b80 10 client.15591 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-05-27T07:26:48.325 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:48.330+0000 7fdf62157b80 20 client.15591 populate_metadata read hostname 'smithi052' 2024-05-27T07:26:48.325 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:48.330+0000 7fdf62157b80 1 --2- 172.21.15.52:0/1546649763 >> [v2:172.21.15.105:6838/500348151,v1:172.21.15.105:6839/500348151] conn(0x562187f98ea0 0x562187f9b2c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:26:48.325 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:48.330+0000 7fdf62157b80 4 client.15591 mds_command: new command op to 24527 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-27T07:26:48.325 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:48.330+0000 7fdf62157b80 1 -- 172.21.15.52:0/1546649763 --> [v2:172.21.15.105:6838/500348151,v1:172.21.15.105:6839/500348151] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x562187eaacc0 con 0x562187f98ea0 2024-05-27T07:26:48.325 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:48.330+0000 7fdf50cdd700 1 --2- 172.21.15.52:0/1546649763 >> [v2:172.21.15.105:6838/500348151,v1:172.21.15.105:6839/500348151] conn(0x562187f98ea0 0x562187f9b2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:26:48.326 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:48.331+0000 7fdf34ff9700 10 client.15591 ms_handle_connect on v2:172.21.15.105:6838/500348151 2024-05-27T07:26:48.326 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:48.331+0000 7fdf50cdd700 1 --2- 172.21.15.52:0/1546649763 >> [v2:172.21.15.105:6838/500348151,v1:172.21.15.105:6839/500348151] conn(0x562187f98ea0 0x562187f9b2c0 crc :-1 s=READY pgs=50 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=3d06c97b1d113219 server_cookie=74bdd227e16af78e in_seq=0 out_seq=0 2024-05-27T07:26:48.326 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:48.331+0000 7fdf34ff9700 1 -- 172.21.15.52:0/1546649763 <== mds.0 v2:172.21.15.105:6838/500348151 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24041 (crc 0 0 0) 0x562187eaacc0 con 0x562187f98ea0 2024-05-27T07:26:48.326 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:48.331+0000 7fdf34ff9700 10 client.15591 handle_command_reply: tid=0 2024-05-27T07:26:48.326 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:48.331+0000 7fdf62157b80 1 client.15591 shutdown 2024-05-27T07:26:48.326 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:48.332+0000 7fdf62157b80 1 -- 172.21.15.52:0/1546649763 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x562187dfbe40 msgr2=0x562187f848a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:26:48.326 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:48.332+0000 7fdf62157b80 1 --2- 172.21.15.52:0/1546649763 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x562187dfbe40 0x562187f848a0 secure :-1 s=READY pgs=462 cs=0 l=1 rev1=1 crypto rx=0x7fdf480046a0 tx=0x7fdf48012f20 comp rx=0 tx=0).stop 2024-05-27T07:26:48.326 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:48.332+0000 7fdf62157b80 1 -- 172.21.15.52:0/1546649763 shutdown_connections 2024-05-27T07:26:48.326 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:48.332+0000 7fdf62157b80 1 --2- 172.21.15.52:0/1546649763 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x562187f81070 0x562187f85320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:26:48.326 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:48.332+0000 7fdf62157b80 1 --2- 172.21.15.52:0/1546649763 >> [v2:172.21.15.105:6838/500348151,v1:172.21.15.105:6839/500348151] conn(0x562187f98ea0 0x562187f9b2c0 crc :-1 s=READY pgs=50 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:26:48.326 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:48.332+0000 7fdf62157b80 1 --2- 172.21.15.52:0/1546649763 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x562187dfbe40 0x562187f848a0 unknown :-1 s=CLOSED pgs=462 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:26:48.326 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:48.332+0000 7fdf62157b80 1 --2- 172.21.15.52:0/1546649763 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x562187f7b120 0x562187f84de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:26:48.327 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:48.332+0000 7fdf62157b80 1 -- 172.21.15.52:0/1546649763 >> 172.21.15.52:0/1546649763 conn(0x562187dfe770 msgr2=0x562187f79bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:26:48.327 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:48.332+0000 7fdf34ff9700 0 client.15591 ms_handle_reset on v2:172.21.15.105:6838/500348151 2024-05-27T07:26:48.327 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:48.332+0000 7fdf62157b80 1 -- 172.21.15.52:0/1546649763 shutdown_connections 2024-05-27T07:26:48.327 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:48.332+0000 7fdf62157b80 1 -- 172.21.15.52:0/1546649763 wait complete. 2024-05-27T07:26:48.327 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:48.332+0000 7fdf62157b80 20 client.15591 trim_cache size 0 max 16384 2024-05-27T07:26:48.342 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:48.352+0000 7fdf62157b80 1 Processor -- start 2024-05-27T07:26:48.342 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:48.352+0000 7fdf62157b80 1 -- start start 2024-05-27T07:26:48.343 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:48.353+0000 7fdf62157b80 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x562187dfbe40 0x562187f952f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:26:48.344 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:48.353+0000 7fdf62157b80 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x562187f7b120 0x562187f7e490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:26:48.344 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:48.353+0000 7fdf62157b80 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x562187f81070 0x562187f7e9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:26:48.344 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:48.353+0000 7fdf62157b80 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x7fdf4c1ae040 con 0x562187f7b120 2024-05-27T07:26:48.344 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:48.353+0000 7fdf62157b80 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7fdf4c1afb80 con 0x562187dfbe40 2024-05-27T07:26:48.344 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:48.353+0000 7fdf62157b80 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7fdf4c1ae710 con 0x562187f81070 2024-05-27T07:26:48.344 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:48.353+0000 7fdf50cdd700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x562187f7b120 0x562187f7e490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:26:48.344 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:48.353+0000 7fdf50cdd700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x562187f7b120 0x562187f7e490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.52:52840/0 (socket says 172.21.15.52:52840) 2024-05-27T07:26:48.344 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:48.353+0000 7fdf50cdd700 1 -- 172.21.15.52:0/2751125921 learned_addr learned my addr 172.21.15.52:0/2751125921 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-05-27T07:26:48.344 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:48.353+0000 7fdf514de700 1 --2- 172.21.15.52:0/2751125921 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x562187dfbe40 0x562187f952f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:26:48.344 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:48.353+0000 7fdf51cdf700 1 --2- 172.21.15.52:0/2751125921 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x562187f81070 0x562187f7e9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:26:48.344 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:48.353+0000 7fdf50cdd700 1 -- 172.21.15.52:0/2751125921 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x562187f81070 msgr2=0x562187f7e9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:26:48.345 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:48.353+0000 7fdf50cdd700 1 --2- 172.21.15.52:0/2751125921 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x562187f81070 0x562187f7e9d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:26:48.345 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:48.353+0000 7fdf50cdd700 1 -- 172.21.15.52:0/2751125921 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x562187dfbe40 msgr2=0x562187f952f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:26:48.345 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:48.353+0000 7fdf50cdd700 1 --2- 172.21.15.52:0/2751125921 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x562187dfbe40 0x562187f952f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:26:48.345 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:48.353+0000 7fdf50cdd700 1 -- 172.21.15.52:0/2751125921 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x562187f82e30 con 0x562187f7b120 2024-05-27T07:26:48.345 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:48.353+0000 7fdf514de700 1 --2- 172.21.15.52:0/2751125921 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x562187dfbe40 0x562187f952f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-27T07:26:48.345 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:48.353+0000 7fdf50cdd700 1 --2- 172.21.15.52:0/2751125921 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x562187f7b120 0x562187f7e490 secure :-1 s=READY pgs=463 cs=0 l=1 rev1=1 crypto rx=0x7fdf3c0e1460 tx=0x7fdf3c0cd040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=829d8c4ae056fe8e server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:26:48.345 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:48.354+0000 7fdf36ffd700 1 -- 172.21.15.52:0/2751125921 <== mon.0 v2:172.21.15.52:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdf3c0aa830 con 0x562187f7b120 2024-05-27T07:26:48.345 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:48.354+0000 7fdf36ffd700 1 -- 172.21.15.52:0/2751125921 <== mon.0 v2:172.21.15.52:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fdf3c0aebe0 con 0x562187f7b120 2024-05-27T07:26:48.345 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:48.354+0000 7fdf36ffd700 1 -- 172.21.15.52:0/2751125921 <== mon.0 v2:172.21.15.52:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdf3c0aacf0 con 0x562187f7b120 2024-05-27T07:26:48.349 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:48.354+0000 7fdf62157b80 1 -- 172.21.15.52:0/2751125921 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x562187f7b120 msgr2=0x562187f7e490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:26:48.349 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:48.354+0000 7fdf62157b80 1 --2- 172.21.15.52:0/2751125921 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x562187f7b120 0x562187f7e490 secure :-1 s=READY pgs=463 cs=0 l=1 rev1=1 crypto rx=0x7fdf3c0e1460 tx=0x7fdf3c0cd040 comp rx=0 tx=0).stop 2024-05-27T07:26:48.349 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:48.354+0000 7fdf62157b80 1 -- 172.21.15.52:0/2751125921 shutdown_connections 2024-05-27T07:26:48.349 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:48.354+0000 7fdf62157b80 1 --2- 172.21.15.52:0/2751125921 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x562187f81070 0x562187f7e9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:26:48.349 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:48.354+0000 7fdf62157b80 1 --2- 172.21.15.52:0/2751125921 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x562187dfbe40 0x562187f952f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:26:48.349 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:48.354+0000 7fdf62157b80 1 --2- 172.21.15.52:0/2751125921 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x562187f7b120 0x562187f7e490 unknown :-1 s=CLOSED pgs=463 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:26:48.349 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:48.354+0000 7fdf62157b80 1 -- 172.21.15.52:0/2751125921 >> 172.21.15.52:0/2751125921 conn(0x562187dfe770 msgr2=0x562187f7f800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:26:48.349 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:48.355+0000 7fdf62157b80 1 -- 172.21.15.52:0/2751125921 shutdown_connections 2024-05-27T07:26:48.349 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:48.355+0000 7fdf62157b80 1 -- 172.21.15.52:0/2751125921 wait complete. 2024-05-27T07:26:48.349 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:48.355+0000 7fdf62157b80 1 -- start start 2024-05-27T07:26:48.349 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:48.355+0000 7fdf62157b80 1 Processor -- start 2024-05-27T07:26:48.350 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:48.355+0000 7fdf62157b80 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x562187dfbe40 0x562187f9eac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:26:48.350 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:48.355+0000 7fdf62157b80 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x562187f7b120 0x562187f9c8a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:26:48.350 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:48.355+0000 7fdf62157b80 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x562187f81070 0x562187f9cde0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:26:48.350 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:48.356+0000 7fdf62157b80 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x7fdf4c1af4b0 con 0x562187f7b120 2024-05-27T07:26:48.350 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:48.356+0000 7fdf62157b80 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7fdf4c1aede0 con 0x562187f81070 2024-05-27T07:26:48.350 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:48.356+0000 7fdf62157b80 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7fdf4c1abfe0 con 0x562187dfbe40 2024-05-27T07:26:48.350 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:48.356+0000 7fdf62157b80 10 client.0 _ll_register_callbacks cb 0x562187e2f5b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T07:26:48.350 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:48.356+0000 7fdf50cdd700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x562187f7b120 0x562187f9c8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:26:48.350 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:48.356+0000 7fdf50cdd700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x562187f7b120 0x562187f9c8a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.52:52850/0 (socket says 172.21.15.52:52850) 2024-05-27T07:26:48.350 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:48.356+0000 7fdf50cdd700 1 -- 172.21.15.52:0/808543346 learned_addr learned my addr 172.21.15.52:0/808543346 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-05-27T07:26:48.350 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:48.356+0000 7fdf514de700 1 --2- 172.21.15.52:0/808543346 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x562187dfbe40 0x562187f9eac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:26:48.350 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:48.356+0000 7fdf51cdf700 1 --2- 172.21.15.52:0/808543346 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x562187f81070 0x562187f9cde0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:26:48.351 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:48.356+0000 7fdf50cdd700 1 -- 172.21.15.52:0/808543346 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x562187dfbe40 msgr2=0x562187f9eac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:26:48.351 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:48.356+0000 7fdf50cdd700 1 --2- 172.21.15.52:0/808543346 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x562187dfbe40 0x562187f9eac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:26:48.351 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:48.356+0000 7fdf50cdd700 1 -- 172.21.15.52:0/808543346 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x562187f81070 msgr2=0x562187f9cde0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:26:48.351 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:48.356+0000 7fdf50cdd700 1 --2- 172.21.15.52:0/808543346 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x562187f81070 0x562187f9cde0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:26:48.351 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:48.356+0000 7fdf50cdd700 1 -- 172.21.15.52:0/808543346 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({osdmap=0}) -- 0x562187f9d520 con 0x562187f7b120 2024-05-27T07:26:48.351 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:48.356+0000 7fdf514de700 1 --2- 172.21.15.52:0/808543346 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x562187dfbe40 0x562187f9eac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-27T07:26:48.351 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:48.356+0000 7fdf50cdd700 1 --2- 172.21.15.52:0/808543346 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x562187f7b120 0x562187f9c8a0 secure :-1 s=READY pgs=464 cs=0 l=1 rev1=1 crypto rx=0x7fdf3c0a6860 tx=0x7fdf3c023f00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=22fd7e4354e36f8e server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:26:48.351 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:48.356+0000 7fdf34ff9700 10 client.0 ms_handle_connect on v2:172.21.15.52:3300/0 2024-05-27T07:26:48.351 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:48.356+0000 7fdf34ff9700 1 -- 172.21.15.52:0/808543346 <== mon.0 v2:172.21.15.52:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdf3c0aa830 con 0x562187f7b120 2024-05-27T07:26:48.351 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:48.356+0000 7fdf34ff9700 1 -- 172.21.15.52:0/808543346 <== mon.0 v2:172.21.15.52:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 9379+0+0 (secure 0 0 0) 0x7fdf3c0aaa90 con 0x562187f7b120 2024-05-27T07:26:48.351 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:48.356+0000 7fdf62157b80 1 -- 172.21.15.52:0/808543346 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fdf3c0aebe0 con 0x562187f7b120 2024-05-27T07:26:48.351 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:48.357+0000 7fdf34ff9700 1 -- 172.21.15.52:0/808543346 <== mon.0 v2:172.21.15.52:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7fdf3c0aac90 con 0x562187f7b120 2024-05-27T07:26:48.351 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:48.357+0000 7fdf62157b80 10 client.15597 fetch_fsmap learned FSMap version 30 2024-05-27T07:26:48.352 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:48.357+0000 7fdf62157b80 1 -- 172.21.15.52:0/808543346 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x5621880035f0 con 0x562187f7b120 2024-05-27T07:26:48.352 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:48.357+0000 7fdf34ff9700 1 -- 172.21.15.52:0/808543346 <== mon.0 v2:172.21.15.52:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fdf3c0aebe0 con 0x562187f7b120 2024-05-27T07:26:48.352 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:48.357+0000 7fdf34ff9700 1 -- 172.21.15.52:0/808543346 <== mon.0 v2:172.21.15.52:3300/0 5 ==== fsmap(e 30) ==== 8362+0+0 (secure 0 0 0) 0x7fdf3c0abb60 con 0x562187f7b120 2024-05-27T07:26:48.352 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:48.357+0000 7fdf62157b80 10 client.15597 fetch_fsmap finished waiting for FSMap version 30 2024-05-27T07:26:48.352 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:48.357+0000 7fdf62157b80 10 client.15597 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-05-27T07:26:48.352 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:48.357+0000 7fdf62157b80 20 client.15597 populate_metadata read hostname 'smithi052' 2024-05-27T07:26:48.352 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:48.357+0000 7fdf62157b80 1 --2- 172.21.15.52:0/808543346 >> [v2:172.21.15.105:6838/500348151,v1:172.21.15.105:6839/500348151] conn(0x562188003b30 0x562188005f50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:26:48.352 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:48.357+0000 7fdf62157b80 4 client.15597 mds_command: new command op to 24527 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-05-27T07:26:48.352 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:48.357+0000 7fdf62157b80 1 -- 172.21.15.52:0/808543346 --> [v2:172.21.15.105:6838/500348151,v1:172.21.15.105:6839/500348151] -- command(tid 0: {"prefix": "scrub status"}) -- 0x562187ea44a0 con 0x562188003b30 2024-05-27T07:26:48.352 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:48.357+0000 7fdf514de700 1 --2- 172.21.15.52:0/808543346 >> [v2:172.21.15.105:6838/500348151,v1:172.21.15.105:6839/500348151] conn(0x562188003b30 0x562188005f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:26:48.352 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:48.358+0000 7fdf34ff9700 10 client.15597 ms_handle_connect on v2:172.21.15.105:6838/500348151 2024-05-27T07:26:48.352 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:48.358+0000 7fdf514de700 1 --2- 172.21.15.52:0/808543346 >> [v2:172.21.15.105:6838/500348151,v1:172.21.15.105:6839/500348151] conn(0x562188003b30 0x562188005f50 crc :-1 s=READY pgs=51 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=5050be090896e295 server_cookie=42b316ac6c5fdcbb in_seq=0 out_seq=0 2024-05-27T07:26:48.352 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:48.358+0000 7fdf34ff9700 1 -- 172.21.15.52:0/808543346 <== mds.0 v2:172.21.15.105:6838/500348151 1 ==== command_reply(tid 0: 0 ) ==== 8+0+63 (crc 0 0 0) 0x562187ea44a0 con 0x562188003b30 2024-05-27T07:26:48.353 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:48.358+0000 7fdf34ff9700 10 client.15597 handle_command_reply: tid=0 2024-05-27T07:26:48.353 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:48.358+0000 7fdf62157b80 1 client.15597 shutdown 2024-05-27T07:26:48.353 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:48.358+0000 7fdf62157b80 1 -- 172.21.15.52:0/808543346 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x562187f7b120 msgr2=0x562187f9c8a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:26:48.353 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:48.358+0000 7fdf62157b80 1 --2- 172.21.15.52:0/808543346 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x562187f7b120 0x562187f9c8a0 secure :-1 s=READY pgs=464 cs=0 l=1 rev1=1 crypto rx=0x7fdf3c0a6860 tx=0x7fdf3c023f00 comp rx=0 tx=0).stop 2024-05-27T07:26:48.353 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:48.358+0000 7fdf62157b80 1 -- 172.21.15.52:0/808543346 shutdown_connections 2024-05-27T07:26:48.353 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:48.358+0000 7fdf62157b80 1 --2- 172.21.15.52:0/808543346 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x562187f81070 0x562187f9cde0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:26:48.353 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:48.358+0000 7fdf62157b80 1 --2- 172.21.15.52:0/808543346 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x562187dfbe40 0x562187f9eac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:26:48.353 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:48.358+0000 7fdf62157b80 1 --2- 172.21.15.52:0/808543346 >> [v2:172.21.15.105:6838/500348151,v1:172.21.15.105:6839/500348151] conn(0x562188003b30 0x562188005f50 crc :-1 s=READY pgs=51 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:26:48.353 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:48.358+0000 7fdf62157b80 1 --2- 172.21.15.52:0/808543346 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x562187f7b120 0x562187f9c8a0 unknown :-1 s=CLOSED pgs=464 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:26:48.353 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:48.358+0000 7fdf62157b80 1 -- 172.21.15.52:0/808543346 >> 172.21.15.52:0/808543346 conn(0x562187dfe770 msgr2=0x562187f7f800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:26:48.353 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:48.358+0000 7fdf34ff9700 0 client.15597 ms_handle_reset on v2:172.21.15.105:6838/500348151 2024-05-27T07:26:48.353 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:48.358+0000 7fdf62157b80 1 -- 172.21.15.52:0/808543346 shutdown_connections 2024-05-27T07:26:48.353 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:48.358+0000 7fdf62157b80 1 -- 172.21.15.52:0/808543346 wait complete. 2024-05-27T07:26:48.354 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:48.358+0000 7fdf62157b80 20 client.15597 trim_cache size 0 max 16384 2024-05-27T07:26:48.354 INFO:teuthology.orchestra.run.smithi052.stdout:{ 2024-05-27T07:26:48.354 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "no active scrubs running", 2024-05-27T07:26:48.354 INFO:teuthology.orchestra.run.smithi052.stdout: "scrubs": {} 2024-05-27T07:26:48.354 INFO:teuthology.orchestra.run.smithi052.stdout:} 2024-05-27T07:26:48.355 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:48.359+0000 7fdf53742700 1 -- 172.21.15.52:0/518760136 >> [v2:172.21.15.52:6800/1822318384,v1:172.21.15.52:6801/1822318384] conn(0x7fdf380b2b90 msgr2=0x7fdf380b50a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:26:48.355 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:48.359+0000 7fdf53742700 1 --2- 172.21.15.52:0/518760136 >> [v2:172.21.15.52:6800/1822318384,v1:172.21.15.52:6801/1822318384] conn(0x7fdf380b2b90 0x7fdf380b50a0 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7fdf4800aca0 tx=0x7fdf4800f6a0 comp rx=0 tx=0).stop 2024-05-27T07:26:48.355 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:48.359+0000 7fdf53742700 1 -- 172.21.15.52:0/518760136 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fdf4c107de0 msgr2=0x7fdf4c19b3a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:26:48.355 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:48.359+0000 7fdf53742700 1 --2- 172.21.15.52:0/518760136 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fdf4c107de0 0x7fdf4c19b3a0 secure :-1 s=READY pgs=460 cs=0 l=1 rev1=1 crypto rx=0x7fdf3c006010 tx=0x7fdf3c0047b0 comp rx=0 tx=0).stop 2024-05-27T07:26:48.355 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:48.359+0000 7fdf53742700 1 -- 172.21.15.52:0/518760136 shutdown_connections 2024-05-27T07:26:48.355 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:48.359+0000 7fdf53742700 1 --2- 172.21.15.52:0/518760136 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fdf4c108730 0x7fdf4c19f630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:26:48.355 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:48.359+0000 7fdf53742700 1 --2- 172.21.15.52:0/518760136 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fdf4c1034b0 0x7fdf4c19ae00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:26:48.355 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:48.359+0000 7fdf53742700 1 --2- 172.21.15.52:0/518760136 >> [v2:172.21.15.52:6800/1822318384,v1:172.21.15.52:6801/1822318384] conn(0x7fdf380b2b90 0x7fdf380b50a0 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:26:48.355 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:48.359+0000 7fdf53742700 1 --2- 172.21.15.52:0/518760136 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fdf4c107de0 0x7fdf4c19b3a0 unknown :-1 s=CLOSED pgs=460 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:26:48.356 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:48.359+0000 7fdf53742700 1 -- 172.21.15.52:0/518760136 >> 172.21.15.52:0/518760136 conn(0x7fdf4c0fedb0 msgr2=0x7fdf4c06b940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:26:48.356 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:48.359+0000 7fdf53742700 1 -- 172.21.15.52:0/518760136 shutdown_connections 2024-05-27T07:26:48.356 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:48.359+0000 7fdf53742700 1 -- 172.21.15.52:0/518760136 wait complete. 2024-05-27T07:26:48.356 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:26:48 smithi173 ceph-mon[37000]: pgmap v28: 129 pgs: 129 active+clean; 1.3 MiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 255 B/s wr, 8 op/s 2024-05-27T07:26:48.439 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:26:48 smithi105 ceph-mon[35977]: pgmap v28: 129 pgs: 129 active+clean; 1.3 MiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 255 B/s wr, 8 op/s 2024-05-27T07:26:49.191 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-05-27T07:26:49.191 DEBUG:teuthology.orchestra.run.smithi052:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:38f9d900c83c63799fbdbe61acc9a11b0d3554a6 shell --fsid 73df2636-1bf8-11ef-bc9b-c7b262605968 -- ceph fs dump --format=json 2024-05-27T07:26:49.444 INFO:teuthology.orchestra.run.smithi052.stderr:Inferring config /var/lib/ceph/73df2636-1bf8-11ef-bc9b-c7b262605968/mon.a/config 2024-05-27T07:26:50.229 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:26:50 smithi052 ceph-mon[31955]: pgmap v29: 129 pgs: 129 active+clean; 1.3 MiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 255 B/s wr, 12 op/s 2024-05-27T07:26:50.351 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:26:50 smithi173 ceph-mon[37000]: pgmap v29: 129 pgs: 129 active+clean; 1.3 MiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 255 B/s wr, 12 op/s 2024-05-27T07:26:50.439 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:26:50 smithi105 ceph-mon[35977]: pgmap v29: 129 pgs: 129 active+clean; 1.3 MiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 255 B/s wr, 12 op/s 2024-05-27T07:26:51.310 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:51.319+0000 7f1aaf7d5700 1 -- 172.21.15.52:0/2078478013 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f1aa8076e30 msgr2=0x7f1aa8077240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:26:51.310 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:51.319+0000 7f1aaf7d5700 1 --2- 172.21.15.52:0/2078478013 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f1aa8076e30 0x7f1aa8077240 secure :-1 s=READY pgs=465 cs=0 l=1 rev1=1 crypto rx=0x7f1aa400b0a0 tx=0x7f1aa400b3b0 comp rx=0 tx=0).stop 2024-05-27T07:26:51.310 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:51.319+0000 7f1aaf7d5700 1 -- 172.21.15.52:0/2078478013 shutdown_connections 2024-05-27T07:26:51.310 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:51.319+0000 7f1aaf7d5700 1 --2- 172.21.15.52:0/2078478013 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f1aa8110b70 0x7f1aa8112fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:26:51.310 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:51.319+0000 7f1aaf7d5700 1 --2- 172.21.15.52:0/2078478013 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f1aa8076e30 0x7f1aa8077240 unknown :-1 s=CLOSED pgs=465 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:26:51.310 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:51.319+0000 7f1aaf7d5700 1 --2- 172.21.15.52:0/2078478013 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f1aa8077780 0x7f1aa8110630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:26:51.310 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:51.319+0000 7f1aaf7d5700 1 -- 172.21.15.52:0/2078478013 >> 172.21.15.52:0/2078478013 conn(0x7f1aa80ffe90 msgr2=0x7f1aa81022e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:26:51.311 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:51.320+0000 7f1aaf7d5700 1 -- 172.21.15.52:0/2078478013 shutdown_connections 2024-05-27T07:26:51.311 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:51.320+0000 7f1aaf7d5700 1 -- 172.21.15.52:0/2078478013 wait complete. 2024-05-27T07:26:51.313 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:51.320+0000 7f1aaf7d5700 1 Processor -- start 2024-05-27T07:26:51.313 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:51.320+0000 7f1aaf7d5700 1 -- start start 2024-05-27T07:26:51.313 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:51.321+0000 7f1aaf7d5700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f1aa8076e30 0x7f1aa819d010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:26:51.313 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:51.321+0000 7f1aaf7d5700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f1aa8077780 0x7f1aa819d5b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:26:51.313 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:51.321+0000 7f1aaf7d5700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f1aa8110b70 0x7f1aa81a1840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:26:51.313 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:51.321+0000 7f1aaf7d5700 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x7f1aa8116660 con 0x7f1aa8110b70 2024-05-27T07:26:51.313 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:51.321+0000 7f1aaf7d5700 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f1aa81164e0 con 0x7f1aa8076e30 2024-05-27T07:26:51.313 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:51.321+0000 7f1aaf7d5700 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f1aa81167e0 con 0x7f1aa8077780 2024-05-27T07:26:51.313 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:51.321+0000 7f1aadd72700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f1aa8110b70 0x7f1aa81a1840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:26:51.313 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:51.321+0000 7f1aadd72700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f1aa8110b70 0x7f1aa81a1840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.52:37258/0 (socket says 172.21.15.52:37258) 2024-05-27T07:26:51.314 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:51.321+0000 7f1aadd72700 1 -- 172.21.15.52:0/3004158280 learned_addr learned my addr 172.21.15.52:0/3004158280 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-05-27T07:26:51.314 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:51.321+0000 7f1aad571700 1 --2- 172.21.15.52:0/3004158280 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f1aa8076e30 0x7f1aa819d010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:26:51.314 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:51.321+0000 7f1aacd70700 1 --2- 172.21.15.52:0/3004158280 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f1aa8077780 0x7f1aa819d5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:26:51.314 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:51.321+0000 7f1aadd72700 1 -- 172.21.15.52:0/3004158280 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f1aa8077780 msgr2=0x7f1aa819d5b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:26:51.314 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:51.321+0000 7f1aadd72700 1 --2- 172.21.15.52:0/3004158280 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f1aa8077780 0x7f1aa819d5b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:26:51.314 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:51.321+0000 7f1aadd72700 1 -- 172.21.15.52:0/3004158280 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f1aa8076e30 msgr2=0x7f1aa819d010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:26:51.314 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:51.321+0000 7f1aadd72700 1 --2- 172.21.15.52:0/3004158280 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f1aa8076e30 0x7f1aa819d010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:26:51.314 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:51.321+0000 7f1aadd72700 1 -- 172.21.15.52:0/3004158280 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1aa81a1e30 con 0x7f1aa8110b70 2024-05-27T07:26:51.315 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:51.321+0000 7f1aacd70700 1 --2- 172.21.15.52:0/3004158280 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f1aa8077780 0x7f1aa819d5b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-27T07:26:51.315 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:51.321+0000 7f1aadd72700 1 --2- 172.21.15.52:0/3004158280 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f1aa8110b70 0x7f1aa81a1840 secure :-1 s=READY pgs=466 cs=0 l=1 rev1=1 crypto rx=0x7f1aa000c8d0 tx=0x7f1aa000cbe0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:26:51.315 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:51.322+0000 7f1a9e7fc700 1 -- 172.21.15.52:0/3004158280 <== mon.0 v2:172.21.15.52:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1aa0004460 con 0x7f1aa8110b70 2024-05-27T07:26:51.315 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:51.322+0000 7f1a9e7fc700 1 -- 172.21.15.52:0/3004158280 <== mon.0 v2:172.21.15.52:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f1aa0003b30 con 0x7f1aa8110b70 2024-05-27T07:26:51.315 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:51.322+0000 7f1a9e7fc700 1 -- 172.21.15.52:0/3004158280 <== mon.0 v2:172.21.15.52:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1aa0007c90 con 0x7f1aa8110b70 2024-05-27T07:26:51.315 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:51.322+0000 7f1aaf7d5700 1 -- 172.21.15.52:0/3004158280 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1aa81a2120 con 0x7f1aa8110b70 2024-05-27T07:26:51.315 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:51.322+0000 7f1aaf7d5700 1 -- 172.21.15.52:0/3004158280 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1aa81a2610 con 0x7f1aa8110b70 2024-05-27T07:26:51.315 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:51.323+0000 7f1aaf7d5700 1 -- 172.21.15.52:0/3004158280 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1aa80512b0 con 0x7f1aa8110b70 2024-05-27T07:26:51.316 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:51.325+0000 7f1a9e7fc700 1 -- 172.21.15.52:0/3004158280 <== mon.0 v2:172.21.15.52:3300/0 4 ==== mgrmap(e 24) ==== 149879+0+0 (secure 0 0 0) 0x7f1aa0003ce0 con 0x7f1aa8110b70 2024-05-27T07:26:51.316 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:51.325+0000 7f1a9e7fc700 1 --2- 172.21.15.52:0/3004158280 >> [v2:172.21.15.52:6800/1822318384,v1:172.21.15.52:6801/1822318384] conn(0x7f1a940b2b10 0x7f1a940b5020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:26:51.316 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:51.325+0000 7f1aad571700 1 --2- 172.21.15.52:0/3004158280 >> [v2:172.21.15.52:6800/1822318384,v1:172.21.15.52:6801/1822318384] conn(0x7f1a940b2b10 0x7f1a940b5020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:26:51.316 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:51.325+0000 7f1a9e7fc700 1 -- 172.21.15.52:0/3004158280 <== mon.0 v2:172.21.15.52:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9379+0+0 (secure 0 0 0) 0x7f1aa00e2be0 con 0x7f1aa8110b70 2024-05-27T07:26:51.316 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:51.326+0000 7f1aad571700 1 --2- 172.21.15.52:0/3004158280 >> [v2:172.21.15.52:6800/1822318384,v1:172.21.15.52:6801/1822318384] conn(0x7f1a940b2b10 0x7f1a940b5020 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f1aa4009e60 tx=0x7f1aa400b7b0 comp rx=0 tx=0).ready entity=mgr.15333 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:26:51.319 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:51.329+0000 7f1a9e7fc700 1 -- 172.21.15.52:0/3004158280 <== mon.0 v2:172.21.15.52:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191963 (secure 0 0 0) 0x7f1aa0013030 con 0x7f1aa8110b70 2024-05-27T07:26:51.581 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:51.592+0000 7f1aaf7d5700 1 -- 172.21.15.52:0/3004158280 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f1aa819e370 con 0x7f1aa8110b70 2024-05-27T07:26:51.582 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:51.593+0000 7f1a9e7fc700 1 -- 172.21.15.52:0/3004158280 <== mon.0 v2:172.21.15.52:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 30 v30) ==== 94+0+11938 (secure 0 0 0) 0x7f1aa00aa0a0 con 0x7f1aa8110b70 2024-05-27T07:26:51.583 INFO:teuthology.orchestra.run.smithi052.stdout: 2024-05-27T07:26:51.583 INFO:teuthology.orchestra.run.smithi052.stdout:{"epoch":30,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14556,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.52:6835/2584733778","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.52:6834","nonce":2584733778},{"type":"v1","addr":"172.21.15.52:6835","nonce":2584733778}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14568,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.52:6837/675445303","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.52:6836","nonce":675445303},{"type":"v1","addr":"172.21.15.52:6837","nonce":675445303}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":30,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-05-27T07:21:50.521783+0000","modified":"2024-05-27T07:25:54.323921+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24527,"mds_1":24497,"mds_2":24475,"mds_3":24473,"mds_4":24466},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14619":{"gid":14619,"name":"g","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.52:6839/4258549559","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.52:6838","nonce":4258549559},{"type":"v1","addr":"172.21.15.52:6839","nonce":4258549559}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_14631":{"gid":14631,"name":"a","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.52:6841/2878618188","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.52:6840","nonce":2878618188},{"type":"v1","addr":"172.21.15.52:6841","nonce":2878618188}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24433":{"gid":24433,"name":"l","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.173:6833/3013589220","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6832","nonce":3013589220},{"type":"v1","addr":"172.21.15.173:6833","nonce":3013589220}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24452":{"gid":24452,"name":"k","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.105:6833/2800678412","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6832","nonce":2800678412},{"type":"v1","addr":"172.21.15.105:6833","nonce":2800678412}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24460":{"gid":24460,"name":"f","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.173:6835/1619486437","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6834","nonce":1619486437},{"type":"v1","addr":"172.21.15.173:6835","nonce":1619486437}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24466":{"gid":24466,"name":"c","rank":4,"incarnation":26,"state":"up:active","state_seq":25,"addr":"172.21.15.173:6837/1396363827","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6836","nonce":1396363827},{"type":"v1","addr":"172.21.15.173:6837","nonce":1396363827}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24473":{"gid":24473,"name":"e","rank":3,"incarnation":24,"state":"up:active","state_seq":26,"addr":"172.21.15.105:6835/3298498384","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6834","nonce":3298498384},{"type":"v1","addr":"172.21.15.105:6835","nonce":3298498384}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24475":{"gid":24475,"name":"i","rank":2,"incarnation":22,"state":"up:active","state_seq":23,"addr":"172.21.15.173:6839/564778293","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6838","nonce":564778293},{"type":"v1","addr":"172.21.15.173:6839","nonce":564778293}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24497":{"gid":24497,"name":"h","rank":1,"incarnation":20,"state":"up:active","state_seq":24,"addr":"172.21.15.105:6837/191393149","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6836","nonce":191393149},{"type":"v1","addr":"172.21.15.105:6837","nonce":191393149}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24527":{"gid":24527,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":5,"addr":"172.21.15.105:6839/500348151","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6838","nonce":500348151},{"type":"v1","addr":"172.21.15.105:6839","nonce":500348151}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24527,"qdb_cluster":[24527,24497,24475,24466,24473]},"id":1}]} 2024-05-27T07:26:51.585 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:51.595+0000 7f1aaf7d5700 1 -- 172.21.15.52:0/3004158280 >> [v2:172.21.15.52:6800/1822318384,v1:172.21.15.52:6801/1822318384] conn(0x7f1a940b2b10 msgr2=0x7f1a940b5020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:26:51.585 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:51.595+0000 7f1aaf7d5700 1 --2- 172.21.15.52:0/3004158280 >> [v2:172.21.15.52:6800/1822318384,v1:172.21.15.52:6801/1822318384] conn(0x7f1a940b2b10 0x7f1a940b5020 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f1aa4009e60 tx=0x7f1aa400b7b0 comp rx=0 tx=0).stop 2024-05-27T07:26:51.585 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:51.595+0000 7f1aaf7d5700 1 -- 172.21.15.52:0/3004158280 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f1aa8110b70 msgr2=0x7f1aa81a1840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:26:51.586 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:51.595+0000 7f1aaf7d5700 1 --2- 172.21.15.52:0/3004158280 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f1aa8110b70 0x7f1aa81a1840 secure :-1 s=READY pgs=466 cs=0 l=1 rev1=1 crypto rx=0x7f1aa000c8d0 tx=0x7f1aa000cbe0 comp rx=0 tx=0).stop 2024-05-27T07:26:51.586 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:51.595+0000 7f1aaf7d5700 1 -- 172.21.15.52:0/3004158280 shutdown_connections 2024-05-27T07:26:51.586 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:51.595+0000 7f1aaf7d5700 1 --2- 172.21.15.52:0/3004158280 >> [v2:172.21.15.52:6800/1822318384,v1:172.21.15.52:6801/1822318384] conn(0x7f1a940b2b10 0x7f1a940b5020 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:26:51.586 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:51.595+0000 7f1aaf7d5700 1 --2- 172.21.15.52:0/3004158280 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f1aa8110b70 0x7f1aa81a1840 unknown :-1 s=CLOSED pgs=466 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:26:51.586 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:51.595+0000 7f1aaf7d5700 1 --2- 172.21.15.52:0/3004158280 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f1aa8076e30 0x7f1aa819d010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:26:51.586 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:51.595+0000 7f1aaf7d5700 1 --2- 172.21.15.52:0/3004158280 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f1aa8077780 0x7f1aa819d5b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:26:51.586 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:51.595+0000 7f1aaf7d5700 1 -- 172.21.15.52:0/3004158280 >> 172.21.15.52:0/3004158280 conn(0x7f1aa80ffe90 msgr2=0x7f1aa8100520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:26:51.587 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:51.595+0000 7f1aaf7d5700 1 -- 172.21.15.52:0/3004158280 shutdown_connections 2024-05-27T07:26:51.587 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:51.595+0000 7f1aaf7d5700 1 -- 172.21.15.52:0/3004158280 wait complete. 2024-05-27T07:26:51.587 INFO:teuthology.orchestra.run.smithi052.stderr:dumped fsmap epoch 30 2024-05-27T07:26:52.229 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:26:52 smithi052 ceph-mon[31955]: pgmap v30: 129 pgs: 129 active+clean; 1.3 MiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 255 B/s wr, 8 op/s 2024-05-27T07:26:52.229 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:26:52 smithi052 ceph-mon[31955]: from='client.? 172.21.15.52:0/3004158280' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-27T07:26:52.276 DEBUG:teuthology.orchestra.run.smithi052:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:38f9d900c83c63799fbdbe61acc9a11b0d3554a6 shell --fsid 73df2636-1bf8-11ef-bc9b-c7b262605968 -- ceph tell mds.1:4 damage ls 2024-05-27T07:26:52.351 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:26:52 smithi173 ceph-mon[37000]: pgmap v30: 129 pgs: 129 active+clean; 1.3 MiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 255 B/s wr, 8 op/s 2024-05-27T07:26:52.352 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:26:52 smithi173 ceph-mon[37000]: from='client.? 172.21.15.52:0/3004158280' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-27T07:26:52.436 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:26:52.447+0000 7f03e68a8640 1 -- 172.21.15.52:0/2304018309 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f03e010e200 msgr2=0x7f03e01106f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:26:52.437 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:26:52.447+0000 7f03e68a8640 1 --2- 172.21.15.52:0/2304018309 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f03e010e200 0x7f03e01106f0 secure :-1 s=READY pgs=467 cs=0 l=1 rev1=1 crypto rx=0x7f03d8009ca0 tx=0x7f03d802a9e0 comp rx=0 tx=0).stop 2024-05-27T07:26:52.437 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:26:52.447+0000 7f03e68a8640 1 -- 172.21.15.52:0/2304018309 shutdown_connections 2024-05-27T07:26:52.437 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:26:52.447+0000 7f03e68a8640 1 --2- 172.21.15.52:0/2304018309 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f03e010e200 0x7f03e01106f0 unknown :-1 s=CLOSED pgs=467 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:26:52.437 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:26:52.447+0000 7f03e68a8640 1 --2- 172.21.15.52:0/2304018309 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f03e010b850 0x7f03e010dc70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:26:52.437 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:26:52.447+0000 7f03e68a8640 1 --2- 172.21.15.52:0/2304018309 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f03e0104df0 0x7f03e0105200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:26:52.437 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:26:52.447+0000 7f03e68a8640 1 -- 172.21.15.52:0/2304018309 >> 172.21.15.52:0/2304018309 conn(0x7f03e00782d0 msgr2=0x7f03e0078710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:26:52.437 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:26:52.447+0000 7f03e68a8640 1 -- 172.21.15.52:0/2304018309 shutdown_connections 2024-05-27T07:26:52.437 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:26:52.447+0000 7f03e68a8640 1 -- 172.21.15.52:0/2304018309 wait complete. 2024-05-27T07:26:52.439 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:26:52.447+0000 7f03e68a8640 1 Processor -- start 2024-05-27T07:26:52.439 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:26:52.447+0000 7f03e68a8640 1 -- start start 2024-05-27T07:26:52.439 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:26:52.448+0000 7f03e68a8640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f03e0104df0 0x7f03e01026b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:26:52.441 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:26:52.448+0000 7f03e68a8640 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f03e010b850 0x7f03e0102bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:26:52.441 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:26:52.448+0000 7f03e68a8640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f03e010e200 0x7f03e0103130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:26:52.441 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:26:52.448+0000 7f03e68a8640 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x7f03e0113e60 con 0x7f03e010b850 2024-05-27T07:26:52.441 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:26:52.448+0000 7f03e68a8640 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f03e0113ce0 con 0x7f03e0104df0 2024-05-27T07:26:52.441 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:26:52.448+0000 7f03e68a8640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f03e0113fe0 con 0x7f03e010e200 2024-05-27T07:26:52.441 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:26:52.448+0000 7f03df7fe640 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f03e010b850 0x7f03e0102bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:26:52.442 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:26:52.448+0000 7f03df7fe640 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f03e010b850 0x7f03e0102bf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.52:37270/0 (socket says 172.21.15.52:37270) 2024-05-27T07:26:52.442 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:26:52.448+0000 7f03df7fe640 1 -- 172.21.15.52:0/293617107 learned_addr learned my addr 172.21.15.52:0/293617107 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-05-27T07:26:52.442 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:26:52.448+0000 7f03dffff640 1 --2- 172.21.15.52:0/293617107 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f03e0104df0 0x7f03e01026b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:26:52.442 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:26:52.448+0000 7f03e4e45640 1 --2- 172.21.15.52:0/293617107 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f03e010e200 0x7f03e0103130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:26:52.442 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:26:52.448+0000 7f03df7fe640 1 -- 172.21.15.52:0/293617107 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f03e010e200 msgr2=0x7f03e0103130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:26:52.442 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:26:52.448+0000 7f03df7fe640 1 --2- 172.21.15.52:0/293617107 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f03e010e200 0x7f03e0103130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:26:52.442 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:26:52.448+0000 7f03df7fe640 1 -- 172.21.15.52:0/293617107 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f03e0104df0 msgr2=0x7f03e01026b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:26:52.442 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:26:52.448+0000 7f03df7fe640 1 --2- 172.21.15.52:0/293617107 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f03e0104df0 0x7f03e01026b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:26:52.442 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:26:52.448+0000 7f03df7fe640 1 -- 172.21.15.52:0/293617107 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f03e01a8b80 con 0x7f03e010b850 2024-05-27T07:26:52.442 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:26:52.448+0000 7f03dffff640 1 --2- 172.21.15.52:0/293617107 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f03e0104df0 0x7f03e01026b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-27T07:26:52.442 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:26:52.449+0000 7f03df7fe640 1 --2- 172.21.15.52:0/293617107 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f03e010b850 0x7f03e0102bf0 secure :-1 s=READY pgs=468 cs=0 l=1 rev1=1 crypto rx=0x7f03d400e9f0 tx=0x7f03d400edf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:26:52.442 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:26:52.449+0000 7f03dd7fa640 1 -- 172.21.15.52:0/293617107 <== mon.0 v2:172.21.15.52:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f03d400cd80 con 0x7f03e010b850 2024-05-27T07:26:52.443 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:26:52.449+0000 7f03dd7fa640 1 -- 172.21.15.52:0/293617107 <== mon.0 v2:172.21.15.52:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f03d40054d0 con 0x7f03e010b850 2024-05-27T07:26:52.443 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:26:52.449+0000 7f03e68a8640 1 -- 172.21.15.52:0/293617107 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f03e01a8e70 con 0x7f03e010b850 2024-05-27T07:26:52.443 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:26:52.449+0000 7f03dd7fa640 1 -- 172.21.15.52:0/293617107 <== mon.0 v2:172.21.15.52:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f03d4010690 con 0x7f03e010b850 2024-05-27T07:26:52.443 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:26:52.449+0000 7f03e68a8640 1 -- 172.21.15.52:0/293617107 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f03e01a92e0 con 0x7f03e010b850 2024-05-27T07:26:52.443 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:26:52 smithi105 ceph-mon[35977]: pgmap v30: 129 pgs: 129 active+clean; 1.3 MiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 255 B/s wr, 8 op/s 2024-05-27T07:26:52.443 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:26:52 smithi105 ceph-mon[35977]: from='client.? 172.21.15.52:0/3004158280' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-27T07:26:52.448 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:26:52.451+0000 7f03ed838740 1 Processor -- start 2024-05-27T07:26:52.449 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:26:52.452+0000 7f03ed838740 1 -- start start 2024-05-27T07:26:52.449 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:26:52.452+0000 7f03ed838740 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55f99d9841b0 0x55f99dd253d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:26:52.449 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:26:52.452+0000 7f03dd7fa640 1 -- 172.21.15.52:0/293617107 <== mon.0 v2:172.21.15.52:3300/0 4 ==== mgrmap(e 24) ==== 149879+0+0 (secure 0 0 0) 0x7f03d4004e90 con 0x7f03e010b850 2024-05-27T07:26:52.449 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:26:52.452+0000 7f03ed838740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55f99d95b2f0 0x55f99dd25910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:26:52.449 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:26:52.452+0000 7f03ed838740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55f99d9634d0 0x55f99dd25e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:26:52.449 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:26:52.452+0000 7f03ed838740 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x55f99dc97c30 con 0x55f99d9841b0 2024-05-27T07:26:52.449 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:26:52.452+0000 7f03ed838740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x55f99dc877d0 con 0x55f99d95b2f0 2024-05-27T07:26:52.449 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:26:52.452+0000 7f03ed838740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x55f99dc8b810 con 0x55f99d9634d0 2024-05-27T07:26:52.449 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:26:52.452+0000 7f03e4e45640 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55f99d9841b0 0x55f99dd253d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:26:52.450 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:26:52.452+0000 7f03e4e45640 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55f99d9841b0 0x55f99dd253d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.52:37280/0 (socket says 172.21.15.52:37280) 2024-05-27T07:26:52.450 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:26:52.452+0000 7f03e4e45640 1 -- 172.21.15.52:0/3039748991 learned_addr learned my addr 172.21.15.52:0/3039748991 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-05-27T07:26:52.450 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:26:52.452+0000 7f03dffff640 1 --2- 172.21.15.52:0/3039748991 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55f99d95b2f0 0x55f99dd25910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:26:52.450 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:26:52.452+0000 7f03df7fe640 1 --2- 172.21.15.52:0/3039748991 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55f99d9634d0 0x55f99dd25e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:26:52.450 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:26:52.452+0000 7f03e4e45640 1 -- 172.21.15.52:0/3039748991 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55f99d9634d0 msgr2=0x55f99dd25e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:26:52.450 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:26:52.452+0000 7f03e4e45640 1 --2- 172.21.15.52:0/3039748991 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55f99d9634d0 0x55f99dd25e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:26:52.450 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:26:52.452+0000 7f03e4e45640 1 -- 172.21.15.52:0/3039748991 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55f99d95b2f0 msgr2=0x55f99dd25910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:26:52.450 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:26:52.452+0000 7f03e4e45640 1 --2- 172.21.15.52:0/3039748991 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55f99d95b2f0 0x55f99dd25910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:26:52.450 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:26:52.452+0000 7f03e4e45640 1 -- 172.21.15.52:0/3039748991 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55f99d99c210 con 0x55f99d9841b0 2024-05-27T07:26:52.450 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:26:52.453+0000 7f03e4e45640 1 --2- 172.21.15.52:0/3039748991 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55f99d9841b0 0x55f99dd253d0 secure :-1 s=READY pgs=469 cs=0 l=1 rev1=1 crypto rx=0x7f03d8009460 tx=0x7f03d8009490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=793e1f02d5d5447a server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:26:52.450 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:26:52.453+0000 7f03dd7fa640 1 --2- 172.21.15.52:0/293617107 >> [v2:172.21.15.52:6800/1822318384,v1:172.21.15.52:6801/1822318384] conn(0x7f03c80b2a70 0x7f03c80b4f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:26:52.450 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:26:52.453+0000 7f03c67fc640 1 -- 172.21.15.52:0/3039748991 <== mon.0 v2:172.21.15.52:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f03d8047070 con 0x55f99d9841b0 2024-05-27T07:26:52.451 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:26:52.453+0000 7f03c67fc640 1 -- 172.21.15.52:0/3039748991 <== mon.0 v2:172.21.15.52:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f03d8003e20 con 0x55f99d9841b0 2024-05-27T07:26:52.451 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:26:52.453+0000 7f03c67fc640 1 -- 172.21.15.52:0/3039748991 <== mon.0 v2:172.21.15.52:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f03d8033050 con 0x55f99d9841b0 2024-05-27T07:26:52.451 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:26:52.453+0000 7f03dffff640 1 --2- 172.21.15.52:0/293617107 >> [v2:172.21.15.52:6800/1822318384,v1:172.21.15.52:6801/1822318384] conn(0x7f03c80b2a70 0x7f03c80b4f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:26:52.451 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:26:52.453+0000 7f03dd7fa640 1 -- 172.21.15.52:0/293617107 <== mon.0 v2:172.21.15.52:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9379+0+0 (secure 0 0 0) 0x7f03d40e3110 con 0x7f03e010b850 2024-05-27T07:26:52.451 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:26:52.454+0000 7f03ed838740 1 -- 172.21.15.52:0/3039748991 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55f99d9841b0 msgr2=0x55f99dd253d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:26:52.451 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:26:52.454+0000 7f03dffff640 1 --2- 172.21.15.52:0/293617107 >> [v2:172.21.15.52:6800/1822318384,v1:172.21.15.52:6801/1822318384] conn(0x7f03c80b2a70 0x7f03c80b4f80 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x55f99dd26780 tx=0x7f03d00094a0 comp rx=0 tx=0).ready entity=mgr.15333 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:26:52.451 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:26:52.454+0000 7f03ed838740 1 --2- 172.21.15.52:0/3039748991 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55f99d9841b0 0x55f99dd253d0 secure :-1 s=READY pgs=469 cs=0 l=1 rev1=1 crypto rx=0x7f03d8009460 tx=0x7f03d8009490 comp rx=0 tx=0).stop 2024-05-27T07:26:52.451 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:26:52.454+0000 7f03ed838740 1 -- 172.21.15.52:0/3039748991 shutdown_connections 2024-05-27T07:26:52.451 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:26:52.454+0000 7f03ed838740 1 --2- 172.21.15.52:0/3039748991 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55f99d9634d0 0x55f99dd25e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:26:52.451 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:26:52.454+0000 7f03ed838740 1 --2- 172.21.15.52:0/3039748991 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55f99d95b2f0 0x55f99dd25910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:26:52.451 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:26:52.454+0000 7f03ed838740 1 --2- 172.21.15.52:0/3039748991 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55f99d9841b0 0x55f99dd253d0 unknown :-1 s=CLOSED pgs=469 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:26:52.451 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:26:52.454+0000 7f03ed838740 1 -- 172.21.15.52:0/3039748991 >> 172.21.15.52:0/3039748991 conn(0x55f99d978300 msgr2=0x55f99dd21b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:26:52.452 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:26:52.454+0000 7f03ed838740 1 -- 172.21.15.52:0/3039748991 shutdown_connections 2024-05-27T07:26:52.452 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:26:52.454+0000 7f03ed838740 1 -- 172.21.15.52:0/3039748991 wait complete. 2024-05-27T07:26:52.452 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:26:52.455+0000 7f03ed838740 1 -- start start 2024-05-27T07:26:52.452 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:26:52.455+0000 7f03ed838740 1 Processor -- start 2024-05-27T07:26:52.452 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:26:52.456+0000 7f03ed838740 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55f99d95b2f0 0x55f99dd314d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:26:52.452 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:26:52.456+0000 7f03ed838740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55f99d9634d0 0x55f99dd31a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:26:52.452 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:26:52.456+0000 7f03ed838740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55f99d9841b0 0x55f99dd2f950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:26:52.452 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:26:52.456+0000 7f03ed838740 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x55f99d96d790 con 0x55f99d95b2f0 2024-05-27T07:26:52.452 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:26:52.456+0000 7f03ed838740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x55f99dc43890 con 0x55f99d9841b0 2024-05-27T07:26:52.452 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:26:52.456+0000 7f03ed838740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x55f99dd324a0 con 0x55f99d9634d0 2024-05-27T07:26:52.452 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:26:52.456+0000 7f03ed838740 10 client.0 _ll_register_callbacks cb 0x55f99d99ba00 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T07:26:52.452 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:26:52.456+0000 7f03dffff640 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55f99d95b2f0 0x55f99dd314d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:26:52.453 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:26:52.456+0000 7f03dffff640 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55f99d95b2f0 0x55f99dd314d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.52:37282/0 (socket says 172.21.15.52:37282) 2024-05-27T07:26:52.453 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:26:52.456+0000 7f03dffff640 1 -- 172.21.15.52:0/3504968578 learned_addr learned my addr 172.21.15.52:0/3504968578 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-05-27T07:26:52.453 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:26:52.456+0000 7f03df7fe640 1 --2- 172.21.15.52:0/3504968578 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55f99d9634d0 0x55f99dd31a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:26:52.453 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:26:52.456+0000 7f03e4e45640 1 --2- 172.21.15.52:0/3504968578 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55f99d9841b0 0x55f99dd2f950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:26:52.453 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:26:52.456+0000 7f03dffff640 1 -- 172.21.15.52:0/3504968578 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55f99d9634d0 msgr2=0x55f99dd31a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:26:52.453 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:26:52.456+0000 7f03dffff640 1 --2- 172.21.15.52:0/3504968578 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55f99d9634d0 0x55f99dd31a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:26:52.453 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:26:52.456+0000 7f03dffff640 1 -- 172.21.15.52:0/3504968578 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55f99d9841b0 msgr2=0x55f99dd2f950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:26:52.453 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:26:52.456+0000 7f03dffff640 1 --2- 172.21.15.52:0/3504968578 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55f99d9841b0 0x55f99dd2f950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:26:52.453 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:26:52.456+0000 7f03dffff640 1 -- 172.21.15.52:0/3504968578 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55f99da68620 con 0x55f99d95b2f0 2024-05-27T07:26:52.453 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:26:52.456+0000 7f03e4e45640 1 --2- 172.21.15.52:0/3504968578 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55f99d9841b0 0x55f99dd2f950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-27T07:26:52.453 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:26:52.456+0000 7f03dffff640 1 --2- 172.21.15.52:0/3504968578 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55f99d95b2f0 0x55f99dd314d0 secure :-1 s=READY pgs=470 cs=0 l=1 rev1=1 crypto rx=0x7f03d0012700 tx=0x7f03d00046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9cb93001111b113d server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:26:52.453 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:26:52.456+0000 7f03bf7fe640 10 client.0 ms_handle_connect on v2:172.21.15.52:3300/0 2024-05-27T07:26:52.453 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:26:52.457+0000 7f03bf7fe640 1 -- 172.21.15.52:0/3504968578 <== mon.0 v2:172.21.15.52:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f03d001f8f0 con 0x55f99d95b2f0 2024-05-27T07:26:52.454 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:26:52.457+0000 7f03bf7fe640 1 -- 172.21.15.52:0/3504968578 <== mon.0 v2:172.21.15.52:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 9379+0+0 (secure 0 0 0) 0x7f03d001fa90 con 0x55f99d95b2f0 2024-05-27T07:26:52.454 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:26:52.457+0000 7f03ed838740 1 -- 172.21.15.52:0/3504968578 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f03d8003e20 con 0x55f99d95b2f0 2024-05-27T07:26:52.454 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:26:52.457+0000 7f03bf7fe640 1 -- 172.21.15.52:0/3504968578 <== mon.0 v2:172.21.15.52:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7f03d0030050 con 0x55f99d95b2f0 2024-05-27T07:26:52.454 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:26:52.457+0000 7f03ed838740 10 client.15615 fetch_fsmap learned FSMap version 30 2024-05-27T07:26:52.454 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:26:52.457+0000 7f03ed838740 1 -- 172.21.15.52:0/3504968578 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x55f99dd3e890 con 0x55f99d95b2f0 2024-05-27T07:26:52.454 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:26:52.457+0000 7f03bf7fe640 1 -- 172.21.15.52:0/3504968578 <== mon.0 v2:172.21.15.52:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f03d8003e20 con 0x55f99d95b2f0 2024-05-27T07:26:52.454 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:26:52.457+0000 7f03bf7fe640 1 -- 172.21.15.52:0/3504968578 <== mon.0 v2:172.21.15.52:3300/0 5 ==== fsmap(e 30) ==== 8362+0+0 (secure 0 0 0) 0x7f03d000f040 con 0x55f99d95b2f0 2024-05-27T07:26:52.454 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:26:52.457+0000 7f03ed838740 10 client.15615 fetch_fsmap finished waiting for FSMap version 30 2024-05-27T07:26:52.454 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:26:52.457+0000 7f03ed838740 10 client.15615 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-05-27T07:26:52.455 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:26:52.457+0000 7f03ed838740 20 client.15615 populate_metadata read hostname 'smithi052' 2024-05-27T07:26:52.455 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:26:52.457+0000 7f03ed838740 1 --2- 172.21.15.52:0/3504968578 >> [v2:172.21.15.105:6838/500348151,v1:172.21.15.105:6839/500348151] conn(0x55f99d942d80 0x55f99dd41dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:26:52.455 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:26:52.457+0000 7f03ed838740 4 client.15615 mds_command: new command op to 24527 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-27T07:26:52.455 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:26:52.457+0000 7f03ed838740 1 -- 172.21.15.52:0/3504968578 --> [v2:172.21.15.105:6838/500348151,v1:172.21.15.105:6839/500348151] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55f99dc99b80 con 0x55f99d942d80 2024-05-27T07:26:52.455 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:26:52.458+0000 7f03df7fe640 1 --2- 172.21.15.52:0/3504968578 >> [v2:172.21.15.105:6838/500348151,v1:172.21.15.105:6839/500348151] conn(0x55f99d942d80 0x55f99dd41dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:26:52.455 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:26:52.458+0000 7f03df7fe640 1 --2- 172.21.15.52:0/3504968578 >> [v2:172.21.15.105:6838/500348151,v1:172.21.15.105:6839/500348151] conn(0x55f99d942d80 0x55f99dd41dc0 crc :-1 s=READY pgs=52 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=fd9d93a4f21df2fa server_cookie=d6fef2909bd57c4a in_seq=0 out_seq=0 2024-05-27T07:26:52.455 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:26:52.458+0000 7f03bf7fe640 10 client.15615 ms_handle_connect on v2:172.21.15.105:6838/500348151 2024-05-27T07:26:52.455 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:26:52.459+0000 7f03bf7fe640 1 -- 172.21.15.52:0/3504968578 <== mds.0 v2:172.21.15.105:6838/500348151 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24041 (crc 0 0 0) 0x55f99dc99b80 con 0x55f99d942d80 2024-05-27T07:26:52.455 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:26:52.459+0000 7f03bf7fe640 10 client.15615 handle_command_reply: tid=0 2024-05-27T07:26:52.455 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:26:52.459+0000 7f03ed838740 1 client.15615 shutdown 2024-05-27T07:26:52.456 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:26:52.459+0000 7f03ed838740 1 -- 172.21.15.52:0/3504968578 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55f99d95b2f0 msgr2=0x55f99dd314d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:26:52.456 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:26:52.459+0000 7f03ed838740 1 --2- 172.21.15.52:0/3504968578 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55f99d95b2f0 0x55f99dd314d0 secure :-1 s=READY pgs=470 cs=0 l=1 rev1=1 crypto rx=0x7f03d0012700 tx=0x7f03d00046c0 comp rx=0 tx=0).stop 2024-05-27T07:26:52.457 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:26:52.459+0000 7f03ed838740 1 -- 172.21.15.52:0/3504968578 shutdown_connections 2024-05-27T07:26:52.457 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:26:52.459+0000 7f03ed838740 1 --2- 172.21.15.52:0/3504968578 >> [v2:172.21.15.105:6838/500348151,v1:172.21.15.105:6839/500348151] conn(0x55f99d942d80 0x55f99dd41dc0 crc :-1 s=READY pgs=52 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:26:52.457 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:26:52.459+0000 7f03ed838740 1 --2- 172.21.15.52:0/3504968578 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55f99d9841b0 0x55f99dd2f950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:26:52.457 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:26:52.459+0000 7f03bf7fe640 0 client.15615 ms_handle_reset on v2:172.21.15.105:6838/500348151 2024-05-27T07:26:52.457 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:26:52.459+0000 7f03ed838740 1 --2- 172.21.15.52:0/3504968578 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55f99d9634d0 0x55f99dd31a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:26:52.457 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:26:52.459+0000 7f03ed838740 1 --2- 172.21.15.52:0/3504968578 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55f99d95b2f0 0x55f99dd314d0 unknown :-1 s=CLOSED pgs=470 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:26:52.457 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:26:52.459+0000 7f03ed838740 1 -- 172.21.15.52:0/3504968578 >> 172.21.15.52:0/3504968578 conn(0x55f99d978300 msgr2=0x55f99dd25200 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:26:52.457 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:26:52.460+0000 7f03ed838740 1 -- 172.21.15.52:0/3504968578 shutdown_connections 2024-05-27T07:26:52.457 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:26:52.460+0000 7f03ed838740 1 -- 172.21.15.52:0/3504968578 wait complete. 2024-05-27T07:26:52.457 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:26:52.460+0000 7f03ed838740 20 client.15615 trim_cache size 0 max 16384 2024-05-27T07:26:52.467 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:26:52.476+0000 7f03ed838740 1 Processor -- start 2024-05-27T07:26:52.467 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:26:52.476+0000 7f03ed838740 1 -- start start 2024-05-27T07:26:52.467 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:26:52.476+0000 7f03ed838740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55f99d95b2f0 0x55f99dd31780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:26:52.467 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:26:52.476+0000 7f03ed838740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55f99d9634d0 0x55f99dd37760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:26:52.468 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:26:52.476+0000 7f03ed838740 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55f99d9841b0 0x55f99dd37ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:26:52.468 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:26:52.476+0000 7f03ed838740 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x7f03e01b3330 con 0x55f99d9841b0 2024-05-27T07:26:52.468 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:26:52.476+0000 7f03ed838740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f03e01b4e70 con 0x55f99d95b2f0 2024-05-27T07:26:52.468 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:26:52.476+0000 7f03ed838740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f03e01b3a00 con 0x55f99d9634d0 2024-05-27T07:26:52.468 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:26:52.477+0000 7f03e4e45640 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55f99d9841b0 0x55f99dd37ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:26:52.468 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:26:52.477+0000 7f03e4e45640 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55f99d9841b0 0x55f99dd37ca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.52:37298/0 (socket says 172.21.15.52:37298) 2024-05-27T07:26:52.468 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:26:52.477+0000 7f03e4e45640 1 -- 172.21.15.52:0/2950471163 learned_addr learned my addr 172.21.15.52:0/2950471163 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-05-27T07:26:52.468 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:26:52.477+0000 7f03dffff640 1 --2- 172.21.15.52:0/2950471163 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55f99d95b2f0 0x55f99dd31780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:26:52.468 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:26:52.477+0000 7f03df7fe640 1 --2- 172.21.15.52:0/2950471163 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55f99d9634d0 0x55f99dd37760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:26:52.468 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:26:52.477+0000 7f03e4e45640 1 -- 172.21.15.52:0/2950471163 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55f99d9634d0 msgr2=0x55f99dd37760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:26:52.468 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:26:52.477+0000 7f03e4e45640 1 --2- 172.21.15.52:0/2950471163 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55f99d9634d0 0x55f99dd37760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:26:52.468 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:26:52.477+0000 7f03e4e45640 1 -- 172.21.15.52:0/2950471163 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55f99d95b2f0 msgr2=0x55f99dd31780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:26:52.468 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:26:52.477+0000 7f03e4e45640 1 --2- 172.21.15.52:0/2950471163 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55f99d95b2f0 0x55f99dd31780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:26:52.469 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:26:52.477+0000 7f03e4e45640 1 -- 172.21.15.52:0/2950471163 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55f99dd26180 con 0x55f99d9841b0 2024-05-27T07:26:52.469 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:26:52.477+0000 7f03e4e45640 1 --2- 172.21.15.52:0/2950471163 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55f99d9841b0 0x55f99dd37ca0 secure :-1 s=READY pgs=471 cs=0 l=1 rev1=1 crypto rx=0x7f03d8009ca0 tx=0x7f03d80095b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8da00035505dccc8 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:26:52.469 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:26:52.478+0000 7f03c5ffb640 1 -- 172.21.15.52:0/2950471163 <== mon.0 v2:172.21.15.52:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f03d8047040 con 0x55f99d9841b0 2024-05-27T07:26:52.469 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:26:52.478+0000 7f03c5ffb640 1 -- 172.21.15.52:0/2950471163 <== mon.0 v2:172.21.15.52:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f03d8003e20 con 0x55f99d9841b0 2024-05-27T07:26:52.469 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:26:52.478+0000 7f03c5ffb640 1 -- 172.21.15.52:0/2950471163 <== mon.0 v2:172.21.15.52:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f03d802f8a0 con 0x55f99d9841b0 2024-05-27T07:26:52.472 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:26:52.478+0000 7f03ed838740 1 -- 172.21.15.52:0/2950471163 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55f99d9841b0 msgr2=0x55f99dd37ca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:26:52.472 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:26:52.478+0000 7f03ed838740 1 --2- 172.21.15.52:0/2950471163 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55f99d9841b0 0x55f99dd37ca0 secure :-1 s=READY pgs=471 cs=0 l=1 rev1=1 crypto rx=0x7f03d8009ca0 tx=0x7f03d80095b0 comp rx=0 tx=0).stop 2024-05-27T07:26:52.473 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:26:52.479+0000 7f03ed838740 1 -- 172.21.15.52:0/2950471163 shutdown_connections 2024-05-27T07:26:52.473 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:26:52.479+0000 7f03ed838740 1 --2- 172.21.15.52:0/2950471163 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55f99d9841b0 0x55f99dd37ca0 unknown :-1 s=CLOSED pgs=471 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:26:52.473 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:26:52.479+0000 7f03ed838740 1 --2- 172.21.15.52:0/2950471163 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55f99d9634d0 0x55f99dd37760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:26:52.473 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:26:52.479+0000 7f03ed838740 1 --2- 172.21.15.52:0/2950471163 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55f99d95b2f0 0x55f99dd31780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:26:52.473 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:26:52.479+0000 7f03ed838740 1 -- 172.21.15.52:0/2950471163 >> 172.21.15.52:0/2950471163 conn(0x55f99d978300 msgr2=0x55f99dd31070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:26:52.473 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:26:52.479+0000 7f03ed838740 1 -- 172.21.15.52:0/2950471163 shutdown_connections 2024-05-27T07:26:52.473 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:26:52.479+0000 7f03ed838740 1 -- 172.21.15.52:0/2950471163 wait complete. 2024-05-27T07:26:52.473 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:26:52.479+0000 7f03ed838740 1 -- start start 2024-05-27T07:26:52.473 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:26:52.479+0000 7f03ed838740 1 Processor -- start 2024-05-27T07:26:52.473 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:26:52.480+0000 7f03ed838740 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55f99d95b2f0 0x55f99dda2c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:26:52.473 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:26:52.480+0000 7f03ed838740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55f99d9634d0 0x55f99dda3170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:26:52.473 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:26:52.480+0000 7f03ed838740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55f99d9841b0 0x55f99dd37130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:26:52.473 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:26:52.480+0000 7f03ed838740 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x7f03e01b47a0 con 0x55f99d95b2f0 2024-05-27T07:26:52.474 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:26:52.480+0000 7f03ed838740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f03e01b40d0 con 0x55f99d9841b0 2024-05-27T07:26:52.474 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:26:52.480+0000 7f03ed838740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f03e01b1310 con 0x55f99d9634d0 2024-05-27T07:26:52.474 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:26:52.480+0000 7f03ed838740 10 client.0 _ll_register_callbacks cb 0x7f03d0009e70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T07:26:52.474 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:26:52.480+0000 7f03dffff640 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55f99d95b2f0 0x55f99dda2c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:26:52.474 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:26:52.480+0000 7f03e4e45640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55f99d9841b0 0x55f99dd37130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:26:52.474 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:26:52.480+0000 7f03dffff640 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55f99d95b2f0 0x55f99dda2c30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.52:37310/0 (socket says 172.21.15.52:37310) 2024-05-27T07:26:52.474 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:26:52.480+0000 7f03dffff640 1 -- 172.21.15.52:0/3031580268 learned_addr learned my addr 172.21.15.52:0/3031580268 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-05-27T07:26:52.474 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:26:52.480+0000 7f03df7fe640 1 --2- 172.21.15.52:0/3031580268 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55f99d9634d0 0x55f99dda3170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:26:52.474 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:26:52.480+0000 7f03e4e45640 1 -- 172.21.15.52:0/3031580268 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55f99d9634d0 msgr2=0x55f99dda3170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:26:52.474 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:26:52.480+0000 7f03e4e45640 1 --2- 172.21.15.52:0/3031580268 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55f99d9634d0 0x55f99dda3170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:26:52.474 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:26:52.480+0000 7f03e4e45640 1 -- 172.21.15.52:0/3031580268 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55f99d95b2f0 msgr2=0x55f99dda2c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:26:52.474 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:26:52.480+0000 7f03e4e45640 1 --2- 172.21.15.52:0/3031580268 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55f99d95b2f0 0x55f99dda2c30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:26:52.475 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:26:52.480+0000 7f03e4e45640 1 -- 172.21.15.52:0/3031580268 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55f99dd267b0 con 0x55f99d9841b0 2024-05-27T07:26:52.475 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:26:52.480+0000 7f03e4e45640 1 --2- 172.21.15.52:0/3031580268 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55f99d9841b0 0x55f99dd37130 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f03d8005e30 tx=0x7f03d8003d70 comp rx=0 tx=0).ready entity=mon.2 client_cookie=1894c2633a7b836 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:26:52.475 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:26:52.480+0000 7f03bf7fe640 10 client.0 ms_handle_connect on v2:172.21.15.105:3300/0 2024-05-27T07:26:52.475 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:26:52.481+0000 7f03bf7fe640 1 -- 172.21.15.52:0/3031580268 <== mon.2 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f03d8033070 con 0x55f99d9841b0 2024-05-27T07:26:52.475 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:26:52.481+0000 7f03bf7fe640 1 -- 172.21.15.52:0/3031580268 <== mon.2 v2:172.21.15.105:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 9379+0+0 (secure 0 0 0) 0x7f03d802e5a0 con 0x55f99d9841b0 2024-05-27T07:26:52.475 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:26:52.481+0000 7f03ed838740 1 -- 172.21.15.52:0/3031580268 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f03d8003e20 con 0x55f99d9841b0 2024-05-27T07:26:52.475 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:26:52.481+0000 7f03bf7fe640 1 -- 172.21.15.52:0/3031580268 <== mon.2 v2:172.21.15.105:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7f03d802eec0 con 0x55f99d9841b0 2024-05-27T07:26:52.475 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:26:52.481+0000 7f03ed838740 10 client.25340 fetch_fsmap learned FSMap version 30 2024-05-27T07:26:52.475 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:26:52.481+0000 7f03ed838740 1 -- 172.21.15.52:0/3031580268 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x55f99ddaa120 con 0x55f99d9841b0 2024-05-27T07:26:52.475 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:26:52.481+0000 7f03bf7fe640 1 -- 172.21.15.52:0/3031580268 <== mon.2 v2:172.21.15.105:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f03d8003e20 con 0x55f99d9841b0 2024-05-27T07:26:52.475 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:26:52.481+0000 7f03bf7fe640 1 -- 172.21.15.52:0/3031580268 <== mon.2 v2:172.21.15.105:3300/0 5 ==== fsmap(e 30) ==== 8362+0+0 (secure 0 0 0) 0x7f03d8004030 con 0x55f99d9841b0 2024-05-27T07:26:52.475 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:26:52.481+0000 7f03ed838740 10 client.25340 fetch_fsmap finished waiting for FSMap version 30 2024-05-27T07:26:52.475 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:26:52.481+0000 7f03ed838740 10 client.25340 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-05-27T07:26:52.476 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:26:52.481+0000 7f03ed838740 20 client.25340 populate_metadata read hostname 'smithi052' 2024-05-27T07:26:52.476 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:26:52.481+0000 7f03ed838740 1 --2- 172.21.15.52:0/3031580268 >> [v2:172.21.15.105:6838/500348151,v1:172.21.15.105:6839/500348151] conn(0x55f99d9181c0 0x55f99ddac360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:26:52.476 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:26:52.481+0000 7f03ed838740 4 client.25340 mds_command: new command op to 24527 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-05-27T07:26:52.476 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:26:52.481+0000 7f03ed838740 1 -- 172.21.15.52:0/3031580268 --> [v2:172.21.15.105:6838/500348151,v1:172.21.15.105:6839/500348151] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55f99dc93360 con 0x55f99d9181c0 2024-05-27T07:26:52.476 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:26:52.482+0000 7f03dffff640 1 --2- 172.21.15.52:0/3031580268 >> [v2:172.21.15.105:6838/500348151,v1:172.21.15.105:6839/500348151] conn(0x55f99d9181c0 0x55f99ddac360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:26:52.476 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:26:52.482+0000 7f03bf7fe640 10 client.25340 ms_handle_connect on v2:172.21.15.105:6838/500348151 2024-05-27T07:26:52.476 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:26:52.482+0000 7f03dffff640 1 --2- 172.21.15.52:0/3031580268 >> [v2:172.21.15.105:6838/500348151,v1:172.21.15.105:6839/500348151] conn(0x55f99d9181c0 0x55f99ddac360 crc :-1 s=READY pgs=53 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=717da9ff1c525d5e server_cookie=a46a4b77eb5aace1 in_seq=0 out_seq=0 2024-05-27T07:26:52.480 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:26:52.489+0000 7f03bf7fe640 1 -- 172.21.15.52:0/3031580268 <== mds.0 v2:172.21.15.105:6838/500348151 1 ==== command_reply(tid 0: 0 ) ==== 8+0+25 (crc 0 0 0) 0x55f99dc93360 con 0x55f99d9181c0 2024-05-27T07:26:52.480 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:26:52.489+0000 7f03bf7fe640 10 client.25340 handle_command_reply: tid=0 2024-05-27T07:26:52.480 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:26:52.489+0000 7f03ed838740 1 client.25340 shutdown 2024-05-27T07:26:52.480 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:26:52.490+0000 7f03ed838740 1 -- 172.21.15.52:0/3031580268 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55f99d9841b0 msgr2=0x55f99dd37130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:26:52.480 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:26:52.490+0000 7f03ed838740 1 --2- 172.21.15.52:0/3031580268 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55f99d9841b0 0x55f99dd37130 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f03d8005e30 tx=0x7f03d8003d70 comp rx=0 tx=0).stop 2024-05-27T07:26:52.480 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:26:52.490+0000 7f03ed838740 1 -- 172.21.15.52:0/3031580268 shutdown_connections 2024-05-27T07:26:52.480 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:26:52.490+0000 7f03ed838740 1 --2- 172.21.15.52:0/3031580268 >> [v2:172.21.15.105:6838/500348151,v1:172.21.15.105:6839/500348151] conn(0x55f99d9181c0 0x55f99ddac360 crc :-1 s=READY pgs=53 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:26:52.480 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:26:52.490+0000 7f03ed838740 1 --2- 172.21.15.52:0/3031580268 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55f99d9841b0 0x55f99dd37130 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:26:52.480 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:26:52.490+0000 7f03ed838740 1 --2- 172.21.15.52:0/3031580268 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55f99d9634d0 0x55f99dda3170 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:26:52.480 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:26:52.490+0000 7f03ed838740 1 --2- 172.21.15.52:0/3031580268 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55f99d95b2f0 0x55f99dda2c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:26:52.480 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:26:52.490+0000 7f03ed838740 1 -- 172.21.15.52:0/3031580268 >> 172.21.15.52:0/3031580268 conn(0x55f99d978300 msgr2=0x55f99dd30ca0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:26:52.480 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:26:52.490+0000 7f03bf7fe640 0 client.25340 ms_handle_reset on v2:172.21.15.105:6838/500348151 2024-05-27T07:26:52.481 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:26:52.490+0000 7f03ed838740 1 -- 172.21.15.52:0/3031580268 shutdown_connections 2024-05-27T07:26:52.481 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:26:52.490+0000 7f03ed838740 1 -- 172.21.15.52:0/3031580268 wait complete. 2024-05-27T07:26:52.481 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:26:52.490+0000 7f03ed838740 20 client.25340 trim_cache size 0 max 16384 2024-05-27T07:26:52.481 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stdout:{ 2024-05-27T07:26:52.481 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stdout: "return_code": 0 2024-05-27T07:26:52.481 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stdout:} 2024-05-27T07:26:52.482 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:26:52.491+0000 7f03e68a8640 1 -- 172.21.15.52:0/293617107 >> [v2:172.21.15.52:6800/1822318384,v1:172.21.15.52:6801/1822318384] conn(0x7f03c80b2a70 msgr2=0x7f03c80b4f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:26:52.482 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:26:52.491+0000 7f03e68a8640 1 --2- 172.21.15.52:0/293617107 >> [v2:172.21.15.52:6800/1822318384,v1:172.21.15.52:6801/1822318384] conn(0x7f03c80b2a70 0x7f03c80b4f80 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x55f99dd26780 tx=0x7f03d00094a0 comp rx=0 tx=0).stop 2024-05-27T07:26:52.482 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:26:52.491+0000 7f03e68a8640 1 -- 172.21.15.52:0/293617107 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f03e010b850 msgr2=0x7f03e0102bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:26:52.482 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:26:52.491+0000 7f03e68a8640 1 --2- 172.21.15.52:0/293617107 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f03e010b850 0x7f03e0102bf0 secure :-1 s=READY pgs=468 cs=0 l=1 rev1=1 crypto rx=0x7f03d400e9f0 tx=0x7f03d400edf0 comp rx=0 tx=0).stop 2024-05-27T07:26:52.482 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:26:52.491+0000 7f03e68a8640 1 -- 172.21.15.52:0/293617107 shutdown_connections 2024-05-27T07:26:52.482 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:26:52.491+0000 7f03e68a8640 1 --2- 172.21.15.52:0/293617107 >> [v2:172.21.15.52:6800/1822318384,v1:172.21.15.52:6801/1822318384] conn(0x7f03c80b2a70 0x7f03c80b4f80 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:26:52.483 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:26:52.491+0000 7f03e68a8640 1 --2- 172.21.15.52:0/293617107 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f03e010e200 0x7f03e0103130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:26:52.483 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:26:52.491+0000 7f03e68a8640 1 --2- 172.21.15.52:0/293617107 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f03e010b850 0x7f03e0102bf0 unknown :-1 s=CLOSED pgs=468 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:26:52.483 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:26:52.491+0000 7f03e68a8640 1 --2- 172.21.15.52:0/293617107 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f03e0104df0 0x7f03e01026b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:26:52.483 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:26:52.491+0000 7f03e68a8640 1 -- 172.21.15.52:0/293617107 >> 172.21.15.52:0/293617107 conn(0x7f03e00782d0 msgr2=0x7f03e01012b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:26:52.483 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:26:52.491+0000 7f03e68a8640 1 -- 172.21.15.52:0/293617107 shutdown_connections 2024-05-27T07:26:52.483 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:26:52.491+0000 7f03e68a8640 1 -- 172.21.15.52:0/293617107 wait complete. 2024-05-27T07:26:52.538 INFO:teuthology.orchestra.run.smithi052.stderr:Inferring config /var/lib/ceph/73df2636-1bf8-11ef-bc9b-c7b262605968/mon.a/config 2024-05-27T07:26:54.313 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:54.323+0000 7f0e1336c700 1 -- 172.21.15.52:0/296030940 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f0e0c108750 msgr2=0x7f0e0c1100b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:26:54.313 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:54.323+0000 7f0e1336c700 1 --2- 172.21.15.52:0/296030940 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f0e0c108750 0x7f0e0c1100b0 secure :-1 s=READY pgs=472 cs=0 l=1 rev1=1 crypto rx=0x7f0e0400b0d0 tx=0x7f0e0400b3e0 comp rx=0 tx=0).stop 2024-05-27T07:26:54.313 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:26:54 smithi052 ceph-mon[31955]: pgmap v31: 129 pgs: 129 active+clean; 1.3 MiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 7.2 KiB/s rd, 1.9 KiB/s wr, 14 op/s 2024-05-27T07:26:54.315 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:54.323+0000 7f0e1336c700 1 -- 172.21.15.52:0/296030940 shutdown_connections 2024-05-27T07:26:54.315 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:54.323+0000 7f0e1336c700 1 --2- 172.21.15.52:0/296030940 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f0e0c108750 0x7f0e0c1100b0 unknown :-1 s=CLOSED pgs=472 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:26:54.315 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:54.323+0000 7f0e1336c700 1 --2- 172.21.15.52:0/296030940 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f0e0c1034d0 0x7f0e0c1038e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:26:54.315 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:54.323+0000 7f0e1336c700 1 --2- 172.21.15.52:0/296030940 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f0e0c107e00 0x7f0e0c108210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:26:54.315 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:54.323+0000 7f0e1336c700 1 -- 172.21.15.52:0/296030940 >> 172.21.15.52:0/296030940 conn(0x7f0e0c0fef20 msgr2=0x7f0e0c101340 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:26:54.315 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:54.323+0000 7f0e1336c700 1 -- 172.21.15.52:0/296030940 shutdown_connections 2024-05-27T07:26:54.315 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:54.323+0000 7f0e1336c700 1 -- 172.21.15.52:0/296030940 wait complete. 2024-05-27T07:26:54.315 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:54.323+0000 7f0e1336c700 1 Processor -- start 2024-05-27T07:26:54.315 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:54.324+0000 7f0e1336c700 1 -- start start 2024-05-27T07:26:54.315 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:54.324+0000 7f0e1336c700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f0e0c1034d0 0x7f0e0c19d0b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:26:54.315 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:54.324+0000 7f0e1336c700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f0e0c107e00 0x7f0e0c19d650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:26:54.316 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:54.324+0000 7f0e1336c700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f0e0c108750 0x7f0e0c1a18e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:26:54.316 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:54.324+0000 7f0e1336c700 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x7f0e0c113440 con 0x7f0e0c1034d0 2024-05-27T07:26:54.316 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:54.324+0000 7f0e1336c700 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f0e0c1132c0 con 0x7f0e0c107e00 2024-05-27T07:26:54.316 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:54.324+0000 7f0e1336c700 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f0e0c1135c0 con 0x7f0e0c108750 2024-05-27T07:26:54.316 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:54.324+0000 7f0e11108700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f0e0c1034d0 0x7f0e0c19d0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:26:54.316 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:54.324+0000 7f0e11108700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f0e0c1034d0 0x7f0e0c19d0b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.52:37324/0 (socket says 172.21.15.52:37324) 2024-05-27T07:26:54.316 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:54.324+0000 7f0e11108700 1 -- 172.21.15.52:0/2191232058 learned_addr learned my addr 172.21.15.52:0/2191232058 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-05-27T07:26:54.316 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:54.324+0000 7f0e10907700 1 --2- 172.21.15.52:0/2191232058 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f0e0c107e00 0x7f0e0c19d650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:26:54.316 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:54.324+0000 7f0e11909700 1 --2- 172.21.15.52:0/2191232058 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f0e0c108750 0x7f0e0c1a18e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:26:54.316 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:54.324+0000 7f0e11108700 1 -- 172.21.15.52:0/2191232058 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f0e0c108750 msgr2=0x7f0e0c1a18e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:26:54.316 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:54.324+0000 7f0e11108700 1 --2- 172.21.15.52:0/2191232058 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f0e0c108750 0x7f0e0c1a18e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:26:54.316 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:54.324+0000 7f0e11108700 1 -- 172.21.15.52:0/2191232058 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f0e0c107e00 msgr2=0x7f0e0c19d650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:26:54.316 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:54.324+0000 7f0e11108700 1 --2- 172.21.15.52:0/2191232058 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f0e0c107e00 0x7f0e0c19d650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:26:54.317 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:54.324+0000 7f0e11108700 1 -- 172.21.15.52:0/2191232058 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0e0c1a1f80 con 0x7f0e0c1034d0 2024-05-27T07:26:54.317 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:54.324+0000 7f0e11909700 1 --2- 172.21.15.52:0/2191232058 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f0e0c108750 0x7f0e0c1a18e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-27T07:26:54.317 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:54.325+0000 7f0e10907700 1 --2- 172.21.15.52:0/2191232058 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f0e0c107e00 0x7f0e0c19d650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-27T07:26:54.317 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:54.325+0000 7f0e11108700 1 --2- 172.21.15.52:0/2191232058 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f0e0c1034d0 0x7f0e0c19d0b0 secure :-1 s=READY pgs=473 cs=0 l=1 rev1=1 crypto rx=0x7f0e0800c920 tx=0x7f0e0800cce0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:26:54.317 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:54.325+0000 7f0e027fc700 1 -- 172.21.15.52:0/2191232058 <== mon.0 v2:172.21.15.52:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0e08007b00 con 0x7f0e0c1034d0 2024-05-27T07:26:54.317 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:54.325+0000 7f0e1336c700 1 -- 172.21.15.52:0/2191232058 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0e0c1a2270 con 0x7f0e0c1034d0 2024-05-27T07:26:54.317 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:54.325+0000 7f0e027fc700 1 -- 172.21.15.52:0/2191232058 <== mon.0 v2:172.21.15.52:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f0e08004ce0 con 0x7f0e0c1034d0 2024-05-27T07:26:54.317 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:54.325+0000 7f0e027fc700 1 -- 172.21.15.52:0/2191232058 <== mon.0 v2:172.21.15.52:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0e08005650 con 0x7f0e0c1034d0 2024-05-27T07:26:54.317 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:54.325+0000 7f0e1336c700 1 -- 172.21.15.52:0/2191232058 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0e0c1a9b90 con 0x7f0e0c1034d0 2024-05-27T07:26:54.318 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:54.329+0000 7f0e027fc700 1 -- 172.21.15.52:0/2191232058 <== mon.0 v2:172.21.15.52:3300/0 4 ==== mgrmap(e 24) ==== 149879+0+0 (secure 0 0 0) 0x7f0e0800f450 con 0x7f0e0c1034d0 2024-05-27T07:26:54.321 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:54.330+0000 7f0e21d81b80 1 Processor -- start 2024-05-27T07:26:54.321 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:54.330+0000 7f0e21d81b80 1 -- start start 2024-05-27T07:26:54.321 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:54.330+0000 7f0e027fc700 1 --2- 172.21.15.52:0/2191232058 >> [v2:172.21.15.52:6800/1822318384,v1:172.21.15.52:6801/1822318384] conn(0x7f0df80b2b10 0x7f0df80b5020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:26:54.321 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:54.330+0000 7f0e21d81b80 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5580b2d3ce40 0x5580b2ec1b30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:26:54.321 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:54.330+0000 7f0e21d81b80 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5580b2ec2070 0x5580b2ebbbe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:26:54.321 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:54.330+0000 7f0e21d81b80 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x5580b2ebc120 0x5580b2ebc530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:26:54.321 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:54.330+0000 7f0e21d81b80 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x5580b2de9d70 con 0x5580b2ebc120 2024-05-27T07:26:54.321 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:54.330+0000 7f0e21d81b80 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x5580b2dd9910 con 0x5580b2d3ce40 2024-05-27T07:26:54.321 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:54.330+0000 7f0e21d81b80 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x5580b2ddd950 con 0x5580b2ec2070 2024-05-27T07:26:54.322 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:54.330+0000 7f0e11909700 1 --2- 172.21.15.52:0/2191232058 >> [v2:172.21.15.52:6800/1822318384,v1:172.21.15.52:6801/1822318384] conn(0x7f0df80b2b10 0x7f0df80b5020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:26:54.322 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:54.330+0000 7f0e027fc700 1 -- 172.21.15.52:0/2191232058 <== mon.0 v2:172.21.15.52:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9379+0+0 (secure 0 0 0) 0x7f0e080e2c30 con 0x7f0e0c1034d0 2024-05-27T07:26:54.322 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:54.330+0000 7f0e11909700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x5580b2ebc120 0x5580b2ebc530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:26:54.322 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:54.330+0000 7f0e11909700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x5580b2ebc120 0x5580b2ebc530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.52:37336/0 (socket says 172.21.15.52:37336) 2024-05-27T07:26:54.322 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:54.330+0000 7f0e11909700 1 -- 172.21.15.52:0/3206395451 learned_addr learned my addr 172.21.15.52:0/3206395451 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-05-27T07:26:54.322 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:54.330+0000 7f0e10907700 1 --2- 172.21.15.52:0/3206395451 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5580b2ec2070 0x5580b2ebbbe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:26:54.322 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:54.330+0000 7f0e11108700 1 --2- 172.21.15.52:0/3206395451 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5580b2d3ce40 0x5580b2ec1b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:26:54.322 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:54.331+0000 7f0e11909700 1 -- 172.21.15.52:0/3206395451 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5580b2ec2070 msgr2=0x5580b2ebbbe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:26:54.322 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:54.331+0000 7f0e11909700 1 --2- 172.21.15.52:0/3206395451 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5580b2ec2070 0x5580b2ebbbe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:26:54.322 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:54.331+0000 7f0e11909700 1 -- 172.21.15.52:0/3206395451 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5580b2d3ce40 msgr2=0x5580b2ec1b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:26:54.322 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:54.331+0000 7f0e11909700 1 --2- 172.21.15.52:0/3206395451 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5580b2d3ce40 0x5580b2ec1b30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:26:54.322 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:54.331+0000 7f0e11909700 1 -- 172.21.15.52:0/3206395451 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5580b2d82fb0 con 0x5580b2ebc120 2024-05-27T07:26:54.323 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:54.331+0000 7f0e11909700 1 --2- 172.21.15.52:0/3206395451 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x5580b2ebc120 0x5580b2ebc530 secure :-1 s=READY pgs=474 cs=0 l=1 rev1=1 crypto rx=0x7f0e04015000 tx=0x7f0e04007910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e58387dda68f5c16 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:26:54.323 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:54.331+0000 7f0e11909700 1 --2- 172.21.15.52:0/2191232058 >> [v2:172.21.15.52:6800/1822318384,v1:172.21.15.52:6801/1822318384] conn(0x7f0df80b2b10 0x7f0df80b5020 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f0e04007980 tx=0x7f0e0401b000 comp rx=0 tx=0).ready entity=mgr.15333 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:26:54.323 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:54.331+0000 7f0df77fe700 1 -- 172.21.15.52:0/3206395451 <== mon.0 v2:172.21.15.52:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0e04005a80 con 0x5580b2ebc120 2024-05-27T07:26:54.323 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:54.331+0000 7f0df77fe700 1 -- 172.21.15.52:0/3206395451 <== mon.0 v2:172.21.15.52:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f0e0402f070 con 0x5580b2ebc120 2024-05-27T07:26:54.323 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:54.331+0000 7f0df77fe700 1 -- 172.21.15.52:0/3206395451 <== mon.0 v2:172.21.15.52:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0e0402a950 con 0x5580b2ebc120 2024-05-27T07:26:54.326 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:54.332+0000 7f0e21d81b80 1 -- 172.21.15.52:0/3206395451 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x5580b2ebc120 msgr2=0x5580b2ebc530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:26:54.326 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:54.332+0000 7f0e21d81b80 1 --2- 172.21.15.52:0/3206395451 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x5580b2ebc120 0x5580b2ebc530 secure :-1 s=READY pgs=474 cs=0 l=1 rev1=1 crypto rx=0x7f0e04015000 tx=0x7f0e04007910 comp rx=0 tx=0).stop 2024-05-27T07:26:54.326 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:54.332+0000 7f0e21d81b80 1 -- 172.21.15.52:0/3206395451 shutdown_connections 2024-05-27T07:26:54.326 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:54.332+0000 7f0e21d81b80 1 --2- 172.21.15.52:0/3206395451 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x5580b2ebc120 0x5580b2ebc530 unknown :-1 s=CLOSED pgs=474 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:26:54.326 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:54.332+0000 7f0e21d81b80 1 --2- 172.21.15.52:0/3206395451 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5580b2d3ce40 0x5580b2ec1b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:26:54.326 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:54.332+0000 7f0e21d81b80 1 --2- 172.21.15.52:0/3206395451 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5580b2ec2070 0x5580b2ebbbe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:26:54.327 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:54.332+0000 7f0e21d81b80 1 -- 172.21.15.52:0/3206395451 >> 172.21.15.52:0/3206395451 conn(0x5580b2d3f770 msgr2=0x5580b2eb6d90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:26:54.327 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:54.332+0000 7f0e21d81b80 1 -- 172.21.15.52:0/3206395451 shutdown_connections 2024-05-27T07:26:54.327 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:54.332+0000 7f0e21d81b80 1 -- 172.21.15.52:0/3206395451 wait complete. 2024-05-27T07:26:54.327 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:54.333+0000 7f0e21d81b80 1 -- start start 2024-05-27T07:26:54.327 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:54.333+0000 7f0e21d81b80 1 Processor -- start 2024-05-27T07:26:54.327 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:54.333+0000 7f0e21d81b80 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5580b2d3ce40 0x5580b2ec58d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:26:54.327 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:54.333+0000 7f0e21d81b80 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x5580b2ebc120 0x5580b2ec5e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:26:54.327 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:54.333+0000 7f0e21d81b80 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5580b2ec2070 0x5580b2ec6350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:26:54.327 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:54.333+0000 7f0e21d81b80 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x5580b2cdffa0 con 0x5580b2ebc120 2024-05-27T07:26:54.327 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:54.333+0000 7f0e21d81b80 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x5580b2d41da0 con 0x5580b2ec2070 2024-05-27T07:26:54.327 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:54.333+0000 7f0e21d81b80 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x5580b2ec8b00 con 0x5580b2d3ce40 2024-05-27T07:26:54.327 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:54.334+0000 7f0e21d81b80 10 client.0 _ll_register_callbacks cb 0x5580b2d705b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T07:26:54.328 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:54.334+0000 7f0e10907700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x5580b2ebc120 0x5580b2ec5e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:26:54.328 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:54.334+0000 7f0e10907700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x5580b2ebc120 0x5580b2ec5e10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.52:37350/0 (socket says 172.21.15.52:37350) 2024-05-27T07:26:54.328 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:54.334+0000 7f0e10907700 1 -- 172.21.15.52:0/2193601026 learned_addr learned my addr 172.21.15.52:0/2193601026 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-05-27T07:26:54.328 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:54.334+0000 7f0e11909700 1 --2- 172.21.15.52:0/2193601026 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5580b2ec2070 0x5580b2ec6350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:26:54.328 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:54.334+0000 7f0e11108700 1 --2- 172.21.15.52:0/2193601026 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5580b2d3ce40 0x5580b2ec58d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:26:54.328 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:54.334+0000 7f0e10907700 1 -- 172.21.15.52:0/2193601026 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5580b2d3ce40 msgr2=0x5580b2ec58d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:26:54.328 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:54.334+0000 7f0e10907700 1 --2- 172.21.15.52:0/2193601026 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5580b2d3ce40 0x5580b2ec58d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:26:54.328 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:54.334+0000 7f0e10907700 1 -- 172.21.15.52:0/2193601026 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5580b2ec2070 msgr2=0x5580b2ec6350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:26:54.328 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:54.334+0000 7f0e10907700 1 --2- 172.21.15.52:0/2193601026 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5580b2ec2070 0x5580b2ec6350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:26:54.328 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:54.334+0000 7f0e10907700 1 -- 172.21.15.52:0/2193601026 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5580b2d298d0 con 0x5580b2ebc120 2024-05-27T07:26:54.328 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:54.334+0000 7f0e11909700 1 --2- 172.21.15.52:0/2193601026 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5580b2ec2070 0x5580b2ec6350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-27T07:26:54.328 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:54.334+0000 7f0e11108700 1 --2- 172.21.15.52:0/2193601026 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5580b2d3ce40 0x5580b2ec58d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-27T07:26:54.328 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:54.334+0000 7f0e10907700 1 --2- 172.21.15.52:0/2193601026 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x5580b2ebc120 0x5580b2ec5e10 secure :-1 s=READY pgs=475 cs=0 l=1 rev1=1 crypto rx=0x7f0dfc010500 tx=0x7f0dfc0108c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fc0ed76ff48c6b0f server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:26:54.329 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:54.334+0000 7f0df4ff9700 10 client.0 ms_handle_connect on v2:172.21.15.52:3300/0 2024-05-27T07:26:54.329 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:54.334+0000 7f0df4ff9700 1 -- 172.21.15.52:0/2193601026 <== mon.0 v2:172.21.15.52:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0dfc018070 con 0x5580b2ebc120 2024-05-27T07:26:54.329 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:54.334+0000 7f0df4ff9700 1 -- 172.21.15.52:0/2193601026 <== mon.0 v2:172.21.15.52:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 9379+0+0 (secure 0 0 0) 0x7f0dfc014640 con 0x5580b2ebc120 2024-05-27T07:26:54.329 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:54.334+0000 7f0e21d81b80 1 -- 172.21.15.52:0/2193601026 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f0e0402f070 con 0x5580b2ebc120 2024-05-27T07:26:54.329 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:54.335+0000 7f0df4ff9700 1 -- 172.21.15.52:0/2193601026 <== mon.0 v2:172.21.15.52:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7f0dfc024050 con 0x5580b2ebc120 2024-05-27T07:26:54.329 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:54.335+0000 7f0e21d81b80 10 client.15633 fetch_fsmap learned FSMap version 30 2024-05-27T07:26:54.329 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:54.335+0000 7f0e21d81b80 1 -- 172.21.15.52:0/2193601026 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x5580b2d93760 con 0x5580b2ebc120 2024-05-27T07:26:54.329 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:54.335+0000 7f0df4ff9700 1 -- 172.21.15.52:0/2193601026 <== mon.0 v2:172.21.15.52:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f0e0402f070 con 0x5580b2ebc120 2024-05-27T07:26:54.329 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:54.335+0000 7f0df4ff9700 1 -- 172.21.15.52:0/2193601026 <== mon.0 v2:172.21.15.52:3300/0 5 ==== fsmap(e 30) ==== 8362+0+0 (secure 0 0 0) 0x7f0dfc013930 con 0x5580b2ebc120 2024-05-27T07:26:54.329 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:54.335+0000 7f0e21d81b80 10 client.15633 fetch_fsmap finished waiting for FSMap version 30 2024-05-27T07:26:54.329 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:54.335+0000 7f0e21d81b80 10 client.15633 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.c 2024-05-27T07:26:54.329 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:54.335+0000 7f0e21d81b80 20 client.15633 populate_metadata read hostname 'smithi052' 2024-05-27T07:26:54.330 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:54.335+0000 7f0e21d81b80 1 --2- 172.21.15.52:0/2193601026 >> [v2:172.21.15.173:6836/1396363827,v1:172.21.15.173:6837/1396363827] conn(0x5580b2ed5a70 0x5580b2ed7e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:26:54.330 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:54.335+0000 7f0e21d81b80 4 client.15633 mds_command: new command op to 24466 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-27T07:26:54.330 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:54.335+0000 7f0e21d81b80 1 -- 172.21.15.52:0/2193601026 --> [v2:172.21.15.173:6836/1396363827,v1:172.21.15.173:6837/1396363827] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5580b2debcc0 con 0x5580b2ed5a70 2024-05-27T07:26:54.330 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:54.336+0000 7f0e11108700 1 --2- 172.21.15.52:0/2193601026 >> [v2:172.21.15.173:6836/1396363827,v1:172.21.15.173:6837/1396363827] conn(0x5580b2ed5a70 0x5580b2ed7e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:26:54.330 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:54.336+0000 7f0df4ff9700 10 client.15633 ms_handle_connect on v2:172.21.15.173:6836/1396363827 2024-05-27T07:26:54.330 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:54.336+0000 7f0e11108700 1 --2- 172.21.15.52:0/2193601026 >> [v2:172.21.15.173:6836/1396363827,v1:172.21.15.173:6837/1396363827] conn(0x5580b2ed5a70 0x5580b2ed7e90 crc :-1 s=READY pgs=27 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=20abb1d2cda0c328 server_cookie=4e2f0592a972eaa4 in_seq=0 out_seq=0 2024-05-27T07:26:54.331 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:54.337+0000 7f0df4ff9700 1 -- 172.21.15.52:0/2193601026 <== mds.4 v2:172.21.15.173:6836/1396363827 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24041 (crc 0 0 0) 0x5580b2debcc0 con 0x5580b2ed5a70 2024-05-27T07:26:54.331 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:54.337+0000 7f0df4ff9700 10 client.15633 handle_command_reply: tid=0 2024-05-27T07:26:54.331 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:54.337+0000 7f0e21d81b80 1 client.15633 shutdown 2024-05-27T07:26:54.332 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:54.337+0000 7f0e21d81b80 1 -- 172.21.15.52:0/2193601026 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x5580b2ebc120 msgr2=0x5580b2ec5e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:26:54.332 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:54.337+0000 7f0e21d81b80 1 --2- 172.21.15.52:0/2193601026 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x5580b2ebc120 0x5580b2ec5e10 secure :-1 s=READY pgs=475 cs=0 l=1 rev1=1 crypto rx=0x7f0dfc010500 tx=0x7f0dfc0108c0 comp rx=0 tx=0).stop 2024-05-27T07:26:54.332 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:54.337+0000 7f0e21d81b80 1 -- 172.21.15.52:0/2193601026 shutdown_connections 2024-05-27T07:26:54.332 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:54.337+0000 7f0e21d81b80 1 --2- 172.21.15.52:0/2193601026 >> [v2:172.21.15.173:6836/1396363827,v1:172.21.15.173:6837/1396363827] conn(0x5580b2ed5a70 0x5580b2ed7e90 crc :-1 s=READY pgs=27 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:26:54.332 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:54.337+0000 7f0e21d81b80 1 --2- 172.21.15.52:0/2193601026 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5580b2ec2070 0x5580b2ec6350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:26:54.332 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:54.337+0000 7f0df4ff9700 0 client.15633 ms_handle_reset on v2:172.21.15.173:6836/1396363827 2024-05-27T07:26:54.332 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:54.337+0000 7f0e21d81b80 1 --2- 172.21.15.52:0/2193601026 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5580b2d3ce40 0x5580b2ec58d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:26:54.332 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:54.337+0000 7f0e21d81b80 1 --2- 172.21.15.52:0/2193601026 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x5580b2ebc120 0x5580b2ec5e10 unknown :-1 s=CLOSED pgs=475 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:26:54.332 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:54.337+0000 7f0e21d81b80 1 -- 172.21.15.52:0/2193601026 >> 172.21.15.52:0/2193601026 conn(0x5580b2d3f770 msgr2=0x5580b2ebabd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:26:54.332 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:54.338+0000 7f0e21d81b80 1 -- 172.21.15.52:0/2193601026 shutdown_connections 2024-05-27T07:26:54.332 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:54.338+0000 7f0e21d81b80 1 -- 172.21.15.52:0/2193601026 wait complete. 2024-05-27T07:26:54.332 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:54.338+0000 7f0e21d81b80 20 client.15633 trim_cache size 0 max 16384 2024-05-27T07:26:54.342 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:54.353+0000 7f0e21d81b80 1 Processor -- start 2024-05-27T07:26:54.343 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:54.353+0000 7f0e21d81b80 1 -- start start 2024-05-27T07:26:54.343 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:54.353+0000 7f0e21d81b80 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x5580b2d3ce40 0x5580b2ebbbe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:26:54.343 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:54.353+0000 7f0e21d81b80 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5580b2ebc120 0x5580b2eb8260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:26:54.343 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:54.353+0000 7f0e21d81b80 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5580b2ec2070 0x5580b2ebfba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:26:54.343 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:54.353+0000 7f0e21d81b80 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x7f0e0c1b26f0 con 0x5580b2d3ce40 2024-05-27T07:26:54.343 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:54.353+0000 7f0e21d81b80 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f0e0c1b4230 con 0x5580b2ec2070 2024-05-27T07:26:54.343 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:54.353+0000 7f0e21d81b80 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f0e0c1b2dc0 con 0x5580b2ebc120 2024-05-27T07:26:54.345 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:54.353+0000 7f0e11108700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x5580b2d3ce40 0x5580b2ebbbe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:26:54.345 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:54.353+0000 7f0e11108700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x5580b2d3ce40 0x5580b2ebbbe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.52:37352/0 (socket says 172.21.15.52:37352) 2024-05-27T07:26:54.345 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:54.353+0000 7f0e11108700 1 -- 172.21.15.52:0/1161757552 learned_addr learned my addr 172.21.15.52:0/1161757552 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-05-27T07:26:54.345 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:54.354+0000 7f0e11108700 1 -- 172.21.15.52:0/1161757552 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5580b2ebc120 msgr2=0x5580b2eb8260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:26:54.345 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:54.354+0000 7f0e10907700 1 --2- 172.21.15.52:0/1161757552 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5580b2ebc120 0x5580b2eb8260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:26:54.345 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:54.354+0000 7f0e11108700 1 --2- 172.21.15.52:0/1161757552 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5580b2ebc120 0x5580b2eb8260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:26:54.345 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:54.354+0000 7f0e11108700 1 -- 172.21.15.52:0/1161757552 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5580b2ec2070 msgr2=0x5580b2ebfba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:26:54.345 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:54.354+0000 7f0e11108700 1 --2- 172.21.15.52:0/1161757552 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5580b2ec2070 0x5580b2ebfba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:26:54.345 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:54.354+0000 7f0e11108700 1 -- 172.21.15.52:0/1161757552 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5580b2ec3e40 con 0x5580b2d3ce40 2024-05-27T07:26:54.346 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:54.354+0000 7f0e10907700 1 --2- 172.21.15.52:0/1161757552 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5580b2ebc120 0x5580b2eb8260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-27T07:26:54.346 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:54.354+0000 7f0e11108700 1 --2- 172.21.15.52:0/1161757552 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x5580b2d3ce40 0x5580b2ebbbe0 secure :-1 s=READY pgs=476 cs=0 l=1 rev1=1 crypto rx=0x7f0e08007fd0 tx=0x7f0e080cc040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8fad5701bba349da server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:26:54.346 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:54.354+0000 7f0df6ffd700 1 -- 172.21.15.52:0/1161757552 <== mon.0 v2:172.21.15.52:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0e0800eec0 con 0x5580b2d3ce40 2024-05-27T07:26:54.346 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:54.354+0000 7f0df6ffd700 1 -- 172.21.15.52:0/1161757552 <== mon.0 v2:172.21.15.52:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f0e080b20a0 con 0x5580b2d3ce40 2024-05-27T07:26:54.346 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:54.354+0000 7f0df6ffd700 1 -- 172.21.15.52:0/1161757552 <== mon.0 v2:172.21.15.52:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0e080b2760 con 0x5580b2d3ce40 2024-05-27T07:26:54.346 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:54.355+0000 7f0e21d81b80 1 -- 172.21.15.52:0/1161757552 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x5580b2d3ce40 msgr2=0x5580b2ebbbe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:26:54.346 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:54.355+0000 7f0e21d81b80 1 --2- 172.21.15.52:0/1161757552 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x5580b2d3ce40 0x5580b2ebbbe0 secure :-1 s=READY pgs=476 cs=0 l=1 rev1=1 crypto rx=0x7f0e08007fd0 tx=0x7f0e080cc040 comp rx=0 tx=0).stop 2024-05-27T07:26:54.346 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:54.355+0000 7f0e21d81b80 1 -- 172.21.15.52:0/1161757552 shutdown_connections 2024-05-27T07:26:54.346 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:54.355+0000 7f0e21d81b80 1 --2- 172.21.15.52:0/1161757552 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5580b2ec2070 0x5580b2ebfba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:26:54.346 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:54.355+0000 7f0e21d81b80 1 --2- 172.21.15.52:0/1161757552 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x5580b2d3ce40 0x5580b2ebbbe0 unknown :-1 s=CLOSED pgs=476 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:26:54.346 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:54.355+0000 7f0e21d81b80 1 --2- 172.21.15.52:0/1161757552 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5580b2ebc120 0x5580b2eb8260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:26:54.346 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:54.355+0000 7f0e21d81b80 1 -- 172.21.15.52:0/1161757552 >> 172.21.15.52:0/1161757552 conn(0x5580b2d3f770 msgr2=0x5580b2ebf490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:26:54.346 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:54.355+0000 7f0e21d81b80 1 -- 172.21.15.52:0/1161757552 shutdown_connections 2024-05-27T07:26:54.347 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:54.355+0000 7f0e21d81b80 1 -- 172.21.15.52:0/1161757552 wait complete. 2024-05-27T07:26:54.347 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:54.355+0000 7f0e21d81b80 1 -- start start 2024-05-27T07:26:54.347 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:54.355+0000 7f0e21d81b80 1 Processor -- start 2024-05-27T07:26:54.350 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:54.356+0000 7f0e21d81b80 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x5580b2d3ce40 0x5580b2ecdf90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:26:54.350 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:54.356+0000 7f0e21d81b80 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5580b2ebc120 0x5580b2ece4d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:26:54.350 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:54.356+0000 7f0e21d81b80 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5580b2ec2070 0x5580b2ebabd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:26:54.350 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:54.356+0000 7f0e21d81b80 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x7f0e0c1b3b60 con 0x5580b2d3ce40 2024-05-27T07:26:54.350 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:54.356+0000 7f0e21d81b80 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f0e0c1b3490 con 0x5580b2ec2070 2024-05-27T07:26:54.350 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:54.356+0000 7f0e21d81b80 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f0e0c1b0690 con 0x5580b2ebc120 2024-05-27T07:26:54.350 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:54.356+0000 7f0e21d81b80 10 client.0 _ll_register_callbacks cb 0x5580b2d705b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T07:26:54.350 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:54.356+0000 7f0e11108700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x5580b2d3ce40 0x5580b2ecdf90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:26:54.351 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:54.356+0000 7f0e11108700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x5580b2d3ce40 0x5580b2ecdf90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.52:37362/0 (socket says 172.21.15.52:37362) 2024-05-27T07:26:54.351 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:54.356+0000 7f0e11108700 1 -- 172.21.15.52:0/1397217571 learned_addr learned my addr 172.21.15.52:0/1397217571 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-05-27T07:26:54.351 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:54.356+0000 7f0e11909700 1 --2- 172.21.15.52:0/1397217571 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5580b2ec2070 0x5580b2ebabd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:26:54.351 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:54.356+0000 7f0e10907700 1 --2- 172.21.15.52:0/1397217571 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5580b2ebc120 0x5580b2ece4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:26:54.351 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:54.356+0000 7f0e11108700 1 -- 172.21.15.52:0/1397217571 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5580b2ebc120 msgr2=0x5580b2ece4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:26:54.351 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:54.356+0000 7f0e11108700 1 --2- 172.21.15.52:0/1397217571 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5580b2ebc120 0x5580b2ece4d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:26:54.351 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:54.356+0000 7f0e11108700 1 -- 172.21.15.52:0/1397217571 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5580b2ec2070 msgr2=0x5580b2ebabd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:26:54.351 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:54.356+0000 7f0e11108700 1 --2- 172.21.15.52:0/1397217571 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5580b2ec2070 0x5580b2ebabd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:26:54.351 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:54.356+0000 7f0e11108700 1 -- 172.21.15.52:0/1397217571 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5580b2ebb350 con 0x5580b2d3ce40 2024-05-27T07:26:54.351 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:54.356+0000 7f0e10907700 1 --2- 172.21.15.52:0/1397217571 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5580b2ebc120 0x5580b2ece4d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-27T07:26:54.351 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:54.356+0000 7f0e11909700 1 --2- 172.21.15.52:0/1397217571 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5580b2ec2070 0x5580b2ebabd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-27T07:26:54.351 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:54.356+0000 7f0e11108700 1 --2- 172.21.15.52:0/1397217571 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x5580b2d3ce40 0x5580b2ecdf90 secure :-1 s=READY pgs=477 cs=0 l=1 rev1=1 crypto rx=0x7f0e080059d0 tx=0x7f0e080050b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d119da9eac5bb44a server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:26:54.352 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:54.356+0000 7f0df4ff9700 10 client.0 ms_handle_connect on v2:172.21.15.52:3300/0 2024-05-27T07:26:54.352 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:54.357+0000 7f0df4ff9700 1 -- 172.21.15.52:0/1397217571 <== mon.0 v2:172.21.15.52:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0e080b2760 con 0x5580b2d3ce40 2024-05-27T07:26:54.352 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:54.357+0000 7f0e21d81b80 1 -- 172.21.15.52:0/1397217571 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x5580b2de54a0 con 0x5580b2d3ce40 2024-05-27T07:26:54.352 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:54.357+0000 7f0df4ff9700 1 -- 172.21.15.52:0/1397217571 <== mon.0 v2:172.21.15.52:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 9379+0+0 (secure 0 0 0) 0x7f0e08004e90 con 0x5580b2d3ce40 2024-05-27T07:26:54.352 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:54.357+0000 7f0df4ff9700 1 -- 172.21.15.52:0/1397217571 <== mon.0 v2:172.21.15.52:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7f0e080adcb0 con 0x5580b2d3ce40 2024-05-27T07:26:54.352 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:54.357+0000 7f0e21d81b80 10 client.15639 fetch_fsmap learned FSMap version 30 2024-05-27T07:26:54.352 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:54.357+0000 7f0e21d81b80 1 -- 172.21.15.52:0/1397217571 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x5580b2f3fe90 con 0x5580b2d3ce40 2024-05-27T07:26:54.352 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:54.357+0000 7f0df4ff9700 1 -- 172.21.15.52:0/1397217571 <== mon.0 v2:172.21.15.52:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x5580b2de54a0 con 0x5580b2d3ce40 2024-05-27T07:26:54.352 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:54.357+0000 7f0df4ff9700 1 -- 172.21.15.52:0/1397217571 <== mon.0 v2:172.21.15.52:3300/0 5 ==== fsmap(e 30) ==== 8362+0+0 (secure 0 0 0) 0x7f0e0801cb90 con 0x5580b2d3ce40 2024-05-27T07:26:54.352 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:54.357+0000 7f0e21d81b80 10 client.15639 fetch_fsmap finished waiting for FSMap version 30 2024-05-27T07:26:54.352 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:54.357+0000 7f0e21d81b80 10 client.15639 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.c 2024-05-27T07:26:54.352 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:54.357+0000 7f0e21d81b80 20 client.15639 populate_metadata read hostname 'smithi052' 2024-05-27T07:26:54.352 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:54.357+0000 7f0e21d81b80 1 --2- 172.21.15.52:0/1397217571 >> [v2:172.21.15.173:6836/1396363827,v1:172.21.15.173:6837/1396363827] conn(0x5580b2ed5a70 0x5580b2f424d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:26:54.353 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:54.357+0000 7f0e21d81b80 4 client.15639 mds_command: new command op to 24466 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-05-27T07:26:54.353 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:54.357+0000 7f0e21d81b80 1 -- 172.21.15.52:0/1397217571 --> [v2:172.21.15.173:6836/1396363827,v1:172.21.15.173:6837/1396363827] -- command(tid 0: {"prefix": "damage ls"}) -- 0x5580b2c5c250 con 0x5580b2ed5a70 2024-05-27T07:26:54.353 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:54.358+0000 7f0e10907700 1 --2- 172.21.15.52:0/1397217571 >> [v2:172.21.15.173:6836/1396363827,v1:172.21.15.173:6837/1396363827] conn(0x5580b2ed5a70 0x5580b2f424d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:26:54.353 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:54.358+0000 7f0e10907700 1 --2- 172.21.15.52:0/1397217571 >> [v2:172.21.15.173:6836/1396363827,v1:172.21.15.173:6837/1396363827] conn(0x5580b2ed5a70 0x5580b2f424d0 crc :-1 s=READY pgs=28 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=ea301c14d04a5dc7 server_cookie=3df1d5f528653c4e in_seq=0 out_seq=0 2024-05-27T07:26:54.353 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:54.358+0000 7f0df4ff9700 10 client.15639 ms_handle_connect on v2:172.21.15.173:6836/1396363827 2024-05-27T07:26:54.353 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:54.358+0000 7f0df4ff9700 1 -- 172.21.15.52:0/1397217571 <== mds.4 v2:172.21.15.173:6836/1396363827 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x5580b2c5c250 con 0x5580b2ed5a70 2024-05-27T07:26:54.353 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:54.358+0000 7f0df4ff9700 10 client.15639 handle_command_reply: tid=0 2024-05-27T07:26:54.353 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:54.358+0000 7f0e21d81b80 1 client.15639 shutdown 2024-05-27T07:26:54.353 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:54.358+0000 7f0e21d81b80 1 -- 172.21.15.52:0/1397217571 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x5580b2d3ce40 msgr2=0x5580b2ecdf90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:26:54.353 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:54.358+0000 7f0e21d81b80 1 --2- 172.21.15.52:0/1397217571 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x5580b2d3ce40 0x5580b2ecdf90 secure :-1 s=READY pgs=477 cs=0 l=1 rev1=1 crypto rx=0x7f0e080059d0 tx=0x7f0e080050b0 comp rx=0 tx=0).stop 2024-05-27T07:26:54.353 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:54.359+0000 7f0e21d81b80 1 -- 172.21.15.52:0/1397217571 shutdown_connections 2024-05-27T07:26:54.353 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:54.359+0000 7f0e21d81b80 1 --2- 172.21.15.52:0/1397217571 >> [v2:172.21.15.173:6836/1396363827,v1:172.21.15.173:6837/1396363827] conn(0x5580b2ed5a70 0x5580b2f424d0 crc :-1 s=READY pgs=28 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:26:54.353 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:54.359+0000 7f0e21d81b80 1 --2- 172.21.15.52:0/1397217571 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5580b2ec2070 0x5580b2ebabd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:26:54.354 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:54.359+0000 7f0e21d81b80 1 --2- 172.21.15.52:0/1397217571 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x5580b2d3ce40 0x5580b2ecdf90 unknown :-1 s=CLOSED pgs=477 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:26:54.354 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:54.359+0000 7f0e21d81b80 1 --2- 172.21.15.52:0/1397217571 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5580b2ebc120 0x5580b2ece4d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:26:54.354 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:54.359+0000 7f0df4ff9700 0 client.15639 ms_handle_reset on v2:172.21.15.173:6836/1396363827 2024-05-27T07:26:54.354 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:54.359+0000 7f0e21d81b80 1 -- 172.21.15.52:0/1397217571 >> 172.21.15.52:0/1397217571 conn(0x5580b2d3f770 msgr2=0x5580b2eb04f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:26:54.354 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:54.359+0000 7f0e21d81b80 1 -- 172.21.15.52:0/1397217571 shutdown_connections 2024-05-27T07:26:54.354 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:54.359+0000 7f0e21d81b80 1 -- 172.21.15.52:0/1397217571 wait complete. 2024-05-27T07:26:54.354 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:54.359+0000 7f0e21d81b80 20 client.15639 trim_cache size 0 max 16384 2024-05-27T07:26:54.354 INFO:teuthology.orchestra.run.smithi052.stdout:[] 2024-05-27T07:26:54.355 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:54.359+0000 7f0e1336c700 1 -- 172.21.15.52:0/2191232058 >> [v2:172.21.15.52:6800/1822318384,v1:172.21.15.52:6801/1822318384] conn(0x7f0df80b2b10 msgr2=0x7f0df80b5020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:26:54.355 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:54.359+0000 7f0e1336c700 1 --2- 172.21.15.52:0/2191232058 >> [v2:172.21.15.52:6800/1822318384,v1:172.21.15.52:6801/1822318384] conn(0x7f0df80b2b10 0x7f0df80b5020 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f0e04007980 tx=0x7f0e0401b000 comp rx=0 tx=0).stop 2024-05-27T07:26:54.355 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:54.359+0000 7f0e1336c700 1 -- 172.21.15.52:0/2191232058 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f0e0c1034d0 msgr2=0x7f0e0c19d0b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:26:54.355 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:54.359+0000 7f0e1336c700 1 --2- 172.21.15.52:0/2191232058 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f0e0c1034d0 0x7f0e0c19d0b0 secure :-1 s=READY pgs=473 cs=0 l=1 rev1=1 crypto rx=0x7f0e0800c920 tx=0x7f0e0800cce0 comp rx=0 tx=0).stop 2024-05-27T07:26:54.355 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:54.359+0000 7f0e1336c700 1 -- 172.21.15.52:0/2191232058 shutdown_connections 2024-05-27T07:26:54.355 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:54.359+0000 7f0e1336c700 1 --2- 172.21.15.52:0/2191232058 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f0e0c108750 0x7f0e0c1a18e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:26:54.355 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:54.359+0000 7f0e1336c700 1 --2- 172.21.15.52:0/2191232058 >> [v2:172.21.15.52:6800/1822318384,v1:172.21.15.52:6801/1822318384] conn(0x7f0df80b2b10 0x7f0df80b5020 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:26:54.356 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:54.359+0000 7f0e1336c700 1 --2- 172.21.15.52:0/2191232058 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f0e0c1034d0 0x7f0e0c19d0b0 unknown :-1 s=CLOSED pgs=473 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:26:54.356 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:54.359+0000 7f0e1336c700 1 --2- 172.21.15.52:0/2191232058 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f0e0c107e00 0x7f0e0c19d650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:26:54.356 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:54.360+0000 7f0e1336c700 1 -- 172.21.15.52:0/2191232058 >> 172.21.15.52:0/2191232058 conn(0x7f0e0c0fef20 msgr2=0x7f0e0c1096b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:26:54.356 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:54.360+0000 7f0e1336c700 1 -- 172.21.15.52:0/2191232058 shutdown_connections 2024-05-27T07:26:54.356 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:54.360+0000 7f0e1336c700 1 -- 172.21.15.52:0/2191232058 wait complete. 2024-05-27T07:26:54.356 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:26:54 smithi173 ceph-mon[37000]: pgmap v31: 129 pgs: 129 active+clean; 1.3 MiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 7.2 KiB/s rd, 1.9 KiB/s wr, 14 op/s 2024-05-27T07:26:54.439 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:26:54 smithi105 ceph-mon[35977]: pgmap v31: 129 pgs: 129 active+clean; 1.3 MiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 7.2 KiB/s rd, 1.9 KiB/s wr, 14 op/s 2024-05-27T07:26:55.012 DEBUG:teuthology.orchestra.run.smithi052:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:38f9d900c83c63799fbdbe61acc9a11b0d3554a6 shell --fsid 73df2636-1bf8-11ef-bc9b-c7b262605968 -- ceph tell mds.1:3 damage ls 2024-05-27T07:26:55.264 INFO:teuthology.orchestra.run.smithi052.stderr:Inferring config /var/lib/ceph/73df2636-1bf8-11ef-bc9b-c7b262605968/mon.a/config 2024-05-27T07:26:56.351 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:26:56 smithi173 ceph-mon[37000]: pgmap v32: 129 pgs: 129 active+clean; 1.3 MiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 1.7 KiB/s wr, 10 op/s 2024-05-27T07:26:56.440 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:26:56 smithi105 ceph-mon[35977]: pgmap v32: 129 pgs: 129 active+clean; 1.3 MiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 1.7 KiB/s wr, 10 op/s 2024-05-27T07:26:56.479 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:26:56 smithi052 ceph-mon[31955]: pgmap v32: 129 pgs: 129 active+clean; 1.3 MiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 1.7 KiB/s wr, 10 op/s 2024-05-27T07:26:57.089 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:57.099+0000 7f120e076700 1 -- 172.21.15.52:0/1659150679 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f12081034d0 msgr2=0x7f12081038e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:26:57.089 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:57.099+0000 7f120e076700 1 --2- 172.21.15.52:0/1659150679 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f12081034d0 0x7f12081038e0 secure :-1 s=READY pgs=478 cs=0 l=1 rev1=1 crypto rx=0x7f11f80099c0 tx=0x7f11f8009cd0 comp rx=0 tx=0).stop 2024-05-27T07:26:57.089 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:57.099+0000 7f120e076700 1 -- 172.21.15.52:0/1659150679 shutdown_connections 2024-05-27T07:26:57.089 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:57.099+0000 7f120e076700 1 --2- 172.21.15.52:0/1659150679 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f1208108750 0x7f12081100b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:26:57.089 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:57.099+0000 7f120e076700 1 --2- 172.21.15.52:0/1659150679 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f12081034d0 0x7f12081038e0 unknown :-1 s=CLOSED pgs=478 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:26:57.090 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:57.099+0000 7f120e076700 1 --2- 172.21.15.52:0/1659150679 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f1208107e00 0x7f1208108210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:26:57.090 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:57.099+0000 7f120e076700 1 -- 172.21.15.52:0/1659150679 >> 172.21.15.52:0/1659150679 conn(0x7f12080fee40 msgr2=0x7f1208101260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:26:57.090 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:57.099+0000 7f120e076700 1 -- 172.21.15.52:0/1659150679 shutdown_connections 2024-05-27T07:26:57.090 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:57.099+0000 7f120e076700 1 -- 172.21.15.52:0/1659150679 wait complete. 2024-05-27T07:26:57.090 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:57.100+0000 7f120e076700 1 Processor -- start 2024-05-27T07:26:57.091 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:57.100+0000 7f120e076700 1 -- start start 2024-05-27T07:26:57.091 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:57.100+0000 7f120e076700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f12081034d0 0x7f120819ad20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:26:57.092 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:57.100+0000 7f120e076700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f1208107e00 0x7f120819b2c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:26:57.092 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:57.100+0000 7f120e076700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f1208108750 0x7f120819f550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:26:57.092 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:57.100+0000 7f120e076700 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x7f1208113380 con 0x7f1208108750 2024-05-27T07:26:57.092 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:57.100+0000 7f120e076700 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f1208113200 con 0x7f12081034d0 2024-05-27T07:26:57.092 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:57.100+0000 7f120e076700 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f1208113500 con 0x7f1208107e00 2024-05-27T07:26:57.092 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:57.100+0000 7f1207fff700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f1208108750 0x7f120819f550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:26:57.092 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:57.100+0000 7f1207fff700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f1208108750 0x7f120819f550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.52:37392/0 (socket says 172.21.15.52:37392) 2024-05-27T07:26:57.092 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:57.100+0000 7f1207fff700 1 -- 172.21.15.52:0/3956247317 learned_addr learned my addr 172.21.15.52:0/3956247317 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-05-27T07:26:57.092 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:57.100+0000 7f12077fe700 1 --2- 172.21.15.52:0/3956247317 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f12081034d0 0x7f120819ad20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:26:57.092 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:57.101+0000 7f1206ffd700 1 --2- 172.21.15.52:0/3956247317 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f1208107e00 0x7f120819b2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:26:57.092 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:57.101+0000 7f1207fff700 1 -- 172.21.15.52:0/3956247317 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f1208107e00 msgr2=0x7f120819b2c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:26:57.092 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:57.101+0000 7f1207fff700 1 --2- 172.21.15.52:0/3956247317 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f1208107e00 0x7f120819b2c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:26:57.093 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:57.101+0000 7f1207fff700 1 -- 172.21.15.52:0/3956247317 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f12081034d0 msgr2=0x7f120819ad20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:26:57.093 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:57.101+0000 7f1207fff700 1 --2- 172.21.15.52:0/3956247317 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f12081034d0 0x7f120819ad20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:26:57.093 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:57.101+0000 7f1207fff700 1 -- 172.21.15.52:0/3956247317 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f120819fb40 con 0x7f1208108750 2024-05-27T07:26:57.093 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:57.101+0000 7f1207fff700 1 --2- 172.21.15.52:0/3956247317 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f1208108750 0x7f120819f550 secure :-1 s=READY pgs=479 cs=0 l=1 rev1=1 crypto rx=0x7f1200007d70 tx=0x7f120000a340 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:26:57.093 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:57.101+0000 7f1204ff9700 1 -- 172.21.15.52:0/3956247317 <== mon.0 v2:172.21.15.52:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1200016020 con 0x7f1208108750 2024-05-27T07:26:57.093 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:57.101+0000 7f120e076700 1 -- 172.21.15.52:0/3956247317 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f120819fe30 con 0x7f1208108750 2024-05-27T07:26:57.093 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:57.101+0000 7f1204ff9700 1 -- 172.21.15.52:0/3956247317 <== mon.0 v2:172.21.15.52:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f120000ac90 con 0x7f1208108750 2024-05-27T07:26:57.093 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:57.101+0000 7f1204ff9700 1 -- 172.21.15.52:0/3956247317 <== mon.0 v2:172.21.15.52:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f12000115e0 con 0x7f1208108750 2024-05-27T07:26:57.093 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:57.101+0000 7f120e076700 1 -- 172.21.15.52:0/3956247317 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f12081a0370 con 0x7f1208108750 2024-05-27T07:26:57.093 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:57.104+0000 7f1204ff9700 1 -- 172.21.15.52:0/3956247317 <== mon.0 v2:172.21.15.52:3300/0 4 ==== mgrmap(e 24) ==== 149879+0+0 (secure 0 0 0) 0x7f1200019020 con 0x7f1208108750 2024-05-27T07:26:57.094 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:57.105+0000 7f1204ff9700 1 --2- 172.21.15.52:0/3956247317 >> [v2:172.21.15.52:6800/1822318384,v1:172.21.15.52:6801/1822318384] conn(0x7f11f00b2b10 0x7f11f00b5020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:26:57.095 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:57.105+0000 7f12077fe700 1 --2- 172.21.15.52:0/3956247317 >> [v2:172.21.15.52:6800/1822318384,v1:172.21.15.52:6801/1822318384] conn(0x7f11f00b2b10 0x7f11f00b5020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:26:57.095 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:57.105+0000 7f1204ff9700 1 -- 172.21.15.52:0/3956247317 <== mon.0 v2:172.21.15.52:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9379+0+0 (secure 0 0 0) 0x7f12000e2a70 con 0x7f1208108750 2024-05-27T07:26:57.095 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:57.105+0000 7f121ca8bb80 1 Processor -- start 2024-05-27T07:26:57.095 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:57.105+0000 7f121ca8bb80 1 -- start start 2024-05-27T07:26:57.097 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:57.106+0000 7f12077fe700 1 --2- 172.21.15.52:0/3956247317 >> [v2:172.21.15.52:6800/1822318384,v1:172.21.15.52:6801/1822318384] conn(0x7f11f00b2b10 0x7f11f00b5020 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f11f8009990 tx=0x7f11f8000bc0 comp rx=0 tx=0).ready entity=mgr.15333 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:26:57.097 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:57.106+0000 7f121ca8bb80 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5568f311fe40 0x5568f32a20f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:26:57.097 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:57.106+0000 7f121ca8bb80 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5568f32a2630 0x5568f32a65d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:26:57.097 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:57.106+0000 7f121ca8bb80 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x5568f32a6b10 0x5568f32a6f20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:26:57.097 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:57.106+0000 7f121ca8bb80 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x5568f31ccc90 con 0x5568f32a6b10 2024-05-27T07:26:57.097 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:57.106+0000 7f121ca8bb80 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x5568f31bc830 con 0x5568f311fe40 2024-05-27T07:26:57.097 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:57.106+0000 7f121ca8bb80 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x5568f31c0870 con 0x5568f32a2630 2024-05-27T07:26:57.097 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:57.106+0000 7f1207fff700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x5568f32a6b10 0x5568f32a6f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:26:57.097 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:57.106+0000 7f1207fff700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x5568f32a6b10 0x5568f32a6f20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.52:37406/0 (socket says 172.21.15.52:37406) 2024-05-27T07:26:57.097 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:57.106+0000 7f1207fff700 1 -- 172.21.15.52:0/536288526 learned_addr learned my addr 172.21.15.52:0/536288526 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-05-27T07:26:57.097 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:57.106+0000 7f1206ffd700 1 --2- 172.21.15.52:0/536288526 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5568f32a2630 0x5568f32a65d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:26:57.097 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:57.106+0000 7f12077fe700 1 --2- 172.21.15.52:0/536288526 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5568f311fe40 0x5568f32a20f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:26:57.098 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:57.106+0000 7f1207fff700 1 -- 172.21.15.52:0/536288526 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5568f32a2630 msgr2=0x5568f32a65d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:26:57.098 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:57.106+0000 7f1207fff700 1 --2- 172.21.15.52:0/536288526 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5568f32a2630 0x5568f32a65d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:26:57.098 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:57.106+0000 7f1207fff700 1 -- 172.21.15.52:0/536288526 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5568f311fe40 msgr2=0x5568f32a20f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:26:57.098 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:57.106+0000 7f1207fff700 1 --2- 172.21.15.52:0/536288526 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5568f311fe40 0x5568f32a20f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:26:57.098 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:57.106+0000 7f1207fff700 1 -- 172.21.15.52:0/536288526 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5568f310c8d0 con 0x5568f32a6b10 2024-05-27T07:26:57.098 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:57.106+0000 7f1207fff700 1 --2- 172.21.15.52:0/536288526 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x5568f32a6b10 0x5568f32a6f20 secure :-1 s=READY pgs=480 cs=0 l=1 rev1=1 crypto rx=0x7f12000a58d0 tx=0x7f12000a9720 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f738498f43ed81b4 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:26:57.098 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:57.107+0000 7f11edffb700 1 -- 172.21.15.52:0/536288526 <== mon.0 v2:172.21.15.52:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f12000a8fc0 con 0x5568f32a6b10 2024-05-27T07:26:57.098 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:57.107+0000 7f11edffb700 1 -- 172.21.15.52:0/536288526 <== mon.0 v2:172.21.15.52:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f120000ae40 con 0x5568f32a6b10 2024-05-27T07:26:57.098 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:57.107+0000 7f11edffb700 1 -- 172.21.15.52:0/536288526 <== mon.0 v2:172.21.15.52:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f12000a9cb0 con 0x5568f32a6b10 2024-05-27T07:26:57.104 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:57.108+0000 7f121ca8bb80 1 -- 172.21.15.52:0/536288526 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x5568f32a6b10 msgr2=0x5568f32a6f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:26:57.104 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:57.108+0000 7f121ca8bb80 1 --2- 172.21.15.52:0/536288526 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x5568f32a6b10 0x5568f32a6f20 secure :-1 s=READY pgs=480 cs=0 l=1 rev1=1 crypto rx=0x7f12000a58d0 tx=0x7f12000a9720 comp rx=0 tx=0).stop 2024-05-27T07:26:57.104 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:57.108+0000 7f121ca8bb80 1 -- 172.21.15.52:0/536288526 shutdown_connections 2024-05-27T07:26:57.104 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:57.108+0000 7f121ca8bb80 1 --2- 172.21.15.52:0/536288526 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x5568f32a6b10 0x5568f32a6f20 unknown :-1 s=CLOSED pgs=480 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:26:57.104 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:57.108+0000 7f121ca8bb80 1 --2- 172.21.15.52:0/536288526 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5568f311fe40 0x5568f32a20f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:26:57.104 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:57.108+0000 7f121ca8bb80 1 --2- 172.21.15.52:0/536288526 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5568f32a2630 0x5568f32a65d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:26:57.104 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:57.108+0000 7f121ca8bb80 1 -- 172.21.15.52:0/536288526 >> 172.21.15.52:0/536288526 conn(0x5568f317aaf0 msgr2=0x5568f3298640 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:26:57.104 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:57.108+0000 7f121ca8bb80 1 -- 172.21.15.52:0/536288526 shutdown_connections 2024-05-27T07:26:57.104 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:57.108+0000 7f121ca8bb80 1 -- 172.21.15.52:0/536288526 wait complete. 2024-05-27T07:26:57.104 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:57.108+0000 7f121ca8bb80 1 -- start start 2024-05-27T07:26:57.104 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:57.108+0000 7f121ca8bb80 1 Processor -- start 2024-05-27T07:26:57.104 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:57.109+0000 7f121ca8bb80 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5568f311fe40 0x5568f32ab850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:26:57.105 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:57.109+0000 7f121ca8bb80 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5568f32a2630 0x5568f32abd90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:26:57.105 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:57.109+0000 7f121ca8bb80 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x5568f32a6b10 0x5568f32ac2d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:26:57.105 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:57.109+0000 7f121ca8bb80 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x5568f30c2fa0 con 0x5568f32a6b10 2024-05-27T07:26:57.105 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:57.109+0000 7f121ca8bb80 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x5568f315e7c0 con 0x5568f311fe40 2024-05-27T07:26:57.105 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:57.109+0000 7f121ca8bb80 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x5568f32aeb20 con 0x5568f32a2630 2024-05-27T07:26:57.105 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:57.109+0000 7f121ca8bb80 10 client.0 _ll_register_callbacks cb 0x5568f3157920 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T07:26:57.105 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:57.109+0000 7f1207fff700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x5568f32a6b10 0x5568f32ac2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:26:57.105 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:57.109+0000 7f1207fff700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x5568f32a6b10 0x5568f32ac2d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.52:37420/0 (socket says 172.21.15.52:37420) 2024-05-27T07:26:57.105 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:57.109+0000 7f1207fff700 1 -- 172.21.15.52:0/1502600427 learned_addr learned my addr 172.21.15.52:0/1502600427 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-05-27T07:26:57.105 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:57.109+0000 7f12077fe700 1 --2- 172.21.15.52:0/1502600427 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5568f311fe40 0x5568f32ab850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:26:57.105 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:57.109+0000 7f1207fff700 1 -- 172.21.15.52:0/1502600427 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5568f32a2630 msgr2=0x5568f32abd90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:26:57.105 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:57.109+0000 7f1207fff700 1 --2- 172.21.15.52:0/1502600427 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5568f32a2630 0x5568f32abd90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:26:57.106 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:57.109+0000 7f1207fff700 1 -- 172.21.15.52:0/1502600427 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5568f311fe40 msgr2=0x5568f32ab850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:26:57.106 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:57.109+0000 7f1207fff700 1 --2- 172.21.15.52:0/1502600427 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5568f311fe40 0x5568f32ab850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:26:57.106 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:57.109+0000 7f1207fff700 1 -- 172.21.15.52:0/1502600427 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5568f315b690 con 0x5568f32a6b10 2024-05-27T07:26:57.106 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:57.109+0000 7f1207fff700 1 --2- 172.21.15.52:0/1502600427 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x5568f32a6b10 0x5568f32ac2d0 secure :-1 s=READY pgs=481 cs=0 l=1 rev1=1 crypto rx=0x7f12000a97e0 tx=0x7f12000a9e10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c8c6e46fd06840c server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:26:57.106 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:57.109+0000 7f11e6ffd700 10 client.0 ms_handle_connect on v2:172.21.15.52:3300/0 2024-05-27T07:26:57.106 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:57.110+0000 7f11e6ffd700 1 -- 172.21.15.52:0/1502600427 <== mon.0 v2:172.21.15.52:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f12000a9850 con 0x5568f32a6b10 2024-05-27T07:26:57.106 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:57.110+0000 7f121ca8bb80 1 -- 172.21.15.52:0/1502600427 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x5568f32699f0 con 0x5568f32a6b10 2024-05-27T07:26:57.106 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:57.110+0000 7f11e6ffd700 1 -- 172.21.15.52:0/1502600427 <== mon.0 v2:172.21.15.52:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 9379+0+0 (secure 0 0 0) 0x7f12000b2430 con 0x5568f32a6b10 2024-05-27T07:26:57.106 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:57.110+0000 7f11e6ffd700 1 -- 172.21.15.52:0/1502600427 <== mon.0 v2:172.21.15.52:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7f12000a9f50 con 0x5568f32a6b10 2024-05-27T07:26:57.106 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:57.110+0000 7f121ca8bb80 10 client.15651 fetch_fsmap learned FSMap version 30 2024-05-27T07:26:57.106 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:57.110+0000 7f121ca8bb80 1 -- 172.21.15.52:0/1502600427 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x5568f32b95d0 con 0x5568f32a6b10 2024-05-27T07:26:57.106 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:57.110+0000 7f11e6ffd700 1 -- 172.21.15.52:0/1502600427 <== mon.0 v2:172.21.15.52:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x5568f32699f0 con 0x5568f32a6b10 2024-05-27T07:26:57.107 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:57.110+0000 7f11e6ffd700 1 -- 172.21.15.52:0/1502600427 <== mon.0 v2:172.21.15.52:3300/0 5 ==== fsmap(e 30) ==== 8362+0+0 (secure 0 0 0) 0x7f12000aaca0 con 0x5568f32a6b10 2024-05-27T07:26:57.107 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:57.111+0000 7f121ca8bb80 10 client.15651 fetch_fsmap finished waiting for FSMap version 30 2024-05-27T07:26:57.107 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:57.111+0000 7f121ca8bb80 10 client.15651 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.e 2024-05-27T07:26:57.107 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:57.111+0000 7f121ca8bb80 20 client.15651 populate_metadata read hostname 'smithi052' 2024-05-27T07:26:57.107 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:57.111+0000 7f121ca8bb80 1 --2- 172.21.15.52:0/1502600427 >> [v2:172.21.15.105:6834/3298498384,v1:172.21.15.105:6835/3298498384] conn(0x5568f32bbaa0 0x5568f32bdec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:26:57.107 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:57.111+0000 7f121ca8bb80 4 client.15651 mds_command: new command op to 24473 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-27T07:26:57.107 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:57.111+0000 7f121ca8bb80 1 -- 172.21.15.52:0/1502600427 --> [v2:172.21.15.105:6834/3298498384,v1:172.21.15.105:6835/3298498384] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5568f31cebe0 con 0x5568f32bbaa0 2024-05-27T07:26:57.107 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:57.111+0000 7f12077fe700 1 --2- 172.21.15.52:0/1502600427 >> [v2:172.21.15.105:6834/3298498384,v1:172.21.15.105:6835/3298498384] conn(0x5568f32bbaa0 0x5568f32bdec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:26:57.107 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:57.112+0000 7f12077fe700 1 --2- 172.21.15.52:0/1502600427 >> [v2:172.21.15.105:6834/3298498384,v1:172.21.15.105:6835/3298498384] conn(0x5568f32bbaa0 0x5568f32bdec0 crc :-1 s=READY pgs=26 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=a8430d35f3272d65 server_cookie=89b0b8963b8c7623 in_seq=0 out_seq=0 2024-05-27T07:26:57.107 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:57.112+0000 7f11e6ffd700 10 client.15651 ms_handle_connect on v2:172.21.15.105:6834/3298498384 2024-05-27T07:26:57.107 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:57.112+0000 7f11e6ffd700 1 -- 172.21.15.52:0/1502600427 <== mds.3 v2:172.21.15.105:6834/3298498384 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24041 (crc 0 0 0) 0x5568f31cebe0 con 0x5568f32bbaa0 2024-05-27T07:26:57.107 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:57.112+0000 7f11e6ffd700 10 client.15651 handle_command_reply: tid=0 2024-05-27T07:26:57.107 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:57.112+0000 7f121ca8bb80 1 client.15651 shutdown 2024-05-27T07:26:57.108 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:57.113+0000 7f121ca8bb80 1 -- 172.21.15.52:0/1502600427 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x5568f32a6b10 msgr2=0x5568f32ac2d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:26:57.108 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:57.113+0000 7f121ca8bb80 1 --2- 172.21.15.52:0/1502600427 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x5568f32a6b10 0x5568f32ac2d0 secure :-1 s=READY pgs=481 cs=0 l=1 rev1=1 crypto rx=0x7f12000a97e0 tx=0x7f12000a9e10 comp rx=0 tx=0).stop 2024-05-27T07:26:57.108 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:57.113+0000 7f121ca8bb80 1 -- 172.21.15.52:0/1502600427 shutdown_connections 2024-05-27T07:26:57.108 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:57.113+0000 7f121ca8bb80 1 --2- 172.21.15.52:0/1502600427 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x5568f32a6b10 0x5568f32ac2d0 unknown :-1 s=CLOSED pgs=481 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:26:57.108 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:57.113+0000 7f121ca8bb80 1 --2- 172.21.15.52:0/1502600427 >> [v2:172.21.15.105:6834/3298498384,v1:172.21.15.105:6835/3298498384] conn(0x5568f32bbaa0 0x5568f32bdec0 crc :-1 s=READY pgs=26 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:26:57.108 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:57.113+0000 7f11e6ffd700 0 client.15651 ms_handle_reset on v2:172.21.15.105:6834/3298498384 2024-05-27T07:26:57.108 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:57.113+0000 7f121ca8bb80 1 --2- 172.21.15.52:0/1502600427 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5568f311fe40 0x5568f32ab850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:26:57.108 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:57.113+0000 7f121ca8bb80 1 --2- 172.21.15.52:0/1502600427 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5568f32a2630 0x5568f32abd90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:26:57.108 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:57.113+0000 7f121ca8bb80 1 -- 172.21.15.52:0/1502600427 >> 172.21.15.52:0/1502600427 conn(0x5568f317aaf0 msgr2=0x5568f329ca30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:26:57.108 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:57.113+0000 7f121ca8bb80 1 -- 172.21.15.52:0/1502600427 shutdown_connections 2024-05-27T07:26:57.108 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:57.113+0000 7f121ca8bb80 1 -- 172.21.15.52:0/1502600427 wait complete. 2024-05-27T07:26:57.108 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:57.113+0000 7f121ca8bb80 20 client.15651 trim_cache size 0 max 16384 2024-05-27T07:26:57.119 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:57.128+0000 7f121ca8bb80 1 Processor -- start 2024-05-27T07:26:57.119 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:57.129+0000 7f121ca8bb80 1 -- start start 2024-05-27T07:26:57.119 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:57.129+0000 7f121ca8bb80 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x5568f311fe40 0x5568f32b7d80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:26:57.119 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:57.129+0000 7f121ca8bb80 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5568f32a2630 0x5568f32a63e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:26:57.120 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:57.129+0000 7f121ca8bb80 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5568f32a6b10 0x5568f32ac110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:26:57.120 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:57.129+0000 7f121ca8bb80 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x7f12081ae180 con 0x5568f311fe40 2024-05-27T07:26:57.120 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:57.129+0000 7f121ca8bb80 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f12081afcc0 con 0x5568f32a2630 2024-05-27T07:26:57.120 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:57.129+0000 7f121ca8bb80 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f12081ae850 con 0x5568f32a6b10 2024-05-27T07:26:57.120 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:57.129+0000 7f12077fe700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x5568f311fe40 0x5568f32b7d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:26:57.120 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:57.129+0000 7f12077fe700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x5568f311fe40 0x5568f32b7d80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.52:37426/0 (socket says 172.21.15.52:37426) 2024-05-27T07:26:57.120 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:57.129+0000 7f12077fe700 1 -- 172.21.15.52:0/419306916 learned_addr learned my addr 172.21.15.52:0/419306916 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-05-27T07:26:57.120 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:57.129+0000 7f1206ffd700 1 --2- 172.21.15.52:0/419306916 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5568f32a2630 0x5568f32a63e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:26:57.120 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:57.129+0000 7f12077fe700 1 -- 172.21.15.52:0/419306916 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5568f32a6b10 msgr2=0x5568f32ac110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:26:57.120 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:57.129+0000 7f12077fe700 1 --2- 172.21.15.52:0/419306916 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5568f32a6b10 0x5568f32ac110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:26:57.120 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:57.129+0000 7f12077fe700 1 -- 172.21.15.52:0/419306916 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5568f32a2630 msgr2=0x5568f32a63e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:26:57.120 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:57.129+0000 7f12077fe700 1 --2- 172.21.15.52:0/419306916 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5568f32a2630 0x5568f32a63e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:26:57.120 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:57.129+0000 7f12077fe700 1 -- 172.21.15.52:0/419306916 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5568f32a2cb0 con 0x5568f311fe40 2024-05-27T07:26:57.121 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:57.129+0000 7f1206ffd700 1 --2- 172.21.15.52:0/419306916 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5568f32a2630 0x5568f32a63e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-27T07:26:57.121 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:57.129+0000 7f12077fe700 1 --2- 172.21.15.52:0/419306916 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x5568f311fe40 0x5568f32b7d80 secure :-1 s=READY pgs=482 cs=0 l=1 rev1=1 crypto rx=0x7f11f800b580 tx=0x7f11f801e000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d9d966ebb885f42b server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:26:57.121 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:57.130+0000 7f11ed7fa700 1 -- 172.21.15.52:0/419306916 <== mon.0 v2:172.21.15.52:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f11f800bdf0 con 0x5568f311fe40 2024-05-27T07:26:57.121 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:57.130+0000 7f11ed7fa700 1 -- 172.21.15.52:0/419306916 <== mon.0 v2:172.21.15.52:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f11f800f910 con 0x5568f311fe40 2024-05-27T07:26:57.121 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:57.130+0000 7f11ed7fa700 1 -- 172.21.15.52:0/419306916 <== mon.0 v2:172.21.15.52:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f11f8031680 con 0x5568f311fe40 2024-05-27T07:26:57.124 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:57.130+0000 7f121ca8bb80 1 -- 172.21.15.52:0/419306916 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x5568f311fe40 msgr2=0x5568f32b7d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:26:57.124 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:57.130+0000 7f121ca8bb80 1 --2- 172.21.15.52:0/419306916 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x5568f311fe40 0x5568f32b7d80 secure :-1 s=READY pgs=482 cs=0 l=1 rev1=1 crypto rx=0x7f11f800b580 tx=0x7f11f801e000 comp rx=0 tx=0).stop 2024-05-27T07:26:57.124 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:57.130+0000 7f121ca8bb80 1 -- 172.21.15.52:0/419306916 shutdown_connections 2024-05-27T07:26:57.124 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:57.130+0000 7f121ca8bb80 1 --2- 172.21.15.52:0/419306916 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5568f32a6b10 0x5568f32ac110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:26:57.124 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:57.130+0000 7f121ca8bb80 1 --2- 172.21.15.52:0/419306916 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x5568f311fe40 0x5568f32b7d80 unknown :-1 s=CLOSED pgs=482 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:26:57.124 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:57.130+0000 7f121ca8bb80 1 --2- 172.21.15.52:0/419306916 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5568f32a2630 0x5568f32a63e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:26:57.124 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:57.130+0000 7f121ca8bb80 1 -- 172.21.15.52:0/419306916 >> 172.21.15.52:0/419306916 conn(0x5568f317aaf0 msgr2=0x5568f329ca30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:26:57.124 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:57.131+0000 7f121ca8bb80 1 -- 172.21.15.52:0/419306916 shutdown_connections 2024-05-27T07:26:57.124 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:57.131+0000 7f121ca8bb80 1 -- 172.21.15.52:0/419306916 wait complete. 2024-05-27T07:26:57.124 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:57.131+0000 7f121ca8bb80 1 -- start start 2024-05-27T07:26:57.125 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:57.131+0000 7f121ca8bb80 1 Processor -- start 2024-05-27T07:26:57.125 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:57.132+0000 7f121ca8bb80 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5568f311fe40 0x5568f331d7f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:26:57.125 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:57.132+0000 7f121ca8bb80 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x5568f32a2630 0x5568f331dd30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:26:57.125 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:57.132+0000 7f121ca8bb80 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5568f32a6b10 0x5568f32c0750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:26:57.125 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:57.132+0000 7f121ca8bb80 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x7f12081af5f0 con 0x5568f32a2630 2024-05-27T07:26:57.125 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:57.132+0000 7f121ca8bb80 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f12081aef20 con 0x5568f311fe40 2024-05-27T07:26:57.125 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:57.132+0000 7f121ca8bb80 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f12081ac120 con 0x5568f32a6b10 2024-05-27T07:26:57.125 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:57.132+0000 7f121ca8bb80 10 client.0 _ll_register_callbacks cb 0x5568f3157920 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T07:26:57.125 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:57.132+0000 7f1206ffd700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x5568f32a2630 0x5568f331dd30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:26:57.125 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:57.132+0000 7f1206ffd700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x5568f32a2630 0x5568f331dd30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.52:37440/0 (socket says 172.21.15.52:37440) 2024-05-27T07:26:57.125 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:57.132+0000 7f1206ffd700 1 -- 172.21.15.52:0/4160672532 learned_addr learned my addr 172.21.15.52:0/4160672532 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-05-27T07:26:57.125 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:57.132+0000 7f1207fff700 1 --2- 172.21.15.52:0/4160672532 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5568f32a6b10 0x5568f32c0750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:26:57.126 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:57.132+0000 7f12077fe700 1 --2- 172.21.15.52:0/4160672532 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5568f311fe40 0x5568f331d7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:26:57.126 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:57.132+0000 7f1206ffd700 1 -- 172.21.15.52:0/4160672532 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5568f32a6b10 msgr2=0x5568f32c0750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:26:57.126 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:57.132+0000 7f1206ffd700 1 --2- 172.21.15.52:0/4160672532 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5568f32a6b10 0x5568f32c0750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:26:57.126 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:57.132+0000 7f1206ffd700 1 -- 172.21.15.52:0/4160672532 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5568f311fe40 msgr2=0x5568f331d7f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:26:57.126 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:57.132+0000 7f1206ffd700 1 --2- 172.21.15.52:0/4160672532 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5568f311fe40 0x5568f331d7f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:26:57.126 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:57.132+0000 7f1206ffd700 1 -- 172.21.15.52:0/4160672532 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5568f32c0d60 con 0x5568f32a2630 2024-05-27T07:26:57.126 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:57.132+0000 7f12077fe700 1 --2- 172.21.15.52:0/4160672532 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5568f311fe40 0x5568f331d7f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-27T07:26:57.126 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:57.132+0000 7f1206ffd700 1 --2- 172.21.15.52:0/4160672532 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x5568f32a2630 0x5568f331dd30 secure :-1 s=READY pgs=483 cs=0 l=1 rev1=1 crypto rx=0x5568f3299e10 tx=0x7f11fc012d20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b241ce9fb3c41ca4 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:26:57.126 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:57.132+0000 7f11e6ffd700 10 client.0 ms_handle_connect on v2:172.21.15.52:3300/0 2024-05-27T07:26:57.126 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:57.133+0000 7f11e6ffd700 1 -- 172.21.15.52:0/4160672532 <== mon.0 v2:172.21.15.52:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f11fc00f760 con 0x5568f32a2630 2024-05-27T07:26:57.126 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:57.133+0000 7f11e6ffd700 1 -- 172.21.15.52:0/4160672532 <== mon.0 v2:172.21.15.52:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 9379+0+0 (secure 0 0 0) 0x7f11fc00f900 con 0x5568f32a2630 2024-05-27T07:26:57.126 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:57.133+0000 7f121ca8bb80 1 -- 172.21.15.52:0/4160672532 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f11f800f910 con 0x5568f32a2630 2024-05-27T07:26:57.126 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:57.133+0000 7f11e6ffd700 1 -- 172.21.15.52:0/4160672532 <== mon.0 v2:172.21.15.52:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7f11fc00fb40 con 0x5568f32a2630 2024-05-27T07:26:57.127 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:57.133+0000 7f121ca8bb80 10 client.15657 fetch_fsmap learned FSMap version 30 2024-05-27T07:26:57.127 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:57.133+0000 7f121ca8bb80 1 -- 172.21.15.52:0/4160672532 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x5568f3325f50 con 0x5568f32a2630 2024-05-27T07:26:57.127 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:57.133+0000 7f11e6ffd700 1 -- 172.21.15.52:0/4160672532 <== mon.0 v2:172.21.15.52:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f11f800f910 con 0x5568f32a2630 2024-05-27T07:26:57.127 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:57.133+0000 7f11e6ffd700 1 -- 172.21.15.52:0/4160672532 <== mon.0 v2:172.21.15.52:3300/0 5 ==== fsmap(e 30) ==== 8362+0+0 (secure 0 0 0) 0x7f11fc01c420 con 0x5568f32a2630 2024-05-27T07:26:57.127 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:57.133+0000 7f121ca8bb80 10 client.15657 fetch_fsmap finished waiting for FSMap version 30 2024-05-27T07:26:57.127 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:57.133+0000 7f121ca8bb80 10 client.15657 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.e 2024-05-27T07:26:57.127 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:57.133+0000 7f121ca8bb80 20 client.15657 populate_metadata read hostname 'smithi052' 2024-05-27T07:26:57.127 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:57.133+0000 7f121ca8bb80 1 --2- 172.21.15.52:0/4160672532 >> [v2:172.21.15.105:6834/3298498384,v1:172.21.15.105:6835/3298498384] conn(0x5568f32bbaa0 0x5568f3328720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:26:57.127 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:57.133+0000 7f121ca8bb80 4 client.15657 mds_command: new command op to 24473 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-05-27T07:26:57.127 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:57.133+0000 7f121ca8bb80 1 -- 172.21.15.52:0/4160672532 --> [v2:172.21.15.105:6834/3298498384,v1:172.21.15.105:6835/3298498384] -- command(tid 0: {"prefix": "damage ls"}) -- 0x5568f31c83c0 con 0x5568f32bbaa0 2024-05-27T07:26:57.127 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:57.134+0000 7f12077fe700 1 --2- 172.21.15.52:0/4160672532 >> [v2:172.21.15.105:6834/3298498384,v1:172.21.15.105:6835/3298498384] conn(0x5568f32bbaa0 0x5568f3328720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:26:57.127 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:57.134+0000 7f12077fe700 1 --2- 172.21.15.52:0/4160672532 >> [v2:172.21.15.105:6834/3298498384,v1:172.21.15.105:6835/3298498384] conn(0x5568f32bbaa0 0x5568f3328720 crc :-1 s=READY pgs=27 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=5554d67b14484eae server_cookie=a0983b235246b309 in_seq=0 out_seq=0 2024-05-27T07:26:57.128 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:57.134+0000 7f11e6ffd700 10 client.15657 ms_handle_connect on v2:172.21.15.105:6834/3298498384 2024-05-27T07:26:57.128 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:57.135+0000 7f11e6ffd700 1 -- 172.21.15.52:0/4160672532 <== mds.3 v2:172.21.15.105:6834/3298498384 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x5568f31c83c0 con 0x5568f32bbaa0 2024-05-27T07:26:57.129 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:57.135+0000 7f11e6ffd700 10 client.15657 handle_command_reply: tid=0 2024-05-27T07:26:57.129 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:57.135+0000 7f121ca8bb80 1 client.15657 shutdown 2024-05-27T07:26:57.129 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:57.135+0000 7f121ca8bb80 1 -- 172.21.15.52:0/4160672532 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x5568f32a2630 msgr2=0x5568f331dd30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:26:57.129 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:57.135+0000 7f121ca8bb80 1 --2- 172.21.15.52:0/4160672532 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x5568f32a2630 0x5568f331dd30 secure :-1 s=READY pgs=483 cs=0 l=1 rev1=1 crypto rx=0x5568f3299e10 tx=0x7f11fc012d20 comp rx=0 tx=0).stop 2024-05-27T07:26:57.129 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:57.135+0000 7f121ca8bb80 1 -- 172.21.15.52:0/4160672532 shutdown_connections 2024-05-27T07:26:57.129 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:57.135+0000 7f121ca8bb80 1 --2- 172.21.15.52:0/4160672532 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5568f32a6b10 0x5568f32c0750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:26:57.129 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:57.135+0000 7f121ca8bb80 1 --2- 172.21.15.52:0/4160672532 >> [v2:172.21.15.105:6834/3298498384,v1:172.21.15.105:6835/3298498384] conn(0x5568f32bbaa0 0x5568f3328720 crc :-1 s=READY pgs=27 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:26:57.129 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:57.135+0000 7f121ca8bb80 1 --2- 172.21.15.52:0/4160672532 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5568f311fe40 0x5568f331d7f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:26:57.129 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:57.135+0000 7f11e6ffd700 0 client.15657 ms_handle_reset on v2:172.21.15.105:6834/3298498384 2024-05-27T07:26:57.129 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:57.135+0000 7f121ca8bb80 1 --2- 172.21.15.52:0/4160672532 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x5568f32a2630 0x5568f331dd30 unknown :-1 s=CLOSED pgs=483 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:26:57.129 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:57.135+0000 7f121ca8bb80 1 -- 172.21.15.52:0/4160672532 >> 172.21.15.52:0/4160672532 conn(0x5568f317aaf0 msgr2=0x5568f32a63e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:26:57.129 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:57.135+0000 7f121ca8bb80 1 -- 172.21.15.52:0/4160672532 shutdown_connections 2024-05-27T07:26:57.130 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:57.135+0000 7f121ca8bb80 1 -- 172.21.15.52:0/4160672532 wait complete. 2024-05-27T07:26:57.130 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:57.135+0000 7f121ca8bb80 20 client.15657 trim_cache size 0 max 16384 2024-05-27T07:26:57.130 INFO:teuthology.orchestra.run.smithi052.stdout:[] 2024-05-27T07:26:57.130 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:57.136+0000 7f120e076700 1 -- 172.21.15.52:0/3956247317 >> [v2:172.21.15.52:6800/1822318384,v1:172.21.15.52:6801/1822318384] conn(0x7f11f00b2b10 msgr2=0x7f11f00b5020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:26:57.131 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:57.136+0000 7f120e076700 1 --2- 172.21.15.52:0/3956247317 >> [v2:172.21.15.52:6800/1822318384,v1:172.21.15.52:6801/1822318384] conn(0x7f11f00b2b10 0x7f11f00b5020 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f11f8009990 tx=0x7f11f8000bc0 comp rx=0 tx=0).stop 2024-05-27T07:26:57.131 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:57.136+0000 7f120e076700 1 -- 172.21.15.52:0/3956247317 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f1208108750 msgr2=0x7f120819f550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:26:57.131 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:57.136+0000 7f120e076700 1 --2- 172.21.15.52:0/3956247317 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f1208108750 0x7f120819f550 secure :-1 s=READY pgs=479 cs=0 l=1 rev1=1 crypto rx=0x7f1200007d70 tx=0x7f120000a340 comp rx=0 tx=0).stop 2024-05-27T07:26:57.131 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:57.136+0000 7f120e076700 1 -- 172.21.15.52:0/3956247317 shutdown_connections 2024-05-27T07:26:57.131 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:57.136+0000 7f120e076700 1 --2- 172.21.15.52:0/3956247317 >> [v2:172.21.15.52:6800/1822318384,v1:172.21.15.52:6801/1822318384] conn(0x7f11f00b2b10 0x7f11f00b5020 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:26:57.131 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:57.136+0000 7f120e076700 1 --2- 172.21.15.52:0/3956247317 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f1208108750 0x7f120819f550 unknown :-1 s=CLOSED pgs=479 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:26:57.131 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:57.136+0000 7f120e076700 1 --2- 172.21.15.52:0/3956247317 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f12081034d0 0x7f120819ad20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:26:57.131 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:57.136+0000 7f120e076700 1 --2- 172.21.15.52:0/3956247317 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f1208107e00 0x7f120819b2c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:26:57.131 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:57.136+0000 7f120e076700 1 -- 172.21.15.52:0/3956247317 >> 172.21.15.52:0/3956247317 conn(0x7f12080fee40 msgr2=0x7f1208075cc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:26:57.131 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:57.136+0000 7f120e076700 1 -- 172.21.15.52:0/3956247317 shutdown_connections 2024-05-27T07:26:57.131 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:57.136+0000 7f120e076700 1 -- 172.21.15.52:0/3956247317 wait complete. 2024-05-27T07:26:57.841 DEBUG:teuthology.orchestra.run.smithi052:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:38f9d900c83c63799fbdbe61acc9a11b0d3554a6 shell --fsid 73df2636-1bf8-11ef-bc9b-c7b262605968 -- ceph tell mds.1:2 damage ls 2024-05-27T07:26:58.110 INFO:teuthology.orchestra.run.smithi052.stderr:Inferring config /var/lib/ceph/73df2636-1bf8-11ef-bc9b-c7b262605968/mon.a/config 2024-05-27T07:26:58.349 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:26:58 smithi052 ceph-mon[31955]: pgmap v33: 129 pgs: 129 active+clean; 1.3 MiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 1.7 KiB/s wr, 10 op/s 2024-05-27T07:26:58.351 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:26:58 smithi173 ceph-mon[37000]: pgmap v33: 129 pgs: 129 active+clean; 1.3 MiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 1.7 KiB/s wr, 10 op/s 2024-05-27T07:26:58.440 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:26:58 smithi105 ceph-mon[35977]: pgmap v33: 129 pgs: 129 active+clean; 1.3 MiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 1.7 KiB/s wr, 10 op/s 2024-05-27T07:26:59.872 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:59.882+0000 7f080a72d700 1 -- 172.21.15.52:0/1885457833 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f0804107de0 msgr2=0x7f08041081f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:26:59.872 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:59.882+0000 7f080a72d700 1 --2- 172.21.15.52:0/1885457833 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f0804107de0 0x7f08041081f0 secure :-1 s=READY pgs=484 cs=0 l=1 rev1=1 crypto rx=0x7f07f80099c0 tx=0x7f07f8009cd0 comp rx=0 tx=0).stop 2024-05-27T07:26:59.872 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:59.882+0000 7f080a72d700 1 -- 172.21.15.52:0/1885457833 shutdown_connections 2024-05-27T07:26:59.872 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:59.882+0000 7f080a72d700 1 --2- 172.21.15.52:0/1885457833 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f0804108730 0x7f0804110090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:26:59.872 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:59.882+0000 7f080a72d700 1 --2- 172.21.15.52:0/1885457833 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f08041034b0 0x7f08041038c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:26:59.872 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:59.882+0000 7f080a72d700 1 --2- 172.21.15.52:0/1885457833 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f0804107de0 0x7f08041081f0 unknown :-1 s=CLOSED pgs=484 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:26:59.872 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:59.882+0000 7f080a72d700 1 -- 172.21.15.52:0/1885457833 >> 172.21.15.52:0/1885457833 conn(0x7f08040fedb0 msgr2=0x7f08041011d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:26:59.872 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:59.882+0000 7f080a72d700 1 -- 172.21.15.52:0/1885457833 shutdown_connections 2024-05-27T07:26:59.872 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:59.882+0000 7f080a72d700 1 -- 172.21.15.52:0/1885457833 wait complete. 2024-05-27T07:26:59.874 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:59.883+0000 7f080a72d700 1 Processor -- start 2024-05-27T07:26:59.874 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:59.883+0000 7f080a72d700 1 -- start start 2024-05-27T07:26:59.874 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:59.883+0000 7f080a72d700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f08041034b0 0x7f0804199a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:26:59.874 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:59.883+0000 7f080a72d700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f0804107de0 0x7f0804199fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:26:59.874 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:59.883+0000 7f080a72d700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f0804108730 0x7f080419a500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:26:59.874 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:59.883+0000 7f080a72d700 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x7f0804113390 con 0x7f0804107de0 2024-05-27T07:26:59.874 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:59.883+0000 7f080a72d700 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f0804113210 con 0x7f08041034b0 2024-05-27T07:26:59.874 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:59.883+0000 7f080a72d700 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f0804113510 con 0x7f0804108730 2024-05-27T07:26:59.874 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:59.883+0000 7f08037fe700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f0804107de0 0x7f0804199fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:26:59.874 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:59.883+0000 7f08037fe700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f0804107de0 0x7f0804199fc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.52:37470/0 (socket says 172.21.15.52:37470) 2024-05-27T07:26:59.874 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:59.884+0000 7f08037fe700 1 -- 172.21.15.52:0/1360449853 learned_addr learned my addr 172.21.15.52:0/1360449853 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-05-27T07:26:59.874 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:59.884+0000 7f0803fff700 1 --2- 172.21.15.52:0/1360449853 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f08041034b0 0x7f0804199a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:26:59.875 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:59.884+0000 7f0808cca700 1 --2- 172.21.15.52:0/1360449853 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f0804108730 0x7f080419a500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:26:59.875 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:59.884+0000 7f08037fe700 1 -- 172.21.15.52:0/1360449853 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f0804108730 msgr2=0x7f080419a500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:26:59.875 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:59.884+0000 7f08037fe700 1 --2- 172.21.15.52:0/1360449853 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f0804108730 0x7f080419a500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:26:59.875 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:59.884+0000 7f08037fe700 1 -- 172.21.15.52:0/1360449853 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f08041034b0 msgr2=0x7f0804199a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:26:59.875 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:59.884+0000 7f08037fe700 1 --2- 172.21.15.52:0/1360449853 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f08041034b0 0x7f0804199a80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:26:59.875 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:59.884+0000 7f08037fe700 1 -- 172.21.15.52:0/1360449853 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f08041a9180 con 0x7f0804107de0 2024-05-27T07:26:59.875 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:59.884+0000 7f08037fe700 1 --2- 172.21.15.52:0/1360449853 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f0804107de0 0x7f0804199fc0 secure :-1 s=READY pgs=485 cs=0 l=1 rev1=1 crypto rx=0x7f07f8009f50 tx=0x7f07f80057c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:26:59.876 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:59.884+0000 7f08017fa700 1 -- 172.21.15.52:0/1360449853 <== mon.0 v2:172.21.15.52:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f07f8027070 con 0x7f0804107de0 2024-05-27T07:26:59.876 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:59.884+0000 7f08017fa700 1 -- 172.21.15.52:0/1360449853 <== mon.0 v2:172.21.15.52:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f07f8003f20 con 0x7f0804107de0 2024-05-27T07:26:59.876 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:59.884+0000 7f08017fa700 1 -- 172.21.15.52:0/1360449853 <== mon.0 v2:172.21.15.52:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f07f8022d10 con 0x7f0804107de0 2024-05-27T07:26:59.876 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:59.885+0000 7f080a72d700 1 -- 172.21.15.52:0/1360449853 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f08041a9410 con 0x7f0804107de0 2024-05-27T07:26:59.876 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:59.885+0000 7f080a72d700 1 -- 172.21.15.52:0/1360449853 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f08041a9810 con 0x7f0804107de0 2024-05-27T07:26:59.877 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:59.888+0000 7f08017fa700 1 -- 172.21.15.52:0/1360449853 <== mon.0 v2:172.21.15.52:3300/0 4 ==== mgrmap(e 24) ==== 149879+0+0 (secure 0 0 0) 0x7f07f802a070 con 0x7f0804107de0 2024-05-27T07:26:59.878 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:59.889+0000 7f08017fa700 1 --2- 172.21.15.52:0/1360449853 >> [v2:172.21.15.52:6800/1822318384,v1:172.21.15.52:6801/1822318384] conn(0x7f07ec0b2b10 0x7f07ec0b5020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:26:59.878 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:59.889+0000 7f08017fa700 1 -- 172.21.15.52:0/1360449853 <== mon.0 v2:172.21.15.52:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9379+0+0 (secure 0 0 0) 0x7f07f80e3a30 con 0x7f0804107de0 2024-05-27T07:26:59.878 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:59.889+0000 7f0803fff700 1 --2- 172.21.15.52:0/1360449853 >> [v2:172.21.15.52:6800/1822318384,v1:172.21.15.52:6801/1822318384] conn(0x7f07ec0b2b10 0x7f07ec0b5020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:26:59.879 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:59.889+0000 7f0803fff700 1 --2- 172.21.15.52:0/1360449853 >> [v2:172.21.15.52:6800/1822318384,v1:172.21.15.52:6801/1822318384] conn(0x7f07ec0b2b10 0x7f07ec0b5020 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f07f4007be0 tx=0x7f07f40073d0 comp rx=0 tx=0).ready entity=mgr.15333 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:26:59.879 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:59.890+0000 7f0819142b80 1 Processor -- start 2024-05-27T07:26:59.879 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:59.890+0000 7f0819142b80 1 -- start start 2024-05-27T07:26:59.881 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:59.890+0000 7f0819142b80 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5592868d1e40 0x559286a55710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:26:59.881 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:59.890+0000 7f0819142b80 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x559286a55c50 0x559286a4fa40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:26:59.881 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:59.890+0000 7f0819142b80 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x559286a4ff80 0x559286a50390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:26:59.881 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:59.890+0000 7f0819142b80 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x55928697ec90 con 0x559286a55c50 2024-05-27T07:26:59.881 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:59.890+0000 7f0819142b80 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x55928696e830 con 0x559286a4ff80 2024-05-27T07:26:59.881 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:59.890+0000 7f0819142b80 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x559286972870 con 0x5592868d1e40 2024-05-27T07:26:59.882 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:59.890+0000 7f08037fe700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x559286a55c50 0x559286a4fa40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:26:59.882 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:59.890+0000 7f08037fe700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x559286a55c50 0x559286a4fa40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.52:37480/0 (socket says 172.21.15.52:37480) 2024-05-27T07:26:59.882 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:59.890+0000 7f08037fe700 1 -- 172.21.15.52:0/888926065 learned_addr learned my addr 172.21.15.52:0/888926065 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-05-27T07:26:59.882 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:59.891+0000 7f0808cca700 1 --2- 172.21.15.52:0/888926065 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x559286a4ff80 0x559286a50390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:26:59.882 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:59.891+0000 7f0803fff700 1 --2- 172.21.15.52:0/888926065 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5592868d1e40 0x559286a55710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:26:59.882 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:59.891+0000 7f08037fe700 1 -- 172.21.15.52:0/888926065 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5592868d1e40 msgr2=0x559286a55710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:26:59.882 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:59.891+0000 7f08037fe700 1 --2- 172.21.15.52:0/888926065 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5592868d1e40 0x559286a55710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:26:59.882 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:59.891+0000 7f08037fe700 1 -- 172.21.15.52:0/888926065 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x559286a4ff80 msgr2=0x559286a50390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:26:59.882 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:59.891+0000 7f08037fe700 1 --2- 172.21.15.52:0/888926065 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x559286a4ff80 0x559286a50390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:26:59.882 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:59.891+0000 7f08037fe700 1 -- 172.21.15.52:0/888926065 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5592868be8d0 con 0x559286a55c50 2024-05-27T07:26:59.882 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:59.891+0000 7f0803fff700 1 --2- 172.21.15.52:0/888926065 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5592868d1e40 0x559286a55710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-27T07:26:59.882 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:59.891+0000 7f08037fe700 1 --2- 172.21.15.52:0/888926065 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x559286a55c50 0x559286a4fa40 secure :-1 s=READY pgs=486 cs=0 l=1 rev1=1 crypto rx=0x7f07f80aa780 tx=0x7f07f80aa270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fd96748b4b24e914 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:26:59.883 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:59.891+0000 7f07ea7fc700 1 -- 172.21.15.52:0/888926065 <== mon.0 v2:172.21.15.52:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f07f80b10f0 con 0x559286a55c50 2024-05-27T07:26:59.883 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:59.891+0000 7f07ea7fc700 1 -- 172.21.15.52:0/888926065 <== mon.0 v2:172.21.15.52:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f07f80aa300 con 0x559286a55c50 2024-05-27T07:26:59.883 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:59.891+0000 7f07ea7fc700 1 -- 172.21.15.52:0/888926065 <== mon.0 v2:172.21.15.52:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f07f80aacb0 con 0x559286a55c50 2024-05-27T07:26:59.885 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:59.892+0000 7f0819142b80 1 -- 172.21.15.52:0/888926065 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x559286a55c50 msgr2=0x559286a4fa40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:26:59.886 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:59.892+0000 7f0819142b80 1 --2- 172.21.15.52:0/888926065 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x559286a55c50 0x559286a4fa40 secure :-1 s=READY pgs=486 cs=0 l=1 rev1=1 crypto rx=0x7f07f80aa780 tx=0x7f07f80aa270 comp rx=0 tx=0).stop 2024-05-27T07:26:59.886 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:59.892+0000 7f0819142b80 1 -- 172.21.15.52:0/888926065 shutdown_connections 2024-05-27T07:26:59.886 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:59.892+0000 7f0819142b80 1 --2- 172.21.15.52:0/888926065 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x559286a4ff80 0x559286a50390 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:26:59.886 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:59.892+0000 7f0819142b80 1 --2- 172.21.15.52:0/888926065 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5592868d1e40 0x559286a55710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:26:59.886 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:59.892+0000 7f0819142b80 1 --2- 172.21.15.52:0/888926065 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x559286a55c50 0x559286a4fa40 unknown :-1 s=CLOSED pgs=486 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:26:59.886 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:59.892+0000 7f0819142b80 1 -- 172.21.15.52:0/888926065 >> 172.21.15.52:0/888926065 conn(0x55928692caf0 msgr2=0x559286a4a640 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:26:59.886 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:59.892+0000 7f0819142b80 1 -- 172.21.15.52:0/888926065 shutdown_connections 2024-05-27T07:26:59.886 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:59.892+0000 7f0819142b80 1 -- 172.21.15.52:0/888926065 wait complete. 2024-05-27T07:26:59.886 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:59.893+0000 7f0819142b80 1 -- start start 2024-05-27T07:26:59.886 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:59.893+0000 7f0819142b80 1 Processor -- start 2024-05-27T07:26:59.886 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:59.893+0000 7f0819142b80 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x5592868d1e40 0x559286a59390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:26:59.887 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:59.893+0000 7f0819142b80 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x559286a4ff80 0x559286a598d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:26:59.887 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:59.893+0000 7f0819142b80 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x559286a55c50 0x559286a59e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:26:59.887 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:59.893+0000 7f0819142b80 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x559286874fa0 con 0x5592868d1e40 2024-05-27T07:26:59.887 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:59.893+0000 7f0819142b80 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x5592869107c0 con 0x559286a4ff80 2024-05-27T07:26:59.887 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:59.893+0000 7f0819142b80 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x559286a5c500 con 0x559286a55c50 2024-05-27T07:26:59.887 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:59.893+0000 7f0819142b80 10 client.0 _ll_register_callbacks cb 0x559286909920 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T07:26:59.887 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:59.894+0000 7f0803fff700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x5592868d1e40 0x559286a59390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:26:59.887 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:59.894+0000 7f0803fff700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x5592868d1e40 0x559286a59390 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.52:37496/0 (socket says 172.21.15.52:37496) 2024-05-27T07:26:59.887 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:59.894+0000 7f0803fff700 1 -- 172.21.15.52:0/2402637421 learned_addr learned my addr 172.21.15.52:0/2402637421 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-05-27T07:26:59.887 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:59.894+0000 7f08037fe700 1 --2- 172.21.15.52:0/2402637421 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x559286a4ff80 0x559286a598d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:26:59.887 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:59.894+0000 7f0808cca700 1 --2- 172.21.15.52:0/2402637421 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x559286a55c50 0x559286a59e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:26:59.887 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:59.894+0000 7f0803fff700 1 -- 172.21.15.52:0/2402637421 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x559286a55c50 msgr2=0x559286a59e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:26:59.888 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:59.894+0000 7f0803fff700 1 --2- 172.21.15.52:0/2402637421 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x559286a55c50 0x559286a59e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:26:59.888 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:59.894+0000 7f0803fff700 1 -- 172.21.15.52:0/2402637421 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x559286a4ff80 msgr2=0x559286a598d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:26:59.888 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:59.894+0000 7f0803fff700 1 --2- 172.21.15.52:0/2402637421 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x559286a4ff80 0x559286a598d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:26:59.888 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:59.894+0000 7f0803fff700 1 -- 172.21.15.52:0/2402637421 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55928690d690 con 0x5592868d1e40 2024-05-27T07:26:59.888 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:59.894+0000 7f0808cca700 1 --2- 172.21.15.52:0/2402637421 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x559286a55c50 0x559286a59e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-27T07:26:59.888 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:59.894+0000 7f0803fff700 1 --2- 172.21.15.52:0/2402637421 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x5592868d1e40 0x559286a59390 secure :-1 s=READY pgs=487 cs=0 l=1 rev1=1 crypto rx=0x7f07f4004360 tx=0x7f07f40047c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=17f045d65f314db6 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:26:59.888 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:59.894+0000 7f07e37fe700 10 client.0 ms_handle_connect on v2:172.21.15.52:3300/0 2024-05-27T07:26:59.888 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:59.894+0000 7f07e37fe700 1 -- 172.21.15.52:0/2402637421 <== mon.0 v2:172.21.15.52:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f07f4024070 con 0x5592868d1e40 2024-05-27T07:26:59.888 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:59.894+0000 7f07e37fe700 1 -- 172.21.15.52:0/2402637421 <== mon.0 v2:172.21.15.52:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 9379+0+0 (secure 0 0 0) 0x7f07f4003d10 con 0x5592868d1e40 2024-05-27T07:26:59.888 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:59.894+0000 7f0819142b80 1 -- 172.21.15.52:0/2402637421 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f07f80aa300 con 0x5592868d1e40 2024-05-27T07:26:59.888 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:59.895+0000 7f07e37fe700 1 -- 172.21.15.52:0/2402637421 <== mon.0 v2:172.21.15.52:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7f07f4030050 con 0x5592868d1e40 2024-05-27T07:26:59.888 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:59.895+0000 7f0819142b80 10 client.15669 fetch_fsmap learned FSMap version 30 2024-05-27T07:26:59.889 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:59.895+0000 7f0819142b80 1 -- 172.21.15.52:0/2402637421 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x559286a66f50 con 0x5592868d1e40 2024-05-27T07:26:59.889 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:59.895+0000 7f07e37fe700 1 -- 172.21.15.52:0/2402637421 <== mon.0 v2:172.21.15.52:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f07f80aa300 con 0x5592868d1e40 2024-05-27T07:26:59.889 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:59.895+0000 7f07e37fe700 1 -- 172.21.15.52:0/2402637421 <== mon.0 v2:172.21.15.52:3300/0 5 ==== fsmap(e 30) ==== 8362+0+0 (secure 0 0 0) 0x7f07f4020410 con 0x5592868d1e40 2024-05-27T07:26:59.889 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:59.895+0000 7f0819142b80 10 client.15669 fetch_fsmap finished waiting for FSMap version 30 2024-05-27T07:26:59.889 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:59.895+0000 7f0819142b80 10 client.15669 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.i 2024-05-27T07:26:59.889 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:59.895+0000 7f0819142b80 20 client.15669 populate_metadata read hostname 'smithi052' 2024-05-27T07:26:59.889 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:59.895+0000 7f0819142b80 1 --2- 172.21.15.52:0/2402637421 >> [v2:172.21.15.173:6838/564778293,v1:172.21.15.173:6839/564778293] conn(0x559286a69530 0x559286a6b950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:26:59.889 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:59.895+0000 7f0819142b80 4 client.15669 mds_command: new command op to 24475 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-27T07:26:59.889 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:59.895+0000 7f0819142b80 1 -- 172.21.15.52:0/2402637421 --> [v2:172.21.15.173:6838/564778293,v1:172.21.15.173:6839/564778293] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x559286a1b9f0 con 0x559286a69530 2024-05-27T07:26:59.889 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:59.896+0000 7f08037fe700 1 --2- 172.21.15.52:0/2402637421 >> [v2:172.21.15.173:6838/564778293,v1:172.21.15.173:6839/564778293] conn(0x559286a69530 0x559286a6b950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:26:59.889 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:59.896+0000 7f07e37fe700 10 client.15669 ms_handle_connect on v2:172.21.15.173:6838/564778293 2024-05-27T07:26:59.889 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:59.896+0000 7f08037fe700 1 --2- 172.21.15.52:0/2402637421 >> [v2:172.21.15.173:6838/564778293,v1:172.21.15.173:6839/564778293] conn(0x559286a69530 0x559286a6b950 crc :-1 s=READY pgs=31 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=8d3c9a2456679878 server_cookie=f9c7a680ddedf6a3 in_seq=0 out_seq=0 2024-05-27T07:26:59.891 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:59.897+0000 7f07e37fe700 1 -- 172.21.15.52:0/2402637421 <== mds.2 v2:172.21.15.173:6838/564778293 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24041 (crc 0 0 0) 0x559286a1b9f0 con 0x559286a69530 2024-05-27T07:26:59.891 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:59.897+0000 7f07e37fe700 10 client.15669 handle_command_reply: tid=0 2024-05-27T07:26:59.891 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:59.897+0000 7f0819142b80 1 client.15669 shutdown 2024-05-27T07:26:59.891 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:59.897+0000 7f0819142b80 1 -- 172.21.15.52:0/2402637421 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x5592868d1e40 msgr2=0x559286a59390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:26:59.891 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:59.897+0000 7f0819142b80 1 --2- 172.21.15.52:0/2402637421 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x5592868d1e40 0x559286a59390 secure :-1 s=READY pgs=487 cs=0 l=1 rev1=1 crypto rx=0x7f07f4004360 tx=0x7f07f40047c0 comp rx=0 tx=0).stop 2024-05-27T07:26:59.891 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:59.897+0000 7f0819142b80 1 -- 172.21.15.52:0/2402637421 shutdown_connections 2024-05-27T07:26:59.891 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:59.897+0000 7f0819142b80 1 --2- 172.21.15.52:0/2402637421 >> [v2:172.21.15.173:6838/564778293,v1:172.21.15.173:6839/564778293] conn(0x559286a69530 0x559286a6b950 crc :-1 s=READY pgs=31 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:26:59.891 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:59.897+0000 7f0819142b80 1 --2- 172.21.15.52:0/2402637421 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x559286a55c50 0x559286a59e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:26:59.891 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:59.897+0000 7f0819142b80 1 --2- 172.21.15.52:0/2402637421 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x5592868d1e40 0x559286a59390 unknown :-1 s=CLOSED pgs=487 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:26:59.891 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:59.897+0000 7f07e37fe700 0 client.15669 ms_handle_reset on v2:172.21.15.173:6838/564778293 2024-05-27T07:26:59.891 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:59.897+0000 7f0819142b80 1 --2- 172.21.15.52:0/2402637421 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x559286a4ff80 0x559286a598d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:26:59.891 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:59.897+0000 7f0819142b80 1 -- 172.21.15.52:0/2402637421 >> 172.21.15.52:0/2402637421 conn(0x55928692caf0 msgr2=0x559286a4ea30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:26:59.892 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:59.897+0000 7f0819142b80 1 -- 172.21.15.52:0/2402637421 shutdown_connections 2024-05-27T07:26:59.892 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:59.897+0000 7f0819142b80 1 -- 172.21.15.52:0/2402637421 wait complete. 2024-05-27T07:26:59.892 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:59.898+0000 7f0819142b80 20 client.15669 trim_cache size 0 max 16384 2024-05-27T07:26:59.904 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:59.913+0000 7f0819142b80 1 Processor -- start 2024-05-27T07:26:59.904 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:59.913+0000 7f0819142b80 1 -- start start 2024-05-27T07:26:59.904 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:59.913+0000 7f0819142b80 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x5592868d1e40 0x559286a4fa40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:26:59.904 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:59.913+0000 7f0819142b80 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x559286a4ff80 0x559286a65560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:26:59.904 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:59.913+0000 7f0819142b80 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x559286a55c50 0x559286a53720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:26:59.904 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:59.914+0000 7f0819142b80 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x7f08041b2560 con 0x5592868d1e40 2024-05-27T07:26:59.904 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:59.914+0000 7f0819142b80 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f08041b40a0 con 0x559286a55c50 2024-05-27T07:26:59.904 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:59.914+0000 7f0819142b80 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f08041b2c30 con 0x559286a4ff80 2024-05-27T07:26:59.904 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:59.914+0000 7f0803fff700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x5592868d1e40 0x559286a4fa40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:26:59.905 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:59.914+0000 7f0803fff700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x5592868d1e40 0x559286a4fa40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.52:37500/0 (socket says 172.21.15.52:37500) 2024-05-27T07:26:59.905 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:59.914+0000 7f0803fff700 1 -- 172.21.15.52:0/1202372472 learned_addr learned my addr 172.21.15.52:0/1202372472 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-05-27T07:26:59.905 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:59.914+0000 7f0808cca700 1 --2- 172.21.15.52:0/1202372472 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x559286a55c50 0x559286a53720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:26:59.905 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:59.914+0000 7f08037fe700 1 --2- 172.21.15.52:0/1202372472 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x559286a4ff80 0x559286a65560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:26:59.905 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:59.914+0000 7f0803fff700 1 -- 172.21.15.52:0/1202372472 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x559286a4ff80 msgr2=0x559286a65560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:26:59.905 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:59.914+0000 7f0803fff700 1 --2- 172.21.15.52:0/1202372472 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x559286a4ff80 0x559286a65560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:26:59.905 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:59.914+0000 7f0803fff700 1 -- 172.21.15.52:0/1202372472 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x559286a55c50 msgr2=0x559286a53720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:26:59.905 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:59.914+0000 7f0803fff700 1 --2- 172.21.15.52:0/1202372472 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x559286a55c50 0x559286a53720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:26:59.905 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:59.914+0000 7f0803fff700 1 -- 172.21.15.52:0/1202372472 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x559286a57900 con 0x5592868d1e40 2024-05-27T07:26:59.905 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:59.914+0000 7f0808cca700 1 --2- 172.21.15.52:0/1202372472 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x559286a55c50 0x559286a53720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-27T07:26:59.905 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:59.914+0000 7f0803fff700 1 --2- 172.21.15.52:0/1202372472 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x5592868d1e40 0x559286a4fa40 secure :-1 s=READY pgs=488 cs=0 l=1 rev1=1 crypto rx=0x7f07f402e040 tx=0x7f07f40042d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fb777d6555b903eb server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:26:59.905 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:59.914+0000 7f07e9ffb700 1 -- 172.21.15.52:0/1202372472 <== mon.0 v2:172.21.15.52:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f07f4024040 con 0x5592868d1e40 2024-05-27T07:26:59.905 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:59.914+0000 7f07e9ffb700 1 -- 172.21.15.52:0/1202372472 <== mon.0 v2:172.21.15.52:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f07f4038070 con 0x5592868d1e40 2024-05-27T07:26:59.906 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:59.914+0000 7f07e9ffb700 1 -- 172.21.15.52:0/1202372472 <== mon.0 v2:172.21.15.52:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f07f4011440 con 0x5592868d1e40 2024-05-27T07:26:59.909 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:59.915+0000 7f0819142b80 1 -- 172.21.15.52:0/1202372472 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x5592868d1e40 msgr2=0x559286a4fa40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:26:59.909 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:59.915+0000 7f0819142b80 1 --2- 172.21.15.52:0/1202372472 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x5592868d1e40 0x559286a4fa40 secure :-1 s=READY pgs=488 cs=0 l=1 rev1=1 crypto rx=0x7f07f402e040 tx=0x7f07f40042d0 comp rx=0 tx=0).stop 2024-05-27T07:26:59.909 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:59.915+0000 7f0819142b80 1 -- 172.21.15.52:0/1202372472 shutdown_connections 2024-05-27T07:26:59.909 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:59.915+0000 7f0819142b80 1 --2- 172.21.15.52:0/1202372472 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x559286a55c50 0x559286a53720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:26:59.910 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:59.915+0000 7f0819142b80 1 --2- 172.21.15.52:0/1202372472 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x5592868d1e40 0x559286a4fa40 unknown :-1 s=CLOSED pgs=488 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:26:59.910 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:59.915+0000 7f0819142b80 1 --2- 172.21.15.52:0/1202372472 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x559286a4ff80 0x559286a65560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:26:59.910 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:59.915+0000 7f0819142b80 1 -- 172.21.15.52:0/1202372472 >> 172.21.15.52:0/1202372472 conn(0x55928692caf0 msgr2=0x559286a53010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:26:59.910 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:59.915+0000 7f0819142b80 1 -- 172.21.15.52:0/1202372472 shutdown_connections 2024-05-27T07:26:59.910 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:59.915+0000 7f0819142b80 1 -- 172.21.15.52:0/1202372472 wait complete. 2024-05-27T07:26:59.910 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:59.916+0000 7f0819142b80 1 -- start start 2024-05-27T07:26:59.910 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:59.916+0000 7f0819142b80 1 Processor -- start 2024-05-27T07:26:59.910 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:59.916+0000 7f0819142b80 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5592868d1e40 0x559286a61950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:26:59.910 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:59.916+0000 7f0819142b80 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x559286a4ff80 0x559286a61e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:26:59.910 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:59.916+0000 7f0819142b80 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x559286a55c50 0x559286a70540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:26:59.910 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:59.916+0000 7f0819142b80 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x7f08041b39d0 con 0x559286a55c50 2024-05-27T07:26:59.910 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:59.916+0000 7f0819142b80 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f08041b3300 con 0x559286a4ff80 2024-05-27T07:26:59.910 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:59.916+0000 7f0819142b80 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f08041b0500 con 0x5592868d1e40 2024-05-27T07:26:59.911 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:59.916+0000 7f0819142b80 10 client.0 _ll_register_callbacks cb 0x559286909920 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T07:26:59.911 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:59.917+0000 7f0808cca700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x559286a55c50 0x559286a70540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:26:59.911 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:59.917+0000 7f0808cca700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x559286a55c50 0x559286a70540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.52:37512/0 (socket says 172.21.15.52:37512) 2024-05-27T07:26:59.911 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:59.917+0000 7f0808cca700 1 -- 172.21.15.52:0/1701537254 learned_addr learned my addr 172.21.15.52:0/1701537254 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-05-27T07:26:59.911 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:59.917+0000 7f0803fff700 1 --2- 172.21.15.52:0/1701537254 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5592868d1e40 0x559286a61950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:26:59.911 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:59.917+0000 7f08037fe700 1 --2- 172.21.15.52:0/1701537254 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x559286a4ff80 0x559286a61e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:26:59.911 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:59.917+0000 7f0808cca700 1 -- 172.21.15.52:0/1701537254 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5592868d1e40 msgr2=0x559286a61950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:26:59.911 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:59.917+0000 7f0808cca700 1 --2- 172.21.15.52:0/1701537254 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5592868d1e40 0x559286a61950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:26:59.911 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:59.917+0000 7f0808cca700 1 -- 172.21.15.52:0/1701537254 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x559286a4ff80 msgr2=0x559286a61e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:26:59.911 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:59.917+0000 7f0808cca700 1 --2- 172.21.15.52:0/1701537254 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x559286a4ff80 0x559286a61e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:26:59.911 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:59.917+0000 7f0808cca700 1 -- 172.21.15.52:0/1701537254 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({osdmap=0}) -- 0x559286a70bb0 con 0x559286a55c50 2024-05-27T07:26:59.911 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:59.917+0000 7f0808cca700 1 --2- 172.21.15.52:0/1701537254 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x559286a55c50 0x559286a70540 secure :-1 s=READY pgs=489 cs=0 l=1 rev1=1 crypto rx=0x7f07fc007cc0 tx=0x7f07fc009f80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a4beb00fb3398928 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:26:59.912 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:59.917+0000 7f07e37fe700 10 client.0 ms_handle_connect on v2:172.21.15.52:3300/0 2024-05-27T07:26:59.912 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:59.917+0000 7f07e37fe700 1 -- 172.21.15.52:0/1701537254 <== mon.0 v2:172.21.15.52:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f07fc01c070 con 0x559286a55c50 2024-05-27T07:26:59.912 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:59.917+0000 7f07e37fe700 1 -- 172.21.15.52:0/1701537254 <== mon.0 v2:172.21.15.52:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 9379+0+0 (secure 0 0 0) 0x7f07fc004db0 con 0x559286a55c50 2024-05-27T07:26:59.912 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:59.917+0000 7f0819142b80 1 -- 172.21.15.52:0/1701537254 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f07f4038070 con 0x559286a55c50 2024-05-27T07:26:59.912 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:59.918+0000 7f07e37fe700 1 -- 172.21.15.52:0/1701537254 <== mon.0 v2:172.21.15.52:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7f07fc029050 con 0x559286a55c50 2024-05-27T07:26:59.912 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:59.918+0000 7f0819142b80 10 client.15675 fetch_fsmap learned FSMap version 30 2024-05-27T07:26:59.912 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:59.918+0000 7f0819142b80 1 -- 172.21.15.52:0/1701537254 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x559286ad3940 con 0x559286a55c50 2024-05-27T07:26:59.912 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:59.918+0000 7f07e37fe700 1 -- 172.21.15.52:0/1701537254 <== mon.0 v2:172.21.15.52:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f07f4038070 con 0x559286a55c50 2024-05-27T07:26:59.912 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:59.918+0000 7f07e37fe700 1 -- 172.21.15.52:0/1701537254 <== mon.0 v2:172.21.15.52:3300/0 5 ==== fsmap(e 30) ==== 8362+0+0 (secure 0 0 0) 0x7f07fc018420 con 0x559286a55c50 2024-05-27T07:26:59.912 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:59.918+0000 7f0819142b80 10 client.15675 fetch_fsmap finished waiting for FSMap version 30 2024-05-27T07:26:59.912 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:59.918+0000 7f0819142b80 10 client.15675 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.i 2024-05-27T07:26:59.912 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:59.918+0000 7f0819142b80 20 client.15675 populate_metadata read hostname 'smithi052' 2024-05-27T07:26:59.912 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:59.918+0000 7f0819142b80 1 --2- 172.21.15.52:0/1701537254 >> [v2:172.21.15.173:6838/564778293,v1:172.21.15.173:6839/564778293] conn(0x559286a69530 0x559286ad60f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:26:59.913 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:59.918+0000 7f0819142b80 4 client.15675 mds_command: new command op to 24475 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-05-27T07:26:59.913 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:59.918+0000 7f0819142b80 1 -- 172.21.15.52:0/1701537254 --> [v2:172.21.15.173:6838/564778293,v1:172.21.15.173:6839/564778293] -- command(tid 0: {"prefix": "damage ls"}) -- 0x559286980be0 con 0x559286a69530 2024-05-27T07:26:59.913 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:59.918+0000 7f0803fff700 1 --2- 172.21.15.52:0/1701537254 >> [v2:172.21.15.173:6838/564778293,v1:172.21.15.173:6839/564778293] conn(0x559286a69530 0x559286ad60f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:26:59.913 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:59.919+0000 7f07e37fe700 10 client.15675 ms_handle_connect on v2:172.21.15.173:6838/564778293 2024-05-27T07:26:59.913 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:59.919+0000 7f0803fff700 1 --2- 172.21.15.52:0/1701537254 >> [v2:172.21.15.173:6838/564778293,v1:172.21.15.173:6839/564778293] conn(0x559286a69530 0x559286ad60f0 crc :-1 s=READY pgs=32 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=c17db54fc2216e71 server_cookie=95002fa02a6f08a1 in_seq=0 out_seq=0 2024-05-27T07:26:59.913 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:59.919+0000 7f07e37fe700 1 -- 172.21.15.52:0/1701537254 <== mds.2 v2:172.21.15.173:6838/564778293 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x559286980be0 con 0x559286a69530 2024-05-27T07:26:59.913 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:59.919+0000 7f07e37fe700 10 client.15675 handle_command_reply: tid=0 2024-05-27T07:26:59.913 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:59.919+0000 7f0819142b80 1 client.15675 shutdown 2024-05-27T07:26:59.913 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:59.919+0000 7f0819142b80 1 -- 172.21.15.52:0/1701537254 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x559286a55c50 msgr2=0x559286a70540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:26:59.913 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:59.919+0000 7f0819142b80 1 --2- 172.21.15.52:0/1701537254 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x559286a55c50 0x559286a70540 secure :-1 s=READY pgs=489 cs=0 l=1 rev1=1 crypto rx=0x7f07fc007cc0 tx=0x7f07fc009f80 comp rx=0 tx=0).stop 2024-05-27T07:26:59.913 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:59.919+0000 7f0819142b80 1 -- 172.21.15.52:0/1701537254 shutdown_connections 2024-05-27T07:26:59.913 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:59.919+0000 7f0819142b80 1 --2- 172.21.15.52:0/1701537254 >> [v2:172.21.15.173:6838/564778293,v1:172.21.15.173:6839/564778293] conn(0x559286a69530 0x559286ad60f0 crc :-1 s=READY pgs=32 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:26:59.913 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:59.919+0000 7f0819142b80 1 --2- 172.21.15.52:0/1701537254 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x559286a55c50 0x559286a70540 unknown :-1 s=CLOSED pgs=489 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:26:59.914 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:59.919+0000 7f0819142b80 1 --2- 172.21.15.52:0/1701537254 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5592868d1e40 0x559286a61950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:26:59.914 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:59.919+0000 7f0819142b80 1 --2- 172.21.15.52:0/1701537254 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x559286a4ff80 0x559286a61e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:26:59.914 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:59.919+0000 7f07e37fe700 0 client.15675 ms_handle_reset on v2:172.21.15.173:6838/564778293 2024-05-27T07:26:59.914 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:59.919+0000 7f0819142b80 1 -- 172.21.15.52:0/1701537254 >> 172.21.15.52:0/1701537254 conn(0x55928692caf0 msgr2=0x559286a54440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:26:59.914 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:59.919+0000 7f0819142b80 1 -- 172.21.15.52:0/1701537254 shutdown_connections 2024-05-27T07:26:59.914 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:59.919+0000 7f0819142b80 1 -- 172.21.15.52:0/1701537254 wait complete. 2024-05-27T07:26:59.914 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:59.920+0000 7f0819142b80 20 client.15675 trim_cache size 0 max 16384 2024-05-27T07:26:59.914 INFO:teuthology.orchestra.run.smithi052.stdout:[] 2024-05-27T07:26:59.915 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:59.920+0000 7f080a72d700 1 -- 172.21.15.52:0/1360449853 >> [v2:172.21.15.52:6800/1822318384,v1:172.21.15.52:6801/1822318384] conn(0x7f07ec0b2b10 msgr2=0x7f07ec0b5020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:26:59.915 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:59.920+0000 7f080a72d700 1 --2- 172.21.15.52:0/1360449853 >> [v2:172.21.15.52:6800/1822318384,v1:172.21.15.52:6801/1822318384] conn(0x7f07ec0b2b10 0x7f07ec0b5020 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f07f4007be0 tx=0x7f07f40073d0 comp rx=0 tx=0).stop 2024-05-27T07:26:59.915 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:59.920+0000 7f080a72d700 1 -- 172.21.15.52:0/1360449853 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f0804107de0 msgr2=0x7f0804199fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:26:59.915 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:59.920+0000 7f080a72d700 1 --2- 172.21.15.52:0/1360449853 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f0804107de0 0x7f0804199fc0 secure :-1 s=READY pgs=485 cs=0 l=1 rev1=1 crypto rx=0x7f07f8009f50 tx=0x7f07f80057c0 comp rx=0 tx=0).stop 2024-05-27T07:26:59.915 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:59.920+0000 7f080a72d700 1 -- 172.21.15.52:0/1360449853 shutdown_connections 2024-05-27T07:26:59.915 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:59.920+0000 7f080a72d700 1 --2- 172.21.15.52:0/1360449853 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f0804108730 0x7f080419a500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:26:59.915 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:59.920+0000 7f080a72d700 1 --2- 172.21.15.52:0/1360449853 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f08041034b0 0x7f0804199a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:26:59.915 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:59.920+0000 7f080a72d700 1 --2- 172.21.15.52:0/1360449853 >> [v2:172.21.15.52:6800/1822318384,v1:172.21.15.52:6801/1822318384] conn(0x7f07ec0b2b10 0x7f07ec0b5020 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:26:59.916 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:59.920+0000 7f080a72d700 1 --2- 172.21.15.52:0/1360449853 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f0804107de0 0x7f0804199fc0 unknown :-1 s=CLOSED pgs=485 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:26:59.916 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:59.920+0000 7f080a72d700 1 -- 172.21.15.52:0/1360449853 >> 172.21.15.52:0/1360449853 conn(0x7f08040fedb0 msgr2=0x7f0804109690 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:26:59.916 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:59.920+0000 7f080a72d700 1 -- 172.21.15.52:0/1360449853 shutdown_connections 2024-05-27T07:26:59.916 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:26:59.920+0000 7f080a72d700 1 -- 172.21.15.52:0/1360449853 wait complete. 2024-05-27T07:27:00.135 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:27:00 smithi052 ceph-mon[31955]: pgmap v34: 129 pgs: 129 active+clean; 1.3 MiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 7.2 KiB/s rd, 1.7 KiB/s wr, 14 op/s 2024-05-27T07:27:00.351 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:27:00 smithi173 ceph-mon[37000]: pgmap v34: 129 pgs: 129 active+clean; 1.3 MiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 7.2 KiB/s rd, 1.7 KiB/s wr, 14 op/s 2024-05-27T07:27:00.439 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:27:00 smithi105 ceph-mon[35977]: pgmap v34: 129 pgs: 129 active+clean; 1.3 MiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 7.2 KiB/s rd, 1.7 KiB/s wr, 14 op/s 2024-05-27T07:27:00.479 INFO:journalctl@ceph.mgr.x.smithi052.stdout:May 27 07:27:00 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mgr-x[32184]: 2024-05-27T07:27:00.145+0000 7fd0a309d700 -1 client.0 error registering admin socket command: (17) File exists 2024-05-27T07:27:00.479 INFO:journalctl@ceph.mgr.x.smithi052.stdout:May 27 07:27:00 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mgr-x[32184]: 2024-05-27T07:27:00.145+0000 7fd0a309d700 -1 client.0 error registering admin socket command: (17) File exists 2024-05-27T07:27:00.479 INFO:journalctl@ceph.mgr.x.smithi052.stdout:May 27 07:27:00 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mgr-x[32184]: 2024-05-27T07:27:00.145+0000 7fd0a309d700 -1 client.0 error registering admin socket command: (17) File exists 2024-05-27T07:27:00.479 INFO:journalctl@ceph.mgr.x.smithi052.stdout:May 27 07:27:00 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mgr-x[32184]: 2024-05-27T07:27:00.145+0000 7fd0a309d700 -1 client.0 error registering admin socket command: (17) File exists 2024-05-27T07:27:00.479 INFO:journalctl@ceph.mgr.x.smithi052.stdout:May 27 07:27:00 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mgr-x[32184]: 2024-05-27T07:27:00.145+0000 7fd0a309d700 -1 client.0 error registering admin socket command: (17) File exists 2024-05-27T07:27:00.697 DEBUG:teuthology.orchestra.run.smithi052:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:38f9d900c83c63799fbdbe61acc9a11b0d3554a6 shell --fsid 73df2636-1bf8-11ef-bc9b-c7b262605968 -- ceph tell mds.1:1 damage ls 2024-05-27T07:27:00.957 INFO:teuthology.orchestra.run.smithi052.stderr:Inferring config /var/lib/ceph/73df2636-1bf8-11ef-bc9b-c7b262605968/mon.a/config 2024-05-27T07:27:02.352 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:27:02 smithi173 ceph-mon[37000]: pgmap v35: 129 pgs: 129 active+clean; 1.3 MiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 1.7 KiB/s wr, 10 op/s 2024-05-27T07:27:02.352 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:27:02 smithi173 ceph-mon[37000]: mgrmap e25: x(active, since 64s), standbys: z, y 2024-05-27T07:27:02.361 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:27:02 smithi052 ceph-mon[31955]: pgmap v35: 129 pgs: 129 active+clean; 1.3 MiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 1.7 KiB/s wr, 10 op/s 2024-05-27T07:27:02.361 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:27:02 smithi052 ceph-mon[31955]: mgrmap e25: x(active, since 64s), standbys: z, y 2024-05-27T07:27:02.439 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:27:02 smithi105 ceph-mon[35977]: pgmap v35: 129 pgs: 129 active+clean; 1.3 MiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 1.7 KiB/s wr, 10 op/s 2024-05-27T07:27:02.440 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:27:02 smithi105 ceph-mon[35977]: mgrmap e25: x(active, since 64s), standbys: z, y 2024-05-27T07:27:02.743 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:02.753+0000 7f8f0b3a4700 1 -- 172.21.15.52:0/3644439704 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f8f041030d0 msgr2=0x7f8f04111160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:27:02.743 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:02.753+0000 7f8f0b3a4700 1 --2- 172.21.15.52:0/3644439704 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f8f041030d0 0x7f8f04111160 secure :-1 s=READY pgs=492 cs=0 l=1 rev1=1 crypto rx=0x7f8efc00b0a0 tx=0x7f8efc00b3b0 comp rx=0 tx=0).stop 2024-05-27T07:27:02.743 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:02.753+0000 7f8f0b3a4700 1 -- 172.21.15.52:0/3644439704 shutdown_connections 2024-05-27T07:27:02.744 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:02.753+0000 7f8f0b3a4700 1 --2- 172.21.15.52:0/3644439704 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f8f041030d0 0x7f8f04111160 unknown :-1 s=CLOSED pgs=492 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:27:02.744 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:02.753+0000 7f8f0b3a4700 1 --2- 172.21.15.52:0/3644439704 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f8f041092f0 0x7f8f04109700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:27:02.744 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:02.753+0000 7f8f0b3a4700 1 --2- 172.21.15.52:0/3644439704 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f8f04102780 0x7f8f04102b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:27:02.744 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:02.753+0000 7f8f0b3a4700 1 -- 172.21.15.52:0/3644439704 >> 172.21.15.52:0/3644439704 conn(0x7f8f040fe4d0 msgr2=0x7f8f04100920 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:27:02.744 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:02.753+0000 7f8f0b3a4700 1 -- 172.21.15.52:0/3644439704 shutdown_connections 2024-05-27T07:27:02.744 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:02.753+0000 7f8f0b3a4700 1 -- 172.21.15.52:0/3644439704 wait complete. 2024-05-27T07:27:02.745 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:02.754+0000 7f8f0b3a4700 1 Processor -- start 2024-05-27T07:27:02.746 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:02.754+0000 7f8f0b3a4700 1 -- start start 2024-05-27T07:27:02.746 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:02.754+0000 7f8f0b3a4700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f8f04102780 0x7f8f0419ae40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:27:02.746 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:02.754+0000 7f8f0b3a4700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f8f041030d0 0x7f8f0419b3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:27:02.746 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:02.754+0000 7f8f0b3a4700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f8f041092f0 0x7f8f0419f670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:27:02.746 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:02.754+0000 7f8f0b3a4700 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x7f8f04114590 con 0x7f8f04102780 2024-05-27T07:27:02.746 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:02.754+0000 7f8f0b3a4700 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f8f04114410 con 0x7f8f041092f0 2024-05-27T07:27:02.746 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:02.754+0000 7f8f0b3a4700 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f8f04114710 con 0x7f8f041030d0 2024-05-27T07:27:02.746 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:02.755+0000 7f8f09140700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f8f04102780 0x7f8f0419ae40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:27:02.746 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:02.755+0000 7f8f09140700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f8f04102780 0x7f8f0419ae40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.52:35760/0 (socket says 172.21.15.52:35760) 2024-05-27T07:27:02.746 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:02.755+0000 7f8f09140700 1 -- 172.21.15.52:0/917430133 learned_addr learned my addr 172.21.15.52:0/917430133 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-05-27T07:27:02.746 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:02.755+0000 7f8f09941700 1 --2- 172.21.15.52:0/917430133 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f8f041092f0 0x7f8f0419f670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:27:02.746 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:02.755+0000 7f8f0893f700 1 --2- 172.21.15.52:0/917430133 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f8f041030d0 0x7f8f0419b3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:27:02.747 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:02.755+0000 7f8f09140700 1 -- 172.21.15.52:0/917430133 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f8f041030d0 msgr2=0x7f8f0419b3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:27:02.747 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:02.755+0000 7f8f09140700 1 --2- 172.21.15.52:0/917430133 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f8f041030d0 0x7f8f0419b3e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:27:02.747 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:02.755+0000 7f8f09140700 1 -- 172.21.15.52:0/917430133 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f8f041092f0 msgr2=0x7f8f0419f670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:27:02.747 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:02.755+0000 7f8f09140700 1 --2- 172.21.15.52:0/917430133 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f8f041092f0 0x7f8f0419f670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:27:02.747 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:02.755+0000 7f8f09140700 1 -- 172.21.15.52:0/917430133 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8f0419fd10 con 0x7f8f04102780 2024-05-27T07:27:02.747 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:02.755+0000 7f8f09140700 1 --2- 172.21.15.52:0/917430133 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f8f04102780 0x7f8f0419ae40 secure :-1 s=READY pgs=493 cs=0 l=1 rev1=1 crypto rx=0x7f8ef800d950 tx=0x7f8ef800dc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:27:02.747 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:02.755+0000 7f8ef67fc700 1 -- 172.21.15.52:0/917430133 <== mon.0 v2:172.21.15.52:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8ef8017070 con 0x7f8f04102780 2024-05-27T07:27:02.747 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:02.755+0000 7f8ef67fc700 1 -- 172.21.15.52:0/917430133 <== mon.0 v2:172.21.15.52:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f8ef800f690 con 0x7f8f04102780 2024-05-27T07:27:02.747 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:02.755+0000 7f8f0b3a4700 1 -- 172.21.15.52:0/917430133 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8f041a0000 con 0x7f8f04102780 2024-05-27T07:27:02.747 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:02.755+0000 7f8ef67fc700 1 -- 172.21.15.52:0/917430133 <== mon.0 v2:172.21.15.52:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8ef8003aa0 con 0x7f8f04102780 2024-05-27T07:27:02.747 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:02.756+0000 7f8f0b3a4700 1 -- 172.21.15.52:0/917430133 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8f041a7920 con 0x7f8f04102780 2024-05-27T07:27:02.748 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:02.758+0000 7f8ef67fc700 1 -- 172.21.15.52:0/917430133 <== mon.0 v2:172.21.15.52:3300/0 4 ==== mgrmap(e 25) ==== 149936+0+0 (secure 0 0 0) 0x7f8ef800f840 con 0x7f8f04102780 2024-05-27T07:27:02.749 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:02.759+0000 7f8ef67fc700 1 --2- 172.21.15.52:0/917430133 >> [v2:172.21.15.52:6800/1822318384,v1:172.21.15.52:6801/1822318384] conn(0x7f8ef00b2ba0 0x7f8ef00b50b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:27:02.749 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:02.759+0000 7f8f0893f700 1 --2- 172.21.15.52:0/917430133 >> [v2:172.21.15.52:6800/1822318384,v1:172.21.15.52:6801/1822318384] conn(0x7f8ef00b2ba0 0x7f8ef00b50b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:27:02.749 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:02.759+0000 7f8ef67fc700 1 -- 172.21.15.52:0/917430133 <== mon.0 v2:172.21.15.52:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9379+0+0 (secure 0 0 0) 0x7f8ef80e26a0 con 0x7f8f04102780 2024-05-27T07:27:02.750 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:02.760+0000 7f8f19db9b80 1 Processor -- start 2024-05-27T07:27:02.750 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:02.760+0000 7f8f19db9b80 1 -- start start 2024-05-27T07:27:02.751 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:02.760+0000 7f8f0893f700 1 --2- 172.21.15.52:0/917430133 >> [v2:172.21.15.52:6800/1822318384,v1:172.21.15.52:6801/1822318384] conn(0x7f8ef00b2ba0 0x7f8ef00b50b0 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f8f04108870 tx=0x7f8f0000a300 comp rx=0 tx=0).ready entity=mgr.15333 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:27:02.751 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:02.760+0000 7f8f19db9b80 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x556bbf1cce40 0x556bbf354b20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:27:02.751 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:02.760+0000 7f8f19db9b80 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x556bbf355060 0x556bbf34ee50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:27:02.751 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:02.760+0000 7f8f19db9b80 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x556bbf34f390 0x556bbf34f7a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:27:02.751 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:02.760+0000 7f8f19db9b80 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x556bbf279c90 con 0x556bbf1cce40 2024-05-27T07:27:02.751 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:02.760+0000 7f8f19db9b80 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x556bbf269830 con 0x556bbf34f390 2024-05-27T07:27:02.751 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:02.760+0000 7f8f19db9b80 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x556bbf26d870 con 0x556bbf355060 2024-05-27T07:27:02.751 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:02.760+0000 7f8f09140700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x556bbf1cce40 0x556bbf354b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:27:02.751 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:02.760+0000 7f8f09140700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x556bbf1cce40 0x556bbf354b20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.52:35774/0 (socket says 172.21.15.52:35774) 2024-05-27T07:27:02.751 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:02.760+0000 7f8f09140700 1 -- 172.21.15.52:0/1691649556 learned_addr learned my addr 172.21.15.52:0/1691649556 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-05-27T07:27:02.751 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:02.760+0000 7f8f09941700 1 --2- 172.21.15.52:0/1691649556 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x556bbf34f390 0x556bbf34f7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:27:02.751 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:02.760+0000 7f8f0893f700 1 --2- 172.21.15.52:0/1691649556 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x556bbf355060 0x556bbf34ee50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:27:02.752 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:02.760+0000 7f8f09140700 1 -- 172.21.15.52:0/1691649556 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x556bbf355060 msgr2=0x556bbf34ee50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:27:02.752 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:02.760+0000 7f8f09140700 1 --2- 172.21.15.52:0/1691649556 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x556bbf355060 0x556bbf34ee50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:27:02.752 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:02.761+0000 7f8f09140700 1 -- 172.21.15.52:0/1691649556 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x556bbf34f390 msgr2=0x556bbf34f7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:27:02.752 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:02.761+0000 7f8f09140700 1 --2- 172.21.15.52:0/1691649556 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x556bbf34f390 0x556bbf34f7a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:27:02.752 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:02.761+0000 7f8f09140700 1 -- 172.21.15.52:0/1691649556 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x556bbf1b98d0 con 0x556bbf1cce40 2024-05-27T07:27:02.752 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:02.761+0000 7f8f09941700 1 --2- 172.21.15.52:0/1691649556 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x556bbf34f390 0x556bbf34f7a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-27T07:27:02.752 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:02.761+0000 7f8f09140700 1 --2- 172.21.15.52:0/1691649556 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x556bbf1cce40 0x556bbf354b20 secure :-1 s=READY pgs=494 cs=0 l=1 rev1=1 crypto rx=0x7f8ef80a6650 tx=0x7f8ef80a8fc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8015deb5374b4e3f server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:27:02.752 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:02.761+0000 7f8eef7fe700 1 -- 172.21.15.52:0/1691649556 <== mon.0 v2:172.21.15.52:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8ef80b2240 con 0x556bbf1cce40 2024-05-27T07:27:02.752 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:02.761+0000 7f8eef7fe700 1 -- 172.21.15.52:0/1691649556 <== mon.0 v2:172.21.15.52:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f8ef800fb60 con 0x556bbf1cce40 2024-05-27T07:27:02.752 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:02.761+0000 7f8eef7fe700 1 -- 172.21.15.52:0/1691649556 <== mon.0 v2:172.21.15.52:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8ef80a9840 con 0x556bbf1cce40 2024-05-27T07:27:02.755 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:02.762+0000 7f8f19db9b80 1 -- 172.21.15.52:0/1691649556 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x556bbf1cce40 msgr2=0x556bbf354b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:27:02.755 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:02.762+0000 7f8f19db9b80 1 --2- 172.21.15.52:0/1691649556 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x556bbf1cce40 0x556bbf354b20 secure :-1 s=READY pgs=494 cs=0 l=1 rev1=1 crypto rx=0x7f8ef80a6650 tx=0x7f8ef80a8fc0 comp rx=0 tx=0).stop 2024-05-27T07:27:02.755 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:02.762+0000 7f8f19db9b80 1 -- 172.21.15.52:0/1691649556 shutdown_connections 2024-05-27T07:27:02.755 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:02.762+0000 7f8f19db9b80 1 --2- 172.21.15.52:0/1691649556 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x556bbf34f390 0x556bbf34f7a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:27:02.755 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:02.762+0000 7f8f19db9b80 1 --2- 172.21.15.52:0/1691649556 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x556bbf1cce40 0x556bbf354b20 unknown :-1 s=CLOSED pgs=494 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:27:02.755 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:02.762+0000 7f8f19db9b80 1 --2- 172.21.15.52:0/1691649556 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x556bbf355060 0x556bbf34ee50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:27:02.755 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:02.762+0000 7f8f19db9b80 1 -- 172.21.15.52:0/1691649556 >> 172.21.15.52:0/1691649556 conn(0x556bbf227af0 msgr2=0x556bbf345640 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:27:02.756 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:02.762+0000 7f8f19db9b80 1 -- 172.21.15.52:0/1691649556 shutdown_connections 2024-05-27T07:27:02.756 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:02.762+0000 7f8f19db9b80 1 -- 172.21.15.52:0/1691649556 wait complete. 2024-05-27T07:27:02.756 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:02.762+0000 7f8f19db9b80 1 -- start start 2024-05-27T07:27:02.756 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:02.762+0000 7f8f19db9b80 1 Processor -- start 2024-05-27T07:27:02.756 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:02.763+0000 7f8f19db9b80 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x556bbf1cce40 0x556bbf3587e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:27:02.756 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:02.763+0000 7f8f19db9b80 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x556bbf34f390 0x556bbf358d20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:27:02.756 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:02.763+0000 7f8f19db9b80 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x556bbf355060 0x556bbf359260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:27:02.756 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:02.763+0000 7f8f19db9b80 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x556bbf16ffa0 con 0x556bbf34f390 2024-05-27T07:27:02.756 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:02.763+0000 7f8f19db9b80 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x556bbf20b7c0 con 0x556bbf1cce40 2024-05-27T07:27:02.756 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:02.763+0000 7f8f19db9b80 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x556bbf35bb50 con 0x556bbf355060 2024-05-27T07:27:02.756 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:02.763+0000 7f8f19db9b80 10 client.0 _ll_register_callbacks cb 0x556bbf204920 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T07:27:02.756 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:02.763+0000 7f8f0893f700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x556bbf34f390 0x556bbf358d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:27:02.757 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:02.763+0000 7f8f0893f700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x556bbf34f390 0x556bbf358d20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.52:35790/0 (socket says 172.21.15.52:35790) 2024-05-27T07:27:02.757 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:02.763+0000 7f8f0893f700 1 -- 172.21.15.52:0/625463775 learned_addr learned my addr 172.21.15.52:0/625463775 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-05-27T07:27:02.757 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:02.763+0000 7f8f09941700 1 --2- 172.21.15.52:0/625463775 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x556bbf355060 0x556bbf359260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:27:02.757 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:02.763+0000 7f8f09140700 1 --2- 172.21.15.52:0/625463775 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x556bbf1cce40 0x556bbf3587e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:27:02.757 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:02.763+0000 7f8f0893f700 1 -- 172.21.15.52:0/625463775 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x556bbf355060 msgr2=0x556bbf359260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:27:02.757 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:02.763+0000 7f8f0893f700 1 --2- 172.21.15.52:0/625463775 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x556bbf355060 0x556bbf359260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:27:02.757 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:02.763+0000 7f8f0893f700 1 -- 172.21.15.52:0/625463775 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x556bbf1cce40 msgr2=0x556bbf3587e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:27:02.757 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:02.763+0000 7f8f0893f700 1 --2- 172.21.15.52:0/625463775 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x556bbf1cce40 0x556bbf3587e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:27:02.757 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:02.763+0000 7f8f0893f700 1 -- 172.21.15.52:0/625463775 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({osdmap=0}) -- 0x556bbf208690 con 0x556bbf34f390 2024-05-27T07:27:02.757 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:02.764+0000 7f8f0893f700 1 --2- 172.21.15.52:0/625463775 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x556bbf34f390 0x556bbf358d20 secure :-1 s=READY pgs=495 cs=0 l=1 rev1=1 crypto rx=0x7f8f0001da60 tx=0x7f8f000047c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ea6a8341c3f83805 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:27:02.757 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:02.764+0000 7f8eecff9700 10 client.0 ms_handle_connect on v2:172.21.15.52:3300/0 2024-05-27T07:27:02.757 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:02.764+0000 7f8eecff9700 1 -- 172.21.15.52:0/625463775 <== mon.0 v2:172.21.15.52:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8f00016070 con 0x556bbf34f390 2024-05-27T07:27:02.757 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:02.764+0000 7f8eecff9700 1 -- 172.21.15.52:0/625463775 <== mon.0 v2:172.21.15.52:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 9379+0+0 (secure 0 0 0) 0x7f8f00004a30 con 0x556bbf34f390 2024-05-27T07:27:02.758 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:02.764+0000 7f8f19db9b80 1 -- 172.21.15.52:0/625463775 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f8ef800fb60 con 0x556bbf34f390 2024-05-27T07:27:02.758 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:02.764+0000 7f8eecff9700 1 -- 172.21.15.52:0/625463775 <== mon.0 v2:172.21.15.52:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7f8f00004c30 con 0x556bbf34f390 2024-05-27T07:27:02.758 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:02.764+0000 7f8f19db9b80 10 client.15693 fetch_fsmap learned FSMap version 30 2024-05-27T07:27:02.758 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:02.764+0000 7f8f19db9b80 1 -- 172.21.15.52:0/625463775 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x556bbf3666a0 con 0x556bbf34f390 2024-05-27T07:27:02.758 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:02.764+0000 7f8eecff9700 1 -- 172.21.15.52:0/625463775 <== mon.0 v2:172.21.15.52:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f8ef800fb60 con 0x556bbf34f390 2024-05-27T07:27:02.758 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:02.765+0000 7f8eecff9700 1 -- 172.21.15.52:0/625463775 <== mon.0 v2:172.21.15.52:3300/0 5 ==== fsmap(e 30) ==== 8362+0+0 (secure 0 0 0) 0x7f8f00010a20 con 0x556bbf34f390 2024-05-27T07:27:02.758 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:02.765+0000 7f8f19db9b80 10 client.15693 fetch_fsmap finished waiting for FSMap version 30 2024-05-27T07:27:02.758 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:02.765+0000 7f8f19db9b80 10 client.15693 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.h 2024-05-27T07:27:02.758 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:02.765+0000 7f8f19db9b80 20 client.15693 populate_metadata read hostname 'smithi052' 2024-05-27T07:27:02.758 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:02.765+0000 7f8f19db9b80 1 --2- 172.21.15.52:0/625463775 >> [v2:172.21.15.105:6836/191393149,v1:172.21.15.105:6837/191393149] conn(0x556bbf368ce0 0x556bbf36b140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:27:02.758 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:02.765+0000 7f8f19db9b80 4 client.15693 mds_command: new command op to 24497 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-27T07:27:02.758 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:02.765+0000 7f8f19db9b80 1 -- 172.21.15.52:0/625463775 --> [v2:172.21.15.105:6836/191393149,v1:172.21.15.105:6837/191393149] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x556bbf3169f0 con 0x556bbf368ce0 2024-05-27T07:27:02.759 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:02.765+0000 7f8f09140700 1 --2- 172.21.15.52:0/625463775 >> [v2:172.21.15.105:6836/191393149,v1:172.21.15.105:6837/191393149] conn(0x556bbf368ce0 0x556bbf36b140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:27:02.759 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:02.765+0000 7f8eecff9700 10 client.15693 ms_handle_connect on v2:172.21.15.105:6836/191393149 2024-05-27T07:27:02.759 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:02.765+0000 7f8f09140700 1 --2- 172.21.15.52:0/625463775 >> [v2:172.21.15.105:6836/191393149,v1:172.21.15.105:6837/191393149] conn(0x556bbf368ce0 0x556bbf36b140 crc :-1 s=READY pgs=30 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=4ad973a7e90bb9f0 server_cookie=13cb611ecb7d45eb in_seq=0 out_seq=0 2024-05-27T07:27:02.760 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:02.766+0000 7f8eecff9700 1 -- 172.21.15.52:0/625463775 <== mds.1 v2:172.21.15.105:6836/191393149 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24041 (crc 0 0 0) 0x556bbf3169f0 con 0x556bbf368ce0 2024-05-27T07:27:02.760 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:02.766+0000 7f8eecff9700 10 client.15693 handle_command_reply: tid=0 2024-05-27T07:27:02.760 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:02.766+0000 7f8f19db9b80 1 client.15693 shutdown 2024-05-27T07:27:02.760 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:02.767+0000 7f8f19db9b80 1 -- 172.21.15.52:0/625463775 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x556bbf34f390 msgr2=0x556bbf358d20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:27:02.760 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:02.767+0000 7f8f19db9b80 1 --2- 172.21.15.52:0/625463775 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x556bbf34f390 0x556bbf358d20 secure :-1 s=READY pgs=495 cs=0 l=1 rev1=1 crypto rx=0x7f8f0001da60 tx=0x7f8f000047c0 comp rx=0 tx=0).stop 2024-05-27T07:27:02.760 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:02.767+0000 7f8f19db9b80 1 -- 172.21.15.52:0/625463775 shutdown_connections 2024-05-27T07:27:02.760 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:02.767+0000 7f8f19db9b80 1 --2- 172.21.15.52:0/625463775 >> [v2:172.21.15.105:6836/191393149,v1:172.21.15.105:6837/191393149] conn(0x556bbf368ce0 0x556bbf36b140 crc :-1 s=READY pgs=30 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:27:02.760 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:02.767+0000 7f8f19db9b80 1 --2- 172.21.15.52:0/625463775 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x556bbf355060 0x556bbf359260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:27:02.760 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:02.767+0000 7f8f19db9b80 1 --2- 172.21.15.52:0/625463775 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x556bbf1cce40 0x556bbf3587e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:27:02.760 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:02.767+0000 7f8f19db9b80 1 --2- 172.21.15.52:0/625463775 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x556bbf34f390 0x556bbf358d20 unknown :-1 s=CLOSED pgs=495 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:27:02.760 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:02.767+0000 7f8eecff9700 0 client.15693 ms_handle_reset on v2:172.21.15.105:6836/191393149 2024-05-27T07:27:02.760 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:02.767+0000 7f8f19db9b80 1 -- 172.21.15.52:0/625463775 >> 172.21.15.52:0/625463775 conn(0x556bbf227af0 msgr2=0x556bbf34de40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:27:02.761 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:02.767+0000 7f8f19db9b80 1 -- 172.21.15.52:0/625463775 shutdown_connections 2024-05-27T07:27:02.761 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:02.767+0000 7f8f19db9b80 1 -- 172.21.15.52:0/625463775 wait complete. 2024-05-27T07:27:02.761 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:02.767+0000 7f8f19db9b80 20 client.15693 trim_cache size 0 max 16384 2024-05-27T07:27:02.773 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:02.782+0000 7f8f19db9b80 1 Processor -- start 2024-05-27T07:27:02.773 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:02.782+0000 7f8f19db9b80 1 -- start start 2024-05-27T07:27:02.773 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:02.782+0000 7f8f19db9b80 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x556bbf1cce40 0x556bbf34ee50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:27:02.773 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:02.782+0000 7f8f19db9b80 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x556bbf34f390 0x556bbf353f00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:27:02.773 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:02.782+0000 7f8f19db9b80 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x556bbf355060 0x556bbf3590a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:27:02.773 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:02.782+0000 7f8f19db9b80 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x7f8f041ae2a0 con 0x556bbf1cce40 2024-05-27T07:27:02.773 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:02.782+0000 7f8f19db9b80 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f8f041afde0 con 0x556bbf355060 2024-05-27T07:27:02.773 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:02.782+0000 7f8f19db9b80 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f8f041ae970 con 0x556bbf34f390 2024-05-27T07:27:02.773 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:02.782+0000 7f8f09140700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x556bbf1cce40 0x556bbf34ee50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:27:02.773 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:02.783+0000 7f8f09140700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x556bbf1cce40 0x556bbf34ee50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.52:35794/0 (socket says 172.21.15.52:35794) 2024-05-27T07:27:02.773 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:02.783+0000 7f8f09140700 1 -- 172.21.15.52:0/1924396471 learned_addr learned my addr 172.21.15.52:0/1924396471 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-05-27T07:27:02.773 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:02.783+0000 7f8f0893f700 1 --2- 172.21.15.52:0/1924396471 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x556bbf34f390 0x556bbf353f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:27:02.774 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:02.783+0000 7f8f09941700 1 --2- 172.21.15.52:0/1924396471 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x556bbf355060 0x556bbf3590a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:27:02.774 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:02.783+0000 7f8f09140700 1 -- 172.21.15.52:0/1924396471 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x556bbf34f390 msgr2=0x556bbf353f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:27:02.774 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:02.783+0000 7f8f09140700 1 --2- 172.21.15.52:0/1924396471 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x556bbf34f390 0x556bbf353f00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:27:02.774 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:02.783+0000 7f8f09140700 1 -- 172.21.15.52:0/1924396471 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x556bbf355060 msgr2=0x556bbf3590a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:27:02.774 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:02.783+0000 7f8f09140700 1 --2- 172.21.15.52:0/1924396471 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x556bbf355060 0x556bbf3590a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:27:02.774 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:02.783+0000 7f8f09140700 1 -- 172.21.15.52:0/1924396471 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x556bbf340410 con 0x556bbf1cce40 2024-05-27T07:27:02.774 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:02.783+0000 7f8f09941700 1 --2- 172.21.15.52:0/1924396471 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x556bbf355060 0x556bbf3590a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-27T07:27:02.774 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:02.783+0000 7f8f09140700 1 --2- 172.21.15.52:0/1924396471 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x556bbf1cce40 0x556bbf34ee50 secure :-1 s=READY pgs=496 cs=0 l=1 rev1=1 crypto rx=0x7f8ef80df940 tx=0x7f8ef8010000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c4c0b4377befd8da server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:27:02.774 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:02.783+0000 7f8eeeffd700 1 -- 172.21.15.52:0/1924396471 <== mon.0 v2:172.21.15.52:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8ef80c2070 con 0x556bbf1cce40 2024-05-27T07:27:02.774 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:02.783+0000 7f8eeeffd700 1 -- 172.21.15.52:0/1924396471 <== mon.0 v2:172.21.15.52:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f8ef80e2320 con 0x556bbf1cce40 2024-05-27T07:27:02.774 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:02.783+0000 7f8eeeffd700 1 -- 172.21.15.52:0/1924396471 <== mon.0 v2:172.21.15.52:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8ef80a98a0 con 0x556bbf1cce40 2024-05-27T07:27:02.778 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:02.784+0000 7f8f19db9b80 1 -- 172.21.15.52:0/1924396471 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x556bbf1cce40 msgr2=0x556bbf34ee50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:27:02.778 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:02.784+0000 7f8f19db9b80 1 --2- 172.21.15.52:0/1924396471 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x556bbf1cce40 0x556bbf34ee50 secure :-1 s=READY pgs=496 cs=0 l=1 rev1=1 crypto rx=0x7f8ef80df940 tx=0x7f8ef8010000 comp rx=0 tx=0).stop 2024-05-27T07:27:02.778 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:02.784+0000 7f8f19db9b80 1 -- 172.21.15.52:0/1924396471 shutdown_connections 2024-05-27T07:27:02.778 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:02.784+0000 7f8f19db9b80 1 --2- 172.21.15.52:0/1924396471 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x556bbf355060 0x556bbf3590a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:27:02.778 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:02.784+0000 7f8f19db9b80 1 --2- 172.21.15.52:0/1924396471 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x556bbf1cce40 0x556bbf34ee50 unknown :-1 s=CLOSED pgs=496 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:27:02.778 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:02.784+0000 7f8f19db9b80 1 --2- 172.21.15.52:0/1924396471 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x556bbf34f390 0x556bbf353f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:27:02.778 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:02.784+0000 7f8f19db9b80 1 -- 172.21.15.52:0/1924396471 >> 172.21.15.52:0/1924396471 conn(0x556bbf227af0 msgr2=0x556bbf3537f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:27:02.778 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:02.784+0000 7f8f19db9b80 1 -- 172.21.15.52:0/1924396471 shutdown_connections 2024-05-27T07:27:02.778 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:02.784+0000 7f8f19db9b80 1 -- 172.21.15.52:0/1924396471 wait complete. 2024-05-27T07:27:02.778 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:02.784+0000 7f8f19db9b80 1 -- start start 2024-05-27T07:27:02.778 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:02.785+0000 7f8f19db9b80 1 Processor -- start 2024-05-27T07:27:02.778 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:02.785+0000 7f8f19db9b80 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x556bbf1cce40 0x556bbf3615f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:27:02.779 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:02.785+0000 7f8f19db9b80 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x556bbf34f390 0x556bbf3702d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:27:02.779 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:02.785+0000 7f8f19db9b80 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x556bbf355060 0x556bbf3708d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:27:02.779 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:02.785+0000 7f8f19db9b80 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x7f8f041af710 con 0x556bbf1cce40 2024-05-27T07:27:02.779 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:02.785+0000 7f8f19db9b80 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f8f041af040 con 0x556bbf355060 2024-05-27T07:27:02.779 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:02.785+0000 7f8f19db9b80 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f8f041ac240 con 0x556bbf34f390 2024-05-27T07:27:02.779 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:02.785+0000 7f8f19db9b80 10 client.0 _ll_register_callbacks cb 0x556bbf204920 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T07:27:02.779 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:02.785+0000 7f8f09140700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x556bbf1cce40 0x556bbf3615f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:27:02.779 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:02.785+0000 7f8f09140700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x556bbf1cce40 0x556bbf3615f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.52:35804/0 (socket says 172.21.15.52:35804) 2024-05-27T07:27:02.779 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:02.785+0000 7f8f09140700 1 -- 172.21.15.52:0/2077278078 learned_addr learned my addr 172.21.15.52:0/2077278078 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-05-27T07:27:02.779 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:02.785+0000 7f8f09941700 1 --2- 172.21.15.52:0/2077278078 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x556bbf355060 0x556bbf3708d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:27:02.779 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:02.785+0000 7f8f09140700 1 -- 172.21.15.52:0/2077278078 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x556bbf34f390 msgr2=0x556bbf3702d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:27:02.779 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:02.785+0000 7f8f0893f700 1 --2- 172.21.15.52:0/2077278078 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x556bbf34f390 0x556bbf3702d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:27:02.779 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:02.785+0000 7f8f09140700 1 --2- 172.21.15.52:0/2077278078 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x556bbf34f390 0x556bbf3702d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:27:02.780 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:02.785+0000 7f8f09140700 1 -- 172.21.15.52:0/2077278078 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x556bbf355060 msgr2=0x556bbf3708d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:27:02.780 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:02.785+0000 7f8f09140700 1 --2- 172.21.15.52:0/2077278078 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x556bbf355060 0x556bbf3708d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:27:02.780 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:02.786+0000 7f8f09140700 1 -- 172.21.15.52:0/2077278078 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({osdmap=0}) -- 0x556bbf3710a0 con 0x556bbf1cce40 2024-05-27T07:27:02.780 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:02.786+0000 7f8f09140700 1 --2- 172.21.15.52:0/2077278078 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x556bbf1cce40 0x556bbf3615f0 secure :-1 s=READY pgs=497 cs=0 l=1 rev1=1 crypto rx=0x7f8ef80a6650 tx=0x7f8ef80a5360 comp rx=0 tx=0).ready entity=mon.0 client_cookie=92880500f5e35723 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:27:02.780 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:02.786+0000 7f8eecff9700 10 client.0 ms_handle_connect on v2:172.21.15.52:3300/0 2024-05-27T07:27:02.780 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:02.786+0000 7f8eecff9700 1 -- 172.21.15.52:0/2077278078 <== mon.0 v2:172.21.15.52:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8ef80c2070 con 0x556bbf1cce40 2024-05-27T07:27:02.780 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:02.786+0000 7f8eecff9700 1 -- 172.21.15.52:0/2077278078 <== mon.0 v2:172.21.15.52:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 9379+0+0 (secure 0 0 0) 0x7f8ef80a59d0 con 0x556bbf1cce40 2024-05-27T07:27:02.780 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:02.786+0000 7f8f19db9b80 1 -- 172.21.15.52:0/2077278078 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f8ef80e2320 con 0x556bbf1cce40 2024-05-27T07:27:02.780 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:02.786+0000 7f8eecff9700 1 -- 172.21.15.52:0/2077278078 <== mon.0 v2:172.21.15.52:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7f8ef80a9640 con 0x556bbf1cce40 2024-05-27T07:27:02.780 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:02.786+0000 7f8f19db9b80 10 client.15699 fetch_fsmap learned FSMap version 30 2024-05-27T07:27:02.780 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:02.786+0000 7f8f19db9b80 1 -- 172.21.15.52:0/2077278078 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x556bbf3d37a0 con 0x556bbf1cce40 2024-05-27T07:27:02.780 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:02.786+0000 7f8eecff9700 1 -- 172.21.15.52:0/2077278078 <== mon.0 v2:172.21.15.52:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f8ef80e2320 con 0x556bbf1cce40 2024-05-27T07:27:02.780 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:02.786+0000 7f8eecff9700 1 -- 172.21.15.52:0/2077278078 <== mon.0 v2:172.21.15.52:3300/0 5 ==== fsmap(e 30) ==== 8362+0+0 (secure 0 0 0) 0x7f8ef80ca200 con 0x556bbf1cce40 2024-05-27T07:27:02.780 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:02.786+0000 7f8f19db9b80 10 client.15699 fetch_fsmap finished waiting for FSMap version 30 2024-05-27T07:27:02.781 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:02.786+0000 7f8f19db9b80 10 client.15699 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.h 2024-05-27T07:27:02.781 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:02.786+0000 7f8f19db9b80 20 client.15699 populate_metadata read hostname 'smithi052' 2024-05-27T07:27:02.781 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:02.786+0000 7f8f19db9b80 1 --2- 172.21.15.52:0/2077278078 >> [v2:172.21.15.105:6836/191393149,v1:172.21.15.105:6837/191393149] conn(0x556bbf3d3970 0x556bbf3d5d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:27:02.781 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:02.787+0000 7f8f19db9b80 4 client.15699 mds_command: new command op to 24497 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-05-27T07:27:02.781 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:02.787+0000 7f8f19db9b80 1 -- 172.21.15.52:0/2077278078 --> [v2:172.21.15.105:6836/191393149,v1:172.21.15.105:6837/191393149] -- command(tid 0: {"prefix": "damage ls"}) -- 0x556bbf27bbe0 con 0x556bbf3d3970 2024-05-27T07:27:02.781 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:02.787+0000 7f8f0893f700 1 --2- 172.21.15.52:0/2077278078 >> [v2:172.21.15.105:6836/191393149,v1:172.21.15.105:6837/191393149] conn(0x556bbf3d3970 0x556bbf3d5d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:27:02.781 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:02.787+0000 7f8f0893f700 1 --2- 172.21.15.52:0/2077278078 >> [v2:172.21.15.105:6836/191393149,v1:172.21.15.105:6837/191393149] conn(0x556bbf3d3970 0x556bbf3d5d90 crc :-1 s=READY pgs=31 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=ad5e3e7457366a33 server_cookie=b91fee055a0a692c in_seq=0 out_seq=0 2024-05-27T07:27:02.781 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:02.787+0000 7f8eecff9700 10 client.15699 ms_handle_connect on v2:172.21.15.105:6836/191393149 2024-05-27T07:27:02.781 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:02.787+0000 7f8eecff9700 1 -- 172.21.15.52:0/2077278078 <== mds.1 v2:172.21.15.105:6836/191393149 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x556bbf27bbe0 con 0x556bbf3d3970 2024-05-27T07:27:02.781 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:02.787+0000 7f8eecff9700 10 client.15699 handle_command_reply: tid=0 2024-05-27T07:27:02.781 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:02.787+0000 7f8f19db9b80 1 client.15699 shutdown 2024-05-27T07:27:02.781 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:02.788+0000 7f8f19db9b80 1 -- 172.21.15.52:0/2077278078 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x556bbf1cce40 msgr2=0x556bbf3615f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:27:02.781 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:02.788+0000 7f8f19db9b80 1 --2- 172.21.15.52:0/2077278078 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x556bbf1cce40 0x556bbf3615f0 secure :-1 s=READY pgs=497 cs=0 l=1 rev1=1 crypto rx=0x7f8ef80a6650 tx=0x7f8ef80a5360 comp rx=0 tx=0).stop 2024-05-27T07:27:02.782 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:02.788+0000 7f8f19db9b80 1 -- 172.21.15.52:0/2077278078 shutdown_connections 2024-05-27T07:27:02.782 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:02.788+0000 7f8f19db9b80 1 --2- 172.21.15.52:0/2077278078 >> [v2:172.21.15.105:6836/191393149,v1:172.21.15.105:6837/191393149] conn(0x556bbf3d3970 0x556bbf3d5d90 crc :-1 s=READY pgs=31 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:27:02.782 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:02.788+0000 7f8f19db9b80 1 --2- 172.21.15.52:0/2077278078 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x556bbf355060 0x556bbf3708d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:27:02.782 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:02.788+0000 7f8eecff9700 0 client.15699 ms_handle_reset on v2:172.21.15.105:6836/191393149 2024-05-27T07:27:02.782 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:02.788+0000 7f8f19db9b80 1 --2- 172.21.15.52:0/2077278078 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x556bbf1cce40 0x556bbf3615f0 unknown :-1 s=CLOSED pgs=497 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:27:02.782 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:02.788+0000 7f8f19db9b80 1 --2- 172.21.15.52:0/2077278078 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x556bbf34f390 0x556bbf3702d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:27:02.782 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:02.788+0000 7f8f19db9b80 1 -- 172.21.15.52:0/2077278078 >> 172.21.15.52:0/2077278078 conn(0x556bbf227af0 msgr2=0x556bbf359d80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:27:02.782 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:02.788+0000 7f8f19db9b80 1 -- 172.21.15.52:0/2077278078 shutdown_connections 2024-05-27T07:27:02.782 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:02.788+0000 7f8f19db9b80 1 -- 172.21.15.52:0/2077278078 wait complete. 2024-05-27T07:27:02.782 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:02.788+0000 7f8f19db9b80 20 client.15699 trim_cache size 0 max 16384 2024-05-27T07:27:02.783 INFO:teuthology.orchestra.run.smithi052.stdout:[] 2024-05-27T07:27:02.783 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:02.789+0000 7f8f0b3a4700 1 -- 172.21.15.52:0/917430133 >> [v2:172.21.15.52:6800/1822318384,v1:172.21.15.52:6801/1822318384] conn(0x7f8ef00b2ba0 msgr2=0x7f8ef00b50b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:27:02.783 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:02.789+0000 7f8f0b3a4700 1 --2- 172.21.15.52:0/917430133 >> [v2:172.21.15.52:6800/1822318384,v1:172.21.15.52:6801/1822318384] conn(0x7f8ef00b2ba0 0x7f8ef00b50b0 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f8f04108870 tx=0x7f8f0000a300 comp rx=0 tx=0).stop 2024-05-27T07:27:02.783 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:02.789+0000 7f8f0b3a4700 1 -- 172.21.15.52:0/917430133 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f8f04102780 msgr2=0x7f8f0419ae40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:27:02.783 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:02.789+0000 7f8f0b3a4700 1 --2- 172.21.15.52:0/917430133 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f8f04102780 0x7f8f0419ae40 secure :-1 s=READY pgs=493 cs=0 l=1 rev1=1 crypto rx=0x7f8ef800d950 tx=0x7f8ef800dc60 comp rx=0 tx=0).stop 2024-05-27T07:27:02.783 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:02.789+0000 7f8f0b3a4700 1 -- 172.21.15.52:0/917430133 shutdown_connections 2024-05-27T07:27:02.783 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:02.789+0000 7f8f0b3a4700 1 --2- 172.21.15.52:0/917430133 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f8f041092f0 0x7f8f0419f670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:27:02.784 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:02.789+0000 7f8f0b3a4700 1 --2- 172.21.15.52:0/917430133 >> [v2:172.21.15.52:6800/1822318384,v1:172.21.15.52:6801/1822318384] conn(0x7f8ef00b2ba0 0x7f8ef00b50b0 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:27:02.784 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:02.789+0000 7f8f0b3a4700 1 --2- 172.21.15.52:0/917430133 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f8f04102780 0x7f8f0419ae40 unknown :-1 s=CLOSED pgs=493 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:27:02.784 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:02.789+0000 7f8f0b3a4700 1 --2- 172.21.15.52:0/917430133 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f8f041030d0 0x7f8f0419b3e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:27:02.784 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:02.789+0000 7f8f0b3a4700 1 -- 172.21.15.52:0/917430133 >> 172.21.15.52:0/917430133 conn(0x7f8f040fe4d0 msgr2=0x7f8f04107020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:27:02.784 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:02.789+0000 7f8f0b3a4700 1 -- 172.21.15.52:0/917430133 shutdown_connections 2024-05-27T07:27:02.784 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:02.789+0000 7f8f0b3a4700 1 -- 172.21.15.52:0/917430133 wait complete. 2024-05-27T07:27:03.511 DEBUG:teuthology.orchestra.run.smithi052:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:38f9d900c83c63799fbdbe61acc9a11b0d3554a6 shell --fsid 73df2636-1bf8-11ef-bc9b-c7b262605968 -- ceph tell mds.1:0 damage ls 2024-05-27T07:27:03.763 INFO:teuthology.orchestra.run.smithi052.stderr:Inferring config /var/lib/ceph/73df2636-1bf8-11ef-bc9b-c7b262605968/mon.a/config 2024-05-27T07:27:04.210 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:27:04 smithi052 ceph-mon[31955]: pgmap v36: 129 pgs: 129 active+clean; 1.4 MiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 24 KiB/s wr, 30 op/s 2024-05-27T07:27:04.352 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:27:04 smithi173 ceph-mon[37000]: pgmap v36: 129 pgs: 129 active+clean; 1.4 MiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 24 KiB/s wr, 30 op/s 2024-05-27T07:27:04.439 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:27:04 smithi105 ceph-mon[35977]: pgmap v36: 129 pgs: 129 active+clean; 1.4 MiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 24 KiB/s wr, 30 op/s 2024-05-27T07:27:05.575 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:05.585+0000 7f474dbaf700 1 -- 172.21.15.52:0/1538494623 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f4748106560 msgr2=0x7f4748106970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:27:05.575 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:05.585+0000 7f474dbaf700 1 --2- 172.21.15.52:0/1538494623 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f4748106560 0x7f4748106970 secure :-1 s=READY pgs=498 cs=0 l=1 rev1=1 crypto rx=0x7f473c009990 tx=0x7f473c009ca0 comp rx=0 tx=0).stop 2024-05-27T07:27:05.575 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:05.585+0000 7f474dbaf700 1 -- 172.21.15.52:0/1538494623 shutdown_connections 2024-05-27T07:27:05.575 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:05.585+0000 7f474dbaf700 1 --2- 172.21.15.52:0/1538494623 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f4748106eb0 0x7f4748113340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:27:05.575 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:05.585+0000 7f474dbaf700 1 --2- 172.21.15.52:0/1538494623 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f474810c5a0 0x7f474810c9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:27:05.575 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:05.585+0000 7f474dbaf700 1 --2- 172.21.15.52:0/1538494623 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f4748106560 0x7f4748106970 unknown :-1 s=CLOSED pgs=498 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:27:05.575 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:05.585+0000 7f474dbaf700 1 -- 172.21.15.52:0/1538494623 >> 172.21.15.52:0/1538494623 conn(0x7f4748102040 msgr2=0x7f4748104490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:27:05.575 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:05.585+0000 7f474dbaf700 1 -- 172.21.15.52:0/1538494623 shutdown_connections 2024-05-27T07:27:05.576 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:05.585+0000 7f474dbaf700 1 -- 172.21.15.52:0/1538494623 wait complete. 2024-05-27T07:27:05.577 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:05.586+0000 7f474dbaf700 1 Processor -- start 2024-05-27T07:27:05.577 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:05.586+0000 7f474dbaf700 1 -- start start 2024-05-27T07:27:05.577 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:05.586+0000 7f474dbaf700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f4748106560 0x7f474819cfd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:27:05.577 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:05.587+0000 7f474dbaf700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f4748106eb0 0x7f474819d570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:27:05.577 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:05.587+0000 7f474dbaf700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f474810c5a0 0x7f47481a1800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:27:05.577 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:05.587+0000 7f47477fe700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f4748106560 0x7f474819cfd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:27:05.577 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:05.587+0000 7f474dbaf700 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x7f4748116680 con 0x7f4748106560 2024-05-27T07:27:05.577 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:05.587+0000 7f474dbaf700 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f4748116500 con 0x7f474810c5a0 2024-05-27T07:27:05.577 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:05.587+0000 7f47477fe700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f4748106560 0x7f474819cfd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.52:35828/0 (socket says 172.21.15.52:35828) 2024-05-27T07:27:05.578 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:05.587+0000 7f474dbaf700 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f4748116800 con 0x7f4748106eb0 2024-05-27T07:27:05.578 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:05.587+0000 7f47477fe700 1 -- 172.21.15.52:0/2137997426 learned_addr learned my addr 172.21.15.52:0/2137997426 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-05-27T07:27:05.578 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:05.587+0000 7f4746ffd700 1 --2- 172.21.15.52:0/2137997426 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f4748106eb0 0x7f474819d570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:27:05.578 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:05.587+0000 7f4747fff700 1 --2- 172.21.15.52:0/2137997426 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f474810c5a0 0x7f47481a1800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:27:05.578 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:05.587+0000 7f47477fe700 1 -- 172.21.15.52:0/2137997426 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f4748106eb0 msgr2=0x7f474819d570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:27:05.578 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:05.587+0000 7f47477fe700 1 --2- 172.21.15.52:0/2137997426 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f4748106eb0 0x7f474819d570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:27:05.578 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:05.587+0000 7f47477fe700 1 -- 172.21.15.52:0/2137997426 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f474810c5a0 msgr2=0x7f47481a1800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:27:05.578 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:05.587+0000 7f47477fe700 1 --2- 172.21.15.52:0/2137997426 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f474810c5a0 0x7f47481a1800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:27:05.578 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:05.587+0000 7f47477fe700 1 -- 172.21.15.52:0/2137997426 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f47481a1ea0 con 0x7f4748106560 2024-05-27T07:27:05.578 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:05.587+0000 7f47477fe700 1 --2- 172.21.15.52:0/2137997426 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f4748106560 0x7f474819cfd0 secure :-1 s=READY pgs=499 cs=0 l=1 rev1=1 crypto rx=0x7f473800e970 tx=0x7f473800ec80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:27:05.579 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:05.588+0000 7f4744ff9700 1 -- 172.21.15.52:0/2137997426 <== mon.0 v2:172.21.15.52:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4738004a30 con 0x7f4748106560 2024-05-27T07:27:05.579 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:05.588+0000 7f4744ff9700 1 -- 172.21.15.52:0/2137997426 <== mon.0 v2:172.21.15.52:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f473800cc90 con 0x7f4748106560 2024-05-27T07:27:05.579 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:05.588+0000 7f474dbaf700 1 -- 172.21.15.52:0/2137997426 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f47481a2190 con 0x7f4748106560 2024-05-27T07:27:05.579 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:05.588+0000 7f4744ff9700 1 -- 172.21.15.52:0/2137997426 <== mon.0 v2:172.21.15.52:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4738005610 con 0x7f4748106560 2024-05-27T07:27:05.579 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:05.588+0000 7f474dbaf700 1 -- 172.21.15.52:0/2137997426 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f47481a9ab0 con 0x7f4748106560 2024-05-27T07:27:05.580 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:05.591+0000 7f4744ff9700 1 -- 172.21.15.52:0/2137997426 <== mon.0 v2:172.21.15.52:3300/0 4 ==== mgrmap(e 25) ==== 149936+0+0 (secure 0 0 0) 0x7f473800c7e0 con 0x7f4748106560 2024-05-27T07:27:05.581 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:05.592+0000 7f4744ff9700 1 --2- 172.21.15.52:0/2137997426 >> [v2:172.21.15.52:6800/1822318384,v1:172.21.15.52:6801/1822318384] conn(0x7f47300b2c20 0x7f47300b5130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:27:05.581 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:05.592+0000 7f4746ffd700 1 --2- 172.21.15.52:0/2137997426 >> [v2:172.21.15.52:6800/1822318384,v1:172.21.15.52:6801/1822318384] conn(0x7f47300b2c20 0x7f47300b5130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:27:05.582 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:05.592+0000 7f4744ff9700 1 -- 172.21.15.52:0/2137997426 <== mon.0 v2:172.21.15.52:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9379+0+0 (secure 0 0 0) 0x7f47380e2ee0 con 0x7f4748106560 2024-05-27T07:27:05.582 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:05.592+0000 7f4746ffd700 1 --2- 172.21.15.52:0/2137997426 >> [v2:172.21.15.52:6800/1822318384,v1:172.21.15.52:6801/1822318384] conn(0x7f47300b2c20 0x7f47300b5130 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f473c006010 tx=0x7f473c00b280 comp rx=0 tx=0).ready entity=mgr.15333 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:27:05.583 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:05.593+0000 7f475c5c4b80 1 Processor -- start 2024-05-27T07:27:05.583 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:05.593+0000 7f475c5c4b80 1 -- start start 2024-05-27T07:27:05.584 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:05.594+0000 7f475c5c4b80 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55a66dbd2e40 0x55a66dd57b30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:27:05.584 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:05.594+0000 7f475c5c4b80 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55a66dd58070 0x55a66dd51be0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:27:05.584 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:05.594+0000 7f475c5c4b80 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55a66dd52120 0x55a66dd52530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:27:05.584 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:05.594+0000 7f475c5c4b80 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x55a66dc7fd70 con 0x55a66dd52120 2024-05-27T07:27:05.584 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:05.594+0000 7f4747fff700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55a66dd52120 0x55a66dd52530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:27:05.585 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:05.594+0000 7f4747fff700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55a66dd52120 0x55a66dd52530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.52:35834/0 (socket says 172.21.15.52:35834) 2024-05-27T07:27:05.585 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:05.594+0000 7f4747fff700 1 -- 172.21.15.52:0/1799495120 learned_addr learned my addr 172.21.15.52:0/1799495120 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-05-27T07:27:05.585 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:05.594+0000 7f475c5c4b80 1 -- 172.21.15.52:0/1799495120 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x55a66dc6f910 con 0x55a66dbd2e40 2024-05-27T07:27:05.585 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:05.594+0000 7f475c5c4b80 1 -- 172.21.15.52:0/1799495120 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x55a66dc73950 con 0x55a66dd58070 2024-05-27T07:27:05.585 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:05.594+0000 7f47477fe700 1 --2- 172.21.15.52:0/1799495120 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55a66dbd2e40 0x55a66dd57b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:27:05.585 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:05.594+0000 7f4746ffd700 1 --2- 172.21.15.52:0/1799495120 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55a66dd58070 0x55a66dd51be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:27:05.585 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:05.594+0000 7f4747fff700 1 -- 172.21.15.52:0/1799495120 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55a66dd58070 msgr2=0x55a66dd51be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:27:05.585 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:05.594+0000 7f4747fff700 1 --2- 172.21.15.52:0/1799495120 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55a66dd58070 0x55a66dd51be0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:27:05.585 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:05.594+0000 7f4747fff700 1 -- 172.21.15.52:0/1799495120 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55a66dbd2e40 msgr2=0x55a66dd57b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:27:05.585 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:05.594+0000 7f4747fff700 1 --2- 172.21.15.52:0/1799495120 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55a66dbd2e40 0x55a66dd57b30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:27:05.585 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:05.594+0000 7f4747fff700 1 -- 172.21.15.52:0/1799495120 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55a66dc18fb0 con 0x55a66dd52120 2024-05-27T07:27:05.585 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:05.594+0000 7f47477fe700 1 --2- 172.21.15.52:0/1799495120 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55a66dbd2e40 0x55a66dd57b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-27T07:27:05.585 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:05.595+0000 7f4747fff700 1 --2- 172.21.15.52:0/1799495120 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55a66dd52120 0x55a66dd52530 secure :-1 s=READY pgs=500 cs=0 l=1 rev1=1 crypto rx=0x7f4740011680 tx=0x7f4740009920 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f356acf3b191e16 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:27:05.586 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:05.595+0000 7f472dffb700 1 -- 172.21.15.52:0/1799495120 <== mon.0 v2:172.21.15.52:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4740019030 con 0x55a66dd52120 2024-05-27T07:27:05.586 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:05.595+0000 7f472dffb700 1 -- 172.21.15.52:0/1799495120 <== mon.0 v2:172.21.15.52:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f47400092e0 con 0x55a66dd52120 2024-05-27T07:27:05.587 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:05.595+0000 7f472dffb700 1 -- 172.21.15.52:0/1799495120 <== mon.0 v2:172.21.15.52:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4740007500 con 0x55a66dd52120 2024-05-27T07:27:05.587 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:05.596+0000 7f475c5c4b80 1 -- 172.21.15.52:0/1799495120 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55a66dd52120 msgr2=0x55a66dd52530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:27:05.587 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:05.596+0000 7f475c5c4b80 1 --2- 172.21.15.52:0/1799495120 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55a66dd52120 0x55a66dd52530 secure :-1 s=READY pgs=500 cs=0 l=1 rev1=1 crypto rx=0x7f4740011680 tx=0x7f4740009920 comp rx=0 tx=0).stop 2024-05-27T07:27:05.587 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:05.596+0000 7f475c5c4b80 1 -- 172.21.15.52:0/1799495120 shutdown_connections 2024-05-27T07:27:05.587 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:05.596+0000 7f475c5c4b80 1 --2- 172.21.15.52:0/1799495120 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55a66dd52120 0x55a66dd52530 unknown :-1 s=CLOSED pgs=500 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:27:05.587 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:05.596+0000 7f475c5c4b80 1 --2- 172.21.15.52:0/1799495120 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55a66dbd2e40 0x55a66dd57b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:27:05.587 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:05.596+0000 7f475c5c4b80 1 --2- 172.21.15.52:0/1799495120 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55a66dd58070 0x55a66dd51be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:27:05.587 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:05.596+0000 7f475c5c4b80 1 -- 172.21.15.52:0/1799495120 >> 172.21.15.52:0/1799495120 conn(0x55a66dbd5770 msgr2=0x55a66dd4cd90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:27:05.587 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:05.597+0000 7f475c5c4b80 1 -- 172.21.15.52:0/1799495120 shutdown_connections 2024-05-27T07:27:05.587 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:05.597+0000 7f475c5c4b80 1 -- 172.21.15.52:0/1799495120 wait complete. 2024-05-27T07:27:05.588 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:05.597+0000 7f475c5c4b80 1 -- start start 2024-05-27T07:27:05.588 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:05.598+0000 7f475c5c4b80 1 Processor -- start 2024-05-27T07:27:05.589 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:05.598+0000 7f475c5c4b80 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55a66dbd2e40 0x55a66dd5b860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:27:05.589 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:05.598+0000 7f47477fe700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55a66dbd2e40 0x55a66dd5b860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:27:05.589 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:05.598+0000 7f47477fe700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55a66dbd2e40 0x55a66dd5b860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.52:35838/0 (socket says 172.21.15.52:35838) 2024-05-27T07:27:05.589 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:05.598+0000 7f475c5c4b80 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55a66dd52120 0x55a66dd5bda0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:27:05.589 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:05.599+0000 7f47477fe700 1 -- 172.21.15.52:0/1777196008 learned_addr learned my addr 172.21.15.52:0/1777196008 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-05-27T07:27:05.589 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:05.599+0000 7f475c5c4b80 1 --2- 172.21.15.52:0/1777196008 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55a66dd58070 0x55a66dd5c2e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:27:05.589 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:05.599+0000 7f475c5c4b80 1 -- 172.21.15.52:0/1777196008 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x55a66db75fa0 con 0x55a66dbd2e40 2024-05-27T07:27:05.589 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:05.599+0000 7f475c5c4b80 1 -- 172.21.15.52:0/1777196008 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x55a66dbd7da0 con 0x55a66dd58070 2024-05-27T07:27:05.589 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:05.599+0000 7f4746ffd700 1 --2- 172.21.15.52:0/1777196008 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55a66dd52120 0x55a66dd5bda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:27:05.589 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:05.599+0000 7f475c5c4b80 1 -- 172.21.15.52:0/1777196008 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x55a66dd5eb60 con 0x55a66dd52120 2024-05-27T07:27:05.589 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:05.599+0000 7f4747fff700 1 --2- 172.21.15.52:0/1777196008 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55a66dd58070 0x55a66dd5c2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:27:05.591 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:05.599+0000 7f47477fe700 1 -- 172.21.15.52:0/1777196008 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55a66dd52120 msgr2=0x55a66dd5bda0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:27:05.591 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:05.599+0000 7f47477fe700 1 --2- 172.21.15.52:0/1777196008 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55a66dd52120 0x55a66dd5bda0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:27:05.592 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:05.599+0000 7f47477fe700 1 -- 172.21.15.52:0/1777196008 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55a66dd58070 msgr2=0x55a66dd5c2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:27:05.592 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:05.599+0000 7f47477fe700 1 --2- 172.21.15.52:0/1777196008 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55a66dd58070 0x55a66dd5c2e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:27:05.592 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:05.599+0000 7f47477fe700 1 -- 172.21.15.52:0/1777196008 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55a66dbbf8d0 con 0x55a66dbd2e40 2024-05-27T07:27:05.592 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:05.599+0000 7f47477fe700 1 --2- 172.21.15.52:0/1777196008 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55a66dbd2e40 0x55a66dd5b860 secure :-1 s=READY pgs=501 cs=0 l=1 rev1=1 crypto rx=0x7f47380e0180 tx=0x7f473801f5f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4c95b6f53fe796f9 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:27:05.592 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:05.599+0000 7f4726ffd700 10 client.0 ms_handle_connect on v2:172.21.15.52:3300/0 2024-05-27T07:27:05.592 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:05.599+0000 7f475c5c4b80 10 client.0 _ll_register_callbacks cb 0x55a66dc065b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T07:27:05.592 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:05.600+0000 7f475c5c4b80 1 -- 172.21.15.52:0/1777196008 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f47400092e0 con 0x55a66dbd2e40 2024-05-27T07:27:05.592 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:05.600+0000 7f4726ffd700 1 -- 172.21.15.52:0/1777196008 <== mon.0 v2:172.21.15.52:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f47380c4060 con 0x55a66dbd2e40 2024-05-27T07:27:05.592 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:05.600+0000 7f4726ffd700 1 -- 172.21.15.52:0/1777196008 <== mon.0 v2:172.21.15.52:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 9379+0+0 (secure 0 0 0) 0x7f47380b2800 con 0x55a66dbd2e40 2024-05-27T07:27:05.592 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:05.600+0000 7f4726ffd700 1 -- 172.21.15.52:0/1777196008 <== mon.0 v2:172.21.15.52:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7f47380a53b0 con 0x55a66dbd2e40 2024-05-27T07:27:05.592 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:05.600+0000 7f475c5c4b80 10 client.15711 fetch_fsmap learned FSMap version 30 2024-05-27T07:27:05.593 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:05.600+0000 7f475c5c4b80 1 -- 172.21.15.52:0/1777196008 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({fsmap=30}) -- 0x55a66dc29760 con 0x55a66dbd2e40 2024-05-27T07:27:05.593 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:05.600+0000 7f4726ffd700 1 -- 172.21.15.52:0/1777196008 <== mon.0 v2:172.21.15.52:3300/0 4 ==== fsmap(e 30) ==== 8362+0+0 (secure 0 0 0) 0x7f47380a8e80 con 0x55a66dbd2e40 2024-05-27T07:27:05.593 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:05.600+0000 7f475c5c4b80 10 client.15711 fetch_fsmap finished waiting for FSMap version 30 2024-05-27T07:27:05.593 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:05.600+0000 7f475c5c4b80 10 client.15711 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-05-27T07:27:05.593 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:05.600+0000 7f475c5c4b80 20 client.15711 populate_metadata read hostname 'smithi052' 2024-05-27T07:27:05.593 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:05.600+0000 7f475c5c4b80 1 --2- 172.21.15.52:0/1777196008 >> [v2:172.21.15.105:6838/500348151,v1:172.21.15.105:6839/500348151] conn(0x55a66dd6b650 0x55a66dd6da70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:27:05.593 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:05.600+0000 7f475c5c4b80 4 client.15711 mds_command: new command op to 24527 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-27T07:27:05.593 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:05.600+0000 7f475c5c4b80 1 -- 172.21.15.52:0/1777196008 --> [v2:172.21.15.105:6838/500348151,v1:172.21.15.105:6839/500348151] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55a66dc81cc0 con 0x55a66dd6b650 2024-05-27T07:27:05.593 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:05.601+0000 7f4746ffd700 1 --2- 172.21.15.52:0/1777196008 >> [v2:172.21.15.105:6838/500348151,v1:172.21.15.105:6839/500348151] conn(0x55a66dd6b650 0x55a66dd6da70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:27:05.593 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:05.601+0000 7f4746ffd700 1 --2- 172.21.15.52:0/1777196008 >> [v2:172.21.15.105:6838/500348151,v1:172.21.15.105:6839/500348151] conn(0x55a66dd6b650 0x55a66dd6da70 crc :-1 s=READY pgs=55 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=8a4e685e1a17f41 server_cookie=f40200e40f0cba55 in_seq=0 out_seq=0 2024-05-27T07:27:05.593 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:05.601+0000 7f4726ffd700 10 client.15711 ms_handle_connect on v2:172.21.15.105:6838/500348151 2024-05-27T07:27:05.593 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:05.602+0000 7f4726ffd700 1 -- 172.21.15.52:0/1777196008 <== mds.0 v2:172.21.15.105:6838/500348151 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24041 (crc 0 0 0) 0x55a66dc81cc0 con 0x55a66dd6b650 2024-05-27T07:27:05.594 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:05.602+0000 7f4726ffd700 10 client.15711 handle_command_reply: tid=0 2024-05-27T07:27:05.594 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:05.602+0000 7f475c5c4b80 1 client.15711 shutdown 2024-05-27T07:27:05.594 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:05.602+0000 7f475c5c4b80 1 -- 172.21.15.52:0/1777196008 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55a66dbd2e40 msgr2=0x55a66dd5b860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:27:05.594 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:05.602+0000 7f475c5c4b80 1 --2- 172.21.15.52:0/1777196008 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55a66dbd2e40 0x55a66dd5b860 secure :-1 s=READY pgs=501 cs=0 l=1 rev1=1 crypto rx=0x7f47380e0180 tx=0x7f473801f5f0 comp rx=0 tx=0).stop 2024-05-27T07:27:05.595 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:05.602+0000 7f475c5c4b80 1 -- 172.21.15.52:0/1777196008 shutdown_connections 2024-05-27T07:27:05.595 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:05.602+0000 7f475c5c4b80 1 --2- 172.21.15.52:0/1777196008 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55a66dd58070 0x55a66dd5c2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:27:05.595 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:05.602+0000 7f475c5c4b80 1 --2- 172.21.15.52:0/1777196008 >> [v2:172.21.15.105:6838/500348151,v1:172.21.15.105:6839/500348151] conn(0x55a66dd6b650 0x55a66dd6da70 crc :-1 s=READY pgs=55 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:27:05.595 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:05.602+0000 7f4726ffd700 0 client.15711 ms_handle_reset on v2:172.21.15.105:6838/500348151 2024-05-27T07:27:05.595 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:05.602+0000 7f475c5c4b80 1 --2- 172.21.15.52:0/1777196008 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55a66dbd2e40 0x55a66dd5b860 unknown :-1 s=CLOSED pgs=501 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:27:05.595 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:05.602+0000 7f475c5c4b80 1 --2- 172.21.15.52:0/1777196008 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55a66dd52120 0x55a66dd5bda0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:27:05.595 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:05.602+0000 7f475c5c4b80 1 -- 172.21.15.52:0/1777196008 >> 172.21.15.52:0/1777196008 conn(0x55a66dbd5770 msgr2=0x55a66dd50bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:27:05.595 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:05.602+0000 7f475c5c4b80 1 -- 172.21.15.52:0/1777196008 shutdown_connections 2024-05-27T07:27:05.595 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:05.602+0000 7f475c5c4b80 1 -- 172.21.15.52:0/1777196008 wait complete. 2024-05-27T07:27:05.595 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:05.603+0000 7f475c5c4b80 20 client.15711 trim_cache size 0 max 16384 2024-05-27T07:27:05.604 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:27:05.614+0000 7f5865884640 1 -- 172.21.15.52:0/1969816342 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f586010e1f0 msgr2=0x7f58601106e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:27:05.604 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:27:05.614+0000 7f5865884640 1 --2- 172.21.15.52:0/1969816342 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f586010e1f0 0x7f58601106e0 secure :-1 s=READY pgs=502 cs=0 l=1 rev1=1 crypto rx=0x7f5858009900 tx=0x7f585802a410 comp rx=0 tx=0).stop 2024-05-27T07:27:05.604 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:27:05.614+0000 7f5865884640 1 -- 172.21.15.52:0/1969816342 shutdown_connections 2024-05-27T07:27:05.604 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:27:05.614+0000 7f5865884640 1 --2- 172.21.15.52:0/1969816342 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f586010e1f0 0x7f58601106e0 unknown :-1 s=CLOSED pgs=502 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:27:05.604 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:27:05.614+0000 7f5865884640 1 --2- 172.21.15.52:0/1969816342 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f586010b800 0x7f586010dc20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:27:05.605 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:27:05.614+0000 7f5865884640 1 --2- 172.21.15.52:0/1969816342 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f5860104da0 0x7f58601051b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:27:05.605 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:27:05.614+0000 7f5865884640 1 -- 172.21.15.52:0/1969816342 >> 172.21.15.52:0/1969816342 conn(0x7f5860078270 msgr2=0x7f58600786b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:27:05.605 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:27:05.614+0000 7f5865884640 1 -- 172.21.15.52:0/1969816342 shutdown_connections 2024-05-27T07:27:05.605 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:27:05.614+0000 7f5865884640 1 -- 172.21.15.52:0/1969816342 wait complete. 2024-05-27T07:27:05.607 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:27:05.615+0000 7f5865884640 1 Processor -- start 2024-05-27T07:27:05.607 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:27:05.615+0000 7f5865884640 1 -- start start 2024-05-27T07:27:05.607 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:27:05.616+0000 7f5865884640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f5860104da0 0x7f58601a04d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:27:05.607 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:27:05.616+0000 7f5865884640 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f586010b800 0x7f58601a0a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:27:05.607 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:27:05.616+0000 7f5865884640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f586010e1f0 0x7f586019a650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:27:05.607 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:27:05.616+0000 7f5865884640 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x7f5860113e50 con 0x7f586010b800 2024-05-27T07:27:05.607 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:27:05.616+0000 7f5865884640 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f5860113cd0 con 0x7f5860104da0 2024-05-27T07:27:05.607 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:27:05.616+0000 7f5865884640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f5860113fd0 con 0x7f586010e1f0 2024-05-27T07:27:05.607 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:27:05.616+0000 7f585e7fc640 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f586010b800 0x7f58601a0a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:27:05.608 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:27:05.616+0000 7f585effd640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f5860104da0 0x7f58601a04d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:27:05.608 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:27:05.616+0000 7f585e7fc640 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f586010b800 0x7f58601a0a10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.52:35858/0 (socket says 172.21.15.52:35858) 2024-05-27T07:27:05.608 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:27:05.616+0000 7f585e7fc640 1 -- 172.21.15.52:0/2541498055 learned_addr learned my addr 172.21.15.52:0/2541498055 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-05-27T07:27:05.608 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:27:05.616+0000 7f585f7fe640 1 --2- 172.21.15.52:0/2541498055 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f586010e1f0 0x7f586019a650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:27:05.608 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:27:05.616+0000 7f585effd640 1 -- 172.21.15.52:0/2541498055 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f586010e1f0 msgr2=0x7f586019a650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:27:05.608 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:27:05.616+0000 7f585effd640 1 --2- 172.21.15.52:0/2541498055 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f586010e1f0 0x7f586019a650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:27:05.608 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:27:05.616+0000 7f585effd640 1 -- 172.21.15.52:0/2541498055 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f586010b800 msgr2=0x7f58601a0a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:27:05.608 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:27:05.616+0000 7f585effd640 1 --2- 172.21.15.52:0/2541498055 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f586010b800 0x7f58601a0a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:27:05.608 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:27:05.616+0000 7f585effd640 1 -- 172.21.15.52:0/2541498055 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f586019ad80 con 0x7f5860104da0 2024-05-27T07:27:05.608 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:27:05.616+0000 7f585e7fc640 1 --2- 172.21.15.52:0/2541498055 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f586010b800 0x7f58601a0a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-27T07:27:05.608 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:27:05.616+0000 7f585effd640 1 --2- 172.21.15.52:0/2541498055 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f5860104da0 0x7f58601a04d0 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f585000eb00 tx=0x7f585000c2d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:27:05.608 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:27:05.617+0000 7f5864882640 1 -- 172.21.15.52:0/2541498055 <== mon.2 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5850010430 con 0x7f5860104da0 2024-05-27T07:27:05.608 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:27:05.617+0000 7f5865884640 1 -- 172.21.15.52:0/2541498055 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f586019b010 con 0x7f5860104da0 2024-05-27T07:27:05.609 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:27:05.617+0000 7f5864882640 1 -- 172.21.15.52:0/2541498055 <== mon.2 v2:172.21.15.105:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f5850010a80 con 0x7f5860104da0 2024-05-27T07:27:05.609 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:27:05.617+0000 7f5864882640 1 -- 172.21.15.52:0/2541498055 <== mon.2 v2:172.21.15.105:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5850017a40 con 0x7f5860104da0 2024-05-27T07:27:05.609 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:27:05.617+0000 7f5865884640 1 -- 172.21.15.52:0/2541498055 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f58601a7270 con 0x7f5860104da0 2024-05-27T07:27:05.610 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:27:05.619+0000 7f586c80a740 1 Processor -- start 2024-05-27T07:27:05.610 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:27:05.619+0000 7f586c80a740 1 -- start start 2024-05-27T07:27:05.610 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:27:05.620+0000 7f586c80a740 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55f2d69aa1b0 0x55f2d6d4b3d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:27:05.611 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:27:05.620+0000 7f586c80a740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55f2d69812f0 0x55f2d6d4b910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:27:05.611 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:27:05.620+0000 7f586c80a740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55f2d69894d0 0x55f2d6d4be50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:27:05.611 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:27:05.620+0000 7f586c80a740 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x55f2d6cbdc30 con 0x55f2d69aa1b0 2024-05-27T07:27:05.611 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:27:05.620+0000 7f586c80a740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x55f2d6cad7d0 con 0x55f2d69894d0 2024-05-27T07:27:05.611 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:27:05.620+0000 7f586c80a740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x55f2d6cb1810 con 0x55f2d69812f0 2024-05-27T07:27:05.611 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:27:05.620+0000 7f585f7fe640 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55f2d69aa1b0 0x55f2d6d4b3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:27:05.611 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:27:05.620+0000 7f585f7fe640 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55f2d69aa1b0 0x55f2d6d4b3d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.52:35864/0 (socket says 172.21.15.52:35864) 2024-05-27T07:27:05.611 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:27:05.620+0000 7f585f7fe640 1 -- 172.21.15.52:0/1724244522 learned_addr learned my addr 172.21.15.52:0/1724244522 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-05-27T07:27:05.611 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:27:05.620+0000 7f585e7fc640 1 --2- 172.21.15.52:0/1724244522 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55f2d69894d0 0x55f2d6d4be50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:27:05.611 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:27:05.620+0000 7f585f7fe640 1 -- 172.21.15.52:0/1724244522 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55f2d69812f0 msgr2=0x55f2d6d4b910 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-05-27T07:27:05.611 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:27:05.620+0000 7f585f7fe640 1 --2- 172.21.15.52:0/1724244522 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55f2d69812f0 0x55f2d6d4b910 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:27:05.611 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:27:05.620+0000 7f585f7fe640 1 -- 172.21.15.52:0/1724244522 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55f2d69894d0 msgr2=0x55f2d6d4be50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:27:05.612 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:27:05.620+0000 7f585f7fe640 1 --2- 172.21.15.52:0/1724244522 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55f2d69894d0 0x55f2d6d4be50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:27:05.612 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:27:05.620+0000 7f585f7fe640 1 -- 172.21.15.52:0/1724244522 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55f2d69c2210 con 0x55f2d69aa1b0 2024-05-27T07:27:05.612 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:27:05.620+0000 7f585e7fc640 1 --2- 172.21.15.52:0/1724244522 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55f2d69894d0 0x55f2d6d4be50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-27T07:27:05.612 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:27:05.621+0000 7f585f7fe640 1 --2- 172.21.15.52:0/1724244522 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55f2d69aa1b0 0x55f2d6d4b3d0 secure :-1 s=READY pgs=503 cs=0 l=1 rev1=1 crypto rx=0x7f5858009900 tx=0x7f5858032bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=df0f87ec6d0ec528 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:27:05.612 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:27:05.621+0000 7f5855ffb640 1 -- 172.21.15.52:0/1724244522 <== mon.0 v2:172.21.15.52:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5858040070 con 0x55f2d69aa1b0 2024-05-27T07:27:05.612 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:27:05.621+0000 7f5855ffb640 1 -- 172.21.15.52:0/1724244522 <== mon.0 v2:172.21.15.52:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f5858004030 con 0x55f2d69aa1b0 2024-05-27T07:27:05.612 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:27:05.621+0000 7f5855ffb640 1 -- 172.21.15.52:0/1724244522 <== mon.0 v2:172.21.15.52:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5858035040 con 0x55f2d69aa1b0 2024-05-27T07:27:05.612 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:27:05.621+0000 7f5864882640 1 -- 172.21.15.52:0/2541498055 <== mon.2 v2:172.21.15.105:3300/0 4 ==== mgrmap(e 25) ==== 149936+0+0 (secure 0 0 0) 0x7f5850017ce0 con 0x7f5860104da0 2024-05-27T07:27:05.613 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:05.621+0000 7f475c5c4b80 1 Processor -- start 2024-05-27T07:27:05.613 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:05.621+0000 7f475c5c4b80 1 -- start start 2024-05-27T07:27:05.613 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:05.622+0000 7f475c5c4b80 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55a66dbd2e40 0x55a66dd51be0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:27:05.613 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:05.622+0000 7f475c5c4b80 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55a66dd52120 0x55a66dd5ab80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:27:05.614 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:27:05.622+0000 7f586c80a740 1 -- 172.21.15.52:0/1724244522 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55f2d69aa1b0 msgr2=0x55f2d6d4b3d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:27:05.614 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:27:05.622+0000 7f586c80a740 1 --2- 172.21.15.52:0/1724244522 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55f2d69aa1b0 0x55f2d6d4b3d0 secure :-1 s=READY pgs=503 cs=0 l=1 rev1=1 crypto rx=0x7f5858009900 tx=0x7f5858032bd0 comp rx=0 tx=0).stop 2024-05-27T07:27:05.614 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:27:05.622+0000 7f586c80a740 1 -- 172.21.15.52:0/1724244522 shutdown_connections 2024-05-27T07:27:05.614 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:27:05.622+0000 7f586c80a740 1 --2- 172.21.15.52:0/1724244522 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55f2d69894d0 0x55f2d6d4be50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:27:05.614 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:27:05.622+0000 7f586c80a740 1 --2- 172.21.15.52:0/1724244522 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55f2d69812f0 0x55f2d6d4b910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:27:05.614 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:27:05.622+0000 7f586c80a740 1 --2- 172.21.15.52:0/1724244522 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55f2d69aa1b0 0x55f2d6d4b3d0 unknown :-1 s=CLOSED pgs=503 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:27:05.614 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:27:05.622+0000 7f586c80a740 1 -- 172.21.15.52:0/1724244522 >> 172.21.15.52:0/1724244522 conn(0x55f2d699e300 msgr2=0x55f2d6d47b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:27:05.614 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:05.622+0000 7f475c5c4b80 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55a66dd58070 0x55a66dd67c00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:27:05.615 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:05.622+0000 7f475c5c4b80 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x7f47481b26d0 con 0x55a66dbd2e40 2024-05-27T07:27:05.615 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:05.622+0000 7f475c5c4b80 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f47481b4210 con 0x55a66dd58070 2024-05-27T07:27:05.615 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:05.622+0000 7f475c5c4b80 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f47481b2da0 con 0x55a66dd52120 2024-05-27T07:27:05.615 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:05.622+0000 7f47477fe700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55a66dbd2e40 0x55a66dd51be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:27:05.615 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:05.622+0000 7f47477fe700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55a66dbd2e40 0x55a66dd51be0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.52:35876/0 (socket says 172.21.15.52:35876) 2024-05-27T07:27:05.615 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:05.622+0000 7f47477fe700 1 -- 172.21.15.52:0/4212827943 learned_addr learned my addr 172.21.15.52:0/4212827943 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-05-27T07:27:05.615 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:27:05.622+0000 7f586c80a740 1 -- 172.21.15.52:0/1724244522 shutdown_connections 2024-05-27T07:27:05.616 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:05.622+0000 7f4747fff700 1 --2- 172.21.15.52:0/4212827943 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55a66dd58070 0x55a66dd67c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:27:05.616 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:05.622+0000 7f4746ffd700 1 --2- 172.21.15.52:0/4212827943 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55a66dd52120 0x55a66dd5ab80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:27:05.616 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:05.622+0000 7f47477fe700 1 -- 172.21.15.52:0/4212827943 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55a66dd52120 msgr2=0x55a66dd5ab80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:27:05.616 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:05.622+0000 7f47477fe700 1 --2- 172.21.15.52:0/4212827943 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55a66dd52120 0x55a66dd5ab80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:27:05.616 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:05.622+0000 7f47477fe700 1 -- 172.21.15.52:0/4212827943 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55a66dd58070 msgr2=0x55a66dd67c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:27:05.616 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:05.622+0000 7f47477fe700 1 --2- 172.21.15.52:0/4212827943 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55a66dd58070 0x55a66dd67c00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:27:05.616 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:05.622+0000 7f47477fe700 1 -- 172.21.15.52:0/4212827943 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55a66dd59800 con 0x55a66dbd2e40 2024-05-27T07:27:05.616 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:05.623+0000 7f4747fff700 1 --2- 172.21.15.52:0/4212827943 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55a66dd58070 0x55a66dd67c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-27T07:27:05.617 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:27:05.622+0000 7f586c80a740 1 -- 172.21.15.52:0/1724244522 wait complete. 2024-05-27T07:27:05.617 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:27:05.622+0000 7f5864882640 1 --2- 172.21.15.52:0/2541498055 >> [v2:172.21.15.52:6800/1822318384,v1:172.21.15.52:6801/1822318384] conn(0x7f584c0b2a10 0x7f584c0b4f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:27:05.617 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:27:05.623+0000 7f585e7fc640 1 --2- 172.21.15.52:0/2541498055 >> [v2:172.21.15.52:6800/1822318384,v1:172.21.15.52:6801/1822318384] conn(0x7f584c0b2a10 0x7f584c0b4f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:27:05.617 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:27:05.623+0000 7f586c80a740 1 -- start start 2024-05-27T07:27:05.617 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:27:05.623+0000 7f5864882640 1 -- 172.21.15.52:0/2541498055 <== mon.2 v2:172.21.15.105:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9379+0+0 (secure 0 0 0) 0x7f58500e3690 con 0x7f5860104da0 2024-05-27T07:27:05.617 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:05.623+0000 7f47477fe700 1 --2- 172.21.15.52:0/4212827943 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55a66dbd2e40 0x55a66dd51be0 secure :-1 s=READY pgs=504 cs=0 l=1 rev1=1 crypto rx=0x7f47380a6e90 tx=0x7f47380ab000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=18ead8d360d742d9 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:27:05.618 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:27:05.623+0000 7f586c80a740 1 Processor -- start 2024-05-27T07:27:05.618 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:05.623+0000 7f472d7fa700 1 -- 172.21.15.52:0/4212827943 <== mon.0 v2:172.21.15.52:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f47380b23c0 con 0x55a66dbd2e40 2024-05-27T07:27:05.618 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:05.623+0000 7f472d7fa700 1 -- 172.21.15.52:0/4212827943 <== mon.0 v2:172.21.15.52:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f47400092e0 con 0x55a66dbd2e40 2024-05-27T07:27:05.618 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:05.623+0000 7f472d7fa700 1 -- 172.21.15.52:0/4212827943 <== mon.0 v2:172.21.15.52:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f47380ae010 con 0x55a66dbd2e40 2024-05-27T07:27:05.619 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:27:05.623+0000 7f585e7fc640 1 --2- 172.21.15.52:0/2541498055 >> [v2:172.21.15.52:6800/1822318384,v1:172.21.15.52:6801/1822318384] conn(0x7f584c0b2a10 0x7f584c0b4f20 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f586019bdf0 tx=0x7f584800b410 comp rx=0 tx=0).ready entity=mgr.15333 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:27:05.620 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:27:05.624+0000 7f586c80a740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55f2d69812f0 0x55f2d6d573f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:27:05.620 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:27:05.624+0000 7f586c80a740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55f2d69894d0 0x55f2d6d57950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:27:05.620 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:27:05.624+0000 7f586c80a740 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55f2d69aa1b0 0x55f2d6d55890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:27:05.620 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:27:05.624+0000 7f586c80a740 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x55f2d6993790 con 0x55f2d69aa1b0 2024-05-27T07:27:05.620 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:27:05.624+0000 7f586c80a740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x55f2d6c69890 con 0x55f2d69812f0 2024-05-27T07:27:05.620 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:27:05.624+0000 7f586c80a740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x55f2d6d584a0 con 0x55f2d69894d0 2024-05-27T07:27:05.620 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:27:05.624+0000 7f586c80a740 10 client.0 _ll_register_callbacks cb 0x55f2d69c1a00 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T07:27:05.620 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:27:05.624+0000 7f585f7fe640 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55f2d69aa1b0 0x55f2d6d55890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:27:05.620 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:27:05.624+0000 7f585f7fe640 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55f2d69aa1b0 0x55f2d6d55890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.52:35890/0 (socket says 172.21.15.52:35890) 2024-05-27T07:27:05.620 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:27:05.624+0000 7f585f7fe640 1 -- 172.21.15.52:0/2169724176 learned_addr learned my addr 172.21.15.52:0/2169724176 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-05-27T07:27:05.620 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:27:05.624+0000 7f585e7fc640 1 --2- 172.21.15.52:0/2169724176 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55f2d69894d0 0x55f2d6d57950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:27:05.620 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:27:05.624+0000 7f585effd640 1 --2- 172.21.15.52:0/2169724176 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55f2d69812f0 0x55f2d6d573f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:27:05.621 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:27:05.624+0000 7f585f7fe640 1 -- 172.21.15.52:0/2169724176 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55f2d69894d0 msgr2=0x55f2d6d57950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:27:05.621 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:27:05.624+0000 7f585f7fe640 1 --2- 172.21.15.52:0/2169724176 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55f2d69894d0 0x55f2d6d57950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:27:05.621 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:27:05.624+0000 7f585f7fe640 1 -- 172.21.15.52:0/2169724176 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55f2d69812f0 msgr2=0x55f2d6d573f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:27:05.621 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:27:05.624+0000 7f585f7fe640 1 --2- 172.21.15.52:0/2169724176 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55f2d69812f0 0x55f2d6d573f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:27:05.621 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:27:05.624+0000 7f585f7fe640 1 -- 172.21.15.52:0/2169724176 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55f2d6a8e620 con 0x55f2d69aa1b0 2024-05-27T07:27:05.621 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:27:05.624+0000 7f585f7fe640 1 --2- 172.21.15.52:0/2169724176 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55f2d69aa1b0 0x55f2d6d55890 secure :-1 s=READY pgs=505 cs=0 l=1 rev1=1 crypto rx=0x7f5858006010 tx=0x7f5858005390 comp rx=0 tx=0).ready entity=mon.0 client_cookie=866336639d857c0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:27:05.621 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:27:05.624+0000 7f583effd640 10 client.0 ms_handle_connect on v2:172.21.15.52:3300/0 2024-05-27T07:27:05.621 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:05.624+0000 7f475c5c4b80 1 -- 172.21.15.52:0/4212827943 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55a66dbd2e40 msgr2=0x55a66dd51be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:27:05.622 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:05.625+0000 7f475c5c4b80 1 --2- 172.21.15.52:0/4212827943 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55a66dbd2e40 0x55a66dd51be0 secure :-1 s=READY pgs=504 cs=0 l=1 rev1=1 crypto rx=0x7f47380a6e90 tx=0x7f47380ab000 comp rx=0 tx=0).stop 2024-05-27T07:27:05.622 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:05.625+0000 7f475c5c4b80 1 -- 172.21.15.52:0/4212827943 shutdown_connections 2024-05-27T07:27:05.622 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:05.625+0000 7f475c5c4b80 1 --2- 172.21.15.52:0/4212827943 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55a66dd58070 0x55a66dd67c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:27:05.622 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:05.625+0000 7f475c5c4b80 1 --2- 172.21.15.52:0/4212827943 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55a66dbd2e40 0x55a66dd51be0 unknown :-1 s=CLOSED pgs=504 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:27:05.622 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:05.625+0000 7f475c5c4b80 1 --2- 172.21.15.52:0/4212827943 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55a66dd52120 0x55a66dd5ab80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:27:05.622 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:05.625+0000 7f475c5c4b80 1 -- 172.21.15.52:0/4212827943 >> 172.21.15.52:0/4212827943 conn(0x55a66dbd5770 msgr2=0x55a66dd55b00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:27:05.622 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:27:05.625+0000 7f583effd640 1 -- 172.21.15.52:0/2169724176 <== mon.0 v2:172.21.15.52:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5858035070 con 0x55f2d69aa1b0 2024-05-27T07:27:05.622 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:27:05.625+0000 7f583effd640 1 -- 172.21.15.52:0/2169724176 <== mon.0 v2:172.21.15.52:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 9379+0+0 (secure 0 0 0) 0x7f58580314a0 con 0x55f2d69aa1b0 2024-05-27T07:27:05.622 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:27:05.625+0000 7f586c80a740 1 -- 172.21.15.52:0/2169724176 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f5858004030 con 0x55f2d69aa1b0 2024-05-27T07:27:05.623 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:05.625+0000 7f475c5c4b80 1 -- 172.21.15.52:0/4212827943 shutdown_connections 2024-05-27T07:27:05.623 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:05.625+0000 7f475c5c4b80 1 -- 172.21.15.52:0/4212827943 wait complete. 2024-05-27T07:27:05.623 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:27:05.625+0000 7f583effd640 1 -- 172.21.15.52:0/2169724176 <== mon.0 v2:172.21.15.52:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7f5858058050 con 0x55f2d69aa1b0 2024-05-27T07:27:05.623 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:27:05.625+0000 7f586c80a740 10 client.15726 fetch_fsmap learned FSMap version 30 2024-05-27T07:27:05.624 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:27:05.625+0000 7f586c80a740 1 -- 172.21.15.52:0/2169724176 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x55f2d6d64870 con 0x55f2d69aa1b0 2024-05-27T07:27:05.624 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:27:05.625+0000 7f583effd640 1 -- 172.21.15.52:0/2169724176 <== mon.0 v2:172.21.15.52:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f5858004030 con 0x55f2d69aa1b0 2024-05-27T07:27:05.624 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:05.625+0000 7f475c5c4b80 1 -- start start 2024-05-27T07:27:05.624 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:27:05.625+0000 7f583effd640 1 -- 172.21.15.52:0/2169724176 <== mon.0 v2:172.21.15.52:3300/0 5 ==== fsmap(e 30) ==== 8362+0+0 (secure 0 0 0) 0x7f5858003d00 con 0x55f2d69aa1b0 2024-05-27T07:27:05.624 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:05.626+0000 7f475c5c4b80 1 Processor -- start 2024-05-27T07:27:05.625 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:27:05.626+0000 7f586c80a740 10 client.15726 fetch_fsmap finished waiting for FSMap version 30 2024-05-27T07:27:05.625 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:27:05.626+0000 7f586c80a740 10 client.15726 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-05-27T07:27:05.625 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:27:05.626+0000 7f586c80a740 20 client.15726 populate_metadata read hostname 'smithi052' 2024-05-27T07:27:05.625 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:27:05.626+0000 7f586c80a740 1 --2- 172.21.15.52:0/2169724176 >> [v2:172.21.15.105:6838/500348151,v1:172.21.15.105:6839/500348151] conn(0x55f2d6968d80 0x55f2d6d67bd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:27:05.626 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:27:05.626+0000 7f586c80a740 4 client.15726 mds_command: new command op to 24527 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-27T07:27:05.626 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:27:05.626+0000 7f586c80a740 1 -- 172.21.15.52:0/2169724176 --> [v2:172.21.15.105:6838/500348151,v1:172.21.15.105:6839/500348151] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55f2d6cbfb80 con 0x55f2d6968d80 2024-05-27T07:27:05.626 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:27:05.626+0000 7f585effd640 1 --2- 172.21.15.52:0/2169724176 >> [v2:172.21.15.105:6838/500348151,v1:172.21.15.105:6839/500348151] conn(0x55f2d6968d80 0x55f2d6d67bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:27:05.627 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:05.626+0000 7f475c5c4b80 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55a66dbd2e40 0x55a66dd63df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:27:05.627 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:05.626+0000 7f475c5c4b80 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55a66dd52120 0x55a66dd64330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:27:05.627 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:05.626+0000 7f475c5c4b80 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55a66dd58070 0x55a66dd60dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:27:05.627 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:05.626+0000 7f47477fe700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55a66dbd2e40 0x55a66dd63df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:27:05.627 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:05.626+0000 7f47477fe700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55a66dbd2e40 0x55a66dd63df0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.52:35904/0 (socket says 172.21.15.52:35904) 2024-05-27T07:27:05.627 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:05.626+0000 7f47477fe700 1 -- 172.21.15.52:0/3766326738 learned_addr learned my addr 172.21.15.52:0/3766326738 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-05-27T07:27:05.627 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:05.626+0000 7f475c5c4b80 1 -- 172.21.15.52:0/3766326738 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x7f47481b3b40 con 0x55a66dbd2e40 2024-05-27T07:27:05.627 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:05.626+0000 7f475c5c4b80 1 -- 172.21.15.52:0/3766326738 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f47481b3470 con 0x55a66dd52120 2024-05-27T07:27:05.627 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:05.626+0000 7f475c5c4b80 1 -- 172.21.15.52:0/3766326738 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f47481b0670 con 0x55a66dd58070 2024-05-27T07:27:05.627 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:05.626+0000 7f4746ffd700 1 --2- 172.21.15.52:0/3766326738 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55a66dd52120 0x55a66dd64330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:27:05.628 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:05.626+0000 7f475c5c4b80 10 client.0 _ll_register_callbacks cb 0x55a66dc065b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T07:27:05.628 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:27:05.627+0000 7f583effd640 10 client.15726 ms_handle_connect on v2:172.21.15.105:6838/500348151 2024-05-27T07:27:05.628 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:27:05.627+0000 7f585effd640 1 --2- 172.21.15.52:0/2169724176 >> [v2:172.21.15.105:6838/500348151,v1:172.21.15.105:6839/500348151] conn(0x55f2d6968d80 0x55f2d6d67bd0 crc :-1 s=READY pgs=56 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=6e64df610778ee85 server_cookie=9d99ccf6c25a119a in_seq=0 out_seq=0 2024-05-27T07:27:05.629 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:05.627+0000 7f4747fff700 1 --2- 172.21.15.52:0/3766326738 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55a66dd58070 0x55a66dd60dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:27:05.629 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:05.627+0000 7f47477fe700 1 -- 172.21.15.52:0/3766326738 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55a66dd58070 msgr2=0x55a66dd60dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:27:05.629 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:05.627+0000 7f47477fe700 1 --2- 172.21.15.52:0/3766326738 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55a66dd58070 0x55a66dd60dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:27:05.629 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:05.627+0000 7f47477fe700 1 -- 172.21.15.52:0/3766326738 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55a66dd52120 msgr2=0x55a66dd64330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:27:05.629 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:05.627+0000 7f47477fe700 1 --2- 172.21.15.52:0/3766326738 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55a66dd52120 0x55a66dd64330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:27:05.629 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:05.627+0000 7f47477fe700 1 -- 172.21.15.52:0/3766326738 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55a66dd61300 con 0x55a66dbd2e40 2024-05-27T07:27:05.629 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:05.627+0000 7f4746ffd700 1 --2- 172.21.15.52:0/3766326738 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55a66dd52120 0x55a66dd64330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-27T07:27:05.629 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:05.627+0000 7f47477fe700 1 --2- 172.21.15.52:0/3766326738 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55a66dbd2e40 0x55a66dd63df0 secure :-1 s=READY pgs=506 cs=0 l=1 rev1=1 crypto rx=0x7f4738000c00 tx=0x7f4738005910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=be8aa2a2054eb977 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:27:05.629 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:05.627+0000 7f4726ffd700 10 client.0 ms_handle_connect on v2:172.21.15.52:3300/0 2024-05-27T07:27:05.629 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:05.627+0000 7f4726ffd700 1 -- 172.21.15.52:0/3766326738 <== mon.0 v2:172.21.15.52:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f47380c4040 con 0x55a66dbd2e40 2024-05-27T07:27:05.630 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:05.627+0000 7f4726ffd700 1 -- 172.21.15.52:0/3766326738 <== mon.0 v2:172.21.15.52:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 9379+0+0 (secure 0 0 0) 0x7f47380a57a0 con 0x55a66dbd2e40 2024-05-27T07:27:05.630 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:05.627+0000 7f475c5c4b80 1 -- 172.21.15.52:0/3766326738 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f47400092e0 con 0x55a66dbd2e40 2024-05-27T07:27:05.630 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:05.627+0000 7f4726ffd700 1 -- 172.21.15.52:0/3766326738 <== mon.0 v2:172.21.15.52:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7f47380b23c0 con 0x55a66dbd2e40 2024-05-27T07:27:05.630 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:27:05.627+0000 7f583effd640 1 -- 172.21.15.52:0/2169724176 <== mds.0 v2:172.21.15.105:6838/500348151 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24041 (crc 0 0 0) 0x55f2d6cbfb80 con 0x55f2d6968d80 2024-05-27T07:27:05.630 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:27:05.627+0000 7f583effd640 10 client.15726 handle_command_reply: tid=0 2024-05-27T07:27:05.630 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:05.627+0000 7f475c5c4b80 10 client.15729 fetch_fsmap learned FSMap version 30 2024-05-27T07:27:05.630 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:05.627+0000 7f475c5c4b80 1 -- 172.21.15.52:0/3766326738 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x55a66dddeef0 con 0x55a66dbd2e40 2024-05-27T07:27:05.631 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:27:05.628+0000 7f586c80a740 1 client.15726 shutdown 2024-05-27T07:27:05.631 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:05.628+0000 7f4726ffd700 1 -- 172.21.15.52:0/3766326738 <== mon.0 v2:172.21.15.52:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f47400092e0 con 0x55a66dbd2e40 2024-05-27T07:27:05.631 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:05.628+0000 7f4726ffd700 1 -- 172.21.15.52:0/3766326738 <== mon.0 v2:172.21.15.52:3300/0 5 ==== fsmap(e 30) ==== 8362+0+0 (secure 0 0 0) 0x7f47380e03a0 con 0x55a66dbd2e40 2024-05-27T07:27:05.632 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:05.628+0000 7f475c5c4b80 10 client.15729 fetch_fsmap finished waiting for FSMap version 30 2024-05-27T07:27:05.632 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:05.628+0000 7f475c5c4b80 10 client.15729 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-05-27T07:27:05.632 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:05.628+0000 7f475c5c4b80 20 client.15729 populate_metadata read hostname 'smithi052' 2024-05-27T07:27:05.632 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:05.628+0000 7f475c5c4b80 1 --2- 172.21.15.52:0/3766326738 >> [v2:172.21.15.105:6838/500348151,v1:172.21.15.105:6839/500348151] conn(0x55a66dd6b650 0x55a66dde13c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:27:05.632 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:05.628+0000 7f475c5c4b80 4 client.15729 mds_command: new command op to 24527 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-05-27T07:27:05.632 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:05.628+0000 7f475c5c4b80 1 -- 172.21.15.52:0/3766326738 --> [v2:172.21.15.105:6838/500348151,v1:172.21.15.105:6839/500348151] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55a66dc7b4a0 con 0x55a66dd6b650 2024-05-27T07:27:05.632 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:05.628+0000 7f4746ffd700 1 --2- 172.21.15.52:0/3766326738 >> [v2:172.21.15.105:6838/500348151,v1:172.21.15.105:6839/500348151] conn(0x55a66dd6b650 0x55a66dde13c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:27:05.633 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:27:05.628+0000 7f586c80a740 1 -- 172.21.15.52:0/2169724176 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55f2d69aa1b0 msgr2=0x55f2d6d55890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:27:05.633 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:27:05.628+0000 7f586c80a740 1 --2- 172.21.15.52:0/2169724176 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55f2d69aa1b0 0x55f2d6d55890 secure :-1 s=READY pgs=505 cs=0 l=1 rev1=1 crypto rx=0x7f5858006010 tx=0x7f5858005390 comp rx=0 tx=0).stop 2024-05-27T07:27:05.633 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:27:05.628+0000 7f586c80a740 1 -- 172.21.15.52:0/2169724176 shutdown_connections 2024-05-27T07:27:05.633 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:27:05.628+0000 7f586c80a740 1 --2- 172.21.15.52:0/2169724176 >> [v2:172.21.15.105:6838/500348151,v1:172.21.15.105:6839/500348151] conn(0x55f2d6968d80 0x55f2d6d67bd0 crc :-1 s=READY pgs=56 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:27:05.633 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:27:05.628+0000 7f586c80a740 1 --2- 172.21.15.52:0/2169724176 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55f2d69aa1b0 0x55f2d6d55890 unknown :-1 s=CLOSED pgs=505 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:27:05.633 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:27:05.628+0000 7f586c80a740 1 --2- 172.21.15.52:0/2169724176 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55f2d69894d0 0x55f2d6d57950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:27:05.633 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:27:05.628+0000 7f583effd640 0 client.15726 ms_handle_reset on v2:172.21.15.105:6838/500348151 2024-05-27T07:27:05.633 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:27:05.628+0000 7f586c80a740 1 --2- 172.21.15.52:0/2169724176 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55f2d69812f0 0x55f2d6d573f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:27:05.633 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:27:05.628+0000 7f586c80a740 1 -- 172.21.15.52:0/2169724176 >> 172.21.15.52:0/2169724176 conn(0x55f2d699e300 msgr2=0x55f2d6d4b1a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:27:05.634 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:27:05.628+0000 7f586c80a740 1 -- 172.21.15.52:0/2169724176 shutdown_connections 2024-05-27T07:27:05.634 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:27:05.628+0000 7f586c80a740 1 -- 172.21.15.52:0/2169724176 wait complete. 2024-05-27T07:27:05.634 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:05.629+0000 7f4726ffd700 10 client.15729 ms_handle_connect on v2:172.21.15.105:6838/500348151 2024-05-27T07:27:05.634 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:27:05.629+0000 7f586c80a740 20 client.15726 trim_cache size 0 max 16384 2024-05-27T07:27:05.635 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:05.629+0000 7f4746ffd700 1 --2- 172.21.15.52:0/3766326738 >> [v2:172.21.15.105:6838/500348151,v1:172.21.15.105:6839/500348151] conn(0x55a66dd6b650 0x55a66dde13c0 crc :-1 s=READY pgs=57 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=1005bdd411ccad5e server_cookie=4b92cf3f787ebaf9 in_seq=0 out_seq=0 2024-05-27T07:27:05.635 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:05.629+0000 7f4726ffd700 1 -- 172.21.15.52:0/3766326738 <== mds.0 v2:172.21.15.105:6838/500348151 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55a66dc7b4a0 con 0x55a66dd6b650 2024-05-27T07:27:05.635 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:05.629+0000 7f4726ffd700 10 client.15729 handle_command_reply: tid=0 2024-05-27T07:27:05.635 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:05.629+0000 7f475c5c4b80 1 client.15729 shutdown 2024-05-27T07:27:05.635 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:05.629+0000 7f475c5c4b80 1 -- 172.21.15.52:0/3766326738 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55a66dbd2e40 msgr2=0x55a66dd63df0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:27:05.636 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:05.629+0000 7f475c5c4b80 1 --2- 172.21.15.52:0/3766326738 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55a66dbd2e40 0x55a66dd63df0 secure :-1 s=READY pgs=506 cs=0 l=1 rev1=1 crypto rx=0x7f4738000c00 tx=0x7f4738005910 comp rx=0 tx=0).stop 2024-05-27T07:27:05.636 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:05.629+0000 7f475c5c4b80 1 -- 172.21.15.52:0/3766326738 shutdown_connections 2024-05-27T07:27:05.636 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:05.629+0000 7f475c5c4b80 1 --2- 172.21.15.52:0/3766326738 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55a66dd58070 0x55a66dd60dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:27:05.636 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:05.629+0000 7f475c5c4b80 1 --2- 172.21.15.52:0/3766326738 >> [v2:172.21.15.105:6838/500348151,v1:172.21.15.105:6839/500348151] conn(0x55a66dd6b650 0x55a66dde13c0 crc :-1 s=READY pgs=57 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:27:05.636 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:05.629+0000 7f475c5c4b80 1 --2- 172.21.15.52:0/3766326738 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55a66dbd2e40 0x55a66dd63df0 unknown :-1 s=CLOSED pgs=506 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:27:05.636 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:05.629+0000 7f475c5c4b80 1 --2- 172.21.15.52:0/3766326738 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55a66dd52120 0x55a66dd64330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:27:05.636 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:05.629+0000 7f475c5c4b80 1 -- 172.21.15.52:0/3766326738 >> 172.21.15.52:0/3766326738 conn(0x55a66dbd5770 msgr2=0x55a66dd464f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:27:05.636 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:05.630+0000 7f4726ffd700 0 client.15729 ms_handle_reset on v2:172.21.15.105:6838/500348151 2024-05-27T07:27:05.636 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:05.630+0000 7f475c5c4b80 1 -- 172.21.15.52:0/3766326738 shutdown_connections 2024-05-27T07:27:05.636 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:05.630+0000 7f475c5c4b80 1 -- 172.21.15.52:0/3766326738 wait complete. 2024-05-27T07:27:05.636 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:05.630+0000 7f475c5c4b80 20 client.15729 trim_cache size 0 max 16384 2024-05-27T07:27:05.637 INFO:teuthology.orchestra.run.smithi052.stdout:[] 2024-05-27T07:27:05.637 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:05.630+0000 7f474dbaf700 1 -- 172.21.15.52:0/2137997426 >> [v2:172.21.15.52:6800/1822318384,v1:172.21.15.52:6801/1822318384] conn(0x7f47300b2c20 msgr2=0x7f47300b5130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:27:05.637 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:05.630+0000 7f474dbaf700 1 --2- 172.21.15.52:0/2137997426 >> [v2:172.21.15.52:6800/1822318384,v1:172.21.15.52:6801/1822318384] conn(0x7f47300b2c20 0x7f47300b5130 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f473c006010 tx=0x7f473c00b280 comp rx=0 tx=0).stop 2024-05-27T07:27:05.637 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:05.630+0000 7f474dbaf700 1 -- 172.21.15.52:0/2137997426 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f4748106560 msgr2=0x7f474819cfd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:27:05.637 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:05.630+0000 7f474dbaf700 1 --2- 172.21.15.52:0/2137997426 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f4748106560 0x7f474819cfd0 secure :-1 s=READY pgs=499 cs=0 l=1 rev1=1 crypto rx=0x7f473800e970 tx=0x7f473800ec80 comp rx=0 tx=0).stop 2024-05-27T07:27:05.637 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:05.631+0000 7f474dbaf700 1 -- 172.21.15.52:0/2137997426 shutdown_connections 2024-05-27T07:27:05.638 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:05.631+0000 7f474dbaf700 1 --2- 172.21.15.52:0/2137997426 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f474810c5a0 0x7f47481a1800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:27:05.638 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:05.631+0000 7f474dbaf700 1 --2- 172.21.15.52:0/2137997426 >> [v2:172.21.15.52:6800/1822318384,v1:172.21.15.52:6801/1822318384] conn(0x7f47300b2c20 0x7f47300b5130 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:27:05.638 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:05.631+0000 7f474dbaf700 1 --2- 172.21.15.52:0/2137997426 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f4748106560 0x7f474819cfd0 unknown :-1 s=CLOSED pgs=499 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:27:05.638 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:05.631+0000 7f474dbaf700 1 --2- 172.21.15.52:0/2137997426 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f4748106eb0 0x7f474819d570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:27:05.638 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:05.631+0000 7f474dbaf700 1 -- 172.21.15.52:0/2137997426 >> 172.21.15.52:0/2137997426 conn(0x7f4748102040 msgr2=0x7f4748103840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:27:05.638 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:05.631+0000 7f474dbaf700 1 -- 172.21.15.52:0/2137997426 shutdown_connections 2024-05-27T07:27:05.638 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:05.631+0000 7f474dbaf700 1 -- 172.21.15.52:0/2137997426 wait complete. 2024-05-27T07:27:05.645 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:27:05.654+0000 7f586c80a740 1 Processor -- start 2024-05-27T07:27:05.645 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:27:05.654+0000 7f586c80a740 1 -- start start 2024-05-27T07:27:05.645 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:27:05.654+0000 7f586c80a740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55f2d69812f0 0x55f2d6d4f020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:27:05.646 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:27:05.654+0000 7f586c80a740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55f2d69894d0 0x55f2d6d4f560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:27:05.646 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:27:05.654+0000 7f586c80a740 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55f2d69aa1b0 0x55f2d6d4b160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:27:05.646 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:27:05.655+0000 7f585f7fe640 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55f2d69aa1b0 0x55f2d6d4b160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:27:05.646 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:27:05.655+0000 7f585f7fe640 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55f2d69aa1b0 0x55f2d6d4b160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.52:35906/0 (socket says 172.21.15.52:35906) 2024-05-27T07:27:05.646 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:27:05.655+0000 7f585f7fe640 1 -- 172.21.15.52:0/1078806599 learned_addr learned my addr 172.21.15.52:0/1078806599 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-05-27T07:27:05.646 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:27:05.655+0000 7f585effd640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55f2d69812f0 0x55f2d6d4f020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:27:05.646 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:27:05.655+0000 7f585e7fc640 1 --2- 172.21.15.52:0/1078806599 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55f2d69894d0 0x55f2d6d4f560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:27:05.646 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:27:05.655+0000 7f586c80a740 1 -- 172.21.15.52:0/1078806599 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x7f58601b3370 con 0x55f2d69aa1b0 2024-05-27T07:27:05.646 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:27:05.655+0000 7f586c80a740 1 -- 172.21.15.52:0/1078806599 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f58601b4eb0 con 0x55f2d69894d0 2024-05-27T07:27:05.646 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:27:05.655+0000 7f586c80a740 1 -- 172.21.15.52:0/1078806599 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f58601b3a40 con 0x55f2d69812f0 2024-05-27T07:27:05.646 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:27:05.655+0000 7f585effd640 1 -- 172.21.15.52:0/1078806599 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55f2d69894d0 msgr2=0x55f2d6d4f560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:27:05.646 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:27:05.655+0000 7f585effd640 1 --2- 172.21.15.52:0/1078806599 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55f2d69894d0 0x55f2d6d4f560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:27:05.647 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:27:05.655+0000 7f585effd640 1 -- 172.21.15.52:0/1078806599 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55f2d69aa1b0 msgr2=0x55f2d6d4b160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:27:05.647 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:27:05.655+0000 7f585effd640 1 --2- 172.21.15.52:0/1078806599 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55f2d69aa1b0 0x55f2d6d4b160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:27:05.647 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:27:05.655+0000 7f585effd640 1 -- 172.21.15.52:0/1078806599 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55f2d6d567b0 con 0x55f2d69812f0 2024-05-27T07:27:05.647 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:27:05.655+0000 7f585f7fe640 1 --2- 172.21.15.52:0/1078806599 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55f2d69aa1b0 0x55f2d6d4b160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-27T07:27:05.647 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:27:05.655+0000 7f585effd640 1 --2- 172.21.15.52:0/1078806599 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55f2d69812f0 0x55f2d6d4f020 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f58500e0840 tx=0x7f585000cf10 comp rx=0 tx=0).ready entity=mon.1 client_cookie=5b8a3b01b2287913 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:27:05.647 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:27:05.656+0000 7f58557fa640 1 -- 172.21.15.52:0/1078806599 <== mon.1 v2:172.21.15.173:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f58500aa3d0 con 0x55f2d69812f0 2024-05-27T07:27:05.647 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:27:05.656+0000 7f58557fa640 1 -- 172.21.15.52:0/1078806599 <== mon.1 v2:172.21.15.173:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f58500c3070 con 0x55f2d69812f0 2024-05-27T07:27:05.647 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:27:05.656+0000 7f58557fa640 1 -- 172.21.15.52:0/1078806599 <== mon.1 v2:172.21.15.173:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f58500b2da0 con 0x55f2d69812f0 2024-05-27T07:27:05.650 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:27:05.657+0000 7f586c80a740 1 -- 172.21.15.52:0/1078806599 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55f2d69812f0 msgr2=0x55f2d6d4f020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:27:05.650 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:27:05.657+0000 7f586c80a740 1 --2- 172.21.15.52:0/1078806599 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55f2d69812f0 0x55f2d6d4f020 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f58500e0840 tx=0x7f585000cf10 comp rx=0 tx=0).stop 2024-05-27T07:27:05.650 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:27:05.657+0000 7f586c80a740 1 -- 172.21.15.52:0/1078806599 shutdown_connections 2024-05-27T07:27:05.650 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:27:05.657+0000 7f586c80a740 1 --2- 172.21.15.52:0/1078806599 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55f2d69aa1b0 0x55f2d6d4b160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:27:05.650 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:27:05.657+0000 7f586c80a740 1 --2- 172.21.15.52:0/1078806599 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55f2d69894d0 0x55f2d6d4f560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:27:05.650 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:27:05.657+0000 7f586c80a740 1 --2- 172.21.15.52:0/1078806599 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55f2d69812f0 0x55f2d6d4f020 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:27:05.651 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:27:05.657+0000 7f586c80a740 1 -- 172.21.15.52:0/1078806599 >> 172.21.15.52:0/1078806599 conn(0x55f2d699e300 msgr2=0x55f2d6d54160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:27:05.651 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:27:05.657+0000 7f586c80a740 1 -- 172.21.15.52:0/1078806599 shutdown_connections 2024-05-27T07:27:05.651 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:27:05.657+0000 7f586c80a740 1 -- 172.21.15.52:0/1078806599 wait complete. 2024-05-27T07:27:05.651 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:27:05.657+0000 7f586c80a740 1 -- start start 2024-05-27T07:27:05.651 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:27:05.658+0000 7f586c80a740 1 Processor -- start 2024-05-27T07:27:05.651 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:27:05.658+0000 7f586c80a740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55f2d69812f0 0x55f2d6dc86f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:27:05.651 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:27:05.658+0000 7f586c80a740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55f2d69894d0 0x55f2d6dc8c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:27:05.651 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:27:05.658+0000 7f586c80a740 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55f2d69aa1b0 0x55f2d6dc2ae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:27:05.651 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:27:05.658+0000 7f586c80a740 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x7f58601b47e0 con 0x55f2d69aa1b0 2024-05-27T07:27:05.651 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:27:05.658+0000 7f586c80a740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f58601b4110 con 0x55f2d69812f0 2024-05-27T07:27:05.651 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:27:05.658+0000 7f586c80a740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f58601b1350 con 0x55f2d69894d0 2024-05-27T07:27:05.651 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:27:05.658+0000 7f586c80a740 10 client.0 _ll_register_callbacks cb 0x7f584800bbc0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T07:27:05.652 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:27:05.658+0000 7f585f7fe640 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55f2d69aa1b0 0x55f2d6dc2ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:27:05.652 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:27:05.658+0000 7f585f7fe640 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55f2d69aa1b0 0x55f2d6dc2ae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.52:35922/0 (socket says 172.21.15.52:35922) 2024-05-27T07:27:05.652 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:27:05.658+0000 7f585f7fe640 1 -- 172.21.15.52:0/145852138 learned_addr learned my addr 172.21.15.52:0/145852138 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-05-27T07:27:05.652 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:27:05.659+0000 7f585e7fc640 1 --2- 172.21.15.52:0/145852138 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55f2d69894d0 0x55f2d6dc8c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:27:05.652 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:27:05.659+0000 7f585effd640 1 --2- 172.21.15.52:0/145852138 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55f2d69812f0 0x55f2d6dc86f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:27:05.652 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:27:05.659+0000 7f585f7fe640 1 -- 172.21.15.52:0/145852138 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55f2d69894d0 msgr2=0x55f2d6dc8c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:27:05.652 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:27:05.659+0000 7f585f7fe640 1 --2- 172.21.15.52:0/145852138 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55f2d69894d0 0x55f2d6dc8c30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:27:05.652 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:27:05.659+0000 7f585f7fe640 1 -- 172.21.15.52:0/145852138 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55f2d69812f0 msgr2=0x55f2d6dc86f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:27:05.652 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:27:05.659+0000 7f585f7fe640 1 --2- 172.21.15.52:0/145852138 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55f2d69812f0 0x55f2d6dc86f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:27:05.652 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:27:05.659+0000 7f585f7fe640 1 -- 172.21.15.52:0/145852138 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55f2d6dc30f0 con 0x55f2d69aa1b0 2024-05-27T07:27:05.652 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:27:05.659+0000 7f585effd640 1 --2- 172.21.15.52:0/145852138 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55f2d69812f0 0x55f2d6dc86f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-27T07:27:05.652 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:27:05.659+0000 7f585f7fe640 1 --2- 172.21.15.52:0/145852138 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55f2d69aa1b0 0x55f2d6dc2ae0 secure :-1 s=READY pgs=507 cs=0 l=1 rev1=1 crypto rx=0x7f5858032d70 tx=0x7f5858009810 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b52f64f6a06019cb server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:27:05.653 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:27:05.659+0000 7f583effd640 10 client.0 ms_handle_connect on v2:172.21.15.52:3300/0 2024-05-27T07:27:05.653 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:27:05.659+0000 7f583effd640 1 -- 172.21.15.52:0/145852138 <== mon.0 v2:172.21.15.52:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5858060020 con 0x55f2d69aa1b0 2024-05-27T07:27:05.653 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:27:05.659+0000 7f583effd640 1 -- 172.21.15.52:0/145852138 <== mon.0 v2:172.21.15.52:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 9379+0+0 (secure 0 0 0) 0x7f58580316a0 con 0x55f2d69aa1b0 2024-05-27T07:27:05.653 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:27:05.659+0000 7f586c80a740 1 -- 172.21.15.52:0/145852138 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f58500c3070 con 0x55f2d69aa1b0 2024-05-27T07:27:05.653 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:27:05.660+0000 7f583effd640 1 -- 172.21.15.52:0/145852138 <== mon.0 v2:172.21.15.52:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7f5858035040 con 0x55f2d69aa1b0 2024-05-27T07:27:05.653 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:27:05.660+0000 7f586c80a740 10 client.15735 fetch_fsmap learned FSMap version 30 2024-05-27T07:27:05.653 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:27:05.660+0000 7f586c80a740 1 -- 172.21.15.52:0/145852138 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x55f2d6dcfda0 con 0x55f2d69aa1b0 2024-05-27T07:27:05.653 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:27:05.660+0000 7f583effd640 1 -- 172.21.15.52:0/145852138 <== mon.0 v2:172.21.15.52:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f58500c3070 con 0x55f2d69aa1b0 2024-05-27T07:27:05.653 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:27:05.660+0000 7f583effd640 1 -- 172.21.15.52:0/145852138 <== mon.0 v2:172.21.15.52:3300/0 5 ==== fsmap(e 30) ==== 8362+0+0 (secure 0 0 0) 0x7f585803c440 con 0x55f2d69aa1b0 2024-05-27T07:27:05.653 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:27:05.660+0000 7f586c80a740 10 client.15735 fetch_fsmap finished waiting for FSMap version 30 2024-05-27T07:27:05.653 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:27:05.660+0000 7f586c80a740 10 client.15735 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-05-27T07:27:05.654 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:27:05.660+0000 7f586c80a740 20 client.15735 populate_metadata read hostname 'smithi052' 2024-05-27T07:27:05.654 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:27:05.660+0000 7f586c80a740 1 --2- 172.21.15.52:0/145852138 >> [v2:172.21.15.105:6838/500348151,v1:172.21.15.105:6839/500348151] conn(0x55f2d693e1c0 0x55f2d6dd2280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:27:05.654 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:27:05.660+0000 7f586c80a740 4 client.15735 mds_command: new command op to 24527 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-05-27T07:27:05.654 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:27:05.660+0000 7f586c80a740 1 -- 172.21.15.52:0/145852138 --> [v2:172.21.15.105:6838/500348151,v1:172.21.15.105:6839/500348151] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55f2d6cb9360 con 0x55f2d693e1c0 2024-05-27T07:27:05.654 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:27:05.661+0000 7f585effd640 1 --2- 172.21.15.52:0/145852138 >> [v2:172.21.15.105:6838/500348151,v1:172.21.15.105:6839/500348151] conn(0x55f2d693e1c0 0x55f2d6dd2280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:27:05.654 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:27:05.661+0000 7f585effd640 1 --2- 172.21.15.52:0/145852138 >> [v2:172.21.15.105:6838/500348151,v1:172.21.15.105:6839/500348151] conn(0x55f2d693e1c0 0x55f2d6dd2280 crc :-1 s=READY pgs=58 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=acd9bddf631cff99 server_cookie=1b3e0ec72180b9c4 in_seq=0 out_seq=0 2024-05-27T07:27:05.654 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:27:05.661+0000 7f583effd640 10 client.15735 ms_handle_connect on v2:172.21.15.105:6838/500348151 2024-05-27T07:27:05.657 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:27:05.667+0000 7f583effd640 1 -- 172.21.15.52:0/145852138 <== mds.0 v2:172.21.15.105:6838/500348151 1 ==== command_reply(tid 0: 0 ) ==== 8+0+25 (crc 0 0 0) 0x55f2d6cb9360 con 0x55f2d693e1c0 2024-05-27T07:27:05.657 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:27:05.667+0000 7f583effd640 10 client.15735 handle_command_reply: tid=0 2024-05-27T07:27:05.657 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:27:05.667+0000 7f586c80a740 1 client.15735 shutdown 2024-05-27T07:27:05.658 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:27:05.668+0000 7f586c80a740 1 -- 172.21.15.52:0/145852138 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55f2d69aa1b0 msgr2=0x55f2d6dc2ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:27:05.658 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:27:05.668+0000 7f586c80a740 1 --2- 172.21.15.52:0/145852138 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55f2d69aa1b0 0x55f2d6dc2ae0 secure :-1 s=READY pgs=507 cs=0 l=1 rev1=1 crypto rx=0x7f5858032d70 tx=0x7f5858009810 comp rx=0 tx=0).stop 2024-05-27T07:27:05.658 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:27:05.668+0000 7f586c80a740 1 -- 172.21.15.52:0/145852138 shutdown_connections 2024-05-27T07:27:05.658 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:27:05.668+0000 7f586c80a740 1 --2- 172.21.15.52:0/145852138 >> [v2:172.21.15.105:6838/500348151,v1:172.21.15.105:6839/500348151] conn(0x55f2d693e1c0 0x55f2d6dd2280 crc :-1 s=READY pgs=58 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:27:05.658 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:27:05.668+0000 7f586c80a740 1 --2- 172.21.15.52:0/145852138 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55f2d69aa1b0 0x55f2d6dc2ae0 unknown :-1 s=CLOSED pgs=507 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:27:05.658 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:27:05.668+0000 7f586c80a740 1 --2- 172.21.15.52:0/145852138 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55f2d69894d0 0x55f2d6dc8c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:27:05.658 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:27:05.668+0000 7f583effd640 0 client.15735 ms_handle_reset on v2:172.21.15.105:6838/500348151 2024-05-27T07:27:05.659 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:27:05.668+0000 7f586c80a740 1 --2- 172.21.15.52:0/145852138 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55f2d69812f0 0x55f2d6dc86f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:27:05.659 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:27:05.668+0000 7f586c80a740 1 -- 172.21.15.52:0/145852138 >> 172.21.15.52:0/145852138 conn(0x55f2d699e300 msgr2=0x55f2d6d4f020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:27:05.659 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:27:05.668+0000 7f586c80a740 1 -- 172.21.15.52:0/145852138 shutdown_connections 2024-05-27T07:27:05.659 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:27:05.668+0000 7f586c80a740 1 -- 172.21.15.52:0/145852138 wait complete. 2024-05-27T07:27:05.659 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:27:05.668+0000 7f586c80a740 20 client.15735 trim_cache size 0 max 16384 2024-05-27T07:27:05.659 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stdout:{ 2024-05-27T07:27:05.659 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stdout: "return_code": 0 2024-05-27T07:27:05.659 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stdout:} 2024-05-27T07:27:05.660 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:27:05.669+0000 7f5865884640 1 -- 172.21.15.52:0/2541498055 >> [v2:172.21.15.52:6800/1822318384,v1:172.21.15.52:6801/1822318384] conn(0x7f584c0b2a10 msgr2=0x7f584c0b4f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:27:05.660 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:27:05.669+0000 7f5865884640 1 --2- 172.21.15.52:0/2541498055 >> [v2:172.21.15.52:6800/1822318384,v1:172.21.15.52:6801/1822318384] conn(0x7f584c0b2a10 0x7f584c0b4f20 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f586019bdf0 tx=0x7f584800b410 comp rx=0 tx=0).stop 2024-05-27T07:27:05.660 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:27:05.669+0000 7f5865884640 1 -- 172.21.15.52:0/2541498055 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f5860104da0 msgr2=0x7f58601a04d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:27:05.660 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:27:05.669+0000 7f5865884640 1 --2- 172.21.15.52:0/2541498055 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f5860104da0 0x7f58601a04d0 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f585000eb00 tx=0x7f585000c2d0 comp rx=0 tx=0).stop 2024-05-27T07:27:05.660 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:27:05.669+0000 7f5865884640 1 -- 172.21.15.52:0/2541498055 shutdown_connections 2024-05-27T07:27:05.660 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:27:05.669+0000 7f5865884640 1 --2- 172.21.15.52:0/2541498055 >> [v2:172.21.15.52:6800/1822318384,v1:172.21.15.52:6801/1822318384] conn(0x7f584c0b2a10 0x7f584c0b4f20 unknown :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:27:05.661 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:27:05.669+0000 7f5865884640 1 --2- 172.21.15.52:0/2541498055 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f586010e1f0 0x7f586019a650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:27:05.661 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:27:05.669+0000 7f5865884640 1 --2- 172.21.15.52:0/2541498055 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f586010b800 0x7f58601a0a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:27:05.661 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:27:05.669+0000 7f5865884640 1 --2- 172.21.15.52:0/2541498055 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f5860104da0 0x7f58601a04d0 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:27:05.661 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:27:05.669+0000 7f5865884640 1 -- 172.21.15.52:0/2541498055 >> 172.21.15.52:0/2541498055 conn(0x7f5860078270 msgr2=0x7f5860101360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:27:05.661 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:27:05.669+0000 7f5865884640 1 -- 172.21.15.52:0/2541498055 shutdown_connections 2024-05-27T07:27:05.661 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:27:05.669+0000 7f5865884640 1 -- 172.21.15.52:0/2541498055 wait complete. 2024-05-27T07:27:06.354 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:27:06 smithi173 ceph-mon[37000]: pgmap v37: 129 pgs: 129 active+clean; 1.4 MiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 22 KiB/s wr, 23 op/s 2024-05-27T07:27:06.439 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:27:06 smithi105 ceph-mon[35977]: pgmap v37: 129 pgs: 129 active+clean; 1.4 MiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 22 KiB/s wr, 23 op/s 2024-05-27T07:27:06.461 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:27:06 smithi052 ceph-mon[31955]: pgmap v37: 129 pgs: 129 active+clean; 1.4 MiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 22 KiB/s wr, 23 op/s 2024-05-27T07:27:07.490 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-05-27T07:27:07.491 DEBUG:teuthology.orchestra.run.smithi052:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:38f9d900c83c63799fbdbe61acc9a11b0d3554a6 shell --fsid 73df2636-1bf8-11ef-bc9b-c7b262605968 -- ceph tell mds.1:0 scrub start / force,recursive 2024-05-27T07:27:07.773 INFO:teuthology.orchestra.run.smithi052.stderr:Inferring config /var/lib/ceph/73df2636-1bf8-11ef-bc9b-c7b262605968/mon.a/config 2024-05-27T07:27:08.165 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:27:08 smithi052 ceph-mon[31955]: pgmap v38: 129 pgs: 129 active+clean; 1.4 MiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 22 KiB/s wr, 23 op/s 2024-05-27T07:27:08.351 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:27:08 smithi173 ceph-mon[37000]: pgmap v38: 129 pgs: 129 active+clean; 1.4 MiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 22 KiB/s wr, 23 op/s 2024-05-27T07:27:08.439 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:27:08 smithi105 ceph-mon[35977]: pgmap v38: 129 pgs: 129 active+clean; 1.4 MiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 22 KiB/s wr, 23 op/s 2024-05-27T07:27:09.592 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:09.600+0000 7efdfd693700 1 -- 172.21.15.52:0/493937034 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7efdf8108ef0 msgr2=0x7efdf8109300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:27:09.592 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:09.600+0000 7efdfd693700 1 --2- 172.21.15.52:0/493937034 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7efdf8108ef0 0x7efdf8109300 secure :-1 s=READY pgs=508 cs=0 l=1 rev1=1 crypto rx=0x7efdec0099c0 tx=0x7efdec009cd0 comp rx=0 tx=0).stop 2024-05-27T07:27:09.592 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:09.601+0000 7efdfd693700 1 -- 172.21.15.52:0/493937034 shutdown_connections 2024-05-27T07:27:09.592 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:09.601+0000 7efdfd693700 1 --2- 172.21.15.52:0/493937034 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7efdf8109840 0x7efdf81111a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:27:09.592 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:09.601+0000 7efdfd693700 1 --2- 172.21.15.52:0/493937034 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7efdf81045c0 0x7efdf81049d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:27:09.592 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:09.601+0000 7efdfd693700 1 --2- 172.21.15.52:0/493937034 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7efdf8108ef0 0x7efdf8109300 unknown :-1 s=CLOSED pgs=508 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:27:09.592 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:09.601+0000 7efdfd693700 1 -- 172.21.15.52:0/493937034 >> 172.21.15.52:0/493937034 conn(0x7efdf80ffe90 msgr2=0x7efdf81022e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:27:09.592 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:09.601+0000 7efdfd693700 1 -- 172.21.15.52:0/493937034 shutdown_connections 2024-05-27T07:27:09.592 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:09.601+0000 7efdfd693700 1 -- 172.21.15.52:0/493937034 wait complete. 2024-05-27T07:27:09.594 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:09.601+0000 7efdfd693700 1 Processor -- start 2024-05-27T07:27:09.594 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:09.602+0000 7efdfd693700 1 -- start start 2024-05-27T07:27:09.594 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:09.602+0000 7efdfd693700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7efdf81045c0 0x7efdf819ae20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:27:09.594 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:09.602+0000 7efdfd693700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7efdf8108ef0 0x7efdf819b3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:27:09.595 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:09.602+0000 7efdfd693700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7efdf8109840 0x7efdf819f650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:27:09.595 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:09.602+0000 7efdfd693700 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x7efdf8114450 con 0x7efdf81045c0 2024-05-27T07:27:09.595 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:09.602+0000 7efdfd693700 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7efdf81142d0 con 0x7efdf8108ef0 2024-05-27T07:27:09.595 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:09.602+0000 7efdfd693700 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7efdf81145d0 con 0x7efdf8109840 2024-05-27T07:27:09.595 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:09.602+0000 7efdf6ffd700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7efdf81045c0 0x7efdf819ae20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:27:09.595 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:09.602+0000 7efdf6ffd700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7efdf81045c0 0x7efdf819ae20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.52:35946/0 (socket says 172.21.15.52:35946) 2024-05-27T07:27:09.595 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:09.602+0000 7efdf6ffd700 1 -- 172.21.15.52:0/2546909497 learned_addr learned my addr 172.21.15.52:0/2546909497 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-05-27T07:27:09.595 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:09.602+0000 7efdf67fc700 1 --2- 172.21.15.52:0/2546909497 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7efdf8108ef0 0x7efdf819b3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:27:09.595 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:09.602+0000 7efdf77fe700 1 --2- 172.21.15.52:0/2546909497 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7efdf8109840 0x7efdf819f650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:27:09.595 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:09.603+0000 7efdf6ffd700 1 -- 172.21.15.52:0/2546909497 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7efdf8109840 msgr2=0x7efdf819f650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:27:09.595 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:09.603+0000 7efdf6ffd700 1 --2- 172.21.15.52:0/2546909497 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7efdf8109840 0x7efdf819f650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:27:09.595 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:09.603+0000 7efdf6ffd700 1 -- 172.21.15.52:0/2546909497 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7efdf8108ef0 msgr2=0x7efdf819b3c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:27:09.595 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:09.603+0000 7efdf6ffd700 1 --2- 172.21.15.52:0/2546909497 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7efdf8108ef0 0x7efdf819b3c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:27:09.595 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:09.603+0000 7efdf6ffd700 1 -- 172.21.15.52:0/2546909497 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7efdf819fcf0 con 0x7efdf81045c0 2024-05-27T07:27:09.596 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:09.603+0000 7efdf77fe700 1 --2- 172.21.15.52:0/2546909497 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7efdf8109840 0x7efdf819f650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-27T07:27:09.596 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:09.603+0000 7efdf6ffd700 1 --2- 172.21.15.52:0/2546909497 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7efdf81045c0 0x7efdf819ae20 secure :-1 s=READY pgs=509 cs=0 l=1 rev1=1 crypto rx=0x7efde800d950 tx=0x7efde800dc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:27:09.596 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:09.603+0000 7efddffff700 1 -- 172.21.15.52:0/2546909497 <== mon.0 v2:172.21.15.52:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efde8016070 con 0x7efdf81045c0 2024-05-27T07:27:09.596 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:09.603+0000 7efdfd693700 1 -- 172.21.15.52:0/2546909497 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7efdf819ffe0 con 0x7efdf81045c0 2024-05-27T07:27:09.596 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:09.603+0000 7efddffff700 1 -- 172.21.15.52:0/2546909497 <== mon.0 v2:172.21.15.52:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7efde8009df0 con 0x7efdf81045c0 2024-05-27T07:27:09.596 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:09.603+0000 7efdfd693700 1 -- 172.21.15.52:0/2546909497 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7efdf81a7900 con 0x7efdf81045c0 2024-05-27T07:27:09.596 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:09.603+0000 7efddffff700 1 -- 172.21.15.52:0/2546909497 <== mon.0 v2:172.21.15.52:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efde800f3c0 con 0x7efdf81045c0 2024-05-27T07:27:09.597 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:09.606+0000 7efddffff700 1 -- 172.21.15.52:0/2546909497 <== mon.0 v2:172.21.15.52:3300/0 4 ==== mgrmap(e 25) ==== 149936+0+0 (secure 0 0 0) 0x7efde8019070 con 0x7efdf81045c0 2024-05-27T07:27:09.597 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:09.607+0000 7efddffff700 1 --2- 172.21.15.52:0/2546909497 >> [v2:172.21.15.52:6800/1822318384,v1:172.21.15.52:6801/1822318384] conn(0x7efde00b2ba0 0x7efde00b50b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:27:09.598 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:09.607+0000 7efdf67fc700 1 --2- 172.21.15.52:0/2546909497 >> [v2:172.21.15.52:6800/1822318384,v1:172.21.15.52:6801/1822318384] conn(0x7efde00b2ba0 0x7efde00b50b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:27:09.598 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:09.607+0000 7efddffff700 1 -- 172.21.15.52:0/2546909497 <== mon.0 v2:172.21.15.52:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9379+0+0 (secure 0 0 0) 0x7efde80e26e0 con 0x7efdf81045c0 2024-05-27T07:27:09.598 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:09.607+0000 7efe0c0a8b80 1 Processor -- start 2024-05-27T07:27:09.598 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:09.607+0000 7efe0c0a8b80 1 -- start start 2024-05-27T07:27:09.599 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:09.608+0000 7efdf67fc700 1 --2- 172.21.15.52:0/2546909497 >> [v2:172.21.15.52:6800/1822318384,v1:172.21.15.52:6801/1822318384] conn(0x7efde00b2ba0 0x7efde00b50b0 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7efdec006010 tx=0x7efdec000bc0 comp rx=0 tx=0).ready entity=mgr.15333 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:27:09.599 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:09.608+0000 7efe0c0a8b80 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55f1f5269ec0 0x55f1f53ed790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:27:09.599 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:09.608+0000 7efe0c0a8b80 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55f1f53edcd0 0x55f1f53e7ac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:27:09.599 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:09.608+0000 7efe0c0a8b80 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55f1f53e8000 0x55f1f53e8410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:27:09.599 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:09.608+0000 7efe0c0a8b80 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x55f1f5316d10 con 0x55f1f5269ec0 2024-05-27T07:27:09.600 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:09.608+0000 7efe0c0a8b80 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x55f1f53068b0 con 0x55f1f53e8000 2024-05-27T07:27:09.600 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:09.608+0000 7efe0c0a8b80 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x55f1f530a8f0 con 0x55f1f53edcd0 2024-05-27T07:27:09.600 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:09.608+0000 7efdf6ffd700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55f1f5269ec0 0x55f1f53ed790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:27:09.600 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:09.608+0000 7efdf6ffd700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55f1f5269ec0 0x55f1f53ed790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.52:35952/0 (socket says 172.21.15.52:35952) 2024-05-27T07:27:09.600 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:09.608+0000 7efdf6ffd700 1 -- 172.21.15.52:0/2980992836 learned_addr learned my addr 172.21.15.52:0/2980992836 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-05-27T07:27:09.600 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:09.608+0000 7efdf77fe700 1 --2- 172.21.15.52:0/2980992836 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55f1f53e8000 0x55f1f53e8410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:27:09.600 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:09.608+0000 7efdf6ffd700 1 -- 172.21.15.52:0/2980992836 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55f1f53edcd0 msgr2=0x55f1f53e7ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:27:09.600 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:09.608+0000 7efdf6ffd700 1 --2- 172.21.15.52:0/2980992836 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55f1f53edcd0 0x55f1f53e7ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:27:09.600 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:09.608+0000 7efdf6ffd700 1 -- 172.21.15.52:0/2980992836 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55f1f53e8000 msgr2=0x55f1f53e8410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:27:09.600 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:09.608+0000 7efdf6ffd700 1 --2- 172.21.15.52:0/2980992836 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55f1f53e8000 0x55f1f53e8410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:27:09.600 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:09.608+0000 7efdf6ffd700 1 -- 172.21.15.52:0/2980992836 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55f1f5256950 con 0x55f1f5269ec0 2024-05-27T07:27:09.600 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:09.608+0000 7efdf77fe700 1 --2- 172.21.15.52:0/2980992836 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55f1f53e8000 0x55f1f53e8410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-27T07:27:09.600 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:09.608+0000 7efdf6ffd700 1 --2- 172.21.15.52:0/2980992836 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55f1f5269ec0 0x55f1f53ed790 secure :-1 s=READY pgs=510 cs=0 l=1 rev1=1 crypto rx=0x7efde8000c00 tx=0x7efde80a9420 comp rx=0 tx=0).ready entity=mon.0 client_cookie=667134538a818bf6 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:27:09.601 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:09.609+0000 7efddd7fa700 1 -- 172.21.15.52:0/2980992836 <== mon.0 v2:172.21.15.52:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efde8020070 con 0x55f1f5269ec0 2024-05-27T07:27:09.601 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:09.609+0000 7efddd7fa700 1 -- 172.21.15.52:0/2980992836 <== mon.0 v2:172.21.15.52:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7efde80e2530 con 0x55f1f5269ec0 2024-05-27T07:27:09.601 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:09.609+0000 7efddd7fa700 1 -- 172.21.15.52:0/2980992836 <== mon.0 v2:172.21.15.52:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efde80a3b40 con 0x55f1f5269ec0 2024-05-27T07:27:09.603 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:09.610+0000 7efe0c0a8b80 1 -- 172.21.15.52:0/2980992836 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55f1f5269ec0 msgr2=0x55f1f53ed790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:27:09.603 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:09.610+0000 7efe0c0a8b80 1 --2- 172.21.15.52:0/2980992836 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55f1f5269ec0 0x55f1f53ed790 secure :-1 s=READY pgs=510 cs=0 l=1 rev1=1 crypto rx=0x7efde8000c00 tx=0x7efde80a9420 comp rx=0 tx=0).stop 2024-05-27T07:27:09.603 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:09.610+0000 7efe0c0a8b80 1 -- 172.21.15.52:0/2980992836 shutdown_connections 2024-05-27T07:27:09.603 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:09.610+0000 7efe0c0a8b80 1 --2- 172.21.15.52:0/2980992836 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55f1f53e8000 0x55f1f53e8410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:27:09.603 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:09.610+0000 7efe0c0a8b80 1 --2- 172.21.15.52:0/2980992836 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55f1f5269ec0 0x55f1f53ed790 unknown :-1 s=CLOSED pgs=510 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:27:09.603 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:09.610+0000 7efe0c0a8b80 1 --2- 172.21.15.52:0/2980992836 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55f1f53edcd0 0x55f1f53e7ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:27:09.603 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:09.610+0000 7efe0c0a8b80 1 -- 172.21.15.52:0/2980992836 >> 172.21.15.52:0/2980992836 conn(0x55f1f52c4b70 msgr2=0x55f1f53e26c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:27:09.603 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:09.610+0000 7efe0c0a8b80 1 -- 172.21.15.52:0/2980992836 shutdown_connections 2024-05-27T07:27:09.603 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:09.610+0000 7efe0c0a8b80 1 -- 172.21.15.52:0/2980992836 wait complete. 2024-05-27T07:27:09.603 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:09.610+0000 7efe0c0a8b80 1 -- start start 2024-05-27T07:27:09.603 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:09.611+0000 7efe0c0a8b80 1 Processor -- start 2024-05-27T07:27:09.604 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:09.611+0000 7efe0c0a8b80 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55f1f5269ec0 0x55f1f53f13c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:27:09.604 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:09.611+0000 7efe0c0a8b80 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55f1f53e8000 0x55f1f53f1900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:27:09.604 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:09.611+0000 7efe0c0a8b80 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55f1f53edcd0 0x55f1f53f1e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:27:09.604 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:09.611+0000 7efe0c0a8b80 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x55f1f520d020 con 0x55f1f5269ec0 2024-05-27T07:27:09.604 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:09.611+0000 7efe0c0a8b80 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x55f1f52a8840 con 0x55f1f53e8000 2024-05-27T07:27:09.604 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:09.611+0000 7efe0c0a8b80 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x55f1f53f47c0 con 0x55f1f53edcd0 2024-05-27T07:27:09.604 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:09.611+0000 7efe0c0a8b80 10 client.0 _ll_register_callbacks cb 0x55f1f52a19a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T07:27:09.604 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:09.611+0000 7efdf6ffd700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55f1f5269ec0 0x55f1f53f13c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:27:09.604 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:09.611+0000 7efdf6ffd700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55f1f5269ec0 0x55f1f53f13c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.52:35968/0 (socket says 172.21.15.52:35968) 2024-05-27T07:27:09.604 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:09.611+0000 7efdf6ffd700 1 -- 172.21.15.52:0/2137191410 learned_addr learned my addr 172.21.15.52:0/2137191410 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-05-27T07:27:09.604 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:09.611+0000 7efdf67fc700 1 --2- 172.21.15.52:0/2137191410 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55f1f53e8000 0x55f1f53f1900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:27:09.604 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:09.611+0000 7efdf77fe700 1 --2- 172.21.15.52:0/2137191410 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55f1f53edcd0 0x55f1f53f1e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:27:09.604 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:09.611+0000 7efdf6ffd700 1 -- 172.21.15.52:0/2137191410 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55f1f53edcd0 msgr2=0x55f1f53f1e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:27:09.605 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:09.611+0000 7efdf6ffd700 1 --2- 172.21.15.52:0/2137191410 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55f1f53edcd0 0x55f1f53f1e40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:27:09.605 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:09.612+0000 7efdf6ffd700 1 -- 172.21.15.52:0/2137191410 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55f1f53e8000 msgr2=0x55f1f53f1900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:27:09.605 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:09.612+0000 7efdf6ffd700 1 --2- 172.21.15.52:0/2137191410 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55f1f53e8000 0x55f1f53f1900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:27:09.605 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:09.612+0000 7efdf6ffd700 1 -- 172.21.15.52:0/2137191410 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55f1f52a5710 con 0x55f1f5269ec0 2024-05-27T07:27:09.605 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:09.612+0000 7efdf77fe700 1 --2- 172.21.15.52:0/2137191410 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55f1f53edcd0 0x55f1f53f1e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-27T07:27:09.605 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:09.612+0000 7efdf6ffd700 1 --2- 172.21.15.52:0/2137191410 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55f1f5269ec0 0x55f1f53f13c0 secure :-1 s=READY pgs=511 cs=0 l=1 rev1=1 crypto rx=0x7efde80a9a80 tx=0x7efde80096c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=27745f08f445bf5f server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:27:09.605 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:09.612+0000 7efddaff5700 10 client.0 ms_handle_connect on v2:172.21.15.52:3300/0 2024-05-27T07:27:09.605 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:09.612+0000 7efddaff5700 1 -- 172.21.15.52:0/2137191410 <== mon.0 v2:172.21.15.52:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efde80c2070 con 0x55f1f5269ec0 2024-05-27T07:27:09.605 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:09.612+0000 7efddaff5700 1 -- 172.21.15.52:0/2137191410 <== mon.0 v2:172.21.15.52:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 9379+0+0 (secure 0 0 0) 0x7efde8009b50 con 0x55f1f5269ec0 2024-05-27T07:27:09.605 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:09.612+0000 7efe0c0a8b80 1 -- 172.21.15.52:0/2137191410 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7efde80e2530 con 0x55f1f5269ec0 2024-05-27T07:27:09.605 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:09.612+0000 7efddaff5700 1 -- 172.21.15.52:0/2137191410 <== mon.0 v2:172.21.15.52:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7efde8020070 con 0x55f1f5269ec0 2024-05-27T07:27:09.605 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:09.612+0000 7efe0c0a8b80 10 client.15747 fetch_fsmap learned FSMap version 30 2024-05-27T07:27:09.605 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:09.612+0000 7efe0c0a8b80 1 -- 172.21.15.52:0/2137191410 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x55f1f53ff1a0 con 0x55f1f5269ec0 2024-05-27T07:27:09.605 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:09.612+0000 7efddaff5700 1 -- 172.21.15.52:0/2137191410 <== mon.0 v2:172.21.15.52:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7efde80e2530 con 0x55f1f5269ec0 2024-05-27T07:27:09.607 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:09.613+0000 7efddaff5700 1 -- 172.21.15.52:0/2137191410 <== mon.0 v2:172.21.15.52:3300/0 5 ==== fsmap(e 30) ==== 8362+0+0 (secure 0 0 0) 0x7efde80a87a0 con 0x55f1f5269ec0 2024-05-27T07:27:09.607 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:09.613+0000 7efe0c0a8b80 10 client.15747 fetch_fsmap finished waiting for FSMap version 30 2024-05-27T07:27:09.607 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:09.613+0000 7efe0c0a8b80 10 client.15747 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-05-27T07:27:09.608 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:09.613+0000 7efe0c0a8b80 20 client.15747 populate_metadata read hostname 'smithi052' 2024-05-27T07:27:09.608 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:09.613+0000 7efe0c0a8b80 1 --2- 172.21.15.52:0/2137191410 >> [v2:172.21.15.105:6838/500348151,v1:172.21.15.105:6839/500348151] conn(0x55f1f5401670 0x55f1f5403a90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:27:09.608 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:09.613+0000 7efe0c0a8b80 4 client.15747 mds_command: new command op to 24527 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-27T07:27:09.608 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:09.613+0000 7efe0c0a8b80 1 -- 172.21.15.52:0/2137191410 --> [v2:172.21.15.105:6838/500348151,v1:172.21.15.105:6839/500348151] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55f1f53b3a70 con 0x55f1f5401670 2024-05-27T07:27:09.608 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:09.613+0000 7efdf67fc700 1 --2- 172.21.15.52:0/2137191410 >> [v2:172.21.15.105:6838/500348151,v1:172.21.15.105:6839/500348151] conn(0x55f1f5401670 0x55f1f5403a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:27:09.608 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:09.613+0000 7efdf67fc700 1 --2- 172.21.15.52:0/2137191410 >> [v2:172.21.15.105:6838/500348151,v1:172.21.15.105:6839/500348151] conn(0x55f1f5401670 0x55f1f5403a90 crc :-1 s=READY pgs=59 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=2e751a675a5d4b5f server_cookie=6f168a1da532bf38 in_seq=0 out_seq=0 2024-05-27T07:27:09.608 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:09.613+0000 7efddaff5700 10 client.15747 ms_handle_connect on v2:172.21.15.105:6838/500348151 2024-05-27T07:27:09.608 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:09.614+0000 7efddaff5700 1 -- 172.21.15.52:0/2137191410 <== mds.0 v2:172.21.15.105:6838/500348151 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24041 (crc 0 0 0) 0x55f1f53b3a70 con 0x55f1f5401670 2024-05-27T07:27:09.608 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:09.614+0000 7efddaff5700 10 client.15747 handle_command_reply: tid=0 2024-05-27T07:27:09.608 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:09.614+0000 7efe0c0a8b80 1 client.15747 shutdown 2024-05-27T07:27:09.608 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:09.614+0000 7efe0c0a8b80 1 -- 172.21.15.52:0/2137191410 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55f1f5269ec0 msgr2=0x55f1f53f13c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:27:09.608 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:09.614+0000 7efe0c0a8b80 1 --2- 172.21.15.52:0/2137191410 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55f1f5269ec0 0x55f1f53f13c0 secure :-1 s=READY pgs=511 cs=0 l=1 rev1=1 crypto rx=0x7efde80a9a80 tx=0x7efde80096c0 comp rx=0 tx=0).stop 2024-05-27T07:27:09.608 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:09.614+0000 7efe0c0a8b80 1 -- 172.21.15.52:0/2137191410 shutdown_connections 2024-05-27T07:27:09.609 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:09.614+0000 7efe0c0a8b80 1 --2- 172.21.15.52:0/2137191410 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55f1f53edcd0 0x55f1f53f1e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:27:09.609 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:09.614+0000 7efe0c0a8b80 1 --2- 172.21.15.52:0/2137191410 >> [v2:172.21.15.105:6838/500348151,v1:172.21.15.105:6839/500348151] conn(0x55f1f5401670 0x55f1f5403a90 crc :-1 s=READY pgs=59 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:27:09.609 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:09.614+0000 7efe0c0a8b80 1 --2- 172.21.15.52:0/2137191410 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55f1f5269ec0 0x55f1f53f13c0 unknown :-1 s=CLOSED pgs=511 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:27:09.609 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:09.614+0000 7efe0c0a8b80 1 --2- 172.21.15.52:0/2137191410 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55f1f53e8000 0x55f1f53f1900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:27:09.609 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:09.614+0000 7efddaff5700 0 client.15747 ms_handle_reset on v2:172.21.15.105:6838/500348151 2024-05-27T07:27:09.609 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:09.614+0000 7efe0c0a8b80 1 -- 172.21.15.52:0/2137191410 >> 172.21.15.52:0/2137191410 conn(0x55f1f52c4b70 msgr2=0x55f1f53e6ab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:27:09.609 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:09.615+0000 7efe0c0a8b80 1 -- 172.21.15.52:0/2137191410 shutdown_connections 2024-05-27T07:27:09.609 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:09.615+0000 7efe0c0a8b80 1 -- 172.21.15.52:0/2137191410 wait complete. 2024-05-27T07:27:09.609 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:09.615+0000 7efe0c0a8b80 20 client.15747 trim_cache size 0 max 16384 2024-05-27T07:27:09.627 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:09.636+0000 7efe0c0a8b80 1 Processor -- start 2024-05-27T07:27:09.627 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:09.636+0000 7efe0c0a8b80 1 -- start start 2024-05-27T07:27:09.628 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:09.636+0000 7efe0c0a8b80 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55f1f5269ec0 0x55f1f53e7ac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:27:09.628 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:09.636+0000 7efe0c0a8b80 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55f1f53e8000 0x55f1f53fdac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:27:09.628 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:09.636+0000 7efe0c0a8b80 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55f1f53edcd0 0x55f1f53eb7a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:27:09.628 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:09.636+0000 7efe0c0a8b80 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x7efde400ec20 con 0x55f1f53e8000 2024-05-27T07:27:09.628 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:09.636+0000 7efe0c0a8b80 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7efde4010760 con 0x55f1f5269ec0 2024-05-27T07:27:09.628 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:09.636+0000 7efe0c0a8b80 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7efde400f2f0 con 0x55f1f53edcd0 2024-05-27T07:27:09.628 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:09.637+0000 7efdf67fc700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55f1f53e8000 0x55f1f53fdac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:27:09.628 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:09.637+0000 7efdf67fc700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55f1f53e8000 0x55f1f53fdac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.52:35972/0 (socket says 172.21.15.52:35972) 2024-05-27T07:27:09.628 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:09.637+0000 7efdf67fc700 1 -- 172.21.15.52:0/884495745 learned_addr learned my addr 172.21.15.52:0/884495745 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-05-27T07:27:09.628 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:09.637+0000 7efdf6ffd700 1 --2- 172.21.15.52:0/884495745 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55f1f5269ec0 0x55f1f53e7ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:27:09.628 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:09.637+0000 7efdf77fe700 1 --2- 172.21.15.52:0/884495745 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55f1f53edcd0 0x55f1f53eb7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:27:09.628 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:09.637+0000 7efdf67fc700 1 -- 172.21.15.52:0/884495745 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55f1f53edcd0 msgr2=0x55f1f53eb7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:27:09.628 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:09.637+0000 7efdf67fc700 1 --2- 172.21.15.52:0/884495745 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55f1f53edcd0 0x55f1f53eb7a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:27:09.629 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:09.637+0000 7efdf67fc700 1 -- 172.21.15.52:0/884495745 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55f1f5269ec0 msgr2=0x55f1f53e7ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:27:09.629 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:09.637+0000 7efdf67fc700 1 --2- 172.21.15.52:0/884495745 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55f1f5269ec0 0x55f1f53e7ac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:27:09.629 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:09.637+0000 7efdf67fc700 1 -- 172.21.15.52:0/884495745 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55f1f53f47f0 con 0x55f1f53e8000 2024-05-27T07:27:09.629 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:09.637+0000 7efdf6ffd700 1 --2- 172.21.15.52:0/884495745 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55f1f5269ec0 0x55f1f53e7ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-27T07:27:09.629 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:09.637+0000 7efdf67fc700 1 --2- 172.21.15.52:0/884495745 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55f1f53e8000 0x55f1f53fdac0 secure :-1 s=READY pgs=512 cs=0 l=1 rev1=1 crypto rx=0x7efdf806c160 tx=0x7efdec01e000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=90222f0aa7f657ec server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:27:09.631 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:09.637+0000 7efddcff9700 1 -- 172.21.15.52:0/884495745 <== mon.0 v2:172.21.15.52:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efdec00b990 con 0x55f1f53e8000 2024-05-27T07:27:09.631 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:09.637+0000 7efddcff9700 1 -- 172.21.15.52:0/884495745 <== mon.0 v2:172.21.15.52:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7efdec003e50 con 0x55f1f53e8000 2024-05-27T07:27:09.631 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:09.637+0000 7efddcff9700 1 -- 172.21.15.52:0/884495745 <== mon.0 v2:172.21.15.52:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efdec031420 con 0x55f1f53e8000 2024-05-27T07:27:09.631 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:09.638+0000 7efe0c0a8b80 1 -- 172.21.15.52:0/884495745 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55f1f53e8000 msgr2=0x55f1f53fdac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:27:09.631 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:09.638+0000 7efe0c0a8b80 1 --2- 172.21.15.52:0/884495745 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55f1f53e8000 0x55f1f53fdac0 secure :-1 s=READY pgs=512 cs=0 l=1 rev1=1 crypto rx=0x7efdf806c160 tx=0x7efdec01e000 comp rx=0 tx=0).stop 2024-05-27T07:27:09.631 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:09.638+0000 7efe0c0a8b80 1 -- 172.21.15.52:0/884495745 shutdown_connections 2024-05-27T07:27:09.631 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:09.638+0000 7efe0c0a8b80 1 --2- 172.21.15.52:0/884495745 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55f1f53edcd0 0x55f1f53eb7a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:27:09.631 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:09.638+0000 7efe0c0a8b80 1 --2- 172.21.15.52:0/884495745 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55f1f5269ec0 0x55f1f53e7ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:27:09.632 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:09.638+0000 7efe0c0a8b80 1 --2- 172.21.15.52:0/884495745 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55f1f53e8000 0x55f1f53fdac0 unknown :-1 s=CLOSED pgs=512 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:27:09.632 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:09.638+0000 7efe0c0a8b80 1 -- 172.21.15.52:0/884495745 >> 172.21.15.52:0/884495745 conn(0x55f1f52c4b70 msgr2=0x55f1f53eb090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:27:09.632 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:09.638+0000 7efe0c0a8b80 1 -- 172.21.15.52:0/884495745 shutdown_connections 2024-05-27T07:27:09.632 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:09.638+0000 7efe0c0a8b80 1 -- 172.21.15.52:0/884495745 wait complete. 2024-05-27T07:27:09.632 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:09.639+0000 7efe0c0a8b80 1 -- start start 2024-05-27T07:27:09.632 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:09.639+0000 7efe0c0a8b80 1 Processor -- start 2024-05-27T07:27:09.632 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:09.639+0000 7efe0c0a8b80 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55f1f5269ec0 0x55f1f5406a90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:27:09.632 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:09.639+0000 7efe0c0a8b80 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55f1f53e8000 0x55f1f5404ec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:27:09.632 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:09.639+0000 7efe0c0a8b80 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55f1f53edcd0 0x55f1f5405400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:27:09.632 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:09.639+0000 7efe0c0a8b80 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x7efde4010090 con 0x55f1f53e8000 2024-05-27T07:27:09.632 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:09.639+0000 7efe0c0a8b80 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7efde400f9c0 con 0x55f1f5269ec0 2024-05-27T07:27:09.632 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:09.639+0000 7efe0c0a8b80 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7efde400cbc0 con 0x55f1f53edcd0 2024-05-27T07:27:09.632 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:09.639+0000 7efe0c0a8b80 10 client.0 _ll_register_callbacks cb 0x55f1f52a19a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T07:27:09.632 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:09.639+0000 7efdf67fc700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55f1f53e8000 0x55f1f5404ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:27:09.633 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:09.639+0000 7efdf67fc700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55f1f53e8000 0x55f1f5404ec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.52:35980/0 (socket says 172.21.15.52:35980) 2024-05-27T07:27:09.633 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:09.639+0000 7efdf67fc700 1 -- 172.21.15.52:0/1255426012 learned_addr learned my addr 172.21.15.52:0/1255426012 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-05-27T07:27:09.633 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:09.640+0000 7efdf6ffd700 1 --2- 172.21.15.52:0/1255426012 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55f1f5269ec0 0x55f1f5406a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:27:09.633 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:09.640+0000 7efdf77fe700 1 --2- 172.21.15.52:0/1255426012 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55f1f53edcd0 0x55f1f5405400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:27:09.633 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:09.640+0000 7efdf67fc700 1 -- 172.21.15.52:0/1255426012 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55f1f53edcd0 msgr2=0x55f1f5405400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:27:09.633 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:09.640+0000 7efdf67fc700 1 --2- 172.21.15.52:0/1255426012 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55f1f53edcd0 0x55f1f5405400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:27:09.633 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:09.640+0000 7efdf67fc700 1 -- 172.21.15.52:0/1255426012 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55f1f5269ec0 msgr2=0x55f1f5406a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:27:09.633 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:09.640+0000 7efdf67fc700 1 --2- 172.21.15.52:0/1255426012 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55f1f5269ec0 0x55f1f5406a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:27:09.633 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:09.640+0000 7efdf67fc700 1 -- 172.21.15.52:0/1255426012 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55f1f5405940 con 0x55f1f53e8000 2024-05-27T07:27:09.633 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:09.640+0000 7efdf6ffd700 1 --2- 172.21.15.52:0/1255426012 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55f1f5269ec0 0x55f1f5406a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-27T07:27:09.633 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:09.640+0000 7efdf67fc700 1 --2- 172.21.15.52:0/1255426012 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55f1f53e8000 0x55f1f5404ec0 secure :-1 s=READY pgs=513 cs=0 l=1 rev1=1 crypto rx=0x7efdec00fb90 tx=0x7efdec010d30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=62f10de7f3cd2926 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:27:09.633 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:09.640+0000 7efddaff5700 10 client.0 ms_handle_connect on v2:172.21.15.52:3300/0 2024-05-27T07:27:09.633 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:09.640+0000 7efddaff5700 1 -- 172.21.15.52:0/1255426012 <== mon.0 v2:172.21.15.52:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efdec043070 con 0x55f1f53e8000 2024-05-27T07:27:09.634 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:09.640+0000 7efddaff5700 1 -- 172.21.15.52:0/1255426012 <== mon.0 v2:172.21.15.52:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 9379+0+0 (secure 0 0 0) 0x7efdec00b6b0 con 0x55f1f53e8000 2024-05-27T07:27:09.634 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:09.640+0000 7efe0c0a8b80 1 -- 172.21.15.52:0/1255426012 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7efdec003e50 con 0x55f1f53e8000 2024-05-27T07:27:09.634 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:09.640+0000 7efddaff5700 1 -- 172.21.15.52:0/1255426012 <== mon.0 v2:172.21.15.52:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7efdec00f930 con 0x55f1f53e8000 2024-05-27T07:27:09.634 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:09.641+0000 7efe0c0a8b80 10 client.15753 fetch_fsmap learned FSMap version 30 2024-05-27T07:27:09.634 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:09.641+0000 7efe0c0a8b80 1 -- 172.21.15.52:0/1255426012 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x55f1f546cb70 con 0x55f1f53e8000 2024-05-27T07:27:09.636 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:09.641+0000 7efddaff5700 1 -- 172.21.15.52:0/1255426012 <== mon.0 v2:172.21.15.52:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7efdec003e50 con 0x55f1f53e8000 2024-05-27T07:27:09.636 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:09.641+0000 7efddaff5700 1 -- 172.21.15.52:0/1255426012 <== mon.0 v2:172.21.15.52:3300/0 5 ==== fsmap(e 30) ==== 8362+0+0 (secure 0 0 0) 0x7efdec03a070 con 0x55f1f53e8000 2024-05-27T07:27:09.636 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:09.641+0000 7efe0c0a8b80 10 client.15753 fetch_fsmap finished waiting for FSMap version 30 2024-05-27T07:27:09.636 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:09.641+0000 7efe0c0a8b80 10 client.15753 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-05-27T07:27:09.636 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:09.641+0000 7efe0c0a8b80 20 client.15753 populate_metadata read hostname 'smithi052' 2024-05-27T07:27:09.636 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:09.641+0000 7efe0c0a8b80 1 --2- 172.21.15.52:0/1255426012 >> [v2:172.21.15.105:6838/500348151,v1:172.21.15.105:6839/500348151] conn(0x55f1f546d220 0x55f1f546f640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:27:09.636 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:09.641+0000 7efe0c0a8b80 4 client.15753 mds_command: new command op to 24527 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-05-27T07:27:09.636 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:09.641+0000 7efe0c0a8b80 1 -- 172.21.15.52:0/1255426012 --> [v2:172.21.15.105:6838/500348151,v1:172.21.15.105:6839/500348151] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) -- 0x55f1f5318c60 con 0x55f1f546d220 2024-05-27T07:27:09.636 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:09.641+0000 7efdf6ffd700 1 --2- 172.21.15.52:0/1255426012 >> [v2:172.21.15.105:6838/500348151,v1:172.21.15.105:6839/500348151] conn(0x55f1f546d220 0x55f1f546f640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:27:09.637 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:09.642+0000 7efddaff5700 10 client.15753 ms_handle_connect on v2:172.21.15.105:6838/500348151 2024-05-27T07:27:09.637 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:09.642+0000 7efdf6ffd700 1 --2- 172.21.15.52:0/1255426012 >> [v2:172.21.15.105:6838/500348151,v1:172.21.15.105:6839/500348151] conn(0x55f1f546d220 0x55f1f546f640 crc :-1 s=READY pgs=60 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=af752c63ad98806d server_cookie=b17e237fdaa38641 in_seq=0 out_seq=0 2024-05-27T07:27:09.637 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:09.643+0000 7efddaff5700 1 -- 172.21.15.52:0/1255426012 <== mds.0 v2:172.21.15.105:6838/500348151 1 ==== command_reply(tid 0: 0 ) ==== 8+0+110 (crc 0 0 0) 0x55f1f5318c60 con 0x55f1f546d220 2024-05-27T07:27:09.637 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:09.643+0000 7efddaff5700 10 client.15753 handle_command_reply: tid=0 2024-05-27T07:27:09.637 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:09.643+0000 7efe0c0a8b80 1 client.15753 shutdown 2024-05-27T07:27:09.637 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:09.643+0000 7efe0c0a8b80 1 -- 172.21.15.52:0/1255426012 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55f1f53e8000 msgr2=0x55f1f5404ec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:27:09.637 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:09.643+0000 7efe0c0a8b80 1 --2- 172.21.15.52:0/1255426012 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55f1f53e8000 0x55f1f5404ec0 secure :-1 s=READY pgs=513 cs=0 l=1 rev1=1 crypto rx=0x7efdec00fb90 tx=0x7efdec010d30 comp rx=0 tx=0).stop 2024-05-27T07:27:09.637 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:09.643+0000 7efe0c0a8b80 1 -- 172.21.15.52:0/1255426012 shutdown_connections 2024-05-27T07:27:09.637 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:09.643+0000 7efe0c0a8b80 1 --2- 172.21.15.52:0/1255426012 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55f1f53edcd0 0x55f1f5405400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:27:09.637 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:09.643+0000 7efe0c0a8b80 1 --2- 172.21.15.52:0/1255426012 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55f1f5269ec0 0x55f1f5406a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:27:09.637 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:09.643+0000 7efe0c0a8b80 1 --2- 172.21.15.52:0/1255426012 >> [v2:172.21.15.105:6838/500348151,v1:172.21.15.105:6839/500348151] conn(0x55f1f546d220 0x55f1f546f640 crc :-1 s=READY pgs=60 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:27:09.638 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:09.643+0000 7efe0c0a8b80 1 --2- 172.21.15.52:0/1255426012 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55f1f53e8000 0x55f1f5404ec0 unknown :-1 s=CLOSED pgs=513 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:27:09.638 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:09.643+0000 7efe0c0a8b80 1 -- 172.21.15.52:0/1255426012 >> 172.21.15.52:0/1255426012 conn(0x55f1f52c4b70 msgr2=0x55f1f53f1380 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:27:09.638 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:09.644+0000 7efddaff5700 0 client.15753 ms_handle_reset on v2:172.21.15.105:6838/500348151 2024-05-27T07:27:09.638 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:09.644+0000 7efe0c0a8b80 1 -- 172.21.15.52:0/1255426012 shutdown_connections 2024-05-27T07:27:09.638 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:09.644+0000 7efe0c0a8b80 1 -- 172.21.15.52:0/1255426012 wait complete. 2024-05-27T07:27:09.638 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:09.644+0000 7efe0c0a8b80 20 client.15753 trim_cache size 0 max 16384 2024-05-27T07:27:09.638 INFO:teuthology.orchestra.run.smithi052.stdout:{ 2024-05-27T07:27:09.638 INFO:teuthology.orchestra.run.smithi052.stdout: "return_code": 0, 2024-05-27T07:27:09.638 INFO:teuthology.orchestra.run.smithi052.stdout: "scrub_tag": "489b8864-d47f-4297-9944-e60a78bbba00", 2024-05-27T07:27:09.639 INFO:teuthology.orchestra.run.smithi052.stdout: "mode": "asynchronous" 2024-05-27T07:27:09.639 INFO:teuthology.orchestra.run.smithi052.stdout:} 2024-05-27T07:27:09.639 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:09.644+0000 7efdfd693700 1 -- 172.21.15.52:0/2546909497 >> [v2:172.21.15.52:6800/1822318384,v1:172.21.15.52:6801/1822318384] conn(0x7efde00b2ba0 msgr2=0x7efde00b50b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:27:09.639 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:09.644+0000 7efdfd693700 1 --2- 172.21.15.52:0/2546909497 >> [v2:172.21.15.52:6800/1822318384,v1:172.21.15.52:6801/1822318384] conn(0x7efde00b2ba0 0x7efde00b50b0 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7efdec006010 tx=0x7efdec000bc0 comp rx=0 tx=0).stop 2024-05-27T07:27:09.639 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:09.644+0000 7efdfd693700 1 -- 172.21.15.52:0/2546909497 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7efdf81045c0 msgr2=0x7efdf819ae20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:27:09.639 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:09.644+0000 7efdfd693700 1 --2- 172.21.15.52:0/2546909497 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7efdf81045c0 0x7efdf819ae20 secure :-1 s=READY pgs=509 cs=0 l=1 rev1=1 crypto rx=0x7efde800d950 tx=0x7efde800dc60 comp rx=0 tx=0).stop 2024-05-27T07:27:09.639 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:09.644+0000 7efdfd693700 1 -- 172.21.15.52:0/2546909497 shutdown_connections 2024-05-27T07:27:09.640 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:09.644+0000 7efdfd693700 1 --2- 172.21.15.52:0/2546909497 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7efdf8109840 0x7efdf819f650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:27:09.640 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:09.644+0000 7efdfd693700 1 --2- 172.21.15.52:0/2546909497 >> [v2:172.21.15.52:6800/1822318384,v1:172.21.15.52:6801/1822318384] conn(0x7efde00b2ba0 0x7efde00b50b0 unknown :-1 s=CLOSED pgs=68 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:27:09.640 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:09.644+0000 7efdfd693700 1 --2- 172.21.15.52:0/2546909497 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7efdf81045c0 0x7efdf819ae20 unknown :-1 s=CLOSED pgs=509 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:27:09.640 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:09.644+0000 7efdfd693700 1 --2- 172.21.15.52:0/2546909497 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7efdf8108ef0 0x7efdf819b3c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:27:09.640 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:09.644+0000 7efdfd693700 1 -- 172.21.15.52:0/2546909497 >> 172.21.15.52:0/2546909497 conn(0x7efdf80ffe90 msgr2=0x7efdf810a7a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:27:09.640 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:09.645+0000 7efdfd693700 1 -- 172.21.15.52:0/2546909497 shutdown_connections 2024-05-27T07:27:09.640 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:09.645+0000 7efdfd693700 1 -- 172.21.15.52:0/2546909497 wait complete. 2024-05-27T07:27:10.351 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:27:10 smithi173 ceph-mon[37000]: pgmap v39: 129 pgs: 129 active+clean; 1.4 MiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 23 KiB/s wr, 28 op/s 2024-05-27T07:27:10.391 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:27:10 smithi052 ceph-mon[31955]: pgmap v39: 129 pgs: 129 active+clean; 1.4 MiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 23 KiB/s wr, 28 op/s 2024-05-27T07:27:10.427 DEBUG:teuthology.orchestra.run.smithi052:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:38f9d900c83c63799fbdbe61acc9a11b0d3554a6 shell --fsid 73df2636-1bf8-11ef-bc9b-c7b262605968 -- ceph tell mds.1:0 scrub status 2024-05-27T07:27:10.439 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:27:10 smithi105 ceph-mon[35977]: pgmap v39: 129 pgs: 129 active+clean; 1.4 MiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 23 KiB/s wr, 28 op/s 2024-05-27T07:27:10.685 INFO:teuthology.orchestra.run.smithi052.stderr:Inferring config /var/lib/ceph/73df2636-1bf8-11ef-bc9b-c7b262605968/mon.a/config 2024-05-27T07:27:11.091 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:27:11 smithi052 ceph-mon[31955]: scrub queued for path: / 2024-05-27T07:27:11.091 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:27:11 smithi052 ceph-mon[31955]: scrub summary: idle+waiting paths [/] 2024-05-27T07:27:11.091 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:27:11 smithi052 ceph-mon[31955]: scrub summary: active paths [/] 2024-05-27T07:27:11.091 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:27:11 smithi052 ceph-mon[31955]: scrub summary: idle+waiting paths [/] 2024-05-27T07:27:11.352 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:27:11 smithi173 ceph-mon[37000]: scrub queued for path: / 2024-05-27T07:27:11.352 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:27:11 smithi173 ceph-mon[37000]: scrub summary: idle+waiting paths [/] 2024-05-27T07:27:11.352 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:27:11 smithi173 ceph-mon[37000]: scrub summary: active paths [/] 2024-05-27T07:27:11.352 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:27:11 smithi173 ceph-mon[37000]: scrub summary: idle+waiting paths [/] 2024-05-27T07:27:11.439 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:27:11 smithi105 ceph-mon[35977]: scrub queued for path: / 2024-05-27T07:27:11.440 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:27:11 smithi105 ceph-mon[35977]: scrub summary: idle+waiting paths [/] 2024-05-27T07:27:11.440 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:27:11 smithi105 ceph-mon[35977]: scrub summary: active paths [/] 2024-05-27T07:27:11.440 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:27:11 smithi105 ceph-mon[35977]: scrub summary: idle+waiting paths [/] 2024-05-27T07:27:12.229 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:27:12 smithi052 ceph-mon[31955]: pgmap v40: 129 pgs: 129 active+clean; 1.4 MiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 23 KiB/s wr, 24 op/s 2024-05-27T07:27:12.229 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:27:12 smithi052 ceph-mon[31955]: scrub summary: idle 2024-05-27T07:27:12.229 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:27:12 smithi052 ceph-mon[31955]: from='mgr.15333 172.21.15.52:0/416389082' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-27T07:27:12.351 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:27:12 smithi173 ceph-mon[37000]: pgmap v40: 129 pgs: 129 active+clean; 1.4 MiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 23 KiB/s wr, 24 op/s 2024-05-27T07:27:12.352 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:27:12 smithi173 ceph-mon[37000]: scrub summary: idle 2024-05-27T07:27:12.352 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:27:12 smithi173 ceph-mon[37000]: from='mgr.15333 172.21.15.52:0/416389082' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-27T07:27:12.439 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:27:12 smithi105 ceph-mon[35977]: pgmap v40: 129 pgs: 129 active+clean; 1.4 MiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 23 KiB/s wr, 24 op/s 2024-05-27T07:27:12.439 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:27:12 smithi105 ceph-mon[35977]: scrub summary: idle 2024-05-27T07:27:12.440 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:27:12 smithi105 ceph-mon[35977]: from='mgr.15333 172.21.15.52:0/416389082' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-27T07:27:12.654 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:12.663+0000 7f0f33e59700 1 -- 172.21.15.52:0/56823761 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f0f2c107de0 msgr2=0x7f0f2c1081f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:27:12.654 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:12.663+0000 7f0f33e59700 1 --2- 172.21.15.52:0/56823761 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f0f2c107de0 0x7f0f2c1081f0 secure :-1 s=READY pgs=514 cs=0 l=1 rev1=1 crypto rx=0x7f0f1c0099c0 tx=0x7f0f1c009cd0 comp rx=0 tx=0).stop 2024-05-27T07:27:12.654 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:12.663+0000 7f0f33e59700 1 -- 172.21.15.52:0/56823761 shutdown_connections 2024-05-27T07:27:12.655 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:12.663+0000 7f0f33e59700 1 --2- 172.21.15.52:0/56823761 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f0f2c108730 0x7f0f2c110090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:27:12.655 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:12.663+0000 7f0f33e59700 1 --2- 172.21.15.52:0/56823761 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f0f2c1034b0 0x7f0f2c1038c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:27:12.655 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:12.663+0000 7f0f33e59700 1 --2- 172.21.15.52:0/56823761 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f0f2c107de0 0x7f0f2c1081f0 unknown :-1 s=CLOSED pgs=514 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:27:12.655 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:12.663+0000 7f0f33e59700 1 -- 172.21.15.52:0/56823761 >> 172.21.15.52:0/56823761 conn(0x7f0f2c0fedb0 msgr2=0x7f0f2c1011d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:27:12.655 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:12.664+0000 7f0f33e59700 1 -- 172.21.15.52:0/56823761 shutdown_connections 2024-05-27T07:27:12.655 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:12.664+0000 7f0f33e59700 1 -- 172.21.15.52:0/56823761 wait complete. 2024-05-27T07:27:12.656 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:12.665+0000 7f0f33e59700 1 Processor -- start 2024-05-27T07:27:12.656 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:12.665+0000 7f0f33e59700 1 -- start start 2024-05-27T07:27:12.656 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:12.665+0000 7f0f33e59700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f0f2c1034b0 0x7f0f2c0ff290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:27:12.657 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:12.665+0000 7f0f33e59700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f0f2c107de0 0x7f0f2c0ff7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:27:12.657 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:12.665+0000 7f0f33e59700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f0f2c108730 0x7f0f2c0ffd10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:27:12.657 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:12.665+0000 7f0f33e59700 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x7f0f2c113340 con 0x7f0f2c108730 2024-05-27T07:27:12.657 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:12.665+0000 7f0f33e59700 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f0f2c1131c0 con 0x7f0f2c1034b0 2024-05-27T07:27:12.657 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:12.665+0000 7f0f33e59700 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f0f2c1134c0 con 0x7f0f2c107de0 2024-05-27T07:27:12.657 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:12.666+0000 7f0f323f6700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f0f2c108730 0x7f0f2c0ffd10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:27:12.657 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:12.666+0000 7f0f31bf5700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f0f2c1034b0 0x7f0f2c0ff290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:27:12.657 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:12.666+0000 7f0f313f4700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f0f2c107de0 0x7f0f2c0ff7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:27:12.657 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:12.666+0000 7f0f323f6700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f0f2c108730 0x7f0f2c0ffd10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.52:51628/0 (socket says 172.21.15.52:51628) 2024-05-27T07:27:12.657 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:12.666+0000 7f0f31bf5700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f0f2c1034b0 0x7f0f2c0ff290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.52:35860/0 (socket says 172.21.15.52:35860) 2024-05-27T07:27:12.657 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:12.666+0000 7f0f323f6700 1 -- 172.21.15.52:0/1514204960 learned_addr learned my addr 172.21.15.52:0/1514204960 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-05-27T07:27:12.658 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:12.666+0000 7f0f323f6700 1 -- 172.21.15.52:0/1514204960 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f0f2c107de0 msgr2=0x7f0f2c0ff7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:27:12.658 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:12.666+0000 7f0f323f6700 1 --2- 172.21.15.52:0/1514204960 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f0f2c107de0 0x7f0f2c0ff7d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:27:12.658 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:12.666+0000 7f0f323f6700 1 -- 172.21.15.52:0/1514204960 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f0f2c1034b0 msgr2=0x7f0f2c0ff290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:27:12.659 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:12.666+0000 7f0f323f6700 1 --2- 172.21.15.52:0/1514204960 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f0f2c1034b0 0x7f0f2c0ff290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:27:12.659 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:12.666+0000 7f0f323f6700 1 -- 172.21.15.52:0/1514204960 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0f2c1a9180 con 0x7f0f2c108730 2024-05-27T07:27:12.659 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:12.666+0000 7f0f323f6700 1 --2- 172.21.15.52:0/1514204960 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f0f2c108730 0x7f0f2c0ffd10 secure :-1 s=READY pgs=515 cs=0 l=1 rev1=1 crypto rx=0x7f0f24007ee0 tx=0x7f0f2400d3b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:27:12.659 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:12.666+0000 7f0f22ffd700 1 -- 172.21.15.52:0/1514204960 <== mon.0 v2:172.21.15.52:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0f24004bb0 con 0x7f0f2c108730 2024-05-27T07:27:12.659 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:12.667+0000 7f0f22ffd700 1 -- 172.21.15.52:0/1514204960 <== mon.0 v2:172.21.15.52:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f0f24004d50 con 0x7f0f2c108730 2024-05-27T07:27:12.659 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:12.667+0000 7f0f22ffd700 1 -- 172.21.15.52:0/1514204960 <== mon.0 v2:172.21.15.52:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0f24005920 con 0x7f0f2c108730 2024-05-27T07:27:12.659 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:12.667+0000 7f0f33e59700 1 -- 172.21.15.52:0/1514204960 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0f2c19aba0 con 0x7f0f2c108730 2024-05-27T07:27:12.659 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:12.667+0000 7f0f33e59700 1 -- 172.21.15.52:0/1514204960 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0f2c19b060 con 0x7f0f2c108730 2024-05-27T07:27:12.662 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:12.672+0000 7f0f22ffd700 1 -- 172.21.15.52:0/1514204960 <== mon.0 v2:172.21.15.52:3300/0 4 ==== mgrmap(e 25) ==== 149936+0+0 (secure 0 0 0) 0x7f0f2401a020 con 0x7f0f2c108730 2024-05-27T07:27:12.663 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:12.673+0000 7f0f4286eb80 1 Processor -- start 2024-05-27T07:27:12.665 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:12.673+0000 7f0f4286eb80 1 -- start start 2024-05-27T07:27:12.665 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:12.673+0000 7f0f22ffd700 1 --2- 172.21.15.52:0/1514204960 >> [v2:172.21.15.52:6800/1822318384,v1:172.21.15.52:6801/1822318384] conn(0x7f0f180b2c20 0x7f0f180b5130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:27:12.665 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:12.673+0000 7f0f4286eb80 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x5626612c7e40 0x56266144cb30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:27:12.665 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:12.673+0000 7f0f4286eb80 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x56266144d070 0x562661446be0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:27:12.665 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:12.673+0000 7f0f4286eb80 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x562661447120 0x562661447530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:27:12.665 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:12.673+0000 7f0f4286eb80 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x562661374d70 con 0x5626612c7e40 2024-05-27T07:27:12.665 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:12.673+0000 7f0f4286eb80 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x562661364910 con 0x562661447120 2024-05-27T07:27:12.665 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:12.673+0000 7f0f4286eb80 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x562661368950 con 0x56266144d070 2024-05-27T07:27:12.665 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:12.673+0000 7f0f22ffd700 1 -- 172.21.15.52:0/1514204960 <== mon.0 v2:172.21.15.52:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9379+0+0 (secure 0 0 0) 0x7f0f240e2200 con 0x7f0f2c108730 2024-05-27T07:27:12.665 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:12.673+0000 7f0f31bf5700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x5626612c7e40 0x56266144cb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:27:12.665 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:12.673+0000 7f0f31bf5700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x5626612c7e40 0x56266144cb30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.52:51636/0 (socket says 172.21.15.52:51636) 2024-05-27T07:27:12.666 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:12.673+0000 7f0f31bf5700 1 -- 172.21.15.52:0/3086816921 learned_addr learned my addr 172.21.15.52:0/3086816921 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-05-27T07:27:12.666 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:12.674+0000 7f0f323f6700 1 --2- 172.21.15.52:0/3086816921 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x562661447120 0x562661447530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:27:12.666 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:12.674+0000 7f0f323f6700 1 -- 172.21.15.52:0/3086816921 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x56266144d070 msgr2=0x562661446be0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-05-27T07:27:12.666 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:12.674+0000 7f0f323f6700 1 --2- 172.21.15.52:0/3086816921 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x56266144d070 0x562661446be0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:27:12.666 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:12.674+0000 7f0f323f6700 1 -- 172.21.15.52:0/3086816921 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x5626612c7e40 msgr2=0x56266144cb30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:27:12.666 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:12.674+0000 7f0f323f6700 1 --2- 172.21.15.52:0/3086816921 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x5626612c7e40 0x56266144cb30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:27:12.666 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:12.674+0000 7f0f323f6700 1 -- 172.21.15.52:0/3086816921 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56266130dfb0 con 0x562661447120 2024-05-27T07:27:12.666 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:12.674+0000 7f0f31bf5700 1 --2- 172.21.15.52:0/3086816921 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x5626612c7e40 0x56266144cb30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-27T07:27:12.666 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:12.674+0000 7f0f313f4700 1 --2- 172.21.15.52:0/1514204960 >> [v2:172.21.15.52:6800/1822318384,v1:172.21.15.52:6801/1822318384] conn(0x7f0f180b2c20 0x7f0f180b5130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:27:12.666 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:12.674+0000 7f0f323f6700 1 --2- 172.21.15.52:0/3086816921 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x562661447120 0x562661447530 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f0f240df3b0 tx=0x7f0f240a8890 comp rx=0 tx=0).ready entity=mon.2 client_cookie=7f7473aceedcc1fd server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:27:12.667 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:12.674+0000 7f0f17fff700 1 -- 172.21.15.52:0/3086816921 <== mon.2 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0f240a8a50 con 0x562661447120 2024-05-27T07:27:12.668 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:12.675+0000 7f0f17fff700 1 -- 172.21.15.52:0/3086816921 <== mon.2 v2:172.21.15.105:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f0f240a9640 con 0x562661447120 2024-05-27T07:27:12.668 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:12.675+0000 7f0f17fff700 1 -- 172.21.15.52:0/3086816921 <== mon.2 v2:172.21.15.105:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0f24051b90 con 0x562661447120 2024-05-27T07:27:12.668 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:12.676+0000 7f0f4286eb80 1 -- 172.21.15.52:0/3086816921 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x562661447120 msgr2=0x562661447530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:27:12.668 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:12.676+0000 7f0f313f4700 1 --2- 172.21.15.52:0/1514204960 >> [v2:172.21.15.52:6800/1822318384,v1:172.21.15.52:6801/1822318384] conn(0x7f0f180b2c20 0x7f0f180b5130 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f0f1c006010 tx=0x7f0f1c024040 comp rx=0 tx=0).ready entity=mgr.15333 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:27:12.668 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:12.676+0000 7f0f4286eb80 1 --2- 172.21.15.52:0/3086816921 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x562661447120 0x562661447530 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f0f240df3b0 tx=0x7f0f240a8890 comp rx=0 tx=0).stop 2024-05-27T07:27:12.668 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:12.676+0000 7f0f4286eb80 1 -- 172.21.15.52:0/3086816921 shutdown_connections 2024-05-27T07:27:12.668 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:12.676+0000 7f0f4286eb80 1 --2- 172.21.15.52:0/3086816921 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x562661447120 0x562661447530 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:27:12.668 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:12.676+0000 7f0f4286eb80 1 --2- 172.21.15.52:0/3086816921 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x5626612c7e40 0x56266144cb30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:27:12.668 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:12.676+0000 7f0f4286eb80 1 --2- 172.21.15.52:0/3086816921 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x56266144d070 0x562661446be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:27:12.668 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:12.676+0000 7f0f4286eb80 1 -- 172.21.15.52:0/3086816921 >> 172.21.15.52:0/3086816921 conn(0x5626612ca770 msgr2=0x562661441d90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:27:12.668 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:12.676+0000 7f0f4286eb80 1 -- 172.21.15.52:0/3086816921 shutdown_connections 2024-05-27T07:27:12.668 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:12.676+0000 7f0f4286eb80 1 -- 172.21.15.52:0/3086816921 wait complete. 2024-05-27T07:27:12.668 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:12.677+0000 7f0f4286eb80 1 -- start start 2024-05-27T07:27:12.669 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:12.677+0000 7f0f4286eb80 1 Processor -- start 2024-05-27T07:27:12.670 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:12.678+0000 7f0f4286eb80 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x5626612c7e40 0x5626614508a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:27:12.670 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:12.678+0000 7f0f4286eb80 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x562661447120 0x562661450de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:27:12.670 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:12.678+0000 7f0f4286eb80 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x56266144d070 0x562661451320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:27:12.670 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:12.678+0000 7f0f4286eb80 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x56266126afa0 con 0x5626612c7e40 2024-05-27T07:27:12.670 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:12.678+0000 7f0f4286eb80 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x5626612ccda0 con 0x56266144d070 2024-05-27T07:27:12.670 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:12.678+0000 7f0f4286eb80 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x562661453b60 con 0x562661447120 2024-05-27T07:27:12.670 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:12.678+0000 7f0f4286eb80 10 client.0 _ll_register_callbacks cb 0x5626612fb5b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T07:27:12.670 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:12.678+0000 7f0f313f4700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x562661447120 0x562661450de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:27:12.670 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:12.678+0000 7f0f31bf5700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x5626612c7e40 0x5626614508a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:27:12.670 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:12.678+0000 7f0f313f4700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x562661447120 0x562661450de0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.173:3300/0 says I am v2:172.21.15.52:52238/0 (socket says 172.21.15.52:52238) 2024-05-27T07:27:12.671 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:12.678+0000 7f0f313f4700 1 -- 172.21.15.52:0/1630757764 learned_addr learned my addr 172.21.15.52:0/1630757764 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-05-27T07:27:12.671 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:12.678+0000 7f0f323f6700 1 --2- 172.21.15.52:0/1630757764 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x56266144d070 0x562661451320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:27:12.671 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:12.678+0000 7f0f313f4700 1 -- 172.21.15.52:0/1630757764 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x56266144d070 msgr2=0x562661451320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:27:12.671 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:12.678+0000 7f0f313f4700 1 --2- 172.21.15.52:0/1630757764 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x56266144d070 0x562661451320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:27:12.671 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:12.678+0000 7f0f313f4700 1 -- 172.21.15.52:0/1630757764 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x5626612c7e40 msgr2=0x5626614508a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:27:12.671 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:12.678+0000 7f0f313f4700 1 --2- 172.21.15.52:0/1630757764 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x5626612c7e40 0x5626614508a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:27:12.671 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:12.678+0000 7f0f313f4700 1 -- 172.21.15.52:0/1630757764 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5626612b48d0 con 0x562661447120 2024-05-27T07:27:12.671 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:12.678+0000 7f0f323f6700 1 --2- 172.21.15.52:0/1630757764 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x56266144d070 0x562661451320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-27T07:27:12.671 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:12.679+0000 7f0f313f4700 1 --2- 172.21.15.52:0/1630757764 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x562661447120 0x562661450de0 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f0f1c0099c0 tx=0x7f0f1c009310 comp rx=0 tx=0).ready entity=mon.1 client_cookie=cc2bf23c01ac1b43 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:27:12.671 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:12.679+0000 7f0f157fa700 10 client.0 ms_handle_connect on v2:172.21.15.173:3300/0 2024-05-27T07:27:12.671 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:12.679+0000 7f0f157fa700 1 -- 172.21.15.52:0/1630757764 <== mon.1 v2:172.21.15.173:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0f1c003e80 con 0x562661447120 2024-05-27T07:27:12.672 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:12.679+0000 7f0f157fa700 1 -- 172.21.15.52:0/1630757764 <== mon.1 v2:172.21.15.173:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 9379+0+0 (secure 0 0 0) 0x7f0f1c00b790 con 0x562661447120 2024-05-27T07:27:12.672 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:12.679+0000 7f0f4286eb80 1 -- 172.21.15.52:0/1630757764 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f0f240a9640 con 0x562661447120 2024-05-27T07:27:12.672 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:12.679+0000 7f0f157fa700 1 -- 172.21.15.52:0/1630757764 <== mon.1 v2:172.21.15.173:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7f0f1c0056c0 con 0x562661447120 2024-05-27T07:27:12.674 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:12.679+0000 7f0f4286eb80 10 client.25399 fetch_fsmap learned FSMap version 30 2024-05-27T07:27:12.674 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:12.679+0000 7f0f4286eb80 1 -- 172.21.15.52:0/1630757764 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x56266131e760 con 0x562661447120 2024-05-27T07:27:12.674 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:12.680+0000 7f0f157fa700 1 -- 172.21.15.52:0/1630757764 <== mon.1 v2:172.21.15.173:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f0f240a9640 con 0x562661447120 2024-05-27T07:27:12.674 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:12.680+0000 7f0f157fa700 1 -- 172.21.15.52:0/1630757764 <== mon.1 v2:172.21.15.173:3300/0 5 ==== fsmap(e 30) ==== 8362+0+0 (secure 0 0 0) 0x7f0f1c036070 con 0x562661447120 2024-05-27T07:27:12.674 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:12.680+0000 7f0f4286eb80 10 client.25399 fetch_fsmap finished waiting for FSMap version 30 2024-05-27T07:27:12.674 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:12.680+0000 7f0f4286eb80 10 client.25399 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-05-27T07:27:12.675 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:12.680+0000 7f0f4286eb80 20 client.25399 populate_metadata read hostname 'smithi052' 2024-05-27T07:27:12.675 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:12.680+0000 7f0f4286eb80 1 --2- 172.21.15.52:0/1630757764 >> [v2:172.21.15.105:6838/500348151,v1:172.21.15.105:6839/500348151] conn(0x562661460af0 0x562661462f10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:27:12.675 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:12.680+0000 7f0f4286eb80 4 client.25399 mds_command: new command op to 24527 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-27T07:27:12.675 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:12.680+0000 7f0f4286eb80 1 -- 172.21.15.52:0/1630757764 --> [v2:172.21.15.105:6838/500348151,v1:172.21.15.105:6839/500348151] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x562661376cc0 con 0x562661460af0 2024-05-27T07:27:12.675 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:12.680+0000 7f0f31bf5700 1 --2- 172.21.15.52:0/1630757764 >> [v2:172.21.15.105:6838/500348151,v1:172.21.15.105:6839/500348151] conn(0x562661460af0 0x562661462f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:27:12.675 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:12.681+0000 7f0f157fa700 10 client.25399 ms_handle_connect on v2:172.21.15.105:6838/500348151 2024-05-27T07:27:12.675 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:12.681+0000 7f0f31bf5700 1 --2- 172.21.15.52:0/1630757764 >> [v2:172.21.15.105:6838/500348151,v1:172.21.15.105:6839/500348151] conn(0x562661460af0 0x562661462f10 crc :-1 s=READY pgs=61 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=179ef9612d915491 server_cookie=884558a260691dcc in_seq=0 out_seq=0 2024-05-27T07:27:12.675 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:12.682+0000 7f0f157fa700 1 -- 172.21.15.52:0/1630757764 <== mds.0 v2:172.21.15.105:6838/500348151 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24041 (crc 0 0 0) 0x562661376cc0 con 0x562661460af0 2024-05-27T07:27:12.675 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:12.682+0000 7f0f157fa700 10 client.25399 handle_command_reply: tid=0 2024-05-27T07:27:12.675 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:12.682+0000 7f0f4286eb80 1 client.25399 shutdown 2024-05-27T07:27:12.676 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:12.682+0000 7f0f4286eb80 1 -- 172.21.15.52:0/1630757764 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x562661447120 msgr2=0x562661450de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:27:12.676 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:12.682+0000 7f0f4286eb80 1 --2- 172.21.15.52:0/1630757764 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x562661447120 0x562661450de0 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f0f1c0099c0 tx=0x7f0f1c009310 comp rx=0 tx=0).stop 2024-05-27T07:27:12.676 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:12.682+0000 7f0f4286eb80 1 -- 172.21.15.52:0/1630757764 shutdown_connections 2024-05-27T07:27:12.676 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:12.682+0000 7f0f4286eb80 1 --2- 172.21.15.52:0/1630757764 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x56266144d070 0x562661451320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:27:12.676 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:12.682+0000 7f0f4286eb80 1 --2- 172.21.15.52:0/1630757764 >> [v2:172.21.15.105:6838/500348151,v1:172.21.15.105:6839/500348151] conn(0x562661460af0 0x562661462f10 crc :-1 s=READY pgs=61 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:27:12.676 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:12.682+0000 7f0f4286eb80 1 --2- 172.21.15.52:0/1630757764 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x5626612c7e40 0x5626614508a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:27:12.676 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:12.682+0000 7f0f4286eb80 1 --2- 172.21.15.52:0/1630757764 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x562661447120 0x562661450de0 unknown :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:27:12.676 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:12.682+0000 7f0f4286eb80 1 -- 172.21.15.52:0/1630757764 >> 172.21.15.52:0/1630757764 conn(0x5626612ca770 msgr2=0x562661445bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:27:12.676 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:12.682+0000 7f0f157fa700 0 client.25399 ms_handle_reset on v2:172.21.15.105:6838/500348151 2024-05-27T07:27:12.676 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:12.683+0000 7f0f4286eb80 1 -- 172.21.15.52:0/1630757764 shutdown_connections 2024-05-27T07:27:12.676 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:12.683+0000 7f0f4286eb80 1 -- 172.21.15.52:0/1630757764 wait complete. 2024-05-27T07:27:12.677 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:12.683+0000 7f0f4286eb80 20 client.25399 trim_cache size 0 max 16384 2024-05-27T07:27:12.706 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:12.714+0000 7f0f4286eb80 1 Processor -- start 2024-05-27T07:27:12.706 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:12.714+0000 7f0f4286eb80 1 -- start start 2024-05-27T07:27:12.706 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:12.715+0000 7f0f4286eb80 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5626612c7e40 0x562661446be0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:27:12.706 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:12.715+0000 7f0f4286eb80 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x562661447120 0x562661450ed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:27:12.706 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:12.715+0000 7f0f4286eb80 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x56266144d070 0x5626614501a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:27:12.706 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:12.715+0000 7f0f4286eb80 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x7f0f2c1ae320 con 0x56266144d070 2024-05-27T07:27:12.706 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:12.715+0000 7f0f4286eb80 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f0f2c1afe60 con 0x562661447120 2024-05-27T07:27:12.707 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:12.715+0000 7f0f4286eb80 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f0f2c1ae9f0 con 0x5626612c7e40 2024-05-27T07:27:12.707 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:12.715+0000 7f0f323f6700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x56266144d070 0x5626614501a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:27:12.707 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:12.715+0000 7f0f323f6700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x56266144d070 0x5626614501a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.52:51654/0 (socket says 172.21.15.52:51654) 2024-05-27T07:27:12.707 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:12.715+0000 7f0f323f6700 1 -- 172.21.15.52:0/237935715 learned_addr learned my addr 172.21.15.52:0/237935715 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-05-27T07:27:12.708 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:12.716+0000 7f0f31bf5700 1 --2- 172.21.15.52:0/237935715 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5626612c7e40 0x562661446be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:27:12.708 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:12.716+0000 7f0f323f6700 1 -- 172.21.15.52:0/237935715 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5626612c7e40 msgr2=0x562661446be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:27:12.708 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:12.716+0000 7f0f323f6700 1 --2- 172.21.15.52:0/237935715 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5626612c7e40 0x562661446be0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:27:12.708 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:12.716+0000 7f0f323f6700 1 -- 172.21.15.52:0/237935715 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x562661447120 msgr2=0x562661450ed0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-05-27T07:27:12.708 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:12.716+0000 7f0f323f6700 1 --2- 172.21.15.52:0/237935715 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x562661447120 0x562661450ed0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:27:12.708 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:12.716+0000 7f0f323f6700 1 -- 172.21.15.52:0/237935715 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56266143daa0 con 0x56266144d070 2024-05-27T07:27:12.708 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:12.716+0000 7f0f323f6700 1 --2- 172.21.15.52:0/237935715 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x56266144d070 0x5626614501a0 secure :-1 s=READY pgs=516 cs=0 l=1 rev1=1 crypto rx=0x7f0f240a8a00 tx=0x7f0f24005b80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=151fc1a4db45c004 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:27:12.708 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:12.716+0000 7f0f177fe700 1 -- 172.21.15.52:0/237935715 <== mon.0 v2:172.21.15.52:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0f24056070 con 0x56266144d070 2024-05-27T07:27:12.708 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:12.716+0000 7f0f177fe700 1 -- 172.21.15.52:0/237935715 <== mon.0 v2:172.21.15.52:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f0f240a8f40 con 0x56266144d070 2024-05-27T07:27:12.708 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:12.716+0000 7f0f177fe700 1 -- 172.21.15.52:0/237935715 <== mon.0 v2:172.21.15.52:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0f240229f0 con 0x56266144d070 2024-05-27T07:27:12.708 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:12.717+0000 7f0f4286eb80 1 -- 172.21.15.52:0/237935715 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x56266144d070 msgr2=0x5626614501a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:27:12.709 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:12.717+0000 7f0f4286eb80 1 --2- 172.21.15.52:0/237935715 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x56266144d070 0x5626614501a0 secure :-1 s=READY pgs=516 cs=0 l=1 rev1=1 crypto rx=0x7f0f240a8a00 tx=0x7f0f24005b80 comp rx=0 tx=0).stop 2024-05-27T07:27:12.709 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:12.718+0000 7f0f4286eb80 1 -- 172.21.15.52:0/237935715 shutdown_connections 2024-05-27T07:27:12.709 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:12.718+0000 7f0f4286eb80 1 --2- 172.21.15.52:0/237935715 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x56266144d070 0x5626614501a0 unknown :-1 s=CLOSED pgs=516 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:27:12.710 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:12.718+0000 7f0f4286eb80 1 --2- 172.21.15.52:0/237935715 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5626612c7e40 0x562661446be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:27:12.710 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:12.718+0000 7f0f4286eb80 1 --2- 172.21.15.52:0/237935715 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x562661447120 0x562661450ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:27:12.710 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:12.718+0000 7f0f4286eb80 1 -- 172.21.15.52:0/237935715 >> 172.21.15.52:0/237935715 conn(0x5626612ca770 msgr2=0x56266144a280 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:27:12.710 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:12.718+0000 7f0f4286eb80 1 -- 172.21.15.52:0/237935715 shutdown_connections 2024-05-27T07:27:12.710 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:12.718+0000 7f0f4286eb80 1 -- 172.21.15.52:0/237935715 wait complete. 2024-05-27T07:27:12.710 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:12.719+0000 7f0f4286eb80 1 -- start start 2024-05-27T07:27:12.710 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:12.719+0000 7f0f4286eb80 1 Processor -- start 2024-05-27T07:27:12.713 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:12.719+0000 7f0f4286eb80 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5626612c7e40 0x562661459440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:27:12.713 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:12.719+0000 7f0f4286eb80 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x562661447120 0x562661467ef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:27:12.713 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:12.720+0000 7f0f4286eb80 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x56266144d070 0x562661468430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:27:12.713 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:12.720+0000 7f0f313f4700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x562661447120 0x562661467ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:27:12.713 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:12.720+0000 7f0f313f4700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x562661447120 0x562661467ef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.52:51666/0 (socket says 172.21.15.52:51666) 2024-05-27T07:27:12.713 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:12.720+0000 7f0f31bf5700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5626612c7e40 0x562661459440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:27:12.713 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:12.720+0000 7f0f313f4700 1 -- 172.21.15.52:0/2229576327 learned_addr learned my addr 172.21.15.52:0/2229576327 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-05-27T07:27:12.713 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:12.720+0000 7f0f4286eb80 1 -- 172.21.15.52:0/2229576327 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x7f0f2c1af790 con 0x562661447120 2024-05-27T07:27:12.714 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:12.720+0000 7f0f4286eb80 1 -- 172.21.15.52:0/2229576327 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f0f2c1af0c0 con 0x56266144d070 2024-05-27T07:27:12.714 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:12.720+0000 7f0f4286eb80 1 -- 172.21.15.52:0/2229576327 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f0f2c1ac2c0 con 0x5626612c7e40 2024-05-27T07:27:12.714 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:12.720+0000 7f0f323f6700 1 --2- 172.21.15.52:0/2229576327 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x56266144d070 0x562661468430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:27:12.714 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:12.720+0000 7f0f4286eb80 10 client.0 _ll_register_callbacks cb 0x5626612fb5b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T07:27:12.714 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:12.720+0000 7f0f323f6700 1 -- 172.21.15.52:0/2229576327 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5626612c7e40 msgr2=0x562661459440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:27:12.714 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:12.721+0000 7f0f323f6700 1 --2- 172.21.15.52:0/2229576327 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5626612c7e40 0x562661459440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:27:12.714 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:12.721+0000 7f0f323f6700 1 -- 172.21.15.52:0/2229576327 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x562661447120 msgr2=0x562661467ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:27:12.714 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:12.721+0000 7f0f323f6700 1 --2- 172.21.15.52:0/2229576327 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x562661447120 0x562661467ef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:27:12.714 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:12.721+0000 7f0f323f6700 1 -- 172.21.15.52:0/2229576327 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({osdmap=0}) -- 0x562661468b60 con 0x56266144d070 2024-05-27T07:27:12.714 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:12.721+0000 7f0f323f6700 1 --2- 172.21.15.52:0/2229576327 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x56266144d070 0x562661468430 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f0f240a8a00 tx=0x7f0f24022ca0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b3b3964157cfb8f4 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:27:12.714 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:12.721+0000 7f0f157fa700 10 client.0 ms_handle_connect on v2:172.21.15.105:3300/0 2024-05-27T07:27:12.714 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:12.721+0000 7f0f157fa700 1 -- 172.21.15.52:0/2229576327 <== mon.2 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0f24056040 con 0x56266144d070 2024-05-27T07:27:12.715 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:12.721+0000 7f0f157fa700 1 -- 172.21.15.52:0/2229576327 <== mon.2 v2:172.21.15.105:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 9379+0+0 (secure 0 0 0) 0x7f0f240a91a0 con 0x56266144d070 2024-05-27T07:27:12.715 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:12.721+0000 7f0f4286eb80 1 -- 172.21.15.52:0/2229576327 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f0f240a8f40 con 0x56266144d070 2024-05-27T07:27:12.715 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:12.721+0000 7f0f157fa700 1 -- 172.21.15.52:0/2229576327 <== mon.2 v2:172.21.15.105:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7f0f240ad2f0 con 0x56266144d070 2024-05-27T07:27:12.715 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:12.721+0000 7f0f4286eb80 10 client.25463 fetch_fsmap learned FSMap version 30 2024-05-27T07:27:12.715 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:12.722+0000 7f0f4286eb80 1 -- 172.21.15.52:0/2229576327 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x5626614cb430 con 0x56266144d070 2024-05-27T07:27:12.715 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:12.722+0000 7f0f157fa700 1 -- 172.21.15.52:0/2229576327 <== mon.2 v2:172.21.15.105:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f0f24051c60 con 0x56266144d070 2024-05-27T07:27:12.715 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:12.722+0000 7f0f157fa700 1 -- 172.21.15.52:0/2229576327 <== mon.2 v2:172.21.15.105:3300/0 5 ==== fsmap(e 30) ==== 8362+0+0 (secure 0 0 0) 0x7f0f240a81a0 con 0x56266144d070 2024-05-27T07:27:12.715 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:12.722+0000 7f0f4286eb80 10 client.25463 fetch_fsmap finished waiting for FSMap version 30 2024-05-27T07:27:12.715 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:12.722+0000 7f0f4286eb80 10 client.25463 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-05-27T07:27:12.715 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:12.722+0000 7f0f4286eb80 20 client.25463 populate_metadata read hostname 'smithi052' 2024-05-27T07:27:12.715 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:12.722+0000 7f0f4286eb80 1 --2- 172.21.15.52:0/2229576327 >> [v2:172.21.15.105:6838/500348151,v1:172.21.15.105:6839/500348151] conn(0x5626614cb600 0x5626614cda20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:27:12.715 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:12.722+0000 7f0f4286eb80 4 client.25463 mds_command: new command op to 24527 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-05-27T07:27:12.716 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:12.722+0000 7f0f4286eb80 1 -- 172.21.15.52:0/2229576327 --> [v2:172.21.15.105:6838/500348151,v1:172.21.15.105:6839/500348151] -- command(tid 0: {"prefix": "scrub status"}) -- 0x7f0f240a8f40 con 0x5626614cb600 2024-05-27T07:27:12.716 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:12.723+0000 7f0f31bf5700 1 --2- 172.21.15.52:0/2229576327 >> [v2:172.21.15.105:6838/500348151,v1:172.21.15.105:6839/500348151] conn(0x5626614cb600 0x5626614cda20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:27:12.717 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:12.723+0000 7f0f157fa700 10 client.25463 ms_handle_connect on v2:172.21.15.105:6838/500348151 2024-05-27T07:27:12.717 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:12.723+0000 7f0f31bf5700 1 --2- 172.21.15.52:0/2229576327 >> [v2:172.21.15.105:6838/500348151,v1:172.21.15.105:6839/500348151] conn(0x5626614cb600 0x5626614cda20 crc :-1 s=READY pgs=62 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=1732f50b48201146 server_cookie=1ff93a5b46e81366 in_seq=0 out_seq=0 2024-05-27T07:27:12.717 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:12.723+0000 7f0f157fa700 1 -- 172.21.15.52:0/2229576327 <== mds.0 v2:172.21.15.105:6838/500348151 1 ==== command_reply(tid 0: 0 ) ==== 8+0+63 (crc 0 0 0) 0x7f0f240a8f40 con 0x5626614cb600 2024-05-27T07:27:12.717 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:12.723+0000 7f0f157fa700 10 client.25463 handle_command_reply: tid=0 2024-05-27T07:27:12.717 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:12.723+0000 7f0f4286eb80 1 client.25463 shutdown 2024-05-27T07:27:12.717 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:12.724+0000 7f0f4286eb80 1 -- 172.21.15.52:0/2229576327 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x56266144d070 msgr2=0x562661468430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:27:12.717 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:12.724+0000 7f0f4286eb80 1 --2- 172.21.15.52:0/2229576327 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x56266144d070 0x562661468430 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f0f240a8a00 tx=0x7f0f24022ca0 comp rx=0 tx=0).stop 2024-05-27T07:27:12.717 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:12.724+0000 7f0f4286eb80 1 -- 172.21.15.52:0/2229576327 shutdown_connections 2024-05-27T07:27:12.717 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:12.724+0000 7f0f4286eb80 1 --2- 172.21.15.52:0/2229576327 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x56266144d070 0x562661468430 unknown :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:27:12.718 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:12.724+0000 7f0f4286eb80 1 --2- 172.21.15.52:0/2229576327 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5626612c7e40 0x562661459440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:27:12.718 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:12.724+0000 7f0f4286eb80 1 --2- 172.21.15.52:0/2229576327 >> [v2:172.21.15.105:6838/500348151,v1:172.21.15.105:6839/500348151] conn(0x5626614cb600 0x5626614cda20 crc :-1 s=READY pgs=62 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:27:12.718 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:12.724+0000 7f0f4286eb80 1 --2- 172.21.15.52:0/2229576327 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x562661447120 0x562661467ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:27:12.718 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:12.724+0000 7f0f4286eb80 1 -- 172.21.15.52:0/2229576327 >> 172.21.15.52:0/2229576327 conn(0x5626612ca770 msgr2=0x56266144acb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:27:12.718 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:12.724+0000 7f0f157fa700 0 client.25463 ms_handle_reset on v2:172.21.15.105:6838/500348151 2024-05-27T07:27:12.718 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:12.724+0000 7f0f4286eb80 1 -- 172.21.15.52:0/2229576327 shutdown_connections 2024-05-27T07:27:12.718 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:12.724+0000 7f0f4286eb80 1 -- 172.21.15.52:0/2229576327 wait complete. 2024-05-27T07:27:12.718 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:12.725+0000 7f0f4286eb80 20 client.25463 trim_cache size 0 max 16384 2024-05-27T07:27:12.718 INFO:teuthology.orchestra.run.smithi052.stdout:{ 2024-05-27T07:27:12.718 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "no active scrubs running", 2024-05-27T07:27:12.719 INFO:teuthology.orchestra.run.smithi052.stdout: "scrubs": {} 2024-05-27T07:27:12.719 INFO:teuthology.orchestra.run.smithi052.stdout:} 2024-05-27T07:27:12.719 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:12.725+0000 7f0f33e59700 1 -- 172.21.15.52:0/1514204960 >> [v2:172.21.15.52:6800/1822318384,v1:172.21.15.52:6801/1822318384] conn(0x7f0f180b2c20 msgr2=0x7f0f180b5130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:27:12.719 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:12.725+0000 7f0f33e59700 1 --2- 172.21.15.52:0/1514204960 >> [v2:172.21.15.52:6800/1822318384,v1:172.21.15.52:6801/1822318384] conn(0x7f0f180b2c20 0x7f0f180b5130 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f0f1c006010 tx=0x7f0f1c024040 comp rx=0 tx=0).stop 2024-05-27T07:27:12.719 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:12.725+0000 7f0f33e59700 1 -- 172.21.15.52:0/1514204960 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f0f2c108730 msgr2=0x7f0f2c0ffd10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:27:12.720 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:12.725+0000 7f0f33e59700 1 --2- 172.21.15.52:0/1514204960 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f0f2c108730 0x7f0f2c0ffd10 secure :-1 s=READY pgs=515 cs=0 l=1 rev1=1 crypto rx=0x7f0f24007ee0 tx=0x7f0f2400d3b0 comp rx=0 tx=0).stop 2024-05-27T07:27:12.720 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:12.726+0000 7f0f33e59700 1 -- 172.21.15.52:0/1514204960 shutdown_connections 2024-05-27T07:27:12.720 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:12.726+0000 7f0f33e59700 1 --2- 172.21.15.52:0/1514204960 >> [v2:172.21.15.52:6800/1822318384,v1:172.21.15.52:6801/1822318384] conn(0x7f0f180b2c20 0x7f0f180b5130 unknown :-1 s=CLOSED pgs=69 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:27:12.720 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:12.726+0000 7f0f33e59700 1 --2- 172.21.15.52:0/1514204960 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f0f2c108730 0x7f0f2c0ffd10 unknown :-1 s=CLOSED pgs=515 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:27:12.720 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:12.726+0000 7f0f33e59700 1 --2- 172.21.15.52:0/1514204960 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f0f2c1034b0 0x7f0f2c0ff290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:27:12.720 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:12.726+0000 7f0f33e59700 1 --2- 172.21.15.52:0/1514204960 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f0f2c107de0 0x7f0f2c0ff7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:27:12.720 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:12.726+0000 7f0f33e59700 1 -- 172.21.15.52:0/1514204960 >> 172.21.15.52:0/1514204960 conn(0x7f0f2c0fedb0 msgr2=0x7f0f2c109690 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:27:12.720 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:12.726+0000 7f0f33e59700 1 -- 172.21.15.52:0/1514204960 shutdown_connections 2024-05-27T07:27:12.720 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:12.726+0000 7f0f33e59700 1 -- 172.21.15.52:0/1514204960 wait complete. 2024-05-27T07:27:13.351 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:27:13 smithi173 ceph-mon[37000]: from='mgr.15333 172.21.15.52:0/416389082' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T07:27:13.352 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:27:13 smithi173 ceph-mon[37000]: from='mgr.15333 172.21.15.52:0/416389082' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-27T07:27:13.352 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:27:13 smithi173 ceph-mon[37000]: from='mgr.15333 172.21.15.52:0/416389082' entity='mgr.x' 2024-05-27T07:27:13.439 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:27:13 smithi105 ceph-mon[35977]: from='mgr.15333 172.21.15.52:0/416389082' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T07:27:13.440 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:27:13 smithi105 ceph-mon[35977]: from='mgr.15333 172.21.15.52:0/416389082' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-27T07:27:13.440 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:27:13 smithi105 ceph-mon[35977]: from='mgr.15333 172.21.15.52:0/416389082' entity='mgr.x' 2024-05-27T07:27:13.479 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:27:13 smithi052 ceph-mon[31955]: from='mgr.15333 172.21.15.52:0/416389082' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T07:27:13.479 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:27:13 smithi052 ceph-mon[31955]: from='mgr.15333 172.21.15.52:0/416389082' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-27T07:27:13.479 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:27:13 smithi052 ceph-mon[31955]: from='mgr.15333 172.21.15.52:0/416389082' entity='mgr.x' 2024-05-27T07:27:13.568 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-05-27T07:27:13.568 DEBUG:teuthology.orchestra.run.smithi052:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:38f9d900c83c63799fbdbe61acc9a11b0d3554a6 shell --fsid 73df2636-1bf8-11ef-bc9b-c7b262605968 -- ceph fs dump --format=json 2024-05-27T07:27:13.890 INFO:teuthology.orchestra.run.smithi052.stderr:Inferring config /var/lib/ceph/73df2636-1bf8-11ef-bc9b-c7b262605968/mon.a/config 2024-05-27T07:27:14.317 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:27:14 smithi052 ceph-mon[31955]: pgmap v41: 129 pgs: 129 active+clean; 1.4 MiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 23 KiB/s wr, 29 op/s 2024-05-27T07:27:14.352 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:27:14 smithi173 ceph-mon[37000]: pgmap v41: 129 pgs: 129 active+clean; 1.4 MiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 23 KiB/s wr, 29 op/s 2024-05-27T07:27:14.439 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:27:14 smithi105 ceph-mon[35977]: pgmap v41: 129 pgs: 129 active+clean; 1.4 MiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 23 KiB/s wr, 29 op/s 2024-05-27T07:27:15.740 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:15.748+0000 7fbf8e9b8700 1 -- 172.21.15.52:0/584255665 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fbf881034b0 msgr2=0x7fbf881038c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:27:15.740 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:15.748+0000 7fbf8e9b8700 1 --2- 172.21.15.52:0/584255665 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fbf881034b0 0x7fbf881038c0 secure :-1 s=READY pgs=517 cs=0 l=1 rev1=1 crypto rx=0x7fbf780099c0 tx=0x7fbf78009cd0 comp rx=0 tx=0).stop 2024-05-27T07:27:15.740 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:15.748+0000 7fbf8e9b8700 1 -- 172.21.15.52:0/584255665 shutdown_connections 2024-05-27T07:27:15.740 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:15.748+0000 7fbf8e9b8700 1 --2- 172.21.15.52:0/584255665 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fbf88108730 0x7fbf88110090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:27:15.740 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:15.748+0000 7fbf8e9b8700 1 --2- 172.21.15.52:0/584255665 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fbf881034b0 0x7fbf881038c0 unknown :-1 s=CLOSED pgs=517 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:27:15.740 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:15.748+0000 7fbf8e9b8700 1 --2- 172.21.15.52:0/584255665 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fbf88107de0 0x7fbf881081f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:27:15.740 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:15.748+0000 7fbf8e9b8700 1 -- 172.21.15.52:0/584255665 >> 172.21.15.52:0/584255665 conn(0x7fbf880fedb0 msgr2=0x7fbf881011d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:27:15.740 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:15.749+0000 7fbf8e9b8700 1 -- 172.21.15.52:0/584255665 shutdown_connections 2024-05-27T07:27:15.740 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:15.749+0000 7fbf8e9b8700 1 -- 172.21.15.52:0/584255665 wait complete. 2024-05-27T07:27:15.741 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:15.749+0000 7fbf8e9b8700 1 Processor -- start 2024-05-27T07:27:15.741 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:15.750+0000 7fbf8e9b8700 1 -- start start 2024-05-27T07:27:15.741 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:15.750+0000 7fbf8e9b8700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fbf881034b0 0x7fbf8819ade0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:27:15.742 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:15.750+0000 7fbf8e9b8700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fbf88107de0 0x7fbf8819b380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:27:15.742 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:15.750+0000 7fbf8e9b8700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fbf88108730 0x7fbf8819f610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:27:15.742 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:15.750+0000 7fbf8e9b8700 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x7fbf881144b0 con 0x7fbf88108730 2024-05-27T07:27:15.742 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:15.750+0000 7fbf8e9b8700 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7fbf88114330 con 0x7fbf88107de0 2024-05-27T07:27:15.742 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:15.750+0000 7fbf8e9b8700 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7fbf88114630 con 0x7fbf881034b0 2024-05-27T07:27:15.742 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:15.750+0000 7fbf8cf55700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fbf88108730 0x7fbf8819f610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:27:15.742 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:15.750+0000 7fbf8cf55700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fbf88108730 0x7fbf8819f610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.52:51674/0 (socket says 172.21.15.52:51674) 2024-05-27T07:27:15.742 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:15.750+0000 7fbf8cf55700 1 -- 172.21.15.52:0/3618380786 learned_addr learned my addr 172.21.15.52:0/3618380786 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-05-27T07:27:15.742 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:15.750+0000 7fbf87fff700 1 --2- 172.21.15.52:0/3618380786 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fbf881034b0 0x7fbf8819ade0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:27:15.742 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:15.751+0000 7fbf877fe700 1 --2- 172.21.15.52:0/3618380786 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fbf88107de0 0x7fbf8819b380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:27:15.742 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:15.751+0000 7fbf87fff700 1 -- 172.21.15.52:0/3618380786 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fbf88107de0 msgr2=0x7fbf8819b380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:27:15.743 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:15.751+0000 7fbf87fff700 1 --2- 172.21.15.52:0/3618380786 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fbf88107de0 0x7fbf8819b380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:27:15.743 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:15.751+0000 7fbf87fff700 1 -- 172.21.15.52:0/3618380786 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fbf88108730 msgr2=0x7fbf8819f610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:27:15.743 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:15.751+0000 7fbf87fff700 1 --2- 172.21.15.52:0/3618380786 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fbf88108730 0x7fbf8819f610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:27:15.743 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:15.751+0000 7fbf87fff700 1 -- 172.21.15.52:0/3618380786 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbf8819fcb0 con 0x7fbf881034b0 2024-05-27T07:27:15.744 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:15.751+0000 7fbf8cf55700 1 --2- 172.21.15.52:0/3618380786 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fbf88108730 0x7fbf8819f610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-27T07:27:15.744 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:15.751+0000 7fbf87fff700 1 --2- 172.21.15.52:0/3618380786 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fbf881034b0 0x7fbf8819ade0 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7fbf780098e0 tx=0x7fbf78009550 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:27:15.744 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:15.751+0000 7fbf857fa700 1 -- 172.21.15.52:0/3618380786 <== mon.1 v2:172.21.15.173:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbf78009e60 con 0x7fbf881034b0 2024-05-27T07:27:15.744 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:15.752+0000 7fbf857fa700 1 -- 172.21.15.52:0/3618380786 <== mon.1 v2:172.21.15.173:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fbf78023070 con 0x7fbf881034b0 2024-05-27T07:27:15.744 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:15.752+0000 7fbf857fa700 1 -- 172.21.15.52:0/3618380786 <== mon.1 v2:172.21.15.173:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbf7801fc70 con 0x7fbf881034b0 2024-05-27T07:27:15.744 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:15.752+0000 7fbf8e9b8700 1 -- 172.21.15.52:0/3618380786 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbf8819ff40 con 0x7fbf881034b0 2024-05-27T07:27:15.744 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:15.752+0000 7fbf8e9b8700 1 -- 172.21.15.52:0/3618380786 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbf881a0420 con 0x7fbf881034b0 2024-05-27T07:27:15.744 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:15.753+0000 7fbf8e9b8700 1 -- 172.21.15.52:0/3618380786 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fbf880512b0 con 0x7fbf881034b0 2024-05-27T07:27:15.746 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:15.755+0000 7fbf857fa700 1 -- 172.21.15.52:0/3618380786 <== mon.1 v2:172.21.15.173:3300/0 4 ==== mgrmap(e 25) ==== 149936+0+0 (secure 0 0 0) 0x7fbf7801f4d0 con 0x7fbf881034b0 2024-05-27T07:27:15.746 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:15.755+0000 7fbf857fa700 1 --2- 172.21.15.52:0/3618380786 >> [v2:172.21.15.52:6800/1822318384,v1:172.21.15.52:6801/1822318384] conn(0x7fbf700b2c20 0x7fbf700b5130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:27:15.746 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:15.756+0000 7fbf857fa700 1 -- 172.21.15.52:0/3618380786 <== mon.1 v2:172.21.15.173:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9379+0+0 (secure 0 0 0) 0x7fbf780e7ef0 con 0x7fbf881034b0 2024-05-27T07:27:15.746 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:15.756+0000 7fbf877fe700 1 --2- 172.21.15.52:0/3618380786 >> [v2:172.21.15.52:6800/1822318384,v1:172.21.15.52:6801/1822318384] conn(0x7fbf700b2c20 0x7fbf700b5130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:27:15.750 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:15.757+0000 7fbf877fe700 1 --2- 172.21.15.52:0/3618380786 >> [v2:172.21.15.52:6800/1822318384,v1:172.21.15.52:6801/1822318384] conn(0x7fbf700b2c20 0x7fbf700b5130 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7fbf88102350 tx=0x7fbf7c009260 comp rx=0 tx=0).ready entity=mgr.15333 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:27:15.750 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:15.759+0000 7fbf857fa700 1 -- 172.21.15.52:0/3618380786 <== mon.1 v2:172.21.15.173:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191963 (secure 0 0 0) 0x7fbf78013070 con 0x7fbf881034b0 2024-05-27T07:27:16.149 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:16.158+0000 7fbf8e9b8700 1 -- 172.21.15.52:0/3618380786 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fbf8802e5e0 con 0x7fbf881034b0 2024-05-27T07:27:16.149 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:16.159+0000 7fbf857fa700 1 -- 172.21.15.52:0/3618380786 <== mon.1 v2:172.21.15.173:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 30 v30) ==== 94+0+11938 (secure 0 0 0) 0x7fbf780af4d0 con 0x7fbf881034b0 2024-05-27T07:27:16.150 INFO:teuthology.orchestra.run.smithi052.stdout: 2024-05-27T07:27:16.150 INFO:teuthology.orchestra.run.smithi052.stdout:{"epoch":30,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14556,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.52:6835/2584733778","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.52:6834","nonce":2584733778},{"type":"v1","addr":"172.21.15.52:6835","nonce":2584733778}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14568,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.52:6837/675445303","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.52:6836","nonce":675445303},{"type":"v1","addr":"172.21.15.52:6837","nonce":675445303}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":30,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-05-27T07:21:50.521783+0000","modified":"2024-05-27T07:25:54.323921+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24527,"mds_1":24497,"mds_2":24475,"mds_3":24473,"mds_4":24466},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14619":{"gid":14619,"name":"g","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.52:6839/4258549559","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.52:6838","nonce":4258549559},{"type":"v1","addr":"172.21.15.52:6839","nonce":4258549559}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_14631":{"gid":14631,"name":"a","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.52:6841/2878618188","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.52:6840","nonce":2878618188},{"type":"v1","addr":"172.21.15.52:6841","nonce":2878618188}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24433":{"gid":24433,"name":"l","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.173:6833/3013589220","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6832","nonce":3013589220},{"type":"v1","addr":"172.21.15.173:6833","nonce":3013589220}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24452":{"gid":24452,"name":"k","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.105:6833/2800678412","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6832","nonce":2800678412},{"type":"v1","addr":"172.21.15.105:6833","nonce":2800678412}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24460":{"gid":24460,"name":"f","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.173:6835/1619486437","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6834","nonce":1619486437},{"type":"v1","addr":"172.21.15.173:6835","nonce":1619486437}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24466":{"gid":24466,"name":"c","rank":4,"incarnation":26,"state":"up:active","state_seq":25,"addr":"172.21.15.173:6837/1396363827","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6836","nonce":1396363827},{"type":"v1","addr":"172.21.15.173:6837","nonce":1396363827}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24473":{"gid":24473,"name":"e","rank":3,"incarnation":24,"state":"up:active","state_seq":26,"addr":"172.21.15.105:6835/3298498384","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6834","nonce":3298498384},{"type":"v1","addr":"172.21.15.105:6835","nonce":3298498384}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24475":{"gid":24475,"name":"i","rank":2,"incarnation":22,"state":"up:active","state_seq":23,"addr":"172.21.15.173:6839/564778293","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6838","nonce":564778293},{"type":"v1","addr":"172.21.15.173:6839","nonce":564778293}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24497":{"gid":24497,"name":"h","rank":1,"incarnation":20,"state":"up:active","state_seq":24,"addr":"172.21.15.105:6837/191393149","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6836","nonce":191393149},{"type":"v1","addr":"172.21.15.105:6837","nonce":191393149}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24527":{"gid":24527,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":5,"addr":"172.21.15.105:6839/500348151","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6838","nonce":500348151},{"type":"v1","addr":"172.21.15.105:6839","nonce":500348151}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24527,"qdb_cluster":[24527,24497,24475,24466,24473]},"id":1}]} 2024-05-27T07:27:16.153 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:16.163+0000 7fbf8e9b8700 1 -- 172.21.15.52:0/3618380786 >> [v2:172.21.15.52:6800/1822318384,v1:172.21.15.52:6801/1822318384] conn(0x7fbf700b2c20 msgr2=0x7fbf700b5130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:27:16.153 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:16.163+0000 7fbf8e9b8700 1 --2- 172.21.15.52:0/3618380786 >> [v2:172.21.15.52:6800/1822318384,v1:172.21.15.52:6801/1822318384] conn(0x7fbf700b2c20 0x7fbf700b5130 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7fbf88102350 tx=0x7fbf7c009260 comp rx=0 tx=0).stop 2024-05-27T07:27:16.154 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:16.163+0000 7fbf8e9b8700 1 -- 172.21.15.52:0/3618380786 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fbf881034b0 msgr2=0x7fbf8819ade0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:27:16.154 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:16.163+0000 7fbf8e9b8700 1 --2- 172.21.15.52:0/3618380786 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fbf881034b0 0x7fbf8819ade0 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7fbf780098e0 tx=0x7fbf78009550 comp rx=0 tx=0).stop 2024-05-27T07:27:16.154 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:16.163+0000 7fbf8e9b8700 1 -- 172.21.15.52:0/3618380786 shutdown_connections 2024-05-27T07:27:16.154 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:16.163+0000 7fbf8e9b8700 1 --2- 172.21.15.52:0/3618380786 >> [v2:172.21.15.52:6800/1822318384,v1:172.21.15.52:6801/1822318384] conn(0x7fbf700b2c20 0x7fbf700b5130 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:27:16.154 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:16.163+0000 7fbf8e9b8700 1 --2- 172.21.15.52:0/3618380786 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fbf88108730 0x7fbf8819f610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:27:16.154 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:16.163+0000 7fbf8e9b8700 1 --2- 172.21.15.52:0/3618380786 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fbf881034b0 0x7fbf8819ade0 unknown :-1 s=CLOSED pgs=75 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:27:16.154 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:16.163+0000 7fbf8e9b8700 1 --2- 172.21.15.52:0/3618380786 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fbf88107de0 0x7fbf8819b380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:27:16.154 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:16.163+0000 7fbf8e9b8700 1 -- 172.21.15.52:0/3618380786 >> 172.21.15.52:0/3618380786 conn(0x7fbf880fedb0 msgr2=0x7fbf88109690 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:27:16.154 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:16.163+0000 7fbf8e9b8700 1 -- 172.21.15.52:0/3618380786 shutdown_connections 2024-05-27T07:27:16.154 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:16.163+0000 7fbf8e9b8700 1 -- 172.21.15.52:0/3618380786 wait complete. 2024-05-27T07:27:16.158 INFO:teuthology.orchestra.run.smithi052.stderr:dumped fsmap epoch 30 2024-05-27T07:27:16.351 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:27:16 smithi173 ceph-mon[37000]: pgmap v42: 129 pgs: 129 active+clean; 1.4 MiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 5.3 KiB/s rd, 511 B/s wr, 10 op/s 2024-05-27T07:27:16.439 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:27:16 smithi105 ceph-mon[35977]: pgmap v42: 129 pgs: 129 active+clean; 1.4 MiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 5.3 KiB/s rd, 511 B/s wr, 10 op/s 2024-05-27T07:27:16.479 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:27:16 smithi052 ceph-mon[31955]: pgmap v42: 129 pgs: 129 active+clean; 1.4 MiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 5.3 KiB/s rd, 511 B/s wr, 10 op/s 2024-05-27T07:27:16.919 DEBUG:teuthology.orchestra.run.smithi052:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:38f9d900c83c63799fbdbe61acc9a11b0d3554a6 shell --fsid 73df2636-1bf8-11ef-bc9b-c7b262605968 -- ceph tell mds.1:4 damage ls 2024-05-27T07:27:17.208 INFO:teuthology.orchestra.run.smithi052.stderr:Inferring config /var/lib/ceph/73df2636-1bf8-11ef-bc9b-c7b262605968/mon.a/config 2024-05-27T07:27:17.229 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:27:17 smithi052 ceph-mon[31955]: from='client.? 172.21.15.52:0/3618380786' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-27T07:27:17.439 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:27:17 smithi105 ceph-mon[35977]: from='client.? 172.21.15.52:0/3618380786' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-27T07:27:17.601 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:27:17 smithi173 ceph-mon[37000]: from='client.? 172.21.15.52:0/3618380786' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-27T07:27:18.439 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:27:18 smithi105 ceph-mon[35977]: pgmap v43: 129 pgs: 129 active+clean; 1.4 MiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 5.3 KiB/s rd, 511 B/s wr, 10 op/s 2024-05-27T07:27:18.479 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:27:18 smithi052 ceph-mon[31955]: pgmap v43: 129 pgs: 129 active+clean; 1.4 MiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 5.3 KiB/s rd, 511 B/s wr, 10 op/s 2024-05-27T07:27:18.601 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:27:18 smithi173 ceph-mon[37000]: pgmap v43: 129 pgs: 129 active+clean; 1.4 MiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 5.3 KiB/s rd, 511 B/s wr, 10 op/s 2024-05-27T07:27:18.815 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:27:18.824+0000 7f115c8fc640 1 -- 172.21.15.52:0/2135755610 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f1150095a50 msgr2=0x7f1150095e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:27:18.815 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:27:18.824+0000 7f115c8fc640 1 --2- 172.21.15.52:0/2135755610 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f1150095a50 0x7f1150095e60 secure :-1 s=READY pgs=518 cs=0 l=1 rev1=1 crypto rx=0x7f114400d910 tx=0x7f114402c380 comp rx=0 tx=0).stop 2024-05-27T07:27:18.815 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:27:18.824+0000 7f115c8fc640 1 -- 172.21.15.52:0/2135755610 shutdown_connections 2024-05-27T07:27:18.815 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:27:18.824+0000 7f115c8fc640 1 --2- 172.21.15.52:0/2135755610 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f11500a0290 0x7f11500a26b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:27:18.815 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:27:18.824+0000 7f115c8fc640 1 --2- 172.21.15.52:0/2135755610 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f1150095a50 0x7f1150095e60 unknown :-1 s=CLOSED pgs=518 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:27:18.815 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:27:18.824+0000 7f115c8fc640 1 --2- 172.21.15.52:0/2135755610 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f115009b930 0x7f1150095510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:27:18.815 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:27:18.824+0000 7f115c8fc640 1 -- 172.21.15.52:0/2135755610 >> 172.21.15.52:0/2135755610 conn(0x7f115000aff0 msgr2=0x7f115000b430 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:27:18.815 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:27:18.824+0000 7f115c8fc640 1 -- 172.21.15.52:0/2135755610 shutdown_connections 2024-05-27T07:27:18.815 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:27:18.824+0000 7f115c8fc640 1 -- 172.21.15.52:0/2135755610 wait complete. 2024-05-27T07:27:18.817 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:27:18.825+0000 7f115c8fc640 1 Processor -- start 2024-05-27T07:27:18.817 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:27:18.825+0000 7f115c8fc640 1 -- start start 2024-05-27T07:27:18.817 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:27:18.826+0000 7f115c8fc640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f1150095a50 0x7f115012b430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:27:18.817 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:27:18.826+0000 7f115c8fc640 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f115009b930 0x7f115012b990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:27:18.817 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:27:18.826+0000 7f115c8fc640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f11500a0290 0x7f115012fc50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:27:18.818 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:27:18.826+0000 7f115c8fc640 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x7f11500a4a30 con 0x7f115009b930 2024-05-27T07:27:18.818 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:27:18.826+0000 7f115c8fc640 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f11500a48b0 con 0x7f11500a0290 2024-05-27T07:27:18.818 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:27:18.826+0000 7f115c8fc640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f11500a4bb0 con 0x7f1150095a50 2024-05-27T07:27:18.818 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:27:18.826+0000 7f1156ffd640 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f115009b930 0x7f115012b990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:27:18.818 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:27:18.826+0000 7f1156ffd640 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f115009b930 0x7f115012b990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.52:51694/0 (socket says 172.21.15.52:51694) 2024-05-27T07:27:18.818 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:27:18.826+0000 7f1156ffd640 1 -- 172.21.15.52:0/1196328276 learned_addr learned my addr 172.21.15.52:0/1196328276 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-05-27T07:27:18.818 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:27:18.826+0000 7f1157fff640 1 --2- 172.21.15.52:0/1196328276 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f11500a0290 0x7f115012fc50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:27:18.818 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:27:18.826+0000 7f11577fe640 1 --2- 172.21.15.52:0/1196328276 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f1150095a50 0x7f115012b430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:27:18.818 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:27:18.826+0000 7f1156ffd640 1 -- 172.21.15.52:0/1196328276 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f1150095a50 msgr2=0x7f115012b430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:27:18.818 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:27:18.826+0000 7f1156ffd640 1 --2- 172.21.15.52:0/1196328276 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f1150095a50 0x7f115012b430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:27:18.818 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:27:18.826+0000 7f1156ffd640 1 -- 172.21.15.52:0/1196328276 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f11500a0290 msgr2=0x7f115012fc50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:27:18.819 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:27:18.826+0000 7f1156ffd640 1 --2- 172.21.15.52:0/1196328276 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f11500a0290 0x7f115012fc50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:27:18.819 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:27:18.826+0000 7f1156ffd640 1 -- 172.21.15.52:0/1196328276 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1150130290 con 0x7f115009b930 2024-05-27T07:27:18.819 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:27:18.826+0000 7f1157fff640 1 --2- 172.21.15.52:0/1196328276 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f11500a0290 0x7f115012fc50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-27T07:27:18.819 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:27:18.826+0000 7f1156ffd640 1 --2- 172.21.15.52:0/1196328276 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f115009b930 0x7f115012b990 secure :-1 s=READY pgs=519 cs=0 l=1 rev1=1 crypto rx=0x7f1144009230 tx=0x7f1144034a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:27:18.819 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:27:18.827+0000 7f1154ff9640 1 -- 172.21.15.52:0/1196328276 <== mon.0 v2:172.21.15.52:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1144034e40 con 0x7f115009b930 2024-05-27T07:27:18.820 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:27:18.827+0000 7f1154ff9640 1 -- 172.21.15.52:0/1196328276 <== mon.0 v2:172.21.15.52:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f114402cd80 con 0x7f115009b930 2024-05-27T07:27:18.820 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:27:18.827+0000 7f115c8fc640 1 -- 172.21.15.52:0/1196328276 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1150130520 con 0x7f115009b930 2024-05-27T07:27:18.820 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:27:18.827+0000 7f1154ff9640 1 -- 172.21.15.52:0/1196328276 <== mon.0 v2:172.21.15.52:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f114403c420 con 0x7f115009b930 2024-05-27T07:27:18.820 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:27:18.827+0000 7f115c8fc640 1 -- 172.21.15.52:0/1196328276 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1150130a60 con 0x7f115009b930 2024-05-27T07:27:18.822 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:27:18.832+0000 7f1164a14740 1 Processor -- start 2024-05-27T07:27:18.823 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:27:18.832+0000 7f1164a14740 1 -- start start 2024-05-27T07:27:18.823 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:27:18.832+0000 7f1164a14740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x56506859e1b0 0x56506893f480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:27:18.823 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:27:18.832+0000 7f1164a14740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5650685752f0 0x56506893f9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:27:18.823 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:27:18.832+0000 7f1164a14740 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x56506857d4d0 0x56506893ff00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:27:18.823 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:27:18.832+0000 7f1164a14740 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x5650688b1c30 con 0x56506857d4d0 2024-05-27T07:27:18.823 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:27:18.832+0000 7f1164a14740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x5650688a17d0 con 0x56506859e1b0 2024-05-27T07:27:18.823 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:27:18.832+0000 7f1164a14740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x5650688a5810 con 0x5650685752f0 2024-05-27T07:27:18.824 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:27:18.833+0000 7f1154ff9640 1 -- 172.21.15.52:0/1196328276 <== mon.0 v2:172.21.15.52:3300/0 4 ==== mgrmap(e 25) ==== 149936+0+0 (secure 0 0 0) 0x7f114400c050 con 0x7f115009b930 2024-05-27T07:27:18.824 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:27:18.833+0000 7f1157fff640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x56506859e1b0 0x56506893f480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:27:18.824 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:27:18.833+0000 7f1157fff640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x56506859e1b0 0x56506893f480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.52:35952/0 (socket says 172.21.15.52:35952) 2024-05-27T07:27:18.824 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:27:18.833+0000 7f1157fff640 1 -- 172.21.15.52:0/878460056 learned_addr learned my addr 172.21.15.52:0/878460056 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-05-27T07:27:18.825 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:27:18.833+0000 7f11577fe640 1 --2- 172.21.15.52:0/878460056 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5650685752f0 0x56506893f9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:27:18.825 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:27:18.833+0000 7f1156ffd640 1 --2- 172.21.15.52:0/878460056 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x56506857d4d0 0x56506893ff00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:27:18.825 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:27:18.833+0000 7f1157fff640 1 -- 172.21.15.52:0/878460056 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5650685752f0 msgr2=0x56506893f9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:27:18.825 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:27:18.833+0000 7f1157fff640 1 --2- 172.21.15.52:0/878460056 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5650685752f0 0x56506893f9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:27:18.825 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:27:18.833+0000 7f1157fff640 1 -- 172.21.15.52:0/878460056 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x56506857d4d0 msgr2=0x56506893ff00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:27:18.825 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:27:18.833+0000 7f1157fff640 1 --2- 172.21.15.52:0/878460056 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x56506857d4d0 0x56506893ff00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:27:18.825 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:27:18.833+0000 7f1157fff640 1 -- 172.21.15.52:0/878460056 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5650685b6210 con 0x56506859e1b0 2024-05-27T07:27:18.825 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:27:18.833+0000 7f1156ffd640 1 --2- 172.21.15.52:0/878460056 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x56506857d4d0 0x56506893ff00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-27T07:27:18.826 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:27:18.833+0000 7f1157fff640 1 --2- 172.21.15.52:0/878460056 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x56506859e1b0 0x56506893f480 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f115802e1e0 tx=0x7f1158068f80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=abf8dcc48eeb1be4 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:27:18.826 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:27:18.834+0000 7f1141ffb640 1 -- 172.21.15.52:0/878460056 <== mon.2 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f115806c170 con 0x56506859e1b0 2024-05-27T07:27:18.826 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:27:18.834+0000 7f1141ffb640 1 -- 172.21.15.52:0/878460056 <== mon.2 v2:172.21.15.105:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f115806c310 con 0x56506859e1b0 2024-05-27T07:27:18.826 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:27:18.834+0000 7f1154ff9640 1 --2- 172.21.15.52:0/1196328276 >> [v2:172.21.15.52:6800/1822318384,v1:172.21.15.52:6801/1822318384] conn(0x7f11480b2b90 0x7f11480b50a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:27:18.826 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:27:18.834+0000 7f11577fe640 1 --2- 172.21.15.52:0/1196328276 >> [v2:172.21.15.52:6800/1822318384,v1:172.21.15.52:6801/1822318384] conn(0x7f11480b2b90 0x7f11480b50a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:27:18.826 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:27:18.834+0000 7f1154ff9640 1 -- 172.21.15.52:0/1196328276 <== mon.0 v2:172.21.15.52:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9379+0+0 (secure 0 0 0) 0x7f1144102070 con 0x7f115009b930 2024-05-27T07:27:18.826 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:27:18.834+0000 7f1141ffb640 1 -- 172.21.15.52:0/878460056 <== mon.2 v2:172.21.15.105:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f115807e3c0 con 0x56506859e1b0 2024-05-27T07:27:18.827 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:27:18.835+0000 7f1164a14740 1 -- 172.21.15.52:0/878460056 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x56506859e1b0 msgr2=0x56506893f480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:27:18.827 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:27:18.835+0000 7f11577fe640 1 --2- 172.21.15.52:0/1196328276 >> [v2:172.21.15.52:6800/1822318384,v1:172.21.15.52:6801/1822318384] conn(0x7f11480b2b90 0x7f11480b50a0 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x56506893d080 tx=0x7f114c005f90 comp rx=0 tx=0).ready entity=mgr.15333 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:27:18.827 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:27:18.835+0000 7f1164a14740 1 --2- 172.21.15.52:0/878460056 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x56506859e1b0 0x56506893f480 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f115802e1e0 tx=0x7f1158068f80 comp rx=0 tx=0).stop 2024-05-27T07:27:18.827 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:27:18.835+0000 7f1164a14740 1 -- 172.21.15.52:0/878460056 shutdown_connections 2024-05-27T07:27:18.827 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:27:18.835+0000 7f1164a14740 1 --2- 172.21.15.52:0/878460056 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x56506857d4d0 0x56506893ff00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:27:18.827 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:27:18.835+0000 7f1164a14740 1 --2- 172.21.15.52:0/878460056 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5650685752f0 0x56506893f9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:27:18.828 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:27:18.835+0000 7f1164a14740 1 --2- 172.21.15.52:0/878460056 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x56506859e1b0 0x56506893f480 unknown :-1 s=CLOSED pgs=75 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:27:18.828 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:27:18.835+0000 7f1164a14740 1 -- 172.21.15.52:0/878460056 >> 172.21.15.52:0/878460056 conn(0x565068592300 msgr2=0x56506893bd40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:27:18.828 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:27:18.835+0000 7f1164a14740 1 -- 172.21.15.52:0/878460056 shutdown_connections 2024-05-27T07:27:18.828 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:27:18.835+0000 7f1164a14740 1 -- 172.21.15.52:0/878460056 wait complete. 2024-05-27T07:27:18.828 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:27:18.836+0000 7f1164a14740 1 -- start start 2024-05-27T07:27:18.828 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:27:18.836+0000 7f1164a14740 1 Processor -- start 2024-05-27T07:27:18.831 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:27:18.837+0000 7f1164a14740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5650685752f0 0x56506894ec50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:27:18.831 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:27:18.837+0000 7f1164a14740 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x56506857d4d0 0x56506894f1b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:27:18.831 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:27:18.837+0000 7f1164a14740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x56506859e1b0 0x56506894a030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:27:18.831 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:27:18.837+0000 7f1156ffd640 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x56506857d4d0 0x56506894f1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:27:18.831 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:27:18.837+0000 7f1156ffd640 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x56506857d4d0 0x56506894f1b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.52:51714/0 (socket says 172.21.15.52:51714) 2024-05-27T07:27:18.831 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:27:18.837+0000 7f11577fe640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5650685752f0 0x56506894ec50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:27:18.831 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:27:18.837+0000 7f11577fe640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5650685752f0 0x56506894ec50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.173:3300/0 says I am v2:172.21.15.52:52312/0 (socket says 172.21.15.52:52312) 2024-05-27T07:27:18.832 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:27:18.837+0000 7f1156ffd640 1 -- 172.21.15.52:0/2444129080 learned_addr learned my addr 172.21.15.52:0/2444129080 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-05-27T07:27:18.832 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:27:18.838+0000 7f1164a14740 1 -- 172.21.15.52:0/2444129080 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x565068587790 con 0x56506857d4d0 2024-05-27T07:27:18.832 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:27:18.838+0000 7f1164a14740 1 -- 172.21.15.52:0/2444129080 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x56506885d890 con 0x56506859e1b0 2024-05-27T07:27:18.832 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:27:18.838+0000 7f1164a14740 1 -- 172.21.15.52:0/2444129080 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x565068942650 con 0x5650685752f0 2024-05-27T07:27:18.832 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:27:18.838+0000 7f1157fff640 1 --2- 172.21.15.52:0/2444129080 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x56506859e1b0 0x56506894a030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:27:18.832 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:27:18.838+0000 7f1164a14740 10 client.0 _ll_register_callbacks cb 0x5650685b5a00 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T07:27:18.832 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:27:18.838+0000 7f1157fff640 1 -- 172.21.15.52:0/2444129080 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5650685752f0 msgr2=0x56506894ec50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:27:18.832 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:27:18.838+0000 7f1157fff640 1 --2- 172.21.15.52:0/2444129080 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5650685752f0 0x56506894ec50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:27:18.832 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:27:18.838+0000 7f1157fff640 1 -- 172.21.15.52:0/2444129080 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x56506857d4d0 msgr2=0x56506894f1b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:27:18.832 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:27:18.838+0000 7f1157fff640 1 --2- 172.21.15.52:0/2444129080 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x56506857d4d0 0x56506894f1b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:27:18.832 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:27:18.838+0000 7f1157fff640 1 -- 172.21.15.52:0/2444129080 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({osdmap=0}) -- 0x565068682620 con 0x56506859e1b0 2024-05-27T07:27:18.832 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:27:18.838+0000 7f1164a14740 1 -- 172.21.15.52:0/2444129080 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f115806c310 con 0x56506859e1b0 2024-05-27T07:27:18.833 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:27:18.838+0000 7f1156ffd640 1 --2- 172.21.15.52:0/2444129080 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x56506857d4d0 0x56506894f1b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-27T07:27:18.833 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:27:18.838+0000 7f1157fff640 1 --2- 172.21.15.52:0/2444129080 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x56506859e1b0 0x56506894a030 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f1158050b30 tx=0x7f1158069740 comp rx=0 tx=0).ready entity=mon.2 client_cookie=61fd44316d1c75eb server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:27:18.833 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:27:18.838+0000 7f1136ffd640 10 client.25481 ms_handle_connect on v2:172.21.15.105:3300/0 2024-05-27T07:27:18.833 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:27:18.839+0000 7f1136ffd640 1 -- 172.21.15.52:0/2444129080 <== mon.2 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f115806c170 con 0x56506859e1b0 2024-05-27T07:27:18.833 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:27:18.839+0000 7f1136ffd640 1 -- 172.21.15.52:0/2444129080 <== mon.2 v2:172.21.15.105:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 9379+0+0 (secure 0 0 0) 0x7f115808a430 con 0x56506859e1b0 2024-05-27T07:27:18.833 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:27:18.839+0000 7f1136ffd640 1 -- 172.21.15.52:0/2444129080 <== mon.2 v2:172.21.15.105:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7f115808a630 con 0x56506859e1b0 2024-05-27T07:27:18.833 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:27:18.839+0000 7f1164a14740 10 client.25481 fetch_fsmap learned FSMap version 30 2024-05-27T07:27:18.833 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:27:18.839+0000 7f1164a14740 1 -- 172.21.15.52:0/2444129080 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({fsmap=30}) -- 0x565068969740 con 0x56506859e1b0 2024-05-27T07:27:18.833 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:27:18.840+0000 7f1136ffd640 1 -- 172.21.15.52:0/2444129080 <== mon.2 v2:172.21.15.105:3300/0 4 ==== fsmap(e 30) ==== 8362+0+0 (secure 0 0 0) 0x7f115808a7d0 con 0x56506859e1b0 2024-05-27T07:27:18.833 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:27:18.840+0000 7f1164a14740 10 client.25481 fetch_fsmap finished waiting for FSMap version 30 2024-05-27T07:27:18.833 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:27:18.840+0000 7f1164a14740 10 client.25481 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-05-27T07:27:18.833 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:27:18.840+0000 7f1164a14740 20 client.25481 populate_metadata read hostname 'smithi052' 2024-05-27T07:27:18.833 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:27:18.840+0000 7f1164a14740 1 --2- 172.21.15.52:0/2444129080 >> [v2:172.21.15.105:6838/500348151,v1:172.21.15.105:6839/500348151] conn(0x56506855cd80 0x56506896caa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:27:18.834 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:27:18.840+0000 7f1164a14740 4 client.25481 mds_command: new command op to 24527 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-27T07:27:18.834 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:27:18.840+0000 7f1164a14740 1 -- 172.21.15.52:0/2444129080 --> [v2:172.21.15.105:6838/500348151,v1:172.21.15.105:6839/500348151] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5650688b3b80 con 0x56506855cd80 2024-05-27T07:27:18.834 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:27:18.840+0000 7f11577fe640 1 --2- 172.21.15.52:0/2444129080 >> [v2:172.21.15.105:6838/500348151,v1:172.21.15.105:6839/500348151] conn(0x56506855cd80 0x56506896caa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:27:18.834 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:27:18.840+0000 7f1136ffd640 10 client.25481 ms_handle_connect on v2:172.21.15.105:6838/500348151 2024-05-27T07:27:18.834 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:27:18.840+0000 7f11577fe640 1 --2- 172.21.15.52:0/2444129080 >> [v2:172.21.15.105:6838/500348151,v1:172.21.15.105:6839/500348151] conn(0x56506855cd80 0x56506896caa0 crc :-1 s=READY pgs=63 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=d2dec4fcf338e4c server_cookie=9d65153ab917c10c in_seq=0 out_seq=0 2024-05-27T07:27:18.835 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:27:18.841+0000 7f1136ffd640 1 -- 172.21.15.52:0/2444129080 <== mds.0 v2:172.21.15.105:6838/500348151 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24041 (crc 0 0 0) 0x5650688b3b80 con 0x56506855cd80 2024-05-27T07:27:18.835 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:27:18.841+0000 7f1136ffd640 10 client.25481 handle_command_reply: tid=0 2024-05-27T07:27:18.835 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:27:18.841+0000 7f1164a14740 1 client.25481 shutdown 2024-05-27T07:27:18.835 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:27:18.841+0000 7f1164a14740 1 -- 172.21.15.52:0/2444129080 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x56506859e1b0 msgr2=0x56506894a030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:27:18.835 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:27:18.841+0000 7f1164a14740 1 --2- 172.21.15.52:0/2444129080 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x56506859e1b0 0x56506894a030 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f1158050b30 tx=0x7f1158069740 comp rx=0 tx=0).stop 2024-05-27T07:27:18.835 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:27:18.842+0000 7f1164a14740 1 -- 172.21.15.52:0/2444129080 shutdown_connections 2024-05-27T07:27:18.835 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:27:18.842+0000 7f1164a14740 1 --2- 172.21.15.52:0/2444129080 >> [v2:172.21.15.105:6838/500348151,v1:172.21.15.105:6839/500348151] conn(0x56506855cd80 0x56506896caa0 crc :-1 s=READY pgs=63 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:27:18.835 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:27:18.842+0000 7f1164a14740 1 --2- 172.21.15.52:0/2444129080 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x56506859e1b0 0x56506894a030 unknown :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:27:18.835 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:27:18.842+0000 7f1164a14740 1 --2- 172.21.15.52:0/2444129080 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x56506857d4d0 0x56506894f1b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:27:18.836 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:27:18.842+0000 7f1136ffd640 0 client.25481 ms_handle_reset on v2:172.21.15.105:6838/500348151 2024-05-27T07:27:18.836 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:27:18.842+0000 7f1164a14740 1 --2- 172.21.15.52:0/2444129080 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5650685752f0 0x56506894ec50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:27:18.836 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:27:18.842+0000 7f1164a14740 1 -- 172.21.15.52:0/2444129080 >> 172.21.15.52:0/2444129080 conn(0x565068592300 msgr2=0x56506893f320 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:27:18.836 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:27:18.842+0000 7f1164a14740 1 -- 172.21.15.52:0/2444129080 shutdown_connections 2024-05-27T07:27:18.836 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:27:18.842+0000 7f1164a14740 1 -- 172.21.15.52:0/2444129080 wait complete. 2024-05-27T07:27:18.836 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:27:18.842+0000 7f1164a14740 20 client.25481 trim_cache size 0 max 16384 2024-05-27T07:27:18.864 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:27:18.872+0000 7f1164a14740 1 Processor -- start 2024-05-27T07:27:18.864 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:27:18.872+0000 7f1164a14740 1 -- start start 2024-05-27T07:27:18.864 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:27:18.873+0000 7f1164a14740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5650685752f0 0x56506893ae40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:27:18.864 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:27:18.873+0000 7f1164a14740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x56506857d4d0 0x56506894e620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:27:18.864 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:27:18.873+0000 7f1164a14740 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x56506859e1b0 0x565068949200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:27:18.864 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:27:18.873+0000 7f1164a14740 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x7f1150141dd0 con 0x56506859e1b0 2024-05-27T07:27:18.864 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:27:18.873+0000 7f1164a14740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f1150143910 con 0x56506857d4d0 2024-05-27T07:27:18.864 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:27:18.873+0000 7f1164a14740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f11501424a0 con 0x5650685752f0 2024-05-27T07:27:18.865 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:27:18.873+0000 7f1157fff640 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x56506859e1b0 0x565068949200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:27:18.865 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:27:18.873+0000 7f1157fff640 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x56506859e1b0 0x565068949200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.52:51722/0 (socket says 172.21.15.52:51722) 2024-05-27T07:27:18.865 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:27:18.873+0000 7f1156ffd640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x56506857d4d0 0x56506894e620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:27:18.865 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:27:18.873+0000 7f1157fff640 1 -- 172.21.15.52:0/3533835340 learned_addr learned my addr 172.21.15.52:0/3533835340 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-05-27T07:27:18.865 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:27:18.873+0000 7f11577fe640 1 --2- 172.21.15.52:0/3533835340 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5650685752f0 0x56506893ae40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:27:18.865 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:27:18.873+0000 7f1157fff640 1 -- 172.21.15.52:0/3533835340 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5650685752f0 msgr2=0x56506893ae40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:27:18.865 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:27:18.873+0000 7f1157fff640 1 --2- 172.21.15.52:0/3533835340 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5650685752f0 0x56506893ae40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:27:18.865 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:27:18.873+0000 7f1157fff640 1 -- 172.21.15.52:0/3533835340 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x56506857d4d0 msgr2=0x56506894e620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:27:18.865 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:27:18.873+0000 7f1157fff640 1 --2- 172.21.15.52:0/3533835340 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x56506857d4d0 0x56506894e620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:27:18.865 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:27:18.873+0000 7f1157fff640 1 -- 172.21.15.52:0/3533835340 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56506894f520 con 0x56506859e1b0 2024-05-27T07:27:18.867 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:27:18.874+0000 7f1157fff640 1 --2- 172.21.15.52:0/3533835340 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x56506859e1b0 0x565068949200 secure :-1 s=READY pgs=520 cs=0 l=1 rev1=1 crypto rx=0x7f1158000b60 tx=0x7f115808e000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=eb5356a92b734b5f server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:27:18.867 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:27:18.874+0000 7f11417fa640 1 -- 172.21.15.52:0/3533835340 <== mon.0 v2:172.21.15.52:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f115806c170 con 0x56506859e1b0 2024-05-27T07:27:18.867 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:27:18.874+0000 7f11417fa640 1 -- 172.21.15.52:0/3533835340 <== mon.0 v2:172.21.15.52:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f115806c310 con 0x56506859e1b0 2024-05-27T07:27:18.867 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:27:18.874+0000 7f11417fa640 1 -- 172.21.15.52:0/3533835340 <== mon.0 v2:172.21.15.52:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f115809e680 con 0x56506859e1b0 2024-05-27T07:27:18.867 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:27:18.876+0000 7f1164a14740 1 -- 172.21.15.52:0/3533835340 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x56506859e1b0 msgr2=0x565068949200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:27:18.867 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:27:18.876+0000 7f1164a14740 1 --2- 172.21.15.52:0/3533835340 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x56506859e1b0 0x565068949200 secure :-1 s=READY pgs=520 cs=0 l=1 rev1=1 crypto rx=0x7f1158000b60 tx=0x7f115808e000 comp rx=0 tx=0).stop 2024-05-27T07:27:18.867 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:27:18.876+0000 7f1164a14740 1 -- 172.21.15.52:0/3533835340 shutdown_connections 2024-05-27T07:27:18.867 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:27:18.876+0000 7f1164a14740 1 --2- 172.21.15.52:0/3533835340 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x56506859e1b0 0x565068949200 unknown :-1 s=CLOSED pgs=520 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:27:18.867 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:27:18.876+0000 7f1164a14740 1 --2- 172.21.15.52:0/3533835340 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x56506857d4d0 0x56506894e620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:27:18.868 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:27:18.876+0000 7f1164a14740 1 --2- 172.21.15.52:0/3533835340 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5650685752f0 0x56506893ae40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:27:18.868 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:27:18.876+0000 7f1164a14740 1 -- 172.21.15.52:0/3533835340 >> 172.21.15.52:0/3533835340 conn(0x565068592300 msgr2=0x5650689412b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:27:18.868 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:27:18.876+0000 7f1164a14740 1 -- 172.21.15.52:0/3533835340 shutdown_connections 2024-05-27T07:27:18.868 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:27:18.876+0000 7f1164a14740 1 -- 172.21.15.52:0/3533835340 wait complete. 2024-05-27T07:27:18.868 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:27:18.876+0000 7f1164a14740 1 -- start start 2024-05-27T07:27:18.872 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:27:18.876+0000 7f1164a14740 1 Processor -- start 2024-05-27T07:27:18.872 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:27:18.877+0000 7f1164a14740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5650685752f0 0x565068978e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:27:18.872 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:27:18.877+0000 7f1164a14740 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x56506857d4d0 0x565068979380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:27:18.872 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:27:18.877+0000 7f1164a14740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x56506859e1b0 0x5650689d2e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:27:18.872 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:27:18.877+0000 7f1164a14740 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x7f1150143240 con 0x56506857d4d0 2024-05-27T07:27:18.872 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:27:18.877+0000 7f1164a14740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f1150142b70 con 0x5650685752f0 2024-05-27T07:27:18.872 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:27:18.877+0000 7f1164a14740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f115013fdd0 con 0x56506859e1b0 2024-05-27T07:27:18.872 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:27:18.877+0000 7f1164a14740 10 client.0 _ll_register_callbacks cb 0x7f114c009590 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T07:27:18.872 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:27:18.877+0000 7f1156ffd640 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x56506857d4d0 0x565068979380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:27:18.873 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:27:18.877+0000 7f1156ffd640 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x56506857d4d0 0x565068979380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.52:51728/0 (socket says 172.21.15.52:51728) 2024-05-27T07:27:18.873 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:27:18.877+0000 7f1156ffd640 1 -- 172.21.15.52:0/2906052722 learned_addr learned my addr 172.21.15.52:0/2906052722 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-05-27T07:27:18.873 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:27:18.877+0000 7f11577fe640 1 --2- 172.21.15.52:0/2906052722 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5650685752f0 0x565068978e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:27:18.873 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:27:18.877+0000 7f1157fff640 1 --2- 172.21.15.52:0/2906052722 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x56506859e1b0 0x5650689d2e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:27:18.873 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:27:18.878+0000 7f11577fe640 1 -- 172.21.15.52:0/2906052722 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x56506859e1b0 msgr2=0x5650689d2e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:27:18.873 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:27:18.878+0000 7f11577fe640 1 --2- 172.21.15.52:0/2906052722 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x56506859e1b0 0x5650689d2e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:27:18.873 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:27:18.878+0000 7f11577fe640 1 -- 172.21.15.52:0/2906052722 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x56506857d4d0 msgr2=0x565068979380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:27:18.873 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:27:18.878+0000 7f11577fe640 1 --2- 172.21.15.52:0/2906052722 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x56506857d4d0 0x565068979380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:27:18.873 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:27:18.878+0000 7f11577fe640 1 -- 172.21.15.52:0/2906052722 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({osdmap=0}) -- 0x565068979bd0 con 0x5650685752f0 2024-05-27T07:27:18.873 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:27:18.878+0000 7f1156ffd640 1 --2- 172.21.15.52:0/2906052722 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x56506857d4d0 0x565068979380 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-27T07:27:18.873 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:27:18.878+0000 7f1157fff640 1 --2- 172.21.15.52:0/2906052722 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x56506859e1b0 0x5650689d2e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-27T07:27:18.873 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:27:18.878+0000 7f11577fe640 1 --2- 172.21.15.52:0/2906052722 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5650685752f0 0x565068978e40 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x56506896d0f0 tx=0x7f114c0046c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c5c5055d7cb5c0e1 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:27:18.874 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:27:18.878+0000 7f1136ffd640 10 client.0 ms_handle_connect on v2:172.21.15.105:3300/0 2024-05-27T07:27:18.874 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:27:18.878+0000 7f1136ffd640 1 -- 172.21.15.52:0/2906052722 <== mon.2 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f114c003cd0 con 0x5650685752f0 2024-05-27T07:27:18.874 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:27:18.878+0000 7f1136ffd640 1 -- 172.21.15.52:0/2906052722 <== mon.2 v2:172.21.15.105:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 9379+0+0 (secure 0 0 0) 0x7f114c02c390 con 0x5650685752f0 2024-05-27T07:27:18.874 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:27:18.878+0000 7f1164a14740 1 -- 172.21.15.52:0/2906052722 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f115806c310 con 0x5650685752f0 2024-05-27T07:27:18.874 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:27:18.879+0000 7f1136ffd640 1 -- 172.21.15.52:0/2906052722 <== mon.2 v2:172.21.15.105:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7f114c02c5b0 con 0x5650685752f0 2024-05-27T07:27:18.874 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:27:18.879+0000 7f1164a14740 10 client.25487 fetch_fsmap learned FSMap version 30 2024-05-27T07:27:18.874 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:27:18.879+0000 7f1164a14740 1 -- 172.21.15.52:0/2906052722 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x5650689d43d0 con 0x5650685752f0 2024-05-27T07:27:18.874 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:27:18.879+0000 7f1136ffd640 1 -- 172.21.15.52:0/2906052722 <== mon.2 v2:172.21.15.105:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f114c02cc00 con 0x5650685752f0 2024-05-27T07:27:18.874 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:27:18.879+0000 7f1136ffd640 1 -- 172.21.15.52:0/2906052722 <== mon.2 v2:172.21.15.105:3300/0 5 ==== fsmap(e 30) ==== 8362+0+0 (secure 0 0 0) 0x7f114c021840 con 0x5650685752f0 2024-05-27T07:27:18.875 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:27:18.879+0000 7f1164a14740 10 client.25487 fetch_fsmap finished waiting for FSMap version 30 2024-05-27T07:27:18.875 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:27:18.879+0000 7f1164a14740 10 client.25487 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-05-27T07:27:18.875 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:27:18.879+0000 7f1164a14740 20 client.25487 populate_metadata read hostname 'smithi052' 2024-05-27T07:27:18.875 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:27:18.879+0000 7f1164a14740 1 --2- 172.21.15.52:0/2906052722 >> [v2:172.21.15.105:6838/500348151,v1:172.21.15.105:6839/500348151] conn(0x5650685321c0 0x5650689d6b90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:27:18.875 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:27:18.879+0000 7f1164a14740 4 client.25487 mds_command: new command op to 24527 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-05-27T07:27:18.875 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:27:18.880+0000 7f1164a14740 1 -- 172.21.15.52:0/2906052722 --> [v2:172.21.15.105:6838/500348151,v1:172.21.15.105:6839/500348151] -- command(tid 0: {"prefix": "flush journal"}) -- 0x7f115806c310 con 0x5650685321c0 2024-05-27T07:27:18.875 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:27:18.880+0000 7f1156ffd640 1 --2- 172.21.15.52:0/2906052722 >> [v2:172.21.15.105:6838/500348151,v1:172.21.15.105:6839/500348151] conn(0x5650685321c0 0x5650689d6b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:27:18.875 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:27:18.880+0000 7f1136ffd640 10 client.25487 ms_handle_connect on v2:172.21.15.105:6838/500348151 2024-05-27T07:27:18.875 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:27:18.880+0000 7f1156ffd640 1 --2- 172.21.15.52:0/2906052722 >> [v2:172.21.15.105:6838/500348151,v1:172.21.15.105:6839/500348151] conn(0x5650685321c0 0x5650689d6b90 crc :-1 s=READY pgs=64 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=549f0f63dadbfc4e server_cookie=6b902af5ddaa03b0 in_seq=0 out_seq=0 2024-05-27T07:27:18.876 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:27:18.884+0000 7f1136ffd640 1 -- 172.21.15.52:0/2906052722 <== mds.0 v2:172.21.15.105:6838/500348151 1 ==== command_reply(tid 0: 0 ) ==== 8+0+25 (crc 0 0 0) 0x7f115806c310 con 0x5650685321c0 2024-05-27T07:27:18.877 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:27:18.884+0000 7f1136ffd640 10 client.25487 handle_command_reply: tid=0 2024-05-27T07:27:18.877 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:27:18.884+0000 7f1164a14740 1 client.25487 shutdown 2024-05-27T07:27:18.877 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:27:18.884+0000 7f1164a14740 1 -- 172.21.15.52:0/2906052722 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5650685752f0 msgr2=0x565068978e40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:27:18.877 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:27:18.884+0000 7f1164a14740 1 --2- 172.21.15.52:0/2906052722 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5650685752f0 0x565068978e40 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x56506896d0f0 tx=0x7f114c0046c0 comp rx=0 tx=0).stop 2024-05-27T07:27:18.877 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:27:18.884+0000 7f1164a14740 1 -- 172.21.15.52:0/2906052722 shutdown_connections 2024-05-27T07:27:18.877 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:27:18.884+0000 7f1164a14740 1 --2- 172.21.15.52:0/2906052722 >> [v2:172.21.15.105:6838/500348151,v1:172.21.15.105:6839/500348151] conn(0x5650685321c0 0x5650689d6b90 crc :-1 s=READY pgs=64 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:27:18.877 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:27:18.884+0000 7f1164a14740 1 --2- 172.21.15.52:0/2906052722 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x56506859e1b0 0x5650689d2e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:27:18.878 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:27:18.884+0000 7f1164a14740 1 --2- 172.21.15.52:0/2906052722 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x56506857d4d0 0x565068979380 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:27:18.878 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:27:18.884+0000 7f1164a14740 1 --2- 172.21.15.52:0/2906052722 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5650685752f0 0x565068978e40 unknown :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:27:18.878 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:27:18.884+0000 7f1164a14740 1 -- 172.21.15.52:0/2906052722 >> 172.21.15.52:0/2906052722 conn(0x565068592300 msgr2=0x565068948f60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:27:18.878 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:27:18.884+0000 7f1136ffd640 0 client.25487 ms_handle_reset on v2:172.21.15.105:6838/500348151 2024-05-27T07:27:18.878 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:27:18.884+0000 7f1164a14740 1 -- 172.21.15.52:0/2906052722 shutdown_connections 2024-05-27T07:27:18.878 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:27:18.884+0000 7f1164a14740 1 -- 172.21.15.52:0/2906052722 wait complete. 2024-05-27T07:27:18.879 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:27:18.884+0000 7f1164a14740 20 client.25487 trim_cache size 0 max 16384 2024-05-27T07:27:18.879 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stdout:{ 2024-05-27T07:27:18.879 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stdout: "return_code": 0 2024-05-27T07:27:18.879 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stdout:} 2024-05-27T07:27:18.881 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:27:18.885+0000 7f115c8fc640 1 -- 172.21.15.52:0/1196328276 >> [v2:172.21.15.52:6800/1822318384,v1:172.21.15.52:6801/1822318384] conn(0x7f11480b2b90 msgr2=0x7f11480b50a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:27:18.881 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:27:18.885+0000 7f115c8fc640 1 --2- 172.21.15.52:0/1196328276 >> [v2:172.21.15.52:6800/1822318384,v1:172.21.15.52:6801/1822318384] conn(0x7f11480b2b90 0x7f11480b50a0 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x56506893d080 tx=0x7f114c005f90 comp rx=0 tx=0).stop 2024-05-27T07:27:18.881 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:27:18.885+0000 7f115c8fc640 1 -- 172.21.15.52:0/1196328276 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f115009b930 msgr2=0x7f115012b990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:27:18.881 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:27:18.885+0000 7f115c8fc640 1 --2- 172.21.15.52:0/1196328276 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f115009b930 0x7f115012b990 secure :-1 s=READY pgs=519 cs=0 l=1 rev1=1 crypto rx=0x7f1144009230 tx=0x7f1144034a40 comp rx=0 tx=0).stop 2024-05-27T07:27:18.881 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:27:18.885+0000 7f115c8fc640 1 -- 172.21.15.52:0/1196328276 shutdown_connections 2024-05-27T07:27:18.881 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:27:18.885+0000 7f115c8fc640 1 --2- 172.21.15.52:0/1196328276 >> [v2:172.21.15.52:6800/1822318384,v1:172.21.15.52:6801/1822318384] conn(0x7f11480b2b90 0x7f11480b50a0 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:27:18.882 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:27:18.885+0000 7f115c8fc640 1 --2- 172.21.15.52:0/1196328276 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f11500a0290 0x7f115012fc50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:27:18.882 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:27:18.885+0000 7f115c8fc640 1 --2- 172.21.15.52:0/1196328276 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f115009b930 0x7f115012b990 unknown :-1 s=CLOSED pgs=519 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:27:18.882 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:27:18.885+0000 7f115c8fc640 1 --2- 172.21.15.52:0/1196328276 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f1150095a50 0x7f115012b430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:27:18.882 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:27:18.885+0000 7f115c8fc640 1 -- 172.21.15.52:0/1196328276 >> 172.21.15.52:0/1196328276 conn(0x7f115000aff0 msgr2=0x7f115009c1e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:27:18.882 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:27:18.885+0000 7f115c8fc640 1 -- 172.21.15.52:0/1196328276 shutdown_connections 2024-05-27T07:27:18.882 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:27:18.886+0000 7f115c8fc640 1 -- 172.21.15.52:0/1196328276 wait complete. 2024-05-27T07:27:19.311 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:19.319+0000 7f5739b56700 1 -- 172.21.15.52:0/3691279108 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f5734108f30 msgr2=0x7f5734109340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:27:19.311 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:19.319+0000 7f5739b56700 1 --2- 172.21.15.52:0/3691279108 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f5734108f30 0x7f5734109340 secure :-1 s=READY pgs=521 cs=0 l=1 rev1=1 crypto rx=0x7f57280099c0 tx=0x7f5728009cd0 comp rx=0 tx=0).stop 2024-05-27T07:27:19.312 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:19.319+0000 7f5739b56700 1 -- 172.21.15.52:0/3691279108 shutdown_connections 2024-05-27T07:27:19.312 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:19.319+0000 7f5739b56700 1 --2- 172.21.15.52:0/3691279108 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f5734109880 0x7f57341111e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:27:19.312 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:19.319+0000 7f5739b56700 1 --2- 172.21.15.52:0/3691279108 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f5734104600 0x7f5734104a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:27:19.312 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:19.319+0000 7f5739b56700 1 --2- 172.21.15.52:0/3691279108 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f5734108f30 0x7f5734109340 unknown :-1 s=CLOSED pgs=521 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:27:19.312 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:19.319+0000 7f5739b56700 1 -- 172.21.15.52:0/3691279108 >> 172.21.15.52:0/3691279108 conn(0x7f573407af60 msgr2=0x7f573407b3a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:27:19.312 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:19.319+0000 7f5739b56700 1 -- 172.21.15.52:0/3691279108 shutdown_connections 2024-05-27T07:27:19.312 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:19.319+0000 7f5739b56700 1 -- 172.21.15.52:0/3691279108 wait complete. 2024-05-27T07:27:19.312 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:19.320+0000 7f5739b56700 1 Processor -- start 2024-05-27T07:27:19.312 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:19.321+0000 7f5739b56700 1 -- start start 2024-05-27T07:27:19.314 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:19.321+0000 7f5739b56700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f5734104600 0x7f57341a2170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:27:19.314 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:19.321+0000 7f5739b56700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f5734108f30 0x7f57341a26b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:27:19.314 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:19.321+0000 7f5739b56700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f5734109880 0x7f57341ad1f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:27:19.314 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:19.321+0000 7f5739b56700 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x7f5734114490 con 0x7f5734109880 2024-05-27T07:27:19.314 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:19.321+0000 7f5739b56700 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f5734114310 con 0x7f5734104600 2024-05-27T07:27:19.314 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:19.321+0000 7f5739b56700 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f5734114610 con 0x7f5734108f30 2024-05-27T07:27:19.315 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:19.321+0000 7f5733fff700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f5734109880 0x7f57341ad1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:27:19.315 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:19.321+0000 7f5732ffd700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f5734108f30 0x7f57341a26b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:27:19.315 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:19.321+0000 7f5733fff700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f5734109880 0x7f57341ad1f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.52:51740/0 (socket says 172.21.15.52:51740) 2024-05-27T07:27:19.315 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:19.321+0000 7f5733fff700 1 -- 172.21.15.52:0/1762830157 learned_addr learned my addr 172.21.15.52:0/1762830157 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-05-27T07:27:19.315 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:19.321+0000 7f57337fe700 1 --2- 172.21.15.52:0/1762830157 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f5734104600 0x7f57341a2170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:27:19.315 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:19.322+0000 7f5733fff700 1 -- 172.21.15.52:0/1762830157 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f5734108f30 msgr2=0x7f57341a26b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:27:19.315 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:19.322+0000 7f5733fff700 1 --2- 172.21.15.52:0/1762830157 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f5734108f30 0x7f57341a26b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:27:19.315 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:19.322+0000 7f5733fff700 1 -- 172.21.15.52:0/1762830157 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f5734104600 msgr2=0x7f57341a2170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:27:19.315 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:19.322+0000 7f5733fff700 1 --2- 172.21.15.52:0/1762830157 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f5734104600 0x7f57341a2170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:27:19.315 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:19.322+0000 7f5733fff700 1 -- 172.21.15.52:0/1762830157 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f57341ad730 con 0x7f5734109880 2024-05-27T07:27:19.315 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:19.322+0000 7f5733fff700 1 --2- 172.21.15.52:0/1762830157 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f5734109880 0x7f57341ad1f0 secure :-1 s=READY pgs=522 cs=0 l=1 rev1=1 crypto rx=0x7f572c00c370 tx=0x7f572c00c680 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:27:19.315 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:19.322+0000 7f5730ff9700 1 -- 172.21.15.52:0/1762830157 <== mon.0 v2:172.21.15.52:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f572c00ce20 con 0x7f5734109880 2024-05-27T07:27:19.315 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:19.322+0000 7f5730ff9700 1 -- 172.21.15.52:0/1762830157 <== mon.0 v2:172.21.15.52:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f572c0090d0 con 0x7f5734109880 2024-05-27T07:27:19.316 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:19.323+0000 7f5730ff9700 1 -- 172.21.15.52:0/1762830157 <== mon.0 v2:172.21.15.52:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f572c0047c0 con 0x7f5734109880 2024-05-27T07:27:19.316 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:19.323+0000 7f5739b56700 1 -- 172.21.15.52:0/1762830157 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f57341ad900 con 0x7f5734109880 2024-05-27T07:27:19.316 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:19.323+0000 7f5739b56700 1 -- 172.21.15.52:0/1762830157 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f57341adca0 con 0x7f5734109880 2024-05-27T07:27:19.316 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:19.326+0000 7f5730ff9700 1 -- 172.21.15.52:0/1762830157 <== mon.0 v2:172.21.15.52:3300/0 4 ==== mgrmap(e 25) ==== 149936+0+0 (secure 0 0 0) 0x7f572c01a020 con 0x7f5734109880 2024-05-27T07:27:19.317 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:19.328+0000 7f5730ff9700 1 --2- 172.21.15.52:0/1762830157 >> [v2:172.21.15.52:6800/1822318384,v1:172.21.15.52:6801/1822318384] conn(0x7f571c0b2c20 0x7f571c0b5130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:27:19.318 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:19.328+0000 7f5730ff9700 1 -- 172.21.15.52:0/1762830157 <== mon.0 v2:172.21.15.52:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9379+0+0 (secure 0 0 0) 0x7f572c0e2460 con 0x7f5734109880 2024-05-27T07:27:19.318 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:19.328+0000 7f57337fe700 1 --2- 172.21.15.52:0/1762830157 >> [v2:172.21.15.52:6800/1822318384,v1:172.21.15.52:6801/1822318384] conn(0x7f571c0b2c20 0x7f571c0b5130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:27:19.318 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:19.328+0000 7f57337fe700 1 --2- 172.21.15.52:0/1762830157 >> [v2:172.21.15.52:6800/1822318384,v1:172.21.15.52:6801/1822318384] conn(0x7f571c0b2c20 0x7f571c0b5130 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f5724005950 tx=0x7f57240058e0 comp rx=0 tx=0).ready entity=mgr.15333 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:27:19.319 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:19.328+0000 7f574856bb80 1 Processor -- start 2024-05-27T07:27:19.319 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:19.328+0000 7f574856bb80 1 -- start start 2024-05-27T07:27:19.320 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:19.329+0000 7f574856bb80 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55e5bf5d6e40 0x55e5bf75a710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:27:19.320 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:19.329+0000 7f574856bb80 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55e5bf75ac50 0x55e5bf754a40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:27:19.321 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:19.329+0000 7f574856bb80 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55e5bf754f80 0x55e5bf755390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:27:19.321 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:19.329+0000 7f574856bb80 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x55e5bf683c90 con 0x55e5bf5d6e40 2024-05-27T07:27:19.321 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:19.329+0000 7f574856bb80 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x55e5bf673830 con 0x55e5bf754f80 2024-05-27T07:27:19.321 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:19.329+0000 7f574856bb80 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x55e5bf677870 con 0x55e5bf75ac50 2024-05-27T07:27:19.321 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:19.329+0000 7f57337fe700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55e5bf5d6e40 0x55e5bf75a710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:27:19.321 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:19.329+0000 7f57337fe700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55e5bf5d6e40 0x55e5bf75a710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.52:51752/0 (socket says 172.21.15.52:51752) 2024-05-27T07:27:19.321 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:19.329+0000 7f57337fe700 1 -- 172.21.15.52:0/3445513037 learned_addr learned my addr 172.21.15.52:0/3445513037 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-05-27T07:27:19.321 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:19.329+0000 7f5733fff700 1 --2- 172.21.15.52:0/3445513037 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55e5bf754f80 0x55e5bf755390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:27:19.321 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:19.330+0000 7f57337fe700 1 -- 172.21.15.52:0/3445513037 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55e5bf75ac50 msgr2=0x55e5bf754a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:27:19.321 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:19.330+0000 7f57337fe700 1 --2- 172.21.15.52:0/3445513037 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55e5bf75ac50 0x55e5bf754a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:27:19.321 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:19.330+0000 7f57337fe700 1 -- 172.21.15.52:0/3445513037 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55e5bf754f80 msgr2=0x55e5bf755390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:27:19.321 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:19.330+0000 7f57337fe700 1 --2- 172.21.15.52:0/3445513037 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55e5bf754f80 0x55e5bf755390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:27:19.321 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:19.330+0000 7f57337fe700 1 -- 172.21.15.52:0/3445513037 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55e5bf5c38d0 con 0x55e5bf5d6e40 2024-05-27T07:27:19.321 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:19.330+0000 7f5733fff700 1 --2- 172.21.15.52:0/3445513037 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55e5bf754f80 0x55e5bf755390 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-27T07:27:19.322 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:19.330+0000 7f57337fe700 1 --2- 172.21.15.52:0/3445513037 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55e5bf5d6e40 0x55e5bf75a710 secure :-1 s=READY pgs=523 cs=0 l=1 rev1=1 crypto rx=0x7f572400fd70 tx=0x7f5724012000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ca18dbdfa3b64ce5 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:27:19.323 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:19.330+0000 7f5719ffb700 1 -- 172.21.15.52:0/3445513037 <== mon.0 v2:172.21.15.52:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5724022070 con 0x55e5bf5d6e40 2024-05-27T07:27:19.323 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:19.330+0000 7f5719ffb700 1 -- 172.21.15.52:0/3445513037 <== mon.0 v2:172.21.15.52:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f5724011c60 con 0x55e5bf5d6e40 2024-05-27T07:27:19.323 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:19.330+0000 7f5719ffb700 1 -- 172.21.15.52:0/3445513037 <== mon.0 v2:172.21.15.52:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f572401d5f0 con 0x55e5bf5d6e40 2024-05-27T07:27:19.323 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:19.331+0000 7f574856bb80 1 -- 172.21.15.52:0/3445513037 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55e5bf5d6e40 msgr2=0x55e5bf75a710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:27:19.323 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:19.331+0000 7f574856bb80 1 --2- 172.21.15.52:0/3445513037 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55e5bf5d6e40 0x55e5bf75a710 secure :-1 s=READY pgs=523 cs=0 l=1 rev1=1 crypto rx=0x7f572400fd70 tx=0x7f5724012000 comp rx=0 tx=0).stop 2024-05-27T07:27:19.323 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:19.331+0000 7f574856bb80 1 -- 172.21.15.52:0/3445513037 shutdown_connections 2024-05-27T07:27:19.323 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:19.331+0000 7f574856bb80 1 --2- 172.21.15.52:0/3445513037 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55e5bf754f80 0x55e5bf755390 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:27:19.323 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:19.331+0000 7f574856bb80 1 --2- 172.21.15.52:0/3445513037 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55e5bf5d6e40 0x55e5bf75a710 unknown :-1 s=CLOSED pgs=523 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:27:19.323 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:19.331+0000 7f574856bb80 1 --2- 172.21.15.52:0/3445513037 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55e5bf75ac50 0x55e5bf754a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:27:19.323 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:19.331+0000 7f574856bb80 1 -- 172.21.15.52:0/3445513037 >> 172.21.15.52:0/3445513037 conn(0x55e5bf631af0 msgr2=0x55e5bf74f640 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:27:19.324 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:19.331+0000 7f574856bb80 1 -- 172.21.15.52:0/3445513037 shutdown_connections 2024-05-27T07:27:19.324 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:19.331+0000 7f574856bb80 1 -- 172.21.15.52:0/3445513037 wait complete. 2024-05-27T07:27:19.324 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:19.332+0000 7f574856bb80 1 -- start start 2024-05-27T07:27:19.324 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:19.332+0000 7f574856bb80 1 Processor -- start 2024-05-27T07:27:19.326 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:19.333+0000 7f574856bb80 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55e5bf5d6e40 0x55e5bf75e2b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:27:19.326 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:19.333+0000 7f574856bb80 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55e5bf754f80 0x55e5bf75e7f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:27:19.326 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:19.333+0000 7f574856bb80 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55e5bf75ac50 0x55e5bf75ed30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:27:19.327 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:19.333+0000 7f574856bb80 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x55e5bf579fa0 con 0x55e5bf5d6e40 2024-05-27T07:27:19.327 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:19.333+0000 7f574856bb80 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x55e5bf6157c0 con 0x55e5bf75ac50 2024-05-27T07:27:19.327 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:19.333+0000 7f574856bb80 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x55e5bf7617e0 con 0x55e5bf754f80 2024-05-27T07:27:19.327 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:19.333+0000 7f574856bb80 10 client.0 _ll_register_callbacks cb 0x55e5bf60e920 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T07:27:19.327 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:19.333+0000 7f57337fe700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55e5bf5d6e40 0x55e5bf75e2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:27:19.327 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:19.333+0000 7f57337fe700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55e5bf5d6e40 0x55e5bf75e2b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.52:51760/0 (socket says 172.21.15.52:51760) 2024-05-27T07:27:19.327 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:19.333+0000 7f57337fe700 1 -- 172.21.15.52:0/344317190 learned_addr learned my addr 172.21.15.52:0/344317190 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-05-27T07:27:19.327 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:19.333+0000 7f5733fff700 1 --2- 172.21.15.52:0/344317190 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55e5bf75ac50 0x55e5bf75ed30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:27:19.327 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:19.333+0000 7f5732ffd700 1 --2- 172.21.15.52:0/344317190 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55e5bf754f80 0x55e5bf75e7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:27:19.327 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:19.333+0000 7f5733fff700 1 -- 172.21.15.52:0/344317190 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55e5bf754f80 msgr2=0x55e5bf75e7f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:27:19.327 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:19.333+0000 7f5733fff700 1 --2- 172.21.15.52:0/344317190 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55e5bf754f80 0x55e5bf75e7f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:27:19.327 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:19.333+0000 7f5733fff700 1 -- 172.21.15.52:0/344317190 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55e5bf5d6e40 msgr2=0x55e5bf75e2b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:27:19.328 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:19.334+0000 7f5733fff700 1 --2- 172.21.15.52:0/344317190 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55e5bf5d6e40 0x55e5bf75e2b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:27:19.328 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:19.334+0000 7f5733fff700 1 -- 172.21.15.52:0/344317190 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55e5bf612690 con 0x55e5bf75ac50 2024-05-27T07:27:19.328 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:19.334+0000 7f5732ffd700 1 --2- 172.21.15.52:0/344317190 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55e5bf754f80 0x55e5bf75e7f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-27T07:27:19.328 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:19.334+0000 7f57337fe700 1 --2- 172.21.15.52:0/344317190 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55e5bf5d6e40 0x55e5bf75e2b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-27T07:27:19.328 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:19.334+0000 7f5733fff700 1 --2- 172.21.15.52:0/344317190 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55e5bf75ac50 0x55e5bf75ed30 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f572c0df6b0 tx=0x7f572c019fb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=5a58042ca7959f47 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:27:19.328 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:19.334+0000 7f5712ffd700 10 client.0 ms_handle_connect on v2:172.21.15.105:3300/0 2024-05-27T07:27:19.328 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:19.334+0000 7f5712ffd700 1 -- 172.21.15.52:0/344317190 <== mon.2 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f572c0a9940 con 0x55e5bf75ac50 2024-05-27T07:27:19.328 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:19.334+0000 7f574856bb80 1 -- 172.21.15.52:0/344317190 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f5724011c60 con 0x55e5bf75ac50 2024-05-27T07:27:19.328 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:19.334+0000 7f5712ffd700 1 -- 172.21.15.52:0/344317190 <== mon.2 v2:172.21.15.105:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 9379+0+0 (secure 0 0 0) 0x7f572c0b1e20 con 0x55e5bf75ac50 2024-05-27T07:27:19.328 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:19.334+0000 7f5712ffd700 1 -- 172.21.15.52:0/344317190 <== mon.2 v2:172.21.15.105:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7f572c0a9240 con 0x55e5bf75ac50 2024-05-27T07:27:19.328 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:19.334+0000 7f574856bb80 10 client.25499 fetch_fsmap learned FSMap version 30 2024-05-27T07:27:19.328 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:19.335+0000 7f574856bb80 1 -- 172.21.15.52:0/344317190 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x55e5bf76c170 con 0x55e5bf75ac50 2024-05-27T07:27:19.328 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:19.335+0000 7f5712ffd700 1 -- 172.21.15.52:0/344317190 <== mon.2 v2:172.21.15.105:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f5724011c60 con 0x55e5bf75ac50 2024-05-27T07:27:19.329 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:19.335+0000 7f5712ffd700 1 -- 172.21.15.52:0/344317190 <== mon.2 v2:172.21.15.105:3300/0 5 ==== fsmap(e 30) ==== 8362+0+0 (secure 0 0 0) 0x7f572c0ad120 con 0x55e5bf75ac50 2024-05-27T07:27:19.329 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:19.335+0000 7f574856bb80 10 client.25499 fetch_fsmap finished waiting for FSMap version 30 2024-05-27T07:27:19.329 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:19.335+0000 7f574856bb80 10 client.25499 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.c 2024-05-27T07:27:19.329 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:19.335+0000 7f574856bb80 20 client.25499 populate_metadata read hostname 'smithi052' 2024-05-27T07:27:19.329 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:19.335+0000 7f574856bb80 1 --2- 172.21.15.52:0/344317190 >> [v2:172.21.15.173:6836/1396363827,v1:172.21.15.173:6837/1396363827] conn(0x55e5bf76e700 0x55e5bf770b70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:27:19.329 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:19.335+0000 7f574856bb80 4 client.25499 mds_command: new command op to 24466 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-27T07:27:19.329 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:19.335+0000 7f574856bb80 1 -- 172.21.15.52:0/344317190 --> [v2:172.21.15.173:6836/1396363827,v1:172.21.15.173:6837/1396363827] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55e5bf7209f0 con 0x55e5bf76e700 2024-05-27T07:27:19.329 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:19.336+0000 7f57337fe700 1 --2- 172.21.15.52:0/344317190 >> [v2:172.21.15.173:6836/1396363827,v1:172.21.15.173:6837/1396363827] conn(0x55e5bf76e700 0x55e5bf770b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:27:19.330 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:19.336+0000 7f5712ffd700 10 client.25499 ms_handle_connect on v2:172.21.15.173:6836/1396363827 2024-05-27T07:27:19.330 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:19.336+0000 7f57337fe700 1 --2- 172.21.15.52:0/344317190 >> [v2:172.21.15.173:6836/1396363827,v1:172.21.15.173:6837/1396363827] conn(0x55e5bf76e700 0x55e5bf770b70 crc :-1 s=READY pgs=29 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=a047a231fcb6e867 server_cookie=cfc3bfcb39bdb748 in_seq=0 out_seq=0 2024-05-27T07:27:19.330 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:19.337+0000 7f5712ffd700 1 -- 172.21.15.52:0/344317190 <== mds.4 v2:172.21.15.173:6836/1396363827 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24041 (crc 0 0 0) 0x55e5bf7209f0 con 0x55e5bf76e700 2024-05-27T07:27:19.330 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:19.337+0000 7f5712ffd700 10 client.25499 handle_command_reply: tid=0 2024-05-27T07:27:19.330 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:19.337+0000 7f574856bb80 1 client.25499 shutdown 2024-05-27T07:27:19.330 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:19.337+0000 7f574856bb80 1 -- 172.21.15.52:0/344317190 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55e5bf75ac50 msgr2=0x55e5bf75ed30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:27:19.330 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:19.337+0000 7f574856bb80 1 --2- 172.21.15.52:0/344317190 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55e5bf75ac50 0x55e5bf75ed30 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f572c0df6b0 tx=0x7f572c019fb0 comp rx=0 tx=0).stop 2024-05-27T07:27:19.330 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:19.337+0000 7f574856bb80 1 -- 172.21.15.52:0/344317190 shutdown_connections 2024-05-27T07:27:19.331 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:19.337+0000 7f574856bb80 1 --2- 172.21.15.52:0/344317190 >> [v2:172.21.15.173:6836/1396363827,v1:172.21.15.173:6837/1396363827] conn(0x55e5bf76e700 0x55e5bf770b70 crc :-1 s=READY pgs=29 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:27:19.331 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:19.337+0000 7f574856bb80 1 --2- 172.21.15.52:0/344317190 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55e5bf75ac50 0x55e5bf75ed30 unknown :-1 s=CLOSED pgs=78 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:27:19.331 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:19.337+0000 7f574856bb80 1 --2- 172.21.15.52:0/344317190 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55e5bf5d6e40 0x55e5bf75e2b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:27:19.331 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:19.337+0000 7f574856bb80 1 --2- 172.21.15.52:0/344317190 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55e5bf754f80 0x55e5bf75e7f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:27:19.331 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:19.337+0000 7f574856bb80 1 -- 172.21.15.52:0/344317190 >> 172.21.15.52:0/344317190 conn(0x55e5bf631af0 msgr2=0x55e5bf753a30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:27:19.331 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:19.337+0000 7f5712ffd700 0 client.25499 ms_handle_reset on v2:172.21.15.173:6836/1396363827 2024-05-27T07:27:19.331 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:19.338+0000 7f574856bb80 1 -- 172.21.15.52:0/344317190 shutdown_connections 2024-05-27T07:27:19.331 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:19.338+0000 7f574856bb80 1 -- 172.21.15.52:0/344317190 wait complete. 2024-05-27T07:27:19.331 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:19.338+0000 7f574856bb80 20 client.25499 trim_cache size 0 max 16384 2024-05-27T07:27:19.358 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:19.367+0000 7f574856bb80 1 Processor -- start 2024-05-27T07:27:19.358 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:19.367+0000 7f574856bb80 1 -- start start 2024-05-27T07:27:19.359 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:19.367+0000 7f574856bb80 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55e5bf5d6e40 0x55e5bf754a40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:27:19.359 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:19.367+0000 7f574856bb80 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55e5bf754f80 0x55e5bf758720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:27:19.359 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:19.367+0000 7f574856bb80 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55e5bf75ac50 0x55e5bf76a880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:27:19.359 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:19.367+0000 7f574856bb80 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x7f57341ae340 con 0x55e5bf5d6e40 2024-05-27T07:27:19.359 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:19.367+0000 7f574856bb80 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f57341afe80 con 0x55e5bf754f80 2024-05-27T07:27:19.360 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:19.367+0000 7f574856bb80 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f57341aea10 con 0x55e5bf75ac50 2024-05-27T07:27:19.360 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:19.368+0000 7f57337fe700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55e5bf5d6e40 0x55e5bf754a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:27:19.360 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:19.368+0000 7f57337fe700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55e5bf5d6e40 0x55e5bf754a40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.52:51774/0 (socket says 172.21.15.52:51774) 2024-05-27T07:27:19.360 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:19.368+0000 7f57337fe700 1 -- 172.21.15.52:0/26963140 learned_addr learned my addr 172.21.15.52:0/26963140 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-05-27T07:27:19.360 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:19.368+0000 7f5733fff700 1 --2- 172.21.15.52:0/26963140 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55e5bf75ac50 0x55e5bf76a880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:27:19.360 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:19.368+0000 7f5732ffd700 1 --2- 172.21.15.52:0/26963140 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55e5bf754f80 0x55e5bf758720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:27:19.360 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:19.368+0000 7f57337fe700 1 -- 172.21.15.52:0/26963140 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55e5bf75ac50 msgr2=0x55e5bf76a880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:27:19.360 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:19.368+0000 7f57337fe700 1 --2- 172.21.15.52:0/26963140 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55e5bf75ac50 0x55e5bf76a880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:27:19.360 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:19.368+0000 7f57337fe700 1 -- 172.21.15.52:0/26963140 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55e5bf754f80 msgr2=0x55e5bf758720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:27:19.360 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:19.368+0000 7f57337fe700 1 --2- 172.21.15.52:0/26963140 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55e5bf754f80 0x55e5bf758720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:27:19.360 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:19.368+0000 7f57337fe700 1 -- 172.21.15.52:0/26963140 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55e5bf74fb70 con 0x55e5bf5d6e40 2024-05-27T07:27:19.360 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:19.368+0000 7f57337fe700 1 --2- 172.21.15.52:0/26963140 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55e5bf5d6e40 0x55e5bf754a40 secure :-1 s=READY pgs=524 cs=0 l=1 rev1=1 crypto rx=0x7f5724014040 tx=0x7f5724015000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c2718991ce3a907f server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:27:19.360 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:19.369+0000 7f57197fa700 1 -- 172.21.15.52:0/26963140 <== mon.0 v2:172.21.15.52:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5724022040 con 0x55e5bf5d6e40 2024-05-27T07:27:19.361 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:19.369+0000 7f57197fa700 1 -- 172.21.15.52:0/26963140 <== mon.0 v2:172.21.15.52:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f57240348f0 con 0x55e5bf5d6e40 2024-05-27T07:27:19.361 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:19.369+0000 7f57197fa700 1 -- 172.21.15.52:0/26963140 <== mon.0 v2:172.21.15.52:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f572401ea70 con 0x55e5bf5d6e40 2024-05-27T07:27:19.362 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:19.370+0000 7f574856bb80 1 -- 172.21.15.52:0/26963140 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55e5bf5d6e40 msgr2=0x55e5bf754a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:27:19.362 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:19.370+0000 7f574856bb80 1 --2- 172.21.15.52:0/26963140 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55e5bf5d6e40 0x55e5bf754a40 secure :-1 s=READY pgs=524 cs=0 l=1 rev1=1 crypto rx=0x7f5724014040 tx=0x7f5724015000 comp rx=0 tx=0).stop 2024-05-27T07:27:19.362 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:19.370+0000 7f574856bb80 1 -- 172.21.15.52:0/26963140 shutdown_connections 2024-05-27T07:27:19.362 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:19.370+0000 7f574856bb80 1 --2- 172.21.15.52:0/26963140 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55e5bf75ac50 0x55e5bf76a880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:27:19.362 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:19.370+0000 7f574856bb80 1 --2- 172.21.15.52:0/26963140 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55e5bf5d6e40 0x55e5bf754a40 unknown :-1 s=CLOSED pgs=524 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:27:19.362 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:19.370+0000 7f574856bb80 1 --2- 172.21.15.52:0/26963140 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55e5bf754f80 0x55e5bf758720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:27:19.362 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:19.370+0000 7f574856bb80 1 -- 172.21.15.52:0/26963140 >> 172.21.15.52:0/26963140 conn(0x55e5bf631af0 msgr2=0x55e5bf758010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:27:19.362 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:19.370+0000 7f574856bb80 1 -- 172.21.15.52:0/26963140 shutdown_connections 2024-05-27T07:27:19.362 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:19.370+0000 7f574856bb80 1 -- 172.21.15.52:0/26963140 wait complete. 2024-05-27T07:27:19.362 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:19.371+0000 7f574856bb80 1 -- start start 2024-05-27T07:27:19.363 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:19.371+0000 7f574856bb80 1 Processor -- start 2024-05-27T07:27:19.364 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:19.372+0000 7f574856bb80 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55e5bf5d6e40 0x55e5bf776b30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:27:19.364 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:19.372+0000 7f574856bb80 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55e5bf754f80 0x55e5bf766a70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:27:19.364 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:19.372+0000 7f574856bb80 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55e5bf75ac50 0x55e5bf7670c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:27:19.364 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:19.372+0000 7f574856bb80 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x7f57341af7b0 con 0x55e5bf754f80 2024-05-27T07:27:19.364 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:19.372+0000 7f574856bb80 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f57341af0e0 con 0x55e5bf5d6e40 2024-05-27T07:27:19.364 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:19.372+0000 7f574856bb80 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f5734073d50 con 0x55e5bf75ac50 2024-05-27T07:27:19.364 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:19.372+0000 7f57337fe700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55e5bf5d6e40 0x55e5bf776b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:27:19.364 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:19.372+0000 7f57337fe700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55e5bf5d6e40 0x55e5bf776b30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.52:36060/0 (socket says 172.21.15.52:36060) 2024-05-27T07:27:19.365 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:19.372+0000 7f57337fe700 1 -- 172.21.15.52:0/324304897 learned_addr learned my addr 172.21.15.52:0/324304897 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-05-27T07:27:19.365 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:19.372+0000 7f5733fff700 1 --2- 172.21.15.52:0/324304897 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55e5bf75ac50 0x55e5bf7670c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:27:19.365 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:19.372+0000 7f5732ffd700 1 --2- 172.21.15.52:0/324304897 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55e5bf754f80 0x55e5bf766a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:27:19.365 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:19.372+0000 7f574856bb80 10 client.0 _ll_register_callbacks cb 0x55e5bf60e920 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T07:27:19.365 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:19.372+0000 7f5733fff700 1 -- 172.21.15.52:0/324304897 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55e5bf5d6e40 msgr2=0x55e5bf776b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:27:19.365 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:19.372+0000 7f5733fff700 1 --2- 172.21.15.52:0/324304897 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55e5bf5d6e40 0x55e5bf776b30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:27:19.365 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:19.372+0000 7f5733fff700 1 -- 172.21.15.52:0/324304897 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55e5bf754f80 msgr2=0x55e5bf766a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:27:19.365 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:19.373+0000 7f5733fff700 1 --2- 172.21.15.52:0/324304897 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55e5bf754f80 0x55e5bf766a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:27:19.365 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:19.373+0000 7f5733fff700 1 -- 172.21.15.52:0/324304897 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55e5bf775370 con 0x55e5bf75ac50 2024-05-27T07:27:19.365 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:19.373+0000 7f5732ffd700 1 --2- 172.21.15.52:0/324304897 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55e5bf754f80 0x55e5bf766a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-27T07:27:19.365 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:19.373+0000 7f5733fff700 1 --2- 172.21.15.52:0/324304897 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55e5bf75ac50 0x55e5bf7670c0 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f572c0df7a0 tx=0x7f572c0a88b0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=681795d2a87db95d server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:27:19.365 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:19.373+0000 7f5712ffd700 10 client.0 ms_handle_connect on v2:172.21.15.173:3300/0 2024-05-27T07:27:19.365 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:19.373+0000 7f5712ffd700 1 -- 172.21.15.52:0/324304897 <== mon.1 v2:172.21.15.173:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f572c0b20a0 con 0x55e5bf75ac50 2024-05-27T07:27:19.366 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:19.373+0000 7f5712ffd700 1 -- 172.21.15.52:0/324304897 <== mon.1 v2:172.21.15.173:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 9379+0+0 (secure 0 0 0) 0x7f572c004bd0 con 0x55e5bf75ac50 2024-05-27T07:27:19.366 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:19.373+0000 7f574856bb80 1 -- 172.21.15.52:0/324304897 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f57240348f0 con 0x55e5bf75ac50 2024-05-27T07:27:19.368 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:19.374+0000 7f5712ffd700 1 -- 172.21.15.52:0/324304897 <== mon.1 v2:172.21.15.173:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7f572c0ad5f0 con 0x55e5bf75ac50 2024-05-27T07:27:19.368 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:19.374+0000 7f574856bb80 10 client.25438 fetch_fsmap learned FSMap version 30 2024-05-27T07:27:19.368 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:19.374+0000 7f574856bb80 1 -- 172.21.15.52:0/324304897 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x55e5bf7d8460 con 0x55e5bf75ac50 2024-05-27T07:27:19.368 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:19.374+0000 7f5712ffd700 1 -- 172.21.15.52:0/324304897 <== mon.1 v2:172.21.15.173:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f57240348f0 con 0x55e5bf75ac50 2024-05-27T07:27:19.368 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:19.374+0000 7f5712ffd700 1 -- 172.21.15.52:0/324304897 <== mon.1 v2:172.21.15.173:3300/0 5 ==== fsmap(e 30) ==== 8362+0+0 (secure 0 0 0) 0x7f572c0a4bb0 con 0x55e5bf75ac50 2024-05-27T07:27:19.368 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:19.374+0000 7f574856bb80 10 client.25438 fetch_fsmap finished waiting for FSMap version 30 2024-05-27T07:27:19.368 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:19.374+0000 7f574856bb80 10 client.25438 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.c 2024-05-27T07:27:19.368 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:19.374+0000 7f574856bb80 20 client.25438 populate_metadata read hostname 'smithi052' 2024-05-27T07:27:19.368 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:19.374+0000 7f574856bb80 1 --2- 172.21.15.52:0/324304897 >> [v2:172.21.15.173:6836/1396363827,v1:172.21.15.173:6837/1396363827] conn(0x55e5bf7d8b90 0x55e5bf7dafb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:27:19.368 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:19.375+0000 7f574856bb80 4 client.25438 mds_command: new command op to 24466 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-05-27T07:27:19.369 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:19.375+0000 7f574856bb80 1 -- 172.21.15.52:0/324304897 --> [v2:172.21.15.173:6836/1396363827,v1:172.21.15.173:6837/1396363827] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55e5bf685be0 con 0x55e5bf7d8b90 2024-05-27T07:27:19.369 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:19.375+0000 7f57337fe700 1 --2- 172.21.15.52:0/324304897 >> [v2:172.21.15.173:6836/1396363827,v1:172.21.15.173:6837/1396363827] conn(0x55e5bf7d8b90 0x55e5bf7dafb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:27:19.369 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:19.375+0000 7f5712ffd700 10 client.25438 ms_handle_connect on v2:172.21.15.173:6836/1396363827 2024-05-27T07:27:19.369 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:19.375+0000 7f57337fe700 1 --2- 172.21.15.52:0/324304897 >> [v2:172.21.15.173:6836/1396363827,v1:172.21.15.173:6837/1396363827] conn(0x55e5bf7d8b90 0x55e5bf7dafb0 crc :-1 s=READY pgs=30 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=b1f7e6534c881261 server_cookie=9c7a4993cf026cd5 in_seq=0 out_seq=0 2024-05-27T07:27:19.369 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:19.376+0000 7f5712ffd700 1 -- 172.21.15.52:0/324304897 <== mds.4 v2:172.21.15.173:6836/1396363827 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55e5bf685be0 con 0x55e5bf7d8b90 2024-05-27T07:27:19.369 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:19.376+0000 7f5712ffd700 10 client.25438 handle_command_reply: tid=0 2024-05-27T07:27:19.369 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:19.376+0000 7f574856bb80 1 client.25438 shutdown 2024-05-27T07:27:19.369 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:19.376+0000 7f574856bb80 1 -- 172.21.15.52:0/324304897 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55e5bf75ac50 msgr2=0x55e5bf7670c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:27:19.369 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:19.376+0000 7f574856bb80 1 --2- 172.21.15.52:0/324304897 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55e5bf75ac50 0x55e5bf7670c0 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f572c0df7a0 tx=0x7f572c0a88b0 comp rx=0 tx=0).stop 2024-05-27T07:27:19.369 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:19.376+0000 7f574856bb80 1 -- 172.21.15.52:0/324304897 shutdown_connections 2024-05-27T07:27:19.369 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:19.376+0000 7f574856bb80 1 --2- 172.21.15.52:0/324304897 >> [v2:172.21.15.173:6836/1396363827,v1:172.21.15.173:6837/1396363827] conn(0x55e5bf7d8b90 0x55e5bf7dafb0 crc :-1 s=READY pgs=30 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:27:19.369 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:19.376+0000 7f5712ffd700 0 client.25438 ms_handle_reset on v2:172.21.15.173:6836/1396363827 2024-05-27T07:27:19.369 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:19.376+0000 7f574856bb80 1 --2- 172.21.15.52:0/324304897 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55e5bf75ac50 0x55e5bf7670c0 unknown :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:27:19.370 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:19.376+0000 7f574856bb80 1 --2- 172.21.15.52:0/324304897 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55e5bf5d6e40 0x55e5bf776b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:27:19.370 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:19.376+0000 7f574856bb80 1 --2- 172.21.15.52:0/324304897 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55e5bf754f80 0x55e5bf766a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:27:19.370 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:19.376+0000 7f574856bb80 1 -- 172.21.15.52:0/324304897 >> 172.21.15.52:0/324304897 conn(0x55e5bf631af0 msgr2=0x55e5bf7496f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:27:19.370 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:19.376+0000 7f574856bb80 1 -- 172.21.15.52:0/324304897 shutdown_connections 2024-05-27T07:27:19.370 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:19.376+0000 7f574856bb80 1 -- 172.21.15.52:0/324304897 wait complete. 2024-05-27T07:27:19.370 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:19.377+0000 7f574856bb80 20 client.25438 trim_cache size 0 max 16384 2024-05-27T07:27:19.370 INFO:teuthology.orchestra.run.smithi052.stdout:[] 2024-05-27T07:27:19.371 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:19.377+0000 7f5739b56700 1 -- 172.21.15.52:0/1762830157 >> [v2:172.21.15.52:6800/1822318384,v1:172.21.15.52:6801/1822318384] conn(0x7f571c0b2c20 msgr2=0x7f571c0b5130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:27:19.371 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:19.377+0000 7f5739b56700 1 --2- 172.21.15.52:0/1762830157 >> [v2:172.21.15.52:6800/1822318384,v1:172.21.15.52:6801/1822318384] conn(0x7f571c0b2c20 0x7f571c0b5130 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f5724005950 tx=0x7f57240058e0 comp rx=0 tx=0).stop 2024-05-27T07:27:19.371 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:19.377+0000 7f5739b56700 1 -- 172.21.15.52:0/1762830157 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f5734109880 msgr2=0x7f57341ad1f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:27:19.371 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:19.377+0000 7f5739b56700 1 --2- 172.21.15.52:0/1762830157 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f5734109880 0x7f57341ad1f0 secure :-1 s=READY pgs=522 cs=0 l=1 rev1=1 crypto rx=0x7f572c00c370 tx=0x7f572c00c680 comp rx=0 tx=0).stop 2024-05-27T07:27:19.371 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:19.378+0000 7f5739b56700 1 -- 172.21.15.52:0/1762830157 shutdown_connections 2024-05-27T07:27:19.372 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:19.378+0000 7f5739b56700 1 --2- 172.21.15.52:0/1762830157 >> [v2:172.21.15.52:6800/1822318384,v1:172.21.15.52:6801/1822318384] conn(0x7f571c0b2c20 0x7f571c0b5130 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:27:19.372 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:19.378+0000 7f5739b56700 1 --2- 172.21.15.52:0/1762830157 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f5734109880 0x7f57341ad1f0 unknown :-1 s=CLOSED pgs=522 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:27:19.372 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:19.378+0000 7f5739b56700 1 --2- 172.21.15.52:0/1762830157 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f5734104600 0x7f57341a2170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:27:19.372 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:19.378+0000 7f5739b56700 1 --2- 172.21.15.52:0/1762830157 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f5734108f30 0x7f57341a26b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:27:19.372 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:19.378+0000 7f5739b56700 1 -- 172.21.15.52:0/1762830157 >> 172.21.15.52:0/1762830157 conn(0x7f573407af60 msgr2=0x7f5734100ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:27:19.372 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:19.378+0000 7f5739b56700 1 -- 172.21.15.52:0/1762830157 shutdown_connections 2024-05-27T07:27:19.372 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:19.378+0000 7f5739b56700 1 -- 172.21.15.52:0/1762830157 wait complete. 2024-05-27T07:27:20.106 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:27:20 smithi052 ceph-mon[31955]: pgmap v44: 129 pgs: 129 active+clean; 1.4 MiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 7.4 KiB/s rd, 511 B/s wr, 14 op/s 2024-05-27T07:27:20.197 DEBUG:teuthology.orchestra.run.smithi052:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:38f9d900c83c63799fbdbe61acc9a11b0d3554a6 shell --fsid 73df2636-1bf8-11ef-bc9b-c7b262605968 -- ceph tell mds.1:3 damage ls 2024-05-27T07:27:20.439 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:27:20 smithi105 ceph-mon[35977]: pgmap v44: 129 pgs: 129 active+clean; 1.4 MiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 7.4 KiB/s rd, 511 B/s wr, 14 op/s 2024-05-27T07:27:20.552 INFO:teuthology.orchestra.run.smithi052.stderr:Inferring config /var/lib/ceph/73df2636-1bf8-11ef-bc9b-c7b262605968/mon.a/config 2024-05-27T07:27:20.601 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:27:20 smithi173 ceph-mon[37000]: pgmap v44: 129 pgs: 129 active+clean; 1.4 MiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 7.4 KiB/s rd, 511 B/s wr, 14 op/s 2024-05-27T07:27:22.358 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:27:22 smithi052 ceph-mon[31955]: pgmap v45: 129 pgs: 129 active+clean; 1.4 MiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 9 op/s 2024-05-27T07:27:22.437 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:22.447+0000 7fc489928700 1 -- 172.21.15.52:0/2581340808 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fc4841034d0 msgr2=0x7fc4841038e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:27:22.437 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:22.447+0000 7fc489928700 1 --2- 172.21.15.52:0/2581340808 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fc4841034d0 0x7fc4841038e0 secure :-1 s=READY pgs=525 cs=0 l=1 rev1=1 crypto rx=0x7fc4740099c0 tx=0x7fc474009cd0 comp rx=0 tx=0).stop 2024-05-27T07:27:22.437 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:22.447+0000 7fc489928700 1 -- 172.21.15.52:0/2581340808 shutdown_connections 2024-05-27T07:27:22.438 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:22.447+0000 7fc489928700 1 --2- 172.21.15.52:0/2581340808 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fc484108750 0x7fc4841100b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:27:22.438 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:22.447+0000 7fc489928700 1 --2- 172.21.15.52:0/2581340808 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fc4841034d0 0x7fc4841038e0 secure :-1 s=CLOSED pgs=525 cs=0 l=1 rev1=1 crypto rx=0x7fc4740099c0 tx=0x7fc474009cd0 comp rx=0 tx=0).stop 2024-05-27T07:27:22.438 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:22.447+0000 7fc489928700 1 --2- 172.21.15.52:0/2581340808 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fc484107e00 0x7fc484108210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:27:22.438 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:22.447+0000 7fc489928700 1 -- 172.21.15.52:0/2581340808 >> 172.21.15.52:0/2581340808 conn(0x7fc4840fef20 msgr2=0x7fc484101340 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:27:22.438 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:22.447+0000 7fc489928700 1 -- 172.21.15.52:0/2581340808 shutdown_connections 2024-05-27T07:27:22.438 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:22.447+0000 7fc489928700 1 -- 172.21.15.52:0/2581340808 wait complete. 2024-05-27T07:27:22.439 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:22.448+0000 7fc489928700 1 Processor -- start 2024-05-27T07:27:22.439 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:22.448+0000 7fc489928700 1 -- start start 2024-05-27T07:27:22.439 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:22.449+0000 7fc489928700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fc484107e00 0x7fc484198eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:27:22.439 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:22.449+0000 7fc489928700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fc484108750 0x7fc484199450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:27:22.439 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:22.449+0000 7fc489928700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fc48419d5d0 0x7fc48419d9e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:27:22.439 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:22.449+0000 7fc489928700 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x7fc4841122e0 con 0x7fc484107e00 2024-05-27T07:27:22.439 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:22.449+0000 7fc489928700 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7fc484112160 con 0x7fc48419d5d0 2024-05-27T07:27:22.440 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:22.449+0000 7fc489928700 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7fc484112460 con 0x7fc484108750 2024-05-27T07:27:22.440 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:22.449+0000 7fc482ffd700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fc484107e00 0x7fc484198eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:27:22.440 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:22.449+0000 7fc4827fc700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fc484108750 0x7fc484199450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:27:22.440 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:22.449+0000 7fc482ffd700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fc484107e00 0x7fc484198eb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.52:47184/0 (socket says 172.21.15.52:47184) 2024-05-27T07:27:22.440 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:22.449+0000 7fc482ffd700 1 -- 172.21.15.52:0/742069193 learned_addr learned my addr 172.21.15.52:0/742069193 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-05-27T07:27:22.440 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:22.449+0000 7fc4837fe700 1 --2- 172.21.15.52:0/742069193 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fc48419d5d0 0x7fc48419d9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:27:22.440 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:22.449+0000 7fc4827fc700 1 -- 172.21.15.52:0/742069193 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fc48419d5d0 msgr2=0x7fc48419d9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:27:22.440 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:22.449+0000 7fc4827fc700 1 --2- 172.21.15.52:0/742069193 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fc48419d5d0 0x7fc48419d9e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:27:22.440 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:22.449+0000 7fc4827fc700 1 -- 172.21.15.52:0/742069193 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fc484107e00 msgr2=0x7fc484198eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:27:22.440 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:22.449+0000 7fc4827fc700 1 --2- 172.21.15.52:0/742069193 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fc484107e00 0x7fc484198eb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:27:22.440 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:22.449+0000 7fc4827fc700 1 -- 172.21.15.52:0/742069193 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc48419e120 con 0x7fc484108750 2024-05-27T07:27:22.441 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:27:22 smithi105 ceph-mon[35977]: pgmap v45: 129 pgs: 129 active+clean; 1.4 MiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 9 op/s 2024-05-27T07:27:22.441 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:22.450+0000 7fc4827fc700 1 --2- 172.21.15.52:0/742069193 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fc484108750 0x7fc484199450 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7fc47800df60 tx=0x7fc47800df90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:27:22.441 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:22.450+0000 7fc488926700 1 -- 172.21.15.52:0/742069193 <== mon.1 v2:172.21.15.173:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc47801a030 con 0x7fc484108750 2024-05-27T07:27:22.442 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:22.450+0000 7fc489928700 1 -- 172.21.15.52:0/742069193 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc48419e410 con 0x7fc484108750 2024-05-27T07:27:22.442 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:22.450+0000 7fc488926700 1 -- 172.21.15.52:0/742069193 <== mon.1 v2:172.21.15.173:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fc478004ce0 con 0x7fc484108750 2024-05-27T07:27:22.442 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:22.450+0000 7fc488926700 1 -- 172.21.15.52:0/742069193 <== mon.1 v2:172.21.15.173:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc478016d50 con 0x7fc484108750 2024-05-27T07:27:22.442 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:22.450+0000 7fc489928700 1 -- 172.21.15.52:0/742069193 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc4841a5c40 con 0x7fc484108750 2024-05-27T07:27:22.444 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:22.454+0000 7fc488926700 1 -- 172.21.15.52:0/742069193 <== mon.1 v2:172.21.15.173:3300/0 4 ==== mgrmap(e 25) ==== 149936+0+0 (secure 0 0 0) 0x7fc47800c070 con 0x7fc484108750 2024-05-27T07:27:22.444 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:22.455+0000 7fc488926700 1 --2- 172.21.15.52:0/742069193 >> [v2:172.21.15.52:6800/1822318384,v1:172.21.15.52:6801/1822318384] conn(0x7fc46c0b2c20 0x7fc46c0b5130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:27:22.445 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:22.455+0000 7fc482ffd700 1 --2- 172.21.15.52:0/742069193 >> [v2:172.21.15.52:6800/1822318384,v1:172.21.15.52:6801/1822318384] conn(0x7fc46c0b2c20 0x7fc46c0b5130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:27:22.445 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:22.455+0000 7fc488926700 1 -- 172.21.15.52:0/742069193 <== mon.1 v2:172.21.15.173:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9379+0+0 (secure 0 0 0) 0x7fc4780e5c00 con 0x7fc484108750 2024-05-27T07:27:22.445 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:22.456+0000 7fc49833db80 1 Processor -- start 2024-05-27T07:27:22.445 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:22.456+0000 7fc482ffd700 1 --2- 172.21.15.52:0/742069193 >> [v2:172.21.15.52:6800/1822318384,v1:172.21.15.52:6801/1822318384] conn(0x7fc46c0b2c20 0x7fc46c0b5130 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7fc474000c00 tx=0x7fc47401a040 comp rx=0 tx=0).ready entity=mgr.15333 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:27:22.446 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:22.456+0000 7fc49833db80 1 -- start start 2024-05-27T07:27:22.446 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:22.457+0000 7fc49833db80 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5653a0abbe40 0x5653a0c3f710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:27:22.447 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:22.457+0000 7fc49833db80 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x5653a0c3fc50 0x5653a0c39a40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:27:22.447 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:22.457+0000 7fc49833db80 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5653a0c39f80 0x5653a0c3a390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:27:22.447 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:22.457+0000 7fc49833db80 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x5653a0b68c90 con 0x5653a0c3fc50 2024-05-27T07:27:22.447 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:22.457+0000 7fc49833db80 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x5653a0b58830 con 0x5653a0abbe40 2024-05-27T07:27:22.447 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:22.457+0000 7fc49833db80 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x5653a0b5c870 con 0x5653a0c39f80 2024-05-27T07:27:22.447 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:22.457+0000 7fc4837fe700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5653a0c39f80 0x5653a0c3a390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:27:22.448 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:22.457+0000 7fc4837fe700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5653a0c39f80 0x5653a0c3a390 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.173:3300/0 says I am v2:172.21.15.52:53694/0 (socket says 172.21.15.52:53694) 2024-05-27T07:27:22.448 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:22.457+0000 7fc4837fe700 1 -- 172.21.15.52:0/80444200 learned_addr learned my addr 172.21.15.52:0/80444200 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-05-27T07:27:22.448 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:22.457+0000 7fc482ffd700 1 --2- 172.21.15.52:0/80444200 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5653a0abbe40 0x5653a0c3f710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:27:22.448 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:22.457+0000 7fc4827fc700 1 --2- 172.21.15.52:0/80444200 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x5653a0c3fc50 0x5653a0c39a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:27:22.448 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:22.457+0000 7fc4837fe700 1 -- 172.21.15.52:0/80444200 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5653a0abbe40 msgr2=0x5653a0c3f710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:27:22.448 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:22.457+0000 7fc4837fe700 1 --2- 172.21.15.52:0/80444200 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5653a0abbe40 0x5653a0c3f710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:27:22.448 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:22.457+0000 7fc4837fe700 1 -- 172.21.15.52:0/80444200 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x5653a0c3fc50 msgr2=0x5653a0c39a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:27:22.448 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:22.457+0000 7fc4837fe700 1 --2- 172.21.15.52:0/80444200 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x5653a0c3fc50 0x5653a0c39a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:27:22.448 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:22.457+0000 7fc4837fe700 1 -- 172.21.15.52:0/80444200 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5653a0aa88d0 con 0x5653a0c39f80 2024-05-27T07:27:22.448 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:22.457+0000 7fc4827fc700 1 --2- 172.21.15.52:0/80444200 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x5653a0c3fc50 0x5653a0c39a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-27T07:27:22.448 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:22.457+0000 7fc482ffd700 1 --2- 172.21.15.52:0/80444200 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5653a0abbe40 0x5653a0c3f710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-27T07:27:22.448 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:22.458+0000 7fc4837fe700 1 --2- 172.21.15.52:0/80444200 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5653a0c39f80 0x5653a0c3a390 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7fc47c011680 tx=0x7fc47c009920 comp rx=0 tx=0).ready entity=mon.1 client_cookie=e6aca526866e2f61 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:27:22.450 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:22.458+0000 7fc469ffb700 1 -- 172.21.15.52:0/80444200 <== mon.1 v2:172.21.15.173:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc47c019030 con 0x5653a0c39f80 2024-05-27T07:27:22.450 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:22.458+0000 7fc469ffb700 1 -- 172.21.15.52:0/80444200 <== mon.1 v2:172.21.15.173:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fc47c0093a0 con 0x5653a0c39f80 2024-05-27T07:27:22.450 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:22.458+0000 7fc469ffb700 1 -- 172.21.15.52:0/80444200 <== mon.1 v2:172.21.15.173:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc47c007500 con 0x5653a0c39f80 2024-05-27T07:27:22.450 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:22.459+0000 7fc49833db80 1 -- 172.21.15.52:0/80444200 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5653a0c39f80 msgr2=0x5653a0c3a390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:27:22.450 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:22.459+0000 7fc49833db80 1 --2- 172.21.15.52:0/80444200 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5653a0c39f80 0x5653a0c3a390 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7fc47c011680 tx=0x7fc47c009920 comp rx=0 tx=0).stop 2024-05-27T07:27:22.450 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:22.460+0000 7fc49833db80 1 -- 172.21.15.52:0/80444200 shutdown_connections 2024-05-27T07:27:22.450 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:22.460+0000 7fc49833db80 1 --2- 172.21.15.52:0/80444200 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5653a0c39f80 0x5653a0c3a390 unknown :-1 s=CLOSED pgs=78 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:27:22.450 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:22.460+0000 7fc49833db80 1 --2- 172.21.15.52:0/80444200 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5653a0abbe40 0x5653a0c3f710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:27:22.450 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:22.460+0000 7fc49833db80 1 --2- 172.21.15.52:0/80444200 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x5653a0c3fc50 0x5653a0c39a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:27:22.450 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:22.460+0000 7fc49833db80 1 -- 172.21.15.52:0/80444200 >> 172.21.15.52:0/80444200 conn(0x5653a0b16af0 msgr2=0x5653a0c34640 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:27:22.451 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:22.460+0000 7fc49833db80 1 -- 172.21.15.52:0/80444200 shutdown_connections 2024-05-27T07:27:22.451 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:22.460+0000 7fc49833db80 1 -- 172.21.15.52:0/80444200 wait complete. 2024-05-27T07:27:22.451 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:22.460+0000 7fc49833db80 1 -- start start 2024-05-27T07:27:22.452 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:22.461+0000 7fc49833db80 1 Processor -- start 2024-05-27T07:27:22.452 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:22.461+0000 7fc49833db80 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5653a0abbe40 0x5653a0c433f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:27:22.452 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:22.461+0000 7fc49833db80 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5653a0c39f80 0x5653a0c43930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:27:22.452 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:22.461+0000 7fc49833db80 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x5653a0c3fc50 0x5653a0c43e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:27:22.452 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:22.461+0000 7fc49833db80 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x5653a0a5efa0 con 0x5653a0c3fc50 2024-05-27T07:27:22.452 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:22.461+0000 7fc49833db80 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x5653a0afa7c0 con 0x5653a0c39f80 2024-05-27T07:27:22.452 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:22.461+0000 7fc49833db80 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x5653a0c46790 con 0x5653a0abbe40 2024-05-27T07:27:22.452 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:22.461+0000 7fc49833db80 10 client.0 _ll_register_callbacks cb 0x5653a0af3920 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T07:27:22.452 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:22.461+0000 7fc482ffd700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5653a0abbe40 0x5653a0c433f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:27:22.452 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:22.461+0000 7fc482ffd700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5653a0abbe40 0x5653a0c433f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.173:3300/0 says I am v2:172.21.15.52:53698/0 (socket says 172.21.15.52:53698) 2024-05-27T07:27:22.452 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:22.462+0000 7fc482ffd700 1 -- 172.21.15.52:0/3298525951 learned_addr learned my addr 172.21.15.52:0/3298525951 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-05-27T07:27:22.453 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:22.462+0000 7fc4837fe700 1 --2- 172.21.15.52:0/3298525951 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x5653a0c3fc50 0x5653a0c43e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:27:22.453 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:22.462+0000 7fc4827fc700 1 --2- 172.21.15.52:0/3298525951 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5653a0c39f80 0x5653a0c43930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:27:22.453 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:22.462+0000 7fc482ffd700 1 -- 172.21.15.52:0/3298525951 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5653a0c39f80 msgr2=0x5653a0c43930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:27:22.453 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:22.462+0000 7fc482ffd700 1 --2- 172.21.15.52:0/3298525951 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5653a0c39f80 0x5653a0c43930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:27:22.453 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:22.462+0000 7fc482ffd700 1 -- 172.21.15.52:0/3298525951 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x5653a0c3fc50 msgr2=0x5653a0c43e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:27:22.453 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:22.462+0000 7fc482ffd700 1 --2- 172.21.15.52:0/3298525951 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x5653a0c3fc50 0x5653a0c43e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:27:22.453 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:22.462+0000 7fc482ffd700 1 -- 172.21.15.52:0/3298525951 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5653a0af7690 con 0x5653a0abbe40 2024-05-27T07:27:22.453 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:22.462+0000 7fc4827fc700 1 --2- 172.21.15.52:0/3298525951 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5653a0c39f80 0x5653a0c43930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-27T07:27:22.453 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:22.462+0000 7fc4837fe700 1 --2- 172.21.15.52:0/3298525951 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x5653a0c3fc50 0x5653a0c43e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-27T07:27:22.454 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:22.463+0000 7fc482ffd700 1 --2- 172.21.15.52:0/3298525951 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5653a0abbe40 0x5653a0c433f0 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7fc474015000 tx=0x7fc474005040 comp rx=0 tx=0).ready entity=mon.1 client_cookie=79e4111b70afc9b9 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:27:22.454 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:22.463+0000 7fc462ffd700 10 client.0 ms_handle_connect on v2:172.21.15.173:3300/0 2024-05-27T07:27:22.454 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:22.463+0000 7fc462ffd700 1 -- 172.21.15.52:0/3298525951 <== mon.1 v2:172.21.15.173:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc47400b9a0 con 0x5653a0abbe40 2024-05-27T07:27:22.454 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:22.463+0000 7fc462ffd700 1 -- 172.21.15.52:0/3298525951 <== mon.1 v2:172.21.15.173:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 9379+0+0 (secure 0 0 0) 0x7fc47400bb40 con 0x5653a0abbe40 2024-05-27T07:27:22.455 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:22.463+0000 7fc49833db80 1 -- 172.21.15.52:0/3298525951 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x5653a0c059f0 con 0x5653a0abbe40 2024-05-27T07:27:22.455 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:22.463+0000 7fc462ffd700 1 -- 172.21.15.52:0/3298525951 <== mon.1 v2:172.21.15.173:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7fc47400bd80 con 0x5653a0abbe40 2024-05-27T07:27:22.455 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:22.463+0000 7fc49833db80 10 client.25450 fetch_fsmap learned FSMap version 30 2024-05-27T07:27:22.455 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:22.463+0000 7fc49833db80 1 -- 172.21.15.52:0/3298525951 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x5653a0c50f40 con 0x5653a0abbe40 2024-05-27T07:27:22.455 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:22.464+0000 7fc462ffd700 1 -- 172.21.15.52:0/3298525951 <== mon.1 v2:172.21.15.173:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x5653a0c059f0 con 0x5653a0abbe40 2024-05-27T07:27:22.455 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:22.465+0000 7fc462ffd700 1 -- 172.21.15.52:0/3298525951 <== mon.1 v2:172.21.15.173:3300/0 5 ==== fsmap(e 30) ==== 8362+0+0 (secure 0 0 0) 0x7fc474021070 con 0x5653a0abbe40 2024-05-27T07:27:22.455 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:22.465+0000 7fc49833db80 10 client.25450 fetch_fsmap finished waiting for FSMap version 30 2024-05-27T07:27:22.455 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:22.465+0000 7fc49833db80 10 client.25450 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.e 2024-05-27T07:27:22.456 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:22.465+0000 7fc49833db80 20 client.25450 populate_metadata read hostname 'smithi052' 2024-05-27T07:27:22.456 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:22.465+0000 7fc49833db80 1 --2- 172.21.15.52:0/3298525951 >> [v2:172.21.15.105:6834/3298498384,v1:172.21.15.105:6835/3298498384] conn(0x5653a0c534d0 0x5653a0c55930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:27:22.456 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:22.465+0000 7fc49833db80 4 client.25450 mds_command: new command op to 24473 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-27T07:27:22.456 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:22.465+0000 7fc49833db80 1 -- 172.21.15.52:0/3298525951 --> [v2:172.21.15.105:6834/3298498384,v1:172.21.15.105:6835/3298498384] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5653a0b6abe0 con 0x5653a0c534d0 2024-05-27T07:27:22.456 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:22.465+0000 7fc4827fc700 1 --2- 172.21.15.52:0/3298525951 >> [v2:172.21.15.105:6834/3298498384,v1:172.21.15.105:6835/3298498384] conn(0x5653a0c534d0 0x5653a0c55930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:27:22.456 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:22.467+0000 7fc4827fc700 1 --2- 172.21.15.52:0/3298525951 >> [v2:172.21.15.105:6834/3298498384,v1:172.21.15.105:6835/3298498384] conn(0x5653a0c534d0 0x5653a0c55930 crc :-1 s=READY pgs=28 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=41d0ee41f051ad8a server_cookie=8b01f6ea2fed77c5 in_seq=0 out_seq=0 2024-05-27T07:27:22.456 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:22.467+0000 7fc462ffd700 10 client.25450 ms_handle_connect on v2:172.21.15.105:6834/3298498384 2024-05-27T07:27:22.457 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:22.467+0000 7fc462ffd700 1 -- 172.21.15.52:0/3298525951 <== mds.3 v2:172.21.15.105:6834/3298498384 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24041 (crc 0 0 0) 0x5653a0b6abe0 con 0x5653a0c534d0 2024-05-27T07:27:22.457 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:22.467+0000 7fc462ffd700 10 client.25450 handle_command_reply: tid=0 2024-05-27T07:27:22.457 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:22.468+0000 7fc49833db80 1 client.25450 shutdown 2024-05-27T07:27:22.458 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:22.468+0000 7fc49833db80 1 -- 172.21.15.52:0/3298525951 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5653a0abbe40 msgr2=0x5653a0c433f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:27:22.458 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:22.468+0000 7fc49833db80 1 --2- 172.21.15.52:0/3298525951 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5653a0abbe40 0x5653a0c433f0 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7fc474015000 tx=0x7fc474005040 comp rx=0 tx=0).stop 2024-05-27T07:27:22.458 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:22.468+0000 7fc49833db80 1 -- 172.21.15.52:0/3298525951 shutdown_connections 2024-05-27T07:27:22.458 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:22.468+0000 7fc49833db80 1 --2- 172.21.15.52:0/3298525951 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x5653a0c3fc50 0x5653a0c43e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:27:22.458 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:22.468+0000 7fc49833db80 1 --2- 172.21.15.52:0/3298525951 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5653a0abbe40 0x5653a0c433f0 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:27:22.458 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:22.468+0000 7fc49833db80 1 --2- 172.21.15.52:0/3298525951 >> [v2:172.21.15.105:6834/3298498384,v1:172.21.15.105:6835/3298498384] conn(0x5653a0c534d0 0x5653a0c55930 crc :-1 s=READY pgs=28 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:27:22.458 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:22.468+0000 7fc462ffd700 0 client.25450 ms_handle_reset on v2:172.21.15.105:6834/3298498384 2024-05-27T07:27:22.458 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:22.468+0000 7fc49833db80 1 --2- 172.21.15.52:0/3298525951 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5653a0c39f80 0x5653a0c43930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:27:22.458 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:22.468+0000 7fc49833db80 1 -- 172.21.15.52:0/3298525951 >> 172.21.15.52:0/3298525951 conn(0x5653a0b16af0 msgr2=0x5653a0c38a30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:27:22.459 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:22.468+0000 7fc49833db80 1 -- 172.21.15.52:0/3298525951 shutdown_connections 2024-05-27T07:27:22.459 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:22.468+0000 7fc49833db80 1 -- 172.21.15.52:0/3298525951 wait complete. 2024-05-27T07:27:22.459 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:22.468+0000 7fc49833db80 20 client.25450 trim_cache size 0 max 16384 2024-05-27T07:27:22.488 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:22.499+0000 7fc49833db80 1 Processor -- start 2024-05-27T07:27:22.489 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:22.499+0000 7fc49833db80 1 -- start start 2024-05-27T07:27:22.489 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:22.499+0000 7fc49833db80 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5653a0abbe40 0x5653a0c39a40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:27:22.489 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:22.499+0000 7fc49833db80 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x5653a0c39f80 0x5653a0c4f830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:27:22.489 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:22.500+0000 7fc49833db80 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5653a0c3fc50 0x5653a0c3d720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:27:22.489 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:22.500+0000 7fc49833db80 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x7fc4841ae810 con 0x5653a0c39f80 2024-05-27T07:27:22.489 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:22.500+0000 7fc49833db80 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7fc4841b0350 con 0x5653a0abbe40 2024-05-27T07:27:22.490 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:22.500+0000 7fc49833db80 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7fc4841aeee0 con 0x5653a0c3fc50 2024-05-27T07:27:22.494 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:22.500+0000 7fc482ffd700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5653a0abbe40 0x5653a0c39a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:27:22.494 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:22.500+0000 7fc4827fc700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x5653a0c39f80 0x5653a0c4f830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:27:22.494 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:22.500+0000 7fc4827fc700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x5653a0c39f80 0x5653a0c4f830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.52:47216/0 (socket says 172.21.15.52:47216) 2024-05-27T07:27:22.494 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:22.500+0000 7fc4827fc700 1 -- 172.21.15.52:0/3569922793 learned_addr learned my addr 172.21.15.52:0/3569922793 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-05-27T07:27:22.494 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:22.500+0000 7fc4837fe700 1 --2- 172.21.15.52:0/3569922793 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5653a0c3fc50 0x5653a0c3d720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:27:22.495 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:22.500+0000 7fc482ffd700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5653a0abbe40 0x5653a0c39a40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.52:52138/0 (socket says 172.21.15.52:52138) 2024-05-27T07:27:22.495 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:22.500+0000 7fc4837fe700 1 -- 172.21.15.52:0/3569922793 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5653a0abbe40 msgr2=0x5653a0c39a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:27:22.495 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:22.500+0000 7fc4837fe700 1 --2- 172.21.15.52:0/3569922793 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5653a0abbe40 0x5653a0c39a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:27:22.495 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:22.500+0000 7fc4837fe700 1 -- 172.21.15.52:0/3569922793 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x5653a0c39f80 msgr2=0x5653a0c4f830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:27:22.495 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:22.500+0000 7fc4837fe700 1 --2- 172.21.15.52:0/3569922793 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x5653a0c39f80 0x5653a0c4f830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:27:22.495 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:22.500+0000 7fc4837fe700 1 -- 172.21.15.52:0/3569922793 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5653a0c34830 con 0x5653a0c3fc50 2024-05-27T07:27:22.495 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:22.501+0000 7fc4837fe700 1 --2- 172.21.15.52:0/3569922793 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5653a0c3fc50 0x5653a0c3d720 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7fc47c011d20 tx=0x7fc47c004290 comp rx=0 tx=0).ready entity=mon.1 client_cookie=eed301ff83081977 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:27:22.495 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:22.501+0000 7fc4697fa700 1 -- 172.21.15.52:0/3569922793 <== mon.1 v2:172.21.15.173:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc47c003c00 con 0x5653a0c3fc50 2024-05-27T07:27:22.495 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:22.501+0000 7fc4697fa700 1 -- 172.21.15.52:0/3569922793 <== mon.1 v2:172.21.15.173:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fc47c007500 con 0x5653a0c3fc50 2024-05-27T07:27:22.495 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:22.501+0000 7fc4697fa700 1 -- 172.21.15.52:0/3569922793 <== mon.1 v2:172.21.15.173:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc47c015670 con 0x5653a0c3fc50 2024-05-27T07:27:22.499 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:22.502+0000 7fc49833db80 1 -- 172.21.15.52:0/3569922793 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5653a0c3fc50 msgr2=0x5653a0c3d720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:27:22.499 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:22.502+0000 7fc49833db80 1 --2- 172.21.15.52:0/3569922793 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5653a0c3fc50 0x5653a0c3d720 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7fc47c011d20 tx=0x7fc47c004290 comp rx=0 tx=0).stop 2024-05-27T07:27:22.499 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:22.502+0000 7fc49833db80 1 -- 172.21.15.52:0/3569922793 shutdown_connections 2024-05-27T07:27:22.499 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:22.502+0000 7fc49833db80 1 --2- 172.21.15.52:0/3569922793 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5653a0c3fc50 0x5653a0c3d720 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:27:22.499 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:22.502+0000 7fc49833db80 1 --2- 172.21.15.52:0/3569922793 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5653a0abbe40 0x5653a0c39a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:27:22.499 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:22.502+0000 7fc49833db80 1 --2- 172.21.15.52:0/3569922793 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x5653a0c39f80 0x5653a0c4f830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:27:22.499 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:22.502+0000 7fc49833db80 1 -- 172.21.15.52:0/3569922793 >> 172.21.15.52:0/3569922793 conn(0x5653a0b16af0 msgr2=0x5653a0c3d010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:27:22.499 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:22.502+0000 7fc49833db80 1 -- 172.21.15.52:0/3569922793 shutdown_connections 2024-05-27T07:27:22.500 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:22.503+0000 7fc49833db80 1 -- 172.21.15.52:0/3569922793 wait complete. 2024-05-27T07:27:22.500 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:22.503+0000 7fc49833db80 1 -- start start 2024-05-27T07:27:22.500 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:22.504+0000 7fc49833db80 1 Processor -- start 2024-05-27T07:27:22.500 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:22.504+0000 7fc49833db80 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5653a0abbe40 0x5653a0c4ba20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:27:22.500 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:22.504+0000 7fc49833db80 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x5653a0c39f80 0x5653a0c4bf60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:27:22.500 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:22.504+0000 7fc49833db80 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5653a0c3fc50 0x5653a0c5a0e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:27:22.500 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:22.505+0000 7fc4827fc700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x5653a0c39f80 0x5653a0c4bf60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:27:22.500 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:22.505+0000 7fc4827fc700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x5653a0c39f80 0x5653a0c4bf60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.52:47230/0 (socket says 172.21.15.52:47230) 2024-05-27T07:27:22.500 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:22.505+0000 7fc4827fc700 1 -- 172.21.15.52:0/1942397609 learned_addr learned my addr 172.21.15.52:0/1942397609 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-05-27T07:27:22.500 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:22.505+0000 7fc49833db80 1 -- 172.21.15.52:0/1942397609 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x7fc4841afc80 con 0x5653a0c39f80 2024-05-27T07:27:22.501 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:22.505+0000 7fc49833db80 1 -- 172.21.15.52:0/1942397609 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7fc4841af5b0 con 0x5653a0abbe40 2024-05-27T07:27:22.501 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:22.505+0000 7fc4837fe700 1 --2- 172.21.15.52:0/1942397609 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5653a0c3fc50 0x5653a0c5a0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:27:22.501 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:22.505+0000 7fc482ffd700 1 --2- 172.21.15.52:0/1942397609 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5653a0abbe40 0x5653a0c4ba20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:27:22.501 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:22.505+0000 7fc49833db80 1 -- 172.21.15.52:0/1942397609 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7fc4841ac7b0 con 0x5653a0c3fc50 2024-05-27T07:27:22.501 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:22.506+0000 7fc49833db80 10 client.0 _ll_register_callbacks cb 0x5653a0c3a8b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T07:27:22.501 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:22.506+0000 7fc4827fc700 1 -- 172.21.15.52:0/1942397609 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5653a0c3fc50 msgr2=0x5653a0c5a0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:27:22.501 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:22.506+0000 7fc4827fc700 1 --2- 172.21.15.52:0/1942397609 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5653a0c3fc50 0x5653a0c5a0e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:27:22.501 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:22.506+0000 7fc4827fc700 1 -- 172.21.15.52:0/1942397609 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5653a0abbe40 msgr2=0x5653a0c4ba20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:27:22.501 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:22.506+0000 7fc4827fc700 1 --2- 172.21.15.52:0/1942397609 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5653a0abbe40 0x5653a0c4ba20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:27:22.501 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:22.506+0000 7fc4827fc700 1 -- 172.21.15.52:0/1942397609 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5653a0c5ba80 con 0x5653a0c39f80 2024-05-27T07:27:22.501 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:22.506+0000 7fc49833db80 1 -- 172.21.15.52:0/1942397609 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fc47c007500 con 0x5653a0c39f80 2024-05-27T07:27:22.502 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:22.506+0000 7fc4827fc700 1 --2- 172.21.15.52:0/1942397609 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x5653a0c39f80 0x5653a0c4bf60 secure :-1 s=READY pgs=526 cs=0 l=1 rev1=1 crypto rx=0x7fc4780ac8d0 tx=0x7fc478016760 comp rx=0 tx=0).ready entity=mon.0 client_cookie=27396c8fb9277b83 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:27:22.502 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:22.506+0000 7fc462ffd700 10 client.15828 ms_handle_connect on v2:172.21.15.52:3300/0 2024-05-27T07:27:22.502 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:22.506+0000 7fc462ffd700 1 -- 172.21.15.52:0/1942397609 <== mon.0 v2:172.21.15.52:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc478092e60 con 0x5653a0c39f80 2024-05-27T07:27:22.502 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:22.506+0000 7fc462ffd700 1 -- 172.21.15.52:0/1942397609 <== mon.0 v2:172.21.15.52:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 9379+0+0 (secure 0 0 0) 0x7fc4780b5520 con 0x5653a0c39f80 2024-05-27T07:27:22.502 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:22.507+0000 7fc462ffd700 1 -- 172.21.15.52:0/1942397609 <== mon.0 v2:172.21.15.52:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7fc478023040 con 0x5653a0c39f80 2024-05-27T07:27:22.502 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:22.507+0000 7fc49833db80 10 client.15828 fetch_fsmap learned FSMap version 30 2024-05-27T07:27:22.502 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:22.507+0000 7fc49833db80 1 -- 172.21.15.52:0/1942397609 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({fsmap=30}) -- 0x5653a0cbd1a0 con 0x5653a0c39f80 2024-05-27T07:27:22.502 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:22.507+0000 7fc462ffd700 1 -- 172.21.15.52:0/1942397609 <== mon.0 v2:172.21.15.52:3300/0 4 ==== fsmap(e 30) ==== 8362+0+0 (secure 0 0 0) 0x7fc4780abba0 con 0x5653a0c39f80 2024-05-27T07:27:22.502 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:22.507+0000 7fc49833db80 10 client.15828 fetch_fsmap finished waiting for FSMap version 30 2024-05-27T07:27:22.502 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:22.507+0000 7fc49833db80 10 client.15828 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.e 2024-05-27T07:27:22.502 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:22.507+0000 7fc49833db80 20 client.15828 populate_metadata read hostname 'smithi052' 2024-05-27T07:27:22.503 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:22.507+0000 7fc49833db80 1 --2- 172.21.15.52:0/1942397609 >> [v2:172.21.15.105:6834/3298498384,v1:172.21.15.105:6835/3298498384] conn(0x5653a0c534d0 0x5653a0cbf870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:27:22.503 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:22.507+0000 7fc49833db80 4 client.15828 mds_command: new command op to 24473 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-05-27T07:27:22.503 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:22.507+0000 7fc49833db80 1 -- 172.21.15.52:0/1942397609 --> [v2:172.21.15.105:6834/3298498384,v1:172.21.15.105:6835/3298498384] -- command(tid 0: {"prefix": "damage ls"}) -- 0x5653a0b643c0 con 0x5653a0c534d0 2024-05-27T07:27:22.503 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:22.508+0000 7fc482ffd700 1 --2- 172.21.15.52:0/1942397609 >> [v2:172.21.15.105:6834/3298498384,v1:172.21.15.105:6835/3298498384] conn(0x5653a0c534d0 0x5653a0cbf870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:27:22.503 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:22.508+0000 7fc482ffd700 1 --2- 172.21.15.52:0/1942397609 >> [v2:172.21.15.105:6834/3298498384,v1:172.21.15.105:6835/3298498384] conn(0x5653a0c534d0 0x5653a0cbf870 crc :-1 s=READY pgs=29 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=fa25562ee9e8c94f server_cookie=a361530487a00194 in_seq=0 out_seq=0 2024-05-27T07:27:22.503 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:22.508+0000 7fc462ffd700 10 client.15828 ms_handle_connect on v2:172.21.15.105:6834/3298498384 2024-05-27T07:27:22.503 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:22.508+0000 7fc462ffd700 1 -- 172.21.15.52:0/1942397609 <== mds.3 v2:172.21.15.105:6834/3298498384 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x5653a0b643c0 con 0x5653a0c534d0 2024-05-27T07:27:22.503 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:22.508+0000 7fc462ffd700 10 client.15828 handle_command_reply: tid=0 2024-05-27T07:27:22.503 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:22.508+0000 7fc49833db80 1 client.15828 shutdown 2024-05-27T07:27:22.503 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:22.509+0000 7fc49833db80 1 -- 172.21.15.52:0/1942397609 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x5653a0c39f80 msgr2=0x5653a0c4bf60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:27:22.503 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:22.509+0000 7fc49833db80 1 --2- 172.21.15.52:0/1942397609 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x5653a0c39f80 0x5653a0c4bf60 secure :-1 s=READY pgs=526 cs=0 l=1 rev1=1 crypto rx=0x7fc4780ac8d0 tx=0x7fc478016760 comp rx=0 tx=0).stop 2024-05-27T07:27:22.504 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:22.509+0000 7fc49833db80 1 -- 172.21.15.52:0/1942397609 shutdown_connections 2024-05-27T07:27:22.504 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:22.509+0000 7fc49833db80 1 --2- 172.21.15.52:0/1942397609 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5653a0c3fc50 0x5653a0c5a0e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:27:22.504 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:22.509+0000 7fc49833db80 1 --2- 172.21.15.52:0/1942397609 >> [v2:172.21.15.105:6834/3298498384,v1:172.21.15.105:6835/3298498384] conn(0x5653a0c534d0 0x5653a0cbf870 crc :-1 s=READY pgs=29 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:27:22.504 INFO:teuthology.orchestra.run.smithi052.stdout:[] 2024-05-27T07:27:22.505 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:22.509+0000 7fc462ffd700 0 client.15828 ms_handle_reset on v2:172.21.15.105:6834/3298498384 2024-05-27T07:27:22.505 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:22.509+0000 7fc49833db80 1 --2- 172.21.15.52:0/1942397609 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5653a0abbe40 0x5653a0c4ba20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:27:22.505 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:22.509+0000 7fc49833db80 1 --2- 172.21.15.52:0/1942397609 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x5653a0c39f80 0x5653a0c4bf60 unknown :-1 s=CLOSED pgs=526 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:27:22.505 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:22.509+0000 7fc49833db80 1 -- 172.21.15.52:0/1942397609 >> 172.21.15.52:0/1942397609 conn(0x5653a0b16af0 msgr2=0x5653a0c449e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:27:22.505 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:22.509+0000 7fc49833db80 1 -- 172.21.15.52:0/1942397609 shutdown_connections 2024-05-27T07:27:22.505 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:22.509+0000 7fc49833db80 1 -- 172.21.15.52:0/1942397609 wait complete. 2024-05-27T07:27:22.505 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:22.509+0000 7fc49833db80 20 client.15828 trim_cache size 0 max 16384 2024-05-27T07:27:22.505 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:22.510+0000 7fc489928700 1 -- 172.21.15.52:0/742069193 >> [v2:172.21.15.52:6800/1822318384,v1:172.21.15.52:6801/1822318384] conn(0x7fc46c0b2c20 msgr2=0x7fc46c0b5130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:27:22.505 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:22.510+0000 7fc489928700 1 --2- 172.21.15.52:0/742069193 >> [v2:172.21.15.52:6800/1822318384,v1:172.21.15.52:6801/1822318384] conn(0x7fc46c0b2c20 0x7fc46c0b5130 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7fc474000c00 tx=0x7fc47401a040 comp rx=0 tx=0).stop 2024-05-27T07:27:22.505 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:22.510+0000 7fc489928700 1 -- 172.21.15.52:0/742069193 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fc484108750 msgr2=0x7fc484199450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:27:22.505 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:22.510+0000 7fc489928700 1 --2- 172.21.15.52:0/742069193 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fc484108750 0x7fc484199450 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7fc47800df60 tx=0x7fc47800df90 comp rx=0 tx=0).stop 2024-05-27T07:27:22.506 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:22.510+0000 7fc489928700 1 -- 172.21.15.52:0/742069193 shutdown_connections 2024-05-27T07:27:22.506 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:22.510+0000 7fc489928700 1 --2- 172.21.15.52:0/742069193 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fc48419d5d0 0x7fc48419d9e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:27:22.506 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:22.510+0000 7fc489928700 1 --2- 172.21.15.52:0/742069193 >> [v2:172.21.15.52:6800/1822318384,v1:172.21.15.52:6801/1822318384] conn(0x7fc46c0b2c20 0x7fc46c0b5130 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:27:22.506 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:22.510+0000 7fc489928700 1 --2- 172.21.15.52:0/742069193 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fc484107e00 0x7fc484198eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:27:22.506 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:22.510+0000 7fc489928700 1 --2- 172.21.15.52:0/742069193 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fc484108750 0x7fc484199450 unknown :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:27:22.506 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:22.510+0000 7fc489928700 1 -- 172.21.15.52:0/742069193 >> 172.21.15.52:0/742069193 conn(0x7fc4840fef20 msgr2=0x7fc484101340 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:27:22.506 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:22.510+0000 7fc489928700 1 -- 172.21.15.52:0/742069193 shutdown_connections 2024-05-27T07:27:22.506 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:22.510+0000 7fc489928700 1 -- 172.21.15.52:0/742069193 wait complete. 2024-05-27T07:27:22.601 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:27:22 smithi173 ceph-mon[37000]: pgmap v45: 129 pgs: 129 active+clean; 1.4 MiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 9 op/s 2024-05-27T07:27:24.440 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:27:24 smithi105 ceph-mon[35977]: pgmap v46: 129 pgs: 129 active+clean; 1.4 MiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 7.4 KiB/s rd, 255 B/s wr, 13 op/s 2024-05-27T07:27:24.479 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:27:24 smithi052 ceph-mon[31955]: pgmap v46: 129 pgs: 129 active+clean; 1.4 MiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 7.4 KiB/s rd, 255 B/s wr, 13 op/s 2024-05-27T07:27:24.601 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:27:24 smithi173 ceph-mon[37000]: pgmap v46: 129 pgs: 129 active+clean; 1.4 MiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 7.4 KiB/s rd, 255 B/s wr, 13 op/s 2024-05-27T07:27:26.233 DEBUG:teuthology.orchestra.run.smithi052:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:38f9d900c83c63799fbdbe61acc9a11b0d3554a6 shell --fsid 73df2636-1bf8-11ef-bc9b-c7b262605968 -- ceph tell mds.1:2 damage ls 2024-05-27T07:27:26.382 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:27:26 smithi052 ceph-mon[31955]: pgmap v47: 129 pgs: 129 active+clean; 1.4 MiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 255 B/s wr, 8 op/s 2024-05-27T07:27:26.439 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:27:26 smithi105 ceph-mon[35977]: pgmap v47: 129 pgs: 129 active+clean; 1.4 MiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 255 B/s wr, 8 op/s 2024-05-27T07:27:26.601 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:27:26 smithi173 ceph-mon[37000]: pgmap v47: 129 pgs: 129 active+clean; 1.4 MiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 255 B/s wr, 8 op/s 2024-05-27T07:27:26.623 INFO:teuthology.orchestra.run.smithi052.stderr:Inferring config /var/lib/ceph/73df2636-1bf8-11ef-bc9b-c7b262605968/mon.a/config 2024-05-27T07:27:28.341 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:27:28 smithi052 ceph-mon[31955]: pgmap v48: 129 pgs: 129 active+clean; 1.4 MiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 255 B/s wr, 8 op/s 2024-05-27T07:27:28.424 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:28.434+0000 7fa40bf29700 1 -- 172.21.15.52:0/861531793 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fa404107de0 msgr2=0x7fa4041081f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:27:28.424 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:28.434+0000 7fa40bf29700 1 --2- 172.21.15.52:0/861531793 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fa404107de0 0x7fa4041081f0 secure :-1 s=READY pgs=527 cs=0 l=1 rev1=1 crypto rx=0x7fa3f40099c0 tx=0x7fa3f4009cd0 comp rx=0 tx=0).stop 2024-05-27T07:27:28.425 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:28.435+0000 7fa40bf29700 1 -- 172.21.15.52:0/861531793 shutdown_connections 2024-05-27T07:27:28.425 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:28.435+0000 7fa40bf29700 1 --2- 172.21.15.52:0/861531793 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fa404108730 0x7fa404110090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:27:28.425 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:28.435+0000 7fa40bf29700 1 --2- 172.21.15.52:0/861531793 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fa4041034b0 0x7fa4041038c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:27:28.425 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:28.435+0000 7fa40bf29700 1 --2- 172.21.15.52:0/861531793 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fa404107de0 0x7fa4041081f0 unknown :-1 s=CLOSED pgs=527 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:27:28.425 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:28.435+0000 7fa40bf29700 1 -- 172.21.15.52:0/861531793 >> 172.21.15.52:0/861531793 conn(0x7fa4040fedb0 msgr2=0x7fa4041011d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:27:28.425 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:28.435+0000 7fa40bf29700 1 -- 172.21.15.52:0/861531793 shutdown_connections 2024-05-27T07:27:28.425 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:28.435+0000 7fa40bf29700 1 -- 172.21.15.52:0/861531793 wait complete. 2024-05-27T07:27:28.426 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:28.436+0000 7fa40bf29700 1 Processor -- start 2024-05-27T07:27:28.426 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:28.436+0000 7fa40bf29700 1 -- start start 2024-05-27T07:27:28.426 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:28.436+0000 7fa40bf29700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fa4041034b0 0x7fa4041a1410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:27:28.426 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:28.436+0000 7fa40bf29700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fa404107de0 0x7fa4041a19b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:27:28.426 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:28.436+0000 7fa40bf29700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fa404108730 0x7fa4041a5c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:27:28.426 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:28.436+0000 7fa40bf29700 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x7fa404113390 con 0x7fa404107de0 2024-05-27T07:27:28.426 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:28.436+0000 7fa40bf29700 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7fa404113210 con 0x7fa404108730 2024-05-27T07:27:28.426 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:28.436+0000 7fa40bf29700 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7fa404113510 con 0x7fa4041034b0 2024-05-27T07:27:28.428 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:28.437+0000 7fa40a4c6700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fa404108730 0x7fa4041a5c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:27:28.428 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:28.437+0000 7fa40a4c6700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fa404108730 0x7fa4041a5c40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.52:52150/0 (socket says 172.21.15.52:52150) 2024-05-27T07:27:28.428 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:28.437+0000 7fa40a4c6700 1 -- 172.21.15.52:0/3263786524 learned_addr learned my addr 172.21.15.52:0/3263786524 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-05-27T07:27:28.428 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:28.437+0000 7fa409cc5700 1 --2- 172.21.15.52:0/3263786524 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fa4041034b0 0x7fa4041a1410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:27:28.428 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:28.437+0000 7fa4094c4700 1 --2- 172.21.15.52:0/3263786524 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fa404107de0 0x7fa4041a19b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:27:28.428 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:28.437+0000 7fa40a4c6700 1 -- 172.21.15.52:0/3263786524 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fa4041034b0 msgr2=0x7fa4041a1410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:27:28.428 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:28.437+0000 7fa40a4c6700 1 --2- 172.21.15.52:0/3263786524 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fa4041034b0 0x7fa4041a1410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:27:28.428 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:28.437+0000 7fa40a4c6700 1 -- 172.21.15.52:0/3263786524 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fa404107de0 msgr2=0x7fa4041a19b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:27:28.428 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:28.437+0000 7fa40a4c6700 1 --2- 172.21.15.52:0/3263786524 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fa404107de0 0x7fa4041a19b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:27:28.428 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:28.437+0000 7fa40a4c6700 1 -- 172.21.15.52:0/3263786524 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa4041a62e0 con 0x7fa404108730 2024-05-27T07:27:28.428 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:28.437+0000 7fa409cc5700 1 --2- 172.21.15.52:0/3263786524 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fa4041034b0 0x7fa4041a1410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-27T07:27:28.428 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:28.437+0000 7fa40a4c6700 1 --2- 172.21.15.52:0/3263786524 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fa404108730 0x7fa4041a5c40 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7fa3fc007ee0 tx=0x7fa3fc00d3b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:27:28.429 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:28.438+0000 7fa3faffd700 1 -- 172.21.15.52:0/3263786524 <== mon.2 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa3fc00dbd0 con 0x7fa404108730 2024-05-27T07:27:28.429 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:28.438+0000 7fa40bf29700 1 -- 172.21.15.52:0/3263786524 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa4041a65d0 con 0x7fa404108730 2024-05-27T07:27:28.429 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:28.438+0000 7fa3faffd700 1 -- 172.21.15.52:0/3263786524 <== mon.2 v2:172.21.15.105:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fa3fc00dd70 con 0x7fa404108730 2024-05-27T07:27:28.429 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:28.438+0000 7fa3faffd700 1 -- 172.21.15.52:0/3263786524 <== mon.2 v2:172.21.15.105:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa3fc005990 con 0x7fa404108730 2024-05-27T07:27:28.429 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:28.438+0000 7fa40bf29700 1 -- 172.21.15.52:0/3263786524 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa40419a870 con 0x7fa404108730 2024-05-27T07:27:28.431 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:28.442+0000 7fa3faffd700 1 -- 172.21.15.52:0/3263786524 <== mon.2 v2:172.21.15.105:3300/0 4 ==== mgrmap(e 25) ==== 149936+0+0 (secure 0 0 0) 0x7fa3fc01a020 con 0x7fa404108730 2024-05-27T07:27:28.432 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:28.443+0000 7fa3faffd700 1 --2- 172.21.15.52:0/3263786524 >> [v2:172.21.15.52:6800/1822318384,v1:172.21.15.52:6801/1822318384] conn(0x7fa3f00b2c20 0x7fa3f00b5130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:27:28.433 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:28.443+0000 7fa3faffd700 1 -- 172.21.15.52:0/3263786524 <== mon.2 v2:172.21.15.105:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9379+0+0 (secure 0 0 0) 0x7fa3fc0e24a0 con 0x7fa404108730 2024-05-27T07:27:28.433 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:28.443+0000 7fa409cc5700 1 --2- 172.21.15.52:0/3263786524 >> [v2:172.21.15.52:6800/1822318384,v1:172.21.15.52:6801/1822318384] conn(0x7fa3f00b2c20 0x7fa3f00b5130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:27:28.433 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:28.443+0000 7fa41a93eb80 1 Processor -- start 2024-05-27T07:27:28.433 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:28.444+0000 7fa41a93eb80 1 -- start start 2024-05-27T07:27:28.434 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:28.444+0000 7fa41a93eb80 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x562459806e40 0x56245998a710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:27:28.434 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:28.444+0000 7fa41a93eb80 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x56245998ac50 0x562459984a40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:27:28.434 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:28.444+0000 7fa41a93eb80 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x562459984f80 0x562459985390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:27:28.434 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:28.444+0000 7fa41a93eb80 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x5624598b3c90 con 0x56245998ac50 2024-05-27T07:27:28.434 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:28.444+0000 7fa41a93eb80 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x5624598a3830 con 0x562459984f80 2024-05-27T07:27:28.434 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:28.444+0000 7fa41a93eb80 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x5624598a7870 con 0x562459806e40 2024-05-27T07:27:28.435 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:28.444+0000 7fa409cc5700 1 --2- 172.21.15.52:0/3263786524 >> [v2:172.21.15.52:6800/1822318384,v1:172.21.15.52:6801/1822318384] conn(0x7fa3f00b2c20 0x7fa3f00b5130 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7fa400009be0 tx=0x7fa400009260 comp rx=0 tx=0).ready entity=mgr.15333 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:27:28.435 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:28.444+0000 7fa40a4c6700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x562459984f80 0x562459985390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:27:28.435 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:28.444+0000 7fa40a4c6700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x562459984f80 0x562459985390 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.52:52160/0 (socket says 172.21.15.52:52160) 2024-05-27T07:27:28.435 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:28.444+0000 7fa40a4c6700 1 -- 172.21.15.52:0/1613081157 learned_addr learned my addr 172.21.15.52:0/1613081157 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-05-27T07:27:28.435 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:28.444+0000 7fa4094c4700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x56245998ac50 0x562459984a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:27:28.436 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:28.445+0000 7fa40a4c6700 1 -- 172.21.15.52:0/1613081157 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x562459806e40 msgr2=0x56245998a710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:27:28.436 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:28.445+0000 7fa409cc5700 1 --2- 172.21.15.52:0/1613081157 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x562459806e40 0x56245998a710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:27:28.436 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:28.445+0000 7fa40a4c6700 1 --2- 172.21.15.52:0/1613081157 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x562459806e40 0x56245998a710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:27:28.436 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:28.445+0000 7fa40a4c6700 1 -- 172.21.15.52:0/1613081157 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x56245998ac50 msgr2=0x562459984a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:27:28.436 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:28.445+0000 7fa40a4c6700 1 --2- 172.21.15.52:0/1613081157 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x56245998ac50 0x562459984a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:27:28.437 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:28.445+0000 7fa40a4c6700 1 -- 172.21.15.52:0/1613081157 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5624597f38d0 con 0x562459984f80 2024-05-27T07:27:28.437 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:28.445+0000 7fa4094c4700 1 --2- 172.21.15.52:0/1613081157 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x56245998ac50 0x562459984a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-27T07:27:28.437 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:28.445+0000 7fa40a4c6700 1 --2- 172.21.15.52:0/1613081157 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x562459984f80 0x562459985390 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7fa3fc0a6450 tx=0x7fa3fc0a89d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b2c5fdf77e33d49c server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:27:28.437 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:28.445+0000 7fa409cc5700 1 --2- 172.21.15.52:0/1613081157 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x562459806e40 0x56245998a710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-27T07:27:28.437 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:28.445+0000 7fa3effff700 1 -- 172.21.15.52:0/1613081157 <== mon.2 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa3fc0a91e0 con 0x562459984f80 2024-05-27T07:27:28.437 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:28.445+0000 7fa3effff700 1 -- 172.21.15.52:0/1613081157 <== mon.2 v2:172.21.15.105:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fa3fc0ad590 con 0x562459984f80 2024-05-27T07:27:28.437 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:28.445+0000 7fa3effff700 1 -- 172.21.15.52:0/1613081157 <== mon.2 v2:172.21.15.105:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa3fc0192c0 con 0x562459984f80 2024-05-27T07:27:28.437 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:28.446+0000 7fa41a93eb80 1 -- 172.21.15.52:0/1613081157 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x562459984f80 msgr2=0x562459985390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:27:28.437 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:28.446+0000 7fa41a93eb80 1 --2- 172.21.15.52:0/1613081157 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x562459984f80 0x562459985390 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7fa3fc0a6450 tx=0x7fa3fc0a89d0 comp rx=0 tx=0).stop 2024-05-27T07:27:28.437 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:28.446+0000 7fa41a93eb80 1 -- 172.21.15.52:0/1613081157 shutdown_connections 2024-05-27T07:27:28.437 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:28.446+0000 7fa41a93eb80 1 --2- 172.21.15.52:0/1613081157 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x562459984f80 0x562459985390 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:27:28.437 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:28.446+0000 7fa41a93eb80 1 --2- 172.21.15.52:0/1613081157 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x562459806e40 0x56245998a710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:27:28.437 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:28.446+0000 7fa41a93eb80 1 --2- 172.21.15.52:0/1613081157 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x56245998ac50 0x562459984a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:27:28.438 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:28.446+0000 7fa41a93eb80 1 -- 172.21.15.52:0/1613081157 >> 172.21.15.52:0/1613081157 conn(0x562459861af0 msgr2=0x56245997f640 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:27:28.438 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:28.446+0000 7fa41a93eb80 1 -- 172.21.15.52:0/1613081157 shutdown_connections 2024-05-27T07:27:28.438 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:28.446+0000 7fa41a93eb80 1 -- 172.21.15.52:0/1613081157 wait complete. 2024-05-27T07:27:28.438 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:28.447+0000 7fa41a93eb80 1 -- start start 2024-05-27T07:27:28.440 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:28.447+0000 7fa41a93eb80 1 Processor -- start 2024-05-27T07:27:28.440 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:28.448+0000 7fa41a93eb80 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x562459806e40 0x56245998e360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:27:28.440 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:28.448+0000 7fa41a93eb80 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x562459984f80 0x56245998e8a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:27:28.440 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:28.448+0000 7fa41a93eb80 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x56245998ac50 0x56245998ede0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:27:28.440 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:28.448+0000 7fa41a93eb80 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x5624597a9fa0 con 0x56245998ac50 2024-05-27T07:27:28.440 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:28.448+0000 7fa41a93eb80 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x5624598457c0 con 0x562459984f80 2024-05-27T07:27:28.440 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:28.448+0000 7fa41a93eb80 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x562459991740 con 0x562459806e40 2024-05-27T07:27:28.440 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:28.448+0000 7fa41a93eb80 10 client.0 _ll_register_callbacks cb 0x56245983e920 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T07:27:28.441 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:28.448+0000 7fa4094c4700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x562459984f80 0x56245998e8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:27:28.441 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:28.448+0000 7fa409cc5700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x562459806e40 0x56245998e360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:27:28.441 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:28.448+0000 7fa40a4c6700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x56245998ac50 0x56245998ede0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:27:28.441 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:28.448+0000 7fa409cc5700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x562459806e40 0x56245998e360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.173:3300/0 says I am v2:172.21.15.52:53748/0 (socket says 172.21.15.52:53748) 2024-05-27T07:27:28.441 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:28.448+0000 7fa409cc5700 1 -- 172.21.15.52:0/2206844707 learned_addr learned my addr 172.21.15.52:0/2206844707 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-05-27T07:27:28.441 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:28.448+0000 7fa4094c4700 1 -- 172.21.15.52:0/2206844707 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x562459806e40 msgr2=0x56245998e360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:27:28.441 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:28.448+0000 7fa4094c4700 1 --2- 172.21.15.52:0/2206844707 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x562459806e40 0x56245998e360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:27:28.441 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:28.448+0000 7fa4094c4700 1 -- 172.21.15.52:0/2206844707 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x56245998ac50 msgr2=0x56245998ede0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:27:28.441 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:28.448+0000 7fa4094c4700 1 --2- 172.21.15.52:0/2206844707 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x56245998ac50 0x56245998ede0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:27:28.441 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:28.448+0000 7fa4094c4700 1 -- 172.21.15.52:0/2206844707 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({osdmap=0}) -- 0x562459842690 con 0x562459984f80 2024-05-27T07:27:28.441 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:28.449+0000 7fa4094c4700 1 --2- 172.21.15.52:0/2206844707 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x562459984f80 0x56245998e8a0 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7fa3f4009990 tx=0x7fa3f4005800 comp rx=0 tx=0).ready entity=mon.2 client_cookie=582cf6f17f5e575a server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:27:28.441 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:28.449+0000 7fa3ed7fa700 10 client.0 ms_handle_connect on v2:172.21.15.105:3300/0 2024-05-27T07:27:28.441 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:28.449+0000 7fa3ed7fa700 1 -- 172.21.15.52:0/2206844707 <== mon.2 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa3f4003bd0 con 0x562459984f80 2024-05-27T07:27:28.442 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:28.449+0000 7fa3ed7fa700 1 -- 172.21.15.52:0/2206844707 <== mon.2 v2:172.21.15.105:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 9379+0+0 (secure 0 0 0) 0x7fa3f4003d70 con 0x562459984f80 2024-05-27T07:27:28.442 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:28.449+0000 7fa41a93eb80 1 -- 172.21.15.52:0/2206844707 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fa3fc0ad590 con 0x562459984f80 2024-05-27T07:27:28.442 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:28.449+0000 7fa3ed7fa700 1 -- 172.21.15.52:0/2206844707 <== mon.2 v2:172.21.15.105:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7fa3f4031050 con 0x562459984f80 2024-05-27T07:27:28.442 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:28.449+0000 7fa41a93eb80 10 client.25535 fetch_fsmap learned FSMap version 30 2024-05-27T07:27:28.442 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:28.449+0000 7fa41a93eb80 1 -- 172.21.15.52:0/2206844707 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x56245999c1b0 con 0x562459984f80 2024-05-27T07:27:28.442 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:28.449+0000 7fa3ed7fa700 1 -- 172.21.15.52:0/2206844707 <== mon.2 v2:172.21.15.105:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fa3f4028070 con 0x562459984f80 2024-05-27T07:27:28.442 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:28.450+0000 7fa3ed7fa700 1 -- 172.21.15.52:0/2206844707 <== mon.2 v2:172.21.15.105:3300/0 5 ==== fsmap(e 30) ==== 8362+0+0 (secure 0 0 0) 0x7fa3f4023700 con 0x562459984f80 2024-05-27T07:27:28.442 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:28.450+0000 7fa41a93eb80 10 client.25535 fetch_fsmap finished waiting for FSMap version 30 2024-05-27T07:27:28.442 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:28.450+0000 7fa41a93eb80 10 client.25535 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.i 2024-05-27T07:27:28.442 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:28.450+0000 7fa41a93eb80 20 client.25535 populate_metadata read hostname 'smithi052' 2024-05-27T07:27:28.442 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:28.450+0000 7fa41a93eb80 1 --2- 172.21.15.52:0/2206844707 >> [v2:172.21.15.173:6838/564778293,v1:172.21.15.173:6839/564778293] conn(0x56245999e890 0x5624599a0cb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:27:28.443 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:28.450+0000 7fa41a93eb80 4 client.25535 mds_command: new command op to 24475 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-27T07:27:28.443 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:28.450+0000 7fa41a93eb80 1 -- 172.21.15.52:0/2206844707 --> [v2:172.21.15.173:6838/564778293,v1:172.21.15.173:6839/564778293] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x7fa3fc0ad590 con 0x56245999e890 2024-05-27T07:27:28.443 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:28.450+0000 7fa409cc5700 1 --2- 172.21.15.52:0/2206844707 >> [v2:172.21.15.173:6838/564778293,v1:172.21.15.173:6839/564778293] conn(0x56245999e890 0x5624599a0cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:27:28.443 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:28.451+0000 7fa409cc5700 1 --2- 172.21.15.52:0/2206844707 >> [v2:172.21.15.173:6838/564778293,v1:172.21.15.173:6839/564778293] conn(0x56245999e890 0x5624599a0cb0 crc :-1 s=READY pgs=34 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=7d6b0dc108fd85ae server_cookie=3885d5f7532e3d52 in_seq=0 out_seq=0 2024-05-27T07:27:28.443 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:28.451+0000 7fa3ed7fa700 10 client.25535 ms_handle_connect on v2:172.21.15.173:6838/564778293 2024-05-27T07:27:28.443 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:27:28 smithi105 ceph-mon[35977]: pgmap v48: 129 pgs: 129 active+clean; 1.4 MiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 255 B/s wr, 8 op/s 2024-05-27T07:27:28.444 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:28.451+0000 7fa3ed7fa700 1 -- 172.21.15.52:0/2206844707 <== mds.2 v2:172.21.15.173:6838/564778293 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24041 (crc 0 0 0) 0x7fa3fc0ad590 con 0x56245999e890 2024-05-27T07:27:28.444 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:28.451+0000 7fa3ed7fa700 10 client.25535 handle_command_reply: tid=0 2024-05-27T07:27:28.444 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:28.451+0000 7fa41a93eb80 1 client.25535 shutdown 2024-05-27T07:27:28.444 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:28.451+0000 7fa41a93eb80 1 -- 172.21.15.52:0/2206844707 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x562459984f80 msgr2=0x56245998e8a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:27:28.445 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:28.451+0000 7fa41a93eb80 1 --2- 172.21.15.52:0/2206844707 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x562459984f80 0x56245998e8a0 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7fa3f4009990 tx=0x7fa3f4005800 comp rx=0 tx=0).stop 2024-05-27T07:27:28.445 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:28.452+0000 7fa41a93eb80 1 -- 172.21.15.52:0/2206844707 shutdown_connections 2024-05-27T07:27:28.445 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:28.452+0000 7fa41a93eb80 1 --2- 172.21.15.52:0/2206844707 >> [v2:172.21.15.173:6838/564778293,v1:172.21.15.173:6839/564778293] conn(0x56245999e890 0x5624599a0cb0 crc :-1 s=READY pgs=34 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:27:28.445 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:28.452+0000 7fa3ed7fa700 0 client.25535 ms_handle_reset on v2:172.21.15.173:6838/564778293 2024-05-27T07:27:28.445 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:28.452+0000 7fa41a93eb80 1 --2- 172.21.15.52:0/2206844707 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x56245998ac50 0x56245998ede0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:27:28.445 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:28.452+0000 7fa41a93eb80 1 --2- 172.21.15.52:0/2206844707 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x562459806e40 0x56245998e360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:27:28.445 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:28.452+0000 7fa41a93eb80 1 --2- 172.21.15.52:0/2206844707 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x562459984f80 0x56245998e8a0 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:27:28.445 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:28.452+0000 7fa41a93eb80 1 -- 172.21.15.52:0/2206844707 >> 172.21.15.52:0/2206844707 conn(0x562459861af0 msgr2=0x562459983a30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:27:28.445 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:28.452+0000 7fa41a93eb80 1 -- 172.21.15.52:0/2206844707 shutdown_connections 2024-05-27T07:27:28.445 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:28.452+0000 7fa41a93eb80 1 -- 172.21.15.52:0/2206844707 wait complete. 2024-05-27T07:27:28.445 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:28.452+0000 7fa41a93eb80 20 client.25535 trim_cache size 0 max 16384 2024-05-27T07:27:28.472 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:28.479+0000 7fa41a93eb80 1 Processor -- start 2024-05-27T07:27:28.472 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:28.479+0000 7fa41a93eb80 1 -- start start 2024-05-27T07:27:28.472 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:28.480+0000 7fa41a93eb80 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x562459806e40 0x562459988a40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:27:28.472 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:28.480+0000 7fa41a93eb80 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x562459984f80 0x56245999a7e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:27:28.472 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:28.480+0000 7fa41a93eb80 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x56245998ac50 0x5624599906c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:27:28.473 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:28.480+0000 7fa41a93eb80 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x7fa4041bf0d0 con 0x56245998ac50 2024-05-27T07:27:28.473 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:28.480+0000 7fa41a93eb80 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7fa4041c0c10 con 0x562459806e40 2024-05-27T07:27:28.473 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:28.480+0000 7fa41a93eb80 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7fa4041bf7a0 con 0x562459984f80 2024-05-27T07:27:28.473 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:28.480+0000 7fa40a4c6700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x56245998ac50 0x5624599906c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:27:28.473 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:28.480+0000 7fa409cc5700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x562459806e40 0x562459988a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:27:28.474 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:28.480+0000 7fa40a4c6700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x56245998ac50 0x5624599906c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.52:47284/0 (socket says 172.21.15.52:47284) 2024-05-27T07:27:28.474 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:28.480+0000 7fa40a4c6700 1 -- 172.21.15.52:0/716200336 learned_addr learned my addr 172.21.15.52:0/716200336 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-05-27T07:27:28.474 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:28.480+0000 7fa4094c4700 1 --2- 172.21.15.52:0/716200336 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x562459984f80 0x56245999a7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:27:28.474 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:28.481+0000 7fa40a4c6700 1 -- 172.21.15.52:0/716200336 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x562459984f80 msgr2=0x56245999a7e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:27:28.474 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:28.481+0000 7fa40a4c6700 1 --2- 172.21.15.52:0/716200336 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x562459984f80 0x56245999a7e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:27:28.474 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:28.481+0000 7fa40a4c6700 1 -- 172.21.15.52:0/716200336 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x562459806e40 msgr2=0x562459988a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:27:28.475 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:28.481+0000 7fa40a4c6700 1 --2- 172.21.15.52:0/716200336 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x562459806e40 0x562459988a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:27:28.475 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:28.481+0000 7fa40a4c6700 1 -- 172.21.15.52:0/716200336 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x562459979fa0 con 0x56245998ac50 2024-05-27T07:27:28.475 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:28.481+0000 7fa40a4c6700 1 --2- 172.21.15.52:0/716200336 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x56245998ac50 0x5624599906c0 secure :-1 s=READY pgs=528 cs=0 l=1 rev1=1 crypto rx=0x7fa3fc009510 tx=0x7fa3fc0a8a60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b0087bb0b9c1d446 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:27:28.475 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:28.481+0000 7fa3ef7fe700 1 -- 172.21.15.52:0/716200336 <== mon.0 v2:172.21.15.52:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa3fc0a91e0 con 0x56245998ac50 2024-05-27T07:27:28.475 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:28.481+0000 7fa3ef7fe700 1 -- 172.21.15.52:0/716200336 <== mon.0 v2:172.21.15.52:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fa3fc0b2040 con 0x56245998ac50 2024-05-27T07:27:28.476 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:28.481+0000 7fa3ef7fe700 1 -- 172.21.15.52:0/716200336 <== mon.0 v2:172.21.15.52:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa3fc0a9440 con 0x56245998ac50 2024-05-27T07:27:28.481 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:28.482+0000 7fa41a93eb80 1 -- 172.21.15.52:0/716200336 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x56245998ac50 msgr2=0x5624599906c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:27:28.481 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:28.482+0000 7fa41a93eb80 1 --2- 172.21.15.52:0/716200336 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x56245998ac50 0x5624599906c0 secure :-1 s=READY pgs=528 cs=0 l=1 rev1=1 crypto rx=0x7fa3fc009510 tx=0x7fa3fc0a8a60 comp rx=0 tx=0).stop 2024-05-27T07:27:28.481 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:28.483+0000 7fa41a93eb80 1 -- 172.21.15.52:0/716200336 shutdown_connections 2024-05-27T07:27:28.481 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:28.483+0000 7fa41a93eb80 1 --2- 172.21.15.52:0/716200336 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x56245998ac50 0x5624599906c0 unknown :-1 s=CLOSED pgs=528 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:27:28.481 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:28.483+0000 7fa41a93eb80 1 --2- 172.21.15.52:0/716200336 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x562459806e40 0x562459988a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:27:28.481 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:28.483+0000 7fa41a93eb80 1 --2- 172.21.15.52:0/716200336 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x562459984f80 0x56245999a7e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:27:28.482 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:28.483+0000 7fa41a93eb80 1 -- 172.21.15.52:0/716200336 >> 172.21.15.52:0/716200336 conn(0x562459861af0 msgr2=0x562459989a80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:27:28.482 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:28.483+0000 7fa41a93eb80 1 -- 172.21.15.52:0/716200336 shutdown_connections 2024-05-27T07:27:28.482 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:28.483+0000 7fa41a93eb80 1 -- 172.21.15.52:0/716200336 wait complete. 2024-05-27T07:27:28.482 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:28.483+0000 7fa41a93eb80 1 -- start start 2024-05-27T07:27:28.482 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:28.483+0000 7fa41a93eb80 1 Processor -- start 2024-05-27T07:27:28.483 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:28.484+0000 7fa41a93eb80 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x562459806e40 0x5624599a60b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:27:28.483 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:28.484+0000 7fa41a93eb80 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x562459984f80 0x5624599a65f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:27:28.483 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:28.484+0000 7fa41a93eb80 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x56245998ac50 0x5624599a6b30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:27:28.483 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:28.484+0000 7fa41a93eb80 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x7fa4041c0540 con 0x56245998ac50 2024-05-27T07:27:28.483 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:28.484+0000 7fa41a93eb80 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7fa4041bfe70 con 0x562459806e40 2024-05-27T07:27:28.483 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:28.484+0000 7fa41a93eb80 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7fa4041bd070 con 0x562459984f80 2024-05-27T07:27:28.484 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:28.484+0000 7fa41a93eb80 10 client.0 _ll_register_callbacks cb 0x56245983e920 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T07:27:28.484 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:28.484+0000 7fa409cc5700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x562459806e40 0x5624599a60b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:27:28.484 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:28.484+0000 7fa409cc5700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x562459806e40 0x5624599a60b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.52:52188/0 (socket says 172.21.15.52:52188) 2024-05-27T07:27:28.484 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:28.484+0000 7fa409cc5700 1 -- 172.21.15.52:0/179138437 learned_addr learned my addr 172.21.15.52:0/179138437 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-05-27T07:27:28.484 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:28.484+0000 7fa40a4c6700 1 --2- 172.21.15.52:0/179138437 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x56245998ac50 0x5624599a6b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:27:28.485 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:28.484+0000 7fa4094c4700 1 --2- 172.21.15.52:0/179138437 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x562459984f80 0x5624599a65f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:27:28.485 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:28.485+0000 7fa409cc5700 1 -- 172.21.15.52:0/179138437 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x562459984f80 msgr2=0x5624599a65f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:27:28.485 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:28.485+0000 7fa409cc5700 1 --2- 172.21.15.52:0/179138437 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x562459984f80 0x5624599a65f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:27:28.485 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:28.485+0000 7fa409cc5700 1 -- 172.21.15.52:0/179138437 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x56245998ac50 msgr2=0x5624599a6b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:27:28.485 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:28.485+0000 7fa409cc5700 1 --2- 172.21.15.52:0/179138437 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x56245998ac50 0x5624599a6b30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:27:28.485 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:28.485+0000 7fa409cc5700 1 -- 172.21.15.52:0/179138437 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({osdmap=0}) -- 0x562459a083d0 con 0x562459806e40 2024-05-27T07:27:28.486 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:28.485+0000 7fa40a4c6700 1 --2- 172.21.15.52:0/179138437 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x56245998ac50 0x5624599a6b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-27T07:27:28.486 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:28.485+0000 7fa409cc5700 1 --2- 172.21.15.52:0/179138437 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x562459806e40 0x5624599a60b0 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7fa400010a70 tx=0x7fa4000047c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=35665e2051c290f8 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:27:28.486 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:28.485+0000 7fa3ed7fa700 10 client.0 ms_handle_connect on v2:172.21.15.105:3300/0 2024-05-27T07:27:28.487 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:28.485+0000 7fa3ed7fa700 1 -- 172.21.15.52:0/179138437 <== mon.2 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa400004440 con 0x562459806e40 2024-05-27T07:27:28.487 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:28.485+0000 7fa3ed7fa700 1 -- 172.21.15.52:0/179138437 <== mon.2 v2:172.21.15.105:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 9379+0+0 (secure 0 0 0) 0x7fa4000128b0 con 0x562459806e40 2024-05-27T07:27:28.487 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:28.485+0000 7fa41a93eb80 1 -- 172.21.15.52:0/179138437 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fa3fc0b2040 con 0x562459806e40 2024-05-27T07:27:28.488 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:28.486+0000 7fa3ed7fa700 1 -- 172.21.15.52:0/179138437 <== mon.2 v2:172.21.15.105:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7fa400039050 con 0x562459806e40 2024-05-27T07:27:28.488 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:28.486+0000 7fa41a93eb80 10 client.25541 fetch_fsmap learned FSMap version 30 2024-05-27T07:27:28.488 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:28.486+0000 7fa41a93eb80 1 -- 172.21.15.52:0/179138437 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x562459a09630 con 0x562459806e40 2024-05-27T07:27:28.488 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:28.486+0000 7fa3ed7fa700 1 -- 172.21.15.52:0/179138437 <== mon.2 v2:172.21.15.105:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fa3fc0b2040 con 0x562459806e40 2024-05-27T07:27:28.489 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:28.486+0000 7fa3ed7fa700 1 -- 172.21.15.52:0/179138437 <== mon.2 v2:172.21.15.105:3300/0 5 ==== fsmap(e 30) ==== 8362+0+0 (secure 0 0 0) 0x7fa40000f040 con 0x562459806e40 2024-05-27T07:27:28.489 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:28.486+0000 7fa41a93eb80 10 client.25541 fetch_fsmap finished waiting for FSMap version 30 2024-05-27T07:27:28.489 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:28.486+0000 7fa41a93eb80 10 client.25541 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.i 2024-05-27T07:27:28.489 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:28.486+0000 7fa41a93eb80 20 client.25541 populate_metadata read hostname 'smithi052' 2024-05-27T07:27:28.489 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:28.486+0000 7fa41a93eb80 1 --2- 172.21.15.52:0/179138437 >> [v2:172.21.15.173:6838/564778293,v1:172.21.15.173:6839/564778293] conn(0x56245999e890 0x562459a0bdc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:27:28.490 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:28.486+0000 7fa41a93eb80 4 client.25541 mds_command: new command op to 24475 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-05-27T07:27:28.490 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:28.486+0000 7fa41a93eb80 1 -- 172.21.15.52:0/179138437 --> [v2:172.21.15.173:6838/564778293,v1:172.21.15.173:6839/564778293] -- command(tid 0: {"prefix": "damage ls"}) -- 0x5624599509f0 con 0x56245999e890 2024-05-27T07:27:28.490 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:28.487+0000 7fa4094c4700 1 --2- 172.21.15.52:0/179138437 >> [v2:172.21.15.173:6838/564778293,v1:172.21.15.173:6839/564778293] conn(0x56245999e890 0x562459a0bdc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:27:28.490 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:28.487+0000 7fa3ed7fa700 10 client.25541 ms_handle_connect on v2:172.21.15.173:6838/564778293 2024-05-27T07:27:28.491 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:28.487+0000 7fa4094c4700 1 --2- 172.21.15.52:0/179138437 >> [v2:172.21.15.173:6838/564778293,v1:172.21.15.173:6839/564778293] conn(0x56245999e890 0x562459a0bdc0 crc :-1 s=READY pgs=35 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=7239211c4077688 server_cookie=b11f9afc5f74c55d in_seq=0 out_seq=0 2024-05-27T07:27:28.491 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:28.487+0000 7fa3ed7fa700 1 -- 172.21.15.52:0/179138437 <== mds.2 v2:172.21.15.173:6838/564778293 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x5624599509f0 con 0x56245999e890 2024-05-27T07:27:28.491 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:28.487+0000 7fa3ed7fa700 10 client.25541 handle_command_reply: tid=0 2024-05-27T07:27:28.491 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:28.487+0000 7fa41a93eb80 1 client.25541 shutdown 2024-05-27T07:27:28.491 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:28.487+0000 7fa41a93eb80 1 -- 172.21.15.52:0/179138437 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x562459806e40 msgr2=0x5624599a60b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:27:28.492 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:28.487+0000 7fa41a93eb80 1 --2- 172.21.15.52:0/179138437 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x562459806e40 0x5624599a60b0 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7fa400010a70 tx=0x7fa4000047c0 comp rx=0 tx=0).stop 2024-05-27T07:27:28.492 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:28.488+0000 7fa41a93eb80 1 -- 172.21.15.52:0/179138437 shutdown_connections 2024-05-27T07:27:28.492 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:28.488+0000 7fa41a93eb80 1 --2- 172.21.15.52:0/179138437 >> [v2:172.21.15.173:6838/564778293,v1:172.21.15.173:6839/564778293] conn(0x56245999e890 0x562459a0bdc0 crc :-1 s=READY pgs=35 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:27:28.492 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:28.488+0000 7fa41a93eb80 1 --2- 172.21.15.52:0/179138437 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x56245998ac50 0x5624599a6b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:27:28.493 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:28.488+0000 7fa41a93eb80 1 --2- 172.21.15.52:0/179138437 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x562459806e40 0x5624599a60b0 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:27:28.493 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:28.488+0000 7fa3ed7fa700 0 client.25541 ms_handle_reset on v2:172.21.15.173:6838/564778293 2024-05-27T07:27:28.493 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:28.488+0000 7fa41a93eb80 1 --2- 172.21.15.52:0/179138437 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x562459984f80 0x5624599a65f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:27:28.493 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:28.488+0000 7fa41a93eb80 1 -- 172.21.15.52:0/179138437 >> 172.21.15.52:0/179138437 conn(0x562459861af0 msgr2=0x562459989a80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:27:28.493 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:28.488+0000 7fa41a93eb80 1 -- 172.21.15.52:0/179138437 shutdown_connections 2024-05-27T07:27:28.494 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:28.488+0000 7fa41a93eb80 1 -- 172.21.15.52:0/179138437 wait complete. 2024-05-27T07:27:28.494 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:28.488+0000 7fa41a93eb80 20 client.25541 trim_cache size 0 max 16384 2024-05-27T07:27:28.495 INFO:teuthology.orchestra.run.smithi052.stdout:[] 2024-05-27T07:27:28.497 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:28.489+0000 7fa40bf29700 1 -- 172.21.15.52:0/3263786524 >> [v2:172.21.15.52:6800/1822318384,v1:172.21.15.52:6801/1822318384] conn(0x7fa3f00b2c20 msgr2=0x7fa3f00b5130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:27:28.497 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:28.489+0000 7fa40bf29700 1 --2- 172.21.15.52:0/3263786524 >> [v2:172.21.15.52:6800/1822318384,v1:172.21.15.52:6801/1822318384] conn(0x7fa3f00b2c20 0x7fa3f00b5130 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7fa400009be0 tx=0x7fa400009260 comp rx=0 tx=0).stop 2024-05-27T07:27:28.497 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:28.489+0000 7fa40bf29700 1 -- 172.21.15.52:0/3263786524 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fa404108730 msgr2=0x7fa4041a5c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:27:28.497 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:28.489+0000 7fa40bf29700 1 --2- 172.21.15.52:0/3263786524 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fa404108730 0x7fa4041a5c40 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7fa3fc007ee0 tx=0x7fa3fc00d3b0 comp rx=0 tx=0).stop 2024-05-27T07:27:28.498 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:28.489+0000 7fa40bf29700 1 -- 172.21.15.52:0/3263786524 shutdown_connections 2024-05-27T07:27:28.498 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:28.489+0000 7fa40bf29700 1 --2- 172.21.15.52:0/3263786524 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fa404108730 0x7fa4041a5c40 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:27:28.498 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:28.489+0000 7fa40bf29700 1 --2- 172.21.15.52:0/3263786524 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fa4041034b0 0x7fa4041a1410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:27:28.498 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:28.489+0000 7fa40bf29700 1 --2- 172.21.15.52:0/3263786524 >> [v2:172.21.15.52:6800/1822318384,v1:172.21.15.52:6801/1822318384] conn(0x7fa3f00b2c20 0x7fa3f00b5130 unknown :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:27:28.499 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:28.489+0000 7fa40bf29700 1 --2- 172.21.15.52:0/3263786524 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fa404107de0 0x7fa4041a19b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:27:28.499 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:28.489+0000 7fa40bf29700 1 -- 172.21.15.52:0/3263786524 >> 172.21.15.52:0/3263786524 conn(0x7fa4040fedb0 msgr2=0x7fa4041011d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:27:28.499 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:28.489+0000 7fa40bf29700 1 -- 172.21.15.52:0/3263786524 shutdown_connections 2024-05-27T07:27:28.499 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:28.489+0000 7fa40bf29700 1 -- 172.21.15.52:0/3263786524 wait complete. 2024-05-27T07:27:28.601 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:27:28 smithi173 ceph-mon[37000]: pgmap v48: 129 pgs: 129 active+clean; 1.4 MiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 255 B/s wr, 8 op/s 2024-05-27T07:27:29.171 DEBUG:teuthology.orchestra.run.smithi052:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:38f9d900c83c63799fbdbe61acc9a11b0d3554a6 shell --fsid 73df2636-1bf8-11ef-bc9b-c7b262605968 -- ceph tell mds.1:1 damage ls 2024-05-27T07:27:29.451 INFO:teuthology.orchestra.run.smithi052.stderr:Inferring config /var/lib/ceph/73df2636-1bf8-11ef-bc9b-c7b262605968/mon.a/config 2024-05-27T07:27:30.229 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:27:30 smithi052 ceph-mon[31955]: pgmap v49: 129 pgs: 129 active+clean; 1.4 MiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 6.7 KiB/s rd, 1.2 KiB/s wr, 14 op/s 2024-05-27T07:27:30.229 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:27:30 smithi052 ceph-mon[31955]: mgrmap e26: x(active, since 92s), standbys: z, y 2024-05-27T07:27:30.440 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:27:30 smithi105 ceph-mon[35977]: pgmap v49: 129 pgs: 129 active+clean; 1.4 MiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 6.7 KiB/s rd, 1.2 KiB/s wr, 14 op/s 2024-05-27T07:27:30.440 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:27:30 smithi105 ceph-mon[35977]: mgrmap e26: x(active, since 92s), standbys: z, y 2024-05-27T07:27:30.602 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:27:30 smithi173 ceph-mon[37000]: pgmap v49: 129 pgs: 129 active+clean; 1.4 MiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 6.7 KiB/s rd, 1.2 KiB/s wr, 14 op/s 2024-05-27T07:27:30.602 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:27:30 smithi173 ceph-mon[37000]: mgrmap e26: x(active, since 92s), standbys: z, y 2024-05-27T07:27:31.234 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:31.244+0000 7f6740f5b700 1 -- 172.21.15.52:0/853275082 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f673c1034d0 msgr2=0x7f673c1038e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:27:31.235 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:31.244+0000 7f6740f5b700 1 --2- 172.21.15.52:0/853275082 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f673c1034d0 0x7f673c1038e0 secure :-1 s=READY pgs=529 cs=0 l=1 rev1=1 crypto rx=0x7f672c0099c0 tx=0x7f672c009cd0 comp rx=0 tx=0).stop 2024-05-27T07:27:31.235 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:31.244+0000 7f6740f5b700 1 -- 172.21.15.52:0/853275082 shutdown_connections 2024-05-27T07:27:31.235 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:31.244+0000 7f6740f5b700 1 --2- 172.21.15.52:0/853275082 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f673c108750 0x7f673c1100b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:27:31.235 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:31.244+0000 7f6740f5b700 1 --2- 172.21.15.52:0/853275082 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f673c1034d0 0x7f673c1038e0 unknown :-1 s=CLOSED pgs=529 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:27:31.235 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:31.244+0000 7f6740f5b700 1 --2- 172.21.15.52:0/853275082 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f673c107e00 0x7f673c108210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:27:31.235 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:31.244+0000 7f6740f5b700 1 -- 172.21.15.52:0/853275082 >> 172.21.15.52:0/853275082 conn(0x7f673c0fef20 msgr2=0x7f673c101340 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:27:31.235 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:31.244+0000 7f6740f5b700 1 -- 172.21.15.52:0/853275082 shutdown_connections 2024-05-27T07:27:31.235 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:31.244+0000 7f6740f5b700 1 -- 172.21.15.52:0/853275082 wait complete. 2024-05-27T07:27:31.236 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:31.246+0000 7f6740f5b700 1 Processor -- start 2024-05-27T07:27:31.236 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:31.246+0000 7f6740f5b700 1 -- start start 2024-05-27T07:27:31.238 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:31.246+0000 7f6740f5b700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f673c1034d0 0x7f673c19ae00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:27:31.238 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:31.246+0000 7f6740f5b700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f673c107e00 0x7f673c19b3a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:27:31.238 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:31.246+0000 7f6740f5b700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f673c108750 0x7f673c19f630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:27:31.238 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:31.247+0000 7f6740f5b700 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x7f673c1144d0 con 0x7f673c108750 2024-05-27T07:27:31.239 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:31.247+0000 7f6740f5b700 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f673c114350 con 0x7f673c107e00 2024-05-27T07:27:31.239 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:31.247+0000 7f6740f5b700 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f673c114650 con 0x7f673c1034d0 2024-05-27T07:27:31.239 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:31.247+0000 7f673ad9d700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f673c108750 0x7f673c19f630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:27:31.239 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:31.247+0000 7f6739d9b700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f673c107e00 0x7f673c19b3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:27:31.239 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:31.247+0000 7f673ad9d700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f673c108750 0x7f673c19f630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.52:53014/0 (socket says 172.21.15.52:53014) 2024-05-27T07:27:31.239 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:31.247+0000 7f673ad9d700 1 -- 172.21.15.52:0/718704019 learned_addr learned my addr 172.21.15.52:0/718704019 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-05-27T07:27:31.239 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:31.247+0000 7f673a59c700 1 --2- 172.21.15.52:0/718704019 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f673c1034d0 0x7f673c19ae00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:27:31.239 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:31.247+0000 7f673ad9d700 1 -- 172.21.15.52:0/718704019 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f673c1034d0 msgr2=0x7f673c19ae00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:27:31.240 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:31.247+0000 7f673ad9d700 1 --2- 172.21.15.52:0/718704019 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f673c1034d0 0x7f673c19ae00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:27:31.240 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:31.247+0000 7f673ad9d700 1 -- 172.21.15.52:0/718704019 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f673c107e00 msgr2=0x7f673c19b3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:27:31.240 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:31.247+0000 7f673ad9d700 1 --2- 172.21.15.52:0/718704019 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f673c107e00 0x7f673c19b3a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:27:31.240 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:31.247+0000 7f673ad9d700 1 -- 172.21.15.52:0/718704019 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f673c19fcd0 con 0x7f673c108750 2024-05-27T07:27:31.240 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:31.248+0000 7f673ad9d700 1 --2- 172.21.15.52:0/718704019 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f673c108750 0x7f673c19f630 secure :-1 s=READY pgs=530 cs=0 l=1 rev1=1 crypto rx=0x7f673400ef30 tx=0x7f673400c3a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:27:31.241 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:31.248+0000 7f672b7fe700 1 -- 172.21.15.52:0/718704019 <== mon.0 v2:172.21.15.52:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f67340092e0 con 0x7f673c108750 2024-05-27T07:27:31.241 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:31.248+0000 7f672b7fe700 1 -- 172.21.15.52:0/718704019 <== mon.0 v2:172.21.15.52:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f6734004bb0 con 0x7f673c108750 2024-05-27T07:27:31.241 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:31.248+0000 7f672b7fe700 1 -- 172.21.15.52:0/718704019 <== mon.0 v2:172.21.15.52:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6734004280 con 0x7f673c108750 2024-05-27T07:27:31.241 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:31.248+0000 7f6740f5b700 1 -- 172.21.15.52:0/718704019 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f673c19ff60 con 0x7f673c108750 2024-05-27T07:27:31.241 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:31.248+0000 7f6740f5b700 1 -- 172.21.15.52:0/718704019 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f673c1a0450 con 0x7f673c108750 2024-05-27T07:27:31.241 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:31.251+0000 7f672b7fe700 1 -- 172.21.15.52:0/718704019 <== mon.0 v2:172.21.15.52:3300/0 4 ==== mgrmap(e 26) ==== 149783+0+0 (secure 0 0 0) 0x7f673401b030 con 0x7f673c108750 2024-05-27T07:27:31.242 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:31.253+0000 7f672b7fe700 1 --2- 172.21.15.52:0/718704019 >> [v2:172.21.15.52:6800/1822318384,v1:172.21.15.52:6801/1822318384] conn(0x7f67240b2af0 0x7f67240b5000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:27:31.243 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:31.253+0000 7f672b7fe700 1 -- 172.21.15.52:0/718704019 <== mon.0 v2:172.21.15.52:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9379+0+0 (secure 0 0 0) 0x7f67340e3320 con 0x7f673c108750 2024-05-27T07:27:31.243 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:31.253+0000 7f673a59c700 1 --2- 172.21.15.52:0/718704019 >> [v2:172.21.15.52:6800/1822318384,v1:172.21.15.52:6801/1822318384] conn(0x7f67240b2af0 0x7f67240b5000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:27:31.243 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:31.254+0000 7f674f970b80 1 Processor -- start 2024-05-27T07:27:31.243 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:31.254+0000 7f674f970b80 1 -- start start 2024-05-27T07:27:31.244 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:31.254+0000 7f674f970b80 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x560ce32bce40 0x560ce3445fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:27:31.244 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:31.254+0000 7f674f970b80 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x560ce34464f0 0x560ce343fff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:27:31.244 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:31.254+0000 7f674f970b80 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x560ce3440530 0x560ce3440940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:27:31.244 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:31.254+0000 7f674f970b80 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x560ce3369d70 con 0x560ce34464f0 2024-05-27T07:27:31.244 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:31.254+0000 7f674f970b80 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x560ce3359910 con 0x560ce3440530 2024-05-27T07:27:31.244 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:31.254+0000 7f674f970b80 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x560ce335d950 con 0x560ce32bce40 2024-05-27T07:27:31.245 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:31.254+0000 7f6739d9b700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x560ce34464f0 0x560ce343fff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:27:31.245 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:31.254+0000 7f6739d9b700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x560ce34464f0 0x560ce343fff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.52:53018/0 (socket says 172.21.15.52:53018) 2024-05-27T07:27:31.245 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:31.254+0000 7f6739d9b700 1 -- 172.21.15.52:0/652040820 learned_addr learned my addr 172.21.15.52:0/652040820 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-05-27T07:27:31.245 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:31.255+0000 7f673ad9d700 1 --2- 172.21.15.52:0/652040820 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x560ce3440530 0x560ce3440940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:27:31.246 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:31.255+0000 7f673a59c700 1 --2- 172.21.15.52:0/718704019 >> [v2:172.21.15.52:6800/1822318384,v1:172.21.15.52:6801/1822318384] conn(0x7f67240b2af0 0x7f67240b5000 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f672c003760 tx=0x7f672c01a040 comp rx=0 tx=0).ready entity=mgr.15333 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:27:31.246 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:31.255+0000 7f6739d9b700 1 -- 172.21.15.52:0/652040820 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x560ce32bce40 msgr2=0x560ce3445fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:27:31.246 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:31.255+0000 7f6739d9b700 1 --2- 172.21.15.52:0/652040820 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x560ce32bce40 0x560ce3445fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:27:31.246 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:31.255+0000 7f6739d9b700 1 -- 172.21.15.52:0/652040820 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x560ce3440530 msgr2=0x560ce3440940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:27:31.246 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:31.255+0000 7f6739d9b700 1 --2- 172.21.15.52:0/652040820 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x560ce3440530 0x560ce3440940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:27:31.246 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:31.255+0000 7f6739d9b700 1 -- 172.21.15.52:0/652040820 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x560ce3302fb0 con 0x560ce34464f0 2024-05-27T07:27:31.246 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:31.255+0000 7f6739d9b700 1 --2- 172.21.15.52:0/652040820 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x560ce34464f0 0x560ce343fff0 secure :-1 s=READY pgs=531 cs=0 l=1 rev1=1 crypto rx=0x7f6730007c90 tx=0x7f67300103d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=10d8ff1969441cce server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:27:31.246 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:31.256+0000 7f6728ff9700 1 -- 172.21.15.52:0/652040820 <== mon.0 v2:172.21.15.52:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6730010cb0 con 0x560ce34464f0 2024-05-27T07:27:31.246 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:31.256+0000 7f6728ff9700 1 -- 172.21.15.52:0/652040820 <== mon.0 v2:172.21.15.52:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f6730015070 con 0x560ce34464f0 2024-05-27T07:27:31.246 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:31.256+0000 7f6728ff9700 1 -- 172.21.15.52:0/652040820 <== mon.0 v2:172.21.15.52:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6730003d90 con 0x560ce34464f0 2024-05-27T07:27:31.247 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:31.257+0000 7f674f970b80 1 -- 172.21.15.52:0/652040820 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x560ce34464f0 msgr2=0x560ce343fff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:27:31.247 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:31.257+0000 7f674f970b80 1 --2- 172.21.15.52:0/652040820 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x560ce34464f0 0x560ce343fff0 secure :-1 s=READY pgs=531 cs=0 l=1 rev1=1 crypto rx=0x7f6730007c90 tx=0x7f67300103d0 comp rx=0 tx=0).stop 2024-05-27T07:27:31.247 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:31.257+0000 7f674f970b80 1 -- 172.21.15.52:0/652040820 shutdown_connections 2024-05-27T07:27:31.247 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:31.257+0000 7f674f970b80 1 --2- 172.21.15.52:0/652040820 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x560ce3440530 0x560ce3440940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:27:31.247 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:31.257+0000 7f674f970b80 1 --2- 172.21.15.52:0/652040820 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x560ce32bce40 0x560ce3445fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:27:31.247 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:31.257+0000 7f674f970b80 1 --2- 172.21.15.52:0/652040820 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x560ce34464f0 0x560ce343fff0 secure :-1 s=CLOSED pgs=531 cs=0 l=1 rev1=1 crypto rx=0x7f6730007c90 tx=0x7f67300103d0 comp rx=0 tx=0).stop 2024-05-27T07:27:31.248 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:31.257+0000 7f674f970b80 1 -- 172.21.15.52:0/652040820 >> 172.21.15.52:0/652040820 conn(0x560ce32bf770 msgr2=0x560ce3436df0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:27:31.248 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:31.257+0000 7f674f970b80 1 -- 172.21.15.52:0/652040820 shutdown_connections 2024-05-27T07:27:31.248 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:31.257+0000 7f674f970b80 1 -- 172.21.15.52:0/652040820 wait complete. 2024-05-27T07:27:31.248 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:31.257+0000 7f674f970b80 1 -- start start 2024-05-27T07:27:31.248 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:31.258+0000 7f674f970b80 1 Processor -- start 2024-05-27T07:27:31.251 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:31.258+0000 7f674f970b80 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x560ce32bce40 0x560ce3449d60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:27:31.251 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:31.258+0000 7f674f970b80 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x560ce3440530 0x560ce344a2a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:27:31.251 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:31.258+0000 7f674f970b80 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x560ce344a7e0 0x560ce3456380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:27:31.251 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:31.258+0000 7f674f970b80 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x560ce325ffa0 con 0x560ce32bce40 2024-05-27T07:27:31.251 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:31.258+0000 7f674f970b80 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x560ce32c1da0 con 0x560ce3440530 2024-05-27T07:27:31.251 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:31.258+0000 7f674f970b80 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x560ce344cfc0 con 0x560ce344a7e0 2024-05-27T07:27:31.251 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:31.258+0000 7f6739d9b700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x560ce3440530 0x560ce344a2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:27:31.251 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:31.258+0000 7f673a59c700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x560ce32bce40 0x560ce3449d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:27:31.251 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:31.258+0000 7f6739d9b700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x560ce3440530 0x560ce344a2a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.52:42158/0 (socket says 172.21.15.52:42158) 2024-05-27T07:27:31.251 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:31.258+0000 7f6739d9b700 1 -- 172.21.15.52:0/268023885 learned_addr learned my addr 172.21.15.52:0/268023885 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-05-27T07:27:31.251 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:31.259+0000 7f673ad9d700 1 --2- 172.21.15.52:0/268023885 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x560ce344a7e0 0x560ce3456380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:27:31.252 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:31.259+0000 7f674f970b80 10 client.0 _ll_register_callbacks cb 0x560ce32f05b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T07:27:31.252 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:31.259+0000 7f6739d9b700 1 -- 172.21.15.52:0/268023885 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x560ce344a7e0 msgr2=0x560ce3456380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:27:31.252 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:31.259+0000 7f6739d9b700 1 --2- 172.21.15.52:0/268023885 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x560ce344a7e0 0x560ce3456380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:27:31.252 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:31.259+0000 7f6739d9b700 1 -- 172.21.15.52:0/268023885 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x560ce32bce40 msgr2=0x560ce3449d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:27:31.252 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:31.259+0000 7f6739d9b700 1 --2- 172.21.15.52:0/268023885 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x560ce32bce40 0x560ce3449d60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:27:31.252 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:31.259+0000 7f6739d9b700 1 -- 172.21.15.52:0/268023885 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({osdmap=0}) -- 0x560ce32a98d0 con 0x560ce3440530 2024-05-27T07:27:31.252 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:31.259+0000 7f673a59c700 1 --2- 172.21.15.52:0/268023885 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x560ce32bce40 0x560ce3449d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-27T07:27:31.252 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:31.259+0000 7f6739d9b700 1 --2- 172.21.15.52:0/268023885 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x560ce3440530 0x560ce344a2a0 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f6730000c00 tx=0x7f6730010d90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c90fcf0f3589e610 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:27:31.252 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:31.259+0000 7f6721ffb700 10 client.25553 ms_handle_connect on v2:172.21.15.105:3300/0 2024-05-27T07:27:31.252 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:31.259+0000 7f674f970b80 1 -- 172.21.15.52:0/268023885 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f6730015070 con 0x560ce3440530 2024-05-27T07:27:31.252 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:31.259+0000 7f6721ffb700 1 -- 172.21.15.52:0/268023885 <== mon.2 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6730028060 con 0x560ce3440530 2024-05-27T07:27:31.252 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:31.259+0000 7f6721ffb700 1 -- 172.21.15.52:0/268023885 <== mon.2 v2:172.21.15.105:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 9379+0+0 (secure 0 0 0) 0x7f6730003c20 con 0x560ce3440530 2024-05-27T07:27:31.252 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:31.260+0000 7f6721ffb700 1 -- 172.21.15.52:0/268023885 <== mon.2 v2:172.21.15.105:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7f6730003e60 con 0x560ce3440530 2024-05-27T07:27:31.253 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:31.260+0000 7f674f970b80 10 client.25553 fetch_fsmap learned FSMap version 30 2024-05-27T07:27:31.253 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:31.260+0000 7f674f970b80 1 -- 172.21.15.52:0/268023885 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({fsmap=30}) -- 0x560ce3313760 con 0x560ce3440530 2024-05-27T07:27:31.253 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:31.260+0000 7f6721ffb700 1 -- 172.21.15.52:0/268023885 <== mon.2 v2:172.21.15.105:3300/0 4 ==== fsmap(e 30) ==== 8362+0+0 (secure 0 0 0) 0x7f6730004050 con 0x560ce3440530 2024-05-27T07:27:31.253 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:31.260+0000 7f674f970b80 10 client.25553 fetch_fsmap finished waiting for FSMap version 30 2024-05-27T07:27:31.253 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:31.260+0000 7f674f970b80 10 client.25553 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.h 2024-05-27T07:27:31.253 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:31.260+0000 7f674f970b80 20 client.25553 populate_metadata read hostname 'smithi052' 2024-05-27T07:27:31.253 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:31.260+0000 7f674f970b80 1 --2- 172.21.15.52:0/268023885 >> [v2:172.21.15.105:6836/191393149,v1:172.21.15.105:6837/191393149] conn(0x560ce345a2f0 0x560ce345c710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:27:31.253 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:31.260+0000 7f674f970b80 4 client.25553 mds_command: new command op to 24497 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-27T07:27:31.253 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:31.260+0000 7f674f970b80 1 -- 172.21.15.52:0/268023885 --> [v2:172.21.15.105:6836/191393149,v1:172.21.15.105:6837/191393149] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x560ce336bcc0 con 0x560ce345a2f0 2024-05-27T07:27:31.253 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:31.260+0000 7f673a59c700 1 --2- 172.21.15.52:0/268023885 >> [v2:172.21.15.105:6836/191393149,v1:172.21.15.105:6837/191393149] conn(0x560ce345a2f0 0x560ce345c710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:27:31.253 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:31.261+0000 7f6721ffb700 10 client.25553 ms_handle_connect on v2:172.21.15.105:6836/191393149 2024-05-27T07:27:31.253 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:31.261+0000 7f673a59c700 1 --2- 172.21.15.52:0/268023885 >> [v2:172.21.15.105:6836/191393149,v1:172.21.15.105:6837/191393149] conn(0x560ce345a2f0 0x560ce345c710 crc :-1 s=READY pgs=32 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=be983df5ffb99bcd server_cookie=f401840c4f065d8a in_seq=0 out_seq=0 2024-05-27T07:27:31.255 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:31.262+0000 7f6721ffb700 1 -- 172.21.15.52:0/268023885 <== mds.1 v2:172.21.15.105:6836/191393149 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24041 (crc 0 0 0) 0x560ce336bcc0 con 0x560ce345a2f0 2024-05-27T07:27:31.255 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:31.262+0000 7f6721ffb700 10 client.25553 handle_command_reply: tid=0 2024-05-27T07:27:31.255 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:31.262+0000 7f674f970b80 1 client.25553 shutdown 2024-05-27T07:27:31.255 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:31.262+0000 7f674f970b80 1 -- 172.21.15.52:0/268023885 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x560ce3440530 msgr2=0x560ce344a2a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:27:31.255 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:31.262+0000 7f674f970b80 1 --2- 172.21.15.52:0/268023885 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x560ce3440530 0x560ce344a2a0 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f6730000c00 tx=0x7f6730010d90 comp rx=0 tx=0).stop 2024-05-27T07:27:31.255 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:31.262+0000 7f674f970b80 1 -- 172.21.15.52:0/268023885 shutdown_connections 2024-05-27T07:27:31.255 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:31.262+0000 7f674f970b80 1 --2- 172.21.15.52:0/268023885 >> [v2:172.21.15.105:6836/191393149,v1:172.21.15.105:6837/191393149] conn(0x560ce345a2f0 0x560ce345c710 crc :-1 s=READY pgs=32 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:27:31.255 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:31.262+0000 7f674f970b80 1 --2- 172.21.15.52:0/268023885 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x560ce344a7e0 0x560ce3456380 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:27:31.255 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:31.262+0000 7f674f970b80 1 --2- 172.21.15.52:0/268023885 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x560ce32bce40 0x560ce3449d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:27:31.255 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:31.262+0000 7f674f970b80 1 --2- 172.21.15.52:0/268023885 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x560ce3440530 0x560ce344a2a0 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:27:31.255 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:31.262+0000 7f674f970b80 1 -- 172.21.15.52:0/268023885 >> 172.21.15.52:0/268023885 conn(0x560ce32bf770 msgr2=0x560ce343efe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:27:31.255 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:31.262+0000 7f6721ffb700 0 client.25553 ms_handle_reset on v2:172.21.15.105:6836/191393149 2024-05-27T07:27:31.256 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:31.262+0000 7f674f970b80 1 -- 172.21.15.52:0/268023885 shutdown_connections 2024-05-27T07:27:31.256 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:31.262+0000 7f674f970b80 1 -- 172.21.15.52:0/268023885 wait complete. 2024-05-27T07:27:31.256 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:31.263+0000 7f674f970b80 20 client.25553 trim_cache size 0 max 16384 2024-05-27T07:27:31.283 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:31.293+0000 7f674f970b80 1 Processor -- start 2024-05-27T07:27:31.283 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:31.293+0000 7f674f970b80 1 -- start start 2024-05-27T07:27:31.283 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:31.293+0000 7f674f970b80 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x560ce32bce40 0x560ce343fff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:27:31.283 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:31.293+0000 7f674f970b80 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x560ce3440530 0x560ce34490a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:27:31.284 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:31.294+0000 7f674f970b80 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x560ce344a7e0 0x560ce34562f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:27:31.284 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:31.294+0000 7f674f970b80 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x7f671c00eb90 con 0x560ce3440530 2024-05-27T07:27:31.284 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:31.294+0000 7f674f970b80 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f671c0106d0 con 0x560ce32bce40 2024-05-27T07:27:31.284 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:31.294+0000 7f674f970b80 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f671c00f260 con 0x560ce344a7e0 2024-05-27T07:27:31.285 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:31.294+0000 7f6739d9b700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x560ce3440530 0x560ce34490a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:27:31.285 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:31.294+0000 7f6739d9b700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x560ce3440530 0x560ce34490a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.52:53036/0 (socket says 172.21.15.52:53036) 2024-05-27T07:27:31.285 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:31.294+0000 7f6739d9b700 1 -- 172.21.15.52:0/4245622005 learned_addr learned my addr 172.21.15.52:0/4245622005 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-05-27T07:27:31.285 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:31.294+0000 7f673a59c700 1 --2- 172.21.15.52:0/4245622005 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x560ce32bce40 0x560ce343fff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:27:31.285 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:31.294+0000 7f673ad9d700 1 --2- 172.21.15.52:0/4245622005 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x560ce344a7e0 0x560ce34562f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:27:31.285 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:31.294+0000 7f6739d9b700 1 -- 172.21.15.52:0/4245622005 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x560ce344a7e0 msgr2=0x560ce34562f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:27:31.285 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:31.294+0000 7f6739d9b700 1 --2- 172.21.15.52:0/4245622005 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x560ce344a7e0 0x560ce34562f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:27:31.285 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:31.294+0000 7f6739d9b700 1 -- 172.21.15.52:0/4245622005 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x560ce32bce40 msgr2=0x560ce343fff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:27:31.285 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:31.294+0000 7f6739d9b700 1 --2- 172.21.15.52:0/4245622005 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x560ce32bce40 0x560ce343fff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:27:31.285 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:31.294+0000 7f6739d9b700 1 -- 172.21.15.52:0/4245622005 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x560ce34411f0 con 0x560ce3440530 2024-05-27T07:27:31.285 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:31.294+0000 7f673a59c700 1 --2- 172.21.15.52:0/4245622005 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x560ce32bce40 0x560ce343fff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-27T07:27:31.285 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:31.295+0000 7f6739d9b700 1 --2- 172.21.15.52:0/4245622005 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x560ce3440530 0x560ce34490a0 secure :-1 s=READY pgs=532 cs=0 l=1 rev1=1 crypto rx=0x7f67300059d0 tx=0x7f673001f000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=12f0953695e6ddb server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:27:31.286 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:31.295+0000 7f6723fff700 1 -- 172.21.15.52:0/4245622005 <== mon.0 v2:172.21.15.52:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6730004000 con 0x560ce3440530 2024-05-27T07:27:31.286 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:31.295+0000 7f6723fff700 1 -- 172.21.15.52:0/4245622005 <== mon.0 v2:172.21.15.52:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f6730015070 con 0x560ce3440530 2024-05-27T07:27:31.286 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:31.295+0000 7f6723fff700 1 -- 172.21.15.52:0/4245622005 <== mon.0 v2:172.21.15.52:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6730004a00 con 0x560ce3440530 2024-05-27T07:27:31.286 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:31.296+0000 7f674f970b80 1 -- 172.21.15.52:0/4245622005 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x560ce3440530 msgr2=0x560ce34490a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:27:31.286 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:31.296+0000 7f674f970b80 1 --2- 172.21.15.52:0/4245622005 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x560ce3440530 0x560ce34490a0 secure :-1 s=READY pgs=532 cs=0 l=1 rev1=1 crypto rx=0x7f67300059d0 tx=0x7f673001f000 comp rx=0 tx=0).stop 2024-05-27T07:27:31.287 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:31.296+0000 7f674f970b80 1 -- 172.21.15.52:0/4245622005 shutdown_connections 2024-05-27T07:27:31.287 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:31.296+0000 7f674f970b80 1 --2- 172.21.15.52:0/4245622005 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x560ce344a7e0 0x560ce34562f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:27:31.287 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:31.296+0000 7f674f970b80 1 --2- 172.21.15.52:0/4245622005 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x560ce32bce40 0x560ce343fff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:27:31.287 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:31.296+0000 7f674f970b80 1 --2- 172.21.15.52:0/4245622005 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x560ce3440530 0x560ce34490a0 unknown :-1 s=CLOSED pgs=532 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:27:31.287 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:31.296+0000 7f674f970b80 1 -- 172.21.15.52:0/4245622005 >> 172.21.15.52:0/4245622005 conn(0x560ce32bf770 msgr2=0x560ce34304f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:27:31.287 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:31.297+0000 7f674f970b80 1 -- 172.21.15.52:0/4245622005 shutdown_connections 2024-05-27T07:27:31.287 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:31.297+0000 7f674f970b80 1 -- 172.21.15.52:0/4245622005 wait complete. 2024-05-27T07:27:31.292 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:31.297+0000 7f674f970b80 1 -- start start 2024-05-27T07:27:31.292 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:31.297+0000 7f674f970b80 1 Processor -- start 2024-05-27T07:27:31.292 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:31.298+0000 7f674f970b80 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x560ce32bce40 0x560ce343c520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:27:31.292 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:31.298+0000 7f674f970b80 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x560ce3440530 0x560ce343a8b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:27:31.292 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:31.298+0000 7f673a59c700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x560ce32bce40 0x560ce343c520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:27:31.292 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:31.298+0000 7f673a59c700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x560ce32bce40 0x560ce343c520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.52:53050/0 (socket says 172.21.15.52:53050) 2024-05-27T07:27:31.292 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:31.298+0000 7f673a59c700 1 -- 172.21.15.52:0/3639317842 learned_addr learned my addr 172.21.15.52:0/3639317842 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-05-27T07:27:31.292 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:31.298+0000 7f674f970b80 1 --2- 172.21.15.52:0/3639317842 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x560ce344a7e0 0x560ce343adf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:27:31.293 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:31.298+0000 7f674f970b80 1 -- 172.21.15.52:0/3639317842 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x7f671c010000 con 0x560ce32bce40 2024-05-27T07:27:31.293 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:31.298+0000 7f674f970b80 1 -- 172.21.15.52:0/3639317842 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f671c00f930 con 0x560ce3440530 2024-05-27T07:27:31.293 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:31.298+0000 7f674f970b80 1 -- 172.21.15.52:0/3639317842 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f671c00cb30 con 0x560ce344a7e0 2024-05-27T07:27:31.293 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:31.298+0000 7f6739d9b700 1 --2- 172.21.15.52:0/3639317842 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x560ce3440530 0x560ce343a8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:27:31.293 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:31.299+0000 7f6739d9b700 1 -- 172.21.15.52:0/3639317842 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x560ce344a7e0 msgr2=0x560ce343adf0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-05-27T07:27:31.293 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:31.299+0000 7f6739d9b700 1 --2- 172.21.15.52:0/3639317842 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x560ce344a7e0 0x560ce343adf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:27:31.293 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:31.299+0000 7f6739d9b700 1 -- 172.21.15.52:0/3639317842 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x560ce32bce40 msgr2=0x560ce343c520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:27:31.293 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:31.299+0000 7f6739d9b700 1 --2- 172.21.15.52:0/3639317842 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x560ce32bce40 0x560ce343c520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:27:31.293 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:31.299+0000 7f6739d9b700 1 -- 172.21.15.52:0/3639317842 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({osdmap=0}) -- 0x560ce343b330 con 0x560ce3440530 2024-05-27T07:27:31.293 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:31.299+0000 7f673a59c700 1 --2- 172.21.15.52:0/3639317842 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x560ce32bce40 0x560ce343c520 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-27T07:27:31.293 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:31.299+0000 7f674f970b80 10 client.0 _ll_register_callbacks cb 0x560ce32f05b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T07:27:31.294 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:31.299+0000 7f674f970b80 1 -- 172.21.15.52:0/3639317842 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f6730015070 con 0x560ce3440530 2024-05-27T07:27:31.294 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:31.299+0000 7f6739d9b700 1 --2- 172.21.15.52:0/3639317842 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x560ce3440530 0x560ce343a8b0 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f6730004520 tx=0x7f673001a6a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d4e7be7a2178e824 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:27:31.294 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:31.299+0000 7f6721ffb700 10 client.25559 ms_handle_connect on v2:172.21.15.105:3300/0 2024-05-27T07:27:31.294 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:31.299+0000 7f6721ffb700 1 -- 172.21.15.52:0/3639317842 <== mon.2 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6730028040 con 0x560ce3440530 2024-05-27T07:27:31.294 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:31.299+0000 7f6721ffb700 1 -- 172.21.15.52:0/3639317842 <== mon.2 v2:172.21.15.105:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 9379+0+0 (secure 0 0 0) 0x7f6730005440 con 0x560ce3440530 2024-05-27T07:27:31.294 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:31.300+0000 7f6721ffb700 1 -- 172.21.15.52:0/3639317842 <== mon.2 v2:172.21.15.105:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7f67300045a0 con 0x560ce3440530 2024-05-27T07:27:31.294 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:31.300+0000 7f674f970b80 10 client.25559 fetch_fsmap learned FSMap version 30 2024-05-27T07:27:31.294 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:31.300+0000 7f674f970b80 1 -- 172.21.15.52:0/3639317842 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({fsmap=30}) -- 0x560ce345f090 con 0x560ce3440530 2024-05-27T07:27:31.294 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:31.300+0000 7f6721ffb700 1 -- 172.21.15.52:0/3639317842 <== mon.2 v2:172.21.15.105:3300/0 4 ==== fsmap(e 30) ==== 8362+0+0 (secure 0 0 0) 0x7f6730004740 con 0x560ce3440530 2024-05-27T07:27:31.294 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:31.300+0000 7f674f970b80 10 client.25559 fetch_fsmap finished waiting for FSMap version 30 2024-05-27T07:27:31.294 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:31.300+0000 7f674f970b80 10 client.25559 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.h 2024-05-27T07:27:31.294 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:31.300+0000 7f674f970b80 20 client.25559 populate_metadata read hostname 'smithi052' 2024-05-27T07:27:31.294 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:31.300+0000 7f674f970b80 1 --2- 172.21.15.52:0/3639317842 >> [v2:172.21.15.105:6836/191393149,v1:172.21.15.105:6837/191393149] conn(0x560ce345f5e0 0x560ce345fa80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:27:31.295 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:31.300+0000 7f674f970b80 4 client.25559 mds_command: new command op to 24497 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-05-27T07:27:31.295 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:31.300+0000 7f674f970b80 1 -- 172.21.15.52:0/3639317842 --> [v2:172.21.15.105:6836/191393149,v1:172.21.15.105:6837/191393149] -- command(tid 0: {"prefix": "damage ls"}) -- 0x560ce33654a0 con 0x560ce345f5e0 2024-05-27T07:27:31.295 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:31.300+0000 7f673a59c700 1 --2- 172.21.15.52:0/3639317842 >> [v2:172.21.15.105:6836/191393149,v1:172.21.15.105:6837/191393149] conn(0x560ce345f5e0 0x560ce345fa80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:27:31.295 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:31.301+0000 7f673a59c700 1 --2- 172.21.15.52:0/3639317842 >> [v2:172.21.15.105:6836/191393149,v1:172.21.15.105:6837/191393149] conn(0x560ce345f5e0 0x560ce345fa80 crc :-1 s=READY pgs=33 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=55d022444fb453ec server_cookie=c45e15b7685245cf in_seq=0 out_seq=0 2024-05-27T07:27:31.295 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:31.301+0000 7f6721ffb700 10 client.25559 ms_handle_connect on v2:172.21.15.105:6836/191393149 2024-05-27T07:27:31.295 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:31.301+0000 7f6721ffb700 1 -- 172.21.15.52:0/3639317842 <== mds.1 v2:172.21.15.105:6836/191393149 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x560ce33654a0 con 0x560ce345f5e0 2024-05-27T07:27:31.295 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:31.301+0000 7f6721ffb700 10 client.25559 handle_command_reply: tid=0 2024-05-27T07:27:31.295 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:31.301+0000 7f674f970b80 1 client.25559 shutdown 2024-05-27T07:27:31.295 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:31.302+0000 7f674f970b80 1 -- 172.21.15.52:0/3639317842 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x560ce3440530 msgr2=0x560ce343a8b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:27:31.295 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:31.302+0000 7f674f970b80 1 --2- 172.21.15.52:0/3639317842 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x560ce3440530 0x560ce343a8b0 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f6730004520 tx=0x7f673001a6a0 comp rx=0 tx=0).stop 2024-05-27T07:27:31.295 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:31.302+0000 7f674f970b80 1 -- 172.21.15.52:0/3639317842 shutdown_connections 2024-05-27T07:27:31.295 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:31.302+0000 7f674f970b80 1 --2- 172.21.15.52:0/3639317842 >> [v2:172.21.15.105:6836/191393149,v1:172.21.15.105:6837/191393149] conn(0x560ce345f5e0 0x560ce345fa80 crc :-1 s=READY pgs=33 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:27:31.295 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:31.302+0000 7f6721ffb700 0 client.25559 ms_handle_reset on v2:172.21.15.105:6836/191393149 2024-05-27T07:27:31.296 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:31.302+0000 7f674f970b80 1 --2- 172.21.15.52:0/3639317842 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x560ce344a7e0 0x560ce343adf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:27:31.296 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:31.302+0000 7f674f970b80 1 --2- 172.21.15.52:0/3639317842 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x560ce32bce40 0x560ce343c520 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:27:31.296 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:31.302+0000 7f674f970b80 1 --2- 172.21.15.52:0/3639317842 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x560ce3440530 0x560ce343a8b0 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:27:31.296 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:31.302+0000 7f674f970b80 1 -- 172.21.15.52:0/3639317842 >> 172.21.15.52:0/3639317842 conn(0x560ce32bf770 msgr2=0x560ce34440c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:27:31.296 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:31.302+0000 7f674f970b80 1 -- 172.21.15.52:0/3639317842 shutdown_connections 2024-05-27T07:27:31.297 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:31.303+0000 7f674f970b80 1 -- 172.21.15.52:0/3639317842 wait complete. 2024-05-27T07:27:31.297 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:31.303+0000 7f674f970b80 20 client.25559 trim_cache size 0 max 16384 2024-05-27T07:27:31.297 INFO:teuthology.orchestra.run.smithi052.stdout:[] 2024-05-27T07:27:31.298 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:31.303+0000 7f6740f5b700 1 -- 172.21.15.52:0/718704019 >> [v2:172.21.15.52:6800/1822318384,v1:172.21.15.52:6801/1822318384] conn(0x7f67240b2af0 msgr2=0x7f67240b5000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:27:31.298 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:31.303+0000 7f6740f5b700 1 --2- 172.21.15.52:0/718704019 >> [v2:172.21.15.52:6800/1822318384,v1:172.21.15.52:6801/1822318384] conn(0x7f67240b2af0 0x7f67240b5000 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f672c003760 tx=0x7f672c01a040 comp rx=0 tx=0).stop 2024-05-27T07:27:31.298 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:31.303+0000 7f6740f5b700 1 -- 172.21.15.52:0/718704019 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f673c108750 msgr2=0x7f673c19f630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:27:31.298 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:31.303+0000 7f6740f5b700 1 --2- 172.21.15.52:0/718704019 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f673c108750 0x7f673c19f630 secure :-1 s=READY pgs=530 cs=0 l=1 rev1=1 crypto rx=0x7f673400ef30 tx=0x7f673400c3a0 comp rx=0 tx=0).stop 2024-05-27T07:27:31.298 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:31.304+0000 7f6740f5b700 1 -- 172.21.15.52:0/718704019 shutdown_connections 2024-05-27T07:27:31.298 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:31.304+0000 7f6740f5b700 1 --2- 172.21.15.52:0/718704019 >> [v2:172.21.15.52:6800/1822318384,v1:172.21.15.52:6801/1822318384] conn(0x7f67240b2af0 0x7f67240b5000 unknown :-1 s=CLOSED pgs=75 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:27:31.298 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:31.304+0000 7f6740f5b700 1 --2- 172.21.15.52:0/718704019 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f673c108750 0x7f673c19f630 unknown :-1 s=CLOSED pgs=530 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:27:31.298 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:31.304+0000 7f6740f5b700 1 --2- 172.21.15.52:0/718704019 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f673c1034d0 0x7f673c19ae00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:27:31.298 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:31.304+0000 7f6740f5b700 1 --2- 172.21.15.52:0/718704019 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f673c107e00 0x7f673c19b3a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:27:31.298 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:31.304+0000 7f6740f5b700 1 -- 172.21.15.52:0/718704019 >> 172.21.15.52:0/718704019 conn(0x7f673c0fef20 msgr2=0x7f673c101dd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:27:31.298 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:31.304+0000 7f6740f5b700 1 -- 172.21.15.52:0/718704019 shutdown_connections 2024-05-27T07:27:31.298 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:31.304+0000 7f6740f5b700 1 -- 172.21.15.52:0/718704019 wait complete. 2024-05-27T07:27:31.987 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:27:31.997+0000 7fc2cca22640 1 -- 172.21.15.52:0/3477389700 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fc2c81084e0 msgr2=0x7fc2c81088f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:27:31.987 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:27:31.997+0000 7fc2cca22640 1 --2- 172.21.15.52:0/3477389700 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fc2c81084e0 0x7fc2c81088f0 secure :-1 s=READY pgs=533 cs=0 l=1 rev1=1 crypto rx=0x7fc2bc009900 tx=0x7fc2bc02a410 comp rx=0 tx=0).stop 2024-05-27T07:27:31.987 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:27:31.997+0000 7fc2cca22640 1 -- 172.21.15.52:0/3477389700 shutdown_connections 2024-05-27T07:27:31.987 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:27:31.997+0000 7fc2cca22640 1 --2- 172.21.15.52:0/3477389700 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fc2c8108f60 0x7fc2c81106e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:27:31.987 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:27:31.997+0000 7fc2cca22640 1 --2- 172.21.15.52:0/3477389700 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fc2c81084e0 0x7fc2c81088f0 unknown :-1 s=CLOSED pgs=533 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:27:31.987 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:27:31.997+0000 7fc2cca22640 1 --2- 172.21.15.52:0/3477389700 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fc2c8102460 0x7fc2c8102870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:27:31.987 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:27:31.997+0000 7fc2cca22640 1 -- 172.21.15.52:0/3477389700 >> 172.21.15.52:0/3477389700 conn(0x7fc2c80779d0 msgr2=0x7fc2c8077e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:27:31.988 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:27:31.997+0000 7fc2cca22640 1 -- 172.21.15.52:0/3477389700 shutdown_connections 2024-05-27T07:27:31.988 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:27:31.997+0000 7fc2cca22640 1 -- 172.21.15.52:0/3477389700 wait complete. 2024-05-27T07:27:31.989 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:27:31.998+0000 7fc2cca22640 1 Processor -- start 2024-05-27T07:27:31.989 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:27:31.998+0000 7fc2cca22640 1 -- start start 2024-05-27T07:27:31.989 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:27:31.998+0000 7fc2cca22640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fc2c8102460 0x7fc2c81a0590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:27:31.989 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:27:31.998+0000 7fc2cca22640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fc2c81084e0 0x7fc2c81a0ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:27:31.990 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:27:31.998+0000 7fc2cca22640 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fc2c8108f60 0x7fc2c819a710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:27:31.990 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:27:31.998+0000 7fc2cca22640 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x7fc2c8113e50 con 0x7fc2c8108f60 2024-05-27T07:27:31.990 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:27:31.998+0000 7fc2cca22640 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7fc2c8113cd0 con 0x7fc2c8102460 2024-05-27T07:27:31.990 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:27:31.998+0000 7fc2cca22640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7fc2c8113fd0 con 0x7fc2c81084e0 2024-05-27T07:27:31.990 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:27:31.999+0000 7fc2c6d9d640 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fc2c8108f60 0x7fc2c819a710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:27:31.990 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:27:31.999+0000 7fc2c6d9d640 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fc2c8108f60 0x7fc2c819a710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.52:53068/0 (socket says 172.21.15.52:53068) 2024-05-27T07:27:31.990 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:27:31.999+0000 7fc2c6d9d640 1 -- 172.21.15.52:0/3433667601 learned_addr learned my addr 172.21.15.52:0/3433667601 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-05-27T07:27:31.990 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:27:31.999+0000 7fc2c659c640 1 --2- 172.21.15.52:0/3433667601 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fc2c8102460 0x7fc2c81a0590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:27:31.990 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:27:31.999+0000 7fc2c5d9b640 1 --2- 172.21.15.52:0/3433667601 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fc2c81084e0 0x7fc2c81a0ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:27:31.990 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:27:31.999+0000 7fc2c659c640 1 -- 172.21.15.52:0/3433667601 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fc2c81084e0 msgr2=0x7fc2c81a0ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:27:31.990 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:27:31.999+0000 7fc2c659c640 1 --2- 172.21.15.52:0/3433667601 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fc2c81084e0 0x7fc2c81a0ad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:27:31.990 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:27:31.999+0000 7fc2c659c640 1 -- 172.21.15.52:0/3433667601 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fc2c8108f60 msgr2=0x7fc2c819a710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:27:31.991 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:27:31.999+0000 7fc2c659c640 1 --2- 172.21.15.52:0/3433667601 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fc2c8108f60 0x7fc2c819a710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:27:31.991 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:27:31.999+0000 7fc2c659c640 1 -- 172.21.15.52:0/3433667601 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc2c819ae40 con 0x7fc2c8102460 2024-05-27T07:27:31.991 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:27:31.999+0000 7fc2c6d9d640 1 --2- 172.21.15.52:0/3433667601 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fc2c8108f60 0x7fc2c819a710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-27T07:27:31.991 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:27:31.999+0000 7fc2c659c640 1 --2- 172.21.15.52:0/3433667601 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fc2c8102460 0x7fc2c81a0590 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7fc2c8069dd0 tx=0x7fc2b800dbb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:27:31.991 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:27:32.000+0000 7fc2b77fe640 1 -- 172.21.15.52:0/3433667601 <== mon.2 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc2b8017070 con 0x7fc2c8102460 2024-05-27T07:27:31.991 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:27:32.000+0000 7fc2b77fe640 1 -- 172.21.15.52:0/3433667601 <== mon.2 v2:172.21.15.105:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fc2b80099d0 con 0x7fc2c8102460 2024-05-27T07:27:31.991 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:27:32.000+0000 7fc2b77fe640 1 -- 172.21.15.52:0/3433667601 <== mon.2 v2:172.21.15.105:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc2b80047a0 con 0x7fc2c8102460 2024-05-27T07:27:31.991 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:27:32.000+0000 7fc2cca22640 1 -- 172.21.15.52:0/3433667601 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc2c819b0d0 con 0x7fc2c8102460 2024-05-27T07:27:31.991 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:27:32.000+0000 7fc2cca22640 1 -- 172.21.15.52:0/3433667601 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc2c81a7330 con 0x7fc2c8102460 2024-05-27T07:27:31.992 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:27:32.002+0000 7fc2d39ab740 1 Processor -- start 2024-05-27T07:27:31.992 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:27:32.003+0000 7fc2d39ab740 1 -- start start 2024-05-27T07:27:31.993 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:27:32.003+0000 7fc2d39ab740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x564c241981b0 0x564c245393d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:27:31.993 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:27:32.003+0000 7fc2d39ab740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x564c2416f2f0 0x564c24539910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:27:31.993 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:27:32.003+0000 7fc2d39ab740 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x564c241774d0 0x564c24539e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:27:31.993 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:27:32.003+0000 7fc2d39ab740 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x564c244abc30 con 0x564c241774d0 2024-05-27T07:27:31.993 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:27:32.003+0000 7fc2d39ab740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x564c2449b7d0 con 0x564c241981b0 2024-05-27T07:27:31.993 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:27:32.003+0000 7fc2d39ab740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x564c2449f810 con 0x564c2416f2f0 2024-05-27T07:27:31.993 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:27:32.003+0000 7fc2c5d9b640 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x564c241774d0 0x564c24539e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:27:31.993 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:27:32.003+0000 7fc2c5d9b640 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x564c241774d0 0x564c24539e50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.52:53070/0 (socket says 172.21.15.52:53070) 2024-05-27T07:27:31.993 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:27:32.003+0000 7fc2c5d9b640 1 -- 172.21.15.52:0/3291691081 learned_addr learned my addr 172.21.15.52:0/3291691081 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-05-27T07:27:31.994 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:27:32.003+0000 7fc2c6d9d640 1 --2- 172.21.15.52:0/3291691081 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x564c241981b0 0x564c245393d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:27:31.994 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:27:32.004+0000 7fc2c5d9b640 1 -- 172.21.15.52:0/3291691081 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x564c2416f2f0 msgr2=0x564c24539910 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-05-27T07:27:31.994 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:27:32.004+0000 7fc2c5d9b640 1 --2- 172.21.15.52:0/3291691081 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x564c2416f2f0 0x564c24539910 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:27:31.994 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:27:32.004+0000 7fc2c5d9b640 1 -- 172.21.15.52:0/3291691081 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x564c241981b0 msgr2=0x564c245393d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:27:31.995 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:27:32.004+0000 7fc2c5d9b640 1 --2- 172.21.15.52:0/3291691081 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x564c241981b0 0x564c245393d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:27:31.995 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:27:32.004+0000 7fc2c5d9b640 1 -- 172.21.15.52:0/3291691081 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x564c241b0210 con 0x564c241774d0 2024-05-27T07:27:31.995 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:27:32.004+0000 7fc2c5d9b640 1 --2- 172.21.15.52:0/3291691081 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x564c241774d0 0x564c24539e50 secure :-1 s=READY pgs=534 cs=0 l=1 rev1=1 crypto rx=0x7fc2c80755c0 tx=0x7fc2bc000bc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fe9816d16ba827b server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:27:31.995 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:27:32.004+0000 7fc2b4ff9640 1 -- 172.21.15.52:0/3291691081 <== mon.0 v2:172.21.15.52:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc2bc040070 con 0x564c241774d0 2024-05-27T07:27:31.995 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:27:32.005+0000 7fc2b4ff9640 1 -- 172.21.15.52:0/3291691081 <== mon.0 v2:172.21.15.52:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fc2bc004030 con 0x564c241774d0 2024-05-27T07:27:31.996 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:27:32.005+0000 7fc2b77fe640 1 -- 172.21.15.52:0/3433667601 <== mon.2 v2:172.21.15.105:3300/0 4 ==== mgrmap(e 26) ==== 149783+0+0 (secure 0 0 0) 0x7fc2b8004940 con 0x7fc2c8102460 2024-05-27T07:27:31.997 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:27:32.005+0000 7fc2b4ff9640 1 -- 172.21.15.52:0/3291691081 <== mon.0 v2:172.21.15.52:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc2bc035040 con 0x564c241774d0 2024-05-27T07:27:31.997 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:27:32.006+0000 7fc2d39ab740 1 -- 172.21.15.52:0/3291691081 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x564c241774d0 msgr2=0x564c24539e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:27:31.997 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:27:32.006+0000 7fc2d39ab740 1 --2- 172.21.15.52:0/3291691081 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x564c241774d0 0x564c24539e50 secure :-1 s=READY pgs=534 cs=0 l=1 rev1=1 crypto rx=0x7fc2c80755c0 tx=0x7fc2bc000bc0 comp rx=0 tx=0).stop 2024-05-27T07:27:31.997 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:27:32.006+0000 7fc2d39ab740 1 -- 172.21.15.52:0/3291691081 shutdown_connections 2024-05-27T07:27:31.997 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:27:32.006+0000 7fc2d39ab740 1 --2- 172.21.15.52:0/3291691081 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x564c241774d0 0x564c24539e50 unknown :-1 s=CLOSED pgs=534 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:27:31.997 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:27:32.006+0000 7fc2d39ab740 1 --2- 172.21.15.52:0/3291691081 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x564c2416f2f0 0x564c24539910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:27:31.997 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:27:32.006+0000 7fc2d39ab740 1 --2- 172.21.15.52:0/3291691081 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x564c241981b0 0x564c245393d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:27:31.997 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:27:32.006+0000 7fc2d39ab740 1 -- 172.21.15.52:0/3291691081 >> 172.21.15.52:0/3291691081 conn(0x564c2418c300 msgr2=0x564c24535b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:27:31.997 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:27:32.006+0000 7fc2b77fe640 1 --2- 172.21.15.52:0/3433667601 >> [v2:172.21.15.52:6800/1822318384,v1:172.21.15.52:6801/1822318384] conn(0x7fc2b00b2950 0x7fc2b00b4e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:27:31.997 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:27:32.006+0000 7fc2d39ab740 1 -- 172.21.15.52:0/3291691081 shutdown_connections 2024-05-27T07:27:31.997 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:27:32.006+0000 7fc2d39ab740 1 -- 172.21.15.52:0/3291691081 wait complete. 2024-05-27T07:27:31.997 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:27:32.006+0000 7fc2b77fe640 1 -- 172.21.15.52:0/3433667601 <== mon.2 v2:172.21.15.105:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9379+0+0 (secure 0 0 0) 0x7fc2b80e2560 con 0x7fc2c8102460 2024-05-27T07:27:31.997 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:27:32.006+0000 7fc2c5d9b640 1 --2- 172.21.15.52:0/3433667601 >> [v2:172.21.15.52:6800/1822318384,v1:172.21.15.52:6801/1822318384] conn(0x7fc2b00b2950 0x7fc2b00b4e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:27:31.998 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:27:32.007+0000 7fc2d39ab740 1 -- start start 2024-05-27T07:27:31.998 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:27:32.007+0000 7fc2c5d9b640 1 --2- 172.21.15.52:0/3433667601 >> [v2:172.21.15.52:6800/1822318384,v1:172.21.15.52:6801/1822318384] conn(0x7fc2b00b2950 0x7fc2b00b4e60 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7fc2bc02a930 tx=0x7fc2bc02a410 comp rx=0 tx=0).ready entity=mgr.15333 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:27:31.998 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:27:32.007+0000 7fc2d39ab740 1 Processor -- start 2024-05-27T07:27:31.999 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:27:32.008+0000 7fc2d39ab740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x564c2416f2f0 0x564c24543970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:27:31.999 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:27:32.008+0000 7fc2d39ab740 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x564c241774d0 0x564c24543eb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:27:31.999 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:27:32.008+0000 7fc2c5d9b640 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x564c241774d0 0x564c24543eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:27:31.999 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:27:32.009+0000 7fc2c659c640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x564c2416f2f0 0x564c24543970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:27:31.999 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:27:32.009+0000 7fc2c659c640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x564c2416f2f0 0x564c24543970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.52:42220/0 (socket says 172.21.15.52:42220) 2024-05-27T07:27:31.999 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:27:32.009+0000 7fc2c5d9b640 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x564c241774d0 0x564c24543eb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.52:53072/0 (socket says 172.21.15.52:53072) 2024-05-27T07:27:31.999 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:27:32.009+0000 7fc2d39ab740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x564c241981b0 0x564c2454f8c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:27:32.000 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:27:32.009+0000 7fc2c5d9b640 1 -- 172.21.15.52:0/1952655957 learned_addr learned my addr 172.21.15.52:0/1952655957 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-05-27T07:27:32.000 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:27:32.009+0000 7fc2d39ab740 1 -- 172.21.15.52:0/1952655957 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x564c24181790 con 0x564c241774d0 2024-05-27T07:27:32.000 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:27:32.009+0000 7fc2d39ab740 1 -- 172.21.15.52:0/1952655957 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x564c24457890 con 0x564c2416f2f0 2024-05-27T07:27:32.000 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:27:32.009+0000 7fc2d39ab740 1 -- 172.21.15.52:0/1952655957 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x564c24546540 con 0x564c241981b0 2024-05-27T07:27:32.000 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:27:32.009+0000 7fc2c6d9d640 1 --2- 172.21.15.52:0/1952655957 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x564c241981b0 0x564c2454f8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:27:32.001 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:27:32.010+0000 7fc2c5d9b640 1 -- 172.21.15.52:0/1952655957 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x564c241981b0 msgr2=0x564c2454f8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:27:32.001 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:27:32.010+0000 7fc2c5d9b640 1 --2- 172.21.15.52:0/1952655957 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x564c241981b0 0x564c2454f8c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:27:32.001 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:27:32.010+0000 7fc2c5d9b640 1 -- 172.21.15.52:0/1952655957 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x564c2416f2f0 msgr2=0x564c24543970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:27:32.001 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:27:32.010+0000 7fc2c5d9b640 1 --2- 172.21.15.52:0/1952655957 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x564c2416f2f0 0x564c24543970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:27:32.002 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:27:32.010+0000 7fc2c5d9b640 1 -- 172.21.15.52:0/1952655957 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({osdmap=0}) -- 0x564c2427c620 con 0x564c241774d0 2024-05-27T07:27:32.002 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:27:32.010+0000 7fc2c659c640 1 --2- 172.21.15.52:0/1952655957 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x564c2416f2f0 0x564c24543970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-27T07:27:32.002 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:27:32.010+0000 7fc2c6d9d640 1 --2- 172.21.15.52:0/1952655957 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x564c241981b0 0x564c2454f8c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-27T07:27:32.002 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:27:32.010+0000 7fc2d39ab740 10 client.0 _ll_register_callbacks cb 0x564c241afa00 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T07:27:32.002 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:27:32.010+0000 7fc2d39ab740 1 -- 172.21.15.52:0/1952655957 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fc2bc004030 con 0x564c241774d0 2024-05-27T07:27:32.002 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:27:32.010+0000 7fc2c5d9b640 1 --2- 172.21.15.52:0/1952655957 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x564c241774d0 0x564c24543eb0 secure :-1 s=READY pgs=535 cs=0 l=1 rev1=1 crypto rx=0x7fc2bc005500 tx=0x7fc2bc000bc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8cb14afbe4a0fcb4 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:27:32.002 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:27:32.010+0000 7fc2adffb640 10 client.15876 ms_handle_connect on v2:172.21.15.52:3300/0 2024-05-27T07:27:32.002 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:27:32.011+0000 7fc2adffb640 1 -- 172.21.15.52:0/1952655957 <== mon.0 v2:172.21.15.52:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc2bc061050 con 0x564c241774d0 2024-05-27T07:27:32.002 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:27:32.011+0000 7fc2adffb640 1 -- 172.21.15.52:0/1952655957 <== mon.0 v2:172.21.15.52:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 9379+0+0 (secure 0 0 0) 0x7fc2bc0316a0 con 0x564c241774d0 2024-05-27T07:27:32.002 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:27:32.011+0000 7fc2adffb640 1 -- 172.21.15.52:0/1952655957 <== mon.0 v2:172.21.15.52:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7fc2bc040040 con 0x564c241774d0 2024-05-27T07:27:32.002 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:27:32.011+0000 7fc2d39ab740 10 client.15876 fetch_fsmap learned FSMap version 30 2024-05-27T07:27:32.002 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:27:32.011+0000 7fc2d39ab740 1 -- 172.21.15.52:0/1952655957 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({fsmap=30}) -- 0x564c245523b0 con 0x564c241774d0 2024-05-27T07:27:32.003 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:27:32.012+0000 7fc2adffb640 1 -- 172.21.15.52:0/1952655957 <== mon.0 v2:172.21.15.52:3300/0 4 ==== fsmap(e 30) ==== 8362+0+0 (secure 0 0 0) 0x7fc2bc004750 con 0x564c241774d0 2024-05-27T07:27:32.003 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:27:32.012+0000 7fc2d39ab740 10 client.15876 fetch_fsmap finished waiting for FSMap version 30 2024-05-27T07:27:32.003 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:27:32.012+0000 7fc2d39ab740 10 client.15876 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-05-27T07:27:32.003 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:27:32.012+0000 7fc2d39ab740 20 client.15876 populate_metadata read hostname 'smithi052' 2024-05-27T07:27:32.003 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:27:32.012+0000 7fc2d39ab740 1 --2- 172.21.15.52:0/1952655957 >> [v2:172.21.15.105:6838/500348151,v1:172.21.15.105:6839/500348151] conn(0x564c24156d80 0x564c24555710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:27:32.003 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:27:32.012+0000 7fc2d39ab740 4 client.15876 mds_command: new command op to 24527 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-27T07:27:32.003 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:27:32.012+0000 7fc2d39ab740 1 -- 172.21.15.52:0/1952655957 --> [v2:172.21.15.105:6838/500348151,v1:172.21.15.105:6839/500348151] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x564c244adb80 con 0x564c24156d80 2024-05-27T07:27:32.003 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:27:32.012+0000 7fc2c659c640 1 --2- 172.21.15.52:0/1952655957 >> [v2:172.21.15.105:6838/500348151,v1:172.21.15.105:6839/500348151] conn(0x564c24156d80 0x564c24555710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:27:32.004 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:27:32.012+0000 7fc2c659c640 1 --2- 172.21.15.52:0/1952655957 >> [v2:172.21.15.105:6838/500348151,v1:172.21.15.105:6839/500348151] conn(0x564c24156d80 0x564c24555710 crc :-1 s=READY pgs=65 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=273b72e5c2fb5261 server_cookie=239ebde787110341 in_seq=0 out_seq=0 2024-05-27T07:27:32.004 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:27:32.012+0000 7fc2adffb640 10 client.15876 ms_handle_connect on v2:172.21.15.105:6838/500348151 2024-05-27T07:27:32.004 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:27:32.013+0000 7fc2adffb640 1 -- 172.21.15.52:0/1952655957 <== mds.0 v2:172.21.15.105:6838/500348151 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24041 (crc 0 0 0) 0x564c244adb80 con 0x564c24156d80 2024-05-27T07:27:32.004 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:27:32.013+0000 7fc2adffb640 10 client.15876 handle_command_reply: tid=0 2024-05-27T07:27:32.004 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:27:32.013+0000 7fc2d39ab740 1 client.15876 shutdown 2024-05-27T07:27:32.004 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:27:32.013+0000 7fc2d39ab740 1 -- 172.21.15.52:0/1952655957 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x564c241774d0 msgr2=0x564c24543eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:27:32.004 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:27:32.013+0000 7fc2d39ab740 1 --2- 172.21.15.52:0/1952655957 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x564c241774d0 0x564c24543eb0 secure :-1 s=READY pgs=535 cs=0 l=1 rev1=1 crypto rx=0x7fc2bc005500 tx=0x7fc2bc000bc0 comp rx=0 tx=0).stop 2024-05-27T07:27:32.005 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:27:32.013+0000 7fc2d39ab740 1 -- 172.21.15.52:0/1952655957 shutdown_connections 2024-05-27T07:27:32.005 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:27:32.013+0000 7fc2d39ab740 1 --2- 172.21.15.52:0/1952655957 >> [v2:172.21.15.105:6838/500348151,v1:172.21.15.105:6839/500348151] conn(0x564c24156d80 0x564c24555710 crc :-1 s=READY pgs=65 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:27:32.005 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:27:32.014+0000 7fc2d39ab740 1 --2- 172.21.15.52:0/1952655957 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x564c241981b0 0x564c2454f8c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:27:32.005 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:27:32.014+0000 7fc2d39ab740 1 --2- 172.21.15.52:0/1952655957 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x564c241774d0 0x564c24543eb0 unknown :-1 s=CLOSED pgs=535 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:27:32.005 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:27:32.014+0000 7fc2adffb640 0 client.15876 ms_handle_reset on v2:172.21.15.105:6838/500348151 2024-05-27T07:27:32.005 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:27:32.014+0000 7fc2d39ab740 1 --2- 172.21.15.52:0/1952655957 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x564c2416f2f0 0x564c24543970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:27:32.005 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:27:32.014+0000 7fc2d39ab740 1 -- 172.21.15.52:0/1952655957 >> 172.21.15.52:0/1952655957 conn(0x564c2418c300 msgr2=0x564c2453b0f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:27:32.005 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:27:32.014+0000 7fc2d39ab740 1 -- 172.21.15.52:0/1952655957 shutdown_connections 2024-05-27T07:27:32.005 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:27:32.014+0000 7fc2d39ab740 1 -- 172.21.15.52:0/1952655957 wait complete. 2024-05-27T07:27:32.005 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:27:32.014+0000 7fc2d39ab740 20 client.15876 trim_cache size 0 max 16384 2024-05-27T07:27:32.019 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:27:32.029+0000 7fc2d39ab740 1 Processor -- start 2024-05-27T07:27:32.019 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:27:32.029+0000 7fc2d39ab740 1 -- start start 2024-05-27T07:27:32.019 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:27:32.030+0000 7fc2d39ab740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x564c2416f2f0 0x564c24543cb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:27:32.019 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:27:32.030+0000 7fc2d39ab740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x564c241774d0 0x564c24539870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:27:32.019 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:27:32.030+0000 7fc2d39ab740 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x564c241981b0 0x564c24539db0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:27:32.020 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:27:32.030+0000 7fc2d39ab740 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x7fc2c81b10e0 con 0x564c241981b0 2024-05-27T07:27:32.020 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:27:32.030+0000 7fc2d39ab740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7fc2c81b2c20 con 0x564c2416f2f0 2024-05-27T07:27:32.020 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:27:32.030+0000 7fc2d39ab740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7fc2c81b17b0 con 0x564c241774d0 2024-05-27T07:27:32.020 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:27:32.030+0000 7fc2c6d9d640 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x564c241981b0 0x564c24539db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:27:32.020 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:27:32.030+0000 7fc2c6d9d640 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x564c241981b0 0x564c24539db0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.52:53086/0 (socket says 172.21.15.52:53086) 2024-05-27T07:27:32.020 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:27:32.030+0000 7fc2c6d9d640 1 -- 172.21.15.52:0/2080874563 learned_addr learned my addr 172.21.15.52:0/2080874563 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-05-27T07:27:32.021 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:27:32.030+0000 7fc2c659c640 1 --2- 172.21.15.52:0/2080874563 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x564c2416f2f0 0x564c24543cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:27:32.021 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:27:32.030+0000 7fc2c5d9b640 1 --2- 172.21.15.52:0/2080874563 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x564c241774d0 0x564c24539870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:27:32.021 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:27:32.030+0000 7fc2c6d9d640 1 -- 172.21.15.52:0/2080874563 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x564c241774d0 msgr2=0x564c24539870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:27:32.021 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:27:32.030+0000 7fc2c6d9d640 1 --2- 172.21.15.52:0/2080874563 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x564c241774d0 0x564c24539870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:27:32.021 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:27:32.030+0000 7fc2c6d9d640 1 -- 172.21.15.52:0/2080874563 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x564c2416f2f0 msgr2=0x564c24543cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:27:32.021 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:27:32.030+0000 7fc2c6d9d640 1 --2- 172.21.15.52:0/2080874563 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x564c2416f2f0 0x564c24543cb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:27:32.021 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:27:32.030+0000 7fc2c6d9d640 1 -- 172.21.15.52:0/2080874563 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x564c245441f0 con 0x564c241981b0 2024-05-27T07:27:32.021 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:27:32.030+0000 7fc2c6d9d640 1 --2- 172.21.15.52:0/2080874563 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x564c241981b0 0x564c24539db0 secure :-1 s=READY pgs=536 cs=0 l=1 rev1=1 crypto rx=0x7fc2c000ea50 tx=0x7fc2c000ef10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=81fd07fd8e3d58a8 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:27:32.021 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:27:32.031+0000 7fc2affff640 1 -- 172.21.15.52:0/2080874563 <== mon.0 v2:172.21.15.52:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc2c0004b20 con 0x564c241981b0 2024-05-27T07:27:32.021 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:27:32.031+0000 7fc2affff640 1 -- 172.21.15.52:0/2080874563 <== mon.0 v2:172.21.15.52:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fc2c0004cc0 con 0x564c241981b0 2024-05-27T07:27:32.021 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:27:32.031+0000 7fc2affff640 1 -- 172.21.15.52:0/2080874563 <== mon.0 v2:172.21.15.52:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc2c0004310 con 0x564c241981b0 2024-05-27T07:27:32.024 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:27:32.032+0000 7fc2d39ab740 1 -- 172.21.15.52:0/2080874563 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x564c241981b0 msgr2=0x564c24539db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:27:32.024 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:27:32.032+0000 7fc2d39ab740 1 --2- 172.21.15.52:0/2080874563 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x564c241981b0 0x564c24539db0 secure :-1 s=READY pgs=536 cs=0 l=1 rev1=1 crypto rx=0x7fc2c000ea50 tx=0x7fc2c000ef10 comp rx=0 tx=0).stop 2024-05-27T07:27:32.024 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:27:32.032+0000 7fc2d39ab740 1 -- 172.21.15.52:0/2080874563 shutdown_connections 2024-05-27T07:27:32.024 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:27:32.032+0000 7fc2d39ab740 1 --2- 172.21.15.52:0/2080874563 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x564c241981b0 0x564c24539db0 unknown :-1 s=CLOSED pgs=536 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:27:32.024 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:27:32.032+0000 7fc2d39ab740 1 --2- 172.21.15.52:0/2080874563 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x564c241774d0 0x564c24539870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:27:32.024 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:27:32.032+0000 7fc2d39ab740 1 --2- 172.21.15.52:0/2080874563 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x564c2416f2f0 0x564c24543cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:27:32.024 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:27:32.032+0000 7fc2d39ab740 1 -- 172.21.15.52:0/2080874563 >> 172.21.15.52:0/2080874563 conn(0x564c2418c300 msgr2=0x564c24539160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:27:32.024 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:27:32.032+0000 7fc2d39ab740 1 -- 172.21.15.52:0/2080874563 shutdown_connections 2024-05-27T07:27:32.024 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:27:32.032+0000 7fc2d39ab740 1 -- 172.21.15.52:0/2080874563 wait complete. 2024-05-27T07:27:32.025 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:27:32.032+0000 7fc2d39ab740 1 -- start start 2024-05-27T07:27:32.025 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:27:32.032+0000 7fc2d39ab740 1 Processor -- start 2024-05-27T07:27:32.025 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:27:32.033+0000 7fc2d39ab740 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x564c2416f2f0 0x564c245b6730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:27:32.025 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:27:32.033+0000 7fc2d39ab740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x564c241774d0 0x564c245b6c70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:27:32.025 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:27:32.033+0000 7fc2d39ab740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x564c241981b0 0x564c2454a810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:27:32.025 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:27:32.033+0000 7fc2d39ab740 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x7fc2c81b2550 con 0x564c2416f2f0 2024-05-27T07:27:32.025 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:27:32.033+0000 7fc2d39ab740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7fc2c81b1e80 con 0x564c241774d0 2024-05-27T07:27:32.025 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:27:32.033+0000 7fc2c659c640 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x564c2416f2f0 0x564c245b6730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:27:32.025 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:27:32.033+0000 7fc2d39ab740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7fc2c81af0e0 con 0x564c241981b0 2024-05-27T07:27:32.025 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:27:32.033+0000 7fc2c659c640 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x564c2416f2f0 0x564c245b6730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.52:53088/0 (socket says 172.21.15.52:53088) 2024-05-27T07:27:32.025 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:27:32.033+0000 7fc2c659c640 1 -- 172.21.15.52:0/2864371063 learned_addr learned my addr 172.21.15.52:0/2864371063 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-05-27T07:27:32.025 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:27:32.033+0000 7fc2d39ab740 10 client.0 _ll_register_callbacks cb 0x7fc2c0009900 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T07:27:32.025 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:27:32.033+0000 7fc2c5d9b640 1 --2- 172.21.15.52:0/2864371063 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x564c241774d0 0x564c245b6c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:27:32.025 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:27:32.033+0000 7fc2c6d9d640 1 --2- 172.21.15.52:0/2864371063 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x564c241981b0 0x564c2454a810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:27:32.026 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:27:32.033+0000 7fc2c659c640 1 -- 172.21.15.52:0/2864371063 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x564c241981b0 msgr2=0x564c2454a810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:27:32.026 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:27:32.033+0000 7fc2c659c640 1 --2- 172.21.15.52:0/2864371063 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x564c241981b0 0x564c2454a810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:27:32.026 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:27:32.033+0000 7fc2c659c640 1 -- 172.21.15.52:0/2864371063 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x564c241774d0 msgr2=0x564c245b6c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:27:32.026 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:27:32.033+0000 7fc2c659c640 1 --2- 172.21.15.52:0/2864371063 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x564c241774d0 0x564c245b6c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:27:32.026 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:27:32.033+0000 7fc2c659c640 1 -- 172.21.15.52:0/2864371063 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({osdmap=0}) -- 0x564c2454ad50 con 0x564c2416f2f0 2024-05-27T07:27:32.026 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:27:32.034+0000 7fc2c659c640 1 --2- 172.21.15.52:0/2864371063 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x564c2416f2f0 0x564c245b6730 secure :-1 s=READY pgs=537 cs=0 l=1 rev1=1 crypto rx=0x7fc2b8009610 tx=0x7fc2b80057e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b1ed4d105d0a7bf8 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:27:32.026 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:27:32.034+0000 7fc2adffb640 10 client.0 ms_handle_connect on v2:172.21.15.52:3300/0 2024-05-27T07:27:32.026 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:27:32.034+0000 7fc2adffb640 1 -- 172.21.15.52:0/2864371063 <== mon.0 v2:172.21.15.52:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc2b80c8060 con 0x564c2416f2f0 2024-05-27T07:27:32.026 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:27:32.034+0000 7fc2adffb640 1 -- 172.21.15.52:0/2864371063 <== mon.0 v2:172.21.15.52:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 9379+0+0 (secure 0 0 0) 0x7fc2b80a5ad0 con 0x564c2416f2f0 2024-05-27T07:27:32.026 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:27:32.034+0000 7fc2d39ab740 1 -- 172.21.15.52:0/2864371063 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fc2c0004cc0 con 0x564c2416f2f0 2024-05-27T07:27:32.026 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:27:32.034+0000 7fc2adffb640 1 -- 172.21.15.52:0/2864371063 <== mon.0 v2:172.21.15.52:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7fc2b8088050 con 0x564c2416f2f0 2024-05-27T07:27:32.026 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:27:32.034+0000 7fc2d39ab740 10 client.15882 fetch_fsmap learned FSMap version 30 2024-05-27T07:27:32.026 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:27:32.034+0000 7fc2d39ab740 1 -- 172.21.15.52:0/2864371063 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x564c2454bfe0 con 0x564c2416f2f0 2024-05-27T07:27:32.027 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:27:32.035+0000 7fc2adffb640 1 -- 172.21.15.52:0/2864371063 <== mon.0 v2:172.21.15.52:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fc2c0004cc0 con 0x564c2416f2f0 2024-05-27T07:27:32.028 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:27:32.035+0000 7fc2adffb640 1 -- 172.21.15.52:0/2864371063 <== mon.0 v2:172.21.15.52:3300/0 5 ==== fsmap(e 30) ==== 8362+0+0 (secure 0 0 0) 0x7fc2b80a5770 con 0x564c2416f2f0 2024-05-27T07:27:32.028 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:27:32.035+0000 7fc2d39ab740 10 client.15882 fetch_fsmap finished waiting for FSMap version 30 2024-05-27T07:27:32.028 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:27:32.035+0000 7fc2d39ab740 10 client.15882 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-05-27T07:27:32.028 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:27:32.035+0000 7fc2d39ab740 20 client.15882 populate_metadata read hostname 'smithi052' 2024-05-27T07:27:32.028 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:27:32.035+0000 7fc2d39ab740 1 --2- 172.21.15.52:0/2864371063 >> [v2:172.21.15.105:6838/500348151,v1:172.21.15.105:6839/500348151] conn(0x564c2412c1c0 0x564c245c8870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:27:32.028 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:27:32.035+0000 7fc2d39ab740 4 client.15882 mds_command: new command op to 24527 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-05-27T07:27:32.028 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:27:32.035+0000 7fc2d39ab740 1 -- 172.21.15.52:0/2864371063 --> [v2:172.21.15.105:6838/500348151,v1:172.21.15.105:6839/500348151] -- command(tid 0: {"prefix": "flush journal"}) -- 0x564c244a7360 con 0x564c2412c1c0 2024-05-27T07:27:32.028 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:27:32.035+0000 7fc2c5d9b640 1 --2- 172.21.15.52:0/2864371063 >> [v2:172.21.15.105:6838/500348151,v1:172.21.15.105:6839/500348151] conn(0x564c2412c1c0 0x564c245c8870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:27:32.028 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:27:32.036+0000 7fc2adffb640 10 client.15882 ms_handle_connect on v2:172.21.15.105:6838/500348151 2024-05-27T07:27:32.028 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:27:32.036+0000 7fc2c5d9b640 1 --2- 172.21.15.52:0/2864371063 >> [v2:172.21.15.105:6838/500348151,v1:172.21.15.105:6839/500348151] conn(0x564c2412c1c0 0x564c245c8870 crc :-1 s=READY pgs=66 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=84447445c34b7fce server_cookie=2544db8f02424ddd in_seq=0 out_seq=0 2024-05-27T07:27:32.032 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:27:32.042+0000 7fc2adffb640 1 -- 172.21.15.52:0/2864371063 <== mds.0 v2:172.21.15.105:6838/500348151 1 ==== command_reply(tid 0: 0 ) ==== 8+0+25 (crc 0 0 0) 0x564c244a7360 con 0x564c2412c1c0 2024-05-27T07:27:32.032 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:27:32.042+0000 7fc2adffb640 10 client.15882 handle_command_reply: tid=0 2024-05-27T07:27:32.032 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:27:32.042+0000 7fc2d39ab740 1 client.15882 shutdown 2024-05-27T07:27:32.032 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:27:32.042+0000 7fc2d39ab740 1 -- 172.21.15.52:0/2864371063 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x564c2416f2f0 msgr2=0x564c245b6730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:27:32.032 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:27:32.042+0000 7fc2d39ab740 1 --2- 172.21.15.52:0/2864371063 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x564c2416f2f0 0x564c245b6730 secure :-1 s=READY pgs=537 cs=0 l=1 rev1=1 crypto rx=0x7fc2b8009610 tx=0x7fc2b80057e0 comp rx=0 tx=0).stop 2024-05-27T07:27:32.032 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:27:32.042+0000 7fc2d39ab740 1 -- 172.21.15.52:0/2864371063 shutdown_connections 2024-05-27T07:27:32.032 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:27:32.042+0000 7fc2d39ab740 1 --2- 172.21.15.52:0/2864371063 >> [v2:172.21.15.105:6838/500348151,v1:172.21.15.105:6839/500348151] conn(0x564c2412c1c0 0x564c245c8870 crc :-1 s=READY pgs=66 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:27:32.033 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:27:32.042+0000 7fc2d39ab740 1 --2- 172.21.15.52:0/2864371063 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x564c241981b0 0x564c2454a810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:27:32.033 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:27:32.042+0000 7fc2d39ab740 1 --2- 172.21.15.52:0/2864371063 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x564c241774d0 0x564c245b6c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:27:32.033 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:27:32.042+0000 7fc2d39ab740 1 --2- 172.21.15.52:0/2864371063 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x564c2416f2f0 0x564c245b6730 unknown :-1 s=CLOSED pgs=537 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:27:32.033 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:27:32.042+0000 7fc2d39ab740 1 -- 172.21.15.52:0/2864371063 >> 172.21.15.52:0/2864371063 conn(0x564c2418c300 msgr2=0x564c24539160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:27:32.033 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:27:32.042+0000 7fc2adffb640 0 client.15882 ms_handle_reset on v2:172.21.15.105:6838/500348151 2024-05-27T07:27:32.033 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:27:32.042+0000 7fc2d39ab740 1 -- 172.21.15.52:0/2864371063 shutdown_connections 2024-05-27T07:27:32.033 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:27:32.042+0000 7fc2d39ab740 1 -- 172.21.15.52:0/2864371063 wait complete. 2024-05-27T07:27:32.033 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:27:32.042+0000 7fc2d39ab740 20 client.15882 trim_cache size 0 max 16384 2024-05-27T07:27:32.033 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stdout:{ 2024-05-27T07:27:32.033 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stdout: "return_code": 0 2024-05-27T07:27:32.033 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stdout:} 2024-05-27T07:27:32.034 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:27:32.043+0000 7fc2cca22640 1 -- 172.21.15.52:0/3433667601 >> [v2:172.21.15.52:6800/1822318384,v1:172.21.15.52:6801/1822318384] conn(0x7fc2b00b2950 msgr2=0x7fc2b00b4e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:27:32.034 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:27:32.043+0000 7fc2cca22640 1 --2- 172.21.15.52:0/3433667601 >> [v2:172.21.15.52:6800/1822318384,v1:172.21.15.52:6801/1822318384] conn(0x7fc2b00b2950 0x7fc2b00b4e60 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7fc2bc02a930 tx=0x7fc2bc02a410 comp rx=0 tx=0).stop 2024-05-27T07:27:32.034 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:27:32.043+0000 7fc2cca22640 1 -- 172.21.15.52:0/3433667601 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fc2c8102460 msgr2=0x7fc2c81a0590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:27:32.035 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:27:32.043+0000 7fc2cca22640 1 --2- 172.21.15.52:0/3433667601 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fc2c8102460 0x7fc2c81a0590 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7fc2c8069dd0 tx=0x7fc2b800dbb0 comp rx=0 tx=0).stop 2024-05-27T07:27:32.035 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:27:32.043+0000 7fc2cca22640 1 -- 172.21.15.52:0/3433667601 shutdown_connections 2024-05-27T07:27:32.035 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:27:32.043+0000 7fc2cca22640 1 --2- 172.21.15.52:0/3433667601 >> [v2:172.21.15.52:6800/1822318384,v1:172.21.15.52:6801/1822318384] conn(0x7fc2b00b2950 0x7fc2b00b4e60 unknown :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:27:32.035 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:27:32.043+0000 7fc2cca22640 1 --2- 172.21.15.52:0/3433667601 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fc2c8108f60 0x7fc2c819a710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:27:32.035 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:27:32.043+0000 7fc2cca22640 1 --2- 172.21.15.52:0/3433667601 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fc2c81084e0 0x7fc2c81a0ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:27:32.035 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:27:32.043+0000 7fc2cca22640 1 --2- 172.21.15.52:0/3433667601 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fc2c8102460 0x7fc2c81a0590 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:27:32.035 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:27:32.043+0000 7fc2cca22640 1 -- 172.21.15.52:0/3433667601 >> 172.21.15.52:0/3433667601 conn(0x7fc2c80779d0 msgr2=0x7fc2c8100720 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:27:32.035 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:27:32.043+0000 7fc2cca22640 1 -- 172.21.15.52:0/3433667601 shutdown_connections 2024-05-27T07:27:32.035 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:27:32.043+0000 7fc2cca22640 1 -- 172.21.15.52:0/3433667601 wait complete. 2024-05-27T07:27:32.131 DEBUG:teuthology.orchestra.run.smithi052:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:38f9d900c83c63799fbdbe61acc9a11b0d3554a6 shell --fsid 73df2636-1bf8-11ef-bc9b-c7b262605968 -- ceph tell mds.1:0 damage ls 2024-05-27T07:27:32.439 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:27:32 smithi105 ceph-mon[35977]: pgmap v50: 129 pgs: 129 active+clean; 1.4 MiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 1.2 KiB/s wr, 10 op/s 2024-05-27T07:27:32.479 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:27:32 smithi052 ceph-mon[31955]: pgmap v50: 129 pgs: 129 active+clean; 1.4 MiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 1.2 KiB/s wr, 10 op/s 2024-05-27T07:27:32.523 INFO:teuthology.orchestra.run.smithi052.stderr:Inferring config /var/lib/ceph/73df2636-1bf8-11ef-bc9b-c7b262605968/mon.a/config 2024-05-27T07:27:32.601 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:27:32 smithi173 ceph-mon[37000]: pgmap v50: 129 pgs: 129 active+clean; 1.4 MiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 1.2 KiB/s wr, 10 op/s 2024-05-27T07:27:34.439 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:27:34 smithi105 ceph-mon[35977]: pgmap v51: 129 pgs: 129 active+clean; 1.4 MiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 6.8 KiB/s rd, 1.5 KiB/s wr, 15 op/s 2024-05-27T07:27:34.454 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:27:34 smithi052 ceph-mon[31955]: pgmap v51: 129 pgs: 129 active+clean; 1.4 MiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 6.8 KiB/s rd, 1.5 KiB/s wr, 15 op/s 2024-05-27T07:27:34.534 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:34.544+0000 7f305eaae700 1 -- 172.21.15.52:0/1394766288 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f30581029b0 msgr2=0x7f3058102dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:27:34.534 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:34.544+0000 7f305eaae700 1 --2- 172.21.15.52:0/1394766288 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f30581029b0 0x7f3058102dc0 secure :-1 s=READY pgs=538 cs=0 l=1 rev1=1 crypto rx=0x7f304800c210 tx=0x7f304800c520 comp rx=0 tx=0).stop 2024-05-27T07:27:34.535 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:34.544+0000 7f305eaae700 1 -- 172.21.15.52:0/1394766288 shutdown_connections 2024-05-27T07:27:34.535 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:34.544+0000 7f305eaae700 1 --2- 172.21.15.52:0/1394766288 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f3058103300 0x7f305810f880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:27:34.535 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:34.544+0000 7f305eaae700 1 --2- 172.21.15.52:0/1394766288 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f30581089e0 0x7f3058108df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:27:34.535 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:34.544+0000 7f305eaae700 1 --2- 172.21.15.52:0/1394766288 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f30581029b0 0x7f3058102dc0 unknown :-1 s=CLOSED pgs=538 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:27:34.535 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:34.544+0000 7f305eaae700 1 -- 172.21.15.52:0/1394766288 >> 172.21.15.52:0/1394766288 conn(0x7f30580fe6e0 msgr2=0x7f3058100b00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:27:34.535 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:34.544+0000 7f305eaae700 1 -- 172.21.15.52:0/1394766288 shutdown_connections 2024-05-27T07:27:34.535 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:34.544+0000 7f305eaae700 1 -- 172.21.15.52:0/1394766288 wait complete. 2024-05-27T07:27:34.536 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:34.545+0000 7f305eaae700 1 Processor -- start 2024-05-27T07:27:34.536 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:34.545+0000 7f305eaae700 1 -- start start 2024-05-27T07:27:34.536 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:34.545+0000 7f305eaae700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f30581029b0 0x7f305819ae50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:27:34.536 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:34.545+0000 7f305eaae700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f3058103300 0x7f305819b3f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:27:34.536 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:34.545+0000 7f305eaae700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f30581089e0 0x7f305819f680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:27:34.536 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:34.546+0000 7f305eaae700 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x7f30581122d0 con 0x7f30581029b0 2024-05-27T07:27:34.536 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:34.546+0000 7f305eaae700 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f3058112150 con 0x7f3058103300 2024-05-27T07:27:34.537 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:34.546+0000 7f305eaae700 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f3058112450 con 0x7f30581089e0 2024-05-27T07:27:34.537 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:34.546+0000 7f305c84a700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f30581029b0 0x7f305819ae50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:27:34.537 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:34.546+0000 7f305c84a700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f30581029b0 0x7f305819ae50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.52:53118/0 (socket says 172.21.15.52:53118) 2024-05-27T07:27:34.537 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:34.546+0000 7f305c84a700 1 -- 172.21.15.52:0/3978290760 learned_addr learned my addr 172.21.15.52:0/3978290760 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-05-27T07:27:34.537 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:34.546+0000 7f305d04b700 1 --2- 172.21.15.52:0/3978290760 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f30581089e0 0x7f305819f680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:27:34.537 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:34.546+0000 7f3057fff700 1 --2- 172.21.15.52:0/3978290760 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f3058103300 0x7f305819b3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:27:34.538 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:34.546+0000 7f305c84a700 1 -- 172.21.15.52:0/3978290760 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f30581089e0 msgr2=0x7f305819f680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:27:34.538 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:34.546+0000 7f305c84a700 1 --2- 172.21.15.52:0/3978290760 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f30581089e0 0x7f305819f680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:27:34.538 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:34.546+0000 7f305c84a700 1 -- 172.21.15.52:0/3978290760 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f3058103300 msgr2=0x7f305819b3f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:27:34.538 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:34.546+0000 7f305c84a700 1 --2- 172.21.15.52:0/3978290760 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f3058103300 0x7f305819b3f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:27:34.538 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:34.546+0000 7f305c84a700 1 -- 172.21.15.52:0/3978290760 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f305819fd20 con 0x7f30581029b0 2024-05-27T07:27:34.538 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:34.546+0000 7f305c84a700 1 --2- 172.21.15.52:0/3978290760 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f30581029b0 0x7f305819ae50 secure :-1 s=READY pgs=539 cs=0 l=1 rev1=1 crypto rx=0x7f305000d980 tx=0x7f305000dc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:27:34.538 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:34.547+0000 7f3055ffb700 1 -- 172.21.15.52:0/3978290760 <== mon.0 v2:172.21.15.52:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3050017070 con 0x7f30581029b0 2024-05-27T07:27:34.538 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:34.547+0000 7f305eaae700 1 -- 172.21.15.52:0/3978290760 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f30581a0010 con 0x7f30581029b0 2024-05-27T07:27:34.538 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:34.547+0000 7f3055ffb700 1 -- 172.21.15.52:0/3978290760 <== mon.0 v2:172.21.15.52:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f3050009de0 con 0x7f30581029b0 2024-05-27T07:27:34.539 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:34.547+0000 7f3055ffb700 1 -- 172.21.15.52:0/3978290760 <== mon.0 v2:172.21.15.52:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f305000f3c0 con 0x7f30581029b0 2024-05-27T07:27:34.539 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:34.548+0000 7f305eaae700 1 -- 172.21.15.52:0/3978290760 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f30581a7930 con 0x7f30581029b0 2024-05-27T07:27:34.541 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:34.551+0000 7f3055ffb700 1 -- 172.21.15.52:0/3978290760 <== mon.0 v2:172.21.15.52:3300/0 4 ==== mgrmap(e 26) ==== 149783+0+0 (secure 0 0 0) 0x7f3050009930 con 0x7f30581029b0 2024-05-27T07:27:34.542 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:34.553+0000 7f3055ffb700 1 --2- 172.21.15.52:0/3978290760 >> [v2:172.21.15.52:6800/1822318384,v1:172.21.15.52:6801/1822318384] conn(0x7f30440b29f0 0x7f30440b4f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:27:34.543 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:34.553+0000 7f3055ffb700 1 -- 172.21.15.52:0/3978290760 <== mon.0 v2:172.21.15.52:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9379+0+0 (secure 0 0 0) 0x7f30500e27b0 con 0x7f30581029b0 2024-05-27T07:27:34.543 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:34.553+0000 7f3057fff700 1 --2- 172.21.15.52:0/3978290760 >> [v2:172.21.15.52:6800/1822318384,v1:172.21.15.52:6801/1822318384] conn(0x7f30440b29f0 0x7f30440b4f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:27:34.543 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:34.553+0000 7f306d4c3b80 1 Processor -- start 2024-05-27T07:27:34.545 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:34.553+0000 7f306d4c3b80 1 -- start start 2024-05-27T07:27:34.545 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:34.554+0000 7f3057fff700 1 --2- 172.21.15.52:0/3978290760 >> [v2:172.21.15.52:6800/1822318384,v1:172.21.15.52:6801/1822318384] conn(0x7f30440b29f0 0x7f30440b4f00 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f3058105ed0 tx=0x7f304800c920 comp rx=0 tx=0).ready entity=mgr.15333 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:27:34.545 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:34.554+0000 7f306d4c3b80 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55b5a52ebe40 0x55b5a546f710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:27:34.545 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:34.554+0000 7f306d4c3b80 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55b5a546fc50 0x55b5a5469a40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:27:34.545 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:34.554+0000 7f306d4c3b80 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55b5a5469f80 0x55b5a546a390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:27:34.545 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:34.555+0000 7f306d4c3b80 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x55b5a5398c90 con 0x55b5a5469f80 2024-05-27T07:27:34.545 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:34.555+0000 7f305d04b700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55b5a5469f80 0x55b5a546a390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:27:34.545 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:34.555+0000 7f305d04b700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55b5a5469f80 0x55b5a546a390 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.52:53120/0 (socket says 172.21.15.52:53120) 2024-05-27T07:27:34.545 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:34.555+0000 7f305d04b700 1 -- 172.21.15.52:0/3995243944 learned_addr learned my addr 172.21.15.52:0/3995243944 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-05-27T07:27:34.545 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:34.555+0000 7f3057fff700 1 --2- 172.21.15.52:0/3995243944 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55b5a546fc50 0x55b5a5469a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:27:34.546 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:34.555+0000 7f306d4c3b80 1 -- 172.21.15.52:0/3995243944 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x55b5a5388830 con 0x55b5a52ebe40 2024-05-27T07:27:34.546 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:34.555+0000 7f306d4c3b80 1 -- 172.21.15.52:0/3995243944 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x55b5a538c870 con 0x55b5a546fc50 2024-05-27T07:27:34.546 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:34.555+0000 7f305d04b700 1 -- 172.21.15.52:0/3995243944 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55b5a546fc50 msgr2=0x55b5a5469a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:27:34.546 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:34.555+0000 7f305d04b700 1 --2- 172.21.15.52:0/3995243944 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55b5a546fc50 0x55b5a5469a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:27:34.546 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:34.555+0000 7f305d04b700 1 -- 172.21.15.52:0/3995243944 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55b5a52ebe40 msgr2=0x55b5a546f710 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-05-27T07:27:34.547 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:34.555+0000 7f305d04b700 1 --2- 172.21.15.52:0/3995243944 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55b5a52ebe40 0x55b5a546f710 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:27:34.547 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:34.555+0000 7f305d04b700 1 -- 172.21.15.52:0/3995243944 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55b5a52d88d0 con 0x55b5a5469f80 2024-05-27T07:27:34.547 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:34.555+0000 7f3057fff700 1 --2- 172.21.15.52:0/3995243944 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55b5a546fc50 0x55b5a5469a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-27T07:27:34.547 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:34.556+0000 7f305d04b700 1 --2- 172.21.15.52:0/3995243944 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55b5a5469f80 0x55b5a546a390 secure :-1 s=READY pgs=540 cs=0 l=1 rev1=1 crypto rx=0x7f304000b6c0 tx=0x7f304000b9d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5a8a4416ecd0d71e server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:27:34.547 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:34.556+0000 7f304effd700 1 -- 172.21.15.52:0/3995243944 <== mon.0 v2:172.21.15.52:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3040004a30 con 0x55b5a5469f80 2024-05-27T07:27:34.547 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:34.556+0000 7f304effd700 1 -- 172.21.15.52:0/3995243944 <== mon.0 v2:172.21.15.52:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f3040015070 con 0x55b5a5469f80 2024-05-27T07:27:34.547 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:34.556+0000 7f304effd700 1 -- 172.21.15.52:0/3995243944 <== mon.0 v2:172.21.15.52:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3040003ab0 con 0x55b5a5469f80 2024-05-27T07:27:34.547 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:34.557+0000 7f306d4c3b80 1 -- 172.21.15.52:0/3995243944 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55b5a5469f80 msgr2=0x55b5a546a390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:27:34.548 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:34.557+0000 7f306d4c3b80 1 --2- 172.21.15.52:0/3995243944 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55b5a5469f80 0x55b5a546a390 secure :-1 s=READY pgs=540 cs=0 l=1 rev1=1 crypto rx=0x7f304000b6c0 tx=0x7f304000b9d0 comp rx=0 tx=0).stop 2024-05-27T07:27:34.548 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:34.558+0000 7f306d4c3b80 1 -- 172.21.15.52:0/3995243944 shutdown_connections 2024-05-27T07:27:34.548 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:34.558+0000 7f306d4c3b80 1 --2- 172.21.15.52:0/3995243944 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55b5a5469f80 0x55b5a546a390 unknown :-1 s=CLOSED pgs=540 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:27:34.548 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:34.558+0000 7f306d4c3b80 1 --2- 172.21.15.52:0/3995243944 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55b5a52ebe40 0x55b5a546f710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:27:34.548 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:34.558+0000 7f306d4c3b80 1 --2- 172.21.15.52:0/3995243944 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55b5a546fc50 0x55b5a5469a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:27:34.548 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:34.558+0000 7f306d4c3b80 1 -- 172.21.15.52:0/3995243944 >> 172.21.15.52:0/3995243944 conn(0x55b5a5346af0 msgr2=0x55b5a5464640 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:27:34.548 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:34.558+0000 7f306d4c3b80 1 -- 172.21.15.52:0/3995243944 shutdown_connections 2024-05-27T07:27:34.548 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:34.558+0000 7f306d4c3b80 1 -- 172.21.15.52:0/3995243944 wait complete. 2024-05-27T07:27:34.549 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:34.558+0000 7f306d4c3b80 1 -- start start 2024-05-27T07:27:34.549 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:34.559+0000 7f306d4c3b80 1 Processor -- start 2024-05-27T07:27:34.550 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:34.559+0000 7f306d4c3b80 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55b5a52ebe40 0x55b5a5473400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:27:34.550 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:34.559+0000 7f306d4c3b80 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55b5a5469f80 0x55b5a5473940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:27:34.550 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:34.559+0000 7f306d4c3b80 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55b5a546fc50 0x55b5a5473e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:27:34.550 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:34.560+0000 7f305c84a700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55b5a52ebe40 0x55b5a5473400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:27:34.550 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:34.560+0000 7f305d04b700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55b5a546fc50 0x55b5a5473e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:27:34.550 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:34.560+0000 7f305d04b700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55b5a546fc50 0x55b5a5473e80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.52:53126/0 (socket says 172.21.15.52:53126) 2024-05-27T07:27:34.550 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:34.560+0000 7f305d04b700 1 -- 172.21.15.52:0/568305857 learned_addr learned my addr 172.21.15.52:0/568305857 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-05-27T07:27:34.550 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:34.560+0000 7f3057fff700 1 --2- 172.21.15.52:0/568305857 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55b5a5469f80 0x55b5a5473940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:27:34.550 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:34.560+0000 7f306d4c3b80 1 -- 172.21.15.52:0/568305857 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x55b5a528efa0 con 0x55b5a546fc50 2024-05-27T07:27:34.550 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:34.560+0000 7f306d4c3b80 1 -- 172.21.15.52:0/568305857 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x55b5a532a7c0 con 0x55b5a5469f80 2024-05-27T07:27:34.550 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:34.560+0000 7f306d4c3b80 1 -- 172.21.15.52:0/568305857 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x55b5a5476740 con 0x55b5a52ebe40 2024-05-27T07:27:34.550 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:34.560+0000 7f306d4c3b80 10 client.0 _ll_register_callbacks cb 0x55b5a5323920 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T07:27:34.552 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:34.560+0000 7f305d04b700 1 -- 172.21.15.52:0/568305857 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55b5a52ebe40 msgr2=0x55b5a5473400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:27:34.552 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:34.560+0000 7f305d04b700 1 --2- 172.21.15.52:0/568305857 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55b5a52ebe40 0x55b5a5473400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:27:34.552 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:34.560+0000 7f305d04b700 1 -- 172.21.15.52:0/568305857 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55b5a5469f80 msgr2=0x55b5a5473940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:27:34.552 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:34.560+0000 7f305d04b700 1 --2- 172.21.15.52:0/568305857 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55b5a5469f80 0x55b5a5473940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:27:34.552 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:34.560+0000 7f305d04b700 1 -- 172.21.15.52:0/568305857 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55b5a5327690 con 0x55b5a546fc50 2024-05-27T07:27:34.552 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:34.560+0000 7f3057fff700 1 --2- 172.21.15.52:0/568305857 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55b5a5469f80 0x55b5a5473940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-27T07:27:34.552 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:34.560+0000 7f305c84a700 1 --2- 172.21.15.52:0/568305857 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55b5a52ebe40 0x55b5a5473400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-27T07:27:34.552 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:34.561+0000 7f305d04b700 1 --2- 172.21.15.52:0/568305857 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55b5a546fc50 0x55b5a5473e80 secure :-1 s=READY pgs=541 cs=0 l=1 rev1=1 crypto rx=0x7f3040004c60 tx=0x7f3040004d40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e26b95a280af22dc server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:27:34.552 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:34.561+0000 7f3037fff700 10 client.0 ms_handle_connect on v2:172.21.15.52:3300/0 2024-05-27T07:27:34.552 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:34.561+0000 7f3037fff700 1 -- 172.21.15.52:0/568305857 <== mon.0 v2:172.21.15.52:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3040028060 con 0x55b5a546fc50 2024-05-27T07:27:34.552 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:34.561+0000 7f3037fff700 1 -- 172.21.15.52:0/568305857 <== mon.0 v2:172.21.15.52:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 9379+0+0 (secure 0 0 0) 0x7f3040005510 con 0x55b5a546fc50 2024-05-27T07:27:34.553 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:34.561+0000 7f306d4c3b80 1 -- 172.21.15.52:0/568305857 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f3040015070 con 0x55b5a546fc50 2024-05-27T07:27:34.553 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:34.562+0000 7f3037fff700 1 -- 172.21.15.52:0/568305857 <== mon.0 v2:172.21.15.52:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7f3040003740 con 0x55b5a546fc50 2024-05-27T07:27:34.553 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:34.562+0000 7f306d4c3b80 10 client.15894 fetch_fsmap learned FSMap version 30 2024-05-27T07:27:34.553 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:34.562+0000 7f306d4c3b80 1 -- 172.21.15.52:0/568305857 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x55b5a5481080 con 0x55b5a546fc50 2024-05-27T07:27:34.553 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:34.562+0000 7f3037fff700 1 -- 172.21.15.52:0/568305857 <== mon.0 v2:172.21.15.52:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f3040015070 con 0x55b5a546fc50 2024-05-27T07:27:34.554 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:34.563+0000 7f3037fff700 1 -- 172.21.15.52:0/568305857 <== mon.0 v2:172.21.15.52:3300/0 5 ==== fsmap(e 30) ==== 8362+0+0 (secure 0 0 0) 0x7f3040011a90 con 0x55b5a546fc50 2024-05-27T07:27:34.554 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:34.563+0000 7f306d4c3b80 10 client.15894 fetch_fsmap finished waiting for FSMap version 30 2024-05-27T07:27:34.554 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:34.563+0000 7f306d4c3b80 10 client.15894 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-05-27T07:27:34.554 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:34.563+0000 7f306d4c3b80 20 client.15894 populate_metadata read hostname 'smithi052' 2024-05-27T07:27:34.554 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:34.563+0000 7f306d4c3b80 1 --2- 172.21.15.52:0/568305857 >> [v2:172.21.15.105:6838/500348151,v1:172.21.15.105:6839/500348151] conn(0x55b5a5483550 0x55b5a5485970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:27:34.554 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:34.563+0000 7f306d4c3b80 4 client.15894 mds_command: new command op to 24527 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-27T07:27:34.554 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:34.563+0000 7f306d4c3b80 1 -- 172.21.15.52:0/568305857 --> [v2:172.21.15.105:6838/500348151,v1:172.21.15.105:6839/500348151] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55b5a54359f0 con 0x55b5a5483550 2024-05-27T07:27:34.554 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:34.563+0000 7f305c84a700 1 --2- 172.21.15.52:0/568305857 >> [v2:172.21.15.105:6838/500348151,v1:172.21.15.105:6839/500348151] conn(0x55b5a5483550 0x55b5a5485970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:27:34.555 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:34.564+0000 7f3037fff700 10 client.15894 ms_handle_connect on v2:172.21.15.105:6838/500348151 2024-05-27T07:27:34.555 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:34.564+0000 7f305c84a700 1 --2- 172.21.15.52:0/568305857 >> [v2:172.21.15.105:6838/500348151,v1:172.21.15.105:6839/500348151] conn(0x55b5a5483550 0x55b5a5485970 crc :-1 s=READY pgs=67 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=c8a695cbbed1cda1 server_cookie=567d8edb4078b0 in_seq=0 out_seq=0 2024-05-27T07:27:34.556 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:34.564+0000 7f3037fff700 1 -- 172.21.15.52:0/568305857 <== mds.0 v2:172.21.15.105:6838/500348151 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24041 (crc 0 0 0) 0x55b5a54359f0 con 0x55b5a5483550 2024-05-27T07:27:34.556 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:34.564+0000 7f3037fff700 10 client.15894 handle_command_reply: tid=0 2024-05-27T07:27:34.556 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:34.565+0000 7f306d4c3b80 1 client.15894 shutdown 2024-05-27T07:27:34.556 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:34.565+0000 7f306d4c3b80 1 -- 172.21.15.52:0/568305857 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55b5a546fc50 msgr2=0x55b5a5473e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:27:34.556 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:34.565+0000 7f306d4c3b80 1 --2- 172.21.15.52:0/568305857 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55b5a546fc50 0x55b5a5473e80 secure :-1 s=READY pgs=541 cs=0 l=1 rev1=1 crypto rx=0x7f3040004c60 tx=0x7f3040004d40 comp rx=0 tx=0).stop 2024-05-27T07:27:34.556 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:34.565+0000 7f306d4c3b80 1 -- 172.21.15.52:0/568305857 shutdown_connections 2024-05-27T07:27:34.557 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:34.565+0000 7f306d4c3b80 1 --2- 172.21.15.52:0/568305857 >> [v2:172.21.15.105:6838/500348151,v1:172.21.15.105:6839/500348151] conn(0x55b5a5483550 0x55b5a5485970 crc :-1 s=READY pgs=67 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:27:34.557 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:34.565+0000 7f3037fff700 0 client.15894 ms_handle_reset on v2:172.21.15.105:6838/500348151 2024-05-27T07:27:34.557 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:34.565+0000 7f306d4c3b80 1 --2- 172.21.15.52:0/568305857 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55b5a546fc50 0x55b5a5473e80 unknown :-1 s=CLOSED pgs=541 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:27:34.557 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:34.565+0000 7f306d4c3b80 1 --2- 172.21.15.52:0/568305857 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55b5a52ebe40 0x55b5a5473400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:27:34.557 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:34.565+0000 7f306d4c3b80 1 --2- 172.21.15.52:0/568305857 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55b5a5469f80 0x55b5a5473940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:27:34.557 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:34.565+0000 7f306d4c3b80 1 -- 172.21.15.52:0/568305857 >> 172.21.15.52:0/568305857 conn(0x55b5a5346af0 msgr2=0x55b5a5468a30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:27:34.557 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:34.565+0000 7f306d4c3b80 1 -- 172.21.15.52:0/568305857 shutdown_connections 2024-05-27T07:27:34.557 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:34.565+0000 7f306d4c3b80 1 -- 172.21.15.52:0/568305857 wait complete. 2024-05-27T07:27:34.557 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:34.566+0000 7f306d4c3b80 20 client.15894 trim_cache size 0 max 16384 2024-05-27T07:27:34.586 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:34.596+0000 7f306d4c3b80 1 Processor -- start 2024-05-27T07:27:34.587 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:34.596+0000 7f306d4c3b80 1 -- start start 2024-05-27T07:27:34.587 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:34.596+0000 7f306d4c3b80 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55b5a52ebe40 0x55b5a5469a40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:27:34.587 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:34.596+0000 7f306d4c3b80 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55b5a5469f80 0x55b5a5472720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:27:34.587 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:34.596+0000 7f306d4c3b80 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55b5a546fc50 0x55b5a5473cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:27:34.587 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:34.597+0000 7f305c84a700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55b5a52ebe40 0x55b5a5469a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:27:34.587 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:34.597+0000 7f305d04b700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55b5a546fc50 0x55b5a5473cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:27:34.587 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:34.597+0000 7f305c84a700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55b5a52ebe40 0x55b5a5469a40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.173:3300/0 says I am v2:172.21.15.52:42422/0 (socket says 172.21.15.52:42422) 2024-05-27T07:27:34.587 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:34.597+0000 7f3057fff700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55b5a5469f80 0x55b5a5472720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:27:34.587 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:34.597+0000 7f305d04b700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55b5a546fc50 0x55b5a5473cc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.52:53128/0 (socket says 172.21.15.52:53128) 2024-05-27T07:27:34.590 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:34.597+0000 7f3057fff700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55b5a5469f80 0x55b5a5472720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.52:42284/0 (socket says 172.21.15.52:42284) 2024-05-27T07:27:34.590 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:34.597+0000 7f3057fff700 1 -- 172.21.15.52:0/3495579951 learned_addr learned my addr 172.21.15.52:0/3495579951 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-05-27T07:27:34.590 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:34.597+0000 7f306d4c3b80 1 -- 172.21.15.52:0/3495579951 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x7f30581b06d0 con 0x55b5a546fc50 2024-05-27T07:27:34.590 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:34.597+0000 7f306d4c3b80 1 -- 172.21.15.52:0/3495579951 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f30581b2210 con 0x55b5a5469f80 2024-05-27T07:27:34.590 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:34.597+0000 7f306d4c3b80 1 -- 172.21.15.52:0/3495579951 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f30581b0da0 con 0x55b5a52ebe40 2024-05-27T07:27:34.590 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:34.597+0000 7f3057fff700 1 -- 172.21.15.52:0/3495579951 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55b5a52ebe40 msgr2=0x55b5a5469a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:27:34.590 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:34.597+0000 7f3057fff700 1 --2- 172.21.15.52:0/3495579951 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55b5a52ebe40 0x55b5a5469a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:27:34.590 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:34.597+0000 7f3057fff700 1 -- 172.21.15.52:0/3495579951 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55b5a546fc50 msgr2=0x55b5a5473cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:27:34.591 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:34.597+0000 7f3057fff700 1 --2- 172.21.15.52:0/3495579951 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55b5a546fc50 0x55b5a5473cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:27:34.591 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:34.597+0000 7f3057fff700 1 -- 172.21.15.52:0/3495579951 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55b5a5471930 con 0x55b5a5469f80 2024-05-27T07:27:34.591 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:34.597+0000 7f305c84a700 1 --2- 172.21.15.52:0/3495579951 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55b5a52ebe40 0x55b5a5469a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-27T07:27:34.591 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:34.597+0000 7f305d04b700 1 --2- 172.21.15.52:0/3495579951 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55b5a546fc50 0x55b5a5473cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-27T07:27:34.591 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:34.597+0000 7f3057fff700 1 --2- 172.21.15.52:0/3495579951 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55b5a5469f80 0x55b5a5472720 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f304800c210 tx=0x7f304801d000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c00c51cbd16e1cce server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:27:34.591 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:34.598+0000 7f304e7fc700 1 -- 172.21.15.52:0/3495579951 <== mon.2 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3048021040 con 0x55b5a5469f80 2024-05-27T07:27:34.591 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:34.598+0000 7f304e7fc700 1 -- 172.21.15.52:0/3495579951 <== mon.2 v2:172.21.15.105:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f304802e070 con 0x55b5a5469f80 2024-05-27T07:27:34.591 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:34.598+0000 7f304e7fc700 1 -- 172.21.15.52:0/3495579951 <== mon.2 v2:172.21.15.105:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3048007950 con 0x55b5a5469f80 2024-05-27T07:27:34.591 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:34.599+0000 7f306d4c3b80 1 -- 172.21.15.52:0/3495579951 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55b5a5469f80 msgr2=0x55b5a5472720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:27:34.591 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:34.599+0000 7f306d4c3b80 1 --2- 172.21.15.52:0/3495579951 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55b5a5469f80 0x55b5a5472720 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f304800c210 tx=0x7f304801d000 comp rx=0 tx=0).stop 2024-05-27T07:27:34.591 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:34.599+0000 7f306d4c3b80 1 -- 172.21.15.52:0/3495579951 shutdown_connections 2024-05-27T07:27:34.591 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:34.599+0000 7f306d4c3b80 1 --2- 172.21.15.52:0/3495579951 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55b5a546fc50 0x55b5a5473cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:27:34.591 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:34.599+0000 7f306d4c3b80 1 --2- 172.21.15.52:0/3495579951 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55b5a52ebe40 0x55b5a5469a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:27:34.592 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:34.599+0000 7f306d4c3b80 1 --2- 172.21.15.52:0/3495579951 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55b5a5469f80 0x55b5a5472720 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:27:34.592 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:34.599+0000 7f306d4c3b80 1 -- 172.21.15.52:0/3495579951 >> 172.21.15.52:0/3495579951 conn(0x55b5a5346af0 msgr2=0x55b5a546eab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:27:34.592 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:34.599+0000 7f306d4c3b80 1 -- 172.21.15.52:0/3495579951 shutdown_connections 2024-05-27T07:27:34.592 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:34.599+0000 7f306d4c3b80 1 -- 172.21.15.52:0/3495579951 wait complete. 2024-05-27T07:27:34.592 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:34.600+0000 7f306d4c3b80 1 -- start start 2024-05-27T07:27:34.592 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:34.600+0000 7f306d4c3b80 1 Processor -- start 2024-05-27T07:27:34.592 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:34.600+0000 7f306d4c3b80 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55b5a52ebe40 0x55b5a5488c20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:27:34.595 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:34.600+0000 7f306d4c3b80 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55b5a5469f80 0x55b5a5489160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:27:34.595 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:34.601+0000 7f306d4c3b80 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55b5a546fc50 0x55b5a54e6730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:27:34.595 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:34.601+0000 7f305c84a700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55b5a52ebe40 0x55b5a5488c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:27:34.595 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:34.601+0000 7f305c84a700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55b5a52ebe40 0x55b5a5488c20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.173:3300/0 says I am v2:172.21.15.52:42426/0 (socket says 172.21.15.52:42426) 2024-05-27T07:27:34.595 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:34.601+0000 7f3057fff700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55b5a5469f80 0x55b5a5489160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:27:34.595 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:34.601+0000 7f3057fff700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55b5a5469f80 0x55b5a5489160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.52:42290/0 (socket says 172.21.15.52:42290) 2024-05-27T07:27:34.595 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:34.601+0000 7f305d04b700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55b5a546fc50 0x55b5a54e6730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:27:34.595 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:34.601+0000 7f305d04b700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55b5a546fc50 0x55b5a54e6730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.52:53134/0 (socket says 172.21.15.52:53134) 2024-05-27T07:27:34.596 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:34.601+0000 7f305c84a700 1 -- 172.21.15.52:0/1813799491 learned_addr learned my addr 172.21.15.52:0/1813799491 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-05-27T07:27:34.596 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:34.601+0000 7f306d4c3b80 1 -- 172.21.15.52:0/1813799491 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x7f30581b1b40 con 0x55b5a546fc50 2024-05-27T07:27:34.596 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:34.601+0000 7f306d4c3b80 1 -- 172.21.15.52:0/1813799491 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f30581b1470 con 0x55b5a5469f80 2024-05-27T07:27:34.596 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:34.601+0000 7f306d4c3b80 1 -- 172.21.15.52:0/1813799491 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f30581ae670 con 0x55b5a52ebe40 2024-05-27T07:27:34.596 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:34.601+0000 7f306d4c3b80 10 client.0 _ll_register_callbacks cb 0x55b5a5323920 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T07:27:34.596 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:34.601+0000 7f3057fff700 1 -- 172.21.15.52:0/1813799491 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55b5a52ebe40 msgr2=0x55b5a5488c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:27:34.596 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:34.601+0000 7f3057fff700 1 --2- 172.21.15.52:0/1813799491 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55b5a52ebe40 0x55b5a5488c20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:27:34.596 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:34.601+0000 7f3057fff700 1 -- 172.21.15.52:0/1813799491 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55b5a546fc50 msgr2=0x55b5a54e6730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:27:34.596 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:34.601+0000 7f3057fff700 1 --2- 172.21.15.52:0/1813799491 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55b5a546fc50 0x55b5a54e6730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:27:34.596 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:34.601+0000 7f3057fff700 1 -- 172.21.15.52:0/1813799491 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55b5a54e6ed0 con 0x55b5a5469f80 2024-05-27T07:27:34.596 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:34.602+0000 7f3057fff700 1 --2- 172.21.15.52:0/1813799491 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55b5a5469f80 0x55b5a5489160 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f30480062a0 tx=0x7f3048008d00 comp rx=0 tx=0).ready entity=mon.2 client_cookie=2a3b729da0d668a5 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:27:34.596 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:34.602+0000 7f3037fff700 10 client.0 ms_handle_connect on v2:172.21.15.105:3300/0 2024-05-27T07:27:34.597 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:34.602+0000 7f3037fff700 1 -- 172.21.15.52:0/1813799491 <== mon.2 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3048021050 con 0x55b5a5469f80 2024-05-27T07:27:34.597 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:34.602+0000 7f306d4c3b80 1 -- 172.21.15.52:0/1813799491 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f304802e070 con 0x55b5a5469f80 2024-05-27T07:27:34.597 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:34.602+0000 7f3037fff700 1 -- 172.21.15.52:0/1813799491 <== mon.2 v2:172.21.15.105:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 9379+0+0 (secure 0 0 0) 0x7f30480077b0 con 0x55b5a5469f80 2024-05-27T07:27:34.597 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:34.602+0000 7f3037fff700 1 -- 172.21.15.52:0/1813799491 <== mon.2 v2:172.21.15.105:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7f30480079b0 con 0x55b5a5469f80 2024-05-27T07:27:34.597 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:34.602+0000 7f306d4c3b80 10 client.25589 fetch_fsmap learned FSMap version 30 2024-05-27T07:27:34.597 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:34.602+0000 7f306d4c3b80 1 -- 172.21.15.52:0/1813799491 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x55b5a5488320 con 0x55b5a5469f80 2024-05-27T07:27:34.597 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:34.603+0000 7f3037fff700 1 -- 172.21.15.52:0/1813799491 <== mon.2 v2:172.21.15.105:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f3048048070 con 0x55b5a5469f80 2024-05-27T07:27:34.597 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:34.603+0000 7f3037fff700 1 -- 172.21.15.52:0/1813799491 <== mon.2 v2:172.21.15.105:3300/0 5 ==== fsmap(e 30) ==== 8362+0+0 (secure 0 0 0) 0x7f3048012070 con 0x55b5a5469f80 2024-05-27T07:27:34.597 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:34.603+0000 7f306d4c3b80 10 client.25589 fetch_fsmap finished waiting for FSMap version 30 2024-05-27T07:27:34.597 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:34.603+0000 7f306d4c3b80 10 client.25589 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-05-27T07:27:34.597 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:34.603+0000 7f306d4c3b80 20 client.25589 populate_metadata read hostname 'smithi052' 2024-05-27T07:27:34.597 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:34.603+0000 7f306d4c3b80 1 --2- 172.21.15.52:0/1813799491 >> [v2:172.21.15.105:6838/500348151,v1:172.21.15.105:6839/500348151] conn(0x55b5a54e81c0 0x55b5a54ea5e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:27:34.597 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:34.603+0000 7f306d4c3b80 4 client.25589 mds_command: new command op to 24527 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-05-27T07:27:34.598 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:34.603+0000 7f306d4c3b80 1 -- 172.21.15.52:0/1813799491 --> [v2:172.21.15.105:6838/500348151,v1:172.21.15.105:6839/500348151] -- command(tid 0: {"prefix": "damage ls"}) -- 0x7f304802e070 con 0x55b5a54e81c0 2024-05-27T07:27:34.598 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:34.603+0000 7f305c84a700 1 --2- 172.21.15.52:0/1813799491 >> [v2:172.21.15.105:6838/500348151,v1:172.21.15.105:6839/500348151] conn(0x55b5a54e81c0 0x55b5a54ea5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:27:34.598 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:34.604+0000 7f3037fff700 10 client.25589 ms_handle_connect on v2:172.21.15.105:6838/500348151 2024-05-27T07:27:34.598 INFO:teuthology.orchestra.run.smithi052.stdout:[] 2024-05-27T07:27:34.600 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:34.604+0000 7f305c84a700 1 --2- 172.21.15.52:0/1813799491 >> [v2:172.21.15.105:6838/500348151,v1:172.21.15.105:6839/500348151] conn(0x55b5a54e81c0 0x55b5a54ea5e0 crc :-1 s=READY pgs=68 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=47466fb7a5888140 server_cookie=4726dc7bf4107f6d in_seq=0 out_seq=0 2024-05-27T07:27:34.600 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:34.604+0000 7f3037fff700 1 -- 172.21.15.52:0/1813799491 <== mds.0 v2:172.21.15.105:6838/500348151 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x7f304802e070 con 0x55b5a54e81c0 2024-05-27T07:27:34.600 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:34.604+0000 7f3037fff700 10 client.25589 handle_command_reply: tid=0 2024-05-27T07:27:34.600 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:34.604+0000 7f306d4c3b80 1 client.25589 shutdown 2024-05-27T07:27:34.600 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:34.604+0000 7f306d4c3b80 1 -- 172.21.15.52:0/1813799491 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55b5a5469f80 msgr2=0x55b5a5489160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:27:34.600 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:34.604+0000 7f306d4c3b80 1 --2- 172.21.15.52:0/1813799491 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55b5a5469f80 0x55b5a5489160 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f30480062a0 tx=0x7f3048008d00 comp rx=0 tx=0).stop 2024-05-27T07:27:34.600 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:34.604+0000 7f306d4c3b80 1 -- 172.21.15.52:0/1813799491 shutdown_connections 2024-05-27T07:27:34.600 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:34.604+0000 7f306d4c3b80 1 --2- 172.21.15.52:0/1813799491 >> [v2:172.21.15.105:6838/500348151,v1:172.21.15.105:6839/500348151] conn(0x55b5a54e81c0 0x55b5a54ea5e0 crc :-1 s=READY pgs=68 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:27:34.600 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:34.604+0000 7f306d4c3b80 1 --2- 172.21.15.52:0/1813799491 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55b5a546fc50 0x55b5a54e6730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:27:34.601 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:34.604+0000 7f306d4c3b80 1 --2- 172.21.15.52:0/1813799491 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55b5a52ebe40 0x55b5a5488c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:27:34.601 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:34.604+0000 7f306d4c3b80 1 --2- 172.21.15.52:0/1813799491 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55b5a5469f80 0x55b5a5489160 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:27:34.601 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:34.604+0000 7f306d4c3b80 1 -- 172.21.15.52:0/1813799491 >> 172.21.15.52:0/1813799491 conn(0x55b5a5346af0 msgr2=0x55b5a546d010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:27:34.601 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:34.604+0000 7f3037fff700 0 client.25589 ms_handle_reset on v2:172.21.15.105:6838/500348151 2024-05-27T07:27:34.601 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:34.605+0000 7f306d4c3b80 1 -- 172.21.15.52:0/1813799491 shutdown_connections 2024-05-27T07:27:34.601 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:34.605+0000 7f306d4c3b80 1 -- 172.21.15.52:0/1813799491 wait complete. 2024-05-27T07:27:34.601 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:34.605+0000 7f306d4c3b80 20 client.25589 trim_cache size 0 max 16384 2024-05-27T07:27:34.601 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:34.605+0000 7f305eaae700 1 -- 172.21.15.52:0/3978290760 >> [v2:172.21.15.52:6800/1822318384,v1:172.21.15.52:6801/1822318384] conn(0x7f30440b29f0 msgr2=0x7f30440b4f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:27:34.601 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:34.605+0000 7f305eaae700 1 --2- 172.21.15.52:0/3978290760 >> [v2:172.21.15.52:6800/1822318384,v1:172.21.15.52:6801/1822318384] conn(0x7f30440b29f0 0x7f30440b4f00 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f3058105ed0 tx=0x7f304800c920 comp rx=0 tx=0).stop 2024-05-27T07:27:34.601 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:34.605+0000 7f305eaae700 1 -- 172.21.15.52:0/3978290760 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f30581029b0 msgr2=0x7f305819ae50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:27:34.601 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:34.605+0000 7f305eaae700 1 --2- 172.21.15.52:0/3978290760 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f30581029b0 0x7f305819ae50 secure :-1 s=READY pgs=539 cs=0 l=1 rev1=1 crypto rx=0x7f305000d980 tx=0x7f305000dc90 comp rx=0 tx=0).stop 2024-05-27T07:27:34.601 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:34.606+0000 7f305eaae700 1 -- 172.21.15.52:0/3978290760 shutdown_connections 2024-05-27T07:27:34.602 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:34.606+0000 7f305eaae700 1 --2- 172.21.15.52:0/3978290760 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f30581089e0 0x7f305819f680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:27:34.602 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:34.606+0000 7f305eaae700 1 --2- 172.21.15.52:0/3978290760 >> [v2:172.21.15.52:6800/1822318384,v1:172.21.15.52:6801/1822318384] conn(0x7f30440b29f0 0x7f30440b4f00 unknown :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:27:34.602 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:34.606+0000 7f305eaae700 1 --2- 172.21.15.52:0/3978290760 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f30581029b0 0x7f305819ae50 unknown :-1 s=CLOSED pgs=539 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:27:34.602 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:34.606+0000 7f305eaae700 1 --2- 172.21.15.52:0/3978290760 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f3058103300 0x7f305819b3f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:27:34.602 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:34.606+0000 7f305eaae700 1 -- 172.21.15.52:0/3978290760 >> 172.21.15.52:0/3978290760 conn(0x7f30580fe6e0 msgr2=0x7f3058107c70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:27:34.602 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:34.606+0000 7f305eaae700 1 -- 172.21.15.52:0/3978290760 shutdown_connections 2024-05-27T07:27:34.602 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:34.606+0000 7f305eaae700 1 -- 172.21.15.52:0/3978290760 wait complete. 2024-05-27T07:27:34.602 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:27:34 smithi173 ceph-mon[37000]: pgmap v51: 129 pgs: 129 active+clean; 1.4 MiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 6.8 KiB/s rd, 1.5 KiB/s wr, 15 op/s 2024-05-27T07:27:36.334 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-05-27T07:27:36.335 DEBUG:teuthology.orchestra.run.smithi052:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:38f9d900c83c63799fbdbe61acc9a11b0d3554a6 shell --fsid 73df2636-1bf8-11ef-bc9b-c7b262605968 -- ceph tell mds.1:0 scrub start / force,recursive 2024-05-27T07:27:36.439 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:27:36 smithi105 ceph-mon[35977]: pgmap v52: 129 pgs: 129 active+clean; 1.4 MiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 1.2 KiB/s wr, 10 op/s 2024-05-27T07:27:36.479 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:27:36 smithi052 ceph-mon[31955]: pgmap v52: 129 pgs: 129 active+clean; 1.4 MiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 1.2 KiB/s wr, 10 op/s 2024-05-27T07:27:36.601 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:27:36 smithi173 ceph-mon[37000]: pgmap v52: 129 pgs: 129 active+clean; 1.4 MiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 1.2 KiB/s wr, 10 op/s 2024-05-27T07:27:36.615 INFO:teuthology.orchestra.run.smithi052.stderr:Inferring config /var/lib/ceph/73df2636-1bf8-11ef-bc9b-c7b262605968/mon.a/config 2024-05-27T07:27:38.390 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:27:38 smithi052 ceph-mon[31955]: pgmap v53: 129 pgs: 129 active+clean; 1.4 MiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 1.2 KiB/s wr, 10 op/s 2024-05-27T07:27:38.439 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:27:38 smithi105 ceph-mon[35977]: pgmap v53: 129 pgs: 129 active+clean; 1.4 MiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 1.2 KiB/s wr, 10 op/s 2024-05-27T07:27:38.566 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:38.576+0000 7f7499e6e700 1 -- 172.21.15.52:0/1661849166 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f74941029b0 msgr2=0x7f7494102dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:27:38.567 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:38.576+0000 7f7499e6e700 1 --2- 172.21.15.52:0/1661849166 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f74941029b0 0x7f7494102dc0 secure :-1 s=READY pgs=542 cs=0 l=1 rev1=1 crypto rx=0x7f748800aaa0 tx=0x7f748800adb0 comp rx=0 tx=0).stop 2024-05-27T07:27:38.567 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:38.576+0000 7f7499e6e700 1 -- 172.21.15.52:0/1661849166 shutdown_connections 2024-05-27T07:27:38.567 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:38.576+0000 7f7499e6e700 1 --2- 172.21.15.52:0/1661849166 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f7494103300 0x7f749410f880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:27:38.567 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:38.576+0000 7f7499e6e700 1 --2- 172.21.15.52:0/1661849166 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f74941089e0 0x7f7494108df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:27:38.567 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:38.576+0000 7f7499e6e700 1 --2- 172.21.15.52:0/1661849166 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f74941029b0 0x7f7494102dc0 unknown :-1 s=CLOSED pgs=542 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:27:38.567 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:38.576+0000 7f7499e6e700 1 -- 172.21.15.52:0/1661849166 >> 172.21.15.52:0/1661849166 conn(0x7f74940fe6e0 msgr2=0x7f7494100b00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:27:38.567 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:38.577+0000 7f7499e6e700 1 -- 172.21.15.52:0/1661849166 shutdown_connections 2024-05-27T07:27:38.567 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:38.577+0000 7f7499e6e700 1 -- 172.21.15.52:0/1661849166 wait complete. 2024-05-27T07:27:38.569 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:38.578+0000 7f7499e6e700 1 Processor -- start 2024-05-27T07:27:38.569 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:38.578+0000 7f7499e6e700 1 -- start start 2024-05-27T07:27:38.569 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:38.578+0000 7f7499e6e700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f74941029b0 0x7f749419af30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:27:38.569 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:38.578+0000 7f7499e6e700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f7494103300 0x7f749419b4d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:27:38.569 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:38.578+0000 7f7492ffd700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f7494103300 0x7f749419b4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:27:38.569 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:38.578+0000 7f7492ffd700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f7494103300 0x7f749419b4d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.52:53154/0 (socket says 172.21.15.52:53154) 2024-05-27T07:27:38.569 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:38.578+0000 7f7499e6e700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f74941089e0 0x7f749419f760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:27:38.569 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:38.579+0000 7f7492ffd700 1 -- 172.21.15.52:0/373209529 learned_addr learned my addr 172.21.15.52:0/373209529 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-05-27T07:27:38.569 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:38.579+0000 7f7499e6e700 1 -- 172.21.15.52:0/373209529 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x7f7494112330 con 0x7f7494103300 2024-05-27T07:27:38.570 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:38.579+0000 7f7499e6e700 1 -- 172.21.15.52:0/373209529 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f74941121b0 con 0x7f74941029b0 2024-05-27T07:27:38.570 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:38.579+0000 7f7499e6e700 1 -- 172.21.15.52:0/373209529 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f74941124b0 con 0x7f74941089e0 2024-05-27T07:27:38.570 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:38.579+0000 7f74937fe700 1 --2- 172.21.15.52:0/373209529 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f74941029b0 0x7f749419af30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:27:38.570 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:38.579+0000 7f7493fff700 1 --2- 172.21.15.52:0/373209529 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f74941089e0 0x7f749419f760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:27:38.570 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:38.579+0000 7f7492ffd700 1 -- 172.21.15.52:0/373209529 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f74941089e0 msgr2=0x7f749419f760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:27:38.570 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:38.579+0000 7f7492ffd700 1 --2- 172.21.15.52:0/373209529 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f74941089e0 0x7f749419f760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:27:38.570 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:38.579+0000 7f7492ffd700 1 -- 172.21.15.52:0/373209529 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f74941029b0 msgr2=0x7f749419af30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:27:38.570 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:38.579+0000 7f7492ffd700 1 --2- 172.21.15.52:0/373209529 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f74941029b0 0x7f749419af30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:27:38.570 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:38.579+0000 7f7492ffd700 1 -- 172.21.15.52:0/373209529 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f749419fe00 con 0x7f7494103300 2024-05-27T07:27:38.570 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:38.579+0000 7f74937fe700 1 --2- 172.21.15.52:0/373209529 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f74941029b0 0x7f749419af30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-27T07:27:38.571 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:38.580+0000 7f7492ffd700 1 --2- 172.21.15.52:0/373209529 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f7494103300 0x7f749419b4d0 secure :-1 s=READY pgs=543 cs=0 l=1 rev1=1 crypto rx=0x7f7488003930 tx=0x7f7488003960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:27:38.571 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:38.580+0000 7f7490ff9700 1 -- 172.21.15.52:0/373209529 <== mon.0 v2:172.21.15.52:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7488003ed0 con 0x7f7494103300 2024-05-27T07:27:38.571 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:38.580+0000 7f7490ff9700 1 -- 172.21.15.52:0/373209529 <== mon.0 v2:172.21.15.52:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f7488004c90 con 0x7f7494103300 2024-05-27T07:27:38.571 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:38.581+0000 7f7490ff9700 1 -- 172.21.15.52:0/373209529 <== mon.0 v2:172.21.15.52:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f74880054a0 con 0x7f7494103300 2024-05-27T07:27:38.571 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:38.581+0000 7f7499e6e700 1 -- 172.21.15.52:0/373209529 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f74941a0090 con 0x7f7494103300 2024-05-27T07:27:38.571 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:38.581+0000 7f7499e6e700 1 -- 172.21.15.52:0/373209529 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f74941a0540 con 0x7f7494103300 2024-05-27T07:27:38.574 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:38.585+0000 7f7490ff9700 1 -- 172.21.15.52:0/373209529 <== mon.0 v2:172.21.15.52:3300/0 4 ==== mgrmap(e 26) ==== 149783+0+0 (secure 0 0 0) 0x7f748801e070 con 0x7f7494103300 2024-05-27T07:27:38.575 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:38.586+0000 7f7490ff9700 1 --2- 172.21.15.52:0/373209529 >> [v2:172.21.15.52:6800/1822318384,v1:172.21.15.52:6801/1822318384] conn(0x7f747c0b28d0 0x7f747c0b4de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:27:38.575 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:38.586+0000 7f74937fe700 1 --2- 172.21.15.52:0/373209529 >> [v2:172.21.15.52:6800/1822318384,v1:172.21.15.52:6801/1822318384] conn(0x7f747c0b28d0 0x7f747c0b4de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:27:38.576 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:38.586+0000 7f7490ff9700 1 -- 172.21.15.52:0/373209529 <== mon.0 v2:172.21.15.52:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9379+0+0 (secure 0 0 0) 0x7f74880e37f0 con 0x7f7494103300 2024-05-27T07:27:38.576 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:38.586+0000 7f74937fe700 1 --2- 172.21.15.52:0/373209529 >> [v2:172.21.15.52:6800/1822318384,v1:172.21.15.52:6801/1822318384] conn(0x7f747c0b28d0 0x7f747c0b4de0 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f7494103af0 tx=0x7f74840073d0 comp rx=0 tx=0).ready entity=mgr.15333 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:27:38.576 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:38.586+0000 7f74a8883b80 1 Processor -- start 2024-05-27T07:27:38.576 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:38.587+0000 7f74a8883b80 1 -- start start 2024-05-27T07:27:38.578 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:38.587+0000 7f74a8883b80 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55fbdbfa9ec0 0x55fbdc12d790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:27:38.578 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:38.587+0000 7f74a8883b80 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55fbdc12dcd0 0x55fbdc127ac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:27:38.578 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:38.587+0000 7f74a8883b80 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55fbdc128000 0x55fbdc128410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:27:38.578 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:38.587+0000 7f74a8883b80 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x55fbdc056d10 con 0x55fbdc12dcd0 2024-05-27T07:27:38.578 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:38.587+0000 7f74a8883b80 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x55fbdc0468b0 con 0x55fbdc128000 2024-05-27T07:27:38.578 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:38.587+0000 7f74a8883b80 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x55fbdc04a8f0 con 0x55fbdbfa9ec0 2024-05-27T07:27:38.578 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:38.587+0000 7f7492ffd700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55fbdc12dcd0 0x55fbdc127ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:27:38.578 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:38.587+0000 7f7492ffd700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55fbdc12dcd0 0x55fbdc127ac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.52:53164/0 (socket says 172.21.15.52:53164) 2024-05-27T07:27:38.578 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:38.587+0000 7f7492ffd700 1 -- 172.21.15.52:0/873378897 learned_addr learned my addr 172.21.15.52:0/873378897 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-05-27T07:27:38.578 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:38.587+0000 7f7493fff700 1 --2- 172.21.15.52:0/873378897 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55fbdc128000 0x55fbdc128410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:27:38.579 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:38.587+0000 7f74937fe700 1 --2- 172.21.15.52:0/873378897 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55fbdbfa9ec0 0x55fbdc12d790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:27:38.579 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:38.588+0000 7f7492ffd700 1 -- 172.21.15.52:0/873378897 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55fbdbfa9ec0 msgr2=0x55fbdc12d790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:27:38.579 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:38.588+0000 7f7492ffd700 1 --2- 172.21.15.52:0/873378897 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55fbdbfa9ec0 0x55fbdc12d790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:27:38.579 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:38.588+0000 7f7492ffd700 1 -- 172.21.15.52:0/873378897 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55fbdc128000 msgr2=0x55fbdc128410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:27:38.579 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:38.588+0000 7f7492ffd700 1 --2- 172.21.15.52:0/873378897 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55fbdc128000 0x55fbdc128410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:27:38.579 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:38.588+0000 7f7492ffd700 1 -- 172.21.15.52:0/873378897 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55fbdbf96950 con 0x55fbdc12dcd0 2024-05-27T07:27:38.579 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:38.588+0000 7f74937fe700 1 --2- 172.21.15.52:0/873378897 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55fbdbfa9ec0 0x55fbdc12d790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-27T07:27:38.579 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:38.588+0000 7f7492ffd700 1 --2- 172.21.15.52:0/873378897 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55fbdc12dcd0 0x55fbdc127ac0 secure :-1 s=READY pgs=544 cs=0 l=1 rev1=1 crypto rx=0x7f74880a6910 tx=0x7f74880aa770 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1c15898e239e0d27 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:27:38.579 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:38.588+0000 7f7479ffb700 1 -- 172.21.15.52:0/873378897 <== mon.0 v2:172.21.15.52:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f74880aaf90 con 0x55fbdc12dcd0 2024-05-27T07:27:38.579 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:38.588+0000 7f7479ffb700 1 -- 172.21.15.52:0/873378897 <== mon.0 v2:172.21.15.52:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f74880aa890 con 0x55fbdc12dcd0 2024-05-27T07:27:38.579 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:38.588+0000 7f7479ffb700 1 -- 172.21.15.52:0/873378897 <== mon.0 v2:172.21.15.52:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7488004570 con 0x55fbdc12dcd0 2024-05-27T07:27:38.580 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:38.589+0000 7f74a8883b80 1 -- 172.21.15.52:0/873378897 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55fbdc12dcd0 msgr2=0x55fbdc127ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:27:38.580 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:38.589+0000 7f74a8883b80 1 --2- 172.21.15.52:0/873378897 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55fbdc12dcd0 0x55fbdc127ac0 secure :-1 s=READY pgs=544 cs=0 l=1 rev1=1 crypto rx=0x7f74880a6910 tx=0x7f74880aa770 comp rx=0 tx=0).stop 2024-05-27T07:27:38.580 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:38.589+0000 7f74a8883b80 1 -- 172.21.15.52:0/873378897 shutdown_connections 2024-05-27T07:27:38.580 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:38.590+0000 7f74a8883b80 1 --2- 172.21.15.52:0/873378897 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55fbdc128000 0x55fbdc128410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:27:38.581 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:38.590+0000 7f74a8883b80 1 --2- 172.21.15.52:0/873378897 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55fbdbfa9ec0 0x55fbdc12d790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:27:38.581 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:38.590+0000 7f74a8883b80 1 --2- 172.21.15.52:0/873378897 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55fbdc12dcd0 0x55fbdc127ac0 unknown :-1 s=CLOSED pgs=544 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:27:38.581 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:38.590+0000 7f74a8883b80 1 -- 172.21.15.52:0/873378897 >> 172.21.15.52:0/873378897 conn(0x55fbdc004b70 msgr2=0x55fbdc1226c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:27:38.581 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:38.590+0000 7f74a8883b80 1 -- 172.21.15.52:0/873378897 shutdown_connections 2024-05-27T07:27:38.581 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:38.590+0000 7f74a8883b80 1 -- 172.21.15.52:0/873378897 wait complete. 2024-05-27T07:27:38.581 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:38.590+0000 7f74a8883b80 1 -- start start 2024-05-27T07:27:38.581 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:38.591+0000 7f74a8883b80 1 Processor -- start 2024-05-27T07:27:38.583 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:38.591+0000 7f74a8883b80 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55fbdbfa9ec0 0x55fbdc131410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:27:38.583 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:38.591+0000 7f74a8883b80 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55fbdc128000 0x55fbdc131950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:27:38.583 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:38.591+0000 7f74a8883b80 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55fbdc12dcd0 0x55fbdc131e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:27:38.583 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:38.592+0000 7f7493fff700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55fbdc12dcd0 0x55fbdc131e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:27:38.583 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:38.592+0000 7f7492ffd700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55fbdc128000 0x55fbdc131950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:27:38.583 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:38.592+0000 7f7493fff700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55fbdc12dcd0 0x55fbdc131e90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.52:53180/0 (socket says 172.21.15.52:53180) 2024-05-27T07:27:38.583 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:38.592+0000 7f7493fff700 1 -- 172.21.15.52:0/774321584 learned_addr learned my addr 172.21.15.52:0/774321584 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-05-27T07:27:38.583 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:38.592+0000 7f7492ffd700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55fbdc128000 0x55fbdc131950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.52:42336/0 (socket says 172.21.15.52:42336) 2024-05-27T07:27:38.583 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:38.592+0000 7f74937fe700 1 --2- 172.21.15.52:0/774321584 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55fbdbfa9ec0 0x55fbdc131410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:27:38.583 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:38.592+0000 7f74a8883b80 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x55fbdbf4d020 con 0x55fbdc12dcd0 2024-05-27T07:27:38.583 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:38.592+0000 7f74a8883b80 1 -- 172.21.15.52:0/774321584 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x55fbdbfe8840 con 0x55fbdc128000 2024-05-27T07:27:38.583 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:38.592+0000 7f74a8883b80 1 -- 172.21.15.52:0/774321584 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x55fbdc056d10 con 0x55fbdbfa9ec0 2024-05-27T07:27:38.584 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:38.592+0000 7f74a8883b80 10 client.0 _ll_register_callbacks cb 0x55fbdbfe19a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T07:27:38.584 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:38.592+0000 7f7492ffd700 1 -- 172.21.15.52:0/774321584 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55fbdbfa9ec0 msgr2=0x55fbdc131410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:27:38.584 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:38.592+0000 7f7492ffd700 1 --2- 172.21.15.52:0/774321584 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55fbdbfa9ec0 0x55fbdc131410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:27:38.584 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:38.592+0000 7f7492ffd700 1 -- 172.21.15.52:0/774321584 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55fbdc12dcd0 msgr2=0x55fbdc131e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:27:38.584 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:38.592+0000 7f7492ffd700 1 --2- 172.21.15.52:0/774321584 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55fbdc12dcd0 0x55fbdc131e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:27:38.584 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:38.592+0000 7f7492ffd700 1 -- 172.21.15.52:0/774321584 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55fbdbfe5710 con 0x55fbdc128000 2024-05-27T07:27:38.584 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:38.592+0000 7f7493fff700 1 --2- 172.21.15.52:0/774321584 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55fbdc12dcd0 0x55fbdc131e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-27T07:27:38.584 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:38.593+0000 7f7492ffd700 1 --2- 172.21.15.52:0/774321584 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55fbdc128000 0x55fbdc131950 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x55fbdc124120 tx=0x7f74880e0be0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=2b7774d1314f674c server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:27:38.584 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:38.593+0000 7f7472ffd700 10 client.0 ms_handle_connect on v2:172.21.15.105:3300/0 2024-05-27T07:27:38.584 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:38.593+0000 7f7472ffd700 1 -- 172.21.15.52:0/774321584 <== mon.2 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f74880b3660 con 0x55fbdc128000 2024-05-27T07:27:38.584 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:38.593+0000 7f74a8883b80 1 -- 172.21.15.52:0/774321584 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f74880aa890 con 0x55fbdc128000 2024-05-27T07:27:38.584 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:38.593+0000 7f7472ffd700 1 -- 172.21.15.52:0/774321584 <== mon.2 v2:172.21.15.105:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 9379+0+0 (secure 0 0 0) 0x7f74880b1040 con 0x55fbdc128000 2024-05-27T07:27:38.584 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:38.593+0000 7f7472ffd700 1 -- 172.21.15.52:0/774321584 <== mon.2 v2:172.21.15.105:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7f7488004e40 con 0x55fbdc128000 2024-05-27T07:27:38.585 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:38.593+0000 7f74a8883b80 10 client.25601 fetch_fsmap learned FSMap version 30 2024-05-27T07:27:38.585 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:38.593+0000 7f74a8883b80 1 -- 172.21.15.52:0/774321584 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x55fbdc13edd0 con 0x55fbdc128000 2024-05-27T07:27:38.586 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:38.593+0000 7f7472ffd700 1 -- 172.21.15.52:0/774321584 <== mon.2 v2:172.21.15.105:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f74880aaf90 con 0x55fbdc128000 2024-05-27T07:27:38.586 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:38.594+0000 7f7472ffd700 1 -- 172.21.15.52:0/774321584 <== mon.2 v2:172.21.15.105:3300/0 5 ==== fsmap(e 30) ==== 8362+0+0 (secure 0 0 0) 0x7f748800b040 con 0x55fbdc128000 2024-05-27T07:27:38.586 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:38.594+0000 7f74a8883b80 10 client.25601 fetch_fsmap finished waiting for FSMap version 30 2024-05-27T07:27:38.586 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:38.594+0000 7f74a8883b80 10 client.25601 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-05-27T07:27:38.586 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:38.594+0000 7f74a8883b80 20 client.25601 populate_metadata read hostname 'smithi052' 2024-05-27T07:27:38.587 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:38.594+0000 7f74a8883b80 1 --2- 172.21.15.52:0/774321584 >> [v2:172.21.15.105:6838/500348151,v1:172.21.15.105:6839/500348151] conn(0x55fbdc1412f0 0x55fbdc143710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:27:38.587 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:38.594+0000 7f74a8883b80 4 client.25601 mds_command: new command op to 24527 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-27T07:27:38.587 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:38.594+0000 7f74a8883b80 1 -- 172.21.15.52:0/774321584 --> [v2:172.21.15.105:6838/500348151,v1:172.21.15.105:6839/500348151] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x7f74880aa890 con 0x55fbdc1412f0 2024-05-27T07:27:38.587 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:38.594+0000 7f74937fe700 1 --2- 172.21.15.52:0/774321584 >> [v2:172.21.15.105:6838/500348151,v1:172.21.15.105:6839/500348151] conn(0x55fbdc1412f0 0x55fbdc143710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:27:38.587 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:38.595+0000 7f74937fe700 1 --2- 172.21.15.52:0/774321584 >> [v2:172.21.15.105:6838/500348151,v1:172.21.15.105:6839/500348151] conn(0x55fbdc1412f0 0x55fbdc143710 crc :-1 s=READY pgs=69 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=c27b1840b5015c1f server_cookie=3feacdcf21317158 in_seq=0 out_seq=0 2024-05-27T07:27:38.587 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:38.595+0000 7f7472ffd700 10 client.25601 ms_handle_connect on v2:172.21.15.105:6838/500348151 2024-05-27T07:27:38.587 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:38.595+0000 7f7472ffd700 1 -- 172.21.15.52:0/774321584 <== mds.0 v2:172.21.15.105:6838/500348151 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24041 (crc 0 0 0) 0x7f74880aa890 con 0x55fbdc1412f0 2024-05-27T07:27:38.587 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:38.595+0000 7f7472ffd700 10 client.25601 handle_command_reply: tid=0 2024-05-27T07:27:38.587 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:38.595+0000 7f74a8883b80 1 client.25601 shutdown 2024-05-27T07:27:38.587 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:38.596+0000 7f74a8883b80 1 -- 172.21.15.52:0/774321584 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55fbdc128000 msgr2=0x55fbdc131950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:27:38.587 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:38.596+0000 7f74a8883b80 1 --2- 172.21.15.52:0/774321584 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55fbdc128000 0x55fbdc131950 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x55fbdc124120 tx=0x7f74880e0be0 comp rx=0 tx=0).stop 2024-05-27T07:27:38.587 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:38.596+0000 7f74a8883b80 1 -- 172.21.15.52:0/774321584 shutdown_connections 2024-05-27T07:27:38.588 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:38.596+0000 7f74a8883b80 1 --2- 172.21.15.52:0/774321584 >> [v2:172.21.15.105:6838/500348151,v1:172.21.15.105:6839/500348151] conn(0x55fbdc1412f0 0x55fbdc143710 crc :-1 s=READY pgs=69 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:27:38.588 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:38.596+0000 7f7472ffd700 0 client.25601 ms_handle_reset on v2:172.21.15.105:6838/500348151 2024-05-27T07:27:38.588 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:38.596+0000 7f74a8883b80 1 --2- 172.21.15.52:0/774321584 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55fbdc12dcd0 0x55fbdc131e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:27:38.588 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:38.596+0000 7f74a8883b80 1 --2- 172.21.15.52:0/774321584 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55fbdbfa9ec0 0x55fbdc131410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:27:38.588 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:38.596+0000 7f74a8883b80 1 --2- 172.21.15.52:0/774321584 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55fbdc128000 0x55fbdc131950 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:27:38.588 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:38.596+0000 7f74a8883b80 1 -- 172.21.15.52:0/774321584 >> 172.21.15.52:0/774321584 conn(0x55fbdc004b70 msgr2=0x55fbdc126ab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:27:38.588 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:38.596+0000 7f74a8883b80 1 -- 172.21.15.52:0/774321584 shutdown_connections 2024-05-27T07:27:38.588 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:38.596+0000 7f74a8883b80 1 -- 172.21.15.52:0/774321584 wait complete. 2024-05-27T07:27:38.588 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:38.596+0000 7f74a8883b80 20 client.25601 trim_cache size 0 max 16384 2024-05-27T07:27:38.601 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:27:38 smithi173 ceph-mon[37000]: pgmap v53: 129 pgs: 129 active+clean; 1.4 MiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 1.2 KiB/s wr, 10 op/s 2024-05-27T07:27:38.616 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:38.627+0000 7f74a8883b80 1 Processor -- start 2024-05-27T07:27:38.617 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:38.627+0000 7f74a8883b80 1 -- start start 2024-05-27T07:27:38.617 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:38.627+0000 7f74a8883b80 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55fbdbfa9ec0 0x55fbdc12cef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:27:38.618 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:38.627+0000 7f74a8883b80 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55fbdc128000 0x55fbdc13d460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:27:38.618 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:38.627+0000 7f74a8883b80 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55fbdc12dcd0 0x55fbdc132fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:27:38.618 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:38.627+0000 7f74a8883b80 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x7f74941adf60 con 0x55fbdc128000 2024-05-27T07:27:38.618 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:38.627+0000 7f74a8883b80 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f74941afaa0 con 0x55fbdc12dcd0 2024-05-27T07:27:38.618 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:38.627+0000 7f74a8883b80 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f74941ae630 con 0x55fbdbfa9ec0 2024-05-27T07:27:38.618 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:38.627+0000 7f7492ffd700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55fbdc128000 0x55fbdc13d460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:27:38.618 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:38.627+0000 7f7492ffd700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55fbdc128000 0x55fbdc13d460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.52:53190/0 (socket says 172.21.15.52:53190) 2024-05-27T07:27:38.618 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:38.627+0000 7f7492ffd700 1 -- 172.21.15.52:0/3969886720 learned_addr learned my addr 172.21.15.52:0/3969886720 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-05-27T07:27:38.618 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:38.627+0000 7f7493fff700 1 --2- 172.21.15.52:0/3969886720 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55fbdc12dcd0 0x55fbdc132fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:27:38.619 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:38.628+0000 7f74937fe700 1 --2- 172.21.15.52:0/3969886720 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55fbdbfa9ec0 0x55fbdc12cef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:27:38.619 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:38.628+0000 7f7492ffd700 1 -- 172.21.15.52:0/3969886720 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55fbdbfa9ec0 msgr2=0x55fbdc12cef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:27:38.619 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:38.628+0000 7f7492ffd700 1 --2- 172.21.15.52:0/3969886720 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55fbdbfa9ec0 0x55fbdc12cef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:27:38.619 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:38.628+0000 7f7492ffd700 1 -- 172.21.15.52:0/3969886720 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55fbdc12dcd0 msgr2=0x55fbdc132fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:27:38.619 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:38.628+0000 7f7492ffd700 1 --2- 172.21.15.52:0/3969886720 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55fbdc12dcd0 0x55fbdc132fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:27:38.619 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:38.628+0000 7f7492ffd700 1 -- 172.21.15.52:0/3969886720 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55fbdc123830 con 0x55fbdc128000 2024-05-27T07:27:38.619 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:38.628+0000 7f74937fe700 1 --2- 172.21.15.52:0/3969886720 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55fbdbfa9ec0 0x55fbdc12cef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-27T07:27:38.619 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:38.628+0000 7f7492ffd700 1 --2- 172.21.15.52:0/3969886720 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55fbdc128000 0x55fbdc13d460 secure :-1 s=READY pgs=545 cs=0 l=1 rev1=1 crypto rx=0x7f74880a6910 tx=0x7f7488003ba0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ed3e8d7b967eddb0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:27:38.619 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:38.628+0000 7f74797fa700 1 -- 172.21.15.52:0/3969886720 <== mon.0 v2:172.21.15.52:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f74880aaf90 con 0x55fbdc128000 2024-05-27T07:27:38.619 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:38.628+0000 7f74797fa700 1 -- 172.21.15.52:0/3969886720 <== mon.0 v2:172.21.15.52:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f74880b31d0 con 0x55fbdc128000 2024-05-27T07:27:38.619 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:38.628+0000 7f74797fa700 1 -- 172.21.15.52:0/3969886720 <== mon.0 v2:172.21.15.52:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f748800c0b0 con 0x55fbdc128000 2024-05-27T07:27:38.621 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:38.629+0000 7f74a8883b80 1 -- 172.21.15.52:0/3969886720 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55fbdc128000 msgr2=0x55fbdc13d460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:27:38.621 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:38.629+0000 7f74a8883b80 1 --2- 172.21.15.52:0/3969886720 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55fbdc128000 0x55fbdc13d460 secure :-1 s=READY pgs=545 cs=0 l=1 rev1=1 crypto rx=0x7f74880a6910 tx=0x7f7488003ba0 comp rx=0 tx=0).stop 2024-05-27T07:27:38.621 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:38.629+0000 7f74a8883b80 1 -- 172.21.15.52:0/3969886720 shutdown_connections 2024-05-27T07:27:38.621 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:38.629+0000 7f74a8883b80 1 --2- 172.21.15.52:0/3969886720 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55fbdc12dcd0 0x55fbdc132fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:27:38.621 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:38.629+0000 7f74a8883b80 1 --2- 172.21.15.52:0/3969886720 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55fbdbfa9ec0 0x55fbdc12cef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:27:38.621 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:38.629+0000 7f74a8883b80 1 --2- 172.21.15.52:0/3969886720 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55fbdc128000 0x55fbdc13d460 unknown :-1 s=CLOSED pgs=545 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:27:38.621 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:38.629+0000 7f74a8883b80 1 -- 172.21.15.52:0/3969886720 >> 172.21.15.52:0/3969886720 conn(0x55fbdc004b70 msgr2=0x55fbdc1328a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:27:38.621 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:38.630+0000 7f74a8883b80 1 -- 172.21.15.52:0/3969886720 shutdown_connections 2024-05-27T07:27:38.622 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:38.630+0000 7f74a8883b80 1 -- 172.21.15.52:0/3969886720 wait complete. 2024-05-27T07:27:38.622 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:38.630+0000 7f74a8883b80 1 -- start start 2024-05-27T07:27:38.622 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:38.630+0000 7f74a8883b80 1 Processor -- start 2024-05-27T07:27:38.622 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:38.631+0000 7f74a8883b80 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55fbdbfa9ec0 0x55fbdc1395f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:27:38.622 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:38.631+0000 7f74937fe700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55fbdbfa9ec0 0x55fbdc1395f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:27:38.622 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:38.631+0000 7f74937fe700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55fbdbfa9ec0 0x55fbdc1395f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.52:53206/0 (socket says 172.21.15.52:53206) 2024-05-27T07:27:38.622 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:38.631+0000 7f74937fe700 1 -- 172.21.15.52:0/211615374 learned_addr learned my addr 172.21.15.52:0/211615374 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-05-27T07:27:38.622 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:38.631+0000 7f74a8883b80 1 --2- 172.21.15.52:0/211615374 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55fbdc128000 0x55fbdc139b30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:27:38.622 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:38.631+0000 7f74a8883b80 1 --2- 172.21.15.52:0/211615374 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55fbdc12dcd0 0x55fbdc137b60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:27:38.622 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:38.631+0000 7f74a8883b80 1 -- 172.21.15.52:0/211615374 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x7f74941af3d0 con 0x55fbdbfa9ec0 2024-05-27T07:27:38.622 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:38.631+0000 7f74a8883b80 1 -- 172.21.15.52:0/211615374 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f74941aed00 con 0x55fbdc128000 2024-05-27T07:27:38.622 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:38.631+0000 7f74a8883b80 1 -- 172.21.15.52:0/211615374 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f74941abf20 con 0x55fbdc12dcd0 2024-05-27T07:27:38.623 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:38.632+0000 7f7492ffd700 1 --2- 172.21.15.52:0/211615374 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55fbdc128000 0x55fbdc139b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:27:38.623 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:38.632+0000 7f7493fff700 1 --2- 172.21.15.52:0/211615374 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55fbdc12dcd0 0x55fbdc137b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:27:38.624 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:38.632+0000 7f74937fe700 1 -- 172.21.15.52:0/211615374 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55fbdc12dcd0 msgr2=0x55fbdc137b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:27:38.624 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:38.632+0000 7f74937fe700 1 --2- 172.21.15.52:0/211615374 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55fbdc12dcd0 0x55fbdc137b60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:27:38.624 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:38.632+0000 7f74937fe700 1 -- 172.21.15.52:0/211615374 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55fbdc128000 msgr2=0x55fbdc139b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:27:38.624 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:38.632+0000 7f74937fe700 1 --2- 172.21.15.52:0/211615374 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55fbdc128000 0x55fbdc139b30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:27:38.624 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:38.632+0000 7f74937fe700 1 -- 172.21.15.52:0/211615374 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55fbdc1381d0 con 0x55fbdbfa9ec0 2024-05-27T07:27:38.624 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:38.632+0000 7f7493fff700 1 --2- 172.21.15.52:0/211615374 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55fbdc12dcd0 0x55fbdc137b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-27T07:27:38.624 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:38.632+0000 7f74a8883b80 10 client.0 _ll_register_callbacks cb 0x55fbdbfe19a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T07:27:38.624 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:38.632+0000 7f74a8883b80 1 -- 172.21.15.52:0/211615374 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f74880b31d0 con 0x55fbdbfa9ec0 2024-05-27T07:27:38.624 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:38.632+0000 7f74937fe700 1 --2- 172.21.15.52:0/211615374 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55fbdbfa9ec0 0x55fbdc1395f0 secure :-1 s=READY pgs=546 cs=0 l=1 rev1=1 crypto rx=0x55fbdc128c30 tx=0x7f74840047c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1d2eba2bd8c6b44e server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:27:38.624 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:38.632+0000 7f7472ffd700 10 client.15918 ms_handle_connect on v2:172.21.15.52:3300/0 2024-05-27T07:27:38.624 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:38.632+0000 7f7472ffd700 1 -- 172.21.15.52:0/211615374 <== mon.0 v2:172.21.15.52:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7484034020 con 0x55fbdbfa9ec0 2024-05-27T07:27:38.625 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:38.632+0000 7f7472ffd700 1 -- 172.21.15.52:0/211615374 <== mon.0 v2:172.21.15.52:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 9379+0+0 (secure 0 0 0) 0x7f7484003a90 con 0x55fbdbfa9ec0 2024-05-27T07:27:38.625 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:38.632+0000 7f7472ffd700 1 -- 172.21.15.52:0/211615374 <== mon.0 v2:172.21.15.52:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7f7484003cb0 con 0x55fbdbfa9ec0 2024-05-27T07:27:38.625 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:38.632+0000 7f74a8883b80 10 client.15918 fetch_fsmap learned FSMap version 30 2024-05-27T07:27:38.625 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:38.633+0000 7f74a8883b80 1 -- 172.21.15.52:0/211615374 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({fsmap=30}) -- 0x55fbdc1ac600 con 0x55fbdbfa9ec0 2024-05-27T07:27:38.625 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:38.633+0000 7f7472ffd700 1 -- 172.21.15.52:0/211615374 <== mon.0 v2:172.21.15.52:3300/0 4 ==== fsmap(e 30) ==== 8362+0+0 (secure 0 0 0) 0x7f7484041020 con 0x55fbdbfa9ec0 2024-05-27T07:27:38.625 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:38.633+0000 7f74a8883b80 10 client.15918 fetch_fsmap finished waiting for FSMap version 30 2024-05-27T07:27:38.625 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:38.633+0000 7f74a8883b80 10 client.15918 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-05-27T07:27:38.625 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:38.633+0000 7f74a8883b80 20 client.15918 populate_metadata read hostname 'smithi052' 2024-05-27T07:27:38.625 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:38.633+0000 7f74a8883b80 1 --2- 172.21.15.52:0/211615374 >> [v2:172.21.15.105:6838/500348151,v1:172.21.15.105:6839/500348151] conn(0x55fbdc1412f0 0x55fbdc1aecd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:27:38.625 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:38.633+0000 7f74a8883b80 4 client.15918 mds_command: new command op to 24527 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-05-27T07:27:38.625 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:38.633+0000 7f74a8883b80 1 -- 172.21.15.52:0/211615374 --> [v2:172.21.15.105:6838/500348151,v1:172.21.15.105:6839/500348151] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) -- 0x55fbdc0f3a70 con 0x55fbdc1412f0 2024-05-27T07:27:38.625 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:38.633+0000 7f7492ffd700 1 --2- 172.21.15.52:0/211615374 >> [v2:172.21.15.105:6838/500348151,v1:172.21.15.105:6839/500348151] conn(0x55fbdc1412f0 0x55fbdc1aecd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:27:38.625 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:38.634+0000 7f7472ffd700 10 client.15918 ms_handle_connect on v2:172.21.15.105:6838/500348151 2024-05-27T07:27:38.626 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:38.634+0000 7f7492ffd700 1 --2- 172.21.15.52:0/211615374 >> [v2:172.21.15.105:6838/500348151,v1:172.21.15.105:6839/500348151] conn(0x55fbdc1412f0 0x55fbdc1aecd0 crc :-1 s=READY pgs=70 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=916abb7626c9f2d0 server_cookie=b752a037c5e1c16c in_seq=0 out_seq=0 2024-05-27T07:27:38.626 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:38.635+0000 7f7472ffd700 1 -- 172.21.15.52:0/211615374 <== mds.0 v2:172.21.15.105:6838/500348151 1 ==== command_reply(tid 0: 0 ) ==== 8+0+110 (crc 0 0 0) 0x55fbdc0f3a70 con 0x55fbdc1412f0 2024-05-27T07:27:38.626 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:38.635+0000 7f7472ffd700 10 client.15918 handle_command_reply: tid=0 2024-05-27T07:27:38.627 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:38.635+0000 7f74a8883b80 1 client.15918 shutdown 2024-05-27T07:27:38.627 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:38.636+0000 7f74a8883b80 1 -- 172.21.15.52:0/211615374 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55fbdbfa9ec0 msgr2=0x55fbdc1395f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:27:38.627 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:38.636+0000 7f74a8883b80 1 --2- 172.21.15.52:0/211615374 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55fbdbfa9ec0 0x55fbdc1395f0 secure :-1 s=READY pgs=546 cs=0 l=1 rev1=1 crypto rx=0x55fbdc128c30 tx=0x7f74840047c0 comp rx=0 tx=0).stop 2024-05-27T07:27:38.627 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:38.636+0000 7f74a8883b80 1 -- 172.21.15.52:0/211615374 shutdown_connections 2024-05-27T07:27:38.627 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:38.636+0000 7f74a8883b80 1 --2- 172.21.15.52:0/211615374 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55fbdc12dcd0 0x55fbdc137b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:27:38.627 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:38.636+0000 7f74a8883b80 1 --2- 172.21.15.52:0/211615374 >> [v2:172.21.15.105:6838/500348151,v1:172.21.15.105:6839/500348151] conn(0x55fbdc1412f0 0x55fbdc1aecd0 crc :-1 s=READY pgs=70 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:27:38.627 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:38.636+0000 7f74a8883b80 1 --2- 172.21.15.52:0/211615374 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55fbdbfa9ec0 0x55fbdc1395f0 unknown :-1 s=CLOSED pgs=546 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:27:38.627 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:38.636+0000 7f7472ffd700 0 client.15918 ms_handle_reset on v2:172.21.15.105:6838/500348151 2024-05-27T07:27:38.627 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:38.636+0000 7f74a8883b80 1 --2- 172.21.15.52:0/211615374 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55fbdc128000 0x55fbdc139b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:27:38.627 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:38.636+0000 7f74a8883b80 1 -- 172.21.15.52:0/211615374 >> 172.21.15.52:0/211615374 conn(0x55fbdc004b70 msgr2=0x55fbdc12cc50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:27:38.627 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:38.636+0000 7f74a8883b80 1 -- 172.21.15.52:0/211615374 shutdown_connections 2024-05-27T07:27:38.627 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:38.636+0000 7f74a8883b80 1 -- 172.21.15.52:0/211615374 wait complete. 2024-05-27T07:27:38.627 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:38.636+0000 7f74a8883b80 20 client.15918 trim_cache size 0 max 16384 2024-05-27T07:27:38.628 INFO:teuthology.orchestra.run.smithi052.stdout:{ 2024-05-27T07:27:38.628 INFO:teuthology.orchestra.run.smithi052.stdout: "return_code": 0, 2024-05-27T07:27:38.628 INFO:teuthology.orchestra.run.smithi052.stdout: "scrub_tag": "f161a4de-d3f0-4469-bf1b-0f1834e40c39", 2024-05-27T07:27:38.628 INFO:teuthology.orchestra.run.smithi052.stdout: "mode": "asynchronous" 2024-05-27T07:27:38.628 INFO:teuthology.orchestra.run.smithi052.stdout:} 2024-05-27T07:27:38.629 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:38.637+0000 7f7499e6e700 1 -- 172.21.15.52:0/373209529 >> [v2:172.21.15.52:6800/1822318384,v1:172.21.15.52:6801/1822318384] conn(0x7f747c0b28d0 msgr2=0x7f747c0b4de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:27:38.629 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:38.637+0000 7f7499e6e700 1 --2- 172.21.15.52:0/373209529 >> [v2:172.21.15.52:6800/1822318384,v1:172.21.15.52:6801/1822318384] conn(0x7f747c0b28d0 0x7f747c0b4de0 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f7494103af0 tx=0x7f74840073d0 comp rx=0 tx=0).stop 2024-05-27T07:27:38.629 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:38.637+0000 7f7499e6e700 1 -- 172.21.15.52:0/373209529 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f7494103300 msgr2=0x7f749419b4d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:27:38.629 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:38.637+0000 7f7499e6e700 1 --2- 172.21.15.52:0/373209529 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f7494103300 0x7f749419b4d0 secure :-1 s=READY pgs=543 cs=0 l=1 rev1=1 crypto rx=0x7f7488003930 tx=0x7f7488003960 comp rx=0 tx=0).stop 2024-05-27T07:27:38.629 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:38.637+0000 7f7499e6e700 1 -- 172.21.15.52:0/373209529 shutdown_connections 2024-05-27T07:27:38.629 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:38.637+0000 7f7499e6e700 1 --2- 172.21.15.52:0/373209529 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f74941089e0 0x7f749419f760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:27:38.629 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:38.637+0000 7f7499e6e700 1 --2- 172.21.15.52:0/373209529 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f74941029b0 0x7f749419af30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:27:38.629 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:38.637+0000 7f7499e6e700 1 --2- 172.21.15.52:0/373209529 >> [v2:172.21.15.52:6800/1822318384,v1:172.21.15.52:6801/1822318384] conn(0x7f747c0b28d0 0x7f747c0b4de0 unknown :-1 s=CLOSED pgs=78 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:27:38.629 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:38.637+0000 7f7499e6e700 1 --2- 172.21.15.52:0/373209529 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f7494103300 0x7f749419b4d0 unknown :-1 s=CLOSED pgs=543 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:27:38.629 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:38.637+0000 7f7499e6e700 1 -- 172.21.15.52:0/373209529 >> 172.21.15.52:0/373209529 conn(0x7f74940fe6e0 msgr2=0x7f7494107c70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:27:38.629 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:38.638+0000 7f7499e6e700 1 -- 172.21.15.52:0/373209529 shutdown_connections 2024-05-27T07:27:38.629 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:38.638+0000 7f7499e6e700 1 -- 172.21.15.52:0/373209529 wait complete. 2024-05-27T07:27:39.437 DEBUG:teuthology.orchestra.run.smithi052:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:38f9d900c83c63799fbdbe61acc9a11b0d3554a6 shell --fsid 73df2636-1bf8-11ef-bc9b-c7b262605968 -- ceph tell mds.1:0 scrub status 2024-05-27T07:27:39.725 INFO:teuthology.orchestra.run.smithi052.stderr:Inferring config /var/lib/ceph/73df2636-1bf8-11ef-bc9b-c7b262605968/mon.a/config 2024-05-27T07:27:40.439 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:27:40 smithi105 ceph-mon[35977]: scrub queued for path: / 2024-05-27T07:27:40.440 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:27:40 smithi105 ceph-mon[35977]: scrub summary: idle+waiting paths [/] 2024-05-27T07:27:40.440 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:27:40 smithi105 ceph-mon[35977]: scrub summary: active paths [/] 2024-05-27T07:27:40.440 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:27:40 smithi105 ceph-mon[35977]: scrub summary: idle+waiting paths [/] 2024-05-27T07:27:40.440 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:27:40 smithi105 ceph-mon[35977]: pgmap v54: 129 pgs: 129 active+clean; 1.4 MiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 6.7 KiB/s rd, 1.2 KiB/s wr, 15 op/s 2024-05-27T07:27:40.479 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:27:40 smithi052 ceph-mon[31955]: scrub queued for path: / 2024-05-27T07:27:40.479 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:27:40 smithi052 ceph-mon[31955]: scrub summary: idle+waiting paths [/] 2024-05-27T07:27:40.479 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:27:40 smithi052 ceph-mon[31955]: scrub summary: active paths [/] 2024-05-27T07:27:40.479 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:27:40 smithi052 ceph-mon[31955]: scrub summary: idle+waiting paths [/] 2024-05-27T07:27:40.479 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:27:40 smithi052 ceph-mon[31955]: pgmap v54: 129 pgs: 129 active+clean; 1.4 MiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 6.7 KiB/s rd, 1.2 KiB/s wr, 15 op/s 2024-05-27T07:27:40.602 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:27:40 smithi173 ceph-mon[37000]: scrub queued for path: / 2024-05-27T07:27:40.602 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:27:40 smithi173 ceph-mon[37000]: scrub summary: idle+waiting paths [/] 2024-05-27T07:27:40.602 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:27:40 smithi173 ceph-mon[37000]: scrub summary: active paths [/] 2024-05-27T07:27:40.602 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:27:40 smithi173 ceph-mon[37000]: scrub summary: idle+waiting paths [/] 2024-05-27T07:27:40.602 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:27:40 smithi173 ceph-mon[37000]: pgmap v54: 129 pgs: 129 active+clean; 1.4 MiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 6.7 KiB/s rd, 1.2 KiB/s wr, 15 op/s 2024-05-27T07:27:41.458 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:41.468+0000 7f9ae00c8700 1 -- 172.21.15.52:0/820933687 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f9ad8102b40 msgr2=0x7f9ad8102f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:27:41.458 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:41.468+0000 7f9ae00c8700 1 --2- 172.21.15.52:0/820933687 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f9ad8102b40 0x7f9ad8102f50 secure :-1 s=READY pgs=547 cs=0 l=1 rev1=1 crypto rx=0x7f9ad40099c0 tx=0x7f9ad4009cd0 comp rx=0 tx=0).stop 2024-05-27T07:27:41.458 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:41.468+0000 7f9ae00c8700 1 -- 172.21.15.52:0/820933687 shutdown_connections 2024-05-27T07:27:41.458 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:41.468+0000 7f9ae00c8700 1 --2- 172.21.15.52:0/820933687 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f9ad8109840 0x7f9ad81111a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:27:41.458 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:41.468+0000 7f9ae00c8700 1 --2- 172.21.15.52:0/820933687 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f9ad8102b40 0x7f9ad8102f50 unknown :-1 s=CLOSED pgs=547 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:27:41.458 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:41.468+0000 7f9ae00c8700 1 --2- 172.21.15.52:0/820933687 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f9ad8108ef0 0x7f9ad8109300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:27:41.458 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:41.468+0000 7f9ae00c8700 1 -- 172.21.15.52:0/820933687 >> 172.21.15.52:0/820933687 conn(0x7f9ad80fe6e0 msgr2=0x7f9ad8100b00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:27:41.458 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:41.468+0000 7f9ae00c8700 1 -- 172.21.15.52:0/820933687 shutdown_connections 2024-05-27T07:27:41.458 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:41.468+0000 7f9ae00c8700 1 -- 172.21.15.52:0/820933687 wait complete. 2024-05-27T07:27:41.459 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:41.469+0000 7f9ae00c8700 1 Processor -- start 2024-05-27T07:27:41.459 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:41.469+0000 7f9ae00c8700 1 -- start start 2024-05-27T07:27:41.460 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:41.469+0000 7f9ae00c8700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f9ad8102b40 0x7f9ad819e850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:27:41.460 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:41.469+0000 7f9ae00c8700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f9ad8108ef0 0x7f9ad819ed90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:27:41.460 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:41.470+0000 7f9add663700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f9ad8108ef0 0x7f9ad819ed90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:27:41.460 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:41.470+0000 7f9add663700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f9ad8108ef0 0x7f9ad819ed90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.52:38472/0 (socket says 172.21.15.52:38472) 2024-05-27T07:27:41.460 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:41.470+0000 7f9add663700 1 -- 172.21.15.52:0/667408378 learned_addr learned my addr 172.21.15.52:0/667408378 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-05-27T07:27:41.460 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:41.470+0000 7f9ae00c8700 1 --2- 172.21.15.52:0/667408378 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f9ad8109840 0x7f9ad8198a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:27:41.460 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:41.470+0000 7f9ae00c8700 1 -- 172.21.15.52:0/667408378 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x7f9ad81122b0 con 0x7f9ad8108ef0 2024-05-27T07:27:41.460 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:41.470+0000 7f9adde64700 1 --2- 172.21.15.52:0/667408378 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f9ad8102b40 0x7f9ad819e850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:27:41.460 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:41.470+0000 7f9ae00c8700 1 -- 172.21.15.52:0/667408378 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f9ad8112130 con 0x7f9ad8109840 2024-05-27T07:27:41.460 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:41.470+0000 7f9ae00c8700 1 -- 172.21.15.52:0/667408378 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f9ad8112430 con 0x7f9ad8102b40 2024-05-27T07:27:41.461 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:41.470+0000 7f9ade665700 1 --2- 172.21.15.52:0/667408378 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f9ad8109840 0x7f9ad8198a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:27:41.461 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:41.470+0000 7f9adde64700 1 -- 172.21.15.52:0/667408378 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f9ad8109840 msgr2=0x7f9ad8198a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:27:41.461 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:41.470+0000 7f9adde64700 1 --2- 172.21.15.52:0/667408378 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f9ad8109840 0x7f9ad8198a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:27:41.461 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:41.470+0000 7f9adde64700 1 -- 172.21.15.52:0/667408378 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f9ad8108ef0 msgr2=0x7f9ad819ed90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:27:41.461 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:41.470+0000 7f9adde64700 1 --2- 172.21.15.52:0/667408378 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f9ad8108ef0 0x7f9ad819ed90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:27:41.462 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:41.470+0000 7f9adde64700 1 -- 172.21.15.52:0/667408378 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9ad8199120 con 0x7f9ad8102b40 2024-05-27T07:27:41.462 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:41.470+0000 7f9add663700 1 --2- 172.21.15.52:0/667408378 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f9ad8108ef0 0x7f9ad819ed90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-27T07:27:41.462 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:41.470+0000 7f9adde64700 1 --2- 172.21.15.52:0/667408378 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f9ad8102b40 0x7f9ad819e850 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f9ad40098e0 tx=0x7f9ad4003680 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:27:41.462 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:41.471+0000 7f9aceffd700 1 -- 172.21.15.52:0/667408378 <== mon.1 v2:172.21.15.173:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9ad40238d0 con 0x7f9ad8102b40 2024-05-27T07:27:41.462 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:41.471+0000 7f9aceffd700 1 -- 172.21.15.52:0/667408378 <== mon.1 v2:172.21.15.173:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f9ad4027070 con 0x7f9ad8102b40 2024-05-27T07:27:41.462 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:41.471+0000 7f9ae00c8700 1 -- 172.21.15.52:0/667408378 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9ad81993b0 con 0x7f9ad8102b40 2024-05-27T07:27:41.462 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:41.471+0000 7f9aceffd700 1 -- 172.21.15.52:0/667408378 <== mon.1 v2:172.21.15.173:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9ad4003e60 con 0x7f9ad8102b40 2024-05-27T07:27:41.462 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:41.471+0000 7f9ae00c8700 1 -- 172.21.15.52:0/667408378 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9ad81997d0 con 0x7f9ad8102b40 2024-05-27T07:27:41.463 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:41.474+0000 7f9aceffd700 1 -- 172.21.15.52:0/667408378 <== mon.1 v2:172.21.15.173:3300/0 4 ==== mgrmap(e 26) ==== 149783+0+0 (secure 0 0 0) 0x7f9ad4003850 con 0x7f9ad8102b40 2024-05-27T07:27:41.464 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:41.474+0000 7f9aceffd700 1 --2- 172.21.15.52:0/667408378 >> [v2:172.21.15.52:6800/1822318384,v1:172.21.15.52:6801/1822318384] conn(0x7f9ac40b29f0 0x7f9ac40b4f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:27:41.464 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:41.475+0000 7f9aceffd700 1 -- 172.21.15.52:0/667408378 <== mon.1 v2:172.21.15.173:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9379+0+0 (secure 0 0 0) 0x7f9ad40ebaa0 con 0x7f9ad8102b40 2024-05-27T07:27:41.464 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:41.475+0000 7f9add663700 1 --2- 172.21.15.52:0/667408378 >> [v2:172.21.15.52:6800/1822318384,v1:172.21.15.52:6801/1822318384] conn(0x7f9ac40b29f0 0x7f9ac40b4f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:27:41.465 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:41.475+0000 7f9add663700 1 --2- 172.21.15.52:0/667408378 >> [v2:172.21.15.52:6800/1822318384,v1:172.21.15.52:6801/1822318384] conn(0x7f9ac40b29f0 0x7f9ac40b4f00 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f9ad81077c0 tx=0x7f9ac8009260 comp rx=0 tx=0).ready entity=mgr.15333 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:27:41.466 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:41.476+0000 7f9aeeaddb80 1 Processor -- start 2024-05-27T07:27:41.466 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:41.477+0000 7f9aeeaddb80 1 -- start start 2024-05-27T07:27:41.467 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:41.477+0000 7f9aeeaddb80 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x5625a529ae40 0x5625a541e710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:27:41.467 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:41.477+0000 7f9aeeaddb80 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5625a541ec50 0x5625a5418a40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:27:41.467 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:41.477+0000 7f9aeeaddb80 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5625a5418f80 0x5625a5419390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:27:41.467 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:41.477+0000 7f9aeeaddb80 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x5625a5347c90 con 0x5625a529ae40 2024-05-27T07:27:41.469 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:41.477+0000 7f9aeeaddb80 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x5625a5337830 con 0x5625a5418f80 2024-05-27T07:27:41.469 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:41.477+0000 7f9aeeaddb80 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x5625a533b870 con 0x5625a541ec50 2024-05-27T07:27:41.469 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:41.477+0000 7f9adde64700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x5625a529ae40 0x5625a541e710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:27:41.469 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:41.477+0000 7f9adde64700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x5625a529ae40 0x5625a541e710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.52:38478/0 (socket says 172.21.15.52:38478) 2024-05-27T07:27:41.469 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:41.478+0000 7f9adde64700 1 -- 172.21.15.52:0/126405977 learned_addr learned my addr 172.21.15.52:0/126405977 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-05-27T07:27:41.469 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:41.478+0000 7f9add663700 1 --2- 172.21.15.52:0/126405977 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5625a541ec50 0x5625a5418a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:27:41.470 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:41.478+0000 7f9adde64700 1 -- 172.21.15.52:0/126405977 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5625a541ec50 msgr2=0x5625a5418a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:27:41.470 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:41.478+0000 7f9adde64700 1 --2- 172.21.15.52:0/126405977 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5625a541ec50 0x5625a5418a40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:27:41.470 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:41.478+0000 7f9adde64700 1 -- 172.21.15.52:0/126405977 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5625a5418f80 msgr2=0x5625a5419390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:27:41.470 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:41.478+0000 7f9adde64700 1 --2- 172.21.15.52:0/126405977 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5625a5418f80 0x5625a5419390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:27:41.470 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:41.478+0000 7f9adde64700 1 -- 172.21.15.52:0/126405977 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5625a52878d0 con 0x5625a529ae40 2024-05-27T07:27:41.470 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:41.478+0000 7f9adde64700 1 --2- 172.21.15.52:0/126405977 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x5625a529ae40 0x5625a541e710 secure :-1 s=READY pgs=548 cs=0 l=1 rev1=1 crypto rx=0x7f9ad4009990 tx=0x7f9ad40b2330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2a5764aaf5c74929 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:27:41.470 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:41.478+0000 7f9ac3fff700 1 -- 172.21.15.52:0/126405977 <== mon.0 v2:172.21.15.52:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9ad40b2bc0 con 0x5625a529ae40 2024-05-27T07:27:41.470 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:41.478+0000 7f9ac3fff700 1 -- 172.21.15.52:0/126405977 <== mon.0 v2:172.21.15.52:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f9ad40b6f70 con 0x5625a529ae40 2024-05-27T07:27:41.470 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:41.478+0000 7f9ac3fff700 1 -- 172.21.15.52:0/126405977 <== mon.0 v2:172.21.15.52:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9ad40b1e20 con 0x5625a529ae40 2024-05-27T07:27:41.470 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:41.479+0000 7f9aeeaddb80 1 -- 172.21.15.52:0/126405977 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x5625a529ae40 msgr2=0x5625a541e710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:27:41.470 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:41.479+0000 7f9aeeaddb80 1 --2- 172.21.15.52:0/126405977 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x5625a529ae40 0x5625a541e710 secure :-1 s=READY pgs=548 cs=0 l=1 rev1=1 crypto rx=0x7f9ad4009990 tx=0x7f9ad40b2330 comp rx=0 tx=0).stop 2024-05-27T07:27:41.470 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:41.479+0000 7f9aeeaddb80 1 -- 172.21.15.52:0/126405977 shutdown_connections 2024-05-27T07:27:41.471 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:41.479+0000 7f9aeeaddb80 1 --2- 172.21.15.52:0/126405977 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5625a5418f80 0x5625a5419390 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:27:41.471 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:41.479+0000 7f9aeeaddb80 1 --2- 172.21.15.52:0/126405977 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x5625a529ae40 0x5625a541e710 unknown :-1 s=CLOSED pgs=548 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:27:41.471 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:41.479+0000 7f9aeeaddb80 1 --2- 172.21.15.52:0/126405977 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5625a541ec50 0x5625a5418a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:27:41.471 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:41.479+0000 7f9aeeaddb80 1 -- 172.21.15.52:0/126405977 >> 172.21.15.52:0/126405977 conn(0x5625a52f5af0 msgr2=0x5625a5413640 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:27:41.471 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:41.480+0000 7f9aeeaddb80 1 -- 172.21.15.52:0/126405977 shutdown_connections 2024-05-27T07:27:41.471 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:41.480+0000 7f9aeeaddb80 1 -- 172.21.15.52:0/126405977 wait complete. 2024-05-27T07:27:41.474 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:41.480+0000 7f9aeeaddb80 1 -- start start 2024-05-27T07:27:41.474 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:41.480+0000 7f9aeeaddb80 1 Processor -- start 2024-05-27T07:27:41.474 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:41.481+0000 7f9aeeaddb80 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x5625a529ae40 0x5625a5422430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:27:41.474 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:41.481+0000 7f9aeeaddb80 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5625a5418f80 0x5625a5422970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:27:41.475 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:41.481+0000 7f9aeeaddb80 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5625a541ec50 0x5625a5422eb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:27:41.475 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:41.481+0000 7f9aeeaddb80 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x5625a523dfa0 con 0x5625a529ae40 2024-05-27T07:27:41.475 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:41.481+0000 7f9aeeaddb80 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x5625a52d97c0 con 0x5625a541ec50 2024-05-27T07:27:41.475 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:41.481+0000 7f9aeeaddb80 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x5625a5425500 con 0x5625a5418f80 2024-05-27T07:27:41.475 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:41.481+0000 7f9adde64700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x5625a529ae40 0x5625a5422430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:27:41.475 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:41.481+0000 7f9aeeaddb80 10 client.0 _ll_register_callbacks cb 0x5625a52d2920 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T07:27:41.475 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:41.481+0000 7f9adde64700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x5625a529ae40 0x5625a5422430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.52:38492/0 (socket says 172.21.15.52:38492) 2024-05-27T07:27:41.475 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:41.481+0000 7f9adde64700 1 -- 172.21.15.52:0/858968659 learned_addr learned my addr 172.21.15.52:0/858968659 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-05-27T07:27:41.475 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:41.481+0000 7f9add663700 1 --2- 172.21.15.52:0/858968659 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5625a5418f80 0x5625a5422970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:27:41.475 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:41.481+0000 7f9ade665700 1 --2- 172.21.15.52:0/858968659 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5625a541ec50 0x5625a5422eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:27:41.475 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:41.481+0000 7f9adde64700 1 -- 172.21.15.52:0/858968659 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5625a5418f80 msgr2=0x5625a5422970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:27:41.475 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:41.481+0000 7f9adde64700 1 --2- 172.21.15.52:0/858968659 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5625a5418f80 0x5625a5422970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:27:41.476 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:41.481+0000 7f9adde64700 1 -- 172.21.15.52:0/858968659 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5625a541ec50 msgr2=0x5625a5422eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:27:41.476 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:41.481+0000 7f9adde64700 1 --2- 172.21.15.52:0/858968659 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5625a541ec50 0x5625a5422eb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:27:41.476 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:41.481+0000 7f9adde64700 1 -- 172.21.15.52:0/858968659 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5625a52d6690 con 0x5625a529ae40 2024-05-27T07:27:41.476 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:41.481+0000 7f9ade665700 1 --2- 172.21.15.52:0/858968659 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5625a541ec50 0x5625a5422eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-27T07:27:41.476 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:41.481+0000 7f9adde64700 1 --2- 172.21.15.52:0/858968659 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x5625a529ae40 0x5625a5422430 secure :-1 s=READY pgs=549 cs=0 l=1 rev1=1 crypto rx=0x7f9ad4009990 tx=0x7f9ad4003bf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=354e80dc14a28d92 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:27:41.476 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:41.481+0000 7f9ac17fa700 10 client.0 ms_handle_connect on v2:172.21.15.52:3300/0 2024-05-27T07:27:41.476 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:41.482+0000 7f9ac17fa700 1 -- 172.21.15.52:0/858968659 <== mon.0 v2:172.21.15.52:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9ad40cc070 con 0x5625a529ae40 2024-05-27T07:27:41.476 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:41.482+0000 7f9ac17fa700 1 -- 172.21.15.52:0/858968659 <== mon.0 v2:172.21.15.52:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 9379+0+0 (secure 0 0 0) 0x7f9ad40bb710 con 0x5625a529ae40 2024-05-27T07:27:41.476 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:41.482+0000 7f9aeeaddb80 1 -- 172.21.15.52:0/858968659 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f9ad40b6f70 con 0x5625a529ae40 2024-05-27T07:27:41.476 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:41.482+0000 7f9ac17fa700 1 -- 172.21.15.52:0/858968659 <== mon.0 v2:172.21.15.52:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7f9ad40b2bc0 con 0x5625a529ae40 2024-05-27T07:27:41.476 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:41.482+0000 7f9aeeaddb80 10 client.15930 fetch_fsmap learned FSMap version 30 2024-05-27T07:27:41.476 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:41.482+0000 7f9aeeaddb80 1 -- 172.21.15.52:0/858968659 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x5625a542ff90 con 0x5625a529ae40 2024-05-27T07:27:41.476 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:41.482+0000 7f9ac17fa700 1 -- 172.21.15.52:0/858968659 <== mon.0 v2:172.21.15.52:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f9ad40bb990 con 0x5625a529ae40 2024-05-27T07:27:41.477 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:41.482+0000 7f9ac17fa700 1 -- 172.21.15.52:0/858968659 <== mon.0 v2:172.21.15.52:3300/0 5 ==== fsmap(e 30) ==== 8362+0+0 (secure 0 0 0) 0x7f9ad4033020 con 0x5625a529ae40 2024-05-27T07:27:41.477 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:41.482+0000 7f9aeeaddb80 10 client.15930 fetch_fsmap finished waiting for FSMap version 30 2024-05-27T07:27:41.477 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:41.482+0000 7f9aeeaddb80 10 client.15930 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-05-27T07:27:41.477 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:41.482+0000 7f9aeeaddb80 20 client.15930 populate_metadata read hostname 'smithi052' 2024-05-27T07:27:41.477 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:41.482+0000 7f9aeeaddb80 1 --2- 172.21.15.52:0/858968659 >> [v2:172.21.15.105:6838/500348151,v1:172.21.15.105:6839/500348151] conn(0x5625a5432460 0x5625a5434880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:27:41.477 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:41.482+0000 7f9aeeaddb80 4 client.15930 mds_command: new command op to 24527 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-27T07:27:41.477 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:41.482+0000 7f9aeeaddb80 1 -- 172.21.15.52:0/858968659 --> [v2:172.21.15.105:6838/500348151,v1:172.21.15.105:6839/500348151] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x7f9ad40b6f70 con 0x5625a5432460 2024-05-27T07:27:41.477 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:41.483+0000 7f9add663700 1 --2- 172.21.15.52:0/858968659 >> [v2:172.21.15.105:6838/500348151,v1:172.21.15.105:6839/500348151] conn(0x5625a5432460 0x5625a5434880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:27:41.477 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:41.483+0000 7f9add663700 1 --2- 172.21.15.52:0/858968659 >> [v2:172.21.15.105:6838/500348151,v1:172.21.15.105:6839/500348151] conn(0x5625a5432460 0x5625a5434880 crc :-1 s=READY pgs=71 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=b2d47176e77b7606 server_cookie=7e1b2ab55ab70290 in_seq=0 out_seq=0 2024-05-27T07:27:41.477 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:41.483+0000 7f9ac17fa700 10 client.15930 ms_handle_connect on v2:172.21.15.105:6838/500348151 2024-05-27T07:27:41.477 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:41.484+0000 7f9ac17fa700 1 -- 172.21.15.52:0/858968659 <== mds.0 v2:172.21.15.105:6838/500348151 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24041 (crc 0 0 0) 0x7f9ad40b6f70 con 0x5625a5432460 2024-05-27T07:27:41.477 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:41.484+0000 7f9ac17fa700 10 client.15930 handle_command_reply: tid=0 2024-05-27T07:27:41.478 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:41.484+0000 7f9aeeaddb80 1 client.15930 shutdown 2024-05-27T07:27:41.478 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:41.484+0000 7f9aeeaddb80 1 -- 172.21.15.52:0/858968659 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x5625a529ae40 msgr2=0x5625a5422430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:27:41.478 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:41.484+0000 7f9aeeaddb80 1 --2- 172.21.15.52:0/858968659 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x5625a529ae40 0x5625a5422430 secure :-1 s=READY pgs=549 cs=0 l=1 rev1=1 crypto rx=0x7f9ad4009990 tx=0x7f9ad4003bf0 comp rx=0 tx=0).stop 2024-05-27T07:27:41.478 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:41.484+0000 7f9aeeaddb80 1 -- 172.21.15.52:0/858968659 shutdown_connections 2024-05-27T07:27:41.478 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:41.484+0000 7f9aeeaddb80 1 --2- 172.21.15.52:0/858968659 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5625a541ec50 0x5625a5422eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:27:41.478 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:41.484+0000 7f9aeeaddb80 1 --2- 172.21.15.52:0/858968659 >> [v2:172.21.15.105:6838/500348151,v1:172.21.15.105:6839/500348151] conn(0x5625a5432460 0x5625a5434880 crc :-1 s=READY pgs=71 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:27:41.478 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:41.484+0000 7f9aeeaddb80 1 --2- 172.21.15.52:0/858968659 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x5625a529ae40 0x5625a5422430 unknown :-1 s=CLOSED pgs=549 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:27:41.478 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:41.484+0000 7f9aeeaddb80 1 --2- 172.21.15.52:0/858968659 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5625a5418f80 0x5625a5422970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:27:41.478 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:41.484+0000 7f9ac17fa700 0 client.15930 ms_handle_reset on v2:172.21.15.105:6838/500348151 2024-05-27T07:27:41.478 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:41.484+0000 7f9aeeaddb80 1 -- 172.21.15.52:0/858968659 >> 172.21.15.52:0/858968659 conn(0x5625a52f5af0 msgr2=0x5625a5417a30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:27:41.478 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:41.484+0000 7f9aeeaddb80 1 -- 172.21.15.52:0/858968659 shutdown_connections 2024-05-27T07:27:41.478 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:41.484+0000 7f9aeeaddb80 1 -- 172.21.15.52:0/858968659 wait complete. 2024-05-27T07:27:41.479 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:41.485+0000 7f9aeeaddb80 20 client.15930 trim_cache size 0 max 16384 2024-05-27T07:27:41.513 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:41.520+0000 7f9aeeaddb80 1 Processor -- start 2024-05-27T07:27:41.513 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:41.520+0000 7f9aeeaddb80 1 -- start start 2024-05-27T07:27:41.514 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:41.521+0000 7f9aeeaddb80 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5625a529ae40 0x5625a5418a40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:27:41.514 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:41.521+0000 7f9aeeaddb80 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x5625a5418f80 0x5625a5420070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:27:41.514 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:41.521+0000 7f9aeeaddb80 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5625a541ec50 0x5625a5421430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:27:41.514 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:41.521+0000 7f9aeeaddb80 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x7f9ad81ae200 con 0x5625a5418f80 2024-05-27T07:27:41.514 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:41.521+0000 7f9aeeaddb80 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f9ad81afd40 con 0x5625a541ec50 2024-05-27T07:27:41.514 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:41.521+0000 7f9aeeaddb80 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f9ad81ae8d0 con 0x5625a529ae40 2024-05-27T07:27:41.514 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:41.521+0000 7f9add663700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x5625a5418f80 0x5625a5420070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:27:41.515 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:41.521+0000 7f9add663700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x5625a5418f80 0x5625a5420070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.52:38504/0 (socket says 172.21.15.52:38504) 2024-05-27T07:27:41.515 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:41.521+0000 7f9add663700 1 -- 172.21.15.52:0/3877278923 learned_addr learned my addr 172.21.15.52:0/3877278923 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-05-27T07:27:41.515 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:41.521+0000 7f9adde64700 1 --2- 172.21.15.52:0/3877278923 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5625a529ae40 0x5625a5418a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:27:41.515 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:41.521+0000 7f9ade665700 1 --2- 172.21.15.52:0/3877278923 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5625a541ec50 0x5625a5421430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:27:41.515 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:41.521+0000 7f9add663700 1 -- 172.21.15.52:0/3877278923 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5625a529ae40 msgr2=0x5625a5418a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:27:41.515 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:41.521+0000 7f9add663700 1 --2- 172.21.15.52:0/3877278923 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5625a529ae40 0x5625a5418a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:27:41.515 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:41.521+0000 7f9add663700 1 -- 172.21.15.52:0/3877278923 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5625a541ec50 msgr2=0x5625a5421430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:27:41.515 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:41.521+0000 7f9add663700 1 --2- 172.21.15.52:0/3877278923 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5625a541ec50 0x5625a5421430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:27:41.516 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:41.521+0000 7f9add663700 1 -- 172.21.15.52:0/3877278923 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5625a54209a0 con 0x5625a5418f80 2024-05-27T07:27:41.516 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:41.522+0000 7f9add663700 1 --2- 172.21.15.52:0/3877278923 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x5625a5418f80 0x5625a5420070 secure :-1 s=READY pgs=550 cs=0 l=1 rev1=1 crypto rx=0x7f9ad8107930 tx=0x7f9ac8014000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e2a57282ad6f9bd7 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:27:41.516 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:41.522+0000 7f9ac37fe700 1 -- 172.21.15.52:0/3877278923 <== mon.0 v2:172.21.15.52:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9ac8004220 con 0x5625a5418f80 2024-05-27T07:27:41.516 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:41.522+0000 7f9ac37fe700 1 -- 172.21.15.52:0/3877278923 <== mon.0 v2:172.21.15.52:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f9ac80298f0 con 0x5625a5418f80 2024-05-27T07:27:41.516 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:41.522+0000 7f9ac37fe700 1 -- 172.21.15.52:0/3877278923 <== mon.0 v2:172.21.15.52:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9ac801e810 con 0x5625a5418f80 2024-05-27T07:27:41.516 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:41.523+0000 7f9aeeaddb80 1 -- 172.21.15.52:0/3877278923 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x5625a5418f80 msgr2=0x5625a5420070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:27:41.516 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:41.523+0000 7f9aeeaddb80 1 --2- 172.21.15.52:0/3877278923 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x5625a5418f80 0x5625a5420070 secure :-1 s=READY pgs=550 cs=0 l=1 rev1=1 crypto rx=0x7f9ad8107930 tx=0x7f9ac8014000 comp rx=0 tx=0).stop 2024-05-27T07:27:41.516 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:41.523+0000 7f9aeeaddb80 1 -- 172.21.15.52:0/3877278923 shutdown_connections 2024-05-27T07:27:41.517 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:41.523+0000 7f9aeeaddb80 1 --2- 172.21.15.52:0/3877278923 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5625a541ec50 0x5625a5421430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:27:41.517 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:41.523+0000 7f9aeeaddb80 1 --2- 172.21.15.52:0/3877278923 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5625a529ae40 0x5625a5418a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:27:41.517 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:41.523+0000 7f9aeeaddb80 1 --2- 172.21.15.52:0/3877278923 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x5625a5418f80 0x5625a5420070 unknown :-1 s=CLOSED pgs=550 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:27:41.517 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:41.523+0000 7f9aeeaddb80 1 -- 172.21.15.52:0/3877278923 >> 172.21.15.52:0/3877278923 conn(0x5625a52f5af0 msgr2=0x5625a5420d20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:27:41.517 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:41.523+0000 7f9aeeaddb80 1 -- 172.21.15.52:0/3877278923 shutdown_connections 2024-05-27T07:27:41.517 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:41.523+0000 7f9aeeaddb80 1 -- 172.21.15.52:0/3877278923 wait complete. 2024-05-27T07:27:41.517 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:41.523+0000 7f9aeeaddb80 1 -- start start 2024-05-27T07:27:41.518 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:41.523+0000 7f9aeeaddb80 1 Processor -- start 2024-05-27T07:27:41.521 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:41.524+0000 7f9aeeaddb80 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5625a529ae40 0x5625a5438160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:27:41.521 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:41.524+0000 7f9aeeaddb80 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x5625a5418f80 0x5625a5435e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:27:41.521 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:41.524+0000 7f9aeeaddb80 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5625a541ec50 0x5625a5436540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:27:41.521 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:41.524+0000 7f9aeeaddb80 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x7f9ad81af670 con 0x5625a5418f80 2024-05-27T07:27:41.521 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:41.524+0000 7f9aeeaddb80 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f9ad81aefa0 con 0x5625a529ae40 2024-05-27T07:27:41.521 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:41.524+0000 7f9aeeaddb80 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f9ad81ac1a0 con 0x5625a541ec50 2024-05-27T07:27:41.522 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:41.524+0000 7f9aeeaddb80 10 client.0 _ll_register_callbacks cb 0x5625a52d2920 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T07:27:41.522 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:41.524+0000 7f9add663700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x5625a5418f80 0x5625a5435e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:27:41.522 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:41.524+0000 7f9add663700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x5625a5418f80 0x5625a5435e80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.52:38518/0 (socket says 172.21.15.52:38518) 2024-05-27T07:27:41.522 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:41.524+0000 7f9add663700 1 -- 172.21.15.52:0/2882386459 learned_addr learned my addr 172.21.15.52:0/2882386459 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-05-27T07:27:41.522 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:41.524+0000 7f9adde64700 1 --2- 172.21.15.52:0/2882386459 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5625a529ae40 0x5625a5438160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:27:41.522 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:41.525+0000 7f9ade665700 1 --2- 172.21.15.52:0/2882386459 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5625a541ec50 0x5625a5436540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:27:41.522 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:41.525+0000 7f9add663700 1 -- 172.21.15.52:0/2882386459 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5625a541ec50 msgr2=0x5625a5436540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:27:41.522 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:41.525+0000 7f9add663700 1 --2- 172.21.15.52:0/2882386459 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5625a541ec50 0x5625a5436540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:27:41.522 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:41.525+0000 7f9add663700 1 -- 172.21.15.52:0/2882386459 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5625a529ae40 msgr2=0x5625a5438160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:27:41.522 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:41.525+0000 7f9add663700 1 --2- 172.21.15.52:0/2882386459 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5625a529ae40 0x5625a5438160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:27:41.522 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:41.525+0000 7f9add663700 1 -- 172.21.15.52:0/2882386459 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5625a549bcd0 con 0x5625a5418f80 2024-05-27T07:27:41.522 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:41.525+0000 7f9adde64700 1 --2- 172.21.15.52:0/2882386459 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5625a529ae40 0x5625a5438160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-27T07:27:41.523 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:41.525+0000 7f9add663700 1 --2- 172.21.15.52:0/2882386459 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x5625a5418f80 0x5625a5435e80 secure :-1 s=READY pgs=551 cs=0 l=1 rev1=1 crypto rx=0x7f9ac8005950 tx=0x7f9ac8003ea0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ec85cef4d798f692 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:27:41.523 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:41.525+0000 7f9ac17fa700 10 client.0 ms_handle_connect on v2:172.21.15.52:3300/0 2024-05-27T07:27:41.523 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:41.525+0000 7f9ac17fa700 1 -- 172.21.15.52:0/2882386459 <== mon.0 v2:172.21.15.52:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9ac8029aa0 con 0x5625a5418f80 2024-05-27T07:27:41.523 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:41.525+0000 7f9ac17fa700 1 -- 172.21.15.52:0/2882386459 <== mon.0 v2:172.21.15.52:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 9379+0+0 (secure 0 0 0) 0x7f9ac8029c40 con 0x5625a5418f80 2024-05-27T07:27:41.523 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:41.525+0000 7f9aeeaddb80 1 -- 172.21.15.52:0/2882386459 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f9ac80298f0 con 0x5625a5418f80 2024-05-27T07:27:41.523 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:41.526+0000 7f9ac17fa700 1 -- 172.21.15.52:0/2882386459 <== mon.0 v2:172.21.15.52:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7f9ac8045050 con 0x5625a5418f80 2024-05-27T07:27:41.523 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:41.526+0000 7f9aeeaddb80 10 client.15936 fetch_fsmap learned FSMap version 30 2024-05-27T07:27:41.523 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:41.526+0000 7f9aeeaddb80 1 -- 172.21.15.52:0/2882386459 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x5625a549cde0 con 0x5625a5418f80 2024-05-27T07:27:41.523 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:41.526+0000 7f9ac17fa700 1 -- 172.21.15.52:0/2882386459 <== mon.0 v2:172.21.15.52:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f9ac80298f0 con 0x5625a5418f80 2024-05-27T07:27:41.523 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:41.526+0000 7f9ac17fa700 1 -- 172.21.15.52:0/2882386459 <== mon.0 v2:172.21.15.52:3300/0 5 ==== fsmap(e 30) ==== 8362+0+0 (secure 0 0 0) 0x7f9ac8032070 con 0x5625a5418f80 2024-05-27T07:27:41.523 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:41.526+0000 7f9aeeaddb80 10 client.15936 fetch_fsmap finished waiting for FSMap version 30 2024-05-27T07:27:41.523 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:41.526+0000 7f9aeeaddb80 10 client.15936 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-05-27T07:27:41.523 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:41.526+0000 7f9aeeaddb80 20 client.15936 populate_metadata read hostname 'smithi052' 2024-05-27T07:27:41.524 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:41.526+0000 7f9aeeaddb80 1 --2- 172.21.15.52:0/2882386459 >> [v2:172.21.15.105:6838/500348151,v1:172.21.15.105:6839/500348151] conn(0x5625a549d3d0 0x5625a549f7f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:27:41.524 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:41.526+0000 7f9aeeaddb80 4 client.15936 mds_command: new command op to 24527 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-05-27T07:27:41.525 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:41.526+0000 7f9aeeaddb80 1 -- 172.21.15.52:0/2882386459 --> [v2:172.21.15.105:6838/500348151,v1:172.21.15.105:6839/500348151] -- command(tid 0: {"prefix": "scrub status"}) -- 0x5625a53e49f0 con 0x5625a549d3d0 2024-05-27T07:27:41.525 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:41.526+0000 7f9adde64700 1 --2- 172.21.15.52:0/2882386459 >> [v2:172.21.15.105:6838/500348151,v1:172.21.15.105:6839/500348151] conn(0x5625a549d3d0 0x5625a549f7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:27:41.525 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:41.527+0000 7f9ac17fa700 10 client.15936 ms_handle_connect on v2:172.21.15.105:6838/500348151 2024-05-27T07:27:41.525 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:41.527+0000 7f9adde64700 1 --2- 172.21.15.52:0/2882386459 >> [v2:172.21.15.105:6838/500348151,v1:172.21.15.105:6839/500348151] conn(0x5625a549d3d0 0x5625a549f7f0 crc :-1 s=READY pgs=72 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=51315220b549b8c6 server_cookie=8e8c2b1f6e97f35e in_seq=0 out_seq=0 2024-05-27T07:27:41.526 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:41.527+0000 7f9ac17fa700 1 -- 172.21.15.52:0/2882386459 <== mds.0 v2:172.21.15.105:6838/500348151 1 ==== command_reply(tid 0: 0 ) ==== 8+0+259 (crc 0 0 0) 0x5625a53e49f0 con 0x5625a549d3d0 2024-05-27T07:27:41.526 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:41.527+0000 7f9ac17fa700 10 client.15936 handle_command_reply: tid=0 2024-05-27T07:27:41.526 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:41.527+0000 7f9aeeaddb80 1 client.15936 shutdown 2024-05-27T07:27:41.526 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:41.527+0000 7f9aeeaddb80 1 -- 172.21.15.52:0/2882386459 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x5625a5418f80 msgr2=0x5625a5435e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:27:41.526 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:41.527+0000 7f9aeeaddb80 1 --2- 172.21.15.52:0/2882386459 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x5625a5418f80 0x5625a5435e80 secure :-1 s=READY pgs=551 cs=0 l=1 rev1=1 crypto rx=0x7f9ac8005950 tx=0x7f9ac8003ea0 comp rx=0 tx=0).stop 2024-05-27T07:27:41.526 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:41.527+0000 7f9aeeaddb80 1 -- 172.21.15.52:0/2882386459 shutdown_connections 2024-05-27T07:27:41.526 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:41.527+0000 7f9aeeaddb80 1 --2- 172.21.15.52:0/2882386459 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5625a541ec50 0x5625a5436540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:27:41.526 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:41.527+0000 7f9aeeaddb80 1 --2- 172.21.15.52:0/2882386459 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5625a529ae40 0x5625a5438160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:27:41.526 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:41.527+0000 7f9aeeaddb80 1 --2- 172.21.15.52:0/2882386459 >> [v2:172.21.15.105:6838/500348151,v1:172.21.15.105:6839/500348151] conn(0x5625a549d3d0 0x5625a549f7f0 crc :-1 s=READY pgs=72 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:27:41.526 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:41.527+0000 7f9aeeaddb80 1 --2- 172.21.15.52:0/2882386459 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x5625a5418f80 0x5625a5435e80 unknown :-1 s=CLOSED pgs=551 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:27:41.526 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:41.527+0000 7f9ac17fa700 0 client.15936 ms_handle_reset on v2:172.21.15.105:6838/500348151 2024-05-27T07:27:41.527 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:41.527+0000 7f9aeeaddb80 1 -- 172.21.15.52:0/2882386459 >> 172.21.15.52:0/2882386459 conn(0x5625a52f5af0 msgr2=0x5625a541c010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:27:41.527 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:41.527+0000 7f9aeeaddb80 1 -- 172.21.15.52:0/2882386459 shutdown_connections 2024-05-27T07:27:41.527 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:41.528+0000 7f9aeeaddb80 1 -- 172.21.15.52:0/2882386459 wait complete. 2024-05-27T07:27:41.527 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:41.528+0000 7f9aeeaddb80 20 client.15936 trim_cache size 0 max 16384 2024-05-27T07:27:41.527 INFO:teuthology.orchestra.run.smithi052.stdout:{ 2024-05-27T07:27:41.527 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "IDLE (waiting for more scrubs)", 2024-05-27T07:27:41.528 INFO:teuthology.orchestra.run.smithi052.stdout: "scrubs": { 2024-05-27T07:27:41.528 INFO:teuthology.orchestra.run.smithi052.stdout: "f161a4de-d3f0-4469-bf1b-0f1834e40c39": { 2024-05-27T07:27:41.528 INFO:teuthology.orchestra.run.smithi052.stdout: "path": "/", 2024-05-27T07:27:41.528 INFO:teuthology.orchestra.run.smithi052.stdout: "tag": "f161a4de-d3f0-4469-bf1b-0f1834e40c39", 2024-05-27T07:27:41.528 INFO:teuthology.orchestra.run.smithi052.stdout: "options": "recursive,force" 2024-05-27T07:27:41.528 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-05-27T07:27:41.528 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-05-27T07:27:41.528 INFO:teuthology.orchestra.run.smithi052.stdout:} 2024-05-27T07:27:41.529 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:41.528+0000 7f9ae00c8700 1 -- 172.21.15.52:0/667408378 >> [v2:172.21.15.52:6800/1822318384,v1:172.21.15.52:6801/1822318384] conn(0x7f9ac40b29f0 msgr2=0x7f9ac40b4f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:27:41.529 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:41.528+0000 7f9ae00c8700 1 --2- 172.21.15.52:0/667408378 >> [v2:172.21.15.52:6800/1822318384,v1:172.21.15.52:6801/1822318384] conn(0x7f9ac40b29f0 0x7f9ac40b4f00 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f9ad81077c0 tx=0x7f9ac8009260 comp rx=0 tx=0).stop 2024-05-27T07:27:41.529 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:41.528+0000 7f9ae00c8700 1 -- 172.21.15.52:0/667408378 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f9ad8102b40 msgr2=0x7f9ad819e850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:27:41.529 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:41.528+0000 7f9ae00c8700 1 --2- 172.21.15.52:0/667408378 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f9ad8102b40 0x7f9ad819e850 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f9ad40098e0 tx=0x7f9ad4003680 comp rx=0 tx=0).stop 2024-05-27T07:27:41.529 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:41.528+0000 7f9ae00c8700 1 -- 172.21.15.52:0/667408378 shutdown_connections 2024-05-27T07:27:41.529 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:41.528+0000 7f9ae00c8700 1 --2- 172.21.15.52:0/667408378 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f9ad8109840 0x7f9ad8198a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:27:41.529 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:41.528+0000 7f9ae00c8700 1 --2- 172.21.15.52:0/667408378 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f9ad8102b40 0x7f9ad819e850 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:27:41.529 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:41.528+0000 7f9ae00c8700 1 --2- 172.21.15.52:0/667408378 >> [v2:172.21.15.52:6800/1822318384,v1:172.21.15.52:6801/1822318384] conn(0x7f9ac40b29f0 0x7f9ac40b4f00 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:27:41.529 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:41.528+0000 7f9ae00c8700 1 --2- 172.21.15.52:0/667408378 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f9ad8108ef0 0x7f9ad819ed90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:27:41.529 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:41.528+0000 7f9ae00c8700 1 -- 172.21.15.52:0/667408378 >> 172.21.15.52:0/667408378 conn(0x7f9ad80fe6e0 msgr2=0x7f9ad810a7a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:27:41.529 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:41.528+0000 7f9ae00c8700 1 -- 172.21.15.52:0/667408378 shutdown_connections 2024-05-27T07:27:41.530 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:27:41.528+0000 7f9ae00c8700 1 -- 172.21.15.52:0/667408378 wait complete. 2024-05-27T07:27:42.229 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:27:42 smithi052 ceph-mon[31955]: pgmap v55: 129 pgs: 129 active+clean; 1.4 MiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 255 B/s wr, 8 op/s 2024-05-27T07:27:42.229 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:27:42 smithi052 ceph-mon[31955]: scrub summary: idle 2024-05-27T07:27:42.313 INFO:tasks.cephfs.filesystem:scrub status for tag:f161a4de-d3f0-4469-bf1b-0f1834e40c39 - {'path': '/', 'tag': 'f161a4de-d3f0-4469-bf1b-0f1834e40c39', 'options': 'recursive,force'} 2024-05-27T07:27:42.439 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:27:42 smithi105 ceph-mon[35977]: pgmap v55: 129 pgs: 129 active+clean; 1.4 MiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 255 B/s wr, 8 op/s 2024-05-27T07:27:42.439 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:27:42 smithi105 ceph-mon[35977]: scrub summary: idle 2024-05-27T07:27:42.602 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:27:42 smithi173 ceph-mon[37000]: pgmap v55: 129 pgs: 129 active+clean; 1.4 MiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 255 B/s wr, 8 op/s 2024-05-27T07:27:42.602 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:27:42 smithi173 ceph-mon[37000]: scrub summary: idle 2024-05-27T07:27:44.389 INFO:tasks.workunit.client.0.smithi052.stderr:Updating files: 93% (11457/12271) Updating files: 94% (11535/12271) Updating files: 95% (11658/12271) Updating files: 96% (11781/12271) Updating files: 97% (11903/12271) Updating files: 98% (12026/12271) Updating files: 99% (12149/12271) Updating files: 100% (12271/12271) Updating files: 100% (12271/12271), done. 2024-05-27T07:27:44.439 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:27:44 smithi105 ceph-mon[35977]: pgmap v56: 129 pgs: 129 active+clean; 1.4 MiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 7.4 KiB/s rd, 255 B/s wr, 14 op/s 2024-05-27T07:27:44.479 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:27:44 smithi052 ceph-mon[31955]: pgmap v56: 129 pgs: 129 active+clean; 1.4 MiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 7.4 KiB/s rd, 255 B/s wr, 14 op/s 2024-05-27T07:27:44.584 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:27:44 smithi173 ceph-mon[37000]: pgmap v56: 129 pgs: 129 active+clean; 1.4 MiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 7.4 KiB/s rd, 255 B/s wr, 14 op/s 2024-05-27T07:27:45.098 INFO:tasks.workunit.client.0.smithi052.stderr:Note: switching to '38f9d900c83c63799fbdbe61acc9a11b0d3554a6'. 2024-05-27T07:27:45.098 INFO:tasks.workunit.client.0.smithi052.stderr: 2024-05-27T07:27:45.098 INFO:tasks.workunit.client.0.smithi052.stderr:You are in 'detached HEAD' state. You can look around, make experimental 2024-05-27T07:27:45.098 INFO:tasks.workunit.client.0.smithi052.stderr:changes and commit them, and you can discard any commits you make in this 2024-05-27T07:27:45.098 INFO:tasks.workunit.client.0.smithi052.stderr:state without impacting any branches by switching back to a branch. 2024-05-27T07:27:45.098 INFO:tasks.workunit.client.0.smithi052.stderr: 2024-05-27T07:27:45.098 INFO:tasks.workunit.client.0.smithi052.stderr:If you want to create a new branch to retain commits you create, you may 2024-05-27T07:27:45.098 INFO:tasks.workunit.client.0.smithi052.stderr:do so (now or later) by using -c with the switch command. Example: 2024-05-27T07:27:45.099 INFO:tasks.workunit.client.0.smithi052.stderr: 2024-05-27T07:27:45.099 INFO:tasks.workunit.client.0.smithi052.stderr: git switch -c 2024-05-27T07:27:45.099 INFO:tasks.workunit.client.0.smithi052.stderr: 2024-05-27T07:27:45.099 INFO:tasks.workunit.client.0.smithi052.stderr:Or undo this operation with: 2024-05-27T07:27:45.099 INFO:tasks.workunit.client.0.smithi052.stderr: 2024-05-27T07:27:45.099 INFO:tasks.workunit.client.0.smithi052.stderr: git switch - 2024-05-27T07:27:45.099 INFO:tasks.workunit.client.0.smithi052.stderr: 2024-05-27T07:27:45.099 INFO:tasks.workunit.client.0.smithi052.stderr:Turn off this advice by setting config variable advice.detachedHead to false 2024-05-27T07:27:45.099 INFO:tasks.workunit.client.0.smithi052.stderr: 2024-05-27T07:27:45.099 INFO:tasks.workunit.client.0.smithi052.stderr:HEAD is now at 38f9d900c83 Merge PR #55758 into wip-vshankar-testing-20240521.063558-debug 2024-05-27T07:27:45.104 DEBUG:teuthology.orchestra.run.smithi052:> cd -- /home/ubuntu/cephtest/clone.client.0/qa/workunits && if test -e Makefile ; then make ; fi && find -executable -type f -printf '%P\0' >/home/ubuntu/cephtest/workunits.list.client.0 2024-05-27T07:27:45.128 INFO:tasks.workunit.client.0.smithi052.stdout:for d in direct_io fs ; do ( cd $d ; make all ) ; done 2024-05-27T07:27:45.131 INFO:tasks.workunit.client.0.smithi052.stdout:make[1]: Entering directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/direct_io' 2024-05-27T07:27:45.131 INFO:tasks.workunit.client.0.smithi052.stdout:cc -Wall -Wextra -D_GNU_SOURCE direct_io_test.c -o direct_io_test 2024-05-27T07:27:45.136 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:27:45.146+0000 7f8892449640 1 -- 172.21.15.52:0/1977934146 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f888c109030 msgr2=0x7f888c102c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:27:45.136 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:27:45.146+0000 7f8892449640 1 --2- 172.21.15.52:0/1977934146 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f888c109030 0x7f888c102c70 secure :-1 s=READY pgs=552 cs=0 l=1 rev1=1 crypto rx=0x7f8874009900 tx=0x7f887402a410 comp rx=0 tx=0).stop 2024-05-27T07:27:45.136 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:27:45.146+0000 7f8892449640 1 -- 172.21.15.52:0/1977934146 shutdown_connections 2024-05-27T07:27:45.136 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:27:45.146+0000 7f8892449640 1 --2- 172.21.15.52:0/1977934146 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f888c1032e0 0x7f888c10f6b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:27:45.136 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:27:45.146+0000 7f8892449640 1 --2- 172.21.15.52:0/1977934146 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f888c109030 0x7f888c102c70 unknown :-1 s=CLOSED pgs=552 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:27:45.136 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:27:45.146+0000 7f8892449640 1 --2- 172.21.15.52:0/1977934146 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f888c108b90 0x7f888c102730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:27:45.137 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:27:45.146+0000 7f8892449640 1 -- 172.21.15.52:0/1977934146 >> 172.21.15.52:0/1977934146 conn(0x7f888c0782d0 msgr2=0x7f888c078710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:27:45.137 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:27:45.146+0000 7f8892449640 1 -- 172.21.15.52:0/1977934146 shutdown_connections 2024-05-27T07:27:45.137 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:27:45.146+0000 7f8892449640 1 -- 172.21.15.52:0/1977934146 wait complete. 2024-05-27T07:27:45.139 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:27:45.147+0000 7f8892449640 1 Processor -- start 2024-05-27T07:27:45.139 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:27:45.147+0000 7f8892449640 1 -- start start 2024-05-27T07:27:45.139 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:27:45.147+0000 7f8892449640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f888c1032e0 0x7f888c1a0f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:27:45.139 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:27:45.147+0000 7f8892449640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f888c108b90 0x7f888c1a14c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:27:45.139 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:27:45.147+0000 7f8892449640 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f888c109030 0x7f888c1a5780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:27:45.139 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:27:45.147+0000 7f8892449640 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x7f888c11a4e0 con 0x7f888c109030 2024-05-27T07:27:45.139 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:27:45.147+0000 7f8892449640 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f888c11a360 con 0x7f888c108b90 2024-05-27T07:27:45.139 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:27:45.147+0000 7f8892449640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f888c11a660 con 0x7f888c1032e0 2024-05-27T07:27:45.139 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:27:45.147+0000 7f88909e6640 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f888c109030 0x7f888c1a5780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:27:45.139 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:27:45.148+0000 7f88909e6640 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f888c109030 0x7f888c1a5780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.52:38530/0 (socket says 172.21.15.52:38530) 2024-05-27T07:27:45.139 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:27:45.148+0000 7f88909e6640 1 -- 172.21.15.52:0/2186669894 learned_addr learned my addr 172.21.15.52:0/2186669894 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-05-27T07:27:45.139 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:27:45.148+0000 7f888bfff640 1 --2- 172.21.15.52:0/2186669894 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f888c1032e0 0x7f888c1a0f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:27:45.140 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:27:45.148+0000 7f8883fff640 1 --2- 172.21.15.52:0/2186669894 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f888c108b90 0x7f888c1a14c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:27:45.140 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:27:45.148+0000 7f88909e6640 1 -- 172.21.15.52:0/2186669894 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f888c1032e0 msgr2=0x7f888c1a0f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:27:45.140 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:27:45.148+0000 7f88909e6640 1 --2- 172.21.15.52:0/2186669894 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f888c1032e0 0x7f888c1a0f60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:27:45.140 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:27:45.148+0000 7f88909e6640 1 -- 172.21.15.52:0/2186669894 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f888c108b90 msgr2=0x7f888c1a14c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:27:45.140 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:27:45.148+0000 7f88909e6640 1 --2- 172.21.15.52:0/2186669894 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f888c108b90 0x7f888c1a14c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:27:45.140 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:27:45.148+0000 7f88909e6640 1 -- 172.21.15.52:0/2186669894 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f888c1a5dc0 con 0x7f888c109030 2024-05-27T07:27:45.140 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:27:45.148+0000 7f88909e6640 1 --2- 172.21.15.52:0/2186669894 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f888c109030 0x7f888c1a5780 secure :-1 s=READY pgs=553 cs=0 l=1 rev1=1 crypto rx=0x7f888400cad0 tx=0x7f888400ced0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:27:45.140 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:27:45.148+0000 7f8889ffb640 1 -- 172.21.15.52:0/2186669894 <== mon.0 v2:172.21.15.52:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f888400ad60 con 0x7f888c109030 2024-05-27T07:27:45.140 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:27:45.148+0000 7f8889ffb640 1 -- 172.21.15.52:0/2186669894 <== mon.0 v2:172.21.15.52:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f8884005020 con 0x7f888c109030 2024-05-27T07:27:45.140 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:27:45.148+0000 7f8889ffb640 1 -- 172.21.15.52:0/2186669894 <== mon.0 v2:172.21.15.52:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f888400f6b0 con 0x7f888c109030 2024-05-27T07:27:45.140 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:27:45.148+0000 7f8892449640 1 -- 172.21.15.52:0/2186669894 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f888c1a60b0 con 0x7f888c109030 2024-05-27T07:27:45.140 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:27:45.148+0000 7f8892449640 1 -- 172.21.15.52:0/2186669894 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f888c1ada30 con 0x7f888c109030 2024-05-27T07:27:45.143 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:27:45.151+0000 7f88993d7740 1 Processor -- start 2024-05-27T07:27:45.143 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:27:45.151+0000 7f8889ffb640 1 -- 172.21.15.52:0/2186669894 <== mon.0 v2:172.21.15.52:3300/0 4 ==== mgrmap(e 26) ==== 149783+0+0 (secure 0 0 0) 0x7f88840051d0 con 0x7f888c109030 2024-05-27T07:27:45.144 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:27:45.151+0000 7f88993d7740 1 -- start start 2024-05-27T07:27:45.144 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:27:45.151+0000 7f88993d7740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x558ef93d41b0 0x558ef97753d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:27:45.144 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:27:45.151+0000 7f88993d7740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x558ef93ab2f0 0x558ef9775910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:27:45.144 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:27:45.151+0000 7f88993d7740 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x558ef93b34d0 0x558ef9775e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:27:45.144 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:27:45.151+0000 7f88993d7740 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x558ef96e7c30 con 0x558ef93b34d0 2024-05-27T07:27:45.144 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:27:45.151+0000 7f88993d7740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x558ef96d77d0 con 0x558ef93d41b0 2024-05-27T07:27:45.144 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:27:45.151+0000 7f88993d7740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x558ef96db810 con 0x558ef93ab2f0 2024-05-27T07:27:45.144 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:27:45.151+0000 7f8883fff640 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x558ef93b34d0 0x558ef9775e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:27:45.144 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:27:45.151+0000 7f8883fff640 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x558ef93b34d0 0x558ef9775e50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.52:38540/0 (socket says 172.21.15.52:38540) 2024-05-27T07:27:45.144 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:27:45.151+0000 7f8883fff640 1 -- 172.21.15.52:0/553898094 learned_addr learned my addr 172.21.15.52:0/553898094 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-05-27T07:27:45.144 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:27:45.152+0000 7f8883fff640 1 --2- 172.21.15.52:0/553898094 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x558ef93d41b0 0x558ef97753d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:27:45.144 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:27:45.152+0000 7f888bfff640 1 --2- 172.21.15.52:0/553898094 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x558ef93ab2f0 0x558ef9775910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:27:45.145 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:27:45.152+0000 7f8883fff640 1 -- 172.21.15.52:0/553898094 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x558ef93ab2f0 msgr2=0x558ef9775910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:27:45.145 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:27:45.152+0000 7f8883fff640 1 --2- 172.21.15.52:0/553898094 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x558ef93ab2f0 0x558ef9775910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:27:45.145 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:27:45.152+0000 7f8883fff640 1 -- 172.21.15.52:0/553898094 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x558ef93d41b0 msgr2=0x558ef97753d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:27:45.145 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:27:45.152+0000 7f8883fff640 1 --2- 172.21.15.52:0/553898094 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x558ef93d41b0 0x558ef97753d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:27:45.145 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:27:45.152+0000 7f8883fff640 1 -- 172.21.15.52:0/553898094 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x558ef93ec210 con 0x558ef93b34d0 2024-05-27T07:27:45.145 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:27:45.152+0000 7f8889ffb640 1 --2- 172.21.15.52:0/2186669894 >> [v2:172.21.15.52:6800/1822318384,v1:172.21.15.52:6801/1822318384] conn(0x7f88780b29c0 0x7f88780b4ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:27:45.145 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:27:45.152+0000 7f8883fff640 1 --2- 172.21.15.52:0/553898094 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x558ef93b34d0 0x558ef9775e50 secure :-1 s=READY pgs=554 cs=0 l=1 rev1=1 crypto rx=0x558ef9772540 tx=0x7f8874009680 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a52ae58031d98d9c server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:27:45.145 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:27:45.152+0000 7f888bfff640 1 --2- 172.21.15.52:0/2186669894 >> [v2:172.21.15.52:6800/1822318384,v1:172.21.15.52:6801/1822318384] conn(0x7f88780b29c0 0x7f88780b4ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:27:45.145 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:27:45.152+0000 7f8889ffb640 1 -- 172.21.15.52:0/2186669894 <== mon.0 v2:172.21.15.52:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9379+0+0 (secure 0 0 0) 0x7f88840e2930 con 0x7f888c109030 2024-05-27T07:27:45.145 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:27:45.152+0000 7f88827fc640 1 -- 172.21.15.52:0/553898094 <== mon.0 v2:172.21.15.52:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8874035040 con 0x558ef93b34d0 2024-05-27T07:27:45.145 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:27:45.152+0000 7f88827fc640 1 -- 172.21.15.52:0/553898094 <== mon.0 v2:172.21.15.52:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f8874005020 con 0x558ef93b34d0 2024-05-27T07:27:45.145 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:27:45.152+0000 7f88827fc640 1 -- 172.21.15.52:0/553898094 <== mon.0 v2:172.21.15.52:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f88740044e0 con 0x558ef93b34d0 2024-05-27T07:27:45.146 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:27:45.153+0000 7f88993d7740 1 -- 172.21.15.52:0/553898094 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x558ef93b34d0 msgr2=0x558ef9775e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:27:45.146 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:27:45.153+0000 7f888bfff640 1 --2- 172.21.15.52:0/2186669894 >> [v2:172.21.15.52:6800/1822318384,v1:172.21.15.52:6801/1822318384] conn(0x7f88780b29c0 0x7f88780b4ed0 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x558ef9776780 tx=0x7f887c009110 comp rx=0 tx=0).ready entity=mgr.15333 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:27:45.146 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:27:45.153+0000 7f88993d7740 1 --2- 172.21.15.52:0/553898094 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x558ef93b34d0 0x558ef9775e50 secure :-1 s=READY pgs=554 cs=0 l=1 rev1=1 crypto rx=0x558ef9772540 tx=0x7f8874009680 comp rx=0 tx=0).stop 2024-05-27T07:27:45.146 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:27:45.153+0000 7f88993d7740 1 -- 172.21.15.52:0/553898094 shutdown_connections 2024-05-27T07:27:45.146 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:27:45.153+0000 7f88993d7740 1 --2- 172.21.15.52:0/553898094 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x558ef93b34d0 0x558ef9775e50 unknown :-1 s=CLOSED pgs=554 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:27:45.146 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:27:45.153+0000 7f88993d7740 1 --2- 172.21.15.52:0/553898094 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x558ef93ab2f0 0x558ef9775910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:27:45.146 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:27:45.153+0000 7f88993d7740 1 --2- 172.21.15.52:0/553898094 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x558ef93d41b0 0x558ef97753d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:27:45.146 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:27:45.153+0000 7f88993d7740 1 -- 172.21.15.52:0/553898094 >> 172.21.15.52:0/553898094 conn(0x558ef93c8300 msgr2=0x558ef9771b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:27:45.146 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:27:45.153+0000 7f88993d7740 1 -- 172.21.15.52:0/553898094 shutdown_connections 2024-05-27T07:27:45.146 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:27:45.153+0000 7f88993d7740 1 -- 172.21.15.52:0/553898094 wait complete. 2024-05-27T07:27:45.146 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:27:45.154+0000 7f88993d7740 1 -- start start 2024-05-27T07:27:45.147 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:27:45.154+0000 7f88993d7740 1 Processor -- start 2024-05-27T07:27:45.149 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:27:45.154+0000 7f88993d7740 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x558ef93ab2f0 0x558ef9781480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:27:45.149 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:27:45.154+0000 7f88993d7740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x558ef93b34d0 0x558ef97819c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:27:45.149 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:27:45.154+0000 7f88993d7740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x558ef93d41b0 0x558ef977f990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:27:45.150 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:27:45.154+0000 7f88993d7740 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x558ef93bd790 con 0x558ef93ab2f0 2024-05-27T07:27:45.150 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:27:45.154+0000 7f88993d7740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x558ef9693890 con 0x558ef93b34d0 2024-05-27T07:27:45.150 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:27:45.154+0000 7f88993d7740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x558ef97824a0 con 0x558ef93d41b0 2024-05-27T07:27:45.150 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:27:45.155+0000 7f88993d7740 10 client.0 _ll_register_callbacks cb 0x558ef93eba00 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T07:27:45.150 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:27:45.155+0000 7f888bfff640 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x558ef93ab2f0 0x558ef9781480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:27:45.150 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:27:45.155+0000 7f888bfff640 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x558ef93ab2f0 0x558ef9781480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.52:38546/0 (socket says 172.21.15.52:38546) 2024-05-27T07:27:45.150 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:27:45.155+0000 7f888bfff640 1 -- 172.21.15.52:0/2676737153 learned_addr learned my addr 172.21.15.52:0/2676737153 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-05-27T07:27:45.150 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:27:45.155+0000 7f8883fff640 1 --2- 172.21.15.52:0/2676737153 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x558ef93b34d0 0x558ef97819c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:27:45.150 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:27:45.155+0000 7f88909e6640 1 --2- 172.21.15.52:0/2676737153 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x558ef93d41b0 0x558ef977f990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:27:45.150 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:27:45.155+0000 7f888bfff640 1 -- 172.21.15.52:0/2676737153 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x558ef93d41b0 msgr2=0x558ef977f990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:27:45.151 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:27:45.155+0000 7f888bfff640 1 --2- 172.21.15.52:0/2676737153 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x558ef93d41b0 0x558ef977f990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:27:45.151 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:27:45.155+0000 7f888bfff640 1 -- 172.21.15.52:0/2676737153 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x558ef93b34d0 msgr2=0x558ef97819c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:27:45.151 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:27:45.155+0000 7f888bfff640 1 --2- 172.21.15.52:0/2676737153 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x558ef93b34d0 0x558ef97819c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:27:45.151 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:27:45.155+0000 7f888bfff640 1 -- 172.21.15.52:0/2676737153 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({osdmap=0}) -- 0x558ef94b8620 con 0x558ef93ab2f0 2024-05-27T07:27:45.151 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:27:45.155+0000 7f8883fff640 1 --2- 172.21.15.52:0/2676737153 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x558ef93b34d0 0x558ef97819c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-27T07:27:45.151 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:27:45.155+0000 7f888bfff640 1 --2- 172.21.15.52:0/2676737153 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x558ef93ab2f0 0x558ef9781480 secure :-1 s=READY pgs=555 cs=0 l=1 rev1=1 crypto rx=0x7f887c006a80 tx=0x7f887c0047d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=28e255daa35c4de5 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:27:45.151 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:27:45.155+0000 7f886b7fe640 10 client.0 ms_handle_connect on v2:172.21.15.52:3300/0 2024-05-27T07:27:45.151 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:27:45.155+0000 7f886b7fe640 1 -- 172.21.15.52:0/2676737153 <== mon.0 v2:172.21.15.52:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f887c025070 con 0x558ef93ab2f0 2024-05-27T07:27:45.151 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:27:45.155+0000 7f886b7fe640 1 -- 172.21.15.52:0/2676737153 <== mon.0 v2:172.21.15.52:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 9379+0+0 (secure 0 0 0) 0x7f887c021600 con 0x558ef93ab2f0 2024-05-27T07:27:45.151 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:27:45.155+0000 7f88993d7740 1 -- 172.21.15.52:0/2676737153 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f8874005020 con 0x558ef93ab2f0 2024-05-27T07:27:45.151 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:27:45.156+0000 7f886b7fe640 1 -- 172.21.15.52:0/2676737153 <== mon.0 v2:172.21.15.52:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7f887c021800 con 0x558ef93ab2f0 2024-05-27T07:27:45.151 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:27:45.156+0000 7f88993d7740 10 client.15948 fetch_fsmap learned FSMap version 30 2024-05-27T07:27:45.151 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:27:45.156+0000 7f88993d7740 1 -- 172.21.15.52:0/2676737153 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x558ef978e8a0 con 0x558ef93ab2f0 2024-05-27T07:27:45.152 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:27:45.159+0000 7f886b7fe640 1 -- 172.21.15.52:0/2676737153 <== mon.0 v2:172.21.15.52:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f8874005020 con 0x558ef93ab2f0 2024-05-27T07:27:45.152 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:27:45.159+0000 7f886b7fe640 1 -- 172.21.15.52:0/2676737153 <== mon.0 v2:172.21.15.52:3300/0 5 ==== fsmap(e 30) ==== 8362+0+0 (secure 0 0 0) 0x7f887c020070 con 0x558ef93ab2f0 2024-05-27T07:27:45.152 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:27:45.159+0000 7f88993d7740 10 client.15948 fetch_fsmap finished waiting for FSMap version 30 2024-05-27T07:27:45.152 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:27:45.159+0000 7f88993d7740 10 client.15948 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-05-27T07:27:45.152 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:27:45.159+0000 7f88993d7740 20 client.15948 populate_metadata read hostname 'smithi052' 2024-05-27T07:27:45.152 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:27:45.159+0000 7f88993d7740 1 --2- 172.21.15.52:0/2676737153 >> [v2:172.21.15.105:6838/500348151,v1:172.21.15.105:6839/500348151] conn(0x558ef9392d80 0x558ef9791e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:27:45.152 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:27:45.159+0000 7f88993d7740 4 client.15948 mds_command: new command op to 24527 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-27T07:27:45.152 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:27:45.159+0000 7f88993d7740 1 -- 172.21.15.52:0/2676737153 --> [v2:172.21.15.105:6838/500348151,v1:172.21.15.105:6839/500348151] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x558ef96e9b80 con 0x558ef9392d80 2024-05-27T07:27:45.152 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:27:45.159+0000 7f8883fff640 1 --2- 172.21.15.52:0/2676737153 >> [v2:172.21.15.105:6838/500348151,v1:172.21.15.105:6839/500348151] conn(0x558ef9392d80 0x558ef9791e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:27:45.152 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:27:45.160+0000 7f886b7fe640 10 client.15948 ms_handle_connect on v2:172.21.15.105:6838/500348151 2024-05-27T07:27:45.152 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:27:45.160+0000 7f8883fff640 1 --2- 172.21.15.52:0/2676737153 >> [v2:172.21.15.105:6838/500348151,v1:172.21.15.105:6839/500348151] conn(0x558ef9392d80 0x558ef9791e30 crc :-1 s=READY pgs=73 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=e1ba6b5ea4d81e7e server_cookie=2386c1e8ba2eae72 in_seq=0 out_seq=0 2024-05-27T07:27:45.153 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:27:45.160+0000 7f886b7fe640 1 -- 172.21.15.52:0/2676737153 <== mds.0 v2:172.21.15.105:6838/500348151 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24041 (crc 0 0 0) 0x558ef96e9b80 con 0x558ef9392d80 2024-05-27T07:27:45.154 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:27:45.160+0000 7f886b7fe640 10 client.15948 handle_command_reply: tid=0 2024-05-27T07:27:45.154 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:27:45.160+0000 7f88993d7740 1 client.15948 shutdown 2024-05-27T07:27:45.154 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:27:45.161+0000 7f88993d7740 1 -- 172.21.15.52:0/2676737153 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x558ef93ab2f0 msgr2=0x558ef9781480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:27:45.154 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:27:45.161+0000 7f88993d7740 1 --2- 172.21.15.52:0/2676737153 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x558ef93ab2f0 0x558ef9781480 secure :-1 s=READY pgs=555 cs=0 l=1 rev1=1 crypto rx=0x7f887c006a80 tx=0x7f887c0047d0 comp rx=0 tx=0).stop 2024-05-27T07:27:45.154 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:27:45.161+0000 7f88993d7740 1 -- 172.21.15.52:0/2676737153 shutdown_connections 2024-05-27T07:27:45.154 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:27:45.161+0000 7f88993d7740 1 --2- 172.21.15.52:0/2676737153 >> [v2:172.21.15.105:6838/500348151,v1:172.21.15.105:6839/500348151] conn(0x558ef9392d80 0x558ef9791e30 crc :-1 s=READY pgs=73 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:27:45.154 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:27:45.161+0000 7f88993d7740 1 --2- 172.21.15.52:0/2676737153 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x558ef93d41b0 0x558ef977f990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:27:45.154 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:27:45.161+0000 7f88993d7740 1 --2- 172.21.15.52:0/2676737153 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x558ef93b34d0 0x558ef97819c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:27:45.154 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:27:45.161+0000 7f886b7fe640 0 client.15948 ms_handle_reset on v2:172.21.15.105:6838/500348151 2024-05-27T07:27:45.154 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:27:45.161+0000 7f88993d7740 1 --2- 172.21.15.52:0/2676737153 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x558ef93ab2f0 0x558ef9781480 unknown :-1 s=CLOSED pgs=555 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:27:45.154 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:27:45.161+0000 7f88993d7740 1 -- 172.21.15.52:0/2676737153 >> 172.21.15.52:0/2676737153 conn(0x558ef93c8300 msgr2=0x558ef97770f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:27:45.154 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:27:45.161+0000 7f88993d7740 1 -- 172.21.15.52:0/2676737153 shutdown_connections 2024-05-27T07:27:45.155 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:27:45.161+0000 7f88993d7740 1 -- 172.21.15.52:0/2676737153 wait complete. 2024-05-27T07:27:45.155 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:27:45.161+0000 7f88993d7740 20 client.15948 trim_cache size 0 max 16384 2024-05-27T07:27:45.167 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:27:45.176+0000 7f88993d7740 1 Processor -- start 2024-05-27T07:27:45.167 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:27:45.177+0000 7f88993d7740 1 -- start start 2024-05-27T07:27:45.168 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:27:45.177+0000 7f88993d7740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x558ef93c8300 0x558ef9777720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:27:45.168 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:27:45.177+0000 7f88993d7740 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x558ef93ab2f0 0x558ef9797630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:27:45.168 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:27:45.177+0000 7f88993d7740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x558ef93b34d0 0x558ef9797b70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:27:45.168 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:27:45.177+0000 7f88993d7740 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x7f888c1b7830 con 0x558ef93ab2f0 2024-05-27T07:27:45.168 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:27:45.177+0000 7f88993d7740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f888c1b9370 con 0x558ef93b34d0 2024-05-27T07:27:45.168 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:27:45.177+0000 7f88993d7740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f888c1b7f00 con 0x558ef93c8300 2024-05-27T07:27:45.168 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:27:45.177+0000 7f8883fff640 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x558ef93ab2f0 0x558ef9797630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:27:45.168 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:27:45.177+0000 7f8883fff640 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x558ef93ab2f0 0x558ef9797630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.52:38554/0 (socket says 172.21.15.52:38554) 2024-05-27T07:27:45.168 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:27:45.177+0000 7f8883fff640 1 -- 172.21.15.52:0/3558734713 learned_addr learned my addr 172.21.15.52:0/3558734713 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-05-27T07:27:45.168 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:27:45.177+0000 7f888bfff640 1 --2- 172.21.15.52:0/3558734713 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x558ef93c8300 0x558ef9777720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:27:45.168 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:27:45.177+0000 7f88909e6640 1 --2- 172.21.15.52:0/3558734713 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x558ef93b34d0 0x558ef9797b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:27:45.168 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:27:45.177+0000 7f8883fff640 1 -- 172.21.15.52:0/3558734713 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x558ef93c8300 msgr2=0x558ef9777720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:27:45.168 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:27:45.177+0000 7f8883fff640 1 --2- 172.21.15.52:0/3558734713 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x558ef93c8300 0x558ef9777720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:27:45.169 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:27:45.177+0000 7f8883fff640 1 -- 172.21.15.52:0/3558734713 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x558ef93b34d0 msgr2=0x558ef9797b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:27:45.169 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:27:45.177+0000 7f8883fff640 1 --2- 172.21.15.52:0/3558734713 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x558ef93b34d0 0x558ef9797b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:27:45.169 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:27:45.177+0000 7f8883fff640 1 -- 172.21.15.52:0/3558734713 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x558ef9775160 con 0x558ef93ab2f0 2024-05-27T07:27:45.169 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:27:45.177+0000 7f88909e6640 1 --2- 172.21.15.52:0/3558734713 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x558ef93b34d0 0x558ef9797b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-27T07:27:45.169 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:27:45.177+0000 7f8883fff640 1 --2- 172.21.15.52:0/3558734713 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x558ef93ab2f0 0x558ef9797630 secure :-1 s=READY pgs=556 cs=0 l=1 rev1=1 crypto rx=0x7f8874009900 tx=0x7f8874005fd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=466e3a8143ce3b21 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:27:45.169 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:27:45.178+0000 7f8881ffb640 1 -- 172.21.15.52:0/3558734713 <== mon.0 v2:172.21.15.52:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8874035040 con 0x558ef93ab2f0 2024-05-27T07:27:45.169 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:27:45.178+0000 7f8881ffb640 1 -- 172.21.15.52:0/3558734713 <== mon.0 v2:172.21.15.52:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f8874056070 con 0x558ef93ab2f0 2024-05-27T07:27:45.169 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:27:45.178+0000 7f8881ffb640 1 -- 172.21.15.52:0/3558734713 <== mon.0 v2:172.21.15.52:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8874051380 con 0x558ef93ab2f0 2024-05-27T07:27:45.172 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:27:45.178+0000 7f88993d7740 1 -- 172.21.15.52:0/3558734713 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x558ef93ab2f0 msgr2=0x558ef9797630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:27:45.172 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:27:45.178+0000 7f88993d7740 1 --2- 172.21.15.52:0/3558734713 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x558ef93ab2f0 0x558ef9797630 secure :-1 s=READY pgs=556 cs=0 l=1 rev1=1 crypto rx=0x7f8874009900 tx=0x7f8874005fd0 comp rx=0 tx=0).stop 2024-05-27T07:27:45.172 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:27:45.179+0000 7f88993d7740 1 -- 172.21.15.52:0/3558734713 shutdown_connections 2024-05-27T07:27:45.173 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:27:45.179+0000 7f88993d7740 1 --2- 172.21.15.52:0/3558734713 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x558ef93b34d0 0x558ef9797b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:27:45.173 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:27:45.179+0000 7f88993d7740 1 --2- 172.21.15.52:0/3558734713 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x558ef93ab2f0 0x558ef9797630 unknown :-1 s=CLOSED pgs=556 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:27:45.173 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:27:45.179+0000 7f88993d7740 1 --2- 172.21.15.52:0/3558734713 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x558ef93c8300 0x558ef9777720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:27:45.173 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:27:45.179+0000 7f88993d7740 1 -- 172.21.15.52:0/3558734713 >> 172.21.15.52:0/3558734713 conn(0x558ef977b310 msgr2=0x558ef9777010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:27:45.173 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:27:45.179+0000 7f88993d7740 1 -- 172.21.15.52:0/3558734713 shutdown_connections 2024-05-27T07:27:45.173 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:27:45.179+0000 7f88993d7740 1 -- 172.21.15.52:0/3558734713 wait complete. 2024-05-27T07:27:45.173 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:27:45.179+0000 7f88993d7740 1 -- start start 2024-05-27T07:27:45.173 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:27:45.179+0000 7f88993d7740 1 Processor -- start 2024-05-27T07:27:45.173 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:27:45.179+0000 7f88993d7740 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x558ef93ab2f0 0x558ef9787820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:27:45.173 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:27:45.179+0000 7f88993d7740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x558ef93b34d0 0x558ef9787d80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:27:45.173 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:27:45.179+0000 7f88993d7740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x558ef93c8300 0x558ef9784c20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:27:45.173 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:27:45.179+0000 7f88993d7740 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x7f888c1b8ca0 con 0x558ef93ab2f0 2024-05-27T07:27:45.173 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:27:45.179+0000 7f88993d7740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f888c1b85d0 con 0x558ef93c8300 2024-05-27T07:27:45.174 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:27:45.179+0000 7f88993d7740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f888c1b5810 con 0x558ef93b34d0 2024-05-27T07:27:45.174 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:27:45.180+0000 7f88993d7740 10 client.0 _ll_register_callbacks cb 0x7f887c009470 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T07:27:45.174 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:27:45.180+0000 7f888bfff640 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x558ef93ab2f0 0x558ef9787820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:27:45.174 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:27:45.180+0000 7f888bfff640 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x558ef93ab2f0 0x558ef9787820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.52:38560/0 (socket says 172.21.15.52:38560) 2024-05-27T07:27:45.174 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:27:45.180+0000 7f888bfff640 1 -- 172.21.15.52:0/812117175 learned_addr learned my addr 172.21.15.52:0/812117175 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-05-27T07:27:45.174 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:27:45.180+0000 7f8883fff640 1 --2- 172.21.15.52:0/812117175 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x558ef93b34d0 0x558ef9787d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:27:45.174 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:27:45.180+0000 7f88909e6640 1 --2- 172.21.15.52:0/812117175 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x558ef93c8300 0x558ef9784c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:27:45.174 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:27:45.180+0000 7f888bfff640 1 -- 172.21.15.52:0/812117175 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x558ef93b34d0 msgr2=0x558ef9787d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:27:45.174 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:27:45.180+0000 7f888bfff640 1 --2- 172.21.15.52:0/812117175 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x558ef93b34d0 0x558ef9787d80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:27:45.174 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:27:45.180+0000 7f888bfff640 1 -- 172.21.15.52:0/812117175 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x558ef93c8300 msgr2=0x558ef9784c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:27:45.174 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:27:45.180+0000 7f888bfff640 1 --2- 172.21.15.52:0/812117175 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x558ef93c8300 0x558ef9784c20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:27:45.174 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:27:45.180+0000 7f888bfff640 1 -- 172.21.15.52:0/812117175 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({osdmap=0}) -- 0x558ef9777c60 con 0x558ef93ab2f0 2024-05-27T07:27:45.175 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:27:45.180+0000 7f88909e6640 1 --2- 172.21.15.52:0/812117175 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x558ef93c8300 0x558ef9784c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-27T07:27:45.175 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:27:45.180+0000 7f888bfff640 1 --2- 172.21.15.52:0/812117175 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x558ef93ab2f0 0x558ef9787820 secure :-1 s=READY pgs=557 cs=0 l=1 rev1=1 crypto rx=0x7f887c006b50 tx=0x7f887c021c50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d9200362a50012d3 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:27:45.175 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:27:45.180+0000 7f886b7fe640 10 client.0 ms_handle_connect on v2:172.21.15.52:3300/0 2024-05-27T07:27:45.175 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:27:45.180+0000 7f886b7fe640 1 -- 172.21.15.52:0/812117175 <== mon.0 v2:172.21.15.52:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f887c03a020 con 0x558ef93ab2f0 2024-05-27T07:27:45.175 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:27:45.180+0000 7f88993d7740 1 -- 172.21.15.52:0/812117175 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f8874056070 con 0x558ef93ab2f0 2024-05-27T07:27:45.175 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:27:45.180+0000 7f886b7fe640 1 -- 172.21.15.52:0/812117175 <== mon.0 v2:172.21.15.52:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 9379+0+0 (secure 0 0 0) 0x7f887c0295c0 con 0x558ef93ab2f0 2024-05-27T07:27:45.175 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:27:45.180+0000 7f886b7fe640 1 -- 172.21.15.52:0/812117175 <== mon.0 v2:172.21.15.52:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7f887c025040 con 0x558ef93ab2f0 2024-05-27T07:27:45.175 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:27:45.181+0000 7f88993d7740 10 client.15954 fetch_fsmap learned FSMap version 30 2024-05-27T07:27:45.175 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:27:45.181+0000 7f88993d7740 1 -- 172.21.15.52:0/812117175 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x558ef97f9cd0 con 0x558ef93ab2f0 2024-05-27T07:27:45.175 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:27:45.181+0000 7f886b7fe640 1 -- 172.21.15.52:0/812117175 <== mon.0 v2:172.21.15.52:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f8874056070 con 0x558ef93ab2f0 2024-05-27T07:27:45.175 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:27:45.181+0000 7f886b7fe640 1 -- 172.21.15.52:0/812117175 <== mon.0 v2:172.21.15.52:3300/0 5 ==== fsmap(e 30) ==== 8362+0+0 (secure 0 0 0) 0x7f887c020040 con 0x558ef93ab2f0 2024-05-27T07:27:45.175 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:27:45.181+0000 7f88993d7740 10 client.15954 fetch_fsmap finished waiting for FSMap version 30 2024-05-27T07:27:45.175 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:27:45.181+0000 7f88993d7740 10 client.15954 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-05-27T07:27:45.176 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:27:45.181+0000 7f88993d7740 20 client.15954 populate_metadata read hostname 'smithi052' 2024-05-27T07:27:45.176 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:27:45.181+0000 7f88993d7740 1 --2- 172.21.15.52:0/812117175 >> [v2:172.21.15.105:6838/500348151,v1:172.21.15.105:6839/500348151] conn(0x558ef93d41b0 0x558ef97fc4d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:27:45.176 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:27:45.181+0000 7f88993d7740 4 client.15954 mds_command: new command op to 24527 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-05-27T07:27:45.176 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:27:45.181+0000 7f88993d7740 1 -- 172.21.15.52:0/812117175 --> [v2:172.21.15.105:6838/500348151,v1:172.21.15.105:6839/500348151] -- command(tid 0: {"prefix": "flush journal"}) -- 0x558ef96e3360 con 0x558ef93d41b0 2024-05-27T07:27:45.176 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:27:45.181+0000 7f8883fff640 1 --2- 172.21.15.52:0/812117175 >> [v2:172.21.15.105:6838/500348151,v1:172.21.15.105:6839/500348151] conn(0x558ef93d41b0 0x558ef97fc4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:27:45.176 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:27:45.182+0000 7f886b7fe640 10 client.15954 ms_handle_connect on v2:172.21.15.105:6838/500348151 2024-05-27T07:27:45.176 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:27:45.182+0000 7f8883fff640 1 --2- 172.21.15.52:0/812117175 >> [v2:172.21.15.105:6838/500348151,v1:172.21.15.105:6839/500348151] conn(0x558ef93d41b0 0x558ef97fc4d0 crc :-1 s=READY pgs=74 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=a6bbefe25e17bc3 server_cookie=f045ccf82c13f048 in_seq=0 out_seq=0 2024-05-27T07:27:45.177 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:27:45.186+0000 7f886b7fe640 1 -- 172.21.15.52:0/812117175 <== mds.0 v2:172.21.15.105:6838/500348151 1 ==== command_reply(tid 0: 0 ) ==== 8+0+25 (crc 0 0 0) 0x558ef96e3360 con 0x558ef93d41b0 2024-05-27T07:27:45.177 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:27:45.186+0000 7f886b7fe640 10 client.15954 handle_command_reply: tid=0 2024-05-27T07:27:45.177 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:27:45.186+0000 7f88993d7740 1 client.15954 shutdown 2024-05-27T07:27:45.177 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:27:45.186+0000 7f88993d7740 1 -- 172.21.15.52:0/812117175 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x558ef93ab2f0 msgr2=0x558ef9787820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:27:45.177 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:27:45.186+0000 7f88993d7740 1 --2- 172.21.15.52:0/812117175 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x558ef93ab2f0 0x558ef9787820 secure :-1 s=READY pgs=557 cs=0 l=1 rev1=1 crypto rx=0x7f887c006b50 tx=0x7f887c021c50 comp rx=0 tx=0).stop 2024-05-27T07:27:45.178 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:27:45.186+0000 7f88993d7740 1 -- 172.21.15.52:0/812117175 shutdown_connections 2024-05-27T07:27:45.178 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:27:45.186+0000 7f88993d7740 1 --2- 172.21.15.52:0/812117175 >> [v2:172.21.15.105:6838/500348151,v1:172.21.15.105:6839/500348151] conn(0x558ef93d41b0 0x558ef97fc4d0 crc :-1 s=READY pgs=74 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:27:45.178 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:27:45.186+0000 7f88993d7740 1 --2- 172.21.15.52:0/812117175 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x558ef93c8300 0x558ef9784c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:27:45.178 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:27:45.186+0000 7f886b7fe640 0 client.15954 ms_handle_reset on v2:172.21.15.105:6838/500348151 2024-05-27T07:27:45.178 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:27:45.186+0000 7f88993d7740 1 --2- 172.21.15.52:0/812117175 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x558ef93b34d0 0x558ef9787d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:27:45.178 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:27:45.186+0000 7f88993d7740 1 --2- 172.21.15.52:0/812117175 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x558ef93ab2f0 0x558ef9787820 unknown :-1 s=CLOSED pgs=557 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:27:45.178 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:27:45.186+0000 7f88993d7740 1 -- 172.21.15.52:0/812117175 >> 172.21.15.52:0/812117175 conn(0x558ef977b310 msgr2=0x558ef9781040 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:27:45.178 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:27:45.186+0000 7f88993d7740 1 -- 172.21.15.52:0/812117175 shutdown_connections 2024-05-27T07:27:45.178 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:27:45.186+0000 7f88993d7740 1 -- 172.21.15.52:0/812117175 wait complete. 2024-05-27T07:27:45.178 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:27:45.186+0000 7f88993d7740 20 client.15954 trim_cache size 0 max 16384 2024-05-27T07:27:45.178 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stdout:{ 2024-05-27T07:27:45.179 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stdout: "return_code": 0 2024-05-27T07:27:45.179 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stdout:} 2024-05-27T07:27:45.179 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:27:45.187+0000 7f8892449640 1 -- 172.21.15.52:0/2186669894 >> [v2:172.21.15.52:6800/1822318384,v1:172.21.15.52:6801/1822318384] conn(0x7f88780b29c0 msgr2=0x7f88780b4ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:27:45.179 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:27:45.187+0000 7f8892449640 1 --2- 172.21.15.52:0/2186669894 >> [v2:172.21.15.52:6800/1822318384,v1:172.21.15.52:6801/1822318384] conn(0x7f88780b29c0 0x7f88780b4ed0 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x558ef9776780 tx=0x7f887c009110 comp rx=0 tx=0).stop 2024-05-27T07:27:45.179 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:27:45.187+0000 7f8892449640 1 -- 172.21.15.52:0/2186669894 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f888c109030 msgr2=0x7f888c1a5780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:27:45.179 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:27:45.187+0000 7f8892449640 1 --2- 172.21.15.52:0/2186669894 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f888c109030 0x7f888c1a5780 secure :-1 s=READY pgs=553 cs=0 l=1 rev1=1 crypto rx=0x7f888400cad0 tx=0x7f888400ced0 comp rx=0 tx=0).stop 2024-05-27T07:27:45.180 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:27:45.187+0000 7f8892449640 1 -- 172.21.15.52:0/2186669894 shutdown_connections 2024-05-27T07:27:45.180 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:27:45.187+0000 7f8892449640 1 --2- 172.21.15.52:0/2186669894 >> [v2:172.21.15.52:6800/1822318384,v1:172.21.15.52:6801/1822318384] conn(0x7f88780b29c0 0x7f88780b4ed0 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:27:45.180 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:27:45.187+0000 7f8892449640 1 --2- 172.21.15.52:0/2186669894 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f888c109030 0x7f888c1a5780 unknown :-1 s=CLOSED pgs=553 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:27:45.180 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:27:45.187+0000 7f8892449640 1 --2- 172.21.15.52:0/2186669894 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f888c108b90 0x7f888c1a14c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:27:45.180 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:27:45.187+0000 7f8892449640 1 --2- 172.21.15.52:0/2186669894 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f888c1032e0 0x7f888c1a0f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:27:45.180 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:27:45.187+0000 7f8892449640 1 -- 172.21.15.52:0/2186669894 >> 172.21.15.52:0/2186669894 conn(0x7f888c0782d0 msgr2=0x7f888c1012b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:27:45.180 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:27:45.187+0000 7f8892449640 1 -- 172.21.15.52:0/2186669894 shutdown_connections 2024-05-27T07:27:45.180 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:27:45.187+0000 7f8892449640 1 -- 172.21.15.52:0/2186669894 wait complete. 2024-05-27T07:27:45.869 INFO:tasks.workunit.client.0.smithi052.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_sync_io.c -o test_sync_io 2024-05-27T07:27:45.926 INFO:tasks.workunit.client.0.smithi052.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_short_dio_read.c -o test_short_dio_read 2024-05-27T07:27:45.968 INFO:tasks.workunit.client.0.smithi052.stdout:make[1]: Leaving directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/direct_io' 2024-05-27T07:27:45.970 INFO:tasks.workunit.client.0.smithi052.stdout:make[1]: Entering directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/fs' 2024-05-27T07:27:45.970 INFO:tasks.workunit.client.0.smithi052.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_o_trunc.c -o test_o_trunc 2024-05-27T07:27:46.010 INFO:tasks.workunit.client.0.smithi052.stdout:make[1]: Leaving directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/fs' 2024-05-27T07:27:46.014 DEBUG:teuthology.orchestra.run.smithi052:> set -ex 2024-05-27T07:27:46.014 DEBUG:teuthology.orchestra.run.smithi052:> dd if=/home/ubuntu/cephtest/workunits.list.client.0 of=/dev/stdout 2024-05-27T07:27:46.071 INFO:tasks.workunit:Running workunits matching suites/blogbench.sh on client.0... 2024-05-27T07:27:46.072 INFO:tasks.workunit:Running workunit suites/blogbench.sh... 2024-05-27T07:27:46.072 DEBUG:teuthology.orchestra.run.smithi052:workunit test suites/blogbench.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=38f9d900c83c63799fbdbe61acc9a11b0d3554a6 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/suites/blogbench.sh 2024-05-27T07:27:46.134 INFO:tasks.workunit.client.0.smithi052.stdout:getting blogbench 2024-05-27T07:27:46.134 INFO:tasks.workunit.client.0.smithi052.stderr:+ echo 'getting blogbench' 2024-05-27T07:27:46.135 INFO:tasks.workunit.client.0.smithi052.stderr:+ wget http://download.ceph.com/qa/blogbench-1.0.tar.bz2 2024-05-27T07:27:46.162 INFO:tasks.workunit.client.0.smithi052.stderr:--2024-05-27 07:27:46-- http://download.ceph.com/qa/blogbench-1.0.tar.bz2 2024-05-27T07:27:46.163 INFO:tasks.workunit.client.0.smithi052.stderr:Resolving download.ceph.com (download.ceph.com)... 158.69.68.124, 2607:5300:201:2000::3:58a1 2024-05-27T07:27:46.188 INFO:tasks.workunit.client.0.smithi052.stderr:Connecting to download.ceph.com (download.ceph.com)|158.69.68.124|:80... connected. 2024-05-27T07:27:46.210 INFO:tasks.workunit.client.0.smithi052.stderr:HTTP request sent, awaiting response... 200 OK 2024-05-27T07:27:46.210 INFO:tasks.workunit.client.0.smithi052.stderr:Length: 103355 (101K) [application/octet-stream] 2024-05-27T07:27:46.211 INFO:tasks.workunit.client.0.smithi052.stderr:Saving to: ‘blogbench-1.0.tar.bz2’ 2024-05-27T07:27:46.214 INFO:tasks.workunit.client.0.smithi052.stderr: 2024-05-27T07:27:46.252 INFO:tasks.workunit.client.0.smithi052.stderr: 0K .......... .......... .......... .......... .......... 49% 1.23M 0s 2024-05-27T07:27:46.278 INFO:tasks.workunit.client.0.smithi052.stderr: 50K .......... .......... .......... .......... .......... 99% 1.83M 0s 2024-05-27T07:27:46.278 INFO:tasks.workunit.client.0.smithi052.stderr: 100K 100% 1.74T=0.07s 2024-05-27T07:27:46.278 INFO:tasks.workunit.client.0.smithi052.stderr: 2024-05-27T07:27:46.278 INFO:tasks.workunit.client.0.smithi052.stderr:2024-05-27 07:27:46 (1.49 MB/s) - ‘blogbench-1.0.tar.bz2’ saved [103355/103355] 2024-05-27T07:27:46.279 INFO:tasks.workunit.client.0.smithi052.stderr: 2024-05-27T07:27:46.279 INFO:tasks.workunit.client.0.smithi052.stderr:+ tar -xvf blogbench-1.0.tar.bz2 2024-05-27T07:27:46.309 INFO:tasks.workunit.client.0.smithi052.stdout:blogbench-1.0/ 2024-05-27T07:27:46.310 INFO:tasks.workunit.client.0.smithi052.stdout:blogbench-1.0/m4/ 2024-05-27T07:27:46.311 INFO:tasks.workunit.client.0.smithi052.stdout:blogbench-1.0/m4/acx_pthread.m4 2024-05-27T07:27:46.312 INFO:tasks.workunit.client.0.smithi052.stdout:blogbench-1.0/man/ 2024-05-27T07:27:46.313 INFO:tasks.workunit.client.0.smithi052.stdout:blogbench-1.0/man/blogbench.8 2024-05-27T07:27:46.314 INFO:tasks.workunit.client.0.smithi052.stdout:blogbench-1.0/man/Makefile.am 2024-05-27T07:27:46.315 INFO:tasks.workunit.client.0.smithi052.stdout:blogbench-1.0/man/Makefile.in 2024-05-27T07:27:46.316 INFO:tasks.workunit.client.0.smithi052.stdout:blogbench-1.0/src/ 2024-05-27T07:27:46.317 INFO:tasks.workunit.client.0.smithi052.stdout:blogbench-1.0/src/mysnprintf.c 2024-05-27T07:27:46.318 INFO:tasks.workunit.client.0.smithi052.stdout:blogbench-1.0/src/mysnprintf.h 2024-05-27T07:27:46.319 INFO:tasks.workunit.client.0.smithi052.stdout:blogbench-1.0/src/blogbench.c 2024-05-27T07:27:46.320 INFO:tasks.workunit.client.0.smithi052.stdout:blogbench-1.0/src/blogbench.h 2024-05-27T07:27:46.321 INFO:tasks.workunit.client.0.smithi052.stdout:blogbench-1.0/src/fakesnprintf.c 2024-05-27T07:27:46.322 INFO:tasks.workunit.client.0.smithi052.stdout:blogbench-1.0/src/fakesnprintf.h 2024-05-27T07:27:46.323 INFO:tasks.workunit.client.0.smithi052.stdout:blogbench-1.0/src/blogbench_p.h 2024-05-27T07:27:46.324 INFO:tasks.workunit.client.0.smithi052.stdout:blogbench-1.0/src/bsd-getopt_long.c 2024-05-27T07:27:46.326 INFO:tasks.workunit.client.0.smithi052.stdout:blogbench-1.0/src/bsd-getopt_long.h 2024-05-27T07:27:46.327 INFO:tasks.workunit.client.0.smithi052.stdout:blogbench-1.0/src/Makefile.am 2024-05-27T07:27:46.328 INFO:tasks.workunit.client.0.smithi052.stdout:blogbench-1.0/src/Makefile.in 2024-05-27T07:27:46.329 INFO:tasks.workunit.client.0.smithi052.stdout:blogbench-1.0/src/process.c 2024-05-27T07:27:46.330 INFO:tasks.workunit.client.0.smithi052.stdout:blogbench-1.0/src/globals.h 2024-05-27T07:27:46.331 INFO:tasks.workunit.client.0.smithi052.stdout:blogbench-1.0/src/commenter.c 2024-05-27T07:27:46.332 INFO:tasks.workunit.client.0.smithi052.stdout:blogbench-1.0/src/rewriter.c 2024-05-27T07:27:46.333 INFO:tasks.workunit.client.0.smithi052.stdout:blogbench-1.0/src/reader.c 2024-05-27T07:27:46.334 INFO:tasks.workunit.client.0.smithi052.stdout:blogbench-1.0/src/writer.c 2024-05-27T07:27:46.335 INFO:tasks.workunit.client.0.smithi052.stdout:blogbench-1.0/src/helpers.c 2024-05-27T07:27:46.336 INFO:tasks.workunit.client.0.smithi052.stdout:blogbench-1.0/NEWS 2024-05-27T07:27:46.337 INFO:tasks.workunit.client.0.smithi052.stdout:blogbench-1.0/compile 2024-05-27T07:27:46.338 INFO:tasks.workunit.client.0.smithi052.stdout:blogbench-1.0/depcomp 2024-05-27T07:27:46.339 INFO:tasks.workunit.client.0.smithi052.stdout:blogbench-1.0/aclocal.m4 2024-05-27T07:27:46.340 INFO:tasks.workunit.client.0.smithi052.stdout:blogbench-1.0/README 2024-05-27T07:27:46.342 INFO:tasks.workunit.client.0.smithi052.stdout:blogbench-1.0/configure 2024-05-27T07:27:46.345 INFO:tasks.workunit.client.0.smithi052.stdout:blogbench-1.0/configure.ac 2024-05-27T07:27:46.347 INFO:tasks.workunit.client.0.smithi052.stdout:blogbench-1.0/config.guess 2024-05-27T07:27:46.348 INFO:tasks.workunit.client.0.smithi052.stdout:blogbench-1.0/install-sh 2024-05-27T07:27:46.349 INFO:tasks.workunit.client.0.smithi052.stdout:blogbench-1.0/config.sub 2024-05-27T07:27:46.350 INFO:tasks.workunit.client.0.smithi052.stdout:blogbench-1.0/missing 2024-05-27T07:27:46.351 INFO:tasks.workunit.client.0.smithi052.stdout:blogbench-1.0/mkinstalldirs 2024-05-27T07:27:46.352 INFO:tasks.workunit.client.0.smithi052.stdout:blogbench-1.0/Makefile.am 2024-05-27T07:27:46.353 INFO:tasks.workunit.client.0.smithi052.stdout:blogbench-1.0/Makefile.in 2024-05-27T07:27:46.354 INFO:tasks.workunit.client.0.smithi052.stdout:blogbench-1.0/config.h.in 2024-05-27T07:27:46.355 INFO:tasks.workunit.client.0.smithi052.stdout:blogbench-1.0/AUTHORS 2024-05-27T07:27:46.356 INFO:tasks.workunit.client.0.smithi052.stdout:blogbench-1.0/INSTALL 2024-05-27T07:27:46.357 INFO:tasks.workunit.client.0.smithi052.stdout:blogbench-1.0/ChangeLog 2024-05-27T07:27:46.358 INFO:tasks.workunit.client.0.smithi052.stdout:blogbench-1.0/COPYING 2024-05-27T07:27:46.359 INFO:tasks.workunit.client.0.smithi052.stderr:+ cd blogbench-1.0/ 2024-05-27T07:27:46.359 INFO:tasks.workunit.client.0.smithi052.stdout:making blogbench 2024-05-27T07:27:46.360 INFO:tasks.workunit.client.0.smithi052.stderr:+ echo 'making blogbench' 2024-05-27T07:27:46.360 INFO:tasks.workunit.client.0.smithi052.stderr:+ ./configure 2024-05-27T07:27:46.439 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:27:46 smithi105 ceph-mon[35977]: pgmap v57: 129 pgs: 129 active+clean; 1.4 MiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 9 op/s 2024-05-27T07:27:46.465 INFO:tasks.workunit.client.0.smithi052.stdout:checking for a BSD-compatible install... /usr/bin/install -c 2024-05-27T07:27:46.479 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:27:46 smithi052 ceph-mon[31955]: pgmap v57: 129 pgs: 129 active+clean; 1.4 MiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 9 op/s 2024-05-27T07:27:46.601 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:27:46 smithi173 ceph-mon[37000]: pgmap v57: 129 pgs: 129 active+clean; 1.4 MiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 9 op/s 2024-05-27T07:27:47.474 INFO:tasks.workunit.client.0.smithi052.stdout:checking whether build environment is sane... yes 2024-05-27T07:27:47.484 INFO:tasks.workunit.client.0.smithi052.stdout:checking for gawk... gawk 2024-05-27T07:27:47.502 INFO:tasks.workunit.client.0.smithi052.stdout:checking whether make sets $(MAKE)... yes 2024-05-27T07:27:47.512 INFO:tasks.workunit.client.0.smithi052.stdout:checking for gcc... gcc 2024-05-27T07:27:47.567 INFO:tasks.workunit.client.0.smithi052.stdout:checking for C compiler default output file name... a.out 2024-05-27T07:27:47.569 INFO:tasks.workunit.client.0.smithi052.stdout:checking whether the C compiler works... yes 2024-05-27T07:27:47.572 INFO:tasks.workunit.client.0.smithi052.stdout:checking whether we are cross compiling... no 2024-05-27T07:27:47.610 INFO:tasks.workunit.client.0.smithi052.stdout:checking for suffix of executables... 2024-05-27T07:27:47.647 INFO:tasks.workunit.client.0.smithi052.stdout:checking for suffix of object files... o 2024-05-27T07:27:47.683 INFO:tasks.workunit.client.0.smithi052.stdout:checking whether we are using the GNU C compiler... yes 2024-05-27T07:27:47.719 INFO:tasks.workunit.client.0.smithi052.stdout:checking whether gcc accepts -g... yes 2024-05-27T07:27:47.790 INFO:tasks.workunit.client.0.smithi052.stdout:checking for gcc option to accept ANSI C... none needed 2024-05-27T07:27:47.838 INFO:tasks.workunit.client.0.smithi052.stdout:checking for style of include used by make... GNU 2024-05-27T07:27:47.938 INFO:tasks.workunit.client.0.smithi052.stdout:checking dependency style of gcc... gcc3 2024-05-27T07:27:47.938 INFO:tasks.workunit.client.0.smithi052.stdout:checking for a BSD-compatible install... /usr/bin/install -c 2024-05-27T07:27:47.994 INFO:tasks.workunit.client.0.smithi052.stdout:checking how to run the C preprocessor... gcc -E 2024-05-27T07:27:48.038 INFO:tasks.workunit.client.0.smithi052.stdout:checking for egrep... grep -E 2024-05-27T07:27:48.047 INFO:tasks.workunit.client.0.smithi052.stdout:checking for AIX... no 2024-05-27T07:27:48.114 INFO:tasks.workunit.client.0.smithi052.stdout:checking for library containing strerror... none required 2024-05-27T07:27:48.331 INFO:tasks.workunit.client.0.smithi052.stdout:checking for ANSI C header files... yes 2024-05-27T07:27:48.369 INFO:tasks.workunit.client.0.smithi052.stdout:checking whether time.h and sys/time.h may both be included... yes 2024-05-27T07:27:48.417 INFO:tasks.workunit.client.0.smithi052.stdout:checking for sys/types.h... yes 2024-05-27T07:27:48.439 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:27:48 smithi105 ceph-mon[35977]: pgmap v58: 129 pgs: 129 active+clean; 1.4 MiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 9 op/s 2024-05-27T07:27:48.467 INFO:tasks.workunit.client.0.smithi052.stdout:checking for sys/stat.h... yes 2024-05-27T07:27:48.478 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:27:48 smithi052 ceph-mon[31955]: pgmap v58: 129 pgs: 129 active+clean; 1.4 MiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 9 op/s 2024-05-27T07:27:48.516 INFO:tasks.workunit.client.0.smithi052.stdout:checking for stdlib.h... yes 2024-05-27T07:27:48.567 INFO:tasks.workunit.client.0.smithi052.stdout:checking for string.h... yes 2024-05-27T07:27:48.601 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:27:48 smithi173 ceph-mon[37000]: pgmap v58: 129 pgs: 129 active+clean; 1.4 MiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 9 op/s 2024-05-27T07:27:48.625 INFO:tasks.workunit.client.0.smithi052.stdout:checking for memory.h... yes 2024-05-27T07:27:48.681 INFO:tasks.workunit.client.0.smithi052.stdout:checking for strings.h... yes 2024-05-27T07:27:48.738 INFO:tasks.workunit.client.0.smithi052.stdout:checking for inttypes.h... yes 2024-05-27T07:27:48.793 INFO:tasks.workunit.client.0.smithi052.stdout:checking for stdint.h... yes 2024-05-27T07:27:48.847 INFO:tasks.workunit.client.0.smithi052.stdout:checking for unistd.h... yes 2024-05-27T07:27:48.860 INFO:tasks.workunit.client.0.smithi052.stdout:checking for memory.h... (cached) yes 2024-05-27T07:27:48.865 INFO:tasks.workunit.client.0.smithi052.stdout:checking for strings.h... (cached) yes 2024-05-27T07:27:48.870 INFO:tasks.workunit.client.0.smithi052.stdout:checking for unistd.h... (cached) yes 2024-05-27T07:27:48.917 INFO:tasks.workunit.client.0.smithi052.stdout:checking locale.h usability... yes 2024-05-27T07:27:48.938 INFO:tasks.workunit.client.0.smithi052.stdout:checking locale.h presence... yes 2024-05-27T07:27:48.939 INFO:tasks.workunit.client.0.smithi052.stdout:checking for locale.h... yes 2024-05-27T07:27:48.954 INFO:tasks.workunit.client.0.smithi052.stdout:checking for stdlib.h... (cached) yes 2024-05-27T07:27:48.959 INFO:tasks.workunit.client.0.smithi052.stdout:checking for string.h... (cached) yes 2024-05-27T07:27:49.008 INFO:tasks.workunit.client.0.smithi052.stdout:checking sys/socket.h usability... yes 2024-05-27T07:27:49.031 INFO:tasks.workunit.client.0.smithi052.stdout:checking sys/socket.h presence... yes 2024-05-27T07:27:49.032 INFO:tasks.workunit.client.0.smithi052.stdout:checking for sys/socket.h... yes 2024-05-27T07:27:49.089 INFO:tasks.workunit.client.0.smithi052.stdout:checking sys/time.h usability... yes 2024-05-27T07:27:49.108 INFO:tasks.workunit.client.0.smithi052.stdout:checking sys/time.h presence... yes 2024-05-27T07:27:49.109 INFO:tasks.workunit.client.0.smithi052.stdout:checking for sys/time.h... yes 2024-05-27T07:27:49.166 INFO:tasks.workunit.client.0.smithi052.stdout:checking fcntl.h usability... yes 2024-05-27T07:27:49.187 INFO:tasks.workunit.client.0.smithi052.stdout:checking fcntl.h presence... yes 2024-05-27T07:27:49.188 INFO:tasks.workunit.client.0.smithi052.stdout:checking for fcntl.h... yes 2024-05-27T07:27:49.254 INFO:tasks.workunit.client.0.smithi052.stdout:checking sys/fcntl.h usability... yes 2024-05-27T07:27:49.275 INFO:tasks.workunit.client.0.smithi052.stdout:checking sys/fcntl.h presence... yes 2024-05-27T07:27:49.276 INFO:tasks.workunit.client.0.smithi052.stdout:checking for sys/fcntl.h... yes 2024-05-27T07:27:49.362 INFO:tasks.workunit.client.0.smithi052.stdout:checking sys/param.h usability... yes 2024-05-27T07:27:49.386 INFO:tasks.workunit.client.0.smithi052.stdout:checking sys/param.h presence... yes 2024-05-27T07:27:49.386 INFO:tasks.workunit.client.0.smithi052.stdout:checking for sys/param.h... yes 2024-05-27T07:27:49.439 INFO:tasks.workunit.client.0.smithi052.stdout:checking ioctl.h usability... no 2024-05-27T07:27:49.460 INFO:tasks.workunit.client.0.smithi052.stdout:checking ioctl.h presence... no 2024-05-27T07:27:49.461 INFO:tasks.workunit.client.0.smithi052.stdout:checking for ioctl.h... no 2024-05-27T07:27:49.519 INFO:tasks.workunit.client.0.smithi052.stdout:checking sys/ioctl.h usability... yes 2024-05-27T07:27:49.540 INFO:tasks.workunit.client.0.smithi052.stdout:checking sys/ioctl.h presence... yes 2024-05-27T07:27:49.540 INFO:tasks.workunit.client.0.smithi052.stdout:checking for sys/ioctl.h... yes 2024-05-27T07:27:49.594 INFO:tasks.workunit.client.0.smithi052.stdout:checking return type of signal handlers... void 2024-05-27T07:27:49.640 INFO:tasks.workunit.client.0.smithi052.stdout:checking for size_t... yes 2024-05-27T07:27:49.680 INFO:tasks.workunit.client.0.smithi052.stdout:checking for dev_t... yes 2024-05-27T07:27:49.726 INFO:tasks.workunit.client.0.smithi052.stdout:checking for short... yes 2024-05-27T07:27:49.807 INFO:tasks.workunit.client.0.smithi052.stdout:checking size of short... 2 2024-05-27T07:27:49.853 INFO:tasks.workunit.client.0.smithi052.stdout:checking for int... yes 2024-05-27T07:27:49.923 INFO:tasks.workunit.client.0.smithi052.stdout:checking size of int... 4 2024-05-27T07:27:49.970 INFO:tasks.workunit.client.0.smithi052.stdout:checking for long... yes 2024-05-27T07:27:50.041 INFO:tasks.workunit.client.0.smithi052.stdout:checking size of long... 8 2024-05-27T07:27:50.088 INFO:tasks.workunit.client.0.smithi052.stdout:checking for long long... yes 2024-05-27T07:27:50.160 INFO:tasks.workunit.client.0.smithi052.stdout:checking size of long long... 8 2024-05-27T07:27:50.205 INFO:tasks.workunit.client.0.smithi052.stdout:checking for ssize_t... yes 2024-05-27T07:27:50.246 INFO:tasks.workunit.client.0.smithi052.stdout:checking for sig_atomic_t... yes 2024-05-27T07:27:50.298 INFO:tasks.workunit.client.0.smithi052.stdout:checking whether gcc needs -traditional... no 2024-05-27T07:27:50.335 INFO:tasks.workunit.client.0.smithi052.stdout:checking for an ANSI C-conforming const... yes 2024-05-27T07:27:50.360 INFO:tasks.workunit.client.0.smithi052.stdout:checking for inline... inline 2024-05-27T07:27:50.427 INFO:tasks.workunit.client.0.smithi052.stdout:checking for working alloca.h... yes 2024-05-27T07:27:50.439 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:27:50 smithi105 ceph-mon[35977]: pgmap v59: 129 pgs: 129 active+clean; 1.8 MiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 37 KiB/s wr, 15 op/s 2024-05-27T07:27:50.479 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:27:50 smithi052 ceph-mon[31955]: pgmap v59: 129 pgs: 129 active+clean; 1.8 MiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 37 KiB/s wr, 15 op/s 2024-05-27T07:27:50.482 INFO:tasks.workunit.client.0.smithi052.stdout:checking for alloca... yes 2024-05-27T07:27:50.556 INFO:tasks.workunit.client.0.smithi052.stdout:checking for working memcmp... yes 2024-05-27T07:27:50.602 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:27:50 smithi173 ceph-mon[37000]: pgmap v59: 129 pgs: 129 active+clean; 1.8 MiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 37 KiB/s wr, 15 op/s 2024-05-27T07:27:50.622 INFO:tasks.workunit.client.0.smithi052.stdout:checking for strftime... yes 2024-05-27T07:27:50.698 INFO:tasks.workunit.client.0.smithi052.stdout:checking whether lstat dereferences a symlink specified with a trailing slash... yes 2024-05-27T07:27:50.774 INFO:tasks.workunit.client.0.smithi052.stdout:checking whether stat accepts an empty string... no 2024-05-27T07:27:50.843 INFO:tasks.workunit.client.0.smithi052.stdout:checking for vprintf... yes 2024-05-27T07:27:50.908 INFO:tasks.workunit.client.0.smithi052.stdout:checking for _doprnt... no 2024-05-27T07:27:50.930 INFO:tasks.workunit.client.0.smithi052.stdout:checking build system type... x86_64-unknown-linux-gnu 2024-05-27T07:27:50.942 INFO:tasks.workunit.client.0.smithi052.stdout:checking host system type... x86_64-unknown-linux-gnu 2024-05-27T07:27:51.007 INFO:tasks.workunit.client.0.smithi052.stdout:checking for the pthreads library -lpthreads... no 2024-05-27T07:27:51.067 INFO:tasks.workunit.client.0.smithi052.stdout:checking whether pthreads work without any flags... yes 2024-05-27T07:27:51.117 INFO:tasks.workunit.client.0.smithi052.stdout:checking for joinable pthread attribute... PTHREAD_CREATE_JOINABLE 2024-05-27T07:27:51.118 INFO:tasks.workunit.client.0.smithi052.stdout:checking if more special flags are required for pthreads... no 2024-05-27T07:27:51.118 INFO:tasks.workunit.client.0.smithi052.stdout:checking for cc_r... gcc 2024-05-27T07:27:51.195 INFO:tasks.workunit.client.0.smithi052.stdout:checking for setrlimit... yes 2024-05-27T07:27:51.260 INFO:tasks.workunit.client.0.smithi052.stdout:checking for setproctitle... no 2024-05-27T07:27:51.320 INFO:tasks.workunit.client.0.smithi052.stdout:checking for getopt_long... yes 2024-05-27T07:27:51.382 INFO:tasks.workunit.client.0.smithi052.stdout:checking for strtoull... yes 2024-05-27T07:27:51.445 INFO:tasks.workunit.client.0.smithi052.stdout:checking for strtoq... yes 2024-05-27T07:27:51.509 INFO:tasks.workunit.client.0.smithi052.stdout:checking for snprintf... yes 2024-05-27T07:27:51.577 INFO:tasks.workunit.client.0.smithi052.stdout:checking for vsnprintf... yes 2024-05-27T07:27:51.690 INFO:tasks.workunit.client.0.smithi052.stdout:checking for vfprintf... yes 2024-05-27T07:27:51.806 INFO:tasks.workunit.client.0.smithi052.stdout:checking for gethostname... yes 2024-05-27T07:27:51.885 INFO:tasks.workunit.client.0.smithi052.stdout:checking for setlocale... yes 2024-05-27T07:27:51.948 INFO:tasks.workunit.client.0.smithi052.stdout:checking for sync... yes 2024-05-27T07:27:52.029 INFO:tasks.workunit.client.0.smithi052.stdout:checking whether snprintf is C99 conformant... done 2024-05-27T07:27:52.047 INFO:tasks.workunit.client.0.smithi052.stdout:configure: creating ./config.status 2024-05-27T07:27:52.231 INFO:tasks.workunit.client.0.smithi052.stdout:config.status: creating Makefile 2024-05-27T07:27:52.249 INFO:tasks.workunit.client.0.smithi052.stdout:config.status: creating src/Makefile 2024-05-27T07:27:52.270 INFO:tasks.workunit.client.0.smithi052.stdout:config.status: creating man/Makefile 2024-05-27T07:27:52.281 INFO:tasks.workunit.client.0.smithi052.stdout:config.status: creating config.h 2024-05-27T07:27:52.334 INFO:tasks.workunit.client.0.smithi052.stdout:config.status: executing depfiles commands 2024-05-27T07:27:52.439 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:27:52 smithi105 ceph-mon[35977]: pgmap v60: 129 pgs: 129 active+clean; 1.8 MiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 37 KiB/s wr, 11 op/s 2024-05-27T07:27:52.470 INFO:tasks.workunit.client.0.smithi052.stderr:+ make 2024-05-27T07:27:52.478 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:27:52 smithi052 ceph-mon[31955]: pgmap v60: 129 pgs: 129 active+clean; 1.8 MiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 37 KiB/s wr, 11 op/s 2024-05-27T07:27:52.480 INFO:tasks.workunit.client.0.smithi052.stdout:make all-recursive 2024-05-27T07:27:52.482 INFO:tasks.workunit.client.0.smithi052.stdout:make[1]: Entering directory '/home/ubuntu/cephtest/mnt.0/client.0/tmp/blogbench-1.0' 2024-05-27T07:27:52.485 INFO:tasks.workunit.client.0.smithi052.stdout:Making all in src 2024-05-27T07:27:52.491 INFO:tasks.workunit.client.0.smithi052.stdout:make[2]: Entering directory '/home/ubuntu/cephtest/mnt.0/client.0/tmp/blogbench-1.0/src' 2024-05-27T07:27:52.491 INFO:tasks.workunit.client.0.smithi052.stdout:if gcc -DHAVE_CONFIG_H -I. -I. -I.. -g -O2 -MT blogbench.o -MD -MP -MF ".deps/blogbench.Tpo" -c -o blogbench.o blogbench.c; \ 2024-05-27T07:27:52.491 INFO:tasks.workunit.client.0.smithi052.stdout:then mv -f ".deps/blogbench.Tpo" ".deps/blogbench.Po"; else rm -f ".deps/blogbench.Tpo"; exit 1; fi 2024-05-27T07:27:52.576 INFO:tasks.workunit.client.0.smithi052.stdout:if gcc -DHAVE_CONFIG_H -I. -I. -I.. -g -O2 -MT process.o -MD -MP -MF ".deps/process.Tpo" -c -o process.o process.c; \ 2024-05-27T07:27:52.576 INFO:tasks.workunit.client.0.smithi052.stdout:then mv -f ".deps/process.Tpo" ".deps/process.Po"; else rm -f ".deps/process.Tpo"; exit 1; fi 2024-05-27T07:27:52.601 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:27:52 smithi173 ceph-mon[37000]: pgmap v60: 129 pgs: 129 active+clean; 1.8 MiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 37 KiB/s wr, 11 op/s 2024-05-27T07:27:52.642 INFO:tasks.workunit.client.0.smithi052.stdout:if gcc -DHAVE_CONFIG_H -I. -I. -I.. -g -O2 -MT writer.o -MD -MP -MF ".deps/writer.Tpo" -c -o writer.o writer.c; \ 2024-05-27T07:27:52.643 INFO:tasks.workunit.client.0.smithi052.stdout:then mv -f ".deps/writer.Tpo" ".deps/writer.Po"; else rm -f ".deps/writer.Tpo"; exit 1; fi 2024-05-27T07:27:52.698 INFO:tasks.workunit.client.0.smithi052.stdout:if gcc -DHAVE_CONFIG_H -I. -I. -I.. -g -O2 -MT rewriter.o -MD -MP -MF ".deps/rewriter.Tpo" -c -o rewriter.o rewriter.c; \ 2024-05-27T07:27:52.698 INFO:tasks.workunit.client.0.smithi052.stdout:then mv -f ".deps/rewriter.Tpo" ".deps/rewriter.Po"; else rm -f ".deps/rewriter.Tpo"; exit 1; fi 2024-05-27T07:27:52.750 INFO:tasks.workunit.client.0.smithi052.stdout:if gcc -DHAVE_CONFIG_H -I. -I. -I.. -g -O2 -MT reader.o -MD -MP -MF ".deps/reader.Tpo" -c -o reader.o reader.c; \ 2024-05-27T07:27:52.751 INFO:tasks.workunit.client.0.smithi052.stdout:then mv -f ".deps/reader.Tpo" ".deps/reader.Po"; else rm -f ".deps/reader.Tpo"; exit 1; fi 2024-05-27T07:27:52.802 INFO:tasks.workunit.client.0.smithi052.stdout:if gcc -DHAVE_CONFIG_H -I. -I. -I.. -g -O2 -MT commenter.o -MD -MP -MF ".deps/commenter.Tpo" -c -o commenter.o commenter.c; \ 2024-05-27T07:27:52.802 INFO:tasks.workunit.client.0.smithi052.stdout:then mv -f ".deps/commenter.Tpo" ".deps/commenter.Po"; else rm -f ".deps/commenter.Tpo"; exit 1; fi 2024-05-27T07:27:52.851 INFO:tasks.workunit.client.0.smithi052.stdout:if gcc -DHAVE_CONFIG_H -I. -I. -I.. -g -O2 -MT helpers.o -MD -MP -MF ".deps/helpers.Tpo" -c -o helpers.o helpers.c; \ 2024-05-27T07:27:52.851 INFO:tasks.workunit.client.0.smithi052.stdout:then mv -f ".deps/helpers.Tpo" ".deps/helpers.Po"; else rm -f ".deps/helpers.Tpo"; exit 1; fi 2024-05-27T07:27:52.926 INFO:tasks.workunit.client.0.smithi052.stdout:if gcc -DHAVE_CONFIG_H -I. -I. -I.. -g -O2 -MT bsd-getopt_long.o -MD -MP -MF ".deps/bsd-getopt_long.Tpo" -c -o bsd-getopt_long.o bsd-getopt_long.c; \ 2024-05-27T07:27:52.926 INFO:tasks.workunit.client.0.smithi052.stdout:then mv -f ".deps/bsd-getopt_long.Tpo" ".deps/bsd-getopt_long.Po"; else rm -f ".deps/bsd-getopt_long.Tpo"; exit 1; fi 2024-05-27T07:27:52.961 INFO:tasks.workunit.client.0.smithi052.stdout:if gcc -DHAVE_CONFIG_H -I. -I. -I.. -g -O2 -MT fakesnprintf.o -MD -MP -MF ".deps/fakesnprintf.Tpo" -c -o fakesnprintf.o fakesnprintf.c; \ 2024-05-27T07:27:52.961 INFO:tasks.workunit.client.0.smithi052.stdout:then mv -f ".deps/fakesnprintf.Tpo" ".deps/fakesnprintf.Po"; else rm -f ".deps/fakesnprintf.Tpo"; exit 1; fi 2024-05-27T07:27:52.999 INFO:tasks.workunit.client.0.smithi052.stdout:if gcc -DHAVE_CONFIG_H -I. -I. -I.. -g -O2 -MT mysnprintf.o -MD -MP -MF ".deps/mysnprintf.Tpo" -c -o mysnprintf.o mysnprintf.c; \ 2024-05-27T07:27:52.999 INFO:tasks.workunit.client.0.smithi052.stdout:then mv -f ".deps/mysnprintf.Tpo" ".deps/mysnprintf.Po"; else rm -f ".deps/mysnprintf.Tpo"; exit 1; fi 2024-05-27T07:27:53.036 INFO:tasks.workunit.client.0.smithi052.stdout:gcc -g -O2 -o blogbench blogbench.o process.o writer.o rewriter.o reader.o commenter.o helpers.o bsd-getopt_long.o fakesnprintf.o mysnprintf.o 2024-05-27T07:27:53.056 INFO:tasks.workunit.client.0.smithi052.stdout:make[2]: Leaving directory '/home/ubuntu/cephtest/mnt.0/client.0/tmp/blogbench-1.0/src' 2024-05-27T07:27:53.056 INFO:tasks.workunit.client.0.smithi052.stdout:Making all in man 2024-05-27T07:27:53.095 INFO:tasks.workunit.client.0.smithi052.stdout:make[2]: Entering directory '/home/ubuntu/cephtest/mnt.0/client.0/tmp/blogbench-1.0/man' 2024-05-27T07:27:53.095 INFO:tasks.workunit.client.0.smithi052.stdout:make[2]: Nothing to be done for 'all'. 2024-05-27T07:27:53.095 INFO:tasks.workunit.client.0.smithi052.stdout:make[2]: Leaving directory '/home/ubuntu/cephtest/mnt.0/client.0/tmp/blogbench-1.0/man' 2024-05-27T07:27:53.096 INFO:tasks.workunit.client.0.smithi052.stdout:make[2]: Entering directory '/home/ubuntu/cephtest/mnt.0/client.0/tmp/blogbench-1.0' 2024-05-27T07:27:53.097 INFO:tasks.workunit.client.0.smithi052.stdout:make[2]: Leaving directory '/home/ubuntu/cephtest/mnt.0/client.0/tmp/blogbench-1.0' 2024-05-27T07:27:53.097 INFO:tasks.workunit.client.0.smithi052.stdout:make[1]: Leaving directory '/home/ubuntu/cephtest/mnt.0/client.0/tmp/blogbench-1.0' 2024-05-27T07:27:53.098 INFO:tasks.workunit.client.0.smithi052.stderr:+ cd src 2024-05-27T07:27:53.098 INFO:tasks.workunit.client.0.smithi052.stderr:+ mkdir blogtest_in 2024-05-27T07:27:53.100 INFO:tasks.workunit.client.0.smithi052.stdout:running blogbench 2024-05-27T07:27:53.101 INFO:tasks.workunit.client.0.smithi052.stderr:+ echo 'running blogbench' 2024-05-27T07:27:53.101 INFO:tasks.workunit.client.0.smithi052.stderr:+ ./blogbench -d blogtest_in 2024-05-27T07:27:53.101 INFO:tasks.workunit.client.0.smithi052.stdout: 2024-05-27T07:27:53.101 INFO:tasks.workunit.client.0.smithi052.stdout:Frequency = 10 secs 2024-05-27T07:27:53.101 INFO:tasks.workunit.client.0.smithi052.stdout:Scratch dir = [blogtest_in] 2024-05-27T07:27:54.439 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:27:54 smithi105 ceph-mon[35977]: pgmap v61: 129 pgs: 129 active+clean; 4.9 MiB data, 378 MiB used, 1.0 TiB / 1.0 TiB avail; 249 KiB/s rd, 379 KiB/s wr, 58 op/s 2024-05-27T07:27:54.479 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:27:54 smithi052 ceph-mon[31955]: pgmap v61: 129 pgs: 129 active+clean; 4.9 MiB data, 378 MiB used, 1.0 TiB / 1.0 TiB avail; 249 KiB/s rd, 379 KiB/s wr, 58 op/s 2024-05-27T07:27:54.601 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:27:54 smithi173 ceph-mon[37000]: pgmap v61: 129 pgs: 129 active+clean; 4.9 MiB data, 378 MiB used, 1.0 TiB / 1.0 TiB avail; 249 KiB/s rd, 379 KiB/s wr, 58 op/s 2024-05-27T07:27:55.201 INFO:tasks.workunit.client.0.smithi052.stdout:Spawning 3 writers... 2024-05-27T07:27:55.202 INFO:tasks.workunit.client.0.smithi052.stdout:Spawning 1 rewriters... 2024-05-27T07:27:55.202 INFO:tasks.workunit.client.0.smithi052.stdout:Spawning 5 commenters... 2024-05-27T07:27:55.202 INFO:tasks.workunit.client.0.smithi052.stdout:Spawning 100 readers... 2024-05-27T07:27:55.204 INFO:tasks.workunit.client.0.smithi052.stdout:Benchmarking for 30 iterations. 2024-05-27T07:27:55.204 INFO:tasks.workunit.client.0.smithi052.stdout:The test will run during 5 minutes. 2024-05-27T07:27:55.204 INFO:tasks.workunit.client.0.smithi052.stdout: 2024-05-27T07:27:55.205 INFO:tasks.workunit.client.0.smithi052.stdout: Nb blogs R articles W articles R pictures W pictures R comments W comments 2024-05-27T07:27:56.462 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:27:56 smithi105 ceph-mon[35977]: pgmap v62: 129 pgs: 129 active+clean; 4.9 MiB data, 378 MiB used, 1.0 TiB / 1.0 TiB avail; 246 KiB/s rd, 379 KiB/s wr, 53 op/s 2024-05-27T07:27:56.479 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:27:56 smithi052 ceph-mon[31955]: pgmap v62: 129 pgs: 129 active+clean; 4.9 MiB data, 378 MiB used, 1.0 TiB / 1.0 TiB avail; 246 KiB/s rd, 379 KiB/s wr, 53 op/s 2024-05-27T07:27:56.601 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:27:56 smithi173 ceph-mon[37000]: pgmap v62: 129 pgs: 129 active+clean; 4.9 MiB data, 378 MiB used, 1.0 TiB / 1.0 TiB avail; 246 KiB/s rd, 379 KiB/s wr, 53 op/s 2024-05-27T07:27:58.289 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:27:58.295+0000 7fd39eed8640 1 -- 172.21.15.52:0/912513756 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fd398108b80 msgr2=0x7fd398102720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:27:58.289 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:27:58.295+0000 7fd39eed8640 1 --2- 172.21.15.52:0/912513756 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fd398108b80 0x7fd398102720 secure :-1 s=READY pgs=558 cs=0 l=1 rev1=1 crypto rx=0x7fd388009900 tx=0x7fd38802a410 comp rx=0 tx=0).stop 2024-05-27T07:27:58.289 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:27:58.295+0000 7fd39eed8640 1 -- 172.21.15.52:0/912513756 shutdown_connections 2024-05-27T07:27:58.289 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:27:58.295+0000 7fd39eed8640 1 --2- 172.21.15.52:0/912513756 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fd3981032d0 0x7fd39810f6a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:27:58.290 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:27:58.295+0000 7fd39eed8640 1 --2- 172.21.15.52:0/912513756 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fd398109020 0x7fd398102c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:27:58.290 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:27:58.295+0000 7fd39eed8640 1 --2- 172.21.15.52:0/912513756 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fd398108b80 0x7fd398102720 unknown :-1 s=CLOSED pgs=558 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:27:58.290 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:27:58.295+0000 7fd39eed8640 1 -- 172.21.15.52:0/912513756 >> 172.21.15.52:0/912513756 conn(0x7fd3980fe380 msgr2=0x7fd3981007a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:27:58.290 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:27:58.295+0000 7fd39eed8640 1 -- 172.21.15.52:0/912513756 shutdown_connections 2024-05-27T07:27:58.290 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:27:58.295+0000 7fd39eed8640 1 -- 172.21.15.52:0/912513756 wait complete. 2024-05-27T07:27:58.290 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:27:58.296+0000 7fd39eed8640 1 Processor -- start 2024-05-27T07:27:58.290 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:27:58.296+0000 7fd39eed8640 1 -- start start 2024-05-27T07:27:58.290 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:27:58.296+0000 7fd39eed8640 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fd3981032d0 0x7fd39819f530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:27:58.290 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:27:58.296+0000 7fd39eed8640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fd398108b80 0x7fd39819fa70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:27:58.291 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:27:58.296+0000 7fd39eed8640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fd398109020 0x7fd3981996b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:27:58.291 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:27:58.296+0000 7fd39eed8640 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x7fd398112dd0 con 0x7fd3981032d0 2024-05-27T07:27:58.291 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:27:58.296+0000 7fd39eed8640 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7fd398112c50 con 0x7fd398109020 2024-05-27T07:27:58.291 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:27:58.296+0000 7fd39eed8640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7fd398112f50 con 0x7fd398108b80 2024-05-27T07:27:58.291 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:27:58.296+0000 7fd39cc74640 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fd3981032d0 0x7fd39819f530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:27:58.291 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:27:58.296+0000 7fd39cc74640 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fd3981032d0 0x7fd39819f530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.52:59644/0 (socket says 172.21.15.52:59644) 2024-05-27T07:27:58.291 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:27:58.296+0000 7fd39cc74640 1 -- 172.21.15.52:0/2762567090 learned_addr learned my addr 172.21.15.52:0/2762567090 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-05-27T07:27:58.291 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:27:58.296+0000 7fd39d475640 1 --2- 172.21.15.52:0/2762567090 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fd398109020 0x7fd3981996b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:27:58.291 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:27:58.297+0000 7fd397fff640 1 --2- 172.21.15.52:0/2762567090 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fd398108b80 0x7fd39819fa70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:27:58.291 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:27:58.297+0000 7fd39cc74640 1 -- 172.21.15.52:0/2762567090 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fd398108b80 msgr2=0x7fd39819fa70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:27:58.292 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:27:58.297+0000 7fd39cc74640 1 --2- 172.21.15.52:0/2762567090 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fd398108b80 0x7fd39819fa70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:27:58.292 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:27:58.297+0000 7fd39cc74640 1 -- 172.21.15.52:0/2762567090 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fd398109020 msgr2=0x7fd3981996b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:27:58.292 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:27:58.297+0000 7fd39cc74640 1 --2- 172.21.15.52:0/2762567090 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fd398109020 0x7fd3981996b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:27:58.292 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:27:58.297+0000 7fd39cc74640 1 -- 172.21.15.52:0/2762567090 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd398199de0 con 0x7fd3981032d0 2024-05-27T07:27:58.292 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:27:58.297+0000 7fd39d475640 1 --2- 172.21.15.52:0/2762567090 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fd398109020 0x7fd3981996b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-27T07:27:58.292 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:27:58.297+0000 7fd39cc74640 1 --2- 172.21.15.52:0/2762567090 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fd3981032d0 0x7fd39819f530 secure :-1 s=READY pgs=559 cs=0 l=1 rev1=1 crypto rx=0x7fd388006010 tx=0x7fd388004060 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:27:58.292 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:27:58.297+0000 7fd395ffb640 1 -- 172.21.15.52:0/2762567090 <== mon.0 v2:172.21.15.52:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd38803f030 con 0x7fd3981032d0 2024-05-27T07:27:58.292 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:27:58.297+0000 7fd395ffb640 1 -- 172.21.15.52:0/2762567090 <== mon.0 v2:172.21.15.52:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fd388003aa0 con 0x7fd3981032d0 2024-05-27T07:27:58.292 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:27:58.297+0000 7fd39eed8640 1 -- 172.21.15.52:0/2762567090 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd39819a010 con 0x7fd3981032d0 2024-05-27T07:27:58.292 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:27:58.297+0000 7fd395ffb640 1 -- 172.21.15.52:0/2762567090 <== mon.0 v2:172.21.15.52:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd388035070 con 0x7fd3981032d0 2024-05-27T07:27:58.293 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:27:58.297+0000 7fd39eed8640 1 -- 172.21.15.52:0/2762567090 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd39819a500 con 0x7fd3981032d0 2024-05-27T07:27:58.299 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:27:58.301+0000 7fd3a5d88740 1 Processor -- start 2024-05-27T07:27:58.299 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:27:58.301+0000 7fd3a5d88740 1 -- start start 2024-05-27T07:27:58.299 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:27:58.301+0000 7fd395ffb640 1 -- 172.21.15.52:0/2762567090 <== mon.0 v2:172.21.15.52:3300/0 4 ==== mgrmap(e 26) ==== 149783+0+0 (secure 0 0 0) 0x7fd388042030 con 0x7fd3981032d0 2024-05-27T07:27:58.300 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:27:58.301+0000 7fd3a5d88740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55f5223ea1b0 0x55f52278b220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:27:58.300 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:27:58.301+0000 7fd3a5d88740 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55f5223c12f0 0x55f52278b760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:27:58.300 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:27:58.301+0000 7fd3a5d88740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55f5223c94d0 0x55f52278bca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:27:58.300 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:27:58.301+0000 7fd3a5d88740 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x55f5226fdc30 con 0x55f5223c12f0 2024-05-27T07:27:58.300 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:27:58.301+0000 7fd3a5d88740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x55f5226ed7d0 con 0x55f5223c94d0 2024-05-27T07:27:58.300 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:27:58.301+0000 7fd3a5d88740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x55f5226f1810 con 0x55f5223ea1b0 2024-05-27T07:27:58.300 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:27:58.301+0000 7fd39cc74640 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55f5223c12f0 0x55f52278b760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:27:58.300 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:27:58.301+0000 7fd397fff640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55f5223c94d0 0x55f52278bca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:27:58.300 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:27:58.301+0000 7fd39cc74640 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55f5223c12f0 0x55f52278b760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.52:59652/0 (socket says 172.21.15.52:59652) 2024-05-27T07:27:58.301 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:27:58.301+0000 7fd39cc74640 1 -- 172.21.15.52:0/1607002966 learned_addr learned my addr 172.21.15.52:0/1607002966 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-05-27T07:27:58.301 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:27:58.301+0000 7fd39d475640 1 --2- 172.21.15.52:0/1607002966 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55f5223ea1b0 0x55f52278b220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:27:58.301 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:27:58.302+0000 7fd397fff640 1 -- 172.21.15.52:0/1607002966 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55f5223ea1b0 msgr2=0x55f52278b220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:27:58.301 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:27:58.302+0000 7fd397fff640 1 --2- 172.21.15.52:0/1607002966 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55f5223ea1b0 0x55f52278b220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:27:58.301 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:27:58.302+0000 7fd397fff640 1 -- 172.21.15.52:0/1607002966 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55f5223c12f0 msgr2=0x55f52278b760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:27:58.301 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:27:58.302+0000 7fd397fff640 1 --2- 172.21.15.52:0/1607002966 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55f5223c12f0 0x55f52278b760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:27:58.301 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:27:58.302+0000 7fd397fff640 1 -- 172.21.15.52:0/1607002966 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55f522402210 con 0x55f5223c94d0 2024-05-27T07:27:58.301 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:27:58.302+0000 7fd397fff640 1 --2- 172.21.15.52:0/1607002966 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55f5223c94d0 0x55f52278bca0 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7fd39819aee0 tx=0x7fd38000c7f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c2c1cbaf7743077e server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:27:58.301 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:27:58.302+0000 7fd38effd640 1 -- 172.21.15.52:0/1607002966 <== mon.2 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd38001a070 con 0x55f5223c94d0 2024-05-27T07:27:58.301 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:27:58.302+0000 7fd395ffb640 1 --2- 172.21.15.52:0/2762567090 >> [v2:172.21.15.52:6800/1822318384,v1:172.21.15.52:6801/1822318384] conn(0x7fd3840b2950 0x7fd3840b4e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:27:58.302 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:27:58.302+0000 7fd38effd640 1 -- 172.21.15.52:0/1607002966 <== mon.2 v2:172.21.15.105:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fd3800158c0 con 0x55f5223c94d0 2024-05-27T07:27:58.302 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:27:58.302+0000 7fd38effd640 1 -- 172.21.15.52:0/1607002966 <== mon.2 v2:172.21.15.105:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd38000a780 con 0x55f5223c94d0 2024-05-27T07:27:58.302 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:27:58.302+0000 7fd39d475640 1 --2- 172.21.15.52:0/2762567090 >> [v2:172.21.15.52:6800/1822318384,v1:172.21.15.52:6801/1822318384] conn(0x7fd3840b2950 0x7fd3840b4e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:27:58.302 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:27:58.302+0000 7fd395ffb640 1 -- 172.21.15.52:0/2762567090 <== mon.0 v2:172.21.15.52:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9379+0+0 (secure 0 0 0) 0x7fd388102df0 con 0x7fd3981032d0 2024-05-27T07:27:58.302 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:27:58.303+0000 7fd3a5d88740 1 -- 172.21.15.52:0/1607002966 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55f5223c94d0 msgr2=0x55f52278bca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:27:58.302 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:27:58.303+0000 7fd39d475640 1 --2- 172.21.15.52:0/2762567090 >> [v2:172.21.15.52:6800/1822318384,v1:172.21.15.52:6801/1822318384] conn(0x7fd3840b2950 0x7fd3840b4e60 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x55f522788de0 tx=0x7fd390005f90 comp rx=0 tx=0).ready entity=mgr.15333 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:27:58.302 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:27:58.303+0000 7fd3a5d88740 1 --2- 172.21.15.52:0/1607002966 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55f5223c94d0 0x55f52278bca0 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7fd39819aee0 tx=0x7fd38000c7f0 comp rx=0 tx=0).stop 2024-05-27T07:27:58.302 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:27:58.303+0000 7fd3a5d88740 1 -- 172.21.15.52:0/1607002966 shutdown_connections 2024-05-27T07:27:58.302 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:27:58.303+0000 7fd3a5d88740 1 --2- 172.21.15.52:0/1607002966 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55f5223c94d0 0x55f52278bca0 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:27:58.303 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:27:58.303+0000 7fd3a5d88740 1 --2- 172.21.15.52:0/1607002966 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55f5223c12f0 0x55f52278b760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:27:58.303 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:27:58.303+0000 7fd3a5d88740 1 --2- 172.21.15.52:0/1607002966 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55f5223ea1b0 0x55f52278b220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:27:58.303 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:27:58.303+0000 7fd3a5d88740 1 -- 172.21.15.52:0/1607002966 >> 172.21.15.52:0/1607002966 conn(0x55f5223de300 msgr2=0x55f522787b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:27:58.303 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:27:58.303+0000 7fd3a5d88740 1 -- 172.21.15.52:0/1607002966 shutdown_connections 2024-05-27T07:27:58.303 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:27:58.303+0000 7fd3a5d88740 1 -- 172.21.15.52:0/1607002966 wait complete. 2024-05-27T07:27:58.303 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:27:58.304+0000 7fd3a5d88740 1 -- start start 2024-05-27T07:27:58.303 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:27:58.304+0000 7fd3a5d88740 1 Processor -- start 2024-05-27T07:27:58.303 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:27:58.304+0000 7fd3a5d88740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55f5223c12f0 0x55f522797cb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:27:58.303 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:27:58.304+0000 7fd3a5d88740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55f5223c94d0 0x55f5227981f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:27:58.303 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:27:58.304+0000 7fd3a5d88740 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55f5223ea1b0 0x55f52279db00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:27:58.303 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:27:58.304+0000 7fd3a5d88740 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x55f5223d3790 con 0x55f5223ea1b0 2024-05-27T07:27:58.304 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:27:58.304+0000 7fd3a5d88740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x55f5226a9890 con 0x55f5223c94d0 2024-05-27T07:27:58.304 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:27:58.304+0000 7fd3a5d88740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x55f52279a770 con 0x55f5223c12f0 2024-05-27T07:27:58.304 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:27:58.304+0000 7fd3a5d88740 10 client.0 _ll_register_callbacks cb 0x55f522401a00 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T07:27:58.304 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:27:58.304+0000 7fd39d475640 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55f5223ea1b0 0x55f52279db00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:27:58.304 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:27:58.304+0000 7fd397fff640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55f5223c94d0 0x55f5227981f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:27:58.304 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:27:58.304+0000 7fd39d475640 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55f5223ea1b0 0x55f52279db00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.52:59660/0 (socket says 172.21.15.52:59660) 2024-05-27T07:27:58.304 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:27:58.304+0000 7fd39d475640 1 -- 172.21.15.52:0/2244433776 learned_addr learned my addr 172.21.15.52:0/2244433776 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-05-27T07:27:58.304 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:27:58.305+0000 7fd39cc74640 1 --2- 172.21.15.52:0/2244433776 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55f5223c12f0 0x55f522797cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:27:58.304 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:27:58.305+0000 7fd39d475640 1 -- 172.21.15.52:0/2244433776 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55f5223c12f0 msgr2=0x55f522797cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:27:58.304 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:27:58.305+0000 7fd39d475640 1 --2- 172.21.15.52:0/2244433776 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55f5223c12f0 0x55f522797cb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:27:58.304 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:27:58.305+0000 7fd39d475640 1 -- 172.21.15.52:0/2244433776 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55f5223c94d0 msgr2=0x55f5227981f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:27:58.304 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:27:58.305+0000 7fd39d475640 1 --2- 172.21.15.52:0/2244433776 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55f5223c94d0 0x55f5227981f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:27:58.305 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:27:58.305+0000 7fd39d475640 1 -- 172.21.15.52:0/2244433776 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55f5224ce620 con 0x55f5223ea1b0 2024-05-27T07:27:58.305 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:27:58.305+0000 7fd397fff640 1 --2- 172.21.15.52:0/2244433776 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55f5223c94d0 0x55f5227981f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-27T07:27:58.305 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:27:58.305+0000 7fd39d475640 1 --2- 172.21.15.52:0/2244433776 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55f5223ea1b0 0x55f52279db00 secure :-1 s=READY pgs=560 cs=0 l=1 rev1=1 crypto rx=0x7fd3900045d0 tx=0x7fd390004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ee09bc94ca548860 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:27:58.305 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:27:58.305+0000 7fd377fff640 10 client.0 ms_handle_connect on v2:172.21.15.52:3300/0 2024-05-27T07:27:58.305 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:27:58.305+0000 7fd377fff640 1 -- 172.21.15.52:0/2244433776 <== mon.0 v2:172.21.15.52:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd39000f830 con 0x55f5223ea1b0 2024-05-27T07:27:58.305 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:27:58.305+0000 7fd3a5d88740 1 -- 172.21.15.52:0/2244433776 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fd3800158c0 con 0x55f5223ea1b0 2024-05-27T07:27:58.305 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:27:58.305+0000 7fd377fff640 1 -- 172.21.15.52:0/2244433776 <== mon.0 v2:172.21.15.52:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 9379+0+0 (secure 0 0 0) 0x7fd39000f9d0 con 0x55f5223ea1b0 2024-05-27T07:27:58.305 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:27:58.305+0000 7fd377fff640 1 -- 172.21.15.52:0/2244433776 <== mon.0 v2:172.21.15.52:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7fd39000fbd0 con 0x55f5223ea1b0 2024-05-27T07:27:58.305 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:27:58.305+0000 7fd3a5d88740 10 client.15966 fetch_fsmap learned FSMap version 31 2024-05-27T07:27:58.305 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:27:58.305+0000 7fd3a5d88740 1 -- 172.21.15.52:0/2244433776 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x55f5227a6ac0 con 0x55f5223ea1b0 2024-05-27T07:27:58.305 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:27:58.306+0000 7fd377fff640 1 -- 172.21.15.52:0/2244433776 <== mon.0 v2:172.21.15.52:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fd3800158c0 con 0x55f5223ea1b0 2024-05-27T07:27:58.306 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:27:58.306+0000 7fd377fff640 1 -- 172.21.15.52:0/2244433776 <== mon.0 v2:172.21.15.52:3300/0 5 ==== fsmap(e 31) ==== 8366+0+0 (secure 0 0 0) 0x7fd390027070 con 0x55f5223ea1b0 2024-05-27T07:27:58.306 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:27:58.306+0000 7fd3a5d88740 10 client.15966 fetch_fsmap finished waiting for FSMap version 31 2024-05-27T07:27:58.306 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:27:58.306+0000 7fd3a5d88740 10 client.15966 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-05-27T07:27:58.306 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:27:58.306+0000 7fd3a5d88740 20 client.15966 populate_metadata read hostname 'smithi052' 2024-05-27T07:27:58.306 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:27:58.306+0000 7fd3a5d88740 1 --2- 172.21.15.52:0/2244433776 >> [v2:172.21.15.105:6838/500348151,v1:172.21.15.105:6839/500348151] conn(0x55f5223a8d80 0x55f5227a9ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:27:58.306 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:27:58.306+0000 7fd3a5d88740 4 client.15966 mds_command: new command op to 24527 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-27T07:27:58.306 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:27:58.306+0000 7fd3a5d88740 1 -- 172.21.15.52:0/2244433776 --> [v2:172.21.15.105:6838/500348151,v1:172.21.15.105:6839/500348151] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55f5226ffb80 con 0x55f5223a8d80 2024-05-27T07:27:58.306 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:27:58.306+0000 7fd39cc74640 1 --2- 172.21.15.52:0/2244433776 >> [v2:172.21.15.105:6838/500348151,v1:172.21.15.105:6839/500348151] conn(0x55f5223a8d80 0x55f5227a9ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:27:58.306 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:27:58.307+0000 7fd377fff640 10 client.15966 ms_handle_connect on v2:172.21.15.105:6838/500348151 2024-05-27T07:27:58.306 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:27:58.307+0000 7fd39cc74640 1 --2- 172.21.15.52:0/2244433776 >> [v2:172.21.15.105:6838/500348151,v1:172.21.15.105:6839/500348151] conn(0x55f5223a8d80 0x55f5227a9ff0 crc :-1 s=READY pgs=75 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=5edb4bffa3495fe6 server_cookie=5fa8b6de7c5a2089 in_seq=0 out_seq=0 2024-05-27T07:27:58.307 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:27:58.310+0000 7fd377fff640 1 -- 172.21.15.52:0/2244433776 <== mds.0 v2:172.21.15.105:6838/500348151 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24041 (crc 0 0 0) 0x55f5226ffb80 con 0x55f5223a8d80 2024-05-27T07:27:58.307 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:27:58.310+0000 7fd377fff640 10 client.15966 handle_command_reply: tid=0 2024-05-27T07:27:58.308 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:27:58.310+0000 7fd3a5d88740 1 client.15966 shutdown 2024-05-27T07:27:58.308 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:27:58.311+0000 7fd3a5d88740 1 -- 172.21.15.52:0/2244433776 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55f5223ea1b0 msgr2=0x55f52279db00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:27:58.308 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:27:58.311+0000 7fd3a5d88740 1 --2- 172.21.15.52:0/2244433776 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55f5223ea1b0 0x55f52279db00 secure :-1 s=READY pgs=560 cs=0 l=1 rev1=1 crypto rx=0x7fd3900045d0 tx=0x7fd390004730 comp rx=0 tx=0).stop 2024-05-27T07:27:58.308 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:27:58.311+0000 7fd3a5d88740 1 -- 172.21.15.52:0/2244433776 shutdown_connections 2024-05-27T07:27:58.308 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:27:58.311+0000 7fd3a5d88740 1 --2- 172.21.15.52:0/2244433776 >> [v2:172.21.15.105:6838/500348151,v1:172.21.15.105:6839/500348151] conn(0x55f5223a8d80 0x55f5227a9ff0 crc :-1 s=READY pgs=75 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:27:58.308 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:27:58.311+0000 7fd3a5d88740 1 --2- 172.21.15.52:0/2244433776 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55f5223ea1b0 0x55f52279db00 unknown :-1 s=CLOSED pgs=560 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:27:58.308 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:27:58.311+0000 7fd3a5d88740 1 --2- 172.21.15.52:0/2244433776 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55f5223c94d0 0x55f5227981f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:27:58.308 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:27:58.311+0000 7fd3a5d88740 1 --2- 172.21.15.52:0/2244433776 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55f5223c12f0 0x55f522797cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:27:58.308 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:27:58.311+0000 7fd377fff640 0 client.15966 ms_handle_reset on v2:172.21.15.105:6838/500348151 2024-05-27T07:27:58.308 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:27:58.311+0000 7fd3a5d88740 1 -- 172.21.15.52:0/2244433776 >> 172.21.15.52:0/2244433776 conn(0x55f5223de300 msgr2=0x55f52278d080 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:27:58.308 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:27:58.311+0000 7fd3a5d88740 1 -- 172.21.15.52:0/2244433776 shutdown_connections 2024-05-27T07:27:58.308 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:27:58.311+0000 7fd3a5d88740 1 -- 172.21.15.52:0/2244433776 wait complete. 2024-05-27T07:27:58.309 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:27:58.311+0000 7fd3a5d88740 20 client.15966 trim_cache size 0 max 16384 2024-05-27T07:27:58.317 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:27:58.326+0000 7fd3a5d88740 1 Processor -- start 2024-05-27T07:27:58.317 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:27:58.326+0000 7fd3a5d88740 1 -- start start 2024-05-27T07:27:58.317 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:27:58.327+0000 7fd3a5d88740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55f5223c12f0 0x55f52278d720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:27:58.318 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:27:58.327+0000 7fd3a5d88740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55f5223c94d0 0x55f5227a4780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:27:58.318 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:27:58.327+0000 7fd3a5d88740 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55f5223ea1b0 0x55f522791410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:27:58.318 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:27:58.327+0000 7fd3a5d88740 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x7fd3981af020 con 0x55f5223ea1b0 2024-05-27T07:27:58.318 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:27:58.327+0000 7fd3a5d88740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7fd3981b0b90 con 0x55f5223c94d0 2024-05-27T07:27:58.318 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:27:58.327+0000 7fd3a5d88740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7fd3981af6f0 con 0x55f5223c12f0 2024-05-27T07:27:58.318 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:27:58.327+0000 7fd39d475640 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55f5223ea1b0 0x55f522791410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:27:58.318 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:27:58.327+0000 7fd397fff640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55f5223c94d0 0x55f5227a4780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:27:58.318 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:27:58.327+0000 7fd39cc74640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55f5223c12f0 0x55f52278d720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:27:58.318 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:27:58.327+0000 7fd39d475640 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55f5223ea1b0 0x55f522791410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.52:59662/0 (socket says 172.21.15.52:59662) 2024-05-27T07:27:58.318 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:27:58.327+0000 7fd39d475640 1 -- 172.21.15.52:0/2417669741 learned_addr learned my addr 172.21.15.52:0/2417669741 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-05-27T07:27:58.319 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:27:58.327+0000 7fd39d475640 1 -- 172.21.15.52:0/2417669741 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55f5223c12f0 msgr2=0x55f52278d720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:27:58.319 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:27:58.327+0000 7fd39d475640 1 --2- 172.21.15.52:0/2417669741 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55f5223c12f0 0x55f52278d720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:27:58.319 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:27:58.327+0000 7fd39d475640 1 -- 172.21.15.52:0/2417669741 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55f5223c94d0 msgr2=0x55f5227a4780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:27:58.319 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:27:58.327+0000 7fd39d475640 1 --2- 172.21.15.52:0/2417669741 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55f5223c94d0 0x55f5227a4780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:27:58.319 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:27:58.327+0000 7fd39d475640 1 -- 172.21.15.52:0/2417669741 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55f522781e60 con 0x55f5223ea1b0 2024-05-27T07:27:58.319 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:27:58.327+0000 7fd39d475640 1 --2- 172.21.15.52:0/2417669741 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55f5223ea1b0 0x55f522791410 secure :-1 s=READY pgs=561 cs=0 l=1 rev1=1 crypto rx=0x7fd390012f40 tx=0x7fd390003fe0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=44178a823aafdc05 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:27:58.319 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:27:58.327+0000 7fd38e7fc640 1 -- 172.21.15.52:0/2417669741 <== mon.0 v2:172.21.15.52:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd39000fa30 con 0x55f5223ea1b0 2024-05-27T07:27:58.323 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:27:58.328+0000 7fd38e7fc640 1 -- 172.21.15.52:0/2417669741 <== mon.0 v2:172.21.15.52:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fd39001fb50 con 0x55f5223ea1b0 2024-05-27T07:27:58.323 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:27:58.328+0000 7fd38e7fc640 1 -- 172.21.15.52:0/2417669741 <== mon.0 v2:172.21.15.52:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd390029a80 con 0x55f5223ea1b0 2024-05-27T07:27:58.323 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:27:58.328+0000 7fd3a5d88740 1 -- 172.21.15.52:0/2417669741 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55f5223ea1b0 msgr2=0x55f522791410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:27:58.324 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:27:58.328+0000 7fd3a5d88740 1 --2- 172.21.15.52:0/2417669741 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55f5223ea1b0 0x55f522791410 secure :-1 s=READY pgs=561 cs=0 l=1 rev1=1 crypto rx=0x7fd390012f40 tx=0x7fd390003fe0 comp rx=0 tx=0).stop 2024-05-27T07:27:58.324 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:27:58.328+0000 7fd3a5d88740 1 -- 172.21.15.52:0/2417669741 shutdown_connections 2024-05-27T07:27:58.324 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:27:58.328+0000 7fd3a5d88740 1 --2- 172.21.15.52:0/2417669741 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55f5223ea1b0 0x55f522791410 unknown :-1 s=CLOSED pgs=561 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:27:58.324 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:27:58.328+0000 7fd3a5d88740 1 --2- 172.21.15.52:0/2417669741 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55f5223c94d0 0x55f5227a4780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:27:58.324 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:27:58.328+0000 7fd3a5d88740 1 --2- 172.21.15.52:0/2417669741 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55f5223c12f0 0x55f52278d720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:27:58.324 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:27:58.328+0000 7fd3a5d88740 1 -- 172.21.15.52:0/2417669741 >> 172.21.15.52:0/2417669741 conn(0x55f5223de300 msgr2=0x55f52278d010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:27:58.324 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:27:58.328+0000 7fd3a5d88740 1 -- 172.21.15.52:0/2417669741 shutdown_connections 2024-05-27T07:27:58.325 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:27:58.328+0000 7fd3a5d88740 1 -- 172.21.15.52:0/2417669741 wait complete. 2024-05-27T07:27:58.325 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:27:58.329+0000 7fd3a5d88740 1 -- start start 2024-05-27T07:27:58.325 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:27:58.329+0000 7fd3a5d88740 1 Processor -- start 2024-05-27T07:27:58.325 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:27:58.329+0000 7fd3a5d88740 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55f5223c12f0 0x55f52279ee40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:27:58.325 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:27:58.329+0000 7fd3a5d88740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55f5223c94d0 0x55f52279f380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:27:58.325 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:27:58.329+0000 7fd3a5d88740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55f5223ea1b0 0x55f52279f8c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:27:58.325 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:27:58.329+0000 7fd3a5d88740 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x7fd3981b0490 con 0x55f5223c12f0 2024-05-27T07:27:58.325 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:27:58.329+0000 7fd3a5d88740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7fd3981afdc0 con 0x55f5223ea1b0 2024-05-27T07:27:58.325 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:27:58.329+0000 7fd3a5d88740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7fd3981ad000 con 0x55f5223c94d0 2024-05-27T07:27:58.326 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:27:58.329+0000 7fd3a5d88740 10 client.0 _ll_register_callbacks cb 0x7fd390009e00 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T07:27:58.326 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:27:58.329+0000 7fd39cc74640 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55f5223c12f0 0x55f52279ee40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:27:58.326 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:27:58.329+0000 7fd39cc74640 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55f5223c12f0 0x55f52279ee40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.52:59672/0 (socket says 172.21.15.52:59672) 2024-05-27T07:27:58.326 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:27:58.329+0000 7fd39cc74640 1 -- 172.21.15.52:0/162898668 learned_addr learned my addr 172.21.15.52:0/162898668 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-05-27T07:27:58.326 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:27:58.329+0000 7fd39d475640 1 --2- 172.21.15.52:0/162898668 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55f5223ea1b0 0x55f52279f8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:27:58.326 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:27:58.330+0000 7fd397fff640 1 --2- 172.21.15.52:0/162898668 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55f5223c94d0 0x55f52279f380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:27:58.326 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:27:58.330+0000 7fd39cc74640 1 -- 172.21.15.52:0/162898668 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55f5223c94d0 msgr2=0x55f52279f380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:27:58.326 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:27:58.330+0000 7fd39cc74640 1 --2- 172.21.15.52:0/162898668 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55f5223c94d0 0x55f52279f380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:27:58.326 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:27:58.330+0000 7fd39cc74640 1 -- 172.21.15.52:0/162898668 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55f5223ea1b0 msgr2=0x55f52279f8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:27:58.327 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:27:58.330+0000 7fd39cc74640 1 --2- 172.21.15.52:0/162898668 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55f5223ea1b0 0x55f52279f8c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:27:58.327 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:27:58.330+0000 7fd39cc74640 1 -- 172.21.15.52:0/162898668 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55f5228108b0 con 0x55f5223c12f0 2024-05-27T07:27:58.327 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:27:58.330+0000 7fd397fff640 1 --2- 172.21.15.52:0/162898668 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55f5223c94d0 0x55f52279f380 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-27T07:27:58.327 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:27:58.330+0000 7fd39cc74640 1 --2- 172.21.15.52:0/162898668 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55f5223c12f0 0x55f52279ee40 secure :-1 s=READY pgs=562 cs=0 l=1 rev1=1 crypto rx=0x7fd3880c5df0 tx=0x7fd388000bc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cad24917bc3387d3 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:27:58.327 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:27:58.330+0000 7fd377fff640 10 client.0 ms_handle_connect on v2:172.21.15.52:3300/0 2024-05-27T07:27:58.327 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:27:58.330+0000 7fd377fff640 1 -- 172.21.15.52:0/162898668 <== mon.0 v2:172.21.15.52:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd388049040 con 0x55f5223c12f0 2024-05-27T07:27:58.327 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:27:58.330+0000 7fd377fff640 1 -- 172.21.15.52:0/162898668 <== mon.0 v2:172.21.15.52:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 9379+0+0 (secure 0 0 0) 0x7fd3880c6000 con 0x55f5223c12f0 2024-05-27T07:27:58.327 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:27:58.330+0000 7fd3a5d88740 1 -- 172.21.15.52:0/162898668 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fd39001fb50 con 0x55f5223c12f0 2024-05-27T07:27:58.327 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:27:58.330+0000 7fd377fff640 1 -- 172.21.15.52:0/162898668 <== mon.0 v2:172.21.15.52:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7fd3880c94e0 con 0x55f5223c12f0 2024-05-27T07:27:58.327 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:27:58.330+0000 7fd3a5d88740 10 client.15972 fetch_fsmap learned FSMap version 31 2024-05-27T07:27:58.328 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:27:58.330+0000 7fd3a5d88740 1 -- 172.21.15.52:0/162898668 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x55f522811b60 con 0x55f5223c12f0 2024-05-27T07:27:58.328 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:27:58.330+0000 7fd377fff640 1 -- 172.21.15.52:0/162898668 <== mon.0 v2:172.21.15.52:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fd39001fb50 con 0x55f5223c12f0 2024-05-27T07:27:58.328 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:27:58.331+0000 7fd377fff640 1 -- 172.21.15.52:0/162898668 <== mon.0 v2:172.21.15.52:3300/0 5 ==== fsmap(e 31) ==== 8366+0+0 (secure 0 0 0) 0x7fd3880c9fd0 con 0x55f5223c12f0 2024-05-27T07:27:58.328 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:27:58.331+0000 7fd3a5d88740 10 client.15972 fetch_fsmap finished waiting for FSMap version 31 2024-05-27T07:27:58.328 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:27:58.331+0000 7fd3a5d88740 10 client.15972 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-05-27T07:27:58.328 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:27:58.331+0000 7fd3a5d88740 20 client.15972 populate_metadata read hostname 'smithi052' 2024-05-27T07:27:58.328 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:27:58.331+0000 7fd3a5d88740 1 --2- 172.21.15.52:0/162898668 >> [v2:172.21.15.105:6838/500348151,v1:172.21.15.105:6839/500348151] conn(0x55f52237e1c0 0x55f522814320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:27:58.328 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:27:58.331+0000 7fd3a5d88740 4 client.15972 mds_command: new command op to 24527 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-05-27T07:27:58.328 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:27:58.331+0000 7fd3a5d88740 1 -- 172.21.15.52:0/162898668 --> [v2:172.21.15.105:6838/500348151,v1:172.21.15.105:6839/500348151] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55f5226f9360 con 0x55f52237e1c0 2024-05-27T07:27:58.328 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:27:58.331+0000 7fd397fff640 1 --2- 172.21.15.52:0/162898668 >> [v2:172.21.15.105:6838/500348151,v1:172.21.15.105:6839/500348151] conn(0x55f52237e1c0 0x55f522814320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:27:58.328 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:27:58.331+0000 7fd377fff640 10 client.15972 ms_handle_connect on v2:172.21.15.105:6838/500348151 2024-05-27T07:27:58.329 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:27:58.331+0000 7fd397fff640 1 --2- 172.21.15.52:0/162898668 >> [v2:172.21.15.105:6838/500348151,v1:172.21.15.105:6839/500348151] conn(0x55f52237e1c0 0x55f522814320 crc :-1 s=READY pgs=76 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=d0c02c672e48f163 server_cookie=c456a25c96239e1f in_seq=0 out_seq=0 2024-05-27T07:27:58.440 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:27:58 smithi105 ceph-mon[35977]: pgmap v63: 129 pgs: 129 active+clean; 4.9 MiB data, 378 MiB used, 1.0 TiB / 1.0 TiB avail; 246 KiB/s rd, 379 KiB/s wr, 53 op/s 2024-05-27T07:27:58.440 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:27:58 smithi105 ceph-mon[35977]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-05-27T07:27:58.479 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:27:58 smithi052 ceph-mon[31955]: pgmap v63: 129 pgs: 129 active+clean; 4.9 MiB data, 378 MiB used, 1.0 TiB / 1.0 TiB avail; 246 KiB/s rd, 379 KiB/s wr, 53 op/s 2024-05-27T07:27:58.479 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:27:58 smithi052 ceph-mon[31955]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-05-27T07:27:58.602 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:27:58 smithi173 ceph-mon[37000]: pgmap v63: 129 pgs: 129 active+clean; 4.9 MiB data, 378 MiB used, 1.0 TiB / 1.0 TiB avail; 246 KiB/s rd, 379 KiB/s wr, 53 op/s 2024-05-27T07:27:58.602 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:27:58 smithi173 ceph-mon[37000]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-05-27T07:28:00.439 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:28:00 smithi105 ceph-mon[35977]: pgmap v64: 129 pgs: 129 active+clean; 9.9 MiB data, 381 MiB used, 1.0 TiB / 1.0 TiB avail; 615 KiB/s rd, 835 KiB/s wr, 69 op/s 2024-05-27T07:28:00.479 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:28:00 smithi052 ceph-mon[31955]: pgmap v64: 129 pgs: 129 active+clean; 9.9 MiB data, 381 MiB used, 1.0 TiB / 1.0 TiB avail; 615 KiB/s rd, 835 KiB/s wr, 69 op/s 2024-05-27T07:28:00.601 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:28:00 smithi173 ceph-mon[37000]: pgmap v64: 129 pgs: 129 active+clean; 9.9 MiB data, 381 MiB used, 1.0 TiB / 1.0 TiB avail; 615 KiB/s rd, 835 KiB/s wr, 69 op/s 2024-05-27T07:28:01.809 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:28:01.819+0000 7fd377fff640 1 -- 172.21.15.52:0/162898668 <== mds.0 v2:172.21.15.105:6838/500348151 1 ==== command_reply(tid 0: 0 ) ==== 8+0+25 (crc 0 0 0) 0x55f5226f9360 con 0x55f52237e1c0 2024-05-27T07:28:01.809 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:28:01.819+0000 7fd377fff640 10 client.15972 handle_command_reply: tid=0 2024-05-27T07:28:01.810 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:28:01.819+0000 7fd3a5d88740 1 client.15972 shutdown 2024-05-27T07:28:01.810 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:28:01.819+0000 7fd3a5d88740 1 -- 172.21.15.52:0/162898668 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55f5223c12f0 msgr2=0x55f52279ee40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:28:01.810 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:28:01.819+0000 7fd3a5d88740 1 --2- 172.21.15.52:0/162898668 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55f5223c12f0 0x55f52279ee40 secure :-1 s=READY pgs=562 cs=0 l=1 rev1=1 crypto rx=0x7fd3880c5df0 tx=0x7fd388000bc0 comp rx=0 tx=0).stop 2024-05-27T07:28:01.810 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:28:01.819+0000 7fd3a5d88740 1 -- 172.21.15.52:0/162898668 shutdown_connections 2024-05-27T07:28:01.810 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:28:01.819+0000 7fd3a5d88740 1 --2- 172.21.15.52:0/162898668 >> [v2:172.21.15.105:6838/500348151,v1:172.21.15.105:6839/500348151] conn(0x55f52237e1c0 0x55f522814320 crc :-1 s=READY pgs=76 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:28:01.810 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:28:01.819+0000 7fd3a5d88740 1 --2- 172.21.15.52:0/162898668 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55f5223ea1b0 0x55f52279f8c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:28:01.810 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:28:01.819+0000 7fd3a5d88740 1 --2- 172.21.15.52:0/162898668 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55f5223c94d0 0x55f52279f380 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:28:01.810 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:28:01.819+0000 7fd377fff640 0 client.15972 ms_handle_reset on v2:172.21.15.105:6838/500348151 2024-05-27T07:28:01.810 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:28:01.819+0000 7fd3a5d88740 1 --2- 172.21.15.52:0/162898668 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55f5223c12f0 0x55f52279ee40 unknown :-1 s=CLOSED pgs=562 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:28:01.810 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:28:01.819+0000 7fd3a5d88740 1 -- 172.21.15.52:0/162898668 >> 172.21.15.52:0/162898668 conn(0x55f5223de300 msgr2=0x55f52278b160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:28:01.811 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:28:01.819+0000 7fd3a5d88740 1 -- 172.21.15.52:0/162898668 shutdown_connections 2024-05-27T07:28:01.811 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:28:01.819+0000 7fd3a5d88740 1 -- 172.21.15.52:0/162898668 wait complete. 2024-05-27T07:28:01.811 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:28:01.819+0000 7fd3a5d88740 20 client.15972 trim_cache size 0 max 16384 2024-05-27T07:28:01.811 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stdout:{ 2024-05-27T07:28:01.811 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stdout: "return_code": 0 2024-05-27T07:28:01.811 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stdout:} 2024-05-27T07:28:01.812 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:28:01.820+0000 7fd39eed8640 1 -- 172.21.15.52:0/2762567090 >> [v2:172.21.15.52:6800/1822318384,v1:172.21.15.52:6801/1822318384] conn(0x7fd3840b2950 msgr2=0x7fd3840b4e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:28:01.812 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:28:01.820+0000 7fd39eed8640 1 --2- 172.21.15.52:0/2762567090 >> [v2:172.21.15.52:6800/1822318384,v1:172.21.15.52:6801/1822318384] conn(0x7fd3840b2950 0x7fd3840b4e60 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x55f522788de0 tx=0x7fd390005f90 comp rx=0 tx=0).stop 2024-05-27T07:28:01.812 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:28:01.820+0000 7fd39eed8640 1 -- 172.21.15.52:0/2762567090 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fd3981032d0 msgr2=0x7fd39819f530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:28:01.812 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:28:01.820+0000 7fd39eed8640 1 --2- 172.21.15.52:0/2762567090 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fd3981032d0 0x7fd39819f530 secure :-1 s=READY pgs=559 cs=0 l=1 rev1=1 crypto rx=0x7fd388006010 tx=0x7fd388004060 comp rx=0 tx=0).stop 2024-05-27T07:28:01.813 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:28:01.820+0000 7fd39eed8640 1 -- 172.21.15.52:0/2762567090 shutdown_connections 2024-05-27T07:28:01.813 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:28:01.820+0000 7fd39eed8640 1 --2- 172.21.15.52:0/2762567090 >> [v2:172.21.15.52:6800/1822318384,v1:172.21.15.52:6801/1822318384] conn(0x7fd3840b2950 0x7fd3840b4e60 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:28:01.813 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:28:01.820+0000 7fd39eed8640 1 --2- 172.21.15.52:0/2762567090 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fd398109020 0x7fd3981996b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:28:01.813 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:28:01.820+0000 7fd39eed8640 1 --2- 172.21.15.52:0/2762567090 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fd398108b80 0x7fd39819fa70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:28:01.813 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:28:01.820+0000 7fd39eed8640 1 --2- 172.21.15.52:0/2762567090 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fd3981032d0 0x7fd39819f530 unknown :-1 s=CLOSED pgs=559 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:28:01.813 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:28:01.820+0000 7fd39eed8640 1 -- 172.21.15.52:0/2762567090 >> 172.21.15.52:0/2762567090 conn(0x7fd3980fe380 msgr2=0x7fd3980fe790 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:28:01.813 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:28:01.820+0000 7fd39eed8640 1 -- 172.21.15.52:0/2762567090 shutdown_connections 2024-05-27T07:28:01.813 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:28:01.820+0000 7fd39eed8640 1 -- 172.21.15.52:0/2762567090 wait complete. 2024-05-27T07:28:02.439 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:28:02 smithi105 ceph-mon[35977]: pgmap v65: 129 pgs: 129 active+clean; 9.9 MiB data, 381 MiB used, 1.0 TiB / 1.0 TiB avail; 601 KiB/s rd, 798 KiB/s wr, 63 op/s 2024-05-27T07:28:02.479 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:28:02 smithi052 ceph-mon[31955]: pgmap v65: 129 pgs: 129 active+clean; 9.9 MiB data, 381 MiB used, 1.0 TiB / 1.0 TiB avail; 601 KiB/s rd, 798 KiB/s wr, 63 op/s 2024-05-27T07:28:02.601 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:28:02 smithi173 ceph-mon[37000]: pgmap v65: 129 pgs: 129 active+clean; 9.9 MiB data, 381 MiB used, 1.0 TiB / 1.0 TiB avail; 601 KiB/s rd, 798 KiB/s wr, 63 op/s 2024-05-27T07:28:03.603 INFO:journalctl@ceph.mds.f.smithi173.stdout:May 27 07:28:03 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-f[67745]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: In function 'std::set SnapClient::filter(const std::set&) const' thread 7fe590f9e700 time 2024-05-27T07:28:03.223098+0000 2024-05-27T07:28:03.603 INFO:journalctl@ceph.mds.f.smithi173.stdout:May 27 07:28:03 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-f[67745]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: 197: FAILED ceph_assert(cached_version > 0) 2024-05-27T07:28:03.603 INFO:journalctl@ceph.mds.f.smithi173.stdout:May 27 07:28:03 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-f[67745]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:28:03.603 INFO:journalctl@ceph.mds.f.smithi173.stdout:May 27 07:28:03 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-f[67745]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x127) [0x7fe5a37e3ba6] 2024-05-27T07:28:03.603 INFO:journalctl@ceph.mds.f.smithi173.stdout:May 27 07:28:03 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-f[67745]: 2: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7fe5a37e3dd1] 2024-05-27T07:28:03.603 INFO:journalctl@ceph.mds.f.smithi173.stdout:May 27 07:28:03 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-f[67745]: 3: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x55578a6d623d] 2024-05-27T07:28:03.603 INFO:journalctl@ceph.mds.f.smithi173.stdout:May 27 07:28:03 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-f[67745]: 4: (SnapRealm::build_snap_set() const+0x323) [0x55578a6c1971] 2024-05-27T07:28:03.603 INFO:journalctl@ceph.mds.f.smithi173.stdout:May 27 07:28:03 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-f[67745]: 5: (SnapRealm::check_cache() const+0x13e) [0x55578a6bffa0] 2024-05-27T07:28:03.604 INFO:journalctl@ceph.mds.f.smithi173.stdout:May 27 07:28:03 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-f[67745]: 6: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x55578a6c0651] 2024-05-27T07:28:03.604 INFO:journalctl@ceph.mds.f.smithi173.stdout:May 27 07:28:03 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-f[67745]: 7: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x55578a655347] 2024-05-27T07:28:03.604 INFO:journalctl@ceph.mds.f.smithi173.stdout:May 27 07:28:03 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-f[67745]: 8: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x55578a35187e] 2024-05-27T07:28:03.604 INFO:journalctl@ceph.mds.f.smithi173.stdout:May 27 07:28:03 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-f[67745]: 9: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x2702) [0x55578a356e94] 2024-05-27T07:28:03.604 INFO:journalctl@ceph.mds.f.smithi173.stdout:May 27 07:28:03 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-f[67745]: 10: (EUpdate::replay(MDSRank*)+0x63) [0x55578a361969] 2024-05-27T07:28:03.604 INFO:journalctl@ceph.mds.f.smithi173.stdout:May 27 07:28:03 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-f[67745]: 11: (MDLog::_replay_thread()+0x1f38) [0x55578a6fa48c] 2024-05-27T07:28:03.604 INFO:journalctl@ceph.mds.f.smithi173.stdout:May 27 07:28:03 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-f[67745]: 12: (MDLog::ReplayThread::entry()+0x11) [0x55578a7039cb] 2024-05-27T07:28:03.604 INFO:journalctl@ceph.mds.f.smithi173.stdout:May 27 07:28:03 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-f[67745]: 13: (Thread::entry_wrapper()+0x43) [0x7fe5a37b272d] 2024-05-27T07:28:03.604 INFO:journalctl@ceph.mds.f.smithi173.stdout:May 27 07:28:03 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-f[67745]: 14: (Thread::_entry_func(void*)+0xd) [0x7fe5a37b2749] 2024-05-27T07:28:03.604 INFO:journalctl@ceph.mds.f.smithi173.stdout:May 27 07:28:03 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-f[67745]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7fe5a23561ca] 2024-05-27T07:28:03.604 INFO:journalctl@ceph.mds.f.smithi173.stdout:May 27 07:28:03 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-f[67745]: 16: clone() 2024-05-27T07:28:03.605 INFO:journalctl@ceph.mds.f.smithi173.stdout:May 27 07:28:03 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-f[67745]: *** Caught signal (Aborted) ** 2024-05-27T07:28:03.605 INFO:journalctl@ceph.mds.f.smithi173.stdout:May 27 07:28:03 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-f[67745]: in thread 7fe590f9e700 thread_name:md_log_replay 2024-05-27T07:28:03.605 INFO:journalctl@ceph.mds.f.smithi173.stdout:May 27 07:28:03 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-f[67745]: 2024-05-27T07:28:03.224+0000 7fe590f9e700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: In function 'std::set SnapClient::filter(const std::set&) const' thread 7fe590f9e700 time 2024-05-27T07:28:03.223098+0000 2024-05-27T07:28:03.605 INFO:journalctl@ceph.mds.f.smithi173.stdout:May 27 07:28:03 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-f[67745]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: 197: FAILED ceph_assert(cached_version > 0) 2024-05-27T07:28:03.605 INFO:journalctl@ceph.mds.f.smithi173.stdout:May 27 07:28:03 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-f[67745]: 2024-05-27T07:28:03.605 INFO:journalctl@ceph.mds.f.smithi173.stdout:May 27 07:28:03 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-f[67745]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:28:03.605 INFO:journalctl@ceph.mds.f.smithi173.stdout:May 27 07:28:03 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-f[67745]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x127) [0x7fe5a37e3ba6] 2024-05-27T07:28:03.605 INFO:journalctl@ceph.mds.f.smithi173.stdout:May 27 07:28:03 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-f[67745]: 2: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7fe5a37e3dd1] 2024-05-27T07:28:03.605 INFO:journalctl@ceph.mds.f.smithi173.stdout:May 27 07:28:03 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-f[67745]: 3: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x55578a6d623d] 2024-05-27T07:28:03.605 INFO:journalctl@ceph.mds.f.smithi173.stdout:May 27 07:28:03 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-f[67745]: 4: (SnapRealm::build_snap_set() const+0x323) [0x55578a6c1971] 2024-05-27T07:28:03.606 INFO:journalctl@ceph.mds.f.smithi173.stdout:May 27 07:28:03 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-f[67745]: 5: (SnapRealm::check_cache() const+0x13e) [0x55578a6bffa0] 2024-05-27T07:28:03.606 INFO:journalctl@ceph.mds.f.smithi173.stdout:May 27 07:28:03 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-f[67745]: 6: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x55578a6c0651] 2024-05-27T07:28:03.606 INFO:journalctl@ceph.mds.f.smithi173.stdout:May 27 07:28:03 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-f[67745]: 7: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x55578a655347] 2024-05-27T07:28:03.606 INFO:journalctl@ceph.mds.f.smithi173.stdout:May 27 07:28:03 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-f[67745]: 8: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x55578a35187e] 2024-05-27T07:28:03.606 INFO:journalctl@ceph.mds.f.smithi173.stdout:May 27 07:28:03 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-f[67745]: 9: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x2702) [0x55578a356e94] 2024-05-27T07:28:03.606 INFO:journalctl@ceph.mds.f.smithi173.stdout:May 27 07:28:03 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-f[67745]: 10: (EUpdate::replay(MDSRank*)+0x63) [0x55578a361969] 2024-05-27T07:28:03.606 INFO:journalctl@ceph.mds.f.smithi173.stdout:May 27 07:28:03 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-f[67745]: 11: (MDLog::_replay_thread()+0x1f38) [0x55578a6fa48c] 2024-05-27T07:28:03.606 INFO:journalctl@ceph.mds.f.smithi173.stdout:May 27 07:28:03 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-f[67745]: 12: (MDLog::ReplayThread::entry()+0x11) [0x55578a7039cb] 2024-05-27T07:28:03.606 INFO:journalctl@ceph.mds.f.smithi173.stdout:May 27 07:28:03 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-f[67745]: 13: (Thread::entry_wrapper()+0x43) [0x7fe5a37b272d] 2024-05-27T07:28:03.606 INFO:journalctl@ceph.mds.f.smithi173.stdout:May 27 07:28:03 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-f[67745]: 14: (Thread::_entry_func(void*)+0xd) [0x7fe5a37b2749] 2024-05-27T07:28:03.606 INFO:journalctl@ceph.mds.f.smithi173.stdout:May 27 07:28:03 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-f[67745]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7fe5a23561ca] 2024-05-27T07:28:03.607 INFO:journalctl@ceph.mds.f.smithi173.stdout:May 27 07:28:03 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-f[67745]: 16: clone() 2024-05-27T07:28:03.607 INFO:journalctl@ceph.mds.f.smithi173.stdout:May 27 07:28:03 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-f[67745]: 2024-05-27T07:28:03.607 INFO:journalctl@ceph.mds.f.smithi173.stdout:May 27 07:28:03 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-f[67745]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:28:03.607 INFO:journalctl@ceph.mds.f.smithi173.stdout:May 27 07:28:03 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-f[67745]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7fe5a2360d20] 2024-05-27T07:28:03.607 INFO:journalctl@ceph.mds.f.smithi173.stdout:May 27 07:28:03 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-f[67745]: 2: gsignal() 2024-05-27T07:28:03.607 INFO:journalctl@ceph.mds.f.smithi173.stdout:May 27 07:28:03 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-f[67745]: 3: abort() 2024-05-27T07:28:03.607 INFO:journalctl@ceph.mds.f.smithi173.stdout:May 27 07:28:03 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-f[67745]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x26e) [0x7fe5a37e3ced] 2024-05-27T07:28:03.607 INFO:journalctl@ceph.mds.f.smithi173.stdout:May 27 07:28:03 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-f[67745]: 5: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7fe5a37e3dd1] 2024-05-27T07:28:03.608 INFO:journalctl@ceph.mds.f.smithi173.stdout:May 27 07:28:03 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-f[67745]: 6: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x55578a6d623d] 2024-05-27T07:28:03.608 INFO:journalctl@ceph.mds.f.smithi173.stdout:May 27 07:28:03 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-f[67745]: 7: (SnapRealm::build_snap_set() const+0x323) [0x55578a6c1971] 2024-05-27T07:28:03.608 INFO:journalctl@ceph.mds.f.smithi173.stdout:May 27 07:28:03 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-f[67745]: 8: (SnapRealm::check_cache() const+0x13e) [0x55578a6bffa0] 2024-05-27T07:28:03.608 INFO:journalctl@ceph.mds.f.smithi173.stdout:May 27 07:28:03 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-f[67745]: 9: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x55578a6c0651] 2024-05-27T07:28:03.608 INFO:journalctl@ceph.mds.f.smithi173.stdout:May 27 07:28:03 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-f[67745]: 10: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x55578a655347] 2024-05-27T07:28:03.608 INFO:journalctl@ceph.mds.f.smithi173.stdout:May 27 07:28:03 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-f[67745]: 11: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x55578a35187e] 2024-05-27T07:28:03.608 INFO:journalctl@ceph.mds.f.smithi173.stdout:May 27 07:28:03 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-f[67745]: 12: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x2702) [0x55578a356e94] 2024-05-27T07:28:03.608 INFO:journalctl@ceph.mds.f.smithi173.stdout:May 27 07:28:03 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-f[67745]: 13: (EUpdate::replay(MDSRank*)+0x63) [0x55578a361969] 2024-05-27T07:28:03.609 INFO:journalctl@ceph.mds.f.smithi173.stdout:May 27 07:28:03 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-f[67745]: 14: (MDLog::_replay_thread()+0x1f38) [0x55578a6fa48c] 2024-05-27T07:28:03.609 INFO:journalctl@ceph.mds.f.smithi173.stdout:May 27 07:28:03 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-f[67745]: 15: (MDLog::ReplayThread::entry()+0x11) [0x55578a7039cb] 2024-05-27T07:28:03.609 INFO:journalctl@ceph.mds.f.smithi173.stdout:May 27 07:28:03 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-f[67745]: 16: (Thread::entry_wrapper()+0x43) [0x7fe5a37b272d] 2024-05-27T07:28:03.609 INFO:journalctl@ceph.mds.f.smithi173.stdout:May 27 07:28:03 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-f[67745]: 17: (Thread::_entry_func(void*)+0xd) [0x7fe5a37b2749] 2024-05-27T07:28:03.609 INFO:journalctl@ceph.mds.f.smithi173.stdout:May 27 07:28:03 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-f[67745]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7fe5a23561ca] 2024-05-27T07:28:03.609 INFO:journalctl@ceph.mds.f.smithi173.stdout:May 27 07:28:03 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-f[67745]: 19: clone() 2024-05-27T07:28:03.609 INFO:journalctl@ceph.mds.f.smithi173.stdout:May 27 07:28:03 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-f[67745]: 2024-05-27T07:28:03.227+0000 7fe590f9e700 -1 *** Caught signal (Aborted) ** 2024-05-27T07:28:03.609 INFO:journalctl@ceph.mds.f.smithi173.stdout:May 27 07:28:03 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-f[67745]: in thread 7fe590f9e700 thread_name:md_log_replay 2024-05-27T07:28:03.609 INFO:journalctl@ceph.mds.f.smithi173.stdout:May 27 07:28:03 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-f[67745]: 2024-05-27T07:28:03.609 INFO:journalctl@ceph.mds.f.smithi173.stdout:May 27 07:28:03 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-f[67745]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:28:03.609 INFO:journalctl@ceph.mds.f.smithi173.stdout:May 27 07:28:03 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-f[67745]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7fe5a2360d20] 2024-05-27T07:28:03.609 INFO:journalctl@ceph.mds.f.smithi173.stdout:May 27 07:28:03 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-f[67745]: 2: gsignal() 2024-05-27T07:28:03.609 INFO:journalctl@ceph.mds.f.smithi173.stdout:May 27 07:28:03 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-f[67745]: 3: abort() 2024-05-27T07:28:03.610 INFO:journalctl@ceph.mds.f.smithi173.stdout:May 27 07:28:03 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-f[67745]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x26e) [0x7fe5a37e3ced] 2024-05-27T07:28:03.610 INFO:journalctl@ceph.mds.f.smithi173.stdout:May 27 07:28:03 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-f[67745]: 5: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7fe5a37e3dd1] 2024-05-27T07:28:03.610 INFO:journalctl@ceph.mds.f.smithi173.stdout:May 27 07:28:03 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-f[67745]: 6: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x55578a6d623d] 2024-05-27T07:28:03.610 INFO:journalctl@ceph.mds.f.smithi173.stdout:May 27 07:28:03 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-f[67745]: 7: (SnapRealm::build_snap_set() const+0x323) [0x55578a6c1971] 2024-05-27T07:28:03.610 INFO:journalctl@ceph.mds.f.smithi173.stdout:May 27 07:28:03 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-f[67745]: 8: (SnapRealm::check_cache() const+0x13e) [0x55578a6bffa0] 2024-05-27T07:28:03.610 INFO:journalctl@ceph.mds.f.smithi173.stdout:May 27 07:28:03 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-f[67745]: 9: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x55578a6c0651] 2024-05-27T07:28:03.610 INFO:journalctl@ceph.mds.f.smithi173.stdout:May 27 07:28:03 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-f[67745]: 10: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x55578a655347] 2024-05-27T07:28:03.610 INFO:journalctl@ceph.mds.f.smithi173.stdout:May 27 07:28:03 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-f[67745]: 11: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x55578a35187e] 2024-05-27T07:28:03.610 INFO:journalctl@ceph.mds.f.smithi173.stdout:May 27 07:28:03 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-f[67745]: 12: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x2702) [0x55578a356e94] 2024-05-27T07:28:03.610 INFO:journalctl@ceph.mds.f.smithi173.stdout:May 27 07:28:03 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-f[67745]: 13: (EUpdate::replay(MDSRank*)+0x63) [0x55578a361969] 2024-05-27T07:28:03.610 INFO:journalctl@ceph.mds.f.smithi173.stdout:May 27 07:28:03 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-f[67745]: 14: (MDLog::_replay_thread()+0x1f38) [0x55578a6fa48c] 2024-05-27T07:28:03.610 INFO:journalctl@ceph.mds.f.smithi173.stdout:May 27 07:28:03 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-f[67745]: 15: (MDLog::ReplayThread::entry()+0x11) [0x55578a7039cb] 2024-05-27T07:28:03.610 INFO:journalctl@ceph.mds.f.smithi173.stdout:May 27 07:28:03 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-f[67745]: 16: (Thread::entry_wrapper()+0x43) [0x7fe5a37b272d] 2024-05-27T07:28:03.611 INFO:journalctl@ceph.mds.f.smithi173.stdout:May 27 07:28:03 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-f[67745]: 17: (Thread::_entry_func(void*)+0xd) [0x7fe5a37b2749] 2024-05-27T07:28:03.611 INFO:journalctl@ceph.mds.f.smithi173.stdout:May 27 07:28:03 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-f[67745]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7fe5a23561ca] 2024-05-27T07:28:03.611 INFO:journalctl@ceph.mds.f.smithi173.stdout:May 27 07:28:03 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-f[67745]: 19: clone() 2024-05-27T07:28:03.611 INFO:journalctl@ceph.mds.f.smithi173.stdout:May 27 07:28:03 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-f[67745]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-05-27T07:28:03.611 INFO:journalctl@ceph.mds.f.smithi173.stdout:May 27 07:28:03 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-f[67745]: 2024-05-27T07:28:03.611 INFO:journalctl@ceph.mds.f.smithi173.stdout:May 27 07:28:03 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-f[67745]: -1> 2024-05-27T07:28:03.224+0000 7fe590f9e700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: In function 'std::set SnapClient::filter(const std::set&) const' thread 7fe590f9e700 time 2024-05-27T07:28:03.223098+0000 2024-05-27T07:28:03.611 INFO:journalctl@ceph.mds.f.smithi173.stdout:May 27 07:28:03 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-f[67745]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: 197: FAILED ceph_assert(cached_version > 0) 2024-05-27T07:28:03.611 INFO:journalctl@ceph.mds.f.smithi173.stdout:May 27 07:28:03 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-f[67745]: 2024-05-27T07:28:03.611 INFO:journalctl@ceph.mds.f.smithi173.stdout:May 27 07:28:03 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-f[67745]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:28:03.611 INFO:journalctl@ceph.mds.f.smithi173.stdout:May 27 07:28:03 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-f[67745]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x127) [0x7fe5a37e3ba6] 2024-05-27T07:28:03.611 INFO:journalctl@ceph.mds.f.smithi173.stdout:May 27 07:28:03 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-f[67745]: 2: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7fe5a37e3dd1] 2024-05-27T07:28:03.611 INFO:journalctl@ceph.mds.f.smithi173.stdout:May 27 07:28:03 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-f[67745]: 3: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x55578a6d623d] 2024-05-27T07:28:03.611 INFO:journalctl@ceph.mds.f.smithi173.stdout:May 27 07:28:03 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-f[67745]: 4: (SnapRealm::build_snap_set() const+0x323) [0x55578a6c1971] 2024-05-27T07:28:03.612 INFO:journalctl@ceph.mds.f.smithi173.stdout:May 27 07:28:03 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-f[67745]: 5: (SnapRealm::check_cache() const+0x13e) [0x55578a6bffa0] 2024-05-27T07:28:03.612 INFO:journalctl@ceph.mds.f.smithi173.stdout:May 27 07:28:03 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-f[67745]: 6: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x55578a6c0651] 2024-05-27T07:28:03.612 INFO:journalctl@ceph.mds.f.smithi173.stdout:May 27 07:28:03 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-f[67745]: 7: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x55578a655347] 2024-05-27T07:28:03.612 INFO:journalctl@ceph.mds.f.smithi173.stdout:May 27 07:28:03 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-f[67745]: 8: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x55578a35187e] 2024-05-27T07:28:03.612 INFO:journalctl@ceph.mds.f.smithi173.stdout:May 27 07:28:03 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-f[67745]: 9: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x2702) [0x55578a356e94] 2024-05-27T07:28:03.612 INFO:journalctl@ceph.mds.f.smithi173.stdout:May 27 07:28:03 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-f[67745]: 10: (EUpdate::replay(MDSRank*)+0x63) [0x55578a361969] 2024-05-27T07:28:03.612 INFO:journalctl@ceph.mds.f.smithi173.stdout:May 27 07:28:03 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-f[67745]: 11: (MDLog::_replay_thread()+0x1f38) [0x55578a6fa48c] 2024-05-27T07:28:03.612 INFO:journalctl@ceph.mds.f.smithi173.stdout:May 27 07:28:03 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-f[67745]: 12: (MDLog::ReplayThread::entry()+0x11) [0x55578a7039cb] 2024-05-27T07:28:03.612 INFO:journalctl@ceph.mds.f.smithi173.stdout:May 27 07:28:03 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-f[67745]: 13: (Thread::entry_wrapper()+0x43) [0x7fe5a37b272d] 2024-05-27T07:28:03.612 INFO:journalctl@ceph.mds.f.smithi173.stdout:May 27 07:28:03 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-f[67745]: 14: (Thread::_entry_func(void*)+0xd) [0x7fe5a37b2749] 2024-05-27T07:28:03.612 INFO:journalctl@ceph.mds.f.smithi173.stdout:May 27 07:28:03 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-f[67745]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7fe5a23561ca] 2024-05-27T07:28:03.612 INFO:journalctl@ceph.mds.f.smithi173.stdout:May 27 07:28:03 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-f[67745]: 16: clone() 2024-05-27T07:28:03.612 INFO:journalctl@ceph.mds.f.smithi173.stdout:May 27 07:28:03 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-f[67745]: 2024-05-27T07:28:03.613 INFO:journalctl@ceph.mds.f.smithi173.stdout:May 27 07:28:03 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-f[67745]: 0> 2024-05-27T07:28:03.227+0000 7fe590f9e700 -1 *** Caught signal (Aborted) ** 2024-05-27T07:28:03.613 INFO:journalctl@ceph.mds.f.smithi173.stdout:May 27 07:28:03 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-f[67745]: in thread 7fe590f9e700 thread_name:md_log_replay 2024-05-27T07:28:03.613 INFO:journalctl@ceph.mds.f.smithi173.stdout:May 27 07:28:03 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-f[67745]: 2024-05-27T07:28:03.613 INFO:journalctl@ceph.mds.f.smithi173.stdout:May 27 07:28:03 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-f[67745]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:28:03.613 INFO:journalctl@ceph.mds.f.smithi173.stdout:May 27 07:28:03 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-f[67745]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7fe5a2360d20] 2024-05-27T07:28:03.613 INFO:journalctl@ceph.mds.f.smithi173.stdout:May 27 07:28:03 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-f[67745]: 2: gsignal() 2024-05-27T07:28:03.613 INFO:journalctl@ceph.mds.f.smithi173.stdout:May 27 07:28:03 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-f[67745]: 3: abort() 2024-05-27T07:28:03.613 INFO:journalctl@ceph.mds.f.smithi173.stdout:May 27 07:28:03 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-f[67745]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x26e) [0x7fe5a37e3ced] 2024-05-27T07:28:03.613 INFO:journalctl@ceph.mds.f.smithi173.stdout:May 27 07:28:03 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-f[67745]: 5: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7fe5a37e3dd1] 2024-05-27T07:28:03.613 INFO:journalctl@ceph.mds.f.smithi173.stdout:May 27 07:28:03 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-f[67745]: 6: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x55578a6d623d] 2024-05-27T07:28:03.613 INFO:journalctl@ceph.mds.f.smithi173.stdout:May 27 07:28:03 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-f[67745]: 7: (SnapRealm::build_snap_set() const+0x323) [0x55578a6c1971] 2024-05-27T07:28:03.613 INFO:journalctl@ceph.mds.f.smithi173.stdout:May 27 07:28:03 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-f[67745]: 8: (SnapRealm::check_cache() const+0x13e) [0x55578a6bffa0] 2024-05-27T07:28:03.613 INFO:journalctl@ceph.mds.f.smithi173.stdout:May 27 07:28:03 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-f[67745]: 9: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x55578a6c0651] 2024-05-27T07:28:03.614 INFO:journalctl@ceph.mds.f.smithi173.stdout:May 27 07:28:03 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-f[67745]: 10: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x55578a655347] 2024-05-27T07:28:03.614 INFO:journalctl@ceph.mds.f.smithi173.stdout:May 27 07:28:03 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-f[67745]: 11: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x55578a35187e] 2024-05-27T07:28:03.614 INFO:journalctl@ceph.mds.f.smithi173.stdout:May 27 07:28:03 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-f[67745]: 12: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x2702) [0x55578a356e94] 2024-05-27T07:28:03.614 INFO:journalctl@ceph.mds.f.smithi173.stdout:May 27 07:28:03 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-f[67745]: 13: (EUpdate::replay(MDSRank*)+0x63) [0x55578a361969] 2024-05-27T07:28:03.614 INFO:journalctl@ceph.mds.f.smithi173.stdout:May 27 07:28:03 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-f[67745]: 14: (MDLog::_replay_thread()+0x1f38) [0x55578a6fa48c] 2024-05-27T07:28:03.614 INFO:journalctl@ceph.mds.f.smithi173.stdout:May 27 07:28:03 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-f[67745]: 15: (MDLog::ReplayThread::entry()+0x11) [0x55578a7039cb] 2024-05-27T07:28:03.614 INFO:journalctl@ceph.mds.f.smithi173.stdout:May 27 07:28:03 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-f[67745]: 16: (Thread::entry_wrapper()+0x43) [0x7fe5a37b272d] 2024-05-27T07:28:03.614 INFO:journalctl@ceph.mds.f.smithi173.stdout:May 27 07:28:03 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-f[67745]: 17: (Thread::_entry_func(void*)+0xd) [0x7fe5a37b2749] 2024-05-27T07:28:03.614 INFO:journalctl@ceph.mds.f.smithi173.stdout:May 27 07:28:03 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-f[67745]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7fe5a23561ca] 2024-05-27T07:28:03.614 INFO:journalctl@ceph.mds.f.smithi173.stdout:May 27 07:28:03 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-f[67745]: 19: clone() 2024-05-27T07:28:03.614 INFO:journalctl@ceph.mds.f.smithi173.stdout:May 27 07:28:03 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-f[67745]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-05-27T07:28:03.614 INFO:journalctl@ceph.mds.f.smithi173.stdout:May 27 07:28:03 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-f[67745]: 2024-05-27T07:28:03.614 INFO:journalctl@ceph.mds.f.smithi173.stdout:May 27 07:28:03 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-f[67745]: -1> 2024-05-27T07:28:03.224+0000 7fe590f9e700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: In function 'std::set SnapClient::filter(const std::set&) const' thread 7fe590f9e700 time 2024-05-27T07:28:03.223098+0000 2024-05-27T07:28:03.615 INFO:journalctl@ceph.mds.f.smithi173.stdout:May 27 07:28:03 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-f[67745]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: 197: FAILED ceph_assert(cached_version > 0) 2024-05-27T07:28:03.615 INFO:journalctl@ceph.mds.f.smithi173.stdout:May 27 07:28:03 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-f[67745]: 2024-05-27T07:28:03.615 INFO:journalctl@ceph.mds.f.smithi173.stdout:May 27 07:28:03 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-f[67745]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:28:03.615 INFO:journalctl@ceph.mds.f.smithi173.stdout:May 27 07:28:03 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-f[67745]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x127) [0x7fe5a37e3ba6] 2024-05-27T07:28:03.615 INFO:journalctl@ceph.mds.f.smithi173.stdout:May 27 07:28:03 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-f[67745]: 2: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7fe5a37e3dd1] 2024-05-27T07:28:03.615 INFO:journalctl@ceph.mds.f.smithi173.stdout:May 27 07:28:03 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-f[67745]: 3: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x55578a6d623d] 2024-05-27T07:28:03.615 INFO:journalctl@ceph.mds.f.smithi173.stdout:May 27 07:28:03 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-f[67745]: 4: (SnapRealm::build_snap_set() const+0x323) [0x55578a6c1971] 2024-05-27T07:28:03.615 INFO:journalctl@ceph.mds.f.smithi173.stdout:May 27 07:28:03 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-f[67745]: 5: (SnapRealm::check_cache() const+0x13e) [0x55578a6bffa0] 2024-05-27T07:28:03.615 INFO:journalctl@ceph.mds.f.smithi173.stdout:May 27 07:28:03 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-f[67745]: 6: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x55578a6c0651] 2024-05-27T07:28:03.615 INFO:journalctl@ceph.mds.f.smithi173.stdout:May 27 07:28:03 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-f[67745]: 7: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x55578a655347] 2024-05-27T07:28:03.615 INFO:journalctl@ceph.mds.f.smithi173.stdout:May 27 07:28:03 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-f[67745]: 8: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x55578a35187e] 2024-05-27T07:28:03.615 INFO:journalctl@ceph.mds.f.smithi173.stdout:May 27 07:28:03 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-f[67745]: 9: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x2702) [0x55578a356e94] 2024-05-27T07:28:03.615 INFO:journalctl@ceph.mds.f.smithi173.stdout:May 27 07:28:03 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-f[67745]: 10: (EUpdate::replay(MDSRank*)+0x63) [0x55578a361969] 2024-05-27T07:28:03.615 INFO:journalctl@ceph.mds.f.smithi173.stdout:May 27 07:28:03 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-f[67745]: 11: (MDLog::_replay_thread()+0x1f38) [0x55578a6fa48c] 2024-05-27T07:28:03.616 INFO:journalctl@ceph.mds.f.smithi173.stdout:May 27 07:28:03 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-f[67745]: 12: (MDLog::ReplayThread::entry()+0x11) [0x55578a7039cb] 2024-05-27T07:28:03.616 INFO:journalctl@ceph.mds.f.smithi173.stdout:May 27 07:28:03 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-f[67745]: 13: (Thread::entry_wrapper()+0x43) [0x7fe5a37b272d] 2024-05-27T07:28:03.616 INFO:journalctl@ceph.mds.f.smithi173.stdout:May 27 07:28:03 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-f[67745]: 14: (Thread::_entry_func(void*)+0xd) [0x7fe5a37b2749] 2024-05-27T07:28:03.616 INFO:journalctl@ceph.mds.f.smithi173.stdout:May 27 07:28:03 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-f[67745]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7fe5a23561ca] 2024-05-27T07:28:03.616 INFO:journalctl@ceph.mds.f.smithi173.stdout:May 27 07:28:03 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-f[67745]: 16: clone() 2024-05-27T07:28:03.616 INFO:journalctl@ceph.mds.f.smithi173.stdout:May 27 07:28:03 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-f[67745]: 2024-05-27T07:28:03.616 INFO:journalctl@ceph.mds.f.smithi173.stdout:May 27 07:28:03 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-f[67745]: 0> 2024-05-27T07:28:03.227+0000 7fe590f9e700 -1 *** Caught signal (Aborted) ** 2024-05-27T07:28:03.616 INFO:journalctl@ceph.mds.f.smithi173.stdout:May 27 07:28:03 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-f[67745]: in thread 7fe590f9e700 thread_name:md_log_replay 2024-05-27T07:28:03.616 INFO:journalctl@ceph.mds.f.smithi173.stdout:May 27 07:28:03 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-f[67745]: 2024-05-27T07:28:03.616 INFO:journalctl@ceph.mds.f.smithi173.stdout:May 27 07:28:03 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-f[67745]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:28:03.616 INFO:journalctl@ceph.mds.f.smithi173.stdout:May 27 07:28:03 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-f[67745]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7fe5a2360d20] 2024-05-27T07:28:03.616 INFO:journalctl@ceph.mds.f.smithi173.stdout:May 27 07:28:03 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-f[67745]: 2: gsignal() 2024-05-27T07:28:03.616 INFO:journalctl@ceph.mds.f.smithi173.stdout:May 27 07:28:03 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-f[67745]: 3: abort() 2024-05-27T07:28:03.617 INFO:journalctl@ceph.mds.f.smithi173.stdout:May 27 07:28:03 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-f[67745]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x26e) [0x7fe5a37e3ced] 2024-05-27T07:28:03.617 INFO:journalctl@ceph.mds.f.smithi173.stdout:May 27 07:28:03 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-f[67745]: 5: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7fe5a37e3dd1] 2024-05-27T07:28:03.617 INFO:journalctl@ceph.mds.f.smithi173.stdout:May 27 07:28:03 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-f[67745]: 6: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x55578a6d623d] 2024-05-27T07:28:03.617 INFO:journalctl@ceph.mds.f.smithi173.stdout:May 27 07:28:03 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-f[67745]: 7: (SnapRealm::build_snap_set() const+0x323) [0x55578a6c1971] 2024-05-27T07:28:03.617 INFO:journalctl@ceph.mds.f.smithi173.stdout:May 27 07:28:03 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-f[67745]: 8: (SnapRealm::check_cache() const+0x13e) [0x55578a6bffa0] 2024-05-27T07:28:03.617 INFO:journalctl@ceph.mds.f.smithi173.stdout:May 27 07:28:03 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-f[67745]: 9: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x55578a6c0651] 2024-05-27T07:28:03.617 INFO:journalctl@ceph.mds.f.smithi173.stdout:May 27 07:28:03 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-f[67745]: 10: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x55578a655347] 2024-05-27T07:28:03.617 INFO:journalctl@ceph.mds.f.smithi173.stdout:May 27 07:28:03 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-f[67745]: 11: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x55578a35187e] 2024-05-27T07:28:03.617 INFO:journalctl@ceph.mds.f.smithi173.stdout:May 27 07:28:03 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-f[67745]: 12: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x2702) [0x55578a356e94] 2024-05-27T07:28:03.617 INFO:journalctl@ceph.mds.f.smithi173.stdout:May 27 07:28:03 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-f[67745]: 13: (EUpdate::replay(MDSRank*)+0x63) [0x55578a361969] 2024-05-27T07:28:03.617 INFO:journalctl@ceph.mds.f.smithi173.stdout:May 27 07:28:03 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-f[67745]: 14: (MDLog::_replay_thread()+0x1f38) [0x55578a6fa48c] 2024-05-27T07:28:03.617 INFO:journalctl@ceph.mds.f.smithi173.stdout:May 27 07:28:03 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-f[67745]: 15: (MDLog::ReplayThread::entry()+0x11) [0x55578a7039cb] 2024-05-27T07:28:03.617 INFO:journalctl@ceph.mds.f.smithi173.stdout:May 27 07:28:03 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-f[67745]: 16: (Thread::entry_wrapper()+0x43) [0x7fe5a37b272d] 2024-05-27T07:28:03.618 INFO:journalctl@ceph.mds.f.smithi173.stdout:May 27 07:28:03 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-f[67745]: 17: (Thread::_entry_func(void*)+0xd) [0x7fe5a37b2749] 2024-05-27T07:28:03.618 INFO:journalctl@ceph.mds.f.smithi173.stdout:May 27 07:28:03 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-f[67745]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7fe5a23561ca] 2024-05-27T07:28:03.618 INFO:journalctl@ceph.mds.f.smithi173.stdout:May 27 07:28:03 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-f[67745]: 19: clone() 2024-05-27T07:28:03.618 INFO:journalctl@ceph.mds.f.smithi173.stdout:May 27 07:28:03 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-f[67745]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-05-27T07:28:03.618 INFO:journalctl@ceph.mds.f.smithi173.stdout:May 27 07:28:03 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-f[67745]: 2024-05-27T07:28:03.618 INFO:journalctl@ceph.mds.f.smithi173.stdout:May 27 07:28:03 smithi173 podman[72175]: 2024-05-27 07:28:03.284618789 +0000 UTC m=+0.019459506 container died e053aecb986487270fcbb179981fb8e9a24093408f8a147f12a6f004f4318f3d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-f, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, org.label-schema.build-date=20240521, GIT_CLEAN=True, ceph=True, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS) 2024-05-27T07:28:04.376 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:28:04 smithi173 ceph-mon[37000]: pgmap v66: 129 pgs: 129 active+clean; 44 MiB data, 435 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 4.2 MiB/s wr, 505 op/s 2024-05-27T07:28:04.377 INFO:journalctl@ceph.mds.f.smithi173.stdout:May 27 07:28:04 smithi173 podman[72175]: 2024-05-27 07:28:04.131815285 +0000 UTC m=+0.866656021 container remove e053aecb986487270fcbb179981fb8e9a24093408f8a147f12a6f004f4318f3d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-f, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240521, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, ceph=True, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, io.buildah.version=1.33.5, GIT_CLEAN=True) 2024-05-27T07:28:04.377 INFO:journalctl@ceph.mds.f.smithi173.stdout:May 27 07:28:04 smithi173 systemd[1]: ceph-73df2636-1bf8-11ef-bc9b-c7b262605968@mds.f.service: Main process exited, code=exited, status=134/n/a 2024-05-27T07:28:04.439 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:28:04 smithi105 ceph-mon[35977]: pgmap v66: 129 pgs: 129 active+clean; 44 MiB data, 435 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 4.2 MiB/s wr, 505 op/s 2024-05-27T07:28:04.479 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:28:04 smithi052 ceph-mon[31955]: pgmap v66: 129 pgs: 129 active+clean; 44 MiB data, 435 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 4.2 MiB/s wr, 505 op/s 2024-05-27T07:28:04.851 INFO:journalctl@ceph.mds.f.smithi173.stdout:May 27 07:28:04 smithi173 systemd[1]: ceph-73df2636-1bf8-11ef-bc9b-c7b262605968@mds.f.service: Failed with result 'exit-code'. 2024-05-27T07:28:04.852 INFO:journalctl@ceph.mds.f.smithi173.stdout:May 27 07:28:04 smithi173 systemd[1]: ceph-73df2636-1bf8-11ef-bc9b-c7b262605968@mds.f.service: Consumed 2.476s CPU time. 2024-05-27T07:28:05.205 INFO:tasks.workunit.client.0.smithi052.stdout: 12 147942 610 96513 695 63116 985 2024-05-27T07:28:06.439 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:28:06 smithi105 ceph-mon[35977]: pgmap v67: 129 pgs: 129 active+clean; 44 MiB data, 435 MiB used, 1.0 TiB / 1.0 TiB avail; 981 KiB/s rd, 3.9 MiB/s wr, 458 op/s 2024-05-27T07:28:06.480 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:28:06 smithi052 ceph-mon[31955]: pgmap v67: 129 pgs: 129 active+clean; 44 MiB data, 435 MiB used, 1.0 TiB / 1.0 TiB avail; 981 KiB/s rd, 3.9 MiB/s wr, 458 op/s 2024-05-27T07:28:06.481 INFO:journalctl@ceph.mds.g.smithi052.stdout:May 27 07:28:06 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-g[73967]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: In function 'std::set SnapClient::filter(const std::set&) const' thread 7fd401699700 time 2024-05-27T07:28:06.367889+0000 2024-05-27T07:28:06.481 INFO:journalctl@ceph.mds.g.smithi052.stdout:May 27 07:28:06 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-g[73967]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: 197: FAILED ceph_assert(cached_version > 0) 2024-05-27T07:28:06.481 INFO:journalctl@ceph.mds.g.smithi052.stdout:May 27 07:28:06 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-g[73967]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:28:06.481 INFO:journalctl@ceph.mds.g.smithi052.stdout:May 27 07:28:06 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-g[73967]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x127) [0x7fd413edeba6] 2024-05-27T07:28:06.481 INFO:journalctl@ceph.mds.g.smithi052.stdout:May 27 07:28:06 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-g[73967]: 2: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7fd413ededd1] 2024-05-27T07:28:06.481 INFO:journalctl@ceph.mds.g.smithi052.stdout:May 27 07:28:06 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-g[73967]: 3: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x55a762d6123d] 2024-05-27T07:28:06.482 INFO:journalctl@ceph.mds.g.smithi052.stdout:May 27 07:28:06 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-g[73967]: 4: (SnapRealm::build_snap_set() const+0x323) [0x55a762d4c971] 2024-05-27T07:28:06.482 INFO:journalctl@ceph.mds.g.smithi052.stdout:May 27 07:28:06 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-g[73967]: 5: (SnapRealm::check_cache() const+0x13e) [0x55a762d4afa0] 2024-05-27T07:28:06.482 INFO:journalctl@ceph.mds.g.smithi052.stdout:May 27 07:28:06 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-g[73967]: 6: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x55a762d4b651] 2024-05-27T07:28:06.482 INFO:journalctl@ceph.mds.g.smithi052.stdout:May 27 07:28:06 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-g[73967]: 7: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x55a762ce0347] 2024-05-27T07:28:06.482 INFO:journalctl@ceph.mds.g.smithi052.stdout:May 27 07:28:06 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-g[73967]: 8: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x55a7629dc87e] 2024-05-27T07:28:06.482 INFO:journalctl@ceph.mds.g.smithi052.stdout:May 27 07:28:06 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-g[73967]: 9: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x2702) [0x55a7629e1e94] 2024-05-27T07:28:06.483 INFO:journalctl@ceph.mds.g.smithi052.stdout:May 27 07:28:06 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-g[73967]: 10: (EUpdate::replay(MDSRank*)+0x63) [0x55a7629ec969] 2024-05-27T07:28:06.483 INFO:journalctl@ceph.mds.g.smithi052.stdout:May 27 07:28:06 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-g[73967]: 11: (MDLog::_replay_thread()+0x1f38) [0x55a762d8548c] 2024-05-27T07:28:06.483 INFO:journalctl@ceph.mds.g.smithi052.stdout:May 27 07:28:06 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-g[73967]: 12: (MDLog::ReplayThread::entry()+0x11) [0x55a762d8e9cb] 2024-05-27T07:28:06.483 INFO:journalctl@ceph.mds.g.smithi052.stdout:May 27 07:28:06 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-g[73967]: 13: (Thread::entry_wrapper()+0x43) [0x7fd413ead72d] 2024-05-27T07:28:06.483 INFO:journalctl@ceph.mds.g.smithi052.stdout:May 27 07:28:06 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-g[73967]: 14: (Thread::_entry_func(void*)+0xd) [0x7fd413ead749] 2024-05-27T07:28:06.484 INFO:journalctl@ceph.mds.g.smithi052.stdout:May 27 07:28:06 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-g[73967]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7fd412a511ca] 2024-05-27T07:28:06.484 INFO:journalctl@ceph.mds.g.smithi052.stdout:May 27 07:28:06 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-g[73967]: 16: clone() 2024-05-27T07:28:06.484 INFO:journalctl@ceph.mds.g.smithi052.stdout:May 27 07:28:06 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-g[73967]: *** Caught signal (Aborted) ** 2024-05-27T07:28:06.484 INFO:journalctl@ceph.mds.g.smithi052.stdout:May 27 07:28:06 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-g[73967]: in thread 7fd401699700 thread_name:md_log_replay 2024-05-27T07:28:06.484 INFO:journalctl@ceph.mds.g.smithi052.stdout:May 27 07:28:06 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-g[73967]: 2024-05-27T07:28:06.369+0000 7fd401699700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: In function 'std::set SnapClient::filter(const std::set&) const' thread 7fd401699700 time 2024-05-27T07:28:06.367889+0000 2024-05-27T07:28:06.484 INFO:journalctl@ceph.mds.g.smithi052.stdout:May 27 07:28:06 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-g[73967]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: 197: FAILED ceph_assert(cached_version > 0) 2024-05-27T07:28:06.485 INFO:journalctl@ceph.mds.g.smithi052.stdout:May 27 07:28:06 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-g[73967]: 2024-05-27T07:28:06.485 INFO:journalctl@ceph.mds.g.smithi052.stdout:May 27 07:28:06 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-g[73967]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:28:06.485 INFO:journalctl@ceph.mds.g.smithi052.stdout:May 27 07:28:06 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-g[73967]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x127) [0x7fd413edeba6] 2024-05-27T07:28:06.485 INFO:journalctl@ceph.mds.g.smithi052.stdout:May 27 07:28:06 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-g[73967]: 2: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7fd413ededd1] 2024-05-27T07:28:06.485 INFO:journalctl@ceph.mds.g.smithi052.stdout:May 27 07:28:06 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-g[73967]: 3: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x55a762d6123d] 2024-05-27T07:28:06.486 INFO:journalctl@ceph.mds.g.smithi052.stdout:May 27 07:28:06 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-g[73967]: 4: (SnapRealm::build_snap_set() const+0x323) [0x55a762d4c971] 2024-05-27T07:28:06.486 INFO:journalctl@ceph.mds.g.smithi052.stdout:May 27 07:28:06 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-g[73967]: 5: (SnapRealm::check_cache() const+0x13e) [0x55a762d4afa0] 2024-05-27T07:28:06.486 INFO:journalctl@ceph.mds.g.smithi052.stdout:May 27 07:28:06 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-g[73967]: 6: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x55a762d4b651] 2024-05-27T07:28:06.486 INFO:journalctl@ceph.mds.g.smithi052.stdout:May 27 07:28:06 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-g[73967]: 7: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x55a762ce0347] 2024-05-27T07:28:06.487 INFO:journalctl@ceph.mds.g.smithi052.stdout:May 27 07:28:06 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-g[73967]: 8: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x55a7629dc87e] 2024-05-27T07:28:06.487 INFO:journalctl@ceph.mds.g.smithi052.stdout:May 27 07:28:06 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-g[73967]: 9: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x2702) [0x55a7629e1e94] 2024-05-27T07:28:06.487 INFO:journalctl@ceph.mds.g.smithi052.stdout:May 27 07:28:06 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-g[73967]: 10: (EUpdate::replay(MDSRank*)+0x63) [0x55a7629ec969] 2024-05-27T07:28:06.487 INFO:journalctl@ceph.mds.g.smithi052.stdout:May 27 07:28:06 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-g[73967]: 11: (MDLog::_replay_thread()+0x1f38) [0x55a762d8548c] 2024-05-27T07:28:06.488 INFO:journalctl@ceph.mds.g.smithi052.stdout:May 27 07:28:06 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-g[73967]: 12: (MDLog::ReplayThread::entry()+0x11) [0x55a762d8e9cb] 2024-05-27T07:28:06.488 INFO:journalctl@ceph.mds.g.smithi052.stdout:May 27 07:28:06 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-g[73967]: 13: (Thread::entry_wrapper()+0x43) [0x7fd413ead72d] 2024-05-27T07:28:06.488 INFO:journalctl@ceph.mds.g.smithi052.stdout:May 27 07:28:06 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-g[73967]: 14: (Thread::_entry_func(void*)+0xd) [0x7fd413ead749] 2024-05-27T07:28:06.488 INFO:journalctl@ceph.mds.g.smithi052.stdout:May 27 07:28:06 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-g[73967]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7fd412a511ca] 2024-05-27T07:28:06.489 INFO:journalctl@ceph.mds.g.smithi052.stdout:May 27 07:28:06 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-g[73967]: 16: clone() 2024-05-27T07:28:06.489 INFO:journalctl@ceph.mds.g.smithi052.stdout:May 27 07:28:06 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-g[73967]: 2024-05-27T07:28:06.489 INFO:journalctl@ceph.mds.g.smithi052.stdout:May 27 07:28:06 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-g[73967]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:28:06.489 INFO:journalctl@ceph.mds.g.smithi052.stdout:May 27 07:28:06 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-g[73967]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7fd412a5bd20] 2024-05-27T07:28:06.489 INFO:journalctl@ceph.mds.g.smithi052.stdout:May 27 07:28:06 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-g[73967]: 2: gsignal() 2024-05-27T07:28:06.490 INFO:journalctl@ceph.mds.g.smithi052.stdout:May 27 07:28:06 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-g[73967]: 3: abort() 2024-05-27T07:28:06.490 INFO:journalctl@ceph.mds.g.smithi052.stdout:May 27 07:28:06 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-g[73967]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x26e) [0x7fd413edeced] 2024-05-27T07:28:06.490 INFO:journalctl@ceph.mds.g.smithi052.stdout:May 27 07:28:06 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-g[73967]: 5: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7fd413ededd1] 2024-05-27T07:28:06.490 INFO:journalctl@ceph.mds.g.smithi052.stdout:May 27 07:28:06 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-g[73967]: 6: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x55a762d6123d] 2024-05-27T07:28:06.491 INFO:journalctl@ceph.mds.g.smithi052.stdout:May 27 07:28:06 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-g[73967]: 7: (SnapRealm::build_snap_set() const+0x323) [0x55a762d4c971] 2024-05-27T07:28:06.491 INFO:journalctl@ceph.mds.g.smithi052.stdout:May 27 07:28:06 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-g[73967]: 8: (SnapRealm::check_cache() const+0x13e) [0x55a762d4afa0] 2024-05-27T07:28:06.491 INFO:journalctl@ceph.mds.g.smithi052.stdout:May 27 07:28:06 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-g[73967]: 9: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x55a762d4b651] 2024-05-27T07:28:06.491 INFO:journalctl@ceph.mds.g.smithi052.stdout:May 27 07:28:06 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-g[73967]: 10: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x55a762ce0347] 2024-05-27T07:28:06.491 INFO:journalctl@ceph.mds.g.smithi052.stdout:May 27 07:28:06 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-g[73967]: 11: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x55a7629dc87e] 2024-05-27T07:28:06.492 INFO:journalctl@ceph.mds.g.smithi052.stdout:May 27 07:28:06 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-g[73967]: 12: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x2702) [0x55a7629e1e94] 2024-05-27T07:28:06.492 INFO:journalctl@ceph.mds.g.smithi052.stdout:May 27 07:28:06 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-g[73967]: 13: (EUpdate::replay(MDSRank*)+0x63) [0x55a7629ec969] 2024-05-27T07:28:06.492 INFO:journalctl@ceph.mds.g.smithi052.stdout:May 27 07:28:06 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-g[73967]: 14: (MDLog::_replay_thread()+0x1f38) [0x55a762d8548c] 2024-05-27T07:28:06.492 INFO:journalctl@ceph.mds.g.smithi052.stdout:May 27 07:28:06 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-g[73967]: 15: (MDLog::ReplayThread::entry()+0x11) [0x55a762d8e9cb] 2024-05-27T07:28:06.493 INFO:journalctl@ceph.mds.g.smithi052.stdout:May 27 07:28:06 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-g[73967]: 16: (Thread::entry_wrapper()+0x43) [0x7fd413ead72d] 2024-05-27T07:28:06.493 INFO:journalctl@ceph.mds.g.smithi052.stdout:May 27 07:28:06 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-g[73967]: 17: (Thread::_entry_func(void*)+0xd) [0x7fd413ead749] 2024-05-27T07:28:06.493 INFO:journalctl@ceph.mds.g.smithi052.stdout:May 27 07:28:06 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-g[73967]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7fd412a511ca] 2024-05-27T07:28:06.493 INFO:journalctl@ceph.mds.g.smithi052.stdout:May 27 07:28:06 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-g[73967]: 19: clone() 2024-05-27T07:28:06.494 INFO:journalctl@ceph.mds.g.smithi052.stdout:May 27 07:28:06 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-g[73967]: 2024-05-27T07:28:06.372+0000 7fd401699700 -1 *** Caught signal (Aborted) ** 2024-05-27T07:28:06.494 INFO:journalctl@ceph.mds.g.smithi052.stdout:May 27 07:28:06 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-g[73967]: in thread 7fd401699700 thread_name:md_log_replay 2024-05-27T07:28:06.494 INFO:journalctl@ceph.mds.g.smithi052.stdout:May 27 07:28:06 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-g[73967]: 2024-05-27T07:28:06.494 INFO:journalctl@ceph.mds.g.smithi052.stdout:May 27 07:28:06 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-g[73967]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:28:06.495 INFO:journalctl@ceph.mds.g.smithi052.stdout:May 27 07:28:06 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-g[73967]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7fd412a5bd20] 2024-05-27T07:28:06.495 INFO:journalctl@ceph.mds.g.smithi052.stdout:May 27 07:28:06 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-g[73967]: 2: gsignal() 2024-05-27T07:28:06.495 INFO:journalctl@ceph.mds.g.smithi052.stdout:May 27 07:28:06 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-g[73967]: 3: abort() 2024-05-27T07:28:06.495 INFO:journalctl@ceph.mds.g.smithi052.stdout:May 27 07:28:06 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-g[73967]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x26e) [0x7fd413edeced] 2024-05-27T07:28:06.496 INFO:journalctl@ceph.mds.g.smithi052.stdout:May 27 07:28:06 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-g[73967]: 5: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7fd413ededd1] 2024-05-27T07:28:06.496 INFO:journalctl@ceph.mds.g.smithi052.stdout:May 27 07:28:06 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-g[73967]: 6: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x55a762d6123d] 2024-05-27T07:28:06.496 INFO:journalctl@ceph.mds.g.smithi052.stdout:May 27 07:28:06 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-g[73967]: 7: (SnapRealm::build_snap_set() const+0x323) [0x55a762d4c971] 2024-05-27T07:28:06.496 INFO:journalctl@ceph.mds.g.smithi052.stdout:May 27 07:28:06 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-g[73967]: 8: (SnapRealm::check_cache() const+0x13e) [0x55a762d4afa0] 2024-05-27T07:28:06.497 INFO:journalctl@ceph.mds.g.smithi052.stdout:May 27 07:28:06 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-g[73967]: 9: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x55a762d4b651] 2024-05-27T07:28:06.497 INFO:journalctl@ceph.mds.g.smithi052.stdout:May 27 07:28:06 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-g[73967]: 10: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x55a762ce0347] 2024-05-27T07:28:06.499 INFO:journalctl@ceph.mds.g.smithi052.stdout:May 27 07:28:06 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-g[73967]: 11: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x55a7629dc87e] 2024-05-27T07:28:06.499 INFO:journalctl@ceph.mds.g.smithi052.stdout:May 27 07:28:06 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-g[73967]: 12: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x2702) [0x55a7629e1e94] 2024-05-27T07:28:06.499 INFO:journalctl@ceph.mds.g.smithi052.stdout:May 27 07:28:06 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-g[73967]: 13: (EUpdate::replay(MDSRank*)+0x63) [0x55a7629ec969] 2024-05-27T07:28:06.499 INFO:journalctl@ceph.mds.g.smithi052.stdout:May 27 07:28:06 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-g[73967]: 14: (MDLog::_replay_thread()+0x1f38) [0x55a762d8548c] 2024-05-27T07:28:06.500 INFO:journalctl@ceph.mds.g.smithi052.stdout:May 27 07:28:06 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-g[73967]: 15: (MDLog::ReplayThread::entry()+0x11) [0x55a762d8e9cb] 2024-05-27T07:28:06.500 INFO:journalctl@ceph.mds.g.smithi052.stdout:May 27 07:28:06 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-g[73967]: 16: (Thread::entry_wrapper()+0x43) [0x7fd413ead72d] 2024-05-27T07:28:06.500 INFO:journalctl@ceph.mds.g.smithi052.stdout:May 27 07:28:06 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-g[73967]: 17: (Thread::_entry_func(void*)+0xd) [0x7fd413ead749] 2024-05-27T07:28:06.500 INFO:journalctl@ceph.mds.g.smithi052.stdout:May 27 07:28:06 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-g[73967]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7fd412a511ca] 2024-05-27T07:28:06.500 INFO:journalctl@ceph.mds.g.smithi052.stdout:May 27 07:28:06 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-g[73967]: 19: clone() 2024-05-27T07:28:06.501 INFO:journalctl@ceph.mds.g.smithi052.stdout:May 27 07:28:06 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-g[73967]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-05-27T07:28:06.501 INFO:journalctl@ceph.mds.g.smithi052.stdout:May 27 07:28:06 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-g[73967]: 2024-05-27T07:28:06.501 INFO:journalctl@ceph.mds.g.smithi052.stdout:May 27 07:28:06 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-g[73967]: -1> 2024-05-27T07:28:06.369+0000 7fd401699700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: In function 'std::set SnapClient::filter(const std::set&) const' thread 7fd401699700 time 2024-05-27T07:28:06.367889+0000 2024-05-27T07:28:06.501 INFO:journalctl@ceph.mds.g.smithi052.stdout:May 27 07:28:06 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-g[73967]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: 197: FAILED ceph_assert(cached_version > 0) 2024-05-27T07:28:06.502 INFO:journalctl@ceph.mds.g.smithi052.stdout:May 27 07:28:06 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-g[73967]: 2024-05-27T07:28:06.502 INFO:journalctl@ceph.mds.g.smithi052.stdout:May 27 07:28:06 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-g[73967]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:28:06.502 INFO:journalctl@ceph.mds.g.smithi052.stdout:May 27 07:28:06 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-g[73967]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x127) [0x7fd413edeba6] 2024-05-27T07:28:06.502 INFO:journalctl@ceph.mds.g.smithi052.stdout:May 27 07:28:06 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-g[73967]: 2: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7fd413ededd1] 2024-05-27T07:28:06.502 INFO:journalctl@ceph.mds.g.smithi052.stdout:May 27 07:28:06 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-g[73967]: 3: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x55a762d6123d] 2024-05-27T07:28:06.503 INFO:journalctl@ceph.mds.g.smithi052.stdout:May 27 07:28:06 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-g[73967]: 4: (SnapRealm::build_snap_set() const+0x323) [0x55a762d4c971] 2024-05-27T07:28:06.503 INFO:journalctl@ceph.mds.g.smithi052.stdout:May 27 07:28:06 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-g[73967]: 5: (SnapRealm::check_cache() const+0x13e) [0x55a762d4afa0] 2024-05-27T07:28:06.503 INFO:journalctl@ceph.mds.g.smithi052.stdout:May 27 07:28:06 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-g[73967]: 6: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x55a762d4b651] 2024-05-27T07:28:06.503 INFO:journalctl@ceph.mds.g.smithi052.stdout:May 27 07:28:06 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-g[73967]: 7: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x55a762ce0347] 2024-05-27T07:28:06.504 INFO:journalctl@ceph.mds.g.smithi052.stdout:May 27 07:28:06 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-g[73967]: 8: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x55a7629dc87e] 2024-05-27T07:28:06.504 INFO:journalctl@ceph.mds.g.smithi052.stdout:May 27 07:28:06 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-g[73967]: 9: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x2702) [0x55a7629e1e94] 2024-05-27T07:28:06.504 INFO:journalctl@ceph.mds.g.smithi052.stdout:May 27 07:28:06 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-g[73967]: 10: (EUpdate::replay(MDSRank*)+0x63) [0x55a7629ec969] 2024-05-27T07:28:06.504 INFO:journalctl@ceph.mds.g.smithi052.stdout:May 27 07:28:06 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-g[73967]: 11: (MDLog::_replay_thread()+0x1f38) [0x55a762d8548c] 2024-05-27T07:28:06.504 INFO:journalctl@ceph.mds.g.smithi052.stdout:May 27 07:28:06 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-g[73967]: 12: (MDLog::ReplayThread::entry()+0x11) [0x55a762d8e9cb] 2024-05-27T07:28:06.505 INFO:journalctl@ceph.mds.g.smithi052.stdout:May 27 07:28:06 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-g[73967]: 13: (Thread::entry_wrapper()+0x43) [0x7fd413ead72d] 2024-05-27T07:28:06.505 INFO:journalctl@ceph.mds.g.smithi052.stdout:May 27 07:28:06 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-g[73967]: 14: (Thread::_entry_func(void*)+0xd) [0x7fd413ead749] 2024-05-27T07:28:06.505 INFO:journalctl@ceph.mds.g.smithi052.stdout:May 27 07:28:06 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-g[73967]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7fd412a511ca] 2024-05-27T07:28:06.505 INFO:journalctl@ceph.mds.g.smithi052.stdout:May 27 07:28:06 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-g[73967]: 16: clone() 2024-05-27T07:28:06.505 INFO:journalctl@ceph.mds.g.smithi052.stdout:May 27 07:28:06 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-g[73967]: 2024-05-27T07:28:06.505 INFO:journalctl@ceph.mds.g.smithi052.stdout:May 27 07:28:06 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-g[73967]: 0> 2024-05-27T07:28:06.372+0000 7fd401699700 -1 *** Caught signal (Aborted) ** 2024-05-27T07:28:06.506 INFO:journalctl@ceph.mds.g.smithi052.stdout:May 27 07:28:06 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-g[73967]: in thread 7fd401699700 thread_name:md_log_replay 2024-05-27T07:28:06.506 INFO:journalctl@ceph.mds.g.smithi052.stdout:May 27 07:28:06 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-g[73967]: 2024-05-27T07:28:06.506 INFO:journalctl@ceph.mds.g.smithi052.stdout:May 27 07:28:06 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-g[73967]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:28:06.506 INFO:journalctl@ceph.mds.g.smithi052.stdout:May 27 07:28:06 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-g[73967]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7fd412a5bd20] 2024-05-27T07:28:06.506 INFO:journalctl@ceph.mds.g.smithi052.stdout:May 27 07:28:06 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-g[73967]: 2: gsignal() 2024-05-27T07:28:06.506 INFO:journalctl@ceph.mds.g.smithi052.stdout:May 27 07:28:06 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-g[73967]: 3: abort() 2024-05-27T07:28:06.507 INFO:journalctl@ceph.mds.g.smithi052.stdout:May 27 07:28:06 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-g[73967]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x26e) [0x7fd413edeced] 2024-05-27T07:28:06.507 INFO:journalctl@ceph.mds.g.smithi052.stdout:May 27 07:28:06 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-g[73967]: 5: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7fd413ededd1] 2024-05-27T07:28:06.507 INFO:journalctl@ceph.mds.g.smithi052.stdout:May 27 07:28:06 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-g[73967]: 6: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x55a762d6123d] 2024-05-27T07:28:06.507 INFO:journalctl@ceph.mds.g.smithi052.stdout:May 27 07:28:06 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-g[73967]: 7: (SnapRealm::build_snap_set() const+0x323) [0x55a762d4c971] 2024-05-27T07:28:06.507 INFO:journalctl@ceph.mds.g.smithi052.stdout:May 27 07:28:06 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-g[73967]: 8: (SnapRealm::check_cache() const+0x13e) [0x55a762d4afa0] 2024-05-27T07:28:06.507 INFO:journalctl@ceph.mds.g.smithi052.stdout:May 27 07:28:06 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-g[73967]: 9: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x55a762d4b651] 2024-05-27T07:28:06.507 INFO:journalctl@ceph.mds.g.smithi052.stdout:May 27 07:28:06 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-g[73967]: 10: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x55a762ce0347] 2024-05-27T07:28:06.508 INFO:journalctl@ceph.mds.g.smithi052.stdout:May 27 07:28:06 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-g[73967]: 11: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x55a7629dc87e] 2024-05-27T07:28:06.508 INFO:journalctl@ceph.mds.g.smithi052.stdout:May 27 07:28:06 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-g[73967]: 12: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x2702) [0x55a7629e1e94] 2024-05-27T07:28:06.508 INFO:journalctl@ceph.mds.g.smithi052.stdout:May 27 07:28:06 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-g[73967]: 13: (EUpdate::replay(MDSRank*)+0x63) [0x55a7629ec969] 2024-05-27T07:28:06.508 INFO:journalctl@ceph.mds.g.smithi052.stdout:May 27 07:28:06 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-g[73967]: 14: (MDLog::_replay_thread()+0x1f38) [0x55a762d8548c] 2024-05-27T07:28:06.508 INFO:journalctl@ceph.mds.g.smithi052.stdout:May 27 07:28:06 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-g[73967]: 15: (MDLog::ReplayThread::entry()+0x11) [0x55a762d8e9cb] 2024-05-27T07:28:06.508 INFO:journalctl@ceph.mds.g.smithi052.stdout:May 27 07:28:06 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-g[73967]: 16: (Thread::entry_wrapper()+0x43) [0x7fd413ead72d] 2024-05-27T07:28:06.509 INFO:journalctl@ceph.mds.g.smithi052.stdout:May 27 07:28:06 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-g[73967]: 17: (Thread::_entry_func(void*)+0xd) [0x7fd413ead749] 2024-05-27T07:28:06.509 INFO:journalctl@ceph.mds.g.smithi052.stdout:May 27 07:28:06 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-g[73967]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7fd412a511ca] 2024-05-27T07:28:06.509 INFO:journalctl@ceph.mds.g.smithi052.stdout:May 27 07:28:06 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-g[73967]: 19: clone() 2024-05-27T07:28:06.509 INFO:journalctl@ceph.mds.g.smithi052.stdout:May 27 07:28:06 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-g[73967]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-05-27T07:28:06.509 INFO:journalctl@ceph.mds.g.smithi052.stdout:May 27 07:28:06 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-g[73967]: 2024-05-27T07:28:06.509 INFO:journalctl@ceph.mds.g.smithi052.stdout:May 27 07:28:06 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-g[73967]: -1> 2024-05-27T07:28:06.369+0000 7fd401699700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: In function 'std::set SnapClient::filter(const std::set&) const' thread 7fd401699700 time 2024-05-27T07:28:06.367889+0000 2024-05-27T07:28:06.510 INFO:journalctl@ceph.mds.g.smithi052.stdout:May 27 07:28:06 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-g[73967]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: 197: FAILED ceph_assert(cached_version > 0) 2024-05-27T07:28:06.510 INFO:journalctl@ceph.mds.g.smithi052.stdout:May 27 07:28:06 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-g[73967]: 2024-05-27T07:28:06.510 INFO:journalctl@ceph.mds.g.smithi052.stdout:May 27 07:28:06 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-g[73967]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:28:06.510 INFO:journalctl@ceph.mds.g.smithi052.stdout:May 27 07:28:06 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-g[73967]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x127) [0x7fd413edeba6] 2024-05-27T07:28:06.510 INFO:journalctl@ceph.mds.g.smithi052.stdout:May 27 07:28:06 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-g[73967]: 2: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7fd413ededd1] 2024-05-27T07:28:06.510 INFO:journalctl@ceph.mds.g.smithi052.stdout:May 27 07:28:06 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-g[73967]: 3: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x55a762d6123d] 2024-05-27T07:28:06.511 INFO:journalctl@ceph.mds.g.smithi052.stdout:May 27 07:28:06 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-g[73967]: 4: (SnapRealm::build_snap_set() const+0x323) [0x55a762d4c971] 2024-05-27T07:28:06.511 INFO:journalctl@ceph.mds.g.smithi052.stdout:May 27 07:28:06 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-g[73967]: 5: (SnapRealm::check_cache() const+0x13e) [0x55a762d4afa0] 2024-05-27T07:28:06.511 INFO:journalctl@ceph.mds.g.smithi052.stdout:May 27 07:28:06 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-g[73967]: 6: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x55a762d4b651] 2024-05-27T07:28:06.511 INFO:journalctl@ceph.mds.g.smithi052.stdout:May 27 07:28:06 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-g[73967]: 7: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x55a762ce0347] 2024-05-27T07:28:06.511 INFO:journalctl@ceph.mds.g.smithi052.stdout:May 27 07:28:06 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-g[73967]: 8: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x55a7629dc87e] 2024-05-27T07:28:06.511 INFO:journalctl@ceph.mds.g.smithi052.stdout:May 27 07:28:06 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-g[73967]: 9: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x2702) [0x55a7629e1e94] 2024-05-27T07:28:06.512 INFO:journalctl@ceph.mds.g.smithi052.stdout:May 27 07:28:06 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-g[73967]: 10: (EUpdate::replay(MDSRank*)+0x63) [0x55a7629ec969] 2024-05-27T07:28:06.512 INFO:journalctl@ceph.mds.g.smithi052.stdout:May 27 07:28:06 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-g[73967]: 11: (MDLog::_replay_thread()+0x1f38) [0x55a762d8548c] 2024-05-27T07:28:06.512 INFO:journalctl@ceph.mds.g.smithi052.stdout:May 27 07:28:06 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-g[73967]: 12: (MDLog::ReplayThread::entry()+0x11) [0x55a762d8e9cb] 2024-05-27T07:28:06.512 INFO:journalctl@ceph.mds.g.smithi052.stdout:May 27 07:28:06 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-g[73967]: 13: (Thread::entry_wrapper()+0x43) [0x7fd413ead72d] 2024-05-27T07:28:06.512 INFO:journalctl@ceph.mds.g.smithi052.stdout:May 27 07:28:06 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-g[73967]: 14: (Thread::_entry_func(void*)+0xd) [0x7fd413ead749] 2024-05-27T07:28:06.512 INFO:journalctl@ceph.mds.g.smithi052.stdout:May 27 07:28:06 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-g[73967]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7fd412a511ca] 2024-05-27T07:28:06.513 INFO:journalctl@ceph.mds.g.smithi052.stdout:May 27 07:28:06 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-g[73967]: 16: clone() 2024-05-27T07:28:06.513 INFO:journalctl@ceph.mds.g.smithi052.stdout:May 27 07:28:06 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-g[73967]: 2024-05-27T07:28:06.513 INFO:journalctl@ceph.mds.g.smithi052.stdout:May 27 07:28:06 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-g[73967]: 0> 2024-05-27T07:28:06.372+0000 7fd401699700 -1 *** Caught signal (Aborted) ** 2024-05-27T07:28:06.513 INFO:journalctl@ceph.mds.g.smithi052.stdout:May 27 07:28:06 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-g[73967]: in thread 7fd401699700 thread_name:md_log_replay 2024-05-27T07:28:06.513 INFO:journalctl@ceph.mds.g.smithi052.stdout:May 27 07:28:06 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-g[73967]: 2024-05-27T07:28:06.513 INFO:journalctl@ceph.mds.g.smithi052.stdout:May 27 07:28:06 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-g[73967]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:28:06.514 INFO:journalctl@ceph.mds.g.smithi052.stdout:May 27 07:28:06 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-g[73967]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7fd412a5bd20] 2024-05-27T07:28:06.514 INFO:journalctl@ceph.mds.g.smithi052.stdout:May 27 07:28:06 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-g[73967]: 2: gsignal() 2024-05-27T07:28:06.514 INFO:journalctl@ceph.mds.g.smithi052.stdout:May 27 07:28:06 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-g[73967]: 3: abort() 2024-05-27T07:28:06.514 INFO:journalctl@ceph.mds.g.smithi052.stdout:May 27 07:28:06 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-g[73967]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x26e) [0x7fd413edeced] 2024-05-27T07:28:06.514 INFO:journalctl@ceph.mds.g.smithi052.stdout:May 27 07:28:06 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-g[73967]: 5: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7fd413ededd1] 2024-05-27T07:28:06.514 INFO:journalctl@ceph.mds.g.smithi052.stdout:May 27 07:28:06 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-g[73967]: 6: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x55a762d6123d] 2024-05-27T07:28:06.515 INFO:journalctl@ceph.mds.g.smithi052.stdout:May 27 07:28:06 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-g[73967]: 7: (SnapRealm::build_snap_set() const+0x323) [0x55a762d4c971] 2024-05-27T07:28:06.515 INFO:journalctl@ceph.mds.g.smithi052.stdout:May 27 07:28:06 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-g[73967]: 8: (SnapRealm::check_cache() const+0x13e) [0x55a762d4afa0] 2024-05-27T07:28:06.515 INFO:journalctl@ceph.mds.g.smithi052.stdout:May 27 07:28:06 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-g[73967]: 9: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x55a762d4b651] 2024-05-27T07:28:06.515 INFO:journalctl@ceph.mds.g.smithi052.stdout:May 27 07:28:06 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-g[73967]: 10: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x55a762ce0347] 2024-05-27T07:28:06.516 INFO:journalctl@ceph.mds.g.smithi052.stdout:May 27 07:28:06 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-g[73967]: 11: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x55a7629dc87e] 2024-05-27T07:28:06.516 INFO:journalctl@ceph.mds.g.smithi052.stdout:May 27 07:28:06 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-g[73967]: 12: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x2702) [0x55a7629e1e94] 2024-05-27T07:28:06.516 INFO:journalctl@ceph.mds.g.smithi052.stdout:May 27 07:28:06 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-g[73967]: 13: (EUpdate::replay(MDSRank*)+0x63) [0x55a7629ec969] 2024-05-27T07:28:06.516 INFO:journalctl@ceph.mds.g.smithi052.stdout:May 27 07:28:06 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-g[73967]: 14: (MDLog::_replay_thread()+0x1f38) [0x55a762d8548c] 2024-05-27T07:28:06.516 INFO:journalctl@ceph.mds.g.smithi052.stdout:May 27 07:28:06 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-g[73967]: 15: (MDLog::ReplayThread::entry()+0x11) [0x55a762d8e9cb] 2024-05-27T07:28:06.517 INFO:journalctl@ceph.mds.g.smithi052.stdout:May 27 07:28:06 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-g[73967]: 16: (Thread::entry_wrapper()+0x43) [0x7fd413ead72d] 2024-05-27T07:28:06.517 INFO:journalctl@ceph.mds.g.smithi052.stdout:May 27 07:28:06 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-g[73967]: 17: (Thread::_entry_func(void*)+0xd) [0x7fd413ead749] 2024-05-27T07:28:06.517 INFO:journalctl@ceph.mds.g.smithi052.stdout:May 27 07:28:06 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-g[73967]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7fd412a511ca] 2024-05-27T07:28:06.517 INFO:journalctl@ceph.mds.g.smithi052.stdout:May 27 07:28:06 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-g[73967]: 19: clone() 2024-05-27T07:28:06.518 INFO:journalctl@ceph.mds.g.smithi052.stdout:May 27 07:28:06 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-g[73967]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-05-27T07:28:06.518 INFO:journalctl@ceph.mds.g.smithi052.stdout:May 27 07:28:06 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-g[73967]: 2024-05-27T07:28:06.518 INFO:journalctl@ceph.mds.g.smithi052.stdout:May 27 07:28:06 smithi052 podman[101779]: 2024-05-27 07:28:06.437269209 +0000 UTC m=+0.020780283 container died e7a40afa0af4a8507385ba0e0c7d06747373a30d90fb7a16b4533f563f7fe719 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-g, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, GIT_CLEAN=True, org.label-schema.build-date=20240521, ceph=True, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, org.label-schema.name=CentOS Stream 8 Base Image) 2024-05-27T07:28:06.602 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:28:06 smithi173 ceph-mon[37000]: pgmap v67: 129 pgs: 129 active+clean; 44 MiB data, 435 MiB used, 1.0 TiB / 1.0 TiB avail; 981 KiB/s rd, 3.9 MiB/s wr, 458 op/s 2024-05-27T07:28:07.430 INFO:journalctl@ceph.mds.g.smithi052.stdout:May 27 07:28:07 smithi052 podman[101779]: 2024-05-27 07:28:07.184125212 +0000 UTC m=+0.767636278 container remove e7a40afa0af4a8507385ba0e0c7d06747373a30d90fb7a16b4533f563f7fe719 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-g, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240521, ceph=True, CEPH_POINT_RELEASE=, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, io.buildah.version=1.33.5, GIT_CLEAN=True, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS) 2024-05-27T07:28:07.431 INFO:journalctl@ceph.mds.g.smithi052.stdout:May 27 07:28:07 smithi052 systemd[1]: ceph-73df2636-1bf8-11ef-bc9b-c7b262605968@mds.g.service: Main process exited, code=exited, status=134/n/a 2024-05-27T07:28:07.729 INFO:journalctl@ceph.mds.g.smithi052.stdout:May 27 07:28:07 smithi052 systemd[1]: ceph-73df2636-1bf8-11ef-bc9b-c7b262605968@mds.g.service: Failed with result 'exit-code'. 2024-05-27T07:28:07.729 INFO:journalctl@ceph.mds.g.smithi052.stdout:May 27 07:28:07 smithi052 systemd[1]: ceph-73df2636-1bf8-11ef-bc9b-c7b262605968@mds.g.service: Consumed 1.179s CPU time. 2024-05-27T07:28:08.439 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:28:08 smithi105 ceph-mon[35977]: pgmap v68: 129 pgs: 129 active+clean; 44 MiB data, 435 MiB used, 1.0 TiB / 1.0 TiB avail; 981 KiB/s rd, 3.9 MiB/s wr, 458 op/s 2024-05-27T07:28:08.439 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:28:08 smithi105 ceph-mon[35977]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-05-27T07:28:08.479 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:28:08 smithi052 ceph-mon[31955]: pgmap v68: 129 pgs: 129 active+clean; 44 MiB data, 435 MiB used, 1.0 TiB / 1.0 TiB avail; 981 KiB/s rd, 3.9 MiB/s wr, 458 op/s 2024-05-27T07:28:08.479 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:28:08 smithi052 ceph-mon[31955]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-05-27T07:28:08.601 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:28:08 smithi173 ceph-mon[37000]: pgmap v68: 129 pgs: 129 active+clean; 44 MiB data, 435 MiB used, 1.0 TiB / 1.0 TiB avail; 981 KiB/s rd, 3.9 MiB/s wr, 458 op/s 2024-05-27T07:28:08.602 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:28:08 smithi173 ceph-mon[37000]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-05-27T07:28:10.439 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:28:10 smithi105 ceph-mon[35977]: pgmap v69: 129 pgs: 129 active+clean; 53 MiB data, 455 MiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 4.7 MiB/s wr, 515 op/s 2024-05-27T07:28:10.479 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:28:10 smithi052 ceph-mon[31955]: pgmap v69: 129 pgs: 129 active+clean; 53 MiB data, 455 MiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 4.7 MiB/s wr, 515 op/s 2024-05-27T07:28:10.601 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:28:10 smithi173 ceph-mon[37000]: pgmap v69: 129 pgs: 129 active+clean; 53 MiB data, 455 MiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 4.7 MiB/s wr, 515 op/s 2024-05-27T07:28:12.314 DEBUG:teuthology.orchestra.run.smithi052:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:38f9d900c83c63799fbdbe61acc9a11b0d3554a6 shell --fsid 73df2636-1bf8-11ef-bc9b-c7b262605968 -- ceph tell mds.1:0 scrub status 2024-05-27T07:28:12.479 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:28:12 smithi052 ceph-mon[31955]: pgmap v70: 129 pgs: 129 active+clean; 53 MiB data, 455 MiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 4.3 MiB/s wr, 499 op/s 2024-05-27T07:28:12.585 INFO:teuthology.orchestra.run.smithi052.stderr:Inferring config /var/lib/ceph/73df2636-1bf8-11ef-bc9b-c7b262605968/mon.a/config 2024-05-27T07:28:12.601 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:28:12 smithi173 ceph-mon[37000]: pgmap v70: 129 pgs: 129 active+clean; 53 MiB data, 455 MiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 4.3 MiB/s wr, 499 op/s 2024-05-27T07:28:12.689 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:28:12 smithi105 ceph-mon[35977]: pgmap v70: 129 pgs: 129 active+clean; 53 MiB data, 455 MiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 4.3 MiB/s wr, 499 op/s 2024-05-27T07:28:13.479 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:28:13 smithi052 ceph-mon[31955]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-05-27T07:28:13.479 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:28:13 smithi052 ceph-mon[31955]: from='mgr.15333 172.21.15.52:0/416389082' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-27T07:28:13.601 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:28:13 smithi173 ceph-mon[37000]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-05-27T07:28:13.601 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:28:13 smithi173 ceph-mon[37000]: from='mgr.15333 172.21.15.52:0/416389082' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-27T07:28:13.689 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:28:13 smithi105 ceph-mon[35977]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-05-27T07:28:13.689 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:28:13 smithi105 ceph-mon[35977]: from='mgr.15333 172.21.15.52:0/416389082' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-27T07:28:14.479 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:28:14 smithi052 ceph-mon[31955]: pgmap v71: 129 pgs: 129 active+clean; 62 MiB data, 461 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 5.1 MiB/s wr, 588 op/s 2024-05-27T07:28:14.601 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:28:14 smithi173 ceph-mon[37000]: pgmap v71: 129 pgs: 129 active+clean; 62 MiB data, 461 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 5.1 MiB/s wr, 588 op/s 2024-05-27T07:28:14.689 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:28:14 smithi105 ceph-mon[35977]: pgmap v71: 129 pgs: 129 active+clean; 62 MiB data, 461 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 5.1 MiB/s wr, 588 op/s 2024-05-27T07:28:14.863 INFO:journalctl@ceph.mds.f.smithi173.stdout:May 27 07:28:14 smithi173 systemd[1]: ceph-73df2636-1bf8-11ef-bc9b-c7b262605968@mds.f.service: Scheduled restart job, restart counter is at 1. 2024-05-27T07:28:14.863 INFO:journalctl@ceph.mds.f.smithi173.stdout:May 27 07:28:14 smithi173 systemd[1]: Stopped Ceph mds.f for 73df2636-1bf8-11ef-bc9b-c7b262605968. 2024-05-27T07:28:14.863 INFO:journalctl@ceph.mds.f.smithi173.stdout:May 27 07:28:14 smithi173 systemd[1]: ceph-73df2636-1bf8-11ef-bc9b-c7b262605968@mds.f.service: Consumed 2.476s CPU time. 2024-05-27T07:28:14.863 INFO:journalctl@ceph.mds.f.smithi173.stdout:May 27 07:28:14 smithi173 systemd[1]: Starting Ceph mds.f for 73df2636-1bf8-11ef-bc9b-c7b262605968... 2024-05-27T07:28:14.863 INFO:journalctl@ceph.mds.f.smithi173.stdout:May 27 07:28:14 smithi173 podman[72395]: 2024-05-27 07:28:14.748925641 +0000 UTC m=+0.017351177 image pull e36ea4fc8a5807e8d2701370e350a0de3856538797f7fba64cfce7e344bd6799 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6 2024-05-27T07:28:14.863 INFO:journalctl@ceph.mds.f.smithi173.stdout:May 27 07:28:14 smithi173 podman[72395]: 2024-05-27 07:28:14.874207175 +0000 UTC m=+0.142632728 container create 11d3b53c90f5ccc17a6b74d217f5c944d8341059a5ed3156713b380c4df8a0df (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-f, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, ceph=True, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.build-date=20240521) 2024-05-27T07:28:14.924 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:28:14.934+0000 7f8b72b86640 1 -- 172.21.15.52:0/755868236 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f8b6c109cd0 msgr2=0x7f8b6c103870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:28:14.924 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:28:14.934+0000 7f8b72b86640 1 --2- 172.21.15.52:0/755868236 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f8b6c109cd0 0x7f8b6c103870 secure :-1 s=READY pgs=563 cs=0 l=1 rev1=1 crypto rx=0x7f8b60009900 tx=0x7f8b6002a3f0 comp rx=0 tx=0).stop 2024-05-27T07:28:14.924 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:28:14.934+0000 7f8b72b86640 1 -- 172.21.15.52:0/755868236 shutdown_connections 2024-05-27T07:28:14.924 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:28:14.934+0000 7f8b72b86640 1 --2- 172.21.15.52:0/755868236 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f8b6c104420 0x7f8b6c1107f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:28:14.925 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:28:14.934+0000 7f8b72b86640 1 --2- 172.21.15.52:0/755868236 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f8b6c10a170 0x7f8b6c103db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:28:14.925 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:28:14.934+0000 7f8b72b86640 1 --2- 172.21.15.52:0/755868236 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f8b6c109cd0 0x7f8b6c103870 unknown :-1 s=CLOSED pgs=563 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:28:14.925 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:28:14.934+0000 7f8b72b86640 1 -- 172.21.15.52:0/755868236 >> 172.21.15.52:0/755868236 conn(0x7f8b6c079440 msgr2=0x7f8b6c079880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:28:14.925 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:28:14.934+0000 7f8b72b86640 1 -- 172.21.15.52:0/755868236 shutdown_connections 2024-05-27T07:28:14.925 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:28:14.934+0000 7f8b72b86640 1 -- 172.21.15.52:0/755868236 wait complete. 2024-05-27T07:28:14.925 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:28:14.935+0000 7f8b72b86640 1 Processor -- start 2024-05-27T07:28:14.925 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:28:14.935+0000 7f8b72b86640 1 -- start start 2024-05-27T07:28:14.927 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:28:14.935+0000 7f8b72b86640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f8b6c104420 0x7f8b6c074fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:28:14.927 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:28:14.935+0000 7f8b72b86640 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f8b6c109cd0 0x7f8b6c075500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:28:14.927 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:28:14.935+0000 7f8b72b86640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f8b6c10a170 0x7f8b6c075a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:28:14.927 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:28:14.935+0000 7f8b72b86640 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x7f8b6c113ea0 con 0x7f8b6c109cd0 2024-05-27T07:28:14.927 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:28:14.935+0000 7f8b72b86640 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f8b6c113d20 con 0x7f8b6c104420 2024-05-27T07:28:14.927 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:28:14.935+0000 7f8b72b86640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f8b6c114020 con 0x7f8b6c10a170 2024-05-27T07:28:14.927 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:28:14.935+0000 7f8b6bfff640 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f8b6c109cd0 0x7f8b6c075500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:28:14.928 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:28:14.935+0000 7f8b6bfff640 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f8b6c109cd0 0x7f8b6c075500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.52:36838/0 (socket says 172.21.15.52:36838) 2024-05-27T07:28:14.928 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:28:14.935+0000 7f8b6bfff640 1 -- 172.21.15.52:0/2575527635 learned_addr learned my addr 172.21.15.52:0/2575527635 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-05-27T07:28:14.928 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:28:14.935+0000 7f8b70922640 1 --2- 172.21.15.52:0/2575527635 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f8b6c104420 0x7f8b6c074fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:28:14.928 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:28:14.935+0000 7f8b71123640 1 --2- 172.21.15.52:0/2575527635 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f8b6c10a170 0x7f8b6c075a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:28:14.928 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:28:14.935+0000 7f8b6bfff640 1 -- 172.21.15.52:0/2575527635 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f8b6c10a170 msgr2=0x7f8b6c075a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:28:14.928 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:28:14.935+0000 7f8b6bfff640 1 --2- 172.21.15.52:0/2575527635 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f8b6c10a170 0x7f8b6c075a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:28:14.928 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:28:14.935+0000 7f8b6bfff640 1 -- 172.21.15.52:0/2575527635 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f8b6c104420 msgr2=0x7f8b6c074fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:28:14.928 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:28:14.935+0000 7f8b6bfff640 1 --2- 172.21.15.52:0/2575527635 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f8b6c104420 0x7f8b6c074fc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:28:14.928 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:28:14.935+0000 7f8b6bfff640 1 -- 172.21.15.52:0/2575527635 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8b6c1a8c30 con 0x7f8b6c109cd0 2024-05-27T07:28:14.928 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:28:14.935+0000 7f8b71123640 1 --2- 172.21.15.52:0/2575527635 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f8b6c10a170 0x7f8b6c075a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-27T07:28:14.928 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:28:14.936+0000 7f8b6bfff640 1 --2- 172.21.15.52:0/2575527635 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f8b6c109cd0 0x7f8b6c075500 secure :-1 s=READY pgs=564 cs=0 l=1 rev1=1 crypto rx=0x7f8b58005fd0 tx=0x7f8b5800dde0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:28:14.928 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:28:14.936+0000 7f8b69ffb640 1 -- 172.21.15.52:0/2575527635 <== mon.0 v2:172.21.15.52:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8b58017070 con 0x7f8b6c109cd0 2024-05-27T07:28:14.929 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:28:14.936+0000 7f8b69ffb640 1 -- 172.21.15.52:0/2575527635 <== mon.0 v2:172.21.15.52:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f8b58009c30 con 0x7f8b6c109cd0 2024-05-27T07:28:14.929 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:28:14.936+0000 7f8b72b86640 1 -- 172.21.15.52:0/2575527635 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8b6c1a8f20 con 0x7f8b6c109cd0 2024-05-27T07:28:14.929 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:28:14.936+0000 7f8b69ffb640 1 -- 172.21.15.52:0/2575527635 <== mon.0 v2:172.21.15.52:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8b58003d10 con 0x7f8b6c109cd0 2024-05-27T07:28:14.929 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:28:14.936+0000 7f8b72b86640 1 -- 172.21.15.52:0/2575527635 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8b6c1a93e0 con 0x7f8b6c109cd0 2024-05-27T07:28:14.934 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:28:14.939+0000 7f8b79a41740 1 Processor -- start 2024-05-27T07:28:14.934 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:28:14.939+0000 7f8b79a41740 1 -- start start 2024-05-27T07:28:14.934 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:28:14.939+0000 7f8b69ffb640 1 -- 172.21.15.52:0/2575527635 <== mon.0 v2:172.21.15.52:3300/0 4 ==== mgrmap(e 26) ==== 149783+0+0 (secure 0 0 0) 0x7f8b58003eb0 con 0x7f8b6c109cd0 2024-05-27T07:28:14.935 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:28:14.939+0000 7f8b79a41740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55c5653c21b0 0x55c5657633d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:28:14.935 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:28:14.939+0000 7f8b79a41740 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55c5653992f0 0x55c565763910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:28:14.935 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:28:14.939+0000 7f8b79a41740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55c5653a14d0 0x55c565763e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:28:14.935 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:28:14.939+0000 7f8b79a41740 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x55c5656d5c30 con 0x55c5653992f0 2024-05-27T07:28:14.935 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:28:14.939+0000 7f8b79a41740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x55c5656c57d0 con 0x55c5653c21b0 2024-05-27T07:28:14.935 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:28:14.939+0000 7f8b79a41740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x55c5656c9810 con 0x55c5653a14d0 2024-05-27T07:28:14.935 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:28:14.939+0000 7f8b71123640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55c5653c21b0 0x55c5657633d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:28:14.935 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:28:14.939+0000 7f8b70922640 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55c5653992f0 0x55c565763910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:28:14.935 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:28:14.939+0000 7f8b71123640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55c5653c21b0 0x55c5657633d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.52:46782/0 (socket says 172.21.15.52:46782) 2024-05-27T07:28:14.935 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:28:14.939+0000 7f8b71123640 1 -- 172.21.15.52:0/3073060905 learned_addr learned my addr 172.21.15.52:0/3073060905 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-05-27T07:28:14.935 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:28:14.939+0000 7f8b6bfff640 1 --2- 172.21.15.52:0/3073060905 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55c5653a14d0 0x55c565763e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:28:14.936 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:28:14.940+0000 7f8b70922640 1 -- 172.21.15.52:0/3073060905 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55c5653a14d0 msgr2=0x55c565763e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:28:14.936 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:28:14.940+0000 7f8b70922640 1 --2- 172.21.15.52:0/3073060905 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55c5653a14d0 0x55c565763e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:28:14.936 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:28:14.940+0000 7f8b70922640 1 -- 172.21.15.52:0/3073060905 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55c5653c21b0 msgr2=0x55c5657633d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:28:14.936 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:28:14.940+0000 7f8b70922640 1 --2- 172.21.15.52:0/3073060905 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55c5653c21b0 0x55c5657633d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:28:14.936 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:28:14.940+0000 7f8b70922640 1 -- 172.21.15.52:0/3073060905 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55c5653da210 con 0x55c5653992f0 2024-05-27T07:28:14.936 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:28:14.940+0000 7f8b71123640 1 --2- 172.21.15.52:0/3073060905 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55c5653c21b0 0x55c5657633d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-27T07:28:14.936 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:28:14.940+0000 7f8b70922640 1 --2- 172.21.15.52:0/3073060905 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55c5653992f0 0x55c565763910 secure :-1 s=READY pgs=565 cs=0 l=1 rev1=1 crypto rx=0x7f8b60006010 tx=0x7f8b60032bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d9b26572b77e9c09 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:28:14.936 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:28:14.940+0000 7f8b56ffd640 1 -- 172.21.15.52:0/3073060905 <== mon.0 v2:172.21.15.52:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8b60040070 con 0x55c5653992f0 2024-05-27T07:28:14.936 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:28:14.940+0000 7f8b56ffd640 1 -- 172.21.15.52:0/3073060905 <== mon.0 v2:172.21.15.52:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f8b60004030 con 0x55c5653992f0 2024-05-27T07:28:14.936 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:28:14.940+0000 7f8b56ffd640 1 -- 172.21.15.52:0/3073060905 <== mon.0 v2:172.21.15.52:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8b60035040 con 0x55c5653992f0 2024-05-27T07:28:14.937 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:28:14.940+0000 7f8b69ffb640 1 --2- 172.21.15.52:0/2575527635 >> [v2:172.21.15.52:6800/1822318384,v1:172.21.15.52:6801/1822318384] conn(0x7f8b5c0b29d0 0x7f8b5c0b4ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:28:14.937 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:28:14.940+0000 7f8b71123640 1 --2- 172.21.15.52:0/2575527635 >> [v2:172.21.15.52:6800/1822318384,v1:172.21.15.52:6801/1822318384] conn(0x7f8b5c0b29d0 0x7f8b5c0b4ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:28:14.937 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:28:14.941+0000 7f8b69ffb640 1 -- 172.21.15.52:0/2575527635 <== mon.0 v2:172.21.15.52:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9379+0+0 (secure 0 0 0) 0x7f8b580e2520 con 0x7f8b6c109cd0 2024-05-27T07:28:14.937 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:28:14.941+0000 7f8b79a41740 1 -- 172.21.15.52:0/3073060905 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55c5653992f0 msgr2=0x55c565763910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:28:14.937 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:28:14.941+0000 7f8b71123640 1 --2- 172.21.15.52:0/2575527635 >> [v2:172.21.15.52:6800/1822318384,v1:172.21.15.52:6801/1822318384] conn(0x7f8b5c0b29d0 0x7f8b5c0b4ee0 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x55c565760540 tx=0x7f8b64009500 comp rx=0 tx=0).ready entity=mgr.15333 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:28:14.937 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:28:14.941+0000 7f8b79a41740 1 --2- 172.21.15.52:0/3073060905 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55c5653992f0 0x55c565763910 secure :-1 s=READY pgs=565 cs=0 l=1 rev1=1 crypto rx=0x7f8b60006010 tx=0x7f8b60032bd0 comp rx=0 tx=0).stop 2024-05-27T07:28:14.937 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:28:14.941+0000 7f8b79a41740 1 -- 172.21.15.52:0/3073060905 shutdown_connections 2024-05-27T07:28:14.937 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:28:14.941+0000 7f8b79a41740 1 --2- 172.21.15.52:0/3073060905 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55c5653a14d0 0x55c565763e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:28:14.937 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:28:14.941+0000 7f8b79a41740 1 --2- 172.21.15.52:0/3073060905 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55c5653992f0 0x55c565763910 unknown :-1 s=CLOSED pgs=565 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:28:14.937 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:28:14.941+0000 7f8b79a41740 1 --2- 172.21.15.52:0/3073060905 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55c5653c21b0 0x55c5657633d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:28:14.937 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:28:14.941+0000 7f8b79a41740 1 -- 172.21.15.52:0/3073060905 >> 172.21.15.52:0/3073060905 conn(0x55c5653b6300 msgr2=0x55c56575fb80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:28:14.937 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:28:14.941+0000 7f8b79a41740 1 -- 172.21.15.52:0/3073060905 shutdown_connections 2024-05-27T07:28:14.938 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:28:14.941+0000 7f8b79a41740 1 -- 172.21.15.52:0/3073060905 wait complete. 2024-05-27T07:28:14.938 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:28:14.942+0000 7f8b79a41740 1 -- start start 2024-05-27T07:28:14.938 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:28:14.942+0000 7f8b79a41740 1 Processor -- start 2024-05-27T07:28:14.938 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:28:14.942+0000 7f8b79a41740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55c5653992f0 0x55c56576f410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:28:14.938 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:28:14.942+0000 7f8b79a41740 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55c5653a14d0 0x55c56576f950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:28:14.938 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:28:14.942+0000 7f8b79a41740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55c5653c21b0 0x55c56576d8e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:28:14.938 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:28:14.942+0000 7f8b79a41740 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x55c5653ab790 con 0x55c5653a14d0 2024-05-27T07:28:14.938 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:28:14.942+0000 7f8b79a41740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x55c565681890 con 0x55c5653992f0 2024-05-27T07:28:14.938 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:28:14.942+0000 7f8b79a41740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x55c5657704a0 con 0x55c5653c21b0 2024-05-27T07:28:14.938 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:28:14.942+0000 7f8b79a41740 10 client.0 _ll_register_callbacks cb 0x55c5653d9a00 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T07:28:14.938 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:28:14.942+0000 7f8b6bfff640 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55c5653a14d0 0x55c56576f950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:28:14.938 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:28:14.942+0000 7f8b70922640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55c5653992f0 0x55c56576f410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:28:14.939 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:28:14.942+0000 7f8b6bfff640 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55c5653a14d0 0x55c56576f950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.52:36850/0 (socket says 172.21.15.52:36850) 2024-05-27T07:28:14.939 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:28:14.942+0000 7f8b70922640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55c5653992f0 0x55c56576f410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.52:46794/0 (socket says 172.21.15.52:46794) 2024-05-27T07:28:14.939 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:28:14.942+0000 7f8b6bfff640 1 -- 172.21.15.52:0/4109007167 learned_addr learned my addr 172.21.15.52:0/4109007167 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-05-27T07:28:14.939 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:28:14.943+0000 7f8b71123640 1 --2- 172.21.15.52:0/4109007167 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55c5653c21b0 0x55c56576d8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:28:14.939 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:28:14.943+0000 7f8b70922640 1 -- 172.21.15.52:0/4109007167 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55c5653c21b0 msgr2=0x55c56576d8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:28:14.939 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:28:14.943+0000 7f8b70922640 1 --2- 172.21.15.52:0/4109007167 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55c5653c21b0 0x55c56576d8e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:28:14.939 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:28:14.943+0000 7f8b70922640 1 -- 172.21.15.52:0/4109007167 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55c5653a14d0 msgr2=0x55c56576f950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:28:14.939 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:28:14.943+0000 7f8b70922640 1 --2- 172.21.15.52:0/4109007167 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55c5653a14d0 0x55c56576f950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:28:14.939 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:28:14.943+0000 7f8b70922640 1 -- 172.21.15.52:0/4109007167 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55c5654a6620 con 0x55c5653992f0 2024-05-27T07:28:14.939 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:28:14.943+0000 7f8b70922640 1 --2- 172.21.15.52:0/4109007167 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55c5653992f0 0x55c56576f410 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x55c565764780 tx=0x7f8b6002ab70 comp rx=0 tx=0).ready entity=mon.2 client_cookie=6e42246437bf6a3f server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:28:14.939 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:28:14.943+0000 7f8b4bfff640 10 client.0 ms_handle_connect on v2:172.21.15.105:3300/0 2024-05-27T07:28:14.939 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:28:14.943+0000 7f8b4bfff640 1 -- 172.21.15.52:0/4109007167 <== mon.2 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8b60052060 con 0x55c5653992f0 2024-05-27T07:28:14.939 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:28:14.943+0000 7f8b79a41740 1 -- 172.21.15.52:0/4109007167 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f8b60004030 con 0x55c5653992f0 2024-05-27T07:28:14.940 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:28:14.943+0000 7f8b4bfff640 1 -- 172.21.15.52:0/4109007167 <== mon.2 v2:172.21.15.105:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 9379+0+0 (secure 0 0 0) 0x7f8b600054c0 con 0x55c5653992f0 2024-05-27T07:28:14.940 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:28:14.944+0000 7f8b4bfff640 1 -- 172.21.15.52:0/4109007167 <== mon.2 v2:172.21.15.105:3300/0 3 ==== mon_get_version_reply(handle=1 version=33) ==== 24+0+0 (secure 0 0 0) 0x7f8b60059050 con 0x55c5653992f0 2024-05-27T07:28:14.940 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:28:14.944+0000 7f8b79a41740 10 client.25661 fetch_fsmap learned FSMap version 33 2024-05-27T07:28:14.940 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:28:14.944+0000 7f8b79a41740 1 -- 172.21.15.52:0/4109007167 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,fsmap=33,monmap=4+}) -- 0x55c56577c7b0 con 0x55c5653992f0 2024-05-27T07:28:14.940 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:28:14.944+0000 7f8b4bfff640 1 -- 172.21.15.52:0/4109007167 <== mon.2 v2:172.21.15.105:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f8b60004030 con 0x55c5653992f0 2024-05-27T07:28:14.940 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:28:14.944+0000 7f8b4bfff640 1 -- 172.21.15.52:0/4109007167 <== mon.2 v2:172.21.15.105:3300/0 5 ==== fsmap(e 33) ==== 8374+0+0 (secure 0 0 0) 0x7f8b60004d70 con 0x55c5653992f0 2024-05-27T07:28:14.940 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:28:14.944+0000 7f8b79a41740 10 client.25661 fetch_fsmap finished waiting for FSMap version 33 2024-05-27T07:28:14.940 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:28:14.944+0000 7f8b79a41740 10 client.25661 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-05-27T07:28:14.940 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:28:14.944+0000 7f8b79a41740 20 client.25661 populate_metadata read hostname 'smithi052' 2024-05-27T07:28:14.940 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:28:14.944+0000 7f8b79a41740 1 --2- 172.21.15.52:0/4109007167 >> [v2:172.21.15.105:6838/500348151,v1:172.21.15.105:6839/500348151] conn(0x55c565380d80 0x55c56577fb10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:28:14.940 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:28:14.944+0000 7f8b79a41740 4 client.25661 mds_command: new command op to 24527 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-27T07:28:14.940 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:28:14.944+0000 7f8b79a41740 1 -- 172.21.15.52:0/4109007167 --> [v2:172.21.15.105:6838/500348151,v1:172.21.15.105:6839/500348151] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55c5656d7b80 con 0x55c565380d80 2024-05-27T07:28:14.940 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:28:14.944+0000 7f8b6bfff640 1 --2- 172.21.15.52:0/4109007167 >> [v2:172.21.15.105:6838/500348151,v1:172.21.15.105:6839/500348151] conn(0x55c565380d80 0x55c56577fb10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:28:14.941 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:28:14.945+0000 7f8b4bfff640 10 client.25661 ms_handle_connect on v2:172.21.15.105:6838/500348151 2024-05-27T07:28:14.941 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:28:14.945+0000 7f8b6bfff640 1 --2- 172.21.15.52:0/4109007167 >> [v2:172.21.15.105:6838/500348151,v1:172.21.15.105:6839/500348151] conn(0x55c565380d80 0x55c56577fb10 crc :-1 s=READY pgs=77 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=85ba7ad794c61d82 server_cookie=9a965d682145e3d2 in_seq=0 out_seq=0 2024-05-27T07:28:14.942 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:28:14.946+0000 7f8b4bfff640 1 -- 172.21.15.52:0/4109007167 <== mds.0 v2:172.21.15.105:6838/500348151 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24041 (crc 0 0 0) 0x55c5656d7b80 con 0x55c565380d80 2024-05-27T07:28:14.942 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:28:14.946+0000 7f8b4bfff640 10 client.25661 handle_command_reply: tid=0 2024-05-27T07:28:14.942 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:28:14.946+0000 7f8b79a41740 1 client.25661 shutdown 2024-05-27T07:28:14.942 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:28:14.947+0000 7f8b79a41740 1 -- 172.21.15.52:0/4109007167 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55c5653992f0 msgr2=0x55c56576f410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:28:14.942 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:28:14.947+0000 7f8b79a41740 1 --2- 172.21.15.52:0/4109007167 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55c5653992f0 0x55c56576f410 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x55c565764780 tx=0x7f8b6002ab70 comp rx=0 tx=0).stop 2024-05-27T07:28:14.942 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:28:14.947+0000 7f8b79a41740 1 -- 172.21.15.52:0/4109007167 shutdown_connections 2024-05-27T07:28:14.942 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:28:14.947+0000 7f8b79a41740 1 --2- 172.21.15.52:0/4109007167 >> [v2:172.21.15.105:6838/500348151,v1:172.21.15.105:6839/500348151] conn(0x55c565380d80 0x55c56577fb10 crc :-1 s=READY pgs=77 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:28:14.943 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:28:14.947+0000 7f8b79a41740 1 --2- 172.21.15.52:0/4109007167 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55c5653c21b0 0x55c56576d8e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:28:14.943 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:28:14.947+0000 7f8b79a41740 1 --2- 172.21.15.52:0/4109007167 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55c5653a14d0 0x55c56576f950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:28:14.943 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:28:14.947+0000 7f8b4bfff640 0 client.25661 ms_handle_reset on v2:172.21.15.105:6838/500348151 2024-05-27T07:28:14.943 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:28:14.947+0000 7f8b79a41740 1 --2- 172.21.15.52:0/4109007167 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55c5653992f0 0x55c56576f410 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:28:14.943 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:28:14.947+0000 7f8b79a41740 1 -- 172.21.15.52:0/4109007167 >> 172.21.15.52:0/4109007167 conn(0x55c5653b6300 msgr2=0x55c5657650f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:28:14.943 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:28:14.947+0000 7f8b79a41740 1 -- 172.21.15.52:0/4109007167 shutdown_connections 2024-05-27T07:28:14.943 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:28:14.947+0000 7f8b79a41740 1 -- 172.21.15.52:0/4109007167 wait complete. 2024-05-27T07:28:14.943 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:28:14.947+0000 7f8b79a41740 20 client.25661 trim_cache size 0 max 16384 2024-05-27T07:28:14.953 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:28:14.962+0000 7f8b79a41740 1 Processor -- start 2024-05-27T07:28:14.953 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:28:14.962+0000 7f8b79a41740 1 -- start start 2024-05-27T07:28:14.954 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:28:14.963+0000 7f8b79a41740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55c5653992f0 0x55c56576f710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:28:14.954 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:28:14.963+0000 7f8b79a41740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55c5653a14d0 0x55c565782b40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:28:14.954 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:28:14.963+0000 7f8b79a41740 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55c5653c21b0 0x55c565783080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:28:14.954 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:28:14.963+0000 7f8b79a41740 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x7f8b6c1b3420 con 0x55c5653c21b0 2024-05-27T07:28:14.954 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:28:14.963+0000 7f8b79a41740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f8b6c1b4ff0 con 0x55c5653992f0 2024-05-27T07:28:14.954 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:28:14.963+0000 7f8b79a41740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f8b6c1b3af0 con 0x55c5653a14d0 2024-05-27T07:28:14.954 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:28:14.963+0000 7f8b71123640 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55c5653c21b0 0x55c565783080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:28:14.954 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:28:14.963+0000 7f8b70922640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55c5653992f0 0x55c56576f710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:28:14.954 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:28:14.963+0000 7f8b71123640 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55c5653c21b0 0x55c565783080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.52:36866/0 (socket says 172.21.15.52:36866) 2024-05-27T07:28:14.954 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:28:14.963+0000 7f8b71123640 1 -- 172.21.15.52:0/19229105 learned_addr learned my addr 172.21.15.52:0/19229105 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-05-27T07:28:14.955 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:28:14.963+0000 7f8b6bfff640 1 --2- 172.21.15.52:0/19229105 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55c5653a14d0 0x55c565782b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:28:14.955 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:28:14.963+0000 7f8b71123640 1 -- 172.21.15.52:0/19229105 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55c5653a14d0 msgr2=0x55c565782b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:28:14.955 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:28:14.963+0000 7f8b71123640 1 --2- 172.21.15.52:0/19229105 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55c5653a14d0 0x55c565782b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:28:14.955 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:28:14.963+0000 7f8b71123640 1 -- 172.21.15.52:0/19229105 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55c5653992f0 msgr2=0x55c56576f710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:28:14.955 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:28:14.963+0000 7f8b71123640 1 --2- 172.21.15.52:0/19229105 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55c5653992f0 0x55c56576f710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:28:14.955 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:28:14.963+0000 7f8b71123640 1 -- 172.21.15.52:0/19229105 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55c56576fc50 con 0x55c5653c21b0 2024-05-27T07:28:14.955 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:28:14.963+0000 7f8b70922640 1 --2- 172.21.15.52:0/19229105 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55c5653992f0 0x55c56576f710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-27T07:28:14.955 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:28:14.963+0000 7f8b71123640 1 --2- 172.21.15.52:0/19229105 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55c5653c21b0 0x55c565783080 secure :-1 s=READY pgs=566 cs=0 l=1 rev1=1 crypto rx=0x7f8b64010ae0 tx=0x7f8b64019000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ff6164e7a8cdbb28 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:28:14.955 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:28:14.964+0000 7f8b567fc640 1 -- 172.21.15.52:0/19229105 <== mon.0 v2:172.21.15.52:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8b64004b20 con 0x55c5653c21b0 2024-05-27T07:28:14.955 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:28:14.964+0000 7f8b567fc640 1 -- 172.21.15.52:0/19229105 <== mon.0 v2:172.21.15.52:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f8b64004cc0 con 0x55c5653c21b0 2024-05-27T07:28:14.956 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:28:14.964+0000 7f8b567fc640 1 -- 172.21.15.52:0/19229105 <== mon.0 v2:172.21.15.52:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8b640059b0 con 0x55c5653c21b0 2024-05-27T07:28:14.959 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:28:14.964+0000 7f8b79a41740 1 -- 172.21.15.52:0/19229105 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55c5653c21b0 msgr2=0x55c565783080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:28:14.959 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:28:14.964+0000 7f8b79a41740 1 --2- 172.21.15.52:0/19229105 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55c5653c21b0 0x55c565783080 secure :-1 s=READY pgs=566 cs=0 l=1 rev1=1 crypto rx=0x7f8b64010ae0 tx=0x7f8b64019000 comp rx=0 tx=0).stop 2024-05-27T07:28:14.959 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:28:14.964+0000 7f8b79a41740 1 -- 172.21.15.52:0/19229105 shutdown_connections 2024-05-27T07:28:14.959 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:28:14.964+0000 7f8b79a41740 1 --2- 172.21.15.52:0/19229105 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55c5653c21b0 0x55c565783080 unknown :-1 s=CLOSED pgs=566 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:28:14.959 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:28:14.964+0000 7f8b79a41740 1 --2- 172.21.15.52:0/19229105 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55c5653a14d0 0x55c565782b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:28:14.959 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:28:14.964+0000 7f8b79a41740 1 --2- 172.21.15.52:0/19229105 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55c5653992f0 0x55c56576f710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:28:14.959 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:28:14.964+0000 7f8b79a41740 1 -- 172.21.15.52:0/19229105 >> 172.21.15.52:0/19229105 conn(0x55c5653b6300 msgr2=0x55c56576f000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:28:14.960 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:28:14.965+0000 7f8b79a41740 1 -- 172.21.15.52:0/19229105 shutdown_connections 2024-05-27T07:28:14.960 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:28:14.965+0000 7f8b79a41740 1 -- 172.21.15.52:0/19229105 wait complete. 2024-05-27T07:28:14.960 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:28:14.965+0000 7f8b79a41740 1 -- start start 2024-05-27T07:28:14.960 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:28:14.965+0000 7f8b79a41740 1 Processor -- start 2024-05-27T07:28:14.960 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:28:14.965+0000 7f8b79a41740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55c5653992f0 0x55c5657dbef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:28:14.960 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:28:14.965+0000 7f8b79a41740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55c5653a14d0 0x55c5657dc430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:28:14.960 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:28:14.965+0000 7f8b79a41740 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55c5653c21b0 0x55c5657dc970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:28:14.960 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:28:14.965+0000 7f8b79a41740 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x7f8b6c1b4890 con 0x55c5653c21b0 2024-05-27T07:28:14.960 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:28:14.965+0000 7f8b79a41740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f8b6c1b41c0 con 0x55c5653a14d0 2024-05-27T07:28:14.960 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:28:14.965+0000 7f8b79a41740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f8b6c1b1400 con 0x55c5653992f0 2024-05-27T07:28:14.960 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:28:14.966+0000 7f8b79a41740 10 client.0 _ll_register_callbacks cb 0x7f8b64009aa0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T07:28:14.960 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:28:14.966+0000 7f8b71123640 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55c5653c21b0 0x55c5657dc970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:28:14.961 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:28:14.966+0000 7f8b6bfff640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55c5653a14d0 0x55c5657dc430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:28:14.961 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:28:14.966+0000 7f8b70922640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55c5653992f0 0x55c5657dbef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:28:14.961 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:28:14.966+0000 7f8b71123640 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55c5653c21b0 0x55c5657dc970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.52:36874/0 (socket says 172.21.15.52:36874) 2024-05-27T07:28:14.961 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:28:14.966+0000 7f8b71123640 1 -- 172.21.15.52:0/1219419566 learned_addr learned my addr 172.21.15.52:0/1219419566 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-05-27T07:28:14.961 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:28:14.966+0000 7f8b71123640 1 -- 172.21.15.52:0/1219419566 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55c5653992f0 msgr2=0x55c5657dbef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:28:14.961 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:28:14.966+0000 7f8b71123640 1 --2- 172.21.15.52:0/1219419566 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55c5653992f0 0x55c5657dbef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:28:14.961 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:28:14.966+0000 7f8b71123640 1 -- 172.21.15.52:0/1219419566 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55c5653a14d0 msgr2=0x55c5657dc430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:28:14.961 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:28:14.966+0000 7f8b71123640 1 --2- 172.21.15.52:0/1219419566 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55c5653a14d0 0x55c5657dc430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:28:14.961 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:28:14.966+0000 7f8b71123640 1 -- 172.21.15.52:0/1219419566 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55c5657e7170 con 0x55c5653c21b0 2024-05-27T07:28:14.961 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:28:14.966+0000 7f8b70922640 1 --2- 172.21.15.52:0/1219419566 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55c5653992f0 0x55c5657dbef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-27T07:28:14.961 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:28:14.966+0000 7f8b6bfff640 1 --2- 172.21.15.52:0/1219419566 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55c5653a14d0 0x55c5657dc430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-27T07:28:14.961 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:28:14.966+0000 7f8b71123640 1 --2- 172.21.15.52:0/1219419566 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55c5653c21b0 0x55c5657dc970 secure :-1 s=READY pgs=567 cs=0 l=1 rev1=1 crypto rx=0x7f8b64018040 tx=0x7f8b64009e40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4e006521bfdeb6ca server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:28:14.961 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:28:14.966+0000 7f8b4bfff640 10 client.0 ms_handle_connect on v2:172.21.15.52:3300/0 2024-05-27T07:28:14.961 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:28:14.966+0000 7f8b4bfff640 1 -- 172.21.15.52:0/1219419566 <== mon.0 v2:172.21.15.52:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8b64036060 con 0x55c5653c21b0 2024-05-27T07:28:14.962 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:28:14.966+0000 7f8b4bfff640 1 -- 172.21.15.52:0/1219419566 <== mon.0 v2:172.21.15.52:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 9379+0+0 (secure 0 0 0) 0x7f8b6401ea60 con 0x55c5653c21b0 2024-05-27T07:28:14.962 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:28:14.966+0000 7f8b79a41740 1 -- 172.21.15.52:0/1219419566 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f8b64004cc0 con 0x55c5653c21b0 2024-05-27T07:28:14.962 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:28:14.967+0000 7f8b4bfff640 1 -- 172.21.15.52:0/1219419566 <== mon.0 v2:172.21.15.52:3300/0 3 ==== mon_get_version_reply(handle=1 version=33) ==== 24+0+0 (secure 0 0 0) 0x7f8b6403d050 con 0x55c5653c21b0 2024-05-27T07:28:14.962 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:28:14.967+0000 7f8b79a41740 10 client.15990 fetch_fsmap learned FSMap version 33 2024-05-27T07:28:14.962 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:28:14.967+0000 7f8b79a41740 1 -- 172.21.15.52:0/1219419566 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({config=0+,fsmap=33,monmap=4+}) -- 0x55c5657e83e0 con 0x55c5653c21b0 2024-05-27T07:28:14.962 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:28:14.967+0000 7f8b4bfff640 1 -- 172.21.15.52:0/1219419566 <== mon.0 v2:172.21.15.52:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f8b64004cc0 con 0x55c5653c21b0 2024-05-27T07:28:14.962 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:28:14.967+0000 7f8b4bfff640 1 -- 172.21.15.52:0/1219419566 <== mon.0 v2:172.21.15.52:3300/0 5 ==== fsmap(e 33) ==== 8374+0+0 (secure 0 0 0) 0x7f8b64030030 con 0x55c5653c21b0 2024-05-27T07:28:14.962 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:28:14.967+0000 7f8b79a41740 10 client.15990 fetch_fsmap finished waiting for FSMap version 33 2024-05-27T07:28:14.962 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:28:14.967+0000 7f8b79a41740 10 client.15990 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-05-27T07:28:14.962 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:28:14.967+0000 7f8b79a41740 20 client.15990 populate_metadata read hostname 'smithi052' 2024-05-27T07:28:14.962 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:28:14.967+0000 7f8b79a41740 1 --2- 172.21.15.52:0/1219419566 >> [v2:172.21.15.105:6838/500348151,v1:172.21.15.105:6839/500348151] conn(0x55c5653561c0 0x55c5657eaaf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:28:14.962 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:28:14.967+0000 7f8b79a41740 4 client.15990 mds_command: new command op to 24527 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-05-27T07:28:14.963 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:28:14.967+0000 7f8b79a41740 1 -- 172.21.15.52:0/1219419566 --> [v2:172.21.15.105:6838/500348151,v1:172.21.15.105:6839/500348151] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55c5656d1360 con 0x55c5653561c0 2024-05-27T07:28:14.963 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:28:14.967+0000 7f8b70922640 1 --2- 172.21.15.52:0/1219419566 >> [v2:172.21.15.105:6838/500348151,v1:172.21.15.105:6839/500348151] conn(0x55c5653561c0 0x55c5657eaaf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:28:14.963 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:28:14.968+0000 7f8b4bfff640 10 client.15990 ms_handle_connect on v2:172.21.15.105:6838/500348151 2024-05-27T07:28:14.963 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:28:14.968+0000 7f8b70922640 1 --2- 172.21.15.52:0/1219419566 >> [v2:172.21.15.105:6838/500348151,v1:172.21.15.105:6839/500348151] conn(0x55c5653561c0 0x55c5657eaaf0 crc :-1 s=READY pgs=78 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=72fc30f849f9a01 server_cookie=d3c5a5735acf160 in_seq=0 out_seq=0 2024-05-27T07:28:15.205 INFO:tasks.workunit.client.0.smithi052.stdout: 21 327847 578 210740 440 177145 1450 2024-05-27T07:28:15.315 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:28:15 smithi052 ceph-mon[31955]: from='mgr.15333 172.21.15.52:0/416389082' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T07:28:15.316 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:28:15 smithi052 ceph-mon[31955]: from='mgr.15333 172.21.15.52:0/416389082' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-27T07:28:15.316 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:28:15 smithi052 ceph-mon[31955]: from='mgr.15333 172.21.15.52:0/416389082' entity='mgr.x' 2024-05-27T07:28:15.481 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:28:15 smithi173 ceph-mon[37000]: from='mgr.15333 172.21.15.52:0/416389082' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T07:28:15.481 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:28:15 smithi173 ceph-mon[37000]: from='mgr.15333 172.21.15.52:0/416389082' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-27T07:28:15.481 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:28:15 smithi173 ceph-mon[37000]: from='mgr.15333 172.21.15.52:0/416389082' entity='mgr.x' 2024-05-27T07:28:15.482 INFO:journalctl@ceph.mds.f.smithi173.stdout:May 27 07:28:15 smithi173 podman[72395]: 2024-05-27 07:28:15.253015925 +0000 UTC m=+0.521441468 container init 11d3b53c90f5ccc17a6b74d217f5c944d8341059a5ed3156713b380c4df8a0df (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-f, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240521, ceph=True, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, GIT_CLEAN=True, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2) 2024-05-27T07:28:15.482 INFO:journalctl@ceph.mds.f.smithi173.stdout:May 27 07:28:15 smithi173 podman[72395]: 2024-05-27 07:28:15.256650093 +0000 UTC m=+0.525075632 container start 11d3b53c90f5ccc17a6b74d217f5c944d8341059a5ed3156713b380c4df8a0df (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-f, org.label-schema.schema-version=1.0, GIT_CLEAN=True, org.label-schema.build-date=20240521, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90) 2024-05-27T07:28:15.482 INFO:journalctl@ceph.mds.f.smithi173.stdout:May 27 07:28:15 smithi173 ceph-mds[72414]: -- 172.21.15.173:0/2822386917 <== mon.1 v2:172.21.15.173:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x558755a0fa40 con 0x558755a57180 2024-05-27T07:28:15.482 INFO:journalctl@ceph.mds.f.smithi173.stdout:May 27 07:28:15 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-f[72409]: starting mds.f at 2024-05-27T07:28:15.482 INFO:journalctl@ceph.mds.f.smithi173.stdout:May 27 07:28:15 smithi173 bash[72395]: 11d3b53c90f5ccc17a6b74d217f5c944d8341059a5ed3156713b380c4df8a0df 2024-05-27T07:28:15.482 INFO:journalctl@ceph.mds.f.smithi173.stdout:May 27 07:28:15 smithi173 systemd[1]: Started Ceph mds.f for 73df2636-1bf8-11ef-bc9b-c7b262605968. 2024-05-27T07:28:15.653 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:15.663+0000 7f1db80d1700 1 -- 172.21.15.52:0/3063311969 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f1db0109840 msgr2=0x7f1db01111a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:28:15.653 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:15.663+0000 7f1db80d1700 1 --2- 172.21.15.52:0/3063311969 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f1db0109840 0x7f1db01111a0 secure :-1 s=READY pgs=568 cs=0 l=1 rev1=1 crypto rx=0x7f1da80099c0 tx=0x7f1da8009cd0 comp rx=0 tx=0).stop 2024-05-27T07:28:15.654 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:15.664+0000 7f1db80d1700 1 -- 172.21.15.52:0/3063311969 shutdown_connections 2024-05-27T07:28:15.654 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:15.664+0000 7f1db80d1700 1 --2- 172.21.15.52:0/3063311969 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f1db0109840 0x7f1db01111a0 unknown :-1 s=CLOSED pgs=568 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:28:15.654 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:15.664+0000 7f1db80d1700 1 --2- 172.21.15.52:0/3063311969 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f1db0102ad0 0x7f1db0102ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:28:15.654 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:15.664+0000 7f1db80d1700 1 --2- 172.21.15.52:0/3063311969 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f1db0108ef0 0x7f1db0109300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:28:15.654 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:15.664+0000 7f1db80d1700 1 -- 172.21.15.52:0/3063311969 >> 172.21.15.52:0/3063311969 conn(0x7f1db00fe4d0 msgr2=0x7f1db0100920 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:28:15.654 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:15.664+0000 7f1db80d1700 1 -- 172.21.15.52:0/3063311969 shutdown_connections 2024-05-27T07:28:15.654 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:15.664+0000 7f1db80d1700 1 -- 172.21.15.52:0/3063311969 wait complete. 2024-05-27T07:28:15.656 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:15.664+0000 7f1db80d1700 1 Processor -- start 2024-05-27T07:28:15.656 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:15.664+0000 7f1db80d1700 1 -- start start 2024-05-27T07:28:15.656 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:15.665+0000 7f1db80d1700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f1db0102ad0 0x7f1db019cfe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:28:15.656 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:15.665+0000 7f1db80d1700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f1db0108ef0 0x7f1db019d580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:28:15.656 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:15.665+0000 7f1db80d1700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f1db0109840 0x7f1db01a1810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:28:15.656 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:15.665+0000 7f1db80d1700 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x7f1db0114470 con 0x7f1db0102ad0 2024-05-27T07:28:15.656 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:15.665+0000 7f1db80d1700 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f1db01142f0 con 0x7f1db0108ef0 2024-05-27T07:28:15.656 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:15.665+0000 7f1db80d1700 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f1db01145f0 con 0x7f1db0109840 2024-05-27T07:28:15.656 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:15.665+0000 7f1db5e6d700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f1db0102ad0 0x7f1db019cfe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:28:15.656 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:15.665+0000 7f1db5e6d700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f1db0102ad0 0x7f1db019cfe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.52:36902/0 (socket says 172.21.15.52:36902) 2024-05-27T07:28:15.657 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:15.665+0000 7f1db5e6d700 1 -- 172.21.15.52:0/3711045425 learned_addr learned my addr 172.21.15.52:0/3711045425 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-05-27T07:28:15.657 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:15.665+0000 7f1db566c700 1 --2- 172.21.15.52:0/3711045425 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f1db0108ef0 0x7f1db019d580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:28:15.657 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:15.665+0000 7f1db666e700 1 --2- 172.21.15.52:0/3711045425 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f1db0109840 0x7f1db01a1810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:28:15.657 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:15.665+0000 7f1db5e6d700 1 -- 172.21.15.52:0/3711045425 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f1db0109840 msgr2=0x7f1db01a1810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:28:15.657 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:15.665+0000 7f1db5e6d700 1 --2- 172.21.15.52:0/3711045425 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f1db0109840 0x7f1db01a1810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:28:15.657 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:15.665+0000 7f1db5e6d700 1 -- 172.21.15.52:0/3711045425 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f1db0108ef0 msgr2=0x7f1db019d580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:28:15.657 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:15.665+0000 7f1db5e6d700 1 --2- 172.21.15.52:0/3711045425 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f1db0108ef0 0x7f1db019d580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:28:15.657 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:15.665+0000 7f1db5e6d700 1 -- 172.21.15.52:0/3711045425 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1db01a1eb0 con 0x7f1db0102ad0 2024-05-27T07:28:15.657 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:15.665+0000 7f1db5e6d700 1 --2- 172.21.15.52:0/3711045425 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f1db0102ad0 0x7f1db019cfe0 secure :-1 s=READY pgs=569 cs=0 l=1 rev1=1 crypto rx=0x7f1da000ef10 tx=0x7f1da000c3a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:28:15.657 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:15.666+0000 7f1daeffd700 1 -- 172.21.15.52:0/3711045425 <== mon.0 v2:172.21.15.52:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1da000c9c0 con 0x7f1db0102ad0 2024-05-27T07:28:15.657 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:15.666+0000 7f1daeffd700 1 -- 172.21.15.52:0/3711045425 <== mon.0 v2:172.21.15.52:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f1da0019070 con 0x7f1db0102ad0 2024-05-27T07:28:15.657 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:15.666+0000 7f1db80d1700 1 -- 172.21.15.52:0/3711045425 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1db01a2140 con 0x7f1db0102ad0 2024-05-27T07:28:15.658 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:15.666+0000 7f1daeffd700 1 -- 172.21.15.52:0/3711045425 <== mon.0 v2:172.21.15.52:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1da0014480 con 0x7f1db0102ad0 2024-05-27T07:28:15.658 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:15.666+0000 7f1db80d1700 1 -- 172.21.15.52:0/3711045425 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1db01a9ac0 con 0x7f1db0102ad0 2024-05-27T07:28:15.658 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:15.668+0000 7f1daeffd700 1 -- 172.21.15.52:0/3711045425 <== mon.0 v2:172.21.15.52:3300/0 4 ==== mgrmap(e 26) ==== 149783+0+0 (secure 0 0 0) 0x7f1da0012050 con 0x7f1db0102ad0 2024-05-27T07:28:15.659 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:15.670+0000 7f1daeffd700 1 --2- 172.21.15.52:0/3711045425 >> [v2:172.21.15.52:6800/1822318384,v1:172.21.15.52:6801/1822318384] conn(0x7f1d980b29f0 0x7f1d980b4f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:28:15.659 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:15.670+0000 7f1db566c700 1 --2- 172.21.15.52:0/3711045425 >> [v2:172.21.15.52:6800/1822318384,v1:172.21.15.52:6801/1822318384] conn(0x7f1d980b29f0 0x7f1d980b4f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:28:15.659 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:15.670+0000 7f1daeffd700 1 -- 172.21.15.52:0/3711045425 <== mon.0 v2:172.21.15.52:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 9379+0+0 (secure 0 0 0) 0x7f1da00e28d0 con 0x7f1db0102ad0 2024-05-27T07:28:15.661 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:15.670+0000 7f1db566c700 1 --2- 172.21.15.52:0/3711045425 >> [v2:172.21.15.52:6800/1822318384,v1:172.21.15.52:6801/1822318384] conn(0x7f1d980b29f0 0x7f1d980b4f00 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f1db0101f30 tx=0x7f1da400a300 comp rx=0 tx=0).ready entity=mgr.15333 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:28:15.661 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:15.670+0000 7f1dc6ae6b80 1 Processor -- start 2024-05-27T07:28:15.661 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:15.670+0000 7f1dc6ae6b80 1 -- start start 2024-05-27T07:28:15.661 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:15.671+0000 7f1dc6ae6b80 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55b459d5de40 0x55b459ee1710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:28:15.661 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:15.671+0000 7f1dc6ae6b80 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55b459ee1c50 0x55b459edba40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:28:15.661 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:15.671+0000 7f1dc6ae6b80 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55b459edbf80 0x55b459edc390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:28:15.661 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:15.671+0000 7f1dc6ae6b80 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x55b459e0ac90 con 0x55b459d5de40 2024-05-27T07:28:15.662 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:15.671+0000 7f1dc6ae6b80 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x55b459dfa830 con 0x55b459ee1c50 2024-05-27T07:28:15.662 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:15.671+0000 7f1dc6ae6b80 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x55b459dfe870 con 0x55b459edbf80 2024-05-27T07:28:15.662 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:15.671+0000 7f1db5e6d700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55b459d5de40 0x55b459ee1710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:28:15.662 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:15.671+0000 7f1db5e6d700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55b459d5de40 0x55b459ee1710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.52:36914/0 (socket says 172.21.15.52:36914) 2024-05-27T07:28:15.662 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:15.671+0000 7f1db5e6d700 1 -- 172.21.15.52:0/2757803922 learned_addr learned my addr 172.21.15.52:0/2757803922 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-05-27T07:28:15.662 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:15.671+0000 7f1db566c700 1 --2- 172.21.15.52:0/2757803922 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55b459ee1c50 0x55b459edba40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:28:15.662 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:15.671+0000 7f1db666e700 1 --2- 172.21.15.52:0/2757803922 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55b459edbf80 0x55b459edc390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:28:15.662 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:15.671+0000 7f1db5e6d700 1 -- 172.21.15.52:0/2757803922 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55b459edbf80 msgr2=0x55b459edc390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:28:15.662 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:15.671+0000 7f1db5e6d700 1 --2- 172.21.15.52:0/2757803922 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55b459edbf80 0x55b459edc390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:28:15.662 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:15.671+0000 7f1db5e6d700 1 -- 172.21.15.52:0/2757803922 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55b459ee1c50 msgr2=0x55b459edba40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:28:15.662 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:15.671+0000 7f1db5e6d700 1 --2- 172.21.15.52:0/2757803922 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55b459ee1c50 0x55b459edba40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:28:15.662 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:15.671+0000 7f1db5e6d700 1 -- 172.21.15.52:0/2757803922 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55b459d4a8d0 con 0x55b459d5de40 2024-05-27T07:28:15.662 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:15.671+0000 7f1db666e700 1 --2- 172.21.15.52:0/2757803922 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55b459edbf80 0x55b459edc390 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-27T07:28:15.663 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:15.671+0000 7f1db5e6d700 1 --2- 172.21.15.52:0/2757803922 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55b459d5de40 0x55b459ee1710 secure :-1 s=READY pgs=570 cs=0 l=1 rev1=1 crypto rx=0x7f1da00a9550 tx=0x7f1da00a9710 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cd6260ef360db53f server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:28:15.663 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:15.672+0000 7f1d97fff700 1 -- 172.21.15.52:0/2757803922 <== mon.0 v2:172.21.15.52:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1da00a9360 con 0x55b459d5de40 2024-05-27T07:28:15.663 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:15.672+0000 7f1d97fff700 1 -- 172.21.15.52:0/2757803922 <== mon.0 v2:172.21.15.52:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f1da00a9f50 con 0x55b459d5de40 2024-05-27T07:28:15.663 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:15.672+0000 7f1d97fff700 1 -- 172.21.15.52:0/2757803922 <== mon.0 v2:172.21.15.52:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1da00a9ab0 con 0x55b459d5de40 2024-05-27T07:28:15.664 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:15.672+0000 7f1dc6ae6b80 1 -- 172.21.15.52:0/2757803922 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55b459d5de40 msgr2=0x55b459ee1710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:28:15.665 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:15.672+0000 7f1dc6ae6b80 1 --2- 172.21.15.52:0/2757803922 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55b459d5de40 0x55b459ee1710 secure :-1 s=READY pgs=570 cs=0 l=1 rev1=1 crypto rx=0x7f1da00a9550 tx=0x7f1da00a9710 comp rx=0 tx=0).stop 2024-05-27T07:28:15.665 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:15.673+0000 7f1dc6ae6b80 1 -- 172.21.15.52:0/2757803922 shutdown_connections 2024-05-27T07:28:15.665 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:15.673+0000 7f1dc6ae6b80 1 --2- 172.21.15.52:0/2757803922 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55b459edbf80 0x55b459edc390 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:28:15.665 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:15.673+0000 7f1dc6ae6b80 1 --2- 172.21.15.52:0/2757803922 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55b459d5de40 0x55b459ee1710 unknown :-1 s=CLOSED pgs=570 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:28:15.665 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:15.673+0000 7f1dc6ae6b80 1 --2- 172.21.15.52:0/2757803922 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55b459ee1c50 0x55b459edba40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:28:15.665 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:15.673+0000 7f1dc6ae6b80 1 -- 172.21.15.52:0/2757803922 >> 172.21.15.52:0/2757803922 conn(0x55b459db8af0 msgr2=0x55b459ed6640 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:28:15.665 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:15.673+0000 7f1dc6ae6b80 1 -- 172.21.15.52:0/2757803922 shutdown_connections 2024-05-27T07:28:15.665 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:15.673+0000 7f1dc6ae6b80 1 -- 172.21.15.52:0/2757803922 wait complete. 2024-05-27T07:28:15.665 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:15.673+0000 7f1dc6ae6b80 1 -- start start 2024-05-27T07:28:15.665 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:15.673+0000 7f1dc6ae6b80 1 Processor -- start 2024-05-27T07:28:15.665 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:15.674+0000 7f1dc6ae6b80 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55b459d5de40 0x55b459ee54f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:28:15.665 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:15.674+0000 7f1dc6ae6b80 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55b459edbf80 0x55b459ee5a30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:28:15.666 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:15.674+0000 7f1dc6ae6b80 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55b459ee1c50 0x55b459ee5f70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:28:15.666 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:15.674+0000 7f1dc6ae6b80 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x55b459d00fa0 con 0x55b459d5de40 2024-05-27T07:28:15.666 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:15.674+0000 7f1dc6ae6b80 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x55b459d9c7c0 con 0x55b459ee1c50 2024-05-27T07:28:15.666 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:15.674+0000 7f1dc6ae6b80 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x55b459ee86e0 con 0x55b459edbf80 2024-05-27T07:28:15.666 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:15.674+0000 7f1dc6ae6b80 10 client.0 _ll_register_callbacks cb 0x55b459d95920 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T07:28:15.666 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:15.674+0000 7f1db5e6d700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55b459d5de40 0x55b459ee54f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:28:15.666 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:15.674+0000 7f1db5e6d700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55b459d5de40 0x55b459ee54f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.52:36926/0 (socket says 172.21.15.52:36926) 2024-05-27T07:28:15.666 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:15.674+0000 7f1db5e6d700 1 -- 172.21.15.52:0/3697704298 learned_addr learned my addr 172.21.15.52:0/3697704298 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-05-27T07:28:15.666 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:15.674+0000 7f1db666e700 1 --2- 172.21.15.52:0/3697704298 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55b459ee1c50 0x55b459ee5f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:28:15.666 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:15.674+0000 7f1db566c700 1 --2- 172.21.15.52:0/3697704298 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55b459edbf80 0x55b459ee5a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:28:15.666 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:15.674+0000 7f1db5e6d700 1 -- 172.21.15.52:0/3697704298 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55b459edbf80 msgr2=0x55b459ee5a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:28:15.666 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:15.674+0000 7f1db5e6d700 1 --2- 172.21.15.52:0/3697704298 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55b459edbf80 0x55b459ee5a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:28:15.666 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:15.674+0000 7f1db5e6d700 1 -- 172.21.15.52:0/3697704298 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55b459ee1c50 msgr2=0x55b459ee5f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:28:15.667 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:15.674+0000 7f1db5e6d700 1 --2- 172.21.15.52:0/3697704298 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55b459ee1c50 0x55b459ee5f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:28:15.667 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:15.674+0000 7f1db5e6d700 1 -- 172.21.15.52:0/3697704298 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55b459d99690 con 0x55b459d5de40 2024-05-27T07:28:15.667 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:15.674+0000 7f1db666e700 1 --2- 172.21.15.52:0/3697704298 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55b459ee1c50 0x55b459ee5f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-27T07:28:15.667 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:15.674+0000 7f1db5e6d700 1 --2- 172.21.15.52:0/3697704298 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55b459d5de40 0x55b459ee54f0 secure :-1 s=READY pgs=571 cs=0 l=1 rev1=1 crypto rx=0x7f1da00a97e0 tx=0x7f1da00a5d10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=de1c0b778dfd2d02 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:28:15.667 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:15.674+0000 7f1d957fa700 10 client.0 ms_handle_connect on v2:172.21.15.52:3300/0 2024-05-27T07:28:15.667 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:15.675+0000 7f1d957fa700 1 -- 172.21.15.52:0/3697704298 <== mon.0 v2:172.21.15.52:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1da00c3060 con 0x55b459d5de40 2024-05-27T07:28:15.667 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:15.675+0000 7f1dc6ae6b80 1 -- 172.21.15.52:0/3697704298 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f1da00a9f50 con 0x55b459d5de40 2024-05-27T07:28:15.667 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:15.675+0000 7f1d957fa700 1 -- 172.21.15.52:0/3697704298 <== mon.0 v2:172.21.15.52:3300/0 2 ==== osd_map(92..92 src has 1..92) ==== 9379+0+0 (secure 0 0 0) 0x7f1da00b2430 con 0x55b459d5de40 2024-05-27T07:28:15.667 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:15.675+0000 7f1d957fa700 1 -- 172.21.15.52:0/3697704298 <== mon.0 v2:172.21.15.52:3300/0 3 ==== mon_get_version_reply(handle=1 version=33) ==== 24+0+0 (secure 0 0 0) 0x7f1da00adc00 con 0x55b459d5de40 2024-05-27T07:28:15.667 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:15.675+0000 7f1dc6ae6b80 10 client.16008 fetch_fsmap learned FSMap version 33 2024-05-27T07:28:15.667 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:15.675+0000 7f1dc6ae6b80 1 -- 172.21.15.52:0/3697704298 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({config=0+,fsmap=33,monmap=4+}) -- 0x55b459ef32b0 con 0x55b459d5de40 2024-05-27T07:28:15.667 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:15.675+0000 7f1d957fa700 1 -- 172.21.15.52:0/3697704298 <== mon.0 v2:172.21.15.52:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f1da00a9f50 con 0x55b459d5de40 2024-05-27T07:28:15.668 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:15.675+0000 7f1d957fa700 1 -- 172.21.15.52:0/3697704298 <== mon.0 v2:172.21.15.52:3300/0 5 ==== fsmap(e 33) ==== 8374+0+0 (secure 0 0 0) 0x7f1da00910d0 con 0x55b459d5de40 2024-05-27T07:28:15.668 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:15.675+0000 7f1dc6ae6b80 10 client.16008 fetch_fsmap finished waiting for FSMap version 33 2024-05-27T07:28:15.668 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:15.675+0000 7f1dc6ae6b80 10 client.16008 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-05-27T07:28:15.668 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:15.675+0000 7f1dc6ae6b80 20 client.16008 populate_metadata read hostname 'smithi052' 2024-05-27T07:28:15.668 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:15.675+0000 7f1dc6ae6b80 1 --2- 172.21.15.52:0/3697704298 >> [v2:172.21.15.105:6838/500348151,v1:172.21.15.105:6839/500348151] conn(0x55b459ef5780 0x55b459ef7ba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:28:15.668 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:15.675+0000 7f1dc6ae6b80 4 client.16008 mds_command: new command op to 24527 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-27T07:28:15.668 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:15.675+0000 7f1dc6ae6b80 1 -- 172.21.15.52:0/3697704298 --> [v2:172.21.15.105:6838/500348151,v1:172.21.15.105:6839/500348151] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55b459ea79f0 con 0x55b459ef5780 2024-05-27T07:28:15.669 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:15.676+0000 7f1db566c700 1 --2- 172.21.15.52:0/3697704298 >> [v2:172.21.15.105:6838/500348151,v1:172.21.15.105:6839/500348151] conn(0x55b459ef5780 0x55b459ef7ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:28:15.669 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:15.677+0000 7f1db566c700 1 --2- 172.21.15.52:0/3697704298 >> [v2:172.21.15.105:6838/500348151,v1:172.21.15.105:6839/500348151] conn(0x55b459ef5780 0x55b459ef7ba0 crc :-1 s=READY pgs=79 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=4bd493b53309a5b3 server_cookie=b3682010a4847b7c in_seq=0 out_seq=0 2024-05-27T07:28:15.669 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:15.677+0000 7f1d957fa700 10 client.16008 ms_handle_connect on v2:172.21.15.105:6838/500348151 2024-05-27T07:28:15.681 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:15.691+0000 7f1d957fa700 1 -- 172.21.15.52:0/3697704298 <== mds.0 v2:172.21.15.105:6838/500348151 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24041 (crc 0 0 0) 0x55b459ea79f0 con 0x55b459ef5780 2024-05-27T07:28:15.682 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:15.691+0000 7f1d957fa700 10 client.16008 handle_command_reply: tid=0 2024-05-27T07:28:15.682 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:15.691+0000 7f1dc6ae6b80 1 client.16008 shutdown 2024-05-27T07:28:15.682 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:15.692+0000 7f1dc6ae6b80 1 -- 172.21.15.52:0/3697704298 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55b459d5de40 msgr2=0x55b459ee54f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:28:15.682 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:15.692+0000 7f1dc6ae6b80 1 --2- 172.21.15.52:0/3697704298 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55b459d5de40 0x55b459ee54f0 secure :-1 s=READY pgs=571 cs=0 l=1 rev1=1 crypto rx=0x7f1da00a97e0 tx=0x7f1da00a5d10 comp rx=0 tx=0).stop 2024-05-27T07:28:15.684 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:15.694+0000 7f1dc6ae6b80 1 -- 172.21.15.52:0/3697704298 shutdown_connections 2024-05-27T07:28:15.684 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:15.694+0000 7f1dc6ae6b80 1 --2- 172.21.15.52:0/3697704298 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55b459ee1c50 0x55b459ee5f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:28:15.684 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:15.694+0000 7f1dc6ae6b80 1 --2- 172.21.15.52:0/3697704298 >> [v2:172.21.15.105:6838/500348151,v1:172.21.15.105:6839/500348151] conn(0x55b459ef5780 0x55b459ef7ba0 crc :-1 s=READY pgs=79 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:28:15.684 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:15.694+0000 7f1dc6ae6b80 1 --2- 172.21.15.52:0/3697704298 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55b459d5de40 0x55b459ee54f0 unknown :-1 s=CLOSED pgs=571 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:28:15.684 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:15.694+0000 7f1d957fa700 0 client.16008 ms_handle_reset on v2:172.21.15.105:6838/500348151 2024-05-27T07:28:15.685 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:15.694+0000 7f1dc6ae6b80 1 --2- 172.21.15.52:0/3697704298 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55b459edbf80 0x55b459ee5a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:28:15.685 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:15.694+0000 7f1dc6ae6b80 1 -- 172.21.15.52:0/3697704298 >> 172.21.15.52:0/3697704298 conn(0x55b459db8af0 msgr2=0x55b459edaa30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:28:15.689 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:28:15 smithi105 ceph-mon[35977]: from='mgr.15333 172.21.15.52:0/416389082' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T07:28:15.689 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:28:15 smithi105 ceph-mon[35977]: from='mgr.15333 172.21.15.52:0/416389082' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-27T07:28:15.689 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:28:15 smithi105 ceph-mon[35977]: from='mgr.15333 172.21.15.52:0/416389082' entity='mgr.x' 2024-05-27T07:28:15.702 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:15.696+0000 7f1dc6ae6b80 1 -- 172.21.15.52:0/3697704298 shutdown_connections 2024-05-27T07:28:15.702 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:15.696+0000 7f1dc6ae6b80 1 -- 172.21.15.52:0/3697704298 wait complete. 2024-05-27T07:28:15.702 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:15.696+0000 7f1dc6ae6b80 20 client.16008 trim_cache size 0 max 16384 2024-05-27T07:28:15.737 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:15.745+0000 7f1dc6ae6b80 1 Processor -- start 2024-05-27T07:28:15.737 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:15.745+0000 7f1dc6ae6b80 1 -- start start 2024-05-27T07:28:15.739 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:15.746+0000 7f1dc6ae6b80 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55b459d5de40 0x55b459edba40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:28:15.739 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:15.746+0000 7f1dc6ae6b80 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55b459edbf80 0x55b459ee3070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:28:15.739 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:15.746+0000 7f1dc6ae6b80 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55b459ee1c50 0x55b459edf010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:28:15.739 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:15.746+0000 7f1dc6ae6b80 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x7f1db01b2710 con 0x55b459edbf80 2024-05-27T07:28:15.739 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:15.746+0000 7f1dc6ae6b80 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f1db01b42e0 con 0x55b459d5de40 2024-05-27T07:28:15.740 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:15.746+0000 7f1dc6ae6b80 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f1db01b2de0 con 0x55b459ee1c50 2024-05-27T07:28:15.740 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:15.748+0000 7f1db666e700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55b459ee1c50 0x55b459edf010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:28:15.740 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:15.748+0000 7f1db666e700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55b459ee1c50 0x55b459edf010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.173:3300/0 says I am v2:172.21.15.52:54632/0 (socket says 172.21.15.52:54632) 2024-05-27T07:28:15.740 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:15.748+0000 7f1db666e700 1 -- 172.21.15.52:0/2618550683 learned_addr learned my addr 172.21.15.52:0/2618550683 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-05-27T07:28:15.740 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:15.748+0000 7f1db5e6d700 1 --2- 172.21.15.52:0/2618550683 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55b459d5de40 0x55b459edba40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:28:15.740 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:15.748+0000 7f1db566c700 1 --2- 172.21.15.52:0/2618550683 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55b459edbf80 0x55b459ee3070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:28:15.741 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:15.751+0000 7f1db666e700 1 -- 172.21.15.52:0/2618550683 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55b459d5de40 msgr2=0x55b459edba40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:28:15.741 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:15.751+0000 7f1db666e700 1 --2- 172.21.15.52:0/2618550683 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55b459d5de40 0x55b459edba40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:28:15.741 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:15.751+0000 7f1db666e700 1 -- 172.21.15.52:0/2618550683 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55b459edbf80 msgr2=0x55b459ee3070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:28:15.741 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:15.751+0000 7f1db666e700 1 --2- 172.21.15.52:0/2618550683 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55b459edbf80 0x55b459ee3070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:28:15.741 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:15.751+0000 7f1db666e700 1 -- 172.21.15.52:0/2618550683 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55b459ed3090 con 0x55b459ee1c50 2024-05-27T07:28:15.745 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:15.755+0000 7f1db666e700 1 --2- 172.21.15.52:0/2618550683 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55b459ee1c50 0x55b459edf010 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f1da8015000 tx=0x7f1da8005040 comp rx=0 tx=0).ready entity=mon.1 client_cookie=f05a26e41bf9b295 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:28:15.758 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:15.763+0000 7f1d977fe700 1 -- 172.21.15.52:0/2618550683 <== mon.1 v2:172.21.15.173:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1da80037c0 con 0x55b459ee1c50 2024-05-27T07:28:15.758 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:15.763+0000 7f1d977fe700 1 -- 172.21.15.52:0/2618550683 <== mon.1 v2:172.21.15.173:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f1da8003e10 con 0x55b459ee1c50 2024-05-27T07:28:15.758 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:15.763+0000 7f1d977fe700 1 -- 172.21.15.52:0/2618550683 <== mon.1 v2:172.21.15.173:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1da800f6c0 con 0x55b459ee1c50 2024-05-27T07:28:15.763 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:15.768+0000 7f1dc6ae6b80 1 -- 172.21.15.52:0/2618550683 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55b459ee1c50 msgr2=0x55b459edf010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:28:15.763 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:15.768+0000 7f1dc6ae6b80 1 --2- 172.21.15.52:0/2618550683 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55b459ee1c50 0x55b459edf010 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f1da8015000 tx=0x7f1da8005040 comp rx=0 tx=0).stop 2024-05-27T07:28:15.763 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:15.771+0000 7f1dc6ae6b80 1 -- 172.21.15.52:0/2618550683 shutdown_connections 2024-05-27T07:28:15.763 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:15.771+0000 7f1dc6ae6b80 1 --2- 172.21.15.52:0/2618550683 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55b459ee1c50 0x55b459edf010 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:28:15.763 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:15.771+0000 7f1dc6ae6b80 1 --2- 172.21.15.52:0/2618550683 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55b459d5de40 0x55b459edba40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:28:15.763 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:15.771+0000 7f1dc6ae6b80 1 --2- 172.21.15.52:0/2618550683 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55b459edbf80 0x55b459ee3070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:28:15.763 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:15.771+0000 7f1dc6ae6b80 1 -- 172.21.15.52:0/2618550683 >> 172.21.15.52:0/2618550683 conn(0x55b459db8af0 msgr2=0x55b459ee3de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:28:15.765 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:15.775+0000 7f1dc6ae6b80 1 -- 172.21.15.52:0/2618550683 shutdown_connections 2024-05-27T07:28:15.766 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:15.775+0000 7f1dc6ae6b80 1 -- 172.21.15.52:0/2618550683 wait complete. 2024-05-27T07:28:15.766 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:15.776+0000 7f1dc6ae6b80 1 -- start start 2024-05-27T07:28:15.766 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:15.776+0000 7f1dc6ae6b80 1 Processor -- start 2024-05-27T07:28:15.768 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:15.777+0000 7f1dc6ae6b80 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55b459d5de40 0x55b459efb3a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:28:15.768 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:15.777+0000 7f1dc6ae6b80 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55b459edbf80 0x55b459ef9180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:28:15.768 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:15.777+0000 7f1dc6ae6b80 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55b459ee1c50 0x55b459ef96c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:28:15.768 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:15.777+0000 7f1db5e6d700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55b459d5de40 0x55b459efb3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:28:15.768 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:15.778+0000 7f1db5e6d700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55b459d5de40 0x55b459efb3a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.52:46862/0 (socket says 172.21.15.52:46862) 2024-05-27T07:28:15.768 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:15.778+0000 7f1db5e6d700 1 -- 172.21.15.52:0/2530078235 learned_addr learned my addr 172.21.15.52:0/2530078235 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-05-27T07:28:15.768 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:15.778+0000 7f1db666e700 1 --2- 172.21.15.52:0/2530078235 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55b459ee1c50 0x55b459ef96c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:28:15.768 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:15.778+0000 7f1dc6ae6b80 1 -- 172.21.15.52:0/2530078235 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x7f1db01b3b80 con 0x55b459ee1c50 2024-05-27T07:28:15.768 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:15.778+0000 7f1dc6ae6b80 1 -- 172.21.15.52:0/2530078235 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f1db01b34b0 con 0x55b459d5de40 2024-05-27T07:28:15.770 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:15.778+0000 7f1dc6ae6b80 1 -- 172.21.15.52:0/2530078235 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f1db01b06b0 con 0x55b459edbf80 2024-05-27T07:28:15.770 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:15.780+0000 7f1dc6ae6b80 10 client.0 _ll_register_callbacks cb 0x55b459d95920 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T07:28:15.770 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:15.781+0000 7f1db566c700 1 --2- 172.21.15.52:0/2530078235 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55b459edbf80 0x55b459ef9180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:28:15.780 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:15.785+0000 7f1db5e6d700 1 -- 172.21.15.52:0/2530078235 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55b459edbf80 msgr2=0x55b459ef9180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:28:15.780 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:15.785+0000 7f1db5e6d700 1 --2- 172.21.15.52:0/2530078235 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55b459edbf80 0x55b459ef9180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:28:15.780 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:15.785+0000 7f1db5e6d700 1 -- 172.21.15.52:0/2530078235 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55b459ee1c50 msgr2=0x55b459ef96c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:28:15.780 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:15.785+0000 7f1db5e6d700 1 --2- 172.21.15.52:0/2530078235 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55b459ee1c50 0x55b459ef96c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:28:15.780 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:15.785+0000 7f1db5e6d700 1 -- 172.21.15.52:0/2530078235 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55b459ef9db0 con 0x55b459d5de40 2024-05-27T07:28:15.781 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:15.786+0000 7f1db5e6d700 1 --2- 172.21.15.52:0/2530078235 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55b459d5de40 0x55b459efb3a0 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f1da00a6ac0 tx=0x7f1da00a9710 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f7d4123aa8a54667 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:28:15.781 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:15.786+0000 7f1d957fa700 10 client.0 ms_handle_connect on v2:172.21.15.105:3300/0 2024-05-27T07:28:15.781 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:15.787+0000 7f1d957fa700 1 -- 172.21.15.52:0/2530078235 <== mon.2 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1da00c3070 con 0x55b459d5de40 2024-05-27T07:28:15.781 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:15.787+0000 7f1d957fa700 1 -- 172.21.15.52:0/2530078235 <== mon.2 v2:172.21.15.105:3300/0 2 ==== osd_map(92..92 src has 1..92) ==== 9379+0+0 (secure 0 0 0) 0x7f1da006f170 con 0x55b459d5de40 2024-05-27T07:28:15.781 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:15.787+0000 7f1dc6ae6b80 1 -- 172.21.15.52:0/2530078235 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f1da8003e10 con 0x55b459d5de40 2024-05-27T07:28:15.781 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:15.787+0000 7f1d957fa700 1 -- 172.21.15.52:0/2530078235 <== mon.2 v2:172.21.15.105:3300/0 3 ==== mon_get_version_reply(handle=1 version=33) ==== 24+0+0 (secure 0 0 0) 0x7f1da00a9f50 con 0x55b459d5de40 2024-05-27T07:28:15.782 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:15.789+0000 7f1dc6ae6b80 10 client.25691 fetch_fsmap learned FSMap version 33 2024-05-27T07:28:15.782 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:15.789+0000 7f1dc6ae6b80 1 -- 172.21.15.52:0/2530078235 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,fsmap=33,monmap=4+}) -- 0x55b459f686a0 con 0x55b459d5de40 2024-05-27T07:28:15.783 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:15.793+0000 7f1d957fa700 1 -- 172.21.15.52:0/2530078235 <== mon.2 v2:172.21.15.105:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f1da00a9850 con 0x55b459d5de40 2024-05-27T07:28:15.783 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:15.793+0000 7f1d957fa700 1 -- 172.21.15.52:0/2530078235 <== mon.2 v2:172.21.15.105:3300/0 5 ==== fsmap(e 33) ==== 8374+0+0 (secure 0 0 0) 0x7f1da00a51c0 con 0x55b459d5de40 2024-05-27T07:28:15.785 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:15.794+0000 7f1dc6ae6b80 10 client.25691 fetch_fsmap finished waiting for FSMap version 33 2024-05-27T07:28:15.785 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:15.794+0000 7f1dc6ae6b80 10 client.25691 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-05-27T07:28:15.785 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:15.794+0000 7f1dc6ae6b80 20 client.25691 populate_metadata read hostname 'smithi052' 2024-05-27T07:28:15.785 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:15.794+0000 7f1dc6ae6b80 1 --2- 172.21.15.52:0/2530078235 >> [v2:172.21.15.105:6838/500348151,v1:172.21.15.105:6839/500348151] conn(0x55b459ef5780 0x55b459f6aa70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:28:15.785 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:15.795+0000 7f1dc6ae6b80 4 client.25691 mds_command: new command op to 24527 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-05-27T07:28:15.785 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:15.795+0000 7f1dc6ae6b80 1 -- 172.21.15.52:0/2530078235 --> [v2:172.21.15.105:6838/500348151,v1:172.21.15.105:6839/500348151] -- command(tid 0: {"prefix": "scrub status"}) -- 0x7f1da8003e10 con 0x55b459ef5780 2024-05-27T07:28:15.785 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:15.796+0000 7f1db566c700 1 --2- 172.21.15.52:0/2530078235 >> [v2:172.21.15.105:6838/500348151,v1:172.21.15.105:6839/500348151] conn(0x55b459ef5780 0x55b459f6aa70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:28:15.796 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:15.807+0000 7f1d957fa700 10 client.25691 ms_handle_connect on v2:172.21.15.105:6838/500348151 2024-05-27T07:28:15.797 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:15.807+0000 7f1db566c700 1 --2- 172.21.15.52:0/2530078235 >> [v2:172.21.15.105:6838/500348151,v1:172.21.15.105:6839/500348151] conn(0x55b459ef5780 0x55b459f6aa70 crc :-1 s=READY pgs=80 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=ec1ccdaefd99918 server_cookie=ef08008bb09b5225 in_seq=0 out_seq=0 2024-05-27T07:28:15.812 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:15.822+0000 7f1d957fa700 1 -- 172.21.15.52:0/2530078235 <== mds.0 v2:172.21.15.105:6838/500348151 1 ==== command_reply(tid 0: 0 ) ==== 8+0+63 (crc 0 0 0) 0x7f1da8003e10 con 0x55b459ef5780 2024-05-27T07:28:15.816 INFO:teuthology.orchestra.run.smithi052.stdout:{ 2024-05-27T07:28:15.816 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "no active scrubs running", 2024-05-27T07:28:15.816 INFO:teuthology.orchestra.run.smithi052.stdout: "scrubs": {} 2024-05-27T07:28:15.816 INFO:teuthology.orchestra.run.smithi052.stdout:} 2024-05-27T07:28:15.816 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:15.822+0000 7f1d957fa700 10 client.25691 handle_command_reply: tid=0 2024-05-27T07:28:15.816 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:15.822+0000 7f1dc6ae6b80 1 client.25691 shutdown 2024-05-27T07:28:15.817 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:15.825+0000 7f1dc6ae6b80 1 -- 172.21.15.52:0/2530078235 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55b459d5de40 msgr2=0x55b459efb3a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:28:15.817 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:15.825+0000 7f1dc6ae6b80 1 --2- 172.21.15.52:0/2530078235 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55b459d5de40 0x55b459efb3a0 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f1da00a6ac0 tx=0x7f1da00a9710 comp rx=0 tx=0).stop 2024-05-27T07:28:15.817 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:15.825+0000 7f1dc6ae6b80 1 -- 172.21.15.52:0/2530078235 shutdown_connections 2024-05-27T07:28:15.817 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:15.825+0000 7f1dc6ae6b80 1 --2- 172.21.15.52:0/2530078235 >> [v2:172.21.15.105:6838/500348151,v1:172.21.15.105:6839/500348151] conn(0x55b459ef5780 0x55b459f6aa70 crc :-1 s=READY pgs=80 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:28:15.817 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:15.825+0000 7f1dc6ae6b80 1 --2- 172.21.15.52:0/2530078235 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55b459ee1c50 0x55b459ef96c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:28:15.817 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:15.825+0000 7f1dc6ae6b80 1 --2- 172.21.15.52:0/2530078235 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55b459d5de40 0x55b459efb3a0 unknown :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:28:15.817 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:15.825+0000 7f1dc6ae6b80 1 --2- 172.21.15.52:0/2530078235 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55b459edbf80 0x55b459ef9180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:28:15.817 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:15.825+0000 7f1d957fa700 0 client.25691 ms_handle_reset on v2:172.21.15.105:6838/500348151 2024-05-27T07:28:15.817 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:15.825+0000 7f1dc6ae6b80 1 -- 172.21.15.52:0/2530078235 >> 172.21.15.52:0/2530078235 conn(0x55b459db8af0 msgr2=0x55b459ee39d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:28:15.817 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:15.825+0000 7f1dc6ae6b80 1 -- 172.21.15.52:0/2530078235 shutdown_connections 2024-05-27T07:28:15.817 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:15.825+0000 7f1dc6ae6b80 1 -- 172.21.15.52:0/2530078235 wait complete. 2024-05-27T07:28:15.817 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:15.826+0000 7f1dc6ae6b80 20 client.25691 trim_cache size 0 max 16384 2024-05-27T07:28:15.820 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:15.828+0000 7f1db80d1700 1 -- 172.21.15.52:0/3711045425 >> [v2:172.21.15.52:6800/1822318384,v1:172.21.15.52:6801/1822318384] conn(0x7f1d980b29f0 msgr2=0x7f1d980b4f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:28:15.820 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:15.828+0000 7f1db80d1700 1 --2- 172.21.15.52:0/3711045425 >> [v2:172.21.15.52:6800/1822318384,v1:172.21.15.52:6801/1822318384] conn(0x7f1d980b29f0 0x7f1d980b4f00 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f1db0101f30 tx=0x7f1da400a300 comp rx=0 tx=0).stop 2024-05-27T07:28:15.820 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:15.828+0000 7f1db80d1700 1 -- 172.21.15.52:0/3711045425 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f1db0102ad0 msgr2=0x7f1db019cfe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:28:15.820 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:15.828+0000 7f1db80d1700 1 --2- 172.21.15.52:0/3711045425 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f1db0102ad0 0x7f1db019cfe0 secure :-1 s=READY pgs=569 cs=0 l=1 rev1=1 crypto rx=0x7f1da000ef10 tx=0x7f1da000c3a0 comp rx=0 tx=0).stop 2024-05-27T07:28:15.821 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:15.832+0000 7f1db80d1700 1 -- 172.21.15.52:0/3711045425 shutdown_connections 2024-05-27T07:28:15.822 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:15.832+0000 7f1db80d1700 1 --2- 172.21.15.52:0/3711045425 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f1db0109840 0x7f1db01a1810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:28:15.822 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:15.832+0000 7f1db80d1700 1 --2- 172.21.15.52:0/3711045425 >> [v2:172.21.15.52:6800/1822318384,v1:172.21.15.52:6801/1822318384] conn(0x7f1d980b29f0 0x7f1d980b4f00 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:28:15.822 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:15.832+0000 7f1db80d1700 1 --2- 172.21.15.52:0/3711045425 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f1db0102ad0 0x7f1db019cfe0 unknown :-1 s=CLOSED pgs=569 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:28:15.822 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:15.832+0000 7f1db80d1700 1 --2- 172.21.15.52:0/3711045425 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f1db0108ef0 0x7f1db019d580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:28:15.822 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:15.832+0000 7f1db80d1700 1 -- 172.21.15.52:0/3711045425 >> 172.21.15.52:0/3711045425 conn(0x7f1db00fe4d0 msgr2=0x7f1db0106d50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:28:15.822 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:15.833+0000 7f1db80d1700 1 -- 172.21.15.52:0/3711045425 shutdown_connections 2024-05-27T07:28:15.822 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:15.833+0000 7f1db80d1700 1 -- 172.21.15.52:0/3711045425 wait complete. 2024-05-27T07:28:16.479 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:28:16 smithi052 ceph-mon[31955]: pgmap v72: 129 pgs: 129 active+clean; 62 MiB data, 461 MiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 1.6 MiB/s wr, 146 op/s 2024-05-27T07:28:16.479 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:28:16 smithi052 ceph-mon[31955]: daemon mds.f restarted 2024-05-27T07:28:16.479 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:28:16 smithi052 ceph-mon[31955]: osdmap e92: 12 total, 12 up, 12 in 2024-05-27T07:28:16.591 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:28:16 smithi105 ceph-mon[35977]: pgmap v72: 129 pgs: 129 active+clean; 62 MiB data, 461 MiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 1.6 MiB/s wr, 146 op/s 2024-05-27T07:28:16.591 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:28:16 smithi105 ceph-mon[35977]: daemon mds.f restarted 2024-05-27T07:28:16.591 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:28:16 smithi105 ceph-mon[35977]: osdmap e92: 12 total, 12 up, 12 in 2024-05-27T07:28:16.601 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:28:16 smithi173 ceph-mon[37000]: pgmap v72: 129 pgs: 129 active+clean; 62 MiB data, 461 MiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 1.6 MiB/s wr, 146 op/s 2024-05-27T07:28:16.601 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:28:16 smithi173 ceph-mon[37000]: daemon mds.f restarted 2024-05-27T07:28:16.602 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:28:16 smithi173 ceph-mon[37000]: osdmap e92: 12 total, 12 up, 12 in 2024-05-27T07:28:16.831 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-05-27T07:28:16.832 DEBUG:teuthology.orchestra.run.smithi052:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:38f9d900c83c63799fbdbe61acc9a11b0d3554a6 shell --fsid 73df2636-1bf8-11ef-bc9b-c7b262605968 -- ceph fs dump --format=json 2024-05-27T07:28:17.389 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:28:17.399+0000 7f8b4bfff640 1 -- 172.21.15.52:0/1219419566 <== mds.0 v2:172.21.15.105:6838/500348151 1 ==== command_reply(tid 0: 0 ) ==== 8+0+25 (crc 0 0 0) 0x55c5656d1360 con 0x55c5653561c0 2024-05-27T07:28:17.389 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:28:17.399+0000 7f8b4bfff640 10 client.15990 handle_command_reply: tid=0 2024-05-27T07:28:17.389 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:28:17.400+0000 7f8b79a41740 1 client.15990 shutdown 2024-05-27T07:28:17.392 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:28:17.403+0000 7f8b79a41740 1 -- 172.21.15.52:0/1219419566 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55c5653c21b0 msgr2=0x55c5657dc970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:28:17.392 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:28:17.403+0000 7f8b79a41740 1 --2- 172.21.15.52:0/1219419566 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55c5653c21b0 0x55c5657dc970 secure :-1 s=READY pgs=567 cs=0 l=1 rev1=1 crypto rx=0x7f8b64018040 tx=0x7f8b64009e40 comp rx=0 tx=0).stop 2024-05-27T07:28:17.393 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:28:17.403+0000 7f8b79a41740 1 -- 172.21.15.52:0/1219419566 shutdown_connections 2024-05-27T07:28:17.393 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:28:17.403+0000 7f8b79a41740 1 --2- 172.21.15.52:0/1219419566 >> [v2:172.21.15.105:6838/500348151,v1:172.21.15.105:6839/500348151] conn(0x55c5653561c0 0x55c5657eaaf0 crc :-1 s=READY pgs=78 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:28:17.393 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:28:17.403+0000 7f8b79a41740 1 --2- 172.21.15.52:0/1219419566 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55c5653c21b0 0x55c5657dc970 unknown :-1 s=CLOSED pgs=567 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:28:17.393 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:28:17.403+0000 7f8b79a41740 1 --2- 172.21.15.52:0/1219419566 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55c5653a14d0 0x55c5657dc430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:28:17.393 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:28:17.403+0000 7f8b79a41740 1 --2- 172.21.15.52:0/1219419566 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55c5653992f0 0x55c5657dbef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:28:17.394 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:28:17.403+0000 7f8b79a41740 1 -- 172.21.15.52:0/1219419566 >> 172.21.15.52:0/1219419566 conn(0x55c5653b6300 msgr2=0x55c565763250 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:28:17.394 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:28:17.404+0000 7f8b4bfff640 0 client.15990 ms_handle_reset on v2:172.21.15.105:6838/500348151 2024-05-27T07:28:17.395 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:28:17.405+0000 7f8b79a41740 1 -- 172.21.15.52:0/1219419566 shutdown_connections 2024-05-27T07:28:17.396 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:28:17.406+0000 7f8b79a41740 1 -- 172.21.15.52:0/1219419566 wait complete. 2024-05-27T07:28:17.397 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:28:17.408+0000 7f8b79a41740 20 client.15990 trim_cache size 0 max 16384 2024-05-27T07:28:17.402 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stdout:{ 2024-05-27T07:28:17.402 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stdout: "return_code": 0 2024-05-27T07:28:17.402 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stdout:} 2024-05-27T07:28:17.404 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:28:17.415+0000 7f8b72b86640 1 -- 172.21.15.52:0/2575527635 >> [v2:172.21.15.52:6800/1822318384,v1:172.21.15.52:6801/1822318384] conn(0x7f8b5c0b29d0 msgr2=0x7f8b5c0b4ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:28:17.405 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:28:17.415+0000 7f8b72b86640 1 --2- 172.21.15.52:0/2575527635 >> [v2:172.21.15.52:6800/1822318384,v1:172.21.15.52:6801/1822318384] conn(0x7f8b5c0b29d0 0x7f8b5c0b4ee0 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x55c565760540 tx=0x7f8b64009500 comp rx=0 tx=0).stop 2024-05-27T07:28:17.405 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:28:17.415+0000 7f8b72b86640 1 -- 172.21.15.52:0/2575527635 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f8b6c109cd0 msgr2=0x7f8b6c075500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:28:17.405 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:28:17.415+0000 7f8b72b86640 1 --2- 172.21.15.52:0/2575527635 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f8b6c109cd0 0x7f8b6c075500 secure :-1 s=READY pgs=564 cs=0 l=1 rev1=1 crypto rx=0x7f8b58005fd0 tx=0x7f8b5800dde0 comp rx=0 tx=0).stop 2024-05-27T07:28:17.407 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:28:17.416+0000 7f8b72b86640 1 -- 172.21.15.52:0/2575527635 shutdown_connections 2024-05-27T07:28:17.407 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:28:17.416+0000 7f8b72b86640 1 --2- 172.21.15.52:0/2575527635 >> [v2:172.21.15.52:6800/1822318384,v1:172.21.15.52:6801/1822318384] conn(0x7f8b5c0b29d0 0x7f8b5c0b4ee0 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:28:17.407 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:28:17.416+0000 7f8b72b86640 1 --2- 172.21.15.52:0/2575527635 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f8b6c10a170 0x7f8b6c075a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:28:17.407 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:28:17.416+0000 7f8b72b86640 1 --2- 172.21.15.52:0/2575527635 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f8b6c109cd0 0x7f8b6c075500 unknown :-1 s=CLOSED pgs=564 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:28:17.407 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:28:17.416+0000 7f8b72b86640 1 --2- 172.21.15.52:0/2575527635 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f8b6c104420 0x7f8b6c074fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:28:17.407 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:28:17.416+0000 7f8b72b86640 1 -- 172.21.15.52:0/2575527635 >> 172.21.15.52:0/2575527635 conn(0x7f8b6c079440 msgr2=0x7f8b6c105370 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:28:17.407 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:28:17.417+0000 7f8b72b86640 1 -- 172.21.15.52:0/2575527635 shutdown_connections 2024-05-27T07:28:17.407 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:28:17.417+0000 7f8b72b86640 1 -- 172.21.15.52:0/2575527635 wait complete. 2024-05-27T07:28:17.475 INFO:teuthology.orchestra.run.smithi052.stderr:Inferring config /var/lib/ceph/73df2636-1bf8-11ef-bc9b-c7b262605968/mon.a/config 2024-05-27T07:28:17.483 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:28:17 smithi052 ceph-mon[31955]: mds.? [v2:172.21.15.173:6834/3771862219,v1:172.21.15.173:6835/3771862219] up:boot 2024-05-27T07:28:17.483 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:28:17 smithi052 ceph-mon[31955]: fsmap cephfs:5 4 up:standby-replay 3 up:standby 5 up:active 2024-05-27T07:28:17.483 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:28:17 smithi052 ceph-mon[31955]: from='mgr.15333 172.21.15.52:0/416389082' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2024-05-27T07:28:17.483 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:28:17 smithi052 ceph-mon[31955]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-05-27T07:28:17.601 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:28:17 smithi173 ceph-mon[37000]: mds.? [v2:172.21.15.173:6834/3771862219,v1:172.21.15.173:6835/3771862219] up:boot 2024-05-27T07:28:17.601 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:28:17 smithi173 ceph-mon[37000]: fsmap cephfs:5 4 up:standby-replay 3 up:standby 5 up:active 2024-05-27T07:28:17.602 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:28:17 smithi173 ceph-mon[37000]: from='mgr.15333 172.21.15.52:0/416389082' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2024-05-27T07:28:17.602 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:28:17 smithi173 ceph-mon[37000]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-05-27T07:28:17.602 INFO:journalctl@ceph.mds.f.smithi173.stdout:May 27 07:28:17 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-f[72409]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: In function 'std::set SnapClient::filter(const std::set&) const' thread 7fd64fa64700 time 2024-05-27T07:28:17.266373+0000 2024-05-27T07:28:17.602 INFO:journalctl@ceph.mds.f.smithi173.stdout:May 27 07:28:17 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-f[72409]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: 197: FAILED ceph_assert(cached_version > 0) 2024-05-27T07:28:17.602 INFO:journalctl@ceph.mds.f.smithi173.stdout:May 27 07:28:17 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-f[72409]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:28:17.602 INFO:journalctl@ceph.mds.f.smithi173.stdout:May 27 07:28:17 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-f[72409]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x127) [0x7fd6622a9ba6] 2024-05-27T07:28:17.602 INFO:journalctl@ceph.mds.f.smithi173.stdout:May 27 07:28:17 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-f[72409]: 2: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7fd6622a9dd1] 2024-05-27T07:28:17.603 INFO:journalctl@ceph.mds.f.smithi173.stdout:May 27 07:28:17 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-f[72409]: 3: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x558752f2e23d] 2024-05-27T07:28:17.603 INFO:journalctl@ceph.mds.f.smithi173.stdout:May 27 07:28:17 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-f[72409]: 4: (SnapRealm::build_snap_set() const+0x323) [0x558752f19971] 2024-05-27T07:28:17.603 INFO:journalctl@ceph.mds.f.smithi173.stdout:May 27 07:28:17 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-f[72409]: 5: (SnapRealm::check_cache() const+0x13e) [0x558752f17fa0] 2024-05-27T07:28:17.603 INFO:journalctl@ceph.mds.f.smithi173.stdout:May 27 07:28:17 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-f[72409]: 6: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x558752f18651] 2024-05-27T07:28:17.603 INFO:journalctl@ceph.mds.f.smithi173.stdout:May 27 07:28:17 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-f[72409]: 7: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x558752ead347] 2024-05-27T07:28:17.603 INFO:journalctl@ceph.mds.f.smithi173.stdout:May 27 07:28:17 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-f[72409]: 8: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x558752ba987e] 2024-05-27T07:28:17.603 INFO:journalctl@ceph.mds.f.smithi173.stdout:May 27 07:28:17 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-f[72409]: 9: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x2702) [0x558752baee94] 2024-05-27T07:28:17.603 INFO:journalctl@ceph.mds.f.smithi173.stdout:May 27 07:28:17 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-f[72409]: 10: (EUpdate::replay(MDSRank*)+0x63) [0x558752bb9969] 2024-05-27T07:28:17.603 INFO:journalctl@ceph.mds.f.smithi173.stdout:May 27 07:28:17 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-f[72409]: 11: (MDLog::_replay_thread()+0x1f38) [0x558752f5248c] 2024-05-27T07:28:17.603 INFO:journalctl@ceph.mds.f.smithi173.stdout:May 27 07:28:17 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-f[72409]: 12: (MDLog::ReplayThread::entry()+0x11) [0x558752f5b9cb] 2024-05-27T07:28:17.603 INFO:journalctl@ceph.mds.f.smithi173.stdout:May 27 07:28:17 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-f[72409]: 13: (Thread::entry_wrapper()+0x43) [0x7fd66227872d] 2024-05-27T07:28:17.603 INFO:journalctl@ceph.mds.f.smithi173.stdout:May 27 07:28:17 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-f[72409]: 14: (Thread::_entry_func(void*)+0xd) [0x7fd662278749] 2024-05-27T07:28:17.604 INFO:journalctl@ceph.mds.f.smithi173.stdout:May 27 07:28:17 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-f[72409]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7fd660e1c1ca] 2024-05-27T07:28:17.604 INFO:journalctl@ceph.mds.f.smithi173.stdout:May 27 07:28:17 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-f[72409]: 16: clone() 2024-05-27T07:28:17.604 INFO:journalctl@ceph.mds.f.smithi173.stdout:May 27 07:28:17 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-f[72409]: 2024-05-27T07:28:17.269+0000 7fd64fa64700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: In function 'std::set SnapClient::filter(const std::set&) const' thread 7fd64fa64700 time 2024-05-27T07:28:17.266373+0000 2024-05-27T07:28:17.604 INFO:journalctl@ceph.mds.f.smithi173.stdout:May 27 07:28:17 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-f[72409]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: 197: FAILED ceph_assert(cached_version > 0) 2024-05-27T07:28:17.604 INFO:journalctl@ceph.mds.f.smithi173.stdout:May 27 07:28:17 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-f[72409]: 2024-05-27T07:28:17.604 INFO:journalctl@ceph.mds.f.smithi173.stdout:May 27 07:28:17 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-f[72409]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:28:17.604 INFO:journalctl@ceph.mds.f.smithi173.stdout:May 27 07:28:17 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-f[72409]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x127) [0x7fd6622a9ba6] 2024-05-27T07:28:17.604 INFO:journalctl@ceph.mds.f.smithi173.stdout:May 27 07:28:17 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-f[72409]: 2: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7fd6622a9dd1] 2024-05-27T07:28:17.604 INFO:journalctl@ceph.mds.f.smithi173.stdout:May 27 07:28:17 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-f[72409]: 3: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x558752f2e23d] 2024-05-27T07:28:17.604 INFO:journalctl@ceph.mds.f.smithi173.stdout:May 27 07:28:17 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-f[72409]: 4: (SnapRealm::build_snap_set() const+0x323) [0x558752f19971] 2024-05-27T07:28:17.605 INFO:journalctl@ceph.mds.f.smithi173.stdout:May 27 07:28:17 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-f[72409]: 5: (SnapRealm::check_cache() const+0x13e) [0x558752f17fa0] 2024-05-27T07:28:17.605 INFO:journalctl@ceph.mds.f.smithi173.stdout:May 27 07:28:17 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-f[72409]: 6: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x558752f18651] 2024-05-27T07:28:17.605 INFO:journalctl@ceph.mds.f.smithi173.stdout:May 27 07:28:17 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-f[72409]: 7: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x558752ead347] 2024-05-27T07:28:17.605 INFO:journalctl@ceph.mds.f.smithi173.stdout:May 27 07:28:17 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-f[72409]: 8: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x558752ba987e] 2024-05-27T07:28:17.605 INFO:journalctl@ceph.mds.f.smithi173.stdout:May 27 07:28:17 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-f[72409]: 9: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x2702) [0x558752baee94] 2024-05-27T07:28:17.605 INFO:journalctl@ceph.mds.f.smithi173.stdout:May 27 07:28:17 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-f[72409]: 10: (EUpdate::replay(MDSRank*)+0x63) [0x558752bb9969] 2024-05-27T07:28:17.605 INFO:journalctl@ceph.mds.f.smithi173.stdout:May 27 07:28:17 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-f[72409]: 11: (MDLog::_replay_thread()+0x1f38) [0x558752f5248c] 2024-05-27T07:28:17.605 INFO:journalctl@ceph.mds.f.smithi173.stdout:May 27 07:28:17 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-f[72409]: 12: (MDLog::ReplayThread::entry()+0x11) [0x558752f5b9cb] 2024-05-27T07:28:17.605 INFO:journalctl@ceph.mds.f.smithi173.stdout:May 27 07:28:17 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-f[72409]: 13: (Thread::entry_wrapper()+0x43) [0x7fd66227872d] 2024-05-27T07:28:17.605 INFO:journalctl@ceph.mds.f.smithi173.stdout:May 27 07:28:17 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-f[72409]: 14: (Thread::_entry_func(void*)+0xd) [0x7fd662278749] 2024-05-27T07:28:17.606 INFO:journalctl@ceph.mds.f.smithi173.stdout:May 27 07:28:17 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-f[72409]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7fd660e1c1ca] 2024-05-27T07:28:17.606 INFO:journalctl@ceph.mds.f.smithi173.stdout:May 27 07:28:17 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-f[72409]: 16: clone() 2024-05-27T07:28:17.606 INFO:journalctl@ceph.mds.f.smithi173.stdout:May 27 07:28:17 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-f[72409]: 2024-05-27T07:28:17.606 INFO:journalctl@ceph.mds.f.smithi173.stdout:May 27 07:28:17 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-f[72409]: *** Caught signal (Aborted) ** 2024-05-27T07:28:17.606 INFO:journalctl@ceph.mds.f.smithi173.stdout:May 27 07:28:17 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-f[72409]: in thread 7fd64fa64700 thread_name:md_log_replay 2024-05-27T07:28:17.606 INFO:journalctl@ceph.mds.f.smithi173.stdout:May 27 07:28:17 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-f[72409]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:28:17.606 INFO:journalctl@ceph.mds.f.smithi173.stdout:May 27 07:28:17 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-f[72409]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7fd660e26d20] 2024-05-27T07:28:17.607 INFO:journalctl@ceph.mds.f.smithi173.stdout:May 27 07:28:17 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-f[72409]: 2: gsignal() 2024-05-27T07:28:17.607 INFO:journalctl@ceph.mds.f.smithi173.stdout:May 27 07:28:17 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-f[72409]: 3: abort() 2024-05-27T07:28:17.607 INFO:journalctl@ceph.mds.f.smithi173.stdout:May 27 07:28:17 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-f[72409]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x26e) [0x7fd6622a9ced] 2024-05-27T07:28:17.607 INFO:journalctl@ceph.mds.f.smithi173.stdout:May 27 07:28:17 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-f[72409]: 5: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7fd6622a9dd1] 2024-05-27T07:28:17.608 INFO:journalctl@ceph.mds.f.smithi173.stdout:May 27 07:28:17 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-f[72409]: 6: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x558752f2e23d] 2024-05-27T07:28:17.608 INFO:journalctl@ceph.mds.f.smithi173.stdout:May 27 07:28:17 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-f[72409]: 7: (SnapRealm::build_snap_set() const+0x323) [0x558752f19971] 2024-05-27T07:28:17.608 INFO:journalctl@ceph.mds.f.smithi173.stdout:May 27 07:28:17 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-f[72409]: 8: (SnapRealm::check_cache() const+0x13e) [0x558752f17fa0] 2024-05-27T07:28:17.608 INFO:journalctl@ceph.mds.f.smithi173.stdout:May 27 07:28:17 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-f[72409]: 9: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x558752f18651] 2024-05-27T07:28:17.608 INFO:journalctl@ceph.mds.f.smithi173.stdout:May 27 07:28:17 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-f[72409]: 10: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x558752ead347] 2024-05-27T07:28:17.608 INFO:journalctl@ceph.mds.f.smithi173.stdout:May 27 07:28:17 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-f[72409]: 11: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x558752ba987e] 2024-05-27T07:28:17.608 INFO:journalctl@ceph.mds.f.smithi173.stdout:May 27 07:28:17 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-f[72409]: 12: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x2702) [0x558752baee94] 2024-05-27T07:28:17.608 INFO:journalctl@ceph.mds.f.smithi173.stdout:May 27 07:28:17 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-f[72409]: 13: (EUpdate::replay(MDSRank*)+0x63) [0x558752bb9969] 2024-05-27T07:28:17.608 INFO:journalctl@ceph.mds.f.smithi173.stdout:May 27 07:28:17 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-f[72409]: 14: (MDLog::_replay_thread()+0x1f38) [0x558752f5248c] 2024-05-27T07:28:17.608 INFO:journalctl@ceph.mds.f.smithi173.stdout:May 27 07:28:17 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-f[72409]: 15: (MDLog::ReplayThread::entry()+0x11) [0x558752f5b9cb] 2024-05-27T07:28:17.608 INFO:journalctl@ceph.mds.f.smithi173.stdout:May 27 07:28:17 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-f[72409]: 16: (Thread::entry_wrapper()+0x43) [0x7fd66227872d] 2024-05-27T07:28:17.609 INFO:journalctl@ceph.mds.f.smithi173.stdout:May 27 07:28:17 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-f[72409]: 17: (Thread::_entry_func(void*)+0xd) [0x7fd662278749] 2024-05-27T07:28:17.609 INFO:journalctl@ceph.mds.f.smithi173.stdout:May 27 07:28:17 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-f[72409]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7fd660e1c1ca] 2024-05-27T07:28:17.609 INFO:journalctl@ceph.mds.f.smithi173.stdout:May 27 07:28:17 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-f[72409]: 19: clone() 2024-05-27T07:28:17.609 INFO:journalctl@ceph.mds.f.smithi173.stdout:May 27 07:28:17 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-f[72409]: 2024-05-27T07:28:17.272+0000 7fd64fa64700 -1 *** Caught signal (Aborted) ** 2024-05-27T07:28:17.609 INFO:journalctl@ceph.mds.f.smithi173.stdout:May 27 07:28:17 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-f[72409]: in thread 7fd64fa64700 thread_name:md_log_replay 2024-05-27T07:28:17.609 INFO:journalctl@ceph.mds.f.smithi173.stdout:May 27 07:28:17 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-f[72409]: 2024-05-27T07:28:17.609 INFO:journalctl@ceph.mds.f.smithi173.stdout:May 27 07:28:17 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-f[72409]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:28:17.609 INFO:journalctl@ceph.mds.f.smithi173.stdout:May 27 07:28:17 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-f[72409]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7fd660e26d20] 2024-05-27T07:28:17.609 INFO:journalctl@ceph.mds.f.smithi173.stdout:May 27 07:28:17 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-f[72409]: 2: gsignal() 2024-05-27T07:28:17.609 INFO:journalctl@ceph.mds.f.smithi173.stdout:May 27 07:28:17 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-f[72409]: 3: abort() 2024-05-27T07:28:17.609 INFO:journalctl@ceph.mds.f.smithi173.stdout:May 27 07:28:17 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-f[72409]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x26e) [0x7fd6622a9ced] 2024-05-27T07:28:17.609 INFO:journalctl@ceph.mds.f.smithi173.stdout:May 27 07:28:17 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-f[72409]: 5: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7fd6622a9dd1] 2024-05-27T07:28:17.609 INFO:journalctl@ceph.mds.f.smithi173.stdout:May 27 07:28:17 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-f[72409]: 6: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x558752f2e23d] 2024-05-27T07:28:17.610 INFO:journalctl@ceph.mds.f.smithi173.stdout:May 27 07:28:17 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-f[72409]: 7: (SnapRealm::build_snap_set() const+0x323) [0x558752f19971] 2024-05-27T07:28:17.610 INFO:journalctl@ceph.mds.f.smithi173.stdout:May 27 07:28:17 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-f[72409]: 8: (SnapRealm::check_cache() const+0x13e) [0x558752f17fa0] 2024-05-27T07:28:17.610 INFO:journalctl@ceph.mds.f.smithi173.stdout:May 27 07:28:17 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-f[72409]: 9: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x558752f18651] 2024-05-27T07:28:17.610 INFO:journalctl@ceph.mds.f.smithi173.stdout:May 27 07:28:17 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-f[72409]: 10: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x558752ead347] 2024-05-27T07:28:17.610 INFO:journalctl@ceph.mds.f.smithi173.stdout:May 27 07:28:17 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-f[72409]: 11: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x558752ba987e] 2024-05-27T07:28:17.610 INFO:journalctl@ceph.mds.f.smithi173.stdout:May 27 07:28:17 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-f[72409]: 12: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x2702) [0x558752baee94] 2024-05-27T07:28:17.610 INFO:journalctl@ceph.mds.f.smithi173.stdout:May 27 07:28:17 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-f[72409]: 13: (EUpdate::replay(MDSRank*)+0x63) [0x558752bb9969] 2024-05-27T07:28:17.610 INFO:journalctl@ceph.mds.f.smithi173.stdout:May 27 07:28:17 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-f[72409]: 14: (MDLog::_replay_thread()+0x1f38) [0x558752f5248c] 2024-05-27T07:28:17.610 INFO:journalctl@ceph.mds.f.smithi173.stdout:May 27 07:28:17 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-f[72409]: 15: (MDLog::ReplayThread::entry()+0x11) [0x558752f5b9cb] 2024-05-27T07:28:17.610 INFO:journalctl@ceph.mds.f.smithi173.stdout:May 27 07:28:17 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-f[72409]: 16: (Thread::entry_wrapper()+0x43) [0x7fd66227872d] 2024-05-27T07:28:17.610 INFO:journalctl@ceph.mds.f.smithi173.stdout:May 27 07:28:17 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-f[72409]: 17: (Thread::_entry_func(void*)+0xd) [0x7fd662278749] 2024-05-27T07:28:17.610 INFO:journalctl@ceph.mds.f.smithi173.stdout:May 27 07:28:17 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-f[72409]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7fd660e1c1ca] 2024-05-27T07:28:17.610 INFO:journalctl@ceph.mds.f.smithi173.stdout:May 27 07:28:17 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-f[72409]: 19: clone() 2024-05-27T07:28:17.611 INFO:journalctl@ceph.mds.f.smithi173.stdout:May 27 07:28:17 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-f[72409]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-05-27T07:28:17.611 INFO:journalctl@ceph.mds.f.smithi173.stdout:May 27 07:28:17 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-f[72409]: 2024-05-27T07:28:17.611 INFO:journalctl@ceph.mds.f.smithi173.stdout:May 27 07:28:17 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-f[72409]: -1> 2024-05-27T07:28:17.269+0000 7fd64fa64700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: In function 'std::set SnapClient::filter(const std::set&) const' thread 7fd64fa64700 time 2024-05-27T07:28:17.266373+0000 2024-05-27T07:28:17.611 INFO:journalctl@ceph.mds.f.smithi173.stdout:May 27 07:28:17 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-f[72409]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: 197: FAILED ceph_assert(cached_version > 0) 2024-05-27T07:28:17.611 INFO:journalctl@ceph.mds.f.smithi173.stdout:May 27 07:28:17 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-f[72409]: 2024-05-27T07:28:17.611 INFO:journalctl@ceph.mds.f.smithi173.stdout:May 27 07:28:17 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-f[72409]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:28:17.611 INFO:journalctl@ceph.mds.f.smithi173.stdout:May 27 07:28:17 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-f[72409]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x127) [0x7fd6622a9ba6] 2024-05-27T07:28:17.611 INFO:journalctl@ceph.mds.f.smithi173.stdout:May 27 07:28:17 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-f[72409]: 2: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7fd6622a9dd1] 2024-05-27T07:28:17.611 INFO:journalctl@ceph.mds.f.smithi173.stdout:May 27 07:28:17 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-f[72409]: 3: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x558752f2e23d] 2024-05-27T07:28:17.611 INFO:journalctl@ceph.mds.f.smithi173.stdout:May 27 07:28:17 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-f[72409]: 4: (SnapRealm::build_snap_set() const+0x323) [0x558752f19971] 2024-05-27T07:28:17.611 INFO:journalctl@ceph.mds.f.smithi173.stdout:May 27 07:28:17 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-f[72409]: 5: (SnapRealm::check_cache() const+0x13e) [0x558752f17fa0] 2024-05-27T07:28:17.611 INFO:journalctl@ceph.mds.f.smithi173.stdout:May 27 07:28:17 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-f[72409]: 6: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x558752f18651] 2024-05-27T07:28:17.611 INFO:journalctl@ceph.mds.f.smithi173.stdout:May 27 07:28:17 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-f[72409]: 7: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x558752ead347] 2024-05-27T07:28:17.612 INFO:journalctl@ceph.mds.f.smithi173.stdout:May 27 07:28:17 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-f[72409]: 8: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x558752ba987e] 2024-05-27T07:28:17.612 INFO:journalctl@ceph.mds.f.smithi173.stdout:May 27 07:28:17 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-f[72409]: 9: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x2702) [0x558752baee94] 2024-05-27T07:28:17.612 INFO:journalctl@ceph.mds.f.smithi173.stdout:May 27 07:28:17 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-f[72409]: 10: (EUpdate::replay(MDSRank*)+0x63) [0x558752bb9969] 2024-05-27T07:28:17.612 INFO:journalctl@ceph.mds.f.smithi173.stdout:May 27 07:28:17 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-f[72409]: 11: (MDLog::_replay_thread()+0x1f38) [0x558752f5248c] 2024-05-27T07:28:17.612 INFO:journalctl@ceph.mds.f.smithi173.stdout:May 27 07:28:17 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-f[72409]: 12: (MDLog::ReplayThread::entry()+0x11) [0x558752f5b9cb] 2024-05-27T07:28:17.612 INFO:journalctl@ceph.mds.f.smithi173.stdout:May 27 07:28:17 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-f[72409]: 13: (Thread::entry_wrapper()+0x43) [0x7fd66227872d] 2024-05-27T07:28:17.612 INFO:journalctl@ceph.mds.f.smithi173.stdout:May 27 07:28:17 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-f[72409]: 14: (Thread::_entry_func(void*)+0xd) [0x7fd662278749] 2024-05-27T07:28:17.612 INFO:journalctl@ceph.mds.f.smithi173.stdout:May 27 07:28:17 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-f[72409]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7fd660e1c1ca] 2024-05-27T07:28:17.613 INFO:journalctl@ceph.mds.f.smithi173.stdout:May 27 07:28:17 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-f[72409]: 16: clone() 2024-05-27T07:28:17.613 INFO:journalctl@ceph.mds.f.smithi173.stdout:May 27 07:28:17 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-f[72409]: 2024-05-27T07:28:17.613 INFO:journalctl@ceph.mds.f.smithi173.stdout:May 27 07:28:17 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-f[72409]: 0> 2024-05-27T07:28:17.272+0000 7fd64fa64700 -1 *** Caught signal (Aborted) ** 2024-05-27T07:28:17.613 INFO:journalctl@ceph.mds.f.smithi173.stdout:May 27 07:28:17 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-f[72409]: in thread 7fd64fa64700 thread_name:md_log_replay 2024-05-27T07:28:17.613 INFO:journalctl@ceph.mds.f.smithi173.stdout:May 27 07:28:17 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-f[72409]: 2024-05-27T07:28:17.613 INFO:journalctl@ceph.mds.f.smithi173.stdout:May 27 07:28:17 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-f[72409]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:28:17.613 INFO:journalctl@ceph.mds.f.smithi173.stdout:May 27 07:28:17 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-f[72409]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7fd660e26d20] 2024-05-27T07:28:17.613 INFO:journalctl@ceph.mds.f.smithi173.stdout:May 27 07:28:17 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-f[72409]: 2: gsignal() 2024-05-27T07:28:17.613 INFO:journalctl@ceph.mds.f.smithi173.stdout:May 27 07:28:17 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-f[72409]: 3: abort() 2024-05-27T07:28:17.613 INFO:journalctl@ceph.mds.f.smithi173.stdout:May 27 07:28:17 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-f[72409]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x26e) [0x7fd6622a9ced] 2024-05-27T07:28:17.613 INFO:journalctl@ceph.mds.f.smithi173.stdout:May 27 07:28:17 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-f[72409]: 5: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7fd6622a9dd1] 2024-05-27T07:28:17.614 INFO:journalctl@ceph.mds.f.smithi173.stdout:May 27 07:28:17 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-f[72409]: 6: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x558752f2e23d] 2024-05-27T07:28:17.614 INFO:journalctl@ceph.mds.f.smithi173.stdout:May 27 07:28:17 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-f[72409]: 7: (SnapRealm::build_snap_set() const+0x323) [0x558752f19971] 2024-05-27T07:28:17.614 INFO:journalctl@ceph.mds.f.smithi173.stdout:May 27 07:28:17 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-f[72409]: 8: (SnapRealm::check_cache() const+0x13e) [0x558752f17fa0] 2024-05-27T07:28:17.614 INFO:journalctl@ceph.mds.f.smithi173.stdout:May 27 07:28:17 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-f[72409]: 9: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x558752f18651] 2024-05-27T07:28:17.614 INFO:journalctl@ceph.mds.f.smithi173.stdout:May 27 07:28:17 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-f[72409]: 10: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x558752ead347] 2024-05-27T07:28:17.614 INFO:journalctl@ceph.mds.f.smithi173.stdout:May 27 07:28:17 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-f[72409]: 11: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x558752ba987e] 2024-05-27T07:28:17.614 INFO:journalctl@ceph.mds.f.smithi173.stdout:May 27 07:28:17 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-f[72409]: 12: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x2702) [0x558752baee94] 2024-05-27T07:28:17.614 INFO:journalctl@ceph.mds.f.smithi173.stdout:May 27 07:28:17 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-f[72409]: 13: (EUpdate::replay(MDSRank*)+0x63) [0x558752bb9969] 2024-05-27T07:28:17.614 INFO:journalctl@ceph.mds.f.smithi173.stdout:May 27 07:28:17 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-f[72409]: 14: (MDLog::_replay_thread()+0x1f38) [0x558752f5248c] 2024-05-27T07:28:17.614 INFO:journalctl@ceph.mds.f.smithi173.stdout:May 27 07:28:17 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-f[72409]: 15: (MDLog::ReplayThread::entry()+0x11) [0x558752f5b9cb] 2024-05-27T07:28:17.614 INFO:journalctl@ceph.mds.f.smithi173.stdout:May 27 07:28:17 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-f[72409]: 16: (Thread::entry_wrapper()+0x43) [0x7fd66227872d] 2024-05-27T07:28:17.614 INFO:journalctl@ceph.mds.f.smithi173.stdout:May 27 07:28:17 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-f[72409]: 17: (Thread::_entry_func(void*)+0xd) [0x7fd662278749] 2024-05-27T07:28:17.614 INFO:journalctl@ceph.mds.f.smithi173.stdout:May 27 07:28:17 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-f[72409]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7fd660e1c1ca] 2024-05-27T07:28:17.615 INFO:journalctl@ceph.mds.f.smithi173.stdout:May 27 07:28:17 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-f[72409]: 19: clone() 2024-05-27T07:28:17.615 INFO:journalctl@ceph.mds.f.smithi173.stdout:May 27 07:28:17 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-f[72409]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-05-27T07:28:17.615 INFO:journalctl@ceph.mds.f.smithi173.stdout:May 27 07:28:17 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-f[72409]: 2024-05-27T07:28:17.615 INFO:journalctl@ceph.mds.f.smithi173.stdout:May 27 07:28:17 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-f[72409]: -1> 2024-05-27T07:28:17.269+0000 7fd64fa64700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: In function 'std::set SnapClient::filter(const std::set&) const' thread 7fd64fa64700 time 2024-05-27T07:28:17.266373+0000 2024-05-27T07:28:17.615 INFO:journalctl@ceph.mds.f.smithi173.stdout:May 27 07:28:17 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-f[72409]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: 197: FAILED ceph_assert(cached_version > 0) 2024-05-27T07:28:17.615 INFO:journalctl@ceph.mds.f.smithi173.stdout:May 27 07:28:17 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-f[72409]: 2024-05-27T07:28:17.615 INFO:journalctl@ceph.mds.f.smithi173.stdout:May 27 07:28:17 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-f[72409]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:28:17.615 INFO:journalctl@ceph.mds.f.smithi173.stdout:May 27 07:28:17 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-f[72409]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x127) [0x7fd6622a9ba6] 2024-05-27T07:28:17.615 INFO:journalctl@ceph.mds.f.smithi173.stdout:May 27 07:28:17 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-f[72409]: 2: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7fd6622a9dd1] 2024-05-27T07:28:17.615 INFO:journalctl@ceph.mds.f.smithi173.stdout:May 27 07:28:17 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-f[72409]: 3: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x558752f2e23d] 2024-05-27T07:28:17.615 INFO:journalctl@ceph.mds.f.smithi173.stdout:May 27 07:28:17 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-f[72409]: 4: (SnapRealm::build_snap_set() const+0x323) [0x558752f19971] 2024-05-27T07:28:17.615 INFO:journalctl@ceph.mds.f.smithi173.stdout:May 27 07:28:17 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-f[72409]: 5: (SnapRealm::check_cache() const+0x13e) [0x558752f17fa0] 2024-05-27T07:28:17.615 INFO:journalctl@ceph.mds.f.smithi173.stdout:May 27 07:28:17 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-f[72409]: 6: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x558752f18651] 2024-05-27T07:28:17.615 INFO:journalctl@ceph.mds.f.smithi173.stdout:May 27 07:28:17 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-f[72409]: 7: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x558752ead347] 2024-05-27T07:28:17.616 INFO:journalctl@ceph.mds.f.smithi173.stdout:May 27 07:28:17 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-f[72409]: 8: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x558752ba987e] 2024-05-27T07:28:17.616 INFO:journalctl@ceph.mds.f.smithi173.stdout:May 27 07:28:17 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-f[72409]: 9: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x2702) [0x558752baee94] 2024-05-27T07:28:17.616 INFO:journalctl@ceph.mds.f.smithi173.stdout:May 27 07:28:17 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-f[72409]: 10: (EUpdate::replay(MDSRank*)+0x63) [0x558752bb9969] 2024-05-27T07:28:17.616 INFO:journalctl@ceph.mds.f.smithi173.stdout:May 27 07:28:17 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-f[72409]: 11: (MDLog::_replay_thread()+0x1f38) [0x558752f5248c] 2024-05-27T07:28:17.616 INFO:journalctl@ceph.mds.f.smithi173.stdout:May 27 07:28:17 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-f[72409]: 12: (MDLog::ReplayThread::entry()+0x11) [0x558752f5b9cb] 2024-05-27T07:28:17.616 INFO:journalctl@ceph.mds.f.smithi173.stdout:May 27 07:28:17 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-f[72409]: 13: (Thread::entry_wrapper()+0x43) [0x7fd66227872d] 2024-05-27T07:28:17.616 INFO:journalctl@ceph.mds.f.smithi173.stdout:May 27 07:28:17 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-f[72409]: 14: (Thread::_entry_func(void*)+0xd) [0x7fd662278749] 2024-05-27T07:28:17.616 INFO:journalctl@ceph.mds.f.smithi173.stdout:May 27 07:28:17 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-f[72409]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7fd660e1c1ca] 2024-05-27T07:28:17.616 INFO:journalctl@ceph.mds.f.smithi173.stdout:May 27 07:28:17 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-f[72409]: 16: clone() 2024-05-27T07:28:17.617 INFO:journalctl@ceph.mds.f.smithi173.stdout:May 27 07:28:17 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-f[72409]: 2024-05-27T07:28:17.617 INFO:journalctl@ceph.mds.f.smithi173.stdout:May 27 07:28:17 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-f[72409]: 0> 2024-05-27T07:28:17.272+0000 7fd64fa64700 -1 *** Caught signal (Aborted) ** 2024-05-27T07:28:17.617 INFO:journalctl@ceph.mds.f.smithi173.stdout:May 27 07:28:17 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-f[72409]: in thread 7fd64fa64700 thread_name:md_log_replay 2024-05-27T07:28:17.617 INFO:journalctl@ceph.mds.f.smithi173.stdout:May 27 07:28:17 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-f[72409]: 2024-05-27T07:28:17.617 INFO:journalctl@ceph.mds.f.smithi173.stdout:May 27 07:28:17 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-f[72409]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:28:17.617 INFO:journalctl@ceph.mds.f.smithi173.stdout:May 27 07:28:17 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-f[72409]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7fd660e26d20] 2024-05-27T07:28:17.617 INFO:journalctl@ceph.mds.f.smithi173.stdout:May 27 07:28:17 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-f[72409]: 2: gsignal() 2024-05-27T07:28:17.617 INFO:journalctl@ceph.mds.f.smithi173.stdout:May 27 07:28:17 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-f[72409]: 3: abort() 2024-05-27T07:28:17.617 INFO:journalctl@ceph.mds.f.smithi173.stdout:May 27 07:28:17 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-f[72409]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x26e) [0x7fd6622a9ced] 2024-05-27T07:28:17.618 INFO:journalctl@ceph.mds.f.smithi173.stdout:May 27 07:28:17 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-f[72409]: 5: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7fd6622a9dd1] 2024-05-27T07:28:17.618 INFO:journalctl@ceph.mds.f.smithi173.stdout:May 27 07:28:17 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-f[72409]: 6: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x558752f2e23d] 2024-05-27T07:28:17.618 INFO:journalctl@ceph.mds.f.smithi173.stdout:May 27 07:28:17 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-f[72409]: 7: (SnapRealm::build_snap_set() const+0x323) [0x558752f19971] 2024-05-27T07:28:17.618 INFO:journalctl@ceph.mds.f.smithi173.stdout:May 27 07:28:17 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-f[72409]: 8: (SnapRealm::check_cache() const+0x13e) [0x558752f17fa0] 2024-05-27T07:28:17.618 INFO:journalctl@ceph.mds.f.smithi173.stdout:May 27 07:28:17 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-f[72409]: 9: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x558752f18651] 2024-05-27T07:28:17.618 INFO:journalctl@ceph.mds.f.smithi173.stdout:May 27 07:28:17 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-f[72409]: 10: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x558752ead347] 2024-05-27T07:28:17.618 INFO:journalctl@ceph.mds.f.smithi173.stdout:May 27 07:28:17 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-f[72409]: 11: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x558752ba987e] 2024-05-27T07:28:17.618 INFO:journalctl@ceph.mds.f.smithi173.stdout:May 27 07:28:17 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-f[72409]: 12: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x2702) [0x558752baee94] 2024-05-27T07:28:17.618 INFO:journalctl@ceph.mds.f.smithi173.stdout:May 27 07:28:17 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-f[72409]: 13: (EUpdate::replay(MDSRank*)+0x63) [0x558752bb9969] 2024-05-27T07:28:17.618 INFO:journalctl@ceph.mds.f.smithi173.stdout:May 27 07:28:17 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-f[72409]: 14: (MDLog::_replay_thread()+0x1f38) [0x558752f5248c] 2024-05-27T07:28:17.618 INFO:journalctl@ceph.mds.f.smithi173.stdout:May 27 07:28:17 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-f[72409]: 15: (MDLog::ReplayThread::entry()+0x11) [0x558752f5b9cb] 2024-05-27T07:28:17.618 INFO:journalctl@ceph.mds.f.smithi173.stdout:May 27 07:28:17 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-f[72409]: 16: (Thread::entry_wrapper()+0x43) [0x7fd66227872d] 2024-05-27T07:28:17.619 INFO:journalctl@ceph.mds.f.smithi173.stdout:May 27 07:28:17 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-f[72409]: 17: (Thread::_entry_func(void*)+0xd) [0x7fd662278749] 2024-05-27T07:28:17.619 INFO:journalctl@ceph.mds.f.smithi173.stdout:May 27 07:28:17 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-f[72409]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7fd660e1c1ca] 2024-05-27T07:28:17.619 INFO:journalctl@ceph.mds.f.smithi173.stdout:May 27 07:28:17 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-f[72409]: 19: clone() 2024-05-27T07:28:17.619 INFO:journalctl@ceph.mds.f.smithi173.stdout:May 27 07:28:17 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-f[72409]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-05-27T07:28:17.619 INFO:journalctl@ceph.mds.f.smithi173.stdout:May 27 07:28:17 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-f[72409]: 2024-05-27T07:28:17.619 INFO:journalctl@ceph.mds.f.smithi173.stdout:May 27 07:28:17 smithi173 podman[72456]: 2024-05-27 07:28:17.387864278 +0000 UTC m=+0.042437048 container died 11d3b53c90f5ccc17a6b74d217f5c944d8341059a5ed3156713b380c4df8a0df (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-f, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, GIT_CLEAN=True, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, org.label-schema.build-date=20240521, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux ) 2024-05-27T07:28:17.689 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:28:17 smithi105 ceph-mon[35977]: mds.? [v2:172.21.15.173:6834/3771862219,v1:172.21.15.173:6835/3771862219] up:boot 2024-05-27T07:28:17.690 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:28:17 smithi105 ceph-mon[35977]: fsmap cephfs:5 4 up:standby-replay 3 up:standby 5 up:active 2024-05-27T07:28:17.690 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:28:17 smithi105 ceph-mon[35977]: from='mgr.15333 172.21.15.52:0/416389082' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2024-05-27T07:28:17.690 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:28:17 smithi105 ceph-mon[35977]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-05-27T07:28:17.782 INFO:journalctl@ceph.mds.g.smithi052.stdout:May 27 07:28:17 smithi052 systemd[1]: ceph-73df2636-1bf8-11ef-bc9b-c7b262605968@mds.g.service: Scheduled restart job, restart counter is at 1. 2024-05-27T07:28:17.783 INFO:journalctl@ceph.mds.g.smithi052.stdout:May 27 07:28:17 smithi052 systemd[1]: Stopped Ceph mds.g for 73df2636-1bf8-11ef-bc9b-c7b262605968. 2024-05-27T07:28:17.783 INFO:journalctl@ceph.mds.g.smithi052.stdout:May 27 07:28:17 smithi052 systemd[1]: ceph-73df2636-1bf8-11ef-bc9b-c7b262605968@mds.g.service: Consumed 1.179s CPU time. 2024-05-27T07:28:17.783 INFO:journalctl@ceph.mds.g.smithi052.stdout:May 27 07:28:17 smithi052 systemd[1]: Starting Ceph mds.g for 73df2636-1bf8-11ef-bc9b-c7b262605968... 2024-05-27T07:28:18.351 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:28:18 smithi173 ceph-mon[37000]: pgmap v74: 129 pgs: 129 active+clean; 62 MiB data, 461 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 1.9 MiB/s wr, 176 op/s 2024-05-27T07:28:18.352 INFO:journalctl@ceph.mds.f.smithi173.stdout:May 27 07:28:18 smithi173 podman[72456]: 2024-05-27 07:28:18.043172495 +0000 UTC m=+0.697745238 container remove 11d3b53c90f5ccc17a6b74d217f5c944d8341059a5ed3156713b380c4df8a0df (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-f, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, org.label-schema.build-date=20240521, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0) 2024-05-27T07:28:18.352 INFO:journalctl@ceph.mds.f.smithi173.stdout:May 27 07:28:18 smithi173 systemd[1]: ceph-73df2636-1bf8-11ef-bc9b-c7b262605968@mds.f.service: Main process exited, code=exited, status=134/n/a 2024-05-27T07:28:18.653 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:28:18 smithi052 ceph-mon[31955]: pgmap v74: 129 pgs: 129 active+clean; 62 MiB data, 461 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 1.9 MiB/s wr, 176 op/s 2024-05-27T07:28:18.689 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:28:18 smithi105 ceph-mon[35977]: pgmap v74: 129 pgs: 129 active+clean; 62 MiB data, 461 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 1.9 MiB/s wr, 176 op/s 2024-05-27T07:28:18.851 INFO:journalctl@ceph.mds.f.smithi173.stdout:May 27 07:28:18 smithi173 systemd[1]: ceph-73df2636-1bf8-11ef-bc9b-c7b262605968@mds.f.service: Failed with result 'exit-code'. 2024-05-27T07:28:18.851 INFO:journalctl@ceph.mds.f.smithi173.stdout:May 27 07:28:18 smithi173 systemd[1]: ceph-73df2636-1bf8-11ef-bc9b-c7b262605968@mds.f.service: Consumed 1.411s CPU time. 2024-05-27T07:28:19.229 INFO:journalctl@ceph.mds.g.smithi052.stdout:May 27 07:28:18 smithi052 podman[102329]: 2024-05-27 07:28:18.883923454 +0000 UTC m=+0.022414522 image pull e36ea4fc8a5807e8d2701370e350a0de3856538797f7fba64cfce7e344bd6799 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6 2024-05-27T07:28:19.230 INFO:journalctl@ceph.mds.g.smithi052.stdout:May 27 07:28:19 smithi052 podman[102329]: 2024-05-27 07:28:19.164502046 +0000 UTC m=+0.302993107 container create ae945548727bd0c8e6eaa92ba1276da2a69dd2e3d7470a92e8aa51ef157a7d4e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-g, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240521, io.buildah.version=1.33.5, GIT_CLEAN=True, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, GIT_BRANCH=HEAD, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, org.label-schema.license=GPLv2) 2024-05-27T07:28:19.513 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:28:19 smithi052 ceph-mon[31955]: pgmap v75: 129 pgs: 129 active+clean; 71 MiB data, 515 MiB used, 1.0 TiB / 1.0 TiB avail; 5.3 MiB/s rd, 5.1 MiB/s wr, 1.33k op/s 2024-05-27T07:28:19.547 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:19.555+0000 7f2b47245700 1 -- 172.21.15.52:0/2355407268 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f2b401092f0 msgr2=0x7f2b40109700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:28:19.547 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:19.555+0000 7f2b47245700 1 --2- 172.21.15.52:0/2355407268 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f2b401092f0 0x7f2b40109700 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f2b34009990 tx=0x7f2b34009ca0 comp rx=0 tx=0).stop 2024-05-27T07:28:19.547 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:19.555+0000 7f2b47245700 1 -- 172.21.15.52:0/2355407268 shutdown_connections 2024-05-27T07:28:19.547 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:19.555+0000 7f2b47245700 1 --2- 172.21.15.52:0/2355407268 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f2b401031c0 0x7f2b40111160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:28:19.547 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:19.555+0000 7f2b47245700 1 --2- 172.21.15.52:0/2355407268 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f2b401092f0 0x7f2b40109700 unknown :-1 s=CLOSED pgs=92 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:28:19.547 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:19.555+0000 7f2b47245700 1 --2- 172.21.15.52:0/2355407268 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f2b40102870 0x7f2b40102c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:28:19.547 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:19.555+0000 7f2b47245700 1 -- 172.21.15.52:0/2355407268 >> 172.21.15.52:0/2355407268 conn(0x7f2b400fe6e0 msgr2=0x7f2b40100b00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:28:19.547 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:19.555+0000 7f2b47245700 1 -- 172.21.15.52:0/2355407268 shutdown_connections 2024-05-27T07:28:19.548 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:19.555+0000 7f2b47245700 1 -- 172.21.15.52:0/2355407268 wait complete. 2024-05-27T07:28:19.548 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:19.556+0000 7f2b47245700 1 Processor -- start 2024-05-27T07:28:19.548 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:19.556+0000 7f2b47245700 1 -- start start 2024-05-27T07:28:19.548 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:19.556+0000 7f2b47245700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f2b40102870 0x7f2b4019af60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:28:19.548 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:19.556+0000 7f2b47245700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f2b401031c0 0x7f2b4019b500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:28:19.548 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:19.556+0000 7f2b47245700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f2b401092f0 0x7f2b4019f790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:28:19.548 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:19.556+0000 7f2b47245700 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x7f2b401145d0 con 0x7f2b401092f0 2024-05-27T07:28:19.548 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:19.556+0000 7f2b47245700 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f2b40114450 con 0x7f2b40102870 2024-05-27T07:28:19.548 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:19.556+0000 7f2b47245700 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f2b40114750 con 0x7f2b401031c0 2024-05-27T07:28:19.548 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:19.556+0000 7f2b3ffff700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f2b401031c0 0x7f2b4019b500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:28:19.548 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:19.556+0000 7f2b457e2700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f2b401092f0 0x7f2b4019f790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:28:19.548 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:19.556+0000 7f2b44fe1700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f2b40102870 0x7f2b4019af60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:28:19.548 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:19.556+0000 7f2b3ffff700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f2b401031c0 0x7f2b4019b500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.173:3300/0 says I am v2:172.21.15.52:54664/0 (socket says 172.21.15.52:54664) 2024-05-27T07:28:19.549 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:19.556+0000 7f2b3ffff700 1 -- 172.21.15.52:0/2274589921 learned_addr learned my addr 172.21.15.52:0/2274589921 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-05-27T07:28:19.549 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:19.556+0000 7f2b44fe1700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f2b40102870 0x7f2b4019af60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.52:46878/0 (socket says 172.21.15.52:46878) 2024-05-27T07:28:19.549 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:19.557+0000 7f2b3ffff700 1 -- 172.21.15.52:0/2274589921 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f2b40102870 msgr2=0x7f2b4019af60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:28:19.549 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:19.557+0000 7f2b3ffff700 1 --2- 172.21.15.52:0/2274589921 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f2b40102870 0x7f2b4019af60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:28:19.549 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:19.557+0000 7f2b3ffff700 1 -- 172.21.15.52:0/2274589921 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f2b401092f0 msgr2=0x7f2b4019f790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:28:19.549 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:19.557+0000 7f2b3ffff700 1 --2- 172.21.15.52:0/2274589921 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f2b401092f0 0x7f2b4019f790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:28:19.549 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:19.557+0000 7f2b3ffff700 1 -- 172.21.15.52:0/2274589921 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2b4019fe30 con 0x7f2b401031c0 2024-05-27T07:28:19.549 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:19.557+0000 7f2b44fe1700 1 --2- 172.21.15.52:0/2274589921 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f2b40102870 0x7f2b4019af60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-27T07:28:19.549 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:19.557+0000 7f2b3ffff700 1 --2- 172.21.15.52:0/2274589921 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f2b401031c0 0x7f2b4019b500 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f2b2c00e970 tx=0x7f2b2c00ec80 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:28:19.549 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:19.557+0000 7f2b3dffb700 1 -- 172.21.15.52:0/2274589921 <== mon.1 v2:172.21.15.173:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2b2c004a30 con 0x7f2b401031c0 2024-05-27T07:28:19.549 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:19.557+0000 7f2b47245700 1 -- 172.21.15.52:0/2274589921 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2b401a0120 con 0x7f2b401031c0 2024-05-27T07:28:19.549 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:19.557+0000 7f2b3dffb700 1 -- 172.21.15.52:0/2274589921 <== mon.1 v2:172.21.15.173:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f2b2c00cc90 con 0x7f2b401031c0 2024-05-27T07:28:19.550 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:19.558+0000 7f2b3dffb700 1 -- 172.21.15.52:0/2274589921 <== mon.1 v2:172.21.15.173:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2b2c005610 con 0x7f2b401031c0 2024-05-27T07:28:19.550 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:19.558+0000 7f2b47245700 1 -- 172.21.15.52:0/2274589921 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2b401a7a40 con 0x7f2b401031c0 2024-05-27T07:28:19.551 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:19.559+0000 7f2b47245700 1 -- 172.21.15.52:0/2274589921 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2b400512b0 con 0x7f2b401031c0 2024-05-27T07:28:19.551 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:19.560+0000 7f2b3dffb700 1 -- 172.21.15.52:0/2274589921 <== mon.1 v2:172.21.15.173:3300/0 4 ==== mgrmap(e 26) ==== 149783+0+0 (secure 0 0 0) 0x7f2b2c00c7e0 con 0x7f2b401031c0 2024-05-27T07:28:19.551 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:19.561+0000 7f2b3dffb700 1 --2- 172.21.15.52:0/2274589921 >> [v2:172.21.15.52:6800/1822318384,v1:172.21.15.52:6801/1822318384] conn(0x7f2b300b29f0 0x7f2b300b4f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:28:19.551 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:19.561+0000 7f2b44fe1700 1 --2- 172.21.15.52:0/2274589921 >> [v2:172.21.15.52:6800/1822318384,v1:172.21.15.52:6801/1822318384] conn(0x7f2b300b29f0 0x7f2b300b4f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:28:19.551 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:19.561+0000 7f2b3dffb700 1 -- 172.21.15.52:0/2274589921 <== mon.1 v2:172.21.15.173:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 9379+0+0 (secure 0 0 0) 0x7f2b2c0e2cc0 con 0x7f2b401031c0 2024-05-27T07:28:19.551 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:19.561+0000 7f2b44fe1700 1 --2- 172.21.15.52:0/2274589921 >> [v2:172.21.15.52:6800/1822318384,v1:172.21.15.52:6801/1822318384] conn(0x7f2b300b29f0 0x7f2b300b4f00 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f2b3400b2e0 tx=0x7f2b34005fd0 comp rx=0 tx=0).ready entity=mgr.15333 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:28:19.554 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:19.564+0000 7f2b3dffb700 1 -- 172.21.15.52:0/2274589921 <== mon.1 v2:172.21.15.173:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191963 (secure 0 0 0) 0x7f2b2c0aa340 con 0x7f2b401031c0 2024-05-27T07:28:19.601 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:28:19 smithi173 ceph-mon[37000]: pgmap v75: 129 pgs: 129 active+clean; 71 MiB data, 515 MiB used, 1.0 TiB / 1.0 TiB avail; 5.3 MiB/s rd, 5.1 MiB/s wr, 1.33k op/s 2024-05-27T07:28:19.689 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:28:19 smithi105 ceph-mon[35977]: pgmap v75: 129 pgs: 129 active+clean; 71 MiB data, 515 MiB used, 1.0 TiB / 1.0 TiB avail; 5.3 MiB/s rd, 5.1 MiB/s wr, 1.33k op/s 2024-05-27T07:28:19.840 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:19.850+0000 7f2b47245700 1 -- 172.21.15.52:0/2274589921 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f2b4010c890 con 0x7f2b401031c0 2024-05-27T07:28:19.841 INFO:journalctl@ceph.mds.g.smithi052.stdout:May 27 07:28:19 smithi052 podman[102329]: 2024-05-27 07:28:19.636208766 +0000 UTC m=+0.774699827 container init ae945548727bd0c8e6eaa92ba1276da2a69dd2e3d7470a92e8aa51ef157a7d4e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-g, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240521, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, ceph=True, GIT_CLEAN=True, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90) 2024-05-27T07:28:19.841 INFO:journalctl@ceph.mds.g.smithi052.stdout:May 27 07:28:19 smithi052 podman[102329]: 2024-05-27 07:28:19.639447282 +0000 UTC m=+0.777938347 container start ae945548727bd0c8e6eaa92ba1276da2a69dd2e3d7470a92e8aa51ef157a7d4e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-g, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240521, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, GIT_CLEAN=True) 2024-05-27T07:28:19.841 INFO:journalctl@ceph.mds.g.smithi052.stdout:May 27 07:28:19 smithi052 ceph-mds[102378]: -- 172.21.15.52:0/188180194 <== mon.0 v2:172.21.15.52:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x555d8ef9da40 con 0x555d8efe5180 2024-05-27T07:28:19.841 INFO:journalctl@ceph.mds.g.smithi052.stdout:May 27 07:28:19 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-g[102352]: starting mds.g at 2024-05-27T07:28:19.841 INFO:journalctl@ceph.mds.g.smithi052.stdout:May 27 07:28:19 smithi052 bash[102329]: ae945548727bd0c8e6eaa92ba1276da2a69dd2e3d7470a92e8aa51ef157a7d4e 2024-05-27T07:28:19.841 INFO:journalctl@ceph.mds.g.smithi052.stdout:May 27 07:28:19 smithi052 systemd[1]: Started Ceph mds.g for 73df2636-1bf8-11ef-bc9b-c7b262605968. 2024-05-27T07:28:19.841 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:19.851+0000 7f2b3dffb700 1 -- 172.21.15.52:0/2274589921 <== mon.1 v2:172.21.15.173:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 35 v35) ==== 94+0+11943 (secure 0 0 0) 0x7f2b2c0a9c40 con 0x7f2b401031c0 2024-05-27T07:28:19.842 INFO:teuthology.orchestra.run.smithi052.stdout: 2024-05-27T07:28:19.842 INFO:teuthology.orchestra.run.smithi052.stdout:{"epoch":35,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14556,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.52:6835/2584733778","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.52:6834","nonce":2584733778},{"type":"v1","addr":"172.21.15.52:6835","nonce":2584733778}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14568,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.52:6837/675445303","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.52:6836","nonce":675445303},{"type":"v1","addr":"172.21.15.52:6837","nonce":675445303}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":35,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-05-27T07:21:50.521783+0000","modified":"2024-05-27T07:28:16.232252+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24527,"mds_1":24497,"mds_2":24475,"mds_3":24473,"mds_4":24466},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14619":{"gid":14619,"name":"g","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.52:6839/4258549559","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.52:6838","nonce":4258549559},{"type":"v1","addr":"172.21.15.52:6839","nonce":4258549559}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_14631":{"gid":14631,"name":"a","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.52:6841/2878618188","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.52:6840","nonce":2878618188},{"type":"v1","addr":"172.21.15.52:6841","nonce":2878618188}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24433":{"gid":24433,"name":"l","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.173:6833/3013589220","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6832","nonce":3013589220},{"type":"v1","addr":"172.21.15.173:6833","nonce":3013589220}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24452":{"gid":24452,"name":"k","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.105:6833/2800678412","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6832","nonce":2800678412},{"type":"v1","addr":"172.21.15.105:6833","nonce":2800678412}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24466":{"gid":24466,"name":"c","rank":4,"incarnation":26,"state":"up:active","state_seq":25,"addr":"172.21.15.173:6837/1396363827","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6836","nonce":1396363827},{"type":"v1","addr":"172.21.15.173:6837","nonce":1396363827}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24473":{"gid":24473,"name":"e","rank":3,"incarnation":24,"state":"up:active","state_seq":26,"addr":"172.21.15.105:6835/3298498384","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6834","nonce":3298498384},{"type":"v1","addr":"172.21.15.105:6835","nonce":3298498384}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24475":{"gid":24475,"name":"i","rank":2,"incarnation":22,"state":"up:active","state_seq":23,"addr":"172.21.15.173:6839/564778293","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6838","nonce":564778293},{"type":"v1","addr":"172.21.15.173:6839","nonce":564778293}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24497":{"gid":24497,"name":"h","rank":1,"incarnation":20,"state":"up:active","state_seq":24,"addr":"172.21.15.105:6837/191393149","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6836","nonce":191393149},{"type":"v1","addr":"172.21.15.105:6837","nonce":191393149}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24527":{"gid":24527,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":5,"addr":"172.21.15.105:6839/500348151","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6838","nonce":500348151},{"type":"v1","addr":"172.21.15.105:6839","nonce":500348151}]},"join_fscid":-1,"export_targets":[1,3,4],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_25588":{"gid":25588,"name":"f","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.173:6835/3771862219","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6834","nonce":3771862219},{"type":"v1","addr":"172.21.15.173:6835","nonce":3771862219}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24527,"qdb_cluster":[24473,24466,24475,24527,24497]},"id":1}]} 2024-05-27T07:28:19.844 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:19.854+0000 7f2b47245700 1 -- 172.21.15.52:0/2274589921 >> [v2:172.21.15.52:6800/1822318384,v1:172.21.15.52:6801/1822318384] conn(0x7f2b300b29f0 msgr2=0x7f2b300b4f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:28:19.844 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:19.854+0000 7f2b47245700 1 --2- 172.21.15.52:0/2274589921 >> [v2:172.21.15.52:6800/1822318384,v1:172.21.15.52:6801/1822318384] conn(0x7f2b300b29f0 0x7f2b300b4f00 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f2b3400b2e0 tx=0x7f2b34005fd0 comp rx=0 tx=0).stop 2024-05-27T07:28:19.844 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:19.854+0000 7f2b47245700 1 -- 172.21.15.52:0/2274589921 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f2b401031c0 msgr2=0x7f2b4019b500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:28:19.844 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:19.854+0000 7f2b47245700 1 --2- 172.21.15.52:0/2274589921 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f2b401031c0 0x7f2b4019b500 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f2b2c00e970 tx=0x7f2b2c00ec80 comp rx=0 tx=0).stop 2024-05-27T07:28:19.844 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:19.855+0000 7f2b47245700 1 -- 172.21.15.52:0/2274589921 shutdown_connections 2024-05-27T07:28:19.844 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:19.855+0000 7f2b47245700 1 --2- 172.21.15.52:0/2274589921 >> [v2:172.21.15.52:6800/1822318384,v1:172.21.15.52:6801/1822318384] conn(0x7f2b300b29f0 0x7f2b300b4f00 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:28:19.845 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:19.855+0000 7f2b47245700 1 --2- 172.21.15.52:0/2274589921 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f2b401092f0 0x7f2b4019f790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:28:19.845 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:19.855+0000 7f2b47245700 1 --2- 172.21.15.52:0/2274589921 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f2b40102870 0x7f2b4019af60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:28:19.845 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:19.855+0000 7f2b47245700 1 --2- 172.21.15.52:0/2274589921 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f2b401031c0 0x7f2b4019b500 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:28:19.845 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:19.855+0000 7f2b47245700 1 -- 172.21.15.52:0/2274589921 >> 172.21.15.52:0/2274589921 conn(0x7f2b400fe6e0 msgr2=0x7f2b4010a780 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:28:19.845 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:19.855+0000 7f2b47245700 1 -- 172.21.15.52:0/2274589921 shutdown_connections 2024-05-27T07:28:19.845 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:19.855+0000 7f2b47245700 1 -- 172.21.15.52:0/2274589921 wait complete. 2024-05-27T07:28:19.846 INFO:teuthology.orchestra.run.smithi052.stderr:dumped fsmap epoch 35 2024-05-27T07:28:20.158 INFO:journalctl@ceph.mds.g.smithi052.stdout:May 27 07:28:20 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-g[102352]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: In function 'std::set SnapClient::filter(const std::set&) const' thread 7f38f313a700 time 2024-05-27T07:28:20.008485+0000 2024-05-27T07:28:20.158 INFO:journalctl@ceph.mds.g.smithi052.stdout:May 27 07:28:20 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-g[102352]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: 197: FAILED ceph_assert(cached_version > 0) 2024-05-27T07:28:20.158 INFO:journalctl@ceph.mds.g.smithi052.stdout:May 27 07:28:20 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-g[102352]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:28:20.158 INFO:journalctl@ceph.mds.g.smithi052.stdout:May 27 07:28:20 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-g[102352]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x127) [0x7f390597fba6] 2024-05-27T07:28:20.158 INFO:journalctl@ceph.mds.g.smithi052.stdout:May 27 07:28:20 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-g[102352]: 2: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7f390597fdd1] 2024-05-27T07:28:20.159 INFO:journalctl@ceph.mds.g.smithi052.stdout:May 27 07:28:20 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-g[102352]: 3: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x555d8d15c23d] 2024-05-27T07:28:20.159 INFO:journalctl@ceph.mds.g.smithi052.stdout:May 27 07:28:20 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-g[102352]: 4: (SnapRealm::build_snap_set() const+0x323) [0x555d8d147971] 2024-05-27T07:28:20.159 INFO:journalctl@ceph.mds.g.smithi052.stdout:May 27 07:28:20 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-g[102352]: 5: (SnapRealm::check_cache() const+0x13e) [0x555d8d145fa0] 2024-05-27T07:28:20.159 INFO:journalctl@ceph.mds.g.smithi052.stdout:May 27 07:28:20 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-g[102352]: 6: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x555d8d146651] 2024-05-27T07:28:20.159 INFO:journalctl@ceph.mds.g.smithi052.stdout:May 27 07:28:20 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-g[102352]: 7: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x555d8d0db347] 2024-05-27T07:28:20.159 INFO:journalctl@ceph.mds.g.smithi052.stdout:May 27 07:28:20 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-g[102352]: 8: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x555d8cdd787e] 2024-05-27T07:28:20.159 INFO:journalctl@ceph.mds.g.smithi052.stdout:May 27 07:28:20 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-g[102352]: 9: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x2702) [0x555d8cddce94] 2024-05-27T07:28:20.159 INFO:journalctl@ceph.mds.g.smithi052.stdout:May 27 07:28:20 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-g[102352]: 10: (EUpdate::replay(MDSRank*)+0x63) [0x555d8cde7969] 2024-05-27T07:28:20.159 INFO:journalctl@ceph.mds.g.smithi052.stdout:May 27 07:28:20 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-g[102352]: 11: (MDLog::_replay_thread()+0x1f38) [0x555d8d18048c] 2024-05-27T07:28:20.159 INFO:journalctl@ceph.mds.g.smithi052.stdout:May 27 07:28:20 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-g[102352]: 12: (MDLog::ReplayThread::entry()+0x11) [0x555d8d1899cb] 2024-05-27T07:28:20.159 INFO:journalctl@ceph.mds.g.smithi052.stdout:May 27 07:28:20 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-g[102352]: 13: (Thread::entry_wrapper()+0x43) [0x7f390594e72d] 2024-05-27T07:28:20.159 INFO:journalctl@ceph.mds.g.smithi052.stdout:May 27 07:28:20 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-g[102352]: 14: (Thread::_entry_func(void*)+0xd) [0x7f390594e749] 2024-05-27T07:28:20.159 INFO:journalctl@ceph.mds.g.smithi052.stdout:May 27 07:28:20 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-g[102352]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7f39044f21ca] 2024-05-27T07:28:20.159 INFO:journalctl@ceph.mds.g.smithi052.stdout:May 27 07:28:20 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-g[102352]: 16: clone() 2024-05-27T07:28:20.160 INFO:journalctl@ceph.mds.g.smithi052.stdout:May 27 07:28:20 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-g[102352]: *** Caught signal (Aborted) ** 2024-05-27T07:28:20.160 INFO:journalctl@ceph.mds.g.smithi052.stdout:May 27 07:28:20 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-g[102352]: in thread 7f38f313a700 thread_name:md_log_replay 2024-05-27T07:28:20.160 INFO:journalctl@ceph.mds.g.smithi052.stdout:May 27 07:28:20 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-g[102352]: 2024-05-27T07:28:20.012+0000 7f38f313a700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: In function 'std::set SnapClient::filter(const std::set&) const' thread 7f38f313a700 time 2024-05-27T07:28:20.008485+0000 2024-05-27T07:28:20.160 INFO:journalctl@ceph.mds.g.smithi052.stdout:May 27 07:28:20 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-g[102352]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: 197: FAILED ceph_assert(cached_version > 0) 2024-05-27T07:28:20.160 INFO:journalctl@ceph.mds.g.smithi052.stdout:May 27 07:28:20 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-g[102352]: 2024-05-27T07:28:20.160 INFO:journalctl@ceph.mds.g.smithi052.stdout:May 27 07:28:20 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-g[102352]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:28:20.161 INFO:journalctl@ceph.mds.g.smithi052.stdout:May 27 07:28:20 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-g[102352]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x127) [0x7f390597fba6] 2024-05-27T07:28:20.161 INFO:journalctl@ceph.mds.g.smithi052.stdout:May 27 07:28:20 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-g[102352]: 2: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7f390597fdd1] 2024-05-27T07:28:20.161 INFO:journalctl@ceph.mds.g.smithi052.stdout:May 27 07:28:20 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-g[102352]: 3: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x555d8d15c23d] 2024-05-27T07:28:20.161 INFO:journalctl@ceph.mds.g.smithi052.stdout:May 27 07:28:20 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-g[102352]: 4: (SnapRealm::build_snap_set() const+0x323) [0x555d8d147971] 2024-05-27T07:28:20.161 INFO:journalctl@ceph.mds.g.smithi052.stdout:May 27 07:28:20 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-g[102352]: 5: (SnapRealm::check_cache() const+0x13e) [0x555d8d145fa0] 2024-05-27T07:28:20.161 INFO:journalctl@ceph.mds.g.smithi052.stdout:May 27 07:28:20 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-g[102352]: 6: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x555d8d146651] 2024-05-27T07:28:20.161 INFO:journalctl@ceph.mds.g.smithi052.stdout:May 27 07:28:20 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-g[102352]: 7: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x555d8d0db347] 2024-05-27T07:28:20.161 INFO:journalctl@ceph.mds.g.smithi052.stdout:May 27 07:28:20 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-g[102352]: 8: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x555d8cdd787e] 2024-05-27T07:28:20.161 INFO:journalctl@ceph.mds.g.smithi052.stdout:May 27 07:28:20 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-g[102352]: 9: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x2702) [0x555d8cddce94] 2024-05-27T07:28:20.161 INFO:journalctl@ceph.mds.g.smithi052.stdout:May 27 07:28:20 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-g[102352]: 10: (EUpdate::replay(MDSRank*)+0x63) [0x555d8cde7969] 2024-05-27T07:28:20.161 INFO:journalctl@ceph.mds.g.smithi052.stdout:May 27 07:28:20 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-g[102352]: 11: (MDLog::_replay_thread()+0x1f38) [0x555d8d18048c] 2024-05-27T07:28:20.161 INFO:journalctl@ceph.mds.g.smithi052.stdout:May 27 07:28:20 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-g[102352]: 12: (MDLog::ReplayThread::entry()+0x11) [0x555d8d1899cb] 2024-05-27T07:28:20.162 INFO:journalctl@ceph.mds.g.smithi052.stdout:May 27 07:28:20 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-g[102352]: 13: (Thread::entry_wrapper()+0x43) [0x7f390594e72d] 2024-05-27T07:28:20.162 INFO:journalctl@ceph.mds.g.smithi052.stdout:May 27 07:28:20 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-g[102352]: 14: (Thread::_entry_func(void*)+0xd) [0x7f390594e749] 2024-05-27T07:28:20.162 INFO:journalctl@ceph.mds.g.smithi052.stdout:May 27 07:28:20 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-g[102352]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7f39044f21ca] 2024-05-27T07:28:20.162 INFO:journalctl@ceph.mds.g.smithi052.stdout:May 27 07:28:20 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-g[102352]: 16: clone() 2024-05-27T07:28:20.162 INFO:journalctl@ceph.mds.g.smithi052.stdout:May 27 07:28:20 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-g[102352]: 2024-05-27T07:28:20.162 INFO:journalctl@ceph.mds.g.smithi052.stdout:May 27 07:28:20 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-g[102352]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:28:20.162 INFO:journalctl@ceph.mds.g.smithi052.stdout:May 27 07:28:20 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-g[102352]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f39044fcd20] 2024-05-27T07:28:20.162 INFO:journalctl@ceph.mds.g.smithi052.stdout:May 27 07:28:20 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-g[102352]: 2: gsignal() 2024-05-27T07:28:20.163 INFO:journalctl@ceph.mds.g.smithi052.stdout:May 27 07:28:20 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-g[102352]: 3: abort() 2024-05-27T07:28:20.163 INFO:journalctl@ceph.mds.g.smithi052.stdout:May 27 07:28:20 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-g[102352]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x26e) [0x7f390597fced] 2024-05-27T07:28:20.163 INFO:journalctl@ceph.mds.g.smithi052.stdout:May 27 07:28:20 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-g[102352]: 5: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7f390597fdd1] 2024-05-27T07:28:20.163 INFO:journalctl@ceph.mds.g.smithi052.stdout:May 27 07:28:20 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-g[102352]: 6: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x555d8d15c23d] 2024-05-27T07:28:20.163 INFO:journalctl@ceph.mds.g.smithi052.stdout:May 27 07:28:20 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-g[102352]: 7: (SnapRealm::build_snap_set() const+0x323) [0x555d8d147971] 2024-05-27T07:28:20.163 INFO:journalctl@ceph.mds.g.smithi052.stdout:May 27 07:28:20 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-g[102352]: 8: (SnapRealm::check_cache() const+0x13e) [0x555d8d145fa0] 2024-05-27T07:28:20.164 INFO:journalctl@ceph.mds.g.smithi052.stdout:May 27 07:28:20 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-g[102352]: 9: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x555d8d146651] 2024-05-27T07:28:20.164 INFO:journalctl@ceph.mds.g.smithi052.stdout:May 27 07:28:20 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-g[102352]: 10: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x555d8d0db347] 2024-05-27T07:28:20.164 INFO:journalctl@ceph.mds.g.smithi052.stdout:May 27 07:28:20 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-g[102352]: 11: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x555d8cdd787e] 2024-05-27T07:28:20.164 INFO:journalctl@ceph.mds.g.smithi052.stdout:May 27 07:28:20 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-g[102352]: 12: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x2702) [0x555d8cddce94] 2024-05-27T07:28:20.164 INFO:journalctl@ceph.mds.g.smithi052.stdout:May 27 07:28:20 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-g[102352]: 13: (EUpdate::replay(MDSRank*)+0x63) [0x555d8cde7969] 2024-05-27T07:28:20.164 INFO:journalctl@ceph.mds.g.smithi052.stdout:May 27 07:28:20 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-g[102352]: 14: (MDLog::_replay_thread()+0x1f38) [0x555d8d18048c] 2024-05-27T07:28:20.165 INFO:journalctl@ceph.mds.g.smithi052.stdout:May 27 07:28:20 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-g[102352]: 15: (MDLog::ReplayThread::entry()+0x11) [0x555d8d1899cb] 2024-05-27T07:28:20.165 INFO:journalctl@ceph.mds.g.smithi052.stdout:May 27 07:28:20 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-g[102352]: 16: (Thread::entry_wrapper()+0x43) [0x7f390594e72d] 2024-05-27T07:28:20.165 INFO:journalctl@ceph.mds.g.smithi052.stdout:May 27 07:28:20 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-g[102352]: 17: (Thread::_entry_func(void*)+0xd) [0x7f390594e749] 2024-05-27T07:28:20.165 INFO:journalctl@ceph.mds.g.smithi052.stdout:May 27 07:28:20 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-g[102352]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7f39044f21ca] 2024-05-27T07:28:20.165 INFO:journalctl@ceph.mds.g.smithi052.stdout:May 27 07:28:20 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-g[102352]: 19: clone() 2024-05-27T07:28:20.165 INFO:journalctl@ceph.mds.g.smithi052.stdout:May 27 07:28:20 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-g[102352]: 2024-05-27T07:28:20.020+0000 7f38f313a700 -1 *** Caught signal (Aborted) ** 2024-05-27T07:28:20.166 INFO:journalctl@ceph.mds.g.smithi052.stdout:May 27 07:28:20 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-g[102352]: in thread 7f38f313a700 thread_name:md_log_replay 2024-05-27T07:28:20.166 INFO:journalctl@ceph.mds.g.smithi052.stdout:May 27 07:28:20 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-g[102352]: 2024-05-27T07:28:20.166 INFO:journalctl@ceph.mds.g.smithi052.stdout:May 27 07:28:20 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-g[102352]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:28:20.166 INFO:journalctl@ceph.mds.g.smithi052.stdout:May 27 07:28:20 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-g[102352]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f39044fcd20] 2024-05-27T07:28:20.166 INFO:journalctl@ceph.mds.g.smithi052.stdout:May 27 07:28:20 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-g[102352]: 2: gsignal() 2024-05-27T07:28:20.166 INFO:journalctl@ceph.mds.g.smithi052.stdout:May 27 07:28:20 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-g[102352]: 3: abort() 2024-05-27T07:28:20.167 INFO:journalctl@ceph.mds.g.smithi052.stdout:May 27 07:28:20 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-g[102352]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x26e) [0x7f390597fced] 2024-05-27T07:28:20.167 INFO:journalctl@ceph.mds.g.smithi052.stdout:May 27 07:28:20 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-g[102352]: 5: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7f390597fdd1] 2024-05-27T07:28:20.167 INFO:journalctl@ceph.mds.g.smithi052.stdout:May 27 07:28:20 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-g[102352]: 6: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x555d8d15c23d] 2024-05-27T07:28:20.167 INFO:journalctl@ceph.mds.g.smithi052.stdout:May 27 07:28:20 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-g[102352]: 7: (SnapRealm::build_snap_set() const+0x323) [0x555d8d147971] 2024-05-27T07:28:20.167 INFO:journalctl@ceph.mds.g.smithi052.stdout:May 27 07:28:20 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-g[102352]: 8: (SnapRealm::check_cache() const+0x13e) [0x555d8d145fa0] 2024-05-27T07:28:20.167 INFO:journalctl@ceph.mds.g.smithi052.stdout:May 27 07:28:20 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-g[102352]: 9: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x555d8d146651] 2024-05-27T07:28:20.169 INFO:journalctl@ceph.mds.g.smithi052.stdout:May 27 07:28:20 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-g[102352]: 10: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x555d8d0db347] 2024-05-27T07:28:20.169 INFO:journalctl@ceph.mds.g.smithi052.stdout:May 27 07:28:20 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-g[102352]: 11: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x555d8cdd787e] 2024-05-27T07:28:20.169 INFO:journalctl@ceph.mds.g.smithi052.stdout:May 27 07:28:20 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-g[102352]: 12: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x2702) [0x555d8cddce94] 2024-05-27T07:28:20.169 INFO:journalctl@ceph.mds.g.smithi052.stdout:May 27 07:28:20 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-g[102352]: 13: (EUpdate::replay(MDSRank*)+0x63) [0x555d8cde7969] 2024-05-27T07:28:20.169 INFO:journalctl@ceph.mds.g.smithi052.stdout:May 27 07:28:20 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-g[102352]: 14: (MDLog::_replay_thread()+0x1f38) [0x555d8d18048c] 2024-05-27T07:28:20.169 INFO:journalctl@ceph.mds.g.smithi052.stdout:May 27 07:28:20 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-g[102352]: 15: (MDLog::ReplayThread::entry()+0x11) [0x555d8d1899cb] 2024-05-27T07:28:20.170 INFO:journalctl@ceph.mds.g.smithi052.stdout:May 27 07:28:20 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-g[102352]: 16: (Thread::entry_wrapper()+0x43) [0x7f390594e72d] 2024-05-27T07:28:20.170 INFO:journalctl@ceph.mds.g.smithi052.stdout:May 27 07:28:20 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-g[102352]: 17: (Thread::_entry_func(void*)+0xd) [0x7f390594e749] 2024-05-27T07:28:20.170 INFO:journalctl@ceph.mds.g.smithi052.stdout:May 27 07:28:20 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-g[102352]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7f39044f21ca] 2024-05-27T07:28:20.170 INFO:journalctl@ceph.mds.g.smithi052.stdout:May 27 07:28:20 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-g[102352]: 19: clone() 2024-05-27T07:28:20.170 INFO:journalctl@ceph.mds.g.smithi052.stdout:May 27 07:28:20 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-g[102352]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-05-27T07:28:20.170 INFO:journalctl@ceph.mds.g.smithi052.stdout:May 27 07:28:20 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-g[102352]: 2024-05-27T07:28:20.170 INFO:journalctl@ceph.mds.g.smithi052.stdout:May 27 07:28:20 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-g[102352]: -1> 2024-05-27T07:28:20.012+0000 7f38f313a700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: In function 'std::set SnapClient::filter(const std::set&) const' thread 7f38f313a700 time 2024-05-27T07:28:20.008485+0000 2024-05-27T07:28:20.171 INFO:journalctl@ceph.mds.g.smithi052.stdout:May 27 07:28:20 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-g[102352]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: 197: FAILED ceph_assert(cached_version > 0) 2024-05-27T07:28:20.171 INFO:journalctl@ceph.mds.g.smithi052.stdout:May 27 07:28:20 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-g[102352]: 2024-05-27T07:28:20.171 INFO:journalctl@ceph.mds.g.smithi052.stdout:May 27 07:28:20 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-g[102352]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:28:20.171 INFO:journalctl@ceph.mds.g.smithi052.stdout:May 27 07:28:20 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-g[102352]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x127) [0x7f390597fba6] 2024-05-27T07:28:20.171 INFO:journalctl@ceph.mds.g.smithi052.stdout:May 27 07:28:20 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-g[102352]: 2: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7f390597fdd1] 2024-05-27T07:28:20.171 INFO:journalctl@ceph.mds.g.smithi052.stdout:May 27 07:28:20 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-g[102352]: 3: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x555d8d15c23d] 2024-05-27T07:28:20.171 INFO:journalctl@ceph.mds.g.smithi052.stdout:May 27 07:28:20 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-g[102352]: 4: (SnapRealm::build_snap_set() const+0x323) [0x555d8d147971] 2024-05-27T07:28:20.171 INFO:journalctl@ceph.mds.g.smithi052.stdout:May 27 07:28:20 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-g[102352]: 5: (SnapRealm::check_cache() const+0x13e) [0x555d8d145fa0] 2024-05-27T07:28:20.172 INFO:journalctl@ceph.mds.g.smithi052.stdout:May 27 07:28:20 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-g[102352]: 6: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x555d8d146651] 2024-05-27T07:28:20.172 INFO:journalctl@ceph.mds.g.smithi052.stdout:May 27 07:28:20 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-g[102352]: 7: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x555d8d0db347] 2024-05-27T07:28:20.172 INFO:journalctl@ceph.mds.g.smithi052.stdout:May 27 07:28:20 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-g[102352]: 8: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x555d8cdd787e] 2024-05-27T07:28:20.172 INFO:journalctl@ceph.mds.g.smithi052.stdout:May 27 07:28:20 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-g[102352]: 9: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x2702) [0x555d8cddce94] 2024-05-27T07:28:20.172 INFO:journalctl@ceph.mds.g.smithi052.stdout:May 27 07:28:20 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-g[102352]: 10: (EUpdate::replay(MDSRank*)+0x63) [0x555d8cde7969] 2024-05-27T07:28:20.172 INFO:journalctl@ceph.mds.g.smithi052.stdout:May 27 07:28:20 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-g[102352]: 11: (MDLog::_replay_thread()+0x1f38) [0x555d8d18048c] 2024-05-27T07:28:20.172 INFO:journalctl@ceph.mds.g.smithi052.stdout:May 27 07:28:20 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-g[102352]: 12: (MDLog::ReplayThread::entry()+0x11) [0x555d8d1899cb] 2024-05-27T07:28:20.172 INFO:journalctl@ceph.mds.g.smithi052.stdout:May 27 07:28:20 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-g[102352]: 13: (Thread::entry_wrapper()+0x43) [0x7f390594e72d] 2024-05-27T07:28:20.173 INFO:journalctl@ceph.mds.g.smithi052.stdout:May 27 07:28:20 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-g[102352]: 14: (Thread::_entry_func(void*)+0xd) [0x7f390594e749] 2024-05-27T07:28:20.173 INFO:journalctl@ceph.mds.g.smithi052.stdout:May 27 07:28:20 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-g[102352]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7f39044f21ca] 2024-05-27T07:28:20.173 INFO:journalctl@ceph.mds.g.smithi052.stdout:May 27 07:28:20 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-g[102352]: 16: clone() 2024-05-27T07:28:20.173 INFO:journalctl@ceph.mds.g.smithi052.stdout:May 27 07:28:20 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-g[102352]: 2024-05-27T07:28:20.173 INFO:journalctl@ceph.mds.g.smithi052.stdout:May 27 07:28:20 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-g[102352]: 0> 2024-05-27T07:28:20.020+0000 7f38f313a700 -1 *** Caught signal (Aborted) ** 2024-05-27T07:28:20.173 INFO:journalctl@ceph.mds.g.smithi052.stdout:May 27 07:28:20 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-g[102352]: in thread 7f38f313a700 thread_name:md_log_replay 2024-05-27T07:28:20.174 INFO:journalctl@ceph.mds.g.smithi052.stdout:May 27 07:28:20 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-g[102352]: 2024-05-27T07:28:20.174 INFO:journalctl@ceph.mds.g.smithi052.stdout:May 27 07:28:20 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-g[102352]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:28:20.174 INFO:journalctl@ceph.mds.g.smithi052.stdout:May 27 07:28:20 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-g[102352]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f39044fcd20] 2024-05-27T07:28:20.174 INFO:journalctl@ceph.mds.g.smithi052.stdout:May 27 07:28:20 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-g[102352]: 2: gsignal() 2024-05-27T07:28:20.174 INFO:journalctl@ceph.mds.g.smithi052.stdout:May 27 07:28:20 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-g[102352]: 3: abort() 2024-05-27T07:28:20.174 INFO:journalctl@ceph.mds.g.smithi052.stdout:May 27 07:28:20 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-g[102352]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x26e) [0x7f390597fced] 2024-05-27T07:28:20.175 INFO:journalctl@ceph.mds.g.smithi052.stdout:May 27 07:28:20 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-g[102352]: 5: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7f390597fdd1] 2024-05-27T07:28:20.175 INFO:journalctl@ceph.mds.g.smithi052.stdout:May 27 07:28:20 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-g[102352]: 6: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x555d8d15c23d] 2024-05-27T07:28:20.175 INFO:journalctl@ceph.mds.g.smithi052.stdout:May 27 07:28:20 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-g[102352]: 7: (SnapRealm::build_snap_set() const+0x323) [0x555d8d147971] 2024-05-27T07:28:20.175 INFO:journalctl@ceph.mds.g.smithi052.stdout:May 27 07:28:20 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-g[102352]: 8: (SnapRealm::check_cache() const+0x13e) [0x555d8d145fa0] 2024-05-27T07:28:20.175 INFO:journalctl@ceph.mds.g.smithi052.stdout:May 27 07:28:20 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-g[102352]: 9: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x555d8d146651] 2024-05-27T07:28:20.175 INFO:journalctl@ceph.mds.g.smithi052.stdout:May 27 07:28:20 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-g[102352]: 10: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x555d8d0db347] 2024-05-27T07:28:20.176 INFO:journalctl@ceph.mds.g.smithi052.stdout:May 27 07:28:20 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-g[102352]: 11: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x555d8cdd787e] 2024-05-27T07:28:20.176 INFO:journalctl@ceph.mds.g.smithi052.stdout:May 27 07:28:20 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-g[102352]: 12: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x2702) [0x555d8cddce94] 2024-05-27T07:28:20.176 INFO:journalctl@ceph.mds.g.smithi052.stdout:May 27 07:28:20 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-g[102352]: 13: (EUpdate::replay(MDSRank*)+0x63) [0x555d8cde7969] 2024-05-27T07:28:20.176 INFO:journalctl@ceph.mds.g.smithi052.stdout:May 27 07:28:20 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-g[102352]: 14: (MDLog::_replay_thread()+0x1f38) [0x555d8d18048c] 2024-05-27T07:28:20.176 INFO:journalctl@ceph.mds.g.smithi052.stdout:May 27 07:28:20 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-g[102352]: 15: (MDLog::ReplayThread::entry()+0x11) [0x555d8d1899cb] 2024-05-27T07:28:20.176 INFO:journalctl@ceph.mds.g.smithi052.stdout:May 27 07:28:20 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-g[102352]: 16: (Thread::entry_wrapper()+0x43) [0x7f390594e72d] 2024-05-27T07:28:20.176 INFO:journalctl@ceph.mds.g.smithi052.stdout:May 27 07:28:20 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-g[102352]: 17: (Thread::_entry_func(void*)+0xd) [0x7f390594e749] 2024-05-27T07:28:20.177 INFO:journalctl@ceph.mds.g.smithi052.stdout:May 27 07:28:20 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-g[102352]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7f39044f21ca] 2024-05-27T07:28:20.177 INFO:journalctl@ceph.mds.g.smithi052.stdout:May 27 07:28:20 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-g[102352]: 19: clone() 2024-05-27T07:28:20.177 INFO:journalctl@ceph.mds.g.smithi052.stdout:May 27 07:28:20 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-g[102352]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-05-27T07:28:20.177 INFO:journalctl@ceph.mds.g.smithi052.stdout:May 27 07:28:20 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-g[102352]: 2024-05-27T07:28:20.177 INFO:journalctl@ceph.mds.g.smithi052.stdout:May 27 07:28:20 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-g[102352]: -1> 2024-05-27T07:28:20.012+0000 7f38f313a700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: In function 'std::set SnapClient::filter(const std::set&) const' thread 7f38f313a700 time 2024-05-27T07:28:20.008485+0000 2024-05-27T07:28:20.177 INFO:journalctl@ceph.mds.g.smithi052.stdout:May 27 07:28:20 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-g[102352]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: 197: FAILED ceph_assert(cached_version > 0) 2024-05-27T07:28:20.177 INFO:journalctl@ceph.mds.g.smithi052.stdout:May 27 07:28:20 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-g[102352]: 2024-05-27T07:28:20.177 INFO:journalctl@ceph.mds.g.smithi052.stdout:May 27 07:28:20 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-g[102352]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:28:20.178 INFO:journalctl@ceph.mds.g.smithi052.stdout:May 27 07:28:20 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-g[102352]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x127) [0x7f390597fba6] 2024-05-27T07:28:20.178 INFO:journalctl@ceph.mds.g.smithi052.stdout:May 27 07:28:20 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-g[102352]: 2: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7f390597fdd1] 2024-05-27T07:28:20.178 INFO:journalctl@ceph.mds.g.smithi052.stdout:May 27 07:28:20 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-g[102352]: 3: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x555d8d15c23d] 2024-05-27T07:28:20.178 INFO:journalctl@ceph.mds.g.smithi052.stdout:May 27 07:28:20 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-g[102352]: 4: (SnapRealm::build_snap_set() const+0x323) [0x555d8d147971] 2024-05-27T07:28:20.178 INFO:journalctl@ceph.mds.g.smithi052.stdout:May 27 07:28:20 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-g[102352]: 5: (SnapRealm::check_cache() const+0x13e) [0x555d8d145fa0] 2024-05-27T07:28:20.179 INFO:journalctl@ceph.mds.g.smithi052.stdout:May 27 07:28:20 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-g[102352]: 6: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x555d8d146651] 2024-05-27T07:28:20.179 INFO:journalctl@ceph.mds.g.smithi052.stdout:May 27 07:28:20 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-g[102352]: 7: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x555d8d0db347] 2024-05-27T07:28:20.179 INFO:journalctl@ceph.mds.g.smithi052.stdout:May 27 07:28:20 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-g[102352]: 8: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x555d8cdd787e] 2024-05-27T07:28:20.179 INFO:journalctl@ceph.mds.g.smithi052.stdout:May 27 07:28:20 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-g[102352]: 9: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x2702) [0x555d8cddce94] 2024-05-27T07:28:20.179 INFO:journalctl@ceph.mds.g.smithi052.stdout:May 27 07:28:20 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-g[102352]: 10: (EUpdate::replay(MDSRank*)+0x63) [0x555d8cde7969] 2024-05-27T07:28:20.179 INFO:journalctl@ceph.mds.g.smithi052.stdout:May 27 07:28:20 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-g[102352]: 11: (MDLog::_replay_thread()+0x1f38) [0x555d8d18048c] 2024-05-27T07:28:20.180 INFO:journalctl@ceph.mds.g.smithi052.stdout:May 27 07:28:20 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-g[102352]: 12: (MDLog::ReplayThread::entry()+0x11) [0x555d8d1899cb] 2024-05-27T07:28:20.180 INFO:journalctl@ceph.mds.g.smithi052.stdout:May 27 07:28:20 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-g[102352]: 13: (Thread::entry_wrapper()+0x43) [0x7f390594e72d] 2024-05-27T07:28:20.180 INFO:journalctl@ceph.mds.g.smithi052.stdout:May 27 07:28:20 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-g[102352]: 14: (Thread::_entry_func(void*)+0xd) [0x7f390594e749] 2024-05-27T07:28:20.180 INFO:journalctl@ceph.mds.g.smithi052.stdout:May 27 07:28:20 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-g[102352]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7f39044f21ca] 2024-05-27T07:28:20.180 INFO:journalctl@ceph.mds.g.smithi052.stdout:May 27 07:28:20 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-g[102352]: 16: clone() 2024-05-27T07:28:20.180 INFO:journalctl@ceph.mds.g.smithi052.stdout:May 27 07:28:20 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-g[102352]: 2024-05-27T07:28:20.181 INFO:journalctl@ceph.mds.g.smithi052.stdout:May 27 07:28:20 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-g[102352]: 0> 2024-05-27T07:28:20.020+0000 7f38f313a700 -1 *** Caught signal (Aborted) ** 2024-05-27T07:28:20.181 INFO:journalctl@ceph.mds.g.smithi052.stdout:May 27 07:28:20 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-g[102352]: in thread 7f38f313a700 thread_name:md_log_replay 2024-05-27T07:28:20.181 INFO:journalctl@ceph.mds.g.smithi052.stdout:May 27 07:28:20 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-g[102352]: 2024-05-27T07:28:20.181 INFO:journalctl@ceph.mds.g.smithi052.stdout:May 27 07:28:20 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-g[102352]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:28:20.181 INFO:journalctl@ceph.mds.g.smithi052.stdout:May 27 07:28:20 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-g[102352]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f39044fcd20] 2024-05-27T07:28:20.181 INFO:journalctl@ceph.mds.g.smithi052.stdout:May 27 07:28:20 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-g[102352]: 2: gsignal() 2024-05-27T07:28:20.181 INFO:journalctl@ceph.mds.g.smithi052.stdout:May 27 07:28:20 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-g[102352]: 3: abort() 2024-05-27T07:28:20.181 INFO:journalctl@ceph.mds.g.smithi052.stdout:May 27 07:28:20 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-g[102352]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x26e) [0x7f390597fced] 2024-05-27T07:28:20.181 INFO:journalctl@ceph.mds.g.smithi052.stdout:May 27 07:28:20 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-g[102352]: 5: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7f390597fdd1] 2024-05-27T07:28:20.181 INFO:journalctl@ceph.mds.g.smithi052.stdout:May 27 07:28:20 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-g[102352]: 6: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x555d8d15c23d] 2024-05-27T07:28:20.182 INFO:journalctl@ceph.mds.g.smithi052.stdout:May 27 07:28:20 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-g[102352]: 7: (SnapRealm::build_snap_set() const+0x323) [0x555d8d147971] 2024-05-27T07:28:20.182 INFO:journalctl@ceph.mds.g.smithi052.stdout:May 27 07:28:20 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-g[102352]: 8: (SnapRealm::check_cache() const+0x13e) [0x555d8d145fa0] 2024-05-27T07:28:20.182 INFO:journalctl@ceph.mds.g.smithi052.stdout:May 27 07:28:20 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-g[102352]: 9: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x555d8d146651] 2024-05-27T07:28:20.182 INFO:journalctl@ceph.mds.g.smithi052.stdout:May 27 07:28:20 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-g[102352]: 10: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x555d8d0db347] 2024-05-27T07:28:20.182 INFO:journalctl@ceph.mds.g.smithi052.stdout:May 27 07:28:20 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-g[102352]: 11: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x555d8cdd787e] 2024-05-27T07:28:20.182 INFO:journalctl@ceph.mds.g.smithi052.stdout:May 27 07:28:20 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-g[102352]: 12: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x2702) [0x555d8cddce94] 2024-05-27T07:28:20.182 INFO:journalctl@ceph.mds.g.smithi052.stdout:May 27 07:28:20 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-g[102352]: 13: (EUpdate::replay(MDSRank*)+0x63) [0x555d8cde7969] 2024-05-27T07:28:20.182 INFO:journalctl@ceph.mds.g.smithi052.stdout:May 27 07:28:20 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-g[102352]: 14: (MDLog::_replay_thread()+0x1f38) [0x555d8d18048c] 2024-05-27T07:28:20.182 INFO:journalctl@ceph.mds.g.smithi052.stdout:May 27 07:28:20 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-g[102352]: 15: (MDLog::ReplayThread::entry()+0x11) [0x555d8d1899cb] 2024-05-27T07:28:20.182 INFO:journalctl@ceph.mds.g.smithi052.stdout:May 27 07:28:20 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-g[102352]: 16: (Thread::entry_wrapper()+0x43) [0x7f390594e72d] 2024-05-27T07:28:20.182 INFO:journalctl@ceph.mds.g.smithi052.stdout:May 27 07:28:20 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-g[102352]: 17: (Thread::_entry_func(void*)+0xd) [0x7f390594e749] 2024-05-27T07:28:20.182 INFO:journalctl@ceph.mds.g.smithi052.stdout:May 27 07:28:20 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-g[102352]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7f39044f21ca] 2024-05-27T07:28:20.183 INFO:journalctl@ceph.mds.g.smithi052.stdout:May 27 07:28:20 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-g[102352]: 19: clone() 2024-05-27T07:28:20.183 INFO:journalctl@ceph.mds.g.smithi052.stdout:May 27 07:28:20 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-g[102352]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-05-27T07:28:20.183 INFO:journalctl@ceph.mds.g.smithi052.stdout:May 27 07:28:20 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-g[102352]: 2024-05-27T07:28:20.479 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:28:20 smithi052 ceph-mon[31955]: daemon mds.g restarted 2024-05-27T07:28:20.479 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:28:20 smithi052 ceph-mon[31955]: from='client.? 172.21.15.52:0/2274589921' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-27T07:28:20.479 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:28:20 smithi052 ceph-mon[31955]: osdmap e93: 12 total, 12 up, 12 in 2024-05-27T07:28:20.479 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:28:20 smithi052 ceph-mon[31955]: mds.? [v2:172.21.15.52:6838/682139508,v1:172.21.15.52:6839/682139508] up:boot 2024-05-27T07:28:20.479 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:28:20 smithi052 ceph-mon[31955]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-05-27T07:28:20.479 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:28:20 smithi052 ceph-mon[31955]: from='mgr.15333 172.21.15.52:0/416389082' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2024-05-27T07:28:20.601 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:28:20 smithi173 ceph-mon[37000]: daemon mds.g restarted 2024-05-27T07:28:20.601 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:28:20 smithi173 ceph-mon[37000]: from='client.? 172.21.15.52:0/2274589921' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-27T07:28:20.602 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:28:20 smithi173 ceph-mon[37000]: osdmap e93: 12 total, 12 up, 12 in 2024-05-27T07:28:20.602 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:28:20 smithi173 ceph-mon[37000]: mds.? [v2:172.21.15.52:6838/682139508,v1:172.21.15.52:6839/682139508] up:boot 2024-05-27T07:28:20.602 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:28:20 smithi173 ceph-mon[37000]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-05-27T07:28:20.602 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:28:20 smithi173 ceph-mon[37000]: from='mgr.15333 172.21.15.52:0/416389082' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2024-05-27T07:28:20.630 DEBUG:teuthology.orchestra.run.smithi052:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:38f9d900c83c63799fbdbe61acc9a11b0d3554a6 shell --fsid 73df2636-1bf8-11ef-bc9b-c7b262605968 -- ceph tell mds.1:4 damage ls 2024-05-27T07:28:20.690 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:28:20 smithi105 ceph-mon[35977]: daemon mds.g restarted 2024-05-27T07:28:20.690 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:28:20 smithi105 ceph-mon[35977]: from='client.? 172.21.15.52:0/2274589921' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-27T07:28:20.690 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:28:20 smithi105 ceph-mon[35977]: osdmap e93: 12 total, 12 up, 12 in 2024-05-27T07:28:20.690 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:28:20 smithi105 ceph-mon[35977]: mds.? [v2:172.21.15.52:6838/682139508,v1:172.21.15.52:6839/682139508] up:boot 2024-05-27T07:28:20.690 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:28:20 smithi105 ceph-mon[35977]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-05-27T07:28:20.690 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:28:20 smithi105 ceph-mon[35977]: from='mgr.15333 172.21.15.52:0/416389082' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2024-05-27T07:28:20.912 INFO:journalctl@ceph.mds.g.smithi052.stdout:May 27 07:28:20 smithi052 podman[102424]: 2024-05-27 07:28:20.650361852 +0000 UTC m=+0.567160787 container died ae945548727bd0c8e6eaa92ba1276da2a69dd2e3d7470a92e8aa51ef157a7d4e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-g, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, GIT_CLEAN=True, org.label-schema.build-date=20240521, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, ceph=True, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, org.label-schema.license=GPLv2) 2024-05-27T07:28:21.479 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:28:21 smithi052 ceph-mon[31955]: pgmap v77: 129 pgs: 129 active+clean; 71 MiB data, 515 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 MiB/s rd, 5.2 MiB/s wr, 1.53k op/s 2024-05-27T07:28:21.482 INFO:journalctl@ceph.mds.g.smithi052.stdout:May 27 07:28:21 smithi052 podman[102424]: 2024-05-27 07:28:21.215923566 +0000 UTC m=+1.132722505 container remove ae945548727bd0c8e6eaa92ba1276da2a69dd2e3d7470a92e8aa51ef157a7d4e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-g, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240521, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, ceph=True, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD) 2024-05-27T07:28:21.483 INFO:journalctl@ceph.mds.g.smithi052.stdout:May 27 07:28:21 smithi052 systemd[1]: ceph-73df2636-1bf8-11ef-bc9b-c7b262605968@mds.g.service: Main process exited, code=exited, status=134/n/a 2024-05-27T07:28:21.534 INFO:teuthology.orchestra.run.smithi052.stderr:Inferring config /var/lib/ceph/73df2636-1bf8-11ef-bc9b-c7b262605968/mon.a/config 2024-05-27T07:28:21.602 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:28:21 smithi173 ceph-mon[37000]: pgmap v77: 129 pgs: 129 active+clean; 71 MiB data, 515 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 MiB/s rd, 5.2 MiB/s wr, 1.53k op/s 2024-05-27T07:28:21.689 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:28:21 smithi105 ceph-mon[35977]: pgmap v77: 129 pgs: 129 active+clean; 71 MiB data, 515 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 MiB/s rd, 5.2 MiB/s wr, 1.53k op/s 2024-05-27T07:28:22.289 INFO:journalctl@ceph.mds.g.smithi052.stdout:May 27 07:28:22 smithi052 podman[102505]: 2024-05-27 07:28:22.113008923 +0000 UTC m=+0.551824685 container died 83ed089aaf3512574b44353c0c39dbcb021d29dc5559563275b425c060a794e4 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:38f9d900c83c63799fbdbe61acc9a11b0d3554a6, name=vibrant_jemison, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, GIT_CLEAN=True, io.buildah.version=1.33.5, org.label-schema.build-date=20240521, org.label-schema.license=GPLv2, ceph=True, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90) 2024-05-27T07:28:22.729 INFO:journalctl@ceph.mds.g.smithi052.stdout:May 27 07:28:22 smithi052 systemd[1]: ceph-73df2636-1bf8-11ef-bc9b-c7b262605968@mds.g.service: Failed with result 'exit-code'. 2024-05-27T07:28:23.519 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:23.528+0000 7f18eb571700 1 -- 172.21.15.52:0/660576448 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f18e4105470 msgr2=0x7f18e4105880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:28:23.519 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:23.528+0000 7f18eb571700 1 --2- 172.21.15.52:0/660576448 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f18e4105470 0x7f18e4105880 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f18e00098f0 tx=0x7f18e0009c00 comp rx=0 tx=0).stop 2024-05-27T07:28:23.519 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:23.529+0000 7f18eb571700 1 -- 172.21.15.52:0/660576448 shutdown_connections 2024-05-27T07:28:23.520 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:23.529+0000 7f18eb571700 1 --2- 172.21.15.52:0/660576448 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f18e410e8e0 0x7f18e4110dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:28:23.520 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:23.529+0000 7f18eb571700 1 --2- 172.21.15.52:0/660576448 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f18e4105470 0x7f18e4105880 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:28:23.520 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:23.529+0000 7f18eb571700 1 --2- 172.21.15.52:0/660576448 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f18e4105dc0 0x7f18e410e310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:28:23.520 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:23.529+0000 7f18eb571700 1 -- 172.21.15.52:0/660576448 >> 172.21.15.52:0/660576448 conn(0x7f18e40fef20 msgr2=0x7f18e4101340 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:28:23.520 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:23.529+0000 7f18eb571700 1 -- 172.21.15.52:0/660576448 shutdown_connections 2024-05-27T07:28:23.520 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:23.529+0000 7f18eb571700 1 -- 172.21.15.52:0/660576448 wait complete. 2024-05-27T07:28:23.522 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:23.530+0000 7f18eb571700 1 Processor -- start 2024-05-27T07:28:23.522 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:23.530+0000 7f18eb571700 1 -- start start 2024-05-27T07:28:23.522 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:23.530+0000 7f18eb571700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f18e4105470 0x7f18e410b4b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:28:23.522 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:23.530+0000 7f18eb571700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f18e4105dc0 0x7f18e410b9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:28:23.522 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:23.530+0000 7f18eb571700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f18e410e8e0 0x7f18e410bf30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:28:23.522 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:23.530+0000 7f18eb571700 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x7f18e4104770 con 0x7f18e410e8e0 2024-05-27T07:28:23.522 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:23.530+0000 7f18eb571700 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f18e41045f0 con 0x7f18e4105dc0 2024-05-27T07:28:23.522 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:23.530+0000 7f18eb571700 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f18e41048f0 con 0x7f18e4105470 2024-05-27T07:28:23.522 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:23.531+0000 7f18e930d700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f18e4105470 0x7f18e410b4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:28:23.522 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:23.531+0000 7f18e8b0c700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f18e4105dc0 0x7f18e410b9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:28:23.522 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:23.531+0000 7f18e8b0c700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f18e4105dc0 0x7f18e410b9f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.52:35672/0 (socket says 172.21.15.52:35672) 2024-05-27T07:28:23.523 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:23.531+0000 7f18e930d700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f18e4105470 0x7f18e410b4b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.173:3300/0 says I am v2:172.21.15.52:37390/0 (socket says 172.21.15.52:37390) 2024-05-27T07:28:23.523 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:23.531+0000 7f18e8b0c700 1 -- 172.21.15.52:0/1226498341 learned_addr learned my addr 172.21.15.52:0/1226498341 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-05-27T07:28:23.523 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:23.531+0000 7f18e9b0e700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f18e410e8e0 0x7f18e410bf30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:28:23.523 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:23.531+0000 7f18e930d700 1 -- 172.21.15.52:0/1226498341 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f18e4105dc0 msgr2=0x7f18e410b9f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:28:23.523 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:23.531+0000 7f18e930d700 1 --2- 172.21.15.52:0/1226498341 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f18e4105dc0 0x7f18e410b9f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:28:23.523 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:23.531+0000 7f18e930d700 1 -- 172.21.15.52:0/1226498341 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f18e410e8e0 msgr2=0x7f18e410bf30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:28:23.523 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:23.531+0000 7f18e930d700 1 --2- 172.21.15.52:0/1226498341 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f18e410e8e0 0x7f18e410bf30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:28:23.523 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:23.531+0000 7f18e930d700 1 -- 172.21.15.52:0/1226498341 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f18e41ad6e0 con 0x7f18e4105470 2024-05-27T07:28:23.523 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:23.531+0000 7f18e930d700 1 --2- 172.21.15.52:0/1226498341 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f18e4105470 0x7f18e410b4b0 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f18e0009230 tx=0x7f18e000fa00 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:28:23.523 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:23.532+0000 7f18da7fc700 1 -- 172.21.15.52:0/1226498341 <== mon.1 v2:172.21.15.173:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f18e0005040 con 0x7f18e4105470 2024-05-27T07:28:23.523 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:23.532+0000 7f18da7fc700 1 -- 172.21.15.52:0/1226498341 <== mon.1 v2:172.21.15.173:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f18e0005690 con 0x7f18e4105470 2024-05-27T07:28:23.523 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:23.532+0000 7f18eb571700 1 -- 172.21.15.52:0/1226498341 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f18e41ad8b0 con 0x7f18e4105470 2024-05-27T07:28:23.524 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:23.532+0000 7f18da7fc700 1 -- 172.21.15.52:0/1226498341 <== mon.1 v2:172.21.15.173:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f18e00038e0 con 0x7f18e4105470 2024-05-27T07:28:23.524 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:23.532+0000 7f18eb571700 1 -- 172.21.15.52:0/1226498341 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f18e41add00 con 0x7f18e4105470 2024-05-27T07:28:23.524 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:23.534+0000 7f18da7fc700 1 -- 172.21.15.52:0/1226498341 <== mon.1 v2:172.21.15.173:3300/0 4 ==== mgrmap(e 26) ==== 149783+0+0 (secure 0 0 0) 0x7f18e0003a80 con 0x7f18e4105470 2024-05-27T07:28:23.525 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:23.536+0000 7f18da7fc700 1 --2- 172.21.15.52:0/1226498341 >> [v2:172.21.15.52:6800/1822318384,v1:172.21.15.52:6801/1822318384] conn(0x7f18d00b29f0 0x7f18d00b4f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:28:23.526 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:23.536+0000 7f18e8b0c700 1 --2- 172.21.15.52:0/1226498341 >> [v2:172.21.15.52:6800/1822318384,v1:172.21.15.52:6801/1822318384] conn(0x7f18d00b29f0 0x7f18d00b4f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:28:23.526 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:23.536+0000 7f18da7fc700 1 -- 172.21.15.52:0/1226498341 <== mon.1 v2:172.21.15.173:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9379+0+0 (secure 0 0 0) 0x7f18e00e37f0 con 0x7f18e4105470 2024-05-27T07:28:23.527 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:23.536+0000 7f18e8b0c700 1 --2- 172.21.15.52:0/1226498341 >> [v2:172.21.15.52:6800/1822318384,v1:172.21.15.52:6801/1822318384] conn(0x7f18d00b29f0 0x7f18d00b4f00 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f18e41020d0 tx=0x7f18d4009260 comp rx=0 tx=0).ready entity=mgr.15333 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:28:23.527 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:23.537+0000 7f18f9f86b80 1 Processor -- start 2024-05-27T07:28:23.527 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:23.537+0000 7f18f9f86b80 1 -- start start 2024-05-27T07:28:23.527 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:23.537+0000 7f18f9f86b80 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55be6260de40 0x55be62791710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:28:23.527 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:23.537+0000 7f18f9f86b80 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55be62791c50 0x55be6278ba40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:28:23.527 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:23.537+0000 7f18f9f86b80 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55be6278bf80 0x55be6278c390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:28:23.527 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:23.537+0000 7f18f9f86b80 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x55be626bac90 con 0x55be62791c50 2024-05-27T07:28:23.528 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:23.537+0000 7f18f9f86b80 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x55be626aa830 con 0x55be6278bf80 2024-05-27T07:28:23.528 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:23.537+0000 7f18f9f86b80 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x55be626ae870 con 0x55be6260de40 2024-05-27T07:28:23.528 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:23.537+0000 7f18e930d700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55be6260de40 0x55be62791710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:28:23.528 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:23.537+0000 7f18e8b0c700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55be62791c50 0x55be6278ba40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:28:23.528 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:23.537+0000 7f18e9b0e700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55be6278bf80 0x55be6278c390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:28:23.528 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:23.537+0000 7f18e930d700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55be6260de40 0x55be62791710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.173:3300/0 says I am v2:172.21.15.52:37392/0 (socket says 172.21.15.52:37392) 2024-05-27T07:28:23.528 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:23.537+0000 7f18e930d700 1 -- 172.21.15.52:0/1926957998 learned_addr learned my addr 172.21.15.52:0/1926957998 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-05-27T07:28:23.528 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:23.538+0000 7f18e930d700 1 -- 172.21.15.52:0/1926957998 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55be6278bf80 msgr2=0x55be6278c390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:28:23.528 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:23.538+0000 7f18e930d700 1 --2- 172.21.15.52:0/1926957998 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55be6278bf80 0x55be6278c390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:28:23.528 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:23.538+0000 7f18e930d700 1 -- 172.21.15.52:0/1926957998 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55be62791c50 msgr2=0x55be6278ba40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:28:23.528 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:23.538+0000 7f18e930d700 1 --2- 172.21.15.52:0/1926957998 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55be62791c50 0x55be6278ba40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:28:23.528 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:23.538+0000 7f18e930d700 1 -- 172.21.15.52:0/1926957998 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55be625fa8d0 con 0x55be6260de40 2024-05-27T07:28:23.529 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:23.538+0000 7f18e9b0e700 1 --2- 172.21.15.52:0/1926957998 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55be6278bf80 0x55be6278c390 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-27T07:28:23.531 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:23.538+0000 7f18e930d700 1 --2- 172.21.15.52:0/1926957998 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55be6260de40 0x55be62791710 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f18e00aa4a0 tx=0x7f18e0009c00 comp rx=0 tx=0).ready entity=mon.1 client_cookie=61cd1058b4a5afa5 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:28:23.531 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:23.538+0000 7f18cf7fe700 1 -- 172.21.15.52:0/1926957998 <== mon.1 v2:172.21.15.173:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f18e00aa2b0 con 0x55be6260de40 2024-05-27T07:28:23.531 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:23.538+0000 7f18cf7fe700 1 -- 172.21.15.52:0/1926957998 <== mon.1 v2:172.21.15.173:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f18e0058070 con 0x55be6260de40 2024-05-27T07:28:23.531 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:23.538+0000 7f18cf7fe700 1 -- 172.21.15.52:0/1926957998 <== mon.1 v2:172.21.15.173:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f18e00b3570 con 0x55be6260de40 2024-05-27T07:28:23.531 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:23.539+0000 7f18f9f86b80 1 -- 172.21.15.52:0/1926957998 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55be6260de40 msgr2=0x55be62791710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:28:23.531 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:23.539+0000 7f18f9f86b80 1 --2- 172.21.15.52:0/1926957998 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55be6260de40 0x55be62791710 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f18e00aa4a0 tx=0x7f18e0009c00 comp rx=0 tx=0).stop 2024-05-27T07:28:23.531 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:23.539+0000 7f18f9f86b80 1 -- 172.21.15.52:0/1926957998 shutdown_connections 2024-05-27T07:28:23.532 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:23.539+0000 7f18f9f86b80 1 --2- 172.21.15.52:0/1926957998 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55be6278bf80 0x55be6278c390 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:28:23.532 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:23.539+0000 7f18f9f86b80 1 --2- 172.21.15.52:0/1926957998 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55be6260de40 0x55be62791710 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:28:23.532 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:23.539+0000 7f18f9f86b80 1 --2- 172.21.15.52:0/1926957998 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55be62791c50 0x55be6278ba40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:28:23.532 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:23.539+0000 7f18f9f86b80 1 -- 172.21.15.52:0/1926957998 >> 172.21.15.52:0/1926957998 conn(0x55be62668af0 msgr2=0x55be62786640 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:28:23.532 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:23.539+0000 7f18f9f86b80 1 -- 172.21.15.52:0/1926957998 shutdown_connections 2024-05-27T07:28:23.532 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:23.539+0000 7f18f9f86b80 1 -- 172.21.15.52:0/1926957998 wait complete. 2024-05-27T07:28:23.532 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:23.540+0000 7f18f9f86b80 1 -- start start 2024-05-27T07:28:23.532 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:23.540+0000 7f18f9f86b80 1 Processor -- start 2024-05-27T07:28:23.532 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:23.540+0000 7f18f9f86b80 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55be6260de40 0x55be62795390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:28:23.532 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:23.540+0000 7f18f9f86b80 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55be6278bf80 0x55be627958d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:28:23.532 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:23.540+0000 7f18f9f86b80 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55be62791c50 0x55be62795e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:28:23.532 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:23.540+0000 7f18f9f86b80 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x55be625b0fa0 con 0x55be62791c50 2024-05-27T07:28:23.532 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:23.540+0000 7f18f9f86b80 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x55be6264c7c0 con 0x55be6260de40 2024-05-27T07:28:23.533 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:23.540+0000 7f18f9f86b80 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x55be62798500 con 0x55be6278bf80 2024-05-27T07:28:23.533 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:23.540+0000 7f18f9f86b80 10 client.0 _ll_register_callbacks cb 0x55be62645920 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T07:28:23.533 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:23.540+0000 7f18e930d700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55be6260de40 0x55be62795390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:28:23.533 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:23.540+0000 7f18e9b0e700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55be62791c50 0x55be62795e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:28:23.533 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:23.540+0000 7f18e930d700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55be6260de40 0x55be62795390 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.52:35686/0 (socket says 172.21.15.52:35686) 2024-05-27T07:28:23.533 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:23.540+0000 7f18e930d700 1 -- 172.21.15.52:0/1290651343 learned_addr learned my addr 172.21.15.52:0/1290651343 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-05-27T07:28:23.533 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:23.541+0000 7f18e8b0c700 1 --2- 172.21.15.52:0/1290651343 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55be6278bf80 0x55be627958d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:28:23.533 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:23.541+0000 7f18e930d700 1 -- 172.21.15.52:0/1290651343 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55be6278bf80 msgr2=0x55be627958d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:28:23.533 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:23.541+0000 7f18e930d700 1 --2- 172.21.15.52:0/1290651343 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55be6278bf80 0x55be627958d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:28:23.533 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:23.541+0000 7f18e930d700 1 -- 172.21.15.52:0/1290651343 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55be62791c50 msgr2=0x55be62795e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:28:23.533 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:23.541+0000 7f18e930d700 1 --2- 172.21.15.52:0/1290651343 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55be62791c50 0x55be62795e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:28:23.533 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:23.541+0000 7f18e930d700 1 -- 172.21.15.52:0/1290651343 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55be62649690 con 0x55be6260de40 2024-05-27T07:28:23.533 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:23.541+0000 7f18e930d700 1 --2- 172.21.15.52:0/1290651343 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55be6260de40 0x55be62795390 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f18e00098c0 tx=0x7f18e00a6cc0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=33ae5d71aa38cf9e server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:28:23.534 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:23.541+0000 7f18ccff9700 10 client.0 ms_handle_connect on v2:172.21.15.105:3300/0 2024-05-27T07:28:23.534 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:23.541+0000 7f18ccff9700 1 -- 172.21.15.52:0/1290651343 <== mon.2 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f18e00aa2b0 con 0x55be6260de40 2024-05-27T07:28:23.534 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:23.541+0000 7f18f9f86b80 1 -- 172.21.15.52:0/1290651343 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f18e0058070 con 0x55be6260de40 2024-05-27T07:28:23.534 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:23.541+0000 7f18ccff9700 1 -- 172.21.15.52:0/1290651343 <== mon.2 v2:172.21.15.105:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9379+0+0 (secure 0 0 0) 0x7f18e00a6d90 con 0x55be6260de40 2024-05-27T07:28:23.534 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:23.542+0000 7f18ccff9700 1 -- 172.21.15.52:0/1290651343 <== mon.2 v2:172.21.15.105:3300/0 3 ==== mon_get_version_reply(handle=1 version=36) ==== 24+0+0 (secure 0 0 0) 0x7f18e00aa7a0 con 0x55be6260de40 2024-05-27T07:28:23.534 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:23.542+0000 7f18f9f86b80 10 client.25715 fetch_fsmap learned FSMap version 36 2024-05-27T07:28:23.534 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:23.542+0000 7f18f9f86b80 1 -- 172.21.15.52:0/1290651343 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,fsmap=36,monmap=4+}) -- 0x55be627a2f70 con 0x55be6260de40 2024-05-27T07:28:23.536 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:23.542+0000 7f18ccff9700 1 -- 172.21.15.52:0/1290651343 <== mon.2 v2:172.21.15.105:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f18e0058070 con 0x55be6260de40 2024-05-27T07:28:23.536 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:23.542+0000 7f18ccff9700 1 -- 172.21.15.52:0/1290651343 <== mon.2 v2:172.21.15.105:3300/0 5 ==== fsmap(e 36) ==== 8374+0+0 (secure 0 0 0) 0x7f18e0090ce0 con 0x55be6260de40 2024-05-27T07:28:23.536 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:23.542+0000 7f18f9f86b80 10 client.25715 fetch_fsmap finished waiting for FSMap version 36 2024-05-27T07:28:23.536 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:23.542+0000 7f18f9f86b80 10 client.25715 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.c 2024-05-27T07:28:23.536 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:23.542+0000 7f18f9f86b80 20 client.25715 populate_metadata read hostname 'smithi052' 2024-05-27T07:28:23.536 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:23.542+0000 7f18f9f86b80 1 --2- 172.21.15.52:0/1290651343 >> [v2:172.21.15.173:6836/1396363827,v1:172.21.15.173:6837/1396363827] conn(0x55be627a5440 0x55be627a7860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:28:23.536 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:23.542+0000 7f18f9f86b80 4 client.25715 mds_command: new command op to 24466 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-27T07:28:23.536 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:23.542+0000 7f18f9f86b80 1 -- 172.21.15.52:0/1290651343 --> [v2:172.21.15.173:6836/1396363827,v1:172.21.15.173:6837/1396363827] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55be627579f0 con 0x55be627a5440 2024-05-27T07:28:23.536 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:23.542+0000 7f18e8b0c700 1 --2- 172.21.15.52:0/1290651343 >> [v2:172.21.15.173:6836/1396363827,v1:172.21.15.173:6837/1396363827] conn(0x55be627a5440 0x55be627a7860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:28:23.537 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:23.543+0000 7f18ccff9700 10 client.25715 ms_handle_connect on v2:172.21.15.173:6836/1396363827 2024-05-27T07:28:23.537 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:23.543+0000 7f18e8b0c700 1 --2- 172.21.15.52:0/1290651343 >> [v2:172.21.15.173:6836/1396363827,v1:172.21.15.173:6837/1396363827] conn(0x55be627a5440 0x55be627a7860 crc :-1 s=READY pgs=35 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=608bc4f9ee3447e5 server_cookie=6ea2ecbd5a3f2102 in_seq=0 out_seq=0 2024-05-27T07:28:23.537 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:23.544+0000 7f18ccff9700 1 -- 172.21.15.52:0/1290651343 <== mds.4 v2:172.21.15.173:6836/1396363827 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24041 (crc 0 0 0) 0x55be627579f0 con 0x55be627a5440 2024-05-27T07:28:23.537 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:23.544+0000 7f18ccff9700 10 client.25715 handle_command_reply: tid=0 2024-05-27T07:28:23.537 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:23.544+0000 7f18f9f86b80 1 client.25715 shutdown 2024-05-27T07:28:23.537 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:23.544+0000 7f18f9f86b80 1 -- 172.21.15.52:0/1290651343 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55be6260de40 msgr2=0x55be62795390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:28:23.537 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:23.544+0000 7f18f9f86b80 1 --2- 172.21.15.52:0/1290651343 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55be6260de40 0x55be62795390 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f18e00098c0 tx=0x7f18e00a6cc0 comp rx=0 tx=0).stop 2024-05-27T07:28:23.537 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:23.544+0000 7f18f9f86b80 1 -- 172.21.15.52:0/1290651343 shutdown_connections 2024-05-27T07:28:23.537 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:23.544+0000 7f18f9f86b80 1 --2- 172.21.15.52:0/1290651343 >> [v2:172.21.15.173:6836/1396363827,v1:172.21.15.173:6837/1396363827] conn(0x55be627a5440 0x55be627a7860 crc :-1 s=READY pgs=35 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:28:23.537 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:23.544+0000 7f18f9f86b80 1 --2- 172.21.15.52:0/1290651343 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55be62791c50 0x55be62795e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:28:23.537 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:23.544+0000 7f18f9f86b80 1 --2- 172.21.15.52:0/1290651343 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55be6260de40 0x55be62795390 unknown :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:28:23.537 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:23.544+0000 7f18ccff9700 0 client.25715 ms_handle_reset on v2:172.21.15.173:6836/1396363827 2024-05-27T07:28:23.537 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:23.544+0000 7f18f9f86b80 1 --2- 172.21.15.52:0/1290651343 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55be6278bf80 0x55be627958d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:28:23.537 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:23.544+0000 7f18f9f86b80 1 -- 172.21.15.52:0/1290651343 >> 172.21.15.52:0/1290651343 conn(0x55be62668af0 msgr2=0x55be6278aa30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:28:23.538 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:23.544+0000 7f18f9f86b80 1 -- 172.21.15.52:0/1290651343 shutdown_connections 2024-05-27T07:28:23.538 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:23.544+0000 7f18f9f86b80 1 -- 172.21.15.52:0/1290651343 wait complete. 2024-05-27T07:28:23.538 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:23.544+0000 7f18f9f86b80 20 client.25715 trim_cache size 0 max 16384 2024-05-27T07:28:23.550 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:23.560+0000 7f18f9f86b80 1 Processor -- start 2024-05-27T07:28:23.551 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:23.560+0000 7f18f9f86b80 1 -- start start 2024-05-27T07:28:23.551 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:23.560+0000 7f18f9f86b80 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55be6260de40 0x55be6278ba40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:28:23.551 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:23.560+0000 7f18f9f86b80 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55be6278bf80 0x55be62795c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:28:23.551 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:23.560+0000 7f18f9f86b80 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55be62791c50 0x55be627a1560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:28:23.551 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:23.560+0000 7f18f9f86b80 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x7f18e41b69c0 con 0x55be62791c50 2024-05-27T07:28:23.551 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:23.560+0000 7f18f9f86b80 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f18e41b8590 con 0x55be6260de40 2024-05-27T07:28:23.551 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:23.560+0000 7f18f9f86b80 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f18e41b7090 con 0x55be6278bf80 2024-05-27T07:28:23.551 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:23.561+0000 7f18e930d700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55be6260de40 0x55be6278ba40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:28:23.551 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:23.561+0000 7f18e9b0e700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55be62791c50 0x55be627a1560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:28:23.551 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:23.561+0000 7f18e930d700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55be6260de40 0x55be6278ba40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.52:35690/0 (socket says 172.21.15.52:35690) 2024-05-27T07:28:23.551 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:23.561+0000 7f18e930d700 1 -- 172.21.15.52:0/3320958830 learned_addr learned my addr 172.21.15.52:0/3320958830 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-05-27T07:28:23.551 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:23.561+0000 7f18e8b0c700 1 --2- 172.21.15.52:0/3320958830 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55be6278bf80 0x55be62795c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:28:23.551 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:23.561+0000 7f18e930d700 1 -- 172.21.15.52:0/3320958830 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55be6278bf80 msgr2=0x55be62795c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:28:23.552 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:23.561+0000 7f18e930d700 1 --2- 172.21.15.52:0/3320958830 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55be6278bf80 0x55be62795c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:28:23.552 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:23.561+0000 7f18e930d700 1 -- 172.21.15.52:0/3320958830 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55be62791c50 msgr2=0x55be627a1560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:28:23.552 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:23.561+0000 7f18e930d700 1 --2- 172.21.15.52:0/3320958830 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55be62791c50 0x55be627a1560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:28:23.552 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:23.561+0000 7f18e930d700 1 -- 172.21.15.52:0/3320958830 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55be627a7bd0 con 0x55be6260de40 2024-05-27T07:28:23.552 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:23.561+0000 7f18e9b0e700 1 --2- 172.21.15.52:0/3320958830 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55be62791c50 0x55be627a1560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-27T07:28:23.552 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:23.561+0000 7f18e930d700 1 --2- 172.21.15.52:0/3320958830 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55be6260de40 0x55be6278ba40 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f18e000bab0 tx=0x7f18e00e0c60 comp rx=0 tx=0).ready entity=mon.2 client_cookie=59cbdedd47fb273c server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:28:23.555 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:23.561+0000 7f18ceffd700 1 -- 172.21.15.52:0/3320958830 <== mon.2 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f18e00e1010 con 0x55be6260de40 2024-05-27T07:28:23.555 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:23.561+0000 7f18ceffd700 1 -- 172.21.15.52:0/3320958830 <== mon.2 v2:172.21.15.105:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f18e0058070 con 0x55be6260de40 2024-05-27T07:28:23.555 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:23.561+0000 7f18ceffd700 1 -- 172.21.15.52:0/3320958830 <== mon.2 v2:172.21.15.105:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f18e00b3570 con 0x55be6260de40 2024-05-27T07:28:23.555 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:23.562+0000 7f18f9f86b80 1 -- 172.21.15.52:0/3320958830 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55be6260de40 msgr2=0x55be6278ba40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:28:23.555 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:23.562+0000 7f18f9f86b80 1 --2- 172.21.15.52:0/3320958830 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55be6260de40 0x55be6278ba40 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f18e000bab0 tx=0x7f18e00e0c60 comp rx=0 tx=0).stop 2024-05-27T07:28:23.555 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:23.563+0000 7f18f9f86b80 1 -- 172.21.15.52:0/3320958830 shutdown_connections 2024-05-27T07:28:23.555 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:23.563+0000 7f18f9f86b80 1 --2- 172.21.15.52:0/3320958830 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55be62791c50 0x55be627a1560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:28:23.555 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:23.563+0000 7f18f9f86b80 1 --2- 172.21.15.52:0/3320958830 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55be6260de40 0x55be6278ba40 unknown :-1 s=CLOSED pgs=94 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:28:23.555 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:23.563+0000 7f18f9f86b80 1 --2- 172.21.15.52:0/3320958830 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55be6278bf80 0x55be62795c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:28:23.555 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:23.563+0000 7f18f9f86b80 1 -- 172.21.15.52:0/3320958830 >> 172.21.15.52:0/3320958830 conn(0x55be62668af0 msgr2=0x55be627861c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:28:23.556 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:23.563+0000 7f18f9f86b80 1 -- 172.21.15.52:0/3320958830 shutdown_connections 2024-05-27T07:28:23.556 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:23.563+0000 7f18f9f86b80 1 -- 172.21.15.52:0/3320958830 wait complete. 2024-05-27T07:28:23.556 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:23.563+0000 7f18f9f86b80 1 -- start start 2024-05-27T07:28:23.556 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:23.563+0000 7f18f9f86b80 1 Processor -- start 2024-05-27T07:28:23.556 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:23.564+0000 7f18f9f86b80 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55be6260de40 0x55be627ad7e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:28:23.556 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:23.564+0000 7f18f9f86b80 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55be6278bf80 0x55be6279d770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:28:23.556 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:23.564+0000 7f18f9f86b80 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55be62791c50 0x55be6279de30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:28:23.556 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:23.564+0000 7f18f9f86b80 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x7f18e41b7ec0 con 0x55be6278bf80 2024-05-27T07:28:23.556 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:23.564+0000 7f18f9f86b80 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f18e41b7760 con 0x55be62791c50 2024-05-27T07:28:23.556 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:23.564+0000 7f18f9f86b80 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f18e41b4960 con 0x55be6260de40 2024-05-27T07:28:23.556 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:23.564+0000 7f18f9f86b80 10 client.0 _ll_register_callbacks cb 0x55be62645920 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T07:28:23.556 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:23.564+0000 7f18e930d700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55be6260de40 0x55be627ad7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:28:23.557 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:23.564+0000 7f18e9b0e700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55be62791c50 0x55be6279de30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:28:23.557 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:23.564+0000 7f18e8b0c700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55be6278bf80 0x55be6279d770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:28:23.557 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:23.564+0000 7f18e930d700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55be6260de40 0x55be627ad7e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.173:3300/0 says I am v2:172.21.15.52:37416/0 (socket says 172.21.15.52:37416) 2024-05-27T07:28:23.557 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:23.564+0000 7f18e930d700 1 -- 172.21.15.52:0/3077946248 learned_addr learned my addr 172.21.15.52:0/3077946248 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-05-27T07:28:23.557 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:23.564+0000 7f18e930d700 1 -- 172.21.15.52:0/3077946248 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55be62791c50 msgr2=0x55be6279de30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:28:23.557 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:23.564+0000 7f18e930d700 1 --2- 172.21.15.52:0/3077946248 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55be62791c50 0x55be6279de30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:28:23.557 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:23.564+0000 7f18e930d700 1 -- 172.21.15.52:0/3077946248 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55be6278bf80 msgr2=0x55be6279d770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:28:23.557 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:23.564+0000 7f18e930d700 1 --2- 172.21.15.52:0/3077946248 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55be6278bf80 0x55be6279d770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:28:23.557 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:23.564+0000 7f18e930d700 1 -- 172.21.15.52:0/3077946248 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55be627ac040 con 0x55be6260de40 2024-05-27T07:28:23.557 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:23.564+0000 7f18e9b0e700 1 --2- 172.21.15.52:0/3077946248 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55be62791c50 0x55be6279de30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-27T07:28:23.557 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:23.564+0000 7f18e8b0c700 1 --2- 172.21.15.52:0/3077946248 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55be6278bf80 0x55be6279d770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-27T07:28:23.557 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:23.564+0000 7f18e930d700 1 --2- 172.21.15.52:0/3077946248 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55be6260de40 0x55be627ad7e0 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f18e0005a80 tx=0x7f18e0009c00 comp rx=0 tx=0).ready entity=mon.1 client_cookie=1f6d966e29a8ecb server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:28:23.557 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:23.564+0000 7f18ccff9700 10 client.0 ms_handle_connect on v2:172.21.15.173:3300/0 2024-05-27T07:28:23.558 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:23.564+0000 7f18ccff9700 1 -- 172.21.15.52:0/3077946248 <== mon.1 v2:172.21.15.173:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f18e00e1010 con 0x55be6260de40 2024-05-27T07:28:23.558 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:23.565+0000 7f18ccff9700 1 -- 172.21.15.52:0/3077946248 <== mon.1 v2:172.21.15.173:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9379+0+0 (secure 0 0 0) 0x7f18e00a6d90 con 0x55be6260de40 2024-05-27T07:28:23.558 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:23.565+0000 7f18f9f86b80 1 -- 172.21.15.52:0/3077946248 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f18e0058070 con 0x55be6260de40 2024-05-27T07:28:23.558 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:23.565+0000 7f18ccff9700 1 -- 172.21.15.52:0/3077946248 <== mon.1 v2:172.21.15.173:3300/0 3 ==== mon_get_version_reply(handle=1 version=36) ==== 24+0+0 (secure 0 0 0) 0x7f18e00a6660 con 0x55be6260de40 2024-05-27T07:28:23.558 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:23.565+0000 7f18f9f86b80 10 client.25630 fetch_fsmap learned FSMap version 36 2024-05-27T07:28:23.558 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:23.565+0000 7f18f9f86b80 1 -- 172.21.15.52:0/3077946248 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({config=0+,fsmap=36,monmap=4+}) -- 0x55be627ac040 con 0x55be6260de40 2024-05-27T07:28:23.558 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:23.565+0000 7f18ccff9700 1 -- 172.21.15.52:0/3077946248 <== mon.1 v2:172.21.15.173:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f18e0058070 con 0x55be6260de40 2024-05-27T07:28:23.558 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:23.565+0000 7f18ccff9700 1 -- 172.21.15.52:0/3077946248 <== mon.1 v2:172.21.15.173:3300/0 5 ==== fsmap(e 36) ==== 8374+0+0 (secure 0 0 0) 0x7f18e0090ce0 con 0x55be6260de40 2024-05-27T07:28:23.558 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:23.565+0000 7f18f9f86b80 10 client.25630 fetch_fsmap finished waiting for FSMap version 36 2024-05-27T07:28:23.558 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:23.565+0000 7f18f9f86b80 10 client.25630 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.c 2024-05-27T07:28:23.558 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:23.565+0000 7f18f9f86b80 20 client.25630 populate_metadata read hostname 'smithi052' 2024-05-27T07:28:23.558 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:23.565+0000 7f18f9f86b80 1 --2- 172.21.15.52:0/3077946248 >> [v2:172.21.15.173:6836/1396363827,v1:172.21.15.173:6837/1396363827] conn(0x55be6280f340 0x55be62811760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:28:23.559 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:23.565+0000 7f18f9f86b80 4 client.25630 mds_command: new command op to 24466 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-05-27T07:28:23.559 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:23.565+0000 7f18f9f86b80 1 -- 172.21.15.52:0/3077946248 --> [v2:172.21.15.173:6836/1396363827,v1:172.21.15.173:6837/1396363827] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55be626bcbe0 con 0x55be6280f340 2024-05-27T07:28:23.559 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:23.565+0000 7f18e8b0c700 1 --2- 172.21.15.52:0/3077946248 >> [v2:172.21.15.173:6836/1396363827,v1:172.21.15.173:6837/1396363827] conn(0x55be6280f340 0x55be62811760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:28:23.560 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:23.566+0000 7f18ccff9700 10 client.25630 ms_handle_connect on v2:172.21.15.173:6836/1396363827 2024-05-27T07:28:23.560 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:23.566+0000 7f18e8b0c700 1 --2- 172.21.15.52:0/3077946248 >> [v2:172.21.15.173:6836/1396363827,v1:172.21.15.173:6837/1396363827] conn(0x55be6280f340 0x55be62811760 crc :-1 s=READY pgs=36 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=528827490777115b server_cookie=1004a2a50091c4c7 in_seq=0 out_seq=0 2024-05-27T07:28:23.560 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:23.566+0000 7f18ccff9700 1 -- 172.21.15.52:0/3077946248 <== mds.4 v2:172.21.15.173:6836/1396363827 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55be626bcbe0 con 0x55be6280f340 2024-05-27T07:28:23.560 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:23.566+0000 7f18ccff9700 10 client.25630 handle_command_reply: tid=0 2024-05-27T07:28:23.560 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:23.566+0000 7f18f9f86b80 1 client.25630 shutdown 2024-05-27T07:28:23.560 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:23.566+0000 7f18f9f86b80 1 -- 172.21.15.52:0/3077946248 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55be6260de40 msgr2=0x55be627ad7e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:28:23.560 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:23.566+0000 7f18f9f86b80 1 --2- 172.21.15.52:0/3077946248 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55be6260de40 0x55be627ad7e0 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f18e0005a80 tx=0x7f18e0009c00 comp rx=0 tx=0).stop 2024-05-27T07:28:23.560 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:23.566+0000 7f18f9f86b80 1 -- 172.21.15.52:0/3077946248 shutdown_connections 2024-05-27T07:28:23.561 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:23.566+0000 7f18f9f86b80 1 --2- 172.21.15.52:0/3077946248 >> [v2:172.21.15.173:6836/1396363827,v1:172.21.15.173:6837/1396363827] conn(0x55be6280f340 0x55be62811760 crc :-1 s=READY pgs=36 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:28:23.561 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:23.566+0000 7f18f9f86b80 1 --2- 172.21.15.52:0/3077946248 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55be62791c50 0x55be6279de30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:28:23.561 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:23.566+0000 7f18f9f86b80 1 --2- 172.21.15.52:0/3077946248 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55be6260de40 0x55be627ad7e0 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:28:23.561 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:23.566+0000 7f18ccff9700 0 client.25630 ms_handle_reset on v2:172.21.15.173:6836/1396363827 2024-05-27T07:28:23.561 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:23.566+0000 7f18f9f86b80 1 --2- 172.21.15.52:0/3077946248 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55be6278bf80 0x55be6279d770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:28:23.561 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:23.566+0000 7f18f9f86b80 1 -- 172.21.15.52:0/3077946248 >> 172.21.15.52:0/3077946248 conn(0x55be62668af0 msgr2=0x55be627861c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:28:23.561 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:23.567+0000 7f18f9f86b80 1 -- 172.21.15.52:0/3077946248 shutdown_connections 2024-05-27T07:28:23.561 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:23.567+0000 7f18f9f86b80 1 -- 172.21.15.52:0/3077946248 wait complete. 2024-05-27T07:28:23.561 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:23.567+0000 7f18f9f86b80 20 client.25630 trim_cache size 0 max 16384 2024-05-27T07:28:23.561 INFO:teuthology.orchestra.run.smithi052.stdout:[] 2024-05-27T07:28:23.562 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:23.567+0000 7f18eb571700 1 -- 172.21.15.52:0/1226498341 >> [v2:172.21.15.52:6800/1822318384,v1:172.21.15.52:6801/1822318384] conn(0x7f18d00b29f0 msgr2=0x7f18d00b4f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:28:23.562 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:23.567+0000 7f18eb571700 1 --2- 172.21.15.52:0/1226498341 >> [v2:172.21.15.52:6800/1822318384,v1:172.21.15.52:6801/1822318384] conn(0x7f18d00b29f0 0x7f18d00b4f00 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f18e41020d0 tx=0x7f18d4009260 comp rx=0 tx=0).stop 2024-05-27T07:28:23.562 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:23.567+0000 7f18eb571700 1 -- 172.21.15.52:0/1226498341 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f18e4105470 msgr2=0x7f18e410b4b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:28:23.562 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:23.567+0000 7f18eb571700 1 --2- 172.21.15.52:0/1226498341 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f18e4105470 0x7f18e410b4b0 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f18e0009230 tx=0x7f18e000fa00 comp rx=0 tx=0).stop 2024-05-27T07:28:23.562 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:23.567+0000 7f18eb571700 1 -- 172.21.15.52:0/1226498341 shutdown_connections 2024-05-27T07:28:23.563 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:23.567+0000 7f18eb571700 1 --2- 172.21.15.52:0/1226498341 >> [v2:172.21.15.52:6800/1822318384,v1:172.21.15.52:6801/1822318384] conn(0x7f18d00b29f0 0x7f18d00b4f00 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:28:23.563 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:23.567+0000 7f18eb571700 1 --2- 172.21.15.52:0/1226498341 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f18e410e8e0 0x7f18e410bf30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:28:23.563 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:23.567+0000 7f18eb571700 1 --2- 172.21.15.52:0/1226498341 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f18e4105470 0x7f18e410b4b0 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:28:23.563 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:23.567+0000 7f18eb571700 1 --2- 172.21.15.52:0/1226498341 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f18e4105dc0 0x7f18e410b9f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:28:23.563 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:23.567+0000 7f18eb571700 1 -- 172.21.15.52:0/1226498341 >> 172.21.15.52:0/1226498341 conn(0x7f18e40fef20 msgr2=0x7f18e40ff330 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:28:23.563 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:23.568+0000 7f18eb571700 1 -- 172.21.15.52:0/1226498341 shutdown_connections 2024-05-27T07:28:23.563 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:23.568+0000 7f18eb571700 1 -- 172.21.15.52:0/1226498341 wait complete. 2024-05-27T07:28:24.232 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:28:24 smithi052 ceph-mon[31955]: pgmap v78: 129 pgs: 129 active+clean; 72 MiB data, 493 MiB used, 1.0 TiB / 1.0 TiB avail; 7.0 MiB/s rd, 5.5 MiB/s wr, 1.69k op/s 2024-05-27T07:28:24.351 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:28:24 smithi173 ceph-mon[37000]: pgmap v78: 129 pgs: 129 active+clean; 72 MiB data, 493 MiB used, 1.0 TiB / 1.0 TiB avail; 7.0 MiB/s rd, 5.5 MiB/s wr, 1.69k op/s 2024-05-27T07:28:24.374 DEBUG:teuthology.orchestra.run.smithi052:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:38f9d900c83c63799fbdbe61acc9a11b0d3554a6 shell --fsid 73df2636-1bf8-11ef-bc9b-c7b262605968 -- ceph tell mds.1:3 damage ls 2024-05-27T07:28:24.439 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:28:24 smithi105 ceph-mon[35977]: pgmap v78: 129 pgs: 129 active+clean; 72 MiB data, 493 MiB used, 1.0 TiB / 1.0 TiB avail; 7.0 MiB/s rd, 5.5 MiB/s wr, 1.69k op/s 2024-05-27T07:28:24.704 INFO:teuthology.orchestra.run.smithi052.stderr:Inferring config /var/lib/ceph/73df2636-1bf8-11ef-bc9b-c7b262605968/mon.a/config 2024-05-27T07:28:25.205 INFO:tasks.workunit.client.0.smithi052.stdout: 25 549726 304 380751 222 336583 699 2024-05-27T07:28:26.229 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:28:26 smithi052 ceph-mon[31955]: pgmap v79: 129 pgs: 129 active+clean; 88 MiB data, 550 MiB used, 1.0 TiB / 1.0 TiB avail; 8.3 MiB/s rd, 7.4 MiB/s wr, 1.80k op/s 2024-05-27T07:28:26.351 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:28:26 smithi173 ceph-mon[37000]: pgmap v79: 129 pgs: 129 active+clean; 88 MiB data, 550 MiB used, 1.0 TiB / 1.0 TiB avail; 8.3 MiB/s rd, 7.4 MiB/s wr, 1.80k op/s 2024-05-27T07:28:26.439 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:28:26 smithi105 ceph-mon[35977]: pgmap v79: 129 pgs: 129 active+clean; 88 MiB data, 550 MiB used, 1.0 TiB / 1.0 TiB avail; 8.3 MiB/s rd, 7.4 MiB/s wr, 1.80k op/s 2024-05-27T07:28:28.351 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:28:28 smithi173 ceph-mon[37000]: pgmap v80: 129 pgs: 129 active+clean; 88 MiB data, 550 MiB used, 1.0 TiB / 1.0 TiB avail; 7.9 MiB/s rd, 7.0 MiB/s wr, 1.70k op/s 2024-05-27T07:28:28.352 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:28:28 smithi052 ceph-mon[31955]: pgmap v80: 129 pgs: 129 active+clean; 88 MiB data, 550 MiB used, 1.0 TiB / 1.0 TiB avail; 7.9 MiB/s rd, 7.0 MiB/s wr, 1.70k op/s 2024-05-27T07:28:28.439 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:28:28 smithi105 ceph-mon[35977]: pgmap v80: 129 pgs: 129 active+clean; 88 MiB data, 550 MiB used, 1.0 TiB / 1.0 TiB avail; 7.9 MiB/s rd, 7.0 MiB/s wr, 1.70k op/s 2024-05-27T07:28:28.968 INFO:journalctl@ceph.mds.f.smithi173.stdout:May 27 07:28:28 smithi173 systemd[1]: ceph-73df2636-1bf8-11ef-bc9b-c7b262605968@mds.f.service: Scheduled restart job, restart counter is at 2. 2024-05-27T07:28:28.968 INFO:journalctl@ceph.mds.f.smithi173.stdout:May 27 07:28:28 smithi173 systemd[1]: Stopped Ceph mds.f for 73df2636-1bf8-11ef-bc9b-c7b262605968. 2024-05-27T07:28:28.968 INFO:journalctl@ceph.mds.f.smithi173.stdout:May 27 07:28:28 smithi173 systemd[1]: ceph-73df2636-1bf8-11ef-bc9b-c7b262605968@mds.f.service: Consumed 1.411s CPU time. 2024-05-27T07:28:28.968 INFO:journalctl@ceph.mds.f.smithi173.stdout:May 27 07:28:28 smithi173 systemd[1]: Starting Ceph mds.f for 73df2636-1bf8-11ef-bc9b-c7b262605968... 2024-05-27T07:28:28.968 INFO:journalctl@ceph.mds.f.smithi173.stdout:May 27 07:28:28 smithi173 podman[72550]: 2024-05-27 07:28:28.742025528 +0000 UTC m=+0.019184212 image pull e36ea4fc8a5807e8d2701370e350a0de3856538797f7fba64cfce7e344bd6799 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6 2024-05-27T07:28:28.968 INFO:journalctl@ceph.mds.f.smithi173.stdout:May 27 07:28:28 smithi173 podman[72550]: 2024-05-27 07:28:28.852430495 +0000 UTC m=+0.129589176 container create 65507d1732d0cf11035e793b7475ea1c7ff3e6f304a3755d2917e9a6b5629e25 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-f, GIT_BRANCH=HEAD, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, org.label-schema.license=GPLv2, GIT_CLEAN=True, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.build-date=20240521, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=) 2024-05-27T07:28:29.045 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:29.052+0000 7fa2599c2700 1 -- 172.21.15.52:0/474295832 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fa254109090 msgr2=0x7fa2541094a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:28:29.045 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:29.052+0000 7fa2599c2700 1 --2- 172.21.15.52:0/474295832 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fa254109090 0x7fa2541094a0 secure :-1 s=READY pgs=574 cs=0 l=1 rev1=1 crypto rx=0x7fa24c00b0a0 tx=0x7fa24c00b3b0 comp rx=0 tx=0).stop 2024-05-27T07:28:29.049 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:29.054+0000 7fa2599c2700 1 -- 172.21.15.52:0/474295832 shutdown_connections 2024-05-27T07:28:29.050 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:29.054+0000 7fa2599c2700 1 --2- 172.21.15.52:0/474295832 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fa254102f20 0x7fa254107d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:28:29.050 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:29.054+0000 7fa2599c2700 1 --2- 172.21.15.52:0/474295832 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fa254109090 0x7fa2541094a0 unknown :-1 s=CLOSED pgs=574 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:28:29.050 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:29.054+0000 7fa2599c2700 1 --2- 172.21.15.52:0/474295832 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fa2541025d0 0x7fa2541029e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:28:29.050 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:29.054+0000 7fa2599c2700 1 -- 172.21.15.52:0/474295832 >> 172.21.15.52:0/474295832 conn(0x7fa2540783d0 msgr2=0x7fa254078810 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:28:29.053 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:29.061+0000 7fa2599c2700 1 -- 172.21.15.52:0/474295832 shutdown_connections 2024-05-27T07:28:29.053 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:29.063+0000 7fa2599c2700 1 -- 172.21.15.52:0/474295832 wait complete. 2024-05-27T07:28:29.058 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:29.068+0000 7fa2599c2700 1 Processor -- start 2024-05-27T07:28:29.062 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:29.071+0000 7fa2599c2700 1 -- start start 2024-05-27T07:28:29.062 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:29.071+0000 7fa2599c2700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fa2541025d0 0x7fa254117d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:28:29.062 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:29.072+0000 7fa2599c2700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fa254102f20 0x7fa254111e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:28:29.063 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:29.072+0000 7fa2599c2700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fa254109090 0x7fa2541123d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:28:29.063 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:29.072+0000 7fa2599c2700 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x7fa25410a600 con 0x7fa254102f20 2024-05-27T07:28:29.063 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:29.072+0000 7fa2599c2700 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7fa25410a480 con 0x7fa2541025d0 2024-05-27T07:28:29.063 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:29.072+0000 7fa2599c2700 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7fa25410a780 con 0x7fa254109090 2024-05-27T07:28:29.063 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:29.072+0000 7fa2527fc700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fa254102f20 0x7fa254111e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:28:29.063 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:29.072+0000 7fa2527fc700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fa254102f20 0x7fa254111e90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.52:33994/0 (socket says 172.21.15.52:33994) 2024-05-27T07:28:29.064 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:29.072+0000 7fa2527fc700 1 -- 172.21.15.52:0/281861882 learned_addr learned my addr 172.21.15.52:0/281861882 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-05-27T07:28:29.064 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:29.073+0000 7fa2527fc700 1 -- 172.21.15.52:0/281861882 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fa254109090 msgr2=0x7fa2541123d0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-05-27T07:28:29.064 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:29.073+0000 7fa2527fc700 1 --2- 172.21.15.52:0/281861882 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fa254109090 0x7fa2541123d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:28:29.064 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:29.073+0000 7fa2527fc700 1 -- 172.21.15.52:0/281861882 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fa2541025d0 msgr2=0x7fa254117d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:28:29.064 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:29.073+0000 7fa2527fc700 1 --2- 172.21.15.52:0/281861882 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fa2541025d0 0x7fa254117d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:28:29.064 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:29.073+0000 7fa2527fc700 1 -- 172.21.15.52:0/281861882 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa254112b10 con 0x7fa254102f20 2024-05-27T07:28:29.064 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:29.073+0000 7fa2527fc700 1 --2- 172.21.15.52:0/281861882 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fa254102f20 0x7fa254111e90 secure :-1 s=READY pgs=575 cs=0 l=1 rev1=1 crypto rx=0x7fa24800e940 tx=0x7fa24800ed00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:28:29.065 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:29.073+0000 7fa2589c0700 1 -- 172.21.15.52:0/281861882 <== mon.0 v2:172.21.15.52:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa24800cbd0 con 0x7fa254102f20 2024-05-27T07:28:29.065 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:29.074+0000 7fa2589c0700 1 -- 172.21.15.52:0/281861882 <== mon.0 v2:172.21.15.52:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fa24800cd70 con 0x7fa254102f20 2024-05-27T07:28:29.065 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:29.074+0000 7fa2589c0700 1 -- 172.21.15.52:0/281861882 <== mon.0 v2:172.21.15.52:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa248018680 con 0x7fa254102f20 2024-05-27T07:28:29.065 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:29.074+0000 7fa2599c2700 1 -- 172.21.15.52:0/281861882 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa2541b5e60 con 0x7fa254102f20 2024-05-27T07:28:29.065 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:29.074+0000 7fa2599c2700 1 -- 172.21.15.52:0/281861882 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa2541b6260 con 0x7fa254102f20 2024-05-27T07:28:29.081 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:29.085+0000 7fa2589c0700 1 -- 172.21.15.52:0/281861882 <== mon.0 v2:172.21.15.52:3300/0 4 ==== mgrmap(e 26) ==== 149783+0+0 (secure 0 0 0) 0x7fa248010430 con 0x7fa254102f20 2024-05-27T07:28:29.081 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:29.086+0000 7fa2589c0700 1 --2- 172.21.15.52:0/281861882 >> [v2:172.21.15.52:6800/1822318384,v1:172.21.15.52:6801/1822318384] conn(0x7fa23c0b2b20 0x7fa23c0b5030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:28:29.081 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:29.087+0000 7fa2589c0700 1 -- 172.21.15.52:0/281861882 <== mon.0 v2:172.21.15.52:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9379+0+0 (secure 0 0 0) 0x7fa2480e2c20 con 0x7fa254102f20 2024-05-27T07:28:29.081 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:29.090+0000 7fa2683d7b80 1 Processor -- start 2024-05-27T07:28:29.081 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:29.090+0000 7fa252ffd700 1 --2- 172.21.15.52:0/281861882 >> [v2:172.21.15.52:6800/1822318384,v1:172.21.15.52:6801/1822318384] conn(0x7fa23c0b2b20 0x7fa23c0b5030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:28:29.081 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:29.090+0000 7fa252ffd700 1 --2- 172.21.15.52:0/281861882 >> [v2:172.21.15.52:6800/1822318384,v1:172.21.15.52:6801/1822318384] conn(0x7fa23c0b2b20 0x7fa23c0b5030 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7fa24c00d010 tx=0x7fa24c00bab0 comp rx=0 tx=0).ready entity=mgr.15333 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:28:29.082 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:29.091+0000 7fa2683d7b80 1 -- start start 2024-05-27T07:28:29.084 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:29.091+0000 7fa2683d7b80 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x556695ca6e40 0x556695e24a40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:28:29.084 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:29.091+0000 7fa2683d7b80 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x556695e24f80 0x556695e25390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:28:29.084 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:29.091+0000 7fa2683d7b80 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x556695e2eb10 0x556695e30f30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:28:29.084 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:29.091+0000 7fa2683d7b80 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x556695d53c90 con 0x556695e2eb10 2024-05-27T07:28:29.085 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:29.091+0000 7fa2683d7b80 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x556695d43830 con 0x556695e24f80 2024-05-27T07:28:29.085 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:29.091+0000 7fa2683d7b80 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x556695d47870 con 0x556695ca6e40 2024-05-27T07:28:29.085 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:29.091+0000 7fa252ffd700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x556695ca6e40 0x556695e24a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:28:29.085 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:29.091+0000 7fa252ffd700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x556695ca6e40 0x556695e24a40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.173:3300/0 says I am v2:172.21.15.52:37424/0 (socket says 172.21.15.52:37424) 2024-05-27T07:28:29.085 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:29.091+0000 7fa252ffd700 1 -- 172.21.15.52:0/1684027691 learned_addr learned my addr 172.21.15.52:0/1684027691 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-05-27T07:28:29.085 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:29.092+0000 7fa2527fc700 1 --2- 172.21.15.52:0/1684027691 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x556695e24f80 0x556695e25390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:28:29.085 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:29.092+0000 7fa252ffd700 1 -- 172.21.15.52:0/1684027691 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x556695e24f80 msgr2=0x556695e25390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:28:29.085 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:29.092+0000 7fa252ffd700 1 --2- 172.21.15.52:0/1684027691 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x556695e24f80 0x556695e25390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:28:29.085 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:29.092+0000 7fa252ffd700 1 -- 172.21.15.52:0/1684027691 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x556695e2eb10 msgr2=0x556695e30f30 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-05-27T07:28:29.085 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:29.092+0000 7fa252ffd700 1 --2- 172.21.15.52:0/1684027691 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x556695e2eb10 0x556695e30f30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:28:29.085 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:29.092+0000 7fa252ffd700 1 -- 172.21.15.52:0/1684027691 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x556695c938d0 con 0x556695ca6e40 2024-05-27T07:28:29.085 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:29.092+0000 7fa252ffd700 1 --2- 172.21.15.52:0/1684027691 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x556695ca6e40 0x556695e24a40 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7fa25406ad20 tx=0x7fa24c01a000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=a2ca85beb0fdd8bd server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:28:29.086 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:29.093+0000 7fa2599c2700 1 -- 172.21.15.52:0/1684027691 <== mon.1 v2:172.21.15.173:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa24c02d070 con 0x556695ca6e40 2024-05-27T07:28:29.086 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:29.093+0000 7fa2599c2700 1 -- 172.21.15.52:0/1684027691 <== mon.1 v2:172.21.15.173:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fa24c024040 con 0x556695ca6e40 2024-05-27T07:28:29.086 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:29.096+0000 7fa2683d7b80 1 -- 172.21.15.52:0/1684027691 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x556695ca6e40 msgr2=0x556695e24a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:28:29.086 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:29.096+0000 7fa2683d7b80 1 --2- 172.21.15.52:0/1684027691 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x556695ca6e40 0x556695e24a40 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7fa25406ad20 tx=0x7fa24c01a000 comp rx=0 tx=0).stop 2024-05-27T07:28:29.090 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:29.096+0000 7fa2683d7b80 1 -- 172.21.15.52:0/1684027691 shutdown_connections 2024-05-27T07:28:29.090 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:29.096+0000 7fa2683d7b80 1 --2- 172.21.15.52:0/1684027691 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x556695e2eb10 0x556695e30f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:28:29.090 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:29.096+0000 7fa2683d7b80 1 --2- 172.21.15.52:0/1684027691 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x556695ca6e40 0x556695e24a40 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:28:29.090 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:29.096+0000 7fa2683d7b80 1 --2- 172.21.15.52:0/1684027691 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x556695e24f80 0x556695e25390 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:28:29.090 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:29.096+0000 7fa2683d7b80 1 -- 172.21.15.52:0/1684027691 >> 172.21.15.52:0/1684027691 conn(0x556695d01af0 msgr2=0x556695e1fda0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:28:29.092 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:29.101+0000 7fa2683d7b80 1 -- 172.21.15.52:0/1684027691 shutdown_connections 2024-05-27T07:28:29.093 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:29.101+0000 7fa2683d7b80 1 -- 172.21.15.52:0/1684027691 wait complete. 2024-05-27T07:28:29.093 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:29.102+0000 7fa2683d7b80 1 -- start start 2024-05-27T07:28:29.094 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:29.103+0000 7fa2683d7b80 1 Processor -- start 2024-05-27T07:28:29.095 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:29.103+0000 7fa2683d7b80 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x556695ca6e40 0x556695e2bac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:28:29.095 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:29.103+0000 7fa2683d7b80 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x556695e24f80 0x556695e2c000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:28:29.095 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:29.104+0000 7fa252ffd700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x556695ca6e40 0x556695e2bac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:28:29.095 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:29.104+0000 7fa252ffd700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x556695ca6e40 0x556695e2bac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.173:3300/0 says I am v2:172.21.15.52:37432/0 (socket says 172.21.15.52:37432) 2024-05-27T07:28:29.095 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:29.104+0000 7fa252ffd700 1 -- 172.21.15.52:0/3242229857 learned_addr learned my addr 172.21.15.52:0/3242229857 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-05-27T07:28:29.095 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:29.104+0000 7fa2527fc700 1 --2- 172.21.15.52:0/3242229857 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x556695e24f80 0x556695e2c000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:28:29.096 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:29.104+0000 7fa2683d7b80 1 --2- 172.21.15.52:0/3242229857 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x556695e2eb10 0x556695e2c540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:28:29.096 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:29.104+0000 7fa2683d7b80 1 -- 172.21.15.52:0/3242229857 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x556695c49fa0 con 0x556695e24f80 2024-05-27T07:28:29.096 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:29.104+0000 7fa2683d7b80 1 -- 172.21.15.52:0/3242229857 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x556695ce57c0 con 0x556695e2eb10 2024-05-27T07:28:29.096 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:29.104+0000 7fa2683d7b80 1 -- 172.21.15.52:0/3242229857 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x556695e42950 con 0x556695ca6e40 2024-05-27T07:28:29.096 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:29.104+0000 7fa2683d7b80 10 client.0 _ll_register_callbacks cb 0x556695cde920 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T07:28:29.099 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:29.108+0000 7fa2527fc700 1 -- 172.21.15.52:0/3242229857 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x556695ca6e40 msgr2=0x556695e2bac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:28:29.099 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:29.108+0000 7fa2527fc700 1 --2- 172.21.15.52:0/3242229857 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x556695ca6e40 0x556695e2bac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:28:29.100 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:29.108+0000 7fa2527fc700 1 -- 172.21.15.52:0/3242229857 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x556695e2eb10 msgr2=0x556695e2c540 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-05-27T07:28:29.100 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:29.108+0000 7fa2527fc700 1 --2- 172.21.15.52:0/3242229857 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x556695e2eb10 0x556695e2c540 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:28:29.100 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:29.108+0000 7fa2527fc700 1 -- 172.21.15.52:0/3242229857 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({osdmap=0}) -- 0x556695ce2690 con 0x556695e24f80 2024-05-27T07:28:29.100 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:29.108+0000 7fa2527fc700 1 --2- 172.21.15.52:0/3242229857 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x556695e24f80 0x556695e2c000 secure :-1 s=READY pgs=576 cs=0 l=1 rev1=1 crypto rx=0x7fa2480dffb0 tx=0x7fa2480e0510 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3cb428e43d01811e server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:28:29.100 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:29.108+0000 7fa232ffd700 10 client.0 ms_handle_connect on v2:172.21.15.52:3300/0 2024-05-27T07:28:29.100 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:29.108+0000 7fa232ffd700 1 -- 172.21.15.52:0/3242229857 <== mon.0 v2:172.21.15.52:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa2480a9b40 con 0x556695e24f80 2024-05-27T07:28:29.100 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:29.109+0000 7fa2683d7b80 1 -- 172.21.15.52:0/3242229857 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fa24c024040 con 0x556695e24f80 2024-05-27T07:28:29.100 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:29.109+0000 7fa232ffd700 1 -- 172.21.15.52:0/3242229857 <== mon.0 v2:172.21.15.52:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9379+0+0 (secure 0 0 0) 0x7fa2480b2720 con 0x556695e24f80 2024-05-27T07:28:29.100 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:29.109+0000 7fa232ffd700 1 -- 172.21.15.52:0/3242229857 <== mon.0 v2:172.21.15.52:3300/0 3 ==== mon_get_version_reply(handle=1 version=36) ==== 24+0+0 (secure 0 0 0) 0x7fa2480adef0 con 0x556695e24f80 2024-05-27T07:28:29.101 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:29.109+0000 7fa2683d7b80 10 client.16053 fetch_fsmap learned FSMap version 36 2024-05-27T07:28:29.101 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:29.109+0000 7fa2683d7b80 1 -- 172.21.15.52:0/3242229857 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({config=0+,fsmap=36,monmap=4+}) -- 0x556695e4d2e0 con 0x556695e24f80 2024-05-27T07:28:29.102 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:29.111+0000 7fa232ffd700 1 -- 172.21.15.52:0/3242229857 <== mon.0 v2:172.21.15.52:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fa2480b29a0 con 0x556695e24f80 2024-05-27T07:28:29.102 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:29.111+0000 7fa232ffd700 1 -- 172.21.15.52:0/3242229857 <== mon.0 v2:172.21.15.52:3300/0 5 ==== fsmap(e 36) ==== 8374+0+0 (secure 0 0 0) 0x7fa2480a55d0 con 0x556695e24f80 2024-05-27T07:28:29.104 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:29.113+0000 7fa2683d7b80 10 client.16053 fetch_fsmap finished waiting for FSMap version 36 2024-05-27T07:28:29.104 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:29.113+0000 7fa2683d7b80 10 client.16053 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.e 2024-05-27T07:28:29.106 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:29.113+0000 7fa2683d7b80 20 client.16053 populate_metadata read hostname 'smithi052' 2024-05-27T07:28:29.106 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:29.113+0000 7fa2683d7b80 1 --2- 172.21.15.52:0/3242229857 >> [v2:172.21.15.105:6834/3298498384,v1:172.21.15.105:6835/3298498384] conn(0x556695e4fcd0 0x556695e52160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:28:29.106 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:29.113+0000 7fa2683d7b80 4 client.16053 mds_command: new command op to 24473 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-27T07:28:29.107 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:29.113+0000 7fa2683d7b80 1 -- 172.21.15.52:0/3242229857 --> [v2:172.21.15.105:6834/3298498384,v1:172.21.15.105:6835/3298498384] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x7fa24c024040 con 0x556695e4fcd0 2024-05-27T07:28:29.107 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:29.113+0000 7fa252ffd700 1 --2- 172.21.15.52:0/3242229857 >> [v2:172.21.15.105:6834/3298498384,v1:172.21.15.105:6835/3298498384] conn(0x556695e4fcd0 0x556695e52160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:28:29.107 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:29.116+0000 7fa252ffd700 1 --2- 172.21.15.52:0/3242229857 >> [v2:172.21.15.105:6834/3298498384,v1:172.21.15.105:6835/3298498384] conn(0x556695e4fcd0 0x556695e52160 crc :-1 s=READY pgs=35 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=305425f307bf061d server_cookie=d13e51375f98134 in_seq=0 out_seq=0 2024-05-27T07:28:29.107 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:29.117+0000 7fa232ffd700 10 client.16053 ms_handle_connect on v2:172.21.15.105:6834/3298498384 2024-05-27T07:28:29.107 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:29.117+0000 7fa232ffd700 1 -- 172.21.15.52:0/3242229857 <== mds.3 v2:172.21.15.105:6834/3298498384 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24041 (crc 0 0 0) 0x7fa24c024040 con 0x556695e4fcd0 2024-05-27T07:28:29.108 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:29.117+0000 7fa232ffd700 10 client.16053 handle_command_reply: tid=0 2024-05-27T07:28:29.108 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:29.117+0000 7fa2683d7b80 1 client.16053 shutdown 2024-05-27T07:28:29.109 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:29.118+0000 7fa2683d7b80 1 -- 172.21.15.52:0/3242229857 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x556695e24f80 msgr2=0x556695e2c000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:28:29.109 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:29.118+0000 7fa2683d7b80 1 --2- 172.21.15.52:0/3242229857 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x556695e24f80 0x556695e2c000 secure :-1 s=READY pgs=576 cs=0 l=1 rev1=1 crypto rx=0x7fa2480dffb0 tx=0x7fa2480e0510 comp rx=0 tx=0).stop 2024-05-27T07:28:29.115 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:29.120+0000 7fa2683d7b80 1 -- 172.21.15.52:0/3242229857 shutdown_connections 2024-05-27T07:28:29.115 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:29.120+0000 7fa2683d7b80 1 --2- 172.21.15.52:0/3242229857 >> [v2:172.21.15.105:6834/3298498384,v1:172.21.15.105:6835/3298498384] conn(0x556695e4fcd0 0x556695e52160 crc :-1 s=READY pgs=35 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:28:29.115 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:29.120+0000 7fa2683d7b80 1 --2- 172.21.15.52:0/3242229857 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x556695e2eb10 0x556695e2c540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:28:29.115 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:29.120+0000 7fa2683d7b80 1 --2- 172.21.15.52:0/3242229857 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x556695ca6e40 0x556695e2bac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:28:29.115 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:29.120+0000 7fa2683d7b80 1 --2- 172.21.15.52:0/3242229857 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x556695e24f80 0x556695e2c000 unknown :-1 s=CLOSED pgs=576 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:28:29.116 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:29.120+0000 7fa2683d7b80 1 -- 172.21.15.52:0/3242229857 >> 172.21.15.52:0/3242229857 conn(0x556695d01af0 msgr2=0x556695e21a20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:28:29.116 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:29.120+0000 7fa232ffd700 0 client.16053 ms_handle_reset on v2:172.21.15.105:6834/3298498384 2024-05-27T07:28:29.116 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:29.122+0000 7fa2683d7b80 1 -- 172.21.15.52:0/3242229857 shutdown_connections 2024-05-27T07:28:29.116 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:29.122+0000 7fa2683d7b80 1 -- 172.21.15.52:0/3242229857 wait complete. 2024-05-27T07:28:29.116 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:29.122+0000 7fa2683d7b80 20 client.16053 trim_cache size 0 max 16384 2024-05-27T07:28:29.163 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:29.172+0000 7fa2683d7b80 1 Processor -- start 2024-05-27T07:28:29.170 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:29.177+0000 7fa2683d7b80 1 -- start start 2024-05-27T07:28:29.171 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:29.177+0000 7fa2683d7b80 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x556695ca6e40 0x556695e24a40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:28:29.171 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:29.177+0000 7fa2683d7b80 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x556695e24f80 0x556695e2ade0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:28:29.171 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:29.177+0000 7fa2683d7b80 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x556695e2eb10 0x556695e2df60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:28:29.171 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:29.177+0000 7fa2683d7b80 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x7fa2541b68f0 con 0x556695e24f80 2024-05-27T07:28:29.171 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:29.177+0000 7fa2683d7b80 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7fa2541b84c0 con 0x556695ca6e40 2024-05-27T07:28:29.171 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:29.177+0000 7fa2683d7b80 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7fa2541b6fc0 con 0x556695e2eb10 2024-05-27T07:28:29.171 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:29.178+0000 7fa2527fc700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x556695e24f80 0x556695e2ade0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:28:29.171 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:29.178+0000 7fa2527fc700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x556695e24f80 0x556695e2ade0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.52:34010/0 (socket says 172.21.15.52:34010) 2024-05-27T07:28:29.171 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:29.178+0000 7fa2527fc700 1 -- 172.21.15.52:0/1420727173 learned_addr learned my addr 172.21.15.52:0/1420727173 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-05-27T07:28:29.171 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:29.179+0000 7fa2527fc700 1 -- 172.21.15.52:0/1420727173 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x556695e2eb10 msgr2=0x556695e2df60 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-05-27T07:28:29.171 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:29.179+0000 7fa2527fc700 1 --2- 172.21.15.52:0/1420727173 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x556695e2eb10 0x556695e2df60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:28:29.171 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:29.179+0000 7fa2527fc700 1 -- 172.21.15.52:0/1420727173 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x556695ca6e40 msgr2=0x556695e24a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:28:29.172 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:29.179+0000 7fa2527fc700 1 --2- 172.21.15.52:0/1420727173 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x556695ca6e40 0x556695e24a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:28:29.172 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:29.179+0000 7fa2527fc700 1 -- 172.21.15.52:0/1420727173 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x556695e29ff0 con 0x556695e24f80 2024-05-27T07:28:29.172 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:29.179+0000 7fa2527fc700 1 --2- 172.21.15.52:0/1420727173 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x556695e24f80 0x556695e2ade0 secure :-1 s=READY pgs=577 cs=0 l=1 rev1=1 crypto rx=0x7fa2480a5bc0 tx=0x7fa248010d20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=343f0b705d1498e9 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:28:29.172 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:29.181+0000 7fa2397fa700 1 -- 172.21.15.52:0/1420727173 <== mon.0 v2:172.21.15.52:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa2480b29a0 con 0x556695e24f80 2024-05-27T07:28:29.172 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:29.181+0000 7fa2397fa700 1 -- 172.21.15.52:0/1420727173 <== mon.0 v2:172.21.15.52:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fa2480a9b40 con 0x556695e24f80 2024-05-27T07:28:29.172 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:29.181+0000 7fa2397fa700 1 -- 172.21.15.52:0/1420727173 <== mon.0 v2:172.21.15.52:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa2480a5710 con 0x556695e24f80 2024-05-27T07:28:29.176 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:29.185+0000 7fa2683d7b80 1 -- 172.21.15.52:0/1420727173 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x556695e24f80 msgr2=0x556695e2ade0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:28:29.177 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:29.185+0000 7fa2683d7b80 1 --2- 172.21.15.52:0/1420727173 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x556695e24f80 0x556695e2ade0 secure :-1 s=READY pgs=577 cs=0 l=1 rev1=1 crypto rx=0x7fa2480a5bc0 tx=0x7fa248010d20 comp rx=0 tx=0).stop 2024-05-27T07:28:29.187 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:29.194+0000 7fa2683d7b80 1 -- 172.21.15.52:0/1420727173 shutdown_connections 2024-05-27T07:28:29.187 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:29.194+0000 7fa2683d7b80 1 --2- 172.21.15.52:0/1420727173 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x556695e2eb10 0x556695e2df60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:28:29.187 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:29.194+0000 7fa2683d7b80 1 --2- 172.21.15.52:0/1420727173 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x556695ca6e40 0x556695e24a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:28:29.187 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:29.194+0000 7fa2683d7b80 1 --2- 172.21.15.52:0/1420727173 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x556695e24f80 0x556695e2ade0 unknown :-1 s=CLOSED pgs=577 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:28:29.187 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:29.194+0000 7fa2683d7b80 1 -- 172.21.15.52:0/1420727173 >> 172.21.15.52:0/1420727173 conn(0x556695d01af0 msgr2=0x556695e270e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:28:29.197 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:29.203+0000 7fa2683d7b80 1 -- 172.21.15.52:0/1420727173 shutdown_connections 2024-05-27T07:28:29.198 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:29.203+0000 7fa2683d7b80 1 -- 172.21.15.52:0/1420727173 wait complete. 2024-05-27T07:28:29.198 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:29.203+0000 7fa2683d7b80 1 -- start start 2024-05-27T07:28:29.198 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:29.204+0000 7fa2683d7b80 1 Processor -- start 2024-05-27T07:28:29.198 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:29.204+0000 7fa2683d7b80 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x556695ca6e40 0x556695e42c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:28:29.198 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:29.205+0000 7fa252ffd700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x556695ca6e40 0x556695e42c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:28:29.198 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:29.205+0000 7fa252ffd700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x556695ca6e40 0x556695e42c90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.52:34018/0 (socket says 172.21.15.52:34018) 2024-05-27T07:28:29.198 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:29.205+0000 7fa252ffd700 1 -- 172.21.15.52:0/1610291677 learned_addr learned my addr 172.21.15.52:0/1610291677 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-05-27T07:28:29.198 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:29.205+0000 7fa2683d7b80 1 --2- 172.21.15.52:0/1610291677 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x556695e24f80 0x556695e431d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:28:29.198 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:29.205+0000 7fa2683d7b80 1 --2- 172.21.15.52:0/1610291677 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x556695e2eb10 0x556695eb8820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:28:29.198 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:29.205+0000 7fa2683d7b80 1 -- 172.21.15.52:0/1610291677 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x7fa2541b7df0 con 0x556695ca6e40 2024-05-27T07:28:29.198 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:29.205+0000 7fa2683d7b80 1 -- 172.21.15.52:0/1610291677 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7fa2541b7690 con 0x556695e2eb10 2024-05-27T07:28:29.198 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:29.205+0000 7fa2683d7b80 1 -- 172.21.15.52:0/1610291677 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7fa254073de0 con 0x556695e24f80 2024-05-27T07:28:29.199 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:29.206+0000 7fa2527fc700 1 --2- 172.21.15.52:0/1610291677 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x556695e24f80 0x556695e431d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:28:29.199 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:29.206+0000 7fa252ffd700 1 -- 172.21.15.52:0/1610291677 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x556695e24f80 msgr2=0x556695e431d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:28:29.199 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:29.206+0000 7fa252ffd700 1 --2- 172.21.15.52:0/1610291677 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x556695e24f80 0x556695e431d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:28:29.199 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:29.206+0000 7fa252ffd700 1 -- 172.21.15.52:0/1610291677 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x556695e2eb10 msgr2=0x556695eb8820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:28:29.199 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:29.206+0000 7fa2537fe700 1 --2- 172.21.15.52:0/1610291677 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x556695e2eb10 0x556695eb8820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:28:29.199 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:29.206+0000 7fa252ffd700 1 --2- 172.21.15.52:0/1610291677 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x556695e2eb10 0x556695eb8820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:28:29.199 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:29.206+0000 7fa252ffd700 1 -- 172.21.15.52:0/1610291677 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({osdmap=0}) -- 0x556695e43a80 con 0x556695ca6e40 2024-05-27T07:28:29.199 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:29.206+0000 7fa2683d7b80 10 client.0 _ll_register_callbacks cb 0x556695cde920 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T07:28:29.199 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:29.206+0000 7fa2683d7b80 1 -- 172.21.15.52:0/1610291677 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fa2480a9b40 con 0x556695ca6e40 2024-05-27T07:28:29.199 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:29.206+0000 7fa252ffd700 1 --2- 172.21.15.52:0/1610291677 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x556695ca6e40 0x556695e42c90 secure :-1 s=READY pgs=578 cs=0 l=1 rev1=1 crypto rx=0x7fa24c009610 tx=0x7fa24c01dca0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=885c6d75b6bd1be1 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:28:29.199 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:29.207+0000 7fa232ffd700 10 client.16062 ms_handle_connect on v2:172.21.15.52:3300/0 2024-05-27T07:28:29.199 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:29.207+0000 7fa232ffd700 1 -- 172.21.15.52:0/1610291677 <== mon.0 v2:172.21.15.52:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa24c02d040 con 0x556695ca6e40 2024-05-27T07:28:29.200 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:29.207+0000 7fa232ffd700 1 -- 172.21.15.52:0/1610291677 <== mon.0 v2:172.21.15.52:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9379+0+0 (secure 0 0 0) 0x7fa24c0075c0 con 0x556695ca6e40 2024-05-27T07:28:29.200 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:29.208+0000 7fa232ffd700 1 -- 172.21.15.52:0/1610291677 <== mon.0 v2:172.21.15.52:3300/0 3 ==== mon_get_version_reply(handle=1 version=36) ==== 24+0+0 (secure 0 0 0) 0x7fa24c043050 con 0x556695ca6e40 2024-05-27T07:28:29.200 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:29.208+0000 7fa2683d7b80 10 client.16062 fetch_fsmap learned FSMap version 36 2024-05-27T07:28:29.200 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:29.208+0000 7fa2683d7b80 1 -- 172.21.15.52:0/1610291677 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({fsmap=36}) -- 0x556695eb9b70 con 0x556695ca6e40 2024-05-27T07:28:29.201 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:29.209+0000 7fa232ffd700 1 -- 172.21.15.52:0/1610291677 <== mon.0 v2:172.21.15.52:3300/0 4 ==== fsmap(e 36) ==== 8374+0+0 (secure 0 0 0) 0x7fa24c011070 con 0x556695ca6e40 2024-05-27T07:28:29.201 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:29.209+0000 7fa2683d7b80 10 client.16062 fetch_fsmap finished waiting for FSMap version 36 2024-05-27T07:28:29.201 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:29.209+0000 7fa2683d7b80 10 client.16062 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.e 2024-05-27T07:28:29.201 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:29.209+0000 7fa2683d7b80 20 client.16062 populate_metadata read hostname 'smithi052' 2024-05-27T07:28:29.201 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:29.209+0000 7fa2683d7b80 1 --2- 172.21.15.52:0/1610291677 >> [v2:172.21.15.105:6834/3298498384,v1:172.21.15.105:6835/3298498384] conn(0x556695eb9da0 0x556695ebc1c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:28:29.201 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:29.209+0000 7fa2683d7b80 4 client.16062 mds_command: new command op to 24473 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-05-27T07:28:29.201 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:29.209+0000 7fa2683d7b80 1 -- 172.21.15.52:0/1610291677 --> [v2:172.21.15.105:6834/3298498384,v1:172.21.15.105:6835/3298498384] -- command(tid 0: {"prefix": "damage ls"}) -- 0x7fa2480a9b40 con 0x556695eb9da0 2024-05-27T07:28:29.201 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:29.210+0000 7fa2527fc700 1 --2- 172.21.15.52:0/1610291677 >> [v2:172.21.15.105:6834/3298498384,v1:172.21.15.105:6835/3298498384] conn(0x556695eb9da0 0x556695ebc1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:28:29.205 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:29.214+0000 7fa2527fc700 1 --2- 172.21.15.52:0/1610291677 >> [v2:172.21.15.105:6834/3298498384,v1:172.21.15.105:6835/3298498384] conn(0x556695eb9da0 0x556695ebc1c0 crc :-1 s=READY pgs=36 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=cabdb6742f3779fb server_cookie=7ef44dd9b676d97c in_seq=0 out_seq=0 2024-05-27T07:28:29.205 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:29.214+0000 7fa232ffd700 10 client.16062 ms_handle_connect on v2:172.21.15.105:6834/3298498384 2024-05-27T07:28:29.205 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:29.214+0000 7fa232ffd700 1 -- 172.21.15.52:0/1610291677 <== mds.3 v2:172.21.15.105:6834/3298498384 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x7fa2480a9b40 con 0x556695eb9da0 2024-05-27T07:28:29.205 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:29.214+0000 7fa232ffd700 10 client.16062 handle_command_reply: tid=0 2024-05-27T07:28:29.205 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:29.214+0000 7fa2683d7b80 1 client.16062 shutdown 2024-05-27T07:28:29.206 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:29.216+0000 7fa2683d7b80 1 -- 172.21.15.52:0/1610291677 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x556695ca6e40 msgr2=0x556695e42c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:28:29.206 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:29.216+0000 7fa2683d7b80 1 --2- 172.21.15.52:0/1610291677 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x556695ca6e40 0x556695e42c90 secure :-1 s=READY pgs=578 cs=0 l=1 rev1=1 crypto rx=0x7fa24c009610 tx=0x7fa24c01dca0 comp rx=0 tx=0).stop 2024-05-27T07:28:29.207 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:29.216+0000 7fa2683d7b80 1 -- 172.21.15.52:0/1610291677 shutdown_connections 2024-05-27T07:28:29.207 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:29.216+0000 7fa2683d7b80 1 --2- 172.21.15.52:0/1610291677 >> [v2:172.21.15.105:6834/3298498384,v1:172.21.15.105:6835/3298498384] conn(0x556695eb9da0 0x556695ebc1c0 crc :-1 s=READY pgs=36 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:28:29.207 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:29.216+0000 7fa2683d7b80 1 --2- 172.21.15.52:0/1610291677 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x556695e2eb10 0x556695eb8820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:28:29.207 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:29.216+0000 7fa2683d7b80 1 --2- 172.21.15.52:0/1610291677 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x556695ca6e40 0x556695e42c90 unknown :-1 s=CLOSED pgs=578 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:28:29.207 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:29.216+0000 7fa2683d7b80 1 --2- 172.21.15.52:0/1610291677 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x556695e24f80 0x556695e431d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:28:29.207 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:29.216+0000 7fa2683d7b80 1 -- 172.21.15.52:0/1610291677 >> 172.21.15.52:0/1610291677 conn(0x556695d01af0 msgr2=0x556695e19650 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:28:29.207 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:29.216+0000 7fa232ffd700 0 client.16062 ms_handle_reset on v2:172.21.15.105:6834/3298498384 2024-05-27T07:28:29.207 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:29.216+0000 7fa2683d7b80 1 -- 172.21.15.52:0/1610291677 shutdown_connections 2024-05-27T07:28:29.208 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:29.216+0000 7fa2683d7b80 1 -- 172.21.15.52:0/1610291677 wait complete. 2024-05-27T07:28:29.208 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:29.216+0000 7fa2683d7b80 20 client.16062 trim_cache size 0 max 16384 2024-05-27T07:28:29.208 INFO:teuthology.orchestra.run.smithi052.stdout:[] 2024-05-27T07:28:29.209 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:29.218+0000 7fa23a7fc700 1 -- 172.21.15.52:0/281861882 >> [v2:172.21.15.52:6800/1822318384,v1:172.21.15.52:6801/1822318384] conn(0x7fa23c0b2b20 msgr2=0x7fa23c0b5030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:28:29.209 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:29.218+0000 7fa23a7fc700 1 --2- 172.21.15.52:0/281861882 >> [v2:172.21.15.52:6800/1822318384,v1:172.21.15.52:6801/1822318384] conn(0x7fa23c0b2b20 0x7fa23c0b5030 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7fa24c00d010 tx=0x7fa24c00bab0 comp rx=0 tx=0).stop 2024-05-27T07:28:29.209 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:29.218+0000 7fa23a7fc700 1 -- 172.21.15.52:0/281861882 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fa254102f20 msgr2=0x7fa254111e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:28:29.209 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:29.218+0000 7fa23a7fc700 1 --2- 172.21.15.52:0/281861882 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fa254102f20 0x7fa254111e90 secure :-1 s=READY pgs=575 cs=0 l=1 rev1=1 crypto rx=0x7fa24800e940 tx=0x7fa24800ed00 comp rx=0 tx=0).stop 2024-05-27T07:28:29.210 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:29.219+0000 7fa23a7fc700 1 -- 172.21.15.52:0/281861882 shutdown_connections 2024-05-27T07:28:29.210 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:29.219+0000 7fa23a7fc700 1 --2- 172.21.15.52:0/281861882 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fa254109090 0x7fa2541123d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:28:29.210 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:29.219+0000 7fa23a7fc700 1 --2- 172.21.15.52:0/281861882 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fa2541025d0 0x7fa254117d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:28:29.210 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:29.219+0000 7fa23a7fc700 1 --2- 172.21.15.52:0/281861882 >> [v2:172.21.15.52:6800/1822318384,v1:172.21.15.52:6801/1822318384] conn(0x7fa23c0b2b20 0x7fa23c0b5030 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:28:29.210 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:29.219+0000 7fa23a7fc700 1 --2- 172.21.15.52:0/281861882 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fa254102f20 0x7fa254111e90 unknown :-1 s=CLOSED pgs=575 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:28:29.210 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:29.219+0000 7fa23a7fc700 1 -- 172.21.15.52:0/281861882 >> 172.21.15.52:0/281861882 conn(0x7fa2540783d0 msgr2=0x7fa254105d90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:28:29.211 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:29.220+0000 7fa23a7fc700 1 -- 172.21.15.52:0/281861882 shutdown_connections 2024-05-27T07:28:29.211 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:29.220+0000 7fa23a7fc700 1 -- 172.21.15.52:0/281861882 wait complete. 2024-05-27T07:28:29.352 INFO:journalctl@ceph.mds.f.smithi173.stdout:May 27 07:28:29 smithi173 podman[72550]: 2024-05-27 07:28:29.097031276 +0000 UTC m=+0.374189956 container init 65507d1732d0cf11035e793b7475ea1c7ff3e6f304a3755d2917e9a6b5629e25 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-f, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, io.buildah.version=1.33.5, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240521) 2024-05-27T07:28:29.352 INFO:journalctl@ceph.mds.f.smithi173.stdout:May 27 07:28:29 smithi173 podman[72550]: 2024-05-27 07:28:29.099764239 +0000 UTC m=+0.376922916 container start 65507d1732d0cf11035e793b7475ea1c7ff3e6f304a3755d2917e9a6b5629e25 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-f, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240521, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0) 2024-05-27T07:28:29.352 INFO:journalctl@ceph.mds.f.smithi173.stdout:May 27 07:28:29 smithi173 ceph-mds[72568]: -- 172.21.15.173:0/99101145 <== mon.1 v2:172.21.15.173:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55a87d755a40 con 0x55a87d79d600 2024-05-27T07:28:29.352 INFO:journalctl@ceph.mds.f.smithi173.stdout:May 27 07:28:29 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-f[72564]: starting mds.f at 2024-05-27T07:28:29.352 INFO:journalctl@ceph.mds.f.smithi173.stdout:May 27 07:28:29 smithi173 bash[72550]: 65507d1732d0cf11035e793b7475ea1c7ff3e6f304a3755d2917e9a6b5629e25 2024-05-27T07:28:29.352 INFO:journalctl@ceph.mds.f.smithi173.stdout:May 27 07:28:29 smithi173 systemd[1]: Started Ceph mds.f for 73df2636-1bf8-11ef-bc9b-c7b262605968. 2024-05-27T07:28:30.298 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:28:30 smithi173 ceph-mon[37000]: pgmap v81: 129 pgs: 129 active+clean; 96 MiB data, 556 MiB used, 1.0 TiB / 1.0 TiB avail; 5.9 MiB/s rd, 3.8 MiB/s wr, 636 op/s 2024-05-27T07:28:30.298 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:28:30 smithi173 ceph-mon[37000]: daemon mds.f restarted 2024-05-27T07:28:30.299 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:28:30 smithi173 ceph-mon[37000]: osdmap e94: 12 total, 12 up, 12 in 2024-05-27T07:28:30.299 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:28:30 smithi173 ceph-mon[37000]: mds.? [v2:172.21.15.173:6834/687320882,v1:172.21.15.173:6835/687320882] up:boot 2024-05-27T07:28:30.299 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:28:30 smithi173 ceph-mon[37000]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-05-27T07:28:30.299 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:28:30 smithi173 ceph-mon[37000]: from='mgr.15333 172.21.15.52:0/416389082' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2024-05-27T07:28:30.299 INFO:journalctl@ceph.mds.f.smithi173.stdout:May 27 07:28:29 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-f[72564]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: In function 'std::set SnapClient::filter(const std::set&) const' thread 7f418bd1d700 time 2024-05-27T07:28:29.963064+0000 2024-05-27T07:28:30.299 INFO:journalctl@ceph.mds.f.smithi173.stdout:May 27 07:28:29 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-f[72564]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: 197: FAILED ceph_assert(cached_version > 0) 2024-05-27T07:28:30.300 INFO:journalctl@ceph.mds.f.smithi173.stdout:May 27 07:28:29 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-f[72564]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:28:30.300 INFO:journalctl@ceph.mds.f.smithi173.stdout:May 27 07:28:29 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-f[72564]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x127) [0x7f419e562ba6] 2024-05-27T07:28:30.300 INFO:journalctl@ceph.mds.f.smithi173.stdout:May 27 07:28:29 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-f[72564]: 2: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7f419e562dd1] 2024-05-27T07:28:30.300 INFO:journalctl@ceph.mds.f.smithi173.stdout:May 27 07:28:29 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-f[72564]: 3: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x55a87af6523d] 2024-05-27T07:28:30.300 INFO:journalctl@ceph.mds.f.smithi173.stdout:May 27 07:28:29 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-f[72564]: 4: (SnapRealm::build_snap_set() const+0x323) [0x55a87af50971] 2024-05-27T07:28:30.300 INFO:journalctl@ceph.mds.f.smithi173.stdout:May 27 07:28:29 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-f[72564]: 5: (SnapRealm::check_cache() const+0x13e) [0x55a87af4efa0] 2024-05-27T07:28:30.300 INFO:journalctl@ceph.mds.f.smithi173.stdout:May 27 07:28:29 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-f[72564]: 6: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x55a87af4f651] 2024-05-27T07:28:30.300 INFO:journalctl@ceph.mds.f.smithi173.stdout:May 27 07:28:29 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-f[72564]: 7: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x55a87aee4347] 2024-05-27T07:28:30.300 INFO:journalctl@ceph.mds.f.smithi173.stdout:May 27 07:28:29 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-f[72564]: 8: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x55a87abe087e] 2024-05-27T07:28:30.300 INFO:journalctl@ceph.mds.f.smithi173.stdout:May 27 07:28:29 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-f[72564]: 9: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x279d) [0x55a87abe5f2f] 2024-05-27T07:28:30.300 INFO:journalctl@ceph.mds.f.smithi173.stdout:May 27 07:28:29 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-f[72564]: 10: (EUpdate::replay(MDSRank*)+0x63) [0x55a87abf0969] 2024-05-27T07:28:30.300 INFO:journalctl@ceph.mds.f.smithi173.stdout:May 27 07:28:29 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-f[72564]: 11: (MDLog::_replay_thread()+0x1f38) [0x55a87af8948c] 2024-05-27T07:28:30.301 INFO:journalctl@ceph.mds.f.smithi173.stdout:May 27 07:28:29 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-f[72564]: 12: (MDLog::ReplayThread::entry()+0x11) [0x55a87af929cb] 2024-05-27T07:28:30.301 INFO:journalctl@ceph.mds.f.smithi173.stdout:May 27 07:28:29 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-f[72564]: 13: (Thread::entry_wrapper()+0x43) [0x7f419e53172d] 2024-05-27T07:28:30.301 INFO:journalctl@ceph.mds.f.smithi173.stdout:May 27 07:28:29 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-f[72564]: 14: (Thread::_entry_func(void*)+0xd) [0x7f419e531749] 2024-05-27T07:28:30.301 INFO:journalctl@ceph.mds.f.smithi173.stdout:May 27 07:28:29 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-f[72564]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7f419d0d51ca] 2024-05-27T07:28:30.301 INFO:journalctl@ceph.mds.f.smithi173.stdout:May 27 07:28:29 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-f[72564]: 16: clone() 2024-05-27T07:28:30.301 INFO:journalctl@ceph.mds.f.smithi173.stdout:May 27 07:28:29 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-f[72564]: *** Caught signal (Aborted) ** 2024-05-27T07:28:30.301 INFO:journalctl@ceph.mds.f.smithi173.stdout:May 27 07:28:29 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-f[72564]: in thread 7f418bd1d700 thread_name:md_log_replay 2024-05-27T07:28:30.301 INFO:journalctl@ceph.mds.f.smithi173.stdout:May 27 07:28:29 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-f[72564]: 2024-05-27T07:28:29.964+0000 7f418bd1d700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: In function 'std::set SnapClient::filter(const std::set&) const' thread 7f418bd1d700 time 2024-05-27T07:28:29.963064+0000 2024-05-27T07:28:30.301 INFO:journalctl@ceph.mds.f.smithi173.stdout:May 27 07:28:29 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-f[72564]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: 197: FAILED ceph_assert(cached_version > 0) 2024-05-27T07:28:30.301 INFO:journalctl@ceph.mds.f.smithi173.stdout:May 27 07:28:29 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-f[72564]: 2024-05-27T07:28:30.301 INFO:journalctl@ceph.mds.f.smithi173.stdout:May 27 07:28:29 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-f[72564]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:28:30.301 INFO:journalctl@ceph.mds.f.smithi173.stdout:May 27 07:28:29 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-f[72564]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x127) [0x7f419e562ba6] 2024-05-27T07:28:30.301 INFO:journalctl@ceph.mds.f.smithi173.stdout:May 27 07:28:29 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-f[72564]: 2: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7f419e562dd1] 2024-05-27T07:28:30.302 INFO:journalctl@ceph.mds.f.smithi173.stdout:May 27 07:28:29 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-f[72564]: 3: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x55a87af6523d] 2024-05-27T07:28:30.302 INFO:journalctl@ceph.mds.f.smithi173.stdout:May 27 07:28:29 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-f[72564]: 4: (SnapRealm::build_snap_set() const+0x323) [0x55a87af50971] 2024-05-27T07:28:30.302 INFO:journalctl@ceph.mds.f.smithi173.stdout:May 27 07:28:29 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-f[72564]: 5: (SnapRealm::check_cache() const+0x13e) [0x55a87af4efa0] 2024-05-27T07:28:30.302 INFO:journalctl@ceph.mds.f.smithi173.stdout:May 27 07:28:29 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-f[72564]: 6: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x55a87af4f651] 2024-05-27T07:28:30.302 INFO:journalctl@ceph.mds.f.smithi173.stdout:May 27 07:28:29 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-f[72564]: 7: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x55a87aee4347] 2024-05-27T07:28:30.302 INFO:journalctl@ceph.mds.f.smithi173.stdout:May 27 07:28:29 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-f[72564]: 8: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x55a87abe087e] 2024-05-27T07:28:30.302 INFO:journalctl@ceph.mds.f.smithi173.stdout:May 27 07:28:29 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-f[72564]: 9: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x279d) [0x55a87abe5f2f] 2024-05-27T07:28:30.302 INFO:journalctl@ceph.mds.f.smithi173.stdout:May 27 07:28:29 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-f[72564]: 10: (EUpdate::replay(MDSRank*)+0x63) [0x55a87abf0969] 2024-05-27T07:28:30.302 INFO:journalctl@ceph.mds.f.smithi173.stdout:May 27 07:28:29 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-f[72564]: 11: (MDLog::_replay_thread()+0x1f38) [0x55a87af8948c] 2024-05-27T07:28:30.302 INFO:journalctl@ceph.mds.f.smithi173.stdout:May 27 07:28:29 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-f[72564]: 12: (MDLog::ReplayThread::entry()+0x11) [0x55a87af929cb] 2024-05-27T07:28:30.302 INFO:journalctl@ceph.mds.f.smithi173.stdout:May 27 07:28:29 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-f[72564]: 13: (Thread::entry_wrapper()+0x43) [0x7f419e53172d] 2024-05-27T07:28:30.302 INFO:journalctl@ceph.mds.f.smithi173.stdout:May 27 07:28:29 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-f[72564]: 14: (Thread::_entry_func(void*)+0xd) [0x7f419e531749] 2024-05-27T07:28:30.303 INFO:journalctl@ceph.mds.f.smithi173.stdout:May 27 07:28:29 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-f[72564]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7f419d0d51ca] 2024-05-27T07:28:30.303 INFO:journalctl@ceph.mds.f.smithi173.stdout:May 27 07:28:29 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-f[72564]: 16: clone() 2024-05-27T07:28:30.303 INFO:journalctl@ceph.mds.f.smithi173.stdout:May 27 07:28:29 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-f[72564]: 2024-05-27T07:28:30.303 INFO:journalctl@ceph.mds.f.smithi173.stdout:May 27 07:28:29 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-f[72564]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:28:30.303 INFO:journalctl@ceph.mds.f.smithi173.stdout:May 27 07:28:29 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-f[72564]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f419d0dfd20] 2024-05-27T07:28:30.303 INFO:journalctl@ceph.mds.f.smithi173.stdout:May 27 07:28:29 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-f[72564]: 2: gsignal() 2024-05-27T07:28:30.303 INFO:journalctl@ceph.mds.f.smithi173.stdout:May 27 07:28:29 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-f[72564]: 3: abort() 2024-05-27T07:28:30.303 INFO:journalctl@ceph.mds.f.smithi173.stdout:May 27 07:28:29 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-f[72564]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x26e) [0x7f419e562ced] 2024-05-27T07:28:30.303 INFO:journalctl@ceph.mds.f.smithi173.stdout:May 27 07:28:29 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-f[72564]: 5: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7f419e562dd1] 2024-05-27T07:28:30.304 INFO:journalctl@ceph.mds.f.smithi173.stdout:May 27 07:28:29 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-f[72564]: 6: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x55a87af6523d] 2024-05-27T07:28:30.304 INFO:journalctl@ceph.mds.f.smithi173.stdout:May 27 07:28:29 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-f[72564]: 7: (SnapRealm::build_snap_set() const+0x323) [0x55a87af50971] 2024-05-27T07:28:30.304 INFO:journalctl@ceph.mds.f.smithi173.stdout:May 27 07:28:29 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-f[72564]: 8: (SnapRealm::check_cache() const+0x13e) [0x55a87af4efa0] 2024-05-27T07:28:30.304 INFO:journalctl@ceph.mds.f.smithi173.stdout:May 27 07:28:29 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-f[72564]: 9: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x55a87af4f651] 2024-05-27T07:28:30.304 INFO:journalctl@ceph.mds.f.smithi173.stdout:May 27 07:28:29 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-f[72564]: 10: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x55a87aee4347] 2024-05-27T07:28:30.304 INFO:journalctl@ceph.mds.f.smithi173.stdout:May 27 07:28:29 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-f[72564]: 11: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x55a87abe087e] 2024-05-27T07:28:30.304 INFO:journalctl@ceph.mds.f.smithi173.stdout:May 27 07:28:29 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-f[72564]: 12: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x279d) [0x55a87abe5f2f] 2024-05-27T07:28:30.305 INFO:journalctl@ceph.mds.f.smithi173.stdout:May 27 07:28:29 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-f[72564]: 13: (EUpdate::replay(MDSRank*)+0x63) [0x55a87abf0969] 2024-05-27T07:28:30.305 INFO:journalctl@ceph.mds.f.smithi173.stdout:May 27 07:28:29 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-f[72564]: 14: (MDLog::_replay_thread()+0x1f38) [0x55a87af8948c] 2024-05-27T07:28:30.305 INFO:journalctl@ceph.mds.f.smithi173.stdout:May 27 07:28:29 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-f[72564]: 15: (MDLog::ReplayThread::entry()+0x11) [0x55a87af929cb] 2024-05-27T07:28:30.305 INFO:journalctl@ceph.mds.f.smithi173.stdout:May 27 07:28:29 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-f[72564]: 16: (Thread::entry_wrapper()+0x43) [0x7f419e53172d] 2024-05-27T07:28:30.305 INFO:journalctl@ceph.mds.f.smithi173.stdout:May 27 07:28:29 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-f[72564]: 17: (Thread::_entry_func(void*)+0xd) [0x7f419e531749] 2024-05-27T07:28:30.305 INFO:journalctl@ceph.mds.f.smithi173.stdout:May 27 07:28:29 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-f[72564]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7f419d0d51ca] 2024-05-27T07:28:30.305 INFO:journalctl@ceph.mds.f.smithi173.stdout:May 27 07:28:29 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-f[72564]: 19: clone() 2024-05-27T07:28:30.305 INFO:journalctl@ceph.mds.f.smithi173.stdout:May 27 07:28:29 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-f[72564]: 2024-05-27T07:28:29.967+0000 7f418bd1d700 -1 *** Caught signal (Aborted) ** 2024-05-27T07:28:30.305 INFO:journalctl@ceph.mds.f.smithi173.stdout:May 27 07:28:29 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-f[72564]: in thread 7f418bd1d700 thread_name:md_log_replay 2024-05-27T07:28:30.305 INFO:journalctl@ceph.mds.f.smithi173.stdout:May 27 07:28:29 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-f[72564]: 2024-05-27T07:28:30.305 INFO:journalctl@ceph.mds.f.smithi173.stdout:May 27 07:28:29 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-f[72564]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:28:30.305 INFO:journalctl@ceph.mds.f.smithi173.stdout:May 27 07:28:29 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-f[72564]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f419d0dfd20] 2024-05-27T07:28:30.305 INFO:journalctl@ceph.mds.f.smithi173.stdout:May 27 07:28:29 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-f[72564]: 2: gsignal() 2024-05-27T07:28:30.306 INFO:journalctl@ceph.mds.f.smithi173.stdout:May 27 07:28:29 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-f[72564]: 3: abort() 2024-05-27T07:28:30.306 INFO:journalctl@ceph.mds.f.smithi173.stdout:May 27 07:28:29 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-f[72564]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x26e) [0x7f419e562ced] 2024-05-27T07:28:30.306 INFO:journalctl@ceph.mds.f.smithi173.stdout:May 27 07:28:29 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-f[72564]: 5: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7f419e562dd1] 2024-05-27T07:28:30.306 INFO:journalctl@ceph.mds.f.smithi173.stdout:May 27 07:28:29 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-f[72564]: 6: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x55a87af6523d] 2024-05-27T07:28:30.306 INFO:journalctl@ceph.mds.f.smithi173.stdout:May 27 07:28:29 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-f[72564]: 7: (SnapRealm::build_snap_set() const+0x323) [0x55a87af50971] 2024-05-27T07:28:30.306 INFO:journalctl@ceph.mds.f.smithi173.stdout:May 27 07:28:29 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-f[72564]: 8: (SnapRealm::check_cache() const+0x13e) [0x55a87af4efa0] 2024-05-27T07:28:30.306 INFO:journalctl@ceph.mds.f.smithi173.stdout:May 27 07:28:29 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-f[72564]: 9: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x55a87af4f651] 2024-05-27T07:28:30.306 INFO:journalctl@ceph.mds.f.smithi173.stdout:May 27 07:28:29 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-f[72564]: 10: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x55a87aee4347] 2024-05-27T07:28:30.306 INFO:journalctl@ceph.mds.f.smithi173.stdout:May 27 07:28:29 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-f[72564]: 11: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x55a87abe087e] 2024-05-27T07:28:30.306 INFO:journalctl@ceph.mds.f.smithi173.stdout:May 27 07:28:29 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-f[72564]: 12: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x279d) [0x55a87abe5f2f] 2024-05-27T07:28:30.306 INFO:journalctl@ceph.mds.f.smithi173.stdout:May 27 07:28:29 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-f[72564]: 13: (EUpdate::replay(MDSRank*)+0x63) [0x55a87abf0969] 2024-05-27T07:28:30.306 INFO:journalctl@ceph.mds.f.smithi173.stdout:May 27 07:28:29 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-f[72564]: 14: (MDLog::_replay_thread()+0x1f38) [0x55a87af8948c] 2024-05-27T07:28:30.307 INFO:journalctl@ceph.mds.f.smithi173.stdout:May 27 07:28:29 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-f[72564]: 15: (MDLog::ReplayThread::entry()+0x11) [0x55a87af929cb] 2024-05-27T07:28:30.307 INFO:journalctl@ceph.mds.f.smithi173.stdout:May 27 07:28:29 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-f[72564]: 16: (Thread::entry_wrapper()+0x43) [0x7f419e53172d] 2024-05-27T07:28:30.307 INFO:journalctl@ceph.mds.f.smithi173.stdout:May 27 07:28:29 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-f[72564]: 17: (Thread::_entry_func(void*)+0xd) [0x7f419e531749] 2024-05-27T07:28:30.307 INFO:journalctl@ceph.mds.f.smithi173.stdout:May 27 07:28:29 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-f[72564]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7f419d0d51ca] 2024-05-27T07:28:30.307 INFO:journalctl@ceph.mds.f.smithi173.stdout:May 27 07:28:29 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-f[72564]: 19: clone() 2024-05-27T07:28:30.307 INFO:journalctl@ceph.mds.f.smithi173.stdout:May 27 07:28:29 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-f[72564]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-05-27T07:28:30.307 INFO:journalctl@ceph.mds.f.smithi173.stdout:May 27 07:28:29 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-f[72564]: 2024-05-27T07:28:30.307 INFO:journalctl@ceph.mds.f.smithi173.stdout:May 27 07:28:29 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-f[72564]: -3> 2024-05-27T07:28:29.964+0000 7f418bd1d700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: In function 'std::set SnapClient::filter(const std::set&) const' thread 7f418bd1d700 time 2024-05-27T07:28:29.963064+0000 2024-05-27T07:28:30.307 INFO:journalctl@ceph.mds.f.smithi173.stdout:May 27 07:28:29 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-f[72564]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: 197: FAILED ceph_assert(cached_version > 0) 2024-05-27T07:28:30.307 INFO:journalctl@ceph.mds.f.smithi173.stdout:May 27 07:28:29 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-f[72564]: 2024-05-27T07:28:30.307 INFO:journalctl@ceph.mds.f.smithi173.stdout:May 27 07:28:29 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-f[72564]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:28:30.307 INFO:journalctl@ceph.mds.f.smithi173.stdout:May 27 07:28:29 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-f[72564]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x127) [0x7f419e562ba6] 2024-05-27T07:28:30.307 INFO:journalctl@ceph.mds.f.smithi173.stdout:May 27 07:28:29 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-f[72564]: 2: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7f419e562dd1] 2024-05-27T07:28:30.307 INFO:journalctl@ceph.mds.f.smithi173.stdout:May 27 07:28:29 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-f[72564]: 3: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x55a87af6523d] 2024-05-27T07:28:30.307 INFO:journalctl@ceph.mds.f.smithi173.stdout:May 27 07:28:29 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-f[72564]: 4: (SnapRealm::build_snap_set() const+0x323) [0x55a87af50971] 2024-05-27T07:28:30.308 INFO:journalctl@ceph.mds.f.smithi173.stdout:May 27 07:28:29 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-f[72564]: 5: (SnapRealm::check_cache() const+0x13e) [0x55a87af4efa0] 2024-05-27T07:28:30.308 INFO:journalctl@ceph.mds.f.smithi173.stdout:May 27 07:28:29 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-f[72564]: 6: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x55a87af4f651] 2024-05-27T07:28:30.308 INFO:journalctl@ceph.mds.f.smithi173.stdout:May 27 07:28:29 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-f[72564]: 7: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x55a87aee4347] 2024-05-27T07:28:30.308 INFO:journalctl@ceph.mds.f.smithi173.stdout:May 27 07:28:29 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-f[72564]: 8: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x55a87abe087e] 2024-05-27T07:28:30.308 INFO:journalctl@ceph.mds.f.smithi173.stdout:May 27 07:28:29 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-f[72564]: 9: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x279d) [0x55a87abe5f2f] 2024-05-27T07:28:30.308 INFO:journalctl@ceph.mds.f.smithi173.stdout:May 27 07:28:29 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-f[72564]: 10: (EUpdate::replay(MDSRank*)+0x63) [0x55a87abf0969] 2024-05-27T07:28:30.308 INFO:journalctl@ceph.mds.f.smithi173.stdout:May 27 07:28:29 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-f[72564]: 11: (MDLog::_replay_thread()+0x1f38) [0x55a87af8948c] 2024-05-27T07:28:30.308 INFO:journalctl@ceph.mds.f.smithi173.stdout:May 27 07:28:29 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-f[72564]: 12: (MDLog::ReplayThread::entry()+0x11) [0x55a87af929cb] 2024-05-27T07:28:30.308 INFO:journalctl@ceph.mds.f.smithi173.stdout:May 27 07:28:29 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-f[72564]: 13: (Thread::entry_wrapper()+0x43) [0x7f419e53172d] 2024-05-27T07:28:30.308 INFO:journalctl@ceph.mds.f.smithi173.stdout:May 27 07:28:29 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-f[72564]: 14: (Thread::_entry_func(void*)+0xd) [0x7f419e531749] 2024-05-27T07:28:30.308 INFO:journalctl@ceph.mds.f.smithi173.stdout:May 27 07:28:29 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-f[72564]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7f419d0d51ca] 2024-05-27T07:28:30.308 INFO:journalctl@ceph.mds.f.smithi173.stdout:May 27 07:28:29 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-f[72564]: 16: clone() 2024-05-27T07:28:30.308 INFO:journalctl@ceph.mds.f.smithi173.stdout:May 27 07:28:29 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-f[72564]: 2024-05-27T07:28:30.309 INFO:journalctl@ceph.mds.f.smithi173.stdout:May 27 07:28:29 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-f[72564]: 0> 2024-05-27T07:28:29.967+0000 7f418bd1d700 -1 *** Caught signal (Aborted) ** 2024-05-27T07:28:30.309 INFO:journalctl@ceph.mds.f.smithi173.stdout:May 27 07:28:29 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-f[72564]: in thread 7f418bd1d700 thread_name:md_log_replay 2024-05-27T07:28:30.309 INFO:journalctl@ceph.mds.f.smithi173.stdout:May 27 07:28:29 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-f[72564]: 2024-05-27T07:28:30.309 INFO:journalctl@ceph.mds.f.smithi173.stdout:May 27 07:28:29 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-f[72564]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:28:30.309 INFO:journalctl@ceph.mds.f.smithi173.stdout:May 27 07:28:29 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-f[72564]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f419d0dfd20] 2024-05-27T07:28:30.309 INFO:journalctl@ceph.mds.f.smithi173.stdout:May 27 07:28:29 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-f[72564]: 2: gsignal() 2024-05-27T07:28:30.309 INFO:journalctl@ceph.mds.f.smithi173.stdout:May 27 07:28:29 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-f[72564]: 3: abort() 2024-05-27T07:28:30.309 INFO:journalctl@ceph.mds.f.smithi173.stdout:May 27 07:28:29 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-f[72564]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x26e) [0x7f419e562ced] 2024-05-27T07:28:30.309 INFO:journalctl@ceph.mds.f.smithi173.stdout:May 27 07:28:29 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-f[72564]: 5: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7f419e562dd1] 2024-05-27T07:28:30.309 INFO:journalctl@ceph.mds.f.smithi173.stdout:May 27 07:28:29 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-f[72564]: 6: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x55a87af6523d] 2024-05-27T07:28:30.309 INFO:journalctl@ceph.mds.f.smithi173.stdout:May 27 07:28:29 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-f[72564]: 7: (SnapRealm::build_snap_set() const+0x323) [0x55a87af50971] 2024-05-27T07:28:30.309 INFO:journalctl@ceph.mds.f.smithi173.stdout:May 27 07:28:29 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-f[72564]: 8: (SnapRealm::check_cache() const+0x13e) [0x55a87af4efa0] 2024-05-27T07:28:30.310 INFO:journalctl@ceph.mds.f.smithi173.stdout:May 27 07:28:29 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-f[72564]: 9: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x55a87af4f651] 2024-05-27T07:28:30.310 INFO:journalctl@ceph.mds.f.smithi173.stdout:May 27 07:28:29 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-f[72564]: 10: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x55a87aee4347] 2024-05-27T07:28:30.310 INFO:journalctl@ceph.mds.f.smithi173.stdout:May 27 07:28:29 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-f[72564]: 11: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x55a87abe087e] 2024-05-27T07:28:30.310 INFO:journalctl@ceph.mds.f.smithi173.stdout:May 27 07:28:29 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-f[72564]: 12: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x279d) [0x55a87abe5f2f] 2024-05-27T07:28:30.310 INFO:journalctl@ceph.mds.f.smithi173.stdout:May 27 07:28:29 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-f[72564]: 13: (EUpdate::replay(MDSRank*)+0x63) [0x55a87abf0969] 2024-05-27T07:28:30.310 INFO:journalctl@ceph.mds.f.smithi173.stdout:May 27 07:28:29 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-f[72564]: 14: (MDLog::_replay_thread()+0x1f38) [0x55a87af8948c] 2024-05-27T07:28:30.310 INFO:journalctl@ceph.mds.f.smithi173.stdout:May 27 07:28:29 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-f[72564]: 15: (MDLog::ReplayThread::entry()+0x11) [0x55a87af929cb] 2024-05-27T07:28:30.310 INFO:journalctl@ceph.mds.f.smithi173.stdout:May 27 07:28:29 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-f[72564]: 16: (Thread::entry_wrapper()+0x43) [0x7f419e53172d] 2024-05-27T07:28:30.310 INFO:journalctl@ceph.mds.f.smithi173.stdout:May 27 07:28:29 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-f[72564]: 17: (Thread::_entry_func(void*)+0xd) [0x7f419e531749] 2024-05-27T07:28:30.310 INFO:journalctl@ceph.mds.f.smithi173.stdout:May 27 07:28:29 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-f[72564]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7f419d0d51ca] 2024-05-27T07:28:30.310 INFO:journalctl@ceph.mds.f.smithi173.stdout:May 27 07:28:29 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-f[72564]: 19: clone() 2024-05-27T07:28:30.310 INFO:journalctl@ceph.mds.f.smithi173.stdout:May 27 07:28:29 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-f[72564]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-05-27T07:28:30.310 INFO:journalctl@ceph.mds.f.smithi173.stdout:May 27 07:28:29 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-f[72564]: 2024-05-27T07:28:30.311 INFO:journalctl@ceph.mds.f.smithi173.stdout:May 27 07:28:29 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-f[72564]: -3> 2024-05-27T07:28:29.964+0000 7f418bd1d700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: In function 'std::set SnapClient::filter(const std::set&) const' thread 7f418bd1d700 time 2024-05-27T07:28:29.963064+0000 2024-05-27T07:28:30.311 INFO:journalctl@ceph.mds.f.smithi173.stdout:May 27 07:28:29 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-f[72564]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: 197: FAILED ceph_assert(cached_version > 0) 2024-05-27T07:28:30.311 INFO:journalctl@ceph.mds.f.smithi173.stdout:May 27 07:28:29 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-f[72564]: 2024-05-27T07:28:30.311 INFO:journalctl@ceph.mds.f.smithi173.stdout:May 27 07:28:29 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-f[72564]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:28:30.311 INFO:journalctl@ceph.mds.f.smithi173.stdout:May 27 07:28:29 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-f[72564]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x127) [0x7f419e562ba6] 2024-05-27T07:28:30.311 INFO:journalctl@ceph.mds.f.smithi173.stdout:May 27 07:28:29 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-f[72564]: 2: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7f419e562dd1] 2024-05-27T07:28:30.311 INFO:journalctl@ceph.mds.f.smithi173.stdout:May 27 07:28:29 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-f[72564]: 3: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x55a87af6523d] 2024-05-27T07:28:30.311 INFO:journalctl@ceph.mds.f.smithi173.stdout:May 27 07:28:29 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-f[72564]: 4: (SnapRealm::build_snap_set() const+0x323) [0x55a87af50971] 2024-05-27T07:28:30.311 INFO:journalctl@ceph.mds.f.smithi173.stdout:May 27 07:28:29 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-f[72564]: 5: (SnapRealm::check_cache() const+0x13e) [0x55a87af4efa0] 2024-05-27T07:28:30.311 INFO:journalctl@ceph.mds.f.smithi173.stdout:May 27 07:28:29 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-f[72564]: 6: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x55a87af4f651] 2024-05-27T07:28:30.311 INFO:journalctl@ceph.mds.f.smithi173.stdout:May 27 07:28:29 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-f[72564]: 7: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x55a87aee4347] 2024-05-27T07:28:30.311 INFO:journalctl@ceph.mds.f.smithi173.stdout:May 27 07:28:29 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-f[72564]: 8: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x55a87abe087e] 2024-05-27T07:28:30.311 INFO:journalctl@ceph.mds.f.smithi173.stdout:May 27 07:28:29 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-f[72564]: 9: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x279d) [0x55a87abe5f2f] 2024-05-27T07:28:30.312 INFO:journalctl@ceph.mds.f.smithi173.stdout:May 27 07:28:29 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-f[72564]: 10: (EUpdate::replay(MDSRank*)+0x63) [0x55a87abf0969] 2024-05-27T07:28:30.312 INFO:journalctl@ceph.mds.f.smithi173.stdout:May 27 07:28:29 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-f[72564]: 11: (MDLog::_replay_thread()+0x1f38) [0x55a87af8948c] 2024-05-27T07:28:30.312 INFO:journalctl@ceph.mds.f.smithi173.stdout:May 27 07:28:29 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-f[72564]: 12: (MDLog::ReplayThread::entry()+0x11) [0x55a87af929cb] 2024-05-27T07:28:30.312 INFO:journalctl@ceph.mds.f.smithi173.stdout:May 27 07:28:29 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-f[72564]: 13: (Thread::entry_wrapper()+0x43) [0x7f419e53172d] 2024-05-27T07:28:30.312 INFO:journalctl@ceph.mds.f.smithi173.stdout:May 27 07:28:29 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-f[72564]: 14: (Thread::_entry_func(void*)+0xd) [0x7f419e531749] 2024-05-27T07:28:30.312 INFO:journalctl@ceph.mds.f.smithi173.stdout:May 27 07:28:29 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-f[72564]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7f419d0d51ca] 2024-05-27T07:28:30.312 INFO:journalctl@ceph.mds.f.smithi173.stdout:May 27 07:28:29 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-f[72564]: 16: clone() 2024-05-27T07:28:30.312 INFO:journalctl@ceph.mds.f.smithi173.stdout:May 27 07:28:29 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-f[72564]: 2024-05-27T07:28:30.312 INFO:journalctl@ceph.mds.f.smithi173.stdout:May 27 07:28:29 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-f[72564]: 0> 2024-05-27T07:28:29.967+0000 7f418bd1d700 -1 *** Caught signal (Aborted) ** 2024-05-27T07:28:30.312 INFO:journalctl@ceph.mds.f.smithi173.stdout:May 27 07:28:29 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-f[72564]: in thread 7f418bd1d700 thread_name:md_log_replay 2024-05-27T07:28:30.312 INFO:journalctl@ceph.mds.f.smithi173.stdout:May 27 07:28:29 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-f[72564]: 2024-05-27T07:28:30.312 INFO:journalctl@ceph.mds.f.smithi173.stdout:May 27 07:28:29 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-f[72564]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:28:30.313 INFO:journalctl@ceph.mds.f.smithi173.stdout:May 27 07:28:29 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-f[72564]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f419d0dfd20] 2024-05-27T07:28:30.313 INFO:journalctl@ceph.mds.f.smithi173.stdout:May 27 07:28:29 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-f[72564]: 2: gsignal() 2024-05-27T07:28:30.313 INFO:journalctl@ceph.mds.f.smithi173.stdout:May 27 07:28:29 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-f[72564]: 3: abort() 2024-05-27T07:28:30.313 INFO:journalctl@ceph.mds.f.smithi173.stdout:May 27 07:28:29 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-f[72564]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x26e) [0x7f419e562ced] 2024-05-27T07:28:30.313 INFO:journalctl@ceph.mds.f.smithi173.stdout:May 27 07:28:29 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-f[72564]: 5: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7f419e562dd1] 2024-05-27T07:28:30.313 INFO:journalctl@ceph.mds.f.smithi173.stdout:May 27 07:28:29 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-f[72564]: 6: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x55a87af6523d] 2024-05-27T07:28:30.313 INFO:journalctl@ceph.mds.f.smithi173.stdout:May 27 07:28:29 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-f[72564]: 7: (SnapRealm::build_snap_set() const+0x323) [0x55a87af50971] 2024-05-27T07:28:30.313 INFO:journalctl@ceph.mds.f.smithi173.stdout:May 27 07:28:29 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-f[72564]: 8: (SnapRealm::check_cache() const+0x13e) [0x55a87af4efa0] 2024-05-27T07:28:30.313 INFO:journalctl@ceph.mds.f.smithi173.stdout:May 27 07:28:29 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-f[72564]: 9: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x55a87af4f651] 2024-05-27T07:28:30.313 INFO:journalctl@ceph.mds.f.smithi173.stdout:May 27 07:28:29 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-f[72564]: 10: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x55a87aee4347] 2024-05-27T07:28:30.313 INFO:journalctl@ceph.mds.f.smithi173.stdout:May 27 07:28:29 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-f[72564]: 11: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x55a87abe087e] 2024-05-27T07:28:30.313 INFO:journalctl@ceph.mds.f.smithi173.stdout:May 27 07:28:29 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-f[72564]: 12: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x279d) [0x55a87abe5f2f] 2024-05-27T07:28:30.313 INFO:journalctl@ceph.mds.f.smithi173.stdout:May 27 07:28:29 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-f[72564]: 13: (EUpdate::replay(MDSRank*)+0x63) [0x55a87abf0969] 2024-05-27T07:28:30.313 INFO:journalctl@ceph.mds.f.smithi173.stdout:May 27 07:28:29 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-f[72564]: 14: (MDLog::_replay_thread()+0x1f38) [0x55a87af8948c] 2024-05-27T07:28:30.314 INFO:journalctl@ceph.mds.f.smithi173.stdout:May 27 07:28:29 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-f[72564]: 15: (MDLog::ReplayThread::entry()+0x11) [0x55a87af929cb] 2024-05-27T07:28:30.314 INFO:journalctl@ceph.mds.f.smithi173.stdout:May 27 07:28:29 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-f[72564]: 16: (Thread::entry_wrapper()+0x43) [0x7f419e53172d] 2024-05-27T07:28:30.314 INFO:journalctl@ceph.mds.f.smithi173.stdout:May 27 07:28:29 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-f[72564]: 17: (Thread::_entry_func(void*)+0xd) [0x7f419e531749] 2024-05-27T07:28:30.314 INFO:journalctl@ceph.mds.f.smithi173.stdout:May 27 07:28:29 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-f[72564]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7f419d0d51ca] 2024-05-27T07:28:30.314 INFO:journalctl@ceph.mds.f.smithi173.stdout:May 27 07:28:29 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-f[72564]: 19: clone() 2024-05-27T07:28:30.314 INFO:journalctl@ceph.mds.f.smithi173.stdout:May 27 07:28:29 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-f[72564]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-05-27T07:28:30.314 INFO:journalctl@ceph.mds.f.smithi173.stdout:May 27 07:28:29 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-f[72564]: 2024-05-27T07:28:30.314 INFO:journalctl@ceph.mds.f.smithi173.stdout:May 27 07:28:30 smithi173 podman[72599]: 2024-05-27 07:28:30.012203859 +0000 UTC m=+0.025280024 container died 65507d1732d0cf11035e793b7475ea1c7ff3e6f304a3755d2917e9a6b5629e25 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-f, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, GIT_CLEAN=True, ceph=True, org.label-schema.license=GPLv2, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, org.label-schema.build-date=20240521, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD) 2024-05-27T07:28:30.315 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:28:30 smithi052 ceph-mon[31955]: pgmap v81: 129 pgs: 129 active+clean; 96 MiB data, 556 MiB used, 1.0 TiB / 1.0 TiB avail; 5.9 MiB/s rd, 3.8 MiB/s wr, 636 op/s 2024-05-27T07:28:30.315 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:28:30 smithi052 ceph-mon[31955]: daemon mds.f restarted 2024-05-27T07:28:30.315 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:28:30 smithi052 ceph-mon[31955]: osdmap e94: 12 total, 12 up, 12 in 2024-05-27T07:28:30.315 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:28:30 smithi052 ceph-mon[31955]: mds.? [v2:172.21.15.173:6834/687320882,v1:172.21.15.173:6835/687320882] up:boot 2024-05-27T07:28:30.315 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:28:30 smithi052 ceph-mon[31955]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-05-27T07:28:30.315 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:28:30 smithi052 ceph-mon[31955]: from='mgr.15333 172.21.15.52:0/416389082' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2024-05-27T07:28:30.316 DEBUG:teuthology.orchestra.run.smithi052:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:38f9d900c83c63799fbdbe61acc9a11b0d3554a6 shell --fsid 73df2636-1bf8-11ef-bc9b-c7b262605968 -- ceph tell mds.1:2 damage ls 2024-05-27T07:28:30.439 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:28:30 smithi105 ceph-mon[35977]: pgmap v81: 129 pgs: 129 active+clean; 96 MiB data, 556 MiB used, 1.0 TiB / 1.0 TiB avail; 5.9 MiB/s rd, 3.8 MiB/s wr, 636 op/s 2024-05-27T07:28:30.440 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:28:30 smithi105 ceph-mon[35977]: daemon mds.f restarted 2024-05-27T07:28:30.440 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:28:30 smithi105 ceph-mon[35977]: osdmap e94: 12 total, 12 up, 12 in 2024-05-27T07:28:30.440 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:28:30 smithi105 ceph-mon[35977]: mds.? [v2:172.21.15.173:6834/687320882,v1:172.21.15.173:6835/687320882] up:boot 2024-05-27T07:28:30.440 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:28:30 smithi105 ceph-mon[35977]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-05-27T07:28:30.440 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:28:30 smithi105 ceph-mon[35977]: from='mgr.15333 172.21.15.52:0/416389082' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2024-05-27T07:28:30.551 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:28:30.559+0000 7f090b99d640 1 -- 172.21.15.52:0/3344299797 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f0904109030 msgr2=0x7f0904102c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:28:30.551 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:28:30.559+0000 7f090b99d640 1 --2- 172.21.15.52:0/3344299797 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f0904109030 0x7f0904102c70 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f08f4009900 tx=0x7f08f402a410 comp rx=0 tx=0).stop 2024-05-27T07:28:30.551 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:28:30.559+0000 7f090b99d640 1 -- 172.21.15.52:0/3344299797 shutdown_connections 2024-05-27T07:28:30.551 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:28:30.559+0000 7f090b99d640 1 --2- 172.21.15.52:0/3344299797 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f09041032e0 0x7f090410f6b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:28:30.551 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:28:30.559+0000 7f090b99d640 1 --2- 172.21.15.52:0/3344299797 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f0904109030 0x7f0904102c70 unknown :-1 s=CLOSED pgs=95 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:28:30.552 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:28:30.559+0000 7f090b99d640 1 --2- 172.21.15.52:0/3344299797 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f0904108b90 0x7f0904102730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:28:30.552 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:28:30.559+0000 7f090b99d640 1 -- 172.21.15.52:0/3344299797 >> 172.21.15.52:0/3344299797 conn(0x7f09040782d0 msgr2=0x7f0904078710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:28:30.552 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:28:30.560+0000 7f090b99d640 1 -- 172.21.15.52:0/3344299797 shutdown_connections 2024-05-27T07:28:30.552 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:28:30.560+0000 7f090b99d640 1 -- 172.21.15.52:0/3344299797 wait complete. 2024-05-27T07:28:30.552 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:28:30.560+0000 7f090b99d640 1 Processor -- start 2024-05-27T07:28:30.552 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:28:30.560+0000 7f090b99d640 1 -- start start 2024-05-27T07:28:30.558 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:28:30.561+0000 7f090b99d640 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f09041032e0 0x7f09041a2710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:28:30.558 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:28:30.561+0000 7f090b99d640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f0904108b90 0x7f09041a2c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:28:30.559 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:28:30.561+0000 7f090b99d640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f0904109030 0x7f090419c970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:28:30.559 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:28:30.561+0000 7f090b99d640 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x7f0904111cc0 con 0x7f09041032e0 2024-05-27T07:28:30.559 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:28:30.561+0000 7f090b99d640 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f0904111b40 con 0x7f0904109030 2024-05-27T07:28:30.559 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:28:30.561+0000 7f090b99d640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f0904111e40 con 0x7f0904108b90 2024-05-27T07:28:30.559 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:28:30.561+0000 7f0909739640 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f09041032e0 0x7f09041a2710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:28:30.559 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:28:30.561+0000 7f0909739640 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f09041032e0 0x7f09041a2710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.52:34046/0 (socket says 172.21.15.52:34046) 2024-05-27T07:28:30.559 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:28:30.561+0000 7f0909f3a640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f0904109030 0x7f090419c970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:28:30.559 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:28:30.561+0000 7f0909739640 1 -- 172.21.15.52:0/777718191 learned_addr learned my addr 172.21.15.52:0/777718191 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-05-27T07:28:30.559 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:28:30.561+0000 7f0908f38640 1 --2- 172.21.15.52:0/777718191 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f0904108b90 0x7f09041a2c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:28:30.559 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:28:30.561+0000 7f0909739640 1 -- 172.21.15.52:0/777718191 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f0904108b90 msgr2=0x7f09041a2c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:28:30.559 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:28:30.561+0000 7f0909739640 1 --2- 172.21.15.52:0/777718191 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f0904108b90 0x7f09041a2c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:28:30.559 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:28:30.561+0000 7f0909739640 1 -- 172.21.15.52:0/777718191 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f0904109030 msgr2=0x7f090419c970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:28:30.560 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:28:30.561+0000 7f0909739640 1 --2- 172.21.15.52:0/777718191 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f0904109030 0x7f090419c970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:28:30.560 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:28:30.561+0000 7f0909739640 1 -- 172.21.15.52:0/777718191 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f090419d010 con 0x7f09041032e0 2024-05-27T07:28:30.560 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:28:30.561+0000 7f0909f3a640 1 --2- 172.21.15.52:0/777718191 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f0904109030 0x7f090419c970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-27T07:28:30.560 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:28:30.562+0000 7f0909739640 1 --2- 172.21.15.52:0/777718191 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f09041032e0 0x7f09041a2710 secure :-1 s=READY pgs=579 cs=0 l=1 rev1=1 crypto rx=0x7f0900009990 tx=0x7f0900009e50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:28:30.560 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:28:30.562+0000 7f08fa7fc640 1 -- 172.21.15.52:0/777718191 <== mon.0 v2:172.21.15.52:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f090000caa0 con 0x7f09041032e0 2024-05-27T07:28:30.560 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:28:30.562+0000 7f08fa7fc640 1 -- 172.21.15.52:0/777718191 <== mon.0 v2:172.21.15.52:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f090000cc40 con 0x7f09041032e0 2024-05-27T07:28:30.560 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:28:30.562+0000 7f08fa7fc640 1 -- 172.21.15.52:0/777718191 <== mon.0 v2:172.21.15.52:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0900003cc0 con 0x7f09041032e0 2024-05-27T07:28:30.560 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:28:30.562+0000 7f090b99d640 1 -- 172.21.15.52:0/777718191 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f090419d300 con 0x7f09041032e0 2024-05-27T07:28:30.560 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:28:30.562+0000 7f090b99d640 1 -- 172.21.15.52:0/777718191 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f09041a9590 con 0x7f09041032e0 2024-05-27T07:28:30.560 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:28:30.565+0000 7f0912853740 1 Processor -- start 2024-05-27T07:28:30.560 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:28:30.565+0000 7f0912853740 1 -- start start 2024-05-27T07:28:30.560 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:28:30.565+0000 7f08fa7fc640 1 -- 172.21.15.52:0/777718191 <== mon.0 v2:172.21.15.52:3300/0 4 ==== mgrmap(e 26) ==== 149783+0+0 (secure 0 0 0) 0x7f0900016410 con 0x7f09041032e0 2024-05-27T07:28:30.560 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:28:30.565+0000 7f0912853740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55eea9e891b0 0x55eeaa22a3d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:28:30.561 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:28:30.565+0000 7f0912853740 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55eea9e602f0 0x55eeaa22a910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:28:30.561 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:28:30.565+0000 7f0912853740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55eea9e684d0 0x55eeaa22ae50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:28:30.561 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:28:30.565+0000 7f0912853740 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x55eeaa19cc30 con 0x55eea9e602f0 2024-05-27T07:28:30.561 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:28:30.565+0000 7f0912853740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x55eeaa18c7d0 con 0x55eea9e891b0 2024-05-27T07:28:30.561 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:28:30.565+0000 7f0912853740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x55eeaa190810 con 0x55eea9e684d0 2024-05-27T07:28:30.561 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:28:30.565+0000 7f0909f3a640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55eea9e891b0 0x55eeaa22a3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:28:30.561 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:28:30.565+0000 7f0909f3a640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55eea9e891b0 0x55eeaa22a3d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.52:35762/0 (socket says 172.21.15.52:35762) 2024-05-27T07:28:30.561 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:28:30.565+0000 7f0909f3a640 1 -- 172.21.15.52:0/1424874643 learned_addr learned my addr 172.21.15.52:0/1424874643 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-05-27T07:28:30.561 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:28:30.566+0000 7f0909739640 1 --2- 172.21.15.52:0/1424874643 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55eea9e602f0 0x55eeaa22a910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:28:30.561 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:28:30.566+0000 7f0908f38640 1 --2- 172.21.15.52:0/1424874643 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55eea9e684d0 0x55eeaa22ae50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:28:30.561 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:28:30.566+0000 7f0909f3a640 1 -- 172.21.15.52:0/1424874643 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55eea9e684d0 msgr2=0x55eeaa22ae50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:28:30.561 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:28:30.566+0000 7f0909f3a640 1 --2- 172.21.15.52:0/1424874643 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55eea9e684d0 0x55eeaa22ae50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:28:30.561 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:28:30.566+0000 7f0909f3a640 1 -- 172.21.15.52:0/1424874643 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55eea9e602f0 msgr2=0x55eeaa22a910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:28:30.562 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:28:30.566+0000 7f0909f3a640 1 --2- 172.21.15.52:0/1424874643 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55eea9e602f0 0x55eeaa22a910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:28:30.562 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:28:30.566+0000 7f0909f3a640 1 -- 172.21.15.52:0/1424874643 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55eea9ea1210 con 0x55eea9e891b0 2024-05-27T07:28:30.562 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:28:30.566+0000 7f0909f3a640 1 --2- 172.21.15.52:0/1424874643 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55eea9e891b0 0x55eeaa22a3d0 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7f08fc0117e0 tx=0x7f08fc011ca0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=1af1cab4049f1152 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:28:30.562 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:28:30.566+0000 7f08fa7fc640 1 --2- 172.21.15.52:0/777718191 >> [v2:172.21.15.52:6800/1822318384,v1:172.21.15.52:6801/1822318384] conn(0x7f08f00b29d0 0x7f08f00b4ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:28:30.562 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:28:30.566+0000 7f08ef7fe640 1 -- 172.21.15.52:0/1424874643 <== mon.2 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f08fc011310 con 0x55eea9e891b0 2024-05-27T07:28:30.562 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:28:30.566+0000 7f08ef7fe640 1 -- 172.21.15.52:0/1424874643 <== mon.2 v2:172.21.15.105:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f08fc003e20 con 0x55eea9e891b0 2024-05-27T07:28:30.562 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:28:30.566+0000 7f08ef7fe640 1 -- 172.21.15.52:0/1424874643 <== mon.2 v2:172.21.15.105:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f08fc004460 con 0x55eea9e891b0 2024-05-27T07:28:30.562 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:28:30.566+0000 7f0909739640 1 --2- 172.21.15.52:0/777718191 >> [v2:172.21.15.52:6800/1822318384,v1:172.21.15.52:6801/1822318384] conn(0x7f08f00b29d0 0x7f08f00b4ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:28:30.562 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:28:30.567+0000 7f08fa7fc640 1 -- 172.21.15.52:0/777718191 <== mon.0 v2:172.21.15.52:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9379+0+0 (secure 0 0 0) 0x7f09000e1db0 con 0x7f09041032e0 2024-05-27T07:28:30.562 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:28:30.567+0000 7f0912853740 1 -- 172.21.15.52:0/1424874643 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55eea9e891b0 msgr2=0x55eeaa22a3d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:28:30.562 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:28:30.567+0000 7f0909739640 1 --2- 172.21.15.52:0/777718191 >> [v2:172.21.15.52:6800/1822318384,v1:172.21.15.52:6801/1822318384] conn(0x7f08f00b29d0 0x7f08f00b4ee0 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f09000df250 tx=0x7f09000e5900 comp rx=0 tx=0).ready entity=mgr.15333 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:28:30.562 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:28:30.567+0000 7f0912853740 1 --2- 172.21.15.52:0/1424874643 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55eea9e891b0 0x55eeaa22a3d0 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7f08fc0117e0 tx=0x7f08fc011ca0 comp rx=0 tx=0).stop 2024-05-27T07:28:30.563 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:28:30.567+0000 7f0912853740 1 -- 172.21.15.52:0/1424874643 shutdown_connections 2024-05-27T07:28:30.563 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:28:30.567+0000 7f0912853740 1 --2- 172.21.15.52:0/1424874643 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55eea9e684d0 0x55eeaa22ae50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:28:30.563 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:28:30.567+0000 7f0912853740 1 --2- 172.21.15.52:0/1424874643 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55eea9e602f0 0x55eeaa22a910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:28:30.563 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:28:30.567+0000 7f0912853740 1 --2- 172.21.15.52:0/1424874643 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55eea9e891b0 0x55eeaa22a3d0 unknown :-1 s=CLOSED pgs=96 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:28:30.563 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:28:30.567+0000 7f0912853740 1 -- 172.21.15.52:0/1424874643 >> 172.21.15.52:0/1424874643 conn(0x55eea9e7d300 msgr2=0x55eeaa226b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:28:30.563 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:28:30.568+0000 7f0912853740 1 -- 172.21.15.52:0/1424874643 shutdown_connections 2024-05-27T07:28:30.563 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:28:30.568+0000 7f0912853740 1 -- 172.21.15.52:0/1424874643 wait complete. 2024-05-27T07:28:30.563 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:28:30.568+0000 7f0912853740 1 -- start start 2024-05-27T07:28:30.563 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:28:30.568+0000 7f0912853740 1 Processor -- start 2024-05-27T07:28:30.568 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:28:30.568+0000 7f0912853740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55eea9e602f0 0x55eeaa236500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:28:30.568 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:28:30.568+0000 7f0912853740 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55eea9e684d0 0x55eeaa236a40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:28:30.568 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:28:30.568+0000 7f0912853740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55eea9e891b0 0x55eeaa234940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:28:30.568 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:28:30.568+0000 7f0912853740 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x55eea9e72790 con 0x55eea9e684d0 2024-05-27T07:28:30.568 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:28:30.569+0000 7f0912853740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x55eeaa148890 con 0x55eea9e891b0 2024-05-27T07:28:30.568 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:28:30.569+0000 7f0912853740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x55eeaa237500 con 0x55eea9e602f0 2024-05-27T07:28:30.568 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:28:30.569+0000 7f0912853740 10 client.0 _ll_register_callbacks cb 0x55eea9ea0a00 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T07:28:30.568 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:28:30.569+0000 7f0908f38640 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55eea9e684d0 0x55eeaa236a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:28:30.568 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:28:30.569+0000 7f0908f38640 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55eea9e684d0 0x55eeaa236a40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.52:34066/0 (socket says 172.21.15.52:34066) 2024-05-27T07:28:30.568 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:28:30.569+0000 7f0908f38640 1 -- 172.21.15.52:0/2529555389 learned_addr learned my addr 172.21.15.52:0/2529555389 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-05-27T07:28:30.568 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:28:30.569+0000 7f0909f3a640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55eea9e891b0 0x55eeaa234940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:28:30.568 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:28:30.569+0000 7f0909f3a640 1 --2- 172.21.15.52:0/2529555389 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55eea9e602f0 0x55eeaa236500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:28:30.569 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:28:30.569+0000 7f0908f38640 1 -- 172.21.15.52:0/2529555389 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55eea9e602f0 msgr2=0x55eeaa236500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:28:30.569 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:28:30.569+0000 7f0908f38640 1 --2- 172.21.15.52:0/2529555389 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55eea9e602f0 0x55eeaa236500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:28:30.569 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:28:30.569+0000 7f0908f38640 1 -- 172.21.15.52:0/2529555389 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55eea9e891b0 msgr2=0x55eeaa234940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:28:30.569 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:28:30.569+0000 7f0908f38640 1 --2- 172.21.15.52:0/2529555389 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55eea9e891b0 0x55eeaa234940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:28:30.569 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:28:30.569+0000 7f0908f38640 1 -- 172.21.15.52:0/2529555389 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55eea9f6d620 con 0x55eea9e684d0 2024-05-27T07:28:30.569 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:28:30.569+0000 7f0908f38640 1 --2- 172.21.15.52:0/2529555389 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55eea9e684d0 0x55eeaa236a40 secure :-1 s=READY pgs=580 cs=0 l=1 rev1=1 crypto rx=0x7f08f4006010 tx=0x7f08f40042c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2489353ac5e6fb2f server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:28:30.569 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:28:30.569+0000 7f08ecff9640 10 client.0 ms_handle_connect on v2:172.21.15.52:3300/0 2024-05-27T07:28:30.569 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:28:30.569+0000 7f08ecff9640 1 -- 172.21.15.52:0/2529555389 <== mon.0 v2:172.21.15.52:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f08f4035040 con 0x55eea9e684d0 2024-05-27T07:28:30.569 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:28:30.569+0000 7f08ecff9640 1 -- 172.21.15.52:0/2529555389 <== mon.0 v2:172.21.15.52:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9379+0+0 (secure 0 0 0) 0x7f08f4005450 con 0x55eea9e684d0 2024-05-27T07:28:30.569 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:28:30.569+0000 7f0912853740 1 -- 172.21.15.52:0/2529555389 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f08fc003e20 con 0x55eea9e684d0 2024-05-27T07:28:30.569 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:28:30.570+0000 7f08ecff9640 1 -- 172.21.15.52:0/2529555389 <== mon.0 v2:172.21.15.52:3300/0 3 ==== mon_get_version_reply(handle=1 version=37) ==== 24+0+0 (secure 0 0 0) 0x7f08f4005650 con 0x55eea9e684d0 2024-05-27T07:28:30.569 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:28:30.570+0000 7f0912853740 10 client.16074 fetch_fsmap learned FSMap version 37 2024-05-27T07:28:30.570 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:28:30.570+0000 7f0912853740 1 -- 172.21.15.52:0/2529555389 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({config=0+,fsmap=37,monmap=4+}) -- 0x55eeaa243840 con 0x55eea9e684d0 2024-05-27T07:28:30.570 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:28:30.570+0000 7f08ecff9640 1 -- 172.21.15.52:0/2529555389 <== mon.0 v2:172.21.15.52:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f08fc003e20 con 0x55eea9e684d0 2024-05-27T07:28:30.570 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:28:30.570+0000 7f08ecff9640 1 -- 172.21.15.52:0/2529555389 <== mon.0 v2:172.21.15.52:3300/0 5 ==== fsmap(e 37) ==== 8374+0+0 (secure 0 0 0) 0x7f08f40038c0 con 0x55eea9e684d0 2024-05-27T07:28:30.570 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:28:30.570+0000 7f0912853740 10 client.16074 fetch_fsmap finished waiting for FSMap version 37 2024-05-27T07:28:30.570 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:28:30.570+0000 7f0912853740 10 client.16074 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-05-27T07:28:30.570 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:28:30.570+0000 7f0912853740 20 client.16074 populate_metadata read hostname 'smithi052' 2024-05-27T07:28:30.570 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:28:30.570+0000 7f0912853740 1 --2- 172.21.15.52:0/2529555389 >> [v2:172.21.15.105:6838/500348151,v1:172.21.15.105:6839/500348151] conn(0x55eea9e47d80 0x55eeaa246e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:28:30.570 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:28:30.570+0000 7f0912853740 4 client.16074 mds_command: new command op to 24527 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-27T07:28:30.570 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:28:30.570+0000 7f0912853740 1 -- 172.21.15.52:0/2529555389 --> [v2:172.21.15.105:6838/500348151,v1:172.21.15.105:6839/500348151] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55eeaa19eb80 con 0x55eea9e47d80 2024-05-27T07:28:30.570 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:28:30.570+0000 7f0909f3a640 1 --2- 172.21.15.52:0/2529555389 >> [v2:172.21.15.105:6838/500348151,v1:172.21.15.105:6839/500348151] conn(0x55eea9e47d80 0x55eeaa246e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:28:30.570 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:28:30.571+0000 7f08ecff9640 10 client.16074 ms_handle_connect on v2:172.21.15.105:6838/500348151 2024-05-27T07:28:30.570 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:28:30.571+0000 7f0909f3a640 1 --2- 172.21.15.52:0/2529555389 >> [v2:172.21.15.105:6838/500348151,v1:172.21.15.105:6839/500348151] conn(0x55eea9e47d80 0x55eeaa246e00 crc :-1 s=READY pgs=81 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=102255e83be2df43 server_cookie=399208aedea8cdf0 in_seq=0 out_seq=0 2024-05-27T07:28:30.570 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:28:30.574+0000 7f08ecff9640 1 -- 172.21.15.52:0/2529555389 <== mds.0 v2:172.21.15.105:6838/500348151 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24041 (crc 0 0 0) 0x55eeaa19eb80 con 0x55eea9e47d80 2024-05-27T07:28:30.571 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:28:30.574+0000 7f08ecff9640 10 client.16074 handle_command_reply: tid=0 2024-05-27T07:28:30.571 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:28:30.574+0000 7f0912853740 1 client.16074 shutdown 2024-05-27T07:28:30.571 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:28:30.575+0000 7f0912853740 1 -- 172.21.15.52:0/2529555389 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55eea9e684d0 msgr2=0x55eeaa236a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:28:30.571 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:28:30.575+0000 7f0912853740 1 --2- 172.21.15.52:0/2529555389 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55eea9e684d0 0x55eeaa236a40 secure :-1 s=READY pgs=580 cs=0 l=1 rev1=1 crypto rx=0x7f08f4006010 tx=0x7f08f40042c0 comp rx=0 tx=0).stop 2024-05-27T07:28:30.571 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:28:30.575+0000 7f0912853740 1 -- 172.21.15.52:0/2529555389 shutdown_connections 2024-05-27T07:28:30.571 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:28:30.575+0000 7f0912853740 1 --2- 172.21.15.52:0/2529555389 >> [v2:172.21.15.105:6838/500348151,v1:172.21.15.105:6839/500348151] conn(0x55eea9e47d80 0x55eeaa246e00 crc :-1 s=READY pgs=81 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:28:30.571 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:28:30.575+0000 7f0912853740 1 --2- 172.21.15.52:0/2529555389 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55eea9e891b0 0x55eeaa234940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:28:30.571 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:28:30.575+0000 7f0912853740 1 --2- 172.21.15.52:0/2529555389 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55eea9e684d0 0x55eeaa236a40 unknown :-1 s=CLOSED pgs=580 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:28:30.571 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:28:30.575+0000 7f0912853740 1 --2- 172.21.15.52:0/2529555389 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55eea9e602f0 0x55eeaa236500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:28:30.571 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:28:30.575+0000 7f0912853740 1 -- 172.21.15.52:0/2529555389 >> 172.21.15.52:0/2529555389 conn(0x55eea9e7d300 msgr2=0x55eeaa22a200 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:28:30.571 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:28:30.575+0000 7f08ecff9640 0 client.16074 ms_handle_reset on v2:172.21.15.105:6838/500348151 2024-05-27T07:28:30.571 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:28:30.575+0000 7f0912853740 1 -- 172.21.15.52:0/2529555389 shutdown_connections 2024-05-27T07:28:30.572 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:28:30.575+0000 7f0912853740 1 -- 172.21.15.52:0/2529555389 wait complete. 2024-05-27T07:28:30.572 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:28:30.575+0000 7f0912853740 20 client.16074 trim_cache size 0 max 16384 2024-05-27T07:28:30.583 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:28:30.591+0000 7f0912853740 1 Processor -- start 2024-05-27T07:28:30.584 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:28:30.591+0000 7f0912853740 1 -- start start 2024-05-27T07:28:30.584 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:28:30.591+0000 7f0912853740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55eea9e602f0 0x55eeaa22a870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:28:30.584 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:28:30.591+0000 7f0912853740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55eea9e684d0 0x55eeaa2340f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:28:30.584 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:28:30.591+0000 7f0912853740 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55eea9e891b0 0x55eeaa234630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:28:30.584 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:28:30.591+0000 7f0912853740 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x7f09041b3540 con 0x55eea9e891b0 2024-05-27T07:28:30.584 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:28:30.591+0000 7f0912853740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f09041b5110 con 0x55eea9e684d0 2024-05-27T07:28:30.584 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:28:30.591+0000 7f0912853740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f09041b3c10 con 0x55eea9e602f0 2024-05-27T07:28:30.584 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:28:30.592+0000 7f0908f38640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55eea9e684d0 0x55eeaa2340f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:28:30.584 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:28:30.592+0000 7f0909f3a640 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55eea9e891b0 0x55eeaa234630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:28:30.584 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:28:30.592+0000 7f0909f3a640 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55eea9e891b0 0x55eeaa234630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.52:34072/0 (socket says 172.21.15.52:34072) 2024-05-27T07:28:30.585 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:28:30.592+0000 7f0909f3a640 1 -- 172.21.15.52:0/2441802936 learned_addr learned my addr 172.21.15.52:0/2441802936 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-05-27T07:28:30.585 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:28:30.592+0000 7f0908f38640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55eea9e684d0 0x55eeaa2340f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.52:35784/0 (socket says 172.21.15.52:35784) 2024-05-27T07:28:30.585 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:28:30.592+0000 7f0909f3a640 1 --2- 172.21.15.52:0/2441802936 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55eea9e602f0 0x55eeaa22a870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:28:30.585 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:28:30.592+0000 7f0908f38640 1 -- 172.21.15.52:0/2441802936 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55eea9e602f0 msgr2=0x55eeaa22a870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:28:30.585 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:28:30.592+0000 7f0908f38640 1 --2- 172.21.15.52:0/2441802936 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55eea9e602f0 0x55eeaa22a870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:28:30.585 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:28:30.592+0000 7f0908f38640 1 -- 172.21.15.52:0/2441802936 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55eea9e891b0 msgr2=0x55eeaa234630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:28:30.585 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:28:30.592+0000 7f0908f38640 1 --2- 172.21.15.52:0/2441802936 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55eea9e891b0 0x55eeaa234630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:28:30.585 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:28:30.592+0000 7f0908f38640 1 -- 172.21.15.52:0/2441802936 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55eeaa2223b0 con 0x55eea9e684d0 2024-05-27T07:28:30.585 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:28:30.592+0000 7f0909f3a640 1 --2- 172.21.15.52:0/2441802936 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55eea9e602f0 0x55eeaa22a870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-27T07:28:30.585 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:28:30.592+0000 7f0908f38640 1 --2- 172.21.15.52:0/2441802936 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55eea9e684d0 0x55eeaa2340f0 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f08f4003e80 tx=0x7f08f4005970 comp rx=0 tx=0).ready entity=mon.2 client_cookie=8708f89c1fd20726 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:28:30.586 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:28:30.592+0000 7f08eeffd640 1 -- 172.21.15.52:0/2441802936 <== mon.2 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f08f4035070 con 0x55eea9e684d0 2024-05-27T07:28:30.586 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:28:30.592+0000 7f08eeffd640 1 -- 172.21.15.52:0/2441802936 <== mon.2 v2:172.21.15.105:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f08f4040070 con 0x55eea9e684d0 2024-05-27T07:28:30.586 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:28:30.592+0000 7f08eeffd640 1 -- 172.21.15.52:0/2441802936 <== mon.2 v2:172.21.15.105:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f08f4031810 con 0x55eea9e684d0 2024-05-27T07:28:30.591 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:28:30.593+0000 7f0912853740 1 -- 172.21.15.52:0/2441802936 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55eea9e684d0 msgr2=0x55eeaa2340f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:28:30.591 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:28:30.593+0000 7f0912853740 1 --2- 172.21.15.52:0/2441802936 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55eea9e684d0 0x55eeaa2340f0 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f08f4003e80 tx=0x7f08f4005970 comp rx=0 tx=0).stop 2024-05-27T07:28:30.591 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:28:30.593+0000 7f0912853740 1 -- 172.21.15.52:0/2441802936 shutdown_connections 2024-05-27T07:28:30.591 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:28:30.593+0000 7f0912853740 1 --2- 172.21.15.52:0/2441802936 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55eea9e891b0 0x55eeaa234630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:28:30.591 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:28:30.593+0000 7f0912853740 1 --2- 172.21.15.52:0/2441802936 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55eea9e684d0 0x55eeaa2340f0 unknown :-1 s=CLOSED pgs=97 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:28:30.592 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:28:30.593+0000 7f0912853740 1 --2- 172.21.15.52:0/2441802936 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55eea9e602f0 0x55eeaa22a870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:28:30.592 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:28:30.593+0000 7f0912853740 1 -- 172.21.15.52:0/2441802936 >> 172.21.15.52:0/2441802936 conn(0x55eea9e7d300 msgr2=0x55eeaa22a160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:28:30.592 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:28:30.593+0000 7f0912853740 1 -- 172.21.15.52:0/2441802936 shutdown_connections 2024-05-27T07:28:30.592 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:28:30.593+0000 7f0912853740 1 -- 172.21.15.52:0/2441802936 wait complete. 2024-05-27T07:28:30.592 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:28:30.594+0000 7f0912853740 1 -- start start 2024-05-27T07:28:30.592 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:28:30.594+0000 7f0912853740 1 Processor -- start 2024-05-27T07:28:30.592 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:28:30.594+0000 7f0912853740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55eea9e602f0 0x55eeaa239630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:28:30.593 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:28:30.594+0000 7f0912853740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55eea9e684d0 0x55eeaa239b70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:28:30.593 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:28:30.594+0000 7f0912853740 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55eea9e891b0 0x55eeaa23a0b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:28:30.593 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:28:30.594+0000 7f0912853740 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x7f09041b4a40 con 0x55eea9e891b0 2024-05-27T07:28:30.593 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:28:30.594+0000 7f0912853740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f09041b42e0 con 0x55eea9e684d0 2024-05-27T07:28:30.593 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:28:30.594+0000 7f0912853740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f09041b1520 con 0x55eea9e602f0 2024-05-27T07:28:30.593 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:28:30.594+0000 7f0912853740 10 client.0 _ll_register_callbacks cb 0x7f0900005e30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T07:28:30.593 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:28:30.594+0000 7f0909f3a640 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55eea9e891b0 0x55eeaa23a0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:28:30.593 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:28:30.595+0000 7f0909f3a640 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55eea9e891b0 0x55eeaa23a0b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.52:34078/0 (socket says 172.21.15.52:34078) 2024-05-27T07:28:30.594 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:28:30.595+0000 7f0908f38640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55eea9e684d0 0x55eeaa239b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:28:30.594 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:28:30.595+0000 7f0909f3a640 1 -- 172.21.15.52:0/1464256265 learned_addr learned my addr 172.21.15.52:0/1464256265 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-05-27T07:28:30.594 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:28:30.595+0000 7f0909f3a640 1 --2- 172.21.15.52:0/1464256265 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55eea9e602f0 0x55eeaa239630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:28:30.594 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:28:30.595+0000 7f0908f38640 1 -- 172.21.15.52:0/1464256265 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55eea9e602f0 msgr2=0x55eeaa239630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:28:30.594 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:28:30.595+0000 7f0908f38640 1 --2- 172.21.15.52:0/1464256265 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55eea9e602f0 0x55eeaa239630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:28:30.594 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:28:30.595+0000 7f0908f38640 1 -- 172.21.15.52:0/1464256265 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55eea9e891b0 msgr2=0x55eeaa23a0b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:28:30.595 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:28:30.595+0000 7f0908f38640 1 --2- 172.21.15.52:0/1464256265 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55eea9e891b0 0x55eeaa23a0b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:28:30.595 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:28:30.595+0000 7f0908f38640 1 -- 172.21.15.52:0/1464256265 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55eeaa2ad540 con 0x55eea9e684d0 2024-05-27T07:28:30.595 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:28:30.595+0000 7f0909f3a640 1 --2- 172.21.15.52:0/1464256265 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55eea9e891b0 0x55eeaa23a0b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-27T07:28:30.595 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:28:30.595+0000 7f0908f38640 1 --2- 172.21.15.52:0/1464256265 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55eea9e684d0 0x55eeaa239b70 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x55eeaa227640 tx=0x7f08f4005e20 comp rx=0 tx=0).ready entity=mon.2 client_cookie=22fd71634e4916ce server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:28:30.595 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:28:30.595+0000 7f08ecff9640 10 client.0 ms_handle_connect on v2:172.21.15.105:3300/0 2024-05-27T07:28:30.595 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:28:30.595+0000 7f08ecff9640 1 -- 172.21.15.52:0/1464256265 <== mon.2 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f08f4035030 con 0x55eea9e684d0 2024-05-27T07:28:30.595 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:28:30.595+0000 7f0912853740 1 -- 172.21.15.52:0/1464256265 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f08f4040070 con 0x55eea9e684d0 2024-05-27T07:28:30.596 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:28:30.595+0000 7f08ecff9640 1 -- 172.21.15.52:0/1464256265 <== mon.2 v2:172.21.15.105:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9379+0+0 (secure 0 0 0) 0x7f08f4031740 con 0x55eea9e684d0 2024-05-27T07:28:30.596 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:28:30.596+0000 7f08ecff9640 1 -- 172.21.15.52:0/1464256265 <== mon.2 v2:172.21.15.105:3300/0 3 ==== mon_get_version_reply(handle=1 version=37) ==== 24+0+0 (secure 0 0 0) 0x7f08f4069050 con 0x55eea9e684d0 2024-05-27T07:28:30.596 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:28:30.596+0000 7f0912853740 10 client.25748 fetch_fsmap learned FSMap version 37 2024-05-27T07:28:30.596 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:28:30.596+0000 7f0912853740 1 -- 172.21.15.52:0/1464256265 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,fsmap=37,monmap=4+}) -- 0x55eeaa2ae7b0 con 0x55eea9e684d0 2024-05-27T07:28:30.596 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:28:30.596+0000 7f08ecff9640 1 -- 172.21.15.52:0/1464256265 <== mon.2 v2:172.21.15.105:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f08f4040070 con 0x55eea9e684d0 2024-05-27T07:28:30.596 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:28:30.596+0000 7f08ecff9640 1 -- 172.21.15.52:0/1464256265 <== mon.2 v2:172.21.15.105:3300/0 5 ==== fsmap(e 37) ==== 8374+0+0 (secure 0 0 0) 0x7f08f40046c0 con 0x55eea9e684d0 2024-05-27T07:28:30.596 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:28:30.596+0000 7f0912853740 10 client.25748 fetch_fsmap finished waiting for FSMap version 37 2024-05-27T07:28:30.596 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:28:30.596+0000 7f0912853740 10 client.25748 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-05-27T07:28:30.597 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:28:30.596+0000 7f0912853740 20 client.25748 populate_metadata read hostname 'smithi052' 2024-05-27T07:28:30.597 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:28:30.596+0000 7f0912853740 1 --2- 172.21.15.52:0/1464256265 >> [v2:172.21.15.105:6838/500348151,v1:172.21.15.105:6839/500348151] conn(0x55eea9e1d1c0 0x55eeaa2b0f30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:28:30.597 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:28:30.596+0000 7f0912853740 4 client.25748 mds_command: new command op to 24527 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-05-27T07:28:30.597 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:28:30.596+0000 7f0912853740 1 -- 172.21.15.52:0/1464256265 --> [v2:172.21.15.105:6838/500348151,v1:172.21.15.105:6839/500348151] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55eeaa198360 con 0x55eea9e1d1c0 2024-05-27T07:28:30.597 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:28:30.596+0000 7f0909f3a640 1 --2- 172.21.15.52:0/1464256265 >> [v2:172.21.15.105:6838/500348151,v1:172.21.15.105:6839/500348151] conn(0x55eea9e1d1c0 0x55eeaa2b0f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:28:30.597 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:28:30.597+0000 7f08ecff9640 10 client.25748 ms_handle_connect on v2:172.21.15.105:6838/500348151 2024-05-27T07:28:30.597 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:28:30.597+0000 7f0909f3a640 1 --2- 172.21.15.52:0/1464256265 >> [v2:172.21.15.105:6838/500348151,v1:172.21.15.105:6839/500348151] conn(0x55eea9e1d1c0 0x55eeaa2b0f30 crc :-1 s=READY pgs=82 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=3e72881e28c3b7ac server_cookie=4befc03822ab8cad in_seq=0 out_seq=0 2024-05-27T07:28:30.691 INFO:teuthology.orchestra.run.smithi052.stderr:Inferring config /var/lib/ceph/73df2636-1bf8-11ef-bc9b-c7b262605968/mon.a/config 2024-05-27T07:28:30.908 INFO:journalctl@ceph.mds.f.smithi173.stdout:May 27 07:28:30 smithi173 podman[72599]: 2024-05-27 07:28:30.669715723 +0000 UTC m=+0.682791861 container remove 65507d1732d0cf11035e793b7475ea1c7ff3e6f304a3755d2917e9a6b5629e25 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-f, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240521, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2) 2024-05-27T07:28:30.909 INFO:journalctl@ceph.mds.f.smithi173.stdout:May 27 07:28:30 smithi173 systemd[1]: ceph-73df2636-1bf8-11ef-bc9b-c7b262605968@mds.f.service: Main process exited, code=exited, status=134/n/a 2024-05-27T07:28:31.352 INFO:journalctl@ceph.mds.f.smithi173.stdout:May 27 07:28:30 smithi173 systemd[1]: ceph-73df2636-1bf8-11ef-bc9b-c7b262605968@mds.f.service: Failed with result 'exit-code'. 2024-05-27T07:28:32.209 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:28:32 smithi052 ceph-mon[31955]: pgmap v83: 129 pgs: 129 active+clean; 96 MiB data, 556 MiB used, 1.0 TiB / 1.0 TiB avail; 5.9 MiB/s rd, 3.8 MiB/s wr, 636 op/s 2024-05-27T07:28:32.352 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:28:32 smithi173 ceph-mon[37000]: pgmap v83: 129 pgs: 129 active+clean; 96 MiB data, 556 MiB used, 1.0 TiB / 1.0 TiB avail; 5.9 MiB/s rd, 3.8 MiB/s wr, 636 op/s 2024-05-27T07:28:32.439 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:28:32 smithi105 ceph-mon[35977]: pgmap v83: 129 pgs: 129 active+clean; 96 MiB data, 556 MiB used, 1.0 TiB / 1.0 TiB avail; 5.9 MiB/s rd, 3.8 MiB/s wr, 636 op/s 2024-05-27T07:28:32.467 INFO:journalctl@ceph.mds.g.smithi052.stdout:May 27 07:28:32 smithi052 systemd[1]: ceph-73df2636-1bf8-11ef-bc9b-c7b262605968@mds.g.service: Scheduled restart job, restart counter is at 2. 2024-05-27T07:28:32.467 INFO:journalctl@ceph.mds.g.smithi052.stdout:May 27 07:28:32 smithi052 systemd[1]: Stopped Ceph mds.g for 73df2636-1bf8-11ef-bc9b-c7b262605968. 2024-05-27T07:28:32.730 INFO:journalctl@ceph.mds.g.smithi052.stdout:May 27 07:28:32 smithi052 systemd[1]: Starting Ceph mds.g for 73df2636-1bf8-11ef-bc9b-c7b262605968... 2024-05-27T07:28:32.879 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:32.885+0000 7f56dae4a700 1 -- 172.21.15.52:0/1269273103 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f56d411d920 msgr2=0x7f56d411fd40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:28:32.879 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:32.885+0000 7f56dae4a700 1 --2- 172.21.15.52:0/1269273103 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f56d411d920 0x7f56d411fd40 secure :-1 s=READY pgs=581 cs=0 l=1 rev1=1 crypto rx=0x7f56cc00b0a0 tx=0x7f56cc00b3b0 comp rx=0 tx=0).stop 2024-05-27T07:28:32.882 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:32.891+0000 7f56dae4a700 1 -- 172.21.15.52:0/1269273103 shutdown_connections 2024-05-27T07:28:32.882 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:32.891+0000 7f56dae4a700 1 --2- 172.21.15.52:0/1269273103 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f56d411d920 0x7f56d411fd40 unknown :-1 s=CLOSED pgs=581 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:28:32.882 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:32.891+0000 7f56dae4a700 1 --2- 172.21.15.52:0/1269273103 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f56d410a920 0x7f56d410ad30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:28:32.882 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:32.891+0000 7f56dae4a700 1 --2- 172.21.15.52:0/1269273103 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f56d410b270 0x7f56d411d3e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:28:32.882 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:32.891+0000 7f56dae4a700 1 -- 172.21.15.52:0/1269273103 >> 172.21.15.52:0/1269273103 conn(0x7f56d406f4c0 msgr2=0x7f56d406f900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:28:32.888 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:32.896+0000 7f56dae4a700 1 -- 172.21.15.52:0/1269273103 shutdown_connections 2024-05-27T07:28:32.901 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:32.897+0000 7f56dae4a700 1 -- 172.21.15.52:0/1269273103 wait complete. 2024-05-27T07:28:32.901 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:32.898+0000 7f56dae4a700 1 Processor -- start 2024-05-27T07:28:32.901 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:32.899+0000 7f56dae4a700 1 -- start start 2024-05-27T07:28:32.901 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:32.899+0000 7f56dae4a700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f56d410a920 0x7f56d41af840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:28:32.901 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:32.899+0000 7f56dae4a700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f56d410b270 0x7f56d41afd80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:28:32.901 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:32.900+0000 7f56dae4a700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f56d411d920 0x7f56d41a9a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:28:32.901 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:32.900+0000 7f56dae4a700 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x7f56d4123430 con 0x7f56d410a920 2024-05-27T07:28:32.901 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:32.900+0000 7f56dae4a700 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f56d41232b0 con 0x7f56d410b270 2024-05-27T07:28:32.901 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:32.900+0000 7f56dae4a700 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f56d41235b0 con 0x7f56d411d920 2024-05-27T07:28:32.902 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:32.900+0000 7f56d93e7700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f56d411d920 0x7f56d41a9a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:28:32.902 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:32.900+0000 7f56d93e7700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f56d411d920 0x7f56d41a9a60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.173:3300/0 says I am v2:172.21.15.52:48668/0 (socket says 172.21.15.52:48668) 2024-05-27T07:28:32.902 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:32.900+0000 7f56d93e7700 1 -- 172.21.15.52:0/3605600626 learned_addr learned my addr 172.21.15.52:0/3605600626 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-05-27T07:28:32.902 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:32.901+0000 7f56d93e7700 1 -- 172.21.15.52:0/3605600626 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f56d410b270 msgr2=0x7f56d41afd80 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-05-27T07:28:32.902 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:32.901+0000 7f56d93e7700 1 --2- 172.21.15.52:0/3605600626 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f56d410b270 0x7f56d41afd80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:28:32.902 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:32.901+0000 7f56d93e7700 1 -- 172.21.15.52:0/3605600626 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f56d410a920 msgr2=0x7f56d41af840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:28:32.902 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:32.901+0000 7f56d93e7700 1 --2- 172.21.15.52:0/3605600626 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f56d410a920 0x7f56d41af840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:28:32.902 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:32.901+0000 7f56d93e7700 1 -- 172.21.15.52:0/3605600626 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f56d41aa130 con 0x7f56d411d920 2024-05-27T07:28:32.902 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:32.901+0000 7f56d93e7700 1 --2- 172.21.15.52:0/3605600626 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f56d411d920 0x7f56d41a9a60 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f56cc009fd0 tx=0x7f56cc00bf70 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:28:32.903 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:32.902+0000 7f56d1ffb700 1 -- 172.21.15.52:0/3605600626 <== mon.1 v2:172.21.15.173:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f56cc022020 con 0x7f56d411d920 2024-05-27T07:28:32.903 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:32.902+0000 7f56dae4a700 1 -- 172.21.15.52:0/3605600626 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f56d41aa3c0 con 0x7f56d411d920 2024-05-27T07:28:32.903 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:32.902+0000 7f56dae4a700 1 -- 172.21.15.52:0/3605600626 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f56d41aa8a0 con 0x7f56d411d920 2024-05-27T07:28:32.903 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:32.902+0000 7f56d1ffb700 1 -- 172.21.15.52:0/3605600626 <== mon.1 v2:172.21.15.173:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f56cc007c20 con 0x7f56d411d920 2024-05-27T07:28:32.903 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:32.903+0000 7f56d1ffb700 1 -- 172.21.15.52:0/3605600626 <== mon.1 v2:172.21.15.173:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f56cc005820 con 0x7f56d411d920 2024-05-27T07:28:32.903 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:32.907+0000 7f56d1ffb700 1 -- 172.21.15.52:0/3605600626 <== mon.1 v2:172.21.15.173:3300/0 4 ==== mgrmap(e 26) ==== 149783+0+0 (secure 0 0 0) 0x7f56cc025020 con 0x7f56d411d920 2024-05-27T07:28:32.903 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:32.908+0000 7f56d1ffb700 1 --2- 172.21.15.52:0/3605600626 >> [v2:172.21.15.52:6800/1822318384,v1:172.21.15.52:6801/1822318384] conn(0x7f56c40b2c00 0x7f56c40b5110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:28:32.906 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:32.909+0000 7f56d1ffb700 1 -- 172.21.15.52:0/3605600626 <== mon.1 v2:172.21.15.173:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9379+0+0 (secure 0 0 0) 0x7f56cc0e3a80 con 0x7f56d411d920 2024-05-27T07:28:32.906 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:32.909+0000 7f56d8be6700 1 --2- 172.21.15.52:0/3605600626 >> [v2:172.21.15.52:6800/1822318384,v1:172.21.15.52:6801/1822318384] conn(0x7f56c40b2c00 0x7f56c40b5110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:28:32.906 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:32.911+0000 7f56d8be6700 1 --2- 172.21.15.52:0/3605600626 >> [v2:172.21.15.52:6800/1822318384,v1:172.21.15.52:6801/1822318384] conn(0x7f56c40b2c00 0x7f56c40b5110 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f56d41aaba0 tx=0x7f56c0006cd0 comp rx=0 tx=0).ready entity=mgr.15333 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:28:32.912 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:32.921+0000 7f56e985fb80 1 Processor -- start 2024-05-27T07:28:32.919 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:32.928+0000 7f56e985fb80 1 -- start start 2024-05-27T07:28:32.919 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:32.928+0000 7f56e985fb80 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x5622f01a6e40 0x5622f032a710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:28:32.919 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:32.929+0000 7f56e985fb80 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5622f032ac50 0x5622f0324a40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:28:32.920 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:32.929+0000 7f56e985fb80 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5622f0324f80 0x5622f0325390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:28:32.920 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:32.929+0000 7f56e985fb80 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x5622f0253c90 con 0x5622f01a6e40 2024-05-27T07:28:32.920 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:32.929+0000 7f56d3fff700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5622f032ac50 0x5622f0324a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:28:32.920 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:32.929+0000 7f56d3fff700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5622f032ac50 0x5622f0324a40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.173:3300/0 says I am v2:172.21.15.52:48682/0 (socket says 172.21.15.52:48682) 2024-05-27T07:28:32.920 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:32.929+0000 7f56d3fff700 1 -- 172.21.15.52:0/1237147309 learned_addr learned my addr 172.21.15.52:0/1237147309 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-05-27T07:28:32.921 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:32.929+0000 7f56d8be6700 1 --2- 172.21.15.52:0/1237147309 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x5622f01a6e40 0x5622f032a710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:28:32.921 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:32.929+0000 7f56e985fb80 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x5622f0243830 con 0x5622f0324f80 2024-05-27T07:28:32.921 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:32.930+0000 7f56e985fb80 1 -- 172.21.15.52:0/1237147309 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x5622f0247870 con 0x5622f032ac50 2024-05-27T07:28:32.921 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:32.931+0000 7f56d93e7700 1 --2- 172.21.15.52:0/1237147309 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5622f0324f80 0x5622f0325390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:28:32.926 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:32.933+0000 7f56d93e7700 1 -- 172.21.15.52:0/1237147309 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5622f032ac50 msgr2=0x5622f0324a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:28:32.926 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:32.933+0000 7f56d93e7700 1 --2- 172.21.15.52:0/1237147309 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5622f032ac50 0x5622f0324a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:28:32.927 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:32.933+0000 7f56d93e7700 1 -- 172.21.15.52:0/1237147309 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x5622f01a6e40 msgr2=0x5622f032a710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:28:32.927 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:32.933+0000 7f56d93e7700 1 --2- 172.21.15.52:0/1237147309 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x5622f01a6e40 0x5622f032a710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:28:32.927 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:32.933+0000 7f56d93e7700 1 -- 172.21.15.52:0/1237147309 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5622f01938d0 con 0x5622f0324f80 2024-05-27T07:28:32.927 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:32.933+0000 7f56d8be6700 1 --2- 172.21.15.52:0/1237147309 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x5622f01a6e40 0x5622f032a710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-27T07:28:32.927 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:32.936+0000 7f56d93e7700 1 --2- 172.21.15.52:0/1237147309 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5622f0324f80 0x5622f0325390 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7f56cc0e0e10 tx=0x7f56cc00b3b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=3873e5911ffa8dc4 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:28:32.932 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:32.940+0000 7f56beffd700 1 -- 172.21.15.52:0/1237147309 <== mon.2 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f56cc01f040 con 0x5622f0324f80 2024-05-27T07:28:32.932 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:32.940+0000 7f56beffd700 1 -- 172.21.15.52:0/1237147309 <== mon.2 v2:172.21.15.105:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f56cc0b3150 con 0x5622f0324f80 2024-05-27T07:28:32.932 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:32.941+0000 7f56e985fb80 1 -- 172.21.15.52:0/1237147309 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5622f0324f80 msgr2=0x5622f0325390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:28:32.932 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:32.941+0000 7f56e985fb80 1 --2- 172.21.15.52:0/1237147309 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5622f0324f80 0x5622f0325390 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7f56cc0e0e10 tx=0x7f56cc00b3b0 comp rx=0 tx=0).stop 2024-05-27T07:28:32.935 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:32.944+0000 7f56e985fb80 1 -- 172.21.15.52:0/1237147309 shutdown_connections 2024-05-27T07:28:32.935 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:32.944+0000 7f56e985fb80 1 --2- 172.21.15.52:0/1237147309 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5622f0324f80 0x5622f0325390 unknown :-1 s=CLOSED pgs=99 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:28:32.935 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:32.944+0000 7f56e985fb80 1 --2- 172.21.15.52:0/1237147309 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x5622f01a6e40 0x5622f032a710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:28:32.935 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:32.944+0000 7f56e985fb80 1 --2- 172.21.15.52:0/1237147309 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5622f032ac50 0x5622f0324a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:28:32.935 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:32.944+0000 7f56e985fb80 1 -- 172.21.15.52:0/1237147309 >> 172.21.15.52:0/1237147309 conn(0x5622f0201af0 msgr2=0x5622f031f640 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:28:32.935 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:32.945+0000 7f56e985fb80 1 -- 172.21.15.52:0/1237147309 shutdown_connections 2024-05-27T07:28:32.938 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:32.947+0000 7f56e985fb80 1 -- 172.21.15.52:0/1237147309 wait complete. 2024-05-27T07:28:32.938 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:32.948+0000 7f56e985fb80 1 -- start start 2024-05-27T07:28:32.939 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:32.948+0000 7f56e985fb80 1 Processor -- start 2024-05-27T07:28:32.940 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:32.949+0000 7f56e985fb80 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5622f01a6e40 0x5622f032e280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:28:32.940 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:32.949+0000 7f56e985fb80 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5622f0324f80 0x5622f032e7c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:28:32.940 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:32.949+0000 7f56e985fb80 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x5622f032ac50 0x5622f032ed00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:28:32.940 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:32.949+0000 7f56d3fff700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5622f0324f80 0x5622f032e7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:28:32.941 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:32.949+0000 7f56d3fff700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5622f0324f80 0x5622f032e7c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.173:3300/0 says I am v2:172.21.15.52:48690/0 (socket says 172.21.15.52:48690) 2024-05-27T07:28:32.941 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:32.949+0000 7f56d3fff700 1 -- 172.21.15.52:0/1616727404 learned_addr learned my addr 172.21.15.52:0/1616727404 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-05-27T07:28:32.941 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:32.949+0000 7f56e985fb80 1 -- 172.21.15.52:0/1616727404 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x5622f0149fa0 con 0x5622f032ac50 2024-05-27T07:28:32.941 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:32.950+0000 7f56e985fb80 1 -- 172.21.15.52:0/1616727404 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x5622f01e57c0 con 0x5622f01a6e40 2024-05-27T07:28:32.941 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:32.950+0000 7f56e985fb80 1 -- 172.21.15.52:0/1616727404 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x5622f0253c90 con 0x5622f0324f80 2024-05-27T07:28:32.941 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:32.950+0000 7f56e985fb80 10 client.0 _ll_register_callbacks cb 0x5622f01de920 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T07:28:32.942 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:32.950+0000 7f56d3fff700 1 -- 172.21.15.52:0/1616727404 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5622f01a6e40 msgr2=0x5622f032e280 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-05-27T07:28:32.942 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:32.950+0000 7f56d3fff700 1 --2- 172.21.15.52:0/1616727404 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5622f01a6e40 0x5622f032e280 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:28:32.942 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:32.950+0000 7f56d3fff700 1 -- 172.21.15.52:0/1616727404 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x5622f032ac50 msgr2=0x5622f032ed00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:28:32.942 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:32.950+0000 7f56d3fff700 1 --2- 172.21.15.52:0/1616727404 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x5622f032ac50 0x5622f032ed00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:28:32.942 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:32.950+0000 7f56d3fff700 1 -- 172.21.15.52:0/1616727404 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5622f01e2690 con 0x5622f0324f80 2024-05-27T07:28:32.943 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:32.950+0000 7f56d3fff700 1 --2- 172.21.15.52:0/1616727404 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5622f0324f80 0x5622f032e7c0 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f56c800bfd0 tx=0x7f56c800d570 comp rx=0 tx=0).ready entity=mon.1 client_cookie=59f52c287678e014 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:28:32.943 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:32.951+0000 7f56b3fff700 10 client.0 ms_handle_connect on v2:172.21.15.173:3300/0 2024-05-27T07:28:32.943 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:32.951+0000 7f56b3fff700 1 -- 172.21.15.52:0/1616727404 <== mon.1 v2:172.21.15.173:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f56c8013480 con 0x5622f0324f80 2024-05-27T07:28:32.943 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:32.951+0000 7f56b3fff700 1 -- 172.21.15.52:0/1616727404 <== mon.1 v2:172.21.15.173:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9379+0+0 (secure 0 0 0) 0x7f56c8013620 con 0x5622f0324f80 2024-05-27T07:28:32.943 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:32.951+0000 7f56e985fb80 1 -- 172.21.15.52:0/1616727404 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f56cc0b3150 con 0x5622f0324f80 2024-05-27T07:28:32.952 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:32.954+0000 7f56b3fff700 1 -- 172.21.15.52:0/1616727404 <== mon.1 v2:172.21.15.173:3300/0 3 ==== mon_get_version_reply(handle=1 version=38) ==== 24+0+0 (secure 0 0 0) 0x7f56c8013820 con 0x5622f0324f80 2024-05-27T07:28:32.952 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:32.954+0000 7f56e985fb80 10 client.25675 fetch_fsmap learned FSMap version 38 2024-05-27T07:28:32.953 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:32.954+0000 7f56e985fb80 1 -- 172.21.15.52:0/1616727404 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({config=0+,fsmap=38,monmap=4+}) -- 0x5622f033be30 con 0x5622f0324f80 2024-05-27T07:28:32.953 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:32.958+0000 7f56b3fff700 1 -- 172.21.15.52:0/1616727404 <== mon.1 v2:172.21.15.173:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f56c8018070 con 0x5622f0324f80 2024-05-27T07:28:32.953 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:32.958+0000 7f56b3fff700 1 -- 172.21.15.52:0/1616727404 <== mon.1 v2:172.21.15.173:3300/0 5 ==== fsmap(e 38) ==== 8378+0+0 (secure 0 0 0) 0x7f56c801dbe0 con 0x5622f0324f80 2024-05-27T07:28:32.953 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:32.958+0000 7f56e985fb80 10 client.25675 fetch_fsmap finished waiting for FSMap version 38 2024-05-27T07:28:32.953 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:32.958+0000 7f56e985fb80 10 client.25675 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.i 2024-05-27T07:28:32.953 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:32.958+0000 7f56e985fb80 20 client.25675 populate_metadata read hostname 'smithi052' 2024-05-27T07:28:32.953 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:32.958+0000 7f56e985fb80 1 --2- 172.21.15.52:0/1616727404 >> [v2:172.21.15.173:6838/564778293,v1:172.21.15.173:6839/564778293] conn(0x5622f033e510 0x5622f0340970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:28:32.953 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:32.958+0000 7f56e985fb80 4 client.25675 mds_command: new command op to 24475 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-27T07:28:32.953 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:32.958+0000 7f56e985fb80 1 -- 172.21.15.52:0/1616727404 --> [v2:172.21.15.173:6838/564778293,v1:172.21.15.173:6839/564778293] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x7f56cc0b3150 con 0x5622f033e510 2024-05-27T07:28:32.953 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:32.961+0000 7f56d8be6700 1 --2- 172.21.15.52:0/1616727404 >> [v2:172.21.15.173:6838/564778293,v1:172.21.15.173:6839/564778293] conn(0x5622f033e510 0x5622f0340970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:28:32.961 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:32.971+0000 7f56d8be6700 1 --2- 172.21.15.52:0/1616727404 >> [v2:172.21.15.173:6838/564778293,v1:172.21.15.173:6839/564778293] conn(0x5622f033e510 0x5622f0340970 crc :-1 s=READY pgs=36 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=b1f01fbfcfab3136 server_cookie=6320180913953b86 in_seq=0 out_seq=0 2024-05-27T07:28:32.961 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:32.971+0000 7f56b3fff700 10 client.25675 ms_handle_connect on v2:172.21.15.173:6838/564778293 2024-05-27T07:28:32.965 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:32.974+0000 7f56b3fff700 1 -- 172.21.15.52:0/1616727404 <== mds.2 v2:172.21.15.173:6838/564778293 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24041 (crc 0 0 0) 0x7f56cc0b3150 con 0x5622f033e510 2024-05-27T07:28:32.965 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:32.974+0000 7f56b3fff700 10 client.25675 handle_command_reply: tid=0 2024-05-27T07:28:32.965 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:32.974+0000 7f56e985fb80 1 client.25675 shutdown 2024-05-27T07:28:32.966 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:32.974+0000 7f56e985fb80 1 -- 172.21.15.52:0/1616727404 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5622f0324f80 msgr2=0x5622f032e7c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:28:32.966 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:32.974+0000 7f56e985fb80 1 --2- 172.21.15.52:0/1616727404 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5622f0324f80 0x5622f032e7c0 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f56c800bfd0 tx=0x7f56c800d570 comp rx=0 tx=0).stop 2024-05-27T07:28:32.966 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:32.974+0000 7f56e985fb80 1 -- 172.21.15.52:0/1616727404 shutdown_connections 2024-05-27T07:28:32.966 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:32.975+0000 7f56e985fb80 1 --2- 172.21.15.52:0/1616727404 >> [v2:172.21.15.173:6838/564778293,v1:172.21.15.173:6839/564778293] conn(0x5622f033e510 0x5622f0340970 crc :-1 s=READY pgs=36 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:28:32.966 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:32.975+0000 7f56b3fff700 0 client.25675 ms_handle_reset on v2:172.21.15.173:6838/564778293 2024-05-27T07:28:32.966 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:32.975+0000 7f56e985fb80 1 --2- 172.21.15.52:0/1616727404 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x5622f032ac50 0x5622f032ed00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:28:32.966 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:32.975+0000 7f56e985fb80 1 --2- 172.21.15.52:0/1616727404 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5622f01a6e40 0x5622f032e280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:28:32.967 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:32.975+0000 7f56e985fb80 1 --2- 172.21.15.52:0/1616727404 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5622f0324f80 0x5622f032e7c0 unknown :-1 s=CLOSED pgs=94 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:28:32.967 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:32.975+0000 7f56e985fb80 1 -- 172.21.15.52:0/1616727404 >> 172.21.15.52:0/1616727404 conn(0x5622f0201af0 msgr2=0x5622f0323a30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:28:32.967 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:32.975+0000 7f56e985fb80 1 -- 172.21.15.52:0/1616727404 shutdown_connections 2024-05-27T07:28:32.967 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:32.975+0000 7f56e985fb80 1 -- 172.21.15.52:0/1616727404 wait complete. 2024-05-27T07:28:32.967 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:32.976+0000 7f56e985fb80 20 client.25675 trim_cache size 0 max 16384 2024-05-27T07:28:33.004 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:33.012+0000 7f56e985fb80 1 Processor -- start 2024-05-27T07:28:33.004 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:33.013+0000 7f56e985fb80 1 -- start start 2024-05-27T07:28:33.004 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:33.013+0000 7f56e985fb80 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5622f01a6e40 0x5622f0324a40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:28:33.005 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:33.013+0000 7f56e985fb80 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5622f0324f80 0x5622f032eb40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:28:33.005 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:33.013+0000 7f56e985fb80 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x5622f032ac50 0x5622f033a520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:28:33.005 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:33.013+0000 7f56e985fb80 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x7f56b80108f0 con 0x5622f032ac50 2024-05-27T07:28:33.005 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:33.013+0000 7f56e985fb80 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f56b800f3c0 con 0x5622f01a6e40 2024-05-27T07:28:33.005 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:33.013+0000 7f56e985fb80 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f56b800fa90 con 0x5622f0324f80 2024-05-27T07:28:33.005 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:33.014+0000 7f56d93e7700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x5622f032ac50 0x5622f033a520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:28:33.005 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:33.014+0000 7f56d93e7700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x5622f032ac50 0x5622f033a520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.52:33996/0 (socket says 172.21.15.52:33996) 2024-05-27T07:28:33.005 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:33.014+0000 7f56d93e7700 1 -- 172.21.15.52:0/1051727988 learned_addr learned my addr 172.21.15.52:0/1051727988 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-05-27T07:28:33.005 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:33.014+0000 7f56d3fff700 1 --2- 172.21.15.52:0/1051727988 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5622f0324f80 0x5622f032eb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:28:33.006 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:33.015+0000 7f56d3fff700 1 -- 172.21.15.52:0/1051727988 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5622f01a6e40 msgr2=0x5622f0324a40 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-05-27T07:28:33.006 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:33.015+0000 7f56d3fff700 1 --2- 172.21.15.52:0/1051727988 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5622f01a6e40 0x5622f0324a40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:28:33.006 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:33.015+0000 7f56d3fff700 1 -- 172.21.15.52:0/1051727988 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x5622f032ac50 msgr2=0x5622f033a520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:28:33.006 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:33.015+0000 7f56d3fff700 1 --2- 172.21.15.52:0/1051727988 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x5622f032ac50 0x5622f033a520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:28:33.006 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:33.015+0000 7f56d3fff700 1 -- 172.21.15.52:0/1051727988 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5622f032c7f0 con 0x5622f0324f80 2024-05-27T07:28:33.009 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:33.019+0000 7f56d3fff700 1 --2- 172.21.15.52:0/1051727988 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5622f0324f80 0x5622f032eb40 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x5622f031c240 tx=0x7f56c800b370 comp rx=0 tx=0).ready entity=mon.1 client_cookie=91d82518c378e55f server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:28:33.012 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:33.020+0000 7f56be7fc700 1 -- 172.21.15.52:0/1051727988 <== mon.1 v2:172.21.15.173:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f56c8012570 con 0x5622f0324f80 2024-05-27T07:28:33.012 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:33.020+0000 7f56be7fc700 1 -- 172.21.15.52:0/1051727988 <== mon.1 v2:172.21.15.173:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f56c8018040 con 0x5622f0324f80 2024-05-27T07:28:33.012 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:33.020+0000 7f56be7fc700 1 -- 172.21.15.52:0/1051727988 <== mon.1 v2:172.21.15.173:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f56c8004040 con 0x5622f0324f80 2024-05-27T07:28:33.013 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:33.021+0000 7f56e985fb80 1 -- 172.21.15.52:0/1051727988 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5622f0324f80 msgr2=0x5622f032eb40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:28:33.014 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:33.021+0000 7f56e985fb80 1 --2- 172.21.15.52:0/1051727988 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5622f0324f80 0x5622f032eb40 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x5622f031c240 tx=0x7f56c800b370 comp rx=0 tx=0).stop 2024-05-27T07:28:33.018 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:33.027+0000 7f56e985fb80 1 -- 172.21.15.52:0/1051727988 shutdown_connections 2024-05-27T07:28:33.018 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:33.027+0000 7f56e985fb80 1 --2- 172.21.15.52:0/1051727988 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x5622f032ac50 0x5622f033a520 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:28:33.018 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:33.027+0000 7f56e985fb80 1 --2- 172.21.15.52:0/1051727988 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5622f01a6e40 0x5622f0324a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:28:33.018 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:33.027+0000 7f56e985fb80 1 --2- 172.21.15.52:0/1051727988 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5622f0324f80 0x5622f032eb40 unknown :-1 s=CLOSED pgs=95 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:28:33.018 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:33.027+0000 7f56e985fb80 1 -- 172.21.15.52:0/1051727988 >> 172.21.15.52:0/1051727988 conn(0x5622f0201af0 msgr2=0x5622f0327e00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:28:33.018 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:33.028+0000 7f56e985fb80 1 -- 172.21.15.52:0/1051727988 shutdown_connections 2024-05-27T07:28:33.018 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:33.028+0000 7f56e985fb80 1 -- 172.21.15.52:0/1051727988 wait complete. 2024-05-27T07:28:33.019 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:33.028+0000 7f56e985fb80 1 -- start start 2024-05-27T07:28:33.019 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:33.029+0000 7f56e985fb80 1 Processor -- start 2024-05-27T07:28:33.022 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:33.030+0000 7f56e985fb80 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x5622f01a6e40 0x5622f0336910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:28:33.022 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:33.030+0000 7f56e985fb80 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5622f0324f80 0x5622f0336e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:28:33.022 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:33.030+0000 7f56e985fb80 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5622f032ac50 0x5622f0345580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:28:33.023 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:33.030+0000 7f56e985fb80 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x7f56b800cc90 con 0x5622f01a6e40 2024-05-27T07:28:33.023 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:33.030+0000 7f56e985fb80 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f56b8010220 con 0x5622f032ac50 2024-05-27T07:28:33.023 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:33.031+0000 7f56e985fb80 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f56b800d300 con 0x5622f0324f80 2024-05-27T07:28:33.023 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:33.031+0000 7f56d93e7700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5622f032ac50 0x5622f0345580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:28:33.023 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:33.031+0000 7f56d3fff700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5622f0324f80 0x5622f0336e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:28:33.023 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:33.031+0000 7f56d3fff700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5622f0324f80 0x5622f0336e50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.173:3300/0 says I am v2:172.21.15.52:48702/0 (socket says 172.21.15.52:48702) 2024-05-27T07:28:33.023 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:33.031+0000 7f56d3fff700 1 -- 172.21.15.52:0/3127267545 learned_addr learned my addr 172.21.15.52:0/3127267545 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-05-27T07:28:33.023 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:33.031+0000 7f56e985fb80 10 client.0 _ll_register_callbacks cb 0x5622f01de920 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T07:28:33.026 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:33.032+0000 7f56d3fff700 1 -- 172.21.15.52:0/3127267545 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5622f032ac50 msgr2=0x5622f0345580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:28:33.026 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:33.032+0000 7f56d3fff700 1 --2- 172.21.15.52:0/3127267545 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5622f032ac50 0x5622f0345580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:28:33.026 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:33.032+0000 7f56d3fff700 1 -- 172.21.15.52:0/3127267545 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x5622f01a6e40 msgr2=0x5622f0336910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:28:33.026 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:33.032+0000 7f56d3fff700 1 --2- 172.21.15.52:0/3127267545 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x5622f01a6e40 0x5622f0336910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:28:33.026 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:33.032+0000 7f56d3fff700 1 -- 172.21.15.52:0/3127267545 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5622f0345d20 con 0x5622f0324f80 2024-05-27T07:28:33.026 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:33.032+0000 7f56d3fff700 1 --2- 172.21.15.52:0/3127267545 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5622f0324f80 0x5622f0336e50 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7f56c801dfd0 tx=0x7f56c8012fa0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=1d115acb487ac0cb server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:28:33.026 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:33.034+0000 7f56b3fff700 10 client.0 ms_handle_connect on v2:172.21.15.173:3300/0 2024-05-27T07:28:33.027 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:33.034+0000 7f56b3fff700 1 -- 172.21.15.52:0/3127267545 <== mon.1 v2:172.21.15.173:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f56c80139c0 con 0x5622f0324f80 2024-05-27T07:28:33.027 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:33.034+0000 7f56b3fff700 1 -- 172.21.15.52:0/3127267545 <== mon.1 v2:172.21.15.173:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9379+0+0 (secure 0 0 0) 0x7f56c8013b60 con 0x5622f0324f80 2024-05-27T07:28:33.027 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:33.035+0000 7f56e985fb80 1 -- 172.21.15.52:0/3127267545 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f56c8018040 con 0x5622f0324f80 2024-05-27T07:28:33.042 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:33.047+0000 7f56b3fff700 1 -- 172.21.15.52:0/3127267545 <== mon.1 v2:172.21.15.173:3300/0 3 ==== mon_get_version_reply(handle=1 version=38) ==== 24+0+0 (secure 0 0 0) 0x7f56c8013d60 con 0x5622f0324f80 2024-05-27T07:28:33.042 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:33.047+0000 7f56e985fb80 10 client.25681 fetch_fsmap learned FSMap version 38 2024-05-27T07:28:33.042 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:33.047+0000 7f56e985fb80 1 -- 172.21.15.52:0/3127267545 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({config=0+,fsmap=38,monmap=4+}) -- 0x5622f03a8a80 con 0x5622f0324f80 2024-05-27T07:28:33.048 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:33.052+0000 7f56b3fff700 1 -- 172.21.15.52:0/3127267545 <== mon.1 v2:172.21.15.173:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f56c8018040 con 0x5622f0324f80 2024-05-27T07:28:33.048 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:33.052+0000 7f56b3fff700 1 -- 172.21.15.52:0/3127267545 <== mon.1 v2:172.21.15.173:3300/0 5 ==== fsmap(e 38) ==== 8378+0+0 (secure 0 0 0) 0x7f56c8025070 con 0x5622f0324f80 2024-05-27T07:28:33.048 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:33.056+0000 7f56e985fb80 10 client.25681 fetch_fsmap finished waiting for FSMap version 38 2024-05-27T07:28:33.049 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:33.056+0000 7f56e985fb80 10 client.25681 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.i 2024-05-27T07:28:33.049 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:33.056+0000 7f56e985fb80 20 client.25681 populate_metadata read hostname 'smithi052' 2024-05-27T07:28:33.049 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:33.056+0000 7f56e985fb80 1 --2- 172.21.15.52:0/3127267545 >> [v2:172.21.15.173:6838/564778293,v1:172.21.15.173:6839/564778293] conn(0x5622f033e510 0x5622f03aaf80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:28:33.049 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:33.056+0000 7f56e985fb80 4 client.25681 mds_command: new command op to 24475 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-05-27T07:28:33.049 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:33.056+0000 7f56e985fb80 1 -- 172.21.15.52:0/3127267545 --> [v2:172.21.15.173:6838/564778293,v1:172.21.15.173:6839/564778293] -- command(tid 0: {"prefix": "damage ls"}) -- 0x5622f02f09f0 con 0x5622f033e510 2024-05-27T07:28:33.049 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:33.057+0000 7f56d8be6700 1 --2- 172.21.15.52:0/3127267545 >> [v2:172.21.15.173:6838/564778293,v1:172.21.15.173:6839/564778293] conn(0x5622f033e510 0x5622f03aaf80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:28:33.053 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:33.061+0000 7f56b3fff700 10 client.25681 ms_handle_connect on v2:172.21.15.173:6838/564778293 2024-05-27T07:28:33.053 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:33.061+0000 7f56d8be6700 1 --2- 172.21.15.52:0/3127267545 >> [v2:172.21.15.173:6838/564778293,v1:172.21.15.173:6839/564778293] conn(0x5622f033e510 0x5622f03aaf80 crc :-1 s=READY pgs=37 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=65720a8aae5d9bd6 server_cookie=647f7b80490de254 in_seq=0 out_seq=0 2024-05-27T07:28:33.055 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:33.064+0000 7f56b3fff700 1 -- 172.21.15.52:0/3127267545 <== mds.2 v2:172.21.15.173:6838/564778293 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x5622f02f09f0 con 0x5622f033e510 2024-05-27T07:28:33.055 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:33.064+0000 7f56b3fff700 10 client.25681 handle_command_reply: tid=0 2024-05-27T07:28:33.055 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:33.064+0000 7f56e985fb80 1 client.25681 shutdown 2024-05-27T07:28:33.056 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:33.065+0000 7f56e985fb80 1 -- 172.21.15.52:0/3127267545 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5622f0324f80 msgr2=0x5622f0336e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:28:33.056 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:33.065+0000 7f56e985fb80 1 --2- 172.21.15.52:0/3127267545 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5622f0324f80 0x5622f0336e50 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7f56c801dfd0 tx=0x7f56c8012fa0 comp rx=0 tx=0).stop 2024-05-27T07:28:33.057 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:33.066+0000 7f56e985fb80 1 -- 172.21.15.52:0/3127267545 shutdown_connections 2024-05-27T07:28:33.057 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:33.066+0000 7f56e985fb80 1 --2- 172.21.15.52:0/3127267545 >> [v2:172.21.15.173:6838/564778293,v1:172.21.15.173:6839/564778293] conn(0x5622f033e510 0x5622f03aaf80 crc :-1 s=READY pgs=37 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:28:33.057 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:33.066+0000 7f56b3fff700 0 client.25681 ms_handle_reset on v2:172.21.15.173:6838/564778293 2024-05-27T07:28:33.057 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:33.066+0000 7f56e985fb80 1 --2- 172.21.15.52:0/3127267545 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5622f032ac50 0x5622f0345580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:28:33.057 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:33.066+0000 7f56e985fb80 1 --2- 172.21.15.52:0/3127267545 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x5622f01a6e40 0x5622f0336910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:28:33.057 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:33.066+0000 7f56e985fb80 1 --2- 172.21.15.52:0/3127267545 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5622f0324f80 0x5622f0336e50 unknown :-1 s=CLOSED pgs=96 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:28:33.057 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:33.066+0000 7f56e985fb80 1 -- 172.21.15.52:0/3127267545 >> 172.21.15.52:0/3127267545 conn(0x5622f0201af0 msgr2=0x5622f0319e80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:28:33.059 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:33.068+0000 7f56e985fb80 1 -- 172.21.15.52:0/3127267545 shutdown_connections 2024-05-27T07:28:33.059 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:33.068+0000 7f56e985fb80 1 -- 172.21.15.52:0/3127267545 wait complete. 2024-05-27T07:28:33.059 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:33.068+0000 7f56e985fb80 20 client.25681 trim_cache size 0 max 16384 2024-05-27T07:28:33.059 INFO:teuthology.orchestra.run.smithi052.stdout:[] 2024-05-27T07:28:33.064 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:33.073+0000 7f56dae4a700 1 -- 172.21.15.52:0/3605600626 >> [v2:172.21.15.52:6800/1822318384,v1:172.21.15.52:6801/1822318384] conn(0x7f56c40b2c00 msgr2=0x7f56c40b5110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:28:33.064 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:33.073+0000 7f56dae4a700 1 --2- 172.21.15.52:0/3605600626 >> [v2:172.21.15.52:6800/1822318384,v1:172.21.15.52:6801/1822318384] conn(0x7f56c40b2c00 0x7f56c40b5110 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f56d41aaba0 tx=0x7f56c0006cd0 comp rx=0 tx=0).stop 2024-05-27T07:28:33.064 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:33.073+0000 7f56dae4a700 1 -- 172.21.15.52:0/3605600626 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f56d411d920 msgr2=0x7f56d41a9a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:28:33.064 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:33.073+0000 7f56dae4a700 1 --2- 172.21.15.52:0/3605600626 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f56d411d920 0x7f56d41a9a60 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f56cc009fd0 tx=0x7f56cc00bf70 comp rx=0 tx=0).stop 2024-05-27T07:28:33.064 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:33.073+0000 7f56dae4a700 1 -- 172.21.15.52:0/3605600626 shutdown_connections 2024-05-27T07:28:33.064 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:33.073+0000 7f56dae4a700 1 --2- 172.21.15.52:0/3605600626 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f56d411d920 0x7f56d41a9a60 unknown :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:28:33.064 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:33.073+0000 7f56dae4a700 1 --2- 172.21.15.52:0/3605600626 >> [v2:172.21.15.52:6800/1822318384,v1:172.21.15.52:6801/1822318384] conn(0x7f56c40b2c00 0x7f56c40b5110 unknown :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:28:33.064 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:33.073+0000 7f56dae4a700 1 --2- 172.21.15.52:0/3605600626 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f56d410a920 0x7f56d41af840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:28:33.064 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:33.073+0000 7f56dae4a700 1 --2- 172.21.15.52:0/3605600626 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f56d410b270 0x7f56d41afd80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:28:33.064 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:33.073+0000 7f56dae4a700 1 -- 172.21.15.52:0/3605600626 >> 172.21.15.52:0/3605600626 conn(0x7f56d406f4c0 msgr2=0x7f56d411b3d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:28:33.065 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:33.074+0000 7f56dae4a700 1 -- 172.21.15.52:0/3605600626 shutdown_connections 2024-05-27T07:28:33.065 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:33.074+0000 7f56dae4a700 1 -- 172.21.15.52:0/3605600626 wait complete. 2024-05-27T07:28:33.093 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:28:33 smithi052 ceph-mon[31955]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-05-27T07:28:33.351 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:28:33 smithi173 ceph-mon[37000]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-05-27T07:28:33.439 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:28:33 smithi105 ceph-mon[35977]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-05-27T07:28:33.900 DEBUG:teuthology.orchestra.run.smithi052:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:38f9d900c83c63799fbdbe61acc9a11b0d3554a6 shell --fsid 73df2636-1bf8-11ef-bc9b-c7b262605968 -- ceph tell mds.1:1 damage ls 2024-05-27T07:28:34.130 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:28:34 smithi052 ceph-mon[31955]: pgmap v84: 129 pgs: 129 active+clean; 98 MiB data, 583 MiB used, 1.0 TiB / 1.0 TiB avail; 5.5 MiB/s rd, 3.9 MiB/s wr, 712 op/s 2024-05-27T07:28:34.131 INFO:journalctl@ceph.mds.g.smithi052.stdout:May 27 07:28:34 smithi052 podman[103102]: 2024-05-27 07:28:34.025859106 +0000 UTC m=+0.029420734 image pull e36ea4fc8a5807e8d2701370e350a0de3856538797f7fba64cfce7e344bd6799 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6 2024-05-27T07:28:34.131 INFO:journalctl@ceph.mds.g.smithi052.stdout:May 27 07:28:34 smithi052 podman[103102]: 2024-05-27 07:28:34.141582214 +0000 UTC m=+0.145143817 container create ff5d245d9ddc6752e3804e7b59f26b44e1aedad2f0ad09bae6127b410f919ae6 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-g, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240521, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, GIT_BRANCH=HEAD, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, org.label-schema.schema-version=1.0, GIT_CLEAN=True, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-05-27T07:28:34.351 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:28:34 smithi173 ceph-mon[37000]: pgmap v84: 129 pgs: 129 active+clean; 98 MiB data, 583 MiB used, 1.0 TiB / 1.0 TiB avail; 5.5 MiB/s rd, 3.9 MiB/s wr, 712 op/s 2024-05-27T07:28:34.439 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:28:34 smithi105 ceph-mon[35977]: pgmap v84: 129 pgs: 129 active+clean; 98 MiB data, 583 MiB used, 1.0 TiB / 1.0 TiB avail; 5.5 MiB/s rd, 3.9 MiB/s wr, 712 op/s 2024-05-27T07:28:34.581 INFO:teuthology.orchestra.run.smithi052.stderr:Inferring config /var/lib/ceph/73df2636-1bf8-11ef-bc9b-c7b262605968/mon.a/config 2024-05-27T07:28:34.706 INFO:journalctl@ceph.mds.g.smithi052.stdout:May 27 07:28:34 smithi052 podman[103102]: 2024-05-27 07:28:34.394701529 +0000 UTC m=+0.398263140 container init ff5d245d9ddc6752e3804e7b59f26b44e1aedad2f0ad09bae6127b410f919ae6 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-g, GIT_CLEAN=True, org.label-schema.license=GPLv2, ceph=True, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240521, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90) 2024-05-27T07:28:34.706 INFO:journalctl@ceph.mds.g.smithi052.stdout:May 27 07:28:34 smithi052 podman[103102]: 2024-05-27 07:28:34.397892633 +0000 UTC m=+0.401454239 container start ff5d245d9ddc6752e3804e7b59f26b44e1aedad2f0ad09bae6127b410f919ae6 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-g, maintainer=Guillaume Abrioux , RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, ceph=True, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240521, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, io.buildah.version=1.33.5) 2024-05-27T07:28:34.706 INFO:journalctl@ceph.mds.g.smithi052.stdout:May 27 07:28:34 smithi052 ceph-mds[103142]: -- 172.21.15.52:0/2857270029 <== mon.0 v2:172.21.15.52:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x559a11723a40 con 0x559a1176b180 2024-05-27T07:28:34.707 INFO:journalctl@ceph.mds.g.smithi052.stdout:May 27 07:28:34 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-g[103127]: starting mds.g at 2024-05-27T07:28:34.707 INFO:journalctl@ceph.mds.g.smithi052.stdout:May 27 07:28:34 smithi052 bash[103102]: ff5d245d9ddc6752e3804e7b59f26b44e1aedad2f0ad09bae6127b410f919ae6 2024-05-27T07:28:34.707 INFO:journalctl@ceph.mds.g.smithi052.stdout:May 27 07:28:34 smithi052 systemd[1]: Started Ceph mds.g for 73df2636-1bf8-11ef-bc9b-c7b262605968. 2024-05-27T07:28:34.957 INFO:journalctl@ceph.mds.g.smithi052.stdout:May 27 07:28:34 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-g[103127]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: In function 'std::set SnapClient::filter(const std::set&) const' thread 7f91a38ea700 time 2024-05-27T07:28:34.965010+0000 2024-05-27T07:28:34.957 INFO:journalctl@ceph.mds.g.smithi052.stdout:May 27 07:28:34 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-g[103127]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: 197: FAILED ceph_assert(cached_version > 0) 2024-05-27T07:28:34.958 INFO:journalctl@ceph.mds.g.smithi052.stdout:May 27 07:28:34 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-g[103127]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:28:34.958 INFO:journalctl@ceph.mds.g.smithi052.stdout:May 27 07:28:34 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-g[103127]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x127) [0x7f91b612fba6] 2024-05-27T07:28:34.958 INFO:journalctl@ceph.mds.g.smithi052.stdout:May 27 07:28:34 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-g[103127]: 2: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7f91b612fdd1] 2024-05-27T07:28:34.958 INFO:journalctl@ceph.mds.g.smithi052.stdout:May 27 07:28:34 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-g[103127]: 3: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x559a0f57d23d] 2024-05-27T07:28:34.958 INFO:journalctl@ceph.mds.g.smithi052.stdout:May 27 07:28:34 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-g[103127]: 4: (SnapRealm::build_snap_set() const+0x323) [0x559a0f568971] 2024-05-27T07:28:34.958 INFO:journalctl@ceph.mds.g.smithi052.stdout:May 27 07:28:34 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-g[103127]: 5: (SnapRealm::check_cache() const+0x13e) [0x559a0f566fa0] 2024-05-27T07:28:34.958 INFO:journalctl@ceph.mds.g.smithi052.stdout:May 27 07:28:34 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-g[103127]: 6: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x559a0f567651] 2024-05-27T07:28:34.958 INFO:journalctl@ceph.mds.g.smithi052.stdout:May 27 07:28:34 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-g[103127]: 7: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x559a0f4fc347] 2024-05-27T07:28:34.958 INFO:journalctl@ceph.mds.g.smithi052.stdout:May 27 07:28:34 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-g[103127]: 8: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x559a0f1f887e] 2024-05-27T07:28:34.958 INFO:journalctl@ceph.mds.g.smithi052.stdout:May 27 07:28:34 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-g[103127]: 9: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x2702) [0x559a0f1fde94] 2024-05-27T07:28:34.958 INFO:journalctl@ceph.mds.g.smithi052.stdout:May 27 07:28:34 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-g[103127]: 10: (EUpdate::replay(MDSRank*)+0x63) [0x559a0f208969] 2024-05-27T07:28:34.958 INFO:journalctl@ceph.mds.g.smithi052.stdout:May 27 07:28:34 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-g[103127]: 11: (MDLog::_replay_thread()+0x1f38) [0x559a0f5a148c] 2024-05-27T07:28:34.958 INFO:journalctl@ceph.mds.g.smithi052.stdout:May 27 07:28:34 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-g[103127]: 12: (MDLog::ReplayThread::entry()+0x11) [0x559a0f5aa9cb] 2024-05-27T07:28:34.959 INFO:journalctl@ceph.mds.g.smithi052.stdout:May 27 07:28:34 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-g[103127]: 13: (Thread::entry_wrapper()+0x43) [0x7f91b60fe72d] 2024-05-27T07:28:34.959 INFO:journalctl@ceph.mds.g.smithi052.stdout:May 27 07:28:34 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-g[103127]: 14: (Thread::_entry_func(void*)+0xd) [0x7f91b60fe749] 2024-05-27T07:28:34.959 INFO:journalctl@ceph.mds.g.smithi052.stdout:May 27 07:28:34 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-g[103127]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7f91b4ca21ca] 2024-05-27T07:28:34.959 INFO:journalctl@ceph.mds.g.smithi052.stdout:May 27 07:28:34 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-g[103127]: 16: clone() 2024-05-27T07:28:34.959 INFO:journalctl@ceph.mds.g.smithi052.stdout:May 27 07:28:34 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-g[103127]: 2024-05-27T07:28:34.966+0000 7f91a38ea700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: In function 'std::set SnapClient::filter(const std::set&) const' thread 7f91a38ea700 time 2024-05-27T07:28:34.965010+0000 2024-05-27T07:28:34.959 INFO:journalctl@ceph.mds.g.smithi052.stdout:May 27 07:28:34 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-g[103127]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: 197: FAILED ceph_assert(cached_version > 0) 2024-05-27T07:28:34.959 INFO:journalctl@ceph.mds.g.smithi052.stdout:May 27 07:28:34 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-g[103127]: 2024-05-27T07:28:34.959 INFO:journalctl@ceph.mds.g.smithi052.stdout:May 27 07:28:34 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-g[103127]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:28:34.960 INFO:journalctl@ceph.mds.g.smithi052.stdout:May 27 07:28:34 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-g[103127]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x127) [0x7f91b612fba6] 2024-05-27T07:28:34.960 INFO:journalctl@ceph.mds.g.smithi052.stdout:May 27 07:28:34 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-g[103127]: 2: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7f91b612fdd1] 2024-05-27T07:28:34.960 INFO:journalctl@ceph.mds.g.smithi052.stdout:May 27 07:28:34 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-g[103127]: 3: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x559a0f57d23d] 2024-05-27T07:28:34.960 INFO:journalctl@ceph.mds.g.smithi052.stdout:May 27 07:28:34 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-g[103127]: 4: (SnapRealm::build_snap_set() const+0x323) [0x559a0f568971] 2024-05-27T07:28:34.960 INFO:journalctl@ceph.mds.g.smithi052.stdout:May 27 07:28:34 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-g[103127]: 5: (SnapRealm::check_cache() const+0x13e) [0x559a0f566fa0] 2024-05-27T07:28:34.960 INFO:journalctl@ceph.mds.g.smithi052.stdout:May 27 07:28:34 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-g[103127]: 6: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x559a0f567651] 2024-05-27T07:28:34.960 INFO:journalctl@ceph.mds.g.smithi052.stdout:May 27 07:28:34 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-g[103127]: 7: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x559a0f4fc347] 2024-05-27T07:28:34.960 INFO:journalctl@ceph.mds.g.smithi052.stdout:May 27 07:28:34 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-g[103127]: 8: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x559a0f1f887e] 2024-05-27T07:28:34.960 INFO:journalctl@ceph.mds.g.smithi052.stdout:May 27 07:28:34 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-g[103127]: 9: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x2702) [0x559a0f1fde94] 2024-05-27T07:28:34.960 INFO:journalctl@ceph.mds.g.smithi052.stdout:May 27 07:28:34 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-g[103127]: 10: (EUpdate::replay(MDSRank*)+0x63) [0x559a0f208969] 2024-05-27T07:28:34.960 INFO:journalctl@ceph.mds.g.smithi052.stdout:May 27 07:28:34 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-g[103127]: 11: (MDLog::_replay_thread()+0x1f38) [0x559a0f5a148c] 2024-05-27T07:28:34.960 INFO:journalctl@ceph.mds.g.smithi052.stdout:May 27 07:28:34 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-g[103127]: 12: (MDLog::ReplayThread::entry()+0x11) [0x559a0f5aa9cb] 2024-05-27T07:28:34.960 INFO:journalctl@ceph.mds.g.smithi052.stdout:May 27 07:28:34 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-g[103127]: 13: (Thread::entry_wrapper()+0x43) [0x7f91b60fe72d] 2024-05-27T07:28:34.960 INFO:journalctl@ceph.mds.g.smithi052.stdout:May 27 07:28:34 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-g[103127]: 14: (Thread::_entry_func(void*)+0xd) [0x7f91b60fe749] 2024-05-27T07:28:34.961 INFO:journalctl@ceph.mds.g.smithi052.stdout:May 27 07:28:34 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-g[103127]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7f91b4ca21ca] 2024-05-27T07:28:34.961 INFO:journalctl@ceph.mds.g.smithi052.stdout:May 27 07:28:34 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-g[103127]: 16: clone() 2024-05-27T07:28:34.961 INFO:journalctl@ceph.mds.g.smithi052.stdout:May 27 07:28:34 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-g[103127]: 2024-05-27T07:28:34.961 INFO:journalctl@ceph.mds.g.smithi052.stdout:May 27 07:28:34 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-g[103127]: *** Caught signal (Aborted) ** 2024-05-27T07:28:34.961 INFO:journalctl@ceph.mds.g.smithi052.stdout:May 27 07:28:34 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-g[103127]: in thread 7f91a38ea700 thread_name:md_log_replay 2024-05-27T07:28:35.205 INFO:tasks.workunit.client.0.smithi052.stdout: 29 778741 226 554305 370 433166 1503 2024-05-27T07:28:35.230 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:28:35 smithi052 ceph-mon[31955]: daemon mds.g restarted 2024-05-27T07:28:35.231 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:28:35 smithi052 ceph-mon[31955]: osdmap e95: 12 total, 12 up, 12 in 2024-05-27T07:28:35.231 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:28:35 smithi052 ceph-mon[31955]: mds.? [v2:172.21.15.52:6838/1275478522,v1:172.21.15.52:6839/1275478522] up:boot 2024-05-27T07:28:35.231 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:28:35 smithi052 ceph-mon[31955]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-05-27T07:28:35.231 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:28:35 smithi052 ceph-mon[31955]: from='mgr.15333 172.21.15.52:0/416389082' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2024-05-27T07:28:35.231 INFO:journalctl@ceph.mds.g.smithi052.stdout:May 27 07:28:34 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-g[103127]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:28:35.231 INFO:journalctl@ceph.mds.g.smithi052.stdout:May 27 07:28:34 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-g[103127]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f91b4cacd20] 2024-05-27T07:28:35.231 INFO:journalctl@ceph.mds.g.smithi052.stdout:May 27 07:28:34 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-g[103127]: 2: gsignal() 2024-05-27T07:28:35.231 INFO:journalctl@ceph.mds.g.smithi052.stdout:May 27 07:28:34 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-g[103127]: 3: abort() 2024-05-27T07:28:35.231 INFO:journalctl@ceph.mds.g.smithi052.stdout:May 27 07:28:34 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-g[103127]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x26e) [0x7f91b612fced] 2024-05-27T07:28:35.231 INFO:journalctl@ceph.mds.g.smithi052.stdout:May 27 07:28:34 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-g[103127]: 5: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7f91b612fdd1] 2024-05-27T07:28:35.232 INFO:journalctl@ceph.mds.g.smithi052.stdout:May 27 07:28:34 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-g[103127]: 6: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x559a0f57d23d] 2024-05-27T07:28:35.232 INFO:journalctl@ceph.mds.g.smithi052.stdout:May 27 07:28:34 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-g[103127]: 7: (SnapRealm::build_snap_set() const+0x323) [0x559a0f568971] 2024-05-27T07:28:35.232 INFO:journalctl@ceph.mds.g.smithi052.stdout:May 27 07:28:34 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-g[103127]: 8: (SnapRealm::check_cache() const+0x13e) [0x559a0f566fa0] 2024-05-27T07:28:35.232 INFO:journalctl@ceph.mds.g.smithi052.stdout:May 27 07:28:34 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-g[103127]: 9: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x559a0f567651] 2024-05-27T07:28:35.232 INFO:journalctl@ceph.mds.g.smithi052.stdout:May 27 07:28:34 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-g[103127]: 10: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x559a0f4fc347] 2024-05-27T07:28:35.232 INFO:journalctl@ceph.mds.g.smithi052.stdout:May 27 07:28:34 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-g[103127]: 11: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x559a0f1f887e] 2024-05-27T07:28:35.232 INFO:journalctl@ceph.mds.g.smithi052.stdout:May 27 07:28:34 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-g[103127]: 12: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x2702) [0x559a0f1fde94] 2024-05-27T07:28:35.233 INFO:journalctl@ceph.mds.g.smithi052.stdout:May 27 07:28:34 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-g[103127]: 13: (EUpdate::replay(MDSRank*)+0x63) [0x559a0f208969] 2024-05-27T07:28:35.233 INFO:journalctl@ceph.mds.g.smithi052.stdout:May 27 07:28:34 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-g[103127]: 14: (MDLog::_replay_thread()+0x1f38) [0x559a0f5a148c] 2024-05-27T07:28:35.233 INFO:journalctl@ceph.mds.g.smithi052.stdout:May 27 07:28:34 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-g[103127]: 15: (MDLog::ReplayThread::entry()+0x11) [0x559a0f5aa9cb] 2024-05-27T07:28:35.233 INFO:journalctl@ceph.mds.g.smithi052.stdout:May 27 07:28:34 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-g[103127]: 16: (Thread::entry_wrapper()+0x43) [0x7f91b60fe72d] 2024-05-27T07:28:35.233 INFO:journalctl@ceph.mds.g.smithi052.stdout:May 27 07:28:34 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-g[103127]: 17: (Thread::_entry_func(void*)+0xd) [0x7f91b60fe749] 2024-05-27T07:28:35.233 INFO:journalctl@ceph.mds.g.smithi052.stdout:May 27 07:28:34 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-g[103127]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7f91b4ca21ca] 2024-05-27T07:28:35.234 INFO:journalctl@ceph.mds.g.smithi052.stdout:May 27 07:28:34 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-g[103127]: 19: clone() 2024-05-27T07:28:35.234 INFO:journalctl@ceph.mds.g.smithi052.stdout:May 27 07:28:34 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-g[103127]: 2024-05-27T07:28:34.968+0000 7f91a38ea700 -1 *** Caught signal (Aborted) ** 2024-05-27T07:28:35.234 INFO:journalctl@ceph.mds.g.smithi052.stdout:May 27 07:28:34 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-g[103127]: in thread 7f91a38ea700 thread_name:md_log_replay 2024-05-27T07:28:35.234 INFO:journalctl@ceph.mds.g.smithi052.stdout:May 27 07:28:34 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-g[103127]: 2024-05-27T07:28:35.234 INFO:journalctl@ceph.mds.g.smithi052.stdout:May 27 07:28:34 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-g[103127]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:28:35.234 INFO:journalctl@ceph.mds.g.smithi052.stdout:May 27 07:28:34 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-g[103127]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f91b4cacd20] 2024-05-27T07:28:35.235 INFO:journalctl@ceph.mds.g.smithi052.stdout:May 27 07:28:34 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-g[103127]: 2: gsignal() 2024-05-27T07:28:35.235 INFO:journalctl@ceph.mds.g.smithi052.stdout:May 27 07:28:34 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-g[103127]: 3: abort() 2024-05-27T07:28:35.235 INFO:journalctl@ceph.mds.g.smithi052.stdout:May 27 07:28:34 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-g[103127]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x26e) [0x7f91b612fced] 2024-05-27T07:28:35.236 INFO:journalctl@ceph.mds.g.smithi052.stdout:May 27 07:28:34 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-g[103127]: 5: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7f91b612fdd1] 2024-05-27T07:28:35.236 INFO:journalctl@ceph.mds.g.smithi052.stdout:May 27 07:28:34 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-g[103127]: 6: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x559a0f57d23d] 2024-05-27T07:28:35.237 INFO:journalctl@ceph.mds.g.smithi052.stdout:May 27 07:28:34 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-g[103127]: 7: (SnapRealm::build_snap_set() const+0x323) [0x559a0f568971] 2024-05-27T07:28:35.237 INFO:journalctl@ceph.mds.g.smithi052.stdout:May 27 07:28:34 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-g[103127]: 8: (SnapRealm::check_cache() const+0x13e) [0x559a0f566fa0] 2024-05-27T07:28:35.237 INFO:journalctl@ceph.mds.g.smithi052.stdout:May 27 07:28:34 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-g[103127]: 9: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x559a0f567651] 2024-05-27T07:28:35.237 INFO:journalctl@ceph.mds.g.smithi052.stdout:May 27 07:28:34 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-g[103127]: 10: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x559a0f4fc347] 2024-05-27T07:28:35.237 INFO:journalctl@ceph.mds.g.smithi052.stdout:May 27 07:28:34 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-g[103127]: 11: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x559a0f1f887e] 2024-05-27T07:28:35.237 INFO:journalctl@ceph.mds.g.smithi052.stdout:May 27 07:28:34 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-g[103127]: 12: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x2702) [0x559a0f1fde94] 2024-05-27T07:28:35.238 INFO:journalctl@ceph.mds.g.smithi052.stdout:May 27 07:28:34 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-g[103127]: 13: (EUpdate::replay(MDSRank*)+0x63) [0x559a0f208969] 2024-05-27T07:28:35.238 INFO:journalctl@ceph.mds.g.smithi052.stdout:May 27 07:28:34 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-g[103127]: 14: (MDLog::_replay_thread()+0x1f38) [0x559a0f5a148c] 2024-05-27T07:28:35.238 INFO:journalctl@ceph.mds.g.smithi052.stdout:May 27 07:28:34 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-g[103127]: 15: (MDLog::ReplayThread::entry()+0x11) [0x559a0f5aa9cb] 2024-05-27T07:28:35.238 INFO:journalctl@ceph.mds.g.smithi052.stdout:May 27 07:28:34 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-g[103127]: 16: (Thread::entry_wrapper()+0x43) [0x7f91b60fe72d] 2024-05-27T07:28:35.238 INFO:journalctl@ceph.mds.g.smithi052.stdout:May 27 07:28:34 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-g[103127]: 17: (Thread::_entry_func(void*)+0xd) [0x7f91b60fe749] 2024-05-27T07:28:35.238 INFO:journalctl@ceph.mds.g.smithi052.stdout:May 27 07:28:34 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-g[103127]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7f91b4ca21ca] 2024-05-27T07:28:35.239 INFO:journalctl@ceph.mds.g.smithi052.stdout:May 27 07:28:34 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-g[103127]: 19: clone() 2024-05-27T07:28:35.239 INFO:journalctl@ceph.mds.g.smithi052.stdout:May 27 07:28:34 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-g[103127]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-05-27T07:28:35.239 INFO:journalctl@ceph.mds.g.smithi052.stdout:May 27 07:28:34 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-g[103127]: 2024-05-27T07:28:35.239 INFO:journalctl@ceph.mds.g.smithi052.stdout:May 27 07:28:34 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-g[103127]: -1> 2024-05-27T07:28:34.966+0000 7f91a38ea700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: In function 'std::set SnapClient::filter(const std::set&) const' thread 7f91a38ea700 time 2024-05-27T07:28:34.965010+0000 2024-05-27T07:28:35.239 INFO:journalctl@ceph.mds.g.smithi052.stdout:May 27 07:28:34 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-g[103127]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: 197: FAILED ceph_assert(cached_version > 0) 2024-05-27T07:28:35.239 INFO:journalctl@ceph.mds.g.smithi052.stdout:May 27 07:28:34 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-g[103127]: 2024-05-27T07:28:35.240 INFO:journalctl@ceph.mds.g.smithi052.stdout:May 27 07:28:34 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-g[103127]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:28:35.240 INFO:journalctl@ceph.mds.g.smithi052.stdout:May 27 07:28:34 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-g[103127]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x127) [0x7f91b612fba6] 2024-05-27T07:28:35.240 INFO:journalctl@ceph.mds.g.smithi052.stdout:May 27 07:28:34 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-g[103127]: 2: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7f91b612fdd1] 2024-05-27T07:28:35.240 INFO:journalctl@ceph.mds.g.smithi052.stdout:May 27 07:28:34 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-g[103127]: 3: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x559a0f57d23d] 2024-05-27T07:28:35.240 INFO:journalctl@ceph.mds.g.smithi052.stdout:May 27 07:28:34 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-g[103127]: 4: (SnapRealm::build_snap_set() const+0x323) [0x559a0f568971] 2024-05-27T07:28:35.240 INFO:journalctl@ceph.mds.g.smithi052.stdout:May 27 07:28:34 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-g[103127]: 5: (SnapRealm::check_cache() const+0x13e) [0x559a0f566fa0] 2024-05-27T07:28:35.241 INFO:journalctl@ceph.mds.g.smithi052.stdout:May 27 07:28:34 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-g[103127]: 6: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x559a0f567651] 2024-05-27T07:28:35.241 INFO:journalctl@ceph.mds.g.smithi052.stdout:May 27 07:28:34 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-g[103127]: 7: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x559a0f4fc347] 2024-05-27T07:28:35.241 INFO:journalctl@ceph.mds.g.smithi052.stdout:May 27 07:28:34 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-g[103127]: 8: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x559a0f1f887e] 2024-05-27T07:28:35.241 INFO:journalctl@ceph.mds.g.smithi052.stdout:May 27 07:28:34 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-g[103127]: 9: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x2702) [0x559a0f1fde94] 2024-05-27T07:28:35.241 INFO:journalctl@ceph.mds.g.smithi052.stdout:May 27 07:28:34 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-g[103127]: 10: (EUpdate::replay(MDSRank*)+0x63) [0x559a0f208969] 2024-05-27T07:28:35.241 INFO:journalctl@ceph.mds.g.smithi052.stdout:May 27 07:28:34 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-g[103127]: 11: (MDLog::_replay_thread()+0x1f38) [0x559a0f5a148c] 2024-05-27T07:28:35.242 INFO:journalctl@ceph.mds.g.smithi052.stdout:May 27 07:28:34 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-g[103127]: 12: (MDLog::ReplayThread::entry()+0x11) [0x559a0f5aa9cb] 2024-05-27T07:28:35.242 INFO:journalctl@ceph.mds.g.smithi052.stdout:May 27 07:28:34 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-g[103127]: 13: (Thread::entry_wrapper()+0x43) [0x7f91b60fe72d] 2024-05-27T07:28:35.242 INFO:journalctl@ceph.mds.g.smithi052.stdout:May 27 07:28:34 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-g[103127]: 14: (Thread::_entry_func(void*)+0xd) [0x7f91b60fe749] 2024-05-27T07:28:35.242 INFO:journalctl@ceph.mds.g.smithi052.stdout:May 27 07:28:34 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-g[103127]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7f91b4ca21ca] 2024-05-27T07:28:35.242 INFO:journalctl@ceph.mds.g.smithi052.stdout:May 27 07:28:34 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-g[103127]: 16: clone() 2024-05-27T07:28:35.242 INFO:journalctl@ceph.mds.g.smithi052.stdout:May 27 07:28:34 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-g[103127]: 2024-05-27T07:28:35.243 INFO:journalctl@ceph.mds.g.smithi052.stdout:May 27 07:28:34 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-g[103127]: 0> 2024-05-27T07:28:34.968+0000 7f91a38ea700 -1 *** Caught signal (Aborted) ** 2024-05-27T07:28:35.243 INFO:journalctl@ceph.mds.g.smithi052.stdout:May 27 07:28:34 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-g[103127]: in thread 7f91a38ea700 thread_name:md_log_replay 2024-05-27T07:28:35.243 INFO:journalctl@ceph.mds.g.smithi052.stdout:May 27 07:28:34 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-g[103127]: 2024-05-27T07:28:35.243 INFO:journalctl@ceph.mds.g.smithi052.stdout:May 27 07:28:34 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-g[103127]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:28:35.243 INFO:journalctl@ceph.mds.g.smithi052.stdout:May 27 07:28:34 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-g[103127]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f91b4cacd20] 2024-05-27T07:28:35.243 INFO:journalctl@ceph.mds.g.smithi052.stdout:May 27 07:28:34 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-g[103127]: 2: gsignal() 2024-05-27T07:28:35.244 INFO:journalctl@ceph.mds.g.smithi052.stdout:May 27 07:28:34 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-g[103127]: 3: abort() 2024-05-27T07:28:35.244 INFO:journalctl@ceph.mds.g.smithi052.stdout:May 27 07:28:34 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-g[103127]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x26e) [0x7f91b612fced] 2024-05-27T07:28:35.244 INFO:journalctl@ceph.mds.g.smithi052.stdout:May 27 07:28:34 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-g[103127]: 5: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7f91b612fdd1] 2024-05-27T07:28:35.244 INFO:journalctl@ceph.mds.g.smithi052.stdout:May 27 07:28:34 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-g[103127]: 6: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x559a0f57d23d] 2024-05-27T07:28:35.244 INFO:journalctl@ceph.mds.g.smithi052.stdout:May 27 07:28:34 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-g[103127]: 7: (SnapRealm::build_snap_set() const+0x323) [0x559a0f568971] 2024-05-27T07:28:35.244 INFO:journalctl@ceph.mds.g.smithi052.stdout:May 27 07:28:34 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-g[103127]: 8: (SnapRealm::check_cache() const+0x13e) [0x559a0f566fa0] 2024-05-27T07:28:35.245 INFO:journalctl@ceph.mds.g.smithi052.stdout:May 27 07:28:34 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-g[103127]: 9: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x559a0f567651] 2024-05-27T07:28:35.245 INFO:journalctl@ceph.mds.g.smithi052.stdout:May 27 07:28:34 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-g[103127]: 10: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x559a0f4fc347] 2024-05-27T07:28:35.245 INFO:journalctl@ceph.mds.g.smithi052.stdout:May 27 07:28:34 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-g[103127]: 11: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x559a0f1f887e] 2024-05-27T07:28:35.245 INFO:journalctl@ceph.mds.g.smithi052.stdout:May 27 07:28:34 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-g[103127]: 12: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x2702) [0x559a0f1fde94] 2024-05-27T07:28:35.245 INFO:journalctl@ceph.mds.g.smithi052.stdout:May 27 07:28:34 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-g[103127]: 13: (EUpdate::replay(MDSRank*)+0x63) [0x559a0f208969] 2024-05-27T07:28:35.245 INFO:journalctl@ceph.mds.g.smithi052.stdout:May 27 07:28:34 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-g[103127]: 14: (MDLog::_replay_thread()+0x1f38) [0x559a0f5a148c] 2024-05-27T07:28:35.246 INFO:journalctl@ceph.mds.g.smithi052.stdout:May 27 07:28:34 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-g[103127]: 15: (MDLog::ReplayThread::entry()+0x11) [0x559a0f5aa9cb] 2024-05-27T07:28:35.246 INFO:journalctl@ceph.mds.g.smithi052.stdout:May 27 07:28:34 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-g[103127]: 16: (Thread::entry_wrapper()+0x43) [0x7f91b60fe72d] 2024-05-27T07:28:35.246 INFO:journalctl@ceph.mds.g.smithi052.stdout:May 27 07:28:34 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-g[103127]: 17: (Thread::_entry_func(void*)+0xd) [0x7f91b60fe749] 2024-05-27T07:28:35.246 INFO:journalctl@ceph.mds.g.smithi052.stdout:May 27 07:28:34 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-g[103127]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7f91b4ca21ca] 2024-05-27T07:28:35.246 INFO:journalctl@ceph.mds.g.smithi052.stdout:May 27 07:28:34 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-g[103127]: 19: clone() 2024-05-27T07:28:35.246 INFO:journalctl@ceph.mds.g.smithi052.stdout:May 27 07:28:34 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-g[103127]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-05-27T07:28:35.247 INFO:journalctl@ceph.mds.g.smithi052.stdout:May 27 07:28:34 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-g[103127]: 2024-05-27T07:28:35.247 INFO:journalctl@ceph.mds.g.smithi052.stdout:May 27 07:28:34 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-g[103127]: -1> 2024-05-27T07:28:34.966+0000 7f91a38ea700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: In function 'std::set SnapClient::filter(const std::set&) const' thread 7f91a38ea700 time 2024-05-27T07:28:34.965010+0000 2024-05-27T07:28:35.247 INFO:journalctl@ceph.mds.g.smithi052.stdout:May 27 07:28:34 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-g[103127]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: 197: FAILED ceph_assert(cached_version > 0) 2024-05-27T07:28:35.247 INFO:journalctl@ceph.mds.g.smithi052.stdout:May 27 07:28:34 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-g[103127]: 2024-05-27T07:28:35.248 INFO:journalctl@ceph.mds.g.smithi052.stdout:May 27 07:28:34 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-g[103127]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:28:35.248 INFO:journalctl@ceph.mds.g.smithi052.stdout:May 27 07:28:34 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-g[103127]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x127) [0x7f91b612fba6] 2024-05-27T07:28:35.248 INFO:journalctl@ceph.mds.g.smithi052.stdout:May 27 07:28:34 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-g[103127]: 2: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7f91b612fdd1] 2024-05-27T07:28:35.248 INFO:journalctl@ceph.mds.g.smithi052.stdout:May 27 07:28:34 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-g[103127]: 3: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x559a0f57d23d] 2024-05-27T07:28:35.248 INFO:journalctl@ceph.mds.g.smithi052.stdout:May 27 07:28:34 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-g[103127]: 4: (SnapRealm::build_snap_set() const+0x323) [0x559a0f568971] 2024-05-27T07:28:35.248 INFO:journalctl@ceph.mds.g.smithi052.stdout:May 27 07:28:34 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-g[103127]: 5: (SnapRealm::check_cache() const+0x13e) [0x559a0f566fa0] 2024-05-27T07:28:35.249 INFO:journalctl@ceph.mds.g.smithi052.stdout:May 27 07:28:34 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-g[103127]: 6: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x559a0f567651] 2024-05-27T07:28:35.249 INFO:journalctl@ceph.mds.g.smithi052.stdout:May 27 07:28:34 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-g[103127]: 7: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x559a0f4fc347] 2024-05-27T07:28:35.249 INFO:journalctl@ceph.mds.g.smithi052.stdout:May 27 07:28:34 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-g[103127]: 8: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x559a0f1f887e] 2024-05-27T07:28:35.249 INFO:journalctl@ceph.mds.g.smithi052.stdout:May 27 07:28:34 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-g[103127]: 9: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x2702) [0x559a0f1fde94] 2024-05-27T07:28:35.250 INFO:journalctl@ceph.mds.g.smithi052.stdout:May 27 07:28:34 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-g[103127]: 10: (EUpdate::replay(MDSRank*)+0x63) [0x559a0f208969] 2024-05-27T07:28:35.250 INFO:journalctl@ceph.mds.g.smithi052.stdout:May 27 07:28:34 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-g[103127]: 11: (MDLog::_replay_thread()+0x1f38) [0x559a0f5a148c] 2024-05-27T07:28:35.250 INFO:journalctl@ceph.mds.g.smithi052.stdout:May 27 07:28:34 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-g[103127]: 12: (MDLog::ReplayThread::entry()+0x11) [0x559a0f5aa9cb] 2024-05-27T07:28:35.250 INFO:journalctl@ceph.mds.g.smithi052.stdout:May 27 07:28:34 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-g[103127]: 13: (Thread::entry_wrapper()+0x43) [0x7f91b60fe72d] 2024-05-27T07:28:35.250 INFO:journalctl@ceph.mds.g.smithi052.stdout:May 27 07:28:34 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-g[103127]: 14: (Thread::_entry_func(void*)+0xd) [0x7f91b60fe749] 2024-05-27T07:28:35.251 INFO:journalctl@ceph.mds.g.smithi052.stdout:May 27 07:28:34 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-g[103127]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7f91b4ca21ca] 2024-05-27T07:28:35.251 INFO:journalctl@ceph.mds.g.smithi052.stdout:May 27 07:28:34 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-g[103127]: 16: clone() 2024-05-27T07:28:35.251 INFO:journalctl@ceph.mds.g.smithi052.stdout:May 27 07:28:34 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-g[103127]: 2024-05-27T07:28:35.251 INFO:journalctl@ceph.mds.g.smithi052.stdout:May 27 07:28:34 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-g[103127]: 0> 2024-05-27T07:28:34.968+0000 7f91a38ea700 -1 *** Caught signal (Aborted) ** 2024-05-27T07:28:35.252 INFO:journalctl@ceph.mds.g.smithi052.stdout:May 27 07:28:34 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-g[103127]: in thread 7f91a38ea700 thread_name:md_log_replay 2024-05-27T07:28:35.252 INFO:journalctl@ceph.mds.g.smithi052.stdout:May 27 07:28:34 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-g[103127]: 2024-05-27T07:28:35.252 INFO:journalctl@ceph.mds.g.smithi052.stdout:May 27 07:28:34 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-g[103127]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:28:35.252 INFO:journalctl@ceph.mds.g.smithi052.stdout:May 27 07:28:34 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-g[103127]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f91b4cacd20] 2024-05-27T07:28:35.252 INFO:journalctl@ceph.mds.g.smithi052.stdout:May 27 07:28:34 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-g[103127]: 2: gsignal() 2024-05-27T07:28:35.253 INFO:journalctl@ceph.mds.g.smithi052.stdout:May 27 07:28:34 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-g[103127]: 3: abort() 2024-05-27T07:28:35.253 INFO:journalctl@ceph.mds.g.smithi052.stdout:May 27 07:28:34 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-g[103127]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x26e) [0x7f91b612fced] 2024-05-27T07:28:35.253 INFO:journalctl@ceph.mds.g.smithi052.stdout:May 27 07:28:34 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-g[103127]: 5: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7f91b612fdd1] 2024-05-27T07:28:35.253 INFO:journalctl@ceph.mds.g.smithi052.stdout:May 27 07:28:34 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-g[103127]: 6: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x559a0f57d23d] 2024-05-27T07:28:35.253 INFO:journalctl@ceph.mds.g.smithi052.stdout:May 27 07:28:34 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-g[103127]: 7: (SnapRealm::build_snap_set() const+0x323) [0x559a0f568971] 2024-05-27T07:28:35.253 INFO:journalctl@ceph.mds.g.smithi052.stdout:May 27 07:28:34 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-g[103127]: 8: (SnapRealm::check_cache() const+0x13e) [0x559a0f566fa0] 2024-05-27T07:28:35.254 INFO:journalctl@ceph.mds.g.smithi052.stdout:May 27 07:28:34 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-g[103127]: 9: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x559a0f567651] 2024-05-27T07:28:35.254 INFO:journalctl@ceph.mds.g.smithi052.stdout:May 27 07:28:34 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-g[103127]: 10: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x559a0f4fc347] 2024-05-27T07:28:35.254 INFO:journalctl@ceph.mds.g.smithi052.stdout:May 27 07:28:34 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-g[103127]: 11: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x559a0f1f887e] 2024-05-27T07:28:35.254 INFO:journalctl@ceph.mds.g.smithi052.stdout:May 27 07:28:34 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-g[103127]: 12: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x2702) [0x559a0f1fde94] 2024-05-27T07:28:35.254 INFO:journalctl@ceph.mds.g.smithi052.stdout:May 27 07:28:34 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-g[103127]: 13: (EUpdate::replay(MDSRank*)+0x63) [0x559a0f208969] 2024-05-27T07:28:35.255 INFO:journalctl@ceph.mds.g.smithi052.stdout:May 27 07:28:34 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-g[103127]: 14: (MDLog::_replay_thread()+0x1f38) [0x559a0f5a148c] 2024-05-27T07:28:35.255 INFO:journalctl@ceph.mds.g.smithi052.stdout:May 27 07:28:34 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-g[103127]: 15: (MDLog::ReplayThread::entry()+0x11) [0x559a0f5aa9cb] 2024-05-27T07:28:35.255 INFO:journalctl@ceph.mds.g.smithi052.stdout:May 27 07:28:34 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-g[103127]: 16: (Thread::entry_wrapper()+0x43) [0x7f91b60fe72d] 2024-05-27T07:28:35.255 INFO:journalctl@ceph.mds.g.smithi052.stdout:May 27 07:28:34 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-g[103127]: 17: (Thread::_entry_func(void*)+0xd) [0x7f91b60fe749] 2024-05-27T07:28:35.255 INFO:journalctl@ceph.mds.g.smithi052.stdout:May 27 07:28:34 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-g[103127]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7f91b4ca21ca] 2024-05-27T07:28:35.255 INFO:journalctl@ceph.mds.g.smithi052.stdout:May 27 07:28:34 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-g[103127]: 19: clone() 2024-05-27T07:28:35.256 INFO:journalctl@ceph.mds.g.smithi052.stdout:May 27 07:28:34 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-g[103127]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-05-27T07:28:35.256 INFO:journalctl@ceph.mds.g.smithi052.stdout:May 27 07:28:34 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-g[103127]: 2024-05-27T07:28:35.351 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:28:35 smithi173 ceph-mon[37000]: daemon mds.g restarted 2024-05-27T07:28:35.351 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:28:35 smithi173 ceph-mon[37000]: osdmap e95: 12 total, 12 up, 12 in 2024-05-27T07:28:35.352 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:28:35 smithi173 ceph-mon[37000]: mds.? [v2:172.21.15.52:6838/1275478522,v1:172.21.15.52:6839/1275478522] up:boot 2024-05-27T07:28:35.352 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:28:35 smithi173 ceph-mon[37000]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-05-27T07:28:35.352 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:28:35 smithi173 ceph-mon[37000]: from='mgr.15333 172.21.15.52:0/416389082' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2024-05-27T07:28:35.439 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:28:35 smithi105 ceph-mon[35977]: daemon mds.g restarted 2024-05-27T07:28:35.439 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:28:35 smithi105 ceph-mon[35977]: osdmap e95: 12 total, 12 up, 12 in 2024-05-27T07:28:35.439 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:28:35 smithi105 ceph-mon[35977]: mds.? [v2:172.21.15.52:6838/1275478522,v1:172.21.15.52:6839/1275478522] up:boot 2024-05-27T07:28:35.440 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:28:35 smithi105 ceph-mon[35977]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-05-27T07:28:35.440 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:28:35 smithi105 ceph-mon[35977]: from='mgr.15333 172.21.15.52:0/416389082' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2024-05-27T07:28:35.953 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:28:35.962+0000 7f08ecff9640 1 -- 172.21.15.52:0/1464256265 <== mds.0 v2:172.21.15.105:6838/500348151 1 ==== command_reply(tid 0: 0 ) ==== 8+0+25 (crc 0 0 0) 0x55eeaa198360 con 0x55eea9e1d1c0 2024-05-27T07:28:35.954 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:28:35.962+0000 7f08ecff9640 10 client.25748 handle_command_reply: tid=0 2024-05-27T07:28:35.954 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:28:35.962+0000 7f0912853740 1 client.25748 shutdown 2024-05-27T07:28:35.954 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:28:35.962+0000 7f0912853740 1 -- 172.21.15.52:0/1464256265 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55eea9e684d0 msgr2=0x55eeaa239b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:28:35.954 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:28:35.962+0000 7f0912853740 1 --2- 172.21.15.52:0/1464256265 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55eea9e684d0 0x55eeaa239b70 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x55eeaa227640 tx=0x7f08f4005e20 comp rx=0 tx=0).stop 2024-05-27T07:28:35.954 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:28:35.962+0000 7f0912853740 1 -- 172.21.15.52:0/1464256265 shutdown_connections 2024-05-27T07:28:35.954 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:28:35.962+0000 7f0912853740 1 --2- 172.21.15.52:0/1464256265 >> [v2:172.21.15.105:6838/500348151,v1:172.21.15.105:6839/500348151] conn(0x55eea9e1d1c0 0x55eeaa2b0f30 crc :-1 s=READY pgs=82 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:28:35.954 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:28:35.962+0000 7f0912853740 1 --2- 172.21.15.52:0/1464256265 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55eea9e891b0 0x55eeaa23a0b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:28:35.955 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:28:35.962+0000 7f0912853740 1 --2- 172.21.15.52:0/1464256265 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55eea9e684d0 0x55eeaa239b70 unknown :-1 s=CLOSED pgs=98 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:28:35.955 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:28:35.962+0000 7f08ecff9640 0 client.25748 ms_handle_reset on v2:172.21.15.105:6838/500348151 2024-05-27T07:28:35.955 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:28:35.962+0000 7f0912853740 1 --2- 172.21.15.52:0/1464256265 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55eea9e602f0 0x55eeaa239630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:28:35.955 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:28:35.963+0000 7f0912853740 1 -- 172.21.15.52:0/1464256265 >> 172.21.15.52:0/1464256265 conn(0x55eea9e7d300 msgr2=0x55eeaa24b5b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:28:35.955 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:28:35.963+0000 7f0912853740 1 -- 172.21.15.52:0/1464256265 shutdown_connections 2024-05-27T07:28:35.955 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:28:35.963+0000 7f0912853740 1 -- 172.21.15.52:0/1464256265 wait complete. 2024-05-27T07:28:35.955 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:28:35.963+0000 7f0912853740 20 client.25748 trim_cache size 0 max 16384 2024-05-27T07:28:35.955 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stdout:{ 2024-05-27T07:28:35.955 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stdout: "return_code": 0 2024-05-27T07:28:35.956 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stdout:} 2024-05-27T07:28:35.956 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:28:35.963+0000 7f090b99d640 1 -- 172.21.15.52:0/777718191 >> [v2:172.21.15.52:6800/1822318384,v1:172.21.15.52:6801/1822318384] conn(0x7f08f00b29d0 msgr2=0x7f08f00b4ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:28:35.956 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:28:35.963+0000 7f090b99d640 1 --2- 172.21.15.52:0/777718191 >> [v2:172.21.15.52:6800/1822318384,v1:172.21.15.52:6801/1822318384] conn(0x7f08f00b29d0 0x7f08f00b4ee0 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f09000df250 tx=0x7f09000e5900 comp rx=0 tx=0).stop 2024-05-27T07:28:35.956 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:28:35.963+0000 7f090b99d640 1 -- 172.21.15.52:0/777718191 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f09041032e0 msgr2=0x7f09041a2710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:28:35.956 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:28:35.963+0000 7f090b99d640 1 --2- 172.21.15.52:0/777718191 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f09041032e0 0x7f09041a2710 secure :-1 s=READY pgs=579 cs=0 l=1 rev1=1 crypto rx=0x7f0900009990 tx=0x7f0900009e50 comp rx=0 tx=0).stop 2024-05-27T07:28:35.957 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:28:35.963+0000 7f090b99d640 1 -- 172.21.15.52:0/777718191 shutdown_connections 2024-05-27T07:28:35.957 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:28:35.963+0000 7f090b99d640 1 --2- 172.21.15.52:0/777718191 >> [v2:172.21.15.52:6800/1822318384,v1:172.21.15.52:6801/1822318384] conn(0x7f08f00b29d0 0x7f08f00b4ee0 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:28:35.957 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:28:35.963+0000 7f090b99d640 1 --2- 172.21.15.52:0/777718191 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f0904109030 0x7f090419c970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:28:35.957 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:28:35.963+0000 7f090b99d640 1 --2- 172.21.15.52:0/777718191 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f0904108b90 0x7f09041a2c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:28:35.957 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:28:35.963+0000 7f090b99d640 1 --2- 172.21.15.52:0/777718191 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f09041032e0 0x7f09041a2710 unknown :-1 s=CLOSED pgs=579 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:28:35.957 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:28:35.964+0000 7f090b99d640 1 -- 172.21.15.52:0/777718191 >> 172.21.15.52:0/777718191 conn(0x7f09040782d0 msgr2=0x7f09041012b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:28:35.957 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:28:35.964+0000 7f090b99d640 1 -- 172.21.15.52:0/777718191 shutdown_connections 2024-05-27T07:28:35.957 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:28:35.964+0000 7f090b99d640 1 -- 172.21.15.52:0/777718191 wait complete. 2024-05-27T07:28:36.351 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:28:36 smithi173 ceph-mon[37000]: pgmap v86: 129 pgs: 129 active+clean; 102 MiB data, 586 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 MiB/s rd, 2.6 MiB/s wr, 1.12k op/s 2024-05-27T07:28:36.436 INFO:journalctl@ceph.mds.g.smithi052.stdout:May 27 07:28:36 smithi052 podman[103206]: 2024-05-27 07:28:36.159770061 +0000 UTC m=+1.162511387 container died ff5d245d9ddc6752e3804e7b59f26b44e1aedad2f0ad09bae6127b410f919ae6 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-g, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, GIT_BRANCH=HEAD, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, org.label-schema.build-date=20240521, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, GIT_CLEAN=True) 2024-05-27T07:28:36.436 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:28:36 smithi052 ceph-mon[31955]: pgmap v86: 129 pgs: 129 active+clean; 102 MiB data, 586 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 MiB/s rd, 2.6 MiB/s wr, 1.12k op/s 2024-05-27T07:28:36.439 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:28:36 smithi105 ceph-mon[35977]: pgmap v86: 129 pgs: 129 active+clean; 102 MiB data, 586 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 MiB/s rd, 2.6 MiB/s wr, 1.12k op/s 2024-05-27T07:28:37.229 INFO:journalctl@ceph.mds.g.smithi052.stdout:May 27 07:28:36 smithi052 podman[103206]: 2024-05-27 07:28:36.823358713 +0000 UTC m=+1.826100022 container remove ff5d245d9ddc6752e3804e7b59f26b44e1aedad2f0ad09bae6127b410f919ae6 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-g, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, org.label-schema.build-date=20240521, GIT_CLEAN=True, org.label-schema.license=GPLv2, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=) 2024-05-27T07:28:37.229 INFO:journalctl@ceph.mds.g.smithi052.stdout:May 27 07:28:36 smithi052 systemd[1]: ceph-73df2636-1bf8-11ef-bc9b-c7b262605968@mds.g.service: Main process exited, code=exited, status=134/n/a 2024-05-27T07:28:37.933 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:37.942+0000 7f085f3f0700 1 -- 172.21.15.52:0/3299096056 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f085810a950 msgr2=0x7f085810ad60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:28:37.933 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:37.942+0000 7f085f3f0700 1 --2- 172.21.15.52:0/3299096056 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f085810a950 0x7f085810ad60 secure :-1 s=READY pgs=584 cs=0 l=1 rev1=1 crypto rx=0x7f0848009990 tx=0x7f0848009ca0 comp rx=0 tx=0).stop 2024-05-27T07:28:37.934 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:37.943+0000 7f085f3f0700 1 -- 172.21.15.52:0/3299096056 shutdown_connections 2024-05-27T07:28:37.934 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:37.943+0000 7f085f3f0700 1 --2- 172.21.15.52:0/3299096056 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f085810b2a0 0x7f0858117820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:28:37.934 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:37.943+0000 7f085f3f0700 1 --2- 172.21.15.52:0/3299096056 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f0858110980 0x7f0858110d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:28:37.934 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:37.943+0000 7f085f3f0700 1 --2- 172.21.15.52:0/3299096056 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f085810a950 0x7f085810ad60 unknown :-1 s=CLOSED pgs=584 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:28:37.934 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:37.943+0000 7f085f3f0700 1 -- 172.21.15.52:0/3299096056 >> 172.21.15.52:0/3299096056 conn(0x7f0858076d20 msgr2=0x7f0858077130 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:28:37.936 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:37.946+0000 7f085f3f0700 1 -- 172.21.15.52:0/3299096056 shutdown_connections 2024-05-27T07:28:37.936 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:37.946+0000 7f085f3f0700 1 -- 172.21.15.52:0/3299096056 wait complete. 2024-05-27T07:28:37.939 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:37.947+0000 7f085f3f0700 1 Processor -- start 2024-05-27T07:28:37.939 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:37.947+0000 7f085f3f0700 1 -- start start 2024-05-27T07:28:37.939 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:37.947+0000 7f085f3f0700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f085810a950 0x7f08581a14d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:28:37.939 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:37.947+0000 7f085f3f0700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f085810b2a0 0x7f08581a1a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:28:37.939 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:37.948+0000 7f085f3f0700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f0858110980 0x7f08581a5d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:28:37.939 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:37.948+0000 7f085f3f0700 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x7f085811ab20 con 0x7f085810b2a0 2024-05-27T07:28:37.939 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:37.948+0000 7f085f3f0700 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f085811a9a0 con 0x7f0858110980 2024-05-27T07:28:37.939 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:37.948+0000 7f085f3f0700 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f085811aca0 con 0x7f085810a950 2024-05-27T07:28:37.939 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:37.948+0000 7f085c98b700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f085810b2a0 0x7f08581a1a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:28:37.939 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:37.948+0000 7f085c98b700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f085810b2a0 0x7f08581a1a70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.52:34042/0 (socket says 172.21.15.52:34042) 2024-05-27T07:28:37.939 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:37.948+0000 7f085c98b700 1 -- 172.21.15.52:0/2173604681 learned_addr learned my addr 172.21.15.52:0/2173604681 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-05-27T07:28:37.940 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:37.948+0000 7f085d98d700 1 --2- 172.21.15.52:0/2173604681 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f0858110980 0x7f08581a5d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:28:37.940 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:37.948+0000 7f085d18c700 1 --2- 172.21.15.52:0/2173604681 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f085810a950 0x7f08581a14d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:28:37.941 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:37.948+0000 7f085c98b700 1 -- 172.21.15.52:0/2173604681 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f085810a950 msgr2=0x7f08581a14d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:28:37.941 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:37.948+0000 7f085c98b700 1 --2- 172.21.15.52:0/2173604681 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f085810a950 0x7f08581a14d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:28:37.941 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:37.948+0000 7f085c98b700 1 -- 172.21.15.52:0/2173604681 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f0858110980 msgr2=0x7f08581a5d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:28:37.941 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:37.948+0000 7f085c98b700 1 --2- 172.21.15.52:0/2173604681 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f0858110980 0x7f08581a5d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:28:37.941 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:37.949+0000 7f085c98b700 1 -- 172.21.15.52:0/2173604681 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f08581a63a0 con 0x7f085810b2a0 2024-05-27T07:28:37.941 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:37.949+0000 7f085d18c700 1 --2- 172.21.15.52:0/2173604681 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f085810a950 0x7f08581a14d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-27T07:28:37.941 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:37.949+0000 7f085c98b700 1 --2- 172.21.15.52:0/2173604681 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f085810b2a0 0x7f08581a1a70 secure :-1 s=READY pgs=585 cs=0 l=1 rev1=1 crypto rx=0x7f08480047e0 tx=0x7f0848005da0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:28:37.941 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:37.950+0000 7f084e7fc700 1 -- 172.21.15.52:0/2173604681 <== mon.0 v2:172.21.15.52:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f084800fb40 con 0x7f085810b2a0 2024-05-27T07:28:37.942 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:37.950+0000 7f084e7fc700 1 -- 172.21.15.52:0/2173604681 <== mon.0 v2:172.21.15.52:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f084800fce0 con 0x7f085810b2a0 2024-05-27T07:28:37.942 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:37.950+0000 7f084e7fc700 1 -- 172.21.15.52:0/2173604681 <== mon.0 v2:172.21.15.52:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f08480056a0 con 0x7f085810b2a0 2024-05-27T07:28:37.942 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:37.950+0000 7f085f3f0700 1 -- 172.21.15.52:0/2173604681 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f08581a6630 con 0x7f085810b2a0 2024-05-27T07:28:37.944 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:37.950+0000 7f085f3f0700 1 -- 172.21.15.52:0/2173604681 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f08581a6a60 con 0x7f085810b2a0 2024-05-27T07:28:37.952 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:37.961+0000 7f084e7fc700 1 -- 172.21.15.52:0/2173604681 <== mon.0 v2:172.21.15.52:3300/0 4 ==== mgrmap(e 26) ==== 149783+0+0 (secure 0 0 0) 0x7f0848028020 con 0x7f085810b2a0 2024-05-27T07:28:37.953 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:37.962+0000 7f086de05b80 1 Processor -- start 2024-05-27T07:28:37.954 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:37.962+0000 7f086de05b80 1 -- start start 2024-05-27T07:28:37.954 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:37.963+0000 7f086de05b80 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x5641f2e6ae40 0x5641f2fee710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:28:37.954 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:37.963+0000 7f086de05b80 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5641f2feec50 0x5641f2fe8a40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:28:37.954 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:37.963+0000 7f086de05b80 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5641f2fe8f80 0x5641f2fe9390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:28:37.954 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:37.963+0000 7f086de05b80 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x5641f2f17c90 con 0x5641f2e6ae40 2024-05-27T07:28:37.954 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:37.963+0000 7f086de05b80 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x5641f2f07830 con 0x5641f2fe8f80 2024-05-27T07:28:37.954 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:37.963+0000 7f086de05b80 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x5641f2f0b870 con 0x5641f2feec50 2024-05-27T07:28:37.954 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:37.963+0000 7f085d98d700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x5641f2e6ae40 0x5641f2fee710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:28:37.954 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:37.963+0000 7f085d98d700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x5641f2e6ae40 0x5641f2fee710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.52:34054/0 (socket says 172.21.15.52:34054) 2024-05-27T07:28:37.954 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:37.963+0000 7f085d98d700 1 -- 172.21.15.52:0/2960000031 learned_addr learned my addr 172.21.15.52:0/2960000031 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-05-27T07:28:37.957 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:37.963+0000 7f085d98d700 1 -- 172.21.15.52:0/2960000031 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5641f2feec50 msgr2=0x5641f2fe8a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:28:37.957 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:37.963+0000 7f085d98d700 1 --2- 172.21.15.52:0/2960000031 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5641f2feec50 0x5641f2fe8a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:28:37.958 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:37.963+0000 7f085d98d700 1 -- 172.21.15.52:0/2960000031 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5641f2fe8f80 msgr2=0x5641f2fe9390 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-05-27T07:28:37.958 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:37.963+0000 7f085d98d700 1 --2- 172.21.15.52:0/2960000031 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5641f2fe8f80 0x5641f2fe9390 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:28:37.958 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:37.963+0000 7f085d98d700 1 -- 172.21.15.52:0/2960000031 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5641f2e578d0 con 0x5641f2e6ae40 2024-05-27T07:28:37.958 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:37.964+0000 7f084e7fc700 1 --2- 172.21.15.52:0/2173604681 >> [v2:172.21.15.52:6800/1822318384,v1:172.21.15.52:6801/1822318384] conn(0x7f08440b2bc0 0x7f08440b50d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:28:37.958 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:37.965+0000 7f084e7fc700 1 -- 172.21.15.52:0/2173604681 <== mon.0 v2:172.21.15.52:3300/0 5 ==== osd_map(95..95 src has 1..95) ==== 9379+0+0 (secure 0 0 0) 0x7f08480e3760 con 0x7f085810b2a0 2024-05-27T07:28:37.958 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:37.965+0000 7f085d98d700 1 --2- 172.21.15.52:0/2960000031 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x5641f2e6ae40 0x5641f2fee710 secure :-1 s=READY pgs=586 cs=0 l=1 rev1=1 crypto rx=0x7f085400b820 tx=0x7f085400bb30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=74d4638ff45937b5 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:28:37.958 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:37.966+0000 7f085d18c700 1 --2- 172.21.15.52:0/2173604681 >> [v2:172.21.15.52:6800/1822318384,v1:172.21.15.52:6801/1822318384] conn(0x7f08440b2bc0 0x7f08440b50d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:28:37.959 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:37.966+0000 7f085d18c700 1 --2- 172.21.15.52:0/2173604681 >> [v2:172.21.15.52:6800/1822318384,v1:172.21.15.52:6801/1822318384] conn(0x7f08440b2bc0 0x7f08440b50d0 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f0850009fd0 tx=0x7f0850009920 comp rx=0 tx=0).ready entity=mgr.15333 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:28:37.959 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:37.967+0000 7f08437fe700 1 -- 172.21.15.52:0/2960000031 <== mon.0 v2:172.21.15.52:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f08540125f0 con 0x5641f2e6ae40 2024-05-27T07:28:37.959 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:37.967+0000 7f08437fe700 1 -- 172.21.15.52:0/2960000031 <== mon.0 v2:172.21.15.52:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f0854019070 con 0x5641f2e6ae40 2024-05-27T07:28:37.959 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:37.967+0000 7f08437fe700 1 -- 172.21.15.52:0/2960000031 <== mon.0 v2:172.21.15.52:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0854014ad0 con 0x5641f2e6ae40 2024-05-27T07:28:37.960 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:37.968+0000 7f086de05b80 1 -- 172.21.15.52:0/2960000031 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x5641f2e6ae40 msgr2=0x5641f2fee710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:28:37.960 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:37.968+0000 7f086de05b80 1 --2- 172.21.15.52:0/2960000031 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x5641f2e6ae40 0x5641f2fee710 secure :-1 s=READY pgs=586 cs=0 l=1 rev1=1 crypto rx=0x7f085400b820 tx=0x7f085400bb30 comp rx=0 tx=0).stop 2024-05-27T07:28:37.960 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:37.969+0000 7f086de05b80 1 -- 172.21.15.52:0/2960000031 shutdown_connections 2024-05-27T07:28:37.960 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:37.969+0000 7f086de05b80 1 --2- 172.21.15.52:0/2960000031 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5641f2fe8f80 0x5641f2fe9390 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:28:37.961 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:37.969+0000 7f086de05b80 1 --2- 172.21.15.52:0/2960000031 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x5641f2e6ae40 0x5641f2fee710 unknown :-1 s=CLOSED pgs=586 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:28:37.961 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:37.969+0000 7f086de05b80 1 --2- 172.21.15.52:0/2960000031 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5641f2feec50 0x5641f2fe8a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:28:37.961 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:37.969+0000 7f086de05b80 1 -- 172.21.15.52:0/2960000031 >> 172.21.15.52:0/2960000031 conn(0x5641f2ec5af0 msgr2=0x5641f2fe3640 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:28:37.962 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:37.970+0000 7f086de05b80 1 -- 172.21.15.52:0/2960000031 shutdown_connections 2024-05-27T07:28:37.966 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:37.975+0000 7f086de05b80 1 -- 172.21.15.52:0/2960000031 wait complete. 2024-05-27T07:28:37.966 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:37.975+0000 7f086de05b80 1 -- start start 2024-05-27T07:28:37.966 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:37.975+0000 7f086de05b80 1 Processor -- start 2024-05-27T07:28:37.975 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:37.976+0000 7f086de05b80 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5641f2e6ae40 0x5641f2ff2340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:28:37.975 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:37.976+0000 7f086de05b80 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5641f2fe8f80 0x5641f2ff2880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:28:37.975 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:37.976+0000 7f086de05b80 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x5641f2feec50 0x5641f2ff2dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:28:37.975 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:37.976+0000 7f085d18c700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5641f2e6ae40 0x5641f2ff2340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:28:37.976 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:37.976+0000 7f085d18c700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5641f2e6ae40 0x5641f2ff2340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.173:3300/0 says I am v2:172.21.15.52:48776/0 (socket says 172.21.15.52:48776) 2024-05-27T07:28:37.976 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:37.977+0000 7f085d18c700 1 -- 172.21.15.52:0/3859165408 learned_addr learned my addr 172.21.15.52:0/3859165408 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-05-27T07:28:37.976 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:37.977+0000 7f086de05b80 1 -- 172.21.15.52:0/3859165408 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x5641f2e0dfa0 con 0x5641f2feec50 2024-05-27T07:28:37.976 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:37.977+0000 7f086de05b80 1 -- 172.21.15.52:0/3859165408 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x5641f2ea97c0 con 0x5641f2fe8f80 2024-05-27T07:28:37.976 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:37.977+0000 7f086de05b80 1 -- 172.21.15.52:0/3859165408 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x5641f3006750 con 0x5641f2e6ae40 2024-05-27T07:28:37.976 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:37.977+0000 7f086de05b80 10 client.0 _ll_register_callbacks cb 0x5641f2ea2920 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T07:28:37.977 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:37.977+0000 7f085d18c700 1 -- 172.21.15.52:0/3859165408 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5641f2fe8f80 msgr2=0x5641f2ff2880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:28:37.977 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:37.977+0000 7f085d18c700 1 --2- 172.21.15.52:0/3859165408 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5641f2fe8f80 0x5641f2ff2880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:28:37.977 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:37.977+0000 7f085d18c700 1 -- 172.21.15.52:0/3859165408 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x5641f2feec50 msgr2=0x5641f2ff2dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:28:37.977 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:37.977+0000 7f085d18c700 1 --2- 172.21.15.52:0/3859165408 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x5641f2feec50 0x5641f2ff2dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:28:37.977 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:37.977+0000 7f085d18c700 1 -- 172.21.15.52:0/3859165408 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5641f2ea6690 con 0x5641f2e6ae40 2024-05-27T07:28:37.977 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:37.977+0000 7f085d18c700 1 --2- 172.21.15.52:0/3859165408 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5641f2e6ae40 0x5641f2ff2340 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f0850004b10 tx=0x7f0850004b40 comp rx=0 tx=0).ready entity=mon.1 client_cookie=9eee5300e55fefdb server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:28:37.977 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:37.978+0000 7f0840ff9700 10 client.0 ms_handle_connect on v2:172.21.15.173:3300/0 2024-05-27T07:28:37.978 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:37.978+0000 7f0840ff9700 1 -- 172.21.15.52:0/3859165408 <== mon.1 v2:172.21.15.173:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0850037060 con 0x5641f2e6ae40 2024-05-27T07:28:37.978 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:37.978+0000 7f086de05b80 1 -- 172.21.15.52:0/3859165408 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f0854019070 con 0x5641f2e6ae40 2024-05-27T07:28:37.978 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:37.978+0000 7f0840ff9700 1 -- 172.21.15.52:0/3859165408 <== mon.1 v2:172.21.15.173:3300/0 2 ==== osd_map(95..95 src has 1..95) ==== 9379+0+0 (secure 0 0 0) 0x7f0850003c10 con 0x5641f2e6ae40 2024-05-27T07:28:37.978 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:37.979+0000 7f0840ff9700 1 -- 172.21.15.52:0/3859165408 <== mon.1 v2:172.21.15.173:3300/0 3 ==== mon_get_version_reply(handle=1 version=39) ==== 24+0+0 (secure 0 0 0) 0x7f085003e050 con 0x5641f2e6ae40 2024-05-27T07:28:37.978 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:37.979+0000 7f086de05b80 10 client.25690 fetch_fsmap learned FSMap version 39 2024-05-27T07:28:37.978 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:37.979+0000 7f086de05b80 1 -- 172.21.15.52:0/3859165408 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({config=0+,fsmap=39,monmap=4+}) -- 0x5641f3011130 con 0x5641f2e6ae40 2024-05-27T07:28:37.979 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:37.980+0000 7f0840ff9700 1 -- 172.21.15.52:0/3859165408 <== mon.1 v2:172.21.15.173:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f0854019070 con 0x5641f2e6ae40 2024-05-27T07:28:37.979 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:37.980+0000 7f0840ff9700 1 -- 172.21.15.52:0/3859165408 <== mon.1 v2:172.21.15.173:3300/0 5 ==== fsmap(e 39) ==== 8378+0+0 (secure 0 0 0) 0x7f08500146f0 con 0x5641f2e6ae40 2024-05-27T07:28:37.979 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:37.980+0000 7f086de05b80 10 client.25690 fetch_fsmap finished waiting for FSMap version 39 2024-05-27T07:28:37.979 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:37.980+0000 7f086de05b80 10 client.25690 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.h 2024-05-27T07:28:37.979 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:37.980+0000 7f086de05b80 20 client.25690 populate_metadata read hostname 'smithi052' 2024-05-27T07:28:37.979 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:37.980+0000 7f086de05b80 1 --2- 172.21.15.52:0/3859165408 >> [v2:172.21.15.105:6836/191393149,v1:172.21.15.105:6837/191393149] conn(0x5641f3013c40 0x5641f3016150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:28:37.979 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:37.980+0000 7f086de05b80 4 client.25690 mds_command: new command op to 24497 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-27T07:28:37.980 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:37.980+0000 7f086de05b80 1 -- 172.21.15.52:0/3859165408 --> [v2:172.21.15.105:6836/191393149,v1:172.21.15.105:6837/191393149] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5641f2fb49f0 con 0x5641f3013c40 2024-05-27T07:28:37.980 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:37.980+0000 7f085c98b700 1 --2- 172.21.15.52:0/3859165408 >> [v2:172.21.15.105:6836/191393149,v1:172.21.15.105:6837/191393149] conn(0x5641f3013c40 0x5641f3016150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:28:37.980 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:37.982+0000 7f085c98b700 1 --2- 172.21.15.52:0/3859165408 >> [v2:172.21.15.105:6836/191393149,v1:172.21.15.105:6837/191393149] conn(0x5641f3013c40 0x5641f3016150 crc :-1 s=READY pgs=35 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=5b5241d6fd79716b server_cookie=e4ab4031bb5d9f80 in_seq=0 out_seq=0 2024-05-27T07:28:37.980 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:37.983+0000 7f0840ff9700 10 client.25690 ms_handle_connect on v2:172.21.15.105:6836/191393149 2024-05-27T07:28:37.980 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:37.984+0000 7f0840ff9700 1 -- 172.21.15.52:0/3859165408 <== mds.1 v2:172.21.15.105:6836/191393149 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24041 (crc 0 0 0) 0x5641f2fb49f0 con 0x5641f3013c40 2024-05-27T07:28:37.980 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:37.984+0000 7f0840ff9700 10 client.25690 handle_command_reply: tid=0 2024-05-27T07:28:37.980 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:37.984+0000 7f086de05b80 1 client.25690 shutdown 2024-05-27T07:28:37.982 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:37.985+0000 7f086de05b80 1 -- 172.21.15.52:0/3859165408 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5641f2e6ae40 msgr2=0x5641f2ff2340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:28:37.982 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:37.985+0000 7f086de05b80 1 --2- 172.21.15.52:0/3859165408 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5641f2e6ae40 0x5641f2ff2340 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f0850004b10 tx=0x7f0850004b40 comp rx=0 tx=0).stop 2024-05-27T07:28:37.982 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:37.986+0000 7f086de05b80 1 -- 172.21.15.52:0/3859165408 shutdown_connections 2024-05-27T07:28:37.982 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:37.986+0000 7f086de05b80 1 --2- 172.21.15.52:0/3859165408 >> [v2:172.21.15.105:6836/191393149,v1:172.21.15.105:6837/191393149] conn(0x5641f3013c40 0x5641f3016150 crc :-1 s=READY pgs=35 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:28:37.983 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:37.986+0000 7f086de05b80 1 --2- 172.21.15.52:0/3859165408 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x5641f2feec50 0x5641f2ff2dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:28:37.983 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:37.986+0000 7f086de05b80 1 --2- 172.21.15.52:0/3859165408 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5641f2e6ae40 0x5641f2ff2340 unknown :-1 s=CLOSED pgs=97 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:28:37.983 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:37.986+0000 7f086de05b80 1 --2- 172.21.15.52:0/3859165408 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5641f2fe8f80 0x5641f2ff2880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:28:37.983 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:37.986+0000 7f086de05b80 1 -- 172.21.15.52:0/3859165408 >> 172.21.15.52:0/3859165408 conn(0x5641f2ec5af0 msgr2=0x5641f2fe7a30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:28:37.983 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:37.987+0000 7f0840ff9700 0 client.25690 ms_handle_reset on v2:172.21.15.105:6836/191393149 2024-05-27T07:28:37.983 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:37.988+0000 7f086de05b80 1 -- 172.21.15.52:0/3859165408 shutdown_connections 2024-05-27T07:28:37.984 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:37.989+0000 7f086de05b80 1 -- 172.21.15.52:0/3859165408 wait complete. 2024-05-27T07:28:37.984 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:37.989+0000 7f086de05b80 20 client.25690 trim_cache size 0 max 16384 2024-05-27T07:28:38.014 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:38.023+0000 7f086de05b80 1 Processor -- start 2024-05-27T07:28:38.014 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:38.023+0000 7f086de05b80 1 -- start start 2024-05-27T07:28:38.014 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:38.023+0000 7f086de05b80 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5641f2e6ae40 0x5641f2ff48b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:28:38.014 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:38.023+0000 7f086de05b80 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x5641f2fe8f80 0x5641f2ff2c00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:28:38.017 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:38.023+0000 7f086de05b80 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5641f2feec50 0x5641f300f880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:28:38.017 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:38.023+0000 7f086de05b80 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x7f083c030d60 con 0x5641f2fe8f80 2024-05-27T07:28:38.017 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:38.023+0000 7f086de05b80 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f083c032960 con 0x5641f2feec50 2024-05-27T07:28:38.018 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:38.023+0000 7f086de05b80 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f083c031430 con 0x5641f2e6ae40 2024-05-27T07:28:38.018 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:38.024+0000 7f085c98b700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x5641f2fe8f80 0x5641f2ff2c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:28:38.018 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:38.024+0000 7f085d18c700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5641f2e6ae40 0x5641f2ff48b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:28:38.018 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:38.024+0000 7f085d18c700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5641f2e6ae40 0x5641f2ff48b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.173:3300/0 says I am v2:172.21.15.52:48780/0 (socket says 172.21.15.52:48780) 2024-05-27T07:28:38.018 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:38.024+0000 7f085d18c700 1 -- 172.21.15.52:0/3158785849 learned_addr learned my addr 172.21.15.52:0/3158785849 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-05-27T07:28:38.018 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:38.024+0000 7f085d98d700 1 --2- 172.21.15.52:0/3158785849 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5641f2feec50 0x5641f300f880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:28:38.018 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:38.024+0000 7f085d18c700 1 -- 172.21.15.52:0/3158785849 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5641f2feec50 msgr2=0x5641f300f880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:28:38.019 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:38.024+0000 7f085d18c700 1 --2- 172.21.15.52:0/3158785849 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5641f2feec50 0x5641f300f880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:28:38.019 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:38.024+0000 7f085d18c700 1 -- 172.21.15.52:0/3158785849 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x5641f2fe8f80 msgr2=0x5641f2ff2c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:28:38.019 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:38.024+0000 7f085d18c700 1 --2- 172.21.15.52:0/3158785849 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x5641f2fe8f80 0x5641f2ff2c00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:28:38.019 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:38.024+0000 7f085d18c700 1 -- 172.21.15.52:0/3158785849 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5641f2ff51c0 con 0x5641f2e6ae40 2024-05-27T07:28:38.019 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:38.025+0000 7f085c98b700 1 --2- 172.21.15.52:0/3158785849 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x5641f2fe8f80 0x5641f2ff2c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-27T07:28:38.019 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:38.025+0000 7f085d18c700 1 --2- 172.21.15.52:0/3158785849 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5641f2e6ae40 0x5641f2ff48b0 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7f0850014c60 tx=0x7f0850014d70 comp rx=0 tx=0).ready entity=mon.1 client_cookie=34c89871e801b0f server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:28:38.020 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:38.025+0000 7f0842ffd700 1 -- 172.21.15.52:0/3158785849 <== mon.1 v2:172.21.15.173:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f085001f070 con 0x5641f2e6ae40 2024-05-27T07:28:38.020 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:38.026+0000 7f0842ffd700 1 -- 172.21.15.52:0/3158785849 <== mon.1 v2:172.21.15.173:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f0850014880 con 0x5641f2e6ae40 2024-05-27T07:28:38.020 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:38.026+0000 7f0842ffd700 1 -- 172.21.15.52:0/3158785849 <== mon.1 v2:172.21.15.173:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0850015d40 con 0x5641f2e6ae40 2024-05-27T07:28:38.024 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:38.027+0000 7f086de05b80 1 -- 172.21.15.52:0/3158785849 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5641f2e6ae40 msgr2=0x5641f2ff48b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:28:38.024 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:38.027+0000 7f086de05b80 1 --2- 172.21.15.52:0/3158785849 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5641f2e6ae40 0x5641f2ff48b0 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7f0850014c60 tx=0x7f0850014d70 comp rx=0 tx=0).stop 2024-05-27T07:28:38.024 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:38.027+0000 7f086de05b80 1 -- 172.21.15.52:0/3158785849 shutdown_connections 2024-05-27T07:28:38.024 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:38.027+0000 7f086de05b80 1 --2- 172.21.15.52:0/3158785849 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5641f2feec50 0x5641f300f880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:28:38.025 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:38.027+0000 7f086de05b80 1 --2- 172.21.15.52:0/3158785849 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5641f2e6ae40 0x5641f2ff48b0 unknown :-1 s=CLOSED pgs=98 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:28:38.025 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:38.027+0000 7f086de05b80 1 --2- 172.21.15.52:0/3158785849 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x5641f2fe8f80 0x5641f2ff2c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:28:38.025 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:38.027+0000 7f086de05b80 1 -- 172.21.15.52:0/3158785849 >> 172.21.15.52:0/3158785849 conn(0x5641f2ec5af0 msgr2=0x5641f3015c50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:28:38.025 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:38.028+0000 7f086de05b80 1 -- 172.21.15.52:0/3158785849 shutdown_connections 2024-05-27T07:28:38.025 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:38.029+0000 7f086de05b80 1 -- 172.21.15.52:0/3158785849 wait complete. 2024-05-27T07:28:38.025 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:38.029+0000 7f086de05b80 1 -- start start 2024-05-27T07:28:38.025 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:38.030+0000 7f086de05b80 1 Processor -- start 2024-05-27T07:28:38.026 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:38.031+0000 7f086de05b80 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5641f2e6ae40 0x5641f3006a30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:28:38.026 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:38.031+0000 7f086de05b80 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x5641f2fe8f80 0x5641f3006f70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:28:38.026 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:38.031+0000 7f085c98b700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x5641f2fe8f80 0x5641f3006f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:28:38.026 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:38.031+0000 7f085c98b700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x5641f2fe8f80 0x5641f3006f70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.52:34092/0 (socket says 172.21.15.52:34092) 2024-05-27T07:28:38.026 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:38.031+0000 7f086de05b80 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5641f2feec50 0x5641f30074b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:28:38.026 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:38.031+0000 7f085c98b700 1 -- 172.21.15.52:0/4057711846 learned_addr learned my addr 172.21.15.52:0/4057711846 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-05-27T07:28:38.026 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:38.032+0000 7f086de05b80 1 -- 172.21.15.52:0/4057711846 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x7f083c032290 con 0x5641f2fe8f80 2024-05-27T07:28:38.027 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:38.032+0000 7f086de05b80 1 -- 172.21.15.52:0/4057711846 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f083c031b00 con 0x5641f2e6ae40 2024-05-27T07:28:38.027 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:38.032+0000 7f085d18c700 1 --2- 172.21.15.52:0/4057711846 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5641f2e6ae40 0x5641f3006a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:28:38.027 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:38.032+0000 7f086de05b80 1 -- 172.21.15.52:0/4057711846 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f083c02ed00 con 0x5641f2feec50 2024-05-27T07:28:38.027 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:38.032+0000 7f086de05b80 10 client.0 _ll_register_callbacks cb 0x7f0850007d50 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T07:28:38.027 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:38.033+0000 7f085d18c700 1 -- 172.21.15.52:0/4057711846 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5641f2feec50 msgr2=0x5641f30074b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:28:38.027 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:38.033+0000 7f085d18c700 1 --2- 172.21.15.52:0/4057711846 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5641f2feec50 0x5641f30074b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:28:38.027 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:38.033+0000 7f085d18c700 1 -- 172.21.15.52:0/4057711846 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x5641f2fe8f80 msgr2=0x5641f3006f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:28:38.028 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:38.033+0000 7f085d18c700 1 --2- 172.21.15.52:0/4057711846 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x5641f2fe8f80 0x5641f3006f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:28:38.028 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:38.033+0000 7f085d18c700 1 -- 172.21.15.52:0/4057711846 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5641f307c810 con 0x5641f2e6ae40 2024-05-27T07:28:38.029 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:38.035+0000 7f085d18c700 1 --2- 172.21.15.52:0/4057711846 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5641f2e6ae40 0x5641f3006a30 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7f0850015420 tx=0x7f0850014680 comp rx=0 tx=0).ready entity=mon.2 client_cookie=84390b83e963f1ae server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:28:38.030 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:38.035+0000 7f0840ff9700 10 client.0 ms_handle_connect on v2:172.21.15.105:3300/0 2024-05-27T07:28:38.030 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:38.037+0000 7f0840ff9700 1 -- 172.21.15.52:0/4057711846 <== mon.2 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f085001f070 con 0x5641f2e6ae40 2024-05-27T07:28:38.030 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:38.037+0000 7f0840ff9700 1 -- 172.21.15.52:0/4057711846 <== mon.2 v2:172.21.15.105:3300/0 2 ==== osd_map(95..95 src has 1..95) ==== 9379+0+0 (secure 0 0 0) 0x7f08500053e0 con 0x5641f2e6ae40 2024-05-27T07:28:38.030 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:38.037+0000 7f086de05b80 1 -- 172.21.15.52:0/4057711846 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f0850014880 con 0x5641f2e6ae40 2024-05-27T07:28:38.030 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:38.038+0000 7f0840ff9700 1 -- 172.21.15.52:0/4057711846 <== mon.2 v2:172.21.15.105:3300/0 3 ==== mon_get_version_reply(handle=1 version=39) ==== 24+0+0 (secure 0 0 0) 0x7f08500055e0 con 0x5641f2e6ae40 2024-05-27T07:28:38.030 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:38.039+0000 7f086de05b80 10 client.25769 fetch_fsmap learned FSMap version 39 2024-05-27T07:28:38.031 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:38.039+0000 7f086de05b80 1 -- 172.21.15.52:0/4057711846 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,fsmap=39,monmap=4+}) -- 0x5641f307d990 con 0x5641f2e6ae40 2024-05-27T07:28:38.032 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:38.040+0000 7f0840ff9700 1 -- 172.21.15.52:0/4057711846 <== mon.2 v2:172.21.15.105:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f0850014880 con 0x5641f2e6ae40 2024-05-27T07:28:38.032 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:38.041+0000 7f0840ff9700 1 -- 172.21.15.52:0/4057711846 <== mon.2 v2:172.21.15.105:3300/0 5 ==== fsmap(e 39) ==== 8378+0+0 (secure 0 0 0) 0x7f0850015cb0 con 0x5641f2e6ae40 2024-05-27T07:28:38.032 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:38.041+0000 7f086de05b80 10 client.25769 fetch_fsmap finished waiting for FSMap version 39 2024-05-27T07:28:38.032 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:38.041+0000 7f086de05b80 10 client.25769 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.h 2024-05-27T07:28:38.032 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:38.041+0000 7f086de05b80 20 client.25769 populate_metadata read hostname 'smithi052' 2024-05-27T07:28:38.033 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:38.041+0000 7f086de05b80 1 --2- 172.21.15.52:0/4057711846 >> [v2:172.21.15.105:6836/191393149,v1:172.21.15.105:6837/191393149] conn(0x5641f307df80 0x5641f30803a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:28:38.033 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:38.041+0000 7f086de05b80 4 client.25769 mds_command: new command op to 24497 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-05-27T07:28:38.033 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:38.041+0000 7f086de05b80 1 -- 172.21.15.52:0/4057711846 --> [v2:172.21.15.105:6836/191393149,v1:172.21.15.105:6837/191393149] -- command(tid 0: {"prefix": "damage ls"}) -- 0x5641f2f19be0 con 0x5641f307df80 2024-05-27T07:28:38.033 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:38.041+0000 7f085c98b700 1 --2- 172.21.15.52:0/4057711846 >> [v2:172.21.15.105:6836/191393149,v1:172.21.15.105:6837/191393149] conn(0x5641f307df80 0x5641f30803a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:28:38.035 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:38.044+0000 7f085c98b700 1 --2- 172.21.15.52:0/4057711846 >> [v2:172.21.15.105:6836/191393149,v1:172.21.15.105:6837/191393149] conn(0x5641f307df80 0x5641f30803a0 crc :-1 s=READY pgs=36 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=25a1d6e931c20440 server_cookie=5867f466401e7a69 in_seq=0 out_seq=0 2024-05-27T07:28:38.035 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:38.044+0000 7f0840ff9700 10 client.25769 ms_handle_connect on v2:172.21.15.105:6836/191393149 2024-05-27T07:28:38.035 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:38.044+0000 7f0840ff9700 1 -- 172.21.15.52:0/4057711846 <== mds.1 v2:172.21.15.105:6836/191393149 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x5641f2f19be0 con 0x5641f307df80 2024-05-27T07:28:38.035 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:38.044+0000 7f0840ff9700 10 client.25769 handle_command_reply: tid=0 2024-05-27T07:28:38.035 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:38.044+0000 7f086de05b80 1 client.25769 shutdown 2024-05-27T07:28:38.037 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:38.046+0000 7f086de05b80 1 -- 172.21.15.52:0/4057711846 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5641f2e6ae40 msgr2=0x5641f3006a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:28:38.037 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:38.046+0000 7f086de05b80 1 --2- 172.21.15.52:0/4057711846 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5641f2e6ae40 0x5641f3006a30 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7f0850015420 tx=0x7f0850014680 comp rx=0 tx=0).stop 2024-05-27T07:28:38.038 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:38.047+0000 7f086de05b80 1 -- 172.21.15.52:0/4057711846 shutdown_connections 2024-05-27T07:28:38.038 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:38.047+0000 7f086de05b80 1 --2- 172.21.15.52:0/4057711846 >> [v2:172.21.15.105:6836/191393149,v1:172.21.15.105:6837/191393149] conn(0x5641f307df80 0x5641f30803a0 crc :-1 s=READY pgs=36 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:28:38.038 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:38.047+0000 7f086de05b80 1 --2- 172.21.15.52:0/4057711846 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5641f2feec50 0x5641f30074b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:28:38.039 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:38.047+0000 7f086de05b80 1 --2- 172.21.15.52:0/4057711846 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5641f2e6ae40 0x5641f3006a30 unknown :-1 s=CLOSED pgs=100 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:28:38.039 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:38.047+0000 7f086de05b80 1 --2- 172.21.15.52:0/4057711846 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x5641f2fe8f80 0x5641f3006f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:28:38.039 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:38.047+0000 7f086de05b80 1 -- 172.21.15.52:0/4057711846 >> 172.21.15.52:0/4057711846 conn(0x5641f2ec5af0 msgr2=0x5641f2fec010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:28:38.039 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:38.047+0000 7f0840ff9700 0 client.25769 ms_handle_reset on v2:172.21.15.105:6836/191393149 2024-05-27T07:28:38.040 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:38.048+0000 7f086de05b80 1 -- 172.21.15.52:0/4057711846 shutdown_connections 2024-05-27T07:28:38.040 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:38.049+0000 7f086de05b80 1 -- 172.21.15.52:0/4057711846 wait complete. 2024-05-27T07:28:38.041 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:38.049+0000 7f086de05b80 20 client.25769 trim_cache size 0 max 16384 2024-05-27T07:28:38.041 INFO:teuthology.orchestra.run.smithi052.stdout:[] 2024-05-27T07:28:38.043 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:38.051+0000 7f085f3f0700 1 -- 172.21.15.52:0/2173604681 >> [v2:172.21.15.52:6800/1822318384,v1:172.21.15.52:6801/1822318384] conn(0x7f08440b2bc0 msgr2=0x7f08440b50d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:28:38.043 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:38.051+0000 7f085f3f0700 1 --2- 172.21.15.52:0/2173604681 >> [v2:172.21.15.52:6800/1822318384,v1:172.21.15.52:6801/1822318384] conn(0x7f08440b2bc0 0x7f08440b50d0 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f0850009fd0 tx=0x7f0850009920 comp rx=0 tx=0).stop 2024-05-27T07:28:38.043 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:38.051+0000 7f085f3f0700 1 -- 172.21.15.52:0/2173604681 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f085810b2a0 msgr2=0x7f08581a1a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:28:38.043 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:38.051+0000 7f085f3f0700 1 --2- 172.21.15.52:0/2173604681 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f085810b2a0 0x7f08581a1a70 secure :-1 s=READY pgs=585 cs=0 l=1 rev1=1 crypto rx=0x7f08480047e0 tx=0x7f0848005da0 comp rx=0 tx=0).stop 2024-05-27T07:28:38.043 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:38.051+0000 7f085f3f0700 1 -- 172.21.15.52:0/2173604681 shutdown_connections 2024-05-27T07:28:38.043 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:38.051+0000 7f085f3f0700 1 --2- 172.21.15.52:0/2173604681 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f0858110980 0x7f08581a5d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:28:38.044 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:38.051+0000 7f085f3f0700 1 --2- 172.21.15.52:0/2173604681 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f085810a950 0x7f08581a14d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:28:38.044 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:38.051+0000 7f085f3f0700 1 --2- 172.21.15.52:0/2173604681 >> [v2:172.21.15.52:6800/1822318384,v1:172.21.15.52:6801/1822318384] conn(0x7f08440b2bc0 0x7f08440b50d0 unknown :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:28:38.044 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:38.051+0000 7f085f3f0700 1 --2- 172.21.15.52:0/2173604681 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f085810b2a0 0x7f08581a1a70 unknown :-1 s=CLOSED pgs=585 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:28:38.044 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:38.051+0000 7f085f3f0700 1 -- 172.21.15.52:0/2173604681 >> 172.21.15.52:0/2173604681 conn(0x7f0858076d20 msgr2=0x7f0858108910 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:28:38.045 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:38.054+0000 7f085f3f0700 1 -- 172.21.15.52:0/2173604681 shutdown_connections 2024-05-27T07:28:38.045 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:38.054+0000 7f085f3f0700 1 -- 172.21.15.52:0/2173604681 wait complete. 2024-05-27T07:28:38.091 INFO:journalctl@ceph.mds.g.smithi052.stdout:May 27 07:28:37 smithi052 systemd[1]: ceph-73df2636-1bf8-11ef-bc9b-c7b262605968@mds.g.service: Failed with result 'exit-code'. 2024-05-27T07:28:38.351 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:28:38 smithi173 ceph-mon[37000]: pgmap v87: 129 pgs: 129 active+clean; 102 MiB data, 586 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 MiB/s rd, 1.3 MiB/s wr, 918 op/s 2024-05-27T07:28:38.439 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:28:38 smithi105 ceph-mon[35977]: pgmap v87: 129 pgs: 129 active+clean; 102 MiB data, 586 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 MiB/s rd, 1.3 MiB/s wr, 918 op/s 2024-05-27T07:28:38.461 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:28:38 smithi052 ceph-mon[31955]: pgmap v87: 129 pgs: 129 active+clean; 102 MiB data, 586 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 MiB/s rd, 1.3 MiB/s wr, 918 op/s 2024-05-27T07:28:38.877 DEBUG:teuthology.orchestra.run.smithi052:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:38f9d900c83c63799fbdbe61acc9a11b0d3554a6 shell --fsid 73df2636-1bf8-11ef-bc9b-c7b262605968 -- ceph tell mds.1:0 damage ls 2024-05-27T07:28:39.313 INFO:teuthology.orchestra.run.smithi052.stderr:Inferring config /var/lib/ceph/73df2636-1bf8-11ef-bc9b-c7b262605968/mon.a/config 2024-05-27T07:28:40.351 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:28:40 smithi173 ceph-mon[37000]: pgmap v88: 129 pgs: 129 active+clean; 100 MiB data, 621 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 MiB/s rd, 3.8 MiB/s wr, 1.13k op/s 2024-05-27T07:28:40.439 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:28:40 smithi105 ceph-mon[35977]: pgmap v88: 129 pgs: 129 active+clean; 100 MiB data, 621 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 MiB/s rd, 3.8 MiB/s wr, 1.13k op/s 2024-05-27T07:28:40.474 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:28:40 smithi052 ceph-mon[31955]: pgmap v88: 129 pgs: 129 active+clean; 100 MiB data, 621 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 MiB/s rd, 3.8 MiB/s wr, 1.13k op/s 2024-05-27T07:28:41.215 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:41.225+0000 7f4d63033700 1 -- 172.21.15.52:0/1199581640 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f4d5c108730 msgr2=0x7f4d5c110090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:28:41.215 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:41.225+0000 7f4d63033700 1 --2- 172.21.15.52:0/1199581640 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f4d5c108730 0x7f4d5c110090 secure :-1 s=READY pgs=587 cs=0 l=1 rev1=1 crypto rx=0x7f4d5400b0d0 tx=0x7f4d5400b3e0 comp rx=0 tx=0).stop 2024-05-27T07:28:41.216 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:41.225+0000 7f4d63033700 1 -- 172.21.15.52:0/1199581640 shutdown_connections 2024-05-27T07:28:41.216 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:41.225+0000 7f4d63033700 1 --2- 172.21.15.52:0/1199581640 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f4d5c108730 0x7f4d5c110090 unknown :-1 s=CLOSED pgs=587 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:28:41.216 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:41.225+0000 7f4d63033700 1 --2- 172.21.15.52:0/1199581640 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f4d5c1034b0 0x7f4d5c1038c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:28:41.216 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:41.225+0000 7f4d63033700 1 --2- 172.21.15.52:0/1199581640 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f4d5c107de0 0x7f4d5c1081f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:28:41.216 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:41.225+0000 7f4d63033700 1 -- 172.21.15.52:0/1199581640 >> 172.21.15.52:0/1199581640 conn(0x7f4d5c0fede0 msgr2=0x7f4d5c101200 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:28:41.216 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:41.225+0000 7f4d63033700 1 -- 172.21.15.52:0/1199581640 shutdown_connections 2024-05-27T07:28:41.216 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:41.225+0000 7f4d63033700 1 -- 172.21.15.52:0/1199581640 wait complete. 2024-05-27T07:28:41.218 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:41.226+0000 7f4d63033700 1 Processor -- start 2024-05-27T07:28:41.218 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:41.226+0000 7f4d63033700 1 -- start start 2024-05-27T07:28:41.218 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:41.227+0000 7f4d63033700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f4d5c1034b0 0x7f4d5c19f880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:28:41.218 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:41.227+0000 7f4d63033700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f4d5c107de0 0x7f4d5c19fdc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:28:41.218 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:41.227+0000 7f4d63033700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f4d5c108730 0x7f4d5c199aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:28:41.218 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:41.227+0000 7f4d63033700 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x7f4d5c113360 con 0x7f4d5c107de0 2024-05-27T07:28:41.218 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:41.227+0000 7f4d63033700 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f4d5c1131e0 con 0x7f4d5c1034b0 2024-05-27T07:28:41.218 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:41.227+0000 7f4d63033700 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f4d5c1134e0 con 0x7f4d5c108730 2024-05-27T07:28:41.218 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:41.227+0000 7f4d5bfff700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f4d5c107de0 0x7f4d5c19fdc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:28:41.218 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:41.227+0000 7f4d60dcf700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f4d5c1034b0 0x7f4d5c19f880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:28:41.219 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:41.227+0000 7f4d60dcf700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f4d5c1034b0 0x7f4d5c19f880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.52:48796/0 (socket says 172.21.15.52:48796) 2024-05-27T07:28:41.219 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:41.227+0000 7f4d60dcf700 1 -- 172.21.15.52:0/648989117 learned_addr learned my addr 172.21.15.52:0/648989117 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-05-27T07:28:41.219 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:41.227+0000 7f4d615d0700 1 --2- 172.21.15.52:0/648989117 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f4d5c108730 0x7f4d5c199aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:28:41.219 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:41.227+0000 7f4d60dcf700 1 -- 172.21.15.52:0/648989117 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f4d5c108730 msgr2=0x7f4d5c199aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:28:41.219 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:41.227+0000 7f4d60dcf700 1 --2- 172.21.15.52:0/648989117 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f4d5c108730 0x7f4d5c199aa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:28:41.219 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:41.227+0000 7f4d60dcf700 1 -- 172.21.15.52:0/648989117 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f4d5c107de0 msgr2=0x7f4d5c19fdc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:28:41.219 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:41.227+0000 7f4d60dcf700 1 --2- 172.21.15.52:0/648989117 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f4d5c107de0 0x7f4d5c19fdc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:28:41.219 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:41.227+0000 7f4d60dcf700 1 -- 172.21.15.52:0/648989117 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4d5c19a150 con 0x7f4d5c1034b0 2024-05-27T07:28:41.219 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:41.227+0000 7f4d5bfff700 1 --2- 172.21.15.52:0/648989117 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f4d5c107de0 0x7f4d5c19fdc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-27T07:28:41.219 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:41.228+0000 7f4d60dcf700 1 --2- 172.21.15.52:0/648989117 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f4d5c1034b0 0x7f4d5c19f880 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7f4d5000eab0 tx=0x7f4d5000edc0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:28:41.219 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:41.228+0000 7f4d59ffb700 1 -- 172.21.15.52:0/648989117 <== mon.2 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4d5001c070 con 0x7f4d5c1034b0 2024-05-27T07:28:41.219 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:41.228+0000 7f4d59ffb700 1 -- 172.21.15.52:0/648989117 <== mon.2 v2:172.21.15.105:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f4d50017410 con 0x7f4d5c1034b0 2024-05-27T07:28:41.219 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:41.228+0000 7f4d63033700 1 -- 172.21.15.52:0/648989117 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4d5c19a440 con 0x7f4d5c1034b0 2024-05-27T07:28:41.220 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:41.228+0000 7f4d59ffb700 1 -- 172.21.15.52:0/648989117 <== mon.2 v2:172.21.15.105:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4d50016610 con 0x7f4d5c1034b0 2024-05-27T07:28:41.220 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:41.228+0000 7f4d63033700 1 -- 172.21.15.52:0/648989117 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4d5c0759e0 con 0x7f4d5c1034b0 2024-05-27T07:28:41.220 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:41.231+0000 7f4d59ffb700 1 -- 172.21.15.52:0/648989117 <== mon.2 v2:172.21.15.105:3300/0 4 ==== mgrmap(e 26) ==== 149783+0+0 (secure 0 0 0) 0x7f4d5001f070 con 0x7f4d5c1034b0 2024-05-27T07:28:41.221 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:41.232+0000 7f4d59ffb700 1 --2- 172.21.15.52:0/648989117 >> [v2:172.21.15.52:6800/1822318384,v1:172.21.15.52:6801/1822318384] conn(0x7f4d4c0b29f0 0x7f4d4c0b4f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:28:41.223 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:41.232+0000 7f4d5bfff700 1 --2- 172.21.15.52:0/648989117 >> [v2:172.21.15.52:6800/1822318384,v1:172.21.15.52:6801/1822318384] conn(0x7f4d4c0b29f0 0x7f4d4c0b4f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:28:41.223 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:41.232+0000 7f4d71a48b80 1 Processor -- start 2024-05-27T07:28:41.223 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:41.232+0000 7f4d59ffb700 1 -- 172.21.15.52:0/648989117 <== mon.2 v2:172.21.15.105:3300/0 5 ==== osd_map(95..95 src has 1..95) ==== 9379+0+0 (secure 0 0 0) 0x7f4d500e3de0 con 0x7f4d5c1034b0 2024-05-27T07:28:41.223 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:41.232+0000 7f4d71a48b80 1 -- start start 2024-05-27T07:28:41.223 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:41.232+0000 7f4d5bfff700 1 --2- 172.21.15.52:0/648989117 >> [v2:172.21.15.52:6800/1822318384,v1:172.21.15.52:6801/1822318384] conn(0x7f4d4c0b29f0 0x7f4d4c0b4f00 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f4d48009680 tx=0x7f4d48009260 comp rx=0 tx=0).ready entity=mgr.15333 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:28:41.224 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:41.232+0000 7f4d71a48b80 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x563923d23e40 0x563923ea8b30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:28:41.224 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:41.232+0000 7f4d71a48b80 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x563923ea9070 0x563923ea2be0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:28:41.224 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:41.233+0000 7f4d71a48b80 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x563923ea3120 0x563923ea3530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:28:41.224 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:41.233+0000 7f4d71a48b80 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x563923dd0d70 con 0x563923d23e40 2024-05-27T07:28:41.224 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:41.233+0000 7f4d71a48b80 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x563923dc0910 con 0x563923ea3120 2024-05-27T07:28:41.224 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:41.233+0000 7f4d71a48b80 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x563923dc4950 con 0x563923ea9070 2024-05-27T07:28:41.224 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:41.233+0000 7f4d60dcf700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x563923d23e40 0x563923ea8b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:28:41.224 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:41.233+0000 7f4d60dcf700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x563923d23e40 0x563923ea8b30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.52:46014/0 (socket says 172.21.15.52:46014) 2024-05-27T07:28:41.224 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:41.233+0000 7f4d60dcf700 1 -- 172.21.15.52:0/2683879288 learned_addr learned my addr 172.21.15.52:0/2683879288 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-05-27T07:28:41.224 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:41.233+0000 7f4d615d0700 1 --2- 172.21.15.52:0/2683879288 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x563923ea3120 0x563923ea3530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:28:41.224 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:41.233+0000 7f4d5bfff700 1 --2- 172.21.15.52:0/2683879288 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x563923ea9070 0x563923ea2be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:28:41.224 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:41.233+0000 7f4d60dcf700 1 -- 172.21.15.52:0/2683879288 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x563923ea9070 msgr2=0x563923ea2be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:28:41.224 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:41.233+0000 7f4d60dcf700 1 --2- 172.21.15.52:0/2683879288 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x563923ea9070 0x563923ea2be0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:28:41.225 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:41.233+0000 7f4d60dcf700 1 -- 172.21.15.52:0/2683879288 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x563923ea3120 msgr2=0x563923ea3530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:28:41.225 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:41.233+0000 7f4d60dcf700 1 --2- 172.21.15.52:0/2683879288 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x563923ea3120 0x563923ea3530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:28:41.225 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:41.233+0000 7f4d60dcf700 1 -- 172.21.15.52:0/2683879288 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x563923d69fb0 con 0x563923d23e40 2024-05-27T07:28:41.225 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:41.233+0000 7f4d5bfff700 1 --2- 172.21.15.52:0/2683879288 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x563923ea9070 0x563923ea2be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-27T07:28:41.225 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:41.233+0000 7f4d60dcf700 1 --2- 172.21.15.52:0/2683879288 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x563923d23e40 0x563923ea8b30 secure :-1 s=READY pgs=588 cs=0 l=1 rev1=1 crypto rx=0x7f4d50019000 tx=0x7f4d500aa840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=abbb4cc07c6d6e7d server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:28:41.225 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:41.233+0000 7f4d46ffd700 1 -- 172.21.15.52:0/2683879288 <== mon.0 v2:172.21.15.52:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4d500c5390 con 0x563923d23e40 2024-05-27T07:28:41.225 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:41.233+0000 7f4d46ffd700 1 -- 172.21.15.52:0/2683879288 <== mon.0 v2:172.21.15.52:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f4d500a7070 con 0x563923d23e40 2024-05-27T07:28:41.225 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:41.233+0000 7f4d46ffd700 1 -- 172.21.15.52:0/2683879288 <== mon.0 v2:172.21.15.52:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4d500a6a40 con 0x563923d23e40 2024-05-27T07:28:41.228 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:41.234+0000 7f4d71a48b80 1 -- 172.21.15.52:0/2683879288 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x563923d23e40 msgr2=0x563923ea8b30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:28:41.228 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:41.234+0000 7f4d71a48b80 1 --2- 172.21.15.52:0/2683879288 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x563923d23e40 0x563923ea8b30 secure :-1 s=READY pgs=588 cs=0 l=1 rev1=1 crypto rx=0x7f4d50019000 tx=0x7f4d500aa840 comp rx=0 tx=0).stop 2024-05-27T07:28:41.228 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:41.234+0000 7f4d71a48b80 1 -- 172.21.15.52:0/2683879288 shutdown_connections 2024-05-27T07:28:41.228 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:41.234+0000 7f4d71a48b80 1 --2- 172.21.15.52:0/2683879288 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x563923ea3120 0x563923ea3530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:28:41.228 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:41.234+0000 7f4d71a48b80 1 --2- 172.21.15.52:0/2683879288 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x563923d23e40 0x563923ea8b30 unknown :-1 s=CLOSED pgs=588 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:28:41.228 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:41.234+0000 7f4d71a48b80 1 --2- 172.21.15.52:0/2683879288 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x563923ea9070 0x563923ea2be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:28:41.228 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:41.234+0000 7f4d71a48b80 1 -- 172.21.15.52:0/2683879288 >> 172.21.15.52:0/2683879288 conn(0x563923d26770 msgr2=0x563923e9dd90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:28:41.228 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:41.234+0000 7f4d71a48b80 1 -- 172.21.15.52:0/2683879288 shutdown_connections 2024-05-27T07:28:41.228 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:41.235+0000 7f4d71a48b80 1 -- 172.21.15.52:0/2683879288 wait complete. 2024-05-27T07:28:41.229 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:41.235+0000 7f4d71a48b80 1 -- start start 2024-05-27T07:28:41.229 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:41.235+0000 7f4d71a48b80 1 Processor -- start 2024-05-27T07:28:41.229 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:41.235+0000 7f4d71a48b80 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x563923d23e40 0x563923eac950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:28:41.229 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:41.235+0000 7f4d71a48b80 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x563923ea3120 0x563923eace90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:28:41.229 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:41.235+0000 7f4d71a48b80 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x563923ea9070 0x563923ead3d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:28:41.229 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:41.235+0000 7f4d71a48b80 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x563923cc6fa0 con 0x563923ea9070 2024-05-27T07:28:41.229 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:41.235+0000 7f4d71a48b80 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x563923d28da0 con 0x563923ea3120 2024-05-27T07:28:41.229 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:41.235+0000 7f4d71a48b80 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x563923eafb60 con 0x563923d23e40 2024-05-27T07:28:41.229 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:41.236+0000 7f4d71a48b80 10 client.0 _ll_register_callbacks cb 0x563923d575b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T07:28:41.229 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:41.236+0000 7f4d615d0700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x563923ea9070 0x563923ead3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:28:41.229 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:41.236+0000 7f4d5bfff700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x563923ea3120 0x563923eace90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:28:41.229 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:41.236+0000 7f4d615d0700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x563923ea9070 0x563923ead3d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.52:46020/0 (socket says 172.21.15.52:46020) 2024-05-27T07:28:41.229 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:41.236+0000 7f4d60dcf700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x563923d23e40 0x563923eac950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:28:41.230 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:41.236+0000 7f4d615d0700 1 -- 172.21.15.52:0/1782750380 learned_addr learned my addr 172.21.15.52:0/1782750380 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-05-27T07:28:41.230 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:41.236+0000 7f4d5bfff700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x563923ea3120 0x563923eace90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.52:48818/0 (socket says 172.21.15.52:48818) 2024-05-27T07:28:41.230 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:41.236+0000 7f4d615d0700 1 -- 172.21.15.52:0/1782750380 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x563923d23e40 msgr2=0x563923eac950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:28:41.230 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:41.236+0000 7f4d615d0700 1 --2- 172.21.15.52:0/1782750380 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x563923d23e40 0x563923eac950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:28:41.230 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:41.236+0000 7f4d615d0700 1 -- 172.21.15.52:0/1782750380 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x563923ea3120 msgr2=0x563923eace90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:28:41.230 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:41.236+0000 7f4d615d0700 1 --2- 172.21.15.52:0/1782750380 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x563923ea3120 0x563923eace90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:28:41.230 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:41.236+0000 7f4d615d0700 1 -- 172.21.15.52:0/1782750380 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({osdmap=0}) -- 0x563923d108d0 con 0x563923ea9070 2024-05-27T07:28:41.230 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:41.236+0000 7f4d5bfff700 1 --2- 172.21.15.52:0/1782750380 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x563923ea3120 0x563923eace90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-27T07:28:41.230 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:41.236+0000 7f4d615d0700 1 --2- 172.21.15.52:0/1782750380 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x563923ea9070 0x563923ead3d0 secure :-1 s=READY pgs=589 cs=0 l=1 rev1=1 crypto rx=0x7f4d54005b60 tx=0x7f4d54005b90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1442b9400015a2ef server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:28:41.230 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:41.236+0000 7f4d3bfff700 10 client.0 ms_handle_connect on v2:172.21.15.52:3300/0 2024-05-27T07:28:41.230 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:41.236+0000 7f4d3bfff700 1 -- 172.21.15.52:0/1782750380 <== mon.0 v2:172.21.15.52:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4d5400bde0 con 0x563923ea9070 2024-05-27T07:28:41.230 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:41.236+0000 7f4d71a48b80 1 -- 172.21.15.52:0/1782750380 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f4d500a7070 con 0x563923ea9070 2024-05-27T07:28:41.231 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:41.236+0000 7f4d3bfff700 1 -- 172.21.15.52:0/1782750380 <== mon.0 v2:172.21.15.52:3300/0 2 ==== osd_map(95..95 src has 1..95) ==== 9379+0+0 (secure 0 0 0) 0x7f4d54003a00 con 0x563923ea9070 2024-05-27T07:28:41.231 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:41.237+0000 7f4d3bfff700 1 -- 172.21.15.52:0/1782750380 <== mon.0 v2:172.21.15.52:3300/0 3 ==== mon_get_version_reply(handle=1 version=39) ==== 24+0+0 (secure 0 0 0) 0x7f4d54003c00 con 0x563923ea9070 2024-05-27T07:28:41.231 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:41.237+0000 7f4d71a48b80 10 client.16119 fetch_fsmap learned FSMap version 39 2024-05-27T07:28:41.231 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:41.237+0000 7f4d71a48b80 1 -- 172.21.15.52:0/1782750380 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({config=0+,fsmap=39,monmap=4+}) -- 0x563923d7a760 con 0x563923ea9070 2024-05-27T07:28:41.231 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:41.237+0000 7f4d3bfff700 1 -- 172.21.15.52:0/1782750380 <== mon.0 v2:172.21.15.52:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f4d500a7070 con 0x563923ea9070 2024-05-27T07:28:41.231 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:41.237+0000 7f4d3bfff700 1 -- 172.21.15.52:0/1782750380 <== mon.0 v2:172.21.15.52:3300/0 5 ==== fsmap(e 39) ==== 8378+0+0 (secure 0 0 0) 0x7f4d54011070 con 0x563923ea9070 2024-05-27T07:28:41.231 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:41.237+0000 7f4d71a48b80 10 client.16119 fetch_fsmap finished waiting for FSMap version 39 2024-05-27T07:28:41.231 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:41.237+0000 7f4d71a48b80 10 client.16119 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-05-27T07:28:41.231 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:41.237+0000 7f4d71a48b80 20 client.16119 populate_metadata read hostname 'smithi052' 2024-05-27T07:28:41.231 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:41.237+0000 7f4d71a48b80 1 --2- 172.21.15.52:0/1782750380 >> [v2:172.21.15.105:6838/500348151,v1:172.21.15.105:6839/500348151] conn(0x563923ebcbf0 0x563923ebf040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:28:41.231 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:41.237+0000 7f4d71a48b80 4 client.16119 mds_command: new command op to 24527 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-27T07:28:41.231 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:41.237+0000 7f4d71a48b80 1 -- 172.21.15.52:0/1782750380 --> [v2:172.21.15.105:6838/500348151,v1:172.21.15.105:6839/500348151] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x563923dd2cc0 con 0x563923ebcbf0 2024-05-27T07:28:41.232 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:41.237+0000 7f4d60dcf700 1 --2- 172.21.15.52:0/1782750380 >> [v2:172.21.15.105:6838/500348151,v1:172.21.15.105:6839/500348151] conn(0x563923ebcbf0 0x563923ebf040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:28:41.232 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:41.238+0000 7f4d60dcf700 1 --2- 172.21.15.52:0/1782750380 >> [v2:172.21.15.105:6838/500348151,v1:172.21.15.105:6839/500348151] conn(0x563923ebcbf0 0x563923ebf040 crc :-1 s=READY pgs=83 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=59e8799b736f1c45 server_cookie=d637b09eeb7dd72f in_seq=0 out_seq=0 2024-05-27T07:28:41.232 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:41.238+0000 7f4d3bfff700 10 client.16119 ms_handle_connect on v2:172.21.15.105:6838/500348151 2024-05-27T07:28:41.233 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:41.242+0000 7f4d3bfff700 1 -- 172.21.15.52:0/1782750380 <== mds.0 v2:172.21.15.105:6838/500348151 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24041 (crc 0 0 0) 0x563923dd2cc0 con 0x563923ebcbf0 2024-05-27T07:28:41.233 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:41.242+0000 7f4d3bfff700 10 client.16119 handle_command_reply: tid=0 2024-05-27T07:28:41.233 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:41.242+0000 7f4d71a48b80 1 client.16119 shutdown 2024-05-27T07:28:41.233 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:41.242+0000 7f4d71a48b80 1 -- 172.21.15.52:0/1782750380 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x563923ea9070 msgr2=0x563923ead3d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:28:41.233 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:41.242+0000 7f4d71a48b80 1 --2- 172.21.15.52:0/1782750380 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x563923ea9070 0x563923ead3d0 secure :-1 s=READY pgs=589 cs=0 l=1 rev1=1 crypto rx=0x7f4d54005b60 tx=0x7f4d54005b90 comp rx=0 tx=0).stop 2024-05-27T07:28:41.233 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:41.242+0000 7f4d71a48b80 1 -- 172.21.15.52:0/1782750380 shutdown_connections 2024-05-27T07:28:41.233 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:41.242+0000 7f4d71a48b80 1 --2- 172.21.15.52:0/1782750380 >> [v2:172.21.15.105:6838/500348151,v1:172.21.15.105:6839/500348151] conn(0x563923ebcbf0 0x563923ebf040 crc :-1 s=READY pgs=83 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:28:41.233 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:41.242+0000 7f4d71a48b80 1 --2- 172.21.15.52:0/1782750380 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x563923ea9070 0x563923ead3d0 unknown :-1 s=CLOSED pgs=589 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:28:41.233 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:41.242+0000 7f4d71a48b80 1 --2- 172.21.15.52:0/1782750380 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x563923d23e40 0x563923eac950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:28:41.233 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:41.242+0000 7f4d3bfff700 0 client.16119 ms_handle_reset on v2:172.21.15.105:6838/500348151 2024-05-27T07:28:41.233 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:41.242+0000 7f4d71a48b80 1 --2- 172.21.15.52:0/1782750380 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x563923ea3120 0x563923eace90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:28:41.234 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:41.242+0000 7f4d71a48b80 1 -- 172.21.15.52:0/1782750380 >> 172.21.15.52:0/1782750380 conn(0x563923d26770 msgr2=0x563923ea1bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:28:41.234 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:41.242+0000 7f4d71a48b80 1 -- 172.21.15.52:0/1782750380 shutdown_connections 2024-05-27T07:28:41.234 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:41.242+0000 7f4d71a48b80 1 -- 172.21.15.52:0/1782750380 wait complete. 2024-05-27T07:28:41.234 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:41.242+0000 7f4d71a48b80 20 client.16119 trim_cache size 0 max 16384 2024-05-27T07:28:41.248 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:41.258+0000 7f4d71a48b80 1 Processor -- start 2024-05-27T07:28:41.248 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:41.258+0000 7f4d71a48b80 1 -- start start 2024-05-27T07:28:41.249 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:41.258+0000 7f4d71a48b80 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x563923d23e40 0x563923ead210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:28:41.249 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:41.258+0000 7f4d71a48b80 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x563923ea3120 0x563923ea2480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:28:41.249 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:41.258+0000 7f4d71a48b80 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x563923ea9070 0x563923ea29c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:28:41.249 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:41.258+0000 7f4d71a48b80 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x7f4d5c1ae270 con 0x563923ea9070 2024-05-27T07:28:41.249 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:41.258+0000 7f4d71a48b80 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f4d5c1afe70 con 0x563923ea3120 2024-05-27T07:28:41.249 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:41.258+0000 7f4d71a48b80 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f4d5c1ae940 con 0x563923d23e40 2024-05-27T07:28:41.249 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:41.258+0000 7f4d615d0700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x563923ea9070 0x563923ea29c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:28:41.249 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:41.258+0000 7f4d5bfff700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x563923ea3120 0x563923ea2480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:28:41.249 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:41.258+0000 7f4d615d0700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x563923ea9070 0x563923ea29c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.52:46024/0 (socket says 172.21.15.52:46024) 2024-05-27T07:28:41.249 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:41.258+0000 7f4d615d0700 1 -- 172.21.15.52:0/3198362442 learned_addr learned my addr 172.21.15.52:0/3198362442 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-05-27T07:28:41.249 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:41.258+0000 7f4d5bfff700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x563923ea3120 0x563923ea2480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.52:48826/0 (socket says 172.21.15.52:48826) 2024-05-27T07:28:41.249 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:41.258+0000 7f4d60dcf700 1 --2- 172.21.15.52:0/3198362442 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x563923d23e40 0x563923ead210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:28:41.249 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:41.258+0000 7f4d615d0700 1 -- 172.21.15.52:0/3198362442 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x563923d23e40 msgr2=0x563923ead210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:28:41.250 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:41.258+0000 7f4d615d0700 1 --2- 172.21.15.52:0/3198362442 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x563923d23e40 0x563923ead210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:28:41.250 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:41.258+0000 7f4d615d0700 1 -- 172.21.15.52:0/3198362442 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x563923ea3120 msgr2=0x563923ea2480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:28:41.250 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:41.258+0000 7f4d615d0700 1 --2- 172.21.15.52:0/3198362442 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x563923ea3120 0x563923ea2480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:28:41.250 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:41.259+0000 7f4d615d0700 1 -- 172.21.15.52:0/3198362442 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x563923eaf1a0 con 0x563923ea9070 2024-05-27T07:28:41.250 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:41.259+0000 7f4d615d0700 1 --2- 172.21.15.52:0/3198362442 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x563923ea9070 0x563923ea29c0 secure :-1 s=READY pgs=590 cs=0 l=1 rev1=1 crypto rx=0x7f4d5400b010 tx=0x7f4d5400bb20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8fd455a7bb37d7fe server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:28:41.250 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:41.259+0000 7f4d467fc700 1 -- 172.21.15.52:0/3198362442 <== mon.0 v2:172.21.15.52:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4d54003a00 con 0x563923ea9070 2024-05-27T07:28:41.250 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:41.259+0000 7f4d467fc700 1 -- 172.21.15.52:0/3198362442 <== mon.0 v2:172.21.15.52:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f4d5400bde0 con 0x563923ea9070 2024-05-27T07:28:41.250 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:41.259+0000 7f4d467fc700 1 -- 172.21.15.52:0/3198362442 <== mon.0 v2:172.21.15.52:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4d54005650 con 0x563923ea9070 2024-05-27T07:28:41.253 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:41.260+0000 7f4d71a48b80 1 -- 172.21.15.52:0/3198362442 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x563923ea9070 msgr2=0x563923ea29c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:28:41.253 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:41.260+0000 7f4d71a48b80 1 --2- 172.21.15.52:0/3198362442 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x563923ea9070 0x563923ea29c0 secure :-1 s=READY pgs=590 cs=0 l=1 rev1=1 crypto rx=0x7f4d5400b010 tx=0x7f4d5400bb20 comp rx=0 tx=0).stop 2024-05-27T07:28:41.253 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:41.260+0000 7f4d71a48b80 1 -- 172.21.15.52:0/3198362442 shutdown_connections 2024-05-27T07:28:41.253 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:41.260+0000 7f4d71a48b80 1 --2- 172.21.15.52:0/3198362442 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x563923ea9070 0x563923ea29c0 unknown :-1 s=CLOSED pgs=590 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:28:41.253 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:41.260+0000 7f4d71a48b80 1 --2- 172.21.15.52:0/3198362442 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x563923d23e40 0x563923ead210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:28:41.253 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:41.260+0000 7f4d71a48b80 1 --2- 172.21.15.52:0/3198362442 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x563923ea3120 0x563923ea2480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:28:41.253 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:41.260+0000 7f4d71a48b80 1 -- 172.21.15.52:0/3198362442 >> 172.21.15.52:0/3198362442 conn(0x563923d26770 msgr2=0x563923ea1bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:28:41.253 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:41.260+0000 7f4d71a48b80 1 -- 172.21.15.52:0/3198362442 shutdown_connections 2024-05-27T07:28:41.253 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:41.260+0000 7f4d71a48b80 1 -- 172.21.15.52:0/3198362442 wait complete. 2024-05-27T07:28:41.253 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:41.260+0000 7f4d71a48b80 1 -- start start 2024-05-27T07:28:41.253 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:41.260+0000 7f4d71a48b80 1 Processor -- start 2024-05-27T07:28:41.254 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:41.261+0000 7f4d71a48b80 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x563923d23e40 0x563923ec4ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:28:41.254 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:41.261+0000 7f4d71a48b80 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x563923ea3120 0x563923eb4df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:28:41.254 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:41.261+0000 7f4d71a48b80 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x563923ea9070 0x563923eb5330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:28:41.254 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:41.261+0000 7f4d71a48b80 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x7f4d5c1af7a0 con 0x563923ea3120 2024-05-27T07:28:41.254 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:41.261+0000 7f4d71a48b80 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f4d5c1af010 con 0x563923d23e40 2024-05-27T07:28:41.254 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:41.261+0000 7f4d71a48b80 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f4d5c1ac210 con 0x563923ea9070 2024-05-27T07:28:41.254 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:41.261+0000 7f4d71a48b80 10 client.0 _ll_register_callbacks cb 0x563923d575b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T07:28:41.254 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:41.261+0000 7f4d60dcf700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x563923d23e40 0x563923ec4ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:28:41.254 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:41.261+0000 7f4d5bfff700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x563923ea3120 0x563923eb4df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:28:41.254 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:41.261+0000 7f4d60dcf700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x563923d23e40 0x563923ec4ca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.52:48832/0 (socket says 172.21.15.52:48832) 2024-05-27T07:28:41.254 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:41.261+0000 7f4d60dcf700 1 -- 172.21.15.52:0/2869275149 learned_addr learned my addr 172.21.15.52:0/2869275149 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-05-27T07:28:41.254 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:41.261+0000 7f4d615d0700 1 --2- 172.21.15.52:0/2869275149 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x563923ea9070 0x563923eb5330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:28:41.254 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:41.261+0000 7f4d5bfff700 1 -- 172.21.15.52:0/2869275149 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x563923ea9070 msgr2=0x563923eb5330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:28:41.255 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:41.261+0000 7f4d5bfff700 1 --2- 172.21.15.52:0/2869275149 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x563923ea9070 0x563923eb5330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:28:41.255 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:41.261+0000 7f4d5bfff700 1 -- 172.21.15.52:0/2869275149 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x563923d23e40 msgr2=0x563923ec4ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:28:41.255 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:41.261+0000 7f4d5bfff700 1 --2- 172.21.15.52:0/2869275149 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x563923d23e40 0x563923ec4ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:28:41.255 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:41.261+0000 7f4d5bfff700 1 -- 172.21.15.52:0/2869275149 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({osdmap=0}) -- 0x563923eb58a0 con 0x563923ea3120 2024-05-27T07:28:41.255 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:41.261+0000 7f4d60dcf700 1 --2- 172.21.15.52:0/2869275149 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x563923d23e40 0x563923ec4ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-27T07:28:41.255 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:41.262+0000 7f4d5bfff700 1 --2- 172.21.15.52:0/2869275149 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x563923ea3120 0x563923eb4df0 secure :-1 s=READY pgs=591 cs=0 l=1 rev1=1 crypto rx=0x563923e9ecf0 tx=0x7f4d480047c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a05bbd20cc4efe92 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:28:41.255 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:41.262+0000 7f4d3bfff700 10 client.0 ms_handle_connect on v2:172.21.15.52:3300/0 2024-05-27T07:28:41.255 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:41.262+0000 7f4d3bfff700 1 -- 172.21.15.52:0/2869275149 <== mon.0 v2:172.21.15.52:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4d4802d060 con 0x563923ea3120 2024-05-27T07:28:41.255 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:41.262+0000 7f4d71a48b80 1 -- 172.21.15.52:0/2869275149 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f4d5400bde0 con 0x563923ea3120 2024-05-27T07:28:41.255 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:41.262+0000 7f4d3bfff700 1 -- 172.21.15.52:0/2869275149 <== mon.0 v2:172.21.15.52:3300/0 2 ==== osd_map(95..95 src has 1..95) ==== 9379+0+0 (secure 0 0 0) 0x7f4d48028680 con 0x563923ea3120 2024-05-27T07:28:41.255 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:41.262+0000 7f4d3bfff700 1 -- 172.21.15.52:0/2869275149 <== mon.0 v2:172.21.15.52:3300/0 3 ==== mon_get_version_reply(handle=1 version=39) ==== 24+0+0 (secure 0 0 0) 0x7f4d48034050 con 0x563923ea3120 2024-05-27T07:28:41.255 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:41.262+0000 7f4d71a48b80 10 client.16125 fetch_fsmap learned FSMap version 39 2024-05-27T07:28:41.255 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:41.262+0000 7f4d71a48b80 1 -- 172.21.15.52:0/2869275149 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({config=0+,fsmap=39,monmap=4+}) -- 0x563923f266b0 con 0x563923ea3120 2024-05-27T07:28:41.256 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:41.262+0000 7f4d3bfff700 1 -- 172.21.15.52:0/2869275149 <== mon.0 v2:172.21.15.52:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f4d5400bde0 con 0x563923ea3120 2024-05-27T07:28:41.256 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:41.262+0000 7f4d3bfff700 1 -- 172.21.15.52:0/2869275149 <== mon.0 v2:172.21.15.52:3300/0 5 ==== fsmap(e 39) ==== 8378+0+0 (secure 0 0 0) 0x7f4d48018040 con 0x563923ea3120 2024-05-27T07:28:41.256 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:41.263+0000 7f4d71a48b80 10 client.16125 fetch_fsmap finished waiting for FSMap version 39 2024-05-27T07:28:41.256 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:41.263+0000 7f4d71a48b80 10 client.16125 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-05-27T07:28:41.256 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:41.263+0000 7f4d71a48b80 20 client.16125 populate_metadata read hostname 'smithi052' 2024-05-27T07:28:41.256 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:41.263+0000 7f4d71a48b80 1 --2- 172.21.15.52:0/2869275149 >> [v2:172.21.15.105:6838/500348151,v1:172.21.15.105:6839/500348151] conn(0x563923f26df0 0x563923f29210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:28:41.256 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:41.263+0000 7f4d71a48b80 4 client.16125 mds_command: new command op to 24527 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-05-27T07:28:41.256 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:41.263+0000 7f4d71a48b80 1 -- 172.21.15.52:0/2869275149 --> [v2:172.21.15.105:6838/500348151,v1:172.21.15.105:6839/500348151] -- command(tid 0: {"prefix": "damage ls"}) -- 0x563923dcc4a0 con 0x563923f26df0 2024-05-27T07:28:41.256 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:41.263+0000 7f4d60dcf700 1 --2- 172.21.15.52:0/2869275149 >> [v2:172.21.15.105:6838/500348151,v1:172.21.15.105:6839/500348151] conn(0x563923f26df0 0x563923f29210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:28:41.256 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:41.263+0000 7f4d3bfff700 10 client.16125 ms_handle_connect on v2:172.21.15.105:6838/500348151 2024-05-27T07:28:41.256 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:41.263+0000 7f4d60dcf700 1 --2- 172.21.15.52:0/2869275149 >> [v2:172.21.15.105:6838/500348151,v1:172.21.15.105:6839/500348151] conn(0x563923f26df0 0x563923f29210 crc :-1 s=READY pgs=84 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=4cbf40f58d87bcc9 server_cookie=66fc1461c34d7a90 in_seq=0 out_seq=0 2024-05-27T07:28:41.259 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:41.268+0000 7f4d3bfff700 1 -- 172.21.15.52:0/2869275149 <== mds.0 v2:172.21.15.105:6838/500348151 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x563923dcc4a0 con 0x563923f26df0 2024-05-27T07:28:41.259 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:41.268+0000 7f4d3bfff700 10 client.16125 handle_command_reply: tid=0 2024-05-27T07:28:41.259 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:41.268+0000 7f4d71a48b80 1 client.16125 shutdown 2024-05-27T07:28:41.259 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:41.269+0000 7f4d71a48b80 1 -- 172.21.15.52:0/2869275149 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x563923ea3120 msgr2=0x563923eb4df0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:28:41.259 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:41.269+0000 7f4d71a48b80 1 --2- 172.21.15.52:0/2869275149 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x563923ea3120 0x563923eb4df0 secure :-1 s=READY pgs=591 cs=0 l=1 rev1=1 crypto rx=0x563923e9ecf0 tx=0x7f4d480047c0 comp rx=0 tx=0).stop 2024-05-27T07:28:41.259 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:41.269+0000 7f4d71a48b80 1 -- 172.21.15.52:0/2869275149 shutdown_connections 2024-05-27T07:28:41.259 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:41.269+0000 7f4d71a48b80 1 --2- 172.21.15.52:0/2869275149 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x563923ea9070 0x563923eb5330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:28:41.259 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:41.269+0000 7f4d71a48b80 1 --2- 172.21.15.52:0/2869275149 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x563923d23e40 0x563923ec4ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:28:41.259 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:41.269+0000 7f4d71a48b80 1 --2- 172.21.15.52:0/2869275149 >> [v2:172.21.15.105:6838/500348151,v1:172.21.15.105:6839/500348151] conn(0x563923f26df0 0x563923f29210 crc :-1 s=READY pgs=84 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:28:41.259 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:41.269+0000 7f4d71a48b80 1 --2- 172.21.15.52:0/2869275149 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x563923ea3120 0x563923eb4df0 unknown :-1 s=CLOSED pgs=591 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:28:41.260 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:41.269+0000 7f4d3bfff700 0 client.16125 ms_handle_reset on v2:172.21.15.105:6838/500348151 2024-05-27T07:28:41.260 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:41.269+0000 7f4d71a48b80 1 -- 172.21.15.52:0/2869275149 >> 172.21.15.52:0/2869275149 conn(0x563923d26770 msgr2=0x563923ea1bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:28:41.260 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:41.269+0000 7f4d71a48b80 1 -- 172.21.15.52:0/2869275149 shutdown_connections 2024-05-27T07:28:41.260 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:41.269+0000 7f4d71a48b80 1 -- 172.21.15.52:0/2869275149 wait complete. 2024-05-27T07:28:41.260 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:41.269+0000 7f4d71a48b80 20 client.16125 trim_cache size 0 max 16384 2024-05-27T07:28:41.260 INFO:teuthology.orchestra.run.smithi052.stdout:[] 2024-05-27T07:28:41.261 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:41.270+0000 7f4d63033700 1 -- 172.21.15.52:0/648989117 >> [v2:172.21.15.52:6800/1822318384,v1:172.21.15.52:6801/1822318384] conn(0x7f4d4c0b29f0 msgr2=0x7f4d4c0b4f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:28:41.261 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:41.270+0000 7f4d63033700 1 --2- 172.21.15.52:0/648989117 >> [v2:172.21.15.52:6800/1822318384,v1:172.21.15.52:6801/1822318384] conn(0x7f4d4c0b29f0 0x7f4d4c0b4f00 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f4d48009680 tx=0x7f4d48009260 comp rx=0 tx=0).stop 2024-05-27T07:28:41.261 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:41.270+0000 7f4d63033700 1 -- 172.21.15.52:0/648989117 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f4d5c1034b0 msgr2=0x7f4d5c19f880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:28:41.261 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:41.270+0000 7f4d63033700 1 --2- 172.21.15.52:0/648989117 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f4d5c1034b0 0x7f4d5c19f880 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7f4d5000eab0 tx=0x7f4d5000edc0 comp rx=0 tx=0).stop 2024-05-27T07:28:41.261 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:41.270+0000 7f4d63033700 1 -- 172.21.15.52:0/648989117 shutdown_connections 2024-05-27T07:28:41.261 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:41.270+0000 7f4d63033700 1 --2- 172.21.15.52:0/648989117 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f4d5c108730 0x7f4d5c199aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:28:41.261 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:41.270+0000 7f4d63033700 1 --2- 172.21.15.52:0/648989117 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f4d5c1034b0 0x7f4d5c19f880 unknown :-1 s=CLOSED pgs=101 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:28:41.261 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:41.270+0000 7f4d63033700 1 --2- 172.21.15.52:0/648989117 >> [v2:172.21.15.52:6800/1822318384,v1:172.21.15.52:6801/1822318384] conn(0x7f4d4c0b29f0 0x7f4d4c0b4f00 unknown :-1 s=CLOSED pgs=94 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:28:41.261 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:41.270+0000 7f4d63033700 1 --2- 172.21.15.52:0/648989117 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f4d5c107de0 0x7f4d5c19fdc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:28:41.261 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:41.270+0000 7f4d63033700 1 -- 172.21.15.52:0/648989117 >> 172.21.15.52:0/648989117 conn(0x7f4d5c0fede0 msgr2=0x7f4d5c107550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:28:41.261 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:41.270+0000 7f4d63033700 1 -- 172.21.15.52:0/648989117 shutdown_connections 2024-05-27T07:28:41.262 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:41.270+0000 7f4d63033700 1 -- 172.21.15.52:0/648989117 wait complete. 2024-05-27T07:28:41.464 INFO:journalctl@ceph.mds.f.smithi173.stdout:May 27 07:28:41 smithi173 systemd[1]: ceph-73df2636-1bf8-11ef-bc9b-c7b262605968@mds.f.service: Scheduled restart job, restart counter is at 3. 2024-05-27T07:28:41.464 INFO:journalctl@ceph.mds.f.smithi173.stdout:May 27 07:28:41 smithi173 systemd[1]: Stopped Ceph mds.f for 73df2636-1bf8-11ef-bc9b-c7b262605968. 2024-05-27T07:28:41.464 INFO:journalctl@ceph.mds.f.smithi173.stdout:May 27 07:28:41 smithi173 systemd[1]: Starting Ceph mds.f for 73df2636-1bf8-11ef-bc9b-c7b262605968... 2024-05-27T07:28:41.464 INFO:journalctl@ceph.mds.f.smithi173.stdout:May 27 07:28:41 smithi173 podman[72691]: 2024-05-27 07:28:41.240824411 +0000 UTC m=+0.017259140 image pull e36ea4fc8a5807e8d2701370e350a0de3856538797f7fba64cfce7e344bd6799 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6 2024-05-27T07:28:41.464 INFO:journalctl@ceph.mds.f.smithi173.stdout:May 27 07:28:41 smithi173 podman[72691]: 2024-05-27 07:28:41.36249957 +0000 UTC m=+0.138934287 container create 56317baf1a0980a654662f6738405bfe86020b01b97b00363257847d5a3a4b76 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-f, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, org.label-schema.build-date=20240521, GIT_CLEAN=True, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, ceph=True, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0) 2024-05-27T07:28:41.852 INFO:journalctl@ceph.mds.f.smithi173.stdout:May 27 07:28:41 smithi173 podman[72691]: 2024-05-27 07:28:41.590126816 +0000 UTC m=+0.366561551 container init 56317baf1a0980a654662f6738405bfe86020b01b97b00363257847d5a3a4b76 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-f, maintainer=Guillaume Abrioux , GIT_CLEAN=True, GIT_BRANCH=HEAD, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, org.label-schema.build-date=20240521, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS) 2024-05-27T07:28:41.852 INFO:journalctl@ceph.mds.f.smithi173.stdout:May 27 07:28:41 smithi173 podman[72691]: 2024-05-27 07:28:41.595916027 +0000 UTC m=+0.372350759 container start 56317baf1a0980a654662f6738405bfe86020b01b97b00363257847d5a3a4b76 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-f, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240521, CEPH_POINT_RELEASE=, GIT_CLEAN=True, GIT_BRANCH=HEAD, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-05-27T07:28:41.852 INFO:journalctl@ceph.mds.f.smithi173.stdout:May 27 07:28:41 smithi173 ceph-mds[72710]: -- 172.21.15.173:0/2381865863 <== mon.1 v2:172.21.15.173:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55e1a1279a40 con 0x55e1a12c1a80 2024-05-27T07:28:41.852 INFO:journalctl@ceph.mds.f.smithi173.stdout:May 27 07:28:41 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-f[72706]: starting mds.f at 2024-05-27T07:28:41.852 INFO:journalctl@ceph.mds.f.smithi173.stdout:May 27 07:28:41 smithi173 bash[72691]: 56317baf1a0980a654662f6738405bfe86020b01b97b00363257847d5a3a4b76 2024-05-27T07:28:41.852 INFO:journalctl@ceph.mds.f.smithi173.stdout:May 27 07:28:41 smithi173 systemd[1]: Started Ceph mds.f for 73df2636-1bf8-11ef-bc9b-c7b262605968. 2024-05-27T07:28:42.229 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:28:42 smithi052 ceph-mon[31955]: pgmap v89: 129 pgs: 129 active+clean; 103 MiB data, 625 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 MiB/s rd, 3.8 MiB/s wr, 1.07k op/s 2024-05-27T07:28:42.229 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:28:42 smithi052 ceph-mon[31955]: daemon mds.f restarted 2024-05-27T07:28:42.351 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:28:42 smithi173 ceph-mon[37000]: pgmap v89: 129 pgs: 129 active+clean; 103 MiB data, 625 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 MiB/s rd, 3.8 MiB/s wr, 1.07k op/s 2024-05-27T07:28:42.351 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:28:42 smithi173 ceph-mon[37000]: daemon mds.f restarted 2024-05-27T07:28:42.352 INFO:journalctl@ceph.mds.f.smithi173.stdout:May 27 07:28:42 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-f[72706]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: In function 'std::set SnapClient::filter(const std::set&) const' thread 7f60548ca700 time 2024-05-27T07:28:42.162784+0000 2024-05-27T07:28:42.352 INFO:journalctl@ceph.mds.f.smithi173.stdout:May 27 07:28:42 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-f[72706]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: 197: FAILED ceph_assert(cached_version > 0) 2024-05-27T07:28:42.352 INFO:journalctl@ceph.mds.f.smithi173.stdout:May 27 07:28:42 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-f[72706]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:28:42.352 INFO:journalctl@ceph.mds.f.smithi173.stdout:May 27 07:28:42 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-f[72706]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x127) [0x7f606710fba6] 2024-05-27T07:28:42.352 INFO:journalctl@ceph.mds.f.smithi173.stdout:May 27 07:28:42 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-f[72706]: 2: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7f606710fdd1] 2024-05-27T07:28:42.353 INFO:journalctl@ceph.mds.f.smithi173.stdout:May 27 07:28:42 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-f[72706]: 3: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x55e1a065423d] 2024-05-27T07:28:42.353 INFO:journalctl@ceph.mds.f.smithi173.stdout:May 27 07:28:42 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-f[72706]: 4: (SnapRealm::build_snap_set() const+0x323) [0x55e1a063f971] 2024-05-27T07:28:42.353 INFO:journalctl@ceph.mds.f.smithi173.stdout:May 27 07:28:42 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-f[72706]: 5: (SnapRealm::check_cache() const+0x13e) [0x55e1a063dfa0] 2024-05-27T07:28:42.353 INFO:journalctl@ceph.mds.f.smithi173.stdout:May 27 07:28:42 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-f[72706]: 6: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x55e1a063e651] 2024-05-27T07:28:42.353 INFO:journalctl@ceph.mds.f.smithi173.stdout:May 27 07:28:42 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-f[72706]: 7: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x55e1a05d3347] 2024-05-27T07:28:42.353 INFO:journalctl@ceph.mds.f.smithi173.stdout:May 27 07:28:42 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-f[72706]: 8: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x55e1a02cf87e] 2024-05-27T07:28:42.353 INFO:journalctl@ceph.mds.f.smithi173.stdout:May 27 07:28:42 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-f[72706]: 9: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x279d) [0x55e1a02d4f2f] 2024-05-27T07:28:42.353 INFO:journalctl@ceph.mds.f.smithi173.stdout:May 27 07:28:42 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-f[72706]: 10: (EUpdate::replay(MDSRank*)+0x63) [0x55e1a02df969] 2024-05-27T07:28:42.353 INFO:journalctl@ceph.mds.f.smithi173.stdout:May 27 07:28:42 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-f[72706]: 11: (MDLog::_replay_thread()+0x1f38) [0x55e1a067848c] 2024-05-27T07:28:42.353 INFO:journalctl@ceph.mds.f.smithi173.stdout:May 27 07:28:42 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-f[72706]: 12: (MDLog::ReplayThread::entry()+0x11) [0x55e1a06819cb] 2024-05-27T07:28:42.353 INFO:journalctl@ceph.mds.f.smithi173.stdout:May 27 07:28:42 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-f[72706]: 13: (Thread::entry_wrapper()+0x43) [0x7f60670de72d] 2024-05-27T07:28:42.353 INFO:journalctl@ceph.mds.f.smithi173.stdout:May 27 07:28:42 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-f[72706]: 14: (Thread::_entry_func(void*)+0xd) [0x7f60670de749] 2024-05-27T07:28:42.354 INFO:journalctl@ceph.mds.f.smithi173.stdout:May 27 07:28:42 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-f[72706]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7f6065c821ca] 2024-05-27T07:28:42.354 INFO:journalctl@ceph.mds.f.smithi173.stdout:May 27 07:28:42 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-f[72706]: 16: clone() 2024-05-27T07:28:42.354 INFO:journalctl@ceph.mds.f.smithi173.stdout:May 27 07:28:42 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-f[72706]: 2024-05-27T07:28:42.164+0000 7f60548ca700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: In function 'std::set SnapClient::filter(const std::set&) const' thread 7f60548ca700 time 2024-05-27T07:28:42.162784+0000 2024-05-27T07:28:42.354 INFO:journalctl@ceph.mds.f.smithi173.stdout:May 27 07:28:42 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-f[72706]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: 197: FAILED ceph_assert(cached_version > 0) 2024-05-27T07:28:42.354 INFO:journalctl@ceph.mds.f.smithi173.stdout:May 27 07:28:42 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-f[72706]: 2024-05-27T07:28:42.354 INFO:journalctl@ceph.mds.f.smithi173.stdout:May 27 07:28:42 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-f[72706]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:28:42.354 INFO:journalctl@ceph.mds.f.smithi173.stdout:May 27 07:28:42 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-f[72706]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x127) [0x7f606710fba6] 2024-05-27T07:28:42.354 INFO:journalctl@ceph.mds.f.smithi173.stdout:May 27 07:28:42 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-f[72706]: 2: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7f606710fdd1] 2024-05-27T07:28:42.354 INFO:journalctl@ceph.mds.f.smithi173.stdout:May 27 07:28:42 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-f[72706]: 3: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x55e1a065423d] 2024-05-27T07:28:42.354 INFO:journalctl@ceph.mds.f.smithi173.stdout:May 27 07:28:42 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-f[72706]: 4: (SnapRealm::build_snap_set() const+0x323) [0x55e1a063f971] 2024-05-27T07:28:42.354 INFO:journalctl@ceph.mds.f.smithi173.stdout:May 27 07:28:42 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-f[72706]: 5: (SnapRealm::check_cache() const+0x13e) [0x55e1a063dfa0] 2024-05-27T07:28:42.355 INFO:journalctl@ceph.mds.f.smithi173.stdout:May 27 07:28:42 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-f[72706]: 6: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x55e1a063e651] 2024-05-27T07:28:42.355 INFO:journalctl@ceph.mds.f.smithi173.stdout:May 27 07:28:42 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-f[72706]: 7: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x55e1a05d3347] 2024-05-27T07:28:42.355 INFO:journalctl@ceph.mds.f.smithi173.stdout:May 27 07:28:42 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-f[72706]: 8: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x55e1a02cf87e] 2024-05-27T07:28:42.355 INFO:journalctl@ceph.mds.f.smithi173.stdout:May 27 07:28:42 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-f[72706]: 9: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x279d) [0x55e1a02d4f2f] 2024-05-27T07:28:42.355 INFO:journalctl@ceph.mds.f.smithi173.stdout:May 27 07:28:42 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-f[72706]: 10: (EUpdate::replay(MDSRank*)+0x63) [0x55e1a02df969] 2024-05-27T07:28:42.355 INFO:journalctl@ceph.mds.f.smithi173.stdout:May 27 07:28:42 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-f[72706]: 11: (MDLog::_replay_thread()+0x1f38) [0x55e1a067848c] 2024-05-27T07:28:42.355 INFO:journalctl@ceph.mds.f.smithi173.stdout:May 27 07:28:42 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-f[72706]: 12: (MDLog::ReplayThread::entry()+0x11) [0x55e1a06819cb] 2024-05-27T07:28:42.355 INFO:journalctl@ceph.mds.f.smithi173.stdout:May 27 07:28:42 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-f[72706]: 13: (Thread::entry_wrapper()+0x43) [0x7f60670de72d] 2024-05-27T07:28:42.355 INFO:journalctl@ceph.mds.f.smithi173.stdout:May 27 07:28:42 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-f[72706]: 14: (Thread::_entry_func(void*)+0xd) [0x7f60670de749] 2024-05-27T07:28:42.355 INFO:journalctl@ceph.mds.f.smithi173.stdout:May 27 07:28:42 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-f[72706]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7f6065c821ca] 2024-05-27T07:28:42.355 INFO:journalctl@ceph.mds.f.smithi173.stdout:May 27 07:28:42 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-f[72706]: 16: clone() 2024-05-27T07:28:42.355 INFO:journalctl@ceph.mds.f.smithi173.stdout:May 27 07:28:42 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-f[72706]: 2024-05-27T07:28:42.355 INFO:journalctl@ceph.mds.f.smithi173.stdout:May 27 07:28:42 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-f[72706]: *** Caught signal (Aborted) ** 2024-05-27T07:28:42.356 INFO:journalctl@ceph.mds.f.smithi173.stdout:May 27 07:28:42 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-f[72706]: in thread 7f60548ca700 thread_name:md_log_replay 2024-05-27T07:28:42.356 INFO:journalctl@ceph.mds.f.smithi173.stdout:May 27 07:28:42 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-f[72706]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:28:42.356 INFO:journalctl@ceph.mds.f.smithi173.stdout:May 27 07:28:42 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-f[72706]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f6065c8cd20] 2024-05-27T07:28:42.356 INFO:journalctl@ceph.mds.f.smithi173.stdout:May 27 07:28:42 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-f[72706]: 2: gsignal() 2024-05-27T07:28:42.356 INFO:journalctl@ceph.mds.f.smithi173.stdout:May 27 07:28:42 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-f[72706]: 3: abort() 2024-05-27T07:28:42.356 INFO:journalctl@ceph.mds.f.smithi173.stdout:May 27 07:28:42 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-f[72706]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x26e) [0x7f606710fced] 2024-05-27T07:28:42.356 INFO:journalctl@ceph.mds.f.smithi173.stdout:May 27 07:28:42 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-f[72706]: 5: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7f606710fdd1] 2024-05-27T07:28:42.357 INFO:journalctl@ceph.mds.f.smithi173.stdout:May 27 07:28:42 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-f[72706]: 6: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x55e1a065423d] 2024-05-27T07:28:42.357 INFO:journalctl@ceph.mds.f.smithi173.stdout:May 27 07:28:42 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-f[72706]: 7: (SnapRealm::build_snap_set() const+0x323) [0x55e1a063f971] 2024-05-27T07:28:42.357 INFO:journalctl@ceph.mds.f.smithi173.stdout:May 27 07:28:42 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-f[72706]: 8: (SnapRealm::check_cache() const+0x13e) [0x55e1a063dfa0] 2024-05-27T07:28:42.357 INFO:journalctl@ceph.mds.f.smithi173.stdout:May 27 07:28:42 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-f[72706]: 9: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x55e1a063e651] 2024-05-27T07:28:42.357 INFO:journalctl@ceph.mds.f.smithi173.stdout:May 27 07:28:42 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-f[72706]: 10: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x55e1a05d3347] 2024-05-27T07:28:42.357 INFO:journalctl@ceph.mds.f.smithi173.stdout:May 27 07:28:42 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-f[72706]: 11: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x55e1a02cf87e] 2024-05-27T07:28:42.357 INFO:journalctl@ceph.mds.f.smithi173.stdout:May 27 07:28:42 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-f[72706]: 12: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x279d) [0x55e1a02d4f2f] 2024-05-27T07:28:42.358 INFO:journalctl@ceph.mds.f.smithi173.stdout:May 27 07:28:42 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-f[72706]: 13: (EUpdate::replay(MDSRank*)+0x63) [0x55e1a02df969] 2024-05-27T07:28:42.358 INFO:journalctl@ceph.mds.f.smithi173.stdout:May 27 07:28:42 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-f[72706]: 14: (MDLog::_replay_thread()+0x1f38) [0x55e1a067848c] 2024-05-27T07:28:42.358 INFO:journalctl@ceph.mds.f.smithi173.stdout:May 27 07:28:42 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-f[72706]: 15: (MDLog::ReplayThread::entry()+0x11) [0x55e1a06819cb] 2024-05-27T07:28:42.358 INFO:journalctl@ceph.mds.f.smithi173.stdout:May 27 07:28:42 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-f[72706]: 16: (Thread::entry_wrapper()+0x43) [0x7f60670de72d] 2024-05-27T07:28:42.358 INFO:journalctl@ceph.mds.f.smithi173.stdout:May 27 07:28:42 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-f[72706]: 17: (Thread::_entry_func(void*)+0xd) [0x7f60670de749] 2024-05-27T07:28:42.358 INFO:journalctl@ceph.mds.f.smithi173.stdout:May 27 07:28:42 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-f[72706]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7f6065c821ca] 2024-05-27T07:28:42.358 INFO:journalctl@ceph.mds.f.smithi173.stdout:May 27 07:28:42 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-f[72706]: 19: clone() 2024-05-27T07:28:42.358 INFO:journalctl@ceph.mds.f.smithi173.stdout:May 27 07:28:42 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-f[72706]: 2024-05-27T07:28:42.167+0000 7f60548ca700 -1 *** Caught signal (Aborted) ** 2024-05-27T07:28:42.358 INFO:journalctl@ceph.mds.f.smithi173.stdout:May 27 07:28:42 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-f[72706]: in thread 7f60548ca700 thread_name:md_log_replay 2024-05-27T07:28:42.358 INFO:journalctl@ceph.mds.f.smithi173.stdout:May 27 07:28:42 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-f[72706]: 2024-05-27T07:28:42.358 INFO:journalctl@ceph.mds.f.smithi173.stdout:May 27 07:28:42 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-f[72706]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:28:42.358 INFO:journalctl@ceph.mds.f.smithi173.stdout:May 27 07:28:42 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-f[72706]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f6065c8cd20] 2024-05-27T07:28:42.358 INFO:journalctl@ceph.mds.f.smithi173.stdout:May 27 07:28:42 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-f[72706]: 2: gsignal() 2024-05-27T07:28:42.359 INFO:journalctl@ceph.mds.f.smithi173.stdout:May 27 07:28:42 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-f[72706]: 3: abort() 2024-05-27T07:28:42.359 INFO:journalctl@ceph.mds.f.smithi173.stdout:May 27 07:28:42 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-f[72706]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x26e) [0x7f606710fced] 2024-05-27T07:28:42.359 INFO:journalctl@ceph.mds.f.smithi173.stdout:May 27 07:28:42 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-f[72706]: 5: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7f606710fdd1] 2024-05-27T07:28:42.359 INFO:journalctl@ceph.mds.f.smithi173.stdout:May 27 07:28:42 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-f[72706]: 6: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x55e1a065423d] 2024-05-27T07:28:42.359 INFO:journalctl@ceph.mds.f.smithi173.stdout:May 27 07:28:42 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-f[72706]: 7: (SnapRealm::build_snap_set() const+0x323) [0x55e1a063f971] 2024-05-27T07:28:42.359 INFO:journalctl@ceph.mds.f.smithi173.stdout:May 27 07:28:42 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-f[72706]: 8: (SnapRealm::check_cache() const+0x13e) [0x55e1a063dfa0] 2024-05-27T07:28:42.359 INFO:journalctl@ceph.mds.f.smithi173.stdout:May 27 07:28:42 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-f[72706]: 9: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x55e1a063e651] 2024-05-27T07:28:42.359 INFO:journalctl@ceph.mds.f.smithi173.stdout:May 27 07:28:42 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-f[72706]: 10: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x55e1a05d3347] 2024-05-27T07:28:42.359 INFO:journalctl@ceph.mds.f.smithi173.stdout:May 27 07:28:42 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-f[72706]: 11: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x55e1a02cf87e] 2024-05-27T07:28:42.359 INFO:journalctl@ceph.mds.f.smithi173.stdout:May 27 07:28:42 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-f[72706]: 12: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x279d) [0x55e1a02d4f2f] 2024-05-27T07:28:42.359 INFO:journalctl@ceph.mds.f.smithi173.stdout:May 27 07:28:42 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-f[72706]: 13: (EUpdate::replay(MDSRank*)+0x63) [0x55e1a02df969] 2024-05-27T07:28:42.359 INFO:journalctl@ceph.mds.f.smithi173.stdout:May 27 07:28:42 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-f[72706]: 14: (MDLog::_replay_thread()+0x1f38) [0x55e1a067848c] 2024-05-27T07:28:42.359 INFO:journalctl@ceph.mds.f.smithi173.stdout:May 27 07:28:42 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-f[72706]: 15: (MDLog::ReplayThread::entry()+0x11) [0x55e1a06819cb] 2024-05-27T07:28:42.360 INFO:journalctl@ceph.mds.f.smithi173.stdout:May 27 07:28:42 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-f[72706]: 16: (Thread::entry_wrapper()+0x43) [0x7f60670de72d] 2024-05-27T07:28:42.360 INFO:journalctl@ceph.mds.f.smithi173.stdout:May 27 07:28:42 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-f[72706]: 17: (Thread::_entry_func(void*)+0xd) [0x7f60670de749] 2024-05-27T07:28:42.360 INFO:journalctl@ceph.mds.f.smithi173.stdout:May 27 07:28:42 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-f[72706]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7f6065c821ca] 2024-05-27T07:28:42.360 INFO:journalctl@ceph.mds.f.smithi173.stdout:May 27 07:28:42 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-f[72706]: 19: clone() 2024-05-27T07:28:42.360 INFO:journalctl@ceph.mds.f.smithi173.stdout:May 27 07:28:42 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-f[72706]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-05-27T07:28:42.360 INFO:journalctl@ceph.mds.f.smithi173.stdout:May 27 07:28:42 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-f[72706]: 2024-05-27T07:28:42.360 INFO:journalctl@ceph.mds.f.smithi173.stdout:May 27 07:28:42 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-f[72706]: -1> 2024-05-27T07:28:42.164+0000 7f60548ca700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: In function 'std::set SnapClient::filter(const std::set&) const' thread 7f60548ca700 time 2024-05-27T07:28:42.162784+0000 2024-05-27T07:28:42.360 INFO:journalctl@ceph.mds.f.smithi173.stdout:May 27 07:28:42 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-f[72706]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: 197: FAILED ceph_assert(cached_version > 0) 2024-05-27T07:28:42.360 INFO:journalctl@ceph.mds.f.smithi173.stdout:May 27 07:28:42 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-f[72706]: 2024-05-27T07:28:42.360 INFO:journalctl@ceph.mds.f.smithi173.stdout:May 27 07:28:42 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-f[72706]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:28:42.360 INFO:journalctl@ceph.mds.f.smithi173.stdout:May 27 07:28:42 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-f[72706]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x127) [0x7f606710fba6] 2024-05-27T07:28:42.360 INFO:journalctl@ceph.mds.f.smithi173.stdout:May 27 07:28:42 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-f[72706]: 2: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7f606710fdd1] 2024-05-27T07:28:42.360 INFO:journalctl@ceph.mds.f.smithi173.stdout:May 27 07:28:42 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-f[72706]: 3: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x55e1a065423d] 2024-05-27T07:28:42.360 INFO:journalctl@ceph.mds.f.smithi173.stdout:May 27 07:28:42 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-f[72706]: 4: (SnapRealm::build_snap_set() const+0x323) [0x55e1a063f971] 2024-05-27T07:28:42.361 INFO:journalctl@ceph.mds.f.smithi173.stdout:May 27 07:28:42 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-f[72706]: 5: (SnapRealm::check_cache() const+0x13e) [0x55e1a063dfa0] 2024-05-27T07:28:42.361 INFO:journalctl@ceph.mds.f.smithi173.stdout:May 27 07:28:42 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-f[72706]: 6: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x55e1a063e651] 2024-05-27T07:28:42.361 INFO:journalctl@ceph.mds.f.smithi173.stdout:May 27 07:28:42 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-f[72706]: 7: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x55e1a05d3347] 2024-05-27T07:28:42.361 INFO:journalctl@ceph.mds.f.smithi173.stdout:May 27 07:28:42 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-f[72706]: 8: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x55e1a02cf87e] 2024-05-27T07:28:42.361 INFO:journalctl@ceph.mds.f.smithi173.stdout:May 27 07:28:42 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-f[72706]: 9: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x279d) [0x55e1a02d4f2f] 2024-05-27T07:28:42.361 INFO:journalctl@ceph.mds.f.smithi173.stdout:May 27 07:28:42 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-f[72706]: 10: (EUpdate::replay(MDSRank*)+0x63) [0x55e1a02df969] 2024-05-27T07:28:42.361 INFO:journalctl@ceph.mds.f.smithi173.stdout:May 27 07:28:42 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-f[72706]: 11: (MDLog::_replay_thread()+0x1f38) [0x55e1a067848c] 2024-05-27T07:28:42.361 INFO:journalctl@ceph.mds.f.smithi173.stdout:May 27 07:28:42 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-f[72706]: 12: (MDLog::ReplayThread::entry()+0x11) [0x55e1a06819cb] 2024-05-27T07:28:42.361 INFO:journalctl@ceph.mds.f.smithi173.stdout:May 27 07:28:42 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-f[72706]: 13: (Thread::entry_wrapper()+0x43) [0x7f60670de72d] 2024-05-27T07:28:42.361 INFO:journalctl@ceph.mds.f.smithi173.stdout:May 27 07:28:42 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-f[72706]: 14: (Thread::_entry_func(void*)+0xd) [0x7f60670de749] 2024-05-27T07:28:42.361 INFO:journalctl@ceph.mds.f.smithi173.stdout:May 27 07:28:42 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-f[72706]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7f6065c821ca] 2024-05-27T07:28:42.361 INFO:journalctl@ceph.mds.f.smithi173.stdout:May 27 07:28:42 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-f[72706]: 16: clone() 2024-05-27T07:28:42.361 INFO:journalctl@ceph.mds.f.smithi173.stdout:May 27 07:28:42 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-f[72706]: 2024-05-27T07:28:42.362 INFO:journalctl@ceph.mds.f.smithi173.stdout:May 27 07:28:42 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-f[72706]: 0> 2024-05-27T07:28:42.167+0000 7f60548ca700 -1 *** Caught signal (Aborted) ** 2024-05-27T07:28:42.362 INFO:journalctl@ceph.mds.f.smithi173.stdout:May 27 07:28:42 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-f[72706]: in thread 7f60548ca700 thread_name:md_log_replay 2024-05-27T07:28:42.362 INFO:journalctl@ceph.mds.f.smithi173.stdout:May 27 07:28:42 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-f[72706]: 2024-05-27T07:28:42.362 INFO:journalctl@ceph.mds.f.smithi173.stdout:May 27 07:28:42 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-f[72706]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:28:42.362 INFO:journalctl@ceph.mds.f.smithi173.stdout:May 27 07:28:42 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-f[72706]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f6065c8cd20] 2024-05-27T07:28:42.362 INFO:journalctl@ceph.mds.f.smithi173.stdout:May 27 07:28:42 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-f[72706]: 2: gsignal() 2024-05-27T07:28:42.362 INFO:journalctl@ceph.mds.f.smithi173.stdout:May 27 07:28:42 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-f[72706]: 3: abort() 2024-05-27T07:28:42.362 INFO:journalctl@ceph.mds.f.smithi173.stdout:May 27 07:28:42 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-f[72706]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x26e) [0x7f606710fced] 2024-05-27T07:28:42.362 INFO:journalctl@ceph.mds.f.smithi173.stdout:May 27 07:28:42 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-f[72706]: 5: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7f606710fdd1] 2024-05-27T07:28:42.362 INFO:journalctl@ceph.mds.f.smithi173.stdout:May 27 07:28:42 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-f[72706]: 6: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x55e1a065423d] 2024-05-27T07:28:42.362 INFO:journalctl@ceph.mds.f.smithi173.stdout:May 27 07:28:42 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-f[72706]: 7: (SnapRealm::build_snap_set() const+0x323) [0x55e1a063f971] 2024-05-27T07:28:42.362 INFO:journalctl@ceph.mds.f.smithi173.stdout:May 27 07:28:42 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-f[72706]: 8: (SnapRealm::check_cache() const+0x13e) [0x55e1a063dfa0] 2024-05-27T07:28:42.362 INFO:journalctl@ceph.mds.f.smithi173.stdout:May 27 07:28:42 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-f[72706]: 9: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x55e1a063e651] 2024-05-27T07:28:42.363 INFO:journalctl@ceph.mds.f.smithi173.stdout:May 27 07:28:42 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-f[72706]: 10: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x55e1a05d3347] 2024-05-27T07:28:42.363 INFO:journalctl@ceph.mds.f.smithi173.stdout:May 27 07:28:42 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-f[72706]: 11: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x55e1a02cf87e] 2024-05-27T07:28:42.363 INFO:journalctl@ceph.mds.f.smithi173.stdout:May 27 07:28:42 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-f[72706]: 12: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x279d) [0x55e1a02d4f2f] 2024-05-27T07:28:42.363 INFO:journalctl@ceph.mds.f.smithi173.stdout:May 27 07:28:42 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-f[72706]: 13: (EUpdate::replay(MDSRank*)+0x63) [0x55e1a02df969] 2024-05-27T07:28:42.363 INFO:journalctl@ceph.mds.f.smithi173.stdout:May 27 07:28:42 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-f[72706]: 14: (MDLog::_replay_thread()+0x1f38) [0x55e1a067848c] 2024-05-27T07:28:42.363 INFO:journalctl@ceph.mds.f.smithi173.stdout:May 27 07:28:42 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-f[72706]: 15: (MDLog::ReplayThread::entry()+0x11) [0x55e1a06819cb] 2024-05-27T07:28:42.363 INFO:journalctl@ceph.mds.f.smithi173.stdout:May 27 07:28:42 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-f[72706]: 16: (Thread::entry_wrapper()+0x43) [0x7f60670de72d] 2024-05-27T07:28:42.363 INFO:journalctl@ceph.mds.f.smithi173.stdout:May 27 07:28:42 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-f[72706]: 17: (Thread::_entry_func(void*)+0xd) [0x7f60670de749] 2024-05-27T07:28:42.363 INFO:journalctl@ceph.mds.f.smithi173.stdout:May 27 07:28:42 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-f[72706]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7f6065c821ca] 2024-05-27T07:28:42.363 INFO:journalctl@ceph.mds.f.smithi173.stdout:May 27 07:28:42 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-f[72706]: 19: clone() 2024-05-27T07:28:42.363 INFO:journalctl@ceph.mds.f.smithi173.stdout:May 27 07:28:42 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-f[72706]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-05-27T07:28:42.363 INFO:journalctl@ceph.mds.f.smithi173.stdout:May 27 07:28:42 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-f[72706]: 2024-05-27T07:28:42.363 INFO:journalctl@ceph.mds.f.smithi173.stdout:May 27 07:28:42 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-f[72706]: -1> 2024-05-27T07:28:42.164+0000 7f60548ca700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: In function 'std::set SnapClient::filter(const std::set&) const' thread 7f60548ca700 time 2024-05-27T07:28:42.162784+0000 2024-05-27T07:28:42.364 INFO:journalctl@ceph.mds.f.smithi173.stdout:May 27 07:28:42 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-f[72706]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: 197: FAILED ceph_assert(cached_version > 0) 2024-05-27T07:28:42.364 INFO:journalctl@ceph.mds.f.smithi173.stdout:May 27 07:28:42 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-f[72706]: 2024-05-27T07:28:42.364 INFO:journalctl@ceph.mds.f.smithi173.stdout:May 27 07:28:42 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-f[72706]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:28:42.364 INFO:journalctl@ceph.mds.f.smithi173.stdout:May 27 07:28:42 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-f[72706]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x127) [0x7f606710fba6] 2024-05-27T07:28:42.364 INFO:journalctl@ceph.mds.f.smithi173.stdout:May 27 07:28:42 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-f[72706]: 2: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7f606710fdd1] 2024-05-27T07:28:42.364 INFO:journalctl@ceph.mds.f.smithi173.stdout:May 27 07:28:42 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-f[72706]: 3: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x55e1a065423d] 2024-05-27T07:28:42.364 INFO:journalctl@ceph.mds.f.smithi173.stdout:May 27 07:28:42 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-f[72706]: 4: (SnapRealm::build_snap_set() const+0x323) [0x55e1a063f971] 2024-05-27T07:28:42.364 INFO:journalctl@ceph.mds.f.smithi173.stdout:May 27 07:28:42 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-f[72706]: 5: (SnapRealm::check_cache() const+0x13e) [0x55e1a063dfa0] 2024-05-27T07:28:42.364 INFO:journalctl@ceph.mds.f.smithi173.stdout:May 27 07:28:42 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-f[72706]: 6: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x55e1a063e651] 2024-05-27T07:28:42.364 INFO:journalctl@ceph.mds.f.smithi173.stdout:May 27 07:28:42 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-f[72706]: 7: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x55e1a05d3347] 2024-05-27T07:28:42.364 INFO:journalctl@ceph.mds.f.smithi173.stdout:May 27 07:28:42 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-f[72706]: 8: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x55e1a02cf87e] 2024-05-27T07:28:42.364 INFO:journalctl@ceph.mds.f.smithi173.stdout:May 27 07:28:42 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-f[72706]: 9: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x279d) [0x55e1a02d4f2f] 2024-05-27T07:28:42.364 INFO:journalctl@ceph.mds.f.smithi173.stdout:May 27 07:28:42 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-f[72706]: 10: (EUpdate::replay(MDSRank*)+0x63) [0x55e1a02df969] 2024-05-27T07:28:42.364 INFO:journalctl@ceph.mds.f.smithi173.stdout:May 27 07:28:42 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-f[72706]: 11: (MDLog::_replay_thread()+0x1f38) [0x55e1a067848c] 2024-05-27T07:28:42.365 INFO:journalctl@ceph.mds.f.smithi173.stdout:May 27 07:28:42 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-f[72706]: 12: (MDLog::ReplayThread::entry()+0x11) [0x55e1a06819cb] 2024-05-27T07:28:42.365 INFO:journalctl@ceph.mds.f.smithi173.stdout:May 27 07:28:42 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-f[72706]: 13: (Thread::entry_wrapper()+0x43) [0x7f60670de72d] 2024-05-27T07:28:42.365 INFO:journalctl@ceph.mds.f.smithi173.stdout:May 27 07:28:42 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-f[72706]: 14: (Thread::_entry_func(void*)+0xd) [0x7f60670de749] 2024-05-27T07:28:42.365 INFO:journalctl@ceph.mds.f.smithi173.stdout:May 27 07:28:42 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-f[72706]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7f6065c821ca] 2024-05-27T07:28:42.365 INFO:journalctl@ceph.mds.f.smithi173.stdout:May 27 07:28:42 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-f[72706]: 16: clone() 2024-05-27T07:28:42.365 INFO:journalctl@ceph.mds.f.smithi173.stdout:May 27 07:28:42 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-f[72706]: 2024-05-27T07:28:42.365 INFO:journalctl@ceph.mds.f.smithi173.stdout:May 27 07:28:42 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-f[72706]: 0> 2024-05-27T07:28:42.167+0000 7f60548ca700 -1 *** Caught signal (Aborted) ** 2024-05-27T07:28:42.365 INFO:journalctl@ceph.mds.f.smithi173.stdout:May 27 07:28:42 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-f[72706]: in thread 7f60548ca700 thread_name:md_log_replay 2024-05-27T07:28:42.365 INFO:journalctl@ceph.mds.f.smithi173.stdout:May 27 07:28:42 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-f[72706]: 2024-05-27T07:28:42.365 INFO:journalctl@ceph.mds.f.smithi173.stdout:May 27 07:28:42 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-f[72706]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:28:42.365 INFO:journalctl@ceph.mds.f.smithi173.stdout:May 27 07:28:42 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-f[72706]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f6065c8cd20] 2024-05-27T07:28:42.365 INFO:journalctl@ceph.mds.f.smithi173.stdout:May 27 07:28:42 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-f[72706]: 2: gsignal() 2024-05-27T07:28:42.365 INFO:journalctl@ceph.mds.f.smithi173.stdout:May 27 07:28:42 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-f[72706]: 3: abort() 2024-05-27T07:28:42.366 INFO:journalctl@ceph.mds.f.smithi173.stdout:May 27 07:28:42 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-f[72706]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x26e) [0x7f606710fced] 2024-05-27T07:28:42.366 INFO:journalctl@ceph.mds.f.smithi173.stdout:May 27 07:28:42 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-f[72706]: 5: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7f606710fdd1] 2024-05-27T07:28:42.366 INFO:journalctl@ceph.mds.f.smithi173.stdout:May 27 07:28:42 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-f[72706]: 6: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x55e1a065423d] 2024-05-27T07:28:42.366 INFO:journalctl@ceph.mds.f.smithi173.stdout:May 27 07:28:42 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-f[72706]: 7: (SnapRealm::build_snap_set() const+0x323) [0x55e1a063f971] 2024-05-27T07:28:42.366 INFO:journalctl@ceph.mds.f.smithi173.stdout:May 27 07:28:42 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-f[72706]: 8: (SnapRealm::check_cache() const+0x13e) [0x55e1a063dfa0] 2024-05-27T07:28:42.366 INFO:journalctl@ceph.mds.f.smithi173.stdout:May 27 07:28:42 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-f[72706]: 9: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x55e1a063e651] 2024-05-27T07:28:42.366 INFO:journalctl@ceph.mds.f.smithi173.stdout:May 27 07:28:42 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-f[72706]: 10: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x55e1a05d3347] 2024-05-27T07:28:42.366 INFO:journalctl@ceph.mds.f.smithi173.stdout:May 27 07:28:42 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-f[72706]: 11: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x55e1a02cf87e] 2024-05-27T07:28:42.366 INFO:journalctl@ceph.mds.f.smithi173.stdout:May 27 07:28:42 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-f[72706]: 12: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x279d) [0x55e1a02d4f2f] 2024-05-27T07:28:42.366 INFO:journalctl@ceph.mds.f.smithi173.stdout:May 27 07:28:42 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-f[72706]: 13: (EUpdate::replay(MDSRank*)+0x63) [0x55e1a02df969] 2024-05-27T07:28:42.366 INFO:journalctl@ceph.mds.f.smithi173.stdout:May 27 07:28:42 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-f[72706]: 14: (MDLog::_replay_thread()+0x1f38) [0x55e1a067848c] 2024-05-27T07:28:42.366 INFO:journalctl@ceph.mds.f.smithi173.stdout:May 27 07:28:42 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-f[72706]: 15: (MDLog::ReplayThread::entry()+0x11) [0x55e1a06819cb] 2024-05-27T07:28:42.366 INFO:journalctl@ceph.mds.f.smithi173.stdout:May 27 07:28:42 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-f[72706]: 16: (Thread::entry_wrapper()+0x43) [0x7f60670de72d] 2024-05-27T07:28:42.367 INFO:journalctl@ceph.mds.f.smithi173.stdout:May 27 07:28:42 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-f[72706]: 17: (Thread::_entry_func(void*)+0xd) [0x7f60670de749] 2024-05-27T07:28:42.367 INFO:journalctl@ceph.mds.f.smithi173.stdout:May 27 07:28:42 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-f[72706]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7f6065c821ca] 2024-05-27T07:28:42.367 INFO:journalctl@ceph.mds.f.smithi173.stdout:May 27 07:28:42 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-f[72706]: 19: clone() 2024-05-27T07:28:42.367 INFO:journalctl@ceph.mds.f.smithi173.stdout:May 27 07:28:42 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-f[72706]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-05-27T07:28:42.367 INFO:journalctl@ceph.mds.f.smithi173.stdout:May 27 07:28:42 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-f[72706]: 2024-05-27T07:28:42.367 INFO:journalctl@ceph.mds.f.smithi173.stdout:May 27 07:28:42 smithi173 podman[72741]: 2024-05-27 07:28:42.231371767 +0000 UTC m=+0.027061776 container died 56317baf1a0980a654662f6738405bfe86020b01b97b00363257847d5a3a4b76 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-f, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, org.label-schema.build-date=20240521, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0) 2024-05-27T07:28:42.439 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:28:42 smithi105 ceph-mon[35977]: pgmap v89: 129 pgs: 129 active+clean; 103 MiB data, 625 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 MiB/s rd, 3.8 MiB/s wr, 1.07k op/s 2024-05-27T07:28:42.439 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:28:42 smithi105 ceph-mon[35977]: daemon mds.f restarted 2024-05-27T07:28:42.986 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-05-27T07:28:42.986 DEBUG:teuthology.orchestra.run.smithi052:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:38f9d900c83c63799fbdbe61acc9a11b0d3554a6 shell --fsid 73df2636-1bf8-11ef-bc9b-c7b262605968 -- ceph tell mds.1:0 scrub start / force,recursive 2024-05-27T07:28:43.095 INFO:journalctl@ceph.mds.f.smithi173.stdout:May 27 07:28:42 smithi173 podman[72741]: 2024-05-27 07:28:42.855027351 +0000 UTC m=+0.650717354 container remove 56317baf1a0980a654662f6738405bfe86020b01b97b00363257847d5a3a4b76 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-f, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, GIT_CLEAN=True, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.build-date=20240521, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-05-27T07:28:43.096 INFO:journalctl@ceph.mds.f.smithi173.stdout:May 27 07:28:42 smithi173 systemd[1]: ceph-73df2636-1bf8-11ef-bc9b-c7b262605968@mds.f.service: Main process exited, code=exited, status=134/n/a 2024-05-27T07:28:43.248 INFO:teuthology.orchestra.run.smithi052.stderr:Inferring config /var/lib/ceph/73df2636-1bf8-11ef-bc9b-c7b262605968/mon.a/config 2024-05-27T07:28:43.351 INFO:journalctl@ceph.mds.f.smithi173.stdout:May 27 07:28:43 smithi173 systemd[1]: ceph-73df2636-1bf8-11ef-bc9b-c7b262605968@mds.f.service: Failed with result 'exit-code'. 2024-05-27T07:28:43.352 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:28:43 smithi173 ceph-mon[37000]: osdmap e96: 12 total, 12 up, 12 in 2024-05-27T07:28:43.352 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:28:43 smithi173 ceph-mon[37000]: mds.? [v2:172.21.15.173:6834/700627779,v1:172.21.15.173:6835/700627779] up:boot 2024-05-27T07:28:43.352 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:28:43 smithi173 ceph-mon[37000]: fsmap cephfs:5 4 up:standby-replay 3 up:standby 5 up:active 2024-05-27T07:28:43.352 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:28:43 smithi173 ceph-mon[37000]: from='mgr.15333 172.21.15.52:0/416389082' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2024-05-27T07:28:43.352 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:28:43 smithi173 ceph-mon[37000]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-05-27T07:28:43.373 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:28:43 smithi052 ceph-mon[31955]: osdmap e96: 12 total, 12 up, 12 in 2024-05-27T07:28:43.373 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:28:43 smithi052 ceph-mon[31955]: mds.? [v2:172.21.15.173:6834/700627779,v1:172.21.15.173:6835/700627779] up:boot 2024-05-27T07:28:43.373 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:28:43 smithi052 ceph-mon[31955]: fsmap cephfs:5 4 up:standby-replay 3 up:standby 5 up:active 2024-05-27T07:28:43.373 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:28:43 smithi052 ceph-mon[31955]: from='mgr.15333 172.21.15.52:0/416389082' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2024-05-27T07:28:43.373 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:28:43 smithi052 ceph-mon[31955]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-05-27T07:28:43.439 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:28:43 smithi105 ceph-mon[35977]: osdmap e96: 12 total, 12 up, 12 in 2024-05-27T07:28:43.440 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:28:43 smithi105 ceph-mon[35977]: mds.? [v2:172.21.15.173:6834/700627779,v1:172.21.15.173:6835/700627779] up:boot 2024-05-27T07:28:43.440 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:28:43 smithi105 ceph-mon[35977]: fsmap cephfs:5 4 up:standby-replay 3 up:standby 5 up:active 2024-05-27T07:28:43.440 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:28:43 smithi105 ceph-mon[35977]: from='mgr.15333 172.21.15.52:0/416389082' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2024-05-27T07:28:43.440 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:28:43 smithi105 ceph-mon[35977]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-05-27T07:28:44.412 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:28:44 smithi052 ceph-mon[31955]: pgmap v91: 129 pgs: 129 active+clean; 116 MiB data, 692 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 5.2 MiB/s wr, 532 op/s 2024-05-27T07:28:44.439 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:28:44 smithi105 ceph-mon[35977]: pgmap v91: 129 pgs: 129 active+clean; 116 MiB data, 692 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 5.2 MiB/s wr, 532 op/s 2024-05-27T07:28:44.601 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:28:44 smithi173 ceph-mon[37000]: pgmap v91: 129 pgs: 129 active+clean; 116 MiB data, 692 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 5.2 MiB/s wr, 532 op/s 2024-05-27T07:28:45.186 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:45.196+0000 7f8d7691f700 1 -- 172.21.15.52:0/4170450827 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f8d701034b0 msgr2=0x7f8d701038c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:28:45.187 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:45.196+0000 7f8d7691f700 1 --2- 172.21.15.52:0/4170450827 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f8d701034b0 0x7f8d701038c0 secure :-1 s=READY pgs=592 cs=0 l=1 rev1=1 crypto rx=0x7f8d600099c0 tx=0x7f8d60009cd0 comp rx=0 tx=0).stop 2024-05-27T07:28:45.187 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:45.197+0000 7f8d7691f700 1 -- 172.21.15.52:0/4170450827 shutdown_connections 2024-05-27T07:28:45.187 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:45.197+0000 7f8d7691f700 1 --2- 172.21.15.52:0/4170450827 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f8d70108730 0x7f8d70110090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:28:45.187 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:45.197+0000 7f8d7691f700 1 --2- 172.21.15.52:0/4170450827 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f8d701034b0 0x7f8d701038c0 unknown :-1 s=CLOSED pgs=592 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:28:45.187 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:45.197+0000 7f8d7691f700 1 --2- 172.21.15.52:0/4170450827 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f8d70107de0 0x7f8d701081f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:28:45.187 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:45.197+0000 7f8d7691f700 1 -- 172.21.15.52:0/4170450827 >> 172.21.15.52:0/4170450827 conn(0x7f8d700fedb0 msgr2=0x7f8d701011d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:28:45.187 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:45.197+0000 7f8d7691f700 1 -- 172.21.15.52:0/4170450827 shutdown_connections 2024-05-27T07:28:45.187 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:45.197+0000 7f8d7691f700 1 -- 172.21.15.52:0/4170450827 wait complete. 2024-05-27T07:28:45.189 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:45.197+0000 7f8d7691f700 1 Processor -- start 2024-05-27T07:28:45.189 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:45.197+0000 7f8d7691f700 1 -- start start 2024-05-27T07:28:45.189 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:45.198+0000 7f8d7691f700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f8d701034b0 0x7f8d7019cfa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:28:45.189 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:45.198+0000 7f8d7691f700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f8d70107de0 0x7f8d7019d540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:28:45.189 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:45.198+0000 7f8d7691f700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f8d70108730 0x7f8d701a17d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:28:45.190 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:45.198+0000 7f8d7691f700 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x7f8d70113360 con 0x7f8d701034b0 2024-05-27T07:28:45.190 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:45.198+0000 7f8d7691f700 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f8d701131e0 con 0x7f8d70107de0 2024-05-27T07:28:45.190 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:45.198+0000 7f8d7691f700 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f8d701134e0 con 0x7f8d70108730 2024-05-27T07:28:45.190 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:45.198+0000 7f8d6ffff700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f8d701034b0 0x7f8d7019cfa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:28:45.190 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:45.198+0000 7f8d6ffff700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f8d701034b0 0x7f8d7019cfa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.52:46062/0 (socket says 172.21.15.52:46062) 2024-05-27T07:28:45.190 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:45.198+0000 7f8d6ffff700 1 -- 172.21.15.52:0/1217755980 learned_addr learned my addr 172.21.15.52:0/1217755980 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-05-27T07:28:45.190 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:45.198+0000 7f8d6f7fe700 1 --2- 172.21.15.52:0/1217755980 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f8d70107de0 0x7f8d7019d540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:28:45.190 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:45.198+0000 7f8d74ebc700 1 --2- 172.21.15.52:0/1217755980 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f8d70108730 0x7f8d701a17d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:28:45.190 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:45.198+0000 7f8d6f7fe700 1 -- 172.21.15.52:0/1217755980 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f8d70108730 msgr2=0x7f8d701a17d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:28:45.190 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:45.198+0000 7f8d6f7fe700 1 --2- 172.21.15.52:0/1217755980 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f8d70108730 0x7f8d701a17d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:28:45.190 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:45.198+0000 7f8d6f7fe700 1 -- 172.21.15.52:0/1217755980 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f8d701034b0 msgr2=0x7f8d7019cfa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:28:45.190 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:45.198+0000 7f8d6f7fe700 1 --2- 172.21.15.52:0/1217755980 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f8d701034b0 0x7f8d7019cfa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:28:45.191 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:45.198+0000 7f8d6f7fe700 1 -- 172.21.15.52:0/1217755980 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8d701a1e70 con 0x7f8d70107de0 2024-05-27T07:28:45.191 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:45.198+0000 7f8d6ffff700 1 --2- 172.21.15.52:0/1217755980 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f8d701034b0 0x7f8d7019cfa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-27T07:28:45.191 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:45.199+0000 7f8d6f7fe700 1 --2- 172.21.15.52:0/1217755980 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f8d70107de0 0x7f8d7019d540 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7f8d6400e9c0 tx=0x7f8d6400ecd0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:28:45.191 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:45.199+0000 7f8d6d7fa700 1 -- 172.21.15.52:0/1217755980 <== mon.2 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8d6400c9c0 con 0x7f8d70107de0 2024-05-27T07:28:45.191 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:45.199+0000 7f8d7691f700 1 -- 172.21.15.52:0/1217755980 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8d701a2160 con 0x7f8d70107de0 2024-05-27T07:28:45.191 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:45.199+0000 7f8d6d7fa700 1 -- 172.21.15.52:0/1217755980 <== mon.2 v2:172.21.15.105:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f8d6400cb60 con 0x7f8d70107de0 2024-05-27T07:28:45.191 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:45.199+0000 7f8d6d7fa700 1 -- 172.21.15.52:0/1217755980 <== mon.2 v2:172.21.15.105:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8d64018680 con 0x7f8d70107de0 2024-05-27T07:28:45.191 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:45.199+0000 7f8d7691f700 1 -- 172.21.15.52:0/1217755980 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8d701a9a80 con 0x7f8d70107de0 2024-05-27T07:28:45.191 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:45.202+0000 7f8d6d7fa700 1 -- 172.21.15.52:0/1217755980 <== mon.2 v2:172.21.15.105:3300/0 4 ==== mgrmap(e 26) ==== 149783+0+0 (secure 0 0 0) 0x7f8d64010430 con 0x7f8d70107de0 2024-05-27T07:28:45.192 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:45.203+0000 7f8d6d7fa700 1 --2- 172.21.15.52:0/1217755980 >> [v2:172.21.15.52:6800/1822318384,v1:172.21.15.52:6801/1822318384] conn(0x7f8d580b29f0 0x7f8d580b4f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:28:45.192 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:45.203+0000 7f8d6ffff700 1 --2- 172.21.15.52:0/1217755980 >> [v2:172.21.15.52:6800/1822318384,v1:172.21.15.52:6801/1822318384] conn(0x7f8d580b29f0 0x7f8d580b4f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:28:45.192 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:45.203+0000 7f8d6d7fa700 1 -- 172.21.15.52:0/1217755980 <== mon.2 v2:172.21.15.105:3300/0 5 ==== osd_map(96..96 src has 1..96) ==== 9379+0+0 (secure 0 0 0) 0x7f8d640e2af0 con 0x7f8d70107de0 2024-05-27T07:28:45.194 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:45.203+0000 7f8d85334b80 1 Processor -- start 2024-05-27T07:28:45.194 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:45.203+0000 7f8d85334b80 1 -- start start 2024-05-27T07:28:45.194 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:45.203+0000 7f8d6ffff700 1 --2- 172.21.15.52:0/1217755980 >> [v2:172.21.15.52:6800/1822318384,v1:172.21.15.52:6801/1822318384] conn(0x7f8d580b29f0 0x7f8d580b4f00 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7f8d60009990 tx=0x7f8d6001a040 comp rx=0 tx=0).ready entity=mgr.15333 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:28:45.194 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:45.204+0000 7f8d85334b80 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x5584d0b20ec0 0x5584d0ca4790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:28:45.194 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:45.204+0000 7f8d85334b80 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5584d0ca4cd0 0x5584d0c9eac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:28:45.194 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:45.204+0000 7f8d85334b80 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5584d0c9f000 0x5584d0c9f410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:28:45.194 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:45.204+0000 7f8d85334b80 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x5584d0bcdd10 con 0x5584d0b20ec0 2024-05-27T07:28:45.194 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:45.204+0000 7f8d85334b80 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x5584d0bbd8b0 con 0x5584d0c9f000 2024-05-27T07:28:45.194 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:45.204+0000 7f8d85334b80 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x5584d0bc18f0 con 0x5584d0ca4cd0 2024-05-27T07:28:45.194 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:45.204+0000 7f8d6ffff700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x5584d0b20ec0 0x5584d0ca4790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:28:45.194 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:45.204+0000 7f8d74ebc700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5584d0c9f000 0x5584d0c9f410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:28:45.195 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:45.204+0000 7f8d6ffff700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x5584d0b20ec0 0x5584d0ca4790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.52:46076/0 (socket says 172.21.15.52:46076) 2024-05-27T07:28:45.195 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:45.204+0000 7f8d6ffff700 1 -- 172.21.15.52:0/4226249696 learned_addr learned my addr 172.21.15.52:0/4226249696 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-05-27T07:28:45.195 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:45.204+0000 7f8d6f7fe700 1 --2- 172.21.15.52:0/4226249696 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5584d0ca4cd0 0x5584d0c9eac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:28:45.195 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:45.204+0000 7f8d74ebc700 1 -- 172.21.15.52:0/4226249696 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5584d0ca4cd0 msgr2=0x5584d0c9eac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:28:45.195 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:45.204+0000 7f8d74ebc700 1 --2- 172.21.15.52:0/4226249696 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5584d0ca4cd0 0x5584d0c9eac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:28:45.195 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:45.204+0000 7f8d74ebc700 1 -- 172.21.15.52:0/4226249696 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x5584d0b20ec0 msgr2=0x5584d0ca4790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:28:45.195 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:45.204+0000 7f8d74ebc700 1 --2- 172.21.15.52:0/4226249696 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x5584d0b20ec0 0x5584d0ca4790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:28:45.195 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:45.204+0000 7f8d74ebc700 1 -- 172.21.15.52:0/4226249696 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5584d0b0d950 con 0x5584d0c9f000 2024-05-27T07:28:45.197 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:45.204+0000 7f8d74ebc700 1 --2- 172.21.15.52:0/4226249696 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5584d0c9f000 0x5584d0c9f410 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7f8d68011580 tx=0x7f8d68009920 comp rx=0 tx=0).ready entity=mon.2 client_cookie=38817f40998d8767 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:28:45.197 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:45.205+0000 7f8d567fc700 1 -- 172.21.15.52:0/4226249696 <== mon.2 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8d68004db0 con 0x5584d0c9f000 2024-05-27T07:28:45.197 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:45.205+0000 7f8d567fc700 1 -- 172.21.15.52:0/4226249696 <== mon.2 v2:172.21.15.105:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f8d68003a00 con 0x5584d0c9f000 2024-05-27T07:28:45.198 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:45.205+0000 7f8d567fc700 1 -- 172.21.15.52:0/4226249696 <== mon.2 v2:172.21.15.105:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8d68004600 con 0x5584d0c9f000 2024-05-27T07:28:45.198 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:45.206+0000 7f8d85334b80 1 -- 172.21.15.52:0/4226249696 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5584d0c9f000 msgr2=0x5584d0c9f410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:28:45.198 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:45.206+0000 7f8d85334b80 1 --2- 172.21.15.52:0/4226249696 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5584d0c9f000 0x5584d0c9f410 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7f8d68011580 tx=0x7f8d68009920 comp rx=0 tx=0).stop 2024-05-27T07:28:45.198 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:45.206+0000 7f8d85334b80 1 -- 172.21.15.52:0/4226249696 shutdown_connections 2024-05-27T07:28:45.198 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:45.206+0000 7f8d85334b80 1 --2- 172.21.15.52:0/4226249696 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5584d0c9f000 0x5584d0c9f410 unknown :-1 s=CLOSED pgs=103 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:28:45.198 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:45.206+0000 7f8d85334b80 1 --2- 172.21.15.52:0/4226249696 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x5584d0b20ec0 0x5584d0ca4790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:28:45.198 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:45.206+0000 7f8d85334b80 1 --2- 172.21.15.52:0/4226249696 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5584d0ca4cd0 0x5584d0c9eac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:28:45.198 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:45.206+0000 7f8d85334b80 1 -- 172.21.15.52:0/4226249696 >> 172.21.15.52:0/4226249696 conn(0x5584d0b7bb70 msgr2=0x5584d0c996c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:28:45.198 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:45.206+0000 7f8d85334b80 1 -- 172.21.15.52:0/4226249696 shutdown_connections 2024-05-27T07:28:45.198 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:45.206+0000 7f8d85334b80 1 -- 172.21.15.52:0/4226249696 wait complete. 2024-05-27T07:28:45.198 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:45.206+0000 7f8d85334b80 1 -- start start 2024-05-27T07:28:45.198 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:45.206+0000 7f8d85334b80 1 Processor -- start 2024-05-27T07:28:45.199 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:45.207+0000 7f8d85334b80 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5584d0b20ec0 0x5584d0ca8360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:28:45.199 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:45.207+0000 7f8d85334b80 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x5584d0c9f000 0x5584d0ca88a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:28:45.199 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:45.207+0000 7f8d85334b80 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5584d0ca4cd0 0x5584d0ca8de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:28:45.199 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:45.207+0000 7f8d85334b80 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x5584d0ac4020 con 0x5584d0c9f000 2024-05-27T07:28:45.199 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:45.207+0000 7f8d85334b80 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x5584d0b5f840 con 0x5584d0ca4cd0 2024-05-27T07:28:45.199 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:45.207+0000 7f8d85334b80 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x5584d0cab7c0 con 0x5584d0b20ec0 2024-05-27T07:28:45.199 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:45.207+0000 7f8d85334b80 10 client.0 _ll_register_callbacks cb 0x5584d0b589a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T07:28:45.199 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:45.207+0000 7f8d6f7fe700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x5584d0c9f000 0x5584d0ca88a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:28:45.199 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:45.207+0000 7f8d6ffff700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5584d0b20ec0 0x5584d0ca8360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:28:45.199 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:45.207+0000 7f8d6f7fe700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x5584d0c9f000 0x5584d0ca88a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.52:46084/0 (socket says 172.21.15.52:46084) 2024-05-27T07:28:45.199 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:45.207+0000 7f8d74ebc700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5584d0ca4cd0 0x5584d0ca8de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:28:45.199 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:45.207+0000 7f8d6f7fe700 1 -- 172.21.15.52:0/4055869471 learned_addr learned my addr 172.21.15.52:0/4055869471 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-05-27T07:28:45.199 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:45.207+0000 7f8d6ffff700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5584d0b20ec0 0x5584d0ca8360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.173:3300/0 says I am v2:172.21.15.52:33704/0 (socket says 172.21.15.52:33704) 2024-05-27T07:28:45.200 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:45.207+0000 7f8d6f7fe700 1 -- 172.21.15.52:0/4055869471 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5584d0b20ec0 msgr2=0x5584d0ca8360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:28:45.200 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:45.207+0000 7f8d6f7fe700 1 --2- 172.21.15.52:0/4055869471 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5584d0b20ec0 0x5584d0ca8360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:28:45.200 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:45.207+0000 7f8d6f7fe700 1 -- 172.21.15.52:0/4055869471 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5584d0ca4cd0 msgr2=0x5584d0ca8de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:28:45.200 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:45.207+0000 7f8d6f7fe700 1 --2- 172.21.15.52:0/4055869471 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5584d0ca4cd0 0x5584d0ca8de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:28:45.200 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:45.207+0000 7f8d6f7fe700 1 -- 172.21.15.52:0/4055869471 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5584d0b5c710 con 0x5584d0c9f000 2024-05-27T07:28:45.200 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:45.207+0000 7f8d74ebc700 1 --2- 172.21.15.52:0/4055869471 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5584d0ca4cd0 0x5584d0ca8de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-27T07:28:45.200 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:45.208+0000 7f8d6f7fe700 1 --2- 172.21.15.52:0/4055869471 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x5584d0c9f000 0x5584d0ca88a0 secure :-1 s=READY pgs=593 cs=0 l=1 rev1=1 crypto rx=0x7f8d640a6ce0 tx=0x7f8d640b2280 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5732059bcd55d322 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:28:45.200 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:45.208+0000 7f8d4f7fe700 10 client.0 ms_handle_connect on v2:172.21.15.52:3300/0 2024-05-27T07:28:45.200 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:45.208+0000 7f8d4f7fe700 1 -- 172.21.15.52:0/4055869471 <== mon.0 v2:172.21.15.52:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8d640aa170 con 0x5584d0c9f000 2024-05-27T07:28:45.200 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:45.208+0000 7f8d4f7fe700 1 -- 172.21.15.52:0/4055869471 <== mon.0 v2:172.21.15.52:3300/0 2 ==== osd_map(96..96 src has 1..96) ==== 9379+0+0 (secure 0 0 0) 0x7f8d640b2650 con 0x5584d0c9f000 2024-05-27T07:28:45.200 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:45.208+0000 7f8d85334b80 1 -- 172.21.15.52:0/4055869471 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f8d68003a00 con 0x5584d0c9f000 2024-05-27T07:28:45.201 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:45.208+0000 7f8d4f7fe700 1 -- 172.21.15.52:0/4055869471 <== mon.0 v2:172.21.15.52:3300/0 3 ==== mon_get_version_reply(handle=1 version=41) ==== 24+0+0 (secure 0 0 0) 0x7f8d640a9a70 con 0x5584d0c9f000 2024-05-27T07:28:45.201 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:45.208+0000 7f8d85334b80 10 client.16143 fetch_fsmap learned FSMap version 41 2024-05-27T07:28:45.202 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:45.208+0000 7f8d85334b80 1 -- 172.21.15.52:0/4055869471 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({config=0+,fsmap=41,monmap=4+}) -- 0x5584d0cb61c0 con 0x5584d0c9f000 2024-05-27T07:28:45.202 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:45.208+0000 7f8d4f7fe700 1 -- 172.21.15.52:0/4055869471 <== mon.0 v2:172.21.15.52:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f8d68003a00 con 0x5584d0c9f000 2024-05-27T07:28:45.202 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:45.208+0000 7f8d4f7fe700 1 -- 172.21.15.52:0/4055869471 <== mon.0 v2:172.21.15.52:3300/0 5 ==== fsmap(e 41) ==== 8378+0+0 (secure 0 0 0) 0x7f8d640a8cd0 con 0x5584d0c9f000 2024-05-27T07:28:45.202 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:45.209+0000 7f8d85334b80 10 client.16143 fetch_fsmap finished waiting for FSMap version 41 2024-05-27T07:28:45.202 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:45.209+0000 7f8d85334b80 10 client.16143 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-05-27T07:28:45.202 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:45.209+0000 7f8d85334b80 20 client.16143 populate_metadata read hostname 'smithi052' 2024-05-27T07:28:45.202 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:45.209+0000 7f8d85334b80 1 --2- 172.21.15.52:0/4055869471 >> [v2:172.21.15.105:6838/500348151,v1:172.21.15.105:6839/500348151] conn(0x5584d0cb8800 0x5584d0cbac20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:28:45.202 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:45.209+0000 7f8d85334b80 4 client.16143 mds_command: new command op to 24527 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-27T07:28:45.202 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:45.209+0000 7f8d85334b80 1 -- 172.21.15.52:0/4055869471 --> [v2:172.21.15.105:6838/500348151,v1:172.21.15.105:6839/500348151] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5584d0c6aa70 con 0x5584d0cb8800 2024-05-27T07:28:45.202 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:45.209+0000 7f8d6ffff700 1 --2- 172.21.15.52:0/4055869471 >> [v2:172.21.15.105:6838/500348151,v1:172.21.15.105:6839/500348151] conn(0x5584d0cb8800 0x5584d0cbac20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:28:45.202 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:45.209+0000 7f8d4f7fe700 10 client.16143 ms_handle_connect on v2:172.21.15.105:6838/500348151 2024-05-27T07:28:45.202 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:45.210+0000 7f8d6ffff700 1 --2- 172.21.15.52:0/4055869471 >> [v2:172.21.15.105:6838/500348151,v1:172.21.15.105:6839/500348151] conn(0x5584d0cb8800 0x5584d0cbac20 crc :-1 s=READY pgs=85 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=2fb2ce88c96082d4 server_cookie=caad4f7fc5ca99bd in_seq=0 out_seq=0 2024-05-27T07:28:45.205 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:45.215+0000 7f8d4f7fe700 1 -- 172.21.15.52:0/4055869471 <== mds.0 v2:172.21.15.105:6838/500348151 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24041 (crc 0 0 0) 0x5584d0c6aa70 con 0x5584d0cb8800 2024-05-27T07:28:45.205 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:45.215+0000 7f8d4f7fe700 10 client.16143 handle_command_reply: tid=0 2024-05-27T07:28:45.205 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:45.215+0000 7f8d85334b80 1 client.16143 shutdown 2024-05-27T07:28:45.205 INFO:tasks.workunit.client.0.smithi052.stdout: 36 205766 378 151224 362 122730 906 2024-05-27T07:28:45.206 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:45.215+0000 7f8d85334b80 1 -- 172.21.15.52:0/4055869471 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x5584d0c9f000 msgr2=0x5584d0ca88a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:28:45.206 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:45.215+0000 7f8d85334b80 1 --2- 172.21.15.52:0/4055869471 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x5584d0c9f000 0x5584d0ca88a0 secure :-1 s=READY pgs=593 cs=0 l=1 rev1=1 crypto rx=0x7f8d640a6ce0 tx=0x7f8d640b2280 comp rx=0 tx=0).stop 2024-05-27T07:28:45.206 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:45.215+0000 7f8d85334b80 1 -- 172.21.15.52:0/4055869471 shutdown_connections 2024-05-27T07:28:45.206 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:45.215+0000 7f8d85334b80 1 --2- 172.21.15.52:0/4055869471 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5584d0ca4cd0 0x5584d0ca8de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:28:45.206 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:45.215+0000 7f8d85334b80 1 --2- 172.21.15.52:0/4055869471 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5584d0b20ec0 0x5584d0ca8360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:28:45.206 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:45.215+0000 7f8d85334b80 1 --2- 172.21.15.52:0/4055869471 >> [v2:172.21.15.105:6838/500348151,v1:172.21.15.105:6839/500348151] conn(0x5584d0cb8800 0x5584d0cbac20 crc :-1 s=READY pgs=85 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:28:45.206 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:45.215+0000 7f8d85334b80 1 --2- 172.21.15.52:0/4055869471 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x5584d0c9f000 0x5584d0ca88a0 unknown :-1 s=CLOSED pgs=593 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:28:45.206 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:45.215+0000 7f8d4f7fe700 0 client.16143 ms_handle_reset on v2:172.21.15.105:6838/500348151 2024-05-27T07:28:45.207 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:45.215+0000 7f8d85334b80 1 -- 172.21.15.52:0/4055869471 >> 172.21.15.52:0/4055869471 conn(0x5584d0b7bb70 msgr2=0x5584d0c9dab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:28:45.207 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:45.216+0000 7f8d85334b80 1 -- 172.21.15.52:0/4055869471 shutdown_connections 2024-05-27T07:28:45.207 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:45.216+0000 7f8d85334b80 1 -- 172.21.15.52:0/4055869471 wait complete. 2024-05-27T07:28:45.207 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:45.216+0000 7f8d85334b80 20 client.16143 trim_cache size 0 max 16384 2024-05-27T07:28:45.228 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:45.237+0000 7f8d85334b80 1 Processor -- start 2024-05-27T07:28:45.228 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:45.237+0000 7f8d85334b80 1 -- start start 2024-05-27T07:28:45.228 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:45.238+0000 7f8d85334b80 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5584d0b20ec0 0x5584d0ca3dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:28:45.228 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:45.238+0000 7f8d85334b80 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5584d0c9f000 0x5584d0ca2950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:28:45.228 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:45.238+0000 7f8d85334b80 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x5584d0ca4cd0 0x5584d0ca8c20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:28:45.228 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:45.238+0000 7f8d85334b80 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x7f8d701b27c0 con 0x5584d0ca4cd0 2024-05-27T07:28:45.228 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:45.238+0000 7f8d85334b80 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f8d701b43c0 con 0x5584d0b20ec0 2024-05-27T07:28:45.228 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:45.238+0000 7f8d85334b80 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f8d701b2e90 con 0x5584d0c9f000 2024-05-27T07:28:45.228 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:45.238+0000 7f8d74ebc700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x5584d0ca4cd0 0x5584d0ca8c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:28:45.228 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:45.238+0000 7f8d6ffff700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5584d0b20ec0 0x5584d0ca3dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:28:45.229 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:45.238+0000 7f8d74ebc700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x5584d0ca4cd0 0x5584d0ca8c20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.52:46100/0 (socket says 172.21.15.52:46100) 2024-05-27T07:28:45.229 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:45.238+0000 7f8d74ebc700 1 -- 172.21.15.52:0/229540856 learned_addr learned my addr 172.21.15.52:0/229540856 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-05-27T07:28:45.229 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:45.238+0000 7f8d6f7fe700 1 --2- 172.21.15.52:0/229540856 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5584d0c9f000 0x5584d0ca2950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:28:45.229 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:45.238+0000 7f8d6ffff700 1 -- 172.21.15.52:0/229540856 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5584d0c9f000 msgr2=0x5584d0ca2950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:28:45.229 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:45.238+0000 7f8d6ffff700 1 --2- 172.21.15.52:0/229540856 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5584d0c9f000 0x5584d0ca2950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:28:45.229 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:45.238+0000 7f8d6ffff700 1 -- 172.21.15.52:0/229540856 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x5584d0ca4cd0 msgr2=0x5584d0ca8c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:28:45.229 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:45.238+0000 7f8d6ffff700 1 --2- 172.21.15.52:0/229540856 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x5584d0ca4cd0 0x5584d0ca8c20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:28:45.229 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:45.238+0000 7f8d6ffff700 1 -- 172.21.15.52:0/229540856 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5584d0c9a840 con 0x5584d0b20ec0 2024-05-27T07:28:45.229 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:45.238+0000 7f8d74ebc700 1 --2- 172.21.15.52:0/229540856 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x5584d0ca4cd0 0x5584d0ca8c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-27T07:28:45.229 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:45.238+0000 7f8d6ffff700 1 --2- 172.21.15.52:0/229540856 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5584d0b20ec0 0x5584d0ca3dd0 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7f8d60003760 tx=0x7f8d6001f000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=46ffa23755e85852 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:28:45.232 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:45.239+0000 7f8d55ffb700 1 -- 172.21.15.52:0/229540856 <== mon.2 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8d60039070 con 0x5584d0b20ec0 2024-05-27T07:28:45.233 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:45.239+0000 7f8d55ffb700 1 -- 172.21.15.52:0/229540856 <== mon.2 v2:172.21.15.105:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f8d600056b0 con 0x5584d0b20ec0 2024-05-27T07:28:45.233 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:45.239+0000 7f8d55ffb700 1 -- 172.21.15.52:0/229540856 <== mon.2 v2:172.21.15.105:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8d6000fd50 con 0x5584d0b20ec0 2024-05-27T07:28:45.233 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:45.239+0000 7f8d85334b80 1 -- 172.21.15.52:0/229540856 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5584d0b20ec0 msgr2=0x5584d0ca3dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:28:45.233 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:45.239+0000 7f8d85334b80 1 --2- 172.21.15.52:0/229540856 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5584d0b20ec0 0x5584d0ca3dd0 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7f8d60003760 tx=0x7f8d6001f000 comp rx=0 tx=0).stop 2024-05-27T07:28:45.233 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:45.240+0000 7f8d85334b80 1 -- 172.21.15.52:0/229540856 shutdown_connections 2024-05-27T07:28:45.233 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:45.240+0000 7f8d85334b80 1 --2- 172.21.15.52:0/229540856 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x5584d0ca4cd0 0x5584d0ca8c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:28:45.233 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:45.240+0000 7f8d85334b80 1 --2- 172.21.15.52:0/229540856 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5584d0b20ec0 0x5584d0ca3dd0 unknown :-1 s=CLOSED pgs=104 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:28:45.233 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:45.240+0000 7f8d85334b80 1 --2- 172.21.15.52:0/229540856 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5584d0c9f000 0x5584d0ca2950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:28:45.233 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:45.240+0000 7f8d85334b80 1 -- 172.21.15.52:0/229540856 >> 172.21.15.52:0/229540856 conn(0x5584d0b7bb70 msgr2=0x5584d0ca9a10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:28:45.233 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:45.240+0000 7f8d85334b80 1 -- 172.21.15.52:0/229540856 shutdown_connections 2024-05-27T07:28:45.233 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:45.240+0000 7f8d85334b80 1 -- 172.21.15.52:0/229540856 wait complete. 2024-05-27T07:28:45.234 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:45.240+0000 7f8d85334b80 1 -- start start 2024-05-27T07:28:45.234 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:45.240+0000 7f8d85334b80 1 Processor -- start 2024-05-27T07:28:45.234 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:45.241+0000 7f8d85334b80 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5584d0b20ec0 0x5584d0cb1120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:28:45.234 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:45.241+0000 7f8d85334b80 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5584d0c9f000 0x5584d0cada20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:28:45.234 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:45.241+0000 7f8d85334b80 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x5584d0ca4cd0 0x5584d0cadf60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:28:45.234 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:45.241+0000 7f8d85334b80 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x7f8d701b3cf0 con 0x5584d0ca4cd0 2024-05-27T07:28:45.234 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:45.241+0000 7f8d85334b80 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f8d701b3560 con 0x5584d0b20ec0 2024-05-27T07:28:45.234 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:45.241+0000 7f8d85334b80 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f8d701b0760 con 0x5584d0c9f000 2024-05-27T07:28:45.234 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:45.241+0000 7f8d85334b80 10 client.0 _ll_register_callbacks cb 0x5584d0b589a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T07:28:45.234 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:45.241+0000 7f8d6ffff700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5584d0b20ec0 0x5584d0cb1120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:28:45.234 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:45.241+0000 7f8d6f7fe700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5584d0c9f000 0x5584d0cada20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:28:45.234 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:45.241+0000 7f8d74ebc700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x5584d0ca4cd0 0x5584d0cadf60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:28:45.235 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:45.241+0000 7f8d6ffff700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5584d0b20ec0 0x5584d0cb1120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.52:48878/0 (socket says 172.21.15.52:48878) 2024-05-27T07:28:45.235 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:45.241+0000 7f8d6ffff700 1 -- 172.21.15.52:0/1405218441 learned_addr learned my addr 172.21.15.52:0/1405218441 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-05-27T07:28:45.235 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:45.241+0000 7f8d6ffff700 1 -- 172.21.15.52:0/1405218441 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5584d0c9f000 msgr2=0x5584d0cada20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:28:45.235 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:45.241+0000 7f8d6ffff700 1 --2- 172.21.15.52:0/1405218441 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5584d0c9f000 0x5584d0cada20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:28:45.235 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:45.241+0000 7f8d6ffff700 1 -- 172.21.15.52:0/1405218441 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x5584d0ca4cd0 msgr2=0x5584d0cadf60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:28:45.235 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:45.241+0000 7f8d6ffff700 1 --2- 172.21.15.52:0/1405218441 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x5584d0ca4cd0 0x5584d0cadf60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:28:45.235 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:45.241+0000 7f8d6ffff700 1 -- 172.21.15.52:0/1405218441 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5584d0cae610 con 0x5584d0b20ec0 2024-05-27T07:28:45.235 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:45.241+0000 7f8d6f7fe700 1 --2- 172.21.15.52:0/1405218441 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5584d0c9f000 0x5584d0cada20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-27T07:28:45.235 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:45.241+0000 7f8d4f7fe700 10 client.0 ms_handle_connect on v2:172.21.15.105:3300/0 2024-05-27T07:28:45.235 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:45.241+0000 7f8d6ffff700 1 --2- 172.21.15.52:0/1405218441 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5584d0b20ec0 0x5584d0cb1120 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7f8d60009600 tx=0x7f8d6000be40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=474d51dead3fd58d server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:28:45.236 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:45.241+0000 7f8d4f7fe700 1 -- 172.21.15.52:0/1405218441 <== mon.2 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8d60042060 con 0x5584d0b20ec0 2024-05-27T07:28:45.236 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:45.242+0000 7f8d4f7fe700 1 -- 172.21.15.52:0/1405218441 <== mon.2 v2:172.21.15.105:3300/0 2 ==== osd_map(96..96 src has 1..96) ==== 9379+0+0 (secure 0 0 0) 0x7f8d6000f6f0 con 0x5584d0b20ec0 2024-05-27T07:28:45.236 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:45.242+0000 7f8d85334b80 1 -- 172.21.15.52:0/1405218441 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f8d600056b0 con 0x5584d0b20ec0 2024-05-27T07:28:45.236 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:45.242+0000 7f8d4f7fe700 1 -- 172.21.15.52:0/1405218441 <== mon.2 v2:172.21.15.105:3300/0 3 ==== mon_get_version_reply(handle=1 version=41) ==== 24+0+0 (secure 0 0 0) 0x7f8d60039030 con 0x5584d0b20ec0 2024-05-27T07:28:45.236 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:45.242+0000 7f8d85334b80 10 client.25805 fetch_fsmap learned FSMap version 41 2024-05-27T07:28:45.236 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:45.242+0000 7f8d85334b80 1 -- 172.21.15.52:0/1405218441 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,fsmap=41,monmap=4+}) -- 0x5584d0d241f0 con 0x5584d0b20ec0 2024-05-27T07:28:45.236 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:45.242+0000 7f8d4f7fe700 1 -- 172.21.15.52:0/1405218441 <== mon.2 v2:172.21.15.105:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f8d600056b0 con 0x5584d0b20ec0 2024-05-27T07:28:45.236 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:45.242+0000 7f8d4f7fe700 1 -- 172.21.15.52:0/1405218441 <== mon.2 v2:172.21.15.105:3300/0 5 ==== fsmap(e 41) ==== 8378+0+0 (secure 0 0 0) 0x7f8d600264c0 con 0x5584d0b20ec0 2024-05-27T07:28:45.236 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:45.242+0000 7f8d85334b80 10 client.25805 fetch_fsmap finished waiting for FSMap version 41 2024-05-27T07:28:45.236 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:45.242+0000 7f8d85334b80 10 client.25805 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-05-27T07:28:45.236 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:45.242+0000 7f8d85334b80 20 client.25805 populate_metadata read hostname 'smithi052' 2024-05-27T07:28:45.236 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:45.242+0000 7f8d85334b80 1 --2- 172.21.15.52:0/1405218441 >> [v2:172.21.15.105:6838/500348151,v1:172.21.15.105:6839/500348151] conn(0x5584d0d247f0 0x5584d0d26c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:28:45.236 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:45.242+0000 7f8d85334b80 4 client.25805 mds_command: new command op to 24527 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-05-27T07:28:45.236 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:45.242+0000 7f8d85334b80 1 -- 172.21.15.52:0/1405218441 --> [v2:172.21.15.105:6838/500348151,v1:172.21.15.105:6839/500348151] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) -- 0x5584d0bcfc60 con 0x5584d0d247f0 2024-05-27T07:28:45.237 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:45.242+0000 7f8d6f7fe700 1 --2- 172.21.15.52:0/1405218441 >> [v2:172.21.15.105:6838/500348151,v1:172.21.15.105:6839/500348151] conn(0x5584d0d247f0 0x5584d0d26c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:28:45.237 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:45.243+0000 7f8d4f7fe700 10 client.25805 ms_handle_connect on v2:172.21.15.105:6838/500348151 2024-05-27T07:28:45.237 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:45.243+0000 7f8d6f7fe700 1 --2- 172.21.15.52:0/1405218441 >> [v2:172.21.15.105:6838/500348151,v1:172.21.15.105:6839/500348151] conn(0x5584d0d247f0 0x5584d0d26c10 crc :-1 s=READY pgs=86 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=a554bf442861cc6f server_cookie=df31be6678d2f28f in_seq=0 out_seq=0 2024-05-27T07:28:45.365 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:45.375+0000 7f8d4f7fe700 1 -- 172.21.15.52:0/1405218441 <== mds.0 v2:172.21.15.105:6838/500348151 1 ==== command_reply(tid 0: 0 ) ==== 8+0+110 (crc 0 0 0) 0x5584d0bcfc60 con 0x5584d0d247f0 2024-05-27T07:28:45.365 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:45.375+0000 7f8d4f7fe700 10 client.25805 handle_command_reply: tid=0 2024-05-27T07:28:45.365 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:45.375+0000 7f8d85334b80 1 client.25805 shutdown 2024-05-27T07:28:45.366 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:45.375+0000 7f8d85334b80 1 -- 172.21.15.52:0/1405218441 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5584d0b20ec0 msgr2=0x5584d0cb1120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:28:45.366 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:45.375+0000 7f8d85334b80 1 --2- 172.21.15.52:0/1405218441 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5584d0b20ec0 0x5584d0cb1120 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7f8d60009600 tx=0x7f8d6000be40 comp rx=0 tx=0).stop 2024-05-27T07:28:45.366 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:45.376+0000 7f8d85334b80 1 -- 172.21.15.52:0/1405218441 shutdown_connections 2024-05-27T07:28:45.366 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:45.376+0000 7f8d85334b80 1 --2- 172.21.15.52:0/1405218441 >> [v2:172.21.15.105:6838/500348151,v1:172.21.15.105:6839/500348151] conn(0x5584d0d247f0 0x5584d0d26c10 crc :-1 s=READY pgs=86 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:28:45.366 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:45.376+0000 7f8d85334b80 1 --2- 172.21.15.52:0/1405218441 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x5584d0ca4cd0 0x5584d0cadf60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:28:45.366 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:45.376+0000 7f8d85334b80 1 --2- 172.21.15.52:0/1405218441 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5584d0b20ec0 0x5584d0cb1120 unknown :-1 s=CLOSED pgs=105 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:28:45.366 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:45.376+0000 7f8d85334b80 1 --2- 172.21.15.52:0/1405218441 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5584d0c9f000 0x5584d0cada20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:28:45.366 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:45.376+0000 7f8d85334b80 1 -- 172.21.15.52:0/1405218441 >> 172.21.15.52:0/1405218441 conn(0x5584d0b7bb70 msgr2=0x5584d0ca3b30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:28:45.366 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:45.376+0000 7f8d4f7fe700 0 client.25805 ms_handle_reset on v2:172.21.15.105:6838/500348151 2024-05-27T07:28:45.366 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:45.376+0000 7f8d85334b80 1 -- 172.21.15.52:0/1405218441 shutdown_connections 2024-05-27T07:28:45.366 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:45.376+0000 7f8d85334b80 1 -- 172.21.15.52:0/1405218441 wait complete. 2024-05-27T07:28:45.366 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:45.376+0000 7f8d85334b80 20 client.25805 trim_cache size 0 max 16384 2024-05-27T07:28:45.367 INFO:teuthology.orchestra.run.smithi052.stdout:{ 2024-05-27T07:28:45.367 INFO:teuthology.orchestra.run.smithi052.stdout: "return_code": 0, 2024-05-27T07:28:45.367 INFO:teuthology.orchestra.run.smithi052.stdout: "scrub_tag": "c250185e-7541-4646-9390-8b76d0627284", 2024-05-27T07:28:45.367 INFO:teuthology.orchestra.run.smithi052.stdout: "mode": "asynchronous" 2024-05-27T07:28:45.367 INFO:teuthology.orchestra.run.smithi052.stdout:} 2024-05-27T07:28:45.368 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:45.377+0000 7f8d7691f700 1 -- 172.21.15.52:0/1217755980 >> [v2:172.21.15.52:6800/1822318384,v1:172.21.15.52:6801/1822318384] conn(0x7f8d580b29f0 msgr2=0x7f8d580b4f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:28:45.368 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:45.377+0000 7f8d7691f700 1 --2- 172.21.15.52:0/1217755980 >> [v2:172.21.15.52:6800/1822318384,v1:172.21.15.52:6801/1822318384] conn(0x7f8d580b29f0 0x7f8d580b4f00 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7f8d60009990 tx=0x7f8d6001a040 comp rx=0 tx=0).stop 2024-05-27T07:28:45.368 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:45.377+0000 7f8d7691f700 1 -- 172.21.15.52:0/1217755980 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f8d70107de0 msgr2=0x7f8d7019d540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:28:45.368 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:45.377+0000 7f8d7691f700 1 --2- 172.21.15.52:0/1217755980 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f8d70107de0 0x7f8d7019d540 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7f8d6400e9c0 tx=0x7f8d6400ecd0 comp rx=0 tx=0).stop 2024-05-27T07:28:45.368 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:45.377+0000 7f8d7691f700 1 -- 172.21.15.52:0/1217755980 shutdown_connections 2024-05-27T07:28:45.368 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:45.377+0000 7f8d7691f700 1 --2- 172.21.15.52:0/1217755980 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f8d70108730 0x7f8d701a17d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:28:45.368 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:45.377+0000 7f8d7691f700 1 --2- 172.21.15.52:0/1217755980 >> [v2:172.21.15.52:6800/1822318384,v1:172.21.15.52:6801/1822318384] conn(0x7f8d580b29f0 0x7f8d580b4f00 unknown :-1 s=CLOSED pgs=96 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:28:45.368 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:45.377+0000 7f8d7691f700 1 --2- 172.21.15.52:0/1217755980 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f8d701034b0 0x7f8d7019cfa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:28:45.368 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:45.377+0000 7f8d7691f700 1 --2- 172.21.15.52:0/1217755980 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f8d70107de0 0x7f8d7019d540 unknown :-1 s=CLOSED pgs=102 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:28:45.368 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:45.377+0000 7f8d7691f700 1 -- 172.21.15.52:0/1217755980 >> 172.21.15.52:0/1217755980 conn(0x7f8d700fedb0 msgr2=0x7f8d70107620 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:28:45.368 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:45.377+0000 7f8d7691f700 1 -- 172.21.15.52:0/1217755980 shutdown_connections 2024-05-27T07:28:45.368 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:45.377+0000 7f8d7691f700 1 -- 172.21.15.52:0/1217755980 wait complete. 2024-05-27T07:28:46.402 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:28:46 smithi173 ceph-mon[37000]: pgmap v92: 129 pgs: 129 active+clean; 139 MiB data, 763 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 MiB/s rd, 7.1 MiB/s wr, 742 op/s 2024-05-27T07:28:46.403 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:28:46 smithi173 ceph-mon[37000]: scrub queued for path: / 2024-05-27T07:28:46.403 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:28:46 smithi173 ceph-mon[37000]: scrub summary: idle+waiting paths [/] 2024-05-27T07:28:46.403 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:28:46 smithi173 ceph-mon[37000]: scrub summary: active paths [/] 2024-05-27T07:28:46.439 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:28:46 smithi105 ceph-mon[35977]: pgmap v92: 129 pgs: 129 active+clean; 139 MiB data, 763 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 MiB/s rd, 7.1 MiB/s wr, 742 op/s 2024-05-27T07:28:46.439 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:28:46 smithi105 ceph-mon[35977]: scrub queued for path: / 2024-05-27T07:28:46.439 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:28:46 smithi105 ceph-mon[35977]: scrub summary: idle+waiting paths [/] 2024-05-27T07:28:46.439 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:28:46 smithi105 ceph-mon[35977]: scrub summary: active paths [/] 2024-05-27T07:28:46.479 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:28:46 smithi052 ceph-mon[31955]: pgmap v92: 129 pgs: 129 active+clean; 139 MiB data, 763 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 MiB/s rd, 7.1 MiB/s wr, 742 op/s 2024-05-27T07:28:46.479 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:28:46 smithi052 ceph-mon[31955]: scrub queued for path: / 2024-05-27T07:28:46.479 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:28:46 smithi052 ceph-mon[31955]: scrub summary: idle+waiting paths [/] 2024-05-27T07:28:46.479 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:28:46 smithi052 ceph-mon[31955]: scrub summary: active paths [/] 2024-05-27T07:28:47.047 DEBUG:teuthology.orchestra.run.smithi052:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:38f9d900c83c63799fbdbe61acc9a11b0d3554a6 shell --fsid 73df2636-1bf8-11ef-bc9b-c7b262605968 -- ceph tell mds.1:0 scrub status 2024-05-27T07:28:47.326 INFO:teuthology.orchestra.run.smithi052.stderr:Inferring config /var/lib/ceph/73df2636-1bf8-11ef-bc9b-c7b262605968/mon.a/config 2024-05-27T07:28:47.420 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:28:47 smithi173 ceph-mon[37000]: scrub summary: active paths [/] 2024-05-27T07:28:47.420 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:28:47 smithi173 ceph-mon[37000]: scrub summary: idle+waiting paths [/] 2024-05-27T07:28:47.421 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:28:47 smithi173 ceph-mon[37000]: scrub summary: active paths [/] 2024-05-27T07:28:47.421 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:28:47 smithi173 ceph-mon[37000]: scrub summary: idle+waiting paths [/] 2024-05-27T07:28:47.421 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:28:47 smithi173 ceph-mon[37000]: scrub summary: active paths [/] 2024-05-27T07:28:47.421 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:28:47 smithi173 ceph-mon[37000]: scrub summary: active paths [/] 2024-05-27T07:28:47.421 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:28:47 smithi173 ceph-mon[37000]: scrub summary: idle+waiting paths [/] 2024-05-27T07:28:47.421 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:28:47 smithi173 ceph-mon[37000]: scrub summary: active paths [/] 2024-05-27T07:28:47.421 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:28:47 smithi173 ceph-mon[37000]: scrub summary: idle+waiting paths [/] 2024-05-27T07:28:47.439 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:28:47 smithi105 ceph-mon[35977]: scrub summary: active paths [/] 2024-05-27T07:28:47.439 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:28:47 smithi105 ceph-mon[35977]: scrub summary: idle+waiting paths [/] 2024-05-27T07:28:47.439 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:28:47 smithi105 ceph-mon[35977]: scrub summary: active paths [/] 2024-05-27T07:28:47.439 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:28:47 smithi105 ceph-mon[35977]: scrub summary: idle+waiting paths [/] 2024-05-27T07:28:47.440 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:28:47 smithi105 ceph-mon[35977]: scrub summary: active paths [/] 2024-05-27T07:28:47.440 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:28:47 smithi105 ceph-mon[35977]: scrub summary: active paths [/] 2024-05-27T07:28:47.440 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:28:47 smithi105 ceph-mon[35977]: scrub summary: idle+waiting paths [/] 2024-05-27T07:28:47.440 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:28:47 smithi105 ceph-mon[35977]: scrub summary: active paths [/] 2024-05-27T07:28:47.440 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:28:47 smithi105 ceph-mon[35977]: scrub summary: idle+waiting paths [/] 2024-05-27T07:28:47.450 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:28:47 smithi052 ceph-mon[31955]: scrub summary: active paths [/] 2024-05-27T07:28:47.450 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:28:47 smithi052 ceph-mon[31955]: scrub summary: idle+waiting paths [/] 2024-05-27T07:28:47.450 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:28:47 smithi052 ceph-mon[31955]: scrub summary: active paths [/] 2024-05-27T07:28:47.450 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:28:47 smithi052 ceph-mon[31955]: scrub summary: idle+waiting paths [/] 2024-05-27T07:28:47.450 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:28:47 smithi052 ceph-mon[31955]: scrub summary: active paths [/] 2024-05-27T07:28:47.450 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:28:47 smithi052 ceph-mon[31955]: scrub summary: active paths [/] 2024-05-27T07:28:47.450 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:28:47 smithi052 ceph-mon[31955]: scrub summary: idle+waiting paths [/] 2024-05-27T07:28:47.451 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:28:47 smithi052 ceph-mon[31955]: scrub summary: active paths [/] 2024-05-27T07:28:47.451 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:28:47 smithi052 ceph-mon[31955]: scrub summary: idle+waiting paths [/] 2024-05-27T07:28:48.119 INFO:journalctl@ceph.mds.g.smithi052.stdout:May 27 07:28:47 smithi052 systemd[1]: ceph-73df2636-1bf8-11ef-bc9b-c7b262605968@mds.g.service: Scheduled restart job, restart counter is at 3. 2024-05-27T07:28:48.119 INFO:journalctl@ceph.mds.g.smithi052.stdout:May 27 07:28:47 smithi052 systemd[1]: Stopped Ceph mds.g for 73df2636-1bf8-11ef-bc9b-c7b262605968. 2024-05-27T07:28:48.119 INFO:journalctl@ceph.mds.g.smithi052.stdout:May 27 07:28:47 smithi052 systemd[1]: Starting Ceph mds.g for 73df2636-1bf8-11ef-bc9b-c7b262605968... 2024-05-27T07:28:48.439 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:28:48 smithi105 ceph-mon[35977]: scrub summary: active paths [/] 2024-05-27T07:28:48.439 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:28:48 smithi105 ceph-mon[35977]: scrub summary: idle+waiting paths [/] 2024-05-27T07:28:48.439 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:28:48 smithi105 ceph-mon[35977]: scrub summary: idle+waiting paths [/] 2024-05-27T07:28:48.440 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:28:48 smithi105 ceph-mon[35977]: pgmap v93: 129 pgs: 129 active+clean; 139 MiB data, 763 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 MiB/s rd, 7.1 MiB/s wr, 742 op/s 2024-05-27T07:28:48.479 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:28:48 smithi052 ceph-mon[31955]: scrub summary: active paths [/] 2024-05-27T07:28:48.479 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:28:48 smithi052 ceph-mon[31955]: scrub summary: idle+waiting paths [/] 2024-05-27T07:28:48.479 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:28:48 smithi052 ceph-mon[31955]: scrub summary: idle+waiting paths [/] 2024-05-27T07:28:48.479 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:28:48 smithi052 ceph-mon[31955]: pgmap v93: 129 pgs: 129 active+clean; 139 MiB data, 763 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 MiB/s rd, 7.1 MiB/s wr, 742 op/s 2024-05-27T07:28:48.601 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:28:48 smithi173 ceph-mon[37000]: scrub summary: active paths [/] 2024-05-27T07:28:48.601 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:28:48 smithi173 ceph-mon[37000]: scrub summary: idle+waiting paths [/] 2024-05-27T07:28:48.601 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:28:48 smithi173 ceph-mon[37000]: scrub summary: idle+waiting paths [/] 2024-05-27T07:28:48.602 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:28:48 smithi173 ceph-mon[37000]: pgmap v93: 129 pgs: 129 active+clean; 139 MiB data, 763 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 MiB/s rd, 7.1 MiB/s wr, 742 op/s 2024-05-27T07:28:49.113 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:28:49.123+0000 7fb747d40640 1 -- 172.21.15.52:0/2808706450 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fb740104410 msgr2=0x7fb7401107e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:28:49.113 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:28:49.123+0000 7fb747d40640 1 --2- 172.21.15.52:0/2808706450 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fb740104410 0x7fb7401107e0 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7fb738009ca0 tx=0x7fb73802a8c0 comp rx=0 tx=0).stop 2024-05-27T07:28:49.113 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:28:49.123+0000 7fb747d40640 1 -- 172.21.15.52:0/2808706450 shutdown_connections 2024-05-27T07:28:49.113 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:28:49.123+0000 7fb747d40640 1 --2- 172.21.15.52:0/2808706450 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fb740104410 0x7fb7401107e0 unknown :-1 s=CLOSED pgs=106 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:28:49.113 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:28:49.123+0000 7fb747d40640 1 --2- 172.21.15.52:0/2808706450 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fb74010a160 0x7fb740103da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:28:49.113 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:28:49.123+0000 7fb747d40640 1 --2- 172.21.15.52:0/2808706450 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fb740109cc0 0x7fb740103860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:28:49.114 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:28:49.123+0000 7fb747d40640 1 -- 172.21.15.52:0/2808706450 >> 172.21.15.52:0/2808706450 conn(0x7fb7400793c0 msgr2=0x7fb740079800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:28:49.114 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:28:49.123+0000 7fb747d40640 1 -- 172.21.15.52:0/2808706450 shutdown_connections 2024-05-27T07:28:49.114 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:28:49.123+0000 7fb747d40640 1 -- 172.21.15.52:0/2808706450 wait complete. 2024-05-27T07:28:49.115 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:28:49.124+0000 7fb747d40640 1 Processor -- start 2024-05-27T07:28:49.116 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:28:49.124+0000 7fb747d40640 1 -- start start 2024-05-27T07:28:49.116 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:28:49.124+0000 7fb747d40640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fb740104410 0x7fb7401a05b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:28:49.116 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:28:49.124+0000 7fb747d40640 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fb740109cc0 0x7fb7401a0af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:28:49.116 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:28:49.124+0000 7fb747d40640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fb74010a160 0x7fb74019a730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:28:49.116 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:28:49.124+0000 7fb747d40640 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x7fb740113ed0 con 0x7fb740109cc0 2024-05-27T07:28:49.116 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:28:49.124+0000 7fb747d40640 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7fb740113d50 con 0x7fb74010a160 2024-05-27T07:28:49.116 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:28:49.124+0000 7fb747d40640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7fb740114050 con 0x7fb740104410 2024-05-27T07:28:49.116 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:28:49.124+0000 7fb7452db640 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fb740109cc0 0x7fb7401a0af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:28:49.116 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:28:49.124+0000 7fb7452db640 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fb740109cc0 0x7fb7401a0af0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.52:46122/0 (socket says 172.21.15.52:46122) 2024-05-27T07:28:49.116 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:28:49.124+0000 7fb7452db640 1 -- 172.21.15.52:0/1871827443 learned_addr learned my addr 172.21.15.52:0/1871827443 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-05-27T07:28:49.116 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:28:49.124+0000 7fb7462dd640 1 --2- 172.21.15.52:0/1871827443 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fb74010a160 0x7fb74019a730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:28:49.116 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:28:49.125+0000 7fb745adc640 1 --2- 172.21.15.52:0/1871827443 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fb740104410 0x7fb7401a05b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:28:49.117 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:28:49.125+0000 7fb7452db640 1 -- 172.21.15.52:0/1871827443 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fb740104410 msgr2=0x7fb7401a05b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:28:49.117 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:28:49.125+0000 7fb7452db640 1 --2- 172.21.15.52:0/1871827443 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fb740104410 0x7fb7401a05b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:28:49.117 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:28:49.125+0000 7fb7452db640 1 -- 172.21.15.52:0/1871827443 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fb74010a160 msgr2=0x7fb74019a730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:28:49.117 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:28:49.125+0000 7fb7452db640 1 --2- 172.21.15.52:0/1871827443 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fb74010a160 0x7fb74019a730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:28:49.117 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:28:49.125+0000 7fb7452db640 1 -- 172.21.15.52:0/1871827443 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb74019ae60 con 0x7fb740109cc0 2024-05-27T07:28:49.117 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:28:49.125+0000 7fb7452db640 1 --2- 172.21.15.52:0/1871827443 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fb740109cc0 0x7fb7401a0af0 secure :-1 s=READY pgs=594 cs=0 l=1 rev1=1 crypto rx=0x7fb73000e9f0 tx=0x7fb73000eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:28:49.117 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:28:49.125+0000 7fb736ffd640 1 -- 172.21.15.52:0/1871827443 <== mon.0 v2:172.21.15.52:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb73000ccc0 con 0x7fb740109cc0 2024-05-27T07:28:49.117 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:28:49.125+0000 7fb736ffd640 1 -- 172.21.15.52:0/1871827443 <== mon.0 v2:172.21.15.52:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fb7300035f0 con 0x7fb740109cc0 2024-05-27T07:28:49.117 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:28:49.125+0000 7fb736ffd640 1 -- 172.21.15.52:0/1871827443 <== mon.0 v2:172.21.15.52:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb73000ce60 con 0x7fb740109cc0 2024-05-27T07:28:49.117 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:28:49.125+0000 7fb747d40640 1 -- 172.21.15.52:0/1871827443 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb74019b0f0 con 0x7fb740109cc0 2024-05-27T07:28:49.117 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:28:49.125+0000 7fb747d40640 1 -- 172.21.15.52:0/1871827443 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb7401a7350 con 0x7fb740109cc0 2024-05-27T07:28:49.118 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:28:49.129+0000 7fb736ffd640 1 -- 172.21.15.52:0/1871827443 <== mon.0 v2:172.21.15.52:3300/0 4 ==== mgrmap(e 26) ==== 149783+0+0 (secure 0 0 0) 0x7fb730003aa0 con 0x7fb740109cc0 2024-05-27T07:28:49.119 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:28:49.129+0000 7fb74ebf0740 1 Processor -- start 2024-05-27T07:28:49.119 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:28:49.130+0000 7fb74ebf0740 1 -- start start 2024-05-27T07:28:49.121 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:28:49.130+0000 7fb74ebf0740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55bcd9cc81b0 0x55bcda0693d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:28:49.121 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:28:49.130+0000 7fb74ebf0740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55bcd9c9f2f0 0x55bcda069910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:28:49.121 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:28:49.130+0000 7fb74ebf0740 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55bcd9ca74d0 0x55bcda069e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:28:49.121 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:28:49.130+0000 7fb74ebf0740 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x55bcd9fdbc30 con 0x55bcd9ca74d0 2024-05-27T07:28:49.121 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:28:49.130+0000 7fb74ebf0740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x55bcd9fcb7d0 con 0x55bcd9c9f2f0 2024-05-27T07:28:49.121 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:28:49.130+0000 7fb74ebf0740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x55bcd9fcf810 con 0x55bcd9cc81b0 2024-05-27T07:28:49.121 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:28:49.130+0000 7fb745adc640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55bcd9c9f2f0 0x55bcda069910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:28:49.122 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:28:49.130+0000 7fb7462dd640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55bcd9cc81b0 0x55bcda0693d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:28:49.122 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:28:49.130+0000 7fb745adc640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55bcd9c9f2f0 0x55bcda069910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.52:48906/0 (socket says 172.21.15.52:48906) 2024-05-27T07:28:49.122 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:28:49.130+0000 7fb745adc640 1 -- 172.21.15.52:0/2822944055 learned_addr learned my addr 172.21.15.52:0/2822944055 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-05-27T07:28:49.122 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:28:49.130+0000 7fb7452db640 1 --2- 172.21.15.52:0/2822944055 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55bcd9ca74d0 0x55bcda069e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:28:49.122 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:28:49.131+0000 7fb7462dd640 1 -- 172.21.15.52:0/2822944055 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55bcd9c9f2f0 msgr2=0x55bcda069910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:28:49.122 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:28:49.131+0000 7fb7462dd640 1 --2- 172.21.15.52:0/2822944055 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55bcd9c9f2f0 0x55bcda069910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:28:49.122 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:28:49.131+0000 7fb7462dd640 1 -- 172.21.15.52:0/2822944055 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55bcd9ca74d0 msgr2=0x55bcda069e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:28:49.122 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:28:49.131+0000 7fb7462dd640 1 --2- 172.21.15.52:0/2822944055 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55bcd9ca74d0 0x55bcda069e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:28:49.122 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:28:49.131+0000 7fb7462dd640 1 -- 172.21.15.52:0/2822944055 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55bcd9ce0210 con 0x55bcd9cc81b0 2024-05-27T07:28:49.122 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:28:49.131+0000 7fb7452db640 1 --2- 172.21.15.52:0/2822944055 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55bcd9ca74d0 0x55bcda069e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-27T07:28:49.122 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:28:49.131+0000 7fb736ffd640 1 --2- 172.21.15.52:0/1871827443 >> [v2:172.21.15.52:6800/1822318384,v1:172.21.15.52:6801/1822318384] conn(0x7fb72c0b2a50 0x7fb72c0b4f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:28:49.122 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:28:49.131+0000 7fb745adc640 1 --2- 172.21.15.52:0/1871827443 >> [v2:172.21.15.52:6800/1822318384,v1:172.21.15.52:6801/1822318384] conn(0x7fb72c0b2a50 0x7fb72c0b4f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:28:49.123 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:28:49.131+0000 7fb7462dd640 1 --2- 172.21.15.52:0/2822944055 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55bcd9cc81b0 0x55bcda0693d0 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7fb738005e30 tx=0x7fb73802f410 comp rx=0 tx=0).ready entity=mon.1 client_cookie=75179debeea52a92 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:28:49.123 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:28:49.131+0000 7fb736ffd640 1 -- 172.21.15.52:0/1871827443 <== mon.0 v2:172.21.15.52:3300/0 5 ==== osd_map(96..96 src has 1..96) ==== 9379+0+0 (secure 0 0 0) 0x7fb7300e2c90 con 0x7fb740109cc0 2024-05-27T07:28:49.123 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:28:49.132+0000 7fb745adc640 1 --2- 172.21.15.52:0/1871827443 >> [v2:172.21.15.52:6800/1822318384,v1:172.21.15.52:6801/1822318384] conn(0x7fb72c0b2a50 0x7fb72c0b4f60 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x55bcda06a780 tx=0x7fb73c005f90 comp rx=0 tx=0).ready entity=mgr.15333 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:28:49.124 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:28:49.133+0000 7fb72bfff640 1 -- 172.21.15.52:0/2822944055 <== mon.1 v2:172.21.15.173:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb738031cf0 con 0x55bcd9cc81b0 2024-05-27T07:28:49.124 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:28:49.133+0000 7fb72bfff640 1 -- 172.21.15.52:0/2822944055 <== mon.1 v2:172.21.15.173:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fb738036030 con 0x55bcd9cc81b0 2024-05-27T07:28:49.124 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:28:49.133+0000 7fb72bfff640 1 -- 172.21.15.52:0/2822944055 <== mon.1 v2:172.21.15.173:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb738004040 con 0x55bcd9cc81b0 2024-05-27T07:28:49.124 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:28:49.134+0000 7fb74ebf0740 1 -- 172.21.15.52:0/2822944055 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55bcd9cc81b0 msgr2=0x55bcda0693d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:28:49.124 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:28:49.134+0000 7fb74ebf0740 1 --2- 172.21.15.52:0/2822944055 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55bcd9cc81b0 0x55bcda0693d0 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7fb738005e30 tx=0x7fb73802f410 comp rx=0 tx=0).stop 2024-05-27T07:28:49.124 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:28:49.134+0000 7fb74ebf0740 1 -- 172.21.15.52:0/2822944055 shutdown_connections 2024-05-27T07:28:49.124 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:28:49.134+0000 7fb74ebf0740 1 --2- 172.21.15.52:0/2822944055 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55bcd9ca74d0 0x55bcda069e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:28:49.124 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:28:49.134+0000 7fb74ebf0740 1 --2- 172.21.15.52:0/2822944055 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55bcd9c9f2f0 0x55bcda069910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:28:49.124 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:28:49.134+0000 7fb74ebf0740 1 --2- 172.21.15.52:0/2822944055 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55bcd9cc81b0 0x55bcda0693d0 unknown :-1 s=CLOSED pgs=101 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:28:49.124 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:28:49.134+0000 7fb74ebf0740 1 -- 172.21.15.52:0/2822944055 >> 172.21.15.52:0/2822944055 conn(0x55bcd9cbc300 msgr2=0x55bcda065b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:28:49.125 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:28:49.134+0000 7fb74ebf0740 1 -- 172.21.15.52:0/2822944055 shutdown_connections 2024-05-27T07:28:49.125 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:28:49.134+0000 7fb74ebf0740 1 -- 172.21.15.52:0/2822944055 wait complete. 2024-05-27T07:28:49.125 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:28:49.135+0000 7fb74ebf0740 1 -- start start 2024-05-27T07:28:49.128 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:28:49.135+0000 7fb74ebf0740 1 Processor -- start 2024-05-27T07:28:49.128 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:28:49.135+0000 7fb74ebf0740 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55bcd9c9f2f0 0x55bcda0754f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:28:49.128 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:28:49.135+0000 7fb74ebf0740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55bcd9ca74d0 0x55bcda075a30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:28:49.128 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:28:49.135+0000 7fb74ebf0740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55bcd9cc81b0 0x55bcda073a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:28:49.128 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:28:49.135+0000 7fb74ebf0740 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x55bcd9cb1790 con 0x55bcd9c9f2f0 2024-05-27T07:28:49.128 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:28:49.135+0000 7fb74ebf0740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x55bcd9f87890 con 0x55bcd9ca74d0 2024-05-27T07:28:49.128 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:28:49.135+0000 7fb74ebf0740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x55bcda076320 con 0x55bcd9cc81b0 2024-05-27T07:28:49.128 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:28:49.135+0000 7fb74ebf0740 10 client.0 _ll_register_callbacks cb 0x55bcd9cdfa00 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T07:28:49.128 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:28:49.135+0000 7fb745adc640 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55bcd9c9f2f0 0x55bcda0754f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:28:49.128 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:28:49.135+0000 7fb7452db640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55bcd9ca74d0 0x55bcda075a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:28:49.128 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:28:49.135+0000 7fb745adc640 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55bcd9c9f2f0 0x55bcda0754f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.52:46148/0 (socket says 172.21.15.52:46148) 2024-05-27T07:28:49.128 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:28:49.135+0000 7fb7452db640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55bcd9ca74d0 0x55bcda075a30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.52:48918/0 (socket says 172.21.15.52:48918) 2024-05-27T07:28:49.129 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:28:49.135+0000 7fb745adc640 1 -- 172.21.15.52:0/1467153682 learned_addr learned my addr 172.21.15.52:0/1467153682 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-05-27T07:28:49.129 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:28:49.135+0000 7fb7462dd640 1 --2- 172.21.15.52:0/1467153682 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55bcd9cc81b0 0x55bcda073a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:28:49.129 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:28:49.136+0000 7fb7452db640 1 -- 172.21.15.52:0/1467153682 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55bcd9cc81b0 msgr2=0x55bcda073a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:28:49.129 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:28:49.136+0000 7fb7452db640 1 --2- 172.21.15.52:0/1467153682 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55bcd9cc81b0 0x55bcda073a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:28:49.129 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:28:49.136+0000 7fb7452db640 1 -- 172.21.15.52:0/1467153682 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55bcd9c9f2f0 msgr2=0x55bcda0754f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:28:49.129 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:28:49.136+0000 7fb7452db640 1 --2- 172.21.15.52:0/1467153682 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55bcd9c9f2f0 0x55bcda0754f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:28:49.129 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:28:49.136+0000 7fb7452db640 1 -- 172.21.15.52:0/1467153682 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55bcd9dac620 con 0x55bcd9ca74d0 2024-05-27T07:28:49.129 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:28:49.136+0000 7fb7452db640 1 --2- 172.21.15.52:0/1467153682 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55bcd9ca74d0 0x55bcda075a30 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7fb7300e0020 tx=0x7fb7300a9ab0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=46bba7ed792e624e server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:28:49.129 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:28:49.136+0000 7fb7297fa640 10 client.0 ms_handle_connect on v2:172.21.15.105:3300/0 2024-05-27T07:28:49.130 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:28:49.136+0000 7fb7297fa640 1 -- 172.21.15.52:0/1467153682 <== mon.2 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb7300cae60 con 0x55bcd9ca74d0 2024-05-27T07:28:49.130 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:28:49.136+0000 7fb7297fa640 1 -- 172.21.15.52:0/1467153682 <== mon.2 v2:172.21.15.105:3300/0 2 ==== osd_map(96..96 src has 1..96) ==== 9379+0+0 (secure 0 0 0) 0x7fb7300b2580 con 0x55bcd9ca74d0 2024-05-27T07:28:49.130 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:28:49.136+0000 7fb74ebf0740 1 -- 172.21.15.52:0/1467153682 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fb738036030 con 0x55bcd9ca74d0 2024-05-27T07:28:49.130 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:28:49.136+0000 7fb7297fa640 1 -- 172.21.15.52:0/1467153682 <== mon.2 v2:172.21.15.105:3300/0 3 ==== mon_get_version_reply(handle=1 version=41) ==== 24+0+0 (secure 0 0 0) 0x7fb7300a96c0 con 0x55bcd9ca74d0 2024-05-27T07:28:49.130 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:28:49.136+0000 7fb74ebf0740 10 client.25817 fetch_fsmap learned FSMap version 41 2024-05-27T07:28:49.130 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:28:49.136+0000 7fb74ebf0740 1 -- 172.21.15.52:0/1467153682 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,fsmap=41,monmap=4+}) -- 0x55bcda082780 con 0x55bcd9ca74d0 2024-05-27T07:28:49.130 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:28:49.137+0000 7fb7297fa640 1 -- 172.21.15.52:0/1467153682 <== mon.2 v2:172.21.15.105:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fb738036030 con 0x55bcd9ca74d0 2024-05-27T07:28:49.130 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:28:49.137+0000 7fb7297fa640 1 -- 172.21.15.52:0/1467153682 <== mon.2 v2:172.21.15.105:3300/0 5 ==== fsmap(e 41) ==== 8378+0+0 (secure 0 0 0) 0x7fb7300c4080 con 0x55bcd9ca74d0 2024-05-27T07:28:49.130 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:28:49.137+0000 7fb74ebf0740 10 client.25817 fetch_fsmap finished waiting for FSMap version 41 2024-05-27T07:28:49.130 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:28:49.137+0000 7fb74ebf0740 10 client.25817 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-05-27T07:28:49.130 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:28:49.137+0000 7fb74ebf0740 20 client.25817 populate_metadata read hostname 'smithi052' 2024-05-27T07:28:49.131 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:28:49.137+0000 7fb74ebf0740 1 --2- 172.21.15.52:0/1467153682 >> [v2:172.21.15.105:6838/500348151,v1:172.21.15.105:6839/500348151] conn(0x55bcd9c86d80 0x55bcda085c60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:28:49.131 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:28:49.137+0000 7fb74ebf0740 4 client.25817 mds_command: new command op to 24527 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-27T07:28:49.131 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:28:49.137+0000 7fb74ebf0740 1 -- 172.21.15.52:0/1467153682 --> [v2:172.21.15.105:6838/500348151,v1:172.21.15.105:6839/500348151] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55bcd9fddb80 con 0x55bcd9c86d80 2024-05-27T07:28:49.131 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:28:49.137+0000 7fb745adc640 1 --2- 172.21.15.52:0/1467153682 >> [v2:172.21.15.105:6838/500348151,v1:172.21.15.105:6839/500348151] conn(0x55bcd9c86d80 0x55bcda085c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:28:49.131 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:28:49.138+0000 7fb7297fa640 10 client.25817 ms_handle_connect on v2:172.21.15.105:6838/500348151 2024-05-27T07:28:49.131 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:28:49.138+0000 7fb745adc640 1 --2- 172.21.15.52:0/1467153682 >> [v2:172.21.15.105:6838/500348151,v1:172.21.15.105:6839/500348151] conn(0x55bcd9c86d80 0x55bcda085c60 crc :-1 s=READY pgs=87 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=d1393b5a43eddec3 server_cookie=b498b86f5a89252b in_seq=0 out_seq=0 2024-05-27T07:28:49.134 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:28:49.144+0000 7fb7297fa640 1 -- 172.21.15.52:0/1467153682 <== mds.0 v2:172.21.15.105:6838/500348151 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24041 (crc 0 0 0) 0x55bcd9fddb80 con 0x55bcd9c86d80 2024-05-27T07:28:49.134 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:28:49.144+0000 7fb7297fa640 10 client.25817 handle_command_reply: tid=0 2024-05-27T07:28:49.134 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:28:49.144+0000 7fb74ebf0740 1 client.25817 shutdown 2024-05-27T07:28:49.134 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:28:49.144+0000 7fb74ebf0740 1 -- 172.21.15.52:0/1467153682 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55bcd9ca74d0 msgr2=0x55bcda075a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:28:49.134 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:28:49.144+0000 7fb74ebf0740 1 --2- 172.21.15.52:0/1467153682 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55bcd9ca74d0 0x55bcda075a30 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7fb7300e0020 tx=0x7fb7300a9ab0 comp rx=0 tx=0).stop 2024-05-27T07:28:49.135 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:28:49.145+0000 7fb74ebf0740 1 -- 172.21.15.52:0/1467153682 shutdown_connections 2024-05-27T07:28:49.135 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:28:49.145+0000 7fb74ebf0740 1 --2- 172.21.15.52:0/1467153682 >> [v2:172.21.15.105:6838/500348151,v1:172.21.15.105:6839/500348151] conn(0x55bcd9c86d80 0x55bcda085c60 crc :-1 s=READY pgs=87 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:28:49.135 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:28:49.145+0000 7fb74ebf0740 1 --2- 172.21.15.52:0/1467153682 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55bcd9cc81b0 0x55bcda073a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:28:49.135 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:28:49.145+0000 7fb74ebf0740 1 --2- 172.21.15.52:0/1467153682 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55bcd9ca74d0 0x55bcda075a30 unknown :-1 s=CLOSED pgs=107 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:28:49.135 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:28:49.145+0000 7fb74ebf0740 1 --2- 172.21.15.52:0/1467153682 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55bcd9c9f2f0 0x55bcda0754f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:28:49.135 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:28:49.145+0000 7fb74ebf0740 1 -- 172.21.15.52:0/1467153682 >> 172.21.15.52:0/1467153682 conn(0x55bcd9cbc300 msgr2=0x55bcda06b0f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:28:49.135 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:28:49.145+0000 7fb7297fa640 0 client.25817 ms_handle_reset on v2:172.21.15.105:6838/500348151 2024-05-27T07:28:49.135 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:28:49.145+0000 7fb74ebf0740 1 -- 172.21.15.52:0/1467153682 shutdown_connections 2024-05-27T07:28:49.135 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:28:49.145+0000 7fb74ebf0740 1 -- 172.21.15.52:0/1467153682 wait complete. 2024-05-27T07:28:49.136 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:28:49.145+0000 7fb74ebf0740 20 client.25817 trim_cache size 0 max 16384 2024-05-27T07:28:49.172 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:28:49.183+0000 7fb74ebf0740 1 Processor -- start 2024-05-27T07:28:49.172 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:28:49.183+0000 7fb74ebf0740 1 -- start start 2024-05-27T07:28:49.173 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:28:49.183+0000 7fb74ebf0740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55bcd9c9f2f0 0x55bcda06d080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:28:49.173 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:28:49.183+0000 7fb74ebf0740 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55bcd9ca74d0 0x55bcda06d5c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:28:49.173 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:28:49.183+0000 7fb74ebf0740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55bcd9cc81b0 0x55bcda065540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:28:49.173 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:28:49.183+0000 7fb74ebf0740 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x7fb7401b11b0 con 0x55bcd9ca74d0 2024-05-27T07:28:49.173 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:28:49.183+0000 7fb74ebf0740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7fb7401b2db0 con 0x55bcd9c9f2f0 2024-05-27T07:28:49.173 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:28:49.183+0000 7fb74ebf0740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7fb7401b1880 con 0x55bcd9cc81b0 2024-05-27T07:28:49.177 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:28:49.187+0000 7fb745adc640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55bcd9c9f2f0 0x55bcda06d080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:28:49.177 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:28:49.187+0000 7fb745adc640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55bcd9c9f2f0 0x55bcda06d080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.52:48934/0 (socket says 172.21.15.52:48934) 2024-05-27T07:28:49.177 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:28:49.187+0000 7fb745adc640 1 -- 172.21.15.52:0/2187780786 learned_addr learned my addr 172.21.15.52:0/2187780786 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-05-27T07:28:49.177 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:28:49.187+0000 7fb7462dd640 1 --2- 172.21.15.52:0/2187780786 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55bcd9cc81b0 0x55bcda065540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:28:49.183 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:28:49.193+0000 7fb745adc640 1 -- 172.21.15.52:0/2187780786 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55bcd9cc81b0 msgr2=0x55bcda065540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:28:49.183 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:28:49.193+0000 7fb745adc640 1 --2- 172.21.15.52:0/2187780786 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55bcd9cc81b0 0x55bcda065540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:28:49.183 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:28:49.193+0000 7fb745adc640 1 -- 172.21.15.52:0/2187780786 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55bcd9ca74d0 msgr2=0x55bcda06d5c0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-05-27T07:28:49.183 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:28:49.193+0000 7fb745adc640 1 --2- 172.21.15.52:0/2187780786 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55bcd9ca74d0 0x55bcda06d5c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:28:49.184 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:28:49.193+0000 7fb745adc640 1 -- 172.21.15.52:0/2187780786 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55bcd9c86fb0 con 0x55bcd9c9f2f0 2024-05-27T07:28:49.185 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:28:49.195+0000 7fb745adc640 1 --2- 172.21.15.52:0/2187780786 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55bcd9c9f2f0 0x55bcda06d080 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7fb73c01f900 tx=0x7fb73c01a000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=75526ee11892f0a0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:28:49.198 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:28:49.205+0000 7fb72b7fe640 1 -- 172.21.15.52:0/2187780786 <== mon.2 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb73c005020 con 0x55bcd9c9f2f0 2024-05-27T07:28:49.198 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:28:49.205+0000 7fb72b7fe640 1 -- 172.21.15.52:0/2187780786 <== mon.2 v2:172.21.15.105:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fb73c0047a0 con 0x55bcd9c9f2f0 2024-05-27T07:28:49.198 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:28:49.205+0000 7fb72b7fe640 1 -- 172.21.15.52:0/2187780786 <== mon.2 v2:172.21.15.105:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb73c02b420 con 0x55bcd9c9f2f0 2024-05-27T07:28:49.199 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:28:49.206+0000 7fb74ebf0740 1 -- 172.21.15.52:0/2187780786 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55bcd9c9f2f0 msgr2=0x55bcda06d080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:28:49.199 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:28:49.206+0000 7fb74ebf0740 1 --2- 172.21.15.52:0/2187780786 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55bcd9c9f2f0 0x55bcda06d080 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7fb73c01f900 tx=0x7fb73c01a000 comp rx=0 tx=0).stop 2024-05-27T07:28:49.199 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:28:49.207+0000 7fb74ebf0740 1 -- 172.21.15.52:0/2187780786 shutdown_connections 2024-05-27T07:28:49.199 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:28:49.207+0000 7fb74ebf0740 1 --2- 172.21.15.52:0/2187780786 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55bcd9cc81b0 0x55bcda065540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:28:49.199 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:28:49.207+0000 7fb74ebf0740 1 --2- 172.21.15.52:0/2187780786 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55bcd9ca74d0 0x55bcda06d5c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:28:49.199 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:28:49.207+0000 7fb74ebf0740 1 --2- 172.21.15.52:0/2187780786 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55bcd9c9f2f0 0x55bcda06d080 unknown :-1 s=CLOSED pgs=108 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:28:49.199 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:28:49.207+0000 7fb74ebf0740 1 -- 172.21.15.52:0/2187780786 >> 172.21.15.52:0/2187780786 conn(0x55bcd9cbc300 msgr2=0x55bcda0722d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:28:49.214 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:28:49.214+0000 7fb74ebf0740 1 -- 172.21.15.52:0/2187780786 shutdown_connections 2024-05-27T07:28:49.214 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:28:49.214+0000 7fb74ebf0740 1 -- 172.21.15.52:0/2187780786 wait complete. 2024-05-27T07:28:49.214 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:28:49.214+0000 7fb74ebf0740 1 -- start start 2024-05-27T07:28:49.214 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:28:49.214+0000 7fb74ebf0740 1 Processor -- start 2024-05-27T07:28:49.214 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:28:49.223+0000 7fb74ebf0740 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55bcd9c9f2f0 0x55bcda0ea010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:28:49.214 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:28:49.223+0000 7fb74ebf0740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55bcd9ca74d0 0x55bcda0e6370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:28:49.214 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:28:49.223+0000 7fb74ebf0740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55bcd9cc81b0 0x55bcda0e68b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:28:49.214 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:28:49.223+0000 7fb74ebf0740 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x7fb7401b26e0 con 0x55bcd9c9f2f0 2024-05-27T07:28:49.214 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:28:49.223+0000 7fb74ebf0740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7fb7401b1f50 con 0x55bcd9ca74d0 2024-05-27T07:28:49.214 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:28:49.223+0000 7fb74ebf0740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7fb7401af190 con 0x55bcd9cc81b0 2024-05-27T07:28:49.214 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:28:49.223+0000 7fb74ebf0740 10 client.0 _ll_register_callbacks cb 0x7fb7300a5b40 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T07:28:49.215 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:28:49.223+0000 7fb745adc640 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55bcd9c9f2f0 0x55bcda0ea010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:28:49.215 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:28:49.223+0000 7fb745adc640 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55bcd9c9f2f0 0x55bcda0ea010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.52:46154/0 (socket says 172.21.15.52:46154) 2024-05-27T07:28:49.215 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:28:49.223+0000 7fb745adc640 1 -- 172.21.15.52:0/780724580 learned_addr learned my addr 172.21.15.52:0/780724580 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-05-27T07:28:49.215 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:28:49.223+0000 7fb7452db640 1 --2- 172.21.15.52:0/780724580 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55bcd9ca74d0 0x55bcda0e6370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:28:49.215 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:28:49.223+0000 7fb7462dd640 1 --2- 172.21.15.52:0/780724580 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55bcd9cc81b0 0x55bcda0e68b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:28:49.215 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:28:49.224+0000 7fb7452db640 1 -- 172.21.15.52:0/780724580 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55bcd9cc81b0 msgr2=0x55bcda0e68b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:28:49.215 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:28:49.224+0000 7fb7452db640 1 --2- 172.21.15.52:0/780724580 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55bcd9cc81b0 0x55bcda0e68b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:28:49.215 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:28:49.224+0000 7fb7452db640 1 -- 172.21.15.52:0/780724580 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55bcd9c9f2f0 msgr2=0x55bcda0ea010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:28:49.215 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:28:49.224+0000 7fb7452db640 1 --2- 172.21.15.52:0/780724580 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55bcd9c9f2f0 0x55bcda0ea010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:28:49.215 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:28:49.224+0000 7fb7452db640 1 -- 172.21.15.52:0/780724580 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55bcda0e7020 con 0x55bcd9ca74d0 2024-05-27T07:28:49.217 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:28:49.224+0000 7fb7452db640 1 --2- 172.21.15.52:0/780724580 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55bcd9ca74d0 0x55bcda0e6370 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7fb7300a5c30 tx=0x7fb7300e0050 comp rx=0 tx=0).ready entity=mon.2 client_cookie=e55a312ef34101fb server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:28:49.220 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:28:49.231+0000 7fb7297fa640 10 client.0 ms_handle_connect on v2:172.21.15.105:3300/0 2024-05-27T07:28:49.220 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:28:49.231+0000 7fb7297fa640 1 -- 172.21.15.52:0/780724580 <== mon.2 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb7300cae60 con 0x55bcd9ca74d0 2024-05-27T07:28:49.220 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:28:49.231+0000 7fb74ebf0740 1 -- 172.21.15.52:0/780724580 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fb73c0047a0 con 0x55bcd9ca74d0 2024-05-27T07:28:49.221 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:28:49.231+0000 7fb7297fa640 1 -- 172.21.15.52:0/780724580 <== mon.2 v2:172.21.15.105:3300/0 2 ==== osd_map(96..96 src has 1..96) ==== 9379+0+0 (secure 0 0 0) 0x7fb7300a5690 con 0x55bcd9ca74d0 2024-05-27T07:28:49.221 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:28:49.231+0000 7fb7297fa640 1 -- 172.21.15.52:0/780724580 <== mon.2 v2:172.21.15.105:3300/0 3 ==== mon_get_version_reply(handle=1 version=41) ==== 24+0+0 (secure 0 0 0) 0x7fb73008c050 con 0x55bcd9ca74d0 2024-05-27T07:28:49.227 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:28:49.238+0000 7fb74ebf0740 10 client.25823 fetch_fsmap learned FSMap version 41 2024-05-27T07:28:49.227 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:28:49.238+0000 7fb74ebf0740 1 -- 172.21.15.52:0/780724580 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,fsmap=41,monmap=4+}) -- 0x55bcda07bdf0 con 0x55bcd9ca74d0 2024-05-27T07:28:49.228 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:28:49.238+0000 7fb7297fa640 1 -- 172.21.15.52:0/780724580 <== mon.2 v2:172.21.15.105:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fb73c0047a0 con 0x55bcd9ca74d0 2024-05-27T07:28:49.228 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:28:49.238+0000 7fb7297fa640 1 -- 172.21.15.52:0/780724580 <== mon.2 v2:172.21.15.105:3300/0 5 ==== fsmap(e 41) ==== 8378+0+0 (secure 0 0 0) 0x7fb730003fb0 con 0x55bcd9ca74d0 2024-05-27T07:28:49.228 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:28:49.238+0000 7fb74ebf0740 10 client.25823 fetch_fsmap finished waiting for FSMap version 41 2024-05-27T07:28:49.228 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:28:49.238+0000 7fb74ebf0740 10 client.25823 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-05-27T07:28:49.228 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:28:49.238+0000 7fb74ebf0740 20 client.25823 populate_metadata read hostname 'smithi052' 2024-05-27T07:28:49.228 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:28:49.238+0000 7fb74ebf0740 1 --2- 172.21.15.52:0/780724580 >> [v2:172.21.15.105:6838/500348151,v1:172.21.15.105:6839/500348151] conn(0x55bcd9c5c1c0 0x55bcda0ec750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:28:49.228 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:28:49.238+0000 7fb74ebf0740 4 client.25823 mds_command: new command op to 24527 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-05-27T07:28:49.229 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:28:49.238+0000 7fb74ebf0740 1 -- 172.21.15.52:0/780724580 --> [v2:172.21.15.105:6838/500348151,v1:172.21.15.105:6839/500348151] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55bcd9fd7360 con 0x55bcd9c5c1c0 2024-05-27T07:28:49.229 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:28:49.239+0000 7fb745adc640 1 --2- 172.21.15.52:0/780724580 >> [v2:172.21.15.105:6838/500348151,v1:172.21.15.105:6839/500348151] conn(0x55bcd9c5c1c0 0x55bcda0ec750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:28:49.229 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:28:49.240+0000 7fb745adc640 1 --2- 172.21.15.52:0/780724580 >> [v2:172.21.15.105:6838/500348151,v1:172.21.15.105:6839/500348151] conn(0x55bcd9c5c1c0 0x55bcda0ec750 crc :-1 s=READY pgs=88 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=42a786487e25356d server_cookie=a468247150c3117e in_seq=0 out_seq=0 2024-05-27T07:28:49.229 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:28:49.240+0000 7fb7297fa640 10 client.25823 ms_handle_connect on v2:172.21.15.105:6838/500348151 2024-05-27T07:28:49.729 INFO:journalctl@ceph.mds.g.smithi052.stdout:May 27 07:28:49 smithi052 podman[103915]: 2024-05-27 07:28:49.398271974 +0000 UTC m=+0.075988188 image pull e36ea4fc8a5807e8d2701370e350a0de3856538797f7fba64cfce7e344bd6799 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6 2024-05-27T07:28:50.390 INFO:journalctl@ceph.mds.g.smithi052.stdout:May 27 07:28:50 smithi052 podman[103915]: 2024-05-27 07:28:50.399215645 +0000 UTC m=+1.076931671 container create 881cedb8d2465d95bc3a2989662221d785f4982234a579c062b0e70889cb4b9a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-g, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240521, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.schema-version=1.0, GIT_CLEAN=True) 2024-05-27T07:28:50.394 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:28:50 smithi052 ceph-mon[31955]: pgmap v94: 129 pgs: 129 active+clean; 143 MiB data, 732 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 MiB/s rd, 5.3 MiB/s wr, 658 op/s 2024-05-27T07:28:50.394 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:28:50 smithi052 ceph-mon[31955]: MDS daemon mds.g is removed because it is dead or otherwise unavailable. 2024-05-27T07:28:50.395 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:28:50 smithi052 ceph-mon[31955]: fsmap cephfs:5 5 up:standby-replay 1 up:standby 5 up:active 2024-05-27T07:28:50.439 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:28:50 smithi105 ceph-mon[35977]: pgmap v94: 129 pgs: 129 active+clean; 143 MiB data, 732 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 MiB/s rd, 5.3 MiB/s wr, 658 op/s 2024-05-27T07:28:50.439 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:28:50 smithi105 ceph-mon[35977]: MDS daemon mds.g is removed because it is dead or otherwise unavailable. 2024-05-27T07:28:50.439 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:28:50 smithi105 ceph-mon[35977]: fsmap cephfs:5 5 up:standby-replay 1 up:standby 5 up:active 2024-05-27T07:28:50.601 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:28:50 smithi173 ceph-mon[37000]: pgmap v94: 129 pgs: 129 active+clean; 143 MiB data, 732 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 MiB/s rd, 5.3 MiB/s wr, 658 op/s 2024-05-27T07:28:50.601 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:28:50 smithi173 ceph-mon[37000]: MDS daemon mds.g is removed because it is dead or otherwise unavailable. 2024-05-27T07:28:50.602 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:28:50 smithi173 ceph-mon[37000]: fsmap cephfs:5 5 up:standby-replay 1 up:standby 5 up:active 2024-05-27T07:28:51.791 INFO:journalctl@ceph.mds.d.smithi052.stdout:May 27 07:28:51 smithi052 systemd[1]: ceph-73df2636-1bf8-11ef-bc9b-c7b262605968@mds.d.service: Main process exited, code=exited, status=134/n/a 2024-05-27T07:28:52.114 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:52.123+0000 7fce4bfff700 1 -- 172.21.15.52:0/2495372382 <== mon.1 v2:172.21.15.173:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fce4c012650 con 0x7fce5410d490 2024-05-27T07:28:52.117 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:52.124+0000 7fce5b901700 1 -- 172.21.15.52:0/2495372382 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fce5410d490 msgr2=0x7fce54119920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:28:52.117 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:52.124+0000 7fce5b901700 1 --2- 172.21.15.52:0/2495372382 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fce5410d490 0x7fce54119920 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7fce4c009f90 tx=0x7fce4c00b330 comp rx=0 tx=0).stop 2024-05-27T07:28:52.117 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:52.127+0000 7fce5b901700 1 -- 172.21.15.52:0/2495372382 shutdown_connections 2024-05-27T07:28:52.117 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:52.127+0000 7fce5b901700 1 --2- 172.21.15.52:0/2495372382 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fce5410d490 0x7fce54119920 unknown :-1 s=CLOSED pgs=102 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:28:52.117 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:52.127+0000 7fce5b901700 1 --2- 172.21.15.52:0/2495372382 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fce54112b80 0x7fce54112f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:28:52.117 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:52.127+0000 7fce5b901700 1 --2- 172.21.15.52:0/2495372382 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fce5410cb40 0x7fce5410cf50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:28:52.118 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:52.127+0000 7fce5b901700 1 -- 172.21.15.52:0/2495372382 >> 172.21.15.52:0/2495372382 conn(0x7fce54070780 msgr2=0x7fce54070b90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:28:52.118 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:52.127+0000 7fce5b901700 1 -- 172.21.15.52:0/2495372382 shutdown_connections 2024-05-27T07:28:52.118 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:52.127+0000 7fce5b901700 1 -- 172.21.15.52:0/2495372382 wait complete. 2024-05-27T07:28:52.121 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:52.129+0000 7fce5b901700 1 Processor -- start 2024-05-27T07:28:52.121 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:52.129+0000 7fce5b901700 1 -- start start 2024-05-27T07:28:52.121 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:52.129+0000 7fce5b901700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fce5410cb40 0x7fce541a21d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:28:52.121 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:52.129+0000 7fce5b901700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fce5410d490 0x7fce541a2710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:28:52.121 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:52.129+0000 7fce5b901700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fce54112b80 0x7fce541ad250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:28:52.121 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:52.129+0000 7fce5b901700 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x7fce5411cd50 con 0x7fce54112b80 2024-05-27T07:28:52.121 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:52.129+0000 7fce5b901700 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7fce5411cbd0 con 0x7fce5410d490 2024-05-27T07:28:52.121 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:52.129+0000 7fce5b901700 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7fce5411ced0 con 0x7fce5410cb40 2024-05-27T07:28:52.121 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:52.129+0000 7fce58e9c700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fce5410d490 0x7fce541a2710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:28:52.121 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:52.129+0000 7fce58e9c700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fce5410d490 0x7fce541a2710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.52:58826/0 (socket says 172.21.15.52:58826) 2024-05-27T07:28:52.121 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:52.129+0000 7fce58e9c700 1 -- 172.21.15.52:0/256029496 learned_addr learned my addr 172.21.15.52:0/256029496 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-05-27T07:28:52.122 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:52.129+0000 7fce59e9e700 1 --2- 172.21.15.52:0/256029496 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fce54112b80 0x7fce541ad250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:28:52.122 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:52.130+0000 7fce5969d700 1 --2- 172.21.15.52:0/256029496 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fce5410cb40 0x7fce541a21d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:28:52.122 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:52.130+0000 7fce59e9e700 1 -- 172.21.15.52:0/256029496 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fce5410cb40 msgr2=0x7fce541a21d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:28:52.122 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:52.130+0000 7fce59e9e700 1 --2- 172.21.15.52:0/256029496 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fce5410cb40 0x7fce541a21d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:28:52.122 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:52.130+0000 7fce59e9e700 1 -- 172.21.15.52:0/256029496 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fce5410d490 msgr2=0x7fce541a2710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:28:52.122 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:52.130+0000 7fce59e9e700 1 --2- 172.21.15.52:0/256029496 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fce5410d490 0x7fce541a2710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:28:52.122 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:52.130+0000 7fce59e9e700 1 -- 172.21.15.52:0/256029496 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fce541a2fd0 con 0x7fce54112b80 2024-05-27T07:28:52.122 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:52.131+0000 7fce59e9e700 1 --2- 172.21.15.52:0/256029496 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fce54112b80 0x7fce541ad250 secure :-1 s=READY pgs=595 cs=0 l=1 rev1=1 crypto rx=0x7fce4c009f90 tx=0x7fce4c00cdb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:28:52.124 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:52.131+0000 7fce4a7fc700 1 -- 172.21.15.52:0/256029496 <== mon.0 v2:172.21.15.52:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fce4c021070 con 0x7fce54112b80 2024-05-27T07:28:52.124 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:52.132+0000 7fce4a7fc700 1 -- 172.21.15.52:0/256029496 <== mon.0 v2:172.21.15.52:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fce4c007b40 con 0x7fce54112b80 2024-05-27T07:28:52.124 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:52.132+0000 7fce4a7fc700 1 -- 172.21.15.52:0/256029496 <== mon.0 v2:172.21.15.52:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fce4c01b680 con 0x7fce54112b80 2024-05-27T07:28:52.124 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:52.132+0000 7fce5b901700 1 -- 172.21.15.52:0/256029496 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fce541ad850 con 0x7fce54112b80 2024-05-27T07:28:52.124 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:52.132+0000 7fce5b901700 1 -- 172.21.15.52:0/256029496 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fce541add40 con 0x7fce54112b80 2024-05-27T07:28:52.140 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:52.141+0000 7fce6a316b80 1 Processor -- start 2024-05-27T07:28:52.141 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:52.141+0000 7fce6a316b80 1 -- start start 2024-05-27T07:28:52.141 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:52.142+0000 7fce6a316b80 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55fdb723ae40 0x55fdb73be710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:28:52.141 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:52.142+0000 7fce6a316b80 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55fdb73bec50 0x55fdb73b8a40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:28:52.141 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:52.142+0000 7fce6a316b80 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55fdb73b8f80 0x55fdb73b9390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:28:52.141 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:52.142+0000 7fce6a316b80 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x55fdb72e7c90 con 0x55fdb73b8f80 2024-05-27T07:28:52.141 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:52.142+0000 7fce6a316b80 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x55fdb72d7830 con 0x55fdb723ae40 2024-05-27T07:28:52.141 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:52.142+0000 7fce6a316b80 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x55fdb72db870 con 0x55fdb73bec50 2024-05-27T07:28:52.141 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:52.142+0000 7fce58e9c700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55fdb723ae40 0x55fdb73be710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:28:52.141 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:52.142+0000 7fce58e9c700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55fdb723ae40 0x55fdb73be710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.52:58834/0 (socket says 172.21.15.52:58834) 2024-05-27T07:28:52.141 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:52.142+0000 7fce58e9c700 1 -- 172.21.15.52:0/751416232 learned_addr learned my addr 172.21.15.52:0/751416232 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-05-27T07:28:52.142 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:52.142+0000 7fce5969d700 1 --2- 172.21.15.52:0/751416232 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55fdb73bec50 0x55fdb73b8a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:28:52.142 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:52.143+0000 7fce58e9c700 1 -- 172.21.15.52:0/751416232 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55fdb73bec50 msgr2=0x55fdb73b8a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:28:52.142 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:52.143+0000 7fce58e9c700 1 --2- 172.21.15.52:0/751416232 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55fdb73bec50 0x55fdb73b8a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:28:52.142 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:52.143+0000 7fce58e9c700 1 -- 172.21.15.52:0/751416232 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55fdb73b8f80 msgr2=0x55fdb73b9390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:28:52.142 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:52.143+0000 7fce58e9c700 1 --2- 172.21.15.52:0/751416232 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55fdb73b8f80 0x55fdb73b9390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:28:52.142 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:52.143+0000 7fce58e9c700 1 -- 172.21.15.52:0/751416232 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55fdb72278d0 con 0x55fdb723ae40 2024-05-27T07:28:52.142 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:52.143+0000 7fce4a7fc700 1 -- 172.21.15.52:0/256029496 <== mon.0 v2:172.21.15.52:3300/0 4 ==== mgrmap(e 26) ==== 149783+0+0 (secure 0 0 0) 0x7fce4c024070 con 0x7fce54112b80 2024-05-27T07:28:52.142 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:52.143+0000 7fce58e9c700 1 --2- 172.21.15.52:0/751416232 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55fdb723ae40 0x55fdb73be710 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7fce50009e10 tx=0x7fce50013740 comp rx=0 tx=0).ready entity=mon.2 client_cookie=5ce86b49ca17474a server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:28:52.142 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:52.144+0000 7fce3f7fe700 1 -- 172.21.15.52:0/751416232 <== mon.2 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fce50010420 con 0x55fdb723ae40 2024-05-27T07:28:52.142 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:52.144+0000 7fce3f7fe700 1 -- 172.21.15.52:0/751416232 <== mon.2 v2:172.21.15.105:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fce50010a70 con 0x55fdb723ae40 2024-05-27T07:28:52.142 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:52.144+0000 7fce3f7fe700 1 -- 172.21.15.52:0/751416232 <== mon.2 v2:172.21.15.105:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fce5001ba50 con 0x55fdb723ae40 2024-05-27T07:28:52.143 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:52.145+0000 7fce6a316b80 1 -- 172.21.15.52:0/751416232 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55fdb723ae40 msgr2=0x55fdb73be710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:28:52.143 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:52.145+0000 7fce6a316b80 1 --2- 172.21.15.52:0/751416232 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55fdb723ae40 0x55fdb73be710 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7fce50009e10 tx=0x7fce50013740 comp rx=0 tx=0).stop 2024-05-27T07:28:52.143 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:52.145+0000 7fce6a316b80 1 -- 172.21.15.52:0/751416232 shutdown_connections 2024-05-27T07:28:52.143 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:52.145+0000 7fce6a316b80 1 --2- 172.21.15.52:0/751416232 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55fdb73b8f80 0x55fdb73b9390 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:28:52.143 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:52.145+0000 7fce6a316b80 1 --2- 172.21.15.52:0/751416232 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55fdb723ae40 0x55fdb73be710 unknown :-1 s=CLOSED pgs=110 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:28:52.143 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:52.145+0000 7fce6a316b80 1 --2- 172.21.15.52:0/751416232 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55fdb73bec50 0x55fdb73b8a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:28:52.143 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:52.145+0000 7fce6a316b80 1 -- 172.21.15.52:0/751416232 >> 172.21.15.52:0/751416232 conn(0x55fdb7295af0 msgr2=0x55fdb73b3640 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:28:52.143 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:52.145+0000 7fce6a316b80 1 -- 172.21.15.52:0/751416232 shutdown_connections 2024-05-27T07:28:52.143 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:52.145+0000 7fce6a316b80 1 -- 172.21.15.52:0/751416232 wait complete. 2024-05-27T07:28:52.143 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:52.146+0000 7fce6a316b80 1 -- start start 2024-05-27T07:28:52.143 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:52.146+0000 7fce6a316b80 1 Processor -- start 2024-05-27T07:28:52.143 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:52.147+0000 7fce6a316b80 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55fdb723ae40 0x55fdb73c23a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:28:52.143 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:52.147+0000 7fce6a316b80 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55fdb73b8f80 0x55fdb73c28e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:28:52.144 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:52.147+0000 7fce58e9c700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55fdb723ae40 0x55fdb73c23a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:28:52.144 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:52.147+0000 7fce58e9c700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55fdb723ae40 0x55fdb73c23a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.52:58844/0 (socket says 172.21.15.52:58844) 2024-05-27T07:28:52.144 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:52.147+0000 7fce5969d700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55fdb73b8f80 0x55fdb73c28e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:28:52.144 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:52.147+0000 7fce58e9c700 1 -- 172.21.15.52:0/1131476100 learned_addr learned my addr 172.21.15.52:0/1131476100 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-05-27T07:28:52.144 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:52.147+0000 7fce6a316b80 1 --2- 172.21.15.52:0/1131476100 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55fdb73bec50 0x55fdb73c2e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:28:52.144 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:52.147+0000 7fce6a316b80 1 -- 172.21.15.52:0/1131476100 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x55fdb71ddfa0 con 0x55fdb73bec50 2024-05-27T07:28:52.144 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:52.147+0000 7fce6a316b80 1 -- 172.21.15.52:0/1131476100 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x55fdb72797c0 con 0x55fdb723ae40 2024-05-27T07:28:52.144 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:52.147+0000 7fce6a316b80 1 -- 172.21.15.52:0/1131476100 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x55fdb73c5740 con 0x55fdb73b8f80 2024-05-27T07:28:52.145 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:52.148+0000 7fce58e9c700 1 -- 172.21.15.52:0/1131476100 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55fdb73b8f80 msgr2=0x55fdb73c28e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:28:52.145 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:52.148+0000 7fce58e9c700 1 --2- 172.21.15.52:0/1131476100 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55fdb73b8f80 0x55fdb73c28e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:28:52.146 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:52.148+0000 7fce58e9c700 1 -- 172.21.15.52:0/1131476100 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55fdb73bec50 msgr2=0x55fdb73c2e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:28:52.146 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:52.148+0000 7fce58e9c700 1 --2- 172.21.15.52:0/1131476100 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55fdb73bec50 0x55fdb73c2e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:28:52.146 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:52.148+0000 7fce58e9c700 1 -- 172.21.15.52:0/1131476100 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55fdb7276690 con 0x55fdb723ae40 2024-05-27T07:28:52.146 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:52.148+0000 7fce58e9c700 1 --2- 172.21.15.52:0/1131476100 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55fdb723ae40 0x55fdb73c23a0 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7fce50005720 tx=0x7fce50005750 comp rx=0 tx=0).ready entity=mon.2 client_cookie=bb1c9680374a0fc4 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:28:52.146 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:52.148+0000 7fce6a316b80 10 client.0 _ll_register_callbacks cb 0x55fdb7272920 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T07:28:52.146 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:52.148+0000 7fce6a316b80 1 -- 172.21.15.52:0/1131476100 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fce50010a70 con 0x55fdb723ae40 2024-05-27T07:28:52.146 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:52.150+0000 7fce4a7fc700 1 --2- 172.21.15.52:0/256029496 >> [v2:172.21.15.52:6800/1822318384,v1:172.21.15.52:6801/1822318384] conn(0x7fce400b2ad0 0x7fce400b4fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:28:52.146 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:52.152+0000 7fce4a7fc700 1 -- 172.21.15.52:0/256029496 <== mon.0 v2:172.21.15.52:3300/0 5 ==== osd_map(96..96 src has 1..96) ==== 9379+0+0 (secure 0 0 0) 0x7fce4c00f020 con 0x7fce54112b80 2024-05-27T07:28:52.146 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:52.152+0000 7fce5969d700 1 --2- 172.21.15.52:0/256029496 >> [v2:172.21.15.52:6800/1822318384,v1:172.21.15.52:6801/1822318384] conn(0x7fce400b2ad0 0x7fce400b4fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:28:52.147 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:52.153+0000 7fce3cff9700 10 client.25835 ms_handle_connect on v2:172.21.15.105:3300/0 2024-05-27T07:28:52.147 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:52.153+0000 7fce3cff9700 1 -- 172.21.15.52:0/1131476100 <== mon.2 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fce5002a060 con 0x55fdb723ae40 2024-05-27T07:28:52.147 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:52.153+0000 7fce3cff9700 1 -- 172.21.15.52:0/1131476100 <== mon.2 v2:172.21.15.105:3300/0 2 ==== osd_map(96..96 src has 1..96) ==== 9379+0+0 (secure 0 0 0) 0x7fce5001bcd0 con 0x55fdb723ae40 2024-05-27T07:28:52.147 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:52.154+0000 7fce3cff9700 1 -- 172.21.15.52:0/1131476100 <== mon.2 v2:172.21.15.105:3300/0 3 ==== mon_get_version_reply(handle=1 version=42) ==== 24+0+0 (secure 0 0 0) 0x7fce50031050 con 0x55fdb723ae40 2024-05-27T07:28:52.147 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:52.157+0000 7fce6a316b80 10 client.25835 fetch_fsmap learned FSMap version 42 2024-05-27T07:28:52.147 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:52.157+0000 7fce6a316b80 1 -- 172.21.15.52:0/1131476100 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({fsmap=42}) -- 0x55fdb73f1eb0 con 0x55fdb723ae40 2024-05-27T07:28:52.148 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:52.157+0000 7fce3cff9700 1 -- 172.21.15.52:0/1131476100 <== mon.2 v2:172.21.15.105:3300/0 4 ==== fsmap(e 42) ==== 7773+0+0 (secure 0 0 0) 0x7fce50024070 con 0x55fdb723ae40 2024-05-27T07:28:52.148 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:52.158+0000 7fce6a316b80 10 client.25835 fetch_fsmap finished waiting for FSMap version 42 2024-05-27T07:28:52.148 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:52.158+0000 7fce6a316b80 10 client.25835 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-05-27T07:28:52.148 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:52.158+0000 7fce5969d700 1 --2- 172.21.15.52:0/256029496 >> [v2:172.21.15.52:6800/1822318384,v1:172.21.15.52:6801/1822318384] conn(0x7fce400b2ad0 0x7fce400b4fe0 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x55fdb73b01b0 tx=0x7fce4400d550 comp rx=0 tx=0).ready entity=mgr.15333 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:28:52.149 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:52.158+0000 7fce6a316b80 20 client.25835 populate_metadata read hostname 'smithi052' 2024-05-27T07:28:52.149 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:52.158+0000 7fce6a316b80 1 --2- 172.21.15.52:0/1131476100 >> [v2:172.21.15.105:6838/500348151,v1:172.21.15.105:6839/500348151] conn(0x55fdb73f4850 0x55fdb73f6d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:28:52.149 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:52.158+0000 7fce6a316b80 4 client.25835 mds_command: new command op to 24527 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-27T07:28:52.149 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:52.158+0000 7fce6a316b80 1 -- 172.21.15.52:0/1131476100 --> [v2:172.21.15.105:6838/500348151,v1:172.21.15.105:6839/500348151] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x7fce50010a70 con 0x55fdb73f4850 2024-05-27T07:28:52.149 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:52.158+0000 7fce58e9c700 1 --2- 172.21.15.52:0/1131476100 >> [v2:172.21.15.105:6838/500348151,v1:172.21.15.105:6839/500348151] conn(0x55fdb73f4850 0x55fdb73f6d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:28:52.149 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:52.159+0000 7fce58e9c700 1 --2- 172.21.15.52:0/1131476100 >> [v2:172.21.15.105:6838/500348151,v1:172.21.15.105:6839/500348151] conn(0x55fdb73f4850 0x55fdb73f6d00 crc :-1 s=READY pgs=89 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=66760cbc86a5d555 server_cookie=d4ee55540292112f in_seq=0 out_seq=0 2024-05-27T07:28:52.149 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:52.159+0000 7fce3cff9700 10 client.25835 ms_handle_connect on v2:172.21.15.105:6838/500348151 2024-05-27T07:28:52.153 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:52.163+0000 7fce3cff9700 1 -- 172.21.15.52:0/1131476100 <== mds.0 v2:172.21.15.105:6838/500348151 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24041 (crc 0 0 0) 0x7fce50010a70 con 0x55fdb73f4850 2024-05-27T07:28:52.154 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:52.163+0000 7fce3cff9700 10 client.25835 handle_command_reply: tid=0 2024-05-27T07:28:52.154 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:52.164+0000 7fce6a316b80 1 client.25835 shutdown 2024-05-27T07:28:52.154 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:52.164+0000 7fce6a316b80 1 -- 172.21.15.52:0/1131476100 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55fdb723ae40 msgr2=0x55fdb73c23a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:28:52.154 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:52.164+0000 7fce6a316b80 1 --2- 172.21.15.52:0/1131476100 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55fdb723ae40 0x55fdb73c23a0 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7fce50005720 tx=0x7fce50005750 comp rx=0 tx=0).stop 2024-05-27T07:28:52.155 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:52.165+0000 7fce6a316b80 1 -- 172.21.15.52:0/1131476100 shutdown_connections 2024-05-27T07:28:52.155 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:52.165+0000 7fce6a316b80 1 --2- 172.21.15.52:0/1131476100 >> [v2:172.21.15.105:6838/500348151,v1:172.21.15.105:6839/500348151] conn(0x55fdb73f4850 0x55fdb73f6d00 crc :-1 s=READY pgs=89 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:28:52.155 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:52.165+0000 7fce6a316b80 1 --2- 172.21.15.52:0/1131476100 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55fdb73bec50 0x55fdb73c2e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:28:52.155 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:52.165+0000 7fce6a316b80 1 --2- 172.21.15.52:0/1131476100 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55fdb723ae40 0x55fdb73c23a0 unknown :-1 s=CLOSED pgs=111 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:28:52.155 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:52.165+0000 7fce6a316b80 1 --2- 172.21.15.52:0/1131476100 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55fdb73b8f80 0x55fdb73c28e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:28:52.155 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:52.165+0000 7fce6a316b80 1 -- 172.21.15.52:0/1131476100 >> 172.21.15.52:0/1131476100 conn(0x55fdb7295af0 msgr2=0x55fdb73b7a30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:28:52.156 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:52.165+0000 7fce3cff9700 0 client.25835 ms_handle_reset on v2:172.21.15.105:6838/500348151 2024-05-27T07:28:52.159 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:52.167+0000 7fce6a316b80 1 -- 172.21.15.52:0/1131476100 shutdown_connections 2024-05-27T07:28:52.159 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:52.167+0000 7fce6a316b80 1 -- 172.21.15.52:0/1131476100 wait complete. 2024-05-27T07:28:52.159 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:52.167+0000 7fce6a316b80 20 client.25835 trim_cache size 0 max 16384 2024-05-27T07:28:52.199 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:52.209+0000 7fce6a316b80 1 Processor -- start 2024-05-27T07:28:52.200 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:52.210+0000 7fce6a316b80 1 -- start start 2024-05-27T07:28:52.200 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:52.210+0000 7fce6a316b80 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55fdb723ae40 0x55fdb73b8a40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:28:52.200 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:52.210+0000 7fce6a316b80 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55fdb73b8f80 0x55fdb73bdaf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:28:52.200 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:52.210+0000 7fce6a316b80 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55fdb73bec50 0x55fdb73c4700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:28:52.200 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:52.210+0000 7fce6a316b80 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x7fce380107f0 con 0x55fdb73bec50 2024-05-27T07:28:52.200 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:52.210+0000 7fce6a316b80 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7fce3800e520 con 0x55fdb73b8f80 2024-05-27T07:28:52.200 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:52.210+0000 7fce6a316b80 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7fce38010120 con 0x55fdb723ae40 2024-05-27T07:28:52.201 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:52.211+0000 7fce58e9c700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55fdb73b8f80 0x55fdb73bdaf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:28:52.201 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:52.211+0000 7fce58e9c700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55fdb73b8f80 0x55fdb73bdaf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.52:58852/0 (socket says 172.21.15.52:58852) 2024-05-27T07:28:52.201 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:52.211+0000 7fce58e9c700 1 -- 172.21.15.52:0/3356189890 learned_addr learned my addr 172.21.15.52:0/3356189890 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-05-27T07:28:52.201 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:52.211+0000 7fce58e9c700 1 -- 172.21.15.52:0/3356189890 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55fdb723ae40 msgr2=0x55fdb73b8a40 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-05-27T07:28:52.201 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:52.211+0000 7fce58e9c700 1 --2- 172.21.15.52:0/3356189890 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55fdb723ae40 0x55fdb73b8a40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:28:52.201 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:52.211+0000 7fce58e9c700 1 -- 172.21.15.52:0/3356189890 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55fdb73bec50 msgr2=0x55fdb73c4700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:28:52.201 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:52.211+0000 7fce58e9c700 1 --2- 172.21.15.52:0/3356189890 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55fdb73bec50 0x55fdb73c4700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:28:52.201 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:52.211+0000 7fce58e9c700 1 -- 172.21.15.52:0/3356189890 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55fdb73c50c0 con 0x55fdb73b8f80 2024-05-27T07:28:52.202 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:52.212+0000 7fce58e9c700 1 --2- 172.21.15.52:0/3356189890 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55fdb73b8f80 0x55fdb73bdaf0 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7fce50015010 tx=0x7fce50023000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=59fdeaa8f27bb2c2 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:28:52.202 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:52.212+0000 7fce3effd700 1 -- 172.21.15.52:0/3356189890 <== mon.2 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fce50042070 con 0x55fdb73b8f80 2024-05-27T07:28:52.202 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:52.212+0000 7fce3effd700 1 -- 172.21.15.52:0/3356189890 <== mon.2 v2:172.21.15.105:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fce50010a30 con 0x55fdb73b8f80 2024-05-27T07:28:52.203 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:52.212+0000 7fce3effd700 1 -- 172.21.15.52:0/3356189890 <== mon.2 v2:172.21.15.105:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fce5001b3f0 con 0x55fdb73b8f80 2024-05-27T07:28:52.203 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:52.213+0000 7fce6a316b80 1 -- 172.21.15.52:0/3356189890 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55fdb73b8f80 msgr2=0x55fdb73bdaf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:28:52.203 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:52.213+0000 7fce6a316b80 1 --2- 172.21.15.52:0/3356189890 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55fdb73b8f80 0x55fdb73bdaf0 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7fce50015010 tx=0x7fce50023000 comp rx=0 tx=0).stop 2024-05-27T07:28:52.204 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:52.214+0000 7fce6a316b80 1 -- 172.21.15.52:0/3356189890 shutdown_connections 2024-05-27T07:28:52.204 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:52.214+0000 7fce6a316b80 1 --2- 172.21.15.52:0/3356189890 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55fdb73bec50 0x55fdb73c4700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:28:52.204 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:52.214+0000 7fce6a316b80 1 --2- 172.21.15.52:0/3356189890 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55fdb723ae40 0x55fdb73b8a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:28:52.204 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:52.214+0000 7fce6a316b80 1 --2- 172.21.15.52:0/3356189890 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55fdb73b8f80 0x55fdb73bdaf0 unknown :-1 s=CLOSED pgs=112 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:28:52.204 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:52.214+0000 7fce6a316b80 1 -- 172.21.15.52:0/3356189890 >> 172.21.15.52:0/3356189890 conn(0x55fdb7295af0 msgr2=0x55fdb73bd3e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:28:52.205 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:52.215+0000 7fce6a316b80 1 -- 172.21.15.52:0/3356189890 shutdown_connections 2024-05-27T07:28:52.205 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:52.215+0000 7fce6a316b80 1 -- 172.21.15.52:0/3356189890 wait complete. 2024-05-27T07:28:52.205 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:52.215+0000 7fce6a316b80 1 -- start start 2024-05-27T07:28:52.205 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:52.215+0000 7fce6a316b80 1 Processor -- start 2024-05-27T07:28:52.206 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:52.216+0000 7fce6a316b80 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55fdb723ae40 0x55fdb73cb1b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:28:52.206 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:52.216+0000 7fce6a316b80 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55fdb73b8f80 0x55fdb73fbdd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:28:52.206 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:52.216+0000 7fce6a316b80 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55fdb73bec50 0x55fdb73fc310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:28:52.206 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:52.216+0000 7fce6a316b80 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x7fce3800f2c0 con 0x55fdb723ae40 2024-05-27T07:28:52.206 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:52.216+0000 7fce6a316b80 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7fce3800ebf0 con 0x55fdb73bec50 2024-05-27T07:28:52.206 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:52.216+0000 7fce6a316b80 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7fce3800fa50 con 0x55fdb73b8f80 2024-05-27T07:28:52.206 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:52.216+0000 7fce6a316b80 10 client.0 _ll_register_callbacks cb 0x55fdb7272920 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T07:28:52.207 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:52.217+0000 7fce5969d700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55fdb723ae40 0x55fdb73cb1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:28:52.207 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:52.217+0000 7fce59e9e700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55fdb73bec50 0x55fdb73fc310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:28:52.207 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:52.217+0000 7fce5969d700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55fdb723ae40 0x55fdb73cb1b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.52:46654/0 (socket says 172.21.15.52:46654) 2024-05-27T07:28:52.207 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:52.217+0000 7fce59e9e700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55fdb73bec50 0x55fdb73fc310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.52:58868/0 (socket says 172.21.15.52:58868) 2024-05-27T07:28:52.207 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:52.217+0000 7fce5969d700 1 -- 172.21.15.52:0/4062879871 learned_addr learned my addr 172.21.15.52:0/4062879871 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-05-27T07:28:52.207 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:52.217+0000 7fce58e9c700 1 --2- 172.21.15.52:0/4062879871 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55fdb73b8f80 0x55fdb73fbdd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:28:52.207 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:52.217+0000 7fce59e9e700 1 -- 172.21.15.52:0/4062879871 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55fdb73b8f80 msgr2=0x55fdb73fbdd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:28:52.207 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:52.217+0000 7fce59e9e700 1 --2- 172.21.15.52:0/4062879871 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55fdb73b8f80 0x55fdb73fbdd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:28:52.208 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:52.217+0000 7fce59e9e700 1 -- 172.21.15.52:0/4062879871 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55fdb723ae40 msgr2=0x55fdb73cb1b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:28:52.208 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:52.217+0000 7fce59e9e700 1 --2- 172.21.15.52:0/4062879871 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55fdb723ae40 0x55fdb73cb1b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:28:52.208 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:52.217+0000 7fce59e9e700 1 -- 172.21.15.52:0/4062879871 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55fdb73fca90 con 0x55fdb73bec50 2024-05-27T07:28:52.208 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:52.219+0000 7fce59e9e700 1 --2- 172.21.15.52:0/4062879871 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55fdb73bec50 0x55fdb73fc310 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7fce4c0e0fd0 tx=0x7fce4c0ab600 comp rx=0 tx=0).ready entity=mon.2 client_cookie=51618bc570b9d1aa server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:28:52.208 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:52.219+0000 7fce3cff9700 10 client.0 ms_handle_connect on v2:172.21.15.105:3300/0 2024-05-27T07:28:52.209 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:52.219+0000 7fce3cff9700 1 -- 172.21.15.52:0/4062879871 <== mon.2 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fce4c0096c0 con 0x55fdb73bec50 2024-05-27T07:28:52.209 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:52.219+0000 7fce6a316b80 1 -- 172.21.15.52:0/4062879871 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fce50010a30 con 0x55fdb73bec50 2024-05-27T07:28:52.209 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:52.219+0000 7fce3cff9700 1 -- 172.21.15.52:0/4062879871 <== mon.2 v2:172.21.15.105:3300/0 2 ==== osd_map(96..96 src has 1..96) ==== 9379+0+0 (secure 0 0 0) 0x7fce4c0ac0b0 con 0x55fdb73bec50 2024-05-27T07:28:52.209 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:52.220+0000 7fce3cff9700 1 -- 172.21.15.52:0/4062879871 <== mon.2 v2:172.21.15.105:3300/0 3 ==== mon_get_version_reply(handle=1 version=42) ==== 24+0+0 (secure 0 0 0) 0x7fce4c0be050 con 0x55fdb73bec50 2024-05-27T07:28:52.210 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:52.220+0000 7fce6a316b80 10 client.25841 fetch_fsmap learned FSMap version 42 2024-05-27T07:28:52.210 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:52.220+0000 7fce6a316b80 1 -- 172.21.15.52:0/4062879871 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,fsmap=42,monmap=4+}) -- 0x55fdb745f3a0 con 0x55fdb73bec50 2024-05-27T07:28:52.210 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:52.220+0000 7fce3cff9700 1 -- 172.21.15.52:0/4062879871 <== mon.2 v2:172.21.15.105:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fce50010a30 con 0x55fdb73bec50 2024-05-27T07:28:52.210 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:52.221+0000 7fce3cff9700 1 -- 172.21.15.52:0/4062879871 <== mon.2 v2:172.21.15.105:3300/0 5 ==== fsmap(e 42) ==== 7773+0+0 (secure 0 0 0) 0x7fce4c011070 con 0x55fdb73bec50 2024-05-27T07:28:52.211 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:52.221+0000 7fce6a316b80 10 client.25841 fetch_fsmap finished waiting for FSMap version 42 2024-05-27T07:28:52.211 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:52.221+0000 7fce6a316b80 10 client.25841 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-05-27T07:28:52.211 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:52.221+0000 7fce6a316b80 20 client.25841 populate_metadata read hostname 'smithi052' 2024-05-27T07:28:52.211 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:52.221+0000 7fce6a316b80 1 --2- 172.21.15.52:0/4062879871 >> [v2:172.21.15.105:6838/500348151,v1:172.21.15.105:6839/500348151] conn(0x55fdb745fbe0 0x55fdb7462000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:28:52.211 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:52.221+0000 7fce6a316b80 4 client.25841 mds_command: new command op to 24527 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-05-27T07:28:52.211 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:52.221+0000 7fce6a316b80 1 -- 172.21.15.52:0/4062879871 --> [v2:172.21.15.105:6838/500348151,v1:172.21.15.105:6839/500348151] -- command(tid 0: {"prefix": "scrub status"}) -- 0x55fdb73849f0 con 0x55fdb745fbe0 2024-05-27T07:28:52.212 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:52.222+0000 7fce5969d700 1 --2- 172.21.15.52:0/4062879871 >> [v2:172.21.15.105:6838/500348151,v1:172.21.15.105:6839/500348151] conn(0x55fdb745fbe0 0x55fdb7462000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:28:52.212 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:52.223+0000 7fce5969d700 1 --2- 172.21.15.52:0/4062879871 >> [v2:172.21.15.105:6838/500348151,v1:172.21.15.105:6839/500348151] conn(0x55fdb745fbe0 0x55fdb7462000 crc :-1 s=READY pgs=90 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=75122fce25b9fbe0 server_cookie=607e58500bb82259 in_seq=0 out_seq=0 2024-05-27T07:28:52.212 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:52.223+0000 7fce3cff9700 10 client.25841 ms_handle_connect on v2:172.21.15.105:6838/500348151 2024-05-27T07:28:52.218 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:52.228+0000 7fce3cff9700 1 -- 172.21.15.52:0/4062879871 <== mds.0 v2:172.21.15.105:6838/500348151 1 ==== command_reply(tid 0: 0 ) ==== 8+0+268 (crc 0 0 0) 0x55fdb73849f0 con 0x55fdb745fbe0 2024-05-27T07:28:52.218 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:52.228+0000 7fce3cff9700 10 client.25841 handle_command_reply: tid=0 2024-05-27T07:28:52.218 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:52.228+0000 7fce6a316b80 1 client.25841 shutdown 2024-05-27T07:28:52.218 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:52.228+0000 7fce6a316b80 1 -- 172.21.15.52:0/4062879871 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55fdb73bec50 msgr2=0x55fdb73fc310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:28:52.218 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:52.228+0000 7fce6a316b80 1 --2- 172.21.15.52:0/4062879871 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55fdb73bec50 0x55fdb73fc310 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7fce4c0e0fd0 tx=0x7fce4c0ab600 comp rx=0 tx=0).stop 2024-05-27T07:28:52.219 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:52.229+0000 7fce6a316b80 1 -- 172.21.15.52:0/4062879871 shutdown_connections 2024-05-27T07:28:52.219 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:52.229+0000 7fce6a316b80 1 --2- 172.21.15.52:0/4062879871 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55fdb73bec50 0x55fdb73fc310 unknown :-1 s=CLOSED pgs=113 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:28:52.219 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:52.229+0000 7fce6a316b80 1 --2- 172.21.15.52:0/4062879871 >> [v2:172.21.15.105:6838/500348151,v1:172.21.15.105:6839/500348151] conn(0x55fdb745fbe0 0x55fdb7462000 crc :-1 s=READY pgs=90 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:28:52.219 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:52.229+0000 7fce6a316b80 1 --2- 172.21.15.52:0/4062879871 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55fdb723ae40 0x55fdb73cb1b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:28:52.219 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:52.229+0000 7fce6a316b80 1 --2- 172.21.15.52:0/4062879871 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55fdb73b8f80 0x55fdb73fbdd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:28:52.219 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:52.229+0000 7fce6a316b80 1 -- 172.21.15.52:0/4062879871 >> 172.21.15.52:0/4062879871 conn(0x55fdb7295af0 msgr2=0x55fdb73c22a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:28:52.219 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:52.229+0000 7fce3cff9700 0 client.25841 ms_handle_reset on v2:172.21.15.105:6838/500348151 2024-05-27T07:28:52.220 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:52.229+0000 7fce6a316b80 1 -- 172.21.15.52:0/4062879871 shutdown_connections 2024-05-27T07:28:52.220 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:52.230+0000 7fce6a316b80 1 -- 172.21.15.52:0/4062879871 wait complete. 2024-05-27T07:28:52.220 INFO:teuthology.orchestra.run.smithi052.stdout:{ 2024-05-27T07:28:52.220 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "scrub active (3799 inodes in the stack)", 2024-05-27T07:28:52.220 INFO:teuthology.orchestra.run.smithi052.stdout: "scrubs": { 2024-05-27T07:28:52.220 INFO:teuthology.orchestra.run.smithi052.stdout: "c250185e-7541-4646-9390-8b76d0627284": { 2024-05-27T07:28:52.220 INFO:teuthology.orchestra.run.smithi052.stdout: "path": "/", 2024-05-27T07:28:52.220 INFO:teuthology.orchestra.run.smithi052.stdout: "tag": "c250185e-7541-4646-9390-8b76d0627284", 2024-05-27T07:28:52.221 INFO:teuthology.orchestra.run.smithi052.stdout: "options": "recursive,force" 2024-05-27T07:28:52.221 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-05-27T07:28:52.221 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-05-27T07:28:52.221 INFO:teuthology.orchestra.run.smithi052.stdout:} 2024-05-27T07:28:52.222 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:52.230+0000 7fce6a316b80 20 client.25841 trim_cache size 0 max 16384 2024-05-27T07:28:52.222 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:52.231+0000 7fce5b901700 1 -- 172.21.15.52:0/256029496 >> [v2:172.21.15.52:6800/1822318384,v1:172.21.15.52:6801/1822318384] conn(0x7fce400b2ad0 msgr2=0x7fce400b4fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:28:52.222 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:52.231+0000 7fce5b901700 1 --2- 172.21.15.52:0/256029496 >> [v2:172.21.15.52:6800/1822318384,v1:172.21.15.52:6801/1822318384] conn(0x7fce400b2ad0 0x7fce400b4fe0 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x55fdb73b01b0 tx=0x7fce4400d550 comp rx=0 tx=0).stop 2024-05-27T07:28:52.222 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:52.231+0000 7fce5b901700 1 -- 172.21.15.52:0/256029496 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fce54112b80 msgr2=0x7fce541ad250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:28:52.222 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:52.231+0000 7fce5b901700 1 --2- 172.21.15.52:0/256029496 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fce54112b80 0x7fce541ad250 secure :-1 s=READY pgs=595 cs=0 l=1 rev1=1 crypto rx=0x7fce4c009f90 tx=0x7fce4c00cdb0 comp rx=0 tx=0).stop 2024-05-27T07:28:52.222 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:52.232+0000 7fce5b901700 1 -- 172.21.15.52:0/256029496 shutdown_connections 2024-05-27T07:28:52.223 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:52.232+0000 7fce5b901700 1 --2- 172.21.15.52:0/256029496 >> [v2:172.21.15.52:6800/1822318384,v1:172.21.15.52:6801/1822318384] conn(0x7fce400b2ad0 0x7fce400b4fe0 unknown :-1 s=CLOSED pgs=98 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:28:52.223 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:52.232+0000 7fce5b901700 1 --2- 172.21.15.52:0/256029496 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fce54112b80 0x7fce541ad250 unknown :-1 s=CLOSED pgs=595 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:28:52.223 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:52.232+0000 7fce5b901700 1 --2- 172.21.15.52:0/256029496 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fce5410cb40 0x7fce541a21d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:28:52.223 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:52.232+0000 7fce5b901700 1 --2- 172.21.15.52:0/256029496 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fce5410d490 0x7fce541a2710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:28:52.223 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:52.232+0000 7fce5b901700 1 -- 172.21.15.52:0/256029496 >> 172.21.15.52:0/256029496 conn(0x7fce54070780 msgr2=0x7fce5410e640 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:28:52.223 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:52.232+0000 7fce5b901700 1 -- 172.21.15.52:0/256029496 shutdown_connections 2024-05-27T07:28:52.223 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:28:52.233+0000 7fce5b901700 1 -- 172.21.15.52:0/256029496 wait complete. 2024-05-27T07:28:52.321 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:28:52 smithi052 ceph-mon[31955]: pgmap v95: 129 pgs: 129 active+clean; 142 MiB data, 731 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 MiB/s rd, 5.0 MiB/s wr, 644 op/s 2024-05-27T07:28:52.321 INFO:journalctl@ceph.mds.g.smithi052.stdout:May 27 07:28:52 smithi052 podman[103915]: 2024-05-27 07:28:52.198587567 +0000 UTC m=+2.876303602 container init 881cedb8d2465d95bc3a2989662221d785f4982234a579c062b0e70889cb4b9a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-g, GIT_CLEAN=True, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240521, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD) 2024-05-27T07:28:52.321 INFO:journalctl@ceph.mds.g.smithi052.stdout:May 27 07:28:52 smithi052 podman[103915]: 2024-05-27 07:28:52.203986553 +0000 UTC m=+2.881702582 container start 881cedb8d2465d95bc3a2989662221d785f4982234a579c062b0e70889cb4b9a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-g, org.label-schema.build-date=20240521, GIT_CLEAN=True, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, ceph=True, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0) 2024-05-27T07:28:52.321 INFO:journalctl@ceph.mds.g.smithi052.stdout:May 27 07:28:52 smithi052 ceph-mds[104015]: -- 172.21.15.52:0/2973239822 <== mon.1 v2:172.21.15.173:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55efb7cb9a40 con 0x55efb7d01180 2024-05-27T07:28:52.321 INFO:journalctl@ceph.mds.g.smithi052.stdout:May 27 07:28:52 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-g[103978]: starting mds.g at 2024-05-27T07:28:52.321 INFO:journalctl@ceph.mds.g.smithi052.stdout:May 27 07:28:52 smithi052 bash[103915]: 881cedb8d2465d95bc3a2989662221d785f4982234a579c062b0e70889cb4b9a 2024-05-27T07:28:52.321 INFO:journalctl@ceph.mds.g.smithi052.stdout:May 27 07:28:52 smithi052 systemd[1]: Started Ceph mds.g for 73df2636-1bf8-11ef-bc9b-c7b262605968. 2024-05-27T07:28:52.439 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:28:52 smithi105 ceph-mon[35977]: pgmap v95: 129 pgs: 129 active+clean; 142 MiB data, 731 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 MiB/s rd, 5.0 MiB/s wr, 644 op/s 2024-05-27T07:28:52.601 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:28:52 smithi173 ceph-mon[37000]: pgmap v95: 129 pgs: 129 active+clean; 142 MiB data, 731 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 MiB/s rd, 5.0 MiB/s wr, 644 op/s 2024-05-27T07:28:52.621 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:28:52.631+0000 7fb7297fa640 1 -- 172.21.15.52:0/780724580 <== mds.0 v2:172.21.15.105:6838/500348151 1 ==== command_reply(tid 0: 0 ) ==== 8+0+25 (crc 0 0 0) 0x55bcd9fd7360 con 0x55bcd9c5c1c0 2024-05-27T07:28:52.621 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:28:52.631+0000 7fb7297fa640 10 client.25823 handle_command_reply: tid=0 2024-05-27T07:28:52.621 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:28:52.631+0000 7fb74ebf0740 1 client.25823 shutdown 2024-05-27T07:28:52.621 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:28:52.631+0000 7fb74ebf0740 1 -- 172.21.15.52:0/780724580 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55bcd9ca74d0 msgr2=0x55bcda0e6370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:28:52.621 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:28:52.631+0000 7fb74ebf0740 1 --2- 172.21.15.52:0/780724580 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55bcd9ca74d0 0x55bcda0e6370 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7fb7300a5c30 tx=0x7fb7300e0050 comp rx=0 tx=0).stop 2024-05-27T07:28:52.621 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:28:52.631+0000 7fb74ebf0740 1 -- 172.21.15.52:0/780724580 shutdown_connections 2024-05-27T07:28:52.621 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:28:52.631+0000 7fb74ebf0740 1 --2- 172.21.15.52:0/780724580 >> [v2:172.21.15.105:6838/500348151,v1:172.21.15.105:6839/500348151] conn(0x55bcd9c5c1c0 0x55bcda0ec750 crc :-1 s=READY pgs=88 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:28:52.621 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:28:52.631+0000 7fb74ebf0740 1 --2- 172.21.15.52:0/780724580 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55bcd9cc81b0 0x55bcda0e68b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:28:52.621 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:28:52.631+0000 7fb74ebf0740 1 --2- 172.21.15.52:0/780724580 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55bcd9ca74d0 0x55bcda0e6370 unknown :-1 s=CLOSED pgs=109 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:28:52.621 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:28:52.631+0000 7fb7297fa640 0 client.25823 ms_handle_reset on v2:172.21.15.105:6838/500348151 2024-05-27T07:28:52.622 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:28:52.631+0000 7fb74ebf0740 1 --2- 172.21.15.52:0/780724580 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55bcd9c9f2f0 0x55bcda0ea010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:28:52.622 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:28:52.631+0000 7fb74ebf0740 1 -- 172.21.15.52:0/780724580 >> 172.21.15.52:0/780724580 conn(0x55bcd9cbc300 msgr2=0x55bcda06b010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:28:52.622 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:28:52.631+0000 7fb74ebf0740 1 -- 172.21.15.52:0/780724580 shutdown_connections 2024-05-27T07:28:52.622 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:28:52.631+0000 7fb74ebf0740 1 -- 172.21.15.52:0/780724580 wait complete. 2024-05-27T07:28:52.622 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:28:52.632+0000 7fb74ebf0740 20 client.25823 trim_cache size 0 max 16384 2024-05-27T07:28:52.622 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stdout:{ 2024-05-27T07:28:52.623 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stdout: "return_code": 0 2024-05-27T07:28:52.623 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stdout:} 2024-05-27T07:28:52.623 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:28:52.632+0000 7fb747d40640 1 -- 172.21.15.52:0/1871827443 >> [v2:172.21.15.52:6800/1822318384,v1:172.21.15.52:6801/1822318384] conn(0x7fb72c0b2a50 msgr2=0x7fb72c0b4f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:28:52.623 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:28:52.632+0000 7fb747d40640 1 --2- 172.21.15.52:0/1871827443 >> [v2:172.21.15.52:6800/1822318384,v1:172.21.15.52:6801/1822318384] conn(0x7fb72c0b2a50 0x7fb72c0b4f60 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x55bcda06a780 tx=0x7fb73c005f90 comp rx=0 tx=0).stop 2024-05-27T07:28:52.623 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:28:52.633+0000 7fb747d40640 1 -- 172.21.15.52:0/1871827443 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fb740109cc0 msgr2=0x7fb7401a0af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:28:52.624 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:28:52.633+0000 7fb747d40640 1 --2- 172.21.15.52:0/1871827443 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fb740109cc0 0x7fb7401a0af0 secure :-1 s=READY pgs=594 cs=0 l=1 rev1=1 crypto rx=0x7fb73000e9f0 tx=0x7fb73000eeb0 comp rx=0 tx=0).stop 2024-05-27T07:28:52.624 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:28:52.633+0000 7fb747d40640 1 -- 172.21.15.52:0/1871827443 shutdown_connections 2024-05-27T07:28:52.624 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:28:52.633+0000 7fb747d40640 1 --2- 172.21.15.52:0/1871827443 >> [v2:172.21.15.52:6800/1822318384,v1:172.21.15.52:6801/1822318384] conn(0x7fb72c0b2a50 0x7fb72c0b4f60 secure :-1 s=CLOSED pgs=97 cs=0 l=1 rev1=1 crypto rx=0x55bcda06a780 tx=0x7fb73c005f90 comp rx=0 tx=0).stop 2024-05-27T07:28:52.624 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:28:52.633+0000 7fb747d40640 1 --2- 172.21.15.52:0/1871827443 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fb74010a160 0x7fb74019a730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:28:52.624 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:28:52.633+0000 7fb747d40640 1 --2- 172.21.15.52:0/1871827443 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fb740109cc0 0x7fb7401a0af0 unknown :-1 s=CLOSED pgs=594 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:28:52.624 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:28:52.633+0000 7fb747d40640 1 --2- 172.21.15.52:0/1871827443 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fb740104410 0x7fb7401a05b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:28:52.624 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:28:52.633+0000 7fb747d40640 1 -- 172.21.15.52:0/1871827443 >> 172.21.15.52:0/1871827443 conn(0x7fb7400793c0 msgr2=0x7fb740100fd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:28:52.624 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:28:52.633+0000 7fb747d40640 1 -- 172.21.15.52:0/1871827443 shutdown_connections 2024-05-27T07:28:52.624 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:28:52.633+0000 7fb747d40640 1 -- 172.21.15.52:0/1871827443 wait complete. 2024-05-27T07:28:53.124 INFO:tasks.cephfs.filesystem:scrub status for tag:c250185e-7541-4646-9390-8b76d0627284 - {'path': '/', 'tag': 'c250185e-7541-4646-9390-8b76d0627284', 'options': 'recursive,force'} 2024-05-27T07:28:53.472 INFO:journalctl@ceph.mds.f.smithi173.stdout:May 27 07:28:53 smithi173 systemd[1]: ceph-73df2636-1bf8-11ef-bc9b-c7b262605968@mds.f.service: Scheduled restart job, restart counter is at 4. 2024-05-27T07:28:53.472 INFO:journalctl@ceph.mds.f.smithi173.stdout:May 27 07:28:53 smithi173 systemd[1]: Stopped Ceph mds.f for 73df2636-1bf8-11ef-bc9b-c7b262605968. 2024-05-27T07:28:53.472 INFO:journalctl@ceph.mds.f.smithi173.stdout:May 27 07:28:53 smithi173 systemd[1]: Starting Ceph mds.f for 73df2636-1bf8-11ef-bc9b-c7b262605968... 2024-05-27T07:28:53.729 INFO:journalctl@ceph.mds.d.smithi052.stdout:May 27 07:28:53 smithi052 systemd[1]: ceph-73df2636-1bf8-11ef-bc9b-c7b262605968@mds.d.service: Failed with result 'exit-code'. 2024-05-27T07:28:53.729 INFO:journalctl@ceph.mds.d.smithi052.stdout:May 27 07:28:53 smithi052 systemd[1]: ceph-73df2636-1bf8-11ef-bc9b-c7b262605968@mds.d.service: Consumed 1.051s CPU time. 2024-05-27T07:28:53.851 INFO:journalctl@ceph.mds.f.smithi173.stdout:May 27 07:28:53 smithi173 podman[72843]: 2024-05-27 07:28:53.389373421 +0000 UTC m=+0.031093453 image pull e36ea4fc8a5807e8d2701370e350a0de3856538797f7fba64cfce7e344bd6799 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6 2024-05-27T07:28:54.431 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:28:54 smithi173 ceph-mon[37000]: pgmap v96: 129 pgs: 129 active+clean; 141 MiB data, 726 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 MiB/s rd, 5.3 MiB/s wr, 815 op/s 2024-05-27T07:28:54.432 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:28:54 smithi173 ceph-mon[37000]: mds.? [v2:172.21.15.52:6836/3590696676,v1:172.21.15.52:6837/3590696676] up:boot 2024-05-27T07:28:54.432 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:28:54 smithi173 ceph-mon[37000]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-05-27T07:28:54.432 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:28:54 smithi173 ceph-mon[37000]: from='mgr.15333 172.21.15.52:0/416389082' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2024-05-27T07:28:54.432 INFO:journalctl@ceph.mds.f.smithi173.stdout:May 27 07:28:54 smithi173 podman[72843]: 2024-05-27 07:28:54.331921666 +0000 UTC m=+0.973641680 container create dc79a0027d89b0f5987c345c7699586f7204bc12c307c5f63fddce46dcb627e7 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-f, ceph=True, org.label-schema.license=GPLv2, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240521, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS) 2024-05-27T07:28:54.439 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:28:54 smithi105 ceph-mon[35977]: pgmap v96: 129 pgs: 129 active+clean; 141 MiB data, 726 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 MiB/s rd, 5.3 MiB/s wr, 815 op/s 2024-05-27T07:28:54.439 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:28:54 smithi105 ceph-mon[35977]: mds.? [v2:172.21.15.52:6836/3590696676,v1:172.21.15.52:6837/3590696676] up:boot 2024-05-27T07:28:54.439 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:28:54 smithi105 ceph-mon[35977]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-05-27T07:28:54.439 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:28:54 smithi105 ceph-mon[35977]: from='mgr.15333 172.21.15.52:0/416389082' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2024-05-27T07:28:54.479 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:28:54 smithi052 ceph-mon[31955]: pgmap v96: 129 pgs: 129 active+clean; 141 MiB data, 726 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 MiB/s rd, 5.3 MiB/s wr, 815 op/s 2024-05-27T07:28:54.479 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:28:54 smithi052 ceph-mon[31955]: mds.? [v2:172.21.15.52:6836/3590696676,v1:172.21.15.52:6837/3590696676] up:boot 2024-05-27T07:28:54.479 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:28:54 smithi052 ceph-mon[31955]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-05-27T07:28:54.479 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:28:54 smithi052 ceph-mon[31955]: from='mgr.15333 172.21.15.52:0/416389082' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2024-05-27T07:28:55.205 INFO:tasks.workunit.client.0.smithi052.stdout: 40 534911 266 379540 317 317237 949 2024-05-27T07:28:56.101 INFO:journalctl@ceph.mds.f.smithi173.stdout:May 27 07:28:55 smithi173 podman[72843]: 2024-05-27 07:28:55.751516475 +0000 UTC m=+2.393236505 container init dc79a0027d89b0f5987c345c7699586f7204bc12c307c5f63fddce46dcb627e7 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-f, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, io.buildah.version=1.33.5, GIT_CLEAN=True, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240521, GIT_BRANCH=HEAD) 2024-05-27T07:28:56.102 INFO:journalctl@ceph.mds.f.smithi173.stdout:May 27 07:28:55 smithi173 podman[72843]: 2024-05-27 07:28:55.754852816 +0000 UTC m=+2.396572833 container start dc79a0027d89b0f5987c345c7699586f7204bc12c307c5f63fddce46dcb627e7 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-f, ceph=True, GIT_BRANCH=HEAD, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, GIT_CLEAN=True, org.label-schema.build-date=20240521, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image) 2024-05-27T07:28:56.102 INFO:journalctl@ceph.mds.f.smithi173.stdout:May 27 07:28:55 smithi173 ceph-mds[72864]: -- 172.21.15.173:0/2041132807 <== mon.1 v2:172.21.15.173:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x560040917a40 con 0x56004095f600 2024-05-27T07:28:56.102 INFO:journalctl@ceph.mds.f.smithi173.stdout:May 27 07:28:55 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-f[72857]: starting mds.f at 2024-05-27T07:28:56.439 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:28:56 smithi105 ceph-mon[35977]: pgmap v97: 129 pgs: 129 active+clean; 134 MiB data, 766 MiB used, 1.0 TiB / 1.0 TiB avail; 2.2 MiB/s rd, 4.9 MiB/s wr, 1.39k op/s 2024-05-27T07:28:56.439 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:28:56 smithi105 ceph-mon[35977]: daemon mds.f restarted 2024-05-27T07:28:56.480 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:28:56 smithi052 ceph-mon[31955]: pgmap v97: 129 pgs: 129 active+clean; 134 MiB data, 766 MiB used, 1.0 TiB / 1.0 TiB avail; 2.2 MiB/s rd, 4.9 MiB/s wr, 1.39k op/s 2024-05-27T07:28:56.480 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:28:56 smithi052 ceph-mon[31955]: daemon mds.f restarted 2024-05-27T07:28:56.602 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:28:56 smithi173 ceph-mon[37000]: pgmap v97: 129 pgs: 129 active+clean; 134 MiB data, 766 MiB used, 1.0 TiB / 1.0 TiB avail; 2.2 MiB/s rd, 4.9 MiB/s wr, 1.39k op/s 2024-05-27T07:28:56.602 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:28:56 smithi173 ceph-mon[37000]: daemon mds.f restarted 2024-05-27T07:28:56.602 INFO:journalctl@ceph.mds.f.smithi173.stdout:May 27 07:28:56 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-f[72857]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: In function 'std::set SnapClient::filter(const std::set&) const' thread 7faaac40d700 time 2024-05-27T07:28:56.286064+0000 2024-05-27T07:28:56.602 INFO:journalctl@ceph.mds.f.smithi173.stdout:May 27 07:28:56 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-f[72857]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: 197: FAILED ceph_assert(cached_version > 0) 2024-05-27T07:28:56.602 INFO:journalctl@ceph.mds.f.smithi173.stdout:May 27 07:28:56 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-f[72857]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:28:56.603 INFO:journalctl@ceph.mds.f.smithi173.stdout:May 27 07:28:56 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-f[72857]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x127) [0x7faabec52ba6] 2024-05-27T07:28:56.603 INFO:journalctl@ceph.mds.f.smithi173.stdout:May 27 07:28:56 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-f[72857]: 2: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7faabec52dd1] 2024-05-27T07:28:56.603 INFO:journalctl@ceph.mds.f.smithi173.stdout:May 27 07:28:56 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-f[72857]: 3: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x56003ef6e23d] 2024-05-27T07:28:56.603 INFO:journalctl@ceph.mds.f.smithi173.stdout:May 27 07:28:56 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-f[72857]: 4: (SnapRealm::build_snap_set() const+0x323) [0x56003ef59971] 2024-05-27T07:28:56.603 INFO:journalctl@ceph.mds.f.smithi173.stdout:May 27 07:28:56 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-f[72857]: 5: (SnapRealm::check_cache() const+0x13e) [0x56003ef57fa0] 2024-05-27T07:28:56.603 INFO:journalctl@ceph.mds.f.smithi173.stdout:May 27 07:28:56 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-f[72857]: 6: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x56003ef58651] 2024-05-27T07:28:56.603 INFO:journalctl@ceph.mds.f.smithi173.stdout:May 27 07:28:56 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-f[72857]: 7: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x56003eeed347] 2024-05-27T07:28:56.603 INFO:journalctl@ceph.mds.f.smithi173.stdout:May 27 07:28:56 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-f[72857]: 8: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x56003ebe987e] 2024-05-27T07:28:56.603 INFO:journalctl@ceph.mds.f.smithi173.stdout:May 27 07:28:56 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-f[72857]: 9: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x279d) [0x56003ebeef2f] 2024-05-27T07:28:56.603 INFO:journalctl@ceph.mds.f.smithi173.stdout:May 27 07:28:56 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-f[72857]: 10: (EUpdate::replay(MDSRank*)+0x63) [0x56003ebf9969] 2024-05-27T07:28:56.603 INFO:journalctl@ceph.mds.f.smithi173.stdout:May 27 07:28:56 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-f[72857]: 11: (MDLog::_replay_thread()+0x1f38) [0x56003ef9248c] 2024-05-27T07:28:56.603 INFO:journalctl@ceph.mds.f.smithi173.stdout:May 27 07:28:56 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-f[72857]: 12: (MDLog::ReplayThread::entry()+0x11) [0x56003ef9b9cb] 2024-05-27T07:28:56.603 INFO:journalctl@ceph.mds.f.smithi173.stdout:May 27 07:28:56 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-f[72857]: 13: (Thread::entry_wrapper()+0x43) [0x7faabec2172d] 2024-05-27T07:28:56.603 INFO:journalctl@ceph.mds.f.smithi173.stdout:May 27 07:28:56 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-f[72857]: 14: (Thread::_entry_func(void*)+0xd) [0x7faabec21749] 2024-05-27T07:28:56.604 INFO:journalctl@ceph.mds.f.smithi173.stdout:May 27 07:28:56 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-f[72857]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7faabd7c51ca] 2024-05-27T07:28:56.604 INFO:journalctl@ceph.mds.f.smithi173.stdout:May 27 07:28:56 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-f[72857]: 16: clone() 2024-05-27T07:28:56.604 INFO:journalctl@ceph.mds.f.smithi173.stdout:May 27 07:28:56 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-f[72857]: *** Caught signal (Aborted) ** 2024-05-27T07:28:56.604 INFO:journalctl@ceph.mds.f.smithi173.stdout:May 27 07:28:56 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-f[72857]: in thread 7faaac40d700 thread_name:md_log_replay 2024-05-27T07:28:56.604 INFO:journalctl@ceph.mds.f.smithi173.stdout:May 27 07:28:56 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-f[72857]: 2024-05-27T07:28:56.286+0000 7faaac40d700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: In function 'std::set SnapClient::filter(const std::set&) const' thread 7faaac40d700 time 2024-05-27T07:28:56.286064+0000 2024-05-27T07:28:56.604 INFO:journalctl@ceph.mds.f.smithi173.stdout:May 27 07:28:56 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-f[72857]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: 197: FAILED ceph_assert(cached_version > 0) 2024-05-27T07:28:56.604 INFO:journalctl@ceph.mds.f.smithi173.stdout:May 27 07:28:56 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-f[72857]: 2024-05-27T07:28:56.604 INFO:journalctl@ceph.mds.f.smithi173.stdout:May 27 07:28:56 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-f[72857]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:28:56.604 INFO:journalctl@ceph.mds.f.smithi173.stdout:May 27 07:28:56 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-f[72857]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x127) [0x7faabec52ba6] 2024-05-27T07:28:56.604 INFO:journalctl@ceph.mds.f.smithi173.stdout:May 27 07:28:56 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-f[72857]: 2: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7faabec52dd1] 2024-05-27T07:28:56.604 INFO:journalctl@ceph.mds.f.smithi173.stdout:May 27 07:28:56 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-f[72857]: 3: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x56003ef6e23d] 2024-05-27T07:28:56.604 INFO:journalctl@ceph.mds.f.smithi173.stdout:May 27 07:28:56 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-f[72857]: 4: (SnapRealm::build_snap_set() const+0x323) [0x56003ef59971] 2024-05-27T07:28:56.605 INFO:journalctl@ceph.mds.f.smithi173.stdout:May 27 07:28:56 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-f[72857]: 5: (SnapRealm::check_cache() const+0x13e) [0x56003ef57fa0] 2024-05-27T07:28:56.605 INFO:journalctl@ceph.mds.f.smithi173.stdout:May 27 07:28:56 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-f[72857]: 6: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x56003ef58651] 2024-05-27T07:28:56.605 INFO:journalctl@ceph.mds.f.smithi173.stdout:May 27 07:28:56 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-f[72857]: 7: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x56003eeed347] 2024-05-27T07:28:56.605 INFO:journalctl@ceph.mds.f.smithi173.stdout:May 27 07:28:56 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-f[72857]: 8: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x56003ebe987e] 2024-05-27T07:28:56.605 INFO:journalctl@ceph.mds.f.smithi173.stdout:May 27 07:28:56 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-f[72857]: 9: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x279d) [0x56003ebeef2f] 2024-05-27T07:28:56.605 INFO:journalctl@ceph.mds.f.smithi173.stdout:May 27 07:28:56 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-f[72857]: 10: (EUpdate::replay(MDSRank*)+0x63) [0x56003ebf9969] 2024-05-27T07:28:56.605 INFO:journalctl@ceph.mds.f.smithi173.stdout:May 27 07:28:56 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-f[72857]: 11: (MDLog::_replay_thread()+0x1f38) [0x56003ef9248c] 2024-05-27T07:28:56.605 INFO:journalctl@ceph.mds.f.smithi173.stdout:May 27 07:28:56 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-f[72857]: 12: (MDLog::ReplayThread::entry()+0x11) [0x56003ef9b9cb] 2024-05-27T07:28:56.605 INFO:journalctl@ceph.mds.f.smithi173.stdout:May 27 07:28:56 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-f[72857]: 13: (Thread::entry_wrapper()+0x43) [0x7faabec2172d] 2024-05-27T07:28:56.605 INFO:journalctl@ceph.mds.f.smithi173.stdout:May 27 07:28:56 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-f[72857]: 14: (Thread::_entry_func(void*)+0xd) [0x7faabec21749] 2024-05-27T07:28:56.605 INFO:journalctl@ceph.mds.f.smithi173.stdout:May 27 07:28:56 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-f[72857]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7faabd7c51ca] 2024-05-27T07:28:56.605 INFO:journalctl@ceph.mds.f.smithi173.stdout:May 27 07:28:56 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-f[72857]: 16: clone() 2024-05-27T07:28:56.605 INFO:journalctl@ceph.mds.f.smithi173.stdout:May 27 07:28:56 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-f[72857]: 2024-05-27T07:28:56.605 INFO:journalctl@ceph.mds.f.smithi173.stdout:May 27 07:28:56 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-f[72857]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:28:56.605 INFO:journalctl@ceph.mds.f.smithi173.stdout:May 27 07:28:56 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-f[72857]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7faabd7cfd20] 2024-05-27T07:28:56.606 INFO:journalctl@ceph.mds.f.smithi173.stdout:May 27 07:28:56 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-f[72857]: 2: gsignal() 2024-05-27T07:28:56.606 INFO:journalctl@ceph.mds.f.smithi173.stdout:May 27 07:28:56 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-f[72857]: 3: abort() 2024-05-27T07:28:56.606 INFO:journalctl@ceph.mds.f.smithi173.stdout:May 27 07:28:56 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-f[72857]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x26e) [0x7faabec52ced] 2024-05-27T07:28:56.606 INFO:journalctl@ceph.mds.f.smithi173.stdout:May 27 07:28:56 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-f[72857]: 5: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7faabec52dd1] 2024-05-27T07:28:56.607 INFO:journalctl@ceph.mds.f.smithi173.stdout:May 27 07:28:56 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-f[72857]: 6: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x56003ef6e23d] 2024-05-27T07:28:56.607 INFO:journalctl@ceph.mds.f.smithi173.stdout:May 27 07:28:56 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-f[72857]: 7: (SnapRealm::build_snap_set() const+0x323) [0x56003ef59971] 2024-05-27T07:28:56.607 INFO:journalctl@ceph.mds.f.smithi173.stdout:May 27 07:28:56 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-f[72857]: 8: (SnapRealm::check_cache() const+0x13e) [0x56003ef57fa0] 2024-05-27T07:28:56.607 INFO:journalctl@ceph.mds.f.smithi173.stdout:May 27 07:28:56 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-f[72857]: 9: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x56003ef58651] 2024-05-27T07:28:56.607 INFO:journalctl@ceph.mds.f.smithi173.stdout:May 27 07:28:56 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-f[72857]: 10: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x56003eeed347] 2024-05-27T07:28:56.607 INFO:journalctl@ceph.mds.f.smithi173.stdout:May 27 07:28:56 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-f[72857]: 11: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x56003ebe987e] 2024-05-27T07:28:56.607 INFO:journalctl@ceph.mds.f.smithi173.stdout:May 27 07:28:56 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-f[72857]: 12: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x279d) [0x56003ebeef2f] 2024-05-27T07:28:56.607 INFO:journalctl@ceph.mds.f.smithi173.stdout:May 27 07:28:56 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-f[72857]: 13: (EUpdate::replay(MDSRank*)+0x63) [0x56003ebf9969] 2024-05-27T07:28:56.607 INFO:journalctl@ceph.mds.f.smithi173.stdout:May 27 07:28:56 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-f[72857]: 14: (MDLog::_replay_thread()+0x1f38) [0x56003ef9248c] 2024-05-27T07:28:56.607 INFO:journalctl@ceph.mds.f.smithi173.stdout:May 27 07:28:56 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-f[72857]: 15: (MDLog::ReplayThread::entry()+0x11) [0x56003ef9b9cb] 2024-05-27T07:28:56.607 INFO:journalctl@ceph.mds.f.smithi173.stdout:May 27 07:28:56 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-f[72857]: 16: (Thread::entry_wrapper()+0x43) [0x7faabec2172d] 2024-05-27T07:28:56.607 INFO:journalctl@ceph.mds.f.smithi173.stdout:May 27 07:28:56 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-f[72857]: 17: (Thread::_entry_func(void*)+0xd) [0x7faabec21749] 2024-05-27T07:28:56.608 INFO:journalctl@ceph.mds.f.smithi173.stdout:May 27 07:28:56 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-f[72857]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7faabd7c51ca] 2024-05-27T07:28:56.608 INFO:journalctl@ceph.mds.f.smithi173.stdout:May 27 07:28:56 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-f[72857]: 19: clone() 2024-05-27T07:28:56.608 INFO:journalctl@ceph.mds.f.smithi173.stdout:May 27 07:28:56 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-f[72857]: 2024-05-27T07:28:56.288+0000 7faaac40d700 -1 *** Caught signal (Aborted) ** 2024-05-27T07:28:56.608 INFO:journalctl@ceph.mds.f.smithi173.stdout:May 27 07:28:56 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-f[72857]: in thread 7faaac40d700 thread_name:md_log_replay 2024-05-27T07:28:56.608 INFO:journalctl@ceph.mds.f.smithi173.stdout:May 27 07:28:56 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-f[72857]: 2024-05-27T07:28:56.608 INFO:journalctl@ceph.mds.f.smithi173.stdout:May 27 07:28:56 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-f[72857]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:28:56.608 INFO:journalctl@ceph.mds.f.smithi173.stdout:May 27 07:28:56 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-f[72857]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7faabd7cfd20] 2024-05-27T07:28:56.608 INFO:journalctl@ceph.mds.f.smithi173.stdout:May 27 07:28:56 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-f[72857]: 2: gsignal() 2024-05-27T07:28:56.608 INFO:journalctl@ceph.mds.f.smithi173.stdout:May 27 07:28:56 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-f[72857]: 3: abort() 2024-05-27T07:28:56.608 INFO:journalctl@ceph.mds.f.smithi173.stdout:May 27 07:28:56 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-f[72857]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x26e) [0x7faabec52ced] 2024-05-27T07:28:56.608 INFO:journalctl@ceph.mds.f.smithi173.stdout:May 27 07:28:56 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-f[72857]: 5: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7faabec52dd1] 2024-05-27T07:28:56.608 INFO:journalctl@ceph.mds.f.smithi173.stdout:May 27 07:28:56 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-f[72857]: 6: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x56003ef6e23d] 2024-05-27T07:28:56.609 INFO:journalctl@ceph.mds.f.smithi173.stdout:May 27 07:28:56 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-f[72857]: 7: (SnapRealm::build_snap_set() const+0x323) [0x56003ef59971] 2024-05-27T07:28:56.609 INFO:journalctl@ceph.mds.f.smithi173.stdout:May 27 07:28:56 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-f[72857]: 8: (SnapRealm::check_cache() const+0x13e) [0x56003ef57fa0] 2024-05-27T07:28:56.609 INFO:journalctl@ceph.mds.f.smithi173.stdout:May 27 07:28:56 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-f[72857]: 9: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x56003ef58651] 2024-05-27T07:28:56.609 INFO:journalctl@ceph.mds.f.smithi173.stdout:May 27 07:28:56 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-f[72857]: 10: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x56003eeed347] 2024-05-27T07:28:56.609 INFO:journalctl@ceph.mds.f.smithi173.stdout:May 27 07:28:56 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-f[72857]: 11: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x56003ebe987e] 2024-05-27T07:28:56.609 INFO:journalctl@ceph.mds.f.smithi173.stdout:May 27 07:28:56 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-f[72857]: 12: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x279d) [0x56003ebeef2f] 2024-05-27T07:28:56.609 INFO:journalctl@ceph.mds.f.smithi173.stdout:May 27 07:28:56 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-f[72857]: 13: (EUpdate::replay(MDSRank*)+0x63) [0x56003ebf9969] 2024-05-27T07:28:56.609 INFO:journalctl@ceph.mds.f.smithi173.stdout:May 27 07:28:56 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-f[72857]: 14: (MDLog::_replay_thread()+0x1f38) [0x56003ef9248c] 2024-05-27T07:28:56.609 INFO:journalctl@ceph.mds.f.smithi173.stdout:May 27 07:28:56 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-f[72857]: 15: (MDLog::ReplayThread::entry()+0x11) [0x56003ef9b9cb] 2024-05-27T07:28:56.609 INFO:journalctl@ceph.mds.f.smithi173.stdout:May 27 07:28:56 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-f[72857]: 16: (Thread::entry_wrapper()+0x43) [0x7faabec2172d] 2024-05-27T07:28:56.609 INFO:journalctl@ceph.mds.f.smithi173.stdout:May 27 07:28:56 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-f[72857]: 17: (Thread::_entry_func(void*)+0xd) [0x7faabec21749] 2024-05-27T07:28:56.609 INFO:journalctl@ceph.mds.f.smithi173.stdout:May 27 07:28:56 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-f[72857]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7faabd7c51ca] 2024-05-27T07:28:56.609 INFO:journalctl@ceph.mds.f.smithi173.stdout:May 27 07:28:56 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-f[72857]: 19: clone() 2024-05-27T07:28:56.609 INFO:journalctl@ceph.mds.f.smithi173.stdout:May 27 07:28:56 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-f[72857]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-05-27T07:28:56.609 INFO:journalctl@ceph.mds.f.smithi173.stdout:May 27 07:28:56 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-f[72857]: 2024-05-27T07:28:56.610 INFO:journalctl@ceph.mds.f.smithi173.stdout:May 27 07:28:56 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-f[72857]: -1> 2024-05-27T07:28:56.286+0000 7faaac40d700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: In function 'std::set SnapClient::filter(const std::set&) const' thread 7faaac40d700 time 2024-05-27T07:28:56.286064+0000 2024-05-27T07:28:56.610 INFO:journalctl@ceph.mds.f.smithi173.stdout:May 27 07:28:56 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-f[72857]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: 197: FAILED ceph_assert(cached_version > 0) 2024-05-27T07:28:56.610 INFO:journalctl@ceph.mds.f.smithi173.stdout:May 27 07:28:56 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-f[72857]: 2024-05-27T07:28:56.610 INFO:journalctl@ceph.mds.f.smithi173.stdout:May 27 07:28:56 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-f[72857]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:28:56.610 INFO:journalctl@ceph.mds.f.smithi173.stdout:May 27 07:28:56 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-f[72857]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x127) [0x7faabec52ba6] 2024-05-27T07:28:56.610 INFO:journalctl@ceph.mds.f.smithi173.stdout:May 27 07:28:56 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-f[72857]: 2: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7faabec52dd1] 2024-05-27T07:28:56.610 INFO:journalctl@ceph.mds.f.smithi173.stdout:May 27 07:28:56 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-f[72857]: 3: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x56003ef6e23d] 2024-05-27T07:28:56.610 INFO:journalctl@ceph.mds.f.smithi173.stdout:May 27 07:28:56 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-f[72857]: 4: (SnapRealm::build_snap_set() const+0x323) [0x56003ef59971] 2024-05-27T07:28:56.610 INFO:journalctl@ceph.mds.f.smithi173.stdout:May 27 07:28:56 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-f[72857]: 5: (SnapRealm::check_cache() const+0x13e) [0x56003ef57fa0] 2024-05-27T07:28:56.610 INFO:journalctl@ceph.mds.f.smithi173.stdout:May 27 07:28:56 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-f[72857]: 6: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x56003ef58651] 2024-05-27T07:28:56.610 INFO:journalctl@ceph.mds.f.smithi173.stdout:May 27 07:28:56 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-f[72857]: 7: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x56003eeed347] 2024-05-27T07:28:56.610 INFO:journalctl@ceph.mds.f.smithi173.stdout:May 27 07:28:56 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-f[72857]: 8: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x56003ebe987e] 2024-05-27T07:28:56.611 INFO:journalctl@ceph.mds.f.smithi173.stdout:May 27 07:28:56 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-f[72857]: 9: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x279d) [0x56003ebeef2f] 2024-05-27T07:28:56.611 INFO:journalctl@ceph.mds.f.smithi173.stdout:May 27 07:28:56 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-f[72857]: 10: (EUpdate::replay(MDSRank*)+0x63) [0x56003ebf9969] 2024-05-27T07:28:56.611 INFO:journalctl@ceph.mds.f.smithi173.stdout:May 27 07:28:56 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-f[72857]: 11: (MDLog::_replay_thread()+0x1f38) [0x56003ef9248c] 2024-05-27T07:28:56.611 INFO:journalctl@ceph.mds.f.smithi173.stdout:May 27 07:28:56 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-f[72857]: 12: (MDLog::ReplayThread::entry()+0x11) [0x56003ef9b9cb] 2024-05-27T07:28:56.611 INFO:journalctl@ceph.mds.f.smithi173.stdout:May 27 07:28:56 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-f[72857]: 13: (Thread::entry_wrapper()+0x43) [0x7faabec2172d] 2024-05-27T07:28:56.611 INFO:journalctl@ceph.mds.f.smithi173.stdout:May 27 07:28:56 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-f[72857]: 14: (Thread::_entry_func(void*)+0xd) [0x7faabec21749] 2024-05-27T07:28:56.611 INFO:journalctl@ceph.mds.f.smithi173.stdout:May 27 07:28:56 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-f[72857]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7faabd7c51ca] 2024-05-27T07:28:56.611 INFO:journalctl@ceph.mds.f.smithi173.stdout:May 27 07:28:56 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-f[72857]: 16: clone() 2024-05-27T07:28:56.611 INFO:journalctl@ceph.mds.f.smithi173.stdout:May 27 07:28:56 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-f[72857]: 2024-05-27T07:28:56.611 INFO:journalctl@ceph.mds.f.smithi173.stdout:May 27 07:28:56 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-f[72857]: 0> 2024-05-27T07:28:56.288+0000 7faaac40d700 -1 *** Caught signal (Aborted) ** 2024-05-27T07:28:56.611 INFO:journalctl@ceph.mds.f.smithi173.stdout:May 27 07:28:56 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-f[72857]: in thread 7faaac40d700 thread_name:md_log_replay 2024-05-27T07:28:56.611 INFO:journalctl@ceph.mds.f.smithi173.stdout:May 27 07:28:56 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-f[72857]: 2024-05-27T07:28:56.611 INFO:journalctl@ceph.mds.f.smithi173.stdout:May 27 07:28:56 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-f[72857]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:28:56.612 INFO:journalctl@ceph.mds.f.smithi173.stdout:May 27 07:28:56 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-f[72857]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7faabd7cfd20] 2024-05-27T07:28:56.612 INFO:journalctl@ceph.mds.f.smithi173.stdout:May 27 07:28:56 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-f[72857]: 2: gsignal() 2024-05-27T07:28:56.612 INFO:journalctl@ceph.mds.f.smithi173.stdout:May 27 07:28:56 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-f[72857]: 3: abort() 2024-05-27T07:28:56.612 INFO:journalctl@ceph.mds.f.smithi173.stdout:May 27 07:28:56 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-f[72857]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x26e) [0x7faabec52ced] 2024-05-27T07:28:56.612 INFO:journalctl@ceph.mds.f.smithi173.stdout:May 27 07:28:56 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-f[72857]: 5: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7faabec52dd1] 2024-05-27T07:28:56.612 INFO:journalctl@ceph.mds.f.smithi173.stdout:May 27 07:28:56 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-f[72857]: 6: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x56003ef6e23d] 2024-05-27T07:28:56.612 INFO:journalctl@ceph.mds.f.smithi173.stdout:May 27 07:28:56 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-f[72857]: 7: (SnapRealm::build_snap_set() const+0x323) [0x56003ef59971] 2024-05-27T07:28:56.612 INFO:journalctl@ceph.mds.f.smithi173.stdout:May 27 07:28:56 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-f[72857]: 8: (SnapRealm::check_cache() const+0x13e) [0x56003ef57fa0] 2024-05-27T07:28:56.612 INFO:journalctl@ceph.mds.f.smithi173.stdout:May 27 07:28:56 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-f[72857]: 9: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x56003ef58651] 2024-05-27T07:28:56.612 INFO:journalctl@ceph.mds.f.smithi173.stdout:May 27 07:28:56 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-f[72857]: 10: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x56003eeed347] 2024-05-27T07:28:56.612 INFO:journalctl@ceph.mds.f.smithi173.stdout:May 27 07:28:56 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-f[72857]: 11: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x56003ebe987e] 2024-05-27T07:28:56.612 INFO:journalctl@ceph.mds.f.smithi173.stdout:May 27 07:28:56 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-f[72857]: 12: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x279d) [0x56003ebeef2f] 2024-05-27T07:28:56.612 INFO:journalctl@ceph.mds.f.smithi173.stdout:May 27 07:28:56 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-f[72857]: 13: (EUpdate::replay(MDSRank*)+0x63) [0x56003ebf9969] 2024-05-27T07:28:56.612 INFO:journalctl@ceph.mds.f.smithi173.stdout:May 27 07:28:56 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-f[72857]: 14: (MDLog::_replay_thread()+0x1f38) [0x56003ef9248c] 2024-05-27T07:28:56.613 INFO:journalctl@ceph.mds.f.smithi173.stdout:May 27 07:28:56 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-f[72857]: 15: (MDLog::ReplayThread::entry()+0x11) [0x56003ef9b9cb] 2024-05-27T07:28:56.613 INFO:journalctl@ceph.mds.f.smithi173.stdout:May 27 07:28:56 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-f[72857]: 16: (Thread::entry_wrapper()+0x43) [0x7faabec2172d] 2024-05-27T07:28:56.613 INFO:journalctl@ceph.mds.f.smithi173.stdout:May 27 07:28:56 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-f[72857]: 17: (Thread::_entry_func(void*)+0xd) [0x7faabec21749] 2024-05-27T07:28:56.613 INFO:journalctl@ceph.mds.f.smithi173.stdout:May 27 07:28:56 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-f[72857]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7faabd7c51ca] 2024-05-27T07:28:56.613 INFO:journalctl@ceph.mds.f.smithi173.stdout:May 27 07:28:56 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-f[72857]: 19: clone() 2024-05-27T07:28:56.613 INFO:journalctl@ceph.mds.f.smithi173.stdout:May 27 07:28:56 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-f[72857]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-05-27T07:28:56.613 INFO:journalctl@ceph.mds.f.smithi173.stdout:May 27 07:28:56 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-f[72857]: 2024-05-27T07:28:56.613 INFO:journalctl@ceph.mds.f.smithi173.stdout:May 27 07:28:56 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-f[72857]: -1> 2024-05-27T07:28:56.286+0000 7faaac40d700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: In function 'std::set SnapClient::filter(const std::set&) const' thread 7faaac40d700 time 2024-05-27T07:28:56.286064+0000 2024-05-27T07:28:56.613 INFO:journalctl@ceph.mds.f.smithi173.stdout:May 27 07:28:56 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-f[72857]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: 197: FAILED ceph_assert(cached_version > 0) 2024-05-27T07:28:56.613 INFO:journalctl@ceph.mds.f.smithi173.stdout:May 27 07:28:56 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-f[72857]: 2024-05-27T07:28:56.613 INFO:journalctl@ceph.mds.f.smithi173.stdout:May 27 07:28:56 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-f[72857]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:28:56.613 INFO:journalctl@ceph.mds.f.smithi173.stdout:May 27 07:28:56 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-f[72857]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x127) [0x7faabec52ba6] 2024-05-27T07:28:56.614 INFO:journalctl@ceph.mds.f.smithi173.stdout:May 27 07:28:56 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-f[72857]: 2: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7faabec52dd1] 2024-05-27T07:28:56.614 INFO:journalctl@ceph.mds.f.smithi173.stdout:May 27 07:28:56 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-f[72857]: 3: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x56003ef6e23d] 2024-05-27T07:28:56.614 INFO:journalctl@ceph.mds.f.smithi173.stdout:May 27 07:28:56 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-f[72857]: 4: (SnapRealm::build_snap_set() const+0x323) [0x56003ef59971] 2024-05-27T07:28:56.614 INFO:journalctl@ceph.mds.f.smithi173.stdout:May 27 07:28:56 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-f[72857]: 5: (SnapRealm::check_cache() const+0x13e) [0x56003ef57fa0] 2024-05-27T07:28:56.614 INFO:journalctl@ceph.mds.f.smithi173.stdout:May 27 07:28:56 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-f[72857]: 6: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x56003ef58651] 2024-05-27T07:28:56.614 INFO:journalctl@ceph.mds.f.smithi173.stdout:May 27 07:28:56 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-f[72857]: 7: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x56003eeed347] 2024-05-27T07:28:56.614 INFO:journalctl@ceph.mds.f.smithi173.stdout:May 27 07:28:56 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-f[72857]: 8: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x56003ebe987e] 2024-05-27T07:28:56.614 INFO:journalctl@ceph.mds.f.smithi173.stdout:May 27 07:28:56 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-f[72857]: 9: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x279d) [0x56003ebeef2f] 2024-05-27T07:28:56.614 INFO:journalctl@ceph.mds.f.smithi173.stdout:May 27 07:28:56 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-f[72857]: 10: (EUpdate::replay(MDSRank*)+0x63) [0x56003ebf9969] 2024-05-27T07:28:56.614 INFO:journalctl@ceph.mds.f.smithi173.stdout:May 27 07:28:56 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-f[72857]: 11: (MDLog::_replay_thread()+0x1f38) [0x56003ef9248c] 2024-05-27T07:28:56.614 INFO:journalctl@ceph.mds.f.smithi173.stdout:May 27 07:28:56 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-f[72857]: 12: (MDLog::ReplayThread::entry()+0x11) [0x56003ef9b9cb] 2024-05-27T07:28:56.614 INFO:journalctl@ceph.mds.f.smithi173.stdout:May 27 07:28:56 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-f[72857]: 13: (Thread::entry_wrapper()+0x43) [0x7faabec2172d] 2024-05-27T07:28:56.614 INFO:journalctl@ceph.mds.f.smithi173.stdout:May 27 07:28:56 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-f[72857]: 14: (Thread::_entry_func(void*)+0xd) [0x7faabec21749] 2024-05-27T07:28:56.615 INFO:journalctl@ceph.mds.f.smithi173.stdout:May 27 07:28:56 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-f[72857]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7faabd7c51ca] 2024-05-27T07:28:56.615 INFO:journalctl@ceph.mds.f.smithi173.stdout:May 27 07:28:56 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-f[72857]: 16: clone() 2024-05-27T07:28:56.615 INFO:journalctl@ceph.mds.f.smithi173.stdout:May 27 07:28:56 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-f[72857]: 2024-05-27T07:28:56.615 INFO:journalctl@ceph.mds.f.smithi173.stdout:May 27 07:28:56 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-f[72857]: 0> 2024-05-27T07:28:56.288+0000 7faaac40d700 -1 *** Caught signal (Aborted) ** 2024-05-27T07:28:56.615 INFO:journalctl@ceph.mds.f.smithi173.stdout:May 27 07:28:56 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-f[72857]: in thread 7faaac40d700 thread_name:md_log_replay 2024-05-27T07:28:56.615 INFO:journalctl@ceph.mds.f.smithi173.stdout:May 27 07:28:56 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-f[72857]: 2024-05-27T07:28:56.615 INFO:journalctl@ceph.mds.f.smithi173.stdout:May 27 07:28:56 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-f[72857]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:28:56.615 INFO:journalctl@ceph.mds.f.smithi173.stdout:May 27 07:28:56 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-f[72857]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7faabd7cfd20] 2024-05-27T07:28:56.615 INFO:journalctl@ceph.mds.f.smithi173.stdout:May 27 07:28:56 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-f[72857]: 2: gsignal() 2024-05-27T07:28:56.615 INFO:journalctl@ceph.mds.f.smithi173.stdout:May 27 07:28:56 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-f[72857]: 3: abort() 2024-05-27T07:28:56.615 INFO:journalctl@ceph.mds.f.smithi173.stdout:May 27 07:28:56 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-f[72857]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x26e) [0x7faabec52ced] 2024-05-27T07:28:56.615 INFO:journalctl@ceph.mds.f.smithi173.stdout:May 27 07:28:56 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-f[72857]: 5: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7faabec52dd1] 2024-05-27T07:28:56.615 INFO:journalctl@ceph.mds.f.smithi173.stdout:May 27 07:28:56 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-f[72857]: 6: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x56003ef6e23d] 2024-05-27T07:28:56.616 INFO:journalctl@ceph.mds.f.smithi173.stdout:May 27 07:28:56 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-f[72857]: 7: (SnapRealm::build_snap_set() const+0x323) [0x56003ef59971] 2024-05-27T07:28:56.616 INFO:journalctl@ceph.mds.f.smithi173.stdout:May 27 07:28:56 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-f[72857]: 8: (SnapRealm::check_cache() const+0x13e) [0x56003ef57fa0] 2024-05-27T07:28:56.616 INFO:journalctl@ceph.mds.f.smithi173.stdout:May 27 07:28:56 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-f[72857]: 9: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x56003ef58651] 2024-05-27T07:28:56.616 INFO:journalctl@ceph.mds.f.smithi173.stdout:May 27 07:28:56 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-f[72857]: 10: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x56003eeed347] 2024-05-27T07:28:56.616 INFO:journalctl@ceph.mds.f.smithi173.stdout:May 27 07:28:56 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-f[72857]: 11: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x56003ebe987e] 2024-05-27T07:28:56.616 INFO:journalctl@ceph.mds.f.smithi173.stdout:May 27 07:28:56 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-f[72857]: 12: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x279d) [0x56003ebeef2f] 2024-05-27T07:28:56.616 INFO:journalctl@ceph.mds.f.smithi173.stdout:May 27 07:28:56 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-f[72857]: 13: (EUpdate::replay(MDSRank*)+0x63) [0x56003ebf9969] 2024-05-27T07:28:56.616 INFO:journalctl@ceph.mds.f.smithi173.stdout:May 27 07:28:56 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-f[72857]: 14: (MDLog::_replay_thread()+0x1f38) [0x56003ef9248c] 2024-05-27T07:28:56.616 INFO:journalctl@ceph.mds.f.smithi173.stdout:May 27 07:28:56 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-f[72857]: 15: (MDLog::ReplayThread::entry()+0x11) [0x56003ef9b9cb] 2024-05-27T07:28:56.616 INFO:journalctl@ceph.mds.f.smithi173.stdout:May 27 07:28:56 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-f[72857]: 16: (Thread::entry_wrapper()+0x43) [0x7faabec2172d] 2024-05-27T07:28:56.616 INFO:journalctl@ceph.mds.f.smithi173.stdout:May 27 07:28:56 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-f[72857]: 17: (Thread::_entry_func(void*)+0xd) [0x7faabec21749] 2024-05-27T07:28:56.616 INFO:journalctl@ceph.mds.f.smithi173.stdout:May 27 07:28:56 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-f[72857]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7faabd7c51ca] 2024-05-27T07:28:56.616 INFO:journalctl@ceph.mds.f.smithi173.stdout:May 27 07:28:56 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-f[72857]: 19: clone() 2024-05-27T07:28:56.617 INFO:journalctl@ceph.mds.f.smithi173.stdout:May 27 07:28:56 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-f[72857]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-05-27T07:28:56.617 INFO:journalctl@ceph.mds.f.smithi173.stdout:May 27 07:28:56 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-f[72857]: 2024-05-27T07:28:57.351 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:28:57 smithi173 ceph-mon[37000]: osdmap e97: 12 total, 12 up, 12 in 2024-05-27T07:28:57.351 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:28:57 smithi173 ceph-mon[37000]: mds.? [v2:172.21.15.173:6834/101717105,v1:172.21.15.173:6835/101717105] up:boot 2024-05-27T07:28:57.352 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:28:57 smithi173 ceph-mon[37000]: fsmap cephfs:5 4 up:standby-replay 3 up:standby 5 up:active 2024-05-27T07:28:57.352 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:28:57 smithi173 ceph-mon[37000]: from='mgr.15333 172.21.15.52:0/416389082' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2024-05-27T07:28:57.352 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:28:57 smithi173 ceph-mon[37000]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-05-27T07:28:57.352 INFO:journalctl@ceph.mds.f.smithi173.stdout:May 27 07:28:57 smithi173 bash[72843]: dc79a0027d89b0f5987c345c7699586f7204bc12c307c5f63fddce46dcb627e7 2024-05-27T07:28:57.352 INFO:journalctl@ceph.mds.f.smithi173.stdout:May 27 07:28:57 smithi173 systemd[1]: Started Ceph mds.f for 73df2636-1bf8-11ef-bc9b-c7b262605968. 2024-05-27T07:28:57.352 INFO:journalctl@ceph.mds.f.smithi173.stdout:May 27 07:28:57 smithi173 podman[72895]: 2024-05-27 07:28:57.165175264 +0000 UTC m=+0.848453242 container died dc79a0027d89b0f5987c345c7699586f7204bc12c307c5f63fddce46dcb627e7 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-f, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, ceph=True, io.buildah.version=1.33.5, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, org.label-schema.build-date=20240521, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD) 2024-05-27T07:28:57.439 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:28:57 smithi105 ceph-mon[35977]: osdmap e97: 12 total, 12 up, 12 in 2024-05-27T07:28:57.439 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:28:57 smithi105 ceph-mon[35977]: mds.? [v2:172.21.15.173:6834/101717105,v1:172.21.15.173:6835/101717105] up:boot 2024-05-27T07:28:57.439 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:28:57 smithi105 ceph-mon[35977]: fsmap cephfs:5 4 up:standby-replay 3 up:standby 5 up:active 2024-05-27T07:28:57.440 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:28:57 smithi105 ceph-mon[35977]: from='mgr.15333 172.21.15.52:0/416389082' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2024-05-27T07:28:57.440 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:28:57 smithi105 ceph-mon[35977]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-05-27T07:28:57.479 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:28:57 smithi052 ceph-mon[31955]: osdmap e97: 12 total, 12 up, 12 in 2024-05-27T07:28:57.479 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:28:57 smithi052 ceph-mon[31955]: mds.? [v2:172.21.15.173:6834/101717105,v1:172.21.15.173:6835/101717105] up:boot 2024-05-27T07:28:57.479 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:28:57 smithi052 ceph-mon[31955]: fsmap cephfs:5 4 up:standby-replay 3 up:standby 5 up:active 2024-05-27T07:28:57.479 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:28:57 smithi052 ceph-mon[31955]: from='mgr.15333 172.21.15.52:0/416389082' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2024-05-27T07:28:57.479 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:28:57 smithi052 ceph-mon[31955]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-05-27T07:28:58.406 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:28:58 smithi173 ceph-mon[37000]: pgmap v99: 129 pgs: 129 active+clean; 134 MiB data, 766 MiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 3.0 MiB/s wr, 1.35k op/s 2024-05-27T07:28:58.439 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:28:58 smithi105 ceph-mon[35977]: pgmap v99: 129 pgs: 129 active+clean; 134 MiB data, 766 MiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 3.0 MiB/s wr, 1.35k op/s 2024-05-27T07:28:58.479 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:28:58 smithi052 ceph-mon[31955]: pgmap v99: 129 pgs: 129 active+clean; 134 MiB data, 766 MiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 3.0 MiB/s wr, 1.35k op/s 2024-05-27T07:28:58.942 INFO:journalctl@ceph.mds.f.smithi173.stdout:May 27 07:28:58 smithi173 podman[72895]: 2024-05-27 07:28:58.701548778 +0000 UTC m=+2.384826733 container remove dc79a0027d89b0f5987c345c7699586f7204bc12c307c5f63fddce46dcb627e7 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-f, io.buildah.version=1.33.5, ceph=True, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240521, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image) 2024-05-27T07:28:58.943 INFO:journalctl@ceph.mds.f.smithi173.stdout:May 27 07:28:58 smithi173 systemd[1]: ceph-73df2636-1bf8-11ef-bc9b-c7b262605968@mds.f.service: Main process exited, code=exited, status=134/n/a 2024-05-27T07:28:59.351 INFO:journalctl@ceph.mds.f.smithi173.stdout:May 27 07:28:58 smithi173 systemd[1]: ceph-73df2636-1bf8-11ef-bc9b-c7b262605968@mds.f.service: Failed with result 'exit-code'. 2024-05-27T07:29:00.439 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:29:00 smithi105 ceph-mon[35977]: pgmap v100: 129 pgs: 129 active+clean; 161 MiB data, 822 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 5.3 MiB/s wr, 1.48k op/s 2024-05-27T07:29:00.479 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:29:00 smithi052 ceph-mon[31955]: pgmap v100: 129 pgs: 129 active+clean; 161 MiB data, 822 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 5.3 MiB/s wr, 1.48k op/s 2024-05-27T07:29:00.601 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:29:00 smithi173 ceph-mon[37000]: pgmap v100: 129 pgs: 129 active+clean; 161 MiB data, 822 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 5.3 MiB/s wr, 1.48k op/s 2024-05-27T07:29:01.987 INFO:journalctl@ceph.mds.l.smithi173.stdout:May 27 07:29:01 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-l[67429]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: In function 'std::set SnapClient::filter(const std::set&) const' thread 7f71be3fd700 time 2024-05-27T07:29:01.638714+0000 2024-05-27T07:29:01.987 INFO:journalctl@ceph.mds.l.smithi173.stdout:May 27 07:29:01 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-l[67429]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: 197: FAILED ceph_assert(cached_version > 0) 2024-05-27T07:29:01.987 INFO:journalctl@ceph.mds.l.smithi173.stdout:May 27 07:29:01 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-l[67429]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:29:01.987 INFO:journalctl@ceph.mds.l.smithi173.stdout:May 27 07:29:01 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-l[67429]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x127) [0x7f71d0c42ba6] 2024-05-27T07:29:01.987 INFO:journalctl@ceph.mds.l.smithi173.stdout:May 27 07:29:01 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-l[67429]: 2: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7f71d0c42dd1] 2024-05-27T07:29:01.987 INFO:journalctl@ceph.mds.l.smithi173.stdout:May 27 07:29:01 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-l[67429]: 3: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x56350a19f23d] 2024-05-27T07:29:01.987 INFO:journalctl@ceph.mds.l.smithi173.stdout:May 27 07:29:01 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-l[67429]: 4: (SnapRealm::build_snap_set() const+0x323) [0x56350a18a971] 2024-05-27T07:29:01.988 INFO:journalctl@ceph.mds.l.smithi173.stdout:May 27 07:29:01 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-l[67429]: 5: (SnapRealm::check_cache() const+0x13e) [0x56350a188fa0] 2024-05-27T07:29:01.988 INFO:journalctl@ceph.mds.l.smithi173.stdout:May 27 07:29:01 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-l[67429]: 6: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x56350a189651] 2024-05-27T07:29:01.988 INFO:journalctl@ceph.mds.l.smithi173.stdout:May 27 07:29:01 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-l[67429]: 7: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x56350a11e347] 2024-05-27T07:29:01.988 INFO:journalctl@ceph.mds.l.smithi173.stdout:May 27 07:29:01 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-l[67429]: 8: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x563509e1a87e] 2024-05-27T07:29:01.988 INFO:journalctl@ceph.mds.l.smithi173.stdout:May 27 07:29:01 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-l[67429]: 9: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x2702) [0x563509e1fe94] 2024-05-27T07:29:01.988 INFO:journalctl@ceph.mds.l.smithi173.stdout:May 27 07:29:01 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-l[67429]: 10: (EUpdate::replay(MDSRank*)+0x63) [0x563509e2a969] 2024-05-27T07:29:01.988 INFO:journalctl@ceph.mds.l.smithi173.stdout:May 27 07:29:01 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-l[67429]: 11: (MDLog::_replay_thread()+0x1f38) [0x56350a1c348c] 2024-05-27T07:29:01.988 INFO:journalctl@ceph.mds.l.smithi173.stdout:May 27 07:29:01 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-l[67429]: 12: (MDLog::ReplayThread::entry()+0x11) [0x56350a1cc9cb] 2024-05-27T07:29:01.988 INFO:journalctl@ceph.mds.l.smithi173.stdout:May 27 07:29:01 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-l[67429]: 13: (Thread::entry_wrapper()+0x43) [0x7f71d0c1172d] 2024-05-27T07:29:01.988 INFO:journalctl@ceph.mds.l.smithi173.stdout:May 27 07:29:01 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-l[67429]: 14: (Thread::_entry_func(void*)+0xd) [0x7f71d0c11749] 2024-05-27T07:29:01.988 INFO:journalctl@ceph.mds.l.smithi173.stdout:May 27 07:29:01 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-l[67429]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7f71cf7b51ca] 2024-05-27T07:29:01.988 INFO:journalctl@ceph.mds.l.smithi173.stdout:May 27 07:29:01 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-l[67429]: 16: clone() 2024-05-27T07:29:01.988 INFO:journalctl@ceph.mds.l.smithi173.stdout:May 27 07:29:01 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-l[67429]: 2024-05-27T07:29:01.639+0000 7f71be3fd700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: In function 'std::set SnapClient::filter(const std::set&) const' thread 7f71be3fd700 time 2024-05-27T07:29:01.638714+0000 2024-05-27T07:29:01.989 INFO:journalctl@ceph.mds.l.smithi173.stdout:May 27 07:29:01 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-l[67429]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: 197: FAILED ceph_assert(cached_version > 0) 2024-05-27T07:29:01.989 INFO:journalctl@ceph.mds.l.smithi173.stdout:May 27 07:29:01 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-l[67429]: 2024-05-27T07:29:01.989 INFO:journalctl@ceph.mds.l.smithi173.stdout:May 27 07:29:01 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-l[67429]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:29:01.989 INFO:journalctl@ceph.mds.l.smithi173.stdout:May 27 07:29:01 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-l[67429]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x127) [0x7f71d0c42ba6] 2024-05-27T07:29:01.990 INFO:journalctl@ceph.mds.l.smithi173.stdout:May 27 07:29:01 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-l[67429]: 2: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7f71d0c42dd1] 2024-05-27T07:29:01.990 INFO:journalctl@ceph.mds.l.smithi173.stdout:May 27 07:29:01 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-l[67429]: 3: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x56350a19f23d] 2024-05-27T07:29:01.990 INFO:journalctl@ceph.mds.l.smithi173.stdout:May 27 07:29:01 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-l[67429]: 4: (SnapRealm::build_snap_set() const+0x323) [0x56350a18a971] 2024-05-27T07:29:01.990 INFO:journalctl@ceph.mds.l.smithi173.stdout:May 27 07:29:01 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-l[67429]: 5: (SnapRealm::check_cache() const+0x13e) [0x56350a188fa0] 2024-05-27T07:29:01.990 INFO:journalctl@ceph.mds.l.smithi173.stdout:May 27 07:29:01 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-l[67429]: 6: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x56350a189651] 2024-05-27T07:29:01.990 INFO:journalctl@ceph.mds.l.smithi173.stdout:May 27 07:29:01 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-l[67429]: 7: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x56350a11e347] 2024-05-27T07:29:01.990 INFO:journalctl@ceph.mds.l.smithi173.stdout:May 27 07:29:01 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-l[67429]: 8: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x563509e1a87e] 2024-05-27T07:29:01.990 INFO:journalctl@ceph.mds.l.smithi173.stdout:May 27 07:29:01 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-l[67429]: 9: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x2702) [0x563509e1fe94] 2024-05-27T07:29:01.990 INFO:journalctl@ceph.mds.l.smithi173.stdout:May 27 07:29:01 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-l[67429]: 10: (EUpdate::replay(MDSRank*)+0x63) [0x563509e2a969] 2024-05-27T07:29:01.990 INFO:journalctl@ceph.mds.l.smithi173.stdout:May 27 07:29:01 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-l[67429]: 11: (MDLog::_replay_thread()+0x1f38) [0x56350a1c348c] 2024-05-27T07:29:01.990 INFO:journalctl@ceph.mds.l.smithi173.stdout:May 27 07:29:01 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-l[67429]: 12: (MDLog::ReplayThread::entry()+0x11) [0x56350a1cc9cb] 2024-05-27T07:29:01.991 INFO:journalctl@ceph.mds.l.smithi173.stdout:May 27 07:29:01 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-l[67429]: 13: (Thread::entry_wrapper()+0x43) [0x7f71d0c1172d] 2024-05-27T07:29:01.991 INFO:journalctl@ceph.mds.l.smithi173.stdout:May 27 07:29:01 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-l[67429]: 14: (Thread::_entry_func(void*)+0xd) [0x7f71d0c11749] 2024-05-27T07:29:01.991 INFO:journalctl@ceph.mds.l.smithi173.stdout:May 27 07:29:01 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-l[67429]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7f71cf7b51ca] 2024-05-27T07:29:01.991 INFO:journalctl@ceph.mds.l.smithi173.stdout:May 27 07:29:01 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-l[67429]: 16: clone() 2024-05-27T07:29:01.991 INFO:journalctl@ceph.mds.l.smithi173.stdout:May 27 07:29:01 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-l[67429]: 2024-05-27T07:29:01.991 INFO:journalctl@ceph.mds.l.smithi173.stdout:May 27 07:29:01 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-l[67429]: *** Caught signal (Aborted) ** 2024-05-27T07:29:01.991 INFO:journalctl@ceph.mds.l.smithi173.stdout:May 27 07:29:01 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-l[67429]: in thread 7f71be3fd700 thread_name:md_log_replay 2024-05-27T07:29:01.991 INFO:journalctl@ceph.mds.l.smithi173.stdout:May 27 07:29:01 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-l[67429]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:29:01.991 INFO:journalctl@ceph.mds.l.smithi173.stdout:May 27 07:29:01 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-l[67429]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f71cf7bfd20] 2024-05-27T07:29:01.991 INFO:journalctl@ceph.mds.l.smithi173.stdout:May 27 07:29:01 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-l[67429]: 2: gsignal() 2024-05-27T07:29:01.991 INFO:journalctl@ceph.mds.l.smithi173.stdout:May 27 07:29:01 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-l[67429]: 3: abort() 2024-05-27T07:29:01.992 INFO:journalctl@ceph.mds.l.smithi173.stdout:May 27 07:29:01 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-l[67429]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x26e) [0x7f71d0c42ced] 2024-05-27T07:29:01.992 INFO:journalctl@ceph.mds.l.smithi173.stdout:May 27 07:29:01 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-l[67429]: 5: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7f71d0c42dd1] 2024-05-27T07:29:01.992 INFO:journalctl@ceph.mds.l.smithi173.stdout:May 27 07:29:01 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-l[67429]: 6: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x56350a19f23d] 2024-05-27T07:29:01.992 INFO:journalctl@ceph.mds.l.smithi173.stdout:May 27 07:29:01 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-l[67429]: 7: (SnapRealm::build_snap_set() const+0x323) [0x56350a18a971] 2024-05-27T07:29:01.992 INFO:journalctl@ceph.mds.l.smithi173.stdout:May 27 07:29:01 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-l[67429]: 8: (SnapRealm::check_cache() const+0x13e) [0x56350a188fa0] 2024-05-27T07:29:01.992 INFO:journalctl@ceph.mds.l.smithi173.stdout:May 27 07:29:01 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-l[67429]: 9: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x56350a189651] 2024-05-27T07:29:01.992 INFO:journalctl@ceph.mds.l.smithi173.stdout:May 27 07:29:01 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-l[67429]: 10: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x56350a11e347] 2024-05-27T07:29:01.992 INFO:journalctl@ceph.mds.l.smithi173.stdout:May 27 07:29:01 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-l[67429]: 11: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x563509e1a87e] 2024-05-27T07:29:01.992 INFO:journalctl@ceph.mds.l.smithi173.stdout:May 27 07:29:01 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-l[67429]: 12: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x2702) [0x563509e1fe94] 2024-05-27T07:29:01.992 INFO:journalctl@ceph.mds.l.smithi173.stdout:May 27 07:29:01 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-l[67429]: 13: (EUpdate::replay(MDSRank*)+0x63) [0x563509e2a969] 2024-05-27T07:29:01.992 INFO:journalctl@ceph.mds.l.smithi173.stdout:May 27 07:29:01 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-l[67429]: 14: (MDLog::_replay_thread()+0x1f38) [0x56350a1c348c] 2024-05-27T07:29:01.992 INFO:journalctl@ceph.mds.l.smithi173.stdout:May 27 07:29:01 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-l[67429]: 15: (MDLog::ReplayThread::entry()+0x11) [0x56350a1cc9cb] 2024-05-27T07:29:01.993 INFO:journalctl@ceph.mds.l.smithi173.stdout:May 27 07:29:01 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-l[67429]: 16: (Thread::entry_wrapper()+0x43) [0x7f71d0c1172d] 2024-05-27T07:29:01.993 INFO:journalctl@ceph.mds.l.smithi173.stdout:May 27 07:29:01 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-l[67429]: 17: (Thread::_entry_func(void*)+0xd) [0x7f71d0c11749] 2024-05-27T07:29:01.993 INFO:journalctl@ceph.mds.l.smithi173.stdout:May 27 07:29:01 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-l[67429]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7f71cf7b51ca] 2024-05-27T07:29:01.993 INFO:journalctl@ceph.mds.l.smithi173.stdout:May 27 07:29:01 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-l[67429]: 19: clone() 2024-05-27T07:29:01.993 INFO:journalctl@ceph.mds.l.smithi173.stdout:May 27 07:29:01 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-l[67429]: 2024-05-27T07:29:01.642+0000 7f71be3fd700 -1 *** Caught signal (Aborted) ** 2024-05-27T07:29:01.993 INFO:journalctl@ceph.mds.l.smithi173.stdout:May 27 07:29:01 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-l[67429]: in thread 7f71be3fd700 thread_name:md_log_replay 2024-05-27T07:29:01.993 INFO:journalctl@ceph.mds.l.smithi173.stdout:May 27 07:29:01 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-l[67429]: 2024-05-27T07:29:01.993 INFO:journalctl@ceph.mds.l.smithi173.stdout:May 27 07:29:01 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-l[67429]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:29:01.993 INFO:journalctl@ceph.mds.l.smithi173.stdout:May 27 07:29:01 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-l[67429]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f71cf7bfd20] 2024-05-27T07:29:01.993 INFO:journalctl@ceph.mds.l.smithi173.stdout:May 27 07:29:01 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-l[67429]: 2: gsignal() 2024-05-27T07:29:01.993 INFO:journalctl@ceph.mds.l.smithi173.stdout:May 27 07:29:01 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-l[67429]: 3: abort() 2024-05-27T07:29:01.993 INFO:journalctl@ceph.mds.l.smithi173.stdout:May 27 07:29:01 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-l[67429]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x26e) [0x7f71d0c42ced] 2024-05-27T07:29:01.993 INFO:journalctl@ceph.mds.l.smithi173.stdout:May 27 07:29:01 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-l[67429]: 5: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7f71d0c42dd1] 2024-05-27T07:29:01.993 INFO:journalctl@ceph.mds.l.smithi173.stdout:May 27 07:29:01 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-l[67429]: 6: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x56350a19f23d] 2024-05-27T07:29:01.994 INFO:journalctl@ceph.mds.l.smithi173.stdout:May 27 07:29:01 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-l[67429]: 7: (SnapRealm::build_snap_set() const+0x323) [0x56350a18a971] 2024-05-27T07:29:01.994 INFO:journalctl@ceph.mds.l.smithi173.stdout:May 27 07:29:01 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-l[67429]: 8: (SnapRealm::check_cache() const+0x13e) [0x56350a188fa0] 2024-05-27T07:29:01.994 INFO:journalctl@ceph.mds.l.smithi173.stdout:May 27 07:29:01 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-l[67429]: 9: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x56350a189651] 2024-05-27T07:29:01.994 INFO:journalctl@ceph.mds.l.smithi173.stdout:May 27 07:29:01 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-l[67429]: 10: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x56350a11e347] 2024-05-27T07:29:01.994 INFO:journalctl@ceph.mds.l.smithi173.stdout:May 27 07:29:01 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-l[67429]: 11: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x563509e1a87e] 2024-05-27T07:29:01.994 INFO:journalctl@ceph.mds.l.smithi173.stdout:May 27 07:29:01 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-l[67429]: 12: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x2702) [0x563509e1fe94] 2024-05-27T07:29:01.994 INFO:journalctl@ceph.mds.l.smithi173.stdout:May 27 07:29:01 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-l[67429]: 13: (EUpdate::replay(MDSRank*)+0x63) [0x563509e2a969] 2024-05-27T07:29:01.994 INFO:journalctl@ceph.mds.l.smithi173.stdout:May 27 07:29:01 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-l[67429]: 14: (MDLog::_replay_thread()+0x1f38) [0x56350a1c348c] 2024-05-27T07:29:01.994 INFO:journalctl@ceph.mds.l.smithi173.stdout:May 27 07:29:01 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-l[67429]: 15: (MDLog::ReplayThread::entry()+0x11) [0x56350a1cc9cb] 2024-05-27T07:29:01.994 INFO:journalctl@ceph.mds.l.smithi173.stdout:May 27 07:29:01 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-l[67429]: 16: (Thread::entry_wrapper()+0x43) [0x7f71d0c1172d] 2024-05-27T07:29:01.995 INFO:journalctl@ceph.mds.l.smithi173.stdout:May 27 07:29:01 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-l[67429]: 17: (Thread::_entry_func(void*)+0xd) [0x7f71d0c11749] 2024-05-27T07:29:01.995 INFO:journalctl@ceph.mds.l.smithi173.stdout:May 27 07:29:01 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-l[67429]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7f71cf7b51ca] 2024-05-27T07:29:01.995 INFO:journalctl@ceph.mds.l.smithi173.stdout:May 27 07:29:01 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-l[67429]: 19: clone() 2024-05-27T07:29:01.995 INFO:journalctl@ceph.mds.l.smithi173.stdout:May 27 07:29:01 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-l[67429]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-05-27T07:29:01.995 INFO:journalctl@ceph.mds.l.smithi173.stdout:May 27 07:29:01 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-l[67429]: 2024-05-27T07:29:01.995 INFO:journalctl@ceph.mds.l.smithi173.stdout:May 27 07:29:01 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-l[67429]: -1> 2024-05-27T07:29:01.639+0000 7f71be3fd700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: In function 'std::set SnapClient::filter(const std::set&) const' thread 7f71be3fd700 time 2024-05-27T07:29:01.638714+0000 2024-05-27T07:29:01.995 INFO:journalctl@ceph.mds.l.smithi173.stdout:May 27 07:29:01 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-l[67429]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: 197: FAILED ceph_assert(cached_version > 0) 2024-05-27T07:29:01.995 INFO:journalctl@ceph.mds.l.smithi173.stdout:May 27 07:29:01 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-l[67429]: 2024-05-27T07:29:01.995 INFO:journalctl@ceph.mds.l.smithi173.stdout:May 27 07:29:01 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-l[67429]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:29:01.995 INFO:journalctl@ceph.mds.l.smithi173.stdout:May 27 07:29:01 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-l[67429]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x127) [0x7f71d0c42ba6] 2024-05-27T07:29:01.995 INFO:journalctl@ceph.mds.l.smithi173.stdout:May 27 07:29:01 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-l[67429]: 2: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7f71d0c42dd1] 2024-05-27T07:29:01.995 INFO:journalctl@ceph.mds.l.smithi173.stdout:May 27 07:29:01 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-l[67429]: 3: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x56350a19f23d] 2024-05-27T07:29:01.996 INFO:journalctl@ceph.mds.l.smithi173.stdout:May 27 07:29:01 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-l[67429]: 4: (SnapRealm::build_snap_set() const+0x323) [0x56350a18a971] 2024-05-27T07:29:01.996 INFO:journalctl@ceph.mds.l.smithi173.stdout:May 27 07:29:01 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-l[67429]: 5: (SnapRealm::check_cache() const+0x13e) [0x56350a188fa0] 2024-05-27T07:29:01.996 INFO:journalctl@ceph.mds.l.smithi173.stdout:May 27 07:29:01 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-l[67429]: 6: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x56350a189651] 2024-05-27T07:29:01.996 INFO:journalctl@ceph.mds.l.smithi173.stdout:May 27 07:29:01 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-l[67429]: 7: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x56350a11e347] 2024-05-27T07:29:01.996 INFO:journalctl@ceph.mds.l.smithi173.stdout:May 27 07:29:01 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-l[67429]: 8: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x563509e1a87e] 2024-05-27T07:29:01.996 INFO:journalctl@ceph.mds.l.smithi173.stdout:May 27 07:29:01 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-l[67429]: 9: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x2702) [0x563509e1fe94] 2024-05-27T07:29:01.996 INFO:journalctl@ceph.mds.l.smithi173.stdout:May 27 07:29:01 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-l[67429]: 10: (EUpdate::replay(MDSRank*)+0x63) [0x563509e2a969] 2024-05-27T07:29:01.996 INFO:journalctl@ceph.mds.l.smithi173.stdout:May 27 07:29:01 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-l[67429]: 11: (MDLog::_replay_thread()+0x1f38) [0x56350a1c348c] 2024-05-27T07:29:01.996 INFO:journalctl@ceph.mds.l.smithi173.stdout:May 27 07:29:01 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-l[67429]: 12: (MDLog::ReplayThread::entry()+0x11) [0x56350a1cc9cb] 2024-05-27T07:29:01.996 INFO:journalctl@ceph.mds.l.smithi173.stdout:May 27 07:29:01 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-l[67429]: 13: (Thread::entry_wrapper()+0x43) [0x7f71d0c1172d] 2024-05-27T07:29:01.996 INFO:journalctl@ceph.mds.l.smithi173.stdout:May 27 07:29:01 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-l[67429]: 14: (Thread::_entry_func(void*)+0xd) [0x7f71d0c11749] 2024-05-27T07:29:01.996 INFO:journalctl@ceph.mds.l.smithi173.stdout:May 27 07:29:01 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-l[67429]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7f71cf7b51ca] 2024-05-27T07:29:01.996 INFO:journalctl@ceph.mds.l.smithi173.stdout:May 27 07:29:01 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-l[67429]: 16: clone() 2024-05-27T07:29:01.997 INFO:journalctl@ceph.mds.l.smithi173.stdout:May 27 07:29:01 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-l[67429]: 2024-05-27T07:29:01.997 INFO:journalctl@ceph.mds.l.smithi173.stdout:May 27 07:29:01 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-l[67429]: 0> 2024-05-27T07:29:01.642+0000 7f71be3fd700 -1 *** Caught signal (Aborted) ** 2024-05-27T07:29:01.997 INFO:journalctl@ceph.mds.l.smithi173.stdout:May 27 07:29:01 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-l[67429]: in thread 7f71be3fd700 thread_name:md_log_replay 2024-05-27T07:29:01.997 INFO:journalctl@ceph.mds.l.smithi173.stdout:May 27 07:29:01 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-l[67429]: 2024-05-27T07:29:01.997 INFO:journalctl@ceph.mds.l.smithi173.stdout:May 27 07:29:01 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-l[67429]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:29:01.997 INFO:journalctl@ceph.mds.l.smithi173.stdout:May 27 07:29:01 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-l[67429]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f71cf7bfd20] 2024-05-27T07:29:01.997 INFO:journalctl@ceph.mds.l.smithi173.stdout:May 27 07:29:01 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-l[67429]: 2: gsignal() 2024-05-27T07:29:01.997 INFO:journalctl@ceph.mds.l.smithi173.stdout:May 27 07:29:01 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-l[67429]: 3: abort() 2024-05-27T07:29:01.997 INFO:journalctl@ceph.mds.l.smithi173.stdout:May 27 07:29:01 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-l[67429]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x26e) [0x7f71d0c42ced] 2024-05-27T07:29:01.997 INFO:journalctl@ceph.mds.l.smithi173.stdout:May 27 07:29:01 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-l[67429]: 5: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7f71d0c42dd1] 2024-05-27T07:29:01.997 INFO:journalctl@ceph.mds.l.smithi173.stdout:May 27 07:29:01 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-l[67429]: 6: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x56350a19f23d] 2024-05-27T07:29:01.997 INFO:journalctl@ceph.mds.l.smithi173.stdout:May 27 07:29:01 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-l[67429]: 7: (SnapRealm::build_snap_set() const+0x323) [0x56350a18a971] 2024-05-27T07:29:01.998 INFO:journalctl@ceph.mds.l.smithi173.stdout:May 27 07:29:01 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-l[67429]: 8: (SnapRealm::check_cache() const+0x13e) [0x56350a188fa0] 2024-05-27T07:29:01.998 INFO:journalctl@ceph.mds.l.smithi173.stdout:May 27 07:29:01 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-l[67429]: 9: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x56350a189651] 2024-05-27T07:29:01.998 INFO:journalctl@ceph.mds.l.smithi173.stdout:May 27 07:29:01 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-l[67429]: 10: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x56350a11e347] 2024-05-27T07:29:01.998 INFO:journalctl@ceph.mds.l.smithi173.stdout:May 27 07:29:01 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-l[67429]: 11: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x563509e1a87e] 2024-05-27T07:29:01.998 INFO:journalctl@ceph.mds.l.smithi173.stdout:May 27 07:29:01 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-l[67429]: 12: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x2702) [0x563509e1fe94] 2024-05-27T07:29:01.998 INFO:journalctl@ceph.mds.l.smithi173.stdout:May 27 07:29:01 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-l[67429]: 13: (EUpdate::replay(MDSRank*)+0x63) [0x563509e2a969] 2024-05-27T07:29:01.998 INFO:journalctl@ceph.mds.l.smithi173.stdout:May 27 07:29:01 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-l[67429]: 14: (MDLog::_replay_thread()+0x1f38) [0x56350a1c348c] 2024-05-27T07:29:01.998 INFO:journalctl@ceph.mds.l.smithi173.stdout:May 27 07:29:01 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-l[67429]: 15: (MDLog::ReplayThread::entry()+0x11) [0x56350a1cc9cb] 2024-05-27T07:29:01.998 INFO:journalctl@ceph.mds.l.smithi173.stdout:May 27 07:29:01 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-l[67429]: 16: (Thread::entry_wrapper()+0x43) [0x7f71d0c1172d] 2024-05-27T07:29:01.999 INFO:journalctl@ceph.mds.l.smithi173.stdout:May 27 07:29:01 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-l[67429]: 17: (Thread::_entry_func(void*)+0xd) [0x7f71d0c11749] 2024-05-27T07:29:01.999 INFO:journalctl@ceph.mds.l.smithi173.stdout:May 27 07:29:01 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-l[67429]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7f71cf7b51ca] 2024-05-27T07:29:01.999 INFO:journalctl@ceph.mds.l.smithi173.stdout:May 27 07:29:01 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-l[67429]: 19: clone() 2024-05-27T07:29:01.999 INFO:journalctl@ceph.mds.l.smithi173.stdout:May 27 07:29:01 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-l[67429]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-05-27T07:29:01.999 INFO:journalctl@ceph.mds.l.smithi173.stdout:May 27 07:29:01 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-l[67429]: 2024-05-27T07:29:01.999 INFO:journalctl@ceph.mds.l.smithi173.stdout:May 27 07:29:01 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-l[67429]: -1> 2024-05-27T07:29:01.639+0000 7f71be3fd700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: In function 'std::set SnapClient::filter(const std::set&) const' thread 7f71be3fd700 time 2024-05-27T07:29:01.638714+0000 2024-05-27T07:29:01.999 INFO:journalctl@ceph.mds.l.smithi173.stdout:May 27 07:29:01 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-l[67429]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: 197: FAILED ceph_assert(cached_version > 0) 2024-05-27T07:29:01.999 INFO:journalctl@ceph.mds.l.smithi173.stdout:May 27 07:29:01 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-l[67429]: 2024-05-27T07:29:01.999 INFO:journalctl@ceph.mds.l.smithi173.stdout:May 27 07:29:01 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-l[67429]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:29:01.999 INFO:journalctl@ceph.mds.l.smithi173.stdout:May 27 07:29:01 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-l[67429]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x127) [0x7f71d0c42ba6] 2024-05-27T07:29:02.000 INFO:journalctl@ceph.mds.l.smithi173.stdout:May 27 07:29:01 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-l[67429]: 2: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7f71d0c42dd1] 2024-05-27T07:29:02.000 INFO:journalctl@ceph.mds.l.smithi173.stdout:May 27 07:29:01 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-l[67429]: 3: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x56350a19f23d] 2024-05-27T07:29:02.000 INFO:journalctl@ceph.mds.l.smithi173.stdout:May 27 07:29:01 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-l[67429]: 4: (SnapRealm::build_snap_set() const+0x323) [0x56350a18a971] 2024-05-27T07:29:02.000 INFO:journalctl@ceph.mds.l.smithi173.stdout:May 27 07:29:01 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-l[67429]: 5: (SnapRealm::check_cache() const+0x13e) [0x56350a188fa0] 2024-05-27T07:29:02.000 INFO:journalctl@ceph.mds.l.smithi173.stdout:May 27 07:29:01 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-l[67429]: 6: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x56350a189651] 2024-05-27T07:29:02.000 INFO:journalctl@ceph.mds.l.smithi173.stdout:May 27 07:29:01 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-l[67429]: 7: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x56350a11e347] 2024-05-27T07:29:02.000 INFO:journalctl@ceph.mds.l.smithi173.stdout:May 27 07:29:01 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-l[67429]: 8: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x563509e1a87e] 2024-05-27T07:29:02.000 INFO:journalctl@ceph.mds.l.smithi173.stdout:May 27 07:29:01 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-l[67429]: 9: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x2702) [0x563509e1fe94] 2024-05-27T07:29:02.000 INFO:journalctl@ceph.mds.l.smithi173.stdout:May 27 07:29:01 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-l[67429]: 10: (EUpdate::replay(MDSRank*)+0x63) [0x563509e2a969] 2024-05-27T07:29:02.000 INFO:journalctl@ceph.mds.l.smithi173.stdout:May 27 07:29:01 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-l[67429]: 11: (MDLog::_replay_thread()+0x1f38) [0x56350a1c348c] 2024-05-27T07:29:02.000 INFO:journalctl@ceph.mds.l.smithi173.stdout:May 27 07:29:01 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-l[67429]: 12: (MDLog::ReplayThread::entry()+0x11) [0x56350a1cc9cb] 2024-05-27T07:29:02.000 INFO:journalctl@ceph.mds.l.smithi173.stdout:May 27 07:29:01 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-l[67429]: 13: (Thread::entry_wrapper()+0x43) [0x7f71d0c1172d] 2024-05-27T07:29:02.000 INFO:journalctl@ceph.mds.l.smithi173.stdout:May 27 07:29:01 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-l[67429]: 14: (Thread::_entry_func(void*)+0xd) [0x7f71d0c11749] 2024-05-27T07:29:02.001 INFO:journalctl@ceph.mds.l.smithi173.stdout:May 27 07:29:01 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-l[67429]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7f71cf7b51ca] 2024-05-27T07:29:02.001 INFO:journalctl@ceph.mds.l.smithi173.stdout:May 27 07:29:01 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-l[67429]: 16: clone() 2024-05-27T07:29:02.001 INFO:journalctl@ceph.mds.l.smithi173.stdout:May 27 07:29:01 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-l[67429]: 2024-05-27T07:29:02.001 INFO:journalctl@ceph.mds.l.smithi173.stdout:May 27 07:29:01 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-l[67429]: 0> 2024-05-27T07:29:01.642+0000 7f71be3fd700 -1 *** Caught signal (Aborted) ** 2024-05-27T07:29:02.001 INFO:journalctl@ceph.mds.l.smithi173.stdout:May 27 07:29:01 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-l[67429]: in thread 7f71be3fd700 thread_name:md_log_replay 2024-05-27T07:29:02.001 INFO:journalctl@ceph.mds.l.smithi173.stdout:May 27 07:29:01 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-l[67429]: 2024-05-27T07:29:02.001 INFO:journalctl@ceph.mds.l.smithi173.stdout:May 27 07:29:01 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-l[67429]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:29:02.001 INFO:journalctl@ceph.mds.l.smithi173.stdout:May 27 07:29:01 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-l[67429]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f71cf7bfd20] 2024-05-27T07:29:02.001 INFO:journalctl@ceph.mds.l.smithi173.stdout:May 27 07:29:01 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-l[67429]: 2: gsignal() 2024-05-27T07:29:02.001 INFO:journalctl@ceph.mds.l.smithi173.stdout:May 27 07:29:01 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-l[67429]: 3: abort() 2024-05-27T07:29:02.002 INFO:journalctl@ceph.mds.l.smithi173.stdout:May 27 07:29:01 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-l[67429]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x26e) [0x7f71d0c42ced] 2024-05-27T07:29:02.002 INFO:journalctl@ceph.mds.l.smithi173.stdout:May 27 07:29:01 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-l[67429]: 5: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7f71d0c42dd1] 2024-05-27T07:29:02.002 INFO:journalctl@ceph.mds.l.smithi173.stdout:May 27 07:29:01 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-l[67429]: 6: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x56350a19f23d] 2024-05-27T07:29:02.002 INFO:journalctl@ceph.mds.l.smithi173.stdout:May 27 07:29:01 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-l[67429]: 7: (SnapRealm::build_snap_set() const+0x323) [0x56350a18a971] 2024-05-27T07:29:02.002 INFO:journalctl@ceph.mds.l.smithi173.stdout:May 27 07:29:01 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-l[67429]: 8: (SnapRealm::check_cache() const+0x13e) [0x56350a188fa0] 2024-05-27T07:29:02.002 INFO:journalctl@ceph.mds.l.smithi173.stdout:May 27 07:29:01 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-l[67429]: 9: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x56350a189651] 2024-05-27T07:29:02.002 INFO:journalctl@ceph.mds.l.smithi173.stdout:May 27 07:29:01 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-l[67429]: 10: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x56350a11e347] 2024-05-27T07:29:02.002 INFO:journalctl@ceph.mds.l.smithi173.stdout:May 27 07:29:01 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-l[67429]: 11: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x563509e1a87e] 2024-05-27T07:29:02.002 INFO:journalctl@ceph.mds.l.smithi173.stdout:May 27 07:29:01 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-l[67429]: 12: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x2702) [0x563509e1fe94] 2024-05-27T07:29:02.002 INFO:journalctl@ceph.mds.l.smithi173.stdout:May 27 07:29:01 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-l[67429]: 13: (EUpdate::replay(MDSRank*)+0x63) [0x563509e2a969] 2024-05-27T07:29:02.002 INFO:journalctl@ceph.mds.l.smithi173.stdout:May 27 07:29:01 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-l[67429]: 14: (MDLog::_replay_thread()+0x1f38) [0x56350a1c348c] 2024-05-27T07:29:02.002 INFO:journalctl@ceph.mds.l.smithi173.stdout:May 27 07:29:01 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-l[67429]: 15: (MDLog::ReplayThread::entry()+0x11) [0x56350a1cc9cb] 2024-05-27T07:29:02.002 INFO:journalctl@ceph.mds.l.smithi173.stdout:May 27 07:29:01 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-l[67429]: 16: (Thread::entry_wrapper()+0x43) [0x7f71d0c1172d] 2024-05-27T07:29:02.003 INFO:journalctl@ceph.mds.l.smithi173.stdout:May 27 07:29:01 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-l[67429]: 17: (Thread::_entry_func(void*)+0xd) [0x7f71d0c11749] 2024-05-27T07:29:02.003 INFO:journalctl@ceph.mds.l.smithi173.stdout:May 27 07:29:01 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-l[67429]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7f71cf7b51ca] 2024-05-27T07:29:02.003 INFO:journalctl@ceph.mds.l.smithi173.stdout:May 27 07:29:01 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-l[67429]: 19: clone() 2024-05-27T07:29:02.003 INFO:journalctl@ceph.mds.l.smithi173.stdout:May 27 07:29:01 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-l[67429]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-05-27T07:29:02.003 INFO:journalctl@ceph.mds.l.smithi173.stdout:May 27 07:29:01 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-l[67429]: 2024-05-27T07:29:02.003 INFO:journalctl@ceph.mds.l.smithi173.stdout:May 27 07:29:01 smithi173 podman[72941]: 2024-05-27 07:29:01.699857068 +0000 UTC m=+0.018780863 container died 4752e4a243d5c61dcd9047e25c945c56cbb10391dc631051e83ed8e7a39f7d44 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-l, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.build-date=20240521, CEPH_POINT_RELEASE=) 2024-05-27T07:29:02.351 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:29:02 smithi173 ceph-mon[37000]: pgmap v101: 129 pgs: 129 active+clean; 161 MiB data, 847 MiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 5.4 MiB/s wr, 1.48k op/s 2024-05-27T07:29:02.439 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:29:02 smithi105 ceph-mon[35977]: pgmap v101: 129 pgs: 129 active+clean; 161 MiB data, 847 MiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 5.4 MiB/s wr, 1.48k op/s 2024-05-27T07:29:02.479 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:29:02 smithi052 ceph-mon[31955]: pgmap v101: 129 pgs: 129 active+clean; 161 MiB data, 847 MiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 5.4 MiB/s wr, 1.48k op/s 2024-05-27T07:29:02.634 INFO:journalctl@ceph.mds.l.smithi173.stdout:May 27 07:29:02 smithi173 podman[72941]: 2024-05-27 07:29:02.366364478 +0000 UTC m=+0.685288266 container remove 4752e4a243d5c61dcd9047e25c945c56cbb10391dc631051e83ed8e7a39f7d44 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-l, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, org.label-schema.build-date=20240521, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, ceph=True, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, GIT_BRANCH=HEAD) 2024-05-27T07:29:02.635 INFO:journalctl@ceph.mds.l.smithi173.stdout:May 27 07:29:02 smithi173 systemd[1]: ceph-73df2636-1bf8-11ef-bc9b-c7b262605968@mds.l.service: Main process exited, code=exited, status=134/n/a 2024-05-27T07:29:03.101 INFO:journalctl@ceph.mds.l.smithi173.stdout:May 27 07:29:02 smithi173 systemd[1]: ceph-73df2636-1bf8-11ef-bc9b-c7b262605968@mds.l.service: Failed with result 'exit-code'. 2024-05-27T07:29:03.102 INFO:journalctl@ceph.mds.l.smithi173.stdout:May 27 07:29:02 smithi173 systemd[1]: ceph-73df2636-1bf8-11ef-bc9b-c7b262605968@mds.l.service: Consumed 1.428s CPU time. 2024-05-27T07:29:03.750 INFO:journalctl@ceph.mds.d.smithi052.stdout:May 27 07:29:03 smithi052 systemd[1]: ceph-73df2636-1bf8-11ef-bc9b-c7b262605968@mds.d.service: Scheduled restart job, restart counter is at 1. 2024-05-27T07:29:03.750 INFO:journalctl@ceph.mds.d.smithi052.stdout:May 27 07:29:03 smithi052 systemd[1]: Stopped Ceph mds.d for 73df2636-1bf8-11ef-bc9b-c7b262605968. 2024-05-27T07:29:03.750 INFO:journalctl@ceph.mds.d.smithi052.stdout:May 27 07:29:03 smithi052 systemd[1]: ceph-73df2636-1bf8-11ef-bc9b-c7b262605968@mds.d.service: Consumed 1.051s CPU time. 2024-05-27T07:29:03.751 INFO:journalctl@ceph.mds.d.smithi052.stdout:May 27 07:29:03 smithi052 systemd[1]: Starting Ceph mds.d for 73df2636-1bf8-11ef-bc9b-c7b262605968... 2024-05-27T07:29:03.940 INFO:journalctl@ceph.mds.k.smithi105.stdout:May 27 07:29:03 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-k[64614]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: In function 'std::set SnapClient::filter(const std::set&) const' thread 7f93d3de4700 time 2024-05-27T07:29:03.671449+0000 2024-05-27T07:29:03.941 INFO:journalctl@ceph.mds.k.smithi105.stdout:May 27 07:29:03 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-k[64614]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: 197: FAILED ceph_assert(cached_version > 0) 2024-05-27T07:29:03.941 INFO:journalctl@ceph.mds.k.smithi105.stdout:May 27 07:29:03 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-k[64614]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:29:03.941 INFO:journalctl@ceph.mds.k.smithi105.stdout:May 27 07:29:03 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-k[64614]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x127) [0x7f93e6629ba6] 2024-05-27T07:29:03.941 INFO:journalctl@ceph.mds.k.smithi105.stdout:May 27 07:29:03 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-k[64614]: 2: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7f93e6629dd1] 2024-05-27T07:29:03.941 INFO:journalctl@ceph.mds.k.smithi105.stdout:May 27 07:29:03 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-k[64614]: 3: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x55b01f05023d] 2024-05-27T07:29:03.941 INFO:journalctl@ceph.mds.k.smithi105.stdout:May 27 07:29:03 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-k[64614]: 4: (SnapRealm::build_snap_set() const+0x323) [0x55b01f03b971] 2024-05-27T07:29:03.941 INFO:journalctl@ceph.mds.k.smithi105.stdout:May 27 07:29:03 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-k[64614]: 5: (SnapRealm::check_cache() const+0x13e) [0x55b01f039fa0] 2024-05-27T07:29:03.941 INFO:journalctl@ceph.mds.k.smithi105.stdout:May 27 07:29:03 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-k[64614]: 6: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x55b01f03a651] 2024-05-27T07:29:03.941 INFO:journalctl@ceph.mds.k.smithi105.stdout:May 27 07:29:03 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-k[64614]: 7: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x55b01efcf347] 2024-05-27T07:29:03.941 INFO:journalctl@ceph.mds.k.smithi105.stdout:May 27 07:29:03 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-k[64614]: 8: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x55b01eccb87e] 2024-05-27T07:29:03.941 INFO:journalctl@ceph.mds.k.smithi105.stdout:May 27 07:29:03 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-k[64614]: 9: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x2702) [0x55b01ecd0e94] 2024-05-27T07:29:03.941 INFO:journalctl@ceph.mds.k.smithi105.stdout:May 27 07:29:03 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-k[64614]: 10: (EUpdate::replay(MDSRank*)+0x63) [0x55b01ecdb969] 2024-05-27T07:29:03.942 INFO:journalctl@ceph.mds.k.smithi105.stdout:May 27 07:29:03 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-k[64614]: 11: (MDLog::_replay_thread()+0x1f38) [0x55b01f07448c] 2024-05-27T07:29:03.942 INFO:journalctl@ceph.mds.k.smithi105.stdout:May 27 07:29:03 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-k[64614]: 12: (MDLog::ReplayThread::entry()+0x11) [0x55b01f07d9cb] 2024-05-27T07:29:03.942 INFO:journalctl@ceph.mds.k.smithi105.stdout:May 27 07:29:03 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-k[64614]: 13: (Thread::entry_wrapper()+0x43) [0x7f93e65f872d] 2024-05-27T07:29:03.942 INFO:journalctl@ceph.mds.k.smithi105.stdout:May 27 07:29:03 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-k[64614]: 14: (Thread::_entry_func(void*)+0xd) [0x7f93e65f8749] 2024-05-27T07:29:03.942 INFO:journalctl@ceph.mds.k.smithi105.stdout:May 27 07:29:03 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-k[64614]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7f93e519c1ca] 2024-05-27T07:29:03.942 INFO:journalctl@ceph.mds.k.smithi105.stdout:May 27 07:29:03 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-k[64614]: 16: clone() 2024-05-27T07:29:03.942 INFO:journalctl@ceph.mds.k.smithi105.stdout:May 27 07:29:03 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-k[64614]: 2024-05-27T07:29:03.673+0000 7f93d3de4700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: In function 'std::set SnapClient::filter(const std::set&) const' thread 7f93d3de4700 time 2024-05-27T07:29:03.671449+0000 2024-05-27T07:29:03.943 INFO:journalctl@ceph.mds.k.smithi105.stdout:May 27 07:29:03 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-k[64614]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: 197: FAILED ceph_assert(cached_version > 0) 2024-05-27T07:29:03.943 INFO:journalctl@ceph.mds.k.smithi105.stdout:May 27 07:29:03 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-k[64614]: 2024-05-27T07:29:03.943 INFO:journalctl@ceph.mds.k.smithi105.stdout:May 27 07:29:03 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-k[64614]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:29:03.943 INFO:journalctl@ceph.mds.k.smithi105.stdout:May 27 07:29:03 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-k[64614]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x127) [0x7f93e6629ba6] 2024-05-27T07:29:03.943 INFO:journalctl@ceph.mds.k.smithi105.stdout:May 27 07:29:03 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-k[64614]: 2: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7f93e6629dd1] 2024-05-27T07:29:03.943 INFO:journalctl@ceph.mds.k.smithi105.stdout:May 27 07:29:03 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-k[64614]: 3: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x55b01f05023d] 2024-05-27T07:29:03.943 INFO:journalctl@ceph.mds.k.smithi105.stdout:May 27 07:29:03 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-k[64614]: 4: (SnapRealm::build_snap_set() const+0x323) [0x55b01f03b971] 2024-05-27T07:29:03.943 INFO:journalctl@ceph.mds.k.smithi105.stdout:May 27 07:29:03 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-k[64614]: 5: (SnapRealm::check_cache() const+0x13e) [0x55b01f039fa0] 2024-05-27T07:29:03.943 INFO:journalctl@ceph.mds.k.smithi105.stdout:May 27 07:29:03 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-k[64614]: 6: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x55b01f03a651] 2024-05-27T07:29:03.943 INFO:journalctl@ceph.mds.k.smithi105.stdout:May 27 07:29:03 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-k[64614]: 7: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x55b01efcf347] 2024-05-27T07:29:03.943 INFO:journalctl@ceph.mds.k.smithi105.stdout:May 27 07:29:03 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-k[64614]: 8: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x55b01eccb87e] 2024-05-27T07:29:03.944 INFO:journalctl@ceph.mds.k.smithi105.stdout:May 27 07:29:03 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-k[64614]: 9: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x2702) [0x55b01ecd0e94] 2024-05-27T07:29:03.944 INFO:journalctl@ceph.mds.k.smithi105.stdout:May 27 07:29:03 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-k[64614]: 10: (EUpdate::replay(MDSRank*)+0x63) [0x55b01ecdb969] 2024-05-27T07:29:03.944 INFO:journalctl@ceph.mds.k.smithi105.stdout:May 27 07:29:03 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-k[64614]: 11: (MDLog::_replay_thread()+0x1f38) [0x55b01f07448c] 2024-05-27T07:29:03.944 INFO:journalctl@ceph.mds.k.smithi105.stdout:May 27 07:29:03 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-k[64614]: 12: (MDLog::ReplayThread::entry()+0x11) [0x55b01f07d9cb] 2024-05-27T07:29:03.944 INFO:journalctl@ceph.mds.k.smithi105.stdout:May 27 07:29:03 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-k[64614]: 13: (Thread::entry_wrapper()+0x43) [0x7f93e65f872d] 2024-05-27T07:29:03.944 INFO:journalctl@ceph.mds.k.smithi105.stdout:May 27 07:29:03 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-k[64614]: 14: (Thread::_entry_func(void*)+0xd) [0x7f93e65f8749] 2024-05-27T07:29:03.944 INFO:journalctl@ceph.mds.k.smithi105.stdout:May 27 07:29:03 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-k[64614]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7f93e519c1ca] 2024-05-27T07:29:03.944 INFO:journalctl@ceph.mds.k.smithi105.stdout:May 27 07:29:03 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-k[64614]: 16: clone() 2024-05-27T07:29:03.944 INFO:journalctl@ceph.mds.k.smithi105.stdout:May 27 07:29:03 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-k[64614]: 2024-05-27T07:29:03.944 INFO:journalctl@ceph.mds.k.smithi105.stdout:May 27 07:29:03 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-k[64614]: *** Caught signal (Aborted) ** 2024-05-27T07:29:03.944 INFO:journalctl@ceph.mds.k.smithi105.stdout:May 27 07:29:03 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-k[64614]: in thread 7f93d3de4700 thread_name:md_log_replay 2024-05-27T07:29:03.944 INFO:journalctl@ceph.mds.k.smithi105.stdout:May 27 07:29:03 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-k[64614]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:29:03.944 INFO:journalctl@ceph.mds.k.smithi105.stdout:May 27 07:29:03 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-k[64614]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f93e51a6d20] 2024-05-27T07:29:03.945 INFO:journalctl@ceph.mds.k.smithi105.stdout:May 27 07:29:03 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-k[64614]: 2: gsignal() 2024-05-27T07:29:03.945 INFO:journalctl@ceph.mds.k.smithi105.stdout:May 27 07:29:03 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-k[64614]: 3: abort() 2024-05-27T07:29:03.945 INFO:journalctl@ceph.mds.k.smithi105.stdout:May 27 07:29:03 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-k[64614]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x26e) [0x7f93e6629ced] 2024-05-27T07:29:03.945 INFO:journalctl@ceph.mds.k.smithi105.stdout:May 27 07:29:03 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-k[64614]: 5: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7f93e6629dd1] 2024-05-27T07:29:03.945 INFO:journalctl@ceph.mds.k.smithi105.stdout:May 27 07:29:03 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-k[64614]: 6: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x55b01f05023d] 2024-05-27T07:29:03.945 INFO:journalctl@ceph.mds.k.smithi105.stdout:May 27 07:29:03 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-k[64614]: 7: (SnapRealm::build_snap_set() const+0x323) [0x55b01f03b971] 2024-05-27T07:29:03.945 INFO:journalctl@ceph.mds.k.smithi105.stdout:May 27 07:29:03 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-k[64614]: 8: (SnapRealm::check_cache() const+0x13e) [0x55b01f039fa0] 2024-05-27T07:29:03.945 INFO:journalctl@ceph.mds.k.smithi105.stdout:May 27 07:29:03 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-k[64614]: 9: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x55b01f03a651] 2024-05-27T07:29:03.945 INFO:journalctl@ceph.mds.k.smithi105.stdout:May 27 07:29:03 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-k[64614]: 10: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x55b01efcf347] 2024-05-27T07:29:03.945 INFO:journalctl@ceph.mds.k.smithi105.stdout:May 27 07:29:03 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-k[64614]: 11: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x55b01eccb87e] 2024-05-27T07:29:03.945 INFO:journalctl@ceph.mds.k.smithi105.stdout:May 27 07:29:03 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-k[64614]: 12: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x2702) [0x55b01ecd0e94] 2024-05-27T07:29:03.945 INFO:journalctl@ceph.mds.k.smithi105.stdout:May 27 07:29:03 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-k[64614]: 13: (EUpdate::replay(MDSRank*)+0x63) [0x55b01ecdb969] 2024-05-27T07:29:03.945 INFO:journalctl@ceph.mds.k.smithi105.stdout:May 27 07:29:03 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-k[64614]: 14: (MDLog::_replay_thread()+0x1f38) [0x55b01f07448c] 2024-05-27T07:29:03.945 INFO:journalctl@ceph.mds.k.smithi105.stdout:May 27 07:29:03 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-k[64614]: 15: (MDLog::ReplayThread::entry()+0x11) [0x55b01f07d9cb] 2024-05-27T07:29:03.946 INFO:journalctl@ceph.mds.k.smithi105.stdout:May 27 07:29:03 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-k[64614]: 16: (Thread::entry_wrapper()+0x43) [0x7f93e65f872d] 2024-05-27T07:29:03.946 INFO:journalctl@ceph.mds.k.smithi105.stdout:May 27 07:29:03 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-k[64614]: 17: (Thread::_entry_func(void*)+0xd) [0x7f93e65f8749] 2024-05-27T07:29:03.946 INFO:journalctl@ceph.mds.k.smithi105.stdout:May 27 07:29:03 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-k[64614]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7f93e519c1ca] 2024-05-27T07:29:03.946 INFO:journalctl@ceph.mds.k.smithi105.stdout:May 27 07:29:03 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-k[64614]: 19: clone() 2024-05-27T07:29:03.946 INFO:journalctl@ceph.mds.k.smithi105.stdout:May 27 07:29:03 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-k[64614]: 2024-05-27T07:29:03.675+0000 7f93d3de4700 -1 *** Caught signal (Aborted) ** 2024-05-27T07:29:03.946 INFO:journalctl@ceph.mds.k.smithi105.stdout:May 27 07:29:03 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-k[64614]: in thread 7f93d3de4700 thread_name:md_log_replay 2024-05-27T07:29:03.946 INFO:journalctl@ceph.mds.k.smithi105.stdout:May 27 07:29:03 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-k[64614]: 2024-05-27T07:29:03.946 INFO:journalctl@ceph.mds.k.smithi105.stdout:May 27 07:29:03 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-k[64614]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:29:03.946 INFO:journalctl@ceph.mds.k.smithi105.stdout:May 27 07:29:03 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-k[64614]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f93e51a6d20] 2024-05-27T07:29:03.946 INFO:journalctl@ceph.mds.k.smithi105.stdout:May 27 07:29:03 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-k[64614]: 2: gsignal() 2024-05-27T07:29:03.946 INFO:journalctl@ceph.mds.k.smithi105.stdout:May 27 07:29:03 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-k[64614]: 3: abort() 2024-05-27T07:29:03.946 INFO:journalctl@ceph.mds.k.smithi105.stdout:May 27 07:29:03 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-k[64614]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x26e) [0x7f93e6629ced] 2024-05-27T07:29:03.946 INFO:journalctl@ceph.mds.k.smithi105.stdout:May 27 07:29:03 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-k[64614]: 5: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7f93e6629dd1] 2024-05-27T07:29:03.947 INFO:journalctl@ceph.mds.k.smithi105.stdout:May 27 07:29:03 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-k[64614]: 6: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x55b01f05023d] 2024-05-27T07:29:03.947 INFO:journalctl@ceph.mds.k.smithi105.stdout:May 27 07:29:03 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-k[64614]: 7: (SnapRealm::build_snap_set() const+0x323) [0x55b01f03b971] 2024-05-27T07:29:03.947 INFO:journalctl@ceph.mds.k.smithi105.stdout:May 27 07:29:03 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-k[64614]: 8: (SnapRealm::check_cache() const+0x13e) [0x55b01f039fa0] 2024-05-27T07:29:03.947 INFO:journalctl@ceph.mds.k.smithi105.stdout:May 27 07:29:03 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-k[64614]: 9: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x55b01f03a651] 2024-05-27T07:29:03.947 INFO:journalctl@ceph.mds.k.smithi105.stdout:May 27 07:29:03 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-k[64614]: 10: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x55b01efcf347] 2024-05-27T07:29:03.948 INFO:journalctl@ceph.mds.k.smithi105.stdout:May 27 07:29:03 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-k[64614]: 11: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x55b01eccb87e] 2024-05-27T07:29:03.948 INFO:journalctl@ceph.mds.k.smithi105.stdout:May 27 07:29:03 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-k[64614]: 12: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x2702) [0x55b01ecd0e94] 2024-05-27T07:29:03.948 INFO:journalctl@ceph.mds.k.smithi105.stdout:May 27 07:29:03 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-k[64614]: 13: (EUpdate::replay(MDSRank*)+0x63) [0x55b01ecdb969] 2024-05-27T07:29:03.948 INFO:journalctl@ceph.mds.k.smithi105.stdout:May 27 07:29:03 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-k[64614]: 14: (MDLog::_replay_thread()+0x1f38) [0x55b01f07448c] 2024-05-27T07:29:03.948 INFO:journalctl@ceph.mds.k.smithi105.stdout:May 27 07:29:03 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-k[64614]: 15: (MDLog::ReplayThread::entry()+0x11) [0x55b01f07d9cb] 2024-05-27T07:29:03.948 INFO:journalctl@ceph.mds.k.smithi105.stdout:May 27 07:29:03 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-k[64614]: 16: (Thread::entry_wrapper()+0x43) [0x7f93e65f872d] 2024-05-27T07:29:03.948 INFO:journalctl@ceph.mds.k.smithi105.stdout:May 27 07:29:03 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-k[64614]: 17: (Thread::_entry_func(void*)+0xd) [0x7f93e65f8749] 2024-05-27T07:29:03.948 INFO:journalctl@ceph.mds.k.smithi105.stdout:May 27 07:29:03 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-k[64614]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7f93e519c1ca] 2024-05-27T07:29:03.948 INFO:journalctl@ceph.mds.k.smithi105.stdout:May 27 07:29:03 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-k[64614]: 19: clone() 2024-05-27T07:29:03.948 INFO:journalctl@ceph.mds.k.smithi105.stdout:May 27 07:29:03 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-k[64614]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-05-27T07:29:03.948 INFO:journalctl@ceph.mds.k.smithi105.stdout:May 27 07:29:03 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-k[64614]: 2024-05-27T07:29:03.948 INFO:journalctl@ceph.mds.k.smithi105.stdout:May 27 07:29:03 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-k[64614]: -1> 2024-05-27T07:29:03.673+0000 7f93d3de4700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: In function 'std::set SnapClient::filter(const std::set&) const' thread 7f93d3de4700 time 2024-05-27T07:29:03.671449+0000 2024-05-27T07:29:03.949 INFO:journalctl@ceph.mds.k.smithi105.stdout:May 27 07:29:03 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-k[64614]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: 197: FAILED ceph_assert(cached_version > 0) 2024-05-27T07:29:03.949 INFO:journalctl@ceph.mds.k.smithi105.stdout:May 27 07:29:03 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-k[64614]: 2024-05-27T07:29:03.949 INFO:journalctl@ceph.mds.k.smithi105.stdout:May 27 07:29:03 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-k[64614]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:29:03.949 INFO:journalctl@ceph.mds.k.smithi105.stdout:May 27 07:29:03 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-k[64614]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x127) [0x7f93e6629ba6] 2024-05-27T07:29:03.949 INFO:journalctl@ceph.mds.k.smithi105.stdout:May 27 07:29:03 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-k[64614]: 2: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7f93e6629dd1] 2024-05-27T07:29:03.949 INFO:journalctl@ceph.mds.k.smithi105.stdout:May 27 07:29:03 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-k[64614]: 3: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x55b01f05023d] 2024-05-27T07:29:03.949 INFO:journalctl@ceph.mds.k.smithi105.stdout:May 27 07:29:03 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-k[64614]: 4: (SnapRealm::build_snap_set() const+0x323) [0x55b01f03b971] 2024-05-27T07:29:03.949 INFO:journalctl@ceph.mds.k.smithi105.stdout:May 27 07:29:03 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-k[64614]: 5: (SnapRealm::check_cache() const+0x13e) [0x55b01f039fa0] 2024-05-27T07:29:03.949 INFO:journalctl@ceph.mds.k.smithi105.stdout:May 27 07:29:03 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-k[64614]: 6: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x55b01f03a651] 2024-05-27T07:29:03.949 INFO:journalctl@ceph.mds.k.smithi105.stdout:May 27 07:29:03 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-k[64614]: 7: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x55b01efcf347] 2024-05-27T07:29:03.949 INFO:journalctl@ceph.mds.k.smithi105.stdout:May 27 07:29:03 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-k[64614]: 8: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x55b01eccb87e] 2024-05-27T07:29:03.949 INFO:journalctl@ceph.mds.k.smithi105.stdout:May 27 07:29:03 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-k[64614]: 9: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x2702) [0x55b01ecd0e94] 2024-05-27T07:29:03.949 INFO:journalctl@ceph.mds.k.smithi105.stdout:May 27 07:29:03 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-k[64614]: 10: (EUpdate::replay(MDSRank*)+0x63) [0x55b01ecdb969] 2024-05-27T07:29:03.950 INFO:journalctl@ceph.mds.k.smithi105.stdout:May 27 07:29:03 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-k[64614]: 11: (MDLog::_replay_thread()+0x1f38) [0x55b01f07448c] 2024-05-27T07:29:03.950 INFO:journalctl@ceph.mds.k.smithi105.stdout:May 27 07:29:03 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-k[64614]: 12: (MDLog::ReplayThread::entry()+0x11) [0x55b01f07d9cb] 2024-05-27T07:29:03.950 INFO:journalctl@ceph.mds.k.smithi105.stdout:May 27 07:29:03 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-k[64614]: 13: (Thread::entry_wrapper()+0x43) [0x7f93e65f872d] 2024-05-27T07:29:03.950 INFO:journalctl@ceph.mds.k.smithi105.stdout:May 27 07:29:03 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-k[64614]: 14: (Thread::_entry_func(void*)+0xd) [0x7f93e65f8749] 2024-05-27T07:29:03.950 INFO:journalctl@ceph.mds.k.smithi105.stdout:May 27 07:29:03 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-k[64614]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7f93e519c1ca] 2024-05-27T07:29:03.950 INFO:journalctl@ceph.mds.k.smithi105.stdout:May 27 07:29:03 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-k[64614]: 16: clone() 2024-05-27T07:29:03.950 INFO:journalctl@ceph.mds.k.smithi105.stdout:May 27 07:29:03 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-k[64614]: 2024-05-27T07:29:03.950 INFO:journalctl@ceph.mds.k.smithi105.stdout:May 27 07:29:03 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-k[64614]: 0> 2024-05-27T07:29:03.675+0000 7f93d3de4700 -1 *** Caught signal (Aborted) ** 2024-05-27T07:29:03.950 INFO:journalctl@ceph.mds.k.smithi105.stdout:May 27 07:29:03 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-k[64614]: in thread 7f93d3de4700 thread_name:md_log_replay 2024-05-27T07:29:03.950 INFO:journalctl@ceph.mds.k.smithi105.stdout:May 27 07:29:03 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-k[64614]: 2024-05-27T07:29:03.950 INFO:journalctl@ceph.mds.k.smithi105.stdout:May 27 07:29:03 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-k[64614]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:29:03.950 INFO:journalctl@ceph.mds.k.smithi105.stdout:May 27 07:29:03 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-k[64614]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f93e51a6d20] 2024-05-27T07:29:03.951 INFO:journalctl@ceph.mds.k.smithi105.stdout:May 27 07:29:03 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-k[64614]: 2: gsignal() 2024-05-27T07:29:03.951 INFO:journalctl@ceph.mds.k.smithi105.stdout:May 27 07:29:03 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-k[64614]: 3: abort() 2024-05-27T07:29:03.951 INFO:journalctl@ceph.mds.k.smithi105.stdout:May 27 07:29:03 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-k[64614]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x26e) [0x7f93e6629ced] 2024-05-27T07:29:03.951 INFO:journalctl@ceph.mds.k.smithi105.stdout:May 27 07:29:03 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-k[64614]: 5: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7f93e6629dd1] 2024-05-27T07:29:03.951 INFO:journalctl@ceph.mds.k.smithi105.stdout:May 27 07:29:03 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-k[64614]: 6: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x55b01f05023d] 2024-05-27T07:29:03.951 INFO:journalctl@ceph.mds.k.smithi105.stdout:May 27 07:29:03 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-k[64614]: 7: (SnapRealm::build_snap_set() const+0x323) [0x55b01f03b971] 2024-05-27T07:29:03.951 INFO:journalctl@ceph.mds.k.smithi105.stdout:May 27 07:29:03 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-k[64614]: 8: (SnapRealm::check_cache() const+0x13e) [0x55b01f039fa0] 2024-05-27T07:29:03.951 INFO:journalctl@ceph.mds.k.smithi105.stdout:May 27 07:29:03 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-k[64614]: 9: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x55b01f03a651] 2024-05-27T07:29:03.951 INFO:journalctl@ceph.mds.k.smithi105.stdout:May 27 07:29:03 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-k[64614]: 10: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x55b01efcf347] 2024-05-27T07:29:03.951 INFO:journalctl@ceph.mds.k.smithi105.stdout:May 27 07:29:03 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-k[64614]: 11: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x55b01eccb87e] 2024-05-27T07:29:03.951 INFO:journalctl@ceph.mds.k.smithi105.stdout:May 27 07:29:03 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-k[64614]: 12: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x2702) [0x55b01ecd0e94] 2024-05-27T07:29:03.951 INFO:journalctl@ceph.mds.k.smithi105.stdout:May 27 07:29:03 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-k[64614]: 13: (EUpdate::replay(MDSRank*)+0x63) [0x55b01ecdb969] 2024-05-27T07:29:03.951 INFO:journalctl@ceph.mds.k.smithi105.stdout:May 27 07:29:03 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-k[64614]: 14: (MDLog::_replay_thread()+0x1f38) [0x55b01f07448c] 2024-05-27T07:29:03.951 INFO:journalctl@ceph.mds.k.smithi105.stdout:May 27 07:29:03 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-k[64614]: 15: (MDLog::ReplayThread::entry()+0x11) [0x55b01f07d9cb] 2024-05-27T07:29:03.952 INFO:journalctl@ceph.mds.k.smithi105.stdout:May 27 07:29:03 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-k[64614]: 16: (Thread::entry_wrapper()+0x43) [0x7f93e65f872d] 2024-05-27T07:29:03.952 INFO:journalctl@ceph.mds.k.smithi105.stdout:May 27 07:29:03 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-k[64614]: 17: (Thread::_entry_func(void*)+0xd) [0x7f93e65f8749] 2024-05-27T07:29:03.952 INFO:journalctl@ceph.mds.k.smithi105.stdout:May 27 07:29:03 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-k[64614]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7f93e519c1ca] 2024-05-27T07:29:03.952 INFO:journalctl@ceph.mds.k.smithi105.stdout:May 27 07:29:03 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-k[64614]: 19: clone() 2024-05-27T07:29:03.952 INFO:journalctl@ceph.mds.k.smithi105.stdout:May 27 07:29:03 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-k[64614]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-05-27T07:29:03.952 INFO:journalctl@ceph.mds.k.smithi105.stdout:May 27 07:29:03 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-k[64614]: 2024-05-27T07:29:03.952 INFO:journalctl@ceph.mds.k.smithi105.stdout:May 27 07:29:03 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-k[64614]: -1> 2024-05-27T07:29:03.673+0000 7f93d3de4700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: In function 'std::set SnapClient::filter(const std::set&) const' thread 7f93d3de4700 time 2024-05-27T07:29:03.671449+0000 2024-05-27T07:29:03.952 INFO:journalctl@ceph.mds.k.smithi105.stdout:May 27 07:29:03 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-k[64614]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: 197: FAILED ceph_assert(cached_version > 0) 2024-05-27T07:29:03.952 INFO:journalctl@ceph.mds.k.smithi105.stdout:May 27 07:29:03 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-k[64614]: 2024-05-27T07:29:03.952 INFO:journalctl@ceph.mds.k.smithi105.stdout:May 27 07:29:03 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-k[64614]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:29:03.952 INFO:journalctl@ceph.mds.k.smithi105.stdout:May 27 07:29:03 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-k[64614]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x127) [0x7f93e6629ba6] 2024-05-27T07:29:03.952 INFO:journalctl@ceph.mds.k.smithi105.stdout:May 27 07:29:03 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-k[64614]: 2: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7f93e6629dd1] 2024-05-27T07:29:03.952 INFO:journalctl@ceph.mds.k.smithi105.stdout:May 27 07:29:03 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-k[64614]: 3: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x55b01f05023d] 2024-05-27T07:29:03.953 INFO:journalctl@ceph.mds.k.smithi105.stdout:May 27 07:29:03 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-k[64614]: 4: (SnapRealm::build_snap_set() const+0x323) [0x55b01f03b971] 2024-05-27T07:29:03.953 INFO:journalctl@ceph.mds.k.smithi105.stdout:May 27 07:29:03 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-k[64614]: 5: (SnapRealm::check_cache() const+0x13e) [0x55b01f039fa0] 2024-05-27T07:29:03.953 INFO:journalctl@ceph.mds.k.smithi105.stdout:May 27 07:29:03 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-k[64614]: 6: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x55b01f03a651] 2024-05-27T07:29:03.953 INFO:journalctl@ceph.mds.k.smithi105.stdout:May 27 07:29:03 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-k[64614]: 7: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x55b01efcf347] 2024-05-27T07:29:03.953 INFO:journalctl@ceph.mds.k.smithi105.stdout:May 27 07:29:03 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-k[64614]: 8: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x55b01eccb87e] 2024-05-27T07:29:03.953 INFO:journalctl@ceph.mds.k.smithi105.stdout:May 27 07:29:03 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-k[64614]: 9: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x2702) [0x55b01ecd0e94] 2024-05-27T07:29:03.953 INFO:journalctl@ceph.mds.k.smithi105.stdout:May 27 07:29:03 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-k[64614]: 10: (EUpdate::replay(MDSRank*)+0x63) [0x55b01ecdb969] 2024-05-27T07:29:03.953 INFO:journalctl@ceph.mds.k.smithi105.stdout:May 27 07:29:03 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-k[64614]: 11: (MDLog::_replay_thread()+0x1f38) [0x55b01f07448c] 2024-05-27T07:29:03.953 INFO:journalctl@ceph.mds.k.smithi105.stdout:May 27 07:29:03 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-k[64614]: 12: (MDLog::ReplayThread::entry()+0x11) [0x55b01f07d9cb] 2024-05-27T07:29:03.953 INFO:journalctl@ceph.mds.k.smithi105.stdout:May 27 07:29:03 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-k[64614]: 13: (Thread::entry_wrapper()+0x43) [0x7f93e65f872d] 2024-05-27T07:29:03.953 INFO:journalctl@ceph.mds.k.smithi105.stdout:May 27 07:29:03 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-k[64614]: 14: (Thread::_entry_func(void*)+0xd) [0x7f93e65f8749] 2024-05-27T07:29:03.953 INFO:journalctl@ceph.mds.k.smithi105.stdout:May 27 07:29:03 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-k[64614]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7f93e519c1ca] 2024-05-27T07:29:03.953 INFO:journalctl@ceph.mds.k.smithi105.stdout:May 27 07:29:03 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-k[64614]: 16: clone() 2024-05-27T07:29:03.954 INFO:journalctl@ceph.mds.k.smithi105.stdout:May 27 07:29:03 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-k[64614]: 2024-05-27T07:29:03.954 INFO:journalctl@ceph.mds.k.smithi105.stdout:May 27 07:29:03 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-k[64614]: 0> 2024-05-27T07:29:03.675+0000 7f93d3de4700 -1 *** Caught signal (Aborted) ** 2024-05-27T07:29:03.954 INFO:journalctl@ceph.mds.k.smithi105.stdout:May 27 07:29:03 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-k[64614]: in thread 7f93d3de4700 thread_name:md_log_replay 2024-05-27T07:29:03.954 INFO:journalctl@ceph.mds.k.smithi105.stdout:May 27 07:29:03 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-k[64614]: 2024-05-27T07:29:03.954 INFO:journalctl@ceph.mds.k.smithi105.stdout:May 27 07:29:03 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-k[64614]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:29:03.954 INFO:journalctl@ceph.mds.k.smithi105.stdout:May 27 07:29:03 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-k[64614]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f93e51a6d20] 2024-05-27T07:29:03.954 INFO:journalctl@ceph.mds.k.smithi105.stdout:May 27 07:29:03 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-k[64614]: 2: gsignal() 2024-05-27T07:29:03.954 INFO:journalctl@ceph.mds.k.smithi105.stdout:May 27 07:29:03 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-k[64614]: 3: abort() 2024-05-27T07:29:03.954 INFO:journalctl@ceph.mds.k.smithi105.stdout:May 27 07:29:03 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-k[64614]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x26e) [0x7f93e6629ced] 2024-05-27T07:29:03.954 INFO:journalctl@ceph.mds.k.smithi105.stdout:May 27 07:29:03 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-k[64614]: 5: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7f93e6629dd1] 2024-05-27T07:29:03.954 INFO:journalctl@ceph.mds.k.smithi105.stdout:May 27 07:29:03 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-k[64614]: 6: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x55b01f05023d] 2024-05-27T07:29:03.954 INFO:journalctl@ceph.mds.k.smithi105.stdout:May 27 07:29:03 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-k[64614]: 7: (SnapRealm::build_snap_set() const+0x323) [0x55b01f03b971] 2024-05-27T07:29:03.955 INFO:journalctl@ceph.mds.k.smithi105.stdout:May 27 07:29:03 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-k[64614]: 8: (SnapRealm::check_cache() const+0x13e) [0x55b01f039fa0] 2024-05-27T07:29:03.955 INFO:journalctl@ceph.mds.k.smithi105.stdout:May 27 07:29:03 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-k[64614]: 9: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x55b01f03a651] 2024-05-27T07:29:03.955 INFO:journalctl@ceph.mds.k.smithi105.stdout:May 27 07:29:03 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-k[64614]: 10: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x55b01efcf347] 2024-05-27T07:29:03.955 INFO:journalctl@ceph.mds.k.smithi105.stdout:May 27 07:29:03 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-k[64614]: 11: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x55b01eccb87e] 2024-05-27T07:29:03.955 INFO:journalctl@ceph.mds.k.smithi105.stdout:May 27 07:29:03 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-k[64614]: 12: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x2702) [0x55b01ecd0e94] 2024-05-27T07:29:03.955 INFO:journalctl@ceph.mds.k.smithi105.stdout:May 27 07:29:03 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-k[64614]: 13: (EUpdate::replay(MDSRank*)+0x63) [0x55b01ecdb969] 2024-05-27T07:29:03.955 INFO:journalctl@ceph.mds.k.smithi105.stdout:May 27 07:29:03 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-k[64614]: 14: (MDLog::_replay_thread()+0x1f38) [0x55b01f07448c] 2024-05-27T07:29:03.955 INFO:journalctl@ceph.mds.k.smithi105.stdout:May 27 07:29:03 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-k[64614]: 15: (MDLog::ReplayThread::entry()+0x11) [0x55b01f07d9cb] 2024-05-27T07:29:03.955 INFO:journalctl@ceph.mds.k.smithi105.stdout:May 27 07:29:03 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-k[64614]: 16: (Thread::entry_wrapper()+0x43) [0x7f93e65f872d] 2024-05-27T07:29:03.955 INFO:journalctl@ceph.mds.k.smithi105.stdout:May 27 07:29:03 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-k[64614]: 17: (Thread::_entry_func(void*)+0xd) [0x7f93e65f8749] 2024-05-27T07:29:03.955 INFO:journalctl@ceph.mds.k.smithi105.stdout:May 27 07:29:03 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-k[64614]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7f93e519c1ca] 2024-05-27T07:29:03.955 INFO:journalctl@ceph.mds.k.smithi105.stdout:May 27 07:29:03 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-k[64614]: 19: clone() 2024-05-27T07:29:03.955 INFO:journalctl@ceph.mds.k.smithi105.stdout:May 27 07:29:03 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-k[64614]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-05-27T07:29:03.955 INFO:journalctl@ceph.mds.k.smithi105.stdout:May 27 07:29:03 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-k[64614]: 2024-05-27T07:29:03.956 INFO:journalctl@ceph.mds.k.smithi105.stdout:May 27 07:29:03 smithi105 podman[69473]: 2024-05-27 07:29:03.746264087 +0000 UTC m=+0.028136285 container died 5d00409929d19fe1d05590684327698fc49e3239f33e4fc0ffad523c2ae62ab3 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-k, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, ceph=True, org.label-schema.build-date=20240521, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image) 2024-05-27T07:29:04.009 INFO:journalctl@ceph.mds.d.smithi052.stdout:May 27 07:29:03 smithi052 podman[104141]: 2024-05-27 07:29:03.663801194 +0000 UTC m=+0.018565565 image pull e36ea4fc8a5807e8d2701370e350a0de3856538797f7fba64cfce7e344bd6799 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6 2024-05-27T07:29:04.009 INFO:journalctl@ceph.mds.d.smithi052.stdout:May 27 07:29:03 smithi052 podman[104141]: 2024-05-27 07:29:03.782101225 +0000 UTC m=+0.136865578 container create 2a4922dadcb331b0acc74cb447d595f8a1728863ff4ff42c8ff52ffaead02447 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-d, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, GIT_BRANCH=HEAD, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, GIT_CLEAN=True, org.label-schema.build-date=20240521, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux ) 2024-05-27T07:29:04.009 INFO:journalctl@ceph.mds.d.smithi052.stdout:May 27 07:29:04 smithi052 podman[104141]: 2024-05-27 07:29:04.010452339 +0000 UTC m=+0.365216715 container init 2a4922dadcb331b0acc74cb447d595f8a1728863ff4ff42c8ff52ffaead02447 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-d, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, GIT_CLEAN=True, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240521, GIT_BRANCH=HEAD, ceph=True, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, io.buildah.version=1.33.5, org.label-schema.license=GPLv2) 2024-05-27T07:29:04.009 INFO:journalctl@ceph.mds.d.smithi052.stdout:May 27 07:29:04 smithi052 podman[104141]: 2024-05-27 07:29:04.016586765 +0000 UTC m=+0.371351133 container start 2a4922dadcb331b0acc74cb447d595f8a1728863ff4ff42c8ff52ffaead02447 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-d, org.label-schema.build-date=20240521, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, ceph=True, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, GIT_CLEAN=True, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=) 2024-05-27T07:29:04.414 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:29:04 smithi052 ceph-mon[31955]: pgmap v102: 129 pgs: 129 active+clean; 161 MiB data, 848 MiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 4.6 MiB/s wr, 1.28k op/s 2024-05-27T07:29:04.414 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:29:04 smithi052 ceph-mon[31955]: daemon mds.d restarted 2024-05-27T07:29:04.415 INFO:journalctl@ceph.mds.d.smithi052.stdout:May 27 07:29:04 smithi052 bash[104141]: 2a4922dadcb331b0acc74cb447d595f8a1728863ff4ff42c8ff52ffaead02447 2024-05-27T07:29:04.415 INFO:journalctl@ceph.mds.d.smithi052.stdout:May 27 07:29:04 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-d[104156]: starting mds.d at 2024-05-27T07:29:04.415 INFO:journalctl@ceph.mds.d.smithi052.stdout:May 27 07:29:04 smithi052 systemd[1]: Started Ceph mds.d for 73df2636-1bf8-11ef-bc9b-c7b262605968. 2024-05-27T07:29:04.415 INFO:journalctl@ceph.mds.d.smithi052.stdout:May 27 07:29:04 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-d[104156]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: In function 'std::set SnapClient::filter(const std::set&) const' thread 7f609d3e8700 time 2024-05-27T07:29:04.425259+0000 2024-05-27T07:29:04.415 INFO:journalctl@ceph.mds.d.smithi052.stdout:May 27 07:29:04 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-d[104156]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: 197: FAILED ceph_assert(cached_version > 0) 2024-05-27T07:29:04.439 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:29:04 smithi105 ceph-mon[35977]: pgmap v102: 129 pgs: 129 active+clean; 161 MiB data, 848 MiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 4.6 MiB/s wr, 1.28k op/s 2024-05-27T07:29:04.439 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:29:04 smithi105 ceph-mon[35977]: daemon mds.d restarted 2024-05-27T07:29:04.601 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:29:04 smithi173 ceph-mon[37000]: pgmap v102: 129 pgs: 129 active+clean; 161 MiB data, 848 MiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 4.6 MiB/s wr, 1.28k op/s 2024-05-27T07:29:04.602 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:29:04 smithi173 ceph-mon[37000]: daemon mds.d restarted 2024-05-27T07:29:04.730 INFO:journalctl@ceph.mds.d.smithi052.stdout:May 27 07:29:04 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-d[104156]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:29:04.730 INFO:journalctl@ceph.mds.d.smithi052.stdout:May 27 07:29:04 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-d[104156]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x127) [0x7f60afc2dba6] 2024-05-27T07:29:04.730 INFO:journalctl@ceph.mds.d.smithi052.stdout:May 27 07:29:04 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-d[104156]: 2: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7f60afc2ddd1] 2024-05-27T07:29:04.730 INFO:journalctl@ceph.mds.d.smithi052.stdout:May 27 07:29:04 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-d[104156]: 3: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x55ac73c1423d] 2024-05-27T07:29:04.730 INFO:journalctl@ceph.mds.d.smithi052.stdout:May 27 07:29:04 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-d[104156]: 4: (SnapRealm::build_snap_set() const+0x323) [0x55ac73bff971] 2024-05-27T07:29:04.730 INFO:journalctl@ceph.mds.d.smithi052.stdout:May 27 07:29:04 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-d[104156]: 5: (SnapRealm::check_cache() const+0x13e) [0x55ac73bfdfa0] 2024-05-27T07:29:04.731 INFO:journalctl@ceph.mds.d.smithi052.stdout:May 27 07:29:04 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-d[104156]: 6: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x55ac73bfe651] 2024-05-27T07:29:04.731 INFO:journalctl@ceph.mds.d.smithi052.stdout:May 27 07:29:04 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-d[104156]: 7: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x55ac73b93347] 2024-05-27T07:29:04.731 INFO:journalctl@ceph.mds.d.smithi052.stdout:May 27 07:29:04 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-d[104156]: 8: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x55ac7388f87e] 2024-05-27T07:29:04.731 INFO:journalctl@ceph.mds.d.smithi052.stdout:May 27 07:29:04 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-d[104156]: 9: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x2702) [0x55ac73894e94] 2024-05-27T07:29:04.731 INFO:journalctl@ceph.mds.d.smithi052.stdout:May 27 07:29:04 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-d[104156]: 10: (EUpdate::replay(MDSRank*)+0x63) [0x55ac7389f969] 2024-05-27T07:29:04.731 INFO:journalctl@ceph.mds.d.smithi052.stdout:May 27 07:29:04 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-d[104156]: 11: (MDLog::_replay_thread()+0x1f38) [0x55ac73c3848c] 2024-05-27T07:29:04.731 INFO:journalctl@ceph.mds.d.smithi052.stdout:May 27 07:29:04 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-d[104156]: 12: (MDLog::ReplayThread::entry()+0x11) [0x55ac73c419cb] 2024-05-27T07:29:04.731 INFO:journalctl@ceph.mds.d.smithi052.stdout:May 27 07:29:04 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-d[104156]: 13: (Thread::entry_wrapper()+0x43) [0x7f60afbfc72d] 2024-05-27T07:29:04.731 INFO:journalctl@ceph.mds.d.smithi052.stdout:May 27 07:29:04 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-d[104156]: 14: (Thread::_entry_func(void*)+0xd) [0x7f60afbfc749] 2024-05-27T07:29:04.731 INFO:journalctl@ceph.mds.d.smithi052.stdout:May 27 07:29:04 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-d[104156]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7f60ae7a01ca] 2024-05-27T07:29:04.731 INFO:journalctl@ceph.mds.d.smithi052.stdout:May 27 07:29:04 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-d[104156]: 16: clone() 2024-05-27T07:29:04.731 INFO:journalctl@ceph.mds.d.smithi052.stdout:May 27 07:29:04 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-d[104156]: 2024-05-27T07:29:04.437+0000 7f609d3e8700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: In function 'std::set SnapClient::filter(const std::set&) const' thread 7f609d3e8700 time 2024-05-27T07:29:04.425259+0000 2024-05-27T07:29:04.731 INFO:journalctl@ceph.mds.d.smithi052.stdout:May 27 07:29:04 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-d[104156]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: 197: FAILED ceph_assert(cached_version > 0) 2024-05-27T07:29:04.731 INFO:journalctl@ceph.mds.d.smithi052.stdout:May 27 07:29:04 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-d[104156]: 2024-05-27T07:29:04.732 INFO:journalctl@ceph.mds.d.smithi052.stdout:May 27 07:29:04 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-d[104156]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:29:04.732 INFO:journalctl@ceph.mds.d.smithi052.stdout:May 27 07:29:04 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-d[104156]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x127) [0x7f60afc2dba6] 2024-05-27T07:29:04.732 INFO:journalctl@ceph.mds.d.smithi052.stdout:May 27 07:29:04 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-d[104156]: 2: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7f60afc2ddd1] 2024-05-27T07:29:04.732 INFO:journalctl@ceph.mds.d.smithi052.stdout:May 27 07:29:04 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-d[104156]: 3: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x55ac73c1423d] 2024-05-27T07:29:04.732 INFO:journalctl@ceph.mds.d.smithi052.stdout:May 27 07:29:04 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-d[104156]: 4: (SnapRealm::build_snap_set() const+0x323) [0x55ac73bff971] 2024-05-27T07:29:04.732 INFO:journalctl@ceph.mds.d.smithi052.stdout:May 27 07:29:04 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-d[104156]: 5: (SnapRealm::check_cache() const+0x13e) [0x55ac73bfdfa0] 2024-05-27T07:29:04.732 INFO:journalctl@ceph.mds.d.smithi052.stdout:May 27 07:29:04 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-d[104156]: 6: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x55ac73bfe651] 2024-05-27T07:29:04.732 INFO:journalctl@ceph.mds.d.smithi052.stdout:May 27 07:29:04 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-d[104156]: 7: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x55ac73b93347] 2024-05-27T07:29:04.732 INFO:journalctl@ceph.mds.d.smithi052.stdout:May 27 07:29:04 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-d[104156]: 8: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x55ac7388f87e] 2024-05-27T07:29:04.733 INFO:journalctl@ceph.mds.d.smithi052.stdout:May 27 07:29:04 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-d[104156]: 9: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x2702) [0x55ac73894e94] 2024-05-27T07:29:04.733 INFO:journalctl@ceph.mds.d.smithi052.stdout:May 27 07:29:04 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-d[104156]: 10: (EUpdate::replay(MDSRank*)+0x63) [0x55ac7389f969] 2024-05-27T07:29:04.733 INFO:journalctl@ceph.mds.d.smithi052.stdout:May 27 07:29:04 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-d[104156]: 11: (MDLog::_replay_thread()+0x1f38) [0x55ac73c3848c] 2024-05-27T07:29:04.733 INFO:journalctl@ceph.mds.d.smithi052.stdout:May 27 07:29:04 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-d[104156]: 12: (MDLog::ReplayThread::entry()+0x11) [0x55ac73c419cb] 2024-05-27T07:29:04.733 INFO:journalctl@ceph.mds.d.smithi052.stdout:May 27 07:29:04 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-d[104156]: 13: (Thread::entry_wrapper()+0x43) [0x7f60afbfc72d] 2024-05-27T07:29:04.733 INFO:journalctl@ceph.mds.d.smithi052.stdout:May 27 07:29:04 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-d[104156]: 14: (Thread::_entry_func(void*)+0xd) [0x7f60afbfc749] 2024-05-27T07:29:04.733 INFO:journalctl@ceph.mds.d.smithi052.stdout:May 27 07:29:04 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-d[104156]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7f60ae7a01ca] 2024-05-27T07:29:04.733 INFO:journalctl@ceph.mds.d.smithi052.stdout:May 27 07:29:04 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-d[104156]: 16: clone() 2024-05-27T07:29:04.733 INFO:journalctl@ceph.mds.d.smithi052.stdout:May 27 07:29:04 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-d[104156]: 2024-05-27T07:29:04.733 INFO:journalctl@ceph.mds.d.smithi052.stdout:May 27 07:29:04 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-d[104156]: *** Caught signal (Aborted) ** 2024-05-27T07:29:04.733 INFO:journalctl@ceph.mds.d.smithi052.stdout:May 27 07:29:04 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-d[104156]: in thread 7f609d3e8700 thread_name:md_log_replay 2024-05-27T07:29:04.733 INFO:journalctl@ceph.mds.d.smithi052.stdout:May 27 07:29:04 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-d[104156]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:29:04.733 INFO:journalctl@ceph.mds.d.smithi052.stdout:May 27 07:29:04 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-d[104156]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f60ae7aad20] 2024-05-27T07:29:04.734 INFO:journalctl@ceph.mds.d.smithi052.stdout:May 27 07:29:04 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-d[104156]: 2: gsignal() 2024-05-27T07:29:04.734 INFO:journalctl@ceph.mds.d.smithi052.stdout:May 27 07:29:04 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-d[104156]: 3: abort() 2024-05-27T07:29:04.734 INFO:journalctl@ceph.mds.d.smithi052.stdout:May 27 07:29:04 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-d[104156]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x26e) [0x7f60afc2dced] 2024-05-27T07:29:04.734 INFO:journalctl@ceph.mds.d.smithi052.stdout:May 27 07:29:04 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-d[104156]: 5: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7f60afc2ddd1] 2024-05-27T07:29:04.734 INFO:journalctl@ceph.mds.d.smithi052.stdout:May 27 07:29:04 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-d[104156]: 6: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x55ac73c1423d] 2024-05-27T07:29:04.734 INFO:journalctl@ceph.mds.d.smithi052.stdout:May 27 07:29:04 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-d[104156]: 7: (SnapRealm::build_snap_set() const+0x323) [0x55ac73bff971] 2024-05-27T07:29:04.734 INFO:journalctl@ceph.mds.d.smithi052.stdout:May 27 07:29:04 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-d[104156]: 8: (SnapRealm::check_cache() const+0x13e) [0x55ac73bfdfa0] 2024-05-27T07:29:04.734 INFO:journalctl@ceph.mds.d.smithi052.stdout:May 27 07:29:04 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-d[104156]: 9: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x55ac73bfe651] 2024-05-27T07:29:04.735 INFO:journalctl@ceph.mds.d.smithi052.stdout:May 27 07:29:04 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-d[104156]: 10: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x55ac73b93347] 2024-05-27T07:29:04.735 INFO:journalctl@ceph.mds.d.smithi052.stdout:May 27 07:29:04 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-d[104156]: 11: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x55ac7388f87e] 2024-05-27T07:29:04.735 INFO:journalctl@ceph.mds.d.smithi052.stdout:May 27 07:29:04 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-d[104156]: 12: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x2702) [0x55ac73894e94] 2024-05-27T07:29:04.735 INFO:journalctl@ceph.mds.d.smithi052.stdout:May 27 07:29:04 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-d[104156]: 13: (EUpdate::replay(MDSRank*)+0x63) [0x55ac7389f969] 2024-05-27T07:29:04.735 INFO:journalctl@ceph.mds.d.smithi052.stdout:May 27 07:29:04 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-d[104156]: 14: (MDLog::_replay_thread()+0x1f38) [0x55ac73c3848c] 2024-05-27T07:29:04.735 INFO:journalctl@ceph.mds.d.smithi052.stdout:May 27 07:29:04 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-d[104156]: 15: (MDLog::ReplayThread::entry()+0x11) [0x55ac73c419cb] 2024-05-27T07:29:04.735 INFO:journalctl@ceph.mds.d.smithi052.stdout:May 27 07:29:04 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-d[104156]: 16: (Thread::entry_wrapper()+0x43) [0x7f60afbfc72d] 2024-05-27T07:29:04.735 INFO:journalctl@ceph.mds.d.smithi052.stdout:May 27 07:29:04 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-d[104156]: 17: (Thread::_entry_func(void*)+0xd) [0x7f60afbfc749] 2024-05-27T07:29:04.735 INFO:journalctl@ceph.mds.d.smithi052.stdout:May 27 07:29:04 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-d[104156]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7f60ae7a01ca] 2024-05-27T07:29:04.735 INFO:journalctl@ceph.mds.d.smithi052.stdout:May 27 07:29:04 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-d[104156]: 19: clone() 2024-05-27T07:29:04.736 INFO:journalctl@ceph.mds.d.smithi052.stdout:May 27 07:29:04 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-d[104156]: 2024-05-27T07:29:04.444+0000 7f609d3e8700 -1 *** Caught signal (Aborted) ** 2024-05-27T07:29:04.736 INFO:journalctl@ceph.mds.d.smithi052.stdout:May 27 07:29:04 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-d[104156]: in thread 7f609d3e8700 thread_name:md_log_replay 2024-05-27T07:29:04.736 INFO:journalctl@ceph.mds.d.smithi052.stdout:May 27 07:29:04 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-d[104156]: 2024-05-27T07:29:04.736 INFO:journalctl@ceph.mds.d.smithi052.stdout:May 27 07:29:04 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-d[104156]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:29:04.736 INFO:journalctl@ceph.mds.d.smithi052.stdout:May 27 07:29:04 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-d[104156]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f60ae7aad20] 2024-05-27T07:29:04.736 INFO:journalctl@ceph.mds.d.smithi052.stdout:May 27 07:29:04 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-d[104156]: 2: gsignal() 2024-05-27T07:29:04.736 INFO:journalctl@ceph.mds.d.smithi052.stdout:May 27 07:29:04 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-d[104156]: 3: abort() 2024-05-27T07:29:04.736 INFO:journalctl@ceph.mds.d.smithi052.stdout:May 27 07:29:04 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-d[104156]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x26e) [0x7f60afc2dced] 2024-05-27T07:29:04.736 INFO:journalctl@ceph.mds.d.smithi052.stdout:May 27 07:29:04 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-d[104156]: 5: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7f60afc2ddd1] 2024-05-27T07:29:04.736 INFO:journalctl@ceph.mds.d.smithi052.stdout:May 27 07:29:04 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-d[104156]: 6: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x55ac73c1423d] 2024-05-27T07:29:04.736 INFO:journalctl@ceph.mds.d.smithi052.stdout:May 27 07:29:04 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-d[104156]: 7: (SnapRealm::build_snap_set() const+0x323) [0x55ac73bff971] 2024-05-27T07:29:04.737 INFO:journalctl@ceph.mds.d.smithi052.stdout:May 27 07:29:04 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-d[104156]: 8: (SnapRealm::check_cache() const+0x13e) [0x55ac73bfdfa0] 2024-05-27T07:29:04.737 INFO:journalctl@ceph.mds.d.smithi052.stdout:May 27 07:29:04 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-d[104156]: 9: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x55ac73bfe651] 2024-05-27T07:29:04.737 INFO:journalctl@ceph.mds.d.smithi052.stdout:May 27 07:29:04 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-d[104156]: 10: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x55ac73b93347] 2024-05-27T07:29:04.737 INFO:journalctl@ceph.mds.d.smithi052.stdout:May 27 07:29:04 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-d[104156]: 11: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x55ac7388f87e] 2024-05-27T07:29:04.737 INFO:journalctl@ceph.mds.d.smithi052.stdout:May 27 07:29:04 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-d[104156]: 12: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x2702) [0x55ac73894e94] 2024-05-27T07:29:04.737 INFO:journalctl@ceph.mds.d.smithi052.stdout:May 27 07:29:04 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-d[104156]: 13: (EUpdate::replay(MDSRank*)+0x63) [0x55ac7389f969] 2024-05-27T07:29:04.737 INFO:journalctl@ceph.mds.d.smithi052.stdout:May 27 07:29:04 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-d[104156]: 14: (MDLog::_replay_thread()+0x1f38) [0x55ac73c3848c] 2024-05-27T07:29:04.737 INFO:journalctl@ceph.mds.d.smithi052.stdout:May 27 07:29:04 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-d[104156]: 15: (MDLog::ReplayThread::entry()+0x11) [0x55ac73c419cb] 2024-05-27T07:29:04.737 INFO:journalctl@ceph.mds.d.smithi052.stdout:May 27 07:29:04 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-d[104156]: 16: (Thread::entry_wrapper()+0x43) [0x7f60afbfc72d] 2024-05-27T07:29:04.737 INFO:journalctl@ceph.mds.d.smithi052.stdout:May 27 07:29:04 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-d[104156]: 17: (Thread::_entry_func(void*)+0xd) [0x7f60afbfc749] 2024-05-27T07:29:04.737 INFO:journalctl@ceph.mds.d.smithi052.stdout:May 27 07:29:04 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-d[104156]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7f60ae7a01ca] 2024-05-27T07:29:04.737 INFO:journalctl@ceph.mds.d.smithi052.stdout:May 27 07:29:04 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-d[104156]: 19: clone() 2024-05-27T07:29:04.738 INFO:journalctl@ceph.mds.d.smithi052.stdout:May 27 07:29:04 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-d[104156]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-05-27T07:29:04.738 INFO:journalctl@ceph.mds.d.smithi052.stdout:May 27 07:29:04 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-d[104156]: 2024-05-27T07:29:04.738 INFO:journalctl@ceph.mds.d.smithi052.stdout:May 27 07:29:04 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-d[104156]: -1> 2024-05-27T07:29:04.437+0000 7f609d3e8700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: In function 'std::set SnapClient::filter(const std::set&) const' thread 7f609d3e8700 time 2024-05-27T07:29:04.425259+0000 2024-05-27T07:29:04.738 INFO:journalctl@ceph.mds.d.smithi052.stdout:May 27 07:29:04 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-d[104156]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: 197: FAILED ceph_assert(cached_version > 0) 2024-05-27T07:29:04.738 INFO:journalctl@ceph.mds.d.smithi052.stdout:May 27 07:29:04 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-d[104156]: 2024-05-27T07:29:04.738 INFO:journalctl@ceph.mds.d.smithi052.stdout:May 27 07:29:04 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-d[104156]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:29:04.738 INFO:journalctl@ceph.mds.d.smithi052.stdout:May 27 07:29:04 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-d[104156]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x127) [0x7f60afc2dba6] 2024-05-27T07:29:04.738 INFO:journalctl@ceph.mds.d.smithi052.stdout:May 27 07:29:04 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-d[104156]: 2: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7f60afc2ddd1] 2024-05-27T07:29:04.738 INFO:journalctl@ceph.mds.d.smithi052.stdout:May 27 07:29:04 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-d[104156]: 3: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x55ac73c1423d] 2024-05-27T07:29:04.738 INFO:journalctl@ceph.mds.d.smithi052.stdout:May 27 07:29:04 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-d[104156]: 4: (SnapRealm::build_snap_set() const+0x323) [0x55ac73bff971] 2024-05-27T07:29:04.738 INFO:journalctl@ceph.mds.d.smithi052.stdout:May 27 07:29:04 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-d[104156]: 5: (SnapRealm::check_cache() const+0x13e) [0x55ac73bfdfa0] 2024-05-27T07:29:04.738 INFO:journalctl@ceph.mds.d.smithi052.stdout:May 27 07:29:04 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-d[104156]: 6: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x55ac73bfe651] 2024-05-27T07:29:04.738 INFO:journalctl@ceph.mds.d.smithi052.stdout:May 27 07:29:04 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-d[104156]: 7: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x55ac73b93347] 2024-05-27T07:29:04.739 INFO:journalctl@ceph.mds.d.smithi052.stdout:May 27 07:29:04 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-d[104156]: 8: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x55ac7388f87e] 2024-05-27T07:29:04.739 INFO:journalctl@ceph.mds.d.smithi052.stdout:May 27 07:29:04 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-d[104156]: 9: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x2702) [0x55ac73894e94] 2024-05-27T07:29:04.739 INFO:journalctl@ceph.mds.d.smithi052.stdout:May 27 07:29:04 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-d[104156]: 10: (EUpdate::replay(MDSRank*)+0x63) [0x55ac7389f969] 2024-05-27T07:29:04.739 INFO:journalctl@ceph.mds.d.smithi052.stdout:May 27 07:29:04 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-d[104156]: 11: (MDLog::_replay_thread()+0x1f38) [0x55ac73c3848c] 2024-05-27T07:29:04.739 INFO:journalctl@ceph.mds.d.smithi052.stdout:May 27 07:29:04 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-d[104156]: 12: (MDLog::ReplayThread::entry()+0x11) [0x55ac73c419cb] 2024-05-27T07:29:04.739 INFO:journalctl@ceph.mds.d.smithi052.stdout:May 27 07:29:04 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-d[104156]: 13: (Thread::entry_wrapper()+0x43) [0x7f60afbfc72d] 2024-05-27T07:29:04.739 INFO:journalctl@ceph.mds.d.smithi052.stdout:May 27 07:29:04 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-d[104156]: 14: (Thread::_entry_func(void*)+0xd) [0x7f60afbfc749] 2024-05-27T07:29:04.739 INFO:journalctl@ceph.mds.d.smithi052.stdout:May 27 07:29:04 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-d[104156]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7f60ae7a01ca] 2024-05-27T07:29:04.739 INFO:journalctl@ceph.mds.d.smithi052.stdout:May 27 07:29:04 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-d[104156]: 16: clone() 2024-05-27T07:29:04.739 INFO:journalctl@ceph.mds.d.smithi052.stdout:May 27 07:29:04 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-d[104156]: 2024-05-27T07:29:04.739 INFO:journalctl@ceph.mds.d.smithi052.stdout:May 27 07:29:04 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-d[104156]: 0> 2024-05-27T07:29:04.444+0000 7f609d3e8700 -1 *** Caught signal (Aborted) ** 2024-05-27T07:29:04.739 INFO:journalctl@ceph.mds.d.smithi052.stdout:May 27 07:29:04 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-d[104156]: in thread 7f609d3e8700 thread_name:md_log_replay 2024-05-27T07:29:04.739 INFO:journalctl@ceph.mds.d.smithi052.stdout:May 27 07:29:04 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-d[104156]: 2024-05-27T07:29:04.740 INFO:journalctl@ceph.mds.d.smithi052.stdout:May 27 07:29:04 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-d[104156]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:29:04.740 INFO:journalctl@ceph.mds.d.smithi052.stdout:May 27 07:29:04 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-d[104156]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f60ae7aad20] 2024-05-27T07:29:04.740 INFO:journalctl@ceph.mds.d.smithi052.stdout:May 27 07:29:04 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-d[104156]: 2: gsignal() 2024-05-27T07:29:04.740 INFO:journalctl@ceph.mds.d.smithi052.stdout:May 27 07:29:04 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-d[104156]: 3: abort() 2024-05-27T07:29:04.740 INFO:journalctl@ceph.mds.d.smithi052.stdout:May 27 07:29:04 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-d[104156]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x26e) [0x7f60afc2dced] 2024-05-27T07:29:04.740 INFO:journalctl@ceph.mds.d.smithi052.stdout:May 27 07:29:04 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-d[104156]: 5: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7f60afc2ddd1] 2024-05-27T07:29:04.740 INFO:journalctl@ceph.mds.d.smithi052.stdout:May 27 07:29:04 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-d[104156]: 6: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x55ac73c1423d] 2024-05-27T07:29:04.740 INFO:journalctl@ceph.mds.d.smithi052.stdout:May 27 07:29:04 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-d[104156]: 7: (SnapRealm::build_snap_set() const+0x323) [0x55ac73bff971] 2024-05-27T07:29:04.740 INFO:journalctl@ceph.mds.d.smithi052.stdout:May 27 07:29:04 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-d[104156]: 8: (SnapRealm::check_cache() const+0x13e) [0x55ac73bfdfa0] 2024-05-27T07:29:04.740 INFO:journalctl@ceph.mds.d.smithi052.stdout:May 27 07:29:04 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-d[104156]: 9: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x55ac73bfe651] 2024-05-27T07:29:04.740 INFO:journalctl@ceph.mds.d.smithi052.stdout:May 27 07:29:04 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-d[104156]: 10: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x55ac73b93347] 2024-05-27T07:29:04.740 INFO:journalctl@ceph.mds.d.smithi052.stdout:May 27 07:29:04 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-d[104156]: 11: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x55ac7388f87e] 2024-05-27T07:29:04.740 INFO:journalctl@ceph.mds.d.smithi052.stdout:May 27 07:29:04 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-d[104156]: 12: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x2702) [0x55ac73894e94] 2024-05-27T07:29:04.741 INFO:journalctl@ceph.mds.d.smithi052.stdout:May 27 07:29:04 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-d[104156]: 13: (EUpdate::replay(MDSRank*)+0x63) [0x55ac7389f969] 2024-05-27T07:29:04.741 INFO:journalctl@ceph.mds.d.smithi052.stdout:May 27 07:29:04 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-d[104156]: 14: (MDLog::_replay_thread()+0x1f38) [0x55ac73c3848c] 2024-05-27T07:29:04.741 INFO:journalctl@ceph.mds.d.smithi052.stdout:May 27 07:29:04 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-d[104156]: 15: (MDLog::ReplayThread::entry()+0x11) [0x55ac73c419cb] 2024-05-27T07:29:04.741 INFO:journalctl@ceph.mds.d.smithi052.stdout:May 27 07:29:04 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-d[104156]: 16: (Thread::entry_wrapper()+0x43) [0x7f60afbfc72d] 2024-05-27T07:29:04.741 INFO:journalctl@ceph.mds.d.smithi052.stdout:May 27 07:29:04 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-d[104156]: 17: (Thread::_entry_func(void*)+0xd) [0x7f60afbfc749] 2024-05-27T07:29:04.741 INFO:journalctl@ceph.mds.d.smithi052.stdout:May 27 07:29:04 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-d[104156]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7f60ae7a01ca] 2024-05-27T07:29:04.741 INFO:journalctl@ceph.mds.d.smithi052.stdout:May 27 07:29:04 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-d[104156]: 19: clone() 2024-05-27T07:29:04.741 INFO:journalctl@ceph.mds.d.smithi052.stdout:May 27 07:29:04 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-d[104156]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-05-27T07:29:04.741 INFO:journalctl@ceph.mds.d.smithi052.stdout:May 27 07:29:04 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-d[104156]: 2024-05-27T07:29:04.742 INFO:journalctl@ceph.mds.d.smithi052.stdout:May 27 07:29:04 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-d[104156]: -1> 2024-05-27T07:29:04.437+0000 7f609d3e8700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: In function 'std::set SnapClient::filter(const std::set&) const' thread 7f609d3e8700 time 2024-05-27T07:29:04.425259+0000 2024-05-27T07:29:04.742 INFO:journalctl@ceph.mds.d.smithi052.stdout:May 27 07:29:04 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-d[104156]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: 197: FAILED ceph_assert(cached_version > 0) 2024-05-27T07:29:04.742 INFO:journalctl@ceph.mds.d.smithi052.stdout:May 27 07:29:04 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-d[104156]: 2024-05-27T07:29:04.742 INFO:journalctl@ceph.mds.d.smithi052.stdout:May 27 07:29:04 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-d[104156]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:29:04.742 INFO:journalctl@ceph.mds.d.smithi052.stdout:May 27 07:29:04 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-d[104156]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x127) [0x7f60afc2dba6] 2024-05-27T07:29:04.742 INFO:journalctl@ceph.mds.d.smithi052.stdout:May 27 07:29:04 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-d[104156]: 2: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7f60afc2ddd1] 2024-05-27T07:29:04.742 INFO:journalctl@ceph.mds.d.smithi052.stdout:May 27 07:29:04 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-d[104156]: 3: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x55ac73c1423d] 2024-05-27T07:29:04.742 INFO:journalctl@ceph.mds.d.smithi052.stdout:May 27 07:29:04 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-d[104156]: 4: (SnapRealm::build_snap_set() const+0x323) [0x55ac73bff971] 2024-05-27T07:29:04.742 INFO:journalctl@ceph.mds.d.smithi052.stdout:May 27 07:29:04 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-d[104156]: 5: (SnapRealm::check_cache() const+0x13e) [0x55ac73bfdfa0] 2024-05-27T07:29:04.742 INFO:journalctl@ceph.mds.d.smithi052.stdout:May 27 07:29:04 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-d[104156]: 6: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x55ac73bfe651] 2024-05-27T07:29:04.742 INFO:journalctl@ceph.mds.d.smithi052.stdout:May 27 07:29:04 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-d[104156]: 7: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x55ac73b93347] 2024-05-27T07:29:04.743 INFO:journalctl@ceph.mds.d.smithi052.stdout:May 27 07:29:04 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-d[104156]: 8: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x55ac7388f87e] 2024-05-27T07:29:04.743 INFO:journalctl@ceph.mds.d.smithi052.stdout:May 27 07:29:04 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-d[104156]: 9: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x2702) [0x55ac73894e94] 2024-05-27T07:29:04.743 INFO:journalctl@ceph.mds.d.smithi052.stdout:May 27 07:29:04 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-d[104156]: 10: (EUpdate::replay(MDSRank*)+0x63) [0x55ac7389f969] 2024-05-27T07:29:04.743 INFO:journalctl@ceph.mds.d.smithi052.stdout:May 27 07:29:04 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-d[104156]: 11: (MDLog::_replay_thread()+0x1f38) [0x55ac73c3848c] 2024-05-27T07:29:04.743 INFO:journalctl@ceph.mds.d.smithi052.stdout:May 27 07:29:04 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-d[104156]: 12: (MDLog::ReplayThread::entry()+0x11) [0x55ac73c419cb] 2024-05-27T07:29:04.743 INFO:journalctl@ceph.mds.d.smithi052.stdout:May 27 07:29:04 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-d[104156]: 13: (Thread::entry_wrapper()+0x43) [0x7f60afbfc72d] 2024-05-27T07:29:04.743 INFO:journalctl@ceph.mds.d.smithi052.stdout:May 27 07:29:04 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-d[104156]: 14: (Thread::_entry_func(void*)+0xd) [0x7f60afbfc749] 2024-05-27T07:29:04.743 INFO:journalctl@ceph.mds.d.smithi052.stdout:May 27 07:29:04 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-d[104156]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7f60ae7a01ca] 2024-05-27T07:29:04.743 INFO:journalctl@ceph.mds.d.smithi052.stdout:May 27 07:29:04 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-d[104156]: 16: clone() 2024-05-27T07:29:04.743 INFO:journalctl@ceph.mds.d.smithi052.stdout:May 27 07:29:04 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-d[104156]: 2024-05-27T07:29:04.743 INFO:journalctl@ceph.mds.d.smithi052.stdout:May 27 07:29:04 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-d[104156]: 0> 2024-05-27T07:29:04.444+0000 7f609d3e8700 -1 *** Caught signal (Aborted) ** 2024-05-27T07:29:04.743 INFO:journalctl@ceph.mds.d.smithi052.stdout:May 27 07:29:04 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-d[104156]: in thread 7f609d3e8700 thread_name:md_log_replay 2024-05-27T07:29:04.743 INFO:journalctl@ceph.mds.d.smithi052.stdout:May 27 07:29:04 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-d[104156]: 2024-05-27T07:29:04.743 INFO:journalctl@ceph.mds.d.smithi052.stdout:May 27 07:29:04 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-d[104156]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:29:04.743 INFO:journalctl@ceph.mds.d.smithi052.stdout:May 27 07:29:04 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-d[104156]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f60ae7aad20] 2024-05-27T07:29:04.744 INFO:journalctl@ceph.mds.d.smithi052.stdout:May 27 07:29:04 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-d[104156]: 2: gsignal() 2024-05-27T07:29:04.744 INFO:journalctl@ceph.mds.d.smithi052.stdout:May 27 07:29:04 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-d[104156]: 3: abort() 2024-05-27T07:29:04.744 INFO:journalctl@ceph.mds.d.smithi052.stdout:May 27 07:29:04 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-d[104156]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x26e) [0x7f60afc2dced] 2024-05-27T07:29:04.744 INFO:journalctl@ceph.mds.d.smithi052.stdout:May 27 07:29:04 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-d[104156]: 5: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7f60afc2ddd1] 2024-05-27T07:29:04.744 INFO:journalctl@ceph.mds.d.smithi052.stdout:May 27 07:29:04 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-d[104156]: 6: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x55ac73c1423d] 2024-05-27T07:29:04.744 INFO:journalctl@ceph.mds.d.smithi052.stdout:May 27 07:29:04 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-d[104156]: 7: (SnapRealm::build_snap_set() const+0x323) [0x55ac73bff971] 2024-05-27T07:29:04.744 INFO:journalctl@ceph.mds.d.smithi052.stdout:May 27 07:29:04 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-d[104156]: 8: (SnapRealm::check_cache() const+0x13e) [0x55ac73bfdfa0] 2024-05-27T07:29:04.744 INFO:journalctl@ceph.mds.d.smithi052.stdout:May 27 07:29:04 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-d[104156]: 9: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x55ac73bfe651] 2024-05-27T07:29:04.744 INFO:journalctl@ceph.mds.d.smithi052.stdout:May 27 07:29:04 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-d[104156]: 10: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x55ac73b93347] 2024-05-27T07:29:04.744 INFO:journalctl@ceph.mds.d.smithi052.stdout:May 27 07:29:04 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-d[104156]: 11: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x55ac7388f87e] 2024-05-27T07:29:04.744 INFO:journalctl@ceph.mds.d.smithi052.stdout:May 27 07:29:04 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-d[104156]: 12: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x2702) [0x55ac73894e94] 2024-05-27T07:29:04.744 INFO:journalctl@ceph.mds.d.smithi052.stdout:May 27 07:29:04 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-d[104156]: 13: (EUpdate::replay(MDSRank*)+0x63) [0x55ac7389f969] 2024-05-27T07:29:04.744 INFO:journalctl@ceph.mds.d.smithi052.stdout:May 27 07:29:04 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-d[104156]: 14: (MDLog::_replay_thread()+0x1f38) [0x55ac73c3848c] 2024-05-27T07:29:04.744 INFO:journalctl@ceph.mds.d.smithi052.stdout:May 27 07:29:04 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-d[104156]: 15: (MDLog::ReplayThread::entry()+0x11) [0x55ac73c419cb] 2024-05-27T07:29:04.744 INFO:journalctl@ceph.mds.d.smithi052.stdout:May 27 07:29:04 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-d[104156]: 16: (Thread::entry_wrapper()+0x43) [0x7f60afbfc72d] 2024-05-27T07:29:04.745 INFO:journalctl@ceph.mds.d.smithi052.stdout:May 27 07:29:04 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-d[104156]: 17: (Thread::_entry_func(void*)+0xd) [0x7f60afbfc749] 2024-05-27T07:29:04.745 INFO:journalctl@ceph.mds.d.smithi052.stdout:May 27 07:29:04 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-d[104156]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7f60ae7a01ca] 2024-05-27T07:29:04.745 INFO:journalctl@ceph.mds.d.smithi052.stdout:May 27 07:29:04 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-d[104156]: 19: clone() 2024-05-27T07:29:04.745 INFO:journalctl@ceph.mds.d.smithi052.stdout:May 27 07:29:04 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-d[104156]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-05-27T07:29:04.745 INFO:journalctl@ceph.mds.d.smithi052.stdout:May 27 07:29:04 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-d[104156]: 2024-05-27T07:29:04.745 INFO:journalctl@ceph.mds.d.smithi052.stdout:May 27 07:29:04 smithi052 podman[104191]: 2024-05-27 07:29:04.62589572 +0000 UTC m=+0.097594981 container died 2a4922dadcb331b0acc74cb447d595f8a1728863ff4ff42c8ff52ffaead02447 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-d, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.build-date=20240521, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.license=GPLv2) 2024-05-27T07:29:05.059 INFO:journalctl@ceph.mds.k.smithi105.stdout:May 27 07:29:04 smithi105 podman[69473]: 2024-05-27 07:29:04.786640648 +0000 UTC m=+1.068512847 container remove 5d00409929d19fe1d05590684327698fc49e3239f33e4fc0ffad523c2ae62ab3 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-k, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, ceph=True, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, org.label-schema.build-date=20240521, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, GIT_BRANCH=HEAD) 2024-05-27T07:29:05.059 INFO:journalctl@ceph.mds.k.smithi105.stdout:May 27 07:29:04 smithi105 systemd[1]: ceph-73df2636-1bf8-11ef-bc9b-c7b262605968@mds.k.service: Main process exited, code=exited, status=134/n/a 2024-05-27T07:29:05.205 INFO:tasks.workunit.client.0.smithi052.stdout: 47 938092 436 671198 381 592288 1387 2024-05-27T07:29:05.229 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:29:05 smithi052 ceph-mon[31955]: osdmap e98: 12 total, 12 up, 12 in 2024-05-27T07:29:05.229 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:29:05 smithi052 ceph-mon[31955]: mds.? [v2:172.21.15.52:6838/1647772367,v1:172.21.15.52:6839/1647772367] up:boot 2024-05-27T07:29:05.229 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:29:05 smithi052 ceph-mon[31955]: fsmap cephfs:5 4 up:standby-replay 3 up:standby 5 up:active 2024-05-27T07:29:05.229 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:29:05 smithi052 ceph-mon[31955]: from='mgr.15333 172.21.15.52:0/416389082' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "d"}]: dispatch 2024-05-27T07:29:05.229 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:29:05 smithi052 ceph-mon[31955]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-05-27T07:29:05.439 INFO:journalctl@ceph.mds.k.smithi105.stdout:May 27 07:29:05 smithi105 systemd[1]: ceph-73df2636-1bf8-11ef-bc9b-c7b262605968@mds.k.service: Failed with result 'exit-code'. 2024-05-27T07:29:05.440 INFO:journalctl@ceph.mds.k.smithi105.stdout:May 27 07:29:05 smithi105 systemd[1]: ceph-73df2636-1bf8-11ef-bc9b-c7b262605968@mds.k.service: Consumed 1.428s CPU time. 2024-05-27T07:29:05.440 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:29:05 smithi105 ceph-mon[35977]: osdmap e98: 12 total, 12 up, 12 in 2024-05-27T07:29:05.440 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:29:05 smithi105 ceph-mon[35977]: mds.? [v2:172.21.15.52:6838/1647772367,v1:172.21.15.52:6839/1647772367] up:boot 2024-05-27T07:29:05.440 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:29:05 smithi105 ceph-mon[35977]: fsmap cephfs:5 4 up:standby-replay 3 up:standby 5 up:active 2024-05-27T07:29:05.440 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:29:05 smithi105 ceph-mon[35977]: from='mgr.15333 172.21.15.52:0/416389082' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "d"}]: dispatch 2024-05-27T07:29:05.440 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:29:05 smithi105 ceph-mon[35977]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-05-27T07:29:05.602 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:29:05 smithi173 ceph-mon[37000]: osdmap e98: 12 total, 12 up, 12 in 2024-05-27T07:29:05.602 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:29:05 smithi173 ceph-mon[37000]: mds.? [v2:172.21.15.52:6838/1647772367,v1:172.21.15.52:6839/1647772367] up:boot 2024-05-27T07:29:05.602 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:29:05 smithi173 ceph-mon[37000]: fsmap cephfs:5 4 up:standby-replay 3 up:standby 5 up:active 2024-05-27T07:29:05.602 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:29:05 smithi173 ceph-mon[37000]: from='mgr.15333 172.21.15.52:0/416389082' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "d"}]: dispatch 2024-05-27T07:29:05.602 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:29:05 smithi173 ceph-mon[37000]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-05-27T07:29:05.667 INFO:journalctl@ceph.mds.d.smithi052.stdout:May 27 07:29:05 smithi052 podman[104191]: 2024-05-27 07:29:05.375102406 +0000 UTC m=+0.846801649 container remove 2a4922dadcb331b0acc74cb447d595f8a1728863ff4ff42c8ff52ffaead02447 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-d, org.label-schema.vendor=CentOS, GIT_CLEAN=True, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, ceph=True, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.build-date=20240521, CEPH_POINT_RELEASE=) 2024-05-27T07:29:05.668 INFO:journalctl@ceph.mds.d.smithi052.stdout:May 27 07:29:05 smithi052 systemd[1]: ceph-73df2636-1bf8-11ef-bc9b-c7b262605968@mds.d.service: Main process exited, code=exited, status=134/n/a 2024-05-27T07:29:05.783 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:29:05.793+0000 7f704f37d640 1 -- 172.21.15.52:0/3214730353 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f7048109040 msgr2=0x7f7048102c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:29:05.783 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:29:05.793+0000 7f704f37d640 1 --2- 172.21.15.52:0/3214730353 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f7048109040 0x7f7048102c80 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7f7038009860 tx=0x7f703802a340 comp rx=0 tx=0).stop 2024-05-27T07:29:05.783 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:29:05.793+0000 7f704f37d640 1 -- 172.21.15.52:0/3214730353 shutdown_connections 2024-05-27T07:29:05.783 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:29:05.793+0000 7f704f37d640 1 --2- 172.21.15.52:0/3214730353 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f70481032f0 0x7f704810f6c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:29:05.783 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:29:05.793+0000 7f704f37d640 1 --2- 172.21.15.52:0/3214730353 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f7048109040 0x7f7048102c80 unknown :-1 s=CLOSED pgs=114 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:29:05.784 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:29:05.793+0000 7f704f37d640 1 --2- 172.21.15.52:0/3214730353 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f7048108ba0 0x7f7048102740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:29:05.784 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:29:05.793+0000 7f704f37d640 1 -- 172.21.15.52:0/3214730353 >> 172.21.15.52:0/3214730353 conn(0x7f7048078270 msgr2=0x7f70480786b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:29:05.784 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:29:05.793+0000 7f704f37d640 1 -- 172.21.15.52:0/3214730353 shutdown_connections 2024-05-27T07:29:05.784 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:29:05.793+0000 7f704f37d640 1 -- 172.21.15.52:0/3214730353 wait complete. 2024-05-27T07:29:05.784 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:29:05.794+0000 7f704f37d640 1 Processor -- start 2024-05-27T07:29:05.785 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:29:05.794+0000 7f704f37d640 1 -- start start 2024-05-27T07:29:05.785 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:29:05.795+0000 7f704f37d640 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f70481032f0 0x7f70481994c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:29:05.785 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:29:05.795+0000 7f704f37d640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f7048108ba0 0x7f7048199a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:29:05.785 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:29:05.795+0000 7f704f37d640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f7048109040 0x7f7048199f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:29:05.785 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:29:05.795+0000 7f704f37d640 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x7f7048112d90 con 0x7f70481032f0 2024-05-27T07:29:05.785 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:29:05.795+0000 7f704f37d640 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f7048112c10 con 0x7f7048109040 2024-05-27T07:29:05.785 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:29:05.795+0000 7f704f37d640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f7048112f10 con 0x7f7048108ba0 2024-05-27T07:29:05.786 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:29:05.795+0000 7f704d91a640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f7048109040 0x7f7048199f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:29:05.786 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:29:05.795+0000 7f704d91a640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f7048109040 0x7f7048199f40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.52:40548/0 (socket says 172.21.15.52:40548) 2024-05-27T07:29:05.787 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:29:05.795+0000 7f704d91a640 1 -- 172.21.15.52:0/3723986285 learned_addr learned my addr 172.21.15.52:0/3723986285 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-05-27T07:29:05.787 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:29:05.795+0000 7f704d119640 1 --2- 172.21.15.52:0/3723986285 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f70481032f0 0x7f70481994c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:29:05.787 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:29:05.796+0000 7f704c918640 1 --2- 172.21.15.52:0/3723986285 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f7048108ba0 0x7f7048199a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:29:05.787 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:29:05.796+0000 7f704d91a640 1 -- 172.21.15.52:0/3723986285 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f7048108ba0 msgr2=0x7f7048199a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:29:05.787 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:29:05.796+0000 7f704d91a640 1 --2- 172.21.15.52:0/3723986285 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f7048108ba0 0x7f7048199a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:29:05.787 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:29:05.796+0000 7f704d91a640 1 -- 172.21.15.52:0/3723986285 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f70481032f0 msgr2=0x7f70481994c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:29:05.787 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:29:05.796+0000 7f704d91a640 1 --2- 172.21.15.52:0/3723986285 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f70481032f0 0x7f70481994c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:29:05.787 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:29:05.796+0000 7f704d91a640 1 -- 172.21.15.52:0/3723986285 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f70481a8b90 con 0x7f7048109040 2024-05-27T07:29:05.787 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:29:05.796+0000 7f704c918640 1 --2- 172.21.15.52:0/3723986285 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f7048108ba0 0x7f7048199a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-27T07:29:05.787 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:29:05.796+0000 7f704d119640 1 --2- 172.21.15.52:0/3723986285 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f70481032f0 0x7f70481994c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-27T07:29:05.787 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:29:05.796+0000 7f704d91a640 1 --2- 172.21.15.52:0/3723986285 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f7048109040 0x7f7048199f40 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7f704000eaa0 tx=0x7f704000ef60 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:29:05.788 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:29:05.796+0000 7f703e7fc640 1 -- 172.21.15.52:0/3723986285 <== mon.2 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f704000cdb0 con 0x7f7048109040 2024-05-27T07:29:05.788 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:29:05.797+0000 7f703e7fc640 1 -- 172.21.15.52:0/3723986285 <== mon.2 v2:172.21.15.105:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f7040005020 con 0x7f7048109040 2024-05-27T07:29:05.788 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:29:05.797+0000 7f703e7fc640 1 -- 172.21.15.52:0/3723986285 <== mon.2 v2:172.21.15.105:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f70400116d0 con 0x7f7048109040 2024-05-27T07:29:05.788 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:29:05.797+0000 7f704f37d640 1 -- 172.21.15.52:0/3723986285 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f70481a8e80 con 0x7f7048109040 2024-05-27T07:29:05.788 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:29:05.797+0000 7f704f37d640 1 -- 172.21.15.52:0/3723986285 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f70481a92f0 con 0x7f7048109040 2024-05-27T07:29:05.793 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:29:05.804+0000 7f703e7fc640 1 -- 172.21.15.52:0/3723986285 <== mon.2 v2:172.21.15.105:3300/0 4 ==== mgrmap(e 26) ==== 149783+0+0 (secure 0 0 0) 0x7f70400054d0 con 0x7f7048109040 2024-05-27T07:29:05.795 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:29:05.805+0000 7f7056234740 1 Processor -- start 2024-05-27T07:29:05.795 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:29:05.805+0000 7f7056234740 1 -- start start 2024-05-27T07:29:05.796 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:29:05.806+0000 7f7056234740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55d670de81b0 0x55d6711893d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:29:05.796 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:29:05.806+0000 7f7056234740 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55d670dbf2f0 0x55d671189910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:29:05.796 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:29:05.806+0000 7f7056234740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55d670dc74d0 0x55d671189e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:29:05.796 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:29:05.806+0000 7f7056234740 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x55d6710fbc30 con 0x55d670dbf2f0 2024-05-27T07:29:05.796 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:29:05.806+0000 7f7056234740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x55d6710eb7d0 con 0x55d670dc74d0 2024-05-27T07:29:05.796 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:29:05.806+0000 7f7056234740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x55d6710ef810 con 0x55d670de81b0 2024-05-27T07:29:05.798 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:29:05.807+0000 7f703e7fc640 1 --2- 172.21.15.52:0/3723986285 >> [v2:172.21.15.52:6800/1822318384,v1:172.21.15.52:6801/1822318384] conn(0x7f70340b2a50 0x7f70340b4f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:29:05.798 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:29:05.807+0000 7f704c918640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55d670de81b0 0x55d6711893d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:29:05.798 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:29:05.807+0000 7f704c918640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55d670de81b0 0x55d6711893d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.173:3300/0 says I am v2:172.21.15.52:59404/0 (socket says 172.21.15.52:59404) 2024-05-27T07:29:05.798 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:29:05.807+0000 7f704c918640 1 -- 172.21.15.52:0/513168532 learned_addr learned my addr 172.21.15.52:0/513168532 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-05-27T07:29:05.798 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:29:05.807+0000 7f704c918640 1 --2- 172.21.15.52:0/513168532 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55d670dc74d0 0x55d671189e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:29:05.798 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:29:05.807+0000 7f704d119640 1 --2- 172.21.15.52:0/513168532 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55d670dbf2f0 0x55d671189910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:29:05.798 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:29:05.807+0000 7f704c918640 1 -- 172.21.15.52:0/513168532 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55d670dc74d0 msgr2=0x55d671189e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:29:05.798 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:29:05.807+0000 7f704c918640 1 --2- 172.21.15.52:0/513168532 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55d670dc74d0 0x55d671189e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:29:05.798 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:29:05.807+0000 7f704c918640 1 -- 172.21.15.52:0/513168532 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55d670dbf2f0 msgr2=0x55d671189910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:29:05.799 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:29:05.807+0000 7f704c918640 1 --2- 172.21.15.52:0/513168532 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55d670dbf2f0 0x55d671189910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:29:05.799 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:29:05.807+0000 7f704c918640 1 -- 172.21.15.52:0/513168532 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55d670e00210 con 0x55d670de81b0 2024-05-27T07:29:05.799 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:29:05.807+0000 7f704d91a640 1 --2- 172.21.15.52:0/3723986285 >> [v2:172.21.15.52:6800/1822318384,v1:172.21.15.52:6801/1822318384] conn(0x7f70340b2a50 0x7f70340b4f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:29:05.799 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:29:05.808+0000 7f704c918640 1 --2- 172.21.15.52:0/513168532 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55d670de81b0 0x55d6711893d0 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f703802a860 tx=0x7f7038009490 comp rx=0 tx=0).ready entity=mon.1 client_cookie=780c85f91c2ee85c server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:29:05.800 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:29:05.808+0000 7f703e7fc640 1 -- 172.21.15.52:0/3723986285 <== mon.2 v2:172.21.15.105:3300/0 5 ==== osd_map(98..98 src has 1..98) ==== 9379+0+0 (secure 0 0 0) 0x7f70400e2bb0 con 0x7f7048109040 2024-05-27T07:29:05.800 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:29:05.809+0000 7f70337fe640 1 -- 172.21.15.52:0/513168532 <== mon.1 v2:172.21.15.173:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7038037070 con 0x55d670de81b0 2024-05-27T07:29:05.800 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:29:05.809+0000 7f704d91a640 1 --2- 172.21.15.52:0/3723986285 >> [v2:172.21.15.52:6800/1822318384,v1:172.21.15.52:6801/1822318384] conn(0x7f70340b2a50 0x7f70340b4f60 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7f70400a97d0 tx=0x7f704000cf50 comp rx=0 tx=0).ready entity=mgr.15333 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:29:05.800 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:29:05.809+0000 7f70337fe640 1 -- 172.21.15.52:0/513168532 <== mon.1 v2:172.21.15.173:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f7038030900 con 0x55d670de81b0 2024-05-27T07:29:05.800 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:29:05.809+0000 7f70337fe640 1 -- 172.21.15.52:0/513168532 <== mon.1 v2:172.21.15.173:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f70380055e0 con 0x55d670de81b0 2024-05-27T07:29:05.801 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:29:05.810+0000 7f7056234740 1 -- 172.21.15.52:0/513168532 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55d670de81b0 msgr2=0x55d6711893d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:29:05.801 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:29:05.810+0000 7f7056234740 1 --2- 172.21.15.52:0/513168532 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55d670de81b0 0x55d6711893d0 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f703802a860 tx=0x7f7038009490 comp rx=0 tx=0).stop 2024-05-27T07:29:05.801 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:29:05.810+0000 7f7056234740 1 -- 172.21.15.52:0/513168532 shutdown_connections 2024-05-27T07:29:05.801 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:29:05.810+0000 7f7056234740 1 --2- 172.21.15.52:0/513168532 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55d670dc74d0 0x55d671189e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:29:05.801 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:29:05.810+0000 7f7056234740 1 --2- 172.21.15.52:0/513168532 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55d670dbf2f0 0x55d671189910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:29:05.801 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:29:05.810+0000 7f7056234740 1 --2- 172.21.15.52:0/513168532 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55d670de81b0 0x55d6711893d0 unknown :-1 s=CLOSED pgs=107 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:29:05.801 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:29:05.810+0000 7f7056234740 1 -- 172.21.15.52:0/513168532 >> 172.21.15.52:0/513168532 conn(0x55d670ddc300 msgr2=0x55d671185b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:29:05.801 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:29:05.810+0000 7f7056234740 1 -- 172.21.15.52:0/513168532 shutdown_connections 2024-05-27T07:29:05.801 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:29:05.810+0000 7f7056234740 1 -- 172.21.15.52:0/513168532 wait complete. 2024-05-27T07:29:05.801 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:29:05.810+0000 7f7056234740 1 -- start start 2024-05-27T07:29:05.801 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:29:05.811+0000 7f7056234740 1 Processor -- start 2024-05-27T07:29:05.804 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:29:05.811+0000 7f7056234740 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55d670dbf2f0 0x55d671198a30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:29:05.804 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:29:05.811+0000 7f7056234740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55d670dc74d0 0x55d671198f90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:29:05.804 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:29:05.811+0000 7f7056234740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55d670de81b0 0x55d671193e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:29:05.804 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:29:05.811+0000 7f7056234740 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x55d670dd1790 con 0x55d670dbf2f0 2024-05-27T07:29:05.804 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:29:05.811+0000 7f7056234740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x55d6710a7890 con 0x55d670de81b0 2024-05-27T07:29:05.804 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:29:05.811+0000 7f7056234740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x55d6711903b0 con 0x55d670dc74d0 2024-05-27T07:29:05.804 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:29:05.811+0000 7f7056234740 10 client.0 _ll_register_callbacks cb 0x55d670dffa00 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T07:29:05.804 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:29:05.811+0000 7f704c918640 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55d670dbf2f0 0x55d671198a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:29:05.805 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:29:05.812+0000 7f704c918640 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55d670dbf2f0 0x55d671198a30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.52:46534/0 (socket says 172.21.15.52:46534) 2024-05-27T07:29:05.805 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:29:05.812+0000 7f704c918640 1 -- 172.21.15.52:0/2591168590 learned_addr learned my addr 172.21.15.52:0/2591168590 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-05-27T07:29:05.805 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:29:05.812+0000 7f704c918640 1 --2- 172.21.15.52:0/2591168590 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55d670de81b0 0x55d671193e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:29:05.805 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:29:05.812+0000 7f704d119640 1 --2- 172.21.15.52:0/2591168590 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55d670dc74d0 0x55d671198f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:29:05.805 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:29:05.812+0000 7f704c918640 1 -- 172.21.15.52:0/2591168590 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55d670dc74d0 msgr2=0x55d671198f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:29:05.805 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:29:05.812+0000 7f704c918640 1 --2- 172.21.15.52:0/2591168590 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55d670dc74d0 0x55d671198f90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:29:05.805 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:29:05.812+0000 7f704c918640 1 -- 172.21.15.52:0/2591168590 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55d670dbf2f0 msgr2=0x55d671198a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:29:05.805 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:29:05.812+0000 7f704c918640 1 --2- 172.21.15.52:0/2591168590 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55d670dbf2f0 0x55d671198a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:29:05.805 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:29:05.812+0000 7f704c918640 1 -- 172.21.15.52:0/2591168590 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55d670ecc620 con 0x55d670de81b0 2024-05-27T07:29:05.805 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:29:05.812+0000 7f704c918640 1 --2- 172.21.15.52:0/2591168590 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55d670de81b0 0x55d671193e10 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7f7038009830 tx=0x7f7038005d10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=3b641dfbe6495f10 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:29:05.805 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:29:05.812+0000 7f7030ff9640 10 client.0 ms_handle_connect on v2:172.21.15.105:3300/0 2024-05-27T07:29:05.805 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:29:05.813+0000 7f7030ff9640 1 -- 172.21.15.52:0/2591168590 <== mon.2 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7038037040 con 0x55d670de81b0 2024-05-27T07:29:05.805 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:29:05.813+0000 7f7030ff9640 1 -- 172.21.15.52:0/2591168590 <== mon.2 v2:172.21.15.105:3300/0 2 ==== osd_map(98..98 src has 1..98) ==== 9379+0+0 (secure 0 0 0) 0x7f70380035f0 con 0x55d670de81b0 2024-05-27T07:29:05.806 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:29:05.813+0000 7f7056234740 1 -- 172.21.15.52:0/2591168590 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f7038030900 con 0x55d670de81b0 2024-05-27T07:29:05.806 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:29:05.813+0000 7f7030ff9640 1 -- 172.21.15.52:0/2591168590 <== mon.2 v2:172.21.15.105:3300/0 3 ==== mon_get_version_reply(handle=1 version=47) ==== 24+0+0 (secure 0 0 0) 0x7f70380037f0 con 0x55d670de81b0 2024-05-27T07:29:05.806 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:29:05.813+0000 7f7056234740 10 client.25865 fetch_fsmap learned FSMap version 47 2024-05-27T07:29:05.806 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:29:05.813+0000 7f7056234740 1 -- 172.21.15.52:0/2591168590 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,fsmap=47,monmap=4+}) -- 0x55d6711a2880 con 0x55d670de81b0 2024-05-27T07:29:05.806 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:29:05.813+0000 7f7030ff9640 1 -- 172.21.15.52:0/2591168590 <== mon.2 v2:172.21.15.105:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f7038030900 con 0x55d670de81b0 2024-05-27T07:29:05.806 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:29:05.814+0000 7f7030ff9640 1 -- 172.21.15.52:0/2591168590 <== mon.2 v2:172.21.15.105:3300/0 5 ==== fsmap(e 47) ==== 8378+0+0 (secure 0 0 0) 0x7f703805c030 con 0x55d670de81b0 2024-05-27T07:29:05.806 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:29:05.814+0000 7f7056234740 10 client.25865 fetch_fsmap finished waiting for FSMap version 47 2024-05-27T07:29:05.806 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:29:05.814+0000 7f7056234740 10 client.25865 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-05-27T07:29:05.806 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:29:05.814+0000 7f7056234740 20 client.25865 populate_metadata read hostname 'smithi052' 2024-05-27T07:29:05.806 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:29:05.814+0000 7f7056234740 1 --2- 172.21.15.52:0/2591168590 >> [v2:172.21.15.105:6838/500348151,v1:172.21.15.105:6839/500348151] conn(0x55d670da6d80 0x55d6711a5f60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:29:05.806 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:29:05.814+0000 7f7056234740 4 client.25865 mds_command: new command op to 24527 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-27T07:29:05.806 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:29:05.814+0000 7f7056234740 1 -- 172.21.15.52:0/2591168590 --> [v2:172.21.15.105:6838/500348151,v1:172.21.15.105:6839/500348151] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55d6710fdb80 con 0x55d670da6d80 2024-05-27T07:29:05.806 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:29:05.814+0000 7f704d119640 1 --2- 172.21.15.52:0/2591168590 >> [v2:172.21.15.105:6838/500348151,v1:172.21.15.105:6839/500348151] conn(0x55d670da6d80 0x55d6711a5f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:29:05.807 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:29:05.815+0000 7f704d119640 1 --2- 172.21.15.52:0/2591168590 >> [v2:172.21.15.105:6838/500348151,v1:172.21.15.105:6839/500348151] conn(0x55d670da6d80 0x55d6711a5f60 crc :-1 s=READY pgs=91 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=21c3cb0bf38039c1 server_cookie=a53051bf79d1a56 in_seq=0 out_seq=0 2024-05-27T07:29:05.807 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:29:05.815+0000 7f7030ff9640 10 client.25865 ms_handle_connect on v2:172.21.15.105:6838/500348151 2024-05-27T07:29:05.810 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:29:05.820+0000 7f7030ff9640 1 -- 172.21.15.52:0/2591168590 <== mds.0 v2:172.21.15.105:6838/500348151 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24041 (crc 0 0 0) 0x55d6710fdb80 con 0x55d670da6d80 2024-05-27T07:29:05.810 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:29:05.820+0000 7f7030ff9640 10 client.25865 handle_command_reply: tid=0 2024-05-27T07:29:05.811 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:29:05.820+0000 7f7056234740 1 client.25865 shutdown 2024-05-27T07:29:05.811 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:29:05.820+0000 7f7056234740 1 -- 172.21.15.52:0/2591168590 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55d670de81b0 msgr2=0x55d671193e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:29:05.811 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:29:05.820+0000 7f7056234740 1 --2- 172.21.15.52:0/2591168590 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55d670de81b0 0x55d671193e10 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7f7038009830 tx=0x7f7038005d10 comp rx=0 tx=0).stop 2024-05-27T07:29:05.811 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:29:05.820+0000 7f7056234740 1 -- 172.21.15.52:0/2591168590 shutdown_connections 2024-05-27T07:29:05.811 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:29:05.820+0000 7f7056234740 1 --2- 172.21.15.52:0/2591168590 >> [v2:172.21.15.105:6838/500348151,v1:172.21.15.105:6839/500348151] conn(0x55d670da6d80 0x55d6711a5f60 crc :-1 s=READY pgs=91 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:29:05.811 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:29:05.820+0000 7f7056234740 1 --2- 172.21.15.52:0/2591168590 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55d670de81b0 0x55d671193e10 unknown :-1 s=CLOSED pgs=116 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:29:05.811 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:29:05.820+0000 7f7056234740 1 --2- 172.21.15.52:0/2591168590 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55d670dc74d0 0x55d671198f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:29:05.811 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:29:05.820+0000 7f7056234740 1 --2- 172.21.15.52:0/2591168590 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55d670dbf2f0 0x55d671198a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:29:05.811 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:29:05.820+0000 7f7056234740 1 -- 172.21.15.52:0/2591168590 >> 172.21.15.52:0/2591168590 conn(0x55d670ddc300 msgr2=0x55d6711891a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:29:05.811 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:29:05.820+0000 7f7030ff9640 0 client.25865 ms_handle_reset on v2:172.21.15.105:6838/500348151 2024-05-27T07:29:05.811 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:29:05.820+0000 7f7056234740 1 -- 172.21.15.52:0/2591168590 shutdown_connections 2024-05-27T07:29:05.811 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:29:05.820+0000 7f7056234740 1 -- 172.21.15.52:0/2591168590 wait complete. 2024-05-27T07:29:05.812 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:29:05.820+0000 7f7056234740 20 client.25865 trim_cache size 0 max 16384 2024-05-27T07:29:05.828 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:29:05.837+0000 7f7056234740 1 Processor -- start 2024-05-27T07:29:05.828 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:29:05.837+0000 7f7056234740 1 -- start start 2024-05-27T07:29:05.828 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:29:05.837+0000 7f7056234740 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55d670dbf2f0 0x55d67118d8c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:29:05.828 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:29:05.837+0000 7f7056234740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55d670dc74d0 0x55d6711982f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:29:05.828 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:29:05.837+0000 7f7056234740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55d670de81b0 0x55d67119dda0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:29:05.828 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:29:05.837+0000 7f7056234740 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x7f70481aefc0 con 0x55d670dbf2f0 2024-05-27T07:29:05.828 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:29:05.837+0000 7f7056234740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f70481b0bc0 con 0x55d670dc74d0 2024-05-27T07:29:05.828 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:29:05.837+0000 7f7056234740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f70481af690 con 0x55d670de81b0 2024-05-27T07:29:05.829 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:29:05.837+0000 7f704c918640 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55d670dbf2f0 0x55d67118d8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:29:05.829 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:29:05.837+0000 7f704c918640 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55d670dbf2f0 0x55d67118d8c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.52:46544/0 (socket says 172.21.15.52:46544) 2024-05-27T07:29:05.829 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:29:05.837+0000 7f704c918640 1 -- 172.21.15.52:0/2140915382 learned_addr learned my addr 172.21.15.52:0/2140915382 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-05-27T07:29:05.829 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:29:05.837+0000 7f704d119640 1 --2- 172.21.15.52:0/2140915382 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55d670dc74d0 0x55d6711982f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:29:05.829 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:29:05.837+0000 7f704c918640 1 --2- 172.21.15.52:0/2140915382 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55d670de81b0 0x55d67119dda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:29:05.829 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:29:05.838+0000 7f704c918640 1 -- 172.21.15.52:0/2140915382 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55d670de81b0 msgr2=0x55d67119dda0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:29:05.829 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:29:05.838+0000 7f704c918640 1 --2- 172.21.15.52:0/2140915382 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55d670de81b0 0x55d67119dda0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:29:05.829 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:29:05.838+0000 7f704c918640 1 -- 172.21.15.52:0/2140915382 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55d670dc74d0 msgr2=0x55d6711982f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:29:05.829 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:29:05.838+0000 7f704c918640 1 --2- 172.21.15.52:0/2140915382 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55d670dc74d0 0x55d6711982f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:29:05.829 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:29:05.838+0000 7f704c918640 1 -- 172.21.15.52:0/2140915382 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55d67118a7b0 con 0x55d670dbf2f0 2024-05-27T07:29:05.829 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:29:05.838+0000 7f704d119640 1 --2- 172.21.15.52:0/2140915382 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55d670dc74d0 0x55d6711982f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-27T07:29:05.829 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:29:05.838+0000 7f704c918640 1 --2- 172.21.15.52:0/2140915382 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55d670dbf2f0 0x55d67118d8c0 secure :-1 s=READY pgs=598 cs=0 l=1 rev1=1 crypto rx=0x7f7038006010 tx=0x7f7038005670 comp rx=0 tx=0).ready entity=mon.0 client_cookie=14aff46b84f00072 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:29:05.829 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:29:05.838+0000 7f7032ffd640 1 -- 172.21.15.52:0/2140915382 <== mon.0 v2:172.21.15.52:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f703802a9c0 con 0x55d670dbf2f0 2024-05-27T07:29:05.830 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:29:05.838+0000 7f7032ffd640 1 -- 172.21.15.52:0/2140915382 <== mon.0 v2:172.21.15.52:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f7038030d00 con 0x55d670dbf2f0 2024-05-27T07:29:05.830 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:29:05.838+0000 7f7032ffd640 1 -- 172.21.15.52:0/2140915382 <== mon.0 v2:172.21.15.52:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7038062440 con 0x55d670dbf2f0 2024-05-27T07:29:05.833 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:29:05.839+0000 7f7056234740 1 -- 172.21.15.52:0/2140915382 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55d670dbf2f0 msgr2=0x55d67118d8c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:29:05.833 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:29:05.839+0000 7f7056234740 1 --2- 172.21.15.52:0/2140915382 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55d670dbf2f0 0x55d67118d8c0 secure :-1 s=READY pgs=598 cs=0 l=1 rev1=1 crypto rx=0x7f7038006010 tx=0x7f7038005670 comp rx=0 tx=0).stop 2024-05-27T07:29:05.833 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:29:05.839+0000 7f7056234740 1 -- 172.21.15.52:0/2140915382 shutdown_connections 2024-05-27T07:29:05.833 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:29:05.839+0000 7f7056234740 1 --2- 172.21.15.52:0/2140915382 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55d670de81b0 0x55d67119dda0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:29:05.833 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:29:05.839+0000 7f7056234740 1 --2- 172.21.15.52:0/2140915382 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55d670dc74d0 0x55d6711982f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:29:05.834 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:29:05.839+0000 7f7056234740 1 --2- 172.21.15.52:0/2140915382 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55d670dbf2f0 0x55d67118d8c0 unknown :-1 s=CLOSED pgs=598 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:29:05.834 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:29:05.839+0000 7f7056234740 1 -- 172.21.15.52:0/2140915382 >> 172.21.15.52:0/2140915382 conn(0x55d670ddc300 msgr2=0x55d67119d690 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:29:05.834 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:29:05.839+0000 7f7056234740 1 -- 172.21.15.52:0/2140915382 shutdown_connections 2024-05-27T07:29:05.834 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:29:05.839+0000 7f7056234740 1 -- 172.21.15.52:0/2140915382 wait complete. 2024-05-27T07:29:05.834 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:29:05.840+0000 7f7056234740 1 -- start start 2024-05-27T07:29:05.834 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:29:05.840+0000 7f7056234740 1 Processor -- start 2024-05-27T07:29:05.834 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:29:05.840+0000 7f7056234740 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55d670dbf2f0 0x55d6711b69a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:29:05.834 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:29:05.840+0000 7f7056234740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55d670dc74d0 0x55d6711b6ee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:29:05.834 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:29:05.840+0000 7f7056234740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55d670de81b0 0x55d6712063d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:29:05.834 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:29:05.841+0000 7f7056234740 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x7f70481b04f0 con 0x55d670dbf2f0 2024-05-27T07:29:05.834 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:29:05.841+0000 7f7056234740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f70481afd60 con 0x55d670dc74d0 2024-05-27T07:29:05.835 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:29:05.841+0000 7f7056234740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f70481acfe0 con 0x55d670de81b0 2024-05-27T07:29:05.835 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:29:05.841+0000 7f7056234740 10 client.0 _ll_register_callbacks cb 0x7f70380096b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T07:29:05.835 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:29:05.841+0000 7f704c918640 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55d670dbf2f0 0x55d6711b69a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:29:05.835 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:29:05.841+0000 7f704d119640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55d670dc74d0 0x55d6711b6ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:29:05.835 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:29:05.841+0000 7f704c918640 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55d670dbf2f0 0x55d6711b69a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.52:46550/0 (socket says 172.21.15.52:46550) 2024-05-27T07:29:05.835 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:29:05.841+0000 7f704c918640 1 -- 172.21.15.52:0/1496268030 learned_addr learned my addr 172.21.15.52:0/1496268030 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-05-27T07:29:05.835 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:29:05.841+0000 7f704c918640 1 --2- 172.21.15.52:0/1496268030 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55d670de81b0 0x55d6712063d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:29:05.835 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:29:05.841+0000 7f704d119640 1 -- 172.21.15.52:0/1496268030 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55d670de81b0 msgr2=0x55d6712063d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:29:05.835 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:29:05.841+0000 7f704d119640 1 --2- 172.21.15.52:0/1496268030 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55d670de81b0 0x55d6712063d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:29:05.835 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:29:05.841+0000 7f704d119640 1 -- 172.21.15.52:0/1496268030 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55d670dbf2f0 msgr2=0x55d6711b69a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:29:05.835 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:29:05.841+0000 7f704d119640 1 --2- 172.21.15.52:0/1496268030 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55d670dbf2f0 0x55d6711b69a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:29:05.835 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:29:05.841+0000 7f704d119640 1 -- 172.21.15.52:0/1496268030 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55d6711946f0 con 0x55d670dc74d0 2024-05-27T07:29:05.835 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:29:05.841+0000 7f704c918640 1 --2- 172.21.15.52:0/1496268030 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55d670dbf2f0 0x55d6711b69a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-27T07:29:05.836 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:29:05.841+0000 7f704d119640 1 --2- 172.21.15.52:0/1496268030 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55d670dc74d0 0x55d6711b6ee0 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x55d67118a780 tx=0x7f70440044d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=4f52a25ab6f8959f server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:29:05.836 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:29:05.841+0000 7f7030ff9640 10 client.0 ms_handle_connect on v2:172.21.15.105:3300/0 2024-05-27T07:29:05.836 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:29:05.841+0000 7f7030ff9640 1 -- 172.21.15.52:0/1496268030 <== mon.2 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7044003df0 con 0x55d670dc74d0 2024-05-27T07:29:05.836 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:29:05.841+0000 7f7030ff9640 1 -- 172.21.15.52:0/1496268030 <== mon.2 v2:172.21.15.105:3300/0 2 ==== osd_map(98..98 src has 1..98) ==== 9379+0+0 (secure 0 0 0) 0x7f7044005050 con 0x55d670dc74d0 2024-05-27T07:29:05.836 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:29:05.841+0000 7f7056234740 1 -- 172.21.15.52:0/1496268030 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f7038030d00 con 0x55d670dc74d0 2024-05-27T07:29:05.836 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:29:05.842+0000 7f7030ff9640 1 -- 172.21.15.52:0/1496268030 <== mon.2 v2:172.21.15.105:3300/0 3 ==== mon_get_version_reply(handle=1 version=47) ==== 24+0+0 (secure 0 0 0) 0x7f704402b050 con 0x55d670dc74d0 2024-05-27T07:29:05.836 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:29:05.842+0000 7f7056234740 10 client.25871 fetch_fsmap learned FSMap version 47 2024-05-27T07:29:05.836 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:29:05.842+0000 7f7056234740 1 -- 172.21.15.52:0/1496268030 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,fsmap=47,monmap=4+}) -- 0x55d671181340 con 0x55d670dc74d0 2024-05-27T07:29:05.836 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:29:05.842+0000 7f7030ff9640 1 -- 172.21.15.52:0/1496268030 <== mon.2 v2:172.21.15.105:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f7038030d00 con 0x55d670dc74d0 2024-05-27T07:29:05.836 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:29:05.842+0000 7f7030ff9640 1 -- 172.21.15.52:0/1496268030 <== mon.2 v2:172.21.15.105:3300/0 5 ==== fsmap(e 47) ==== 8378+0+0 (secure 0 0 0) 0x7f704401c410 con 0x55d670dc74d0 2024-05-27T07:29:05.836 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:29:05.842+0000 7f7056234740 10 client.25871 fetch_fsmap finished waiting for FSMap version 47 2024-05-27T07:29:05.836 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:29:05.842+0000 7f7056234740 10 client.25871 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-05-27T07:29:05.837 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:29:05.842+0000 7f7056234740 20 client.25871 populate_metadata read hostname 'smithi052' 2024-05-27T07:29:05.837 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:29:05.842+0000 7f7056234740 1 --2- 172.21.15.52:0/1496268030 >> [v2:172.21.15.105:6838/500348151,v1:172.21.15.105:6839/500348151] conn(0x55d670d7c1c0 0x55d6712101e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:29:05.837 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:29:05.842+0000 7f7056234740 4 client.25871 mds_command: new command op to 24527 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-05-27T07:29:05.837 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:29:05.842+0000 7f7056234740 1 -- 172.21.15.52:0/1496268030 --> [v2:172.21.15.105:6838/500348151,v1:172.21.15.105:6839/500348151] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55d6710f7360 con 0x55d670d7c1c0 2024-05-27T07:29:05.837 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:29:05.842+0000 7f704c918640 1 --2- 172.21.15.52:0/1496268030 >> [v2:172.21.15.105:6838/500348151,v1:172.21.15.105:6839/500348151] conn(0x55d670d7c1c0 0x55d6712101e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:29:05.837 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:29:05.843+0000 7f7030ff9640 10 client.25871 ms_handle_connect on v2:172.21.15.105:6838/500348151 2024-05-27T07:29:05.837 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:29:05.843+0000 7f704c918640 1 --2- 172.21.15.52:0/1496268030 >> [v2:172.21.15.105:6838/500348151,v1:172.21.15.105:6839/500348151] conn(0x55d670d7c1c0 0x55d6712101e0 crc :-1 s=READY pgs=92 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=eaa85727f947b4b server_cookie=66fb0f9d3d918c42 in_seq=0 out_seq=0 2024-05-27T07:29:05.979 INFO:journalctl@ceph.mds.d.smithi052.stdout:May 27 07:29:05 smithi052 systemd[1]: ceph-73df2636-1bf8-11ef-bc9b-c7b262605968@mds.d.service: Failed with result 'exit-code'. 2024-05-27T07:29:06.479 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:29:06 smithi052 ceph-mon[31955]: pgmap v104: 129 pgs: 129 active+clean; 165 MiB data, 869 MiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 4.4 MiB/s wr, 685 op/s 2024-05-27T07:29:06.479 INFO:journalctl@ceph.mds.a.smithi052.stdout:May 27 07:29:06 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-a[74428]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: In function 'std::set SnapClient::filter(const std::set&) const' thread 7f74e54c7700 time 2024-05-27T07:29:06.275381+0000 2024-05-27T07:29:06.479 INFO:journalctl@ceph.mds.a.smithi052.stdout:May 27 07:29:06 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-a[74428]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: 197: FAILED ceph_assert(cached_version > 0) 2024-05-27T07:29:06.480 INFO:journalctl@ceph.mds.a.smithi052.stdout:May 27 07:29:06 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-a[74428]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:29:06.480 INFO:journalctl@ceph.mds.a.smithi052.stdout:May 27 07:29:06 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-a[74428]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x127) [0x7f74f7d0cba6] 2024-05-27T07:29:06.480 INFO:journalctl@ceph.mds.a.smithi052.stdout:May 27 07:29:06 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-a[74428]: 2: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7f74f7d0cdd1] 2024-05-27T07:29:06.480 INFO:journalctl@ceph.mds.a.smithi052.stdout:May 27 07:29:06 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-a[74428]: 3: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x5629e062023d] 2024-05-27T07:29:06.480 INFO:journalctl@ceph.mds.a.smithi052.stdout:May 27 07:29:06 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-a[74428]: 4: (SnapRealm::build_snap_set() const+0x323) [0x5629e060b971] 2024-05-27T07:29:06.480 INFO:journalctl@ceph.mds.a.smithi052.stdout:May 27 07:29:06 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-a[74428]: 5: (SnapRealm::check_cache() const+0x13e) [0x5629e0609fa0] 2024-05-27T07:29:06.480 INFO:journalctl@ceph.mds.a.smithi052.stdout:May 27 07:29:06 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-a[74428]: 6: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x5629e060a651] 2024-05-27T07:29:06.480 INFO:journalctl@ceph.mds.a.smithi052.stdout:May 27 07:29:06 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-a[74428]: 7: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x5629e059f347] 2024-05-27T07:29:06.480 INFO:journalctl@ceph.mds.a.smithi052.stdout:May 27 07:29:06 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-a[74428]: 8: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x5629e029b87e] 2024-05-27T07:29:06.480 INFO:journalctl@ceph.mds.a.smithi052.stdout:May 27 07:29:06 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-a[74428]: 9: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x2702) [0x5629e02a0e94] 2024-05-27T07:29:06.480 INFO:journalctl@ceph.mds.a.smithi052.stdout:May 27 07:29:06 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-a[74428]: 10: (EUpdate::replay(MDSRank*)+0x63) [0x5629e02ab969] 2024-05-27T07:29:06.480 INFO:journalctl@ceph.mds.a.smithi052.stdout:May 27 07:29:06 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-a[74428]: 11: (MDLog::_replay_thread()+0x1f38) [0x5629e064448c] 2024-05-27T07:29:06.480 INFO:journalctl@ceph.mds.a.smithi052.stdout:May 27 07:29:06 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-a[74428]: 12: (MDLog::ReplayThread::entry()+0x11) [0x5629e064d9cb] 2024-05-27T07:29:06.480 INFO:journalctl@ceph.mds.a.smithi052.stdout:May 27 07:29:06 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-a[74428]: 13: (Thread::entry_wrapper()+0x43) [0x7f74f7cdb72d] 2024-05-27T07:29:06.481 INFO:journalctl@ceph.mds.a.smithi052.stdout:May 27 07:29:06 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-a[74428]: 14: (Thread::_entry_func(void*)+0xd) [0x7f74f7cdb749] 2024-05-27T07:29:06.481 INFO:journalctl@ceph.mds.a.smithi052.stdout:May 27 07:29:06 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-a[74428]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7f74f687f1ca] 2024-05-27T07:29:06.481 INFO:journalctl@ceph.mds.a.smithi052.stdout:May 27 07:29:06 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-a[74428]: 16: clone() 2024-05-27T07:29:06.481 INFO:journalctl@ceph.mds.a.smithi052.stdout:May 27 07:29:06 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-a[74428]: *** Caught signal (Aborted) ** 2024-05-27T07:29:06.481 INFO:journalctl@ceph.mds.a.smithi052.stdout:May 27 07:29:06 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-a[74428]: in thread 7f74e54c7700 thread_name:md_log_replay 2024-05-27T07:29:06.481 INFO:journalctl@ceph.mds.a.smithi052.stdout:May 27 07:29:06 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-a[74428]: 2024-05-27T07:29:06.277+0000 7f74e54c7700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: In function 'std::set SnapClient::filter(const std::set&) const' thread 7f74e54c7700 time 2024-05-27T07:29:06.275381+0000 2024-05-27T07:29:06.482 INFO:journalctl@ceph.mds.a.smithi052.stdout:May 27 07:29:06 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-a[74428]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: 197: FAILED ceph_assert(cached_version > 0) 2024-05-27T07:29:06.482 INFO:journalctl@ceph.mds.a.smithi052.stdout:May 27 07:29:06 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-a[74428]: 2024-05-27T07:29:06.482 INFO:journalctl@ceph.mds.a.smithi052.stdout:May 27 07:29:06 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-a[74428]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:29:06.482 INFO:journalctl@ceph.mds.a.smithi052.stdout:May 27 07:29:06 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-a[74428]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x127) [0x7f74f7d0cba6] 2024-05-27T07:29:06.482 INFO:journalctl@ceph.mds.a.smithi052.stdout:May 27 07:29:06 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-a[74428]: 2: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7f74f7d0cdd1] 2024-05-27T07:29:06.482 INFO:journalctl@ceph.mds.a.smithi052.stdout:May 27 07:29:06 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-a[74428]: 3: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x5629e062023d] 2024-05-27T07:29:06.482 INFO:journalctl@ceph.mds.a.smithi052.stdout:May 27 07:29:06 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-a[74428]: 4: (SnapRealm::build_snap_set() const+0x323) [0x5629e060b971] 2024-05-27T07:29:06.482 INFO:journalctl@ceph.mds.a.smithi052.stdout:May 27 07:29:06 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-a[74428]: 5: (SnapRealm::check_cache() const+0x13e) [0x5629e0609fa0] 2024-05-27T07:29:06.482 INFO:journalctl@ceph.mds.a.smithi052.stdout:May 27 07:29:06 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-a[74428]: 6: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x5629e060a651] 2024-05-27T07:29:06.482 INFO:journalctl@ceph.mds.a.smithi052.stdout:May 27 07:29:06 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-a[74428]: 7: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x5629e059f347] 2024-05-27T07:29:06.482 INFO:journalctl@ceph.mds.a.smithi052.stdout:May 27 07:29:06 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-a[74428]: 8: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x5629e029b87e] 2024-05-27T07:29:06.482 INFO:journalctl@ceph.mds.a.smithi052.stdout:May 27 07:29:06 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-a[74428]: 9: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x2702) [0x5629e02a0e94] 2024-05-27T07:29:06.483 INFO:journalctl@ceph.mds.a.smithi052.stdout:May 27 07:29:06 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-a[74428]: 10: (EUpdate::replay(MDSRank*)+0x63) [0x5629e02ab969] 2024-05-27T07:29:06.483 INFO:journalctl@ceph.mds.a.smithi052.stdout:May 27 07:29:06 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-a[74428]: 11: (MDLog::_replay_thread()+0x1f38) [0x5629e064448c] 2024-05-27T07:29:06.483 INFO:journalctl@ceph.mds.a.smithi052.stdout:May 27 07:29:06 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-a[74428]: 12: (MDLog::ReplayThread::entry()+0x11) [0x5629e064d9cb] 2024-05-27T07:29:06.483 INFO:journalctl@ceph.mds.a.smithi052.stdout:May 27 07:29:06 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-a[74428]: 13: (Thread::entry_wrapper()+0x43) [0x7f74f7cdb72d] 2024-05-27T07:29:06.483 INFO:journalctl@ceph.mds.a.smithi052.stdout:May 27 07:29:06 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-a[74428]: 14: (Thread::_entry_func(void*)+0xd) [0x7f74f7cdb749] 2024-05-27T07:29:06.483 INFO:journalctl@ceph.mds.a.smithi052.stdout:May 27 07:29:06 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-a[74428]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7f74f687f1ca] 2024-05-27T07:29:06.483 INFO:journalctl@ceph.mds.a.smithi052.stdout:May 27 07:29:06 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-a[74428]: 16: clone() 2024-05-27T07:29:06.483 INFO:journalctl@ceph.mds.a.smithi052.stdout:May 27 07:29:06 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-a[74428]: 2024-05-27T07:29:06.483 INFO:journalctl@ceph.mds.a.smithi052.stdout:May 27 07:29:06 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-a[74428]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:29:06.483 INFO:journalctl@ceph.mds.a.smithi052.stdout:May 27 07:29:06 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-a[74428]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f74f6889d20] 2024-05-27T07:29:06.483 INFO:journalctl@ceph.mds.a.smithi052.stdout:May 27 07:29:06 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-a[74428]: 2: gsignal() 2024-05-27T07:29:06.483 INFO:journalctl@ceph.mds.a.smithi052.stdout:May 27 07:29:06 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-a[74428]: 3: abort() 2024-05-27T07:29:06.483 INFO:journalctl@ceph.mds.a.smithi052.stdout:May 27 07:29:06 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-a[74428]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x26e) [0x7f74f7d0cced] 2024-05-27T07:29:06.484 INFO:journalctl@ceph.mds.a.smithi052.stdout:May 27 07:29:06 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-a[74428]: 5: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7f74f7d0cdd1] 2024-05-27T07:29:06.484 INFO:journalctl@ceph.mds.a.smithi052.stdout:May 27 07:29:06 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-a[74428]: 6: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x5629e062023d] 2024-05-27T07:29:06.484 INFO:journalctl@ceph.mds.a.smithi052.stdout:May 27 07:29:06 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-a[74428]: 7: (SnapRealm::build_snap_set() const+0x323) [0x5629e060b971] 2024-05-27T07:29:06.484 INFO:journalctl@ceph.mds.a.smithi052.stdout:May 27 07:29:06 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-a[74428]: 8: (SnapRealm::check_cache() const+0x13e) [0x5629e0609fa0] 2024-05-27T07:29:06.484 INFO:journalctl@ceph.mds.a.smithi052.stdout:May 27 07:29:06 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-a[74428]: 9: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x5629e060a651] 2024-05-27T07:29:06.484 INFO:journalctl@ceph.mds.a.smithi052.stdout:May 27 07:29:06 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-a[74428]: 10: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x5629e059f347] 2024-05-27T07:29:06.484 INFO:journalctl@ceph.mds.a.smithi052.stdout:May 27 07:29:06 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-a[74428]: 11: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x5629e029b87e] 2024-05-27T07:29:06.484 INFO:journalctl@ceph.mds.a.smithi052.stdout:May 27 07:29:06 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-a[74428]: 12: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x2702) [0x5629e02a0e94] 2024-05-27T07:29:06.484 INFO:journalctl@ceph.mds.a.smithi052.stdout:May 27 07:29:06 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-a[74428]: 13: (EUpdate::replay(MDSRank*)+0x63) [0x5629e02ab969] 2024-05-27T07:29:06.484 INFO:journalctl@ceph.mds.a.smithi052.stdout:May 27 07:29:06 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-a[74428]: 14: (MDLog::_replay_thread()+0x1f38) [0x5629e064448c] 2024-05-27T07:29:06.484 INFO:journalctl@ceph.mds.a.smithi052.stdout:May 27 07:29:06 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-a[74428]: 15: (MDLog::ReplayThread::entry()+0x11) [0x5629e064d9cb] 2024-05-27T07:29:06.484 INFO:journalctl@ceph.mds.a.smithi052.stdout:May 27 07:29:06 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-a[74428]: 16: (Thread::entry_wrapper()+0x43) [0x7f74f7cdb72d] 2024-05-27T07:29:06.484 INFO:journalctl@ceph.mds.a.smithi052.stdout:May 27 07:29:06 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-a[74428]: 17: (Thread::_entry_func(void*)+0xd) [0x7f74f7cdb749] 2024-05-27T07:29:06.484 INFO:journalctl@ceph.mds.a.smithi052.stdout:May 27 07:29:06 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-a[74428]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7f74f687f1ca] 2024-05-27T07:29:06.485 INFO:journalctl@ceph.mds.a.smithi052.stdout:May 27 07:29:06 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-a[74428]: 19: clone() 2024-05-27T07:29:06.485 INFO:journalctl@ceph.mds.a.smithi052.stdout:May 27 07:29:06 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-a[74428]: 2024-05-27T07:29:06.280+0000 7f74e54c7700 -1 *** Caught signal (Aborted) ** 2024-05-27T07:29:06.485 INFO:journalctl@ceph.mds.a.smithi052.stdout:May 27 07:29:06 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-a[74428]: in thread 7f74e54c7700 thread_name:md_log_replay 2024-05-27T07:29:06.485 INFO:journalctl@ceph.mds.a.smithi052.stdout:May 27 07:29:06 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-a[74428]: 2024-05-27T07:29:06.485 INFO:journalctl@ceph.mds.a.smithi052.stdout:May 27 07:29:06 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-a[74428]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:29:06.485 INFO:journalctl@ceph.mds.a.smithi052.stdout:May 27 07:29:06 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-a[74428]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f74f6889d20] 2024-05-27T07:29:06.485 INFO:journalctl@ceph.mds.a.smithi052.stdout:May 27 07:29:06 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-a[74428]: 2: gsignal() 2024-05-27T07:29:06.485 INFO:journalctl@ceph.mds.a.smithi052.stdout:May 27 07:29:06 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-a[74428]: 3: abort() 2024-05-27T07:29:06.485 INFO:journalctl@ceph.mds.a.smithi052.stdout:May 27 07:29:06 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-a[74428]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x26e) [0x7f74f7d0cced] 2024-05-27T07:29:06.485 INFO:journalctl@ceph.mds.a.smithi052.stdout:May 27 07:29:06 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-a[74428]: 5: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7f74f7d0cdd1] 2024-05-27T07:29:06.485 INFO:journalctl@ceph.mds.a.smithi052.stdout:May 27 07:29:06 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-a[74428]: 6: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x5629e062023d] 2024-05-27T07:29:06.485 INFO:journalctl@ceph.mds.a.smithi052.stdout:May 27 07:29:06 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-a[74428]: 7: (SnapRealm::build_snap_set() const+0x323) [0x5629e060b971] 2024-05-27T07:29:06.486 INFO:journalctl@ceph.mds.a.smithi052.stdout:May 27 07:29:06 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-a[74428]: 8: (SnapRealm::check_cache() const+0x13e) [0x5629e0609fa0] 2024-05-27T07:29:06.486 INFO:journalctl@ceph.mds.a.smithi052.stdout:May 27 07:29:06 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-a[74428]: 9: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x5629e060a651] 2024-05-27T07:29:06.486 INFO:journalctl@ceph.mds.a.smithi052.stdout:May 27 07:29:06 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-a[74428]: 10: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x5629e059f347] 2024-05-27T07:29:06.486 INFO:journalctl@ceph.mds.a.smithi052.stdout:May 27 07:29:06 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-a[74428]: 11: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x5629e029b87e] 2024-05-27T07:29:06.486 INFO:journalctl@ceph.mds.a.smithi052.stdout:May 27 07:29:06 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-a[74428]: 12: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x2702) [0x5629e02a0e94] 2024-05-27T07:29:06.486 INFO:journalctl@ceph.mds.a.smithi052.stdout:May 27 07:29:06 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-a[74428]: 13: (EUpdate::replay(MDSRank*)+0x63) [0x5629e02ab969] 2024-05-27T07:29:06.486 INFO:journalctl@ceph.mds.a.smithi052.stdout:May 27 07:29:06 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-a[74428]: 14: (MDLog::_replay_thread()+0x1f38) [0x5629e064448c] 2024-05-27T07:29:06.486 INFO:journalctl@ceph.mds.a.smithi052.stdout:May 27 07:29:06 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-a[74428]: 15: (MDLog::ReplayThread::entry()+0x11) [0x5629e064d9cb] 2024-05-27T07:29:06.486 INFO:journalctl@ceph.mds.a.smithi052.stdout:May 27 07:29:06 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-a[74428]: 16: (Thread::entry_wrapper()+0x43) [0x7f74f7cdb72d] 2024-05-27T07:29:06.486 INFO:journalctl@ceph.mds.a.smithi052.stdout:May 27 07:29:06 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-a[74428]: 17: (Thread::_entry_func(void*)+0xd) [0x7f74f7cdb749] 2024-05-27T07:29:06.486 INFO:journalctl@ceph.mds.a.smithi052.stdout:May 27 07:29:06 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-a[74428]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7f74f687f1ca] 2024-05-27T07:29:06.486 INFO:journalctl@ceph.mds.a.smithi052.stdout:May 27 07:29:06 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-a[74428]: 19: clone() 2024-05-27T07:29:06.486 INFO:journalctl@ceph.mds.a.smithi052.stdout:May 27 07:29:06 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-a[74428]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-05-27T07:29:06.487 INFO:journalctl@ceph.mds.a.smithi052.stdout:May 27 07:29:06 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-a[74428]: 2024-05-27T07:29:06.487 INFO:journalctl@ceph.mds.a.smithi052.stdout:May 27 07:29:06 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-a[74428]: -1> 2024-05-27T07:29:06.277+0000 7f74e54c7700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: In function 'std::set SnapClient::filter(const std::set&) const' thread 7f74e54c7700 time 2024-05-27T07:29:06.275381+0000 2024-05-27T07:29:06.487 INFO:journalctl@ceph.mds.a.smithi052.stdout:May 27 07:29:06 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-a[74428]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: 197: FAILED ceph_assert(cached_version > 0) 2024-05-27T07:29:06.487 INFO:journalctl@ceph.mds.a.smithi052.stdout:May 27 07:29:06 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-a[74428]: 2024-05-27T07:29:06.487 INFO:journalctl@ceph.mds.a.smithi052.stdout:May 27 07:29:06 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-a[74428]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:29:06.487 INFO:journalctl@ceph.mds.a.smithi052.stdout:May 27 07:29:06 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-a[74428]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x127) [0x7f74f7d0cba6] 2024-05-27T07:29:06.487 INFO:journalctl@ceph.mds.a.smithi052.stdout:May 27 07:29:06 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-a[74428]: 2: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7f74f7d0cdd1] 2024-05-27T07:29:06.487 INFO:journalctl@ceph.mds.a.smithi052.stdout:May 27 07:29:06 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-a[74428]: 3: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x5629e062023d] 2024-05-27T07:29:06.487 INFO:journalctl@ceph.mds.a.smithi052.stdout:May 27 07:29:06 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-a[74428]: 4: (SnapRealm::build_snap_set() const+0x323) [0x5629e060b971] 2024-05-27T07:29:06.487 INFO:journalctl@ceph.mds.a.smithi052.stdout:May 27 07:29:06 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-a[74428]: 5: (SnapRealm::check_cache() const+0x13e) [0x5629e0609fa0] 2024-05-27T07:29:06.487 INFO:journalctl@ceph.mds.a.smithi052.stdout:May 27 07:29:06 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-a[74428]: 6: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x5629e060a651] 2024-05-27T07:29:06.487 INFO:journalctl@ceph.mds.a.smithi052.stdout:May 27 07:29:06 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-a[74428]: 7: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x5629e059f347] 2024-05-27T07:29:06.488 INFO:journalctl@ceph.mds.a.smithi052.stdout:May 27 07:29:06 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-a[74428]: 8: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x5629e029b87e] 2024-05-27T07:29:06.488 INFO:journalctl@ceph.mds.a.smithi052.stdout:May 27 07:29:06 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-a[74428]: 9: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x2702) [0x5629e02a0e94] 2024-05-27T07:29:06.488 INFO:journalctl@ceph.mds.a.smithi052.stdout:May 27 07:29:06 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-a[74428]: 10: (EUpdate::replay(MDSRank*)+0x63) [0x5629e02ab969] 2024-05-27T07:29:06.488 INFO:journalctl@ceph.mds.a.smithi052.stdout:May 27 07:29:06 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-a[74428]: 11: (MDLog::_replay_thread()+0x1f38) [0x5629e064448c] 2024-05-27T07:29:06.488 INFO:journalctl@ceph.mds.a.smithi052.stdout:May 27 07:29:06 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-a[74428]: 12: (MDLog::ReplayThread::entry()+0x11) [0x5629e064d9cb] 2024-05-27T07:29:06.489 INFO:journalctl@ceph.mds.a.smithi052.stdout:May 27 07:29:06 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-a[74428]: 13: (Thread::entry_wrapper()+0x43) [0x7f74f7cdb72d] 2024-05-27T07:29:06.489 INFO:journalctl@ceph.mds.a.smithi052.stdout:May 27 07:29:06 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-a[74428]: 14: (Thread::_entry_func(void*)+0xd) [0x7f74f7cdb749] 2024-05-27T07:29:06.489 INFO:journalctl@ceph.mds.a.smithi052.stdout:May 27 07:29:06 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-a[74428]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7f74f687f1ca] 2024-05-27T07:29:06.489 INFO:journalctl@ceph.mds.a.smithi052.stdout:May 27 07:29:06 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-a[74428]: 16: clone() 2024-05-27T07:29:06.489 INFO:journalctl@ceph.mds.a.smithi052.stdout:May 27 07:29:06 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-a[74428]: 2024-05-27T07:29:06.489 INFO:journalctl@ceph.mds.a.smithi052.stdout:May 27 07:29:06 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-a[74428]: 0> 2024-05-27T07:29:06.280+0000 7f74e54c7700 -1 *** Caught signal (Aborted) ** 2024-05-27T07:29:06.489 INFO:journalctl@ceph.mds.a.smithi052.stdout:May 27 07:29:06 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-a[74428]: in thread 7f74e54c7700 thread_name:md_log_replay 2024-05-27T07:29:06.489 INFO:journalctl@ceph.mds.a.smithi052.stdout:May 27 07:29:06 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-a[74428]: 2024-05-27T07:29:06.489 INFO:journalctl@ceph.mds.a.smithi052.stdout:May 27 07:29:06 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-a[74428]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:29:06.489 INFO:journalctl@ceph.mds.a.smithi052.stdout:May 27 07:29:06 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-a[74428]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f74f6889d20] 2024-05-27T07:29:06.490 INFO:journalctl@ceph.mds.a.smithi052.stdout:May 27 07:29:06 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-a[74428]: 2: gsignal() 2024-05-27T07:29:06.490 INFO:journalctl@ceph.mds.a.smithi052.stdout:May 27 07:29:06 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-a[74428]: 3: abort() 2024-05-27T07:29:06.490 INFO:journalctl@ceph.mds.a.smithi052.stdout:May 27 07:29:06 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-a[74428]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x26e) [0x7f74f7d0cced] 2024-05-27T07:29:06.490 INFO:journalctl@ceph.mds.a.smithi052.stdout:May 27 07:29:06 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-a[74428]: 5: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7f74f7d0cdd1] 2024-05-27T07:29:06.490 INFO:journalctl@ceph.mds.a.smithi052.stdout:May 27 07:29:06 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-a[74428]: 6: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x5629e062023d] 2024-05-27T07:29:06.490 INFO:journalctl@ceph.mds.a.smithi052.stdout:May 27 07:29:06 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-a[74428]: 7: (SnapRealm::build_snap_set() const+0x323) [0x5629e060b971] 2024-05-27T07:29:06.490 INFO:journalctl@ceph.mds.a.smithi052.stdout:May 27 07:29:06 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-a[74428]: 8: (SnapRealm::check_cache() const+0x13e) [0x5629e0609fa0] 2024-05-27T07:29:06.490 INFO:journalctl@ceph.mds.a.smithi052.stdout:May 27 07:29:06 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-a[74428]: 9: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x5629e060a651] 2024-05-27T07:29:06.490 INFO:journalctl@ceph.mds.a.smithi052.stdout:May 27 07:29:06 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-a[74428]: 10: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x5629e059f347] 2024-05-27T07:29:06.490 INFO:journalctl@ceph.mds.a.smithi052.stdout:May 27 07:29:06 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-a[74428]: 11: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x5629e029b87e] 2024-05-27T07:29:06.491 INFO:journalctl@ceph.mds.a.smithi052.stdout:May 27 07:29:06 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-a[74428]: 12: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x2702) [0x5629e02a0e94] 2024-05-27T07:29:06.491 INFO:journalctl@ceph.mds.a.smithi052.stdout:May 27 07:29:06 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-a[74428]: 13: (EUpdate::replay(MDSRank*)+0x63) [0x5629e02ab969] 2024-05-27T07:29:06.491 INFO:journalctl@ceph.mds.a.smithi052.stdout:May 27 07:29:06 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-a[74428]: 14: (MDLog::_replay_thread()+0x1f38) [0x5629e064448c] 2024-05-27T07:29:06.491 INFO:journalctl@ceph.mds.a.smithi052.stdout:May 27 07:29:06 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-a[74428]: 15: (MDLog::ReplayThread::entry()+0x11) [0x5629e064d9cb] 2024-05-27T07:29:06.491 INFO:journalctl@ceph.mds.a.smithi052.stdout:May 27 07:29:06 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-a[74428]: 16: (Thread::entry_wrapper()+0x43) [0x7f74f7cdb72d] 2024-05-27T07:29:06.491 INFO:journalctl@ceph.mds.a.smithi052.stdout:May 27 07:29:06 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-a[74428]: 17: (Thread::_entry_func(void*)+0xd) [0x7f74f7cdb749] 2024-05-27T07:29:06.491 INFO:journalctl@ceph.mds.a.smithi052.stdout:May 27 07:29:06 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-a[74428]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7f74f687f1ca] 2024-05-27T07:29:06.491 INFO:journalctl@ceph.mds.a.smithi052.stdout:May 27 07:29:06 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-a[74428]: 19: clone() 2024-05-27T07:29:06.491 INFO:journalctl@ceph.mds.a.smithi052.stdout:May 27 07:29:06 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-a[74428]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-05-27T07:29:06.491 INFO:journalctl@ceph.mds.a.smithi052.stdout:May 27 07:29:06 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-a[74428]: 2024-05-27T07:29:06.492 INFO:journalctl@ceph.mds.a.smithi052.stdout:May 27 07:29:06 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-a[74428]: -1> 2024-05-27T07:29:06.277+0000 7f74e54c7700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: In function 'std::set SnapClient::filter(const std::set&) const' thread 7f74e54c7700 time 2024-05-27T07:29:06.275381+0000 2024-05-27T07:29:06.492 INFO:journalctl@ceph.mds.a.smithi052.stdout:May 27 07:29:06 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-a[74428]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: 197: FAILED ceph_assert(cached_version > 0) 2024-05-27T07:29:06.492 INFO:journalctl@ceph.mds.a.smithi052.stdout:May 27 07:29:06 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-a[74428]: 2024-05-27T07:29:06.492 INFO:journalctl@ceph.mds.a.smithi052.stdout:May 27 07:29:06 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-a[74428]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:29:06.492 INFO:journalctl@ceph.mds.a.smithi052.stdout:May 27 07:29:06 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-a[74428]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x127) [0x7f74f7d0cba6] 2024-05-27T07:29:06.492 INFO:journalctl@ceph.mds.a.smithi052.stdout:May 27 07:29:06 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-a[74428]: 2: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7f74f7d0cdd1] 2024-05-27T07:29:06.492 INFO:journalctl@ceph.mds.a.smithi052.stdout:May 27 07:29:06 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-a[74428]: 3: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x5629e062023d] 2024-05-27T07:29:06.492 INFO:journalctl@ceph.mds.a.smithi052.stdout:May 27 07:29:06 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-a[74428]: 4: (SnapRealm::build_snap_set() const+0x323) [0x5629e060b971] 2024-05-27T07:29:06.492 INFO:journalctl@ceph.mds.a.smithi052.stdout:May 27 07:29:06 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-a[74428]: 5: (SnapRealm::check_cache() const+0x13e) [0x5629e0609fa0] 2024-05-27T07:29:06.492 INFO:journalctl@ceph.mds.a.smithi052.stdout:May 27 07:29:06 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-a[74428]: 6: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x5629e060a651] 2024-05-27T07:29:06.492 INFO:journalctl@ceph.mds.a.smithi052.stdout:May 27 07:29:06 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-a[74428]: 7: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x5629e059f347] 2024-05-27T07:29:06.492 INFO:journalctl@ceph.mds.a.smithi052.stdout:May 27 07:29:06 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-a[74428]: 8: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x5629e029b87e] 2024-05-27T07:29:06.493 INFO:journalctl@ceph.mds.a.smithi052.stdout:May 27 07:29:06 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-a[74428]: 9: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x2702) [0x5629e02a0e94] 2024-05-27T07:29:06.493 INFO:journalctl@ceph.mds.a.smithi052.stdout:May 27 07:29:06 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-a[74428]: 10: (EUpdate::replay(MDSRank*)+0x63) [0x5629e02ab969] 2024-05-27T07:29:06.493 INFO:journalctl@ceph.mds.a.smithi052.stdout:May 27 07:29:06 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-a[74428]: 11: (MDLog::_replay_thread()+0x1f38) [0x5629e064448c] 2024-05-27T07:29:06.493 INFO:journalctl@ceph.mds.a.smithi052.stdout:May 27 07:29:06 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-a[74428]: 12: (MDLog::ReplayThread::entry()+0x11) [0x5629e064d9cb] 2024-05-27T07:29:06.493 INFO:journalctl@ceph.mds.a.smithi052.stdout:May 27 07:29:06 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-a[74428]: 13: (Thread::entry_wrapper()+0x43) [0x7f74f7cdb72d] 2024-05-27T07:29:06.493 INFO:journalctl@ceph.mds.a.smithi052.stdout:May 27 07:29:06 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-a[74428]: 14: (Thread::_entry_func(void*)+0xd) [0x7f74f7cdb749] 2024-05-27T07:29:06.493 INFO:journalctl@ceph.mds.a.smithi052.stdout:May 27 07:29:06 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-a[74428]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7f74f687f1ca] 2024-05-27T07:29:06.493 INFO:journalctl@ceph.mds.a.smithi052.stdout:May 27 07:29:06 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-a[74428]: 16: clone() 2024-05-27T07:29:06.493 INFO:journalctl@ceph.mds.a.smithi052.stdout:May 27 07:29:06 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-a[74428]: 2024-05-27T07:29:06.493 INFO:journalctl@ceph.mds.a.smithi052.stdout:May 27 07:29:06 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-a[74428]: 0> 2024-05-27T07:29:06.280+0000 7f74e54c7700 -1 *** Caught signal (Aborted) ** 2024-05-27T07:29:06.493 INFO:journalctl@ceph.mds.a.smithi052.stdout:May 27 07:29:06 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-a[74428]: in thread 7f74e54c7700 thread_name:md_log_replay 2024-05-27T07:29:06.493 INFO:journalctl@ceph.mds.a.smithi052.stdout:May 27 07:29:06 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-a[74428]: 2024-05-27T07:29:06.493 INFO:journalctl@ceph.mds.a.smithi052.stdout:May 27 07:29:06 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-a[74428]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:29:06.493 INFO:journalctl@ceph.mds.a.smithi052.stdout:May 27 07:29:06 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-a[74428]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f74f6889d20] 2024-05-27T07:29:06.494 INFO:journalctl@ceph.mds.a.smithi052.stdout:May 27 07:29:06 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-a[74428]: 2: gsignal() 2024-05-27T07:29:06.494 INFO:journalctl@ceph.mds.a.smithi052.stdout:May 27 07:29:06 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-a[74428]: 3: abort() 2024-05-27T07:29:06.494 INFO:journalctl@ceph.mds.a.smithi052.stdout:May 27 07:29:06 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-a[74428]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x26e) [0x7f74f7d0cced] 2024-05-27T07:29:06.494 INFO:journalctl@ceph.mds.a.smithi052.stdout:May 27 07:29:06 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-a[74428]: 5: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7f74f7d0cdd1] 2024-05-27T07:29:06.494 INFO:journalctl@ceph.mds.a.smithi052.stdout:May 27 07:29:06 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-a[74428]: 6: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x5629e062023d] 2024-05-27T07:29:06.494 INFO:journalctl@ceph.mds.a.smithi052.stdout:May 27 07:29:06 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-a[74428]: 7: (SnapRealm::build_snap_set() const+0x323) [0x5629e060b971] 2024-05-27T07:29:06.494 INFO:journalctl@ceph.mds.a.smithi052.stdout:May 27 07:29:06 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-a[74428]: 8: (SnapRealm::check_cache() const+0x13e) [0x5629e0609fa0] 2024-05-27T07:29:06.494 INFO:journalctl@ceph.mds.a.smithi052.stdout:May 27 07:29:06 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-a[74428]: 9: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x5629e060a651] 2024-05-27T07:29:06.494 INFO:journalctl@ceph.mds.a.smithi052.stdout:May 27 07:29:06 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-a[74428]: 10: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x5629e059f347] 2024-05-27T07:29:06.494 INFO:journalctl@ceph.mds.a.smithi052.stdout:May 27 07:29:06 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-a[74428]: 11: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x5629e029b87e] 2024-05-27T07:29:06.494 INFO:journalctl@ceph.mds.a.smithi052.stdout:May 27 07:29:06 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-a[74428]: 12: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x2702) [0x5629e02a0e94] 2024-05-27T07:29:06.494 INFO:journalctl@ceph.mds.a.smithi052.stdout:May 27 07:29:06 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-a[74428]: 13: (EUpdate::replay(MDSRank*)+0x63) [0x5629e02ab969] 2024-05-27T07:29:06.494 INFO:journalctl@ceph.mds.a.smithi052.stdout:May 27 07:29:06 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-a[74428]: 14: (MDLog::_replay_thread()+0x1f38) [0x5629e064448c] 2024-05-27T07:29:06.495 INFO:journalctl@ceph.mds.a.smithi052.stdout:May 27 07:29:06 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-a[74428]: 15: (MDLog::ReplayThread::entry()+0x11) [0x5629e064d9cb] 2024-05-27T07:29:06.495 INFO:journalctl@ceph.mds.a.smithi052.stdout:May 27 07:29:06 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-a[74428]: 16: (Thread::entry_wrapper()+0x43) [0x7f74f7cdb72d] 2024-05-27T07:29:06.495 INFO:journalctl@ceph.mds.a.smithi052.stdout:May 27 07:29:06 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-a[74428]: 17: (Thread::_entry_func(void*)+0xd) [0x7f74f7cdb749] 2024-05-27T07:29:06.495 INFO:journalctl@ceph.mds.a.smithi052.stdout:May 27 07:29:06 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-a[74428]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7f74f687f1ca] 2024-05-27T07:29:06.495 INFO:journalctl@ceph.mds.a.smithi052.stdout:May 27 07:29:06 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-a[74428]: 19: clone() 2024-05-27T07:29:06.495 INFO:journalctl@ceph.mds.a.smithi052.stdout:May 27 07:29:06 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-a[74428]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-05-27T07:29:06.495 INFO:journalctl@ceph.mds.a.smithi052.stdout:May 27 07:29:06 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-a[74428]: 2024-05-27T07:29:06.495 INFO:journalctl@ceph.mds.a.smithi052.stdout:May 27 07:29:06 smithi052 podman[104276]: 2024-05-27 07:29:06.379038156 +0000 UTC m=+0.036690456 container died eda21eb53157d0b0ae4ffa197709e0eea08539346544e8fa541a514e350a59cf (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-a, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240521, ceph=True, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD) 2024-05-27T07:29:06.602 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:29:06 smithi173 ceph-mon[37000]: pgmap v104: 129 pgs: 129 active+clean; 165 MiB data, 869 MiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 4.4 MiB/s wr, 685 op/s 2024-05-27T07:29:06.689 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:29:06 smithi105 ceph-mon[35977]: pgmap v104: 129 pgs: 129 active+clean; 165 MiB data, 869 MiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 4.4 MiB/s wr, 685 op/s 2024-05-27T07:29:07.390 INFO:journalctl@ceph.mds.a.smithi052.stdout:May 27 07:29:07 smithi052 podman[104276]: 2024-05-27 07:29:07.106962282 +0000 UTC m=+0.764614572 container remove eda21eb53157d0b0ae4ffa197709e0eea08539346544e8fa541a514e350a59cf (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-a, GIT_BRANCH=HEAD, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.build-date=20240521, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image) 2024-05-27T07:29:07.391 INFO:journalctl@ceph.mds.a.smithi052.stdout:May 27 07:29:07 smithi052 systemd[1]: ceph-73df2636-1bf8-11ef-bc9b-c7b262605968@mds.a.service: Main process exited, code=exited, status=134/n/a 2024-05-27T07:29:07.729 INFO:journalctl@ceph.mds.a.smithi052.stdout:May 27 07:29:07 smithi052 systemd[1]: ceph-73df2636-1bf8-11ef-bc9b-c7b262605968@mds.a.service: Failed with result 'exit-code'. 2024-05-27T07:29:07.729 INFO:journalctl@ceph.mds.a.smithi052.stdout:May 27 07:29:07 smithi052 systemd[1]: ceph-73df2636-1bf8-11ef-bc9b-c7b262605968@mds.a.service: Consumed 1.517s CPU time. 2024-05-27T07:29:08.479 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:29:08 smithi052 ceph-mon[31955]: pgmap v105: 129 pgs: 129 active+clean; 165 MiB data, 869 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 3.9 MiB/s wr, 607 op/s 2024-05-27T07:29:08.601 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:29:08 smithi173 ceph-mon[37000]: pgmap v105: 129 pgs: 129 active+clean; 165 MiB data, 869 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 3.9 MiB/s wr, 607 op/s 2024-05-27T07:29:08.689 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:29:08 smithi105 ceph-mon[35977]: pgmap v105: 129 pgs: 129 active+clean; 165 MiB data, 869 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 3.9 MiB/s wr, 607 op/s 2024-05-27T07:29:09.427 INFO:journalctl@ceph.mds.f.smithi173.stdout:May 27 07:29:09 smithi173 systemd[1]: ceph-73df2636-1bf8-11ef-bc9b-c7b262605968@mds.f.service: Scheduled restart job, restart counter is at 5. 2024-05-27T07:29:09.427 INFO:journalctl@ceph.mds.f.smithi173.stdout:May 27 07:29:09 smithi173 systemd[1]: Stopped Ceph mds.f for 73df2636-1bf8-11ef-bc9b-c7b262605968. 2024-05-27T07:29:09.427 INFO:journalctl@ceph.mds.f.smithi173.stdout:May 27 07:29:09 smithi173 systemd[1]: Starting Ceph mds.f for 73df2636-1bf8-11ef-bc9b-c7b262605968... 2024-05-27T07:29:09.701 INFO:journalctl@ceph.mds.f.smithi173.stdout:May 27 07:29:09 smithi173 podman[73024]: 2024-05-27 07:29:09.341930645 +0000 UTC m=+0.030039873 image pull e36ea4fc8a5807e8d2701370e350a0de3856538797f7fba64cfce7e344bd6799 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6 2024-05-27T07:29:09.701 INFO:journalctl@ceph.mds.f.smithi173.stdout:May 27 07:29:09 smithi173 podman[73024]: 2024-05-27 07:29:09.492646643 +0000 UTC m=+0.180755872 container create 58968c237f2e2e320318fa2b66b26679470663234b1e13652788a79b15dc06e0 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-f, ceph=True, org.label-schema.build-date=20240521, GIT_CLEAN=True, org.label-schema.vendor=CentOS, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5) 2024-05-27T07:29:10.102 INFO:journalctl@ceph.mds.f.smithi173.stdout:May 27 07:29:09 smithi173 podman[73024]: 2024-05-27 07:29:09.71216095 +0000 UTC m=+0.400270181 container init 58968c237f2e2e320318fa2b66b26679470663234b1e13652788a79b15dc06e0 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-f, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, ceph=True, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240521, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, io.buildah.version=1.33.5, GIT_CLEAN=True, maintainer=Guillaume Abrioux ) 2024-05-27T07:29:10.102 INFO:journalctl@ceph.mds.f.smithi173.stdout:May 27 07:29:09 smithi173 podman[73024]: 2024-05-27 07:29:09.715501155 +0000 UTC m=+0.403610383 container start 58968c237f2e2e320318fa2b66b26679470663234b1e13652788a79b15dc06e0 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-f, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, org.label-schema.license=GPLv2, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240521) 2024-05-27T07:29:10.102 INFO:journalctl@ceph.mds.f.smithi173.stdout:May 27 07:29:09 smithi173 ceph-mds[73042]: -- 172.21.15.173:0/2159641150 <== mon.1 v2:172.21.15.173:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x5595825f7a40 con 0x55958263f180 2024-05-27T07:29:10.102 INFO:journalctl@ceph.mds.f.smithi173.stdout:May 27 07:29:09 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-f[73038]: starting mds.f at 2024-05-27T07:29:10.102 INFO:journalctl@ceph.mds.f.smithi173.stdout:May 27 07:29:09 smithi173 bash[73024]: 58968c237f2e2e320318fa2b66b26679470663234b1e13652788a79b15dc06e0 2024-05-27T07:29:10.102 INFO:journalctl@ceph.mds.f.smithi173.stdout:May 27 07:29:09 smithi173 systemd[1]: Started Ceph mds.f for 73df2636-1bf8-11ef-bc9b-c7b262605968. 2024-05-27T07:29:10.479 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:29:10 smithi052 ceph-mon[31955]: pgmap v106: 129 pgs: 129 active+clean; 176 MiB data, 952 MiB used, 1.0 TiB / 1.0 TiB avail; 426 KiB/s rd, 2.3 MiB/s wr, 786 op/s 2024-05-27T07:29:10.479 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:29:10 smithi052 ceph-mon[31955]: daemon mds.f restarted 2024-05-27T07:29:10.479 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:29:10 smithi052 ceph-mon[31955]: osdmap e99: 12 total, 12 up, 12 in 2024-05-27T07:29:10.480 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:29:10 smithi052 ceph-mon[31955]: mds.? [v2:172.21.15.173:6832/3660570821,v1:172.21.15.173:6833/3660570821] up:boot 2024-05-27T07:29:10.480 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:29:10 smithi052 ceph-mon[31955]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-05-27T07:29:10.480 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:29:10 smithi052 ceph-mon[31955]: from='mgr.15333 172.21.15.52:0/416389082' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2024-05-27T07:29:10.483 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:29:10 smithi173 ceph-mon[37000]: pgmap v106: 129 pgs: 129 active+clean; 176 MiB data, 952 MiB used, 1.0 TiB / 1.0 TiB avail; 426 KiB/s rd, 2.3 MiB/s wr, 786 op/s 2024-05-27T07:29:10.483 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:29:10 smithi173 ceph-mon[37000]: daemon mds.f restarted 2024-05-27T07:29:10.483 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:29:10 smithi173 ceph-mon[37000]: osdmap e99: 12 total, 12 up, 12 in 2024-05-27T07:29:10.483 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:29:10 smithi173 ceph-mon[37000]: mds.? [v2:172.21.15.173:6832/3660570821,v1:172.21.15.173:6833/3660570821] up:boot 2024-05-27T07:29:10.483 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:29:10 smithi173 ceph-mon[37000]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-05-27T07:29:10.483 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:29:10 smithi173 ceph-mon[37000]: from='mgr.15333 172.21.15.52:0/416389082' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2024-05-27T07:29:10.690 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:29:10 smithi105 ceph-mon[35977]: pgmap v106: 129 pgs: 129 active+clean; 176 MiB data, 952 MiB used, 1.0 TiB / 1.0 TiB avail; 426 KiB/s rd, 2.3 MiB/s wr, 786 op/s 2024-05-27T07:29:10.690 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:29:10 smithi105 ceph-mon[35977]: daemon mds.f restarted 2024-05-27T07:29:10.690 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:29:10 smithi105 ceph-mon[35977]: osdmap e99: 12 total, 12 up, 12 in 2024-05-27T07:29:10.690 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:29:10 smithi105 ceph-mon[35977]: mds.? [v2:172.21.15.173:6832/3660570821,v1:172.21.15.173:6833/3660570821] up:boot 2024-05-27T07:29:10.690 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:29:10 smithi105 ceph-mon[35977]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-05-27T07:29:10.690 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:29:10 smithi105 ceph-mon[35977]: from='mgr.15333 172.21.15.52:0/416389082' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2024-05-27T07:29:10.823 INFO:journalctl@ceph.mds.f.smithi173.stdout:May 27 07:29:10 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-f[73038]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: In function 'std::set SnapClient::filter(const std::set&) const' thread 7f7990227700 time 2024-05-27T07:29:10.494072+0000 2024-05-27T07:29:10.823 INFO:journalctl@ceph.mds.f.smithi173.stdout:May 27 07:29:10 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-f[73038]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: 197: FAILED ceph_assert(cached_version > 0) 2024-05-27T07:29:10.823 INFO:journalctl@ceph.mds.f.smithi173.stdout:May 27 07:29:10 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-f[73038]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:29:10.824 INFO:journalctl@ceph.mds.f.smithi173.stdout:May 27 07:29:10 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-f[73038]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x127) [0x7f79a2a6cba6] 2024-05-27T07:29:10.824 INFO:journalctl@ceph.mds.f.smithi173.stdout:May 27 07:29:10 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-f[73038]: 2: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7f79a2a6cdd1] 2024-05-27T07:29:10.824 INFO:journalctl@ceph.mds.f.smithi173.stdout:May 27 07:29:10 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-f[73038]: 3: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x55958101c23d] 2024-05-27T07:29:10.824 INFO:journalctl@ceph.mds.f.smithi173.stdout:May 27 07:29:10 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-f[73038]: 4: (SnapRealm::build_snap_set() const+0x323) [0x559581007971] 2024-05-27T07:29:10.824 INFO:journalctl@ceph.mds.f.smithi173.stdout:May 27 07:29:10 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-f[73038]: 5: (SnapRealm::check_cache() const+0x13e) [0x559581005fa0] 2024-05-27T07:29:10.824 INFO:journalctl@ceph.mds.f.smithi173.stdout:May 27 07:29:10 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-f[73038]: 6: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x559581006651] 2024-05-27T07:29:10.824 INFO:journalctl@ceph.mds.f.smithi173.stdout:May 27 07:29:10 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-f[73038]: 7: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x559580f9b347] 2024-05-27T07:29:10.824 INFO:journalctl@ceph.mds.f.smithi173.stdout:May 27 07:29:10 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-f[73038]: 8: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x559580c9787e] 2024-05-27T07:29:10.824 INFO:journalctl@ceph.mds.f.smithi173.stdout:May 27 07:29:10 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-f[73038]: 9: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x279d) [0x559580c9cf2f] 2024-05-27T07:29:10.824 INFO:journalctl@ceph.mds.f.smithi173.stdout:May 27 07:29:10 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-f[73038]: 10: (EUpdate::replay(MDSRank*)+0x63) [0x559580ca7969] 2024-05-27T07:29:10.824 INFO:journalctl@ceph.mds.f.smithi173.stdout:May 27 07:29:10 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-f[73038]: 11: (MDLog::_replay_thread()+0x1f38) [0x55958104048c] 2024-05-27T07:29:10.824 INFO:journalctl@ceph.mds.f.smithi173.stdout:May 27 07:29:10 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-f[73038]: 12: (MDLog::ReplayThread::entry()+0x11) [0x5595810499cb] 2024-05-27T07:29:10.824 INFO:journalctl@ceph.mds.f.smithi173.stdout:May 27 07:29:10 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-f[73038]: 13: (Thread::entry_wrapper()+0x43) [0x7f79a2a3b72d] 2024-05-27T07:29:10.824 INFO:journalctl@ceph.mds.f.smithi173.stdout:May 27 07:29:10 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-f[73038]: 14: (Thread::_entry_func(void*)+0xd) [0x7f79a2a3b749] 2024-05-27T07:29:10.825 INFO:journalctl@ceph.mds.f.smithi173.stdout:May 27 07:29:10 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-f[73038]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7f79a15df1ca] 2024-05-27T07:29:10.825 INFO:journalctl@ceph.mds.f.smithi173.stdout:May 27 07:29:10 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-f[73038]: 16: clone() 2024-05-27T07:29:10.825 INFO:journalctl@ceph.mds.f.smithi173.stdout:May 27 07:29:10 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-f[73038]: 2024-05-27T07:29:10.495+0000 7f7990227700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: In function 'std::set SnapClient::filter(const std::set&) const' thread 7f7990227700 time 2024-05-27T07:29:10.494072+0000 2024-05-27T07:29:10.826 INFO:journalctl@ceph.mds.f.smithi173.stdout:May 27 07:29:10 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-f[73038]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: 197: FAILED ceph_assert(cached_version > 0) 2024-05-27T07:29:10.826 INFO:journalctl@ceph.mds.f.smithi173.stdout:May 27 07:29:10 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-f[73038]: 2024-05-27T07:29:10.826 INFO:journalctl@ceph.mds.f.smithi173.stdout:May 27 07:29:10 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-f[73038]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:29:10.826 INFO:journalctl@ceph.mds.f.smithi173.stdout:May 27 07:29:10 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-f[73038]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x127) [0x7f79a2a6cba6] 2024-05-27T07:29:10.826 INFO:journalctl@ceph.mds.f.smithi173.stdout:May 27 07:29:10 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-f[73038]: 2: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7f79a2a6cdd1] 2024-05-27T07:29:10.826 INFO:journalctl@ceph.mds.f.smithi173.stdout:May 27 07:29:10 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-f[73038]: 3: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x55958101c23d] 2024-05-27T07:29:10.826 INFO:journalctl@ceph.mds.f.smithi173.stdout:May 27 07:29:10 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-f[73038]: 4: (SnapRealm::build_snap_set() const+0x323) [0x559581007971] 2024-05-27T07:29:10.826 INFO:journalctl@ceph.mds.f.smithi173.stdout:May 27 07:29:10 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-f[73038]: 5: (SnapRealm::check_cache() const+0x13e) [0x559581005fa0] 2024-05-27T07:29:10.826 INFO:journalctl@ceph.mds.f.smithi173.stdout:May 27 07:29:10 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-f[73038]: 6: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x559581006651] 2024-05-27T07:29:10.826 INFO:journalctl@ceph.mds.f.smithi173.stdout:May 27 07:29:10 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-f[73038]: 7: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x559580f9b347] 2024-05-27T07:29:10.826 INFO:journalctl@ceph.mds.f.smithi173.stdout:May 27 07:29:10 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-f[73038]: 8: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x559580c9787e] 2024-05-27T07:29:10.826 INFO:journalctl@ceph.mds.f.smithi173.stdout:May 27 07:29:10 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-f[73038]: 9: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x279d) [0x559580c9cf2f] 2024-05-27T07:29:10.826 INFO:journalctl@ceph.mds.f.smithi173.stdout:May 27 07:29:10 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-f[73038]: 10: (EUpdate::replay(MDSRank*)+0x63) [0x559580ca7969] 2024-05-27T07:29:10.827 INFO:journalctl@ceph.mds.f.smithi173.stdout:May 27 07:29:10 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-f[73038]: 11: (MDLog::_replay_thread()+0x1f38) [0x55958104048c] 2024-05-27T07:29:10.827 INFO:journalctl@ceph.mds.f.smithi173.stdout:May 27 07:29:10 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-f[73038]: 12: (MDLog::ReplayThread::entry()+0x11) [0x5595810499cb] 2024-05-27T07:29:10.827 INFO:journalctl@ceph.mds.f.smithi173.stdout:May 27 07:29:10 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-f[73038]: 13: (Thread::entry_wrapper()+0x43) [0x7f79a2a3b72d] 2024-05-27T07:29:10.827 INFO:journalctl@ceph.mds.f.smithi173.stdout:May 27 07:29:10 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-f[73038]: 14: (Thread::_entry_func(void*)+0xd) [0x7f79a2a3b749] 2024-05-27T07:29:10.827 INFO:journalctl@ceph.mds.f.smithi173.stdout:May 27 07:29:10 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-f[73038]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7f79a15df1ca] 2024-05-27T07:29:10.827 INFO:journalctl@ceph.mds.f.smithi173.stdout:May 27 07:29:10 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-f[73038]: 16: clone() 2024-05-27T07:29:10.827 INFO:journalctl@ceph.mds.f.smithi173.stdout:May 27 07:29:10 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-f[73038]: 2024-05-27T07:29:10.827 INFO:journalctl@ceph.mds.f.smithi173.stdout:May 27 07:29:10 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-f[73038]: *** Caught signal (Aborted) ** 2024-05-27T07:29:10.827 INFO:journalctl@ceph.mds.f.smithi173.stdout:May 27 07:29:10 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-f[73038]: in thread 7f7990227700 thread_name:md_log_replay 2024-05-27T07:29:10.827 INFO:journalctl@ceph.mds.f.smithi173.stdout:May 27 07:29:10 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-f[73038]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:29:10.827 INFO:journalctl@ceph.mds.f.smithi173.stdout:May 27 07:29:10 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-f[73038]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f79a15e9d20] 2024-05-27T07:29:10.827 INFO:journalctl@ceph.mds.f.smithi173.stdout:May 27 07:29:10 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-f[73038]: 2: gsignal() 2024-05-27T07:29:10.827 INFO:journalctl@ceph.mds.f.smithi173.stdout:May 27 07:29:10 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-f[73038]: 3: abort() 2024-05-27T07:29:10.827 INFO:journalctl@ceph.mds.f.smithi173.stdout:May 27 07:29:10 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-f[73038]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x26e) [0x7f79a2a6cced] 2024-05-27T07:29:10.828 INFO:journalctl@ceph.mds.f.smithi173.stdout:May 27 07:29:10 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-f[73038]: 5: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7f79a2a6cdd1] 2024-05-27T07:29:10.828 INFO:journalctl@ceph.mds.f.smithi173.stdout:May 27 07:29:10 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-f[73038]: 6: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x55958101c23d] 2024-05-27T07:29:10.828 INFO:journalctl@ceph.mds.f.smithi173.stdout:May 27 07:29:10 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-f[73038]: 7: (SnapRealm::build_snap_set() const+0x323) [0x559581007971] 2024-05-27T07:29:10.828 INFO:journalctl@ceph.mds.f.smithi173.stdout:May 27 07:29:10 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-f[73038]: 8: (SnapRealm::check_cache() const+0x13e) [0x559581005fa0] 2024-05-27T07:29:10.828 INFO:journalctl@ceph.mds.f.smithi173.stdout:May 27 07:29:10 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-f[73038]: 9: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x559581006651] 2024-05-27T07:29:10.828 INFO:journalctl@ceph.mds.f.smithi173.stdout:May 27 07:29:10 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-f[73038]: 10: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x559580f9b347] 2024-05-27T07:29:10.828 INFO:journalctl@ceph.mds.f.smithi173.stdout:May 27 07:29:10 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-f[73038]: 11: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x559580c9787e] 2024-05-27T07:29:10.828 INFO:journalctl@ceph.mds.f.smithi173.stdout:May 27 07:29:10 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-f[73038]: 12: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x279d) [0x559580c9cf2f] 2024-05-27T07:29:10.828 INFO:journalctl@ceph.mds.f.smithi173.stdout:May 27 07:29:10 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-f[73038]: 13: (EUpdate::replay(MDSRank*)+0x63) [0x559580ca7969] 2024-05-27T07:29:10.828 INFO:journalctl@ceph.mds.f.smithi173.stdout:May 27 07:29:10 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-f[73038]: 14: (MDLog::_replay_thread()+0x1f38) [0x55958104048c] 2024-05-27T07:29:10.828 INFO:journalctl@ceph.mds.f.smithi173.stdout:May 27 07:29:10 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-f[73038]: 15: (MDLog::ReplayThread::entry()+0x11) [0x5595810499cb] 2024-05-27T07:29:10.828 INFO:journalctl@ceph.mds.f.smithi173.stdout:May 27 07:29:10 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-f[73038]: 16: (Thread::entry_wrapper()+0x43) [0x7f79a2a3b72d] 2024-05-27T07:29:10.828 INFO:journalctl@ceph.mds.f.smithi173.stdout:May 27 07:29:10 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-f[73038]: 17: (Thread::_entry_func(void*)+0xd) [0x7f79a2a3b749] 2024-05-27T07:29:10.828 INFO:journalctl@ceph.mds.f.smithi173.stdout:May 27 07:29:10 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-f[73038]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7f79a15df1ca] 2024-05-27T07:29:10.828 INFO:journalctl@ceph.mds.f.smithi173.stdout:May 27 07:29:10 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-f[73038]: 19: clone() 2024-05-27T07:29:10.829 INFO:journalctl@ceph.mds.f.smithi173.stdout:May 27 07:29:10 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-f[73038]: 2024-05-27T07:29:10.498+0000 7f7990227700 -1 *** Caught signal (Aborted) ** 2024-05-27T07:29:10.829 INFO:journalctl@ceph.mds.f.smithi173.stdout:May 27 07:29:10 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-f[73038]: in thread 7f7990227700 thread_name:md_log_replay 2024-05-27T07:29:10.829 INFO:journalctl@ceph.mds.f.smithi173.stdout:May 27 07:29:10 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-f[73038]: 2024-05-27T07:29:10.829 INFO:journalctl@ceph.mds.f.smithi173.stdout:May 27 07:29:10 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-f[73038]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:29:10.829 INFO:journalctl@ceph.mds.f.smithi173.stdout:May 27 07:29:10 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-f[73038]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f79a15e9d20] 2024-05-27T07:29:10.829 INFO:journalctl@ceph.mds.f.smithi173.stdout:May 27 07:29:10 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-f[73038]: 2: gsignal() 2024-05-27T07:29:10.829 INFO:journalctl@ceph.mds.f.smithi173.stdout:May 27 07:29:10 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-f[73038]: 3: abort() 2024-05-27T07:29:10.829 INFO:journalctl@ceph.mds.f.smithi173.stdout:May 27 07:29:10 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-f[73038]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x26e) [0x7f79a2a6cced] 2024-05-27T07:29:10.829 INFO:journalctl@ceph.mds.f.smithi173.stdout:May 27 07:29:10 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-f[73038]: 5: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7f79a2a6cdd1] 2024-05-27T07:29:10.829 INFO:journalctl@ceph.mds.f.smithi173.stdout:May 27 07:29:10 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-f[73038]: 6: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x55958101c23d] 2024-05-27T07:29:10.829 INFO:journalctl@ceph.mds.f.smithi173.stdout:May 27 07:29:10 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-f[73038]: 7: (SnapRealm::build_snap_set() const+0x323) [0x559581007971] 2024-05-27T07:29:10.829 INFO:journalctl@ceph.mds.f.smithi173.stdout:May 27 07:29:10 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-f[73038]: 8: (SnapRealm::check_cache() const+0x13e) [0x559581005fa0] 2024-05-27T07:29:10.829 INFO:journalctl@ceph.mds.f.smithi173.stdout:May 27 07:29:10 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-f[73038]: 9: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x559581006651] 2024-05-27T07:29:10.829 INFO:journalctl@ceph.mds.f.smithi173.stdout:May 27 07:29:10 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-f[73038]: 10: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x559580f9b347] 2024-05-27T07:29:10.829 INFO:journalctl@ceph.mds.f.smithi173.stdout:May 27 07:29:10 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-f[73038]: 11: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x559580c9787e] 2024-05-27T07:29:10.830 INFO:journalctl@ceph.mds.f.smithi173.stdout:May 27 07:29:10 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-f[73038]: 12: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x279d) [0x559580c9cf2f] 2024-05-27T07:29:10.830 INFO:journalctl@ceph.mds.f.smithi173.stdout:May 27 07:29:10 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-f[73038]: 13: (EUpdate::replay(MDSRank*)+0x63) [0x559580ca7969] 2024-05-27T07:29:10.830 INFO:journalctl@ceph.mds.f.smithi173.stdout:May 27 07:29:10 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-f[73038]: 14: (MDLog::_replay_thread()+0x1f38) [0x55958104048c] 2024-05-27T07:29:10.830 INFO:journalctl@ceph.mds.f.smithi173.stdout:May 27 07:29:10 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-f[73038]: 15: (MDLog::ReplayThread::entry()+0x11) [0x5595810499cb] 2024-05-27T07:29:10.830 INFO:journalctl@ceph.mds.f.smithi173.stdout:May 27 07:29:10 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-f[73038]: 16: (Thread::entry_wrapper()+0x43) [0x7f79a2a3b72d] 2024-05-27T07:29:10.830 INFO:journalctl@ceph.mds.f.smithi173.stdout:May 27 07:29:10 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-f[73038]: 17: (Thread::_entry_func(void*)+0xd) [0x7f79a2a3b749] 2024-05-27T07:29:10.830 INFO:journalctl@ceph.mds.f.smithi173.stdout:May 27 07:29:10 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-f[73038]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7f79a15df1ca] 2024-05-27T07:29:10.830 INFO:journalctl@ceph.mds.f.smithi173.stdout:May 27 07:29:10 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-f[73038]: 19: clone() 2024-05-27T07:29:10.830 INFO:journalctl@ceph.mds.f.smithi173.stdout:May 27 07:29:10 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-f[73038]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-05-27T07:29:10.830 INFO:journalctl@ceph.mds.f.smithi173.stdout:May 27 07:29:10 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-f[73038]: 2024-05-27T07:29:10.830 INFO:journalctl@ceph.mds.f.smithi173.stdout:May 27 07:29:10 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-f[73038]: -1> 2024-05-27T07:29:10.495+0000 7f7990227700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: In function 'std::set SnapClient::filter(const std::set&) const' thread 7f7990227700 time 2024-05-27T07:29:10.494072+0000 2024-05-27T07:29:10.830 INFO:journalctl@ceph.mds.f.smithi173.stdout:May 27 07:29:10 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-f[73038]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: 197: FAILED ceph_assert(cached_version > 0) 2024-05-27T07:29:10.830 INFO:journalctl@ceph.mds.f.smithi173.stdout:May 27 07:29:10 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-f[73038]: 2024-05-27T07:29:10.830 INFO:journalctl@ceph.mds.f.smithi173.stdout:May 27 07:29:10 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-f[73038]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:29:10.831 INFO:journalctl@ceph.mds.f.smithi173.stdout:May 27 07:29:10 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-f[73038]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x127) [0x7f79a2a6cba6] 2024-05-27T07:29:10.831 INFO:journalctl@ceph.mds.f.smithi173.stdout:May 27 07:29:10 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-f[73038]: 2: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7f79a2a6cdd1] 2024-05-27T07:29:10.831 INFO:journalctl@ceph.mds.f.smithi173.stdout:May 27 07:29:10 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-f[73038]: 3: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x55958101c23d] 2024-05-27T07:29:10.831 INFO:journalctl@ceph.mds.f.smithi173.stdout:May 27 07:29:10 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-f[73038]: 4: (SnapRealm::build_snap_set() const+0x323) [0x559581007971] 2024-05-27T07:29:10.831 INFO:journalctl@ceph.mds.f.smithi173.stdout:May 27 07:29:10 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-f[73038]: 5: (SnapRealm::check_cache() const+0x13e) [0x559581005fa0] 2024-05-27T07:29:10.831 INFO:journalctl@ceph.mds.f.smithi173.stdout:May 27 07:29:10 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-f[73038]: 6: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x559581006651] 2024-05-27T07:29:10.831 INFO:journalctl@ceph.mds.f.smithi173.stdout:May 27 07:29:10 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-f[73038]: 7: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x559580f9b347] 2024-05-27T07:29:10.831 INFO:journalctl@ceph.mds.f.smithi173.stdout:May 27 07:29:10 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-f[73038]: 8: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x559580c9787e] 2024-05-27T07:29:10.831 INFO:journalctl@ceph.mds.f.smithi173.stdout:May 27 07:29:10 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-f[73038]: 9: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x279d) [0x559580c9cf2f] 2024-05-27T07:29:10.831 INFO:journalctl@ceph.mds.f.smithi173.stdout:May 27 07:29:10 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-f[73038]: 10: (EUpdate::replay(MDSRank*)+0x63) [0x559580ca7969] 2024-05-27T07:29:10.831 INFO:journalctl@ceph.mds.f.smithi173.stdout:May 27 07:29:10 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-f[73038]: 11: (MDLog::_replay_thread()+0x1f38) [0x55958104048c] 2024-05-27T07:29:10.831 INFO:journalctl@ceph.mds.f.smithi173.stdout:May 27 07:29:10 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-f[73038]: 12: (MDLog::ReplayThread::entry()+0x11) [0x5595810499cb] 2024-05-27T07:29:10.831 INFO:journalctl@ceph.mds.f.smithi173.stdout:May 27 07:29:10 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-f[73038]: 13: (Thread::entry_wrapper()+0x43) [0x7f79a2a3b72d] 2024-05-27T07:29:10.831 INFO:journalctl@ceph.mds.f.smithi173.stdout:May 27 07:29:10 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-f[73038]: 14: (Thread::_entry_func(void*)+0xd) [0x7f79a2a3b749] 2024-05-27T07:29:10.831 INFO:journalctl@ceph.mds.f.smithi173.stdout:May 27 07:29:10 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-f[73038]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7f79a15df1ca] 2024-05-27T07:29:10.832 INFO:journalctl@ceph.mds.f.smithi173.stdout:May 27 07:29:10 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-f[73038]: 16: clone() 2024-05-27T07:29:10.832 INFO:journalctl@ceph.mds.f.smithi173.stdout:May 27 07:29:10 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-f[73038]: 2024-05-27T07:29:10.832 INFO:journalctl@ceph.mds.f.smithi173.stdout:May 27 07:29:10 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-f[73038]: 0> 2024-05-27T07:29:10.498+0000 7f7990227700 -1 *** Caught signal (Aborted) ** 2024-05-27T07:29:10.832 INFO:journalctl@ceph.mds.f.smithi173.stdout:May 27 07:29:10 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-f[73038]: in thread 7f7990227700 thread_name:md_log_replay 2024-05-27T07:29:10.832 INFO:journalctl@ceph.mds.f.smithi173.stdout:May 27 07:29:10 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-f[73038]: 2024-05-27T07:29:10.832 INFO:journalctl@ceph.mds.f.smithi173.stdout:May 27 07:29:10 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-f[73038]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:29:10.832 INFO:journalctl@ceph.mds.f.smithi173.stdout:May 27 07:29:10 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-f[73038]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f79a15e9d20] 2024-05-27T07:29:10.832 INFO:journalctl@ceph.mds.f.smithi173.stdout:May 27 07:29:10 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-f[73038]: 2: gsignal() 2024-05-27T07:29:10.832 INFO:journalctl@ceph.mds.f.smithi173.stdout:May 27 07:29:10 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-f[73038]: 3: abort() 2024-05-27T07:29:10.832 INFO:journalctl@ceph.mds.f.smithi173.stdout:May 27 07:29:10 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-f[73038]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x26e) [0x7f79a2a6cced] 2024-05-27T07:29:10.832 INFO:journalctl@ceph.mds.f.smithi173.stdout:May 27 07:29:10 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-f[73038]: 5: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7f79a2a6cdd1] 2024-05-27T07:29:10.832 INFO:journalctl@ceph.mds.f.smithi173.stdout:May 27 07:29:10 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-f[73038]: 6: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x55958101c23d] 2024-05-27T07:29:10.832 INFO:journalctl@ceph.mds.f.smithi173.stdout:May 27 07:29:10 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-f[73038]: 7: (SnapRealm::build_snap_set() const+0x323) [0x559581007971] 2024-05-27T07:29:10.833 INFO:journalctl@ceph.mds.f.smithi173.stdout:May 27 07:29:10 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-f[73038]: 8: (SnapRealm::check_cache() const+0x13e) [0x559581005fa0] 2024-05-27T07:29:10.833 INFO:journalctl@ceph.mds.f.smithi173.stdout:May 27 07:29:10 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-f[73038]: 9: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x559581006651] 2024-05-27T07:29:10.833 INFO:journalctl@ceph.mds.f.smithi173.stdout:May 27 07:29:10 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-f[73038]: 10: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x559580f9b347] 2024-05-27T07:29:10.833 INFO:journalctl@ceph.mds.f.smithi173.stdout:May 27 07:29:10 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-f[73038]: 11: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x559580c9787e] 2024-05-27T07:29:10.833 INFO:journalctl@ceph.mds.f.smithi173.stdout:May 27 07:29:10 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-f[73038]: 12: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x279d) [0x559580c9cf2f] 2024-05-27T07:29:10.833 INFO:journalctl@ceph.mds.f.smithi173.stdout:May 27 07:29:10 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-f[73038]: 13: (EUpdate::replay(MDSRank*)+0x63) [0x559580ca7969] 2024-05-27T07:29:10.833 INFO:journalctl@ceph.mds.f.smithi173.stdout:May 27 07:29:10 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-f[73038]: 14: (MDLog::_replay_thread()+0x1f38) [0x55958104048c] 2024-05-27T07:29:10.833 INFO:journalctl@ceph.mds.f.smithi173.stdout:May 27 07:29:10 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-f[73038]: 15: (MDLog::ReplayThread::entry()+0x11) [0x5595810499cb] 2024-05-27T07:29:10.833 INFO:journalctl@ceph.mds.f.smithi173.stdout:May 27 07:29:10 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-f[73038]: 16: (Thread::entry_wrapper()+0x43) [0x7f79a2a3b72d] 2024-05-27T07:29:10.833 INFO:journalctl@ceph.mds.f.smithi173.stdout:May 27 07:29:10 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-f[73038]: 17: (Thread::_entry_func(void*)+0xd) [0x7f79a2a3b749] 2024-05-27T07:29:10.833 INFO:journalctl@ceph.mds.f.smithi173.stdout:May 27 07:29:10 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-f[73038]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7f79a15df1ca] 2024-05-27T07:29:10.833 INFO:journalctl@ceph.mds.f.smithi173.stdout:May 27 07:29:10 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-f[73038]: 19: clone() 2024-05-27T07:29:10.833 INFO:journalctl@ceph.mds.f.smithi173.stdout:May 27 07:29:10 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-f[73038]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-05-27T07:29:10.833 INFO:journalctl@ceph.mds.f.smithi173.stdout:May 27 07:29:10 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-f[73038]: 2024-05-27T07:29:10.833 INFO:journalctl@ceph.mds.f.smithi173.stdout:May 27 07:29:10 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-f[73038]: -1> 2024-05-27T07:29:10.495+0000 7f7990227700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: In function 'std::set SnapClient::filter(const std::set&) const' thread 7f7990227700 time 2024-05-27T07:29:10.494072+0000 2024-05-27T07:29:10.834 INFO:journalctl@ceph.mds.f.smithi173.stdout:May 27 07:29:10 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-f[73038]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: 197: FAILED ceph_assert(cached_version > 0) 2024-05-27T07:29:10.834 INFO:journalctl@ceph.mds.f.smithi173.stdout:May 27 07:29:10 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-f[73038]: 2024-05-27T07:29:10.834 INFO:journalctl@ceph.mds.f.smithi173.stdout:May 27 07:29:10 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-f[73038]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:29:10.834 INFO:journalctl@ceph.mds.f.smithi173.stdout:May 27 07:29:10 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-f[73038]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x127) [0x7f79a2a6cba6] 2024-05-27T07:29:10.834 INFO:journalctl@ceph.mds.f.smithi173.stdout:May 27 07:29:10 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-f[73038]: 2: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7f79a2a6cdd1] 2024-05-27T07:29:10.834 INFO:journalctl@ceph.mds.f.smithi173.stdout:May 27 07:29:10 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-f[73038]: 3: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x55958101c23d] 2024-05-27T07:29:10.834 INFO:journalctl@ceph.mds.f.smithi173.stdout:May 27 07:29:10 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-f[73038]: 4: (SnapRealm::build_snap_set() const+0x323) [0x559581007971] 2024-05-27T07:29:10.834 INFO:journalctl@ceph.mds.f.smithi173.stdout:May 27 07:29:10 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-f[73038]: 5: (SnapRealm::check_cache() const+0x13e) [0x559581005fa0] 2024-05-27T07:29:10.834 INFO:journalctl@ceph.mds.f.smithi173.stdout:May 27 07:29:10 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-f[73038]: 6: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x559581006651] 2024-05-27T07:29:10.834 INFO:journalctl@ceph.mds.f.smithi173.stdout:May 27 07:29:10 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-f[73038]: 7: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x559580f9b347] 2024-05-27T07:29:10.834 INFO:journalctl@ceph.mds.f.smithi173.stdout:May 27 07:29:10 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-f[73038]: 8: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x559580c9787e] 2024-05-27T07:29:10.834 INFO:journalctl@ceph.mds.f.smithi173.stdout:May 27 07:29:10 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-f[73038]: 9: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x279d) [0x559580c9cf2f] 2024-05-27T07:29:10.834 INFO:journalctl@ceph.mds.f.smithi173.stdout:May 27 07:29:10 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-f[73038]: 10: (EUpdate::replay(MDSRank*)+0x63) [0x559580ca7969] 2024-05-27T07:29:10.834 INFO:journalctl@ceph.mds.f.smithi173.stdout:May 27 07:29:10 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-f[73038]: 11: (MDLog::_replay_thread()+0x1f38) [0x55958104048c] 2024-05-27T07:29:10.835 INFO:journalctl@ceph.mds.f.smithi173.stdout:May 27 07:29:10 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-f[73038]: 12: (MDLog::ReplayThread::entry()+0x11) [0x5595810499cb] 2024-05-27T07:29:10.835 INFO:journalctl@ceph.mds.f.smithi173.stdout:May 27 07:29:10 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-f[73038]: 13: (Thread::entry_wrapper()+0x43) [0x7f79a2a3b72d] 2024-05-27T07:29:10.835 INFO:journalctl@ceph.mds.f.smithi173.stdout:May 27 07:29:10 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-f[73038]: 14: (Thread::_entry_func(void*)+0xd) [0x7f79a2a3b749] 2024-05-27T07:29:10.835 INFO:journalctl@ceph.mds.f.smithi173.stdout:May 27 07:29:10 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-f[73038]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7f79a15df1ca] 2024-05-27T07:29:10.835 INFO:journalctl@ceph.mds.f.smithi173.stdout:May 27 07:29:10 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-f[73038]: 16: clone() 2024-05-27T07:29:10.835 INFO:journalctl@ceph.mds.f.smithi173.stdout:May 27 07:29:10 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-f[73038]: 2024-05-27T07:29:10.835 INFO:journalctl@ceph.mds.f.smithi173.stdout:May 27 07:29:10 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-f[73038]: 0> 2024-05-27T07:29:10.498+0000 7f7990227700 -1 *** Caught signal (Aborted) ** 2024-05-27T07:29:10.835 INFO:journalctl@ceph.mds.f.smithi173.stdout:May 27 07:29:10 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-f[73038]: in thread 7f7990227700 thread_name:md_log_replay 2024-05-27T07:29:10.835 INFO:journalctl@ceph.mds.f.smithi173.stdout:May 27 07:29:10 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-f[73038]: 2024-05-27T07:29:10.835 INFO:journalctl@ceph.mds.f.smithi173.stdout:May 27 07:29:10 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-f[73038]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:29:10.835 INFO:journalctl@ceph.mds.f.smithi173.stdout:May 27 07:29:10 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-f[73038]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f79a15e9d20] 2024-05-27T07:29:10.836 INFO:journalctl@ceph.mds.f.smithi173.stdout:May 27 07:29:10 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-f[73038]: 2: gsignal() 2024-05-27T07:29:10.836 INFO:journalctl@ceph.mds.f.smithi173.stdout:May 27 07:29:10 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-f[73038]: 3: abort() 2024-05-27T07:29:10.836 INFO:journalctl@ceph.mds.f.smithi173.stdout:May 27 07:29:10 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-f[73038]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x26e) [0x7f79a2a6cced] 2024-05-27T07:29:10.836 INFO:journalctl@ceph.mds.f.smithi173.stdout:May 27 07:29:10 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-f[73038]: 5: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7f79a2a6cdd1] 2024-05-27T07:29:10.836 INFO:journalctl@ceph.mds.f.smithi173.stdout:May 27 07:29:10 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-f[73038]: 6: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x55958101c23d] 2024-05-27T07:29:10.836 INFO:journalctl@ceph.mds.f.smithi173.stdout:May 27 07:29:10 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-f[73038]: 7: (SnapRealm::build_snap_set() const+0x323) [0x559581007971] 2024-05-27T07:29:10.836 INFO:journalctl@ceph.mds.f.smithi173.stdout:May 27 07:29:10 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-f[73038]: 8: (SnapRealm::check_cache() const+0x13e) [0x559581005fa0] 2024-05-27T07:29:10.836 INFO:journalctl@ceph.mds.f.smithi173.stdout:May 27 07:29:10 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-f[73038]: 9: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x559581006651] 2024-05-27T07:29:10.836 INFO:journalctl@ceph.mds.f.smithi173.stdout:May 27 07:29:10 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-f[73038]: 10: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x559580f9b347] 2024-05-27T07:29:10.836 INFO:journalctl@ceph.mds.f.smithi173.stdout:May 27 07:29:10 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-f[73038]: 11: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x559580c9787e] 2024-05-27T07:29:10.836 INFO:journalctl@ceph.mds.f.smithi173.stdout:May 27 07:29:10 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-f[73038]: 12: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x279d) [0x559580c9cf2f] 2024-05-27T07:29:10.836 INFO:journalctl@ceph.mds.f.smithi173.stdout:May 27 07:29:10 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-f[73038]: 13: (EUpdate::replay(MDSRank*)+0x63) [0x559580ca7969] 2024-05-27T07:29:10.836 INFO:journalctl@ceph.mds.f.smithi173.stdout:May 27 07:29:10 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-f[73038]: 14: (MDLog::_replay_thread()+0x1f38) [0x55958104048c] 2024-05-27T07:29:10.836 INFO:journalctl@ceph.mds.f.smithi173.stdout:May 27 07:29:10 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-f[73038]: 15: (MDLog::ReplayThread::entry()+0x11) [0x5595810499cb] 2024-05-27T07:29:10.837 INFO:journalctl@ceph.mds.f.smithi173.stdout:May 27 07:29:10 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-f[73038]: 16: (Thread::entry_wrapper()+0x43) [0x7f79a2a3b72d] 2024-05-27T07:29:10.837 INFO:journalctl@ceph.mds.f.smithi173.stdout:May 27 07:29:10 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-f[73038]: 17: (Thread::_entry_func(void*)+0xd) [0x7f79a2a3b749] 2024-05-27T07:29:10.837 INFO:journalctl@ceph.mds.f.smithi173.stdout:May 27 07:29:10 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-f[73038]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7f79a15df1ca] 2024-05-27T07:29:10.837 INFO:journalctl@ceph.mds.f.smithi173.stdout:May 27 07:29:10 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-f[73038]: 19: clone() 2024-05-27T07:29:10.837 INFO:journalctl@ceph.mds.f.smithi173.stdout:May 27 07:29:10 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-f[73038]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-05-27T07:29:10.837 INFO:journalctl@ceph.mds.f.smithi173.stdout:May 27 07:29:10 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-f[73038]: 2024-05-27T07:29:10.837 INFO:journalctl@ceph.mds.f.smithi173.stdout:May 27 07:29:10 smithi173 podman[73073]: 2024-05-27 07:29:10.556367311 +0000 UTC m=+0.025827652 container died 58968c237f2e2e320318fa2b66b26679470663234b1e13652788a79b15dc06e0 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-f, GIT_CLEAN=True, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240521, CEPH_POINT_RELEASE=, ceph=True, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2) 2024-05-27T07:29:11.479 INFO:journalctl@ceph.mds.f.smithi173.stdout:May 27 07:29:11 smithi173 podman[73073]: 2024-05-27 07:29:11.176941607 +0000 UTC m=+0.646401923 container remove 58968c237f2e2e320318fa2b66b26679470663234b1e13652788a79b15dc06e0 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-f, ceph=True, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.build-date=20240521) 2024-05-27T07:29:11.479 INFO:journalctl@ceph.mds.f.smithi173.stdout:May 27 07:29:11 smithi173 systemd[1]: ceph-73df2636-1bf8-11ef-bc9b-c7b262605968@mds.f.service: Main process exited, code=exited, status=134/n/a 2024-05-27T07:29:11.852 INFO:journalctl@ceph.mds.f.smithi173.stdout:May 27 07:29:11 smithi173 systemd[1]: ceph-73df2636-1bf8-11ef-bc9b-c7b262605968@mds.f.service: Failed with result 'exit-code'. 2024-05-27T07:29:12.479 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:29:12 smithi052 ceph-mon[31955]: pgmap v108: 129 pgs: 129 active+clean; 179 MiB data, 961 MiB used, 1.0 TiB / 1.0 TiB avail; 677 KiB/s rd, 3.1 MiB/s wr, 1.15k op/s 2024-05-27T07:29:12.602 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:29:12 smithi173 ceph-mon[37000]: pgmap v108: 129 pgs: 129 active+clean; 179 MiB data, 961 MiB used, 1.0 TiB / 1.0 TiB avail; 677 KiB/s rd, 3.1 MiB/s wr, 1.15k op/s 2024-05-27T07:29:12.689 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:29:12 smithi105 ceph-mon[35977]: pgmap v108: 129 pgs: 129 active+clean; 179 MiB data, 961 MiB used, 1.0 TiB / 1.0 TiB avail; 677 KiB/s rd, 3.1 MiB/s wr, 1.15k op/s 2024-05-27T07:29:13.128 INFO:journalctl@ceph.mds.l.smithi173.stdout:May 27 07:29:12 smithi173 systemd[1]: ceph-73df2636-1bf8-11ef-bc9b-c7b262605968@mds.l.service: Scheduled restart job, restart counter is at 1. 2024-05-27T07:29:13.129 INFO:journalctl@ceph.mds.l.smithi173.stdout:May 27 07:29:12 smithi173 systemd[1]: Stopped Ceph mds.l for 73df2636-1bf8-11ef-bc9b-c7b262605968. 2024-05-27T07:29:13.129 INFO:journalctl@ceph.mds.l.smithi173.stdout:May 27 07:29:12 smithi173 systemd[1]: ceph-73df2636-1bf8-11ef-bc9b-c7b262605968@mds.l.service: Consumed 1.428s CPU time. 2024-05-27T07:29:13.129 INFO:journalctl@ceph.mds.l.smithi173.stdout:May 27 07:29:12 smithi173 systemd[1]: Starting Ceph mds.l for 73df2636-1bf8-11ef-bc9b-c7b262605968... 2024-05-27T07:29:13.384 INFO:journalctl@ceph.mds.l.smithi173.stdout:May 27 07:29:13 smithi173 podman[73165]: 2024-05-27 07:29:13.043117286 +0000 UTC m=+0.033007416 image pull e36ea4fc8a5807e8d2701370e350a0de3856538797f7fba64cfce7e344bd6799 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6 2024-05-27T07:29:13.646 INFO:journalctl@ceph.mds.l.smithi173.stdout:May 27 07:29:13 smithi173 podman[73165]: 2024-05-27 07:29:13.394787613 +0000 UTC m=+0.384677722 container create 8b5d2550a74e1afcc42419f2b5079ccf25e4d0aa8ae654fc35c85b6f7e325d3d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-l, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, GIT_CLEAN=True, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, ceph=True, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240521) 2024-05-27T07:29:13.922 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:29:13.931+0000 7f7030ff9640 1 -- 172.21.15.52:0/1496268030 <== mds.0 v2:172.21.15.105:6838/500348151 1 ==== command_reply(tid 0: 0 ) ==== 8+0+25 (crc 0 0 0) 0x55d6710f7360 con 0x55d670d7c1c0 2024-05-27T07:29:13.922 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:29:13.931+0000 7f7030ff9640 10 client.25871 handle_command_reply: tid=0 2024-05-27T07:29:13.922 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:29:13.931+0000 7f7056234740 1 client.25871 shutdown 2024-05-27T07:29:13.922 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:29:13.931+0000 7f7056234740 1 -- 172.21.15.52:0/1496268030 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55d670dc74d0 msgr2=0x55d6711b6ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:29:13.922 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:29:13.931+0000 7f7056234740 1 --2- 172.21.15.52:0/1496268030 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55d670dc74d0 0x55d6711b6ee0 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x55d67118a780 tx=0x7f70440044d0 comp rx=0 tx=0).stop 2024-05-27T07:29:13.922 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:29:13.931+0000 7f7056234740 1 -- 172.21.15.52:0/1496268030 shutdown_connections 2024-05-27T07:29:13.922 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:29:13.931+0000 7f7056234740 1 --2- 172.21.15.52:0/1496268030 >> [v2:172.21.15.105:6838/500348151,v1:172.21.15.105:6839/500348151] conn(0x55d670d7c1c0 0x55d6712101e0 crc :-1 s=READY pgs=92 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:29:13.922 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:29:13.931+0000 7f7056234740 1 --2- 172.21.15.52:0/1496268030 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55d670de81b0 0x55d6712063d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:29:13.922 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:29:13.931+0000 7f7056234740 1 --2- 172.21.15.52:0/1496268030 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55d670dc74d0 0x55d6711b6ee0 unknown :-1 s=CLOSED pgs=117 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:29:13.923 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:29:13.931+0000 7f7030ff9640 0 client.25871 ms_handle_reset on v2:172.21.15.105:6838/500348151 2024-05-27T07:29:13.923 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:29:13.931+0000 7f7056234740 1 --2- 172.21.15.52:0/1496268030 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55d670dbf2f0 0x55d6711b69a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:29:13.923 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:29:13.931+0000 7f7056234740 1 -- 172.21.15.52:0/1496268030 >> 172.21.15.52:0/1496268030 conn(0x55d670ddc300 msgr2=0x55d67118d620 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:29:13.923 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:29:13.932+0000 7f7056234740 1 -- 172.21.15.52:0/1496268030 shutdown_connections 2024-05-27T07:29:13.923 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:29:13.932+0000 7f7056234740 1 -- 172.21.15.52:0/1496268030 wait complete. 2024-05-27T07:29:13.923 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:29:13.932+0000 7f7056234740 20 client.25871 trim_cache size 0 max 16384 2024-05-27T07:29:13.923 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stdout:{ 2024-05-27T07:29:13.923 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stdout: "return_code": 0 2024-05-27T07:29:13.923 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stdout:} 2024-05-27T07:29:13.924 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:29:13.932+0000 7f704f37d640 1 -- 172.21.15.52:0/3723986285 >> [v2:172.21.15.52:6800/1822318384,v1:172.21.15.52:6801/1822318384] conn(0x7f70340b2a50 msgr2=0x7f70340b4f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:29:13.924 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:29:13.932+0000 7f704f37d640 1 --2- 172.21.15.52:0/3723986285 >> [v2:172.21.15.52:6800/1822318384,v1:172.21.15.52:6801/1822318384] conn(0x7f70340b2a50 0x7f70340b4f60 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7f70400a97d0 tx=0x7f704000cf50 comp rx=0 tx=0).stop 2024-05-27T07:29:13.924 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:29:13.932+0000 7f704f37d640 1 -- 172.21.15.52:0/3723986285 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f7048109040 msgr2=0x7f7048199f40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:29:13.924 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:29:13.932+0000 7f704f37d640 1 --2- 172.21.15.52:0/3723986285 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f7048109040 0x7f7048199f40 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7f704000eaa0 tx=0x7f704000ef60 comp rx=0 tx=0).stop 2024-05-27T07:29:13.925 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:29:13.932+0000 7f704f37d640 1 -- 172.21.15.52:0/3723986285 shutdown_connections 2024-05-27T07:29:13.925 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:29:13.932+0000 7f704f37d640 1 --2- 172.21.15.52:0/3723986285 >> [v2:172.21.15.52:6800/1822318384,v1:172.21.15.52:6801/1822318384] conn(0x7f70340b2a50 0x7f70340b4f60 unknown :-1 s=CLOSED pgs=102 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:29:13.925 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:29:13.932+0000 7f704f37d640 1 --2- 172.21.15.52:0/3723986285 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f7048109040 0x7f7048199f40 unknown :-1 s=CLOSED pgs=115 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:29:13.925 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:29:13.932+0000 7f704f37d640 1 --2- 172.21.15.52:0/3723986285 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f7048108ba0 0x7f7048199a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:29:13.925 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:29:13.932+0000 7f704f37d640 1 --2- 172.21.15.52:0/3723986285 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f70481032f0 0x7f70481994c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:29:13.925 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:29:13.932+0000 7f704f37d640 1 -- 172.21.15.52:0/3723986285 >> 172.21.15.52:0/3723986285 conn(0x7f7048078270 msgr2=0x7f7048101360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:29:13.925 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:29:13.932+0000 7f704f37d640 1 -- 172.21.15.52:0/3723986285 shutdown_connections 2024-05-27T07:29:13.925 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:29:13.932+0000 7f704f37d640 1 -- 172.21.15.52:0/3723986285 wait complete. 2024-05-27T07:29:14.102 INFO:journalctl@ceph.mds.l.smithi173.stdout:May 27 07:29:13 smithi173 podman[73165]: 2024-05-27 07:29:13.656809669 +0000 UTC m=+0.646699786 container init 8b5d2550a74e1afcc42419f2b5079ccf25e4d0aa8ae654fc35c85b6f7e325d3d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-l, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, io.buildah.version=1.33.5, GIT_CLEAN=True, org.label-schema.build-date=20240521, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2) 2024-05-27T07:29:14.102 INFO:journalctl@ceph.mds.l.smithi173.stdout:May 27 07:29:13 smithi173 podman[73165]: 2024-05-27 07:29:13.660258305 +0000 UTC m=+0.650148427 container start 8b5d2550a74e1afcc42419f2b5079ccf25e4d0aa8ae654fc35c85b6f7e325d3d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-l, CEPH_POINT_RELEASE=, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, org.label-schema.build-date=20240521, GIT_BRANCH=HEAD, GIT_CLEAN=True, ceph=True, io.buildah.version=1.33.5) 2024-05-27T07:29:14.102 INFO:journalctl@ceph.mds.l.smithi173.stdout:May 27 07:29:13 smithi173 ceph-mds[73184]: -- 172.21.15.173:0/3683475663 <== mon.2 v2:172.21.15.105:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x561ba34c7a40 con 0x561ba350fa80 2024-05-27T07:29:14.102 INFO:journalctl@ceph.mds.l.smithi173.stdout:May 27 07:29:13 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-l[73180]: starting mds.l at 2024-05-27T07:29:14.102 INFO:journalctl@ceph.mds.l.smithi173.stdout:May 27 07:29:13 smithi173 bash[73165]: 8b5d2550a74e1afcc42419f2b5079ccf25e4d0aa8ae654fc35c85b6f7e325d3d 2024-05-27T07:29:14.102 INFO:journalctl@ceph.mds.l.smithi173.stdout:May 27 07:29:13 smithi173 systemd[1]: Started Ceph mds.l for 73df2636-1bf8-11ef-bc9b-c7b262605968. 2024-05-27T07:29:14.479 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:29:14 smithi052 ceph-mon[31955]: pgmap v109: 129 pgs: 129 active+clean; 182 MiB data, 957 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 2.3 MiB/s wr, 836 op/s 2024-05-27T07:29:14.479 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:29:14 smithi052 ceph-mon[31955]: daemon mds.l restarted 2024-05-27T07:29:14.601 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:29:14 smithi173 ceph-mon[37000]: pgmap v109: 129 pgs: 129 active+clean; 182 MiB data, 957 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 2.3 MiB/s wr, 836 op/s 2024-05-27T07:29:14.602 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:29:14 smithi173 ceph-mon[37000]: daemon mds.l restarted 2024-05-27T07:29:14.602 INFO:journalctl@ceph.mds.l.smithi173.stdout:May 27 07:29:14 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-l[73180]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: In function 'std::set SnapClient::filter(const std::set&) const' thread 7fee2d7fe700 time 2024-05-27T07:29:14.305580+0000 2024-05-27T07:29:14.602 INFO:journalctl@ceph.mds.l.smithi173.stdout:May 27 07:29:14 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-l[73180]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: 197: FAILED ceph_assert(cached_version > 0) 2024-05-27T07:29:14.602 INFO:journalctl@ceph.mds.l.smithi173.stdout:May 27 07:29:14 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-l[73180]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:29:14.602 INFO:journalctl@ceph.mds.l.smithi173.stdout:May 27 07:29:14 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-l[73180]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x127) [0x7fee40043ba6] 2024-05-27T07:29:14.602 INFO:journalctl@ceph.mds.l.smithi173.stdout:May 27 07:29:14 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-l[73180]: 2: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7fee40043dd1] 2024-05-27T07:29:14.603 INFO:journalctl@ceph.mds.l.smithi173.stdout:May 27 07:29:14 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-l[73180]: 3: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x561ba1ed323d] 2024-05-27T07:29:14.603 INFO:journalctl@ceph.mds.l.smithi173.stdout:May 27 07:29:14 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-l[73180]: 4: (SnapRealm::build_snap_set() const+0x323) [0x561ba1ebe971] 2024-05-27T07:29:14.603 INFO:journalctl@ceph.mds.l.smithi173.stdout:May 27 07:29:14 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-l[73180]: 5: (SnapRealm::check_cache() const+0x13e) [0x561ba1ebcfa0] 2024-05-27T07:29:14.603 INFO:journalctl@ceph.mds.l.smithi173.stdout:May 27 07:29:14 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-l[73180]: 6: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x561ba1ebd651] 2024-05-27T07:29:14.603 INFO:journalctl@ceph.mds.l.smithi173.stdout:May 27 07:29:14 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-l[73180]: 7: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x561ba1e52347] 2024-05-27T07:29:14.603 INFO:journalctl@ceph.mds.l.smithi173.stdout:May 27 07:29:14 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-l[73180]: 8: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x561ba1b4e87e] 2024-05-27T07:29:14.603 INFO:journalctl@ceph.mds.l.smithi173.stdout:May 27 07:29:14 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-l[73180]: 9: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x2702) [0x561ba1b53e94] 2024-05-27T07:29:14.603 INFO:journalctl@ceph.mds.l.smithi173.stdout:May 27 07:29:14 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-l[73180]: 10: (EUpdate::replay(MDSRank*)+0x63) [0x561ba1b5e969] 2024-05-27T07:29:14.603 INFO:journalctl@ceph.mds.l.smithi173.stdout:May 27 07:29:14 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-l[73180]: 11: (MDLog::_replay_thread()+0x1f38) [0x561ba1ef748c] 2024-05-27T07:29:14.603 INFO:journalctl@ceph.mds.l.smithi173.stdout:May 27 07:29:14 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-l[73180]: 12: (MDLog::ReplayThread::entry()+0x11) [0x561ba1f009cb] 2024-05-27T07:29:14.603 INFO:journalctl@ceph.mds.l.smithi173.stdout:May 27 07:29:14 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-l[73180]: 13: (Thread::entry_wrapper()+0x43) [0x7fee4001272d] 2024-05-27T07:29:14.603 INFO:journalctl@ceph.mds.l.smithi173.stdout:May 27 07:29:14 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-l[73180]: 14: (Thread::_entry_func(void*)+0xd) [0x7fee40012749] 2024-05-27T07:29:14.603 INFO:journalctl@ceph.mds.l.smithi173.stdout:May 27 07:29:14 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-l[73180]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7fee3ebb61ca] 2024-05-27T07:29:14.604 INFO:journalctl@ceph.mds.l.smithi173.stdout:May 27 07:29:14 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-l[73180]: 16: clone() 2024-05-27T07:29:14.604 INFO:journalctl@ceph.mds.l.smithi173.stdout:May 27 07:29:14 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-l[73180]: *** Caught signal (Aborted) ** 2024-05-27T07:29:14.604 INFO:journalctl@ceph.mds.l.smithi173.stdout:May 27 07:29:14 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-l[73180]: in thread 7fee2d7fe700 thread_name:md_log_replay 2024-05-27T07:29:14.604 INFO:journalctl@ceph.mds.l.smithi173.stdout:May 27 07:29:14 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-l[73180]: 2024-05-27T07:29:14.306+0000 7fee2d7fe700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: In function 'std::set SnapClient::filter(const std::set&) const' thread 7fee2d7fe700 time 2024-05-27T07:29:14.305580+0000 2024-05-27T07:29:14.604 INFO:journalctl@ceph.mds.l.smithi173.stdout:May 27 07:29:14 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-l[73180]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: 197: FAILED ceph_assert(cached_version > 0) 2024-05-27T07:29:14.605 INFO:journalctl@ceph.mds.l.smithi173.stdout:May 27 07:29:14 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-l[73180]: 2024-05-27T07:29:14.605 INFO:journalctl@ceph.mds.l.smithi173.stdout:May 27 07:29:14 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-l[73180]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:29:14.605 INFO:journalctl@ceph.mds.l.smithi173.stdout:May 27 07:29:14 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-l[73180]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x127) [0x7fee40043ba6] 2024-05-27T07:29:14.605 INFO:journalctl@ceph.mds.l.smithi173.stdout:May 27 07:29:14 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-l[73180]: 2: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7fee40043dd1] 2024-05-27T07:29:14.605 INFO:journalctl@ceph.mds.l.smithi173.stdout:May 27 07:29:14 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-l[73180]: 3: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x561ba1ed323d] 2024-05-27T07:29:14.605 INFO:journalctl@ceph.mds.l.smithi173.stdout:May 27 07:29:14 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-l[73180]: 4: (SnapRealm::build_snap_set() const+0x323) [0x561ba1ebe971] 2024-05-27T07:29:14.605 INFO:journalctl@ceph.mds.l.smithi173.stdout:May 27 07:29:14 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-l[73180]: 5: (SnapRealm::check_cache() const+0x13e) [0x561ba1ebcfa0] 2024-05-27T07:29:14.605 INFO:journalctl@ceph.mds.l.smithi173.stdout:May 27 07:29:14 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-l[73180]: 6: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x561ba1ebd651] 2024-05-27T07:29:14.605 INFO:journalctl@ceph.mds.l.smithi173.stdout:May 27 07:29:14 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-l[73180]: 7: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x561ba1e52347] 2024-05-27T07:29:14.605 INFO:journalctl@ceph.mds.l.smithi173.stdout:May 27 07:29:14 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-l[73180]: 8: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x561ba1b4e87e] 2024-05-27T07:29:14.605 INFO:journalctl@ceph.mds.l.smithi173.stdout:May 27 07:29:14 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-l[73180]: 9: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x2702) [0x561ba1b53e94] 2024-05-27T07:29:14.606 INFO:journalctl@ceph.mds.l.smithi173.stdout:May 27 07:29:14 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-l[73180]: 10: (EUpdate::replay(MDSRank*)+0x63) [0x561ba1b5e969] 2024-05-27T07:29:14.606 INFO:journalctl@ceph.mds.l.smithi173.stdout:May 27 07:29:14 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-l[73180]: 11: (MDLog::_replay_thread()+0x1f38) [0x561ba1ef748c] 2024-05-27T07:29:14.606 INFO:journalctl@ceph.mds.l.smithi173.stdout:May 27 07:29:14 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-l[73180]: 12: (MDLog::ReplayThread::entry()+0x11) [0x561ba1f009cb] 2024-05-27T07:29:14.606 INFO:journalctl@ceph.mds.l.smithi173.stdout:May 27 07:29:14 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-l[73180]: 13: (Thread::entry_wrapper()+0x43) [0x7fee4001272d] 2024-05-27T07:29:14.606 INFO:journalctl@ceph.mds.l.smithi173.stdout:May 27 07:29:14 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-l[73180]: 14: (Thread::_entry_func(void*)+0xd) [0x7fee40012749] 2024-05-27T07:29:14.606 INFO:journalctl@ceph.mds.l.smithi173.stdout:May 27 07:29:14 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-l[73180]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7fee3ebb61ca] 2024-05-27T07:29:14.606 INFO:journalctl@ceph.mds.l.smithi173.stdout:May 27 07:29:14 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-l[73180]: 16: clone() 2024-05-27T07:29:14.606 INFO:journalctl@ceph.mds.l.smithi173.stdout:May 27 07:29:14 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-l[73180]: 2024-05-27T07:29:14.606 INFO:journalctl@ceph.mds.l.smithi173.stdout:May 27 07:29:14 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-l[73180]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:29:14.606 INFO:journalctl@ceph.mds.l.smithi173.stdout:May 27 07:29:14 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-l[73180]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7fee3ebc0d20] 2024-05-27T07:29:14.606 INFO:journalctl@ceph.mds.l.smithi173.stdout:May 27 07:29:14 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-l[73180]: 2: gsignal() 2024-05-27T07:29:14.606 INFO:journalctl@ceph.mds.l.smithi173.stdout:May 27 07:29:14 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-l[73180]: 3: abort() 2024-05-27T07:29:14.606 INFO:journalctl@ceph.mds.l.smithi173.stdout:May 27 07:29:14 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-l[73180]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x26e) [0x7fee40043ced] 2024-05-27T07:29:14.607 INFO:journalctl@ceph.mds.l.smithi173.stdout:May 27 07:29:14 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-l[73180]: 5: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7fee40043dd1] 2024-05-27T07:29:14.607 INFO:journalctl@ceph.mds.l.smithi173.stdout:May 27 07:29:14 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-l[73180]: 6: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x561ba1ed323d] 2024-05-27T07:29:14.607 INFO:journalctl@ceph.mds.l.smithi173.stdout:May 27 07:29:14 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-l[73180]: 7: (SnapRealm::build_snap_set() const+0x323) [0x561ba1ebe971] 2024-05-27T07:29:14.607 INFO:journalctl@ceph.mds.l.smithi173.stdout:May 27 07:29:14 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-l[73180]: 8: (SnapRealm::check_cache() const+0x13e) [0x561ba1ebcfa0] 2024-05-27T07:29:14.607 INFO:journalctl@ceph.mds.l.smithi173.stdout:May 27 07:29:14 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-l[73180]: 9: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x561ba1ebd651] 2024-05-27T07:29:14.607 INFO:journalctl@ceph.mds.l.smithi173.stdout:May 27 07:29:14 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-l[73180]: 10: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x561ba1e52347] 2024-05-27T07:29:14.607 INFO:journalctl@ceph.mds.l.smithi173.stdout:May 27 07:29:14 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-l[73180]: 11: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x561ba1b4e87e] 2024-05-27T07:29:14.607 INFO:journalctl@ceph.mds.l.smithi173.stdout:May 27 07:29:14 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-l[73180]: 12: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x2702) [0x561ba1b53e94] 2024-05-27T07:29:14.607 INFO:journalctl@ceph.mds.l.smithi173.stdout:May 27 07:29:14 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-l[73180]: 13: (EUpdate::replay(MDSRank*)+0x63) [0x561ba1b5e969] 2024-05-27T07:29:14.607 INFO:journalctl@ceph.mds.l.smithi173.stdout:May 27 07:29:14 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-l[73180]: 14: (MDLog::_replay_thread()+0x1f38) [0x561ba1ef748c] 2024-05-27T07:29:14.607 INFO:journalctl@ceph.mds.l.smithi173.stdout:May 27 07:29:14 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-l[73180]: 15: (MDLog::ReplayThread::entry()+0x11) [0x561ba1f009cb] 2024-05-27T07:29:14.607 INFO:journalctl@ceph.mds.l.smithi173.stdout:May 27 07:29:14 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-l[73180]: 16: (Thread::entry_wrapper()+0x43) [0x7fee4001272d] 2024-05-27T07:29:14.607 INFO:journalctl@ceph.mds.l.smithi173.stdout:May 27 07:29:14 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-l[73180]: 17: (Thread::_entry_func(void*)+0xd) [0x7fee40012749] 2024-05-27T07:29:14.607 INFO:journalctl@ceph.mds.l.smithi173.stdout:May 27 07:29:14 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-l[73180]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7fee3ebb61ca] 2024-05-27T07:29:14.607 INFO:journalctl@ceph.mds.l.smithi173.stdout:May 27 07:29:14 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-l[73180]: 19: clone() 2024-05-27T07:29:14.608 INFO:journalctl@ceph.mds.l.smithi173.stdout:May 27 07:29:14 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-l[73180]: 2024-05-27T07:29:14.309+0000 7fee2d7fe700 -1 *** Caught signal (Aborted) ** 2024-05-27T07:29:14.608 INFO:journalctl@ceph.mds.l.smithi173.stdout:May 27 07:29:14 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-l[73180]: in thread 7fee2d7fe700 thread_name:md_log_replay 2024-05-27T07:29:14.608 INFO:journalctl@ceph.mds.l.smithi173.stdout:May 27 07:29:14 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-l[73180]: 2024-05-27T07:29:14.608 INFO:journalctl@ceph.mds.l.smithi173.stdout:May 27 07:29:14 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-l[73180]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:29:14.608 INFO:journalctl@ceph.mds.l.smithi173.stdout:May 27 07:29:14 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-l[73180]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7fee3ebc0d20] 2024-05-27T07:29:14.608 INFO:journalctl@ceph.mds.l.smithi173.stdout:May 27 07:29:14 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-l[73180]: 2: gsignal() 2024-05-27T07:29:14.608 INFO:journalctl@ceph.mds.l.smithi173.stdout:May 27 07:29:14 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-l[73180]: 3: abort() 2024-05-27T07:29:14.608 INFO:journalctl@ceph.mds.l.smithi173.stdout:May 27 07:29:14 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-l[73180]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x26e) [0x7fee40043ced] 2024-05-27T07:29:14.608 INFO:journalctl@ceph.mds.l.smithi173.stdout:May 27 07:29:14 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-l[73180]: 5: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7fee40043dd1] 2024-05-27T07:29:14.608 INFO:journalctl@ceph.mds.l.smithi173.stdout:May 27 07:29:14 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-l[73180]: 6: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x561ba1ed323d] 2024-05-27T07:29:14.608 INFO:journalctl@ceph.mds.l.smithi173.stdout:May 27 07:29:14 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-l[73180]: 7: (SnapRealm::build_snap_set() const+0x323) [0x561ba1ebe971] 2024-05-27T07:29:14.608 INFO:journalctl@ceph.mds.l.smithi173.stdout:May 27 07:29:14 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-l[73180]: 8: (SnapRealm::check_cache() const+0x13e) [0x561ba1ebcfa0] 2024-05-27T07:29:14.608 INFO:journalctl@ceph.mds.l.smithi173.stdout:May 27 07:29:14 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-l[73180]: 9: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x561ba1ebd651] 2024-05-27T07:29:14.608 INFO:journalctl@ceph.mds.l.smithi173.stdout:May 27 07:29:14 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-l[73180]: 10: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x561ba1e52347] 2024-05-27T07:29:14.608 INFO:journalctl@ceph.mds.l.smithi173.stdout:May 27 07:29:14 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-l[73180]: 11: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x561ba1b4e87e] 2024-05-27T07:29:14.609 INFO:journalctl@ceph.mds.l.smithi173.stdout:May 27 07:29:14 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-l[73180]: 12: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x2702) [0x561ba1b53e94] 2024-05-27T07:29:14.609 INFO:journalctl@ceph.mds.l.smithi173.stdout:May 27 07:29:14 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-l[73180]: 13: (EUpdate::replay(MDSRank*)+0x63) [0x561ba1b5e969] 2024-05-27T07:29:14.609 INFO:journalctl@ceph.mds.l.smithi173.stdout:May 27 07:29:14 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-l[73180]: 14: (MDLog::_replay_thread()+0x1f38) [0x561ba1ef748c] 2024-05-27T07:29:14.609 INFO:journalctl@ceph.mds.l.smithi173.stdout:May 27 07:29:14 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-l[73180]: 15: (MDLog::ReplayThread::entry()+0x11) [0x561ba1f009cb] 2024-05-27T07:29:14.609 INFO:journalctl@ceph.mds.l.smithi173.stdout:May 27 07:29:14 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-l[73180]: 16: (Thread::entry_wrapper()+0x43) [0x7fee4001272d] 2024-05-27T07:29:14.609 INFO:journalctl@ceph.mds.l.smithi173.stdout:May 27 07:29:14 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-l[73180]: 17: (Thread::_entry_func(void*)+0xd) [0x7fee40012749] 2024-05-27T07:29:14.609 INFO:journalctl@ceph.mds.l.smithi173.stdout:May 27 07:29:14 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-l[73180]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7fee3ebb61ca] 2024-05-27T07:29:14.609 INFO:journalctl@ceph.mds.l.smithi173.stdout:May 27 07:29:14 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-l[73180]: 19: clone() 2024-05-27T07:29:14.609 INFO:journalctl@ceph.mds.l.smithi173.stdout:May 27 07:29:14 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-l[73180]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-05-27T07:29:14.609 INFO:journalctl@ceph.mds.l.smithi173.stdout:May 27 07:29:14 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-l[73180]: 2024-05-27T07:29:14.609 INFO:journalctl@ceph.mds.l.smithi173.stdout:May 27 07:29:14 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-l[73180]: -1> 2024-05-27T07:29:14.306+0000 7fee2d7fe700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: In function 'std::set SnapClient::filter(const std::set&) const' thread 7fee2d7fe700 time 2024-05-27T07:29:14.305580+0000 2024-05-27T07:29:14.609 INFO:journalctl@ceph.mds.l.smithi173.stdout:May 27 07:29:14 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-l[73180]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: 197: FAILED ceph_assert(cached_version > 0) 2024-05-27T07:29:14.609 INFO:journalctl@ceph.mds.l.smithi173.stdout:May 27 07:29:14 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-l[73180]: 2024-05-27T07:29:14.609 INFO:journalctl@ceph.mds.l.smithi173.stdout:May 27 07:29:14 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-l[73180]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:29:14.610 INFO:journalctl@ceph.mds.l.smithi173.stdout:May 27 07:29:14 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-l[73180]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x127) [0x7fee40043ba6] 2024-05-27T07:29:14.610 INFO:journalctl@ceph.mds.l.smithi173.stdout:May 27 07:29:14 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-l[73180]: 2: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7fee40043dd1] 2024-05-27T07:29:14.610 INFO:journalctl@ceph.mds.l.smithi173.stdout:May 27 07:29:14 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-l[73180]: 3: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x561ba1ed323d] 2024-05-27T07:29:14.610 INFO:journalctl@ceph.mds.l.smithi173.stdout:May 27 07:29:14 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-l[73180]: 4: (SnapRealm::build_snap_set() const+0x323) [0x561ba1ebe971] 2024-05-27T07:29:14.610 INFO:journalctl@ceph.mds.l.smithi173.stdout:May 27 07:29:14 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-l[73180]: 5: (SnapRealm::check_cache() const+0x13e) [0x561ba1ebcfa0] 2024-05-27T07:29:14.610 INFO:journalctl@ceph.mds.l.smithi173.stdout:May 27 07:29:14 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-l[73180]: 6: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x561ba1ebd651] 2024-05-27T07:29:14.610 INFO:journalctl@ceph.mds.l.smithi173.stdout:May 27 07:29:14 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-l[73180]: 7: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x561ba1e52347] 2024-05-27T07:29:14.610 INFO:journalctl@ceph.mds.l.smithi173.stdout:May 27 07:29:14 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-l[73180]: 8: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x561ba1b4e87e] 2024-05-27T07:29:14.610 INFO:journalctl@ceph.mds.l.smithi173.stdout:May 27 07:29:14 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-l[73180]: 9: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x2702) [0x561ba1b53e94] 2024-05-27T07:29:14.610 INFO:journalctl@ceph.mds.l.smithi173.stdout:May 27 07:29:14 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-l[73180]: 10: (EUpdate::replay(MDSRank*)+0x63) [0x561ba1b5e969] 2024-05-27T07:29:14.610 INFO:journalctl@ceph.mds.l.smithi173.stdout:May 27 07:29:14 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-l[73180]: 11: (MDLog::_replay_thread()+0x1f38) [0x561ba1ef748c] 2024-05-27T07:29:14.610 INFO:journalctl@ceph.mds.l.smithi173.stdout:May 27 07:29:14 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-l[73180]: 12: (MDLog::ReplayThread::entry()+0x11) [0x561ba1f009cb] 2024-05-27T07:29:14.611 INFO:journalctl@ceph.mds.l.smithi173.stdout:May 27 07:29:14 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-l[73180]: 13: (Thread::entry_wrapper()+0x43) [0x7fee4001272d] 2024-05-27T07:29:14.611 INFO:journalctl@ceph.mds.l.smithi173.stdout:May 27 07:29:14 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-l[73180]: 14: (Thread::_entry_func(void*)+0xd) [0x7fee40012749] 2024-05-27T07:29:14.611 INFO:journalctl@ceph.mds.l.smithi173.stdout:May 27 07:29:14 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-l[73180]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7fee3ebb61ca] 2024-05-27T07:29:14.611 INFO:journalctl@ceph.mds.l.smithi173.stdout:May 27 07:29:14 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-l[73180]: 16: clone() 2024-05-27T07:29:14.611 INFO:journalctl@ceph.mds.l.smithi173.stdout:May 27 07:29:14 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-l[73180]: 2024-05-27T07:29:14.611 INFO:journalctl@ceph.mds.l.smithi173.stdout:May 27 07:29:14 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-l[73180]: 0> 2024-05-27T07:29:14.309+0000 7fee2d7fe700 -1 *** Caught signal (Aborted) ** 2024-05-27T07:29:14.611 INFO:journalctl@ceph.mds.l.smithi173.stdout:May 27 07:29:14 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-l[73180]: in thread 7fee2d7fe700 thread_name:md_log_replay 2024-05-27T07:29:14.611 INFO:journalctl@ceph.mds.l.smithi173.stdout:May 27 07:29:14 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-l[73180]: 2024-05-27T07:29:14.611 INFO:journalctl@ceph.mds.l.smithi173.stdout:May 27 07:29:14 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-l[73180]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:29:14.611 INFO:journalctl@ceph.mds.l.smithi173.stdout:May 27 07:29:14 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-l[73180]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7fee3ebc0d20] 2024-05-27T07:29:14.611 INFO:journalctl@ceph.mds.l.smithi173.stdout:May 27 07:29:14 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-l[73180]: 2: gsignal() 2024-05-27T07:29:14.611 INFO:journalctl@ceph.mds.l.smithi173.stdout:May 27 07:29:14 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-l[73180]: 3: abort() 2024-05-27T07:29:14.611 INFO:journalctl@ceph.mds.l.smithi173.stdout:May 27 07:29:14 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-l[73180]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x26e) [0x7fee40043ced] 2024-05-27T07:29:14.611 INFO:journalctl@ceph.mds.l.smithi173.stdout:May 27 07:29:14 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-l[73180]: 5: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7fee40043dd1] 2024-05-27T07:29:14.611 INFO:journalctl@ceph.mds.l.smithi173.stdout:May 27 07:29:14 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-l[73180]: 6: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x561ba1ed323d] 2024-05-27T07:29:14.612 INFO:journalctl@ceph.mds.l.smithi173.stdout:May 27 07:29:14 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-l[73180]: 7: (SnapRealm::build_snap_set() const+0x323) [0x561ba1ebe971] 2024-05-27T07:29:14.612 INFO:journalctl@ceph.mds.l.smithi173.stdout:May 27 07:29:14 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-l[73180]: 8: (SnapRealm::check_cache() const+0x13e) [0x561ba1ebcfa0] 2024-05-27T07:29:14.612 INFO:journalctl@ceph.mds.l.smithi173.stdout:May 27 07:29:14 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-l[73180]: 9: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x561ba1ebd651] 2024-05-27T07:29:14.612 INFO:journalctl@ceph.mds.l.smithi173.stdout:May 27 07:29:14 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-l[73180]: 10: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x561ba1e52347] 2024-05-27T07:29:14.612 INFO:journalctl@ceph.mds.l.smithi173.stdout:May 27 07:29:14 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-l[73180]: 11: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x561ba1b4e87e] 2024-05-27T07:29:14.612 INFO:journalctl@ceph.mds.l.smithi173.stdout:May 27 07:29:14 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-l[73180]: 12: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x2702) [0x561ba1b53e94] 2024-05-27T07:29:14.612 INFO:journalctl@ceph.mds.l.smithi173.stdout:May 27 07:29:14 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-l[73180]: 13: (EUpdate::replay(MDSRank*)+0x63) [0x561ba1b5e969] 2024-05-27T07:29:14.612 INFO:journalctl@ceph.mds.l.smithi173.stdout:May 27 07:29:14 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-l[73180]: 14: (MDLog::_replay_thread()+0x1f38) [0x561ba1ef748c] 2024-05-27T07:29:14.612 INFO:journalctl@ceph.mds.l.smithi173.stdout:May 27 07:29:14 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-l[73180]: 15: (MDLog::ReplayThread::entry()+0x11) [0x561ba1f009cb] 2024-05-27T07:29:14.612 INFO:journalctl@ceph.mds.l.smithi173.stdout:May 27 07:29:14 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-l[73180]: 16: (Thread::entry_wrapper()+0x43) [0x7fee4001272d] 2024-05-27T07:29:14.613 INFO:journalctl@ceph.mds.l.smithi173.stdout:May 27 07:29:14 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-l[73180]: 17: (Thread::_entry_func(void*)+0xd) [0x7fee40012749] 2024-05-27T07:29:14.613 INFO:journalctl@ceph.mds.l.smithi173.stdout:May 27 07:29:14 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-l[73180]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7fee3ebb61ca] 2024-05-27T07:29:14.613 INFO:journalctl@ceph.mds.l.smithi173.stdout:May 27 07:29:14 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-l[73180]: 19: clone() 2024-05-27T07:29:14.613 INFO:journalctl@ceph.mds.l.smithi173.stdout:May 27 07:29:14 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-l[73180]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-05-27T07:29:14.613 INFO:journalctl@ceph.mds.l.smithi173.stdout:May 27 07:29:14 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-l[73180]: 2024-05-27T07:29:14.613 INFO:journalctl@ceph.mds.l.smithi173.stdout:May 27 07:29:14 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-l[73180]: -1> 2024-05-27T07:29:14.306+0000 7fee2d7fe700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: In function 'std::set SnapClient::filter(const std::set&) const' thread 7fee2d7fe700 time 2024-05-27T07:29:14.305580+0000 2024-05-27T07:29:14.613 INFO:journalctl@ceph.mds.l.smithi173.stdout:May 27 07:29:14 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-l[73180]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: 197: FAILED ceph_assert(cached_version > 0) 2024-05-27T07:29:14.613 INFO:journalctl@ceph.mds.l.smithi173.stdout:May 27 07:29:14 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-l[73180]: 2024-05-27T07:29:14.613 INFO:journalctl@ceph.mds.l.smithi173.stdout:May 27 07:29:14 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-l[73180]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:29:14.613 INFO:journalctl@ceph.mds.l.smithi173.stdout:May 27 07:29:14 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-l[73180]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x127) [0x7fee40043ba6] 2024-05-27T07:29:14.613 INFO:journalctl@ceph.mds.l.smithi173.stdout:May 27 07:29:14 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-l[73180]: 2: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7fee40043dd1] 2024-05-27T07:29:14.614 INFO:journalctl@ceph.mds.l.smithi173.stdout:May 27 07:29:14 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-l[73180]: 3: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x561ba1ed323d] 2024-05-27T07:29:14.614 INFO:journalctl@ceph.mds.l.smithi173.stdout:May 27 07:29:14 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-l[73180]: 4: (SnapRealm::build_snap_set() const+0x323) [0x561ba1ebe971] 2024-05-27T07:29:14.614 INFO:journalctl@ceph.mds.l.smithi173.stdout:May 27 07:29:14 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-l[73180]: 5: (SnapRealm::check_cache() const+0x13e) [0x561ba1ebcfa0] 2024-05-27T07:29:14.614 INFO:journalctl@ceph.mds.l.smithi173.stdout:May 27 07:29:14 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-l[73180]: 6: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x561ba1ebd651] 2024-05-27T07:29:14.614 INFO:journalctl@ceph.mds.l.smithi173.stdout:May 27 07:29:14 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-l[73180]: 7: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x561ba1e52347] 2024-05-27T07:29:14.614 INFO:journalctl@ceph.mds.l.smithi173.stdout:May 27 07:29:14 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-l[73180]: 8: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x561ba1b4e87e] 2024-05-27T07:29:14.614 INFO:journalctl@ceph.mds.l.smithi173.stdout:May 27 07:29:14 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-l[73180]: 9: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x2702) [0x561ba1b53e94] 2024-05-27T07:29:14.614 INFO:journalctl@ceph.mds.l.smithi173.stdout:May 27 07:29:14 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-l[73180]: 10: (EUpdate::replay(MDSRank*)+0x63) [0x561ba1b5e969] 2024-05-27T07:29:14.614 INFO:journalctl@ceph.mds.l.smithi173.stdout:May 27 07:29:14 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-l[73180]: 11: (MDLog::_replay_thread()+0x1f38) [0x561ba1ef748c] 2024-05-27T07:29:14.614 INFO:journalctl@ceph.mds.l.smithi173.stdout:May 27 07:29:14 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-l[73180]: 12: (MDLog::ReplayThread::entry()+0x11) [0x561ba1f009cb] 2024-05-27T07:29:14.614 INFO:journalctl@ceph.mds.l.smithi173.stdout:May 27 07:29:14 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-l[73180]: 13: (Thread::entry_wrapper()+0x43) [0x7fee4001272d] 2024-05-27T07:29:14.614 INFO:journalctl@ceph.mds.l.smithi173.stdout:May 27 07:29:14 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-l[73180]: 14: (Thread::_entry_func(void*)+0xd) [0x7fee40012749] 2024-05-27T07:29:14.615 INFO:journalctl@ceph.mds.l.smithi173.stdout:May 27 07:29:14 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-l[73180]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7fee3ebb61ca] 2024-05-27T07:29:14.615 INFO:journalctl@ceph.mds.l.smithi173.stdout:May 27 07:29:14 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-l[73180]: 16: clone() 2024-05-27T07:29:14.615 INFO:journalctl@ceph.mds.l.smithi173.stdout:May 27 07:29:14 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-l[73180]: 2024-05-27T07:29:14.615 INFO:journalctl@ceph.mds.l.smithi173.stdout:May 27 07:29:14 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-l[73180]: 0> 2024-05-27T07:29:14.309+0000 7fee2d7fe700 -1 *** Caught signal (Aborted) ** 2024-05-27T07:29:14.615 INFO:journalctl@ceph.mds.l.smithi173.stdout:May 27 07:29:14 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-l[73180]: in thread 7fee2d7fe700 thread_name:md_log_replay 2024-05-27T07:29:14.615 INFO:journalctl@ceph.mds.l.smithi173.stdout:May 27 07:29:14 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-l[73180]: 2024-05-27T07:29:14.615 INFO:journalctl@ceph.mds.l.smithi173.stdout:May 27 07:29:14 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-l[73180]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:29:14.615 INFO:journalctl@ceph.mds.l.smithi173.stdout:May 27 07:29:14 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-l[73180]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7fee3ebc0d20] 2024-05-27T07:29:14.615 INFO:journalctl@ceph.mds.l.smithi173.stdout:May 27 07:29:14 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-l[73180]: 2: gsignal() 2024-05-27T07:29:14.615 INFO:journalctl@ceph.mds.l.smithi173.stdout:May 27 07:29:14 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-l[73180]: 3: abort() 2024-05-27T07:29:14.615 INFO:journalctl@ceph.mds.l.smithi173.stdout:May 27 07:29:14 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-l[73180]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x26e) [0x7fee40043ced] 2024-05-27T07:29:14.615 INFO:journalctl@ceph.mds.l.smithi173.stdout:May 27 07:29:14 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-l[73180]: 5: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7fee40043dd1] 2024-05-27T07:29:14.615 INFO:journalctl@ceph.mds.l.smithi173.stdout:May 27 07:29:14 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-l[73180]: 6: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x561ba1ed323d] 2024-05-27T07:29:14.615 INFO:journalctl@ceph.mds.l.smithi173.stdout:May 27 07:29:14 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-l[73180]: 7: (SnapRealm::build_snap_set() const+0x323) [0x561ba1ebe971] 2024-05-27T07:29:14.615 INFO:journalctl@ceph.mds.l.smithi173.stdout:May 27 07:29:14 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-l[73180]: 8: (SnapRealm::check_cache() const+0x13e) [0x561ba1ebcfa0] 2024-05-27T07:29:14.616 INFO:journalctl@ceph.mds.l.smithi173.stdout:May 27 07:29:14 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-l[73180]: 9: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x561ba1ebd651] 2024-05-27T07:29:14.616 INFO:journalctl@ceph.mds.l.smithi173.stdout:May 27 07:29:14 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-l[73180]: 10: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x561ba1e52347] 2024-05-27T07:29:14.616 INFO:journalctl@ceph.mds.l.smithi173.stdout:May 27 07:29:14 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-l[73180]: 11: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x561ba1b4e87e] 2024-05-27T07:29:14.616 INFO:journalctl@ceph.mds.l.smithi173.stdout:May 27 07:29:14 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-l[73180]: 12: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x2702) [0x561ba1b53e94] 2024-05-27T07:29:14.616 INFO:journalctl@ceph.mds.l.smithi173.stdout:May 27 07:29:14 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-l[73180]: 13: (EUpdate::replay(MDSRank*)+0x63) [0x561ba1b5e969] 2024-05-27T07:29:14.616 INFO:journalctl@ceph.mds.l.smithi173.stdout:May 27 07:29:14 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-l[73180]: 14: (MDLog::_replay_thread()+0x1f38) [0x561ba1ef748c] 2024-05-27T07:29:14.616 INFO:journalctl@ceph.mds.l.smithi173.stdout:May 27 07:29:14 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-l[73180]: 15: (MDLog::ReplayThread::entry()+0x11) [0x561ba1f009cb] 2024-05-27T07:29:14.616 INFO:journalctl@ceph.mds.l.smithi173.stdout:May 27 07:29:14 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-l[73180]: 16: (Thread::entry_wrapper()+0x43) [0x7fee4001272d] 2024-05-27T07:29:14.616 INFO:journalctl@ceph.mds.l.smithi173.stdout:May 27 07:29:14 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-l[73180]: 17: (Thread::_entry_func(void*)+0xd) [0x7fee40012749] 2024-05-27T07:29:14.616 INFO:journalctl@ceph.mds.l.smithi173.stdout:May 27 07:29:14 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-l[73180]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7fee3ebb61ca] 2024-05-27T07:29:14.616 INFO:journalctl@ceph.mds.l.smithi173.stdout:May 27 07:29:14 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-l[73180]: 19: clone() 2024-05-27T07:29:14.616 INFO:journalctl@ceph.mds.l.smithi173.stdout:May 27 07:29:14 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-l[73180]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-05-27T07:29:14.616 INFO:journalctl@ceph.mds.l.smithi173.stdout:May 27 07:29:14 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-l[73180]: 2024-05-27T07:29:14.616 INFO:journalctl@ceph.mds.l.smithi173.stdout:May 27 07:29:14 smithi173 podman[73215]: 2024-05-27 07:29:14.381127817 +0000 UTC m=+0.027853881 container died 8b5d2550a74e1afcc42419f2b5079ccf25e4d0aa8ae654fc35c85b6f7e325d3d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-l, org.label-schema.vendor=CentOS, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, org.label-schema.build-date=20240521, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , GIT_CLEAN=True, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.license=GPLv2) 2024-05-27T07:29:14.689 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:29:14 smithi105 ceph-mon[35977]: pgmap v109: 129 pgs: 129 active+clean; 182 MiB data, 957 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 2.3 MiB/s wr, 836 op/s 2024-05-27T07:29:14.690 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:29:14 smithi105 ceph-mon[35977]: daemon mds.l restarted 2024-05-27T07:29:15.205 INFO:tasks.workunit.client.0.smithi052.stdout: 50 367481 261 257754 121 217795 718 2024-05-27T07:29:15.298 INFO:journalctl@ceph.mds.l.smithi173.stdout:May 27 07:29:15 smithi173 podman[73215]: 2024-05-27 07:29:15.046587795 +0000 UTC m=+0.693313845 container remove 8b5d2550a74e1afcc42419f2b5079ccf25e4d0aa8ae654fc35c85b6f7e325d3d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-l, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, org.label-schema.build-date=20240521, ceph=True) 2024-05-27T07:29:15.298 INFO:journalctl@ceph.mds.l.smithi173.stdout:May 27 07:29:15 smithi173 systemd[1]: ceph-73df2636-1bf8-11ef-bc9b-c7b262605968@mds.l.service: Main process exited, code=exited, status=134/n/a 2024-05-27T07:29:15.299 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:29:15 smithi173 ceph-mon[37000]: osdmap e100: 12 total, 12 up, 12 in 2024-05-27T07:29:15.299 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:29:15 smithi173 ceph-mon[37000]: mds.? [v2:172.21.15.173:6832/2348995049,v1:172.21.15.173:6833/2348995049] up:boot 2024-05-27T07:29:15.299 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:29:15 smithi173 ceph-mon[37000]: fsmap cephfs:5 4 up:standby-replay 3 up:standby 5 up:active 2024-05-27T07:29:15.299 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:29:15 smithi173 ceph-mon[37000]: from='mgr.15333 172.21.15.52:0/416389082' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "l"}]: dispatch 2024-05-27T07:29:15.299 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:29:15 smithi173 ceph-mon[37000]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-05-27T07:29:15.299 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:29:15 smithi173 ceph-mon[37000]: from='mgr.15333 172.21.15.52:0/416389082' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-27T07:29:15.480 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:29:15 smithi052 ceph-mon[31955]: osdmap e100: 12 total, 12 up, 12 in 2024-05-27T07:29:15.480 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:29:15 smithi052 ceph-mon[31955]: mds.? [v2:172.21.15.173:6832/2348995049,v1:172.21.15.173:6833/2348995049] up:boot 2024-05-27T07:29:15.480 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:29:15 smithi052 ceph-mon[31955]: fsmap cephfs:5 4 up:standby-replay 3 up:standby 5 up:active 2024-05-27T07:29:15.480 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:29:15 smithi052 ceph-mon[31955]: from='mgr.15333 172.21.15.52:0/416389082' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "l"}]: dispatch 2024-05-27T07:29:15.480 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:29:15 smithi052 ceph-mon[31955]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-05-27T07:29:15.481 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:29:15 smithi052 ceph-mon[31955]: from='mgr.15333 172.21.15.52:0/416389082' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-27T07:29:15.602 INFO:journalctl@ceph.mds.l.smithi173.stdout:May 27 07:29:15 smithi173 systemd[1]: ceph-73df2636-1bf8-11ef-bc9b-c7b262605968@mds.l.service: Failed with result 'exit-code'. 2024-05-27T07:29:15.690 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:29:15 smithi105 ceph-mon[35977]: osdmap e100: 12 total, 12 up, 12 in 2024-05-27T07:29:15.690 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:29:15 smithi105 ceph-mon[35977]: mds.? [v2:172.21.15.173:6832/2348995049,v1:172.21.15.173:6833/2348995049] up:boot 2024-05-27T07:29:15.690 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:29:15 smithi105 ceph-mon[35977]: fsmap cephfs:5 4 up:standby-replay 3 up:standby 5 up:active 2024-05-27T07:29:15.690 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:29:15 smithi105 ceph-mon[35977]: from='mgr.15333 172.21.15.52:0/416389082' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "l"}]: dispatch 2024-05-27T07:29:15.690 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:29:15 smithi105 ceph-mon[35977]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-05-27T07:29:15.690 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:29:15 smithi105 ceph-mon[35977]: from='mgr.15333 172.21.15.52:0/416389082' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-27T07:29:15.690 INFO:journalctl@ceph.mds.k.smithi105.stdout:May 27 07:29:15 smithi105 systemd[1]: ceph-73df2636-1bf8-11ef-bc9b-c7b262605968@mds.k.service: Scheduled restart job, restart counter is at 1. 2024-05-27T07:29:15.691 INFO:journalctl@ceph.mds.k.smithi105.stdout:May 27 07:29:15 smithi105 systemd[1]: Stopped Ceph mds.k for 73df2636-1bf8-11ef-bc9b-c7b262605968. 2024-05-27T07:29:15.691 INFO:journalctl@ceph.mds.k.smithi105.stdout:May 27 07:29:15 smithi105 systemd[1]: ceph-73df2636-1bf8-11ef-bc9b-c7b262605968@mds.k.service: Consumed 1.428s CPU time. 2024-05-27T07:29:15.691 INFO:journalctl@ceph.mds.k.smithi105.stdout:May 27 07:29:15 smithi105 systemd[1]: Starting Ceph mds.k for 73df2636-1bf8-11ef-bc9b-c7b262605968... 2024-05-27T07:29:15.691 INFO:journalctl@ceph.mds.k.smithi105.stdout:May 27 07:29:15 smithi105 podman[69637]: 2024-05-27 07:29:15.351254472 +0000 UTC m=+0.019860959 image pull e36ea4fc8a5807e8d2701370e350a0de3856538797f7fba64cfce7e344bd6799 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6 2024-05-27T07:29:16.001 INFO:journalctl@ceph.mds.d.smithi052.stdout:May 27 07:29:15 smithi052 systemd[1]: ceph-73df2636-1bf8-11ef-bc9b-c7b262605968@mds.d.service: Scheduled restart job, restart counter is at 2. 2024-05-27T07:29:16.002 INFO:journalctl@ceph.mds.d.smithi052.stdout:May 27 07:29:15 smithi052 systemd[1]: Stopped Ceph mds.d for 73df2636-1bf8-11ef-bc9b-c7b262605968. 2024-05-27T07:29:16.002 INFO:journalctl@ceph.mds.d.smithi052.stdout:May 27 07:29:15 smithi052 systemd[1]: Starting Ceph mds.d for 73df2636-1bf8-11ef-bc9b-c7b262605968... 2024-05-27T07:29:16.359 INFO:journalctl@ceph.mds.d.smithi052.stdout:May 27 07:29:16 smithi052 podman[104449]: 2024-05-27 07:29:15.915410859 +0000 UTC m=+0.028036387 image pull e36ea4fc8a5807e8d2701370e350a0de3856538797f7fba64cfce7e344bd6799 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6 2024-05-27T07:29:16.359 INFO:journalctl@ceph.mds.d.smithi052.stdout:May 27 07:29:16 smithi052 podman[104449]: 2024-05-27 07:29:16.041188099 +0000 UTC m=+0.153813629 container create 1440a1635f6e4feeff38e9cb90a021107e7444f5c84852004669f8f823b1af19 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-d, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240521, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, GIT_CLEAN=True, org.label-schema.schema-version=1.0, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90) 2024-05-27T07:29:16.359 INFO:journalctl@ceph.mds.d.smithi052.stdout:May 27 07:29:16 smithi052 podman[104449]: 2024-05-27 07:29:16.36954765 +0000 UTC m=+0.482173189 container init 1440a1635f6e4feeff38e9cb90a021107e7444f5c84852004669f8f823b1af19 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-d, org.label-schema.build-date=20240521, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, ceph=True, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-05-27T07:29:16.359 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:29:16 smithi052 ceph-mon[31955]: scrub summary: idle+waiting paths [/] 2024-05-27T07:29:16.360 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:29:16 smithi052 ceph-mon[31955]: pgmap v111: 129 pgs: 129 active+clean; 213 MiB data, 1016 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 MiB/s rd, 7.4 MiB/s wr, 1.96k op/s 2024-05-27T07:29:16.360 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:29:16 smithi052 ceph-mon[31955]: from='mgr.15333 172.21.15.52:0/416389082' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T07:29:16.360 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:29:16 smithi052 ceph-mon[31955]: from='mgr.15333 172.21.15.52:0/416389082' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-27T07:29:16.360 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:29:16 smithi052 ceph-mon[31955]: from='mgr.15333 172.21.15.52:0/416389082' entity='mgr.x' 2024-05-27T07:29:16.503 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:29:16 smithi173 ceph-mon[37000]: scrub summary: idle+waiting paths [/] 2024-05-27T07:29:16.503 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:29:16 smithi173 ceph-mon[37000]: pgmap v111: 129 pgs: 129 active+clean; 213 MiB data, 1016 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 MiB/s rd, 7.4 MiB/s wr, 1.96k op/s 2024-05-27T07:29:16.503 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:29:16 smithi173 ceph-mon[37000]: from='mgr.15333 172.21.15.52:0/416389082' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T07:29:16.503 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:29:16 smithi173 ceph-mon[37000]: from='mgr.15333 172.21.15.52:0/416389082' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-27T07:29:16.503 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:29:16 smithi173 ceph-mon[37000]: from='mgr.15333 172.21.15.52:0/416389082' entity='mgr.x' 2024-05-27T07:29:16.591 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:29:16 smithi105 ceph-mon[35977]: scrub summary: idle+waiting paths [/] 2024-05-27T07:29:16.591 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:29:16 smithi105 ceph-mon[35977]: pgmap v111: 129 pgs: 129 active+clean; 213 MiB data, 1016 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 MiB/s rd, 7.4 MiB/s wr, 1.96k op/s 2024-05-27T07:29:16.591 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:29:16 smithi105 ceph-mon[35977]: from='mgr.15333 172.21.15.52:0/416389082' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T07:29:16.591 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:29:16 smithi105 ceph-mon[35977]: from='mgr.15333 172.21.15.52:0/416389082' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-27T07:29:16.591 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:29:16 smithi105 ceph-mon[35977]: from='mgr.15333 172.21.15.52:0/416389082' entity='mgr.x' 2024-05-27T07:29:16.729 INFO:journalctl@ceph.mds.d.smithi052.stdout:May 27 07:29:16 smithi052 podman[104449]: 2024-05-27 07:29:16.375040647 +0000 UTC m=+0.487666178 container start 1440a1635f6e4feeff38e9cb90a021107e7444f5c84852004669f8f823b1af19 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-d, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, org.label-schema.build-date=20240521, GIT_BRANCH=HEAD, ceph=True, GIT_CLEAN=True) 2024-05-27T07:29:16.730 INFO:journalctl@ceph.mds.d.smithi052.stdout:May 27 07:29:16 smithi052 ceph-mds[104516]: -- 172.21.15.52:0/1141988904 <== mon.0 v2:172.21.15.52:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x558fcc515a40 con 0x558fcc55d600 2024-05-27T07:29:16.730 INFO:journalctl@ceph.mds.d.smithi052.stdout:May 27 07:29:16 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-d[104512]: starting mds.d at 2024-05-27T07:29:16.730 INFO:journalctl@ceph.mds.d.smithi052.stdout:May 27 07:29:16 smithi052 bash[104449]: 1440a1635f6e4feeff38e9cb90a021107e7444f5c84852004669f8f823b1af19 2024-05-27T07:29:16.730 INFO:journalctl@ceph.mds.d.smithi052.stdout:May 27 07:29:16 smithi052 systemd[1]: Started Ceph mds.d for 73df2636-1bf8-11ef-bc9b-c7b262605968. 2024-05-27T07:29:16.940 INFO:journalctl@ceph.mds.k.smithi105.stdout:May 27 07:29:16 smithi105 podman[69637]: 2024-05-27 07:29:16.64769566 +0000 UTC m=+1.316302143 container create 63d34243be402599f9f82855c72ddf43c4fc940256782a2b543ffdcdb8f8c5f6 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-k, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.license=GPLv2, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240521, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0) 2024-05-27T07:29:17.501 INFO:journalctl@ceph.mds.a.smithi052.stdout:May 27 07:29:17 smithi052 systemd[1]: ceph-73df2636-1bf8-11ef-bc9b-c7b262605968@mds.a.service: Scheduled restart job, restart counter is at 1. 2024-05-27T07:29:17.501 INFO:journalctl@ceph.mds.a.smithi052.stdout:May 27 07:29:17 smithi052 systemd[1]: Stopped Ceph mds.a for 73df2636-1bf8-11ef-bc9b-c7b262605968. 2024-05-27T07:29:17.501 INFO:journalctl@ceph.mds.a.smithi052.stdout:May 27 07:29:17 smithi052 systemd[1]: ceph-73df2636-1bf8-11ef-bc9b-c7b262605968@mds.a.service: Consumed 1.517s CPU time. 2024-05-27T07:29:17.501 INFO:journalctl@ceph.mds.a.smithi052.stdout:May 27 07:29:17 smithi052 systemd[1]: Starting Ceph mds.a for 73df2636-1bf8-11ef-bc9b-c7b262605968... 2024-05-27T07:29:17.509 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:29:17 smithi052 ceph-mon[31955]: daemon mds.d restarted 2024-05-27T07:29:17.509 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:29:17 smithi052 ceph-mon[31955]: scrub summary: idle 2024-05-27T07:29:17.509 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:29:17 smithi052 ceph-mon[31955]: scrub summary: idle 2024-05-27T07:29:17.509 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:29:17 smithi052 ceph-mon[31955]: scrub summary: idle 2024-05-27T07:29:17.509 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:29:17 smithi052 ceph-mon[31955]: scrub summary: idle 2024-05-27T07:29:17.602 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:29:17 smithi173 ceph-mon[37000]: daemon mds.d restarted 2024-05-27T07:29:17.602 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:29:17 smithi173 ceph-mon[37000]: scrub summary: idle 2024-05-27T07:29:17.602 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:29:17 smithi173 ceph-mon[37000]: scrub summary: idle 2024-05-27T07:29:17.602 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:29:17 smithi173 ceph-mon[37000]: scrub summary: idle 2024-05-27T07:29:17.602 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:29:17 smithi173 ceph-mon[37000]: scrub summary: idle 2024-05-27T07:29:17.690 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:29:17 smithi105 ceph-mon[35977]: daemon mds.d restarted 2024-05-27T07:29:17.690 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:29:17 smithi105 ceph-mon[35977]: scrub summary: idle 2024-05-27T07:29:17.690 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:29:17 smithi105 ceph-mon[35977]: scrub summary: idle 2024-05-27T07:29:17.690 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:29:17 smithi105 ceph-mon[35977]: scrub summary: idle 2024-05-27T07:29:17.690 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:29:17 smithi105 ceph-mon[35977]: scrub summary: idle 2024-05-27T07:29:17.835 INFO:journalctl@ceph.mds.g.smithi052.stdout:May 27 07:29:17 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-g[103978]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: In function 'std::set SnapClient::filter(const std::set&) const' thread 7f98c3e25700 time 2024-05-27T07:29:17.528719+0000 2024-05-27T07:29:17.835 INFO:journalctl@ceph.mds.g.smithi052.stdout:May 27 07:29:17 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-g[103978]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: 197: FAILED ceph_assert(cached_version > 0) 2024-05-27T07:29:17.835 INFO:journalctl@ceph.mds.g.smithi052.stdout:May 27 07:29:17 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-g[103978]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:29:17.835 INFO:journalctl@ceph.mds.g.smithi052.stdout:May 27 07:29:17 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-g[103978]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x127) [0x7f98d666aba6] 2024-05-27T07:29:17.835 INFO:journalctl@ceph.mds.g.smithi052.stdout:May 27 07:29:17 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-g[103978]: 2: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7f98d666add1] 2024-05-27T07:29:17.835 INFO:journalctl@ceph.mds.g.smithi052.stdout:May 27 07:29:17 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-g[103978]: 3: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x55efb558123d] 2024-05-27T07:29:17.836 INFO:journalctl@ceph.mds.g.smithi052.stdout:May 27 07:29:17 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-g[103978]: 4: (SnapRealm::build_snap_set() const+0x323) [0x55efb556c971] 2024-05-27T07:29:17.836 INFO:journalctl@ceph.mds.g.smithi052.stdout:May 27 07:29:17 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-g[103978]: 5: (SnapRealm::check_cache() const+0x13e) [0x55efb556afa0] 2024-05-27T07:29:17.836 INFO:journalctl@ceph.mds.g.smithi052.stdout:May 27 07:29:17 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-g[103978]: 6: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x55efb556b651] 2024-05-27T07:29:17.836 INFO:journalctl@ceph.mds.g.smithi052.stdout:May 27 07:29:17 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-g[103978]: 7: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x55efb5500347] 2024-05-27T07:29:17.836 INFO:journalctl@ceph.mds.g.smithi052.stdout:May 27 07:29:17 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-g[103978]: 8: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x55efb51fc87e] 2024-05-27T07:29:17.836 INFO:journalctl@ceph.mds.g.smithi052.stdout:May 27 07:29:17 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-g[103978]: 9: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x2702) [0x55efb5201e94] 2024-05-27T07:29:17.836 INFO:journalctl@ceph.mds.g.smithi052.stdout:May 27 07:29:17 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-g[103978]: 10: (EUpdate::replay(MDSRank*)+0x63) [0x55efb520c969] 2024-05-27T07:29:17.836 INFO:journalctl@ceph.mds.g.smithi052.stdout:May 27 07:29:17 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-g[103978]: 11: (MDLog::_replay_thread()+0x1f38) [0x55efb55a548c] 2024-05-27T07:29:17.836 INFO:journalctl@ceph.mds.g.smithi052.stdout:May 27 07:29:17 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-g[103978]: 12: (MDLog::ReplayThread::entry()+0x11) [0x55efb55ae9cb] 2024-05-27T07:29:17.836 INFO:journalctl@ceph.mds.g.smithi052.stdout:May 27 07:29:17 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-g[103978]: 13: (Thread::entry_wrapper()+0x43) [0x7f98d663972d] 2024-05-27T07:29:17.837 INFO:journalctl@ceph.mds.g.smithi052.stdout:May 27 07:29:17 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-g[103978]: 14: (Thread::_entry_func(void*)+0xd) [0x7f98d6639749] 2024-05-27T07:29:17.837 INFO:journalctl@ceph.mds.g.smithi052.stdout:May 27 07:29:17 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-g[103978]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7f98d51dd1ca] 2024-05-27T07:29:17.837 INFO:journalctl@ceph.mds.g.smithi052.stdout:May 27 07:29:17 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-g[103978]: 16: clone() 2024-05-27T07:29:17.837 INFO:journalctl@ceph.mds.g.smithi052.stdout:May 27 07:29:17 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-g[103978]: 2024-05-27T07:29:17.531+0000 7f98c3e25700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: In function 'std::set SnapClient::filter(const std::set&) const' thread 7f98c3e25700 time 2024-05-27T07:29:17.528719+0000 2024-05-27T07:29:17.837 INFO:journalctl@ceph.mds.g.smithi052.stdout:May 27 07:29:17 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-g[103978]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: 197: FAILED ceph_assert(cached_version > 0) 2024-05-27T07:29:17.837 INFO:journalctl@ceph.mds.g.smithi052.stdout:May 27 07:29:17 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-g[103978]: 2024-05-27T07:29:17.837 INFO:journalctl@ceph.mds.g.smithi052.stdout:May 27 07:29:17 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-g[103978]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:29:17.837 INFO:journalctl@ceph.mds.g.smithi052.stdout:May 27 07:29:17 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-g[103978]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x127) [0x7f98d666aba6] 2024-05-27T07:29:17.837 INFO:journalctl@ceph.mds.g.smithi052.stdout:May 27 07:29:17 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-g[103978]: 2: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7f98d666add1] 2024-05-27T07:29:17.837 INFO:journalctl@ceph.mds.g.smithi052.stdout:May 27 07:29:17 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-g[103978]: 3: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x55efb558123d] 2024-05-27T07:29:17.837 INFO:journalctl@ceph.mds.g.smithi052.stdout:May 27 07:29:17 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-g[103978]: 4: (SnapRealm::build_snap_set() const+0x323) [0x55efb556c971] 2024-05-27T07:29:17.837 INFO:journalctl@ceph.mds.g.smithi052.stdout:May 27 07:29:17 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-g[103978]: 5: (SnapRealm::check_cache() const+0x13e) [0x55efb556afa0] 2024-05-27T07:29:17.838 INFO:journalctl@ceph.mds.g.smithi052.stdout:May 27 07:29:17 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-g[103978]: 6: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x55efb556b651] 2024-05-27T07:29:17.838 INFO:journalctl@ceph.mds.g.smithi052.stdout:May 27 07:29:17 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-g[103978]: 7: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x55efb5500347] 2024-05-27T07:29:17.838 INFO:journalctl@ceph.mds.g.smithi052.stdout:May 27 07:29:17 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-g[103978]: 8: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x55efb51fc87e] 2024-05-27T07:29:17.838 INFO:journalctl@ceph.mds.g.smithi052.stdout:May 27 07:29:17 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-g[103978]: 9: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x2702) [0x55efb5201e94] 2024-05-27T07:29:17.838 INFO:journalctl@ceph.mds.g.smithi052.stdout:May 27 07:29:17 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-g[103978]: 10: (EUpdate::replay(MDSRank*)+0x63) [0x55efb520c969] 2024-05-27T07:29:17.838 INFO:journalctl@ceph.mds.g.smithi052.stdout:May 27 07:29:17 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-g[103978]: 11: (MDLog::_replay_thread()+0x1f38) [0x55efb55a548c] 2024-05-27T07:29:17.838 INFO:journalctl@ceph.mds.g.smithi052.stdout:May 27 07:29:17 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-g[103978]: 12: (MDLog::ReplayThread::entry()+0x11) [0x55efb55ae9cb] 2024-05-27T07:29:17.838 INFO:journalctl@ceph.mds.g.smithi052.stdout:May 27 07:29:17 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-g[103978]: 13: (Thread::entry_wrapper()+0x43) [0x7f98d663972d] 2024-05-27T07:29:17.838 INFO:journalctl@ceph.mds.g.smithi052.stdout:May 27 07:29:17 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-g[103978]: 14: (Thread::_entry_func(void*)+0xd) [0x7f98d6639749] 2024-05-27T07:29:17.838 INFO:journalctl@ceph.mds.g.smithi052.stdout:May 27 07:29:17 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-g[103978]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7f98d51dd1ca] 2024-05-27T07:29:17.838 INFO:journalctl@ceph.mds.g.smithi052.stdout:May 27 07:29:17 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-g[103978]: 16: clone() 2024-05-27T07:29:17.838 INFO:journalctl@ceph.mds.g.smithi052.stdout:May 27 07:29:17 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-g[103978]: 2024-05-27T07:29:17.838 INFO:journalctl@ceph.mds.g.smithi052.stdout:May 27 07:29:17 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-g[103978]: *** Caught signal (Aborted) ** 2024-05-27T07:29:17.839 INFO:journalctl@ceph.mds.g.smithi052.stdout:May 27 07:29:17 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-g[103978]: in thread 7f98c3e25700 thread_name:md_log_replay 2024-05-27T07:29:17.839 INFO:journalctl@ceph.mds.g.smithi052.stdout:May 27 07:29:17 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-g[103978]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:29:17.839 INFO:journalctl@ceph.mds.g.smithi052.stdout:May 27 07:29:17 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-g[103978]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f98d51e7d20] 2024-05-27T07:29:17.839 INFO:journalctl@ceph.mds.g.smithi052.stdout:May 27 07:29:17 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-g[103978]: 2: gsignal() 2024-05-27T07:29:17.839 INFO:journalctl@ceph.mds.g.smithi052.stdout:May 27 07:29:17 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-g[103978]: 3: abort() 2024-05-27T07:29:17.839 INFO:journalctl@ceph.mds.g.smithi052.stdout:May 27 07:29:17 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-g[103978]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x26e) [0x7f98d666aced] 2024-05-27T07:29:17.839 INFO:journalctl@ceph.mds.g.smithi052.stdout:May 27 07:29:17 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-g[103978]: 5: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7f98d666add1] 2024-05-27T07:29:17.839 INFO:journalctl@ceph.mds.g.smithi052.stdout:May 27 07:29:17 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-g[103978]: 6: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x55efb558123d] 2024-05-27T07:29:17.839 INFO:journalctl@ceph.mds.g.smithi052.stdout:May 27 07:29:17 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-g[103978]: 7: (SnapRealm::build_snap_set() const+0x323) [0x55efb556c971] 2024-05-27T07:29:17.839 INFO:journalctl@ceph.mds.g.smithi052.stdout:May 27 07:29:17 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-g[103978]: 8: (SnapRealm::check_cache() const+0x13e) [0x55efb556afa0] 2024-05-27T07:29:17.839 INFO:journalctl@ceph.mds.g.smithi052.stdout:May 27 07:29:17 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-g[103978]: 9: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x55efb556b651] 2024-05-27T07:29:17.839 INFO:journalctl@ceph.mds.g.smithi052.stdout:May 27 07:29:17 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-g[103978]: 10: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x55efb5500347] 2024-05-27T07:29:17.839 INFO:journalctl@ceph.mds.g.smithi052.stdout:May 27 07:29:17 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-g[103978]: 11: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x55efb51fc87e] 2024-05-27T07:29:17.840 INFO:journalctl@ceph.mds.g.smithi052.stdout:May 27 07:29:17 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-g[103978]: 12: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x2702) [0x55efb5201e94] 2024-05-27T07:29:17.840 INFO:journalctl@ceph.mds.g.smithi052.stdout:May 27 07:29:17 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-g[103978]: 13: (EUpdate::replay(MDSRank*)+0x63) [0x55efb520c969] 2024-05-27T07:29:17.840 INFO:journalctl@ceph.mds.g.smithi052.stdout:May 27 07:29:17 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-g[103978]: 14: (MDLog::_replay_thread()+0x1f38) [0x55efb55a548c] 2024-05-27T07:29:17.840 INFO:journalctl@ceph.mds.g.smithi052.stdout:May 27 07:29:17 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-g[103978]: 15: (MDLog::ReplayThread::entry()+0x11) [0x55efb55ae9cb] 2024-05-27T07:29:17.840 INFO:journalctl@ceph.mds.g.smithi052.stdout:May 27 07:29:17 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-g[103978]: 16: (Thread::entry_wrapper()+0x43) [0x7f98d663972d] 2024-05-27T07:29:17.840 INFO:journalctl@ceph.mds.g.smithi052.stdout:May 27 07:29:17 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-g[103978]: 17: (Thread::_entry_func(void*)+0xd) [0x7f98d6639749] 2024-05-27T07:29:17.840 INFO:journalctl@ceph.mds.g.smithi052.stdout:May 27 07:29:17 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-g[103978]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7f98d51dd1ca] 2024-05-27T07:29:17.840 INFO:journalctl@ceph.mds.g.smithi052.stdout:May 27 07:29:17 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-g[103978]: 19: clone() 2024-05-27T07:29:17.840 INFO:journalctl@ceph.mds.g.smithi052.stdout:May 27 07:29:17 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-g[103978]: 2024-05-27T07:29:17.548+0000 7f98c3e25700 -1 *** Caught signal (Aborted) ** 2024-05-27T07:29:17.840 INFO:journalctl@ceph.mds.g.smithi052.stdout:May 27 07:29:17 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-g[103978]: in thread 7f98c3e25700 thread_name:md_log_replay 2024-05-27T07:29:17.840 INFO:journalctl@ceph.mds.g.smithi052.stdout:May 27 07:29:17 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-g[103978]: 2024-05-27T07:29:17.840 INFO:journalctl@ceph.mds.g.smithi052.stdout:May 27 07:29:17 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-g[103978]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:29:17.840 INFO:journalctl@ceph.mds.g.smithi052.stdout:May 27 07:29:17 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-g[103978]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f98d51e7d20] 2024-05-27T07:29:17.840 INFO:journalctl@ceph.mds.g.smithi052.stdout:May 27 07:29:17 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-g[103978]: 2: gsignal() 2024-05-27T07:29:17.841 INFO:journalctl@ceph.mds.g.smithi052.stdout:May 27 07:29:17 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-g[103978]: 3: abort() 2024-05-27T07:29:17.841 INFO:journalctl@ceph.mds.g.smithi052.stdout:May 27 07:29:17 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-g[103978]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x26e) [0x7f98d666aced] 2024-05-27T07:29:17.841 INFO:journalctl@ceph.mds.g.smithi052.stdout:May 27 07:29:17 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-g[103978]: 5: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7f98d666add1] 2024-05-27T07:29:17.841 INFO:journalctl@ceph.mds.g.smithi052.stdout:May 27 07:29:17 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-g[103978]: 6: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x55efb558123d] 2024-05-27T07:29:17.841 INFO:journalctl@ceph.mds.g.smithi052.stdout:May 27 07:29:17 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-g[103978]: 7: (SnapRealm::build_snap_set() const+0x323) [0x55efb556c971] 2024-05-27T07:29:17.841 INFO:journalctl@ceph.mds.g.smithi052.stdout:May 27 07:29:17 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-g[103978]: 8: (SnapRealm::check_cache() const+0x13e) [0x55efb556afa0] 2024-05-27T07:29:17.841 INFO:journalctl@ceph.mds.g.smithi052.stdout:May 27 07:29:17 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-g[103978]: 9: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x55efb556b651] 2024-05-27T07:29:17.842 INFO:journalctl@ceph.mds.g.smithi052.stdout:May 27 07:29:17 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-g[103978]: 10: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x55efb5500347] 2024-05-27T07:29:17.842 INFO:journalctl@ceph.mds.g.smithi052.stdout:May 27 07:29:17 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-g[103978]: 11: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x55efb51fc87e] 2024-05-27T07:29:17.842 INFO:journalctl@ceph.mds.g.smithi052.stdout:May 27 07:29:17 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-g[103978]: 12: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x2702) [0x55efb5201e94] 2024-05-27T07:29:17.842 INFO:journalctl@ceph.mds.g.smithi052.stdout:May 27 07:29:17 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-g[103978]: 13: (EUpdate::replay(MDSRank*)+0x63) [0x55efb520c969] 2024-05-27T07:29:17.842 INFO:journalctl@ceph.mds.g.smithi052.stdout:May 27 07:29:17 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-g[103978]: 14: (MDLog::_replay_thread()+0x1f38) [0x55efb55a548c] 2024-05-27T07:29:17.842 INFO:journalctl@ceph.mds.g.smithi052.stdout:May 27 07:29:17 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-g[103978]: 15: (MDLog::ReplayThread::entry()+0x11) [0x55efb55ae9cb] 2024-05-27T07:29:17.842 INFO:journalctl@ceph.mds.g.smithi052.stdout:May 27 07:29:17 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-g[103978]: 16: (Thread::entry_wrapper()+0x43) [0x7f98d663972d] 2024-05-27T07:29:17.842 INFO:journalctl@ceph.mds.g.smithi052.stdout:May 27 07:29:17 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-g[103978]: 17: (Thread::_entry_func(void*)+0xd) [0x7f98d6639749] 2024-05-27T07:29:17.842 INFO:journalctl@ceph.mds.g.smithi052.stdout:May 27 07:29:17 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-g[103978]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7f98d51dd1ca] 2024-05-27T07:29:17.843 INFO:journalctl@ceph.mds.g.smithi052.stdout:May 27 07:29:17 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-g[103978]: 19: clone() 2024-05-27T07:29:17.843 INFO:journalctl@ceph.mds.g.smithi052.stdout:May 27 07:29:17 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-g[103978]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-05-27T07:29:17.843 INFO:journalctl@ceph.mds.g.smithi052.stdout:May 27 07:29:17 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-g[103978]: 2024-05-27T07:29:17.843 INFO:journalctl@ceph.mds.g.smithi052.stdout:May 27 07:29:17 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-g[103978]: -1> 2024-05-27T07:29:17.531+0000 7f98c3e25700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: In function 'std::set SnapClient::filter(const std::set&) const' thread 7f98c3e25700 time 2024-05-27T07:29:17.528719+0000 2024-05-27T07:29:17.843 INFO:journalctl@ceph.mds.g.smithi052.stdout:May 27 07:29:17 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-g[103978]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: 197: FAILED ceph_assert(cached_version > 0) 2024-05-27T07:29:17.843 INFO:journalctl@ceph.mds.g.smithi052.stdout:May 27 07:29:17 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-g[103978]: 2024-05-27T07:29:17.843 INFO:journalctl@ceph.mds.g.smithi052.stdout:May 27 07:29:17 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-g[103978]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:29:17.843 INFO:journalctl@ceph.mds.g.smithi052.stdout:May 27 07:29:17 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-g[103978]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x127) [0x7f98d666aba6] 2024-05-27T07:29:17.843 INFO:journalctl@ceph.mds.g.smithi052.stdout:May 27 07:29:17 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-g[103978]: 2: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7f98d666add1] 2024-05-27T07:29:17.843 INFO:journalctl@ceph.mds.g.smithi052.stdout:May 27 07:29:17 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-g[103978]: 3: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x55efb558123d] 2024-05-27T07:29:17.843 INFO:journalctl@ceph.mds.g.smithi052.stdout:May 27 07:29:17 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-g[103978]: 4: (SnapRealm::build_snap_set() const+0x323) [0x55efb556c971] 2024-05-27T07:29:17.843 INFO:journalctl@ceph.mds.g.smithi052.stdout:May 27 07:29:17 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-g[103978]: 5: (SnapRealm::check_cache() const+0x13e) [0x55efb556afa0] 2024-05-27T07:29:17.843 INFO:journalctl@ceph.mds.g.smithi052.stdout:May 27 07:29:17 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-g[103978]: 6: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x55efb556b651] 2024-05-27T07:29:17.844 INFO:journalctl@ceph.mds.g.smithi052.stdout:May 27 07:29:17 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-g[103978]: 7: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x55efb5500347] 2024-05-27T07:29:17.844 INFO:journalctl@ceph.mds.g.smithi052.stdout:May 27 07:29:17 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-g[103978]: 8: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x55efb51fc87e] 2024-05-27T07:29:17.844 INFO:journalctl@ceph.mds.g.smithi052.stdout:May 27 07:29:17 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-g[103978]: 9: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x2702) [0x55efb5201e94] 2024-05-27T07:29:17.844 INFO:journalctl@ceph.mds.g.smithi052.stdout:May 27 07:29:17 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-g[103978]: 10: (EUpdate::replay(MDSRank*)+0x63) [0x55efb520c969] 2024-05-27T07:29:17.844 INFO:journalctl@ceph.mds.g.smithi052.stdout:May 27 07:29:17 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-g[103978]: 11: (MDLog::_replay_thread()+0x1f38) [0x55efb55a548c] 2024-05-27T07:29:17.844 INFO:journalctl@ceph.mds.g.smithi052.stdout:May 27 07:29:17 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-g[103978]: 12: (MDLog::ReplayThread::entry()+0x11) [0x55efb55ae9cb] 2024-05-27T07:29:17.844 INFO:journalctl@ceph.mds.g.smithi052.stdout:May 27 07:29:17 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-g[103978]: 13: (Thread::entry_wrapper()+0x43) [0x7f98d663972d] 2024-05-27T07:29:17.844 INFO:journalctl@ceph.mds.g.smithi052.stdout:May 27 07:29:17 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-g[103978]: 14: (Thread::_entry_func(void*)+0xd) [0x7f98d6639749] 2024-05-27T07:29:17.844 INFO:journalctl@ceph.mds.g.smithi052.stdout:May 27 07:29:17 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-g[103978]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7f98d51dd1ca] 2024-05-27T07:29:17.844 INFO:journalctl@ceph.mds.g.smithi052.stdout:May 27 07:29:17 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-g[103978]: 16: clone() 2024-05-27T07:29:17.844 INFO:journalctl@ceph.mds.g.smithi052.stdout:May 27 07:29:17 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-g[103978]: 2024-05-27T07:29:17.844 INFO:journalctl@ceph.mds.g.smithi052.stdout:May 27 07:29:17 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-g[103978]: 0> 2024-05-27T07:29:17.548+0000 7f98c3e25700 -1 *** Caught signal (Aborted) ** 2024-05-27T07:29:17.844 INFO:journalctl@ceph.mds.g.smithi052.stdout:May 27 07:29:17 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-g[103978]: in thread 7f98c3e25700 thread_name:md_log_replay 2024-05-27T07:29:17.845 INFO:journalctl@ceph.mds.g.smithi052.stdout:May 27 07:29:17 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-g[103978]: 2024-05-27T07:29:17.845 INFO:journalctl@ceph.mds.g.smithi052.stdout:May 27 07:29:17 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-g[103978]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:29:17.845 INFO:journalctl@ceph.mds.g.smithi052.stdout:May 27 07:29:17 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-g[103978]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f98d51e7d20] 2024-05-27T07:29:17.845 INFO:journalctl@ceph.mds.g.smithi052.stdout:May 27 07:29:17 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-g[103978]: 2: gsignal() 2024-05-27T07:29:17.845 INFO:journalctl@ceph.mds.g.smithi052.stdout:May 27 07:29:17 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-g[103978]: 3: abort() 2024-05-27T07:29:17.845 INFO:journalctl@ceph.mds.g.smithi052.stdout:May 27 07:29:17 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-g[103978]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x26e) [0x7f98d666aced] 2024-05-27T07:29:17.845 INFO:journalctl@ceph.mds.g.smithi052.stdout:May 27 07:29:17 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-g[103978]: 5: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7f98d666add1] 2024-05-27T07:29:17.845 INFO:journalctl@ceph.mds.g.smithi052.stdout:May 27 07:29:17 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-g[103978]: 6: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x55efb558123d] 2024-05-27T07:29:17.845 INFO:journalctl@ceph.mds.g.smithi052.stdout:May 27 07:29:17 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-g[103978]: 7: (SnapRealm::build_snap_set() const+0x323) [0x55efb556c971] 2024-05-27T07:29:17.845 INFO:journalctl@ceph.mds.g.smithi052.stdout:May 27 07:29:17 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-g[103978]: 8: (SnapRealm::check_cache() const+0x13e) [0x55efb556afa0] 2024-05-27T07:29:17.845 INFO:journalctl@ceph.mds.g.smithi052.stdout:May 27 07:29:17 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-g[103978]: 9: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x55efb556b651] 2024-05-27T07:29:17.845 INFO:journalctl@ceph.mds.g.smithi052.stdout:May 27 07:29:17 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-g[103978]: 10: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x55efb5500347] 2024-05-27T07:29:17.846 INFO:journalctl@ceph.mds.g.smithi052.stdout:May 27 07:29:17 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-g[103978]: 11: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x55efb51fc87e] 2024-05-27T07:29:17.846 INFO:journalctl@ceph.mds.g.smithi052.stdout:May 27 07:29:17 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-g[103978]: 12: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x2702) [0x55efb5201e94] 2024-05-27T07:29:17.846 INFO:journalctl@ceph.mds.g.smithi052.stdout:May 27 07:29:17 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-g[103978]: 13: (EUpdate::replay(MDSRank*)+0x63) [0x55efb520c969] 2024-05-27T07:29:17.846 INFO:journalctl@ceph.mds.g.smithi052.stdout:May 27 07:29:17 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-g[103978]: 14: (MDLog::_replay_thread()+0x1f38) [0x55efb55a548c] 2024-05-27T07:29:17.846 INFO:journalctl@ceph.mds.g.smithi052.stdout:May 27 07:29:17 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-g[103978]: 15: (MDLog::ReplayThread::entry()+0x11) [0x55efb55ae9cb] 2024-05-27T07:29:17.846 INFO:journalctl@ceph.mds.g.smithi052.stdout:May 27 07:29:17 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-g[103978]: 16: (Thread::entry_wrapper()+0x43) [0x7f98d663972d] 2024-05-27T07:29:17.846 INFO:journalctl@ceph.mds.g.smithi052.stdout:May 27 07:29:17 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-g[103978]: 17: (Thread::_entry_func(void*)+0xd) [0x7f98d6639749] 2024-05-27T07:29:17.846 INFO:journalctl@ceph.mds.g.smithi052.stdout:May 27 07:29:17 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-g[103978]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7f98d51dd1ca] 2024-05-27T07:29:17.846 INFO:journalctl@ceph.mds.g.smithi052.stdout:May 27 07:29:17 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-g[103978]: 19: clone() 2024-05-27T07:29:17.846 INFO:journalctl@ceph.mds.g.smithi052.stdout:May 27 07:29:17 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-g[103978]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-05-27T07:29:17.846 INFO:journalctl@ceph.mds.g.smithi052.stdout:May 27 07:29:17 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-g[103978]: 2024-05-27T07:29:17.846 INFO:journalctl@ceph.mds.g.smithi052.stdout:May 27 07:29:17 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-g[103978]: -1> 2024-05-27T07:29:17.531+0000 7f98c3e25700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: In function 'std::set SnapClient::filter(const std::set&) const' thread 7f98c3e25700 time 2024-05-27T07:29:17.528719+0000 2024-05-27T07:29:17.847 INFO:journalctl@ceph.mds.g.smithi052.stdout:May 27 07:29:17 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-g[103978]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: 197: FAILED ceph_assert(cached_version > 0) 2024-05-27T07:29:17.847 INFO:journalctl@ceph.mds.g.smithi052.stdout:May 27 07:29:17 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-g[103978]: 2024-05-27T07:29:17.847 INFO:journalctl@ceph.mds.g.smithi052.stdout:May 27 07:29:17 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-g[103978]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:29:17.847 INFO:journalctl@ceph.mds.g.smithi052.stdout:May 27 07:29:17 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-g[103978]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x127) [0x7f98d666aba6] 2024-05-27T07:29:17.847 INFO:journalctl@ceph.mds.g.smithi052.stdout:May 27 07:29:17 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-g[103978]: 2: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7f98d666add1] 2024-05-27T07:29:17.847 INFO:journalctl@ceph.mds.g.smithi052.stdout:May 27 07:29:17 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-g[103978]: 3: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x55efb558123d] 2024-05-27T07:29:17.847 INFO:journalctl@ceph.mds.g.smithi052.stdout:May 27 07:29:17 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-g[103978]: 4: (SnapRealm::build_snap_set() const+0x323) [0x55efb556c971] 2024-05-27T07:29:17.847 INFO:journalctl@ceph.mds.g.smithi052.stdout:May 27 07:29:17 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-g[103978]: 5: (SnapRealm::check_cache() const+0x13e) [0x55efb556afa0] 2024-05-27T07:29:17.847 INFO:journalctl@ceph.mds.g.smithi052.stdout:May 27 07:29:17 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-g[103978]: 6: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x55efb556b651] 2024-05-27T07:29:17.847 INFO:journalctl@ceph.mds.g.smithi052.stdout:May 27 07:29:17 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-g[103978]: 7: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x55efb5500347] 2024-05-27T07:29:17.847 INFO:journalctl@ceph.mds.g.smithi052.stdout:May 27 07:29:17 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-g[103978]: 8: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x55efb51fc87e] 2024-05-27T07:29:17.847 INFO:journalctl@ceph.mds.g.smithi052.stdout:May 27 07:29:17 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-g[103978]: 9: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x2702) [0x55efb5201e94] 2024-05-27T07:29:17.847 INFO:journalctl@ceph.mds.g.smithi052.stdout:May 27 07:29:17 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-g[103978]: 10: (EUpdate::replay(MDSRank*)+0x63) [0x55efb520c969] 2024-05-27T07:29:17.848 INFO:journalctl@ceph.mds.g.smithi052.stdout:May 27 07:29:17 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-g[103978]: 11: (MDLog::_replay_thread()+0x1f38) [0x55efb55a548c] 2024-05-27T07:29:17.848 INFO:journalctl@ceph.mds.g.smithi052.stdout:May 27 07:29:17 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-g[103978]: 12: (MDLog::ReplayThread::entry()+0x11) [0x55efb55ae9cb] 2024-05-27T07:29:17.848 INFO:journalctl@ceph.mds.g.smithi052.stdout:May 27 07:29:17 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-g[103978]: 13: (Thread::entry_wrapper()+0x43) [0x7f98d663972d] 2024-05-27T07:29:17.848 INFO:journalctl@ceph.mds.g.smithi052.stdout:May 27 07:29:17 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-g[103978]: 14: (Thread::_entry_func(void*)+0xd) [0x7f98d6639749] 2024-05-27T07:29:17.848 INFO:journalctl@ceph.mds.g.smithi052.stdout:May 27 07:29:17 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-g[103978]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7f98d51dd1ca] 2024-05-27T07:29:17.848 INFO:journalctl@ceph.mds.g.smithi052.stdout:May 27 07:29:17 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-g[103978]: 16: clone() 2024-05-27T07:29:17.848 INFO:journalctl@ceph.mds.g.smithi052.stdout:May 27 07:29:17 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-g[103978]: 2024-05-27T07:29:17.848 INFO:journalctl@ceph.mds.g.smithi052.stdout:May 27 07:29:17 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-g[103978]: 0> 2024-05-27T07:29:17.548+0000 7f98c3e25700 -1 *** Caught signal (Aborted) ** 2024-05-27T07:29:17.848 INFO:journalctl@ceph.mds.g.smithi052.stdout:May 27 07:29:17 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-g[103978]: in thread 7f98c3e25700 thread_name:md_log_replay 2024-05-27T07:29:17.848 INFO:journalctl@ceph.mds.g.smithi052.stdout:May 27 07:29:17 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-g[103978]: 2024-05-27T07:29:17.848 INFO:journalctl@ceph.mds.g.smithi052.stdout:May 27 07:29:17 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-g[103978]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:29:17.848 INFO:journalctl@ceph.mds.g.smithi052.stdout:May 27 07:29:17 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-g[103978]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f98d51e7d20] 2024-05-27T07:29:17.848 INFO:journalctl@ceph.mds.g.smithi052.stdout:May 27 07:29:17 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-g[103978]: 2: gsignal() 2024-05-27T07:29:17.848 INFO:journalctl@ceph.mds.g.smithi052.stdout:May 27 07:29:17 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-g[103978]: 3: abort() 2024-05-27T07:29:17.849 INFO:journalctl@ceph.mds.g.smithi052.stdout:May 27 07:29:17 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-g[103978]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x26e) [0x7f98d666aced] 2024-05-27T07:29:17.849 INFO:journalctl@ceph.mds.g.smithi052.stdout:May 27 07:29:17 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-g[103978]: 5: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7f98d666add1] 2024-05-27T07:29:17.849 INFO:journalctl@ceph.mds.g.smithi052.stdout:May 27 07:29:17 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-g[103978]: 6: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x55efb558123d] 2024-05-27T07:29:17.849 INFO:journalctl@ceph.mds.g.smithi052.stdout:May 27 07:29:17 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-g[103978]: 7: (SnapRealm::build_snap_set() const+0x323) [0x55efb556c971] 2024-05-27T07:29:17.849 INFO:journalctl@ceph.mds.g.smithi052.stdout:May 27 07:29:17 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-g[103978]: 8: (SnapRealm::check_cache() const+0x13e) [0x55efb556afa0] 2024-05-27T07:29:17.849 INFO:journalctl@ceph.mds.g.smithi052.stdout:May 27 07:29:17 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-g[103978]: 9: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x55efb556b651] 2024-05-27T07:29:17.849 INFO:journalctl@ceph.mds.g.smithi052.stdout:May 27 07:29:17 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-g[103978]: 10: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x55efb5500347] 2024-05-27T07:29:17.849 INFO:journalctl@ceph.mds.g.smithi052.stdout:May 27 07:29:17 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-g[103978]: 11: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x55efb51fc87e] 2024-05-27T07:29:17.849 INFO:journalctl@ceph.mds.g.smithi052.stdout:May 27 07:29:17 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-g[103978]: 12: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x2702) [0x55efb5201e94] 2024-05-27T07:29:17.849 INFO:journalctl@ceph.mds.g.smithi052.stdout:May 27 07:29:17 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-g[103978]: 13: (EUpdate::replay(MDSRank*)+0x63) [0x55efb520c969] 2024-05-27T07:29:17.849 INFO:journalctl@ceph.mds.g.smithi052.stdout:May 27 07:29:17 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-g[103978]: 14: (MDLog::_replay_thread()+0x1f38) [0x55efb55a548c] 2024-05-27T07:29:17.849 INFO:journalctl@ceph.mds.g.smithi052.stdout:May 27 07:29:17 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-g[103978]: 15: (MDLog::ReplayThread::entry()+0x11) [0x55efb55ae9cb] 2024-05-27T07:29:17.849 INFO:journalctl@ceph.mds.g.smithi052.stdout:May 27 07:29:17 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-g[103978]: 16: (Thread::entry_wrapper()+0x43) [0x7f98d663972d] 2024-05-27T07:29:17.849 INFO:journalctl@ceph.mds.g.smithi052.stdout:May 27 07:29:17 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-g[103978]: 17: (Thread::_entry_func(void*)+0xd) [0x7f98d6639749] 2024-05-27T07:29:17.849 INFO:journalctl@ceph.mds.g.smithi052.stdout:May 27 07:29:17 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-g[103978]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7f98d51dd1ca] 2024-05-27T07:29:17.850 INFO:journalctl@ceph.mds.g.smithi052.stdout:May 27 07:29:17 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-g[103978]: 19: clone() 2024-05-27T07:29:17.850 INFO:journalctl@ceph.mds.g.smithi052.stdout:May 27 07:29:17 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-g[103978]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-05-27T07:29:17.850 INFO:journalctl@ceph.mds.g.smithi052.stdout:May 27 07:29:17 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-g[103978]: 2024-05-27T07:29:17.850 INFO:journalctl@ceph.mds.g.smithi052.stdout:May 27 07:29:17 smithi052 podman[104570]: 2024-05-27 07:29:17.841545163 +0000 UTC m=+0.181737536 container died 881cedb8d2465d95bc3a2989662221d785f4982234a579c062b0e70889cb4b9a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-g, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, ceph=True, org.label-schema.build-date=20240521, org.label-schema.schema-version=1.0, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, GIT_CLEAN=True, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS) 2024-05-27T07:29:18.105 INFO:journalctl@ceph.mds.d.smithi052.stdout:May 27 07:29:18 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-d[104512]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: In function 'std::set SnapClient::filter(const std::set&) const' thread 7f5263e9b700 time 2024-05-27T07:29:18.072979+0000 2024-05-27T07:29:18.106 INFO:journalctl@ceph.mds.d.smithi052.stdout:May 27 07:29:18 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-d[104512]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: 197: FAILED ceph_assert(cached_version > 0) 2024-05-27T07:29:18.106 INFO:journalctl@ceph.mds.d.smithi052.stdout:May 27 07:29:18 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-d[104512]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:29:18.106 INFO:journalctl@ceph.mds.d.smithi052.stdout:May 27 07:29:18 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-d[104512]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x127) [0x7f52766e0ba6] 2024-05-27T07:29:18.106 INFO:journalctl@ceph.mds.d.smithi052.stdout:May 27 07:29:18 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-d[104512]: 2: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7f52766e0dd1] 2024-05-27T07:29:18.106 INFO:journalctl@ceph.mds.d.smithi052.stdout:May 27 07:29:18 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-d[104512]: 3: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x558fca00b23d] 2024-05-27T07:29:18.106 INFO:journalctl@ceph.mds.d.smithi052.stdout:May 27 07:29:18 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-d[104512]: 4: (SnapRealm::build_snap_set() const+0x323) [0x558fc9ff6971] 2024-05-27T07:29:18.106 INFO:journalctl@ceph.mds.d.smithi052.stdout:May 27 07:29:18 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-d[104512]: 5: (SnapRealm::check_cache() const+0x13e) [0x558fc9ff4fa0] 2024-05-27T07:29:18.106 INFO:journalctl@ceph.mds.d.smithi052.stdout:May 27 07:29:18 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-d[104512]: 6: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x558fc9ff5651] 2024-05-27T07:29:18.106 INFO:journalctl@ceph.mds.d.smithi052.stdout:May 27 07:29:18 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-d[104512]: 7: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x558fc9f8a347] 2024-05-27T07:29:18.106 INFO:journalctl@ceph.mds.d.smithi052.stdout:May 27 07:29:18 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-d[104512]: 8: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x558fc9c8687e] 2024-05-27T07:29:18.106 INFO:journalctl@ceph.mds.d.smithi052.stdout:May 27 07:29:18 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-d[104512]: 9: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x2702) [0x558fc9c8be94] 2024-05-27T07:29:18.107 INFO:journalctl@ceph.mds.d.smithi052.stdout:May 27 07:29:18 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-d[104512]: 10: (EUpdate::replay(MDSRank*)+0x63) [0x558fc9c96969] 2024-05-27T07:29:18.107 INFO:journalctl@ceph.mds.d.smithi052.stdout:May 27 07:29:18 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-d[104512]: 11: (MDLog::_replay_thread()+0x1f38) [0x558fca02f48c] 2024-05-27T07:29:18.107 INFO:journalctl@ceph.mds.d.smithi052.stdout:May 27 07:29:18 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-d[104512]: 12: (MDLog::ReplayThread::entry()+0x11) [0x558fca0389cb] 2024-05-27T07:29:18.107 INFO:journalctl@ceph.mds.d.smithi052.stdout:May 27 07:29:18 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-d[104512]: 13: (Thread::entry_wrapper()+0x43) [0x7f52766af72d] 2024-05-27T07:29:18.107 INFO:journalctl@ceph.mds.d.smithi052.stdout:May 27 07:29:18 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-d[104512]: 14: (Thread::_entry_func(void*)+0xd) [0x7f52766af749] 2024-05-27T07:29:18.107 INFO:journalctl@ceph.mds.d.smithi052.stdout:May 27 07:29:18 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-d[104512]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7f52752531ca] 2024-05-27T07:29:18.107 INFO:journalctl@ceph.mds.d.smithi052.stdout:May 27 07:29:18 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-d[104512]: 16: clone() 2024-05-27T07:29:18.107 INFO:journalctl@ceph.mds.d.smithi052.stdout:May 27 07:29:18 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-d[104512]: 2024-05-27T07:29:18.075+0000 7f5263e9b700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: In function 'std::set SnapClient::filter(const std::set&) const' thread 7f5263e9b700 time 2024-05-27T07:29:18.072979+0000 2024-05-27T07:29:18.107 INFO:journalctl@ceph.mds.d.smithi052.stdout:May 27 07:29:18 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-d[104512]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: 197: FAILED ceph_assert(cached_version > 0) 2024-05-27T07:29:18.108 INFO:journalctl@ceph.mds.d.smithi052.stdout:May 27 07:29:18 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-d[104512]: 2024-05-27T07:29:18.108 INFO:journalctl@ceph.mds.d.smithi052.stdout:May 27 07:29:18 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-d[104512]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:29:18.108 INFO:journalctl@ceph.mds.d.smithi052.stdout:May 27 07:29:18 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-d[104512]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x127) [0x7f52766e0ba6] 2024-05-27T07:29:18.108 INFO:journalctl@ceph.mds.d.smithi052.stdout:May 27 07:29:18 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-d[104512]: 2: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7f52766e0dd1] 2024-05-27T07:29:18.108 INFO:journalctl@ceph.mds.d.smithi052.stdout:May 27 07:29:18 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-d[104512]: 3: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x558fca00b23d] 2024-05-27T07:29:18.108 INFO:journalctl@ceph.mds.d.smithi052.stdout:May 27 07:29:18 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-d[104512]: 4: (SnapRealm::build_snap_set() const+0x323) [0x558fc9ff6971] 2024-05-27T07:29:18.108 INFO:journalctl@ceph.mds.d.smithi052.stdout:May 27 07:29:18 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-d[104512]: 5: (SnapRealm::check_cache() const+0x13e) [0x558fc9ff4fa0] 2024-05-27T07:29:18.108 INFO:journalctl@ceph.mds.d.smithi052.stdout:May 27 07:29:18 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-d[104512]: 6: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x558fc9ff5651] 2024-05-27T07:29:18.108 INFO:journalctl@ceph.mds.d.smithi052.stdout:May 27 07:29:18 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-d[104512]: 7: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x558fc9f8a347] 2024-05-27T07:29:18.108 INFO:journalctl@ceph.mds.d.smithi052.stdout:May 27 07:29:18 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-d[104512]: 8: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x558fc9c8687e] 2024-05-27T07:29:18.108 INFO:journalctl@ceph.mds.d.smithi052.stdout:May 27 07:29:18 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-d[104512]: 9: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x2702) [0x558fc9c8be94] 2024-05-27T07:29:18.108 INFO:journalctl@ceph.mds.d.smithi052.stdout:May 27 07:29:18 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-d[104512]: 10: (EUpdate::replay(MDSRank*)+0x63) [0x558fc9c96969] 2024-05-27T07:29:18.108 INFO:journalctl@ceph.mds.d.smithi052.stdout:May 27 07:29:18 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-d[104512]: 11: (MDLog::_replay_thread()+0x1f38) [0x558fca02f48c] 2024-05-27T07:29:18.108 INFO:journalctl@ceph.mds.d.smithi052.stdout:May 27 07:29:18 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-d[104512]: 12: (MDLog::ReplayThread::entry()+0x11) [0x558fca0389cb] 2024-05-27T07:29:18.108 INFO:journalctl@ceph.mds.d.smithi052.stdout:May 27 07:29:18 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-d[104512]: 13: (Thread::entry_wrapper()+0x43) [0x7f52766af72d] 2024-05-27T07:29:18.109 INFO:journalctl@ceph.mds.d.smithi052.stdout:May 27 07:29:18 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-d[104512]: 14: (Thread::_entry_func(void*)+0xd) [0x7f52766af749] 2024-05-27T07:29:18.109 INFO:journalctl@ceph.mds.d.smithi052.stdout:May 27 07:29:18 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-d[104512]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7f52752531ca] 2024-05-27T07:29:18.109 INFO:journalctl@ceph.mds.d.smithi052.stdout:May 27 07:29:18 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-d[104512]: 16: clone() 2024-05-27T07:29:18.109 INFO:journalctl@ceph.mds.d.smithi052.stdout:May 27 07:29:18 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-d[104512]: 2024-05-27T07:29:18.109 INFO:journalctl@ceph.mds.d.smithi052.stdout:May 27 07:29:18 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-d[104512]: *** Caught signal (Aborted) ** 2024-05-27T07:29:18.109 INFO:journalctl@ceph.mds.d.smithi052.stdout:May 27 07:29:18 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-d[104512]: in thread 7f5263e9b700 thread_name:md_log_replay 2024-05-27T07:29:18.109 INFO:journalctl@ceph.mds.d.smithi052.stdout:May 27 07:29:18 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-d[104512]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:29:18.109 INFO:journalctl@ceph.mds.d.smithi052.stdout:May 27 07:29:18 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-d[104512]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f527525dd20] 2024-05-27T07:29:18.109 INFO:journalctl@ceph.mds.d.smithi052.stdout:May 27 07:29:18 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-d[104512]: 2: gsignal() 2024-05-27T07:29:18.109 INFO:journalctl@ceph.mds.d.smithi052.stdout:May 27 07:29:18 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-d[104512]: 3: abort() 2024-05-27T07:29:18.109 INFO:journalctl@ceph.mds.d.smithi052.stdout:May 27 07:29:18 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-d[104512]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x26e) [0x7f52766e0ced] 2024-05-27T07:29:18.110 INFO:journalctl@ceph.mds.d.smithi052.stdout:May 27 07:29:18 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-d[104512]: 5: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7f52766e0dd1] 2024-05-27T07:29:18.110 INFO:journalctl@ceph.mds.d.smithi052.stdout:May 27 07:29:18 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-d[104512]: 6: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x558fca00b23d] 2024-05-27T07:29:18.110 INFO:journalctl@ceph.mds.d.smithi052.stdout:May 27 07:29:18 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-d[104512]: 7: (SnapRealm::build_snap_set() const+0x323) [0x558fc9ff6971] 2024-05-27T07:29:18.110 INFO:journalctl@ceph.mds.d.smithi052.stdout:May 27 07:29:18 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-d[104512]: 8: (SnapRealm::check_cache() const+0x13e) [0x558fc9ff4fa0] 2024-05-27T07:29:18.110 INFO:journalctl@ceph.mds.d.smithi052.stdout:May 27 07:29:18 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-d[104512]: 9: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x558fc9ff5651] 2024-05-27T07:29:18.110 INFO:journalctl@ceph.mds.d.smithi052.stdout:May 27 07:29:18 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-d[104512]: 10: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x558fc9f8a347] 2024-05-27T07:29:18.110 INFO:journalctl@ceph.mds.d.smithi052.stdout:May 27 07:29:18 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-d[104512]: 11: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x558fc9c8687e] 2024-05-27T07:29:18.110 INFO:journalctl@ceph.mds.d.smithi052.stdout:May 27 07:29:18 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-d[104512]: 12: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x2702) [0x558fc9c8be94] 2024-05-27T07:29:18.111 INFO:journalctl@ceph.mds.d.smithi052.stdout:May 27 07:29:18 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-d[104512]: 13: (EUpdate::replay(MDSRank*)+0x63) [0x558fc9c96969] 2024-05-27T07:29:18.111 INFO:journalctl@ceph.mds.d.smithi052.stdout:May 27 07:29:18 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-d[104512]: 14: (MDLog::_replay_thread()+0x1f38) [0x558fca02f48c] 2024-05-27T07:29:18.111 INFO:journalctl@ceph.mds.d.smithi052.stdout:May 27 07:29:18 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-d[104512]: 15: (MDLog::ReplayThread::entry()+0x11) [0x558fca0389cb] 2024-05-27T07:29:18.111 INFO:journalctl@ceph.mds.d.smithi052.stdout:May 27 07:29:18 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-d[104512]: 16: (Thread::entry_wrapper()+0x43) [0x7f52766af72d] 2024-05-27T07:29:18.111 INFO:journalctl@ceph.mds.d.smithi052.stdout:May 27 07:29:18 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-d[104512]: 17: (Thread::_entry_func(void*)+0xd) [0x7f52766af749] 2024-05-27T07:29:18.111 INFO:journalctl@ceph.mds.d.smithi052.stdout:May 27 07:29:18 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-d[104512]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7f52752531ca] 2024-05-27T07:29:18.111 INFO:journalctl@ceph.mds.d.smithi052.stdout:May 27 07:29:18 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-d[104512]: 19: clone() 2024-05-27T07:29:18.111 INFO:journalctl@ceph.mds.d.smithi052.stdout:May 27 07:29:18 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-d[104512]: 2024-05-27T07:29:18.079+0000 7f5263e9b700 -1 *** Caught signal (Aborted) ** 2024-05-27T07:29:18.111 INFO:journalctl@ceph.mds.d.smithi052.stdout:May 27 07:29:18 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-d[104512]: in thread 7f5263e9b700 thread_name:md_log_replay 2024-05-27T07:29:18.111 INFO:journalctl@ceph.mds.d.smithi052.stdout:May 27 07:29:18 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-d[104512]: 2024-05-27T07:29:18.111 INFO:journalctl@ceph.mds.d.smithi052.stdout:May 27 07:29:18 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-d[104512]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:29:18.111 INFO:journalctl@ceph.mds.d.smithi052.stdout:May 27 07:29:18 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-d[104512]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f527525dd20] 2024-05-27T07:29:18.111 INFO:journalctl@ceph.mds.d.smithi052.stdout:May 27 07:29:18 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-d[104512]: 2: gsignal() 2024-05-27T07:29:18.111 INFO:journalctl@ceph.mds.d.smithi052.stdout:May 27 07:29:18 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-d[104512]: 3: abort() 2024-05-27T07:29:18.112 INFO:journalctl@ceph.mds.d.smithi052.stdout:May 27 07:29:18 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-d[104512]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x26e) [0x7f52766e0ced] 2024-05-27T07:29:18.112 INFO:journalctl@ceph.mds.d.smithi052.stdout:May 27 07:29:18 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-d[104512]: 5: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7f52766e0dd1] 2024-05-27T07:29:18.112 INFO:journalctl@ceph.mds.d.smithi052.stdout:May 27 07:29:18 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-d[104512]: 6: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x558fca00b23d] 2024-05-27T07:29:18.112 INFO:journalctl@ceph.mds.d.smithi052.stdout:May 27 07:29:18 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-d[104512]: 7: (SnapRealm::build_snap_set() const+0x323) [0x558fc9ff6971] 2024-05-27T07:29:18.112 INFO:journalctl@ceph.mds.d.smithi052.stdout:May 27 07:29:18 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-d[104512]: 8: (SnapRealm::check_cache() const+0x13e) [0x558fc9ff4fa0] 2024-05-27T07:29:18.112 INFO:journalctl@ceph.mds.d.smithi052.stdout:May 27 07:29:18 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-d[104512]: 9: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x558fc9ff5651] 2024-05-27T07:29:18.112 INFO:journalctl@ceph.mds.d.smithi052.stdout:May 27 07:29:18 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-d[104512]: 10: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x558fc9f8a347] 2024-05-27T07:29:18.112 INFO:journalctl@ceph.mds.d.smithi052.stdout:May 27 07:29:18 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-d[104512]: 11: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x558fc9c8687e] 2024-05-27T07:29:18.112 INFO:journalctl@ceph.mds.d.smithi052.stdout:May 27 07:29:18 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-d[104512]: 12: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x2702) [0x558fc9c8be94] 2024-05-27T07:29:18.112 INFO:journalctl@ceph.mds.d.smithi052.stdout:May 27 07:29:18 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-d[104512]: 13: (EUpdate::replay(MDSRank*)+0x63) [0x558fc9c96969] 2024-05-27T07:29:18.112 INFO:journalctl@ceph.mds.d.smithi052.stdout:May 27 07:29:18 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-d[104512]: 14: (MDLog::_replay_thread()+0x1f38) [0x558fca02f48c] 2024-05-27T07:29:18.112 INFO:journalctl@ceph.mds.d.smithi052.stdout:May 27 07:29:18 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-d[104512]: 15: (MDLog::ReplayThread::entry()+0x11) [0x558fca0389cb] 2024-05-27T07:29:18.112 INFO:journalctl@ceph.mds.d.smithi052.stdout:May 27 07:29:18 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-d[104512]: 16: (Thread::entry_wrapper()+0x43) [0x7f52766af72d] 2024-05-27T07:29:18.113 INFO:journalctl@ceph.mds.d.smithi052.stdout:May 27 07:29:18 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-d[104512]: 17: (Thread::_entry_func(void*)+0xd) [0x7f52766af749] 2024-05-27T07:29:18.113 INFO:journalctl@ceph.mds.d.smithi052.stdout:May 27 07:29:18 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-d[104512]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7f52752531ca] 2024-05-27T07:29:18.113 INFO:journalctl@ceph.mds.d.smithi052.stdout:May 27 07:29:18 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-d[104512]: 19: clone() 2024-05-27T07:29:18.113 INFO:journalctl@ceph.mds.d.smithi052.stdout:May 27 07:29:18 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-d[104512]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-05-27T07:29:18.113 INFO:journalctl@ceph.mds.d.smithi052.stdout:May 27 07:29:18 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-d[104512]: 2024-05-27T07:29:18.113 INFO:journalctl@ceph.mds.d.smithi052.stdout:May 27 07:29:18 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-d[104512]: -1> 2024-05-27T07:29:18.075+0000 7f5263e9b700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: In function 'std::set SnapClient::filter(const std::set&) const' thread 7f5263e9b700 time 2024-05-27T07:29:18.072979+0000 2024-05-27T07:29:18.113 INFO:journalctl@ceph.mds.d.smithi052.stdout:May 27 07:29:18 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-d[104512]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: 197: FAILED ceph_assert(cached_version > 0) 2024-05-27T07:29:18.113 INFO:journalctl@ceph.mds.d.smithi052.stdout:May 27 07:29:18 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-d[104512]: 2024-05-27T07:29:18.113 INFO:journalctl@ceph.mds.d.smithi052.stdout:May 27 07:29:18 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-d[104512]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:29:18.113 INFO:journalctl@ceph.mds.d.smithi052.stdout:May 27 07:29:18 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-d[104512]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x127) [0x7f52766e0ba6] 2024-05-27T07:29:18.113 INFO:journalctl@ceph.mds.d.smithi052.stdout:May 27 07:29:18 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-d[104512]: 2: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7f52766e0dd1] 2024-05-27T07:29:18.113 INFO:journalctl@ceph.mds.d.smithi052.stdout:May 27 07:29:18 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-d[104512]: 3: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x558fca00b23d] 2024-05-27T07:29:18.114 INFO:journalctl@ceph.mds.d.smithi052.stdout:May 27 07:29:18 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-d[104512]: 4: (SnapRealm::build_snap_set() const+0x323) [0x558fc9ff6971] 2024-05-27T07:29:18.114 INFO:journalctl@ceph.mds.d.smithi052.stdout:May 27 07:29:18 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-d[104512]: 5: (SnapRealm::check_cache() const+0x13e) [0x558fc9ff4fa0] 2024-05-27T07:29:18.114 INFO:journalctl@ceph.mds.d.smithi052.stdout:May 27 07:29:18 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-d[104512]: 6: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x558fc9ff5651] 2024-05-27T07:29:18.114 INFO:journalctl@ceph.mds.d.smithi052.stdout:May 27 07:29:18 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-d[104512]: 7: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x558fc9f8a347] 2024-05-27T07:29:18.114 INFO:journalctl@ceph.mds.d.smithi052.stdout:May 27 07:29:18 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-d[104512]: 8: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x558fc9c8687e] 2024-05-27T07:29:18.122 INFO:journalctl@ceph.mds.d.smithi052.stdout:May 27 07:29:18 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-d[104512]: 9: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x2702) [0x558fc9c8be94] 2024-05-27T07:29:18.122 INFO:journalctl@ceph.mds.d.smithi052.stdout:May 27 07:29:18 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-d[104512]: 10: (EUpdate::replay(MDSRank*)+0x63) [0x558fc9c96969] 2024-05-27T07:29:18.123 INFO:journalctl@ceph.mds.d.smithi052.stdout:May 27 07:29:18 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-d[104512]: 11: (MDLog::_replay_thread()+0x1f38) [0x558fca02f48c] 2024-05-27T07:29:18.123 INFO:journalctl@ceph.mds.d.smithi052.stdout:May 27 07:29:18 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-d[104512]: 12: (MDLog::ReplayThread::entry()+0x11) [0x558fca0389cb] 2024-05-27T07:29:18.123 INFO:journalctl@ceph.mds.d.smithi052.stdout:May 27 07:29:18 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-d[104512]: 13: (Thread::entry_wrapper()+0x43) [0x7f52766af72d] 2024-05-27T07:29:18.123 INFO:journalctl@ceph.mds.d.smithi052.stdout:May 27 07:29:18 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-d[104512]: 14: (Thread::_entry_func(void*)+0xd) [0x7f52766af749] 2024-05-27T07:29:18.123 INFO:journalctl@ceph.mds.d.smithi052.stdout:May 27 07:29:18 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-d[104512]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7f52752531ca] 2024-05-27T07:29:18.123 INFO:journalctl@ceph.mds.d.smithi052.stdout:May 27 07:29:18 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-d[104512]: 16: clone() 2024-05-27T07:29:18.123 INFO:journalctl@ceph.mds.d.smithi052.stdout:May 27 07:29:18 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-d[104512]: 2024-05-27T07:29:18.123 INFO:journalctl@ceph.mds.d.smithi052.stdout:May 27 07:29:18 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-d[104512]: 0> 2024-05-27T07:29:18.079+0000 7f5263e9b700 -1 *** Caught signal (Aborted) ** 2024-05-27T07:29:18.123 INFO:journalctl@ceph.mds.d.smithi052.stdout:May 27 07:29:18 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-d[104512]: in thread 7f5263e9b700 thread_name:md_log_replay 2024-05-27T07:29:18.123 INFO:journalctl@ceph.mds.d.smithi052.stdout:May 27 07:29:18 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-d[104512]: 2024-05-27T07:29:18.123 INFO:journalctl@ceph.mds.d.smithi052.stdout:May 27 07:29:18 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-d[104512]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:29:18.124 INFO:journalctl@ceph.mds.d.smithi052.stdout:May 27 07:29:18 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-d[104512]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f527525dd20] 2024-05-27T07:29:18.124 INFO:journalctl@ceph.mds.d.smithi052.stdout:May 27 07:29:18 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-d[104512]: 2: gsignal() 2024-05-27T07:29:18.124 INFO:journalctl@ceph.mds.d.smithi052.stdout:May 27 07:29:18 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-d[104512]: 3: abort() 2024-05-27T07:29:18.124 INFO:journalctl@ceph.mds.d.smithi052.stdout:May 27 07:29:18 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-d[104512]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x26e) [0x7f52766e0ced] 2024-05-27T07:29:18.124 INFO:journalctl@ceph.mds.d.smithi052.stdout:May 27 07:29:18 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-d[104512]: 5: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7f52766e0dd1] 2024-05-27T07:29:18.124 INFO:journalctl@ceph.mds.d.smithi052.stdout:May 27 07:29:18 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-d[104512]: 6: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x558fca00b23d] 2024-05-27T07:29:18.124 INFO:journalctl@ceph.mds.d.smithi052.stdout:May 27 07:29:18 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-d[104512]: 7: (SnapRealm::build_snap_set() const+0x323) [0x558fc9ff6971] 2024-05-27T07:29:18.124 INFO:journalctl@ceph.mds.d.smithi052.stdout:May 27 07:29:18 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-d[104512]: 8: (SnapRealm::check_cache() const+0x13e) [0x558fc9ff4fa0] 2024-05-27T07:29:18.124 INFO:journalctl@ceph.mds.d.smithi052.stdout:May 27 07:29:18 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-d[104512]: 9: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x558fc9ff5651] 2024-05-27T07:29:18.125 INFO:journalctl@ceph.mds.d.smithi052.stdout:May 27 07:29:18 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-d[104512]: 10: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x558fc9f8a347] 2024-05-27T07:29:18.125 INFO:journalctl@ceph.mds.d.smithi052.stdout:May 27 07:29:18 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-d[104512]: 11: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x558fc9c8687e] 2024-05-27T07:29:18.125 INFO:journalctl@ceph.mds.d.smithi052.stdout:May 27 07:29:18 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-d[104512]: 12: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x2702) [0x558fc9c8be94] 2024-05-27T07:29:18.125 INFO:journalctl@ceph.mds.d.smithi052.stdout:May 27 07:29:18 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-d[104512]: 13: (EUpdate::replay(MDSRank*)+0x63) [0x558fc9c96969] 2024-05-27T07:29:18.125 INFO:journalctl@ceph.mds.d.smithi052.stdout:May 27 07:29:18 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-d[104512]: 14: (MDLog::_replay_thread()+0x1f38) [0x558fca02f48c] 2024-05-27T07:29:18.125 INFO:journalctl@ceph.mds.d.smithi052.stdout:May 27 07:29:18 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-d[104512]: 15: (MDLog::ReplayThread::entry()+0x11) [0x558fca0389cb] 2024-05-27T07:29:18.125 INFO:journalctl@ceph.mds.d.smithi052.stdout:May 27 07:29:18 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-d[104512]: 16: (Thread::entry_wrapper()+0x43) [0x7f52766af72d] 2024-05-27T07:29:18.125 INFO:journalctl@ceph.mds.d.smithi052.stdout:May 27 07:29:18 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-d[104512]: 17: (Thread::_entry_func(void*)+0xd) [0x7f52766af749] 2024-05-27T07:29:18.125 INFO:journalctl@ceph.mds.d.smithi052.stdout:May 27 07:29:18 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-d[104512]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7f52752531ca] 2024-05-27T07:29:18.125 INFO:journalctl@ceph.mds.d.smithi052.stdout:May 27 07:29:18 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-d[104512]: 19: clone() 2024-05-27T07:29:18.125 INFO:journalctl@ceph.mds.d.smithi052.stdout:May 27 07:29:18 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-d[104512]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-05-27T07:29:18.125 INFO:journalctl@ceph.mds.d.smithi052.stdout:May 27 07:29:18 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-d[104512]: 2024-05-27T07:29:18.481 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:29:18 smithi052 ceph-mon[31955]: scrub summary: idle 2024-05-27T07:29:18.481 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:29:18 smithi052 ceph-mon[31955]: pgmap v112: 129 pgs: 129 active+clean; 213 MiB data, 1016 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 MiB/s rd, 5.7 MiB/s wr, 1.31k op/s 2024-05-27T07:29:18.481 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:29:18 smithi052 ceph-mon[31955]: osdmap e101: 12 total, 12 up, 12 in 2024-05-27T07:29:18.481 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:29:18 smithi052 ceph-mon[31955]: mds.? [v2:172.21.15.52:6838/4186787748,v1:172.21.15.52:6839/4186787748] up:boot 2024-05-27T07:29:18.481 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:29:18 smithi052 ceph-mon[31955]: MDS daemon mds.k is removed because it is dead or otherwise unavailable. 2024-05-27T07:29:18.482 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:29:18 smithi052 ceph-mon[31955]: fsmap cephfs:5 4 up:standby-replay 3 up:standby 5 up:active 2024-05-27T07:29:18.482 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:29:18 smithi052 ceph-mon[31955]: from='mgr.15333 172.21.15.52:0/416389082' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "d"}]: dispatch 2024-05-27T07:29:18.482 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:29:18 smithi052 ceph-mon[31955]: fsmap cephfs:5 5 up:standby-replay 1 up:standby 5 up:active 2024-05-27T07:29:18.482 INFO:journalctl@ceph.mds.d.smithi052.stdout:May 27 07:29:18 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-d[104512]: -1> 2024-05-27T07:29:18.075+0000 7f5263e9b700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: In function 'std::set SnapClient::filter(const std::set&) const' thread 7f5263e9b700 time 2024-05-27T07:29:18.072979+0000 2024-05-27T07:29:18.482 INFO:journalctl@ceph.mds.d.smithi052.stdout:May 27 07:29:18 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-d[104512]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: 197: FAILED ceph_assert(cached_version > 0) 2024-05-27T07:29:18.483 INFO:journalctl@ceph.mds.d.smithi052.stdout:May 27 07:29:18 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-d[104512]: 2024-05-27T07:29:18.483 INFO:journalctl@ceph.mds.d.smithi052.stdout:May 27 07:29:18 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-d[104512]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:29:18.483 INFO:journalctl@ceph.mds.d.smithi052.stdout:May 27 07:29:18 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-d[104512]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x127) [0x7f52766e0ba6] 2024-05-27T07:29:18.483 INFO:journalctl@ceph.mds.d.smithi052.stdout:May 27 07:29:18 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-d[104512]: 2: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7f52766e0dd1] 2024-05-27T07:29:18.483 INFO:journalctl@ceph.mds.d.smithi052.stdout:May 27 07:29:18 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-d[104512]: 3: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x558fca00b23d] 2024-05-27T07:29:18.483 INFO:journalctl@ceph.mds.d.smithi052.stdout:May 27 07:29:18 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-d[104512]: 4: (SnapRealm::build_snap_set() const+0x323) [0x558fc9ff6971] 2024-05-27T07:29:18.483 INFO:journalctl@ceph.mds.d.smithi052.stdout:May 27 07:29:18 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-d[104512]: 5: (SnapRealm::check_cache() const+0x13e) [0x558fc9ff4fa0] 2024-05-27T07:29:18.483 INFO:journalctl@ceph.mds.d.smithi052.stdout:May 27 07:29:18 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-d[104512]: 6: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x558fc9ff5651] 2024-05-27T07:29:18.483 INFO:journalctl@ceph.mds.d.smithi052.stdout:May 27 07:29:18 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-d[104512]: 7: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x558fc9f8a347] 2024-05-27T07:29:18.483 INFO:journalctl@ceph.mds.d.smithi052.stdout:May 27 07:29:18 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-d[104512]: 8: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x558fc9c8687e] 2024-05-27T07:29:18.483 INFO:journalctl@ceph.mds.d.smithi052.stdout:May 27 07:29:18 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-d[104512]: 9: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x2702) [0x558fc9c8be94] 2024-05-27T07:29:18.483 INFO:journalctl@ceph.mds.d.smithi052.stdout:May 27 07:29:18 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-d[104512]: 10: (EUpdate::replay(MDSRank*)+0x63) [0x558fc9c96969] 2024-05-27T07:29:18.484 INFO:journalctl@ceph.mds.d.smithi052.stdout:May 27 07:29:18 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-d[104512]: 11: (MDLog::_replay_thread()+0x1f38) [0x558fca02f48c] 2024-05-27T07:29:18.484 INFO:journalctl@ceph.mds.d.smithi052.stdout:May 27 07:29:18 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-d[104512]: 12: (MDLog::ReplayThread::entry()+0x11) [0x558fca0389cb] 2024-05-27T07:29:18.484 INFO:journalctl@ceph.mds.d.smithi052.stdout:May 27 07:29:18 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-d[104512]: 13: (Thread::entry_wrapper()+0x43) [0x7f52766af72d] 2024-05-27T07:29:18.484 INFO:journalctl@ceph.mds.d.smithi052.stdout:May 27 07:29:18 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-d[104512]: 14: (Thread::_entry_func(void*)+0xd) [0x7f52766af749] 2024-05-27T07:29:18.484 INFO:journalctl@ceph.mds.d.smithi052.stdout:May 27 07:29:18 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-d[104512]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7f52752531ca] 2024-05-27T07:29:18.484 INFO:journalctl@ceph.mds.d.smithi052.stdout:May 27 07:29:18 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-d[104512]: 16: clone() 2024-05-27T07:29:18.484 INFO:journalctl@ceph.mds.d.smithi052.stdout:May 27 07:29:18 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-d[104512]: 2024-05-27T07:29:18.484 INFO:journalctl@ceph.mds.d.smithi052.stdout:May 27 07:29:18 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-d[104512]: 0> 2024-05-27T07:29:18.079+0000 7f5263e9b700 -1 *** Caught signal (Aborted) ** 2024-05-27T07:29:18.484 INFO:journalctl@ceph.mds.d.smithi052.stdout:May 27 07:29:18 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-d[104512]: in thread 7f5263e9b700 thread_name:md_log_replay 2024-05-27T07:29:18.484 INFO:journalctl@ceph.mds.d.smithi052.stdout:May 27 07:29:18 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-d[104512]: 2024-05-27T07:29:18.484 INFO:journalctl@ceph.mds.d.smithi052.stdout:May 27 07:29:18 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-d[104512]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:29:18.484 INFO:journalctl@ceph.mds.d.smithi052.stdout:May 27 07:29:18 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-d[104512]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f527525dd20] 2024-05-27T07:29:18.485 INFO:journalctl@ceph.mds.d.smithi052.stdout:May 27 07:29:18 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-d[104512]: 2: gsignal() 2024-05-27T07:29:18.485 INFO:journalctl@ceph.mds.d.smithi052.stdout:May 27 07:29:18 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-d[104512]: 3: abort() 2024-05-27T07:29:18.485 INFO:journalctl@ceph.mds.d.smithi052.stdout:May 27 07:29:18 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-d[104512]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x26e) [0x7f52766e0ced] 2024-05-27T07:29:18.485 INFO:journalctl@ceph.mds.d.smithi052.stdout:May 27 07:29:18 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-d[104512]: 5: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7f52766e0dd1] 2024-05-27T07:29:18.485 INFO:journalctl@ceph.mds.d.smithi052.stdout:May 27 07:29:18 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-d[104512]: 6: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x558fca00b23d] 2024-05-27T07:29:18.485 INFO:journalctl@ceph.mds.d.smithi052.stdout:May 27 07:29:18 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-d[104512]: 7: (SnapRealm::build_snap_set() const+0x323) [0x558fc9ff6971] 2024-05-27T07:29:18.485 INFO:journalctl@ceph.mds.d.smithi052.stdout:May 27 07:29:18 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-d[104512]: 8: (SnapRealm::check_cache() const+0x13e) [0x558fc9ff4fa0] 2024-05-27T07:29:18.485 INFO:journalctl@ceph.mds.d.smithi052.stdout:May 27 07:29:18 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-d[104512]: 9: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x558fc9ff5651] 2024-05-27T07:29:18.485 INFO:journalctl@ceph.mds.d.smithi052.stdout:May 27 07:29:18 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-d[104512]: 10: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x558fc9f8a347] 2024-05-27T07:29:18.485 INFO:journalctl@ceph.mds.d.smithi052.stdout:May 27 07:29:18 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-d[104512]: 11: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x558fc9c8687e] 2024-05-27T07:29:18.485 INFO:journalctl@ceph.mds.d.smithi052.stdout:May 27 07:29:18 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-d[104512]: 12: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x2702) [0x558fc9c8be94] 2024-05-27T07:29:18.485 INFO:journalctl@ceph.mds.d.smithi052.stdout:May 27 07:29:18 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-d[104512]: 13: (EUpdate::replay(MDSRank*)+0x63) [0x558fc9c96969] 2024-05-27T07:29:18.486 INFO:journalctl@ceph.mds.d.smithi052.stdout:May 27 07:29:18 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-d[104512]: 14: (MDLog::_replay_thread()+0x1f38) [0x558fca02f48c] 2024-05-27T07:29:18.486 INFO:journalctl@ceph.mds.d.smithi052.stdout:May 27 07:29:18 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-d[104512]: 15: (MDLog::ReplayThread::entry()+0x11) [0x558fca0389cb] 2024-05-27T07:29:18.486 INFO:journalctl@ceph.mds.d.smithi052.stdout:May 27 07:29:18 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-d[104512]: 16: (Thread::entry_wrapper()+0x43) [0x7f52766af72d] 2024-05-27T07:29:18.486 INFO:journalctl@ceph.mds.d.smithi052.stdout:May 27 07:29:18 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-d[104512]: 17: (Thread::_entry_func(void*)+0xd) [0x7f52766af749] 2024-05-27T07:29:18.486 INFO:journalctl@ceph.mds.d.smithi052.stdout:May 27 07:29:18 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-d[104512]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7f52752531ca] 2024-05-27T07:29:18.486 INFO:journalctl@ceph.mds.d.smithi052.stdout:May 27 07:29:18 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-d[104512]: 19: clone() 2024-05-27T07:29:18.486 INFO:journalctl@ceph.mds.d.smithi052.stdout:May 27 07:29:18 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-d[104512]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-05-27T07:29:18.486 INFO:journalctl@ceph.mds.d.smithi052.stdout:May 27 07:29:18 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-d[104512]: 2024-05-27T07:29:18.602 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:29:18 smithi173 ceph-mon[37000]: scrub summary: idle 2024-05-27T07:29:18.602 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:29:18 smithi173 ceph-mon[37000]: pgmap v112: 129 pgs: 129 active+clean; 213 MiB data, 1016 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 MiB/s rd, 5.7 MiB/s wr, 1.31k op/s 2024-05-27T07:29:18.602 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:29:18 smithi173 ceph-mon[37000]: osdmap e101: 12 total, 12 up, 12 in 2024-05-27T07:29:18.602 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:29:18 smithi173 ceph-mon[37000]: mds.? [v2:172.21.15.52:6838/4186787748,v1:172.21.15.52:6839/4186787748] up:boot 2024-05-27T07:29:18.602 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:29:18 smithi173 ceph-mon[37000]: MDS daemon mds.k is removed because it is dead or otherwise unavailable. 2024-05-27T07:29:18.602 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:29:18 smithi173 ceph-mon[37000]: fsmap cephfs:5 4 up:standby-replay 3 up:standby 5 up:active 2024-05-27T07:29:18.602 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:29:18 smithi173 ceph-mon[37000]: from='mgr.15333 172.21.15.52:0/416389082' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "d"}]: dispatch 2024-05-27T07:29:18.602 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:29:18 smithi173 ceph-mon[37000]: fsmap cephfs:5 5 up:standby-replay 1 up:standby 5 up:active 2024-05-27T07:29:18.690 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:29:18 smithi105 ceph-mon[35977]: scrub summary: idle 2024-05-27T07:29:18.690 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:29:18 smithi105 ceph-mon[35977]: pgmap v112: 129 pgs: 129 active+clean; 213 MiB data, 1016 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 MiB/s rd, 5.7 MiB/s wr, 1.31k op/s 2024-05-27T07:29:18.690 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:29:18 smithi105 ceph-mon[35977]: osdmap e101: 12 total, 12 up, 12 in 2024-05-27T07:29:18.690 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:29:18 smithi105 ceph-mon[35977]: mds.? [v2:172.21.15.52:6838/4186787748,v1:172.21.15.52:6839/4186787748] up:boot 2024-05-27T07:29:18.690 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:29:18 smithi105 ceph-mon[35977]: MDS daemon mds.k is removed because it is dead or otherwise unavailable. 2024-05-27T07:29:18.690 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:29:18 smithi105 ceph-mon[35977]: fsmap cephfs:5 4 up:standby-replay 3 up:standby 5 up:active 2024-05-27T07:29:18.690 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:29:18 smithi105 ceph-mon[35977]: from='mgr.15333 172.21.15.52:0/416389082' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "d"}]: dispatch 2024-05-27T07:29:18.690 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:29:18 smithi105 ceph-mon[35977]: fsmap cephfs:5 5 up:standby-replay 1 up:standby 5 up:active 2024-05-27T07:29:18.973 INFO:journalctl@ceph.mds.g.smithi052.stdout:May 27 07:29:18 smithi052 podman[104570]: 2024-05-27 07:29:18.645229379 +0000 UTC m=+0.985421744 container remove 881cedb8d2465d95bc3a2989662221d785f4982234a579c062b0e70889cb4b9a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-g, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, GIT_BRANCH=HEAD, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.build-date=20240521, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2) 2024-05-27T07:29:18.973 INFO:journalctl@ceph.mds.g.smithi052.stdout:May 27 07:29:18 smithi052 systemd[1]: ceph-73df2636-1bf8-11ef-bc9b-c7b262605968@mds.g.service: Main process exited, code=exited, status=134/n/a 2024-05-27T07:29:18.976 INFO:journalctl@ceph.mds.d.smithi052.stdout:May 27 07:29:18 smithi052 podman[104609]: 2024-05-27 07:29:18.683669985 +0000 UTC m=+0.512486708 container died 1440a1635f6e4feeff38e9cb90a021107e7444f5c84852004669f8f823b1af19 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-d, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240521, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, GIT_CLEAN=True, io.buildah.version=1.33.5, ceph=True, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD) 2024-05-27T07:29:19.440 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:29:19 smithi105 ceph-mon[35977]: pgmap v114: 129 pgs: 129 active+clean; 198 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 4.7 MiB/s rd, 6.4 MiB/s wr, 1.21k op/s 2024-05-27T07:29:19.440 INFO:journalctl@ceph.mds.k.smithi105.stdout:May 27 07:29:19 smithi105 podman[69637]: 2024-05-27 07:29:19.144053633 +0000 UTC m=+3.812660108 container init 63d34243be402599f9f82855c72ddf43c4fc940256782a2b543ffdcdb8f8c5f6 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-k, org.label-schema.build-date=20240521, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, maintainer=Guillaume Abrioux , ceph=True, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90) 2024-05-27T07:29:19.440 INFO:journalctl@ceph.mds.k.smithi105.stdout:May 27 07:29:19 smithi105 podman[69637]: 2024-05-27 07:29:19.14671383 +0000 UTC m=+3.815320313 container start 63d34243be402599f9f82855c72ddf43c4fc940256782a2b543ffdcdb8f8c5f6 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-k, GIT_CLEAN=True, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, org.label-schema.build-date=20240521, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, CEPH_POINT_RELEASE=) 2024-05-27T07:29:19.440 INFO:journalctl@ceph.mds.k.smithi105.stdout:May 27 07:29:19 smithi105 ceph-mds[69703]: -- 172.21.15.105:0/342358672 <== mon.2 v2:172.21.15.105:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x563e07491a40 con 0x563e074d9600 2024-05-27T07:29:19.440 INFO:journalctl@ceph.mds.k.smithi105.stdout:May 27 07:29:19 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-k[69699]: starting mds.k at 2024-05-27T07:29:19.602 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:29:19 smithi173 ceph-mon[37000]: pgmap v114: 129 pgs: 129 active+clean; 198 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 4.7 MiB/s rd, 6.4 MiB/s wr, 1.21k op/s 2024-05-27T07:29:19.607 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:29:19 smithi052 ceph-mon[31955]: pgmap v114: 129 pgs: 129 active+clean; 198 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 4.7 MiB/s rd, 6.4 MiB/s wr, 1.21k op/s 2024-05-27T07:29:19.608 INFO:journalctl@ceph.mds.d.smithi052.stdout:May 27 07:29:19 smithi052 podman[104609]: 2024-05-27 07:29:19.375526309 +0000 UTC m=+1.204343029 container remove 1440a1635f6e4feeff38e9cb90a021107e7444f5c84852004669f8f823b1af19 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-d, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, ceph=True, org.label-schema.schema-version=1.0, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, org.label-schema.build-date=20240521) 2024-05-27T07:29:19.608 INFO:journalctl@ceph.mds.d.smithi052.stdout:May 27 07:29:19 smithi052 systemd[1]: ceph-73df2636-1bf8-11ef-bc9b-c7b262605968@mds.d.service: Main process exited, code=exited, status=134/n/a 2024-05-27T07:29:19.902 INFO:journalctl@ceph.mds.a.smithi052.stdout:May 27 07:29:19 smithi052 podman[104638]: 2024-05-27 07:29:19.526071926 +0000 UTC m=+0.083325495 image pull e36ea4fc8a5807e8d2701370e350a0de3856538797f7fba64cfce7e344bd6799 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6 2024-05-27T07:29:19.902 INFO:journalctl@ceph.mds.a.smithi052.stdout:May 27 07:29:19 smithi052 podman[104638]: 2024-05-27 07:29:19.626358665 +0000 UTC m=+0.183612215 container create 46c806e123bbc1bcf4df0fc9657e67eba98bcf204f960943c2efea78585f35c4 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-a, maintainer=Guillaume Abrioux , ceph=True, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, org.label-schema.build-date=20240521, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True) 2024-05-27T07:29:19.902 INFO:journalctl@ceph.mds.a.smithi052.stdout:May 27 07:29:19 smithi052 podman[104638]: 2024-05-27 07:29:19.91274105 +0000 UTC m=+0.469994612 container init 46c806e123bbc1bcf4df0fc9657e67eba98bcf204f960943c2efea78585f35c4 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-a, org.label-schema.license=GPLv2, ceph=True, GIT_CLEAN=True, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240521, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-05-27T07:29:20.229 INFO:journalctl@ceph.mds.a.smithi052.stdout:May 27 07:29:19 smithi052 podman[104638]: 2024-05-27 07:29:19.919365717 +0000 UTC m=+0.476619276 container start 46c806e123bbc1bcf4df0fc9657e67eba98bcf204f960943c2efea78585f35c4 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-a, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240521, org.label-schema.schema-version=1.0, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, ceph=True, org.label-schema.license=GPLv2, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0) 2024-05-27T07:29:20.230 INFO:journalctl@ceph.mds.a.smithi052.stdout:May 27 07:29:20 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-a[104660]: starting mds.a at 2024-05-27T07:29:20.230 INFO:journalctl@ceph.mds.a.smithi052.stdout:May 27 07:29:20 smithi052 bash[104638]: 46c806e123bbc1bcf4df0fc9657e67eba98bcf204f960943c2efea78585f35c4 2024-05-27T07:29:20.230 INFO:journalctl@ceph.mds.a.smithi052.stdout:May 27 07:29:20 smithi052 systemd[1]: Started Ceph mds.a for 73df2636-1bf8-11ef-bc9b-c7b262605968. 2024-05-27T07:29:20.602 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:29:20 smithi173 ceph-mon[37000]: mds.? [v2:172.21.15.105:6832/1948236635,v1:172.21.15.105:6833/1948236635] up:boot 2024-05-27T07:29:20.602 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:29:20 smithi173 ceph-mon[37000]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-05-27T07:29:20.602 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:29:20 smithi173 ceph-mon[37000]: from='mgr.15333 172.21.15.52:0/416389082' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "k"}]: dispatch 2024-05-27T07:29:20.602 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:29:20 smithi173 ceph-mon[37000]: daemon mds.a restarted 2024-05-27T07:29:20.690 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:29:20 smithi105 ceph-mon[35977]: mds.? [v2:172.21.15.105:6832/1948236635,v1:172.21.15.105:6833/1948236635] up:boot 2024-05-27T07:29:20.690 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:29:20 smithi105 ceph-mon[35977]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-05-27T07:29:20.690 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:29:20 smithi105 ceph-mon[35977]: from='mgr.15333 172.21.15.52:0/416389082' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "k"}]: dispatch 2024-05-27T07:29:20.690 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:29:20 smithi105 ceph-mon[35977]: daemon mds.a restarted 2024-05-27T07:29:20.691 INFO:journalctl@ceph.mds.k.smithi105.stdout:May 27 07:29:20 smithi105 bash[69637]: 63d34243be402599f9f82855c72ddf43c4fc940256782a2b543ffdcdb8f8c5f6 2024-05-27T07:29:20.691 INFO:journalctl@ceph.mds.k.smithi105.stdout:May 27 07:29:20 smithi105 systemd[1]: Started Ceph mds.k for 73df2636-1bf8-11ef-bc9b-c7b262605968. 2024-05-27T07:29:20.691 INFO:journalctl@ceph.mds.k.smithi105.stdout:May 27 07:29:20 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-k[69699]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: In function 'std::set SnapClient::filter(const std::set&) const' thread 7f7c6121e700 time 2024-05-27T07:29:20.466140+0000 2024-05-27T07:29:20.691 INFO:journalctl@ceph.mds.k.smithi105.stdout:May 27 07:29:20 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-k[69699]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: 197: FAILED ceph_assert(cached_version > 0) 2024-05-27T07:29:20.691 INFO:journalctl@ceph.mds.k.smithi105.stdout:May 27 07:29:20 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-k[69699]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:29:20.691 INFO:journalctl@ceph.mds.k.smithi105.stdout:May 27 07:29:20 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-k[69699]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x127) [0x7f7c73a63ba6] 2024-05-27T07:29:20.691 INFO:journalctl@ceph.mds.k.smithi105.stdout:May 27 07:29:20 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-k[69699]: 2: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7f7c73a63dd1] 2024-05-27T07:29:20.691 INFO:journalctl@ceph.mds.k.smithi105.stdout:May 27 07:29:20 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-k[69699]: 3: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x563e04c8123d] 2024-05-27T07:29:20.691 INFO:journalctl@ceph.mds.k.smithi105.stdout:May 27 07:29:20 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-k[69699]: 4: (SnapRealm::build_snap_set() const+0x323) [0x563e04c6c971] 2024-05-27T07:29:20.691 INFO:journalctl@ceph.mds.k.smithi105.stdout:May 27 07:29:20 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-k[69699]: 5: (SnapRealm::check_cache() const+0x13e) [0x563e04c6afa0] 2024-05-27T07:29:20.692 INFO:journalctl@ceph.mds.k.smithi105.stdout:May 27 07:29:20 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-k[69699]: 6: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x563e04c6b651] 2024-05-27T07:29:20.692 INFO:journalctl@ceph.mds.k.smithi105.stdout:May 27 07:29:20 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-k[69699]: 7: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x563e04c00347] 2024-05-27T07:29:20.692 INFO:journalctl@ceph.mds.k.smithi105.stdout:May 27 07:29:20 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-k[69699]: 8: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x563e048fc87e] 2024-05-27T07:29:20.692 INFO:journalctl@ceph.mds.k.smithi105.stdout:May 27 07:29:20 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-k[69699]: 9: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x2702) [0x563e04901e94] 2024-05-27T07:29:20.692 INFO:journalctl@ceph.mds.k.smithi105.stdout:May 27 07:29:20 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-k[69699]: 10: (EUpdate::replay(MDSRank*)+0x63) [0x563e0490c969] 2024-05-27T07:29:20.692 INFO:journalctl@ceph.mds.k.smithi105.stdout:May 27 07:29:20 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-k[69699]: 11: (MDLog::_replay_thread()+0x1f38) [0x563e04ca548c] 2024-05-27T07:29:20.692 INFO:journalctl@ceph.mds.k.smithi105.stdout:May 27 07:29:20 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-k[69699]: 12: (MDLog::ReplayThread::entry()+0x11) [0x563e04cae9cb] 2024-05-27T07:29:20.692 INFO:journalctl@ceph.mds.k.smithi105.stdout:May 27 07:29:20 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-k[69699]: 13: (Thread::entry_wrapper()+0x43) [0x7f7c73a3272d] 2024-05-27T07:29:20.692 INFO:journalctl@ceph.mds.k.smithi105.stdout:May 27 07:29:20 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-k[69699]: 14: (Thread::_entry_func(void*)+0xd) [0x7f7c73a32749] 2024-05-27T07:29:20.692 INFO:journalctl@ceph.mds.k.smithi105.stdout:May 27 07:29:20 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-k[69699]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7f7c725d61ca] 2024-05-27T07:29:20.692 INFO:journalctl@ceph.mds.k.smithi105.stdout:May 27 07:29:20 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-k[69699]: 16: clone() 2024-05-27T07:29:20.693 INFO:journalctl@ceph.mds.k.smithi105.stdout:May 27 07:29:20 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-k[69699]: 2024-05-27T07:29:20.468+0000 7f7c6121e700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: In function 'std::set SnapClient::filter(const std::set&) const' thread 7f7c6121e700 time 2024-05-27T07:29:20.466140+0000 2024-05-27T07:29:20.693 INFO:journalctl@ceph.mds.k.smithi105.stdout:May 27 07:29:20 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-k[69699]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: 197: FAILED ceph_assert(cached_version > 0) 2024-05-27T07:29:20.693 INFO:journalctl@ceph.mds.k.smithi105.stdout:May 27 07:29:20 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-k[69699]: 2024-05-27T07:29:20.693 INFO:journalctl@ceph.mds.k.smithi105.stdout:May 27 07:29:20 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-k[69699]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:29:20.693 INFO:journalctl@ceph.mds.k.smithi105.stdout:May 27 07:29:20 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-k[69699]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x127) [0x7f7c73a63ba6] 2024-05-27T07:29:20.693 INFO:journalctl@ceph.mds.k.smithi105.stdout:May 27 07:29:20 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-k[69699]: 2: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7f7c73a63dd1] 2024-05-27T07:29:20.693 INFO:journalctl@ceph.mds.k.smithi105.stdout:May 27 07:29:20 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-k[69699]: 3: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x563e04c8123d] 2024-05-27T07:29:20.693 INFO:journalctl@ceph.mds.k.smithi105.stdout:May 27 07:29:20 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-k[69699]: 4: (SnapRealm::build_snap_set() const+0x323) [0x563e04c6c971] 2024-05-27T07:29:20.693 INFO:journalctl@ceph.mds.k.smithi105.stdout:May 27 07:29:20 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-k[69699]: 5: (SnapRealm::check_cache() const+0x13e) [0x563e04c6afa0] 2024-05-27T07:29:20.693 INFO:journalctl@ceph.mds.k.smithi105.stdout:May 27 07:29:20 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-k[69699]: 6: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x563e04c6b651] 2024-05-27T07:29:20.693 INFO:journalctl@ceph.mds.k.smithi105.stdout:May 27 07:29:20 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-k[69699]: 7: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x563e04c00347] 2024-05-27T07:29:20.693 INFO:journalctl@ceph.mds.k.smithi105.stdout:May 27 07:29:20 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-k[69699]: 8: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x563e048fc87e] 2024-05-27T07:29:20.693 INFO:journalctl@ceph.mds.k.smithi105.stdout:May 27 07:29:20 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-k[69699]: 9: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x2702) [0x563e04901e94] 2024-05-27T07:29:20.694 INFO:journalctl@ceph.mds.k.smithi105.stdout:May 27 07:29:20 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-k[69699]: 10: (EUpdate::replay(MDSRank*)+0x63) [0x563e0490c969] 2024-05-27T07:29:20.694 INFO:journalctl@ceph.mds.k.smithi105.stdout:May 27 07:29:20 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-k[69699]: 11: (MDLog::_replay_thread()+0x1f38) [0x563e04ca548c] 2024-05-27T07:29:20.694 INFO:journalctl@ceph.mds.k.smithi105.stdout:May 27 07:29:20 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-k[69699]: 12: (MDLog::ReplayThread::entry()+0x11) [0x563e04cae9cb] 2024-05-27T07:29:20.694 INFO:journalctl@ceph.mds.k.smithi105.stdout:May 27 07:29:20 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-k[69699]: 13: (Thread::entry_wrapper()+0x43) [0x7f7c73a3272d] 2024-05-27T07:29:20.694 INFO:journalctl@ceph.mds.k.smithi105.stdout:May 27 07:29:20 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-k[69699]: 14: (Thread::_entry_func(void*)+0xd) [0x7f7c73a32749] 2024-05-27T07:29:20.694 INFO:journalctl@ceph.mds.k.smithi105.stdout:May 27 07:29:20 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-k[69699]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7f7c725d61ca] 2024-05-27T07:29:20.694 INFO:journalctl@ceph.mds.k.smithi105.stdout:May 27 07:29:20 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-k[69699]: 16: clone() 2024-05-27T07:29:20.694 INFO:journalctl@ceph.mds.k.smithi105.stdout:May 27 07:29:20 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-k[69699]: 2024-05-27T07:29:20.694 INFO:journalctl@ceph.mds.k.smithi105.stdout:May 27 07:29:20 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-k[69699]: *** Caught signal (Aborted) ** 2024-05-27T07:29:20.694 INFO:journalctl@ceph.mds.k.smithi105.stdout:May 27 07:29:20 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-k[69699]: in thread 7f7c6121e700 thread_name:md_log_replay 2024-05-27T07:29:20.694 INFO:journalctl@ceph.mds.k.smithi105.stdout:May 27 07:29:20 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-k[69699]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:29:20.694 INFO:journalctl@ceph.mds.k.smithi105.stdout:May 27 07:29:20 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-k[69699]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f7c725e0d20] 2024-05-27T07:29:20.695 INFO:journalctl@ceph.mds.k.smithi105.stdout:May 27 07:29:20 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-k[69699]: 2: gsignal() 2024-05-27T07:29:20.695 INFO:journalctl@ceph.mds.k.smithi105.stdout:May 27 07:29:20 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-k[69699]: 3: abort() 2024-05-27T07:29:20.695 INFO:journalctl@ceph.mds.k.smithi105.stdout:May 27 07:29:20 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-k[69699]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x26e) [0x7f7c73a63ced] 2024-05-27T07:29:20.695 INFO:journalctl@ceph.mds.k.smithi105.stdout:May 27 07:29:20 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-k[69699]: 5: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7f7c73a63dd1] 2024-05-27T07:29:20.696 INFO:journalctl@ceph.mds.k.smithi105.stdout:May 27 07:29:20 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-k[69699]: 6: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x563e04c8123d] 2024-05-27T07:29:20.696 INFO:journalctl@ceph.mds.k.smithi105.stdout:May 27 07:29:20 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-k[69699]: 7: (SnapRealm::build_snap_set() const+0x323) [0x563e04c6c971] 2024-05-27T07:29:20.696 INFO:journalctl@ceph.mds.k.smithi105.stdout:May 27 07:29:20 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-k[69699]: 8: (SnapRealm::check_cache() const+0x13e) [0x563e04c6afa0] 2024-05-27T07:29:20.696 INFO:journalctl@ceph.mds.k.smithi105.stdout:May 27 07:29:20 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-k[69699]: 9: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x563e04c6b651] 2024-05-27T07:29:20.696 INFO:journalctl@ceph.mds.k.smithi105.stdout:May 27 07:29:20 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-k[69699]: 10: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x563e04c00347] 2024-05-27T07:29:20.696 INFO:journalctl@ceph.mds.k.smithi105.stdout:May 27 07:29:20 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-k[69699]: 11: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x563e048fc87e] 2024-05-27T07:29:20.696 INFO:journalctl@ceph.mds.k.smithi105.stdout:May 27 07:29:20 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-k[69699]: 12: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x2702) [0x563e04901e94] 2024-05-27T07:29:20.696 INFO:journalctl@ceph.mds.k.smithi105.stdout:May 27 07:29:20 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-k[69699]: 13: (EUpdate::replay(MDSRank*)+0x63) [0x563e0490c969] 2024-05-27T07:29:20.696 INFO:journalctl@ceph.mds.k.smithi105.stdout:May 27 07:29:20 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-k[69699]: 14: (MDLog::_replay_thread()+0x1f38) [0x563e04ca548c] 2024-05-27T07:29:20.696 INFO:journalctl@ceph.mds.k.smithi105.stdout:May 27 07:29:20 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-k[69699]: 15: (MDLog::ReplayThread::entry()+0x11) [0x563e04cae9cb] 2024-05-27T07:29:20.697 INFO:journalctl@ceph.mds.k.smithi105.stdout:May 27 07:29:20 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-k[69699]: 16: (Thread::entry_wrapper()+0x43) [0x7f7c73a3272d] 2024-05-27T07:29:20.697 INFO:journalctl@ceph.mds.k.smithi105.stdout:May 27 07:29:20 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-k[69699]: 17: (Thread::_entry_func(void*)+0xd) [0x7f7c73a32749] 2024-05-27T07:29:20.697 INFO:journalctl@ceph.mds.k.smithi105.stdout:May 27 07:29:20 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-k[69699]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7f7c725d61ca] 2024-05-27T07:29:20.697 INFO:journalctl@ceph.mds.k.smithi105.stdout:May 27 07:29:20 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-k[69699]: 19: clone() 2024-05-27T07:29:20.697 INFO:journalctl@ceph.mds.k.smithi105.stdout:May 27 07:29:20 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-k[69699]: 2024-05-27T07:29:20.471+0000 7f7c6121e700 -1 *** Caught signal (Aborted) ** 2024-05-27T07:29:20.697 INFO:journalctl@ceph.mds.k.smithi105.stdout:May 27 07:29:20 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-k[69699]: in thread 7f7c6121e700 thread_name:md_log_replay 2024-05-27T07:29:20.697 INFO:journalctl@ceph.mds.k.smithi105.stdout:May 27 07:29:20 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-k[69699]: 2024-05-27T07:29:20.697 INFO:journalctl@ceph.mds.k.smithi105.stdout:May 27 07:29:20 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-k[69699]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:29:20.697 INFO:journalctl@ceph.mds.k.smithi105.stdout:May 27 07:29:20 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-k[69699]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f7c725e0d20] 2024-05-27T07:29:20.697 INFO:journalctl@ceph.mds.k.smithi105.stdout:May 27 07:29:20 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-k[69699]: 2: gsignal() 2024-05-27T07:29:20.697 INFO:journalctl@ceph.mds.k.smithi105.stdout:May 27 07:29:20 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-k[69699]: 3: abort() 2024-05-27T07:29:20.697 INFO:journalctl@ceph.mds.k.smithi105.stdout:May 27 07:29:20 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-k[69699]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x26e) [0x7f7c73a63ced] 2024-05-27T07:29:20.697 INFO:journalctl@ceph.mds.k.smithi105.stdout:May 27 07:29:20 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-k[69699]: 5: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7f7c73a63dd1] 2024-05-27T07:29:20.698 INFO:journalctl@ceph.mds.k.smithi105.stdout:May 27 07:29:20 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-k[69699]: 6: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x563e04c8123d] 2024-05-27T07:29:20.698 INFO:journalctl@ceph.mds.k.smithi105.stdout:May 27 07:29:20 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-k[69699]: 7: (SnapRealm::build_snap_set() const+0x323) [0x563e04c6c971] 2024-05-27T07:29:20.698 INFO:journalctl@ceph.mds.k.smithi105.stdout:May 27 07:29:20 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-k[69699]: 8: (SnapRealm::check_cache() const+0x13e) [0x563e04c6afa0] 2024-05-27T07:29:20.698 INFO:journalctl@ceph.mds.k.smithi105.stdout:May 27 07:29:20 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-k[69699]: 9: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x563e04c6b651] 2024-05-27T07:29:20.698 INFO:journalctl@ceph.mds.k.smithi105.stdout:May 27 07:29:20 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-k[69699]: 10: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x563e04c00347] 2024-05-27T07:29:20.698 INFO:journalctl@ceph.mds.k.smithi105.stdout:May 27 07:29:20 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-k[69699]: 11: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x563e048fc87e] 2024-05-27T07:29:20.698 INFO:journalctl@ceph.mds.k.smithi105.stdout:May 27 07:29:20 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-k[69699]: 12: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x2702) [0x563e04901e94] 2024-05-27T07:29:20.698 INFO:journalctl@ceph.mds.k.smithi105.stdout:May 27 07:29:20 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-k[69699]: 13: (EUpdate::replay(MDSRank*)+0x63) [0x563e0490c969] 2024-05-27T07:29:20.698 INFO:journalctl@ceph.mds.k.smithi105.stdout:May 27 07:29:20 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-k[69699]: 14: (MDLog::_replay_thread()+0x1f38) [0x563e04ca548c] 2024-05-27T07:29:20.698 INFO:journalctl@ceph.mds.k.smithi105.stdout:May 27 07:29:20 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-k[69699]: 15: (MDLog::ReplayThread::entry()+0x11) [0x563e04cae9cb] 2024-05-27T07:29:20.698 INFO:journalctl@ceph.mds.k.smithi105.stdout:May 27 07:29:20 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-k[69699]: 16: (Thread::entry_wrapper()+0x43) [0x7f7c73a3272d] 2024-05-27T07:29:20.698 INFO:journalctl@ceph.mds.k.smithi105.stdout:May 27 07:29:20 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-k[69699]: 17: (Thread::_entry_func(void*)+0xd) [0x7f7c73a32749] 2024-05-27T07:29:20.699 INFO:journalctl@ceph.mds.k.smithi105.stdout:May 27 07:29:20 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-k[69699]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7f7c725d61ca] 2024-05-27T07:29:20.699 INFO:journalctl@ceph.mds.k.smithi105.stdout:May 27 07:29:20 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-k[69699]: 19: clone() 2024-05-27T07:29:20.699 INFO:journalctl@ceph.mds.k.smithi105.stdout:May 27 07:29:20 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-k[69699]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-05-27T07:29:20.699 INFO:journalctl@ceph.mds.k.smithi105.stdout:May 27 07:29:20 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-k[69699]: 2024-05-27T07:29:20.699 INFO:journalctl@ceph.mds.k.smithi105.stdout:May 27 07:29:20 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-k[69699]: -1> 2024-05-27T07:29:20.468+0000 7f7c6121e700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: In function 'std::set SnapClient::filter(const std::set&) const' thread 7f7c6121e700 time 2024-05-27T07:29:20.466140+0000 2024-05-27T07:29:20.699 INFO:journalctl@ceph.mds.k.smithi105.stdout:May 27 07:29:20 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-k[69699]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: 197: FAILED ceph_assert(cached_version > 0) 2024-05-27T07:29:20.699 INFO:journalctl@ceph.mds.k.smithi105.stdout:May 27 07:29:20 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-k[69699]: 2024-05-27T07:29:20.699 INFO:journalctl@ceph.mds.k.smithi105.stdout:May 27 07:29:20 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-k[69699]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:29:20.699 INFO:journalctl@ceph.mds.k.smithi105.stdout:May 27 07:29:20 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-k[69699]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x127) [0x7f7c73a63ba6] 2024-05-27T07:29:20.699 INFO:journalctl@ceph.mds.k.smithi105.stdout:May 27 07:29:20 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-k[69699]: 2: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7f7c73a63dd1] 2024-05-27T07:29:20.700 INFO:journalctl@ceph.mds.k.smithi105.stdout:May 27 07:29:20 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-k[69699]: 3: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x563e04c8123d] 2024-05-27T07:29:20.700 INFO:journalctl@ceph.mds.k.smithi105.stdout:May 27 07:29:20 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-k[69699]: 4: (SnapRealm::build_snap_set() const+0x323) [0x563e04c6c971] 2024-05-27T07:29:20.700 INFO:journalctl@ceph.mds.k.smithi105.stdout:May 27 07:29:20 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-k[69699]: 5: (SnapRealm::check_cache() const+0x13e) [0x563e04c6afa0] 2024-05-27T07:29:20.700 INFO:journalctl@ceph.mds.k.smithi105.stdout:May 27 07:29:20 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-k[69699]: 6: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x563e04c6b651] 2024-05-27T07:29:20.700 INFO:journalctl@ceph.mds.k.smithi105.stdout:May 27 07:29:20 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-k[69699]: 7: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x563e04c00347] 2024-05-27T07:29:20.700 INFO:journalctl@ceph.mds.k.smithi105.stdout:May 27 07:29:20 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-k[69699]: 8: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x563e048fc87e] 2024-05-27T07:29:20.700 INFO:journalctl@ceph.mds.k.smithi105.stdout:May 27 07:29:20 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-k[69699]: 9: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x2702) [0x563e04901e94] 2024-05-27T07:29:20.700 INFO:journalctl@ceph.mds.k.smithi105.stdout:May 27 07:29:20 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-k[69699]: 10: (EUpdate::replay(MDSRank*)+0x63) [0x563e0490c969] 2024-05-27T07:29:20.700 INFO:journalctl@ceph.mds.k.smithi105.stdout:May 27 07:29:20 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-k[69699]: 11: (MDLog::_replay_thread()+0x1f38) [0x563e04ca548c] 2024-05-27T07:29:20.700 INFO:journalctl@ceph.mds.k.smithi105.stdout:May 27 07:29:20 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-k[69699]: 12: (MDLog::ReplayThread::entry()+0x11) [0x563e04cae9cb] 2024-05-27T07:29:20.700 INFO:journalctl@ceph.mds.k.smithi105.stdout:May 27 07:29:20 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-k[69699]: 13: (Thread::entry_wrapper()+0x43) [0x7f7c73a3272d] 2024-05-27T07:29:20.700 INFO:journalctl@ceph.mds.k.smithi105.stdout:May 27 07:29:20 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-k[69699]: 14: (Thread::_entry_func(void*)+0xd) [0x7f7c73a32749] 2024-05-27T07:29:20.700 INFO:journalctl@ceph.mds.k.smithi105.stdout:May 27 07:29:20 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-k[69699]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7f7c725d61ca] 2024-05-27T07:29:20.701 INFO:journalctl@ceph.mds.k.smithi105.stdout:May 27 07:29:20 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-k[69699]: 16: clone() 2024-05-27T07:29:20.701 INFO:journalctl@ceph.mds.k.smithi105.stdout:May 27 07:29:20 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-k[69699]: 2024-05-27T07:29:20.701 INFO:journalctl@ceph.mds.k.smithi105.stdout:May 27 07:29:20 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-k[69699]: 0> 2024-05-27T07:29:20.471+0000 7f7c6121e700 -1 *** Caught signal (Aborted) ** 2024-05-27T07:29:20.701 INFO:journalctl@ceph.mds.k.smithi105.stdout:May 27 07:29:20 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-k[69699]: in thread 7f7c6121e700 thread_name:md_log_replay 2024-05-27T07:29:20.701 INFO:journalctl@ceph.mds.k.smithi105.stdout:May 27 07:29:20 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-k[69699]: 2024-05-27T07:29:20.701 INFO:journalctl@ceph.mds.k.smithi105.stdout:May 27 07:29:20 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-k[69699]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:29:20.701 INFO:journalctl@ceph.mds.k.smithi105.stdout:May 27 07:29:20 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-k[69699]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f7c725e0d20] 2024-05-27T07:29:20.701 INFO:journalctl@ceph.mds.k.smithi105.stdout:May 27 07:29:20 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-k[69699]: 2: gsignal() 2024-05-27T07:29:20.701 INFO:journalctl@ceph.mds.k.smithi105.stdout:May 27 07:29:20 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-k[69699]: 3: abort() 2024-05-27T07:29:20.701 INFO:journalctl@ceph.mds.k.smithi105.stdout:May 27 07:29:20 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-k[69699]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x26e) [0x7f7c73a63ced] 2024-05-27T07:29:20.701 INFO:journalctl@ceph.mds.k.smithi105.stdout:May 27 07:29:20 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-k[69699]: 5: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7f7c73a63dd1] 2024-05-27T07:29:20.701 INFO:journalctl@ceph.mds.k.smithi105.stdout:May 27 07:29:20 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-k[69699]: 6: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x563e04c8123d] 2024-05-27T07:29:20.702 INFO:journalctl@ceph.mds.k.smithi105.stdout:May 27 07:29:20 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-k[69699]: 7: (SnapRealm::build_snap_set() const+0x323) [0x563e04c6c971] 2024-05-27T07:29:20.702 INFO:journalctl@ceph.mds.k.smithi105.stdout:May 27 07:29:20 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-k[69699]: 8: (SnapRealm::check_cache() const+0x13e) [0x563e04c6afa0] 2024-05-27T07:29:20.702 INFO:journalctl@ceph.mds.k.smithi105.stdout:May 27 07:29:20 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-k[69699]: 9: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x563e04c6b651] 2024-05-27T07:29:20.702 INFO:journalctl@ceph.mds.k.smithi105.stdout:May 27 07:29:20 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-k[69699]: 10: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x563e04c00347] 2024-05-27T07:29:20.702 INFO:journalctl@ceph.mds.k.smithi105.stdout:May 27 07:29:20 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-k[69699]: 11: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x563e048fc87e] 2024-05-27T07:29:20.702 INFO:journalctl@ceph.mds.k.smithi105.stdout:May 27 07:29:20 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-k[69699]: 12: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x2702) [0x563e04901e94] 2024-05-27T07:29:20.702 INFO:journalctl@ceph.mds.k.smithi105.stdout:May 27 07:29:20 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-k[69699]: 13: (EUpdate::replay(MDSRank*)+0x63) [0x563e0490c969] 2024-05-27T07:29:20.702 INFO:journalctl@ceph.mds.k.smithi105.stdout:May 27 07:29:20 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-k[69699]: 14: (MDLog::_replay_thread()+0x1f38) [0x563e04ca548c] 2024-05-27T07:29:20.702 INFO:journalctl@ceph.mds.k.smithi105.stdout:May 27 07:29:20 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-k[69699]: 15: (MDLog::ReplayThread::entry()+0x11) [0x563e04cae9cb] 2024-05-27T07:29:20.702 INFO:journalctl@ceph.mds.k.smithi105.stdout:May 27 07:29:20 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-k[69699]: 16: (Thread::entry_wrapper()+0x43) [0x7f7c73a3272d] 2024-05-27T07:29:20.702 INFO:journalctl@ceph.mds.k.smithi105.stdout:May 27 07:29:20 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-k[69699]: 17: (Thread::_entry_func(void*)+0xd) [0x7f7c73a32749] 2024-05-27T07:29:20.702 INFO:journalctl@ceph.mds.k.smithi105.stdout:May 27 07:29:20 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-k[69699]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7f7c725d61ca] 2024-05-27T07:29:20.703 INFO:journalctl@ceph.mds.k.smithi105.stdout:May 27 07:29:20 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-k[69699]: 19: clone() 2024-05-27T07:29:20.703 INFO:journalctl@ceph.mds.k.smithi105.stdout:May 27 07:29:20 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-k[69699]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-05-27T07:29:20.703 INFO:journalctl@ceph.mds.k.smithi105.stdout:May 27 07:29:20 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-k[69699]: 2024-05-27T07:29:20.703 INFO:journalctl@ceph.mds.k.smithi105.stdout:May 27 07:29:20 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-k[69699]: -1> 2024-05-27T07:29:20.468+0000 7f7c6121e700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: In function 'std::set SnapClient::filter(const std::set&) const' thread 7f7c6121e700 time 2024-05-27T07:29:20.466140+0000 2024-05-27T07:29:20.703 INFO:journalctl@ceph.mds.k.smithi105.stdout:May 27 07:29:20 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-k[69699]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: 197: FAILED ceph_assert(cached_version > 0) 2024-05-27T07:29:20.703 INFO:journalctl@ceph.mds.k.smithi105.stdout:May 27 07:29:20 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-k[69699]: 2024-05-27T07:29:20.703 INFO:journalctl@ceph.mds.k.smithi105.stdout:May 27 07:29:20 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-k[69699]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:29:20.703 INFO:journalctl@ceph.mds.k.smithi105.stdout:May 27 07:29:20 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-k[69699]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x127) [0x7f7c73a63ba6] 2024-05-27T07:29:20.703 INFO:journalctl@ceph.mds.k.smithi105.stdout:May 27 07:29:20 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-k[69699]: 2: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7f7c73a63dd1] 2024-05-27T07:29:20.703 INFO:journalctl@ceph.mds.k.smithi105.stdout:May 27 07:29:20 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-k[69699]: 3: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x563e04c8123d] 2024-05-27T07:29:20.703 INFO:journalctl@ceph.mds.k.smithi105.stdout:May 27 07:29:20 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-k[69699]: 4: (SnapRealm::build_snap_set() const+0x323) [0x563e04c6c971] 2024-05-27T07:29:20.703 INFO:journalctl@ceph.mds.k.smithi105.stdout:May 27 07:29:20 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-k[69699]: 5: (SnapRealm::check_cache() const+0x13e) [0x563e04c6afa0] 2024-05-27T07:29:20.703 INFO:journalctl@ceph.mds.k.smithi105.stdout:May 27 07:29:20 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-k[69699]: 6: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x563e04c6b651] 2024-05-27T07:29:20.703 INFO:journalctl@ceph.mds.k.smithi105.stdout:May 27 07:29:20 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-k[69699]: 7: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x563e04c00347] 2024-05-27T07:29:20.704 INFO:journalctl@ceph.mds.k.smithi105.stdout:May 27 07:29:20 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-k[69699]: 8: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x563e048fc87e] 2024-05-27T07:29:20.704 INFO:journalctl@ceph.mds.k.smithi105.stdout:May 27 07:29:20 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-k[69699]: 9: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x2702) [0x563e04901e94] 2024-05-27T07:29:20.704 INFO:journalctl@ceph.mds.k.smithi105.stdout:May 27 07:29:20 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-k[69699]: 10: (EUpdate::replay(MDSRank*)+0x63) [0x563e0490c969] 2024-05-27T07:29:20.704 INFO:journalctl@ceph.mds.k.smithi105.stdout:May 27 07:29:20 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-k[69699]: 11: (MDLog::_replay_thread()+0x1f38) [0x563e04ca548c] 2024-05-27T07:29:20.704 INFO:journalctl@ceph.mds.k.smithi105.stdout:May 27 07:29:20 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-k[69699]: 12: (MDLog::ReplayThread::entry()+0x11) [0x563e04cae9cb] 2024-05-27T07:29:20.704 INFO:journalctl@ceph.mds.k.smithi105.stdout:May 27 07:29:20 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-k[69699]: 13: (Thread::entry_wrapper()+0x43) [0x7f7c73a3272d] 2024-05-27T07:29:20.704 INFO:journalctl@ceph.mds.k.smithi105.stdout:May 27 07:29:20 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-k[69699]: 14: (Thread::_entry_func(void*)+0xd) [0x7f7c73a32749] 2024-05-27T07:29:20.704 INFO:journalctl@ceph.mds.k.smithi105.stdout:May 27 07:29:20 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-k[69699]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7f7c725d61ca] 2024-05-27T07:29:20.704 INFO:journalctl@ceph.mds.k.smithi105.stdout:May 27 07:29:20 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-k[69699]: 16: clone() 2024-05-27T07:29:20.704 INFO:journalctl@ceph.mds.k.smithi105.stdout:May 27 07:29:20 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-k[69699]: 2024-05-27T07:29:20.704 INFO:journalctl@ceph.mds.k.smithi105.stdout:May 27 07:29:20 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-k[69699]: 0> 2024-05-27T07:29:20.471+0000 7f7c6121e700 -1 *** Caught signal (Aborted) ** 2024-05-27T07:29:20.704 INFO:journalctl@ceph.mds.k.smithi105.stdout:May 27 07:29:20 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-k[69699]: in thread 7f7c6121e700 thread_name:md_log_replay 2024-05-27T07:29:20.704 INFO:journalctl@ceph.mds.k.smithi105.stdout:May 27 07:29:20 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-k[69699]: 2024-05-27T07:29:20.705 INFO:journalctl@ceph.mds.k.smithi105.stdout:May 27 07:29:20 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-k[69699]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:29:20.705 INFO:journalctl@ceph.mds.k.smithi105.stdout:May 27 07:29:20 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-k[69699]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f7c725e0d20] 2024-05-27T07:29:20.705 INFO:journalctl@ceph.mds.k.smithi105.stdout:May 27 07:29:20 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-k[69699]: 2: gsignal() 2024-05-27T07:29:20.705 INFO:journalctl@ceph.mds.k.smithi105.stdout:May 27 07:29:20 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-k[69699]: 3: abort() 2024-05-27T07:29:20.705 INFO:journalctl@ceph.mds.k.smithi105.stdout:May 27 07:29:20 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-k[69699]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x26e) [0x7f7c73a63ced] 2024-05-27T07:29:20.705 INFO:journalctl@ceph.mds.k.smithi105.stdout:May 27 07:29:20 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-k[69699]: 5: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7f7c73a63dd1] 2024-05-27T07:29:20.705 INFO:journalctl@ceph.mds.k.smithi105.stdout:May 27 07:29:20 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-k[69699]: 6: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x563e04c8123d] 2024-05-27T07:29:20.705 INFO:journalctl@ceph.mds.k.smithi105.stdout:May 27 07:29:20 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-k[69699]: 7: (SnapRealm::build_snap_set() const+0x323) [0x563e04c6c971] 2024-05-27T07:29:20.705 INFO:journalctl@ceph.mds.k.smithi105.stdout:May 27 07:29:20 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-k[69699]: 8: (SnapRealm::check_cache() const+0x13e) [0x563e04c6afa0] 2024-05-27T07:29:20.705 INFO:journalctl@ceph.mds.k.smithi105.stdout:May 27 07:29:20 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-k[69699]: 9: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x563e04c6b651] 2024-05-27T07:29:20.705 INFO:journalctl@ceph.mds.k.smithi105.stdout:May 27 07:29:20 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-k[69699]: 10: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x563e04c00347] 2024-05-27T07:29:20.705 INFO:journalctl@ceph.mds.k.smithi105.stdout:May 27 07:29:20 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-k[69699]: 11: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x563e048fc87e] 2024-05-27T07:29:20.705 INFO:journalctl@ceph.mds.k.smithi105.stdout:May 27 07:29:20 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-k[69699]: 12: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x2702) [0x563e04901e94] 2024-05-27T07:29:20.705 INFO:journalctl@ceph.mds.k.smithi105.stdout:May 27 07:29:20 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-k[69699]: 13: (EUpdate::replay(MDSRank*)+0x63) [0x563e0490c969] 2024-05-27T07:29:20.706 INFO:journalctl@ceph.mds.k.smithi105.stdout:May 27 07:29:20 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-k[69699]: 14: (MDLog::_replay_thread()+0x1f38) [0x563e04ca548c] 2024-05-27T07:29:20.706 INFO:journalctl@ceph.mds.k.smithi105.stdout:May 27 07:29:20 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-k[69699]: 15: (MDLog::ReplayThread::entry()+0x11) [0x563e04cae9cb] 2024-05-27T07:29:20.706 INFO:journalctl@ceph.mds.k.smithi105.stdout:May 27 07:29:20 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-k[69699]: 16: (Thread::entry_wrapper()+0x43) [0x7f7c73a3272d] 2024-05-27T07:29:20.706 INFO:journalctl@ceph.mds.k.smithi105.stdout:May 27 07:29:20 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-k[69699]: 17: (Thread::_entry_func(void*)+0xd) [0x7f7c73a32749] 2024-05-27T07:29:20.706 INFO:journalctl@ceph.mds.k.smithi105.stdout:May 27 07:29:20 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-k[69699]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7f7c725d61ca] 2024-05-27T07:29:20.706 INFO:journalctl@ceph.mds.k.smithi105.stdout:May 27 07:29:20 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-k[69699]: 19: clone() 2024-05-27T07:29:20.706 INFO:journalctl@ceph.mds.k.smithi105.stdout:May 27 07:29:20 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-k[69699]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-05-27T07:29:20.706 INFO:journalctl@ceph.mds.k.smithi105.stdout:May 27 07:29:20 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-k[69699]: 2024-05-27T07:29:20.706 INFO:journalctl@ceph.mds.k.smithi105.stdout:May 27 07:29:20 smithi105 podman[69734]: 2024-05-27 07:29:20.551442084 +0000 UTC m=+0.027771611 container died 63d34243be402599f9f82855c72ddf43c4fc940256782a2b543ffdcdb8f8c5f6 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-k, org.label-schema.build-date=20240521, CEPH_POINT_RELEASE=, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_CLEAN=True, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, ceph=True, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2) 2024-05-27T07:29:20.733 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:29:20 smithi052 ceph-mon[31955]: mds.? [v2:172.21.15.105:6832/1948236635,v1:172.21.15.105:6833/1948236635] up:boot 2024-05-27T07:29:20.734 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:29:20 smithi052 ceph-mon[31955]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-05-27T07:29:20.734 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:29:20 smithi052 ceph-mon[31955]: from='mgr.15333 172.21.15.52:0/416389082' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "k"}]: dispatch 2024-05-27T07:29:20.734 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:29:20 smithi052 ceph-mon[31955]: daemon mds.a restarted 2024-05-27T07:29:20.734 INFO:journalctl@ceph.mds.d.smithi052.stdout:May 27 07:29:20 smithi052 systemd[1]: ceph-73df2636-1bf8-11ef-bc9b-c7b262605968@mds.d.service: Failed with result 'exit-code'. 2024-05-27T07:29:20.734 INFO:journalctl@ceph.mds.d.smithi052.stdout:May 27 07:29:20 smithi052 systemd[1]: ceph-73df2636-1bf8-11ef-bc9b-c7b262605968@mds.d.service: Consumed 1.383s CPU time. 2024-05-27T07:29:20.734 INFO:journalctl@ceph.mds.g.smithi052.stdout:May 27 07:29:20 smithi052 systemd[1]: ceph-73df2636-1bf8-11ef-bc9b-c7b262605968@mds.g.service: Failed with result 'exit-code'. 2024-05-27T07:29:20.734 INFO:journalctl@ceph.mds.g.smithi052.stdout:May 27 07:29:20 smithi052 systemd[1]: ceph-73df2636-1bf8-11ef-bc9b-c7b262605968@mds.g.service: Consumed 1.077s CPU time. 2024-05-27T07:29:21.602 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:29:21 smithi173 ceph-mon[37000]: osdmap e102: 12 total, 12 up, 12 in 2024-05-27T07:29:21.602 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:29:21 smithi173 ceph-mon[37000]: mds.? [v2:172.21.15.52:6836/1075341446,v1:172.21.15.52:6837/1075341446] up:boot 2024-05-27T07:29:21.602 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:29:21 smithi173 ceph-mon[37000]: fsmap cephfs:5 4 up:standby-replay 3 up:standby 5 up:active 2024-05-27T07:29:21.602 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:29:21 smithi173 ceph-mon[37000]: from='mgr.15333 172.21.15.52:0/416389082' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "a"}]: dispatch 2024-05-27T07:29:21.602 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:29:21 smithi173 ceph-mon[37000]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-05-27T07:29:21.602 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:29:21 smithi173 ceph-mon[37000]: pgmap v116: 129 pgs: 129 active+clean; 198 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 2.0 MiB/s wr, 370 op/s 2024-05-27T07:29:21.690 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:29:21 smithi105 ceph-mon[35977]: osdmap e102: 12 total, 12 up, 12 in 2024-05-27T07:29:21.690 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:29:21 smithi105 ceph-mon[35977]: mds.? [v2:172.21.15.52:6836/1075341446,v1:172.21.15.52:6837/1075341446] up:boot 2024-05-27T07:29:21.690 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:29:21 smithi105 ceph-mon[35977]: fsmap cephfs:5 4 up:standby-replay 3 up:standby 5 up:active 2024-05-27T07:29:21.690 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:29:21 smithi105 ceph-mon[35977]: from='mgr.15333 172.21.15.52:0/416389082' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "a"}]: dispatch 2024-05-27T07:29:21.690 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:29:21 smithi105 ceph-mon[35977]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-05-27T07:29:21.690 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:29:21 smithi105 ceph-mon[35977]: pgmap v116: 129 pgs: 129 active+clean; 198 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 2.0 MiB/s wr, 370 op/s 2024-05-27T07:29:21.729 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:29:21 smithi052 ceph-mon[31955]: osdmap e102: 12 total, 12 up, 12 in 2024-05-27T07:29:21.729 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:29:21 smithi052 ceph-mon[31955]: mds.? [v2:172.21.15.52:6836/1075341446,v1:172.21.15.52:6837/1075341446] up:boot 2024-05-27T07:29:21.729 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:29:21 smithi052 ceph-mon[31955]: fsmap cephfs:5 4 up:standby-replay 3 up:standby 5 up:active 2024-05-27T07:29:21.729 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:29:21 smithi052 ceph-mon[31955]: from='mgr.15333 172.21.15.52:0/416389082' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "a"}]: dispatch 2024-05-27T07:29:21.730 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:29:21 smithi052 ceph-mon[31955]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-05-27T07:29:21.730 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:29:21 smithi052 ceph-mon[31955]: pgmap v116: 129 pgs: 129 active+clean; 198 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 2.0 MiB/s wr, 370 op/s 2024-05-27T07:29:22.102 INFO:journalctl@ceph.mds.f.smithi173.stdout:May 27 07:29:21 smithi173 systemd[1]: ceph-73df2636-1bf8-11ef-bc9b-c7b262605968@mds.f.service: Scheduled restart job, restart counter is at 6. 2024-05-27T07:29:22.102 INFO:journalctl@ceph.mds.f.smithi173.stdout:May 27 07:29:21 smithi173 systemd[1]: Stopped Ceph mds.f for 73df2636-1bf8-11ef-bc9b-c7b262605968. 2024-05-27T07:29:22.102 INFO:journalctl@ceph.mds.f.smithi173.stdout:May 27 07:29:21 smithi173 systemd[1]: ceph-73df2636-1bf8-11ef-bc9b-c7b262605968@mds.f.service: Start request repeated too quickly. 2024-05-27T07:29:22.102 INFO:journalctl@ceph.mds.f.smithi173.stdout:May 27 07:29:21 smithi173 systemd[1]: ceph-73df2636-1bf8-11ef-bc9b-c7b262605968@mds.f.service: Failed with result 'exit-code'. 2024-05-27T07:29:22.102 INFO:journalctl@ceph.mds.f.smithi173.stdout:May 27 07:29:21 smithi173 systemd[1]: Failed to start Ceph mds.f for 73df2636-1bf8-11ef-bc9b-c7b262605968. 2024-05-27T07:29:23.125 DEBUG:teuthology.orchestra.run.smithi052:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:38f9d900c83c63799fbdbe61acc9a11b0d3554a6 shell --fsid 73df2636-1bf8-11ef-bc9b-c7b262605968 -- ceph tell mds.1:0 scrub status 2024-05-27T07:29:23.745 INFO:teuthology.orchestra.run.smithi052.stderr:Inferring config /var/lib/ceph/73df2636-1bf8-11ef-bc9b-c7b262605968/mon.a/config 2024-05-27T07:29:24.229 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:29:24 smithi052 ceph-mon[31955]: pgmap v117: 129 pgs: 129 active+clean; 198 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 778 KiB/s rd, 1.2 MiB/s wr, 180 op/s 2024-05-27T07:29:24.354 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:29:24 smithi173 ceph-mon[37000]: pgmap v117: 129 pgs: 129 active+clean; 198 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 778 KiB/s rd, 1.2 MiB/s wr, 180 op/s 2024-05-27T07:29:24.440 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:29:24 smithi105 ceph-mon[35977]: pgmap v117: 129 pgs: 129 active+clean; 198 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 778 KiB/s rd, 1.2 MiB/s wr, 180 op/s 2024-05-27T07:29:25.206 INFO:tasks.workunit.client.0.smithi052.stdout: 53 1225027 356 847111 320 756409 934 2024-05-27T07:29:25.352 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:29:25 smithi173 ceph-mon[37000]: MDS daemon mds.f is removed because it is dead or otherwise unavailable. 2024-05-27T07:29:25.352 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:29:25 smithi173 ceph-mon[37000]: fsmap cephfs:5 5 up:standby-replay 1 up:standby 5 up:active 2024-05-27T07:29:25.440 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:29:25 smithi105 ceph-mon[35977]: MDS daemon mds.f is removed because it is dead or otherwise unavailable. 2024-05-27T07:29:25.440 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:29:25 smithi105 ceph-mon[35977]: fsmap cephfs:5 5 up:standby-replay 1 up:standby 5 up:active 2024-05-27T07:29:25.480 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:29:25 smithi052 ceph-mon[31955]: MDS daemon mds.f is removed because it is dead or otherwise unavailable. 2024-05-27T07:29:25.480 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:29:25 smithi052 ceph-mon[31955]: fsmap cephfs:5 5 up:standby-replay 1 up:standby 5 up:active 2024-05-27T07:29:25.481 INFO:journalctl@ceph.mds.a.smithi052.stdout:May 27 07:29:25 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-a[104660]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: In function 'std::set SnapClient::filter(const std::set&) const' thread 7f0cbf9a3700 time 2024-05-27T07:29:25.032361+0000 2024-05-27T07:29:25.481 INFO:journalctl@ceph.mds.a.smithi052.stdout:May 27 07:29:25 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-a[104660]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: 197: FAILED ceph_assert(cached_version > 0) 2024-05-27T07:29:25.481 INFO:journalctl@ceph.mds.a.smithi052.stdout:May 27 07:29:25 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-a[104660]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:29:25.481 INFO:journalctl@ceph.mds.a.smithi052.stdout:May 27 07:29:25 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-a[104660]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x127) [0x7f0cd21e8ba6] 2024-05-27T07:29:25.481 INFO:journalctl@ceph.mds.a.smithi052.stdout:May 27 07:29:25 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-a[104660]: 2: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7f0cd21e8dd1] 2024-05-27T07:29:25.481 INFO:journalctl@ceph.mds.a.smithi052.stdout:May 27 07:29:25 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-a[104660]: 3: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x55cd59b2923d] 2024-05-27T07:29:25.481 INFO:journalctl@ceph.mds.a.smithi052.stdout:May 27 07:29:25 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-a[104660]: 4: (SnapRealm::build_snap_set() const+0x323) [0x55cd59b14971] 2024-05-27T07:29:25.481 INFO:journalctl@ceph.mds.a.smithi052.stdout:May 27 07:29:25 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-a[104660]: 5: (SnapRealm::check_cache() const+0x13e) [0x55cd59b12fa0] 2024-05-27T07:29:25.482 INFO:journalctl@ceph.mds.a.smithi052.stdout:May 27 07:29:25 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-a[104660]: 6: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x55cd59b13651] 2024-05-27T07:29:25.482 INFO:journalctl@ceph.mds.a.smithi052.stdout:May 27 07:29:25 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-a[104660]: 7: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x55cd59aa8347] 2024-05-27T07:29:25.482 INFO:journalctl@ceph.mds.a.smithi052.stdout:May 27 07:29:25 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-a[104660]: 8: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x55cd597a487e] 2024-05-27T07:29:25.482 INFO:journalctl@ceph.mds.a.smithi052.stdout:May 27 07:29:25 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-a[104660]: 9: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x279d) [0x55cd597a9f2f] 2024-05-27T07:29:25.482 INFO:journalctl@ceph.mds.a.smithi052.stdout:May 27 07:29:25 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-a[104660]: 10: (EUpdate::replay(MDSRank*)+0x63) [0x55cd597b4969] 2024-05-27T07:29:25.482 INFO:journalctl@ceph.mds.a.smithi052.stdout:May 27 07:29:25 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-a[104660]: 11: (MDLog::_replay_thread()+0x1f38) [0x55cd59b4d48c] 2024-05-27T07:29:25.482 INFO:journalctl@ceph.mds.a.smithi052.stdout:May 27 07:29:25 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-a[104660]: 12: (MDLog::ReplayThread::entry()+0x11) [0x55cd59b569cb] 2024-05-27T07:29:25.483 INFO:journalctl@ceph.mds.a.smithi052.stdout:May 27 07:29:25 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-a[104660]: 13: (Thread::entry_wrapper()+0x43) [0x7f0cd21b772d] 2024-05-27T07:29:25.483 INFO:journalctl@ceph.mds.a.smithi052.stdout:May 27 07:29:25 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-a[104660]: 14: (Thread::_entry_func(void*)+0xd) [0x7f0cd21b7749] 2024-05-27T07:29:25.483 INFO:journalctl@ceph.mds.a.smithi052.stdout:May 27 07:29:25 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-a[104660]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7f0cd0d5b1ca] 2024-05-27T07:29:25.483 INFO:journalctl@ceph.mds.a.smithi052.stdout:May 27 07:29:25 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-a[104660]: 16: clone() 2024-05-27T07:29:25.483 INFO:journalctl@ceph.mds.a.smithi052.stdout:May 27 07:29:25 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-a[104660]: 2024-05-27T07:29:25.034+0000 7f0cbf9a3700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: In function 'std::set SnapClient::filter(const std::set&) const' thread 7f0cbf9a3700 time 2024-05-27T07:29:25.032361+0000 2024-05-27T07:29:25.483 INFO:journalctl@ceph.mds.a.smithi052.stdout:May 27 07:29:25 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-a[104660]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: 197: FAILED ceph_assert(cached_version > 0) 2024-05-27T07:29:25.483 INFO:journalctl@ceph.mds.a.smithi052.stdout:May 27 07:29:25 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-a[104660]: 2024-05-27T07:29:25.483 INFO:journalctl@ceph.mds.a.smithi052.stdout:May 27 07:29:25 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-a[104660]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:29:25.484 INFO:journalctl@ceph.mds.a.smithi052.stdout:May 27 07:29:25 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-a[104660]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x127) [0x7f0cd21e8ba6] 2024-05-27T07:29:25.484 INFO:journalctl@ceph.mds.a.smithi052.stdout:May 27 07:29:25 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-a[104660]: 2: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7f0cd21e8dd1] 2024-05-27T07:29:25.484 INFO:journalctl@ceph.mds.a.smithi052.stdout:May 27 07:29:25 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-a[104660]: 3: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x55cd59b2923d] 2024-05-27T07:29:25.484 INFO:journalctl@ceph.mds.a.smithi052.stdout:May 27 07:29:25 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-a[104660]: 4: (SnapRealm::build_snap_set() const+0x323) [0x55cd59b14971] 2024-05-27T07:29:25.484 INFO:journalctl@ceph.mds.a.smithi052.stdout:May 27 07:29:25 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-a[104660]: 5: (SnapRealm::check_cache() const+0x13e) [0x55cd59b12fa0] 2024-05-27T07:29:25.484 INFO:journalctl@ceph.mds.a.smithi052.stdout:May 27 07:29:25 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-a[104660]: 6: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x55cd59b13651] 2024-05-27T07:29:25.484 INFO:journalctl@ceph.mds.a.smithi052.stdout:May 27 07:29:25 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-a[104660]: 7: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x55cd59aa8347] 2024-05-27T07:29:25.484 INFO:journalctl@ceph.mds.a.smithi052.stdout:May 27 07:29:25 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-a[104660]: 8: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x55cd597a487e] 2024-05-27T07:29:25.485 INFO:journalctl@ceph.mds.a.smithi052.stdout:May 27 07:29:25 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-a[104660]: 9: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x279d) [0x55cd597a9f2f] 2024-05-27T07:29:25.485 INFO:journalctl@ceph.mds.a.smithi052.stdout:May 27 07:29:25 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-a[104660]: 10: (EUpdate::replay(MDSRank*)+0x63) [0x55cd597b4969] 2024-05-27T07:29:25.485 INFO:journalctl@ceph.mds.a.smithi052.stdout:May 27 07:29:25 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-a[104660]: 11: (MDLog::_replay_thread()+0x1f38) [0x55cd59b4d48c] 2024-05-27T07:29:25.485 INFO:journalctl@ceph.mds.a.smithi052.stdout:May 27 07:29:25 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-a[104660]: 12: (MDLog::ReplayThread::entry()+0x11) [0x55cd59b569cb] 2024-05-27T07:29:25.485 INFO:journalctl@ceph.mds.a.smithi052.stdout:May 27 07:29:25 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-a[104660]: 13: (Thread::entry_wrapper()+0x43) [0x7f0cd21b772d] 2024-05-27T07:29:25.485 INFO:journalctl@ceph.mds.a.smithi052.stdout:May 27 07:29:25 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-a[104660]: 14: (Thread::_entry_func(void*)+0xd) [0x7f0cd21b7749] 2024-05-27T07:29:25.485 INFO:journalctl@ceph.mds.a.smithi052.stdout:May 27 07:29:25 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-a[104660]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7f0cd0d5b1ca] 2024-05-27T07:29:25.485 INFO:journalctl@ceph.mds.a.smithi052.stdout:May 27 07:29:25 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-a[104660]: 16: clone() 2024-05-27T07:29:25.486 INFO:journalctl@ceph.mds.a.smithi052.stdout:May 27 07:29:25 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-a[104660]: 2024-05-27T07:29:25.486 INFO:journalctl@ceph.mds.a.smithi052.stdout:May 27 07:29:25 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-a[104660]: *** Caught signal (Aborted) ** 2024-05-27T07:29:25.486 INFO:journalctl@ceph.mds.a.smithi052.stdout:May 27 07:29:25 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-a[104660]: in thread 7f0cbf9a3700 thread_name:md_log_replay 2024-05-27T07:29:25.486 INFO:journalctl@ceph.mds.a.smithi052.stdout:May 27 07:29:25 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-a[104660]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:29:25.486 INFO:journalctl@ceph.mds.a.smithi052.stdout:May 27 07:29:25 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-a[104660]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f0cd0d65d20] 2024-05-27T07:29:25.486 INFO:journalctl@ceph.mds.a.smithi052.stdout:May 27 07:29:25 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-a[104660]: 2: gsignal() 2024-05-27T07:29:25.486 INFO:journalctl@ceph.mds.a.smithi052.stdout:May 27 07:29:25 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-a[104660]: 3: abort() 2024-05-27T07:29:25.486 INFO:journalctl@ceph.mds.a.smithi052.stdout:May 27 07:29:25 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-a[104660]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x26e) [0x7f0cd21e8ced] 2024-05-27T07:29:25.488 INFO:journalctl@ceph.mds.a.smithi052.stdout:May 27 07:29:25 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-a[104660]: 5: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7f0cd21e8dd1] 2024-05-27T07:29:25.488 INFO:journalctl@ceph.mds.a.smithi052.stdout:May 27 07:29:25 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-a[104660]: 6: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x55cd59b2923d] 2024-05-27T07:29:25.488 INFO:journalctl@ceph.mds.a.smithi052.stdout:May 27 07:29:25 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-a[104660]: 7: (SnapRealm::build_snap_set() const+0x323) [0x55cd59b14971] 2024-05-27T07:29:25.488 INFO:journalctl@ceph.mds.a.smithi052.stdout:May 27 07:29:25 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-a[104660]: 8: (SnapRealm::check_cache() const+0x13e) [0x55cd59b12fa0] 2024-05-27T07:29:25.488 INFO:journalctl@ceph.mds.a.smithi052.stdout:May 27 07:29:25 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-a[104660]: 9: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x55cd59b13651] 2024-05-27T07:29:25.488 INFO:journalctl@ceph.mds.a.smithi052.stdout:May 27 07:29:25 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-a[104660]: 10: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x55cd59aa8347] 2024-05-27T07:29:25.489 INFO:journalctl@ceph.mds.a.smithi052.stdout:May 27 07:29:25 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-a[104660]: 11: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x55cd597a487e] 2024-05-27T07:29:25.489 INFO:journalctl@ceph.mds.a.smithi052.stdout:May 27 07:29:25 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-a[104660]: 12: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x279d) [0x55cd597a9f2f] 2024-05-27T07:29:25.489 INFO:journalctl@ceph.mds.a.smithi052.stdout:May 27 07:29:25 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-a[104660]: 13: (EUpdate::replay(MDSRank*)+0x63) [0x55cd597b4969] 2024-05-27T07:29:25.489 INFO:journalctl@ceph.mds.a.smithi052.stdout:May 27 07:29:25 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-a[104660]: 14: (MDLog::_replay_thread()+0x1f38) [0x55cd59b4d48c] 2024-05-27T07:29:25.489 INFO:journalctl@ceph.mds.a.smithi052.stdout:May 27 07:29:25 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-a[104660]: 15: (MDLog::ReplayThread::entry()+0x11) [0x55cd59b569cb] 2024-05-27T07:29:25.489 INFO:journalctl@ceph.mds.a.smithi052.stdout:May 27 07:29:25 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-a[104660]: 16: (Thread::entry_wrapper()+0x43) [0x7f0cd21b772d] 2024-05-27T07:29:25.489 INFO:journalctl@ceph.mds.a.smithi052.stdout:May 27 07:29:25 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-a[104660]: 17: (Thread::_entry_func(void*)+0xd) [0x7f0cd21b7749] 2024-05-27T07:29:25.489 INFO:journalctl@ceph.mds.a.smithi052.stdout:May 27 07:29:25 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-a[104660]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7f0cd0d5b1ca] 2024-05-27T07:29:25.489 INFO:journalctl@ceph.mds.a.smithi052.stdout:May 27 07:29:25 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-a[104660]: 19: clone() 2024-05-27T07:29:25.490 INFO:journalctl@ceph.mds.a.smithi052.stdout:May 27 07:29:25 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-a[104660]: 2024-05-27T07:29:25.038+0000 7f0cbf9a3700 -1 *** Caught signal (Aborted) ** 2024-05-27T07:29:25.490 INFO:journalctl@ceph.mds.a.smithi052.stdout:May 27 07:29:25 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-a[104660]: in thread 7f0cbf9a3700 thread_name:md_log_replay 2024-05-27T07:29:25.490 INFO:journalctl@ceph.mds.a.smithi052.stdout:May 27 07:29:25 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-a[104660]: 2024-05-27T07:29:25.490 INFO:journalctl@ceph.mds.a.smithi052.stdout:May 27 07:29:25 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-a[104660]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:29:25.490 INFO:journalctl@ceph.mds.a.smithi052.stdout:May 27 07:29:25 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-a[104660]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f0cd0d65d20] 2024-05-27T07:29:25.490 INFO:journalctl@ceph.mds.a.smithi052.stdout:May 27 07:29:25 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-a[104660]: 2: gsignal() 2024-05-27T07:29:25.490 INFO:journalctl@ceph.mds.a.smithi052.stdout:May 27 07:29:25 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-a[104660]: 3: abort() 2024-05-27T07:29:25.491 INFO:journalctl@ceph.mds.a.smithi052.stdout:May 27 07:29:25 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-a[104660]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x26e) [0x7f0cd21e8ced] 2024-05-27T07:29:25.491 INFO:journalctl@ceph.mds.a.smithi052.stdout:May 27 07:29:25 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-a[104660]: 5: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7f0cd21e8dd1] 2024-05-27T07:29:25.491 INFO:journalctl@ceph.mds.a.smithi052.stdout:May 27 07:29:25 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-a[104660]: 6: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x55cd59b2923d] 2024-05-27T07:29:25.491 INFO:journalctl@ceph.mds.a.smithi052.stdout:May 27 07:29:25 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-a[104660]: 7: (SnapRealm::build_snap_set() const+0x323) [0x55cd59b14971] 2024-05-27T07:29:25.491 INFO:journalctl@ceph.mds.a.smithi052.stdout:May 27 07:29:25 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-a[104660]: 8: (SnapRealm::check_cache() const+0x13e) [0x55cd59b12fa0] 2024-05-27T07:29:25.491 INFO:journalctl@ceph.mds.a.smithi052.stdout:May 27 07:29:25 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-a[104660]: 9: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x55cd59b13651] 2024-05-27T07:29:25.491 INFO:journalctl@ceph.mds.a.smithi052.stdout:May 27 07:29:25 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-a[104660]: 10: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x55cd59aa8347] 2024-05-27T07:29:25.491 INFO:journalctl@ceph.mds.a.smithi052.stdout:May 27 07:29:25 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-a[104660]: 11: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x55cd597a487e] 2024-05-27T07:29:25.491 INFO:journalctl@ceph.mds.a.smithi052.stdout:May 27 07:29:25 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-a[104660]: 12: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x279d) [0x55cd597a9f2f] 2024-05-27T07:29:25.492 INFO:journalctl@ceph.mds.a.smithi052.stdout:May 27 07:29:25 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-a[104660]: 13: (EUpdate::replay(MDSRank*)+0x63) [0x55cd597b4969] 2024-05-27T07:29:25.492 INFO:journalctl@ceph.mds.a.smithi052.stdout:May 27 07:29:25 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-a[104660]: 14: (MDLog::_replay_thread()+0x1f38) [0x55cd59b4d48c] 2024-05-27T07:29:25.492 INFO:journalctl@ceph.mds.a.smithi052.stdout:May 27 07:29:25 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-a[104660]: 15: (MDLog::ReplayThread::entry()+0x11) [0x55cd59b569cb] 2024-05-27T07:29:25.492 INFO:journalctl@ceph.mds.a.smithi052.stdout:May 27 07:29:25 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-a[104660]: 16: (Thread::entry_wrapper()+0x43) [0x7f0cd21b772d] 2024-05-27T07:29:25.492 INFO:journalctl@ceph.mds.a.smithi052.stdout:May 27 07:29:25 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-a[104660]: 17: (Thread::_entry_func(void*)+0xd) [0x7f0cd21b7749] 2024-05-27T07:29:25.492 INFO:journalctl@ceph.mds.a.smithi052.stdout:May 27 07:29:25 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-a[104660]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7f0cd0d5b1ca] 2024-05-27T07:29:25.492 INFO:journalctl@ceph.mds.a.smithi052.stdout:May 27 07:29:25 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-a[104660]: 19: clone() 2024-05-27T07:29:25.492 INFO:journalctl@ceph.mds.a.smithi052.stdout:May 27 07:29:25 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-a[104660]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-05-27T07:29:25.493 INFO:journalctl@ceph.mds.a.smithi052.stdout:May 27 07:29:25 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-a[104660]: 2024-05-27T07:29:25.493 INFO:journalctl@ceph.mds.a.smithi052.stdout:May 27 07:29:25 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-a[104660]: -2> 2024-05-27T07:29:25.034+0000 7f0cbf9a3700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: In function 'std::set SnapClient::filter(const std::set&) const' thread 7f0cbf9a3700 time 2024-05-27T07:29:25.032361+0000 2024-05-27T07:29:25.493 INFO:journalctl@ceph.mds.a.smithi052.stdout:May 27 07:29:25 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-a[104660]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: 197: FAILED ceph_assert(cached_version > 0) 2024-05-27T07:29:25.493 INFO:journalctl@ceph.mds.a.smithi052.stdout:May 27 07:29:25 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-a[104660]: 2024-05-27T07:29:25.493 INFO:journalctl@ceph.mds.a.smithi052.stdout:May 27 07:29:25 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-a[104660]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:29:25.493 INFO:journalctl@ceph.mds.a.smithi052.stdout:May 27 07:29:25 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-a[104660]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x127) [0x7f0cd21e8ba6] 2024-05-27T07:29:25.493 INFO:journalctl@ceph.mds.a.smithi052.stdout:May 27 07:29:25 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-a[104660]: 2: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7f0cd21e8dd1] 2024-05-27T07:29:25.493 INFO:journalctl@ceph.mds.a.smithi052.stdout:May 27 07:29:25 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-a[104660]: 3: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x55cd59b2923d] 2024-05-27T07:29:25.493 INFO:journalctl@ceph.mds.a.smithi052.stdout:May 27 07:29:25 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-a[104660]: 4: (SnapRealm::build_snap_set() const+0x323) [0x55cd59b14971] 2024-05-27T07:29:25.494 INFO:journalctl@ceph.mds.a.smithi052.stdout:May 27 07:29:25 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-a[104660]: 5: (SnapRealm::check_cache() const+0x13e) [0x55cd59b12fa0] 2024-05-27T07:29:25.494 INFO:journalctl@ceph.mds.a.smithi052.stdout:May 27 07:29:25 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-a[104660]: 6: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x55cd59b13651] 2024-05-27T07:29:25.494 INFO:journalctl@ceph.mds.a.smithi052.stdout:May 27 07:29:25 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-a[104660]: 7: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x55cd59aa8347] 2024-05-27T07:29:25.494 INFO:journalctl@ceph.mds.a.smithi052.stdout:May 27 07:29:25 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-a[104660]: 8: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x55cd597a487e] 2024-05-27T07:29:25.494 INFO:journalctl@ceph.mds.a.smithi052.stdout:May 27 07:29:25 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-a[104660]: 9: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x279d) [0x55cd597a9f2f] 2024-05-27T07:29:25.494 INFO:journalctl@ceph.mds.a.smithi052.stdout:May 27 07:29:25 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-a[104660]: 10: (EUpdate::replay(MDSRank*)+0x63) [0x55cd597b4969] 2024-05-27T07:29:25.494 INFO:journalctl@ceph.mds.a.smithi052.stdout:May 27 07:29:25 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-a[104660]: 11: (MDLog::_replay_thread()+0x1f38) [0x55cd59b4d48c] 2024-05-27T07:29:25.494 INFO:journalctl@ceph.mds.a.smithi052.stdout:May 27 07:29:25 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-a[104660]: 12: (MDLog::ReplayThread::entry()+0x11) [0x55cd59b569cb] 2024-05-27T07:29:25.495 INFO:journalctl@ceph.mds.a.smithi052.stdout:May 27 07:29:25 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-a[104660]: 13: (Thread::entry_wrapper()+0x43) [0x7f0cd21b772d] 2024-05-27T07:29:25.495 INFO:journalctl@ceph.mds.a.smithi052.stdout:May 27 07:29:25 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-a[104660]: 14: (Thread::_entry_func(void*)+0xd) [0x7f0cd21b7749] 2024-05-27T07:29:25.495 INFO:journalctl@ceph.mds.a.smithi052.stdout:May 27 07:29:25 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-a[104660]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7f0cd0d5b1ca] 2024-05-27T07:29:25.495 INFO:journalctl@ceph.mds.a.smithi052.stdout:May 27 07:29:25 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-a[104660]: 16: clone() 2024-05-27T07:29:25.495 INFO:journalctl@ceph.mds.a.smithi052.stdout:May 27 07:29:25 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-a[104660]: 2024-05-27T07:29:25.495 INFO:journalctl@ceph.mds.a.smithi052.stdout:May 27 07:29:25 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-a[104660]: 0> 2024-05-27T07:29:25.038+0000 7f0cbf9a3700 -1 *** Caught signal (Aborted) ** 2024-05-27T07:29:25.495 INFO:journalctl@ceph.mds.a.smithi052.stdout:May 27 07:29:25 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-a[104660]: in thread 7f0cbf9a3700 thread_name:md_log_replay 2024-05-27T07:29:25.495 INFO:journalctl@ceph.mds.a.smithi052.stdout:May 27 07:29:25 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-a[104660]: 2024-05-27T07:29:25.496 INFO:journalctl@ceph.mds.a.smithi052.stdout:May 27 07:29:25 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-a[104660]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:29:25.496 INFO:journalctl@ceph.mds.a.smithi052.stdout:May 27 07:29:25 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-a[104660]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f0cd0d65d20] 2024-05-27T07:29:25.496 INFO:journalctl@ceph.mds.a.smithi052.stdout:May 27 07:29:25 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-a[104660]: 2: gsignal() 2024-05-27T07:29:25.496 INFO:journalctl@ceph.mds.a.smithi052.stdout:May 27 07:29:25 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-a[104660]: 3: abort() 2024-05-27T07:29:25.496 INFO:journalctl@ceph.mds.a.smithi052.stdout:May 27 07:29:25 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-a[104660]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x26e) [0x7f0cd21e8ced] 2024-05-27T07:29:25.496 INFO:journalctl@ceph.mds.a.smithi052.stdout:May 27 07:29:25 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-a[104660]: 5: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7f0cd21e8dd1] 2024-05-27T07:29:25.496 INFO:journalctl@ceph.mds.a.smithi052.stdout:May 27 07:29:25 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-a[104660]: 6: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x55cd59b2923d] 2024-05-27T07:29:25.496 INFO:journalctl@ceph.mds.a.smithi052.stdout:May 27 07:29:25 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-a[104660]: 7: (SnapRealm::build_snap_set() const+0x323) [0x55cd59b14971] 2024-05-27T07:29:25.497 INFO:journalctl@ceph.mds.a.smithi052.stdout:May 27 07:29:25 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-a[104660]: 8: (SnapRealm::check_cache() const+0x13e) [0x55cd59b12fa0] 2024-05-27T07:29:25.497 INFO:journalctl@ceph.mds.a.smithi052.stdout:May 27 07:29:25 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-a[104660]: 9: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x55cd59b13651] 2024-05-27T07:29:25.497 INFO:journalctl@ceph.mds.a.smithi052.stdout:May 27 07:29:25 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-a[104660]: 10: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x55cd59aa8347] 2024-05-27T07:29:25.497 INFO:journalctl@ceph.mds.a.smithi052.stdout:May 27 07:29:25 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-a[104660]: 11: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x55cd597a487e] 2024-05-27T07:29:25.497 INFO:journalctl@ceph.mds.a.smithi052.stdout:May 27 07:29:25 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-a[104660]: 12: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x279d) [0x55cd597a9f2f] 2024-05-27T07:29:25.497 INFO:journalctl@ceph.mds.a.smithi052.stdout:May 27 07:29:25 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-a[104660]: 13: (EUpdate::replay(MDSRank*)+0x63) [0x55cd597b4969] 2024-05-27T07:29:25.497 INFO:journalctl@ceph.mds.a.smithi052.stdout:May 27 07:29:25 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-a[104660]: 14: (MDLog::_replay_thread()+0x1f38) [0x55cd59b4d48c] 2024-05-27T07:29:25.497 INFO:journalctl@ceph.mds.a.smithi052.stdout:May 27 07:29:25 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-a[104660]: 15: (MDLog::ReplayThread::entry()+0x11) [0x55cd59b569cb] 2024-05-27T07:29:25.498 INFO:journalctl@ceph.mds.a.smithi052.stdout:May 27 07:29:25 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-a[104660]: 16: (Thread::entry_wrapper()+0x43) [0x7f0cd21b772d] 2024-05-27T07:29:25.498 INFO:journalctl@ceph.mds.a.smithi052.stdout:May 27 07:29:25 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-a[104660]: 17: (Thread::_entry_func(void*)+0xd) [0x7f0cd21b7749] 2024-05-27T07:29:25.498 INFO:journalctl@ceph.mds.a.smithi052.stdout:May 27 07:29:25 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-a[104660]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7f0cd0d5b1ca] 2024-05-27T07:29:25.498 INFO:journalctl@ceph.mds.a.smithi052.stdout:May 27 07:29:25 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-a[104660]: 19: clone() 2024-05-27T07:29:25.498 INFO:journalctl@ceph.mds.a.smithi052.stdout:May 27 07:29:25 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-a[104660]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-05-27T07:29:25.498 INFO:journalctl@ceph.mds.a.smithi052.stdout:May 27 07:29:25 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-a[104660]: 2024-05-27T07:29:25.498 INFO:journalctl@ceph.mds.a.smithi052.stdout:May 27 07:29:25 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-a[104660]: -3> 2024-05-27T07:29:25.034+0000 7f0cbf9a3700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: In function 'std::set SnapClient::filter(const std::set&) const' thread 7f0cbf9a3700 time 2024-05-27T07:29:25.032361+0000 2024-05-27T07:29:25.498 INFO:journalctl@ceph.mds.a.smithi052.stdout:May 27 07:29:25 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-a[104660]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: 197: FAILED ceph_assert(cached_version > 0) 2024-05-27T07:29:25.499 INFO:journalctl@ceph.mds.a.smithi052.stdout:May 27 07:29:25 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-a[104660]: 2024-05-27T07:29:25.499 INFO:journalctl@ceph.mds.a.smithi052.stdout:May 27 07:29:25 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-a[104660]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:29:25.499 INFO:journalctl@ceph.mds.a.smithi052.stdout:May 27 07:29:25 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-a[104660]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x127) [0x7f0cd21e8ba6] 2024-05-27T07:29:25.499 INFO:journalctl@ceph.mds.a.smithi052.stdout:May 27 07:29:25 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-a[104660]: 2: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7f0cd21e8dd1] 2024-05-27T07:29:25.499 INFO:journalctl@ceph.mds.a.smithi052.stdout:May 27 07:29:25 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-a[104660]: 3: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x55cd59b2923d] 2024-05-27T07:29:25.499 INFO:journalctl@ceph.mds.a.smithi052.stdout:May 27 07:29:25 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-a[104660]: 4: (SnapRealm::build_snap_set() const+0x323) [0x55cd59b14971] 2024-05-27T07:29:25.499 INFO:journalctl@ceph.mds.a.smithi052.stdout:May 27 07:29:25 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-a[104660]: 5: (SnapRealm::check_cache() const+0x13e) [0x55cd59b12fa0] 2024-05-27T07:29:25.499 INFO:journalctl@ceph.mds.a.smithi052.stdout:May 27 07:29:25 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-a[104660]: 6: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x55cd59b13651] 2024-05-27T07:29:25.500 INFO:journalctl@ceph.mds.a.smithi052.stdout:May 27 07:29:25 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-a[104660]: 7: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x55cd59aa8347] 2024-05-27T07:29:25.500 INFO:journalctl@ceph.mds.a.smithi052.stdout:May 27 07:29:25 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-a[104660]: 8: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x55cd597a487e] 2024-05-27T07:29:25.500 INFO:journalctl@ceph.mds.a.smithi052.stdout:May 27 07:29:25 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-a[104660]: 9: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x279d) [0x55cd597a9f2f] 2024-05-27T07:29:25.500 INFO:journalctl@ceph.mds.a.smithi052.stdout:May 27 07:29:25 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-a[104660]: 10: (EUpdate::replay(MDSRank*)+0x63) [0x55cd597b4969] 2024-05-27T07:29:25.500 INFO:journalctl@ceph.mds.a.smithi052.stdout:May 27 07:29:25 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-a[104660]: 11: (MDLog::_replay_thread()+0x1f38) [0x55cd59b4d48c] 2024-05-27T07:29:25.500 INFO:journalctl@ceph.mds.a.smithi052.stdout:May 27 07:29:25 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-a[104660]: 12: (MDLog::ReplayThread::entry()+0x11) [0x55cd59b569cb] 2024-05-27T07:29:25.500 INFO:journalctl@ceph.mds.a.smithi052.stdout:May 27 07:29:25 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-a[104660]: 13: (Thread::entry_wrapper()+0x43) [0x7f0cd21b772d] 2024-05-27T07:29:25.500 INFO:journalctl@ceph.mds.a.smithi052.stdout:May 27 07:29:25 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-a[104660]: 14: (Thread::_entry_func(void*)+0xd) [0x7f0cd21b7749] 2024-05-27T07:29:25.500 INFO:journalctl@ceph.mds.a.smithi052.stdout:May 27 07:29:25 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-a[104660]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7f0cd0d5b1ca] 2024-05-27T07:29:25.501 INFO:journalctl@ceph.mds.a.smithi052.stdout:May 27 07:29:25 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-a[104660]: 16: clone() 2024-05-27T07:29:25.501 INFO:journalctl@ceph.mds.a.smithi052.stdout:May 27 07:29:25 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-a[104660]: 2024-05-27T07:29:25.501 INFO:journalctl@ceph.mds.a.smithi052.stdout:May 27 07:29:25 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-a[104660]: -1> 2024-05-27T07:29:25.038+0000 7f0cbf9a3700 -1 *** Caught signal (Aborted) ** 2024-05-27T07:29:25.501 INFO:journalctl@ceph.mds.a.smithi052.stdout:May 27 07:29:25 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-a[104660]: in thread 7f0cbf9a3700 thread_name:md_log_replay 2024-05-27T07:29:25.501 INFO:journalctl@ceph.mds.a.smithi052.stdout:May 27 07:29:25 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-a[104660]: 2024-05-27T07:29:25.501 INFO:journalctl@ceph.mds.a.smithi052.stdout:May 27 07:29:25 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-a[104660]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:29:25.501 INFO:journalctl@ceph.mds.a.smithi052.stdout:May 27 07:29:25 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-a[104660]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f0cd0d65d20] 2024-05-27T07:29:25.502 INFO:journalctl@ceph.mds.a.smithi052.stdout:May 27 07:29:25 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-a[104660]: 2: gsignal() 2024-05-27T07:29:25.502 INFO:journalctl@ceph.mds.a.smithi052.stdout:May 27 07:29:25 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-a[104660]: 3: abort() 2024-05-27T07:29:25.502 INFO:journalctl@ceph.mds.a.smithi052.stdout:May 27 07:29:25 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-a[104660]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x26e) [0x7f0cd21e8ced] 2024-05-27T07:29:25.502 INFO:journalctl@ceph.mds.a.smithi052.stdout:May 27 07:29:25 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-a[104660]: 5: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7f0cd21e8dd1] 2024-05-27T07:29:25.502 INFO:journalctl@ceph.mds.a.smithi052.stdout:May 27 07:29:25 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-a[104660]: 6: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x55cd59b2923d] 2024-05-27T07:29:25.502 INFO:journalctl@ceph.mds.a.smithi052.stdout:May 27 07:29:25 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-a[104660]: 7: (SnapRealm::build_snap_set() const+0x323) [0x55cd59b14971] 2024-05-27T07:29:25.502 INFO:journalctl@ceph.mds.a.smithi052.stdout:May 27 07:29:25 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-a[104660]: 8: (SnapRealm::check_cache() const+0x13e) [0x55cd59b12fa0] 2024-05-27T07:29:25.502 INFO:journalctl@ceph.mds.a.smithi052.stdout:May 27 07:29:25 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-a[104660]: 9: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x55cd59b13651] 2024-05-27T07:29:25.503 INFO:journalctl@ceph.mds.a.smithi052.stdout:May 27 07:29:25 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-a[104660]: 10: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x55cd59aa8347] 2024-05-27T07:29:25.503 INFO:journalctl@ceph.mds.a.smithi052.stdout:May 27 07:29:25 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-a[104660]: 11: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x55cd597a487e] 2024-05-27T07:29:25.503 INFO:journalctl@ceph.mds.a.smithi052.stdout:May 27 07:29:25 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-a[104660]: 12: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x279d) [0x55cd597a9f2f] 2024-05-27T07:29:25.503 INFO:journalctl@ceph.mds.a.smithi052.stdout:May 27 07:29:25 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-a[104660]: 13: (EUpdate::replay(MDSRank*)+0x63) [0x55cd597b4969] 2024-05-27T07:29:25.503 INFO:journalctl@ceph.mds.a.smithi052.stdout:May 27 07:29:25 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-a[104660]: 14: (MDLog::_replay_thread()+0x1f38) [0x55cd59b4d48c] 2024-05-27T07:29:25.503 INFO:journalctl@ceph.mds.a.smithi052.stdout:May 27 07:29:25 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-a[104660]: 15: (MDLog::ReplayThread::entry()+0x11) [0x55cd59b569cb] 2024-05-27T07:29:25.503 INFO:journalctl@ceph.mds.a.smithi052.stdout:May 27 07:29:25 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-a[104660]: 16: (Thread::entry_wrapper()+0x43) [0x7f0cd21b772d] 2024-05-27T07:29:25.503 INFO:journalctl@ceph.mds.a.smithi052.stdout:May 27 07:29:25 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-a[104660]: 17: (Thread::_entry_func(void*)+0xd) [0x7f0cd21b7749] 2024-05-27T07:29:25.504 INFO:journalctl@ceph.mds.a.smithi052.stdout:May 27 07:29:25 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-a[104660]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7f0cd0d5b1ca] 2024-05-27T07:29:25.504 INFO:journalctl@ceph.mds.a.smithi052.stdout:May 27 07:29:25 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-a[104660]: 19: clone() 2024-05-27T07:29:25.504 INFO:journalctl@ceph.mds.a.smithi052.stdout:May 27 07:29:25 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-a[104660]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-05-27T07:29:25.504 INFO:journalctl@ceph.mds.a.smithi052.stdout:May 27 07:29:25 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-a[104660]: 2024-05-27T07:29:25.504 INFO:journalctl@ceph.mds.a.smithi052.stdout:May 27 07:29:25 smithi052 podman[104807]: 2024-05-27 07:29:25.264413416 +0000 UTC m=+0.198933280 container died 46c806e123bbc1bcf4df0fc9657e67eba98bcf204f960943c2efea78585f35c4 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-a, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, GIT_CLEAN=True, ceph=True, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240521, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90) 2024-05-27T07:29:25.687 INFO:journalctl@ceph.mds.l.smithi173.stdout:May 27 07:29:25 smithi173 systemd[1]: ceph-73df2636-1bf8-11ef-bc9b-c7b262605968@mds.l.service: Scheduled restart job, restart counter is at 2. 2024-05-27T07:29:25.687 INFO:journalctl@ceph.mds.l.smithi173.stdout:May 27 07:29:25 smithi173 systemd[1]: Stopped Ceph mds.l for 73df2636-1bf8-11ef-bc9b-c7b262605968. 2024-05-27T07:29:25.687 INFO:journalctl@ceph.mds.l.smithi173.stdout:May 27 07:29:25 smithi173 systemd[1]: Starting Ceph mds.l for 73df2636-1bf8-11ef-bc9b-c7b262605968... 2024-05-27T07:29:25.687 INFO:journalctl@ceph.mds.l.smithi173.stdout:May 27 07:29:25 smithi173 podman[73434]: 2024-05-27 07:29:25.598126246 +0000 UTC m=+0.111422798 container create 686d1400cad7235ed70ce609ed1204dc003f1dfb6ad6c3e84cea045a4e9fff4d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-l, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240521, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, GIT_CLEAN=True) 2024-05-27T07:29:25.687 INFO:journalctl@ceph.mds.l.smithi173.stdout:May 27 07:29:25 smithi173 podman[73434]: 2024-05-27 07:29:25.503927926 +0000 UTC m=+0.017224491 image pull e36ea4fc8a5807e8d2701370e350a0de3856538797f7fba64cfce7e344bd6799 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6 2024-05-27T07:29:25.952 INFO:journalctl@ceph.mds.l.smithi173.stdout:May 27 07:29:25 smithi173 podman[73434]: 2024-05-27 07:29:25.808973676 +0000 UTC m=+0.322270237 container init 686d1400cad7235ed70ce609ed1204dc003f1dfb6ad6c3e84cea045a4e9fff4d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-l, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.build-date=20240521, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, CEPH_POINT_RELEASE=, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, io.buildah.version=1.33.5, ceph=True, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-05-27T07:29:25.952 INFO:journalctl@ceph.mds.l.smithi173.stdout:May 27 07:29:25 smithi173 podman[73434]: 2024-05-27 07:29:25.811565596 +0000 UTC m=+0.324862154 container start 686d1400cad7235ed70ce609ed1204dc003f1dfb6ad6c3e84cea045a4e9fff4d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-l, io.buildah.version=1.33.5, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, GIT_CLEAN=True, ceph=True, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240521, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0) 2024-05-27T07:29:25.952 INFO:journalctl@ceph.mds.l.smithi173.stdout:May 27 07:29:25 smithi173 ceph-mds[73453]: -- 172.21.15.173:0/2750343774 <== mon.1 v2:172.21.15.173:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x563084ff1a40 con 0x563085039180 2024-05-27T07:29:25.952 INFO:journalctl@ceph.mds.l.smithi173.stdout:May 27 07:29:25 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-l[73448]: starting mds.l at 2024-05-27T07:29:25.952 INFO:journalctl@ceph.mds.l.smithi173.stdout:May 27 07:29:25 smithi173 bash[73434]: 686d1400cad7235ed70ce609ed1204dc003f1dfb6ad6c3e84cea045a4e9fff4d 2024-05-27T07:29:25.952 INFO:journalctl@ceph.mds.l.smithi173.stdout:May 27 07:29:25 smithi173 systemd[1]: Started Ceph mds.l for 73df2636-1bf8-11ef-bc9b-c7b262605968. 2024-05-27T07:29:26.140 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:29:26 smithi173 ceph-mon[37000]: pgmap v118: 129 pgs: 129 active+clean; 209 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 2.8 MiB/s wr, 344 op/s 2024-05-27T07:29:26.140 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:29:26 smithi173 ceph-mon[37000]: daemon mds.l restarted 2024-05-27T07:29:26.141 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:29:26 smithi173 ceph-mon[37000]: osdmap e103: 12 total, 12 up, 12 in 2024-05-27T07:29:26.141 INFO:journalctl@ceph.mds.l.smithi173.stdout:May 27 07:29:26 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-l[73448]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: In function 'std::set SnapClient::filter(const std::set&) const' thread 7fd784e58700 time 2024-05-27T07:29:26.145123+0000 2024-05-27T07:29:26.141 INFO:journalctl@ceph.mds.l.smithi173.stdout:May 27 07:29:26 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-l[73448]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: 197: FAILED ceph_assert(cached_version > 0) 2024-05-27T07:29:26.141 INFO:journalctl@ceph.mds.l.smithi173.stdout:May 27 07:29:26 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-l[73448]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:29:26.141 INFO:journalctl@ceph.mds.l.smithi173.stdout:May 27 07:29:26 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-l[73448]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x127) [0x7fd79769dba6] 2024-05-27T07:29:26.141 INFO:journalctl@ceph.mds.l.smithi173.stdout:May 27 07:29:26 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-l[73448]: 2: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7fd79769ddd1] 2024-05-27T07:29:26.141 INFO:journalctl@ceph.mds.l.smithi173.stdout:May 27 07:29:26 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-l[73448]: 3: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x563082efa23d] 2024-05-27T07:29:26.141 INFO:journalctl@ceph.mds.l.smithi173.stdout:May 27 07:29:26 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-l[73448]: 4: (SnapRealm::build_snap_set() const+0x323) [0x563082ee5971] 2024-05-27T07:29:26.142 INFO:journalctl@ceph.mds.l.smithi173.stdout:May 27 07:29:26 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-l[73448]: 5: (SnapRealm::check_cache() const+0x13e) [0x563082ee3fa0] 2024-05-27T07:29:26.142 INFO:journalctl@ceph.mds.l.smithi173.stdout:May 27 07:29:26 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-l[73448]: 6: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x563082ee4651] 2024-05-27T07:29:26.142 INFO:journalctl@ceph.mds.l.smithi173.stdout:May 27 07:29:26 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-l[73448]: 7: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x563082e79347] 2024-05-27T07:29:26.142 INFO:journalctl@ceph.mds.l.smithi173.stdout:May 27 07:29:26 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-l[73448]: 8: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x563082b7587e] 2024-05-27T07:29:26.142 INFO:journalctl@ceph.mds.l.smithi173.stdout:May 27 07:29:26 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-l[73448]: 9: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x2702) [0x563082b7ae94] 2024-05-27T07:29:26.142 INFO:journalctl@ceph.mds.l.smithi173.stdout:May 27 07:29:26 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-l[73448]: 10: (EUpdate::replay(MDSRank*)+0x63) [0x563082b85969] 2024-05-27T07:29:26.142 INFO:journalctl@ceph.mds.l.smithi173.stdout:May 27 07:29:26 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-l[73448]: 11: (MDLog::_replay_thread()+0x1f38) [0x563082f1e48c] 2024-05-27T07:29:26.142 INFO:journalctl@ceph.mds.l.smithi173.stdout:May 27 07:29:26 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-l[73448]: 12: (MDLog::ReplayThread::entry()+0x11) [0x563082f279cb] 2024-05-27T07:29:26.142 INFO:journalctl@ceph.mds.l.smithi173.stdout:May 27 07:29:26 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-l[73448]: 13: (Thread::entry_wrapper()+0x43) [0x7fd79766c72d] 2024-05-27T07:29:26.142 INFO:journalctl@ceph.mds.l.smithi173.stdout:May 27 07:29:26 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-l[73448]: 14: (Thread::_entry_func(void*)+0xd) [0x7fd79766c749] 2024-05-27T07:29:26.142 INFO:journalctl@ceph.mds.l.smithi173.stdout:May 27 07:29:26 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-l[73448]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7fd7962101ca] 2024-05-27T07:29:26.142 INFO:journalctl@ceph.mds.l.smithi173.stdout:May 27 07:29:26 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-l[73448]: 16: clone() 2024-05-27T07:29:26.142 INFO:journalctl@ceph.mds.l.smithi173.stdout:May 27 07:29:26 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-l[73448]: 2024-05-27T07:29:26.146+0000 7fd784e58700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: In function 'std::set SnapClient::filter(const std::set&) const' thread 7fd784e58700 time 2024-05-27T07:29:26.145123+0000 2024-05-27T07:29:26.143 INFO:journalctl@ceph.mds.l.smithi173.stdout:May 27 07:29:26 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-l[73448]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: 197: FAILED ceph_assert(cached_version > 0) 2024-05-27T07:29:26.143 INFO:journalctl@ceph.mds.l.smithi173.stdout:May 27 07:29:26 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-l[73448]: 2024-05-27T07:29:26.143 INFO:journalctl@ceph.mds.l.smithi173.stdout:May 27 07:29:26 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-l[73448]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:29:26.143 INFO:journalctl@ceph.mds.l.smithi173.stdout:May 27 07:29:26 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-l[73448]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x127) [0x7fd79769dba6] 2024-05-27T07:29:26.143 INFO:journalctl@ceph.mds.l.smithi173.stdout:May 27 07:29:26 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-l[73448]: 2: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7fd79769ddd1] 2024-05-27T07:29:26.144 INFO:journalctl@ceph.mds.l.smithi173.stdout:May 27 07:29:26 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-l[73448]: 3: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x563082efa23d] 2024-05-27T07:29:26.144 INFO:journalctl@ceph.mds.l.smithi173.stdout:May 27 07:29:26 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-l[73448]: 4: (SnapRealm::build_snap_set() const+0x323) [0x563082ee5971] 2024-05-27T07:29:26.144 INFO:journalctl@ceph.mds.l.smithi173.stdout:May 27 07:29:26 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-l[73448]: 5: (SnapRealm::check_cache() const+0x13e) [0x563082ee3fa0] 2024-05-27T07:29:26.144 INFO:journalctl@ceph.mds.l.smithi173.stdout:May 27 07:29:26 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-l[73448]: 6: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x563082ee4651] 2024-05-27T07:29:26.144 INFO:journalctl@ceph.mds.l.smithi173.stdout:May 27 07:29:26 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-l[73448]: 7: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x563082e79347] 2024-05-27T07:29:26.144 INFO:journalctl@ceph.mds.l.smithi173.stdout:May 27 07:29:26 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-l[73448]: 8: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x563082b7587e] 2024-05-27T07:29:26.144 INFO:journalctl@ceph.mds.l.smithi173.stdout:May 27 07:29:26 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-l[73448]: 9: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x2702) [0x563082b7ae94] 2024-05-27T07:29:26.144 INFO:journalctl@ceph.mds.l.smithi173.stdout:May 27 07:29:26 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-l[73448]: 10: (EUpdate::replay(MDSRank*)+0x63) [0x563082b85969] 2024-05-27T07:29:26.144 INFO:journalctl@ceph.mds.l.smithi173.stdout:May 27 07:29:26 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-l[73448]: 11: (MDLog::_replay_thread()+0x1f38) [0x563082f1e48c] 2024-05-27T07:29:26.144 INFO:journalctl@ceph.mds.l.smithi173.stdout:May 27 07:29:26 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-l[73448]: 12: (MDLog::ReplayThread::entry()+0x11) [0x563082f279cb] 2024-05-27T07:29:26.144 INFO:journalctl@ceph.mds.l.smithi173.stdout:May 27 07:29:26 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-l[73448]: 13: (Thread::entry_wrapper()+0x43) [0x7fd79766c72d] 2024-05-27T07:29:26.144 INFO:journalctl@ceph.mds.l.smithi173.stdout:May 27 07:29:26 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-l[73448]: 14: (Thread::_entry_func(void*)+0xd) [0x7fd79766c749] 2024-05-27T07:29:26.144 INFO:journalctl@ceph.mds.l.smithi173.stdout:May 27 07:29:26 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-l[73448]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7fd7962101ca] 2024-05-27T07:29:26.145 INFO:journalctl@ceph.mds.l.smithi173.stdout:May 27 07:29:26 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-l[73448]: 16: clone() 2024-05-27T07:29:26.145 INFO:journalctl@ceph.mds.l.smithi173.stdout:May 27 07:29:26 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-l[73448]: 2024-05-27T07:29:26.145 INFO:journalctl@ceph.mds.l.smithi173.stdout:May 27 07:29:26 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-l[73448]: *** Caught signal (Aborted) ** 2024-05-27T07:29:26.145 INFO:journalctl@ceph.mds.l.smithi173.stdout:May 27 07:29:26 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-l[73448]: in thread 7fd784e58700 thread_name:md_log_replay 2024-05-27T07:29:26.145 INFO:journalctl@ceph.mds.l.smithi173.stdout:May 27 07:29:26 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-l[73448]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:29:26.145 INFO:journalctl@ceph.mds.l.smithi173.stdout:May 27 07:29:26 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-l[73448]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7fd79621ad20] 2024-05-27T07:29:26.145 INFO:journalctl@ceph.mds.l.smithi173.stdout:May 27 07:29:26 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-l[73448]: 2: gsignal() 2024-05-27T07:29:26.145 INFO:journalctl@ceph.mds.l.smithi173.stdout:May 27 07:29:26 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-l[73448]: 3: abort() 2024-05-27T07:29:26.145 INFO:journalctl@ceph.mds.l.smithi173.stdout:May 27 07:29:26 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-l[73448]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x26e) [0x7fd79769dced] 2024-05-27T07:29:26.145 INFO:journalctl@ceph.mds.l.smithi173.stdout:May 27 07:29:26 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-l[73448]: 5: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7fd79769ddd1] 2024-05-27T07:29:26.145 INFO:journalctl@ceph.mds.l.smithi173.stdout:May 27 07:29:26 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-l[73448]: 6: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x563082efa23d] 2024-05-27T07:29:26.145 INFO:journalctl@ceph.mds.l.smithi173.stdout:May 27 07:29:26 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-l[73448]: 7: (SnapRealm::build_snap_set() const+0x323) [0x563082ee5971] 2024-05-27T07:29:26.145 INFO:journalctl@ceph.mds.l.smithi173.stdout:May 27 07:29:26 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-l[73448]: 8: (SnapRealm::check_cache() const+0x13e) [0x563082ee3fa0] 2024-05-27T07:29:26.146 INFO:journalctl@ceph.mds.l.smithi173.stdout:May 27 07:29:26 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-l[73448]: 9: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x563082ee4651] 2024-05-27T07:29:26.146 INFO:journalctl@ceph.mds.l.smithi173.stdout:May 27 07:29:26 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-l[73448]: 10: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x563082e79347] 2024-05-27T07:29:26.146 INFO:journalctl@ceph.mds.l.smithi173.stdout:May 27 07:29:26 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-l[73448]: 11: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x563082b7587e] 2024-05-27T07:29:26.146 INFO:journalctl@ceph.mds.l.smithi173.stdout:May 27 07:29:26 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-l[73448]: 12: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x2702) [0x563082b7ae94] 2024-05-27T07:29:26.146 INFO:journalctl@ceph.mds.l.smithi173.stdout:May 27 07:29:26 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-l[73448]: 13: (EUpdate::replay(MDSRank*)+0x63) [0x563082b85969] 2024-05-27T07:29:26.146 INFO:journalctl@ceph.mds.l.smithi173.stdout:May 27 07:29:26 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-l[73448]: 14: (MDLog::_replay_thread()+0x1f38) [0x563082f1e48c] 2024-05-27T07:29:26.146 INFO:journalctl@ceph.mds.l.smithi173.stdout:May 27 07:29:26 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-l[73448]: 15: (MDLog::ReplayThread::entry()+0x11) [0x563082f279cb] 2024-05-27T07:29:26.146 INFO:journalctl@ceph.mds.l.smithi173.stdout:May 27 07:29:26 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-l[73448]: 16: (Thread::entry_wrapper()+0x43) [0x7fd79766c72d] 2024-05-27T07:29:26.146 INFO:journalctl@ceph.mds.l.smithi173.stdout:May 27 07:29:26 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-l[73448]: 17: (Thread::_entry_func(void*)+0xd) [0x7fd79766c749] 2024-05-27T07:29:26.146 INFO:journalctl@ceph.mds.l.smithi173.stdout:May 27 07:29:26 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-l[73448]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7fd7962101ca] 2024-05-27T07:29:26.146 INFO:journalctl@ceph.mds.l.smithi173.stdout:May 27 07:29:26 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-l[73448]: 19: clone() 2024-05-27T07:29:26.146 INFO:journalctl@ceph.mds.l.smithi173.stdout:May 27 07:29:26 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-l[73448]: 2024-05-27T07:29:26.148+0000 7fd784e58700 -1 *** Caught signal (Aborted) ** 2024-05-27T07:29:26.146 INFO:journalctl@ceph.mds.l.smithi173.stdout:May 27 07:29:26 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-l[73448]: in thread 7fd784e58700 thread_name:md_log_replay 2024-05-27T07:29:26.146 INFO:journalctl@ceph.mds.l.smithi173.stdout:May 27 07:29:26 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-l[73448]: 2024-05-27T07:29:26.147 INFO:journalctl@ceph.mds.l.smithi173.stdout:May 27 07:29:26 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-l[73448]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:29:26.147 INFO:journalctl@ceph.mds.l.smithi173.stdout:May 27 07:29:26 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-l[73448]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7fd79621ad20] 2024-05-27T07:29:26.147 INFO:journalctl@ceph.mds.l.smithi173.stdout:May 27 07:29:26 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-l[73448]: 2: gsignal() 2024-05-27T07:29:26.147 INFO:journalctl@ceph.mds.l.smithi173.stdout:May 27 07:29:26 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-l[73448]: 3: abort() 2024-05-27T07:29:26.147 INFO:journalctl@ceph.mds.l.smithi173.stdout:May 27 07:29:26 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-l[73448]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x26e) [0x7fd79769dced] 2024-05-27T07:29:26.147 INFO:journalctl@ceph.mds.l.smithi173.stdout:May 27 07:29:26 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-l[73448]: 5: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7fd79769ddd1] 2024-05-27T07:29:26.147 INFO:journalctl@ceph.mds.l.smithi173.stdout:May 27 07:29:26 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-l[73448]: 6: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x563082efa23d] 2024-05-27T07:29:26.147 INFO:journalctl@ceph.mds.l.smithi173.stdout:May 27 07:29:26 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-l[73448]: 7: (SnapRealm::build_snap_set() const+0x323) [0x563082ee5971] 2024-05-27T07:29:26.147 INFO:journalctl@ceph.mds.l.smithi173.stdout:May 27 07:29:26 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-l[73448]: 8: (SnapRealm::check_cache() const+0x13e) [0x563082ee3fa0] 2024-05-27T07:29:26.147 INFO:journalctl@ceph.mds.l.smithi173.stdout:May 27 07:29:26 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-l[73448]: 9: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x563082ee4651] 2024-05-27T07:29:26.147 INFO:journalctl@ceph.mds.l.smithi173.stdout:May 27 07:29:26 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-l[73448]: 10: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x563082e79347] 2024-05-27T07:29:26.147 INFO:journalctl@ceph.mds.l.smithi173.stdout:May 27 07:29:26 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-l[73448]: 11: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x563082b7587e] 2024-05-27T07:29:26.147 INFO:journalctl@ceph.mds.l.smithi173.stdout:May 27 07:29:26 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-l[73448]: 12: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x2702) [0x563082b7ae94] 2024-05-27T07:29:26.148 INFO:journalctl@ceph.mds.l.smithi173.stdout:May 27 07:29:26 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-l[73448]: 13: (EUpdate::replay(MDSRank*)+0x63) [0x563082b85969] 2024-05-27T07:29:26.148 INFO:journalctl@ceph.mds.l.smithi173.stdout:May 27 07:29:26 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-l[73448]: 14: (MDLog::_replay_thread()+0x1f38) [0x563082f1e48c] 2024-05-27T07:29:26.148 INFO:journalctl@ceph.mds.l.smithi173.stdout:May 27 07:29:26 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-l[73448]: 15: (MDLog::ReplayThread::entry()+0x11) [0x563082f279cb] 2024-05-27T07:29:26.148 INFO:journalctl@ceph.mds.l.smithi173.stdout:May 27 07:29:26 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-l[73448]: 16: (Thread::entry_wrapper()+0x43) [0x7fd79766c72d] 2024-05-27T07:29:26.148 INFO:journalctl@ceph.mds.l.smithi173.stdout:May 27 07:29:26 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-l[73448]: 17: (Thread::_entry_func(void*)+0xd) [0x7fd79766c749] 2024-05-27T07:29:26.148 INFO:journalctl@ceph.mds.l.smithi173.stdout:May 27 07:29:26 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-l[73448]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7fd7962101ca] 2024-05-27T07:29:26.148 INFO:journalctl@ceph.mds.l.smithi173.stdout:May 27 07:29:26 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-l[73448]: 19: clone() 2024-05-27T07:29:26.148 INFO:journalctl@ceph.mds.l.smithi173.stdout:May 27 07:29:26 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-l[73448]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-05-27T07:29:26.148 INFO:journalctl@ceph.mds.l.smithi173.stdout:May 27 07:29:26 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-l[73448]: 2024-05-27T07:29:26.149 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:29:26 smithi052 ceph-mon[31955]: pgmap v118: 129 pgs: 129 active+clean; 209 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 2.8 MiB/s wr, 344 op/s 2024-05-27T07:29:26.149 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:29:26 smithi052 ceph-mon[31955]: daemon mds.l restarted 2024-05-27T07:29:26.149 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:29:26 smithi052 ceph-mon[31955]: osdmap e103: 12 total, 12 up, 12 in 2024-05-27T07:29:26.149 INFO:journalctl@ceph.mds.a.smithi052.stdout:May 27 07:29:25 smithi052 podman[104807]: 2024-05-27 07:29:25.904083416 +0000 UTC m=+0.838603269 container remove 46c806e123bbc1bcf4df0fc9657e67eba98bcf204f960943c2efea78585f35c4 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-a, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, org.label-schema.build-date=20240521, org.label-schema.schema-version=1.0, GIT_CLEAN=True, org.label-schema.vendor=CentOS, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5) 2024-05-27T07:29:26.149 INFO:journalctl@ceph.mds.a.smithi052.stdout:May 27 07:29:25 smithi052 systemd[1]: ceph-73df2636-1bf8-11ef-bc9b-c7b262605968@mds.a.service: Main process exited, code=exited, status=134/n/a 2024-05-27T07:29:26.352 INFO:journalctl@ceph.mds.l.smithi173.stdout:May 27 07:29:26 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-l[73448]: -1> 2024-05-27T07:29:26.146+0000 7fd784e58700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: In function 'std::set SnapClient::filter(const std::set&) const' thread 7fd784e58700 time 2024-05-27T07:29:26.145123+0000 2024-05-27T07:29:26.353 INFO:journalctl@ceph.mds.l.smithi173.stdout:May 27 07:29:26 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-l[73448]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: 197: FAILED ceph_assert(cached_version > 0) 2024-05-27T07:29:26.353 INFO:journalctl@ceph.mds.l.smithi173.stdout:May 27 07:29:26 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-l[73448]: 2024-05-27T07:29:26.353 INFO:journalctl@ceph.mds.l.smithi173.stdout:May 27 07:29:26 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-l[73448]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:29:26.353 INFO:journalctl@ceph.mds.l.smithi173.stdout:May 27 07:29:26 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-l[73448]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x127) [0x7fd79769dba6] 2024-05-27T07:29:26.353 INFO:journalctl@ceph.mds.l.smithi173.stdout:May 27 07:29:26 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-l[73448]: 2: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7fd79769ddd1] 2024-05-27T07:29:26.353 INFO:journalctl@ceph.mds.l.smithi173.stdout:May 27 07:29:26 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-l[73448]: 3: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x563082efa23d] 2024-05-27T07:29:26.353 INFO:journalctl@ceph.mds.l.smithi173.stdout:May 27 07:29:26 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-l[73448]: 4: (SnapRealm::build_snap_set() const+0x323) [0x563082ee5971] 2024-05-27T07:29:26.353 INFO:journalctl@ceph.mds.l.smithi173.stdout:May 27 07:29:26 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-l[73448]: 5: (SnapRealm::check_cache() const+0x13e) [0x563082ee3fa0] 2024-05-27T07:29:26.353 INFO:journalctl@ceph.mds.l.smithi173.stdout:May 27 07:29:26 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-l[73448]: 6: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x563082ee4651] 2024-05-27T07:29:26.353 INFO:journalctl@ceph.mds.l.smithi173.stdout:May 27 07:29:26 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-l[73448]: 7: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x563082e79347] 2024-05-27T07:29:26.353 INFO:journalctl@ceph.mds.l.smithi173.stdout:May 27 07:29:26 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-l[73448]: 8: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x563082b7587e] 2024-05-27T07:29:26.354 INFO:journalctl@ceph.mds.l.smithi173.stdout:May 27 07:29:26 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-l[73448]: 9: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x2702) [0x563082b7ae94] 2024-05-27T07:29:26.354 INFO:journalctl@ceph.mds.l.smithi173.stdout:May 27 07:29:26 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-l[73448]: 10: (EUpdate::replay(MDSRank*)+0x63) [0x563082b85969] 2024-05-27T07:29:26.354 INFO:journalctl@ceph.mds.l.smithi173.stdout:May 27 07:29:26 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-l[73448]: 11: (MDLog::_replay_thread()+0x1f38) [0x563082f1e48c] 2024-05-27T07:29:26.354 INFO:journalctl@ceph.mds.l.smithi173.stdout:May 27 07:29:26 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-l[73448]: 12: (MDLog::ReplayThread::entry()+0x11) [0x563082f279cb] 2024-05-27T07:29:26.354 INFO:journalctl@ceph.mds.l.smithi173.stdout:May 27 07:29:26 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-l[73448]: 13: (Thread::entry_wrapper()+0x43) [0x7fd79766c72d] 2024-05-27T07:29:26.354 INFO:journalctl@ceph.mds.l.smithi173.stdout:May 27 07:29:26 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-l[73448]: 14: (Thread::_entry_func(void*)+0xd) [0x7fd79766c749] 2024-05-27T07:29:26.354 INFO:journalctl@ceph.mds.l.smithi173.stdout:May 27 07:29:26 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-l[73448]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7fd7962101ca] 2024-05-27T07:29:26.354 INFO:journalctl@ceph.mds.l.smithi173.stdout:May 27 07:29:26 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-l[73448]: 16: clone() 2024-05-27T07:29:26.354 INFO:journalctl@ceph.mds.l.smithi173.stdout:May 27 07:29:26 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-l[73448]: 2024-05-27T07:29:26.354 INFO:journalctl@ceph.mds.l.smithi173.stdout:May 27 07:29:26 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-l[73448]: 0> 2024-05-27T07:29:26.148+0000 7fd784e58700 -1 *** Caught signal (Aborted) ** 2024-05-27T07:29:26.354 INFO:journalctl@ceph.mds.l.smithi173.stdout:May 27 07:29:26 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-l[73448]: in thread 7fd784e58700 thread_name:md_log_replay 2024-05-27T07:29:26.354 INFO:journalctl@ceph.mds.l.smithi173.stdout:May 27 07:29:26 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-l[73448]: 2024-05-27T07:29:26.355 INFO:journalctl@ceph.mds.l.smithi173.stdout:May 27 07:29:26 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-l[73448]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:29:26.355 INFO:journalctl@ceph.mds.l.smithi173.stdout:May 27 07:29:26 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-l[73448]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7fd79621ad20] 2024-05-27T07:29:26.355 INFO:journalctl@ceph.mds.l.smithi173.stdout:May 27 07:29:26 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-l[73448]: 2: gsignal() 2024-05-27T07:29:26.355 INFO:journalctl@ceph.mds.l.smithi173.stdout:May 27 07:29:26 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-l[73448]: 3: abort() 2024-05-27T07:29:26.355 INFO:journalctl@ceph.mds.l.smithi173.stdout:May 27 07:29:26 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-l[73448]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x26e) [0x7fd79769dced] 2024-05-27T07:29:26.355 INFO:journalctl@ceph.mds.l.smithi173.stdout:May 27 07:29:26 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-l[73448]: 5: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7fd79769ddd1] 2024-05-27T07:29:26.356 INFO:journalctl@ceph.mds.l.smithi173.stdout:May 27 07:29:26 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-l[73448]: 6: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x563082efa23d] 2024-05-27T07:29:26.356 INFO:journalctl@ceph.mds.l.smithi173.stdout:May 27 07:29:26 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-l[73448]: 7: (SnapRealm::build_snap_set() const+0x323) [0x563082ee5971] 2024-05-27T07:29:26.356 INFO:journalctl@ceph.mds.l.smithi173.stdout:May 27 07:29:26 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-l[73448]: 8: (SnapRealm::check_cache() const+0x13e) [0x563082ee3fa0] 2024-05-27T07:29:26.356 INFO:journalctl@ceph.mds.l.smithi173.stdout:May 27 07:29:26 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-l[73448]: 9: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x563082ee4651] 2024-05-27T07:29:26.356 INFO:journalctl@ceph.mds.l.smithi173.stdout:May 27 07:29:26 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-l[73448]: 10: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x563082e79347] 2024-05-27T07:29:26.356 INFO:journalctl@ceph.mds.l.smithi173.stdout:May 27 07:29:26 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-l[73448]: 11: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x563082b7587e] 2024-05-27T07:29:26.356 INFO:journalctl@ceph.mds.l.smithi173.stdout:May 27 07:29:26 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-l[73448]: 12: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x2702) [0x563082b7ae94] 2024-05-27T07:29:26.356 INFO:journalctl@ceph.mds.l.smithi173.stdout:May 27 07:29:26 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-l[73448]: 13: (EUpdate::replay(MDSRank*)+0x63) [0x563082b85969] 2024-05-27T07:29:26.356 INFO:journalctl@ceph.mds.l.smithi173.stdout:May 27 07:29:26 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-l[73448]: 14: (MDLog::_replay_thread()+0x1f38) [0x563082f1e48c] 2024-05-27T07:29:26.356 INFO:journalctl@ceph.mds.l.smithi173.stdout:May 27 07:29:26 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-l[73448]: 15: (MDLog::ReplayThread::entry()+0x11) [0x563082f279cb] 2024-05-27T07:29:26.356 INFO:journalctl@ceph.mds.l.smithi173.stdout:May 27 07:29:26 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-l[73448]: 16: (Thread::entry_wrapper()+0x43) [0x7fd79766c72d] 2024-05-27T07:29:26.356 INFO:journalctl@ceph.mds.l.smithi173.stdout:May 27 07:29:26 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-l[73448]: 17: (Thread::_entry_func(void*)+0xd) [0x7fd79766c749] 2024-05-27T07:29:26.356 INFO:journalctl@ceph.mds.l.smithi173.stdout:May 27 07:29:26 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-l[73448]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7fd7962101ca] 2024-05-27T07:29:26.357 INFO:journalctl@ceph.mds.l.smithi173.stdout:May 27 07:29:26 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-l[73448]: 19: clone() 2024-05-27T07:29:26.357 INFO:journalctl@ceph.mds.l.smithi173.stdout:May 27 07:29:26 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-l[73448]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-05-27T07:29:26.357 INFO:journalctl@ceph.mds.l.smithi173.stdout:May 27 07:29:26 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-l[73448]: 2024-05-27T07:29:26.357 INFO:journalctl@ceph.mds.l.smithi173.stdout:May 27 07:29:26 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-l[73448]: -1> 2024-05-27T07:29:26.146+0000 7fd784e58700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: In function 'std::set SnapClient::filter(const std::set&) const' thread 7fd784e58700 time 2024-05-27T07:29:26.145123+0000 2024-05-27T07:29:26.357 INFO:journalctl@ceph.mds.l.smithi173.stdout:May 27 07:29:26 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-l[73448]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: 197: FAILED ceph_assert(cached_version > 0) 2024-05-27T07:29:26.357 INFO:journalctl@ceph.mds.l.smithi173.stdout:May 27 07:29:26 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-l[73448]: 2024-05-27T07:29:26.357 INFO:journalctl@ceph.mds.l.smithi173.stdout:May 27 07:29:26 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-l[73448]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:29:26.357 INFO:journalctl@ceph.mds.l.smithi173.stdout:May 27 07:29:26 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-l[73448]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x127) [0x7fd79769dba6] 2024-05-27T07:29:26.357 INFO:journalctl@ceph.mds.l.smithi173.stdout:May 27 07:29:26 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-l[73448]: 2: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7fd79769ddd1] 2024-05-27T07:29:26.357 INFO:journalctl@ceph.mds.l.smithi173.stdout:May 27 07:29:26 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-l[73448]: 3: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x563082efa23d] 2024-05-27T07:29:26.357 INFO:journalctl@ceph.mds.l.smithi173.stdout:May 27 07:29:26 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-l[73448]: 4: (SnapRealm::build_snap_set() const+0x323) [0x563082ee5971] 2024-05-27T07:29:26.357 INFO:journalctl@ceph.mds.l.smithi173.stdout:May 27 07:29:26 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-l[73448]: 5: (SnapRealm::check_cache() const+0x13e) [0x563082ee3fa0] 2024-05-27T07:29:26.358 INFO:journalctl@ceph.mds.l.smithi173.stdout:May 27 07:29:26 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-l[73448]: 6: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x563082ee4651] 2024-05-27T07:29:26.358 INFO:journalctl@ceph.mds.l.smithi173.stdout:May 27 07:29:26 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-l[73448]: 7: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x563082e79347] 2024-05-27T07:29:26.358 INFO:journalctl@ceph.mds.l.smithi173.stdout:May 27 07:29:26 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-l[73448]: 8: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x563082b7587e] 2024-05-27T07:29:26.358 INFO:journalctl@ceph.mds.l.smithi173.stdout:May 27 07:29:26 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-l[73448]: 9: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x2702) [0x563082b7ae94] 2024-05-27T07:29:26.358 INFO:journalctl@ceph.mds.l.smithi173.stdout:May 27 07:29:26 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-l[73448]: 10: (EUpdate::replay(MDSRank*)+0x63) [0x563082b85969] 2024-05-27T07:29:26.358 INFO:journalctl@ceph.mds.l.smithi173.stdout:May 27 07:29:26 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-l[73448]: 11: (MDLog::_replay_thread()+0x1f38) [0x563082f1e48c] 2024-05-27T07:29:26.358 INFO:journalctl@ceph.mds.l.smithi173.stdout:May 27 07:29:26 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-l[73448]: 12: (MDLog::ReplayThread::entry()+0x11) [0x563082f279cb] 2024-05-27T07:29:26.358 INFO:journalctl@ceph.mds.l.smithi173.stdout:May 27 07:29:26 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-l[73448]: 13: (Thread::entry_wrapper()+0x43) [0x7fd79766c72d] 2024-05-27T07:29:26.358 INFO:journalctl@ceph.mds.l.smithi173.stdout:May 27 07:29:26 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-l[73448]: 14: (Thread::_entry_func(void*)+0xd) [0x7fd79766c749] 2024-05-27T07:29:26.358 INFO:journalctl@ceph.mds.l.smithi173.stdout:May 27 07:29:26 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-l[73448]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7fd7962101ca] 2024-05-27T07:29:26.358 INFO:journalctl@ceph.mds.l.smithi173.stdout:May 27 07:29:26 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-l[73448]: 16: clone() 2024-05-27T07:29:26.358 INFO:journalctl@ceph.mds.l.smithi173.stdout:May 27 07:29:26 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-l[73448]: 2024-05-27T07:29:26.358 INFO:journalctl@ceph.mds.l.smithi173.stdout:May 27 07:29:26 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-l[73448]: 0> 2024-05-27T07:29:26.148+0000 7fd784e58700 -1 *** Caught signal (Aborted) ** 2024-05-27T07:29:26.359 INFO:journalctl@ceph.mds.l.smithi173.stdout:May 27 07:29:26 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-l[73448]: in thread 7fd784e58700 thread_name:md_log_replay 2024-05-27T07:29:26.359 INFO:journalctl@ceph.mds.l.smithi173.stdout:May 27 07:29:26 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-l[73448]: 2024-05-27T07:29:26.359 INFO:journalctl@ceph.mds.l.smithi173.stdout:May 27 07:29:26 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-l[73448]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:29:26.359 INFO:journalctl@ceph.mds.l.smithi173.stdout:May 27 07:29:26 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-l[73448]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7fd79621ad20] 2024-05-27T07:29:26.359 INFO:journalctl@ceph.mds.l.smithi173.stdout:May 27 07:29:26 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-l[73448]: 2: gsignal() 2024-05-27T07:29:26.359 INFO:journalctl@ceph.mds.l.smithi173.stdout:May 27 07:29:26 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-l[73448]: 3: abort() 2024-05-27T07:29:26.359 INFO:journalctl@ceph.mds.l.smithi173.stdout:May 27 07:29:26 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-l[73448]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x26e) [0x7fd79769dced] 2024-05-27T07:29:26.359 INFO:journalctl@ceph.mds.l.smithi173.stdout:May 27 07:29:26 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-l[73448]: 5: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7fd79769ddd1] 2024-05-27T07:29:26.359 INFO:journalctl@ceph.mds.l.smithi173.stdout:May 27 07:29:26 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-l[73448]: 6: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x563082efa23d] 2024-05-27T07:29:26.359 INFO:journalctl@ceph.mds.l.smithi173.stdout:May 27 07:29:26 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-l[73448]: 7: (SnapRealm::build_snap_set() const+0x323) [0x563082ee5971] 2024-05-27T07:29:26.359 INFO:journalctl@ceph.mds.l.smithi173.stdout:May 27 07:29:26 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-l[73448]: 8: (SnapRealm::check_cache() const+0x13e) [0x563082ee3fa0] 2024-05-27T07:29:26.359 INFO:journalctl@ceph.mds.l.smithi173.stdout:May 27 07:29:26 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-l[73448]: 9: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x563082ee4651] 2024-05-27T07:29:26.359 INFO:journalctl@ceph.mds.l.smithi173.stdout:May 27 07:29:26 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-l[73448]: 10: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x563082e79347] 2024-05-27T07:29:26.360 INFO:journalctl@ceph.mds.l.smithi173.stdout:May 27 07:29:26 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-l[73448]: 11: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x563082b7587e] 2024-05-27T07:29:26.360 INFO:journalctl@ceph.mds.l.smithi173.stdout:May 27 07:29:26 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-l[73448]: 12: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x2702) [0x563082b7ae94] 2024-05-27T07:29:26.360 INFO:journalctl@ceph.mds.l.smithi173.stdout:May 27 07:29:26 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-l[73448]: 13: (EUpdate::replay(MDSRank*)+0x63) [0x563082b85969] 2024-05-27T07:29:26.360 INFO:journalctl@ceph.mds.l.smithi173.stdout:May 27 07:29:26 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-l[73448]: 14: (MDLog::_replay_thread()+0x1f38) [0x563082f1e48c] 2024-05-27T07:29:26.360 INFO:journalctl@ceph.mds.l.smithi173.stdout:May 27 07:29:26 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-l[73448]: 15: (MDLog::ReplayThread::entry()+0x11) [0x563082f279cb] 2024-05-27T07:29:26.360 INFO:journalctl@ceph.mds.l.smithi173.stdout:May 27 07:29:26 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-l[73448]: 16: (Thread::entry_wrapper()+0x43) [0x7fd79766c72d] 2024-05-27T07:29:26.360 INFO:journalctl@ceph.mds.l.smithi173.stdout:May 27 07:29:26 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-l[73448]: 17: (Thread::_entry_func(void*)+0xd) [0x7fd79766c749] 2024-05-27T07:29:26.360 INFO:journalctl@ceph.mds.l.smithi173.stdout:May 27 07:29:26 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-l[73448]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7fd7962101ca] 2024-05-27T07:29:26.360 INFO:journalctl@ceph.mds.l.smithi173.stdout:May 27 07:29:26 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-l[73448]: 19: clone() 2024-05-27T07:29:26.360 INFO:journalctl@ceph.mds.l.smithi173.stdout:May 27 07:29:26 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-l[73448]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-05-27T07:29:26.360 INFO:journalctl@ceph.mds.l.smithi173.stdout:May 27 07:29:26 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-l[73448]: 2024-05-27T07:29:26.360 INFO:journalctl@ceph.mds.l.smithi173.stdout:May 27 07:29:26 smithi173 podman[73492]: 2024-05-27 07:29:26.20991799 +0000 UTC m=+0.019274925 container died 686d1400cad7235ed70ce609ed1204dc003f1dfb6ad6c3e84cea045a4e9fff4d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-l, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240521, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, ceph=True, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD) 2024-05-27T07:29:26.410 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:29:26 smithi105 ceph-mon[35977]: pgmap v118: 129 pgs: 129 active+clean; 209 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 2.8 MiB/s wr, 344 op/s 2024-05-27T07:29:26.411 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:29:26 smithi105 ceph-mon[35977]: daemon mds.l restarted 2024-05-27T07:29:26.411 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:29:26 smithi105 ceph-mon[35977]: osdmap e103: 12 total, 12 up, 12 in 2024-05-27T07:29:26.652 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:29:26.420+0000 7f419505f700 1 -- 172.21.15.52:0/3590117640 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f41901034b0 msgr2=0x7f41901038c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:29:26.652 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:29:26.420+0000 7f419505f700 1 --2- 172.21.15.52:0/3590117640 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f41901034b0 0x7f41901038c0 secure :-1 s=READY pgs=602 cs=0 l=1 rev1=1 crypto rx=0x7f41800099c0 tx=0x7f4180009cd0 comp rx=0 tx=0).stop 2024-05-27T07:29:26.652 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:29:26.420+0000 7f419505f700 1 -- 172.21.15.52:0/3590117640 shutdown_connections 2024-05-27T07:29:26.652 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:29:26.420+0000 7f419505f700 1 --2- 172.21.15.52:0/3590117640 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f4190108730 0x7f4190110090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:29:26.652 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:29:26.420+0000 7f419505f700 1 --2- 172.21.15.52:0/3590117640 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f41901034b0 0x7f41901038c0 unknown :-1 s=CLOSED pgs=602 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:29:26.652 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:29:26.420+0000 7f419505f700 1 --2- 172.21.15.52:0/3590117640 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f4190107de0 0x7f41901081f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:29:26.652 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:29:26.420+0000 7f419505f700 1 -- 172.21.15.52:0/3590117640 >> 172.21.15.52:0/3590117640 conn(0x7f41900fede0 msgr2=0x7f4190101200 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:29:26.652 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:29:26.420+0000 7f419505f700 1 -- 172.21.15.52:0/3590117640 shutdown_connections 2024-05-27T07:29:26.652 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:29:26.420+0000 7f419505f700 1 -- 172.21.15.52:0/3590117640 wait complete. 2024-05-27T07:29:26.652 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:29:26.421+0000 7f419505f700 1 Processor -- start 2024-05-27T07:29:26.652 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:29:26.421+0000 7f419505f700 1 -- start start 2024-05-27T07:29:26.653 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:29:26.421+0000 7f419505f700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f41901034b0 0x7f419019ae20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:29:26.653 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:29:26.421+0000 7f419505f700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f4190107de0 0x7f419019b3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:29:26.653 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:29:26.421+0000 7f419505f700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f4190108730 0x7f419019f650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:29:26.653 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:29:26.421+0000 7f419505f700 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x7f4190113360 con 0x7f4190107de0 2024-05-27T07:29:26.653 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:29:26.421+0000 7f419505f700 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f41901131e0 con 0x7f41901034b0 2024-05-27T07:29:26.653 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:29:26.421+0000 7f419505f700 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f41901134e0 con 0x7f4190108730 2024-05-27T07:29:26.653 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:29:26.422+0000 7f418e59c700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f4190107de0 0x7f419019b3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:29:26.653 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:29:26.422+0000 7f418e59c700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f4190107de0 0x7f419019b3c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.52:54352/0 (socket says 172.21.15.52:54352) 2024-05-27T07:29:26.653 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:29:26.422+0000 7f418ed9d700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f41901034b0 0x7f419019ae20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:29:26.653 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:29:26.422+0000 7f418e59c700 1 -- 172.21.15.52:0/1672060335 learned_addr learned my addr 172.21.15.52:0/1672060335 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-05-27T07:29:26.653 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:29:26.422+0000 7f418f59e700 1 --2- 172.21.15.52:0/1672060335 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f4190108730 0x7f419019f650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:29:26.653 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:29:26.422+0000 7f418e59c700 1 -- 172.21.15.52:0/1672060335 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f4190108730 msgr2=0x7f419019f650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:29:26.653 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:29:26.422+0000 7f418e59c700 1 --2- 172.21.15.52:0/1672060335 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f4190108730 0x7f419019f650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:29:26.654 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:29:26.422+0000 7f418e59c700 1 -- 172.21.15.52:0/1672060335 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f41901034b0 msgr2=0x7f419019ae20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:29:26.654 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:29:26.422+0000 7f418e59c700 1 --2- 172.21.15.52:0/1672060335 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f41901034b0 0x7f419019ae20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:29:26.654 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:29:26.422+0000 7f418e59c700 1 -- 172.21.15.52:0/1672060335 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f419019fcf0 con 0x7f4190107de0 2024-05-27T07:29:26.654 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:29:26.422+0000 7f418ed9d700 1 --2- 172.21.15.52:0/1672060335 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f41901034b0 0x7f419019ae20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-27T07:29:26.654 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:29:26.422+0000 7f418e59c700 1 --2- 172.21.15.52:0/1672060335 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f4190107de0 0x7f419019b3c0 secure :-1 s=READY pgs=603 cs=0 l=1 rev1=1 crypto rx=0x7f418400e970 tx=0x7f418400ec80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:29:26.654 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:29:26.422+0000 7f4177fff700 1 -- 172.21.15.52:0/1672060335 <== mon.0 v2:172.21.15.52:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4184004a30 con 0x7f4190107de0 2024-05-27T07:29:26.654 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:29:26.422+0000 7f419505f700 1 -- 172.21.15.52:0/1672060335 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f419019ffe0 con 0x7f4190107de0 2024-05-27T07:29:26.654 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:29:26.422+0000 7f4177fff700 1 -- 172.21.15.52:0/1672060335 <== mon.0 v2:172.21.15.52:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f418400cc90 con 0x7f4190107de0 2024-05-27T07:29:26.654 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:29:26.422+0000 7f4177fff700 1 -- 172.21.15.52:0/1672060335 <== mon.0 v2:172.21.15.52:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4184005610 con 0x7f4190107de0 2024-05-27T07:29:26.654 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:29:26.422+0000 7f419505f700 1 -- 172.21.15.52:0/1672060335 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f41901a7900 con 0x7f4190107de0 2024-05-27T07:29:26.654 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:29:26.425+0000 7f4177fff700 1 -- 172.21.15.52:0/1672060335 <== mon.0 v2:172.21.15.52:3300/0 4 ==== mgrmap(e 26) ==== 149783+0+0 (secure 0 0 0) 0x7f418400c7e0 con 0x7f4190107de0 2024-05-27T07:29:26.654 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:29:26.426+0000 7f4177fff700 1 --2- 172.21.15.52:0/1672060335 >> [v2:172.21.15.52:6800/1822318384,v1:172.21.15.52:6801/1822318384] conn(0x7f41780b29f0 0x7f41780b4f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:29:26.655 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:29:26.426+0000 7f4177fff700 1 -- 172.21.15.52:0/1672060335 <== mon.0 v2:172.21.15.52:3300/0 5 ==== osd_map(103..103 src has 1..103) ==== 9379+0+0 (secure 0 0 0) 0x7f41840e2ad0 con 0x7f4190107de0 2024-05-27T07:29:26.655 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:29:26.426+0000 7f418ed9d700 1 --2- 172.21.15.52:0/1672060335 >> [v2:172.21.15.52:6800/1822318384,v1:172.21.15.52:6801/1822318384] conn(0x7f41780b29f0 0x7f41780b4f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:29:26.655 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:29:26.427+0000 7f418ed9d700 1 --2- 172.21.15.52:0/1672060335 >> [v2:172.21.15.52:6800/1822318384,v1:172.21.15.52:6801/1822318384] conn(0x7f41780b29f0 0x7f41780b4f00 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7f4180000c00 tx=0x7f418001a040 comp rx=0 tx=0).ready entity=mgr.15333 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:29:26.655 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:29:26.427+0000 7f41a3a74b80 1 Processor -- start 2024-05-27T07:29:26.655 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:29:26.427+0000 7f41a3a74b80 1 -- start start 2024-05-27T07:29:26.655 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:29:26.427+0000 7f41a3a74b80 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55acfb2d2e40 0x55acfb456710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:29:26.655 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:29:26.427+0000 7f41a3a74b80 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55acfb456c50 0x55acfb450a40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:29:26.655 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:29:26.427+0000 7f41a3a74b80 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55acfb450f80 0x55acfb451390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:29:26.655 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:29:26.427+0000 7f41a3a74b80 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x55acfb37fc90 con 0x55acfb2d2e40 2024-05-27T07:29:26.655 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:29:26.427+0000 7f41a3a74b80 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x55acfb36f830 con 0x55acfb456c50 2024-05-27T07:29:26.655 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:29:26.427+0000 7f41a3a74b80 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x55acfb373870 con 0x55acfb450f80 2024-05-27T07:29:26.655 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:29:26.427+0000 7f418ed9d700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55acfb2d2e40 0x55acfb456710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:29:26.656 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:29:26.427+0000 7f418ed9d700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55acfb2d2e40 0x55acfb456710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.52:54358/0 (socket says 172.21.15.52:54358) 2024-05-27T07:29:26.656 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:29:26.427+0000 7f418ed9d700 1 -- 172.21.15.52:0/947814782 learned_addr learned my addr 172.21.15.52:0/947814782 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-05-27T07:29:26.656 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:29:26.427+0000 7f418e59c700 1 --2- 172.21.15.52:0/947814782 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55acfb456c50 0x55acfb450a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:29:26.656 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:29:26.427+0000 7f418f59e700 1 --2- 172.21.15.52:0/947814782 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55acfb450f80 0x55acfb451390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:29:26.656 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:29:26.428+0000 7f418ed9d700 1 -- 172.21.15.52:0/947814782 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55acfb450f80 msgr2=0x55acfb451390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:29:26.656 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:29:26.428+0000 7f418ed9d700 1 --2- 172.21.15.52:0/947814782 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55acfb450f80 0x55acfb451390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:29:26.656 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:29:26.428+0000 7f418ed9d700 1 -- 172.21.15.52:0/947814782 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55acfb456c50 msgr2=0x55acfb450a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:29:26.656 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:29:26.428+0000 7f418ed9d700 1 --2- 172.21.15.52:0/947814782 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55acfb456c50 0x55acfb450a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:29:26.656 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:29:26.428+0000 7f418ed9d700 1 -- 172.21.15.52:0/947814782 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55acfb2bf8d0 con 0x55acfb2d2e40 2024-05-27T07:29:26.656 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:29:26.428+0000 7f418f59e700 1 --2- 172.21.15.52:0/947814782 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55acfb450f80 0x55acfb451390 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-27T07:29:26.656 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:29:26.428+0000 7f418ed9d700 1 --2- 172.21.15.52:0/947814782 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55acfb2d2e40 0x55acfb456710 secure :-1 s=READY pgs=604 cs=0 l=1 rev1=1 crypto rx=0x7f41800099c0 tx=0x7f4180022000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dc321cc54b38d3f2 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:29:26.656 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:29:26.428+0000 7f41757fa700 1 -- 172.21.15.52:0/947814782 <== mon.0 v2:172.21.15.52:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f41800037c0 con 0x55acfb2d2e40 2024-05-27T07:29:26.657 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:29:26.428+0000 7f41757fa700 1 -- 172.21.15.52:0/947814782 <== mon.0 v2:172.21.15.52:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f4180003e10 con 0x55acfb2d2e40 2024-05-27T07:29:26.657 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:29:26.428+0000 7f41757fa700 1 -- 172.21.15.52:0/947814782 <== mon.0 v2:172.21.15.52:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f418000fbd0 con 0x55acfb2d2e40 2024-05-27T07:29:26.657 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:29:26.429+0000 7f41a3a74b80 1 -- 172.21.15.52:0/947814782 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55acfb2d2e40 msgr2=0x55acfb456710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:29:26.657 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:29:26.429+0000 7f41a3a74b80 1 --2- 172.21.15.52:0/947814782 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55acfb2d2e40 0x55acfb456710 secure :-1 s=READY pgs=604 cs=0 l=1 rev1=1 crypto rx=0x7f41800099c0 tx=0x7f4180022000 comp rx=0 tx=0).stop 2024-05-27T07:29:26.657 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:29:26.429+0000 7f41a3a74b80 1 -- 172.21.15.52:0/947814782 shutdown_connections 2024-05-27T07:29:26.657 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:29:26.429+0000 7f41a3a74b80 1 --2- 172.21.15.52:0/947814782 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55acfb450f80 0x55acfb451390 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:29:26.657 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:29:26.429+0000 7f41a3a74b80 1 --2- 172.21.15.52:0/947814782 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55acfb2d2e40 0x55acfb456710 unknown :-1 s=CLOSED pgs=604 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:29:26.657 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:29:26.429+0000 7f41a3a74b80 1 --2- 172.21.15.52:0/947814782 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55acfb456c50 0x55acfb450a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:29:26.657 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:29:26.429+0000 7f41a3a74b80 1 -- 172.21.15.52:0/947814782 >> 172.21.15.52:0/947814782 conn(0x55acfb32daf0 msgr2=0x55acfb44b640 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:29:26.657 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:29:26.429+0000 7f41a3a74b80 1 -- 172.21.15.52:0/947814782 shutdown_connections 2024-05-27T07:29:26.657 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:29:26.429+0000 7f41a3a74b80 1 -- 172.21.15.52:0/947814782 wait complete. 2024-05-27T07:29:26.657 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:29:26.430+0000 7f41a3a74b80 1 -- start start 2024-05-27T07:29:26.658 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:29:26.430+0000 7f41a3a74b80 1 Processor -- start 2024-05-27T07:29:26.658 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:29:26.430+0000 7f41a3a74b80 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55acfb2d2e40 0x55acfb464970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:29:26.658 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:29:26.430+0000 7f41a3a74b80 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55acfb450f80 0x55acfb464eb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:29:26.658 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:29:26.430+0000 7f41a3a74b80 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55acfb456c50 0x55acfb45ed70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:29:26.658 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:29:26.430+0000 7f418f59e700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55acfb456c50 0x55acfb45ed70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:29:26.658 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:29:26.430+0000 7f418e59c700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55acfb450f80 0x55acfb464eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:29:26.658 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:29:26.430+0000 7f41a3a74b80 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x55acfb275fa0 con 0x55acfb456c50 2024-05-27T07:29:26.658 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:29:26.430+0000 7f41a3a74b80 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x55acfb3117c0 con 0x55acfb450f80 2024-05-27T07:29:26.658 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:29:26.430+0000 7f418f59e700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55acfb456c50 0x55acfb45ed70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.52:54362/0 (socket says 172.21.15.52:54362) 2024-05-27T07:29:26.658 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:29:26.430+0000 7f418e59c700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55acfb450f80 0x55acfb464eb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.52:42958/0 (socket says 172.21.15.52:42958) 2024-05-27T07:29:26.658 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:29:26.430+0000 7f418f59e700 1 -- 172.21.15.52:0/334686769 learned_addr learned my addr 172.21.15.52:0/334686769 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-05-27T07:29:26.658 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:29:26.430+0000 7f418ed9d700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55acfb2d2e40 0x55acfb464970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:29:26.659 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:29:26.430+0000 7f41a3a74b80 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x55acfb45d6e0 con 0x55acfb2d2e40 2024-05-27T07:29:26.659 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:29:26.431+0000 7f41a3a74b80 10 client.0 _ll_register_callbacks cb 0x55acfb30a920 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T07:29:26.659 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:29:26.431+0000 7f418ed9d700 1 -- 172.21.15.52:0/334686769 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55acfb450f80 msgr2=0x55acfb464eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:29:26.659 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:29:26.431+0000 7f418ed9d700 1 --2- 172.21.15.52:0/334686769 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55acfb450f80 0x55acfb464eb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:29:26.659 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:29:26.431+0000 7f418ed9d700 1 -- 172.21.15.52:0/334686769 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55acfb456c50 msgr2=0x55acfb45ed70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:29:26.659 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:29:26.431+0000 7f418ed9d700 1 --2- 172.21.15.52:0/334686769 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55acfb456c50 0x55acfb45ed70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:29:26.659 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:29:26.431+0000 7f418ed9d700 1 -- 172.21.15.52:0/334686769 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55acfb30e690 con 0x55acfb2d2e40 2024-05-27T07:29:26.659 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:29:26.431+0000 7f418ed9d700 1 --2- 172.21.15.52:0/334686769 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55acfb2d2e40 0x55acfb464970 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7f4180009990 tx=0x7f418000f690 comp rx=0 tx=0).ready entity=mon.1 client_cookie=da5bdd9520f4fe30 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:29:26.659 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:29:26.431+0000 7f4172ff5700 10 client.0 ms_handle_connect on v2:172.21.15.173:3300/0 2024-05-27T07:29:26.659 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:29:26.431+0000 7f4172ff5700 1 -- 172.21.15.52:0/334686769 <== mon.1 v2:172.21.15.173:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f418000fa80 con 0x55acfb2d2e40 2024-05-27T07:29:26.659 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:29:26.431+0000 7f4172ff5700 1 -- 172.21.15.52:0/334686769 <== mon.1 v2:172.21.15.173:3300/0 2 ==== osd_map(103..103 src has 1..103) ==== 9379+0+0 (secure 0 0 0) 0x7f418000fc20 con 0x55acfb2d2e40 2024-05-27T07:29:26.659 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:29:26.431+0000 7f41a3a74b80 1 -- 172.21.15.52:0/334686769 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f4180003e10 con 0x55acfb2d2e40 2024-05-27T07:29:26.659 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:29:26.432+0000 7f4172ff5700 1 -- 172.21.15.52:0/334686769 <== mon.1 v2:172.21.15.173:3300/0 3 ==== mon_get_version_reply(handle=1 version=58) ==== 24+0+0 (secure 0 0 0) 0x7f41800037c0 con 0x55acfb2d2e40 2024-05-27T07:29:26.659 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:29:26.432+0000 7f41a3a74b80 10 client.25840 fetch_fsmap learned FSMap version 58 2024-05-27T07:29:26.660 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:29:26.432+0000 7f41a3a74b80 1 -- 172.21.15.52:0/334686769 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({config=0+,fsmap=58,monmap=4+}) -- 0x55acfb46c550 con 0x55acfb2d2e40 2024-05-27T07:29:26.660 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:29:26.432+0000 7f4172ff5700 1 -- 172.21.15.52:0/334686769 <== mon.1 v2:172.21.15.173:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f4180030480 con 0x55acfb2d2e40 2024-05-27T07:29:26.660 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:29:26.432+0000 7f4172ff5700 1 -- 172.21.15.52:0/334686769 <== mon.1 v2:172.21.15.173:3300/0 5 ==== fsmap(e 58) ==== 7773+0+0 (secure 0 0 0) 0x7f418003a460 con 0x55acfb2d2e40 2024-05-27T07:29:26.660 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:29:26.432+0000 7f41a3a74b80 10 client.25840 fetch_fsmap finished waiting for FSMap version 58 2024-05-27T07:29:26.660 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:29:26.432+0000 7f41a3a74b80 10 client.25840 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-05-27T07:29:26.660 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:29:26.432+0000 7f41a3a74b80 20 client.25840 populate_metadata read hostname 'smithi052' 2024-05-27T07:29:26.660 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:29:26.432+0000 7f41a3a74b80 1 --2- 172.21.15.52:0/334686769 >> [v2:172.21.15.105:6838/500348151,v1:172.21.15.105:6839/500348151] conn(0x55acfb46ea20 0x55acfb470e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:29:26.660 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:29:26.432+0000 7f41a3a74b80 4 client.25840 mds_command: new command op to 24527 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-27T07:29:26.660 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:29:26.432+0000 7f41a3a74b80 1 -- 172.21.15.52:0/334686769 --> [v2:172.21.15.105:6838/500348151,v1:172.21.15.105:6839/500348151] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x7f4180003e10 con 0x55acfb46ea20 2024-05-27T07:29:26.660 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:29:26.433+0000 7f418e59c700 1 --2- 172.21.15.52:0/334686769 >> [v2:172.21.15.105:6838/500348151,v1:172.21.15.105:6839/500348151] conn(0x55acfb46ea20 0x55acfb470e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:29:26.660 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:29:26.433+0000 7f4172ff5700 10 client.25840 ms_handle_connect on v2:172.21.15.105:6838/500348151 2024-05-27T07:29:26.660 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:29:26.433+0000 7f418e59c700 1 --2- 172.21.15.52:0/334686769 >> [v2:172.21.15.105:6838/500348151,v1:172.21.15.105:6839/500348151] conn(0x55acfb46ea20 0x55acfb470e40 crc :-1 s=READY pgs=93 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=b85da19234c17627 server_cookie=c223f9e5874a0478 in_seq=0 out_seq=0 2024-05-27T07:29:26.661 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:29:26.436+0000 7f4172ff5700 1 -- 172.21.15.52:0/334686769 <== mds.0 v2:172.21.15.105:6838/500348151 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24041 (crc 0 0 0) 0x7f4180003e10 con 0x55acfb46ea20 2024-05-27T07:29:26.661 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:29:26.436+0000 7f4172ff5700 10 client.25840 handle_command_reply: tid=0 2024-05-27T07:29:26.661 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:29:26.436+0000 7f41a3a74b80 1 client.25840 shutdown 2024-05-27T07:29:26.661 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:29:26.436+0000 7f41a3a74b80 1 -- 172.21.15.52:0/334686769 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55acfb2d2e40 msgr2=0x55acfb464970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:29:26.661 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:29:26.436+0000 7f41a3a74b80 1 --2- 172.21.15.52:0/334686769 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55acfb2d2e40 0x55acfb464970 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7f4180009990 tx=0x7f418000f690 comp rx=0 tx=0).stop 2024-05-27T07:29:26.661 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:29:26.436+0000 7f41a3a74b80 1 -- 172.21.15.52:0/334686769 shutdown_connections 2024-05-27T07:29:26.661 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:29:26.436+0000 7f41a3a74b80 1 --2- 172.21.15.52:0/334686769 >> [v2:172.21.15.105:6838/500348151,v1:172.21.15.105:6839/500348151] conn(0x55acfb46ea20 0x55acfb470e40 crc :-1 s=READY pgs=93 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:29:26.661 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:29:26.436+0000 7f41a3a74b80 1 --2- 172.21.15.52:0/334686769 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55acfb456c50 0x55acfb45ed70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:29:26.661 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:29:26.436+0000 7f41a3a74b80 1 --2- 172.21.15.52:0/334686769 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55acfb2d2e40 0x55acfb464970 unknown :-1 s=CLOSED pgs=113 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:29:26.661 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:29:26.436+0000 7f4172ff5700 0 client.25840 ms_handle_reset on v2:172.21.15.105:6838/500348151 2024-05-27T07:29:26.661 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:29:26.436+0000 7f41a3a74b80 1 --2- 172.21.15.52:0/334686769 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55acfb450f80 0x55acfb464eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:29:26.661 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:29:26.436+0000 7f41a3a74b80 1 -- 172.21.15.52:0/334686769 >> 172.21.15.52:0/334686769 conn(0x55acfb32daf0 msgr2=0x55acfb44fa30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:29:26.661 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:29:26.436+0000 7f41a3a74b80 1 -- 172.21.15.52:0/334686769 shutdown_connections 2024-05-27T07:29:26.661 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:29:26.436+0000 7f41a3a74b80 1 -- 172.21.15.52:0/334686769 wait complete. 2024-05-27T07:29:26.662 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:29:26.436+0000 7f41a3a74b80 20 client.25840 trim_cache size 0 max 16384 2024-05-27T07:29:26.662 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:29:26.459+0000 7f41a3a74b80 1 Processor -- start 2024-05-27T07:29:26.662 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:29:26.459+0000 7f41a3a74b80 1 -- start start 2024-05-27T07:29:26.662 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:29:26.459+0000 7f41a3a74b80 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55acfb2d2e40 0x55acfb45c6d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:29:26.662 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:29:26.459+0000 7f41a3a74b80 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55acfb450f80 0x55acfb45cc10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:29:26.662 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:29:26.459+0000 7f41a3a74b80 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55acfb456c50 0x55acfb464620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:29:26.662 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:29:26.459+0000 7f41a3a74b80 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x7f41901afe10 con 0x55acfb456c50 2024-05-27T07:29:26.662 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:29:26.459+0000 7f41a3a74b80 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f41901adb40 con 0x55acfb450f80 2024-05-27T07:29:26.662 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:29:26.459+0000 7f41a3a74b80 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f41901af740 con 0x55acfb2d2e40 2024-05-27T07:29:26.662 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:29:26.459+0000 7f418f59e700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55acfb456c50 0x55acfb464620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:29:26.662 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:29:26.459+0000 7f418f59e700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55acfb456c50 0x55acfb464620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.52:54372/0 (socket says 172.21.15.52:54372) 2024-05-27T07:29:26.662 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:29:26.459+0000 7f418f59e700 1 -- 172.21.15.52:0/2987209299 learned_addr learned my addr 172.21.15.52:0/2987209299 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-05-27T07:29:26.662 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:29:26.459+0000 7f418e59c700 1 --2- 172.21.15.52:0/2987209299 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55acfb450f80 0x55acfb45cc10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:29:26.663 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:29:26.459+0000 7f418ed9d700 1 --2- 172.21.15.52:0/2987209299 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55acfb2d2e40 0x55acfb45c6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:29:26.663 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:29:26.460+0000 7f418f59e700 1 -- 172.21.15.52:0/2987209299 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55acfb2d2e40 msgr2=0x55acfb45c6d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:29:26.663 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:29:26.460+0000 7f418f59e700 1 --2- 172.21.15.52:0/2987209299 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55acfb2d2e40 0x55acfb45c6d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:29:26.663 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:29:26.460+0000 7f418f59e700 1 -- 172.21.15.52:0/2987209299 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55acfb450f80 msgr2=0x55acfb45cc10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:29:26.663 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:29:26.460+0000 7f418f59e700 1 --2- 172.21.15.52:0/2987209299 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55acfb450f80 0x55acfb45cc10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:29:26.663 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:29:26.460+0000 7f418f59e700 1 -- 172.21.15.52:0/2987209299 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55acfb455a80 con 0x55acfb456c50 2024-05-27T07:29:26.663 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:29:26.460+0000 7f418f59e700 1 --2- 172.21.15.52:0/2987209299 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55acfb456c50 0x55acfb464620 secure :-1 s=READY pgs=605 cs=0 l=1 rev1=1 crypto rx=0x7f4188003d50 tx=0x7f4188009920 comp rx=0 tx=0).ready entity=mon.0 client_cookie=10b1c9d04daa5a8c server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:29:26.663 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:29:26.460+0000 7f4174ff9700 1 -- 172.21.15.52:0/2987209299 <== mon.0 v2:172.21.15.52:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4188004db0 con 0x55acfb456c50 2024-05-27T07:29:26.663 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:29:26.460+0000 7f4174ff9700 1 -- 172.21.15.52:0/2987209299 <== mon.0 v2:172.21.15.52:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f41880054d0 con 0x55acfb456c50 2024-05-27T07:29:26.663 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:29:26.460+0000 7f4174ff9700 1 -- 172.21.15.52:0/2987209299 <== mon.0 v2:172.21.15.52:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4188016960 con 0x55acfb456c50 2024-05-27T07:29:26.663 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:29:26.461+0000 7f41a3a74b80 1 -- 172.21.15.52:0/2987209299 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55acfb456c50 msgr2=0x55acfb464620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:29:26.663 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:29:26.461+0000 7f41a3a74b80 1 --2- 172.21.15.52:0/2987209299 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55acfb456c50 0x55acfb464620 secure :-1 s=READY pgs=605 cs=0 l=1 rev1=1 crypto rx=0x7f4188003d50 tx=0x7f4188009920 comp rx=0 tx=0).stop 2024-05-27T07:29:26.663 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:29:26.461+0000 7f41a3a74b80 1 -- 172.21.15.52:0/2987209299 shutdown_connections 2024-05-27T07:29:26.664 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:29:26.461+0000 7f41a3a74b80 1 --2- 172.21.15.52:0/2987209299 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55acfb456c50 0x55acfb464620 unknown :-1 s=CLOSED pgs=605 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:29:26.664 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:29:26.461+0000 7f41a3a74b80 1 --2- 172.21.15.52:0/2987209299 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55acfb2d2e40 0x55acfb45c6d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:29:26.664 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:29:26.461+0000 7f41a3a74b80 1 --2- 172.21.15.52:0/2987209299 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55acfb450f80 0x55acfb45cc10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:29:26.664 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:29:26.461+0000 7f41a3a74b80 1 -- 172.21.15.52:0/2987209299 >> 172.21.15.52:0/2987209299 conn(0x55acfb32daf0 msgr2=0x55acfb45e620 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:29:26.664 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:29:26.461+0000 7f41a3a74b80 1 -- 172.21.15.52:0/2987209299 shutdown_connections 2024-05-27T07:29:26.664 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:29:26.461+0000 7f41a3a74b80 1 -- 172.21.15.52:0/2987209299 wait complete. 2024-05-27T07:29:26.664 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:29:26.462+0000 7f41a3a74b80 1 -- start start 2024-05-27T07:29:26.664 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:29:26.462+0000 7f41a3a74b80 1 Processor -- start 2024-05-27T07:29:26.664 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:29:26.462+0000 7f41a3a74b80 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55acfb2d2e40 0x55acfb468d40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:29:26.664 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:29:26.462+0000 7f41a3a74b80 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55acfb450f80 0x55acfb469280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:29:26.664 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:29:26.462+0000 7f41a3a74b80 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55acfb456c50 0x55acfb4d8290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:29:26.664 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:29:26.462+0000 7f41a3a74b80 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x7f41901ae9a0 con 0x55acfb456c50 2024-05-27T07:29:26.664 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:29:26.462+0000 7f41a3a74b80 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f41901ae210 con 0x55acfb450f80 2024-05-27T07:29:26.665 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:29:26.462+0000 7f41a3a74b80 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f41901af070 con 0x55acfb2d2e40 2024-05-27T07:29:26.665 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:29:26.462+0000 7f41a3a74b80 10 client.0 _ll_register_callbacks cb 0x55acfb30a920 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T07:29:26.665 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:29:26.462+0000 7f418ed9d700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55acfb2d2e40 0x55acfb468d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:29:26.665 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:29:26.462+0000 7f418e59c700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55acfb450f80 0x55acfb469280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:29:26.665 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:29:26.462+0000 7f418f59e700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55acfb456c50 0x55acfb4d8290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:29:26.665 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:29:26.462+0000 7f418ed9d700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55acfb2d2e40 0x55acfb468d40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.173:3300/0 says I am v2:172.21.15.52:36776/0 (socket says 172.21.15.52:36776) 2024-05-27T07:29:26.665 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:29:26.462+0000 7f418ed9d700 1 -- 172.21.15.52:0/3443638571 learned_addr learned my addr 172.21.15.52:0/3443638571 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-05-27T07:29:26.665 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:29:26.463+0000 7f418ed9d700 1 -- 172.21.15.52:0/3443638571 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55acfb450f80 msgr2=0x55acfb469280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:29:26.665 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:29:26.463+0000 7f418ed9d700 1 --2- 172.21.15.52:0/3443638571 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55acfb450f80 0x55acfb469280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:29:26.665 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:29:26.463+0000 7f418ed9d700 1 -- 172.21.15.52:0/3443638571 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55acfb456c50 msgr2=0x55acfb4d8290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:29:26.665 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:29:26.463+0000 7f418ed9d700 1 --2- 172.21.15.52:0/3443638571 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55acfb456c50 0x55acfb4d8290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:29:26.665 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:29:26.463+0000 7f418ed9d700 1 -- 172.21.15.52:0/3443638571 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55acfb469b10 con 0x55acfb2d2e40 2024-05-27T07:29:26.665 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:29:26.463+0000 7f418e59c700 1 --2- 172.21.15.52:0/3443638571 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55acfb450f80 0x55acfb469280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-27T07:29:26.666 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:29:26.463+0000 7f418ed9d700 1 --2- 172.21.15.52:0/3443638571 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55acfb2d2e40 0x55acfb468d40 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7f418003aa20 tx=0x7f418000fcb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=e3a406a53cd41655 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:29:26.666 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:29:26.463+0000 7f4172ff5700 10 client.0 ms_handle_connect on v2:172.21.15.173:3300/0 2024-05-27T07:29:26.666 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:29:26.463+0000 7f4172ff5700 1 -- 172.21.15.52:0/3443638571 <== mon.1 v2:172.21.15.173:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f418003acb0 con 0x55acfb2d2e40 2024-05-27T07:29:26.666 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:29:26.463+0000 7f41a3a74b80 1 -- 172.21.15.52:0/3443638571 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f41880054d0 con 0x55acfb2d2e40 2024-05-27T07:29:26.666 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:29:26.463+0000 7f4172ff5700 1 -- 172.21.15.52:0/3443638571 <== mon.1 v2:172.21.15.173:3300/0 2 ==== osd_map(103..103 src has 1..103) ==== 9379+0+0 (secure 0 0 0) 0x7f4180003960 con 0x55acfb2d2e40 2024-05-27T07:29:26.666 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:29:26.463+0000 7f4172ff5700 1 -- 172.21.15.52:0/3443638571 <== mon.1 v2:172.21.15.173:3300/0 3 ==== mon_get_version_reply(handle=1 version=58) ==== 24+0+0 (secure 0 0 0) 0x7f4180003b60 con 0x55acfb2d2e40 2024-05-27T07:29:26.666 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:29:26.463+0000 7f41a3a74b80 10 client.25846 fetch_fsmap learned FSMap version 58 2024-05-27T07:29:26.666 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:29:26.463+0000 7f41a3a74b80 1 -- 172.21.15.52:0/3443638571 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({config=0+,fsmap=58,monmap=4+}) -- 0x55acfb4d9800 con 0x55acfb2d2e40 2024-05-27T07:29:26.666 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:29:26.464+0000 7f4172ff5700 1 -- 172.21.15.52:0/3443638571 <== mon.1 v2:172.21.15.173:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f41880054d0 con 0x55acfb2d2e40 2024-05-27T07:29:26.666 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:29:26.464+0000 7f4172ff5700 1 -- 172.21.15.52:0/3443638571 <== mon.1 v2:172.21.15.173:3300/0 5 ==== fsmap(e 58) ==== 7773+0+0 (secure 0 0 0) 0x7f4180030c60 con 0x55acfb2d2e40 2024-05-27T07:29:26.666 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:29:26.464+0000 7f41a3a74b80 10 client.25846 fetch_fsmap finished waiting for FSMap version 58 2024-05-27T07:29:26.666 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:29:26.464+0000 7f41a3a74b80 10 client.25846 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-05-27T07:29:26.667 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:29:26.464+0000 7f41a3a74b80 20 client.25846 populate_metadata read hostname 'smithi052' 2024-05-27T07:29:26.667 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:29:26.464+0000 7f41a3a74b80 1 --2- 172.21.15.52:0/3443638571 >> [v2:172.21.15.105:6838/500348151,v1:172.21.15.105:6839/500348151] conn(0x55acfb4d9d40 0x55acfb4dc160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:29:26.667 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:29:26.464+0000 7f41a3a74b80 4 client.25846 mds_command: new command op to 24527 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-05-27T07:29:26.667 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:29:26.464+0000 7f41a3a74b80 1 -- 172.21.15.52:0/3443638571 --> [v2:172.21.15.105:6838/500348151,v1:172.21.15.105:6839/500348151] -- command(tid 0: {"prefix": "scrub status"}) -- 0x55acfb41c9f0 con 0x55acfb4d9d40 2024-05-27T07:29:26.667 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:29:26.464+0000 7f418e59c700 1 --2- 172.21.15.52:0/3443638571 >> [v2:172.21.15.105:6838/500348151,v1:172.21.15.105:6839/500348151] conn(0x55acfb4d9d40 0x55acfb4dc160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:29:26.667 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:29:26.464+0000 7f4172ff5700 10 client.25846 ms_handle_connect on v2:172.21.15.105:6838/500348151 2024-05-27T07:29:26.667 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:29:26.464+0000 7f418e59c700 1 --2- 172.21.15.52:0/3443638571 >> [v2:172.21.15.105:6838/500348151,v1:172.21.15.105:6839/500348151] conn(0x55acfb4d9d40 0x55acfb4dc160 crc :-1 s=READY pgs=94 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=5f449a4624c81433 server_cookie=65bb76cd55fceeda in_seq=0 out_seq=0 2024-05-27T07:29:26.667 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:29:26.466+0000 7f4172ff5700 1 -- 172.21.15.52:0/3443638571 <== mds.0 v2:172.21.15.105:6838/500348151 1 ==== command_reply(tid 0: 0 ) ==== 8+0+63 (crc 0 0 0) 0x55acfb41c9f0 con 0x55acfb4d9d40 2024-05-27T07:29:26.667 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:29:26.466+0000 7f4172ff5700 10 client.25846 handle_command_reply: tid=0 2024-05-27T07:29:26.667 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:29:26.466+0000 7f41a3a74b80 1 client.25846 shutdown 2024-05-27T07:29:26.667 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:29:26.466+0000 7f41a3a74b80 1 -- 172.21.15.52:0/3443638571 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55acfb2d2e40 msgr2=0x55acfb468d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:29:26.667 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:29:26.466+0000 7f41a3a74b80 1 --2- 172.21.15.52:0/3443638571 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55acfb2d2e40 0x55acfb468d40 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7f418003aa20 tx=0x7f418000fcb0 comp rx=0 tx=0).stop 2024-05-27T07:29:26.668 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:29:26.466+0000 7f41a3a74b80 1 -- 172.21.15.52:0/3443638571 shutdown_connections 2024-05-27T07:29:26.668 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:29:26.466+0000 7f41a3a74b80 1 --2- 172.21.15.52:0/3443638571 >> [v2:172.21.15.105:6838/500348151,v1:172.21.15.105:6839/500348151] conn(0x55acfb4d9d40 0x55acfb4dc160 crc :-1 s=READY pgs=94 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:29:26.668 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:29:26.466+0000 7f41a3a74b80 1 --2- 172.21.15.52:0/3443638571 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55acfb456c50 0x55acfb4d8290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:29:26.668 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:29:26.466+0000 7f41a3a74b80 1 --2- 172.21.15.52:0/3443638571 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55acfb2d2e40 0x55acfb468d40 unknown :-1 s=CLOSED pgs=114 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:29:26.668 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:29:26.466+0000 7f41a3a74b80 1 --2- 172.21.15.52:0/3443638571 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55acfb450f80 0x55acfb469280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:29:26.668 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:29:26.466+0000 7f4172ff5700 0 client.25846 ms_handle_reset on v2:172.21.15.105:6838/500348151 2024-05-27T07:29:26.668 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:29:26.466+0000 7f41a3a74b80 1 -- 172.21.15.52:0/3443638571 >> 172.21.15.52:0/3443638571 conn(0x55acfb32daf0 msgr2=0x55acfb4645b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:29:26.668 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:29:26.466+0000 7f41a3a74b80 1 -- 172.21.15.52:0/3443638571 shutdown_connections 2024-05-27T07:29:26.668 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:29:26.466+0000 7f41a3a74b80 1 -- 172.21.15.52:0/3443638571 wait complete. 2024-05-27T07:29:26.669 INFO:teuthology.orchestra.run.smithi052.stdout:{ 2024-05-27T07:29:26.669 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "no active scrubs running", 2024-05-27T07:29:26.669 INFO:teuthology.orchestra.run.smithi052.stdout: "scrubs": {} 2024-05-27T07:29:26.669 INFO:teuthology.orchestra.run.smithi052.stdout:} 2024-05-27T07:29:26.669 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:29:26.467+0000 7f41a3a74b80 20 client.25846 trim_cache size 0 max 16384 2024-05-27T07:29:26.669 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:29:26.467+0000 7f419505f700 1 -- 172.21.15.52:0/1672060335 >> [v2:172.21.15.52:6800/1822318384,v1:172.21.15.52:6801/1822318384] conn(0x7f41780b29f0 msgr2=0x7f41780b4f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:29:26.669 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:29:26.467+0000 7f419505f700 1 --2- 172.21.15.52:0/1672060335 >> [v2:172.21.15.52:6800/1822318384,v1:172.21.15.52:6801/1822318384] conn(0x7f41780b29f0 0x7f41780b4f00 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7f4180000c00 tx=0x7f418001a040 comp rx=0 tx=0).stop 2024-05-27T07:29:26.669 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:29:26.467+0000 7f419505f700 1 -- 172.21.15.52:0/1672060335 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f4190107de0 msgr2=0x7f419019b3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:29:26.670 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:29:26.467+0000 7f419505f700 1 --2- 172.21.15.52:0/1672060335 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f4190107de0 0x7f419019b3c0 secure :-1 s=READY pgs=603 cs=0 l=1 rev1=1 crypto rx=0x7f418400e970 tx=0x7f418400ec80 comp rx=0 tx=0).stop 2024-05-27T07:29:26.670 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:29:26.467+0000 7f419505f700 1 -- 172.21.15.52:0/1672060335 shutdown_connections 2024-05-27T07:29:26.670 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:29:26.467+0000 7f419505f700 1 --2- 172.21.15.52:0/1672060335 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f4190108730 0x7f419019f650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:29:26.670 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:29:26.467+0000 7f419505f700 1 --2- 172.21.15.52:0/1672060335 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f41901034b0 0x7f419019ae20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:29:26.670 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:29:26.467+0000 7f419505f700 1 --2- 172.21.15.52:0/1672060335 >> [v2:172.21.15.52:6800/1822318384,v1:172.21.15.52:6801/1822318384] conn(0x7f41780b29f0 0x7f41780b4f00 unknown :-1 s=CLOSED pgs=109 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:29:26.670 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:29:26.467+0000 7f419505f700 1 --2- 172.21.15.52:0/1672060335 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f4190107de0 0x7f419019b3c0 unknown :-1 s=CLOSED pgs=603 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:29:26.670 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:29:26.467+0000 7f419505f700 1 -- 172.21.15.52:0/1672060335 >> 172.21.15.52:0/1672060335 conn(0x7f41900fede0 msgr2=0x7f4190075d20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:29:26.670 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:29:26.468+0000 7f419505f700 1 -- 172.21.15.52:0/1672060335 shutdown_connections 2024-05-27T07:29:26.670 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:29:26.468+0000 7f419505f700 1 -- 172.21.15.52:0/1672060335 wait complete. 2024-05-27T07:29:26.690 INFO:journalctl@ceph.mds.k.smithi105.stdout:May 27 07:29:26 smithi105 podman[69734]: 2024-05-27 07:29:26.421374347 +0000 UTC m=+5.897703882 container remove 63d34243be402599f9f82855c72ddf43c4fc940256782a2b543ffdcdb8f8c5f6 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-k, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, GIT_CLEAN=True, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, org.label-schema.build-date=20240521, ceph=True) 2024-05-27T07:29:26.690 INFO:journalctl@ceph.mds.k.smithi105.stdout:May 27 07:29:26 smithi105 systemd[1]: ceph-73df2636-1bf8-11ef-bc9b-c7b262605968@mds.k.service: Main process exited, code=exited, status=134/n/a 2024-05-27T07:29:26.842 INFO:journalctl@ceph.mds.l.smithi173.stdout:May 27 07:29:26 smithi173 podman[73492]: 2024-05-27 07:29:26.840274528 +0000 UTC m=+0.649631453 container remove 686d1400cad7235ed70ce609ed1204dc003f1dfb6ad6c3e84cea045a4e9fff4d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-l, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240521, ceph=True, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, GIT_CLEAN=True, io.buildah.version=1.33.5) 2024-05-27T07:29:26.842 INFO:journalctl@ceph.mds.l.smithi173.stdout:May 27 07:29:26 smithi173 systemd[1]: ceph-73df2636-1bf8-11ef-bc9b-c7b262605968@mds.l.service: Main process exited, code=exited, status=134/n/a 2024-05-27T07:29:27.038 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:29:27.048+0000 7f3c199d5640 1 -- 172.21.15.52:0/701977351 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f3c141032e0 msgr2=0x7f3c1410f6b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:29:27.038 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:29:27.048+0000 7f3c199d5640 1 --2- 172.21.15.52:0/701977351 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f3c141032e0 0x7f3c1410f6b0 secure :-1 s=READY pgs=606 cs=0 l=1 rev1=1 crypto rx=0x7f3c0c009ca0 tx=0x7f3c0c02a8c0 comp rx=0 tx=0).stop 2024-05-27T07:29:27.039 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:29:27.048+0000 7f3c199d5640 1 -- 172.21.15.52:0/701977351 shutdown_connections 2024-05-27T07:29:27.039 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:29:27.048+0000 7f3c199d5640 1 --2- 172.21.15.52:0/701977351 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f3c141032e0 0x7f3c1410f6b0 unknown :-1 s=CLOSED pgs=606 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:29:27.039 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:29:27.048+0000 7f3c199d5640 1 --2- 172.21.15.52:0/701977351 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f3c14109030 0x7f3c14102c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:29:27.039 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:29:27.048+0000 7f3c199d5640 1 --2- 172.21.15.52:0/701977351 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f3c14108b90 0x7f3c14102730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:29:27.039 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:29:27.048+0000 7f3c199d5640 1 -- 172.21.15.52:0/701977351 >> 172.21.15.52:0/701977351 conn(0x7f3c140782d0 msgr2=0x7f3c14078710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:29:27.039 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:29:27.048+0000 7f3c199d5640 1 -- 172.21.15.52:0/701977351 shutdown_connections 2024-05-27T07:29:27.039 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:29:27.048+0000 7f3c199d5640 1 -- 172.21.15.52:0/701977351 wait complete. 2024-05-27T07:29:27.042 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:29:27.049+0000 7f3c199d5640 1 Processor -- start 2024-05-27T07:29:27.042 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:29:27.049+0000 7f3c199d5640 1 -- start start 2024-05-27T07:29:27.042 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:29:27.049+0000 7f3c199d5640 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f3c141032e0 0x7f3c140ffaf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:29:27.042 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:29:27.049+0000 7f3c199d5640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f3c14108b90 0x7f3c14100030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:29:27.042 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:29:27.049+0000 7f3c199d5640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f3c14109030 0x7f3c14100570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:29:27.042 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:29:27.049+0000 7f3c199d5640 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x7f3c14112df0 con 0x7f3c141032e0 2024-05-27T07:29:27.042 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:29:27.049+0000 7f3c199d5640 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f3c14112c70 con 0x7f3c14108b90 2024-05-27T07:29:27.042 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:29:27.049+0000 7f3c199d5640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f3c14112f70 con 0x7f3c14109030 2024-05-27T07:29:27.042 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:29:27.049+0000 7f3c12ffd640 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f3c141032e0 0x7f3c140ffaf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:29:27.042 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:29:27.049+0000 7f3c12ffd640 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f3c141032e0 0x7f3c140ffaf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.52:54394/0 (socket says 172.21.15.52:54394) 2024-05-27T07:29:27.042 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:29:27.049+0000 7f3c12ffd640 1 -- 172.21.15.52:0/1042421655 learned_addr learned my addr 172.21.15.52:0/1042421655 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-05-27T07:29:27.042 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:29:27.049+0000 7f3c127fc640 1 --2- 172.21.15.52:0/1042421655 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f3c14108b90 0x7f3c14100030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:29:27.043 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:29:27.049+0000 7f3c137fe640 1 --2- 172.21.15.52:0/1042421655 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f3c14109030 0x7f3c14100570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:29:27.043 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:29:27.050+0000 7f3c12ffd640 1 -- 172.21.15.52:0/1042421655 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f3c14109030 msgr2=0x7f3c14100570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:29:27.043 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:29:27.050+0000 7f3c12ffd640 1 --2- 172.21.15.52:0/1042421655 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f3c14109030 0x7f3c14100570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:29:27.043 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:29:27.050+0000 7f3c12ffd640 1 -- 172.21.15.52:0/1042421655 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f3c14108b90 msgr2=0x7f3c14100030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:29:27.043 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:29:27.050+0000 7f3c12ffd640 1 --2- 172.21.15.52:0/1042421655 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f3c14108b90 0x7f3c14100030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:29:27.043 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:29:27.050+0000 7f3c12ffd640 1 -- 172.21.15.52:0/1042421655 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3c141a8be0 con 0x7f3c141032e0 2024-05-27T07:29:27.043 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:29:27.050+0000 7f3c127fc640 1 --2- 172.21.15.52:0/1042421655 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f3c14108b90 0x7f3c14100030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-27T07:29:27.043 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:29:27.050+0000 7f3c12ffd640 1 --2- 172.21.15.52:0/1042421655 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f3c141032e0 0x7f3c140ffaf0 secure :-1 s=READY pgs=607 cs=0 l=1 rev1=1 crypto rx=0x7f3c0400e9f0 tx=0x7f3c0400eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:29:27.043 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:29:27.050+0000 7f3c189d3640 1 -- 172.21.15.52:0/1042421655 <== mon.0 v2:172.21.15.52:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3c0400ccc0 con 0x7f3c141032e0 2024-05-27T07:29:27.043 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:29:27.050+0000 7f3c189d3640 1 -- 172.21.15.52:0/1042421655 <== mon.0 v2:172.21.15.52:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f3c040054d0 con 0x7f3c141032e0 2024-05-27T07:29:27.044 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:29:27.050+0000 7f3c199d5640 1 -- 172.21.15.52:0/1042421655 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3c141a8ed0 con 0x7f3c141032e0 2024-05-27T07:29:27.044 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:29:27.050+0000 7f3c189d3640 1 -- 172.21.15.52:0/1042421655 <== mon.0 v2:172.21.15.52:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3c04010430 con 0x7f3c141032e0 2024-05-27T07:29:27.044 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:29:27.050+0000 7f3c199d5640 1 -- 172.21.15.52:0/1042421655 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3c141a9330 con 0x7f3c141032e0 2024-05-27T07:29:27.049 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:29:27.053+0000 7f3c20967740 1 Processor -- start 2024-05-27T07:29:27.049 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:29:27.053+0000 7f3c20967740 1 -- start start 2024-05-27T07:29:27.049 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:29:27.053+0000 7f3c20967740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5557fdcde1b0 0x5557fe07f3d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:29:27.049 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:29:27.053+0000 7f3c20967740 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x5557fdcb52f0 0x5557fe07f910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:29:27.049 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:29:27.053+0000 7f3c189d3640 1 -- 172.21.15.52:0/1042421655 <== mon.0 v2:172.21.15.52:3300/0 4 ==== mgrmap(e 26) ==== 149783+0+0 (secure 0 0 0) 0x7f3c040105d0 con 0x7f3c141032e0 2024-05-27T07:29:27.050 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:29:27.053+0000 7f3c20967740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5557fdcbd4d0 0x5557fe07fe50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:29:27.050 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:29:27.053+0000 7f3c20967740 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x5557fdff1c30 con 0x5557fdcb52f0 2024-05-27T07:29:27.050 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:29:27.053+0000 7f3c20967740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x5557fdfe17d0 con 0x5557fdcbd4d0 2024-05-27T07:29:27.050 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:29:27.053+0000 7f3c20967740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x5557fdfe5810 con 0x5557fdcde1b0 2024-05-27T07:29:27.050 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:29:27.054+0000 7f3c12ffd640 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x5557fdcb52f0 0x5557fe07f910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:29:27.050 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:29:27.054+0000 7f3c137fe640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5557fdcde1b0 0x5557fe07f3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:29:27.050 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:29:27.054+0000 7f3c127fc640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5557fdcbd4d0 0x5557fe07fe50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:29:27.050 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:29:27.054+0000 7f3c137fe640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5557fdcde1b0 0x5557fe07f3d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.173:3300/0 says I am v2:172.21.15.52:36810/0 (socket says 172.21.15.52:36810) 2024-05-27T07:29:27.050 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:29:27.054+0000 7f3c137fe640 1 -- 172.21.15.52:0/2826657681 learned_addr learned my addr 172.21.15.52:0/2826657681 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-05-27T07:29:27.050 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:29:27.054+0000 7f3c127fc640 1 -- 172.21.15.52:0/2826657681 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5557fdcde1b0 msgr2=0x5557fe07f3d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:29:27.050 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:29:27.054+0000 7f3c127fc640 1 --2- 172.21.15.52:0/2826657681 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5557fdcde1b0 0x5557fe07f3d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:29:27.050 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:29:27.054+0000 7f3c127fc640 1 -- 172.21.15.52:0/2826657681 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x5557fdcb52f0 msgr2=0x5557fe07f910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:29:27.051 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:29:27.054+0000 7f3c127fc640 1 --2- 172.21.15.52:0/2826657681 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x5557fdcb52f0 0x5557fe07f910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:29:27.051 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:29:27.054+0000 7f3c127fc640 1 -- 172.21.15.52:0/2826657681 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5557fdcf6210 con 0x5557fdcbd4d0 2024-05-27T07:29:27.051 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:29:27.054+0000 7f3c127fc640 1 --2- 172.21.15.52:0/2826657681 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5557fdcbd4d0 0x5557fe07fe50 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7f3c14109460 tx=0x7f3c08011810 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b9c1a2fcc8635fdd server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:29:27.051 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:29:27.054+0000 7f3bf9ffb640 1 -- 172.21.15.52:0/2826657681 <== mon.2 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3c08004310 con 0x5557fdcbd4d0 2024-05-27T07:29:27.051 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:29:27.054+0000 7f3c189d3640 1 --2- 172.21.15.52:0/1042421655 >> [v2:172.21.15.52:6800/1822318384,v1:172.21.15.52:6801/1822318384] conn(0x7f3bfc0b2950 0x7f3bfc0b4e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:29:27.051 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:29:27.055+0000 7f3bf9ffb640 1 -- 172.21.15.52:0/2826657681 <== mon.2 v2:172.21.15.105:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f3c080044b0 con 0x5557fdcbd4d0 2024-05-27T07:29:27.051 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:29:27.055+0000 7f3bf9ffb640 1 -- 172.21.15.52:0/2826657681 <== mon.2 v2:172.21.15.105:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3c080035f0 con 0x5557fdcbd4d0 2024-05-27T07:29:27.051 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:29:27.055+0000 7f3c137fe640 1 --2- 172.21.15.52:0/1042421655 >> [v2:172.21.15.52:6800/1822318384,v1:172.21.15.52:6801/1822318384] conn(0x7f3bfc0b2950 0x7f3bfc0b4e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:29:27.051 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:29:27.055+0000 7f3c189d3640 1 -- 172.21.15.52:0/1042421655 <== mon.0 v2:172.21.15.52:3300/0 5 ==== osd_map(103..103 src has 1..103) ==== 9379+0+0 (secure 0 0 0) 0x7f3c040e2d10 con 0x7f3c141032e0 2024-05-27T07:29:27.051 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:29:27.055+0000 7f3c20967740 1 -- 172.21.15.52:0/2826657681 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5557fdcbd4d0 msgr2=0x5557fe07fe50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:29:27.051 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:29:27.055+0000 7f3c20967740 1 --2- 172.21.15.52:0/2826657681 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5557fdcbd4d0 0x5557fe07fe50 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7f3c14109460 tx=0x7f3c08011810 comp rx=0 tx=0).stop 2024-05-27T07:29:27.051 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:29:27.055+0000 7f3c137fe640 1 --2- 172.21.15.52:0/1042421655 >> [v2:172.21.15.52:6800/1822318384,v1:172.21.15.52:6801/1822318384] conn(0x7f3bfc0b2950 0x7f3bfc0b4e60 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x5557fe07c540 tx=0x7f3c0c005d80 comp rx=0 tx=0).ready entity=mgr.15333 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:29:27.052 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:29:27.055+0000 7f3c20967740 1 -- 172.21.15.52:0/2826657681 shutdown_connections 2024-05-27T07:29:27.052 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:29:27.055+0000 7f3c20967740 1 --2- 172.21.15.52:0/2826657681 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5557fdcbd4d0 0x5557fe07fe50 unknown :-1 s=CLOSED pgs=122 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:29:27.052 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:29:27.055+0000 7f3c20967740 1 --2- 172.21.15.52:0/2826657681 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x5557fdcb52f0 0x5557fe07f910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:29:27.052 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:29:27.055+0000 7f3c20967740 1 --2- 172.21.15.52:0/2826657681 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5557fdcde1b0 0x5557fe07f3d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:29:27.052 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:29:27.055+0000 7f3c20967740 1 -- 172.21.15.52:0/2826657681 >> 172.21.15.52:0/2826657681 conn(0x5557fdcd2300 msgr2=0x5557fe07bb80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:29:27.052 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:29:27.056+0000 7f3c20967740 1 -- 172.21.15.52:0/2826657681 shutdown_connections 2024-05-27T07:29:27.052 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:29:27.056+0000 7f3c20967740 1 -- 172.21.15.52:0/2826657681 wait complete. 2024-05-27T07:29:27.052 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:29:27.056+0000 7f3c20967740 1 -- start start 2024-05-27T07:29:27.052 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:29:27.056+0000 7f3c20967740 1 Processor -- start 2024-05-27T07:29:27.052 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:29:27.057+0000 7f3c20967740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5557fdcb52f0 0x5557fe089a60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:29:27.052 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:29:27.057+0000 7f3c20967740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5557fdcbd4d0 0x5557fe089fa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:29:27.053 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:29:27.057+0000 7f3c20967740 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x5557fdcde1b0 0x5557fe08f800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:29:27.053 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:29:27.057+0000 7f3c20967740 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x5557fdcc7790 con 0x5557fdcde1b0 2024-05-27T07:29:27.053 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:29:27.057+0000 7f3c20967740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x5557fdf9d890 con 0x5557fdcbd4d0 2024-05-27T07:29:27.053 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:29:27.057+0000 7f3c20967740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x5557fe08c500 con 0x5557fdcb52f0 2024-05-27T07:29:27.053 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:29:27.057+0000 7f3c20967740 10 client.0 _ll_register_callbacks cb 0x5557fdcf5a00 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T07:29:27.053 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:29:27.057+0000 7f3c137fe640 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x5557fdcde1b0 0x5557fe08f800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:29:27.053 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:29:27.057+0000 7f3c12ffd640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5557fdcb52f0 0x5557fe089a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:29:27.053 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:29:27.057+0000 7f3c137fe640 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x5557fdcde1b0 0x5557fe08f800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.52:54404/0 (socket says 172.21.15.52:54404) 2024-05-27T07:29:27.053 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:29:27.057+0000 7f3c12ffd640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5557fdcb52f0 0x5557fe089a60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.173:3300/0 says I am v2:172.21.15.52:36826/0 (socket says 172.21.15.52:36826) 2024-05-27T07:29:27.053 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:29:27.057+0000 7f3c137fe640 1 -- 172.21.15.52:0/3120273603 learned_addr learned my addr 172.21.15.52:0/3120273603 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-05-27T07:29:27.053 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:29:27.057+0000 7f3c127fc640 1 --2- 172.21.15.52:0/3120273603 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5557fdcbd4d0 0x5557fe089fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:29:27.053 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:29:27.057+0000 7f3c137fe640 1 -- 172.21.15.52:0/3120273603 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5557fdcb52f0 msgr2=0x5557fe089a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:29:27.054 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:29:27.057+0000 7f3c137fe640 1 --2- 172.21.15.52:0/3120273603 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5557fdcb52f0 0x5557fe089a60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:29:27.054 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:29:27.057+0000 7f3c137fe640 1 -- 172.21.15.52:0/3120273603 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5557fdcbd4d0 msgr2=0x5557fe089fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:29:27.054 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:29:27.057+0000 7f3c137fe640 1 --2- 172.21.15.52:0/3120273603 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5557fdcbd4d0 0x5557fe089fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:29:27.054 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:29:27.057+0000 7f3c137fe640 1 -- 172.21.15.52:0/3120273603 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5557fddc2620 con 0x5557fdcde1b0 2024-05-27T07:29:27.054 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:29:27.057+0000 7f3c12ffd640 1 --2- 172.21.15.52:0/3120273603 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5557fdcb52f0 0x5557fe089a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-27T07:29:27.054 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:29:27.057+0000 7f3c137fe640 1 --2- 172.21.15.52:0/3120273603 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x5557fdcde1b0 0x5557fe08f800 secure :-1 s=READY pgs=608 cs=0 l=1 rev1=1 crypto rx=0x7f3c0c02f540 tx=0x7f3c0c02a8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d0e15473af5ceaa2 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:29:27.054 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:29:27.057+0000 7f3bf2ffd640 10 client.0 ms_handle_connect on v2:172.21.15.52:3300/0 2024-05-27T07:29:27.054 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:29:27.057+0000 7f3bf2ffd640 1 -- 172.21.15.52:0/3120273603 <== mon.0 v2:172.21.15.52:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3c0c02f610 con 0x5557fdcde1b0 2024-05-27T07:29:27.054 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:29:27.057+0000 7f3bf2ffd640 1 -- 172.21.15.52:0/3120273603 <== mon.0 v2:172.21.15.52:3300/0 2 ==== osd_map(103..103 src has 1..103) ==== 9379+0+0 (secure 0 0 0) 0x7f3c0c02f7b0 con 0x5557fdcde1b0 2024-05-27T07:29:27.054 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:29:27.058+0000 7f3c20967740 1 -- 172.21.15.52:0/3120273603 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f3c080044b0 con 0x5557fdcde1b0 2024-05-27T07:29:27.054 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:29:27.058+0000 7f3bf2ffd640 1 -- 172.21.15.52:0/3120273603 <== mon.0 v2:172.21.15.52:3300/0 3 ==== mon_get_version_reply(handle=1 version=58) ==== 24+0+0 (secure 0 0 0) 0x7f3c0c02f9b0 con 0x5557fdcde1b0 2024-05-27T07:29:27.055 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:29:27.058+0000 7f3c20967740 10 client.16269 fetch_fsmap learned FSMap version 58 2024-05-27T07:29:27.055 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:29:27.058+0000 7f3c20967740 1 -- 172.21.15.52:0/3120273603 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({config=0+,fsmap=58,monmap=4+}) -- 0x5557fe098760 con 0x5557fdcde1b0 2024-05-27T07:29:27.055 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:29:27.058+0000 7f3bf2ffd640 1 -- 172.21.15.52:0/3120273603 <== mon.0 v2:172.21.15.52:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f3c080044b0 con 0x5557fdcde1b0 2024-05-27T07:29:27.055 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:29:27.058+0000 7f3bf2ffd640 1 -- 172.21.15.52:0/3120273603 <== mon.0 v2:172.21.15.52:3300/0 5 ==== fsmap(e 58) ==== 7773+0+0 (secure 0 0 0) 0x7f3c0c051070 con 0x5557fdcde1b0 2024-05-27T07:29:27.055 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:29:27.058+0000 7f3c20967740 10 client.16269 fetch_fsmap finished waiting for FSMap version 58 2024-05-27T07:29:27.055 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:29:27.058+0000 7f3c20967740 10 client.16269 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-05-27T07:29:27.055 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:29:27.058+0000 7f3c20967740 20 client.16269 populate_metadata read hostname 'smithi052' 2024-05-27T07:29:27.055 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:29:27.058+0000 7f3c20967740 1 --2- 172.21.15.52:0/3120273603 >> [v2:172.21.15.105:6838/500348151,v1:172.21.15.105:6839/500348151] conn(0x5557fdc9cd80 0x5557fe09be50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:29:27.055 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:29:27.058+0000 7f3c20967740 4 client.16269 mds_command: new command op to 24527 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-27T07:29:27.055 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:29:27.058+0000 7f3c20967740 1 -- 172.21.15.52:0/3120273603 --> [v2:172.21.15.105:6838/500348151,v1:172.21.15.105:6839/500348151] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5557fdff3b80 con 0x5557fdc9cd80 2024-05-27T07:29:27.055 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:29:27.059+0000 7f3c12ffd640 1 --2- 172.21.15.52:0/3120273603 >> [v2:172.21.15.105:6838/500348151,v1:172.21.15.105:6839/500348151] conn(0x5557fdc9cd80 0x5557fe09be50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:29:27.055 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:29:27.059+0000 7f3bf2ffd640 10 client.16269 ms_handle_connect on v2:172.21.15.105:6838/500348151 2024-05-27T07:29:27.055 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:29:27.059+0000 7f3c12ffd640 1 --2- 172.21.15.52:0/3120273603 >> [v2:172.21.15.105:6838/500348151,v1:172.21.15.105:6839/500348151] conn(0x5557fdc9cd80 0x5557fe09be50 crc :-1 s=READY pgs=95 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=637b6dcbeade8d9c server_cookie=7aee34606daa4bbc in_seq=0 out_seq=0 2024-05-27T07:29:27.057 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:29:27.061+0000 7f3bf2ffd640 1 -- 172.21.15.52:0/3120273603 <== mds.0 v2:172.21.15.105:6838/500348151 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24041 (crc 0 0 0) 0x5557fdff3b80 con 0x5557fdc9cd80 2024-05-27T07:29:27.057 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:29:27.061+0000 7f3bf2ffd640 10 client.16269 handle_command_reply: tid=0 2024-05-27T07:29:27.057 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:29:27.061+0000 7f3c20967740 1 client.16269 shutdown 2024-05-27T07:29:27.057 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:29:27.061+0000 7f3c20967740 1 -- 172.21.15.52:0/3120273603 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x5557fdcde1b0 msgr2=0x5557fe08f800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:29:27.057 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:29:27.061+0000 7f3c20967740 1 --2- 172.21.15.52:0/3120273603 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x5557fdcde1b0 0x5557fe08f800 secure :-1 s=READY pgs=608 cs=0 l=1 rev1=1 crypto rx=0x7f3c0c02f540 tx=0x7f3c0c02a8c0 comp rx=0 tx=0).stop 2024-05-27T07:29:27.057 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:29:27.061+0000 7f3c20967740 1 -- 172.21.15.52:0/3120273603 shutdown_connections 2024-05-27T07:29:27.057 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:29:27.061+0000 7f3c20967740 1 --2- 172.21.15.52:0/3120273603 >> [v2:172.21.15.105:6838/500348151,v1:172.21.15.105:6839/500348151] conn(0x5557fdc9cd80 0x5557fe09be50 crc :-1 s=READY pgs=95 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:29:27.057 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:29:27.061+0000 7f3c20967740 1 --2- 172.21.15.52:0/3120273603 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x5557fdcde1b0 0x5557fe08f800 unknown :-1 s=CLOSED pgs=608 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:29:27.057 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:29:27.061+0000 7f3c20967740 1 --2- 172.21.15.52:0/3120273603 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5557fdcbd4d0 0x5557fe089fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:29:27.057 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:29:27.061+0000 7f3c20967740 1 --2- 172.21.15.52:0/3120273603 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5557fdcb52f0 0x5557fe089a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:29:27.057 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:29:27.061+0000 7f3bf2ffd640 0 client.16269 ms_handle_reset on v2:172.21.15.105:6838/500348151 2024-05-27T07:29:27.058 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:29:27.061+0000 7f3c20967740 1 -- 172.21.15.52:0/3120273603 >> 172.21.15.52:0/3120273603 conn(0x5557fdcd2300 msgr2=0x5557fe0810f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:29:27.058 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:29:27.061+0000 7f3c20967740 1 -- 172.21.15.52:0/3120273603 shutdown_connections 2024-05-27T07:29:27.058 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:29:27.061+0000 7f3c20967740 1 -- 172.21.15.52:0/3120273603 wait complete. 2024-05-27T07:29:27.058 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:29:27.061+0000 7f3c20967740 20 client.16269 trim_cache size 0 max 16384 2024-05-27T07:29:27.069 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:29:27.078+0000 7f3c20967740 1 Processor -- start 2024-05-27T07:29:27.069 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:29:27.078+0000 7f3c20967740 1 -- start start 2024-05-27T07:29:27.069 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:29:27.078+0000 7f3c20967740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5557fdcb52f0 0x5557fe089130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:29:27.069 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:29:27.078+0000 7f3c20967740 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x5557fdcbd4d0 0x5557fe07fb90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:29:27.070 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:29:27.078+0000 7f3c20967740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5557fdcde1b0 0x5557fe08f770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:29:27.070 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:29:27.078+0000 7f3c20967740 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x7f3c141b4f20 con 0x5557fdcbd4d0 2024-05-27T07:29:27.070 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:29:27.078+0000 7f3c20967740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f3c141b2c50 con 0x5557fdcb52f0 2024-05-27T07:29:27.070 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:29:27.078+0000 7f3c20967740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f3c141b4850 con 0x5557fdcde1b0 2024-05-27T07:29:27.070 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:29:27.078+0000 7f3c127fc640 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x5557fdcbd4d0 0x5557fe07fb90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:29:27.070 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:29:27.078+0000 7f3c12ffd640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5557fdcb52f0 0x5557fe089130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:29:27.070 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:29:27.078+0000 7f3c127fc640 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x5557fdcbd4d0 0x5557fe07fb90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.52:54408/0 (socket says 172.21.15.52:54408) 2024-05-27T07:29:27.070 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:29:27.078+0000 7f3c127fc640 1 -- 172.21.15.52:0/3383186500 learned_addr learned my addr 172.21.15.52:0/3383186500 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-05-27T07:29:27.070 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:29:27.078+0000 7f3c137fe640 1 --2- 172.21.15.52:0/3383186500 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5557fdcde1b0 0x5557fe08f770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:29:27.070 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:29:27.078+0000 7f3c127fc640 1 -- 172.21.15.52:0/3383186500 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5557fdcde1b0 msgr2=0x5557fe08f770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:29:27.071 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:29:27.078+0000 7f3c127fc640 1 --2- 172.21.15.52:0/3383186500 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5557fdcde1b0 0x5557fe08f770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:29:27.071 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:29:27.078+0000 7f3c127fc640 1 -- 172.21.15.52:0/3383186500 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5557fdcb52f0 msgr2=0x5557fe089130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:29:27.071 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:29:27.078+0000 7f3c127fc640 1 --2- 172.21.15.52:0/3383186500 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5557fdcb52f0 0x5557fe089130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:29:27.071 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:29:27.078+0000 7f3c127fc640 1 -- 172.21.15.52:0/3383186500 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5557fe087da0 con 0x5557fdcbd4d0 2024-05-27T07:29:27.071 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:29:27.078+0000 7f3c12ffd640 1 --2- 172.21.15.52:0/3383186500 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5557fdcb52f0 0x5557fe089130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-27T07:29:27.071 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:29:27.079+0000 7f3c127fc640 1 --2- 172.21.15.52:0/3383186500 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x5557fdcbd4d0 0x5557fe07fb90 secure :-1 s=READY pgs=609 cs=0 l=1 rev1=1 crypto rx=0x7f3c08004a90 tx=0x7f3c08004ac0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=38bcdbf90b28be79 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:29:27.071 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:29:27.079+0000 7f3bf97fa640 1 -- 172.21.15.52:0/3383186500 <== mon.0 v2:172.21.15.52:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3c08029020 con 0x5557fdcbd4d0 2024-05-27T07:29:27.071 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:29:27.079+0000 7f3bf97fa640 1 -- 172.21.15.52:0/3383186500 <== mon.0 v2:172.21.15.52:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f3c0801f040 con 0x5557fdcbd4d0 2024-05-27T07:29:27.071 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:29:27.079+0000 7f3bf97fa640 1 -- 172.21.15.52:0/3383186500 <== mon.0 v2:172.21.15.52:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3c0801e040 con 0x5557fdcbd4d0 2024-05-27T07:29:27.075 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:29:27.080+0000 7f3c20967740 1 -- 172.21.15.52:0/3383186500 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x5557fdcbd4d0 msgr2=0x5557fe07fb90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:29:27.075 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:29:27.080+0000 7f3c20967740 1 --2- 172.21.15.52:0/3383186500 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x5557fdcbd4d0 0x5557fe07fb90 secure :-1 s=READY pgs=609 cs=0 l=1 rev1=1 crypto rx=0x7f3c08004a90 tx=0x7f3c08004ac0 comp rx=0 tx=0).stop 2024-05-27T07:29:27.075 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:29:27.080+0000 7f3c20967740 1 -- 172.21.15.52:0/3383186500 shutdown_connections 2024-05-27T07:29:27.075 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:29:27.080+0000 7f3c20967740 1 --2- 172.21.15.52:0/3383186500 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5557fdcde1b0 0x5557fe08f770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:29:27.075 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:29:27.080+0000 7f3c20967740 1 --2- 172.21.15.52:0/3383186500 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x5557fdcbd4d0 0x5557fe07fb90 unknown :-1 s=CLOSED pgs=609 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:29:27.075 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:29:27.080+0000 7f3c20967740 1 --2- 172.21.15.52:0/3383186500 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5557fdcb52f0 0x5557fe089130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:29:27.075 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:29:27.080+0000 7f3c20967740 1 -- 172.21.15.52:0/3383186500 >> 172.21.15.52:0/3383186500 conn(0x5557fdcd2300 msgr2=0x5557fe08ba40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:29:27.075 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:29:27.080+0000 7f3c20967740 1 -- 172.21.15.52:0/3383186500 shutdown_connections 2024-05-27T07:29:27.076 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:29:27.080+0000 7f3c20967740 1 -- 172.21.15.52:0/3383186500 wait complete. 2024-05-27T07:29:27.076 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:29:27.080+0000 7f3c20967740 1 -- start start 2024-05-27T07:29:27.076 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:29:27.081+0000 7f3c20967740 1 Processor -- start 2024-05-27T07:29:27.076 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:29:27.081+0000 7f3c20967740 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x5557fdcb52f0 0x5557fe096040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:29:27.076 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:29:27.081+0000 7f3c20967740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5557fdcbd4d0 0x5557fe091390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:29:27.076 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:29:27.081+0000 7f3c20967740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5557fdcde1b0 0x5557fe0918d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:29:27.076 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:29:27.081+0000 7f3c20967740 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x7f3c141b3ab0 con 0x5557fdcb52f0 2024-05-27T07:29:27.076 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:29:27.081+0000 7f3c20967740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f3c141b3320 con 0x5557fdcbd4d0 2024-05-27T07:29:27.076 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:29:27.081+0000 7f3c20967740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f3c141b4180 con 0x5557fdcde1b0 2024-05-27T07:29:27.076 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:29:27.081+0000 7f3c20967740 10 client.0 _ll_register_callbacks cb 0x7f3c0c009b20 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T07:29:27.076 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:29:27.081+0000 7f3c12ffd640 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x5557fdcb52f0 0x5557fe096040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:29:27.077 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:29:27.081+0000 7f3c127fc640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5557fdcbd4d0 0x5557fe091390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:29:27.077 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:29:27.081+0000 7f3c12ffd640 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x5557fdcb52f0 0x5557fe096040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.52:54410/0 (socket says 172.21.15.52:54410) 2024-05-27T07:29:27.077 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:29:27.081+0000 7f3c12ffd640 1 -- 172.21.15.52:0/95148456 learned_addr learned my addr 172.21.15.52:0/95148456 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-05-27T07:29:27.077 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:29:27.081+0000 7f3c137fe640 1 --2- 172.21.15.52:0/95148456 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5557fdcde1b0 0x5557fe0918d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:29:27.077 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:29:27.082+0000 7f3c127fc640 1 -- 172.21.15.52:0/95148456 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5557fdcde1b0 msgr2=0x5557fe0918d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:29:27.077 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:29:27.082+0000 7f3c127fc640 1 --2- 172.21.15.52:0/95148456 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5557fdcde1b0 0x5557fe0918d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:29:27.077 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:29:27.082+0000 7f3c127fc640 1 -- 172.21.15.52:0/95148456 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x5557fdcb52f0 msgr2=0x5557fe096040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:29:27.077 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:29:27.082+0000 7f3c127fc640 1 --2- 172.21.15.52:0/95148456 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x5557fdcb52f0 0x5557fe096040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:29:27.077 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:29:27.082+0000 7f3c127fc640 1 -- 172.21.15.52:0/95148456 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5557fe091e10 con 0x5557fdcbd4d0 2024-05-27T07:29:27.077 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:29:27.082+0000 7f3c12ffd640 1 --2- 172.21.15.52:0/95148456 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x5557fdcb52f0 0x5557fe096040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-27T07:29:27.077 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:29:27.082+0000 7f3c137fe640 1 --2- 172.21.15.52:0/95148456 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5557fdcde1b0 0x5557fe0918d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-27T07:29:27.077 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:29:27.082+0000 7f3c127fc640 1 --2- 172.21.15.52:0/95148456 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5557fdcbd4d0 0x5557fe091390 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7f3c08004a90 tx=0x7f3c08004850 comp rx=0 tx=0).ready entity=mon.2 client_cookie=184476270c4076d1 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:29:27.077 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:29:27.082+0000 7f3bf2ffd640 10 client.0 ms_handle_connect on v2:172.21.15.105:3300/0 2024-05-27T07:29:27.078 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:29:27.082+0000 7f3bf2ffd640 1 -- 172.21.15.52:0/95148456 <== mon.2 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3c08032070 con 0x5557fdcbd4d0 2024-05-27T07:29:27.078 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:29:27.082+0000 7f3c20967740 1 -- 172.21.15.52:0/95148456 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f3c0801f040 con 0x5557fdcbd4d0 2024-05-27T07:29:27.078 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:29:27.083+0000 7f3bf2ffd640 1 -- 172.21.15.52:0/95148456 <== mon.2 v2:172.21.15.105:3300/0 2 ==== osd_map(103..103 src has 1..103) ==== 9379+0+0 (secure 0 0 0) 0x7f3c08003a00 con 0x5557fdcbd4d0 2024-05-27T07:29:27.078 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:29:27.083+0000 7f3bf2ffd640 1 -- 172.21.15.52:0/95148456 <== mon.2 v2:172.21.15.105:3300/0 3 ==== mon_get_version_reply(handle=1 version=58) ==== 24+0+0 (secure 0 0 0) 0x7f3c08040050 con 0x5557fdcbd4d0 2024-05-27T07:29:27.078 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:29:27.083+0000 7f3c20967740 10 client.25943 fetch_fsmap learned FSMap version 58 2024-05-27T07:29:27.078 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:29:27.083+0000 7f3c20967740 1 -- 172.21.15.52:0/95148456 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,fsmap=58,monmap=4+}) -- 0x5557fe104090 con 0x5557fdcbd4d0 2024-05-27T07:29:27.078 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:29:27.083+0000 7f3bf2ffd640 1 -- 172.21.15.52:0/95148456 <== mon.2 v2:172.21.15.105:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f3c0801f040 con 0x5557fdcbd4d0 2024-05-27T07:29:27.078 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:29:27.083+0000 7f3bf2ffd640 1 -- 172.21.15.52:0/95148456 <== mon.2 v2:172.21.15.105:3300/0 5 ==== fsmap(e 58) ==== 7773+0+0 (secure 0 0 0) 0x7f3c08024040 con 0x5557fdcbd4d0 2024-05-27T07:29:27.078 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:29:27.083+0000 7f3c20967740 10 client.25943 fetch_fsmap finished waiting for FSMap version 58 2024-05-27T07:29:27.078 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:29:27.083+0000 7f3c20967740 10 client.25943 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-05-27T07:29:27.078 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:29:27.083+0000 7f3c20967740 20 client.25943 populate_metadata read hostname 'smithi052' 2024-05-27T07:29:27.078 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:29:27.083+0000 7f3c20967740 1 --2- 172.21.15.52:0/95148456 >> [v2:172.21.15.105:6838/500348151,v1:172.21.15.105:6839/500348151] conn(0x5557fdc721c0 0x5557fe106620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:29:27.078 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:29:27.083+0000 7f3c20967740 4 client.25943 mds_command: new command op to 24527 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-05-27T07:29:27.079 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:29:27.083+0000 7f3c20967740 1 -- 172.21.15.52:0/95148456 --> [v2:172.21.15.105:6838/500348151,v1:172.21.15.105:6839/500348151] -- command(tid 0: {"prefix": "flush journal"}) -- 0x5557fdfed360 con 0x5557fdc721c0 2024-05-27T07:29:27.079 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:29:27.084+0000 7f3c12ffd640 1 --2- 172.21.15.52:0/95148456 >> [v2:172.21.15.105:6838/500348151,v1:172.21.15.105:6839/500348151] conn(0x5557fdc721c0 0x5557fe106620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:29:27.079 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:29:27.084+0000 7f3bf2ffd640 10 client.25943 ms_handle_connect on v2:172.21.15.105:6838/500348151 2024-05-27T07:29:27.079 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:29:27.084+0000 7f3c12ffd640 1 --2- 172.21.15.52:0/95148456 >> [v2:172.21.15.105:6838/500348151,v1:172.21.15.105:6839/500348151] conn(0x5557fdc721c0 0x5557fe106620 crc :-1 s=READY pgs=96 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=5e565225f0fe9859 server_cookie=8daaa87f4792e0c in_seq=0 out_seq=0 2024-05-27T07:29:27.079 INFO:journalctl@ceph.mds.k.smithi105.stdout:May 27 07:29:26 smithi105 systemd[1]: ceph-73df2636-1bf8-11ef-bc9b-c7b262605968@mds.k.service: Failed with result 'exit-code'. 2024-05-27T07:29:27.352 INFO:journalctl@ceph.mds.l.smithi173.stdout:May 27 07:29:27 smithi173 systemd[1]: ceph-73df2636-1bf8-11ef-bc9b-c7b262605968@mds.l.service: Failed with result 'exit-code'. 2024-05-27T07:29:27.352 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:29:27 smithi173 ceph-mon[37000]: mds.? [v2:172.21.15.173:6832/453411960,v1:172.21.15.173:6833/453411960] up:boot 2024-05-27T07:29:27.353 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:29:27 smithi173 ceph-mon[37000]: fsmap cephfs:5 4 up:standby-replay 2 up:standby 5 up:active 2024-05-27T07:29:27.353 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:29:27 smithi173 ceph-mon[37000]: from='mgr.15333 172.21.15.52:0/416389082' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "l"}]: dispatch 2024-05-27T07:29:27.353 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:29:27 smithi173 ceph-mon[37000]: fsmap cephfs:5 5 up:standby-replay 1 up:standby 5 up:active 2024-05-27T07:29:27.440 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:29:27 smithi105 ceph-mon[35977]: mds.? [v2:172.21.15.173:6832/453411960,v1:172.21.15.173:6833/453411960] up:boot 2024-05-27T07:29:27.440 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:29:27 smithi105 ceph-mon[35977]: fsmap cephfs:5 4 up:standby-replay 2 up:standby 5 up:active 2024-05-27T07:29:27.440 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:29:27 smithi105 ceph-mon[35977]: from='mgr.15333 172.21.15.52:0/416389082' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "l"}]: dispatch 2024-05-27T07:29:27.440 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:29:27 smithi105 ceph-mon[35977]: fsmap cephfs:5 5 up:standby-replay 1 up:standby 5 up:active 2024-05-27T07:29:27.479 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:29:27 smithi052 ceph-mon[31955]: mds.? [v2:172.21.15.173:6832/453411960,v1:172.21.15.173:6833/453411960] up:boot 2024-05-27T07:29:27.480 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:29:27 smithi052 ceph-mon[31955]: fsmap cephfs:5 4 up:standby-replay 2 up:standby 5 up:active 2024-05-27T07:29:27.480 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:29:27 smithi052 ceph-mon[31955]: from='mgr.15333 172.21.15.52:0/416389082' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "l"}]: dispatch 2024-05-27T07:29:27.480 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:29:27 smithi052 ceph-mon[31955]: fsmap cephfs:5 5 up:standby-replay 1 up:standby 5 up:active 2024-05-27T07:29:28.079 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-05-27T07:29:28.080 DEBUG:teuthology.orchestra.run.smithi052:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:38f9d900c83c63799fbdbe61acc9a11b0d3554a6 shell --fsid 73df2636-1bf8-11ef-bc9b-c7b262605968 -- ceph fs dump --format=json 2024-05-27T07:29:28.341 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:29:28 smithi052 ceph-mon[31955]: pgmap v120: 129 pgs: 129 active+clean; 209 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 690 KiB/s rd, 1.6 MiB/s wr, 196 op/s 2024-05-27T07:29:28.352 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:29:28 smithi173 ceph-mon[37000]: pgmap v120: 129 pgs: 129 active+clean; 209 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 690 KiB/s rd, 1.6 MiB/s wr, 196 op/s 2024-05-27T07:29:28.388 INFO:teuthology.orchestra.run.smithi052.stderr:Inferring config /var/lib/ceph/73df2636-1bf8-11ef-bc9b-c7b262605968/mon.a/config 2024-05-27T07:29:28.440 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:29:28 smithi105 ceph-mon[35977]: pgmap v120: 129 pgs: 129 active+clean; 209 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 690 KiB/s rd, 1.6 MiB/s wr, 196 op/s 2024-05-27T07:29:28.698 INFO:journalctl@ceph.mds.a.smithi052.stdout:May 27 07:29:28 smithi052 systemd[1]: ceph-73df2636-1bf8-11ef-bc9b-c7b262605968@mds.a.service: Failed with result 'exit-code'. 2024-05-27T07:29:30.352 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:29:30 smithi173 ceph-mon[37000]: pgmap v121: 129 pgs: 129 active+clean; 221 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 MiB/s rd, 2.9 MiB/s wr, 342 op/s 2024-05-27T07:29:30.440 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:29:30 smithi105 ceph-mon[35977]: pgmap v121: 129 pgs: 129 active+clean; 221 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 MiB/s rd, 2.9 MiB/s wr, 342 op/s 2024-05-27T07:29:30.482 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:29:30 smithi052 ceph-mon[31955]: pgmap v121: 129 pgs: 129 active+clean; 221 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 MiB/s rd, 2.9 MiB/s wr, 342 op/s 2024-05-27T07:29:30.881 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:29:30.890+0000 7f3bf2ffd640 1 -- 172.21.15.52:0/95148456 <== mds.0 v2:172.21.15.105:6838/500348151 1 ==== command_reply(tid 0: 0 ) ==== 8+0+25 (crc 0 0 0) 0x5557fdfed360 con 0x5557fdc721c0 2024-05-27T07:29:30.881 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:29:30.890+0000 7f3bf2ffd640 10 client.25943 handle_command_reply: tid=0 2024-05-27T07:29:30.881 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:29:30.890+0000 7f3c20967740 1 client.25943 shutdown 2024-05-27T07:29:30.882 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:29:30.891+0000 7f3c20967740 1 -- 172.21.15.52:0/95148456 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5557fdcbd4d0 msgr2=0x5557fe091390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:29:30.882 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:29:30.891+0000 7f3c20967740 1 --2- 172.21.15.52:0/95148456 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5557fdcbd4d0 0x5557fe091390 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7f3c08004a90 tx=0x7f3c08004850 comp rx=0 tx=0).stop 2024-05-27T07:29:30.882 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:29:30.891+0000 7f3c20967740 1 -- 172.21.15.52:0/95148456 shutdown_connections 2024-05-27T07:29:30.882 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:29:30.891+0000 7f3c20967740 1 --2- 172.21.15.52:0/95148456 >> [v2:172.21.15.105:6838/500348151,v1:172.21.15.105:6839/500348151] conn(0x5557fdc721c0 0x5557fe106620 crc :-1 s=READY pgs=96 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:29:30.882 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:29:30.891+0000 7f3c20967740 1 --2- 172.21.15.52:0/95148456 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5557fdcde1b0 0x5557fe0918d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:29:30.882 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:29:30.891+0000 7f3c20967740 1 --2- 172.21.15.52:0/95148456 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5557fdcbd4d0 0x5557fe091390 unknown :-1 s=CLOSED pgs=123 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:29:30.882 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:29:30.891+0000 7f3c20967740 1 --2- 172.21.15.52:0/95148456 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x5557fdcb52f0 0x5557fe096040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:29:30.882 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:29:30.891+0000 7f3c20967740 1 -- 172.21.15.52:0/95148456 >> 172.21.15.52:0/95148456 conn(0x5557fdcd2300 msgr2=0x5557fe08ba40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:29:30.882 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:29:30.891+0000 7f3bf2ffd640 0 client.25943 ms_handle_reset on v2:172.21.15.105:6838/500348151 2024-05-27T07:29:30.883 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stdout:{ 2024-05-27T07:29:30.883 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stdout: "return_code": 0 2024-05-27T07:29:30.883 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stdout:} 2024-05-27T07:29:30.883 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:29:30.891+0000 7f3c20967740 1 -- 172.21.15.52:0/95148456 shutdown_connections 2024-05-27T07:29:30.883 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:29:30.891+0000 7f3c20967740 1 -- 172.21.15.52:0/95148456 wait complete. 2024-05-27T07:29:30.883 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:29:30.891+0000 7f3c20967740 20 client.25943 trim_cache size 0 max 16384 2024-05-27T07:29:30.919 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:29:30.928+0000 7f3c199d5640 1 -- 172.21.15.52:0/1042421655 >> [v2:172.21.15.52:6800/1822318384,v1:172.21.15.52:6801/1822318384] conn(0x7f3bfc0b2950 msgr2=0x7f3bfc0b4e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:29:30.919 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:29:30.928+0000 7f3c199d5640 1 --2- 172.21.15.52:0/1042421655 >> [v2:172.21.15.52:6800/1822318384,v1:172.21.15.52:6801/1822318384] conn(0x7f3bfc0b2950 0x7f3bfc0b4e60 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x5557fe07c540 tx=0x7f3c0c005d80 comp rx=0 tx=0).stop 2024-05-27T07:29:30.919 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:29:30.928+0000 7f3c199d5640 1 -- 172.21.15.52:0/1042421655 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f3c141032e0 msgr2=0x7f3c140ffaf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:29:30.919 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:29:30.928+0000 7f3c199d5640 1 --2- 172.21.15.52:0/1042421655 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f3c141032e0 0x7f3c140ffaf0 secure :-1 s=READY pgs=607 cs=0 l=1 rev1=1 crypto rx=0x7f3c0400e9f0 tx=0x7f3c0400eeb0 comp rx=0 tx=0).stop 2024-05-27T07:29:30.919 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:29:30.929+0000 7f3c199d5640 1 -- 172.21.15.52:0/1042421655 shutdown_connections 2024-05-27T07:29:30.919 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:29:30.929+0000 7f3c199d5640 1 --2- 172.21.15.52:0/1042421655 >> [v2:172.21.15.52:6800/1822318384,v1:172.21.15.52:6801/1822318384] conn(0x7f3bfc0b2950 0x7f3bfc0b4e60 unknown :-1 s=CLOSED pgs=110 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:29:30.920 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:29:30.929+0000 7f3c199d5640 1 --2- 172.21.15.52:0/1042421655 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f3c14109030 0x7f3c14100570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:29:30.920 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:29:30.929+0000 7f3c199d5640 1 --2- 172.21.15.52:0/1042421655 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f3c14108b90 0x7f3c14100030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:29:30.920 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:29:30.929+0000 7f3c199d5640 1 --2- 172.21.15.52:0/1042421655 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f3c141032e0 0x7f3c140ffaf0 unknown :-1 s=CLOSED pgs=607 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:29:30.920 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:29:30.929+0000 7f3c199d5640 1 -- 172.21.15.52:0/1042421655 >> 172.21.15.52:0/1042421655 conn(0x7f3c140782d0 msgr2=0x7f3c1406a920 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:29:30.920 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:29:30.930+0000 7f3c199d5640 1 -- 172.21.15.52:0/1042421655 shutdown_connections 2024-05-27T07:29:30.920 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:29:30.930+0000 7f3c199d5640 1 -- 172.21.15.52:0/1042421655 wait complete. 2024-05-27T07:29:30.980 INFO:journalctl@ceph.mds.d.smithi052.stdout:May 27 07:29:30 smithi052 systemd[1]: ceph-73df2636-1bf8-11ef-bc9b-c7b262605968@mds.d.service: Scheduled restart job, restart counter is at 3. 2024-05-27T07:29:30.980 INFO:journalctl@ceph.mds.d.smithi052.stdout:May 27 07:29:30 smithi052 systemd[1]: Stopped Ceph mds.d for 73df2636-1bf8-11ef-bc9b-c7b262605968. 2024-05-27T07:29:30.980 INFO:journalctl@ceph.mds.d.smithi052.stdout:May 27 07:29:30 smithi052 systemd[1]: ceph-73df2636-1bf8-11ef-bc9b-c7b262605968@mds.d.service: Consumed 1.383s CPU time. 2024-05-27T07:29:30.980 INFO:journalctl@ceph.mds.d.smithi052.stdout:May 27 07:29:30 smithi052 systemd[1]: Starting Ceph mds.d for 73df2636-1bf8-11ef-bc9b-c7b262605968... 2024-05-27T07:29:30.980 INFO:journalctl@ceph.mds.g.smithi052.stdout:May 27 07:29:30 smithi052 systemd[1]: ceph-73df2636-1bf8-11ef-bc9b-c7b262605968@mds.g.service: Scheduled restart job, restart counter is at 4. 2024-05-27T07:29:30.980 INFO:journalctl@ceph.mds.g.smithi052.stdout:May 27 07:29:30 smithi052 systemd[1]: Stopped Ceph mds.g for 73df2636-1bf8-11ef-bc9b-c7b262605968. 2024-05-27T07:29:30.981 INFO:journalctl@ceph.mds.g.smithi052.stdout:May 27 07:29:30 smithi052 systemd[1]: ceph-73df2636-1bf8-11ef-bc9b-c7b262605968@mds.g.service: Consumed 1.077s CPU time. 2024-05-27T07:29:30.981 INFO:journalctl@ceph.mds.g.smithi052.stdout:May 27 07:29:30 smithi052 systemd[1]: Starting Ceph mds.g for 73df2636-1bf8-11ef-bc9b-c7b262605968... 2024-05-27T07:29:32.189 INFO:journalctl@ceph.mds.d.smithi052.stdout:May 27 07:29:31 smithi052 podman[105189]: 2024-05-27 07:29:31.900977921 +0000 UTC m=+0.206175544 image pull e36ea4fc8a5807e8d2701370e350a0de3856538797f7fba64cfce7e344bd6799 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6 2024-05-27T07:29:32.190 INFO:journalctl@ceph.mds.d.smithi052.stdout:May 27 07:29:32 smithi052 podman[105189]: 2024-05-27 07:29:32.071556837 +0000 UTC m=+0.376754456 container create 935d31972b66528c0799298a9629bb77353b540e9609ecda523a8fd9079cabc0 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-d, org.label-schema.license=GPLv2, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, maintainer=Guillaume Abrioux , GIT_CLEAN=True, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, org.label-schema.build-date=20240521, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS) 2024-05-27T07:29:32.190 INFO:journalctl@ceph.mds.g.smithi052.stdout:May 27 07:29:31 smithi052 podman[105188]: 2024-05-27 07:29:31.900858301 +0000 UTC m=+0.207045449 image pull e36ea4fc8a5807e8d2701370e350a0de3856538797f7fba64cfce7e344bd6799 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6 2024-05-27T07:29:32.190 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:29:32 smithi052 ceph-mon[31955]: pgmap v122: 129 pgs: 129 active+clean; 243 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 3.6 MiB/s rd, 5.0 MiB/s wr, 751 op/s 2024-05-27T07:29:32.352 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:29:32 smithi173 ceph-mon[37000]: pgmap v122: 129 pgs: 129 active+clean; 243 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 3.6 MiB/s rd, 5.0 MiB/s wr, 751 op/s 2024-05-27T07:29:32.440 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:29:32 smithi105 ceph-mon[35977]: pgmap v122: 129 pgs: 129 active+clean; 243 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 3.6 MiB/s rd, 5.0 MiB/s wr, 751 op/s 2024-05-27T07:29:32.480 INFO:journalctl@ceph.mds.d.smithi052.stdout:May 27 07:29:32 smithi052 podman[105189]: 2024-05-27 07:29:32.32451016 +0000 UTC m=+0.629707799 container init 935d31972b66528c0799298a9629bb77353b540e9609ecda523a8fd9079cabc0 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-d, GIT_BRANCH=HEAD, org.label-schema.build-date=20240521, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, ceph=True, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, CEPH_POINT_RELEASE=, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, GIT_CLEAN=True) 2024-05-27T07:29:32.480 INFO:journalctl@ceph.mds.d.smithi052.stdout:May 27 07:29:32 smithi052 podman[105189]: 2024-05-27 07:29:32.327866241 +0000 UTC m=+0.633063868 container start 935d31972b66528c0799298a9629bb77353b540e9609ecda523a8fd9079cabc0 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-d, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.build-date=20240521, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, GIT_CLEAN=True, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image) 2024-05-27T07:29:32.480 INFO:journalctl@ceph.mds.d.smithi052.stdout:May 27 07:29:32 smithi052 ceph-mds[105225]: -- 172.21.15.52:0/2762823206 <== mon.0 v2:172.21.15.52:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x5650d9f33a40 con 0x5650d9f7b180 2024-05-27T07:29:32.480 INFO:journalctl@ceph.mds.d.smithi052.stdout:May 27 07:29:32 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-d[105221]: starting mds.d at 2024-05-27T07:29:32.480 INFO:journalctl@ceph.mds.d.smithi052.stdout:May 27 07:29:32 smithi052 bash[105189]: 935d31972b66528c0799298a9629bb77353b540e9609ecda523a8fd9079cabc0 2024-05-27T07:29:32.480 INFO:journalctl@ceph.mds.d.smithi052.stdout:May 27 07:29:32 smithi052 systemd[1]: Started Ceph mds.d for 73df2636-1bf8-11ef-bc9b-c7b262605968. 2024-05-27T07:29:32.787 INFO:journalctl@ceph.mds.g.smithi052.stdout:May 27 07:29:32 smithi052 podman[105188]: 2024-05-27 07:29:32.542271928 +0000 UTC m=+0.848459062 container create 64363f89b5da86618bd5c5ce360a83faf2f428ec1d1fca15fef1b39bb68b23e0 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-g, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, ceph=True, org.label-schema.license=GPLv2, GIT_CLEAN=True, maintainer=Guillaume Abrioux , GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, io.buildah.version=1.33.5, org.label-schema.build-date=20240521, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0) 2024-05-27T07:29:33.014 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:29:33.022+0000 7f044aed2700 1 -- 172.21.15.52:0/330949174 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f0444106570 msgr2=0x7f0444106980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:29:33.014 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:29:33.022+0000 7f044aed2700 1 --2- 172.21.15.52:0/330949174 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f0444106570 0x7f0444106980 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7f0438009990 tx=0x7f0438009ca0 comp rx=0 tx=0).stop 2024-05-27T07:29:33.014 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:29:33.022+0000 7f044aed2700 1 -- 172.21.15.52:0/330949174 shutdown_connections 2024-05-27T07:29:33.014 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:29:33.022+0000 7f044aed2700 1 --2- 172.21.15.52:0/330949174 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f0444106ec0 0x7f0444113350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:29:33.014 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:29:33.022+0000 7f044aed2700 1 --2- 172.21.15.52:0/330949174 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f044410c5b0 0x7f044410c9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:29:33.014 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:29:33.022+0000 7f044aed2700 1 --2- 172.21.15.52:0/330949174 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f0444106570 0x7f0444106980 unknown :-1 s=CLOSED pgs=126 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:29:33.014 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:29:33.022+0000 7f044aed2700 1 -- 172.21.15.52:0/330949174 >> 172.21.15.52:0/330949174 conn(0x7f0444102070 msgr2=0x7f04441044c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:29:33.015 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:29:33.023+0000 7f044aed2700 1 -- 172.21.15.52:0/330949174 shutdown_connections 2024-05-27T07:29:33.015 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:29:33.023+0000 7f044aed2700 1 -- 172.21.15.52:0/330949174 wait complete. 2024-05-27T07:29:33.015 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:29:33.024+0000 7f044aed2700 1 Processor -- start 2024-05-27T07:29:33.015 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:29:33.025+0000 7f044aed2700 1 -- start start 2024-05-27T07:29:33.016 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:29:33.025+0000 7f044aed2700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f0444106570 0x7f0444075440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:29:33.016 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:29:33.025+0000 7f044aed2700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f0444106ec0 0x7f044406e570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:29:33.016 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:29:33.025+0000 7f044aed2700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f044406eb70 0x7f044406f040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:29:33.016 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:29:33.025+0000 7f044aed2700 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x7f0444116760 con 0x7f0444106570 2024-05-27T07:29:33.016 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:29:33.025+0000 7f044aed2700 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f04441165e0 con 0x7f044406eb70 2024-05-27T07:29:33.017 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:29:33.025+0000 7f044aed2700 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f04441168e0 con 0x7f0444106ec0 2024-05-27T07:29:33.017 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:29:33.026+0000 7f0443fff700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f0444106ec0 0x7f044406e570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:29:33.017 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:29:33.026+0000 7f0443fff700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f0444106ec0 0x7f044406e570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.173:3300/0 says I am v2:172.21.15.52:60176/0 (socket says 172.21.15.52:60176) 2024-05-27T07:29:33.017 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:29:33.026+0000 7f0443fff700 1 -- 172.21.15.52:0/966372404 learned_addr learned my addr 172.21.15.52:0/966372404 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-05-27T07:29:33.017 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:29:33.026+0000 7f0443fff700 1 -- 172.21.15.52:0/966372404 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f044406eb70 msgr2=0x7f044406f040 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-05-27T07:29:33.017 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:29:33.026+0000 7f0443fff700 1 --2- 172.21.15.52:0/966372404 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f044406eb70 0x7f044406f040 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:29:33.017 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:29:33.026+0000 7f0443fff700 1 -- 172.21.15.52:0/966372404 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f0444106570 msgr2=0x7f0444075440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:29:33.018 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:29:33.026+0000 7f0443fff700 1 --2- 172.21.15.52:0/966372404 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f0444106570 0x7f0444075440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:29:33.018 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:29:33.026+0000 7f0443fff700 1 -- 172.21.15.52:0/966372404 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0444073290 con 0x7f0444106ec0 2024-05-27T07:29:33.018 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:29:33.027+0000 7f0443fff700 1 --2- 172.21.15.52:0/966372404 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f0444106ec0 0x7f044406e570 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7f043800b5c0 tx=0x7f0438005590 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:29:33.018 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:29:33.027+0000 7f0441ffb700 1 -- 172.21.15.52:0/966372404 <== mon.1 v2:172.21.15.173:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0438026070 con 0x7f0444106ec0 2024-05-27T07:29:33.018 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:29:33.027+0000 7f044aed2700 1 -- 172.21.15.52:0/966372404 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0444073520 con 0x7f0444106ec0 2024-05-27T07:29:33.018 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:29:33.028+0000 7f044aed2700 1 -- 172.21.15.52:0/966372404 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0444073a50 con 0x7f0444106ec0 2024-05-27T07:29:33.020 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:29:33.029+0000 7f044aed2700 1 -- 172.21.15.52:0/966372404 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0444073d70 con 0x7f0444106ec0 2024-05-27T07:29:33.020 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:29:33.030+0000 7f0441ffb700 1 -- 172.21.15.52:0/966372404 <== mon.1 v2:172.21.15.173:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f043800f910 con 0x7f0444106ec0 2024-05-27T07:29:33.020 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:29:33.030+0000 7f0441ffb700 1 -- 172.21.15.52:0/966372404 <== mon.1 v2:172.21.15.173:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f04380218b0 con 0x7f0444106ec0 2024-05-27T07:29:33.023 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:29:33.032+0000 7f0441ffb700 1 -- 172.21.15.52:0/966372404 <== mon.1 v2:172.21.15.173:3300/0 4 ==== mgrmap(e 26) ==== 149783+0+0 (secure 0 0 0) 0x7f04380050d0 con 0x7f0444106ec0 2024-05-27T07:29:33.024 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:29:33.033+0000 7f0441ffb700 1 --2- 172.21.15.52:0/966372404 >> [v2:172.21.15.52:6800/1822318384,v1:172.21.15.52:6801/1822318384] conn(0x7f042c0b2a70 0x7f042c0b4f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:29:33.024 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:29:33.034+0000 7f0441ffb700 1 -- 172.21.15.52:0/966372404 <== mon.1 v2:172.21.15.173:3300/0 5 ==== osd_map(103..103 src has 1..103) ==== 9379+0+0 (secure 0 0 0) 0x7f04380e34b0 con 0x7f0444106ec0 2024-05-27T07:29:33.024 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:29:33.034+0000 7f0448c6e700 1 --2- 172.21.15.52:0/966372404 >> [v2:172.21.15.52:6800/1822318384,v1:172.21.15.52:6801/1822318384] conn(0x7f042c0b2a70 0x7f042c0b4f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:29:33.025 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:29:33.035+0000 7f0448c6e700 1 --2- 172.21.15.52:0/966372404 >> [v2:172.21.15.52:6800/1822318384,v1:172.21.15.52:6801/1822318384] conn(0x7f042c0b2a70 0x7f042c0b4f80 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7f044406fdc0 tx=0x7f043c009250 comp rx=0 tx=0).ready entity=mgr.15333 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:29:33.028 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:29:33.037+0000 7f0441ffb700 1 -- 172.21.15.52:0/966372404 <== mon.1 v2:172.21.15.173:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191963 (secure 0 0 0) 0x7f0438013070 con 0x7f0444106ec0 2024-05-27T07:29:33.097 INFO:journalctl@ceph.mds.g.smithi052.stdout:May 27 07:29:32 smithi052 podman[105188]: 2024-05-27 07:29:32.879450253 +0000 UTC m=+1.185637408 container init 64363f89b5da86618bd5c5ce360a83faf2f428ec1d1fca15fef1b39bb68b23e0 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-g, org.label-schema.build-date=20240521, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, CEPH_POINT_RELEASE=, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, GIT_CLEAN=True) 2024-05-27T07:29:33.097 INFO:journalctl@ceph.mds.g.smithi052.stdout:May 27 07:29:32 smithi052 podman[105188]: 2024-05-27 07:29:32.885752187 +0000 UTC m=+1.191939319 container start 64363f89b5da86618bd5c5ce360a83faf2f428ec1d1fca15fef1b39bb68b23e0 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-g, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240521, ceph=True, org.label-schema.schema-version=1.0, GIT_CLEAN=True, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2) 2024-05-27T07:29:33.097 INFO:journalctl@ceph.mds.g.smithi052.stdout:May 27 07:29:32 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-g[105245]: starting mds.g at 2024-05-27T07:29:33.352 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:29:33 smithi173 ceph-mon[37000]: daemon mds.d restarted 2024-05-27T07:29:33.378 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:29:33 smithi052 ceph-mon[31955]: daemon mds.d restarted 2024-05-27T07:29:33.379 INFO:journalctl@ceph.mds.d.smithi052.stdout:May 27 07:29:33 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-d[105221]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: In function 'std::set SnapClient::filter(const std::set&) const' thread 7f11297cc700 time 2024-05-27T07:29:33.169136+0000 2024-05-27T07:29:33.379 INFO:journalctl@ceph.mds.d.smithi052.stdout:May 27 07:29:33 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-d[105221]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: 197: FAILED ceph_assert(cached_version > 0) 2024-05-27T07:29:33.379 INFO:journalctl@ceph.mds.d.smithi052.stdout:May 27 07:29:33 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-d[105221]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:29:33.379 INFO:journalctl@ceph.mds.d.smithi052.stdout:May 27 07:29:33 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-d[105221]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x127) [0x7f113c011ba6] 2024-05-27T07:29:33.379 INFO:journalctl@ceph.mds.d.smithi052.stdout:May 27 07:29:33 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-d[105221]: 2: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7f113c011dd1] 2024-05-27T07:29:33.379 INFO:journalctl@ceph.mds.d.smithi052.stdout:May 27 07:29:33 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-d[105221]: 3: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x5650d7d0c23d] 2024-05-27T07:29:33.379 INFO:journalctl@ceph.mds.d.smithi052.stdout:May 27 07:29:33 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-d[105221]: 4: (SnapRealm::build_snap_set() const+0x323) [0x5650d7cf7971] 2024-05-27T07:29:33.379 INFO:journalctl@ceph.mds.d.smithi052.stdout:May 27 07:29:33 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-d[105221]: 5: (SnapRealm::check_cache() const+0x13e) [0x5650d7cf5fa0] 2024-05-27T07:29:33.379 INFO:journalctl@ceph.mds.d.smithi052.stdout:May 27 07:29:33 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-d[105221]: 6: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x5650d7cf6651] 2024-05-27T07:29:33.379 INFO:journalctl@ceph.mds.d.smithi052.stdout:May 27 07:29:33 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-d[105221]: 7: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x5650d7c8b347] 2024-05-27T07:29:33.380 INFO:journalctl@ceph.mds.d.smithi052.stdout:May 27 07:29:33 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-d[105221]: 8: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x5650d798787e] 2024-05-27T07:29:33.380 INFO:journalctl@ceph.mds.d.smithi052.stdout:May 27 07:29:33 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-d[105221]: 9: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x2702) [0x5650d798ce94] 2024-05-27T07:29:33.380 INFO:journalctl@ceph.mds.d.smithi052.stdout:May 27 07:29:33 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-d[105221]: 10: (EUpdate::replay(MDSRank*)+0x63) [0x5650d7997969] 2024-05-27T07:29:33.380 INFO:journalctl@ceph.mds.d.smithi052.stdout:May 27 07:29:33 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-d[105221]: 11: (MDLog::_replay_thread()+0x1f38) [0x5650d7d3048c] 2024-05-27T07:29:33.380 INFO:journalctl@ceph.mds.d.smithi052.stdout:May 27 07:29:33 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-d[105221]: 12: (MDLog::ReplayThread::entry()+0x11) [0x5650d7d399cb] 2024-05-27T07:29:33.380 INFO:journalctl@ceph.mds.d.smithi052.stdout:May 27 07:29:33 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-d[105221]: 13: (Thread::entry_wrapper()+0x43) [0x7f113bfe072d] 2024-05-27T07:29:33.380 INFO:journalctl@ceph.mds.d.smithi052.stdout:May 27 07:29:33 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-d[105221]: 14: (Thread::_entry_func(void*)+0xd) [0x7f113bfe0749] 2024-05-27T07:29:33.380 INFO:journalctl@ceph.mds.d.smithi052.stdout:May 27 07:29:33 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-d[105221]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7f113ab841ca] 2024-05-27T07:29:33.380 INFO:journalctl@ceph.mds.d.smithi052.stdout:May 27 07:29:33 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-d[105221]: 16: clone() 2024-05-27T07:29:33.380 INFO:journalctl@ceph.mds.d.smithi052.stdout:May 27 07:29:33 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-d[105221]: 2024-05-27T07:29:33.171+0000 7f11297cc700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: In function 'std::set SnapClient::filter(const std::set&) const' thread 7f11297cc700 time 2024-05-27T07:29:33.169136+0000 2024-05-27T07:29:33.381 INFO:journalctl@ceph.mds.g.smithi052.stdout:May 27 07:29:33 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-g[105245]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: In function 'std::set SnapClient::filter(const std::set&) const' thread 7ff463c87700 time 2024-05-27T07:29:33.220576+0000 2024-05-27T07:29:33.381 INFO:journalctl@ceph.mds.g.smithi052.stdout:May 27 07:29:33 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-g[105245]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: 197: FAILED ceph_assert(cached_version > 0) 2024-05-27T07:29:33.381 INFO:journalctl@ceph.mds.g.smithi052.stdout:May 27 07:29:33 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-g[105245]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:29:33.381 INFO:journalctl@ceph.mds.g.smithi052.stdout:May 27 07:29:33 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-g[105245]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x127) [0x7ff4764ccba6] 2024-05-27T07:29:33.381 INFO:journalctl@ceph.mds.g.smithi052.stdout:May 27 07:29:33 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-g[105245]: 2: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7ff4764ccdd1] 2024-05-27T07:29:33.381 INFO:journalctl@ceph.mds.g.smithi052.stdout:May 27 07:29:33 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-g[105245]: 3: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x55ce81f4d23d] 2024-05-27T07:29:33.381 INFO:journalctl@ceph.mds.g.smithi052.stdout:May 27 07:29:33 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-g[105245]: 4: (SnapRealm::build_snap_set() const+0x323) [0x55ce81f38971] 2024-05-27T07:29:33.381 INFO:journalctl@ceph.mds.g.smithi052.stdout:May 27 07:29:33 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-g[105245]: 5: (SnapRealm::check_cache() const+0x13e) [0x55ce81f36fa0] 2024-05-27T07:29:33.382 INFO:journalctl@ceph.mds.g.smithi052.stdout:May 27 07:29:33 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-g[105245]: 6: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x55ce81f37651] 2024-05-27T07:29:33.382 INFO:journalctl@ceph.mds.g.smithi052.stdout:May 27 07:29:33 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-g[105245]: 7: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x55ce81ecc347] 2024-05-27T07:29:33.382 INFO:journalctl@ceph.mds.g.smithi052.stdout:May 27 07:29:33 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-g[105245]: 8: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x55ce81bc887e] 2024-05-27T07:29:33.382 INFO:journalctl@ceph.mds.g.smithi052.stdout:May 27 07:29:33 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-g[105245]: 9: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x2702) [0x55ce81bcde94] 2024-05-27T07:29:33.382 INFO:journalctl@ceph.mds.g.smithi052.stdout:May 27 07:29:33 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-g[105245]: 10: (EUpdate::replay(MDSRank*)+0x63) [0x55ce81bd8969] 2024-05-27T07:29:33.382 INFO:journalctl@ceph.mds.g.smithi052.stdout:May 27 07:29:33 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-g[105245]: 11: (MDLog::_replay_thread()+0x1f38) [0x55ce81f7148c] 2024-05-27T07:29:33.382 INFO:journalctl@ceph.mds.g.smithi052.stdout:May 27 07:29:33 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-g[105245]: 12: (MDLog::ReplayThread::entry()+0x11) [0x55ce81f7a9cb] 2024-05-27T07:29:33.382 INFO:journalctl@ceph.mds.g.smithi052.stdout:May 27 07:29:33 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-g[105245]: 13: (Thread::entry_wrapper()+0x43) [0x7ff47649b72d] 2024-05-27T07:29:33.382 INFO:journalctl@ceph.mds.g.smithi052.stdout:May 27 07:29:33 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-g[105245]: 14: (Thread::_entry_func(void*)+0xd) [0x7ff47649b749] 2024-05-27T07:29:33.382 INFO:journalctl@ceph.mds.g.smithi052.stdout:May 27 07:29:33 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-g[105245]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7ff47503f1ca] 2024-05-27T07:29:33.382 INFO:journalctl@ceph.mds.g.smithi052.stdout:May 27 07:29:33 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-g[105245]: 16: clone() 2024-05-27T07:29:33.382 INFO:journalctl@ceph.mds.g.smithi052.stdout:May 27 07:29:33 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-g[105245]: *** Caught signal (Aborted) ** 2024-05-27T07:29:33.383 INFO:journalctl@ceph.mds.g.smithi052.stdout:May 27 07:29:33 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-g[105245]: in thread 7ff463c87700 thread_name:md_log_replay 2024-05-27T07:29:33.383 INFO:journalctl@ceph.mds.d.smithi052.stdout:May 27 07:29:33 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-d[105221]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: 197: FAILED ceph_assert(cached_version > 0) 2024-05-27T07:29:33.383 INFO:journalctl@ceph.mds.d.smithi052.stdout:May 27 07:29:33 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-d[105221]: 2024-05-27T07:29:33.383 INFO:journalctl@ceph.mds.d.smithi052.stdout:May 27 07:29:33 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-d[105221]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:29:33.383 INFO:journalctl@ceph.mds.d.smithi052.stdout:May 27 07:29:33 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-d[105221]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x127) [0x7f113c011ba6] 2024-05-27T07:29:33.383 INFO:journalctl@ceph.mds.d.smithi052.stdout:May 27 07:29:33 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-d[105221]: 2: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7f113c011dd1] 2024-05-27T07:29:33.383 INFO:journalctl@ceph.mds.d.smithi052.stdout:May 27 07:29:33 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-d[105221]: 3: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x5650d7d0c23d] 2024-05-27T07:29:33.384 INFO:journalctl@ceph.mds.d.smithi052.stdout:May 27 07:29:33 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-d[105221]: 4: (SnapRealm::build_snap_set() const+0x323) [0x5650d7cf7971] 2024-05-27T07:29:33.384 INFO:journalctl@ceph.mds.d.smithi052.stdout:May 27 07:29:33 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-d[105221]: 5: (SnapRealm::check_cache() const+0x13e) [0x5650d7cf5fa0] 2024-05-27T07:29:33.384 INFO:journalctl@ceph.mds.d.smithi052.stdout:May 27 07:29:33 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-d[105221]: 6: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x5650d7cf6651] 2024-05-27T07:29:33.384 INFO:journalctl@ceph.mds.d.smithi052.stdout:May 27 07:29:33 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-d[105221]: 7: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x5650d7c8b347] 2024-05-27T07:29:33.384 INFO:journalctl@ceph.mds.d.smithi052.stdout:May 27 07:29:33 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-d[105221]: 8: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x5650d798787e] 2024-05-27T07:29:33.384 INFO:journalctl@ceph.mds.d.smithi052.stdout:May 27 07:29:33 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-d[105221]: 9: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x2702) [0x5650d798ce94] 2024-05-27T07:29:33.384 INFO:journalctl@ceph.mds.d.smithi052.stdout:May 27 07:29:33 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-d[105221]: 10: (EUpdate::replay(MDSRank*)+0x63) [0x5650d7997969] 2024-05-27T07:29:33.384 INFO:journalctl@ceph.mds.d.smithi052.stdout:May 27 07:29:33 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-d[105221]: 11: (MDLog::_replay_thread()+0x1f38) [0x5650d7d3048c] 2024-05-27T07:29:33.384 INFO:journalctl@ceph.mds.d.smithi052.stdout:May 27 07:29:33 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-d[105221]: 12: (MDLog::ReplayThread::entry()+0x11) [0x5650d7d399cb] 2024-05-27T07:29:33.384 INFO:journalctl@ceph.mds.d.smithi052.stdout:May 27 07:29:33 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-d[105221]: 13: (Thread::entry_wrapper()+0x43) [0x7f113bfe072d] 2024-05-27T07:29:33.384 INFO:journalctl@ceph.mds.d.smithi052.stdout:May 27 07:29:33 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-d[105221]: 14: (Thread::_entry_func(void*)+0xd) [0x7f113bfe0749] 2024-05-27T07:29:33.384 INFO:journalctl@ceph.mds.d.smithi052.stdout:May 27 07:29:33 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-d[105221]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7f113ab841ca] 2024-05-27T07:29:33.384 INFO:journalctl@ceph.mds.d.smithi052.stdout:May 27 07:29:33 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-d[105221]: 16: clone() 2024-05-27T07:29:33.385 INFO:journalctl@ceph.mds.d.smithi052.stdout:May 27 07:29:33 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-d[105221]: 2024-05-27T07:29:33.385 INFO:journalctl@ceph.mds.d.smithi052.stdout:May 27 07:29:33 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-d[105221]: *** Caught signal (Aborted) ** 2024-05-27T07:29:33.385 INFO:journalctl@ceph.mds.d.smithi052.stdout:May 27 07:29:33 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-d[105221]: in thread 7f11297cc700 thread_name:md_log_replay 2024-05-27T07:29:33.385 INFO:journalctl@ceph.mds.d.smithi052.stdout:May 27 07:29:33 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-d[105221]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:29:33.385 INFO:journalctl@ceph.mds.d.smithi052.stdout:May 27 07:29:33 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-d[105221]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f113ab8ed20] 2024-05-27T07:29:33.385 INFO:journalctl@ceph.mds.d.smithi052.stdout:May 27 07:29:33 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-d[105221]: 2: gsignal() 2024-05-27T07:29:33.385 INFO:journalctl@ceph.mds.d.smithi052.stdout:May 27 07:29:33 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-d[105221]: 3: abort() 2024-05-27T07:29:33.385 INFO:journalctl@ceph.mds.d.smithi052.stdout:May 27 07:29:33 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-d[105221]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x26e) [0x7f113c011ced] 2024-05-27T07:29:33.385 INFO:journalctl@ceph.mds.d.smithi052.stdout:May 27 07:29:33 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-d[105221]: 5: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7f113c011dd1] 2024-05-27T07:29:33.385 INFO:journalctl@ceph.mds.d.smithi052.stdout:May 27 07:29:33 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-d[105221]: 6: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x5650d7d0c23d] 2024-05-27T07:29:33.385 INFO:journalctl@ceph.mds.d.smithi052.stdout:May 27 07:29:33 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-d[105221]: 7: (SnapRealm::build_snap_set() const+0x323) [0x5650d7cf7971] 2024-05-27T07:29:33.385 INFO:journalctl@ceph.mds.d.smithi052.stdout:May 27 07:29:33 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-d[105221]: 8: (SnapRealm::check_cache() const+0x13e) [0x5650d7cf5fa0] 2024-05-27T07:29:33.385 INFO:journalctl@ceph.mds.d.smithi052.stdout:May 27 07:29:33 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-d[105221]: 9: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x5650d7cf6651] 2024-05-27T07:29:33.385 INFO:journalctl@ceph.mds.d.smithi052.stdout:May 27 07:29:33 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-d[105221]: 10: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x5650d7c8b347] 2024-05-27T07:29:33.386 INFO:journalctl@ceph.mds.d.smithi052.stdout:May 27 07:29:33 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-d[105221]: 11: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x5650d798787e] 2024-05-27T07:29:33.386 INFO:journalctl@ceph.mds.d.smithi052.stdout:May 27 07:29:33 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-d[105221]: 12: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x2702) [0x5650d798ce94] 2024-05-27T07:29:33.386 INFO:journalctl@ceph.mds.d.smithi052.stdout:May 27 07:29:33 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-d[105221]: 13: (EUpdate::replay(MDSRank*)+0x63) [0x5650d7997969] 2024-05-27T07:29:33.386 INFO:journalctl@ceph.mds.d.smithi052.stdout:May 27 07:29:33 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-d[105221]: 14: (MDLog::_replay_thread()+0x1f38) [0x5650d7d3048c] 2024-05-27T07:29:33.386 INFO:journalctl@ceph.mds.d.smithi052.stdout:May 27 07:29:33 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-d[105221]: 15: (MDLog::ReplayThread::entry()+0x11) [0x5650d7d399cb] 2024-05-27T07:29:33.386 INFO:journalctl@ceph.mds.d.smithi052.stdout:May 27 07:29:33 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-d[105221]: 16: (Thread::entry_wrapper()+0x43) [0x7f113bfe072d] 2024-05-27T07:29:33.386 INFO:journalctl@ceph.mds.d.smithi052.stdout:May 27 07:29:33 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-d[105221]: 17: (Thread::_entry_func(void*)+0xd) [0x7f113bfe0749] 2024-05-27T07:29:33.386 INFO:journalctl@ceph.mds.d.smithi052.stdout:May 27 07:29:33 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-d[105221]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7f113ab841ca] 2024-05-27T07:29:33.386 INFO:journalctl@ceph.mds.d.smithi052.stdout:May 27 07:29:33 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-d[105221]: 19: clone() 2024-05-27T07:29:33.386 INFO:journalctl@ceph.mds.d.smithi052.stdout:May 27 07:29:33 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-d[105221]: 2024-05-27T07:29:33.174+0000 7f11297cc700 -1 *** Caught signal (Aborted) ** 2024-05-27T07:29:33.386 INFO:journalctl@ceph.mds.d.smithi052.stdout:May 27 07:29:33 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-d[105221]: in thread 7f11297cc700 thread_name:md_log_replay 2024-05-27T07:29:33.386 INFO:journalctl@ceph.mds.d.smithi052.stdout:May 27 07:29:33 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-d[105221]: 2024-05-27T07:29:33.386 INFO:journalctl@ceph.mds.d.smithi052.stdout:May 27 07:29:33 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-d[105221]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:29:33.387 INFO:journalctl@ceph.mds.d.smithi052.stdout:May 27 07:29:33 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-d[105221]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f113ab8ed20] 2024-05-27T07:29:33.387 INFO:journalctl@ceph.mds.d.smithi052.stdout:May 27 07:29:33 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-d[105221]: 2: gsignal() 2024-05-27T07:29:33.387 INFO:journalctl@ceph.mds.d.smithi052.stdout:May 27 07:29:33 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-d[105221]: 3: abort() 2024-05-27T07:29:33.387 INFO:journalctl@ceph.mds.d.smithi052.stdout:May 27 07:29:33 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-d[105221]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x26e) [0x7f113c011ced] 2024-05-27T07:29:33.387 INFO:journalctl@ceph.mds.d.smithi052.stdout:May 27 07:29:33 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-d[105221]: 5: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7f113c011dd1] 2024-05-27T07:29:33.387 INFO:journalctl@ceph.mds.d.smithi052.stdout:May 27 07:29:33 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-d[105221]: 6: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x5650d7d0c23d] 2024-05-27T07:29:33.387 INFO:journalctl@ceph.mds.d.smithi052.stdout:May 27 07:29:33 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-d[105221]: 7: (SnapRealm::build_snap_set() const+0x323) [0x5650d7cf7971] 2024-05-27T07:29:33.387 INFO:journalctl@ceph.mds.d.smithi052.stdout:May 27 07:29:33 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-d[105221]: 8: (SnapRealm::check_cache() const+0x13e) [0x5650d7cf5fa0] 2024-05-27T07:29:33.387 INFO:journalctl@ceph.mds.d.smithi052.stdout:May 27 07:29:33 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-d[105221]: 9: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x5650d7cf6651] 2024-05-27T07:29:33.387 INFO:journalctl@ceph.mds.g.smithi052.stdout:May 27 07:29:33 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-g[105245]: 2024-05-27T07:29:33.222+0000 7ff463c87700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: In function 'std::set SnapClient::filter(const std::set&) const' thread 7ff463c87700 time 2024-05-27T07:29:33.220576+0000 2024-05-27T07:29:33.388 INFO:journalctl@ceph.mds.g.smithi052.stdout:May 27 07:29:33 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-g[105245]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: 197: FAILED ceph_assert(cached_version > 0) 2024-05-27T07:29:33.388 INFO:journalctl@ceph.mds.g.smithi052.stdout:May 27 07:29:33 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-g[105245]: 2024-05-27T07:29:33.388 INFO:journalctl@ceph.mds.g.smithi052.stdout:May 27 07:29:33 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-g[105245]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:29:33.388 INFO:journalctl@ceph.mds.g.smithi052.stdout:May 27 07:29:33 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-g[105245]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x127) [0x7ff4764ccba6] 2024-05-27T07:29:33.388 INFO:journalctl@ceph.mds.g.smithi052.stdout:May 27 07:29:33 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-g[105245]: 2: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7ff4764ccdd1] 2024-05-27T07:29:33.388 INFO:journalctl@ceph.mds.g.smithi052.stdout:May 27 07:29:33 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-g[105245]: 3: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x55ce81f4d23d] 2024-05-27T07:29:33.388 INFO:journalctl@ceph.mds.g.smithi052.stdout:May 27 07:29:33 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-g[105245]: 4: (SnapRealm::build_snap_set() const+0x323) [0x55ce81f38971] 2024-05-27T07:29:33.388 INFO:journalctl@ceph.mds.g.smithi052.stdout:May 27 07:29:33 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-g[105245]: 5: (SnapRealm::check_cache() const+0x13e) [0x55ce81f36fa0] 2024-05-27T07:29:33.388 INFO:journalctl@ceph.mds.g.smithi052.stdout:May 27 07:29:33 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-g[105245]: 6: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x55ce81f37651] 2024-05-27T07:29:33.388 INFO:journalctl@ceph.mds.g.smithi052.stdout:May 27 07:29:33 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-g[105245]: 7: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x55ce81ecc347] 2024-05-27T07:29:33.388 INFO:journalctl@ceph.mds.g.smithi052.stdout:May 27 07:29:33 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-g[105245]: 8: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x55ce81bc887e] 2024-05-27T07:29:33.388 INFO:journalctl@ceph.mds.g.smithi052.stdout:May 27 07:29:33 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-g[105245]: 9: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x2702) [0x55ce81bcde94] 2024-05-27T07:29:33.389 INFO:journalctl@ceph.mds.g.smithi052.stdout:May 27 07:29:33 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-g[105245]: 10: (EUpdate::replay(MDSRank*)+0x63) [0x55ce81bd8969] 2024-05-27T07:29:33.389 INFO:journalctl@ceph.mds.g.smithi052.stdout:May 27 07:29:33 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-g[105245]: 11: (MDLog::_replay_thread()+0x1f38) [0x55ce81f7148c] 2024-05-27T07:29:33.389 INFO:journalctl@ceph.mds.g.smithi052.stdout:May 27 07:29:33 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-g[105245]: 12: (MDLog::ReplayThread::entry()+0x11) [0x55ce81f7a9cb] 2024-05-27T07:29:33.389 INFO:journalctl@ceph.mds.g.smithi052.stdout:May 27 07:29:33 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-g[105245]: 13: (Thread::entry_wrapper()+0x43) [0x7ff47649b72d] 2024-05-27T07:29:33.389 INFO:journalctl@ceph.mds.g.smithi052.stdout:May 27 07:29:33 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-g[105245]: 14: (Thread::_entry_func(void*)+0xd) [0x7ff47649b749] 2024-05-27T07:29:33.389 INFO:journalctl@ceph.mds.g.smithi052.stdout:May 27 07:29:33 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-g[105245]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7ff47503f1ca] 2024-05-27T07:29:33.389 INFO:journalctl@ceph.mds.g.smithi052.stdout:May 27 07:29:33 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-g[105245]: 16: clone() 2024-05-27T07:29:33.389 INFO:journalctl@ceph.mds.g.smithi052.stdout:May 27 07:29:33 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-g[105245]: 2024-05-27T07:29:33.389 INFO:journalctl@ceph.mds.g.smithi052.stdout:May 27 07:29:33 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-g[105245]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:29:33.389 INFO:journalctl@ceph.mds.g.smithi052.stdout:May 27 07:29:33 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-g[105245]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7ff475049d20] 2024-05-27T07:29:33.389 INFO:journalctl@ceph.mds.g.smithi052.stdout:May 27 07:29:33 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-g[105245]: 2: gsignal() 2024-05-27T07:29:33.389 INFO:journalctl@ceph.mds.g.smithi052.stdout:May 27 07:29:33 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-g[105245]: 3: abort() 2024-05-27T07:29:33.390 INFO:journalctl@ceph.mds.g.smithi052.stdout:May 27 07:29:33 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-g[105245]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x26e) [0x7ff4764ccced] 2024-05-27T07:29:33.390 INFO:journalctl@ceph.mds.d.smithi052.stdout:May 27 07:29:33 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-d[105221]: 10: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x5650d7c8b347] 2024-05-27T07:29:33.390 INFO:journalctl@ceph.mds.d.smithi052.stdout:May 27 07:29:33 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-d[105221]: 11: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x5650d798787e] 2024-05-27T07:29:33.390 INFO:journalctl@ceph.mds.d.smithi052.stdout:May 27 07:29:33 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-d[105221]: 12: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x2702) [0x5650d798ce94] 2024-05-27T07:29:33.390 INFO:journalctl@ceph.mds.d.smithi052.stdout:May 27 07:29:33 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-d[105221]: 13: (EUpdate::replay(MDSRank*)+0x63) [0x5650d7997969] 2024-05-27T07:29:33.390 INFO:journalctl@ceph.mds.d.smithi052.stdout:May 27 07:29:33 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-d[105221]: 14: (MDLog::_replay_thread()+0x1f38) [0x5650d7d3048c] 2024-05-27T07:29:33.390 INFO:journalctl@ceph.mds.d.smithi052.stdout:May 27 07:29:33 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-d[105221]: 15: (MDLog::ReplayThread::entry()+0x11) [0x5650d7d399cb] 2024-05-27T07:29:33.390 INFO:journalctl@ceph.mds.d.smithi052.stdout:May 27 07:29:33 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-d[105221]: 16: (Thread::entry_wrapper()+0x43) [0x7f113bfe072d] 2024-05-27T07:29:33.390 INFO:journalctl@ceph.mds.d.smithi052.stdout:May 27 07:29:33 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-d[105221]: 17: (Thread::_entry_func(void*)+0xd) [0x7f113bfe0749] 2024-05-27T07:29:33.390 INFO:journalctl@ceph.mds.d.smithi052.stdout:May 27 07:29:33 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-d[105221]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7f113ab841ca] 2024-05-27T07:29:33.391 INFO:journalctl@ceph.mds.d.smithi052.stdout:May 27 07:29:33 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-d[105221]: 19: clone() 2024-05-27T07:29:33.391 INFO:journalctl@ceph.mds.d.smithi052.stdout:May 27 07:29:33 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-d[105221]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-05-27T07:29:33.391 INFO:journalctl@ceph.mds.d.smithi052.stdout:May 27 07:29:33 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-d[105221]: 2024-05-27T07:29:33.391 INFO:journalctl@ceph.mds.d.smithi052.stdout:May 27 07:29:33 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-d[105221]: -1> 2024-05-27T07:29:33.171+0000 7f11297cc700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: In function 'std::set SnapClient::filter(const std::set&) const' thread 7f11297cc700 time 2024-05-27T07:29:33.169136+0000 2024-05-27T07:29:33.391 INFO:journalctl@ceph.mds.d.smithi052.stdout:May 27 07:29:33 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-d[105221]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: 197: FAILED ceph_assert(cached_version > 0) 2024-05-27T07:29:33.391 INFO:journalctl@ceph.mds.d.smithi052.stdout:May 27 07:29:33 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-d[105221]: 2024-05-27T07:29:33.391 INFO:journalctl@ceph.mds.d.smithi052.stdout:May 27 07:29:33 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-d[105221]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:29:33.391 INFO:journalctl@ceph.mds.d.smithi052.stdout:May 27 07:29:33 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-d[105221]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x127) [0x7f113c011ba6] 2024-05-27T07:29:33.391 INFO:journalctl@ceph.mds.d.smithi052.stdout:May 27 07:29:33 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-d[105221]: 2: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7f113c011dd1] 2024-05-27T07:29:33.391 INFO:journalctl@ceph.mds.d.smithi052.stdout:May 27 07:29:33 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-d[105221]: 3: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x5650d7d0c23d] 2024-05-27T07:29:33.391 INFO:journalctl@ceph.mds.d.smithi052.stdout:May 27 07:29:33 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-d[105221]: 4: (SnapRealm::build_snap_set() const+0x323) [0x5650d7cf7971] 2024-05-27T07:29:33.391 INFO:journalctl@ceph.mds.d.smithi052.stdout:May 27 07:29:33 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-d[105221]: 5: (SnapRealm::check_cache() const+0x13e) [0x5650d7cf5fa0] 2024-05-27T07:29:33.391 INFO:journalctl@ceph.mds.d.smithi052.stdout:May 27 07:29:33 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-d[105221]: 6: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x5650d7cf6651] 2024-05-27T07:29:33.392 INFO:journalctl@ceph.mds.d.smithi052.stdout:May 27 07:29:33 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-d[105221]: 7: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x5650d7c8b347] 2024-05-27T07:29:33.392 INFO:journalctl@ceph.mds.d.smithi052.stdout:May 27 07:29:33 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-d[105221]: 8: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x5650d798787e] 2024-05-27T07:29:33.392 INFO:journalctl@ceph.mds.g.smithi052.stdout:May 27 07:29:33 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-g[105245]: 5: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7ff4764ccdd1] 2024-05-27T07:29:33.392 INFO:journalctl@ceph.mds.g.smithi052.stdout:May 27 07:29:33 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-g[105245]: 6: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x55ce81f4d23d] 2024-05-27T07:29:33.392 INFO:journalctl@ceph.mds.g.smithi052.stdout:May 27 07:29:33 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-g[105245]: 7: (SnapRealm::build_snap_set() const+0x323) [0x55ce81f38971] 2024-05-27T07:29:33.392 INFO:journalctl@ceph.mds.g.smithi052.stdout:May 27 07:29:33 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-g[105245]: 8: (SnapRealm::check_cache() const+0x13e) [0x55ce81f36fa0] 2024-05-27T07:29:33.392 INFO:journalctl@ceph.mds.g.smithi052.stdout:May 27 07:29:33 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-g[105245]: 9: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x55ce81f37651] 2024-05-27T07:29:33.392 INFO:journalctl@ceph.mds.g.smithi052.stdout:May 27 07:29:33 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-g[105245]: 10: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x55ce81ecc347] 2024-05-27T07:29:33.392 INFO:journalctl@ceph.mds.g.smithi052.stdout:May 27 07:29:33 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-g[105245]: 11: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x55ce81bc887e] 2024-05-27T07:29:33.393 INFO:journalctl@ceph.mds.g.smithi052.stdout:May 27 07:29:33 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-g[105245]: 12: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x2702) [0x55ce81bcde94] 2024-05-27T07:29:33.393 INFO:journalctl@ceph.mds.g.smithi052.stdout:May 27 07:29:33 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-g[105245]: 13: (EUpdate::replay(MDSRank*)+0x63) [0x55ce81bd8969] 2024-05-27T07:29:33.393 INFO:journalctl@ceph.mds.g.smithi052.stdout:May 27 07:29:33 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-g[105245]: 14: (MDLog::_replay_thread()+0x1f38) [0x55ce81f7148c] 2024-05-27T07:29:33.393 INFO:journalctl@ceph.mds.g.smithi052.stdout:May 27 07:29:33 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-g[105245]: 15: (MDLog::ReplayThread::entry()+0x11) [0x55ce81f7a9cb] 2024-05-27T07:29:33.393 INFO:journalctl@ceph.mds.g.smithi052.stdout:May 27 07:29:33 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-g[105245]: 16: (Thread::entry_wrapper()+0x43) [0x7ff47649b72d] 2024-05-27T07:29:33.393 INFO:journalctl@ceph.mds.g.smithi052.stdout:May 27 07:29:33 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-g[105245]: 17: (Thread::_entry_func(void*)+0xd) [0x7ff47649b749] 2024-05-27T07:29:33.393 INFO:journalctl@ceph.mds.g.smithi052.stdout:May 27 07:29:33 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-g[105245]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7ff47503f1ca] 2024-05-27T07:29:33.393 INFO:journalctl@ceph.mds.g.smithi052.stdout:May 27 07:29:33 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-g[105245]: 19: clone() 2024-05-27T07:29:33.393 INFO:journalctl@ceph.mds.g.smithi052.stdout:May 27 07:29:33 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-g[105245]: 2024-05-27T07:29:33.225+0000 7ff463c87700 -1 *** Caught signal (Aborted) ** 2024-05-27T07:29:33.393 INFO:journalctl@ceph.mds.g.smithi052.stdout:May 27 07:29:33 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-g[105245]: in thread 7ff463c87700 thread_name:md_log_replay 2024-05-27T07:29:33.393 INFO:journalctl@ceph.mds.g.smithi052.stdout:May 27 07:29:33 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-g[105245]: 2024-05-27T07:29:33.393 INFO:journalctl@ceph.mds.g.smithi052.stdout:May 27 07:29:33 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-g[105245]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:29:33.393 INFO:journalctl@ceph.mds.g.smithi052.stdout:May 27 07:29:33 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-g[105245]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7ff475049d20] 2024-05-27T07:29:33.394 INFO:journalctl@ceph.mds.g.smithi052.stdout:May 27 07:29:33 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-g[105245]: 2: gsignal() 2024-05-27T07:29:33.394 INFO:journalctl@ceph.mds.g.smithi052.stdout:May 27 07:29:33 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-g[105245]: 3: abort() 2024-05-27T07:29:33.394 INFO:journalctl@ceph.mds.g.smithi052.stdout:May 27 07:29:33 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-g[105245]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x26e) [0x7ff4764ccced] 2024-05-27T07:29:33.394 INFO:journalctl@ceph.mds.g.smithi052.stdout:May 27 07:29:33 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-g[105245]: 5: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7ff4764ccdd1] 2024-05-27T07:29:33.394 INFO:journalctl@ceph.mds.g.smithi052.stdout:May 27 07:29:33 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-g[105245]: 6: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x55ce81f4d23d] 2024-05-27T07:29:33.394 INFO:journalctl@ceph.mds.g.smithi052.stdout:May 27 07:29:33 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-g[105245]: 7: (SnapRealm::build_snap_set() const+0x323) [0x55ce81f38971] 2024-05-27T07:29:33.394 INFO:journalctl@ceph.mds.g.smithi052.stdout:May 27 07:29:33 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-g[105245]: 8: (SnapRealm::check_cache() const+0x13e) [0x55ce81f36fa0] 2024-05-27T07:29:33.394 INFO:journalctl@ceph.mds.g.smithi052.stdout:May 27 07:29:33 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-g[105245]: 9: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x55ce81f37651] 2024-05-27T07:29:33.394 INFO:journalctl@ceph.mds.d.smithi052.stdout:May 27 07:29:33 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-d[105221]: 9: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x2702) [0x5650d798ce94] 2024-05-27T07:29:33.395 INFO:journalctl@ceph.mds.d.smithi052.stdout:May 27 07:29:33 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-d[105221]: 10: (EUpdate::replay(MDSRank*)+0x63) [0x5650d7997969] 2024-05-27T07:29:33.395 INFO:journalctl@ceph.mds.d.smithi052.stdout:May 27 07:29:33 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-d[105221]: 11: (MDLog::_replay_thread()+0x1f38) [0x5650d7d3048c] 2024-05-27T07:29:33.395 INFO:journalctl@ceph.mds.d.smithi052.stdout:May 27 07:29:33 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-d[105221]: 12: (MDLog::ReplayThread::entry()+0x11) [0x5650d7d399cb] 2024-05-27T07:29:33.395 INFO:journalctl@ceph.mds.d.smithi052.stdout:May 27 07:29:33 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-d[105221]: 13: (Thread::entry_wrapper()+0x43) [0x7f113bfe072d] 2024-05-27T07:29:33.395 INFO:journalctl@ceph.mds.d.smithi052.stdout:May 27 07:29:33 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-d[105221]: 14: (Thread::_entry_func(void*)+0xd) [0x7f113bfe0749] 2024-05-27T07:29:33.395 INFO:journalctl@ceph.mds.d.smithi052.stdout:May 27 07:29:33 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-d[105221]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7f113ab841ca] 2024-05-27T07:29:33.395 INFO:journalctl@ceph.mds.d.smithi052.stdout:May 27 07:29:33 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-d[105221]: 16: clone() 2024-05-27T07:29:33.395 INFO:journalctl@ceph.mds.d.smithi052.stdout:May 27 07:29:33 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-d[105221]: 2024-05-27T07:29:33.395 INFO:journalctl@ceph.mds.d.smithi052.stdout:May 27 07:29:33 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-d[105221]: 0> 2024-05-27T07:29:33.174+0000 7f11297cc700 -1 *** Caught signal (Aborted) ** 2024-05-27T07:29:33.395 INFO:journalctl@ceph.mds.d.smithi052.stdout:May 27 07:29:33 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-d[105221]: in thread 7f11297cc700 thread_name:md_log_replay 2024-05-27T07:29:33.395 INFO:journalctl@ceph.mds.d.smithi052.stdout:May 27 07:29:33 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-d[105221]: 2024-05-27T07:29:33.395 INFO:journalctl@ceph.mds.d.smithi052.stdout:May 27 07:29:33 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-d[105221]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:29:33.396 INFO:journalctl@ceph.mds.d.smithi052.stdout:May 27 07:29:33 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-d[105221]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f113ab8ed20] 2024-05-27T07:29:33.396 INFO:journalctl@ceph.mds.d.smithi052.stdout:May 27 07:29:33 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-d[105221]: 2: gsignal() 2024-05-27T07:29:33.396 INFO:journalctl@ceph.mds.d.smithi052.stdout:May 27 07:29:33 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-d[105221]: 3: abort() 2024-05-27T07:29:33.396 INFO:journalctl@ceph.mds.d.smithi052.stdout:May 27 07:29:33 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-d[105221]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x26e) [0x7f113c011ced] 2024-05-27T07:29:33.396 INFO:journalctl@ceph.mds.d.smithi052.stdout:May 27 07:29:33 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-d[105221]: 5: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7f113c011dd1] 2024-05-27T07:29:33.396 INFO:journalctl@ceph.mds.d.smithi052.stdout:May 27 07:29:33 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-d[105221]: 6: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x5650d7d0c23d] 2024-05-27T07:29:33.396 INFO:journalctl@ceph.mds.d.smithi052.stdout:May 27 07:29:33 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-d[105221]: 7: (SnapRealm::build_snap_set() const+0x323) [0x5650d7cf7971] 2024-05-27T07:29:33.396 INFO:journalctl@ceph.mds.d.smithi052.stdout:May 27 07:29:33 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-d[105221]: 8: (SnapRealm::check_cache() const+0x13e) [0x5650d7cf5fa0] 2024-05-27T07:29:33.396 INFO:journalctl@ceph.mds.d.smithi052.stdout:May 27 07:29:33 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-d[105221]: 9: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x5650d7cf6651] 2024-05-27T07:29:33.396 INFO:journalctl@ceph.mds.d.smithi052.stdout:May 27 07:29:33 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-d[105221]: 10: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x5650d7c8b347] 2024-05-27T07:29:33.396 INFO:journalctl@ceph.mds.d.smithi052.stdout:May 27 07:29:33 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-d[105221]: 11: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x5650d798787e] 2024-05-27T07:29:33.396 INFO:journalctl@ceph.mds.d.smithi052.stdout:May 27 07:29:33 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-d[105221]: 12: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x2702) [0x5650d798ce94] 2024-05-27T07:29:33.396 INFO:journalctl@ceph.mds.d.smithi052.stdout:May 27 07:29:33 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-d[105221]: 13: (EUpdate::replay(MDSRank*)+0x63) [0x5650d7997969] 2024-05-27T07:29:33.396 INFO:journalctl@ceph.mds.d.smithi052.stdout:May 27 07:29:33 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-d[105221]: 14: (MDLog::_replay_thread()+0x1f38) [0x5650d7d3048c] 2024-05-27T07:29:33.397 INFO:journalctl@ceph.mds.d.smithi052.stdout:May 27 07:29:33 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-d[105221]: 15: (MDLog::ReplayThread::entry()+0x11) [0x5650d7d399cb] 2024-05-27T07:29:33.397 INFO:journalctl@ceph.mds.g.smithi052.stdout:May 27 07:29:33 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-g[105245]: 10: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x55ce81ecc347] 2024-05-27T07:29:33.397 INFO:journalctl@ceph.mds.g.smithi052.stdout:May 27 07:29:33 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-g[105245]: 11: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x55ce81bc887e] 2024-05-27T07:29:33.397 INFO:journalctl@ceph.mds.g.smithi052.stdout:May 27 07:29:33 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-g[105245]: 12: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x2702) [0x55ce81bcde94] 2024-05-27T07:29:33.397 INFO:journalctl@ceph.mds.g.smithi052.stdout:May 27 07:29:33 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-g[105245]: 13: (EUpdate::replay(MDSRank*)+0x63) [0x55ce81bd8969] 2024-05-27T07:29:33.397 INFO:journalctl@ceph.mds.g.smithi052.stdout:May 27 07:29:33 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-g[105245]: 14: (MDLog::_replay_thread()+0x1f38) [0x55ce81f7148c] 2024-05-27T07:29:33.397 INFO:journalctl@ceph.mds.g.smithi052.stdout:May 27 07:29:33 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-g[105245]: 15: (MDLog::ReplayThread::entry()+0x11) [0x55ce81f7a9cb] 2024-05-27T07:29:33.397 INFO:journalctl@ceph.mds.g.smithi052.stdout:May 27 07:29:33 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-g[105245]: 16: (Thread::entry_wrapper()+0x43) [0x7ff47649b72d] 2024-05-27T07:29:33.398 INFO:journalctl@ceph.mds.g.smithi052.stdout:May 27 07:29:33 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-g[105245]: 17: (Thread::_entry_func(void*)+0xd) [0x7ff47649b749] 2024-05-27T07:29:33.398 INFO:journalctl@ceph.mds.g.smithi052.stdout:May 27 07:29:33 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-g[105245]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7ff47503f1ca] 2024-05-27T07:29:33.398 INFO:journalctl@ceph.mds.g.smithi052.stdout:May 27 07:29:33 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-g[105245]: 19: clone() 2024-05-27T07:29:33.398 INFO:journalctl@ceph.mds.g.smithi052.stdout:May 27 07:29:33 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-g[105245]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-05-27T07:29:33.398 INFO:journalctl@ceph.mds.g.smithi052.stdout:May 27 07:29:33 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-g[105245]: 2024-05-27T07:29:33.398 INFO:journalctl@ceph.mds.g.smithi052.stdout:May 27 07:29:33 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-g[105245]: -1> 2024-05-27T07:29:33.222+0000 7ff463c87700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: In function 'std::set SnapClient::filter(const std::set&) const' thread 7ff463c87700 time 2024-05-27T07:29:33.220576+0000 2024-05-27T07:29:33.398 INFO:journalctl@ceph.mds.g.smithi052.stdout:May 27 07:29:33 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-g[105245]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: 197: FAILED ceph_assert(cached_version > 0) 2024-05-27T07:29:33.398 INFO:journalctl@ceph.mds.g.smithi052.stdout:May 27 07:29:33 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-g[105245]: 2024-05-27T07:29:33.398 INFO:journalctl@ceph.mds.g.smithi052.stdout:May 27 07:29:33 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-g[105245]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:29:33.398 INFO:journalctl@ceph.mds.g.smithi052.stdout:May 27 07:29:33 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-g[105245]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x127) [0x7ff4764ccba6] 2024-05-27T07:29:33.398 INFO:journalctl@ceph.mds.g.smithi052.stdout:May 27 07:29:33 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-g[105245]: 2: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7ff4764ccdd1] 2024-05-27T07:29:33.398 INFO:journalctl@ceph.mds.g.smithi052.stdout:May 27 07:29:33 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-g[105245]: 3: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x55ce81f4d23d] 2024-05-27T07:29:33.399 INFO:journalctl@ceph.mds.g.smithi052.stdout:May 27 07:29:33 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-g[105245]: 4: (SnapRealm::build_snap_set() const+0x323) [0x55ce81f38971] 2024-05-27T07:29:33.399 INFO:journalctl@ceph.mds.g.smithi052.stdout:May 27 07:29:33 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-g[105245]: 5: (SnapRealm::check_cache() const+0x13e) [0x55ce81f36fa0] 2024-05-27T07:29:33.399 INFO:journalctl@ceph.mds.g.smithi052.stdout:May 27 07:29:33 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-g[105245]: 6: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x55ce81f37651] 2024-05-27T07:29:33.399 INFO:journalctl@ceph.mds.g.smithi052.stdout:May 27 07:29:33 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-g[105245]: 7: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x55ce81ecc347] 2024-05-27T07:29:33.399 INFO:journalctl@ceph.mds.g.smithi052.stdout:May 27 07:29:33 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-g[105245]: 8: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x55ce81bc887e] 2024-05-27T07:29:33.399 INFO:journalctl@ceph.mds.d.smithi052.stdout:May 27 07:29:33 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-d[105221]: 16: (Thread::entry_wrapper()+0x43) [0x7f113bfe072d] 2024-05-27T07:29:33.399 INFO:journalctl@ceph.mds.d.smithi052.stdout:May 27 07:29:33 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-d[105221]: 17: (Thread::_entry_func(void*)+0xd) [0x7f113bfe0749] 2024-05-27T07:29:33.399 INFO:journalctl@ceph.mds.d.smithi052.stdout:May 27 07:29:33 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-d[105221]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7f113ab841ca] 2024-05-27T07:29:33.399 INFO:journalctl@ceph.mds.d.smithi052.stdout:May 27 07:29:33 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-d[105221]: 19: clone() 2024-05-27T07:29:33.400 INFO:journalctl@ceph.mds.d.smithi052.stdout:May 27 07:29:33 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-d[105221]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-05-27T07:29:33.400 INFO:journalctl@ceph.mds.d.smithi052.stdout:May 27 07:29:33 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-d[105221]: 2024-05-27T07:29:33.400 INFO:journalctl@ceph.mds.d.smithi052.stdout:May 27 07:29:33 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-d[105221]: -1> 2024-05-27T07:29:33.171+0000 7f11297cc700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: In function 'std::set SnapClient::filter(const std::set&) const' thread 7f11297cc700 time 2024-05-27T07:29:33.169136+0000 2024-05-27T07:29:33.400 INFO:journalctl@ceph.mds.d.smithi052.stdout:May 27 07:29:33 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-d[105221]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: 197: FAILED ceph_assert(cached_version > 0) 2024-05-27T07:29:33.400 INFO:journalctl@ceph.mds.d.smithi052.stdout:May 27 07:29:33 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-d[105221]: 2024-05-27T07:29:33.400 INFO:journalctl@ceph.mds.d.smithi052.stdout:May 27 07:29:33 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-d[105221]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:29:33.400 INFO:journalctl@ceph.mds.d.smithi052.stdout:May 27 07:29:33 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-d[105221]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x127) [0x7f113c011ba6] 2024-05-27T07:29:33.400 INFO:journalctl@ceph.mds.d.smithi052.stdout:May 27 07:29:33 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-d[105221]: 2: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7f113c011dd1] 2024-05-27T07:29:33.400 INFO:journalctl@ceph.mds.d.smithi052.stdout:May 27 07:29:33 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-d[105221]: 3: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x5650d7d0c23d] 2024-05-27T07:29:33.400 INFO:journalctl@ceph.mds.d.smithi052.stdout:May 27 07:29:33 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-d[105221]: 4: (SnapRealm::build_snap_set() const+0x323) [0x5650d7cf7971] 2024-05-27T07:29:33.400 INFO:journalctl@ceph.mds.d.smithi052.stdout:May 27 07:29:33 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-d[105221]: 5: (SnapRealm::check_cache() const+0x13e) [0x5650d7cf5fa0] 2024-05-27T07:29:33.400 INFO:journalctl@ceph.mds.d.smithi052.stdout:May 27 07:29:33 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-d[105221]: 6: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x5650d7cf6651] 2024-05-27T07:29:33.401 INFO:journalctl@ceph.mds.d.smithi052.stdout:May 27 07:29:33 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-d[105221]: 7: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x5650d7c8b347] 2024-05-27T07:29:33.401 INFO:journalctl@ceph.mds.d.smithi052.stdout:May 27 07:29:33 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-d[105221]: 8: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x5650d798787e] 2024-05-27T07:29:33.401 INFO:journalctl@ceph.mds.d.smithi052.stdout:May 27 07:29:33 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-d[105221]: 9: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x2702) [0x5650d798ce94] 2024-05-27T07:29:33.401 INFO:journalctl@ceph.mds.d.smithi052.stdout:May 27 07:29:33 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-d[105221]: 10: (EUpdate::replay(MDSRank*)+0x63) [0x5650d7997969] 2024-05-27T07:29:33.401 INFO:journalctl@ceph.mds.d.smithi052.stdout:May 27 07:29:33 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-d[105221]: 11: (MDLog::_replay_thread()+0x1f38) [0x5650d7d3048c] 2024-05-27T07:29:33.401 INFO:journalctl@ceph.mds.d.smithi052.stdout:May 27 07:29:33 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-d[105221]: 12: (MDLog::ReplayThread::entry()+0x11) [0x5650d7d399cb] 2024-05-27T07:29:33.401 INFO:journalctl@ceph.mds.d.smithi052.stdout:May 27 07:29:33 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-d[105221]: 13: (Thread::entry_wrapper()+0x43) [0x7f113bfe072d] 2024-05-27T07:29:33.401 INFO:journalctl@ceph.mds.d.smithi052.stdout:May 27 07:29:33 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-d[105221]: 14: (Thread::_entry_func(void*)+0xd) [0x7f113bfe0749] 2024-05-27T07:29:33.401 INFO:journalctl@ceph.mds.g.smithi052.stdout:May 27 07:29:33 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-g[105245]: 9: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x2702) [0x55ce81bcde94] 2024-05-27T07:29:33.401 INFO:journalctl@ceph.mds.g.smithi052.stdout:May 27 07:29:33 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-g[105245]: 10: (EUpdate::replay(MDSRank*)+0x63) [0x55ce81bd8969] 2024-05-27T07:29:33.401 INFO:journalctl@ceph.mds.g.smithi052.stdout:May 27 07:29:33 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-g[105245]: 11: (MDLog::_replay_thread()+0x1f38) [0x55ce81f7148c] 2024-05-27T07:29:33.401 INFO:journalctl@ceph.mds.g.smithi052.stdout:May 27 07:29:33 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-g[105245]: 12: (MDLog::ReplayThread::entry()+0x11) [0x55ce81f7a9cb] 2024-05-27T07:29:33.402 INFO:journalctl@ceph.mds.g.smithi052.stdout:May 27 07:29:33 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-g[105245]: 13: (Thread::entry_wrapper()+0x43) [0x7ff47649b72d] 2024-05-27T07:29:33.402 INFO:journalctl@ceph.mds.g.smithi052.stdout:May 27 07:29:33 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-g[105245]: 14: (Thread::_entry_func(void*)+0xd) [0x7ff47649b749] 2024-05-27T07:29:33.402 INFO:journalctl@ceph.mds.g.smithi052.stdout:May 27 07:29:33 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-g[105245]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7ff47503f1ca] 2024-05-27T07:29:33.402 INFO:journalctl@ceph.mds.g.smithi052.stdout:May 27 07:29:33 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-g[105245]: 16: clone() 2024-05-27T07:29:33.402 INFO:journalctl@ceph.mds.g.smithi052.stdout:May 27 07:29:33 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-g[105245]: 2024-05-27T07:29:33.402 INFO:journalctl@ceph.mds.g.smithi052.stdout:May 27 07:29:33 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-g[105245]: 0> 2024-05-27T07:29:33.225+0000 7ff463c87700 -1 *** Caught signal (Aborted) ** 2024-05-27T07:29:33.402 INFO:journalctl@ceph.mds.g.smithi052.stdout:May 27 07:29:33 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-g[105245]: in thread 7ff463c87700 thread_name:md_log_replay 2024-05-27T07:29:33.402 INFO:journalctl@ceph.mds.g.smithi052.stdout:May 27 07:29:33 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-g[105245]: 2024-05-27T07:29:33.402 INFO:journalctl@ceph.mds.g.smithi052.stdout:May 27 07:29:33 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-g[105245]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:29:33.402 INFO:journalctl@ceph.mds.g.smithi052.stdout:May 27 07:29:33 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-g[105245]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7ff475049d20] 2024-05-27T07:29:33.402 INFO:journalctl@ceph.mds.g.smithi052.stdout:May 27 07:29:33 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-g[105245]: 2: gsignal() 2024-05-27T07:29:33.402 INFO:journalctl@ceph.mds.g.smithi052.stdout:May 27 07:29:33 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-g[105245]: 3: abort() 2024-05-27T07:29:33.403 INFO:journalctl@ceph.mds.g.smithi052.stdout:May 27 07:29:33 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-g[105245]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x26e) [0x7ff4764ccced] 2024-05-27T07:29:33.403 INFO:journalctl@ceph.mds.g.smithi052.stdout:May 27 07:29:33 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-g[105245]: 5: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7ff4764ccdd1] 2024-05-27T07:29:33.403 INFO:journalctl@ceph.mds.g.smithi052.stdout:May 27 07:29:33 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-g[105245]: 6: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x55ce81f4d23d] 2024-05-27T07:29:33.403 INFO:journalctl@ceph.mds.g.smithi052.stdout:May 27 07:29:33 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-g[105245]: 7: (SnapRealm::build_snap_set() const+0x323) [0x55ce81f38971] 2024-05-27T07:29:33.403 INFO:journalctl@ceph.mds.g.smithi052.stdout:May 27 07:29:33 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-g[105245]: 8: (SnapRealm::check_cache() const+0x13e) [0x55ce81f36fa0] 2024-05-27T07:29:33.403 INFO:journalctl@ceph.mds.g.smithi052.stdout:May 27 07:29:33 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-g[105245]: 9: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x55ce81f37651] 2024-05-27T07:29:33.403 INFO:journalctl@ceph.mds.g.smithi052.stdout:May 27 07:29:33 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-g[105245]: 10: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x55ce81ecc347] 2024-05-27T07:29:33.403 INFO:journalctl@ceph.mds.g.smithi052.stdout:May 27 07:29:33 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-g[105245]: 11: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x55ce81bc887e] 2024-05-27T07:29:33.403 INFO:journalctl@ceph.mds.g.smithi052.stdout:May 27 07:29:33 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-g[105245]: 12: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x2702) [0x55ce81bcde94] 2024-05-27T07:29:33.403 INFO:journalctl@ceph.mds.g.smithi052.stdout:May 27 07:29:33 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-g[105245]: 13: (EUpdate::replay(MDSRank*)+0x63) [0x55ce81bd8969] 2024-05-27T07:29:33.403 INFO:journalctl@ceph.mds.g.smithi052.stdout:May 27 07:29:33 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-g[105245]: 14: (MDLog::_replay_thread()+0x1f38) [0x55ce81f7148c] 2024-05-27T07:29:33.403 INFO:journalctl@ceph.mds.g.smithi052.stdout:May 27 07:29:33 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-g[105245]: 15: (MDLog::ReplayThread::entry()+0x11) [0x55ce81f7a9cb] 2024-05-27T07:29:33.404 INFO:journalctl@ceph.mds.d.smithi052.stdout:May 27 07:29:33 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-d[105221]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7f113ab841ca] 2024-05-27T07:29:33.404 INFO:journalctl@ceph.mds.d.smithi052.stdout:May 27 07:29:33 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-d[105221]: 16: clone() 2024-05-27T07:29:33.404 INFO:journalctl@ceph.mds.d.smithi052.stdout:May 27 07:29:33 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-d[105221]: 2024-05-27T07:29:33.404 INFO:journalctl@ceph.mds.d.smithi052.stdout:May 27 07:29:33 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-d[105221]: 0> 2024-05-27T07:29:33.174+0000 7f11297cc700 -1 *** Caught signal (Aborted) ** 2024-05-27T07:29:33.404 INFO:journalctl@ceph.mds.d.smithi052.stdout:May 27 07:29:33 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-d[105221]: in thread 7f11297cc700 thread_name:md_log_replay 2024-05-27T07:29:33.404 INFO:journalctl@ceph.mds.d.smithi052.stdout:May 27 07:29:33 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-d[105221]: 2024-05-27T07:29:33.404 INFO:journalctl@ceph.mds.d.smithi052.stdout:May 27 07:29:33 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-d[105221]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:29:33.404 INFO:journalctl@ceph.mds.d.smithi052.stdout:May 27 07:29:33 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-d[105221]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f113ab8ed20] 2024-05-27T07:29:33.404 INFO:journalctl@ceph.mds.d.smithi052.stdout:May 27 07:29:33 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-d[105221]: 2: gsignal() 2024-05-27T07:29:33.404 INFO:journalctl@ceph.mds.d.smithi052.stdout:May 27 07:29:33 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-d[105221]: 3: abort() 2024-05-27T07:29:33.404 INFO:journalctl@ceph.mds.d.smithi052.stdout:May 27 07:29:33 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-d[105221]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x26e) [0x7f113c011ced] 2024-05-27T07:29:33.405 INFO:journalctl@ceph.mds.d.smithi052.stdout:May 27 07:29:33 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-d[105221]: 5: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7f113c011dd1] 2024-05-27T07:29:33.405 INFO:journalctl@ceph.mds.d.smithi052.stdout:May 27 07:29:33 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-d[105221]: 6: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x5650d7d0c23d] 2024-05-27T07:29:33.405 INFO:journalctl@ceph.mds.d.smithi052.stdout:May 27 07:29:33 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-d[105221]: 7: (SnapRealm::build_snap_set() const+0x323) [0x5650d7cf7971] 2024-05-27T07:29:33.405 INFO:journalctl@ceph.mds.d.smithi052.stdout:May 27 07:29:33 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-d[105221]: 8: (SnapRealm::check_cache() const+0x13e) [0x5650d7cf5fa0] 2024-05-27T07:29:33.405 INFO:journalctl@ceph.mds.d.smithi052.stdout:May 27 07:29:33 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-d[105221]: 9: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x5650d7cf6651] 2024-05-27T07:29:33.405 INFO:journalctl@ceph.mds.d.smithi052.stdout:May 27 07:29:33 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-d[105221]: 10: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x5650d7c8b347] 2024-05-27T07:29:33.405 INFO:journalctl@ceph.mds.d.smithi052.stdout:May 27 07:29:33 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-d[105221]: 11: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x5650d798787e] 2024-05-27T07:29:33.405 INFO:journalctl@ceph.mds.d.smithi052.stdout:May 27 07:29:33 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-d[105221]: 12: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x2702) [0x5650d798ce94] 2024-05-27T07:29:33.405 INFO:journalctl@ceph.mds.d.smithi052.stdout:May 27 07:29:33 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-d[105221]: 13: (EUpdate::replay(MDSRank*)+0x63) [0x5650d7997969] 2024-05-27T07:29:33.405 INFO:journalctl@ceph.mds.d.smithi052.stdout:May 27 07:29:33 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-d[105221]: 14: (MDLog::_replay_thread()+0x1f38) [0x5650d7d3048c] 2024-05-27T07:29:33.405 INFO:journalctl@ceph.mds.d.smithi052.stdout:May 27 07:29:33 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-d[105221]: 15: (MDLog::ReplayThread::entry()+0x11) [0x5650d7d399cb] 2024-05-27T07:29:33.405 INFO:journalctl@ceph.mds.d.smithi052.stdout:May 27 07:29:33 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-d[105221]: 16: (Thread::entry_wrapper()+0x43) [0x7f113bfe072d] 2024-05-27T07:29:33.405 INFO:journalctl@ceph.mds.d.smithi052.stdout:May 27 07:29:33 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-d[105221]: 17: (Thread::_entry_func(void*)+0xd) [0x7f113bfe0749] 2024-05-27T07:29:33.405 INFO:journalctl@ceph.mds.d.smithi052.stdout:May 27 07:29:33 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-d[105221]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7f113ab841ca] 2024-05-27T07:29:33.406 INFO:journalctl@ceph.mds.d.smithi052.stdout:May 27 07:29:33 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-d[105221]: 19: clone() 2024-05-27T07:29:33.406 INFO:journalctl@ceph.mds.d.smithi052.stdout:May 27 07:29:33 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-d[105221]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-05-27T07:29:33.406 INFO:journalctl@ceph.mds.d.smithi052.stdout:May 27 07:29:33 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-d[105221]: 2024-05-27T07:29:33.406 INFO:journalctl@ceph.mds.g.smithi052.stdout:May 27 07:29:33 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-g[105245]: 16: (Thread::entry_wrapper()+0x43) [0x7ff47649b72d] 2024-05-27T07:29:33.406 INFO:journalctl@ceph.mds.g.smithi052.stdout:May 27 07:29:33 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-g[105245]: 17: (Thread::_entry_func(void*)+0xd) [0x7ff47649b749] 2024-05-27T07:29:33.406 INFO:journalctl@ceph.mds.g.smithi052.stdout:May 27 07:29:33 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-g[105245]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7ff47503f1ca] 2024-05-27T07:29:33.406 INFO:journalctl@ceph.mds.g.smithi052.stdout:May 27 07:29:33 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-g[105245]: 19: clone() 2024-05-27T07:29:33.406 INFO:journalctl@ceph.mds.g.smithi052.stdout:May 27 07:29:33 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-g[105245]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-05-27T07:29:33.407 INFO:journalctl@ceph.mds.g.smithi052.stdout:May 27 07:29:33 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-g[105245]: 2024-05-27T07:29:33.407 INFO:journalctl@ceph.mds.g.smithi052.stdout:May 27 07:29:33 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-g[105245]: -1> 2024-05-27T07:29:33.222+0000 7ff463c87700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: In function 'std::set SnapClient::filter(const std::set&) const' thread 7ff463c87700 time 2024-05-27T07:29:33.220576+0000 2024-05-27T07:29:33.407 INFO:journalctl@ceph.mds.g.smithi052.stdout:May 27 07:29:33 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-g[105245]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: 197: FAILED ceph_assert(cached_version > 0) 2024-05-27T07:29:33.407 INFO:journalctl@ceph.mds.g.smithi052.stdout:May 27 07:29:33 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-g[105245]: 2024-05-27T07:29:33.407 INFO:journalctl@ceph.mds.g.smithi052.stdout:May 27 07:29:33 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-g[105245]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:29:33.407 INFO:journalctl@ceph.mds.g.smithi052.stdout:May 27 07:29:33 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-g[105245]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x127) [0x7ff4764ccba6] 2024-05-27T07:29:33.407 INFO:journalctl@ceph.mds.g.smithi052.stdout:May 27 07:29:33 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-g[105245]: 2: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7ff4764ccdd1] 2024-05-27T07:29:33.407 INFO:journalctl@ceph.mds.g.smithi052.stdout:May 27 07:29:33 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-g[105245]: 3: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x55ce81f4d23d] 2024-05-27T07:29:33.407 INFO:journalctl@ceph.mds.g.smithi052.stdout:May 27 07:29:33 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-g[105245]: 4: (SnapRealm::build_snap_set() const+0x323) [0x55ce81f38971] 2024-05-27T07:29:33.407 INFO:journalctl@ceph.mds.g.smithi052.stdout:May 27 07:29:33 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-g[105245]: 5: (SnapRealm::check_cache() const+0x13e) [0x55ce81f36fa0] 2024-05-27T07:29:33.407 INFO:journalctl@ceph.mds.g.smithi052.stdout:May 27 07:29:33 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-g[105245]: 6: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x55ce81f37651] 2024-05-27T07:29:33.407 INFO:journalctl@ceph.mds.g.smithi052.stdout:May 27 07:29:33 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-g[105245]: 7: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x55ce81ecc347] 2024-05-27T07:29:33.407 INFO:journalctl@ceph.mds.g.smithi052.stdout:May 27 07:29:33 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-g[105245]: 8: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x55ce81bc887e] 2024-05-27T07:29:33.408 INFO:journalctl@ceph.mds.g.smithi052.stdout:May 27 07:29:33 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-g[105245]: 9: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x2702) [0x55ce81bcde94] 2024-05-27T07:29:33.408 INFO:journalctl@ceph.mds.g.smithi052.stdout:May 27 07:29:33 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-g[105245]: 10: (EUpdate::replay(MDSRank*)+0x63) [0x55ce81bd8969] 2024-05-27T07:29:33.408 INFO:journalctl@ceph.mds.g.smithi052.stdout:May 27 07:29:33 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-g[105245]: 11: (MDLog::_replay_thread()+0x1f38) [0x55ce81f7148c] 2024-05-27T07:29:33.408 INFO:journalctl@ceph.mds.g.smithi052.stdout:May 27 07:29:33 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-g[105245]: 12: (MDLog::ReplayThread::entry()+0x11) [0x55ce81f7a9cb] 2024-05-27T07:29:33.408 INFO:journalctl@ceph.mds.g.smithi052.stdout:May 27 07:29:33 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-g[105245]: 13: (Thread::entry_wrapper()+0x43) [0x7ff47649b72d] 2024-05-27T07:29:33.408 INFO:journalctl@ceph.mds.g.smithi052.stdout:May 27 07:29:33 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-g[105245]: 14: (Thread::_entry_func(void*)+0xd) [0x7ff47649b749] 2024-05-27T07:29:33.408 INFO:journalctl@ceph.mds.g.smithi052.stdout:May 27 07:29:33 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-g[105245]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7ff47503f1ca] 2024-05-27T07:29:33.408 INFO:journalctl@ceph.mds.g.smithi052.stdout:May 27 07:29:33 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-g[105245]: 16: clone() 2024-05-27T07:29:33.408 INFO:journalctl@ceph.mds.g.smithi052.stdout:May 27 07:29:33 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-g[105245]: 2024-05-27T07:29:33.408 INFO:journalctl@ceph.mds.g.smithi052.stdout:May 27 07:29:33 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-g[105245]: 0> 2024-05-27T07:29:33.225+0000 7ff463c87700 -1 *** Caught signal (Aborted) ** 2024-05-27T07:29:33.409 INFO:journalctl@ceph.mds.g.smithi052.stdout:May 27 07:29:33 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-g[105245]: in thread 7ff463c87700 thread_name:md_log_replay 2024-05-27T07:29:33.409 INFO:journalctl@ceph.mds.g.smithi052.stdout:May 27 07:29:33 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-g[105245]: 2024-05-27T07:29:33.409 INFO:journalctl@ceph.mds.g.smithi052.stdout:May 27 07:29:33 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-g[105245]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:29:33.409 INFO:journalctl@ceph.mds.g.smithi052.stdout:May 27 07:29:33 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-g[105245]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7ff475049d20] 2024-05-27T07:29:33.409 INFO:journalctl@ceph.mds.g.smithi052.stdout:May 27 07:29:33 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-g[105245]: 2: gsignal() 2024-05-27T07:29:33.409 INFO:journalctl@ceph.mds.g.smithi052.stdout:May 27 07:29:33 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-g[105245]: 3: abort() 2024-05-27T07:29:33.409 INFO:journalctl@ceph.mds.g.smithi052.stdout:May 27 07:29:33 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-g[105245]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x26e) [0x7ff4764ccced] 2024-05-27T07:29:33.409 INFO:journalctl@ceph.mds.g.smithi052.stdout:May 27 07:29:33 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-g[105245]: 5: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7ff4764ccdd1] 2024-05-27T07:29:33.409 INFO:journalctl@ceph.mds.g.smithi052.stdout:May 27 07:29:33 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-g[105245]: 6: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x55ce81f4d23d] 2024-05-27T07:29:33.409 INFO:journalctl@ceph.mds.g.smithi052.stdout:May 27 07:29:33 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-g[105245]: 7: (SnapRealm::build_snap_set() const+0x323) [0x55ce81f38971] 2024-05-27T07:29:33.409 INFO:journalctl@ceph.mds.g.smithi052.stdout:May 27 07:29:33 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-g[105245]: 8: (SnapRealm::check_cache() const+0x13e) [0x55ce81f36fa0] 2024-05-27T07:29:33.409 INFO:journalctl@ceph.mds.g.smithi052.stdout:May 27 07:29:33 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-g[105245]: 9: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x55ce81f37651] 2024-05-27T07:29:33.409 INFO:journalctl@ceph.mds.g.smithi052.stdout:May 27 07:29:33 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-g[105245]: 10: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x55ce81ecc347] 2024-05-27T07:29:33.409 INFO:journalctl@ceph.mds.g.smithi052.stdout:May 27 07:29:33 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-g[105245]: 11: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x55ce81bc887e] 2024-05-27T07:29:33.410 INFO:journalctl@ceph.mds.g.smithi052.stdout:May 27 07:29:33 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-g[105245]: 12: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x2702) [0x55ce81bcde94] 2024-05-27T07:29:33.410 INFO:journalctl@ceph.mds.g.smithi052.stdout:May 27 07:29:33 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-g[105245]: 13: (EUpdate::replay(MDSRank*)+0x63) [0x55ce81bd8969] 2024-05-27T07:29:33.410 INFO:journalctl@ceph.mds.g.smithi052.stdout:May 27 07:29:33 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-g[105245]: 14: (MDLog::_replay_thread()+0x1f38) [0x55ce81f7148c] 2024-05-27T07:29:33.410 INFO:journalctl@ceph.mds.g.smithi052.stdout:May 27 07:29:33 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-g[105245]: 15: (MDLog::ReplayThread::entry()+0x11) [0x55ce81f7a9cb] 2024-05-27T07:29:33.410 INFO:journalctl@ceph.mds.g.smithi052.stdout:May 27 07:29:33 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-g[105245]: 16: (Thread::entry_wrapper()+0x43) [0x7ff47649b72d] 2024-05-27T07:29:33.410 INFO:journalctl@ceph.mds.g.smithi052.stdout:May 27 07:29:33 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-g[105245]: 17: (Thread::_entry_func(void*)+0xd) [0x7ff47649b749] 2024-05-27T07:29:33.410 INFO:journalctl@ceph.mds.g.smithi052.stdout:May 27 07:29:33 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-g[105245]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7ff47503f1ca] 2024-05-27T07:29:33.410 INFO:journalctl@ceph.mds.g.smithi052.stdout:May 27 07:29:33 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-g[105245]: 19: clone() 2024-05-27T07:29:33.410 INFO:journalctl@ceph.mds.g.smithi052.stdout:May 27 07:29:33 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-g[105245]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-05-27T07:29:33.410 INFO:journalctl@ceph.mds.g.smithi052.stdout:May 27 07:29:33 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-g[105245]: 2024-05-27T07:29:33.410 INFO:journalctl@ceph.mds.g.smithi052.stdout:May 27 07:29:33 smithi052 bash[105188]: 64363f89b5da86618bd5c5ce360a83faf2f428ec1d1fca15fef1b39bb68b23e0 2024-05-27T07:29:33.410 INFO:journalctl@ceph.mds.g.smithi052.stdout:May 27 07:29:33 smithi052 systemd[1]: Started Ceph mds.g for 73df2636-1bf8-11ef-bc9b-c7b262605968. 2024-05-27T07:29:33.411 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:29:33.397+0000 7f044aed2700 1 -- 172.21.15.52:0/966372404 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f044406faa0 con 0x7f0444106ec0 2024-05-27T07:29:33.411 INFO:teuthology.orchestra.run.smithi052.stdout: 2024-05-27T07:29:33.411 INFO:teuthology.orchestra.run.smithi052.stdout:{"epoch":60,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14556,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.52:6835/2584733778","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.52:6834","nonce":2584733778},{"type":"v1","addr":"172.21.15.52:6835","nonce":2584733778}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4}],"filesystems":[{"mdsmap":{"epoch":60,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-05-27T07:21:50.521783+0000","modified":"2024-05-27T07:29:33.116188+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24527,"mds_1":24497,"mds_2":24475,"mds_3":24473,"mds_4":24466},"failed":[],"damaged":[],"stopped":[],"info":{"gid_16236":{"gid":16236,"name":"a","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.52:6837/1075341446","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.52:6836","nonce":1075341446},{"type":"v1","addr":"172.21.15.52:6837","nonce":1075341446}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24466":{"gid":24466,"name":"c","rank":4,"incarnation":26,"state":"up:active","state_seq":25,"addr":"172.21.15.173:6837/1396363827","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6836","nonce":1396363827},{"type":"v1","addr":"172.21.15.173:6837","nonce":1396363827}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24473":{"gid":24473,"name":"e","rank":3,"incarnation":24,"state":"up:active","state_seq":26,"addr":"172.21.15.105:6835/3298498384","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6834","nonce":3298498384},{"type":"v1","addr":"172.21.15.105:6835","nonce":3298498384}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24475":{"gid":24475,"name":"i","rank":2,"incarnation":22,"state":"up:active","state_seq":23,"addr":"172.21.15.173:6839/564778293","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6838","nonce":564778293},{"type":"v1","addr":"172.21.15.173:6839","nonce":564778293}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24497":{"gid":24497,"name":"h","rank":1,"incarnation":20,"state":"up:active","state_seq":24,"addr":"172.21.15.105:6837/191393149","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6836","nonce":191393149},{"type":"v1","addr":"172.21.15.105:6837","nonce":191393149}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24527":{"gid":24527,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":5,"addr":"172.21.15.105:6839/500348151","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6838","nonce":500348151},{"type":"v1","addr":"172.21.15.105:6839","nonce":500348151}]},"join_fscid":-1,"export_targets":[1,2,3,4],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_25828":{"gid":25828,"name":"l","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.173:6833/453411960","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6832","nonce":453411960},{"type":"v1","addr":"172.21.15.173:6833","nonce":453411960}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_25895":{"gid":25895,"name":"k","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.105:6833/1948236635","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6832","nonce":1948236635},{"type":"v1","addr":"172.21.15.105:6833","nonce":1948236635}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_25949":{"gid":25949,"name":"d","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.52:6837/3586900688","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.52:6836","nonce":3586900688},{"type":"v1","addr":"172.21.15.52:6837","nonce":3586900688}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_25952":{"gid":25952,"name":"g","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.52:6839/3507425174","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.52:6838","nonce":3507425174},{"type":"v1","addr":"172.21.15.52:6839","nonce":3507425174}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24527,"qdb_cluster":[24497,24527,24475,24473,24466]},"id":1}]} 2024-05-27T07:29:33.412 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:29:33.398+0000 7f0441ffb700 1 -- 172.21.15.52:0/966372404 <== mon.1 v2:172.21.15.173:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 60 v60) ==== 94+0+11117 (secure 0 0 0) 0x7f04380aabc0 con 0x7f0444106ec0 2024-05-27T07:29:33.412 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:29:33.400+0000 7f044aed2700 1 -- 172.21.15.52:0/966372404 >> [v2:172.21.15.52:6800/1822318384,v1:172.21.15.52:6801/1822318384] conn(0x7f042c0b2a70 msgr2=0x7f042c0b4f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:29:33.412 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:29:33.400+0000 7f044aed2700 1 --2- 172.21.15.52:0/966372404 >> [v2:172.21.15.52:6800/1822318384,v1:172.21.15.52:6801/1822318384] conn(0x7f042c0b2a70 0x7f042c0b4f80 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7f044406fdc0 tx=0x7f043c009250 comp rx=0 tx=0).stop 2024-05-27T07:29:33.412 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:29:33.400+0000 7f044aed2700 1 -- 172.21.15.52:0/966372404 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f0444106ec0 msgr2=0x7f044406e570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:29:33.412 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:29:33.400+0000 7f044aed2700 1 --2- 172.21.15.52:0/966372404 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f0444106ec0 0x7f044406e570 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7f043800b5c0 tx=0x7f0438005590 comp rx=0 tx=0).stop 2024-05-27T07:29:33.412 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:29:33.400+0000 7f044aed2700 1 -- 172.21.15.52:0/966372404 shutdown_connections 2024-05-27T07:29:33.412 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:29:33.400+0000 7f044aed2700 1 --2- 172.21.15.52:0/966372404 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f044406eb70 0x7f044406f040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:29:33.413 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:29:33.400+0000 7f044aed2700 1 --2- 172.21.15.52:0/966372404 >> [v2:172.21.15.52:6800/1822318384,v1:172.21.15.52:6801/1822318384] conn(0x7f042c0b2a70 0x7f042c0b4f80 unknown :-1 s=CLOSED pgs=111 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:29:33.413 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:29:33.400+0000 7f044aed2700 1 --2- 172.21.15.52:0/966372404 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f0444106570 0x7f0444075440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:29:33.413 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:29:33.400+0000 7f044aed2700 1 --2- 172.21.15.52:0/966372404 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f0444106ec0 0x7f044406e570 unknown :-1 s=CLOSED pgs=115 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:29:33.413 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:29:33.400+0000 7f044aed2700 1 -- 172.21.15.52:0/966372404 >> 172.21.15.52:0/966372404 conn(0x7f0444102070 msgr2=0x7f04441047e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:29:33.413 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:29:33.401+0000 7f044aed2700 1 -- 172.21.15.52:0/966372404 shutdown_connections 2024-05-27T07:29:33.413 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:29:33.401+0000 7f044aed2700 1 -- 172.21.15.52:0/966372404 wait complete. 2024-05-27T07:29:33.413 INFO:teuthology.orchestra.run.smithi052.stderr:dumped fsmap epoch 60 2024-05-27T07:29:33.440 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:29:33 smithi105 ceph-mon[35977]: daemon mds.d restarted 2024-05-27T07:29:33.732 INFO:journalctl@ceph.mds.g.smithi052.stdout:May 27 07:29:33 smithi052 podman[105325]: 2024-05-27 07:29:33.412223176 +0000 UTC m=+0.129609181 container died 64363f89b5da86618bd5c5ce360a83faf2f428ec1d1fca15fef1b39bb68b23e0 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-g, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, GIT_CLEAN=True, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240521, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, org.label-schema.vendor=CentOS) 2024-05-27T07:29:34.230 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:29:34 smithi052 ceph-mon[31955]: pgmap v123: 129 pgs: 129 active+clean; 235 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 3.6 MiB/s rd, 5.1 MiB/s wr, 778 op/s 2024-05-27T07:29:34.230 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:29:34 smithi052 ceph-mon[31955]: osdmap e104: 12 total, 12 up, 12 in 2024-05-27T07:29:34.230 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:29:34 smithi052 ceph-mon[31955]: mds.? [v2:172.21.15.52:6836/3586900688,v1:172.21.15.52:6837/3586900688] up:boot 2024-05-27T07:29:34.230 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:29:34 smithi052 ceph-mon[31955]: daemon mds.g restarted 2024-05-27T07:29:34.231 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:29:34 smithi052 ceph-mon[31955]: fsmap cephfs:5 4 up:standby-replay 2 up:standby 5 up:active 2024-05-27T07:29:34.231 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:29:34 smithi052 ceph-mon[31955]: from='mgr.15333 172.21.15.52:0/416389082' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "d"}]: dispatch 2024-05-27T07:29:34.231 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:29:34 smithi052 ceph-mon[31955]: osdmap e105: 12 total, 12 up, 12 in 2024-05-27T07:29:34.231 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:29:34 smithi052 ceph-mon[31955]: mds.? [v2:172.21.15.52:6838/3507425174,v1:172.21.15.52:6839/3507425174] up:boot 2024-05-27T07:29:34.231 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:29:34 smithi052 ceph-mon[31955]: fsmap cephfs:5 5 up:standby-replay 1 up:standby 5 up:active 2024-05-27T07:29:34.231 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:29:34 smithi052 ceph-mon[31955]: from='mgr.15333 172.21.15.52:0/416389082' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2024-05-27T07:29:34.231 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:29:34 smithi052 ceph-mon[31955]: from='client.? 172.21.15.52:0/966372404' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-27T07:29:34.353 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:29:34 smithi173 ceph-mon[37000]: pgmap v123: 129 pgs: 129 active+clean; 235 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 3.6 MiB/s rd, 5.1 MiB/s wr, 778 op/s 2024-05-27T07:29:34.353 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:29:34 smithi173 ceph-mon[37000]: osdmap e104: 12 total, 12 up, 12 in 2024-05-27T07:29:34.353 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:29:34 smithi173 ceph-mon[37000]: mds.? [v2:172.21.15.52:6836/3586900688,v1:172.21.15.52:6837/3586900688] up:boot 2024-05-27T07:29:34.353 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:29:34 smithi173 ceph-mon[37000]: daemon mds.g restarted 2024-05-27T07:29:34.353 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:29:34 smithi173 ceph-mon[37000]: fsmap cephfs:5 4 up:standby-replay 2 up:standby 5 up:active 2024-05-27T07:29:34.353 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:29:34 smithi173 ceph-mon[37000]: from='mgr.15333 172.21.15.52:0/416389082' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "d"}]: dispatch 2024-05-27T07:29:34.353 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:29:34 smithi173 ceph-mon[37000]: osdmap e105: 12 total, 12 up, 12 in 2024-05-27T07:29:34.354 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:29:34 smithi173 ceph-mon[37000]: mds.? [v2:172.21.15.52:6838/3507425174,v1:172.21.15.52:6839/3507425174] up:boot 2024-05-27T07:29:34.354 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:29:34 smithi173 ceph-mon[37000]: fsmap cephfs:5 5 up:standby-replay 1 up:standby 5 up:active 2024-05-27T07:29:34.354 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:29:34 smithi173 ceph-mon[37000]: from='mgr.15333 172.21.15.52:0/416389082' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2024-05-27T07:29:34.354 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:29:34 smithi173 ceph-mon[37000]: from='client.? 172.21.15.52:0/966372404' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-27T07:29:34.441 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:29:34 smithi105 ceph-mon[35977]: pgmap v123: 129 pgs: 129 active+clean; 235 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 3.6 MiB/s rd, 5.1 MiB/s wr, 778 op/s 2024-05-27T07:29:34.442 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:29:34 smithi105 ceph-mon[35977]: osdmap e104: 12 total, 12 up, 12 in 2024-05-27T07:29:34.442 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:29:34 smithi105 ceph-mon[35977]: mds.? [v2:172.21.15.52:6836/3586900688,v1:172.21.15.52:6837/3586900688] up:boot 2024-05-27T07:29:34.442 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:29:34 smithi105 ceph-mon[35977]: daemon mds.g restarted 2024-05-27T07:29:34.442 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:29:34 smithi105 ceph-mon[35977]: fsmap cephfs:5 4 up:standby-replay 2 up:standby 5 up:active 2024-05-27T07:29:34.442 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:29:34 smithi105 ceph-mon[35977]: from='mgr.15333 172.21.15.52:0/416389082' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "d"}]: dispatch 2024-05-27T07:29:34.442 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:29:34 smithi105 ceph-mon[35977]: osdmap e105: 12 total, 12 up, 12 in 2024-05-27T07:29:34.442 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:29:34 smithi105 ceph-mon[35977]: mds.? [v2:172.21.15.52:6838/3507425174,v1:172.21.15.52:6839/3507425174] up:boot 2024-05-27T07:29:34.442 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:29:34 smithi105 ceph-mon[35977]: fsmap cephfs:5 5 up:standby-replay 1 up:standby 5 up:active 2024-05-27T07:29:34.442 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:29:34 smithi105 ceph-mon[35977]: from='mgr.15333 172.21.15.52:0/416389082' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2024-05-27T07:29:34.443 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:29:34 smithi105 ceph-mon[35977]: from='client.? 172.21.15.52:0/966372404' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-27T07:29:34.853 INFO:journalctl@ceph.mds.g.smithi052.stdout:May 27 07:29:34 smithi052 podman[105325]: 2024-05-27 07:29:34.549049048 +0000 UTC m=+1.266435044 container remove 64363f89b5da86618bd5c5ce360a83faf2f428ec1d1fca15fef1b39bb68b23e0 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-g, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, ceph=True, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, org.label-schema.build-date=20240521, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2) 2024-05-27T07:29:34.854 INFO:journalctl@ceph.mds.g.smithi052.stdout:May 27 07:29:34 smithi052 systemd[1]: ceph-73df2636-1bf8-11ef-bc9b-c7b262605968@mds.g.service: Main process exited, code=exited, status=134/n/a 2024-05-27T07:29:34.854 INFO:journalctl@ceph.mds.d.smithi052.stdout:May 27 07:29:34 smithi052 podman[105314]: 2024-05-27 07:29:34.577288418 +0000 UTC m=+1.361400869 container died 935d31972b66528c0799298a9629bb77353b540e9609ecda523a8fd9079cabc0 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-d, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240521, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, GIT_CLEAN=True) 2024-05-27T07:29:35.188 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:29:35 smithi052 ceph-mon[31955]: MDS daemon mds.k is removed because it is dead or otherwise unavailable. 2024-05-27T07:29:35.188 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:29:35 smithi052 ceph-mon[31955]: fsmap cephfs:5 5 up:standby-replay 5 up:active 2024-05-27T07:29:35.206 INFO:tasks.workunit.client.0.smithi052.stdout: 61 583659 351 403862 315 362705 1114 2024-05-27T07:29:35.352 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:29:35 smithi173 ceph-mon[37000]: MDS daemon mds.k is removed because it is dead or otherwise unavailable. 2024-05-27T07:29:35.352 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:29:35 smithi173 ceph-mon[37000]: fsmap cephfs:5 5 up:standby-replay 5 up:active 2024-05-27T07:29:35.440 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:29:35 smithi105 ceph-mon[35977]: MDS daemon mds.k is removed because it is dead or otherwise unavailable. 2024-05-27T07:29:35.440 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:29:35 smithi105 ceph-mon[35977]: fsmap cephfs:5 5 up:standby-replay 5 up:active 2024-05-27T07:29:35.443 INFO:journalctl@ceph.mds.d.smithi052.stdout:May 27 07:29:35 smithi052 podman[105314]: 2024-05-27 07:29:35.196070246 +0000 UTC m=+1.980182685 container remove 935d31972b66528c0799298a9629bb77353b540e9609ecda523a8fd9079cabc0 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-d, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, GIT_BRANCH=HEAD, org.label-schema.build-date=20240521, org.label-schema.license=GPLv2, GIT_CLEAN=True, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, io.buildah.version=1.33.5, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0) 2024-05-27T07:29:35.443 INFO:journalctl@ceph.mds.d.smithi052.stdout:May 27 07:29:35 smithi052 systemd[1]: ceph-73df2636-1bf8-11ef-bc9b-c7b262605968@mds.d.service: Main process exited, code=exited, status=134/n/a 2024-05-27T07:29:35.444 INFO:journalctl@ceph.mds.j.smithi052.stdout:May 27 07:29:35 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-j[72721]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: In function 'std::set SnapClient::filter(const std::set&) const' thread 7fa501aa8700 time 2024-05-27T07:29:35.244135+0000 2024-05-27T07:29:35.444 INFO:journalctl@ceph.mds.j.smithi052.stdout:May 27 07:29:35 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-j[72721]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: 197: FAILED ceph_assert(cached_version > 0) 2024-05-27T07:29:35.444 INFO:journalctl@ceph.mds.j.smithi052.stdout:May 27 07:29:35 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-j[72721]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:29:35.444 INFO:journalctl@ceph.mds.j.smithi052.stdout:May 27 07:29:35 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-j[72721]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x127) [0x7fa5142edba6] 2024-05-27T07:29:35.444 INFO:journalctl@ceph.mds.j.smithi052.stdout:May 27 07:29:35 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-j[72721]: 2: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7fa5142eddd1] 2024-05-27T07:29:35.444 INFO:journalctl@ceph.mds.j.smithi052.stdout:May 27 07:29:35 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-j[72721]: 3: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x55b5c75b823d] 2024-05-27T07:29:35.445 INFO:journalctl@ceph.mds.j.smithi052.stdout:May 27 07:29:35 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-j[72721]: 4: (SnapRealm::build_snap_set() const+0x323) [0x55b5c75a3971] 2024-05-27T07:29:35.445 INFO:journalctl@ceph.mds.j.smithi052.stdout:May 27 07:29:35 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-j[72721]: 5: (SnapRealm::check_cache() const+0x13e) [0x55b5c75a1fa0] 2024-05-27T07:29:35.445 INFO:journalctl@ceph.mds.j.smithi052.stdout:May 27 07:29:35 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-j[72721]: 6: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x55b5c75a2651] 2024-05-27T07:29:35.445 INFO:journalctl@ceph.mds.j.smithi052.stdout:May 27 07:29:35 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-j[72721]: 7: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x55b5c7537347] 2024-05-27T07:29:35.445 INFO:journalctl@ceph.mds.j.smithi052.stdout:May 27 07:29:35 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-j[72721]: 8: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x55b5c723387e] 2024-05-27T07:29:35.445 INFO:journalctl@ceph.mds.j.smithi052.stdout:May 27 07:29:35 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-j[72721]: 9: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x2702) [0x55b5c7238e94] 2024-05-27T07:29:35.445 INFO:journalctl@ceph.mds.j.smithi052.stdout:May 27 07:29:35 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-j[72721]: 10: (EUpdate::replay(MDSRank*)+0x63) [0x55b5c7243969] 2024-05-27T07:29:35.445 INFO:journalctl@ceph.mds.j.smithi052.stdout:May 27 07:29:35 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-j[72721]: 11: (MDLog::_replay_thread()+0x1f38) [0x55b5c75dc48c] 2024-05-27T07:29:35.445 INFO:journalctl@ceph.mds.j.smithi052.stdout:May 27 07:29:35 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-j[72721]: 12: (MDLog::ReplayThread::entry()+0x11) [0x55b5c75e59cb] 2024-05-27T07:29:35.445 INFO:journalctl@ceph.mds.j.smithi052.stdout:May 27 07:29:35 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-j[72721]: 13: (Thread::entry_wrapper()+0x43) [0x7fa5142bc72d] 2024-05-27T07:29:35.445 INFO:journalctl@ceph.mds.j.smithi052.stdout:May 27 07:29:35 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-j[72721]: 14: (Thread::_entry_func(void*)+0xd) [0x7fa5142bc749] 2024-05-27T07:29:35.445 INFO:journalctl@ceph.mds.j.smithi052.stdout:May 27 07:29:35 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-j[72721]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7fa512e601ca] 2024-05-27T07:29:35.445 INFO:journalctl@ceph.mds.j.smithi052.stdout:May 27 07:29:35 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-j[72721]: 16: clone() 2024-05-27T07:29:35.446 INFO:journalctl@ceph.mds.j.smithi052.stdout:May 27 07:29:35 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-j[72721]: *** Caught signal (Aborted) ** 2024-05-27T07:29:35.446 INFO:journalctl@ceph.mds.j.smithi052.stdout:May 27 07:29:35 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-j[72721]: in thread 7fa501aa8700 thread_name:md_log_replay 2024-05-27T07:29:35.446 INFO:journalctl@ceph.mds.j.smithi052.stdout:May 27 07:29:35 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-j[72721]: 2024-05-27T07:29:35.245+0000 7fa501aa8700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: In function 'std::set SnapClient::filter(const std::set&) const' thread 7fa501aa8700 time 2024-05-27T07:29:35.244135+0000 2024-05-27T07:29:35.446 INFO:journalctl@ceph.mds.j.smithi052.stdout:May 27 07:29:35 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-j[72721]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: 197: FAILED ceph_assert(cached_version > 0) 2024-05-27T07:29:35.446 INFO:journalctl@ceph.mds.j.smithi052.stdout:May 27 07:29:35 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-j[72721]: 2024-05-27T07:29:35.446 INFO:journalctl@ceph.mds.j.smithi052.stdout:May 27 07:29:35 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-j[72721]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:29:35.446 INFO:journalctl@ceph.mds.j.smithi052.stdout:May 27 07:29:35 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-j[72721]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x127) [0x7fa5142edba6] 2024-05-27T07:29:35.446 INFO:journalctl@ceph.mds.j.smithi052.stdout:May 27 07:29:35 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-j[72721]: 2: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7fa5142eddd1] 2024-05-27T07:29:35.446 INFO:journalctl@ceph.mds.j.smithi052.stdout:May 27 07:29:35 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-j[72721]: 3: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x55b5c75b823d] 2024-05-27T07:29:35.447 INFO:journalctl@ceph.mds.j.smithi052.stdout:May 27 07:29:35 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-j[72721]: 4: (SnapRealm::build_snap_set() const+0x323) [0x55b5c75a3971] 2024-05-27T07:29:35.447 INFO:journalctl@ceph.mds.j.smithi052.stdout:May 27 07:29:35 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-j[72721]: 5: (SnapRealm::check_cache() const+0x13e) [0x55b5c75a1fa0] 2024-05-27T07:29:35.447 INFO:journalctl@ceph.mds.j.smithi052.stdout:May 27 07:29:35 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-j[72721]: 6: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x55b5c75a2651] 2024-05-27T07:29:35.447 INFO:journalctl@ceph.mds.j.smithi052.stdout:May 27 07:29:35 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-j[72721]: 7: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x55b5c7537347] 2024-05-27T07:29:35.447 INFO:journalctl@ceph.mds.j.smithi052.stdout:May 27 07:29:35 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-j[72721]: 8: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x55b5c723387e] 2024-05-27T07:29:35.447 INFO:journalctl@ceph.mds.j.smithi052.stdout:May 27 07:29:35 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-j[72721]: 9: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x2702) [0x55b5c7238e94] 2024-05-27T07:29:35.447 INFO:journalctl@ceph.mds.j.smithi052.stdout:May 27 07:29:35 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-j[72721]: 10: (EUpdate::replay(MDSRank*)+0x63) [0x55b5c7243969] 2024-05-27T07:29:35.447 INFO:journalctl@ceph.mds.j.smithi052.stdout:May 27 07:29:35 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-j[72721]: 11: (MDLog::_replay_thread()+0x1f38) [0x55b5c75dc48c] 2024-05-27T07:29:35.447 INFO:journalctl@ceph.mds.j.smithi052.stdout:May 27 07:29:35 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-j[72721]: 12: (MDLog::ReplayThread::entry()+0x11) [0x55b5c75e59cb] 2024-05-27T07:29:35.447 INFO:journalctl@ceph.mds.j.smithi052.stdout:May 27 07:29:35 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-j[72721]: 13: (Thread::entry_wrapper()+0x43) [0x7fa5142bc72d] 2024-05-27T07:29:35.447 INFO:journalctl@ceph.mds.j.smithi052.stdout:May 27 07:29:35 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-j[72721]: 14: (Thread::_entry_func(void*)+0xd) [0x7fa5142bc749] 2024-05-27T07:29:35.447 INFO:journalctl@ceph.mds.j.smithi052.stdout:May 27 07:29:35 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-j[72721]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7fa512e601ca] 2024-05-27T07:29:35.447 INFO:journalctl@ceph.mds.j.smithi052.stdout:May 27 07:29:35 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-j[72721]: 16: clone() 2024-05-27T07:29:35.448 INFO:journalctl@ceph.mds.j.smithi052.stdout:May 27 07:29:35 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-j[72721]: 2024-05-27T07:29:35.448 INFO:journalctl@ceph.mds.j.smithi052.stdout:May 27 07:29:35 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-j[72721]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:29:35.448 INFO:journalctl@ceph.mds.j.smithi052.stdout:May 27 07:29:35 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-j[72721]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7fa512e6ad20] 2024-05-27T07:29:35.448 INFO:journalctl@ceph.mds.j.smithi052.stdout:May 27 07:29:35 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-j[72721]: 2: gsignal() 2024-05-27T07:29:35.448 INFO:journalctl@ceph.mds.j.smithi052.stdout:May 27 07:29:35 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-j[72721]: 3: abort() 2024-05-27T07:29:35.448 INFO:journalctl@ceph.mds.j.smithi052.stdout:May 27 07:29:35 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-j[72721]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x26e) [0x7fa5142edced] 2024-05-27T07:29:35.448 INFO:journalctl@ceph.mds.j.smithi052.stdout:May 27 07:29:35 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-j[72721]: 5: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7fa5142eddd1] 2024-05-27T07:29:35.449 INFO:journalctl@ceph.mds.j.smithi052.stdout:May 27 07:29:35 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-j[72721]: 6: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x55b5c75b823d] 2024-05-27T07:29:35.449 INFO:journalctl@ceph.mds.j.smithi052.stdout:May 27 07:29:35 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-j[72721]: 7: (SnapRealm::build_snap_set() const+0x323) [0x55b5c75a3971] 2024-05-27T07:29:35.449 INFO:journalctl@ceph.mds.j.smithi052.stdout:May 27 07:29:35 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-j[72721]: 8: (SnapRealm::check_cache() const+0x13e) [0x55b5c75a1fa0] 2024-05-27T07:29:35.449 INFO:journalctl@ceph.mds.j.smithi052.stdout:May 27 07:29:35 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-j[72721]: 9: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x55b5c75a2651] 2024-05-27T07:29:35.449 INFO:journalctl@ceph.mds.j.smithi052.stdout:May 27 07:29:35 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-j[72721]: 10: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x55b5c7537347] 2024-05-27T07:29:35.449 INFO:journalctl@ceph.mds.j.smithi052.stdout:May 27 07:29:35 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-j[72721]: 11: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x55b5c723387e] 2024-05-27T07:29:35.449 INFO:journalctl@ceph.mds.j.smithi052.stdout:May 27 07:29:35 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-j[72721]: 12: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x2702) [0x55b5c7238e94] 2024-05-27T07:29:35.449 INFO:journalctl@ceph.mds.j.smithi052.stdout:May 27 07:29:35 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-j[72721]: 13: (EUpdate::replay(MDSRank*)+0x63) [0x55b5c7243969] 2024-05-27T07:29:35.449 INFO:journalctl@ceph.mds.j.smithi052.stdout:May 27 07:29:35 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-j[72721]: 14: (MDLog::_replay_thread()+0x1f38) [0x55b5c75dc48c] 2024-05-27T07:29:35.449 INFO:journalctl@ceph.mds.j.smithi052.stdout:May 27 07:29:35 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-j[72721]: 15: (MDLog::ReplayThread::entry()+0x11) [0x55b5c75e59cb] 2024-05-27T07:29:35.449 INFO:journalctl@ceph.mds.j.smithi052.stdout:May 27 07:29:35 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-j[72721]: 16: (Thread::entry_wrapper()+0x43) [0x7fa5142bc72d] 2024-05-27T07:29:35.449 INFO:journalctl@ceph.mds.j.smithi052.stdout:May 27 07:29:35 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-j[72721]: 17: (Thread::_entry_func(void*)+0xd) [0x7fa5142bc749] 2024-05-27T07:29:35.450 INFO:journalctl@ceph.mds.j.smithi052.stdout:May 27 07:29:35 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-j[72721]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7fa512e601ca] 2024-05-27T07:29:35.450 INFO:journalctl@ceph.mds.j.smithi052.stdout:May 27 07:29:35 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-j[72721]: 19: clone() 2024-05-27T07:29:35.450 INFO:journalctl@ceph.mds.j.smithi052.stdout:May 27 07:29:35 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-j[72721]: 2024-05-27T07:29:35.248+0000 7fa501aa8700 -1 *** Caught signal (Aborted) ** 2024-05-27T07:29:35.450 INFO:journalctl@ceph.mds.j.smithi052.stdout:May 27 07:29:35 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-j[72721]: in thread 7fa501aa8700 thread_name:md_log_replay 2024-05-27T07:29:35.450 INFO:journalctl@ceph.mds.j.smithi052.stdout:May 27 07:29:35 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-j[72721]: 2024-05-27T07:29:35.450 INFO:journalctl@ceph.mds.j.smithi052.stdout:May 27 07:29:35 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-j[72721]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:29:35.450 INFO:journalctl@ceph.mds.j.smithi052.stdout:May 27 07:29:35 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-j[72721]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7fa512e6ad20] 2024-05-27T07:29:35.450 INFO:journalctl@ceph.mds.j.smithi052.stdout:May 27 07:29:35 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-j[72721]: 2: gsignal() 2024-05-27T07:29:35.450 INFO:journalctl@ceph.mds.j.smithi052.stdout:May 27 07:29:35 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-j[72721]: 3: abort() 2024-05-27T07:29:35.450 INFO:journalctl@ceph.mds.j.smithi052.stdout:May 27 07:29:35 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-j[72721]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x26e) [0x7fa5142edced] 2024-05-27T07:29:35.450 INFO:journalctl@ceph.mds.j.smithi052.stdout:May 27 07:29:35 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-j[72721]: 5: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7fa5142eddd1] 2024-05-27T07:29:35.450 INFO:journalctl@ceph.mds.j.smithi052.stdout:May 27 07:29:35 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-j[72721]: 6: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x55b5c75b823d] 2024-05-27T07:29:35.450 INFO:journalctl@ceph.mds.j.smithi052.stdout:May 27 07:29:35 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-j[72721]: 7: (SnapRealm::build_snap_set() const+0x323) [0x55b5c75a3971] 2024-05-27T07:29:35.451 INFO:journalctl@ceph.mds.j.smithi052.stdout:May 27 07:29:35 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-j[72721]: 8: (SnapRealm::check_cache() const+0x13e) [0x55b5c75a1fa0] 2024-05-27T07:29:35.451 INFO:journalctl@ceph.mds.j.smithi052.stdout:May 27 07:29:35 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-j[72721]: 9: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x55b5c75a2651] 2024-05-27T07:29:35.451 INFO:journalctl@ceph.mds.j.smithi052.stdout:May 27 07:29:35 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-j[72721]: 10: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x55b5c7537347] 2024-05-27T07:29:35.451 INFO:journalctl@ceph.mds.j.smithi052.stdout:May 27 07:29:35 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-j[72721]: 11: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x55b5c723387e] 2024-05-27T07:29:35.451 INFO:journalctl@ceph.mds.j.smithi052.stdout:May 27 07:29:35 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-j[72721]: 12: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x2702) [0x55b5c7238e94] 2024-05-27T07:29:35.451 INFO:journalctl@ceph.mds.j.smithi052.stdout:May 27 07:29:35 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-j[72721]: 13: (EUpdate::replay(MDSRank*)+0x63) [0x55b5c7243969] 2024-05-27T07:29:35.451 INFO:journalctl@ceph.mds.j.smithi052.stdout:May 27 07:29:35 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-j[72721]: 14: (MDLog::_replay_thread()+0x1f38) [0x55b5c75dc48c] 2024-05-27T07:29:35.451 INFO:journalctl@ceph.mds.j.smithi052.stdout:May 27 07:29:35 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-j[72721]: 15: (MDLog::ReplayThread::entry()+0x11) [0x55b5c75e59cb] 2024-05-27T07:29:35.451 INFO:journalctl@ceph.mds.j.smithi052.stdout:May 27 07:29:35 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-j[72721]: 16: (Thread::entry_wrapper()+0x43) [0x7fa5142bc72d] 2024-05-27T07:29:35.451 INFO:journalctl@ceph.mds.j.smithi052.stdout:May 27 07:29:35 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-j[72721]: 17: (Thread::_entry_func(void*)+0xd) [0x7fa5142bc749] 2024-05-27T07:29:35.451 INFO:journalctl@ceph.mds.j.smithi052.stdout:May 27 07:29:35 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-j[72721]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7fa512e601ca] 2024-05-27T07:29:35.451 INFO:journalctl@ceph.mds.j.smithi052.stdout:May 27 07:29:35 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-j[72721]: 19: clone() 2024-05-27T07:29:35.452 INFO:journalctl@ceph.mds.j.smithi052.stdout:May 27 07:29:35 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-j[72721]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-05-27T07:29:35.452 INFO:journalctl@ceph.mds.j.smithi052.stdout:May 27 07:29:35 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-j[72721]: 2024-05-27T07:29:35.452 INFO:journalctl@ceph.mds.j.smithi052.stdout:May 27 07:29:35 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-j[72721]: -1> 2024-05-27T07:29:35.245+0000 7fa501aa8700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: In function 'std::set SnapClient::filter(const std::set&) const' thread 7fa501aa8700 time 2024-05-27T07:29:35.244135+0000 2024-05-27T07:29:35.452 INFO:journalctl@ceph.mds.j.smithi052.stdout:May 27 07:29:35 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-j[72721]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: 197: FAILED ceph_assert(cached_version > 0) 2024-05-27T07:29:35.452 INFO:journalctl@ceph.mds.j.smithi052.stdout:May 27 07:29:35 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-j[72721]: 2024-05-27T07:29:35.452 INFO:journalctl@ceph.mds.j.smithi052.stdout:May 27 07:29:35 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-j[72721]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:29:35.452 INFO:journalctl@ceph.mds.j.smithi052.stdout:May 27 07:29:35 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-j[72721]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x127) [0x7fa5142edba6] 2024-05-27T07:29:35.452 INFO:journalctl@ceph.mds.j.smithi052.stdout:May 27 07:29:35 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-j[72721]: 2: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7fa5142eddd1] 2024-05-27T07:29:35.452 INFO:journalctl@ceph.mds.j.smithi052.stdout:May 27 07:29:35 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-j[72721]: 3: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x55b5c75b823d] 2024-05-27T07:29:35.452 INFO:journalctl@ceph.mds.j.smithi052.stdout:May 27 07:29:35 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-j[72721]: 4: (SnapRealm::build_snap_set() const+0x323) [0x55b5c75a3971] 2024-05-27T07:29:35.452 INFO:journalctl@ceph.mds.j.smithi052.stdout:May 27 07:29:35 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-j[72721]: 5: (SnapRealm::check_cache() const+0x13e) [0x55b5c75a1fa0] 2024-05-27T07:29:35.452 INFO:journalctl@ceph.mds.j.smithi052.stdout:May 27 07:29:35 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-j[72721]: 6: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x55b5c75a2651] 2024-05-27T07:29:35.452 INFO:journalctl@ceph.mds.j.smithi052.stdout:May 27 07:29:35 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-j[72721]: 7: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x55b5c7537347] 2024-05-27T07:29:35.453 INFO:journalctl@ceph.mds.j.smithi052.stdout:May 27 07:29:35 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-j[72721]: 8: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x55b5c723387e] 2024-05-27T07:29:35.453 INFO:journalctl@ceph.mds.j.smithi052.stdout:May 27 07:29:35 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-j[72721]: 9: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x2702) [0x55b5c7238e94] 2024-05-27T07:29:35.453 INFO:journalctl@ceph.mds.j.smithi052.stdout:May 27 07:29:35 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-j[72721]: 10: (EUpdate::replay(MDSRank*)+0x63) [0x55b5c7243969] 2024-05-27T07:29:35.453 INFO:journalctl@ceph.mds.j.smithi052.stdout:May 27 07:29:35 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-j[72721]: 11: (MDLog::_replay_thread()+0x1f38) [0x55b5c75dc48c] 2024-05-27T07:29:35.454 INFO:journalctl@ceph.mds.j.smithi052.stdout:May 27 07:29:35 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-j[72721]: 12: (MDLog::ReplayThread::entry()+0x11) [0x55b5c75e59cb] 2024-05-27T07:29:35.454 INFO:journalctl@ceph.mds.j.smithi052.stdout:May 27 07:29:35 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-j[72721]: 13: (Thread::entry_wrapper()+0x43) [0x7fa5142bc72d] 2024-05-27T07:29:35.454 INFO:journalctl@ceph.mds.j.smithi052.stdout:May 27 07:29:35 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-j[72721]: 14: (Thread::_entry_func(void*)+0xd) [0x7fa5142bc749] 2024-05-27T07:29:35.454 INFO:journalctl@ceph.mds.j.smithi052.stdout:May 27 07:29:35 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-j[72721]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7fa512e601ca] 2024-05-27T07:29:35.454 INFO:journalctl@ceph.mds.j.smithi052.stdout:May 27 07:29:35 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-j[72721]: 16: clone() 2024-05-27T07:29:35.454 INFO:journalctl@ceph.mds.j.smithi052.stdout:May 27 07:29:35 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-j[72721]: 2024-05-27T07:29:35.454 INFO:journalctl@ceph.mds.j.smithi052.stdout:May 27 07:29:35 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-j[72721]: 0> 2024-05-27T07:29:35.248+0000 7fa501aa8700 -1 *** Caught signal (Aborted) ** 2024-05-27T07:29:35.454 INFO:journalctl@ceph.mds.j.smithi052.stdout:May 27 07:29:35 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-j[72721]: in thread 7fa501aa8700 thread_name:md_log_replay 2024-05-27T07:29:35.454 INFO:journalctl@ceph.mds.j.smithi052.stdout:May 27 07:29:35 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-j[72721]: 2024-05-27T07:29:35.454 INFO:journalctl@ceph.mds.j.smithi052.stdout:May 27 07:29:35 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-j[72721]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:29:35.454 INFO:journalctl@ceph.mds.j.smithi052.stdout:May 27 07:29:35 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-j[72721]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7fa512e6ad20] 2024-05-27T07:29:35.454 INFO:journalctl@ceph.mds.j.smithi052.stdout:May 27 07:29:35 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-j[72721]: 2: gsignal() 2024-05-27T07:29:35.455 INFO:journalctl@ceph.mds.j.smithi052.stdout:May 27 07:29:35 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-j[72721]: 3: abort() 2024-05-27T07:29:35.455 INFO:journalctl@ceph.mds.j.smithi052.stdout:May 27 07:29:35 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-j[72721]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x26e) [0x7fa5142edced] 2024-05-27T07:29:35.455 INFO:journalctl@ceph.mds.j.smithi052.stdout:May 27 07:29:35 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-j[72721]: 5: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7fa5142eddd1] 2024-05-27T07:29:35.455 INFO:journalctl@ceph.mds.j.smithi052.stdout:May 27 07:29:35 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-j[72721]: 6: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x55b5c75b823d] 2024-05-27T07:29:35.455 INFO:journalctl@ceph.mds.j.smithi052.stdout:May 27 07:29:35 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-j[72721]: 7: (SnapRealm::build_snap_set() const+0x323) [0x55b5c75a3971] 2024-05-27T07:29:35.455 INFO:journalctl@ceph.mds.j.smithi052.stdout:May 27 07:29:35 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-j[72721]: 8: (SnapRealm::check_cache() const+0x13e) [0x55b5c75a1fa0] 2024-05-27T07:29:35.455 INFO:journalctl@ceph.mds.j.smithi052.stdout:May 27 07:29:35 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-j[72721]: 9: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x55b5c75a2651] 2024-05-27T07:29:35.455 INFO:journalctl@ceph.mds.j.smithi052.stdout:May 27 07:29:35 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-j[72721]: 10: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x55b5c7537347] 2024-05-27T07:29:35.455 INFO:journalctl@ceph.mds.j.smithi052.stdout:May 27 07:29:35 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-j[72721]: 11: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x55b5c723387e] 2024-05-27T07:29:35.455 INFO:journalctl@ceph.mds.j.smithi052.stdout:May 27 07:29:35 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-j[72721]: 12: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x2702) [0x55b5c7238e94] 2024-05-27T07:29:35.455 INFO:journalctl@ceph.mds.j.smithi052.stdout:May 27 07:29:35 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-j[72721]: 13: (EUpdate::replay(MDSRank*)+0x63) [0x55b5c7243969] 2024-05-27T07:29:35.455 INFO:journalctl@ceph.mds.j.smithi052.stdout:May 27 07:29:35 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-j[72721]: 14: (MDLog::_replay_thread()+0x1f38) [0x55b5c75dc48c] 2024-05-27T07:29:35.456 INFO:journalctl@ceph.mds.j.smithi052.stdout:May 27 07:29:35 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-j[72721]: 15: (MDLog::ReplayThread::entry()+0x11) [0x55b5c75e59cb] 2024-05-27T07:29:35.456 INFO:journalctl@ceph.mds.j.smithi052.stdout:May 27 07:29:35 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-j[72721]: 16: (Thread::entry_wrapper()+0x43) [0x7fa5142bc72d] 2024-05-27T07:29:35.456 INFO:journalctl@ceph.mds.j.smithi052.stdout:May 27 07:29:35 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-j[72721]: 17: (Thread::_entry_func(void*)+0xd) [0x7fa5142bc749] 2024-05-27T07:29:35.456 INFO:journalctl@ceph.mds.j.smithi052.stdout:May 27 07:29:35 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-j[72721]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7fa512e601ca] 2024-05-27T07:29:35.456 INFO:journalctl@ceph.mds.j.smithi052.stdout:May 27 07:29:35 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-j[72721]: 19: clone() 2024-05-27T07:29:35.456 INFO:journalctl@ceph.mds.j.smithi052.stdout:May 27 07:29:35 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-j[72721]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-05-27T07:29:35.456 INFO:journalctl@ceph.mds.j.smithi052.stdout:May 27 07:29:35 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-j[72721]: 2024-05-27T07:29:35.456 INFO:journalctl@ceph.mds.j.smithi052.stdout:May 27 07:29:35 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-j[72721]: -1> 2024-05-27T07:29:35.245+0000 7fa501aa8700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: In function 'std::set SnapClient::filter(const std::set&) const' thread 7fa501aa8700 time 2024-05-27T07:29:35.244135+0000 2024-05-27T07:29:35.456 INFO:journalctl@ceph.mds.j.smithi052.stdout:May 27 07:29:35 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-j[72721]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: 197: FAILED ceph_assert(cached_version > 0) 2024-05-27T07:29:35.456 INFO:journalctl@ceph.mds.j.smithi052.stdout:May 27 07:29:35 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-j[72721]: 2024-05-27T07:29:35.456 INFO:journalctl@ceph.mds.j.smithi052.stdout:May 27 07:29:35 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-j[72721]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:29:35.456 INFO:journalctl@ceph.mds.j.smithi052.stdout:May 27 07:29:35 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-j[72721]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x127) [0x7fa5142edba6] 2024-05-27T07:29:35.456 INFO:journalctl@ceph.mds.j.smithi052.stdout:May 27 07:29:35 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-j[72721]: 2: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7fa5142eddd1] 2024-05-27T07:29:35.457 INFO:journalctl@ceph.mds.j.smithi052.stdout:May 27 07:29:35 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-j[72721]: 3: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x55b5c75b823d] 2024-05-27T07:29:35.457 INFO:journalctl@ceph.mds.j.smithi052.stdout:May 27 07:29:35 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-j[72721]: 4: (SnapRealm::build_snap_set() const+0x323) [0x55b5c75a3971] 2024-05-27T07:29:35.457 INFO:journalctl@ceph.mds.j.smithi052.stdout:May 27 07:29:35 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-j[72721]: 5: (SnapRealm::check_cache() const+0x13e) [0x55b5c75a1fa0] 2024-05-27T07:29:35.457 INFO:journalctl@ceph.mds.j.smithi052.stdout:May 27 07:29:35 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-j[72721]: 6: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x55b5c75a2651] 2024-05-27T07:29:35.457 INFO:journalctl@ceph.mds.j.smithi052.stdout:May 27 07:29:35 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-j[72721]: 7: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x55b5c7537347] 2024-05-27T07:29:35.457 INFO:journalctl@ceph.mds.j.smithi052.stdout:May 27 07:29:35 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-j[72721]: 8: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x55b5c723387e] 2024-05-27T07:29:35.457 INFO:journalctl@ceph.mds.j.smithi052.stdout:May 27 07:29:35 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-j[72721]: 9: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x2702) [0x55b5c7238e94] 2024-05-27T07:29:35.457 INFO:journalctl@ceph.mds.j.smithi052.stdout:May 27 07:29:35 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-j[72721]: 10: (EUpdate::replay(MDSRank*)+0x63) [0x55b5c7243969] 2024-05-27T07:29:35.457 INFO:journalctl@ceph.mds.j.smithi052.stdout:May 27 07:29:35 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-j[72721]: 11: (MDLog::_replay_thread()+0x1f38) [0x55b5c75dc48c] 2024-05-27T07:29:35.457 INFO:journalctl@ceph.mds.j.smithi052.stdout:May 27 07:29:35 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-j[72721]: 12: (MDLog::ReplayThread::entry()+0x11) [0x55b5c75e59cb] 2024-05-27T07:29:35.457 INFO:journalctl@ceph.mds.j.smithi052.stdout:May 27 07:29:35 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-j[72721]: 13: (Thread::entry_wrapper()+0x43) [0x7fa5142bc72d] 2024-05-27T07:29:35.457 INFO:journalctl@ceph.mds.j.smithi052.stdout:May 27 07:29:35 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-j[72721]: 14: (Thread::_entry_func(void*)+0xd) [0x7fa5142bc749] 2024-05-27T07:29:35.457 INFO:journalctl@ceph.mds.j.smithi052.stdout:May 27 07:29:35 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-j[72721]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7fa512e601ca] 2024-05-27T07:29:35.458 INFO:journalctl@ceph.mds.j.smithi052.stdout:May 27 07:29:35 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-j[72721]: 16: clone() 2024-05-27T07:29:35.458 INFO:journalctl@ceph.mds.j.smithi052.stdout:May 27 07:29:35 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-j[72721]: 2024-05-27T07:29:35.458 INFO:journalctl@ceph.mds.j.smithi052.stdout:May 27 07:29:35 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-j[72721]: 0> 2024-05-27T07:29:35.248+0000 7fa501aa8700 -1 *** Caught signal (Aborted) ** 2024-05-27T07:29:35.458 INFO:journalctl@ceph.mds.j.smithi052.stdout:May 27 07:29:35 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-j[72721]: in thread 7fa501aa8700 thread_name:md_log_replay 2024-05-27T07:29:35.458 INFO:journalctl@ceph.mds.j.smithi052.stdout:May 27 07:29:35 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-j[72721]: 2024-05-27T07:29:35.458 INFO:journalctl@ceph.mds.j.smithi052.stdout:May 27 07:29:35 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-j[72721]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:29:35.458 INFO:journalctl@ceph.mds.j.smithi052.stdout:May 27 07:29:35 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-j[72721]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7fa512e6ad20] 2024-05-27T07:29:35.458 INFO:journalctl@ceph.mds.j.smithi052.stdout:May 27 07:29:35 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-j[72721]: 2: gsignal() 2024-05-27T07:29:35.458 INFO:journalctl@ceph.mds.j.smithi052.stdout:May 27 07:29:35 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-j[72721]: 3: abort() 2024-05-27T07:29:35.458 INFO:journalctl@ceph.mds.j.smithi052.stdout:May 27 07:29:35 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-j[72721]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x26e) [0x7fa5142edced] 2024-05-27T07:29:35.458 INFO:journalctl@ceph.mds.j.smithi052.stdout:May 27 07:29:35 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-j[72721]: 5: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7fa5142eddd1] 2024-05-27T07:29:35.458 INFO:journalctl@ceph.mds.j.smithi052.stdout:May 27 07:29:35 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-j[72721]: 6: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x55b5c75b823d] 2024-05-27T07:29:35.458 INFO:journalctl@ceph.mds.j.smithi052.stdout:May 27 07:29:35 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-j[72721]: 7: (SnapRealm::build_snap_set() const+0x323) [0x55b5c75a3971] 2024-05-27T07:29:35.459 INFO:journalctl@ceph.mds.j.smithi052.stdout:May 27 07:29:35 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-j[72721]: 8: (SnapRealm::check_cache() const+0x13e) [0x55b5c75a1fa0] 2024-05-27T07:29:35.459 INFO:journalctl@ceph.mds.j.smithi052.stdout:May 27 07:29:35 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-j[72721]: 9: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x55b5c75a2651] 2024-05-27T07:29:35.459 INFO:journalctl@ceph.mds.j.smithi052.stdout:May 27 07:29:35 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-j[72721]: 10: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x55b5c7537347] 2024-05-27T07:29:35.459 INFO:journalctl@ceph.mds.j.smithi052.stdout:May 27 07:29:35 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-j[72721]: 11: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x55b5c723387e] 2024-05-27T07:29:35.459 INFO:journalctl@ceph.mds.j.smithi052.stdout:May 27 07:29:35 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-j[72721]: 12: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x2702) [0x55b5c7238e94] 2024-05-27T07:29:35.459 INFO:journalctl@ceph.mds.j.smithi052.stdout:May 27 07:29:35 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-j[72721]: 13: (EUpdate::replay(MDSRank*)+0x63) [0x55b5c7243969] 2024-05-27T07:29:35.459 INFO:journalctl@ceph.mds.j.smithi052.stdout:May 27 07:29:35 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-j[72721]: 14: (MDLog::_replay_thread()+0x1f38) [0x55b5c75dc48c] 2024-05-27T07:29:35.459 INFO:journalctl@ceph.mds.j.smithi052.stdout:May 27 07:29:35 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-j[72721]: 15: (MDLog::ReplayThread::entry()+0x11) [0x55b5c75e59cb] 2024-05-27T07:29:35.459 INFO:journalctl@ceph.mds.j.smithi052.stdout:May 27 07:29:35 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-j[72721]: 16: (Thread::entry_wrapper()+0x43) [0x7fa5142bc72d] 2024-05-27T07:29:35.459 INFO:journalctl@ceph.mds.j.smithi052.stdout:May 27 07:29:35 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-j[72721]: 17: (Thread::_entry_func(void*)+0xd) [0x7fa5142bc749] 2024-05-27T07:29:35.459 INFO:journalctl@ceph.mds.j.smithi052.stdout:May 27 07:29:35 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-j[72721]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7fa512e601ca] 2024-05-27T07:29:35.459 INFO:journalctl@ceph.mds.j.smithi052.stdout:May 27 07:29:35 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-j[72721]: 19: clone() 2024-05-27T07:29:35.459 INFO:journalctl@ceph.mds.j.smithi052.stdout:May 27 07:29:35 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-j[72721]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-05-27T07:29:35.460 INFO:journalctl@ceph.mds.j.smithi052.stdout:May 27 07:29:35 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-j[72721]: 2024-05-27T07:29:35.730 INFO:journalctl@ceph.mds.g.smithi052.stdout:May 27 07:29:35 smithi052 podman[105353]: 2024-05-27 07:29:35.45399793 +0000 UTC m=+0.875303535 container died 107b05d83164e934771756afdb36f84df595611dba00940359c933ea5ddee931 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-j, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, org.label-schema.build-date=20240521, org.label-schema.license=GPLv2, GIT_CLEAN=True, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, ceph=True, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-05-27T07:29:36.105 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:29:36 smithi052 ceph-mon[31955]: pgmap v126: 129 pgs: 129 active+clean; 213 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 5.7 MiB/s rd, 6.0 MiB/s wr, 1.53k op/s 2024-05-27T07:29:36.283 DEBUG:teuthology.orchestra.run.smithi052:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:38f9d900c83c63799fbdbe61acc9a11b0d3554a6 shell --fsid 73df2636-1bf8-11ef-bc9b-c7b262605968 -- ceph tell mds.1:4 damage ls 2024-05-27T07:29:36.394 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:29:36 smithi173 ceph-mon[37000]: pgmap v126: 129 pgs: 129 active+clean; 213 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 5.7 MiB/s rd, 6.0 MiB/s wr, 1.53k op/s 2024-05-27T07:29:36.440 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:29:36 smithi105 ceph-mon[35977]: pgmap v126: 129 pgs: 129 active+clean; 213 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 5.7 MiB/s rd, 6.0 MiB/s wr, 1.53k op/s 2024-05-27T07:29:36.873 INFO:teuthology.orchestra.run.smithi052.stderr:Inferring config /var/lib/ceph/73df2636-1bf8-11ef-bc9b-c7b262605968/mon.a/config 2024-05-27T07:29:37.024 INFO:journalctl@ceph.mds.d.smithi052.stdout:May 27 07:29:36 smithi052 systemd[1]: ceph-73df2636-1bf8-11ef-bc9b-c7b262605968@mds.d.service: Failed with result 'exit-code'. 2024-05-27T07:29:37.025 INFO:journalctl@ceph.mds.g.smithi052.stdout:May 27 07:29:36 smithi052 systemd[1]: ceph-73df2636-1bf8-11ef-bc9b-c7b262605968@mds.g.service: Failed with result 'exit-code'. 2024-05-27T07:29:37.025 INFO:journalctl@ceph.mds.g.smithi052.stdout:May 27 07:29:36 smithi052 systemd[1]: ceph-73df2636-1bf8-11ef-bc9b-c7b262605968@mds.g.service: Consumed 1.028s CPU time. 2024-05-27T07:29:37.025 INFO:journalctl@ceph.mds.j.smithi052.stdout:May 27 07:29:36 smithi052 podman[105390]: 2024-05-27 07:29:36.775509931 +0000 UTC m=+1.481120296 container remove 107b05d83164e934771756afdb36f84df595611dba00940359c933ea5ddee931 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-j, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, GIT_CLEAN=True, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240521, GIT_BRANCH=HEAD, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0) 2024-05-27T07:29:37.025 INFO:journalctl@ceph.mds.j.smithi052.stdout:May 27 07:29:36 smithi052 systemd[1]: ceph-73df2636-1bf8-11ef-bc9b-c7b262605968@mds.j.service: Main process exited, code=exited, status=134/n/a 2024-05-27T07:29:37.352 INFO:journalctl@ceph.mds.l.smithi173.stdout:May 27 07:29:37 smithi173 systemd[1]: ceph-73df2636-1bf8-11ef-bc9b-c7b262605968@mds.l.service: Scheduled restart job, restart counter is at 3. 2024-05-27T07:29:37.353 INFO:journalctl@ceph.mds.l.smithi173.stdout:May 27 07:29:37 smithi173 systemd[1]: Stopped Ceph mds.l for 73df2636-1bf8-11ef-bc9b-c7b262605968. 2024-05-27T07:29:37.353 INFO:journalctl@ceph.mds.l.smithi173.stdout:May 27 07:29:37 smithi173 systemd[1]: Starting Ceph mds.l for 73df2636-1bf8-11ef-bc9b-c7b262605968... 2024-05-27T07:29:37.353 INFO:journalctl@ceph.mds.l.smithi173.stdout:May 27 07:29:37 smithi173 podman[73584]: 2024-05-27 07:29:37.237696881 +0000 UTC m=+0.017788269 image pull e36ea4fc8a5807e8d2701370e350a0de3856538797f7fba64cfce7e344bd6799 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6 2024-05-27T07:29:37.353 INFO:journalctl@ceph.mds.l.smithi173.stdout:May 27 07:29:37 smithi173 podman[73584]: 2024-05-27 07:29:37.341823972 +0000 UTC m=+0.121915325 container create b569910ccacdea0c5f88cd2446dd19414cf2167c1bfcea2209be3f26c4bb9c0f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-l, GIT_CLEAN=True, org.label-schema.license=GPLv2, ceph=True, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, org.label-schema.build-date=20240521, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0) 2024-05-27T07:29:37.440 INFO:journalctl@ceph.mds.k.smithi105.stdout:May 27 07:29:36 smithi105 systemd[1]: ceph-73df2636-1bf8-11ef-bc9b-c7b262605968@mds.k.service: Scheduled restart job, restart counter is at 2. 2024-05-27T07:29:37.440 INFO:journalctl@ceph.mds.k.smithi105.stdout:May 27 07:29:36 smithi105 systemd[1]: Stopped Ceph mds.k for 73df2636-1bf8-11ef-bc9b-c7b262605968. 2024-05-27T07:29:37.440 INFO:journalctl@ceph.mds.k.smithi105.stdout:May 27 07:29:36 smithi105 systemd[1]: Starting Ceph mds.k for 73df2636-1bf8-11ef-bc9b-c7b262605968... 2024-05-27T07:29:37.441 INFO:journalctl@ceph.mds.k.smithi105.stdout:May 27 07:29:37 smithi105 podman[69824]: 2024-05-27 07:29:37.07768686 +0000 UTC m=+0.017306194 image pull e36ea4fc8a5807e8d2701370e350a0de3856538797f7fba64cfce7e344bd6799 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6 2024-05-27T07:29:37.852 INFO:journalctl@ceph.mds.l.smithi173.stdout:May 27 07:29:37 smithi173 podman[73584]: 2024-05-27 07:29:37.603475194 +0000 UTC m=+0.383566554 container init b569910ccacdea0c5f88cd2446dd19414cf2167c1bfcea2209be3f26c4bb9c0f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-l, ceph=True, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, GIT_CLEAN=True, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, org.label-schema.build-date=20240521) 2024-05-27T07:29:37.852 INFO:journalctl@ceph.mds.l.smithi173.stdout:May 27 07:29:37 smithi173 podman[73584]: 2024-05-27 07:29:37.606379442 +0000 UTC m=+0.386470815 container start b569910ccacdea0c5f88cd2446dd19414cf2167c1bfcea2209be3f26c4bb9c0f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-l, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, GIT_CLEAN=True, ceph=True, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, org.label-schema.build-date=20240521, maintainer=Guillaume Abrioux ) 2024-05-27T07:29:37.853 INFO:journalctl@ceph.mds.l.smithi173.stdout:May 27 07:29:37 smithi173 ceph-mds[73605]: -- 172.21.15.173:0/2631933570 <== mon.1 v2:172.21.15.173:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x5645f15e3a40 con 0x5645f162b600 2024-05-27T07:29:37.853 INFO:journalctl@ceph.mds.l.smithi173.stdout:May 27 07:29:37 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-l[73601]: starting mds.l at 2024-05-27T07:29:37.853 INFO:journalctl@ceph.mds.l.smithi173.stdout:May 27 07:29:37 smithi173 bash[73584]: b569910ccacdea0c5f88cd2446dd19414cf2167c1bfcea2209be3f26c4bb9c0f 2024-05-27T07:29:37.853 INFO:journalctl@ceph.mds.l.smithi173.stdout:May 27 07:29:37 smithi173 systemd[1]: Started Ceph mds.l for 73df2636-1bf8-11ef-bc9b-c7b262605968. 2024-05-27T07:29:38.367 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:29:38 smithi105 ceph-mon[35977]: pgmap v127: 129 pgs: 129 active+clean; 213 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 MiB/s rd, 4.4 MiB/s wr, 1.32k op/s 2024-05-27T07:29:38.367 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:29:38 smithi105 ceph-mon[35977]: daemon mds.l restarted 2024-05-27T07:29:38.367 INFO:journalctl@ceph.mds.k.smithi105.stdout:May 27 07:29:38 smithi105 podman[69824]: 2024-05-27 07:29:38.089871982 +0000 UTC m=+1.029491295 container create 08bbfdf102e0774d695c3546b67fd704eb12702e615eb396bd27fb68becc408e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-k, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_CLEAN=True, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, maintainer=Guillaume Abrioux , RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, org.label-schema.build-date=20240521, io.buildah.version=1.33.5, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2) 2024-05-27T07:29:38.367 INFO:journalctl@ceph.mds.k.smithi105.stdout:May 27 07:29:38 smithi105 podman[69824]: 2024-05-27 07:29:38.334772552 +0000 UTC m=+1.274391860 container init 08bbfdf102e0774d695c3546b67fd704eb12702e615eb396bd27fb68becc408e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-k, org.label-schema.build-date=20240521, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, GIT_CLEAN=True, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, ceph=True, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=) 2024-05-27T07:29:38.367 INFO:journalctl@ceph.mds.k.smithi105.stdout:May 27 07:29:38 smithi105 podman[69824]: 2024-05-27 07:29:38.338149508 +0000 UTC m=+1.277768830 container start 08bbfdf102e0774d695c3546b67fd704eb12702e615eb396bd27fb68becc408e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-k, org.label-schema.license=GPLv2, org.label-schema.build-date=20240521, GIT_CLEAN=True, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image) 2024-05-27T07:29:38.479 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:29:38 smithi052 ceph-mon[31955]: pgmap v127: 129 pgs: 129 active+clean; 213 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 MiB/s rd, 4.4 MiB/s wr, 1.32k op/s 2024-05-27T07:29:38.479 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:29:38 smithi052 ceph-mon[31955]: daemon mds.l restarted 2024-05-27T07:29:38.514 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:29:38 smithi173 ceph-mon[37000]: pgmap v127: 129 pgs: 129 active+clean; 213 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 MiB/s rd, 4.4 MiB/s wr, 1.32k op/s 2024-05-27T07:29:38.514 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:29:38 smithi173 ceph-mon[37000]: daemon mds.l restarted 2024-05-27T07:29:38.514 INFO:journalctl@ceph.mds.l.smithi173.stdout:May 27 07:29:38 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-l[73601]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: In function 'std::set SnapClient::filter(const std::set&) const' thread 7f396c15c700 time 2024-05-27T07:29:38.185282+0000 2024-05-27T07:29:38.514 INFO:journalctl@ceph.mds.l.smithi173.stdout:May 27 07:29:38 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-l[73601]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: 197: FAILED ceph_assert(cached_version > 0) 2024-05-27T07:29:38.514 INFO:journalctl@ceph.mds.l.smithi173.stdout:May 27 07:29:38 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-l[73601]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:29:38.514 INFO:journalctl@ceph.mds.l.smithi173.stdout:May 27 07:29:38 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-l[73601]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x127) [0x7f397e9a1ba6] 2024-05-27T07:29:38.515 INFO:journalctl@ceph.mds.l.smithi173.stdout:May 27 07:29:38 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-l[73601]: 2: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7f397e9a1dd1] 2024-05-27T07:29:38.515 INFO:journalctl@ceph.mds.l.smithi173.stdout:May 27 07:29:38 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-l[73601]: 3: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x5645f012e23d] 2024-05-27T07:29:38.515 INFO:journalctl@ceph.mds.l.smithi173.stdout:May 27 07:29:38 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-l[73601]: 4: (SnapRealm::build_snap_set() const+0x323) [0x5645f0119971] 2024-05-27T07:29:38.515 INFO:journalctl@ceph.mds.l.smithi173.stdout:May 27 07:29:38 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-l[73601]: 5: (SnapRealm::check_cache() const+0x13e) [0x5645f0117fa0] 2024-05-27T07:29:38.515 INFO:journalctl@ceph.mds.l.smithi173.stdout:May 27 07:29:38 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-l[73601]: 6: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x5645f0118651] 2024-05-27T07:29:38.515 INFO:journalctl@ceph.mds.l.smithi173.stdout:May 27 07:29:38 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-l[73601]: 7: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x5645f00ad347] 2024-05-27T07:29:38.515 INFO:journalctl@ceph.mds.l.smithi173.stdout:May 27 07:29:38 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-l[73601]: 8: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x5645efda987e] 2024-05-27T07:29:38.515 INFO:journalctl@ceph.mds.l.smithi173.stdout:May 27 07:29:38 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-l[73601]: 9: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x2702) [0x5645efdaee94] 2024-05-27T07:29:38.515 INFO:journalctl@ceph.mds.l.smithi173.stdout:May 27 07:29:38 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-l[73601]: 10: (EUpdate::replay(MDSRank*)+0x63) [0x5645efdb9969] 2024-05-27T07:29:38.515 INFO:journalctl@ceph.mds.l.smithi173.stdout:May 27 07:29:38 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-l[73601]: 11: (MDLog::_replay_thread()+0x1f38) [0x5645f015248c] 2024-05-27T07:29:38.515 INFO:journalctl@ceph.mds.l.smithi173.stdout:May 27 07:29:38 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-l[73601]: 12: (MDLog::ReplayThread::entry()+0x11) [0x5645f015b9cb] 2024-05-27T07:29:38.515 INFO:journalctl@ceph.mds.l.smithi173.stdout:May 27 07:29:38 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-l[73601]: 13: (Thread::entry_wrapper()+0x43) [0x7f397e97072d] 2024-05-27T07:29:38.515 INFO:journalctl@ceph.mds.l.smithi173.stdout:May 27 07:29:38 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-l[73601]: 14: (Thread::_entry_func(void*)+0xd) [0x7f397e970749] 2024-05-27T07:29:38.516 INFO:journalctl@ceph.mds.l.smithi173.stdout:May 27 07:29:38 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-l[73601]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7f397d5141ca] 2024-05-27T07:29:38.516 INFO:journalctl@ceph.mds.l.smithi173.stdout:May 27 07:29:38 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-l[73601]: 16: clone() 2024-05-27T07:29:38.516 INFO:journalctl@ceph.mds.l.smithi173.stdout:May 27 07:29:38 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-l[73601]: 2024-05-27T07:29:38.186+0000 7f396c15c700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: In function 'std::set SnapClient::filter(const std::set&) const' thread 7f396c15c700 time 2024-05-27T07:29:38.185282+0000 2024-05-27T07:29:38.516 INFO:journalctl@ceph.mds.l.smithi173.stdout:May 27 07:29:38 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-l[73601]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: 197: FAILED ceph_assert(cached_version > 0) 2024-05-27T07:29:38.516 INFO:journalctl@ceph.mds.l.smithi173.stdout:May 27 07:29:38 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-l[73601]: 2024-05-27T07:29:38.516 INFO:journalctl@ceph.mds.l.smithi173.stdout:May 27 07:29:38 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-l[73601]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:29:38.517 INFO:journalctl@ceph.mds.l.smithi173.stdout:May 27 07:29:38 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-l[73601]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x127) [0x7f397e9a1ba6] 2024-05-27T07:29:38.517 INFO:journalctl@ceph.mds.l.smithi173.stdout:May 27 07:29:38 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-l[73601]: 2: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7f397e9a1dd1] 2024-05-27T07:29:38.517 INFO:journalctl@ceph.mds.l.smithi173.stdout:May 27 07:29:38 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-l[73601]: 3: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x5645f012e23d] 2024-05-27T07:29:38.517 INFO:journalctl@ceph.mds.l.smithi173.stdout:May 27 07:29:38 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-l[73601]: 4: (SnapRealm::build_snap_set() const+0x323) [0x5645f0119971] 2024-05-27T07:29:38.517 INFO:journalctl@ceph.mds.l.smithi173.stdout:May 27 07:29:38 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-l[73601]: 5: (SnapRealm::check_cache() const+0x13e) [0x5645f0117fa0] 2024-05-27T07:29:38.517 INFO:journalctl@ceph.mds.l.smithi173.stdout:May 27 07:29:38 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-l[73601]: 6: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x5645f0118651] 2024-05-27T07:29:38.517 INFO:journalctl@ceph.mds.l.smithi173.stdout:May 27 07:29:38 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-l[73601]: 7: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x5645f00ad347] 2024-05-27T07:29:38.517 INFO:journalctl@ceph.mds.l.smithi173.stdout:May 27 07:29:38 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-l[73601]: 8: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x5645efda987e] 2024-05-27T07:29:38.517 INFO:journalctl@ceph.mds.l.smithi173.stdout:May 27 07:29:38 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-l[73601]: 9: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x2702) [0x5645efdaee94] 2024-05-27T07:29:38.517 INFO:journalctl@ceph.mds.l.smithi173.stdout:May 27 07:29:38 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-l[73601]: 10: (EUpdate::replay(MDSRank*)+0x63) [0x5645efdb9969] 2024-05-27T07:29:38.517 INFO:journalctl@ceph.mds.l.smithi173.stdout:May 27 07:29:38 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-l[73601]: 11: (MDLog::_replay_thread()+0x1f38) [0x5645f015248c] 2024-05-27T07:29:38.517 INFO:journalctl@ceph.mds.l.smithi173.stdout:May 27 07:29:38 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-l[73601]: 12: (MDLog::ReplayThread::entry()+0x11) [0x5645f015b9cb] 2024-05-27T07:29:38.517 INFO:journalctl@ceph.mds.l.smithi173.stdout:May 27 07:29:38 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-l[73601]: 13: (Thread::entry_wrapper()+0x43) [0x7f397e97072d] 2024-05-27T07:29:38.518 INFO:journalctl@ceph.mds.l.smithi173.stdout:May 27 07:29:38 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-l[73601]: 14: (Thread::_entry_func(void*)+0xd) [0x7f397e970749] 2024-05-27T07:29:38.518 INFO:journalctl@ceph.mds.l.smithi173.stdout:May 27 07:29:38 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-l[73601]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7f397d5141ca] 2024-05-27T07:29:38.518 INFO:journalctl@ceph.mds.l.smithi173.stdout:May 27 07:29:38 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-l[73601]: 16: clone() 2024-05-27T07:29:38.518 INFO:journalctl@ceph.mds.l.smithi173.stdout:May 27 07:29:38 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-l[73601]: 2024-05-27T07:29:38.518 INFO:journalctl@ceph.mds.l.smithi173.stdout:May 27 07:29:38 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-l[73601]: *** Caught signal (Aborted) ** 2024-05-27T07:29:38.518 INFO:journalctl@ceph.mds.l.smithi173.stdout:May 27 07:29:38 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-l[73601]: in thread 7f396c15c700 thread_name:md_log_replay 2024-05-27T07:29:38.518 INFO:journalctl@ceph.mds.l.smithi173.stdout:May 27 07:29:38 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-l[73601]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:29:38.518 INFO:journalctl@ceph.mds.l.smithi173.stdout:May 27 07:29:38 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-l[73601]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f397d51ed20] 2024-05-27T07:29:38.518 INFO:journalctl@ceph.mds.l.smithi173.stdout:May 27 07:29:38 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-l[73601]: 2: gsignal() 2024-05-27T07:29:38.518 INFO:journalctl@ceph.mds.l.smithi173.stdout:May 27 07:29:38 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-l[73601]: 3: abort() 2024-05-27T07:29:38.518 INFO:journalctl@ceph.mds.l.smithi173.stdout:May 27 07:29:38 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-l[73601]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x26e) [0x7f397e9a1ced] 2024-05-27T07:29:38.518 INFO:journalctl@ceph.mds.l.smithi173.stdout:May 27 07:29:38 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-l[73601]: 5: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7f397e9a1dd1] 2024-05-27T07:29:38.518 INFO:journalctl@ceph.mds.l.smithi173.stdout:May 27 07:29:38 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-l[73601]: 6: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x5645f012e23d] 2024-05-27T07:29:38.518 INFO:journalctl@ceph.mds.l.smithi173.stdout:May 27 07:29:38 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-l[73601]: 7: (SnapRealm::build_snap_set() const+0x323) [0x5645f0119971] 2024-05-27T07:29:38.519 INFO:journalctl@ceph.mds.l.smithi173.stdout:May 27 07:29:38 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-l[73601]: 8: (SnapRealm::check_cache() const+0x13e) [0x5645f0117fa0] 2024-05-27T07:29:38.519 INFO:journalctl@ceph.mds.l.smithi173.stdout:May 27 07:29:38 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-l[73601]: 9: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x5645f0118651] 2024-05-27T07:29:38.519 INFO:journalctl@ceph.mds.l.smithi173.stdout:May 27 07:29:38 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-l[73601]: 10: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x5645f00ad347] 2024-05-27T07:29:38.519 INFO:journalctl@ceph.mds.l.smithi173.stdout:May 27 07:29:38 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-l[73601]: 11: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x5645efda987e] 2024-05-27T07:29:38.519 INFO:journalctl@ceph.mds.l.smithi173.stdout:May 27 07:29:38 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-l[73601]: 12: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x2702) [0x5645efdaee94] 2024-05-27T07:29:38.519 INFO:journalctl@ceph.mds.l.smithi173.stdout:May 27 07:29:38 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-l[73601]: 13: (EUpdate::replay(MDSRank*)+0x63) [0x5645efdb9969] 2024-05-27T07:29:38.519 INFO:journalctl@ceph.mds.l.smithi173.stdout:May 27 07:29:38 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-l[73601]: 14: (MDLog::_replay_thread()+0x1f38) [0x5645f015248c] 2024-05-27T07:29:38.519 INFO:journalctl@ceph.mds.l.smithi173.stdout:May 27 07:29:38 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-l[73601]: 15: (MDLog::ReplayThread::entry()+0x11) [0x5645f015b9cb] 2024-05-27T07:29:38.519 INFO:journalctl@ceph.mds.l.smithi173.stdout:May 27 07:29:38 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-l[73601]: 16: (Thread::entry_wrapper()+0x43) [0x7f397e97072d] 2024-05-27T07:29:38.519 INFO:journalctl@ceph.mds.l.smithi173.stdout:May 27 07:29:38 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-l[73601]: 17: (Thread::_entry_func(void*)+0xd) [0x7f397e970749] 2024-05-27T07:29:38.519 INFO:journalctl@ceph.mds.l.smithi173.stdout:May 27 07:29:38 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-l[73601]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7f397d5141ca] 2024-05-27T07:29:38.519 INFO:journalctl@ceph.mds.l.smithi173.stdout:May 27 07:29:38 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-l[73601]: 19: clone() 2024-05-27T07:29:38.519 INFO:journalctl@ceph.mds.l.smithi173.stdout:May 27 07:29:38 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-l[73601]: 2024-05-27T07:29:38.188+0000 7f396c15c700 -1 *** Caught signal (Aborted) ** 2024-05-27T07:29:38.519 INFO:journalctl@ceph.mds.l.smithi173.stdout:May 27 07:29:38 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-l[73601]: in thread 7f396c15c700 thread_name:md_log_replay 2024-05-27T07:29:38.520 INFO:journalctl@ceph.mds.l.smithi173.stdout:May 27 07:29:38 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-l[73601]: 2024-05-27T07:29:38.520 INFO:journalctl@ceph.mds.l.smithi173.stdout:May 27 07:29:38 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-l[73601]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:29:38.520 INFO:journalctl@ceph.mds.l.smithi173.stdout:May 27 07:29:38 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-l[73601]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f397d51ed20] 2024-05-27T07:29:38.520 INFO:journalctl@ceph.mds.l.smithi173.stdout:May 27 07:29:38 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-l[73601]: 2: gsignal() 2024-05-27T07:29:38.520 INFO:journalctl@ceph.mds.l.smithi173.stdout:May 27 07:29:38 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-l[73601]: 3: abort() 2024-05-27T07:29:38.520 INFO:journalctl@ceph.mds.l.smithi173.stdout:May 27 07:29:38 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-l[73601]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x26e) [0x7f397e9a1ced] 2024-05-27T07:29:38.520 INFO:journalctl@ceph.mds.l.smithi173.stdout:May 27 07:29:38 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-l[73601]: 5: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7f397e9a1dd1] 2024-05-27T07:29:38.520 INFO:journalctl@ceph.mds.l.smithi173.stdout:May 27 07:29:38 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-l[73601]: 6: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x5645f012e23d] 2024-05-27T07:29:38.520 INFO:journalctl@ceph.mds.l.smithi173.stdout:May 27 07:29:38 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-l[73601]: 7: (SnapRealm::build_snap_set() const+0x323) [0x5645f0119971] 2024-05-27T07:29:38.520 INFO:journalctl@ceph.mds.l.smithi173.stdout:May 27 07:29:38 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-l[73601]: 8: (SnapRealm::check_cache() const+0x13e) [0x5645f0117fa0] 2024-05-27T07:29:38.520 INFO:journalctl@ceph.mds.l.smithi173.stdout:May 27 07:29:38 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-l[73601]: 9: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x5645f0118651] 2024-05-27T07:29:38.520 INFO:journalctl@ceph.mds.l.smithi173.stdout:May 27 07:29:38 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-l[73601]: 10: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x5645f00ad347] 2024-05-27T07:29:38.520 INFO:journalctl@ceph.mds.l.smithi173.stdout:May 27 07:29:38 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-l[73601]: 11: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x5645efda987e] 2024-05-27T07:29:38.520 INFO:journalctl@ceph.mds.l.smithi173.stdout:May 27 07:29:38 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-l[73601]: 12: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x2702) [0x5645efdaee94] 2024-05-27T07:29:38.520 INFO:journalctl@ceph.mds.l.smithi173.stdout:May 27 07:29:38 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-l[73601]: 13: (EUpdate::replay(MDSRank*)+0x63) [0x5645efdb9969] 2024-05-27T07:29:38.521 INFO:journalctl@ceph.mds.l.smithi173.stdout:May 27 07:29:38 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-l[73601]: 14: (MDLog::_replay_thread()+0x1f38) [0x5645f015248c] 2024-05-27T07:29:38.521 INFO:journalctl@ceph.mds.l.smithi173.stdout:May 27 07:29:38 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-l[73601]: 15: (MDLog::ReplayThread::entry()+0x11) [0x5645f015b9cb] 2024-05-27T07:29:38.521 INFO:journalctl@ceph.mds.l.smithi173.stdout:May 27 07:29:38 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-l[73601]: 16: (Thread::entry_wrapper()+0x43) [0x7f397e97072d] 2024-05-27T07:29:38.521 INFO:journalctl@ceph.mds.l.smithi173.stdout:May 27 07:29:38 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-l[73601]: 17: (Thread::_entry_func(void*)+0xd) [0x7f397e970749] 2024-05-27T07:29:38.521 INFO:journalctl@ceph.mds.l.smithi173.stdout:May 27 07:29:38 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-l[73601]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7f397d5141ca] 2024-05-27T07:29:38.521 INFO:journalctl@ceph.mds.l.smithi173.stdout:May 27 07:29:38 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-l[73601]: 19: clone() 2024-05-27T07:29:38.521 INFO:journalctl@ceph.mds.l.smithi173.stdout:May 27 07:29:38 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-l[73601]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-05-27T07:29:38.521 INFO:journalctl@ceph.mds.l.smithi173.stdout:May 27 07:29:38 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-l[73601]: 2024-05-27T07:29:38.521 INFO:journalctl@ceph.mds.l.smithi173.stdout:May 27 07:29:38 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-l[73601]: -1> 2024-05-27T07:29:38.186+0000 7f396c15c700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: In function 'std::set SnapClient::filter(const std::set&) const' thread 7f396c15c700 time 2024-05-27T07:29:38.185282+0000 2024-05-27T07:29:38.521 INFO:journalctl@ceph.mds.l.smithi173.stdout:May 27 07:29:38 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-l[73601]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: 197: FAILED ceph_assert(cached_version > 0) 2024-05-27T07:29:38.521 INFO:journalctl@ceph.mds.l.smithi173.stdout:May 27 07:29:38 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-l[73601]: 2024-05-27T07:29:38.521 INFO:journalctl@ceph.mds.l.smithi173.stdout:May 27 07:29:38 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-l[73601]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:29:38.521 INFO:journalctl@ceph.mds.l.smithi173.stdout:May 27 07:29:38 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-l[73601]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x127) [0x7f397e9a1ba6] 2024-05-27T07:29:38.521 INFO:journalctl@ceph.mds.l.smithi173.stdout:May 27 07:29:38 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-l[73601]: 2: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7f397e9a1dd1] 2024-05-27T07:29:38.522 INFO:journalctl@ceph.mds.l.smithi173.stdout:May 27 07:29:38 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-l[73601]: 3: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x5645f012e23d] 2024-05-27T07:29:38.522 INFO:journalctl@ceph.mds.l.smithi173.stdout:May 27 07:29:38 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-l[73601]: 4: (SnapRealm::build_snap_set() const+0x323) [0x5645f0119971] 2024-05-27T07:29:38.522 INFO:journalctl@ceph.mds.l.smithi173.stdout:May 27 07:29:38 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-l[73601]: 5: (SnapRealm::check_cache() const+0x13e) [0x5645f0117fa0] 2024-05-27T07:29:38.522 INFO:journalctl@ceph.mds.l.smithi173.stdout:May 27 07:29:38 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-l[73601]: 6: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x5645f0118651] 2024-05-27T07:29:38.522 INFO:journalctl@ceph.mds.l.smithi173.stdout:May 27 07:29:38 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-l[73601]: 7: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x5645f00ad347] 2024-05-27T07:29:38.522 INFO:journalctl@ceph.mds.l.smithi173.stdout:May 27 07:29:38 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-l[73601]: 8: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x5645efda987e] 2024-05-27T07:29:38.522 INFO:journalctl@ceph.mds.l.smithi173.stdout:May 27 07:29:38 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-l[73601]: 9: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x2702) [0x5645efdaee94] 2024-05-27T07:29:38.522 INFO:journalctl@ceph.mds.l.smithi173.stdout:May 27 07:29:38 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-l[73601]: 10: (EUpdate::replay(MDSRank*)+0x63) [0x5645efdb9969] 2024-05-27T07:29:38.522 INFO:journalctl@ceph.mds.l.smithi173.stdout:May 27 07:29:38 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-l[73601]: 11: (MDLog::_replay_thread()+0x1f38) [0x5645f015248c] 2024-05-27T07:29:38.522 INFO:journalctl@ceph.mds.l.smithi173.stdout:May 27 07:29:38 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-l[73601]: 12: (MDLog::ReplayThread::entry()+0x11) [0x5645f015b9cb] 2024-05-27T07:29:38.522 INFO:journalctl@ceph.mds.l.smithi173.stdout:May 27 07:29:38 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-l[73601]: 13: (Thread::entry_wrapper()+0x43) [0x7f397e97072d] 2024-05-27T07:29:38.522 INFO:journalctl@ceph.mds.l.smithi173.stdout:May 27 07:29:38 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-l[73601]: 14: (Thread::_entry_func(void*)+0xd) [0x7f397e970749] 2024-05-27T07:29:38.523 INFO:journalctl@ceph.mds.l.smithi173.stdout:May 27 07:29:38 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-l[73601]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7f397d5141ca] 2024-05-27T07:29:38.523 INFO:journalctl@ceph.mds.l.smithi173.stdout:May 27 07:29:38 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-l[73601]: 16: clone() 2024-05-27T07:29:38.523 INFO:journalctl@ceph.mds.l.smithi173.stdout:May 27 07:29:38 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-l[73601]: 2024-05-27T07:29:38.523 INFO:journalctl@ceph.mds.l.smithi173.stdout:May 27 07:29:38 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-l[73601]: 0> 2024-05-27T07:29:38.188+0000 7f396c15c700 -1 *** Caught signal (Aborted) ** 2024-05-27T07:29:38.523 INFO:journalctl@ceph.mds.l.smithi173.stdout:May 27 07:29:38 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-l[73601]: in thread 7f396c15c700 thread_name:md_log_replay 2024-05-27T07:29:38.523 INFO:journalctl@ceph.mds.l.smithi173.stdout:May 27 07:29:38 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-l[73601]: 2024-05-27T07:29:38.523 INFO:journalctl@ceph.mds.l.smithi173.stdout:May 27 07:29:38 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-l[73601]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:29:38.523 INFO:journalctl@ceph.mds.l.smithi173.stdout:May 27 07:29:38 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-l[73601]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f397d51ed20] 2024-05-27T07:29:38.523 INFO:journalctl@ceph.mds.l.smithi173.stdout:May 27 07:29:38 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-l[73601]: 2: gsignal() 2024-05-27T07:29:38.523 INFO:journalctl@ceph.mds.l.smithi173.stdout:May 27 07:29:38 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-l[73601]: 3: abort() 2024-05-27T07:29:38.523 INFO:journalctl@ceph.mds.l.smithi173.stdout:May 27 07:29:38 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-l[73601]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x26e) [0x7f397e9a1ced] 2024-05-27T07:29:38.523 INFO:journalctl@ceph.mds.l.smithi173.stdout:May 27 07:29:38 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-l[73601]: 5: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7f397e9a1dd1] 2024-05-27T07:29:38.523 INFO:journalctl@ceph.mds.l.smithi173.stdout:May 27 07:29:38 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-l[73601]: 6: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x5645f012e23d] 2024-05-27T07:29:38.523 INFO:journalctl@ceph.mds.l.smithi173.stdout:May 27 07:29:38 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-l[73601]: 7: (SnapRealm::build_snap_set() const+0x323) [0x5645f0119971] 2024-05-27T07:29:38.523 INFO:journalctl@ceph.mds.l.smithi173.stdout:May 27 07:29:38 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-l[73601]: 8: (SnapRealm::check_cache() const+0x13e) [0x5645f0117fa0] 2024-05-27T07:29:38.524 INFO:journalctl@ceph.mds.l.smithi173.stdout:May 27 07:29:38 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-l[73601]: 9: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x5645f0118651] 2024-05-27T07:29:38.524 INFO:journalctl@ceph.mds.l.smithi173.stdout:May 27 07:29:38 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-l[73601]: 10: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x5645f00ad347] 2024-05-27T07:29:38.524 INFO:journalctl@ceph.mds.l.smithi173.stdout:May 27 07:29:38 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-l[73601]: 11: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x5645efda987e] 2024-05-27T07:29:38.524 INFO:journalctl@ceph.mds.l.smithi173.stdout:May 27 07:29:38 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-l[73601]: 12: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x2702) [0x5645efdaee94] 2024-05-27T07:29:38.524 INFO:journalctl@ceph.mds.l.smithi173.stdout:May 27 07:29:38 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-l[73601]: 13: (EUpdate::replay(MDSRank*)+0x63) [0x5645efdb9969] 2024-05-27T07:29:38.524 INFO:journalctl@ceph.mds.l.smithi173.stdout:May 27 07:29:38 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-l[73601]: 14: (MDLog::_replay_thread()+0x1f38) [0x5645f015248c] 2024-05-27T07:29:38.524 INFO:journalctl@ceph.mds.l.smithi173.stdout:May 27 07:29:38 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-l[73601]: 15: (MDLog::ReplayThread::entry()+0x11) [0x5645f015b9cb] 2024-05-27T07:29:38.524 INFO:journalctl@ceph.mds.l.smithi173.stdout:May 27 07:29:38 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-l[73601]: 16: (Thread::entry_wrapper()+0x43) [0x7f397e97072d] 2024-05-27T07:29:38.524 INFO:journalctl@ceph.mds.l.smithi173.stdout:May 27 07:29:38 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-l[73601]: 17: (Thread::_entry_func(void*)+0xd) [0x7f397e970749] 2024-05-27T07:29:38.525 INFO:journalctl@ceph.mds.l.smithi173.stdout:May 27 07:29:38 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-l[73601]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7f397d5141ca] 2024-05-27T07:29:38.525 INFO:journalctl@ceph.mds.l.smithi173.stdout:May 27 07:29:38 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-l[73601]: 19: clone() 2024-05-27T07:29:38.525 INFO:journalctl@ceph.mds.l.smithi173.stdout:May 27 07:29:38 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-l[73601]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-05-27T07:29:38.525 INFO:journalctl@ceph.mds.l.smithi173.stdout:May 27 07:29:38 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-l[73601]: 2024-05-27T07:29:38.525 INFO:journalctl@ceph.mds.l.smithi173.stdout:May 27 07:29:38 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-l[73601]: -1> 2024-05-27T07:29:38.186+0000 7f396c15c700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: In function 'std::set SnapClient::filter(const std::set&) const' thread 7f396c15c700 time 2024-05-27T07:29:38.185282+0000 2024-05-27T07:29:38.525 INFO:journalctl@ceph.mds.l.smithi173.stdout:May 27 07:29:38 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-l[73601]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: 197: FAILED ceph_assert(cached_version > 0) 2024-05-27T07:29:38.525 INFO:journalctl@ceph.mds.l.smithi173.stdout:May 27 07:29:38 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-l[73601]: 2024-05-27T07:29:38.525 INFO:journalctl@ceph.mds.l.smithi173.stdout:May 27 07:29:38 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-l[73601]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:29:38.525 INFO:journalctl@ceph.mds.l.smithi173.stdout:May 27 07:29:38 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-l[73601]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x127) [0x7f397e9a1ba6] 2024-05-27T07:29:38.525 INFO:journalctl@ceph.mds.l.smithi173.stdout:May 27 07:29:38 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-l[73601]: 2: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7f397e9a1dd1] 2024-05-27T07:29:38.525 INFO:journalctl@ceph.mds.l.smithi173.stdout:May 27 07:29:38 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-l[73601]: 3: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x5645f012e23d] 2024-05-27T07:29:38.525 INFO:journalctl@ceph.mds.l.smithi173.stdout:May 27 07:29:38 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-l[73601]: 4: (SnapRealm::build_snap_set() const+0x323) [0x5645f0119971] 2024-05-27T07:29:38.525 INFO:journalctl@ceph.mds.l.smithi173.stdout:May 27 07:29:38 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-l[73601]: 5: (SnapRealm::check_cache() const+0x13e) [0x5645f0117fa0] 2024-05-27T07:29:38.526 INFO:journalctl@ceph.mds.l.smithi173.stdout:May 27 07:29:38 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-l[73601]: 6: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x5645f0118651] 2024-05-27T07:29:38.526 INFO:journalctl@ceph.mds.l.smithi173.stdout:May 27 07:29:38 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-l[73601]: 7: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x5645f00ad347] 2024-05-27T07:29:38.526 INFO:journalctl@ceph.mds.l.smithi173.stdout:May 27 07:29:38 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-l[73601]: 8: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x5645efda987e] 2024-05-27T07:29:38.526 INFO:journalctl@ceph.mds.l.smithi173.stdout:May 27 07:29:38 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-l[73601]: 9: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x2702) [0x5645efdaee94] 2024-05-27T07:29:38.526 INFO:journalctl@ceph.mds.l.smithi173.stdout:May 27 07:29:38 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-l[73601]: 10: (EUpdate::replay(MDSRank*)+0x63) [0x5645efdb9969] 2024-05-27T07:29:38.526 INFO:journalctl@ceph.mds.l.smithi173.stdout:May 27 07:29:38 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-l[73601]: 11: (MDLog::_replay_thread()+0x1f38) [0x5645f015248c] 2024-05-27T07:29:38.526 INFO:journalctl@ceph.mds.l.smithi173.stdout:May 27 07:29:38 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-l[73601]: 12: (MDLog::ReplayThread::entry()+0x11) [0x5645f015b9cb] 2024-05-27T07:29:38.526 INFO:journalctl@ceph.mds.l.smithi173.stdout:May 27 07:29:38 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-l[73601]: 13: (Thread::entry_wrapper()+0x43) [0x7f397e97072d] 2024-05-27T07:29:38.526 INFO:journalctl@ceph.mds.l.smithi173.stdout:May 27 07:29:38 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-l[73601]: 14: (Thread::_entry_func(void*)+0xd) [0x7f397e970749] 2024-05-27T07:29:38.526 INFO:journalctl@ceph.mds.l.smithi173.stdout:May 27 07:29:38 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-l[73601]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7f397d5141ca] 2024-05-27T07:29:38.526 INFO:journalctl@ceph.mds.l.smithi173.stdout:May 27 07:29:38 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-l[73601]: 16: clone() 2024-05-27T07:29:38.526 INFO:journalctl@ceph.mds.l.smithi173.stdout:May 27 07:29:38 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-l[73601]: 2024-05-27T07:29:38.526 INFO:journalctl@ceph.mds.l.smithi173.stdout:May 27 07:29:38 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-l[73601]: 0> 2024-05-27T07:29:38.188+0000 7f396c15c700 -1 *** Caught signal (Aborted) ** 2024-05-27T07:29:38.527 INFO:journalctl@ceph.mds.l.smithi173.stdout:May 27 07:29:38 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-l[73601]: in thread 7f396c15c700 thread_name:md_log_replay 2024-05-27T07:29:38.527 INFO:journalctl@ceph.mds.l.smithi173.stdout:May 27 07:29:38 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-l[73601]: 2024-05-27T07:29:38.527 INFO:journalctl@ceph.mds.l.smithi173.stdout:May 27 07:29:38 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-l[73601]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:29:38.527 INFO:journalctl@ceph.mds.l.smithi173.stdout:May 27 07:29:38 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-l[73601]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f397d51ed20] 2024-05-27T07:29:38.527 INFO:journalctl@ceph.mds.l.smithi173.stdout:May 27 07:29:38 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-l[73601]: 2: gsignal() 2024-05-27T07:29:38.527 INFO:journalctl@ceph.mds.l.smithi173.stdout:May 27 07:29:38 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-l[73601]: 3: abort() 2024-05-27T07:29:38.527 INFO:journalctl@ceph.mds.l.smithi173.stdout:May 27 07:29:38 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-l[73601]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x26e) [0x7f397e9a1ced] 2024-05-27T07:29:38.527 INFO:journalctl@ceph.mds.l.smithi173.stdout:May 27 07:29:38 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-l[73601]: 5: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7f397e9a1dd1] 2024-05-27T07:29:38.527 INFO:journalctl@ceph.mds.l.smithi173.stdout:May 27 07:29:38 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-l[73601]: 6: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x5645f012e23d] 2024-05-27T07:29:38.527 INFO:journalctl@ceph.mds.l.smithi173.stdout:May 27 07:29:38 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-l[73601]: 7: (SnapRealm::build_snap_set() const+0x323) [0x5645f0119971] 2024-05-27T07:29:38.527 INFO:journalctl@ceph.mds.l.smithi173.stdout:May 27 07:29:38 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-l[73601]: 8: (SnapRealm::check_cache() const+0x13e) [0x5645f0117fa0] 2024-05-27T07:29:38.527 INFO:journalctl@ceph.mds.l.smithi173.stdout:May 27 07:29:38 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-l[73601]: 9: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x5645f0118651] 2024-05-27T07:29:38.527 INFO:journalctl@ceph.mds.l.smithi173.stdout:May 27 07:29:38 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-l[73601]: 10: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x5645f00ad347] 2024-05-27T07:29:38.527 INFO:journalctl@ceph.mds.l.smithi173.stdout:May 27 07:29:38 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-l[73601]: 11: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x5645efda987e] 2024-05-27T07:29:38.528 INFO:journalctl@ceph.mds.l.smithi173.stdout:May 27 07:29:38 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-l[73601]: 12: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x2702) [0x5645efdaee94] 2024-05-27T07:29:38.528 INFO:journalctl@ceph.mds.l.smithi173.stdout:May 27 07:29:38 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-l[73601]: 13: (EUpdate::replay(MDSRank*)+0x63) [0x5645efdb9969] 2024-05-27T07:29:38.528 INFO:journalctl@ceph.mds.l.smithi173.stdout:May 27 07:29:38 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-l[73601]: 14: (MDLog::_replay_thread()+0x1f38) [0x5645f015248c] 2024-05-27T07:29:38.528 INFO:journalctl@ceph.mds.l.smithi173.stdout:May 27 07:29:38 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-l[73601]: 15: (MDLog::ReplayThread::entry()+0x11) [0x5645f015b9cb] 2024-05-27T07:29:38.528 INFO:journalctl@ceph.mds.l.smithi173.stdout:May 27 07:29:38 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-l[73601]: 16: (Thread::entry_wrapper()+0x43) [0x7f397e97072d] 2024-05-27T07:29:38.528 INFO:journalctl@ceph.mds.l.smithi173.stdout:May 27 07:29:38 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-l[73601]: 17: (Thread::_entry_func(void*)+0xd) [0x7f397e970749] 2024-05-27T07:29:38.528 INFO:journalctl@ceph.mds.l.smithi173.stdout:May 27 07:29:38 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-l[73601]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7f397d5141ca] 2024-05-27T07:29:38.528 INFO:journalctl@ceph.mds.l.smithi173.stdout:May 27 07:29:38 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-l[73601]: 19: clone() 2024-05-27T07:29:38.528 INFO:journalctl@ceph.mds.l.smithi173.stdout:May 27 07:29:38 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-l[73601]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-05-27T07:29:38.528 INFO:journalctl@ceph.mds.l.smithi173.stdout:May 27 07:29:38 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-l[73601]: 2024-05-27T07:29:38.528 INFO:journalctl@ceph.mds.l.smithi173.stdout:May 27 07:29:38 smithi173 podman[73636]: 2024-05-27 07:29:38.246155281 +0000 UTC m=+0.018564719 container died b569910ccacdea0c5f88cd2446dd19414cf2167c1bfcea2209be3f26c4bb9c0f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-l, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, GIT_BRANCH=HEAD, ceph=True, org.label-schema.build-date=20240521, org.label-schema.license=GPLv2) 2024-05-27T07:29:38.690 INFO:journalctl@ceph.mds.k.smithi105.stdout:May 27 07:29:38 smithi105 ceph-mds[69842]: -- 172.21.15.105:0/2645272436 <== mon.2 v2:172.21.15.105:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55f62aa17a40 con 0x55f62aa5fa80 2024-05-27T07:29:38.690 INFO:journalctl@ceph.mds.k.smithi105.stdout:May 27 07:29:38 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-k[69838]: starting mds.k at 2024-05-27T07:29:38.690 INFO:journalctl@ceph.mds.k.smithi105.stdout:May 27 07:29:38 smithi105 bash[69824]: 08bbfdf102e0774d695c3546b67fd704eb12702e615eb396bd27fb68becc408e 2024-05-27T07:29:38.690 INFO:journalctl@ceph.mds.k.smithi105.stdout:May 27 07:29:38 smithi105 systemd[1]: Started Ceph mds.k for 73df2636-1bf8-11ef-bc9b-c7b262605968. 2024-05-27T07:29:38.822 INFO:journalctl@ceph.mds.j.smithi052.stdout:May 27 07:29:38 smithi052 systemd[1]: ceph-73df2636-1bf8-11ef-bc9b-c7b262605968@mds.j.service: Failed with result 'exit-code'. 2024-05-27T07:29:38.822 INFO:journalctl@ceph.mds.j.smithi052.stdout:May 27 07:29:38 smithi052 systemd[1]: ceph-73df2636-1bf8-11ef-bc9b-c7b262605968@mds.j.service: Consumed 1.162s CPU time. 2024-05-27T07:29:38.822 INFO:journalctl@ceph.mds.a.smithi052.stdout:May 27 07:29:38 smithi052 systemd[1]: ceph-73df2636-1bf8-11ef-bc9b-c7b262605968@mds.a.service: Scheduled restart job, restart counter is at 2. 2024-05-27T07:29:38.822 INFO:journalctl@ceph.mds.a.smithi052.stdout:May 27 07:29:38 smithi052 systemd[1]: Stopped Ceph mds.a for 73df2636-1bf8-11ef-bc9b-c7b262605968. 2024-05-27T07:29:38.823 INFO:journalctl@ceph.mds.a.smithi052.stdout:May 27 07:29:38 smithi052 systemd[1]: Starting Ceph mds.a for 73df2636-1bf8-11ef-bc9b-c7b262605968... 2024-05-27T07:29:39.102 INFO:journalctl@ceph.mds.l.smithi173.stdout:May 27 07:29:38 smithi173 podman[73636]: 2024-05-27 07:29:38.817724936 +0000 UTC m=+0.590134374 container remove b569910ccacdea0c5f88cd2446dd19414cf2167c1bfcea2209be3f26c4bb9c0f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-l, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, org.label-schema.build-date=20240521, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, org.label-schema.schema-version=1.0, GIT_CLEAN=True, CEPH_POINT_RELEASE=) 2024-05-27T07:29:39.102 INFO:journalctl@ceph.mds.l.smithi173.stdout:May 27 07:29:38 smithi173 systemd[1]: ceph-73df2636-1bf8-11ef-bc9b-c7b262605968@mds.l.service: Main process exited, code=exited, status=134/n/a 2024-05-27T07:29:39.102 INFO:journalctl@ceph.mds.l.smithi173.stdout:May 27 07:29:39 smithi173 systemd[1]: ceph-73df2636-1bf8-11ef-bc9b-c7b262605968@mds.l.service: Failed with result 'exit-code'. 2024-05-27T07:29:39.435 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:29:39 smithi105 ceph-mon[35977]: osdmap e106: 12 total, 12 up, 12 in 2024-05-27T07:29:39.436 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:29:39 smithi105 ceph-mon[35977]: mds.? [v2:172.21.15.173:6832/1239741309,v1:172.21.15.173:6833/1239741309] up:boot 2024-05-27T07:29:39.436 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:29:39 smithi105 ceph-mon[35977]: fsmap cephfs:5 4 up:standby-replay 1 up:standby 5 up:active 2024-05-27T07:29:39.436 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:29:39 smithi105 ceph-mon[35977]: from='mgr.15333 172.21.15.52:0/416389082' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "l"}]: dispatch 2024-05-27T07:29:39.436 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:29:39 smithi105 ceph-mon[35977]: fsmap cephfs:5 5 up:standby-replay 5 up:active 2024-05-27T07:29:39.441 INFO:journalctl@ceph.mds.k.smithi105.stdout:May 27 07:29:39 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-k[69838]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: In function 'std::set SnapClient::filter(const std::set&) const' thread 7f7cbf3c6700 time 2024-05-27T07:29:39.158903+0000 2024-05-27T07:29:39.441 INFO:journalctl@ceph.mds.k.smithi105.stdout:May 27 07:29:39 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-k[69838]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: 197: FAILED ceph_assert(cached_version > 0) 2024-05-27T07:29:39.442 INFO:journalctl@ceph.mds.k.smithi105.stdout:May 27 07:29:39 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-k[69838]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:29:39.442 INFO:journalctl@ceph.mds.k.smithi105.stdout:May 27 07:29:39 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-k[69838]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x127) [0x7f7cd1c0bba6] 2024-05-27T07:29:39.442 INFO:journalctl@ceph.mds.k.smithi105.stdout:May 27 07:29:39 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-k[69838]: 2: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7f7cd1c0bdd1] 2024-05-27T07:29:39.442 INFO:journalctl@ceph.mds.k.smithi105.stdout:May 27 07:29:39 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-k[69838]: 3: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x55f62923b23d] 2024-05-27T07:29:39.442 INFO:journalctl@ceph.mds.k.smithi105.stdout:May 27 07:29:39 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-k[69838]: 4: (SnapRealm::build_snap_set() const+0x323) [0x55f629226971] 2024-05-27T07:29:39.442 INFO:journalctl@ceph.mds.k.smithi105.stdout:May 27 07:29:39 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-k[69838]: 5: (SnapRealm::check_cache() const+0x13e) [0x55f629224fa0] 2024-05-27T07:29:39.442 INFO:journalctl@ceph.mds.k.smithi105.stdout:May 27 07:29:39 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-k[69838]: 6: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x55f629225651] 2024-05-27T07:29:39.442 INFO:journalctl@ceph.mds.k.smithi105.stdout:May 27 07:29:39 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-k[69838]: 7: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x55f6291ba347] 2024-05-27T07:29:39.442 INFO:journalctl@ceph.mds.k.smithi105.stdout:May 27 07:29:39 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-k[69838]: 8: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x55f628eb687e] 2024-05-27T07:29:39.442 INFO:journalctl@ceph.mds.k.smithi105.stdout:May 27 07:29:39 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-k[69838]: 9: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x279d) [0x55f628ebbf2f] 2024-05-27T07:29:39.442 INFO:journalctl@ceph.mds.k.smithi105.stdout:May 27 07:29:39 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-k[69838]: 10: (EUpdate::replay(MDSRank*)+0x63) [0x55f628ec6969] 2024-05-27T07:29:39.443 INFO:journalctl@ceph.mds.k.smithi105.stdout:May 27 07:29:39 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-k[69838]: 11: (MDLog::_replay_thread()+0x1f38) [0x55f62925f48c] 2024-05-27T07:29:39.443 INFO:journalctl@ceph.mds.k.smithi105.stdout:May 27 07:29:39 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-k[69838]: 12: (MDLog::ReplayThread::entry()+0x11) [0x55f6292689cb] 2024-05-27T07:29:39.443 INFO:journalctl@ceph.mds.k.smithi105.stdout:May 27 07:29:39 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-k[69838]: 13: (Thread::entry_wrapper()+0x43) [0x7f7cd1bda72d] 2024-05-27T07:29:39.443 INFO:journalctl@ceph.mds.k.smithi105.stdout:May 27 07:29:39 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-k[69838]: 14: (Thread::_entry_func(void*)+0xd) [0x7f7cd1bda749] 2024-05-27T07:29:39.443 INFO:journalctl@ceph.mds.k.smithi105.stdout:May 27 07:29:39 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-k[69838]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7f7cd077e1ca] 2024-05-27T07:29:39.443 INFO:journalctl@ceph.mds.k.smithi105.stdout:May 27 07:29:39 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-k[69838]: 16: clone() 2024-05-27T07:29:39.443 INFO:journalctl@ceph.mds.k.smithi105.stdout:May 27 07:29:39 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-k[69838]: 2024-05-27T07:29:39.160+0000 7f7cbf3c6700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: In function 'std::set SnapClient::filter(const std::set&) const' thread 7f7cbf3c6700 time 2024-05-27T07:29:39.158903+0000 2024-05-27T07:29:39.443 INFO:journalctl@ceph.mds.k.smithi105.stdout:May 27 07:29:39 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-k[69838]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: 197: FAILED ceph_assert(cached_version > 0) 2024-05-27T07:29:39.443 INFO:journalctl@ceph.mds.k.smithi105.stdout:May 27 07:29:39 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-k[69838]: 2024-05-27T07:29:39.443 INFO:journalctl@ceph.mds.k.smithi105.stdout:May 27 07:29:39 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-k[69838]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:29:39.443 INFO:journalctl@ceph.mds.k.smithi105.stdout:May 27 07:29:39 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-k[69838]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x127) [0x7f7cd1c0bba6] 2024-05-27T07:29:39.444 INFO:journalctl@ceph.mds.k.smithi105.stdout:May 27 07:29:39 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-k[69838]: 2: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7f7cd1c0bdd1] 2024-05-27T07:29:39.444 INFO:journalctl@ceph.mds.k.smithi105.stdout:May 27 07:29:39 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-k[69838]: 3: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x55f62923b23d] 2024-05-27T07:29:39.444 INFO:journalctl@ceph.mds.k.smithi105.stdout:May 27 07:29:39 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-k[69838]: 4: (SnapRealm::build_snap_set() const+0x323) [0x55f629226971] 2024-05-27T07:29:39.444 INFO:journalctl@ceph.mds.k.smithi105.stdout:May 27 07:29:39 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-k[69838]: 5: (SnapRealm::check_cache() const+0x13e) [0x55f629224fa0] 2024-05-27T07:29:39.444 INFO:journalctl@ceph.mds.k.smithi105.stdout:May 27 07:29:39 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-k[69838]: 6: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x55f629225651] 2024-05-27T07:29:39.444 INFO:journalctl@ceph.mds.k.smithi105.stdout:May 27 07:29:39 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-k[69838]: 7: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x55f6291ba347] 2024-05-27T07:29:39.444 INFO:journalctl@ceph.mds.k.smithi105.stdout:May 27 07:29:39 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-k[69838]: 8: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x55f628eb687e] 2024-05-27T07:29:39.444 INFO:journalctl@ceph.mds.k.smithi105.stdout:May 27 07:29:39 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-k[69838]: 9: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x279d) [0x55f628ebbf2f] 2024-05-27T07:29:39.444 INFO:journalctl@ceph.mds.k.smithi105.stdout:May 27 07:29:39 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-k[69838]: 10: (EUpdate::replay(MDSRank*)+0x63) [0x55f628ec6969] 2024-05-27T07:29:39.445 INFO:journalctl@ceph.mds.k.smithi105.stdout:May 27 07:29:39 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-k[69838]: 11: (MDLog::_replay_thread()+0x1f38) [0x55f62925f48c] 2024-05-27T07:29:39.445 INFO:journalctl@ceph.mds.k.smithi105.stdout:May 27 07:29:39 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-k[69838]: 12: (MDLog::ReplayThread::entry()+0x11) [0x55f6292689cb] 2024-05-27T07:29:39.445 INFO:journalctl@ceph.mds.k.smithi105.stdout:May 27 07:29:39 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-k[69838]: 13: (Thread::entry_wrapper()+0x43) [0x7f7cd1bda72d] 2024-05-27T07:29:39.445 INFO:journalctl@ceph.mds.k.smithi105.stdout:May 27 07:29:39 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-k[69838]: 14: (Thread::_entry_func(void*)+0xd) [0x7f7cd1bda749] 2024-05-27T07:29:39.445 INFO:journalctl@ceph.mds.k.smithi105.stdout:May 27 07:29:39 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-k[69838]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7f7cd077e1ca] 2024-05-27T07:29:39.445 INFO:journalctl@ceph.mds.k.smithi105.stdout:May 27 07:29:39 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-k[69838]: 16: clone() 2024-05-27T07:29:39.445 INFO:journalctl@ceph.mds.k.smithi105.stdout:May 27 07:29:39 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-k[69838]: 2024-05-27T07:29:39.445 INFO:journalctl@ceph.mds.k.smithi105.stdout:May 27 07:29:39 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-k[69838]: *** Caught signal (Aborted) ** 2024-05-27T07:29:39.445 INFO:journalctl@ceph.mds.k.smithi105.stdout:May 27 07:29:39 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-k[69838]: in thread 7f7cbf3c6700 thread_name:md_log_replay 2024-05-27T07:29:39.445 INFO:journalctl@ceph.mds.k.smithi105.stdout:May 27 07:29:39 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-k[69838]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:29:39.445 INFO:journalctl@ceph.mds.k.smithi105.stdout:May 27 07:29:39 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-k[69838]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f7cd0788d20] 2024-05-27T07:29:39.445 INFO:journalctl@ceph.mds.k.smithi105.stdout:May 27 07:29:39 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-k[69838]: 2: gsignal() 2024-05-27T07:29:39.446 INFO:journalctl@ceph.mds.k.smithi105.stdout:May 27 07:29:39 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-k[69838]: 3: abort() 2024-05-27T07:29:39.446 INFO:journalctl@ceph.mds.k.smithi105.stdout:May 27 07:29:39 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-k[69838]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x26e) [0x7f7cd1c0bced] 2024-05-27T07:29:39.446 INFO:journalctl@ceph.mds.k.smithi105.stdout:May 27 07:29:39 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-k[69838]: 5: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7f7cd1c0bdd1] 2024-05-27T07:29:39.446 INFO:journalctl@ceph.mds.k.smithi105.stdout:May 27 07:29:39 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-k[69838]: 6: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x55f62923b23d] 2024-05-27T07:29:39.446 INFO:journalctl@ceph.mds.k.smithi105.stdout:May 27 07:29:39 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-k[69838]: 7: (SnapRealm::build_snap_set() const+0x323) [0x55f629226971] 2024-05-27T07:29:39.446 INFO:journalctl@ceph.mds.k.smithi105.stdout:May 27 07:29:39 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-k[69838]: 8: (SnapRealm::check_cache() const+0x13e) [0x55f629224fa0] 2024-05-27T07:29:39.446 INFO:journalctl@ceph.mds.k.smithi105.stdout:May 27 07:29:39 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-k[69838]: 9: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x55f629225651] 2024-05-27T07:29:39.446 INFO:journalctl@ceph.mds.k.smithi105.stdout:May 27 07:29:39 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-k[69838]: 10: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x55f6291ba347] 2024-05-27T07:29:39.446 INFO:journalctl@ceph.mds.k.smithi105.stdout:May 27 07:29:39 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-k[69838]: 11: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x55f628eb687e] 2024-05-27T07:29:39.446 INFO:journalctl@ceph.mds.k.smithi105.stdout:May 27 07:29:39 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-k[69838]: 12: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x279d) [0x55f628ebbf2f] 2024-05-27T07:29:39.447 INFO:journalctl@ceph.mds.k.smithi105.stdout:May 27 07:29:39 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-k[69838]: 13: (EUpdate::replay(MDSRank*)+0x63) [0x55f628ec6969] 2024-05-27T07:29:39.447 INFO:journalctl@ceph.mds.k.smithi105.stdout:May 27 07:29:39 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-k[69838]: 14: (MDLog::_replay_thread()+0x1f38) [0x55f62925f48c] 2024-05-27T07:29:39.447 INFO:journalctl@ceph.mds.k.smithi105.stdout:May 27 07:29:39 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-k[69838]: 15: (MDLog::ReplayThread::entry()+0x11) [0x55f6292689cb] 2024-05-27T07:29:39.447 INFO:journalctl@ceph.mds.k.smithi105.stdout:May 27 07:29:39 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-k[69838]: 16: (Thread::entry_wrapper()+0x43) [0x7f7cd1bda72d] 2024-05-27T07:29:39.447 INFO:journalctl@ceph.mds.k.smithi105.stdout:May 27 07:29:39 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-k[69838]: 17: (Thread::_entry_func(void*)+0xd) [0x7f7cd1bda749] 2024-05-27T07:29:39.447 INFO:journalctl@ceph.mds.k.smithi105.stdout:May 27 07:29:39 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-k[69838]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7f7cd077e1ca] 2024-05-27T07:29:39.447 INFO:journalctl@ceph.mds.k.smithi105.stdout:May 27 07:29:39 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-k[69838]: 19: clone() 2024-05-27T07:29:39.447 INFO:journalctl@ceph.mds.k.smithi105.stdout:May 27 07:29:39 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-k[69838]: 2024-05-27T07:29:39.163+0000 7f7cbf3c6700 -1 *** Caught signal (Aborted) ** 2024-05-27T07:29:39.447 INFO:journalctl@ceph.mds.k.smithi105.stdout:May 27 07:29:39 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-k[69838]: in thread 7f7cbf3c6700 thread_name:md_log_replay 2024-05-27T07:29:39.447 INFO:journalctl@ceph.mds.k.smithi105.stdout:May 27 07:29:39 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-k[69838]: 2024-05-27T07:29:39.447 INFO:journalctl@ceph.mds.k.smithi105.stdout:May 27 07:29:39 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-k[69838]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:29:39.447 INFO:journalctl@ceph.mds.k.smithi105.stdout:May 27 07:29:39 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-k[69838]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f7cd0788d20] 2024-05-27T07:29:39.447 INFO:journalctl@ceph.mds.k.smithi105.stdout:May 27 07:29:39 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-k[69838]: 2: gsignal() 2024-05-27T07:29:39.447 INFO:journalctl@ceph.mds.k.smithi105.stdout:May 27 07:29:39 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-k[69838]: 3: abort() 2024-05-27T07:29:39.448 INFO:journalctl@ceph.mds.k.smithi105.stdout:May 27 07:29:39 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-k[69838]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x26e) [0x7f7cd1c0bced] 2024-05-27T07:29:39.448 INFO:journalctl@ceph.mds.k.smithi105.stdout:May 27 07:29:39 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-k[69838]: 5: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7f7cd1c0bdd1] 2024-05-27T07:29:39.448 INFO:journalctl@ceph.mds.k.smithi105.stdout:May 27 07:29:39 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-k[69838]: 6: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x55f62923b23d] 2024-05-27T07:29:39.448 INFO:journalctl@ceph.mds.k.smithi105.stdout:May 27 07:29:39 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-k[69838]: 7: (SnapRealm::build_snap_set() const+0x323) [0x55f629226971] 2024-05-27T07:29:39.448 INFO:journalctl@ceph.mds.k.smithi105.stdout:May 27 07:29:39 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-k[69838]: 8: (SnapRealm::check_cache() const+0x13e) [0x55f629224fa0] 2024-05-27T07:29:39.448 INFO:journalctl@ceph.mds.k.smithi105.stdout:May 27 07:29:39 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-k[69838]: 9: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x55f629225651] 2024-05-27T07:29:39.448 INFO:journalctl@ceph.mds.k.smithi105.stdout:May 27 07:29:39 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-k[69838]: 10: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x55f6291ba347] 2024-05-27T07:29:39.448 INFO:journalctl@ceph.mds.k.smithi105.stdout:May 27 07:29:39 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-k[69838]: 11: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x55f628eb687e] 2024-05-27T07:29:39.448 INFO:journalctl@ceph.mds.k.smithi105.stdout:May 27 07:29:39 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-k[69838]: 12: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x279d) [0x55f628ebbf2f] 2024-05-27T07:29:39.448 INFO:journalctl@ceph.mds.k.smithi105.stdout:May 27 07:29:39 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-k[69838]: 13: (EUpdate::replay(MDSRank*)+0x63) [0x55f628ec6969] 2024-05-27T07:29:39.448 INFO:journalctl@ceph.mds.k.smithi105.stdout:May 27 07:29:39 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-k[69838]: 14: (MDLog::_replay_thread()+0x1f38) [0x55f62925f48c] 2024-05-27T07:29:39.448 INFO:journalctl@ceph.mds.k.smithi105.stdout:May 27 07:29:39 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-k[69838]: 15: (MDLog::ReplayThread::entry()+0x11) [0x55f6292689cb] 2024-05-27T07:29:39.448 INFO:journalctl@ceph.mds.k.smithi105.stdout:May 27 07:29:39 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-k[69838]: 16: (Thread::entry_wrapper()+0x43) [0x7f7cd1bda72d] 2024-05-27T07:29:39.449 INFO:journalctl@ceph.mds.k.smithi105.stdout:May 27 07:29:39 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-k[69838]: 17: (Thread::_entry_func(void*)+0xd) [0x7f7cd1bda749] 2024-05-27T07:29:39.449 INFO:journalctl@ceph.mds.k.smithi105.stdout:May 27 07:29:39 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-k[69838]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7f7cd077e1ca] 2024-05-27T07:29:39.449 INFO:journalctl@ceph.mds.k.smithi105.stdout:May 27 07:29:39 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-k[69838]: 19: clone() 2024-05-27T07:29:39.449 INFO:journalctl@ceph.mds.k.smithi105.stdout:May 27 07:29:39 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-k[69838]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-05-27T07:29:39.449 INFO:journalctl@ceph.mds.k.smithi105.stdout:May 27 07:29:39 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-k[69838]: 2024-05-27T07:29:39.449 INFO:journalctl@ceph.mds.k.smithi105.stdout:May 27 07:29:39 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-k[69838]: -3> 2024-05-27T07:29:39.160+0000 7f7cbf3c6700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: In function 'std::set SnapClient::filter(const std::set&) const' thread 7f7cbf3c6700 time 2024-05-27T07:29:39.158903+0000 2024-05-27T07:29:39.449 INFO:journalctl@ceph.mds.k.smithi105.stdout:May 27 07:29:39 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-k[69838]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: 197: FAILED ceph_assert(cached_version > 0) 2024-05-27T07:29:39.449 INFO:journalctl@ceph.mds.k.smithi105.stdout:May 27 07:29:39 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-k[69838]: 2024-05-27T07:29:39.449 INFO:journalctl@ceph.mds.k.smithi105.stdout:May 27 07:29:39 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-k[69838]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:29:39.449 INFO:journalctl@ceph.mds.k.smithi105.stdout:May 27 07:29:39 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-k[69838]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x127) [0x7f7cd1c0bba6] 2024-05-27T07:29:39.449 INFO:journalctl@ceph.mds.k.smithi105.stdout:May 27 07:29:39 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-k[69838]: 2: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7f7cd1c0bdd1] 2024-05-27T07:29:39.449 INFO:journalctl@ceph.mds.k.smithi105.stdout:May 27 07:29:39 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-k[69838]: 3: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x55f62923b23d] 2024-05-27T07:29:39.449 INFO:journalctl@ceph.mds.k.smithi105.stdout:May 27 07:29:39 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-k[69838]: 4: (SnapRealm::build_snap_set() const+0x323) [0x55f629226971] 2024-05-27T07:29:39.449 INFO:journalctl@ceph.mds.k.smithi105.stdout:May 27 07:29:39 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-k[69838]: 5: (SnapRealm::check_cache() const+0x13e) [0x55f629224fa0] 2024-05-27T07:29:39.450 INFO:journalctl@ceph.mds.k.smithi105.stdout:May 27 07:29:39 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-k[69838]: 6: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x55f629225651] 2024-05-27T07:29:39.450 INFO:journalctl@ceph.mds.k.smithi105.stdout:May 27 07:29:39 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-k[69838]: 7: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x55f6291ba347] 2024-05-27T07:29:39.450 INFO:journalctl@ceph.mds.k.smithi105.stdout:May 27 07:29:39 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-k[69838]: 8: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x55f628eb687e] 2024-05-27T07:29:39.450 INFO:journalctl@ceph.mds.k.smithi105.stdout:May 27 07:29:39 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-k[69838]: 9: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x279d) [0x55f628ebbf2f] 2024-05-27T07:29:39.450 INFO:journalctl@ceph.mds.k.smithi105.stdout:May 27 07:29:39 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-k[69838]: 10: (EUpdate::replay(MDSRank*)+0x63) [0x55f628ec6969] 2024-05-27T07:29:39.450 INFO:journalctl@ceph.mds.k.smithi105.stdout:May 27 07:29:39 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-k[69838]: 11: (MDLog::_replay_thread()+0x1f38) [0x55f62925f48c] 2024-05-27T07:29:39.450 INFO:journalctl@ceph.mds.k.smithi105.stdout:May 27 07:29:39 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-k[69838]: 12: (MDLog::ReplayThread::entry()+0x11) [0x55f6292689cb] 2024-05-27T07:29:39.450 INFO:journalctl@ceph.mds.k.smithi105.stdout:May 27 07:29:39 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-k[69838]: 13: (Thread::entry_wrapper()+0x43) [0x7f7cd1bda72d] 2024-05-27T07:29:39.450 INFO:journalctl@ceph.mds.k.smithi105.stdout:May 27 07:29:39 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-k[69838]: 14: (Thread::_entry_func(void*)+0xd) [0x7f7cd1bda749] 2024-05-27T07:29:39.450 INFO:journalctl@ceph.mds.k.smithi105.stdout:May 27 07:29:39 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-k[69838]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7f7cd077e1ca] 2024-05-27T07:29:39.450 INFO:journalctl@ceph.mds.k.smithi105.stdout:May 27 07:29:39 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-k[69838]: 16: clone() 2024-05-27T07:29:39.450 INFO:journalctl@ceph.mds.k.smithi105.stdout:May 27 07:29:39 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-k[69838]: 2024-05-27T07:29:39.450 INFO:journalctl@ceph.mds.k.smithi105.stdout:May 27 07:29:39 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-k[69838]: 0> 2024-05-27T07:29:39.163+0000 7f7cbf3c6700 -1 *** Caught signal (Aborted) ** 2024-05-27T07:29:39.451 INFO:journalctl@ceph.mds.k.smithi105.stdout:May 27 07:29:39 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-k[69838]: in thread 7f7cbf3c6700 thread_name:md_log_replay 2024-05-27T07:29:39.451 INFO:journalctl@ceph.mds.k.smithi105.stdout:May 27 07:29:39 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-k[69838]: 2024-05-27T07:29:39.451 INFO:journalctl@ceph.mds.k.smithi105.stdout:May 27 07:29:39 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-k[69838]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:29:39.451 INFO:journalctl@ceph.mds.k.smithi105.stdout:May 27 07:29:39 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-k[69838]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f7cd0788d20] 2024-05-27T07:29:39.451 INFO:journalctl@ceph.mds.k.smithi105.stdout:May 27 07:29:39 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-k[69838]: 2: gsignal() 2024-05-27T07:29:39.451 INFO:journalctl@ceph.mds.k.smithi105.stdout:May 27 07:29:39 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-k[69838]: 3: abort() 2024-05-27T07:29:39.451 INFO:journalctl@ceph.mds.k.smithi105.stdout:May 27 07:29:39 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-k[69838]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x26e) [0x7f7cd1c0bced] 2024-05-27T07:29:39.451 INFO:journalctl@ceph.mds.k.smithi105.stdout:May 27 07:29:39 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-k[69838]: 5: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7f7cd1c0bdd1] 2024-05-27T07:29:39.451 INFO:journalctl@ceph.mds.k.smithi105.stdout:May 27 07:29:39 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-k[69838]: 6: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x55f62923b23d] 2024-05-27T07:29:39.451 INFO:journalctl@ceph.mds.k.smithi105.stdout:May 27 07:29:39 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-k[69838]: 7: (SnapRealm::build_snap_set() const+0x323) [0x55f629226971] 2024-05-27T07:29:39.451 INFO:journalctl@ceph.mds.k.smithi105.stdout:May 27 07:29:39 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-k[69838]: 8: (SnapRealm::check_cache() const+0x13e) [0x55f629224fa0] 2024-05-27T07:29:39.451 INFO:journalctl@ceph.mds.k.smithi105.stdout:May 27 07:29:39 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-k[69838]: 9: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x55f629225651] 2024-05-27T07:29:39.451 INFO:journalctl@ceph.mds.k.smithi105.stdout:May 27 07:29:39 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-k[69838]: 10: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x55f6291ba347] 2024-05-27T07:29:39.451 INFO:journalctl@ceph.mds.k.smithi105.stdout:May 27 07:29:39 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-k[69838]: 11: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x55f628eb687e] 2024-05-27T07:29:39.452 INFO:journalctl@ceph.mds.k.smithi105.stdout:May 27 07:29:39 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-k[69838]: 12: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x279d) [0x55f628ebbf2f] 2024-05-27T07:29:39.452 INFO:journalctl@ceph.mds.k.smithi105.stdout:May 27 07:29:39 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-k[69838]: 13: (EUpdate::replay(MDSRank*)+0x63) [0x55f628ec6969] 2024-05-27T07:29:39.452 INFO:journalctl@ceph.mds.k.smithi105.stdout:May 27 07:29:39 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-k[69838]: 14: (MDLog::_replay_thread()+0x1f38) [0x55f62925f48c] 2024-05-27T07:29:39.452 INFO:journalctl@ceph.mds.k.smithi105.stdout:May 27 07:29:39 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-k[69838]: 15: (MDLog::ReplayThread::entry()+0x11) [0x55f6292689cb] 2024-05-27T07:29:39.452 INFO:journalctl@ceph.mds.k.smithi105.stdout:May 27 07:29:39 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-k[69838]: 16: (Thread::entry_wrapper()+0x43) [0x7f7cd1bda72d] 2024-05-27T07:29:39.452 INFO:journalctl@ceph.mds.k.smithi105.stdout:May 27 07:29:39 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-k[69838]: 17: (Thread::_entry_func(void*)+0xd) [0x7f7cd1bda749] 2024-05-27T07:29:39.452 INFO:journalctl@ceph.mds.k.smithi105.stdout:May 27 07:29:39 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-k[69838]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7f7cd077e1ca] 2024-05-27T07:29:39.452 INFO:journalctl@ceph.mds.k.smithi105.stdout:May 27 07:29:39 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-k[69838]: 19: clone() 2024-05-27T07:29:39.452 INFO:journalctl@ceph.mds.k.smithi105.stdout:May 27 07:29:39 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-k[69838]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-05-27T07:29:39.452 INFO:journalctl@ceph.mds.k.smithi105.stdout:May 27 07:29:39 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-k[69838]: 2024-05-27T07:29:39.452 INFO:journalctl@ceph.mds.k.smithi105.stdout:May 27 07:29:39 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-k[69838]: -3> 2024-05-27T07:29:39.160+0000 7f7cbf3c6700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: In function 'std::set SnapClient::filter(const std::set&) const' thread 7f7cbf3c6700 time 2024-05-27T07:29:39.158903+0000 2024-05-27T07:29:39.452 INFO:journalctl@ceph.mds.k.smithi105.stdout:May 27 07:29:39 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-k[69838]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: 197: FAILED ceph_assert(cached_version > 0) 2024-05-27T07:29:39.452 INFO:journalctl@ceph.mds.k.smithi105.stdout:May 27 07:29:39 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-k[69838]: 2024-05-27T07:29:39.452 INFO:journalctl@ceph.mds.k.smithi105.stdout:May 27 07:29:39 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-k[69838]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:29:39.452 INFO:journalctl@ceph.mds.k.smithi105.stdout:May 27 07:29:39 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-k[69838]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x127) [0x7f7cd1c0bba6] 2024-05-27T07:29:39.453 INFO:journalctl@ceph.mds.k.smithi105.stdout:May 27 07:29:39 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-k[69838]: 2: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7f7cd1c0bdd1] 2024-05-27T07:29:39.453 INFO:journalctl@ceph.mds.k.smithi105.stdout:May 27 07:29:39 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-k[69838]: 3: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x55f62923b23d] 2024-05-27T07:29:39.453 INFO:journalctl@ceph.mds.k.smithi105.stdout:May 27 07:29:39 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-k[69838]: 4: (SnapRealm::build_snap_set() const+0x323) [0x55f629226971] 2024-05-27T07:29:39.453 INFO:journalctl@ceph.mds.k.smithi105.stdout:May 27 07:29:39 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-k[69838]: 5: (SnapRealm::check_cache() const+0x13e) [0x55f629224fa0] 2024-05-27T07:29:39.453 INFO:journalctl@ceph.mds.k.smithi105.stdout:May 27 07:29:39 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-k[69838]: 6: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x55f629225651] 2024-05-27T07:29:39.453 INFO:journalctl@ceph.mds.k.smithi105.stdout:May 27 07:29:39 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-k[69838]: 7: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x55f6291ba347] 2024-05-27T07:29:39.453 INFO:journalctl@ceph.mds.k.smithi105.stdout:May 27 07:29:39 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-k[69838]: 8: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x55f628eb687e] 2024-05-27T07:29:39.453 INFO:journalctl@ceph.mds.k.smithi105.stdout:May 27 07:29:39 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-k[69838]: 9: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x279d) [0x55f628ebbf2f] 2024-05-27T07:29:39.453 INFO:journalctl@ceph.mds.k.smithi105.stdout:May 27 07:29:39 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-k[69838]: 10: (EUpdate::replay(MDSRank*)+0x63) [0x55f628ec6969] 2024-05-27T07:29:39.453 INFO:journalctl@ceph.mds.k.smithi105.stdout:May 27 07:29:39 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-k[69838]: 11: (MDLog::_replay_thread()+0x1f38) [0x55f62925f48c] 2024-05-27T07:29:39.453 INFO:journalctl@ceph.mds.k.smithi105.stdout:May 27 07:29:39 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-k[69838]: 12: (MDLog::ReplayThread::entry()+0x11) [0x55f6292689cb] 2024-05-27T07:29:39.453 INFO:journalctl@ceph.mds.k.smithi105.stdout:May 27 07:29:39 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-k[69838]: 13: (Thread::entry_wrapper()+0x43) [0x7f7cd1bda72d] 2024-05-27T07:29:39.453 INFO:journalctl@ceph.mds.k.smithi105.stdout:May 27 07:29:39 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-k[69838]: 14: (Thread::_entry_func(void*)+0xd) [0x7f7cd1bda749] 2024-05-27T07:29:39.453 INFO:journalctl@ceph.mds.k.smithi105.stdout:May 27 07:29:39 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-k[69838]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7f7cd077e1ca] 2024-05-27T07:29:39.453 INFO:journalctl@ceph.mds.k.smithi105.stdout:May 27 07:29:39 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-k[69838]: 16: clone() 2024-05-27T07:29:39.454 INFO:journalctl@ceph.mds.k.smithi105.stdout:May 27 07:29:39 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-k[69838]: 2024-05-27T07:29:39.454 INFO:journalctl@ceph.mds.k.smithi105.stdout:May 27 07:29:39 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-k[69838]: 0> 2024-05-27T07:29:39.163+0000 7f7cbf3c6700 -1 *** Caught signal (Aborted) ** 2024-05-27T07:29:39.454 INFO:journalctl@ceph.mds.k.smithi105.stdout:May 27 07:29:39 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-k[69838]: in thread 7f7cbf3c6700 thread_name:md_log_replay 2024-05-27T07:29:39.454 INFO:journalctl@ceph.mds.k.smithi105.stdout:May 27 07:29:39 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-k[69838]: 2024-05-27T07:29:39.454 INFO:journalctl@ceph.mds.k.smithi105.stdout:May 27 07:29:39 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-k[69838]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:29:39.454 INFO:journalctl@ceph.mds.k.smithi105.stdout:May 27 07:29:39 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-k[69838]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f7cd0788d20] 2024-05-27T07:29:39.454 INFO:journalctl@ceph.mds.k.smithi105.stdout:May 27 07:29:39 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-k[69838]: 2: gsignal() 2024-05-27T07:29:39.454 INFO:journalctl@ceph.mds.k.smithi105.stdout:May 27 07:29:39 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-k[69838]: 3: abort() 2024-05-27T07:29:39.455 INFO:journalctl@ceph.mds.k.smithi105.stdout:May 27 07:29:39 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-k[69838]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x26e) [0x7f7cd1c0bced] 2024-05-27T07:29:39.455 INFO:journalctl@ceph.mds.k.smithi105.stdout:May 27 07:29:39 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-k[69838]: 5: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7f7cd1c0bdd1] 2024-05-27T07:29:39.455 INFO:journalctl@ceph.mds.k.smithi105.stdout:May 27 07:29:39 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-k[69838]: 6: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x55f62923b23d] 2024-05-27T07:29:39.455 INFO:journalctl@ceph.mds.k.smithi105.stdout:May 27 07:29:39 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-k[69838]: 7: (SnapRealm::build_snap_set() const+0x323) [0x55f629226971] 2024-05-27T07:29:39.455 INFO:journalctl@ceph.mds.k.smithi105.stdout:May 27 07:29:39 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-k[69838]: 8: (SnapRealm::check_cache() const+0x13e) [0x55f629224fa0] 2024-05-27T07:29:39.455 INFO:journalctl@ceph.mds.k.smithi105.stdout:May 27 07:29:39 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-k[69838]: 9: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x55f629225651] 2024-05-27T07:29:39.455 INFO:journalctl@ceph.mds.k.smithi105.stdout:May 27 07:29:39 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-k[69838]: 10: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x55f6291ba347] 2024-05-27T07:29:39.455 INFO:journalctl@ceph.mds.k.smithi105.stdout:May 27 07:29:39 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-k[69838]: 11: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x55f628eb687e] 2024-05-27T07:29:39.455 INFO:journalctl@ceph.mds.k.smithi105.stdout:May 27 07:29:39 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-k[69838]: 12: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x279d) [0x55f628ebbf2f] 2024-05-27T07:29:39.455 INFO:journalctl@ceph.mds.k.smithi105.stdout:May 27 07:29:39 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-k[69838]: 13: (EUpdate::replay(MDSRank*)+0x63) [0x55f628ec6969] 2024-05-27T07:29:39.455 INFO:journalctl@ceph.mds.k.smithi105.stdout:May 27 07:29:39 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-k[69838]: 14: (MDLog::_replay_thread()+0x1f38) [0x55f62925f48c] 2024-05-27T07:29:39.455 INFO:journalctl@ceph.mds.k.smithi105.stdout:May 27 07:29:39 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-k[69838]: 15: (MDLog::ReplayThread::entry()+0x11) [0x55f6292689cb] 2024-05-27T07:29:39.455 INFO:journalctl@ceph.mds.k.smithi105.stdout:May 27 07:29:39 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-k[69838]: 16: (Thread::entry_wrapper()+0x43) [0x7f7cd1bda72d] 2024-05-27T07:29:39.455 INFO:journalctl@ceph.mds.k.smithi105.stdout:May 27 07:29:39 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-k[69838]: 17: (Thread::_entry_func(void*)+0xd) [0x7f7cd1bda749] 2024-05-27T07:29:39.456 INFO:journalctl@ceph.mds.k.smithi105.stdout:May 27 07:29:39 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-k[69838]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7f7cd077e1ca] 2024-05-27T07:29:39.456 INFO:journalctl@ceph.mds.k.smithi105.stdout:May 27 07:29:39 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-k[69838]: 19: clone() 2024-05-27T07:29:39.456 INFO:journalctl@ceph.mds.k.smithi105.stdout:May 27 07:29:39 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-k[69838]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-05-27T07:29:39.456 INFO:journalctl@ceph.mds.k.smithi105.stdout:May 27 07:29:39 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-k[69838]: 2024-05-27T07:29:39.456 INFO:journalctl@ceph.mds.k.smithi105.stdout:May 27 07:29:39 smithi105 podman[69873]: 2024-05-27 07:29:39.230264879 +0000 UTC m=+0.044815230 container died 08bbfdf102e0774d695c3546b67fd704eb12702e615eb396bd27fb68becc408e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-k, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, ceph=True, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, org.label-schema.build-date=20240521, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux ) 2024-05-27T07:29:39.491 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:29:39 smithi052 ceph-mon[31955]: osdmap e106: 12 total, 12 up, 12 in 2024-05-27T07:29:39.491 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:29:39 smithi052 ceph-mon[31955]: mds.? [v2:172.21.15.173:6832/1239741309,v1:172.21.15.173:6833/1239741309] up:boot 2024-05-27T07:29:39.492 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:29:39 smithi052 ceph-mon[31955]: fsmap cephfs:5 4 up:standby-replay 1 up:standby 5 up:active 2024-05-27T07:29:39.492 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:29:39 smithi052 ceph-mon[31955]: from='mgr.15333 172.21.15.52:0/416389082' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "l"}]: dispatch 2024-05-27T07:29:39.492 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:29:39 smithi052 ceph-mon[31955]: fsmap cephfs:5 5 up:standby-replay 5 up:active 2024-05-27T07:29:39.492 INFO:journalctl@ceph.mds.a.smithi052.stdout:May 27 07:29:39 smithi052 podman[105616]: 2024-05-27 07:29:39.010496649 +0000 UTC m=+0.027957144 image pull e36ea4fc8a5807e8d2701370e350a0de3856538797f7fba64cfce7e344bd6799 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6 2024-05-27T07:29:39.492 INFO:journalctl@ceph.mds.a.smithi052.stdout:May 27 07:29:39 smithi052 podman[105616]: 2024-05-27 07:29:39.13955834 +0000 UTC m=+0.157018805 container create c5447791e4a6f6b5a3928fe2dcc2df1539dc407ed035bafd6e3e0e3e6691c4ef (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-a, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240521, CEPH_POINT_RELEASE=, ceph=True, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, GIT_BRANCH=HEAD) 2024-05-27T07:29:39.602 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:29:39 smithi173 ceph-mon[37000]: osdmap e106: 12 total, 12 up, 12 in 2024-05-27T07:29:39.602 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:29:39 smithi173 ceph-mon[37000]: mds.? [v2:172.21.15.173:6832/1239741309,v1:172.21.15.173:6833/1239741309] up:boot 2024-05-27T07:29:39.602 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:29:39 smithi173 ceph-mon[37000]: fsmap cephfs:5 4 up:standby-replay 1 up:standby 5 up:active 2024-05-27T07:29:39.602 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:29:39 smithi173 ceph-mon[37000]: from='mgr.15333 172.21.15.52:0/416389082' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "l"}]: dispatch 2024-05-27T07:29:39.603 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:29:39 smithi173 ceph-mon[37000]: fsmap cephfs:5 5 up:standby-replay 5 up:active 2024-05-27T07:29:39.799 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:29:39.806+0000 7f7de93fa700 1 -- 172.21.15.52:0/4195179959 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f7de411d920 msgr2=0x7f7de411fd40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:29:39.799 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:29:39.806+0000 7f7de93fa700 1 --2- 172.21.15.52:0/4195179959 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f7de411d920 0x7f7de411fd40 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7f7ddc00b0a0 tx=0x7f7ddc00b3b0 comp rx=0 tx=0).stop 2024-05-27T07:29:39.802 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:29:39.809+0000 7f7de93fa700 1 -- 172.21.15.52:0/4195179959 shutdown_connections 2024-05-27T07:29:39.802 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:29:39.809+0000 7f7de93fa700 1 --2- 172.21.15.52:0/4195179959 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f7de411d920 0x7f7de411fd40 unknown :-1 s=CLOSED pgs=129 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:29:39.802 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:29:39.809+0000 7f7de93fa700 1 --2- 172.21.15.52:0/4195179959 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f7de410a920 0x7f7de410ad30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:29:39.802 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:29:39.809+0000 7f7de93fa700 1 --2- 172.21.15.52:0/4195179959 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f7de410b270 0x7f7de411d3e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:29:39.802 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:29:39.809+0000 7f7de93fa700 1 -- 172.21.15.52:0/4195179959 >> 172.21.15.52:0/4195179959 conn(0x7f7de406f4c0 msgr2=0x7f7de406f900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:29:39.803 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:29:39.810+0000 7f7de93fa700 1 -- 172.21.15.52:0/4195179959 shutdown_connections 2024-05-27T07:29:39.806 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:29:39.814+0000 7f7de93fa700 1 -- 172.21.15.52:0/4195179959 wait complete. 2024-05-27T07:29:39.806 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:29:39.815+0000 7f7de93fa700 1 Processor -- start 2024-05-27T07:29:39.811 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:29:39.817+0000 7f7de93fa700 1 -- start start 2024-05-27T07:29:39.811 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:29:39.817+0000 7f7de93fa700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f7de410a920 0x7f7de41af7f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:29:39.811 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:29:39.817+0000 7f7de93fa700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f7de410b270 0x7f7de41afd30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:29:39.812 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:29:39.817+0000 7f7de93fa700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f7de411d920 0x7f7de41a9a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:29:39.812 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:29:39.817+0000 7f7de93fa700 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x7f7de41232b0 con 0x7f7de410b270 2024-05-27T07:29:39.812 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:29:39.817+0000 7f7de93fa700 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f7de4123130 con 0x7f7de410a920 2024-05-27T07:29:39.812 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:29:39.817+0000 7f7de93fa700 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f7de4123430 con 0x7f7de411d920 2024-05-27T07:29:39.812 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:29:39.818+0000 7f7de27fc700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f7de410b270 0x7f7de41afd30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:29:39.812 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:29:39.818+0000 7f7de27fc700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f7de410b270 0x7f7de41afd30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.52:35840/0 (socket says 172.21.15.52:35840) 2024-05-27T07:29:39.812 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:29:39.818+0000 7f7de27fc700 1 -- 172.21.15.52:0/1066560918 learned_addr learned my addr 172.21.15.52:0/1066560918 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-05-27T07:29:39.812 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:29:39.819+0000 7f7de2ffd700 1 --2- 172.21.15.52:0/1066560918 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f7de410a920 0x7f7de41af7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:29:39.812 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:29:39.819+0000 7f7de37fe700 1 --2- 172.21.15.52:0/1066560918 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f7de411d920 0x7f7de41a9a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:29:39.812 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:29:39.820+0000 7f7de27fc700 1 -- 172.21.15.52:0/1066560918 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f7de411d920 msgr2=0x7f7de41a9a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:29:39.812 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:29:39.820+0000 7f7de27fc700 1 --2- 172.21.15.52:0/1066560918 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f7de411d920 0x7f7de41a9a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:29:39.812 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:29:39.820+0000 7f7de27fc700 1 -- 172.21.15.52:0/1066560918 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f7de410a920 msgr2=0x7f7de41af7f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:29:39.812 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:29:39.820+0000 7f7de27fc700 1 --2- 172.21.15.52:0/1066560918 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f7de410a920 0x7f7de41af7f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:29:39.813 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:29:39.820+0000 7f7de27fc700 1 -- 172.21.15.52:0/1066560918 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7de41aa0e0 con 0x7f7de410b270 2024-05-27T07:29:39.821 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:29:39.831+0000 7f7de27fc700 1 --2- 172.21.15.52:0/1066560918 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f7de410b270 0x7f7de41afd30 secure :-1 s=READY pgs=612 cs=0 l=1 rev1=1 crypto rx=0x7f7dd800bf20 tx=0x7f7dd800dc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:29:39.827 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:29:39.835+0000 7f7dcbfff700 1 -- 172.21.15.52:0/1066560918 <== mon.0 v2:172.21.15.52:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7dd8038070 con 0x7f7de410b270 2024-05-27T07:29:39.827 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:29:39.835+0000 7f7dcbfff700 1 -- 172.21.15.52:0/1066560918 <== mon.0 v2:172.21.15.52:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f7dd800de70 con 0x7f7de410b270 2024-05-27T07:29:39.827 INFO:journalctl@ceph.mds.a.smithi052.stdout:May 27 07:29:39 smithi052 podman[105616]: 2024-05-27 07:29:39.77841227 +0000 UTC m=+0.795872756 container init c5447791e4a6f6b5a3928fe2dcc2df1539dc407ed035bafd6e3e0e3e6691c4ef (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-a, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, org.label-schema.build-date=20240521, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, io.buildah.version=1.33.5, GIT_CLEAN=True, ceph=True) 2024-05-27T07:29:39.827 INFO:journalctl@ceph.mds.a.smithi052.stdout:May 27 07:29:39 smithi052 podman[105616]: 2024-05-27 07:29:39.783917935 +0000 UTC m=+0.801378412 container start c5447791e4a6f6b5a3928fe2dcc2df1539dc407ed035bafd6e3e0e3e6691c4ef (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-a, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, org.label-schema.build-date=20240521, org.label-schema.vendor=CentOS, GIT_CLEAN=True, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, ceph=True, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0) 2024-05-27T07:29:39.828 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:29:39.836+0000 7f7dcbfff700 1 -- 172.21.15.52:0/1066560918 <== mon.0 v2:172.21.15.52:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7dd8031630 con 0x7f7de410b270 2024-05-27T07:29:39.828 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:29:39.836+0000 7f7de93fa700 1 -- 172.21.15.52:0/1066560918 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7de41aa3d0 con 0x7f7de410b270 2024-05-27T07:29:39.828 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:29:39.837+0000 7f7de93fa700 1 -- 172.21.15.52:0/1066560918 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7de4112bc0 con 0x7f7de410b270 2024-05-27T07:29:39.835 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:29:39.842+0000 7f7df7e0fb80 1 Processor -- start 2024-05-27T07:29:39.835 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:29:39.842+0000 7f7df7e0fb80 1 -- start start 2024-05-27T07:29:39.835 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:29:39.842+0000 7f7df7e0fb80 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x5636f9e0be40 0x5636f9f90b30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:29:39.835 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:29:39.842+0000 7f7df7e0fb80 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5636f9f91070 0x5636f9f8abe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:29:39.835 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:29:39.842+0000 7f7de37fe700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x5636f9e0be40 0x5636f9f90b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:29:39.835 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:29:39.843+0000 7f7de37fe700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x5636f9e0be40 0x5636f9f90b30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.52:35854/0 (socket says 172.21.15.52:35854) 2024-05-27T07:29:39.835 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:29:39.843+0000 7f7de37fe700 1 -- 172.21.15.52:0/531955204 learned_addr learned my addr 172.21.15.52:0/531955204 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-05-27T07:29:39.835 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:29:39.843+0000 7f7df7e0fb80 1 --2- 172.21.15.52:0/531955204 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5636f9f8b120 0x5636f9f8b530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:29:39.835 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:29:39.843+0000 7f7df7e0fb80 1 -- 172.21.15.52:0/531955204 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x5636f9eb8d70 con 0x5636f9e0be40 2024-05-27T07:29:39.835 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:29:39.843+0000 7f7df7e0fb80 1 -- 172.21.15.52:0/531955204 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x5636f9ea8910 con 0x5636f9f91070 2024-05-27T07:29:39.836 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:29:39.843+0000 7f7df7e0fb80 1 -- 172.21.15.52:0/531955204 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x5636f9eac950 con 0x5636f9f8b120 2024-05-27T07:29:39.836 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:29:39.844+0000 7f7de2ffd700 1 --2- 172.21.15.52:0/531955204 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5636f9f91070 0x5636f9f8abe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:29:39.836 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:29:39.845+0000 7f7de37fe700 1 -- 172.21.15.52:0/531955204 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5636f9f8b120 msgr2=0x5636f9f8b530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:29:39.836 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:29:39.845+0000 7f7de37fe700 1 --2- 172.21.15.52:0/531955204 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5636f9f8b120 0x5636f9f8b530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:29:39.836 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:29:39.845+0000 7f7de37fe700 1 -- 172.21.15.52:0/531955204 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5636f9f91070 msgr2=0x5636f9f8abe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:29:39.836 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:29:39.845+0000 7f7de37fe700 1 --2- 172.21.15.52:0/531955204 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5636f9f91070 0x5636f9f8abe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:29:39.836 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:29:39.845+0000 7f7de37fe700 1 -- 172.21.15.52:0/531955204 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5636f9e51fb0 con 0x5636f9e0be40 2024-05-27T07:29:39.840 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:29:39.848+0000 7f7dcbfff700 1 -- 172.21.15.52:0/1066560918 <== mon.0 v2:172.21.15.52:3300/0 4 ==== mgrmap(e 26) ==== 149783+0+0 (secure 0 0 0) 0x7f7dd803b070 con 0x7f7de410b270 2024-05-27T07:29:39.840 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:29:39.849+0000 7f7dcbfff700 1 --2- 172.21.15.52:0/1066560918 >> [v2:172.21.15.52:6800/1822318384,v1:172.21.15.52:6801/1822318384] conn(0x7f7dcc0b2a20 0x7f7dcc0b4f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:29:39.840 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:29:39.850+0000 7f7dcbfff700 1 -- 172.21.15.52:0/1066560918 <== mon.0 v2:172.21.15.52:3300/0 5 ==== osd_map(106..106 src has 1..106) ==== 9379+0+0 (secure 0 0 0) 0x7f7dd80c9960 con 0x7f7de410b270 2024-05-27T07:29:39.841 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:29:39.850+0000 7f7de2ffd700 1 --2- 172.21.15.52:0/1066560918 >> [v2:172.21.15.52:6800/1822318384,v1:172.21.15.52:6801/1822318384] conn(0x7f7dcc0b2a20 0x7f7dcc0b4f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:29:39.842 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:29:39.851+0000 7f7de2ffd700 1 --2- 172.21.15.52:0/1066560918 >> [v2:172.21.15.52:6800/1822318384,v1:172.21.15.52:6801/1822318384] conn(0x7f7dcc0b2a20 0x7f7dcc0b4f30 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x5636f9f872a0 tx=0x7f7dd4009260 comp rx=0 tx=0).ready entity=mgr.15333 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:29:39.844 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:29:39.852+0000 7f7de37fe700 1 --2- 172.21.15.52:0/531955204 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x5636f9e0be40 0x5636f9f90b30 secure :-1 s=READY pgs=613 cs=0 l=1 rev1=1 crypto rx=0x7f7ddc00b780 tx=0x7f7ddc009830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=704b51b28095b087 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:29:39.848 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:29:39.857+0000 7f7dc97fa700 1 -- 172.21.15.52:0/531955204 <== mon.0 v2:172.21.15.52:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7ddc00bcb0 con 0x5636f9e0be40 2024-05-27T07:29:39.848 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:29:39.857+0000 7f7dc97fa700 1 -- 172.21.15.52:0/531955204 <== mon.0 v2:172.21.15.52:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f7ddc0075c0 con 0x5636f9e0be40 2024-05-27T07:29:39.848 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:29:39.857+0000 7f7dc97fa700 1 -- 172.21.15.52:0/531955204 <== mon.0 v2:172.21.15.52:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7ddc005260 con 0x5636f9e0be40 2024-05-27T07:29:39.850 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:29:39.859+0000 7f7df7e0fb80 1 -- 172.21.15.52:0/531955204 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x5636f9e0be40 msgr2=0x5636f9f90b30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:29:39.850 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:29:39.859+0000 7f7df7e0fb80 1 --2- 172.21.15.52:0/531955204 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x5636f9e0be40 0x5636f9f90b30 secure :-1 s=READY pgs=613 cs=0 l=1 rev1=1 crypto rx=0x7f7ddc00b780 tx=0x7f7ddc009830 comp rx=0 tx=0).stop 2024-05-27T07:29:39.850 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:29:39.859+0000 7f7df7e0fb80 1 -- 172.21.15.52:0/531955204 shutdown_connections 2024-05-27T07:29:39.850 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:29:39.859+0000 7f7df7e0fb80 1 --2- 172.21.15.52:0/531955204 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5636f9f8b120 0x5636f9f8b530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:29:39.850 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:29:39.859+0000 7f7df7e0fb80 1 --2- 172.21.15.52:0/531955204 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x5636f9e0be40 0x5636f9f90b30 unknown :-1 s=CLOSED pgs=613 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:29:39.850 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:29:39.859+0000 7f7df7e0fb80 1 --2- 172.21.15.52:0/531955204 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5636f9f91070 0x5636f9f8abe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:29:39.850 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:29:39.859+0000 7f7df7e0fb80 1 -- 172.21.15.52:0/531955204 >> 172.21.15.52:0/531955204 conn(0x5636f9e0e770 msgr2=0x5636f9f85d90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:29:39.851 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:29:39.860+0000 7f7df7e0fb80 1 -- 172.21.15.52:0/531955204 shutdown_connections 2024-05-27T07:29:39.852 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:29:39.861+0000 7f7df7e0fb80 1 -- 172.21.15.52:0/531955204 wait complete. 2024-05-27T07:29:39.852 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:29:39.862+0000 7f7df7e0fb80 1 -- start start 2024-05-27T07:29:39.853 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:29:39.863+0000 7f7df7e0fb80 1 Processor -- start 2024-05-27T07:29:39.854 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:29:39.863+0000 7f7df7e0fb80 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5636f9e0be40 0x5636f9f94880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:29:39.854 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:29:39.863+0000 7f7df7e0fb80 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5636f9f8b120 0x5636f9f94dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:29:39.854 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:29:39.863+0000 7f7df7e0fb80 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x5636f9f91070 0x5636f9f95300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:29:39.854 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:29:39.863+0000 7f7df7e0fb80 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x5636f9daefa0 con 0x5636f9f91070 2024-05-27T07:29:39.854 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:29:39.863+0000 7f7df7e0fb80 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x5636f9e10da0 con 0x5636f9f8b120 2024-05-27T07:29:39.854 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:29:39.863+0000 7f7df7e0fb80 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x5636f9f97b00 con 0x5636f9e0be40 2024-05-27T07:29:39.854 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:29:39.863+0000 7f7df7e0fb80 10 client.0 _ll_register_callbacks cb 0x5636f9e3f5b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T07:29:39.855 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:29:39.864+0000 7f7de37fe700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x5636f9f91070 0x5636f9f95300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:29:39.855 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:29:39.865+0000 7f7de37fe700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x5636f9f91070 0x5636f9f95300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.52:35874/0 (socket says 172.21.15.52:35874) 2024-05-27T07:29:39.855 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:29:39.865+0000 7f7de37fe700 1 -- 172.21.15.52:0/1622834940 learned_addr learned my addr 172.21.15.52:0/1622834940 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-05-27T07:29:39.855 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:29:39.865+0000 7f7de27fc700 1 --2- 172.21.15.52:0/1622834940 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5636f9f8b120 0x5636f9f94dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:29:39.855 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:29:39.865+0000 7f7de2ffd700 1 --2- 172.21.15.52:0/1622834940 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5636f9e0be40 0x5636f9f94880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:29:39.856 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:29:39.866+0000 7f7de37fe700 1 -- 172.21.15.52:0/1622834940 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5636f9e0be40 msgr2=0x5636f9f94880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:29:39.857 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:29:39.866+0000 7f7de37fe700 1 --2- 172.21.15.52:0/1622834940 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5636f9e0be40 0x5636f9f94880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:29:39.857 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:29:39.866+0000 7f7de37fe700 1 -- 172.21.15.52:0/1622834940 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5636f9f8b120 msgr2=0x5636f9f94dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:29:39.857 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:29:39.866+0000 7f7de37fe700 1 --2- 172.21.15.52:0/1622834940 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5636f9f8b120 0x5636f9f94dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:29:39.857 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:29:39.866+0000 7f7de37fe700 1 -- 172.21.15.52:0/1622834940 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5636f9df88d0 con 0x5636f9f91070 2024-05-27T07:29:39.858 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:29:39.868+0000 7f7de37fe700 1 --2- 172.21.15.52:0/1622834940 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x5636f9f91070 0x5636f9f95300 secure :-1 s=READY pgs=614 cs=0 l=1 rev1=1 crypto rx=0x7f7ddc009900 tx=0x7f7ddc0099e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=81bef8fc3a9f3d79 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:29:39.859 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:29:39.868+0000 7f7dc6ff5700 10 client.0 ms_handle_connect on v2:172.21.15.52:3300/0 2024-05-27T07:29:39.859 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:29:39.869+0000 7f7dc6ff5700 1 -- 172.21.15.52:0/1622834940 <== mon.0 v2:172.21.15.52:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7ddc050070 con 0x5636f9f91070 2024-05-27T07:29:39.859 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:29:39.869+0000 7f7df7e0fb80 1 -- 172.21.15.52:0/1622834940 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f7ddc0075c0 con 0x5636f9f91070 2024-05-27T07:29:39.859 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:29:39.869+0000 7f7dc6ff5700 1 -- 172.21.15.52:0/1622834940 <== mon.0 v2:172.21.15.52:3300/0 2 ==== osd_map(106..106 src has 1..106) ==== 9379+0+0 (secure 0 0 0) 0x7f7ddc007770 con 0x5636f9f91070 2024-05-27T07:29:39.861 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:29:39.870+0000 7f7dc6ff5700 1 -- 172.21.15.52:0/1622834940 <== mon.0 v2:172.21.15.52:3300/0 3 ==== mon_get_version_reply(handle=1 version=65) ==== 24+0+0 (secure 0 0 0) 0x7f7ddc05c050 con 0x5636f9f91070 2024-05-27T07:29:39.863 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:29:39.872+0000 7f7df7e0fb80 10 client.16311 fetch_fsmap learned FSMap version 65 2024-05-27T07:29:39.863 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:29:39.872+0000 7f7df7e0fb80 1 -- 172.21.15.52:0/1622834940 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({config=0+,fsmap=65,monmap=4+}) -- 0x5636f9e62760 con 0x5636f9f91070 2024-05-27T07:29:39.866 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:29:39.876+0000 7f7dc6ff5700 1 -- 172.21.15.52:0/1622834940 <== mon.0 v2:172.21.15.52:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f7ddc0075c0 con 0x5636f9f91070 2024-05-27T07:29:39.866 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:29:39.876+0000 7f7dc6ff5700 1 -- 172.21.15.52:0/1622834940 <== mon.0 v2:172.21.15.52:3300/0 5 ==== fsmap(e 65) ==== 7168+0+0 (secure 0 0 0) 0x7f7ddc011070 con 0x5636f9f91070 2024-05-27T07:29:39.866 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:29:39.876+0000 7f7df7e0fb80 10 client.16311 fetch_fsmap finished waiting for FSMap version 65 2024-05-27T07:29:39.866 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:29:39.876+0000 7f7df7e0fb80 10 client.16311 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.c 2024-05-27T07:29:39.866 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:29:39.876+0000 7f7df7e0fb80 20 client.16311 populate_metadata read hostname 'smithi052' 2024-05-27T07:29:39.867 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:29:39.876+0000 7f7df7e0fb80 1 --2- 172.21.15.52:0/1622834940 >> [v2:172.21.15.173:6836/1396363827,v1:172.21.15.173:6837/1396363827] conn(0x5636f9fa4740 0x5636f9fa6b60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:29:39.867 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:29:39.876+0000 7f7df7e0fb80 4 client.16311 mds_command: new command op to 24466 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-27T07:29:39.867 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:29:39.876+0000 7f7df7e0fb80 1 -- 172.21.15.52:0/1622834940 --> [v2:172.21.15.173:6836/1396363827,v1:172.21.15.173:6837/1396363827] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5636f9ebacc0 con 0x5636f9fa4740 2024-05-27T07:29:39.870 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:29:39.878+0000 7f7de2ffd700 1 --2- 172.21.15.52:0/1622834940 >> [v2:172.21.15.173:6836/1396363827,v1:172.21.15.173:6837/1396363827] conn(0x5636f9fa4740 0x5636f9fa6b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:29:39.875 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:29:39.885+0000 7f7de2ffd700 1 --2- 172.21.15.52:0/1622834940 >> [v2:172.21.15.173:6836/1396363827,v1:172.21.15.173:6837/1396363827] conn(0x5636f9fa4740 0x5636f9fa6b60 crc :-1 s=READY pgs=38 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=ad932fcbfe03f434 server_cookie=407e6b6455807fa8 in_seq=0 out_seq=0 2024-05-27T07:29:39.875 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:29:39.885+0000 7f7dc6ff5700 10 client.16311 ms_handle_connect on v2:172.21.15.173:6836/1396363827 2024-05-27T07:29:39.876 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:29:39.885+0000 7f7dc6ff5700 1 -- 172.21.15.52:0/1622834940 <== mds.4 v2:172.21.15.173:6836/1396363827 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24041 (crc 0 0 0) 0x5636f9ebacc0 con 0x5636f9fa4740 2024-05-27T07:29:39.876 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:29:39.885+0000 7f7dc6ff5700 10 client.16311 handle_command_reply: tid=0 2024-05-27T07:29:39.876 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:29:39.886+0000 7f7df7e0fb80 1 client.16311 shutdown 2024-05-27T07:29:39.880 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:29:39.890+0000 7f7df7e0fb80 1 -- 172.21.15.52:0/1622834940 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x5636f9f91070 msgr2=0x5636f9f95300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:29:39.880 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:29:39.890+0000 7f7df7e0fb80 1 --2- 172.21.15.52:0/1622834940 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x5636f9f91070 0x5636f9f95300 secure :-1 s=READY pgs=614 cs=0 l=1 rev1=1 crypto rx=0x7f7ddc009900 tx=0x7f7ddc0099e0 comp rx=0 tx=0).stop 2024-05-27T07:29:39.882 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:29:39.892+0000 7f7df7e0fb80 1 -- 172.21.15.52:0/1622834940 shutdown_connections 2024-05-27T07:29:39.883 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:29:39.892+0000 7f7df7e0fb80 1 --2- 172.21.15.52:0/1622834940 >> [v2:172.21.15.173:6836/1396363827,v1:172.21.15.173:6837/1396363827] conn(0x5636f9fa4740 0x5636f9fa6b60 crc :-1 s=READY pgs=38 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:29:39.883 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:29:39.892+0000 7f7df7e0fb80 1 --2- 172.21.15.52:0/1622834940 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x5636f9f91070 0x5636f9f95300 unknown :-1 s=CLOSED pgs=614 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:29:39.883 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:29:39.892+0000 7f7df7e0fb80 1 --2- 172.21.15.52:0/1622834940 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5636f9e0be40 0x5636f9f94880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:29:39.883 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:29:39.892+0000 7f7df7e0fb80 1 --2- 172.21.15.52:0/1622834940 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5636f9f8b120 0x5636f9f94dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:29:39.883 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:29:39.892+0000 7f7df7e0fb80 1 -- 172.21.15.52:0/1622834940 >> 172.21.15.52:0/1622834940 conn(0x5636f9e0e770 msgr2=0x5636f9f89bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:29:39.883 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:29:39.892+0000 7f7dc6ff5700 0 client.16311 ms_handle_reset on v2:172.21.15.173:6836/1396363827 2024-05-27T07:29:39.883 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:29:39.892+0000 7f7df7e0fb80 1 -- 172.21.15.52:0/1622834940 shutdown_connections 2024-05-27T07:29:39.883 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:29:39.892+0000 7f7df7e0fb80 1 -- 172.21.15.52:0/1622834940 wait complete. 2024-05-27T07:29:39.884 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:29:39.893+0000 7f7df7e0fb80 20 client.16311 trim_cache size 0 max 16384 2024-05-27T07:29:39.917 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:29:39.923+0000 7f7df7e0fb80 1 Processor -- start 2024-05-27T07:29:39.917 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:29:39.923+0000 7f7df7e0fb80 1 -- start start 2024-05-27T07:29:39.917 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:29:39.923+0000 7f7df7e0fb80 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5636f9e0be40 0x5636f9f8abe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:29:39.917 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:29:39.923+0000 7f7df7e0fb80 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x5636f9f8b120 0x5636f9f8fef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:29:39.917 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:29:39.924+0000 7f7df7e0fb80 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5636f9f91070 0x5636f9fa0ba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:29:39.917 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:29:39.924+0000 7f7de2ffd700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5636f9e0be40 0x5636f9f8abe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:29:39.917 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:29:39.924+0000 7f7de2ffd700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5636f9e0be40 0x5636f9f8abe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.173:3300/0 says I am v2:172.21.15.52:60224/0 (socket says 172.21.15.52:60224) 2024-05-27T07:29:39.917 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:29:39.924+0000 7f7de2ffd700 1 -- 172.21.15.52:0/3095042446 learned_addr learned my addr 172.21.15.52:0/3095042446 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-05-27T07:29:39.917 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:29:39.924+0000 7f7df7e0fb80 1 -- 172.21.15.52:0/3095042446 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x7f7de411a110 con 0x5636f9f8b120 2024-05-27T07:29:39.918 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:29:39.924+0000 7f7df7e0fb80 1 -- 172.21.15.52:0/3095042446 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f7de4077100 con 0x5636f9f91070 2024-05-27T07:29:39.918 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:29:39.924+0000 7f7df7e0fb80 1 -- 172.21.15.52:0/3095042446 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f7de411a7e0 con 0x5636f9e0be40 2024-05-27T07:29:39.918 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:29:39.924+0000 7f7de37fe700 1 --2- 172.21.15.52:0/3095042446 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5636f9f91070 0x5636f9fa0ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:29:39.918 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:29:39.925+0000 7f7de37fe700 1 -- 172.21.15.52:0/3095042446 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5636f9e0be40 msgr2=0x5636f9f8abe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:29:39.918 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:29:39.925+0000 7f7de37fe700 1 --2- 172.21.15.52:0/3095042446 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5636f9e0be40 0x5636f9f8abe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:29:39.918 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:29:39.925+0000 7f7de37fe700 1 -- 172.21.15.52:0/3095042446 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x5636f9f8b120 msgr2=0x5636f9f8fef0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-05-27T07:29:39.918 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:29:39.925+0000 7f7de37fe700 1 --2- 172.21.15.52:0/3095042446 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x5636f9f8b120 0x5636f9f8fef0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:29:39.918 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:29:39.925+0000 7f7de37fe700 1 -- 172.21.15.52:0/3095042446 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5636f9f93ba0 con 0x5636f9f91070 2024-05-27T07:29:39.918 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:29:39.925+0000 7f7de2ffd700 1 --2- 172.21.15.52:0/3095042446 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5636f9e0be40 0x5636f9f8abe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-27T07:29:39.918 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:29:39.925+0000 7f7de37fe700 1 --2- 172.21.15.52:0/3095042446 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5636f9f91070 0x5636f9fa0ba0 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7f7ddc0119c0 tx=0x7f7ddc00b780 comp rx=0 tx=0).ready entity=mon.2 client_cookie=e1a1d3c488287de1 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:29:39.918 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:29:39.925+0000 7f7dc8ff9700 1 -- 172.21.15.52:0/3095042446 <== mon.2 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7ddc059020 con 0x5636f9f91070 2024-05-27T07:29:39.918 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:29:39.926+0000 7f7dc8ff9700 1 -- 172.21.15.52:0/3095042446 <== mon.2 v2:172.21.15.105:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f7ddc005e60 con 0x5636f9f91070 2024-05-27T07:29:39.920 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:29:39.927+0000 7f7df7e0fb80 1 -- 172.21.15.52:0/3095042446 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5636f9f91070 msgr2=0x5636f9fa0ba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:29:39.921 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:29:39.927+0000 7f7df7e0fb80 1 --2- 172.21.15.52:0/3095042446 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5636f9f91070 0x5636f9fa0ba0 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7f7ddc0119c0 tx=0x7f7ddc00b780 comp rx=0 tx=0).stop 2024-05-27T07:29:39.921 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:29:39.927+0000 7f7df7e0fb80 1 -- 172.21.15.52:0/3095042446 shutdown_connections 2024-05-27T07:29:39.921 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:29:39.927+0000 7f7df7e0fb80 1 --2- 172.21.15.52:0/3095042446 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5636f9f91070 0x5636f9fa0ba0 unknown :-1 s=CLOSED pgs=131 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:29:39.921 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:29:39.927+0000 7f7df7e0fb80 1 --2- 172.21.15.52:0/3095042446 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5636f9e0be40 0x5636f9f8abe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:29:39.921 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:29:39.927+0000 7f7df7e0fb80 1 --2- 172.21.15.52:0/3095042446 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x5636f9f8b120 0x5636f9f8fef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:29:39.921 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:29:39.927+0000 7f7df7e0fb80 1 -- 172.21.15.52:0/3095042446 >> 172.21.15.52:0/3095042446 conn(0x5636f9e0e770 msgr2=0x5636f9f8f7e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:29:39.926 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:29:39.932+0000 7f7df7e0fb80 1 -- 172.21.15.52:0/3095042446 shutdown_connections 2024-05-27T07:29:39.929 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:29:39.938+0000 7f7df7e0fb80 1 -- 172.21.15.52:0/3095042446 wait complete. 2024-05-27T07:29:39.929 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:29:39.938+0000 7f7df7e0fb80 1 -- start start 2024-05-27T07:29:39.929 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:29:39.938+0000 7f7df7e0fb80 1 Processor -- start 2024-05-27T07:29:39.930 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:29:39.939+0000 7f7df7e0fb80 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5636f9e0be40 0x5636f9fac890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:29:39.930 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:29:39.939+0000 7f7df7e0fb80 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5636f9f8b120 0x5636f9f9cd90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:29:39.930 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:29:39.939+0000 7f7df7e0fb80 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x5636f9f91070 0x5636f9f9d2d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:29:39.930 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:29:39.939+0000 7f7df7e0fb80 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x7f7de411af70 con 0x5636f9f91070 2024-05-27T07:29:39.930 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:29:39.939+0000 7f7df7e0fb80 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f7de4076a30 con 0x5636f9e0be40 2024-05-27T07:29:39.931 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:29:39.939+0000 7f7df7e0fb80 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f7de4075c90 con 0x5636f9f8b120 2024-05-27T07:29:39.931 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:29:39.939+0000 7f7df7e0fb80 10 client.0 _ll_register_callbacks cb 0x5636f9e3f5b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T07:29:39.931 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:29:39.941+0000 7f7de37fe700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x5636f9f91070 0x5636f9f9d2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:29:39.931 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:29:39.941+0000 7f7de37fe700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x5636f9f91070 0x5636f9f9d2d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.52:35882/0 (socket says 172.21.15.52:35882) 2024-05-27T07:29:39.931 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:29:39.941+0000 7f7de37fe700 1 -- 172.21.15.52:0/2899785314 learned_addr learned my addr 172.21.15.52:0/2899785314 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-05-27T07:29:39.934 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:29:39.941+0000 7f7de2ffd700 1 --2- 172.21.15.52:0/2899785314 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5636f9e0be40 0x5636f9fac890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:29:39.934 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:29:39.942+0000 7f7de2ffd700 1 -- 172.21.15.52:0/2899785314 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5636f9f8b120 msgr2=0x5636f9f9cd90 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-05-27T07:29:39.934 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:29:39.942+0000 7f7de2ffd700 1 --2- 172.21.15.52:0/2899785314 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5636f9f8b120 0x5636f9f9cd90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:29:39.934 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:29:39.942+0000 7f7de2ffd700 1 -- 172.21.15.52:0/2899785314 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x5636f9f91070 msgr2=0x5636f9f9d2d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:29:39.934 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:29:39.942+0000 7f7de2ffd700 1 --2- 172.21.15.52:0/2899785314 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x5636f9f91070 0x5636f9f9d2d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:29:39.934 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:29:39.942+0000 7f7de2ffd700 1 -- 172.21.15.52:0/2899785314 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5636f9f9d810 con 0x5636f9e0be40 2024-05-27T07:29:39.934 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:29:39.943+0000 7f7de2ffd700 1 --2- 172.21.15.52:0/2899785314 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5636f9e0be40 0x5636f9fac890 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7f7dd4015010 tx=0x7f7dd4015040 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b695f88485886c47 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:29:39.934 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:29:39.943+0000 7f7dc6ff5700 10 client.0 ms_handle_connect on v2:172.21.15.105:3300/0 2024-05-27T07:29:39.939 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:29:39.944+0000 7f7dc6ff5700 1 -- 172.21.15.52:0/2899785314 <== mon.2 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7dd4004230 con 0x5636f9e0be40 2024-05-27T07:29:39.939 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:29:39.944+0000 7f7dc6ff5700 1 -- 172.21.15.52:0/2899785314 <== mon.2 v2:172.21.15.105:3300/0 2 ==== osd_map(106..106 src has 1..106) ==== 9379+0+0 (secure 0 0 0) 0x7f7dd40043d0 con 0x5636f9e0be40 2024-05-27T07:29:39.939 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:29:39.945+0000 7f7df7e0fb80 1 -- 172.21.15.52:0/2899785314 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f7ddc005e60 con 0x5636f9e0be40 2024-05-27T07:29:39.939 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:29:39.945+0000 7f7dc6ff5700 1 -- 172.21.15.52:0/2899785314 <== mon.2 v2:172.21.15.105:3300/0 3 ==== mon_get_version_reply(handle=1 version=65) ==== 24+0+0 (secure 0 0 0) 0x7f7dd4029700 con 0x5636f9e0be40 2024-05-27T07:29:39.940 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:29:39.948+0000 7f7df7e0fb80 10 client.25991 fetch_fsmap learned FSMap version 65 2024-05-27T07:29:39.940 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:29:39.948+0000 7f7df7e0fb80 1 -- 172.21.15.52:0/2899785314 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,fsmap=65,monmap=4+}) -- 0x5636fa00e210 con 0x5636f9e0be40 2024-05-27T07:29:39.940 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:29:39.950+0000 7f7dc6ff5700 1 -- 172.21.15.52:0/2899785314 <== mon.2 v2:172.21.15.105:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f7ddc005e60 con 0x5636f9e0be40 2024-05-27T07:29:39.940 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:29:39.950+0000 7f7dc6ff5700 1 -- 172.21.15.52:0/2899785314 <== mon.2 v2:172.21.15.105:3300/0 5 ==== fsmap(e 65) ==== 7168+0+0 (secure 0 0 0) 0x7f7dd401a040 con 0x5636f9e0be40 2024-05-27T07:29:39.943 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:29:39.953+0000 7f7df7e0fb80 10 client.25991 fetch_fsmap finished waiting for FSMap version 65 2024-05-27T07:29:39.943 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:29:39.953+0000 7f7df7e0fb80 10 client.25991 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.c 2024-05-27T07:29:39.943 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:29:39.953+0000 7f7df7e0fb80 20 client.25991 populate_metadata read hostname 'smithi052' 2024-05-27T07:29:39.943 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:29:39.953+0000 7f7df7e0fb80 1 --2- 172.21.15.52:0/2899785314 >> [v2:172.21.15.173:6836/1396363827,v1:172.21.15.173:6837/1396363827] conn(0x5636fa00e780 0x5636fa010ba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:29:39.943 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:29:39.953+0000 7f7df7e0fb80 4 client.25991 mds_command: new command op to 24466 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-05-27T07:29:39.944 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:29:39.953+0000 7f7df7e0fb80 1 -- 172.21.15.52:0/2899785314 --> [v2:172.21.15.173:6836/1396363827,v1:172.21.15.173:6837/1396363827] -- command(tid 0: {"prefix": "damage ls"}) -- 0x5636f9eb44a0 con 0x5636fa00e780 2024-05-27T07:29:39.944 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:29:39.953+0000 7f7de27fc700 1 --2- 172.21.15.52:0/2899785314 >> [v2:172.21.15.173:6836/1396363827,v1:172.21.15.173:6837/1396363827] conn(0x5636fa00e780 0x5636fa010ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:29:39.952 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:29:39.960+0000 7f7dc6ff5700 10 client.25991 ms_handle_connect on v2:172.21.15.173:6836/1396363827 2024-05-27T07:29:39.952 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:29:39.960+0000 7f7de27fc700 1 --2- 172.21.15.52:0/2899785314 >> [v2:172.21.15.173:6836/1396363827,v1:172.21.15.173:6837/1396363827] conn(0x5636fa00e780 0x5636fa010ba0 crc :-1 s=READY pgs=39 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=27188f19de541545 server_cookie=11f6edd70d16193a in_seq=0 out_seq=0 2024-05-27T07:29:39.953 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:29:39.960+0000 7f7dc6ff5700 1 -- 172.21.15.52:0/2899785314 <== mds.4 v2:172.21.15.173:6836/1396363827 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x5636f9eb44a0 con 0x5636fa00e780 2024-05-27T07:29:39.953 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:29:39.960+0000 7f7dc6ff5700 10 client.25991 handle_command_reply: tid=0 2024-05-27T07:29:39.953 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:29:39.962+0000 7f7df7e0fb80 1 client.25991 shutdown 2024-05-27T07:29:39.953 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:29:39.962+0000 7f7df7e0fb80 1 -- 172.21.15.52:0/2899785314 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5636f9e0be40 msgr2=0x5636f9fac890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:29:39.953 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:29:39.962+0000 7f7df7e0fb80 1 --2- 172.21.15.52:0/2899785314 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5636f9e0be40 0x5636f9fac890 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7f7dd4015010 tx=0x7f7dd4015040 comp rx=0 tx=0).stop 2024-05-27T07:29:39.954 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:29:39.963+0000 7f7df7e0fb80 1 -- 172.21.15.52:0/2899785314 shutdown_connections 2024-05-27T07:29:39.954 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:29:39.963+0000 7f7df7e0fb80 1 --2- 172.21.15.52:0/2899785314 >> [v2:172.21.15.173:6836/1396363827,v1:172.21.15.173:6837/1396363827] conn(0x5636fa00e780 0x5636fa010ba0 crc :-1 s=READY pgs=39 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:29:39.954 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:29:39.963+0000 7f7df7e0fb80 1 --2- 172.21.15.52:0/2899785314 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x5636f9f91070 0x5636f9f9d2d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:29:39.954 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:29:39.963+0000 7f7df7e0fb80 1 --2- 172.21.15.52:0/2899785314 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5636f9e0be40 0x5636f9fac890 unknown :-1 s=CLOSED pgs=132 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:29:39.954 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:29:39.963+0000 7f7df7e0fb80 1 --2- 172.21.15.52:0/2899785314 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5636f9f8b120 0x5636f9f9cd90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:29:39.954 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:29:39.963+0000 7f7dc6ff5700 0 client.25991 ms_handle_reset on v2:172.21.15.173:6836/1396363827 2024-05-27T07:29:39.955 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:29:39.963+0000 7f7df7e0fb80 1 -- 172.21.15.52:0/2899785314 >> 172.21.15.52:0/2899785314 conn(0x5636f9e0e770 msgr2=0x5636f9f8e490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:29:39.955 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:29:39.963+0000 7f7df7e0fb80 1 -- 172.21.15.52:0/2899785314 shutdown_connections 2024-05-27T07:29:39.955 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:29:39.964+0000 7f7df7e0fb80 1 -- 172.21.15.52:0/2899785314 wait complete. 2024-05-27T07:29:39.955 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:29:39.964+0000 7f7df7e0fb80 20 client.25991 trim_cache size 0 max 16384 2024-05-27T07:29:39.956 INFO:teuthology.orchestra.run.smithi052.stdout:[] 2024-05-27T07:29:39.956 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:29:39.965+0000 7f7de93fa700 1 -- 172.21.15.52:0/1066560918 >> [v2:172.21.15.52:6800/1822318384,v1:172.21.15.52:6801/1822318384] conn(0x7f7dcc0b2a20 msgr2=0x7f7dcc0b4f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:29:39.957 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:29:39.965+0000 7f7de93fa700 1 --2- 172.21.15.52:0/1066560918 >> [v2:172.21.15.52:6800/1822318384,v1:172.21.15.52:6801/1822318384] conn(0x7f7dcc0b2a20 0x7f7dcc0b4f30 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x5636f9f872a0 tx=0x7f7dd4009260 comp rx=0 tx=0).stop 2024-05-27T07:29:39.957 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:29:39.965+0000 7f7de93fa700 1 -- 172.21.15.52:0/1066560918 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f7de410b270 msgr2=0x7f7de41afd30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:29:39.957 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:29:39.965+0000 7f7de93fa700 1 --2- 172.21.15.52:0/1066560918 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f7de410b270 0x7f7de41afd30 secure :-1 s=READY pgs=612 cs=0 l=1 rev1=1 crypto rx=0x7f7dd800bf20 tx=0x7f7dd800dc30 comp rx=0 tx=0).stop 2024-05-27T07:29:39.957 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:29:39.965+0000 7f7de93fa700 1 -- 172.21.15.52:0/1066560918 shutdown_connections 2024-05-27T07:29:39.957 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:29:39.965+0000 7f7de93fa700 1 --2- 172.21.15.52:0/1066560918 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f7de411d920 0x7f7de41a9a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:29:39.957 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:29:39.965+0000 7f7de93fa700 1 --2- 172.21.15.52:0/1066560918 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f7de410a920 0x7f7de41af7f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:29:39.957 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:29:39.965+0000 7f7de93fa700 1 --2- 172.21.15.52:0/1066560918 >> [v2:172.21.15.52:6800/1822318384,v1:172.21.15.52:6801/1822318384] conn(0x7f7dcc0b2a20 0x7f7dcc0b4f30 unknown :-1 s=CLOSED pgs=116 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:29:39.957 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:29:39.965+0000 7f7de93fa700 1 --2- 172.21.15.52:0/1066560918 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f7de410b270 0x7f7de41afd30 unknown :-1 s=CLOSED pgs=612 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:29:39.957 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:29:39.965+0000 7f7de93fa700 1 -- 172.21.15.52:0/1066560918 >> 172.21.15.52:0/1066560918 conn(0x7f7de406f4c0 msgr2=0x7f7de411b3d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:29:39.957 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:29:39.966+0000 7f7de93fa700 1 -- 172.21.15.52:0/1066560918 shutdown_connections 2024-05-27T07:29:39.957 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:29:39.966+0000 7f7de93fa700 1 -- 172.21.15.52:0/1066560918 wait complete. 2024-05-27T07:29:40.119 INFO:journalctl@ceph.mds.a.smithi052.stdout:May 27 07:29:39 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-a[105634]: starting mds.a at 2024-05-27T07:29:40.119 INFO:journalctl@ceph.mds.a.smithi052.stdout:May 27 07:29:39 smithi052 bash[105616]: c5447791e4a6f6b5a3928fe2dcc2df1539dc407ed035bafd6e3e0e3e6691c4ef 2024-05-27T07:29:40.119 INFO:journalctl@ceph.mds.a.smithi052.stdout:May 27 07:29:39 smithi052 systemd[1]: Started Ceph mds.a for 73df2636-1bf8-11ef-bc9b-c7b262605968. 2024-05-27T07:29:40.412 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:29:40 smithi052 ceph-mon[31955]: pgmap v129: 129 pgs: 129 active+clean; 215 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.8 MiB/s rd, 2.1 MiB/s wr, 1.26k op/s 2024-05-27T07:29:40.412 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:29:40 smithi052 ceph-mon[31955]: mds.? [v2:172.21.15.105:6832/4264391145,v1:172.21.15.105:6833/4264391145] up:boot 2024-05-27T07:29:40.413 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:29:40 smithi052 ceph-mon[31955]: MDS daemon mds.a is removed because it is dead or otherwise unavailable. 2024-05-27T07:29:40.413 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:29:40 smithi052 ceph-mon[31955]: fsmap cephfs:5 5 up:standby-replay 1 up:standby 5 up:active 2024-05-27T07:29:40.413 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:29:40 smithi052 ceph-mon[31955]: from='mgr.15333 172.21.15.52:0/416389082' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "k"}]: dispatch 2024-05-27T07:29:40.413 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:29:40 smithi052 ceph-mon[31955]: fsmap cephfs:5 5 up:standby-replay 5 up:active 2024-05-27T07:29:40.440 INFO:journalctl@ceph.mds.k.smithi105.stdout:May 27 07:29:40 smithi105 podman[69873]: 2024-05-27 07:29:40.142912651 +0000 UTC m=+0.957462998 container remove 08bbfdf102e0774d695c3546b67fd704eb12702e615eb396bd27fb68becc408e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-k, ceph=True, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, GIT_CLEAN=True, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, org.label-schema.build-date=20240521, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0) 2024-05-27T07:29:40.440 INFO:journalctl@ceph.mds.k.smithi105.stdout:May 27 07:29:40 smithi105 systemd[1]: ceph-73df2636-1bf8-11ef-bc9b-c7b262605968@mds.k.service: Main process exited, code=exited, status=134/n/a 2024-05-27T07:29:40.440 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:29:40 smithi105 ceph-mon[35977]: pgmap v129: 129 pgs: 129 active+clean; 215 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.8 MiB/s rd, 2.1 MiB/s wr, 1.26k op/s 2024-05-27T07:29:40.440 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:29:40 smithi105 ceph-mon[35977]: mds.? [v2:172.21.15.105:6832/4264391145,v1:172.21.15.105:6833/4264391145] up:boot 2024-05-27T07:29:40.440 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:29:40 smithi105 ceph-mon[35977]: MDS daemon mds.a is removed because it is dead or otherwise unavailable. 2024-05-27T07:29:40.441 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:29:40 smithi105 ceph-mon[35977]: fsmap cephfs:5 5 up:standby-replay 1 up:standby 5 up:active 2024-05-27T07:29:40.441 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:29:40 smithi105 ceph-mon[35977]: from='mgr.15333 172.21.15.52:0/416389082' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "k"}]: dispatch 2024-05-27T07:29:40.441 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:29:40 smithi105 ceph-mon[35977]: fsmap cephfs:5 5 up:standby-replay 5 up:active 2024-05-27T07:29:40.602 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:29:40 smithi173 ceph-mon[37000]: pgmap v129: 129 pgs: 129 active+clean; 215 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.8 MiB/s rd, 2.1 MiB/s wr, 1.26k op/s 2024-05-27T07:29:40.602 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:29:40 smithi173 ceph-mon[37000]: mds.? [v2:172.21.15.105:6832/4264391145,v1:172.21.15.105:6833/4264391145] up:boot 2024-05-27T07:29:40.602 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:29:40 smithi173 ceph-mon[37000]: MDS daemon mds.a is removed because it is dead or otherwise unavailable. 2024-05-27T07:29:40.602 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:29:40 smithi173 ceph-mon[37000]: fsmap cephfs:5 5 up:standby-replay 1 up:standby 5 up:active 2024-05-27T07:29:40.602 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:29:40 smithi173 ceph-mon[37000]: from='mgr.15333 172.21.15.52:0/416389082' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "k"}]: dispatch 2024-05-27T07:29:40.603 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:29:40 smithi173 ceph-mon[37000]: fsmap cephfs:5 5 up:standby-replay 5 up:active 2024-05-27T07:29:40.830 DEBUG:teuthology.orchestra.run.smithi052:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:38f9d900c83c63799fbdbe61acc9a11b0d3554a6 shell --fsid 73df2636-1bf8-11ef-bc9b-c7b262605968 -- ceph tell mds.1:3 damage ls 2024-05-27T07:29:40.940 INFO:journalctl@ceph.mds.k.smithi105.stdout:May 27 07:29:40 smithi105 systemd[1]: ceph-73df2636-1bf8-11ef-bc9b-c7b262605968@mds.k.service: Failed with result 'exit-code'. 2024-05-27T07:29:41.097 INFO:teuthology.orchestra.run.smithi052.stderr:Inferring config /var/lib/ceph/73df2636-1bf8-11ef-bc9b-c7b262605968/mon.a/config 2024-05-27T07:29:41.440 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:29:41 smithi105 ceph-mon[35977]: mds.? [v2:172.21.15.52:6834/1711818766,v1:172.21.15.52:6835/1711818766] up:boot 2024-05-27T07:29:41.440 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:29:41 smithi105 ceph-mon[35977]: fsmap cephfs:5 5 up:standby-replay 1 up:standby 5 up:active 2024-05-27T07:29:41.440 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:29:41 smithi105 ceph-mon[35977]: from='mgr.15333 172.21.15.52:0/416389082' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "a"}]: dispatch 2024-05-27T07:29:41.480 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:29:41 smithi052 ceph-mon[31955]: mds.? [v2:172.21.15.52:6834/1711818766,v1:172.21.15.52:6835/1711818766] up:boot 2024-05-27T07:29:41.480 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:29:41 smithi052 ceph-mon[31955]: fsmap cephfs:5 5 up:standby-replay 1 up:standby 5 up:active 2024-05-27T07:29:41.480 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:29:41 smithi052 ceph-mon[31955]: from='mgr.15333 172.21.15.52:0/416389082' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "a"}]: dispatch 2024-05-27T07:29:41.602 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:29:41 smithi173 ceph-mon[37000]: mds.? [v2:172.21.15.52:6834/1711818766,v1:172.21.15.52:6835/1711818766] up:boot 2024-05-27T07:29:41.602 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:29:41 smithi173 ceph-mon[37000]: fsmap cephfs:5 5 up:standby-replay 1 up:standby 5 up:active 2024-05-27T07:29:41.602 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:29:41 smithi173 ceph-mon[37000]: from='mgr.15333 172.21.15.52:0/416389082' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "a"}]: dispatch 2024-05-27T07:29:42.440 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:29:42 smithi105 ceph-mon[35977]: pgmap v130: 129 pgs: 129 active+clean; 224 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 8.6 MiB/s rd, 2.3 MiB/s wr, 884 op/s 2024-05-27T07:29:42.480 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:29:42 smithi052 ceph-mon[31955]: pgmap v130: 129 pgs: 129 active+clean; 224 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 8.6 MiB/s rd, 2.3 MiB/s wr, 884 op/s 2024-05-27T07:29:42.602 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:29:42 smithi173 ceph-mon[37000]: pgmap v130: 129 pgs: 129 active+clean; 224 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 8.6 MiB/s rd, 2.3 MiB/s wr, 884 op/s 2024-05-27T07:29:43.383 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:29:43.392+0000 7f4ea4f39700 1 -- 172.21.15.52:0/2248612387 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f4ea0108ef0 msgr2=0x7f4ea0109300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:29:43.383 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:29:43.392+0000 7f4ea4f39700 1 --2- 172.21.15.52:0/2248612387 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f4ea0108ef0 0x7f4ea0109300 secure :-1 s=READY pgs=615 cs=0 l=1 rev1=1 crypto rx=0x7f4e940099c0 tx=0x7f4e94009cd0 comp rx=0 tx=0).stop 2024-05-27T07:29:43.383 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:29:43.392+0000 7f4ea4f39700 1 -- 172.21.15.52:0/2248612387 shutdown_connections 2024-05-27T07:29:43.383 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:29:43.392+0000 7f4ea4f39700 1 --2- 172.21.15.52:0/2248612387 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f4ea0109840 0x7f4ea01111a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:29:43.384 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:29:43.392+0000 7f4ea4f39700 1 --2- 172.21.15.52:0/2248612387 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f4ea01045c0 0x7f4ea01049d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:29:43.384 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:29:43.392+0000 7f4ea4f39700 1 --2- 172.21.15.52:0/2248612387 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f4ea0108ef0 0x7f4ea0109300 unknown :-1 s=CLOSED pgs=615 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:29:43.384 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:29:43.392+0000 7f4ea4f39700 1 -- 172.21.15.52:0/2248612387 >> 172.21.15.52:0/2248612387 conn(0x7f4ea00ffe90 msgr2=0x7f4ea01022e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:29:43.384 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:29:43.392+0000 7f4ea4f39700 1 -- 172.21.15.52:0/2248612387 shutdown_connections 2024-05-27T07:29:43.384 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:29:43.392+0000 7f4ea4f39700 1 -- 172.21.15.52:0/2248612387 wait complete. 2024-05-27T07:29:43.385 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:29:43.393+0000 7f4ea4f39700 1 Processor -- start 2024-05-27T07:29:43.385 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:29:43.393+0000 7f4ea4f39700 1 -- start start 2024-05-27T07:29:43.385 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:29:43.394+0000 7f4ea4f39700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f4ea01045c0 0x7f4ea019ae80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:29:43.386 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:29:43.394+0000 7f4ea4f39700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f4ea0108ef0 0x7f4ea019b420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:29:43.386 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:29:43.394+0000 7f4ea4f39700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f4ea0109840 0x7f4ea019f6b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:29:43.386 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:29:43.394+0000 7f4ea4f39700 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x7f4ea0114450 con 0x7f4ea0108ef0 2024-05-27T07:29:43.386 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:29:43.394+0000 7f4ea4f39700 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f4ea01142d0 con 0x7f4ea0109840 2024-05-27T07:29:43.386 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:29:43.394+0000 7f4ea4f39700 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f4ea01145d0 con 0x7f4ea01045c0 2024-05-27T07:29:43.386 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:29:43.394+0000 7f4e9dd9b700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f4ea0108ef0 0x7f4ea019b420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:29:43.386 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:29:43.394+0000 7f4e9ed9d700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f4ea0109840 0x7f4ea019f6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:29:43.386 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:29:43.394+0000 7f4e9e59c700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f4ea01045c0 0x7f4ea019ae80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:29:43.386 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:29:43.394+0000 7f4e9ed9d700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f4ea0109840 0x7f4ea019f6b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.52:45238/0 (socket says 172.21.15.52:45238) 2024-05-27T07:29:43.386 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:29:43.394+0000 7f4e9dd9b700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f4ea0108ef0 0x7f4ea019b420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.52:36360/0 (socket says 172.21.15.52:36360) 2024-05-27T07:29:43.386 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:29:43.394+0000 7f4e9ed9d700 1 -- 172.21.15.52:0/3944493490 learned_addr learned my addr 172.21.15.52:0/3944493490 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-05-27T07:29:43.386 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:29:43.394+0000 7f4e9e59c700 1 -- 172.21.15.52:0/3944493490 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f4ea0109840 msgr2=0x7f4ea019f6b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:29:43.387 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:29:43.394+0000 7f4e9e59c700 1 --2- 172.21.15.52:0/3944493490 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f4ea0109840 0x7f4ea019f6b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:29:43.387 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:29:43.394+0000 7f4e9e59c700 1 -- 172.21.15.52:0/3944493490 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f4ea0108ef0 msgr2=0x7f4ea019b420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:29:43.387 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:29:43.394+0000 7f4e9e59c700 1 --2- 172.21.15.52:0/3944493490 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f4ea0108ef0 0x7f4ea019b420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:29:43.387 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:29:43.394+0000 7f4e9e59c700 1 -- 172.21.15.52:0/3944493490 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4ea019fd50 con 0x7f4ea01045c0 2024-05-27T07:29:43.387 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:29:43.394+0000 7f4e9ed9d700 1 --2- 172.21.15.52:0/3944493490 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f4ea0109840 0x7f4ea019f6b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-27T07:29:43.387 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:29:43.394+0000 7f4e9dd9b700 1 --2- 172.21.15.52:0/3944493490 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f4ea0108ef0 0x7f4ea019b420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-27T07:29:43.387 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:29:43.394+0000 7f4e9e59c700 1 --2- 172.21.15.52:0/3944493490 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f4ea01045c0 0x7f4ea019ae80 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7f4e9000c970 tx=0x7f4e9000cc80 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:29:43.387 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:29:43.395+0000 7f4e8f7fe700 1 -- 172.21.15.52:0/3944493490 <== mon.1 v2:172.21.15.173:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4e90004a30 con 0x7f4ea01045c0 2024-05-27T07:29:43.387 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:29:43.395+0000 7f4ea4f39700 1 -- 172.21.15.52:0/3944493490 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4ea01a0040 con 0x7f4ea01045c0 2024-05-27T07:29:43.387 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:29:43.395+0000 7f4e8f7fe700 1 -- 172.21.15.52:0/3944493490 <== mon.1 v2:172.21.15.173:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f4e900054d0 con 0x7f4ea01045c0 2024-05-27T07:29:43.387 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:29:43.395+0000 7f4e8f7fe700 1 -- 172.21.15.52:0/3944493490 <== mon.1 v2:172.21.15.173:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4e9000eb40 con 0x7f4ea01045c0 2024-05-27T07:29:43.387 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:29:43.395+0000 7f4ea4f39700 1 -- 172.21.15.52:0/3944493490 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4ea01a7960 con 0x7f4ea01045c0 2024-05-27T07:29:43.388 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:29:43.397+0000 7f4e8f7fe700 1 -- 172.21.15.52:0/3944493490 <== mon.1 v2:172.21.15.173:3300/0 4 ==== mgrmap(e 26) ==== 149783+0+0 (secure 0 0 0) 0x7f4e9000ed00 con 0x7f4ea01045c0 2024-05-27T07:29:43.389 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:29:43.398+0000 7f4e8f7fe700 1 --2- 172.21.15.52:0/3944493490 >> [v2:172.21.15.52:6800/1822318384,v1:172.21.15.52:6801/1822318384] conn(0x7f4e880b29f0 0x7f4e880b4f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:29:43.389 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:29:43.398+0000 7f4eb394eb80 1 Processor -- start 2024-05-27T07:29:43.389 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:29:43.398+0000 7f4e9dd9b700 1 --2- 172.21.15.52:0/3944493490 >> [v2:172.21.15.52:6800/1822318384,v1:172.21.15.52:6801/1822318384] conn(0x7f4e880b29f0 0x7f4e880b4f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:29:43.389 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:29:43.399+0000 7f4e8f7fe700 1 -- 172.21.15.52:0/3944493490 <== mon.1 v2:172.21.15.173:3300/0 5 ==== osd_map(106..106 src has 1..106) ==== 9379+0+0 (secure 0 0 0) 0x7f4e900e2400 con 0x7f4ea01045c0 2024-05-27T07:29:43.389 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:29:43.399+0000 7f4eb394eb80 1 -- start start 2024-05-27T07:29:43.390 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:29:43.399+0000 7f4eb394eb80 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55c57a787e40 0x55c57a90b710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:29:43.391 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:29:43.399+0000 7f4e9dd9b700 1 --2- 172.21.15.52:0/3944493490 >> [v2:172.21.15.52:6800/1822318384,v1:172.21.15.52:6801/1822318384] conn(0x7f4e880b29f0 0x7f4e880b4f00 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f4e94000c00 tx=0x7f4e9401a040 comp rx=0 tx=0).ready entity=mgr.15333 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:29:43.391 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:29:43.399+0000 7f4eb394eb80 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55c57a90bc50 0x55c57a905a40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:29:43.391 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:29:43.399+0000 7f4eb394eb80 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55c57a905f80 0x55c57a906390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:29:43.391 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:29:43.399+0000 7f4eb394eb80 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x55c57a834c90 con 0x55c57a905f80 2024-05-27T07:29:43.391 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:29:43.399+0000 7f4eb394eb80 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x55c57a824830 con 0x55c57a90bc50 2024-05-27T07:29:43.391 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:29:43.399+0000 7f4eb394eb80 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x55c57a828870 con 0x55c57a787e40 2024-05-27T07:29:43.391 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:29:43.399+0000 7f4e9dd9b700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55c57a90bc50 0x55c57a905a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:29:43.391 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:29:43.399+0000 7f4e9dd9b700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55c57a90bc50 0x55c57a905a40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.52:45244/0 (socket says 172.21.15.52:45244) 2024-05-27T07:29:43.391 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:29:43.399+0000 7f4e9ed9d700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55c57a905f80 0x55c57a906390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:29:43.391 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:29:43.399+0000 7f4e9dd9b700 1 -- 172.21.15.52:0/2058468890 learned_addr learned my addr 172.21.15.52:0/2058468890 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-05-27T07:29:43.391 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:29:43.399+0000 7f4e9e59c700 1 --2- 172.21.15.52:0/2058468890 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55c57a787e40 0x55c57a90b710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:29:43.391 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:29:43.399+0000 7f4e9dd9b700 1 -- 172.21.15.52:0/2058468890 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55c57a787e40 msgr2=0x55c57a90b710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:29:43.392 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:29:43.399+0000 7f4e9dd9b700 1 --2- 172.21.15.52:0/2058468890 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55c57a787e40 0x55c57a90b710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:29:43.392 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:29:43.399+0000 7f4e9dd9b700 1 -- 172.21.15.52:0/2058468890 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55c57a905f80 msgr2=0x55c57a906390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:29:43.392 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:29:43.399+0000 7f4e9dd9b700 1 --2- 172.21.15.52:0/2058468890 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55c57a905f80 0x55c57a906390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:29:43.392 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:29:43.399+0000 7f4e9dd9b700 1 -- 172.21.15.52:0/2058468890 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55c57a7748d0 con 0x55c57a90bc50 2024-05-27T07:29:43.392 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:29:43.400+0000 7f4e9dd9b700 1 --2- 172.21.15.52:0/2058468890 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55c57a90bc50 0x55c57a905a40 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7f4e94009750 tx=0x7f4e94022000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=179339ae271351c7 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:29:43.392 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:29:43.400+0000 7f4e8cff9700 1 -- 172.21.15.52:0/2058468890 <== mon.2 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4e940037c0 con 0x55c57a90bc50 2024-05-27T07:29:43.392 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:29:43.400+0000 7f4e8cff9700 1 -- 172.21.15.52:0/2058468890 <== mon.2 v2:172.21.15.105:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f4e94003e10 con 0x55c57a90bc50 2024-05-27T07:29:43.392 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:29:43.400+0000 7f4e8cff9700 1 -- 172.21.15.52:0/2058468890 <== mon.2 v2:172.21.15.105:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4e9400f460 con 0x55c57a90bc50 2024-05-27T07:29:43.395 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:29:43.401+0000 7f4eb394eb80 1 -- 172.21.15.52:0/2058468890 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55c57a90bc50 msgr2=0x55c57a905a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:29:43.395 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:29:43.401+0000 7f4eb394eb80 1 --2- 172.21.15.52:0/2058468890 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55c57a90bc50 0x55c57a905a40 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7f4e94009750 tx=0x7f4e94022000 comp rx=0 tx=0).stop 2024-05-27T07:29:43.395 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:29:43.401+0000 7f4eb394eb80 1 -- 172.21.15.52:0/2058468890 shutdown_connections 2024-05-27T07:29:43.395 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:29:43.401+0000 7f4eb394eb80 1 --2- 172.21.15.52:0/2058468890 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55c57a905f80 0x55c57a906390 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:29:43.395 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:29:43.401+0000 7f4eb394eb80 1 --2- 172.21.15.52:0/2058468890 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55c57a787e40 0x55c57a90b710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:29:43.395 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:29:43.401+0000 7f4eb394eb80 1 --2- 172.21.15.52:0/2058468890 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55c57a90bc50 0x55c57a905a40 unknown :-1 s=CLOSED pgs=133 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:29:43.396 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:29:43.401+0000 7f4eb394eb80 1 -- 172.21.15.52:0/2058468890 >> 172.21.15.52:0/2058468890 conn(0x55c57a7e2af0 msgr2=0x55c57a900640 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:29:43.396 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:29:43.401+0000 7f4eb394eb80 1 -- 172.21.15.52:0/2058468890 shutdown_connections 2024-05-27T07:29:43.396 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:29:43.401+0000 7f4eb394eb80 1 -- 172.21.15.52:0/2058468890 wait complete. 2024-05-27T07:29:43.396 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:29:43.401+0000 7f4eb394eb80 1 -- start start 2024-05-27T07:29:43.396 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:29:43.402+0000 7f4eb394eb80 1 Processor -- start 2024-05-27T07:29:43.396 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:29:43.402+0000 7f4eb394eb80 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55c57a787e40 0x55c57a90f310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:29:43.396 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:29:43.402+0000 7f4eb394eb80 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55c57a905f80 0x55c57a90f850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:29:43.396 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:29:43.402+0000 7f4eb394eb80 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55c57a90bc50 0x55c57a90fd90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:29:43.396 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:29:43.402+0000 7f4eb394eb80 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x55c57a72afa0 con 0x55c57a90bc50 2024-05-27T07:29:43.396 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:29:43.402+0000 7f4eb394eb80 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x55c57a7c67c0 con 0x55c57a905f80 2024-05-27T07:29:43.396 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:29:43.402+0000 7f4eb394eb80 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x55c57a9126e0 con 0x55c57a787e40 2024-05-27T07:29:43.396 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:29:43.402+0000 7f4eb394eb80 10 client.0 _ll_register_callbacks cb 0x55c57a7bf920 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T07:29:43.396 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:29:43.402+0000 7f4e9dd9b700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55c57a905f80 0x55c57a90f850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:29:43.397 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:29:43.402+0000 7f4e9ed9d700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55c57a90bc50 0x55c57a90fd90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:29:43.397 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:29:43.402+0000 7f4e9dd9b700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55c57a905f80 0x55c57a90f850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.52:45260/0 (socket says 172.21.15.52:45260) 2024-05-27T07:29:43.397 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:29:43.402+0000 7f4e9dd9b700 1 -- 172.21.15.52:0/2745403023 learned_addr learned my addr 172.21.15.52:0/2745403023 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-05-27T07:29:43.397 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:29:43.402+0000 7f4e9e59c700 1 --2- 172.21.15.52:0/2745403023 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55c57a787e40 0x55c57a90f310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:29:43.397 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:29:43.403+0000 7f4e9ed9d700 1 -- 172.21.15.52:0/2745403023 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55c57a787e40 msgr2=0x55c57a90f310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:29:43.397 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:29:43.403+0000 7f4e9ed9d700 1 --2- 172.21.15.52:0/2745403023 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55c57a787e40 0x55c57a90f310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:29:43.397 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:29:43.403+0000 7f4e9ed9d700 1 -- 172.21.15.52:0/2745403023 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55c57a905f80 msgr2=0x55c57a90f850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:29:43.397 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:29:43.403+0000 7f4e9ed9d700 1 --2- 172.21.15.52:0/2745403023 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55c57a905f80 0x55c57a90f850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:29:43.397 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:29:43.403+0000 7f4e9ed9d700 1 -- 172.21.15.52:0/2745403023 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55c57a7c3690 con 0x55c57a90bc50 2024-05-27T07:29:43.397 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:29:43.403+0000 7f4e9dd9b700 1 --2- 172.21.15.52:0/2745403023 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55c57a905f80 0x55c57a90f850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-27T07:29:43.397 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:29:43.403+0000 7f4e9ed9d700 1 --2- 172.21.15.52:0/2745403023 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55c57a90bc50 0x55c57a90fd90 secure :-1 s=READY pgs=616 cs=0 l=1 rev1=1 crypto rx=0x7f4e98007fb0 tx=0x7f4e9800e7d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=80a795b4f20ef32f server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:29:43.397 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:29:43.403+0000 7f4e85ffb700 10 client.0 ms_handle_connect on v2:172.21.15.52:3300/0 2024-05-27T07:29:43.397 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:29:43.403+0000 7f4e85ffb700 1 -- 172.21.15.52:0/2745403023 <== mon.0 v2:172.21.15.52:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4e9801a070 con 0x55c57a90bc50 2024-05-27T07:29:43.397 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:29:43.403+0000 7f4e85ffb700 1 -- 172.21.15.52:0/2745403023 <== mon.0 v2:172.21.15.52:3300/0 2 ==== osd_map(106..106 src has 1..106) ==== 9379+0+0 (secure 0 0 0) 0x7f4e98004db0 con 0x55c57a90bc50 2024-05-27T07:29:43.398 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:29:43.403+0000 7f4eb394eb80 1 -- 172.21.15.52:0/2745403023 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f4e94003e10 con 0x55c57a90bc50 2024-05-27T07:29:43.398 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:29:43.403+0000 7f4e85ffb700 1 -- 172.21.15.52:0/2745403023 <== mon.0 v2:172.21.15.52:3300/0 3 ==== mon_get_version_reply(handle=1 version=66) ==== 24+0+0 (secure 0 0 0) 0x7f4e98026050 con 0x55c57a90bc50 2024-05-27T07:29:43.398 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:29:43.403+0000 7f4eb394eb80 10 client.16329 fetch_fsmap learned FSMap version 66 2024-05-27T07:29:43.398 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:29:43.403+0000 7f4eb394eb80 1 -- 172.21.15.52:0/2745403023 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({config=0+,fsmap=66,monmap=4+}) -- 0x55c57a91d170 con 0x55c57a90bc50 2024-05-27T07:29:43.398 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:29:43.403+0000 7f4e85ffb700 1 -- 172.21.15.52:0/2745403023 <== mon.0 v2:172.21.15.52:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f4e94003e10 con 0x55c57a90bc50 2024-05-27T07:29:43.398 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:29:43.404+0000 7f4e85ffb700 1 -- 172.21.15.52:0/2745403023 <== mon.0 v2:172.21.15.52:3300/0 5 ==== fsmap(e 66) ==== 7773+0+0 (secure 0 0 0) 0x7f4e98004590 con 0x55c57a90bc50 2024-05-27T07:29:43.398 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:29:43.404+0000 7f4eb394eb80 10 client.16329 fetch_fsmap finished waiting for FSMap version 66 2024-05-27T07:29:43.398 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:29:43.404+0000 7f4eb394eb80 10 client.16329 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.e 2024-05-27T07:29:43.398 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:29:43.404+0000 7f4eb394eb80 20 client.16329 populate_metadata read hostname 'smithi052' 2024-05-27T07:29:43.398 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:29:43.404+0000 7f4eb394eb80 1 --2- 172.21.15.52:0/2745403023 >> [v2:172.21.15.105:6834/3298498384,v1:172.21.15.105:6835/3298498384] conn(0x55c57a91f850 0x55c57a921c70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:29:43.398 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:29:43.404+0000 7f4eb394eb80 4 client.16329 mds_command: new command op to 24473 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-27T07:29:43.398 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:29:43.404+0000 7f4eb394eb80 1 -- 172.21.15.52:0/2745403023 --> [v2:172.21.15.105:6834/3298498384,v1:172.21.15.105:6835/3298498384] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55c57a8d19f0 con 0x55c57a91f850 2024-05-27T07:29:43.398 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:29:43.404+0000 7f4e9e59c700 1 --2- 172.21.15.52:0/2745403023 >> [v2:172.21.15.105:6834/3298498384,v1:172.21.15.105:6835/3298498384] conn(0x55c57a91f850 0x55c57a921c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:29:43.399 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:29:43.404+0000 7f4e9e59c700 1 --2- 172.21.15.52:0/2745403023 >> [v2:172.21.15.105:6834/3298498384,v1:172.21.15.105:6835/3298498384] conn(0x55c57a91f850 0x55c57a921c70 crc :-1 s=READY pgs=38 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=1488d7fef616436d server_cookie=56264400fbdd8a41 in_seq=0 out_seq=0 2024-05-27T07:29:43.399 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:29:43.404+0000 7f4e85ffb700 10 client.16329 ms_handle_connect on v2:172.21.15.105:6834/3298498384 2024-05-27T07:29:43.399 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:29:43.405+0000 7f4e85ffb700 1 -- 172.21.15.52:0/2745403023 <== mds.3 v2:172.21.15.105:6834/3298498384 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24041 (crc 0 0 0) 0x55c57a8d19f0 con 0x55c57a91f850 2024-05-27T07:29:43.400 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:29:43.405+0000 7f4e85ffb700 10 client.16329 handle_command_reply: tid=0 2024-05-27T07:29:43.400 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:29:43.405+0000 7f4eb394eb80 1 client.16329 shutdown 2024-05-27T07:29:43.400 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:29:43.405+0000 7f4eb394eb80 1 -- 172.21.15.52:0/2745403023 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55c57a90bc50 msgr2=0x55c57a90fd90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:29:43.400 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:29:43.405+0000 7f4eb394eb80 1 --2- 172.21.15.52:0/2745403023 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55c57a90bc50 0x55c57a90fd90 secure :-1 s=READY pgs=616 cs=0 l=1 rev1=1 crypto rx=0x7f4e98007fb0 tx=0x7f4e9800e7d0 comp rx=0 tx=0).stop 2024-05-27T07:29:43.400 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:29:43.405+0000 7f4eb394eb80 1 -- 172.21.15.52:0/2745403023 shutdown_connections 2024-05-27T07:29:43.400 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:29:43.405+0000 7f4eb394eb80 1 --2- 172.21.15.52:0/2745403023 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55c57a90bc50 0x55c57a90fd90 unknown :-1 s=CLOSED pgs=616 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:29:43.400 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:29:43.405+0000 7f4eb394eb80 1 --2- 172.21.15.52:0/2745403023 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55c57a787e40 0x55c57a90f310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:29:43.400 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:29:43.405+0000 7f4eb394eb80 1 --2- 172.21.15.52:0/2745403023 >> [v2:172.21.15.105:6834/3298498384,v1:172.21.15.105:6835/3298498384] conn(0x55c57a91f850 0x55c57a921c70 crc :-1 s=READY pgs=38 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:29:43.400 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:29:43.405+0000 7f4eb394eb80 1 --2- 172.21.15.52:0/2745403023 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55c57a905f80 0x55c57a90f850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:29:43.400 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:29:43.405+0000 7f4eb394eb80 1 -- 172.21.15.52:0/2745403023 >> 172.21.15.52:0/2745403023 conn(0x55c57a7e2af0 msgr2=0x55c57a904a30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:29:43.400 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:29:43.405+0000 7f4e85ffb700 0 client.16329 ms_handle_reset on v2:172.21.15.105:6834/3298498384 2024-05-27T07:29:43.400 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:29:43.406+0000 7f4eb394eb80 1 -- 172.21.15.52:0/2745403023 shutdown_connections 2024-05-27T07:29:43.400 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:29:43.406+0000 7f4eb394eb80 1 -- 172.21.15.52:0/2745403023 wait complete. 2024-05-27T07:29:43.401 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:29:43.406+0000 7f4eb394eb80 20 client.16329 trim_cache size 0 max 16384 2024-05-27T07:29:43.413 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:29:43.421+0000 7f4eb394eb80 1 Processor -- start 2024-05-27T07:29:43.413 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:29:43.422+0000 7f4eb394eb80 1 -- start start 2024-05-27T07:29:43.413 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:29:43.422+0000 7f4eb394eb80 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55c57a787e40 0x55c57a905a40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:29:43.413 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:29:43.422+0000 7f4eb394eb80 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55c57a905f80 0x55c57a90fbd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:29:43.413 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:29:43.422+0000 7f4eb394eb80 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55c57a90bc50 0x55c57a8fae00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:29:43.413 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:29:43.422+0000 7f4eb394eb80 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x7f4ea01b2030 con 0x55c57a905f80 2024-05-27T07:29:43.413 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:29:43.422+0000 7f4eb394eb80 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f4ea01afd60 con 0x55c57a787e40 2024-05-27T07:29:43.413 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:29:43.422+0000 7f4eb394eb80 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f4ea01b1960 con 0x55c57a90bc50 2024-05-27T07:29:43.413 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:29:43.422+0000 7f4e9dd9b700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55c57a905f80 0x55c57a90fbd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:29:43.413 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:29:43.422+0000 7f4e9e59c700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55c57a787e40 0x55c57a905a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:29:43.413 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:29:43.422+0000 7f4e9dd9b700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55c57a905f80 0x55c57a90fbd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.52:36396/0 (socket says 172.21.15.52:36396) 2024-05-27T07:29:43.414 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:29:43.422+0000 7f4e9dd9b700 1 -- 172.21.15.52:0/366486784 learned_addr learned my addr 172.21.15.52:0/366486784 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-05-27T07:29:43.414 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:29:43.422+0000 7f4e9ed9d700 1 --2- 172.21.15.52:0/366486784 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55c57a90bc50 0x55c57a8fae00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:29:43.414 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:29:43.422+0000 7f4e9dd9b700 1 -- 172.21.15.52:0/366486784 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55c57a90bc50 msgr2=0x55c57a8fae00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:29:43.414 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:29:43.422+0000 7f4e9dd9b700 1 --2- 172.21.15.52:0/366486784 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55c57a90bc50 0x55c57a8fae00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:29:43.414 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:29:43.422+0000 7f4e9dd9b700 1 -- 172.21.15.52:0/366486784 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55c57a787e40 msgr2=0x55c57a905a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:29:43.414 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:29:43.422+0000 7f4e9dd9b700 1 --2- 172.21.15.52:0/366486784 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55c57a787e40 0x55c57a905a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:29:43.414 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:29:43.422+0000 7f4e9dd9b700 1 -- 172.21.15.52:0/366486784 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55c57a910190 con 0x55c57a905f80 2024-05-27T07:29:43.414 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:29:43.422+0000 7f4e9ed9d700 1 --2- 172.21.15.52:0/366486784 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55c57a90bc50 0x55c57a8fae00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-27T07:29:43.414 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:29:43.422+0000 7f4e9dd9b700 1 --2- 172.21.15.52:0/366486784 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55c57a905f80 0x55c57a90fbd0 secure :-1 s=READY pgs=617 cs=0 l=1 rev1=1 crypto rx=0x7f4e94005010 tx=0x7f4e94030a60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fcbbe9d3b6fce8a9 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:29:43.418 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:29:43.423+0000 7f4e87fff700 1 -- 172.21.15.52:0/366486784 <== mon.0 v2:172.21.15.52:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4e94030480 con 0x55c57a905f80 2024-05-27T07:29:43.418 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:29:43.423+0000 7f4e87fff700 1 -- 172.21.15.52:0/366486784 <== mon.0 v2:172.21.15.52:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f4e9403d070 con 0x55c57a905f80 2024-05-27T07:29:43.418 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:29:43.423+0000 7f4e87fff700 1 -- 172.21.15.52:0/366486784 <== mon.0 v2:172.21.15.52:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4e9400f9c0 con 0x55c57a905f80 2024-05-27T07:29:43.418 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:29:43.423+0000 7f4eb394eb80 1 -- 172.21.15.52:0/366486784 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55c57a905f80 msgr2=0x55c57a90fbd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:29:43.418 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:29:43.423+0000 7f4eb394eb80 1 --2- 172.21.15.52:0/366486784 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55c57a905f80 0x55c57a90fbd0 secure :-1 s=READY pgs=617 cs=0 l=1 rev1=1 crypto rx=0x7f4e94005010 tx=0x7f4e94030a60 comp rx=0 tx=0).stop 2024-05-27T07:29:43.418 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:29:43.423+0000 7f4eb394eb80 1 -- 172.21.15.52:0/366486784 shutdown_connections 2024-05-27T07:29:43.418 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:29:43.423+0000 7f4eb394eb80 1 --2- 172.21.15.52:0/366486784 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55c57a90bc50 0x55c57a8fae00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:29:43.418 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:29:43.423+0000 7f4eb394eb80 1 --2- 172.21.15.52:0/366486784 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55c57a787e40 0x55c57a905a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:29:43.418 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:29:43.423+0000 7f4eb394eb80 1 --2- 172.21.15.52:0/366486784 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55c57a905f80 0x55c57a90fbd0 unknown :-1 s=CLOSED pgs=617 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:29:43.419 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:29:43.423+0000 7f4eb394eb80 1 -- 172.21.15.52:0/366486784 >> 172.21.15.52:0/366486784 conn(0x55c57a7e2af0 msgr2=0x55c57a8fa6f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:29:43.419 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:29:43.424+0000 7f4eb394eb80 1 -- 172.21.15.52:0/366486784 shutdown_connections 2024-05-27T07:29:43.419 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:29:43.424+0000 7f4eb394eb80 1 -- 172.21.15.52:0/366486784 wait complete. 2024-05-27T07:29:43.419 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:29:43.424+0000 7f4eb394eb80 1 -- start start 2024-05-27T07:29:43.419 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:29:43.424+0000 7f4eb394eb80 1 Processor -- start 2024-05-27T07:29:43.419 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:29:43.424+0000 7f4eb394eb80 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55c57a787e40 0x55c57a917970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:29:43.419 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:29:43.424+0000 7f4eb394eb80 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55c57a905f80 0x55c57a917eb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:29:43.419 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:29:43.424+0000 7f4eb394eb80 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55c57a90bc50 0x55c57a926440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:29:43.419 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:29:43.424+0000 7f4eb394eb80 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x7f4ea01b0bc0 con 0x55c57a90bc50 2024-05-27T07:29:43.419 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:29:43.424+0000 7f4eb394eb80 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f4ea01b04f0 con 0x55c57a905f80 2024-05-27T07:29:43.419 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:29:43.424+0000 7f4eb394eb80 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f4ea01b1290 con 0x55c57a787e40 2024-05-27T07:29:43.419 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:29:43.424+0000 7f4eb394eb80 10 client.0 _ll_register_callbacks cb 0x55c57a7bf920 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T07:29:43.419 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:29:43.424+0000 7f4e9dd9b700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55c57a905f80 0x55c57a917eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:29:43.420 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:29:43.424+0000 7f4e9ed9d700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55c57a90bc50 0x55c57a926440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:29:43.420 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:29:43.424+0000 7f4e9dd9b700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55c57a905f80 0x55c57a917eb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.52:45276/0 (socket says 172.21.15.52:45276) 2024-05-27T07:29:43.420 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:29:43.425+0000 7f4e9dd9b700 1 -- 172.21.15.52:0/2133161883 learned_addr learned my addr 172.21.15.52:0/2133161883 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-05-27T07:29:43.420 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:29:43.425+0000 7f4e9e59c700 1 --2- 172.21.15.52:0/2133161883 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55c57a787e40 0x55c57a917970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:29:43.420 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:29:43.425+0000 7f4e9dd9b700 1 -- 172.21.15.52:0/2133161883 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55c57a787e40 msgr2=0x55c57a917970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:29:43.420 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:29:43.425+0000 7f4e9dd9b700 1 --2- 172.21.15.52:0/2133161883 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55c57a787e40 0x55c57a917970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:29:43.420 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:29:43.425+0000 7f4e9dd9b700 1 -- 172.21.15.52:0/2133161883 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55c57a90bc50 msgr2=0x55c57a926440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:29:43.420 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:29:43.425+0000 7f4e9dd9b700 1 --2- 172.21.15.52:0/2133161883 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55c57a90bc50 0x55c57a926440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:29:43.420 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:29:43.425+0000 7f4e9dd9b700 1 -- 172.21.15.52:0/2133161883 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55c57a9184f0 con 0x55c57a905f80 2024-05-27T07:29:43.420 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:29:43.425+0000 7f4e9ed9d700 1 --2- 172.21.15.52:0/2133161883 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55c57a90bc50 0x55c57a926440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-27T07:29:43.420 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:29:43.425+0000 7f4e9dd9b700 1 --2- 172.21.15.52:0/2133161883 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55c57a905f80 0x55c57a917eb0 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7f4e94033010 tx=0x7f4e940306b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c9cca8510f4d79e6 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:29:43.420 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:29:43.425+0000 7f4e85ffb700 10 client.0 ms_handle_connect on v2:172.21.15.105:3300/0 2024-05-27T07:29:43.420 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:29:43.425+0000 7f4e85ffb700 1 -- 172.21.15.52:0/2133161883 <== mon.2 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4e94047070 con 0x55c57a905f80 2024-05-27T07:29:43.420 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:29:43.425+0000 7f4e85ffb700 1 -- 172.21.15.52:0/2133161883 <== mon.2 v2:172.21.15.105:3300/0 2 ==== osd_map(106..106 src has 1..106) ==== 9379+0+0 (secure 0 0 0) 0x7f4e94003d90 con 0x55c57a905f80 2024-05-27T07:29:43.421 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:29:43.425+0000 7f4eb394eb80 1 -- 172.21.15.52:0/2133161883 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f4e9403d070 con 0x55c57a905f80 2024-05-27T07:29:43.421 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:29:43.426+0000 7f4e85ffb700 1 -- 172.21.15.52:0/2133161883 <== mon.2 v2:172.21.15.105:3300/0 3 ==== mon_get_version_reply(handle=1 version=66) ==== 24+0+0 (secure 0 0 0) 0x7f4e94053050 con 0x55c57a905f80 2024-05-27T07:29:43.421 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:29:43.426+0000 7f4eb394eb80 10 client.26009 fetch_fsmap learned FSMap version 66 2024-05-27T07:29:43.421 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:29:43.426+0000 7f4eb394eb80 1 -- 172.21.15.52:0/2133161883 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,fsmap=66,monmap=4+}) -- 0x55c57a989820 con 0x55c57a905f80 2024-05-27T07:29:43.421 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:29:43.426+0000 7f4e85ffb700 1 -- 172.21.15.52:0/2133161883 <== mon.2 v2:172.21.15.105:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f4e9403d070 con 0x55c57a905f80 2024-05-27T07:29:43.421 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:29:43.426+0000 7f4e85ffb700 1 -- 172.21.15.52:0/2133161883 <== mon.2 v2:172.21.15.105:3300/0 5 ==== fsmap(e 66) ==== 7773+0+0 (secure 0 0 0) 0x7f4e94043800 con 0x55c57a905f80 2024-05-27T07:29:43.421 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:29:43.426+0000 7f4eb394eb80 10 client.26009 fetch_fsmap finished waiting for FSMap version 66 2024-05-27T07:29:43.421 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:29:43.426+0000 7f4eb394eb80 10 client.26009 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.e 2024-05-27T07:29:43.421 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:29:43.426+0000 7f4eb394eb80 20 client.26009 populate_metadata read hostname 'smithi052' 2024-05-27T07:29:43.421 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:29:43.426+0000 7f4eb394eb80 1 --2- 172.21.15.52:0/2133161883 >> [v2:172.21.15.105:6834/3298498384,v1:172.21.15.105:6835/3298498384] conn(0x55c57a989a50 0x55c57a98be70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:29:43.421 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:29:43.426+0000 7f4eb394eb80 4 client.26009 mds_command: new command op to 24473 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-05-27T07:29:43.421 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:29:43.426+0000 7f4eb394eb80 1 -- 172.21.15.52:0/2133161883 --> [v2:172.21.15.105:6834/3298498384,v1:172.21.15.105:6835/3298498384] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55c57a836be0 con 0x55c57a989a50 2024-05-27T07:29:43.421 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:29:43.426+0000 7f4e9e59c700 1 --2- 172.21.15.52:0/2133161883 >> [v2:172.21.15.105:6834/3298498384,v1:172.21.15.105:6835/3298498384] conn(0x55c57a989a50 0x55c57a98be70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:29:43.422 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:29:43.427+0000 7f4e9e59c700 1 --2- 172.21.15.52:0/2133161883 >> [v2:172.21.15.105:6834/3298498384,v1:172.21.15.105:6835/3298498384] conn(0x55c57a989a50 0x55c57a98be70 crc :-1 s=READY pgs=39 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=29f1e4dbe80969eb server_cookie=bb5efc92c4b708e1 in_seq=0 out_seq=0 2024-05-27T07:29:43.422 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:29:43.427+0000 7f4e85ffb700 10 client.26009 ms_handle_connect on v2:172.21.15.105:6834/3298498384 2024-05-27T07:29:43.422 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:29:43.427+0000 7f4e85ffb700 1 -- 172.21.15.52:0/2133161883 <== mds.3 v2:172.21.15.105:6834/3298498384 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55c57a836be0 con 0x55c57a989a50 2024-05-27T07:29:43.422 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:29:43.427+0000 7f4e85ffb700 10 client.26009 handle_command_reply: tid=0 2024-05-27T07:29:43.422 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:29:43.427+0000 7f4eb394eb80 1 client.26009 shutdown 2024-05-27T07:29:43.422 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:29:43.427+0000 7f4eb394eb80 1 -- 172.21.15.52:0/2133161883 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55c57a905f80 msgr2=0x55c57a917eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:29:43.422 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:29:43.427+0000 7f4eb394eb80 1 --2- 172.21.15.52:0/2133161883 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55c57a905f80 0x55c57a917eb0 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7f4e94033010 tx=0x7f4e940306b0 comp rx=0 tx=0).stop 2024-05-27T07:29:43.422 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:29:43.427+0000 7f4eb394eb80 1 -- 172.21.15.52:0/2133161883 shutdown_connections 2024-05-27T07:29:43.422 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:29:43.427+0000 7f4eb394eb80 1 --2- 172.21.15.52:0/2133161883 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55c57a90bc50 0x55c57a926440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:29:43.422 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:29:43.427+0000 7f4eb394eb80 1 --2- 172.21.15.52:0/2133161883 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55c57a787e40 0x55c57a917970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:29:43.422 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:29:43.427+0000 7f4eb394eb80 1 --2- 172.21.15.52:0/2133161883 >> [v2:172.21.15.105:6834/3298498384,v1:172.21.15.105:6835/3298498384] conn(0x55c57a989a50 0x55c57a98be70 crc :-1 s=READY pgs=39 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:29:43.422 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:29:43.427+0000 7f4eb394eb80 1 --2- 172.21.15.52:0/2133161883 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55c57a905f80 0x55c57a917eb0 unknown :-1 s=CLOSED pgs=134 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:29:43.423 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:29:43.427+0000 7f4e85ffb700 0 client.26009 ms_handle_reset on v2:172.21.15.105:6834/3298498384 2024-05-27T07:29:43.423 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:29:43.427+0000 7f4eb394eb80 1 -- 172.21.15.52:0/2133161883 >> 172.21.15.52:0/2133161883 conn(0x55c57a7e2af0 msgr2=0x55c57a90a350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:29:43.423 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:29:43.427+0000 7f4eb394eb80 1 -- 172.21.15.52:0/2133161883 shutdown_connections 2024-05-27T07:29:43.423 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:29:43.427+0000 7f4eb394eb80 1 -- 172.21.15.52:0/2133161883 wait complete. 2024-05-27T07:29:43.423 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:29:43.427+0000 7f4eb394eb80 20 client.26009 trim_cache size 0 max 16384 2024-05-27T07:29:43.423 INFO:teuthology.orchestra.run.smithi052.stdout:[] 2024-05-27T07:29:43.424 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:29:43.428+0000 7f4ea4f39700 1 -- 172.21.15.52:0/3944493490 >> [v2:172.21.15.52:6800/1822318384,v1:172.21.15.52:6801/1822318384] conn(0x7f4e880b29f0 msgr2=0x7f4e880b4f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:29:43.424 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:29:43.428+0000 7f4ea4f39700 1 --2- 172.21.15.52:0/3944493490 >> [v2:172.21.15.52:6800/1822318384,v1:172.21.15.52:6801/1822318384] conn(0x7f4e880b29f0 0x7f4e880b4f00 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f4e94000c00 tx=0x7f4e9401a040 comp rx=0 tx=0).stop 2024-05-27T07:29:43.424 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:29:43.428+0000 7f4ea4f39700 1 -- 172.21.15.52:0/3944493490 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f4ea01045c0 msgr2=0x7f4ea019ae80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:29:43.424 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:29:43.428+0000 7f4ea4f39700 1 --2- 172.21.15.52:0/3944493490 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f4ea01045c0 0x7f4ea019ae80 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7f4e9000c970 tx=0x7f4e9000cc80 comp rx=0 tx=0).stop 2024-05-27T07:29:43.424 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:29:43.428+0000 7f4ea4f39700 1 -- 172.21.15.52:0/3944493490 shutdown_connections 2024-05-27T07:29:43.424 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:29:43.428+0000 7f4ea4f39700 1 --2- 172.21.15.52:0/3944493490 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f4ea0109840 0x7f4ea019f6b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:29:43.424 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:29:43.428+0000 7f4ea4f39700 1 --2- 172.21.15.52:0/3944493490 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f4ea01045c0 0x7f4ea019ae80 unknown :-1 s=CLOSED pgs=119 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:29:43.424 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:29:43.428+0000 7f4ea4f39700 1 --2- 172.21.15.52:0/3944493490 >> [v2:172.21.15.52:6800/1822318384,v1:172.21.15.52:6801/1822318384] conn(0x7f4e880b29f0 0x7f4e880b4f00 unknown :-1 s=CLOSED pgs=118 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:29:43.424 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:29:43.428+0000 7f4ea4f39700 1 --2- 172.21.15.52:0/3944493490 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f4ea0108ef0 0x7f4ea019b420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:29:43.425 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:29:43.428+0000 7f4ea4f39700 1 -- 172.21.15.52:0/3944493490 >> 172.21.15.52:0/3944493490 conn(0x7f4ea00ffe90 msgr2=0x7f4ea010a7a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:29:43.425 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:29:43.428+0000 7f4ea4f39700 1 -- 172.21.15.52:0/3944493490 shutdown_connections 2024-05-27T07:29:43.425 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:29:43.428+0000 7f4ea4f39700 1 -- 172.21.15.52:0/3944493490 wait complete. 2024-05-27T07:29:44.125 DEBUG:teuthology.orchestra.run.smithi052:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:38f9d900c83c63799fbdbe61acc9a11b0d3554a6 shell --fsid 73df2636-1bf8-11ef-bc9b-c7b262605968 -- ceph tell mds.1:2 damage ls 2024-05-27T07:29:44.188 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:29:44.197+0000 7fa975c5a640 1 -- 172.21.15.52:0/3507961302 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fa97010a150 msgr2=0x7fa970103d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:29:44.188 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:29:44.197+0000 7fa975c5a640 1 --2- 172.21.15.52:0/3507961302 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fa97010a150 0x7fa970103d90 secure :-1 s=READY pgs=618 cs=0 l=1 rev1=1 crypto rx=0x7fa958009900 tx=0x7fa95802a410 comp rx=0 tx=0).stop 2024-05-27T07:29:44.188 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:29:44.197+0000 7fa975c5a640 1 -- 172.21.15.52:0/3507961302 shutdown_connections 2024-05-27T07:29:44.189 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:29:44.197+0000 7fa975c5a640 1 --2- 172.21.15.52:0/3507961302 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fa970104400 0x7fa9701107d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:29:44.189 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:29:44.197+0000 7fa975c5a640 1 --2- 172.21.15.52:0/3507961302 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fa97010a150 0x7fa970103d90 unknown :-1 s=CLOSED pgs=618 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:29:44.189 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:29:44.197+0000 7fa975c5a640 1 --2- 172.21.15.52:0/3507961302 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fa970109cb0 0x7fa970103850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:29:44.189 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:29:44.197+0000 7fa975c5a640 1 -- 172.21.15.52:0/3507961302 >> 172.21.15.52:0/3507961302 conn(0x7fa970079310 msgr2=0x7fa970079750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:29:44.189 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:29:44.197+0000 7fa975c5a640 1 -- 172.21.15.52:0/3507961302 shutdown_connections 2024-05-27T07:29:44.189 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:29:44.197+0000 7fa975c5a640 1 -- 172.21.15.52:0/3507961302 wait complete. 2024-05-27T07:29:44.191 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:29:44.198+0000 7fa975c5a640 1 Processor -- start 2024-05-27T07:29:44.191 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:29:44.198+0000 7fa975c5a640 1 -- start start 2024-05-27T07:29:44.191 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:29:44.199+0000 7fa975c5a640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fa970104400 0x7fa970074fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:29:44.191 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:29:44.199+0000 7fa975c5a640 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fa970109cb0 0x7fa970075500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:29:44.191 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:29:44.199+0000 7fa975c5a640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fa97010a150 0x7fa970075a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:29:44.192 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:29:44.199+0000 7fa975c5a640 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x7fa970113ea0 con 0x7fa970109cb0 2024-05-27T07:29:44.192 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:29:44.199+0000 7fa975c5a640 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7fa970113d20 con 0x7fa97010a150 2024-05-27T07:29:44.192 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:29:44.199+0000 7fa975c5a640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7fa970114020 con 0x7fa970104400 2024-05-27T07:29:44.192 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:29:44.199+0000 7fa96effd640 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fa970109cb0 0x7fa970075500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:29:44.192 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:29:44.199+0000 7fa96effd640 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fa970109cb0 0x7fa970075500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.52:36410/0 (socket says 172.21.15.52:36410) 2024-05-27T07:29:44.192 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:29:44.199+0000 7fa96effd640 1 -- 172.21.15.52:0/1671073841 learned_addr learned my addr 172.21.15.52:0/1671073841 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-05-27T07:29:44.192 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:29:44.199+0000 7fa96ffff640 1 --2- 172.21.15.52:0/1671073841 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fa97010a150 0x7fa970075a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:29:44.192 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:29:44.199+0000 7fa96f7fe640 1 --2- 172.21.15.52:0/1671073841 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fa970104400 0x7fa970074fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:29:44.192 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:29:44.199+0000 7fa96effd640 1 -- 172.21.15.52:0/1671073841 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fa970104400 msgr2=0x7fa970074fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:29:44.192 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:29:44.199+0000 7fa96effd640 1 --2- 172.21.15.52:0/1671073841 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fa970104400 0x7fa970074fc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:29:44.192 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:29:44.199+0000 7fa96effd640 1 -- 172.21.15.52:0/1671073841 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fa97010a150 msgr2=0x7fa970075a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:29:44.192 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:29:44.199+0000 7fa96effd640 1 --2- 172.21.15.52:0/1671073841 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fa97010a150 0x7fa970075a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:29:44.193 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:29:44.199+0000 7fa96effd640 1 -- 172.21.15.52:0/1671073841 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa9701a8ca0 con 0x7fa970109cb0 2024-05-27T07:29:44.193 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:29:44.199+0000 7fa96ffff640 1 --2- 172.21.15.52:0/1671073841 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fa97010a150 0x7fa970075a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-27T07:29:44.193 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:29:44.199+0000 7fa96effd640 1 --2- 172.21.15.52:0/1671073841 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fa970109cb0 0x7fa970075500 secure :-1 s=READY pgs=619 cs=0 l=1 rev1=1 crypto rx=0x7fa958006010 tx=0x7fa958004060 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:29:44.193 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:29:44.200+0000 7fa96cff9640 1 -- 172.21.15.52:0/1671073841 <== mon.0 v2:172.21.15.52:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa95803f030 con 0x7fa970109cb0 2024-05-27T07:29:44.193 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:29:44.200+0000 7fa96cff9640 1 -- 172.21.15.52:0/1671073841 <== mon.0 v2:172.21.15.52:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fa958003aa0 con 0x7fa970109cb0 2024-05-27T07:29:44.193 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:29:44.200+0000 7fa975c5a640 1 -- 172.21.15.52:0/1671073841 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa9701a8f30 con 0x7fa970109cb0 2024-05-27T07:29:44.193 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:29:44.200+0000 7fa96cff9640 1 -- 172.21.15.52:0/1671073841 <== mon.0 v2:172.21.15.52:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa958035070 con 0x7fa970109cb0 2024-05-27T07:29:44.193 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:29:44.200+0000 7fa975c5a640 1 -- 172.21.15.52:0/1671073841 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa9701a9330 con 0x7fa970109cb0 2024-05-27T07:29:44.196 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:29:44.203+0000 7fa97cbea740 1 Processor -- start 2024-05-27T07:29:44.196 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:29:44.203+0000 7fa97cbea740 1 -- start start 2024-05-27T07:29:44.196 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:29:44.203+0000 7fa97cbea740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55ad8ca261b0 0x55ad8cdc73d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:29:44.196 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:29:44.203+0000 7fa97cbea740 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55ad8c9fd2f0 0x55ad8cdc7910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:29:44.196 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:29:44.203+0000 7fa97cbea740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55ad8ca054d0 0x55ad8cdc7e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:29:44.196 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:29:44.203+0000 7fa97cbea740 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x55ad8cd39c30 con 0x55ad8c9fd2f0 2024-05-27T07:29:44.196 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:29:44.203+0000 7fa97cbea740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x55ad8cd297d0 con 0x55ad8ca054d0 2024-05-27T07:29:44.197 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:29:44.203+0000 7fa97cbea740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x55ad8cd2d810 con 0x55ad8ca261b0 2024-05-27T07:29:44.197 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:29:44.203+0000 7fa96f7fe640 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55ad8c9fd2f0 0x55ad8cdc7910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:29:44.197 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:29:44.203+0000 7fa96f7fe640 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55ad8c9fd2f0 0x55ad8cdc7910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.52:36426/0 (socket says 172.21.15.52:36426) 2024-05-27T07:29:44.197 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:29:44.203+0000 7fa96f7fe640 1 -- 172.21.15.52:0/875307544 learned_addr learned my addr 172.21.15.52:0/875307544 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-05-27T07:29:44.197 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:29:44.203+0000 7fa96ffff640 1 --2- 172.21.15.52:0/875307544 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55ad8ca261b0 0x55ad8cdc73d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:29:44.197 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:29:44.204+0000 7fa96cff9640 1 -- 172.21.15.52:0/1671073841 <== mon.0 v2:172.21.15.52:3300/0 4 ==== mgrmap(e 26) ==== 149783+0+0 (secure 0 0 0) 0x7fa958042030 con 0x7fa970109cb0 2024-05-27T07:29:44.197 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:29:44.204+0000 7fa96f7fe640 1 -- 172.21.15.52:0/875307544 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55ad8ca261b0 msgr2=0x55ad8cdc73d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:29:44.197 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:29:44.204+0000 7fa96f7fe640 1 --2- 172.21.15.52:0/875307544 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55ad8ca261b0 0x55ad8cdc73d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:29:44.197 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:29:44.204+0000 7fa96f7fe640 1 -- 172.21.15.52:0/875307544 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55ad8ca054d0 msgr2=0x55ad8cdc7e50 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-05-27T07:29:44.197 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:29:44.204+0000 7fa96f7fe640 1 --2- 172.21.15.52:0/875307544 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55ad8ca054d0 0x55ad8cdc7e50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:29:44.197 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:29:44.204+0000 7fa96f7fe640 1 -- 172.21.15.52:0/875307544 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55ad8ca3e210 con 0x55ad8c9fd2f0 2024-05-27T07:29:44.197 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:29:44.204+0000 7fa96f7fe640 1 --2- 172.21.15.52:0/875307544 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55ad8c9fd2f0 0x55ad8cdc7910 secure :-1 s=READY pgs=620 cs=0 l=1 rev1=1 crypto rx=0x7fa960009c10 tx=0x7fa960011810 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3a2a4313708b1d19 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:29:44.198 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:29:44.204+0000 7fa965ffb640 1 -- 172.21.15.52:0/875307544 <== mon.0 v2:172.21.15.52:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa960004230 con 0x55ad8c9fd2f0 2024-05-27T07:29:44.198 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:29:44.204+0000 7fa965ffb640 1 -- 172.21.15.52:0/875307544 <== mon.0 v2:172.21.15.52:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fa9600043d0 con 0x55ad8c9fd2f0 2024-05-27T07:29:44.198 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:29:44.204+0000 7fa965ffb640 1 -- 172.21.15.52:0/875307544 <== mon.0 v2:172.21.15.52:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa960004ce0 con 0x55ad8c9fd2f0 2024-05-27T07:29:44.198 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:29:44.205+0000 7fa96cff9640 1 --2- 172.21.15.52:0/1671073841 >> [v2:172.21.15.52:6800/1822318384,v1:172.21.15.52:6801/1822318384] conn(0x7fa95c0b2950 0x7fa95c0b4e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:29:44.198 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:29:44.205+0000 7fa96ffff640 1 --2- 172.21.15.52:0/1671073841 >> [v2:172.21.15.52:6800/1822318384,v1:172.21.15.52:6801/1822318384] conn(0x7fa95c0b2950 0x7fa95c0b4e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:29:44.198 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:29:44.205+0000 7fa96cff9640 1 -- 172.21.15.52:0/1671073841 <== mon.0 v2:172.21.15.52:3300/0 5 ==== osd_map(106..106 src has 1..106) ==== 9379+0+0 (secure 0 0 0) 0x7fa958102d60 con 0x7fa970109cb0 2024-05-27T07:29:44.198 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:29:44.205+0000 7fa97cbea740 1 -- 172.21.15.52:0/875307544 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55ad8c9fd2f0 msgr2=0x55ad8cdc7910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:29:44.198 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:29:44.205+0000 7fa97cbea740 1 --2- 172.21.15.52:0/875307544 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55ad8c9fd2f0 0x55ad8cdc7910 secure :-1 s=READY pgs=620 cs=0 l=1 rev1=1 crypto rx=0x7fa960009c10 tx=0x7fa960011810 comp rx=0 tx=0).stop 2024-05-27T07:29:44.198 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:29:44.205+0000 7fa97cbea740 1 -- 172.21.15.52:0/875307544 shutdown_connections 2024-05-27T07:29:44.198 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:29:44.205+0000 7fa97cbea740 1 --2- 172.21.15.52:0/875307544 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55ad8ca054d0 0x55ad8cdc7e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:29:44.198 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:29:44.205+0000 7fa97cbea740 1 --2- 172.21.15.52:0/875307544 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55ad8c9fd2f0 0x55ad8cdc7910 unknown :-1 s=CLOSED pgs=620 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:29:44.198 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:29:44.205+0000 7fa97cbea740 1 --2- 172.21.15.52:0/875307544 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55ad8ca261b0 0x55ad8cdc73d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:29:44.198 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:29:44.205+0000 7fa97cbea740 1 -- 172.21.15.52:0/875307544 >> 172.21.15.52:0/875307544 conn(0x55ad8ca1a300 msgr2=0x55ad8cdc3b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:29:44.199 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:29:44.205+0000 7fa96ffff640 1 --2- 172.21.15.52:0/1671073841 >> [v2:172.21.15.52:6800/1822318384,v1:172.21.15.52:6801/1822318384] conn(0x7fa95c0b2950 0x7fa95c0b4e60 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x55ad8cdc4540 tx=0x7fa9680096a0 comp rx=0 tx=0).ready entity=mgr.15333 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:29:44.199 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:29:44.205+0000 7fa97cbea740 1 -- 172.21.15.52:0/875307544 shutdown_connections 2024-05-27T07:29:44.199 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:29:44.205+0000 7fa97cbea740 1 -- 172.21.15.52:0/875307544 wait complete. 2024-05-27T07:29:44.199 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:29:44.206+0000 7fa97cbea740 1 -- start start 2024-05-27T07:29:44.201 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:29:44.206+0000 7fa97cbea740 1 Processor -- start 2024-05-27T07:29:44.201 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:29:44.206+0000 7fa97cbea740 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55ad8c9fd2f0 0x55ad8cdd1ad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:29:44.202 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:29:44.206+0000 7fa97cbea740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55ad8ca054d0 0x55ad8cdd2030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:29:44.202 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:29:44.206+0000 7fa97cbea740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55ad8ca261b0 0x55ad8cdd78a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:29:44.202 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:29:44.206+0000 7fa97cbea740 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x55ad8ca0f790 con 0x55ad8c9fd2f0 2024-05-27T07:29:44.202 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:29:44.206+0000 7fa97cbea740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x55ad8cce5890 con 0x55ad8ca054d0 2024-05-27T07:29:44.202 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:29:44.206+0000 7fa97cbea740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x55ad8cdd4500 con 0x55ad8ca261b0 2024-05-27T07:29:44.202 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:29:44.207+0000 7fa96f7fe640 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55ad8c9fd2f0 0x55ad8cdd1ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:29:44.202 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:29:44.207+0000 7fa97cbea740 10 client.0 _ll_register_callbacks cb 0x55ad8ca3da00 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T07:29:44.202 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:29:44.207+0000 7fa96effd640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55ad8ca054d0 0x55ad8cdd2030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:29:44.202 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:29:44.207+0000 7fa96f7fe640 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55ad8c9fd2f0 0x55ad8cdd1ad0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.52:36428/0 (socket says 172.21.15.52:36428) 2024-05-27T07:29:44.202 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:29:44.207+0000 7fa96f7fe640 1 -- 172.21.15.52:0/532167926 learned_addr learned my addr 172.21.15.52:0/532167926 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-05-27T07:29:44.202 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:29:44.207+0000 7fa96ffff640 1 --2- 172.21.15.52:0/532167926 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55ad8ca261b0 0x55ad8cdd78a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:29:44.202 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:29:44.207+0000 7fa96effd640 1 -- 172.21.15.52:0/532167926 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55ad8ca261b0 msgr2=0x55ad8cdd78a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:29:44.203 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:29:44.207+0000 7fa96effd640 1 --2- 172.21.15.52:0/532167926 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55ad8ca261b0 0x55ad8cdd78a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:29:44.203 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:29:44.207+0000 7fa96effd640 1 -- 172.21.15.52:0/532167926 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55ad8c9fd2f0 msgr2=0x55ad8cdd1ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:29:44.203 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:29:44.207+0000 7fa96effd640 1 --2- 172.21.15.52:0/532167926 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55ad8c9fd2f0 0x55ad8cdd1ad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:29:44.203 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:29:44.207+0000 7fa96effd640 1 -- 172.21.15.52:0/532167926 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55ad8cb0a620 con 0x55ad8ca054d0 2024-05-27T07:29:44.203 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:29:44.207+0000 7fa96f7fe640 1 --2- 172.21.15.52:0/532167926 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55ad8c9fd2f0 0x55ad8cdd1ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-27T07:29:44.203 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:29:44.207+0000 7fa96effd640 1 --2- 172.21.15.52:0/532167926 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55ad8ca054d0 0x55ad8cdd2030 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7fa958100150 tx=0x7fa958000bc0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f1b7f34980aeb9d3 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:29:44.203 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:29:44.207+0000 7fa94effd640 10 client.0 ms_handle_connect on v2:172.21.15.105:3300/0 2024-05-27T07:29:44.203 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:29:44.207+0000 7fa94effd640 1 -- 172.21.15.52:0/532167926 <== mon.2 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa958049040 con 0x55ad8ca054d0 2024-05-27T07:29:44.203 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:29:44.207+0000 7fa94effd640 1 -- 172.21.15.52:0/532167926 <== mon.2 v2:172.21.15.105:3300/0 2 ==== osd_map(106..106 src has 1..106) ==== 9379+0+0 (secure 0 0 0) 0x7fa9580c6380 con 0x55ad8ca054d0 2024-05-27T07:29:44.203 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:29:44.207+0000 7fa97cbea740 1 -- 172.21.15.52:0/532167926 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fa9600043d0 con 0x55ad8ca054d0 2024-05-27T07:29:44.203 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:29:44.207+0000 7fa94effd640 1 -- 172.21.15.52:0/532167926 <== mon.2 v2:172.21.15.105:3300/0 3 ==== mon_get_version_reply(handle=1 version=66) ==== 24+0+0 (secure 0 0 0) 0x7fa9580c98a0 con 0x55ad8ca054d0 2024-05-27T07:29:44.204 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:29:44.208+0000 7fa97cbea740 10 client.26018 fetch_fsmap learned FSMap version 66 2024-05-27T07:29:44.204 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:29:44.208+0000 7fa97cbea740 1 -- 172.21.15.52:0/532167926 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,fsmap=66,monmap=4+}) -- 0x55ad8cde0870 con 0x55ad8ca054d0 2024-05-27T07:29:44.204 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:29:44.208+0000 7fa94effd640 1 -- 172.21.15.52:0/532167926 <== mon.2 v2:172.21.15.105:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fa9600043d0 con 0x55ad8ca054d0 2024-05-27T07:29:44.204 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:29:44.208+0000 7fa94effd640 1 -- 172.21.15.52:0/532167926 <== mon.2 v2:172.21.15.105:3300/0 5 ==== fsmap(e 66) ==== 7773+0+0 (secure 0 0 0) 0x7fa9580c6020 con 0x55ad8ca054d0 2024-05-27T07:29:44.204 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:29:44.208+0000 7fa97cbea740 10 client.26018 fetch_fsmap finished waiting for FSMap version 66 2024-05-27T07:29:44.204 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:29:44.208+0000 7fa97cbea740 10 client.26018 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-05-27T07:29:44.204 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:29:44.208+0000 7fa97cbea740 20 client.26018 populate_metadata read hostname 'smithi052' 2024-05-27T07:29:44.204 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:29:44.208+0000 7fa97cbea740 1 --2- 172.21.15.52:0/532167926 >> [v2:172.21.15.105:6838/500348151,v1:172.21.15.105:6839/500348151] conn(0x55ad8c9e4d80 0x55ad8cde3ed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:29:44.204 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:29:44.208+0000 7fa97cbea740 4 client.26018 mds_command: new command op to 24527 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-27T07:29:44.204 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:29:44.208+0000 7fa97cbea740 1 -- 172.21.15.52:0/532167926 --> [v2:172.21.15.105:6838/500348151,v1:172.21.15.105:6839/500348151] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55ad8cd3bb80 con 0x55ad8c9e4d80 2024-05-27T07:29:44.204 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:29:44.208+0000 7fa96f7fe640 1 --2- 172.21.15.52:0/532167926 >> [v2:172.21.15.105:6838/500348151,v1:172.21.15.105:6839/500348151] conn(0x55ad8c9e4d80 0x55ad8cde3ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:29:44.204 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:29:44.209+0000 7fa94effd640 10 client.26018 ms_handle_connect on v2:172.21.15.105:6838/500348151 2024-05-27T07:29:44.205 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:29:44.209+0000 7fa96f7fe640 1 --2- 172.21.15.52:0/532167926 >> [v2:172.21.15.105:6838/500348151,v1:172.21.15.105:6839/500348151] conn(0x55ad8c9e4d80 0x55ad8cde3ed0 crc :-1 s=READY pgs=97 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=dec2519b53f19e6f server_cookie=4b2cbc941f6f927d in_seq=0 out_seq=0 2024-05-27T07:29:44.206 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:29:44.213+0000 7fa94effd640 1 -- 172.21.15.52:0/532167926 <== mds.0 v2:172.21.15.105:6838/500348151 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24041 (crc 0 0 0) 0x55ad8cd3bb80 con 0x55ad8c9e4d80 2024-05-27T07:29:44.206 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:29:44.213+0000 7fa94effd640 10 client.26018 handle_command_reply: tid=0 2024-05-27T07:29:44.206 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:29:44.213+0000 7fa97cbea740 1 client.26018 shutdown 2024-05-27T07:29:44.206 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:29:44.213+0000 7fa97cbea740 1 -- 172.21.15.52:0/532167926 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55ad8ca054d0 msgr2=0x55ad8cdd2030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:29:44.206 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:29:44.213+0000 7fa97cbea740 1 --2- 172.21.15.52:0/532167926 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55ad8ca054d0 0x55ad8cdd2030 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7fa958100150 tx=0x7fa958000bc0 comp rx=0 tx=0).stop 2024-05-27T07:29:44.206 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:29:44.213+0000 7fa97cbea740 1 -- 172.21.15.52:0/532167926 shutdown_connections 2024-05-27T07:29:44.206 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:29:44.213+0000 7fa97cbea740 1 --2- 172.21.15.52:0/532167926 >> [v2:172.21.15.105:6838/500348151,v1:172.21.15.105:6839/500348151] conn(0x55ad8c9e4d80 0x55ad8cde3ed0 crc :-1 s=READY pgs=97 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:29:44.206 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:29:44.213+0000 7fa97cbea740 1 --2- 172.21.15.52:0/532167926 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55ad8ca261b0 0x55ad8cdd78a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:29:44.206 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:29:44.213+0000 7fa97cbea740 1 --2- 172.21.15.52:0/532167926 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55ad8ca054d0 0x55ad8cdd2030 unknown :-1 s=CLOSED pgs=135 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:29:44.207 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:29:44.213+0000 7fa94effd640 0 client.26018 ms_handle_reset on v2:172.21.15.105:6838/500348151 2024-05-27T07:29:44.207 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:29:44.213+0000 7fa97cbea740 1 --2- 172.21.15.52:0/532167926 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55ad8c9fd2f0 0x55ad8cdd1ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:29:44.207 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:29:44.213+0000 7fa97cbea740 1 -- 172.21.15.52:0/532167926 >> 172.21.15.52:0/532167926 conn(0x55ad8ca1a300 msgr2=0x55ad8cdc90f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:29:44.207 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:29:44.213+0000 7fa97cbea740 1 -- 172.21.15.52:0/532167926 shutdown_connections 2024-05-27T07:29:44.207 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:29:44.213+0000 7fa97cbea740 1 -- 172.21.15.52:0/532167926 wait complete. 2024-05-27T07:29:44.207 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:29:44.213+0000 7fa97cbea740 20 client.26018 trim_cache size 0 max 16384 2024-05-27T07:29:44.220 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:29:44.229+0000 7fa97cbea740 1 Processor -- start 2024-05-27T07:29:44.220 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:29:44.229+0000 7fa97cbea740 1 -- start start 2024-05-27T07:29:44.221 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:29:44.230+0000 7fa97cbea740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55ad8c9e4d80 0x55ad8cdd77a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:29:44.221 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:29:44.230+0000 7fa97cbea740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55ad8c9fd2f0 0x55ad8cdc3c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:29:44.221 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:29:44.230+0000 7fa97cbea740 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55ad8ca054d0 0x55ad8cdcb150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:29:44.221 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:29:44.230+0000 7fa97cbea740 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x7fa9701b4f60 con 0x55ad8ca054d0 2024-05-27T07:29:44.221 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:29:44.230+0000 7fa97cbea740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7fa9701b2c90 con 0x55ad8c9fd2f0 2024-05-27T07:29:44.221 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:29:44.230+0000 7fa96ffff640 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55ad8ca054d0 0x55ad8cdcb150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:29:44.221 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:29:44.230+0000 7fa97cbea740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7fa9701b4890 con 0x55ad8c9e4d80 2024-05-27T07:29:44.221 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:29:44.230+0000 7fa96ffff640 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55ad8ca054d0 0x55ad8cdcb150 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.52:36440/0 (socket says 172.21.15.52:36440) 2024-05-27T07:29:44.221 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:29:44.230+0000 7fa96ffff640 1 -- 172.21.15.52:0/3074095531 learned_addr learned my addr 172.21.15.52:0/3074095531 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-05-27T07:29:44.221 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:29:44.230+0000 7fa96f7fe640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55ad8c9e4d80 0x55ad8cdd77a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:29:44.221 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:29:44.230+0000 7fa96effd640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55ad8c9fd2f0 0x55ad8cdc3c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:29:44.223 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:29:44.230+0000 7fa96ffff640 1 -- 172.21.15.52:0/3074095531 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55ad8c9e4d80 msgr2=0x55ad8cdd77a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:29:44.223 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:29:44.230+0000 7fa96ffff640 1 --2- 172.21.15.52:0/3074095531 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55ad8c9e4d80 0x55ad8cdd77a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:29:44.223 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:29:44.230+0000 7fa96ffff640 1 -- 172.21.15.52:0/3074095531 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55ad8c9fd2f0 msgr2=0x55ad8cdc3c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:29:44.223 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:29:44.230+0000 7fa96ffff640 1 --2- 172.21.15.52:0/3074095531 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55ad8c9fd2f0 0x55ad8cdc3c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:29:44.223 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:29:44.230+0000 7fa96ffff640 1 -- 172.21.15.52:0/3074095531 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55ad8cdd0bc0 con 0x55ad8ca054d0 2024-05-27T07:29:44.223 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:29:44.230+0000 7fa96f7fe640 1 --2- 172.21.15.52:0/3074095531 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55ad8c9e4d80 0x55ad8cdd77a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-27T07:29:44.223 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:29:44.231+0000 7fa96ffff640 1 --2- 172.21.15.52:0/3074095531 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55ad8ca054d0 0x55ad8cdcb150 secure :-1 s=READY pgs=621 cs=0 l=1 rev1=1 crypto rx=0x7fa968012b20 tx=0x7fa968015000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5bbebd8a20220e3e server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:29:44.223 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:29:44.231+0000 7fa9657fa640 1 -- 172.21.15.52:0/3074095531 <== mon.0 v2:172.21.15.52:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa968004d20 con 0x55ad8ca054d0 2024-05-27T07:29:44.223 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:29:44.231+0000 7fa9657fa640 1 -- 172.21.15.52:0/3074095531 <== mon.0 v2:172.21.15.52:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fa96802d070 con 0x55ad8ca054d0 2024-05-27T07:29:44.223 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:29:44.231+0000 7fa9657fa640 1 -- 172.21.15.52:0/3074095531 <== mon.0 v2:172.21.15.52:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa968031420 con 0x55ad8ca054d0 2024-05-27T07:29:44.224 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:29:44.232+0000 7fa97cbea740 1 -- 172.21.15.52:0/3074095531 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55ad8ca054d0 msgr2=0x55ad8cdcb150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:29:44.224 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:29:44.232+0000 7fa97cbea740 1 --2- 172.21.15.52:0/3074095531 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55ad8ca054d0 0x55ad8cdcb150 secure :-1 s=READY pgs=621 cs=0 l=1 rev1=1 crypto rx=0x7fa968012b20 tx=0x7fa968015000 comp rx=0 tx=0).stop 2024-05-27T07:29:44.224 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:29:44.232+0000 7fa97cbea740 1 -- 172.21.15.52:0/3074095531 shutdown_connections 2024-05-27T07:29:44.224 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:29:44.232+0000 7fa97cbea740 1 --2- 172.21.15.52:0/3074095531 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55ad8ca054d0 0x55ad8cdcb150 unknown :-1 s=CLOSED pgs=621 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:29:44.224 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:29:44.232+0000 7fa97cbea740 1 --2- 172.21.15.52:0/3074095531 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55ad8c9fd2f0 0x55ad8cdc3c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:29:44.224 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:29:44.232+0000 7fa97cbea740 1 --2- 172.21.15.52:0/3074095531 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55ad8c9e4d80 0x55ad8cdd77a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:29:44.224 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:29:44.232+0000 7fa97cbea740 1 -- 172.21.15.52:0/3074095531 >> 172.21.15.52:0/3074095531 conn(0x55ad8ca1a300 msgr2=0x55ad8cdc3540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:29:44.224 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:29:44.232+0000 7fa97cbea740 1 -- 172.21.15.52:0/3074095531 shutdown_connections 2024-05-27T07:29:44.224 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:29:44.232+0000 7fa97cbea740 1 -- 172.21.15.52:0/3074095531 wait complete. 2024-05-27T07:29:44.224 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:29:44.232+0000 7fa97cbea740 1 -- start start 2024-05-27T07:29:44.224 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:29:44.233+0000 7fa97cbea740 1 Processor -- start 2024-05-27T07:29:44.227 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:29:44.233+0000 7fa97cbea740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55ad8c9e4d80 0x55ad8cdc31a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:29:44.227 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:29:44.233+0000 7fa97cbea740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55ad8c9fd2f0 0x55ad8cdc36e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:29:44.227 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:29:44.233+0000 7fa97cbea740 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55ad8ca054d0 0x55ad8cdc3c20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:29:44.227 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:29:44.233+0000 7fa97cbea740 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x7fa9701b3af0 con 0x55ad8ca054d0 2024-05-27T07:29:44.227 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:29:44.233+0000 7fa97cbea740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7fa9701b3420 con 0x55ad8c9e4d80 2024-05-27T07:29:44.227 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:29:44.233+0000 7fa97cbea740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7fa9701b41c0 con 0x55ad8c9fd2f0 2024-05-27T07:29:44.227 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:29:44.233+0000 7fa97cbea740 10 client.0 _ll_register_callbacks cb 0x7fa958005ef0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T07:29:44.228 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:29:44.233+0000 7fa96ffff640 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55ad8ca054d0 0x55ad8cdc3c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:29:44.228 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:29:44.233+0000 7fa96ffff640 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55ad8ca054d0 0x55ad8cdc3c20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.52:36442/0 (socket says 172.21.15.52:36442) 2024-05-27T07:29:44.228 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:29:44.233+0000 7fa96ffff640 1 -- 172.21.15.52:0/1820869022 learned_addr learned my addr 172.21.15.52:0/1820869022 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-05-27T07:29:44.228 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:29:44.233+0000 7fa96f7fe640 1 --2- 172.21.15.52:0/1820869022 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55ad8c9e4d80 0x55ad8cdc31a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:29:44.228 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:29:44.233+0000 7fa96effd640 1 --2- 172.21.15.52:0/1820869022 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55ad8c9fd2f0 0x55ad8cdc36e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:29:44.228 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:29:44.233+0000 7fa96f7fe640 1 -- 172.21.15.52:0/1820869022 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55ad8c9fd2f0 msgr2=0x55ad8cdc36e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:29:44.228 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:29:44.233+0000 7fa96f7fe640 1 --2- 172.21.15.52:0/1820869022 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55ad8c9fd2f0 0x55ad8cdc36e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:29:44.228 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:29:44.233+0000 7fa96f7fe640 1 -- 172.21.15.52:0/1820869022 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55ad8ca054d0 msgr2=0x55ad8cdc3c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:29:44.228 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:29:44.233+0000 7fa96f7fe640 1 --2- 172.21.15.52:0/1820869022 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55ad8ca054d0 0x55ad8cdc3c20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:29:44.228 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:29:44.233+0000 7fa96f7fe640 1 -- 172.21.15.52:0/1820869022 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55ad8ce4a700 con 0x55ad8c9e4d80 2024-05-27T07:29:44.228 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:29:44.233+0000 7fa96ffff640 1 --2- 172.21.15.52:0/1820869022 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55ad8ca054d0 0x55ad8cdc3c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-27T07:29:44.228 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:29:44.234+0000 7fa96f7fe640 1 --2- 172.21.15.52:0/1820869022 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55ad8c9e4d80 0x55ad8cdc31a0 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7fa960004880 tx=0x7fa960009870 comp rx=0 tx=0).ready entity=mon.2 client_cookie=3590797fe9f0593e server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:29:44.228 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:29:44.234+0000 7fa94effd640 10 client.0 ms_handle_connect on v2:172.21.15.105:3300/0 2024-05-27T07:29:44.229 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:29:44.234+0000 7fa94effd640 1 -- 172.21.15.52:0/1820869022 <== mon.2 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa96001e040 con 0x55ad8c9e4d80 2024-05-27T07:29:44.229 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:29:44.234+0000 7fa97cbea740 1 -- 172.21.15.52:0/1820869022 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fa96802d070 con 0x55ad8c9e4d80 2024-05-27T07:29:44.229 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:29:44.234+0000 7fa94effd640 1 -- 172.21.15.52:0/1820869022 <== mon.2 v2:172.21.15.105:3300/0 2 ==== osd_map(106..106 src has 1..106) ==== 9379+0+0 (secure 0 0 0) 0x7fa96001aae0 con 0x55ad8c9e4d80 2024-05-27T07:29:44.229 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:29:44.234+0000 7fa94effd640 1 -- 172.21.15.52:0/1820869022 <== mon.2 v2:172.21.15.105:3300/0 3 ==== mon_get_version_reply(handle=1 version=66) ==== 24+0+0 (secure 0 0 0) 0x7fa96003c050 con 0x55ad8c9e4d80 2024-05-27T07:29:44.229 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:29:44.234+0000 7fa97cbea740 10 client.26024 fetch_fsmap learned FSMap version 66 2024-05-27T07:29:44.229 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:29:44.234+0000 7fa97cbea740 1 -- 172.21.15.52:0/1820869022 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,fsmap=66,monmap=4+}) -- 0x55ad8ce4b950 con 0x55ad8c9e4d80 2024-05-27T07:29:44.229 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:29:44.234+0000 7fa94effd640 1 -- 172.21.15.52:0/1820869022 <== mon.2 v2:172.21.15.105:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fa96802d070 con 0x55ad8c9e4d80 2024-05-27T07:29:44.229 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:29:44.235+0000 7fa94effd640 1 -- 172.21.15.52:0/1820869022 <== mon.2 v2:172.21.15.105:3300/0 5 ==== fsmap(e 66) ==== 7773+0+0 (secure 0 0 0) 0x7fa960024040 con 0x55ad8c9e4d80 2024-05-27T07:29:44.229 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:29:44.235+0000 7fa97cbea740 10 client.26024 fetch_fsmap finished waiting for FSMap version 66 2024-05-27T07:29:44.230 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:29:44.235+0000 7fa97cbea740 10 client.26024 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-05-27T07:29:44.230 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:29:44.235+0000 7fa97cbea740 20 client.26024 populate_metadata read hostname 'smithi052' 2024-05-27T07:29:44.230 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:29:44.235+0000 7fa97cbea740 1 --2- 172.21.15.52:0/1820869022 >> [v2:172.21.15.105:6838/500348151,v1:172.21.15.105:6839/500348151] conn(0x55ad8ca261b0 0x55ad8ce4df70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:29:44.230 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:29:44.235+0000 7fa97cbea740 4 client.26024 mds_command: new command op to 24527 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-05-27T07:29:44.230 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:29:44.235+0000 7fa97cbea740 1 -- 172.21.15.52:0/1820869022 --> [v2:172.21.15.105:6838/500348151,v1:172.21.15.105:6839/500348151] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55ad8cd35360 con 0x55ad8ca261b0 2024-05-27T07:29:44.230 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:29:44.235+0000 7fa96effd640 1 --2- 172.21.15.52:0/1820869022 >> [v2:172.21.15.105:6838/500348151,v1:172.21.15.105:6839/500348151] conn(0x55ad8ca261b0 0x55ad8ce4df70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:29:44.230 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:29:44.235+0000 7fa96effd640 1 --2- 172.21.15.52:0/1820869022 >> [v2:172.21.15.105:6838/500348151,v1:172.21.15.105:6839/500348151] conn(0x55ad8ca261b0 0x55ad8ce4df70 crc :-1 s=READY pgs=98 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=ab000299ef584b1c server_cookie=8049f53e6de1ed0e in_seq=0 out_seq=0 2024-05-27T07:29:44.230 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:29:44.235+0000 7fa94effd640 10 client.26024 ms_handle_connect on v2:172.21.15.105:6838/500348151 2024-05-27T07:29:44.388 INFO:teuthology.orchestra.run.smithi052.stderr:Inferring config /var/lib/ceph/73df2636-1bf8-11ef-bc9b-c7b262605968/mon.a/config 2024-05-27T07:29:44.440 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:29:44 smithi105 ceph-mon[35977]: pgmap v131: 129 pgs: 129 active+clean; 224 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.9 MiB/s rd, 1.8 MiB/s wr, 708 op/s 2024-05-27T07:29:44.480 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:29:44 smithi052 ceph-mon[31955]: pgmap v131: 129 pgs: 129 active+clean; 224 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.9 MiB/s rd, 1.8 MiB/s wr, 708 op/s 2024-05-27T07:29:44.602 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:29:44 smithi173 ceph-mon[37000]: pgmap v131: 129 pgs: 129 active+clean; 224 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.9 MiB/s rd, 1.8 MiB/s wr, 708 op/s 2024-05-27T07:29:45.206 INFO:tasks.workunit.client.0.smithi052.stdout: 67 472323 391 327690 413 293949 881 2024-05-27T07:29:46.230 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:29:46 smithi052 ceph-mon[31955]: pgmap v132: 129 pgs: 129 active+clean; 263 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 6.3 MiB/s rd, 5.4 MiB/s wr, 592 op/s 2024-05-27T07:29:46.404 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:29:46 smithi173 ceph-mon[37000]: pgmap v132: 129 pgs: 129 active+clean; 263 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 6.3 MiB/s rd, 5.4 MiB/s wr, 592 op/s 2024-05-27T07:29:46.440 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:29:46 smithi105 ceph-mon[35977]: pgmap v132: 129 pgs: 129 active+clean; 263 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 6.3 MiB/s rd, 5.4 MiB/s wr, 592 op/s 2024-05-27T07:29:46.883 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:29:46.885+0000 7fe50871b700 1 -- 172.21.15.52:0/1508884570 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fe50011d920 msgr2=0x7fe50011fd40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:29:46.883 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:29:46.885+0000 7fe50871b700 1 --2- 172.21.15.52:0/1508884570 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fe50011d920 0x7fe50011fd40 secure :-1 s=READY pgs=622 cs=0 l=1 rev1=1 crypto rx=0x7fe4f800b0a0 tx=0x7fe4f800b3b0 comp rx=0 tx=0).stop 2024-05-27T07:29:46.883 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:29:46.885+0000 7fe5054b5700 1 -- 172.21.15.52:0/1508884570 <== mon.0 v2:172.21.15.52:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe4f8012650 con 0x7fe50011d920 2024-05-27T07:29:46.883 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:29:46.887+0000 7fe50871b700 1 -- 172.21.15.52:0/1508884570 shutdown_connections 2024-05-27T07:29:46.883 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:29:46.887+0000 7fe50871b700 1 --2- 172.21.15.52:0/1508884570 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fe50011d920 0x7fe50011fd40 unknown :-1 s=CLOSED pgs=622 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:29:46.883 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:29:46.887+0000 7fe50871b700 1 --2- 172.21.15.52:0/1508884570 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fe50010a920 0x7fe50010ad30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:29:46.883 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:29:46.887+0000 7fe50871b700 1 --2- 172.21.15.52:0/1508884570 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fe50010b270 0x7fe50011d3e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:29:46.883 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:29:46.887+0000 7fe50871b700 1 -- 172.21.15.52:0/1508884570 >> 172.21.15.52:0/1508884570 conn(0x7fe50006f4c0 msgr2=0x7fe50006f900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:29:46.884 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:29:46.892+0000 7fe50871b700 1 -- 172.21.15.52:0/1508884570 shutdown_connections 2024-05-27T07:29:46.884 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:29:46.893+0000 7fe50871b700 1 -- 172.21.15.52:0/1508884570 wait complete. 2024-05-27T07:29:46.885 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:29:46.894+0000 7fe50871b700 1 Processor -- start 2024-05-27T07:29:46.886 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:29:46.894+0000 7fe50871b700 1 -- start start 2024-05-27T07:29:46.886 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:29:46.894+0000 7fe50871b700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fe50010a920 0x7fe5001a9d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:29:46.886 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:29:46.894+0000 7fe50871b700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fe50010b270 0x7fe5001aa300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:29:46.886 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:29:46.895+0000 7fe50871b700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fe50011d920 0x7fe5001ae590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:29:46.886 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:29:46.895+0000 7fe50871b700 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x7fe500123430 con 0x7fe50011d920 2024-05-27T07:29:46.886 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:29:46.895+0000 7fe50871b700 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7fe5001232b0 con 0x7fe50010b270 2024-05-27T07:29:46.886 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:29:46.895+0000 7fe50871b700 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7fe5001235b0 con 0x7fe50010a920 2024-05-27T07:29:46.887 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:29:46.895+0000 7fe506cb8700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fe50011d920 0x7fe5001ae590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:29:46.887 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:29:46.895+0000 7fe506cb8700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fe50011d920 0x7fe5001ae590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.52:36460/0 (socket says 172.21.15.52:36460) 2024-05-27T07:29:46.887 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:29:46.895+0000 7fe506cb8700 1 -- 172.21.15.52:0/721306121 learned_addr learned my addr 172.21.15.52:0/721306121 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-05-27T07:29:46.887 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:29:46.896+0000 7fe506cb8700 1 -- 172.21.15.52:0/721306121 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fe50010a920 msgr2=0x7fe5001a9d60 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-05-27T07:29:46.887 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:29:46.896+0000 7fe506cb8700 1 --2- 172.21.15.52:0/721306121 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fe50010a920 0x7fe5001a9d60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:29:46.887 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:29:46.896+0000 7fe506cb8700 1 -- 172.21.15.52:0/721306121 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fe50010b270 msgr2=0x7fe5001aa300 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-05-27T07:29:46.887 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:29:46.896+0000 7fe506cb8700 1 --2- 172.21.15.52:0/721306121 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fe50010b270 0x7fe5001aa300 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:29:46.888 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:29:46.896+0000 7fe506cb8700 1 -- 172.21.15.52:0/721306121 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe5001aec30 con 0x7fe50011d920 2024-05-27T07:29:46.888 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:29:46.897+0000 7fe506cb8700 1 --2- 172.21.15.52:0/721306121 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fe50011d920 0x7fe5001ae590 secure :-1 s=READY pgs=623 cs=0 l=1 rev1=1 crypto rx=0x7fe4f8009bd0 tx=0x7fe4f800cf70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:29:46.891 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:29:46.900+0000 7fe4f37fe700 1 -- 172.21.15.52:0/721306121 <== mon.0 v2:172.21.15.52:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe4f8021070 con 0x7fe50011d920 2024-05-27T07:29:46.891 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:29:46.900+0000 7fe50871b700 1 -- 172.21.15.52:0/721306121 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe5001aeec0 con 0x7fe50011d920 2024-05-27T07:29:46.891 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:29:46.900+0000 7fe4f37fe700 1 -- 172.21.15.52:0/721306121 <== mon.0 v2:172.21.15.52:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fe4f8007d30 con 0x7fe50011d920 2024-05-27T07:29:46.891 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:29:46.900+0000 7fe4f37fe700 1 -- 172.21.15.52:0/721306121 <== mon.0 v2:172.21.15.52:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe4f801d450 con 0x7fe50011d920 2024-05-27T07:29:46.891 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:29:46.900+0000 7fe50871b700 1 -- 172.21.15.52:0/721306121 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe500112bc0 con 0x7fe50011d920 2024-05-27T07:29:46.907 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:29:46.913+0000 7fe517130b80 1 Processor -- start 2024-05-27T07:29:46.907 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:29:46.913+0000 7fe517130b80 1 -- start start 2024-05-27T07:29:46.907 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:29:46.914+0000 7fe517130b80 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55f2935d6e40 0x55f29375bb30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:29:46.907 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:29:46.914+0000 7fe517130b80 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55f29375c070 0x55f293755be0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:29:46.907 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:29:46.915+0000 7fe517130b80 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55f293756120 0x55f293756530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:29:46.907 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:29:46.915+0000 7fe517130b80 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x55f293683d70 con 0x55f293756120 2024-05-27T07:29:46.907 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:29:46.915+0000 7fe517130b80 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x55f293673910 con 0x55f29375c070 2024-05-27T07:29:46.907 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:29:46.915+0000 7fe517130b80 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x55f293677950 con 0x55f2935d6e40 2024-05-27T07:29:46.908 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:29:46.915+0000 7fe4f37fe700 1 -- 172.21.15.52:0/721306121 <== mon.0 v2:172.21.15.52:3300/0 4 ==== mgrmap(e 26) ==== 149783+0+0 (secure 0 0 0) 0x7fe4f8003a00 con 0x7fe50011d920 2024-05-27T07:29:46.908 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:29:46.916+0000 7fe505cb6700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55f2935d6e40 0x55f29375bb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:29:46.908 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:29:46.916+0000 7fe505cb6700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55f2935d6e40 0x55f29375bb30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.173:3300/0 says I am v2:172.21.15.52:39264/0 (socket says 172.21.15.52:39264) 2024-05-27T07:29:46.908 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:29:46.916+0000 7fe505cb6700 1 -- 172.21.15.52:0/412886468 learned_addr learned my addr 172.21.15.52:0/412886468 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-05-27T07:29:46.908 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:29:46.916+0000 7fe505cb6700 1 --2- 172.21.15.52:0/412886468 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55f293756120 0x55f293756530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:29:46.909 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:29:46.917+0000 7fe505cb6700 1 -- 172.21.15.52:0/412886468 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55f29375c070 msgr2=0x55f293755be0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-05-27T07:29:46.909 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:29:46.917+0000 7fe505cb6700 1 --2- 172.21.15.52:0/412886468 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55f29375c070 0x55f293755be0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:29:46.909 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:29:46.917+0000 7fe505cb6700 1 -- 172.21.15.52:0/412886468 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55f293756120 msgr2=0x55f293756530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:29:46.909 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:29:46.917+0000 7fe505cb6700 1 --2- 172.21.15.52:0/412886468 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55f293756120 0x55f293756530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:29:46.909 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:29:46.917+0000 7fe505cb6700 1 -- 172.21.15.52:0/412886468 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55f29361cfb0 con 0x55f2935d6e40 2024-05-27T07:29:46.909 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:29:46.918+0000 7fe505cb6700 1 --2- 172.21.15.52:0/412886468 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55f2935d6e40 0x55f29375bb30 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7fe4fc001cf0 tx=0x7fe4fc009ec0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=65115e271a34dc30 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:29:46.909 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:29:46.918+0000 7fe4f0ff9700 1 -- 172.21.15.52:0/412886468 <== mon.1 v2:172.21.15.173:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe4fc012040 con 0x55f2935d6e40 2024-05-27T07:29:46.909 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:29:46.918+0000 7fe4f0ff9700 1 -- 172.21.15.52:0/412886468 <== mon.1 v2:172.21.15.173:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fe4fc001d20 con 0x55f2935d6e40 2024-05-27T07:29:46.910 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:29:46.920+0000 7fe4f37fe700 1 --2- 172.21.15.52:0/721306121 >> [v2:172.21.15.52:6800/1822318384,v1:172.21.15.52:6801/1822318384] conn(0x7fe4ec0b2b80 0x7fe4ec0b5090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:29:46.911 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:29:46.920+0000 7fe517130b80 1 -- 172.21.15.52:0/412886468 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55f2935d6e40 msgr2=0x55f29375bb30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:29:46.911 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:29:46.920+0000 7fe517130b80 1 --2- 172.21.15.52:0/412886468 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55f2935d6e40 0x55f29375bb30 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7fe4fc001cf0 tx=0x7fe4fc009ec0 comp rx=0 tx=0).stop 2024-05-27T07:29:46.911 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:29:46.920+0000 7fe517130b80 1 -- 172.21.15.52:0/412886468 shutdown_connections 2024-05-27T07:29:46.912 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:29:46.920+0000 7fe517130b80 1 --2- 172.21.15.52:0/412886468 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55f293756120 0x55f293756530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:29:46.912 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:29:46.920+0000 7fe517130b80 1 --2- 172.21.15.52:0/412886468 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55f2935d6e40 0x55f29375bb30 unknown :-1 s=CLOSED pgs=120 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:29:46.912 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:29:46.920+0000 7fe517130b80 1 --2- 172.21.15.52:0/412886468 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55f29375c070 0x55f293755be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:29:46.912 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:29:46.920+0000 7fe517130b80 1 -- 172.21.15.52:0/412886468 >> 172.21.15.52:0/412886468 conn(0x55f2935d9770 msgr2=0x55f293750d90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:29:46.912 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:29:46.920+0000 7fe517130b80 1 -- 172.21.15.52:0/412886468 shutdown_connections 2024-05-27T07:29:46.912 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:29:46.920+0000 7fe517130b80 1 -- 172.21.15.52:0/412886468 wait complete. 2024-05-27T07:29:46.912 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:29:46.921+0000 7fe5064b7700 1 --2- 172.21.15.52:0/721306121 >> [v2:172.21.15.52:6800/1822318384,v1:172.21.15.52:6801/1822318384] conn(0x7fe4ec0b2b80 0x7fe4ec0b5090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:29:46.912 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:29:46.921+0000 7fe4f37fe700 1 -- 172.21.15.52:0/721306121 <== mon.0 v2:172.21.15.52:3300/0 5 ==== osd_map(106..106 src has 1..106) ==== 9379+0+0 (secure 0 0 0) 0x7fe4f80e7060 con 0x7fe50011d920 2024-05-27T07:29:46.913 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:29:46.921+0000 7fe5064b7700 1 --2- 172.21.15.52:0/721306121 >> [v2:172.21.15.52:6800/1822318384,v1:172.21.15.52:6801/1822318384] conn(0x7fe4ec0b2b80 0x7fe4ec0b5090 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x55f2937522a0 tx=0x7fe4f4006c50 comp rx=0 tx=0).ready entity=mgr.15333 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:29:46.913 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:29:46.921+0000 7fe517130b80 1 -- start start 2024-05-27T07:29:46.913 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:29:46.922+0000 7fe517130b80 1 Processor -- start 2024-05-27T07:29:46.915 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:29:46.922+0000 7fe517130b80 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55f293756120 0x55f2937621b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:29:46.915 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:29:46.922+0000 7fe517130b80 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55f29375c070 0x55f29375f470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:29:46.915 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:29:46.922+0000 7fe517130b80 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55f29375f9b0 0x55f29375fde0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:29:46.915 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:29:46.923+0000 7fe5064b7700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55f293756120 0x55f2937621b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:29:46.915 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:29:46.923+0000 7fe5064b7700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55f293756120 0x55f2937621b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.173:3300/0 says I am v2:172.21.15.52:39280/0 (socket says 172.21.15.52:39280) 2024-05-27T07:29:46.915 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:29:46.923+0000 7fe506cb8700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55f29375f9b0 0x55f29375fde0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:29:46.915 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:29:46.923+0000 7fe506cb8700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55f29375f9b0 0x55f29375fde0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.52:36466/0 (socket says 172.21.15.52:36466) 2024-05-27T07:29:46.916 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:29:46.923+0000 7fe505cb6700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55f29375c070 0x55f29375f470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:29:46.916 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:29:46.923+0000 7fe5064b7700 1 -- 172.21.15.52:0/3703603109 learned_addr learned my addr 172.21.15.52:0/3703603109 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-05-27T07:29:46.916 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:29:46.923+0000 7fe517130b80 1 -- 172.21.15.52:0/3703603109 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x55f293579fa0 con 0x55f29375f9b0 2024-05-27T07:29:46.916 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:29:46.923+0000 7fe517130b80 1 -- 172.21.15.52:0/3703603109 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x55f2935dbda0 con 0x55f29375c070 2024-05-27T07:29:46.916 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:29:46.923+0000 7fe517130b80 1 -- 172.21.15.52:0/3703603109 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x55f293762c00 con 0x55f293756120 2024-05-27T07:29:46.916 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:29:46.924+0000 7fe5064b7700 1 -- 172.21.15.52:0/3703603109 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55f29375c070 msgr2=0x55f29375f470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:29:46.916 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:29:46.924+0000 7fe5064b7700 1 --2- 172.21.15.52:0/3703603109 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55f29375c070 0x55f29375f470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:29:46.916 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:29:46.924+0000 7fe5064b7700 1 -- 172.21.15.52:0/3703603109 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55f29375f9b0 msgr2=0x55f29375fde0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:29:46.916 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:29:46.924+0000 7fe5064b7700 1 --2- 172.21.15.52:0/3703603109 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55f29375f9b0 0x55f29375fde0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:29:46.916 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:29:46.924+0000 7fe5064b7700 1 -- 172.21.15.52:0/3703603109 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55f2935c38d0 con 0x55f293756120 2024-05-27T07:29:46.916 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:29:46.924+0000 7fe506cb8700 1 --2- 172.21.15.52:0/3703603109 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55f29375f9b0 0x55f29375fde0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-27T07:29:46.917 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:29:46.924+0000 7fe505cb6700 1 --2- 172.21.15.52:0/3703603109 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55f29375c070 0x55f29375f470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-27T07:29:46.917 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:29:46.924+0000 7fe5064b7700 1 --2- 172.21.15.52:0/3703603109 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55f293756120 0x55f2937621b0 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7fe4f4013fd0 tx=0x7fe4f4004750 comp rx=0 tx=0).ready entity=mon.1 client_cookie=7c952a0557a0be20 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:29:46.917 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:29:46.925+0000 7fe4e9ffb700 10 client.0 ms_handle_connect on v2:172.21.15.173:3300/0 2024-05-27T07:29:46.917 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:29:46.925+0000 7fe517130b80 10 client.0 _ll_register_callbacks cb 0x55f29360a5b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T07:29:46.917 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:29:46.925+0000 7fe4e9ffb700 1 -- 172.21.15.52:0/3703603109 <== mon.1 v2:172.21.15.173:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe4f401d3f0 con 0x55f293756120 2024-05-27T07:29:46.917 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:29:46.925+0000 7fe4e9ffb700 1 -- 172.21.15.52:0/3703603109 <== mon.1 v2:172.21.15.173:3300/0 2 ==== osd_map(106..106 src has 1..106) ==== 9379+0+0 (secure 0 0 0) 0x7fe4f401d590 con 0x55f293756120 2024-05-27T07:29:46.917 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:29:46.925+0000 7fe517130b80 1 -- 172.21.15.52:0/3703603109 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x55f293685cc0 con 0x55f293756120 2024-05-27T07:29:46.920 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:29:46.929+0000 7fe4e9ffb700 1 -- 172.21.15.52:0/3703603109 <== mon.1 v2:172.21.15.173:3300/0 3 ==== mon_get_version_reply(handle=1 version=66) ==== 24+0+0 (secure 0 0 0) 0x7fe4f402b050 con 0x55f293756120 2024-05-27T07:29:46.921 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:29:46.930+0000 7fe517130b80 10 client.25942 fetch_fsmap learned FSMap version 66 2024-05-27T07:29:46.921 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:29:46.930+0000 7fe517130b80 1 -- 172.21.15.52:0/3703603109 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({fsmap=66}) -- 0x55f29362d760 con 0x55f293756120 2024-05-27T07:29:46.921 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:29:46.931+0000 7fe4e9ffb700 1 -- 172.21.15.52:0/3703603109 <== mon.1 v2:172.21.15.173:3300/0 4 ==== fsmap(e 66) ==== 7773+0+0 (secure 0 0 0) 0x7fe4f4012070 con 0x55f293756120 2024-05-27T07:29:46.921 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:29:46.931+0000 7fe517130b80 10 client.25942 fetch_fsmap finished waiting for FSMap version 66 2024-05-27T07:29:46.921 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:29:46.931+0000 7fe517130b80 10 client.25942 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.i 2024-05-27T07:29:46.922 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:29:46.931+0000 7fe517130b80 20 client.25942 populate_metadata read hostname 'smithi052' 2024-05-27T07:29:46.922 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:29:46.931+0000 7fe517130b80 1 --2- 172.21.15.52:0/3703603109 >> [v2:172.21.15.173:6838/564778293,v1:172.21.15.173:6839/564778293] conn(0x55f29376fb80 0x55f293772090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:29:46.922 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:29:46.931+0000 7fe517130b80 4 client.25942 mds_command: new command op to 24475 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-27T07:29:46.922 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:29:46.931+0000 7fe517130b80 1 -- 172.21.15.52:0/3703603109 --> [v2:172.21.15.173:6838/564778293,v1:172.21.15.173:6839/564778293] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55f293685cc0 con 0x55f29376fb80 2024-05-27T07:29:46.922 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:29:46.931+0000 7fe505cb6700 1 --2- 172.21.15.52:0/3703603109 >> [v2:172.21.15.173:6838/564778293,v1:172.21.15.173:6839/564778293] conn(0x55f29376fb80 0x55f293772090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:29:46.923 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:29:46.933+0000 7fe505cb6700 1 --2- 172.21.15.52:0/3703603109 >> [v2:172.21.15.173:6838/564778293,v1:172.21.15.173:6839/564778293] conn(0x55f29376fb80 0x55f293772090 crc :-1 s=READY pgs=42 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=c30541160fd66ae4 server_cookie=f12b77cff4e3dba4 in_seq=0 out_seq=0 2024-05-27T07:29:46.923 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:29:46.933+0000 7fe4e9ffb700 10 client.25942 ms_handle_connect on v2:172.21.15.173:6838/564778293 2024-05-27T07:29:46.924 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:29:46.934+0000 7fe4e9ffb700 1 -- 172.21.15.52:0/3703603109 <== mds.2 v2:172.21.15.173:6838/564778293 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24041 (crc 0 0 0) 0x55f293685cc0 con 0x55f29376fb80 2024-05-27T07:29:46.924 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:29:46.934+0000 7fe4e9ffb700 10 client.25942 handle_command_reply: tid=0 2024-05-27T07:29:46.924 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:29:46.934+0000 7fe517130b80 1 client.25942 shutdown 2024-05-27T07:29:46.925 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:29:46.934+0000 7fe517130b80 1 -- 172.21.15.52:0/3703603109 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55f293756120 msgr2=0x55f2937621b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:29:46.925 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:29:46.934+0000 7fe517130b80 1 --2- 172.21.15.52:0/3703603109 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55f293756120 0x55f2937621b0 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7fe4f4013fd0 tx=0x7fe4f4004750 comp rx=0 tx=0).stop 2024-05-27T07:29:46.925 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:29:46.935+0000 7fe517130b80 1 -- 172.21.15.52:0/3703603109 shutdown_connections 2024-05-27T07:29:46.925 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:29:46.935+0000 7fe517130b80 1 --2- 172.21.15.52:0/3703603109 >> [v2:172.21.15.173:6838/564778293,v1:172.21.15.173:6839/564778293] conn(0x55f29376fb80 0x55f293772090 crc :-1 s=READY pgs=42 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:29:46.926 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:29:46.935+0000 7fe517130b80 1 --2- 172.21.15.52:0/3703603109 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55f29375f9b0 0x55f29375fde0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:29:46.926 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:29:46.935+0000 7fe517130b80 1 --2- 172.21.15.52:0/3703603109 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55f293756120 0x55f2937621b0 unknown :-1 s=CLOSED pgs=121 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:29:46.926 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:29:46.935+0000 7fe517130b80 1 --2- 172.21.15.52:0/3703603109 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55f29375c070 0x55f29375f470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:29:46.926 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:29:46.935+0000 7fe517130b80 1 -- 172.21.15.52:0/3703603109 >> 172.21.15.52:0/3703603109 conn(0x55f2935d9770 msgr2=0x55f293754bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:29:46.926 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:29:46.935+0000 7fe4e9ffb700 0 client.25942 ms_handle_reset on v2:172.21.15.173:6838/564778293 2024-05-27T07:29:46.926 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:29:46.935+0000 7fe517130b80 1 -- 172.21.15.52:0/3703603109 shutdown_connections 2024-05-27T07:29:46.926 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:29:46.935+0000 7fe517130b80 1 -- 172.21.15.52:0/3703603109 wait complete. 2024-05-27T07:29:46.927 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:29:46.936+0000 7fe517130b80 20 client.25942 trim_cache size 0 max 16384 2024-05-27T07:29:46.954 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:29:46.963+0000 7fe517130b80 1 Processor -- start 2024-05-27T07:29:46.954 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:29:46.963+0000 7fe517130b80 1 -- start start 2024-05-27T07:29:46.955 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:29:46.963+0000 7fe517130b80 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55f293756120 0x55f293755be0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:29:46.955 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:29:46.963+0000 7fe517130b80 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55f29375c070 0x55f293750720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:29:46.955 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:29:46.963+0000 7fe517130b80 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55f29375f9b0 0x55f293750c60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:29:46.955 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:29:46.963+0000 7fe517130b80 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x7fe4e4010760 con 0x55f29375c070 2024-05-27T07:29:46.956 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:29:46.963+0000 7fe517130b80 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7fe4e400e490 con 0x55f29375f9b0 2024-05-27T07:29:46.956 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:29:46.963+0000 7fe517130b80 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7fe4e4010090 con 0x55f293756120 2024-05-27T07:29:46.956 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:29:46.963+0000 7fe505cb6700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55f29375c070 0x55f293750720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:29:46.956 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:29:46.963+0000 7fe505cb6700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55f29375c070 0x55f293750720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.52:36472/0 (socket says 172.21.15.52:36472) 2024-05-27T07:29:46.956 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:29:46.964+0000 7fe505cb6700 1 -- 172.21.15.52:0/267129653 learned_addr learned my addr 172.21.15.52:0/267129653 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-05-27T07:29:46.956 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:29:46.964+0000 7fe5064b7700 1 --2- 172.21.15.52:0/267129653 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55f293756120 0x55f293755be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:29:46.956 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:29:46.964+0000 7fe5064b7700 1 -- 172.21.15.52:0/267129653 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55f29375f9b0 msgr2=0x55f293750c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:29:46.956 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:29:46.964+0000 7fe5064b7700 1 --2- 172.21.15.52:0/267129653 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55f29375f9b0 0x55f293750c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:29:46.956 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:29:46.964+0000 7fe5064b7700 1 -- 172.21.15.52:0/267129653 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55f29375c070 msgr2=0x55f293750720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:29:46.956 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:29:46.964+0000 7fe5064b7700 1 --2- 172.21.15.52:0/267129653 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55f29375c070 0x55f293750720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:29:46.956 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:29:46.964+0000 7fe5064b7700 1 -- 172.21.15.52:0/267129653 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55f293761d10 con 0x55f293756120 2024-05-27T07:29:46.957 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:29:46.964+0000 7fe505cb6700 1 --2- 172.21.15.52:0/267129653 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55f29375c070 0x55f293750720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-27T07:29:46.957 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:29:46.964+0000 7fe5064b7700 1 --2- 172.21.15.52:0/267129653 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55f293756120 0x55f293755be0 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7fe4f4006fb0 tx=0x7fe4f401a000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=51caf35dff34cdab server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:29:46.957 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:29:46.965+0000 7fe4ebfff700 1 -- 172.21.15.52:0/267129653 <== mon.1 v2:172.21.15.173:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe4f4036020 con 0x55f293756120 2024-05-27T07:29:46.957 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:29:46.965+0000 7fe4ebfff700 1 -- 172.21.15.52:0/267129653 <== mon.1 v2:172.21.15.173:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fe4f4022070 con 0x55f293756120 2024-05-27T07:29:46.957 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:29:46.965+0000 7fe4ebfff700 1 -- 172.21.15.52:0/267129653 <== mon.1 v2:172.21.15.173:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe4f4005dd0 con 0x55f293756120 2024-05-27T07:29:46.958 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:29:46.966+0000 7fe517130b80 1 -- 172.21.15.52:0/267129653 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55f293756120 msgr2=0x55f293755be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:29:46.959 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:29:46.966+0000 7fe517130b80 1 --2- 172.21.15.52:0/267129653 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55f293756120 0x55f293755be0 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7fe4f4006fb0 tx=0x7fe4f401a000 comp rx=0 tx=0).stop 2024-05-27T07:29:46.959 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:29:46.966+0000 7fe517130b80 1 -- 172.21.15.52:0/267129653 shutdown_connections 2024-05-27T07:29:46.959 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:29:46.966+0000 7fe517130b80 1 --2- 172.21.15.52:0/267129653 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55f29375f9b0 0x55f293750c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:29:46.959 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:29:46.966+0000 7fe517130b80 1 --2- 172.21.15.52:0/267129653 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55f293756120 0x55f293755be0 unknown :-1 s=CLOSED pgs=122 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:29:46.959 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:29:46.966+0000 7fe517130b80 1 --2- 172.21.15.52:0/267129653 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55f29375c070 0x55f293750720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:29:46.959 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:29:46.966+0000 7fe517130b80 1 -- 172.21.15.52:0/267129653 >> 172.21.15.52:0/267129653 conn(0x55f2935d9770 msgr2=0x55f29375dd60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:29:46.959 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:29:46.967+0000 7fe517130b80 1 -- 172.21.15.52:0/267129653 shutdown_connections 2024-05-27T07:29:46.959 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:29:46.967+0000 7fe517130b80 1 -- 172.21.15.52:0/267129653 wait complete. 2024-05-27T07:29:46.959 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:29:46.967+0000 7fe517130b80 1 -- start start 2024-05-27T07:29:46.959 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:29:46.967+0000 7fe517130b80 1 Processor -- start 2024-05-27T07:29:46.959 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:29:46.968+0000 7fe517130b80 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55f293756120 0x55f293767f40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:29:46.959 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:29:46.968+0000 7fe517130b80 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55f29375c070 0x55f2937684c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:29:46.960 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:29:46.968+0000 7fe517130b80 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55f29375f9b0 0x55f293754bd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:29:46.960 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:29:46.968+0000 7fe517130b80 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x7fe4e400f2f0 con 0x55f29375f9b0 2024-05-27T07:29:46.961 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:29:46.968+0000 7fe517130b80 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7fe4e400ec20 con 0x55f293756120 2024-05-27T07:29:46.961 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:29:46.968+0000 7fe517130b80 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7fe4e400f9c0 con 0x55f29375c070 2024-05-27T07:29:46.961 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:29:46.968+0000 7fe517130b80 10 client.0 _ll_register_callbacks cb 0x55f29360a5b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T07:29:46.961 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:29:46.968+0000 7fe5064b7700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55f293756120 0x55f293767f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:29:46.961 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:29:46.968+0000 7fe506cb8700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55f29375f9b0 0x55f293754bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:29:46.961 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:29:46.969+0000 7fe505cb6700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55f29375c070 0x55f2937684c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:29:46.961 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:29:46.969+0000 7fe506cb8700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55f29375f9b0 0x55f293754bd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.52:36480/0 (socket says 172.21.15.52:36480) 2024-05-27T07:29:46.961 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:29:46.969+0000 7fe505cb6700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55f29375c070 0x55f2937684c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.173:3300/0 says I am v2:172.21.15.52:39304/0 (socket says 172.21.15.52:39304) 2024-05-27T07:29:46.961 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:29:46.969+0000 7fe506cb8700 1 -- 172.21.15.52:0/1720005679 learned_addr learned my addr 172.21.15.52:0/1720005679 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-05-27T07:29:46.961 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:29:46.969+0000 7fe505cb6700 1 -- 172.21.15.52:0/1720005679 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55f293756120 msgr2=0x55f293767f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:29:46.961 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:29:46.969+0000 7fe505cb6700 1 --2- 172.21.15.52:0/1720005679 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55f293756120 0x55f293767f40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:29:46.961 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:29:46.969+0000 7fe505cb6700 1 -- 172.21.15.52:0/1720005679 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55f29375f9b0 msgr2=0x55f293754bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:29:46.962 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:29:46.969+0000 7fe505cb6700 1 --2- 172.21.15.52:0/1720005679 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55f29375f9b0 0x55f293754bd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:29:46.962 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:29:46.969+0000 7fe505cb6700 1 -- 172.21.15.52:0/1720005679 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55f2937551c0 con 0x55f29375c070 2024-05-27T07:29:46.962 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:29:46.969+0000 7fe506cb8700 1 --2- 172.21.15.52:0/1720005679 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55f29375f9b0 0x55f293754bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-27T07:29:46.962 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:29:46.970+0000 7fe505cb6700 1 --2- 172.21.15.52:0/1720005679 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55f29375c070 0x55f2937684c0 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7fe4fc001cc0 tx=0x7fe4fc0017a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=525e46320d2d3b8a server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:29:46.962 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:29:46.970+0000 7fe4e9ffb700 10 client.0 ms_handle_connect on v2:172.21.15.173:3300/0 2024-05-27T07:29:46.963 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:29:46.970+0000 7fe4e9ffb700 1 -- 172.21.15.52:0/1720005679 <== mon.1 v2:172.21.15.173:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe4fc013db0 con 0x55f29375c070 2024-05-27T07:29:46.963 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:29:46.970+0000 7fe517130b80 1 -- 172.21.15.52:0/1720005679 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fe4f4022070 con 0x55f29375c070 2024-05-27T07:29:46.963 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:29:46.971+0000 7fe4e9ffb700 1 -- 172.21.15.52:0/1720005679 <== mon.1 v2:172.21.15.173:3300/0 2 ==== osd_map(106..106 src has 1..106) ==== 9379+0+0 (secure 0 0 0) 0x7fe4fc028030 con 0x55f29375c070 2024-05-27T07:29:46.963 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:29:46.971+0000 7fe4e9ffb700 1 -- 172.21.15.52:0/1720005679 <== mon.1 v2:172.21.15.173:3300/0 3 ==== mon_get_version_reply(handle=1 version=66) ==== 24+0+0 (secure 0 0 0) 0x7fe4fc013db0 con 0x55f29375c070 2024-05-27T07:29:46.963 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:29:46.971+0000 7fe517130b80 10 client.25948 fetch_fsmap learned FSMap version 66 2024-05-27T07:29:46.963 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:29:46.971+0000 7fe517130b80 1 -- 172.21.15.52:0/1720005679 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({config=0+,fsmap=66,monmap=4+}) -- 0x55f2937d9fa0 con 0x55f29375c070 2024-05-27T07:29:46.963 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:29:46.972+0000 7fe4e9ffb700 1 -- 172.21.15.52:0/1720005679 <== mon.1 v2:172.21.15.173:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fe4f4022070 con 0x55f29375c070 2024-05-27T07:29:46.963 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:29:46.972+0000 7fe4e9ffb700 1 -- 172.21.15.52:0/1720005679 <== mon.1 v2:172.21.15.173:3300/0 5 ==== fsmap(e 66) ==== 7773+0+0 (secure 0 0 0) 0x7fe4fc034080 con 0x55f29375c070 2024-05-27T07:29:46.964 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:29:46.973+0000 7fe517130b80 10 client.25948 fetch_fsmap finished waiting for FSMap version 66 2024-05-27T07:29:46.964 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:29:46.973+0000 7fe517130b80 10 client.25948 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.i 2024-05-27T07:29:46.964 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:29:46.973+0000 7fe517130b80 20 client.25948 populate_metadata read hostname 'smithi052' 2024-05-27T07:29:46.964 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:29:46.973+0000 7fe517130b80 1 --2- 172.21.15.52:0/1720005679 >> [v2:172.21.15.173:6838/564778293,v1:172.21.15.173:6839/564778293] conn(0x55f2937da550 0x55f2937dc970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:29:46.965 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:29:46.973+0000 7fe517130b80 4 client.25948 mds_command: new command op to 24475 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-05-27T07:29:46.965 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:29:46.973+0000 7fe517130b80 1 -- 172.21.15.52:0/1720005679 --> [v2:172.21.15.173:6838/564778293,v1:172.21.15.173:6839/564778293] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55f29367f4a0 con 0x55f2937da550 2024-05-27T07:29:46.965 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:29:46.973+0000 7fe5064b7700 1 --2- 172.21.15.52:0/1720005679 >> [v2:172.21.15.173:6838/564778293,v1:172.21.15.173:6839/564778293] conn(0x55f2937da550 0x55f2937dc970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:29:46.965 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:29:46.974+0000 7fe5064b7700 1 --2- 172.21.15.52:0/1720005679 >> [v2:172.21.15.173:6838/564778293,v1:172.21.15.173:6839/564778293] conn(0x55f2937da550 0x55f2937dc970 crc :-1 s=READY pgs=43 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=b652903973ade554 server_cookie=123531aafbcb30c2 in_seq=0 out_seq=0 2024-05-27T07:29:46.965 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:29:46.974+0000 7fe4e9ffb700 10 client.25948 ms_handle_connect on v2:172.21.15.173:6838/564778293 2024-05-27T07:29:46.965 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:29:46.974+0000 7fe4e9ffb700 1 -- 172.21.15.52:0/1720005679 <== mds.2 v2:172.21.15.173:6838/564778293 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55f29367f4a0 con 0x55f2937da550 2024-05-27T07:29:46.965 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:29:46.974+0000 7fe4e9ffb700 10 client.25948 handle_command_reply: tid=0 2024-05-27T07:29:46.966 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:29:46.974+0000 7fe517130b80 1 client.25948 shutdown 2024-05-27T07:29:46.966 INFO:teuthology.orchestra.run.smithi052.stdout:[] 2024-05-27T07:29:46.967 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:29:46.974+0000 7fe517130b80 1 -- 172.21.15.52:0/1720005679 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55f29375c070 msgr2=0x55f2937684c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:29:46.967 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:29:46.974+0000 7fe517130b80 1 --2- 172.21.15.52:0/1720005679 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55f29375c070 0x55f2937684c0 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7fe4fc001cc0 tx=0x7fe4fc0017a0 comp rx=0 tx=0).stop 2024-05-27T07:29:46.967 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:29:46.974+0000 7fe517130b80 1 -- 172.21.15.52:0/1720005679 shutdown_connections 2024-05-27T07:29:46.967 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:29:46.974+0000 7fe517130b80 1 --2- 172.21.15.52:0/1720005679 >> [v2:172.21.15.173:6838/564778293,v1:172.21.15.173:6839/564778293] conn(0x55f2937da550 0x55f2937dc970 crc :-1 s=READY pgs=43 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:29:46.967 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:29:46.974+0000 7fe517130b80 1 --2- 172.21.15.52:0/1720005679 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55f29375f9b0 0x55f293754bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:29:46.968 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:29:46.974+0000 7fe517130b80 1 --2- 172.21.15.52:0/1720005679 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55f293756120 0x55f293767f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:29:46.968 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:29:46.974+0000 7fe517130b80 1 --2- 172.21.15.52:0/1720005679 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55f29375c070 0x55f2937684c0 unknown :-1 s=CLOSED pgs=123 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:29:46.968 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:29:46.974+0000 7fe517130b80 1 -- 172.21.15.52:0/1720005679 >> 172.21.15.52:0/1720005679 conn(0x55f2935d9770 msgr2=0x55f293750720 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:29:46.968 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:29:46.974+0000 7fe4e9ffb700 0 client.25948 ms_handle_reset on v2:172.21.15.173:6838/564778293 2024-05-27T07:29:46.968 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:29:46.974+0000 7fe517130b80 1 -- 172.21.15.52:0/1720005679 shutdown_connections 2024-05-27T07:29:46.968 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:29:46.975+0000 7fe517130b80 1 -- 172.21.15.52:0/1720005679 wait complete. 2024-05-27T07:29:46.968 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:29:46.975+0000 7fe517130b80 20 client.25948 trim_cache size 0 max 16384 2024-05-27T07:29:46.968 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:29:46.976+0000 7fe50871b700 1 -- 172.21.15.52:0/721306121 >> [v2:172.21.15.52:6800/1822318384,v1:172.21.15.52:6801/1822318384] conn(0x7fe4ec0b2b80 msgr2=0x7fe4ec0b5090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:29:46.969 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:29:46.976+0000 7fe50871b700 1 --2- 172.21.15.52:0/721306121 >> [v2:172.21.15.52:6800/1822318384,v1:172.21.15.52:6801/1822318384] conn(0x7fe4ec0b2b80 0x7fe4ec0b5090 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x55f2937522a0 tx=0x7fe4f4006c50 comp rx=0 tx=0).stop 2024-05-27T07:29:46.969 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:29:46.976+0000 7fe50871b700 1 -- 172.21.15.52:0/721306121 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fe50011d920 msgr2=0x7fe5001ae590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:29:46.969 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:29:46.976+0000 7fe50871b700 1 --2- 172.21.15.52:0/721306121 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fe50011d920 0x7fe5001ae590 secure :-1 s=READY pgs=623 cs=0 l=1 rev1=1 crypto rx=0x7fe4f8009bd0 tx=0x7fe4f800cf70 comp rx=0 tx=0).stop 2024-05-27T07:29:46.969 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:29:46.976+0000 7fe50871b700 1 -- 172.21.15.52:0/721306121 shutdown_connections 2024-05-27T07:29:46.969 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:29:46.976+0000 7fe50871b700 1 --2- 172.21.15.52:0/721306121 >> [v2:172.21.15.52:6800/1822318384,v1:172.21.15.52:6801/1822318384] conn(0x7fe4ec0b2b80 0x7fe4ec0b5090 unknown :-1 s=CLOSED pgs=120 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:29:46.969 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:29:46.976+0000 7fe50871b700 1 --2- 172.21.15.52:0/721306121 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fe50011d920 0x7fe5001ae590 unknown :-1 s=CLOSED pgs=623 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:29:46.969 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:29:46.976+0000 7fe50871b700 1 --2- 172.21.15.52:0/721306121 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fe50010a920 0x7fe5001a9d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:29:46.969 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:29:46.976+0000 7fe50871b700 1 --2- 172.21.15.52:0/721306121 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fe50010b270 0x7fe5001aa300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:29:46.969 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:29:46.976+0000 7fe50871b700 1 -- 172.21.15.52:0/721306121 >> 172.21.15.52:0/721306121 conn(0x7fe50006f4c0 msgr2=0x7fe50011b3d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:29:46.969 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:29:46.976+0000 7fe50871b700 1 -- 172.21.15.52:0/721306121 shutdown_connections 2024-05-27T07:29:46.969 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:29:46.976+0000 7fe50871b700 1 -- 172.21.15.52:0/721306121 wait complete. 2024-05-27T07:29:47.141 INFO:journalctl@ceph.mds.d.smithi052.stdout:May 27 07:29:46 smithi052 systemd[1]: ceph-73df2636-1bf8-11ef-bc9b-c7b262605968@mds.d.service: Scheduled restart job, restart counter is at 4. 2024-05-27T07:29:47.141 INFO:journalctl@ceph.mds.d.smithi052.stdout:May 27 07:29:46 smithi052 systemd[1]: Stopped Ceph mds.d for 73df2636-1bf8-11ef-bc9b-c7b262605968. 2024-05-27T07:29:47.141 INFO:journalctl@ceph.mds.d.smithi052.stdout:May 27 07:29:47 smithi052 systemd[1]: Starting Ceph mds.d for 73df2636-1bf8-11ef-bc9b-c7b262605968... 2024-05-27T07:29:47.141 INFO:journalctl@ceph.mds.g.smithi052.stdout:May 27 07:29:46 smithi052 systemd[1]: ceph-73df2636-1bf8-11ef-bc9b-c7b262605968@mds.g.service: Scheduled restart job, restart counter is at 5. 2024-05-27T07:29:47.141 INFO:journalctl@ceph.mds.g.smithi052.stdout:May 27 07:29:47 smithi052 systemd[1]: Stopped Ceph mds.g for 73df2636-1bf8-11ef-bc9b-c7b262605968. 2024-05-27T07:29:47.142 INFO:journalctl@ceph.mds.g.smithi052.stdout:May 27 07:29:47 smithi052 systemd[1]: ceph-73df2636-1bf8-11ef-bc9b-c7b262605968@mds.g.service: Consumed 1.028s CPU time. 2024-05-27T07:29:47.142 INFO:journalctl@ceph.mds.g.smithi052.stdout:May 27 07:29:47 smithi052 systemd[1]: Starting Ceph mds.g for 73df2636-1bf8-11ef-bc9b-c7b262605968... 2024-05-27T07:29:47.744 DEBUG:teuthology.orchestra.run.smithi052:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:38f9d900c83c63799fbdbe61acc9a11b0d3554a6 shell --fsid 73df2636-1bf8-11ef-bc9b-c7b262605968 -- ceph tell mds.1:1 damage ls 2024-05-27T07:29:48.293 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:29:48 smithi052 ceph-mon[31955]: pgmap v133: 129 pgs: 129 active+clean; 263 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 6.3 MiB/s rd, 5.4 MiB/s wr, 592 op/s 2024-05-27T07:29:48.293 INFO:journalctl@ceph.mds.g.smithi052.stdout:May 27 07:29:48 smithi052 podman[106228]: 2024-05-27 07:29:47.932089766 +0000 UTC m=+0.033274244 image pull e36ea4fc8a5807e8d2701370e350a0de3856538797f7fba64cfce7e344bd6799 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6 2024-05-27T07:29:48.294 INFO:journalctl@ceph.mds.d.smithi052.stdout:May 27 07:29:48 smithi052 podman[106221]: 2024-05-27 07:29:47.922135937 +0000 UTC m=+0.031158910 image pull e36ea4fc8a5807e8d2701370e350a0de3856538797f7fba64cfce7e344bd6799 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6 2024-05-27T07:29:48.294 INFO:journalctl@ceph.mds.d.smithi052.stdout:May 27 07:29:48 smithi052 podman[106221]: 2024-05-27 07:29:48.076079137 +0000 UTC m=+0.185102094 container create 377781175b2cb23f4f9fbb362b5b24d7361ffdd72d17fa2e26d16676442aa5c2 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-d, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, ceph=True, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, GIT_CLEAN=True, org.label-schema.build-date=20240521, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD) 2024-05-27T07:29:48.441 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:29:48 smithi105 ceph-mon[35977]: pgmap v133: 129 pgs: 129 active+clean; 263 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 6.3 MiB/s rd, 5.4 MiB/s wr, 592 op/s 2024-05-27T07:29:48.555 INFO:journalctl@ceph.mds.g.smithi052.stdout:May 27 07:29:48 smithi052 podman[106228]: 2024-05-27 07:29:48.30337463 +0000 UTC m=+0.404559074 container create 366d6f4b82cae2e92c867e97cde83ae500f5633b53b558e9f83dca56e4b4b222 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-g, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, GIT_CLEAN=True, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240521, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0) 2024-05-27T07:29:48.602 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:29:48 smithi173 ceph-mon[37000]: pgmap v133: 129 pgs: 129 active+clean; 263 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 6.3 MiB/s rd, 5.4 MiB/s wr, 592 op/s 2024-05-27T07:29:48.807 INFO:journalctl@ceph.mds.j.smithi052.stdout:May 27 07:29:48 smithi052 systemd[1]: ceph-73df2636-1bf8-11ef-bc9b-c7b262605968@mds.j.service: Scheduled restart job, restart counter is at 1. 2024-05-27T07:29:48.807 INFO:journalctl@ceph.mds.j.smithi052.stdout:May 27 07:29:48 smithi052 systemd[1]: Stopped Ceph mds.j for 73df2636-1bf8-11ef-bc9b-c7b262605968. 2024-05-27T07:29:48.807 INFO:journalctl@ceph.mds.j.smithi052.stdout:May 27 07:29:48 smithi052 systemd[1]: ceph-73df2636-1bf8-11ef-bc9b-c7b262605968@mds.j.service: Consumed 1.162s CPU time. 2024-05-27T07:29:48.807 INFO:journalctl@ceph.mds.j.smithi052.stdout:May 27 07:29:48 smithi052 systemd[1]: Starting Ceph mds.j for 73df2636-1bf8-11ef-bc9b-c7b262605968... 2024-05-27T07:29:48.807 INFO:journalctl@ceph.mds.g.smithi052.stdout:May 27 07:29:48 smithi052 podman[106228]: 2024-05-27 07:29:48.69362243 +0000 UTC m=+0.794806894 container init 366d6f4b82cae2e92c867e97cde83ae500f5633b53b558e9f83dca56e4b4b222 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-g, ceph=True, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240521, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90) 2024-05-27T07:29:48.807 INFO:journalctl@ceph.mds.g.smithi052.stdout:May 27 07:29:48 smithi052 podman[106228]: 2024-05-27 07:29:48.708996234 +0000 UTC m=+0.810180681 container start 366d6f4b82cae2e92c867e97cde83ae500f5633b53b558e9f83dca56e4b4b222 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-g, GIT_CLEAN=True, CEPH_POINT_RELEASE=, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.build-date=20240521, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS) 2024-05-27T07:29:48.814 INFO:journalctl@ceph.mds.d.smithi052.stdout:May 27 07:29:48 smithi052 podman[106221]: 2024-05-27 07:29:48.603927884 +0000 UTC m=+0.712950838 container init 377781175b2cb23f4f9fbb362b5b24d7361ffdd72d17fa2e26d16676442aa5c2 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-d, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, ceph=True, org.label-schema.build-date=20240521, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True) 2024-05-27T07:29:48.814 INFO:journalctl@ceph.mds.d.smithi052.stdout:May 27 07:29:48 smithi052 podman[106221]: 2024-05-27 07:29:48.629203949 +0000 UTC m=+0.738226904 container start 377781175b2cb23f4f9fbb362b5b24d7361ffdd72d17fa2e26d16676442aa5c2 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-d, io.buildah.version=1.33.5, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, org.label-schema.build-date=20240521) 2024-05-27T07:29:48.814 INFO:journalctl@ceph.mds.d.smithi052.stdout:May 27 07:29:48 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-d[106255]: starting mds.d at 2024-05-27T07:29:49.165 INFO:journalctl@ceph.mds.d.smithi052.stdout:May 27 07:29:48 smithi052 bash[106221]: 377781175b2cb23f4f9fbb362b5b24d7361ffdd72d17fa2e26d16676442aa5c2 2024-05-27T07:29:49.165 INFO:journalctl@ceph.mds.d.smithi052.stdout:May 27 07:29:48 smithi052 systemd[1]: Started Ceph mds.d for 73df2636-1bf8-11ef-bc9b-c7b262605968. 2024-05-27T07:29:49.166 INFO:journalctl@ceph.mds.g.smithi052.stdout:May 27 07:29:48 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-g[106273]: starting mds.g at 2024-05-27T07:29:49.166 INFO:journalctl@ceph.mds.g.smithi052.stdout:May 27 07:29:48 smithi052 bash[106228]: 366d6f4b82cae2e92c867e97cde83ae500f5633b53b558e9f83dca56e4b4b222 2024-05-27T07:29:49.166 INFO:journalctl@ceph.mds.g.smithi052.stdout:May 27 07:29:48 smithi052 systemd[1]: Started Ceph mds.g for 73df2636-1bf8-11ef-bc9b-c7b262605968. 2024-05-27T07:29:49.170 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:29:49 smithi052 ceph-mon[31955]: daemon mds.d restarted 2024-05-27T07:29:49.330 INFO:teuthology.orchestra.run.smithi052.stderr:Inferring config /var/lib/ceph/73df2636-1bf8-11ef-bc9b-c7b262605968/mon.a/config 2024-05-27T07:29:49.408 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:29:49 smithi173 ceph-mon[37000]: daemon mds.d restarted 2024-05-27T07:29:49.409 INFO:journalctl@ceph.mds.l.smithi173.stdout:May 27 07:29:49 smithi173 systemd[1]: ceph-73df2636-1bf8-11ef-bc9b-c7b262605968@mds.l.service: Scheduled restart job, restart counter is at 4. 2024-05-27T07:29:49.409 INFO:journalctl@ceph.mds.l.smithi173.stdout:May 27 07:29:49 smithi173 systemd[1]: Stopped Ceph mds.l for 73df2636-1bf8-11ef-bc9b-c7b262605968. 2024-05-27T07:29:49.409 INFO:journalctl@ceph.mds.l.smithi173.stdout:May 27 07:29:49 smithi173 systemd[1]: Starting Ceph mds.l for 73df2636-1bf8-11ef-bc9b-c7b262605968... 2024-05-27T07:29:49.409 INFO:journalctl@ceph.mds.l.smithi173.stdout:May 27 07:29:49 smithi173 podman[73728]: 2024-05-27 07:29:49.257753076 +0000 UTC m=+0.017566328 image pull e36ea4fc8a5807e8d2701370e350a0de3856538797f7fba64cfce7e344bd6799 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6 2024-05-27T07:29:49.440 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:29:49 smithi105 ceph-mon[35977]: daemon mds.d restarted 2024-05-27T07:29:49.692 INFO:journalctl@ceph.mds.l.smithi173.stdout:May 27 07:29:49 smithi173 podman[73728]: 2024-05-27 07:29:49.418989915 +0000 UTC m=+0.178803146 container create d70a52e699b5e512e537a44cc9d1f077b52180fbafd7465fb3950b042a8eb975 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-l, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240521, ceph=True, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, GIT_CLEAN=True, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, org.label-schema.schema-version=1.0, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, maintainer=Guillaume Abrioux ) 2024-05-27T07:29:49.692 INFO:journalctl@ceph.mds.l.smithi173.stdout:May 27 07:29:49 smithi173 podman[73728]: 2024-05-27 07:29:49.663435773 +0000 UTC m=+0.423249013 container init d70a52e699b5e512e537a44cc9d1f077b52180fbafd7465fb3950b042a8eb975 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-l, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240521, org.label-schema.vendor=CentOS, GIT_CLEAN=True, org.label-schema.license=GPLv2, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5) 2024-05-27T07:29:49.692 INFO:journalctl@ceph.mds.l.smithi173.stdout:May 27 07:29:49 smithi173 podman[73728]: 2024-05-27 07:29:49.666904322 +0000 UTC m=+0.426717560 container start d70a52e699b5e512e537a44cc9d1f077b52180fbafd7465fb3950b042a8eb975 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-l, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, GIT_CLEAN=True, org.label-schema.build-date=20240521, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90) 2024-05-27T07:29:49.852 INFO:journalctl@ceph.mds.j.smithi052.stdout:May 27 07:29:49 smithi052 podman[106408]: 2024-05-27 07:29:49.493815744 +0000 UTC m=+0.079573765 image pull e36ea4fc8a5807e8d2701370e350a0de3856538797f7fba64cfce7e344bd6799 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6 2024-05-27T07:29:49.852 INFO:journalctl@ceph.mds.j.smithi052.stdout:May 27 07:29:49 smithi052 podman[106408]: 2024-05-27 07:29:49.729050508 +0000 UTC m=+0.314808526 container create d03f6d3925720a9a1b1ac9712defc07799179d80b7913701e4ef645e7a0b5cd1 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-j, org.label-schema.vendor=CentOS, GIT_CLEAN=True, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240521, ceph=True, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90) 2024-05-27T07:29:50.068 INFO:journalctl@ceph.mds.l.smithi173.stdout:May 27 07:29:49 smithi173 ceph-mds[73747]: -- 172.21.15.173:0/2891728611 <== mon.1 v2:172.21.15.173:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x557201487a40 con 0x5572014cfa80 2024-05-27T07:29:50.068 INFO:journalctl@ceph.mds.l.smithi173.stdout:May 27 07:29:49 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-l[73743]: starting mds.l at 2024-05-27T07:29:50.068 INFO:journalctl@ceph.mds.l.smithi173.stdout:May 27 07:29:49 smithi173 bash[73728]: d70a52e699b5e512e537a44cc9d1f077b52180fbafd7465fb3950b042a8eb975 2024-05-27T07:29:50.068 INFO:journalctl@ceph.mds.l.smithi173.stdout:May 27 07:29:49 smithi173 systemd[1]: Started Ceph mds.l for 73df2636-1bf8-11ef-bc9b-c7b262605968. 2024-05-27T07:29:50.147 INFO:journalctl@ceph.mds.a.smithi052.stdout:May 27 07:29:49 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-a[105634]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: In function 'std::set SnapClient::filter(const std::set&) const' thread 7fd926779700 time 2024-05-27T07:29:49.955460+0000 2024-05-27T07:29:50.147 INFO:journalctl@ceph.mds.a.smithi052.stdout:May 27 07:29:49 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-a[105634]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: 197: FAILED ceph_assert(cached_version > 0) 2024-05-27T07:29:50.147 INFO:journalctl@ceph.mds.a.smithi052.stdout:May 27 07:29:49 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-a[105634]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:29:50.147 INFO:journalctl@ceph.mds.a.smithi052.stdout:May 27 07:29:49 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-a[105634]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x127) [0x7fd938fbeba6] 2024-05-27T07:29:50.148 INFO:journalctl@ceph.mds.a.smithi052.stdout:May 27 07:29:49 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-a[105634]: 2: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7fd938fbedd1] 2024-05-27T07:29:50.148 INFO:journalctl@ceph.mds.a.smithi052.stdout:May 27 07:29:49 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-a[105634]: 3: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x564afde5423d] 2024-05-27T07:29:50.148 INFO:journalctl@ceph.mds.a.smithi052.stdout:May 27 07:29:49 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-a[105634]: 4: (SnapRealm::build_snap_set() const+0x323) [0x564afde3f971] 2024-05-27T07:29:50.148 INFO:journalctl@ceph.mds.a.smithi052.stdout:May 27 07:29:49 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-a[105634]: 5: (SnapRealm::check_cache() const+0x13e) [0x564afde3dfa0] 2024-05-27T07:29:50.148 INFO:journalctl@ceph.mds.a.smithi052.stdout:May 27 07:29:49 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-a[105634]: 6: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x564afde3e651] 2024-05-27T07:29:50.148 INFO:journalctl@ceph.mds.a.smithi052.stdout:May 27 07:29:49 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-a[105634]: 7: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x564afddd3347] 2024-05-27T07:29:50.148 INFO:journalctl@ceph.mds.a.smithi052.stdout:May 27 07:29:49 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-a[105634]: 8: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x564afdacf87e] 2024-05-27T07:29:50.148 INFO:journalctl@ceph.mds.a.smithi052.stdout:May 27 07:29:49 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-a[105634]: 9: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x2702) [0x564afdad4e94] 2024-05-27T07:29:50.148 INFO:journalctl@ceph.mds.a.smithi052.stdout:May 27 07:29:49 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-a[105634]: 10: (EUpdate::replay(MDSRank*)+0x63) [0x564afdadf969] 2024-05-27T07:29:50.148 INFO:journalctl@ceph.mds.a.smithi052.stdout:May 27 07:29:49 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-a[105634]: 11: (MDLog::_replay_thread()+0x1f38) [0x564afde7848c] 2024-05-27T07:29:50.148 INFO:journalctl@ceph.mds.a.smithi052.stdout:May 27 07:29:49 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-a[105634]: 12: (MDLog::ReplayThread::entry()+0x11) [0x564afde819cb] 2024-05-27T07:29:50.148 INFO:journalctl@ceph.mds.a.smithi052.stdout:May 27 07:29:49 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-a[105634]: 13: (Thread::entry_wrapper()+0x43) [0x7fd938f8d72d] 2024-05-27T07:29:50.149 INFO:journalctl@ceph.mds.a.smithi052.stdout:May 27 07:29:49 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-a[105634]: 14: (Thread::_entry_func(void*)+0xd) [0x7fd938f8d749] 2024-05-27T07:29:50.149 INFO:journalctl@ceph.mds.a.smithi052.stdout:May 27 07:29:49 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-a[105634]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7fd937b311ca] 2024-05-27T07:29:50.149 INFO:journalctl@ceph.mds.a.smithi052.stdout:May 27 07:29:49 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-a[105634]: 16: clone() 2024-05-27T07:29:50.149 INFO:journalctl@ceph.mds.a.smithi052.stdout:May 27 07:29:49 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-a[105634]: *** Caught signal (Aborted) ** 2024-05-27T07:29:50.149 INFO:journalctl@ceph.mds.a.smithi052.stdout:May 27 07:29:49 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-a[105634]: in thread 7fd926779700 thread_name:md_log_replay 2024-05-27T07:29:50.149 INFO:journalctl@ceph.mds.a.smithi052.stdout:May 27 07:29:49 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-a[105634]: 2024-05-27T07:29:49.959+0000 7fd926779700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: In function 'std::set SnapClient::filter(const std::set&) const' thread 7fd926779700 time 2024-05-27T07:29:49.955460+0000 2024-05-27T07:29:50.149 INFO:journalctl@ceph.mds.a.smithi052.stdout:May 27 07:29:49 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-a[105634]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: 197: FAILED ceph_assert(cached_version > 0) 2024-05-27T07:29:50.149 INFO:journalctl@ceph.mds.a.smithi052.stdout:May 27 07:29:49 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-a[105634]: 2024-05-27T07:29:50.149 INFO:journalctl@ceph.mds.a.smithi052.stdout:May 27 07:29:49 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-a[105634]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:29:50.149 INFO:journalctl@ceph.mds.a.smithi052.stdout:May 27 07:29:49 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-a[105634]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x127) [0x7fd938fbeba6] 2024-05-27T07:29:50.149 INFO:journalctl@ceph.mds.a.smithi052.stdout:May 27 07:29:49 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-a[105634]: 2: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7fd938fbedd1] 2024-05-27T07:29:50.150 INFO:journalctl@ceph.mds.a.smithi052.stdout:May 27 07:29:49 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-a[105634]: 3: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x564afde5423d] 2024-05-27T07:29:50.150 INFO:journalctl@ceph.mds.a.smithi052.stdout:May 27 07:29:49 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-a[105634]: 4: (SnapRealm::build_snap_set() const+0x323) [0x564afde3f971] 2024-05-27T07:29:50.150 INFO:journalctl@ceph.mds.a.smithi052.stdout:May 27 07:29:49 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-a[105634]: 5: (SnapRealm::check_cache() const+0x13e) [0x564afde3dfa0] 2024-05-27T07:29:50.150 INFO:journalctl@ceph.mds.a.smithi052.stdout:May 27 07:29:49 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-a[105634]: 6: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x564afde3e651] 2024-05-27T07:29:50.150 INFO:journalctl@ceph.mds.a.smithi052.stdout:May 27 07:29:49 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-a[105634]: 7: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x564afddd3347] 2024-05-27T07:29:50.150 INFO:journalctl@ceph.mds.a.smithi052.stdout:May 27 07:29:49 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-a[105634]: 8: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x564afdacf87e] 2024-05-27T07:29:50.150 INFO:journalctl@ceph.mds.a.smithi052.stdout:May 27 07:29:49 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-a[105634]: 9: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x2702) [0x564afdad4e94] 2024-05-27T07:29:50.150 INFO:journalctl@ceph.mds.a.smithi052.stdout:May 27 07:29:49 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-a[105634]: 10: (EUpdate::replay(MDSRank*)+0x63) [0x564afdadf969] 2024-05-27T07:29:50.150 INFO:journalctl@ceph.mds.a.smithi052.stdout:May 27 07:29:49 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-a[105634]: 11: (MDLog::_replay_thread()+0x1f38) [0x564afde7848c] 2024-05-27T07:29:50.150 INFO:journalctl@ceph.mds.a.smithi052.stdout:May 27 07:29:49 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-a[105634]: 12: (MDLog::ReplayThread::entry()+0x11) [0x564afde819cb] 2024-05-27T07:29:50.150 INFO:journalctl@ceph.mds.a.smithi052.stdout:May 27 07:29:49 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-a[105634]: 13: (Thread::entry_wrapper()+0x43) [0x7fd938f8d72d] 2024-05-27T07:29:50.151 INFO:journalctl@ceph.mds.a.smithi052.stdout:May 27 07:29:49 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-a[105634]: 14: (Thread::_entry_func(void*)+0xd) [0x7fd938f8d749] 2024-05-27T07:29:50.151 INFO:journalctl@ceph.mds.a.smithi052.stdout:May 27 07:29:49 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-a[105634]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7fd937b311ca] 2024-05-27T07:29:50.151 INFO:journalctl@ceph.mds.a.smithi052.stdout:May 27 07:29:49 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-a[105634]: 16: clone() 2024-05-27T07:29:50.151 INFO:journalctl@ceph.mds.a.smithi052.stdout:May 27 07:29:49 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-a[105634]: 2024-05-27T07:29:50.151 INFO:journalctl@ceph.mds.a.smithi052.stdout:May 27 07:29:49 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-a[105634]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:29:50.151 INFO:journalctl@ceph.mds.a.smithi052.stdout:May 27 07:29:49 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-a[105634]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7fd937b3bd20] 2024-05-27T07:29:50.151 INFO:journalctl@ceph.mds.a.smithi052.stdout:May 27 07:29:49 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-a[105634]: 2: gsignal() 2024-05-27T07:29:50.151 INFO:journalctl@ceph.mds.a.smithi052.stdout:May 27 07:29:49 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-a[105634]: 3: abort() 2024-05-27T07:29:50.151 INFO:journalctl@ceph.mds.a.smithi052.stdout:May 27 07:29:49 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-a[105634]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x26e) [0x7fd938fbeced] 2024-05-27T07:29:50.151 INFO:journalctl@ceph.mds.a.smithi052.stdout:May 27 07:29:49 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-a[105634]: 5: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7fd938fbedd1] 2024-05-27T07:29:50.152 INFO:journalctl@ceph.mds.a.smithi052.stdout:May 27 07:29:49 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-a[105634]: 6: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x564afde5423d] 2024-05-27T07:29:50.152 INFO:journalctl@ceph.mds.a.smithi052.stdout:May 27 07:29:49 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-a[105634]: 7: (SnapRealm::build_snap_set() const+0x323) [0x564afde3f971] 2024-05-27T07:29:50.152 INFO:journalctl@ceph.mds.a.smithi052.stdout:May 27 07:29:49 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-a[105634]: 8: (SnapRealm::check_cache() const+0x13e) [0x564afde3dfa0] 2024-05-27T07:29:50.152 INFO:journalctl@ceph.mds.a.smithi052.stdout:May 27 07:29:49 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-a[105634]: 9: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x564afde3e651] 2024-05-27T07:29:50.152 INFO:journalctl@ceph.mds.a.smithi052.stdout:May 27 07:29:49 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-a[105634]: 10: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x564afddd3347] 2024-05-27T07:29:50.152 INFO:journalctl@ceph.mds.a.smithi052.stdout:May 27 07:29:49 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-a[105634]: 11: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x564afdacf87e] 2024-05-27T07:29:50.152 INFO:journalctl@ceph.mds.a.smithi052.stdout:May 27 07:29:49 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-a[105634]: 12: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x2702) [0x564afdad4e94] 2024-05-27T07:29:50.152 INFO:journalctl@ceph.mds.a.smithi052.stdout:May 27 07:29:49 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-a[105634]: 13: (EUpdate::replay(MDSRank*)+0x63) [0x564afdadf969] 2024-05-27T07:29:50.152 INFO:journalctl@ceph.mds.a.smithi052.stdout:May 27 07:29:49 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-a[105634]: 14: (MDLog::_replay_thread()+0x1f38) [0x564afde7848c] 2024-05-27T07:29:50.152 INFO:journalctl@ceph.mds.a.smithi052.stdout:May 27 07:29:49 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-a[105634]: 15: (MDLog::ReplayThread::entry()+0x11) [0x564afde819cb] 2024-05-27T07:29:50.152 INFO:journalctl@ceph.mds.a.smithi052.stdout:May 27 07:29:49 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-a[105634]: 16: (Thread::entry_wrapper()+0x43) [0x7fd938f8d72d] 2024-05-27T07:29:50.152 INFO:journalctl@ceph.mds.a.smithi052.stdout:May 27 07:29:49 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-a[105634]: 17: (Thread::_entry_func(void*)+0xd) [0x7fd938f8d749] 2024-05-27T07:29:50.153 INFO:journalctl@ceph.mds.a.smithi052.stdout:May 27 07:29:49 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-a[105634]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7fd937b311ca] 2024-05-27T07:29:50.153 INFO:journalctl@ceph.mds.a.smithi052.stdout:May 27 07:29:49 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-a[105634]: 19: clone() 2024-05-27T07:29:50.153 INFO:journalctl@ceph.mds.a.smithi052.stdout:May 27 07:29:49 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-a[105634]: 2024-05-27T07:29:49.968+0000 7fd926779700 -1 *** Caught signal (Aborted) ** 2024-05-27T07:29:50.153 INFO:journalctl@ceph.mds.a.smithi052.stdout:May 27 07:29:49 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-a[105634]: in thread 7fd926779700 thread_name:md_log_replay 2024-05-27T07:29:50.153 INFO:journalctl@ceph.mds.a.smithi052.stdout:May 27 07:29:49 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-a[105634]: 2024-05-27T07:29:50.153 INFO:journalctl@ceph.mds.a.smithi052.stdout:May 27 07:29:49 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-a[105634]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:29:50.153 INFO:journalctl@ceph.mds.a.smithi052.stdout:May 27 07:29:49 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-a[105634]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7fd937b3bd20] 2024-05-27T07:29:50.153 INFO:journalctl@ceph.mds.a.smithi052.stdout:May 27 07:29:49 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-a[105634]: 2: gsignal() 2024-05-27T07:29:50.153 INFO:journalctl@ceph.mds.a.smithi052.stdout:May 27 07:29:49 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-a[105634]: 3: abort() 2024-05-27T07:29:50.153 INFO:journalctl@ceph.mds.a.smithi052.stdout:May 27 07:29:49 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-a[105634]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x26e) [0x7fd938fbeced] 2024-05-27T07:29:50.153 INFO:journalctl@ceph.mds.a.smithi052.stdout:May 27 07:29:49 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-a[105634]: 5: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7fd938fbedd1] 2024-05-27T07:29:50.153 INFO:journalctl@ceph.mds.a.smithi052.stdout:May 27 07:29:49 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-a[105634]: 6: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x564afde5423d] 2024-05-27T07:29:50.153 INFO:journalctl@ceph.mds.a.smithi052.stdout:May 27 07:29:49 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-a[105634]: 7: (SnapRealm::build_snap_set() const+0x323) [0x564afde3f971] 2024-05-27T07:29:50.154 INFO:journalctl@ceph.mds.a.smithi052.stdout:May 27 07:29:49 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-a[105634]: 8: (SnapRealm::check_cache() const+0x13e) [0x564afde3dfa0] 2024-05-27T07:29:50.154 INFO:journalctl@ceph.mds.a.smithi052.stdout:May 27 07:29:49 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-a[105634]: 9: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x564afde3e651] 2024-05-27T07:29:50.154 INFO:journalctl@ceph.mds.g.smithi052.stdout:May 27 07:29:49 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-g[106273]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: In function 'std::set SnapClient::filter(const std::set&) const' thread 7f3be1ec5700 time 2024-05-27T07:29:49.858274+0000 2024-05-27T07:29:50.154 INFO:journalctl@ceph.mds.g.smithi052.stdout:May 27 07:29:49 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-g[106273]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: 197: FAILED ceph_assert(cached_version > 0) 2024-05-27T07:29:50.154 INFO:journalctl@ceph.mds.g.smithi052.stdout:May 27 07:29:49 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-g[106273]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:29:50.154 INFO:journalctl@ceph.mds.g.smithi052.stdout:May 27 07:29:49 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-g[106273]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x127) [0x7f3bf470aba6] 2024-05-27T07:29:50.154 INFO:journalctl@ceph.mds.g.smithi052.stdout:May 27 07:29:49 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-g[106273]: 2: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7f3bf470add1] 2024-05-27T07:29:50.154 INFO:journalctl@ceph.mds.g.smithi052.stdout:May 27 07:29:49 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-g[106273]: 3: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x55a76ce6023d] 2024-05-27T07:29:50.154 INFO:journalctl@ceph.mds.g.smithi052.stdout:May 27 07:29:49 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-g[106273]: 4: (SnapRealm::build_snap_set() const+0x323) [0x55a76ce4b971] 2024-05-27T07:29:50.155 INFO:journalctl@ceph.mds.g.smithi052.stdout:May 27 07:29:49 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-g[106273]: 5: (SnapRealm::check_cache() const+0x13e) [0x55a76ce49fa0] 2024-05-27T07:29:50.155 INFO:journalctl@ceph.mds.g.smithi052.stdout:May 27 07:29:49 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-g[106273]: 6: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x55a76ce4a651] 2024-05-27T07:29:50.155 INFO:journalctl@ceph.mds.g.smithi052.stdout:May 27 07:29:49 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-g[106273]: 7: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x55a76cddf347] 2024-05-27T07:29:50.155 INFO:journalctl@ceph.mds.g.smithi052.stdout:May 27 07:29:49 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-g[106273]: 8: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x55a76cadb87e] 2024-05-27T07:29:50.155 INFO:journalctl@ceph.mds.g.smithi052.stdout:May 27 07:29:49 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-g[106273]: 9: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x2702) [0x55a76cae0e94] 2024-05-27T07:29:50.155 INFO:journalctl@ceph.mds.g.smithi052.stdout:May 27 07:29:49 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-g[106273]: 10: (EUpdate::replay(MDSRank*)+0x63) [0x55a76caeb969] 2024-05-27T07:29:50.155 INFO:journalctl@ceph.mds.g.smithi052.stdout:May 27 07:29:49 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-g[106273]: 11: (MDLog::_replay_thread()+0x1f38) [0x55a76ce8448c] 2024-05-27T07:29:50.155 INFO:journalctl@ceph.mds.g.smithi052.stdout:May 27 07:29:49 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-g[106273]: 12: (MDLog::ReplayThread::entry()+0x11) [0x55a76ce8d9cb] 2024-05-27T07:29:50.155 INFO:journalctl@ceph.mds.g.smithi052.stdout:May 27 07:29:49 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-g[106273]: 13: (Thread::entry_wrapper()+0x43) [0x7f3bf46d972d] 2024-05-27T07:29:50.155 INFO:journalctl@ceph.mds.g.smithi052.stdout:May 27 07:29:49 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-g[106273]: 14: (Thread::_entry_func(void*)+0xd) [0x7f3bf46d9749] 2024-05-27T07:29:50.155 INFO:journalctl@ceph.mds.g.smithi052.stdout:May 27 07:29:49 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-g[106273]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7f3bf327d1ca] 2024-05-27T07:29:50.155 INFO:journalctl@ceph.mds.g.smithi052.stdout:May 27 07:29:49 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-g[106273]: 16: clone() 2024-05-27T07:29:50.155 INFO:journalctl@ceph.mds.g.smithi052.stdout:May 27 07:29:49 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-g[106273]: 2024-05-27T07:29:49.862+0000 7f3be1ec5700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: In function 'std::set SnapClient::filter(const std::set&) const' thread 7f3be1ec5700 time 2024-05-27T07:29:49.858274+0000 2024-05-27T07:29:50.156 INFO:journalctl@ceph.mds.g.smithi052.stdout:May 27 07:29:49 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-g[106273]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: 197: FAILED ceph_assert(cached_version > 0) 2024-05-27T07:29:50.156 INFO:journalctl@ceph.mds.g.smithi052.stdout:May 27 07:29:49 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-g[106273]: 2024-05-27T07:29:50.156 INFO:journalctl@ceph.mds.g.smithi052.stdout:May 27 07:29:49 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-g[106273]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:29:50.156 INFO:journalctl@ceph.mds.g.smithi052.stdout:May 27 07:29:49 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-g[106273]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x127) [0x7f3bf470aba6] 2024-05-27T07:29:50.156 INFO:journalctl@ceph.mds.g.smithi052.stdout:May 27 07:29:49 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-g[106273]: 2: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7f3bf470add1] 2024-05-27T07:29:50.156 INFO:journalctl@ceph.mds.g.smithi052.stdout:May 27 07:29:49 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-g[106273]: 3: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x55a76ce6023d] 2024-05-27T07:29:50.156 INFO:journalctl@ceph.mds.g.smithi052.stdout:May 27 07:29:49 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-g[106273]: 4: (SnapRealm::build_snap_set() const+0x323) [0x55a76ce4b971] 2024-05-27T07:29:50.156 INFO:journalctl@ceph.mds.g.smithi052.stdout:May 27 07:29:49 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-g[106273]: 5: (SnapRealm::check_cache() const+0x13e) [0x55a76ce49fa0] 2024-05-27T07:29:50.156 INFO:journalctl@ceph.mds.g.smithi052.stdout:May 27 07:29:49 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-g[106273]: 6: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x55a76ce4a651] 2024-05-27T07:29:50.156 INFO:journalctl@ceph.mds.g.smithi052.stdout:May 27 07:29:49 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-g[106273]: 7: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x55a76cddf347] 2024-05-27T07:29:50.156 INFO:journalctl@ceph.mds.g.smithi052.stdout:May 27 07:29:49 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-g[106273]: 8: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x55a76cadb87e] 2024-05-27T07:29:50.156 INFO:journalctl@ceph.mds.g.smithi052.stdout:May 27 07:29:49 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-g[106273]: 9: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x2702) [0x55a76cae0e94] 2024-05-27T07:29:50.156 INFO:journalctl@ceph.mds.g.smithi052.stdout:May 27 07:29:49 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-g[106273]: 10: (EUpdate::replay(MDSRank*)+0x63) [0x55a76caeb969] 2024-05-27T07:29:50.157 INFO:journalctl@ceph.mds.g.smithi052.stdout:May 27 07:29:49 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-g[106273]: 11: (MDLog::_replay_thread()+0x1f38) [0x55a76ce8448c] 2024-05-27T07:29:50.157 INFO:journalctl@ceph.mds.g.smithi052.stdout:May 27 07:29:49 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-g[106273]: 12: (MDLog::ReplayThread::entry()+0x11) [0x55a76ce8d9cb] 2024-05-27T07:29:50.157 INFO:journalctl@ceph.mds.g.smithi052.stdout:May 27 07:29:49 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-g[106273]: 13: (Thread::entry_wrapper()+0x43) [0x7f3bf46d972d] 2024-05-27T07:29:50.157 INFO:journalctl@ceph.mds.g.smithi052.stdout:May 27 07:29:49 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-g[106273]: 14: (Thread::_entry_func(void*)+0xd) [0x7f3bf46d9749] 2024-05-27T07:29:50.157 INFO:journalctl@ceph.mds.g.smithi052.stdout:May 27 07:29:49 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-g[106273]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7f3bf327d1ca] 2024-05-27T07:29:50.157 INFO:journalctl@ceph.mds.g.smithi052.stdout:May 27 07:29:49 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-g[106273]: 16: clone() 2024-05-27T07:29:50.157 INFO:journalctl@ceph.mds.g.smithi052.stdout:May 27 07:29:49 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-g[106273]: 2024-05-27T07:29:50.157 INFO:journalctl@ceph.mds.g.smithi052.stdout:May 27 07:29:49 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-g[106273]: *** Caught signal (Aborted) ** 2024-05-27T07:29:50.157 INFO:journalctl@ceph.mds.g.smithi052.stdout:May 27 07:29:49 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-g[106273]: in thread 7f3be1ec5700 thread_name:md_log_replay 2024-05-27T07:29:50.157 INFO:journalctl@ceph.mds.g.smithi052.stdout:May 27 07:29:49 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-g[106273]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:29:50.157 INFO:journalctl@ceph.mds.g.smithi052.stdout:May 27 07:29:49 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-g[106273]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f3bf3287d20] 2024-05-27T07:29:50.157 INFO:journalctl@ceph.mds.g.smithi052.stdout:May 27 07:29:49 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-g[106273]: 2: gsignal() 2024-05-27T07:29:50.157 INFO:journalctl@ceph.mds.g.smithi052.stdout:May 27 07:29:49 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-g[106273]: 3: abort() 2024-05-27T07:29:50.158 INFO:journalctl@ceph.mds.g.smithi052.stdout:May 27 07:29:49 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-g[106273]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x26e) [0x7f3bf470aced] 2024-05-27T07:29:50.158 INFO:journalctl@ceph.mds.a.smithi052.stdout:May 27 07:29:49 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-a[105634]: 10: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x564afddd3347] 2024-05-27T07:29:50.158 INFO:journalctl@ceph.mds.a.smithi052.stdout:May 27 07:29:49 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-a[105634]: 11: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x564afdacf87e] 2024-05-27T07:29:50.158 INFO:journalctl@ceph.mds.a.smithi052.stdout:May 27 07:29:49 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-a[105634]: 12: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x2702) [0x564afdad4e94] 2024-05-27T07:29:50.158 INFO:journalctl@ceph.mds.a.smithi052.stdout:May 27 07:29:49 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-a[105634]: 13: (EUpdate::replay(MDSRank*)+0x63) [0x564afdadf969] 2024-05-27T07:29:50.158 INFO:journalctl@ceph.mds.a.smithi052.stdout:May 27 07:29:49 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-a[105634]: 14: (MDLog::_replay_thread()+0x1f38) [0x564afde7848c] 2024-05-27T07:29:50.158 INFO:journalctl@ceph.mds.a.smithi052.stdout:May 27 07:29:49 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-a[105634]: 15: (MDLog::ReplayThread::entry()+0x11) [0x564afde819cb] 2024-05-27T07:29:50.158 INFO:journalctl@ceph.mds.a.smithi052.stdout:May 27 07:29:49 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-a[105634]: 16: (Thread::entry_wrapper()+0x43) [0x7fd938f8d72d] 2024-05-27T07:29:50.158 INFO:journalctl@ceph.mds.a.smithi052.stdout:May 27 07:29:49 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-a[105634]: 17: (Thread::_entry_func(void*)+0xd) [0x7fd938f8d749] 2024-05-27T07:29:50.159 INFO:journalctl@ceph.mds.a.smithi052.stdout:May 27 07:29:49 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-a[105634]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7fd937b311ca] 2024-05-27T07:29:50.159 INFO:journalctl@ceph.mds.a.smithi052.stdout:May 27 07:29:49 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-a[105634]: 19: clone() 2024-05-27T07:29:50.159 INFO:journalctl@ceph.mds.a.smithi052.stdout:May 27 07:29:49 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-a[105634]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-05-27T07:29:50.159 INFO:journalctl@ceph.mds.a.smithi052.stdout:May 27 07:29:49 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-a[105634]: 2024-05-27T07:29:50.159 INFO:journalctl@ceph.mds.a.smithi052.stdout:May 27 07:29:50 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-a[105634]: -1> 2024-05-27T07:29:49.959+0000 7fd926779700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: In function 'std::set SnapClient::filter(const std::set&) const' thread 7fd926779700 time 2024-05-27T07:29:49.955460+0000 2024-05-27T07:29:50.159 INFO:journalctl@ceph.mds.a.smithi052.stdout:May 27 07:29:50 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-a[105634]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: 197: FAILED ceph_assert(cached_version > 0) 2024-05-27T07:29:50.159 INFO:journalctl@ceph.mds.a.smithi052.stdout:May 27 07:29:50 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-a[105634]: 2024-05-27T07:29:50.159 INFO:journalctl@ceph.mds.a.smithi052.stdout:May 27 07:29:50 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-a[105634]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:29:50.159 INFO:journalctl@ceph.mds.a.smithi052.stdout:May 27 07:29:50 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-a[105634]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x127) [0x7fd938fbeba6] 2024-05-27T07:29:50.159 INFO:journalctl@ceph.mds.a.smithi052.stdout:May 27 07:29:50 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-a[105634]: 2: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7fd938fbedd1] 2024-05-27T07:29:50.159 INFO:journalctl@ceph.mds.a.smithi052.stdout:May 27 07:29:50 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-a[105634]: 3: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x564afde5423d] 2024-05-27T07:29:50.159 INFO:journalctl@ceph.mds.a.smithi052.stdout:May 27 07:29:50 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-a[105634]: 4: (SnapRealm::build_snap_set() const+0x323) [0x564afde3f971] 2024-05-27T07:29:50.159 INFO:journalctl@ceph.mds.a.smithi052.stdout:May 27 07:29:50 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-a[105634]: 5: (SnapRealm::check_cache() const+0x13e) [0x564afde3dfa0] 2024-05-27T07:29:50.160 INFO:journalctl@ceph.mds.a.smithi052.stdout:May 27 07:29:50 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-a[105634]: 6: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x564afde3e651] 2024-05-27T07:29:50.160 INFO:journalctl@ceph.mds.a.smithi052.stdout:May 27 07:29:50 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-a[105634]: 7: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x564afddd3347] 2024-05-27T07:29:50.160 INFO:journalctl@ceph.mds.a.smithi052.stdout:May 27 07:29:50 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-a[105634]: 8: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x564afdacf87e] 2024-05-27T07:29:50.160 INFO:journalctl@ceph.mds.a.smithi052.stdout:May 27 07:29:50 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-a[105634]: 9: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x2702) [0x564afdad4e94] 2024-05-27T07:29:50.160 INFO:journalctl@ceph.mds.a.smithi052.stdout:May 27 07:29:50 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-a[105634]: 10: (EUpdate::replay(MDSRank*)+0x63) [0x564afdadf969] 2024-05-27T07:29:50.160 INFO:journalctl@ceph.mds.a.smithi052.stdout:May 27 07:29:50 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-a[105634]: 11: (MDLog::_replay_thread()+0x1f38) [0x564afde7848c] 2024-05-27T07:29:50.160 INFO:journalctl@ceph.mds.a.smithi052.stdout:May 27 07:29:50 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-a[105634]: 12: (MDLog::ReplayThread::entry()+0x11) [0x564afde819cb] 2024-05-27T07:29:50.160 INFO:journalctl@ceph.mds.a.smithi052.stdout:May 27 07:29:50 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-a[105634]: 13: (Thread::entry_wrapper()+0x43) [0x7fd938f8d72d] 2024-05-27T07:29:50.160 INFO:journalctl@ceph.mds.a.smithi052.stdout:May 27 07:29:50 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-a[105634]: 14: (Thread::_entry_func(void*)+0xd) [0x7fd938f8d749] 2024-05-27T07:29:50.160 INFO:journalctl@ceph.mds.a.smithi052.stdout:May 27 07:29:50 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-a[105634]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7fd937b311ca] 2024-05-27T07:29:50.160 INFO:journalctl@ceph.mds.a.smithi052.stdout:May 27 07:29:50 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-a[105634]: 16: clone() 2024-05-27T07:29:50.160 INFO:journalctl@ceph.mds.a.smithi052.stdout:May 27 07:29:50 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-a[105634]: 2024-05-27T07:29:50.160 INFO:journalctl@ceph.mds.a.smithi052.stdout:May 27 07:29:50 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-a[105634]: 0> 2024-05-27T07:29:49.968+0000 7fd926779700 -1 *** Caught signal (Aborted) ** 2024-05-27T07:29:50.160 INFO:journalctl@ceph.mds.a.smithi052.stdout:May 27 07:29:50 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-a[105634]: in thread 7fd926779700 thread_name:md_log_replay 2024-05-27T07:29:50.161 INFO:journalctl@ceph.mds.a.smithi052.stdout:May 27 07:29:50 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-a[105634]: 2024-05-27T07:29:50.161 INFO:journalctl@ceph.mds.a.smithi052.stdout:May 27 07:29:50 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-a[105634]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:29:50.161 INFO:journalctl@ceph.mds.a.smithi052.stdout:May 27 07:29:50 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-a[105634]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7fd937b3bd20] 2024-05-27T07:29:50.161 INFO:journalctl@ceph.mds.a.smithi052.stdout:May 27 07:29:50 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-a[105634]: 2: gsignal() 2024-05-27T07:29:50.161 INFO:journalctl@ceph.mds.a.smithi052.stdout:May 27 07:29:50 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-a[105634]: 3: abort() 2024-05-27T07:29:50.161 INFO:journalctl@ceph.mds.a.smithi052.stdout:May 27 07:29:50 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-a[105634]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x26e) [0x7fd938fbeced] 2024-05-27T07:29:50.161 INFO:journalctl@ceph.mds.a.smithi052.stdout:May 27 07:29:50 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-a[105634]: 5: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7fd938fbedd1] 2024-05-27T07:29:50.161 INFO:journalctl@ceph.mds.a.smithi052.stdout:May 27 07:29:50 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-a[105634]: 6: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x564afde5423d] 2024-05-27T07:29:50.161 INFO:journalctl@ceph.mds.a.smithi052.stdout:May 27 07:29:50 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-a[105634]: 7: (SnapRealm::build_snap_set() const+0x323) [0x564afde3f971] 2024-05-27T07:29:50.161 INFO:journalctl@ceph.mds.a.smithi052.stdout:May 27 07:29:50 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-a[105634]: 8: (SnapRealm::check_cache() const+0x13e) [0x564afde3dfa0] 2024-05-27T07:29:50.161 INFO:journalctl@ceph.mds.a.smithi052.stdout:May 27 07:29:50 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-a[105634]: 9: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x564afde3e651] 2024-05-27T07:29:50.161 INFO:journalctl@ceph.mds.a.smithi052.stdout:May 27 07:29:50 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-a[105634]: 10: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x564afddd3347] 2024-05-27T07:29:50.161 INFO:journalctl@ceph.mds.a.smithi052.stdout:May 27 07:29:50 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-a[105634]: 11: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x564afdacf87e] 2024-05-27T07:29:50.161 INFO:journalctl@ceph.mds.a.smithi052.stdout:May 27 07:29:50 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-a[105634]: 12: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x2702) [0x564afdad4e94] 2024-05-27T07:29:50.162 INFO:journalctl@ceph.mds.a.smithi052.stdout:May 27 07:29:50 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-a[105634]: 13: (EUpdate::replay(MDSRank*)+0x63) [0x564afdadf969] 2024-05-27T07:29:50.162 INFO:journalctl@ceph.mds.a.smithi052.stdout:May 27 07:29:50 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-a[105634]: 14: (MDLog::_replay_thread()+0x1f38) [0x564afde7848c] 2024-05-27T07:29:50.162 INFO:journalctl@ceph.mds.a.smithi052.stdout:May 27 07:29:50 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-a[105634]: 15: (MDLog::ReplayThread::entry()+0x11) [0x564afde819cb] 2024-05-27T07:29:50.162 INFO:journalctl@ceph.mds.g.smithi052.stdout:May 27 07:29:49 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-g[106273]: 5: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7f3bf470add1] 2024-05-27T07:29:50.162 INFO:journalctl@ceph.mds.g.smithi052.stdout:May 27 07:29:49 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-g[106273]: 6: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x55a76ce6023d] 2024-05-27T07:29:50.163 INFO:journalctl@ceph.mds.g.smithi052.stdout:May 27 07:29:49 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-g[106273]: 7: (SnapRealm::build_snap_set() const+0x323) [0x55a76ce4b971] 2024-05-27T07:29:50.163 INFO:journalctl@ceph.mds.g.smithi052.stdout:May 27 07:29:49 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-g[106273]: 8: (SnapRealm::check_cache() const+0x13e) [0x55a76ce49fa0] 2024-05-27T07:29:50.163 INFO:journalctl@ceph.mds.g.smithi052.stdout:May 27 07:29:49 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-g[106273]: 9: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x55a76ce4a651] 2024-05-27T07:29:50.163 INFO:journalctl@ceph.mds.g.smithi052.stdout:May 27 07:29:49 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-g[106273]: 10: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x55a76cddf347] 2024-05-27T07:29:50.163 INFO:journalctl@ceph.mds.g.smithi052.stdout:May 27 07:29:49 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-g[106273]: 11: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x55a76cadb87e] 2024-05-27T07:29:50.163 INFO:journalctl@ceph.mds.g.smithi052.stdout:May 27 07:29:49 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-g[106273]: 12: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x2702) [0x55a76cae0e94] 2024-05-27T07:29:50.163 INFO:journalctl@ceph.mds.g.smithi052.stdout:May 27 07:29:49 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-g[106273]: 13: (EUpdate::replay(MDSRank*)+0x63) [0x55a76caeb969] 2024-05-27T07:29:50.163 INFO:journalctl@ceph.mds.g.smithi052.stdout:May 27 07:29:49 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-g[106273]: 14: (MDLog::_replay_thread()+0x1f38) [0x55a76ce8448c] 2024-05-27T07:29:50.163 INFO:journalctl@ceph.mds.g.smithi052.stdout:May 27 07:29:49 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-g[106273]: 15: (MDLog::ReplayThread::entry()+0x11) [0x55a76ce8d9cb] 2024-05-27T07:29:50.163 INFO:journalctl@ceph.mds.g.smithi052.stdout:May 27 07:29:49 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-g[106273]: 16: (Thread::entry_wrapper()+0x43) [0x7f3bf46d972d] 2024-05-27T07:29:50.163 INFO:journalctl@ceph.mds.g.smithi052.stdout:May 27 07:29:49 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-g[106273]: 17: (Thread::_entry_func(void*)+0xd) [0x7f3bf46d9749] 2024-05-27T07:29:50.163 INFO:journalctl@ceph.mds.g.smithi052.stdout:May 27 07:29:49 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-g[106273]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7f3bf327d1ca] 2024-05-27T07:29:50.164 INFO:journalctl@ceph.mds.g.smithi052.stdout:May 27 07:29:49 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-g[106273]: 19: clone() 2024-05-27T07:29:50.164 INFO:journalctl@ceph.mds.g.smithi052.stdout:May 27 07:29:49 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-g[106273]: 2024-05-27T07:29:49.866+0000 7f3be1ec5700 -1 *** Caught signal (Aborted) ** 2024-05-27T07:29:50.164 INFO:journalctl@ceph.mds.g.smithi052.stdout:May 27 07:29:49 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-g[106273]: in thread 7f3be1ec5700 thread_name:md_log_replay 2024-05-27T07:29:50.164 INFO:journalctl@ceph.mds.g.smithi052.stdout:May 27 07:29:49 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-g[106273]: 2024-05-27T07:29:50.164 INFO:journalctl@ceph.mds.g.smithi052.stdout:May 27 07:29:49 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-g[106273]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:29:50.164 INFO:journalctl@ceph.mds.g.smithi052.stdout:May 27 07:29:49 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-g[106273]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f3bf3287d20] 2024-05-27T07:29:50.164 INFO:journalctl@ceph.mds.g.smithi052.stdout:May 27 07:29:49 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-g[106273]: 2: gsignal() 2024-05-27T07:29:50.164 INFO:journalctl@ceph.mds.g.smithi052.stdout:May 27 07:29:49 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-g[106273]: 3: abort() 2024-05-27T07:29:50.164 INFO:journalctl@ceph.mds.g.smithi052.stdout:May 27 07:29:49 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-g[106273]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x26e) [0x7f3bf470aced] 2024-05-27T07:29:50.164 INFO:journalctl@ceph.mds.g.smithi052.stdout:May 27 07:29:49 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-g[106273]: 5: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7f3bf470add1] 2024-05-27T07:29:50.164 INFO:journalctl@ceph.mds.g.smithi052.stdout:May 27 07:29:49 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-g[106273]: 6: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x55a76ce6023d] 2024-05-27T07:29:50.164 INFO:journalctl@ceph.mds.g.smithi052.stdout:May 27 07:29:49 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-g[106273]: 7: (SnapRealm::build_snap_set() const+0x323) [0x55a76ce4b971] 2024-05-27T07:29:50.164 INFO:journalctl@ceph.mds.g.smithi052.stdout:May 27 07:29:49 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-g[106273]: 8: (SnapRealm::check_cache() const+0x13e) [0x55a76ce49fa0] 2024-05-27T07:29:50.165 INFO:journalctl@ceph.mds.g.smithi052.stdout:May 27 07:29:49 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-g[106273]: 9: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x55a76ce4a651] 2024-05-27T07:29:50.165 INFO:journalctl@ceph.mds.g.smithi052.stdout:May 27 07:29:49 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-g[106273]: 10: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x55a76cddf347] 2024-05-27T07:29:50.165 INFO:journalctl@ceph.mds.g.smithi052.stdout:May 27 07:29:49 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-g[106273]: 11: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x55a76cadb87e] 2024-05-27T07:29:50.165 INFO:journalctl@ceph.mds.g.smithi052.stdout:May 27 07:29:49 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-g[106273]: 12: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x2702) [0x55a76cae0e94] 2024-05-27T07:29:50.165 INFO:journalctl@ceph.mds.g.smithi052.stdout:May 27 07:29:49 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-g[106273]: 13: (EUpdate::replay(MDSRank*)+0x63) [0x55a76caeb969] 2024-05-27T07:29:50.165 INFO:journalctl@ceph.mds.g.smithi052.stdout:May 27 07:29:49 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-g[106273]: 14: (MDLog::_replay_thread()+0x1f38) [0x55a76ce8448c] 2024-05-27T07:29:50.165 INFO:journalctl@ceph.mds.g.smithi052.stdout:May 27 07:29:49 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-g[106273]: 15: (MDLog::ReplayThread::entry()+0x11) [0x55a76ce8d9cb] 2024-05-27T07:29:50.165 INFO:journalctl@ceph.mds.g.smithi052.stdout:May 27 07:29:49 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-g[106273]: 16: (Thread::entry_wrapper()+0x43) [0x7f3bf46d972d] 2024-05-27T07:29:50.165 INFO:journalctl@ceph.mds.g.smithi052.stdout:May 27 07:29:49 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-g[106273]: 17: (Thread::_entry_func(void*)+0xd) [0x7f3bf46d9749] 2024-05-27T07:29:50.165 INFO:journalctl@ceph.mds.g.smithi052.stdout:May 27 07:29:49 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-g[106273]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7f3bf327d1ca] 2024-05-27T07:29:50.165 INFO:journalctl@ceph.mds.g.smithi052.stdout:May 27 07:29:49 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-g[106273]: 19: clone() 2024-05-27T07:29:50.165 INFO:journalctl@ceph.mds.g.smithi052.stdout:May 27 07:29:49 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-g[106273]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-05-27T07:29:50.165 INFO:journalctl@ceph.mds.g.smithi052.stdout:May 27 07:29:49 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-g[106273]: 2024-05-27T07:29:50.165 INFO:journalctl@ceph.mds.g.smithi052.stdout:May 27 07:29:49 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-g[106273]: -1> 2024-05-27T07:29:49.862+0000 7f3be1ec5700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: In function 'std::set SnapClient::filter(const std::set&) const' thread 7f3be1ec5700 time 2024-05-27T07:29:49.858274+0000 2024-05-27T07:29:50.166 INFO:journalctl@ceph.mds.g.smithi052.stdout:May 27 07:29:49 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-g[106273]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: 197: FAILED ceph_assert(cached_version > 0) 2024-05-27T07:29:50.166 INFO:journalctl@ceph.mds.g.smithi052.stdout:May 27 07:29:49 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-g[106273]: 2024-05-27T07:29:50.166 INFO:journalctl@ceph.mds.g.smithi052.stdout:May 27 07:29:49 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-g[106273]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:29:50.166 INFO:journalctl@ceph.mds.g.smithi052.stdout:May 27 07:29:49 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-g[106273]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x127) [0x7f3bf470aba6] 2024-05-27T07:29:50.166 INFO:journalctl@ceph.mds.g.smithi052.stdout:May 27 07:29:49 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-g[106273]: 2: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7f3bf470add1] 2024-05-27T07:29:50.166 INFO:journalctl@ceph.mds.g.smithi052.stdout:May 27 07:29:49 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-g[106273]: 3: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x55a76ce6023d] 2024-05-27T07:29:50.166 INFO:journalctl@ceph.mds.g.smithi052.stdout:May 27 07:29:49 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-g[106273]: 4: (SnapRealm::build_snap_set() const+0x323) [0x55a76ce4b971] 2024-05-27T07:29:50.166 INFO:journalctl@ceph.mds.g.smithi052.stdout:May 27 07:29:49 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-g[106273]: 5: (SnapRealm::check_cache() const+0x13e) [0x55a76ce49fa0] 2024-05-27T07:29:50.166 INFO:journalctl@ceph.mds.g.smithi052.stdout:May 27 07:29:49 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-g[106273]: 6: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x55a76ce4a651] 2024-05-27T07:29:50.166 INFO:journalctl@ceph.mds.g.smithi052.stdout:May 27 07:29:49 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-g[106273]: 7: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x55a76cddf347] 2024-05-27T07:29:50.167 INFO:journalctl@ceph.mds.g.smithi052.stdout:May 27 07:29:49 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-g[106273]: 8: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x55a76cadb87e] 2024-05-27T07:29:50.167 INFO:journalctl@ceph.mds.g.smithi052.stdout:May 27 07:29:49 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-g[106273]: 9: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x2702) [0x55a76cae0e94] 2024-05-27T07:29:50.167 INFO:journalctl@ceph.mds.g.smithi052.stdout:May 27 07:29:49 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-g[106273]: 10: (EUpdate::replay(MDSRank*)+0x63) [0x55a76caeb969] 2024-05-27T07:29:50.167 INFO:journalctl@ceph.mds.g.smithi052.stdout:May 27 07:29:49 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-g[106273]: 11: (MDLog::_replay_thread()+0x1f38) [0x55a76ce8448c] 2024-05-27T07:29:50.167 INFO:journalctl@ceph.mds.g.smithi052.stdout:May 27 07:29:49 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-g[106273]: 12: (MDLog::ReplayThread::entry()+0x11) [0x55a76ce8d9cb] 2024-05-27T07:29:50.167 INFO:journalctl@ceph.mds.g.smithi052.stdout:May 27 07:29:49 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-g[106273]: 13: (Thread::entry_wrapper()+0x43) [0x7f3bf46d972d] 2024-05-27T07:29:50.167 INFO:journalctl@ceph.mds.g.smithi052.stdout:May 27 07:29:49 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-g[106273]: 14: (Thread::_entry_func(void*)+0xd) [0x7f3bf46d9749] 2024-05-27T07:29:50.167 INFO:journalctl@ceph.mds.g.smithi052.stdout:May 27 07:29:49 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-g[106273]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7f3bf327d1ca] 2024-05-27T07:29:50.167 INFO:journalctl@ceph.mds.g.smithi052.stdout:May 27 07:29:49 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-g[106273]: 16: clone() 2024-05-27T07:29:50.167 INFO:journalctl@ceph.mds.g.smithi052.stdout:May 27 07:29:49 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-g[106273]: 2024-05-27T07:29:50.167 INFO:journalctl@ceph.mds.g.smithi052.stdout:May 27 07:29:49 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-g[106273]: 0> 2024-05-27T07:29:49.866+0000 7f3be1ec5700 -1 *** Caught signal (Aborted) ** 2024-05-27T07:29:50.167 INFO:journalctl@ceph.mds.g.smithi052.stdout:May 27 07:29:49 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-g[106273]: in thread 7f3be1ec5700 thread_name:md_log_replay 2024-05-27T07:29:50.168 INFO:journalctl@ceph.mds.g.smithi052.stdout:May 27 07:29:49 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-g[106273]: 2024-05-27T07:29:50.168 INFO:journalctl@ceph.mds.g.smithi052.stdout:May 27 07:29:49 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-g[106273]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:29:50.168 INFO:journalctl@ceph.mds.g.smithi052.stdout:May 27 07:29:49 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-g[106273]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f3bf3287d20] 2024-05-27T07:29:50.168 INFO:journalctl@ceph.mds.g.smithi052.stdout:May 27 07:29:49 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-g[106273]: 2: gsignal() 2024-05-27T07:29:50.168 INFO:journalctl@ceph.mds.g.smithi052.stdout:May 27 07:29:49 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-g[106273]: 3: abort() 2024-05-27T07:29:50.168 INFO:journalctl@ceph.mds.g.smithi052.stdout:May 27 07:29:49 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-g[106273]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x26e) [0x7f3bf470aced] 2024-05-27T07:29:50.168 INFO:journalctl@ceph.mds.g.smithi052.stdout:May 27 07:29:49 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-g[106273]: 5: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7f3bf470add1] 2024-05-27T07:29:50.168 INFO:journalctl@ceph.mds.g.smithi052.stdout:May 27 07:29:49 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-g[106273]: 6: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x55a76ce6023d] 2024-05-27T07:29:50.168 INFO:journalctl@ceph.mds.g.smithi052.stdout:May 27 07:29:49 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-g[106273]: 7: (SnapRealm::build_snap_set() const+0x323) [0x55a76ce4b971] 2024-05-27T07:29:50.168 INFO:journalctl@ceph.mds.g.smithi052.stdout:May 27 07:29:49 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-g[106273]: 8: (SnapRealm::check_cache() const+0x13e) [0x55a76ce49fa0] 2024-05-27T07:29:50.168 INFO:journalctl@ceph.mds.g.smithi052.stdout:May 27 07:29:49 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-g[106273]: 9: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x55a76ce4a651] 2024-05-27T07:29:50.168 INFO:journalctl@ceph.mds.g.smithi052.stdout:May 27 07:29:49 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-g[106273]: 10: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x55a76cddf347] 2024-05-27T07:29:50.168 INFO:journalctl@ceph.mds.g.smithi052.stdout:May 27 07:29:49 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-g[106273]: 11: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x55a76cadb87e] 2024-05-27T07:29:50.169 INFO:journalctl@ceph.mds.g.smithi052.stdout:May 27 07:29:49 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-g[106273]: 12: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x2702) [0x55a76cae0e94] 2024-05-27T07:29:50.169 INFO:journalctl@ceph.mds.g.smithi052.stdout:May 27 07:29:49 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-g[106273]: 13: (EUpdate::replay(MDSRank*)+0x63) [0x55a76caeb969] 2024-05-27T07:29:50.169 INFO:journalctl@ceph.mds.g.smithi052.stdout:May 27 07:29:49 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-g[106273]: 14: (MDLog::_replay_thread()+0x1f38) [0x55a76ce8448c] 2024-05-27T07:29:50.169 INFO:journalctl@ceph.mds.g.smithi052.stdout:May 27 07:29:49 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-g[106273]: 15: (MDLog::ReplayThread::entry()+0x11) [0x55a76ce8d9cb] 2024-05-27T07:29:50.169 INFO:journalctl@ceph.mds.g.smithi052.stdout:May 27 07:29:49 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-g[106273]: 16: (Thread::entry_wrapper()+0x43) [0x7f3bf46d972d] 2024-05-27T07:29:50.169 INFO:journalctl@ceph.mds.g.smithi052.stdout:May 27 07:29:49 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-g[106273]: 17: (Thread::_entry_func(void*)+0xd) [0x7f3bf46d9749] 2024-05-27T07:29:50.169 INFO:journalctl@ceph.mds.g.smithi052.stdout:May 27 07:29:49 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-g[106273]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7f3bf327d1ca] 2024-05-27T07:29:50.169 INFO:journalctl@ceph.mds.g.smithi052.stdout:May 27 07:29:49 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-g[106273]: 19: clone() 2024-05-27T07:29:50.169 INFO:journalctl@ceph.mds.g.smithi052.stdout:May 27 07:29:49 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-g[106273]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-05-27T07:29:50.169 INFO:journalctl@ceph.mds.g.smithi052.stdout:May 27 07:29:49 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-g[106273]: 2024-05-27T07:29:50.169 INFO:journalctl@ceph.mds.g.smithi052.stdout:May 27 07:29:49 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-g[106273]: -1> 2024-05-27T07:29:49.862+0000 7f3be1ec5700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: In function 'std::set SnapClient::filter(const std::set&) const' thread 7f3be1ec5700 time 2024-05-27T07:29:49.858274+0000 2024-05-27T07:29:50.169 INFO:journalctl@ceph.mds.g.smithi052.stdout:May 27 07:29:49 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-g[106273]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: 197: FAILED ceph_assert(cached_version > 0) 2024-05-27T07:29:50.169 INFO:journalctl@ceph.mds.g.smithi052.stdout:May 27 07:29:49 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-g[106273]: 2024-05-27T07:29:50.169 INFO:journalctl@ceph.mds.g.smithi052.stdout:May 27 07:29:49 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-g[106273]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:29:50.170 INFO:journalctl@ceph.mds.g.smithi052.stdout:May 27 07:29:49 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-g[106273]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x127) [0x7f3bf470aba6] 2024-05-27T07:29:50.170 INFO:journalctl@ceph.mds.g.smithi052.stdout:May 27 07:29:49 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-g[106273]: 2: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7f3bf470add1] 2024-05-27T07:29:50.170 INFO:journalctl@ceph.mds.g.smithi052.stdout:May 27 07:29:49 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-g[106273]: 3: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x55a76ce6023d] 2024-05-27T07:29:50.170 INFO:journalctl@ceph.mds.g.smithi052.stdout:May 27 07:29:49 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-g[106273]: 4: (SnapRealm::build_snap_set() const+0x323) [0x55a76ce4b971] 2024-05-27T07:29:50.170 INFO:journalctl@ceph.mds.g.smithi052.stdout:May 27 07:29:49 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-g[106273]: 5: (SnapRealm::check_cache() const+0x13e) [0x55a76ce49fa0] 2024-05-27T07:29:50.170 INFO:journalctl@ceph.mds.g.smithi052.stdout:May 27 07:29:49 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-g[106273]: 6: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x55a76ce4a651] 2024-05-27T07:29:50.170 INFO:journalctl@ceph.mds.g.smithi052.stdout:May 27 07:29:49 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-g[106273]: 7: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x55a76cddf347] 2024-05-27T07:29:50.170 INFO:journalctl@ceph.mds.g.smithi052.stdout:May 27 07:29:49 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-g[106273]: 8: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x55a76cadb87e] 2024-05-27T07:29:50.170 INFO:journalctl@ceph.mds.g.smithi052.stdout:May 27 07:29:49 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-g[106273]: 9: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x2702) [0x55a76cae0e94] 2024-05-27T07:29:50.171 INFO:journalctl@ceph.mds.g.smithi052.stdout:May 27 07:29:49 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-g[106273]: 10: (EUpdate::replay(MDSRank*)+0x63) [0x55a76caeb969] 2024-05-27T07:29:50.171 INFO:journalctl@ceph.mds.g.smithi052.stdout:May 27 07:29:49 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-g[106273]: 11: (MDLog::_replay_thread()+0x1f38) [0x55a76ce8448c] 2024-05-27T07:29:50.171 INFO:journalctl@ceph.mds.g.smithi052.stdout:May 27 07:29:49 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-g[106273]: 12: (MDLog::ReplayThread::entry()+0x11) [0x55a76ce8d9cb] 2024-05-27T07:29:50.171 INFO:journalctl@ceph.mds.g.smithi052.stdout:May 27 07:29:49 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-g[106273]: 13: (Thread::entry_wrapper()+0x43) [0x7f3bf46d972d] 2024-05-27T07:29:50.171 INFO:journalctl@ceph.mds.g.smithi052.stdout:May 27 07:29:49 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-g[106273]: 14: (Thread::_entry_func(void*)+0xd) [0x7f3bf46d9749] 2024-05-27T07:29:50.171 INFO:journalctl@ceph.mds.g.smithi052.stdout:May 27 07:29:49 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-g[106273]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7f3bf327d1ca] 2024-05-27T07:29:50.172 INFO:journalctl@ceph.mds.g.smithi052.stdout:May 27 07:29:49 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-g[106273]: 16: clone() 2024-05-27T07:29:50.172 INFO:journalctl@ceph.mds.g.smithi052.stdout:May 27 07:29:49 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-g[106273]: 2024-05-27T07:29:50.172 INFO:journalctl@ceph.mds.g.smithi052.stdout:May 27 07:29:49 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-g[106273]: 0> 2024-05-27T07:29:49.866+0000 7f3be1ec5700 -1 *** Caught signal (Aborted) ** 2024-05-27T07:29:50.172 INFO:journalctl@ceph.mds.g.smithi052.stdout:May 27 07:29:49 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-g[106273]: in thread 7f3be1ec5700 thread_name:md_log_replay 2024-05-27T07:29:50.172 INFO:journalctl@ceph.mds.g.smithi052.stdout:May 27 07:29:49 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-g[106273]: 2024-05-27T07:29:50.172 INFO:journalctl@ceph.mds.g.smithi052.stdout:May 27 07:29:49 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-g[106273]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:29:50.173 INFO:journalctl@ceph.mds.g.smithi052.stdout:May 27 07:29:49 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-g[106273]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f3bf3287d20] 2024-05-27T07:29:50.173 INFO:journalctl@ceph.mds.g.smithi052.stdout:May 27 07:29:49 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-g[106273]: 2: gsignal() 2024-05-27T07:29:50.173 INFO:journalctl@ceph.mds.g.smithi052.stdout:May 27 07:29:49 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-g[106273]: 3: abort() 2024-05-27T07:29:50.173 INFO:journalctl@ceph.mds.g.smithi052.stdout:May 27 07:29:49 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-g[106273]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x26e) [0x7f3bf470aced] 2024-05-27T07:29:50.173 INFO:journalctl@ceph.mds.g.smithi052.stdout:May 27 07:29:49 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-g[106273]: 5: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7f3bf470add1] 2024-05-27T07:29:50.173 INFO:journalctl@ceph.mds.g.smithi052.stdout:May 27 07:29:49 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-g[106273]: 6: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x55a76ce6023d] 2024-05-27T07:29:50.173 INFO:journalctl@ceph.mds.g.smithi052.stdout:May 27 07:29:49 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-g[106273]: 7: (SnapRealm::build_snap_set() const+0x323) [0x55a76ce4b971] 2024-05-27T07:29:50.173 INFO:journalctl@ceph.mds.g.smithi052.stdout:May 27 07:29:49 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-g[106273]: 8: (SnapRealm::check_cache() const+0x13e) [0x55a76ce49fa0] 2024-05-27T07:29:50.173 INFO:journalctl@ceph.mds.g.smithi052.stdout:May 27 07:29:49 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-g[106273]: 9: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x55a76ce4a651] 2024-05-27T07:29:50.173 INFO:journalctl@ceph.mds.g.smithi052.stdout:May 27 07:29:49 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-g[106273]: 10: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x55a76cddf347] 2024-05-27T07:29:50.173 INFO:journalctl@ceph.mds.g.smithi052.stdout:May 27 07:29:49 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-g[106273]: 11: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x55a76cadb87e] 2024-05-27T07:29:50.173 INFO:journalctl@ceph.mds.g.smithi052.stdout:May 27 07:29:49 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-g[106273]: 12: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x2702) [0x55a76cae0e94] 2024-05-27T07:29:50.173 INFO:journalctl@ceph.mds.g.smithi052.stdout:May 27 07:29:49 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-g[106273]: 13: (EUpdate::replay(MDSRank*)+0x63) [0x55a76caeb969] 2024-05-27T07:29:50.173 INFO:journalctl@ceph.mds.g.smithi052.stdout:May 27 07:29:49 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-g[106273]: 14: (MDLog::_replay_thread()+0x1f38) [0x55a76ce8448c] 2024-05-27T07:29:50.174 INFO:journalctl@ceph.mds.g.smithi052.stdout:May 27 07:29:49 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-g[106273]: 15: (MDLog::ReplayThread::entry()+0x11) [0x55a76ce8d9cb] 2024-05-27T07:29:50.174 INFO:journalctl@ceph.mds.g.smithi052.stdout:May 27 07:29:49 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-g[106273]: 16: (Thread::entry_wrapper()+0x43) [0x7f3bf46d972d] 2024-05-27T07:29:50.174 INFO:journalctl@ceph.mds.g.smithi052.stdout:May 27 07:29:49 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-g[106273]: 17: (Thread::_entry_func(void*)+0xd) [0x7f3bf46d9749] 2024-05-27T07:29:50.174 INFO:journalctl@ceph.mds.g.smithi052.stdout:May 27 07:29:49 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-g[106273]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7f3bf327d1ca] 2024-05-27T07:29:50.174 INFO:journalctl@ceph.mds.g.smithi052.stdout:May 27 07:29:49 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-g[106273]: 19: clone() 2024-05-27T07:29:50.174 INFO:journalctl@ceph.mds.g.smithi052.stdout:May 27 07:29:49 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-g[106273]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-05-27T07:29:50.174 INFO:journalctl@ceph.mds.g.smithi052.stdout:May 27 07:29:49 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-g[106273]: 2024-05-27T07:29:50.175 INFO:journalctl@ceph.mds.a.smithi052.stdout:May 27 07:29:50 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-a[105634]: 16: (Thread::entry_wrapper()+0x43) [0x7fd938f8d72d] 2024-05-27T07:29:50.175 INFO:journalctl@ceph.mds.a.smithi052.stdout:May 27 07:29:50 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-a[105634]: 17: (Thread::_entry_func(void*)+0xd) [0x7fd938f8d749] 2024-05-27T07:29:50.175 INFO:journalctl@ceph.mds.a.smithi052.stdout:May 27 07:29:50 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-a[105634]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7fd937b311ca] 2024-05-27T07:29:50.175 INFO:journalctl@ceph.mds.a.smithi052.stdout:May 27 07:29:50 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-a[105634]: 19: clone() 2024-05-27T07:29:50.176 INFO:journalctl@ceph.mds.a.smithi052.stdout:May 27 07:29:50 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-a[105634]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-05-27T07:29:50.176 INFO:journalctl@ceph.mds.a.smithi052.stdout:May 27 07:29:50 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-a[105634]: 2024-05-27T07:29:50.176 INFO:journalctl@ceph.mds.a.smithi052.stdout:May 27 07:29:50 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-a[105634]: -2> 2024-05-27T07:29:49.959+0000 7fd926779700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: In function 'std::set SnapClient::filter(const std::set&) const' thread 7fd926779700 time 2024-05-27T07:29:49.955460+0000 2024-05-27T07:29:50.176 INFO:journalctl@ceph.mds.a.smithi052.stdout:May 27 07:29:50 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-a[105634]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: 197: FAILED ceph_assert(cached_version > 0) 2024-05-27T07:29:50.176 INFO:journalctl@ceph.mds.a.smithi052.stdout:May 27 07:29:50 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-a[105634]: 2024-05-27T07:29:50.176 INFO:journalctl@ceph.mds.a.smithi052.stdout:May 27 07:29:50 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-a[105634]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:29:50.176 INFO:journalctl@ceph.mds.a.smithi052.stdout:May 27 07:29:50 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-a[105634]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x127) [0x7fd938fbeba6] 2024-05-27T07:29:50.176 INFO:journalctl@ceph.mds.a.smithi052.stdout:May 27 07:29:50 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-a[105634]: 2: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7fd938fbedd1] 2024-05-27T07:29:50.176 INFO:journalctl@ceph.mds.a.smithi052.stdout:May 27 07:29:50 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-a[105634]: 3: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x564afde5423d] 2024-05-27T07:29:50.176 INFO:journalctl@ceph.mds.a.smithi052.stdout:May 27 07:29:50 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-a[105634]: 4: (SnapRealm::build_snap_set() const+0x323) [0x564afde3f971] 2024-05-27T07:29:50.176 INFO:journalctl@ceph.mds.a.smithi052.stdout:May 27 07:29:50 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-a[105634]: 5: (SnapRealm::check_cache() const+0x13e) [0x564afde3dfa0] 2024-05-27T07:29:50.176 INFO:journalctl@ceph.mds.a.smithi052.stdout:May 27 07:29:50 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-a[105634]: 6: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x564afde3e651] 2024-05-27T07:29:50.177 INFO:journalctl@ceph.mds.a.smithi052.stdout:May 27 07:29:50 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-a[105634]: 7: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x564afddd3347] 2024-05-27T07:29:50.177 INFO:journalctl@ceph.mds.a.smithi052.stdout:May 27 07:29:50 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-a[105634]: 8: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x564afdacf87e] 2024-05-27T07:29:50.177 INFO:journalctl@ceph.mds.a.smithi052.stdout:May 27 07:29:50 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-a[105634]: 9: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x2702) [0x564afdad4e94] 2024-05-27T07:29:50.177 INFO:journalctl@ceph.mds.a.smithi052.stdout:May 27 07:29:50 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-a[105634]: 10: (EUpdate::replay(MDSRank*)+0x63) [0x564afdadf969] 2024-05-27T07:29:50.177 INFO:journalctl@ceph.mds.a.smithi052.stdout:May 27 07:29:50 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-a[105634]: 11: (MDLog::_replay_thread()+0x1f38) [0x564afde7848c] 2024-05-27T07:29:50.177 INFO:journalctl@ceph.mds.a.smithi052.stdout:May 27 07:29:50 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-a[105634]: 12: (MDLog::ReplayThread::entry()+0x11) [0x564afde819cb] 2024-05-27T07:29:50.177 INFO:journalctl@ceph.mds.a.smithi052.stdout:May 27 07:29:50 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-a[105634]: 13: (Thread::entry_wrapper()+0x43) [0x7fd938f8d72d] 2024-05-27T07:29:50.177 INFO:journalctl@ceph.mds.a.smithi052.stdout:May 27 07:29:50 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-a[105634]: 14: (Thread::_entry_func(void*)+0xd) [0x7fd938f8d749] 2024-05-27T07:29:50.177 INFO:journalctl@ceph.mds.a.smithi052.stdout:May 27 07:29:50 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-a[105634]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7fd937b311ca] 2024-05-27T07:29:50.177 INFO:journalctl@ceph.mds.a.smithi052.stdout:May 27 07:29:50 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-a[105634]: 16: clone() 2024-05-27T07:29:50.177 INFO:journalctl@ceph.mds.a.smithi052.stdout:May 27 07:29:50 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-a[105634]: 2024-05-27T07:29:50.177 INFO:journalctl@ceph.mds.a.smithi052.stdout:May 27 07:29:50 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-a[105634]: -1> 2024-05-27T07:29:49.968+0000 7fd926779700 -1 *** Caught signal (Aborted) ** 2024-05-27T07:29:50.177 INFO:journalctl@ceph.mds.a.smithi052.stdout:May 27 07:29:50 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-a[105634]: in thread 7fd926779700 thread_name:md_log_replay 2024-05-27T07:29:50.178 INFO:journalctl@ceph.mds.a.smithi052.stdout:May 27 07:29:50 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-a[105634]: 2024-05-27T07:29:50.178 INFO:journalctl@ceph.mds.a.smithi052.stdout:May 27 07:29:50 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-a[105634]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:29:50.178 INFO:journalctl@ceph.mds.a.smithi052.stdout:May 27 07:29:50 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-a[105634]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7fd937b3bd20] 2024-05-27T07:29:50.178 INFO:journalctl@ceph.mds.a.smithi052.stdout:May 27 07:29:50 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-a[105634]: 2: gsignal() 2024-05-27T07:29:50.178 INFO:journalctl@ceph.mds.a.smithi052.stdout:May 27 07:29:50 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-a[105634]: 3: abort() 2024-05-27T07:29:50.178 INFO:journalctl@ceph.mds.a.smithi052.stdout:May 27 07:29:50 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-a[105634]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x26e) [0x7fd938fbeced] 2024-05-27T07:29:50.178 INFO:journalctl@ceph.mds.a.smithi052.stdout:May 27 07:29:50 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-a[105634]: 5: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7fd938fbedd1] 2024-05-27T07:29:50.178 INFO:journalctl@ceph.mds.a.smithi052.stdout:May 27 07:29:50 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-a[105634]: 6: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x564afde5423d] 2024-05-27T07:29:50.178 INFO:journalctl@ceph.mds.a.smithi052.stdout:May 27 07:29:50 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-a[105634]: 7: (SnapRealm::build_snap_set() const+0x323) [0x564afde3f971] 2024-05-27T07:29:50.178 INFO:journalctl@ceph.mds.a.smithi052.stdout:May 27 07:29:50 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-a[105634]: 8: (SnapRealm::check_cache() const+0x13e) [0x564afde3dfa0] 2024-05-27T07:29:50.178 INFO:journalctl@ceph.mds.a.smithi052.stdout:May 27 07:29:50 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-a[105634]: 9: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x564afde3e651] 2024-05-27T07:29:50.179 INFO:journalctl@ceph.mds.a.smithi052.stdout:May 27 07:29:50 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-a[105634]: 10: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x564afddd3347] 2024-05-27T07:29:50.179 INFO:journalctl@ceph.mds.a.smithi052.stdout:May 27 07:29:50 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-a[105634]: 11: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x564afdacf87e] 2024-05-27T07:29:50.179 INFO:journalctl@ceph.mds.a.smithi052.stdout:May 27 07:29:50 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-a[105634]: 12: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x2702) [0x564afdad4e94] 2024-05-27T07:29:50.179 INFO:journalctl@ceph.mds.a.smithi052.stdout:May 27 07:29:50 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-a[105634]: 13: (EUpdate::replay(MDSRank*)+0x63) [0x564afdadf969] 2024-05-27T07:29:50.179 INFO:journalctl@ceph.mds.a.smithi052.stdout:May 27 07:29:50 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-a[105634]: 14: (MDLog::_replay_thread()+0x1f38) [0x564afde7848c] 2024-05-27T07:29:50.179 INFO:journalctl@ceph.mds.a.smithi052.stdout:May 27 07:29:50 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-a[105634]: 15: (MDLog::ReplayThread::entry()+0x11) [0x564afde819cb] 2024-05-27T07:29:50.179 INFO:journalctl@ceph.mds.a.smithi052.stdout:May 27 07:29:50 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-a[105634]: 16: (Thread::entry_wrapper()+0x43) [0x7fd938f8d72d] 2024-05-27T07:29:50.179 INFO:journalctl@ceph.mds.a.smithi052.stdout:May 27 07:29:50 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-a[105634]: 17: (Thread::_entry_func(void*)+0xd) [0x7fd938f8d749] 2024-05-27T07:29:50.179 INFO:journalctl@ceph.mds.a.smithi052.stdout:May 27 07:29:50 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-a[105634]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7fd937b311ca] 2024-05-27T07:29:50.179 INFO:journalctl@ceph.mds.a.smithi052.stdout:May 27 07:29:50 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-a[105634]: 19: clone() 2024-05-27T07:29:50.179 INFO:journalctl@ceph.mds.a.smithi052.stdout:May 27 07:29:50 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-a[105634]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-05-27T07:29:50.179 INFO:journalctl@ceph.mds.a.smithi052.stdout:May 27 07:29:50 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-a[105634]: 2024-05-27T07:29:50.353 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:29:50 smithi173 ceph-mon[37000]: pgmap v134: 129 pgs: 129 active+clean; 267 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.9 MiB/s rd, 5.2 MiB/s wr, 560 op/s 2024-05-27T07:29:50.353 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:29:50 smithi173 ceph-mon[37000]: osdmap e107: 12 total, 12 up, 12 in 2024-05-27T07:29:50.353 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:29:50 smithi173 ceph-mon[37000]: mds.? [v2:172.21.15.52:6836/2351486841,v1:172.21.15.52:6837/2351486841] up:boot 2024-05-27T07:29:50.353 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:29:50 smithi173 ceph-mon[37000]: daemon mds.g restarted 2024-05-27T07:29:50.353 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:29:50 smithi173 ceph-mon[37000]: fsmap cephfs:5 4 up:standby-replay 2 up:standby 5 up:active 2024-05-27T07:29:50.353 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:29:50 smithi173 ceph-mon[37000]: from='mgr.15333 172.21.15.52:0/416389082' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "d"}]: dispatch 2024-05-27T07:29:50.353 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:29:50 smithi173 ceph-mon[37000]: osdmap e108: 12 total, 12 up, 12 in 2024-05-27T07:29:50.353 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:29:50 smithi173 ceph-mon[37000]: mds.? [v2:172.21.15.52:6838/3051157409,v1:172.21.15.52:6839/3051157409] up:boot 2024-05-27T07:29:50.353 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:29:50 smithi173 ceph-mon[37000]: MDS daemon mds.j is removed because it is dead or otherwise unavailable. 2024-05-27T07:29:50.353 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:29:50 smithi173 ceph-mon[37000]: fsmap cephfs:5 5 up:standby-replay 1 up:standby 5 up:active 2024-05-27T07:29:50.354 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:29:50 smithi173 ceph-mon[37000]: from='mgr.15333 172.21.15.52:0/416389082' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2024-05-27T07:29:50.354 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:29:50 smithi173 ceph-mon[37000]: fsmap cephfs:5 5 up:standby-replay 5 up:active 2024-05-27T07:29:50.354 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:29:50 smithi173 ceph-mon[37000]: daemon mds.l restarted 2024-05-27T07:29:50.354 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:29:50 smithi173 ceph-mon[37000]: osdmap e109: 12 total, 12 up, 12 in 2024-05-27T07:29:50.354 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:29:50 smithi173 ceph-mon[37000]: mds.? [v2:172.21.15.173:6832/1177755063,v1:172.21.15.173:6833/1177755063] up:boot 2024-05-27T07:29:50.354 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:29:50 smithi173 ceph-mon[37000]: fsmap cephfs:5 5 up:standby-replay 5 up:active 2024-05-27T07:29:50.354 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:29:50 smithi173 ceph-mon[37000]: from='mgr.15333 172.21.15.52:0/416389082' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "l"}]: dispatch 2024-05-27T07:29:50.355 INFO:journalctl@ceph.mds.l.smithi173.stdout:May 27 07:29:50 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-l[73743]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: In function 'std::set SnapClient::filter(const std::set&) const' thread 7fa8082cc700 time 2024-05-27T07:29:50.078215+0000 2024-05-27T07:29:50.355 INFO:journalctl@ceph.mds.l.smithi173.stdout:May 27 07:29:50 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-l[73743]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: 197: FAILED ceph_assert(cached_version > 0) 2024-05-27T07:29:50.355 INFO:journalctl@ceph.mds.l.smithi173.stdout:May 27 07:29:50 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-l[73743]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:29:50.355 INFO:journalctl@ceph.mds.l.smithi173.stdout:May 27 07:29:50 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-l[73743]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x127) [0x7fa81ab11ba6] 2024-05-27T07:29:50.355 INFO:journalctl@ceph.mds.l.smithi173.stdout:May 27 07:29:50 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-l[73743]: 2: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7fa81ab11dd1] 2024-05-27T07:29:50.355 INFO:journalctl@ceph.mds.l.smithi173.stdout:May 27 07:29:50 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-l[73743]: 3: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x5571fe90223d] 2024-05-27T07:29:50.355 INFO:journalctl@ceph.mds.l.smithi173.stdout:May 27 07:29:50 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-l[73743]: 4: (SnapRealm::build_snap_set() const+0x323) [0x5571fe8ed971] 2024-05-27T07:29:50.355 INFO:journalctl@ceph.mds.l.smithi173.stdout:May 27 07:29:50 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-l[73743]: 5: (SnapRealm::check_cache() const+0x13e) [0x5571fe8ebfa0] 2024-05-27T07:29:50.355 INFO:journalctl@ceph.mds.l.smithi173.stdout:May 27 07:29:50 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-l[73743]: 6: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x5571fe8ec651] 2024-05-27T07:29:50.356 INFO:journalctl@ceph.mds.l.smithi173.stdout:May 27 07:29:50 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-l[73743]: 7: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x5571fe881347] 2024-05-27T07:29:50.356 INFO:journalctl@ceph.mds.l.smithi173.stdout:May 27 07:29:50 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-l[73743]: 8: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x5571fe57d87e] 2024-05-27T07:29:50.356 INFO:journalctl@ceph.mds.l.smithi173.stdout:May 27 07:29:50 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-l[73743]: 9: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x2702) [0x5571fe582e94] 2024-05-27T07:29:50.356 INFO:journalctl@ceph.mds.l.smithi173.stdout:May 27 07:29:50 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-l[73743]: 10: (EUpdate::replay(MDSRank*)+0x63) [0x5571fe58d969] 2024-05-27T07:29:50.356 INFO:journalctl@ceph.mds.l.smithi173.stdout:May 27 07:29:50 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-l[73743]: 11: (MDLog::_replay_thread()+0x1f38) [0x5571fe92648c] 2024-05-27T07:29:50.356 INFO:journalctl@ceph.mds.l.smithi173.stdout:May 27 07:29:50 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-l[73743]: 12: (MDLog::ReplayThread::entry()+0x11) [0x5571fe92f9cb] 2024-05-27T07:29:50.356 INFO:journalctl@ceph.mds.l.smithi173.stdout:May 27 07:29:50 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-l[73743]: 13: (Thread::entry_wrapper()+0x43) [0x7fa81aae072d] 2024-05-27T07:29:50.356 INFO:journalctl@ceph.mds.l.smithi173.stdout:May 27 07:29:50 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-l[73743]: 14: (Thread::_entry_func(void*)+0xd) [0x7fa81aae0749] 2024-05-27T07:29:50.357 INFO:journalctl@ceph.mds.l.smithi173.stdout:May 27 07:29:50 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-l[73743]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7fa8196841ca] 2024-05-27T07:29:50.357 INFO:journalctl@ceph.mds.l.smithi173.stdout:May 27 07:29:50 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-l[73743]: 16: clone() 2024-05-27T07:29:50.357 INFO:journalctl@ceph.mds.l.smithi173.stdout:May 27 07:29:50 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-l[73743]: 2024-05-27T07:29:50.079+0000 7fa8082cc700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: In function 'std::set SnapClient::filter(const std::set&) const' thread 7fa8082cc700 time 2024-05-27T07:29:50.078215+0000 2024-05-27T07:29:50.357 INFO:journalctl@ceph.mds.l.smithi173.stdout:May 27 07:29:50 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-l[73743]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: 197: FAILED ceph_assert(cached_version > 0) 2024-05-27T07:29:50.357 INFO:journalctl@ceph.mds.l.smithi173.stdout:May 27 07:29:50 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-l[73743]: 2024-05-27T07:29:50.357 INFO:journalctl@ceph.mds.l.smithi173.stdout:May 27 07:29:50 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-l[73743]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:29:50.357 INFO:journalctl@ceph.mds.l.smithi173.stdout:May 27 07:29:50 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-l[73743]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x127) [0x7fa81ab11ba6] 2024-05-27T07:29:50.357 INFO:journalctl@ceph.mds.l.smithi173.stdout:May 27 07:29:50 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-l[73743]: 2: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7fa81ab11dd1] 2024-05-27T07:29:50.357 INFO:journalctl@ceph.mds.l.smithi173.stdout:May 27 07:29:50 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-l[73743]: 3: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x5571fe90223d] 2024-05-27T07:29:50.357 INFO:journalctl@ceph.mds.l.smithi173.stdout:May 27 07:29:50 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-l[73743]: 4: (SnapRealm::build_snap_set() const+0x323) [0x5571fe8ed971] 2024-05-27T07:29:50.357 INFO:journalctl@ceph.mds.l.smithi173.stdout:May 27 07:29:50 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-l[73743]: 5: (SnapRealm::check_cache() const+0x13e) [0x5571fe8ebfa0] 2024-05-27T07:29:50.358 INFO:journalctl@ceph.mds.l.smithi173.stdout:May 27 07:29:50 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-l[73743]: 6: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x5571fe8ec651] 2024-05-27T07:29:50.358 INFO:journalctl@ceph.mds.l.smithi173.stdout:May 27 07:29:50 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-l[73743]: 7: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x5571fe881347] 2024-05-27T07:29:50.358 INFO:journalctl@ceph.mds.l.smithi173.stdout:May 27 07:29:50 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-l[73743]: 8: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x5571fe57d87e] 2024-05-27T07:29:50.358 INFO:journalctl@ceph.mds.l.smithi173.stdout:May 27 07:29:50 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-l[73743]: 9: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x2702) [0x5571fe582e94] 2024-05-27T07:29:50.358 INFO:journalctl@ceph.mds.l.smithi173.stdout:May 27 07:29:50 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-l[73743]: 10: (EUpdate::replay(MDSRank*)+0x63) [0x5571fe58d969] 2024-05-27T07:29:50.359 INFO:journalctl@ceph.mds.l.smithi173.stdout:May 27 07:29:50 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-l[73743]: 11: (MDLog::_replay_thread()+0x1f38) [0x5571fe92648c] 2024-05-27T07:29:50.359 INFO:journalctl@ceph.mds.l.smithi173.stdout:May 27 07:29:50 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-l[73743]: 12: (MDLog::ReplayThread::entry()+0x11) [0x5571fe92f9cb] 2024-05-27T07:29:50.359 INFO:journalctl@ceph.mds.l.smithi173.stdout:May 27 07:29:50 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-l[73743]: 13: (Thread::entry_wrapper()+0x43) [0x7fa81aae072d] 2024-05-27T07:29:50.359 INFO:journalctl@ceph.mds.l.smithi173.stdout:May 27 07:29:50 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-l[73743]: 14: (Thread::_entry_func(void*)+0xd) [0x7fa81aae0749] 2024-05-27T07:29:50.359 INFO:journalctl@ceph.mds.l.smithi173.stdout:May 27 07:29:50 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-l[73743]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7fa8196841ca] 2024-05-27T07:29:50.359 INFO:journalctl@ceph.mds.l.smithi173.stdout:May 27 07:29:50 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-l[73743]: 16: clone() 2024-05-27T07:29:50.359 INFO:journalctl@ceph.mds.l.smithi173.stdout:May 27 07:29:50 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-l[73743]: 2024-05-27T07:29:50.359 INFO:journalctl@ceph.mds.l.smithi173.stdout:May 27 07:29:50 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-l[73743]: *** Caught signal (Aborted) ** 2024-05-27T07:29:50.359 INFO:journalctl@ceph.mds.l.smithi173.stdout:May 27 07:29:50 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-l[73743]: in thread 7fa8082cc700 thread_name:md_log_replay 2024-05-27T07:29:50.359 INFO:journalctl@ceph.mds.l.smithi173.stdout:May 27 07:29:50 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-l[73743]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:29:50.360 INFO:journalctl@ceph.mds.l.smithi173.stdout:May 27 07:29:50 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-l[73743]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7fa81968ed20] 2024-05-27T07:29:50.360 INFO:journalctl@ceph.mds.l.smithi173.stdout:May 27 07:29:50 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-l[73743]: 2: gsignal() 2024-05-27T07:29:50.360 INFO:journalctl@ceph.mds.l.smithi173.stdout:May 27 07:29:50 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-l[73743]: 3: abort() 2024-05-27T07:29:50.360 INFO:journalctl@ceph.mds.l.smithi173.stdout:May 27 07:29:50 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-l[73743]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x26e) [0x7fa81ab11ced] 2024-05-27T07:29:50.360 INFO:journalctl@ceph.mds.l.smithi173.stdout:May 27 07:29:50 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-l[73743]: 5: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7fa81ab11dd1] 2024-05-27T07:29:50.360 INFO:journalctl@ceph.mds.l.smithi173.stdout:May 27 07:29:50 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-l[73743]: 6: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x5571fe90223d] 2024-05-27T07:29:50.360 INFO:journalctl@ceph.mds.l.smithi173.stdout:May 27 07:29:50 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-l[73743]: 7: (SnapRealm::build_snap_set() const+0x323) [0x5571fe8ed971] 2024-05-27T07:29:50.360 INFO:journalctl@ceph.mds.l.smithi173.stdout:May 27 07:29:50 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-l[73743]: 8: (SnapRealm::check_cache() const+0x13e) [0x5571fe8ebfa0] 2024-05-27T07:29:50.360 INFO:journalctl@ceph.mds.l.smithi173.stdout:May 27 07:29:50 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-l[73743]: 9: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x5571fe8ec651] 2024-05-27T07:29:50.360 INFO:journalctl@ceph.mds.l.smithi173.stdout:May 27 07:29:50 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-l[73743]: 10: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x5571fe881347] 2024-05-27T07:29:50.360 INFO:journalctl@ceph.mds.l.smithi173.stdout:May 27 07:29:50 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-l[73743]: 11: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x5571fe57d87e] 2024-05-27T07:29:50.360 INFO:journalctl@ceph.mds.l.smithi173.stdout:May 27 07:29:50 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-l[73743]: 12: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x2702) [0x5571fe582e94] 2024-05-27T07:29:50.361 INFO:journalctl@ceph.mds.l.smithi173.stdout:May 27 07:29:50 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-l[73743]: 13: (EUpdate::replay(MDSRank*)+0x63) [0x5571fe58d969] 2024-05-27T07:29:50.361 INFO:journalctl@ceph.mds.l.smithi173.stdout:May 27 07:29:50 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-l[73743]: 14: (MDLog::_replay_thread()+0x1f38) [0x5571fe92648c] 2024-05-27T07:29:50.361 INFO:journalctl@ceph.mds.l.smithi173.stdout:May 27 07:29:50 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-l[73743]: 15: (MDLog::ReplayThread::entry()+0x11) [0x5571fe92f9cb] 2024-05-27T07:29:50.361 INFO:journalctl@ceph.mds.l.smithi173.stdout:May 27 07:29:50 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-l[73743]: 16: (Thread::entry_wrapper()+0x43) [0x7fa81aae072d] 2024-05-27T07:29:50.361 INFO:journalctl@ceph.mds.l.smithi173.stdout:May 27 07:29:50 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-l[73743]: 17: (Thread::_entry_func(void*)+0xd) [0x7fa81aae0749] 2024-05-27T07:29:50.361 INFO:journalctl@ceph.mds.l.smithi173.stdout:May 27 07:29:50 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-l[73743]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7fa8196841ca] 2024-05-27T07:29:50.361 INFO:journalctl@ceph.mds.l.smithi173.stdout:May 27 07:29:50 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-l[73743]: 19: clone() 2024-05-27T07:29:50.361 INFO:journalctl@ceph.mds.l.smithi173.stdout:May 27 07:29:50 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-l[73743]: 2024-05-27T07:29:50.082+0000 7fa8082cc700 -1 *** Caught signal (Aborted) ** 2024-05-27T07:29:50.361 INFO:journalctl@ceph.mds.l.smithi173.stdout:May 27 07:29:50 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-l[73743]: in thread 7fa8082cc700 thread_name:md_log_replay 2024-05-27T07:29:50.361 INFO:journalctl@ceph.mds.l.smithi173.stdout:May 27 07:29:50 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-l[73743]: 2024-05-27T07:29:50.361 INFO:journalctl@ceph.mds.l.smithi173.stdout:May 27 07:29:50 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-l[73743]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:29:50.361 INFO:journalctl@ceph.mds.l.smithi173.stdout:May 27 07:29:50 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-l[73743]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7fa81968ed20] 2024-05-27T07:29:50.362 INFO:journalctl@ceph.mds.l.smithi173.stdout:May 27 07:29:50 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-l[73743]: 2: gsignal() 2024-05-27T07:29:50.362 INFO:journalctl@ceph.mds.l.smithi173.stdout:May 27 07:29:50 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-l[73743]: 3: abort() 2024-05-27T07:29:50.362 INFO:journalctl@ceph.mds.l.smithi173.stdout:May 27 07:29:50 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-l[73743]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x26e) [0x7fa81ab11ced] 2024-05-27T07:29:50.362 INFO:journalctl@ceph.mds.l.smithi173.stdout:May 27 07:29:50 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-l[73743]: 5: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7fa81ab11dd1] 2024-05-27T07:29:50.362 INFO:journalctl@ceph.mds.l.smithi173.stdout:May 27 07:29:50 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-l[73743]: 6: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x5571fe90223d] 2024-05-27T07:29:50.362 INFO:journalctl@ceph.mds.l.smithi173.stdout:May 27 07:29:50 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-l[73743]: 7: (SnapRealm::build_snap_set() const+0x323) [0x5571fe8ed971] 2024-05-27T07:29:50.362 INFO:journalctl@ceph.mds.l.smithi173.stdout:May 27 07:29:50 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-l[73743]: 8: (SnapRealm::check_cache() const+0x13e) [0x5571fe8ebfa0] 2024-05-27T07:29:50.362 INFO:journalctl@ceph.mds.l.smithi173.stdout:May 27 07:29:50 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-l[73743]: 9: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x5571fe8ec651] 2024-05-27T07:29:50.362 INFO:journalctl@ceph.mds.l.smithi173.stdout:May 27 07:29:50 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-l[73743]: 10: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x5571fe881347] 2024-05-27T07:29:50.362 INFO:journalctl@ceph.mds.l.smithi173.stdout:May 27 07:29:50 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-l[73743]: 11: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x5571fe57d87e] 2024-05-27T07:29:50.362 INFO:journalctl@ceph.mds.l.smithi173.stdout:May 27 07:29:50 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-l[73743]: 12: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x2702) [0x5571fe582e94] 2024-05-27T07:29:50.362 INFO:journalctl@ceph.mds.l.smithi173.stdout:May 27 07:29:50 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-l[73743]: 13: (EUpdate::replay(MDSRank*)+0x63) [0x5571fe58d969] 2024-05-27T07:29:50.362 INFO:journalctl@ceph.mds.l.smithi173.stdout:May 27 07:29:50 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-l[73743]: 14: (MDLog::_replay_thread()+0x1f38) [0x5571fe92648c] 2024-05-27T07:29:50.363 INFO:journalctl@ceph.mds.l.smithi173.stdout:May 27 07:29:50 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-l[73743]: 15: (MDLog::ReplayThread::entry()+0x11) [0x5571fe92f9cb] 2024-05-27T07:29:50.363 INFO:journalctl@ceph.mds.l.smithi173.stdout:May 27 07:29:50 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-l[73743]: 16: (Thread::entry_wrapper()+0x43) [0x7fa81aae072d] 2024-05-27T07:29:50.363 INFO:journalctl@ceph.mds.l.smithi173.stdout:May 27 07:29:50 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-l[73743]: 17: (Thread::_entry_func(void*)+0xd) [0x7fa81aae0749] 2024-05-27T07:29:50.363 INFO:journalctl@ceph.mds.l.smithi173.stdout:May 27 07:29:50 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-l[73743]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7fa8196841ca] 2024-05-27T07:29:50.363 INFO:journalctl@ceph.mds.l.smithi173.stdout:May 27 07:29:50 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-l[73743]: 19: clone() 2024-05-27T07:29:50.363 INFO:journalctl@ceph.mds.l.smithi173.stdout:May 27 07:29:50 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-l[73743]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-05-27T07:29:50.363 INFO:journalctl@ceph.mds.l.smithi173.stdout:May 27 07:29:50 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-l[73743]: 2024-05-27T07:29:50.363 INFO:journalctl@ceph.mds.l.smithi173.stdout:May 27 07:29:50 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-l[73743]: -1> 2024-05-27T07:29:50.079+0000 7fa8082cc700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: In function 'std::set SnapClient::filter(const std::set&) const' thread 7fa8082cc700 time 2024-05-27T07:29:50.078215+0000 2024-05-27T07:29:50.363 INFO:journalctl@ceph.mds.l.smithi173.stdout:May 27 07:29:50 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-l[73743]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: 197: FAILED ceph_assert(cached_version > 0) 2024-05-27T07:29:50.363 INFO:journalctl@ceph.mds.l.smithi173.stdout:May 27 07:29:50 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-l[73743]: 2024-05-27T07:29:50.363 INFO:journalctl@ceph.mds.l.smithi173.stdout:May 27 07:29:50 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-l[73743]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:29:50.363 INFO:journalctl@ceph.mds.l.smithi173.stdout:May 27 07:29:50 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-l[73743]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x127) [0x7fa81ab11ba6] 2024-05-27T07:29:50.364 INFO:journalctl@ceph.mds.l.smithi173.stdout:May 27 07:29:50 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-l[73743]: 2: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7fa81ab11dd1] 2024-05-27T07:29:50.364 INFO:journalctl@ceph.mds.l.smithi173.stdout:May 27 07:29:50 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-l[73743]: 3: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x5571fe90223d] 2024-05-27T07:29:50.364 INFO:journalctl@ceph.mds.l.smithi173.stdout:May 27 07:29:50 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-l[73743]: 4: (SnapRealm::build_snap_set() const+0x323) [0x5571fe8ed971] 2024-05-27T07:29:50.364 INFO:journalctl@ceph.mds.l.smithi173.stdout:May 27 07:29:50 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-l[73743]: 5: (SnapRealm::check_cache() const+0x13e) [0x5571fe8ebfa0] 2024-05-27T07:29:50.364 INFO:journalctl@ceph.mds.l.smithi173.stdout:May 27 07:29:50 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-l[73743]: 6: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x5571fe8ec651] 2024-05-27T07:29:50.364 INFO:journalctl@ceph.mds.l.smithi173.stdout:May 27 07:29:50 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-l[73743]: 7: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x5571fe881347] 2024-05-27T07:29:50.364 INFO:journalctl@ceph.mds.l.smithi173.stdout:May 27 07:29:50 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-l[73743]: 8: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x5571fe57d87e] 2024-05-27T07:29:50.365 INFO:journalctl@ceph.mds.l.smithi173.stdout:May 27 07:29:50 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-l[73743]: 9: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x2702) [0x5571fe582e94] 2024-05-27T07:29:50.365 INFO:journalctl@ceph.mds.l.smithi173.stdout:May 27 07:29:50 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-l[73743]: 10: (EUpdate::replay(MDSRank*)+0x63) [0x5571fe58d969] 2024-05-27T07:29:50.365 INFO:journalctl@ceph.mds.l.smithi173.stdout:May 27 07:29:50 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-l[73743]: 11: (MDLog::_replay_thread()+0x1f38) [0x5571fe92648c] 2024-05-27T07:29:50.365 INFO:journalctl@ceph.mds.l.smithi173.stdout:May 27 07:29:50 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-l[73743]: 12: (MDLog::ReplayThread::entry()+0x11) [0x5571fe92f9cb] 2024-05-27T07:29:50.365 INFO:journalctl@ceph.mds.l.smithi173.stdout:May 27 07:29:50 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-l[73743]: 13: (Thread::entry_wrapper()+0x43) [0x7fa81aae072d] 2024-05-27T07:29:50.365 INFO:journalctl@ceph.mds.l.smithi173.stdout:May 27 07:29:50 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-l[73743]: 14: (Thread::_entry_func(void*)+0xd) [0x7fa81aae0749] 2024-05-27T07:29:50.365 INFO:journalctl@ceph.mds.l.smithi173.stdout:May 27 07:29:50 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-l[73743]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7fa8196841ca] 2024-05-27T07:29:50.365 INFO:journalctl@ceph.mds.l.smithi173.stdout:May 27 07:29:50 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-l[73743]: 16: clone() 2024-05-27T07:29:50.365 INFO:journalctl@ceph.mds.l.smithi173.stdout:May 27 07:29:50 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-l[73743]: 2024-05-27T07:29:50.366 INFO:journalctl@ceph.mds.l.smithi173.stdout:May 27 07:29:50 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-l[73743]: 0> 2024-05-27T07:29:50.082+0000 7fa8082cc700 -1 *** Caught signal (Aborted) ** 2024-05-27T07:29:50.366 INFO:journalctl@ceph.mds.l.smithi173.stdout:May 27 07:29:50 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-l[73743]: in thread 7fa8082cc700 thread_name:md_log_replay 2024-05-27T07:29:50.366 INFO:journalctl@ceph.mds.l.smithi173.stdout:May 27 07:29:50 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-l[73743]: 2024-05-27T07:29:50.366 INFO:journalctl@ceph.mds.l.smithi173.stdout:May 27 07:29:50 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-l[73743]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:29:50.366 INFO:journalctl@ceph.mds.l.smithi173.stdout:May 27 07:29:50 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-l[73743]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7fa81968ed20] 2024-05-27T07:29:50.367 INFO:journalctl@ceph.mds.l.smithi173.stdout:May 27 07:29:50 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-l[73743]: 2: gsignal() 2024-05-27T07:29:50.367 INFO:journalctl@ceph.mds.l.smithi173.stdout:May 27 07:29:50 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-l[73743]: 3: abort() 2024-05-27T07:29:50.367 INFO:journalctl@ceph.mds.l.smithi173.stdout:May 27 07:29:50 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-l[73743]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x26e) [0x7fa81ab11ced] 2024-05-27T07:29:50.367 INFO:journalctl@ceph.mds.l.smithi173.stdout:May 27 07:29:50 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-l[73743]: 5: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7fa81ab11dd1] 2024-05-27T07:29:50.367 INFO:journalctl@ceph.mds.l.smithi173.stdout:May 27 07:29:50 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-l[73743]: 6: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x5571fe90223d] 2024-05-27T07:29:50.367 INFO:journalctl@ceph.mds.l.smithi173.stdout:May 27 07:29:50 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-l[73743]: 7: (SnapRealm::build_snap_set() const+0x323) [0x5571fe8ed971] 2024-05-27T07:29:50.367 INFO:journalctl@ceph.mds.l.smithi173.stdout:May 27 07:29:50 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-l[73743]: 8: (SnapRealm::check_cache() const+0x13e) [0x5571fe8ebfa0] 2024-05-27T07:29:50.367 INFO:journalctl@ceph.mds.l.smithi173.stdout:May 27 07:29:50 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-l[73743]: 9: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x5571fe8ec651] 2024-05-27T07:29:50.367 INFO:journalctl@ceph.mds.l.smithi173.stdout:May 27 07:29:50 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-l[73743]: 10: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x5571fe881347] 2024-05-27T07:29:50.368 INFO:journalctl@ceph.mds.l.smithi173.stdout:May 27 07:29:50 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-l[73743]: 11: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x5571fe57d87e] 2024-05-27T07:29:50.368 INFO:journalctl@ceph.mds.l.smithi173.stdout:May 27 07:29:50 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-l[73743]: 12: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x2702) [0x5571fe582e94] 2024-05-27T07:29:50.368 INFO:journalctl@ceph.mds.l.smithi173.stdout:May 27 07:29:50 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-l[73743]: 13: (EUpdate::replay(MDSRank*)+0x63) [0x5571fe58d969] 2024-05-27T07:29:50.368 INFO:journalctl@ceph.mds.l.smithi173.stdout:May 27 07:29:50 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-l[73743]: 14: (MDLog::_replay_thread()+0x1f38) [0x5571fe92648c] 2024-05-27T07:29:50.368 INFO:journalctl@ceph.mds.l.smithi173.stdout:May 27 07:29:50 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-l[73743]: 15: (MDLog::ReplayThread::entry()+0x11) [0x5571fe92f9cb] 2024-05-27T07:29:50.368 INFO:journalctl@ceph.mds.l.smithi173.stdout:May 27 07:29:50 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-l[73743]: 16: (Thread::entry_wrapper()+0x43) [0x7fa81aae072d] 2024-05-27T07:29:50.368 INFO:journalctl@ceph.mds.l.smithi173.stdout:May 27 07:29:50 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-l[73743]: 17: (Thread::_entry_func(void*)+0xd) [0x7fa81aae0749] 2024-05-27T07:29:50.368 INFO:journalctl@ceph.mds.l.smithi173.stdout:May 27 07:29:50 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-l[73743]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7fa8196841ca] 2024-05-27T07:29:50.369 INFO:journalctl@ceph.mds.l.smithi173.stdout:May 27 07:29:50 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-l[73743]: 19: clone() 2024-05-27T07:29:50.369 INFO:journalctl@ceph.mds.l.smithi173.stdout:May 27 07:29:50 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-l[73743]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-05-27T07:29:50.369 INFO:journalctl@ceph.mds.l.smithi173.stdout:May 27 07:29:50 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-l[73743]: 2024-05-27T07:29:50.369 INFO:journalctl@ceph.mds.l.smithi173.stdout:May 27 07:29:50 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-l[73743]: -1> 2024-05-27T07:29:50.079+0000 7fa8082cc700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: In function 'std::set SnapClient::filter(const std::set&) const' thread 7fa8082cc700 time 2024-05-27T07:29:50.078215+0000 2024-05-27T07:29:50.369 INFO:journalctl@ceph.mds.l.smithi173.stdout:May 27 07:29:50 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-l[73743]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: 197: FAILED ceph_assert(cached_version > 0) 2024-05-27T07:29:50.369 INFO:journalctl@ceph.mds.l.smithi173.stdout:May 27 07:29:50 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-l[73743]: 2024-05-27T07:29:50.369 INFO:journalctl@ceph.mds.l.smithi173.stdout:May 27 07:29:50 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-l[73743]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:29:50.369 INFO:journalctl@ceph.mds.l.smithi173.stdout:May 27 07:29:50 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-l[73743]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x127) [0x7fa81ab11ba6] 2024-05-27T07:29:50.369 INFO:journalctl@ceph.mds.l.smithi173.stdout:May 27 07:29:50 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-l[73743]: 2: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7fa81ab11dd1] 2024-05-27T07:29:50.369 INFO:journalctl@ceph.mds.l.smithi173.stdout:May 27 07:29:50 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-l[73743]: 3: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x5571fe90223d] 2024-05-27T07:29:50.369 INFO:journalctl@ceph.mds.l.smithi173.stdout:May 27 07:29:50 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-l[73743]: 4: (SnapRealm::build_snap_set() const+0x323) [0x5571fe8ed971] 2024-05-27T07:29:50.369 INFO:journalctl@ceph.mds.l.smithi173.stdout:May 27 07:29:50 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-l[73743]: 5: (SnapRealm::check_cache() const+0x13e) [0x5571fe8ebfa0] 2024-05-27T07:29:50.369 INFO:journalctl@ceph.mds.l.smithi173.stdout:May 27 07:29:50 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-l[73743]: 6: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x5571fe8ec651] 2024-05-27T07:29:50.370 INFO:journalctl@ceph.mds.l.smithi173.stdout:May 27 07:29:50 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-l[73743]: 7: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x5571fe881347] 2024-05-27T07:29:50.370 INFO:journalctl@ceph.mds.l.smithi173.stdout:May 27 07:29:50 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-l[73743]: 8: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x5571fe57d87e] 2024-05-27T07:29:50.370 INFO:journalctl@ceph.mds.l.smithi173.stdout:May 27 07:29:50 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-l[73743]: 9: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x2702) [0x5571fe582e94] 2024-05-27T07:29:50.370 INFO:journalctl@ceph.mds.l.smithi173.stdout:May 27 07:29:50 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-l[73743]: 10: (EUpdate::replay(MDSRank*)+0x63) [0x5571fe58d969] 2024-05-27T07:29:50.370 INFO:journalctl@ceph.mds.l.smithi173.stdout:May 27 07:29:50 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-l[73743]: 11: (MDLog::_replay_thread()+0x1f38) [0x5571fe92648c] 2024-05-27T07:29:50.370 INFO:journalctl@ceph.mds.l.smithi173.stdout:May 27 07:29:50 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-l[73743]: 12: (MDLog::ReplayThread::entry()+0x11) [0x5571fe92f9cb] 2024-05-27T07:29:50.370 INFO:journalctl@ceph.mds.l.smithi173.stdout:May 27 07:29:50 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-l[73743]: 13: (Thread::entry_wrapper()+0x43) [0x7fa81aae072d] 2024-05-27T07:29:50.370 INFO:journalctl@ceph.mds.l.smithi173.stdout:May 27 07:29:50 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-l[73743]: 14: (Thread::_entry_func(void*)+0xd) [0x7fa81aae0749] 2024-05-27T07:29:50.370 INFO:journalctl@ceph.mds.l.smithi173.stdout:May 27 07:29:50 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-l[73743]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7fa8196841ca] 2024-05-27T07:29:50.370 INFO:journalctl@ceph.mds.l.smithi173.stdout:May 27 07:29:50 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-l[73743]: 16: clone() 2024-05-27T07:29:50.370 INFO:journalctl@ceph.mds.l.smithi173.stdout:May 27 07:29:50 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-l[73743]: 2024-05-27T07:29:50.371 INFO:journalctl@ceph.mds.l.smithi173.stdout:May 27 07:29:50 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-l[73743]: 0> 2024-05-27T07:29:50.082+0000 7fa8082cc700 -1 *** Caught signal (Aborted) ** 2024-05-27T07:29:50.371 INFO:journalctl@ceph.mds.l.smithi173.stdout:May 27 07:29:50 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-l[73743]: in thread 7fa8082cc700 thread_name:md_log_replay 2024-05-27T07:29:50.371 INFO:journalctl@ceph.mds.l.smithi173.stdout:May 27 07:29:50 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-l[73743]: 2024-05-27T07:29:50.371 INFO:journalctl@ceph.mds.l.smithi173.stdout:May 27 07:29:50 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-l[73743]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:29:50.371 INFO:journalctl@ceph.mds.l.smithi173.stdout:May 27 07:29:50 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-l[73743]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7fa81968ed20] 2024-05-27T07:29:50.371 INFO:journalctl@ceph.mds.l.smithi173.stdout:May 27 07:29:50 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-l[73743]: 2: gsignal() 2024-05-27T07:29:50.371 INFO:journalctl@ceph.mds.l.smithi173.stdout:May 27 07:29:50 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-l[73743]: 3: abort() 2024-05-27T07:29:50.371 INFO:journalctl@ceph.mds.l.smithi173.stdout:May 27 07:29:50 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-l[73743]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x26e) [0x7fa81ab11ced] 2024-05-27T07:29:50.371 INFO:journalctl@ceph.mds.l.smithi173.stdout:May 27 07:29:50 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-l[73743]: 5: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7fa81ab11dd1] 2024-05-27T07:29:50.371 INFO:journalctl@ceph.mds.l.smithi173.stdout:May 27 07:29:50 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-l[73743]: 6: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x5571fe90223d] 2024-05-27T07:29:50.371 INFO:journalctl@ceph.mds.l.smithi173.stdout:May 27 07:29:50 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-l[73743]: 7: (SnapRealm::build_snap_set() const+0x323) [0x5571fe8ed971] 2024-05-27T07:29:50.371 INFO:journalctl@ceph.mds.l.smithi173.stdout:May 27 07:29:50 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-l[73743]: 8: (SnapRealm::check_cache() const+0x13e) [0x5571fe8ebfa0] 2024-05-27T07:29:50.371 INFO:journalctl@ceph.mds.l.smithi173.stdout:May 27 07:29:50 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-l[73743]: 9: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x5571fe8ec651] 2024-05-27T07:29:50.371 INFO:journalctl@ceph.mds.l.smithi173.stdout:May 27 07:29:50 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-l[73743]: 10: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x5571fe881347] 2024-05-27T07:29:50.372 INFO:journalctl@ceph.mds.l.smithi173.stdout:May 27 07:29:50 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-l[73743]: 11: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x5571fe57d87e] 2024-05-27T07:29:50.372 INFO:journalctl@ceph.mds.l.smithi173.stdout:May 27 07:29:50 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-l[73743]: 12: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x2702) [0x5571fe582e94] 2024-05-27T07:29:50.372 INFO:journalctl@ceph.mds.l.smithi173.stdout:May 27 07:29:50 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-l[73743]: 13: (EUpdate::replay(MDSRank*)+0x63) [0x5571fe58d969] 2024-05-27T07:29:50.372 INFO:journalctl@ceph.mds.l.smithi173.stdout:May 27 07:29:50 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-l[73743]: 14: (MDLog::_replay_thread()+0x1f38) [0x5571fe92648c] 2024-05-27T07:29:50.372 INFO:journalctl@ceph.mds.l.smithi173.stdout:May 27 07:29:50 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-l[73743]: 15: (MDLog::ReplayThread::entry()+0x11) [0x5571fe92f9cb] 2024-05-27T07:29:50.372 INFO:journalctl@ceph.mds.l.smithi173.stdout:May 27 07:29:50 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-l[73743]: 16: (Thread::entry_wrapper()+0x43) [0x7fa81aae072d] 2024-05-27T07:29:50.372 INFO:journalctl@ceph.mds.l.smithi173.stdout:May 27 07:29:50 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-l[73743]: 17: (Thread::_entry_func(void*)+0xd) [0x7fa81aae0749] 2024-05-27T07:29:50.372 INFO:journalctl@ceph.mds.l.smithi173.stdout:May 27 07:29:50 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-l[73743]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7fa8196841ca] 2024-05-27T07:29:50.372 INFO:journalctl@ceph.mds.l.smithi173.stdout:May 27 07:29:50 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-l[73743]: 19: clone() 2024-05-27T07:29:50.372 INFO:journalctl@ceph.mds.l.smithi173.stdout:May 27 07:29:50 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-l[73743]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-05-27T07:29:50.372 INFO:journalctl@ceph.mds.l.smithi173.stdout:May 27 07:29:50 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-l[73743]: 2024-05-27T07:29:50.373 INFO:journalctl@ceph.mds.l.smithi173.stdout:May 27 07:29:50 smithi173 podman[73778]: 2024-05-27 07:29:50.150926488 +0000 UTC m=+0.022840624 container died d70a52e699b5e512e537a44cc9d1f077b52180fbafd7465fb3950b042a8eb975 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-l, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, org.label-schema.build-date=20240521, GIT_BRANCH=HEAD, ceph=True, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, maintainer=Guillaume Abrioux ) 2024-05-27T07:29:50.441 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:29:50 smithi105 ceph-mon[35977]: pgmap v134: 129 pgs: 129 active+clean; 267 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.9 MiB/s rd, 5.2 MiB/s wr, 560 op/s 2024-05-27T07:29:50.441 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:29:50 smithi105 ceph-mon[35977]: osdmap e107: 12 total, 12 up, 12 in 2024-05-27T07:29:50.441 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:29:50 smithi105 ceph-mon[35977]: mds.? [v2:172.21.15.52:6836/2351486841,v1:172.21.15.52:6837/2351486841] up:boot 2024-05-27T07:29:50.441 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:29:50 smithi105 ceph-mon[35977]: daemon mds.g restarted 2024-05-27T07:29:50.441 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:29:50 smithi105 ceph-mon[35977]: fsmap cephfs:5 4 up:standby-replay 2 up:standby 5 up:active 2024-05-27T07:29:50.441 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:29:50 smithi105 ceph-mon[35977]: from='mgr.15333 172.21.15.52:0/416389082' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "d"}]: dispatch 2024-05-27T07:29:50.441 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:29:50 smithi105 ceph-mon[35977]: osdmap e108: 12 total, 12 up, 12 in 2024-05-27T07:29:50.441 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:29:50 smithi105 ceph-mon[35977]: mds.? [v2:172.21.15.52:6838/3051157409,v1:172.21.15.52:6839/3051157409] up:boot 2024-05-27T07:29:50.441 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:29:50 smithi105 ceph-mon[35977]: MDS daemon mds.j is removed because it is dead or otherwise unavailable. 2024-05-27T07:29:50.442 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:29:50 smithi105 ceph-mon[35977]: fsmap cephfs:5 5 up:standby-replay 1 up:standby 5 up:active 2024-05-27T07:29:50.442 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:29:50 smithi105 ceph-mon[35977]: from='mgr.15333 172.21.15.52:0/416389082' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2024-05-27T07:29:50.442 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:29:50 smithi105 ceph-mon[35977]: fsmap cephfs:5 5 up:standby-replay 5 up:active 2024-05-27T07:29:50.442 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:29:50 smithi105 ceph-mon[35977]: daemon mds.l restarted 2024-05-27T07:29:50.442 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:29:50 smithi105 ceph-mon[35977]: osdmap e109: 12 total, 12 up, 12 in 2024-05-27T07:29:50.442 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:29:50 smithi105 ceph-mon[35977]: mds.? [v2:172.21.15.173:6832/1177755063,v1:172.21.15.173:6833/1177755063] up:boot 2024-05-27T07:29:50.442 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:29:50 smithi105 ceph-mon[35977]: fsmap cephfs:5 5 up:standby-replay 5 up:active 2024-05-27T07:29:50.442 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:29:50 smithi105 ceph-mon[35977]: from='mgr.15333 172.21.15.52:0/416389082' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "l"}]: dispatch 2024-05-27T07:29:50.480 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:29:50 smithi052 ceph-mon[31955]: pgmap v134: 129 pgs: 129 active+clean; 267 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.9 MiB/s rd, 5.2 MiB/s wr, 560 op/s 2024-05-27T07:29:50.481 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:29:50 smithi052 ceph-mon[31955]: osdmap e107: 12 total, 12 up, 12 in 2024-05-27T07:29:50.481 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:29:50 smithi052 ceph-mon[31955]: mds.? [v2:172.21.15.52:6836/2351486841,v1:172.21.15.52:6837/2351486841] up:boot 2024-05-27T07:29:50.481 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:29:50 smithi052 ceph-mon[31955]: daemon mds.g restarted 2024-05-27T07:29:50.481 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:29:50 smithi052 ceph-mon[31955]: fsmap cephfs:5 4 up:standby-replay 2 up:standby 5 up:active 2024-05-27T07:29:50.481 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:29:50 smithi052 ceph-mon[31955]: from='mgr.15333 172.21.15.52:0/416389082' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "d"}]: dispatch 2024-05-27T07:29:50.481 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:29:50 smithi052 ceph-mon[31955]: osdmap e108: 12 total, 12 up, 12 in 2024-05-27T07:29:50.481 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:29:50 smithi052 ceph-mon[31955]: mds.? [v2:172.21.15.52:6838/3051157409,v1:172.21.15.52:6839/3051157409] up:boot 2024-05-27T07:29:50.481 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:29:50 smithi052 ceph-mon[31955]: MDS daemon mds.j is removed because it is dead or otherwise unavailable. 2024-05-27T07:29:50.481 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:29:50 smithi052 ceph-mon[31955]: fsmap cephfs:5 5 up:standby-replay 1 up:standby 5 up:active 2024-05-27T07:29:50.481 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:29:50 smithi052 ceph-mon[31955]: from='mgr.15333 172.21.15.52:0/416389082' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2024-05-27T07:29:50.481 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:29:50 smithi052 ceph-mon[31955]: fsmap cephfs:5 5 up:standby-replay 5 up:active 2024-05-27T07:29:50.481 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:29:50 smithi052 ceph-mon[31955]: daemon mds.l restarted 2024-05-27T07:29:50.481 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:29:50 smithi052 ceph-mon[31955]: osdmap e109: 12 total, 12 up, 12 in 2024-05-27T07:29:50.482 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:29:50 smithi052 ceph-mon[31955]: mds.? [v2:172.21.15.173:6832/1177755063,v1:172.21.15.173:6833/1177755063] up:boot 2024-05-27T07:29:50.482 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:29:50 smithi052 ceph-mon[31955]: fsmap cephfs:5 5 up:standby-replay 5 up:active 2024-05-27T07:29:50.482 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:29:50 smithi052 ceph-mon[31955]: from='mgr.15333 172.21.15.52:0/416389082' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "l"}]: dispatch 2024-05-27T07:29:50.482 INFO:journalctl@ceph.mds.a.smithi052.stdout:May 27 07:29:50 smithi052 podman[106444]: 2024-05-27 07:29:50.325329623 +0000 UTC m=+0.241605496 container died c5447791e4a6f6b5a3928fe2dcc2df1539dc407ed035bafd6e3e0e3e6691c4ef (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-a, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, GIT_CLEAN=True, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240521, maintainer=Guillaume Abrioux ) 2024-05-27T07:29:50.482 INFO:journalctl@ceph.mds.j.smithi052.stdout:May 27 07:29:50 smithi052 podman[106408]: 2024-05-27 07:29:50.152739995 +0000 UTC m=+0.738498020 container init d03f6d3925720a9a1b1ac9712defc07799179d80b7913701e4ef645e7a0b5cd1 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-j, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, ceph=True, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240521, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image) 2024-05-27T07:29:50.482 INFO:journalctl@ceph.mds.j.smithi052.stdout:May 27 07:29:50 smithi052 podman[106408]: 2024-05-27 07:29:50.163967416 +0000 UTC m=+0.749725433 container start d03f6d3925720a9a1b1ac9712defc07799179d80b7913701e4ef645e7a0b5cd1 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-j, maintainer=Guillaume Abrioux , GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, ceph=True, org.label-schema.build-date=20240521, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD) 2024-05-27T07:29:50.483 INFO:journalctl@ceph.mds.j.smithi052.stdout:May 27 07:29:50 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-j[106435]: starting mds.j at 2024-05-27T07:29:50.483 INFO:journalctl@ceph.mds.j.smithi052.stdout:May 27 07:29:50 smithi052 bash[106408]: d03f6d3925720a9a1b1ac9712defc07799179d80b7913701e4ef645e7a0b5cd1 2024-05-27T07:29:50.483 INFO:journalctl@ceph.mds.j.smithi052.stdout:May 27 07:29:50 smithi052 systemd[1]: Started Ceph mds.j for 73df2636-1bf8-11ef-bc9b-c7b262605968. 2024-05-27T07:29:50.949 INFO:journalctl@ceph.mds.k.smithi105.stdout:May 27 07:29:50 smithi105 systemd[1]: ceph-73df2636-1bf8-11ef-bc9b-c7b262605968@mds.k.service: Scheduled restart job, restart counter is at 3. 2024-05-27T07:29:50.949 INFO:journalctl@ceph.mds.k.smithi105.stdout:May 27 07:29:50 smithi105 systemd[1]: Stopped Ceph mds.k for 73df2636-1bf8-11ef-bc9b-c7b262605968. 2024-05-27T07:29:50.949 INFO:journalctl@ceph.mds.k.smithi105.stdout:May 27 07:29:50 smithi105 systemd[1]: Starting Ceph mds.k for 73df2636-1bf8-11ef-bc9b-c7b262605968... 2024-05-27T07:29:51.206 INFO:journalctl@ceph.mds.k.smithi105.stdout:May 27 07:29:50 smithi105 podman[69974]: 2024-05-27 07:29:50.863305822 +0000 UTC m=+0.025849156 image pull e36ea4fc8a5807e8d2701370e350a0de3856538797f7fba64cfce7e344bd6799 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6 2024-05-27T07:29:51.206 INFO:journalctl@ceph.mds.k.smithi105.stdout:May 27 07:29:50 smithi105 podman[69974]: 2024-05-27 07:29:50.994701511 +0000 UTC m=+0.157244842 container create 65fd1707409f955d437095b2ebc4b654df22761089d5fcee3ce3884251d520be (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-k, ceph=True, CEPH_POINT_RELEASE=, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, io.buildah.version=1.33.5, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240521, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-05-27T07:29:51.228 INFO:journalctl@ceph.mds.a.smithi052.stdout:May 27 07:29:51 smithi052 podman[106444]: 2024-05-27 07:29:51.237925568 +0000 UTC m=+1.154201442 container remove c5447791e4a6f6b5a3928fe2dcc2df1539dc407ed035bafd6e3e0e3e6691c4ef (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-a, GIT_CLEAN=True, org.label-schema.build-date=20240521, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, maintainer=Guillaume Abrioux ) 2024-05-27T07:29:51.229 INFO:journalctl@ceph.mds.d.smithi052.stdout:May 27 07:29:50 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-d[106255]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: In function 'std::set SnapClient::filter(const std::set&) const' thread 7f058bb25700 time 2024-05-27T07:29:50.872338+0000 2024-05-27T07:29:51.230 INFO:journalctl@ceph.mds.d.smithi052.stdout:May 27 07:29:50 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-d[106255]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: 197: FAILED ceph_assert(cached_version > 0) 2024-05-27T07:29:51.230 INFO:journalctl@ceph.mds.d.smithi052.stdout:May 27 07:29:50 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-d[106255]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:29:51.230 INFO:journalctl@ceph.mds.d.smithi052.stdout:May 27 07:29:50 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-d[106255]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x127) [0x7f059e36aba6] 2024-05-27T07:29:51.230 INFO:journalctl@ceph.mds.d.smithi052.stdout:May 27 07:29:50 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-d[106255]: 2: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7f059e36add1] 2024-05-27T07:29:51.230 INFO:journalctl@ceph.mds.d.smithi052.stdout:May 27 07:29:50 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-d[106255]: 3: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x5620dd72123d] 2024-05-27T07:29:51.230 INFO:journalctl@ceph.mds.d.smithi052.stdout:May 27 07:29:50 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-d[106255]: 4: (SnapRealm::build_snap_set() const+0x323) [0x5620dd70c971] 2024-05-27T07:29:51.230 INFO:journalctl@ceph.mds.d.smithi052.stdout:May 27 07:29:50 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-d[106255]: 5: (SnapRealm::check_cache() const+0x13e) [0x5620dd70afa0] 2024-05-27T07:29:51.230 INFO:journalctl@ceph.mds.d.smithi052.stdout:May 27 07:29:50 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-d[106255]: 6: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x5620dd70b651] 2024-05-27T07:29:51.230 INFO:journalctl@ceph.mds.d.smithi052.stdout:May 27 07:29:50 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-d[106255]: 7: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x5620dd6a0347] 2024-05-27T07:29:51.230 INFO:journalctl@ceph.mds.d.smithi052.stdout:May 27 07:29:50 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-d[106255]: 8: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x5620dd39c87e] 2024-05-27T07:29:51.230 INFO:journalctl@ceph.mds.d.smithi052.stdout:May 27 07:29:50 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-d[106255]: 9: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x2702) [0x5620dd3a1e94] 2024-05-27T07:29:51.230 INFO:journalctl@ceph.mds.d.smithi052.stdout:May 27 07:29:50 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-d[106255]: 10: (EUpdate::replay(MDSRank*)+0x63) [0x5620dd3ac969] 2024-05-27T07:29:51.230 INFO:journalctl@ceph.mds.d.smithi052.stdout:May 27 07:29:50 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-d[106255]: 11: (MDLog::_replay_thread()+0x1f38) [0x5620dd74548c] 2024-05-27T07:29:51.231 INFO:journalctl@ceph.mds.d.smithi052.stdout:May 27 07:29:50 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-d[106255]: 12: (MDLog::ReplayThread::entry()+0x11) [0x5620dd74e9cb] 2024-05-27T07:29:51.231 INFO:journalctl@ceph.mds.d.smithi052.stdout:May 27 07:29:50 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-d[106255]: 13: (Thread::entry_wrapper()+0x43) [0x7f059e33972d] 2024-05-27T07:29:51.231 INFO:journalctl@ceph.mds.d.smithi052.stdout:May 27 07:29:50 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-d[106255]: 14: (Thread::_entry_func(void*)+0xd) [0x7f059e339749] 2024-05-27T07:29:51.231 INFO:journalctl@ceph.mds.d.smithi052.stdout:May 27 07:29:50 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-d[106255]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7f059cedd1ca] 2024-05-27T07:29:51.231 INFO:journalctl@ceph.mds.d.smithi052.stdout:May 27 07:29:50 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-d[106255]: 16: clone() 2024-05-27T07:29:51.231 INFO:journalctl@ceph.mds.d.smithi052.stdout:May 27 07:29:50 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-d[106255]: *** Caught signal (Aborted) ** 2024-05-27T07:29:51.231 INFO:journalctl@ceph.mds.d.smithi052.stdout:May 27 07:29:50 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-d[106255]: in thread 7f058bb25700 thread_name:md_log_replay 2024-05-27T07:29:51.233 INFO:journalctl@ceph.mds.d.smithi052.stdout:May 27 07:29:50 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-d[106255]: 2024-05-27T07:29:50.876+0000 7f058bb25700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: In function 'std::set SnapClient::filter(const std::set&) const' thread 7f058bb25700 time 2024-05-27T07:29:50.872338+0000 2024-05-27T07:29:51.233 INFO:journalctl@ceph.mds.d.smithi052.stdout:May 27 07:29:50 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-d[106255]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: 197: FAILED ceph_assert(cached_version > 0) 2024-05-27T07:29:51.233 INFO:journalctl@ceph.mds.d.smithi052.stdout:May 27 07:29:50 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-d[106255]: 2024-05-27T07:29:51.233 INFO:journalctl@ceph.mds.d.smithi052.stdout:May 27 07:29:50 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-d[106255]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:29:51.233 INFO:journalctl@ceph.mds.d.smithi052.stdout:May 27 07:29:50 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-d[106255]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x127) [0x7f059e36aba6] 2024-05-27T07:29:51.233 INFO:journalctl@ceph.mds.d.smithi052.stdout:May 27 07:29:50 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-d[106255]: 2: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7f059e36add1] 2024-05-27T07:29:51.233 INFO:journalctl@ceph.mds.d.smithi052.stdout:May 27 07:29:50 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-d[106255]: 3: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x5620dd72123d] 2024-05-27T07:29:51.233 INFO:journalctl@ceph.mds.d.smithi052.stdout:May 27 07:29:50 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-d[106255]: 4: (SnapRealm::build_snap_set() const+0x323) [0x5620dd70c971] 2024-05-27T07:29:51.233 INFO:journalctl@ceph.mds.d.smithi052.stdout:May 27 07:29:50 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-d[106255]: 5: (SnapRealm::check_cache() const+0x13e) [0x5620dd70afa0] 2024-05-27T07:29:51.233 INFO:journalctl@ceph.mds.d.smithi052.stdout:May 27 07:29:50 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-d[106255]: 6: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x5620dd70b651] 2024-05-27T07:29:51.233 INFO:journalctl@ceph.mds.d.smithi052.stdout:May 27 07:29:50 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-d[106255]: 7: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x5620dd6a0347] 2024-05-27T07:29:51.233 INFO:journalctl@ceph.mds.d.smithi052.stdout:May 27 07:29:50 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-d[106255]: 8: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x5620dd39c87e] 2024-05-27T07:29:51.234 INFO:journalctl@ceph.mds.d.smithi052.stdout:May 27 07:29:50 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-d[106255]: 9: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x2702) [0x5620dd3a1e94] 2024-05-27T07:29:51.234 INFO:journalctl@ceph.mds.d.smithi052.stdout:May 27 07:29:50 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-d[106255]: 10: (EUpdate::replay(MDSRank*)+0x63) [0x5620dd3ac969] 2024-05-27T07:29:51.234 INFO:journalctl@ceph.mds.d.smithi052.stdout:May 27 07:29:50 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-d[106255]: 11: (MDLog::_replay_thread()+0x1f38) [0x5620dd74548c] 2024-05-27T07:29:51.234 INFO:journalctl@ceph.mds.d.smithi052.stdout:May 27 07:29:50 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-d[106255]: 12: (MDLog::ReplayThread::entry()+0x11) [0x5620dd74e9cb] 2024-05-27T07:29:51.234 INFO:journalctl@ceph.mds.d.smithi052.stdout:May 27 07:29:50 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-d[106255]: 13: (Thread::entry_wrapper()+0x43) [0x7f059e33972d] 2024-05-27T07:29:51.234 INFO:journalctl@ceph.mds.d.smithi052.stdout:May 27 07:29:50 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-d[106255]: 14: (Thread::_entry_func(void*)+0xd) [0x7f059e339749] 2024-05-27T07:29:51.234 INFO:journalctl@ceph.mds.d.smithi052.stdout:May 27 07:29:50 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-d[106255]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7f059cedd1ca] 2024-05-27T07:29:51.234 INFO:journalctl@ceph.mds.d.smithi052.stdout:May 27 07:29:50 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-d[106255]: 16: clone() 2024-05-27T07:29:51.234 INFO:journalctl@ceph.mds.d.smithi052.stdout:May 27 07:29:50 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-d[106255]: 2024-05-27T07:29:51.234 INFO:journalctl@ceph.mds.d.smithi052.stdout:May 27 07:29:50 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-d[106255]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:29:51.234 INFO:journalctl@ceph.mds.d.smithi052.stdout:May 27 07:29:50 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-d[106255]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f059cee7d20] 2024-05-27T07:29:51.234 INFO:journalctl@ceph.mds.d.smithi052.stdout:May 27 07:29:50 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-d[106255]: 2: gsignal() 2024-05-27T07:29:51.234 INFO:journalctl@ceph.mds.d.smithi052.stdout:May 27 07:29:50 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-d[106255]: 3: abort() 2024-05-27T07:29:51.235 INFO:journalctl@ceph.mds.d.smithi052.stdout:May 27 07:29:50 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-d[106255]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x26e) [0x7f059e36aced] 2024-05-27T07:29:51.236 INFO:journalctl@ceph.mds.d.smithi052.stdout:May 27 07:29:50 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-d[106255]: 5: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7f059e36add1] 2024-05-27T07:29:51.236 INFO:journalctl@ceph.mds.d.smithi052.stdout:May 27 07:29:50 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-d[106255]: 6: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x5620dd72123d] 2024-05-27T07:29:51.236 INFO:journalctl@ceph.mds.d.smithi052.stdout:May 27 07:29:50 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-d[106255]: 7: (SnapRealm::build_snap_set() const+0x323) [0x5620dd70c971] 2024-05-27T07:29:51.236 INFO:journalctl@ceph.mds.d.smithi052.stdout:May 27 07:29:50 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-d[106255]: 8: (SnapRealm::check_cache() const+0x13e) [0x5620dd70afa0] 2024-05-27T07:29:51.236 INFO:journalctl@ceph.mds.d.smithi052.stdout:May 27 07:29:50 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-d[106255]: 9: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x5620dd70b651] 2024-05-27T07:29:51.236 INFO:journalctl@ceph.mds.d.smithi052.stdout:May 27 07:29:50 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-d[106255]: 10: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x5620dd6a0347] 2024-05-27T07:29:51.237 INFO:journalctl@ceph.mds.d.smithi052.stdout:May 27 07:29:50 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-d[106255]: 11: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x5620dd39c87e] 2024-05-27T07:29:51.237 INFO:journalctl@ceph.mds.d.smithi052.stdout:May 27 07:29:50 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-d[106255]: 12: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x2702) [0x5620dd3a1e94] 2024-05-27T07:29:51.237 INFO:journalctl@ceph.mds.d.smithi052.stdout:May 27 07:29:50 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-d[106255]: 13: (EUpdate::replay(MDSRank*)+0x63) [0x5620dd3ac969] 2024-05-27T07:29:51.237 INFO:journalctl@ceph.mds.d.smithi052.stdout:May 27 07:29:50 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-d[106255]: 14: (MDLog::_replay_thread()+0x1f38) [0x5620dd74548c] 2024-05-27T07:29:51.237 INFO:journalctl@ceph.mds.d.smithi052.stdout:May 27 07:29:50 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-d[106255]: 15: (MDLog::ReplayThread::entry()+0x11) [0x5620dd74e9cb] 2024-05-27T07:29:51.237 INFO:journalctl@ceph.mds.d.smithi052.stdout:May 27 07:29:50 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-d[106255]: 16: (Thread::entry_wrapper()+0x43) [0x7f059e33972d] 2024-05-27T07:29:51.237 INFO:journalctl@ceph.mds.d.smithi052.stdout:May 27 07:29:50 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-d[106255]: 17: (Thread::_entry_func(void*)+0xd) [0x7f059e339749] 2024-05-27T07:29:51.237 INFO:journalctl@ceph.mds.d.smithi052.stdout:May 27 07:29:50 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-d[106255]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7f059cedd1ca] 2024-05-27T07:29:51.237 INFO:journalctl@ceph.mds.d.smithi052.stdout:May 27 07:29:50 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-d[106255]: 19: clone() 2024-05-27T07:29:51.237 INFO:journalctl@ceph.mds.d.smithi052.stdout:May 27 07:29:50 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-d[106255]: 2024-05-27T07:29:50.883+0000 7f058bb25700 -1 *** Caught signal (Aborted) ** 2024-05-27T07:29:51.237 INFO:journalctl@ceph.mds.d.smithi052.stdout:May 27 07:29:50 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-d[106255]: in thread 7f058bb25700 thread_name:md_log_replay 2024-05-27T07:29:51.237 INFO:journalctl@ceph.mds.d.smithi052.stdout:May 27 07:29:50 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-d[106255]: 2024-05-27T07:29:51.237 INFO:journalctl@ceph.mds.d.smithi052.stdout:May 27 07:29:50 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-d[106255]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:29:51.238 INFO:journalctl@ceph.mds.d.smithi052.stdout:May 27 07:29:50 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-d[106255]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f059cee7d20] 2024-05-27T07:29:51.238 INFO:journalctl@ceph.mds.d.smithi052.stdout:May 27 07:29:50 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-d[106255]: 2: gsignal() 2024-05-27T07:29:51.238 INFO:journalctl@ceph.mds.d.smithi052.stdout:May 27 07:29:50 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-d[106255]: 3: abort() 2024-05-27T07:29:51.238 INFO:journalctl@ceph.mds.d.smithi052.stdout:May 27 07:29:50 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-d[106255]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x26e) [0x7f059e36aced] 2024-05-27T07:29:51.238 INFO:journalctl@ceph.mds.d.smithi052.stdout:May 27 07:29:50 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-d[106255]: 5: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7f059e36add1] 2024-05-27T07:29:51.238 INFO:journalctl@ceph.mds.d.smithi052.stdout:May 27 07:29:50 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-d[106255]: 6: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x5620dd72123d] 2024-05-27T07:29:51.238 INFO:journalctl@ceph.mds.d.smithi052.stdout:May 27 07:29:50 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-d[106255]: 7: (SnapRealm::build_snap_set() const+0x323) [0x5620dd70c971] 2024-05-27T07:29:51.238 INFO:journalctl@ceph.mds.d.smithi052.stdout:May 27 07:29:50 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-d[106255]: 8: (SnapRealm::check_cache() const+0x13e) [0x5620dd70afa0] 2024-05-27T07:29:51.238 INFO:journalctl@ceph.mds.d.smithi052.stdout:May 27 07:29:50 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-d[106255]: 9: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x5620dd70b651] 2024-05-27T07:29:51.238 INFO:journalctl@ceph.mds.d.smithi052.stdout:May 27 07:29:50 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-d[106255]: 10: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x5620dd6a0347] 2024-05-27T07:29:51.238 INFO:journalctl@ceph.mds.d.smithi052.stdout:May 27 07:29:50 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-d[106255]: 11: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x5620dd39c87e] 2024-05-27T07:29:51.238 INFO:journalctl@ceph.mds.d.smithi052.stdout:May 27 07:29:50 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-d[106255]: 12: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x2702) [0x5620dd3a1e94] 2024-05-27T07:29:51.238 INFO:journalctl@ceph.mds.d.smithi052.stdout:May 27 07:29:50 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-d[106255]: 13: (EUpdate::replay(MDSRank*)+0x63) [0x5620dd3ac969] 2024-05-27T07:29:51.239 INFO:journalctl@ceph.mds.d.smithi052.stdout:May 27 07:29:50 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-d[106255]: 14: (MDLog::_replay_thread()+0x1f38) [0x5620dd74548c] 2024-05-27T07:29:51.239 INFO:journalctl@ceph.mds.d.smithi052.stdout:May 27 07:29:50 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-d[106255]: 15: (MDLog::ReplayThread::entry()+0x11) [0x5620dd74e9cb] 2024-05-27T07:29:51.239 INFO:journalctl@ceph.mds.d.smithi052.stdout:May 27 07:29:50 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-d[106255]: 16: (Thread::entry_wrapper()+0x43) [0x7f059e33972d] 2024-05-27T07:29:51.239 INFO:journalctl@ceph.mds.d.smithi052.stdout:May 27 07:29:50 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-d[106255]: 17: (Thread::_entry_func(void*)+0xd) [0x7f059e339749] 2024-05-27T07:29:51.239 INFO:journalctl@ceph.mds.d.smithi052.stdout:May 27 07:29:50 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-d[106255]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7f059cedd1ca] 2024-05-27T07:29:51.239 INFO:journalctl@ceph.mds.d.smithi052.stdout:May 27 07:29:50 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-d[106255]: 19: clone() 2024-05-27T07:29:51.239 INFO:journalctl@ceph.mds.d.smithi052.stdout:May 27 07:29:50 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-d[106255]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-05-27T07:29:51.239 INFO:journalctl@ceph.mds.d.smithi052.stdout:May 27 07:29:50 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-d[106255]: 2024-05-27T07:29:51.239 INFO:journalctl@ceph.mds.d.smithi052.stdout:May 27 07:29:50 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-d[106255]: -1> 2024-05-27T07:29:50.876+0000 7f058bb25700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: In function 'std::set SnapClient::filter(const std::set&) const' thread 7f058bb25700 time 2024-05-27T07:29:50.872338+0000 2024-05-27T07:29:51.239 INFO:journalctl@ceph.mds.d.smithi052.stdout:May 27 07:29:50 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-d[106255]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: 197: FAILED ceph_assert(cached_version > 0) 2024-05-27T07:29:51.239 INFO:journalctl@ceph.mds.d.smithi052.stdout:May 27 07:29:50 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-d[106255]: 2024-05-27T07:29:51.239 INFO:journalctl@ceph.mds.d.smithi052.stdout:May 27 07:29:50 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-d[106255]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:29:51.239 INFO:journalctl@ceph.mds.d.smithi052.stdout:May 27 07:29:50 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-d[106255]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x127) [0x7f059e36aba6] 2024-05-27T07:29:51.239 INFO:journalctl@ceph.mds.d.smithi052.stdout:May 27 07:29:50 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-d[106255]: 2: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7f059e36add1] 2024-05-27T07:29:51.240 INFO:journalctl@ceph.mds.d.smithi052.stdout:May 27 07:29:50 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-d[106255]: 3: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x5620dd72123d] 2024-05-27T07:29:51.240 INFO:journalctl@ceph.mds.d.smithi052.stdout:May 27 07:29:50 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-d[106255]: 4: (SnapRealm::build_snap_set() const+0x323) [0x5620dd70c971] 2024-05-27T07:29:51.240 INFO:journalctl@ceph.mds.d.smithi052.stdout:May 27 07:29:50 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-d[106255]: 5: (SnapRealm::check_cache() const+0x13e) [0x5620dd70afa0] 2024-05-27T07:29:51.240 INFO:journalctl@ceph.mds.d.smithi052.stdout:May 27 07:29:50 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-d[106255]: 6: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x5620dd70b651] 2024-05-27T07:29:51.240 INFO:journalctl@ceph.mds.d.smithi052.stdout:May 27 07:29:50 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-d[106255]: 7: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x5620dd6a0347] 2024-05-27T07:29:51.240 INFO:journalctl@ceph.mds.d.smithi052.stdout:May 27 07:29:50 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-d[106255]: 8: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x5620dd39c87e] 2024-05-27T07:29:51.240 INFO:journalctl@ceph.mds.d.smithi052.stdout:May 27 07:29:50 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-d[106255]: 9: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x2702) [0x5620dd3a1e94] 2024-05-27T07:29:51.240 INFO:journalctl@ceph.mds.d.smithi052.stdout:May 27 07:29:50 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-d[106255]: 10: (EUpdate::replay(MDSRank*)+0x63) [0x5620dd3ac969] 2024-05-27T07:29:51.240 INFO:journalctl@ceph.mds.d.smithi052.stdout:May 27 07:29:50 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-d[106255]: 11: (MDLog::_replay_thread()+0x1f38) [0x5620dd74548c] 2024-05-27T07:29:51.240 INFO:journalctl@ceph.mds.d.smithi052.stdout:May 27 07:29:50 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-d[106255]: 12: (MDLog::ReplayThread::entry()+0x11) [0x5620dd74e9cb] 2024-05-27T07:29:51.240 INFO:journalctl@ceph.mds.d.smithi052.stdout:May 27 07:29:50 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-d[106255]: 13: (Thread::entry_wrapper()+0x43) [0x7f059e33972d] 2024-05-27T07:29:51.240 INFO:journalctl@ceph.mds.d.smithi052.stdout:May 27 07:29:50 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-d[106255]: 14: (Thread::_entry_func(void*)+0xd) [0x7f059e339749] 2024-05-27T07:29:51.241 INFO:journalctl@ceph.mds.d.smithi052.stdout:May 27 07:29:50 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-d[106255]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7f059cedd1ca] 2024-05-27T07:29:51.241 INFO:journalctl@ceph.mds.d.smithi052.stdout:May 27 07:29:50 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-d[106255]: 16: clone() 2024-05-27T07:29:51.241 INFO:journalctl@ceph.mds.d.smithi052.stdout:May 27 07:29:50 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-d[106255]: 2024-05-27T07:29:51.241 INFO:journalctl@ceph.mds.d.smithi052.stdout:May 27 07:29:50 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-d[106255]: 0> 2024-05-27T07:29:50.883+0000 7f058bb25700 -1 *** Caught signal (Aborted) ** 2024-05-27T07:29:51.241 INFO:journalctl@ceph.mds.d.smithi052.stdout:May 27 07:29:50 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-d[106255]: in thread 7f058bb25700 thread_name:md_log_replay 2024-05-27T07:29:51.241 INFO:journalctl@ceph.mds.d.smithi052.stdout:May 27 07:29:50 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-d[106255]: 2024-05-27T07:29:51.241 INFO:journalctl@ceph.mds.d.smithi052.stdout:May 27 07:29:50 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-d[106255]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:29:51.241 INFO:journalctl@ceph.mds.d.smithi052.stdout:May 27 07:29:50 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-d[106255]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f059cee7d20] 2024-05-27T07:29:51.241 INFO:journalctl@ceph.mds.d.smithi052.stdout:May 27 07:29:50 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-d[106255]: 2: gsignal() 2024-05-27T07:29:51.241 INFO:journalctl@ceph.mds.d.smithi052.stdout:May 27 07:29:50 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-d[106255]: 3: abort() 2024-05-27T07:29:51.241 INFO:journalctl@ceph.mds.d.smithi052.stdout:May 27 07:29:50 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-d[106255]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x26e) [0x7f059e36aced] 2024-05-27T07:29:51.241 INFO:journalctl@ceph.mds.d.smithi052.stdout:May 27 07:29:50 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-d[106255]: 5: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7f059e36add1] 2024-05-27T07:29:51.241 INFO:journalctl@ceph.mds.d.smithi052.stdout:May 27 07:29:50 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-d[106255]: 6: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x5620dd72123d] 2024-05-27T07:29:51.241 INFO:journalctl@ceph.mds.d.smithi052.stdout:May 27 07:29:50 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-d[106255]: 7: (SnapRealm::build_snap_set() const+0x323) [0x5620dd70c971] 2024-05-27T07:29:51.241 INFO:journalctl@ceph.mds.d.smithi052.stdout:May 27 07:29:50 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-d[106255]: 8: (SnapRealm::check_cache() const+0x13e) [0x5620dd70afa0] 2024-05-27T07:29:51.242 INFO:journalctl@ceph.mds.d.smithi052.stdout:May 27 07:29:50 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-d[106255]: 9: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x5620dd70b651] 2024-05-27T07:29:51.242 INFO:journalctl@ceph.mds.d.smithi052.stdout:May 27 07:29:50 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-d[106255]: 10: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x5620dd6a0347] 2024-05-27T07:29:51.242 INFO:journalctl@ceph.mds.d.smithi052.stdout:May 27 07:29:50 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-d[106255]: 11: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x5620dd39c87e] 2024-05-27T07:29:51.242 INFO:journalctl@ceph.mds.d.smithi052.stdout:May 27 07:29:50 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-d[106255]: 12: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x2702) [0x5620dd3a1e94] 2024-05-27T07:29:51.242 INFO:journalctl@ceph.mds.d.smithi052.stdout:May 27 07:29:50 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-d[106255]: 13: (EUpdate::replay(MDSRank*)+0x63) [0x5620dd3ac969] 2024-05-27T07:29:51.242 INFO:journalctl@ceph.mds.d.smithi052.stdout:May 27 07:29:50 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-d[106255]: 14: (MDLog::_replay_thread()+0x1f38) [0x5620dd74548c] 2024-05-27T07:29:51.242 INFO:journalctl@ceph.mds.d.smithi052.stdout:May 27 07:29:50 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-d[106255]: 15: (MDLog::ReplayThread::entry()+0x11) [0x5620dd74e9cb] 2024-05-27T07:29:51.242 INFO:journalctl@ceph.mds.d.smithi052.stdout:May 27 07:29:50 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-d[106255]: 16: (Thread::entry_wrapper()+0x43) [0x7f059e33972d] 2024-05-27T07:29:51.243 INFO:journalctl@ceph.mds.d.smithi052.stdout:May 27 07:29:50 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-d[106255]: 17: (Thread::_entry_func(void*)+0xd) [0x7f059e339749] 2024-05-27T07:29:51.243 INFO:journalctl@ceph.mds.d.smithi052.stdout:May 27 07:29:50 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-d[106255]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7f059cedd1ca] 2024-05-27T07:29:51.243 INFO:journalctl@ceph.mds.d.smithi052.stdout:May 27 07:29:50 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-d[106255]: 19: clone() 2024-05-27T07:29:51.243 INFO:journalctl@ceph.mds.d.smithi052.stdout:May 27 07:29:50 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-d[106255]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-05-27T07:29:51.243 INFO:journalctl@ceph.mds.d.smithi052.stdout:May 27 07:29:50 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-d[106255]: 2024-05-27T07:29:51.243 INFO:journalctl@ceph.mds.d.smithi052.stdout:May 27 07:29:50 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-d[106255]: -1> 2024-05-27T07:29:50.876+0000 7f058bb25700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: In function 'std::set SnapClient::filter(const std::set&) const' thread 7f058bb25700 time 2024-05-27T07:29:50.872338+0000 2024-05-27T07:29:51.243 INFO:journalctl@ceph.mds.d.smithi052.stdout:May 27 07:29:50 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-d[106255]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: 197: FAILED ceph_assert(cached_version > 0) 2024-05-27T07:29:51.243 INFO:journalctl@ceph.mds.d.smithi052.stdout:May 27 07:29:50 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-d[106255]: 2024-05-27T07:29:51.243 INFO:journalctl@ceph.mds.d.smithi052.stdout:May 27 07:29:50 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-d[106255]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:29:51.243 INFO:journalctl@ceph.mds.d.smithi052.stdout:May 27 07:29:50 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-d[106255]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x127) [0x7f059e36aba6] 2024-05-27T07:29:51.243 INFO:journalctl@ceph.mds.d.smithi052.stdout:May 27 07:29:50 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-d[106255]: 2: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7f059e36add1] 2024-05-27T07:29:51.243 INFO:journalctl@ceph.mds.d.smithi052.stdout:May 27 07:29:50 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-d[106255]: 3: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x5620dd72123d] 2024-05-27T07:29:51.243 INFO:journalctl@ceph.mds.d.smithi052.stdout:May 27 07:29:50 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-d[106255]: 4: (SnapRealm::build_snap_set() const+0x323) [0x5620dd70c971] 2024-05-27T07:29:51.244 INFO:journalctl@ceph.mds.d.smithi052.stdout:May 27 07:29:50 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-d[106255]: 5: (SnapRealm::check_cache() const+0x13e) [0x5620dd70afa0] 2024-05-27T07:29:51.244 INFO:journalctl@ceph.mds.d.smithi052.stdout:May 27 07:29:50 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-d[106255]: 6: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x5620dd70b651] 2024-05-27T07:29:51.244 INFO:journalctl@ceph.mds.d.smithi052.stdout:May 27 07:29:50 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-d[106255]: 7: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x5620dd6a0347] 2024-05-27T07:29:51.244 INFO:journalctl@ceph.mds.d.smithi052.stdout:May 27 07:29:50 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-d[106255]: 8: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x5620dd39c87e] 2024-05-27T07:29:51.244 INFO:journalctl@ceph.mds.d.smithi052.stdout:May 27 07:29:50 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-d[106255]: 9: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x2702) [0x5620dd3a1e94] 2024-05-27T07:29:51.244 INFO:journalctl@ceph.mds.d.smithi052.stdout:May 27 07:29:50 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-d[106255]: 10: (EUpdate::replay(MDSRank*)+0x63) [0x5620dd3ac969] 2024-05-27T07:29:51.244 INFO:journalctl@ceph.mds.d.smithi052.stdout:May 27 07:29:50 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-d[106255]: 11: (MDLog::_replay_thread()+0x1f38) [0x5620dd74548c] 2024-05-27T07:29:51.244 INFO:journalctl@ceph.mds.d.smithi052.stdout:May 27 07:29:50 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-d[106255]: 12: (MDLog::ReplayThread::entry()+0x11) [0x5620dd74e9cb] 2024-05-27T07:29:51.244 INFO:journalctl@ceph.mds.d.smithi052.stdout:May 27 07:29:50 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-d[106255]: 13: (Thread::entry_wrapper()+0x43) [0x7f059e33972d] 2024-05-27T07:29:51.244 INFO:journalctl@ceph.mds.d.smithi052.stdout:May 27 07:29:50 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-d[106255]: 14: (Thread::_entry_func(void*)+0xd) [0x7f059e339749] 2024-05-27T07:29:51.244 INFO:journalctl@ceph.mds.d.smithi052.stdout:May 27 07:29:50 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-d[106255]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7f059cedd1ca] 2024-05-27T07:29:51.244 INFO:journalctl@ceph.mds.d.smithi052.stdout:May 27 07:29:50 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-d[106255]: 16: clone() 2024-05-27T07:29:51.244 INFO:journalctl@ceph.mds.d.smithi052.stdout:May 27 07:29:50 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-d[106255]: 2024-05-27T07:29:51.245 INFO:journalctl@ceph.mds.d.smithi052.stdout:May 27 07:29:50 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-d[106255]: 0> 2024-05-27T07:29:50.883+0000 7f058bb25700 -1 *** Caught signal (Aborted) ** 2024-05-27T07:29:51.245 INFO:journalctl@ceph.mds.d.smithi052.stdout:May 27 07:29:50 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-d[106255]: in thread 7f058bb25700 thread_name:md_log_replay 2024-05-27T07:29:51.245 INFO:journalctl@ceph.mds.d.smithi052.stdout:May 27 07:29:50 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-d[106255]: 2024-05-27T07:29:51.245 INFO:journalctl@ceph.mds.d.smithi052.stdout:May 27 07:29:50 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-d[106255]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:29:51.245 INFO:journalctl@ceph.mds.d.smithi052.stdout:May 27 07:29:50 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-d[106255]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f059cee7d20] 2024-05-27T07:29:51.245 INFO:journalctl@ceph.mds.d.smithi052.stdout:May 27 07:29:50 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-d[106255]: 2: gsignal() 2024-05-27T07:29:51.245 INFO:journalctl@ceph.mds.d.smithi052.stdout:May 27 07:29:50 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-d[106255]: 3: abort() 2024-05-27T07:29:51.245 INFO:journalctl@ceph.mds.d.smithi052.stdout:May 27 07:29:50 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-d[106255]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x26e) [0x7f059e36aced] 2024-05-27T07:29:51.245 INFO:journalctl@ceph.mds.d.smithi052.stdout:May 27 07:29:50 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-d[106255]: 5: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7f059e36add1] 2024-05-27T07:29:51.245 INFO:journalctl@ceph.mds.d.smithi052.stdout:May 27 07:29:50 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-d[106255]: 6: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x5620dd72123d] 2024-05-27T07:29:51.245 INFO:journalctl@ceph.mds.d.smithi052.stdout:May 27 07:29:50 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-d[106255]: 7: (SnapRealm::build_snap_set() const+0x323) [0x5620dd70c971] 2024-05-27T07:29:51.245 INFO:journalctl@ceph.mds.d.smithi052.stdout:May 27 07:29:50 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-d[106255]: 8: (SnapRealm::check_cache() const+0x13e) [0x5620dd70afa0] 2024-05-27T07:29:51.245 INFO:journalctl@ceph.mds.d.smithi052.stdout:May 27 07:29:50 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-d[106255]: 9: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x5620dd70b651] 2024-05-27T07:29:51.245 INFO:journalctl@ceph.mds.d.smithi052.stdout:May 27 07:29:50 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-d[106255]: 10: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x5620dd6a0347] 2024-05-27T07:29:51.246 INFO:journalctl@ceph.mds.d.smithi052.stdout:May 27 07:29:50 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-d[106255]: 11: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x5620dd39c87e] 2024-05-27T07:29:51.246 INFO:journalctl@ceph.mds.d.smithi052.stdout:May 27 07:29:50 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-d[106255]: 12: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x2702) [0x5620dd3a1e94] 2024-05-27T07:29:51.246 INFO:journalctl@ceph.mds.d.smithi052.stdout:May 27 07:29:50 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-d[106255]: 13: (EUpdate::replay(MDSRank*)+0x63) [0x5620dd3ac969] 2024-05-27T07:29:51.246 INFO:journalctl@ceph.mds.d.smithi052.stdout:May 27 07:29:50 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-d[106255]: 14: (MDLog::_replay_thread()+0x1f38) [0x5620dd74548c] 2024-05-27T07:29:51.246 INFO:journalctl@ceph.mds.d.smithi052.stdout:May 27 07:29:50 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-d[106255]: 15: (MDLog::ReplayThread::entry()+0x11) [0x5620dd74e9cb] 2024-05-27T07:29:51.246 INFO:journalctl@ceph.mds.d.smithi052.stdout:May 27 07:29:50 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-d[106255]: 16: (Thread::entry_wrapper()+0x43) [0x7f059e33972d] 2024-05-27T07:29:51.246 INFO:journalctl@ceph.mds.d.smithi052.stdout:May 27 07:29:50 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-d[106255]: 17: (Thread::_entry_func(void*)+0xd) [0x7f059e339749] 2024-05-27T07:29:51.246 INFO:journalctl@ceph.mds.d.smithi052.stdout:May 27 07:29:50 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-d[106255]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7f059cedd1ca] 2024-05-27T07:29:51.246 INFO:journalctl@ceph.mds.d.smithi052.stdout:May 27 07:29:50 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-d[106255]: 19: clone() 2024-05-27T07:29:51.246 INFO:journalctl@ceph.mds.d.smithi052.stdout:May 27 07:29:50 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-d[106255]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-05-27T07:29:51.246 INFO:journalctl@ceph.mds.d.smithi052.stdout:May 27 07:29:50 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-d[106255]: 2024-05-27T07:29:51.481 INFO:journalctl@ceph.mds.a.smithi052.stdout:May 27 07:29:51 smithi052 systemd[1]: ceph-73df2636-1bf8-11ef-bc9b-c7b262605968@mds.a.service: Main process exited, code=exited, status=134/n/a 2024-05-27T07:29:51.481 INFO:journalctl@ceph.mds.g.smithi052.stdout:May 27 07:29:51 smithi052 podman[106427]: 2024-05-27 07:29:51.308283817 +0000 UTC m=+1.333160414 container died 366d6f4b82cae2e92c867e97cde83ae500f5633b53b558e9f83dca56e4b4b222 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-g, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, org.label-schema.license=GPLv2, GIT_CLEAN=True, org.label-schema.build-date=20240521, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True) 2024-05-27T07:29:51.516 INFO:journalctl@ceph.mds.l.smithi173.stdout:May 27 07:29:51 smithi173 podman[73778]: 2024-05-27 07:29:51.263115869 +0000 UTC m=+1.135030040 container remove d70a52e699b5e512e537a44cc9d1f077b52180fbafd7465fb3950b042a8eb975 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-l, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240521, org.label-schema.license=GPLv2, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, ceph=True, maintainer=Guillaume Abrioux ) 2024-05-27T07:29:51.516 INFO:journalctl@ceph.mds.l.smithi173.stdout:May 27 07:29:51 smithi173 systemd[1]: ceph-73df2636-1bf8-11ef-bc9b-c7b262605968@mds.l.service: Main process exited, code=exited, status=134/n/a 2024-05-27T07:29:51.852 INFO:journalctl@ceph.mds.l.smithi173.stdout:May 27 07:29:51 smithi173 systemd[1]: ceph-73df2636-1bf8-11ef-bc9b-c7b262605968@mds.l.service: Failed with result 'exit-code'. 2024-05-27T07:29:52.191 INFO:journalctl@ceph.mds.k.smithi105.stdout:May 27 07:29:51 smithi105 podman[69974]: 2024-05-27 07:29:51.924286107 +0000 UTC m=+1.086829435 container init 65fd1707409f955d437095b2ebc4b654df22761089d5fcee3ce3884251d520be (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-k, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.vendor=CentOS, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, org.label-schema.build-date=20240521, org.label-schema.license=GPLv2, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, org.label-schema.schema-version=1.0) 2024-05-27T07:29:52.191 INFO:journalctl@ceph.mds.k.smithi105.stdout:May 27 07:29:51 smithi105 podman[69974]: 2024-05-27 07:29:51.929084725 +0000 UTC m=+1.091628050 container start 65fd1707409f955d437095b2ebc4b654df22761089d5fcee3ce3884251d520be (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-k, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, org.label-schema.build-date=20240521, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, GIT_CLEAN=True, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, org.label-schema.schema-version=1.0, ceph=True, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS) 2024-05-27T07:29:52.191 INFO:journalctl@ceph.mds.k.smithi105.stdout:May 27 07:29:51 smithi105 ceph-mds[69994]: -- 172.21.15.105:0/1385537513 <== mon.2 v2:172.21.15.105:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55c5de119a40 con 0x55c5de161a80 2024-05-27T07:29:52.191 INFO:journalctl@ceph.mds.k.smithi105.stdout:May 27 07:29:51 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-k[69990]: starting mds.k at 2024-05-27T07:29:52.191 INFO:journalctl@ceph.mds.k.smithi105.stdout:May 27 07:29:52 smithi105 bash[69974]: 65fd1707409f955d437095b2ebc4b654df22761089d5fcee3ce3884251d520be 2024-05-27T07:29:52.191 INFO:journalctl@ceph.mds.k.smithi105.stdout:May 27 07:29:52 smithi105 systemd[1]: Started Ceph mds.k for 73df2636-1bf8-11ef-bc9b-c7b262605968. 2024-05-27T07:29:52.480 INFO:journalctl@ceph.mds.g.smithi052.stdout:May 27 07:29:52 smithi052 podman[106427]: 2024-05-27 07:29:52.183441304 +0000 UTC m=+2.208317896 container remove 366d6f4b82cae2e92c867e97cde83ae500f5633b53b558e9f83dca56e4b4b222 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-g, GIT_CLEAN=True, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, ceph=True, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240521, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0) 2024-05-27T07:29:52.480 INFO:journalctl@ceph.mds.g.smithi052.stdout:May 27 07:29:52 smithi052 systemd[1]: ceph-73df2636-1bf8-11ef-bc9b-c7b262605968@mds.g.service: Main process exited, code=exited, status=134/n/a 2024-05-27T07:29:52.481 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:29:52 smithi052 ceph-mon[31955]: pgmap v138: 129 pgs: 129 active+clean; 269 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 605 KiB/s rd, 1.6 MiB/s wr, 735 op/s 2024-05-27T07:29:52.481 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:29:52 smithi052 ceph-mon[31955]: mds.? [v2:172.21.15.52:6834/69093683,v1:172.21.15.52:6835/69093683] up:boot 2024-05-27T07:29:52.481 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:29:52 smithi052 ceph-mon[31955]: fsmap cephfs:5 5 up:standby-replay 1 up:standby 5 up:active 2024-05-27T07:29:52.481 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:29:52 smithi052 ceph-mon[31955]: from='mgr.15333 172.21.15.52:0/416389082' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "j"}]: dispatch 2024-05-27T07:29:52.481 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:29:52 smithi052 ceph-mon[31955]: daemon mds.k restarted 2024-05-27T07:29:52.603 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:29:52 smithi173 ceph-mon[37000]: pgmap v138: 129 pgs: 129 active+clean; 269 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 605 KiB/s rd, 1.6 MiB/s wr, 735 op/s 2024-05-27T07:29:52.603 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:29:52 smithi173 ceph-mon[37000]: mds.? [v2:172.21.15.52:6834/69093683,v1:172.21.15.52:6835/69093683] up:boot 2024-05-27T07:29:52.603 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:29:52 smithi173 ceph-mon[37000]: fsmap cephfs:5 5 up:standby-replay 1 up:standby 5 up:active 2024-05-27T07:29:52.603 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:29:52 smithi173 ceph-mon[37000]: from='mgr.15333 172.21.15.52:0/416389082' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "j"}]: dispatch 2024-05-27T07:29:52.603 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:29:52 smithi173 ceph-mon[37000]: daemon mds.k restarted 2024-05-27T07:29:52.690 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:29:52 smithi105 ceph-mon[35977]: pgmap v138: 129 pgs: 129 active+clean; 269 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 605 KiB/s rd, 1.6 MiB/s wr, 735 op/s 2024-05-27T07:29:52.690 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:29:52 smithi105 ceph-mon[35977]: mds.? [v2:172.21.15.52:6834/69093683,v1:172.21.15.52:6835/69093683] up:boot 2024-05-27T07:29:52.691 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:29:52 smithi105 ceph-mon[35977]: fsmap cephfs:5 5 up:standby-replay 1 up:standby 5 up:active 2024-05-27T07:29:52.691 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:29:52 smithi105 ceph-mon[35977]: from='mgr.15333 172.21.15.52:0/416389082' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "j"}]: dispatch 2024-05-27T07:29:52.691 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:29:52 smithi105 ceph-mon[35977]: daemon mds.k restarted 2024-05-27T07:29:52.691 INFO:journalctl@ceph.mds.k.smithi105.stdout:May 27 07:29:52 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-k[69990]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: In function 'std::set SnapClient::filter(const std::set&) const' thread 7f1bea666700 time 2024-05-27T07:29:52.348263+0000 2024-05-27T07:29:52.691 INFO:journalctl@ceph.mds.k.smithi105.stdout:May 27 07:29:52 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-k[69990]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: 197: FAILED ceph_assert(cached_version > 0) 2024-05-27T07:29:52.691 INFO:journalctl@ceph.mds.k.smithi105.stdout:May 27 07:29:52 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-k[69990]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:29:52.691 INFO:journalctl@ceph.mds.k.smithi105.stdout:May 27 07:29:52 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-k[69990]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x127) [0x7f1bfceabba6] 2024-05-27T07:29:52.691 INFO:journalctl@ceph.mds.k.smithi105.stdout:May 27 07:29:52 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-k[69990]: 2: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7f1bfceabdd1] 2024-05-27T07:29:52.692 INFO:journalctl@ceph.mds.k.smithi105.stdout:May 27 07:29:52 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-k[69990]: 3: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x55c5dcbf323d] 2024-05-27T07:29:52.692 INFO:journalctl@ceph.mds.k.smithi105.stdout:May 27 07:29:52 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-k[69990]: 4: (SnapRealm::build_snap_set() const+0x323) [0x55c5dcbde971] 2024-05-27T07:29:52.692 INFO:journalctl@ceph.mds.k.smithi105.stdout:May 27 07:29:52 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-k[69990]: 5: (SnapRealm::check_cache() const+0x13e) [0x55c5dcbdcfa0] 2024-05-27T07:29:52.692 INFO:journalctl@ceph.mds.k.smithi105.stdout:May 27 07:29:52 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-k[69990]: 6: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x55c5dcbdd651] 2024-05-27T07:29:52.692 INFO:journalctl@ceph.mds.k.smithi105.stdout:May 27 07:29:52 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-k[69990]: 7: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x55c5dcb72347] 2024-05-27T07:29:52.692 INFO:journalctl@ceph.mds.k.smithi105.stdout:May 27 07:29:52 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-k[69990]: 8: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x55c5dc86e87e] 2024-05-27T07:29:52.692 INFO:journalctl@ceph.mds.k.smithi105.stdout:May 27 07:29:52 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-k[69990]: 9: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x279d) [0x55c5dc873f2f] 2024-05-27T07:29:52.692 INFO:journalctl@ceph.mds.k.smithi105.stdout:May 27 07:29:52 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-k[69990]: 10: (EUpdate::replay(MDSRank*)+0x63) [0x55c5dc87e969] 2024-05-27T07:29:52.692 INFO:journalctl@ceph.mds.k.smithi105.stdout:May 27 07:29:52 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-k[69990]: 11: (MDLog::_replay_thread()+0x1f38) [0x55c5dcc1748c] 2024-05-27T07:29:52.692 INFO:journalctl@ceph.mds.k.smithi105.stdout:May 27 07:29:52 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-k[69990]: 12: (MDLog::ReplayThread::entry()+0x11) [0x55c5dcc209cb] 2024-05-27T07:29:52.692 INFO:journalctl@ceph.mds.k.smithi105.stdout:May 27 07:29:52 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-k[69990]: 13: (Thread::entry_wrapper()+0x43) [0x7f1bfce7a72d] 2024-05-27T07:29:52.692 INFO:journalctl@ceph.mds.k.smithi105.stdout:May 27 07:29:52 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-k[69990]: 14: (Thread::_entry_func(void*)+0xd) [0x7f1bfce7a749] 2024-05-27T07:29:52.692 INFO:journalctl@ceph.mds.k.smithi105.stdout:May 27 07:29:52 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-k[69990]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7f1bfba1e1ca] 2024-05-27T07:29:52.692 INFO:journalctl@ceph.mds.k.smithi105.stdout:May 27 07:29:52 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-k[69990]: 16: clone() 2024-05-27T07:29:52.693 INFO:journalctl@ceph.mds.k.smithi105.stdout:May 27 07:29:52 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-k[69990]: 2024-05-27T07:29:52.350+0000 7f1bea666700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: In function 'std::set SnapClient::filter(const std::set&) const' thread 7f1bea666700 time 2024-05-27T07:29:52.348263+0000 2024-05-27T07:29:52.693 INFO:journalctl@ceph.mds.k.smithi105.stdout:May 27 07:29:52 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-k[69990]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: 197: FAILED ceph_assert(cached_version > 0) 2024-05-27T07:29:52.693 INFO:journalctl@ceph.mds.k.smithi105.stdout:May 27 07:29:52 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-k[69990]: 2024-05-27T07:29:52.693 INFO:journalctl@ceph.mds.k.smithi105.stdout:May 27 07:29:52 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-k[69990]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:29:52.693 INFO:journalctl@ceph.mds.k.smithi105.stdout:May 27 07:29:52 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-k[69990]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x127) [0x7f1bfceabba6] 2024-05-27T07:29:52.693 INFO:journalctl@ceph.mds.k.smithi105.stdout:May 27 07:29:52 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-k[69990]: 2: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7f1bfceabdd1] 2024-05-27T07:29:52.693 INFO:journalctl@ceph.mds.k.smithi105.stdout:May 27 07:29:52 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-k[69990]: 3: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x55c5dcbf323d] 2024-05-27T07:29:52.693 INFO:journalctl@ceph.mds.k.smithi105.stdout:May 27 07:29:52 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-k[69990]: 4: (SnapRealm::build_snap_set() const+0x323) [0x55c5dcbde971] 2024-05-27T07:29:52.693 INFO:journalctl@ceph.mds.k.smithi105.stdout:May 27 07:29:52 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-k[69990]: 5: (SnapRealm::check_cache() const+0x13e) [0x55c5dcbdcfa0] 2024-05-27T07:29:52.694 INFO:journalctl@ceph.mds.k.smithi105.stdout:May 27 07:29:52 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-k[69990]: 6: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x55c5dcbdd651] 2024-05-27T07:29:52.694 INFO:journalctl@ceph.mds.k.smithi105.stdout:May 27 07:29:52 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-k[69990]: 7: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x55c5dcb72347] 2024-05-27T07:29:52.694 INFO:journalctl@ceph.mds.k.smithi105.stdout:May 27 07:29:52 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-k[69990]: 8: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x55c5dc86e87e] 2024-05-27T07:29:52.694 INFO:journalctl@ceph.mds.k.smithi105.stdout:May 27 07:29:52 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-k[69990]: 9: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x279d) [0x55c5dc873f2f] 2024-05-27T07:29:52.694 INFO:journalctl@ceph.mds.k.smithi105.stdout:May 27 07:29:52 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-k[69990]: 10: (EUpdate::replay(MDSRank*)+0x63) [0x55c5dc87e969] 2024-05-27T07:29:52.694 INFO:journalctl@ceph.mds.k.smithi105.stdout:May 27 07:29:52 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-k[69990]: 11: (MDLog::_replay_thread()+0x1f38) [0x55c5dcc1748c] 2024-05-27T07:29:52.694 INFO:journalctl@ceph.mds.k.smithi105.stdout:May 27 07:29:52 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-k[69990]: 12: (MDLog::ReplayThread::entry()+0x11) [0x55c5dcc209cb] 2024-05-27T07:29:52.694 INFO:journalctl@ceph.mds.k.smithi105.stdout:May 27 07:29:52 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-k[69990]: 13: (Thread::entry_wrapper()+0x43) [0x7f1bfce7a72d] 2024-05-27T07:29:52.694 INFO:journalctl@ceph.mds.k.smithi105.stdout:May 27 07:29:52 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-k[69990]: 14: (Thread::_entry_func(void*)+0xd) [0x7f1bfce7a749] 2024-05-27T07:29:52.694 INFO:journalctl@ceph.mds.k.smithi105.stdout:May 27 07:29:52 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-k[69990]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7f1bfba1e1ca] 2024-05-27T07:29:52.694 INFO:journalctl@ceph.mds.k.smithi105.stdout:May 27 07:29:52 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-k[69990]: 16: clone() 2024-05-27T07:29:52.694 INFO:journalctl@ceph.mds.k.smithi105.stdout:May 27 07:29:52 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-k[69990]: 2024-05-27T07:29:52.694 INFO:journalctl@ceph.mds.k.smithi105.stdout:May 27 07:29:52 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-k[69990]: *** Caught signal (Aborted) ** 2024-05-27T07:29:52.695 INFO:journalctl@ceph.mds.k.smithi105.stdout:May 27 07:29:52 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-k[69990]: in thread 7f1bea666700 thread_name:md_log_replay 2024-05-27T07:29:52.695 INFO:journalctl@ceph.mds.k.smithi105.stdout:May 27 07:29:52 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-k[69990]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:29:52.695 INFO:journalctl@ceph.mds.k.smithi105.stdout:May 27 07:29:52 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-k[69990]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f1bfba28d20] 2024-05-27T07:29:52.695 INFO:journalctl@ceph.mds.k.smithi105.stdout:May 27 07:29:52 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-k[69990]: 2: gsignal() 2024-05-27T07:29:52.695 INFO:journalctl@ceph.mds.k.smithi105.stdout:May 27 07:29:52 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-k[69990]: 3: abort() 2024-05-27T07:29:52.695 INFO:journalctl@ceph.mds.k.smithi105.stdout:May 27 07:29:52 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-k[69990]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x26e) [0x7f1bfceabced] 2024-05-27T07:29:52.695 INFO:journalctl@ceph.mds.k.smithi105.stdout:May 27 07:29:52 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-k[69990]: 5: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7f1bfceabdd1] 2024-05-27T07:29:52.696 INFO:journalctl@ceph.mds.k.smithi105.stdout:May 27 07:29:52 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-k[69990]: 6: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x55c5dcbf323d] 2024-05-27T07:29:52.696 INFO:journalctl@ceph.mds.k.smithi105.stdout:May 27 07:29:52 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-k[69990]: 7: (SnapRealm::build_snap_set() const+0x323) [0x55c5dcbde971] 2024-05-27T07:29:52.696 INFO:journalctl@ceph.mds.k.smithi105.stdout:May 27 07:29:52 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-k[69990]: 8: (SnapRealm::check_cache() const+0x13e) [0x55c5dcbdcfa0] 2024-05-27T07:29:52.696 INFO:journalctl@ceph.mds.k.smithi105.stdout:May 27 07:29:52 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-k[69990]: 9: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x55c5dcbdd651] 2024-05-27T07:29:52.696 INFO:journalctl@ceph.mds.k.smithi105.stdout:May 27 07:29:52 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-k[69990]: 10: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x55c5dcb72347] 2024-05-27T07:29:52.696 INFO:journalctl@ceph.mds.k.smithi105.stdout:May 27 07:29:52 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-k[69990]: 11: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x55c5dc86e87e] 2024-05-27T07:29:52.696 INFO:journalctl@ceph.mds.k.smithi105.stdout:May 27 07:29:52 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-k[69990]: 12: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x279d) [0x55c5dc873f2f] 2024-05-27T07:29:52.696 INFO:journalctl@ceph.mds.k.smithi105.stdout:May 27 07:29:52 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-k[69990]: 13: (EUpdate::replay(MDSRank*)+0x63) [0x55c5dc87e969] 2024-05-27T07:29:52.696 INFO:journalctl@ceph.mds.k.smithi105.stdout:May 27 07:29:52 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-k[69990]: 14: (MDLog::_replay_thread()+0x1f38) [0x55c5dcc1748c] 2024-05-27T07:29:52.696 INFO:journalctl@ceph.mds.k.smithi105.stdout:May 27 07:29:52 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-k[69990]: 15: (MDLog::ReplayThread::entry()+0x11) [0x55c5dcc209cb] 2024-05-27T07:29:52.696 INFO:journalctl@ceph.mds.k.smithi105.stdout:May 27 07:29:52 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-k[69990]: 16: (Thread::entry_wrapper()+0x43) [0x7f1bfce7a72d] 2024-05-27T07:29:52.696 INFO:journalctl@ceph.mds.k.smithi105.stdout:May 27 07:29:52 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-k[69990]: 17: (Thread::_entry_func(void*)+0xd) [0x7f1bfce7a749] 2024-05-27T07:29:52.697 INFO:journalctl@ceph.mds.k.smithi105.stdout:May 27 07:29:52 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-k[69990]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7f1bfba1e1ca] 2024-05-27T07:29:52.697 INFO:journalctl@ceph.mds.k.smithi105.stdout:May 27 07:29:52 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-k[69990]: 19: clone() 2024-05-27T07:29:52.697 INFO:journalctl@ceph.mds.k.smithi105.stdout:May 27 07:29:52 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-k[69990]: 2024-05-27T07:29:52.353+0000 7f1bea666700 -1 *** Caught signal (Aborted) ** 2024-05-27T07:29:52.697 INFO:journalctl@ceph.mds.k.smithi105.stdout:May 27 07:29:52 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-k[69990]: in thread 7f1bea666700 thread_name:md_log_replay 2024-05-27T07:29:52.697 INFO:journalctl@ceph.mds.k.smithi105.stdout:May 27 07:29:52 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-k[69990]: 2024-05-27T07:29:52.697 INFO:journalctl@ceph.mds.k.smithi105.stdout:May 27 07:29:52 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-k[69990]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:29:52.697 INFO:journalctl@ceph.mds.k.smithi105.stdout:May 27 07:29:52 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-k[69990]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f1bfba28d20] 2024-05-27T07:29:52.697 INFO:journalctl@ceph.mds.k.smithi105.stdout:May 27 07:29:52 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-k[69990]: 2: gsignal() 2024-05-27T07:29:52.697 INFO:journalctl@ceph.mds.k.smithi105.stdout:May 27 07:29:52 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-k[69990]: 3: abort() 2024-05-27T07:29:52.697 INFO:journalctl@ceph.mds.k.smithi105.stdout:May 27 07:29:52 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-k[69990]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x26e) [0x7f1bfceabced] 2024-05-27T07:29:52.697 INFO:journalctl@ceph.mds.k.smithi105.stdout:May 27 07:29:52 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-k[69990]: 5: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7f1bfceabdd1] 2024-05-27T07:29:52.697 INFO:journalctl@ceph.mds.k.smithi105.stdout:May 27 07:29:52 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-k[69990]: 6: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x55c5dcbf323d] 2024-05-27T07:29:52.698 INFO:journalctl@ceph.mds.k.smithi105.stdout:May 27 07:29:52 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-k[69990]: 7: (SnapRealm::build_snap_set() const+0x323) [0x55c5dcbde971] 2024-05-27T07:29:52.698 INFO:journalctl@ceph.mds.k.smithi105.stdout:May 27 07:29:52 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-k[69990]: 8: (SnapRealm::check_cache() const+0x13e) [0x55c5dcbdcfa0] 2024-05-27T07:29:52.698 INFO:journalctl@ceph.mds.k.smithi105.stdout:May 27 07:29:52 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-k[69990]: 9: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x55c5dcbdd651] 2024-05-27T07:29:52.698 INFO:journalctl@ceph.mds.k.smithi105.stdout:May 27 07:29:52 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-k[69990]: 10: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x55c5dcb72347] 2024-05-27T07:29:52.698 INFO:journalctl@ceph.mds.k.smithi105.stdout:May 27 07:29:52 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-k[69990]: 11: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x55c5dc86e87e] 2024-05-27T07:29:52.698 INFO:journalctl@ceph.mds.k.smithi105.stdout:May 27 07:29:52 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-k[69990]: 12: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x279d) [0x55c5dc873f2f] 2024-05-27T07:29:52.698 INFO:journalctl@ceph.mds.k.smithi105.stdout:May 27 07:29:52 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-k[69990]: 13: (EUpdate::replay(MDSRank*)+0x63) [0x55c5dc87e969] 2024-05-27T07:29:52.698 INFO:journalctl@ceph.mds.k.smithi105.stdout:May 27 07:29:52 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-k[69990]: 14: (MDLog::_replay_thread()+0x1f38) [0x55c5dcc1748c] 2024-05-27T07:29:52.698 INFO:journalctl@ceph.mds.k.smithi105.stdout:May 27 07:29:52 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-k[69990]: 15: (MDLog::ReplayThread::entry()+0x11) [0x55c5dcc209cb] 2024-05-27T07:29:52.698 INFO:journalctl@ceph.mds.k.smithi105.stdout:May 27 07:29:52 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-k[69990]: 16: (Thread::entry_wrapper()+0x43) [0x7f1bfce7a72d] 2024-05-27T07:29:52.698 INFO:journalctl@ceph.mds.k.smithi105.stdout:May 27 07:29:52 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-k[69990]: 17: (Thread::_entry_func(void*)+0xd) [0x7f1bfce7a749] 2024-05-27T07:29:52.698 INFO:journalctl@ceph.mds.k.smithi105.stdout:May 27 07:29:52 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-k[69990]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7f1bfba1e1ca] 2024-05-27T07:29:52.698 INFO:journalctl@ceph.mds.k.smithi105.stdout:May 27 07:29:52 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-k[69990]: 19: clone() 2024-05-27T07:29:52.699 INFO:journalctl@ceph.mds.k.smithi105.stdout:May 27 07:29:52 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-k[69990]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-05-27T07:29:52.699 INFO:journalctl@ceph.mds.k.smithi105.stdout:May 27 07:29:52 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-k[69990]: 2024-05-27T07:29:52.699 INFO:journalctl@ceph.mds.k.smithi105.stdout:May 27 07:29:52 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-k[69990]: -2> 2024-05-27T07:29:52.350+0000 7f1bea666700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: In function 'std::set SnapClient::filter(const std::set&) const' thread 7f1bea666700 time 2024-05-27T07:29:52.348263+0000 2024-05-27T07:29:52.699 INFO:journalctl@ceph.mds.k.smithi105.stdout:May 27 07:29:52 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-k[69990]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: 197: FAILED ceph_assert(cached_version > 0) 2024-05-27T07:29:52.699 INFO:journalctl@ceph.mds.k.smithi105.stdout:May 27 07:29:52 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-k[69990]: 2024-05-27T07:29:52.699 INFO:journalctl@ceph.mds.k.smithi105.stdout:May 27 07:29:52 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-k[69990]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:29:52.699 INFO:journalctl@ceph.mds.k.smithi105.stdout:May 27 07:29:52 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-k[69990]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x127) [0x7f1bfceabba6] 2024-05-27T07:29:52.699 INFO:journalctl@ceph.mds.k.smithi105.stdout:May 27 07:29:52 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-k[69990]: 2: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7f1bfceabdd1] 2024-05-27T07:29:52.699 INFO:journalctl@ceph.mds.k.smithi105.stdout:May 27 07:29:52 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-k[69990]: 3: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x55c5dcbf323d] 2024-05-27T07:29:52.699 INFO:journalctl@ceph.mds.k.smithi105.stdout:May 27 07:29:52 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-k[69990]: 4: (SnapRealm::build_snap_set() const+0x323) [0x55c5dcbde971] 2024-05-27T07:29:52.699 INFO:journalctl@ceph.mds.k.smithi105.stdout:May 27 07:29:52 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-k[69990]: 5: (SnapRealm::check_cache() const+0x13e) [0x55c5dcbdcfa0] 2024-05-27T07:29:52.699 INFO:journalctl@ceph.mds.k.smithi105.stdout:May 27 07:29:52 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-k[69990]: 6: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x55c5dcbdd651] 2024-05-27T07:29:52.699 INFO:journalctl@ceph.mds.k.smithi105.stdout:May 27 07:29:52 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-k[69990]: 7: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x55c5dcb72347] 2024-05-27T07:29:52.700 INFO:journalctl@ceph.mds.k.smithi105.stdout:May 27 07:29:52 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-k[69990]: 8: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x55c5dc86e87e] 2024-05-27T07:29:52.700 INFO:journalctl@ceph.mds.k.smithi105.stdout:May 27 07:29:52 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-k[69990]: 9: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x279d) [0x55c5dc873f2f] 2024-05-27T07:29:52.700 INFO:journalctl@ceph.mds.k.smithi105.stdout:May 27 07:29:52 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-k[69990]: 10: (EUpdate::replay(MDSRank*)+0x63) [0x55c5dc87e969] 2024-05-27T07:29:52.700 INFO:journalctl@ceph.mds.k.smithi105.stdout:May 27 07:29:52 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-k[69990]: 11: (MDLog::_replay_thread()+0x1f38) [0x55c5dcc1748c] 2024-05-27T07:29:52.700 INFO:journalctl@ceph.mds.k.smithi105.stdout:May 27 07:29:52 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-k[69990]: 12: (MDLog::ReplayThread::entry()+0x11) [0x55c5dcc209cb] 2024-05-27T07:29:52.700 INFO:journalctl@ceph.mds.k.smithi105.stdout:May 27 07:29:52 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-k[69990]: 13: (Thread::entry_wrapper()+0x43) [0x7f1bfce7a72d] 2024-05-27T07:29:52.700 INFO:journalctl@ceph.mds.k.smithi105.stdout:May 27 07:29:52 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-k[69990]: 14: (Thread::_entry_func(void*)+0xd) [0x7f1bfce7a749] 2024-05-27T07:29:52.700 INFO:journalctl@ceph.mds.k.smithi105.stdout:May 27 07:29:52 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-k[69990]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7f1bfba1e1ca] 2024-05-27T07:29:52.700 INFO:journalctl@ceph.mds.k.smithi105.stdout:May 27 07:29:52 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-k[69990]: 16: clone() 2024-05-27T07:29:52.700 INFO:journalctl@ceph.mds.k.smithi105.stdout:May 27 07:29:52 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-k[69990]: 2024-05-27T07:29:52.700 INFO:journalctl@ceph.mds.k.smithi105.stdout:May 27 07:29:52 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-k[69990]: 0> 2024-05-27T07:29:52.353+0000 7f1bea666700 -1 *** Caught signal (Aborted) ** 2024-05-27T07:29:52.700 INFO:journalctl@ceph.mds.k.smithi105.stdout:May 27 07:29:52 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-k[69990]: in thread 7f1bea666700 thread_name:md_log_replay 2024-05-27T07:29:52.700 INFO:journalctl@ceph.mds.k.smithi105.stdout:May 27 07:29:52 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-k[69990]: 2024-05-27T07:29:52.701 INFO:journalctl@ceph.mds.k.smithi105.stdout:May 27 07:29:52 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-k[69990]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:29:52.701 INFO:journalctl@ceph.mds.k.smithi105.stdout:May 27 07:29:52 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-k[69990]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f1bfba28d20] 2024-05-27T07:29:52.701 INFO:journalctl@ceph.mds.k.smithi105.stdout:May 27 07:29:52 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-k[69990]: 2: gsignal() 2024-05-27T07:29:52.701 INFO:journalctl@ceph.mds.k.smithi105.stdout:May 27 07:29:52 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-k[69990]: 3: abort() 2024-05-27T07:29:52.701 INFO:journalctl@ceph.mds.k.smithi105.stdout:May 27 07:29:52 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-k[69990]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x26e) [0x7f1bfceabced] 2024-05-27T07:29:52.701 INFO:journalctl@ceph.mds.k.smithi105.stdout:May 27 07:29:52 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-k[69990]: 5: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7f1bfceabdd1] 2024-05-27T07:29:52.701 INFO:journalctl@ceph.mds.k.smithi105.stdout:May 27 07:29:52 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-k[69990]: 6: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x55c5dcbf323d] 2024-05-27T07:29:52.701 INFO:journalctl@ceph.mds.k.smithi105.stdout:May 27 07:29:52 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-k[69990]: 7: (SnapRealm::build_snap_set() const+0x323) [0x55c5dcbde971] 2024-05-27T07:29:52.701 INFO:journalctl@ceph.mds.k.smithi105.stdout:May 27 07:29:52 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-k[69990]: 8: (SnapRealm::check_cache() const+0x13e) [0x55c5dcbdcfa0] 2024-05-27T07:29:52.701 INFO:journalctl@ceph.mds.k.smithi105.stdout:May 27 07:29:52 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-k[69990]: 9: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x55c5dcbdd651] 2024-05-27T07:29:52.701 INFO:journalctl@ceph.mds.k.smithi105.stdout:May 27 07:29:52 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-k[69990]: 10: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x55c5dcb72347] 2024-05-27T07:29:52.701 INFO:journalctl@ceph.mds.k.smithi105.stdout:May 27 07:29:52 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-k[69990]: 11: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x55c5dc86e87e] 2024-05-27T07:29:52.701 INFO:journalctl@ceph.mds.k.smithi105.stdout:May 27 07:29:52 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-k[69990]: 12: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x279d) [0x55c5dc873f2f] 2024-05-27T07:29:52.701 INFO:journalctl@ceph.mds.k.smithi105.stdout:May 27 07:29:52 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-k[69990]: 13: (EUpdate::replay(MDSRank*)+0x63) [0x55c5dc87e969] 2024-05-27T07:29:52.702 INFO:journalctl@ceph.mds.k.smithi105.stdout:May 27 07:29:52 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-k[69990]: 14: (MDLog::_replay_thread()+0x1f38) [0x55c5dcc1748c] 2024-05-27T07:29:52.702 INFO:journalctl@ceph.mds.k.smithi105.stdout:May 27 07:29:52 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-k[69990]: 15: (MDLog::ReplayThread::entry()+0x11) [0x55c5dcc209cb] 2024-05-27T07:29:52.702 INFO:journalctl@ceph.mds.k.smithi105.stdout:May 27 07:29:52 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-k[69990]: 16: (Thread::entry_wrapper()+0x43) [0x7f1bfce7a72d] 2024-05-27T07:29:52.702 INFO:journalctl@ceph.mds.k.smithi105.stdout:May 27 07:29:52 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-k[69990]: 17: (Thread::_entry_func(void*)+0xd) [0x7f1bfce7a749] 2024-05-27T07:29:52.702 INFO:journalctl@ceph.mds.k.smithi105.stdout:May 27 07:29:52 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-k[69990]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7f1bfba1e1ca] 2024-05-27T07:29:52.703 INFO:journalctl@ceph.mds.k.smithi105.stdout:May 27 07:29:52 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-k[69990]: 19: clone() 2024-05-27T07:29:52.703 INFO:journalctl@ceph.mds.k.smithi105.stdout:May 27 07:29:52 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-k[69990]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-05-27T07:29:52.703 INFO:journalctl@ceph.mds.k.smithi105.stdout:May 27 07:29:52 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-k[69990]: 2024-05-27T07:29:52.703 INFO:journalctl@ceph.mds.k.smithi105.stdout:May 27 07:29:52 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-k[69990]: -4> 2024-05-27T07:29:52.350+0000 7f1bea666700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: In function 'std::set SnapClient::filter(const std::set&) const' thread 7f1bea666700 time 2024-05-27T07:29:52.348263+0000 2024-05-27T07:29:52.703 INFO:journalctl@ceph.mds.k.smithi105.stdout:May 27 07:29:52 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-k[69990]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: 197: FAILED ceph_assert(cached_version > 0) 2024-05-27T07:29:52.703 INFO:journalctl@ceph.mds.k.smithi105.stdout:May 27 07:29:52 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-k[69990]: 2024-05-27T07:29:52.703 INFO:journalctl@ceph.mds.k.smithi105.stdout:May 27 07:29:52 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-k[69990]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:29:52.703 INFO:journalctl@ceph.mds.k.smithi105.stdout:May 27 07:29:52 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-k[69990]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x127) [0x7f1bfceabba6] 2024-05-27T07:29:52.703 INFO:journalctl@ceph.mds.k.smithi105.stdout:May 27 07:29:52 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-k[69990]: 2: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7f1bfceabdd1] 2024-05-27T07:29:52.703 INFO:journalctl@ceph.mds.k.smithi105.stdout:May 27 07:29:52 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-k[69990]: 3: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x55c5dcbf323d] 2024-05-27T07:29:52.703 INFO:journalctl@ceph.mds.k.smithi105.stdout:May 27 07:29:52 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-k[69990]: 4: (SnapRealm::build_snap_set() const+0x323) [0x55c5dcbde971] 2024-05-27T07:29:52.703 INFO:journalctl@ceph.mds.k.smithi105.stdout:May 27 07:29:52 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-k[69990]: 5: (SnapRealm::check_cache() const+0x13e) [0x55c5dcbdcfa0] 2024-05-27T07:29:52.704 INFO:journalctl@ceph.mds.k.smithi105.stdout:May 27 07:29:52 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-k[69990]: 6: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x55c5dcbdd651] 2024-05-27T07:29:52.704 INFO:journalctl@ceph.mds.k.smithi105.stdout:May 27 07:29:52 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-k[69990]: 7: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x55c5dcb72347] 2024-05-27T07:29:52.704 INFO:journalctl@ceph.mds.k.smithi105.stdout:May 27 07:29:52 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-k[69990]: 8: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x55c5dc86e87e] 2024-05-27T07:29:52.704 INFO:journalctl@ceph.mds.k.smithi105.stdout:May 27 07:29:52 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-k[69990]: 9: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x279d) [0x55c5dc873f2f] 2024-05-27T07:29:52.704 INFO:journalctl@ceph.mds.k.smithi105.stdout:May 27 07:29:52 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-k[69990]: 10: (EUpdate::replay(MDSRank*)+0x63) [0x55c5dc87e969] 2024-05-27T07:29:52.704 INFO:journalctl@ceph.mds.k.smithi105.stdout:May 27 07:29:52 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-k[69990]: 11: (MDLog::_replay_thread()+0x1f38) [0x55c5dcc1748c] 2024-05-27T07:29:52.704 INFO:journalctl@ceph.mds.k.smithi105.stdout:May 27 07:29:52 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-k[69990]: 12: (MDLog::ReplayThread::entry()+0x11) [0x55c5dcc209cb] 2024-05-27T07:29:52.704 INFO:journalctl@ceph.mds.k.smithi105.stdout:May 27 07:29:52 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-k[69990]: 13: (Thread::entry_wrapper()+0x43) [0x7f1bfce7a72d] 2024-05-27T07:29:52.704 INFO:journalctl@ceph.mds.k.smithi105.stdout:May 27 07:29:52 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-k[69990]: 14: (Thread::_entry_func(void*)+0xd) [0x7f1bfce7a749] 2024-05-27T07:29:52.704 INFO:journalctl@ceph.mds.k.smithi105.stdout:May 27 07:29:52 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-k[69990]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7f1bfba1e1ca] 2024-05-27T07:29:52.704 INFO:journalctl@ceph.mds.k.smithi105.stdout:May 27 07:29:52 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-k[69990]: 16: clone() 2024-05-27T07:29:52.704 INFO:journalctl@ceph.mds.k.smithi105.stdout:May 27 07:29:52 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-k[69990]: 2024-05-27T07:29:52.704 INFO:journalctl@ceph.mds.k.smithi105.stdout:May 27 07:29:52 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-k[69990]: -2> 2024-05-27T07:29:52.353+0000 7f1bea666700 -1 *** Caught signal (Aborted) ** 2024-05-27T07:29:52.705 INFO:journalctl@ceph.mds.k.smithi105.stdout:May 27 07:29:52 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-k[69990]: in thread 7f1bea666700 thread_name:md_log_replay 2024-05-27T07:29:52.705 INFO:journalctl@ceph.mds.k.smithi105.stdout:May 27 07:29:52 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-k[69990]: 2024-05-27T07:29:52.705 INFO:journalctl@ceph.mds.k.smithi105.stdout:May 27 07:29:52 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-k[69990]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:29:52.705 INFO:journalctl@ceph.mds.k.smithi105.stdout:May 27 07:29:52 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-k[69990]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f1bfba28d20] 2024-05-27T07:29:52.705 INFO:journalctl@ceph.mds.k.smithi105.stdout:May 27 07:29:52 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-k[69990]: 2: gsignal() 2024-05-27T07:29:52.705 INFO:journalctl@ceph.mds.k.smithi105.stdout:May 27 07:29:52 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-k[69990]: 3: abort() 2024-05-27T07:29:52.705 INFO:journalctl@ceph.mds.k.smithi105.stdout:May 27 07:29:52 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-k[69990]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x26e) [0x7f1bfceabced] 2024-05-27T07:29:52.705 INFO:journalctl@ceph.mds.k.smithi105.stdout:May 27 07:29:52 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-k[69990]: 5: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7f1bfceabdd1] 2024-05-27T07:29:52.705 INFO:journalctl@ceph.mds.k.smithi105.stdout:May 27 07:29:52 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-k[69990]: 6: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x55c5dcbf323d] 2024-05-27T07:29:52.705 INFO:journalctl@ceph.mds.k.smithi105.stdout:May 27 07:29:52 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-k[69990]: 7: (SnapRealm::build_snap_set() const+0x323) [0x55c5dcbde971] 2024-05-27T07:29:52.705 INFO:journalctl@ceph.mds.k.smithi105.stdout:May 27 07:29:52 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-k[69990]: 8: (SnapRealm::check_cache() const+0x13e) [0x55c5dcbdcfa0] 2024-05-27T07:29:52.705 INFO:journalctl@ceph.mds.k.smithi105.stdout:May 27 07:29:52 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-k[69990]: 9: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x55c5dcbdd651] 2024-05-27T07:29:52.705 INFO:journalctl@ceph.mds.k.smithi105.stdout:May 27 07:29:52 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-k[69990]: 10: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x55c5dcb72347] 2024-05-27T07:29:52.705 INFO:journalctl@ceph.mds.k.smithi105.stdout:May 27 07:29:52 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-k[69990]: 11: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x55c5dc86e87e] 2024-05-27T07:29:52.706 INFO:journalctl@ceph.mds.k.smithi105.stdout:May 27 07:29:52 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-k[69990]: 12: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x279d) [0x55c5dc873f2f] 2024-05-27T07:29:52.706 INFO:journalctl@ceph.mds.k.smithi105.stdout:May 27 07:29:52 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-k[69990]: 13: (EUpdate::replay(MDSRank*)+0x63) [0x55c5dc87e969] 2024-05-27T07:29:52.706 INFO:journalctl@ceph.mds.k.smithi105.stdout:May 27 07:29:52 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-k[69990]: 14: (MDLog::_replay_thread()+0x1f38) [0x55c5dcc1748c] 2024-05-27T07:29:52.706 INFO:journalctl@ceph.mds.k.smithi105.stdout:May 27 07:29:52 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-k[69990]: 15: (MDLog::ReplayThread::entry()+0x11) [0x55c5dcc209cb] 2024-05-27T07:29:52.706 INFO:journalctl@ceph.mds.k.smithi105.stdout:May 27 07:29:52 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-k[69990]: 16: (Thread::entry_wrapper()+0x43) [0x7f1bfce7a72d] 2024-05-27T07:29:52.706 INFO:journalctl@ceph.mds.k.smithi105.stdout:May 27 07:29:52 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-k[69990]: 17: (Thread::_entry_func(void*)+0xd) [0x7f1bfce7a749] 2024-05-27T07:29:52.706 INFO:journalctl@ceph.mds.k.smithi105.stdout:May 27 07:29:52 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-k[69990]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7f1bfba1e1ca] 2024-05-27T07:29:52.706 INFO:journalctl@ceph.mds.k.smithi105.stdout:May 27 07:29:52 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-k[69990]: 19: clone() 2024-05-27T07:29:52.706 INFO:journalctl@ceph.mds.k.smithi105.stdout:May 27 07:29:52 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-k[69990]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-05-27T07:29:52.706 INFO:journalctl@ceph.mds.k.smithi105.stdout:May 27 07:29:52 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-k[69990]: 2024-05-27T07:29:52.706 INFO:journalctl@ceph.mds.k.smithi105.stdout:May 27 07:29:52 smithi105 podman[70025]: 2024-05-27 07:29:52.395967303 +0000 UTC m=+0.021227310 container died 65fd1707409f955d437095b2ebc4b654df22761089d5fcee3ce3884251d520be (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-k, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.build-date=20240521, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, GIT_CLEAN=True, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS) 2024-05-27T07:29:53.197 INFO:journalctl@ceph.mds.d.smithi052.stdout:May 27 07:29:52 smithi052 podman[106485]: 2024-05-27 07:29:52.935844261 +0000 UTC m=+1.977813651 container died 377781175b2cb23f4f9fbb362b5b24d7361ffdd72d17fa2e26d16676442aa5c2 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-d, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, org.label-schema.build-date=20240521, GIT_BRANCH=HEAD, ceph=True, org.label-schema.schema-version=1.0, GIT_CLEAN=True, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, org.label-schema.name=CentOS Stream 8 Base Image) 2024-05-27T07:29:53.477 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:29:53.484+0000 7fa94effd640 1 -- 172.21.15.52:0/1820869022 <== mds.0 v2:172.21.15.105:6838/500348151 1 ==== command_reply(tid 0: 0 ) ==== 8+0+25 (crc 0 0 0) 0x55ad8cd35360 con 0x55ad8ca261b0 2024-05-27T07:29:53.477 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:29:53.484+0000 7fa94effd640 10 client.26024 handle_command_reply: tid=0 2024-05-27T07:29:53.477 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:29:53.484+0000 7fa97cbea740 1 client.26024 shutdown 2024-05-27T07:29:53.478 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:29:53.485+0000 7fa97cbea740 1 -- 172.21.15.52:0/1820869022 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55ad8c9e4d80 msgr2=0x55ad8cdc31a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:29:53.478 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:29:53.485+0000 7fa97cbea740 1 --2- 172.21.15.52:0/1820869022 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55ad8c9e4d80 0x55ad8cdc31a0 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7fa960004880 tx=0x7fa960009870 comp rx=0 tx=0).stop 2024-05-27T07:29:53.478 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:29:53.485+0000 7fa97cbea740 1 -- 172.21.15.52:0/1820869022 shutdown_connections 2024-05-27T07:29:53.478 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:29:53.485+0000 7fa97cbea740 1 --2- 172.21.15.52:0/1820869022 >> [v2:172.21.15.105:6838/500348151,v1:172.21.15.105:6839/500348151] conn(0x55ad8ca261b0 0x55ad8ce4df70 crc :-1 s=READY pgs=98 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:29:53.478 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:29:53.485+0000 7fa97cbea740 1 --2- 172.21.15.52:0/1820869022 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55ad8ca054d0 0x55ad8cdc3c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:29:53.478 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:29:53.485+0000 7fa97cbea740 1 --2- 172.21.15.52:0/1820869022 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55ad8c9fd2f0 0x55ad8cdc36e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:29:53.478 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:29:53.485+0000 7fa97cbea740 1 --2- 172.21.15.52:0/1820869022 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55ad8c9e4d80 0x55ad8cdc31a0 unknown :-1 s=CLOSED pgs=136 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:29:53.478 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:29:53.485+0000 7fa97cbea740 1 -- 172.21.15.52:0/1820869022 >> 172.21.15.52:0/1820869022 conn(0x55ad8ca1a300 msgr2=0x55ad8cdd2d80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:29:53.478 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:29:53.485+0000 7fa94effd640 0 client.26024 ms_handle_reset on v2:172.21.15.105:6838/500348151 2024-05-27T07:29:53.478 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:29:53.485+0000 7fa97cbea740 1 -- 172.21.15.52:0/1820869022 shutdown_connections 2024-05-27T07:29:53.478 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:29:53.485+0000 7fa97cbea740 1 -- 172.21.15.52:0/1820869022 wait complete. 2024-05-27T07:29:53.478 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:29:53.485+0000 7fa97cbea740 20 client.26024 trim_cache size 0 max 16384 2024-05-27T07:29:53.479 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stdout:{ 2024-05-27T07:29:53.479 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stdout: "return_code": 0 2024-05-27T07:29:53.479 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stdout:} 2024-05-27T07:29:53.480 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:29:53 smithi052 ceph-mon[31955]: osdmap e110: 12 total, 12 up, 12 in 2024-05-27T07:29:53.480 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:29:53 smithi052 ceph-mon[31955]: mds.? [v2:172.21.15.105:6832/789451115,v1:172.21.15.105:6833/789451115] up:boot 2024-05-27T07:29:53.480 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:29:53 smithi052 ceph-mon[31955]: fsmap cephfs:5 4 up:standby-replay 2 up:standby 5 up:active 2024-05-27T07:29:53.480 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:29:53 smithi052 ceph-mon[31955]: from='mgr.15333 172.21.15.52:0/416389082' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "k"}]: dispatch 2024-05-27T07:29:53.480 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:29:53 smithi052 ceph-mon[31955]: fsmap cephfs:5 5 up:standby-replay 1 up:standby 5 up:active 2024-05-27T07:29:53.495 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:29:53.503+0000 7fa975c5a640 1 -- 172.21.15.52:0/1671073841 >> [v2:172.21.15.52:6800/1822318384,v1:172.21.15.52:6801/1822318384] conn(0x7fa95c0b2950 msgr2=0x7fa95c0b4e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:29:53.495 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:29:53.503+0000 7fa975c5a640 1 --2- 172.21.15.52:0/1671073841 >> [v2:172.21.15.52:6800/1822318384,v1:172.21.15.52:6801/1822318384] conn(0x7fa95c0b2950 0x7fa95c0b4e60 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x55ad8cdc4540 tx=0x7fa9680096a0 comp rx=0 tx=0).stop 2024-05-27T07:29:53.495 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:29:53.503+0000 7fa975c5a640 1 -- 172.21.15.52:0/1671073841 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fa970109cb0 msgr2=0x7fa970075500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:29:53.495 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:29:53.503+0000 7fa975c5a640 1 --2- 172.21.15.52:0/1671073841 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fa970109cb0 0x7fa970075500 secure :-1 s=READY pgs=619 cs=0 l=1 rev1=1 crypto rx=0x7fa958006010 tx=0x7fa958004060 comp rx=0 tx=0).stop 2024-05-27T07:29:53.496 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:29:53.504+0000 7fa975c5a640 1 -- 172.21.15.52:0/1671073841 shutdown_connections 2024-05-27T07:29:53.496 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:29:53.504+0000 7fa975c5a640 1 --2- 172.21.15.52:0/1671073841 >> [v2:172.21.15.52:6800/1822318384,v1:172.21.15.52:6801/1822318384] conn(0x7fa95c0b2950 0x7fa95c0b4e60 unknown :-1 s=CLOSED pgs=119 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:29:53.496 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:29:53.504+0000 7fa975c5a640 1 --2- 172.21.15.52:0/1671073841 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fa97010a150 0x7fa970075a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:29:53.496 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:29:53.504+0000 7fa975c5a640 1 --2- 172.21.15.52:0/1671073841 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fa970109cb0 0x7fa970075500 unknown :-1 s=CLOSED pgs=619 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:29:53.496 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:29:53.504+0000 7fa975c5a640 1 --2- 172.21.15.52:0/1671073841 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fa970104400 0x7fa970074fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:29:53.496 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:29:53.504+0000 7fa975c5a640 1 -- 172.21.15.52:0/1671073841 >> 172.21.15.52:0/1671073841 conn(0x7fa970079310 msgr2=0x7fa970102450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:29:53.498 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:29:53.506+0000 7fa975c5a640 1 -- 172.21.15.52:0/1671073841 shutdown_connections 2024-05-27T07:29:53.498 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:29:53.506+0000 7fa975c5a640 1 -- 172.21.15.52:0/1671073841 wait complete. 2024-05-27T07:29:53.602 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:29:53 smithi173 ceph-mon[37000]: osdmap e110: 12 total, 12 up, 12 in 2024-05-27T07:29:53.603 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:29:53 smithi173 ceph-mon[37000]: mds.? [v2:172.21.15.105:6832/789451115,v1:172.21.15.105:6833/789451115] up:boot 2024-05-27T07:29:53.603 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:29:53 smithi173 ceph-mon[37000]: fsmap cephfs:5 4 up:standby-replay 2 up:standby 5 up:active 2024-05-27T07:29:53.603 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:29:53 smithi173 ceph-mon[37000]: from='mgr.15333 172.21.15.52:0/416389082' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "k"}]: dispatch 2024-05-27T07:29:53.603 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:29:53 smithi173 ceph-mon[37000]: fsmap cephfs:5 5 up:standby-replay 1 up:standby 5 up:active 2024-05-27T07:29:53.690 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:29:53 smithi105 ceph-mon[35977]: osdmap e110: 12 total, 12 up, 12 in 2024-05-27T07:29:53.691 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:29:53 smithi105 ceph-mon[35977]: mds.? [v2:172.21.15.105:6832/789451115,v1:172.21.15.105:6833/789451115] up:boot 2024-05-27T07:29:53.691 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:29:53 smithi105 ceph-mon[35977]: fsmap cephfs:5 4 up:standby-replay 2 up:standby 5 up:active 2024-05-27T07:29:53.691 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:29:53 smithi105 ceph-mon[35977]: from='mgr.15333 172.21.15.52:0/416389082' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "k"}]: dispatch 2024-05-27T07:29:53.691 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:29:53 smithi105 ceph-mon[35977]: fsmap cephfs:5 5 up:standby-replay 1 up:standby 5 up:active 2024-05-27T07:29:53.936 INFO:journalctl@ceph.mds.d.smithi052.stdout:May 27 07:29:53 smithi052 podman[106485]: 2024-05-27 07:29:53.627924937 +0000 UTC m=+2.669894325 container remove 377781175b2cb23f4f9fbb362b5b24d7361ffdd72d17fa2e26d16676442aa5c2 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-d, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240521, GIT_CLEAN=True, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, ceph=True, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0) 2024-05-27T07:29:53.936 INFO:journalctl@ceph.mds.d.smithi052.stdout:May 27 07:29:53 smithi052 systemd[1]: ceph-73df2636-1bf8-11ef-bc9b-c7b262605968@mds.d.service: Main process exited, code=exited, status=134/n/a 2024-05-27T07:29:54.415 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:29:54.422+0000 7f2071701700 1 -- 172.21.15.52:0/2925000309 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f206c108760 msgr2=0x7f206c1100c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:29:54.415 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:29:54.422+0000 7f2071701700 1 --2- 172.21.15.52:0/2925000309 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f206c108760 0x7f206c1100c0 secure :-1 s=READY pgs=627 cs=0 l=1 rev1=1 crypto rx=0x7f206400b0d0 tx=0x7f206400b3e0 comp rx=0 tx=0).stop 2024-05-27T07:29:54.415 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:29:54.422+0000 7f2071701700 1 -- 172.21.15.52:0/2925000309 shutdown_connections 2024-05-27T07:29:54.415 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:29:54.422+0000 7f2071701700 1 --2- 172.21.15.52:0/2925000309 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f206c108760 0x7f206c1100c0 unknown :-1 s=CLOSED pgs=627 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:29:54.415 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:29:54.422+0000 7f2071701700 1 --2- 172.21.15.52:0/2925000309 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f206c103c10 0x7f206c104020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:29:54.415 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:29:54.422+0000 7f2071701700 1 --2- 172.21.15.52:0/2925000309 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f206c107e10 0x7f206c108220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:29:54.416 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:29:54.422+0000 7f2071701700 1 -- 172.21.15.52:0/2925000309 >> 172.21.15.52:0/2925000309 conn(0x7f206c0fe6e0 msgr2=0x7f206c100b00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:29:54.416 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:29:54.423+0000 7f2071701700 1 -- 172.21.15.52:0/2925000309 shutdown_connections 2024-05-27T07:29:54.416 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:29:54.423+0000 7f2071701700 1 -- 172.21.15.52:0/2925000309 wait complete. 2024-05-27T07:29:54.416 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:29:54.423+0000 7f2071701700 1 Processor -- start 2024-05-27T07:29:54.418 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:29:54.424+0000 7f2071701700 1 -- start start 2024-05-27T07:29:54.418 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:29:54.424+0000 7f2071701700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f206c103c10 0x7f206c19ae40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:29:54.418 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:29:54.424+0000 7f2071701700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f206c107e10 0x7f206c19b3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:29:54.418 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:29:54.424+0000 7f2071701700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f206c108760 0x7f206c19f670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:29:54.418 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:29:54.424+0000 7f2071701700 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x7f206c113400 con 0x7f206c107e10 2024-05-27T07:29:54.418 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:29:54.425+0000 7f2071701700 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f206c113280 con 0x7f206c108760 2024-05-27T07:29:54.418 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:29:54.425+0000 7f206a7fc700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f206c107e10 0x7f206c19b3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:29:54.418 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:29:54.425+0000 7f206b7fe700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f206c108760 0x7f206c19f670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:29:54.419 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:29:54.425+0000 7f206affd700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f206c103c10 0x7f206c19ae40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:29:54.419 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:29:54.425+0000 7f206b7fe700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f206c108760 0x7f206c19f670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.52:34956/0 (socket says 172.21.15.52:34956) 2024-05-27T07:29:54.419 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:29:54.425+0000 7f206b7fe700 1 -- 172.21.15.52:0/3429381078 learned_addr learned my addr 172.21.15.52:0/3429381078 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-05-27T07:29:54.419 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:29:54.425+0000 7f2071701700 1 -- 172.21.15.52:0/3429381078 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f206c113580 con 0x7f206c103c10 2024-05-27T07:29:54.419 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:29:54.425+0000 7f206affd700 1 -- 172.21.15.52:0/3429381078 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f206c108760 msgr2=0x7f206c19f670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:29:54.419 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:29:54.425+0000 7f206affd700 1 --2- 172.21.15.52:0/3429381078 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f206c108760 0x7f206c19f670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:29:54.419 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:29:54.425+0000 7f206affd700 1 -- 172.21.15.52:0/3429381078 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f206c107e10 msgr2=0x7f206c19b3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:29:54.419 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:29:54.425+0000 7f206affd700 1 --2- 172.21.15.52:0/3429381078 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f206c107e10 0x7f206c19b3e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:29:54.419 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:29:54.425+0000 7f206affd700 1 -- 172.21.15.52:0/3429381078 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f206c19fd10 con 0x7f206c103c10 2024-05-27T07:29:54.419 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:29:54.425+0000 7f206affd700 1 --2- 172.21.15.52:0/3429381078 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f206c103c10 0x7f206c19ae40 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7f205c00e9c0 tx=0x7f205c00ecd0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:29:54.419 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:29:54.426+0000 7f2053fff700 1 -- 172.21.15.52:0/3429381078 <== mon.1 v2:172.21.15.173:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f205c00c9c0 con 0x7f206c103c10 2024-05-27T07:29:54.419 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:29:54.426+0000 7f2071701700 1 -- 172.21.15.52:0/3429381078 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f206c1a0000 con 0x7f206c103c10 2024-05-27T07:29:54.419 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:29:54.426+0000 7f2071701700 1 -- 172.21.15.52:0/3429381078 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f206c10a090 con 0x7f206c103c10 2024-05-27T07:29:54.420 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:29:54.426+0000 7f2053fff700 1 -- 172.21.15.52:0/3429381078 <== mon.1 v2:172.21.15.173:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f205c00cb60 con 0x7f206c103c10 2024-05-27T07:29:54.420 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:29:54.426+0000 7f2053fff700 1 -- 172.21.15.52:0/3429381078 <== mon.1 v2:172.21.15.173:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f205c018680 con 0x7f206c103c10 2024-05-27T07:29:54.420 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:29:54.429+0000 7f2053fff700 1 -- 172.21.15.52:0/3429381078 <== mon.1 v2:172.21.15.173:3300/0 4 ==== mgrmap(e 26) ==== 149783+0+0 (secure 0 0 0) 0x7f205c018820 con 0x7f206c103c10 2024-05-27T07:29:54.421 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:29:54.429+0000 7f2053fff700 1 --2- 172.21.15.52:0/3429381078 >> [v2:172.21.15.52:6800/1822318384,v1:172.21.15.52:6801/1822318384] conn(0x7f20540b29f0 0x7f20540b4f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:29:54.422 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:29:54.430+0000 7f206a7fc700 1 --2- 172.21.15.52:0/3429381078 >> [v2:172.21.15.52:6800/1822318384,v1:172.21.15.52:6801/1822318384] conn(0x7f20540b29f0 0x7f20540b4f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:29:54.422 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:29:54.430+0000 7f2053fff700 1 -- 172.21.15.52:0/3429381078 <== mon.1 v2:172.21.15.173:3300/0 5 ==== osd_map(110..110 src has 1..110) ==== 9379+0+0 (secure 0 0 0) 0x7f205c0e2ae0 con 0x7f206c103c10 2024-05-27T07:29:54.422 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:29:54.430+0000 7f2080116b80 1 Processor -- start 2024-05-27T07:29:54.422 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:29:54.430+0000 7f2080116b80 1 -- start start 2024-05-27T07:29:54.422 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:29:54.430+0000 7f206a7fc700 1 --2- 172.21.15.52:0/3429381078 >> [v2:172.21.15.52:6800/1822318384,v1:172.21.15.52:6801/1822318384] conn(0x7f20540b29f0 0x7f20540b4f00 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7f206c109860 tx=0x7f2060009260 comp rx=0 tx=0).ready entity=mgr.15333 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:29:54.422 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:29:54.430+0000 7f2080116b80 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55718caf8e40 0x55718cc7db30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:29:54.422 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:29:54.430+0000 7f2080116b80 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55718cc7e070 0x55718cc77be0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:29:54.423 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:29:54.430+0000 7f2080116b80 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55718cc78120 0x55718cc78530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:29:54.423 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:29:54.430+0000 7f2080116b80 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x55718cba5d70 con 0x55718cc78120 2024-05-27T07:29:54.423 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:29:54.430+0000 7f2080116b80 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x55718cb95910 con 0x55718cc7e070 2024-05-27T07:29:54.423 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:29:54.430+0000 7f2080116b80 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x55718cb99950 con 0x55718caf8e40 2024-05-27T07:29:54.423 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:29:54.430+0000 7f206b7fe700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55718cc78120 0x55718cc78530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:29:54.424 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:29:54.430+0000 7f206a7fc700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55718cc7e070 0x55718cc77be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:29:54.424 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:29:54.430+0000 7f206a7fc700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55718cc7e070 0x55718cc77be0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.52:34958/0 (socket says 172.21.15.52:34958) 2024-05-27T07:29:54.425 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:29:54.430+0000 7f206a7fc700 1 -- 172.21.15.52:0/1605989328 learned_addr learned my addr 172.21.15.52:0/1605989328 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-05-27T07:29:54.425 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:29:54.430+0000 7f206b7fe700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55718cc78120 0x55718cc78530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.52:39606/0 (socket says 172.21.15.52:39606) 2024-05-27T07:29:54.425 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:29:54.431+0000 7f206affd700 1 --2- 172.21.15.52:0/1605989328 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55718caf8e40 0x55718cc7db30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:29:54.425 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:29:54.431+0000 7f206b7fe700 1 -- 172.21.15.52:0/1605989328 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55718caf8e40 msgr2=0x55718cc7db30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:29:54.425 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:29:54.431+0000 7f206b7fe700 1 --2- 172.21.15.52:0/1605989328 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55718caf8e40 0x55718cc7db30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:29:54.425 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:29:54.431+0000 7f206b7fe700 1 -- 172.21.15.52:0/1605989328 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55718cc7e070 msgr2=0x55718cc77be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:29:54.425 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:29:54.431+0000 7f206b7fe700 1 --2- 172.21.15.52:0/1605989328 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55718cc7e070 0x55718cc77be0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:29:54.425 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:29:54.431+0000 7f206b7fe700 1 -- 172.21.15.52:0/1605989328 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55718cb3efb0 con 0x55718cc78120 2024-05-27T07:29:54.425 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:29:54.431+0000 7f206a7fc700 1 --2- 172.21.15.52:0/1605989328 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55718cc7e070 0x55718cc77be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-27T07:29:54.425 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:29:54.431+0000 7f206b7fe700 1 --2- 172.21.15.52:0/1605989328 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55718cc78120 0x55718cc78530 secure :-1 s=READY pgs=628 cs=0 l=1 rev1=1 crypto rx=0x7f2064009d10 tx=0x7f206400b040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=36c21201a350b466 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:29:54.425 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:29:54.431+0000 7f20517fa700 1 -- 172.21.15.52:0/1605989328 <== mon.0 v2:172.21.15.52:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f206402b070 con 0x55718cc78120 2024-05-27T07:29:54.425 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:29:54.431+0000 7f20517fa700 1 -- 172.21.15.52:0/1605989328 <== mon.0 v2:172.21.15.52:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f2064003a00 con 0x55718cc78120 2024-05-27T07:29:54.426 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:29:54.431+0000 7f20517fa700 1 -- 172.21.15.52:0/1605989328 <== mon.0 v2:172.21.15.52:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f206401cb20 con 0x55718cc78120 2024-05-27T07:29:54.426 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:29:54.432+0000 7f2080116b80 1 -- 172.21.15.52:0/1605989328 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55718cc78120 msgr2=0x55718cc78530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:29:54.426 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:29:54.432+0000 7f2080116b80 1 --2- 172.21.15.52:0/1605989328 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55718cc78120 0x55718cc78530 secure :-1 s=READY pgs=628 cs=0 l=1 rev1=1 crypto rx=0x7f2064009d10 tx=0x7f206400b040 comp rx=0 tx=0).stop 2024-05-27T07:29:54.426 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:29:54.432+0000 7f2080116b80 1 -- 172.21.15.52:0/1605989328 shutdown_connections 2024-05-27T07:29:54.426 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:29:54.432+0000 7f2080116b80 1 --2- 172.21.15.52:0/1605989328 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55718cc78120 0x55718cc78530 unknown :-1 s=CLOSED pgs=628 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:29:54.426 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:29:54.432+0000 7f2080116b80 1 --2- 172.21.15.52:0/1605989328 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55718caf8e40 0x55718cc7db30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:29:54.426 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:29:54.432+0000 7f2080116b80 1 --2- 172.21.15.52:0/1605989328 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55718cc7e070 0x55718cc77be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:29:54.426 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:29:54.432+0000 7f2080116b80 1 -- 172.21.15.52:0/1605989328 >> 172.21.15.52:0/1605989328 conn(0x55718cafb770 msgr2=0x55718cc72d90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:29:54.426 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:29:54.432+0000 7f2080116b80 1 -- 172.21.15.52:0/1605989328 shutdown_connections 2024-05-27T07:29:54.426 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:29:54.432+0000 7f2080116b80 1 -- 172.21.15.52:0/1605989328 wait complete. 2024-05-27T07:29:54.426 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:29:54.433+0000 7f2080116b80 1 -- start start 2024-05-27T07:29:54.427 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:29:54.433+0000 7f2080116b80 1 Processor -- start 2024-05-27T07:29:54.430 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:29:54.433+0000 7f2080116b80 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55718caf8e40 0x55718cc81890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:29:54.430 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:29:54.433+0000 7f2080116b80 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55718cc78120 0x55718cc81dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:29:54.430 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:29:54.434+0000 7f206a7fc700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55718cc78120 0x55718cc81dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:29:54.431 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:29:54.434+0000 7f2080116b80 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55718cc7e070 0x55718cc82310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:29:54.431 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:29:54.434+0000 7f2080116b80 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x55718ca9bfa0 con 0x55718cc78120 2024-05-27T07:29:54.431 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:29:54.434+0000 7f2080116b80 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x55718cafdda0 con 0x55718caf8e40 2024-05-27T07:29:54.431 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:29:54.434+0000 7f2080116b80 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x55718cc84b00 con 0x55718cc7e070 2024-05-27T07:29:54.431 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:29:54.434+0000 7f206affd700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55718caf8e40 0x55718cc81890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:29:54.431 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:29:54.434+0000 7f2080116b80 10 client.0 _ll_register_callbacks cb 0x55718cb2c5b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T07:29:54.431 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:29:54.434+0000 7f206a7fc700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55718cc78120 0x55718cc81dd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.52:39618/0 (socket says 172.21.15.52:39618) 2024-05-27T07:29:54.431 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:29:54.434+0000 7f206a7fc700 1 -- 172.21.15.52:0/2917160979 learned_addr learned my addr 172.21.15.52:0/2917160979 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-05-27T07:29:54.431 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:29:54.434+0000 7f206b7fe700 1 --2- 172.21.15.52:0/2917160979 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55718cc7e070 0x55718cc82310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:29:54.431 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:29:54.434+0000 7f206a7fc700 1 -- 172.21.15.52:0/2917160979 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55718cc7e070 msgr2=0x55718cc82310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:29:54.431 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:29:54.434+0000 7f206a7fc700 1 --2- 172.21.15.52:0/2917160979 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55718cc7e070 0x55718cc82310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:29:54.431 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:29:54.434+0000 7f206a7fc700 1 -- 172.21.15.52:0/2917160979 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55718caf8e40 msgr2=0x55718cc81890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:29:54.432 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:29:54.434+0000 7f206a7fc700 1 --2- 172.21.15.52:0/2917160979 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55718caf8e40 0x55718cc81890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:29:54.432 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:29:54.434+0000 7f206a7fc700 1 -- 172.21.15.52:0/2917160979 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55718cae58d0 con 0x55718cc78120 2024-05-27T07:29:54.432 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:29:54.435+0000 7f206a7fc700 1 --2- 172.21.15.52:0/2917160979 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55718cc78120 0x55718cc81dd0 secure :-1 s=READY pgs=629 cs=0 l=1 rev1=1 crypto rx=0x7f206001f910 tx=0x7f20600047c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d65f2fa8aceaae3c server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:29:54.432 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:29:54.435+0000 7f204eff5700 10 client.0 ms_handle_connect on v2:172.21.15.52:3300/0 2024-05-27T07:29:54.432 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:29:54.435+0000 7f204eff5700 1 -- 172.21.15.52:0/2917160979 <== mon.0 v2:172.21.15.52:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f206002c060 con 0x55718cc78120 2024-05-27T07:29:54.432 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:29:54.435+0000 7f204eff5700 1 -- 172.21.15.52:0/2917160979 <== mon.0 v2:172.21.15.52:3300/0 2 ==== osd_map(110..110 src has 1..110) ==== 9379+0+0 (secure 0 0 0) 0x7f2060009040 con 0x55718cc78120 2024-05-27T07:29:54.432 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:29:54.435+0000 7f2080116b80 1 -- 172.21.15.52:0/2917160979 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f2064003a00 con 0x55718cc78120 2024-05-27T07:29:54.432 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:29:54.435+0000 7f204eff5700 1 -- 172.21.15.52:0/2917160979 <== mon.0 v2:172.21.15.52:3300/0 3 ==== mon_get_version_reply(handle=1 version=73) ==== 24+0+0 (secure 0 0 0) 0x7f2060004020 con 0x55718cc78120 2024-05-27T07:29:54.432 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:29:54.435+0000 7f2080116b80 10 client.16404 fetch_fsmap learned FSMap version 73 2024-05-27T07:29:54.432 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:29:54.435+0000 7f2080116b80 1 -- 172.21.15.52:0/2917160979 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({config=0+,fsmap=73,monmap=4+}) -- 0x55718cb4f760 con 0x55718cc78120 2024-05-27T07:29:54.432 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:29:54.436+0000 7f204eff5700 1 -- 172.21.15.52:0/2917160979 <== mon.0 v2:172.21.15.52:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f206001d740 con 0x55718cc78120 2024-05-27T07:29:54.432 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:29:54.436+0000 7f204eff5700 1 -- 172.21.15.52:0/2917160979 <== mon.0 v2:172.21.15.52:3300/0 5 ==== fsmap(e 73) ==== 7773+0+0 (secure 0 0 0) 0x7f2060003860 con 0x55718cc78120 2024-05-27T07:29:54.432 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:29:54.436+0000 7f2080116b80 10 client.16404 fetch_fsmap finished waiting for FSMap version 73 2024-05-27T07:29:54.433 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:29:54.436+0000 7f2080116b80 10 client.16404 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.h 2024-05-27T07:29:54.433 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:29:54.436+0000 7f2080116b80 20 client.16404 populate_metadata read hostname 'smithi052' 2024-05-27T07:29:54.433 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:29:54.436+0000 7f2080116b80 1 --2- 172.21.15.52:0/2917160979 >> [v2:172.21.15.105:6836/191393149,v1:172.21.15.105:6837/191393149] conn(0x55718cc91af0 0x55718cc93f10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:29:54.433 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:29:54.436+0000 7f2080116b80 4 client.16404 mds_command: new command op to 24497 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-27T07:29:54.433 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:29:54.436+0000 7f2080116b80 1 -- 172.21.15.52:0/2917160979 --> [v2:172.21.15.105:6836/191393149,v1:172.21.15.105:6837/191393149] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x7f2064003a00 con 0x55718cc91af0 2024-05-27T07:29:54.433 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:29:54.436+0000 7f206affd700 1 --2- 172.21.15.52:0/2917160979 >> [v2:172.21.15.105:6836/191393149,v1:172.21.15.105:6837/191393149] conn(0x55718cc91af0 0x55718cc93f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:29:54.433 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:29:54.437+0000 7f206affd700 1 --2- 172.21.15.52:0/2917160979 >> [v2:172.21.15.105:6836/191393149,v1:172.21.15.105:6837/191393149] conn(0x55718cc91af0 0x55718cc93f10 crc :-1 s=READY pgs=43 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=3c03f253de25c924 server_cookie=d326680f305d2a69 in_seq=0 out_seq=0 2024-05-27T07:29:54.433 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:29:54.437+0000 7f204eff5700 10 client.16404 ms_handle_connect on v2:172.21.15.105:6836/191393149 2024-05-27T07:29:54.433 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:29:54.437+0000 7f204eff5700 1 -- 172.21.15.52:0/2917160979 <== mds.1 v2:172.21.15.105:6836/191393149 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24041 (crc 0 0 0) 0x7f2064003a00 con 0x55718cc91af0 2024-05-27T07:29:54.433 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:29:54.437+0000 7f204eff5700 10 client.16404 handle_command_reply: tid=0 2024-05-27T07:29:54.433 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:29:54.437+0000 7f2080116b80 1 client.16404 shutdown 2024-05-27T07:29:54.433 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:29:54.438+0000 7f2080116b80 1 -- 172.21.15.52:0/2917160979 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55718cc78120 msgr2=0x55718cc81dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:29:54.434 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:29:54.438+0000 7f2080116b80 1 --2- 172.21.15.52:0/2917160979 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55718cc78120 0x55718cc81dd0 secure :-1 s=READY pgs=629 cs=0 l=1 rev1=1 crypto rx=0x7f206001f910 tx=0x7f20600047c0 comp rx=0 tx=0).stop 2024-05-27T07:29:54.434 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:29:54.438+0000 7f2080116b80 1 -- 172.21.15.52:0/2917160979 shutdown_connections 2024-05-27T07:29:54.434 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:29:54.438+0000 7f2080116b80 1 --2- 172.21.15.52:0/2917160979 >> [v2:172.21.15.105:6836/191393149,v1:172.21.15.105:6837/191393149] conn(0x55718cc91af0 0x55718cc93f10 crc :-1 s=READY pgs=43 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:29:54.434 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:29:54.438+0000 7f2080116b80 1 --2- 172.21.15.52:0/2917160979 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55718cc7e070 0x55718cc82310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:29:54.434 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:29:54.438+0000 7f2080116b80 1 --2- 172.21.15.52:0/2917160979 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55718caf8e40 0x55718cc81890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:29:54.434 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:29:54.438+0000 7f2080116b80 1 --2- 172.21.15.52:0/2917160979 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55718cc78120 0x55718cc81dd0 unknown :-1 s=CLOSED pgs=629 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:29:54.434 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:29:54.438+0000 7f204eff5700 0 client.16404 ms_handle_reset on v2:172.21.15.105:6836/191393149 2024-05-27T07:29:54.434 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:29:54.438+0000 7f2080116b80 1 -- 172.21.15.52:0/2917160979 >> 172.21.15.52:0/2917160979 conn(0x55718cafb770 msgr2=0x55718cc76bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:29:54.434 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:29:54.438+0000 7f2080116b80 1 -- 172.21.15.52:0/2917160979 shutdown_connections 2024-05-27T07:29:54.434 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:29:54.438+0000 7f2080116b80 1 -- 172.21.15.52:0/2917160979 wait complete. 2024-05-27T07:29:54.434 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:29:54.438+0000 7f2080116b80 20 client.16404 trim_cache size 0 max 16384 2024-05-27T07:29:54.447 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:29:54.455+0000 7f2080116b80 1 Processor -- start 2024-05-27T07:29:54.447 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:29:54.455+0000 7f2080116b80 1 -- start start 2024-05-27T07:29:54.447 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:29:54.455+0000 7f2080116b80 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55718cafb770 0x55718cc77be0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:29:54.447 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:29:54.455+0000 7f2080116b80 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55718caf8e40 0x55718cc807f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:29:54.447 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:29:54.455+0000 7f2080116b80 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55718cc78120 0x55718cc6cc00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:29:54.447 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:29:54.455+0000 7f2080116b80 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x7f206c1afde0 con 0x55718cc78120 2024-05-27T07:29:54.447 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:29:54.455+0000 7f2080116b80 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f206c1adbd0 con 0x55718caf8e40 2024-05-27T07:29:54.447 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:29:54.455+0000 7f2080116b80 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f206c1af710 con 0x55718cafb770 2024-05-27T07:29:54.447 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:29:54.455+0000 7f206a7fc700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55718caf8e40 0x55718cc807f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:29:54.447 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:29:54.455+0000 7f206a7fc700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55718caf8e40 0x55718cc807f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.52:34970/0 (socket says 172.21.15.52:34970) 2024-05-27T07:29:54.448 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:29:54.455+0000 7f206b7fe700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55718cc78120 0x55718cc6cc00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:29:54.448 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:29:54.455+0000 7f206a7fc700 1 -- 172.21.15.52:0/3027238271 learned_addr learned my addr 172.21.15.52:0/3027238271 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-05-27T07:29:54.448 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:29:54.455+0000 7f206affd700 1 --2- 172.21.15.52:0/3027238271 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55718cafb770 0x55718cc77be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:29:54.449 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:29:54.455+0000 7f206b7fe700 1 -- 172.21.15.52:0/3027238271 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55718cafb770 msgr2=0x55718cc77be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:29:54.449 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:29:54.455+0000 7f206b7fe700 1 --2- 172.21.15.52:0/3027238271 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55718cafb770 0x55718cc77be0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:29:54.449 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:29:54.455+0000 7f206b7fe700 1 -- 172.21.15.52:0/3027238271 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55718caf8e40 msgr2=0x55718cc807f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:29:54.449 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:29:54.456+0000 7f206b7fe700 1 --2- 172.21.15.52:0/3027238271 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55718caf8e40 0x55718cc807f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:29:54.450 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:29:54.456+0000 7f206b7fe700 1 -- 172.21.15.52:0/3027238271 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55718cc84130 con 0x55718cc78120 2024-05-27T07:29:54.450 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:29:54.456+0000 7f206a7fc700 1 --2- 172.21.15.52:0/3027238271 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55718caf8e40 0x55718cc807f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-27T07:29:54.450 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:29:54.456+0000 7f206affd700 1 --2- 172.21.15.52:0/3027238271 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55718cafb770 0x55718cc77be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-27T07:29:54.450 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:29:54.456+0000 7f206b7fe700 1 --2- 172.21.15.52:0/3027238271 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55718cc78120 0x55718cc6cc00 secure :-1 s=READY pgs=630 cs=0 l=1 rev1=1 crypto rx=0x7f2064029040 tx=0x7f2064009860 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d9a322fce41303dd server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:29:54.450 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:29:54.456+0000 7f2050ff9700 1 -- 172.21.15.52:0/3027238271 <== mon.0 v2:172.21.15.52:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f206402b040 con 0x55718cc78120 2024-05-27T07:29:54.450 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:29:54.456+0000 7f2050ff9700 1 -- 172.21.15.52:0/3027238271 <== mon.0 v2:172.21.15.52:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f206400bca0 con 0x55718cc78120 2024-05-27T07:29:54.450 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:29:54.456+0000 7f2050ff9700 1 -- 172.21.15.52:0/3027238271 <== mon.0 v2:172.21.15.52:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f206401cd80 con 0x55718cc78120 2024-05-27T07:29:54.450 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:29:54.457+0000 7f2080116b80 1 -- 172.21.15.52:0/3027238271 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55718cc78120 msgr2=0x55718cc6cc00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:29:54.450 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:29:54.457+0000 7f2080116b80 1 --2- 172.21.15.52:0/3027238271 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55718cc78120 0x55718cc6cc00 secure :-1 s=READY pgs=630 cs=0 l=1 rev1=1 crypto rx=0x7f2064029040 tx=0x7f2064009860 comp rx=0 tx=0).stop 2024-05-27T07:29:54.450 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:29:54.457+0000 7f2080116b80 1 -- 172.21.15.52:0/3027238271 shutdown_connections 2024-05-27T07:29:54.450 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:29:54.457+0000 7f2080116b80 1 --2- 172.21.15.52:0/3027238271 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55718cc78120 0x55718cc6cc00 unknown :-1 s=CLOSED pgs=630 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:29:54.450 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:29:54.457+0000 7f2080116b80 1 --2- 172.21.15.52:0/3027238271 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55718cafb770 0x55718cc77be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:29:54.451 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:29:54.457+0000 7f2080116b80 1 --2- 172.21.15.52:0/3027238271 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55718caf8e40 0x55718cc807f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:29:54.451 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:29:54.457+0000 7f2080116b80 1 -- 172.21.15.52:0/3027238271 >> 172.21.15.52:0/3027238271 conn(0x55718cc74480 msgr2=0x55718cc6c4f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:29:54.451 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:29:54.457+0000 7f2080116b80 1 -- 172.21.15.52:0/3027238271 shutdown_connections 2024-05-27T07:29:54.451 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:29:54.457+0000 7f2080116b80 1 -- 172.21.15.52:0/3027238271 wait complete. 2024-05-27T07:29:54.451 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:29:54.457+0000 7f2080116b80 1 -- start start 2024-05-27T07:29:54.451 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:29:54.458+0000 7f2080116b80 1 Processor -- start 2024-05-27T07:29:54.454 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:29:54.458+0000 7f2080116b80 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55718caf8e40 0x55718cc76cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:29:54.454 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:29:54.458+0000 7f2080116b80 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55718cafb770 0x55718cc77210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:29:54.455 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:29:54.458+0000 7f2080116b80 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55718cc78120 0x55718cc77750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:29:54.455 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:29:54.458+0000 7f2080116b80 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x7f206c1ae970 con 0x55718caf8e40 2024-05-27T07:29:54.455 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:29:54.458+0000 7f2080116b80 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f206c1ae2a0 con 0x55718cc78120 2024-05-27T07:29:54.455 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:29:54.458+0000 7f2080116b80 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f206c1af040 con 0x55718cafb770 2024-05-27T07:29:54.455 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:29:54.458+0000 7f2080116b80 10 client.0 _ll_register_callbacks cb 0x55718cb2c5b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T07:29:54.455 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:29:54.458+0000 7f206affd700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55718caf8e40 0x55718cc76cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:29:54.455 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:29:54.458+0000 7f206affd700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55718caf8e40 0x55718cc76cd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.52:39636/0 (socket says 172.21.15.52:39636) 2024-05-27T07:29:54.455 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:29:54.458+0000 7f206affd700 1 -- 172.21.15.52:0/3358744075 learned_addr learned my addr 172.21.15.52:0/3358744075 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-05-27T07:29:54.455 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:29:54.458+0000 7f206b7fe700 1 --2- 172.21.15.52:0/3358744075 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55718cc78120 0x55718cc77750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:29:54.455 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:29:54.458+0000 7f206a7fc700 1 --2- 172.21.15.52:0/3358744075 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55718cafb770 0x55718cc77210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:29:54.455 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:29:54.458+0000 7f206affd700 1 -- 172.21.15.52:0/3358744075 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55718cafb770 msgr2=0x55718cc77210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:29:54.455 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:29:54.458+0000 7f206affd700 1 --2- 172.21.15.52:0/3358744075 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55718cafb770 0x55718cc77210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:29:54.455 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:29:54.458+0000 7f206affd700 1 -- 172.21.15.52:0/3358744075 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55718cc78120 msgr2=0x55718cc77750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:29:54.456 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:29:54.458+0000 7f206affd700 1 --2- 172.21.15.52:0/3358744075 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55718cc78120 0x55718cc77750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:29:54.456 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:29:54.458+0000 7f206affd700 1 -- 172.21.15.52:0/3358744075 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55718cc8a950 con 0x55718caf8e40 2024-05-27T07:29:54.456 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:29:54.459+0000 7f206affd700 1 --2- 172.21.15.52:0/3358744075 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55718caf8e40 0x55718cc76cd0 secure :-1 s=READY pgs=631 cs=0 l=1 rev1=1 crypto rx=0x7f205c0dff30 tx=0x7f205c0cc040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ebf9b3affd24c510 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:29:54.456 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:29:54.459+0000 7f204eff5700 10 client.0 ms_handle_connect on v2:172.21.15.52:3300/0 2024-05-27T07:29:54.456 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:29:54.459+0000 7f204eff5700 1 -- 172.21.15.52:0/3358744075 <== mon.0 v2:172.21.15.52:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f205c0c2070 con 0x55718caf8e40 2024-05-27T07:29:54.456 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:29:54.459+0000 7f204eff5700 1 -- 172.21.15.52:0/3358744075 <== mon.0 v2:172.21.15.52:3300/0 2 ==== osd_map(110..110 src has 1..110) ==== 9379+0+0 (secure 0 0 0) 0x7f205c0a9cc0 con 0x55718caf8e40 2024-05-27T07:29:54.456 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:29:54.459+0000 7f2080116b80 1 -- 172.21.15.52:0/3358744075 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f206400bca0 con 0x55718caf8e40 2024-05-27T07:29:54.456 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:29:54.459+0000 7f204eff5700 1 -- 172.21.15.52:0/3358744075 <== mon.0 v2:172.21.15.52:3300/0 3 ==== mon_get_version_reply(handle=1 version=73) ==== 24+0+0 (secure 0 0 0) 0x7f205c0a9a60 con 0x55718caf8e40 2024-05-27T07:29:54.456 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:29:54.459+0000 7f2080116b80 10 client.16410 fetch_fsmap learned FSMap version 73 2024-05-27T07:29:54.456 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:29:54.459+0000 7f2080116b80 1 -- 172.21.15.52:0/3358744075 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({config=0+,fsmap=73,monmap=4+}) -- 0x55718ccfc640 con 0x55718caf8e40 2024-05-27T07:29:54.456 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:29:54.459+0000 7f204eff5700 1 -- 172.21.15.52:0/3358744075 <== mon.0 v2:172.21.15.52:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f206400bca0 con 0x55718caf8e40 2024-05-27T07:29:54.456 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:29:54.459+0000 7f204eff5700 1 -- 172.21.15.52:0/3358744075 <== mon.0 v2:172.21.15.52:3300/0 5 ==== fsmap(e 73) ==== 7773+0+0 (secure 0 0 0) 0x7f205c070200 con 0x55718caf8e40 2024-05-27T07:29:54.456 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:29:54.459+0000 7f2080116b80 10 client.16410 fetch_fsmap finished waiting for FSMap version 73 2024-05-27T07:29:54.457 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:29:54.459+0000 7f2080116b80 10 client.16410 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.h 2024-05-27T07:29:54.457 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:29:54.459+0000 7f2080116b80 20 client.16410 populate_metadata read hostname 'smithi052' 2024-05-27T07:29:54.457 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:29:54.459+0000 7f2080116b80 1 --2- 172.21.15.52:0/3358744075 >> [v2:172.21.15.105:6836/191393149,v1:172.21.15.105:6837/191393149] conn(0x55718cc7e070 0x55718ccfed20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:29:54.457 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:29:54.459+0000 7f2080116b80 4 client.16410 mds_command: new command op to 24497 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-05-27T07:29:54.457 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:29:54.459+0000 7f2080116b80 1 -- 172.21.15.52:0/3358744075 --> [v2:172.21.15.105:6836/191393149,v1:172.21.15.105:6837/191393149] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55718cba7cc0 con 0x55718cc7e070 2024-05-27T07:29:54.457 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:29:54.460+0000 7f206a7fc700 1 --2- 172.21.15.52:0/3358744075 >> [v2:172.21.15.105:6836/191393149,v1:172.21.15.105:6837/191393149] conn(0x55718cc7e070 0x55718ccfed20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:29:54.457 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:29:54.460+0000 7f206a7fc700 1 --2- 172.21.15.52:0/3358744075 >> [v2:172.21.15.105:6836/191393149,v1:172.21.15.105:6837/191393149] conn(0x55718cc7e070 0x55718ccfed20 crc :-1 s=READY pgs=44 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=ad373eced71ed3b4 server_cookie=4f53750c2306516f in_seq=0 out_seq=0 2024-05-27T07:29:54.457 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:29:54.460+0000 7f204eff5700 10 client.16410 ms_handle_connect on v2:172.21.15.105:6836/191393149 2024-05-27T07:29:54.457 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:29:54.460+0000 7f204eff5700 1 -- 172.21.15.52:0/3358744075 <== mds.1 v2:172.21.15.105:6836/191393149 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55718cba7cc0 con 0x55718cc7e070 2024-05-27T07:29:54.457 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:29:54.460+0000 7f204eff5700 10 client.16410 handle_command_reply: tid=0 2024-05-27T07:29:54.457 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:29:54.460+0000 7f2080116b80 1 client.16410 shutdown 2024-05-27T07:29:54.457 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:29:54.460+0000 7f2080116b80 1 -- 172.21.15.52:0/3358744075 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55718caf8e40 msgr2=0x55718cc76cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:29:54.458 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:29:54.460+0000 7f2080116b80 1 --2- 172.21.15.52:0/3358744075 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55718caf8e40 0x55718cc76cd0 secure :-1 s=READY pgs=631 cs=0 l=1 rev1=1 crypto rx=0x7f205c0dff30 tx=0x7f205c0cc040 comp rx=0 tx=0).stop 2024-05-27T07:29:54.458 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:29:54.460+0000 7f2080116b80 1 -- 172.21.15.52:0/3358744075 shutdown_connections 2024-05-27T07:29:54.458 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:29:54.460+0000 7f2080116b80 1 --2- 172.21.15.52:0/3358744075 >> [v2:172.21.15.105:6836/191393149,v1:172.21.15.105:6837/191393149] conn(0x55718cc7e070 0x55718ccfed20 crc :-1 s=READY pgs=44 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:29:54.458 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:29:54.460+0000 7f2080116b80 1 --2- 172.21.15.52:0/3358744075 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55718cc78120 0x55718cc77750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:29:54.458 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:29:54.460+0000 7f2080116b80 1 --2- 172.21.15.52:0/3358744075 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55718caf8e40 0x55718cc76cd0 unknown :-1 s=CLOSED pgs=631 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:29:54.458 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:29:54.460+0000 7f2080116b80 1 --2- 172.21.15.52:0/3358744075 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55718cafb770 0x55718cc77210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:29:54.458 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:29:54.460+0000 7f2080116b80 1 -- 172.21.15.52:0/3358744075 >> 172.21.15.52:0/3358744075 conn(0x55718cc74480 msgr2=0x55718cc7d100 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:29:54.458 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:29:54.461+0000 7f204eff5700 0 client.16410 ms_handle_reset on v2:172.21.15.105:6836/191393149 2024-05-27T07:29:54.458 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:29:54.461+0000 7f2080116b80 1 -- 172.21.15.52:0/3358744075 shutdown_connections 2024-05-27T07:29:54.458 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:29:54.461+0000 7f2080116b80 1 -- 172.21.15.52:0/3358744075 wait complete. 2024-05-27T07:29:54.458 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:29:54.461+0000 7f2080116b80 20 client.16410 trim_cache size 0 max 16384 2024-05-27T07:29:54.459 INFO:teuthology.orchestra.run.smithi052.stdout:[] 2024-05-27T07:29:54.460 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:29:54.461+0000 7f2071701700 1 -- 172.21.15.52:0/3429381078 >> [v2:172.21.15.52:6800/1822318384,v1:172.21.15.52:6801/1822318384] conn(0x7f20540b29f0 msgr2=0x7f20540b4f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:29:54.460 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:29:54.461+0000 7f2071701700 1 --2- 172.21.15.52:0/3429381078 >> [v2:172.21.15.52:6800/1822318384,v1:172.21.15.52:6801/1822318384] conn(0x7f20540b29f0 0x7f20540b4f00 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7f206c109860 tx=0x7f2060009260 comp rx=0 tx=0).stop 2024-05-27T07:29:54.460 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:29:54.461+0000 7f2071701700 1 -- 172.21.15.52:0/3429381078 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f206c103c10 msgr2=0x7f206c19ae40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:29:54.460 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:29:54.461+0000 7f2071701700 1 --2- 172.21.15.52:0/3429381078 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f206c103c10 0x7f206c19ae40 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7f205c00e9c0 tx=0x7f205c00ecd0 comp rx=0 tx=0).stop 2024-05-27T07:29:54.460 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:29:54.461+0000 7f2071701700 1 -- 172.21.15.52:0/3429381078 shutdown_connections 2024-05-27T07:29:54.460 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:29:54.461+0000 7f2071701700 1 --2- 172.21.15.52:0/3429381078 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f206c108760 0x7f206c19f670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:29:54.460 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:29:54.461+0000 7f2071701700 1 --2- 172.21.15.52:0/3429381078 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f206c103c10 0x7f206c19ae40 unknown :-1 s=CLOSED pgs=127 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:29:54.460 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:29:54.461+0000 7f2071701700 1 --2- 172.21.15.52:0/3429381078 >> [v2:172.21.15.52:6800/1822318384,v1:172.21.15.52:6801/1822318384] conn(0x7f20540b29f0 0x7f20540b4f00 unknown :-1 s=CLOSED pgs=126 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:29:54.460 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:29:54.461+0000 7f2071701700 1 --2- 172.21.15.52:0/3429381078 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f206c107e10 0x7f206c19b3e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:29:54.460 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:29:54.462+0000 7f2071701700 1 -- 172.21.15.52:0/3429381078 >> 172.21.15.52:0/3429381078 conn(0x7f206c0fe6e0 msgr2=0x7f206c100b00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:29:54.460 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:29:54.462+0000 7f2071701700 1 -- 172.21.15.52:0/3429381078 shutdown_connections 2024-05-27T07:29:54.460 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:29:54.462+0000 7f2071701700 1 -- 172.21.15.52:0/3429381078 wait complete. 2024-05-27T07:29:54.480 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:29:54 smithi052 ceph-mon[31955]: pgmap v140: 129 pgs: 129 active+clean; 269 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 1.1 MiB/s wr, 953 op/s 2024-05-27T07:29:54.480 INFO:journalctl@ceph.mds.g.smithi052.stdout:May 27 07:29:54 smithi052 systemd[1]: ceph-73df2636-1bf8-11ef-bc9b-c7b262605968@mds.g.service: Failed with result 'exit-code'. 2024-05-27T07:29:54.480 INFO:journalctl@ceph.mds.g.smithi052.stdout:May 27 07:29:54 smithi052 systemd[1]: ceph-73df2636-1bf8-11ef-bc9b-c7b262605968@mds.g.service: Consumed 1.196s CPU time. 2024-05-27T07:29:54.480 INFO:journalctl@ceph.mds.a.smithi052.stdout:May 27 07:29:54 smithi052 systemd[1]: ceph-73df2636-1bf8-11ef-bc9b-c7b262605968@mds.a.service: Failed with result 'exit-code'. 2024-05-27T07:29:54.480 INFO:journalctl@ceph.mds.a.smithi052.stdout:May 27 07:29:54 smithi052 systemd[1]: ceph-73df2636-1bf8-11ef-bc9b-c7b262605968@mds.a.service: Consumed 1.283s CPU time. 2024-05-27T07:29:54.602 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:29:54 smithi173 ceph-mon[37000]: pgmap v140: 129 pgs: 129 active+clean; 269 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 1.1 MiB/s wr, 953 op/s 2024-05-27T07:29:54.690 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:29:54 smithi105 ceph-mon[35977]: pgmap v140: 129 pgs: 129 active+clean; 269 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 1.1 MiB/s wr, 953 op/s 2024-05-27T07:29:55.207 INFO:tasks.workunit.client.0.smithi052.stdout: 72 682092 384 487418 412 417353 889 2024-05-27T07:29:55.730 INFO:journalctl@ceph.mds.d.smithi052.stdout:May 27 07:29:55 smithi052 systemd[1]: ceph-73df2636-1bf8-11ef-bc9b-c7b262605968@mds.d.service: Failed with result 'exit-code'. 2024-05-27T07:29:55.877 DEBUG:teuthology.orchestra.run.smithi052:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:38f9d900c83c63799fbdbe61acc9a11b0d3554a6 shell --fsid 73df2636-1bf8-11ef-bc9b-c7b262605968 -- ceph tell mds.1:0 damage ls 2024-05-27T07:29:56.602 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:29:56 smithi173 ceph-mon[37000]: pgmap v141: 129 pgs: 129 active+clean; 265 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 7.7 MiB/s rd, 1.8 MiB/s wr, 1.67k op/s 2024-05-27T07:29:56.642 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:29:56 smithi105 ceph-mon[35977]: pgmap v141: 129 pgs: 129 active+clean; 265 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 7.7 MiB/s rd, 1.8 MiB/s wr, 1.67k op/s 2024-05-27T07:29:56.730 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:29:56 smithi052 ceph-mon[31955]: pgmap v141: 129 pgs: 129 active+clean; 265 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 7.7 MiB/s rd, 1.8 MiB/s wr, 1.67k op/s 2024-05-27T07:29:56.940 INFO:journalctl@ceph.mds.k.smithi105.stdout:May 27 07:29:56 smithi105 podman[70025]: 2024-05-27 07:29:56.651967408 +0000 UTC m=+4.277227405 container remove 65fd1707409f955d437095b2ebc4b654df22761089d5fcee3ce3884251d520be (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-k, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.build-date=20240521, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, org.label-schema.schema-version=1.0) 2024-05-27T07:29:56.940 INFO:journalctl@ceph.mds.k.smithi105.stdout:May 27 07:29:56 smithi105 systemd[1]: ceph-73df2636-1bf8-11ef-bc9b-c7b262605968@mds.k.service: Main process exited, code=exited, status=134/n/a 2024-05-27T07:29:57.440 INFO:journalctl@ceph.mds.k.smithi105.stdout:May 27 07:29:56 smithi105 systemd[1]: ceph-73df2636-1bf8-11ef-bc9b-c7b262605968@mds.k.service: Failed with result 'exit-code'. 2024-05-27T07:29:58.602 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:29:58 smithi173 ceph-mon[37000]: pgmap v142: 129 pgs: 129 active+clean; 260 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 6.8 MiB/s rd, 1.4 MiB/s wr, 1.38k op/s 2024-05-27T07:29:58.690 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:29:58 smithi105 ceph-mon[35977]: pgmap v142: 129 pgs: 129 active+clean; 260 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 6.8 MiB/s rd, 1.4 MiB/s wr, 1.38k op/s 2024-05-27T07:29:58.730 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:29:58 smithi052 ceph-mon[31955]: pgmap v142: 129 pgs: 129 active+clean; 260 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 6.8 MiB/s rd, 1.4 MiB/s wr, 1.38k op/s 2024-05-27T07:29:59.412 INFO:teuthology.orchestra.run.smithi052.stderr:Inferring config /var/lib/ceph/73df2636-1bf8-11ef-bc9b-c7b262605968/mon.a/config 2024-05-27T07:30:00.480 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:30:00 smithi052 ceph-mon[31955]: pgmap v143: 129 pgs: 129 active+clean; 249 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 5.6 MiB/s rd, 1.3 MiB/s wr, 879 op/s 2024-05-27T07:30:00.480 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:30:00 smithi052 ceph-mon[31955]: overall HEALTH_OK 2024-05-27T07:30:00.602 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:30:00 smithi173 ceph-mon[37000]: pgmap v143: 129 pgs: 129 active+clean; 249 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 5.6 MiB/s rd, 1.3 MiB/s wr, 879 op/s 2024-05-27T07:30:00.603 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:30:00 smithi173 ceph-mon[37000]: overall HEALTH_OK 2024-05-27T07:30:00.690 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:30:00 smithi105 ceph-mon[35977]: pgmap v143: 129 pgs: 129 active+clean; 249 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 5.6 MiB/s rd, 1.3 MiB/s wr, 879 op/s 2024-05-27T07:30:00.691 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:30:00 smithi105 ceph-mon[35977]: overall HEALTH_OK 2024-05-27T07:30:01.434 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:30:01.440+0000 7f8a58ab1700 1 -- 172.21.15.52:0/3947325255 <== mon.1 v2:172.21.15.173:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8a50017400 con 0x7f8a54107de0 2024-05-27T07:30:01.434 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:30:01.440+0000 7f8a5bd17700 1 -- 172.21.15.52:0/3947325255 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f8a54107de0 msgr2=0x7f8a541081f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:30:01.434 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:30:01.440+0000 7f8a5bd17700 1 --2- 172.21.15.52:0/3947325255 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f8a54107de0 0x7f8a541081f0 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7f8a50009920 tx=0x7f8a50009c30 comp rx=0 tx=0).stop 2024-05-27T07:30:01.434 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:30:01.440+0000 7f8a5bd17700 1 -- 172.21.15.52:0/3947325255 shutdown_connections 2024-05-27T07:30:01.434 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:30:01.440+0000 7f8a5bd17700 1 --2- 172.21.15.52:0/3947325255 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f8a54108730 0x7f8a54110090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:30:01.435 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:30:01.440+0000 7f8a5bd17700 1 --2- 172.21.15.52:0/3947325255 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f8a541034b0 0x7f8a541038c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:30:01.435 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:30:01.440+0000 7f8a5bd17700 1 --2- 172.21.15.52:0/3947325255 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f8a54107de0 0x7f8a541081f0 unknown :-1 s=CLOSED pgs=128 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:30:01.435 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:30:01.440+0000 7f8a5bd17700 1 -- 172.21.15.52:0/3947325255 >> 172.21.15.52:0/3947325255 conn(0x7f8a540fedb0 msgr2=0x7f8a541011d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:30:01.435 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:30:01.441+0000 7f8a5bd17700 1 -- 172.21.15.52:0/3947325255 shutdown_connections 2024-05-27T07:30:01.435 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:30:01.441+0000 7f8a5bd17700 1 -- 172.21.15.52:0/3947325255 wait complete. 2024-05-27T07:30:01.438 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:30:01.442+0000 7f8a5bd17700 1 Processor -- start 2024-05-27T07:30:01.439 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:30:01.443+0000 7f8a5bd17700 1 -- start start 2024-05-27T07:30:01.439 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:30:01.443+0000 7f8a5bd17700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f8a541034b0 0x7f8a54199b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:30:01.439 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:30:01.443+0000 7f8a5bd17700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f8a54107de0 0x7f8a5419a070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:30:01.439 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:30:01.443+0000 7f8a5bd17700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f8a54108730 0x7f8a5419a5f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:30:01.439 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:30:01.443+0000 7f8a5bd17700 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x7f8a541133e0 con 0x7f8a54107de0 2024-05-27T07:30:01.439 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:30:01.443+0000 7f8a5bd17700 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f8a54113260 con 0x7f8a54108730 2024-05-27T07:30:01.440 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:30:01.443+0000 7f8a5bd17700 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f8a54113560 con 0x7f8a541034b0 2024-05-27T07:30:01.440 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:30:01.444+0000 7f8a5a2b4700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f8a54108730 0x7f8a5419a5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:30:01.440 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:30:01.444+0000 7f8a5a2b4700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f8a54108730 0x7f8a5419a5f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.52:54100/0 (socket says 172.21.15.52:54100) 2024-05-27T07:30:01.440 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:30:01.444+0000 7f8a5a2b4700 1 -- 172.21.15.52:0/2753348598 learned_addr learned my addr 172.21.15.52:0/2753348598 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-05-27T07:30:01.440 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:30:01.444+0000 7f8a592b2700 1 --2- 172.21.15.52:0/2753348598 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f8a54107de0 0x7f8a5419a070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:30:01.440 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:30:01.445+0000 7f8a59ab3700 1 --2- 172.21.15.52:0/2753348598 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f8a541034b0 0x7f8a54199b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:30:01.441 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:30:01.445+0000 7f8a592b2700 1 -- 172.21.15.52:0/2753348598 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f8a541034b0 msgr2=0x7f8a54199b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:30:01.441 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:30:01.445+0000 7f8a592b2700 1 --2- 172.21.15.52:0/2753348598 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f8a541034b0 0x7f8a54199b30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:30:01.441 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:30:01.445+0000 7f8a592b2700 1 -- 172.21.15.52:0/2753348598 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f8a54108730 msgr2=0x7f8a5419a5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:30:01.441 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:30:01.445+0000 7f8a592b2700 1 --2- 172.21.15.52:0/2753348598 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f8a54108730 0x7f8a5419a5f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:30:01.441 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:30:01.445+0000 7f8a592b2700 1 -- 172.21.15.52:0/2753348598 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8a5419ee70 con 0x7f8a54107de0 2024-05-27T07:30:01.442 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:30:01.445+0000 7f8a59ab3700 1 --2- 172.21.15.52:0/2753348598 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f8a541034b0 0x7f8a54199b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-27T07:30:01.442 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:30:01.445+0000 7f8a592b2700 1 --2- 172.21.15.52:0/2753348598 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f8a54107de0 0x7f8a5419a070 secure :-1 s=READY pgs=632 cs=0 l=1 rev1=1 crypto rx=0x7f8a50000c00 tx=0x7f8a5000f950 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:30:01.442 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:30:01.446+0000 7f8a4affd700 1 -- 172.21.15.52:0/2753348598 <== mon.0 v2:172.21.15.52:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8a50027070 con 0x7f8a54107de0 2024-05-27T07:30:01.442 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:30:01.446+0000 7f8a4affd700 1 -- 172.21.15.52:0/2753348598 <== mon.0 v2:172.21.15.52:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f8a50005570 con 0x7f8a54107de0 2024-05-27T07:30:01.442 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:30:01.446+0000 7f8a5bd17700 1 -- 172.21.15.52:0/2753348598 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8a5419f100 con 0x7f8a54107de0 2024-05-27T07:30:01.442 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:30:01.446+0000 7f8a4affd700 1 -- 172.21.15.52:0/2753348598 <== mon.0 v2:172.21.15.52:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8a50004930 con 0x7f8a54107de0 2024-05-27T07:30:01.443 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:30:01.446+0000 7f8a5bd17700 1 -- 172.21.15.52:0/2753348598 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8a5419f590 con 0x7f8a54107de0 2024-05-27T07:30:01.444 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:30:01.452+0000 7f8a4affd700 1 -- 172.21.15.52:0/2753348598 <== mon.0 v2:172.21.15.52:3300/0 4 ==== mgrmap(e 26) ==== 149783+0+0 (secure 0 0 0) 0x7f8a5002a070 con 0x7f8a54107de0 2024-05-27T07:30:01.444 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:30:01.452+0000 7f8a6a72cb80 1 Processor -- start 2024-05-27T07:30:01.445 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:30:01.453+0000 7f8a6a72cb80 1 -- start start 2024-05-27T07:30:01.449 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:30:01.453+0000 7f8a6a72cb80 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5651f09a1e40 0x5651f0b25710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:30:01.450 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:30:01.453+0000 7f8a6a72cb80 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5651f0b25c50 0x5651f0b1fa40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:30:01.450 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:30:01.453+0000 7f8a5a2b4700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5651f09a1e40 0x5651f0b25710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:30:01.450 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:30:01.453+0000 7f8a5a2b4700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5651f09a1e40 0x5651f0b25710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.52:54102/0 (socket says 172.21.15.52:54102) 2024-05-27T07:30:01.450 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:30:01.453+0000 7f8a6a72cb80 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x5651f0b1ff80 0x5651f0b20390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:30:01.450 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:30:01.453+0000 7f8a5a2b4700 1 -- 172.21.15.52:0/1908067893 learned_addr learned my addr 172.21.15.52:0/1908067893 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-05-27T07:30:01.450 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:30:01.453+0000 7f8a6a72cb80 1 -- 172.21.15.52:0/1908067893 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x5651f0a4ec90 con 0x5651f0b1ff80 2024-05-27T07:30:01.451 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:30:01.453+0000 7f8a6a72cb80 1 -- 172.21.15.52:0/1908067893 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x5651f0a3e830 con 0x5651f09a1e40 2024-05-27T07:30:01.451 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:30:01.453+0000 7f8a6a72cb80 1 -- 172.21.15.52:0/1908067893 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x5651f0a42870 con 0x5651f0b25c50 2024-05-27T07:30:01.451 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:30:01.453+0000 7f8a4affd700 1 --2- 172.21.15.52:0/2753348598 >> [v2:172.21.15.52:6800/1822318384,v1:172.21.15.52:6801/1822318384] conn(0x7f8a400b2b40 0x7f8a400b5050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:30:01.451 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:30:01.453+0000 7f8a59ab3700 1 --2- 172.21.15.52:0/1908067893 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5651f0b25c50 0x5651f0b1fa40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:30:01.451 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:30:01.454+0000 7f8a5a2b4700 1 --2- 172.21.15.52:0/2753348598 >> [v2:172.21.15.52:6800/1822318384,v1:172.21.15.52:6801/1822318384] conn(0x7f8a400b2b40 0x7f8a400b5050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:30:01.451 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:30:01.454+0000 7f8a4affd700 1 -- 172.21.15.52:0/2753348598 <== mon.0 v2:172.21.15.52:3300/0 5 ==== osd_map(110..110 src has 1..110) ==== 9379+0+0 (secure 0 0 0) 0x7f8a500e32e0 con 0x7f8a54107de0 2024-05-27T07:30:01.452 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:30:01.454+0000 7f8a5a2b4700 1 -- 172.21.15.52:0/1908067893 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5651f0b25c50 msgr2=0x5651f0b1fa40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:30:01.452 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:30:01.454+0000 7f8a5a2b4700 1 --2- 172.21.15.52:0/1908067893 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5651f0b25c50 0x5651f0b1fa40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:30:01.452 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:30:01.454+0000 7f8a5a2b4700 1 -- 172.21.15.52:0/1908067893 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x5651f0b1ff80 msgr2=0x5651f0b20390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:30:01.452 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:30:01.454+0000 7f8a5a2b4700 1 --2- 172.21.15.52:0/1908067893 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x5651f0b1ff80 0x5651f0b20390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:30:01.452 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:30:01.454+0000 7f8a5a2b4700 1 -- 172.21.15.52:0/1908067893 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5651f098e8d0 con 0x5651f09a1e40 2024-05-27T07:30:01.452 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:30:01.454+0000 7f8a5a2b4700 1 --2- 172.21.15.52:0/1908067893 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5651f09a1e40 0x5651f0b25710 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7f8a4c00ece0 tx=0x7f8a4c018360 comp rx=0 tx=0).ready entity=mon.2 client_cookie=829b2b68a8135738 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:30:01.453 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:30:01.455+0000 7f8a5a2b4700 1 --2- 172.21.15.52:0/2753348598 >> [v2:172.21.15.52:6800/1822318384,v1:172.21.15.52:6801/1822318384] conn(0x7f8a400b2b40 0x7f8a400b5050 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7f8a4c022000 tx=0x7f8a4c020000 comp rx=0 tx=0).ready entity=mgr.15333 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:30:01.453 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:30:01.455+0000 7f8a3ffff700 1 -- 172.21.15.52:0/1908067893 <== mon.2 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8a4c009370 con 0x5651f09a1e40 2024-05-27T07:30:01.453 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:30:01.455+0000 7f8a3ffff700 1 -- 172.21.15.52:0/1908067893 <== mon.2 v2:172.21.15.105:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f8a4c007c90 con 0x5651f09a1e40 2024-05-27T07:30:01.453 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:30:01.456+0000 7f8a6a72cb80 1 -- 172.21.15.52:0/1908067893 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5651f09a1e40 msgr2=0x5651f0b25710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:30:01.453 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:30:01.456+0000 7f8a6a72cb80 1 --2- 172.21.15.52:0/1908067893 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5651f09a1e40 0x5651f0b25710 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7f8a4c00ece0 tx=0x7f8a4c018360 comp rx=0 tx=0).stop 2024-05-27T07:30:01.454 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:30:01.456+0000 7f8a6a72cb80 1 -- 172.21.15.52:0/1908067893 shutdown_connections 2024-05-27T07:30:01.454 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:30:01.456+0000 7f8a6a72cb80 1 --2- 172.21.15.52:0/1908067893 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x5651f0b1ff80 0x5651f0b20390 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:30:01.454 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:30:01.456+0000 7f8a6a72cb80 1 --2- 172.21.15.52:0/1908067893 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5651f09a1e40 0x5651f0b25710 unknown :-1 s=CLOSED pgs=141 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:30:01.454 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:30:01.456+0000 7f8a6a72cb80 1 --2- 172.21.15.52:0/1908067893 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5651f0b25c50 0x5651f0b1fa40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:30:01.454 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:30:01.456+0000 7f8a6a72cb80 1 -- 172.21.15.52:0/1908067893 >> 172.21.15.52:0/1908067893 conn(0x5651f09fcaf0 msgr2=0x5651f0b1a640 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:30:01.454 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:30:01.457+0000 7f8a6a72cb80 1 -- 172.21.15.52:0/1908067893 shutdown_connections 2024-05-27T07:30:01.455 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:30:01.457+0000 7f8a6a72cb80 1 -- 172.21.15.52:0/1908067893 wait complete. 2024-05-27T07:30:01.455 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:30:01.457+0000 7f8a6a72cb80 1 -- start start 2024-05-27T07:30:01.455 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:30:01.457+0000 7f8a6a72cb80 1 Processor -- start 2024-05-27T07:30:01.461 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:30:01.458+0000 7f8a6a72cb80 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5651f09a1e40 0x5651f0b29370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:30:01.461 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:30:01.458+0000 7f8a6a72cb80 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5651f0b1ff80 0x5651f0b298b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:30:01.461 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:30:01.458+0000 7f8a59ab3700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5651f09a1e40 0x5651f0b29370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:30:01.461 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:30:01.458+0000 7f8a59ab3700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5651f09a1e40 0x5651f0b29370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.52:54110/0 (socket says 172.21.15.52:54110) 2024-05-27T07:30:01.462 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:30:01.458+0000 7f8a592b2700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5651f0b1ff80 0x5651f0b298b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:30:01.462 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:30:01.458+0000 7f8a592b2700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5651f0b1ff80 0x5651f0b298b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.173:3300/0 says I am v2:172.21.15.52:53842/0 (socket says 172.21.15.52:53842) 2024-05-27T07:30:01.462 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:30:01.458+0000 7f8a592b2700 1 -- 172.21.15.52:0/3326565009 learned_addr learned my addr 172.21.15.52:0/3326565009 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-05-27T07:30:01.462 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:30:01.459+0000 7f8a6a72cb80 1 --2- 172.21.15.52:0/3326565009 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x5651f0b25c50 0x5651f0b29df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:30:01.462 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:30:01.459+0000 7f8a6a72cb80 1 -- 172.21.15.52:0/3326565009 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x5651f0944fa0 con 0x5651f0b25c50 2024-05-27T07:30:01.462 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:30:01.459+0000 7f8a6a72cb80 1 -- 172.21.15.52:0/3326565009 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x5651f09e07c0 con 0x5651f09a1e40 2024-05-27T07:30:01.463 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:30:01.459+0000 7f8a6a72cb80 1 -- 172.21.15.52:0/3326565009 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x5651f0b2c790 con 0x5651f0b1ff80 2024-05-27T07:30:01.463 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:30:01.459+0000 7f8a6a72cb80 10 client.0 _ll_register_callbacks cb 0x5651f09d9920 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T07:30:01.463 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:30:01.459+0000 7f8a5a2b4700 1 --2- 172.21.15.52:0/3326565009 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x5651f0b25c50 0x5651f0b29df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:30:01.463 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:30:01.459+0000 7f8a592b2700 1 -- 172.21.15.52:0/3326565009 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5651f09a1e40 msgr2=0x5651f0b29370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:30:01.463 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:30:01.459+0000 7f8a592b2700 1 --2- 172.21.15.52:0/3326565009 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5651f09a1e40 0x5651f0b29370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:30:01.463 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:30:01.459+0000 7f8a592b2700 1 -- 172.21.15.52:0/3326565009 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x5651f0b25c50 msgr2=0x5651f0b29df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:30:01.464 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:30:01.459+0000 7f8a592b2700 1 --2- 172.21.15.52:0/3326565009 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x5651f0b25c50 0x5651f0b29df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:30:01.464 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:30:01.460+0000 7f8a592b2700 1 -- 172.21.15.52:0/3326565009 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5651f09dd690 con 0x5651f0b1ff80 2024-05-27T07:30:01.464 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:30:01.460+0000 7f8a592b2700 1 --2- 172.21.15.52:0/3326565009 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5651f0b1ff80 0x5651f0b298b0 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7f8a500098f0 tx=0x7f8a500aa0c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=12db72ce3f2f3030 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:30:01.464 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:30:01.460+0000 7f8a3d7fa700 10 client.0 ms_handle_connect on v2:172.21.15.173:3300/0 2024-05-27T07:30:01.464 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:30:01.461+0000 7f8a3d7fa700 1 -- 172.21.15.52:0/3326565009 <== mon.1 v2:172.21.15.173:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8a5006dc30 con 0x5651f0b1ff80 2024-05-27T07:30:01.464 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:30:01.461+0000 7f8a3d7fa700 1 -- 172.21.15.52:0/3326565009 <== mon.1 v2:172.21.15.173:3300/0 2 ==== osd_map(110..110 src has 1..110) ==== 9379+0+0 (secure 0 0 0) 0x7f8a5000bd20 con 0x5651f0b1ff80 2024-05-27T07:30:01.465 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:30:01.461+0000 7f8a6a72cb80 1 -- 172.21.15.52:0/3326565009 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f8a4c007c90 con 0x5651f0b1ff80 2024-05-27T07:30:01.465 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:30:01.461+0000 7f8a3d7fa700 1 -- 172.21.15.52:0/3326565009 <== mon.1 v2:172.21.15.173:3300/0 3 ==== mon_get_version_reply(handle=1 version=73) ==== 24+0+0 (secure 0 0 0) 0x7f8a5006cce0 con 0x5651f0b1ff80 2024-05-27T07:30:01.465 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:30:01.461+0000 7f8a6a72cb80 10 client.25990 fetch_fsmap learned FSMap version 73 2024-05-27T07:30:01.465 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:30:01.461+0000 7f8a6a72cb80 1 -- 172.21.15.52:0/3326565009 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({config=0+,fsmap=73,monmap=4+}) -- 0x5651f0b370a0 con 0x5651f0b1ff80 2024-05-27T07:30:01.465 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:30:01.461+0000 7f8a3d7fa700 1 -- 172.21.15.52:0/3326565009 <== mon.1 v2:172.21.15.173:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f8a500a9e00 con 0x5651f0b1ff80 2024-05-27T07:30:01.465 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:30:01.461+0000 7f8a3d7fa700 1 -- 172.21.15.52:0/3326565009 <== mon.1 v2:172.21.15.173:3300/0 5 ==== fsmap(e 73) ==== 7773+0+0 (secure 0 0 0) 0x7f8a500aa550 con 0x5651f0b1ff80 2024-05-27T07:30:01.466 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:30:01.461+0000 7f8a6a72cb80 10 client.25990 fetch_fsmap finished waiting for FSMap version 73 2024-05-27T07:30:01.466 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:30:01.461+0000 7f8a6a72cb80 10 client.25990 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-05-27T07:30:01.466 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:30:01.461+0000 7f8a6a72cb80 20 client.25990 populate_metadata read hostname 'smithi052' 2024-05-27T07:30:01.466 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:30:01.461+0000 7f8a6a72cb80 1 --2- 172.21.15.52:0/3326565009 >> [v2:172.21.15.105:6838/500348151,v1:172.21.15.105:6839/500348151] conn(0x5651f0b39660 0x5651f0b3bab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:30:01.466 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:30:01.461+0000 7f8a6a72cb80 4 client.25990 mds_command: new command op to 24527 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-27T07:30:01.466 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:30:01.461+0000 7f8a6a72cb80 1 -- 172.21.15.52:0/3326565009 --> [v2:172.21.15.105:6838/500348151,v1:172.21.15.105:6839/500348151] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x7f8a4c007c90 con 0x5651f0b39660 2024-05-27T07:30:01.467 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:30:01.462+0000 7f8a59ab3700 1 --2- 172.21.15.52:0/3326565009 >> [v2:172.21.15.105:6838/500348151,v1:172.21.15.105:6839/500348151] conn(0x5651f0b39660 0x5651f0b3bab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:30:01.467 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:30:01.462+0000 7f8a3d7fa700 10 client.25990 ms_handle_connect on v2:172.21.15.105:6838/500348151 2024-05-27T07:30:01.467 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:30:01.462+0000 7f8a59ab3700 1 --2- 172.21.15.52:0/3326565009 >> [v2:172.21.15.105:6838/500348151,v1:172.21.15.105:6839/500348151] conn(0x5651f0b39660 0x5651f0b3bab0 crc :-1 s=READY pgs=99 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=3da3f504b7c724f9 server_cookie=8cb4232e5a1552d0 in_seq=0 out_seq=0 2024-05-27T07:30:01.467 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:30:01.465+0000 7f8a3d7fa700 1 -- 172.21.15.52:0/3326565009 <== mds.0 v2:172.21.15.105:6838/500348151 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24041 (crc 0 0 0) 0x7f8a4c007c90 con 0x5651f0b39660 2024-05-27T07:30:01.467 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:30:01.465+0000 7f8a3d7fa700 10 client.25990 handle_command_reply: tid=0 2024-05-27T07:30:01.467 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:30:01.465+0000 7f8a6a72cb80 1 client.25990 shutdown 2024-05-27T07:30:01.468 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:30:01.465+0000 7f8a6a72cb80 1 -- 172.21.15.52:0/3326565009 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5651f0b1ff80 msgr2=0x5651f0b298b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:30:01.468 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:30:01.465+0000 7f8a6a72cb80 1 --2- 172.21.15.52:0/3326565009 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5651f0b1ff80 0x5651f0b298b0 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7f8a500098f0 tx=0x7f8a500aa0c0 comp rx=0 tx=0).stop 2024-05-27T07:30:01.468 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:30:01.465+0000 7f8a6a72cb80 1 -- 172.21.15.52:0/3326565009 shutdown_connections 2024-05-27T07:30:01.468 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:30:01.465+0000 7f8a6a72cb80 1 --2- 172.21.15.52:0/3326565009 >> [v2:172.21.15.105:6838/500348151,v1:172.21.15.105:6839/500348151] conn(0x5651f0b39660 0x5651f0b3bab0 crc :-1 s=READY pgs=99 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:30:01.468 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:30:01.465+0000 7f8a6a72cb80 1 --2- 172.21.15.52:0/3326565009 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x5651f0b25c50 0x5651f0b29df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:30:01.468 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:30:01.465+0000 7f8a6a72cb80 1 --2- 172.21.15.52:0/3326565009 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5651f09a1e40 0x5651f0b29370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:30:01.469 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:30:01.465+0000 7f8a6a72cb80 1 --2- 172.21.15.52:0/3326565009 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5651f0b1ff80 0x5651f0b298b0 unknown :-1 s=CLOSED pgs=129 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:30:01.469 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:30:01.465+0000 7f8a6a72cb80 1 -- 172.21.15.52:0/3326565009 >> 172.21.15.52:0/3326565009 conn(0x5651f09fcaf0 msgr2=0x5651f0b1ea30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:30:01.469 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:30:01.465+0000 7f8a3d7fa700 0 client.25990 ms_handle_reset on v2:172.21.15.105:6838/500348151 2024-05-27T07:30:01.469 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:30:01.465+0000 7f8a6a72cb80 1 -- 172.21.15.52:0/3326565009 shutdown_connections 2024-05-27T07:30:01.469 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:30:01.465+0000 7f8a6a72cb80 1 -- 172.21.15.52:0/3326565009 wait complete. 2024-05-27T07:30:01.469 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:30:01.466+0000 7f8a6a72cb80 20 client.25990 trim_cache size 0 max 16384 2024-05-27T07:30:01.486 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:30:01.494+0000 7f8a6a72cb80 1 Processor -- start 2024-05-27T07:30:01.489 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:30:01.495+0000 7f8a6a72cb80 1 -- start start 2024-05-27T07:30:01.489 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:30:01.495+0000 7f8a6a72cb80 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x5651f09a1e40 0x5651f0b1fa40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:30:01.489 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:30:01.495+0000 7f8a6a72cb80 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5651f0b1ff80 0x5651f0b28b40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:30:01.489 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:30:01.495+0000 7f8a6a72cb80 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5651f0b25c50 0x5651f0b23720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:30:01.489 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:30:01.495+0000 7f8a6a72cb80 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x7f8a541b8590 con 0x5651f09a1e40 2024-05-27T07:30:01.489 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:30:01.495+0000 7f8a6a72cb80 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f8a541b6380 con 0x5651f0b25c50 2024-05-27T07:30:01.490 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:30:01.495+0000 7f8a6a72cb80 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f8a541b7ec0 con 0x5651f0b1ff80 2024-05-27T07:30:01.490 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:30:01.496+0000 7f8a5a2b4700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5651f0b25c50 0x5651f0b23720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:30:01.490 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:30:01.496+0000 7f8a5a2b4700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5651f0b25c50 0x5651f0b23720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.52:54118/0 (socket says 172.21.15.52:54118) 2024-05-27T07:30:01.490 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:30:01.496+0000 7f8a5a2b4700 1 -- 172.21.15.52:0/1872853538 learned_addr learned my addr 172.21.15.52:0/1872853538 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-05-27T07:30:01.490 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:30:01.496+0000 7f8a592b2700 1 --2- 172.21.15.52:0/1872853538 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5651f0b1ff80 0x5651f0b28b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:30:01.490 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:30:01.497+0000 7f8a5a2b4700 1 -- 172.21.15.52:0/1872853538 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5651f0b1ff80 msgr2=0x5651f0b28b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:30:01.491 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:30:01.497+0000 7f8a5a2b4700 1 --2- 172.21.15.52:0/1872853538 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5651f0b1ff80 0x5651f0b28b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:30:01.491 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:30:01.497+0000 7f8a5a2b4700 1 -- 172.21.15.52:0/1872853538 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x5651f09a1e40 msgr2=0x5651f0b1fa40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:30:01.491 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:30:01.497+0000 7f8a5a2b4700 1 --2- 172.21.15.52:0/1872853538 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x5651f09a1e40 0x5651f0b1fa40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:30:01.491 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:30:01.497+0000 7f8a5a2b4700 1 -- 172.21.15.52:0/1872853538 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5651f0b27330 con 0x5651f0b25c50 2024-05-27T07:30:01.492 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:30:01.497+0000 7f8a5a2b4700 1 --2- 172.21.15.52:0/1872853538 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5651f0b25c50 0x5651f0b23720 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7f8a4c0186b0 tx=0x7f8a4c023470 comp rx=0 tx=0).ready entity=mon.2 client_cookie=80acceac1fb81140 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:30:01.492 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:30:01.498+0000 7f8a3f7fe700 1 -- 172.21.15.52:0/1872853538 <== mon.2 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8a4c023d80 con 0x5651f0b25c50 2024-05-27T07:30:01.493 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:30:01.498+0000 7f8a3f7fe700 1 -- 172.21.15.52:0/1872853538 <== mon.2 v2:172.21.15.105:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f8a4c02f020 con 0x5651f0b25c50 2024-05-27T07:30:01.493 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:30:01.498+0000 7f8a3f7fe700 1 -- 172.21.15.52:0/1872853538 <== mon.2 v2:172.21.15.105:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8a4c007c50 con 0x5651f0b25c50 2024-05-27T07:30:01.493 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:30:01.500+0000 7f8a6a72cb80 1 -- 172.21.15.52:0/1872853538 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5651f0b25c50 msgr2=0x5651f0b23720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:30:01.493 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:30:01.500+0000 7f8a6a72cb80 1 --2- 172.21.15.52:0/1872853538 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5651f0b25c50 0x5651f0b23720 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7f8a4c0186b0 tx=0x7f8a4c023470 comp rx=0 tx=0).stop 2024-05-27T07:30:01.494 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:30:01.502+0000 7f8a6a72cb80 1 -- 172.21.15.52:0/1872853538 shutdown_connections 2024-05-27T07:30:01.494 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:30:01.502+0000 7f8a6a72cb80 1 --2- 172.21.15.52:0/1872853538 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5651f0b25c50 0x5651f0b23720 unknown :-1 s=CLOSED pgs=142 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:30:01.494 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:30:01.502+0000 7f8a6a72cb80 1 --2- 172.21.15.52:0/1872853538 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x5651f09a1e40 0x5651f0b1fa40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:30:01.495 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:30:01.502+0000 7f8a6a72cb80 1 --2- 172.21.15.52:0/1872853538 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5651f0b1ff80 0x5651f0b28b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:30:01.495 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:30:01.502+0000 7f8a6a72cb80 1 -- 172.21.15.52:0/1872853538 >> 172.21.15.52:0/1872853538 conn(0x5651f09fcaf0 msgr2=0x5651f0b23010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:30:01.498 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:30:01.506+0000 7f8a6a72cb80 1 -- 172.21.15.52:0/1872853538 shutdown_connections 2024-05-27T07:30:01.499 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:30:01.506+0000 7f8a6a72cb80 1 -- 172.21.15.52:0/1872853538 wait complete. 2024-05-27T07:30:01.499 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:30:01.507+0000 7f8a6a72cb80 1 -- start start 2024-05-27T07:30:01.499 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:30:01.507+0000 7f8a6a72cb80 1 Processor -- start 2024-05-27T07:30:01.501 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:30:01.508+0000 7f8a6a72cb80 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5651f09a1e40 0x5651f0b9f6a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:30:01.501 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:30:01.508+0000 7f8a6a72cb80 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5651f0b1ff80 0x5651f0b9fbe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:30:01.501 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:30:01.508+0000 7f8a6a72cb80 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x5651f0b25c50 0x5651f0b31c20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:30:01.501 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:30:01.508+0000 7f8a59ab3700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5651f09a1e40 0x5651f0b9f6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:30:01.502 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:30:01.508+0000 7f8a59ab3700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5651f09a1e40 0x5651f0b9f6a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.173:3300/0 says I am v2:172.21.15.52:53868/0 (socket says 172.21.15.52:53868) 2024-05-27T07:30:01.502 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:30:01.508+0000 7f8a592b2700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5651f0b1ff80 0x5651f0b9fbe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:30:01.502 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:30:01.508+0000 7f8a592b2700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5651f0b1ff80 0x5651f0b9fbe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.52:54122/0 (socket says 172.21.15.52:54122) 2024-05-27T07:30:01.502 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:30:01.508+0000 7f8a6a72cb80 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x7f8a541b7120 con 0x5651f0b25c50 2024-05-27T07:30:01.502 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:30:01.508+0000 7f8a6a72cb80 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f8a541b6a50 con 0x5651f0b1ff80 2024-05-27T07:30:01.502 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:30:01.508+0000 7f8a6a72cb80 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f8a541b77f0 con 0x5651f09a1e40 2024-05-27T07:30:01.503 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:30:01.509+0000 7f8a6a72cb80 10 client.0 _ll_register_callbacks cb 0x5651f09d9920 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T07:30:01.504 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:30:01.510+0000 7f8a59ab3700 1 -- 172.21.15.52:0/3110726027 learned_addr learned my addr 172.21.15.52:0/3110726027 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-05-27T07:30:01.505 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:30:01.510+0000 7f8a5a2b4700 1 --2- 172.21.15.52:0/3110726027 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x5651f0b25c50 0x5651f0b31c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:30:01.505 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:30:01.512+0000 7f8a5a2b4700 1 -- 172.21.15.52:0/3110726027 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5651f09a1e40 msgr2=0x5651f0b9f6a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:30:01.505 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:30:01.512+0000 7f8a5a2b4700 1 --2- 172.21.15.52:0/3110726027 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5651f09a1e40 0x5651f0b9f6a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:30:01.505 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:30:01.512+0000 7f8a5a2b4700 1 -- 172.21.15.52:0/3110726027 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5651f0b1ff80 msgr2=0x5651f0b9fbe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:30:01.505 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:30:01.512+0000 7f8a5a2b4700 1 --2- 172.21.15.52:0/3110726027 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5651f0b1ff80 0x5651f0b9fbe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:30:01.505 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:30:01.512+0000 7f8a5a2b4700 1 -- 172.21.15.52:0/3110726027 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5651f0b323c0 con 0x5651f0b25c50 2024-05-27T07:30:01.506 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:30:01.512+0000 7f8a5a2b4700 1 --2- 172.21.15.52:0/3110726027 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x5651f0b25c50 0x5651f0b31c20 secure :-1 s=READY pgs=633 cs=0 l=1 rev1=1 crypto rx=0x7f8a4c00f010 tx=0x7f8a4c0091f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=554c9d58dcab64d2 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:30:01.506 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:30:01.512+0000 7f8a3d7fa700 10 client.0 ms_handle_connect on v2:172.21.15.52:3300/0 2024-05-27T07:30:01.509 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:30:01.513+0000 7f8a3d7fa700 1 -- 172.21.15.52:0/3110726027 <== mon.0 v2:172.21.15.52:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8a4c0077e0 con 0x5651f0b25c50 2024-05-27T07:30:01.509 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:30:01.513+0000 7f8a3d7fa700 1 -- 172.21.15.52:0/3110726027 <== mon.0 v2:172.21.15.52:3300/0 2 ==== osd_map(110..110 src has 1..110) ==== 9379+0+0 (secure 0 0 0) 0x7f8a4c030070 con 0x5651f0b25c50 2024-05-27T07:30:01.509 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:30:01.513+0000 7f8a6a72cb80 1 -- 172.21.15.52:0/3110726027 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f8a4c02f020 con 0x5651f0b25c50 2024-05-27T07:30:01.509 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:30:01.513+0000 7f8a3d7fa700 1 -- 172.21.15.52:0/3110726027 <== mon.0 v2:172.21.15.52:3300/0 3 ==== mon_get_version_reply(handle=1 version=73) ==== 24+0+0 (secure 0 0 0) 0x7f8a4c023d80 con 0x5651f0b25c50 2024-05-27T07:30:01.509 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:30:01.513+0000 7f8a6a72cb80 10 client.16422 fetch_fsmap learned FSMap version 73 2024-05-27T07:30:01.510 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:30:01.513+0000 7f8a6a72cb80 1 -- 172.21.15.52:0/3110726027 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({config=0+,fsmap=73,monmap=4+}) -- 0x5651f0b416b0 con 0x5651f0b25c50 2024-05-27T07:30:01.510 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:30:01.514+0000 7f8a3d7fa700 1 -- 172.21.15.52:0/3110726027 <== mon.0 v2:172.21.15.52:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f8a4c004020 con 0x5651f0b25c50 2024-05-27T07:30:01.510 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:30:01.515+0000 7f8a3d7fa700 1 -- 172.21.15.52:0/3110726027 <== mon.0 v2:172.21.15.52:3300/0 5 ==== fsmap(e 73) ==== 7773+0+0 (secure 0 0 0) 0x7f8a4c01a440 con 0x5651f0b25c50 2024-05-27T07:30:01.510 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:30:01.516+0000 7f8a6a72cb80 10 client.16422 fetch_fsmap finished waiting for FSMap version 73 2024-05-27T07:30:01.510 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:30:01.516+0000 7f8a6a72cb80 10 client.16422 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-05-27T07:30:01.510 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:30:01.516+0000 7f8a6a72cb80 20 client.16422 populate_metadata read hostname 'smithi052' 2024-05-27T07:30:01.511 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:30:01.516+0000 7f8a6a72cb80 1 --2- 172.21.15.52:0/3110726027 >> [v2:172.21.15.105:6838/500348151,v1:172.21.15.105:6839/500348151] conn(0x5651f0bb5410 0x5651f0bb7830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:30:01.511 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:30:01.516+0000 7f8a6a72cb80 4 client.16422 mds_command: new command op to 24527 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-05-27T07:30:01.511 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:30:01.516+0000 7f8a6a72cb80 1 -- 172.21.15.52:0/3110726027 --> [v2:172.21.15.105:6838/500348151,v1:172.21.15.105:6839/500348151] -- command(tid 0: {"prefix": "damage ls"}) -- 0x7f8a4c02f020 con 0x5651f0bb5410 2024-05-27T07:30:01.511 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:30:01.516+0000 7f8a59ab3700 1 --2- 172.21.15.52:0/3110726027 >> [v2:172.21.15.105:6838/500348151,v1:172.21.15.105:6839/500348151] conn(0x5651f0bb5410 0x5651f0bb7830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:30:01.511 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:30:01.517+0000 7f8a3d7fa700 10 client.16422 ms_handle_connect on v2:172.21.15.105:6838/500348151 2024-05-27T07:30:01.511 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:30:01.517+0000 7f8a59ab3700 1 --2- 172.21.15.52:0/3110726027 >> [v2:172.21.15.105:6838/500348151,v1:172.21.15.105:6839/500348151] conn(0x5651f0bb5410 0x5651f0bb7830 crc :-1 s=READY pgs=100 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=ba61b34edf020e35 server_cookie=9f666f4e7e93c73d in_seq=0 out_seq=0 2024-05-27T07:30:01.514 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:30:01.519+0000 7f8a3d7fa700 1 -- 172.21.15.52:0/3110726027 <== mds.0 v2:172.21.15.105:6838/500348151 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x7f8a4c02f020 con 0x5651f0bb5410 2024-05-27T07:30:01.514 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:30:01.519+0000 7f8a3d7fa700 10 client.16422 handle_command_reply: tid=0 2024-05-27T07:30:01.514 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:30:01.519+0000 7f8a6a72cb80 1 client.16422 shutdown 2024-05-27T07:30:01.515 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:30:01.519+0000 7f8a6a72cb80 1 -- 172.21.15.52:0/3110726027 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x5651f0b25c50 msgr2=0x5651f0b31c20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:30:01.515 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:30:01.519+0000 7f8a6a72cb80 1 --2- 172.21.15.52:0/3110726027 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x5651f0b25c50 0x5651f0b31c20 secure :-1 s=READY pgs=633 cs=0 l=1 rev1=1 crypto rx=0x7f8a4c00f010 tx=0x7f8a4c0091f0 comp rx=0 tx=0).stop 2024-05-27T07:30:01.515 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:30:01.520+0000 7f8a6a72cb80 1 -- 172.21.15.52:0/3110726027 shutdown_connections 2024-05-27T07:30:01.515 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:30:01.520+0000 7f8a6a72cb80 1 --2- 172.21.15.52:0/3110726027 >> [v2:172.21.15.105:6838/500348151,v1:172.21.15.105:6839/500348151] conn(0x5651f0bb5410 0x5651f0bb7830 crc :-1 s=READY pgs=100 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:30:01.515 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:30:01.520+0000 7f8a6a72cb80 1 --2- 172.21.15.52:0/3110726027 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x5651f0b25c50 0x5651f0b31c20 unknown :-1 s=CLOSED pgs=633 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:30:01.515 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:30:01.520+0000 7f8a6a72cb80 1 --2- 172.21.15.52:0/3110726027 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5651f09a1e40 0x5651f0b9f6a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:30:01.516 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:30:01.520+0000 7f8a6a72cb80 1 --2- 172.21.15.52:0/3110726027 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5651f0b1ff80 0x5651f0b9fbe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:30:01.516 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:30:01.520+0000 7f8a6a72cb80 1 -- 172.21.15.52:0/3110726027 >> 172.21.15.52:0/3110726027 conn(0x5651f09fcaf0 msgr2=0x5651f0b26320 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:30:01.516 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:30:01.520+0000 7f8a3d7fa700 0 client.16422 ms_handle_reset on v2:172.21.15.105:6838/500348151 2024-05-27T07:30:01.516 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:30:01.521+0000 7f8a6a72cb80 1 -- 172.21.15.52:0/3110726027 shutdown_connections 2024-05-27T07:30:01.516 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:30:01.521+0000 7f8a6a72cb80 1 -- 172.21.15.52:0/3110726027 wait complete. 2024-05-27T07:30:01.516 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:30:01.521+0000 7f8a6a72cb80 20 client.16422 trim_cache size 0 max 16384 2024-05-27T07:30:01.517 INFO:teuthology.orchestra.run.smithi052.stdout:[] 2024-05-27T07:30:01.519 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:30:01.522+0000 7f8a5bd17700 1 -- 172.21.15.52:0/2753348598 >> [v2:172.21.15.52:6800/1822318384,v1:172.21.15.52:6801/1822318384] conn(0x7f8a400b2b40 msgr2=0x7f8a400b5050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:30:01.520 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:30:01.522+0000 7f8a5bd17700 1 --2- 172.21.15.52:0/2753348598 >> [v2:172.21.15.52:6800/1822318384,v1:172.21.15.52:6801/1822318384] conn(0x7f8a400b2b40 0x7f8a400b5050 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7f8a4c022000 tx=0x7f8a4c020000 comp rx=0 tx=0).stop 2024-05-27T07:30:01.520 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:30:01.522+0000 7f8a5bd17700 1 -- 172.21.15.52:0/2753348598 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f8a54107de0 msgr2=0x7f8a5419a070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:30:01.520 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:30:01.522+0000 7f8a5bd17700 1 --2- 172.21.15.52:0/2753348598 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f8a54107de0 0x7f8a5419a070 secure :-1 s=READY pgs=632 cs=0 l=1 rev1=1 crypto rx=0x7f8a50000c00 tx=0x7f8a5000f950 comp rx=0 tx=0).stop 2024-05-27T07:30:01.520 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:30:01.522+0000 7f8a5bd17700 1 -- 172.21.15.52:0/2753348598 shutdown_connections 2024-05-27T07:30:01.521 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:30:01.522+0000 7f8a5bd17700 1 --2- 172.21.15.52:0/2753348598 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f8a54108730 0x7f8a5419a5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:30:01.521 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:30:01.522+0000 7f8a5bd17700 1 --2- 172.21.15.52:0/2753348598 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f8a541034b0 0x7f8a54199b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:30:01.521 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:30:01.522+0000 7f8a5bd17700 1 --2- 172.21.15.52:0/2753348598 >> [v2:172.21.15.52:6800/1822318384,v1:172.21.15.52:6801/1822318384] conn(0x7f8a400b2b40 0x7f8a400b5050 unknown :-1 s=CLOSED pgs=127 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:30:01.521 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:30:01.522+0000 7f8a5bd17700 1 --2- 172.21.15.52:0/2753348598 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f8a54107de0 0x7f8a5419a070 unknown :-1 s=CLOSED pgs=632 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:30:01.521 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:30:01.522+0000 7f8a5bd17700 1 -- 172.21.15.52:0/2753348598 >> 172.21.15.52:0/2753348598 conn(0x7f8a540fedb0 msgr2=0x7f8a541011d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:30:01.522 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:30:01.523+0000 7f8a5bd17700 1 -- 172.21.15.52:0/2753348598 shutdown_connections 2024-05-27T07:30:01.522 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:30:01.523+0000 7f8a5bd17700 1 -- 172.21.15.52:0/2753348598 wait complete. 2024-05-27T07:30:01.871 INFO:journalctl@ceph.mds.l.smithi173.stdout:May 27 07:30:01 smithi173 systemd[1]: ceph-73df2636-1bf8-11ef-bc9b-c7b262605968@mds.l.service: Scheduled restart job, restart counter is at 5. 2024-05-27T07:30:01.871 INFO:journalctl@ceph.mds.l.smithi173.stdout:May 27 07:30:01 smithi173 systemd[1]: Stopped Ceph mds.l for 73df2636-1bf8-11ef-bc9b-c7b262605968. 2024-05-27T07:30:01.871 INFO:journalctl@ceph.mds.l.smithi173.stdout:May 27 07:30:01 smithi173 systemd[1]: Starting Ceph mds.l for 73df2636-1bf8-11ef-bc9b-c7b262605968... 2024-05-27T07:30:01.871 INFO:journalctl@ceph.mds.l.smithi173.stdout:May 27 07:30:01 smithi173 podman[73867]: 2024-05-27 07:30:01.763625192 +0000 UTC m=+0.019734855 image pull e36ea4fc8a5807e8d2701370e350a0de3856538797f7fba64cfce7e344bd6799 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6 2024-05-27T07:30:01.871 INFO:journalctl@ceph.mds.l.smithi173.stdout:May 27 07:30:01 smithi173 podman[73867]: 2024-05-27 07:30:01.880727154 +0000 UTC m=+0.136836802 container create c45c2b94f64ea76e5af0feebc3984337c003cbad4af4a2f3b0ca75a8c5e3d596 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-l, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, GIT_CLEAN=True, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, org.label-schema.build-date=20240521) 2024-05-27T07:30:02.166 INFO:journalctl@ceph.mds.l.smithi173.stdout:May 27 07:30:02 smithi173 podman[73867]: 2024-05-27 07:30:02.083508702 +0000 UTC m=+0.339618349 container init c45c2b94f64ea76e5af0feebc3984337c003cbad4af4a2f3b0ca75a8c5e3d596 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-l, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, org.label-schema.license=GPLv2, org.label-schema.build-date=20240521, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, ceph=True, io.buildah.version=1.33.5, GIT_CLEAN=True) 2024-05-27T07:30:02.166 INFO:journalctl@ceph.mds.l.smithi173.stdout:May 27 07:30:02 smithi173 podman[73867]: 2024-05-27 07:30:02.086272476 +0000 UTC m=+0.342382127 container start c45c2b94f64ea76e5af0feebc3984337c003cbad4af4a2f3b0ca75a8c5e3d596 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-l, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, ceph=True, GIT_CLEAN=True, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240521) 2024-05-27T07:30:02.166 INFO:journalctl@ceph.mds.l.smithi173.stdout:May 27 07:30:02 smithi173 ceph-mds[73887]: -- 172.21.15.173:0/350221541 <== mon.1 v2:172.21.15.173:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55c70b4d5a40 con 0x55c70b51d600 2024-05-27T07:30:02.166 INFO:journalctl@ceph.mds.l.smithi173.stdout:May 27 07:30:02 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-l[73883]: starting mds.l at 2024-05-27T07:30:02.438 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:30:02 smithi173 ceph-mon[37000]: pgmap v144: 129 pgs: 129 active+clean; 260 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 5.2 MiB/s rd, 2.3 MiB/s wr, 878 op/s 2024-05-27T07:30:02.439 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:30:02 smithi173 ceph-mon[37000]: daemon mds.l restarted 2024-05-27T07:30:02.439 INFO:journalctl@ceph.mds.l.smithi173.stdout:May 27 07:30:02 smithi173 bash[73867]: c45c2b94f64ea76e5af0feebc3984337c003cbad4af4a2f3b0ca75a8c5e3d596 2024-05-27T07:30:02.439 INFO:journalctl@ceph.mds.l.smithi173.stdout:May 27 07:30:02 smithi173 systemd[1]: Started Ceph mds.l for 73df2636-1bf8-11ef-bc9b-c7b262605968. 2024-05-27T07:30:02.439 INFO:journalctl@ceph.mds.l.smithi173.stdout:May 27 07:30:02 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-l[73883]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: In function 'std::set SnapClient::filter(const std::set&) const' thread 7f780faa2700 time 2024-05-27T07:30:02.389255+0000 2024-05-27T07:30:02.439 INFO:journalctl@ceph.mds.l.smithi173.stdout:May 27 07:30:02 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-l[73883]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: 197: FAILED ceph_assert(cached_version > 0) 2024-05-27T07:30:02.439 INFO:journalctl@ceph.mds.l.smithi173.stdout:May 27 07:30:02 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-l[73883]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:30:02.440 INFO:journalctl@ceph.mds.l.smithi173.stdout:May 27 07:30:02 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-l[73883]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x127) [0x7f78222e7ba6] 2024-05-27T07:30:02.440 INFO:journalctl@ceph.mds.l.smithi173.stdout:May 27 07:30:02 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-l[73883]: 2: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7f78222e7dd1] 2024-05-27T07:30:02.440 INFO:journalctl@ceph.mds.l.smithi173.stdout:May 27 07:30:02 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-l[73883]: 3: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x55c708a9723d] 2024-05-27T07:30:02.440 INFO:journalctl@ceph.mds.l.smithi173.stdout:May 27 07:30:02 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-l[73883]: 4: (SnapRealm::build_snap_set() const+0x323) [0x55c708a82971] 2024-05-27T07:30:02.440 INFO:journalctl@ceph.mds.l.smithi173.stdout:May 27 07:30:02 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-l[73883]: 5: (SnapRealm::check_cache() const+0x13e) [0x55c708a80fa0] 2024-05-27T07:30:02.440 INFO:journalctl@ceph.mds.l.smithi173.stdout:May 27 07:30:02 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-l[73883]: 6: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x55c708a81651] 2024-05-27T07:30:02.440 INFO:journalctl@ceph.mds.l.smithi173.stdout:May 27 07:30:02 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-l[73883]: 7: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x55c708a16347] 2024-05-27T07:30:02.440 INFO:journalctl@ceph.mds.l.smithi173.stdout:May 27 07:30:02 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-l[73883]: 8: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x55c70871287e] 2024-05-27T07:30:02.440 INFO:journalctl@ceph.mds.l.smithi173.stdout:May 27 07:30:02 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-l[73883]: 9: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x2702) [0x55c708717e94] 2024-05-27T07:30:02.440 INFO:journalctl@ceph.mds.l.smithi173.stdout:May 27 07:30:02 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-l[73883]: 10: (EUpdate::replay(MDSRank*)+0x63) [0x55c708722969] 2024-05-27T07:30:02.441 INFO:journalctl@ceph.mds.l.smithi173.stdout:May 27 07:30:02 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-l[73883]: 11: (MDLog::_replay_thread()+0x1f38) [0x55c708abb48c] 2024-05-27T07:30:02.441 INFO:journalctl@ceph.mds.l.smithi173.stdout:May 27 07:30:02 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-l[73883]: 12: (MDLog::ReplayThread::entry()+0x11) [0x55c708ac49cb] 2024-05-27T07:30:02.441 INFO:journalctl@ceph.mds.l.smithi173.stdout:May 27 07:30:02 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-l[73883]: 13: (Thread::entry_wrapper()+0x43) [0x7f78222b672d] 2024-05-27T07:30:02.441 INFO:journalctl@ceph.mds.l.smithi173.stdout:May 27 07:30:02 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-l[73883]: 14: (Thread::_entry_func(void*)+0xd) [0x7f78222b6749] 2024-05-27T07:30:02.441 INFO:journalctl@ceph.mds.l.smithi173.stdout:May 27 07:30:02 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-l[73883]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7f7820e5a1ca] 2024-05-27T07:30:02.441 INFO:journalctl@ceph.mds.l.smithi173.stdout:May 27 07:30:02 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-l[73883]: 16: clone() 2024-05-27T07:30:02.441 INFO:journalctl@ceph.mds.l.smithi173.stdout:May 27 07:30:02 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-l[73883]: 2024-05-27T07:30:02.390+0000 7f780faa2700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: In function 'std::set SnapClient::filter(const std::set&) const' thread 7f780faa2700 time 2024-05-27T07:30:02.389255+0000 2024-05-27T07:30:02.441 INFO:journalctl@ceph.mds.l.smithi173.stdout:May 27 07:30:02 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-l[73883]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: 197: FAILED ceph_assert(cached_version > 0) 2024-05-27T07:30:02.441 INFO:journalctl@ceph.mds.l.smithi173.stdout:May 27 07:30:02 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-l[73883]: 2024-05-27T07:30:02.441 INFO:journalctl@ceph.mds.l.smithi173.stdout:May 27 07:30:02 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-l[73883]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:30:02.442 INFO:journalctl@ceph.mds.l.smithi173.stdout:May 27 07:30:02 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-l[73883]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x127) [0x7f78222e7ba6] 2024-05-27T07:30:02.442 INFO:journalctl@ceph.mds.l.smithi173.stdout:May 27 07:30:02 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-l[73883]: 2: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7f78222e7dd1] 2024-05-27T07:30:02.442 INFO:journalctl@ceph.mds.l.smithi173.stdout:May 27 07:30:02 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-l[73883]: 3: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x55c708a9723d] 2024-05-27T07:30:02.442 INFO:journalctl@ceph.mds.l.smithi173.stdout:May 27 07:30:02 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-l[73883]: 4: (SnapRealm::build_snap_set() const+0x323) [0x55c708a82971] 2024-05-27T07:30:02.442 INFO:journalctl@ceph.mds.l.smithi173.stdout:May 27 07:30:02 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-l[73883]: 5: (SnapRealm::check_cache() const+0x13e) [0x55c708a80fa0] 2024-05-27T07:30:02.442 INFO:journalctl@ceph.mds.l.smithi173.stdout:May 27 07:30:02 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-l[73883]: 6: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x55c708a81651] 2024-05-27T07:30:02.442 INFO:journalctl@ceph.mds.l.smithi173.stdout:May 27 07:30:02 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-l[73883]: 7: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x55c708a16347] 2024-05-27T07:30:02.442 INFO:journalctl@ceph.mds.l.smithi173.stdout:May 27 07:30:02 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-l[73883]: 8: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x55c70871287e] 2024-05-27T07:30:02.442 INFO:journalctl@ceph.mds.l.smithi173.stdout:May 27 07:30:02 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-l[73883]: 9: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x2702) [0x55c708717e94] 2024-05-27T07:30:02.442 INFO:journalctl@ceph.mds.l.smithi173.stdout:May 27 07:30:02 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-l[73883]: 10: (EUpdate::replay(MDSRank*)+0x63) [0x55c708722969] 2024-05-27T07:30:02.442 INFO:journalctl@ceph.mds.l.smithi173.stdout:May 27 07:30:02 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-l[73883]: 11: (MDLog::_replay_thread()+0x1f38) [0x55c708abb48c] 2024-05-27T07:30:02.443 INFO:journalctl@ceph.mds.l.smithi173.stdout:May 27 07:30:02 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-l[73883]: 12: (MDLog::ReplayThread::entry()+0x11) [0x55c708ac49cb] 2024-05-27T07:30:02.443 INFO:journalctl@ceph.mds.l.smithi173.stdout:May 27 07:30:02 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-l[73883]: 13: (Thread::entry_wrapper()+0x43) [0x7f78222b672d] 2024-05-27T07:30:02.443 INFO:journalctl@ceph.mds.l.smithi173.stdout:May 27 07:30:02 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-l[73883]: 14: (Thread::_entry_func(void*)+0xd) [0x7f78222b6749] 2024-05-27T07:30:02.443 INFO:journalctl@ceph.mds.l.smithi173.stdout:May 27 07:30:02 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-l[73883]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7f7820e5a1ca] 2024-05-27T07:30:02.443 INFO:journalctl@ceph.mds.l.smithi173.stdout:May 27 07:30:02 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-l[73883]: 16: clone() 2024-05-27T07:30:02.443 INFO:journalctl@ceph.mds.l.smithi173.stdout:May 27 07:30:02 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-l[73883]: 2024-05-27T07:30:02.443 INFO:journalctl@ceph.mds.l.smithi173.stdout:May 27 07:30:02 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-l[73883]: *** Caught signal (Aborted) ** 2024-05-27T07:30:02.443 INFO:journalctl@ceph.mds.l.smithi173.stdout:May 27 07:30:02 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-l[73883]: in thread 7f780faa2700 thread_name:md_log_replay 2024-05-27T07:30:02.443 INFO:journalctl@ceph.mds.l.smithi173.stdout:May 27 07:30:02 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-l[73883]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:30:02.443 INFO:journalctl@ceph.mds.l.smithi173.stdout:May 27 07:30:02 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-l[73883]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f7820e64d20] 2024-05-27T07:30:02.444 INFO:journalctl@ceph.mds.l.smithi173.stdout:May 27 07:30:02 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-l[73883]: 2: gsignal() 2024-05-27T07:30:02.444 INFO:journalctl@ceph.mds.l.smithi173.stdout:May 27 07:30:02 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-l[73883]: 3: abort() 2024-05-27T07:30:02.444 INFO:journalctl@ceph.mds.l.smithi173.stdout:May 27 07:30:02 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-l[73883]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x26e) [0x7f78222e7ced] 2024-05-27T07:30:02.445 INFO:journalctl@ceph.mds.l.smithi173.stdout:May 27 07:30:02 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-l[73883]: 5: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7f78222e7dd1] 2024-05-27T07:30:02.445 INFO:journalctl@ceph.mds.l.smithi173.stdout:May 27 07:30:02 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-l[73883]: 6: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x55c708a9723d] 2024-05-27T07:30:02.445 INFO:journalctl@ceph.mds.l.smithi173.stdout:May 27 07:30:02 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-l[73883]: 7: (SnapRealm::build_snap_set() const+0x323) [0x55c708a82971] 2024-05-27T07:30:02.445 INFO:journalctl@ceph.mds.l.smithi173.stdout:May 27 07:30:02 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-l[73883]: 8: (SnapRealm::check_cache() const+0x13e) [0x55c708a80fa0] 2024-05-27T07:30:02.445 INFO:journalctl@ceph.mds.l.smithi173.stdout:May 27 07:30:02 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-l[73883]: 9: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x55c708a81651] 2024-05-27T07:30:02.445 INFO:journalctl@ceph.mds.l.smithi173.stdout:May 27 07:30:02 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-l[73883]: 10: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x55c708a16347] 2024-05-27T07:30:02.445 INFO:journalctl@ceph.mds.l.smithi173.stdout:May 27 07:30:02 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-l[73883]: 11: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x55c70871287e] 2024-05-27T07:30:02.445 INFO:journalctl@ceph.mds.l.smithi173.stdout:May 27 07:30:02 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-l[73883]: 12: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x2702) [0x55c708717e94] 2024-05-27T07:30:02.445 INFO:journalctl@ceph.mds.l.smithi173.stdout:May 27 07:30:02 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-l[73883]: 13: (EUpdate::replay(MDSRank*)+0x63) [0x55c708722969] 2024-05-27T07:30:02.445 INFO:journalctl@ceph.mds.l.smithi173.stdout:May 27 07:30:02 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-l[73883]: 14: (MDLog::_replay_thread()+0x1f38) [0x55c708abb48c] 2024-05-27T07:30:02.446 INFO:journalctl@ceph.mds.l.smithi173.stdout:May 27 07:30:02 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-l[73883]: 15: (MDLog::ReplayThread::entry()+0x11) [0x55c708ac49cb] 2024-05-27T07:30:02.446 INFO:journalctl@ceph.mds.l.smithi173.stdout:May 27 07:30:02 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-l[73883]: 16: (Thread::entry_wrapper()+0x43) [0x7f78222b672d] 2024-05-27T07:30:02.446 INFO:journalctl@ceph.mds.l.smithi173.stdout:May 27 07:30:02 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-l[73883]: 17: (Thread::_entry_func(void*)+0xd) [0x7f78222b6749] 2024-05-27T07:30:02.446 INFO:journalctl@ceph.mds.l.smithi173.stdout:May 27 07:30:02 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-l[73883]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7f7820e5a1ca] 2024-05-27T07:30:02.446 INFO:journalctl@ceph.mds.l.smithi173.stdout:May 27 07:30:02 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-l[73883]: 19: clone() 2024-05-27T07:30:02.446 INFO:journalctl@ceph.mds.l.smithi173.stdout:May 27 07:30:02 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-l[73883]: 2024-05-27T07:30:02.392+0000 7f780faa2700 -1 *** Caught signal (Aborted) ** 2024-05-27T07:30:02.446 INFO:journalctl@ceph.mds.l.smithi173.stdout:May 27 07:30:02 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-l[73883]: in thread 7f780faa2700 thread_name:md_log_replay 2024-05-27T07:30:02.446 INFO:journalctl@ceph.mds.l.smithi173.stdout:May 27 07:30:02 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-l[73883]: 2024-05-27T07:30:02.446 INFO:journalctl@ceph.mds.l.smithi173.stdout:May 27 07:30:02 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-l[73883]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:30:02.446 INFO:journalctl@ceph.mds.l.smithi173.stdout:May 27 07:30:02 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-l[73883]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f7820e64d20] 2024-05-27T07:30:02.447 INFO:journalctl@ceph.mds.l.smithi173.stdout:May 27 07:30:02 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-l[73883]: 2: gsignal() 2024-05-27T07:30:02.447 INFO:journalctl@ceph.mds.l.smithi173.stdout:May 27 07:30:02 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-l[73883]: 3: abort() 2024-05-27T07:30:02.447 INFO:journalctl@ceph.mds.l.smithi173.stdout:May 27 07:30:02 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-l[73883]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x26e) [0x7f78222e7ced] 2024-05-27T07:30:02.447 INFO:journalctl@ceph.mds.l.smithi173.stdout:May 27 07:30:02 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-l[73883]: 5: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7f78222e7dd1] 2024-05-27T07:30:02.447 INFO:journalctl@ceph.mds.l.smithi173.stdout:May 27 07:30:02 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-l[73883]: 6: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x55c708a9723d] 2024-05-27T07:30:02.447 INFO:journalctl@ceph.mds.l.smithi173.stdout:May 27 07:30:02 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-l[73883]: 7: (SnapRealm::build_snap_set() const+0x323) [0x55c708a82971] 2024-05-27T07:30:02.447 INFO:journalctl@ceph.mds.l.smithi173.stdout:May 27 07:30:02 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-l[73883]: 8: (SnapRealm::check_cache() const+0x13e) [0x55c708a80fa0] 2024-05-27T07:30:02.447 INFO:journalctl@ceph.mds.l.smithi173.stdout:May 27 07:30:02 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-l[73883]: 9: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x55c708a81651] 2024-05-27T07:30:02.447 INFO:journalctl@ceph.mds.l.smithi173.stdout:May 27 07:30:02 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-l[73883]: 10: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x55c708a16347] 2024-05-27T07:30:02.447 INFO:journalctl@ceph.mds.l.smithi173.stdout:May 27 07:30:02 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-l[73883]: 11: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x55c70871287e] 2024-05-27T07:30:02.448 INFO:journalctl@ceph.mds.l.smithi173.stdout:May 27 07:30:02 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-l[73883]: 12: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x2702) [0x55c708717e94] 2024-05-27T07:30:02.448 INFO:journalctl@ceph.mds.l.smithi173.stdout:May 27 07:30:02 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-l[73883]: 13: (EUpdate::replay(MDSRank*)+0x63) [0x55c708722969] 2024-05-27T07:30:02.448 INFO:journalctl@ceph.mds.l.smithi173.stdout:May 27 07:30:02 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-l[73883]: 14: (MDLog::_replay_thread()+0x1f38) [0x55c708abb48c] 2024-05-27T07:30:02.448 INFO:journalctl@ceph.mds.l.smithi173.stdout:May 27 07:30:02 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-l[73883]: 15: (MDLog::ReplayThread::entry()+0x11) [0x55c708ac49cb] 2024-05-27T07:30:02.448 INFO:journalctl@ceph.mds.l.smithi173.stdout:May 27 07:30:02 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-l[73883]: 16: (Thread::entry_wrapper()+0x43) [0x7f78222b672d] 2024-05-27T07:30:02.448 INFO:journalctl@ceph.mds.l.smithi173.stdout:May 27 07:30:02 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-l[73883]: 17: (Thread::_entry_func(void*)+0xd) [0x7f78222b6749] 2024-05-27T07:30:02.448 INFO:journalctl@ceph.mds.l.smithi173.stdout:May 27 07:30:02 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-l[73883]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7f7820e5a1ca] 2024-05-27T07:30:02.448 INFO:journalctl@ceph.mds.l.smithi173.stdout:May 27 07:30:02 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-l[73883]: 19: clone() 2024-05-27T07:30:02.448 INFO:journalctl@ceph.mds.l.smithi173.stdout:May 27 07:30:02 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-l[73883]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-05-27T07:30:02.448 INFO:journalctl@ceph.mds.l.smithi173.stdout:May 27 07:30:02 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-l[73883]: 2024-05-27T07:30:02.448 INFO:journalctl@ceph.mds.l.smithi173.stdout:May 27 07:30:02 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-l[73883]: -1> 2024-05-27T07:30:02.390+0000 7f780faa2700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: In function 'std::set SnapClient::filter(const std::set&) const' thread 7f780faa2700 time 2024-05-27T07:30:02.389255+0000 2024-05-27T07:30:02.448 INFO:journalctl@ceph.mds.l.smithi173.stdout:May 27 07:30:02 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-l[73883]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: 197: FAILED ceph_assert(cached_version > 0) 2024-05-27T07:30:02.449 INFO:journalctl@ceph.mds.l.smithi173.stdout:May 27 07:30:02 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-l[73883]: 2024-05-27T07:30:02.449 INFO:journalctl@ceph.mds.l.smithi173.stdout:May 27 07:30:02 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-l[73883]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:30:02.449 INFO:journalctl@ceph.mds.l.smithi173.stdout:May 27 07:30:02 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-l[73883]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x127) [0x7f78222e7ba6] 2024-05-27T07:30:02.449 INFO:journalctl@ceph.mds.l.smithi173.stdout:May 27 07:30:02 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-l[73883]: 2: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7f78222e7dd1] 2024-05-27T07:30:02.449 INFO:journalctl@ceph.mds.l.smithi173.stdout:May 27 07:30:02 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-l[73883]: 3: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x55c708a9723d] 2024-05-27T07:30:02.449 INFO:journalctl@ceph.mds.l.smithi173.stdout:May 27 07:30:02 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-l[73883]: 4: (SnapRealm::build_snap_set() const+0x323) [0x55c708a82971] 2024-05-27T07:30:02.449 INFO:journalctl@ceph.mds.l.smithi173.stdout:May 27 07:30:02 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-l[73883]: 5: (SnapRealm::check_cache() const+0x13e) [0x55c708a80fa0] 2024-05-27T07:30:02.449 INFO:journalctl@ceph.mds.l.smithi173.stdout:May 27 07:30:02 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-l[73883]: 6: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x55c708a81651] 2024-05-27T07:30:02.449 INFO:journalctl@ceph.mds.l.smithi173.stdout:May 27 07:30:02 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-l[73883]: 7: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x55c708a16347] 2024-05-27T07:30:02.449 INFO:journalctl@ceph.mds.l.smithi173.stdout:May 27 07:30:02 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-l[73883]: 8: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x55c70871287e] 2024-05-27T07:30:02.450 INFO:journalctl@ceph.mds.l.smithi173.stdout:May 27 07:30:02 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-l[73883]: 9: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x2702) [0x55c708717e94] 2024-05-27T07:30:02.450 INFO:journalctl@ceph.mds.l.smithi173.stdout:May 27 07:30:02 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-l[73883]: 10: (EUpdate::replay(MDSRank*)+0x63) [0x55c708722969] 2024-05-27T07:30:02.450 INFO:journalctl@ceph.mds.l.smithi173.stdout:May 27 07:30:02 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-l[73883]: 11: (MDLog::_replay_thread()+0x1f38) [0x55c708abb48c] 2024-05-27T07:30:02.450 INFO:journalctl@ceph.mds.l.smithi173.stdout:May 27 07:30:02 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-l[73883]: 12: (MDLog::ReplayThread::entry()+0x11) [0x55c708ac49cb] 2024-05-27T07:30:02.450 INFO:journalctl@ceph.mds.l.smithi173.stdout:May 27 07:30:02 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-l[73883]: 13: (Thread::entry_wrapper()+0x43) [0x7f78222b672d] 2024-05-27T07:30:02.450 INFO:journalctl@ceph.mds.l.smithi173.stdout:May 27 07:30:02 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-l[73883]: 14: (Thread::_entry_func(void*)+0xd) [0x7f78222b6749] 2024-05-27T07:30:02.450 INFO:journalctl@ceph.mds.l.smithi173.stdout:May 27 07:30:02 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-l[73883]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7f7820e5a1ca] 2024-05-27T07:30:02.450 INFO:journalctl@ceph.mds.l.smithi173.stdout:May 27 07:30:02 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-l[73883]: 16: clone() 2024-05-27T07:30:02.450 INFO:journalctl@ceph.mds.l.smithi173.stdout:May 27 07:30:02 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-l[73883]: 2024-05-27T07:30:02.450 INFO:journalctl@ceph.mds.l.smithi173.stdout:May 27 07:30:02 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-l[73883]: 0> 2024-05-27T07:30:02.392+0000 7f780faa2700 -1 *** Caught signal (Aborted) ** 2024-05-27T07:30:02.450 INFO:journalctl@ceph.mds.l.smithi173.stdout:May 27 07:30:02 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-l[73883]: in thread 7f780faa2700 thread_name:md_log_replay 2024-05-27T07:30:02.450 INFO:journalctl@ceph.mds.l.smithi173.stdout:May 27 07:30:02 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-l[73883]: 2024-05-27T07:30:02.451 INFO:journalctl@ceph.mds.l.smithi173.stdout:May 27 07:30:02 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-l[73883]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:30:02.451 INFO:journalctl@ceph.mds.l.smithi173.stdout:May 27 07:30:02 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-l[73883]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f7820e64d20] 2024-05-27T07:30:02.451 INFO:journalctl@ceph.mds.l.smithi173.stdout:May 27 07:30:02 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-l[73883]: 2: gsignal() 2024-05-27T07:30:02.451 INFO:journalctl@ceph.mds.l.smithi173.stdout:May 27 07:30:02 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-l[73883]: 3: abort() 2024-05-27T07:30:02.451 INFO:journalctl@ceph.mds.l.smithi173.stdout:May 27 07:30:02 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-l[73883]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x26e) [0x7f78222e7ced] 2024-05-27T07:30:02.451 INFO:journalctl@ceph.mds.l.smithi173.stdout:May 27 07:30:02 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-l[73883]: 5: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7f78222e7dd1] 2024-05-27T07:30:02.451 INFO:journalctl@ceph.mds.l.smithi173.stdout:May 27 07:30:02 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-l[73883]: 6: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x55c708a9723d] 2024-05-27T07:30:02.451 INFO:journalctl@ceph.mds.l.smithi173.stdout:May 27 07:30:02 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-l[73883]: 7: (SnapRealm::build_snap_set() const+0x323) [0x55c708a82971] 2024-05-27T07:30:02.451 INFO:journalctl@ceph.mds.l.smithi173.stdout:May 27 07:30:02 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-l[73883]: 8: (SnapRealm::check_cache() const+0x13e) [0x55c708a80fa0] 2024-05-27T07:30:02.451 INFO:journalctl@ceph.mds.l.smithi173.stdout:May 27 07:30:02 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-l[73883]: 9: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x55c708a81651] 2024-05-27T07:30:02.452 INFO:journalctl@ceph.mds.l.smithi173.stdout:May 27 07:30:02 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-l[73883]: 10: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x55c708a16347] 2024-05-27T07:30:02.452 INFO:journalctl@ceph.mds.l.smithi173.stdout:May 27 07:30:02 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-l[73883]: 11: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x55c70871287e] 2024-05-27T07:30:02.452 INFO:journalctl@ceph.mds.l.smithi173.stdout:May 27 07:30:02 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-l[73883]: 12: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x2702) [0x55c708717e94] 2024-05-27T07:30:02.452 INFO:journalctl@ceph.mds.l.smithi173.stdout:May 27 07:30:02 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-l[73883]: 13: (EUpdate::replay(MDSRank*)+0x63) [0x55c708722969] 2024-05-27T07:30:02.452 INFO:journalctl@ceph.mds.l.smithi173.stdout:May 27 07:30:02 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-l[73883]: 14: (MDLog::_replay_thread()+0x1f38) [0x55c708abb48c] 2024-05-27T07:30:02.452 INFO:journalctl@ceph.mds.l.smithi173.stdout:May 27 07:30:02 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-l[73883]: 15: (MDLog::ReplayThread::entry()+0x11) [0x55c708ac49cb] 2024-05-27T07:30:02.452 INFO:journalctl@ceph.mds.l.smithi173.stdout:May 27 07:30:02 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-l[73883]: 16: (Thread::entry_wrapper()+0x43) [0x7f78222b672d] 2024-05-27T07:30:02.452 INFO:journalctl@ceph.mds.l.smithi173.stdout:May 27 07:30:02 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-l[73883]: 17: (Thread::_entry_func(void*)+0xd) [0x7f78222b6749] 2024-05-27T07:30:02.452 INFO:journalctl@ceph.mds.l.smithi173.stdout:May 27 07:30:02 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-l[73883]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7f7820e5a1ca] 2024-05-27T07:30:02.452 INFO:journalctl@ceph.mds.l.smithi173.stdout:May 27 07:30:02 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-l[73883]: 19: clone() 2024-05-27T07:30:02.452 INFO:journalctl@ceph.mds.l.smithi173.stdout:May 27 07:30:02 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-l[73883]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-05-27T07:30:02.452 INFO:journalctl@ceph.mds.l.smithi173.stdout:May 27 07:30:02 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-l[73883]: 2024-05-27T07:30:02.452 INFO:journalctl@ceph.mds.l.smithi173.stdout:May 27 07:30:02 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-l[73883]: -1> 2024-05-27T07:30:02.390+0000 7f780faa2700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: In function 'std::set SnapClient::filter(const std::set&) const' thread 7f780faa2700 time 2024-05-27T07:30:02.389255+0000 2024-05-27T07:30:02.453 INFO:journalctl@ceph.mds.l.smithi173.stdout:May 27 07:30:02 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-l[73883]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: 197: FAILED ceph_assert(cached_version > 0) 2024-05-27T07:30:02.453 INFO:journalctl@ceph.mds.l.smithi173.stdout:May 27 07:30:02 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-l[73883]: 2024-05-27T07:30:02.453 INFO:journalctl@ceph.mds.l.smithi173.stdout:May 27 07:30:02 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-l[73883]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:30:02.453 INFO:journalctl@ceph.mds.l.smithi173.stdout:May 27 07:30:02 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-l[73883]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x127) [0x7f78222e7ba6] 2024-05-27T07:30:02.453 INFO:journalctl@ceph.mds.l.smithi173.stdout:May 27 07:30:02 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-l[73883]: 2: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7f78222e7dd1] 2024-05-27T07:30:02.453 INFO:journalctl@ceph.mds.l.smithi173.stdout:May 27 07:30:02 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-l[73883]: 3: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x55c708a9723d] 2024-05-27T07:30:02.453 INFO:journalctl@ceph.mds.l.smithi173.stdout:May 27 07:30:02 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-l[73883]: 4: (SnapRealm::build_snap_set() const+0x323) [0x55c708a82971] 2024-05-27T07:30:02.453 INFO:journalctl@ceph.mds.l.smithi173.stdout:May 27 07:30:02 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-l[73883]: 5: (SnapRealm::check_cache() const+0x13e) [0x55c708a80fa0] 2024-05-27T07:30:02.453 INFO:journalctl@ceph.mds.l.smithi173.stdout:May 27 07:30:02 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-l[73883]: 6: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x55c708a81651] 2024-05-27T07:30:02.453 INFO:journalctl@ceph.mds.l.smithi173.stdout:May 27 07:30:02 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-l[73883]: 7: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x55c708a16347] 2024-05-27T07:30:02.453 INFO:journalctl@ceph.mds.l.smithi173.stdout:May 27 07:30:02 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-l[73883]: 8: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x55c70871287e] 2024-05-27T07:30:02.454 INFO:journalctl@ceph.mds.l.smithi173.stdout:May 27 07:30:02 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-l[73883]: 9: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x2702) [0x55c708717e94] 2024-05-27T07:30:02.454 INFO:journalctl@ceph.mds.l.smithi173.stdout:May 27 07:30:02 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-l[73883]: 10: (EUpdate::replay(MDSRank*)+0x63) [0x55c708722969] 2024-05-27T07:30:02.454 INFO:journalctl@ceph.mds.l.smithi173.stdout:May 27 07:30:02 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-l[73883]: 11: (MDLog::_replay_thread()+0x1f38) [0x55c708abb48c] 2024-05-27T07:30:02.454 INFO:journalctl@ceph.mds.l.smithi173.stdout:May 27 07:30:02 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-l[73883]: 12: (MDLog::ReplayThread::entry()+0x11) [0x55c708ac49cb] 2024-05-27T07:30:02.454 INFO:journalctl@ceph.mds.l.smithi173.stdout:May 27 07:30:02 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-l[73883]: 13: (Thread::entry_wrapper()+0x43) [0x7f78222b672d] 2024-05-27T07:30:02.454 INFO:journalctl@ceph.mds.l.smithi173.stdout:May 27 07:30:02 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-l[73883]: 14: (Thread::_entry_func(void*)+0xd) [0x7f78222b6749] 2024-05-27T07:30:02.454 INFO:journalctl@ceph.mds.l.smithi173.stdout:May 27 07:30:02 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-l[73883]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7f7820e5a1ca] 2024-05-27T07:30:02.454 INFO:journalctl@ceph.mds.l.smithi173.stdout:May 27 07:30:02 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-l[73883]: 16: clone() 2024-05-27T07:30:02.454 INFO:journalctl@ceph.mds.l.smithi173.stdout:May 27 07:30:02 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-l[73883]: 2024-05-27T07:30:02.454 INFO:journalctl@ceph.mds.l.smithi173.stdout:May 27 07:30:02 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-l[73883]: 0> 2024-05-27T07:30:02.392+0000 7f780faa2700 -1 *** Caught signal (Aborted) ** 2024-05-27T07:30:02.455 INFO:journalctl@ceph.mds.l.smithi173.stdout:May 27 07:30:02 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-l[73883]: in thread 7f780faa2700 thread_name:md_log_replay 2024-05-27T07:30:02.455 INFO:journalctl@ceph.mds.l.smithi173.stdout:May 27 07:30:02 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-l[73883]: 2024-05-27T07:30:02.455 INFO:journalctl@ceph.mds.l.smithi173.stdout:May 27 07:30:02 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-l[73883]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:30:02.455 INFO:journalctl@ceph.mds.l.smithi173.stdout:May 27 07:30:02 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-l[73883]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f7820e64d20] 2024-05-27T07:30:02.455 INFO:journalctl@ceph.mds.l.smithi173.stdout:May 27 07:30:02 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-l[73883]: 2: gsignal() 2024-05-27T07:30:02.455 INFO:journalctl@ceph.mds.l.smithi173.stdout:May 27 07:30:02 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-l[73883]: 3: abort() 2024-05-27T07:30:02.455 INFO:journalctl@ceph.mds.l.smithi173.stdout:May 27 07:30:02 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-l[73883]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x26e) [0x7f78222e7ced] 2024-05-27T07:30:02.455 INFO:journalctl@ceph.mds.l.smithi173.stdout:May 27 07:30:02 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-l[73883]: 5: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7f78222e7dd1] 2024-05-27T07:30:02.455 INFO:journalctl@ceph.mds.l.smithi173.stdout:May 27 07:30:02 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-l[73883]: 6: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x55c708a9723d] 2024-05-27T07:30:02.455 INFO:journalctl@ceph.mds.l.smithi173.stdout:May 27 07:30:02 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-l[73883]: 7: (SnapRealm::build_snap_set() const+0x323) [0x55c708a82971] 2024-05-27T07:30:02.455 INFO:journalctl@ceph.mds.l.smithi173.stdout:May 27 07:30:02 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-l[73883]: 8: (SnapRealm::check_cache() const+0x13e) [0x55c708a80fa0] 2024-05-27T07:30:02.456 INFO:journalctl@ceph.mds.l.smithi173.stdout:May 27 07:30:02 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-l[73883]: 9: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x55c708a81651] 2024-05-27T07:30:02.456 INFO:journalctl@ceph.mds.l.smithi173.stdout:May 27 07:30:02 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-l[73883]: 10: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x55c708a16347] 2024-05-27T07:30:02.456 INFO:journalctl@ceph.mds.l.smithi173.stdout:May 27 07:30:02 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-l[73883]: 11: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x55c70871287e] 2024-05-27T07:30:02.456 INFO:journalctl@ceph.mds.l.smithi173.stdout:May 27 07:30:02 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-l[73883]: 12: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x2702) [0x55c708717e94] 2024-05-27T07:30:02.456 INFO:journalctl@ceph.mds.l.smithi173.stdout:May 27 07:30:02 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-l[73883]: 13: (EUpdate::replay(MDSRank*)+0x63) [0x55c708722969] 2024-05-27T07:30:02.456 INFO:journalctl@ceph.mds.l.smithi173.stdout:May 27 07:30:02 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-l[73883]: 14: (MDLog::_replay_thread()+0x1f38) [0x55c708abb48c] 2024-05-27T07:30:02.456 INFO:journalctl@ceph.mds.l.smithi173.stdout:May 27 07:30:02 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-l[73883]: 15: (MDLog::ReplayThread::entry()+0x11) [0x55c708ac49cb] 2024-05-27T07:30:02.456 INFO:journalctl@ceph.mds.l.smithi173.stdout:May 27 07:30:02 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-l[73883]: 16: (Thread::entry_wrapper()+0x43) [0x7f78222b672d] 2024-05-27T07:30:02.456 INFO:journalctl@ceph.mds.l.smithi173.stdout:May 27 07:30:02 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-l[73883]: 17: (Thread::_entry_func(void*)+0xd) [0x7f78222b6749] 2024-05-27T07:30:02.456 INFO:journalctl@ceph.mds.l.smithi173.stdout:May 27 07:30:02 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-l[73883]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7f7820e5a1ca] 2024-05-27T07:30:02.456 INFO:journalctl@ceph.mds.l.smithi173.stdout:May 27 07:30:02 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-l[73883]: 19: clone() 2024-05-27T07:30:02.457 INFO:journalctl@ceph.mds.l.smithi173.stdout:May 27 07:30:02 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-l[73883]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-05-27T07:30:02.457 INFO:journalctl@ceph.mds.l.smithi173.stdout:May 27 07:30:02 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-l[73883]: 2024-05-27T07:30:02.457 INFO:journalctl@ceph.mds.l.smithi173.stdout:May 27 07:30:02 smithi173 podman[73918]: 2024-05-27 07:30:02.448485232 +0000 UTC m=+0.017822722 container died c45c2b94f64ea76e5af0feebc3984337c003cbad4af4a2f3b0ca75a8c5e3d596 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-l, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240521, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, GIT_CLEAN=True, io.buildah.version=1.33.5, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS) 2024-05-27T07:30:02.690 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:30:02 smithi105 ceph-mon[35977]: pgmap v144: 129 pgs: 129 active+clean; 260 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 5.2 MiB/s rd, 2.3 MiB/s wr, 878 op/s 2024-05-27T07:30:02.691 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:30:02 smithi105 ceph-mon[35977]: daemon mds.l restarted 2024-05-27T07:30:02.730 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:30:02 smithi052 ceph-mon[31955]: pgmap v144: 129 pgs: 129 active+clean; 260 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 5.2 MiB/s rd, 2.3 MiB/s wr, 878 op/s 2024-05-27T07:30:02.730 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:30:02 smithi052 ceph-mon[31955]: daemon mds.l restarted 2024-05-27T07:30:03.260 INFO:journalctl@ceph.mds.l.smithi173.stdout:May 27 07:30:03 smithi173 podman[73918]: 2024-05-27 07:30:03.07315309 +0000 UTC m=+0.642490572 container remove c45c2b94f64ea76e5af0feebc3984337c003cbad4af4a2f3b0ca75a8c5e3d596 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-l, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240521, GIT_CLEAN=True, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, ceph=True, GIT_BRANCH=HEAD, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-05-27T07:30:03.261 INFO:journalctl@ceph.mds.l.smithi173.stdout:May 27 07:30:03 smithi173 systemd[1]: ceph-73df2636-1bf8-11ef-bc9b-c7b262605968@mds.l.service: Main process exited, code=exited, status=134/n/a 2024-05-27T07:30:03.478 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-05-27T07:30:03.478 DEBUG:teuthology.orchestra.run.smithi052:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:38f9d900c83c63799fbdbe61acc9a11b0d3554a6 shell --fsid 73df2636-1bf8-11ef-bc9b-c7b262605968 -- ceph tell mds.1:0 scrub start / force,recursive 2024-05-27T07:30:03.603 INFO:journalctl@ceph.mds.l.smithi173.stdout:May 27 07:30:03 smithi173 systemd[1]: ceph-73df2636-1bf8-11ef-bc9b-c7b262605968@mds.l.service: Failed with result 'exit-code'. 2024-05-27T07:30:03.603 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:30:03 smithi173 ceph-mon[37000]: osdmap e111: 12 total, 12 up, 12 in 2024-05-27T07:30:03.603 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:30:03 smithi173 ceph-mon[37000]: mds.? [v2:172.21.15.173:6832/3278129804,v1:172.21.15.173:6833/3278129804] up:boot 2024-05-27T07:30:03.603 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:30:03 smithi173 ceph-mon[37000]: fsmap cephfs:5 4 up:standby-replay 2 up:standby 5 up:active 2024-05-27T07:30:03.603 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:30:03 smithi173 ceph-mon[37000]: from='mgr.15333 172.21.15.52:0/416389082' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "l"}]: dispatch 2024-05-27T07:30:03.603 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:30:03 smithi173 ceph-mon[37000]: fsmap cephfs:5 5 up:standby-replay 1 up:standby 5 up:active 2024-05-27T07:30:03.690 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:30:03 smithi105 ceph-mon[35977]: osdmap e111: 12 total, 12 up, 12 in 2024-05-27T07:30:03.691 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:30:03 smithi105 ceph-mon[35977]: mds.? [v2:172.21.15.173:6832/3278129804,v1:172.21.15.173:6833/3278129804] up:boot 2024-05-27T07:30:03.691 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:30:03 smithi105 ceph-mon[35977]: fsmap cephfs:5 4 up:standby-replay 2 up:standby 5 up:active 2024-05-27T07:30:03.691 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:30:03 smithi105 ceph-mon[35977]: from='mgr.15333 172.21.15.52:0/416389082' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "l"}]: dispatch 2024-05-27T07:30:03.691 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:30:03 smithi105 ceph-mon[35977]: fsmap cephfs:5 5 up:standby-replay 1 up:standby 5 up:active 2024-05-27T07:30:03.730 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:30:03 smithi052 ceph-mon[31955]: osdmap e111: 12 total, 12 up, 12 in 2024-05-27T07:30:03.730 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:30:03 smithi052 ceph-mon[31955]: mds.? [v2:172.21.15.173:6832/3278129804,v1:172.21.15.173:6833/3278129804] up:boot 2024-05-27T07:30:03.730 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:30:03 smithi052 ceph-mon[31955]: fsmap cephfs:5 4 up:standby-replay 2 up:standby 5 up:active 2024-05-27T07:30:03.730 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:30:03 smithi052 ceph-mon[31955]: from='mgr.15333 172.21.15.52:0/416389082' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "l"}]: dispatch 2024-05-27T07:30:03.731 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:30:03 smithi052 ceph-mon[31955]: fsmap cephfs:5 5 up:standby-replay 1 up:standby 5 up:active 2024-05-27T07:30:03.760 INFO:teuthology.orchestra.run.smithi052.stderr:Inferring config /var/lib/ceph/73df2636-1bf8-11ef-bc9b-c7b262605968/mon.a/config 2024-05-27T07:30:04.480 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:30:04 smithi052 ceph-mon[31955]: pgmap v146: 129 pgs: 129 active+clean; 260 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.8 MiB/s rd, 2.3 MiB/s wr, 849 op/s 2024-05-27T07:30:04.602 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:30:04 smithi173 ceph-mon[37000]: pgmap v146: 129 pgs: 129 active+clean; 260 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.8 MiB/s rd, 2.3 MiB/s wr, 849 op/s 2024-05-27T07:30:04.690 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:30:04 smithi105 ceph-mon[35977]: pgmap v146: 129 pgs: 129 active+clean; 260 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.8 MiB/s rd, 2.3 MiB/s wr, 849 op/s 2024-05-27T07:30:04.985 INFO:journalctl@ceph.mds.a.smithi052.stdout:May 27 07:30:04 smithi052 systemd[1]: ceph-73df2636-1bf8-11ef-bc9b-c7b262605968@mds.a.service: Scheduled restart job, restart counter is at 3. 2024-05-27T07:30:05.037 INFO:journalctl@ceph.mds.a.smithi052.stdout:May 27 07:30:04 smithi052 systemd[1]: Stopped Ceph mds.a for 73df2636-1bf8-11ef-bc9b-c7b262605968. 2024-05-27T07:30:05.037 INFO:journalctl@ceph.mds.a.smithi052.stdout:May 27 07:30:04 smithi052 systemd[1]: ceph-73df2636-1bf8-11ef-bc9b-c7b262605968@mds.a.service: Consumed 1.283s CPU time. 2024-05-27T07:30:05.037 INFO:journalctl@ceph.mds.a.smithi052.stdout:May 27 07:30:04 smithi052 systemd[1]: Starting Ceph mds.a for 73df2636-1bf8-11ef-bc9b-c7b262605968... 2024-05-27T07:30:05.038 INFO:journalctl@ceph.mds.g.smithi052.stdout:May 27 07:30:04 smithi052 systemd[1]: ceph-73df2636-1bf8-11ef-bc9b-c7b262605968@mds.g.service: Scheduled restart job, restart counter is at 6. 2024-05-27T07:30:05.038 INFO:journalctl@ceph.mds.g.smithi052.stdout:May 27 07:30:04 smithi052 systemd[1]: Stopped Ceph mds.g for 73df2636-1bf8-11ef-bc9b-c7b262605968. 2024-05-27T07:30:05.038 INFO:journalctl@ceph.mds.g.smithi052.stdout:May 27 07:30:04 smithi052 systemd[1]: ceph-73df2636-1bf8-11ef-bc9b-c7b262605968@mds.g.service: Consumed 1.196s CPU time. 2024-05-27T07:30:05.038 INFO:journalctl@ceph.mds.g.smithi052.stdout:May 27 07:30:04 smithi052 systemd[1]: ceph-73df2636-1bf8-11ef-bc9b-c7b262605968@mds.g.service: Start request repeated too quickly. 2024-05-27T07:30:05.038 INFO:journalctl@ceph.mds.g.smithi052.stdout:May 27 07:30:04 smithi052 systemd[1]: ceph-73df2636-1bf8-11ef-bc9b-c7b262605968@mds.g.service: Failed with result 'exit-code'. 2024-05-27T07:30:05.039 INFO:journalctl@ceph.mds.g.smithi052.stdout:May 27 07:30:04 smithi052 systemd[1]: Failed to start Ceph mds.g for 73df2636-1bf8-11ef-bc9b-c7b262605968. 2024-05-27T07:30:05.208 INFO:tasks.workunit.client.0.smithi052.stdout: 77 896877 419 636015 178 542596 752 2024-05-27T07:30:05.565 INFO:journalctl@ceph.mds.d.smithi052.stdout:May 27 07:30:05 smithi052 systemd[1]: ceph-73df2636-1bf8-11ef-bc9b-c7b262605968@mds.d.service: Scheduled restart job, restart counter is at 5. 2024-05-27T07:30:05.566 INFO:journalctl@ceph.mds.d.smithi052.stdout:May 27 07:30:05 smithi052 systemd[1]: Stopped Ceph mds.d for 73df2636-1bf8-11ef-bc9b-c7b262605968. 2024-05-27T07:30:05.566 INFO:journalctl@ceph.mds.d.smithi052.stdout:May 27 07:30:05 smithi052 systemd[1]: Starting Ceph mds.d for 73df2636-1bf8-11ef-bc9b-c7b262605968... 2024-05-27T07:30:05.568 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:30:05 smithi052 ceph-mon[31955]: MDS daemon mds.a is removed because it is dead or otherwise unavailable. 2024-05-27T07:30:05.568 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:30:05 smithi052 ceph-mon[31955]: MDS daemon mds.d is removed because it is dead or otherwise unavailable. 2024-05-27T07:30:05.569 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:30:05 smithi052 ceph-mon[31955]: MDS daemon mds.g is removed because it is dead or otherwise unavailable. 2024-05-27T07:30:05.569 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:30:05 smithi052 ceph-mon[31955]: fsmap cephfs:5 3 up:standby-replay 5 up:active 2024-05-27T07:30:05.571 INFO:journalctl@ceph.mds.j.smithi052.stdout:May 27 07:30:05 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-j[106435]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: In function 'std::set SnapClient::filter(const std::set&) const' thread 7f49cffc8700 time 2024-05-27T07:30:05.482481+0000 2024-05-27T07:30:05.571 INFO:journalctl@ceph.mds.j.smithi052.stdout:May 27 07:30:05 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-j[106435]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: 197: FAILED ceph_assert(cached_version > 0) 2024-05-27T07:30:05.571 INFO:journalctl@ceph.mds.j.smithi052.stdout:May 27 07:30:05 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-j[106435]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:30:05.571 INFO:journalctl@ceph.mds.j.smithi052.stdout:May 27 07:30:05 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-j[106435]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x127) [0x7f49e280dba6] 2024-05-27T07:30:05.571 INFO:journalctl@ceph.mds.j.smithi052.stdout:May 27 07:30:05 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-j[106435]: 2: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7f49e280ddd1] 2024-05-27T07:30:05.571 INFO:journalctl@ceph.mds.j.smithi052.stdout:May 27 07:30:05 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-j[106435]: 3: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x56028365423d] 2024-05-27T07:30:05.571 INFO:journalctl@ceph.mds.j.smithi052.stdout:May 27 07:30:05 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-j[106435]: 4: (SnapRealm::build_snap_set() const+0x323) [0x56028363f971] 2024-05-27T07:30:05.571 INFO:journalctl@ceph.mds.j.smithi052.stdout:May 27 07:30:05 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-j[106435]: 5: (SnapRealm::check_cache() const+0x13e) [0x56028363dfa0] 2024-05-27T07:30:05.572 INFO:journalctl@ceph.mds.j.smithi052.stdout:May 27 07:30:05 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-j[106435]: 6: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x56028363e651] 2024-05-27T07:30:05.572 INFO:journalctl@ceph.mds.j.smithi052.stdout:May 27 07:30:05 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-j[106435]: 7: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x5602835d3347] 2024-05-27T07:30:05.572 INFO:journalctl@ceph.mds.j.smithi052.stdout:May 27 07:30:05 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-j[106435]: 8: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x5602832cf87e] 2024-05-27T07:30:05.572 INFO:journalctl@ceph.mds.j.smithi052.stdout:May 27 07:30:05 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-j[106435]: 9: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x2702) [0x5602832d4e94] 2024-05-27T07:30:05.572 INFO:journalctl@ceph.mds.j.smithi052.stdout:May 27 07:30:05 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-j[106435]: 10: (EUpdate::replay(MDSRank*)+0x63) [0x5602832df969] 2024-05-27T07:30:05.572 INFO:journalctl@ceph.mds.j.smithi052.stdout:May 27 07:30:05 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-j[106435]: 11: (MDLog::_replay_thread()+0x1f38) [0x56028367848c] 2024-05-27T07:30:05.572 INFO:journalctl@ceph.mds.j.smithi052.stdout:May 27 07:30:05 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-j[106435]: 12: (MDLog::ReplayThread::entry()+0x11) [0x5602836819cb] 2024-05-27T07:30:05.572 INFO:journalctl@ceph.mds.j.smithi052.stdout:May 27 07:30:05 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-j[106435]: 13: (Thread::entry_wrapper()+0x43) [0x7f49e27dc72d] 2024-05-27T07:30:05.572 INFO:journalctl@ceph.mds.j.smithi052.stdout:May 27 07:30:05 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-j[106435]: 14: (Thread::_entry_func(void*)+0xd) [0x7f49e27dc749] 2024-05-27T07:30:05.572 INFO:journalctl@ceph.mds.j.smithi052.stdout:May 27 07:30:05 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-j[106435]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7f49e13801ca] 2024-05-27T07:30:05.572 INFO:journalctl@ceph.mds.j.smithi052.stdout:May 27 07:30:05 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-j[106435]: 16: clone() 2024-05-27T07:30:05.572 INFO:journalctl@ceph.mds.j.smithi052.stdout:May 27 07:30:05 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-j[106435]: 2024-05-27T07:30:05.492+0000 7f49cffc8700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: In function 'std::set SnapClient::filter(const std::set&) const' thread 7f49cffc8700 time 2024-05-27T07:30:05.482481+0000 2024-05-27T07:30:05.572 INFO:journalctl@ceph.mds.j.smithi052.stdout:May 27 07:30:05 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-j[106435]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: 197: FAILED ceph_assert(cached_version > 0) 2024-05-27T07:30:05.573 INFO:journalctl@ceph.mds.j.smithi052.stdout:May 27 07:30:05 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-j[106435]: 2024-05-27T07:30:05.573 INFO:journalctl@ceph.mds.j.smithi052.stdout:May 27 07:30:05 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-j[106435]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:30:05.573 INFO:journalctl@ceph.mds.j.smithi052.stdout:May 27 07:30:05 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-j[106435]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x127) [0x7f49e280dba6] 2024-05-27T07:30:05.573 INFO:journalctl@ceph.mds.j.smithi052.stdout:May 27 07:30:05 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-j[106435]: 2: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7f49e280ddd1] 2024-05-27T07:30:05.573 INFO:journalctl@ceph.mds.j.smithi052.stdout:May 27 07:30:05 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-j[106435]: 3: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x56028365423d] 2024-05-27T07:30:05.573 INFO:journalctl@ceph.mds.j.smithi052.stdout:May 27 07:30:05 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-j[106435]: 4: (SnapRealm::build_snap_set() const+0x323) [0x56028363f971] 2024-05-27T07:30:05.573 INFO:journalctl@ceph.mds.j.smithi052.stdout:May 27 07:30:05 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-j[106435]: 5: (SnapRealm::check_cache() const+0x13e) [0x56028363dfa0] 2024-05-27T07:30:05.573 INFO:journalctl@ceph.mds.j.smithi052.stdout:May 27 07:30:05 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-j[106435]: 6: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x56028363e651] 2024-05-27T07:30:05.573 INFO:journalctl@ceph.mds.j.smithi052.stdout:May 27 07:30:05 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-j[106435]: 7: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x5602835d3347] 2024-05-27T07:30:05.573 INFO:journalctl@ceph.mds.j.smithi052.stdout:May 27 07:30:05 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-j[106435]: 8: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x5602832cf87e] 2024-05-27T07:30:05.573 INFO:journalctl@ceph.mds.j.smithi052.stdout:May 27 07:30:05 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-j[106435]: 9: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x2702) [0x5602832d4e94] 2024-05-27T07:30:05.573 INFO:journalctl@ceph.mds.j.smithi052.stdout:May 27 07:30:05 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-j[106435]: 10: (EUpdate::replay(MDSRank*)+0x63) [0x5602832df969] 2024-05-27T07:30:05.573 INFO:journalctl@ceph.mds.j.smithi052.stdout:May 27 07:30:05 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-j[106435]: 11: (MDLog::_replay_thread()+0x1f38) [0x56028367848c] 2024-05-27T07:30:05.574 INFO:journalctl@ceph.mds.j.smithi052.stdout:May 27 07:30:05 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-j[106435]: 12: (MDLog::ReplayThread::entry()+0x11) [0x5602836819cb] 2024-05-27T07:30:05.574 INFO:journalctl@ceph.mds.j.smithi052.stdout:May 27 07:30:05 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-j[106435]: 13: (Thread::entry_wrapper()+0x43) [0x7f49e27dc72d] 2024-05-27T07:30:05.574 INFO:journalctl@ceph.mds.j.smithi052.stdout:May 27 07:30:05 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-j[106435]: 14: (Thread::_entry_func(void*)+0xd) [0x7f49e27dc749] 2024-05-27T07:30:05.574 INFO:journalctl@ceph.mds.j.smithi052.stdout:May 27 07:30:05 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-j[106435]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7f49e13801ca] 2024-05-27T07:30:05.574 INFO:journalctl@ceph.mds.j.smithi052.stdout:May 27 07:30:05 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-j[106435]: 16: clone() 2024-05-27T07:30:05.574 INFO:journalctl@ceph.mds.j.smithi052.stdout:May 27 07:30:05 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-j[106435]: 2024-05-27T07:30:05.574 INFO:journalctl@ceph.mds.j.smithi052.stdout:May 27 07:30:05 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-j[106435]: *** Caught signal (Aborted) ** 2024-05-27T07:30:05.574 INFO:journalctl@ceph.mds.j.smithi052.stdout:May 27 07:30:05 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-j[106435]: in thread 7f49cffc8700 thread_name:md_log_replay 2024-05-27T07:30:05.574 INFO:journalctl@ceph.mds.j.smithi052.stdout:May 27 07:30:05 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-j[106435]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:30:05.574 INFO:journalctl@ceph.mds.j.smithi052.stdout:May 27 07:30:05 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-j[106435]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f49e138ad20] 2024-05-27T07:30:05.574 INFO:journalctl@ceph.mds.j.smithi052.stdout:May 27 07:30:05 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-j[106435]: 2: gsignal() 2024-05-27T07:30:05.574 INFO:journalctl@ceph.mds.j.smithi052.stdout:May 27 07:30:05 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-j[106435]: 3: abort() 2024-05-27T07:30:05.574 INFO:journalctl@ceph.mds.j.smithi052.stdout:May 27 07:30:05 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-j[106435]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x26e) [0x7f49e280dced] 2024-05-27T07:30:05.575 INFO:journalctl@ceph.mds.j.smithi052.stdout:May 27 07:30:05 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-j[106435]: 5: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7f49e280ddd1] 2024-05-27T07:30:05.575 INFO:journalctl@ceph.mds.j.smithi052.stdout:May 27 07:30:05 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-j[106435]: 6: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x56028365423d] 2024-05-27T07:30:05.575 INFO:journalctl@ceph.mds.j.smithi052.stdout:May 27 07:30:05 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-j[106435]: 7: (SnapRealm::build_snap_set() const+0x323) [0x56028363f971] 2024-05-27T07:30:05.575 INFO:journalctl@ceph.mds.j.smithi052.stdout:May 27 07:30:05 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-j[106435]: 8: (SnapRealm::check_cache() const+0x13e) [0x56028363dfa0] 2024-05-27T07:30:05.575 INFO:journalctl@ceph.mds.j.smithi052.stdout:May 27 07:30:05 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-j[106435]: 9: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x56028363e651] 2024-05-27T07:30:05.575 INFO:journalctl@ceph.mds.j.smithi052.stdout:May 27 07:30:05 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-j[106435]: 10: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x5602835d3347] 2024-05-27T07:30:05.575 INFO:journalctl@ceph.mds.j.smithi052.stdout:May 27 07:30:05 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-j[106435]: 11: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x5602832cf87e] 2024-05-27T07:30:05.575 INFO:journalctl@ceph.mds.j.smithi052.stdout:May 27 07:30:05 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-j[106435]: 12: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x2702) [0x5602832d4e94] 2024-05-27T07:30:05.575 INFO:journalctl@ceph.mds.j.smithi052.stdout:May 27 07:30:05 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-j[106435]: 13: (EUpdate::replay(MDSRank*)+0x63) [0x5602832df969] 2024-05-27T07:30:05.575 INFO:journalctl@ceph.mds.j.smithi052.stdout:May 27 07:30:05 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-j[106435]: 14: (MDLog::_replay_thread()+0x1f38) [0x56028367848c] 2024-05-27T07:30:05.575 INFO:journalctl@ceph.mds.j.smithi052.stdout:May 27 07:30:05 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-j[106435]: 15: (MDLog::ReplayThread::entry()+0x11) [0x5602836819cb] 2024-05-27T07:30:05.575 INFO:journalctl@ceph.mds.j.smithi052.stdout:May 27 07:30:05 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-j[106435]: 16: (Thread::entry_wrapper()+0x43) [0x7f49e27dc72d] 2024-05-27T07:30:05.576 INFO:journalctl@ceph.mds.j.smithi052.stdout:May 27 07:30:05 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-j[106435]: 17: (Thread::_entry_func(void*)+0xd) [0x7f49e27dc749] 2024-05-27T07:30:05.576 INFO:journalctl@ceph.mds.j.smithi052.stdout:May 27 07:30:05 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-j[106435]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7f49e13801ca] 2024-05-27T07:30:05.576 INFO:journalctl@ceph.mds.j.smithi052.stdout:May 27 07:30:05 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-j[106435]: 19: clone() 2024-05-27T07:30:05.576 INFO:journalctl@ceph.mds.j.smithi052.stdout:May 27 07:30:05 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-j[106435]: 2024-05-27T07:30:05.496+0000 7f49cffc8700 -1 *** Caught signal (Aborted) ** 2024-05-27T07:30:05.576 INFO:journalctl@ceph.mds.j.smithi052.stdout:May 27 07:30:05 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-j[106435]: in thread 7f49cffc8700 thread_name:md_log_replay 2024-05-27T07:30:05.576 INFO:journalctl@ceph.mds.j.smithi052.stdout:May 27 07:30:05 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-j[106435]: 2024-05-27T07:30:05.576 INFO:journalctl@ceph.mds.j.smithi052.stdout:May 27 07:30:05 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-j[106435]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:30:05.576 INFO:journalctl@ceph.mds.j.smithi052.stdout:May 27 07:30:05 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-j[106435]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f49e138ad20] 2024-05-27T07:30:05.576 INFO:journalctl@ceph.mds.j.smithi052.stdout:May 27 07:30:05 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-j[106435]: 2: gsignal() 2024-05-27T07:30:05.576 INFO:journalctl@ceph.mds.j.smithi052.stdout:May 27 07:30:05 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-j[106435]: 3: abort() 2024-05-27T07:30:05.576 INFO:journalctl@ceph.mds.j.smithi052.stdout:May 27 07:30:05 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-j[106435]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x26e) [0x7f49e280dced] 2024-05-27T07:30:05.576 INFO:journalctl@ceph.mds.j.smithi052.stdout:May 27 07:30:05 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-j[106435]: 5: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7f49e280ddd1] 2024-05-27T07:30:05.576 INFO:journalctl@ceph.mds.j.smithi052.stdout:May 27 07:30:05 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-j[106435]: 6: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x56028365423d] 2024-05-27T07:30:05.576 INFO:journalctl@ceph.mds.j.smithi052.stdout:May 27 07:30:05 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-j[106435]: 7: (SnapRealm::build_snap_set() const+0x323) [0x56028363f971] 2024-05-27T07:30:05.577 INFO:journalctl@ceph.mds.j.smithi052.stdout:May 27 07:30:05 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-j[106435]: 8: (SnapRealm::check_cache() const+0x13e) [0x56028363dfa0] 2024-05-27T07:30:05.577 INFO:journalctl@ceph.mds.j.smithi052.stdout:May 27 07:30:05 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-j[106435]: 9: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x56028363e651] 2024-05-27T07:30:05.577 INFO:journalctl@ceph.mds.j.smithi052.stdout:May 27 07:30:05 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-j[106435]: 10: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x5602835d3347] 2024-05-27T07:30:05.577 INFO:journalctl@ceph.mds.j.smithi052.stdout:May 27 07:30:05 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-j[106435]: 11: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x5602832cf87e] 2024-05-27T07:30:05.577 INFO:journalctl@ceph.mds.j.smithi052.stdout:May 27 07:30:05 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-j[106435]: 12: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x2702) [0x5602832d4e94] 2024-05-27T07:30:05.577 INFO:journalctl@ceph.mds.j.smithi052.stdout:May 27 07:30:05 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-j[106435]: 13: (EUpdate::replay(MDSRank*)+0x63) [0x5602832df969] 2024-05-27T07:30:05.577 INFO:journalctl@ceph.mds.j.smithi052.stdout:May 27 07:30:05 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-j[106435]: 14: (MDLog::_replay_thread()+0x1f38) [0x56028367848c] 2024-05-27T07:30:05.577 INFO:journalctl@ceph.mds.j.smithi052.stdout:May 27 07:30:05 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-j[106435]: 15: (MDLog::ReplayThread::entry()+0x11) [0x5602836819cb] 2024-05-27T07:30:05.577 INFO:journalctl@ceph.mds.j.smithi052.stdout:May 27 07:30:05 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-j[106435]: 16: (Thread::entry_wrapper()+0x43) [0x7f49e27dc72d] 2024-05-27T07:30:05.577 INFO:journalctl@ceph.mds.j.smithi052.stdout:May 27 07:30:05 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-j[106435]: 17: (Thread::_entry_func(void*)+0xd) [0x7f49e27dc749] 2024-05-27T07:30:05.577 INFO:journalctl@ceph.mds.j.smithi052.stdout:May 27 07:30:05 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-j[106435]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7f49e13801ca] 2024-05-27T07:30:05.577 INFO:journalctl@ceph.mds.j.smithi052.stdout:May 27 07:30:05 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-j[106435]: 19: clone() 2024-05-27T07:30:05.577 INFO:journalctl@ceph.mds.j.smithi052.stdout:May 27 07:30:05 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-j[106435]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-05-27T07:30:05.578 INFO:journalctl@ceph.mds.j.smithi052.stdout:May 27 07:30:05 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-j[106435]: 2024-05-27T07:30:05.578 INFO:journalctl@ceph.mds.j.smithi052.stdout:May 27 07:30:05 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-j[106435]: -1> 2024-05-27T07:30:05.492+0000 7f49cffc8700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: In function 'std::set SnapClient::filter(const std::set&) const' thread 7f49cffc8700 time 2024-05-27T07:30:05.482481+0000 2024-05-27T07:30:05.578 INFO:journalctl@ceph.mds.j.smithi052.stdout:May 27 07:30:05 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-j[106435]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: 197: FAILED ceph_assert(cached_version > 0) 2024-05-27T07:30:05.578 INFO:journalctl@ceph.mds.j.smithi052.stdout:May 27 07:30:05 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-j[106435]: 2024-05-27T07:30:05.578 INFO:journalctl@ceph.mds.j.smithi052.stdout:May 27 07:30:05 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-j[106435]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:30:05.578 INFO:journalctl@ceph.mds.j.smithi052.stdout:May 27 07:30:05 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-j[106435]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x127) [0x7f49e280dba6] 2024-05-27T07:30:05.578 INFO:journalctl@ceph.mds.j.smithi052.stdout:May 27 07:30:05 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-j[106435]: 2: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7f49e280ddd1] 2024-05-27T07:30:05.578 INFO:journalctl@ceph.mds.j.smithi052.stdout:May 27 07:30:05 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-j[106435]: 3: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x56028365423d] 2024-05-27T07:30:05.578 INFO:journalctl@ceph.mds.j.smithi052.stdout:May 27 07:30:05 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-j[106435]: 4: (SnapRealm::build_snap_set() const+0x323) [0x56028363f971] 2024-05-27T07:30:05.578 INFO:journalctl@ceph.mds.j.smithi052.stdout:May 27 07:30:05 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-j[106435]: 5: (SnapRealm::check_cache() const+0x13e) [0x56028363dfa0] 2024-05-27T07:30:05.578 INFO:journalctl@ceph.mds.j.smithi052.stdout:May 27 07:30:05 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-j[106435]: 6: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x56028363e651] 2024-05-27T07:30:05.578 INFO:journalctl@ceph.mds.j.smithi052.stdout:May 27 07:30:05 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-j[106435]: 7: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x5602835d3347] 2024-05-27T07:30:05.579 INFO:journalctl@ceph.mds.j.smithi052.stdout:May 27 07:30:05 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-j[106435]: 8: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x5602832cf87e] 2024-05-27T07:30:05.579 INFO:journalctl@ceph.mds.j.smithi052.stdout:May 27 07:30:05 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-j[106435]: 9: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x2702) [0x5602832d4e94] 2024-05-27T07:30:05.579 INFO:journalctl@ceph.mds.j.smithi052.stdout:May 27 07:30:05 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-j[106435]: 10: (EUpdate::replay(MDSRank*)+0x63) [0x5602832df969] 2024-05-27T07:30:05.579 INFO:journalctl@ceph.mds.j.smithi052.stdout:May 27 07:30:05 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-j[106435]: 11: (MDLog::_replay_thread()+0x1f38) [0x56028367848c] 2024-05-27T07:30:05.579 INFO:journalctl@ceph.mds.j.smithi052.stdout:May 27 07:30:05 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-j[106435]: 12: (MDLog::ReplayThread::entry()+0x11) [0x5602836819cb] 2024-05-27T07:30:05.579 INFO:journalctl@ceph.mds.j.smithi052.stdout:May 27 07:30:05 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-j[106435]: 13: (Thread::entry_wrapper()+0x43) [0x7f49e27dc72d] 2024-05-27T07:30:05.579 INFO:journalctl@ceph.mds.j.smithi052.stdout:May 27 07:30:05 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-j[106435]: 14: (Thread::_entry_func(void*)+0xd) [0x7f49e27dc749] 2024-05-27T07:30:05.579 INFO:journalctl@ceph.mds.j.smithi052.stdout:May 27 07:30:05 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-j[106435]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7f49e13801ca] 2024-05-27T07:30:05.579 INFO:journalctl@ceph.mds.j.smithi052.stdout:May 27 07:30:05 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-j[106435]: 16: clone() 2024-05-27T07:30:05.579 INFO:journalctl@ceph.mds.j.smithi052.stdout:May 27 07:30:05 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-j[106435]: 2024-05-27T07:30:05.579 INFO:journalctl@ceph.mds.j.smithi052.stdout:May 27 07:30:05 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-j[106435]: 0> 2024-05-27T07:30:05.496+0000 7f49cffc8700 -1 *** Caught signal (Aborted) ** 2024-05-27T07:30:05.579 INFO:journalctl@ceph.mds.j.smithi052.stdout:May 27 07:30:05 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-j[106435]: in thread 7f49cffc8700 thread_name:md_log_replay 2024-05-27T07:30:05.579 INFO:journalctl@ceph.mds.j.smithi052.stdout:May 27 07:30:05 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-j[106435]: 2024-05-27T07:30:05.580 INFO:journalctl@ceph.mds.j.smithi052.stdout:May 27 07:30:05 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-j[106435]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:30:05.580 INFO:journalctl@ceph.mds.j.smithi052.stdout:May 27 07:30:05 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-j[106435]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f49e138ad20] 2024-05-27T07:30:05.580 INFO:journalctl@ceph.mds.j.smithi052.stdout:May 27 07:30:05 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-j[106435]: 2: gsignal() 2024-05-27T07:30:05.580 INFO:journalctl@ceph.mds.j.smithi052.stdout:May 27 07:30:05 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-j[106435]: 3: abort() 2024-05-27T07:30:05.580 INFO:journalctl@ceph.mds.j.smithi052.stdout:May 27 07:30:05 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-j[106435]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x26e) [0x7f49e280dced] 2024-05-27T07:30:05.580 INFO:journalctl@ceph.mds.j.smithi052.stdout:May 27 07:30:05 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-j[106435]: 5: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7f49e280ddd1] 2024-05-27T07:30:05.580 INFO:journalctl@ceph.mds.j.smithi052.stdout:May 27 07:30:05 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-j[106435]: 6: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x56028365423d] 2024-05-27T07:30:05.580 INFO:journalctl@ceph.mds.j.smithi052.stdout:May 27 07:30:05 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-j[106435]: 7: (SnapRealm::build_snap_set() const+0x323) [0x56028363f971] 2024-05-27T07:30:05.580 INFO:journalctl@ceph.mds.j.smithi052.stdout:May 27 07:30:05 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-j[106435]: 8: (SnapRealm::check_cache() const+0x13e) [0x56028363dfa0] 2024-05-27T07:30:05.580 INFO:journalctl@ceph.mds.j.smithi052.stdout:May 27 07:30:05 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-j[106435]: 9: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x56028363e651] 2024-05-27T07:30:05.580 INFO:journalctl@ceph.mds.j.smithi052.stdout:May 27 07:30:05 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-j[106435]: 10: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x5602835d3347] 2024-05-27T07:30:05.580 INFO:journalctl@ceph.mds.j.smithi052.stdout:May 27 07:30:05 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-j[106435]: 11: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x5602832cf87e] 2024-05-27T07:30:05.580 INFO:journalctl@ceph.mds.j.smithi052.stdout:May 27 07:30:05 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-j[106435]: 12: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x2702) [0x5602832d4e94] 2024-05-27T07:30:05.581 INFO:journalctl@ceph.mds.j.smithi052.stdout:May 27 07:30:05 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-j[106435]: 13: (EUpdate::replay(MDSRank*)+0x63) [0x5602832df969] 2024-05-27T07:30:05.581 INFO:journalctl@ceph.mds.j.smithi052.stdout:May 27 07:30:05 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-j[106435]: 14: (MDLog::_replay_thread()+0x1f38) [0x56028367848c] 2024-05-27T07:30:05.581 INFO:journalctl@ceph.mds.j.smithi052.stdout:May 27 07:30:05 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-j[106435]: 15: (MDLog::ReplayThread::entry()+0x11) [0x5602836819cb] 2024-05-27T07:30:05.581 INFO:journalctl@ceph.mds.j.smithi052.stdout:May 27 07:30:05 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-j[106435]: 16: (Thread::entry_wrapper()+0x43) [0x7f49e27dc72d] 2024-05-27T07:30:05.581 INFO:journalctl@ceph.mds.j.smithi052.stdout:May 27 07:30:05 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-j[106435]: 17: (Thread::_entry_func(void*)+0xd) [0x7f49e27dc749] 2024-05-27T07:30:05.581 INFO:journalctl@ceph.mds.j.smithi052.stdout:May 27 07:30:05 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-j[106435]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7f49e13801ca] 2024-05-27T07:30:05.581 INFO:journalctl@ceph.mds.j.smithi052.stdout:May 27 07:30:05 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-j[106435]: 19: clone() 2024-05-27T07:30:05.581 INFO:journalctl@ceph.mds.j.smithi052.stdout:May 27 07:30:05 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-j[106435]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-05-27T07:30:05.581 INFO:journalctl@ceph.mds.j.smithi052.stdout:May 27 07:30:05 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-j[106435]: 2024-05-27T07:30:05.612 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:30:05 smithi173 ceph-mon[37000]: MDS daemon mds.a is removed because it is dead or otherwise unavailable. 2024-05-27T07:30:05.612 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:30:05 smithi173 ceph-mon[37000]: MDS daemon mds.d is removed because it is dead or otherwise unavailable. 2024-05-27T07:30:05.612 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:30:05 smithi173 ceph-mon[37000]: MDS daemon mds.g is removed because it is dead or otherwise unavailable. 2024-05-27T07:30:05.612 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:30:05 smithi173 ceph-mon[37000]: fsmap cephfs:5 3 up:standby-replay 5 up:active 2024-05-27T07:30:05.698 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:30:05 smithi105 ceph-mon[35977]: MDS daemon mds.a is removed because it is dead or otherwise unavailable. 2024-05-27T07:30:05.698 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:30:05 smithi105 ceph-mon[35977]: MDS daemon mds.d is removed because it is dead or otherwise unavailable. 2024-05-27T07:30:05.698 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:30:05 smithi105 ceph-mon[35977]: MDS daemon mds.g is removed because it is dead or otherwise unavailable. 2024-05-27T07:30:05.698 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:30:05 smithi105 ceph-mon[35977]: fsmap cephfs:5 3 up:standby-replay 5 up:active 2024-05-27T07:30:05.981 INFO:journalctl@ceph.mds.j.smithi052.stdout:May 27 07:30:05 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-j[106435]: -1> 2024-05-27T07:30:05.492+0000 7f49cffc8700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: In function 'std::set SnapClient::filter(const std::set&) const' thread 7f49cffc8700 time 2024-05-27T07:30:05.482481+0000 2024-05-27T07:30:05.981 INFO:journalctl@ceph.mds.j.smithi052.stdout:May 27 07:30:05 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-j[106435]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: 197: FAILED ceph_assert(cached_version > 0) 2024-05-27T07:30:05.981 INFO:journalctl@ceph.mds.j.smithi052.stdout:May 27 07:30:05 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-j[106435]: 2024-05-27T07:30:05.982 INFO:journalctl@ceph.mds.j.smithi052.stdout:May 27 07:30:05 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-j[106435]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:30:05.982 INFO:journalctl@ceph.mds.j.smithi052.stdout:May 27 07:30:05 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-j[106435]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x127) [0x7f49e280dba6] 2024-05-27T07:30:05.982 INFO:journalctl@ceph.mds.j.smithi052.stdout:May 27 07:30:05 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-j[106435]: 2: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7f49e280ddd1] 2024-05-27T07:30:05.982 INFO:journalctl@ceph.mds.j.smithi052.stdout:May 27 07:30:05 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-j[106435]: 3: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x56028365423d] 2024-05-27T07:30:05.982 INFO:journalctl@ceph.mds.j.smithi052.stdout:May 27 07:30:05 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-j[106435]: 4: (SnapRealm::build_snap_set() const+0x323) [0x56028363f971] 2024-05-27T07:30:05.982 INFO:journalctl@ceph.mds.j.smithi052.stdout:May 27 07:30:05 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-j[106435]: 5: (SnapRealm::check_cache() const+0x13e) [0x56028363dfa0] 2024-05-27T07:30:05.982 INFO:journalctl@ceph.mds.j.smithi052.stdout:May 27 07:30:05 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-j[106435]: 6: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x56028363e651] 2024-05-27T07:30:05.982 INFO:journalctl@ceph.mds.j.smithi052.stdout:May 27 07:30:05 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-j[106435]: 7: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x5602835d3347] 2024-05-27T07:30:05.982 INFO:journalctl@ceph.mds.j.smithi052.stdout:May 27 07:30:05 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-j[106435]: 8: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x5602832cf87e] 2024-05-27T07:30:05.982 INFO:journalctl@ceph.mds.j.smithi052.stdout:May 27 07:30:05 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-j[106435]: 9: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x2702) [0x5602832d4e94] 2024-05-27T07:30:05.982 INFO:journalctl@ceph.mds.j.smithi052.stdout:May 27 07:30:05 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-j[106435]: 10: (EUpdate::replay(MDSRank*)+0x63) [0x5602832df969] 2024-05-27T07:30:05.982 INFO:journalctl@ceph.mds.j.smithi052.stdout:May 27 07:30:05 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-j[106435]: 11: (MDLog::_replay_thread()+0x1f38) [0x56028367848c] 2024-05-27T07:30:05.983 INFO:journalctl@ceph.mds.j.smithi052.stdout:May 27 07:30:05 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-j[106435]: 12: (MDLog::ReplayThread::entry()+0x11) [0x5602836819cb] 2024-05-27T07:30:05.983 INFO:journalctl@ceph.mds.j.smithi052.stdout:May 27 07:30:05 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-j[106435]: 13: (Thread::entry_wrapper()+0x43) [0x7f49e27dc72d] 2024-05-27T07:30:05.983 INFO:journalctl@ceph.mds.j.smithi052.stdout:May 27 07:30:05 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-j[106435]: 14: (Thread::_entry_func(void*)+0xd) [0x7f49e27dc749] 2024-05-27T07:30:05.983 INFO:journalctl@ceph.mds.j.smithi052.stdout:May 27 07:30:05 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-j[106435]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7f49e13801ca] 2024-05-27T07:30:05.983 INFO:journalctl@ceph.mds.j.smithi052.stdout:May 27 07:30:05 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-j[106435]: 16: clone() 2024-05-27T07:30:05.983 INFO:journalctl@ceph.mds.j.smithi052.stdout:May 27 07:30:05 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-j[106435]: 2024-05-27T07:30:05.983 INFO:journalctl@ceph.mds.j.smithi052.stdout:May 27 07:30:05 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-j[106435]: 0> 2024-05-27T07:30:05.496+0000 7f49cffc8700 -1 *** Caught signal (Aborted) ** 2024-05-27T07:30:05.983 INFO:journalctl@ceph.mds.j.smithi052.stdout:May 27 07:30:05 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-j[106435]: in thread 7f49cffc8700 thread_name:md_log_replay 2024-05-27T07:30:05.983 INFO:journalctl@ceph.mds.j.smithi052.stdout:May 27 07:30:05 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-j[106435]: 2024-05-27T07:30:05.983 INFO:journalctl@ceph.mds.j.smithi052.stdout:May 27 07:30:05 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-j[106435]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:30:05.984 INFO:journalctl@ceph.mds.j.smithi052.stdout:May 27 07:30:05 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-j[106435]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f49e138ad20] 2024-05-27T07:30:05.984 INFO:journalctl@ceph.mds.j.smithi052.stdout:May 27 07:30:05 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-j[106435]: 2: gsignal() 2024-05-27T07:30:05.984 INFO:journalctl@ceph.mds.j.smithi052.stdout:May 27 07:30:05 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-j[106435]: 3: abort() 2024-05-27T07:30:05.984 INFO:journalctl@ceph.mds.j.smithi052.stdout:May 27 07:30:05 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-j[106435]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x26e) [0x7f49e280dced] 2024-05-27T07:30:05.984 INFO:journalctl@ceph.mds.j.smithi052.stdout:May 27 07:30:05 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-j[106435]: 5: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7f49e280ddd1] 2024-05-27T07:30:05.984 INFO:journalctl@ceph.mds.j.smithi052.stdout:May 27 07:30:05 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-j[106435]: 6: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x56028365423d] 2024-05-27T07:30:05.984 INFO:journalctl@ceph.mds.j.smithi052.stdout:May 27 07:30:05 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-j[106435]: 7: (SnapRealm::build_snap_set() const+0x323) [0x56028363f971] 2024-05-27T07:30:05.984 INFO:journalctl@ceph.mds.j.smithi052.stdout:May 27 07:30:05 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-j[106435]: 8: (SnapRealm::check_cache() const+0x13e) [0x56028363dfa0] 2024-05-27T07:30:05.984 INFO:journalctl@ceph.mds.j.smithi052.stdout:May 27 07:30:05 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-j[106435]: 9: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x56028363e651] 2024-05-27T07:30:05.984 INFO:journalctl@ceph.mds.j.smithi052.stdout:May 27 07:30:05 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-j[106435]: 10: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x5602835d3347] 2024-05-27T07:30:05.984 INFO:journalctl@ceph.mds.j.smithi052.stdout:May 27 07:30:05 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-j[106435]: 11: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x5602832cf87e] 2024-05-27T07:30:05.984 INFO:journalctl@ceph.mds.j.smithi052.stdout:May 27 07:30:05 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-j[106435]: 12: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x2702) [0x5602832d4e94] 2024-05-27T07:30:05.985 INFO:journalctl@ceph.mds.j.smithi052.stdout:May 27 07:30:05 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-j[106435]: 13: (EUpdate::replay(MDSRank*)+0x63) [0x5602832df969] 2024-05-27T07:30:05.985 INFO:journalctl@ceph.mds.j.smithi052.stdout:May 27 07:30:05 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-j[106435]: 14: (MDLog::_replay_thread()+0x1f38) [0x56028367848c] 2024-05-27T07:30:05.985 INFO:journalctl@ceph.mds.j.smithi052.stdout:May 27 07:30:05 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-j[106435]: 15: (MDLog::ReplayThread::entry()+0x11) [0x5602836819cb] 2024-05-27T07:30:05.985 INFO:journalctl@ceph.mds.j.smithi052.stdout:May 27 07:30:05 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-j[106435]: 16: (Thread::entry_wrapper()+0x43) [0x7f49e27dc72d] 2024-05-27T07:30:05.985 INFO:journalctl@ceph.mds.j.smithi052.stdout:May 27 07:30:05 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-j[106435]: 17: (Thread::_entry_func(void*)+0xd) [0x7f49e27dc749] 2024-05-27T07:30:05.985 INFO:journalctl@ceph.mds.j.smithi052.stdout:May 27 07:30:05 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-j[106435]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7f49e13801ca] 2024-05-27T07:30:05.985 INFO:journalctl@ceph.mds.j.smithi052.stdout:May 27 07:30:05 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-j[106435]: 19: clone() 2024-05-27T07:30:05.985 INFO:journalctl@ceph.mds.j.smithi052.stdout:May 27 07:30:05 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-j[106435]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-05-27T07:30:05.985 INFO:journalctl@ceph.mds.j.smithi052.stdout:May 27 07:30:05 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-j[106435]: 2024-05-27T07:30:06.481 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:30:06 smithi052 ceph-mon[31955]: pgmap v147: 129 pgs: 129 active+clean; 301 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1023 KiB/s rd, 6.2 MiB/s wr, 533 op/s 2024-05-27T07:30:06.482 INFO:journalctl@ceph.mds.j.smithi052.stdout:May 27 07:30:06 smithi052 podman[106985]: 2024-05-27 07:30:06.079433208 +0000 UTC m=+0.458127843 container died d03f6d3925720a9a1b1ac9712defc07799179d80b7913701e4ef645e7a0b5cd1 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-j, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, GIT_CLEAN=True, ceph=True, org.label-schema.build-date=20240521, org.label-schema.vendor=CentOS, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-05-27T07:30:06.603 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:30:06 smithi173 ceph-mon[37000]: pgmap v147: 129 pgs: 129 active+clean; 301 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1023 KiB/s rd, 6.2 MiB/s wr, 533 op/s 2024-05-27T07:30:06.690 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:30:06 smithi105 ceph-mon[35977]: pgmap v147: 129 pgs: 129 active+clean; 301 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1023 KiB/s rd, 6.2 MiB/s wr, 533 op/s 2024-05-27T07:30:06.916 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:30:06.925+0000 7ff789b1f640 1 -- 172.21.15.52:0/4134424941 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff784103300 msgr2=0x7ff78410f6d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:30:06.916 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:30:06.925+0000 7ff789b1f640 1 --2- 172.21.15.52:0/4134424941 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff784103300 0x7ff78410f6d0 secure :-1 s=READY pgs=634 cs=0 l=1 rev1=1 crypto rx=0x7ff77c009ca0 tx=0x7ff77c02a8c0 comp rx=0 tx=0).stop 2024-05-27T07:30:06.916 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:30:06.925+0000 7ff789b1f640 1 -- 172.21.15.52:0/4134424941 shutdown_connections 2024-05-27T07:30:06.916 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:30:06.925+0000 7ff789b1f640 1 --2- 172.21.15.52:0/4134424941 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff784103300 0x7ff78410f6d0 unknown :-1 s=CLOSED pgs=634 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:30:06.916 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:30:06.925+0000 7ff789b1f640 1 --2- 172.21.15.52:0/4134424941 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7ff784109050 0x7ff784102c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:30:06.916 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:30:06.925+0000 7ff789b1f640 1 --2- 172.21.15.52:0/4134424941 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7ff784108bb0 0x7ff784102750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:30:06.916 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:30:06.925+0000 7ff789b1f640 1 -- 172.21.15.52:0/4134424941 >> 172.21.15.52:0/4134424941 conn(0x7ff7840781a0 msgr2=0x7ff784078600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:30:06.917 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:30:06.925+0000 7ff789b1f640 1 -- 172.21.15.52:0/4134424941 shutdown_connections 2024-05-27T07:30:06.917 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:30:06.925+0000 7ff789b1f640 1 -- 172.21.15.52:0/4134424941 wait complete. 2024-05-27T07:30:06.919 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:30:06.925+0000 7ff789b1f640 1 Processor -- start 2024-05-27T07:30:06.919 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:30:06.926+0000 7ff789b1f640 1 -- start start 2024-05-27T07:30:06.919 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:30:06.926+0000 7ff789b1f640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7ff784103300 0x7ff78419b640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:30:06.919 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:30:06.926+0000 7ff789b1f640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7ff784108bb0 0x7ff78419bb80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:30:06.919 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:30:06.926+0000 7ff789b1f640 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff784109050 0x7ff7841a66c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:30:06.919 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:30:06.926+0000 7ff789b1f640 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x7ff784112dc0 con 0x7ff784109050 2024-05-27T07:30:06.919 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:30:06.926+0000 7ff789b1f640 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7ff784112c40 con 0x7ff784108bb0 2024-05-27T07:30:06.920 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:30:06.926+0000 7ff789b1f640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7ff784112f40 con 0x7ff784103300 2024-05-27T07:30:06.920 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:30:06.926+0000 7ff782ffd640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7ff784108bb0 0x7ff78419bb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:30:06.920 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:30:06.926+0000 7ff783fff640 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff784109050 0x7ff7841a66c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:30:06.920 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:30:06.926+0000 7ff782ffd640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7ff784108bb0 0x7ff78419bb80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.52:54132/0 (socket says 172.21.15.52:54132) 2024-05-27T07:30:06.920 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:30:06.926+0000 7ff782ffd640 1 -- 172.21.15.52:0/1220248260 learned_addr learned my addr 172.21.15.52:0/1220248260 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-05-27T07:30:06.920 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:30:06.926+0000 7ff7837fe640 1 --2- 172.21.15.52:0/1220248260 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7ff784103300 0x7ff78419b640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:30:06.920 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:30:06.926+0000 7ff782ffd640 1 -- 172.21.15.52:0/1220248260 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7ff784103300 msgr2=0x7ff78419b640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:30:06.920 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:30:06.926+0000 7ff782ffd640 1 --2- 172.21.15.52:0/1220248260 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7ff784103300 0x7ff78419b640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:30:06.920 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:30:06.926+0000 7ff782ffd640 1 -- 172.21.15.52:0/1220248260 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff784109050 msgr2=0x7ff7841a66c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:30:06.920 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:30:06.926+0000 7ff782ffd640 1 --2- 172.21.15.52:0/1220248260 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff784109050 0x7ff7841a66c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:30:06.920 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:30:06.926+0000 7ff782ffd640 1 -- 172.21.15.52:0/1220248260 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff78419c400 con 0x7ff784108bb0 2024-05-27T07:30:06.920 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:30:06.926+0000 7ff7837fe640 1 --2- 172.21.15.52:0/1220248260 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7ff784103300 0x7ff78419b640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-27T07:30:06.920 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:30:06.927+0000 7ff782ffd640 1 --2- 172.21.15.52:0/1220248260 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7ff784108bb0 0x7ff78419bb80 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7ff778005d00 tx=0x7ff77800dde0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:30:06.921 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:30:06.927+0000 7ff780ff9640 1 -- 172.21.15.52:0/1220248260 <== mon.2 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff778017070 con 0x7ff784108bb0 2024-05-27T07:30:06.921 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:30:06.927+0000 7ff780ff9640 1 -- 172.21.15.52:0/1220248260 <== mon.2 v2:172.21.15.105:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ff778009c30 con 0x7ff784108bb0 2024-05-27T07:30:06.921 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:30:06.927+0000 7ff789b1f640 1 -- 172.21.15.52:0/1220248260 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff7841a6cc0 con 0x7ff784108bb0 2024-05-27T07:30:06.921 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:30:06.927+0000 7ff780ff9640 1 -- 172.21.15.52:0/1220248260 <== mon.2 v2:172.21.15.105:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff77800f620 con 0x7ff784108bb0 2024-05-27T07:30:06.921 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:30:06.927+0000 7ff789b1f640 1 -- 172.21.15.52:0/1220248260 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff7841a71c0 con 0x7ff784108bb0 2024-05-27T07:30:06.924 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:30:06.930+0000 7ff7909d3740 1 Processor -- start 2024-05-27T07:30:06.924 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:30:06.930+0000 7ff7909d3740 1 -- start start 2024-05-27T07:30:06.924 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:30:06.930+0000 7ff7909d3740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55fae84f61b0 0x55fae88973d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:30:06.924 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:30:06.930+0000 7ff7909d3740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55fae84cd2f0 0x55fae8897910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:30:06.924 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:30:06.930+0000 7ff7909d3740 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55fae84d54d0 0x55fae8897e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:30:06.924 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:30:06.930+0000 7ff7909d3740 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x55fae8809c30 con 0x55fae84d54d0 2024-05-27T07:30:06.924 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:30:06.930+0000 7ff7909d3740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x55fae87f97d0 con 0x55fae84cd2f0 2024-05-27T07:30:06.924 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:30:06.930+0000 7ff7909d3740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x55fae87fd810 con 0x55fae84f61b0 2024-05-27T07:30:06.924 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:30:06.930+0000 7ff780ff9640 1 -- 172.21.15.52:0/1220248260 <== mon.2 v2:172.21.15.105:3300/0 4 ==== mgrmap(e 26) ==== 149783+0+0 (secure 0 0 0) 0x7ff77801a070 con 0x7ff784108bb0 2024-05-27T07:30:06.924 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:30:06.930+0000 7ff7837fe640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55fae84cd2f0 0x55fae8897910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:30:06.924 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:30:06.930+0000 7ff7837fe640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55fae84cd2f0 0x55fae8897910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.52:54142/0 (socket says 172.21.15.52:54142) 2024-05-27T07:30:06.924 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:30:06.930+0000 7ff7837fe640 1 -- 172.21.15.52:0/2725192838 learned_addr learned my addr 172.21.15.52:0/2725192838 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-05-27T07:30:06.925 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:30:06.930+0000 7ff782ffd640 1 --2- 172.21.15.52:0/2725192838 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55fae84d54d0 0x55fae8897e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:30:06.925 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:30:06.930+0000 7ff783fff640 1 --2- 172.21.15.52:0/2725192838 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55fae84f61b0 0x55fae88973d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:30:06.925 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:30:06.930+0000 7ff7837fe640 1 -- 172.21.15.52:0/2725192838 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55fae84f61b0 msgr2=0x55fae88973d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:30:06.925 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:30:06.930+0000 7ff7837fe640 1 --2- 172.21.15.52:0/2725192838 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55fae84f61b0 0x55fae88973d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:30:06.925 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:30:06.930+0000 7ff7837fe640 1 -- 172.21.15.52:0/2725192838 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55fae84d54d0 msgr2=0x55fae8897e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:30:06.925 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:30:06.930+0000 7ff7837fe640 1 --2- 172.21.15.52:0/2725192838 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55fae84d54d0 0x55fae8897e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:30:06.925 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:30:06.930+0000 7ff7837fe640 1 -- 172.21.15.52:0/2725192838 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55fae850e210 con 0x55fae84cd2f0 2024-05-27T07:30:06.925 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:30:06.931+0000 7ff783fff640 1 --2- 172.21.15.52:0/2725192838 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55fae84f61b0 0x55fae88973d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-27T07:30:06.925 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:30:06.931+0000 7ff7837fe640 1 --2- 172.21.15.52:0/2725192838 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55fae84cd2f0 0x55fae8897910 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7ff774009c00 tx=0x7ff774010740 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c9cdb5d51985ee0f server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:30:06.925 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:30:06.931+0000 7ff769ffb640 1 -- 172.21.15.52:0/2725192838 <== mon.2 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff774005020 con 0x55fae84cd2f0 2024-05-27T07:30:06.925 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:30:06.931+0000 7ff769ffb640 1 -- 172.21.15.52:0/2725192838 <== mon.2 v2:172.21.15.105:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ff7740047a0 con 0x55fae84cd2f0 2024-05-27T07:30:06.925 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:30:06.931+0000 7ff780ff9640 1 --2- 172.21.15.52:0/1220248260 >> [v2:172.21.15.52:6800/1822318384,v1:172.21.15.52:6801/1822318384] conn(0x7ff76c0b2950 0x7ff76c0b4e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:30:06.925 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:30:06.931+0000 7ff769ffb640 1 -- 172.21.15.52:0/2725192838 <== mon.2 v2:172.21.15.105:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff774012670 con 0x55fae84cd2f0 2024-05-27T07:30:06.926 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:30:06.931+0000 7ff783fff640 1 --2- 172.21.15.52:0/1220248260 >> [v2:172.21.15.52:6800/1822318384,v1:172.21.15.52:6801/1822318384] conn(0x7ff76c0b2950 0x7ff76c0b4e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:30:06.926 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:30:06.931+0000 7ff780ff9640 1 -- 172.21.15.52:0/1220248260 <== mon.2 v2:172.21.15.105:3300/0 5 ==== osd_map(111..111 src has 1..111) ==== 9379+0+0 (secure 0 0 0) 0x7ff7780e2370 con 0x7ff784108bb0 2024-05-27T07:30:06.926 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:30:06.932+0000 7ff7909d3740 1 -- 172.21.15.52:0/2725192838 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55fae84cd2f0 msgr2=0x55fae8897910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:30:06.926 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:30:06.932+0000 7ff783fff640 1 --2- 172.21.15.52:0/1220248260 >> [v2:172.21.15.52:6800/1822318384,v1:172.21.15.52:6801/1822318384] conn(0x7ff76c0b2950 0x7ff76c0b4e60 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x55fae8894540 tx=0x7ff77c0094a0 comp rx=0 tx=0).ready entity=mgr.15333 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:30:06.926 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:30:06.932+0000 7ff7909d3740 1 --2- 172.21.15.52:0/2725192838 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55fae84cd2f0 0x55fae8897910 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7ff774009c00 tx=0x7ff774010740 comp rx=0 tx=0).stop 2024-05-27T07:30:06.926 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:30:06.932+0000 7ff7909d3740 1 -- 172.21.15.52:0/2725192838 shutdown_connections 2024-05-27T07:30:06.926 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:30:06.932+0000 7ff7909d3740 1 --2- 172.21.15.52:0/2725192838 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55fae84d54d0 0x55fae8897e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:30:06.926 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:30:06.932+0000 7ff7909d3740 1 --2- 172.21.15.52:0/2725192838 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55fae84cd2f0 0x55fae8897910 unknown :-1 s=CLOSED pgs=144 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:30:06.926 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:30:06.932+0000 7ff7909d3740 1 --2- 172.21.15.52:0/2725192838 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55fae84f61b0 0x55fae88973d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:30:06.926 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:30:06.932+0000 7ff7909d3740 1 -- 172.21.15.52:0/2725192838 >> 172.21.15.52:0/2725192838 conn(0x55fae84ea300 msgr2=0x55fae8893b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:30:06.926 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:30:06.932+0000 7ff7909d3740 1 -- 172.21.15.52:0/2725192838 shutdown_connections 2024-05-27T07:30:06.926 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:30:06.932+0000 7ff7909d3740 1 -- 172.21.15.52:0/2725192838 wait complete. 2024-05-27T07:30:06.927 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:30:06.933+0000 7ff7909d3740 1 -- start start 2024-05-27T07:30:06.927 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:30:06.933+0000 7ff7909d3740 1 Processor -- start 2024-05-27T07:30:06.929 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:30:06.933+0000 7ff7909d3740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55fae84cd2f0 0x55fae88a3410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:30:06.930 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:30:06.933+0000 7ff7909d3740 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55fae84d54d0 0x55fae88a3950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:30:06.930 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:30:06.933+0000 7ff7909d3740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55fae84f61b0 0x55fae88a18e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:30:06.930 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:30:06.933+0000 7ff7909d3740 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x55fae84df790 con 0x55fae84d54d0 2024-05-27T07:30:06.930 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:30:06.933+0000 7ff7909d3740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x55fae87b5890 con 0x55fae84cd2f0 2024-05-27T07:30:06.930 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:30:06.933+0000 7ff7909d3740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x55fae88a4500 con 0x55fae84f61b0 2024-05-27T07:30:06.930 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:30:06.934+0000 7ff7909d3740 10 client.0 _ll_register_callbacks cb 0x55fae850da00 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T07:30:06.930 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:30:06.934+0000 7ff782ffd640 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55fae84d54d0 0x55fae88a3950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:30:06.930 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:30:06.934+0000 7ff7837fe640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55fae84cd2f0 0x55fae88a3410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:30:06.930 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:30:06.934+0000 7ff782ffd640 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55fae84d54d0 0x55fae88a3950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.52:37954/0 (socket says 172.21.15.52:37954) 2024-05-27T07:30:06.930 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:30:06.934+0000 7ff782ffd640 1 -- 172.21.15.52:0/2815140268 learned_addr learned my addr 172.21.15.52:0/2815140268 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-05-27T07:30:06.930 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:30:06.934+0000 7ff783fff640 1 --2- 172.21.15.52:0/2815140268 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55fae84f61b0 0x55fae88a18e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:30:06.931 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:30:06.934+0000 7ff7837fe640 1 -- 172.21.15.52:0/2815140268 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55fae84f61b0 msgr2=0x55fae88a18e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:30:06.931 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:30:06.934+0000 7ff7837fe640 1 --2- 172.21.15.52:0/2815140268 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55fae84f61b0 0x55fae88a18e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:30:06.931 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:30:06.934+0000 7ff7837fe640 1 -- 172.21.15.52:0/2815140268 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55fae84d54d0 msgr2=0x55fae88a3950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:30:06.931 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:30:06.934+0000 7ff7837fe640 1 --2- 172.21.15.52:0/2815140268 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55fae84d54d0 0x55fae88a3950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:30:06.931 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:30:06.934+0000 7ff7837fe640 1 -- 172.21.15.52:0/2815140268 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55fae85da620 con 0x55fae84cd2f0 2024-05-27T07:30:06.931 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:30:06.934+0000 7ff782ffd640 1 --2- 172.21.15.52:0/2815140268 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55fae84d54d0 0x55fae88a3950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-27T07:30:06.931 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:30:06.934+0000 7ff7837fe640 1 --2- 172.21.15.52:0/2815140268 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55fae84cd2f0 0x55fae88a3410 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x55fae8898780 tx=0x7ff774005890 comp rx=0 tx=0).ready entity=mon.2 client_cookie=dcdd84a332eb74ce server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:30:06.931 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:30:06.934+0000 7ff762ffd640 10 client.0 ms_handle_connect on v2:172.21.15.105:3300/0 2024-05-27T07:30:06.931 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:30:06.934+0000 7ff762ffd640 1 -- 172.21.15.52:0/2815140268 <== mon.2 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff7740097f0 con 0x55fae84cd2f0 2024-05-27T07:30:06.931 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:30:06.935+0000 7ff7909d3740 1 -- 172.21.15.52:0/2815140268 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7ff7740047a0 con 0x55fae84cd2f0 2024-05-27T07:30:06.931 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:30:06.935+0000 7ff762ffd640 1 -- 172.21.15.52:0/2815140268 <== mon.2 v2:172.21.15.105:3300/0 2 ==== osd_map(111..111 src has 1..111) ==== 9379+0+0 (secure 0 0 0) 0x7ff774005110 con 0x55fae84cd2f0 2024-05-27T07:30:06.931 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:30:06.935+0000 7ff762ffd640 1 -- 172.21.15.52:0/2815140268 <== mon.2 v2:172.21.15.105:3300/0 3 ==== mon_get_version_reply(handle=1 version=76) ==== 24+0+0 (secure 0 0 0) 0x7ff774005310 con 0x55fae84cd2f0 2024-05-27T07:30:06.931 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:30:06.935+0000 7ff7909d3740 10 client.26102 fetch_fsmap learned FSMap version 76 2024-05-27T07:30:06.932 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:30:06.935+0000 7ff7909d3740 1 -- 172.21.15.52:0/2815140268 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,fsmap=76,monmap=4+}) -- 0x55fae88b07e0 con 0x55fae84cd2f0 2024-05-27T07:30:06.932 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:30:06.935+0000 7ff762ffd640 1 -- 172.21.15.52:0/2815140268 <== mon.2 v2:172.21.15.105:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ff7740047a0 con 0x55fae84cd2f0 2024-05-27T07:30:06.932 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:30:06.935+0000 7ff762ffd640 1 -- 172.21.15.52:0/2815140268 <== mon.2 v2:172.21.15.105:3300/0 5 ==== fsmap(e 76) ==== 5982+0+0 (secure 0 0 0) 0x7ff77400f040 con 0x55fae84cd2f0 2024-05-27T07:30:06.932 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:30:06.935+0000 7ff7909d3740 10 client.26102 fetch_fsmap finished waiting for FSMap version 76 2024-05-27T07:30:06.932 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:30:06.935+0000 7ff7909d3740 10 client.26102 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-05-27T07:30:06.932 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:30:06.935+0000 7ff7909d3740 20 client.26102 populate_metadata read hostname 'smithi052' 2024-05-27T07:30:06.932 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:30:06.935+0000 7ff7909d3740 1 --2- 172.21.15.52:0/2815140268 >> [v2:172.21.15.105:6838/500348151,v1:172.21.15.105:6839/500348151] conn(0x55fae84b4d80 0x55fae88b4120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:30:06.932 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:30:06.935+0000 7ff7909d3740 4 client.26102 mds_command: new command op to 24527 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-27T07:30:06.932 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:30:06.935+0000 7ff7909d3740 1 -- 172.21.15.52:0/2815140268 --> [v2:172.21.15.105:6838/500348151,v1:172.21.15.105:6839/500348151] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55fae880bb80 con 0x55fae84b4d80 2024-05-27T07:30:06.932 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:30:06.936+0000 7ff782ffd640 1 --2- 172.21.15.52:0/2815140268 >> [v2:172.21.15.105:6838/500348151,v1:172.21.15.105:6839/500348151] conn(0x55fae84b4d80 0x55fae88b4120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:30:06.932 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:30:06.936+0000 7ff762ffd640 10 client.26102 ms_handle_connect on v2:172.21.15.105:6838/500348151 2024-05-27T07:30:06.932 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:30:06.936+0000 7ff782ffd640 1 --2- 172.21.15.52:0/2815140268 >> [v2:172.21.15.105:6838/500348151,v1:172.21.15.105:6839/500348151] conn(0x55fae84b4d80 0x55fae88b4120 crc :-1 s=READY pgs=101 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=6ba9c88a80b48e37 server_cookie=3caf03b1997be900 in_seq=0 out_seq=0 2024-05-27T07:30:06.933 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:30:06.941+0000 7ff762ffd640 1 -- 172.21.15.52:0/2815140268 <== mds.0 v2:172.21.15.105:6838/500348151 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24041 (crc 0 0 0) 0x55fae880bb80 con 0x55fae84b4d80 2024-05-27T07:30:06.934 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:30:06.941+0000 7ff762ffd640 10 client.26102 handle_command_reply: tid=0 2024-05-27T07:30:06.934 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:30:06.941+0000 7ff7909d3740 1 client.26102 shutdown 2024-05-27T07:30:06.934 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:30:06.941+0000 7ff7909d3740 1 -- 172.21.15.52:0/2815140268 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55fae84cd2f0 msgr2=0x55fae88a3410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:30:06.934 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:30:06.941+0000 7ff7909d3740 1 --2- 172.21.15.52:0/2815140268 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55fae84cd2f0 0x55fae88a3410 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x55fae8898780 tx=0x7ff774005890 comp rx=0 tx=0).stop 2024-05-27T07:30:06.934 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:30:06.941+0000 7ff7909d3740 1 -- 172.21.15.52:0/2815140268 shutdown_connections 2024-05-27T07:30:06.934 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:30:06.941+0000 7ff7909d3740 1 --2- 172.21.15.52:0/2815140268 >> [v2:172.21.15.105:6838/500348151,v1:172.21.15.105:6839/500348151] conn(0x55fae84b4d80 0x55fae88b4120 crc :-1 s=READY pgs=101 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:30:06.934 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:30:06.941+0000 7ff7909d3740 1 --2- 172.21.15.52:0/2815140268 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55fae84f61b0 0x55fae88a18e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:30:06.934 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:30:06.941+0000 7ff762ffd640 0 client.26102 ms_handle_reset on v2:172.21.15.105:6838/500348151 2024-05-27T07:30:06.934 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:30:06.941+0000 7ff7909d3740 1 --2- 172.21.15.52:0/2815140268 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55fae84d54d0 0x55fae88a3950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:30:06.934 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:30:06.941+0000 7ff7909d3740 1 --2- 172.21.15.52:0/2815140268 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55fae84cd2f0 0x55fae88a3410 unknown :-1 s=CLOSED pgs=145 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:30:06.934 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:30:06.942+0000 7ff7909d3740 1 -- 172.21.15.52:0/2815140268 >> 172.21.15.52:0/2815140268 conn(0x55fae84ea300 msgr2=0x55fae88990f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:30:06.934 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:30:06.942+0000 7ff7909d3740 1 -- 172.21.15.52:0/2815140268 shutdown_connections 2024-05-27T07:30:06.935 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:30:06.942+0000 7ff7909d3740 1 -- 172.21.15.52:0/2815140268 wait complete. 2024-05-27T07:30:06.935 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:30:06.942+0000 7ff7909d3740 20 client.26102 trim_cache size 0 max 16384 2024-05-27T07:30:06.951 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:30:06.958+0000 7ff7909d3740 1 Processor -- start 2024-05-27T07:30:06.951 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:30:06.958+0000 7ff7909d3740 1 -- start start 2024-05-27T07:30:06.951 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:30:06.959+0000 7ff7909d3740 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55fae84cd2f0 0x55fae88a08c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:30:06.951 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:30:06.959+0000 7ff7909d3740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55fae84d54d0 0x55fae8893160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:30:06.951 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:30:06.959+0000 7ff7909d3740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55fae84f61b0 0x55fae88936a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:30:06.951 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:30:06.959+0000 7ff7909d3740 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x7ff7841b1f10 con 0x55fae84cd2f0 2024-05-27T07:30:06.951 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:30:06.959+0000 7ff7909d3740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7ff7841b25e0 con 0x55fae84f61b0 2024-05-27T07:30:06.951 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:30:06.959+0000 7ff7909d3740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7ff7841afc40 con 0x55fae84d54d0 2024-05-27T07:30:06.952 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:30:06.959+0000 7ff783fff640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55fae84f61b0 0x55fae88936a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:30:06.952 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:30:06.959+0000 7ff7837fe640 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55fae84cd2f0 0x55fae88a08c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:30:06.952 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:30:06.959+0000 7ff783fff640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55fae84f61b0 0x55fae88936a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.52:54156/0 (socket says 172.21.15.52:54156) 2024-05-27T07:30:06.952 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:30:06.959+0000 7ff7837fe640 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55fae84cd2f0 0x55fae88a08c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.52:37970/0 (socket says 172.21.15.52:37970) 2024-05-27T07:30:06.952 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:30:06.959+0000 7ff783fff640 1 -- 172.21.15.52:0/4063623940 learned_addr learned my addr 172.21.15.52:0/4063623940 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-05-27T07:30:06.952 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:30:06.959+0000 7ff782ffd640 1 --2- 172.21.15.52:0/4063623940 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55fae84d54d0 0x55fae8893160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:30:06.952 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:30:06.959+0000 7ff783fff640 1 -- 172.21.15.52:0/4063623940 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55fae84d54d0 msgr2=0x55fae8893160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:30:06.952 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:30:06.959+0000 7ff783fff640 1 --2- 172.21.15.52:0/4063623940 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55fae84d54d0 0x55fae8893160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:30:06.952 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:30:06.959+0000 7ff783fff640 1 -- 172.21.15.52:0/4063623940 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55fae84cd2f0 msgr2=0x55fae88a08c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:30:06.952 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:30:06.959+0000 7ff783fff640 1 --2- 172.21.15.52:0/4063623940 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55fae84cd2f0 0x55fae88a08c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:30:06.952 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:30:06.959+0000 7ff783fff640 1 -- 172.21.15.52:0/4063623940 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55fae888de60 con 0x55fae84f61b0 2024-05-27T07:30:06.952 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:30:06.959+0000 7ff7837fe640 1 --2- 172.21.15.52:0/4063623940 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55fae84cd2f0 0x55fae88a08c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-27T07:30:06.953 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:30:06.960+0000 7ff783fff640 1 --2- 172.21.15.52:0/4063623940 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55fae84f61b0 0x55fae88936a0 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7ff77c035000 tx=0x7ff77c044000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=6ca0d58533c27f1d server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:30:06.953 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:30:06.960+0000 7ff7697fa640 1 -- 172.21.15.52:0/4063623940 <== mon.2 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff77c003970 con 0x55fae84f61b0 2024-05-27T07:30:06.953 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:30:06.960+0000 7ff7697fa640 1 -- 172.21.15.52:0/4063623940 <== mon.2 v2:172.21.15.105:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ff77c036050 con 0x55fae84f61b0 2024-05-27T07:30:06.953 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:30:06.960+0000 7ff7697fa640 1 -- 172.21.15.52:0/4063623940 <== mon.2 v2:172.21.15.105:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff77c02f660 con 0x55fae84f61b0 2024-05-27T07:30:06.956 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:30:06.961+0000 7ff7909d3740 1 -- 172.21.15.52:0/4063623940 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55fae84f61b0 msgr2=0x55fae88936a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:30:06.956 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:30:06.961+0000 7ff7909d3740 1 --2- 172.21.15.52:0/4063623940 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55fae84f61b0 0x55fae88936a0 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7ff77c035000 tx=0x7ff77c044000 comp rx=0 tx=0).stop 2024-05-27T07:30:06.956 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:30:06.961+0000 7ff7909d3740 1 -- 172.21.15.52:0/4063623940 shutdown_connections 2024-05-27T07:30:06.956 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:30:06.961+0000 7ff7909d3740 1 --2- 172.21.15.52:0/4063623940 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55fae84f61b0 0x55fae88936a0 unknown :-1 s=CLOSED pgs=146 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:30:06.957 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:30:06.961+0000 7ff7909d3740 1 --2- 172.21.15.52:0/4063623940 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55fae84d54d0 0x55fae8893160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:30:06.957 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:30:06.961+0000 7ff7909d3740 1 --2- 172.21.15.52:0/4063623940 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55fae84cd2f0 0x55fae88a08c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:30:06.957 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:30:06.961+0000 7ff7909d3740 1 -- 172.21.15.52:0/4063623940 >> 172.21.15.52:0/4063623940 conn(0x55fae84ea300 msgr2=0x55fae88a01b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:30:06.957 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:30:06.961+0000 7ff7909d3740 1 -- 172.21.15.52:0/4063623940 shutdown_connections 2024-05-27T07:30:06.957 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:30:06.961+0000 7ff7909d3740 1 -- 172.21.15.52:0/4063623940 wait complete. 2024-05-27T07:30:06.957 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:30:06.961+0000 7ff7909d3740 1 -- start start 2024-05-27T07:30:06.957 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:30:06.962+0000 7ff7909d3740 1 Processor -- start 2024-05-27T07:30:06.957 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:30:06.962+0000 7ff7909d3740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55fae84cd2f0 0x55fae88a6db0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:30:06.957 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:30:06.962+0000 7ff7909d3740 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55fae84d54d0 0x55fae88a72f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:30:06.957 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:30:06.962+0000 7ff7909d3740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55fae84f61b0 0x55fae89147f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:30:06.957 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:30:06.962+0000 7ff7909d3740 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x7ff7841b0310 con 0x55fae84d54d0 2024-05-27T07:30:06.957 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:30:06.962+0000 7ff7909d3740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7ff7841b1840 con 0x55fae84f61b0 2024-05-27T07:30:06.958 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:30:06.962+0000 7ff7909d3740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7ff7841b0aa0 con 0x55fae84cd2f0 2024-05-27T07:30:06.958 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:30:06.962+0000 7ff7909d3740 10 client.0 _ll_register_callbacks cb 0x7ff774009a50 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T07:30:06.958 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:30:06.962+0000 7ff782ffd640 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55fae84d54d0 0x55fae88a72f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:30:06.958 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:30:06.962+0000 7ff783fff640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55fae84f61b0 0x55fae89147f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:30:06.958 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:30:06.962+0000 7ff783fff640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55fae84f61b0 0x55fae89147f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.52:54158/0 (socket says 172.21.15.52:54158) 2024-05-27T07:30:06.958 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:30:06.962+0000 7ff782ffd640 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55fae84d54d0 0x55fae88a72f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.52:37974/0 (socket says 172.21.15.52:37974) 2024-05-27T07:30:06.958 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:30:06.962+0000 7ff783fff640 1 -- 172.21.15.52:0/421624937 learned_addr learned my addr 172.21.15.52:0/421624937 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-05-27T07:30:06.958 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:30:06.962+0000 7ff7837fe640 1 --2- 172.21.15.52:0/421624937 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55fae84cd2f0 0x55fae88a6db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:30:06.958 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:30:06.963+0000 7ff783fff640 1 -- 172.21.15.52:0/421624937 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55fae84cd2f0 msgr2=0x55fae88a6db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:30:06.958 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:30:06.963+0000 7ff783fff640 1 --2- 172.21.15.52:0/421624937 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55fae84cd2f0 0x55fae88a6db0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:30:06.958 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:30:06.963+0000 7ff783fff640 1 -- 172.21.15.52:0/421624937 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55fae84d54d0 msgr2=0x55fae88a72f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:30:06.959 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:30:06.963+0000 7ff783fff640 1 --2- 172.21.15.52:0/421624937 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55fae84d54d0 0x55fae88a72f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:30:06.959 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:30:06.963+0000 7ff783fff640 1 -- 172.21.15.52:0/421624937 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55fae8914d30 con 0x55fae84f61b0 2024-05-27T07:30:06.959 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:30:06.963+0000 7ff783fff640 1 --2- 172.21.15.52:0/421624937 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55fae84f61b0 0x55fae89147f0 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7ff77c043040 tx=0x7ff77c004730 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d88105139ae7e0b4 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:30:06.959 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:30:06.963+0000 7ff762ffd640 10 client.0 ms_handle_connect on v2:172.21.15.105:3300/0 2024-05-27T07:30:06.959 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:30:06.963+0000 7ff762ffd640 1 -- 172.21.15.52:0/421624937 <== mon.2 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff77c061060 con 0x55fae84f61b0 2024-05-27T07:30:06.959 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:30:06.963+0000 7ff7909d3740 1 -- 172.21.15.52:0/421624937 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7ff77c036050 con 0x55fae84f61b0 2024-05-27T07:30:06.959 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:30:06.963+0000 7ff762ffd640 1 -- 172.21.15.52:0/421624937 <== mon.2 v2:172.21.15.105:3300/0 2 ==== osd_map(111..111 src has 1..111) ==== 9379+0+0 (secure 0 0 0) 0x7ff77c004450 con 0x55fae84f61b0 2024-05-27T07:30:06.959 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:30:06.963+0000 7ff762ffd640 1 -- 172.21.15.52:0/421624937 <== mon.2 v2:172.21.15.105:3300/0 3 ==== mon_get_version_reply(handle=1 version=76) ==== 24+0+0 (secure 0 0 0) 0x7ff77c068050 con 0x55fae84f61b0 2024-05-27T07:30:06.959 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:30:06.963+0000 7ff7909d3740 10 client.26108 fetch_fsmap learned FSMap version 76 2024-05-27T07:30:06.959 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:30:06.963+0000 7ff7909d3740 1 -- 172.21.15.52:0/421624937 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,fsmap=76,monmap=4+}) -- 0x55fae891c030 con 0x55fae84f61b0 2024-05-27T07:30:06.959 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:30:06.964+0000 7ff762ffd640 1 -- 172.21.15.52:0/421624937 <== mon.2 v2:172.21.15.105:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ff77c036050 con 0x55fae84f61b0 2024-05-27T07:30:06.959 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:30:06.964+0000 7ff762ffd640 1 -- 172.21.15.52:0/421624937 <== mon.2 v2:172.21.15.105:3300/0 5 ==== fsmap(e 76) ==== 5982+0+0 (secure 0 0 0) 0x7ff77c02fbb0 con 0x55fae84f61b0 2024-05-27T07:30:06.959 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:30:06.964+0000 7ff7909d3740 10 client.26108 fetch_fsmap finished waiting for FSMap version 76 2024-05-27T07:30:06.960 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:30:06.964+0000 7ff7909d3740 10 client.26108 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-05-27T07:30:06.960 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:30:06.964+0000 7ff7909d3740 20 client.26108 populate_metadata read hostname 'smithi052' 2024-05-27T07:30:06.960 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:30:06.964+0000 7ff7909d3740 1 --2- 172.21.15.52:0/421624937 >> [v2:172.21.15.105:6838/500348151,v1:172.21.15.105:6839/500348151] conn(0x55fae848a1c0 0x55fae891e710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:30:06.960 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:30:06.964+0000 7ff7909d3740 4 client.26108 mds_command: new command op to 24527 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-05-27T07:30:06.960 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:30:06.964+0000 7ff7909d3740 1 -- 172.21.15.52:0/421624937 --> [v2:172.21.15.105:6838/500348151,v1:172.21.15.105:6839/500348151] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55fae8805360 con 0x55fae848a1c0 2024-05-27T07:30:06.960 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:30:06.964+0000 7ff7837fe640 1 --2- 172.21.15.52:0/421624937 >> [v2:172.21.15.105:6838/500348151,v1:172.21.15.105:6839/500348151] conn(0x55fae848a1c0 0x55fae891e710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:30:06.960 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:30:06.965+0000 7ff762ffd640 10 client.26108 ms_handle_connect on v2:172.21.15.105:6838/500348151 2024-05-27T07:30:06.960 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:30:06.965+0000 7ff7837fe640 1 --2- 172.21.15.52:0/421624937 >> [v2:172.21.15.105:6838/500348151,v1:172.21.15.105:6839/500348151] conn(0x55fae848a1c0 0x55fae891e710 crc :-1 s=READY pgs=102 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=aa56698874fe16df server_cookie=79a5bda779f74df1 in_seq=0 out_seq=0 2024-05-27T07:30:07.230 INFO:journalctl@ceph.mds.j.smithi052.stdout:May 27 07:30:06 smithi052 podman[106985]: 2024-05-27 07:30:06.943614642 +0000 UTC m=+1.322309285 container remove d03f6d3925720a9a1b1ac9712defc07799179d80b7913701e4ef645e7a0b5cd1 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-j, org.label-schema.schema-version=1.0, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, GIT_CLEAN=True, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, org.label-schema.build-date=20240521, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, maintainer=Guillaume Abrioux ) 2024-05-27T07:30:07.230 INFO:journalctl@ceph.mds.j.smithi052.stdout:May 27 07:30:06 smithi052 systemd[1]: ceph-73df2636-1bf8-11ef-bc9b-c7b262605968@mds.j.service: Main process exited, code=exited, status=134/n/a 2024-05-27T07:30:07.476 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:30:07 smithi105 ceph-mon[35977]: pgmap v148: 129 pgs: 129 active+clean; 313 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 475 KiB/s rd, 7.4 MiB/s wr, 563 op/s 2024-05-27T07:30:07.476 INFO:journalctl@ceph.mds.k.smithi105.stdout:May 27 07:30:07 smithi105 systemd[1]: ceph-73df2636-1bf8-11ef-bc9b-c7b262605968@mds.k.service: Scheduled restart job, restart counter is at 4. 2024-05-27T07:30:07.476 INFO:journalctl@ceph.mds.k.smithi105.stdout:May 27 07:30:07 smithi105 systemd[1]: Stopped Ceph mds.k for 73df2636-1bf8-11ef-bc9b-c7b262605968. 2024-05-27T07:30:07.476 INFO:journalctl@ceph.mds.k.smithi105.stdout:May 27 07:30:07 smithi105 systemd[1]: Starting Ceph mds.k for 73df2636-1bf8-11ef-bc9b-c7b262605968... 2024-05-27T07:30:07.563 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:30:07 smithi052 ceph-mon[31955]: pgmap v148: 129 pgs: 129 active+clean; 313 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 475 KiB/s rd, 7.4 MiB/s wr, 563 op/s 2024-05-27T07:30:07.593 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:30:07.602+0000 7fbc9c7ee700 1 -- 172.21.15.52:0/4067149850 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fbc941034d0 msgr2=0x7fbc941038e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:30:07.593 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:30:07.602+0000 7fbc9c7ee700 1 --2- 172.21.15.52:0/4067149850 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fbc941034d0 0x7fbc941038e0 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7fbc90009990 tx=0x7fbc90009ca0 comp rx=0 tx=0).stop 2024-05-27T07:30:07.593 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:30:07.602+0000 7fbc9c7ee700 1 -- 172.21.15.52:0/4067149850 shutdown_connections 2024-05-27T07:30:07.593 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:30:07.602+0000 7fbc9c7ee700 1 --2- 172.21.15.52:0/4067149850 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fbc94108750 0x7fbc941100b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:30:07.594 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:30:07.602+0000 7fbc9c7ee700 1 --2- 172.21.15.52:0/4067149850 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fbc941034d0 0x7fbc941038e0 unknown :-1 s=CLOSED pgs=148 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:30:07.594 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:30:07.602+0000 7fbc9c7ee700 1 --2- 172.21.15.52:0/4067149850 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fbc94107e00 0x7fbc94108210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:30:07.594 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:30:07.602+0000 7fbc9c7ee700 1 -- 172.21.15.52:0/4067149850 >> 172.21.15.52:0/4067149850 conn(0x7fbc940fef20 msgr2=0x7fbc94101340 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:30:07.594 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:30:07.602+0000 7fbc9c7ee700 1 -- 172.21.15.52:0/4067149850 shutdown_connections 2024-05-27T07:30:07.594 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:30:07.602+0000 7fbc9c7ee700 1 -- 172.21.15.52:0/4067149850 wait complete. 2024-05-27T07:30:07.595 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:30:07.603+0000 7fbc9c7ee700 1 Processor -- start 2024-05-27T07:30:07.595 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:30:07.603+0000 7fbc9c7ee700 1 -- start start 2024-05-27T07:30:07.595 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:30:07.604+0000 7fbc9c7ee700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fbc941034d0 0x7fbc9419d090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:30:07.595 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:30:07.604+0000 7fbc9c7ee700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fbc94107e00 0x7fbc9419d630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:30:07.595 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:30:07.604+0000 7fbc9c7ee700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fbc94108750 0x7fbc941a18c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:30:07.595 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:30:07.604+0000 7fbc9a58a700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fbc941034d0 0x7fbc9419d090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:30:07.595 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:30:07.604+0000 7fbc9a58a700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fbc941034d0 0x7fbc9419d090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.52:37990/0 (socket says 172.21.15.52:37990) 2024-05-27T07:30:07.595 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:30:07.604+0000 7fbc9a58a700 1 -- 172.21.15.52:0/1626648419 learned_addr learned my addr 172.21.15.52:0/1626648419 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-05-27T07:30:07.596 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:30:07.604+0000 7fbc9c7ee700 1 -- 172.21.15.52:0/1626648419 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x7fbc94113380 con 0x7fbc941034d0 2024-05-27T07:30:07.596 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:30:07.604+0000 7fbc9c7ee700 1 -- 172.21.15.52:0/1626648419 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7fbc94113200 con 0x7fbc94108750 2024-05-27T07:30:07.596 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:30:07.604+0000 7fbc9c7ee700 1 -- 172.21.15.52:0/1626648419 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7fbc94113500 con 0x7fbc94107e00 2024-05-27T07:30:07.596 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:30:07.604+0000 7fbc99d89700 1 --2- 172.21.15.52:0/1626648419 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fbc94107e00 0x7fbc9419d630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:30:07.597 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:30:07.604+0000 7fbc9ad8b700 1 --2- 172.21.15.52:0/1626648419 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fbc94108750 0x7fbc941a18c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:30:07.597 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:30:07.604+0000 7fbc99d89700 1 -- 172.21.15.52:0/1626648419 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fbc94108750 msgr2=0x7fbc941a18c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:30:07.597 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:30:07.604+0000 7fbc99d89700 1 --2- 172.21.15.52:0/1626648419 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fbc94108750 0x7fbc941a18c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:30:07.597 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:30:07.604+0000 7fbc99d89700 1 -- 172.21.15.52:0/1626648419 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fbc941034d0 msgr2=0x7fbc9419d090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:30:07.597 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:30:07.604+0000 7fbc99d89700 1 --2- 172.21.15.52:0/1626648419 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fbc941034d0 0x7fbc9419d090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:30:07.597 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:30:07.604+0000 7fbc99d89700 1 -- 172.21.15.52:0/1626648419 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbc941a1f60 con 0x7fbc94107e00 2024-05-27T07:30:07.597 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:30:07.605+0000 7fbc99d89700 1 --2- 172.21.15.52:0/1626648419 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fbc94107e00 0x7fbc9419d630 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7fbc8400e9c0 tx=0x7fbc8400ecd0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:30:07.597 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:30:07.605+0000 7fbc8b7fe700 1 -- 172.21.15.52:0/1626648419 <== mon.1 v2:172.21.15.173:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbc8400cbd0 con 0x7fbc94107e00 2024-05-27T07:30:07.597 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:30:07.605+0000 7fbc8b7fe700 1 -- 172.21.15.52:0/1626648419 <== mon.1 v2:172.21.15.173:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fbc8400cd70 con 0x7fbc94107e00 2024-05-27T07:30:07.598 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:30:07.606+0000 7fbc8b7fe700 1 -- 172.21.15.52:0/1626648419 <== mon.1 v2:172.21.15.173:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbc84018680 con 0x7fbc94107e00 2024-05-27T07:30:07.598 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:30:07.606+0000 7fbc9c7ee700 1 -- 172.21.15.52:0/1626648419 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbc941a2250 con 0x7fbc94107e00 2024-05-27T07:30:07.598 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:30:07.606+0000 7fbc9c7ee700 1 -- 172.21.15.52:0/1626648419 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbc941a2680 con 0x7fbc94107e00 2024-05-27T07:30:07.601 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:30:07.610+0000 7fbc8b7fe700 1 -- 172.21.15.52:0/1626648419 <== mon.1 v2:172.21.15.173:3300/0 4 ==== mgrmap(e 26) ==== 149783+0+0 (secure 0 0 0) 0x7fbc84010430 con 0x7fbc94107e00 2024-05-27T07:30:07.602 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:30:07 smithi173 ceph-mon[37000]: pgmap v148: 129 pgs: 129 active+clean; 313 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 475 KiB/s rd, 7.4 MiB/s wr, 563 op/s 2024-05-27T07:30:07.604 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:30:07.611+0000 7fbcab203b80 1 Processor -- start 2024-05-27T07:30:07.604 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:30:07.612+0000 7fbcab203b80 1 -- start start 2024-05-27T07:30:07.605 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:30:07.612+0000 7fbcab203b80 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x564876172ec0 0x5648762f57e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:30:07.605 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:30:07.612+0000 7fbcab203b80 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x5648762f5d20 0x5648762f3430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:30:07.605 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:30:07.612+0000 7fbc9ad8b700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x564876172ec0 0x5648762f57e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:30:07.605 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:30:07.612+0000 7fbcab203b80 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5648762f3970 0x5648762f3d80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:30:07.605 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:30:07.612+0000 7fbc9ad8b700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x564876172ec0 0x5648762f57e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.52:54184/0 (socket says 172.21.15.52:54184) 2024-05-27T07:30:07.605 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:30:07.612+0000 7fbc9ad8b700 1 -- 172.21.15.52:0/2597441605 learned_addr learned my addr 172.21.15.52:0/2597441605 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-05-27T07:30:07.605 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:30:07.613+0000 7fbcab203b80 1 -- 172.21.15.52:0/2597441605 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x56487621fd10 con 0x5648762f5d20 2024-05-27T07:30:07.605 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:30:07.613+0000 7fbcab203b80 1 -- 172.21.15.52:0/2597441605 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x56487620f8b0 con 0x564876172ec0 2024-05-27T07:30:07.605 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:30:07.613+0000 7fbcab203b80 1 -- 172.21.15.52:0/2597441605 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x5648762138f0 con 0x5648762f3970 2024-05-27T07:30:07.605 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:30:07.613+0000 7fbc99d89700 1 --2- 172.21.15.52:0/2597441605 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5648762f3970 0x5648762f3d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:30:07.605 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:30:07.613+0000 7fbc8b7fe700 1 --2- 172.21.15.52:0/1626648419 >> [v2:172.21.15.52:6800/1822318384,v1:172.21.15.52:6801/1822318384] conn(0x7fbc800b2af0 0x7fbc800b5000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:30:07.606 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:30:07.614+0000 7fbc9ad8b700 1 --2- 172.21.15.52:0/1626648419 >> [v2:172.21.15.52:6800/1822318384,v1:172.21.15.52:6801/1822318384] conn(0x7fbc800b2af0 0x7fbc800b5000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:30:07.606 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:30:07.614+0000 7fbc9a58a700 1 --2- 172.21.15.52:0/2597441605 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x5648762f5d20 0x5648762f3430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:30:07.607 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:30:07.614+0000 7fbc9ad8b700 1 -- 172.21.15.52:0/2597441605 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5648762f3970 msgr2=0x5648762f3d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:30:07.607 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:30:07.614+0000 7fbc9ad8b700 1 --2- 172.21.15.52:0/2597441605 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5648762f3970 0x5648762f3d80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:30:07.607 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:30:07.614+0000 7fbc9ad8b700 1 -- 172.21.15.52:0/2597441605 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x5648762f5d20 msgr2=0x5648762f3430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:30:07.607 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:30:07.614+0000 7fbc9ad8b700 1 --2- 172.21.15.52:0/2597441605 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x5648762f5d20 0x5648762f3430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:30:07.607 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:30:07.614+0000 7fbc9ad8b700 1 -- 172.21.15.52:0/2597441605 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56487615f950 con 0x564876172ec0 2024-05-27T07:30:07.607 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:30:07.614+0000 7fbc9ad8b700 1 --2- 172.21.15.52:0/2597441605 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x564876172ec0 0x5648762f57e0 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7fbc8c0078c0 tx=0x7fbc8c007bd0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=829ff7b99e03d9a7 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:30:07.607 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:30:07.615+0000 7fbc8b7fe700 1 -- 172.21.15.52:0/1626648419 <== mon.1 v2:172.21.15.173:3300/0 5 ==== osd_map(111..111 src has 1..111) ==== 9379+0+0 (secure 0 0 0) 0x7fbc840e2b20 con 0x7fbc94107e00 2024-05-27T07:30:07.607 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:30:07.615+0000 7fbc88ff9700 1 -- 172.21.15.52:0/2597441605 <== mon.2 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbc8c040030 con 0x564876172ec0 2024-05-27T07:30:07.607 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:30:07.615+0000 7fbc88ff9700 1 -- 172.21.15.52:0/2597441605 <== mon.2 v2:172.21.15.105:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fbc8c036040 con 0x564876172ec0 2024-05-27T07:30:07.607 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:30:07.615+0000 7fbc88ff9700 1 -- 172.21.15.52:0/2597441605 <== mon.2 v2:172.21.15.105:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbc8c035050 con 0x564876172ec0 2024-05-27T07:30:07.608 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:30:07.617+0000 7fbc9ad8b700 1 --2- 172.21.15.52:0/1626648419 >> [v2:172.21.15.52:6800/1822318384,v1:172.21.15.52:6801/1822318384] conn(0x7fbc800b2af0 0x7fbc800b5000 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7fbc8c043000 tx=0x7fbc8c0446a0 comp rx=0 tx=0).ready entity=mgr.15333 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:30:07.608 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:30:07.617+0000 7fbcab203b80 1 -- 172.21.15.52:0/2597441605 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x564876172ec0 msgr2=0x5648762f57e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:30:07.608 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:30:07.617+0000 7fbcab203b80 1 --2- 172.21.15.52:0/2597441605 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x564876172ec0 0x5648762f57e0 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7fbc8c0078c0 tx=0x7fbc8c007bd0 comp rx=0 tx=0).stop 2024-05-27T07:30:07.608 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:30:07.617+0000 7fbcab203b80 1 -- 172.21.15.52:0/2597441605 shutdown_connections 2024-05-27T07:30:07.608 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:30:07.617+0000 7fbcab203b80 1 --2- 172.21.15.52:0/2597441605 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5648762f3970 0x5648762f3d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:30:07.608 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:30:07.617+0000 7fbcab203b80 1 --2- 172.21.15.52:0/2597441605 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x564876172ec0 0x5648762f57e0 unknown :-1 s=CLOSED pgs=149 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:30:07.609 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:30:07.617+0000 7fbcab203b80 1 --2- 172.21.15.52:0/2597441605 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x5648762f5d20 0x5648762f3430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:30:07.609 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:30:07.617+0000 7fbcab203b80 1 -- 172.21.15.52:0/2597441605 >> 172.21.15.52:0/2597441605 conn(0x5648761cdb70 msgr2=0x5648762eb6c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:30:07.609 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:30:07.617+0000 7fbcab203b80 1 -- 172.21.15.52:0/2597441605 shutdown_connections 2024-05-27T07:30:07.609 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:30:07.617+0000 7fbcab203b80 1 -- 172.21.15.52:0/2597441605 wait complete. 2024-05-27T07:30:07.609 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:30:07.618+0000 7fbcab203b80 1 -- start start 2024-05-27T07:30:07.609 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:30:07.619+0000 7fbcab203b80 1 Processor -- start 2024-05-27T07:30:07.611 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:30:07.619+0000 7fbcab203b80 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x564876172ec0 0x564876300690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:30:07.611 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:30:07.619+0000 7fbcab203b80 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x5648762f3970 0x564876300bd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:30:07.611 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:30:07.619+0000 7fbcab203b80 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5648762f5d20 0x5648762faa90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:30:07.611 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:30:07.619+0000 7fbcab203b80 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x564876116020 con 0x5648762f3970 2024-05-27T07:30:07.611 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:30:07.619+0000 7fbcab203b80 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x5648761b1840 con 0x5648762f5d20 2024-05-27T07:30:07.611 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:30:07.619+0000 7fbcab203b80 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x5648762edf60 con 0x564876172ec0 2024-05-27T07:30:07.611 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:30:07.619+0000 7fbcab203b80 10 client.0 _ll_register_callbacks cb 0x5648761aa9a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T07:30:07.611 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:30:07.620+0000 7fbc99d89700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x5648762f3970 0x564876300bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:30:07.612 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:30:07.620+0000 7fbc99d89700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x5648762f3970 0x564876300bd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.52:38004/0 (socket says 172.21.15.52:38004) 2024-05-27T07:30:07.612 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:30:07.620+0000 7fbc99d89700 1 -- 172.21.15.52:0/1086433902 learned_addr learned my addr 172.21.15.52:0/1086433902 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-05-27T07:30:07.612 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:30:07.620+0000 7fbc9ad8b700 1 --2- 172.21.15.52:0/1086433902 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5648762f5d20 0x5648762faa90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:30:07.612 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:30:07.620+0000 7fbc9a58a700 1 --2- 172.21.15.52:0/1086433902 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x564876172ec0 0x564876300690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:30:07.614 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:30:07.620+0000 7fbc9ad8b700 1 -- 172.21.15.52:0/1086433902 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x564876172ec0 msgr2=0x564876300690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:30:07.614 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:30:07.620+0000 7fbc9ad8b700 1 --2- 172.21.15.52:0/1086433902 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x564876172ec0 0x564876300690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:30:07.614 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:30:07.620+0000 7fbc9ad8b700 1 -- 172.21.15.52:0/1086433902 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x5648762f3970 msgr2=0x564876300bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:30:07.614 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:30:07.620+0000 7fbc9ad8b700 1 --2- 172.21.15.52:0/1086433902 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x5648762f3970 0x564876300bd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:30:07.614 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:30:07.620+0000 7fbc9ad8b700 1 -- 172.21.15.52:0/1086433902 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5648761ae710 con 0x5648762f5d20 2024-05-27T07:30:07.614 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:30:07.620+0000 7fbc9a58a700 1 --2- 172.21.15.52:0/1086433902 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x564876172ec0 0x564876300690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-27T07:30:07.614 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:30:07.621+0000 7fbc9ad8b700 1 --2- 172.21.15.52:0/1086433902 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5648762f5d20 0x5648762faa90 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7fbc8c00b010 tx=0x7fbc8c009920 comp rx=0 tx=0).ready entity=mon.2 client_cookie=5cf87a9b069227c4 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:30:07.614 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:30:07.621+0000 7fbc7dffb700 10 client.0 ms_handle_connect on v2:172.21.15.105:3300/0 2024-05-27T07:30:07.614 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:30:07.621+0000 7fbc7dffb700 1 -- 172.21.15.52:0/1086433902 <== mon.2 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbc8c054060 con 0x5648762f5d20 2024-05-27T07:30:07.615 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:30:07.621+0000 7fbc7dffb700 1 -- 172.21.15.52:0/1086433902 <== mon.2 v2:172.21.15.105:3300/0 2 ==== osd_map(111..111 src has 1..111) ==== 9379+0+0 (secure 0 0 0) 0x7fbc8c004ad0 con 0x5648762f5d20 2024-05-27T07:30:07.615 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:30:07.621+0000 7fbcab203b80 1 -- 172.21.15.52:0/1086433902 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fbc8c036040 con 0x5648762f5d20 2024-05-27T07:30:07.615 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:30:07.621+0000 7fbc7dffb700 1 -- 172.21.15.52:0/1086433902 <== mon.2 v2:172.21.15.105:3300/0 3 ==== mon_get_version_reply(handle=1 version=76) ==== 24+0+0 (secure 0 0 0) 0x7fbc8c040050 con 0x5648762f5d20 2024-05-27T07:30:07.615 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:30:07.622+0000 7fbcab203b80 10 client.26117 fetch_fsmap learned FSMap version 76 2024-05-27T07:30:07.615 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:30:07.622+0000 7fbcab203b80 1 -- 172.21.15.52:0/1086433902 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,fsmap=76,monmap=4+}) -- 0x5648763080a0 con 0x5648762f5d20 2024-05-27T07:30:07.615 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:30:07.622+0000 7fbc7dffb700 1 -- 172.21.15.52:0/1086433902 <== mon.2 v2:172.21.15.105:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fbc8c036040 con 0x5648762f5d20 2024-05-27T07:30:07.615 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:30:07.622+0000 7fbc7dffb700 1 -- 172.21.15.52:0/1086433902 <== mon.2 v2:172.21.15.105:3300/0 5 ==== fsmap(e 76) ==== 5982+0+0 (secure 0 0 0) 0x7fbc8c03b070 con 0x5648762f5d20 2024-05-27T07:30:07.615 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:30:07.622+0000 7fbcab203b80 10 client.26117 fetch_fsmap finished waiting for FSMap version 76 2024-05-27T07:30:07.615 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:30:07.622+0000 7fbcab203b80 10 client.26117 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-05-27T07:30:07.615 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:30:07.622+0000 7fbcab203b80 20 client.26117 populate_metadata read hostname 'smithi052' 2024-05-27T07:30:07.615 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:30:07.622+0000 7fbcab203b80 1 --2- 172.21.15.52:0/1086433902 >> [v2:172.21.15.105:6838/500348151,v1:172.21.15.105:6839/500348151] conn(0x56487630a950 0x56487630cdc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:30:07.615 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:30:07.622+0000 7fbcab203b80 4 client.26117 mds_command: new command op to 24527 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-27T07:30:07.616 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:30:07.622+0000 7fbcab203b80 1 -- 172.21.15.52:0/1086433902 --> [v2:172.21.15.105:6838/500348151,v1:172.21.15.105:6839/500348151] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5648762bca70 con 0x56487630a950 2024-05-27T07:30:07.616 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:30:07.622+0000 7fbc9a58a700 1 --2- 172.21.15.52:0/1086433902 >> [v2:172.21.15.105:6838/500348151,v1:172.21.15.105:6839/500348151] conn(0x56487630a950 0x56487630cdc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:30:07.616 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:30:07.623+0000 7fbc7dffb700 10 client.26117 ms_handle_connect on v2:172.21.15.105:6838/500348151 2024-05-27T07:30:07.616 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:30:07.623+0000 7fbc9a58a700 1 --2- 172.21.15.52:0/1086433902 >> [v2:172.21.15.105:6838/500348151,v1:172.21.15.105:6839/500348151] conn(0x56487630a950 0x56487630cdc0 crc :-1 s=READY pgs=103 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=23f86dfc8618a8af server_cookie=695cda90f8b1153c in_seq=0 out_seq=0 2024-05-27T07:30:07.617 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:30:07.626+0000 7fbc7dffb700 1 -- 172.21.15.52:0/1086433902 <== mds.0 v2:172.21.15.105:6838/500348151 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24041 (crc 0 0 0) 0x5648762bca70 con 0x56487630a950 2024-05-27T07:30:07.617 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:30:07.626+0000 7fbc7dffb700 10 client.26117 handle_command_reply: tid=0 2024-05-27T07:30:07.617 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:30:07.626+0000 7fbcab203b80 1 client.26117 shutdown 2024-05-27T07:30:07.618 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:30:07.626+0000 7fbcab203b80 1 -- 172.21.15.52:0/1086433902 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5648762f5d20 msgr2=0x5648762faa90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:30:07.618 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:30:07.626+0000 7fbcab203b80 1 --2- 172.21.15.52:0/1086433902 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5648762f5d20 0x5648762faa90 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7fbc8c00b010 tx=0x7fbc8c009920 comp rx=0 tx=0).stop 2024-05-27T07:30:07.619 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:30:07.626+0000 7fbcab203b80 1 -- 172.21.15.52:0/1086433902 shutdown_connections 2024-05-27T07:30:07.619 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:30:07.626+0000 7fbcab203b80 1 --2- 172.21.15.52:0/1086433902 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5648762f5d20 0x5648762faa90 unknown :-1 s=CLOSED pgs=150 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:30:07.619 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:30:07.626+0000 7fbcab203b80 1 --2- 172.21.15.52:0/1086433902 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x564876172ec0 0x564876300690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:30:07.619 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:30:07.626+0000 7fbcab203b80 1 --2- 172.21.15.52:0/1086433902 >> [v2:172.21.15.105:6838/500348151,v1:172.21.15.105:6839/500348151] conn(0x56487630a950 0x56487630cdc0 crc :-1 s=READY pgs=103 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:30:07.619 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:30:07.626+0000 7fbcab203b80 1 --2- 172.21.15.52:0/1086433902 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x5648762f3970 0x564876300bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:30:07.620 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:30:07.626+0000 7fbcab203b80 1 -- 172.21.15.52:0/1086433902 >> 172.21.15.52:0/1086433902 conn(0x5648761cdb70 msgr2=0x5648762ee230 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:30:07.620 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:30:07.626+0000 7fbc7dffb700 0 client.26117 ms_handle_reset on v2:172.21.15.105:6838/500348151 2024-05-27T07:30:07.620 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:30:07.627+0000 7fbcab203b80 1 -- 172.21.15.52:0/1086433902 shutdown_connections 2024-05-27T07:30:07.620 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:30:07.627+0000 7fbcab203b80 1 -- 172.21.15.52:0/1086433902 wait complete. 2024-05-27T07:30:07.620 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:30:07.627+0000 7fbcab203b80 20 client.26117 trim_cache size 0 max 16384 2024-05-27T07:30:07.649 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:30:07.658+0000 7fbcab203b80 1 Processor -- start 2024-05-27T07:30:07.649 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:30:07.658+0000 7fbcab203b80 1 -- start start 2024-05-27T07:30:07.650 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:30:07.658+0000 7fbcab203b80 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x564876172ec0 0x5648762e5e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:30:07.650 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:30:07.659+0000 7fbcab203b80 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x5648762f3970 0x564876312580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:30:07.651 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:30:07.659+0000 7fbcab203b80 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5648762f5d20 0x564876312ac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:30:07.651 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:30:07.659+0000 7fbcab203b80 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x7fbc941aefe0 con 0x5648762f3970 2024-05-27T07:30:07.651 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:30:07.659+0000 7fbcab203b80 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7fbc941af6b0 con 0x564876172ec0 2024-05-27T07:30:07.651 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:30:07.659+0000 7fbcab203b80 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7fbc941acd10 con 0x5648762f5d20 2024-05-27T07:30:07.651 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:30:07.659+0000 7fbc9a58a700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x564876172ec0 0x5648762e5e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:30:07.651 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:30:07.659+0000 7fbc9a58a700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x564876172ec0 0x5648762e5e80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.52:54208/0 (socket says 172.21.15.52:54208) 2024-05-27T07:30:07.651 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:30:07.659+0000 7fbc9a58a700 1 -- 172.21.15.52:0/583666772 learned_addr learned my addr 172.21.15.52:0/583666772 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-05-27T07:30:07.651 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:30:07.659+0000 7fbc99d89700 1 --2- 172.21.15.52:0/583666772 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x5648762f3970 0x564876312580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:30:07.651 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:30:07.659+0000 7fbc9ad8b700 1 --2- 172.21.15.52:0/583666772 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5648762f5d20 0x564876312ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:30:07.652 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:30:07.659+0000 7fbc9a58a700 1 -- 172.21.15.52:0/583666772 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5648762f5d20 msgr2=0x564876312ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:30:07.652 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:30:07.659+0000 7fbc9a58a700 1 --2- 172.21.15.52:0/583666772 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5648762f5d20 0x564876312ac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:30:07.652 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:30:07.659+0000 7fbc9a58a700 1 -- 172.21.15.52:0/583666772 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x5648762f3970 msgr2=0x564876312580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:30:07.652 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:30:07.659+0000 7fbc9a58a700 1 --2- 172.21.15.52:0/583666772 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x5648762f3970 0x564876312580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:30:07.652 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:30:07.659+0000 7fbc9a58a700 1 -- 172.21.15.52:0/583666772 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5648762ee8e0 con 0x564876172ec0 2024-05-27T07:30:07.652 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:30:07.659+0000 7fbc9ad8b700 1 --2- 172.21.15.52:0/583666772 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5648762f5d20 0x564876312ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-27T07:30:07.652 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:30:07.659+0000 7fbc9a58a700 1 --2- 172.21.15.52:0/583666772 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x564876172ec0 0x5648762e5e80 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7fbc90000c00 tx=0x7fbc900053b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=e1f618fcbff02867 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:30:07.652 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:30:07.660+0000 7fbc7ffff700 1 -- 172.21.15.52:0/583666772 <== mon.2 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbc9000b7f0 con 0x564876172ec0 2024-05-27T07:30:07.652 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:30:07.660+0000 7fbc7ffff700 1 -- 172.21.15.52:0/583666772 <== mon.2 v2:172.21.15.105:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fbc9002b070 con 0x564876172ec0 2024-05-27T07:30:07.652 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:30:07.660+0000 7fbc7ffff700 1 -- 172.21.15.52:0/583666772 <== mon.2 v2:172.21.15.105:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbc9000fa90 con 0x564876172ec0 2024-05-27T07:30:07.656 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:30:07.661+0000 7fbcab203b80 1 -- 172.21.15.52:0/583666772 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x564876172ec0 msgr2=0x5648762e5e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:30:07.656 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:30:07.661+0000 7fbcab203b80 1 --2- 172.21.15.52:0/583666772 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x564876172ec0 0x5648762e5e80 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7fbc90000c00 tx=0x7fbc900053b0 comp rx=0 tx=0).stop 2024-05-27T07:30:07.656 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:30:07.661+0000 7fbcab203b80 1 -- 172.21.15.52:0/583666772 shutdown_connections 2024-05-27T07:30:07.656 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:30:07.661+0000 7fbcab203b80 1 --2- 172.21.15.52:0/583666772 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5648762f5d20 0x564876312ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:30:07.656 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:30:07.661+0000 7fbcab203b80 1 --2- 172.21.15.52:0/583666772 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x564876172ec0 0x5648762e5e80 unknown :-1 s=CLOSED pgs=151 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:30:07.656 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:30:07.661+0000 7fbcab203b80 1 --2- 172.21.15.52:0/583666772 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x5648762f3970 0x564876312580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:30:07.656 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:30:07.661+0000 7fbcab203b80 1 -- 172.21.15.52:0/583666772 >> 172.21.15.52:0/583666772 conn(0x5648761cdb70 msgr2=0x5648762e5770 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:30:07.656 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:30:07.661+0000 7fbcab203b80 1 -- 172.21.15.52:0/583666772 shutdown_connections 2024-05-27T07:30:07.656 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:30:07.661+0000 7fbcab203b80 1 -- 172.21.15.52:0/583666772 wait complete. 2024-05-27T07:30:07.656 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:30:07.662+0000 7fbcab203b80 1 -- start start 2024-05-27T07:30:07.656 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:30:07.662+0000 7fbcab203b80 1 Processor -- start 2024-05-27T07:30:07.657 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:30:07.662+0000 7fbcab203b80 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x564876172ec0 0x5648763047d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:30:07.657 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:30:07.662+0000 7fbcab203b80 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5648762f3970 0x564876304d10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:30:07.657 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:30:07.662+0000 7fbcab203b80 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x5648762f5d20 0x564876374460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:30:07.657 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:30:07.662+0000 7fbcab203b80 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x7fbc941ad3e0 con 0x5648762f5d20 2024-05-27T07:30:07.657 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:30:07.662+0000 7fbcab203b80 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7fbc941ae910 con 0x5648762f3970 2024-05-27T07:30:07.657 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:30:07.662+0000 7fbcab203b80 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7fbc941adb70 con 0x564876172ec0 2024-05-27T07:30:07.657 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:30:07.662+0000 7fbc9a58a700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x564876172ec0 0x5648763047d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:30:07.657 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:30:07.663+0000 7fbc9a58a700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x564876172ec0 0x5648763047d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.173:3300/0 says I am v2:172.21.15.52:53970/0 (socket says 172.21.15.52:53970) 2024-05-27T07:30:07.657 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:30:07.663+0000 7fbc9a58a700 1 -- 172.21.15.52:0/1635833554 learned_addr learned my addr 172.21.15.52:0/1635833554 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-05-27T07:30:07.657 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:30:07.663+0000 7fbc9ad8b700 1 --2- 172.21.15.52:0/1635833554 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x5648762f5d20 0x564876374460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:30:07.657 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:30:07.663+0000 7fbcab203b80 10 client.0 _ll_register_callbacks cb 0x5648761aa9a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T07:30:07.658 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:30:07.663+0000 7fbc99d89700 1 --2- 172.21.15.52:0/1635833554 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5648762f3970 0x564876304d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:30:07.658 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:30:07.663+0000 7fbc9ad8b700 1 -- 172.21.15.52:0/1635833554 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x564876172ec0 msgr2=0x5648763047d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:30:07.658 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:30:07.663+0000 7fbc9ad8b700 1 --2- 172.21.15.52:0/1635833554 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x564876172ec0 0x5648763047d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:30:07.658 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:30:07.663+0000 7fbc9ad8b700 1 -- 172.21.15.52:0/1635833554 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5648762f3970 msgr2=0x564876304d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:30:07.658 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:30:07.663+0000 7fbc9ad8b700 1 --2- 172.21.15.52:0/1635833554 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5648762f3970 0x564876304d10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:30:07.658 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:30:07.663+0000 7fbc9ad8b700 1 -- 172.21.15.52:0/1635833554 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5648763749a0 con 0x5648762f5d20 2024-05-27T07:30:07.658 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:30:07.664+0000 7fbc99d89700 1 --2- 172.21.15.52:0/1635833554 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5648762f3970 0x564876304d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-27T07:30:07.658 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:30:07.664+0000 7fbc9ad8b700 1 --2- 172.21.15.52:0/1635833554 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x5648762f5d20 0x564876374460 secure :-1 s=READY pgs=635 cs=0 l=1 rev1=1 crypto rx=0x7fbc8c057bc0 tx=0x7fbc8c007810 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a6677af00dd51141 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:30:07.658 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:30:07.664+0000 7fbc7dffb700 10 client.0 ms_handle_connect on v2:172.21.15.52:3300/0 2024-05-27T07:30:07.658 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:30:07.664+0000 7fbc7dffb700 1 -- 172.21.15.52:0/1635833554 <== mon.0 v2:172.21.15.52:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbc8c035070 con 0x5648762f5d20 2024-05-27T07:30:07.658 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:30:07.664+0000 7fbc7dffb700 1 -- 172.21.15.52:0/1635833554 <== mon.0 v2:172.21.15.52:3300/0 2 ==== osd_map(111..111 src has 1..111) ==== 9379+0+0 (secure 0 0 0) 0x7fbc8c004a30 con 0x5648762f5d20 2024-05-27T07:30:07.659 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:30:07.664+0000 7fbcab203b80 1 -- 172.21.15.52:0/1635833554 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fbc9002b070 con 0x5648762f5d20 2024-05-27T07:30:07.659 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:30:07.665+0000 7fbc7dffb700 1 -- 172.21.15.52:0/1635833554 <== mon.0 v2:172.21.15.52:3300/0 3 ==== mon_get_version_reply(handle=1 version=76) ==== 24+0+0 (secure 0 0 0) 0x7fbc8c067050 con 0x5648762f5d20 2024-05-27T07:30:07.659 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:30:07.665+0000 7fbcab203b80 10 client.16461 fetch_fsmap learned FSMap version 76 2024-05-27T07:30:07.659 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:30:07.665+0000 7fbcab203b80 1 -- 172.21.15.52:0/1635833554 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({config=0+,fsmap=76,monmap=4+}) -- 0x564876375aa0 con 0x5648762f5d20 2024-05-27T07:30:07.659 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:30:07.665+0000 7fbc7dffb700 1 -- 172.21.15.52:0/1635833554 <== mon.0 v2:172.21.15.52:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fbc8c049070 con 0x5648762f5d20 2024-05-27T07:30:07.660 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:30:07.665+0000 7fbc7dffb700 1 -- 172.21.15.52:0/1635833554 <== mon.0 v2:172.21.15.52:3300/0 5 ==== fsmap(e 76) ==== 5982+0+0 (secure 0 0 0) 0x7fbc8c03b040 con 0x5648762f5d20 2024-05-27T07:30:07.660 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:30:07.665+0000 7fbcab203b80 10 client.16461 fetch_fsmap finished waiting for FSMap version 76 2024-05-27T07:30:07.660 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:30:07.665+0000 7fbcab203b80 10 client.16461 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-05-27T07:30:07.660 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:30:07.665+0000 7fbcab203b80 20 client.16461 populate_metadata read hostname 'smithi052' 2024-05-27T07:30:07.660 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:30:07.665+0000 7fbcab203b80 1 --2- 172.21.15.52:0/1635833554 >> [v2:172.21.15.105:6838/500348151,v1:172.21.15.105:6839/500348151] conn(0x564876376180 0x5648763785a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:30:07.660 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:30:07.665+0000 7fbcab203b80 4 client.16461 mds_command: new command op to 24527 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-05-27T07:30:07.660 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:30:07.665+0000 7fbcab203b80 1 -- 172.21.15.52:0/1635833554 --> [v2:172.21.15.105:6838/500348151,v1:172.21.15.105:6839/500348151] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) -- 0x7fbc9002b070 con 0x564876376180 2024-05-27T07:30:07.660 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:30:07.666+0000 7fbc9a58a700 1 --2- 172.21.15.52:0/1635833554 >> [v2:172.21.15.105:6838/500348151,v1:172.21.15.105:6839/500348151] conn(0x564876376180 0x5648763785a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:30:07.660 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:30:07.666+0000 7fbc7dffb700 10 client.16461 ms_handle_connect on v2:172.21.15.105:6838/500348151 2024-05-27T07:30:07.660 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:30:07.666+0000 7fbc9a58a700 1 --2- 172.21.15.52:0/1635833554 >> [v2:172.21.15.105:6838/500348151,v1:172.21.15.105:6839/500348151] conn(0x564876376180 0x5648763785a0 crc :-1 s=READY pgs=104 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=55e5e3c07800b5da server_cookie=1dc4301e7f55e480 in_seq=0 out_seq=0 2024-05-27T07:30:07.752 INFO:journalctl@ceph.mds.k.smithi105.stdout:May 27 07:30:07 smithi105 podman[70110]: 2024-05-27 07:30:07.388612943 +0000 UTC m=+0.018936641 image pull e36ea4fc8a5807e8d2701370e350a0de3856538797f7fba64cfce7e344bd6799 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6 2024-05-27T07:30:07.861 INFO:journalctl@ceph.mds.a.smithi052.stdout:May 27 07:30:07 smithi052 podman[107170]: 2024-05-27 07:30:07.64572598 +0000 UTC m=+0.025520275 image pull e36ea4fc8a5807e8d2701370e350a0de3856538797f7fba64cfce7e344bd6799 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6 2024-05-27T07:30:07.861 INFO:journalctl@ceph.mds.a.smithi052.stdout:May 27 07:30:07 smithi052 podman[107170]: 2024-05-27 07:30:07.782640913 +0000 UTC m=+0.162435202 container create e5f4c04c08e94f426d7470f4fb1ace3e1702e3663fcdc999a50fd8cdce849ac9 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-a, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, org.label-schema.build-date=20240521, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, GIT_CLEAN=True, ceph=True) 2024-05-27T07:30:07.864 INFO:teuthology.orchestra.run.smithi052.stdout:{ 2024-05-27T07:30:07.865 INFO:teuthology.orchestra.run.smithi052.stdout: "return_code": 0, 2024-05-27T07:30:07.865 INFO:teuthology.orchestra.run.smithi052.stdout: "scrub_tag": "8ab50389-7f33-4519-a5fc-243c0e6b3763", 2024-05-27T07:30:07.865 INFO:teuthology.orchestra.run.smithi052.stdout: "mode": "asynchronous" 2024-05-27T07:30:07.865 INFO:teuthology.orchestra.run.smithi052.stdout:} 2024-05-27T07:30:07.865 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:30:07.866+0000 7fbc7dffb700 1 -- 172.21.15.52:0/1635833554 <== mds.0 v2:172.21.15.105:6838/500348151 1 ==== command_reply(tid 0: 0 ) ==== 8+0+110 (crc 0 0 0) 0x7fbc9002b070 con 0x564876376180 2024-05-27T07:30:07.865 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:30:07.867+0000 7fbc7dffb700 10 client.16461 handle_command_reply: tid=0 2024-05-27T07:30:07.865 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:30:07.867+0000 7fbcab203b80 1 client.16461 shutdown 2024-05-27T07:30:07.865 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:30:07.867+0000 7fbcab203b80 1 -- 172.21.15.52:0/1635833554 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x5648762f5d20 msgr2=0x564876374460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:30:07.866 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:30:07.867+0000 7fbcab203b80 1 --2- 172.21.15.52:0/1635833554 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x5648762f5d20 0x564876374460 secure :-1 s=READY pgs=635 cs=0 l=1 rev1=1 crypto rx=0x7fbc8c057bc0 tx=0x7fbc8c007810 comp rx=0 tx=0).stop 2024-05-27T07:30:07.866 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:30:07.868+0000 7fbcab203b80 1 -- 172.21.15.52:0/1635833554 shutdown_connections 2024-05-27T07:30:07.866 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:30:07.868+0000 7fbcab203b80 1 --2- 172.21.15.52:0/1635833554 >> [v2:172.21.15.105:6838/500348151,v1:172.21.15.105:6839/500348151] conn(0x564876376180 0x5648763785a0 crc :-1 s=READY pgs=104 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:30:07.866 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:30:07.868+0000 7fbcab203b80 1 --2- 172.21.15.52:0/1635833554 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x5648762f5d20 0x564876374460 unknown :-1 s=CLOSED pgs=635 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:30:07.866 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:30:07.868+0000 7fbc7dffb700 0 client.16461 ms_handle_reset on v2:172.21.15.105:6838/500348151 2024-05-27T07:30:07.866 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:30:07.868+0000 7fbcab203b80 1 --2- 172.21.15.52:0/1635833554 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x564876172ec0 0x5648763047d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:30:07.866 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:30:07.868+0000 7fbcab203b80 1 --2- 172.21.15.52:0/1635833554 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5648762f3970 0x564876304d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:30:07.866 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:30:07.868+0000 7fbcab203b80 1 -- 172.21.15.52:0/1635833554 >> 172.21.15.52:0/1635833554 conn(0x5648761cdb70 msgr2=0x5648762f9ff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:30:07.866 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:30:07.868+0000 7fbcab203b80 1 -- 172.21.15.52:0/1635833554 shutdown_connections 2024-05-27T07:30:07.867 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:30:07.868+0000 7fbcab203b80 1 -- 172.21.15.52:0/1635833554 wait complete. 2024-05-27T07:30:07.867 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:30:07.869+0000 7fbcab203b80 20 client.16461 trim_cache size 0 max 16384 2024-05-27T07:30:07.868 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:30:07.874+0000 7fbc9c7ee700 1 -- 172.21.15.52:0/1626648419 >> [v2:172.21.15.52:6800/1822318384,v1:172.21.15.52:6801/1822318384] conn(0x7fbc800b2af0 msgr2=0x7fbc800b5000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:30:07.868 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:30:07.874+0000 7fbc9c7ee700 1 --2- 172.21.15.52:0/1626648419 >> [v2:172.21.15.52:6800/1822318384,v1:172.21.15.52:6801/1822318384] conn(0x7fbc800b2af0 0x7fbc800b5000 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7fbc8c043000 tx=0x7fbc8c0446a0 comp rx=0 tx=0).stop 2024-05-27T07:30:07.868 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:30:07.874+0000 7fbc9c7ee700 1 -- 172.21.15.52:0/1626648419 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fbc94107e00 msgr2=0x7fbc9419d630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:30:07.868 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:30:07.874+0000 7fbc9c7ee700 1 --2- 172.21.15.52:0/1626648419 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fbc94107e00 0x7fbc9419d630 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7fbc8400e9c0 tx=0x7fbc8400ecd0 comp rx=0 tx=0).stop 2024-05-27T07:30:07.869 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:30:07.875+0000 7fbc9c7ee700 1 -- 172.21.15.52:0/1626648419 shutdown_connections 2024-05-27T07:30:07.869 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:30:07.876+0000 7fbc9c7ee700 1 --2- 172.21.15.52:0/1626648419 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fbc94108750 0x7fbc941a18c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:30:07.869 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:30:07.876+0000 7fbc9c7ee700 1 --2- 172.21.15.52:0/1626648419 >> [v2:172.21.15.52:6800/1822318384,v1:172.21.15.52:6801/1822318384] conn(0x7fbc800b2af0 0x7fbc800b5000 unknown :-1 s=CLOSED pgs=130 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:30:07.869 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:30:07.876+0000 7fbc9c7ee700 1 --2- 172.21.15.52:0/1626648419 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fbc941034d0 0x7fbc9419d090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:30:07.869 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:30:07.876+0000 7fbc9c7ee700 1 --2- 172.21.15.52:0/1626648419 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fbc94107e00 0x7fbc9419d630 unknown :-1 s=CLOSED pgs=132 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:30:07.869 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:30:07.876+0000 7fbc9c7ee700 1 -- 172.21.15.52:0/1626648419 >> 172.21.15.52:0/1626648419 conn(0x7fbc940fef20 msgr2=0x7fbc941065b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:30:07.870 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:30:07.876+0000 7fbc9c7ee700 1 -- 172.21.15.52:0/1626648419 shutdown_connections 2024-05-27T07:30:07.870 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:30:07.876+0000 7fbc9c7ee700 1 -- 172.21.15.52:0/1626648419 wait complete. 2024-05-27T07:30:08.190 INFO:journalctl@ceph.mds.k.smithi105.stdout:May 27 07:30:07 smithi105 podman[70110]: 2024-05-27 07:30:07.7623515 +0000 UTC m=+0.392675187 container create ef69d02153ef7bfcd8563b2cac6fc29312888b9219ed7f50788b3fe56d33d46f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-k, org.label-schema.build-date=20240521, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, io.buildah.version=1.33.5) 2024-05-27T07:30:08.657 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:30:08 smithi105 ceph-mon[35977]: daemon mds.k restarted 2024-05-27T07:30:08.658 INFO:journalctl@ceph.mds.k.smithi105.stdout:May 27 07:30:08 smithi105 podman[70110]: 2024-05-27 07:30:08.382992523 +0000 UTC m=+1.013316218 container init ef69d02153ef7bfcd8563b2cac6fc29312888b9219ed7f50788b3fe56d33d46f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-k, org.label-schema.build-date=20240521, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, ceph=True, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2) 2024-05-27T07:30:08.658 INFO:journalctl@ceph.mds.k.smithi105.stdout:May 27 07:30:08 smithi105 podman[70110]: 2024-05-27 07:30:08.385773909 +0000 UTC m=+1.016097604 container start ef69d02153ef7bfcd8563b2cac6fc29312888b9219ed7f50788b3fe56d33d46f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-k, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, GIT_CLEAN=True, ceph=True, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240521, GIT_BRANCH=HEAD) 2024-05-27T07:30:08.658 INFO:journalctl@ceph.mds.k.smithi105.stdout:May 27 07:30:08 smithi105 ceph-mds[70128]: -- 172.21.15.105:0/1549008951 <== mon.2 v2:172.21.15.105:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x565466905a40 con 0x56546694da80 2024-05-27T07:30:08.658 INFO:journalctl@ceph.mds.k.smithi105.stdout:May 27 07:30:08 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-k[70124]: starting mds.k at 2024-05-27T07:30:08.658 INFO:journalctl@ceph.mds.k.smithi105.stdout:May 27 07:30:08 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-k[70124]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: In function 'std::set SnapClient::filter(const std::set&) const' thread 7f847dd5f700 time 2024-05-27T07:30:08.550288+0000 2024-05-27T07:30:08.658 INFO:journalctl@ceph.mds.k.smithi105.stdout:May 27 07:30:08 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-k[70124]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: 197: FAILED ceph_assert(cached_version > 0) 2024-05-27T07:30:08.658 INFO:journalctl@ceph.mds.k.smithi105.stdout:May 27 07:30:08 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-k[70124]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:30:08.658 INFO:journalctl@ceph.mds.k.smithi105.stdout:May 27 07:30:08 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-k[70124]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x127) [0x7f84905a4ba6] 2024-05-27T07:30:08.658 INFO:journalctl@ceph.mds.k.smithi105.stdout:May 27 07:30:08 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-k[70124]: 2: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7f84905a4dd1] 2024-05-27T07:30:08.658 INFO:journalctl@ceph.mds.k.smithi105.stdout:May 27 07:30:08 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-k[70124]: 3: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x56546478423d] 2024-05-27T07:30:08.659 INFO:journalctl@ceph.mds.k.smithi105.stdout:May 27 07:30:08 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-k[70124]: 4: (SnapRealm::build_snap_set() const+0x323) [0x56546476f971] 2024-05-27T07:30:08.659 INFO:journalctl@ceph.mds.k.smithi105.stdout:May 27 07:30:08 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-k[70124]: 5: (SnapRealm::check_cache() const+0x13e) [0x56546476dfa0] 2024-05-27T07:30:08.659 INFO:journalctl@ceph.mds.k.smithi105.stdout:May 27 07:30:08 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-k[70124]: 6: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x56546476e651] 2024-05-27T07:30:08.659 INFO:journalctl@ceph.mds.k.smithi105.stdout:May 27 07:30:08 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-k[70124]: 7: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x565464703347] 2024-05-27T07:30:08.659 INFO:journalctl@ceph.mds.k.smithi105.stdout:May 27 07:30:08 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-k[70124]: 8: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x5654643ff87e] 2024-05-27T07:30:08.659 INFO:journalctl@ceph.mds.k.smithi105.stdout:May 27 07:30:08 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-k[70124]: 9: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x279d) [0x565464404f2f] 2024-05-27T07:30:08.659 INFO:journalctl@ceph.mds.k.smithi105.stdout:May 27 07:30:08 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-k[70124]: 10: (EUpdate::replay(MDSRank*)+0x63) [0x56546440f969] 2024-05-27T07:30:08.659 INFO:journalctl@ceph.mds.k.smithi105.stdout:May 27 07:30:08 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-k[70124]: 11: (MDLog::_replay_thread()+0x1f38) [0x5654647a848c] 2024-05-27T07:30:08.659 INFO:journalctl@ceph.mds.k.smithi105.stdout:May 27 07:30:08 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-k[70124]: 12: (MDLog::ReplayThread::entry()+0x11) [0x5654647b19cb] 2024-05-27T07:30:08.660 INFO:journalctl@ceph.mds.k.smithi105.stdout:May 27 07:30:08 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-k[70124]: 13: (Thread::entry_wrapper()+0x43) [0x7f849057372d] 2024-05-27T07:30:08.660 INFO:journalctl@ceph.mds.k.smithi105.stdout:May 27 07:30:08 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-k[70124]: 14: (Thread::_entry_func(void*)+0xd) [0x7f8490573749] 2024-05-27T07:30:08.660 INFO:journalctl@ceph.mds.k.smithi105.stdout:May 27 07:30:08 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-k[70124]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7f848f1171ca] 2024-05-27T07:30:08.660 INFO:journalctl@ceph.mds.k.smithi105.stdout:May 27 07:30:08 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-k[70124]: 16: clone() 2024-05-27T07:30:08.660 INFO:journalctl@ceph.mds.k.smithi105.stdout:May 27 07:30:08 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-k[70124]: *** Caught signal (Aborted) ** 2024-05-27T07:30:08.660 INFO:journalctl@ceph.mds.k.smithi105.stdout:May 27 07:30:08 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-k[70124]: in thread 7f847dd5f700 thread_name:md_log_replay 2024-05-27T07:30:08.660 INFO:journalctl@ceph.mds.k.smithi105.stdout:May 27 07:30:08 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-k[70124]: 2024-05-27T07:30:08.551+0000 7f847dd5f700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: In function 'std::set SnapClient::filter(const std::set&) const' thread 7f847dd5f700 time 2024-05-27T07:30:08.550288+0000 2024-05-27T07:30:08.660 INFO:journalctl@ceph.mds.k.smithi105.stdout:May 27 07:30:08 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-k[70124]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: 197: FAILED ceph_assert(cached_version > 0) 2024-05-27T07:30:08.660 INFO:journalctl@ceph.mds.k.smithi105.stdout:May 27 07:30:08 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-k[70124]: 2024-05-27T07:30:08.660 INFO:journalctl@ceph.mds.k.smithi105.stdout:May 27 07:30:08 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-k[70124]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:30:08.660 INFO:journalctl@ceph.mds.k.smithi105.stdout:May 27 07:30:08 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-k[70124]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x127) [0x7f84905a4ba6] 2024-05-27T07:30:08.660 INFO:journalctl@ceph.mds.k.smithi105.stdout:May 27 07:30:08 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-k[70124]: 2: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7f84905a4dd1] 2024-05-27T07:30:08.660 INFO:journalctl@ceph.mds.k.smithi105.stdout:May 27 07:30:08 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-k[70124]: 3: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x56546478423d] 2024-05-27T07:30:08.660 INFO:journalctl@ceph.mds.k.smithi105.stdout:May 27 07:30:08 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-k[70124]: 4: (SnapRealm::build_snap_set() const+0x323) [0x56546476f971] 2024-05-27T07:30:08.661 INFO:journalctl@ceph.mds.k.smithi105.stdout:May 27 07:30:08 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-k[70124]: 5: (SnapRealm::check_cache() const+0x13e) [0x56546476dfa0] 2024-05-27T07:30:08.661 INFO:journalctl@ceph.mds.k.smithi105.stdout:May 27 07:30:08 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-k[70124]: 6: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x56546476e651] 2024-05-27T07:30:08.661 INFO:journalctl@ceph.mds.k.smithi105.stdout:May 27 07:30:08 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-k[70124]: 7: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x565464703347] 2024-05-27T07:30:08.661 INFO:journalctl@ceph.mds.k.smithi105.stdout:May 27 07:30:08 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-k[70124]: 8: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x5654643ff87e] 2024-05-27T07:30:08.661 INFO:journalctl@ceph.mds.k.smithi105.stdout:May 27 07:30:08 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-k[70124]: 9: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x279d) [0x565464404f2f] 2024-05-27T07:30:08.661 INFO:journalctl@ceph.mds.k.smithi105.stdout:May 27 07:30:08 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-k[70124]: 10: (EUpdate::replay(MDSRank*)+0x63) [0x56546440f969] 2024-05-27T07:30:08.661 INFO:journalctl@ceph.mds.k.smithi105.stdout:May 27 07:30:08 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-k[70124]: 11: (MDLog::_replay_thread()+0x1f38) [0x5654647a848c] 2024-05-27T07:30:08.662 INFO:journalctl@ceph.mds.k.smithi105.stdout:May 27 07:30:08 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-k[70124]: 12: (MDLog::ReplayThread::entry()+0x11) [0x5654647b19cb] 2024-05-27T07:30:08.662 INFO:journalctl@ceph.mds.k.smithi105.stdout:May 27 07:30:08 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-k[70124]: 13: (Thread::entry_wrapper()+0x43) [0x7f849057372d] 2024-05-27T07:30:08.662 INFO:journalctl@ceph.mds.k.smithi105.stdout:May 27 07:30:08 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-k[70124]: 14: (Thread::_entry_func(void*)+0xd) [0x7f8490573749] 2024-05-27T07:30:08.662 INFO:journalctl@ceph.mds.k.smithi105.stdout:May 27 07:30:08 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-k[70124]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7f848f1171ca] 2024-05-27T07:30:08.662 INFO:journalctl@ceph.mds.k.smithi105.stdout:May 27 07:30:08 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-k[70124]: 16: clone() 2024-05-27T07:30:08.662 INFO:journalctl@ceph.mds.k.smithi105.stdout:May 27 07:30:08 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-k[70124]: 2024-05-27T07:30:08.662 INFO:journalctl@ceph.mds.k.smithi105.stdout:May 27 07:30:08 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-k[70124]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:30:08.662 INFO:journalctl@ceph.mds.k.smithi105.stdout:May 27 07:30:08 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-k[70124]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f848f121d20] 2024-05-27T07:30:08.662 INFO:journalctl@ceph.mds.k.smithi105.stdout:May 27 07:30:08 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-k[70124]: 2: gsignal() 2024-05-27T07:30:08.663 INFO:journalctl@ceph.mds.k.smithi105.stdout:May 27 07:30:08 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-k[70124]: 3: abort() 2024-05-27T07:30:08.663 INFO:journalctl@ceph.mds.k.smithi105.stdout:May 27 07:30:08 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-k[70124]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x26e) [0x7f84905a4ced] 2024-05-27T07:30:08.663 INFO:journalctl@ceph.mds.k.smithi105.stdout:May 27 07:30:08 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-k[70124]: 5: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7f84905a4dd1] 2024-05-27T07:30:08.663 INFO:journalctl@ceph.mds.k.smithi105.stdout:May 27 07:30:08 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-k[70124]: 6: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x56546478423d] 2024-05-27T07:30:08.663 INFO:journalctl@ceph.mds.k.smithi105.stdout:May 27 07:30:08 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-k[70124]: 7: (SnapRealm::build_snap_set() const+0x323) [0x56546476f971] 2024-05-27T07:30:08.663 INFO:journalctl@ceph.mds.k.smithi105.stdout:May 27 07:30:08 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-k[70124]: 8: (SnapRealm::check_cache() const+0x13e) [0x56546476dfa0] 2024-05-27T07:30:08.663 INFO:journalctl@ceph.mds.k.smithi105.stdout:May 27 07:30:08 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-k[70124]: 9: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x56546476e651] 2024-05-27T07:30:08.663 INFO:journalctl@ceph.mds.k.smithi105.stdout:May 27 07:30:08 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-k[70124]: 10: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x565464703347] 2024-05-27T07:30:08.663 INFO:journalctl@ceph.mds.k.smithi105.stdout:May 27 07:30:08 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-k[70124]: 11: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x5654643ff87e] 2024-05-27T07:30:08.663 INFO:journalctl@ceph.mds.k.smithi105.stdout:May 27 07:30:08 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-k[70124]: 12: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x279d) [0x565464404f2f] 2024-05-27T07:30:08.663 INFO:journalctl@ceph.mds.k.smithi105.stdout:May 27 07:30:08 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-k[70124]: 13: (EUpdate::replay(MDSRank*)+0x63) [0x56546440f969] 2024-05-27T07:30:08.663 INFO:journalctl@ceph.mds.k.smithi105.stdout:May 27 07:30:08 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-k[70124]: 14: (MDLog::_replay_thread()+0x1f38) [0x5654647a848c] 2024-05-27T07:30:08.664 INFO:journalctl@ceph.mds.k.smithi105.stdout:May 27 07:30:08 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-k[70124]: 15: (MDLog::ReplayThread::entry()+0x11) [0x5654647b19cb] 2024-05-27T07:30:08.664 INFO:journalctl@ceph.mds.k.smithi105.stdout:May 27 07:30:08 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-k[70124]: 16: (Thread::entry_wrapper()+0x43) [0x7f849057372d] 2024-05-27T07:30:08.664 INFO:journalctl@ceph.mds.k.smithi105.stdout:May 27 07:30:08 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-k[70124]: 17: (Thread::_entry_func(void*)+0xd) [0x7f8490573749] 2024-05-27T07:30:08.664 INFO:journalctl@ceph.mds.k.smithi105.stdout:May 27 07:30:08 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-k[70124]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7f848f1171ca] 2024-05-27T07:30:08.664 INFO:journalctl@ceph.mds.k.smithi105.stdout:May 27 07:30:08 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-k[70124]: 19: clone() 2024-05-27T07:30:08.664 INFO:journalctl@ceph.mds.k.smithi105.stdout:May 27 07:30:08 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-k[70124]: 2024-05-27T07:30:08.553+0000 7f847dd5f700 -1 *** Caught signal (Aborted) ** 2024-05-27T07:30:08.664 INFO:journalctl@ceph.mds.k.smithi105.stdout:May 27 07:30:08 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-k[70124]: in thread 7f847dd5f700 thread_name:md_log_replay 2024-05-27T07:30:08.664 INFO:journalctl@ceph.mds.k.smithi105.stdout:May 27 07:30:08 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-k[70124]: 2024-05-27T07:30:08.664 INFO:journalctl@ceph.mds.k.smithi105.stdout:May 27 07:30:08 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-k[70124]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:30:08.664 INFO:journalctl@ceph.mds.k.smithi105.stdout:May 27 07:30:08 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-k[70124]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f848f121d20] 2024-05-27T07:30:08.664 INFO:journalctl@ceph.mds.k.smithi105.stdout:May 27 07:30:08 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-k[70124]: 2: gsignal() 2024-05-27T07:30:08.664 INFO:journalctl@ceph.mds.k.smithi105.stdout:May 27 07:30:08 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-k[70124]: 3: abort() 2024-05-27T07:30:08.664 INFO:journalctl@ceph.mds.k.smithi105.stdout:May 27 07:30:08 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-k[70124]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x26e) [0x7f84905a4ced] 2024-05-27T07:30:08.665 INFO:journalctl@ceph.mds.k.smithi105.stdout:May 27 07:30:08 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-k[70124]: 5: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7f84905a4dd1] 2024-05-27T07:30:08.665 INFO:journalctl@ceph.mds.k.smithi105.stdout:May 27 07:30:08 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-k[70124]: 6: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x56546478423d] 2024-05-27T07:30:08.665 INFO:journalctl@ceph.mds.k.smithi105.stdout:May 27 07:30:08 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-k[70124]: 7: (SnapRealm::build_snap_set() const+0x323) [0x56546476f971] 2024-05-27T07:30:08.665 INFO:journalctl@ceph.mds.k.smithi105.stdout:May 27 07:30:08 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-k[70124]: 8: (SnapRealm::check_cache() const+0x13e) [0x56546476dfa0] 2024-05-27T07:30:08.665 INFO:journalctl@ceph.mds.k.smithi105.stdout:May 27 07:30:08 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-k[70124]: 9: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x56546476e651] 2024-05-27T07:30:08.665 INFO:journalctl@ceph.mds.k.smithi105.stdout:May 27 07:30:08 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-k[70124]: 10: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x565464703347] 2024-05-27T07:30:08.665 INFO:journalctl@ceph.mds.k.smithi105.stdout:May 27 07:30:08 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-k[70124]: 11: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x5654643ff87e] 2024-05-27T07:30:08.665 INFO:journalctl@ceph.mds.k.smithi105.stdout:May 27 07:30:08 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-k[70124]: 12: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x279d) [0x565464404f2f] 2024-05-27T07:30:08.665 INFO:journalctl@ceph.mds.k.smithi105.stdout:May 27 07:30:08 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-k[70124]: 13: (EUpdate::replay(MDSRank*)+0x63) [0x56546440f969] 2024-05-27T07:30:08.665 INFO:journalctl@ceph.mds.k.smithi105.stdout:May 27 07:30:08 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-k[70124]: 14: (MDLog::_replay_thread()+0x1f38) [0x5654647a848c] 2024-05-27T07:30:08.665 INFO:journalctl@ceph.mds.k.smithi105.stdout:May 27 07:30:08 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-k[70124]: 15: (MDLog::ReplayThread::entry()+0x11) [0x5654647b19cb] 2024-05-27T07:30:08.665 INFO:journalctl@ceph.mds.k.smithi105.stdout:May 27 07:30:08 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-k[70124]: 16: (Thread::entry_wrapper()+0x43) [0x7f849057372d] 2024-05-27T07:30:08.665 INFO:journalctl@ceph.mds.k.smithi105.stdout:May 27 07:30:08 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-k[70124]: 17: (Thread::_entry_func(void*)+0xd) [0x7f8490573749] 2024-05-27T07:30:08.666 INFO:journalctl@ceph.mds.k.smithi105.stdout:May 27 07:30:08 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-k[70124]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7f848f1171ca] 2024-05-27T07:30:08.666 INFO:journalctl@ceph.mds.k.smithi105.stdout:May 27 07:30:08 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-k[70124]: 19: clone() 2024-05-27T07:30:08.666 INFO:journalctl@ceph.mds.k.smithi105.stdout:May 27 07:30:08 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-k[70124]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-05-27T07:30:08.666 INFO:journalctl@ceph.mds.k.smithi105.stdout:May 27 07:30:08 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-k[70124]: 2024-05-27T07:30:08.666 INFO:journalctl@ceph.mds.k.smithi105.stdout:May 27 07:30:08 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-k[70124]: -1> 2024-05-27T07:30:08.551+0000 7f847dd5f700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: In function 'std::set SnapClient::filter(const std::set&) const' thread 7f847dd5f700 time 2024-05-27T07:30:08.550288+0000 2024-05-27T07:30:08.667 INFO:journalctl@ceph.mds.k.smithi105.stdout:May 27 07:30:08 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-k[70124]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: 197: FAILED ceph_assert(cached_version > 0) 2024-05-27T07:30:08.667 INFO:journalctl@ceph.mds.k.smithi105.stdout:May 27 07:30:08 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-k[70124]: 2024-05-27T07:30:08.667 INFO:journalctl@ceph.mds.k.smithi105.stdout:May 27 07:30:08 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-k[70124]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:30:08.667 INFO:journalctl@ceph.mds.k.smithi105.stdout:May 27 07:30:08 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-k[70124]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x127) [0x7f84905a4ba6] 2024-05-27T07:30:08.667 INFO:journalctl@ceph.mds.k.smithi105.stdout:May 27 07:30:08 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-k[70124]: 2: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7f84905a4dd1] 2024-05-27T07:30:08.667 INFO:journalctl@ceph.mds.k.smithi105.stdout:May 27 07:30:08 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-k[70124]: 3: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x56546478423d] 2024-05-27T07:30:08.667 INFO:journalctl@ceph.mds.k.smithi105.stdout:May 27 07:30:08 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-k[70124]: 4: (SnapRealm::build_snap_set() const+0x323) [0x56546476f971] 2024-05-27T07:30:08.667 INFO:journalctl@ceph.mds.k.smithi105.stdout:May 27 07:30:08 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-k[70124]: 5: (SnapRealm::check_cache() const+0x13e) [0x56546476dfa0] 2024-05-27T07:30:08.667 INFO:journalctl@ceph.mds.k.smithi105.stdout:May 27 07:30:08 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-k[70124]: 6: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x56546476e651] 2024-05-27T07:30:08.667 INFO:journalctl@ceph.mds.k.smithi105.stdout:May 27 07:30:08 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-k[70124]: 7: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x565464703347] 2024-05-27T07:30:08.667 INFO:journalctl@ceph.mds.k.smithi105.stdout:May 27 07:30:08 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-k[70124]: 8: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x5654643ff87e] 2024-05-27T07:30:08.668 INFO:journalctl@ceph.mds.k.smithi105.stdout:May 27 07:30:08 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-k[70124]: 9: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x279d) [0x565464404f2f] 2024-05-27T07:30:08.668 INFO:journalctl@ceph.mds.k.smithi105.stdout:May 27 07:30:08 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-k[70124]: 10: (EUpdate::replay(MDSRank*)+0x63) [0x56546440f969] 2024-05-27T07:30:08.668 INFO:journalctl@ceph.mds.k.smithi105.stdout:May 27 07:30:08 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-k[70124]: 11: (MDLog::_replay_thread()+0x1f38) [0x5654647a848c] 2024-05-27T07:30:08.668 INFO:journalctl@ceph.mds.k.smithi105.stdout:May 27 07:30:08 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-k[70124]: 12: (MDLog::ReplayThread::entry()+0x11) [0x5654647b19cb] 2024-05-27T07:30:08.668 INFO:journalctl@ceph.mds.k.smithi105.stdout:May 27 07:30:08 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-k[70124]: 13: (Thread::entry_wrapper()+0x43) [0x7f849057372d] 2024-05-27T07:30:08.668 INFO:journalctl@ceph.mds.k.smithi105.stdout:May 27 07:30:08 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-k[70124]: 14: (Thread::_entry_func(void*)+0xd) [0x7f8490573749] 2024-05-27T07:30:08.668 INFO:journalctl@ceph.mds.k.smithi105.stdout:May 27 07:30:08 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-k[70124]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7f848f1171ca] 2024-05-27T07:30:08.668 INFO:journalctl@ceph.mds.k.smithi105.stdout:May 27 07:30:08 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-k[70124]: 16: clone() 2024-05-27T07:30:08.668 INFO:journalctl@ceph.mds.k.smithi105.stdout:May 27 07:30:08 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-k[70124]: 2024-05-27T07:30:08.668 INFO:journalctl@ceph.mds.k.smithi105.stdout:May 27 07:30:08 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-k[70124]: 0> 2024-05-27T07:30:08.553+0000 7f847dd5f700 -1 *** Caught signal (Aborted) ** 2024-05-27T07:30:08.668 INFO:journalctl@ceph.mds.k.smithi105.stdout:May 27 07:30:08 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-k[70124]: in thread 7f847dd5f700 thread_name:md_log_replay 2024-05-27T07:30:08.668 INFO:journalctl@ceph.mds.k.smithi105.stdout:May 27 07:30:08 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-k[70124]: 2024-05-27T07:30:08.668 INFO:journalctl@ceph.mds.k.smithi105.stdout:May 27 07:30:08 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-k[70124]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:30:08.669 INFO:journalctl@ceph.mds.k.smithi105.stdout:May 27 07:30:08 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-k[70124]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f848f121d20] 2024-05-27T07:30:08.669 INFO:journalctl@ceph.mds.k.smithi105.stdout:May 27 07:30:08 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-k[70124]: 2: gsignal() 2024-05-27T07:30:08.669 INFO:journalctl@ceph.mds.k.smithi105.stdout:May 27 07:30:08 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-k[70124]: 3: abort() 2024-05-27T07:30:08.669 INFO:journalctl@ceph.mds.k.smithi105.stdout:May 27 07:30:08 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-k[70124]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x26e) [0x7f84905a4ced] 2024-05-27T07:30:08.669 INFO:journalctl@ceph.mds.k.smithi105.stdout:May 27 07:30:08 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-k[70124]: 5: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7f84905a4dd1] 2024-05-27T07:30:08.669 INFO:journalctl@ceph.mds.k.smithi105.stdout:May 27 07:30:08 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-k[70124]: 6: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x56546478423d] 2024-05-27T07:30:08.669 INFO:journalctl@ceph.mds.k.smithi105.stdout:May 27 07:30:08 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-k[70124]: 7: (SnapRealm::build_snap_set() const+0x323) [0x56546476f971] 2024-05-27T07:30:08.669 INFO:journalctl@ceph.mds.k.smithi105.stdout:May 27 07:30:08 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-k[70124]: 8: (SnapRealm::check_cache() const+0x13e) [0x56546476dfa0] 2024-05-27T07:30:08.669 INFO:journalctl@ceph.mds.k.smithi105.stdout:May 27 07:30:08 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-k[70124]: 9: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x56546476e651] 2024-05-27T07:30:08.669 INFO:journalctl@ceph.mds.k.smithi105.stdout:May 27 07:30:08 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-k[70124]: 10: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x565464703347] 2024-05-27T07:30:08.669 INFO:journalctl@ceph.mds.k.smithi105.stdout:May 27 07:30:08 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-k[70124]: 11: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x5654643ff87e] 2024-05-27T07:30:08.669 INFO:journalctl@ceph.mds.k.smithi105.stdout:May 27 07:30:08 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-k[70124]: 12: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x279d) [0x565464404f2f] 2024-05-27T07:30:08.669 INFO:journalctl@ceph.mds.k.smithi105.stdout:May 27 07:30:08 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-k[70124]: 13: (EUpdate::replay(MDSRank*)+0x63) [0x56546440f969] 2024-05-27T07:30:08.670 INFO:journalctl@ceph.mds.k.smithi105.stdout:May 27 07:30:08 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-k[70124]: 14: (MDLog::_replay_thread()+0x1f38) [0x5654647a848c] 2024-05-27T07:30:08.670 INFO:journalctl@ceph.mds.k.smithi105.stdout:May 27 07:30:08 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-k[70124]: 15: (MDLog::ReplayThread::entry()+0x11) [0x5654647b19cb] 2024-05-27T07:30:08.670 INFO:journalctl@ceph.mds.k.smithi105.stdout:May 27 07:30:08 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-k[70124]: 16: (Thread::entry_wrapper()+0x43) [0x7f849057372d] 2024-05-27T07:30:08.670 INFO:journalctl@ceph.mds.k.smithi105.stdout:May 27 07:30:08 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-k[70124]: 17: (Thread::_entry_func(void*)+0xd) [0x7f8490573749] 2024-05-27T07:30:08.670 INFO:journalctl@ceph.mds.k.smithi105.stdout:May 27 07:30:08 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-k[70124]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7f848f1171ca] 2024-05-27T07:30:08.670 INFO:journalctl@ceph.mds.k.smithi105.stdout:May 27 07:30:08 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-k[70124]: 19: clone() 2024-05-27T07:30:08.670 INFO:journalctl@ceph.mds.k.smithi105.stdout:May 27 07:30:08 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-k[70124]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-05-27T07:30:08.670 INFO:journalctl@ceph.mds.k.smithi105.stdout:May 27 07:30:08 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-k[70124]: 2024-05-27T07:30:08.670 INFO:journalctl@ceph.mds.k.smithi105.stdout:May 27 07:30:08 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-k[70124]: -1> 2024-05-27T07:30:08.551+0000 7f847dd5f700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: In function 'std::set SnapClient::filter(const std::set&) const' thread 7f847dd5f700 time 2024-05-27T07:30:08.550288+0000 2024-05-27T07:30:08.670 INFO:journalctl@ceph.mds.k.smithi105.stdout:May 27 07:30:08 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-k[70124]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: 197: FAILED ceph_assert(cached_version > 0) 2024-05-27T07:30:08.670 INFO:journalctl@ceph.mds.k.smithi105.stdout:May 27 07:30:08 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-k[70124]: 2024-05-27T07:30:08.670 INFO:journalctl@ceph.mds.k.smithi105.stdout:May 27 07:30:08 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-k[70124]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:30:08.670 INFO:journalctl@ceph.mds.k.smithi105.stdout:May 27 07:30:08 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-k[70124]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x127) [0x7f84905a4ba6] 2024-05-27T07:30:08.671 INFO:journalctl@ceph.mds.k.smithi105.stdout:May 27 07:30:08 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-k[70124]: 2: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7f84905a4dd1] 2024-05-27T07:30:08.671 INFO:journalctl@ceph.mds.k.smithi105.stdout:May 27 07:30:08 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-k[70124]: 3: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x56546478423d] 2024-05-27T07:30:08.671 INFO:journalctl@ceph.mds.k.smithi105.stdout:May 27 07:30:08 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-k[70124]: 4: (SnapRealm::build_snap_set() const+0x323) [0x56546476f971] 2024-05-27T07:30:08.671 INFO:journalctl@ceph.mds.k.smithi105.stdout:May 27 07:30:08 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-k[70124]: 5: (SnapRealm::check_cache() const+0x13e) [0x56546476dfa0] 2024-05-27T07:30:08.671 INFO:journalctl@ceph.mds.k.smithi105.stdout:May 27 07:30:08 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-k[70124]: 6: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x56546476e651] 2024-05-27T07:30:08.671 INFO:journalctl@ceph.mds.k.smithi105.stdout:May 27 07:30:08 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-k[70124]: 7: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x565464703347] 2024-05-27T07:30:08.671 INFO:journalctl@ceph.mds.k.smithi105.stdout:May 27 07:30:08 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-k[70124]: 8: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x5654643ff87e] 2024-05-27T07:30:08.671 INFO:journalctl@ceph.mds.k.smithi105.stdout:May 27 07:30:08 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-k[70124]: 9: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x279d) [0x565464404f2f] 2024-05-27T07:30:08.671 INFO:journalctl@ceph.mds.k.smithi105.stdout:May 27 07:30:08 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-k[70124]: 10: (EUpdate::replay(MDSRank*)+0x63) [0x56546440f969] 2024-05-27T07:30:08.671 INFO:journalctl@ceph.mds.k.smithi105.stdout:May 27 07:30:08 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-k[70124]: 11: (MDLog::_replay_thread()+0x1f38) [0x5654647a848c] 2024-05-27T07:30:08.671 INFO:journalctl@ceph.mds.k.smithi105.stdout:May 27 07:30:08 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-k[70124]: 12: (MDLog::ReplayThread::entry()+0x11) [0x5654647b19cb] 2024-05-27T07:30:08.671 INFO:journalctl@ceph.mds.k.smithi105.stdout:May 27 07:30:08 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-k[70124]: 13: (Thread::entry_wrapper()+0x43) [0x7f849057372d] 2024-05-27T07:30:08.672 INFO:journalctl@ceph.mds.k.smithi105.stdout:May 27 07:30:08 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-k[70124]: 14: (Thread::_entry_func(void*)+0xd) [0x7f8490573749] 2024-05-27T07:30:08.672 INFO:journalctl@ceph.mds.k.smithi105.stdout:May 27 07:30:08 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-k[70124]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7f848f1171ca] 2024-05-27T07:30:08.672 INFO:journalctl@ceph.mds.k.smithi105.stdout:May 27 07:30:08 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-k[70124]: 16: clone() 2024-05-27T07:30:08.672 INFO:journalctl@ceph.mds.k.smithi105.stdout:May 27 07:30:08 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-k[70124]: 2024-05-27T07:30:08.672 INFO:journalctl@ceph.mds.k.smithi105.stdout:May 27 07:30:08 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-k[70124]: 0> 2024-05-27T07:30:08.553+0000 7f847dd5f700 -1 *** Caught signal (Aborted) ** 2024-05-27T07:30:08.672 INFO:journalctl@ceph.mds.k.smithi105.stdout:May 27 07:30:08 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-k[70124]: in thread 7f847dd5f700 thread_name:md_log_replay 2024-05-27T07:30:08.672 INFO:journalctl@ceph.mds.k.smithi105.stdout:May 27 07:30:08 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-k[70124]: 2024-05-27T07:30:08.672 INFO:journalctl@ceph.mds.k.smithi105.stdout:May 27 07:30:08 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-k[70124]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:30:08.672 INFO:journalctl@ceph.mds.k.smithi105.stdout:May 27 07:30:08 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-k[70124]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f848f121d20] 2024-05-27T07:30:08.672 INFO:journalctl@ceph.mds.k.smithi105.stdout:May 27 07:30:08 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-k[70124]: 2: gsignal() 2024-05-27T07:30:08.672 INFO:journalctl@ceph.mds.k.smithi105.stdout:May 27 07:30:08 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-k[70124]: 3: abort() 2024-05-27T07:30:08.672 INFO:journalctl@ceph.mds.k.smithi105.stdout:May 27 07:30:08 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-k[70124]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x26e) [0x7f84905a4ced] 2024-05-27T07:30:08.672 INFO:journalctl@ceph.mds.k.smithi105.stdout:May 27 07:30:08 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-k[70124]: 5: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7f84905a4dd1] 2024-05-27T07:30:08.673 INFO:journalctl@ceph.mds.k.smithi105.stdout:May 27 07:30:08 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-k[70124]: 6: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x56546478423d] 2024-05-27T07:30:08.673 INFO:journalctl@ceph.mds.k.smithi105.stdout:May 27 07:30:08 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-k[70124]: 7: (SnapRealm::build_snap_set() const+0x323) [0x56546476f971] 2024-05-27T07:30:08.673 INFO:journalctl@ceph.mds.k.smithi105.stdout:May 27 07:30:08 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-k[70124]: 8: (SnapRealm::check_cache() const+0x13e) [0x56546476dfa0] 2024-05-27T07:30:08.673 INFO:journalctl@ceph.mds.k.smithi105.stdout:May 27 07:30:08 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-k[70124]: 9: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x56546476e651] 2024-05-27T07:30:08.673 INFO:journalctl@ceph.mds.k.smithi105.stdout:May 27 07:30:08 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-k[70124]: 10: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x565464703347] 2024-05-27T07:30:08.673 INFO:journalctl@ceph.mds.k.smithi105.stdout:May 27 07:30:08 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-k[70124]: 11: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x5654643ff87e] 2024-05-27T07:30:08.673 INFO:journalctl@ceph.mds.k.smithi105.stdout:May 27 07:30:08 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-k[70124]: 12: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x279d) [0x565464404f2f] 2024-05-27T07:30:08.673 INFO:journalctl@ceph.mds.k.smithi105.stdout:May 27 07:30:08 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-k[70124]: 13: (EUpdate::replay(MDSRank*)+0x63) [0x56546440f969] 2024-05-27T07:30:08.673 INFO:journalctl@ceph.mds.k.smithi105.stdout:May 27 07:30:08 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-k[70124]: 14: (MDLog::_replay_thread()+0x1f38) [0x5654647a848c] 2024-05-27T07:30:08.673 INFO:journalctl@ceph.mds.k.smithi105.stdout:May 27 07:30:08 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-k[70124]: 15: (MDLog::ReplayThread::entry()+0x11) [0x5654647b19cb] 2024-05-27T07:30:08.673 INFO:journalctl@ceph.mds.k.smithi105.stdout:May 27 07:30:08 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-k[70124]: 16: (Thread::entry_wrapper()+0x43) [0x7f849057372d] 2024-05-27T07:30:08.673 INFO:journalctl@ceph.mds.k.smithi105.stdout:May 27 07:30:08 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-k[70124]: 17: (Thread::_entry_func(void*)+0xd) [0x7f8490573749] 2024-05-27T07:30:08.673 INFO:journalctl@ceph.mds.k.smithi105.stdout:May 27 07:30:08 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-k[70124]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7f848f1171ca] 2024-05-27T07:30:08.674 INFO:journalctl@ceph.mds.k.smithi105.stdout:May 27 07:30:08 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-k[70124]: 19: clone() 2024-05-27T07:30:08.674 INFO:journalctl@ceph.mds.k.smithi105.stdout:May 27 07:30:08 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-k[70124]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-05-27T07:30:08.674 INFO:journalctl@ceph.mds.k.smithi105.stdout:May 27 07:30:08 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-k[70124]: 2024-05-27T07:30:08.674 INFO:journalctl@ceph.mds.k.smithi105.stdout:May 27 07:30:08 smithi105 bash[70110]: ef69d02153ef7bfcd8563b2cac6fc29312888b9219ed7f50788b3fe56d33d46f 2024-05-27T07:30:08.674 INFO:journalctl@ceph.mds.k.smithi105.stdout:May 27 07:30:08 smithi105 systemd[1]: Started Ceph mds.k for 73df2636-1bf8-11ef-bc9b-c7b262605968. 2024-05-27T07:30:08.730 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:30:08 smithi052 ceph-mon[31955]: daemon mds.k restarted 2024-05-27T07:30:08.853 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:30:08 smithi173 ceph-mon[37000]: daemon mds.k restarted 2024-05-27T07:30:08.941 INFO:journalctl@ceph.mds.k.smithi105.stdout:May 27 07:30:08 smithi105 podman[70159]: 2024-05-27 07:30:08.696496526 +0000 UTC m=+0.101478043 container died ef69d02153ef7bfcd8563b2cac6fc29312888b9219ed7f50788b3fe56d33d46f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-k, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, org.label-schema.license=GPLv2, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, org.label-schema.build-date=20240521, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, org.label-schema.schema-version=1.0) 2024-05-27T07:30:09.182 INFO:journalctl@ceph.mds.d.smithi052.stdout:May 27 07:30:08 smithi052 podman[107194]: 2024-05-27 07:30:08.80125288 +0000 UTC m=+1.154736563 image pull e36ea4fc8a5807e8d2701370e350a0de3856538797f7fba64cfce7e344bd6799 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6 2024-05-27T07:30:09.182 INFO:journalctl@ceph.mds.d.smithi052.stdout:May 27 07:30:08 smithi052 podman[107194]: 2024-05-27 07:30:08.92591631 +0000 UTC m=+1.279399982 container create 0f66c93719468a265ba834a48f395ac973351eab29410ab9f063e4c1561b9e34 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-d, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240521, io.buildah.version=1.33.5, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, ceph=True, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, GIT_CLEAN=True) 2024-05-27T07:30:09.487 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:30:09 smithi052 ceph-mon[31955]: scrub queued for path: / 2024-05-27T07:30:09.487 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:30:09 smithi052 ceph-mon[31955]: scrub summary: idle+waiting paths [/] 2024-05-27T07:30:09.487 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:30:09 smithi052 ceph-mon[31955]: scrub summary: active paths [/] 2024-05-27T07:30:09.487 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:30:09 smithi052 ceph-mon[31955]: osdmap e112: 12 total, 12 up, 12 in 2024-05-27T07:30:09.487 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:30:09 smithi052 ceph-mon[31955]: mds.? [v2:172.21.15.105:6832/4278076901,v1:172.21.15.105:6833/4278076901] up:boot 2024-05-27T07:30:09.487 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:30:09 smithi052 ceph-mon[31955]: fsmap cephfs:5 2 up:standby-replay 1 up:standby 5 up:active 2024-05-27T07:30:09.487 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:30:09 smithi052 ceph-mon[31955]: from='mgr.15333 172.21.15.52:0/416389082' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "k"}]: dispatch 2024-05-27T07:30:09.488 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:30:09 smithi052 ceph-mon[31955]: fsmap cephfs:5 3 up:standby-replay 5 up:active 2024-05-27T07:30:09.488 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:30:09 smithi052 ceph-mon[31955]: pgmap v150: 129 pgs: 129 active+clean; 316 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 7.7 MiB/s wr, 580 op/s 2024-05-27T07:30:09.488 INFO:journalctl@ceph.mds.a.smithi052.stdout:May 27 07:30:09 smithi052 podman[107170]: 2024-05-27 07:30:09.192189704 +0000 UTC m=+1.571984000 container init e5f4c04c08e94f426d7470f4fb1ace3e1702e3663fcdc999a50fd8cdce849ac9 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-a, ceph=True, org.label-schema.build-date=20240521, io.buildah.version=1.33.5, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, org.label-schema.vendor=CentOS, GIT_CLEAN=True) 2024-05-27T07:30:09.488 INFO:journalctl@ceph.mds.a.smithi052.stdout:May 27 07:30:09 smithi052 podman[107170]: 2024-05-27 07:30:09.201021816 +0000 UTC m=+1.580816105 container start e5f4c04c08e94f426d7470f4fb1ace3e1702e3663fcdc999a50fd8cdce849ac9 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-a, GIT_CLEAN=True, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, GIT_BRANCH=HEAD, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, org.label-schema.build-date=20240521, io.buildah.version=1.33.5) 2024-05-27T07:30:09.488 INFO:journalctl@ceph.mds.a.smithi052.stdout:May 27 07:30:09 smithi052 ceph-mds[107243]: -- 172.21.15.52:0/3916708063 <== mon.0 v2:172.21.15.52:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x563e2e191a40 con 0x563e2e1d9600 2024-05-27T07:30:09.488 INFO:journalctl@ceph.mds.a.smithi052.stdout:May 27 07:30:09 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-a[107232]: starting mds.a at 2024-05-27T07:30:09.488 INFO:journalctl@ceph.mds.a.smithi052.stdout:May 27 07:30:09 smithi052 bash[107170]: e5f4c04c08e94f426d7470f4fb1ace3e1702e3663fcdc999a50fd8cdce849ac9 2024-05-27T07:30:09.488 INFO:journalctl@ceph.mds.a.smithi052.stdout:May 27 07:30:09 smithi052 systemd[1]: Started Ceph mds.a for 73df2636-1bf8-11ef-bc9b-c7b262605968. 2024-05-27T07:30:09.691 INFO:journalctl@ceph.mds.k.smithi105.stdout:May 27 07:30:09 smithi105 podman[70159]: 2024-05-27 07:30:09.365040672 +0000 UTC m=+0.770022167 container remove ef69d02153ef7bfcd8563b2cac6fc29312888b9219ed7f50788b3fe56d33d46f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-k, GIT_CLEAN=True, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, org.label-schema.build-date=20240521, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, ceph=True, maintainer=Guillaume Abrioux ) 2024-05-27T07:30:09.691 INFO:journalctl@ceph.mds.k.smithi105.stdout:May 27 07:30:09 smithi105 systemd[1]: ceph-73df2636-1bf8-11ef-bc9b-c7b262605968@mds.k.service: Main process exited, code=exited, status=134/n/a 2024-05-27T07:30:09.691 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:30:09 smithi105 ceph-mon[35977]: scrub queued for path: / 2024-05-27T07:30:09.691 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:30:09 smithi105 ceph-mon[35977]: scrub summary: idle+waiting paths [/] 2024-05-27T07:30:09.691 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:30:09 smithi105 ceph-mon[35977]: scrub summary: active paths [/] 2024-05-27T07:30:09.691 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:30:09 smithi105 ceph-mon[35977]: osdmap e112: 12 total, 12 up, 12 in 2024-05-27T07:30:09.691 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:30:09 smithi105 ceph-mon[35977]: mds.? [v2:172.21.15.105:6832/4278076901,v1:172.21.15.105:6833/4278076901] up:boot 2024-05-27T07:30:09.692 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:30:09 smithi105 ceph-mon[35977]: fsmap cephfs:5 2 up:standby-replay 1 up:standby 5 up:active 2024-05-27T07:30:09.692 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:30:09 smithi105 ceph-mon[35977]: from='mgr.15333 172.21.15.52:0/416389082' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "k"}]: dispatch 2024-05-27T07:30:09.692 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:30:09 smithi105 ceph-mon[35977]: fsmap cephfs:5 3 up:standby-replay 5 up:active 2024-05-27T07:30:09.692 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:30:09 smithi105 ceph-mon[35977]: pgmap v150: 129 pgs: 129 active+clean; 316 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 7.7 MiB/s wr, 580 op/s 2024-05-27T07:30:09.734 INFO:journalctl@ceph.mds.d.smithi052.stdout:May 27 07:30:09 smithi052 podman[107194]: 2024-05-27 07:30:09.638674476 +0000 UTC m=+1.992158155 container init 0f66c93719468a265ba834a48f395ac973351eab29410ab9f063e4c1561b9e34 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-d, io.buildah.version=1.33.5, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.build-date=20240521, ceph=True, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, GIT_CLEAN=True, org.label-schema.vendor=CentOS) 2024-05-27T07:30:09.734 INFO:journalctl@ceph.mds.d.smithi052.stdout:May 27 07:30:09 smithi052 podman[107194]: 2024-05-27 07:30:09.652391186 +0000 UTC m=+2.005874856 container start 0f66c93719468a265ba834a48f395ac973351eab29410ab9f063e4c1561b9e34 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-d, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, GIT_CLEAN=True, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, org.label-schema.build-date=20240521, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, org.label-schema.schema-version=1.0, ceph=True, org.label-schema.vendor=CentOS) 2024-05-27T07:30:09.734 INFO:journalctl@ceph.mds.d.smithi052.stdout:May 27 07:30:09 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-d[107239]: starting mds.d at 2024-05-27T07:30:09.853 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:30:09 smithi173 ceph-mon[37000]: scrub queued for path: / 2024-05-27T07:30:09.853 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:30:09 smithi173 ceph-mon[37000]: scrub summary: idle+waiting paths [/] 2024-05-27T07:30:09.853 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:30:09 smithi173 ceph-mon[37000]: scrub summary: active paths [/] 2024-05-27T07:30:09.853 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:30:09 smithi173 ceph-mon[37000]: osdmap e112: 12 total, 12 up, 12 in 2024-05-27T07:30:09.853 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:30:09 smithi173 ceph-mon[37000]: mds.? [v2:172.21.15.105:6832/4278076901,v1:172.21.15.105:6833/4278076901] up:boot 2024-05-27T07:30:09.853 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:30:09 smithi173 ceph-mon[37000]: fsmap cephfs:5 2 up:standby-replay 1 up:standby 5 up:active 2024-05-27T07:30:09.853 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:30:09 smithi173 ceph-mon[37000]: from='mgr.15333 172.21.15.52:0/416389082' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "k"}]: dispatch 2024-05-27T07:30:09.854 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:30:09 smithi173 ceph-mon[37000]: fsmap cephfs:5 3 up:standby-replay 5 up:active 2024-05-27T07:30:09.854 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:30:09 smithi173 ceph-mon[37000]: pgmap v150: 129 pgs: 129 active+clean; 316 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 7.7 MiB/s wr, 580 op/s 2024-05-27T07:30:10.163 INFO:journalctl@ceph.mds.d.smithi052.stdout:May 27 07:30:09 smithi052 bash[107194]: 0f66c93719468a265ba834a48f395ac973351eab29410ab9f063e4c1561b9e34 2024-05-27T07:30:10.163 INFO:journalctl@ceph.mds.d.smithi052.stdout:May 27 07:30:09 smithi052 systemd[1]: Started Ceph mds.d for 73df2636-1bf8-11ef-bc9b-c7b262605968. 2024-05-27T07:30:10.191 INFO:journalctl@ceph.mds.k.smithi105.stdout:May 27 07:30:09 smithi105 systemd[1]: ceph-73df2636-1bf8-11ef-bc9b-c7b262605968@mds.k.service: Failed with result 'exit-code'. 2024-05-27T07:30:10.344 DEBUG:teuthology.orchestra.run.smithi052:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:38f9d900c83c63799fbdbe61acc9a11b0d3554a6 shell --fsid 73df2636-1bf8-11ef-bc9b-c7b262605968 -- ceph tell mds.1:0 scrub status 2024-05-27T07:30:10.468 INFO:journalctl@ceph.mds.j.smithi052.stdout:May 27 07:30:10 smithi052 systemd[1]: ceph-73df2636-1bf8-11ef-bc9b-c7b262605968@mds.j.service: Failed with result 'exit-code'. 2024-05-27T07:30:10.468 INFO:journalctl@ceph.mds.j.smithi052.stdout:May 27 07:30:10 smithi052 systemd[1]: ceph-73df2636-1bf8-11ef-bc9b-c7b262605968@mds.j.service: Consumed 1.226s CPU time. 2024-05-27T07:30:10.731 INFO:journalctl@ceph.mds.a.smithi052.stdout:May 27 07:30:10 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-a[107232]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: In function 'std::set SnapClient::filter(const std::set&) const' thread 7f5fe83d7700 time 2024-05-27T07:30:10.500154+0000 2024-05-27T07:30:10.731 INFO:journalctl@ceph.mds.a.smithi052.stdout:May 27 07:30:10 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-a[107232]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: 197: FAILED ceph_assert(cached_version > 0) 2024-05-27T07:30:10.731 INFO:journalctl@ceph.mds.a.smithi052.stdout:May 27 07:30:10 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-a[107232]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:30:10.732 INFO:journalctl@ceph.mds.a.smithi052.stdout:May 27 07:30:10 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-a[107232]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x127) [0x7f5ffac1cba6] 2024-05-27T07:30:10.732 INFO:journalctl@ceph.mds.a.smithi052.stdout:May 27 07:30:10 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-a[107232]: 2: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7f5ffac1cdd1] 2024-05-27T07:30:10.732 INFO:journalctl@ceph.mds.a.smithi052.stdout:May 27 07:30:10 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-a[107232]: 3: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x563e2cf9b23d] 2024-05-27T07:30:10.732 INFO:journalctl@ceph.mds.a.smithi052.stdout:May 27 07:30:10 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-a[107232]: 4: (SnapRealm::build_snap_set() const+0x323) [0x563e2cf86971] 2024-05-27T07:30:10.732 INFO:journalctl@ceph.mds.a.smithi052.stdout:May 27 07:30:10 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-a[107232]: 5: (SnapRealm::check_cache() const+0x13e) [0x563e2cf84fa0] 2024-05-27T07:30:10.732 INFO:journalctl@ceph.mds.a.smithi052.stdout:May 27 07:30:10 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-a[107232]: 6: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x563e2cf85651] 2024-05-27T07:30:10.732 INFO:journalctl@ceph.mds.a.smithi052.stdout:May 27 07:30:10 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-a[107232]: 7: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x563e2cf1a347] 2024-05-27T07:30:10.732 INFO:journalctl@ceph.mds.a.smithi052.stdout:May 27 07:30:10 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-a[107232]: 8: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x563e2cc1687e] 2024-05-27T07:30:10.732 INFO:journalctl@ceph.mds.a.smithi052.stdout:May 27 07:30:10 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-a[107232]: 9: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x2702) [0x563e2cc1be94] 2024-05-27T07:30:10.732 INFO:journalctl@ceph.mds.a.smithi052.stdout:May 27 07:30:10 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-a[107232]: 10: (EUpdate::replay(MDSRank*)+0x63) [0x563e2cc26969] 2024-05-27T07:30:10.732 INFO:journalctl@ceph.mds.a.smithi052.stdout:May 27 07:30:10 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-a[107232]: 11: (MDLog::_replay_thread()+0x1f38) [0x563e2cfbf48c] 2024-05-27T07:30:10.732 INFO:journalctl@ceph.mds.a.smithi052.stdout:May 27 07:30:10 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-a[107232]: 12: (MDLog::ReplayThread::entry()+0x11) [0x563e2cfc89cb] 2024-05-27T07:30:10.732 INFO:journalctl@ceph.mds.a.smithi052.stdout:May 27 07:30:10 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-a[107232]: 13: (Thread::entry_wrapper()+0x43) [0x7f5ffabeb72d] 2024-05-27T07:30:10.733 INFO:journalctl@ceph.mds.a.smithi052.stdout:May 27 07:30:10 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-a[107232]: 14: (Thread::_entry_func(void*)+0xd) [0x7f5ffabeb749] 2024-05-27T07:30:10.733 INFO:journalctl@ceph.mds.a.smithi052.stdout:May 27 07:30:10 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-a[107232]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7f5ff978f1ca] 2024-05-27T07:30:10.733 INFO:journalctl@ceph.mds.a.smithi052.stdout:May 27 07:30:10 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-a[107232]: 16: clone() 2024-05-27T07:30:10.733 INFO:journalctl@ceph.mds.a.smithi052.stdout:May 27 07:30:10 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-a[107232]: *** Caught signal (Aborted) ** 2024-05-27T07:30:10.733 INFO:journalctl@ceph.mds.a.smithi052.stdout:May 27 07:30:10 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-a[107232]: in thread 7f5fe83d7700 thread_name:md_log_replay 2024-05-27T07:30:10.733 INFO:journalctl@ceph.mds.a.smithi052.stdout:May 27 07:30:10 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-a[107232]: 2024-05-27T07:30:10.503+0000 7f5fe83d7700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: In function 'std::set SnapClient::filter(const std::set&) const' thread 7f5fe83d7700 time 2024-05-27T07:30:10.500154+0000 2024-05-27T07:30:10.733 INFO:journalctl@ceph.mds.a.smithi052.stdout:May 27 07:30:10 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-a[107232]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: 197: FAILED ceph_assert(cached_version > 0) 2024-05-27T07:30:10.733 INFO:journalctl@ceph.mds.a.smithi052.stdout:May 27 07:30:10 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-a[107232]: 2024-05-27T07:30:10.734 INFO:journalctl@ceph.mds.a.smithi052.stdout:May 27 07:30:10 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-a[107232]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:30:10.734 INFO:journalctl@ceph.mds.a.smithi052.stdout:May 27 07:30:10 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-a[107232]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x127) [0x7f5ffac1cba6] 2024-05-27T07:30:10.734 INFO:journalctl@ceph.mds.a.smithi052.stdout:May 27 07:30:10 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-a[107232]: 2: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7f5ffac1cdd1] 2024-05-27T07:30:10.734 INFO:journalctl@ceph.mds.a.smithi052.stdout:May 27 07:30:10 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-a[107232]: 3: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x563e2cf9b23d] 2024-05-27T07:30:10.734 INFO:journalctl@ceph.mds.a.smithi052.stdout:May 27 07:30:10 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-a[107232]: 4: (SnapRealm::build_snap_set() const+0x323) [0x563e2cf86971] 2024-05-27T07:30:10.734 INFO:journalctl@ceph.mds.a.smithi052.stdout:May 27 07:30:10 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-a[107232]: 5: (SnapRealm::check_cache() const+0x13e) [0x563e2cf84fa0] 2024-05-27T07:30:10.734 INFO:journalctl@ceph.mds.a.smithi052.stdout:May 27 07:30:10 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-a[107232]: 6: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x563e2cf85651] 2024-05-27T07:30:10.734 INFO:journalctl@ceph.mds.a.smithi052.stdout:May 27 07:30:10 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-a[107232]: 7: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x563e2cf1a347] 2024-05-27T07:30:10.734 INFO:journalctl@ceph.mds.a.smithi052.stdout:May 27 07:30:10 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-a[107232]: 8: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x563e2cc1687e] 2024-05-27T07:30:10.734 INFO:journalctl@ceph.mds.a.smithi052.stdout:May 27 07:30:10 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-a[107232]: 9: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x2702) [0x563e2cc1be94] 2024-05-27T07:30:10.734 INFO:journalctl@ceph.mds.a.smithi052.stdout:May 27 07:30:10 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-a[107232]: 10: (EUpdate::replay(MDSRank*)+0x63) [0x563e2cc26969] 2024-05-27T07:30:10.735 INFO:journalctl@ceph.mds.a.smithi052.stdout:May 27 07:30:10 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-a[107232]: 11: (MDLog::_replay_thread()+0x1f38) [0x563e2cfbf48c] 2024-05-27T07:30:10.735 INFO:journalctl@ceph.mds.a.smithi052.stdout:May 27 07:30:10 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-a[107232]: 12: (MDLog::ReplayThread::entry()+0x11) [0x563e2cfc89cb] 2024-05-27T07:30:10.735 INFO:journalctl@ceph.mds.a.smithi052.stdout:May 27 07:30:10 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-a[107232]: 13: (Thread::entry_wrapper()+0x43) [0x7f5ffabeb72d] 2024-05-27T07:30:10.735 INFO:journalctl@ceph.mds.a.smithi052.stdout:May 27 07:30:10 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-a[107232]: 14: (Thread::_entry_func(void*)+0xd) [0x7f5ffabeb749] 2024-05-27T07:30:10.735 INFO:journalctl@ceph.mds.a.smithi052.stdout:May 27 07:30:10 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-a[107232]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7f5ff978f1ca] 2024-05-27T07:30:10.735 INFO:journalctl@ceph.mds.a.smithi052.stdout:May 27 07:30:10 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-a[107232]: 16: clone() 2024-05-27T07:30:10.735 INFO:journalctl@ceph.mds.a.smithi052.stdout:May 27 07:30:10 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-a[107232]: 2024-05-27T07:30:10.735 INFO:journalctl@ceph.mds.a.smithi052.stdout:May 27 07:30:10 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-a[107232]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:30:10.735 INFO:journalctl@ceph.mds.a.smithi052.stdout:May 27 07:30:10 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-a[107232]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f5ff9799d20] 2024-05-27T07:30:10.735 INFO:journalctl@ceph.mds.a.smithi052.stdout:May 27 07:30:10 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-a[107232]: 2: gsignal() 2024-05-27T07:30:10.735 INFO:journalctl@ceph.mds.a.smithi052.stdout:May 27 07:30:10 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-a[107232]: 3: abort() 2024-05-27T07:30:10.735 INFO:journalctl@ceph.mds.a.smithi052.stdout:May 27 07:30:10 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-a[107232]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x26e) [0x7f5ffac1cced] 2024-05-27T07:30:10.736 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:30:10 smithi052 ceph-mon[31955]: mds.? [v2:172.21.15.52:6834/3479317649,v1:172.21.15.52:6835/3479317649] up:boot 2024-05-27T07:30:10.736 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:30:10 smithi052 ceph-mon[31955]: fsmap cephfs:5 3 up:standby-replay 1 up:standby 5 up:active 2024-05-27T07:30:10.736 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:30:10 smithi052 ceph-mon[31955]: from='mgr.15333 172.21.15.52:0/416389082' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "a"}]: dispatch 2024-05-27T07:30:10.736 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:30:10 smithi052 ceph-mon[31955]: fsmap cephfs:5 4 up:standby-replay 5 up:active 2024-05-27T07:30:10.736 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:30:10 smithi052 ceph-mon[31955]: mds.? [v2:172.21.15.52:6836/75944465,v1:172.21.15.52:6837/75944465] up:boot 2024-05-27T07:30:10.736 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:30:10 smithi052 ceph-mon[31955]: fsmap cephfs:5 5 up:standby-replay 5 up:active 2024-05-27T07:30:10.736 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:30:10 smithi052 ceph-mon[31955]: from='mgr.15333 172.21.15.52:0/416389082' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "d"}]: dispatch 2024-05-27T07:30:10.737 INFO:journalctl@ceph.mds.a.smithi052.stdout:May 27 07:30:10 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-a[107232]: 5: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7f5ffac1cdd1] 2024-05-27T07:30:10.737 INFO:journalctl@ceph.mds.a.smithi052.stdout:May 27 07:30:10 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-a[107232]: 6: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x563e2cf9b23d] 2024-05-27T07:30:10.737 INFO:journalctl@ceph.mds.a.smithi052.stdout:May 27 07:30:10 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-a[107232]: 7: (SnapRealm::build_snap_set() const+0x323) [0x563e2cf86971] 2024-05-27T07:30:10.737 INFO:journalctl@ceph.mds.a.smithi052.stdout:May 27 07:30:10 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-a[107232]: 8: (SnapRealm::check_cache() const+0x13e) [0x563e2cf84fa0] 2024-05-27T07:30:10.737 INFO:journalctl@ceph.mds.a.smithi052.stdout:May 27 07:30:10 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-a[107232]: 9: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x563e2cf85651] 2024-05-27T07:30:10.737 INFO:journalctl@ceph.mds.a.smithi052.stdout:May 27 07:30:10 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-a[107232]: 10: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x563e2cf1a347] 2024-05-27T07:30:10.737 INFO:journalctl@ceph.mds.a.smithi052.stdout:May 27 07:30:10 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-a[107232]: 11: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x563e2cc1687e] 2024-05-27T07:30:10.737 INFO:journalctl@ceph.mds.a.smithi052.stdout:May 27 07:30:10 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-a[107232]: 12: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x2702) [0x563e2cc1be94] 2024-05-27T07:30:10.737 INFO:journalctl@ceph.mds.a.smithi052.stdout:May 27 07:30:10 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-a[107232]: 13: (EUpdate::replay(MDSRank*)+0x63) [0x563e2cc26969] 2024-05-27T07:30:10.737 INFO:journalctl@ceph.mds.a.smithi052.stdout:May 27 07:30:10 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-a[107232]: 14: (MDLog::_replay_thread()+0x1f38) [0x563e2cfbf48c] 2024-05-27T07:30:10.737 INFO:journalctl@ceph.mds.a.smithi052.stdout:May 27 07:30:10 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-a[107232]: 15: (MDLog::ReplayThread::entry()+0x11) [0x563e2cfc89cb] 2024-05-27T07:30:10.738 INFO:journalctl@ceph.mds.a.smithi052.stdout:May 27 07:30:10 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-a[107232]: 16: (Thread::entry_wrapper()+0x43) [0x7f5ffabeb72d] 2024-05-27T07:30:10.738 INFO:journalctl@ceph.mds.a.smithi052.stdout:May 27 07:30:10 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-a[107232]: 17: (Thread::_entry_func(void*)+0xd) [0x7f5ffabeb749] 2024-05-27T07:30:10.738 INFO:journalctl@ceph.mds.a.smithi052.stdout:May 27 07:30:10 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-a[107232]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7f5ff978f1ca] 2024-05-27T07:30:10.738 INFO:journalctl@ceph.mds.a.smithi052.stdout:May 27 07:30:10 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-a[107232]: 19: clone() 2024-05-27T07:30:10.738 INFO:journalctl@ceph.mds.a.smithi052.stdout:May 27 07:30:10 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-a[107232]: 2024-05-27T07:30:10.507+0000 7f5fe83d7700 -1 *** Caught signal (Aborted) ** 2024-05-27T07:30:10.738 INFO:journalctl@ceph.mds.a.smithi052.stdout:May 27 07:30:10 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-a[107232]: in thread 7f5fe83d7700 thread_name:md_log_replay 2024-05-27T07:30:10.738 INFO:journalctl@ceph.mds.a.smithi052.stdout:May 27 07:30:10 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-a[107232]: 2024-05-27T07:30:10.738 INFO:journalctl@ceph.mds.a.smithi052.stdout:May 27 07:30:10 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-a[107232]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:30:10.738 INFO:journalctl@ceph.mds.a.smithi052.stdout:May 27 07:30:10 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-a[107232]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f5ff9799d20] 2024-05-27T07:30:10.738 INFO:journalctl@ceph.mds.a.smithi052.stdout:May 27 07:30:10 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-a[107232]: 2: gsignal() 2024-05-27T07:30:10.738 INFO:journalctl@ceph.mds.a.smithi052.stdout:May 27 07:30:10 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-a[107232]: 3: abort() 2024-05-27T07:30:10.738 INFO:journalctl@ceph.mds.a.smithi052.stdout:May 27 07:30:10 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-a[107232]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x26e) [0x7f5ffac1cced] 2024-05-27T07:30:10.738 INFO:journalctl@ceph.mds.a.smithi052.stdout:May 27 07:30:10 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-a[107232]: 5: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7f5ffac1cdd1] 2024-05-27T07:30:10.739 INFO:journalctl@ceph.mds.a.smithi052.stdout:May 27 07:30:10 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-a[107232]: 6: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x563e2cf9b23d] 2024-05-27T07:30:10.739 INFO:journalctl@ceph.mds.a.smithi052.stdout:May 27 07:30:10 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-a[107232]: 7: (SnapRealm::build_snap_set() const+0x323) [0x563e2cf86971] 2024-05-27T07:30:10.739 INFO:journalctl@ceph.mds.a.smithi052.stdout:May 27 07:30:10 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-a[107232]: 8: (SnapRealm::check_cache() const+0x13e) [0x563e2cf84fa0] 2024-05-27T07:30:10.739 INFO:journalctl@ceph.mds.a.smithi052.stdout:May 27 07:30:10 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-a[107232]: 9: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x563e2cf85651] 2024-05-27T07:30:10.739 INFO:journalctl@ceph.mds.a.smithi052.stdout:May 27 07:30:10 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-a[107232]: 10: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x563e2cf1a347] 2024-05-27T07:30:10.739 INFO:journalctl@ceph.mds.a.smithi052.stdout:May 27 07:30:10 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-a[107232]: 11: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x563e2cc1687e] 2024-05-27T07:30:10.739 INFO:journalctl@ceph.mds.a.smithi052.stdout:May 27 07:30:10 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-a[107232]: 12: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x2702) [0x563e2cc1be94] 2024-05-27T07:30:10.739 INFO:journalctl@ceph.mds.a.smithi052.stdout:May 27 07:30:10 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-a[107232]: 13: (EUpdate::replay(MDSRank*)+0x63) [0x563e2cc26969] 2024-05-27T07:30:10.739 INFO:journalctl@ceph.mds.a.smithi052.stdout:May 27 07:30:10 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-a[107232]: 14: (MDLog::_replay_thread()+0x1f38) [0x563e2cfbf48c] 2024-05-27T07:30:10.739 INFO:journalctl@ceph.mds.a.smithi052.stdout:May 27 07:30:10 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-a[107232]: 15: (MDLog::ReplayThread::entry()+0x11) [0x563e2cfc89cb] 2024-05-27T07:30:10.739 INFO:journalctl@ceph.mds.a.smithi052.stdout:May 27 07:30:10 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-a[107232]: 16: (Thread::entry_wrapper()+0x43) [0x7f5ffabeb72d] 2024-05-27T07:30:10.739 INFO:journalctl@ceph.mds.a.smithi052.stdout:May 27 07:30:10 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-a[107232]: 17: (Thread::_entry_func(void*)+0xd) [0x7f5ffabeb749] 2024-05-27T07:30:10.740 INFO:journalctl@ceph.mds.a.smithi052.stdout:May 27 07:30:10 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-a[107232]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7f5ff978f1ca] 2024-05-27T07:30:10.740 INFO:journalctl@ceph.mds.a.smithi052.stdout:May 27 07:30:10 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-a[107232]: 19: clone() 2024-05-27T07:30:10.740 INFO:journalctl@ceph.mds.a.smithi052.stdout:May 27 07:30:10 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-a[107232]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-05-27T07:30:10.740 INFO:journalctl@ceph.mds.a.smithi052.stdout:May 27 07:30:10 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-a[107232]: 2024-05-27T07:30:10.740 INFO:journalctl@ceph.mds.a.smithi052.stdout:May 27 07:30:10 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-a[107232]: -1> 2024-05-27T07:30:10.503+0000 7f5fe83d7700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: In function 'std::set SnapClient::filter(const std::set&) const' thread 7f5fe83d7700 time 2024-05-27T07:30:10.500154+0000 2024-05-27T07:30:10.740 INFO:journalctl@ceph.mds.a.smithi052.stdout:May 27 07:30:10 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-a[107232]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: 197: FAILED ceph_assert(cached_version > 0) 2024-05-27T07:30:10.740 INFO:journalctl@ceph.mds.a.smithi052.stdout:May 27 07:30:10 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-a[107232]: 2024-05-27T07:30:10.740 INFO:journalctl@ceph.mds.a.smithi052.stdout:May 27 07:30:10 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-a[107232]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:30:10.740 INFO:journalctl@ceph.mds.a.smithi052.stdout:May 27 07:30:10 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-a[107232]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x127) [0x7f5ffac1cba6] 2024-05-27T07:30:10.741 INFO:journalctl@ceph.mds.a.smithi052.stdout:May 27 07:30:10 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-a[107232]: 2: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7f5ffac1cdd1] 2024-05-27T07:30:10.741 INFO:journalctl@ceph.mds.a.smithi052.stdout:May 27 07:30:10 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-a[107232]: 3: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x563e2cf9b23d] 2024-05-27T07:30:10.741 INFO:journalctl@ceph.mds.a.smithi052.stdout:May 27 07:30:10 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-a[107232]: 4: (SnapRealm::build_snap_set() const+0x323) [0x563e2cf86971] 2024-05-27T07:30:10.741 INFO:journalctl@ceph.mds.a.smithi052.stdout:May 27 07:30:10 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-a[107232]: 5: (SnapRealm::check_cache() const+0x13e) [0x563e2cf84fa0] 2024-05-27T07:30:10.741 INFO:journalctl@ceph.mds.a.smithi052.stdout:May 27 07:30:10 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-a[107232]: 6: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x563e2cf85651] 2024-05-27T07:30:10.741 INFO:journalctl@ceph.mds.a.smithi052.stdout:May 27 07:30:10 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-a[107232]: 7: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x563e2cf1a347] 2024-05-27T07:30:10.741 INFO:journalctl@ceph.mds.a.smithi052.stdout:May 27 07:30:10 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-a[107232]: 8: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x563e2cc1687e] 2024-05-27T07:30:10.741 INFO:journalctl@ceph.mds.a.smithi052.stdout:May 27 07:30:10 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-a[107232]: 9: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x2702) [0x563e2cc1be94] 2024-05-27T07:30:10.741 INFO:journalctl@ceph.mds.a.smithi052.stdout:May 27 07:30:10 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-a[107232]: 10: (EUpdate::replay(MDSRank*)+0x63) [0x563e2cc26969] 2024-05-27T07:30:10.741 INFO:journalctl@ceph.mds.a.smithi052.stdout:May 27 07:30:10 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-a[107232]: 11: (MDLog::_replay_thread()+0x1f38) [0x563e2cfbf48c] 2024-05-27T07:30:10.741 INFO:journalctl@ceph.mds.a.smithi052.stdout:May 27 07:30:10 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-a[107232]: 12: (MDLog::ReplayThread::entry()+0x11) [0x563e2cfc89cb] 2024-05-27T07:30:10.741 INFO:journalctl@ceph.mds.a.smithi052.stdout:May 27 07:30:10 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-a[107232]: 13: (Thread::entry_wrapper()+0x43) [0x7f5ffabeb72d] 2024-05-27T07:30:10.741 INFO:journalctl@ceph.mds.a.smithi052.stdout:May 27 07:30:10 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-a[107232]: 14: (Thread::_entry_func(void*)+0xd) [0x7f5ffabeb749] 2024-05-27T07:30:10.742 INFO:journalctl@ceph.mds.a.smithi052.stdout:May 27 07:30:10 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-a[107232]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7f5ff978f1ca] 2024-05-27T07:30:10.742 INFO:journalctl@ceph.mds.a.smithi052.stdout:May 27 07:30:10 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-a[107232]: 16: clone() 2024-05-27T07:30:10.742 INFO:journalctl@ceph.mds.a.smithi052.stdout:May 27 07:30:10 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-a[107232]: 2024-05-27T07:30:10.742 INFO:journalctl@ceph.mds.a.smithi052.stdout:May 27 07:30:10 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-a[107232]: 0> 2024-05-27T07:30:10.507+0000 7f5fe83d7700 -1 *** Caught signal (Aborted) ** 2024-05-27T07:30:10.742 INFO:journalctl@ceph.mds.a.smithi052.stdout:May 27 07:30:10 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-a[107232]: in thread 7f5fe83d7700 thread_name:md_log_replay 2024-05-27T07:30:10.742 INFO:journalctl@ceph.mds.a.smithi052.stdout:May 27 07:30:10 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-a[107232]: 2024-05-27T07:30:10.742 INFO:journalctl@ceph.mds.a.smithi052.stdout:May 27 07:30:10 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-a[107232]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:30:10.742 INFO:journalctl@ceph.mds.a.smithi052.stdout:May 27 07:30:10 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-a[107232]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f5ff9799d20] 2024-05-27T07:30:10.742 INFO:journalctl@ceph.mds.a.smithi052.stdout:May 27 07:30:10 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-a[107232]: 2: gsignal() 2024-05-27T07:30:10.742 INFO:journalctl@ceph.mds.a.smithi052.stdout:May 27 07:30:10 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-a[107232]: 3: abort() 2024-05-27T07:30:10.742 INFO:journalctl@ceph.mds.a.smithi052.stdout:May 27 07:30:10 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-a[107232]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x26e) [0x7f5ffac1cced] 2024-05-27T07:30:10.742 INFO:journalctl@ceph.mds.a.smithi052.stdout:May 27 07:30:10 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-a[107232]: 5: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7f5ffac1cdd1] 2024-05-27T07:30:10.742 INFO:journalctl@ceph.mds.a.smithi052.stdout:May 27 07:30:10 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-a[107232]: 6: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x563e2cf9b23d] 2024-05-27T07:30:10.743 INFO:journalctl@ceph.mds.a.smithi052.stdout:May 27 07:30:10 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-a[107232]: 7: (SnapRealm::build_snap_set() const+0x323) [0x563e2cf86971] 2024-05-27T07:30:10.743 INFO:journalctl@ceph.mds.a.smithi052.stdout:May 27 07:30:10 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-a[107232]: 8: (SnapRealm::check_cache() const+0x13e) [0x563e2cf84fa0] 2024-05-27T07:30:10.743 INFO:journalctl@ceph.mds.a.smithi052.stdout:May 27 07:30:10 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-a[107232]: 9: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x563e2cf85651] 2024-05-27T07:30:10.743 INFO:journalctl@ceph.mds.a.smithi052.stdout:May 27 07:30:10 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-a[107232]: 10: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x563e2cf1a347] 2024-05-27T07:30:10.743 INFO:journalctl@ceph.mds.a.smithi052.stdout:May 27 07:30:10 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-a[107232]: 11: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x563e2cc1687e] 2024-05-27T07:30:10.743 INFO:journalctl@ceph.mds.a.smithi052.stdout:May 27 07:30:10 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-a[107232]: 12: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x2702) [0x563e2cc1be94] 2024-05-27T07:30:10.743 INFO:journalctl@ceph.mds.a.smithi052.stdout:May 27 07:30:10 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-a[107232]: 13: (EUpdate::replay(MDSRank*)+0x63) [0x563e2cc26969] 2024-05-27T07:30:10.743 INFO:journalctl@ceph.mds.a.smithi052.stdout:May 27 07:30:10 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-a[107232]: 14: (MDLog::_replay_thread()+0x1f38) [0x563e2cfbf48c] 2024-05-27T07:30:10.743 INFO:journalctl@ceph.mds.a.smithi052.stdout:May 27 07:30:10 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-a[107232]: 15: (MDLog::ReplayThread::entry()+0x11) [0x563e2cfc89cb] 2024-05-27T07:30:10.743 INFO:journalctl@ceph.mds.a.smithi052.stdout:May 27 07:30:10 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-a[107232]: 16: (Thread::entry_wrapper()+0x43) [0x7f5ffabeb72d] 2024-05-27T07:30:10.743 INFO:journalctl@ceph.mds.a.smithi052.stdout:May 27 07:30:10 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-a[107232]: 17: (Thread::_entry_func(void*)+0xd) [0x7f5ffabeb749] 2024-05-27T07:30:10.743 INFO:journalctl@ceph.mds.a.smithi052.stdout:May 27 07:30:10 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-a[107232]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7f5ff978f1ca] 2024-05-27T07:30:10.743 INFO:journalctl@ceph.mds.a.smithi052.stdout:May 27 07:30:10 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-a[107232]: 19: clone() 2024-05-27T07:30:10.744 INFO:journalctl@ceph.mds.a.smithi052.stdout:May 27 07:30:10 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-a[107232]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-05-27T07:30:10.744 INFO:journalctl@ceph.mds.a.smithi052.stdout:May 27 07:30:10 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-a[107232]: 2024-05-27T07:30:10.744 INFO:journalctl@ceph.mds.a.smithi052.stdout:May 27 07:30:10 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-a[107232]: -1> 2024-05-27T07:30:10.503+0000 7f5fe83d7700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: In function 'std::set SnapClient::filter(const std::set&) const' thread 7f5fe83d7700 time 2024-05-27T07:30:10.500154+0000 2024-05-27T07:30:10.744 INFO:journalctl@ceph.mds.a.smithi052.stdout:May 27 07:30:10 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-a[107232]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: 197: FAILED ceph_assert(cached_version > 0) 2024-05-27T07:30:10.744 INFO:journalctl@ceph.mds.a.smithi052.stdout:May 27 07:30:10 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-a[107232]: 2024-05-27T07:30:10.744 INFO:journalctl@ceph.mds.a.smithi052.stdout:May 27 07:30:10 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-a[107232]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:30:10.744 INFO:journalctl@ceph.mds.a.smithi052.stdout:May 27 07:30:10 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-a[107232]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x127) [0x7f5ffac1cba6] 2024-05-27T07:30:10.744 INFO:journalctl@ceph.mds.a.smithi052.stdout:May 27 07:30:10 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-a[107232]: 2: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7f5ffac1cdd1] 2024-05-27T07:30:10.744 INFO:journalctl@ceph.mds.a.smithi052.stdout:May 27 07:30:10 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-a[107232]: 3: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x563e2cf9b23d] 2024-05-27T07:30:10.744 INFO:journalctl@ceph.mds.a.smithi052.stdout:May 27 07:30:10 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-a[107232]: 4: (SnapRealm::build_snap_set() const+0x323) [0x563e2cf86971] 2024-05-27T07:30:10.744 INFO:journalctl@ceph.mds.a.smithi052.stdout:May 27 07:30:10 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-a[107232]: 5: (SnapRealm::check_cache() const+0x13e) [0x563e2cf84fa0] 2024-05-27T07:30:10.744 INFO:journalctl@ceph.mds.a.smithi052.stdout:May 27 07:30:10 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-a[107232]: 6: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x563e2cf85651] 2024-05-27T07:30:10.744 INFO:journalctl@ceph.mds.a.smithi052.stdout:May 27 07:30:10 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-a[107232]: 7: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x563e2cf1a347] 2024-05-27T07:30:10.745 INFO:journalctl@ceph.mds.a.smithi052.stdout:May 27 07:30:10 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-a[107232]: 8: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x563e2cc1687e] 2024-05-27T07:30:10.745 INFO:journalctl@ceph.mds.a.smithi052.stdout:May 27 07:30:10 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-a[107232]: 9: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x2702) [0x563e2cc1be94] 2024-05-27T07:30:10.745 INFO:journalctl@ceph.mds.a.smithi052.stdout:May 27 07:30:10 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-a[107232]: 10: (EUpdate::replay(MDSRank*)+0x63) [0x563e2cc26969] 2024-05-27T07:30:10.745 INFO:journalctl@ceph.mds.a.smithi052.stdout:May 27 07:30:10 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-a[107232]: 11: (MDLog::_replay_thread()+0x1f38) [0x563e2cfbf48c] 2024-05-27T07:30:10.745 INFO:journalctl@ceph.mds.a.smithi052.stdout:May 27 07:30:10 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-a[107232]: 12: (MDLog::ReplayThread::entry()+0x11) [0x563e2cfc89cb] 2024-05-27T07:30:10.745 INFO:journalctl@ceph.mds.a.smithi052.stdout:May 27 07:30:10 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-a[107232]: 13: (Thread::entry_wrapper()+0x43) [0x7f5ffabeb72d] 2024-05-27T07:30:10.745 INFO:journalctl@ceph.mds.a.smithi052.stdout:May 27 07:30:10 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-a[107232]: 14: (Thread::_entry_func(void*)+0xd) [0x7f5ffabeb749] 2024-05-27T07:30:10.745 INFO:journalctl@ceph.mds.a.smithi052.stdout:May 27 07:30:10 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-a[107232]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7f5ff978f1ca] 2024-05-27T07:30:10.745 INFO:journalctl@ceph.mds.a.smithi052.stdout:May 27 07:30:10 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-a[107232]: 16: clone() 2024-05-27T07:30:10.746 INFO:journalctl@ceph.mds.a.smithi052.stdout:May 27 07:30:10 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-a[107232]: 2024-05-27T07:30:10.746 INFO:journalctl@ceph.mds.a.smithi052.stdout:May 27 07:30:10 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-a[107232]: 0> 2024-05-27T07:30:10.507+0000 7f5fe83d7700 -1 *** Caught signal (Aborted) ** 2024-05-27T07:30:10.746 INFO:journalctl@ceph.mds.a.smithi052.stdout:May 27 07:30:10 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-a[107232]: in thread 7f5fe83d7700 thread_name:md_log_replay 2024-05-27T07:30:10.746 INFO:journalctl@ceph.mds.a.smithi052.stdout:May 27 07:30:10 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-a[107232]: 2024-05-27T07:30:10.746 INFO:journalctl@ceph.mds.a.smithi052.stdout:May 27 07:30:10 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-a[107232]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:30:10.746 INFO:journalctl@ceph.mds.a.smithi052.stdout:May 27 07:30:10 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-a[107232]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f5ff9799d20] 2024-05-27T07:30:10.746 INFO:journalctl@ceph.mds.a.smithi052.stdout:May 27 07:30:10 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-a[107232]: 2: gsignal() 2024-05-27T07:30:10.746 INFO:journalctl@ceph.mds.a.smithi052.stdout:May 27 07:30:10 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-a[107232]: 3: abort() 2024-05-27T07:30:10.746 INFO:journalctl@ceph.mds.a.smithi052.stdout:May 27 07:30:10 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-a[107232]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x26e) [0x7f5ffac1cced] 2024-05-27T07:30:10.746 INFO:journalctl@ceph.mds.a.smithi052.stdout:May 27 07:30:10 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-a[107232]: 5: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7f5ffac1cdd1] 2024-05-27T07:30:10.746 INFO:journalctl@ceph.mds.a.smithi052.stdout:May 27 07:30:10 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-a[107232]: 6: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x563e2cf9b23d] 2024-05-27T07:30:10.746 INFO:journalctl@ceph.mds.a.smithi052.stdout:May 27 07:30:10 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-a[107232]: 7: (SnapRealm::build_snap_set() const+0x323) [0x563e2cf86971] 2024-05-27T07:30:10.746 INFO:journalctl@ceph.mds.a.smithi052.stdout:May 27 07:30:10 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-a[107232]: 8: (SnapRealm::check_cache() const+0x13e) [0x563e2cf84fa0] 2024-05-27T07:30:10.747 INFO:journalctl@ceph.mds.a.smithi052.stdout:May 27 07:30:10 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-a[107232]: 9: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x563e2cf85651] 2024-05-27T07:30:10.747 INFO:journalctl@ceph.mds.a.smithi052.stdout:May 27 07:30:10 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-a[107232]: 10: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x563e2cf1a347] 2024-05-27T07:30:10.747 INFO:journalctl@ceph.mds.a.smithi052.stdout:May 27 07:30:10 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-a[107232]: 11: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x563e2cc1687e] 2024-05-27T07:30:10.747 INFO:journalctl@ceph.mds.a.smithi052.stdout:May 27 07:30:10 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-a[107232]: 12: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x2702) [0x563e2cc1be94] 2024-05-27T07:30:10.747 INFO:journalctl@ceph.mds.a.smithi052.stdout:May 27 07:30:10 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-a[107232]: 13: (EUpdate::replay(MDSRank*)+0x63) [0x563e2cc26969] 2024-05-27T07:30:10.747 INFO:journalctl@ceph.mds.a.smithi052.stdout:May 27 07:30:10 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-a[107232]: 14: (MDLog::_replay_thread()+0x1f38) [0x563e2cfbf48c] 2024-05-27T07:30:10.747 INFO:journalctl@ceph.mds.a.smithi052.stdout:May 27 07:30:10 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-a[107232]: 15: (MDLog::ReplayThread::entry()+0x11) [0x563e2cfc89cb] 2024-05-27T07:30:10.747 INFO:journalctl@ceph.mds.a.smithi052.stdout:May 27 07:30:10 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-a[107232]: 16: (Thread::entry_wrapper()+0x43) [0x7f5ffabeb72d] 2024-05-27T07:30:10.747 INFO:journalctl@ceph.mds.a.smithi052.stdout:May 27 07:30:10 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-a[107232]: 17: (Thread::_entry_func(void*)+0xd) [0x7f5ffabeb749] 2024-05-27T07:30:10.747 INFO:journalctl@ceph.mds.a.smithi052.stdout:May 27 07:30:10 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-a[107232]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7f5ff978f1ca] 2024-05-27T07:30:10.747 INFO:journalctl@ceph.mds.a.smithi052.stdout:May 27 07:30:10 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-a[107232]: 19: clone() 2024-05-27T07:30:10.747 INFO:journalctl@ceph.mds.a.smithi052.stdout:May 27 07:30:10 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-a[107232]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-05-27T07:30:10.747 INFO:journalctl@ceph.mds.a.smithi052.stdout:May 27 07:30:10 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-a[107232]: 2024-05-27T07:30:10.748 INFO:journalctl@ceph.mds.a.smithi052.stdout:May 27 07:30:10 smithi052 podman[107335]: 2024-05-27 07:30:10.668505484 +0000 UTC m=+0.063895479 container died e5f4c04c08e94f426d7470f4fb1ace3e1702e3663fcdc999a50fd8cdce849ac9 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-a, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, org.label-schema.build-date=20240521, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=) 2024-05-27T07:30:10.853 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:30:10 smithi173 ceph-mon[37000]: mds.? [v2:172.21.15.52:6834/3479317649,v1:172.21.15.52:6835/3479317649] up:boot 2024-05-27T07:30:10.853 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:30:10 smithi173 ceph-mon[37000]: fsmap cephfs:5 3 up:standby-replay 1 up:standby 5 up:active 2024-05-27T07:30:10.853 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:30:10 smithi173 ceph-mon[37000]: from='mgr.15333 172.21.15.52:0/416389082' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "a"}]: dispatch 2024-05-27T07:30:10.853 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:30:10 smithi173 ceph-mon[37000]: fsmap cephfs:5 4 up:standby-replay 5 up:active 2024-05-27T07:30:10.853 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:30:10 smithi173 ceph-mon[37000]: mds.? [v2:172.21.15.52:6836/75944465,v1:172.21.15.52:6837/75944465] up:boot 2024-05-27T07:30:10.853 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:30:10 smithi173 ceph-mon[37000]: fsmap cephfs:5 5 up:standby-replay 5 up:active 2024-05-27T07:30:10.853 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:30:10 smithi173 ceph-mon[37000]: from='mgr.15333 172.21.15.52:0/416389082' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "d"}]: dispatch 2024-05-27T07:30:10.941 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:30:10 smithi105 ceph-mon[35977]: mds.? [v2:172.21.15.52:6834/3479317649,v1:172.21.15.52:6835/3479317649] up:boot 2024-05-27T07:30:10.941 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:30:10 smithi105 ceph-mon[35977]: fsmap cephfs:5 3 up:standby-replay 1 up:standby 5 up:active 2024-05-27T07:30:10.941 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:30:10 smithi105 ceph-mon[35977]: from='mgr.15333 172.21.15.52:0/416389082' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "a"}]: dispatch 2024-05-27T07:30:10.941 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:30:10 smithi105 ceph-mon[35977]: fsmap cephfs:5 4 up:standby-replay 5 up:active 2024-05-27T07:30:10.941 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:30:10 smithi105 ceph-mon[35977]: mds.? [v2:172.21.15.52:6836/75944465,v1:172.21.15.52:6837/75944465] up:boot 2024-05-27T07:30:10.941 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:30:10 smithi105 ceph-mon[35977]: fsmap cephfs:5 5 up:standby-replay 5 up:active 2024-05-27T07:30:10.941 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:30:10 smithi105 ceph-mon[35977]: from='mgr.15333 172.21.15.52:0/416389082' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "d"}]: dispatch 2024-05-27T07:30:11.482 INFO:journalctl@ceph.mds.d.smithi052.stdout:May 27 07:30:11 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-d[107239]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: In function 'std::set SnapClient::filter(const std::set&) const' thread 7f495b2aa700 time 2024-05-27T07:30:11.144715+0000 2024-05-27T07:30:11.482 INFO:journalctl@ceph.mds.d.smithi052.stdout:May 27 07:30:11 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-d[107239]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: 197: FAILED ceph_assert(cached_version > 0) 2024-05-27T07:30:11.483 INFO:journalctl@ceph.mds.d.smithi052.stdout:May 27 07:30:11 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-d[107239]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:30:11.483 INFO:journalctl@ceph.mds.d.smithi052.stdout:May 27 07:30:11 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-d[107239]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x127) [0x7f496daefba6] 2024-05-27T07:30:11.483 INFO:journalctl@ceph.mds.d.smithi052.stdout:May 27 07:30:11 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-d[107239]: 2: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7f496daefdd1] 2024-05-27T07:30:11.483 INFO:journalctl@ceph.mds.d.smithi052.stdout:May 27 07:30:11 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-d[107239]: 3: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x55dfad04523d] 2024-05-27T07:30:11.483 INFO:journalctl@ceph.mds.d.smithi052.stdout:May 27 07:30:11 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-d[107239]: 4: (SnapRealm::build_snap_set() const+0x323) [0x55dfad030971] 2024-05-27T07:30:11.483 INFO:journalctl@ceph.mds.d.smithi052.stdout:May 27 07:30:11 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-d[107239]: 5: (SnapRealm::check_cache() const+0x13e) [0x55dfad02efa0] 2024-05-27T07:30:11.483 INFO:journalctl@ceph.mds.d.smithi052.stdout:May 27 07:30:11 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-d[107239]: 6: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x55dfad02f651] 2024-05-27T07:30:11.483 INFO:journalctl@ceph.mds.d.smithi052.stdout:May 27 07:30:11 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-d[107239]: 7: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x55dfacfc4347] 2024-05-27T07:30:11.483 INFO:journalctl@ceph.mds.d.smithi052.stdout:May 27 07:30:11 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-d[107239]: 8: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x55dfaccc087e] 2024-05-27T07:30:11.483 INFO:journalctl@ceph.mds.d.smithi052.stdout:May 27 07:30:11 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-d[107239]: 9: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x2702) [0x55dfaccc5e94] 2024-05-27T07:30:11.483 INFO:journalctl@ceph.mds.d.smithi052.stdout:May 27 07:30:11 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-d[107239]: 10: (EUpdate::replay(MDSRank*)+0x63) [0x55dfaccd0969] 2024-05-27T07:30:11.483 INFO:journalctl@ceph.mds.d.smithi052.stdout:May 27 07:30:11 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-d[107239]: 11: (MDLog::_replay_thread()+0x1f38) [0x55dfad06948c] 2024-05-27T07:30:11.483 INFO:journalctl@ceph.mds.d.smithi052.stdout:May 27 07:30:11 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-d[107239]: 12: (MDLog::ReplayThread::entry()+0x11) [0x55dfad0729cb] 2024-05-27T07:30:11.484 INFO:journalctl@ceph.mds.d.smithi052.stdout:May 27 07:30:11 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-d[107239]: 13: (Thread::entry_wrapper()+0x43) [0x7f496dabe72d] 2024-05-27T07:30:11.484 INFO:journalctl@ceph.mds.d.smithi052.stdout:May 27 07:30:11 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-d[107239]: 14: (Thread::_entry_func(void*)+0xd) [0x7f496dabe749] 2024-05-27T07:30:11.484 INFO:journalctl@ceph.mds.d.smithi052.stdout:May 27 07:30:11 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-d[107239]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7f496c6621ca] 2024-05-27T07:30:11.484 INFO:journalctl@ceph.mds.d.smithi052.stdout:May 27 07:30:11 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-d[107239]: 16: clone() 2024-05-27T07:30:11.484 INFO:journalctl@ceph.mds.d.smithi052.stdout:May 27 07:30:11 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-d[107239]: 2024-05-27T07:30:11.149+0000 7f495b2aa700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: In function 'std::set SnapClient::filter(const std::set&) const' thread 7f495b2aa700 time 2024-05-27T07:30:11.144715+0000 2024-05-27T07:30:11.484 INFO:journalctl@ceph.mds.d.smithi052.stdout:May 27 07:30:11 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-d[107239]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: 197: FAILED ceph_assert(cached_version > 0) 2024-05-27T07:30:11.484 INFO:journalctl@ceph.mds.d.smithi052.stdout:May 27 07:30:11 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-d[107239]: 2024-05-27T07:30:11.485 INFO:journalctl@ceph.mds.d.smithi052.stdout:May 27 07:30:11 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-d[107239]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:30:11.485 INFO:journalctl@ceph.mds.d.smithi052.stdout:May 27 07:30:11 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-d[107239]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x127) [0x7f496daefba6] 2024-05-27T07:30:11.485 INFO:journalctl@ceph.mds.d.smithi052.stdout:May 27 07:30:11 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-d[107239]: 2: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7f496daefdd1] 2024-05-27T07:30:11.485 INFO:journalctl@ceph.mds.d.smithi052.stdout:May 27 07:30:11 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-d[107239]: 3: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x55dfad04523d] 2024-05-27T07:30:11.485 INFO:journalctl@ceph.mds.d.smithi052.stdout:May 27 07:30:11 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-d[107239]: 4: (SnapRealm::build_snap_set() const+0x323) [0x55dfad030971] 2024-05-27T07:30:11.485 INFO:journalctl@ceph.mds.d.smithi052.stdout:May 27 07:30:11 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-d[107239]: 5: (SnapRealm::check_cache() const+0x13e) [0x55dfad02efa0] 2024-05-27T07:30:11.485 INFO:journalctl@ceph.mds.d.smithi052.stdout:May 27 07:30:11 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-d[107239]: 6: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x55dfad02f651] 2024-05-27T07:30:11.485 INFO:journalctl@ceph.mds.d.smithi052.stdout:May 27 07:30:11 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-d[107239]: 7: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x55dfacfc4347] 2024-05-27T07:30:11.485 INFO:journalctl@ceph.mds.d.smithi052.stdout:May 27 07:30:11 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-d[107239]: 8: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x55dfaccc087e] 2024-05-27T07:30:11.485 INFO:journalctl@ceph.mds.d.smithi052.stdout:May 27 07:30:11 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-d[107239]: 9: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x2702) [0x55dfaccc5e94] 2024-05-27T07:30:11.485 INFO:journalctl@ceph.mds.d.smithi052.stdout:May 27 07:30:11 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-d[107239]: 10: (EUpdate::replay(MDSRank*)+0x63) [0x55dfaccd0969] 2024-05-27T07:30:11.486 INFO:journalctl@ceph.mds.d.smithi052.stdout:May 27 07:30:11 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-d[107239]: 11: (MDLog::_replay_thread()+0x1f38) [0x55dfad06948c] 2024-05-27T07:30:11.486 INFO:journalctl@ceph.mds.d.smithi052.stdout:May 27 07:30:11 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-d[107239]: 12: (MDLog::ReplayThread::entry()+0x11) [0x55dfad0729cb] 2024-05-27T07:30:11.486 INFO:journalctl@ceph.mds.d.smithi052.stdout:May 27 07:30:11 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-d[107239]: 13: (Thread::entry_wrapper()+0x43) [0x7f496dabe72d] 2024-05-27T07:30:11.486 INFO:journalctl@ceph.mds.d.smithi052.stdout:May 27 07:30:11 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-d[107239]: 14: (Thread::_entry_func(void*)+0xd) [0x7f496dabe749] 2024-05-27T07:30:11.486 INFO:journalctl@ceph.mds.d.smithi052.stdout:May 27 07:30:11 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-d[107239]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7f496c6621ca] 2024-05-27T07:30:11.486 INFO:journalctl@ceph.mds.d.smithi052.stdout:May 27 07:30:11 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-d[107239]: 16: clone() 2024-05-27T07:30:11.486 INFO:journalctl@ceph.mds.d.smithi052.stdout:May 27 07:30:11 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-d[107239]: 2024-05-27T07:30:11.486 INFO:journalctl@ceph.mds.d.smithi052.stdout:May 27 07:30:11 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-d[107239]: *** Caught signal (Aborted) ** 2024-05-27T07:30:11.486 INFO:journalctl@ceph.mds.d.smithi052.stdout:May 27 07:30:11 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-d[107239]: in thread 7f495b2aa700 thread_name:md_log_replay 2024-05-27T07:30:11.486 INFO:journalctl@ceph.mds.d.smithi052.stdout:May 27 07:30:11 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-d[107239]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:30:11.486 INFO:journalctl@ceph.mds.d.smithi052.stdout:May 27 07:30:11 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-d[107239]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f496c66cd20] 2024-05-27T07:30:11.487 INFO:journalctl@ceph.mds.d.smithi052.stdout:May 27 07:30:11 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-d[107239]: 2: gsignal() 2024-05-27T07:30:11.487 INFO:journalctl@ceph.mds.d.smithi052.stdout:May 27 07:30:11 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-d[107239]: 3: abort() 2024-05-27T07:30:11.487 INFO:journalctl@ceph.mds.d.smithi052.stdout:May 27 07:30:11 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-d[107239]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x26e) [0x7f496daefced] 2024-05-27T07:30:11.487 INFO:journalctl@ceph.mds.d.smithi052.stdout:May 27 07:30:11 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-d[107239]: 5: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7f496daefdd1] 2024-05-27T07:30:11.487 INFO:journalctl@ceph.mds.d.smithi052.stdout:May 27 07:30:11 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-d[107239]: 6: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x55dfad04523d] 2024-05-27T07:30:11.487 INFO:journalctl@ceph.mds.d.smithi052.stdout:May 27 07:30:11 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-d[107239]: 7: (SnapRealm::build_snap_set() const+0x323) [0x55dfad030971] 2024-05-27T07:30:11.487 INFO:journalctl@ceph.mds.d.smithi052.stdout:May 27 07:30:11 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-d[107239]: 8: (SnapRealm::check_cache() const+0x13e) [0x55dfad02efa0] 2024-05-27T07:30:11.487 INFO:journalctl@ceph.mds.d.smithi052.stdout:May 27 07:30:11 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-d[107239]: 9: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x55dfad02f651] 2024-05-27T07:30:11.487 INFO:journalctl@ceph.mds.d.smithi052.stdout:May 27 07:30:11 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-d[107239]: 10: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x55dfacfc4347] 2024-05-27T07:30:11.487 INFO:journalctl@ceph.mds.d.smithi052.stdout:May 27 07:30:11 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-d[107239]: 11: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x55dfaccc087e] 2024-05-27T07:30:11.487 INFO:journalctl@ceph.mds.d.smithi052.stdout:May 27 07:30:11 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-d[107239]: 12: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x2702) [0x55dfaccc5e94] 2024-05-27T07:30:11.487 INFO:journalctl@ceph.mds.d.smithi052.stdout:May 27 07:30:11 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-d[107239]: 13: (EUpdate::replay(MDSRank*)+0x63) [0x55dfaccd0969] 2024-05-27T07:30:11.487 INFO:journalctl@ceph.mds.d.smithi052.stdout:May 27 07:30:11 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-d[107239]: 14: (MDLog::_replay_thread()+0x1f38) [0x55dfad06948c] 2024-05-27T07:30:11.488 INFO:journalctl@ceph.mds.d.smithi052.stdout:May 27 07:30:11 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-d[107239]: 15: (MDLog::ReplayThread::entry()+0x11) [0x55dfad0729cb] 2024-05-27T07:30:11.488 INFO:journalctl@ceph.mds.d.smithi052.stdout:May 27 07:30:11 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-d[107239]: 16: (Thread::entry_wrapper()+0x43) [0x7f496dabe72d] 2024-05-27T07:30:11.488 INFO:journalctl@ceph.mds.d.smithi052.stdout:May 27 07:30:11 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-d[107239]: 17: (Thread::_entry_func(void*)+0xd) [0x7f496dabe749] 2024-05-27T07:30:11.488 INFO:journalctl@ceph.mds.d.smithi052.stdout:May 27 07:30:11 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-d[107239]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7f496c6621ca] 2024-05-27T07:30:11.488 INFO:journalctl@ceph.mds.d.smithi052.stdout:May 27 07:30:11 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-d[107239]: 19: clone() 2024-05-27T07:30:11.488 INFO:journalctl@ceph.mds.d.smithi052.stdout:May 27 07:30:11 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-d[107239]: 2024-05-27T07:30:11.156+0000 7f495b2aa700 -1 *** Caught signal (Aborted) ** 2024-05-27T07:30:11.488 INFO:journalctl@ceph.mds.d.smithi052.stdout:May 27 07:30:11 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-d[107239]: in thread 7f495b2aa700 thread_name:md_log_replay 2024-05-27T07:30:11.488 INFO:journalctl@ceph.mds.d.smithi052.stdout:May 27 07:30:11 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-d[107239]: 2024-05-27T07:30:11.488 INFO:journalctl@ceph.mds.d.smithi052.stdout:May 27 07:30:11 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-d[107239]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:30:11.488 INFO:journalctl@ceph.mds.d.smithi052.stdout:May 27 07:30:11 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-d[107239]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f496c66cd20] 2024-05-27T07:30:11.488 INFO:journalctl@ceph.mds.d.smithi052.stdout:May 27 07:30:11 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-d[107239]: 2: gsignal() 2024-05-27T07:30:11.488 INFO:journalctl@ceph.mds.d.smithi052.stdout:May 27 07:30:11 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-d[107239]: 3: abort() 2024-05-27T07:30:11.488 INFO:journalctl@ceph.mds.d.smithi052.stdout:May 27 07:30:11 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-d[107239]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x26e) [0x7f496daefced] 2024-05-27T07:30:11.489 INFO:journalctl@ceph.mds.d.smithi052.stdout:May 27 07:30:11 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-d[107239]: 5: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7f496daefdd1] 2024-05-27T07:30:11.489 INFO:journalctl@ceph.mds.d.smithi052.stdout:May 27 07:30:11 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-d[107239]: 6: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x55dfad04523d] 2024-05-27T07:30:11.489 INFO:journalctl@ceph.mds.d.smithi052.stdout:May 27 07:30:11 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-d[107239]: 7: (SnapRealm::build_snap_set() const+0x323) [0x55dfad030971] 2024-05-27T07:30:11.489 INFO:journalctl@ceph.mds.d.smithi052.stdout:May 27 07:30:11 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-d[107239]: 8: (SnapRealm::check_cache() const+0x13e) [0x55dfad02efa0] 2024-05-27T07:30:11.489 INFO:journalctl@ceph.mds.d.smithi052.stdout:May 27 07:30:11 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-d[107239]: 9: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x55dfad02f651] 2024-05-27T07:30:11.490 INFO:journalctl@ceph.mds.d.smithi052.stdout:May 27 07:30:11 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-d[107239]: 10: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x55dfacfc4347] 2024-05-27T07:30:11.490 INFO:journalctl@ceph.mds.d.smithi052.stdout:May 27 07:30:11 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-d[107239]: 11: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x55dfaccc087e] 2024-05-27T07:30:11.490 INFO:journalctl@ceph.mds.d.smithi052.stdout:May 27 07:30:11 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-d[107239]: 12: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x2702) [0x55dfaccc5e94] 2024-05-27T07:30:11.490 INFO:journalctl@ceph.mds.d.smithi052.stdout:May 27 07:30:11 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-d[107239]: 13: (EUpdate::replay(MDSRank*)+0x63) [0x55dfaccd0969] 2024-05-27T07:30:11.490 INFO:journalctl@ceph.mds.d.smithi052.stdout:May 27 07:30:11 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-d[107239]: 14: (MDLog::_replay_thread()+0x1f38) [0x55dfad06948c] 2024-05-27T07:30:11.490 INFO:journalctl@ceph.mds.d.smithi052.stdout:May 27 07:30:11 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-d[107239]: 15: (MDLog::ReplayThread::entry()+0x11) [0x55dfad0729cb] 2024-05-27T07:30:11.490 INFO:journalctl@ceph.mds.d.smithi052.stdout:May 27 07:30:11 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-d[107239]: 16: (Thread::entry_wrapper()+0x43) [0x7f496dabe72d] 2024-05-27T07:30:11.490 INFO:journalctl@ceph.mds.d.smithi052.stdout:May 27 07:30:11 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-d[107239]: 17: (Thread::_entry_func(void*)+0xd) [0x7f496dabe749] 2024-05-27T07:30:11.490 INFO:journalctl@ceph.mds.d.smithi052.stdout:May 27 07:30:11 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-d[107239]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7f496c6621ca] 2024-05-27T07:30:11.490 INFO:journalctl@ceph.mds.d.smithi052.stdout:May 27 07:30:11 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-d[107239]: 19: clone() 2024-05-27T07:30:11.490 INFO:journalctl@ceph.mds.d.smithi052.stdout:May 27 07:30:11 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-d[107239]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-05-27T07:30:11.490 INFO:journalctl@ceph.mds.d.smithi052.stdout:May 27 07:30:11 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-d[107239]: 2024-05-27T07:30:11.491 INFO:journalctl@ceph.mds.d.smithi052.stdout:May 27 07:30:11 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-d[107239]: -1> 2024-05-27T07:30:11.149+0000 7f495b2aa700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: In function 'std::set SnapClient::filter(const std::set&) const' thread 7f495b2aa700 time 2024-05-27T07:30:11.144715+0000 2024-05-27T07:30:11.491 INFO:journalctl@ceph.mds.d.smithi052.stdout:May 27 07:30:11 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-d[107239]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: 197: FAILED ceph_assert(cached_version > 0) 2024-05-27T07:30:11.491 INFO:journalctl@ceph.mds.d.smithi052.stdout:May 27 07:30:11 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-d[107239]: 2024-05-27T07:30:11.491 INFO:journalctl@ceph.mds.d.smithi052.stdout:May 27 07:30:11 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-d[107239]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:30:11.491 INFO:journalctl@ceph.mds.d.smithi052.stdout:May 27 07:30:11 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-d[107239]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x127) [0x7f496daefba6] 2024-05-27T07:30:11.491 INFO:journalctl@ceph.mds.d.smithi052.stdout:May 27 07:30:11 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-d[107239]: 2: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7f496daefdd1] 2024-05-27T07:30:11.491 INFO:journalctl@ceph.mds.d.smithi052.stdout:May 27 07:30:11 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-d[107239]: 3: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x55dfad04523d] 2024-05-27T07:30:11.491 INFO:journalctl@ceph.mds.d.smithi052.stdout:May 27 07:30:11 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-d[107239]: 4: (SnapRealm::build_snap_set() const+0x323) [0x55dfad030971] 2024-05-27T07:30:11.491 INFO:journalctl@ceph.mds.d.smithi052.stdout:May 27 07:30:11 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-d[107239]: 5: (SnapRealm::check_cache() const+0x13e) [0x55dfad02efa0] 2024-05-27T07:30:11.491 INFO:journalctl@ceph.mds.d.smithi052.stdout:May 27 07:30:11 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-d[107239]: 6: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x55dfad02f651] 2024-05-27T07:30:11.491 INFO:journalctl@ceph.mds.d.smithi052.stdout:May 27 07:30:11 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-d[107239]: 7: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x55dfacfc4347] 2024-05-27T07:30:11.491 INFO:journalctl@ceph.mds.d.smithi052.stdout:May 27 07:30:11 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-d[107239]: 8: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x55dfaccc087e] 2024-05-27T07:30:11.492 INFO:journalctl@ceph.mds.d.smithi052.stdout:May 27 07:30:11 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-d[107239]: 9: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x2702) [0x55dfaccc5e94] 2024-05-27T07:30:11.492 INFO:journalctl@ceph.mds.d.smithi052.stdout:May 27 07:30:11 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-d[107239]: 10: (EUpdate::replay(MDSRank*)+0x63) [0x55dfaccd0969] 2024-05-27T07:30:11.492 INFO:journalctl@ceph.mds.d.smithi052.stdout:May 27 07:30:11 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-d[107239]: 11: (MDLog::_replay_thread()+0x1f38) [0x55dfad06948c] 2024-05-27T07:30:11.492 INFO:journalctl@ceph.mds.d.smithi052.stdout:May 27 07:30:11 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-d[107239]: 12: (MDLog::ReplayThread::entry()+0x11) [0x55dfad0729cb] 2024-05-27T07:30:11.492 INFO:journalctl@ceph.mds.d.smithi052.stdout:May 27 07:30:11 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-d[107239]: 13: (Thread::entry_wrapper()+0x43) [0x7f496dabe72d] 2024-05-27T07:30:11.492 INFO:journalctl@ceph.mds.d.smithi052.stdout:May 27 07:30:11 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-d[107239]: 14: (Thread::_entry_func(void*)+0xd) [0x7f496dabe749] 2024-05-27T07:30:11.492 INFO:journalctl@ceph.mds.d.smithi052.stdout:May 27 07:30:11 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-d[107239]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7f496c6621ca] 2024-05-27T07:30:11.492 INFO:journalctl@ceph.mds.d.smithi052.stdout:May 27 07:30:11 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-d[107239]: 16: clone() 2024-05-27T07:30:11.492 INFO:journalctl@ceph.mds.d.smithi052.stdout:May 27 07:30:11 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-d[107239]: 2024-05-27T07:30:11.492 INFO:journalctl@ceph.mds.d.smithi052.stdout:May 27 07:30:11 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-d[107239]: 0> 2024-05-27T07:30:11.156+0000 7f495b2aa700 -1 *** Caught signal (Aborted) ** 2024-05-27T07:30:11.492 INFO:journalctl@ceph.mds.d.smithi052.stdout:May 27 07:30:11 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-d[107239]: in thread 7f495b2aa700 thread_name:md_log_replay 2024-05-27T07:30:11.492 INFO:journalctl@ceph.mds.d.smithi052.stdout:May 27 07:30:11 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-d[107239]: 2024-05-27T07:30:11.493 INFO:journalctl@ceph.mds.d.smithi052.stdout:May 27 07:30:11 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-d[107239]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:30:11.493 INFO:journalctl@ceph.mds.d.smithi052.stdout:May 27 07:30:11 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-d[107239]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f496c66cd20] 2024-05-27T07:30:11.493 INFO:journalctl@ceph.mds.d.smithi052.stdout:May 27 07:30:11 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-d[107239]: 2: gsignal() 2024-05-27T07:30:11.493 INFO:journalctl@ceph.mds.d.smithi052.stdout:May 27 07:30:11 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-d[107239]: 3: abort() 2024-05-27T07:30:11.493 INFO:journalctl@ceph.mds.d.smithi052.stdout:May 27 07:30:11 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-d[107239]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x26e) [0x7f496daefced] 2024-05-27T07:30:11.493 INFO:journalctl@ceph.mds.d.smithi052.stdout:May 27 07:30:11 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-d[107239]: 5: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7f496daefdd1] 2024-05-27T07:30:11.493 INFO:journalctl@ceph.mds.d.smithi052.stdout:May 27 07:30:11 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-d[107239]: 6: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x55dfad04523d] 2024-05-27T07:30:11.493 INFO:journalctl@ceph.mds.d.smithi052.stdout:May 27 07:30:11 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-d[107239]: 7: (SnapRealm::build_snap_set() const+0x323) [0x55dfad030971] 2024-05-27T07:30:11.493 INFO:journalctl@ceph.mds.d.smithi052.stdout:May 27 07:30:11 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-d[107239]: 8: (SnapRealm::check_cache() const+0x13e) [0x55dfad02efa0] 2024-05-27T07:30:11.493 INFO:journalctl@ceph.mds.d.smithi052.stdout:May 27 07:30:11 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-d[107239]: 9: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x55dfad02f651] 2024-05-27T07:30:11.493 INFO:journalctl@ceph.mds.d.smithi052.stdout:May 27 07:30:11 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-d[107239]: 10: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x55dfacfc4347] 2024-05-27T07:30:11.493 INFO:journalctl@ceph.mds.d.smithi052.stdout:May 27 07:30:11 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-d[107239]: 11: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x55dfaccc087e] 2024-05-27T07:30:11.494 INFO:journalctl@ceph.mds.d.smithi052.stdout:May 27 07:30:11 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-d[107239]: 12: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x2702) [0x55dfaccc5e94] 2024-05-27T07:30:11.494 INFO:journalctl@ceph.mds.d.smithi052.stdout:May 27 07:30:11 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-d[107239]: 13: (EUpdate::replay(MDSRank*)+0x63) [0x55dfaccd0969] 2024-05-27T07:30:11.494 INFO:journalctl@ceph.mds.d.smithi052.stdout:May 27 07:30:11 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-d[107239]: 14: (MDLog::_replay_thread()+0x1f38) [0x55dfad06948c] 2024-05-27T07:30:11.494 INFO:journalctl@ceph.mds.d.smithi052.stdout:May 27 07:30:11 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-d[107239]: 15: (MDLog::ReplayThread::entry()+0x11) [0x55dfad0729cb] 2024-05-27T07:30:11.494 INFO:journalctl@ceph.mds.d.smithi052.stdout:May 27 07:30:11 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-d[107239]: 16: (Thread::entry_wrapper()+0x43) [0x7f496dabe72d] 2024-05-27T07:30:11.494 INFO:journalctl@ceph.mds.d.smithi052.stdout:May 27 07:30:11 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-d[107239]: 17: (Thread::_entry_func(void*)+0xd) [0x7f496dabe749] 2024-05-27T07:30:11.494 INFO:journalctl@ceph.mds.d.smithi052.stdout:May 27 07:30:11 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-d[107239]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7f496c6621ca] 2024-05-27T07:30:11.494 INFO:journalctl@ceph.mds.d.smithi052.stdout:May 27 07:30:11 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-d[107239]: 19: clone() 2024-05-27T07:30:11.494 INFO:journalctl@ceph.mds.d.smithi052.stdout:May 27 07:30:11 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-d[107239]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-05-27T07:30:11.494 INFO:journalctl@ceph.mds.d.smithi052.stdout:May 27 07:30:11 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-d[107239]: 2024-05-27T07:30:11.494 INFO:journalctl@ceph.mds.d.smithi052.stdout:May 27 07:30:11 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-d[107239]: -1> 2024-05-27T07:30:11.149+0000 7f495b2aa700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: In function 'std::set SnapClient::filter(const std::set&) const' thread 7f495b2aa700 time 2024-05-27T07:30:11.144715+0000 2024-05-27T07:30:11.494 INFO:journalctl@ceph.mds.d.smithi052.stdout:May 27 07:30:11 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-d[107239]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: 197: FAILED ceph_assert(cached_version > 0) 2024-05-27T07:30:11.495 INFO:journalctl@ceph.mds.d.smithi052.stdout:May 27 07:30:11 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-d[107239]: 2024-05-27T07:30:11.495 INFO:journalctl@ceph.mds.d.smithi052.stdout:May 27 07:30:11 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-d[107239]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:30:11.495 INFO:journalctl@ceph.mds.d.smithi052.stdout:May 27 07:30:11 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-d[107239]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x127) [0x7f496daefba6] 2024-05-27T07:30:11.495 INFO:journalctl@ceph.mds.d.smithi052.stdout:May 27 07:30:11 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-d[107239]: 2: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7f496daefdd1] 2024-05-27T07:30:11.495 INFO:journalctl@ceph.mds.d.smithi052.stdout:May 27 07:30:11 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-d[107239]: 3: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x55dfad04523d] 2024-05-27T07:30:11.495 INFO:journalctl@ceph.mds.d.smithi052.stdout:May 27 07:30:11 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-d[107239]: 4: (SnapRealm::build_snap_set() const+0x323) [0x55dfad030971] 2024-05-27T07:30:11.495 INFO:journalctl@ceph.mds.d.smithi052.stdout:May 27 07:30:11 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-d[107239]: 5: (SnapRealm::check_cache() const+0x13e) [0x55dfad02efa0] 2024-05-27T07:30:11.495 INFO:journalctl@ceph.mds.d.smithi052.stdout:May 27 07:30:11 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-d[107239]: 6: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x55dfad02f651] 2024-05-27T07:30:11.495 INFO:journalctl@ceph.mds.d.smithi052.stdout:May 27 07:30:11 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-d[107239]: 7: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x55dfacfc4347] 2024-05-27T07:30:11.495 INFO:journalctl@ceph.mds.d.smithi052.stdout:May 27 07:30:11 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-d[107239]: 8: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x55dfaccc087e] 2024-05-27T07:30:11.495 INFO:journalctl@ceph.mds.d.smithi052.stdout:May 27 07:30:11 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-d[107239]: 9: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x2702) [0x55dfaccc5e94] 2024-05-27T07:30:11.495 INFO:journalctl@ceph.mds.d.smithi052.stdout:May 27 07:30:11 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-d[107239]: 10: (EUpdate::replay(MDSRank*)+0x63) [0x55dfaccd0969] 2024-05-27T07:30:11.495 INFO:journalctl@ceph.mds.d.smithi052.stdout:May 27 07:30:11 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-d[107239]: 11: (MDLog::_replay_thread()+0x1f38) [0x55dfad06948c] 2024-05-27T07:30:11.496 INFO:journalctl@ceph.mds.d.smithi052.stdout:May 27 07:30:11 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-d[107239]: 12: (MDLog::ReplayThread::entry()+0x11) [0x55dfad0729cb] 2024-05-27T07:30:11.496 INFO:journalctl@ceph.mds.d.smithi052.stdout:May 27 07:30:11 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-d[107239]: 13: (Thread::entry_wrapper()+0x43) [0x7f496dabe72d] 2024-05-27T07:30:11.496 INFO:journalctl@ceph.mds.d.smithi052.stdout:May 27 07:30:11 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-d[107239]: 14: (Thread::_entry_func(void*)+0xd) [0x7f496dabe749] 2024-05-27T07:30:11.496 INFO:journalctl@ceph.mds.d.smithi052.stdout:May 27 07:30:11 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-d[107239]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7f496c6621ca] 2024-05-27T07:30:11.496 INFO:journalctl@ceph.mds.d.smithi052.stdout:May 27 07:30:11 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-d[107239]: 16: clone() 2024-05-27T07:30:11.496 INFO:journalctl@ceph.mds.d.smithi052.stdout:May 27 07:30:11 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-d[107239]: 2024-05-27T07:30:11.496 INFO:journalctl@ceph.mds.d.smithi052.stdout:May 27 07:30:11 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-d[107239]: 0> 2024-05-27T07:30:11.156+0000 7f495b2aa700 -1 *** Caught signal (Aborted) ** 2024-05-27T07:30:11.496 INFO:journalctl@ceph.mds.d.smithi052.stdout:May 27 07:30:11 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-d[107239]: in thread 7f495b2aa700 thread_name:md_log_replay 2024-05-27T07:30:11.496 INFO:journalctl@ceph.mds.d.smithi052.stdout:May 27 07:30:11 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-d[107239]: 2024-05-27T07:30:11.496 INFO:journalctl@ceph.mds.d.smithi052.stdout:May 27 07:30:11 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-d[107239]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:30:11.497 INFO:journalctl@ceph.mds.d.smithi052.stdout:May 27 07:30:11 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-d[107239]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f496c66cd20] 2024-05-27T07:30:11.497 INFO:journalctl@ceph.mds.d.smithi052.stdout:May 27 07:30:11 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-d[107239]: 2: gsignal() 2024-05-27T07:30:11.497 INFO:journalctl@ceph.mds.d.smithi052.stdout:May 27 07:30:11 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-d[107239]: 3: abort() 2024-05-27T07:30:11.497 INFO:journalctl@ceph.mds.d.smithi052.stdout:May 27 07:30:11 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-d[107239]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x26e) [0x7f496daefced] 2024-05-27T07:30:11.497 INFO:journalctl@ceph.mds.d.smithi052.stdout:May 27 07:30:11 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-d[107239]: 5: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7f496daefdd1] 2024-05-27T07:30:11.497 INFO:journalctl@ceph.mds.d.smithi052.stdout:May 27 07:30:11 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-d[107239]: 6: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x55dfad04523d] 2024-05-27T07:30:11.497 INFO:journalctl@ceph.mds.d.smithi052.stdout:May 27 07:30:11 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-d[107239]: 7: (SnapRealm::build_snap_set() const+0x323) [0x55dfad030971] 2024-05-27T07:30:11.497 INFO:journalctl@ceph.mds.d.smithi052.stdout:May 27 07:30:11 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-d[107239]: 8: (SnapRealm::check_cache() const+0x13e) [0x55dfad02efa0] 2024-05-27T07:30:11.497 INFO:journalctl@ceph.mds.d.smithi052.stdout:May 27 07:30:11 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-d[107239]: 9: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x55dfad02f651] 2024-05-27T07:30:11.497 INFO:journalctl@ceph.mds.d.smithi052.stdout:May 27 07:30:11 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-d[107239]: 10: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x55dfacfc4347] 2024-05-27T07:30:11.497 INFO:journalctl@ceph.mds.d.smithi052.stdout:May 27 07:30:11 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-d[107239]: 11: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x55dfaccc087e] 2024-05-27T07:30:11.497 INFO:journalctl@ceph.mds.d.smithi052.stdout:May 27 07:30:11 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-d[107239]: 12: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x2702) [0x55dfaccc5e94] 2024-05-27T07:30:11.497 INFO:journalctl@ceph.mds.d.smithi052.stdout:May 27 07:30:11 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-d[107239]: 13: (EUpdate::replay(MDSRank*)+0x63) [0x55dfaccd0969] 2024-05-27T07:30:11.498 INFO:journalctl@ceph.mds.d.smithi052.stdout:May 27 07:30:11 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-d[107239]: 14: (MDLog::_replay_thread()+0x1f38) [0x55dfad06948c] 2024-05-27T07:30:11.498 INFO:journalctl@ceph.mds.d.smithi052.stdout:May 27 07:30:11 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-d[107239]: 15: (MDLog::ReplayThread::entry()+0x11) [0x55dfad0729cb] 2024-05-27T07:30:11.498 INFO:journalctl@ceph.mds.d.smithi052.stdout:May 27 07:30:11 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-d[107239]: 16: (Thread::entry_wrapper()+0x43) [0x7f496dabe72d] 2024-05-27T07:30:11.498 INFO:journalctl@ceph.mds.d.smithi052.stdout:May 27 07:30:11 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-d[107239]: 17: (Thread::_entry_func(void*)+0xd) [0x7f496dabe749] 2024-05-27T07:30:11.498 INFO:journalctl@ceph.mds.d.smithi052.stdout:May 27 07:30:11 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-d[107239]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7f496c6621ca] 2024-05-27T07:30:11.498 INFO:journalctl@ceph.mds.d.smithi052.stdout:May 27 07:30:11 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-d[107239]: 19: clone() 2024-05-27T07:30:11.498 INFO:journalctl@ceph.mds.d.smithi052.stdout:May 27 07:30:11 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-d[107239]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-05-27T07:30:11.498 INFO:journalctl@ceph.mds.d.smithi052.stdout:May 27 07:30:11 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-d[107239]: 2024-05-27T07:30:11.852 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:30:11 smithi173 ceph-mon[37000]: pgmap v151: 129 pgs: 129 active+clean; 323 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 MiB/s rd, 7.9 MiB/s wr, 754 op/s 2024-05-27T07:30:11.941 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:30:11 smithi105 ceph-mon[35977]: pgmap v151: 129 pgs: 129 active+clean; 323 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 MiB/s rd, 7.9 MiB/s wr, 754 op/s 2024-05-27T07:30:11.980 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:30:11 smithi052 ceph-mon[31955]: pgmap v151: 129 pgs: 129 active+clean; 323 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 MiB/s rd, 7.9 MiB/s wr, 754 op/s 2024-05-27T07:30:11.981 INFO:journalctl@ceph.mds.a.smithi052.stdout:May 27 07:30:11 smithi052 podman[107335]: 2024-05-27 07:30:11.686441482 +0000 UTC m=+1.081831455 container remove e5f4c04c08e94f426d7470f4fb1ace3e1702e3663fcdc999a50fd8cdce849ac9 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-a, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240521, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, GIT_CLEAN=True, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0) 2024-05-27T07:30:11.981 INFO:journalctl@ceph.mds.a.smithi052.stdout:May 27 07:30:11 smithi052 systemd[1]: ceph-73df2636-1bf8-11ef-bc9b-c7b262605968@mds.a.service: Main process exited, code=exited, status=134/n/a 2024-05-27T07:30:11.981 INFO:journalctl@ceph.mds.d.smithi052.stdout:May 27 07:30:11 smithi052 podman[107357]: 2024-05-27 07:30:11.715454588 +0000 UTC m=+0.505927057 container died 0f66c93719468a265ba834a48f395ac973351eab29410ab9f063e4c1561b9e34 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-d, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, CEPH_POINT_RELEASE=, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, org.label-schema.vendor=CentOS, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240521) 2024-05-27T07:30:12.853 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:30:12 smithi173 ceph-mon[37000]: scrub summary: active paths [/] 2024-05-27T07:30:12.941 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:30:12 smithi105 ceph-mon[35977]: scrub summary: active paths [/] 2024-05-27T07:30:12.941 INFO:journalctl@ceph.mds.b.smithi105.stdout:May 27 07:30:12 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-b[65543]: 2024-05-27T07:30:12.722+0000 7f33fced0700 -1 mds.0.scrubstack _validate_inode_done scrub error on inode [inode 0x10000003fad [...6,head] /volumes/qa/sv_1/95e9ac73-825f-48e3-998b-133d94d96559/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-58/ auth{4=1} v189176 ap=2 RANDEPHEMERALPIN f(v3 m2024-05-27T07:30:04.935831+0000 75=75+0) n(v3 rc2024-05-27T07:30:04.935831+0000 b1945549 76=75+1) old_inodes=1 (inest sync->mix g=4) caps={25060=pAsLsXsFs/pAsLsXsFsx@209} | dirtyscattered=0 request=0 lock=0 dirfrag=1 caps=1 dirtyparent=0 replicated=1 dirty=0 waiter=0 authpin=1 scrubqueue=0 randepin 0x55ee54bf8b00]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":true,"read_ret_val":0,"ondisk_value":"(2)0x10000003fad:[<0x10000000399/blog-58 v152699>,<0x10000000208/blogtest_in v814>,<0x10000000201/src v2182>,<0x10000000006/blogbench-1.0 v1931>,<0x10000000005/tmp v231>,<0x100000001fb/client.0 v208>,<0x100000001fa/95e9ac73-825f-48e3-998b-133d94d96559 v218>,<0x10000000001/sv_1 v230>,<0x10000000000/qa v222>,<0x1/volumes v215>]//[]","memoryvalue":"(2)0x10000003fad:[<0x10000000399/blog-58 v189176>,<0x10000000208/blogtest_in v953>,<0x10000000201/src v2231>,<0x10000000006/blogbench-1.0 v1977>,<0x10000000005/tmp v277>,<0x100000001fb/client.0 v247>,<0x100000001fa/95e9ac73-825f-48e3-998b-133d94d96559 v260>,<0x10000000001/sv_1 v271>,<0x10000000000/qa v261>,<0x1/volumes v257>]//[]","error_str":""},"raw_stats":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f(v0 m2024-05-27T07:29:42.380092+0000 73=73+0)","ondisk_value.rstat":"n(v0 rc2024-05-27T07:30:04.935831+0000 b1945549 76=75+1)","memory_value.dirstat":"f(v3 m2024-05-27T07:30:04.935831+0000 75=75+0)","memory_value.rstat":"n(v3 rc2024-05-27T07:30:04.935831+0000 b1945549 76=75+1)","error_str":"freshly-calculated rstats don't match existing ones"},"return_code":0} 2024-05-27T07:30:12.984 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:30:12 smithi052 ceph-mon[31955]: scrub summary: active paths [/] 2024-05-27T07:30:13.854 INFO:journalctl@ceph.mds.l.smithi173.stdout:May 27 07:30:13 smithi173 systemd[1]: ceph-73df2636-1bf8-11ef-bc9b-c7b262605968@mds.l.service: Scheduled restart job, restart counter is at 6. 2024-05-27T07:30:13.854 INFO:journalctl@ceph.mds.l.smithi173.stdout:May 27 07:30:13 smithi173 systemd[1]: Stopped Ceph mds.l for 73df2636-1bf8-11ef-bc9b-c7b262605968. 2024-05-27T07:30:13.854 INFO:journalctl@ceph.mds.l.smithi173.stdout:May 27 07:30:13 smithi173 systemd[1]: ceph-73df2636-1bf8-11ef-bc9b-c7b262605968@mds.l.service: Start request repeated too quickly. 2024-05-27T07:30:13.855 INFO:journalctl@ceph.mds.l.smithi173.stdout:May 27 07:30:13 smithi173 systemd[1]: ceph-73df2636-1bf8-11ef-bc9b-c7b262605968@mds.l.service: Failed with result 'exit-code'. 2024-05-27T07:30:13.855 INFO:journalctl@ceph.mds.l.smithi173.stdout:May 27 07:30:13 smithi173 systemd[1]: Failed to start Ceph mds.l for 73df2636-1bf8-11ef-bc9b-c7b262605968. 2024-05-27T07:30:13.855 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:30:13 smithi173 ceph-mon[37000]: scrub summary: idle+waiting paths [/] 2024-05-27T07:30:13.855 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:30:13 smithi173 ceph-mon[37000]: scrub summary: active paths [/] 2024-05-27T07:30:13.855 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:30:13 smithi173 ceph-mon[37000]: scrub summary: idle+waiting paths [/] 2024-05-27T07:30:13.855 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:30:13 smithi173 ceph-mon[37000]: scrub summary: active paths [/] 2024-05-27T07:30:13.855 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:30:13 smithi173 ceph-mon[37000]: scrub summary: idle+waiting paths [/] 2024-05-27T07:30:13.855 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:30:13 smithi173 ceph-mon[37000]: scrub summary: active paths [/] 2024-05-27T07:30:13.855 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:30:13 smithi173 ceph-mon[37000]: scrub summary: idle+waiting paths [/] 2024-05-27T07:30:13.855 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:30:13 smithi173 ceph-mon[37000]: scrub summary: active paths [/] 2024-05-27T07:30:13.855 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:30:13 smithi173 ceph-mon[37000]: scrub summary: active paths [/] 2024-05-27T07:30:13.855 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:30:13 smithi173 ceph-mon[37000]: scrub summary: idle+waiting paths [/] 2024-05-27T07:30:13.856 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:30:13 smithi173 ceph-mon[37000]: scrub summary: active paths [/] 2024-05-27T07:30:13.856 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:30:13 smithi173 ceph-mon[37000]: scrub summary: idle+waiting paths [/] 2024-05-27T07:30:13.856 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:30:13 smithi173 ceph-mon[37000]: scrub summary: idle+waiting paths [/] 2024-05-27T07:30:13.856 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:30:13 smithi173 ceph-mon[37000]: scrub summary: active paths [/] 2024-05-27T07:30:13.856 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:30:13 smithi173 ceph-mon[37000]: scrub summary: idle+waiting paths [/] 2024-05-27T07:30:13.856 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:30:13 smithi173 ceph-mon[37000]: pgmap v152: 129 pgs: 129 active+clean; 325 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 MiB/s rd, 7.3 MiB/s wr, 840 op/s 2024-05-27T07:30:13.925 INFO:teuthology.orchestra.run.smithi052.stderr:Inferring config /var/lib/ceph/73df2636-1bf8-11ef-bc9b-c7b262605968/mon.a/config 2024-05-27T07:30:13.941 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:30:13 smithi105 ceph-mon[35977]: scrub summary: idle+waiting paths [/] 2024-05-27T07:30:13.941 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:30:13 smithi105 ceph-mon[35977]: scrub summary: active paths [/] 2024-05-27T07:30:13.941 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:30:13 smithi105 ceph-mon[35977]: scrub summary: idle+waiting paths [/] 2024-05-27T07:30:13.941 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:30:13 smithi105 ceph-mon[35977]: scrub summary: active paths [/] 2024-05-27T07:30:13.941 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:30:13 smithi105 ceph-mon[35977]: scrub summary: idle+waiting paths [/] 2024-05-27T07:30:13.941 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:30:13 smithi105 ceph-mon[35977]: scrub summary: active paths [/] 2024-05-27T07:30:13.941 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:30:13 smithi105 ceph-mon[35977]: scrub summary: idle+waiting paths [/] 2024-05-27T07:30:13.941 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:30:13 smithi105 ceph-mon[35977]: scrub summary: active paths [/] 2024-05-27T07:30:13.941 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:30:13 smithi105 ceph-mon[35977]: scrub summary: active paths [/] 2024-05-27T07:30:13.942 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:30:13 smithi105 ceph-mon[35977]: scrub summary: idle+waiting paths [/] 2024-05-27T07:30:13.942 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:30:13 smithi105 ceph-mon[35977]: scrub summary: active paths [/] 2024-05-27T07:30:13.942 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:30:13 smithi105 ceph-mon[35977]: scrub summary: idle+waiting paths [/] 2024-05-27T07:30:13.942 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:30:13 smithi105 ceph-mon[35977]: scrub summary: idle+waiting paths [/] 2024-05-27T07:30:13.942 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:30:13 smithi105 ceph-mon[35977]: scrub summary: active paths [/] 2024-05-27T07:30:13.942 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:30:13 smithi105 ceph-mon[35977]: scrub summary: idle+waiting paths [/] 2024-05-27T07:30:13.942 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:30:13 smithi105 ceph-mon[35977]: pgmap v152: 129 pgs: 129 active+clean; 325 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 MiB/s rd, 7.3 MiB/s wr, 840 op/s 2024-05-27T07:30:13.981 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:30:13 smithi052 ceph-mon[31955]: scrub summary: idle+waiting paths [/] 2024-05-27T07:30:13.981 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:30:13 smithi052 ceph-mon[31955]: scrub summary: active paths [/] 2024-05-27T07:30:13.981 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:30:13 smithi052 ceph-mon[31955]: scrub summary: idle+waiting paths [/] 2024-05-27T07:30:13.981 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:30:13 smithi052 ceph-mon[31955]: scrub summary: active paths [/] 2024-05-27T07:30:13.981 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:30:13 smithi052 ceph-mon[31955]: scrub summary: idle+waiting paths [/] 2024-05-27T07:30:13.981 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:30:13 smithi052 ceph-mon[31955]: scrub summary: active paths [/] 2024-05-27T07:30:13.981 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:30:13 smithi052 ceph-mon[31955]: scrub summary: idle+waiting paths [/] 2024-05-27T07:30:13.981 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:30:13 smithi052 ceph-mon[31955]: scrub summary: active paths [/] 2024-05-27T07:30:13.981 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:30:13 smithi052 ceph-mon[31955]: scrub summary: active paths [/] 2024-05-27T07:30:13.981 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:30:13 smithi052 ceph-mon[31955]: scrub summary: idle+waiting paths [/] 2024-05-27T07:30:13.981 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:30:13 smithi052 ceph-mon[31955]: scrub summary: active paths [/] 2024-05-27T07:30:13.982 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:30:13 smithi052 ceph-mon[31955]: scrub summary: idle+waiting paths [/] 2024-05-27T07:30:13.982 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:30:13 smithi052 ceph-mon[31955]: scrub summary: idle+waiting paths [/] 2024-05-27T07:30:13.982 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:30:13 smithi052 ceph-mon[31955]: scrub summary: active paths [/] 2024-05-27T07:30:13.982 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:30:13 smithi052 ceph-mon[31955]: scrub summary: idle+waiting paths [/] 2024-05-27T07:30:13.982 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:30:13 smithi052 ceph-mon[31955]: pgmap v152: 129 pgs: 129 active+clean; 325 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 MiB/s rd, 7.3 MiB/s wr, 840 op/s 2024-05-27T07:30:13.984 INFO:journalctl@ceph.mds.d.smithi052.stdout:May 27 07:30:13 smithi052 podman[107357]: 2024-05-27 07:30:13.622581133 +0000 UTC m=+2.413053574 container remove 0f66c93719468a265ba834a48f395ac973351eab29410ab9f063e4c1561b9e34 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-d, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , GIT_CLEAN=True, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, org.label-schema.build-date=20240521, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-05-27T07:30:13.984 INFO:journalctl@ceph.mds.d.smithi052.stdout:May 27 07:30:13 smithi052 systemd[1]: ceph-73df2636-1bf8-11ef-bc9b-c7b262605968@mds.d.service: Main process exited, code=exited, status=134/n/a 2024-05-27T07:30:14.853 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:30:14 smithi173 ceph-mon[37000]: scrub summary: active paths [/] 2024-05-27T07:30:14.853 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:30:14 smithi173 ceph-mon[37000]: Scrub error on inode 0x10000003fad (/volumes/qa/sv_1/95e9ac73-825f-48e3-998b-133d94d96559/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-58) see mds.b log and `damage ls` output for details 2024-05-27T07:30:14.853 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:30:14 smithi173 ceph-mon[37000]: scrub summary: idle+waiting paths [/] 2024-05-27T07:30:14.860 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:30:14 smithi052 ceph-mon[31955]: scrub summary: active paths [/] 2024-05-27T07:30:14.860 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:30:14 smithi052 ceph-mon[31955]: Scrub error on inode 0x10000003fad (/volumes/qa/sv_1/95e9ac73-825f-48e3-998b-133d94d96559/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-58) see mds.b log and `damage ls` output for details 2024-05-27T07:30:14.860 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:30:14 smithi052 ceph-mon[31955]: scrub summary: idle+waiting paths [/] 2024-05-27T07:30:14.941 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:30:14 smithi105 ceph-mon[35977]: scrub summary: active paths [/] 2024-05-27T07:30:14.941 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:30:14 smithi105 ceph-mon[35977]: Scrub error on inode 0x10000003fad (/volumes/qa/sv_1/95e9ac73-825f-48e3-998b-133d94d96559/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-58) see mds.b log and `damage ls` output for details 2024-05-27T07:30:14.941 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:30:14 smithi105 ceph-mon[35977]: scrub summary: idle+waiting paths [/] 2024-05-27T07:30:15.208 INFO:tasks.workunit.client.0.smithi052.stdout: 83 1015054 376 689358 320 596899 1069 2024-05-27T07:30:15.231 INFO:journalctl@ceph.mds.a.smithi052.stdout:May 27 07:30:14 smithi052 podman[107416]: 2024-05-27 07:30:14.869255765 +0000 UTC m=+0.909091125 container died 16db595f4624542fb7d0ece3a23f74f0963bbc85f5c28833cf3d56e7b833c441 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:38f9d900c83c63799fbdbe61acc9a11b0d3554a6, name=nifty_elgamal, maintainer=Guillaume Abrioux , GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, ceph=True, org.label-schema.license=GPLv2, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240521, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90) 2024-05-27T07:30:15.231 INFO:journalctl@ceph.mds.a.smithi052.stdout:May 27 07:30:15 smithi052 systemd[1]: ceph-73df2636-1bf8-11ef-bc9b-c7b262605968@mds.a.service: Failed with result 'exit-code'. 2024-05-27T07:30:15.231 INFO:journalctl@ceph.mds.a.smithi052.stdout:May 27 07:30:15 smithi052 systemd[1]: ceph-73df2636-1bf8-11ef-bc9b-c7b262605968@mds.a.service: Consumed 1.433s CPU time. 2024-05-27T07:30:15.232 INFO:journalctl@ceph.mds.d.smithi052.stdout:May 27 07:30:15 smithi052 systemd[1]: ceph-73df2636-1bf8-11ef-bc9b-c7b262605968@mds.d.service: Failed with result 'exit-code'. 2024-05-27T07:30:15.232 INFO:journalctl@ceph.mds.d.smithi052.stdout:May 27 07:30:15 smithi052 systemd[1]: ceph-73df2636-1bf8-11ef-bc9b-c7b262605968@mds.d.service: Consumed 1.070s CPU time. 2024-05-27T07:30:15.735 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:30:15 smithi052 ceph-mon[31955]: pgmap v153: 129 pgs: 129 active+clean; 326 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 5.2 MiB/s rd, 3.1 MiB/s wr, 1.06k op/s 2024-05-27T07:30:15.853 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:30:15 smithi173 ceph-mon[37000]: pgmap v153: 129 pgs: 129 active+clean; 326 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 5.2 MiB/s rd, 3.1 MiB/s wr, 1.06k op/s 2024-05-27T07:30:15.941 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:30:15 smithi105 ceph-mon[35977]: pgmap v153: 129 pgs: 129 active+clean; 326 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 5.2 MiB/s rd, 3.1 MiB/s wr, 1.06k op/s 2024-05-27T07:30:16.465 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:30:16.473+0000 7f6db8756700 1 -- 172.21.15.52:0/2086955463 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f6db0076670 msgr2=0x7f6db0110650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:30:16.466 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:30:16.474+0000 7f6db8756700 1 --2- 172.21.15.52:0/2086955463 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f6db0076670 0x7f6db0110650 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7f6dac009990 tx=0x7f6dac009ca0 comp rx=0 tx=0).stop 2024-05-27T07:30:16.466 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:30:16.474+0000 7f6db8756700 1 -- 172.21.15.52:0/2086955463 shutdown_connections 2024-05-27T07:30:16.466 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:30:16.474+0000 7f6db8756700 1 --2- 172.21.15.52:0/2086955463 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f6db0110b90 0x7f6db0112ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:30:16.466 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:30:16.474+0000 7f6db8756700 1 --2- 172.21.15.52:0/2086955463 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f6db0075d20 0x7f6db0076130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:30:16.466 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:30:16.474+0000 7f6db8756700 1 --2- 172.21.15.52:0/2086955463 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f6db0076670 0x7f6db0110650 unknown :-1 s=CLOSED pgs=155 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:30:16.466 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:30:16.474+0000 7f6db8756700 1 -- 172.21.15.52:0/2086955463 >> 172.21.15.52:0/2086955463 conn(0x7f6db0100030 msgr2=0x7f6db0102450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:30:16.469 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:30:16.478+0000 7f6db8756700 1 -- 172.21.15.52:0/2086955463 shutdown_connections 2024-05-27T07:30:16.469 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:30:16.478+0000 7f6db8756700 1 -- 172.21.15.52:0/2086955463 wait complete. 2024-05-27T07:30:16.470 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:30:16.479+0000 7f6db8756700 1 Processor -- start 2024-05-27T07:30:16.470 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:30:16.479+0000 7f6db8756700 1 -- start start 2024-05-27T07:30:16.471 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:30:16.480+0000 7f6db8756700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f6db0075d20 0x7f6db019e8a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:30:16.471 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:30:16.480+0000 7f6db8756700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f6db0076670 0x7f6db019ee00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:30:16.472 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:30:16.481+0000 7f6db8756700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f6db0110b90 0x7f6db0198c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:30:16.473 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:30:16.481+0000 7f6db5cf1700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f6db0076670 0x7f6db019ee00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:30:16.473 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:30:16.481+0000 7f6db5cf1700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f6db0076670 0x7f6db019ee00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.52:58212/0 (socket says 172.21.15.52:58212) 2024-05-27T07:30:16.473 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:30:16.481+0000 7f6db6cf3700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f6db0110b90 0x7f6db0198c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:30:16.473 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:30:16.481+0000 7f6db6cf3700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f6db0110b90 0x7f6db0198c70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.52:39582/0 (socket says 172.21.15.52:39582) 2024-05-27T07:30:16.473 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:30:16.481+0000 7f6db6cf3700 1 -- 172.21.15.52:0/2783287891 learned_addr learned my addr 172.21.15.52:0/2783287891 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-05-27T07:30:16.473 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:30:16.481+0000 7f6db8756700 1 -- 172.21.15.52:0/2783287891 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x7f6db01069f0 con 0x7f6db0076670 2024-05-27T07:30:16.473 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:30:16.481+0000 7f6db8756700 1 -- 172.21.15.52:0/2783287891 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f6db0106870 con 0x7f6db0110b90 2024-05-27T07:30:16.473 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:30:16.481+0000 7f6db8756700 1 -- 172.21.15.52:0/2783287891 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f6db0106b70 con 0x7f6db0075d20 2024-05-27T07:30:16.475 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:30:16.482+0000 7f6db64f2700 1 --2- 172.21.15.52:0/2783287891 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f6db0075d20 0x7f6db019e8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:30:16.475 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:30:16.482+0000 7f6db6cf3700 1 -- 172.21.15.52:0/2783287891 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f6db0075d20 msgr2=0x7f6db019e8a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:30:16.475 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:30:16.482+0000 7f6db6cf3700 1 --2- 172.21.15.52:0/2783287891 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f6db0075d20 0x7f6db019e8a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:30:16.475 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:30:16.482+0000 7f6db6cf3700 1 -- 172.21.15.52:0/2783287891 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f6db0076670 msgr2=0x7f6db019ee00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:30:16.475 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:30:16.482+0000 7f6db6cf3700 1 --2- 172.21.15.52:0/2783287891 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f6db0076670 0x7f6db019ee00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:30:16.475 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:30:16.482+0000 7f6db6cf3700 1 -- 172.21.15.52:0/2783287891 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6db0199350 con 0x7f6db0110b90 2024-05-27T07:30:16.475 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:30:16.482+0000 7f6db64f2700 1 --2- 172.21.15.52:0/2783287891 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f6db0075d20 0x7f6db019e8a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-27T07:30:16.475 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:30:16.483+0000 7f6db6cf3700 1 --2- 172.21.15.52:0/2783287891 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f6db0110b90 0x7f6db0198c70 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7f6da000e970 tx=0x7f6da000ec80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:30:16.475 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:30:16.483+0000 7f6da77fe700 1 -- 172.21.15.52:0/2783287891 <== mon.2 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6da000cbd0 con 0x7f6db0110b90 2024-05-27T07:30:16.475 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:30:16.483+0000 7f6db8756700 1 -- 172.21.15.52:0/2783287891 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6db0199640 con 0x7f6db0110b90 2024-05-27T07:30:16.475 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:30:16.483+0000 7f6da77fe700 1 -- 172.21.15.52:0/2783287891 <== mon.2 v2:172.21.15.105:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f6da000ee60 con 0x7f6db0110b90 2024-05-27T07:30:16.476 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:30:16.483+0000 7f6da77fe700 1 -- 172.21.15.52:0/2783287891 <== mon.2 v2:172.21.15.105:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6da0005590 con 0x7f6db0110b90 2024-05-27T07:30:16.476 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:30:16.483+0000 7f6db8756700 1 -- 172.21.15.52:0/2783287891 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6db01a5750 con 0x7f6db0110b90 2024-05-27T07:30:16.484 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:30:16.492+0000 7f6dc716bb80 1 Processor -- start 2024-05-27T07:30:16.485 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:30:16.494+0000 7f6da77fe700 1 -- 172.21.15.52:0/2783287891 <== mon.2 v2:172.21.15.105:3300/0 4 ==== mgrmap(e 26) ==== 149783+0+0 (secure 0 0 0) 0x7f6da0021440 con 0x7f6db0110b90 2024-05-27T07:30:16.487 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:30:16.496+0000 7f6da77fe700 1 --2- 172.21.15.52:0/2783287891 >> [v2:172.21.15.52:6800/1822318384,v1:172.21.15.52:6801/1822318384] conn(0x7f6d9c0b2bc0 0x7f6d9c0b50d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:30:16.487 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:30:16.496+0000 7f6dc716bb80 1 -- start start 2024-05-27T07:30:16.487 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:30:16.496+0000 7f6da77fe700 1 -- 172.21.15.52:0/2783287891 <== mon.2 v2:172.21.15.105:3300/0 5 ==== osd_map(112..112 src has 1..112) ==== 9379+0+0 (secure 0 0 0) 0x7f6da00e2a70 con 0x7f6db0110b90 2024-05-27T07:30:16.487 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:30:16.496+0000 7f6dc716bb80 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55b32c6dde40 0x55b32c862b30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:30:16.489 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:30:16.496+0000 7f6dc716bb80 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55b32c863070 0x55b32c85cbe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:30:16.489 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:30:16.496+0000 7f6dc716bb80 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55b32c85d120 0x55b32c85d530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:30:16.489 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:30:16.496+0000 7f6dc716bb80 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x55b32c78ad70 con 0x55b32c85d120 2024-05-27T07:30:16.489 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:30:16.496+0000 7f6dc716bb80 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x55b32c77a910 con 0x55b32c863070 2024-05-27T07:30:16.489 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:30:16.497+0000 7f6dc716bb80 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x55b32c77e950 con 0x55b32c6dde40 2024-05-27T07:30:16.489 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:30:16.497+0000 7f6db6cf3700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55b32c85d120 0x55b32c85d530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:30:16.489 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:30:16.497+0000 7f6db6cf3700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55b32c85d120 0x55b32c85d530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.52:58224/0 (socket says 172.21.15.52:58224) 2024-05-27T07:30:16.489 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:30:16.497+0000 7f6db6cf3700 1 -- 172.21.15.52:0/4265476099 learned_addr learned my addr 172.21.15.52:0/4265476099 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-05-27T07:30:16.489 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:30:16.497+0000 7f6db6cf3700 1 -- 172.21.15.52:0/4265476099 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55b32c6dde40 msgr2=0x55b32c862b30 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-05-27T07:30:16.490 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:30:16.497+0000 7f6db6cf3700 1 --2- 172.21.15.52:0/4265476099 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55b32c6dde40 0x55b32c862b30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:30:16.490 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:30:16.497+0000 7f6db6cf3700 1 -- 172.21.15.52:0/4265476099 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55b32c863070 msgr2=0x55b32c85cbe0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-05-27T07:30:16.490 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:30:16.497+0000 7f6db6cf3700 1 --2- 172.21.15.52:0/4265476099 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55b32c863070 0x55b32c85cbe0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:30:16.490 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:30:16.497+0000 7f6db6cf3700 1 -- 172.21.15.52:0/4265476099 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55b32c723fb0 con 0x55b32c85d120 2024-05-27T07:30:16.490 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:30:16.497+0000 7f6db5cf1700 1 --2- 172.21.15.52:0/2783287891 >> [v2:172.21.15.52:6800/1822318384,v1:172.21.15.52:6801/1822318384] conn(0x7f6d9c0b2bc0 0x7f6d9c0b50d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:30:16.490 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:30:16.498+0000 7f6db6cf3700 1 --2- 172.21.15.52:0/4265476099 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55b32c85d120 0x55b32c85d530 secure :-1 s=READY pgs=639 cs=0 l=1 rev1=1 crypto rx=0x7f6da000e940 tx=0x7f6da00a9100 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1f856bc9e2b9e860 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:30:16.490 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:30:16.499+0000 7f6db5cf1700 1 --2- 172.21.15.52:0/2783287891 >> [v2:172.21.15.52:6800/1822318384,v1:172.21.15.52:6801/1822318384] conn(0x7f6d9c0b2bc0 0x7f6d9c0b50d0 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7f6dac01e040 tx=0x7f6dac0058e0 comp rx=0 tx=0).ready entity=mgr.15333 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:30:16.494 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:30:16.503+0000 7f6da4ff9700 1 -- 172.21.15.52:0/4265476099 <== mon.0 v2:172.21.15.52:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6da00c3070 con 0x55b32c85d120 2024-05-27T07:30:16.494 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:30:16.503+0000 7f6da4ff9700 1 -- 172.21.15.52:0/4265476099 <== mon.0 v2:172.21.15.52:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f6da00b2140 con 0x55b32c85d120 2024-05-27T07:30:16.494 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:30:16.503+0000 7f6da4ff9700 1 -- 172.21.15.52:0/4265476099 <== mon.0 v2:172.21.15.52:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6da00cc0d0 con 0x55b32c85d120 2024-05-27T07:30:16.496 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:30:16.504+0000 7f6dc716bb80 1 -- 172.21.15.52:0/4265476099 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55b32c85d120 msgr2=0x55b32c85d530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:30:16.496 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:30:16.504+0000 7f6dc716bb80 1 --2- 172.21.15.52:0/4265476099 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55b32c85d120 0x55b32c85d530 secure :-1 s=READY pgs=639 cs=0 l=1 rev1=1 crypto rx=0x7f6da000e940 tx=0x7f6da00a9100 comp rx=0 tx=0).stop 2024-05-27T07:30:16.496 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:30:16.504+0000 7f6dc716bb80 1 -- 172.21.15.52:0/4265476099 shutdown_connections 2024-05-27T07:30:16.496 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:30:16.504+0000 7f6dc716bb80 1 --2- 172.21.15.52:0/4265476099 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55b32c85d120 0x55b32c85d530 unknown :-1 s=CLOSED pgs=639 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:30:16.496 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:30:16.504+0000 7f6dc716bb80 1 --2- 172.21.15.52:0/4265476099 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55b32c6dde40 0x55b32c862b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:30:16.496 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:30:16.504+0000 7f6dc716bb80 1 --2- 172.21.15.52:0/4265476099 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55b32c863070 0x55b32c85cbe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:30:16.496 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:30:16.504+0000 7f6dc716bb80 1 -- 172.21.15.52:0/4265476099 >> 172.21.15.52:0/4265476099 conn(0x55b32c6e0770 msgr2=0x55b32c857d90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:30:16.496 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:30:16.505+0000 7f6dc716bb80 1 -- 172.21.15.52:0/4265476099 shutdown_connections 2024-05-27T07:30:16.496 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:30:16.505+0000 7f6dc716bb80 1 -- 172.21.15.52:0/4265476099 wait complete. 2024-05-27T07:30:16.497 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:30:16.505+0000 7f6dc716bb80 1 -- start start 2024-05-27T07:30:16.497 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:30:16.505+0000 7f6dc716bb80 1 Processor -- start 2024-05-27T07:30:16.497 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:30:16.506+0000 7f6dc716bb80 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55b32c6dde40 0x55b32c8668a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:30:16.498 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:30:16.506+0000 7f6dc716bb80 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55b32c85d120 0x55b32c866de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:30:16.498 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:30:16.506+0000 7f6dc716bb80 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55b32c863070 0x55b32c867320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:30:16.498 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:30:16.506+0000 7f6dc716bb80 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x55b32c680fa0 con 0x55b32c6dde40 2024-05-27T07:30:16.498 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:30:16.506+0000 7f6dc716bb80 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x55b32c6e2da0 con 0x55b32c85d120 2024-05-27T07:30:16.498 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:30:16.506+0000 7f6dc716bb80 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x55b32c869b60 con 0x55b32c863070 2024-05-27T07:30:16.498 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:30:16.506+0000 7f6dc716bb80 10 client.0 _ll_register_callbacks cb 0x55b32c7115b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T07:30:16.498 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:30:16.506+0000 7f6db6cf3700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55b32c863070 0x55b32c867320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:30:16.498 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:30:16.507+0000 7f6db6cf3700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55b32c863070 0x55b32c867320 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.173:3300/0 says I am v2:172.21.15.52:37112/0 (socket says 172.21.15.52:37112) 2024-05-27T07:30:16.498 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:30:16.507+0000 7f6db6cf3700 1 -- 172.21.15.52:0/2266166208 learned_addr learned my addr 172.21.15.52:0/2266166208 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-05-27T07:30:16.500 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:30:16.508+0000 7f6db5cf1700 1 --2- 172.21.15.52:0/2266166208 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55b32c85d120 0x55b32c866de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:30:16.500 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:30:16.508+0000 7f6db6cf3700 1 -- 172.21.15.52:0/2266166208 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55b32c85d120 msgr2=0x55b32c866de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:30:16.500 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:30:16.508+0000 7f6db6cf3700 1 --2- 172.21.15.52:0/2266166208 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55b32c85d120 0x55b32c866de0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:30:16.500 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:30:16.508+0000 7f6db6cf3700 1 -- 172.21.15.52:0/2266166208 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55b32c6dde40 msgr2=0x55b32c8668a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:30:16.500 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:30:16.508+0000 7f6db6cf3700 1 --2- 172.21.15.52:0/2266166208 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55b32c6dde40 0x55b32c8668a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:30:16.500 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:30:16.508+0000 7f6db6cf3700 1 -- 172.21.15.52:0/2266166208 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55b32c6ca8d0 con 0x55b32c863070 2024-05-27T07:30:16.502 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:30:16.510+0000 7f6db6cf3700 1 --2- 172.21.15.52:0/2266166208 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55b32c863070 0x55b32c867320 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7f6da00a59c0 tx=0x7f6da0009810 comp rx=0 tx=0).ready entity=mon.1 client_cookie=b1ac86e4a4e8aefd server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:30:16.502 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:30:16.510+0000 7f6d99ffb700 10 client.0 ms_handle_connect on v2:172.21.15.173:3300/0 2024-05-27T07:30:16.502 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:30:16.511+0000 7f6d99ffb700 1 -- 172.21.15.52:0/2266166208 <== mon.1 v2:172.21.15.173:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6da00b7060 con 0x55b32c863070 2024-05-27T07:30:16.502 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:30:16.511+0000 7f6d99ffb700 1 -- 172.21.15.52:0/2266166208 <== mon.1 v2:172.21.15.173:3300/0 2 ==== osd_map(112..112 src has 1..112) ==== 9379+0+0 (secure 0 0 0) 0x7f6da00a9e00 con 0x55b32c863070 2024-05-27T07:30:16.503 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:30:16.511+0000 7f6dc716bb80 1 -- 172.21.15.52:0/2266166208 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f6da00b2140 con 0x55b32c863070 2024-05-27T07:30:16.503 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:30:16.511+0000 7f6d99ffb700 1 -- 172.21.15.52:0/2266166208 <== mon.1 v2:172.21.15.173:3300/0 3 ==== mon_get_version_reply(handle=1 version=81) ==== 24+0+0 (secure 0 0 0) 0x7f6da00c3040 con 0x55b32c863070 2024-05-27T07:30:16.503 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:30:16.512+0000 7f6dc716bb80 10 client.26044 fetch_fsmap learned FSMap version 81 2024-05-27T07:30:16.503 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:30:16.512+0000 7f6dc716bb80 1 -- 172.21.15.52:0/2266166208 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({config=0+,fsmap=81,monmap=4+}) -- 0x55b32c734760 con 0x55b32c863070 2024-05-27T07:30:16.504 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:30:16.513+0000 7f6d99ffb700 1 -- 172.21.15.52:0/2266166208 <== mon.1 v2:172.21.15.173:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f6da00b2140 con 0x55b32c863070 2024-05-27T07:30:16.506 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:30:16.514+0000 7f6d99ffb700 1 -- 172.21.15.52:0/2266166208 <== mon.1 v2:172.21.15.173:3300/0 5 ==== fsmap(e 81) ==== 7168+0+0 (secure 0 0 0) 0x7f6da00a5300 con 0x55b32c863070 2024-05-27T07:30:16.506 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:30:16.514+0000 7f6dc716bb80 10 client.26044 fetch_fsmap finished waiting for FSMap version 81 2024-05-27T07:30:16.506 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:30:16.514+0000 7f6dc716bb80 10 client.26044 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-05-27T07:30:16.506 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:30:16.514+0000 7f6dc716bb80 20 client.26044 populate_metadata read hostname 'smithi052' 2024-05-27T07:30:16.506 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:30:16.514+0000 7f6dc716bb80 1 --2- 172.21.15.52:0/2266166208 >> [v2:172.21.15.105:6838/500348151,v1:172.21.15.105:6839/500348151] conn(0x55b32c876930 0x55b32c878d50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:30:16.506 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:30:16.514+0000 7f6dc716bb80 4 client.26044 mds_command: new command op to 24527 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-27T07:30:16.506 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:30:16.514+0000 7f6dc716bb80 1 -- 172.21.15.52:0/2266166208 --> [v2:172.21.15.105:6838/500348151,v1:172.21.15.105:6839/500348151] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55b32c78ccc0 con 0x55b32c876930 2024-05-27T07:30:16.506 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:30:16.515+0000 7f6db64f2700 1 --2- 172.21.15.52:0/2266166208 >> [v2:172.21.15.105:6838/500348151,v1:172.21.15.105:6839/500348151] conn(0x55b32c876930 0x55b32c878d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:30:16.508 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:30:16.516+0000 7f6db64f2700 1 --2- 172.21.15.52:0/2266166208 >> [v2:172.21.15.105:6838/500348151,v1:172.21.15.105:6839/500348151] conn(0x55b32c876930 0x55b32c878d50 crc :-1 s=READY pgs=105 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=cd631f9917d5d064 server_cookie=e84879063ae4ce22 in_seq=0 out_seq=0 2024-05-27T07:30:16.508 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:30:16.516+0000 7f6d99ffb700 10 client.26044 ms_handle_connect on v2:172.21.15.105:6838/500348151 2024-05-27T07:30:16.511 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:30:16.519+0000 7f6d99ffb700 1 -- 172.21.15.52:0/2266166208 <== mds.0 v2:172.21.15.105:6838/500348151 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24041 (crc 0 0 0) 0x55b32c78ccc0 con 0x55b32c876930 2024-05-27T07:30:16.511 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:30:16.520+0000 7f6d99ffb700 10 client.26044 handle_command_reply: tid=0 2024-05-27T07:30:16.512 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:30:16.520+0000 7f6dc716bb80 1 client.26044 shutdown 2024-05-27T07:30:16.513 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:30:16.521+0000 7f6dc716bb80 1 -- 172.21.15.52:0/2266166208 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55b32c863070 msgr2=0x55b32c867320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:30:16.513 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:30:16.521+0000 7f6dc716bb80 1 --2- 172.21.15.52:0/2266166208 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55b32c863070 0x55b32c867320 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7f6da00a59c0 tx=0x7f6da0009810 comp rx=0 tx=0).stop 2024-05-27T07:30:16.513 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:30:16.521+0000 7f6dc716bb80 1 -- 172.21.15.52:0/2266166208 shutdown_connections 2024-05-27T07:30:16.513 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:30:16.521+0000 7f6dc716bb80 1 --2- 172.21.15.52:0/2266166208 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55b32c863070 0x55b32c867320 unknown :-1 s=CLOSED pgs=133 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:30:16.513 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:30:16.521+0000 7f6dc716bb80 1 --2- 172.21.15.52:0/2266166208 >> [v2:172.21.15.105:6838/500348151,v1:172.21.15.105:6839/500348151] conn(0x55b32c876930 0x55b32c878d50 crc :-1 s=READY pgs=105 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:30:16.513 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:30:16.521+0000 7f6dc716bb80 1 --2- 172.21.15.52:0/2266166208 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55b32c6dde40 0x55b32c8668a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:30:16.513 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:30:16.521+0000 7f6dc716bb80 1 --2- 172.21.15.52:0/2266166208 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55b32c85d120 0x55b32c866de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:30:16.513 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:30:16.521+0000 7f6dc716bb80 1 -- 172.21.15.52:0/2266166208 >> 172.21.15.52:0/2266166208 conn(0x55b32c6e0770 msgr2=0x55b32c85bbd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:30:16.513 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:30:16.521+0000 7f6d99ffb700 0 client.26044 ms_handle_reset on v2:172.21.15.105:6838/500348151 2024-05-27T07:30:16.514 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:30:16.522+0000 7f6dc716bb80 1 -- 172.21.15.52:0/2266166208 shutdown_connections 2024-05-27T07:30:16.514 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:30:16.522+0000 7f6dc716bb80 1 -- 172.21.15.52:0/2266166208 wait complete. 2024-05-27T07:30:16.514 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:30:16.522+0000 7f6dc716bb80 20 client.26044 trim_cache size 0 max 16384 2024-05-27T07:30:16.551 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:30:16.560+0000 7f6dc716bb80 1 Processor -- start 2024-05-27T07:30:16.552 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:30:16.560+0000 7f6dc716bb80 1 -- start start 2024-05-27T07:30:16.552 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:30:16.561+0000 7f6dc716bb80 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55b32c6dde40 0x55b32c85cbe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:30:16.553 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:30:16.561+0000 7f6dc716bb80 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55b32c85d120 0x55b32c858860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:30:16.553 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:30:16.561+0000 7f6dc716bb80 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55b32c863070 0x55b32c860ba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:30:16.553 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:30:16.561+0000 7f6db64f2700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55b32c6dde40 0x55b32c85cbe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:30:16.553 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:30:16.561+0000 7f6db64f2700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55b32c6dde40 0x55b32c85cbe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.52:39616/0 (socket says 172.21.15.52:39616) 2024-05-27T07:30:16.553 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:30:16.561+0000 7f6db64f2700 1 -- 172.21.15.52:0/2549671831 learned_addr learned my addr 172.21.15.52:0/2549671831 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-05-27T07:30:16.553 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:30:16.561+0000 7f6dc716bb80 1 -- 172.21.15.52:0/2549671831 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x7f6d9400d780 con 0x55b32c863070 2024-05-27T07:30:16.553 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:30:16.561+0000 7f6dc716bb80 1 -- 172.21.15.52:0/2549671831 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f6d94010120 con 0x55b32c6dde40 2024-05-27T07:30:16.553 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:30:16.561+0000 7f6dc716bb80 1 -- 172.21.15.52:0/2549671831 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f6d9400de50 con 0x55b32c85d120 2024-05-27T07:30:16.554 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:30:16.561+0000 7f6db6cf3700 1 --2- 172.21.15.52:0/2549671831 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55b32c863070 0x55b32c860ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:30:16.554 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:30:16.561+0000 7f6db5cf1700 1 --2- 172.21.15.52:0/2549671831 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55b32c85d120 0x55b32c858860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:30:16.554 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:30:16.561+0000 7f6db64f2700 1 -- 172.21.15.52:0/2549671831 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55b32c85d120 msgr2=0x55b32c858860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:30:16.554 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:30:16.561+0000 7f6db64f2700 1 --2- 172.21.15.52:0/2549671831 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55b32c85d120 0x55b32c858860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:30:16.554 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:30:16.561+0000 7f6db64f2700 1 -- 172.21.15.52:0/2549671831 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55b32c863070 msgr2=0x55b32c860ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:30:16.554 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:30:16.561+0000 7f6db64f2700 1 --2- 172.21.15.52:0/2549671831 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55b32c863070 0x55b32c860ba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:30:16.554 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:30:16.561+0000 7f6db64f2700 1 -- 172.21.15.52:0/2549671831 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55b32c85dc30 con 0x55b32c6dde40 2024-05-27T07:30:16.554 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:30:16.562+0000 7f6db5cf1700 1 --2- 172.21.15.52:0/2549671831 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55b32c85d120 0x55b32c858860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-27T07:30:16.554 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:30:16.562+0000 7f6db64f2700 1 --2- 172.21.15.52:0/2549671831 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55b32c6dde40 0x55b32c85cbe0 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7f6da8008fd0 tx=0x7f6da8008330 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ac3a095cfb9409bc server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:30:16.555 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:30:16.562+0000 7f6d9bfff700 1 -- 172.21.15.52:0/2549671831 <== mon.2 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6da80119f0 con 0x55b32c6dde40 2024-05-27T07:30:16.555 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:30:16.562+0000 7f6d9bfff700 1 -- 172.21.15.52:0/2549671831 <== mon.2 v2:172.21.15.105:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f6da801b030 con 0x55b32c6dde40 2024-05-27T07:30:16.555 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:30:16.562+0000 7f6d9bfff700 1 -- 172.21.15.52:0/2549671831 <== mon.2 v2:172.21.15.105:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6da801f430 con 0x55b32c6dde40 2024-05-27T07:30:16.555 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:30:16.564+0000 7f6dc716bb80 1 -- 172.21.15.52:0/2549671831 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55b32c6dde40 msgr2=0x55b32c85cbe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:30:16.555 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:30:16.564+0000 7f6dc716bb80 1 --2- 172.21.15.52:0/2549671831 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55b32c6dde40 0x55b32c85cbe0 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7f6da8008fd0 tx=0x7f6da8008330 comp rx=0 tx=0).stop 2024-05-27T07:30:16.556 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:30:16.565+0000 7f6dc716bb80 1 -- 172.21.15.52:0/2549671831 shutdown_connections 2024-05-27T07:30:16.557 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:30:16.565+0000 7f6dc716bb80 1 --2- 172.21.15.52:0/2549671831 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55b32c863070 0x55b32c860ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:30:16.557 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:30:16.565+0000 7f6dc716bb80 1 --2- 172.21.15.52:0/2549671831 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55b32c6dde40 0x55b32c85cbe0 unknown :-1 s=CLOSED pgs=157 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:30:16.557 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:30:16.565+0000 7f6dc716bb80 1 --2- 172.21.15.52:0/2549671831 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55b32c85d120 0x55b32c858860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:30:16.557 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:30:16.565+0000 7f6dc716bb80 1 -- 172.21.15.52:0/2549671831 >> 172.21.15.52:0/2549671831 conn(0x55b32c6e0770 msgr2=0x55b32c860490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:30:16.557 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:30:16.565+0000 7f6dc716bb80 1 -- 172.21.15.52:0/2549671831 shutdown_connections 2024-05-27T07:30:16.557 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:30:16.565+0000 7f6dc716bb80 1 -- 172.21.15.52:0/2549671831 wait complete. 2024-05-27T07:30:16.558 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:30:16.566+0000 7f6dc716bb80 1 -- start start 2024-05-27T07:30:16.558 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:30:16.566+0000 7f6dc716bb80 1 Processor -- start 2024-05-27T07:30:16.558 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:30:16.567+0000 7f6dc716bb80 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55b32c6dde40 0x55b32c86edf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:30:16.559 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:30:16.567+0000 7f6dc716bb80 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55b32c85d120 0x55b32c86f330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:30:16.560 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:30:16.567+0000 7f6dc716bb80 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55b32c863070 0x55b32c87d310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:30:16.560 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:30:16.568+0000 7f6db5cf1700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55b32c85d120 0x55b32c86f330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:30:16.560 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:30:16.568+0000 7f6db5cf1700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55b32c85d120 0x55b32c86f330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.52:39622/0 (socket says 172.21.15.52:39622) 2024-05-27T07:30:16.560 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:30:16.568+0000 7f6db5cf1700 1 -- 172.21.15.52:0/1714340685 learned_addr learned my addr 172.21.15.52:0/1714340685 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-05-27T07:30:16.560 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:30:16.568+0000 7f6db64f2700 1 --2- 172.21.15.52:0/1714340685 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55b32c6dde40 0x55b32c86edf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:30:16.560 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:30:16.568+0000 7f6dc716bb80 1 -- 172.21.15.52:0/1714340685 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x7f6d9400e520 con 0x55b32c6dde40 2024-05-27T07:30:16.560 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:30:16.568+0000 7f6dc716bb80 1 -- 172.21.15.52:0/1714340685 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f6d9400f990 con 0x55b32c85d120 2024-05-27T07:30:16.560 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:30:16.568+0000 7f6dc716bb80 1 -- 172.21.15.52:0/1714340685 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f6d9400ebf0 con 0x55b32c863070 2024-05-27T07:30:16.560 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:30:16.568+0000 7f6dc716bb80 10 client.0 _ll_register_callbacks cb 0x55b32c7115b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T07:30:16.560 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:30:16.568+0000 7f6db6cf3700 1 --2- 172.21.15.52:0/1714340685 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55b32c863070 0x55b32c87d310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:30:16.560 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:30:16.569+0000 7f6db5cf1700 1 -- 172.21.15.52:0/1714340685 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55b32c863070 msgr2=0x55b32c87d310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:30:16.561 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:30:16.569+0000 7f6db5cf1700 1 --2- 172.21.15.52:0/1714340685 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55b32c863070 0x55b32c87d310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:30:16.561 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:30:16.569+0000 7f6db5cf1700 1 -- 172.21.15.52:0/1714340685 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55b32c6dde40 msgr2=0x55b32c86edf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:30:16.561 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:30:16.569+0000 7f6db5cf1700 1 --2- 172.21.15.52:0/1714340685 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55b32c6dde40 0x55b32c86edf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:30:16.561 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:30:16.569+0000 7f6db5cf1700 1 -- 172.21.15.52:0/1714340685 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55b32c87d850 con 0x55b32c85d120 2024-05-27T07:30:16.561 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:30:16.569+0000 7f6db6cf3700 1 --2- 172.21.15.52:0/1714340685 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55b32c863070 0x55b32c87d310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-27T07:30:16.561 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:30:16.569+0000 7f6db5cf1700 1 --2- 172.21.15.52:0/1714340685 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55b32c85d120 0x55b32c86f330 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7f6dac000c00 tx=0x7f6dac009310 comp rx=0 tx=0).ready entity=mon.2 client_cookie=37e0d14eb218aa74 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:30:16.563 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:30:16.569+0000 7f6d99ffb700 10 client.0 ms_handle_connect on v2:172.21.15.105:3300/0 2024-05-27T07:30:16.563 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:30:16.569+0000 7f6d99ffb700 1 -- 172.21.15.52:0/1714340685 <== mon.2 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6dac0148b0 con 0x55b32c85d120 2024-05-27T07:30:16.564 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:30:16.569+0000 7f6d99ffb700 1 -- 172.21.15.52:0/1714340685 <== mon.2 v2:172.21.15.105:3300/0 2 ==== osd_map(112..112 src has 1..112) ==== 9379+0+0 (secure 0 0 0) 0x7f6dac00b4f0 con 0x55b32c85d120 2024-05-27T07:30:16.564 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:30:16.570+0000 7f6dc716bb80 1 -- 172.21.15.52:0/1714340685 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f6da801b030 con 0x55b32c85d120 2024-05-27T07:30:16.564 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:30:16.570+0000 7f6d99ffb700 1 -- 172.21.15.52:0/1714340685 <== mon.2 v2:172.21.15.105:3300/0 3 ==== mon_get_version_reply(handle=1 version=81) ==== 24+0+0 (secure 0 0 0) 0x7f6dac0056b0 con 0x55b32c85d120 2024-05-27T07:30:16.564 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:30:16.571+0000 7f6dc716bb80 10 client.26144 fetch_fsmap learned FSMap version 81 2024-05-27T07:30:16.564 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:30:16.571+0000 7f6dc716bb80 1 -- 172.21.15.52:0/1714340685 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,fsmap=81,monmap=4+}) -- 0x55b32c8f1840 con 0x55b32c85d120 2024-05-27T07:30:16.564 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:30:16.572+0000 7f6d99ffb700 1 -- 172.21.15.52:0/1714340685 <== mon.2 v2:172.21.15.105:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f6dac02c040 con 0x55b32c85d120 2024-05-27T07:30:16.564 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:30:16.572+0000 7f6d99ffb700 1 -- 172.21.15.52:0/1714340685 <== mon.2 v2:172.21.15.105:3300/0 5 ==== fsmap(e 81) ==== 7168+0+0 (secure 0 0 0) 0x7f6dac038030 con 0x55b32c85d120 2024-05-27T07:30:16.564 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:30:16.572+0000 7f6dc716bb80 10 client.26144 fetch_fsmap finished waiting for FSMap version 81 2024-05-27T07:30:16.564 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:30:16.572+0000 7f6dc716bb80 10 client.26144 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-05-27T07:30:16.564 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:30:16.572+0000 7f6dc716bb80 20 client.26144 populate_metadata read hostname 'smithi052' 2024-05-27T07:30:16.564 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:30:16.572+0000 7f6dc716bb80 1 --2- 172.21.15.52:0/1714340685 >> [v2:172.21.15.105:6838/500348151,v1:172.21.15.105:6839/500348151] conn(0x55b32c8f2090 0x55b32c8f44b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:30:16.564 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:30:16.572+0000 7f6dc716bb80 4 client.26144 mds_command: new command op to 24527 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-05-27T07:30:16.565 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:30:16.572+0000 7f6dc716bb80 1 -- 172.21.15.52:0/1714340685 --> [v2:172.21.15.105:6838/500348151,v1:172.21.15.105:6839/500348151] -- command(tid 0: {"prefix": "scrub status"}) -- 0x7f6da801b030 con 0x55b32c8f2090 2024-05-27T07:30:16.565 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:30:16.572+0000 7f6db64f2700 1 --2- 172.21.15.52:0/1714340685 >> [v2:172.21.15.105:6838/500348151,v1:172.21.15.105:6839/500348151] conn(0x55b32c8f2090 0x55b32c8f44b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:30:16.565 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:30:16.574+0000 7f6db64f2700 1 --2- 172.21.15.52:0/1714340685 >> [v2:172.21.15.105:6838/500348151,v1:172.21.15.105:6839/500348151] conn(0x55b32c8f2090 0x55b32c8f44b0 crc :-1 s=READY pgs=106 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=b34fd8e8b11671d3 server_cookie=9d7e80ccc27c8f5a in_seq=0 out_seq=0 2024-05-27T07:30:16.565 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:30:16.574+0000 7f6d99ffb700 10 client.26144 ms_handle_connect on v2:172.21.15.105:6838/500348151 2024-05-27T07:30:16.569 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:30:16.577+0000 7f6d99ffb700 1 -- 172.21.15.52:0/1714340685 <== mds.0 v2:172.21.15.105:6838/500348151 1 ==== command_reply(tid 0: 0 ) ==== 8+0+269 (crc 0 0 0) 0x7f6da801b030 con 0x55b32c8f2090 2024-05-27T07:30:16.569 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:30:16.577+0000 7f6d99ffb700 10 client.26144 handle_command_reply: tid=0 2024-05-27T07:30:16.569 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:30:16.578+0000 7f6dc716bb80 1 client.26144 shutdown 2024-05-27T07:30:16.573 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:30:16.582+0000 7f6dc716bb80 1 -- 172.21.15.52:0/1714340685 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55b32c85d120 msgr2=0x55b32c86f330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:30:16.573 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:30:16.582+0000 7f6dc716bb80 1 --2- 172.21.15.52:0/1714340685 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55b32c85d120 0x55b32c86f330 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7f6dac000c00 tx=0x7f6dac009310 comp rx=0 tx=0).stop 2024-05-27T07:30:16.574 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:30:16.583+0000 7f6dc716bb80 1 -- 172.21.15.52:0/1714340685 shutdown_connections 2024-05-27T07:30:16.574 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:30:16.583+0000 7f6dc716bb80 1 --2- 172.21.15.52:0/1714340685 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55b32c863070 0x55b32c87d310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:30:16.574 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:30:16.583+0000 7f6dc716bb80 1 --2- 172.21.15.52:0/1714340685 >> [v2:172.21.15.105:6838/500348151,v1:172.21.15.105:6839/500348151] conn(0x55b32c8f2090 0x55b32c8f44b0 crc :-1 s=READY pgs=106 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:30:16.574 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:30:16.583+0000 7f6d99ffb700 0 client.26144 ms_handle_reset on v2:172.21.15.105:6838/500348151 2024-05-27T07:30:16.575 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:30:16.583+0000 7f6dc716bb80 1 --2- 172.21.15.52:0/1714340685 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55b32c6dde40 0x55b32c86edf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:30:16.575 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:30:16.583+0000 7f6dc716bb80 1 --2- 172.21.15.52:0/1714340685 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55b32c85d120 0x55b32c86f330 unknown :-1 s=CLOSED pgs=158 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:30:16.575 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:30:16.583+0000 7f6dc716bb80 1 -- 172.21.15.52:0/1714340685 >> 172.21.15.52:0/1714340685 conn(0x55b32c6e0770 msgr2=0x55b32c8514f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:30:16.575 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:30:16.583+0000 7f6dc716bb80 1 -- 172.21.15.52:0/1714340685 shutdown_connections 2024-05-27T07:30:16.575 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:30:16.583+0000 7f6dc716bb80 1 -- 172.21.15.52:0/1714340685 wait complete. 2024-05-27T07:30:16.575 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:30:16.584+0000 7f6dc716bb80 20 client.26144 trim_cache size 0 max 16384 2024-05-27T07:30:16.576 INFO:teuthology.orchestra.run.smithi052.stdout:{ 2024-05-27T07:30:16.576 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "scrub active (11265 inodes in the stack)", 2024-05-27T07:30:16.577 INFO:teuthology.orchestra.run.smithi052.stdout: "scrubs": { 2024-05-27T07:30:16.577 INFO:teuthology.orchestra.run.smithi052.stdout: "8ab50389-7f33-4519-a5fc-243c0e6b3763": { 2024-05-27T07:30:16.577 INFO:teuthology.orchestra.run.smithi052.stdout: "path": "/", 2024-05-27T07:30:16.577 INFO:teuthology.orchestra.run.smithi052.stdout: "tag": "8ab50389-7f33-4519-a5fc-243c0e6b3763", 2024-05-27T07:30:16.577 INFO:teuthology.orchestra.run.smithi052.stdout: "options": "recursive,force" 2024-05-27T07:30:16.577 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-05-27T07:30:16.577 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-05-27T07:30:16.577 INFO:teuthology.orchestra.run.smithi052.stdout:} 2024-05-27T07:30:16.578 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:30:16.585+0000 7f6db8756700 1 -- 172.21.15.52:0/2783287891 >> [v2:172.21.15.52:6800/1822318384,v1:172.21.15.52:6801/1822318384] conn(0x7f6d9c0b2bc0 msgr2=0x7f6d9c0b50d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:30:16.578 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:30:16.585+0000 7f6db8756700 1 --2- 172.21.15.52:0/2783287891 >> [v2:172.21.15.52:6800/1822318384,v1:172.21.15.52:6801/1822318384] conn(0x7f6d9c0b2bc0 0x7f6d9c0b50d0 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7f6dac01e040 tx=0x7f6dac0058e0 comp rx=0 tx=0).stop 2024-05-27T07:30:16.578 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:30:16.586+0000 7f6db8756700 1 -- 172.21.15.52:0/2783287891 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f6db0110b90 msgr2=0x7f6db0198c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:30:16.578 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:30:16.586+0000 7f6db8756700 1 --2- 172.21.15.52:0/2783287891 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f6db0110b90 0x7f6db0198c70 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7f6da000e970 tx=0x7f6da000ec80 comp rx=0 tx=0).stop 2024-05-27T07:30:16.579 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:30:16.587+0000 7f6db8756700 1 -- 172.21.15.52:0/2783287891 shutdown_connections 2024-05-27T07:30:16.579 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:30:16.587+0000 7f6db8756700 1 --2- 172.21.15.52:0/2783287891 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f6db0110b90 0x7f6db0198c70 unknown :-1 s=CLOSED pgs=156 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:30:16.579 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:30:16.587+0000 7f6db8756700 1 --2- 172.21.15.52:0/2783287891 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f6db0075d20 0x7f6db019e8a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:30:16.579 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:30:16.587+0000 7f6db8756700 1 --2- 172.21.15.52:0/2783287891 >> [v2:172.21.15.52:6800/1822318384,v1:172.21.15.52:6801/1822318384] conn(0x7f6d9c0b2bc0 0x7f6d9c0b50d0 unknown :-1 s=CLOSED pgs=134 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:30:16.579 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:30:16.587+0000 7f6db8756700 1 --2- 172.21.15.52:0/2783287891 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f6db0076670 0x7f6db019ee00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:30:16.579 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:30:16.587+0000 7f6db8756700 1 -- 172.21.15.52:0/2783287891 >> 172.21.15.52:0/2783287891 conn(0x7f6db0100030 msgr2=0x7f6db0101dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:30:16.579 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:30:16.587+0000 7f6db8756700 1 -- 172.21.15.52:0/2783287891 shutdown_connections 2024-05-27T07:30:16.579 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:30:16.587+0000 7f6db8756700 1 -- 172.21.15.52:0/2783287891 wait complete. 2024-05-27T07:30:16.842 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:30:16 smithi105 ceph-mon[35977]: from='mgr.15333 172.21.15.52:0/416389082' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-27T07:30:16.853 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:30:16 smithi173 ceph-mon[37000]: from='mgr.15333 172.21.15.52:0/416389082' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-27T07:30:16.926 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:30:16 smithi052 ceph-mon[31955]: from='mgr.15333 172.21.15.52:0/416389082' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-27T07:30:17.387 INFO:tasks.cephfs.filesystem:scrub status for tag:8ab50389-7f33-4519-a5fc-243c0e6b3763 - {'path': '/', 'tag': '8ab50389-7f33-4519-a5fc-243c0e6b3763', 'options': 'recursive,force'} 2024-05-27T07:30:17.730 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:30:17 smithi052 ceph-mon[31955]: pgmap v154: 129 pgs: 129 active+clean; 332 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 5.8 MiB/s rd, 2.4 MiB/s wr, 1.43k op/s 2024-05-27T07:30:17.853 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:30:17 smithi173 ceph-mon[37000]: pgmap v154: 129 pgs: 129 active+clean; 332 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 5.8 MiB/s rd, 2.4 MiB/s wr, 1.43k op/s 2024-05-27T07:30:17.941 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:30:17 smithi105 ceph-mon[35977]: pgmap v154: 129 pgs: 129 active+clean; 332 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 5.8 MiB/s rd, 2.4 MiB/s wr, 1.43k op/s 2024-05-27T07:30:18.853 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:30:18 smithi173 ceph-mon[37000]: from='mgr.15333 172.21.15.52:0/416389082' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T07:30:18.853 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:30:18 smithi173 ceph-mon[37000]: from='mgr.15333 172.21.15.52:0/416389082' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-27T07:30:18.853 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:30:18 smithi173 ceph-mon[37000]: from='mgr.15333 172.21.15.52:0/416389082' entity='mgr.x' 2024-05-27T07:30:18.941 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:30:18 smithi105 ceph-mon[35977]: from='mgr.15333 172.21.15.52:0/416389082' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T07:30:18.941 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:30:18 smithi105 ceph-mon[35977]: from='mgr.15333 172.21.15.52:0/416389082' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-27T07:30:18.941 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:30:18 smithi105 ceph-mon[35977]: from='mgr.15333 172.21.15.52:0/416389082' entity='mgr.x' 2024-05-27T07:30:18.980 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:30:18 smithi052 ceph-mon[31955]: from='mgr.15333 172.21.15.52:0/416389082' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T07:30:18.980 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:30:18 smithi052 ceph-mon[31955]: from='mgr.15333 172.21.15.52:0/416389082' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-27T07:30:18.981 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:30:18 smithi052 ceph-mon[31955]: from='mgr.15333 172.21.15.52:0/416389082' entity='mgr.x' 2024-05-27T07:30:19.853 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:30:19 smithi173 ceph-mon[37000]: pgmap v155: 129 pgs: 129 active+clean; 347 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 5.5 MiB/s rd, 3.4 MiB/s wr, 1.44k op/s 2024-05-27T07:30:19.941 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:30:19 smithi105 ceph-mon[35977]: pgmap v155: 129 pgs: 129 active+clean; 347 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 5.5 MiB/s rd, 3.4 MiB/s wr, 1.44k op/s 2024-05-27T07:30:19.980 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:30:19 smithi052 ceph-mon[31955]: pgmap v155: 129 pgs: 129 active+clean; 347 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 5.5 MiB/s rd, 3.4 MiB/s wr, 1.44k op/s 2024-05-27T07:30:20.195 INFO:journalctl@ceph.mds.k.smithi105.stdout:May 27 07:30:19 smithi105 systemd[1]: ceph-73df2636-1bf8-11ef-bc9b-c7b262605968@mds.k.service: Scheduled restart job, restart counter is at 5. 2024-05-27T07:30:20.195 INFO:journalctl@ceph.mds.k.smithi105.stdout:May 27 07:30:19 smithi105 systemd[1]: Stopped Ceph mds.k for 73df2636-1bf8-11ef-bc9b-c7b262605968. 2024-05-27T07:30:20.195 INFO:journalctl@ceph.mds.k.smithi105.stdout:May 27 07:30:19 smithi105 systemd[1]: Starting Ceph mds.k for 73df2636-1bf8-11ef-bc9b-c7b262605968... 2024-05-27T07:30:20.486 INFO:journalctl@ceph.mds.k.smithi105.stdout:May 27 07:30:20 smithi105 podman[70376]: 2024-05-27 07:30:20.107348721 +0000 UTC m=+0.026986205 image pull e36ea4fc8a5807e8d2701370e350a0de3856538797f7fba64cfce7e344bd6799 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6 2024-05-27T07:30:20.486 INFO:journalctl@ceph.mds.k.smithi105.stdout:May 27 07:30:20 smithi105 podman[70376]: 2024-05-27 07:30:20.225213311 +0000 UTC m=+0.144850767 container create 7ecb8856db56e0f00d5b025b01a7c69ab17ac2a1672e92156e1ba5122cefd159 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-k, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, GIT_CLEAN=True, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, ceph=True, org.label-schema.build-date=20240521, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS) 2024-05-27T07:30:20.535 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:30:20.543+0000 7ff762ffd640 1 -- 172.21.15.52:0/421624937 <== mds.0 v2:172.21.15.105:6838/500348151 1 ==== command_reply(tid 0: 0 ) ==== 8+0+25 (crc 0 0 0) 0x55fae8805360 con 0x55fae848a1c0 2024-05-27T07:30:20.535 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:30:20.543+0000 7ff762ffd640 10 client.26108 handle_command_reply: tid=0 2024-05-27T07:30:20.535 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:30:20.543+0000 7ff7909d3740 1 client.26108 shutdown 2024-05-27T07:30:20.535 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:30:20.543+0000 7ff7909d3740 1 -- 172.21.15.52:0/421624937 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55fae84f61b0 msgr2=0x55fae89147f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:30:20.535 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:30:20.543+0000 7ff7909d3740 1 --2- 172.21.15.52:0/421624937 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55fae84f61b0 0x55fae89147f0 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7ff77c043040 tx=0x7ff77c004730 comp rx=0 tx=0).stop 2024-05-27T07:30:20.536 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:30:20.543+0000 7ff7909d3740 1 -- 172.21.15.52:0/421624937 shutdown_connections 2024-05-27T07:30:20.536 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:30:20.544+0000 7ff7909d3740 1 --2- 172.21.15.52:0/421624937 >> [v2:172.21.15.105:6838/500348151,v1:172.21.15.105:6839/500348151] conn(0x55fae848a1c0 0x55fae891e710 crc :-1 s=READY pgs=102 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:30:20.536 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:30:20.544+0000 7ff7909d3740 1 --2- 172.21.15.52:0/421624937 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55fae84f61b0 0x55fae89147f0 unknown :-1 s=CLOSED pgs=147 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:30:20.536 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:30:20.544+0000 7ff7909d3740 1 --2- 172.21.15.52:0/421624937 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55fae84d54d0 0x55fae88a72f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:30:20.536 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:30:20.544+0000 7ff7909d3740 1 --2- 172.21.15.52:0/421624937 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55fae84cd2f0 0x55fae88a6db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:30:20.536 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:30:20.544+0000 7ff762ffd640 0 client.26108 ms_handle_reset on v2:172.21.15.105:6838/500348151 2024-05-27T07:30:20.536 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:30:20.544+0000 7ff7909d3740 1 -- 172.21.15.52:0/421624937 >> 172.21.15.52:0/421624937 conn(0x55fae84ea300 msgr2=0x55fae88a3810 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:30:20.536 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:30:20.544+0000 7ff7909d3740 1 -- 172.21.15.52:0/421624937 shutdown_connections 2024-05-27T07:30:20.536 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:30:20.544+0000 7ff7909d3740 1 -- 172.21.15.52:0/421624937 wait complete. 2024-05-27T07:30:20.536 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:30:20.544+0000 7ff7909d3740 20 client.26108 trim_cache size 0 max 16384 2024-05-27T07:30:20.537 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stdout:{ 2024-05-27T07:30:20.537 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stdout: "return_code": 0 2024-05-27T07:30:20.537 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stdout:} 2024-05-27T07:30:20.537 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:30:20.544+0000 7ff789b1f640 1 -- 172.21.15.52:0/1220248260 >> [v2:172.21.15.52:6800/1822318384,v1:172.21.15.52:6801/1822318384] conn(0x7ff76c0b2950 msgr2=0x7ff76c0b4e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:30:20.538 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:30:20.544+0000 7ff789b1f640 1 --2- 172.21.15.52:0/1220248260 >> [v2:172.21.15.52:6800/1822318384,v1:172.21.15.52:6801/1822318384] conn(0x7ff76c0b2950 0x7ff76c0b4e60 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x55fae8894540 tx=0x7ff77c0094a0 comp rx=0 tx=0).stop 2024-05-27T07:30:20.538 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:30:20.544+0000 7ff789b1f640 1 -- 172.21.15.52:0/1220248260 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7ff784108bb0 msgr2=0x7ff78419bb80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:30:20.538 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:30:20.544+0000 7ff789b1f640 1 --2- 172.21.15.52:0/1220248260 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7ff784108bb0 0x7ff78419bb80 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7ff778005d00 tx=0x7ff77800dde0 comp rx=0 tx=0).stop 2024-05-27T07:30:20.538 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:30:20.544+0000 7ff789b1f640 1 -- 172.21.15.52:0/1220248260 shutdown_connections 2024-05-27T07:30:20.538 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:30:20.544+0000 7ff789b1f640 1 --2- 172.21.15.52:0/1220248260 >> [v2:172.21.15.52:6800/1822318384,v1:172.21.15.52:6801/1822318384] conn(0x7ff76c0b2950 0x7ff76c0b4e60 unknown :-1 s=CLOSED pgs=129 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:30:20.538 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:30:20.544+0000 7ff789b1f640 1 --2- 172.21.15.52:0/1220248260 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff784109050 0x7ff7841a66c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:30:20.538 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:30:20.544+0000 7ff789b1f640 1 --2- 172.21.15.52:0/1220248260 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7ff784108bb0 0x7ff78419bb80 unknown :-1 s=CLOSED pgs=143 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:30:20.538 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:30:20.544+0000 7ff789b1f640 1 --2- 172.21.15.52:0/1220248260 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7ff784103300 0x7ff78419b640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:30:20.538 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:30:20.544+0000 7ff789b1f640 1 -- 172.21.15.52:0/1220248260 >> 172.21.15.52:0/1220248260 conn(0x7ff7840781a0 msgr2=0x7ff784100e50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:30:20.538 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:30:20.545+0000 7ff789b1f640 1 -- 172.21.15.52:0/1220248260 shutdown_connections 2024-05-27T07:30:20.538 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:30:20.545+0000 7ff789b1f640 1 -- 172.21.15.52:0/1220248260 wait complete. 2024-05-27T07:30:20.736 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:30:20 smithi052 ceph-mon[31955]: MDS daemon mds.j is removed because it is dead or otherwise unavailable. 2024-05-27T07:30:20.736 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:30:20 smithi052 ceph-mon[31955]: MDS daemon mds.l is removed because it is dead or otherwise unavailable. 2024-05-27T07:30:20.736 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:30:20 smithi052 ceph-mon[31955]: fsmap cephfs:5 3 up:standby-replay 5 up:active 2024-05-27T07:30:20.736 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:30:20 smithi052 ceph-mon[31955]: daemon mds.k restarted 2024-05-27T07:30:20.736 INFO:journalctl@ceph.mds.j.smithi052.stdout:May 27 07:30:20 smithi052 systemd[1]: ceph-73df2636-1bf8-11ef-bc9b-c7b262605968@mds.j.service: Scheduled restart job, restart counter is at 2. 2024-05-27T07:30:20.737 INFO:journalctl@ceph.mds.j.smithi052.stdout:May 27 07:30:20 smithi052 systemd[1]: Stopped Ceph mds.j for 73df2636-1bf8-11ef-bc9b-c7b262605968. 2024-05-27T07:30:20.737 INFO:journalctl@ceph.mds.j.smithi052.stdout:May 27 07:30:20 smithi052 systemd[1]: ceph-73df2636-1bf8-11ef-bc9b-c7b262605968@mds.j.service: Consumed 1.226s CPU time. 2024-05-27T07:30:20.737 INFO:journalctl@ceph.mds.j.smithi052.stdout:May 27 07:30:20 smithi052 systemd[1]: Starting Ceph mds.j for 73df2636-1bf8-11ef-bc9b-c7b262605968... 2024-05-27T07:30:20.737 INFO:journalctl@ceph.mds.j.smithi052.stdout:May 27 07:30:20 smithi052 podman[107731]: 2024-05-27 07:30:20.633099307 +0000 UTC m=+0.019559653 image pull e36ea4fc8a5807e8d2701370e350a0de3856538797f7fba64cfce7e344bd6799 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6 2024-05-27T07:30:20.737 INFO:journalctl@ceph.mds.j.smithi052.stdout:May 27 07:30:20 smithi052 podman[107731]: 2024-05-27 07:30:20.745561316 +0000 UTC m=+0.132021640 container create 332ab7d985b17bcd2765ea36a1f8438aec4b0d8a6a4108995ab221d58bbcb935 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-j, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_CLEAN=True, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, GIT_BRANCH=HEAD, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240521, org.label-schema.name=CentOS Stream 8 Base Image) 2024-05-27T07:30:20.853 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:30:20 smithi173 ceph-mon[37000]: MDS daemon mds.j is removed because it is dead or otherwise unavailable. 2024-05-27T07:30:20.853 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:30:20 smithi173 ceph-mon[37000]: MDS daemon mds.l is removed because it is dead or otherwise unavailable. 2024-05-27T07:30:20.853 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:30:20 smithi173 ceph-mon[37000]: fsmap cephfs:5 3 up:standby-replay 5 up:active 2024-05-27T07:30:20.853 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:30:20 smithi173 ceph-mon[37000]: daemon mds.k restarted 2024-05-27T07:30:20.941 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:30:20 smithi105 ceph-mon[35977]: MDS daemon mds.j is removed because it is dead or otherwise unavailable. 2024-05-27T07:30:20.941 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:30:20 smithi105 ceph-mon[35977]: MDS daemon mds.l is removed because it is dead or otherwise unavailable. 2024-05-27T07:30:20.941 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:30:20 smithi105 ceph-mon[35977]: fsmap cephfs:5 3 up:standby-replay 5 up:active 2024-05-27T07:30:20.941 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:30:20 smithi105 ceph-mon[35977]: daemon mds.k restarted 2024-05-27T07:30:20.941 INFO:journalctl@ceph.mds.k.smithi105.stdout:May 27 07:30:20 smithi105 podman[70376]: 2024-05-27 07:30:20.495711913 +0000 UTC m=+0.415349391 container init 7ecb8856db56e0f00d5b025b01a7c69ab17ac2a1672e92156e1ba5122cefd159 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-k, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, ceph=True, org.label-schema.build-date=20240521, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, org.label-schema.license=GPLv2) 2024-05-27T07:30:20.941 INFO:journalctl@ceph.mds.k.smithi105.stdout:May 27 07:30:20 smithi105 podman[70376]: 2024-05-27 07:30:20.499386096 +0000 UTC m=+0.419023568 container start 7ecb8856db56e0f00d5b025b01a7c69ab17ac2a1672e92156e1ba5122cefd159 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-k, GIT_CLEAN=True, ceph=True, org.label-schema.build-date=20240521, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90) 2024-05-27T07:30:20.941 INFO:journalctl@ceph.mds.k.smithi105.stdout:May 27 07:30:20 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-k[70390]: starting mds.k at 2024-05-27T07:30:20.942 INFO:journalctl@ceph.mds.k.smithi105.stdout:May 27 07:30:20 smithi105 bash[70376]: 7ecb8856db56e0f00d5b025b01a7c69ab17ac2a1672e92156e1ba5122cefd159 2024-05-27T07:30:20.942 INFO:journalctl@ceph.mds.k.smithi105.stdout:May 27 07:30:20 smithi105 systemd[1]: Started Ceph mds.k for 73df2636-1bf8-11ef-bc9b-c7b262605968. 2024-05-27T07:30:21.442 INFO:journalctl@ceph.mds.k.smithi105.stdout:May 27 07:30:21 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-k[70390]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: In function 'std::set SnapClient::filter(const std::set&) const' thread 7f866cfb6700 time 2024-05-27T07:30:21.009708+0000 2024-05-27T07:30:21.442 INFO:journalctl@ceph.mds.k.smithi105.stdout:May 27 07:30:21 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-k[70390]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: 197: FAILED ceph_assert(cached_version > 0) 2024-05-27T07:30:21.442 INFO:journalctl@ceph.mds.k.smithi105.stdout:May 27 07:30:21 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-k[70390]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:30:21.442 INFO:journalctl@ceph.mds.k.smithi105.stdout:May 27 07:30:21 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-k[70390]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x127) [0x7f867f7fbba6] 2024-05-27T07:30:21.443 INFO:journalctl@ceph.mds.k.smithi105.stdout:May 27 07:30:21 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-k[70390]: 2: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7f867f7fbdd1] 2024-05-27T07:30:21.443 INFO:journalctl@ceph.mds.k.smithi105.stdout:May 27 07:30:21 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-k[70390]: 3: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x5647e7e7623d] 2024-05-27T07:30:21.443 INFO:journalctl@ceph.mds.k.smithi105.stdout:May 27 07:30:21 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-k[70390]: 4: (SnapRealm::build_snap_set() const+0x323) [0x5647e7e61971] 2024-05-27T07:30:21.443 INFO:journalctl@ceph.mds.k.smithi105.stdout:May 27 07:30:21 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-k[70390]: 5: (SnapRealm::check_cache() const+0x13e) [0x5647e7e5ffa0] 2024-05-27T07:30:21.443 INFO:journalctl@ceph.mds.k.smithi105.stdout:May 27 07:30:21 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-k[70390]: 6: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x5647e7e60651] 2024-05-27T07:30:21.443 INFO:journalctl@ceph.mds.k.smithi105.stdout:May 27 07:30:21 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-k[70390]: 7: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x5647e7df5347] 2024-05-27T07:30:21.443 INFO:journalctl@ceph.mds.k.smithi105.stdout:May 27 07:30:21 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-k[70390]: 8: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x5647e7af187e] 2024-05-27T07:30:21.443 INFO:journalctl@ceph.mds.k.smithi105.stdout:May 27 07:30:21 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-k[70390]: 9: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x279d) [0x5647e7af6f2f] 2024-05-27T07:30:21.443 INFO:journalctl@ceph.mds.k.smithi105.stdout:May 27 07:30:21 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-k[70390]: 10: (EUpdate::replay(MDSRank*)+0x63) [0x5647e7b01969] 2024-05-27T07:30:21.443 INFO:journalctl@ceph.mds.k.smithi105.stdout:May 27 07:30:21 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-k[70390]: 11: (MDLog::_replay_thread()+0x1f38) [0x5647e7e9a48c] 2024-05-27T07:30:21.444 INFO:journalctl@ceph.mds.k.smithi105.stdout:May 27 07:30:21 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-k[70390]: 12: (MDLog::ReplayThread::entry()+0x11) [0x5647e7ea39cb] 2024-05-27T07:30:21.444 INFO:journalctl@ceph.mds.k.smithi105.stdout:May 27 07:30:21 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-k[70390]: 13: (Thread::entry_wrapper()+0x43) [0x7f867f7ca72d] 2024-05-27T07:30:21.444 INFO:journalctl@ceph.mds.k.smithi105.stdout:May 27 07:30:21 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-k[70390]: 14: (Thread::_entry_func(void*)+0xd) [0x7f867f7ca749] 2024-05-27T07:30:21.444 INFO:journalctl@ceph.mds.k.smithi105.stdout:May 27 07:30:21 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-k[70390]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7f867e36e1ca] 2024-05-27T07:30:21.444 INFO:journalctl@ceph.mds.k.smithi105.stdout:May 27 07:30:21 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-k[70390]: 16: clone() 2024-05-27T07:30:21.444 INFO:journalctl@ceph.mds.k.smithi105.stdout:May 27 07:30:21 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-k[70390]: 2024-05-27T07:30:21.010+0000 7f866cfb6700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: In function 'std::set SnapClient::filter(const std::set&) const' thread 7f866cfb6700 time 2024-05-27T07:30:21.009708+0000 2024-05-27T07:30:21.444 INFO:journalctl@ceph.mds.k.smithi105.stdout:May 27 07:30:21 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-k[70390]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: 197: FAILED ceph_assert(cached_version > 0) 2024-05-27T07:30:21.444 INFO:journalctl@ceph.mds.k.smithi105.stdout:May 27 07:30:21 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-k[70390]: 2024-05-27T07:30:21.444 INFO:journalctl@ceph.mds.k.smithi105.stdout:May 27 07:30:21 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-k[70390]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:30:21.444 INFO:journalctl@ceph.mds.k.smithi105.stdout:May 27 07:30:21 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-k[70390]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x127) [0x7f867f7fbba6] 2024-05-27T07:30:21.444 INFO:journalctl@ceph.mds.k.smithi105.stdout:May 27 07:30:21 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-k[70390]: 2: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7f867f7fbdd1] 2024-05-27T07:30:21.445 INFO:journalctl@ceph.mds.k.smithi105.stdout:May 27 07:30:21 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-k[70390]: 3: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x5647e7e7623d] 2024-05-27T07:30:21.445 INFO:journalctl@ceph.mds.k.smithi105.stdout:May 27 07:30:21 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-k[70390]: 4: (SnapRealm::build_snap_set() const+0x323) [0x5647e7e61971] 2024-05-27T07:30:21.445 INFO:journalctl@ceph.mds.k.smithi105.stdout:May 27 07:30:21 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-k[70390]: 5: (SnapRealm::check_cache() const+0x13e) [0x5647e7e5ffa0] 2024-05-27T07:30:21.445 INFO:journalctl@ceph.mds.k.smithi105.stdout:May 27 07:30:21 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-k[70390]: 6: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x5647e7e60651] 2024-05-27T07:30:21.445 INFO:journalctl@ceph.mds.k.smithi105.stdout:May 27 07:30:21 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-k[70390]: 7: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x5647e7df5347] 2024-05-27T07:30:21.445 INFO:journalctl@ceph.mds.k.smithi105.stdout:May 27 07:30:21 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-k[70390]: 8: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x5647e7af187e] 2024-05-27T07:30:21.445 INFO:journalctl@ceph.mds.k.smithi105.stdout:May 27 07:30:21 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-k[70390]: 9: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x279d) [0x5647e7af6f2f] 2024-05-27T07:30:21.445 INFO:journalctl@ceph.mds.k.smithi105.stdout:May 27 07:30:21 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-k[70390]: 10: (EUpdate::replay(MDSRank*)+0x63) [0x5647e7b01969] 2024-05-27T07:30:21.445 INFO:journalctl@ceph.mds.k.smithi105.stdout:May 27 07:30:21 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-k[70390]: 11: (MDLog::_replay_thread()+0x1f38) [0x5647e7e9a48c] 2024-05-27T07:30:21.445 INFO:journalctl@ceph.mds.k.smithi105.stdout:May 27 07:30:21 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-k[70390]: 12: (MDLog::ReplayThread::entry()+0x11) [0x5647e7ea39cb] 2024-05-27T07:30:21.445 INFO:journalctl@ceph.mds.k.smithi105.stdout:May 27 07:30:21 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-k[70390]: 13: (Thread::entry_wrapper()+0x43) [0x7f867f7ca72d] 2024-05-27T07:30:21.445 INFO:journalctl@ceph.mds.k.smithi105.stdout:May 27 07:30:21 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-k[70390]: 14: (Thread::_entry_func(void*)+0xd) [0x7f867f7ca749] 2024-05-27T07:30:21.445 INFO:journalctl@ceph.mds.k.smithi105.stdout:May 27 07:30:21 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-k[70390]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7f867e36e1ca] 2024-05-27T07:30:21.446 INFO:journalctl@ceph.mds.k.smithi105.stdout:May 27 07:30:21 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-k[70390]: 16: clone() 2024-05-27T07:30:21.446 INFO:journalctl@ceph.mds.k.smithi105.stdout:May 27 07:30:21 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-k[70390]: 2024-05-27T07:30:21.446 INFO:journalctl@ceph.mds.k.smithi105.stdout:May 27 07:30:21 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-k[70390]: *** Caught signal (Aborted) ** 2024-05-27T07:30:21.446 INFO:journalctl@ceph.mds.k.smithi105.stdout:May 27 07:30:21 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-k[70390]: in thread 7f866cfb6700 thread_name:md_log_replay 2024-05-27T07:30:21.446 INFO:journalctl@ceph.mds.k.smithi105.stdout:May 27 07:30:21 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-k[70390]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:30:21.446 INFO:journalctl@ceph.mds.k.smithi105.stdout:May 27 07:30:21 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-k[70390]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f867e378d20] 2024-05-27T07:30:21.446 INFO:journalctl@ceph.mds.k.smithi105.stdout:May 27 07:30:21 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-k[70390]: 2: gsignal() 2024-05-27T07:30:21.446 INFO:journalctl@ceph.mds.k.smithi105.stdout:May 27 07:30:21 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-k[70390]: 3: abort() 2024-05-27T07:30:21.446 INFO:journalctl@ceph.mds.k.smithi105.stdout:May 27 07:30:21 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-k[70390]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x26e) [0x7f867f7fbced] 2024-05-27T07:30:21.446 INFO:journalctl@ceph.mds.k.smithi105.stdout:May 27 07:30:21 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-k[70390]: 5: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7f867f7fbdd1] 2024-05-27T07:30:21.447 INFO:journalctl@ceph.mds.k.smithi105.stdout:May 27 07:30:21 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-k[70390]: 6: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x5647e7e7623d] 2024-05-27T07:30:21.447 INFO:journalctl@ceph.mds.k.smithi105.stdout:May 27 07:30:21 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-k[70390]: 7: (SnapRealm::build_snap_set() const+0x323) [0x5647e7e61971] 2024-05-27T07:30:21.447 INFO:journalctl@ceph.mds.k.smithi105.stdout:May 27 07:30:21 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-k[70390]: 8: (SnapRealm::check_cache() const+0x13e) [0x5647e7e5ffa0] 2024-05-27T07:30:21.448 INFO:journalctl@ceph.mds.k.smithi105.stdout:May 27 07:30:21 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-k[70390]: 9: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x5647e7e60651] 2024-05-27T07:30:21.448 INFO:journalctl@ceph.mds.k.smithi105.stdout:May 27 07:30:21 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-k[70390]: 10: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x5647e7df5347] 2024-05-27T07:30:21.448 INFO:journalctl@ceph.mds.k.smithi105.stdout:May 27 07:30:21 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-k[70390]: 11: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x5647e7af187e] 2024-05-27T07:30:21.448 INFO:journalctl@ceph.mds.k.smithi105.stdout:May 27 07:30:21 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-k[70390]: 12: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x279d) [0x5647e7af6f2f] 2024-05-27T07:30:21.448 INFO:journalctl@ceph.mds.k.smithi105.stdout:May 27 07:30:21 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-k[70390]: 13: (EUpdate::replay(MDSRank*)+0x63) [0x5647e7b01969] 2024-05-27T07:30:21.448 INFO:journalctl@ceph.mds.k.smithi105.stdout:May 27 07:30:21 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-k[70390]: 14: (MDLog::_replay_thread()+0x1f38) [0x5647e7e9a48c] 2024-05-27T07:30:21.448 INFO:journalctl@ceph.mds.k.smithi105.stdout:May 27 07:30:21 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-k[70390]: 15: (MDLog::ReplayThread::entry()+0x11) [0x5647e7ea39cb] 2024-05-27T07:30:21.448 INFO:journalctl@ceph.mds.k.smithi105.stdout:May 27 07:30:21 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-k[70390]: 16: (Thread::entry_wrapper()+0x43) [0x7f867f7ca72d] 2024-05-27T07:30:21.448 INFO:journalctl@ceph.mds.k.smithi105.stdout:May 27 07:30:21 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-k[70390]: 17: (Thread::_entry_func(void*)+0xd) [0x7f867f7ca749] 2024-05-27T07:30:21.448 INFO:journalctl@ceph.mds.k.smithi105.stdout:May 27 07:30:21 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-k[70390]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7f867e36e1ca] 2024-05-27T07:30:21.448 INFO:journalctl@ceph.mds.k.smithi105.stdout:May 27 07:30:21 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-k[70390]: 19: clone() 2024-05-27T07:30:21.448 INFO:journalctl@ceph.mds.k.smithi105.stdout:May 27 07:30:21 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-k[70390]: 2024-05-27T07:30:21.014+0000 7f866cfb6700 -1 *** Caught signal (Aborted) ** 2024-05-27T07:30:21.448 INFO:journalctl@ceph.mds.k.smithi105.stdout:May 27 07:30:21 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-k[70390]: in thread 7f866cfb6700 thread_name:md_log_replay 2024-05-27T07:30:21.448 INFO:journalctl@ceph.mds.k.smithi105.stdout:May 27 07:30:21 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-k[70390]: 2024-05-27T07:30:21.449 INFO:journalctl@ceph.mds.k.smithi105.stdout:May 27 07:30:21 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-k[70390]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:30:21.449 INFO:journalctl@ceph.mds.k.smithi105.stdout:May 27 07:30:21 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-k[70390]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f867e378d20] 2024-05-27T07:30:21.449 INFO:journalctl@ceph.mds.k.smithi105.stdout:May 27 07:30:21 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-k[70390]: 2: gsignal() 2024-05-27T07:30:21.449 INFO:journalctl@ceph.mds.k.smithi105.stdout:May 27 07:30:21 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-k[70390]: 3: abort() 2024-05-27T07:30:21.449 INFO:journalctl@ceph.mds.k.smithi105.stdout:May 27 07:30:21 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-k[70390]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x26e) [0x7f867f7fbced] 2024-05-27T07:30:21.449 INFO:journalctl@ceph.mds.k.smithi105.stdout:May 27 07:30:21 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-k[70390]: 5: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7f867f7fbdd1] 2024-05-27T07:30:21.449 INFO:journalctl@ceph.mds.k.smithi105.stdout:May 27 07:30:21 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-k[70390]: 6: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x5647e7e7623d] 2024-05-27T07:30:21.449 INFO:journalctl@ceph.mds.k.smithi105.stdout:May 27 07:30:21 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-k[70390]: 7: (SnapRealm::build_snap_set() const+0x323) [0x5647e7e61971] 2024-05-27T07:30:21.449 INFO:journalctl@ceph.mds.k.smithi105.stdout:May 27 07:30:21 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-k[70390]: 8: (SnapRealm::check_cache() const+0x13e) [0x5647e7e5ffa0] 2024-05-27T07:30:21.449 INFO:journalctl@ceph.mds.k.smithi105.stdout:May 27 07:30:21 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-k[70390]: 9: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x5647e7e60651] 2024-05-27T07:30:21.449 INFO:journalctl@ceph.mds.k.smithi105.stdout:May 27 07:30:21 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-k[70390]: 10: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x5647e7df5347] 2024-05-27T07:30:21.449 INFO:journalctl@ceph.mds.k.smithi105.stdout:May 27 07:30:21 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-k[70390]: 11: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x5647e7af187e] 2024-05-27T07:30:21.449 INFO:journalctl@ceph.mds.k.smithi105.stdout:May 27 07:30:21 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-k[70390]: 12: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x279d) [0x5647e7af6f2f] 2024-05-27T07:30:21.450 INFO:journalctl@ceph.mds.k.smithi105.stdout:May 27 07:30:21 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-k[70390]: 13: (EUpdate::replay(MDSRank*)+0x63) [0x5647e7b01969] 2024-05-27T07:30:21.450 INFO:journalctl@ceph.mds.k.smithi105.stdout:May 27 07:30:21 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-k[70390]: 14: (MDLog::_replay_thread()+0x1f38) [0x5647e7e9a48c] 2024-05-27T07:30:21.450 INFO:journalctl@ceph.mds.k.smithi105.stdout:May 27 07:30:21 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-k[70390]: 15: (MDLog::ReplayThread::entry()+0x11) [0x5647e7ea39cb] 2024-05-27T07:30:21.450 INFO:journalctl@ceph.mds.k.smithi105.stdout:May 27 07:30:21 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-k[70390]: 16: (Thread::entry_wrapper()+0x43) [0x7f867f7ca72d] 2024-05-27T07:30:21.450 INFO:journalctl@ceph.mds.k.smithi105.stdout:May 27 07:30:21 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-k[70390]: 17: (Thread::_entry_func(void*)+0xd) [0x7f867f7ca749] 2024-05-27T07:30:21.450 INFO:journalctl@ceph.mds.k.smithi105.stdout:May 27 07:30:21 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-k[70390]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7f867e36e1ca] 2024-05-27T07:30:21.450 INFO:journalctl@ceph.mds.k.smithi105.stdout:May 27 07:30:21 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-k[70390]: 19: clone() 2024-05-27T07:30:21.450 INFO:journalctl@ceph.mds.k.smithi105.stdout:May 27 07:30:21 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-k[70390]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-05-27T07:30:21.450 INFO:journalctl@ceph.mds.k.smithi105.stdout:May 27 07:30:21 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-k[70390]: 2024-05-27T07:30:21.450 INFO:journalctl@ceph.mds.k.smithi105.stdout:May 27 07:30:21 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-k[70390]: -2> 2024-05-27T07:30:21.010+0000 7f866cfb6700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: In function 'std::set SnapClient::filter(const std::set&) const' thread 7f866cfb6700 time 2024-05-27T07:30:21.009708+0000 2024-05-27T07:30:21.450 INFO:journalctl@ceph.mds.k.smithi105.stdout:May 27 07:30:21 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-k[70390]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: 197: FAILED ceph_assert(cached_version > 0) 2024-05-27T07:30:21.450 INFO:journalctl@ceph.mds.k.smithi105.stdout:May 27 07:30:21 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-k[70390]: 2024-05-27T07:30:21.451 INFO:journalctl@ceph.mds.k.smithi105.stdout:May 27 07:30:21 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-k[70390]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:30:21.451 INFO:journalctl@ceph.mds.k.smithi105.stdout:May 27 07:30:21 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-k[70390]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x127) [0x7f867f7fbba6] 2024-05-27T07:30:21.451 INFO:journalctl@ceph.mds.k.smithi105.stdout:May 27 07:30:21 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-k[70390]: 2: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7f867f7fbdd1] 2024-05-27T07:30:21.451 INFO:journalctl@ceph.mds.k.smithi105.stdout:May 27 07:30:21 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-k[70390]: 3: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x5647e7e7623d] 2024-05-27T07:30:21.451 INFO:journalctl@ceph.mds.k.smithi105.stdout:May 27 07:30:21 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-k[70390]: 4: (SnapRealm::build_snap_set() const+0x323) [0x5647e7e61971] 2024-05-27T07:30:21.451 INFO:journalctl@ceph.mds.k.smithi105.stdout:May 27 07:30:21 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-k[70390]: 5: (SnapRealm::check_cache() const+0x13e) [0x5647e7e5ffa0] 2024-05-27T07:30:21.451 INFO:journalctl@ceph.mds.k.smithi105.stdout:May 27 07:30:21 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-k[70390]: 6: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x5647e7e60651] 2024-05-27T07:30:21.451 INFO:journalctl@ceph.mds.k.smithi105.stdout:May 27 07:30:21 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-k[70390]: 7: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x5647e7df5347] 2024-05-27T07:30:21.451 INFO:journalctl@ceph.mds.k.smithi105.stdout:May 27 07:30:21 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-k[70390]: 8: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x5647e7af187e] 2024-05-27T07:30:21.451 INFO:journalctl@ceph.mds.k.smithi105.stdout:May 27 07:30:21 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-k[70390]: 9: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x279d) [0x5647e7af6f2f] 2024-05-27T07:30:21.451 INFO:journalctl@ceph.mds.k.smithi105.stdout:May 27 07:30:21 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-k[70390]: 10: (EUpdate::replay(MDSRank*)+0x63) [0x5647e7b01969] 2024-05-27T07:30:21.451 INFO:journalctl@ceph.mds.k.smithi105.stdout:May 27 07:30:21 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-k[70390]: 11: (MDLog::_replay_thread()+0x1f38) [0x5647e7e9a48c] 2024-05-27T07:30:21.451 INFO:journalctl@ceph.mds.k.smithi105.stdout:May 27 07:30:21 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-k[70390]: 12: (MDLog::ReplayThread::entry()+0x11) [0x5647e7ea39cb] 2024-05-27T07:30:21.452 INFO:journalctl@ceph.mds.k.smithi105.stdout:May 27 07:30:21 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-k[70390]: 13: (Thread::entry_wrapper()+0x43) [0x7f867f7ca72d] 2024-05-27T07:30:21.452 INFO:journalctl@ceph.mds.k.smithi105.stdout:May 27 07:30:21 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-k[70390]: 14: (Thread::_entry_func(void*)+0xd) [0x7f867f7ca749] 2024-05-27T07:30:21.452 INFO:journalctl@ceph.mds.k.smithi105.stdout:May 27 07:30:21 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-k[70390]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7f867e36e1ca] 2024-05-27T07:30:21.452 INFO:journalctl@ceph.mds.k.smithi105.stdout:May 27 07:30:21 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-k[70390]: 16: clone() 2024-05-27T07:30:21.452 INFO:journalctl@ceph.mds.k.smithi105.stdout:May 27 07:30:21 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-k[70390]: 2024-05-27T07:30:21.452 INFO:journalctl@ceph.mds.k.smithi105.stdout:May 27 07:30:21 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-k[70390]: 0> 2024-05-27T07:30:21.014+0000 7f866cfb6700 -1 *** Caught signal (Aborted) ** 2024-05-27T07:30:21.452 INFO:journalctl@ceph.mds.k.smithi105.stdout:May 27 07:30:21 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-k[70390]: in thread 7f866cfb6700 thread_name:md_log_replay 2024-05-27T07:30:21.452 INFO:journalctl@ceph.mds.k.smithi105.stdout:May 27 07:30:21 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-k[70390]: 2024-05-27T07:30:21.452 INFO:journalctl@ceph.mds.k.smithi105.stdout:May 27 07:30:21 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-k[70390]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:30:21.452 INFO:journalctl@ceph.mds.k.smithi105.stdout:May 27 07:30:21 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-k[70390]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f867e378d20] 2024-05-27T07:30:21.452 INFO:journalctl@ceph.mds.k.smithi105.stdout:May 27 07:30:21 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-k[70390]: 2: gsignal() 2024-05-27T07:30:21.453 INFO:journalctl@ceph.mds.k.smithi105.stdout:May 27 07:30:21 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-k[70390]: 3: abort() 2024-05-27T07:30:21.453 INFO:journalctl@ceph.mds.k.smithi105.stdout:May 27 07:30:21 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-k[70390]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x26e) [0x7f867f7fbced] 2024-05-27T07:30:21.453 INFO:journalctl@ceph.mds.k.smithi105.stdout:May 27 07:30:21 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-k[70390]: 5: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7f867f7fbdd1] 2024-05-27T07:30:21.453 INFO:journalctl@ceph.mds.k.smithi105.stdout:May 27 07:30:21 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-k[70390]: 6: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x5647e7e7623d] 2024-05-27T07:30:21.453 INFO:journalctl@ceph.mds.k.smithi105.stdout:May 27 07:30:21 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-k[70390]: 7: (SnapRealm::build_snap_set() const+0x323) [0x5647e7e61971] 2024-05-27T07:30:21.453 INFO:journalctl@ceph.mds.k.smithi105.stdout:May 27 07:30:21 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-k[70390]: 8: (SnapRealm::check_cache() const+0x13e) [0x5647e7e5ffa0] 2024-05-27T07:30:21.453 INFO:journalctl@ceph.mds.k.smithi105.stdout:May 27 07:30:21 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-k[70390]: 9: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x5647e7e60651] 2024-05-27T07:30:21.453 INFO:journalctl@ceph.mds.k.smithi105.stdout:May 27 07:30:21 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-k[70390]: 10: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x5647e7df5347] 2024-05-27T07:30:21.453 INFO:journalctl@ceph.mds.k.smithi105.stdout:May 27 07:30:21 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-k[70390]: 11: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x5647e7af187e] 2024-05-27T07:30:21.453 INFO:journalctl@ceph.mds.k.smithi105.stdout:May 27 07:30:21 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-k[70390]: 12: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x279d) [0x5647e7af6f2f] 2024-05-27T07:30:21.453 INFO:journalctl@ceph.mds.k.smithi105.stdout:May 27 07:30:21 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-k[70390]: 13: (EUpdate::replay(MDSRank*)+0x63) [0x5647e7b01969] 2024-05-27T07:30:21.454 INFO:journalctl@ceph.mds.k.smithi105.stdout:May 27 07:30:21 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-k[70390]: 14: (MDLog::_replay_thread()+0x1f38) [0x5647e7e9a48c] 2024-05-27T07:30:21.454 INFO:journalctl@ceph.mds.k.smithi105.stdout:May 27 07:30:21 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-k[70390]: 15: (MDLog::ReplayThread::entry()+0x11) [0x5647e7ea39cb] 2024-05-27T07:30:21.454 INFO:journalctl@ceph.mds.k.smithi105.stdout:May 27 07:30:21 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-k[70390]: 16: (Thread::entry_wrapper()+0x43) [0x7f867f7ca72d] 2024-05-27T07:30:21.454 INFO:journalctl@ceph.mds.k.smithi105.stdout:May 27 07:30:21 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-k[70390]: 17: (Thread::_entry_func(void*)+0xd) [0x7f867f7ca749] 2024-05-27T07:30:21.454 INFO:journalctl@ceph.mds.k.smithi105.stdout:May 27 07:30:21 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-k[70390]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7f867e36e1ca] 2024-05-27T07:30:21.454 INFO:journalctl@ceph.mds.k.smithi105.stdout:May 27 07:30:21 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-k[70390]: 19: clone() 2024-05-27T07:30:21.454 INFO:journalctl@ceph.mds.k.smithi105.stdout:May 27 07:30:21 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-k[70390]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-05-27T07:30:21.454 INFO:journalctl@ceph.mds.k.smithi105.stdout:May 27 07:30:21 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-k[70390]: 2024-05-27T07:30:21.454 INFO:journalctl@ceph.mds.k.smithi105.stdout:May 27 07:30:21 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-k[70390]: -3> 2024-05-27T07:30:21.010+0000 7f866cfb6700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: In function 'std::set SnapClient::filter(const std::set&) const' thread 7f866cfb6700 time 2024-05-27T07:30:21.009708+0000 2024-05-27T07:30:21.454 INFO:journalctl@ceph.mds.k.smithi105.stdout:May 27 07:30:21 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-k[70390]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: 197: FAILED ceph_assert(cached_version > 0) 2024-05-27T07:30:21.455 INFO:journalctl@ceph.mds.k.smithi105.stdout:May 27 07:30:21 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-k[70390]: 2024-05-27T07:30:21.455 INFO:journalctl@ceph.mds.k.smithi105.stdout:May 27 07:30:21 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-k[70390]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:30:21.455 INFO:journalctl@ceph.mds.k.smithi105.stdout:May 27 07:30:21 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-k[70390]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x127) [0x7f867f7fbba6] 2024-05-27T07:30:21.455 INFO:journalctl@ceph.mds.k.smithi105.stdout:May 27 07:30:21 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-k[70390]: 2: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7f867f7fbdd1] 2024-05-27T07:30:21.455 INFO:journalctl@ceph.mds.k.smithi105.stdout:May 27 07:30:21 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-k[70390]: 3: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x5647e7e7623d] 2024-05-27T07:30:21.455 INFO:journalctl@ceph.mds.k.smithi105.stdout:May 27 07:30:21 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-k[70390]: 4: (SnapRealm::build_snap_set() const+0x323) [0x5647e7e61971] 2024-05-27T07:30:21.455 INFO:journalctl@ceph.mds.k.smithi105.stdout:May 27 07:30:21 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-k[70390]: 5: (SnapRealm::check_cache() const+0x13e) [0x5647e7e5ffa0] 2024-05-27T07:30:21.455 INFO:journalctl@ceph.mds.k.smithi105.stdout:May 27 07:30:21 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-k[70390]: 6: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x5647e7e60651] 2024-05-27T07:30:21.455 INFO:journalctl@ceph.mds.k.smithi105.stdout:May 27 07:30:21 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-k[70390]: 7: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x5647e7df5347] 2024-05-27T07:30:21.455 INFO:journalctl@ceph.mds.k.smithi105.stdout:May 27 07:30:21 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-k[70390]: 8: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x5647e7af187e] 2024-05-27T07:30:21.455 INFO:journalctl@ceph.mds.k.smithi105.stdout:May 27 07:30:21 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-k[70390]: 9: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x279d) [0x5647e7af6f2f] 2024-05-27T07:30:21.455 INFO:journalctl@ceph.mds.k.smithi105.stdout:May 27 07:30:21 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-k[70390]: 10: (EUpdate::replay(MDSRank*)+0x63) [0x5647e7b01969] 2024-05-27T07:30:21.455 INFO:journalctl@ceph.mds.k.smithi105.stdout:May 27 07:30:21 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-k[70390]: 11: (MDLog::_replay_thread()+0x1f38) [0x5647e7e9a48c] 2024-05-27T07:30:21.456 INFO:journalctl@ceph.mds.k.smithi105.stdout:May 27 07:30:21 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-k[70390]: 12: (MDLog::ReplayThread::entry()+0x11) [0x5647e7ea39cb] 2024-05-27T07:30:21.456 INFO:journalctl@ceph.mds.k.smithi105.stdout:May 27 07:30:21 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-k[70390]: 13: (Thread::entry_wrapper()+0x43) [0x7f867f7ca72d] 2024-05-27T07:30:21.456 INFO:journalctl@ceph.mds.k.smithi105.stdout:May 27 07:30:21 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-k[70390]: 14: (Thread::_entry_func(void*)+0xd) [0x7f867f7ca749] 2024-05-27T07:30:21.456 INFO:journalctl@ceph.mds.k.smithi105.stdout:May 27 07:30:21 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-k[70390]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7f867e36e1ca] 2024-05-27T07:30:21.456 INFO:journalctl@ceph.mds.k.smithi105.stdout:May 27 07:30:21 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-k[70390]: 16: clone() 2024-05-27T07:30:21.456 INFO:journalctl@ceph.mds.k.smithi105.stdout:May 27 07:30:21 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-k[70390]: 2024-05-27T07:30:21.456 INFO:journalctl@ceph.mds.k.smithi105.stdout:May 27 07:30:21 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-k[70390]: -1> 2024-05-27T07:30:21.014+0000 7f866cfb6700 -1 *** Caught signal (Aborted) ** 2024-05-27T07:30:21.456 INFO:journalctl@ceph.mds.k.smithi105.stdout:May 27 07:30:21 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-k[70390]: in thread 7f866cfb6700 thread_name:md_log_replay 2024-05-27T07:30:21.456 INFO:journalctl@ceph.mds.k.smithi105.stdout:May 27 07:30:21 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-k[70390]: 2024-05-27T07:30:21.456 INFO:journalctl@ceph.mds.k.smithi105.stdout:May 27 07:30:21 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-k[70390]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:30:21.457 INFO:journalctl@ceph.mds.k.smithi105.stdout:May 27 07:30:21 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-k[70390]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f867e378d20] 2024-05-27T07:30:21.457 INFO:journalctl@ceph.mds.k.smithi105.stdout:May 27 07:30:21 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-k[70390]: 2: gsignal() 2024-05-27T07:30:21.457 INFO:journalctl@ceph.mds.k.smithi105.stdout:May 27 07:30:21 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-k[70390]: 3: abort() 2024-05-27T07:30:21.457 INFO:journalctl@ceph.mds.k.smithi105.stdout:May 27 07:30:21 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-k[70390]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x26e) [0x7f867f7fbced] 2024-05-27T07:30:21.457 INFO:journalctl@ceph.mds.k.smithi105.stdout:May 27 07:30:21 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-k[70390]: 5: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7f867f7fbdd1] 2024-05-27T07:30:21.457 INFO:journalctl@ceph.mds.k.smithi105.stdout:May 27 07:30:21 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-k[70390]: 6: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x5647e7e7623d] 2024-05-27T07:30:21.457 INFO:journalctl@ceph.mds.k.smithi105.stdout:May 27 07:30:21 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-k[70390]: 7: (SnapRealm::build_snap_set() const+0x323) [0x5647e7e61971] 2024-05-27T07:30:21.458 INFO:journalctl@ceph.mds.k.smithi105.stdout:May 27 07:30:21 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-k[70390]: 8: (SnapRealm::check_cache() const+0x13e) [0x5647e7e5ffa0] 2024-05-27T07:30:21.458 INFO:journalctl@ceph.mds.k.smithi105.stdout:May 27 07:30:21 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-k[70390]: 9: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x5647e7e60651] 2024-05-27T07:30:21.458 INFO:journalctl@ceph.mds.k.smithi105.stdout:May 27 07:30:21 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-k[70390]: 10: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x5647e7df5347] 2024-05-27T07:30:21.458 INFO:journalctl@ceph.mds.k.smithi105.stdout:May 27 07:30:21 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-k[70390]: 11: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x5647e7af187e] 2024-05-27T07:30:21.458 INFO:journalctl@ceph.mds.k.smithi105.stdout:May 27 07:30:21 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-k[70390]: 12: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x279d) [0x5647e7af6f2f] 2024-05-27T07:30:21.458 INFO:journalctl@ceph.mds.k.smithi105.stdout:May 27 07:30:21 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-k[70390]: 13: (EUpdate::replay(MDSRank*)+0x63) [0x5647e7b01969] 2024-05-27T07:30:21.458 INFO:journalctl@ceph.mds.k.smithi105.stdout:May 27 07:30:21 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-k[70390]: 14: (MDLog::_replay_thread()+0x1f38) [0x5647e7e9a48c] 2024-05-27T07:30:21.458 INFO:journalctl@ceph.mds.k.smithi105.stdout:May 27 07:30:21 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-k[70390]: 15: (MDLog::ReplayThread::entry()+0x11) [0x5647e7ea39cb] 2024-05-27T07:30:21.458 INFO:journalctl@ceph.mds.k.smithi105.stdout:May 27 07:30:21 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-k[70390]: 16: (Thread::entry_wrapper()+0x43) [0x7f867f7ca72d] 2024-05-27T07:30:21.458 INFO:journalctl@ceph.mds.k.smithi105.stdout:May 27 07:30:21 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-k[70390]: 17: (Thread::_entry_func(void*)+0xd) [0x7f867f7ca749] 2024-05-27T07:30:21.458 INFO:journalctl@ceph.mds.k.smithi105.stdout:May 27 07:30:21 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-k[70390]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7f867e36e1ca] 2024-05-27T07:30:21.459 INFO:journalctl@ceph.mds.k.smithi105.stdout:May 27 07:30:21 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-k[70390]: 19: clone() 2024-05-27T07:30:21.459 INFO:journalctl@ceph.mds.k.smithi105.stdout:May 27 07:30:21 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-k[70390]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-05-27T07:30:21.459 INFO:journalctl@ceph.mds.k.smithi105.stdout:May 27 07:30:21 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-k[70390]: 2024-05-27T07:30:21.459 INFO:journalctl@ceph.mds.k.smithi105.stdout:May 27 07:30:21 smithi105 podman[70425]: 2024-05-27 07:30:21.058367342 +0000 UTC m=+0.018694248 container died 7ecb8856db56e0f00d5b025b01a7c69ab17ac2a1672e92156e1ba5122cefd159 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-k, org.label-schema.build-date=20240521, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, org.label-schema.schema-version=1.0, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=) 2024-05-27T07:30:21.482 INFO:journalctl@ceph.mds.j.smithi052.stdout:May 27 07:30:21 smithi052 podman[107731]: 2024-05-27 07:30:21.023860136 +0000 UTC m=+0.410320491 container init 332ab7d985b17bcd2765ea36a1f8438aec4b0d8a6a4108995ab221d58bbcb935 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-j, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, GIT_CLEAN=True, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240521, org.label-schema.name=CentOS Stream 8 Base Image) 2024-05-27T07:30:21.482 INFO:journalctl@ceph.mds.j.smithi052.stdout:May 27 07:30:21 smithi052 podman[107731]: 2024-05-27 07:30:21.026828702 +0000 UTC m=+0.413289032 container start 332ab7d985b17bcd2765ea36a1f8438aec4b0d8a6a4108995ab221d58bbcb935 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-j, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, GIT_CLEAN=True, org.label-schema.build-date=20240521, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True) 2024-05-27T07:30:21.482 INFO:journalctl@ceph.mds.j.smithi052.stdout:May 27 07:30:21 smithi052 ceph-mds[107750]: -- 172.21.15.52:0/3587031688 <== mon.0 v2:172.21.15.52:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55b0be387a40 con 0x55b0be3cfa80 2024-05-27T07:30:21.482 INFO:journalctl@ceph.mds.j.smithi052.stdout:May 27 07:30:21 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-j[107746]: starting mds.j at 2024-05-27T07:30:21.482 INFO:journalctl@ceph.mds.j.smithi052.stdout:May 27 07:30:21 smithi052 bash[107731]: 332ab7d985b17bcd2765ea36a1f8438aec4b0d8a6a4108995ab221d58bbcb935 2024-05-27T07:30:21.482 INFO:journalctl@ceph.mds.j.smithi052.stdout:May 27 07:30:21 smithi052 systemd[1]: Started Ceph mds.j for 73df2636-1bf8-11ef-bc9b-c7b262605968. 2024-05-27T07:30:21.852 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:30:21 smithi105 ceph-mon[35977]: osdmap e113: 12 total, 12 up, 12 in 2024-05-27T07:30:21.852 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:30:21 smithi105 ceph-mon[35977]: mds.? [v2:172.21.15.105:6832/1939755335,v1:172.21.15.105:6833/1939755335] up:boot 2024-05-27T07:30:21.852 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:30:21 smithi105 ceph-mon[35977]: fsmap cephfs:5 2 up:standby-replay 1 up:standby 5 up:active 2024-05-27T07:30:21.852 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:30:21 smithi105 ceph-mon[35977]: from='mgr.15333 172.21.15.52:0/416389082' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "k"}]: dispatch 2024-05-27T07:30:21.852 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:30:21 smithi105 ceph-mon[35977]: fsmap cephfs:5 3 up:standby-replay 5 up:active 2024-05-27T07:30:21.852 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:30:21 smithi105 ceph-mon[35977]: pgmap v157: 129 pgs: 129 active+clean; 366 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 MiB/s rd, 4.8 MiB/s wr, 1.45k op/s 2024-05-27T07:30:21.853 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:30:21 smithi173 ceph-mon[37000]: osdmap e113: 12 total, 12 up, 12 in 2024-05-27T07:30:21.853 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:30:21 smithi173 ceph-mon[37000]: mds.? [v2:172.21.15.105:6832/1939755335,v1:172.21.15.105:6833/1939755335] up:boot 2024-05-27T07:30:21.853 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:30:21 smithi173 ceph-mon[37000]: fsmap cephfs:5 2 up:standby-replay 1 up:standby 5 up:active 2024-05-27T07:30:21.853 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:30:21 smithi173 ceph-mon[37000]: from='mgr.15333 172.21.15.52:0/416389082' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "k"}]: dispatch 2024-05-27T07:30:21.853 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:30:21 smithi173 ceph-mon[37000]: fsmap cephfs:5 3 up:standby-replay 5 up:active 2024-05-27T07:30:21.853 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:30:21 smithi173 ceph-mon[37000]: pgmap v157: 129 pgs: 129 active+clean; 366 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 MiB/s rd, 4.8 MiB/s wr, 1.45k op/s 2024-05-27T07:30:21.980 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:30:21 smithi052 ceph-mon[31955]: osdmap e113: 12 total, 12 up, 12 in 2024-05-27T07:30:21.981 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:30:21 smithi052 ceph-mon[31955]: mds.? [v2:172.21.15.105:6832/1939755335,v1:172.21.15.105:6833/1939755335] up:boot 2024-05-27T07:30:21.981 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:30:21 smithi052 ceph-mon[31955]: fsmap cephfs:5 2 up:standby-replay 1 up:standby 5 up:active 2024-05-27T07:30:21.981 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:30:21 smithi052 ceph-mon[31955]: from='mgr.15333 172.21.15.52:0/416389082' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "k"}]: dispatch 2024-05-27T07:30:21.981 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:30:21 smithi052 ceph-mon[31955]: fsmap cephfs:5 3 up:standby-replay 5 up:active 2024-05-27T07:30:21.981 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:30:21 smithi052 ceph-mon[31955]: pgmap v157: 129 pgs: 129 active+clean; 366 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 MiB/s rd, 4.8 MiB/s wr, 1.45k op/s 2024-05-27T07:30:22.982 INFO:journalctl@ceph.mds.j.smithi052.stdout:May 27 07:30:22 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-j[107746]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: In function 'std::set SnapClient::filter(const std::set&) const' thread 7fb29f53e700 time 2024-05-27T07:30:22.491858+0000 2024-05-27T07:30:22.982 INFO:journalctl@ceph.mds.j.smithi052.stdout:May 27 07:30:22 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-j[107746]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: 197: FAILED ceph_assert(cached_version > 0) 2024-05-27T07:30:22.982 INFO:journalctl@ceph.mds.j.smithi052.stdout:May 27 07:30:22 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-j[107746]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:30:22.982 INFO:journalctl@ceph.mds.j.smithi052.stdout:May 27 07:30:22 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-j[107746]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x127) [0x7fb2b1d83ba6] 2024-05-27T07:30:22.982 INFO:journalctl@ceph.mds.j.smithi052.stdout:May 27 07:30:22 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-j[107746]: 2: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7fb2b1d83dd1] 2024-05-27T07:30:22.982 INFO:journalctl@ceph.mds.j.smithi052.stdout:May 27 07:30:22 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-j[107746]: 3: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x55b0bc72c23d] 2024-05-27T07:30:22.982 INFO:journalctl@ceph.mds.j.smithi052.stdout:May 27 07:30:22 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-j[107746]: 4: (SnapRealm::build_snap_set() const+0x323) [0x55b0bc717971] 2024-05-27T07:30:22.982 INFO:journalctl@ceph.mds.j.smithi052.stdout:May 27 07:30:22 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-j[107746]: 5: (SnapRealm::check_cache() const+0x13e) [0x55b0bc715fa0] 2024-05-27T07:30:22.983 INFO:journalctl@ceph.mds.j.smithi052.stdout:May 27 07:30:22 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-j[107746]: 6: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x55b0bc716651] 2024-05-27T07:30:22.983 INFO:journalctl@ceph.mds.j.smithi052.stdout:May 27 07:30:22 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-j[107746]: 7: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x55b0bc6ab347] 2024-05-27T07:30:22.983 INFO:journalctl@ceph.mds.j.smithi052.stdout:May 27 07:30:22 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-j[107746]: 8: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x55b0bc3a787e] 2024-05-27T07:30:22.983 INFO:journalctl@ceph.mds.j.smithi052.stdout:May 27 07:30:22 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-j[107746]: 9: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x2702) [0x55b0bc3ace94] 2024-05-27T07:30:22.983 INFO:journalctl@ceph.mds.j.smithi052.stdout:May 27 07:30:22 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-j[107746]: 10: (EUpdate::replay(MDSRank*)+0x63) [0x55b0bc3b7969] 2024-05-27T07:30:22.983 INFO:journalctl@ceph.mds.j.smithi052.stdout:May 27 07:30:22 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-j[107746]: 11: (MDLog::_replay_thread()+0x1f38) [0x55b0bc75048c] 2024-05-27T07:30:22.983 INFO:journalctl@ceph.mds.j.smithi052.stdout:May 27 07:30:22 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-j[107746]: 12: (MDLog::ReplayThread::entry()+0x11) [0x55b0bc7599cb] 2024-05-27T07:30:22.983 INFO:journalctl@ceph.mds.j.smithi052.stdout:May 27 07:30:22 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-j[107746]: 13: (Thread::entry_wrapper()+0x43) [0x7fb2b1d5272d] 2024-05-27T07:30:22.983 INFO:journalctl@ceph.mds.j.smithi052.stdout:May 27 07:30:22 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-j[107746]: 14: (Thread::_entry_func(void*)+0xd) [0x7fb2b1d52749] 2024-05-27T07:30:22.983 INFO:journalctl@ceph.mds.j.smithi052.stdout:May 27 07:30:22 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-j[107746]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7fb2b08f61ca] 2024-05-27T07:30:22.983 INFO:journalctl@ceph.mds.j.smithi052.stdout:May 27 07:30:22 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-j[107746]: 16: clone() 2024-05-27T07:30:22.983 INFO:journalctl@ceph.mds.j.smithi052.stdout:May 27 07:30:22 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-j[107746]: 2024-05-27T07:30:22.494+0000 7fb29f53e700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: In function 'std::set SnapClient::filter(const std::set&) const' thread 7fb29f53e700 time 2024-05-27T07:30:22.491858+0000 2024-05-27T07:30:22.984 INFO:journalctl@ceph.mds.j.smithi052.stdout:May 27 07:30:22 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-j[107746]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: 197: FAILED ceph_assert(cached_version > 0) 2024-05-27T07:30:22.984 INFO:journalctl@ceph.mds.j.smithi052.stdout:May 27 07:30:22 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-j[107746]: 2024-05-27T07:30:22.984 INFO:journalctl@ceph.mds.j.smithi052.stdout:May 27 07:30:22 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-j[107746]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:30:22.984 INFO:journalctl@ceph.mds.j.smithi052.stdout:May 27 07:30:22 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-j[107746]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x127) [0x7fb2b1d83ba6] 2024-05-27T07:30:22.984 INFO:journalctl@ceph.mds.j.smithi052.stdout:May 27 07:30:22 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-j[107746]: 2: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7fb2b1d83dd1] 2024-05-27T07:30:22.984 INFO:journalctl@ceph.mds.j.smithi052.stdout:May 27 07:30:22 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-j[107746]: 3: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x55b0bc72c23d] 2024-05-27T07:30:22.984 INFO:journalctl@ceph.mds.j.smithi052.stdout:May 27 07:30:22 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-j[107746]: 4: (SnapRealm::build_snap_set() const+0x323) [0x55b0bc717971] 2024-05-27T07:30:22.985 INFO:journalctl@ceph.mds.j.smithi052.stdout:May 27 07:30:22 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-j[107746]: 5: (SnapRealm::check_cache() const+0x13e) [0x55b0bc715fa0] 2024-05-27T07:30:22.985 INFO:journalctl@ceph.mds.j.smithi052.stdout:May 27 07:30:22 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-j[107746]: 6: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x55b0bc716651] 2024-05-27T07:30:22.985 INFO:journalctl@ceph.mds.j.smithi052.stdout:May 27 07:30:22 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-j[107746]: 7: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x55b0bc6ab347] 2024-05-27T07:30:22.985 INFO:journalctl@ceph.mds.j.smithi052.stdout:May 27 07:30:22 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-j[107746]: 8: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x55b0bc3a787e] 2024-05-27T07:30:22.985 INFO:journalctl@ceph.mds.j.smithi052.stdout:May 27 07:30:22 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-j[107746]: 9: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x2702) [0x55b0bc3ace94] 2024-05-27T07:30:22.985 INFO:journalctl@ceph.mds.j.smithi052.stdout:May 27 07:30:22 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-j[107746]: 10: (EUpdate::replay(MDSRank*)+0x63) [0x55b0bc3b7969] 2024-05-27T07:30:22.985 INFO:journalctl@ceph.mds.j.smithi052.stdout:May 27 07:30:22 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-j[107746]: 11: (MDLog::_replay_thread()+0x1f38) [0x55b0bc75048c] 2024-05-27T07:30:22.985 INFO:journalctl@ceph.mds.j.smithi052.stdout:May 27 07:30:22 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-j[107746]: 12: (MDLog::ReplayThread::entry()+0x11) [0x55b0bc7599cb] 2024-05-27T07:30:22.985 INFO:journalctl@ceph.mds.j.smithi052.stdout:May 27 07:30:22 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-j[107746]: 13: (Thread::entry_wrapper()+0x43) [0x7fb2b1d5272d] 2024-05-27T07:30:22.985 INFO:journalctl@ceph.mds.j.smithi052.stdout:May 27 07:30:22 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-j[107746]: 14: (Thread::_entry_func(void*)+0xd) [0x7fb2b1d52749] 2024-05-27T07:30:22.985 INFO:journalctl@ceph.mds.j.smithi052.stdout:May 27 07:30:22 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-j[107746]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7fb2b08f61ca] 2024-05-27T07:30:22.986 INFO:journalctl@ceph.mds.j.smithi052.stdout:May 27 07:30:22 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-j[107746]: 16: clone() 2024-05-27T07:30:22.986 INFO:journalctl@ceph.mds.j.smithi052.stdout:May 27 07:30:22 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-j[107746]: 2024-05-27T07:30:22.986 INFO:journalctl@ceph.mds.j.smithi052.stdout:May 27 07:30:22 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-j[107746]: *** Caught signal (Aborted) ** 2024-05-27T07:30:22.986 INFO:journalctl@ceph.mds.j.smithi052.stdout:May 27 07:30:22 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-j[107746]: in thread 7fb29f53e700 thread_name:md_log_replay 2024-05-27T07:30:22.986 INFO:journalctl@ceph.mds.j.smithi052.stdout:May 27 07:30:22 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-j[107746]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:30:22.986 INFO:journalctl@ceph.mds.j.smithi052.stdout:May 27 07:30:22 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-j[107746]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7fb2b0900d20] 2024-05-27T07:30:22.986 INFO:journalctl@ceph.mds.j.smithi052.stdout:May 27 07:30:22 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-j[107746]: 2: gsignal() 2024-05-27T07:30:22.986 INFO:journalctl@ceph.mds.j.smithi052.stdout:May 27 07:30:22 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-j[107746]: 3: abort() 2024-05-27T07:30:22.986 INFO:journalctl@ceph.mds.j.smithi052.stdout:May 27 07:30:22 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-j[107746]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x26e) [0x7fb2b1d83ced] 2024-05-27T07:30:22.986 INFO:journalctl@ceph.mds.j.smithi052.stdout:May 27 07:30:22 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-j[107746]: 5: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7fb2b1d83dd1] 2024-05-27T07:30:22.986 INFO:journalctl@ceph.mds.j.smithi052.stdout:May 27 07:30:22 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-j[107746]: 6: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x55b0bc72c23d] 2024-05-27T07:30:22.987 INFO:journalctl@ceph.mds.j.smithi052.stdout:May 27 07:30:22 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-j[107746]: 7: (SnapRealm::build_snap_set() const+0x323) [0x55b0bc717971] 2024-05-27T07:30:22.987 INFO:journalctl@ceph.mds.j.smithi052.stdout:May 27 07:30:22 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-j[107746]: 8: (SnapRealm::check_cache() const+0x13e) [0x55b0bc715fa0] 2024-05-27T07:30:22.987 INFO:journalctl@ceph.mds.j.smithi052.stdout:May 27 07:30:22 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-j[107746]: 9: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x55b0bc716651] 2024-05-27T07:30:22.987 INFO:journalctl@ceph.mds.j.smithi052.stdout:May 27 07:30:22 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-j[107746]: 10: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x55b0bc6ab347] 2024-05-27T07:30:22.987 INFO:journalctl@ceph.mds.j.smithi052.stdout:May 27 07:30:22 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-j[107746]: 11: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x55b0bc3a787e] 2024-05-27T07:30:22.987 INFO:journalctl@ceph.mds.j.smithi052.stdout:May 27 07:30:22 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-j[107746]: 12: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x2702) [0x55b0bc3ace94] 2024-05-27T07:30:22.987 INFO:journalctl@ceph.mds.j.smithi052.stdout:May 27 07:30:22 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-j[107746]: 13: (EUpdate::replay(MDSRank*)+0x63) [0x55b0bc3b7969] 2024-05-27T07:30:22.987 INFO:journalctl@ceph.mds.j.smithi052.stdout:May 27 07:30:22 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-j[107746]: 14: (MDLog::_replay_thread()+0x1f38) [0x55b0bc75048c] 2024-05-27T07:30:22.987 INFO:journalctl@ceph.mds.j.smithi052.stdout:May 27 07:30:22 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-j[107746]: 15: (MDLog::ReplayThread::entry()+0x11) [0x55b0bc7599cb] 2024-05-27T07:30:22.987 INFO:journalctl@ceph.mds.j.smithi052.stdout:May 27 07:30:22 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-j[107746]: 16: (Thread::entry_wrapper()+0x43) [0x7fb2b1d5272d] 2024-05-27T07:30:22.987 INFO:journalctl@ceph.mds.j.smithi052.stdout:May 27 07:30:22 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-j[107746]: 17: (Thread::_entry_func(void*)+0xd) [0x7fb2b1d52749] 2024-05-27T07:30:22.987 INFO:journalctl@ceph.mds.j.smithi052.stdout:May 27 07:30:22 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-j[107746]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7fb2b08f61ca] 2024-05-27T07:30:22.987 INFO:journalctl@ceph.mds.j.smithi052.stdout:May 27 07:30:22 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-j[107746]: 19: clone() 2024-05-27T07:30:22.988 INFO:journalctl@ceph.mds.j.smithi052.stdout:May 27 07:30:22 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-j[107746]: 2024-05-27T07:30:22.498+0000 7fb29f53e700 -1 *** Caught signal (Aborted) ** 2024-05-27T07:30:22.988 INFO:journalctl@ceph.mds.j.smithi052.stdout:May 27 07:30:22 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-j[107746]: in thread 7fb29f53e700 thread_name:md_log_replay 2024-05-27T07:30:22.988 INFO:journalctl@ceph.mds.j.smithi052.stdout:May 27 07:30:22 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-j[107746]: 2024-05-27T07:30:22.988 INFO:journalctl@ceph.mds.j.smithi052.stdout:May 27 07:30:22 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-j[107746]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:30:22.988 INFO:journalctl@ceph.mds.j.smithi052.stdout:May 27 07:30:22 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-j[107746]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7fb2b0900d20] 2024-05-27T07:30:22.988 INFO:journalctl@ceph.mds.j.smithi052.stdout:May 27 07:30:22 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-j[107746]: 2: gsignal() 2024-05-27T07:30:22.988 INFO:journalctl@ceph.mds.j.smithi052.stdout:May 27 07:30:22 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-j[107746]: 3: abort() 2024-05-27T07:30:22.988 INFO:journalctl@ceph.mds.j.smithi052.stdout:May 27 07:30:22 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-j[107746]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x26e) [0x7fb2b1d83ced] 2024-05-27T07:30:22.988 INFO:journalctl@ceph.mds.j.smithi052.stdout:May 27 07:30:22 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-j[107746]: 5: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7fb2b1d83dd1] 2024-05-27T07:30:22.988 INFO:journalctl@ceph.mds.j.smithi052.stdout:May 27 07:30:22 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-j[107746]: 6: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x55b0bc72c23d] 2024-05-27T07:30:22.988 INFO:journalctl@ceph.mds.j.smithi052.stdout:May 27 07:30:22 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-j[107746]: 7: (SnapRealm::build_snap_set() const+0x323) [0x55b0bc717971] 2024-05-27T07:30:22.988 INFO:journalctl@ceph.mds.j.smithi052.stdout:May 27 07:30:22 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-j[107746]: 8: (SnapRealm::check_cache() const+0x13e) [0x55b0bc715fa0] 2024-05-27T07:30:22.988 INFO:journalctl@ceph.mds.j.smithi052.stdout:May 27 07:30:22 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-j[107746]: 9: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x55b0bc716651] 2024-05-27T07:30:22.989 INFO:journalctl@ceph.mds.j.smithi052.stdout:May 27 07:30:22 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-j[107746]: 10: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x55b0bc6ab347] 2024-05-27T07:30:22.989 INFO:journalctl@ceph.mds.j.smithi052.stdout:May 27 07:30:22 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-j[107746]: 11: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x55b0bc3a787e] 2024-05-27T07:30:22.989 INFO:journalctl@ceph.mds.j.smithi052.stdout:May 27 07:30:22 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-j[107746]: 12: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x2702) [0x55b0bc3ace94] 2024-05-27T07:30:22.990 INFO:journalctl@ceph.mds.j.smithi052.stdout:May 27 07:30:22 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-j[107746]: 13: (EUpdate::replay(MDSRank*)+0x63) [0x55b0bc3b7969] 2024-05-27T07:30:22.990 INFO:journalctl@ceph.mds.j.smithi052.stdout:May 27 07:30:22 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-j[107746]: 14: (MDLog::_replay_thread()+0x1f38) [0x55b0bc75048c] 2024-05-27T07:30:22.990 INFO:journalctl@ceph.mds.j.smithi052.stdout:May 27 07:30:22 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-j[107746]: 15: (MDLog::ReplayThread::entry()+0x11) [0x55b0bc7599cb] 2024-05-27T07:30:22.990 INFO:journalctl@ceph.mds.j.smithi052.stdout:May 27 07:30:22 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-j[107746]: 16: (Thread::entry_wrapper()+0x43) [0x7fb2b1d5272d] 2024-05-27T07:30:22.990 INFO:journalctl@ceph.mds.j.smithi052.stdout:May 27 07:30:22 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-j[107746]: 17: (Thread::_entry_func(void*)+0xd) [0x7fb2b1d52749] 2024-05-27T07:30:22.990 INFO:journalctl@ceph.mds.j.smithi052.stdout:May 27 07:30:22 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-j[107746]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7fb2b08f61ca] 2024-05-27T07:30:22.990 INFO:journalctl@ceph.mds.j.smithi052.stdout:May 27 07:30:22 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-j[107746]: 19: clone() 2024-05-27T07:30:22.990 INFO:journalctl@ceph.mds.j.smithi052.stdout:May 27 07:30:22 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-j[107746]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-05-27T07:30:22.990 INFO:journalctl@ceph.mds.j.smithi052.stdout:May 27 07:30:22 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-j[107746]: 2024-05-27T07:30:22.990 INFO:journalctl@ceph.mds.j.smithi052.stdout:May 27 07:30:22 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-j[107746]: -1> 2024-05-27T07:30:22.494+0000 7fb29f53e700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: In function 'std::set SnapClient::filter(const std::set&) const' thread 7fb29f53e700 time 2024-05-27T07:30:22.491858+0000 2024-05-27T07:30:22.990 INFO:journalctl@ceph.mds.j.smithi052.stdout:May 27 07:30:22 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-j[107746]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: 197: FAILED ceph_assert(cached_version > 0) 2024-05-27T07:30:22.990 INFO:journalctl@ceph.mds.j.smithi052.stdout:May 27 07:30:22 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-j[107746]: 2024-05-27T07:30:22.991 INFO:journalctl@ceph.mds.j.smithi052.stdout:May 27 07:30:22 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-j[107746]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:30:22.991 INFO:journalctl@ceph.mds.j.smithi052.stdout:May 27 07:30:22 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-j[107746]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x127) [0x7fb2b1d83ba6] 2024-05-27T07:30:22.991 INFO:journalctl@ceph.mds.j.smithi052.stdout:May 27 07:30:22 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-j[107746]: 2: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7fb2b1d83dd1] 2024-05-27T07:30:22.991 INFO:journalctl@ceph.mds.j.smithi052.stdout:May 27 07:30:22 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-j[107746]: 3: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x55b0bc72c23d] 2024-05-27T07:30:22.991 INFO:journalctl@ceph.mds.j.smithi052.stdout:May 27 07:30:22 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-j[107746]: 4: (SnapRealm::build_snap_set() const+0x323) [0x55b0bc717971] 2024-05-27T07:30:22.991 INFO:journalctl@ceph.mds.j.smithi052.stdout:May 27 07:30:22 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-j[107746]: 5: (SnapRealm::check_cache() const+0x13e) [0x55b0bc715fa0] 2024-05-27T07:30:22.991 INFO:journalctl@ceph.mds.j.smithi052.stdout:May 27 07:30:22 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-j[107746]: 6: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x55b0bc716651] 2024-05-27T07:30:22.991 INFO:journalctl@ceph.mds.j.smithi052.stdout:May 27 07:30:22 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-j[107746]: 7: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x55b0bc6ab347] 2024-05-27T07:30:22.991 INFO:journalctl@ceph.mds.j.smithi052.stdout:May 27 07:30:22 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-j[107746]: 8: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x55b0bc3a787e] 2024-05-27T07:30:22.991 INFO:journalctl@ceph.mds.j.smithi052.stdout:May 27 07:30:22 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-j[107746]: 9: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x2702) [0x55b0bc3ace94] 2024-05-27T07:30:22.991 INFO:journalctl@ceph.mds.j.smithi052.stdout:May 27 07:30:22 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-j[107746]: 10: (EUpdate::replay(MDSRank*)+0x63) [0x55b0bc3b7969] 2024-05-27T07:30:22.991 INFO:journalctl@ceph.mds.j.smithi052.stdout:May 27 07:30:22 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-j[107746]: 11: (MDLog::_replay_thread()+0x1f38) [0x55b0bc75048c] 2024-05-27T07:30:22.992 INFO:journalctl@ceph.mds.j.smithi052.stdout:May 27 07:30:22 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-j[107746]: 12: (MDLog::ReplayThread::entry()+0x11) [0x55b0bc7599cb] 2024-05-27T07:30:22.992 INFO:journalctl@ceph.mds.j.smithi052.stdout:May 27 07:30:22 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-j[107746]: 13: (Thread::entry_wrapper()+0x43) [0x7fb2b1d5272d] 2024-05-27T07:30:22.992 INFO:journalctl@ceph.mds.j.smithi052.stdout:May 27 07:30:22 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-j[107746]: 14: (Thread::_entry_func(void*)+0xd) [0x7fb2b1d52749] 2024-05-27T07:30:22.992 INFO:journalctl@ceph.mds.j.smithi052.stdout:May 27 07:30:22 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-j[107746]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7fb2b08f61ca] 2024-05-27T07:30:22.992 INFO:journalctl@ceph.mds.j.smithi052.stdout:May 27 07:30:22 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-j[107746]: 16: clone() 2024-05-27T07:30:22.992 INFO:journalctl@ceph.mds.j.smithi052.stdout:May 27 07:30:22 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-j[107746]: 2024-05-27T07:30:22.992 INFO:journalctl@ceph.mds.j.smithi052.stdout:May 27 07:30:22 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-j[107746]: 0> 2024-05-27T07:30:22.498+0000 7fb29f53e700 -1 *** Caught signal (Aborted) ** 2024-05-27T07:30:22.992 INFO:journalctl@ceph.mds.j.smithi052.stdout:May 27 07:30:22 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-j[107746]: in thread 7fb29f53e700 thread_name:md_log_replay 2024-05-27T07:30:22.992 INFO:journalctl@ceph.mds.j.smithi052.stdout:May 27 07:30:22 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-j[107746]: 2024-05-27T07:30:22.992 INFO:journalctl@ceph.mds.j.smithi052.stdout:May 27 07:30:22 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-j[107746]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:30:22.992 INFO:journalctl@ceph.mds.j.smithi052.stdout:May 27 07:30:22 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-j[107746]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7fb2b0900d20] 2024-05-27T07:30:22.992 INFO:journalctl@ceph.mds.j.smithi052.stdout:May 27 07:30:22 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-j[107746]: 2: gsignal() 2024-05-27T07:30:22.992 INFO:journalctl@ceph.mds.j.smithi052.stdout:May 27 07:30:22 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-j[107746]: 3: abort() 2024-05-27T07:30:22.993 INFO:journalctl@ceph.mds.j.smithi052.stdout:May 27 07:30:22 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-j[107746]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x26e) [0x7fb2b1d83ced] 2024-05-27T07:30:22.993 INFO:journalctl@ceph.mds.j.smithi052.stdout:May 27 07:30:22 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-j[107746]: 5: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7fb2b1d83dd1] 2024-05-27T07:30:22.993 INFO:journalctl@ceph.mds.j.smithi052.stdout:May 27 07:30:22 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-j[107746]: 6: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x55b0bc72c23d] 2024-05-27T07:30:22.993 INFO:journalctl@ceph.mds.j.smithi052.stdout:May 27 07:30:22 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-j[107746]: 7: (SnapRealm::build_snap_set() const+0x323) [0x55b0bc717971] 2024-05-27T07:30:22.993 INFO:journalctl@ceph.mds.j.smithi052.stdout:May 27 07:30:22 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-j[107746]: 8: (SnapRealm::check_cache() const+0x13e) [0x55b0bc715fa0] 2024-05-27T07:30:22.993 INFO:journalctl@ceph.mds.j.smithi052.stdout:May 27 07:30:22 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-j[107746]: 9: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x55b0bc716651] 2024-05-27T07:30:22.993 INFO:journalctl@ceph.mds.j.smithi052.stdout:May 27 07:30:22 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-j[107746]: 10: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x55b0bc6ab347] 2024-05-27T07:30:22.993 INFO:journalctl@ceph.mds.j.smithi052.stdout:May 27 07:30:22 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-j[107746]: 11: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x55b0bc3a787e] 2024-05-27T07:30:22.993 INFO:journalctl@ceph.mds.j.smithi052.stdout:May 27 07:30:22 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-j[107746]: 12: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x2702) [0x55b0bc3ace94] 2024-05-27T07:30:22.993 INFO:journalctl@ceph.mds.j.smithi052.stdout:May 27 07:30:22 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-j[107746]: 13: (EUpdate::replay(MDSRank*)+0x63) [0x55b0bc3b7969] 2024-05-27T07:30:22.993 INFO:journalctl@ceph.mds.j.smithi052.stdout:May 27 07:30:22 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-j[107746]: 14: (MDLog::_replay_thread()+0x1f38) [0x55b0bc75048c] 2024-05-27T07:30:22.993 INFO:journalctl@ceph.mds.j.smithi052.stdout:May 27 07:30:22 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-j[107746]: 15: (MDLog::ReplayThread::entry()+0x11) [0x55b0bc7599cb] 2024-05-27T07:30:22.993 INFO:journalctl@ceph.mds.j.smithi052.stdout:May 27 07:30:22 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-j[107746]: 16: (Thread::entry_wrapper()+0x43) [0x7fb2b1d5272d] 2024-05-27T07:30:22.994 INFO:journalctl@ceph.mds.j.smithi052.stdout:May 27 07:30:22 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-j[107746]: 17: (Thread::_entry_func(void*)+0xd) [0x7fb2b1d52749] 2024-05-27T07:30:22.994 INFO:journalctl@ceph.mds.j.smithi052.stdout:May 27 07:30:22 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-j[107746]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7fb2b08f61ca] 2024-05-27T07:30:22.994 INFO:journalctl@ceph.mds.j.smithi052.stdout:May 27 07:30:22 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-j[107746]: 19: clone() 2024-05-27T07:30:22.994 INFO:journalctl@ceph.mds.j.smithi052.stdout:May 27 07:30:22 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-j[107746]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-05-27T07:30:22.994 INFO:journalctl@ceph.mds.j.smithi052.stdout:May 27 07:30:22 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-j[107746]: 2024-05-27T07:30:22.994 INFO:journalctl@ceph.mds.j.smithi052.stdout:May 27 07:30:22 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-j[107746]: -1> 2024-05-27T07:30:22.494+0000 7fb29f53e700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: In function 'std::set SnapClient::filter(const std::set&) const' thread 7fb29f53e700 time 2024-05-27T07:30:22.491858+0000 2024-05-27T07:30:22.994 INFO:journalctl@ceph.mds.j.smithi052.stdout:May 27 07:30:22 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-j[107746]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: 197: FAILED ceph_assert(cached_version > 0) 2024-05-27T07:30:22.994 INFO:journalctl@ceph.mds.j.smithi052.stdout:May 27 07:30:22 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-j[107746]: 2024-05-27T07:30:22.994 INFO:journalctl@ceph.mds.j.smithi052.stdout:May 27 07:30:22 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-j[107746]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:30:22.994 INFO:journalctl@ceph.mds.j.smithi052.stdout:May 27 07:30:22 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-j[107746]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x127) [0x7fb2b1d83ba6] 2024-05-27T07:30:22.994 INFO:journalctl@ceph.mds.j.smithi052.stdout:May 27 07:30:22 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-j[107746]: 2: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7fb2b1d83dd1] 2024-05-27T07:30:22.994 INFO:journalctl@ceph.mds.j.smithi052.stdout:May 27 07:30:22 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-j[107746]: 3: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x55b0bc72c23d] 2024-05-27T07:30:22.995 INFO:journalctl@ceph.mds.j.smithi052.stdout:May 27 07:30:22 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-j[107746]: 4: (SnapRealm::build_snap_set() const+0x323) [0x55b0bc717971] 2024-05-27T07:30:22.995 INFO:journalctl@ceph.mds.j.smithi052.stdout:May 27 07:30:22 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-j[107746]: 5: (SnapRealm::check_cache() const+0x13e) [0x55b0bc715fa0] 2024-05-27T07:30:22.995 INFO:journalctl@ceph.mds.j.smithi052.stdout:May 27 07:30:22 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-j[107746]: 6: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x55b0bc716651] 2024-05-27T07:30:22.995 INFO:journalctl@ceph.mds.j.smithi052.stdout:May 27 07:30:22 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-j[107746]: 7: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x55b0bc6ab347] 2024-05-27T07:30:22.995 INFO:journalctl@ceph.mds.j.smithi052.stdout:May 27 07:30:22 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-j[107746]: 8: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x55b0bc3a787e] 2024-05-27T07:30:22.995 INFO:journalctl@ceph.mds.j.smithi052.stdout:May 27 07:30:22 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-j[107746]: 9: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x2702) [0x55b0bc3ace94] 2024-05-27T07:30:22.995 INFO:journalctl@ceph.mds.j.smithi052.stdout:May 27 07:30:22 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-j[107746]: 10: (EUpdate::replay(MDSRank*)+0x63) [0x55b0bc3b7969] 2024-05-27T07:30:22.995 INFO:journalctl@ceph.mds.j.smithi052.stdout:May 27 07:30:22 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-j[107746]: 11: (MDLog::_replay_thread()+0x1f38) [0x55b0bc75048c] 2024-05-27T07:30:22.995 INFO:journalctl@ceph.mds.j.smithi052.stdout:May 27 07:30:22 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-j[107746]: 12: (MDLog::ReplayThread::entry()+0x11) [0x55b0bc7599cb] 2024-05-27T07:30:22.995 INFO:journalctl@ceph.mds.j.smithi052.stdout:May 27 07:30:22 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-j[107746]: 13: (Thread::entry_wrapper()+0x43) [0x7fb2b1d5272d] 2024-05-27T07:30:22.995 INFO:journalctl@ceph.mds.j.smithi052.stdout:May 27 07:30:22 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-j[107746]: 14: (Thread::_entry_func(void*)+0xd) [0x7fb2b1d52749] 2024-05-27T07:30:22.995 INFO:journalctl@ceph.mds.j.smithi052.stdout:May 27 07:30:22 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-j[107746]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7fb2b08f61ca] 2024-05-27T07:30:22.996 INFO:journalctl@ceph.mds.j.smithi052.stdout:May 27 07:30:22 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-j[107746]: 16: clone() 2024-05-27T07:30:22.996 INFO:journalctl@ceph.mds.j.smithi052.stdout:May 27 07:30:22 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-j[107746]: 2024-05-27T07:30:22.996 INFO:journalctl@ceph.mds.j.smithi052.stdout:May 27 07:30:22 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-j[107746]: 0> 2024-05-27T07:30:22.498+0000 7fb29f53e700 -1 *** Caught signal (Aborted) ** 2024-05-27T07:30:22.996 INFO:journalctl@ceph.mds.j.smithi052.stdout:May 27 07:30:22 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-j[107746]: in thread 7fb29f53e700 thread_name:md_log_replay 2024-05-27T07:30:22.996 INFO:journalctl@ceph.mds.j.smithi052.stdout:May 27 07:30:22 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-j[107746]: 2024-05-27T07:30:22.996 INFO:journalctl@ceph.mds.j.smithi052.stdout:May 27 07:30:22 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-j[107746]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:30:22.996 INFO:journalctl@ceph.mds.j.smithi052.stdout:May 27 07:30:22 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-j[107746]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7fb2b0900d20] 2024-05-27T07:30:22.996 INFO:journalctl@ceph.mds.j.smithi052.stdout:May 27 07:30:22 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-j[107746]: 2: gsignal() 2024-05-27T07:30:22.996 INFO:journalctl@ceph.mds.j.smithi052.stdout:May 27 07:30:22 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-j[107746]: 3: abort() 2024-05-27T07:30:22.996 INFO:journalctl@ceph.mds.j.smithi052.stdout:May 27 07:30:22 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-j[107746]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x26e) [0x7fb2b1d83ced] 2024-05-27T07:30:22.996 INFO:journalctl@ceph.mds.j.smithi052.stdout:May 27 07:30:22 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-j[107746]: 5: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7fb2b1d83dd1] 2024-05-27T07:30:22.996 INFO:journalctl@ceph.mds.j.smithi052.stdout:May 27 07:30:22 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-j[107746]: 6: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x55b0bc72c23d] 2024-05-27T07:30:22.996 INFO:journalctl@ceph.mds.j.smithi052.stdout:May 27 07:30:22 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-j[107746]: 7: (SnapRealm::build_snap_set() const+0x323) [0x55b0bc717971] 2024-05-27T07:30:22.996 INFO:journalctl@ceph.mds.j.smithi052.stdout:May 27 07:30:22 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-j[107746]: 8: (SnapRealm::check_cache() const+0x13e) [0x55b0bc715fa0] 2024-05-27T07:30:22.997 INFO:journalctl@ceph.mds.j.smithi052.stdout:May 27 07:30:22 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-j[107746]: 9: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x55b0bc716651] 2024-05-27T07:30:22.997 INFO:journalctl@ceph.mds.j.smithi052.stdout:May 27 07:30:22 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-j[107746]: 10: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x55b0bc6ab347] 2024-05-27T07:30:22.997 INFO:journalctl@ceph.mds.j.smithi052.stdout:May 27 07:30:22 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-j[107746]: 11: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x55b0bc3a787e] 2024-05-27T07:30:22.997 INFO:journalctl@ceph.mds.j.smithi052.stdout:May 27 07:30:22 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-j[107746]: 12: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x2702) [0x55b0bc3ace94] 2024-05-27T07:30:22.997 INFO:journalctl@ceph.mds.j.smithi052.stdout:May 27 07:30:22 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-j[107746]: 13: (EUpdate::replay(MDSRank*)+0x63) [0x55b0bc3b7969] 2024-05-27T07:30:22.997 INFO:journalctl@ceph.mds.j.smithi052.stdout:May 27 07:30:22 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-j[107746]: 14: (MDLog::_replay_thread()+0x1f38) [0x55b0bc75048c] 2024-05-27T07:30:22.997 INFO:journalctl@ceph.mds.j.smithi052.stdout:May 27 07:30:22 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-j[107746]: 15: (MDLog::ReplayThread::entry()+0x11) [0x55b0bc7599cb] 2024-05-27T07:30:22.997 INFO:journalctl@ceph.mds.j.smithi052.stdout:May 27 07:30:22 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-j[107746]: 16: (Thread::entry_wrapper()+0x43) [0x7fb2b1d5272d] 2024-05-27T07:30:22.997 INFO:journalctl@ceph.mds.j.smithi052.stdout:May 27 07:30:22 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-j[107746]: 17: (Thread::_entry_func(void*)+0xd) [0x7fb2b1d52749] 2024-05-27T07:30:22.997 INFO:journalctl@ceph.mds.j.smithi052.stdout:May 27 07:30:22 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-j[107746]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7fb2b08f61ca] 2024-05-27T07:30:22.997 INFO:journalctl@ceph.mds.j.smithi052.stdout:May 27 07:30:22 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-j[107746]: 19: clone() 2024-05-27T07:30:22.997 INFO:journalctl@ceph.mds.j.smithi052.stdout:May 27 07:30:22 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-j[107746]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-05-27T07:30:22.998 INFO:journalctl@ceph.mds.j.smithi052.stdout:May 27 07:30:22 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-j[107746]: 2024-05-27T07:30:22.998 INFO:journalctl@ceph.mds.j.smithi052.stdout:May 27 07:30:22 smithi052 podman[107781]: 2024-05-27 07:30:22.686772192 +0000 UTC m=+0.101183057 container died 332ab7d985b17bcd2765ea36a1f8438aec4b0d8a6a4108995ab221d58bbcb935 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-j, org.label-schema.build-date=20240521, GIT_CLEAN=True, ceph=True, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS) 2024-05-27T07:30:23.353 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:30:22 smithi173 ceph-mon[37000]: mds.? [v2:172.21.15.52:6834/575031775,v1:172.21.15.52:6835/575031775] up:boot 2024-05-27T07:30:23.353 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:30:22 smithi173 ceph-mon[37000]: fsmap cephfs:5 3 up:standby-replay 1 up:standby 5 up:active 2024-05-27T07:30:23.353 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:30:22 smithi173 ceph-mon[37000]: from='mgr.15333 172.21.15.52:0/416389082' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "j"}]: dispatch 2024-05-27T07:30:23.353 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:30:22 smithi173 ceph-mon[37000]: fsmap cephfs:5 4 up:standby-replay 5 up:active 2024-05-27T07:30:23.441 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:30:22 smithi105 ceph-mon[35977]: mds.? [v2:172.21.15.52:6834/575031775,v1:172.21.15.52:6835/575031775] up:boot 2024-05-27T07:30:23.441 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:30:22 smithi105 ceph-mon[35977]: fsmap cephfs:5 3 up:standby-replay 1 up:standby 5 up:active 2024-05-27T07:30:23.441 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:30:22 smithi105 ceph-mon[35977]: from='mgr.15333 172.21.15.52:0/416389082' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "j"}]: dispatch 2024-05-27T07:30:23.441 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:30:22 smithi105 ceph-mon[35977]: fsmap cephfs:5 4 up:standby-replay 5 up:active 2024-05-27T07:30:23.480 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:30:22 smithi052 ceph-mon[31955]: mds.? [v2:172.21.15.52:6834/575031775,v1:172.21.15.52:6835/575031775] up:boot 2024-05-27T07:30:23.480 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:30:22 smithi052 ceph-mon[31955]: fsmap cephfs:5 3 up:standby-replay 1 up:standby 5 up:active 2024-05-27T07:30:23.481 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:30:22 smithi052 ceph-mon[31955]: from='mgr.15333 172.21.15.52:0/416389082' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "j"}]: dispatch 2024-05-27T07:30:23.481 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:30:22 smithi052 ceph-mon[31955]: fsmap cephfs:5 4 up:standby-replay 5 up:active 2024-05-27T07:30:23.994 INFO:journalctl@ceph.mds.j.smithi052.stdout:May 27 07:30:23 smithi052 podman[107781]: 2024-05-27 07:30:23.746994893 +0000 UTC m=+1.161405751 container remove 332ab7d985b17bcd2765ea36a1f8438aec4b0d8a6a4108995ab221d58bbcb935 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-j, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, org.label-schema.build-date=20240521, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90) 2024-05-27T07:30:23.995 INFO:journalctl@ceph.mds.j.smithi052.stdout:May 27 07:30:23 smithi052 systemd[1]: ceph-73df2636-1bf8-11ef-bc9b-c7b262605968@mds.j.service: Main process exited, code=exited, status=134/n/a 2024-05-27T07:30:23.995 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:30:24 smithi052 ceph-mon[31955]: pgmap v158: 129 pgs: 129 active+clean; 364 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 MiB/s rd, 5.1 MiB/s wr, 1.32k op/s 2024-05-27T07:30:24.353 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:30:24 smithi173 ceph-mon[37000]: pgmap v158: 129 pgs: 129 active+clean; 364 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 MiB/s rd, 5.1 MiB/s wr, 1.32k op/s 2024-05-27T07:30:24.441 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:30:24 smithi105 ceph-mon[35977]: pgmap v158: 129 pgs: 129 active+clean; 364 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 MiB/s rd, 5.1 MiB/s wr, 1.32k op/s 2024-05-27T07:30:24.731 INFO:journalctl@ceph.mds.j.smithi052.stdout:May 27 07:30:24 smithi052 systemd[1]: ceph-73df2636-1bf8-11ef-bc9b-c7b262605968@mds.j.service: Failed with result 'exit-code'. 2024-05-27T07:30:24.731 INFO:journalctl@ceph.mds.j.smithi052.stdout:May 27 07:30:24 smithi052 systemd[1]: ceph-73df2636-1bf8-11ef-bc9b-c7b262605968@mds.j.service: Consumed 1.333s CPU time. 2024-05-27T07:30:25.209 INFO:tasks.workunit.client.0.smithi052.stdout: 98 1186579 576 839296 847 672291 501 2024-05-27T07:30:25.251 INFO:journalctl@ceph.mds.d.smithi052.stdout:May 27 07:30:25 smithi052 systemd[1]: ceph-73df2636-1bf8-11ef-bc9b-c7b262605968@mds.d.service: Scheduled restart job, restart counter is at 6. 2024-05-27T07:30:25.251 INFO:journalctl@ceph.mds.d.smithi052.stdout:May 27 07:30:25 smithi052 systemd[1]: Stopped Ceph mds.d for 73df2636-1bf8-11ef-bc9b-c7b262605968. 2024-05-27T07:30:25.251 INFO:journalctl@ceph.mds.d.smithi052.stdout:May 27 07:30:25 smithi052 systemd[1]: ceph-73df2636-1bf8-11ef-bc9b-c7b262605968@mds.d.service: Consumed 1.070s CPU time. 2024-05-27T07:30:25.251 INFO:journalctl@ceph.mds.d.smithi052.stdout:May 27 07:30:25 smithi052 systemd[1]: ceph-73df2636-1bf8-11ef-bc9b-c7b262605968@mds.d.service: Start request repeated too quickly. 2024-05-27T07:30:25.251 INFO:journalctl@ceph.mds.d.smithi052.stdout:May 27 07:30:25 smithi052 systemd[1]: ceph-73df2636-1bf8-11ef-bc9b-c7b262605968@mds.d.service: Failed with result 'exit-code'. 2024-05-27T07:30:25.251 INFO:journalctl@ceph.mds.d.smithi052.stdout:May 27 07:30:25 smithi052 systemd[1]: Failed to start Ceph mds.d for 73df2636-1bf8-11ef-bc9b-c7b262605968. 2024-05-27T07:30:25.252 INFO:journalctl@ceph.mds.a.smithi052.stdout:May 27 07:30:25 smithi052 systemd[1]: ceph-73df2636-1bf8-11ef-bc9b-c7b262605968@mds.a.service: Scheduled restart job, restart counter is at 4. 2024-05-27T07:30:25.252 INFO:journalctl@ceph.mds.a.smithi052.stdout:May 27 07:30:25 smithi052 systemd[1]: Stopped Ceph mds.a for 73df2636-1bf8-11ef-bc9b-c7b262605968. 2024-05-27T07:30:25.252 INFO:journalctl@ceph.mds.a.smithi052.stdout:May 27 07:30:25 smithi052 systemd[1]: ceph-73df2636-1bf8-11ef-bc9b-c7b262605968@mds.a.service: Consumed 1.433s CPU time. 2024-05-27T07:30:25.252 INFO:journalctl@ceph.mds.a.smithi052.stdout:May 27 07:30:25 smithi052 systemd[1]: Starting Ceph mds.a for 73df2636-1bf8-11ef-bc9b-c7b262605968... 2024-05-27T07:30:25.252 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:30:25 smithi052 ceph-mon[31955]: MDS daemon mds.d is removed because it is dead or otherwise unavailable. 2024-05-27T07:30:25.252 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:30:25 smithi052 ceph-mon[31955]: MDS daemon mds.a is removed because it is dead or otherwise unavailable. 2024-05-27T07:30:25.252 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:30:25 smithi052 ceph-mon[31955]: fsmap cephfs:5 2 up:standby-replay 5 up:active 2024-05-27T07:30:25.353 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:30:25 smithi173 ceph-mon[37000]: MDS daemon mds.d is removed because it is dead or otherwise unavailable. 2024-05-27T07:30:25.353 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:30:25 smithi173 ceph-mon[37000]: MDS daemon mds.a is removed because it is dead or otherwise unavailable. 2024-05-27T07:30:25.353 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:30:25 smithi173 ceph-mon[37000]: fsmap cephfs:5 2 up:standby-replay 5 up:active 2024-05-27T07:30:25.441 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:30:25 smithi105 ceph-mon[35977]: MDS daemon mds.d is removed because it is dead or otherwise unavailable. 2024-05-27T07:30:25.441 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:30:25 smithi105 ceph-mon[35977]: MDS daemon mds.a is removed because it is dead or otherwise unavailable. 2024-05-27T07:30:25.441 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:30:25 smithi105 ceph-mon[35977]: fsmap cephfs:5 2 up:standby-replay 5 up:active 2024-05-27T07:30:25.441 INFO:journalctl@ceph.mds.k.smithi105.stdout:May 27 07:30:25 smithi105 podman[70425]: 2024-05-27 07:30:25.223709705 +0000 UTC m=+4.184036616 container remove 7ecb8856db56e0f00d5b025b01a7c69ab17ac2a1672e92156e1ba5122cefd159 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-k, org.label-schema.build-date=20240521, GIT_BRANCH=HEAD, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, org.label-schema.license=GPLv2, ceph=True, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90) 2024-05-27T07:30:25.442 INFO:journalctl@ceph.mds.k.smithi105.stdout:May 27 07:30:25 smithi105 systemd[1]: ceph-73df2636-1bf8-11ef-bc9b-c7b262605968@mds.k.service: Main process exited, code=exited, status=134/n/a 2024-05-27T07:30:25.941 INFO:journalctl@ceph.mds.k.smithi105.stdout:May 27 07:30:25 smithi105 systemd[1]: ceph-73df2636-1bf8-11ef-bc9b-c7b262605968@mds.k.service: Failed with result 'exit-code'. 2024-05-27T07:30:26.103 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:30:26 smithi052 ceph-mon[31955]: pgmap v159: 129 pgs: 129 active+clean; 348 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 MiB/s rd, 5.3 MiB/s wr, 837 op/s 2024-05-27T07:30:26.103 INFO:journalctl@ceph.mds.a.smithi052.stdout:May 27 07:30:25 smithi052 podman[107855]: 2024-05-27 07:30:25.715708669 +0000 UTC m=+0.088519771 image pull e36ea4fc8a5807e8d2701370e350a0de3856538797f7fba64cfce7e344bd6799 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6 2024-05-27T07:30:26.104 INFO:journalctl@ceph.mds.a.smithi052.stdout:May 27 07:30:25 smithi052 podman[107855]: 2024-05-27 07:30:25.833726383 +0000 UTC m=+0.206537461 container create 893ffb3422f9f447e60922f01570ff78e7dafcf0c65edf706bc1782ef760a66d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-a, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.schema-version=1.0, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, GIT_BRANCH=HEAD, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, org.label-schema.build-date=20240521, org.label-schema.vendor=CentOS) 2024-05-27T07:30:26.104 INFO:journalctl@ceph.mds.a.smithi052.stdout:May 27 07:30:26 smithi052 podman[107855]: 2024-05-27 07:30:26.112162766 +0000 UTC m=+0.484973856 container init 893ffb3422f9f447e60922f01570ff78e7dafcf0c65edf706bc1782ef760a66d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-a, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.build-date=20240521, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2) 2024-05-27T07:30:26.353 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:30:26 smithi173 ceph-mon[37000]: pgmap v159: 129 pgs: 129 active+clean; 348 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 MiB/s rd, 5.3 MiB/s wr, 837 op/s 2024-05-27T07:30:26.441 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:30:26 smithi105 ceph-mon[35977]: pgmap v159: 129 pgs: 129 active+clean; 348 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 MiB/s rd, 5.3 MiB/s wr, 837 op/s 2024-05-27T07:30:26.481 INFO:journalctl@ceph.mds.a.smithi052.stdout:May 27 07:30:26 smithi052 podman[107855]: 2024-05-27 07:30:26.11869325 +0000 UTC m=+0.491504339 container start 893ffb3422f9f447e60922f01570ff78e7dafcf0c65edf706bc1782ef760a66d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-a, org.label-schema.build-date=20240521, GIT_BRANCH=HEAD, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, maintainer=Guillaume Abrioux , GIT_CLEAN=True, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-05-27T07:30:26.481 INFO:journalctl@ceph.mds.a.smithi052.stdout:May 27 07:30:26 smithi052 bash[107855]: 893ffb3422f9f447e60922f01570ff78e7dafcf0c65edf706bc1782ef760a66d 2024-05-27T07:30:26.481 INFO:journalctl@ceph.mds.a.smithi052.stdout:May 27 07:30:26 smithi052 systemd[1]: Started Ceph mds.a for 73df2636-1bf8-11ef-bc9b-c7b262605968. 2024-05-27T07:30:26.481 INFO:journalctl@ceph.mds.a.smithi052.stdout:May 27 07:30:26 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-a[107870]: starting mds.a at 2024-05-27T07:30:27.987 INFO:journalctl@ceph.mds.a.smithi052.stdout:May 27 07:30:27 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-a[107870]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: In function 'std::set SnapClient::filter(const std::set&) const' thread 7fde2c30f700 time 2024-05-27T07:30:27.510364+0000 2024-05-27T07:30:27.987 INFO:journalctl@ceph.mds.a.smithi052.stdout:May 27 07:30:27 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-a[107870]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: 197: FAILED ceph_assert(cached_version > 0) 2024-05-27T07:30:27.988 INFO:journalctl@ceph.mds.a.smithi052.stdout:May 27 07:30:27 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-a[107870]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:30:27.988 INFO:journalctl@ceph.mds.a.smithi052.stdout:May 27 07:30:27 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-a[107870]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x127) [0x7fde3eb54ba6] 2024-05-27T07:30:27.988 INFO:journalctl@ceph.mds.a.smithi052.stdout:May 27 07:30:27 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-a[107870]: 2: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7fde3eb54dd1] 2024-05-27T07:30:27.988 INFO:journalctl@ceph.mds.a.smithi052.stdout:May 27 07:30:27 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-a[107870]: 3: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x557d2b26923d] 2024-05-27T07:30:27.988 INFO:journalctl@ceph.mds.a.smithi052.stdout:May 27 07:30:27 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-a[107870]: 4: (SnapRealm::build_snap_set() const+0x323) [0x557d2b254971] 2024-05-27T07:30:27.988 INFO:journalctl@ceph.mds.a.smithi052.stdout:May 27 07:30:27 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-a[107870]: 5: (SnapRealm::check_cache() const+0x13e) [0x557d2b252fa0] 2024-05-27T07:30:27.988 INFO:journalctl@ceph.mds.a.smithi052.stdout:May 27 07:30:27 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-a[107870]: 6: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x557d2b253651] 2024-05-27T07:30:27.989 INFO:journalctl@ceph.mds.a.smithi052.stdout:May 27 07:30:27 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-a[107870]: 7: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x557d2b1e8347] 2024-05-27T07:30:27.989 INFO:journalctl@ceph.mds.a.smithi052.stdout:May 27 07:30:27 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-a[107870]: 8: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x557d2aee487e] 2024-05-27T07:30:27.989 INFO:journalctl@ceph.mds.a.smithi052.stdout:May 27 07:30:27 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-a[107870]: 9: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x2702) [0x557d2aee9e94] 2024-05-27T07:30:27.989 INFO:journalctl@ceph.mds.a.smithi052.stdout:May 27 07:30:27 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-a[107870]: 10: (EUpdate::replay(MDSRank*)+0x63) [0x557d2aef4969] 2024-05-27T07:30:27.989 INFO:journalctl@ceph.mds.a.smithi052.stdout:May 27 07:30:27 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-a[107870]: 11: (MDLog::_replay_thread()+0x1f38) [0x557d2b28d48c] 2024-05-27T07:30:27.989 INFO:journalctl@ceph.mds.a.smithi052.stdout:May 27 07:30:27 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-a[107870]: 12: (MDLog::ReplayThread::entry()+0x11) [0x557d2b2969cb] 2024-05-27T07:30:27.989 INFO:journalctl@ceph.mds.a.smithi052.stdout:May 27 07:30:27 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-a[107870]: 13: (Thread::entry_wrapper()+0x43) [0x7fde3eb2372d] 2024-05-27T07:30:27.989 INFO:journalctl@ceph.mds.a.smithi052.stdout:May 27 07:30:27 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-a[107870]: 14: (Thread::_entry_func(void*)+0xd) [0x7fde3eb23749] 2024-05-27T07:30:27.990 INFO:journalctl@ceph.mds.a.smithi052.stdout:May 27 07:30:27 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-a[107870]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7fde3d6c71ca] 2024-05-27T07:30:27.990 INFO:journalctl@ceph.mds.a.smithi052.stdout:May 27 07:30:27 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-a[107870]: 16: clone() 2024-05-27T07:30:27.990 INFO:journalctl@ceph.mds.a.smithi052.stdout:May 27 07:30:27 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-a[107870]: 2024-05-27T07:30:27.512+0000 7fde2c30f700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: In function 'std::set SnapClient::filter(const std::set&) const' thread 7fde2c30f700 time 2024-05-27T07:30:27.510364+0000 2024-05-27T07:30:27.990 INFO:journalctl@ceph.mds.a.smithi052.stdout:May 27 07:30:27 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-a[107870]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: 197: FAILED ceph_assert(cached_version > 0) 2024-05-27T07:30:27.990 INFO:journalctl@ceph.mds.a.smithi052.stdout:May 27 07:30:27 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-a[107870]: 2024-05-27T07:30:27.990 INFO:journalctl@ceph.mds.a.smithi052.stdout:May 27 07:30:27 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-a[107870]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:30:27.990 INFO:journalctl@ceph.mds.a.smithi052.stdout:May 27 07:30:27 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-a[107870]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x127) [0x7fde3eb54ba6] 2024-05-27T07:30:27.990 INFO:journalctl@ceph.mds.a.smithi052.stdout:May 27 07:30:27 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-a[107870]: 2: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7fde3eb54dd1] 2024-05-27T07:30:27.991 INFO:journalctl@ceph.mds.a.smithi052.stdout:May 27 07:30:27 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-a[107870]: 3: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x557d2b26923d] 2024-05-27T07:30:27.991 INFO:journalctl@ceph.mds.a.smithi052.stdout:May 27 07:30:27 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-a[107870]: 4: (SnapRealm::build_snap_set() const+0x323) [0x557d2b254971] 2024-05-27T07:30:27.991 INFO:journalctl@ceph.mds.a.smithi052.stdout:May 27 07:30:27 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-a[107870]: 5: (SnapRealm::check_cache() const+0x13e) [0x557d2b252fa0] 2024-05-27T07:30:27.991 INFO:journalctl@ceph.mds.a.smithi052.stdout:May 27 07:30:27 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-a[107870]: 6: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x557d2b253651] 2024-05-27T07:30:27.991 INFO:journalctl@ceph.mds.a.smithi052.stdout:May 27 07:30:27 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-a[107870]: 7: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x557d2b1e8347] 2024-05-27T07:30:27.991 INFO:journalctl@ceph.mds.a.smithi052.stdout:May 27 07:30:27 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-a[107870]: 8: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x557d2aee487e] 2024-05-27T07:30:27.991 INFO:journalctl@ceph.mds.a.smithi052.stdout:May 27 07:30:27 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-a[107870]: 9: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x2702) [0x557d2aee9e94] 2024-05-27T07:30:27.992 INFO:journalctl@ceph.mds.a.smithi052.stdout:May 27 07:30:27 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-a[107870]: 10: (EUpdate::replay(MDSRank*)+0x63) [0x557d2aef4969] 2024-05-27T07:30:27.992 INFO:journalctl@ceph.mds.a.smithi052.stdout:May 27 07:30:27 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-a[107870]: 11: (MDLog::_replay_thread()+0x1f38) [0x557d2b28d48c] 2024-05-27T07:30:27.992 INFO:journalctl@ceph.mds.a.smithi052.stdout:May 27 07:30:27 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-a[107870]: 12: (MDLog::ReplayThread::entry()+0x11) [0x557d2b2969cb] 2024-05-27T07:30:27.992 INFO:journalctl@ceph.mds.a.smithi052.stdout:May 27 07:30:27 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-a[107870]: 13: (Thread::entry_wrapper()+0x43) [0x7fde3eb2372d] 2024-05-27T07:30:27.992 INFO:journalctl@ceph.mds.a.smithi052.stdout:May 27 07:30:27 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-a[107870]: 14: (Thread::_entry_func(void*)+0xd) [0x7fde3eb23749] 2024-05-27T07:30:27.992 INFO:journalctl@ceph.mds.a.smithi052.stdout:May 27 07:30:27 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-a[107870]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7fde3d6c71ca] 2024-05-27T07:30:27.992 INFO:journalctl@ceph.mds.a.smithi052.stdout:May 27 07:30:27 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-a[107870]: 16: clone() 2024-05-27T07:30:27.992 INFO:journalctl@ceph.mds.a.smithi052.stdout:May 27 07:30:27 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-a[107870]: 2024-05-27T07:30:27.992 INFO:journalctl@ceph.mds.a.smithi052.stdout:May 27 07:30:27 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-a[107870]: *** Caught signal (Aborted) ** 2024-05-27T07:30:27.993 INFO:journalctl@ceph.mds.a.smithi052.stdout:May 27 07:30:27 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-a[107870]: in thread 7fde2c30f700 thread_name:md_log_replay 2024-05-27T07:30:27.993 INFO:journalctl@ceph.mds.a.smithi052.stdout:May 27 07:30:27 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-a[107870]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:30:27.993 INFO:journalctl@ceph.mds.a.smithi052.stdout:May 27 07:30:27 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-a[107870]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7fde3d6d1d20] 2024-05-27T07:30:27.993 INFO:journalctl@ceph.mds.a.smithi052.stdout:May 27 07:30:27 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-a[107870]: 2: gsignal() 2024-05-27T07:30:27.993 INFO:journalctl@ceph.mds.a.smithi052.stdout:May 27 07:30:27 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-a[107870]: 3: abort() 2024-05-27T07:30:27.993 INFO:journalctl@ceph.mds.a.smithi052.stdout:May 27 07:30:27 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-a[107870]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x26e) [0x7fde3eb54ced] 2024-05-27T07:30:27.994 INFO:journalctl@ceph.mds.a.smithi052.stdout:May 27 07:30:27 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-a[107870]: 5: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7fde3eb54dd1] 2024-05-27T07:30:27.995 INFO:journalctl@ceph.mds.a.smithi052.stdout:May 27 07:30:27 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-a[107870]: 6: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x557d2b26923d] 2024-05-27T07:30:27.995 INFO:journalctl@ceph.mds.a.smithi052.stdout:May 27 07:30:27 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-a[107870]: 7: (SnapRealm::build_snap_set() const+0x323) [0x557d2b254971] 2024-05-27T07:30:27.995 INFO:journalctl@ceph.mds.a.smithi052.stdout:May 27 07:30:27 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-a[107870]: 8: (SnapRealm::check_cache() const+0x13e) [0x557d2b252fa0] 2024-05-27T07:30:27.995 INFO:journalctl@ceph.mds.a.smithi052.stdout:May 27 07:30:27 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-a[107870]: 9: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x557d2b253651] 2024-05-27T07:30:27.995 INFO:journalctl@ceph.mds.a.smithi052.stdout:May 27 07:30:27 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-a[107870]: 10: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x557d2b1e8347] 2024-05-27T07:30:27.995 INFO:journalctl@ceph.mds.a.smithi052.stdout:May 27 07:30:27 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-a[107870]: 11: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x557d2aee487e] 2024-05-27T07:30:27.995 INFO:journalctl@ceph.mds.a.smithi052.stdout:May 27 07:30:27 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-a[107870]: 12: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x2702) [0x557d2aee9e94] 2024-05-27T07:30:27.995 INFO:journalctl@ceph.mds.a.smithi052.stdout:May 27 07:30:27 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-a[107870]: 13: (EUpdate::replay(MDSRank*)+0x63) [0x557d2aef4969] 2024-05-27T07:30:27.996 INFO:journalctl@ceph.mds.a.smithi052.stdout:May 27 07:30:27 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-a[107870]: 14: (MDLog::_replay_thread()+0x1f38) [0x557d2b28d48c] 2024-05-27T07:30:27.996 INFO:journalctl@ceph.mds.a.smithi052.stdout:May 27 07:30:27 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-a[107870]: 15: (MDLog::ReplayThread::entry()+0x11) [0x557d2b2969cb] 2024-05-27T07:30:27.996 INFO:journalctl@ceph.mds.a.smithi052.stdout:May 27 07:30:27 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-a[107870]: 16: (Thread::entry_wrapper()+0x43) [0x7fde3eb2372d] 2024-05-27T07:30:27.996 INFO:journalctl@ceph.mds.a.smithi052.stdout:May 27 07:30:27 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-a[107870]: 17: (Thread::_entry_func(void*)+0xd) [0x7fde3eb23749] 2024-05-27T07:30:27.996 INFO:journalctl@ceph.mds.a.smithi052.stdout:May 27 07:30:27 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-a[107870]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7fde3d6c71ca] 2024-05-27T07:30:27.996 INFO:journalctl@ceph.mds.a.smithi052.stdout:May 27 07:30:27 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-a[107870]: 19: clone() 2024-05-27T07:30:27.996 INFO:journalctl@ceph.mds.a.smithi052.stdout:May 27 07:30:27 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-a[107870]: 2024-05-27T07:30:27.516+0000 7fde2c30f700 -1 *** Caught signal (Aborted) ** 2024-05-27T07:30:27.996 INFO:journalctl@ceph.mds.a.smithi052.stdout:May 27 07:30:27 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-a[107870]: in thread 7fde2c30f700 thread_name:md_log_replay 2024-05-27T07:30:27.996 INFO:journalctl@ceph.mds.a.smithi052.stdout:May 27 07:30:27 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-a[107870]: 2024-05-27T07:30:27.996 INFO:journalctl@ceph.mds.a.smithi052.stdout:May 27 07:30:27 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-a[107870]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:30:27.996 INFO:journalctl@ceph.mds.a.smithi052.stdout:May 27 07:30:27 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-a[107870]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7fde3d6d1d20] 2024-05-27T07:30:27.996 INFO:journalctl@ceph.mds.a.smithi052.stdout:May 27 07:30:27 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-a[107870]: 2: gsignal() 2024-05-27T07:30:27.997 INFO:journalctl@ceph.mds.a.smithi052.stdout:May 27 07:30:27 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-a[107870]: 3: abort() 2024-05-27T07:30:27.997 INFO:journalctl@ceph.mds.a.smithi052.stdout:May 27 07:30:27 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-a[107870]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x26e) [0x7fde3eb54ced] 2024-05-27T07:30:27.997 INFO:journalctl@ceph.mds.a.smithi052.stdout:May 27 07:30:27 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-a[107870]: 5: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7fde3eb54dd1] 2024-05-27T07:30:27.997 INFO:journalctl@ceph.mds.a.smithi052.stdout:May 27 07:30:27 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-a[107870]: 6: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x557d2b26923d] 2024-05-27T07:30:27.997 INFO:journalctl@ceph.mds.a.smithi052.stdout:May 27 07:30:27 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-a[107870]: 7: (SnapRealm::build_snap_set() const+0x323) [0x557d2b254971] 2024-05-27T07:30:27.997 INFO:journalctl@ceph.mds.a.smithi052.stdout:May 27 07:30:27 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-a[107870]: 8: (SnapRealm::check_cache() const+0x13e) [0x557d2b252fa0] 2024-05-27T07:30:27.997 INFO:journalctl@ceph.mds.a.smithi052.stdout:May 27 07:30:27 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-a[107870]: 9: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x557d2b253651] 2024-05-27T07:30:27.997 INFO:journalctl@ceph.mds.a.smithi052.stdout:May 27 07:30:27 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-a[107870]: 10: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x557d2b1e8347] 2024-05-27T07:30:27.997 INFO:journalctl@ceph.mds.a.smithi052.stdout:May 27 07:30:27 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-a[107870]: 11: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x557d2aee487e] 2024-05-27T07:30:27.997 INFO:journalctl@ceph.mds.a.smithi052.stdout:May 27 07:30:27 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-a[107870]: 12: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x2702) [0x557d2aee9e94] 2024-05-27T07:30:27.997 INFO:journalctl@ceph.mds.a.smithi052.stdout:May 27 07:30:27 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-a[107870]: 13: (EUpdate::replay(MDSRank*)+0x63) [0x557d2aef4969] 2024-05-27T07:30:27.997 INFO:journalctl@ceph.mds.a.smithi052.stdout:May 27 07:30:27 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-a[107870]: 14: (MDLog::_replay_thread()+0x1f38) [0x557d2b28d48c] 2024-05-27T07:30:27.997 INFO:journalctl@ceph.mds.a.smithi052.stdout:May 27 07:30:27 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-a[107870]: 15: (MDLog::ReplayThread::entry()+0x11) [0x557d2b2969cb] 2024-05-27T07:30:27.998 INFO:journalctl@ceph.mds.a.smithi052.stdout:May 27 07:30:27 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-a[107870]: 16: (Thread::entry_wrapper()+0x43) [0x7fde3eb2372d] 2024-05-27T07:30:27.998 INFO:journalctl@ceph.mds.a.smithi052.stdout:May 27 07:30:27 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-a[107870]: 17: (Thread::_entry_func(void*)+0xd) [0x7fde3eb23749] 2024-05-27T07:30:27.998 INFO:journalctl@ceph.mds.a.smithi052.stdout:May 27 07:30:27 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-a[107870]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7fde3d6c71ca] 2024-05-27T07:30:27.998 INFO:journalctl@ceph.mds.a.smithi052.stdout:May 27 07:30:27 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-a[107870]: 19: clone() 2024-05-27T07:30:27.998 INFO:journalctl@ceph.mds.a.smithi052.stdout:May 27 07:30:27 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-a[107870]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-05-27T07:30:27.998 INFO:journalctl@ceph.mds.a.smithi052.stdout:May 27 07:30:27 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-a[107870]: 2024-05-27T07:30:27.998 INFO:journalctl@ceph.mds.a.smithi052.stdout:May 27 07:30:27 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-a[107870]: -1> 2024-05-27T07:30:27.512+0000 7fde2c30f700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: In function 'std::set SnapClient::filter(const std::set&) const' thread 7fde2c30f700 time 2024-05-27T07:30:27.510364+0000 2024-05-27T07:30:27.998 INFO:journalctl@ceph.mds.a.smithi052.stdout:May 27 07:30:27 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-a[107870]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: 197: FAILED ceph_assert(cached_version > 0) 2024-05-27T07:30:27.998 INFO:journalctl@ceph.mds.a.smithi052.stdout:May 27 07:30:27 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-a[107870]: 2024-05-27T07:30:27.998 INFO:journalctl@ceph.mds.a.smithi052.stdout:May 27 07:30:27 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-a[107870]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:30:27.998 INFO:journalctl@ceph.mds.a.smithi052.stdout:May 27 07:30:27 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-a[107870]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x127) [0x7fde3eb54ba6] 2024-05-27T07:30:27.998 INFO:journalctl@ceph.mds.a.smithi052.stdout:May 27 07:30:27 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-a[107870]: 2: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7fde3eb54dd1] 2024-05-27T07:30:27.999 INFO:journalctl@ceph.mds.a.smithi052.stdout:May 27 07:30:27 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-a[107870]: 3: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x557d2b26923d] 2024-05-27T07:30:27.999 INFO:journalctl@ceph.mds.a.smithi052.stdout:May 27 07:30:27 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-a[107870]: 4: (SnapRealm::build_snap_set() const+0x323) [0x557d2b254971] 2024-05-27T07:30:27.999 INFO:journalctl@ceph.mds.a.smithi052.stdout:May 27 07:30:27 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-a[107870]: 5: (SnapRealm::check_cache() const+0x13e) [0x557d2b252fa0] 2024-05-27T07:30:27.999 INFO:journalctl@ceph.mds.a.smithi052.stdout:May 27 07:30:27 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-a[107870]: 6: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x557d2b253651] 2024-05-27T07:30:27.999 INFO:journalctl@ceph.mds.a.smithi052.stdout:May 27 07:30:27 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-a[107870]: 7: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x557d2b1e8347] 2024-05-27T07:30:27.999 INFO:journalctl@ceph.mds.a.smithi052.stdout:May 27 07:30:27 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-a[107870]: 8: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x557d2aee487e] 2024-05-27T07:30:27.999 INFO:journalctl@ceph.mds.a.smithi052.stdout:May 27 07:30:27 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-a[107870]: 9: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x2702) [0x557d2aee9e94] 2024-05-27T07:30:27.999 INFO:journalctl@ceph.mds.a.smithi052.stdout:May 27 07:30:27 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-a[107870]: 10: (EUpdate::replay(MDSRank*)+0x63) [0x557d2aef4969] 2024-05-27T07:30:27.999 INFO:journalctl@ceph.mds.a.smithi052.stdout:May 27 07:30:27 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-a[107870]: 11: (MDLog::_replay_thread()+0x1f38) [0x557d2b28d48c] 2024-05-27T07:30:27.999 INFO:journalctl@ceph.mds.a.smithi052.stdout:May 27 07:30:27 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-a[107870]: 12: (MDLog::ReplayThread::entry()+0x11) [0x557d2b2969cb] 2024-05-27T07:30:27.999 INFO:journalctl@ceph.mds.a.smithi052.stdout:May 27 07:30:27 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-a[107870]: 13: (Thread::entry_wrapper()+0x43) [0x7fde3eb2372d] 2024-05-27T07:30:27.999 INFO:journalctl@ceph.mds.a.smithi052.stdout:May 27 07:30:27 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-a[107870]: 14: (Thread::_entry_func(void*)+0xd) [0x7fde3eb23749] 2024-05-27T07:30:28.000 INFO:journalctl@ceph.mds.a.smithi052.stdout:May 27 07:30:27 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-a[107870]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7fde3d6c71ca] 2024-05-27T07:30:28.000 INFO:journalctl@ceph.mds.a.smithi052.stdout:May 27 07:30:27 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-a[107870]: 16: clone() 2024-05-27T07:30:28.000 INFO:journalctl@ceph.mds.a.smithi052.stdout:May 27 07:30:27 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-a[107870]: 2024-05-27T07:30:28.000 INFO:journalctl@ceph.mds.a.smithi052.stdout:May 27 07:30:27 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-a[107870]: 0> 2024-05-27T07:30:27.516+0000 7fde2c30f700 -1 *** Caught signal (Aborted) ** 2024-05-27T07:30:28.000 INFO:journalctl@ceph.mds.a.smithi052.stdout:May 27 07:30:27 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-a[107870]: in thread 7fde2c30f700 thread_name:md_log_replay 2024-05-27T07:30:28.000 INFO:journalctl@ceph.mds.a.smithi052.stdout:May 27 07:30:27 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-a[107870]: 2024-05-27T07:30:28.000 INFO:journalctl@ceph.mds.a.smithi052.stdout:May 27 07:30:27 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-a[107870]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:30:28.000 INFO:journalctl@ceph.mds.a.smithi052.stdout:May 27 07:30:27 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-a[107870]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7fde3d6d1d20] 2024-05-27T07:30:28.000 INFO:journalctl@ceph.mds.a.smithi052.stdout:May 27 07:30:27 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-a[107870]: 2: gsignal() 2024-05-27T07:30:28.000 INFO:journalctl@ceph.mds.a.smithi052.stdout:May 27 07:30:27 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-a[107870]: 3: abort() 2024-05-27T07:30:28.000 INFO:journalctl@ceph.mds.a.smithi052.stdout:May 27 07:30:27 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-a[107870]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x26e) [0x7fde3eb54ced] 2024-05-27T07:30:28.000 INFO:journalctl@ceph.mds.a.smithi052.stdout:May 27 07:30:27 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-a[107870]: 5: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7fde3eb54dd1] 2024-05-27T07:30:28.000 INFO:journalctl@ceph.mds.a.smithi052.stdout:May 27 07:30:27 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-a[107870]: 6: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x557d2b26923d] 2024-05-27T07:30:28.001 INFO:journalctl@ceph.mds.a.smithi052.stdout:May 27 07:30:27 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-a[107870]: 7: (SnapRealm::build_snap_set() const+0x323) [0x557d2b254971] 2024-05-27T07:30:28.001 INFO:journalctl@ceph.mds.a.smithi052.stdout:May 27 07:30:27 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-a[107870]: 8: (SnapRealm::check_cache() const+0x13e) [0x557d2b252fa0] 2024-05-27T07:30:28.001 INFO:journalctl@ceph.mds.a.smithi052.stdout:May 27 07:30:27 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-a[107870]: 9: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x557d2b253651] 2024-05-27T07:30:28.001 INFO:journalctl@ceph.mds.a.smithi052.stdout:May 27 07:30:27 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-a[107870]: 10: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x557d2b1e8347] 2024-05-27T07:30:28.001 INFO:journalctl@ceph.mds.a.smithi052.stdout:May 27 07:30:27 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-a[107870]: 11: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x557d2aee487e] 2024-05-27T07:30:28.001 INFO:journalctl@ceph.mds.a.smithi052.stdout:May 27 07:30:27 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-a[107870]: 12: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x2702) [0x557d2aee9e94] 2024-05-27T07:30:28.001 INFO:journalctl@ceph.mds.a.smithi052.stdout:May 27 07:30:27 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-a[107870]: 13: (EUpdate::replay(MDSRank*)+0x63) [0x557d2aef4969] 2024-05-27T07:30:28.001 INFO:journalctl@ceph.mds.a.smithi052.stdout:May 27 07:30:27 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-a[107870]: 14: (MDLog::_replay_thread()+0x1f38) [0x557d2b28d48c] 2024-05-27T07:30:28.001 INFO:journalctl@ceph.mds.a.smithi052.stdout:May 27 07:30:27 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-a[107870]: 15: (MDLog::ReplayThread::entry()+0x11) [0x557d2b2969cb] 2024-05-27T07:30:28.001 INFO:journalctl@ceph.mds.a.smithi052.stdout:May 27 07:30:27 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-a[107870]: 16: (Thread::entry_wrapper()+0x43) [0x7fde3eb2372d] 2024-05-27T07:30:28.001 INFO:journalctl@ceph.mds.a.smithi052.stdout:May 27 07:30:27 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-a[107870]: 17: (Thread::_entry_func(void*)+0xd) [0x7fde3eb23749] 2024-05-27T07:30:28.001 INFO:journalctl@ceph.mds.a.smithi052.stdout:May 27 07:30:27 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-a[107870]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7fde3d6c71ca] 2024-05-27T07:30:28.001 INFO:journalctl@ceph.mds.a.smithi052.stdout:May 27 07:30:27 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-a[107870]: 19: clone() 2024-05-27T07:30:28.002 INFO:journalctl@ceph.mds.a.smithi052.stdout:May 27 07:30:27 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-a[107870]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-05-27T07:30:28.002 INFO:journalctl@ceph.mds.a.smithi052.stdout:May 27 07:30:27 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-a[107870]: 2024-05-27T07:30:28.002 INFO:journalctl@ceph.mds.a.smithi052.stdout:May 27 07:30:27 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-a[107870]: -1> 2024-05-27T07:30:27.512+0000 7fde2c30f700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: In function 'std::set SnapClient::filter(const std::set&) const' thread 7fde2c30f700 time 2024-05-27T07:30:27.510364+0000 2024-05-27T07:30:28.002 INFO:journalctl@ceph.mds.a.smithi052.stdout:May 27 07:30:27 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-a[107870]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: 197: FAILED ceph_assert(cached_version > 0) 2024-05-27T07:30:28.002 INFO:journalctl@ceph.mds.a.smithi052.stdout:May 27 07:30:27 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-a[107870]: 2024-05-27T07:30:28.002 INFO:journalctl@ceph.mds.a.smithi052.stdout:May 27 07:30:27 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-a[107870]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:30:28.002 INFO:journalctl@ceph.mds.a.smithi052.stdout:May 27 07:30:27 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-a[107870]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x127) [0x7fde3eb54ba6] 2024-05-27T07:30:28.002 INFO:journalctl@ceph.mds.a.smithi052.stdout:May 27 07:30:27 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-a[107870]: 2: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7fde3eb54dd1] 2024-05-27T07:30:28.002 INFO:journalctl@ceph.mds.a.smithi052.stdout:May 27 07:30:27 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-a[107870]: 3: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x557d2b26923d] 2024-05-27T07:30:28.002 INFO:journalctl@ceph.mds.a.smithi052.stdout:May 27 07:30:27 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-a[107870]: 4: (SnapRealm::build_snap_set() const+0x323) [0x557d2b254971] 2024-05-27T07:30:28.002 INFO:journalctl@ceph.mds.a.smithi052.stdout:May 27 07:30:27 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-a[107870]: 5: (SnapRealm::check_cache() const+0x13e) [0x557d2b252fa0] 2024-05-27T07:30:28.002 INFO:journalctl@ceph.mds.a.smithi052.stdout:May 27 07:30:27 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-a[107870]: 6: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x557d2b253651] 2024-05-27T07:30:28.003 INFO:journalctl@ceph.mds.a.smithi052.stdout:May 27 07:30:27 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-a[107870]: 7: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x557d2b1e8347] 2024-05-27T07:30:28.003 INFO:journalctl@ceph.mds.a.smithi052.stdout:May 27 07:30:27 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-a[107870]: 8: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x557d2aee487e] 2024-05-27T07:30:28.003 INFO:journalctl@ceph.mds.a.smithi052.stdout:May 27 07:30:27 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-a[107870]: 9: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x2702) [0x557d2aee9e94] 2024-05-27T07:30:28.003 INFO:journalctl@ceph.mds.a.smithi052.stdout:May 27 07:30:27 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-a[107870]: 10: (EUpdate::replay(MDSRank*)+0x63) [0x557d2aef4969] 2024-05-27T07:30:28.003 INFO:journalctl@ceph.mds.a.smithi052.stdout:May 27 07:30:27 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-a[107870]: 11: (MDLog::_replay_thread()+0x1f38) [0x557d2b28d48c] 2024-05-27T07:30:28.003 INFO:journalctl@ceph.mds.a.smithi052.stdout:May 27 07:30:27 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-a[107870]: 12: (MDLog::ReplayThread::entry()+0x11) [0x557d2b2969cb] 2024-05-27T07:30:28.003 INFO:journalctl@ceph.mds.a.smithi052.stdout:May 27 07:30:27 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-a[107870]: 13: (Thread::entry_wrapper()+0x43) [0x7fde3eb2372d] 2024-05-27T07:30:28.003 INFO:journalctl@ceph.mds.a.smithi052.stdout:May 27 07:30:27 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-a[107870]: 14: (Thread::_entry_func(void*)+0xd) [0x7fde3eb23749] 2024-05-27T07:30:28.003 INFO:journalctl@ceph.mds.a.smithi052.stdout:May 27 07:30:27 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-a[107870]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7fde3d6c71ca] 2024-05-27T07:30:28.003 INFO:journalctl@ceph.mds.a.smithi052.stdout:May 27 07:30:27 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-a[107870]: 16: clone() 2024-05-27T07:30:28.003 INFO:journalctl@ceph.mds.a.smithi052.stdout:May 27 07:30:27 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-a[107870]: 2024-05-27T07:30:28.003 INFO:journalctl@ceph.mds.a.smithi052.stdout:May 27 07:30:27 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-a[107870]: 0> 2024-05-27T07:30:27.516+0000 7fde2c30f700 -1 *** Caught signal (Aborted) ** 2024-05-27T07:30:28.003 INFO:journalctl@ceph.mds.a.smithi052.stdout:May 27 07:30:27 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-a[107870]: in thread 7fde2c30f700 thread_name:md_log_replay 2024-05-27T07:30:28.004 INFO:journalctl@ceph.mds.a.smithi052.stdout:May 27 07:30:27 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-a[107870]: 2024-05-27T07:30:28.004 INFO:journalctl@ceph.mds.a.smithi052.stdout:May 27 07:30:27 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-a[107870]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:30:28.004 INFO:journalctl@ceph.mds.a.smithi052.stdout:May 27 07:30:27 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-a[107870]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7fde3d6d1d20] 2024-05-27T07:30:28.004 INFO:journalctl@ceph.mds.a.smithi052.stdout:May 27 07:30:27 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-a[107870]: 2: gsignal() 2024-05-27T07:30:28.004 INFO:journalctl@ceph.mds.a.smithi052.stdout:May 27 07:30:27 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-a[107870]: 3: abort() 2024-05-27T07:30:28.004 INFO:journalctl@ceph.mds.a.smithi052.stdout:May 27 07:30:27 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-a[107870]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x26e) [0x7fde3eb54ced] 2024-05-27T07:30:28.004 INFO:journalctl@ceph.mds.a.smithi052.stdout:May 27 07:30:27 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-a[107870]: 5: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7fde3eb54dd1] 2024-05-27T07:30:28.004 INFO:journalctl@ceph.mds.a.smithi052.stdout:May 27 07:30:27 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-a[107870]: 6: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x557d2b26923d] 2024-05-27T07:30:28.004 INFO:journalctl@ceph.mds.a.smithi052.stdout:May 27 07:30:27 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-a[107870]: 7: (SnapRealm::build_snap_set() const+0x323) [0x557d2b254971] 2024-05-27T07:30:28.004 INFO:journalctl@ceph.mds.a.smithi052.stdout:May 27 07:30:27 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-a[107870]: 8: (SnapRealm::check_cache() const+0x13e) [0x557d2b252fa0] 2024-05-27T07:30:28.004 INFO:journalctl@ceph.mds.a.smithi052.stdout:May 27 07:30:27 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-a[107870]: 9: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x557d2b253651] 2024-05-27T07:30:28.004 INFO:journalctl@ceph.mds.a.smithi052.stdout:May 27 07:30:27 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-a[107870]: 10: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x557d2b1e8347] 2024-05-27T07:30:28.004 INFO:journalctl@ceph.mds.a.smithi052.stdout:May 27 07:30:27 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-a[107870]: 11: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x557d2aee487e] 2024-05-27T07:30:28.005 INFO:journalctl@ceph.mds.a.smithi052.stdout:May 27 07:30:27 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-a[107870]: 12: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x2702) [0x557d2aee9e94] 2024-05-27T07:30:28.005 INFO:journalctl@ceph.mds.a.smithi052.stdout:May 27 07:30:27 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-a[107870]: 13: (EUpdate::replay(MDSRank*)+0x63) [0x557d2aef4969] 2024-05-27T07:30:28.005 INFO:journalctl@ceph.mds.a.smithi052.stdout:May 27 07:30:27 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-a[107870]: 14: (MDLog::_replay_thread()+0x1f38) [0x557d2b28d48c] 2024-05-27T07:30:28.005 INFO:journalctl@ceph.mds.a.smithi052.stdout:May 27 07:30:27 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-a[107870]: 15: (MDLog::ReplayThread::entry()+0x11) [0x557d2b2969cb] 2024-05-27T07:30:28.005 INFO:journalctl@ceph.mds.a.smithi052.stdout:May 27 07:30:27 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-a[107870]: 16: (Thread::entry_wrapper()+0x43) [0x7fde3eb2372d] 2024-05-27T07:30:28.005 INFO:journalctl@ceph.mds.a.smithi052.stdout:May 27 07:30:27 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-a[107870]: 17: (Thread::_entry_func(void*)+0xd) [0x7fde3eb23749] 2024-05-27T07:30:28.005 INFO:journalctl@ceph.mds.a.smithi052.stdout:May 27 07:30:27 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-a[107870]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7fde3d6c71ca] 2024-05-27T07:30:28.005 INFO:journalctl@ceph.mds.a.smithi052.stdout:May 27 07:30:27 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-a[107870]: 19: clone() 2024-05-27T07:30:28.005 INFO:journalctl@ceph.mds.a.smithi052.stdout:May 27 07:30:27 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-a[107870]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-05-27T07:30:28.005 INFO:journalctl@ceph.mds.a.smithi052.stdout:May 27 07:30:27 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-a[107870]: 2024-05-27T07:30:28.005 INFO:journalctl@ceph.mds.a.smithi052.stdout:May 27 07:30:27 smithi052 podman[107914]: 2024-05-27 07:30:27.686285126 +0000 UTC m=+0.085030073 container died 893ffb3422f9f447e60922f01570ff78e7dafcf0c65edf706bc1782ef760a66d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-a, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, org.label-schema.build-date=20240521, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, org.label-schema.license=GPLv2) 2024-05-27T07:30:28.353 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:30:28 smithi173 ceph-mon[37000]: mds.? [v2:172.21.15.52:6834/51372710,v1:172.21.15.52:6835/51372710] up:boot 2024-05-27T07:30:28.353 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:30:28 smithi173 ceph-mon[37000]: fsmap cephfs:5 2 up:standby-replay 1 up:standby 5 up:active 2024-05-27T07:30:28.353 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:30:28 smithi173 ceph-mon[37000]: from='mgr.15333 172.21.15.52:0/416389082' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "a"}]: dispatch 2024-05-27T07:30:28.353 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:30:28 smithi173 ceph-mon[37000]: pgmap v160: 129 pgs: 129 active+clean; 342 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 2.8 MiB/s rd, 5.0 MiB/s wr, 371 op/s 2024-05-27T07:30:28.353 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:30:28 smithi173 ceph-mon[37000]: fsmap cephfs:5 3 up:standby-replay 5 up:active 2024-05-27T07:30:28.441 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:30:28 smithi105 ceph-mon[35977]: mds.? [v2:172.21.15.52:6834/51372710,v1:172.21.15.52:6835/51372710] up:boot 2024-05-27T07:30:28.441 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:30:28 smithi105 ceph-mon[35977]: fsmap cephfs:5 2 up:standby-replay 1 up:standby 5 up:active 2024-05-27T07:30:28.441 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:30:28 smithi105 ceph-mon[35977]: from='mgr.15333 172.21.15.52:0/416389082' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "a"}]: dispatch 2024-05-27T07:30:28.441 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:30:28 smithi105 ceph-mon[35977]: pgmap v160: 129 pgs: 129 active+clean; 342 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 2.8 MiB/s rd, 5.0 MiB/s wr, 371 op/s 2024-05-27T07:30:28.442 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:30:28 smithi105 ceph-mon[35977]: fsmap cephfs:5 3 up:standby-replay 5 up:active 2024-05-27T07:30:28.481 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:30:28 smithi052 ceph-mon[31955]: mds.? [v2:172.21.15.52:6834/51372710,v1:172.21.15.52:6835/51372710] up:boot 2024-05-27T07:30:28.481 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:30:28 smithi052 ceph-mon[31955]: fsmap cephfs:5 2 up:standby-replay 1 up:standby 5 up:active 2024-05-27T07:30:28.481 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:30:28 smithi052 ceph-mon[31955]: from='mgr.15333 172.21.15.52:0/416389082' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "a"}]: dispatch 2024-05-27T07:30:28.481 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:30:28 smithi052 ceph-mon[31955]: pgmap v160: 129 pgs: 129 active+clean; 342 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 2.8 MiB/s rd, 5.0 MiB/s wr, 371 op/s 2024-05-27T07:30:28.481 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:30:28 smithi052 ceph-mon[31955]: fsmap cephfs:5 3 up:standby-replay 5 up:active 2024-05-27T07:30:28.981 INFO:journalctl@ceph.mds.a.smithi052.stdout:May 27 07:30:28 smithi052 podman[107914]: 2024-05-27 07:30:28.585366006 +0000 UTC m=+0.984110952 container remove 893ffb3422f9f447e60922f01570ff78e7dafcf0c65edf706bc1782ef760a66d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-a, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, org.label-schema.build-date=20240521, org.label-schema.schema-version=1.0) 2024-05-27T07:30:28.981 INFO:journalctl@ceph.mds.a.smithi052.stdout:May 27 07:30:28 smithi052 systemd[1]: ceph-73df2636-1bf8-11ef-bc9b-c7b262605968@mds.a.service: Main process exited, code=exited, status=134/n/a 2024-05-27T07:30:29.731 INFO:journalctl@ceph.mds.a.smithi052.stdout:May 27 07:30:29 smithi052 systemd[1]: ceph-73df2636-1bf8-11ef-bc9b-c7b262605968@mds.a.service: Failed with result 'exit-code'. 2024-05-27T07:30:29.731 INFO:journalctl@ceph.mds.a.smithi052.stdout:May 27 07:30:29 smithi052 systemd[1]: ceph-73df2636-1bf8-11ef-bc9b-c7b262605968@mds.a.service: Consumed 1.432s CPU time. 2024-05-27T07:30:30.353 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:30:30 smithi173 ceph-mon[37000]: pgmap v161: 129 pgs: 129 active+clean; 342 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 2.8 MiB/s rd, 3.4 MiB/s wr, 274 op/s 2024-05-27T07:30:30.441 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:30:30 smithi105 ceph-mon[35977]: pgmap v161: 129 pgs: 129 active+clean; 342 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 2.8 MiB/s rd, 3.4 MiB/s wr, 274 op/s 2024-05-27T07:30:30.481 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:30:30 smithi052 ceph-mon[31955]: pgmap v161: 129 pgs: 129 active+clean; 342 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 2.8 MiB/s rd, 3.4 MiB/s wr, 274 op/s 2024-05-27T07:30:32.353 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:30:32 smithi173 ceph-mon[37000]: pgmap v162: 129 pgs: 129 active+clean; 348 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 MiB/s rd, 2.8 MiB/s wr, 365 op/s 2024-05-27T07:30:32.441 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:30:32 smithi105 ceph-mon[35977]: pgmap v162: 129 pgs: 129 active+clean; 348 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 MiB/s rd, 2.8 MiB/s wr, 365 op/s 2024-05-27T07:30:32.481 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:30:32 smithi052 ceph-mon[31955]: pgmap v162: 129 pgs: 129 active+clean; 348 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 MiB/s rd, 2.8 MiB/s wr, 365 op/s 2024-05-27T07:30:33.757 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:30:33.765+0000 7fce2673f640 1 -- 172.21.15.52:0/281458226 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fce201032e0 msgr2=0x7fce2010f6b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:30:33.757 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:30:33.765+0000 7fce2673f640 1 --2- 172.21.15.52:0/281458226 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fce201032e0 0x7fce2010f6b0 secure :-1 s=READY pgs=643 cs=0 l=1 rev1=1 crypto rx=0x7fce18009ca0 tx=0x7fce1802a8c0 comp rx=0 tx=0).stop 2024-05-27T07:30:33.758 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:30:33.765+0000 7fce2673f640 1 -- 172.21.15.52:0/281458226 shutdown_connections 2024-05-27T07:30:33.758 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:30:33.765+0000 7fce2673f640 1 --2- 172.21.15.52:0/281458226 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fce201032e0 0x7fce2010f6b0 unknown :-1 s=CLOSED pgs=643 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:30:33.758 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:30:33.765+0000 7fce2673f640 1 --2- 172.21.15.52:0/281458226 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fce20109030 0x7fce20102c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:30:33.758 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:30:33.765+0000 7fce2673f640 1 --2- 172.21.15.52:0/281458226 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fce20108b90 0x7fce20102730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:30:33.758 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:30:33.765+0000 7fce2673f640 1 -- 172.21.15.52:0/281458226 >> 172.21.15.52:0/281458226 conn(0x7fce200782d0 msgr2=0x7fce20078710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:30:33.758 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:30:33.766+0000 7fce2673f640 1 -- 172.21.15.52:0/281458226 shutdown_connections 2024-05-27T07:30:33.758 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:30:33.766+0000 7fce2673f640 1 -- 172.21.15.52:0/281458226 wait complete. 2024-05-27T07:30:33.760 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:30:33.766+0000 7fce2673f640 1 Processor -- start 2024-05-27T07:30:33.760 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:30:33.766+0000 7fce2673f640 1 -- start start 2024-05-27T07:30:33.760 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:30:33.767+0000 7fce2673f640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fce201032e0 0x7fce2019b6e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:30:33.760 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:30:33.767+0000 7fce2673f640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fce20108b90 0x7fce2019bc20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:30:33.760 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:30:33.767+0000 7fce2673f640 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fce20109030 0x7fce2019c160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:30:33.761 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:30:33.767+0000 7fce2673f640 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x7fce20112da0 con 0x7fce20109030 2024-05-27T07:30:33.761 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:30:33.767+0000 7fce2673f640 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7fce20112c20 con 0x7fce20108b90 2024-05-27T07:30:33.761 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:30:33.767+0000 7fce2673f640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7fce20112f20 con 0x7fce201032e0 2024-05-27T07:30:33.761 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:30:33.767+0000 7fce24cdc640 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fce20109030 0x7fce2019c160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:30:33.761 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:30:33.767+0000 7fce24cdc640 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fce20109030 0x7fce2019c160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.52:40138/0 (socket says 172.21.15.52:40138) 2024-05-27T07:30:33.761 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:30:33.767+0000 7fce24cdc640 1 -- 172.21.15.52:0/3834476284 learned_addr learned my addr 172.21.15.52:0/3834476284 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-05-27T07:30:33.761 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:30:33.767+0000 7fce1f7fe640 1 --2- 172.21.15.52:0/3834476284 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fce20108b90 0x7fce2019bc20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:30:33.761 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:30:33.767+0000 7fce1ffff640 1 --2- 172.21.15.52:0/3834476284 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fce201032e0 0x7fce2019b6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:30:33.761 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:30:33.767+0000 7fce24cdc640 1 -- 172.21.15.52:0/3834476284 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fce201032e0 msgr2=0x7fce2019b6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:30:33.761 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:30:33.767+0000 7fce24cdc640 1 --2- 172.21.15.52:0/3834476284 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fce201032e0 0x7fce2019b6e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:30:33.761 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:30:33.767+0000 7fce24cdc640 1 -- 172.21.15.52:0/3834476284 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fce20108b90 msgr2=0x7fce2019bc20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:30:33.761 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:30:33.767+0000 7fce24cdc640 1 --2- 172.21.15.52:0/3834476284 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fce20108b90 0x7fce2019bc20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:30:33.762 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:30:33.767+0000 7fce24cdc640 1 -- 172.21.15.52:0/3834476284 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fce201a69f0 con 0x7fce20109030 2024-05-27T07:30:33.762 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:30:33.767+0000 7fce1f7fe640 1 --2- 172.21.15.52:0/3834476284 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fce20108b90 0x7fce2019bc20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-27T07:30:33.762 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:30:33.768+0000 7fce24cdc640 1 --2- 172.21.15.52:0/3834476284 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fce20109030 0x7fce2019c160 secure :-1 s=READY pgs=644 cs=0 l=1 rev1=1 crypto rx=0x7fce180098d0 tx=0x7fce180317b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:30:33.762 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:30:33.768+0000 7fce1d7fa640 1 -- 172.21.15.52:0/3834476284 <== mon.0 v2:172.21.15.52:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fce18003b70 con 0x7fce20109030 2024-05-27T07:30:33.762 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:30:33.768+0000 7fce2673f640 1 -- 172.21.15.52:0/3834476284 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fce201a6c80 con 0x7fce20109030 2024-05-27T07:30:33.762 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:30:33.768+0000 7fce1d7fa640 1 -- 172.21.15.52:0/3834476284 <== mon.0 v2:172.21.15.52:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fce18003d10 con 0x7fce20109030 2024-05-27T07:30:33.762 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:30:33.768+0000 7fce1d7fa640 1 -- 172.21.15.52:0/3834476284 <== mon.0 v2:172.21.15.52:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fce1802fa20 con 0x7fce20109030 2024-05-27T07:30:33.762 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:30:33.768+0000 7fce2673f640 1 -- 172.21.15.52:0/3834476284 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fce201a71d0 con 0x7fce20109030 2024-05-27T07:30:33.764 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:30:33.771+0000 7fce2d5f9740 1 Processor -- start 2024-05-27T07:30:33.764 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:30:33.771+0000 7fce1d7fa640 1 -- 172.21.15.52:0/3834476284 <== mon.0 v2:172.21.15.52:3300/0 4 ==== mgrmap(e 26) ==== 149783+0+0 (secure 0 0 0) 0x7fce1802fc40 con 0x7fce20109030 2024-05-27T07:30:33.764 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:30:33.771+0000 7fce2d5f9740 1 -- start start 2024-05-27T07:30:33.764 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:30:33.771+0000 7fce2d5f9740 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x562c892fa1b0 0x562c8969b3d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:30:33.764 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:30:33.771+0000 7fce2d5f9740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x562c892d12f0 0x562c8969b910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:30:33.765 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:30:33.771+0000 7fce2d5f9740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x562c892d94d0 0x562c8969be50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:30:33.765 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:30:33.771+0000 7fce2d5f9740 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x562c8960dc30 con 0x562c892fa1b0 2024-05-27T07:30:33.765 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:30:33.771+0000 7fce2d5f9740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x562c895fd7d0 con 0x562c892d94d0 2024-05-27T07:30:33.765 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:30:33.771+0000 7fce2d5f9740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x562c89601810 con 0x562c892d12f0 2024-05-27T07:30:33.765 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:30:33.771+0000 7fce1f7fe640 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x562c892fa1b0 0x562c8969b3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:30:33.765 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:30:33.771+0000 7fce1f7fe640 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x562c892fa1b0 0x562c8969b3d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.52:40150/0 (socket says 172.21.15.52:40150) 2024-05-27T07:30:33.765 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:30:33.771+0000 7fce1f7fe640 1 -- 172.21.15.52:0/724730062 learned_addr learned my addr 172.21.15.52:0/724730062 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-05-27T07:30:33.765 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:30:33.771+0000 7fce1ffff640 1 --2- 172.21.15.52:0/724730062 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x562c892d12f0 0x562c8969b910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:30:33.765 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:30:33.772+0000 7fce1d7fa640 1 --2- 172.21.15.52:0/3834476284 >> [v2:172.21.15.52:6800/1822318384,v1:172.21.15.52:6801/1822318384] conn(0x7fce080b2950 0x7fce080b4e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:30:33.766 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:30:33.772+0000 7fce1f7fe640 1 --2- 172.21.15.52:0/724730062 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x562c892d94d0 0x562c8969be50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:30:33.766 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:30:33.772+0000 7fce1f7fe640 1 -- 172.21.15.52:0/724730062 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x562c892d12f0 msgr2=0x562c8969b910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:30:33.766 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:30:33.772+0000 7fce24cdc640 1 --2- 172.21.15.52:0/3834476284 >> [v2:172.21.15.52:6800/1822318384,v1:172.21.15.52:6801/1822318384] conn(0x7fce080b2950 0x7fce080b4e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:30:33.766 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:30:33.772+0000 7fce1f7fe640 1 --2- 172.21.15.52:0/724730062 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x562c892d12f0 0x562c8969b910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:30:33.766 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:30:33.772+0000 7fce1f7fe640 1 -- 172.21.15.52:0/724730062 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x562c892d94d0 msgr2=0x562c8969be50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:30:33.766 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:30:33.772+0000 7fce1f7fe640 1 --2- 172.21.15.52:0/724730062 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x562c892d94d0 0x562c8969be50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:30:33.766 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:30:33.772+0000 7fce1f7fe640 1 -- 172.21.15.52:0/724730062 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x562c89312210 con 0x562c892fa1b0 2024-05-27T07:30:33.766 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:30:33.772+0000 7fce1d7fa640 1 -- 172.21.15.52:0/3834476284 <== mon.0 v2:172.21.15.52:3300/0 5 ==== osd_map(113..113 src has 1..113) ==== 9379+0+0 (secure 0 0 0) 0x7fce18103db0 con 0x7fce20109030 2024-05-27T07:30:33.766 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:30:33.772+0000 7fce1f7fe640 1 --2- 172.21.15.52:0/724730062 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x562c892fa1b0 0x562c8969b3d0 secure :-1 s=READY pgs=645 cs=0 l=1 rev1=1 crypto rx=0x7fce140058c0 tx=0x7fce1400bb10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e1fadf02b2a67828 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:30:33.766 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:30:33.772+0000 7fce067fc640 1 -- 172.21.15.52:0/724730062 <== mon.0 v2:172.21.15.52:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fce140122d0 con 0x562c892fa1b0 2024-05-27T07:30:33.766 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:30:33.772+0000 7fce067fc640 1 -- 172.21.15.52:0/724730062 <== mon.0 v2:172.21.15.52:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fce140054d0 con 0x562c892fa1b0 2024-05-27T07:30:33.766 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:30:33.772+0000 7fce067fc640 1 -- 172.21.15.52:0/724730062 <== mon.0 v2:172.21.15.52:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fce14004a00 con 0x562c892fa1b0 2024-05-27T07:30:33.767 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:30:33.772+0000 7fce24cdc640 1 --2- 172.21.15.52:0/3834476284 >> [v2:172.21.15.52:6800/1822318384,v1:172.21.15.52:6801/1822318384] conn(0x7fce080b2950 0x7fce080b4e60 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7fce180c80b0 tx=0x7fce180b1040 comp rx=0 tx=0).ready entity=mgr.15333 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:30:33.770 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:30:33.773+0000 7fce2d5f9740 1 -- 172.21.15.52:0/724730062 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x562c892fa1b0 msgr2=0x562c8969b3d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:30:33.770 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:30:33.773+0000 7fce2d5f9740 1 --2- 172.21.15.52:0/724730062 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x562c892fa1b0 0x562c8969b3d0 secure :-1 s=READY pgs=645 cs=0 l=1 rev1=1 crypto rx=0x7fce140058c0 tx=0x7fce1400bb10 comp rx=0 tx=0).stop 2024-05-27T07:30:33.771 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:30:33.773+0000 7fce2d5f9740 1 -- 172.21.15.52:0/724730062 shutdown_connections 2024-05-27T07:30:33.771 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:30:33.773+0000 7fce2d5f9740 1 --2- 172.21.15.52:0/724730062 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x562c892d94d0 0x562c8969be50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:30:33.771 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:30:33.773+0000 7fce2d5f9740 1 --2- 172.21.15.52:0/724730062 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x562c892d12f0 0x562c8969b910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:30:33.771 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:30:33.773+0000 7fce2d5f9740 1 --2- 172.21.15.52:0/724730062 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x562c892fa1b0 0x562c8969b3d0 unknown :-1 s=CLOSED pgs=645 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:30:33.771 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:30:33.773+0000 7fce2d5f9740 1 -- 172.21.15.52:0/724730062 >> 172.21.15.52:0/724730062 conn(0x562c892ee300 msgr2=0x562c89697b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:30:33.771 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:30:33.774+0000 7fce2d5f9740 1 -- 172.21.15.52:0/724730062 shutdown_connections 2024-05-27T07:30:33.771 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:30:33.774+0000 7fce2d5f9740 1 -- 172.21.15.52:0/724730062 wait complete. 2024-05-27T07:30:33.771 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:30:33.774+0000 7fce2d5f9740 1 -- start start 2024-05-27T07:30:33.771 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:30:33.774+0000 7fce2d5f9740 1 Processor -- start 2024-05-27T07:30:33.771 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:30:33.775+0000 7fce2d5f9740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x562c892d12f0 0x562c896a5ac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:30:33.772 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:30:33.775+0000 7fce2d5f9740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x562c892d94d0 0x562c896a6000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:30:33.772 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:30:33.775+0000 7fce2d5f9740 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x562c892fa1b0 0x562c896ab940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:30:33.772 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:30:33.775+0000 7fce2d5f9740 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x562c892e3790 con 0x562c892fa1b0 2024-05-27T07:30:33.772 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:30:33.775+0000 7fce2d5f9740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x562c895b9890 con 0x562c892d12f0 2024-05-27T07:30:33.772 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:30:33.775+0000 7fce2d5f9740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x562c896a8500 con 0x562c892d94d0 2024-05-27T07:30:33.772 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:30:33.775+0000 7fce2d5f9740 10 client.0 _ll_register_callbacks cb 0x562c89311a00 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T07:30:33.772 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:30:33.775+0000 7fce1f7fe640 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x562c892fa1b0 0x562c896ab940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:30:33.772 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:30:33.775+0000 7fce1f7fe640 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x562c892fa1b0 0x562c896ab940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.52:40158/0 (socket says 172.21.15.52:40158) 2024-05-27T07:30:33.772 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:30:33.775+0000 7fce1f7fe640 1 -- 172.21.15.52:0/1053825564 learned_addr learned my addr 172.21.15.52:0/1053825564 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-05-27T07:30:33.772 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:30:33.775+0000 7fce1f7fe640 1 --2- 172.21.15.52:0/1053825564 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x562c892d12f0 0x562c896a5ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:30:33.772 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:30:33.775+0000 7fce1ffff640 1 --2- 172.21.15.52:0/1053825564 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x562c892d94d0 0x562c896a6000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:30:33.772 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:30:33.776+0000 7fce1f7fe640 1 -- 172.21.15.52:0/1053825564 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x562c892d94d0 msgr2=0x562c896a6000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:30:33.773 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:30:33.776+0000 7fce1f7fe640 1 --2- 172.21.15.52:0/1053825564 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x562c892d94d0 0x562c896a6000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:30:33.773 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:30:33.776+0000 7fce1f7fe640 1 -- 172.21.15.52:0/1053825564 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x562c892d12f0 msgr2=0x562c896a5ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:30:33.773 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:30:33.776+0000 7fce1f7fe640 1 --2- 172.21.15.52:0/1053825564 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x562c892d12f0 0x562c896a5ac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:30:33.773 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:30:33.776+0000 7fce1f7fe640 1 -- 172.21.15.52:0/1053825564 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({osdmap=0}) -- 0x562c893de620 con 0x562c892fa1b0 2024-05-27T07:30:33.773 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:30:33.776+0000 7fce1f7fe640 1 --2- 172.21.15.52:0/1053825564 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x562c892fa1b0 0x562c896ab940 secure :-1 s=READY pgs=646 cs=0 l=1 rev1=1 crypto rx=0x562c896a5110 tx=0x7fce1400eda0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=71b4baa2d9c118ff server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:30:33.773 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:30:33.776+0000 7fcdff7fe640 10 client.0 ms_handle_connect on v2:172.21.15.52:3300/0 2024-05-27T07:30:33.773 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:30:33.776+0000 7fcdff7fe640 1 -- 172.21.15.52:0/1053825564 <== mon.0 v2:172.21.15.52:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fce14027040 con 0x562c892fa1b0 2024-05-27T07:30:33.773 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:30:33.776+0000 7fcdff7fe640 1 -- 172.21.15.52:0/1053825564 <== mon.0 v2:172.21.15.52:3300/0 2 ==== osd_map(113..113 src has 1..113) ==== 9379+0+0 (secure 0 0 0) 0x7fce140051b0 con 0x562c892fa1b0 2024-05-27T07:30:33.773 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:30:33.776+0000 7fce2d5f9740 1 -- 172.21.15.52:0/1053825564 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fce140054d0 con 0x562c892fa1b0 2024-05-27T07:30:33.773 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:30:33.777+0000 7fcdff7fe640 1 -- 172.21.15.52:0/1053825564 <== mon.0 v2:172.21.15.52:3300/0 3 ==== mon_get_version_reply(handle=1 version=89) ==== 24+0+0 (secure 0 0 0) 0x7fce140120d0 con 0x562c892fa1b0 2024-05-27T07:30:33.773 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:30:33.777+0000 7fce2d5f9740 10 client.16518 fetch_fsmap learned FSMap version 89 2024-05-27T07:30:33.773 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:30:33.777+0000 7fce2d5f9740 1 -- 172.21.15.52:0/1053825564 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({config=0+,fsmap=89,monmap=4+}) -- 0x562c896b4880 con 0x562c892fa1b0 2024-05-27T07:30:33.773 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:30:33.777+0000 7fcdff7fe640 1 -- 172.21.15.52:0/1053825564 <== mon.0 v2:172.21.15.52:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fce140054d0 con 0x562c892fa1b0 2024-05-27T07:30:33.774 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:30:33.777+0000 7fcdff7fe640 1 -- 172.21.15.52:0/1053825564 <== mon.0 v2:172.21.15.52:3300/0 5 ==== fsmap(e 89) ==== 5982+0+0 (secure 0 0 0) 0x7fce1401c020 con 0x562c892fa1b0 2024-05-27T07:30:33.774 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:30:33.777+0000 7fce2d5f9740 10 client.16518 fetch_fsmap finished waiting for FSMap version 89 2024-05-27T07:30:33.774 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:30:33.777+0000 7fce2d5f9740 10 client.16518 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-05-27T07:30:33.774 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:30:33.777+0000 7fce2d5f9740 20 client.16518 populate_metadata read hostname 'smithi052' 2024-05-27T07:30:33.774 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:30:33.777+0000 7fce2d5f9740 1 --2- 172.21.15.52:0/1053825564 >> [v2:172.21.15.105:6838/500348151,v1:172.21.15.105:6839/500348151] conn(0x562c892b8d80 0x562c896b7ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:30:33.774 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:30:33.777+0000 7fce2d5f9740 4 client.16518 mds_command: new command op to 24527 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-27T07:30:33.774 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:30:33.777+0000 7fce2d5f9740 1 -- 172.21.15.52:0/1053825564 --> [v2:172.21.15.105:6838/500348151,v1:172.21.15.105:6839/500348151] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x562c8960fb80 con 0x562c892b8d80 2024-05-27T07:30:33.774 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:30:33.777+0000 7fce1ffff640 1 --2- 172.21.15.52:0/1053825564 >> [v2:172.21.15.105:6838/500348151,v1:172.21.15.105:6839/500348151] conn(0x562c892b8d80 0x562c896b7ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:30:33.774 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:30:33.778+0000 7fcdff7fe640 10 client.16518 ms_handle_connect on v2:172.21.15.105:6838/500348151 2024-05-27T07:30:33.774 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:30:33.778+0000 7fce1ffff640 1 --2- 172.21.15.52:0/1053825564 >> [v2:172.21.15.105:6838/500348151,v1:172.21.15.105:6839/500348151] conn(0x562c892b8d80 0x562c896b7ff0 crc :-1 s=READY pgs=107 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=2f0c8b0104c264d9 server_cookie=862356045b3d3e4a in_seq=0 out_seq=0 2024-05-27T07:30:33.775 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:30:33.782+0000 7fcdff7fe640 1 -- 172.21.15.52:0/1053825564 <== mds.0 v2:172.21.15.105:6838/500348151 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24041 (crc 0 0 0) 0x562c8960fb80 con 0x562c892b8d80 2024-05-27T07:30:33.775 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:30:33.782+0000 7fcdff7fe640 10 client.16518 handle_command_reply: tid=0 2024-05-27T07:30:33.776 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:30:33.782+0000 7fce2d5f9740 1 client.16518 shutdown 2024-05-27T07:30:33.776 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:30:33.782+0000 7fce2d5f9740 1 -- 172.21.15.52:0/1053825564 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x562c892fa1b0 msgr2=0x562c896ab940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:30:33.776 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:30:33.782+0000 7fce2d5f9740 1 --2- 172.21.15.52:0/1053825564 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x562c892fa1b0 0x562c896ab940 secure :-1 s=READY pgs=646 cs=0 l=1 rev1=1 crypto rx=0x562c896a5110 tx=0x7fce1400eda0 comp rx=0 tx=0).stop 2024-05-27T07:30:33.776 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:30:33.782+0000 7fce2d5f9740 1 -- 172.21.15.52:0/1053825564 shutdown_connections 2024-05-27T07:30:33.776 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:30:33.782+0000 7fce2d5f9740 1 --2- 172.21.15.52:0/1053825564 >> [v2:172.21.15.105:6838/500348151,v1:172.21.15.105:6839/500348151] conn(0x562c892b8d80 0x562c896b7ff0 crc :-1 s=READY pgs=107 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:30:33.776 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:30:33.782+0000 7fce2d5f9740 1 --2- 172.21.15.52:0/1053825564 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x562c892fa1b0 0x562c896ab940 unknown :-1 s=CLOSED pgs=646 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:30:33.776 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:30:33.782+0000 7fce2d5f9740 1 --2- 172.21.15.52:0/1053825564 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x562c892d94d0 0x562c896a6000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:30:33.776 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:30:33.782+0000 7fcdff7fe640 0 client.16518 ms_handle_reset on v2:172.21.15.105:6838/500348151 2024-05-27T07:30:33.776 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:30:33.782+0000 7fce2d5f9740 1 --2- 172.21.15.52:0/1053825564 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x562c892d12f0 0x562c896a5ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:30:33.776 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:30:33.782+0000 7fce2d5f9740 1 -- 172.21.15.52:0/1053825564 >> 172.21.15.52:0/1053825564 conn(0x562c892ee300 msgr2=0x562c8969b1a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:30:33.776 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:30:33.783+0000 7fce2d5f9740 1 -- 172.21.15.52:0/1053825564 shutdown_connections 2024-05-27T07:30:33.776 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:30:33.783+0000 7fce2d5f9740 1 -- 172.21.15.52:0/1053825564 wait complete. 2024-05-27T07:30:33.776 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:30:33.783+0000 7fce2d5f9740 20 client.16518 trim_cache size 0 max 16384 2024-05-27T07:30:33.791 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:30:33.799+0000 7fce2d5f9740 1 Processor -- start 2024-05-27T07:30:33.791 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:30:33.799+0000 7fce2d5f9740 1 -- start start 2024-05-27T07:30:33.792 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:30:33.799+0000 7fce2d5f9740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x562c892b8d80 0x562c896ab7d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:30:33.792 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:30:33.799+0000 7fce2d5f9740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x562c892d12f0 0x562c896b2610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:30:33.792 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:30:33.799+0000 7fce2d5f9740 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x562c892d94d0 0x562c896abd10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:30:33.792 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:30:33.799+0000 7fce2d5f9740 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x7fce201b1ec0 con 0x562c892d94d0 2024-05-27T07:30:33.792 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:30:33.799+0000 7fce2d5f9740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7fce201b2590 con 0x562c892d12f0 2024-05-27T07:30:33.792 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:30:33.799+0000 7fce2d5f9740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7fce201afbf0 con 0x562c892b8d80 2024-05-27T07:30:33.792 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:30:33.799+0000 7fce1ffff640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x562c892d12f0 0x562c896b2610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:30:33.792 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:30:33.799+0000 7fce1ffff640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x562c892d12f0 0x562c896b2610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.52:36924/0 (socket says 172.21.15.52:36924) 2024-05-27T07:30:33.792 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:30:33.799+0000 7fce1ffff640 1 -- 172.21.15.52:0/4268896440 learned_addr learned my addr 172.21.15.52:0/4268896440 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-05-27T07:30:33.792 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:30:33.799+0000 7fce1f7fe640 1 --2- 172.21.15.52:0/4268896440 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x562c892d94d0 0x562c896abd10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:30:33.792 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:30:33.799+0000 7fce1f7fe640 1 --2- 172.21.15.52:0/4268896440 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x562c892b8d80 0x562c896ab7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:30:33.793 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:30:33.800+0000 7fce1ffff640 1 -- 172.21.15.52:0/4268896440 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x562c892b8d80 msgr2=0x562c896ab7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:30:33.793 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:30:33.800+0000 7fce1ffff640 1 --2- 172.21.15.52:0/4268896440 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x562c892b8d80 0x562c896ab7d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:30:33.793 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:30:33.800+0000 7fce1ffff640 1 -- 172.21.15.52:0/4268896440 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x562c892d94d0 msgr2=0x562c896abd10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:30:33.793 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:30:33.800+0000 7fce1ffff640 1 --2- 172.21.15.52:0/4268896440 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x562c892d94d0 0x562c896abd10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:30:33.793 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:30:33.800+0000 7fce1ffff640 1 -- 172.21.15.52:0/4268896440 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x562c896a6d70 con 0x562c892d12f0 2024-05-27T07:30:33.793 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:30:33.800+0000 7fce1ffff640 1 --2- 172.21.15.52:0/4268896440 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x562c892d12f0 0x562c896b2610 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x562c89698b10 tx=0x7fce10005f90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c40f9b782914b249 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:30:33.793 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:30:33.800+0000 7fce05ffb640 1 -- 172.21.15.52:0/4268896440 <== mon.2 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fce10004220 con 0x562c892d12f0 2024-05-27T07:30:33.793 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:30:33.800+0000 7fce05ffb640 1 -- 172.21.15.52:0/4268896440 <== mon.2 v2:172.21.15.105:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fce10012e50 con 0x562c892d12f0 2024-05-27T07:30:33.793 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:30:33.800+0000 7fce05ffb640 1 -- 172.21.15.52:0/4268896440 <== mon.2 v2:172.21.15.105:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fce10028880 con 0x562c892d12f0 2024-05-27T07:30:33.797 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:30:33.801+0000 7fce2d5f9740 1 -- 172.21.15.52:0/4268896440 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x562c892d12f0 msgr2=0x562c896b2610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:30:33.797 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:30:33.802+0000 7fce2d5f9740 1 --2- 172.21.15.52:0/4268896440 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x562c892d12f0 0x562c896b2610 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x562c89698b10 tx=0x7fce10005f90 comp rx=0 tx=0).stop 2024-05-27T07:30:33.797 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:30:33.802+0000 7fce2d5f9740 1 -- 172.21.15.52:0/4268896440 shutdown_connections 2024-05-27T07:30:33.797 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:30:33.802+0000 7fce2d5f9740 1 --2- 172.21.15.52:0/4268896440 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x562c892d94d0 0x562c896abd10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:30:33.797 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:30:33.802+0000 7fce2d5f9740 1 --2- 172.21.15.52:0/4268896440 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x562c892d12f0 0x562c896b2610 unknown :-1 s=CLOSED pgs=162 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:30:33.797 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:30:33.802+0000 7fce2d5f9740 1 --2- 172.21.15.52:0/4268896440 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x562c892b8d80 0x562c896ab7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:30:33.797 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:30:33.802+0000 7fce2d5f9740 1 -- 172.21.15.52:0/4268896440 >> 172.21.15.52:0/4268896440 conn(0x562c892ee300 msgr2=0x562c896a4180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:30:33.797 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:30:33.802+0000 7fce2d5f9740 1 -- 172.21.15.52:0/4268896440 shutdown_connections 2024-05-27T07:30:33.797 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:30:33.802+0000 7fce2d5f9740 1 -- 172.21.15.52:0/4268896440 wait complete. 2024-05-27T07:30:33.797 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:30:33.802+0000 7fce2d5f9740 1 -- start start 2024-05-27T07:30:33.798 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:30:33.803+0000 7fce2d5f9740 1 Processor -- start 2024-05-27T07:30:33.798 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:30:33.803+0000 7fce2d5f9740 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x562c892b8d80 0x562c896b1ee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:30:33.798 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:30:33.803+0000 7fce2d5f9740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x562c892d12f0 0x562c896b2420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:30:33.798 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:30:33.803+0000 7fce2d5f9740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x562c892d94d0 0x562c896ad2f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:30:33.798 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:30:33.803+0000 7fce2d5f9740 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x7fce201b02c0 con 0x562c892b8d80 2024-05-27T07:30:33.798 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:30:33.803+0000 7fce2d5f9740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7fce201b17f0 con 0x562c892d12f0 2024-05-27T07:30:33.798 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:30:33.803+0000 7fce2d5f9740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7fce201b0a50 con 0x562c892d94d0 2024-05-27T07:30:33.798 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:30:33.803+0000 7fce2d5f9740 10 client.0 _ll_register_callbacks cb 0x7fce100094f0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T07:30:33.798 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:30:33.803+0000 7fce1f7fe640 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x562c892b8d80 0x562c896b1ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:30:33.798 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:30:33.803+0000 7fce1ffff640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x562c892d12f0 0x562c896b2420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:30:33.798 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:30:33.803+0000 7fce1ffff640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x562c892d12f0 0x562c896b2420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.52:36932/0 (socket says 172.21.15.52:36932) 2024-05-27T07:30:33.798 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:30:33.803+0000 7fce1ffff640 1 -- 172.21.15.52:0/1351519904 learned_addr learned my addr 172.21.15.52:0/1351519904 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-05-27T07:30:33.798 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:30:33.803+0000 7fce1f7fe640 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x562c892b8d80 0x562c896b1ee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.52:40180/0 (socket says 172.21.15.52:40180) 2024-05-27T07:30:33.799 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:30:33.803+0000 7fce1f7fe640 1 --2- 172.21.15.52:0/1351519904 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x562c892d94d0 0x562c896ad2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:30:33.799 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:30:33.804+0000 7fce1ffff640 1 -- 172.21.15.52:0/1351519904 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x562c892d94d0 msgr2=0x562c896ad2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:30:33.799 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:30:33.804+0000 7fce1ffff640 1 --2- 172.21.15.52:0/1351519904 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x562c892d94d0 0x562c896ad2f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:30:33.799 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:30:33.804+0000 7fce1ffff640 1 -- 172.21.15.52:0/1351519904 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x562c892b8d80 msgr2=0x562c896b1ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:30:33.799 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:30:33.804+0000 7fce1ffff640 1 --2- 172.21.15.52:0/1351519904 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x562c892b8d80 0x562c896b1ee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:30:33.799 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:30:33.804+0000 7fce1ffff640 1 -- 172.21.15.52:0/1351519904 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({osdmap=0}) -- 0x562c896ad9e0 con 0x562c892d12f0 2024-05-27T07:30:33.799 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:30:33.804+0000 7fce1f7fe640 1 --2- 172.21.15.52:0/1351519904 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x562c892d94d0 0x562c896ad2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-27T07:30:33.799 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:30:33.804+0000 7fce1ffff640 1 --2- 172.21.15.52:0/1351519904 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x562c892d12f0 0x562c896b2420 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7fce100284c0 tx=0x7fce10004160 comp rx=0 tx=0).ready entity=mon.2 client_cookie=7275d5806252364c server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:30:33.799 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:30:33.804+0000 7fcdff7fe640 10 client.0 ms_handle_connect on v2:172.21.15.105:3300/0 2024-05-27T07:30:33.799 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:30:33.804+0000 7fcdff7fe640 1 -- 172.21.15.52:0/1351519904 <== mon.2 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fce100037f0 con 0x562c892d12f0 2024-05-27T07:30:33.799 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:30:33.804+0000 7fce2d5f9740 1 -- 172.21.15.52:0/1351519904 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fce10012e50 con 0x562c892d12f0 2024-05-27T07:30:33.799 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:30:33.804+0000 7fcdff7fe640 1 -- 172.21.15.52:0/1351519904 <== mon.2 v2:172.21.15.105:3300/0 2 ==== osd_map(113..113 src has 1..113) ==== 9379+0+0 (secure 0 0 0) 0x7fce10003990 con 0x562c892d12f0 2024-05-27T07:30:33.799 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:30:33.804+0000 7fcdff7fe640 1 -- 172.21.15.52:0/1351519904 <== mon.2 v2:172.21.15.105:3300/0 3 ==== mon_get_version_reply(handle=1 version=89) ==== 24+0+0 (secure 0 0 0) 0x7fce10003b90 con 0x562c892d12f0 2024-05-27T07:30:33.800 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:30:33.804+0000 7fce2d5f9740 10 client.26177 fetch_fsmap learned FSMap version 89 2024-05-27T07:30:33.800 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:30:33.804+0000 7fce2d5f9740 1 -- 172.21.15.52:0/1351519904 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,fsmap=89,monmap=4+}) -- 0x562c897200a0 con 0x562c892d12f0 2024-05-27T07:30:33.800 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:30:33.804+0000 7fcdff7fe640 1 -- 172.21.15.52:0/1351519904 <== mon.2 v2:172.21.15.105:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fce10012e50 con 0x562c892d12f0 2024-05-27T07:30:33.800 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:30:33.805+0000 7fcdff7fe640 1 -- 172.21.15.52:0/1351519904 <== mon.2 v2:172.21.15.105:3300/0 5 ==== fsmap(e 89) ==== 5982+0+0 (secure 0 0 0) 0x7fce1001f500 con 0x562c892d12f0 2024-05-27T07:30:33.800 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:30:33.805+0000 7fce2d5f9740 10 client.26177 fetch_fsmap finished waiting for FSMap version 89 2024-05-27T07:30:33.800 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:30:33.805+0000 7fce2d5f9740 10 client.26177 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-05-27T07:30:33.800 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:30:33.805+0000 7fce2d5f9740 20 client.26177 populate_metadata read hostname 'smithi052' 2024-05-27T07:30:33.800 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:30:33.805+0000 7fce2d5f9740 1 --2- 172.21.15.52:0/1351519904 >> [v2:172.21.15.105:6838/500348151,v1:172.21.15.105:6839/500348151] conn(0x562c892fa1b0 0x562c89722950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:30:33.800 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:30:33.805+0000 7fce2d5f9740 4 client.26177 mds_command: new command op to 24527 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-05-27T07:30:33.800 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:30:33.805+0000 7fce2d5f9740 1 -- 172.21.15.52:0/1351519904 --> [v2:172.21.15.105:6838/500348151,v1:172.21.15.105:6839/500348151] -- command(tid 0: {"prefix": "flush journal"}) -- 0x562c89609360 con 0x562c892fa1b0 2024-05-27T07:30:33.800 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:30:33.805+0000 7fce1f7fe640 1 --2- 172.21.15.52:0/1351519904 >> [v2:172.21.15.105:6838/500348151,v1:172.21.15.105:6839/500348151] conn(0x562c892fa1b0 0x562c89722950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:30:33.800 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:30:33.805+0000 7fcdff7fe640 10 client.26177 ms_handle_connect on v2:172.21.15.105:6838/500348151 2024-05-27T07:30:33.801 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:30:33.805+0000 7fce1f7fe640 1 --2- 172.21.15.52:0/1351519904 >> [v2:172.21.15.105:6838/500348151,v1:172.21.15.105:6839/500348151] conn(0x562c892fa1b0 0x562c89722950 crc :-1 s=READY pgs=108 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=54f8b43ed421b4f8 server_cookie=dd576bd2377555e1 in_seq=0 out_seq=0 2024-05-27T07:30:34.353 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:30:34 smithi173 ceph-mon[37000]: pgmap v163: 129 pgs: 129 active+clean; 352 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 MiB/s rd, 2.0 MiB/s wr, 399 op/s 2024-05-27T07:30:34.441 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:30:34 smithi105 ceph-mon[35977]: pgmap v163: 129 pgs: 129 active+clean; 352 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 MiB/s rd, 2.0 MiB/s wr, 399 op/s 2024-05-27T07:30:34.489 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:30:34 smithi052 ceph-mon[31955]: pgmap v163: 129 pgs: 129 active+clean; 352 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 MiB/s rd, 2.0 MiB/s wr, 399 op/s 2024-05-27T07:30:34.981 INFO:journalctl@ceph.mds.j.smithi052.stdout:May 27 07:30:34 smithi052 systemd[1]: ceph-73df2636-1bf8-11ef-bc9b-c7b262605968@mds.j.service: Scheduled restart job, restart counter is at 3. 2024-05-27T07:30:34.981 INFO:journalctl@ceph.mds.j.smithi052.stdout:May 27 07:30:34 smithi052 systemd[1]: Stopped Ceph mds.j for 73df2636-1bf8-11ef-bc9b-c7b262605968. 2024-05-27T07:30:34.981 INFO:journalctl@ceph.mds.j.smithi052.stdout:May 27 07:30:34 smithi052 systemd[1]: ceph-73df2636-1bf8-11ef-bc9b-c7b262605968@mds.j.service: Consumed 1.333s CPU time. 2024-05-27T07:30:34.981 INFO:journalctl@ceph.mds.j.smithi052.stdout:May 27 07:30:34 smithi052 systemd[1]: Starting Ceph mds.j for 73df2636-1bf8-11ef-bc9b-c7b262605968... 2024-05-27T07:30:35.209 INFO:tasks.workunit.client.0.smithi052.stdout: 105 1534085 433 1096489 424 837440 1169 2024-05-27T07:30:35.446 INFO:journalctl@ceph.mds.j.smithi052.stdout:May 27 07:30:35 smithi052 podman[108044]: 2024-05-27 07:30:34.992037594 +0000 UTC m=+0.085899899 image pull e36ea4fc8a5807e8d2701370e350a0de3856538797f7fba64cfce7e344bd6799 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6 2024-05-27T07:30:35.446 INFO:journalctl@ceph.mds.j.smithi052.stdout:May 27 07:30:35 smithi052 podman[108044]: 2024-05-27 07:30:35.303556095 +0000 UTC m=+0.397418373 container create a24bb58d90f111c96437ec4e2d4bf19550bfe06ab1d48ffd2e6055a142538066 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-j, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, ceph=True, org.label-schema.license=GPLv2, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, org.label-schema.build-date=20240521, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5) 2024-05-27T07:30:35.710 INFO:journalctl@ceph.mds.j.smithi052.stdout:May 27 07:30:35 smithi052 podman[108044]: 2024-05-27 07:30:35.586210546 +0000 UTC m=+0.680072841 container init a24bb58d90f111c96437ec4e2d4bf19550bfe06ab1d48ffd2e6055a142538066 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-j, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240521, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, GIT_CLEAN=True, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image) 2024-05-27T07:30:35.711 INFO:journalctl@ceph.mds.j.smithi052.stdout:May 27 07:30:35 smithi052 podman[108044]: 2024-05-27 07:30:35.605154766 +0000 UTC m=+0.699017056 container start a24bb58d90f111c96437ec4e2d4bf19550bfe06ab1d48ffd2e6055a142538066 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-j, GIT_CLEAN=True, org.label-schema.build-date=20240521, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, ceph=True, maintainer=Guillaume Abrioux , GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0) 2024-05-27T07:30:35.711 INFO:journalctl@ceph.mds.j.smithi052.stdout:May 27 07:30:35 smithi052 bash[108044]: a24bb58d90f111c96437ec4e2d4bf19550bfe06ab1d48ffd2e6055a142538066 2024-05-27T07:30:35.711 INFO:journalctl@ceph.mds.j.smithi052.stdout:May 27 07:30:35 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-j[108055]: starting mds.j at 2024-05-27T07:30:35.711 INFO:journalctl@ceph.mds.j.smithi052.stdout:May 27 07:30:35 smithi052 systemd[1]: Started Ceph mds.j for 73df2636-1bf8-11ef-bc9b-c7b262605968. 2024-05-27T07:30:36.092 INFO:journalctl@ceph.mds.k.smithi105.stdout:May 27 07:30:35 smithi105 systemd[1]: ceph-73df2636-1bf8-11ef-bc9b-c7b262605968@mds.k.service: Scheduled restart job, restart counter is at 6. 2024-05-27T07:30:36.092 INFO:journalctl@ceph.mds.k.smithi105.stdout:May 27 07:30:35 smithi105 systemd[1]: Stopped Ceph mds.k for 73df2636-1bf8-11ef-bc9b-c7b262605968. 2024-05-27T07:30:36.092 INFO:journalctl@ceph.mds.k.smithi105.stdout:May 27 07:30:35 smithi105 systemd[1]: ceph-73df2636-1bf8-11ef-bc9b-c7b262605968@mds.k.service: Start request repeated too quickly. 2024-05-27T07:30:36.092 INFO:journalctl@ceph.mds.k.smithi105.stdout:May 27 07:30:35 smithi105 systemd[1]: ceph-73df2636-1bf8-11ef-bc9b-c7b262605968@mds.k.service: Failed with result 'exit-code'. 2024-05-27T07:30:36.093 INFO:journalctl@ceph.mds.k.smithi105.stdout:May 27 07:30:35 smithi105 systemd[1]: Failed to start Ceph mds.k for 73df2636-1bf8-11ef-bc9b-c7b262605968. 2024-05-27T07:30:36.353 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:30:36 smithi173 ceph-mon[37000]: pgmap v164: 129 pgs: 129 active+clean; 355 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 1.8 MiB/s wr, 632 op/s 2024-05-27T07:30:36.353 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:30:36 smithi173 ceph-mon[37000]: daemon mds.j restarted 2024-05-27T07:30:36.441 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:30:36 smithi105 ceph-mon[35977]: pgmap v164: 129 pgs: 129 active+clean; 355 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 1.8 MiB/s wr, 632 op/s 2024-05-27T07:30:36.441 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:30:36 smithi105 ceph-mon[35977]: daemon mds.j restarted 2024-05-27T07:30:36.481 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:30:36 smithi052 ceph-mon[31955]: pgmap v164: 129 pgs: 129 active+clean; 355 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 1.8 MiB/s wr, 632 op/s 2024-05-27T07:30:36.481 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:30:36 smithi052 ceph-mon[31955]: daemon mds.j restarted 2024-05-27T07:30:37.353 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:30:37 smithi173 ceph-mon[37000]: osdmap e114: 12 total, 12 up, 12 in 2024-05-27T07:30:37.354 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:30:37 smithi173 ceph-mon[37000]: mds.? [v2:172.21.15.52:6834/2692336430,v1:172.21.15.52:6835/2692336430] up:boot 2024-05-27T07:30:37.354 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:30:37 smithi173 ceph-mon[37000]: MDS daemon mds.k is removed because it is dead or otherwise unavailable. 2024-05-27T07:30:37.354 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:30:37 smithi173 ceph-mon[37000]: fsmap cephfs:5 2 up:standby-replay 1 up:standby 5 up:active 2024-05-27T07:30:37.354 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:30:37 smithi173 ceph-mon[37000]: from='mgr.15333 172.21.15.52:0/416389082' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "j"}]: dispatch 2024-05-27T07:30:37.354 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:30:37 smithi173 ceph-mon[37000]: fsmap cephfs:5 2 up:standby-replay 5 up:active 2024-05-27T07:30:37.441 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:30:37 smithi105 ceph-mon[35977]: osdmap e114: 12 total, 12 up, 12 in 2024-05-27T07:30:37.441 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:30:37 smithi105 ceph-mon[35977]: mds.? [v2:172.21.15.52:6834/2692336430,v1:172.21.15.52:6835/2692336430] up:boot 2024-05-27T07:30:37.442 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:30:37 smithi105 ceph-mon[35977]: MDS daemon mds.k is removed because it is dead or otherwise unavailable. 2024-05-27T07:30:37.442 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:30:37 smithi105 ceph-mon[35977]: fsmap cephfs:5 2 up:standby-replay 1 up:standby 5 up:active 2024-05-27T07:30:37.442 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:30:37 smithi105 ceph-mon[35977]: from='mgr.15333 172.21.15.52:0/416389082' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "j"}]: dispatch 2024-05-27T07:30:37.442 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:30:37 smithi105 ceph-mon[35977]: fsmap cephfs:5 2 up:standby-replay 5 up:active 2024-05-27T07:30:37.488 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:30:37 smithi052 ceph-mon[31955]: osdmap e114: 12 total, 12 up, 12 in 2024-05-27T07:30:37.488 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:30:37 smithi052 ceph-mon[31955]: mds.? [v2:172.21.15.52:6834/2692336430,v1:172.21.15.52:6835/2692336430] up:boot 2024-05-27T07:30:37.488 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:30:37 smithi052 ceph-mon[31955]: MDS daemon mds.k is removed because it is dead or otherwise unavailable. 2024-05-27T07:30:37.488 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:30:37 smithi052 ceph-mon[31955]: fsmap cephfs:5 2 up:standby-replay 1 up:standby 5 up:active 2024-05-27T07:30:37.488 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:30:37 smithi052 ceph-mon[31955]: from='mgr.15333 172.21.15.52:0/416389082' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "j"}]: dispatch 2024-05-27T07:30:37.488 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:30:37 smithi052 ceph-mon[31955]: fsmap cephfs:5 2 up:standby-replay 5 up:active 2024-05-27T07:30:38.404 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:30:38 smithi052 ceph-mon[31955]: pgmap v166: 129 pgs: 129 active+clean; 361 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 2.2 MiB/s wr, 984 op/s 2024-05-27T07:30:38.404 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:30:38 smithi052 ceph-mon[31955]: fsmap cephfs:5 2 up:standby-replay 5 up:active 2024-05-27T07:30:38.441 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:30:38 smithi105 ceph-mon[35977]: pgmap v166: 129 pgs: 129 active+clean; 361 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 2.2 MiB/s wr, 984 op/s 2024-05-27T07:30:38.441 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:30:38 smithi105 ceph-mon[35977]: fsmap cephfs:5 2 up:standby-replay 5 up:active 2024-05-27T07:30:38.603 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:30:38 smithi173 ceph-mon[37000]: pgmap v166: 129 pgs: 129 active+clean; 361 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 2.2 MiB/s wr, 984 op/s 2024-05-27T07:30:38.603 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:30:38 smithi173 ceph-mon[37000]: fsmap cephfs:5 2 up:standby-replay 5 up:active 2024-05-27T07:30:38.729 INFO:journalctl@ceph.mds.j.smithi052.stdout:May 27 07:30:38 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-j[108055]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: In function 'std::set SnapClient::filter(const std::set&) const' thread 7f22e5f04700 time 2024-05-27T07:30:38.413007+0000 2024-05-27T07:30:38.729 INFO:journalctl@ceph.mds.j.smithi052.stdout:May 27 07:30:38 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-j[108055]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: 197: FAILED ceph_assert(cached_version > 0) 2024-05-27T07:30:38.729 INFO:journalctl@ceph.mds.j.smithi052.stdout:May 27 07:30:38 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-j[108055]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:30:38.729 INFO:journalctl@ceph.mds.j.smithi052.stdout:May 27 07:30:38 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-j[108055]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x127) [0x7f22f8749ba6] 2024-05-27T07:30:38.729 INFO:journalctl@ceph.mds.j.smithi052.stdout:May 27 07:30:38 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-j[108055]: 2: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7f22f8749dd1] 2024-05-27T07:30:38.729 INFO:journalctl@ceph.mds.j.smithi052.stdout:May 27 07:30:38 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-j[108055]: 3: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x55b486fa823d] 2024-05-27T07:30:38.729 INFO:journalctl@ceph.mds.j.smithi052.stdout:May 27 07:30:38 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-j[108055]: 4: (SnapRealm::build_snap_set() const+0x323) [0x55b486f93971] 2024-05-27T07:30:38.729 INFO:journalctl@ceph.mds.j.smithi052.stdout:May 27 07:30:38 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-j[108055]: 5: (SnapRealm::check_cache() const+0x13e) [0x55b486f91fa0] 2024-05-27T07:30:38.730 INFO:journalctl@ceph.mds.j.smithi052.stdout:May 27 07:30:38 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-j[108055]: 6: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x55b486f92651] 2024-05-27T07:30:38.730 INFO:journalctl@ceph.mds.j.smithi052.stdout:May 27 07:30:38 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-j[108055]: 7: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x55b486f27347] 2024-05-27T07:30:38.730 INFO:journalctl@ceph.mds.j.smithi052.stdout:May 27 07:30:38 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-j[108055]: 8: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x55b486c2387e] 2024-05-27T07:30:38.730 INFO:journalctl@ceph.mds.j.smithi052.stdout:May 27 07:30:38 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-j[108055]: 9: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x279d) [0x55b486c28f2f] 2024-05-27T07:30:38.730 INFO:journalctl@ceph.mds.j.smithi052.stdout:May 27 07:30:38 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-j[108055]: 10: (EUpdate::replay(MDSRank*)+0x63) [0x55b486c33969] 2024-05-27T07:30:38.730 INFO:journalctl@ceph.mds.j.smithi052.stdout:May 27 07:30:38 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-j[108055]: 11: (MDLog::_replay_thread()+0x1f38) [0x55b486fcc48c] 2024-05-27T07:30:38.730 INFO:journalctl@ceph.mds.j.smithi052.stdout:May 27 07:30:38 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-j[108055]: 12: (MDLog::ReplayThread::entry()+0x11) [0x55b486fd59cb] 2024-05-27T07:30:38.730 INFO:journalctl@ceph.mds.j.smithi052.stdout:May 27 07:30:38 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-j[108055]: 13: (Thread::entry_wrapper()+0x43) [0x7f22f871872d] 2024-05-27T07:30:38.730 INFO:journalctl@ceph.mds.j.smithi052.stdout:May 27 07:30:38 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-j[108055]: 14: (Thread::_entry_func(void*)+0xd) [0x7f22f8718749] 2024-05-27T07:30:38.730 INFO:journalctl@ceph.mds.j.smithi052.stdout:May 27 07:30:38 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-j[108055]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7f22f72bc1ca] 2024-05-27T07:30:38.730 INFO:journalctl@ceph.mds.j.smithi052.stdout:May 27 07:30:38 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-j[108055]: 16: clone() 2024-05-27T07:30:38.730 INFO:journalctl@ceph.mds.j.smithi052.stdout:May 27 07:30:38 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-j[108055]: 2024-05-27T07:30:38.416+0000 7f22e5f04700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: In function 'std::set SnapClient::filter(const std::set&) const' thread 7f22e5f04700 time 2024-05-27T07:30:38.413007+0000 2024-05-27T07:30:38.731 INFO:journalctl@ceph.mds.j.smithi052.stdout:May 27 07:30:38 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-j[108055]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: 197: FAILED ceph_assert(cached_version > 0) 2024-05-27T07:30:38.731 INFO:journalctl@ceph.mds.j.smithi052.stdout:May 27 07:30:38 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-j[108055]: 2024-05-27T07:30:38.731 INFO:journalctl@ceph.mds.j.smithi052.stdout:May 27 07:30:38 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-j[108055]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:30:38.731 INFO:journalctl@ceph.mds.j.smithi052.stdout:May 27 07:30:38 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-j[108055]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x127) [0x7f22f8749ba6] 2024-05-27T07:30:38.731 INFO:journalctl@ceph.mds.j.smithi052.stdout:May 27 07:30:38 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-j[108055]: 2: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7f22f8749dd1] 2024-05-27T07:30:38.731 INFO:journalctl@ceph.mds.j.smithi052.stdout:May 27 07:30:38 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-j[108055]: 3: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x55b486fa823d] 2024-05-27T07:30:38.731 INFO:journalctl@ceph.mds.j.smithi052.stdout:May 27 07:30:38 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-j[108055]: 4: (SnapRealm::build_snap_set() const+0x323) [0x55b486f93971] 2024-05-27T07:30:38.732 INFO:journalctl@ceph.mds.j.smithi052.stdout:May 27 07:30:38 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-j[108055]: 5: (SnapRealm::check_cache() const+0x13e) [0x55b486f91fa0] 2024-05-27T07:30:38.732 INFO:journalctl@ceph.mds.j.smithi052.stdout:May 27 07:30:38 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-j[108055]: 6: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x55b486f92651] 2024-05-27T07:30:38.732 INFO:journalctl@ceph.mds.j.smithi052.stdout:May 27 07:30:38 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-j[108055]: 7: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x55b486f27347] 2024-05-27T07:30:38.732 INFO:journalctl@ceph.mds.j.smithi052.stdout:May 27 07:30:38 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-j[108055]: 8: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x55b486c2387e] 2024-05-27T07:30:38.732 INFO:journalctl@ceph.mds.j.smithi052.stdout:May 27 07:30:38 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-j[108055]: 9: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x279d) [0x55b486c28f2f] 2024-05-27T07:30:38.732 INFO:journalctl@ceph.mds.j.smithi052.stdout:May 27 07:30:38 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-j[108055]: 10: (EUpdate::replay(MDSRank*)+0x63) [0x55b486c33969] 2024-05-27T07:30:38.732 INFO:journalctl@ceph.mds.j.smithi052.stdout:May 27 07:30:38 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-j[108055]: 11: (MDLog::_replay_thread()+0x1f38) [0x55b486fcc48c] 2024-05-27T07:30:38.732 INFO:journalctl@ceph.mds.j.smithi052.stdout:May 27 07:30:38 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-j[108055]: 12: (MDLog::ReplayThread::entry()+0x11) [0x55b486fd59cb] 2024-05-27T07:30:38.732 INFO:journalctl@ceph.mds.j.smithi052.stdout:May 27 07:30:38 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-j[108055]: 13: (Thread::entry_wrapper()+0x43) [0x7f22f871872d] 2024-05-27T07:30:38.732 INFO:journalctl@ceph.mds.j.smithi052.stdout:May 27 07:30:38 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-j[108055]: 14: (Thread::_entry_func(void*)+0xd) [0x7f22f8718749] 2024-05-27T07:30:38.732 INFO:journalctl@ceph.mds.j.smithi052.stdout:May 27 07:30:38 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-j[108055]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7f22f72bc1ca] 2024-05-27T07:30:38.732 INFO:journalctl@ceph.mds.j.smithi052.stdout:May 27 07:30:38 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-j[108055]: 16: clone() 2024-05-27T07:30:38.732 INFO:journalctl@ceph.mds.j.smithi052.stdout:May 27 07:30:38 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-j[108055]: 2024-05-27T07:30:38.733 INFO:journalctl@ceph.mds.j.smithi052.stdout:May 27 07:30:38 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-j[108055]: *** Caught signal (Aborted) ** 2024-05-27T07:30:38.733 INFO:journalctl@ceph.mds.j.smithi052.stdout:May 27 07:30:38 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-j[108055]: in thread 7f22e5f04700 thread_name:md_log_replay 2024-05-27T07:30:38.733 INFO:journalctl@ceph.mds.j.smithi052.stdout:May 27 07:30:38 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-j[108055]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:30:38.733 INFO:journalctl@ceph.mds.j.smithi052.stdout:May 27 07:30:38 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-j[108055]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f22f72c6d20] 2024-05-27T07:30:38.733 INFO:journalctl@ceph.mds.j.smithi052.stdout:May 27 07:30:38 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-j[108055]: 2: gsignal() 2024-05-27T07:30:38.733 INFO:journalctl@ceph.mds.j.smithi052.stdout:May 27 07:30:38 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-j[108055]: 3: abort() 2024-05-27T07:30:38.733 INFO:journalctl@ceph.mds.j.smithi052.stdout:May 27 07:30:38 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-j[108055]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x26e) [0x7f22f8749ced] 2024-05-27T07:30:38.733 INFO:journalctl@ceph.mds.j.smithi052.stdout:May 27 07:30:38 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-j[108055]: 5: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7f22f8749dd1] 2024-05-27T07:30:38.733 INFO:journalctl@ceph.mds.j.smithi052.stdout:May 27 07:30:38 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-j[108055]: 6: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x55b486fa823d] 2024-05-27T07:30:38.733 INFO:journalctl@ceph.mds.j.smithi052.stdout:May 27 07:30:38 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-j[108055]: 7: (SnapRealm::build_snap_set() const+0x323) [0x55b486f93971] 2024-05-27T07:30:38.733 INFO:journalctl@ceph.mds.j.smithi052.stdout:May 27 07:30:38 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-j[108055]: 8: (SnapRealm::check_cache() const+0x13e) [0x55b486f91fa0] 2024-05-27T07:30:38.733 INFO:journalctl@ceph.mds.j.smithi052.stdout:May 27 07:30:38 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-j[108055]: 9: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x55b486f92651] 2024-05-27T07:30:38.733 INFO:journalctl@ceph.mds.j.smithi052.stdout:May 27 07:30:38 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-j[108055]: 10: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x55b486f27347] 2024-05-27T07:30:38.734 INFO:journalctl@ceph.mds.j.smithi052.stdout:May 27 07:30:38 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-j[108055]: 11: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x55b486c2387e] 2024-05-27T07:30:38.734 INFO:journalctl@ceph.mds.j.smithi052.stdout:May 27 07:30:38 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-j[108055]: 12: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x279d) [0x55b486c28f2f] 2024-05-27T07:30:38.734 INFO:journalctl@ceph.mds.j.smithi052.stdout:May 27 07:30:38 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-j[108055]: 13: (EUpdate::replay(MDSRank*)+0x63) [0x55b486c33969] 2024-05-27T07:30:38.734 INFO:journalctl@ceph.mds.j.smithi052.stdout:May 27 07:30:38 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-j[108055]: 14: (MDLog::_replay_thread()+0x1f38) [0x55b486fcc48c] 2024-05-27T07:30:38.734 INFO:journalctl@ceph.mds.j.smithi052.stdout:May 27 07:30:38 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-j[108055]: 15: (MDLog::ReplayThread::entry()+0x11) [0x55b486fd59cb] 2024-05-27T07:30:38.734 INFO:journalctl@ceph.mds.j.smithi052.stdout:May 27 07:30:38 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-j[108055]: 16: (Thread::entry_wrapper()+0x43) [0x7f22f871872d] 2024-05-27T07:30:38.734 INFO:journalctl@ceph.mds.j.smithi052.stdout:May 27 07:30:38 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-j[108055]: 17: (Thread::_entry_func(void*)+0xd) [0x7f22f8718749] 2024-05-27T07:30:38.734 INFO:journalctl@ceph.mds.j.smithi052.stdout:May 27 07:30:38 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-j[108055]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7f22f72bc1ca] 2024-05-27T07:30:38.734 INFO:journalctl@ceph.mds.j.smithi052.stdout:May 27 07:30:38 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-j[108055]: 19: clone() 2024-05-27T07:30:38.734 INFO:journalctl@ceph.mds.j.smithi052.stdout:May 27 07:30:38 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-j[108055]: 2024-05-27T07:30:38.421+0000 7f22e5f04700 -1 *** Caught signal (Aborted) ** 2024-05-27T07:30:38.734 INFO:journalctl@ceph.mds.j.smithi052.stdout:May 27 07:30:38 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-j[108055]: in thread 7f22e5f04700 thread_name:md_log_replay 2024-05-27T07:30:38.734 INFO:journalctl@ceph.mds.j.smithi052.stdout:May 27 07:30:38 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-j[108055]: 2024-05-27T07:30:38.734 INFO:journalctl@ceph.mds.j.smithi052.stdout:May 27 07:30:38 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-j[108055]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:30:38.735 INFO:journalctl@ceph.mds.j.smithi052.stdout:May 27 07:30:38 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-j[108055]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f22f72c6d20] 2024-05-27T07:30:38.735 INFO:journalctl@ceph.mds.j.smithi052.stdout:May 27 07:30:38 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-j[108055]: 2: gsignal() 2024-05-27T07:30:38.735 INFO:journalctl@ceph.mds.j.smithi052.stdout:May 27 07:30:38 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-j[108055]: 3: abort() 2024-05-27T07:30:38.735 INFO:journalctl@ceph.mds.j.smithi052.stdout:May 27 07:30:38 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-j[108055]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x26e) [0x7f22f8749ced] 2024-05-27T07:30:38.735 INFO:journalctl@ceph.mds.j.smithi052.stdout:May 27 07:30:38 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-j[108055]: 5: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7f22f8749dd1] 2024-05-27T07:30:38.735 INFO:journalctl@ceph.mds.j.smithi052.stdout:May 27 07:30:38 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-j[108055]: 6: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x55b486fa823d] 2024-05-27T07:30:38.735 INFO:journalctl@ceph.mds.j.smithi052.stdout:May 27 07:30:38 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-j[108055]: 7: (SnapRealm::build_snap_set() const+0x323) [0x55b486f93971] 2024-05-27T07:30:38.735 INFO:journalctl@ceph.mds.j.smithi052.stdout:May 27 07:30:38 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-j[108055]: 8: (SnapRealm::check_cache() const+0x13e) [0x55b486f91fa0] 2024-05-27T07:30:38.735 INFO:journalctl@ceph.mds.j.smithi052.stdout:May 27 07:30:38 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-j[108055]: 9: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x55b486f92651] 2024-05-27T07:30:38.736 INFO:journalctl@ceph.mds.j.smithi052.stdout:May 27 07:30:38 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-j[108055]: 10: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x55b486f27347] 2024-05-27T07:30:38.736 INFO:journalctl@ceph.mds.j.smithi052.stdout:May 27 07:30:38 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-j[108055]: 11: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x55b486c2387e] 2024-05-27T07:30:38.736 INFO:journalctl@ceph.mds.j.smithi052.stdout:May 27 07:30:38 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-j[108055]: 12: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x279d) [0x55b486c28f2f] 2024-05-27T07:30:38.736 INFO:journalctl@ceph.mds.j.smithi052.stdout:May 27 07:30:38 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-j[108055]: 13: (EUpdate::replay(MDSRank*)+0x63) [0x55b486c33969] 2024-05-27T07:30:38.736 INFO:journalctl@ceph.mds.j.smithi052.stdout:May 27 07:30:38 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-j[108055]: 14: (MDLog::_replay_thread()+0x1f38) [0x55b486fcc48c] 2024-05-27T07:30:38.736 INFO:journalctl@ceph.mds.j.smithi052.stdout:May 27 07:30:38 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-j[108055]: 15: (MDLog::ReplayThread::entry()+0x11) [0x55b486fd59cb] 2024-05-27T07:30:38.736 INFO:journalctl@ceph.mds.j.smithi052.stdout:May 27 07:30:38 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-j[108055]: 16: (Thread::entry_wrapper()+0x43) [0x7f22f871872d] 2024-05-27T07:30:38.736 INFO:journalctl@ceph.mds.j.smithi052.stdout:May 27 07:30:38 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-j[108055]: 17: (Thread::_entry_func(void*)+0xd) [0x7f22f8718749] 2024-05-27T07:30:38.736 INFO:journalctl@ceph.mds.j.smithi052.stdout:May 27 07:30:38 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-j[108055]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7f22f72bc1ca] 2024-05-27T07:30:38.736 INFO:journalctl@ceph.mds.j.smithi052.stdout:May 27 07:30:38 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-j[108055]: 19: clone() 2024-05-27T07:30:38.737 INFO:journalctl@ceph.mds.j.smithi052.stdout:May 27 07:30:38 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-j[108055]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-05-27T07:30:38.737 INFO:journalctl@ceph.mds.j.smithi052.stdout:May 27 07:30:38 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-j[108055]: 2024-05-27T07:30:38.737 INFO:journalctl@ceph.mds.j.smithi052.stdout:May 27 07:30:38 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-j[108055]: -3> 2024-05-27T07:30:38.416+0000 7f22e5f04700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: In function 'std::set SnapClient::filter(const std::set&) const' thread 7f22e5f04700 time 2024-05-27T07:30:38.413007+0000 2024-05-27T07:30:38.737 INFO:journalctl@ceph.mds.j.smithi052.stdout:May 27 07:30:38 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-j[108055]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: 197: FAILED ceph_assert(cached_version > 0) 2024-05-27T07:30:38.737 INFO:journalctl@ceph.mds.j.smithi052.stdout:May 27 07:30:38 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-j[108055]: 2024-05-27T07:30:38.737 INFO:journalctl@ceph.mds.j.smithi052.stdout:May 27 07:30:38 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-j[108055]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:30:38.737 INFO:journalctl@ceph.mds.j.smithi052.stdout:May 27 07:30:38 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-j[108055]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x127) [0x7f22f8749ba6] 2024-05-27T07:30:38.737 INFO:journalctl@ceph.mds.j.smithi052.stdout:May 27 07:30:38 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-j[108055]: 2: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7f22f8749dd1] 2024-05-27T07:30:38.737 INFO:journalctl@ceph.mds.j.smithi052.stdout:May 27 07:30:38 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-j[108055]: 3: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x55b486fa823d] 2024-05-27T07:30:38.737 INFO:journalctl@ceph.mds.j.smithi052.stdout:May 27 07:30:38 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-j[108055]: 4: (SnapRealm::build_snap_set() const+0x323) [0x55b486f93971] 2024-05-27T07:30:38.737 INFO:journalctl@ceph.mds.j.smithi052.stdout:May 27 07:30:38 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-j[108055]: 5: (SnapRealm::check_cache() const+0x13e) [0x55b486f91fa0] 2024-05-27T07:30:38.737 INFO:journalctl@ceph.mds.j.smithi052.stdout:May 27 07:30:38 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-j[108055]: 6: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x55b486f92651] 2024-05-27T07:30:38.737 INFO:journalctl@ceph.mds.j.smithi052.stdout:May 27 07:30:38 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-j[108055]: 7: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x55b486f27347] 2024-05-27T07:30:38.737 INFO:journalctl@ceph.mds.j.smithi052.stdout:May 27 07:30:38 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-j[108055]: 8: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x55b486c2387e] 2024-05-27T07:30:38.738 INFO:journalctl@ceph.mds.j.smithi052.stdout:May 27 07:30:38 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-j[108055]: 9: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x279d) [0x55b486c28f2f] 2024-05-27T07:30:38.738 INFO:journalctl@ceph.mds.j.smithi052.stdout:May 27 07:30:38 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-j[108055]: 10: (EUpdate::replay(MDSRank*)+0x63) [0x55b486c33969] 2024-05-27T07:30:38.738 INFO:journalctl@ceph.mds.j.smithi052.stdout:May 27 07:30:38 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-j[108055]: 11: (MDLog::_replay_thread()+0x1f38) [0x55b486fcc48c] 2024-05-27T07:30:38.738 INFO:journalctl@ceph.mds.j.smithi052.stdout:May 27 07:30:38 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-j[108055]: 12: (MDLog::ReplayThread::entry()+0x11) [0x55b486fd59cb] 2024-05-27T07:30:38.738 INFO:journalctl@ceph.mds.j.smithi052.stdout:May 27 07:30:38 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-j[108055]: 13: (Thread::entry_wrapper()+0x43) [0x7f22f871872d] 2024-05-27T07:30:38.738 INFO:journalctl@ceph.mds.j.smithi052.stdout:May 27 07:30:38 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-j[108055]: 14: (Thread::_entry_func(void*)+0xd) [0x7f22f8718749] 2024-05-27T07:30:38.738 INFO:journalctl@ceph.mds.j.smithi052.stdout:May 27 07:30:38 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-j[108055]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7f22f72bc1ca] 2024-05-27T07:30:38.738 INFO:journalctl@ceph.mds.j.smithi052.stdout:May 27 07:30:38 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-j[108055]: 16: clone() 2024-05-27T07:30:38.738 INFO:journalctl@ceph.mds.j.smithi052.stdout:May 27 07:30:38 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-j[108055]: 2024-05-27T07:30:38.738 INFO:journalctl@ceph.mds.j.smithi052.stdout:May 27 07:30:38 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-j[108055]: 0> 2024-05-27T07:30:38.421+0000 7f22e5f04700 -1 *** Caught signal (Aborted) ** 2024-05-27T07:30:38.738 INFO:journalctl@ceph.mds.j.smithi052.stdout:May 27 07:30:38 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-j[108055]: in thread 7f22e5f04700 thread_name:md_log_replay 2024-05-27T07:30:38.738 INFO:journalctl@ceph.mds.j.smithi052.stdout:May 27 07:30:38 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-j[108055]: 2024-05-27T07:30:38.738 INFO:journalctl@ceph.mds.j.smithi052.stdout:May 27 07:30:38 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-j[108055]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:30:38.739 INFO:journalctl@ceph.mds.j.smithi052.stdout:May 27 07:30:38 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-j[108055]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f22f72c6d20] 2024-05-27T07:30:38.739 INFO:journalctl@ceph.mds.j.smithi052.stdout:May 27 07:30:38 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-j[108055]: 2: gsignal() 2024-05-27T07:30:38.739 INFO:journalctl@ceph.mds.j.smithi052.stdout:May 27 07:30:38 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-j[108055]: 3: abort() 2024-05-27T07:30:38.739 INFO:journalctl@ceph.mds.j.smithi052.stdout:May 27 07:30:38 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-j[108055]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x26e) [0x7f22f8749ced] 2024-05-27T07:30:38.739 INFO:journalctl@ceph.mds.j.smithi052.stdout:May 27 07:30:38 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-j[108055]: 5: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7f22f8749dd1] 2024-05-27T07:30:38.739 INFO:journalctl@ceph.mds.j.smithi052.stdout:May 27 07:30:38 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-j[108055]: 6: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x55b486fa823d] 2024-05-27T07:30:38.739 INFO:journalctl@ceph.mds.j.smithi052.stdout:May 27 07:30:38 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-j[108055]: 7: (SnapRealm::build_snap_set() const+0x323) [0x55b486f93971] 2024-05-27T07:30:38.739 INFO:journalctl@ceph.mds.j.smithi052.stdout:May 27 07:30:38 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-j[108055]: 8: (SnapRealm::check_cache() const+0x13e) [0x55b486f91fa0] 2024-05-27T07:30:38.739 INFO:journalctl@ceph.mds.j.smithi052.stdout:May 27 07:30:38 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-j[108055]: 9: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x55b486f92651] 2024-05-27T07:30:38.739 INFO:journalctl@ceph.mds.j.smithi052.stdout:May 27 07:30:38 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-j[108055]: 10: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x55b486f27347] 2024-05-27T07:30:38.739 INFO:journalctl@ceph.mds.j.smithi052.stdout:May 27 07:30:38 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-j[108055]: 11: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x55b486c2387e] 2024-05-27T07:30:38.739 INFO:journalctl@ceph.mds.j.smithi052.stdout:May 27 07:30:38 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-j[108055]: 12: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x279d) [0x55b486c28f2f] 2024-05-27T07:30:38.739 INFO:journalctl@ceph.mds.j.smithi052.stdout:May 27 07:30:38 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-j[108055]: 13: (EUpdate::replay(MDSRank*)+0x63) [0x55b486c33969] 2024-05-27T07:30:38.739 INFO:journalctl@ceph.mds.j.smithi052.stdout:May 27 07:30:38 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-j[108055]: 14: (MDLog::_replay_thread()+0x1f38) [0x55b486fcc48c] 2024-05-27T07:30:38.739 INFO:journalctl@ceph.mds.j.smithi052.stdout:May 27 07:30:38 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-j[108055]: 15: (MDLog::ReplayThread::entry()+0x11) [0x55b486fd59cb] 2024-05-27T07:30:38.740 INFO:journalctl@ceph.mds.j.smithi052.stdout:May 27 07:30:38 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-j[108055]: 16: (Thread::entry_wrapper()+0x43) [0x7f22f871872d] 2024-05-27T07:30:38.740 INFO:journalctl@ceph.mds.j.smithi052.stdout:May 27 07:30:38 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-j[108055]: 17: (Thread::_entry_func(void*)+0xd) [0x7f22f8718749] 2024-05-27T07:30:38.740 INFO:journalctl@ceph.mds.j.smithi052.stdout:May 27 07:30:38 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-j[108055]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7f22f72bc1ca] 2024-05-27T07:30:38.740 INFO:journalctl@ceph.mds.j.smithi052.stdout:May 27 07:30:38 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-j[108055]: 19: clone() 2024-05-27T07:30:38.740 INFO:journalctl@ceph.mds.j.smithi052.stdout:May 27 07:30:38 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-j[108055]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-05-27T07:30:38.740 INFO:journalctl@ceph.mds.j.smithi052.stdout:May 27 07:30:38 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-j[108055]: 2024-05-27T07:30:38.740 INFO:journalctl@ceph.mds.j.smithi052.stdout:May 27 07:30:38 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-j[108055]: -4> 2024-05-27T07:30:38.416+0000 7f22e5f04700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: In function 'std::set SnapClient::filter(const std::set&) const' thread 7f22e5f04700 time 2024-05-27T07:30:38.413007+0000 2024-05-27T07:30:38.740 INFO:journalctl@ceph.mds.j.smithi052.stdout:May 27 07:30:38 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-j[108055]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: 197: FAILED ceph_assert(cached_version > 0) 2024-05-27T07:30:38.740 INFO:journalctl@ceph.mds.j.smithi052.stdout:May 27 07:30:38 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-j[108055]: 2024-05-27T07:30:38.740 INFO:journalctl@ceph.mds.j.smithi052.stdout:May 27 07:30:38 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-j[108055]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:30:38.740 INFO:journalctl@ceph.mds.j.smithi052.stdout:May 27 07:30:38 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-j[108055]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x127) [0x7f22f8749ba6] 2024-05-27T07:30:38.740 INFO:journalctl@ceph.mds.j.smithi052.stdout:May 27 07:30:38 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-j[108055]: 2: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7f22f8749dd1] 2024-05-27T07:30:38.740 INFO:journalctl@ceph.mds.j.smithi052.stdout:May 27 07:30:38 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-j[108055]: 3: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x55b486fa823d] 2024-05-27T07:30:38.741 INFO:journalctl@ceph.mds.j.smithi052.stdout:May 27 07:30:38 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-j[108055]: 4: (SnapRealm::build_snap_set() const+0x323) [0x55b486f93971] 2024-05-27T07:30:38.741 INFO:journalctl@ceph.mds.j.smithi052.stdout:May 27 07:30:38 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-j[108055]: 5: (SnapRealm::check_cache() const+0x13e) [0x55b486f91fa0] 2024-05-27T07:30:38.741 INFO:journalctl@ceph.mds.j.smithi052.stdout:May 27 07:30:38 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-j[108055]: 6: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x55b486f92651] 2024-05-27T07:30:38.741 INFO:journalctl@ceph.mds.j.smithi052.stdout:May 27 07:30:38 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-j[108055]: 7: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x55b486f27347] 2024-05-27T07:30:38.741 INFO:journalctl@ceph.mds.j.smithi052.stdout:May 27 07:30:38 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-j[108055]: 8: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x55b486c2387e] 2024-05-27T07:30:38.741 INFO:journalctl@ceph.mds.j.smithi052.stdout:May 27 07:30:38 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-j[108055]: 9: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x279d) [0x55b486c28f2f] 2024-05-27T07:30:38.741 INFO:journalctl@ceph.mds.j.smithi052.stdout:May 27 07:30:38 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-j[108055]: 10: (EUpdate::replay(MDSRank*)+0x63) [0x55b486c33969] 2024-05-27T07:30:38.741 INFO:journalctl@ceph.mds.j.smithi052.stdout:May 27 07:30:38 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-j[108055]: 11: (MDLog::_replay_thread()+0x1f38) [0x55b486fcc48c] 2024-05-27T07:30:38.741 INFO:journalctl@ceph.mds.j.smithi052.stdout:May 27 07:30:38 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-j[108055]: 12: (MDLog::ReplayThread::entry()+0x11) [0x55b486fd59cb] 2024-05-27T07:30:38.741 INFO:journalctl@ceph.mds.j.smithi052.stdout:May 27 07:30:38 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-j[108055]: 13: (Thread::entry_wrapper()+0x43) [0x7f22f871872d] 2024-05-27T07:30:38.741 INFO:journalctl@ceph.mds.j.smithi052.stdout:May 27 07:30:38 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-j[108055]: 14: (Thread::_entry_func(void*)+0xd) [0x7f22f8718749] 2024-05-27T07:30:38.741 INFO:journalctl@ceph.mds.j.smithi052.stdout:May 27 07:30:38 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-j[108055]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7f22f72bc1ca] 2024-05-27T07:30:38.741 INFO:journalctl@ceph.mds.j.smithi052.stdout:May 27 07:30:38 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-j[108055]: 16: clone() 2024-05-27T07:30:38.741 INFO:journalctl@ceph.mds.j.smithi052.stdout:May 27 07:30:38 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-j[108055]: 2024-05-27T07:30:38.742 INFO:journalctl@ceph.mds.j.smithi052.stdout:May 27 07:30:38 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-j[108055]: -1> 2024-05-27T07:30:38.421+0000 7f22e5f04700 -1 *** Caught signal (Aborted) ** 2024-05-27T07:30:38.742 INFO:journalctl@ceph.mds.j.smithi052.stdout:May 27 07:30:38 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-j[108055]: in thread 7f22e5f04700 thread_name:md_log_replay 2024-05-27T07:30:38.742 INFO:journalctl@ceph.mds.j.smithi052.stdout:May 27 07:30:38 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-j[108055]: 2024-05-27T07:30:38.742 INFO:journalctl@ceph.mds.j.smithi052.stdout:May 27 07:30:38 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-j[108055]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:30:38.742 INFO:journalctl@ceph.mds.j.smithi052.stdout:May 27 07:30:38 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-j[108055]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f22f72c6d20] 2024-05-27T07:30:38.742 INFO:journalctl@ceph.mds.j.smithi052.stdout:May 27 07:30:38 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-j[108055]: 2: gsignal() 2024-05-27T07:30:38.742 INFO:journalctl@ceph.mds.j.smithi052.stdout:May 27 07:30:38 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-j[108055]: 3: abort() 2024-05-27T07:30:38.742 INFO:journalctl@ceph.mds.j.smithi052.stdout:May 27 07:30:38 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-j[108055]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x26e) [0x7f22f8749ced] 2024-05-27T07:30:38.742 INFO:journalctl@ceph.mds.j.smithi052.stdout:May 27 07:30:38 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-j[108055]: 5: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7f22f8749dd1] 2024-05-27T07:30:38.742 INFO:journalctl@ceph.mds.j.smithi052.stdout:May 27 07:30:38 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-j[108055]: 6: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x55b486fa823d] 2024-05-27T07:30:38.742 INFO:journalctl@ceph.mds.j.smithi052.stdout:May 27 07:30:38 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-j[108055]: 7: (SnapRealm::build_snap_set() const+0x323) [0x55b486f93971] 2024-05-27T07:30:38.742 INFO:journalctl@ceph.mds.j.smithi052.stdout:May 27 07:30:38 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-j[108055]: 8: (SnapRealm::check_cache() const+0x13e) [0x55b486f91fa0] 2024-05-27T07:30:38.742 INFO:journalctl@ceph.mds.j.smithi052.stdout:May 27 07:30:38 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-j[108055]: 9: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x55b486f92651] 2024-05-27T07:30:38.742 INFO:journalctl@ceph.mds.j.smithi052.stdout:May 27 07:30:38 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-j[108055]: 10: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x55b486f27347] 2024-05-27T07:30:38.743 INFO:journalctl@ceph.mds.j.smithi052.stdout:May 27 07:30:38 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-j[108055]: 11: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x55b486c2387e] 2024-05-27T07:30:38.743 INFO:journalctl@ceph.mds.j.smithi052.stdout:May 27 07:30:38 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-j[108055]: 12: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x279d) [0x55b486c28f2f] 2024-05-27T07:30:38.743 INFO:journalctl@ceph.mds.j.smithi052.stdout:May 27 07:30:38 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-j[108055]: 13: (EUpdate::replay(MDSRank*)+0x63) [0x55b486c33969] 2024-05-27T07:30:38.743 INFO:journalctl@ceph.mds.j.smithi052.stdout:May 27 07:30:38 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-j[108055]: 14: (MDLog::_replay_thread()+0x1f38) [0x55b486fcc48c] 2024-05-27T07:30:38.743 INFO:journalctl@ceph.mds.j.smithi052.stdout:May 27 07:30:38 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-j[108055]: 15: (MDLog::ReplayThread::entry()+0x11) [0x55b486fd59cb] 2024-05-27T07:30:38.743 INFO:journalctl@ceph.mds.j.smithi052.stdout:May 27 07:30:38 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-j[108055]: 16: (Thread::entry_wrapper()+0x43) [0x7f22f871872d] 2024-05-27T07:30:38.743 INFO:journalctl@ceph.mds.j.smithi052.stdout:May 27 07:30:38 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-j[108055]: 17: (Thread::_entry_func(void*)+0xd) [0x7f22f8718749] 2024-05-27T07:30:38.743 INFO:journalctl@ceph.mds.j.smithi052.stdout:May 27 07:30:38 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-j[108055]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7f22f72bc1ca] 2024-05-27T07:30:38.743 INFO:journalctl@ceph.mds.j.smithi052.stdout:May 27 07:30:38 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-j[108055]: 19: clone() 2024-05-27T07:30:38.743 INFO:journalctl@ceph.mds.j.smithi052.stdout:May 27 07:30:38 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-j[108055]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-05-27T07:30:38.743 INFO:journalctl@ceph.mds.j.smithi052.stdout:May 27 07:30:38 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-j[108055]: 2024-05-27T07:30:38.743 INFO:journalctl@ceph.mds.j.smithi052.stdout:May 27 07:30:38 smithi052 podman[108090]: 2024-05-27 07:30:38.542727609 +0000 UTC m=+0.071254084 container died a24bb58d90f111c96437ec4e2d4bf19550bfe06ab1d48ffd2e6055a142538066 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-j, org.label-schema.vendor=CentOS, ceph=True, org.label-schema.build-date=20240521, GIT_CLEAN=True, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90) 2024-05-27T07:30:39.981 INFO:journalctl@ceph.mds.j.smithi052.stdout:May 27 07:30:39 smithi052 podman[108090]: 2024-05-27 07:30:39.647686271 +0000 UTC m=+1.176212728 container remove a24bb58d90f111c96437ec4e2d4bf19550bfe06ab1d48ffd2e6055a142538066 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-j, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, ceph=True, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, org.label-schema.build-date=20240521, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, org.label-schema.license=GPLv2, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, GIT_CLEAN=True, maintainer=Guillaume Abrioux ) 2024-05-27T07:30:39.981 INFO:journalctl@ceph.mds.j.smithi052.stdout:May 27 07:30:39 smithi052 systemd[1]: ceph-73df2636-1bf8-11ef-bc9b-c7b262605968@mds.j.service: Main process exited, code=exited, status=134/n/a 2024-05-27T07:30:39.981 INFO:journalctl@ceph.mds.a.smithi052.stdout:May 27 07:30:39 smithi052 systemd[1]: ceph-73df2636-1bf8-11ef-bc9b-c7b262605968@mds.a.service: Scheduled restart job, restart counter is at 5. 2024-05-27T07:30:39.981 INFO:journalctl@ceph.mds.a.smithi052.stdout:May 27 07:30:39 smithi052 systemd[1]: Stopped Ceph mds.a for 73df2636-1bf8-11ef-bc9b-c7b262605968. 2024-05-27T07:30:39.982 INFO:journalctl@ceph.mds.a.smithi052.stdout:May 27 07:30:39 smithi052 systemd[1]: ceph-73df2636-1bf8-11ef-bc9b-c7b262605968@mds.a.service: Consumed 1.432s CPU time. 2024-05-27T07:30:39.982 INFO:journalctl@ceph.mds.a.smithi052.stdout:May 27 07:30:39 smithi052 systemd[1]: ceph-73df2636-1bf8-11ef-bc9b-c7b262605968@mds.a.service: Start request repeated too quickly. 2024-05-27T07:30:39.982 INFO:journalctl@ceph.mds.a.smithi052.stdout:May 27 07:30:39 smithi052 systemd[1]: ceph-73df2636-1bf8-11ef-bc9b-c7b262605968@mds.a.service: Failed with result 'exit-code'. 2024-05-27T07:30:39.982 INFO:journalctl@ceph.mds.a.smithi052.stdout:May 27 07:30:39 smithi052 systemd[1]: Failed to start Ceph mds.a for 73df2636-1bf8-11ef-bc9b-c7b262605968. 2024-05-27T07:30:40.441 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:30:40 smithi105 ceph-mon[35977]: pgmap v167: 129 pgs: 129 active+clean; 383 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 4.7 MiB/s wr, 1.60k op/s 2024-05-27T07:30:40.481 INFO:journalctl@ceph.mds.j.smithi052.stdout:May 27 07:30:40 smithi052 systemd[1]: ceph-73df2636-1bf8-11ef-bc9b-c7b262605968@mds.j.service: Failed with result 'exit-code'. 2024-05-27T07:30:40.481 INFO:journalctl@ceph.mds.j.smithi052.stdout:May 27 07:30:40 smithi052 systemd[1]: ceph-73df2636-1bf8-11ef-bc9b-c7b262605968@mds.j.service: Consumed 1.096s CPU time. 2024-05-27T07:30:40.481 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:30:40 smithi052 ceph-mon[31955]: pgmap v167: 129 pgs: 129 active+clean; 383 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 4.7 MiB/s wr, 1.60k op/s 2024-05-27T07:30:40.603 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:30:40 smithi173 ceph-mon[37000]: pgmap v167: 129 pgs: 129 active+clean; 383 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 4.7 MiB/s wr, 1.60k op/s 2024-05-27T07:30:41.391 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:30:41.399+0000 7fcdff7fe640 1 -- 172.21.15.52:0/1351519904 <== mds.0 v2:172.21.15.105:6838/500348151 1 ==== command_reply(tid 0: 0 ) ==== 8+0+25 (crc 0 0 0) 0x562c89609360 con 0x562c892fa1b0 2024-05-27T07:30:41.392 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:30:41.399+0000 7fcdff7fe640 10 client.26177 handle_command_reply: tid=0 2024-05-27T07:30:41.392 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:30:41.399+0000 7fce2d5f9740 1 client.26177 shutdown 2024-05-27T07:30:41.392 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:30:41.399+0000 7fce2d5f9740 1 -- 172.21.15.52:0/1351519904 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x562c892d12f0 msgr2=0x562c896b2420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:30:41.392 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:30:41.399+0000 7fce2d5f9740 1 --2- 172.21.15.52:0/1351519904 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x562c892d12f0 0x562c896b2420 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7fce100284c0 tx=0x7fce10004160 comp rx=0 tx=0).stop 2024-05-27T07:30:41.392 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:30:41.399+0000 7fce2d5f9740 1 -- 172.21.15.52:0/1351519904 shutdown_connections 2024-05-27T07:30:41.392 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:30:41.399+0000 7fce2d5f9740 1 --2- 172.21.15.52:0/1351519904 >> [v2:172.21.15.105:6838/500348151,v1:172.21.15.105:6839/500348151] conn(0x562c892fa1b0 0x562c89722950 crc :-1 s=READY pgs=108 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:30:41.392 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:30:41.399+0000 7fce2d5f9740 1 --2- 172.21.15.52:0/1351519904 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x562c892d94d0 0x562c896ad2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:30:41.392 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:30:41.399+0000 7fce2d5f9740 1 --2- 172.21.15.52:0/1351519904 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x562c892d12f0 0x562c896b2420 unknown :-1 s=CLOSED pgs=163 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:30:41.392 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:30:41.399+0000 7fce2d5f9740 1 --2- 172.21.15.52:0/1351519904 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x562c892b8d80 0x562c896b1ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:30:41.392 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:30:41.399+0000 7fce2d5f9740 1 -- 172.21.15.52:0/1351519904 >> 172.21.15.52:0/1351519904 conn(0x562c892ee300 msgr2=0x562c8969b160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:30:41.393 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:30:41.399+0000 7fcdff7fe640 0 client.26177 ms_handle_reset on v2:172.21.15.105:6838/500348151 2024-05-27T07:30:41.393 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:30:41.400+0000 7fce2d5f9740 1 -- 172.21.15.52:0/1351519904 shutdown_connections 2024-05-27T07:30:41.393 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:30:41.400+0000 7fce2d5f9740 1 -- 172.21.15.52:0/1351519904 wait complete. 2024-05-27T07:30:41.393 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:30:41.400+0000 7fce2d5f9740 20 client.26177 trim_cache size 0 max 16384 2024-05-27T07:30:41.393 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stdout:{ 2024-05-27T07:30:41.393 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stdout: "return_code": 0 2024-05-27T07:30:41.393 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stdout:} 2024-05-27T07:30:41.394 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:30:41.400+0000 7fce2673f640 1 -- 172.21.15.52:0/3834476284 >> [v2:172.21.15.52:6800/1822318384,v1:172.21.15.52:6801/1822318384] conn(0x7fce080b2950 msgr2=0x7fce080b4e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:30:41.394 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:30:41.400+0000 7fce2673f640 1 --2- 172.21.15.52:0/3834476284 >> [v2:172.21.15.52:6800/1822318384,v1:172.21.15.52:6801/1822318384] conn(0x7fce080b2950 0x7fce080b4e60 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7fce180c80b0 tx=0x7fce180b1040 comp rx=0 tx=0).stop 2024-05-27T07:30:41.394 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:30:41.400+0000 7fce2673f640 1 -- 172.21.15.52:0/3834476284 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fce20109030 msgr2=0x7fce2019c160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:30:41.394 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:30:41.400+0000 7fce2673f640 1 --2- 172.21.15.52:0/3834476284 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fce20109030 0x7fce2019c160 secure :-1 s=READY pgs=644 cs=0 l=1 rev1=1 crypto rx=0x7fce180098d0 tx=0x7fce180317b0 comp rx=0 tx=0).stop 2024-05-27T07:30:41.394 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:30:41.401+0000 7fce2673f640 1 -- 172.21.15.52:0/3834476284 shutdown_connections 2024-05-27T07:30:41.394 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:30:41.401+0000 7fce2673f640 1 --2- 172.21.15.52:0/3834476284 >> [v2:172.21.15.52:6800/1822318384,v1:172.21.15.52:6801/1822318384] conn(0x7fce080b2950 0x7fce080b4e60 unknown :-1 s=CLOSED pgs=138 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:30:41.395 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:30:41.401+0000 7fce2673f640 1 --2- 172.21.15.52:0/3834476284 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fce20109030 0x7fce2019c160 unknown :-1 s=CLOSED pgs=644 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:30:41.395 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:30:41.401+0000 7fce2673f640 1 --2- 172.21.15.52:0/3834476284 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fce20108b90 0x7fce2019bc20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:30:41.395 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:30:41.401+0000 7fce2673f640 1 --2- 172.21.15.52:0/3834476284 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fce201032e0 0x7fce2019b6e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:30:41.395 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:30:41.401+0000 7fce2673f640 1 -- 172.21.15.52:0/3834476284 >> 172.21.15.52:0/3834476284 conn(0x7fce200782d0 msgr2=0x7fce20100e60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:30:41.395 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:30:41.401+0000 7fce2673f640 1 -- 172.21.15.52:0/3834476284 shutdown_connections 2024-05-27T07:30:41.395 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:30:41.401+0000 7fce2673f640 1 -- 172.21.15.52:0/3834476284 wait complete. 2024-05-27T07:30:42.441 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:30:42 smithi105 ceph-mon[35977]: pgmap v168: 129 pgs: 129 active+clean; 417 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 MiB/s rd, 7.6 MiB/s wr, 2.23k op/s 2024-05-27T07:30:42.483 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:30:42 smithi052 ceph-mon[31955]: pgmap v168: 129 pgs: 129 active+clean; 417 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 MiB/s rd, 7.6 MiB/s wr, 2.23k op/s 2024-05-27T07:30:42.603 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:30:42 smithi173 ceph-mon[37000]: pgmap v168: 129 pgs: 129 active+clean; 417 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 MiB/s rd, 7.6 MiB/s wr, 2.23k op/s 2024-05-27T07:30:44.441 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:30:44 smithi105 ceph-mon[35977]: pgmap v169: 129 pgs: 129 active+clean; 421 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 4.4 MiB/s rd, 9.3 MiB/s wr, 2.46k op/s 2024-05-27T07:30:44.481 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:30:44 smithi052 ceph-mon[31955]: pgmap v169: 129 pgs: 129 active+clean; 421 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 4.4 MiB/s rd, 9.3 MiB/s wr, 2.46k op/s 2024-05-27T07:30:44.603 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:30:44 smithi173 ceph-mon[37000]: pgmap v169: 129 pgs: 129 active+clean; 421 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 4.4 MiB/s rd, 9.3 MiB/s wr, 2.46k op/s 2024-05-27T07:30:45.210 INFO:tasks.workunit.client.0.smithi052.stdout: 105 771539 139 552880 156 437943 427 2024-05-27T07:30:45.441 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:30:45 smithi105 ceph-mon[35977]: MDS daemon mds.a is removed because it is dead or otherwise unavailable. 2024-05-27T07:30:45.441 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:30:45 smithi105 ceph-mon[35977]: fsmap cephfs:5 1 up:standby-replay 5 up:active 2024-05-27T07:30:45.481 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:30:45 smithi052 ceph-mon[31955]: MDS daemon mds.a is removed because it is dead or otherwise unavailable. 2024-05-27T07:30:45.481 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:30:45 smithi052 ceph-mon[31955]: fsmap cephfs:5 1 up:standby-replay 5 up:active 2024-05-27T07:30:45.603 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:30:45 smithi173 ceph-mon[37000]: MDS daemon mds.a is removed because it is dead or otherwise unavailable. 2024-05-27T07:30:45.603 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:30:45 smithi173 ceph-mon[37000]: fsmap cephfs:5 1 up:standby-replay 5 up:active 2024-05-27T07:30:46.441 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:30:46 smithi105 ceph-mon[35977]: pgmap v170: 129 pgs: 129 active+clean; 401 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 4.3 MiB/s rd, 9.0 MiB/s wr, 2.36k op/s 2024-05-27T07:30:46.481 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:30:46 smithi052 ceph-mon[31955]: pgmap v170: 129 pgs: 129 active+clean; 401 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 4.3 MiB/s rd, 9.0 MiB/s wr, 2.36k op/s 2024-05-27T07:30:46.504 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:30:46 smithi173 ceph-mon[37000]: pgmap v170: 129 pgs: 129 active+clean; 401 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 4.3 MiB/s rd, 9.0 MiB/s wr, 2.36k op/s 2024-05-27T07:30:47.388 DEBUG:teuthology.orchestra.run.smithi052:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:38f9d900c83c63799fbdbe61acc9a11b0d3554a6 shell --fsid 73df2636-1bf8-11ef-bc9b-c7b262605968 -- ceph tell mds.1:0 scrub status 2024-05-27T07:30:48.441 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:30:48 smithi105 ceph-mon[35977]: pgmap v171: 129 pgs: 129 active+clean; 397 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 3.9 MiB/s rd, 8.4 MiB/s wr, 2.13k op/s 2024-05-27T07:30:48.481 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:30:48 smithi052 ceph-mon[31955]: pgmap v171: 129 pgs: 129 active+clean; 397 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 3.9 MiB/s rd, 8.4 MiB/s wr, 2.13k op/s 2024-05-27T07:30:48.540 INFO:teuthology.orchestra.run.smithi052.stderr:Inferring config /var/lib/ceph/73df2636-1bf8-11ef-bc9b-c7b262605968/mon.a/config 2024-05-27T07:30:48.603 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:30:48 smithi173 ceph-mon[37000]: pgmap v171: 129 pgs: 129 active+clean; 397 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 3.9 MiB/s rd, 8.4 MiB/s wr, 2.13k op/s 2024-05-27T07:30:50.237 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:30:50 smithi052 ceph-mon[31955]: pgmap v172: 129 pgs: 129 active+clean; 397 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 3.6 MiB/s rd, 7.5 MiB/s wr, 1.98k op/s 2024-05-27T07:30:50.441 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:30:50 smithi105 ceph-mon[35977]: pgmap v172: 129 pgs: 129 active+clean; 397 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 3.6 MiB/s rd, 7.5 MiB/s wr, 1.98k op/s 2024-05-27T07:30:50.486 INFO:journalctl@ceph.mds.j.smithi052.stdout:May 27 07:30:50 smithi052 systemd[1]: ceph-73df2636-1bf8-11ef-bc9b-c7b262605968@mds.j.service: Scheduled restart job, restart counter is at 4. 2024-05-27T07:30:50.487 INFO:journalctl@ceph.mds.j.smithi052.stdout:May 27 07:30:50 smithi052 systemd[1]: Stopped Ceph mds.j for 73df2636-1bf8-11ef-bc9b-c7b262605968. 2024-05-27T07:30:50.487 INFO:journalctl@ceph.mds.j.smithi052.stdout:May 27 07:30:50 smithi052 systemd[1]: ceph-73df2636-1bf8-11ef-bc9b-c7b262605968@mds.j.service: Consumed 1.096s CPU time. 2024-05-27T07:30:50.555 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:30:50.561+0000 7fccba2ac700 1 -- 172.21.15.52:0/3774039385 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fccb4110750 msgr2=0x7fccb4110b60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:30:50.555 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:30:50.561+0000 7fccba2ac700 1 --2- 172.21.15.52:0/3774039385 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fccb4110750 0x7fccb4110b60 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7fcca40098f0 tx=0x7fcca4009c00 comp rx=0 tx=0).stop 2024-05-27T07:30:50.557 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:30:50.565+0000 7fccba2ac700 1 -- 172.21.15.52:0/3774039385 shutdown_connections 2024-05-27T07:30:50.557 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:30:50.565+0000 7fccba2ac700 1 --2- 172.21.15.52:0/3774039385 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fccb41110a0 0x7fccb4117800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:30:50.557 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:30:50.565+0000 7fccba2ac700 1 --2- 172.21.15.52:0/3774039385 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fccb410b990 0x7fccb410bda0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:30:50.557 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:30:50.565+0000 7fccba2ac700 1 --2- 172.21.15.52:0/3774039385 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fccb4110750 0x7fccb4110b60 unknown :-1 s=CLOSED pgs=135 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:30:50.558 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:30:50.565+0000 7fccba2ac700 1 -- 172.21.15.52:0/3774039385 >> 172.21.15.52:0/3774039385 conn(0x7fccb4079e60 msgr2=0x7fccb407c2b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:30:50.558 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:30:50.565+0000 7fccba2ac700 1 -- 172.21.15.52:0/3774039385 shutdown_connections 2024-05-27T07:30:50.560 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:30:50.567+0000 7fccba2ac700 1 -- 172.21.15.52:0/3774039385 wait complete. 2024-05-27T07:30:50.560 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:30:50.568+0000 7fccba2ac700 1 Processor -- start 2024-05-27T07:30:50.561 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:30:50.569+0000 7fccba2ac700 1 -- start start 2024-05-27T07:30:50.562 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:30:50.570+0000 7fccba2ac700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fccb410b990 0x7fccb40872d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:30:50.562 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:30:50.570+0000 7fccba2ac700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fccb4110750 0x7fccb4080400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:30:50.562 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:30:50.570+0000 7fccba2ac700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fccb41110a0 0x7fccb4080a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:30:50.564 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:30:50.570+0000 7fccba2ac700 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x7fccb411ab90 con 0x7fccb41110a0 2024-05-27T07:30:50.564 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:30:50.570+0000 7fccba2ac700 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7fccb411aa10 con 0x7fccb410b990 2024-05-27T07:30:50.564 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:30:50.571+0000 7fccba2ac700 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7fccb411ad10 con 0x7fccb4110750 2024-05-27T07:30:50.566 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:30:50.571+0000 7fccb3fff700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fccb410b990 0x7fccb40872d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:30:50.566 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:30:50.571+0000 7fccb3fff700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fccb410b990 0x7fccb40872d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.52:36938/0 (socket says 172.21.15.52:36938) 2024-05-27T07:30:50.566 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:30:50.571+0000 7fccb3fff700 1 -- 172.21.15.52:0/408951316 learned_addr learned my addr 172.21.15.52:0/408951316 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-05-27T07:30:50.566 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:30:50.571+0000 7fccb8849700 1 --2- 172.21.15.52:0/408951316 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fccb41110a0 0x7fccb4080a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:30:50.566 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:30:50.572+0000 7fccb37fe700 1 --2- 172.21.15.52:0/408951316 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fccb4110750 0x7fccb4080400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:30:50.566 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:30:50.572+0000 7fccb3fff700 1 -- 172.21.15.52:0/408951316 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fccb4110750 msgr2=0x7fccb4080400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:30:50.566 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:30:50.572+0000 7fccb3fff700 1 --2- 172.21.15.52:0/408951316 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fccb4110750 0x7fccb4080400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:30:50.566 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:30:50.572+0000 7fccb3fff700 1 -- 172.21.15.52:0/408951316 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fccb41110a0 msgr2=0x7fccb4080a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:30:50.566 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:30:50.572+0000 7fccb3fff700 1 --2- 172.21.15.52:0/408951316 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fccb41110a0 0x7fccb4080a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:30:50.566 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:30:50.572+0000 7fccb3fff700 1 -- 172.21.15.52:0/408951316 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fccb4084e00 con 0x7fccb410b990 2024-05-27T07:30:50.567 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:30:50.572+0000 7fccb3fff700 1 --2- 172.21.15.52:0/408951316 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fccb410b990 0x7fccb40872d0 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7fccac00c0c0 tx=0x7fccac00c3d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:30:50.567 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:30:50.573+0000 7fccb17fa700 1 -- 172.21.15.52:0/408951316 <== mon.2 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fccac00cd10 con 0x7fccb410b990 2024-05-27T07:30:50.567 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:30:50.573+0000 7fccb17fa700 1 -- 172.21.15.52:0/408951316 <== mon.2 v2:172.21.15.105:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fccac0075c0 con 0x7fccb410b990 2024-05-27T07:30:50.567 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:30:50.573+0000 7fccba2ac700 1 -- 172.21.15.52:0/408951316 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fccb4085030 con 0x7fccb410b990 2024-05-27T07:30:50.567 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:30:50.573+0000 7fccba2ac700 1 -- 172.21.15.52:0/408951316 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fccb4085480 con 0x7fccb410b990 2024-05-27T07:30:50.578 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:30:50.580+0000 7fccb17fa700 1 -- 172.21.15.52:0/408951316 <== mon.2 v2:172.21.15.105:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fccac0146c0 con 0x7fccb410b990 2024-05-27T07:30:50.578 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:30:50.586+0000 7fccb17fa700 1 -- 172.21.15.52:0/408951316 <== mon.2 v2:172.21.15.105:3300/0 4 ==== mgrmap(e 26) ==== 149783+0+0 (secure 0 0 0) 0x7fccac01b020 con 0x7fccb410b990 2024-05-27T07:30:50.578 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:30:50.586+0000 7fccc8cc1b80 1 Processor -- start 2024-05-27T07:30:50.578 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:30:50.586+0000 7fccc8cc1b80 1 -- start start 2024-05-27T07:30:50.596 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:30:50.588+0000 7fccc8cc1b80 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5640b9120e40 0x5640b92a4710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:30:50.596 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:30:50.588+0000 7fccc8cc1b80 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x5640b92a4c50 0x5640b929ea40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:30:50.596 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:30:50.588+0000 7fccc8cc1b80 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5640b929ef80 0x5640b929f390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:30:50.596 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:30:50.589+0000 7fccb8849700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5640b9120e40 0x5640b92a4710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:30:50.596 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:30:50.589+0000 7fccb8849700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5640b9120e40 0x5640b92a4710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.52:36948/0 (socket says 172.21.15.52:36948) 2024-05-27T07:30:50.596 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:30:50.589+0000 7fccc8cc1b80 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x5640b91cdc90 con 0x5640b92a4c50 2024-05-27T07:30:50.596 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:30:50.589+0000 7fccc8cc1b80 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x5640b91bd830 con 0x5640b9120e40 2024-05-27T07:30:50.596 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:30:50.589+0000 7fccc8cc1b80 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x5640b91c1870 con 0x5640b929ef80 2024-05-27T07:30:50.596 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:30:50.589+0000 7fccb8849700 1 -- 172.21.15.52:0/2868201427 learned_addr learned my addr 172.21.15.52:0/2868201427 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-05-27T07:30:50.597 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:30:50.589+0000 7fccb3fff700 1 --2- 172.21.15.52:0/2868201427 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x5640b92a4c50 0x5640b929ea40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:30:50.597 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:30:50.589+0000 7fccb37fe700 1 --2- 172.21.15.52:0/2868201427 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5640b929ef80 0x5640b929f390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:30:50.597 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:30:50.590+0000 7fccb8849700 1 -- 172.21.15.52:0/2868201427 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5640b929ef80 msgr2=0x5640b929f390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:30:50.597 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:30:50.590+0000 7fccb8849700 1 --2- 172.21.15.52:0/2868201427 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5640b929ef80 0x5640b929f390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:30:50.597 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:30:50.590+0000 7fccb8849700 1 -- 172.21.15.52:0/2868201427 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x5640b92a4c50 msgr2=0x5640b929ea40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:30:50.597 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:30:50.590+0000 7fccb8849700 1 --2- 172.21.15.52:0/2868201427 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x5640b92a4c50 0x5640b929ea40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:30:50.597 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:30:50.590+0000 7fccb8849700 1 -- 172.21.15.52:0/2868201427 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5640b910d8d0 con 0x5640b9120e40 2024-05-27T07:30:50.597 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:30:50.592+0000 7fccb37fe700 1 --2- 172.21.15.52:0/2868201427 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5640b929ef80 0x5640b929f390 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-27T07:30:50.597 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:30:50.592+0000 7fccb8849700 1 --2- 172.21.15.52:0/2868201427 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5640b9120e40 0x5640b92a4710 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7fcca8009a60 tx=0x7fcca8009d70 comp rx=0 tx=0).ready entity=mon.2 client_cookie=6cc990a2974a384 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:30:50.597 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:30:50.594+0000 7fccb17fa700 1 --2- 172.21.15.52:0/408951316 >> [v2:172.21.15.52:6800/1822318384,v1:172.21.15.52:6801/1822318384] conn(0x7fcc9c0b2b80 0x7fcc9c0b5090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:30:50.597 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:30:50.594+0000 7fccb17fa700 1 -- 172.21.15.52:0/408951316 <== mon.2 v2:172.21.15.105:3300/0 5 ==== osd_map(114..114 src has 1..114) ==== 9379+0+0 (secure 0 0 0) 0x7fccac0e3670 con 0x7fccb410b990 2024-05-27T07:30:50.597 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:30:50.595+0000 7fcc9a7fc700 1 -- 172.21.15.52:0/2868201427 <== mon.2 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcca8017070 con 0x5640b9120e40 2024-05-27T07:30:50.597 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:30:50.595+0000 7fcc9a7fc700 1 -- 172.21.15.52:0/2868201427 <== mon.2 v2:172.21.15.105:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fcca800ed80 con 0x5640b9120e40 2024-05-27T07:30:50.598 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:30:50.596+0000 7fccc8cc1b80 1 -- 172.21.15.52:0/2868201427 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5640b9120e40 msgr2=0x5640b92a4710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:30:50.598 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:30:50.596+0000 7fccc8cc1b80 1 --2- 172.21.15.52:0/2868201427 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5640b9120e40 0x5640b92a4710 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7fcca8009a60 tx=0x7fcca8009d70 comp rx=0 tx=0).stop 2024-05-27T07:30:50.598 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:30:50.597+0000 7fccc8cc1b80 1 -- 172.21.15.52:0/2868201427 shutdown_connections 2024-05-27T07:30:50.598 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:30:50.597+0000 7fccc8cc1b80 1 --2- 172.21.15.52:0/2868201427 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5640b929ef80 0x5640b929f390 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:30:50.598 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:30:50.597+0000 7fccc8cc1b80 1 --2- 172.21.15.52:0/2868201427 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5640b9120e40 0x5640b92a4710 unknown :-1 s=CLOSED pgs=165 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:30:50.598 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:30:50.597+0000 7fccc8cc1b80 1 --2- 172.21.15.52:0/2868201427 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x5640b92a4c50 0x5640b929ea40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:30:50.598 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:30:50.597+0000 7fccc8cc1b80 1 -- 172.21.15.52:0/2868201427 >> 172.21.15.52:0/2868201427 conn(0x5640b917baf0 msgr2=0x5640b9299640 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:30:50.598 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:30:50.601+0000 7fccb3fff700 1 --2- 172.21.15.52:0/408951316 >> [v2:172.21.15.52:6800/1822318384,v1:172.21.15.52:6801/1822318384] conn(0x7fcc9c0b2b80 0x7fcc9c0b5090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:30:50.598 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:30:50.602+0000 7fccb3fff700 1 --2- 172.21.15.52:0/408951316 >> [v2:172.21.15.52:6800/1822318384,v1:172.21.15.52:6801/1822318384] conn(0x7fcc9c0b2b80 0x7fcc9c0b5090 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7fccac00efd0 tx=0x7fccac013000 comp rx=0 tx=0).ready entity=mgr.15333 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:30:50.599 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:30:50.606+0000 7fccc8cc1b80 1 -- 172.21.15.52:0/2868201427 shutdown_connections 2024-05-27T07:30:50.602 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:30:50.607+0000 7fccc8cc1b80 1 -- 172.21.15.52:0/2868201427 wait complete. 2024-05-27T07:30:50.602 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:30:50.607+0000 7fccc8cc1b80 1 -- start start 2024-05-27T07:30:50.603 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:30:50 smithi173 ceph-mon[37000]: pgmap v172: 129 pgs: 129 active+clean; 397 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 3.6 MiB/s rd, 7.5 MiB/s wr, 1.98k op/s 2024-05-27T07:30:50.604 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:30:50.612+0000 7fccc8cc1b80 1 Processor -- start 2024-05-27T07:30:50.605 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:30:50.613+0000 7fccc8cc1b80 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5640b9120e40 0x5640b92a82e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:30:50.605 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:30:50.614+0000 7fccc8cc1b80 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5640b929ef80 0x5640b92a8820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:30:50.606 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:30:50.614+0000 7fccc8cc1b80 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x5640b92a4c50 0x5640b92a8d60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:30:50.606 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:30:50.614+0000 7fccc8cc1b80 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x5640b90c3fa0 con 0x5640b92a4c50 2024-05-27T07:30:50.606 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:30:50.614+0000 7fccc8cc1b80 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x5640b915f7c0 con 0x5640b9120e40 2024-05-27T07:30:50.606 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:30:50.614+0000 7fccc8cc1b80 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x5640b92ab790 con 0x5640b929ef80 2024-05-27T07:30:50.606 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:30:50.614+0000 7fccc8cc1b80 10 client.0 _ll_register_callbacks cb 0x5640b9158920 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T07:30:50.608 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:30:50.616+0000 7fccb37fe700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5640b929ef80 0x5640b92a8820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:30:50.608 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:30:50.616+0000 7fccb8849700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5640b9120e40 0x5640b92a82e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:30:50.608 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:30:50.616+0000 7fccb8849700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5640b9120e40 0x5640b92a82e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.52:36952/0 (socket says 172.21.15.52:36952) 2024-05-27T07:30:50.608 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:30:50.616+0000 7fccb8849700 1 -- 172.21.15.52:0/2221591682 learned_addr learned my addr 172.21.15.52:0/2221591682 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-05-27T07:30:50.608 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:30:50.616+0000 7fccb8849700 1 --2- 172.21.15.52:0/2221591682 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x5640b92a4c50 0x5640b92a8d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:30:50.609 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:30:50.617+0000 7fccb8849700 1 -- 172.21.15.52:0/2221591682 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5640b929ef80 msgr2=0x5640b92a8820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:30:50.609 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:30:50.617+0000 7fccb8849700 1 --2- 172.21.15.52:0/2221591682 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5640b929ef80 0x5640b92a8820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:30:50.609 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:30:50.617+0000 7fccb8849700 1 -- 172.21.15.52:0/2221591682 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x5640b92a4c50 msgr2=0x5640b92a8d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:30:50.609 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:30:50.617+0000 7fccb8849700 1 --2- 172.21.15.52:0/2221591682 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x5640b92a4c50 0x5640b92a8d60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:30:50.609 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:30:50.617+0000 7fccb8849700 1 -- 172.21.15.52:0/2221591682 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5640b915c690 con 0x5640b9120e40 2024-05-27T07:30:50.610 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:30:50.618+0000 7fccb8849700 1 --2- 172.21.15.52:0/2221591682 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5640b9120e40 0x5640b92a82e0 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7fcca8013a10 tx=0x7fcca8013b90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=47e69fe24c7c2dad server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:30:50.610 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:30:50.618+0000 7fcc937fe700 10 client.0 ms_handle_connect on v2:172.21.15.105:3300/0 2024-05-27T07:30:50.610 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:30:50.618+0000 7fcc937fe700 1 -- 172.21.15.52:0/2221591682 <== mon.2 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcca8017040 con 0x5640b9120e40 2024-05-27T07:30:50.610 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:30:50.618+0000 7fccc8cc1b80 1 -- 172.21.15.52:0/2221591682 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fcca800ed80 con 0x5640b9120e40 2024-05-27T07:30:50.610 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:30:50.619+0000 7fcc937fe700 1 -- 172.21.15.52:0/2221591682 <== mon.2 v2:172.21.15.105:3300/0 2 ==== osd_map(114..114 src has 1..114) ==== 9379+0+0 (secure 0 0 0) 0x7fcca8004040 con 0x5640b9120e40 2024-05-27T07:30:50.611 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:30:50.619+0000 7fcc937fe700 1 -- 172.21.15.52:0/2221591682 <== mon.2 v2:172.21.15.105:3300/0 3 ==== mon_get_version_reply(handle=1 version=93) ==== 24+0+0 (secure 0 0 0) 0x7fcca8032050 con 0x5640b9120e40 2024-05-27T07:30:50.611 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:30:50.619+0000 7fccc8cc1b80 10 client.26189 fetch_fsmap learned FSMap version 93 2024-05-27T07:30:50.611 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:30:50.619+0000 7fccc8cc1b80 1 -- 172.21.15.52:0/2221591682 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,fsmap=93,monmap=4+}) -- 0x5640b92b6240 con 0x5640b9120e40 2024-05-27T07:30:50.612 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:30:50.620+0000 7fcc937fe700 1 -- 172.21.15.52:0/2221591682 <== mon.2 v2:172.21.15.105:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fcca800ed80 con 0x5640b9120e40 2024-05-27T07:30:50.612 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:30:50.620+0000 7fcc937fe700 1 -- 172.21.15.52:0/2221591682 <== mon.2 v2:172.21.15.105:3300/0 5 ==== fsmap(e 93) ==== 4792+0+0 (secure 0 0 0) 0x7fcca8012630 con 0x5640b9120e40 2024-05-27T07:30:50.613 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:30:50.620+0000 7fccc8cc1b80 10 client.26189 fetch_fsmap finished waiting for FSMap version 93 2024-05-27T07:30:50.613 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:30:50.620+0000 7fccc8cc1b80 10 client.26189 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-05-27T07:30:50.613 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:30:50.620+0000 7fccc8cc1b80 20 client.26189 populate_metadata read hostname 'smithi052' 2024-05-27T07:30:50.613 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:30:50.620+0000 7fccc8cc1b80 1 --2- 172.21.15.52:0/2221591682 >> [v2:172.21.15.105:6838/500348151,v1:172.21.15.105:6839/500348151] conn(0x5640b92b8b20 0x5640b92bafb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:30:50.613 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:30:50.620+0000 7fccc8cc1b80 4 client.26189 mds_command: new command op to 24527 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-27T07:30:50.613 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:30:50.620+0000 7fccc8cc1b80 1 -- 172.21.15.52:0/2221591682 --> [v2:172.21.15.105:6838/500348151,v1:172.21.15.105:6839/500348151] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5640b926a9f0 con 0x5640b92b8b20 2024-05-27T07:30:50.616 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:30:50.625+0000 7fccb37fe700 1 --2- 172.21.15.52:0/2221591682 >> [v2:172.21.15.105:6838/500348151,v1:172.21.15.105:6839/500348151] conn(0x5640b92b8b20 0x5640b92bafb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:30:50.627 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:30:50.636+0000 7fccb37fe700 1 --2- 172.21.15.52:0/2221591682 >> [v2:172.21.15.105:6838/500348151,v1:172.21.15.105:6839/500348151] conn(0x5640b92b8b20 0x5640b92bafb0 crc :-1 s=READY pgs=109 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=f4145b0d787e06e7 server_cookie=e9dae2681b246c13 in_seq=0 out_seq=0 2024-05-27T07:30:50.628 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:30:50.636+0000 7fcc937fe700 10 client.26189 ms_handle_connect on v2:172.21.15.105:6838/500348151 2024-05-27T07:30:50.633 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:30:50.641+0000 7fcc937fe700 1 -- 172.21.15.52:0/2221591682 <== mds.0 v2:172.21.15.105:6838/500348151 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24041 (crc 0 0 0) 0x5640b926a9f0 con 0x5640b92b8b20 2024-05-27T07:30:50.633 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:30:50.641+0000 7fcc937fe700 10 client.26189 handle_command_reply: tid=0 2024-05-27T07:30:50.633 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:30:50.641+0000 7fccc8cc1b80 1 client.26189 shutdown 2024-05-27T07:30:50.634 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:30:50.642+0000 7fccc8cc1b80 1 -- 172.21.15.52:0/2221591682 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5640b9120e40 msgr2=0x5640b92a82e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:30:50.634 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:30:50.642+0000 7fccc8cc1b80 1 --2- 172.21.15.52:0/2221591682 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5640b9120e40 0x5640b92a82e0 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7fcca8013a10 tx=0x7fcca8013b90 comp rx=0 tx=0).stop 2024-05-27T07:30:50.634 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:30:50.642+0000 7fccc8cc1b80 1 -- 172.21.15.52:0/2221591682 shutdown_connections 2024-05-27T07:30:50.635 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:30:50.642+0000 7fccc8cc1b80 1 --2- 172.21.15.52:0/2221591682 >> [v2:172.21.15.105:6838/500348151,v1:172.21.15.105:6839/500348151] conn(0x5640b92b8b20 0x5640b92bafb0 crc :-1 s=READY pgs=109 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:30:50.635 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:30:50.642+0000 7fccc8cc1b80 1 --2- 172.21.15.52:0/2221591682 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x5640b92a4c50 0x5640b92a8d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:30:50.635 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:30:50.642+0000 7fccc8cc1b80 1 --2- 172.21.15.52:0/2221591682 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5640b9120e40 0x5640b92a82e0 unknown :-1 s=CLOSED pgs=166 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:30:50.635 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:30:50.642+0000 7fccc8cc1b80 1 --2- 172.21.15.52:0/2221591682 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5640b929ef80 0x5640b92a8820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:30:50.635 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:30:50.642+0000 7fccc8cc1b80 1 -- 172.21.15.52:0/2221591682 >> 172.21.15.52:0/2221591682 conn(0x5640b917baf0 msgr2=0x5640b929da30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:30:50.635 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:30:50.642+0000 7fcc937fe700 0 client.26189 ms_handle_reset on v2:172.21.15.105:6838/500348151 2024-05-27T07:30:50.637 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:30:50.646+0000 7fccc8cc1b80 1 -- 172.21.15.52:0/2221591682 shutdown_connections 2024-05-27T07:30:50.638 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:30:50.647+0000 7fccc8cc1b80 1 -- 172.21.15.52:0/2221591682 wait complete. 2024-05-27T07:30:50.638 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:30:50.647+0000 7fccc8cc1b80 20 client.26189 trim_cache size 0 max 16384 2024-05-27T07:30:50.715 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:30:50.714+0000 7fccc8cc1b80 1 Processor -- start 2024-05-27T07:30:50.715 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:30:50.715+0000 7fccc8cc1b80 1 -- start start 2024-05-27T07:30:50.715 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:30:50.715+0000 7fccc8cc1b80 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x5640b9120e40 0x5640b92a2720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:30:50.716 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:30:50.716+0000 7fccc8cc1b80 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5640b929ef80 0x5640b92b48e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:30:50.716 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:30:50.716+0000 7fccc8cc1b80 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5640b92a4c50 0x5640b92aa850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:30:50.716 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:30:50.716+0000 7fccb37fe700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5640b929ef80 0x5640b92b48e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:30:50.716 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:30:50.716+0000 7fccb37fe700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5640b929ef80 0x5640b92b48e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.52:37310/0 (socket says 172.21.15.52:37310) 2024-05-27T07:30:50.716 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:30:50.716+0000 7fccb8849700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x5640b9120e40 0x5640b92a2720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:30:50.716 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:30:50.716+0000 7fccb8849700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x5640b9120e40 0x5640b92a2720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.52:35388/0 (socket says 172.21.15.52:35388) 2024-05-27T07:30:50.716 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:30:50.717+0000 7fccb37fe700 1 -- 172.21.15.52:0/3175468261 learned_addr learned my addr 172.21.15.52:0/3175468261 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-05-27T07:30:50.716 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:30:50.717+0000 7fccc8cc1b80 1 -- 172.21.15.52:0/3175468261 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x7fcca000f9c0 con 0x5640b9120e40 2024-05-27T07:30:50.716 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:30:50.717+0000 7fccc8cc1b80 1 -- 172.21.15.52:0/3175468261 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7fcca000d720 con 0x5640b929ef80 2024-05-27T07:30:50.716 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:30:50.717+0000 7fccc8cc1b80 1 -- 172.21.15.52:0/3175468261 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7fcca000f2f0 con 0x5640b92a4c50 2024-05-27T07:30:50.717 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:30:50.717+0000 7fccb8849700 1 --2- 172.21.15.52:0/3175468261 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5640b92a4c50 0x5640b92aa850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:30:50.717 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:30:50.718+0000 7fccb37fe700 1 -- 172.21.15.52:0/3175468261 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5640b92a4c50 msgr2=0x5640b92aa850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:30:50.717 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:30:50.718+0000 7fccb37fe700 1 --2- 172.21.15.52:0/3175468261 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5640b92a4c50 0x5640b92aa850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:30:50.717 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:30:50.718+0000 7fccb37fe700 1 -- 172.21.15.52:0/3175468261 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x5640b9120e40 msgr2=0x5640b92a2720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:30:50.717 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:30:50.718+0000 7fccb37fe700 1 --2- 172.21.15.52:0/3175468261 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x5640b9120e40 0x5640b92a2720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:30:50.717 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:30:50.718+0000 7fccb37fe700 1 -- 172.21.15.52:0/3175468261 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5640b929f9f0 con 0x5640b929ef80 2024-05-27T07:30:50.717 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:30:50.718+0000 7fccb37fe700 1 --2- 172.21.15.52:0/3175468261 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5640b929ef80 0x5640b92b48e0 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7fcca40092a0 tx=0x7fcca40058e0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=23177483a5679fce server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:30:50.717 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:30:50.720+0000 7fcc99ffb700 1 -- 172.21.15.52:0/3175468261 <== mon.2 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcca400f720 con 0x5640b929ef80 2024-05-27T07:30:50.717 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:30:50.720+0000 7fcc99ffb700 1 -- 172.21.15.52:0/3175468261 <== mon.2 v2:172.21.15.105:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fcca400f8c0 con 0x5640b929ef80 2024-05-27T07:30:50.717 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:30:50.720+0000 7fcc99ffb700 1 -- 172.21.15.52:0/3175468261 <== mon.2 v2:172.21.15.105:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcca4004c50 con 0x5640b929ef80 2024-05-27T07:30:50.717 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:30:50.721+0000 7fccc8cc1b80 1 -- 172.21.15.52:0/3175468261 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5640b929ef80 msgr2=0x5640b92b48e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:30:50.717 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:30:50.721+0000 7fccc8cc1b80 1 --2- 172.21.15.52:0/3175468261 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5640b929ef80 0x5640b92b48e0 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7fcca40092a0 tx=0x7fcca40058e0 comp rx=0 tx=0).stop 2024-05-27T07:30:50.718 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:30:50.722+0000 7fccc8cc1b80 1 -- 172.21.15.52:0/3175468261 shutdown_connections 2024-05-27T07:30:50.718 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:30:50.722+0000 7fccc8cc1b80 1 --2- 172.21.15.52:0/3175468261 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5640b92a4c50 0x5640b92aa850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:30:50.718 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:30:50.722+0000 7fccc8cc1b80 1 --2- 172.21.15.52:0/3175468261 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x5640b9120e40 0x5640b92a2720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:30:50.718 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:30:50.722+0000 7fccc8cc1b80 1 --2- 172.21.15.52:0/3175468261 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5640b929ef80 0x5640b92b48e0 unknown :-1 s=CLOSED pgs=167 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:30:50.718 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:30:50.722+0000 7fccc8cc1b80 1 -- 172.21.15.52:0/3175468261 >> 172.21.15.52:0/3175468261 conn(0x5640b917baf0 msgr2=0x5640b92a2010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:30:50.718 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:30:50.725+0000 7fccc8cc1b80 1 -- 172.21.15.52:0/3175468261 shutdown_connections 2024-05-27T07:30:50.725 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:30:50.729+0000 7fccc8cc1b80 1 -- 172.21.15.52:0/3175468261 wait complete. 2024-05-27T07:30:50.725 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:30:50.730+0000 7fccc8cc1b80 1 -- start start 2024-05-27T07:30:50.725 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:30:50.730+0000 7fccc8cc1b80 1 Processor -- start 2024-05-27T07:30:50.725 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:30:50.731+0000 7fccc8cc1b80 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5640b9120e40 0x5640b92b0d40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:30:50.726 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:30:50.732+0000 7fccc8cc1b80 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x5640b929ef80 0x5640b92b1280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:30:50.726 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:30:50.732+0000 7fccc8cc1b80 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5640b92a4c50 0x5640b92bfe00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:30:50.726 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:30:50.732+0000 7fccb8849700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5640b9120e40 0x5640b92b0d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:30:50.726 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:30:50.732+0000 7fccb8849700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5640b9120e40 0x5640b92b0d40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.52:37312/0 (socket says 172.21.15.52:37312) 2024-05-27T07:30:50.726 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:30:50.732+0000 7fccb8849700 1 -- 172.21.15.52:0/2238870944 learned_addr learned my addr 172.21.15.52:0/2238870944 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-05-27T07:30:50.726 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:30:50.733+0000 7fccb37fe700 1 --2- 172.21.15.52:0/2238870944 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x5640b929ef80 0x5640b92b1280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:30:50.726 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:30:50.733+0000 7fccc8cc1b80 1 -- 172.21.15.52:0/2238870944 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x7fcca000e4c0 con 0x5640b929ef80 2024-05-27T07:30:50.726 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:30:50.733+0000 7fccc8cc1b80 1 -- 172.21.15.52:0/2238870944 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7fcca000ddf0 con 0x5640b9120e40 2024-05-27T07:30:50.726 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:30:50.733+0000 7fccc8cc1b80 1 -- 172.21.15.52:0/2238870944 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7fcca000eb90 con 0x5640b92a4c50 2024-05-27T07:30:50.727 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:30:50.733+0000 7fccc8cc1b80 10 client.0 _ll_register_callbacks cb 0x5640b9158920 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T07:30:50.727 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:30:50.733+0000 7fccb8849700 1 -- 172.21.15.52:0/2238870944 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5640b92a4c50 msgr2=0x5640b92bfe00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:30:50.727 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:30:50.733+0000 7fccb8849700 1 --2- 172.21.15.52:0/2238870944 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5640b92a4c50 0x5640b92bfe00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:30:50.727 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:30:50.733+0000 7fccb8849700 1 -- 172.21.15.52:0/2238870944 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x5640b929ef80 msgr2=0x5640b92b1280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:30:50.727 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:30:50.733+0000 7fccb8849700 1 --2- 172.21.15.52:0/2238870944 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x5640b929ef80 0x5640b92b1280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:30:50.727 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:30:50.733+0000 7fccb8849700 1 -- 172.21.15.52:0/2238870944 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5640b92c05a0 con 0x5640b9120e40 2024-05-27T07:30:50.727 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:30:50.734+0000 7fccb8849700 1 --2- 172.21.15.52:0/2238870944 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5640b9120e40 0x5640b92b0d40 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7fcca8013c00 tx=0x7fcca8000bc0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=cf6b5ec7e68d2310 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:30:50.727 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:30:50.734+0000 7fcc937fe700 10 client.0 ms_handle_connect on v2:172.21.15.105:3300/0 2024-05-27T07:30:50.728 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:30:50.736+0000 7fcc937fe700 1 -- 172.21.15.52:0/2238870944 <== mon.2 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcca8017040 con 0x5640b9120e40 2024-05-27T07:30:50.728 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:30:50.736+0000 7fccc8cc1b80 1 -- 172.21.15.52:0/2238870944 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fcca400f8c0 con 0x5640b9120e40 2024-05-27T07:30:50.728 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:30:50.736+0000 7fcc937fe700 1 -- 172.21.15.52:0/2238870944 <== mon.2 v2:172.21.15.105:3300/0 2 ==== osd_map(114..114 src has 1..114) ==== 9379+0+0 (secure 0 0 0) 0x7fcca802f040 con 0x5640b9120e40 2024-05-27T07:30:50.729 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:30:50.737+0000 7fcc937fe700 1 -- 172.21.15.52:0/2238870944 <== mon.2 v2:172.21.15.105:3300/0 3 ==== mon_get_version_reply(handle=1 version=93) ==== 24+0+0 (secure 0 0 0) 0x7fcca8012780 con 0x5640b9120e40 2024-05-27T07:30:50.733 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:30:50.741+0000 7fccc8cc1b80 10 client.26195 fetch_fsmap learned FSMap version 93 2024-05-27T07:30:50.734 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:30:50.741+0000 7fccc8cc1b80 1 -- 172.21.15.52:0/2238870944 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,fsmap=93,monmap=4+}) -- 0x5640b9323090 con 0x5640b9120e40 2024-05-27T07:30:50.751 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:30:50.747+0000 7fcc937fe700 1 -- 172.21.15.52:0/2238870944 <== mon.2 v2:172.21.15.105:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fcca80044d0 con 0x5640b9120e40 2024-05-27T07:30:50.751 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:30:50.747+0000 7fcc937fe700 1 -- 172.21.15.52:0/2238870944 <== mon.2 v2:172.21.15.105:3300/0 5 ==== fsmap(e 93) ==== 4792+0+0 (secure 0 0 0) 0x7fcca801c450 con 0x5640b9120e40 2024-05-27T07:30:50.751 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:30:50.747+0000 7fccc8cc1b80 10 client.26195 fetch_fsmap finished waiting for FSMap version 93 2024-05-27T07:30:50.751 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:30:50.747+0000 7fccc8cc1b80 10 client.26195 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-05-27T07:30:50.751 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:30:50.747+0000 7fccc8cc1b80 20 client.26195 populate_metadata read hostname 'smithi052' 2024-05-27T07:30:50.751 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:30:50.747+0000 7fccc8cc1b80 1 --2- 172.21.15.52:0/2238870944 >> [v2:172.21.15.105:6838/500348151,v1:172.21.15.105:6839/500348151] conn(0x5640b9323660 0x5640b9325a80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:30:50.751 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:30:50.747+0000 7fccc8cc1b80 4 client.26195 mds_command: new command op to 24527 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-05-27T07:30:50.751 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:30:50.747+0000 7fccc8cc1b80 1 -- 172.21.15.52:0/2238870944 --> [v2:172.21.15.105:6838/500348151,v1:172.21.15.105:6839/500348151] -- command(tid 0: {"prefix": "scrub status"}) -- 0x7fcca400f8c0 con 0x5640b9323660 2024-05-27T07:30:50.751 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:30:50.749+0000 7fccb37fe700 1 --2- 172.21.15.52:0/2238870944 >> [v2:172.21.15.105:6838/500348151,v1:172.21.15.105:6839/500348151] conn(0x5640b9323660 0x5640b9325a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:30:50.751 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:30:50.751+0000 7fccb37fe700 1 --2- 172.21.15.52:0/2238870944 >> [v2:172.21.15.105:6838/500348151,v1:172.21.15.105:6839/500348151] conn(0x5640b9323660 0x5640b9325a80 crc :-1 s=READY pgs=110 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=e455df0df1fa6066 server_cookie=e28fdf6e1cef9a54 in_seq=0 out_seq=0 2024-05-27T07:30:50.752 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:30:50.751+0000 7fcc937fe700 10 client.26195 ms_handle_connect on v2:172.21.15.105:6838/500348151 2024-05-27T07:30:50.752 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:30:50.755+0000 7fcc937fe700 1 -- 172.21.15.52:0/2238870944 <== mds.0 v2:172.21.15.105:6838/500348151 1 ==== command_reply(tid 0: 0 ) ==== 8+0+268 (crc 0 0 0) 0x7fcca400f8c0 con 0x5640b9323660 2024-05-27T07:30:50.752 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:30:50.755+0000 7fcc937fe700 10 client.26195 handle_command_reply: tid=0 2024-05-27T07:30:50.752 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:30:50.755+0000 7fccc8cc1b80 1 client.26195 shutdown 2024-05-27T07:30:50.752 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:30:50.757+0000 7fccc8cc1b80 1 -- 172.21.15.52:0/2238870944 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5640b9120e40 msgr2=0x5640b92b0d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:30:50.752 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:30:50.757+0000 7fccc8cc1b80 1 --2- 172.21.15.52:0/2238870944 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5640b9120e40 0x5640b92b0d40 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7fcca8013c00 tx=0x7fcca8000bc0 comp rx=0 tx=0).stop 2024-05-27T07:30:50.752 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:30:50.758+0000 7fccc8cc1b80 1 -- 172.21.15.52:0/2238870944 shutdown_connections 2024-05-27T07:30:50.752 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:30:50.758+0000 7fccc8cc1b80 1 --2- 172.21.15.52:0/2238870944 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5640b92a4c50 0x5640b92bfe00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:30:50.752 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:30:50.758+0000 7fccc8cc1b80 1 --2- 172.21.15.52:0/2238870944 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5640b9120e40 0x5640b92b0d40 unknown :-1 s=CLOSED pgs=168 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:30:50.752 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:30:50.758+0000 7fccc8cc1b80 1 --2- 172.21.15.52:0/2238870944 >> [v2:172.21.15.105:6838/500348151,v1:172.21.15.105:6839/500348151] conn(0x5640b9323660 0x5640b9325a80 crc :-1 s=READY pgs=110 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:30:50.752 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:30:50.758+0000 7fccc8cc1b80 1 --2- 172.21.15.52:0/2238870944 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x5640b929ef80 0x5640b92b1280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:30:50.752 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:30:50.758+0000 7fccc8cc1b80 1 -- 172.21.15.52:0/2238870944 >> 172.21.15.52:0/2238870944 conn(0x5640b917baf0 msgr2=0x5640b92a3320 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:30:50.753 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:30:50.759+0000 7fcc937fe700 0 client.26195 ms_handle_reset on v2:172.21.15.105:6838/500348151 2024-05-27T07:30:50.753 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:30:50.760+0000 7fccc8cc1b80 1 -- 172.21.15.52:0/2238870944 shutdown_connections 2024-05-27T07:30:50.756 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:30:50.762+0000 7fccc8cc1b80 1 -- 172.21.15.52:0/2238870944 wait complete. 2024-05-27T07:30:50.765 INFO:journalctl@ceph.mds.j.smithi052.stdout:May 27 07:30:50 smithi052 systemd[1]: Starting Ceph mds.j for 73df2636-1bf8-11ef-bc9b-c7b262605968... 2024-05-27T07:30:50.765 INFO:teuthology.orchestra.run.smithi052.stdout:{ 2024-05-27T07:30:50.765 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "scrub active (1019 inodes in the stack)", 2024-05-27T07:30:50.765 INFO:teuthology.orchestra.run.smithi052.stdout: "scrubs": { 2024-05-27T07:30:50.765 INFO:teuthology.orchestra.run.smithi052.stdout: "8ab50389-7f33-4519-a5fc-243c0e6b3763": { 2024-05-27T07:30:50.765 INFO:teuthology.orchestra.run.smithi052.stdout: "path": "/", 2024-05-27T07:30:50.766 INFO:teuthology.orchestra.run.smithi052.stdout: "tag": "8ab50389-7f33-4519-a5fc-243c0e6b3763", 2024-05-27T07:30:50.766 INFO:teuthology.orchestra.run.smithi052.stdout: "options": "recursive,force" 2024-05-27T07:30:50.766 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-05-27T07:30:50.766 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-05-27T07:30:50.766 INFO:teuthology.orchestra.run.smithi052.stdout:} 2024-05-27T07:30:50.766 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:30:50.767+0000 7fccc8cc1b80 20 client.26195 trim_cache size 0 max 16384 2024-05-27T07:30:50.766 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:30:50.768+0000 7fccba2ac700 1 -- 172.21.15.52:0/408951316 >> [v2:172.21.15.52:6800/1822318384,v1:172.21.15.52:6801/1822318384] conn(0x7fcc9c0b2b80 msgr2=0x7fcc9c0b5090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:30:50.767 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:30:50.768+0000 7fccba2ac700 1 --2- 172.21.15.52:0/408951316 >> [v2:172.21.15.52:6800/1822318384,v1:172.21.15.52:6801/1822318384] conn(0x7fcc9c0b2b80 0x7fcc9c0b5090 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7fccac00efd0 tx=0x7fccac013000 comp rx=0 tx=0).stop 2024-05-27T07:30:50.767 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:30:50.768+0000 7fccba2ac700 1 -- 172.21.15.52:0/408951316 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fccb410b990 msgr2=0x7fccb40872d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:30:50.767 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:30:50.768+0000 7fccba2ac700 1 --2- 172.21.15.52:0/408951316 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fccb410b990 0x7fccb40872d0 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7fccac00c0c0 tx=0x7fccac00c3d0 comp rx=0 tx=0).stop 2024-05-27T07:30:50.767 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:30:50.772+0000 7fccba2ac700 1 -- 172.21.15.52:0/408951316 shutdown_connections 2024-05-27T07:30:50.767 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:30:50.772+0000 7fccba2ac700 1 --2- 172.21.15.52:0/408951316 >> [v2:172.21.15.52:6800/1822318384,v1:172.21.15.52:6801/1822318384] conn(0x7fcc9c0b2b80 0x7fcc9c0b5090 unknown :-1 s=CLOSED pgs=140 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:30:50.767 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:30:50.772+0000 7fccba2ac700 1 --2- 172.21.15.52:0/408951316 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fccb41110a0 0x7fccb4080a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:30:50.767 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:30:50.772+0000 7fccba2ac700 1 --2- 172.21.15.52:0/408951316 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fccb410b990 0x7fccb40872d0 unknown :-1 s=CLOSED pgs=164 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:30:50.768 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:30:50.772+0000 7fccba2ac700 1 --2- 172.21.15.52:0/408951316 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fccb4110750 0x7fccb4080400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:30:50.768 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:30:50.772+0000 7fccba2ac700 1 -- 172.21.15.52:0/408951316 >> 172.21.15.52:0/408951316 conn(0x7fccb4079e60 msgr2=0x7fccb407b550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:30:50.768 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:30:50.774+0000 7fccba2ac700 1 -- 172.21.15.52:0/408951316 shutdown_connections 2024-05-27T07:30:50.768 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:30:50.774+0000 7fccba2ac700 1 -- 172.21.15.52:0/408951316 wait complete. 2024-05-27T07:30:51.776 INFO:tasks.cephfs.filesystem:scrub status for tag:8ab50389-7f33-4519-a5fc-243c0e6b3763 - {'path': '/', 'tag': '8ab50389-7f33-4519-a5fc-243c0e6b3763', 'options': 'recursive,force'} 2024-05-27T07:30:52.136 INFO:journalctl@ceph.mds.j.smithi052.stdout:May 27 07:30:51 smithi052 podman[108354]: 2024-05-27 07:30:51.844959984 +0000 UTC m=+0.141714906 image pull e36ea4fc8a5807e8d2701370e350a0de3856538797f7fba64cfce7e344bd6799 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6 2024-05-27T07:30:52.137 INFO:journalctl@ceph.mds.j.smithi052.stdout:May 27 07:30:51 smithi052 podman[108354]: 2024-05-27 07:30:51.970388383 +0000 UTC m=+0.267143296 container create 1797abb68ea79dde47fc28d12c9cdbd9da61ed99aa86f51e26ab091b595655a6 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-j, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, ceph=True, GIT_BRANCH=HEAD, org.label-schema.build-date=20240521, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux ) 2024-05-27T07:30:52.428 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:30:52 smithi052 ceph-mon[31955]: pgmap v173: 129 pgs: 129 active+clean; 399 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 3.6 MiB/s rd, 5.6 MiB/s wr, 1.51k op/s 2024-05-27T07:30:52.428 INFO:journalctl@ceph.mds.j.smithi052.stdout:May 27 07:30:52 smithi052 podman[108354]: 2024-05-27 07:30:52.311604793 +0000 UTC m=+0.608359705 container init 1797abb68ea79dde47fc28d12c9cdbd9da61ed99aa86f51e26ab091b595655a6 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-j, ceph=True, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, maintainer=Guillaume Abrioux , GIT_CLEAN=True, GIT_BRANCH=HEAD, org.label-schema.build-date=20240521, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-05-27T07:30:52.428 INFO:journalctl@ceph.mds.j.smithi052.stdout:May 27 07:30:52 smithi052 podman[108354]: 2024-05-27 07:30:52.321651529 +0000 UTC m=+0.618406442 container start 1797abb68ea79dde47fc28d12c9cdbd9da61ed99aa86f51e26ab091b595655a6 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-j, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, org.label-schema.license=GPLv2, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.build-date=20240521, ceph=True, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.vendor=CentOS) 2024-05-27T07:30:52.428 INFO:journalctl@ceph.mds.j.smithi052.stdout:May 27 07:30:52 smithi052 bash[108354]: 1797abb68ea79dde47fc28d12c9cdbd9da61ed99aa86f51e26ab091b595655a6 2024-05-27T07:30:52.441 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:30:52 smithi105 ceph-mon[35977]: pgmap v173: 129 pgs: 129 active+clean; 399 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 3.6 MiB/s rd, 5.6 MiB/s wr, 1.51k op/s 2024-05-27T07:30:52.604 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:30:52 smithi173 ceph-mon[37000]: pgmap v173: 129 pgs: 129 active+clean; 399 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 3.6 MiB/s rd, 5.6 MiB/s wr, 1.51k op/s 2024-05-27T07:30:52.732 INFO:journalctl@ceph.mds.j.smithi052.stdout:May 27 07:30:52 smithi052 systemd[1]: Started Ceph mds.j for 73df2636-1bf8-11ef-bc9b-c7b262605968. 2024-05-27T07:30:52.732 INFO:journalctl@ceph.mds.j.smithi052.stdout:May 27 07:30:52 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-j[108368]: starting mds.j at 2024-05-27T07:30:53.197 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:30:53 smithi052 ceph-mon[31955]: daemon mds.j restarted 2024-05-27T07:30:53.450 INFO:journalctl@ceph.mds.j.smithi052.stdout:May 27 07:30:53 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-j[108368]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: In function 'std::set SnapClient::filter(const std::set&) const' thread 7fb95747c700 time 2024-05-27T07:30:53.447580+0000 2024-05-27T07:30:53.451 INFO:journalctl@ceph.mds.j.smithi052.stdout:May 27 07:30:53 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-j[108368]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: 197: FAILED ceph_assert(cached_version > 0) 2024-05-27T07:30:53.451 INFO:journalctl@ceph.mds.j.smithi052.stdout:May 27 07:30:53 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-j[108368]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:30:53.451 INFO:journalctl@ceph.mds.j.smithi052.stdout:May 27 07:30:53 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-j[108368]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x127) [0x7fb969cc1ba6] 2024-05-27T07:30:53.451 INFO:journalctl@ceph.mds.j.smithi052.stdout:May 27 07:30:53 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-j[108368]: 2: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7fb969cc1dd1] 2024-05-27T07:30:53.451 INFO:journalctl@ceph.mds.j.smithi052.stdout:May 27 07:30:53 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-j[108368]: 3: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x555dc771b23d] 2024-05-27T07:30:53.451 INFO:journalctl@ceph.mds.j.smithi052.stdout:May 27 07:30:53 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-j[108368]: 4: (SnapRealm::build_snap_set() const+0x323) [0x555dc7706971] 2024-05-27T07:30:53.451 INFO:journalctl@ceph.mds.j.smithi052.stdout:May 27 07:30:53 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-j[108368]: 5: (SnapRealm::check_cache() const+0x13e) [0x555dc7704fa0] 2024-05-27T07:30:53.451 INFO:journalctl@ceph.mds.j.smithi052.stdout:May 27 07:30:53 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-j[108368]: 6: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x555dc7705651] 2024-05-27T07:30:53.451 INFO:journalctl@ceph.mds.j.smithi052.stdout:May 27 07:30:53 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-j[108368]: 7: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x555dc769a347] 2024-05-27T07:30:53.451 INFO:journalctl@ceph.mds.j.smithi052.stdout:May 27 07:30:53 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-j[108368]: 8: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x555dc739687e] 2024-05-27T07:30:53.451 INFO:journalctl@ceph.mds.j.smithi052.stdout:May 27 07:30:53 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-j[108368]: 9: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x279d) [0x555dc739bf2f] 2024-05-27T07:30:53.452 INFO:journalctl@ceph.mds.j.smithi052.stdout:May 27 07:30:53 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-j[108368]: 10: (EUpdate::replay(MDSRank*)+0x63) [0x555dc73a6969] 2024-05-27T07:30:53.452 INFO:journalctl@ceph.mds.j.smithi052.stdout:May 27 07:30:53 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-j[108368]: 11: (MDLog::_replay_thread()+0x1f38) [0x555dc773f48c] 2024-05-27T07:30:53.452 INFO:journalctl@ceph.mds.j.smithi052.stdout:May 27 07:30:53 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-j[108368]: 12: (MDLog::ReplayThread::entry()+0x11) [0x555dc77489cb] 2024-05-27T07:30:53.452 INFO:journalctl@ceph.mds.j.smithi052.stdout:May 27 07:30:53 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-j[108368]: 13: (Thread::entry_wrapper()+0x43) [0x7fb969c9072d] 2024-05-27T07:30:53.452 INFO:journalctl@ceph.mds.j.smithi052.stdout:May 27 07:30:53 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-j[108368]: 14: (Thread::_entry_func(void*)+0xd) [0x7fb969c90749] 2024-05-27T07:30:53.452 INFO:journalctl@ceph.mds.j.smithi052.stdout:May 27 07:30:53 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-j[108368]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7fb9688341ca] 2024-05-27T07:30:53.452 INFO:journalctl@ceph.mds.j.smithi052.stdout:May 27 07:30:53 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-j[108368]: 16: clone() 2024-05-27T07:30:53.452 INFO:journalctl@ceph.mds.j.smithi052.stdout:May 27 07:30:53 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-j[108368]: 2024-05-27T07:30:53.456+0000 7fb95747c700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: In function 'std::set SnapClient::filter(const std::set&) const' thread 7fb95747c700 time 2024-05-27T07:30:53.447580+0000 2024-05-27T07:30:53.452 INFO:journalctl@ceph.mds.j.smithi052.stdout:May 27 07:30:53 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-j[108368]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: 197: FAILED ceph_assert(cached_version > 0) 2024-05-27T07:30:53.452 INFO:journalctl@ceph.mds.j.smithi052.stdout:May 27 07:30:53 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-j[108368]: 2024-05-27T07:30:53.452 INFO:journalctl@ceph.mds.j.smithi052.stdout:May 27 07:30:53 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-j[108368]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:30:53.452 INFO:journalctl@ceph.mds.j.smithi052.stdout:May 27 07:30:53 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-j[108368]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x127) [0x7fb969cc1ba6] 2024-05-27T07:30:53.452 INFO:journalctl@ceph.mds.j.smithi052.stdout:May 27 07:30:53 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-j[108368]: 2: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7fb969cc1dd1] 2024-05-27T07:30:53.452 INFO:journalctl@ceph.mds.j.smithi052.stdout:May 27 07:30:53 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-j[108368]: 3: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x555dc771b23d] 2024-05-27T07:30:53.453 INFO:journalctl@ceph.mds.j.smithi052.stdout:May 27 07:30:53 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-j[108368]: 4: (SnapRealm::build_snap_set() const+0x323) [0x555dc7706971] 2024-05-27T07:30:53.453 INFO:journalctl@ceph.mds.j.smithi052.stdout:May 27 07:30:53 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-j[108368]: 5: (SnapRealm::check_cache() const+0x13e) [0x555dc7704fa0] 2024-05-27T07:30:53.453 INFO:journalctl@ceph.mds.j.smithi052.stdout:May 27 07:30:53 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-j[108368]: 6: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x555dc7705651] 2024-05-27T07:30:53.453 INFO:journalctl@ceph.mds.j.smithi052.stdout:May 27 07:30:53 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-j[108368]: 7: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x555dc769a347] 2024-05-27T07:30:53.453 INFO:journalctl@ceph.mds.j.smithi052.stdout:May 27 07:30:53 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-j[108368]: 8: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x555dc739687e] 2024-05-27T07:30:53.453 INFO:journalctl@ceph.mds.j.smithi052.stdout:May 27 07:30:53 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-j[108368]: 9: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x279d) [0x555dc739bf2f] 2024-05-27T07:30:53.453 INFO:journalctl@ceph.mds.j.smithi052.stdout:May 27 07:30:53 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-j[108368]: 10: (EUpdate::replay(MDSRank*)+0x63) [0x555dc73a6969] 2024-05-27T07:30:53.453 INFO:journalctl@ceph.mds.j.smithi052.stdout:May 27 07:30:53 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-j[108368]: 11: (MDLog::_replay_thread()+0x1f38) [0x555dc773f48c] 2024-05-27T07:30:53.453 INFO:journalctl@ceph.mds.j.smithi052.stdout:May 27 07:30:53 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-j[108368]: 12: (MDLog::ReplayThread::entry()+0x11) [0x555dc77489cb] 2024-05-27T07:30:53.453 INFO:journalctl@ceph.mds.j.smithi052.stdout:May 27 07:30:53 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-j[108368]: 13: (Thread::entry_wrapper()+0x43) [0x7fb969c9072d] 2024-05-27T07:30:53.453 INFO:journalctl@ceph.mds.j.smithi052.stdout:May 27 07:30:53 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-j[108368]: 14: (Thread::_entry_func(void*)+0xd) [0x7fb969c90749] 2024-05-27T07:30:53.453 INFO:journalctl@ceph.mds.j.smithi052.stdout:May 27 07:30:53 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-j[108368]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7fb9688341ca] 2024-05-27T07:30:53.454 INFO:journalctl@ceph.mds.j.smithi052.stdout:May 27 07:30:53 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-j[108368]: 16: clone() 2024-05-27T07:30:53.454 INFO:journalctl@ceph.mds.j.smithi052.stdout:May 27 07:30:53 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-j[108368]: 2024-05-27T07:30:53.454 INFO:journalctl@ceph.mds.j.smithi052.stdout:May 27 07:30:53 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-j[108368]: *** Caught signal (Aborted) ** 2024-05-27T07:30:53.454 INFO:journalctl@ceph.mds.j.smithi052.stdout:May 27 07:30:53 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-j[108368]: in thread 7fb95747c700 thread_name:md_log_replay 2024-05-27T07:30:53.603 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:30:53 smithi173 ceph-mon[37000]: daemon mds.j restarted 2024-05-27T07:30:53.691 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:30:53 smithi105 ceph-mon[35977]: daemon mds.j restarted 2024-05-27T07:30:53.734 INFO:journalctl@ceph.mds.j.smithi052.stdout:May 27 07:30:53 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-j[108368]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:30:53.735 INFO:journalctl@ceph.mds.j.smithi052.stdout:May 27 07:30:53 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-j[108368]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7fb96883ed20] 2024-05-27T07:30:53.735 INFO:journalctl@ceph.mds.j.smithi052.stdout:May 27 07:30:53 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-j[108368]: 2: gsignal() 2024-05-27T07:30:53.735 INFO:journalctl@ceph.mds.j.smithi052.stdout:May 27 07:30:53 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-j[108368]: 3: abort() 2024-05-27T07:30:53.735 INFO:journalctl@ceph.mds.j.smithi052.stdout:May 27 07:30:53 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-j[108368]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x26e) [0x7fb969cc1ced] 2024-05-27T07:30:53.735 INFO:journalctl@ceph.mds.j.smithi052.stdout:May 27 07:30:53 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-j[108368]: 5: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7fb969cc1dd1] 2024-05-27T07:30:53.735 INFO:journalctl@ceph.mds.j.smithi052.stdout:May 27 07:30:53 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-j[108368]: 6: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x555dc771b23d] 2024-05-27T07:30:53.735 INFO:journalctl@ceph.mds.j.smithi052.stdout:May 27 07:30:53 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-j[108368]: 7: (SnapRealm::build_snap_set() const+0x323) [0x555dc7706971] 2024-05-27T07:30:53.735 INFO:journalctl@ceph.mds.j.smithi052.stdout:May 27 07:30:53 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-j[108368]: 8: (SnapRealm::check_cache() const+0x13e) [0x555dc7704fa0] 2024-05-27T07:30:53.735 INFO:journalctl@ceph.mds.j.smithi052.stdout:May 27 07:30:53 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-j[108368]: 9: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x555dc7705651] 2024-05-27T07:30:53.735 INFO:journalctl@ceph.mds.j.smithi052.stdout:May 27 07:30:53 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-j[108368]: 10: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x555dc769a347] 2024-05-27T07:30:53.735 INFO:journalctl@ceph.mds.j.smithi052.stdout:May 27 07:30:53 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-j[108368]: 11: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x555dc739687e] 2024-05-27T07:30:53.735 INFO:journalctl@ceph.mds.j.smithi052.stdout:May 27 07:30:53 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-j[108368]: 12: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x279d) [0x555dc739bf2f] 2024-05-27T07:30:53.736 INFO:journalctl@ceph.mds.j.smithi052.stdout:May 27 07:30:53 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-j[108368]: 13: (EUpdate::replay(MDSRank*)+0x63) [0x555dc73a6969] 2024-05-27T07:30:53.736 INFO:journalctl@ceph.mds.j.smithi052.stdout:May 27 07:30:53 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-j[108368]: 14: (MDLog::_replay_thread()+0x1f38) [0x555dc773f48c] 2024-05-27T07:30:53.736 INFO:journalctl@ceph.mds.j.smithi052.stdout:May 27 07:30:53 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-j[108368]: 15: (MDLog::ReplayThread::entry()+0x11) [0x555dc77489cb] 2024-05-27T07:30:53.736 INFO:journalctl@ceph.mds.j.smithi052.stdout:May 27 07:30:53 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-j[108368]: 16: (Thread::entry_wrapper()+0x43) [0x7fb969c9072d] 2024-05-27T07:30:53.736 INFO:journalctl@ceph.mds.j.smithi052.stdout:May 27 07:30:53 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-j[108368]: 17: (Thread::_entry_func(void*)+0xd) [0x7fb969c90749] 2024-05-27T07:30:53.736 INFO:journalctl@ceph.mds.j.smithi052.stdout:May 27 07:30:53 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-j[108368]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7fb9688341ca] 2024-05-27T07:30:53.736 INFO:journalctl@ceph.mds.j.smithi052.stdout:May 27 07:30:53 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-j[108368]: 19: clone() 2024-05-27T07:30:53.736 INFO:journalctl@ceph.mds.j.smithi052.stdout:May 27 07:30:53 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-j[108368]: 2024-05-27T07:30:53.465+0000 7fb95747c700 -1 *** Caught signal (Aborted) ** 2024-05-27T07:30:53.736 INFO:journalctl@ceph.mds.j.smithi052.stdout:May 27 07:30:53 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-j[108368]: in thread 7fb95747c700 thread_name:md_log_replay 2024-05-27T07:30:53.736 INFO:journalctl@ceph.mds.j.smithi052.stdout:May 27 07:30:53 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-j[108368]: 2024-05-27T07:30:53.736 INFO:journalctl@ceph.mds.j.smithi052.stdout:May 27 07:30:53 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-j[108368]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:30:53.736 INFO:journalctl@ceph.mds.j.smithi052.stdout:May 27 07:30:53 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-j[108368]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7fb96883ed20] 2024-05-27T07:30:53.737 INFO:journalctl@ceph.mds.j.smithi052.stdout:May 27 07:30:53 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-j[108368]: 2: gsignal() 2024-05-27T07:30:53.737 INFO:journalctl@ceph.mds.j.smithi052.stdout:May 27 07:30:53 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-j[108368]: 3: abort() 2024-05-27T07:30:53.737 INFO:journalctl@ceph.mds.j.smithi052.stdout:May 27 07:30:53 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-j[108368]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x26e) [0x7fb969cc1ced] 2024-05-27T07:30:53.737 INFO:journalctl@ceph.mds.j.smithi052.stdout:May 27 07:30:53 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-j[108368]: 5: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7fb969cc1dd1] 2024-05-27T07:30:53.737 INFO:journalctl@ceph.mds.j.smithi052.stdout:May 27 07:30:53 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-j[108368]: 6: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x555dc771b23d] 2024-05-27T07:30:53.737 INFO:journalctl@ceph.mds.j.smithi052.stdout:May 27 07:30:53 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-j[108368]: 7: (SnapRealm::build_snap_set() const+0x323) [0x555dc7706971] 2024-05-27T07:30:53.737 INFO:journalctl@ceph.mds.j.smithi052.stdout:May 27 07:30:53 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-j[108368]: 8: (SnapRealm::check_cache() const+0x13e) [0x555dc7704fa0] 2024-05-27T07:30:53.737 INFO:journalctl@ceph.mds.j.smithi052.stdout:May 27 07:30:53 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-j[108368]: 9: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x555dc7705651] 2024-05-27T07:30:53.737 INFO:journalctl@ceph.mds.j.smithi052.stdout:May 27 07:30:53 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-j[108368]: 10: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x555dc769a347] 2024-05-27T07:30:53.737 INFO:journalctl@ceph.mds.j.smithi052.stdout:May 27 07:30:53 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-j[108368]: 11: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x555dc739687e] 2024-05-27T07:30:53.737 INFO:journalctl@ceph.mds.j.smithi052.stdout:May 27 07:30:53 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-j[108368]: 12: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x279d) [0x555dc739bf2f] 2024-05-27T07:30:53.737 INFO:journalctl@ceph.mds.j.smithi052.stdout:May 27 07:30:53 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-j[108368]: 13: (EUpdate::replay(MDSRank*)+0x63) [0x555dc73a6969] 2024-05-27T07:30:53.738 INFO:journalctl@ceph.mds.j.smithi052.stdout:May 27 07:30:53 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-j[108368]: 14: (MDLog::_replay_thread()+0x1f38) [0x555dc773f48c] 2024-05-27T07:30:53.738 INFO:journalctl@ceph.mds.j.smithi052.stdout:May 27 07:30:53 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-j[108368]: 15: (MDLog::ReplayThread::entry()+0x11) [0x555dc77489cb] 2024-05-27T07:30:53.738 INFO:journalctl@ceph.mds.j.smithi052.stdout:May 27 07:30:53 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-j[108368]: 16: (Thread::entry_wrapper()+0x43) [0x7fb969c9072d] 2024-05-27T07:30:53.738 INFO:journalctl@ceph.mds.j.smithi052.stdout:May 27 07:30:53 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-j[108368]: 17: (Thread::_entry_func(void*)+0xd) [0x7fb969c90749] 2024-05-27T07:30:53.738 INFO:journalctl@ceph.mds.j.smithi052.stdout:May 27 07:30:53 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-j[108368]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7fb9688341ca] 2024-05-27T07:30:53.738 INFO:journalctl@ceph.mds.j.smithi052.stdout:May 27 07:30:53 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-j[108368]: 19: clone() 2024-05-27T07:30:53.738 INFO:journalctl@ceph.mds.j.smithi052.stdout:May 27 07:30:53 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-j[108368]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-05-27T07:30:53.738 INFO:journalctl@ceph.mds.j.smithi052.stdout:May 27 07:30:53 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-j[108368]: 2024-05-27T07:30:53.738 INFO:journalctl@ceph.mds.j.smithi052.stdout:May 27 07:30:53 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-j[108368]: -2> 2024-05-27T07:30:53.456+0000 7fb95747c700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: In function 'std::set SnapClient::filter(const std::set&) const' thread 7fb95747c700 time 2024-05-27T07:30:53.447580+0000 2024-05-27T07:30:53.738 INFO:journalctl@ceph.mds.j.smithi052.stdout:May 27 07:30:53 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-j[108368]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: 197: FAILED ceph_assert(cached_version > 0) 2024-05-27T07:30:53.738 INFO:journalctl@ceph.mds.j.smithi052.stdout:May 27 07:30:53 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-j[108368]: 2024-05-27T07:30:53.739 INFO:journalctl@ceph.mds.j.smithi052.stdout:May 27 07:30:53 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-j[108368]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:30:53.739 INFO:journalctl@ceph.mds.j.smithi052.stdout:May 27 07:30:53 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-j[108368]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x127) [0x7fb969cc1ba6] 2024-05-27T07:30:53.739 INFO:journalctl@ceph.mds.j.smithi052.stdout:May 27 07:30:53 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-j[108368]: 2: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7fb969cc1dd1] 2024-05-27T07:30:53.739 INFO:journalctl@ceph.mds.j.smithi052.stdout:May 27 07:30:53 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-j[108368]: 3: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x555dc771b23d] 2024-05-27T07:30:53.739 INFO:journalctl@ceph.mds.j.smithi052.stdout:May 27 07:30:53 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-j[108368]: 4: (SnapRealm::build_snap_set() const+0x323) [0x555dc7706971] 2024-05-27T07:30:53.739 INFO:journalctl@ceph.mds.j.smithi052.stdout:May 27 07:30:53 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-j[108368]: 5: (SnapRealm::check_cache() const+0x13e) [0x555dc7704fa0] 2024-05-27T07:30:53.739 INFO:journalctl@ceph.mds.j.smithi052.stdout:May 27 07:30:53 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-j[108368]: 6: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x555dc7705651] 2024-05-27T07:30:53.739 INFO:journalctl@ceph.mds.j.smithi052.stdout:May 27 07:30:53 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-j[108368]: 7: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x555dc769a347] 2024-05-27T07:30:53.740 INFO:journalctl@ceph.mds.j.smithi052.stdout:May 27 07:30:53 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-j[108368]: 8: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x555dc739687e] 2024-05-27T07:30:53.740 INFO:journalctl@ceph.mds.j.smithi052.stdout:May 27 07:30:53 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-j[108368]: 9: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x279d) [0x555dc739bf2f] 2024-05-27T07:30:53.740 INFO:journalctl@ceph.mds.j.smithi052.stdout:May 27 07:30:53 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-j[108368]: 10: (EUpdate::replay(MDSRank*)+0x63) [0x555dc73a6969] 2024-05-27T07:30:53.740 INFO:journalctl@ceph.mds.j.smithi052.stdout:May 27 07:30:53 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-j[108368]: 11: (MDLog::_replay_thread()+0x1f38) [0x555dc773f48c] 2024-05-27T07:30:53.740 INFO:journalctl@ceph.mds.j.smithi052.stdout:May 27 07:30:53 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-j[108368]: 12: (MDLog::ReplayThread::entry()+0x11) [0x555dc77489cb] 2024-05-27T07:30:53.740 INFO:journalctl@ceph.mds.j.smithi052.stdout:May 27 07:30:53 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-j[108368]: 13: (Thread::entry_wrapper()+0x43) [0x7fb969c9072d] 2024-05-27T07:30:53.740 INFO:journalctl@ceph.mds.j.smithi052.stdout:May 27 07:30:53 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-j[108368]: 14: (Thread::_entry_func(void*)+0xd) [0x7fb969c90749] 2024-05-27T07:30:53.740 INFO:journalctl@ceph.mds.j.smithi052.stdout:May 27 07:30:53 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-j[108368]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7fb9688341ca] 2024-05-27T07:30:53.740 INFO:journalctl@ceph.mds.j.smithi052.stdout:May 27 07:30:53 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-j[108368]: 16: clone() 2024-05-27T07:30:53.740 INFO:journalctl@ceph.mds.j.smithi052.stdout:May 27 07:30:53 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-j[108368]: 2024-05-27T07:30:53.740 INFO:journalctl@ceph.mds.j.smithi052.stdout:May 27 07:30:53 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-j[108368]: 0> 2024-05-27T07:30:53.465+0000 7fb95747c700 -1 *** Caught signal (Aborted) ** 2024-05-27T07:30:53.740 INFO:journalctl@ceph.mds.j.smithi052.stdout:May 27 07:30:53 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-j[108368]: in thread 7fb95747c700 thread_name:md_log_replay 2024-05-27T07:30:53.740 INFO:journalctl@ceph.mds.j.smithi052.stdout:May 27 07:30:53 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-j[108368]: 2024-05-27T07:30:53.741 INFO:journalctl@ceph.mds.j.smithi052.stdout:May 27 07:30:53 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-j[108368]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:30:53.741 INFO:journalctl@ceph.mds.j.smithi052.stdout:May 27 07:30:53 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-j[108368]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7fb96883ed20] 2024-05-27T07:30:53.741 INFO:journalctl@ceph.mds.j.smithi052.stdout:May 27 07:30:53 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-j[108368]: 2: gsignal() 2024-05-27T07:30:53.741 INFO:journalctl@ceph.mds.j.smithi052.stdout:May 27 07:30:53 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-j[108368]: 3: abort() 2024-05-27T07:30:53.741 INFO:journalctl@ceph.mds.j.smithi052.stdout:May 27 07:30:53 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-j[108368]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x26e) [0x7fb969cc1ced] 2024-05-27T07:30:53.741 INFO:journalctl@ceph.mds.j.smithi052.stdout:May 27 07:30:53 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-j[108368]: 5: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7fb969cc1dd1] 2024-05-27T07:30:53.741 INFO:journalctl@ceph.mds.j.smithi052.stdout:May 27 07:30:53 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-j[108368]: 6: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x555dc771b23d] 2024-05-27T07:30:53.741 INFO:journalctl@ceph.mds.j.smithi052.stdout:May 27 07:30:53 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-j[108368]: 7: (SnapRealm::build_snap_set() const+0x323) [0x555dc7706971] 2024-05-27T07:30:53.741 INFO:journalctl@ceph.mds.j.smithi052.stdout:May 27 07:30:53 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-j[108368]: 8: (SnapRealm::check_cache() const+0x13e) [0x555dc7704fa0] 2024-05-27T07:30:53.741 INFO:journalctl@ceph.mds.j.smithi052.stdout:May 27 07:30:53 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-j[108368]: 9: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x555dc7705651] 2024-05-27T07:30:53.741 INFO:journalctl@ceph.mds.j.smithi052.stdout:May 27 07:30:53 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-j[108368]: 10: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x555dc769a347] 2024-05-27T07:30:53.741 INFO:journalctl@ceph.mds.j.smithi052.stdout:May 27 07:30:53 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-j[108368]: 11: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x555dc739687e] 2024-05-27T07:30:53.741 INFO:journalctl@ceph.mds.j.smithi052.stdout:May 27 07:30:53 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-j[108368]: 12: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x279d) [0x555dc739bf2f] 2024-05-27T07:30:53.742 INFO:journalctl@ceph.mds.j.smithi052.stdout:May 27 07:30:53 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-j[108368]: 13: (EUpdate::replay(MDSRank*)+0x63) [0x555dc73a6969] 2024-05-27T07:30:53.742 INFO:journalctl@ceph.mds.j.smithi052.stdout:May 27 07:30:53 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-j[108368]: 14: (MDLog::_replay_thread()+0x1f38) [0x555dc773f48c] 2024-05-27T07:30:53.742 INFO:journalctl@ceph.mds.j.smithi052.stdout:May 27 07:30:53 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-j[108368]: 15: (MDLog::ReplayThread::entry()+0x11) [0x555dc77489cb] 2024-05-27T07:30:53.742 INFO:journalctl@ceph.mds.j.smithi052.stdout:May 27 07:30:53 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-j[108368]: 16: (Thread::entry_wrapper()+0x43) [0x7fb969c9072d] 2024-05-27T07:30:53.742 INFO:journalctl@ceph.mds.j.smithi052.stdout:May 27 07:30:53 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-j[108368]: 17: (Thread::_entry_func(void*)+0xd) [0x7fb969c90749] 2024-05-27T07:30:53.742 INFO:journalctl@ceph.mds.j.smithi052.stdout:May 27 07:30:53 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-j[108368]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7fb9688341ca] 2024-05-27T07:30:53.742 INFO:journalctl@ceph.mds.j.smithi052.stdout:May 27 07:30:53 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-j[108368]: 19: clone() 2024-05-27T07:30:53.742 INFO:journalctl@ceph.mds.j.smithi052.stdout:May 27 07:30:53 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-j[108368]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-05-27T07:30:53.742 INFO:journalctl@ceph.mds.j.smithi052.stdout:May 27 07:30:53 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-j[108368]: 2024-05-27T07:30:53.742 INFO:journalctl@ceph.mds.j.smithi052.stdout:May 27 07:30:53 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-j[108368]: -3> 2024-05-27T07:30:53.456+0000 7fb95747c700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: In function 'std::set SnapClient::filter(const std::set&) const' thread 7fb95747c700 time 2024-05-27T07:30:53.447580+0000 2024-05-27T07:30:53.742 INFO:journalctl@ceph.mds.j.smithi052.stdout:May 27 07:30:53 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-j[108368]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: 197: FAILED ceph_assert(cached_version > 0) 2024-05-27T07:30:53.742 INFO:journalctl@ceph.mds.j.smithi052.stdout:May 27 07:30:53 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-j[108368]: 2024-05-27T07:30:53.743 INFO:journalctl@ceph.mds.j.smithi052.stdout:May 27 07:30:53 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-j[108368]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:30:53.743 INFO:journalctl@ceph.mds.j.smithi052.stdout:May 27 07:30:53 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-j[108368]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x127) [0x7fb969cc1ba6] 2024-05-27T07:30:53.743 INFO:journalctl@ceph.mds.j.smithi052.stdout:May 27 07:30:53 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-j[108368]: 2: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7fb969cc1dd1] 2024-05-27T07:30:53.743 INFO:journalctl@ceph.mds.j.smithi052.stdout:May 27 07:30:53 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-j[108368]: 3: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x555dc771b23d] 2024-05-27T07:30:53.743 INFO:journalctl@ceph.mds.j.smithi052.stdout:May 27 07:30:53 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-j[108368]: 4: (SnapRealm::build_snap_set() const+0x323) [0x555dc7706971] 2024-05-27T07:30:53.743 INFO:journalctl@ceph.mds.j.smithi052.stdout:May 27 07:30:53 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-j[108368]: 5: (SnapRealm::check_cache() const+0x13e) [0x555dc7704fa0] 2024-05-27T07:30:53.743 INFO:journalctl@ceph.mds.j.smithi052.stdout:May 27 07:30:53 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-j[108368]: 6: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x555dc7705651] 2024-05-27T07:30:53.743 INFO:journalctl@ceph.mds.j.smithi052.stdout:May 27 07:30:53 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-j[108368]: 7: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x555dc769a347] 2024-05-27T07:30:53.743 INFO:journalctl@ceph.mds.j.smithi052.stdout:May 27 07:30:53 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-j[108368]: 8: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x555dc739687e] 2024-05-27T07:30:53.743 INFO:journalctl@ceph.mds.j.smithi052.stdout:May 27 07:30:53 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-j[108368]: 9: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x279d) [0x555dc739bf2f] 2024-05-27T07:30:53.743 INFO:journalctl@ceph.mds.j.smithi052.stdout:May 27 07:30:53 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-j[108368]: 10: (EUpdate::replay(MDSRank*)+0x63) [0x555dc73a6969] 2024-05-27T07:30:53.743 INFO:journalctl@ceph.mds.j.smithi052.stdout:May 27 07:30:53 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-j[108368]: 11: (MDLog::_replay_thread()+0x1f38) [0x555dc773f48c] 2024-05-27T07:30:53.743 INFO:journalctl@ceph.mds.j.smithi052.stdout:May 27 07:30:53 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-j[108368]: 12: (MDLog::ReplayThread::entry()+0x11) [0x555dc77489cb] 2024-05-27T07:30:53.744 INFO:journalctl@ceph.mds.j.smithi052.stdout:May 27 07:30:53 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-j[108368]: 13: (Thread::entry_wrapper()+0x43) [0x7fb969c9072d] 2024-05-27T07:30:53.744 INFO:journalctl@ceph.mds.j.smithi052.stdout:May 27 07:30:53 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-j[108368]: 14: (Thread::_entry_func(void*)+0xd) [0x7fb969c90749] 2024-05-27T07:30:53.744 INFO:journalctl@ceph.mds.j.smithi052.stdout:May 27 07:30:53 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-j[108368]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7fb9688341ca] 2024-05-27T07:30:53.744 INFO:journalctl@ceph.mds.j.smithi052.stdout:May 27 07:30:53 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-j[108368]: 16: clone() 2024-05-27T07:30:53.744 INFO:journalctl@ceph.mds.j.smithi052.stdout:May 27 07:30:53 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-j[108368]: 2024-05-27T07:30:53.744 INFO:journalctl@ceph.mds.j.smithi052.stdout:May 27 07:30:53 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-j[108368]: -1> 2024-05-27T07:30:53.465+0000 7fb95747c700 -1 *** Caught signal (Aborted) ** 2024-05-27T07:30:53.744 INFO:journalctl@ceph.mds.j.smithi052.stdout:May 27 07:30:53 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-j[108368]: in thread 7fb95747c700 thread_name:md_log_replay 2024-05-27T07:30:53.744 INFO:journalctl@ceph.mds.j.smithi052.stdout:May 27 07:30:53 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-j[108368]: 2024-05-27T07:30:53.744 INFO:journalctl@ceph.mds.j.smithi052.stdout:May 27 07:30:53 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-j[108368]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:30:53.744 INFO:journalctl@ceph.mds.j.smithi052.stdout:May 27 07:30:53 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-j[108368]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7fb96883ed20] 2024-05-27T07:30:53.744 INFO:journalctl@ceph.mds.j.smithi052.stdout:May 27 07:30:53 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-j[108368]: 2: gsignal() 2024-05-27T07:30:53.744 INFO:journalctl@ceph.mds.j.smithi052.stdout:May 27 07:30:53 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-j[108368]: 3: abort() 2024-05-27T07:30:53.744 INFO:journalctl@ceph.mds.j.smithi052.stdout:May 27 07:30:53 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-j[108368]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x26e) [0x7fb969cc1ced] 2024-05-27T07:30:53.745 INFO:journalctl@ceph.mds.j.smithi052.stdout:May 27 07:30:53 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-j[108368]: 5: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7fb969cc1dd1] 2024-05-27T07:30:53.745 INFO:journalctl@ceph.mds.j.smithi052.stdout:May 27 07:30:53 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-j[108368]: 6: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x555dc771b23d] 2024-05-27T07:30:53.745 INFO:journalctl@ceph.mds.j.smithi052.stdout:May 27 07:30:53 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-j[108368]: 7: (SnapRealm::build_snap_set() const+0x323) [0x555dc7706971] 2024-05-27T07:30:53.745 INFO:journalctl@ceph.mds.j.smithi052.stdout:May 27 07:30:53 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-j[108368]: 8: (SnapRealm::check_cache() const+0x13e) [0x555dc7704fa0] 2024-05-27T07:30:53.745 INFO:journalctl@ceph.mds.j.smithi052.stdout:May 27 07:30:53 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-j[108368]: 9: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x555dc7705651] 2024-05-27T07:30:53.745 INFO:journalctl@ceph.mds.j.smithi052.stdout:May 27 07:30:53 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-j[108368]: 10: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x555dc769a347] 2024-05-27T07:30:53.745 INFO:journalctl@ceph.mds.j.smithi052.stdout:May 27 07:30:53 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-j[108368]: 11: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x555dc739687e] 2024-05-27T07:30:53.745 INFO:journalctl@ceph.mds.j.smithi052.stdout:May 27 07:30:53 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-j[108368]: 12: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x279d) [0x555dc739bf2f] 2024-05-27T07:30:53.745 INFO:journalctl@ceph.mds.j.smithi052.stdout:May 27 07:30:53 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-j[108368]: 13: (EUpdate::replay(MDSRank*)+0x63) [0x555dc73a6969] 2024-05-27T07:30:53.745 INFO:journalctl@ceph.mds.j.smithi052.stdout:May 27 07:30:53 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-j[108368]: 14: (MDLog::_replay_thread()+0x1f38) [0x555dc773f48c] 2024-05-27T07:30:53.745 INFO:journalctl@ceph.mds.j.smithi052.stdout:May 27 07:30:53 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-j[108368]: 15: (MDLog::ReplayThread::entry()+0x11) [0x555dc77489cb] 2024-05-27T07:30:53.745 INFO:journalctl@ceph.mds.j.smithi052.stdout:May 27 07:30:53 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-j[108368]: 16: (Thread::entry_wrapper()+0x43) [0x7fb969c9072d] 2024-05-27T07:30:53.745 INFO:journalctl@ceph.mds.j.smithi052.stdout:May 27 07:30:53 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-j[108368]: 17: (Thread::_entry_func(void*)+0xd) [0x7fb969c90749] 2024-05-27T07:30:53.745 INFO:journalctl@ceph.mds.j.smithi052.stdout:May 27 07:30:53 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-j[108368]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7fb9688341ca] 2024-05-27T07:30:53.746 INFO:journalctl@ceph.mds.j.smithi052.stdout:May 27 07:30:53 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-j[108368]: 19: clone() 2024-05-27T07:30:53.746 INFO:journalctl@ceph.mds.j.smithi052.stdout:May 27 07:30:53 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-j[108368]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-05-27T07:30:53.746 INFO:journalctl@ceph.mds.j.smithi052.stdout:May 27 07:30:53 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-j[108368]: 2024-05-27T07:30:53.746 INFO:journalctl@ceph.mds.j.smithi052.stdout:May 27 07:30:53 smithi052 podman[108404]: 2024-05-27 07:30:53.640236783 +0000 UTC m=+0.112067842 container died 1797abb68ea79dde47fc28d12c9cdbd9da61ed99aa86f51e26ab091b595655a6 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-j, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.build-date=20240521) 2024-05-27T07:30:54.231 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:30:54 smithi052 ceph-mon[31955]: pgmap v174: 129 pgs: 129 active+clean; 403 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 3.0 MiB/s wr, 874 op/s 2024-05-27T07:30:54.231 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:30:54 smithi052 ceph-mon[31955]: osdmap e115: 12 total, 12 up, 12 in 2024-05-27T07:30:54.231 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:30:54 smithi052 ceph-mon[31955]: mds.? [v2:172.21.15.52:6834/134081853,v1:172.21.15.52:6835/134081853] up:boot 2024-05-27T07:30:54.231 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:30:54 smithi052 ceph-mon[31955]: fsmap cephfs:5 1 up:standby 5 up:active 2024-05-27T07:30:54.231 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:30:54 smithi052 ceph-mon[31955]: from='mgr.15333 172.21.15.52:0/416389082' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "j"}]: dispatch 2024-05-27T07:30:54.231 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:30:54 smithi052 ceph-mon[31955]: fsmap cephfs:5 1 up:standby-replay 5 up:active 2024-05-27T07:30:54.603 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:30:54 smithi173 ceph-mon[37000]: pgmap v174: 129 pgs: 129 active+clean; 403 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 3.0 MiB/s wr, 874 op/s 2024-05-27T07:30:54.604 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:30:54 smithi173 ceph-mon[37000]: osdmap e115: 12 total, 12 up, 12 in 2024-05-27T07:30:54.604 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:30:54 smithi173 ceph-mon[37000]: mds.? [v2:172.21.15.52:6834/134081853,v1:172.21.15.52:6835/134081853] up:boot 2024-05-27T07:30:54.604 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:30:54 smithi173 ceph-mon[37000]: fsmap cephfs:5 1 up:standby 5 up:active 2024-05-27T07:30:54.604 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:30:54 smithi173 ceph-mon[37000]: from='mgr.15333 172.21.15.52:0/416389082' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "j"}]: dispatch 2024-05-27T07:30:54.604 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:30:54 smithi173 ceph-mon[37000]: fsmap cephfs:5 1 up:standby-replay 5 up:active 2024-05-27T07:30:54.691 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:30:54 smithi105 ceph-mon[35977]: pgmap v174: 129 pgs: 129 active+clean; 403 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 3.0 MiB/s wr, 874 op/s 2024-05-27T07:30:54.692 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:30:54 smithi105 ceph-mon[35977]: osdmap e115: 12 total, 12 up, 12 in 2024-05-27T07:30:54.692 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:30:54 smithi105 ceph-mon[35977]: mds.? [v2:172.21.15.52:6834/134081853,v1:172.21.15.52:6835/134081853] up:boot 2024-05-27T07:30:54.692 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:30:54 smithi105 ceph-mon[35977]: fsmap cephfs:5 1 up:standby 5 up:active 2024-05-27T07:30:54.692 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:30:54 smithi105 ceph-mon[35977]: from='mgr.15333 172.21.15.52:0/416389082' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "j"}]: dispatch 2024-05-27T07:30:54.692 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:30:54 smithi105 ceph-mon[35977]: fsmap cephfs:5 1 up:standby-replay 5 up:active 2024-05-27T07:30:54.731 INFO:journalctl@ceph.mds.j.smithi052.stdout:May 27 07:30:54 smithi052 podman[108404]: 2024-05-27 07:30:54.288563334 +0000 UTC m=+0.760394379 container remove 1797abb68ea79dde47fc28d12c9cdbd9da61ed99aa86f51e26ab091b595655a6 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-j, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240521, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True) 2024-05-27T07:30:54.731 INFO:journalctl@ceph.mds.j.smithi052.stdout:May 27 07:30:54 smithi052 systemd[1]: ceph-73df2636-1bf8-11ef-bc9b-c7b262605968@mds.j.service: Main process exited, code=exited, status=134/n/a 2024-05-27T07:30:55.212 INFO:tasks.workunit.client.0.smithi052.stdout: 114 1508444 494 1071085 368 851224 1309 2024-05-27T07:30:55.234 INFO:journalctl@ceph.mds.j.smithi052.stdout:May 27 07:30:54 smithi052 systemd[1]: ceph-73df2636-1bf8-11ef-bc9b-c7b262605968@mds.j.service: Failed with result 'exit-code'. 2024-05-27T07:30:55.234 INFO:journalctl@ceph.mds.j.smithi052.stdout:May 27 07:30:54 smithi052 systemd[1]: ceph-73df2636-1bf8-11ef-bc9b-c7b262605968@mds.j.service: Consumed 1.230s CPU time. 2024-05-27T07:30:55.613 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:30:55.621+0000 7f52e7fff640 1 -- 172.21.15.52:0/2838898220 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f52e810a200 msgr2=0x7f52e810a690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:30:55.613 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:30:55.621+0000 7f52e7fff640 1 --2- 172.21.15.52:0/2838898220 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f52e810a200 0x7f52e810a690 secure :-1 s=READY pgs=649 cs=0 l=1 rev1=1 crypto rx=0x7f52d8009950 tx=0x7f52d802a480 comp rx=0 tx=0).stop 2024-05-27T07:30:55.626 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:30:55.634+0000 7f52e7fff640 1 -- 172.21.15.52:0/2838898220 shutdown_connections 2024-05-27T07:30:55.626 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:30:55.634+0000 7f52e7fff640 1 --2- 172.21.15.52:0/2838898220 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f52e810a200 0x7f52e810a690 unknown :-1 s=CLOSED pgs=649 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:30:55.627 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:30:55.634+0000 7f52e7fff640 1 --2- 172.21.15.52:0/2838898220 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f52e81103e0 0x7f52e8109cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:30:55.627 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:30:55.634+0000 7f52e7fff640 1 --2- 172.21.15.52:0/2838898220 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f52e810fa00 0x7f52e810fe10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:30:55.627 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:30:55.634+0000 7f52e7fff640 1 -- 172.21.15.52:0/2838898220 >> 172.21.15.52:0/2838898220 conn(0x7f52e806f800 msgr2=0x7f52e806fc10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:30:55.627 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:30:55.635+0000 7f52e7fff640 1 -- 172.21.15.52:0/2838898220 shutdown_connections 2024-05-27T07:30:55.627 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:30:55.635+0000 7f52e7fff640 1 -- 172.21.15.52:0/2838898220 wait complete. 2024-05-27T07:30:55.627 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:30:55.636+0000 7f52e7fff640 1 Processor -- start 2024-05-27T07:30:55.629 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:30:55.636+0000 7f52e7fff640 1 -- start start 2024-05-27T07:30:55.629 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:30:55.637+0000 7f52e7fff640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f52e810a200 0x7f52e81a95d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:30:55.629 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:30:55.637+0000 7f52e7fff640 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f52e810fa00 0x7f52e81a9b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:30:55.629 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:30:55.637+0000 7f52e7fff640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f52e81103e0 0x7f52e81addf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:30:55.629 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:30:55.637+0000 7f52e7fff640 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x7f52e8122b90 con 0x7f52e810fa00 2024-05-27T07:30:55.629 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:30:55.637+0000 7f52e7fff640 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f52e8122a10 con 0x7f52e810a200 2024-05-27T07:30:55.629 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:30:55.637+0000 7f52e7fff640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f52e8122d10 con 0x7f52e81103e0 2024-05-27T07:30:55.630 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:30:55.637+0000 7f52e6ffd640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f52e810a200 0x7f52e81a95d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:30:55.630 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:30:55.637+0000 7f52e6ffd640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f52e810a200 0x7f52e81a95d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.52:37324/0 (socket says 172.21.15.52:37324) 2024-05-27T07:30:55.630 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:30:55.637+0000 7f52e6ffd640 1 -- 172.21.15.52:0/1912007294 learned_addr learned my addr 172.21.15.52:0/1912007294 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-05-27T07:30:55.630 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:30:55.637+0000 7f52e67fc640 1 --2- 172.21.15.52:0/1912007294 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f52e810fa00 0x7f52e81a9b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:30:55.630 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:30:55.637+0000 7f52e77fe640 1 --2- 172.21.15.52:0/1912007294 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f52e81103e0 0x7f52e81addf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:30:55.630 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:30:55.638+0000 7f52e67fc640 1 -- 172.21.15.52:0/1912007294 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f52e81103e0 msgr2=0x7f52e81addf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:30:55.631 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:30:55.638+0000 7f52e67fc640 1 --2- 172.21.15.52:0/1912007294 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f52e81103e0 0x7f52e81addf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:30:55.631 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:30:55.638+0000 7f52e67fc640 1 -- 172.21.15.52:0/1912007294 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f52e810a200 msgr2=0x7f52e81a95d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:30:55.631 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:30:55.638+0000 7f52e67fc640 1 --2- 172.21.15.52:0/1912007294 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f52e810a200 0x7f52e81a95d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:30:55.631 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:30:55.638+0000 7f52e67fc640 1 -- 172.21.15.52:0/1912007294 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f52e81ae430 con 0x7f52e810fa00 2024-05-27T07:30:55.631 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:30:55.638+0000 7f52e77fe640 1 --2- 172.21.15.52:0/1912007294 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f52e81103e0 0x7f52e81addf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-27T07:30:55.631 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:30:55.638+0000 7f52e6ffd640 1 --2- 172.21.15.52:0/1912007294 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f52e810a200 0x7f52e81a95d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-27T07:30:55.631 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:30:55.638+0000 7f52e67fc640 1 --2- 172.21.15.52:0/1912007294 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f52e810fa00 0x7f52e81a9b30 secure :-1 s=READY pgs=650 cs=0 l=1 rev1=1 crypto rx=0x7f52dc00b750 tx=0x7f52dc00bc10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:30:55.641 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:30:55.649+0000 7f52d7fff640 1 -- 172.21.15.52:0/1912007294 <== mon.0 v2:172.21.15.52:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f52dc019070 con 0x7f52e810fa00 2024-05-27T07:30:55.647 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:30:55.649+0000 7f52e7fff640 1 -- 172.21.15.52:0/1912007294 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f52e81ae720 con 0x7f52e810fa00 2024-05-27T07:30:55.647 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:30:55.649+0000 7f52e7fff640 1 -- 172.21.15.52:0/1912007294 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f52e81120d0 con 0x7f52e810fa00 2024-05-27T07:30:55.649 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:30:55.657+0000 7f52f4268740 1 Processor -- start 2024-05-27T07:30:55.651 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:30:55.660+0000 7f52f4268740 1 -- start start 2024-05-27T07:30:55.654 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:30:55.663+0000 7f52f4268740 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x556b6a0d61b0 0x556b6a477590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:30:55.655 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:30:55.663+0000 7f52e77fe640 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x556b6a0d61b0 0x556b6a477590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:30:55.655 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:30:55.663+0000 7f52e77fe640 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x556b6a0d61b0 0x556b6a477590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.52:35444/0 (socket says 172.21.15.52:35444) 2024-05-27T07:30:55.655 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:30:55.663+0000 7f52e77fe640 1 -- 172.21.15.52:0/1969272217 learned_addr learned my addr 172.21.15.52:0/1969272217 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-05-27T07:30:55.655 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:30:55.664+0000 7f52f4268740 1 --2- 172.21.15.52:0/1969272217 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x556b6a0ad2f0 0x556b6a477ad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:30:55.656 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:30:55.664+0000 7f52f4268740 1 --2- 172.21.15.52:0/1969272217 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x556b6a0b54d0 0x556b6a478010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:30:55.656 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:30:55.664+0000 7f52e6ffd640 1 --2- 172.21.15.52:0/1969272217 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x556b6a0ad2f0 0x556b6a477ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:30:55.657 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:30:55.664+0000 7f52f4268740 1 -- 172.21.15.52:0/1969272217 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x556b6a3e9c30 con 0x556b6a0d61b0 2024-05-27T07:30:55.657 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:30:55.664+0000 7f52f4268740 1 -- 172.21.15.52:0/1969272217 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x556b6a3d97d0 con 0x556b6a0ad2f0 2024-05-27T07:30:55.657 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:30:55.664+0000 7f52f4268740 1 -- 172.21.15.52:0/1969272217 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x556b6a3dd810 con 0x556b6a0b54d0 2024-05-27T07:30:55.657 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:30:55.664+0000 7f52e67fc640 1 --2- 172.21.15.52:0/1969272217 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x556b6a0b54d0 0x556b6a478010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:30:55.657 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:30:55.665+0000 7f52e6ffd640 1 -- 172.21.15.52:0/1969272217 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x556b6a0b54d0 msgr2=0x556b6a478010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:30:55.657 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:30:55.665+0000 7f52e6ffd640 1 --2- 172.21.15.52:0/1969272217 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x556b6a0b54d0 0x556b6a478010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:30:55.657 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:30:55.665+0000 7f52e6ffd640 1 -- 172.21.15.52:0/1969272217 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x556b6a0d61b0 msgr2=0x556b6a477590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:30:55.657 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:30:55.665+0000 7f52e6ffd640 1 --2- 172.21.15.52:0/1969272217 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x556b6a0d61b0 0x556b6a477590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:30:55.657 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:30:55.665+0000 7f52e6ffd640 1 -- 172.21.15.52:0/1969272217 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x556b6a0ee210 con 0x556b6a0ad2f0 2024-05-27T07:30:55.658 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:30:55.665+0000 7f52e77fe640 1 --2- 172.21.15.52:0/1969272217 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x556b6a0d61b0 0x556b6a477590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-27T07:30:55.658 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:30:55.665+0000 7f52e6ffd640 1 --2- 172.21.15.52:0/1969272217 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x556b6a0ad2f0 0x556b6a477ad0 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7f52e000e560 tx=0x7f52e0009790 comp rx=0 tx=0).ready entity=mon.2 client_cookie=4d14d40770867655 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:30:55.658 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:30:55.666+0000 7f52d57fa640 1 -- 172.21.15.52:0/1969272217 <== mon.2 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f52e0017020 con 0x556b6a0ad2f0 2024-05-27T07:30:55.658 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:30:55.666+0000 7f52d57fa640 1 -- 172.21.15.52:0/1969272217 <== mon.2 v2:172.21.15.105:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f52e0013d80 con 0x556b6a0ad2f0 2024-05-27T07:30:55.658 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:30:55.666+0000 7f52d57fa640 1 -- 172.21.15.52:0/1969272217 <== mon.2 v2:172.21.15.105:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f52e0012360 con 0x556b6a0ad2f0 2024-05-27T07:30:55.659 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:30:55.668+0000 7f52f4268740 1 -- 172.21.15.52:0/1969272217 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x556b6a0ad2f0 msgr2=0x556b6a477ad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:30:55.659 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:30:55.668+0000 7f52f4268740 1 --2- 172.21.15.52:0/1969272217 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x556b6a0ad2f0 0x556b6a477ad0 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7f52e000e560 tx=0x7f52e0009790 comp rx=0 tx=0).stop 2024-05-27T07:30:55.663 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:30:55.671+0000 7f52f4268740 1 -- 172.21.15.52:0/1969272217 shutdown_connections 2024-05-27T07:30:55.663 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:30:55.671+0000 7f52f4268740 1 --2- 172.21.15.52:0/1969272217 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x556b6a0b54d0 0x556b6a478010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:30:55.663 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:30:55.671+0000 7f52f4268740 1 --2- 172.21.15.52:0/1969272217 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x556b6a0ad2f0 0x556b6a477ad0 unknown :-1 s=CLOSED pgs=170 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:30:55.663 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:30:55.671+0000 7f52f4268740 1 --2- 172.21.15.52:0/1969272217 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x556b6a0d61b0 0x556b6a477590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:30:55.663 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:30:55.671+0000 7f52f4268740 1 -- 172.21.15.52:0/1969272217 >> 172.21.15.52:0/1969272217 conn(0x556b6a0ca300 msgr2=0x556b6a473d40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:30:55.665 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:30:55.674+0000 7f52d7fff640 1 -- 172.21.15.52:0/1912007294 <== mon.0 v2:172.21.15.52:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f52dc00cbd0 con 0x7f52e810fa00 2024-05-27T07:30:55.665 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:30:55.674+0000 7f52d7fff640 1 -- 172.21.15.52:0/1912007294 <== mon.0 v2:172.21.15.52:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f52dc014da0 con 0x7f52e810fa00 2024-05-27T07:30:55.667 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:30:55.674+0000 7f52d7fff640 1 -- 172.21.15.52:0/1912007294 <== mon.0 v2:172.21.15.52:3300/0 4 ==== mgrmap(e 26) ==== 149783+0+0 (secure 0 0 0) 0x7f52dc01c070 con 0x7f52e810fa00 2024-05-27T07:30:55.667 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:30:55.675+0000 7f52d7fff640 1 --2- 172.21.15.52:0/1912007294 >> [v2:172.21.15.52:6800/1822318384,v1:172.21.15.52:6801/1822318384] conn(0x7f52d00b2aa0 0x7f52d00b4fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:30:55.671 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:30:55.679+0000 7f52e6ffd640 1 --2- 172.21.15.52:0/1912007294 >> [v2:172.21.15.52:6800/1822318384,v1:172.21.15.52:6801/1822318384] conn(0x7f52d00b2aa0 0x7f52d00b4fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:30:55.672 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:30:55.680+0000 7f52f4268740 1 -- 172.21.15.52:0/1969272217 shutdown_connections 2024-05-27T07:30:55.673 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:30:55.681+0000 7f52e6ffd640 1 --2- 172.21.15.52:0/1912007294 >> [v2:172.21.15.52:6800/1822318384,v1:172.21.15.52:6801/1822318384] conn(0x7f52d00b2aa0 0x7f52d00b4fb0 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7f52e0007930 tx=0x7f52e0007a30 comp rx=0 tx=0).ready entity=mgr.15333 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:30:55.673 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:30:55.681+0000 7f52f4268740 1 -- 172.21.15.52:0/1969272217 wait complete. 2024-05-27T07:30:55.673 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:30:55.682+0000 7f52f4268740 1 -- start start 2024-05-27T07:30:55.674 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:30:55.682+0000 7f52f4268740 1 Processor -- start 2024-05-27T07:30:55.674 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:30:55.682+0000 7f52d7fff640 1 -- 172.21.15.52:0/1912007294 <== mon.0 v2:172.21.15.52:3300/0 5 ==== osd_map(115..115 src has 1..115) ==== 9379+0+0 (secure 0 0 0) 0x7f52dc0e2410 con 0x7f52e810fa00 2024-05-27T07:30:55.675 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:30:55.683+0000 7f52f4268740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x556b6a0ad2f0 0x556b6a481c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:30:55.675 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:30:55.683+0000 7f52f4268740 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x556b6a0b54d0 0x556b6a482180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:30:55.675 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:30:55.683+0000 7f52f4268740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x556b6a0d61b0 0x556b6a487a90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:30:55.676 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:30:55.684+0000 7f52f4268740 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x556b6a0bf790 con 0x556b6a0b54d0 2024-05-27T07:30:55.676 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:30:55.684+0000 7f52e77fe640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x556b6a0d61b0 0x556b6a487a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:30:55.676 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:30:55.684+0000 7f52e77fe640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x556b6a0d61b0 0x556b6a487a90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.52:37342/0 (socket says 172.21.15.52:37342) 2024-05-27T07:30:55.676 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:30:55.684+0000 7f52e77fe640 1 -- 172.21.15.52:0/1560737730 learned_addr learned my addr 172.21.15.52:0/1560737730 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-05-27T07:30:55.676 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:30:55.684+0000 7f52e67fc640 1 --2- 172.21.15.52:0/1560737730 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x556b6a0b54d0 0x556b6a482180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:30:55.676 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:30:55.684+0000 7f52f4268740 1 -- 172.21.15.52:0/1560737730 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x556b6a395890 con 0x556b6a0d61b0 2024-05-27T07:30:55.676 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:30:55.684+0000 7f52f4268740 1 -- 172.21.15.52:0/1560737730 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x556b6a4846c0 con 0x556b6a0ad2f0 2024-05-27T07:30:55.677 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:30:55.684+0000 7f52f4268740 10 client.0 _ll_register_callbacks cb 0x556b6a0eda00 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T07:30:55.677 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:30:55.685+0000 7f52e77fe640 1 -- 172.21.15.52:0/1560737730 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x556b6a0ad2f0 msgr2=0x556b6a481c40 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-05-27T07:30:55.677 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:30:55.685+0000 7f52e77fe640 1 --2- 172.21.15.52:0/1560737730 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x556b6a0ad2f0 0x556b6a481c40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:30:55.677 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:30:55.685+0000 7f52e77fe640 1 -- 172.21.15.52:0/1560737730 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x556b6a0b54d0 msgr2=0x556b6a482180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:30:55.677 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:30:55.685+0000 7f52e77fe640 1 --2- 172.21.15.52:0/1560737730 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x556b6a0b54d0 0x556b6a482180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:30:55.677 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:30:55.685+0000 7f52e77fe640 1 -- 172.21.15.52:0/1560737730 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({osdmap=0}) -- 0x556b6a1ba620 con 0x556b6a0d61b0 2024-05-27T07:30:55.677 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:30:55.685+0000 7f52e67fc640 1 --2- 172.21.15.52:0/1560737730 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x556b6a0b54d0 0x556b6a482180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-27T07:30:55.678 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:30:55.685+0000 7f52e77fe640 1 --2- 172.21.15.52:0/1560737730 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x556b6a0d61b0 0x556b6a487a90 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7f52d802aa70 tx=0x7f52d8003b30 comp rx=0 tx=0).ready entity=mon.2 client_cookie=e4f460a1bc1a502b server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:30:55.678 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:30:55.686+0000 7f52ce7fc640 10 client.0 ms_handle_connect on v2:172.21.15.105:3300/0 2024-05-27T07:30:55.678 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:30:55.686+0000 7f52ce7fc640 1 -- 172.21.15.52:0/1560737730 <== mon.2 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f52d8035040 con 0x556b6a0d61b0 2024-05-27T07:30:55.679 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:30:55.686+0000 7f52f4268740 1 -- 172.21.15.52:0/1560737730 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f52e0013d80 con 0x556b6a0d61b0 2024-05-27T07:30:55.679 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:30:55.686+0000 7f52ce7fc640 1 -- 172.21.15.52:0/1560737730 <== mon.2 v2:172.21.15.105:3300/0 2 ==== osd_map(115..115 src has 1..115) ==== 9379+0+0 (secure 0 0 0) 0x7f52d8005220 con 0x556b6a0d61b0 2024-05-27T07:30:55.679 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:30:55.687+0000 7f52ce7fc640 1 -- 172.21.15.52:0/1560737730 <== mon.2 v2:172.21.15.105:3300/0 3 ==== mon_get_version_reply(handle=1 version=95) ==== 24+0+0 (secure 0 0 0) 0x7f52d804e050 con 0x556b6a0d61b0 2024-05-27T07:30:55.679 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:30:55.687+0000 7f52f4268740 10 client.26207 fetch_fsmap learned FSMap version 95 2024-05-27T07:30:55.679 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:30:55.687+0000 7f52f4268740 1 -- 172.21.15.52:0/1560737730 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,fsmap=95,monmap=4+}) -- 0x556b6a490a20 con 0x556b6a0d61b0 2024-05-27T07:30:55.679 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:30:55.687+0000 7f52ce7fc640 1 -- 172.21.15.52:0/1560737730 <== mon.2 v2:172.21.15.105:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f52d8003e30 con 0x556b6a0d61b0 2024-05-27T07:30:55.681 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:30:55.689+0000 7f52ce7fc640 1 -- 172.21.15.52:0/1560737730 <== mon.2 v2:172.21.15.105:3300/0 5 ==== fsmap(e 95) ==== 4792+0+0 (secure 0 0 0) 0x7f52d80316d0 con 0x556b6a0d61b0 2024-05-27T07:30:55.681 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:30:55.689+0000 7f52f4268740 10 client.26207 fetch_fsmap finished waiting for FSMap version 95 2024-05-27T07:30:55.681 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:30:55.689+0000 7f52f4268740 10 client.26207 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-05-27T07:30:55.681 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:30:55.689+0000 7f52f4268740 20 client.26207 populate_metadata read hostname 'smithi052' 2024-05-27T07:30:55.681 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:30:55.689+0000 7f52f4268740 1 --2- 172.21.15.52:0/1560737730 >> [v2:172.21.15.105:6838/500348151,v1:172.21.15.105:6839/500348151] conn(0x556b6a094d80 0x556b6a494470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:30:55.681 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:30:55.689+0000 7f52f4268740 4 client.26207 mds_command: new command op to 24527 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-27T07:30:55.681 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:30:55.689+0000 7f52f4268740 1 -- 172.21.15.52:0/1560737730 --> [v2:172.21.15.105:6838/500348151,v1:172.21.15.105:6839/500348151] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x7f52e0013d80 con 0x556b6a094d80 2024-05-27T07:30:55.682 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:30:55.690+0000 7f52e6ffd640 1 --2- 172.21.15.52:0/1560737730 >> [v2:172.21.15.105:6838/500348151,v1:172.21.15.105:6839/500348151] conn(0x556b6a094d80 0x556b6a494470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:30:55.682 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:30:55.690+0000 7f52e6ffd640 1 --2- 172.21.15.52:0/1560737730 >> [v2:172.21.15.105:6838/500348151,v1:172.21.15.105:6839/500348151] conn(0x556b6a094d80 0x556b6a494470 crc :-1 s=READY pgs=111 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=f226d8eb603f0119 server_cookie=2cd786213ff059d5 in_seq=0 out_seq=0 2024-05-27T07:30:55.682 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:30:55.691+0000 7f52ce7fc640 10 client.26207 ms_handle_connect on v2:172.21.15.105:6838/500348151 2024-05-27T07:30:55.685 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:30:55.693+0000 7f52ce7fc640 1 -- 172.21.15.52:0/1560737730 <== mds.0 v2:172.21.15.105:6838/500348151 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24041 (crc 0 0 0) 0x7f52e0013d80 con 0x556b6a094d80 2024-05-27T07:30:55.685 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:30:55.693+0000 7f52ce7fc640 10 client.26207 handle_command_reply: tid=0 2024-05-27T07:30:55.685 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:30:55.694+0000 7f52f4268740 1 client.26207 shutdown 2024-05-27T07:30:55.689 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:30:55.697+0000 7f52f4268740 1 -- 172.21.15.52:0/1560737730 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x556b6a0d61b0 msgr2=0x556b6a487a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:30:55.689 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:30:55.697+0000 7f52f4268740 1 --2- 172.21.15.52:0/1560737730 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x556b6a0d61b0 0x556b6a487a90 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7f52d802aa70 tx=0x7f52d8003b30 comp rx=0 tx=0).stop 2024-05-27T07:30:55.690 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:30:55.698+0000 7f52f4268740 1 -- 172.21.15.52:0/1560737730 shutdown_connections 2024-05-27T07:30:55.690 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:30:55.698+0000 7f52f4268740 1 --2- 172.21.15.52:0/1560737730 >> [v2:172.21.15.105:6838/500348151,v1:172.21.15.105:6839/500348151] conn(0x556b6a094d80 0x556b6a494470 crc :-1 s=READY pgs=111 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:30:55.690 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:30:55.698+0000 7f52f4268740 1 --2- 172.21.15.52:0/1560737730 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x556b6a0d61b0 0x556b6a487a90 unknown :-1 s=CLOSED pgs=171 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:30:55.690 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:30:55.698+0000 7f52f4268740 1 --2- 172.21.15.52:0/1560737730 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x556b6a0b54d0 0x556b6a482180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:30:55.690 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:30:55.698+0000 7f52f4268740 1 --2- 172.21.15.52:0/1560737730 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x556b6a0ad2f0 0x556b6a481c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:30:55.690 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:30:55.698+0000 7f52f4268740 1 -- 172.21.15.52:0/1560737730 >> 172.21.15.52:0/1560737730 conn(0x556b6a0ca300 msgr2=0x556b6a4792b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:30:55.690 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:30:55.698+0000 7f52ce7fc640 0 client.26207 ms_handle_reset on v2:172.21.15.105:6838/500348151 2024-05-27T07:30:55.690 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:30:55.698+0000 7f52f4268740 1 -- 172.21.15.52:0/1560737730 shutdown_connections 2024-05-27T07:30:55.691 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:30:55.698+0000 7f52f4268740 1 -- 172.21.15.52:0/1560737730 wait complete. 2024-05-27T07:30:55.691 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:30:55.698+0000 7f52f4268740 20 client.26207 trim_cache size 0 max 16384 2024-05-27T07:30:55.724 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:30:55.732+0000 7f52f4268740 1 Processor -- start 2024-05-27T07:30:55.724 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:30:55.732+0000 7f52f4268740 1 -- start start 2024-05-27T07:30:55.725 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:30:55.733+0000 7f52f4268740 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x556b6a0ad2f0 0x556b6a481680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:30:55.726 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:30:55.734+0000 7f52e6ffd640 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x556b6a0ad2f0 0x556b6a481680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:30:55.726 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:30:55.734+0000 7f52e6ffd640 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x556b6a0ad2f0 0x556b6a481680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.52:35460/0 (socket says 172.21.15.52:35460) 2024-05-27T07:30:55.726 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:30:55.734+0000 7f52e6ffd640 1 -- 172.21.15.52:0/1632345851 learned_addr learned my addr 172.21.15.52:0/1632345851 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-05-27T07:30:55.726 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:30:55.734+0000 7f52f4268740 1 --2- 172.21.15.52:0/1632345851 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x556b6a0b54d0 0x556b6a480970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:30:55.726 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:30:55.734+0000 7f52f4268740 1 --2- 172.21.15.52:0/1632345851 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x556b6a0d61b0 0x556b6a487a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:30:55.727 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:30:55.735+0000 7f52e67fc640 1 --2- 172.21.15.52:0/1632345851 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x556b6a0b54d0 0x556b6a480970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:30:55.727 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:30:55.735+0000 7f52e77fe640 1 --2- 172.21.15.52:0/1632345851 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x556b6a0d61b0 0x556b6a487a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:30:55.727 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:30:55.735+0000 7f52f4268740 1 -- 172.21.15.52:0/1632345851 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x7f52c8010880 con 0x556b6a0ad2f0 2024-05-27T07:30:55.727 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:30:55.735+0000 7f52f4268740 1 -- 172.21.15.52:0/1632345851 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f52c800e5e0 con 0x556b6a0d61b0 2024-05-27T07:30:55.727 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:30:55.735+0000 7f52f4268740 1 -- 172.21.15.52:0/1632345851 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f52c80101b0 con 0x556b6a0b54d0 2024-05-27T07:30:55.728 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:30:55.736+0000 7f52e6ffd640 1 -- 172.21.15.52:0/1632345851 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x556b6a0b54d0 msgr2=0x556b6a480970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:30:55.728 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:30:55.736+0000 7f52e6ffd640 1 --2- 172.21.15.52:0/1632345851 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x556b6a0b54d0 0x556b6a480970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:30:55.728 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:30:55.736+0000 7f52e6ffd640 1 -- 172.21.15.52:0/1632345851 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x556b6a0d61b0 msgr2=0x556b6a487a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:30:55.728 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:30:55.736+0000 7f52e6ffd640 1 --2- 172.21.15.52:0/1632345851 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x556b6a0d61b0 0x556b6a487a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:30:55.728 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:30:55.736+0000 7f52e6ffd640 1 -- 172.21.15.52:0/1632345851 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x556b6a478970 con 0x556b6a0ad2f0 2024-05-27T07:30:55.732 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:30:55.737+0000 7f52e6ffd640 1 --2- 172.21.15.52:0/1632345851 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x556b6a0ad2f0 0x556b6a481680 secure :-1 s=READY pgs=651 cs=0 l=1 rev1=1 crypto rx=0x7f52e0007a00 tx=0x7f52e001f040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=29c22d726f6368bb server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:30:55.732 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:30:55.740+0000 7f52d4ff9640 1 -- 172.21.15.52:0/1632345851 <== mon.0 v2:172.21.15.52:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f52e0017050 con 0x556b6a0ad2f0 2024-05-27T07:30:55.732 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:30:55.741+0000 7f52d4ff9640 1 -- 172.21.15.52:0/1632345851 <== mon.0 v2:172.21.15.52:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f52e0037070 con 0x556b6a0ad2f0 2024-05-27T07:30:55.736 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:30:55.744+0000 7f52f4268740 1 -- 172.21.15.52:0/1632345851 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x556b6a0ad2f0 msgr2=0x556b6a481680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:30:55.736 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:30:55.744+0000 7f52f4268740 1 --2- 172.21.15.52:0/1632345851 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x556b6a0ad2f0 0x556b6a481680 secure :-1 s=READY pgs=651 cs=0 l=1 rev1=1 crypto rx=0x7f52e0007a00 tx=0x7f52e001f040 comp rx=0 tx=0).stop 2024-05-27T07:30:55.737 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:30:55.745+0000 7f52f4268740 1 -- 172.21.15.52:0/1632345851 shutdown_connections 2024-05-27T07:30:55.737 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:30:55.745+0000 7f52f4268740 1 --2- 172.21.15.52:0/1632345851 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x556b6a0d61b0 0x556b6a487a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:30:55.737 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:30:55.745+0000 7f52f4268740 1 --2- 172.21.15.52:0/1632345851 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x556b6a0b54d0 0x556b6a480970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:30:55.737 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:30:55.745+0000 7f52f4268740 1 --2- 172.21.15.52:0/1632345851 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x556b6a0ad2f0 0x556b6a481680 unknown :-1 s=CLOSED pgs=651 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:30:55.737 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:30:55.745+0000 7f52f4268740 1 -- 172.21.15.52:0/1632345851 >> 172.21.15.52:0/1632345851 conn(0x556b6a0ca300 msgr2=0x556b6a480f70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:30:55.737 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:30:55.745+0000 7f52f4268740 1 -- 172.21.15.52:0/1632345851 shutdown_connections 2024-05-27T07:30:55.737 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:30:55.745+0000 7f52f4268740 1 -- 172.21.15.52:0/1632345851 wait complete. 2024-05-27T07:30:55.738 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:30:55.746+0000 7f52f4268740 1 -- start start 2024-05-27T07:30:55.738 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:30:55.746+0000 7f52f4268740 1 Processor -- start 2024-05-27T07:30:55.739 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:30:55.746+0000 7f52f4268740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x556b6a0ad2f0 0x556b6a48e1e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:30:55.739 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:30:55.747+0000 7f52f4268740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x556b6a0b54d0 0x556b6a488520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:30:55.739 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:30:55.747+0000 7f52f4268740 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x556b6a0d61b0 0x556b6a488a60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:30:55.739 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:30:55.747+0000 7f52f4268740 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x7f52c800f380 con 0x556b6a0d61b0 2024-05-27T07:30:55.739 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:30:55.747+0000 7f52f4268740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f52c800ecb0 con 0x556b6a0ad2f0 2024-05-27T07:30:55.739 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:30:55.747+0000 7f52f4268740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f52c800fa50 con 0x556b6a0b54d0 2024-05-27T07:30:55.740 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:30:55.747+0000 7f52f4268740 10 client.0 _ll_register_callbacks cb 0x7f52d8009640 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T07:30:55.740 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:30:55.747+0000 7f52e6ffd640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x556b6a0ad2f0 0x556b6a48e1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:30:55.740 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:30:55.747+0000 7f52e6ffd640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x556b6a0ad2f0 0x556b6a48e1e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.52:37360/0 (socket says 172.21.15.52:37360) 2024-05-27T07:30:55.740 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:30:55.747+0000 7f52e6ffd640 1 -- 172.21.15.52:0/2935442986 learned_addr learned my addr 172.21.15.52:0/2935442986 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-05-27T07:30:55.740 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:30:55.747+0000 7f52e77fe640 1 --2- 172.21.15.52:0/2935442986 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x556b6a0d61b0 0x556b6a488a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:30:55.740 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:30:55.748+0000 7f52e6ffd640 1 -- 172.21.15.52:0/2935442986 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x556b6a0b54d0 msgr2=0x556b6a488520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:30:55.740 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:30:55.748+0000 7f52e6ffd640 1 --2- 172.21.15.52:0/2935442986 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x556b6a0b54d0 0x556b6a488520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:30:55.740 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:30:55.748+0000 7f52e6ffd640 1 -- 172.21.15.52:0/2935442986 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x556b6a0d61b0 msgr2=0x556b6a488a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:30:55.742 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:30:55.748+0000 7f52e6ffd640 1 --2- 172.21.15.52:0/2935442986 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x556b6a0d61b0 0x556b6a488a60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:30:55.742 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:30:55.749+0000 7f52e6ffd640 1 -- 172.21.15.52:0/2935442986 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({osdmap=0}) -- 0x556b6a488fa0 con 0x556b6a0ad2f0 2024-05-27T07:30:55.742 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:30:55.749+0000 7f52e77fe640 1 --2- 172.21.15.52:0/2935442986 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x556b6a0d61b0 0x556b6a488a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-27T07:30:55.744 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:30:55.752+0000 7f52e6ffd640 1 --2- 172.21.15.52:0/2935442986 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x556b6a0ad2f0 0x556b6a48e1e0 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7f52e0007980 tx=0x7f52e0009710 comp rx=0 tx=0).ready entity=mon.2 client_cookie=13ecd749f4a16f7b server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:30:55.744 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:30:55.753+0000 7f52ce7fc640 10 client.0 ms_handle_connect on v2:172.21.15.105:3300/0 2024-05-27T07:30:55.745 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:30:55.753+0000 7f52ce7fc640 1 -- 172.21.15.52:0/2935442986 <== mon.2 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f52e0032320 con 0x556b6a0ad2f0 2024-05-27T07:30:55.745 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:30:55.753+0000 7f52f4268740 1 -- 172.21.15.52:0/2935442986 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f52e0037070 con 0x556b6a0ad2f0 2024-05-27T07:30:55.745 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:30:55.753+0000 7f52ce7fc640 1 -- 172.21.15.52:0/2935442986 <== mon.2 v2:172.21.15.105:3300/0 2 ==== osd_map(115..115 src has 1..115) ==== 9379+0+0 (secure 0 0 0) 0x7f52e0030070 con 0x556b6a0ad2f0 2024-05-27T07:30:55.746 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:30:55.754+0000 7f52ce7fc640 1 -- 172.21.15.52:0/2935442986 <== mon.2 v2:172.21.15.105:3300/0 3 ==== mon_get_version_reply(handle=1 version=95) ==== 24+0+0 (secure 0 0 0) 0x7f52e0013790 con 0x556b6a0ad2f0 2024-05-27T07:30:55.746 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:30:55.754+0000 7f52f4268740 10 client.26213 fetch_fsmap learned FSMap version 95 2024-05-27T07:30:55.746 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:30:55.754+0000 7f52f4268740 1 -- 172.21.15.52:0/2935442986 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,fsmap=95,monmap=4+}) -- 0x556b6a4fb6c0 con 0x556b6a0ad2f0 2024-05-27T07:30:55.747 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:30:55.754+0000 7f52ce7fc640 1 -- 172.21.15.52:0/2935442986 <== mon.2 v2:172.21.15.105:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f52e0037070 con 0x556b6a0ad2f0 2024-05-27T07:30:55.747 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:30:55.754+0000 7f52ce7fc640 1 -- 172.21.15.52:0/2935442986 <== mon.2 v2:172.21.15.105:3300/0 5 ==== fsmap(e 95) ==== 4792+0+0 (secure 0 0 0) 0x7f52e003bb20 con 0x556b6a0ad2f0 2024-05-27T07:30:55.747 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:30:55.754+0000 7f52f4268740 10 client.26213 fetch_fsmap finished waiting for FSMap version 95 2024-05-27T07:30:55.747 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:30:55.755+0000 7f52f4268740 10 client.26213 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-05-27T07:30:55.747 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:30:55.755+0000 7f52f4268740 20 client.26213 populate_metadata read hostname 'smithi052' 2024-05-27T07:30:55.747 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:30:55.755+0000 7f52f4268740 1 --2- 172.21.15.52:0/2935442986 >> [v2:172.21.15.105:6838/500348151,v1:172.21.15.105:6839/500348151] conn(0x556b6a06a1c0 0x556b6a4fddc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:30:55.747 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:30:55.755+0000 7f52f4268740 4 client.26213 mds_command: new command op to 24527 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-05-27T07:30:55.747 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:30:55.755+0000 7f52f4268740 1 -- 172.21.15.52:0/2935442986 --> [v2:172.21.15.105:6838/500348151,v1:172.21.15.105:6839/500348151] -- command(tid 0: {"prefix": "flush journal"}) -- 0x556b6a3ebb80 con 0x556b6a06a1c0 2024-05-27T07:30:55.747 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:30:55.755+0000 7f52e67fc640 1 --2- 172.21.15.52:0/2935442986 >> [v2:172.21.15.105:6838/500348151,v1:172.21.15.105:6839/500348151] conn(0x556b6a06a1c0 0x556b6a4fddc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:30:55.748 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:30:55.755+0000 7f52ce7fc640 10 client.26213 ms_handle_connect on v2:172.21.15.105:6838/500348151 2024-05-27T07:30:55.748 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:30:55.756+0000 7f52e67fc640 1 --2- 172.21.15.52:0/2935442986 >> [v2:172.21.15.105:6838/500348151,v1:172.21.15.105:6839/500348151] conn(0x556b6a06a1c0 0x556b6a4fddc0 crc :-1 s=READY pgs=112 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=c377b3204146803d server_cookie=54ff881854833e5e in_seq=0 out_seq=0 2024-05-27T07:30:56.481 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:30:56 smithi052 ceph-mon[31955]: pgmap v176: 129 pgs: 129 active+clean; 410 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 2.2 MiB/s wr, 471 op/s 2024-05-27T07:30:56.603 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:30:56 smithi173 ceph-mon[37000]: pgmap v176: 129 pgs: 129 active+clean; 410 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 2.2 MiB/s wr, 471 op/s 2024-05-27T07:30:56.691 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:30:56 smithi105 ceph-mon[35977]: pgmap v176: 129 pgs: 129 active+clean; 410 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 2.2 MiB/s wr, 471 op/s 2024-05-27T07:30:58.481 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:30:58 smithi052 ceph-mon[31955]: pgmap v177: 129 pgs: 129 active+clean; 414 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 MiB/s rd, 2.0 MiB/s wr, 345 op/s 2024-05-27T07:30:58.603 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:30:58 smithi173 ceph-mon[37000]: pgmap v177: 129 pgs: 129 active+clean; 414 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 MiB/s rd, 2.0 MiB/s wr, 345 op/s 2024-05-27T07:30:58.691 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:30:58 smithi105 ceph-mon[35977]: pgmap v177: 129 pgs: 129 active+clean; 414 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 MiB/s rd, 2.0 MiB/s wr, 345 op/s 2024-05-27T07:31:00.603 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:31:00 smithi173 ceph-mon[37000]: pgmap v178: 129 pgs: 129 active+clean; 415 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 MiB/s rd, 2.4 MiB/s wr, 787 op/s 2024-05-27T07:31:00.667 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:31:00.674+0000 7f52ce7fc640 1 -- 172.21.15.52:0/2935442986 <== mds.0 v2:172.21.15.105:6838/500348151 1 ==== command_reply(tid 0: 0 ) ==== 8+0+25 (crc 0 0 0) 0x556b6a3ebb80 con 0x556b6a06a1c0 2024-05-27T07:31:00.667 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:31:00.674+0000 7f52ce7fc640 10 client.26213 handle_command_reply: tid=0 2024-05-27T07:31:00.667 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:31:00.674+0000 7f52f4268740 1 client.26213 shutdown 2024-05-27T07:31:00.667 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:31:00.674+0000 7f52f4268740 1 -- 172.21.15.52:0/2935442986 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x556b6a0ad2f0 msgr2=0x556b6a48e1e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:31:00.667 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:31:00.674+0000 7f52f4268740 1 --2- 172.21.15.52:0/2935442986 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x556b6a0ad2f0 0x556b6a48e1e0 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7f52e0007980 tx=0x7f52e0009710 comp rx=0 tx=0).stop 2024-05-27T07:31:00.667 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:31:00.674+0000 7f52f4268740 1 -- 172.21.15.52:0/2935442986 shutdown_connections 2024-05-27T07:31:00.668 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:31:00.674+0000 7f52f4268740 1 --2- 172.21.15.52:0/2935442986 >> [v2:172.21.15.105:6838/500348151,v1:172.21.15.105:6839/500348151] conn(0x556b6a06a1c0 0x556b6a4fddc0 crc :-1 s=READY pgs=112 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:31:00.668 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:31:00.674+0000 7f52f4268740 1 --2- 172.21.15.52:0/2935442986 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x556b6a0d61b0 0x556b6a488a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:31:00.668 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:31:00.674+0000 7f52f4268740 1 --2- 172.21.15.52:0/2935442986 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x556b6a0b54d0 0x556b6a488520 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:31:00.668 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:31:00.674+0000 7f52f4268740 1 --2- 172.21.15.52:0/2935442986 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x556b6a0ad2f0 0x556b6a48e1e0 unknown :-1 s=CLOSED pgs=172 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:31:00.668 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:31:00.674+0000 7f52f4268740 1 -- 172.21.15.52:0/2935442986 >> 172.21.15.52:0/2935442986 conn(0x556b6a0ca300 msgr2=0x556b6a483000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:31:00.668 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:31:00.674+0000 7f52ce7fc640 0 client.26213 ms_handle_reset on v2:172.21.15.105:6838/500348151 2024-05-27T07:31:00.668 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:31:00.675+0000 7f52f4268740 1 -- 172.21.15.52:0/2935442986 shutdown_connections 2024-05-27T07:31:00.668 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:31:00.675+0000 7f52f4268740 1 -- 172.21.15.52:0/2935442986 wait complete. 2024-05-27T07:31:00.668 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:31:00.675+0000 7f52f4268740 20 client.26213 trim_cache size 0 max 16384 2024-05-27T07:31:00.672 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stdout:{ 2024-05-27T07:31:00.672 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stdout: "return_code": 0 2024-05-27T07:31:00.672 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stdout:} 2024-05-27T07:31:00.682 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:31:00.690+0000 7f52e7fff640 1 -- 172.21.15.52:0/1912007294 >> [v2:172.21.15.52:6800/1822318384,v1:172.21.15.52:6801/1822318384] conn(0x7f52d00b2aa0 msgr2=0x7f52d00b4fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:31:00.682 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:31:00.690+0000 7f52e7fff640 1 --2- 172.21.15.52:0/1912007294 >> [v2:172.21.15.52:6800/1822318384,v1:172.21.15.52:6801/1822318384] conn(0x7f52d00b2aa0 0x7f52d00b4fb0 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7f52e0007930 tx=0x7f52e0007a30 comp rx=0 tx=0).stop 2024-05-27T07:31:00.682 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:31:00.690+0000 7f52e7fff640 1 -- 172.21.15.52:0/1912007294 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f52e810fa00 msgr2=0x7f52e81a9b30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:31:00.683 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:31:00.690+0000 7f52e7fff640 1 --2- 172.21.15.52:0/1912007294 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f52e810fa00 0x7f52e81a9b30 secure :-1 s=READY pgs=650 cs=0 l=1 rev1=1 crypto rx=0x7f52dc00b750 tx=0x7f52dc00bc10 comp rx=0 tx=0).stop 2024-05-27T07:31:00.686 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:31:00.693+0000 7f52e7fff640 1 -- 172.21.15.52:0/1912007294 shutdown_connections 2024-05-27T07:31:00.686 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:31:00.693+0000 7f52e7fff640 1 --2- 172.21.15.52:0/1912007294 >> [v2:172.21.15.52:6800/1822318384,v1:172.21.15.52:6801/1822318384] conn(0x7f52d00b2aa0 0x7f52d00b4fb0 unknown :-1 s=CLOSED pgs=142 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:31:00.686 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:31:00.693+0000 7f52e7fff640 1 --2- 172.21.15.52:0/1912007294 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f52e81103e0 0x7f52e81addf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:31:00.686 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:31:00.694+0000 7f52e7fff640 1 --2- 172.21.15.52:0/1912007294 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f52e810fa00 0x7f52e81a9b30 unknown :-1 s=CLOSED pgs=650 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:31:00.686 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:31:00.694+0000 7f52e7fff640 1 --2- 172.21.15.52:0/1912007294 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f52e810a200 0x7f52e81a95d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:31:00.686 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:31:00.694+0000 7f52e7fff640 1 -- 172.21.15.52:0/1912007294 >> 172.21.15.52:0/1912007294 conn(0x7f52e806f800 msgr2=0x7f52e810d2a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:31:00.686 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:31:00.694+0000 7f52e7fff640 1 -- 172.21.15.52:0/1912007294 shutdown_connections 2024-05-27T07:31:00.686 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:31:00.694+0000 7f52e7fff640 1 -- 172.21.15.52:0/1912007294 wait complete. 2024-05-27T07:31:00.692 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:31:00 smithi105 ceph-mon[35977]: pgmap v178: 129 pgs: 129 active+clean; 415 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 MiB/s rd, 2.4 MiB/s wr, 787 op/s 2024-05-27T07:31:00.731 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:31:00 smithi052 ceph-mon[31955]: pgmap v178: 129 pgs: 129 active+clean; 415 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 MiB/s rd, 2.4 MiB/s wr, 787 op/s 2024-05-27T07:31:02.604 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:31:02 smithi173 ceph-mon[37000]: pgmap v179: 129 pgs: 129 active+clean; 415 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 MiB/s rd, 2.3 MiB/s wr, 1.22k op/s 2024-05-27T07:31:02.691 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:31:02 smithi105 ceph-mon[35977]: pgmap v179: 129 pgs: 129 active+clean; 415 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 MiB/s rd, 2.3 MiB/s wr, 1.22k op/s 2024-05-27T07:31:02.731 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:31:02 smithi052 ceph-mon[31955]: pgmap v179: 129 pgs: 129 active+clean; 415 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 MiB/s rd, 2.3 MiB/s wr, 1.22k op/s 2024-05-27T07:31:03.603 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:31:03 smithi173 ceph-mon[37000]: fsmap cephfs:5 1 up:standby-replay 5 up:active 2024-05-27T07:31:03.692 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:31:03 smithi105 ceph-mon[35977]: fsmap cephfs:5 1 up:standby-replay 5 up:active 2024-05-27T07:31:03.731 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:31:03 smithi052 ceph-mon[31955]: fsmap cephfs:5 1 up:standby-replay 5 up:active 2024-05-27T07:31:04.604 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:31:04 smithi173 ceph-mon[37000]: pgmap v180: 129 pgs: 129 active+clean; 407 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 MiB/s rd, 2.7 MiB/s wr, 1.45k op/s 2024-05-27T07:31:04.692 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:31:04 smithi105 ceph-mon[35977]: pgmap v180: 129 pgs: 129 active+clean; 407 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 MiB/s rd, 2.7 MiB/s wr, 1.45k op/s 2024-05-27T07:31:04.731 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:31:04 smithi052 ceph-mon[31955]: pgmap v180: 129 pgs: 129 active+clean; 407 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 MiB/s rd, 2.7 MiB/s wr, 1.45k op/s 2024-05-27T07:31:05.212 INFO:tasks.workunit.client.0.smithi052.stdout: 116 928573 280 652844 157 526411 516 2024-05-27T07:31:05.239 INFO:journalctl@ceph.mds.j.smithi052.stdout:May 27 07:31:04 smithi052 systemd[1]: ceph-73df2636-1bf8-11ef-bc9b-c7b262605968@mds.j.service: Scheduled restart job, restart counter is at 5. 2024-05-27T07:31:05.239 INFO:journalctl@ceph.mds.j.smithi052.stdout:May 27 07:31:04 smithi052 systemd[1]: Stopped Ceph mds.j for 73df2636-1bf8-11ef-bc9b-c7b262605968. 2024-05-27T07:31:05.239 INFO:journalctl@ceph.mds.j.smithi052.stdout:May 27 07:31:04 smithi052 systemd[1]: ceph-73df2636-1bf8-11ef-bc9b-c7b262605968@mds.j.service: Consumed 1.230s CPU time. 2024-05-27T07:31:05.239 INFO:journalctl@ceph.mds.j.smithi052.stdout:May 27 07:31:05 smithi052 systemd[1]: Starting Ceph mds.j for 73df2636-1bf8-11ef-bc9b-c7b262605968... 2024-05-27T07:31:05.239 INFO:journalctl@ceph.mds.j.smithi052.stdout:May 27 07:31:05 smithi052 podman[108539]: 2024-05-27 07:31:05.132457591 +0000 UTC m=+0.018244542 image pull e36ea4fc8a5807e8d2701370e350a0de3856538797f7fba64cfce7e344bd6799 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6 2024-05-27T07:31:05.239 INFO:journalctl@ceph.mds.j.smithi052.stdout:May 27 07:31:05 smithi052 podman[108539]: 2024-05-27 07:31:05.248063292 +0000 UTC m=+0.133850226 container create 3c3b023744ef806cc320db3e4bb680a307035d3e387dc1ca9704917e5d7bfdcf (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-j, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240521, GIT_CLEAN=True, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , ceph=True, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, CEPH_POINT_RELEASE=) 2024-05-27T07:31:05.521 INFO:journalctl@ceph.mds.j.smithi052.stdout:May 27 07:31:05 smithi052 podman[108539]: 2024-05-27 07:31:05.492892435 +0000 UTC m=+0.378679390 container init 3c3b023744ef806cc320db3e4bb680a307035d3e387dc1ca9704917e5d7bfdcf (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-j, org.label-schema.build-date=20240521, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , ceph=True, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, GIT_CLEAN=True, org.label-schema.license=GPLv2, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, org.label-schema.name=CentOS Stream 8 Base Image) 2024-05-27T07:31:05.521 INFO:journalctl@ceph.mds.j.smithi052.stdout:May 27 07:31:05 smithi052 podman[108539]: 2024-05-27 07:31:05.496038475 +0000 UTC m=+0.381825424 container start 3c3b023744ef806cc320db3e4bb680a307035d3e387dc1ca9704917e5d7bfdcf (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-j, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, maintainer=Guillaume Abrioux , GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240521, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0) 2024-05-27T07:31:05.981 INFO:journalctl@ceph.mds.j.smithi052.stdout:May 27 07:31:05 smithi052 ceph-mds[108558]: -- 172.21.15.52:0/3734609847 <== mon.0 v2:172.21.15.52:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x555565613a40 con 0x55556565b600 2024-05-27T07:31:05.981 INFO:journalctl@ceph.mds.j.smithi052.stdout:May 27 07:31:05 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-j[108554]: starting mds.j at 2024-05-27T07:31:05.982 INFO:journalctl@ceph.mds.j.smithi052.stdout:May 27 07:31:05 smithi052 bash[108539]: 3c3b023744ef806cc320db3e4bb680a307035d3e387dc1ca9704917e5d7bfdcf 2024-05-27T07:31:05.982 INFO:journalctl@ceph.mds.j.smithi052.stdout:May 27 07:31:05 smithi052 systemd[1]: Started Ceph mds.j for 73df2636-1bf8-11ef-bc9b-c7b262605968. 2024-05-27T07:31:06.603 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:31:06 smithi173 ceph-mon[37000]: pgmap v181: 129 pgs: 129 active+clean; 416 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 3.4 MiB/s rd, 4.4 MiB/s wr, 1.36k op/s 2024-05-27T07:31:06.604 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:31:06 smithi173 ceph-mon[37000]: daemon mds.j restarted 2024-05-27T07:31:06.671 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:31:06 smithi052 ceph-mon[31955]: pgmap v181: 129 pgs: 129 active+clean; 416 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 3.4 MiB/s rd, 4.4 MiB/s wr, 1.36k op/s 2024-05-27T07:31:06.671 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:31:06 smithi052 ceph-mon[31955]: daemon mds.j restarted 2024-05-27T07:31:06.672 INFO:journalctl@ceph.mds.j.smithi052.stdout:May 27 07:31:06 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-j[108554]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: In function 'std::set SnapClient::filter(const std::set&) const' thread 7f04b29af700 time 2024-05-27T07:31:06.342977+0000 2024-05-27T07:31:06.672 INFO:journalctl@ceph.mds.j.smithi052.stdout:May 27 07:31:06 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-j[108554]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: 197: FAILED ceph_assert(cached_version > 0) 2024-05-27T07:31:06.672 INFO:journalctl@ceph.mds.j.smithi052.stdout:May 27 07:31:06 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-j[108554]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:31:06.672 INFO:journalctl@ceph.mds.j.smithi052.stdout:May 27 07:31:06 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-j[108554]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x127) [0x7f04c51f4ba6] 2024-05-27T07:31:06.672 INFO:journalctl@ceph.mds.j.smithi052.stdout:May 27 07:31:06 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-j[108554]: 2: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7f04c51f4dd1] 2024-05-27T07:31:06.672 INFO:journalctl@ceph.mds.j.smithi052.stdout:May 27 07:31:06 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-j[108554]: 3: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x55556422a23d] 2024-05-27T07:31:06.672 INFO:journalctl@ceph.mds.j.smithi052.stdout:May 27 07:31:06 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-j[108554]: 4: (SnapRealm::build_snap_set() const+0x323) [0x555564215971] 2024-05-27T07:31:06.672 INFO:journalctl@ceph.mds.j.smithi052.stdout:May 27 07:31:06 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-j[108554]: 5: (SnapRealm::check_cache() const+0x13e) [0x555564213fa0] 2024-05-27T07:31:06.672 INFO:journalctl@ceph.mds.j.smithi052.stdout:May 27 07:31:06 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-j[108554]: 6: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x555564214651] 2024-05-27T07:31:06.672 INFO:journalctl@ceph.mds.j.smithi052.stdout:May 27 07:31:06 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-j[108554]: 7: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x5555641a9347] 2024-05-27T07:31:06.672 INFO:journalctl@ceph.mds.j.smithi052.stdout:May 27 07:31:06 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-j[108554]: 8: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x555563ea587e] 2024-05-27T07:31:06.673 INFO:journalctl@ceph.mds.j.smithi052.stdout:May 27 07:31:06 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-j[108554]: 9: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x279d) [0x555563eaaf2f] 2024-05-27T07:31:06.673 INFO:journalctl@ceph.mds.j.smithi052.stdout:May 27 07:31:06 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-j[108554]: 10: (EUpdate::replay(MDSRank*)+0x63) [0x555563eb5969] 2024-05-27T07:31:06.673 INFO:journalctl@ceph.mds.j.smithi052.stdout:May 27 07:31:06 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-j[108554]: 11: (MDLog::_replay_thread()+0x1f38) [0x55556424e48c] 2024-05-27T07:31:06.673 INFO:journalctl@ceph.mds.j.smithi052.stdout:May 27 07:31:06 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-j[108554]: 12: (MDLog::ReplayThread::entry()+0x11) [0x5555642579cb] 2024-05-27T07:31:06.673 INFO:journalctl@ceph.mds.j.smithi052.stdout:May 27 07:31:06 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-j[108554]: 13: (Thread::entry_wrapper()+0x43) [0x7f04c51c372d] 2024-05-27T07:31:06.673 INFO:journalctl@ceph.mds.j.smithi052.stdout:May 27 07:31:06 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-j[108554]: 14: (Thread::_entry_func(void*)+0xd) [0x7f04c51c3749] 2024-05-27T07:31:06.673 INFO:journalctl@ceph.mds.j.smithi052.stdout:May 27 07:31:06 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-j[108554]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7f04c3d671ca] 2024-05-27T07:31:06.673 INFO:journalctl@ceph.mds.j.smithi052.stdout:May 27 07:31:06 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-j[108554]: 16: clone() 2024-05-27T07:31:06.673 INFO:journalctl@ceph.mds.j.smithi052.stdout:May 27 07:31:06 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-j[108554]: *** Caught signal (Aborted) ** 2024-05-27T07:31:06.673 INFO:journalctl@ceph.mds.j.smithi052.stdout:May 27 07:31:06 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-j[108554]: in thread 7f04b29af700 thread_name:md_log_replay 2024-05-27T07:31:06.674 INFO:journalctl@ceph.mds.j.smithi052.stdout:May 27 07:31:06 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-j[108554]: 2024-05-27T07:31:06.344+0000 7f04b29af700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: In function 'std::set SnapClient::filter(const std::set&) const' thread 7f04b29af700 time 2024-05-27T07:31:06.342977+0000 2024-05-27T07:31:06.674 INFO:journalctl@ceph.mds.j.smithi052.stdout:May 27 07:31:06 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-j[108554]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: 197: FAILED ceph_assert(cached_version > 0) 2024-05-27T07:31:06.674 INFO:journalctl@ceph.mds.j.smithi052.stdout:May 27 07:31:06 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-j[108554]: 2024-05-27T07:31:06.674 INFO:journalctl@ceph.mds.j.smithi052.stdout:May 27 07:31:06 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-j[108554]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:31:06.674 INFO:journalctl@ceph.mds.j.smithi052.stdout:May 27 07:31:06 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-j[108554]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x127) [0x7f04c51f4ba6] 2024-05-27T07:31:06.674 INFO:journalctl@ceph.mds.j.smithi052.stdout:May 27 07:31:06 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-j[108554]: 2: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7f04c51f4dd1] 2024-05-27T07:31:06.675 INFO:journalctl@ceph.mds.j.smithi052.stdout:May 27 07:31:06 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-j[108554]: 3: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x55556422a23d] 2024-05-27T07:31:06.675 INFO:journalctl@ceph.mds.j.smithi052.stdout:May 27 07:31:06 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-j[108554]: 4: (SnapRealm::build_snap_set() const+0x323) [0x555564215971] 2024-05-27T07:31:06.675 INFO:journalctl@ceph.mds.j.smithi052.stdout:May 27 07:31:06 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-j[108554]: 5: (SnapRealm::check_cache() const+0x13e) [0x555564213fa0] 2024-05-27T07:31:06.675 INFO:journalctl@ceph.mds.j.smithi052.stdout:May 27 07:31:06 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-j[108554]: 6: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x555564214651] 2024-05-27T07:31:06.675 INFO:journalctl@ceph.mds.j.smithi052.stdout:May 27 07:31:06 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-j[108554]: 7: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x5555641a9347] 2024-05-27T07:31:06.675 INFO:journalctl@ceph.mds.j.smithi052.stdout:May 27 07:31:06 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-j[108554]: 8: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x555563ea587e] 2024-05-27T07:31:06.675 INFO:journalctl@ceph.mds.j.smithi052.stdout:May 27 07:31:06 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-j[108554]: 9: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x279d) [0x555563eaaf2f] 2024-05-27T07:31:06.675 INFO:journalctl@ceph.mds.j.smithi052.stdout:May 27 07:31:06 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-j[108554]: 10: (EUpdate::replay(MDSRank*)+0x63) [0x555563eb5969] 2024-05-27T07:31:06.675 INFO:journalctl@ceph.mds.j.smithi052.stdout:May 27 07:31:06 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-j[108554]: 11: (MDLog::_replay_thread()+0x1f38) [0x55556424e48c] 2024-05-27T07:31:06.675 INFO:journalctl@ceph.mds.j.smithi052.stdout:May 27 07:31:06 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-j[108554]: 12: (MDLog::ReplayThread::entry()+0x11) [0x5555642579cb] 2024-05-27T07:31:06.675 INFO:journalctl@ceph.mds.j.smithi052.stdout:May 27 07:31:06 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-j[108554]: 13: (Thread::entry_wrapper()+0x43) [0x7f04c51c372d] 2024-05-27T07:31:06.675 INFO:journalctl@ceph.mds.j.smithi052.stdout:May 27 07:31:06 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-j[108554]: 14: (Thread::_entry_func(void*)+0xd) [0x7f04c51c3749] 2024-05-27T07:31:06.676 INFO:journalctl@ceph.mds.j.smithi052.stdout:May 27 07:31:06 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-j[108554]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7f04c3d671ca] 2024-05-27T07:31:06.676 INFO:journalctl@ceph.mds.j.smithi052.stdout:May 27 07:31:06 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-j[108554]: 16: clone() 2024-05-27T07:31:06.676 INFO:journalctl@ceph.mds.j.smithi052.stdout:May 27 07:31:06 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-j[108554]: 2024-05-27T07:31:06.676 INFO:journalctl@ceph.mds.j.smithi052.stdout:May 27 07:31:06 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-j[108554]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:31:06.676 INFO:journalctl@ceph.mds.j.smithi052.stdout:May 27 07:31:06 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-j[108554]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f04c3d71d20] 2024-05-27T07:31:06.676 INFO:journalctl@ceph.mds.j.smithi052.stdout:May 27 07:31:06 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-j[108554]: 2: gsignal() 2024-05-27T07:31:06.676 INFO:journalctl@ceph.mds.j.smithi052.stdout:May 27 07:31:06 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-j[108554]: 3: abort() 2024-05-27T07:31:06.676 INFO:journalctl@ceph.mds.j.smithi052.stdout:May 27 07:31:06 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-j[108554]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x26e) [0x7f04c51f4ced] 2024-05-27T07:31:06.676 INFO:journalctl@ceph.mds.j.smithi052.stdout:May 27 07:31:06 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-j[108554]: 5: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7f04c51f4dd1] 2024-05-27T07:31:06.676 INFO:journalctl@ceph.mds.j.smithi052.stdout:May 27 07:31:06 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-j[108554]: 6: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x55556422a23d] 2024-05-27T07:31:06.676 INFO:journalctl@ceph.mds.j.smithi052.stdout:May 27 07:31:06 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-j[108554]: 7: (SnapRealm::build_snap_set() const+0x323) [0x555564215971] 2024-05-27T07:31:06.676 INFO:journalctl@ceph.mds.j.smithi052.stdout:May 27 07:31:06 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-j[108554]: 8: (SnapRealm::check_cache() const+0x13e) [0x555564213fa0] 2024-05-27T07:31:06.676 INFO:journalctl@ceph.mds.j.smithi052.stdout:May 27 07:31:06 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-j[108554]: 9: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x555564214651] 2024-05-27T07:31:06.677 INFO:journalctl@ceph.mds.j.smithi052.stdout:May 27 07:31:06 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-j[108554]: 10: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x5555641a9347] 2024-05-27T07:31:06.677 INFO:journalctl@ceph.mds.j.smithi052.stdout:May 27 07:31:06 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-j[108554]: 11: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x555563ea587e] 2024-05-27T07:31:06.677 INFO:journalctl@ceph.mds.j.smithi052.stdout:May 27 07:31:06 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-j[108554]: 12: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x279d) [0x555563eaaf2f] 2024-05-27T07:31:06.677 INFO:journalctl@ceph.mds.j.smithi052.stdout:May 27 07:31:06 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-j[108554]: 13: (EUpdate::replay(MDSRank*)+0x63) [0x555563eb5969] 2024-05-27T07:31:06.677 INFO:journalctl@ceph.mds.j.smithi052.stdout:May 27 07:31:06 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-j[108554]: 14: (MDLog::_replay_thread()+0x1f38) [0x55556424e48c] 2024-05-27T07:31:06.677 INFO:journalctl@ceph.mds.j.smithi052.stdout:May 27 07:31:06 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-j[108554]: 15: (MDLog::ReplayThread::entry()+0x11) [0x5555642579cb] 2024-05-27T07:31:06.677 INFO:journalctl@ceph.mds.j.smithi052.stdout:May 27 07:31:06 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-j[108554]: 16: (Thread::entry_wrapper()+0x43) [0x7f04c51c372d] 2024-05-27T07:31:06.677 INFO:journalctl@ceph.mds.j.smithi052.stdout:May 27 07:31:06 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-j[108554]: 17: (Thread::_entry_func(void*)+0xd) [0x7f04c51c3749] 2024-05-27T07:31:06.677 INFO:journalctl@ceph.mds.j.smithi052.stdout:May 27 07:31:06 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-j[108554]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7f04c3d671ca] 2024-05-27T07:31:06.677 INFO:journalctl@ceph.mds.j.smithi052.stdout:May 27 07:31:06 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-j[108554]: 19: clone() 2024-05-27T07:31:06.677 INFO:journalctl@ceph.mds.j.smithi052.stdout:May 27 07:31:06 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-j[108554]: 2024-05-27T07:31:06.346+0000 7f04b29af700 -1 *** Caught signal (Aborted) ** 2024-05-27T07:31:06.677 INFO:journalctl@ceph.mds.j.smithi052.stdout:May 27 07:31:06 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-j[108554]: in thread 7f04b29af700 thread_name:md_log_replay 2024-05-27T07:31:06.677 INFO:journalctl@ceph.mds.j.smithi052.stdout:May 27 07:31:06 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-j[108554]: 2024-05-27T07:31:06.678 INFO:journalctl@ceph.mds.j.smithi052.stdout:May 27 07:31:06 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-j[108554]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:31:06.678 INFO:journalctl@ceph.mds.j.smithi052.stdout:May 27 07:31:06 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-j[108554]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f04c3d71d20] 2024-05-27T07:31:06.678 INFO:journalctl@ceph.mds.j.smithi052.stdout:May 27 07:31:06 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-j[108554]: 2: gsignal() 2024-05-27T07:31:06.678 INFO:journalctl@ceph.mds.j.smithi052.stdout:May 27 07:31:06 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-j[108554]: 3: abort() 2024-05-27T07:31:06.678 INFO:journalctl@ceph.mds.j.smithi052.stdout:May 27 07:31:06 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-j[108554]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x26e) [0x7f04c51f4ced] 2024-05-27T07:31:06.678 INFO:journalctl@ceph.mds.j.smithi052.stdout:May 27 07:31:06 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-j[108554]: 5: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7f04c51f4dd1] 2024-05-27T07:31:06.678 INFO:journalctl@ceph.mds.j.smithi052.stdout:May 27 07:31:06 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-j[108554]: 6: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x55556422a23d] 2024-05-27T07:31:06.678 INFO:journalctl@ceph.mds.j.smithi052.stdout:May 27 07:31:06 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-j[108554]: 7: (SnapRealm::build_snap_set() const+0x323) [0x555564215971] 2024-05-27T07:31:06.678 INFO:journalctl@ceph.mds.j.smithi052.stdout:May 27 07:31:06 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-j[108554]: 8: (SnapRealm::check_cache() const+0x13e) [0x555564213fa0] 2024-05-27T07:31:06.678 INFO:journalctl@ceph.mds.j.smithi052.stdout:May 27 07:31:06 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-j[108554]: 9: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x555564214651] 2024-05-27T07:31:06.678 INFO:journalctl@ceph.mds.j.smithi052.stdout:May 27 07:31:06 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-j[108554]: 10: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x5555641a9347] 2024-05-27T07:31:06.678 INFO:journalctl@ceph.mds.j.smithi052.stdout:May 27 07:31:06 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-j[108554]: 11: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x555563ea587e] 2024-05-27T07:31:06.679 INFO:journalctl@ceph.mds.j.smithi052.stdout:May 27 07:31:06 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-j[108554]: 12: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x279d) [0x555563eaaf2f] 2024-05-27T07:31:06.679 INFO:journalctl@ceph.mds.j.smithi052.stdout:May 27 07:31:06 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-j[108554]: 13: (EUpdate::replay(MDSRank*)+0x63) [0x555563eb5969] 2024-05-27T07:31:06.679 INFO:journalctl@ceph.mds.j.smithi052.stdout:May 27 07:31:06 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-j[108554]: 14: (MDLog::_replay_thread()+0x1f38) [0x55556424e48c] 2024-05-27T07:31:06.679 INFO:journalctl@ceph.mds.j.smithi052.stdout:May 27 07:31:06 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-j[108554]: 15: (MDLog::ReplayThread::entry()+0x11) [0x5555642579cb] 2024-05-27T07:31:06.679 INFO:journalctl@ceph.mds.j.smithi052.stdout:May 27 07:31:06 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-j[108554]: 16: (Thread::entry_wrapper()+0x43) [0x7f04c51c372d] 2024-05-27T07:31:06.679 INFO:journalctl@ceph.mds.j.smithi052.stdout:May 27 07:31:06 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-j[108554]: 17: (Thread::_entry_func(void*)+0xd) [0x7f04c51c3749] 2024-05-27T07:31:06.679 INFO:journalctl@ceph.mds.j.smithi052.stdout:May 27 07:31:06 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-j[108554]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7f04c3d671ca] 2024-05-27T07:31:06.679 INFO:journalctl@ceph.mds.j.smithi052.stdout:May 27 07:31:06 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-j[108554]: 19: clone() 2024-05-27T07:31:06.679 INFO:journalctl@ceph.mds.j.smithi052.stdout:May 27 07:31:06 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-j[108554]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-05-27T07:31:06.679 INFO:journalctl@ceph.mds.j.smithi052.stdout:May 27 07:31:06 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-j[108554]: 2024-05-27T07:31:06.679 INFO:journalctl@ceph.mds.j.smithi052.stdout:May 27 07:31:06 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-j[108554]: -1> 2024-05-27T07:31:06.344+0000 7f04b29af700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: In function 'std::set SnapClient::filter(const std::set&) const' thread 7f04b29af700 time 2024-05-27T07:31:06.342977+0000 2024-05-27T07:31:06.679 INFO:journalctl@ceph.mds.j.smithi052.stdout:May 27 07:31:06 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-j[108554]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: 197: FAILED ceph_assert(cached_version > 0) 2024-05-27T07:31:06.679 INFO:journalctl@ceph.mds.j.smithi052.stdout:May 27 07:31:06 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-j[108554]: 2024-05-27T07:31:06.680 INFO:journalctl@ceph.mds.j.smithi052.stdout:May 27 07:31:06 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-j[108554]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:31:06.680 INFO:journalctl@ceph.mds.j.smithi052.stdout:May 27 07:31:06 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-j[108554]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x127) [0x7f04c51f4ba6] 2024-05-27T07:31:06.680 INFO:journalctl@ceph.mds.j.smithi052.stdout:May 27 07:31:06 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-j[108554]: 2: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7f04c51f4dd1] 2024-05-27T07:31:06.680 INFO:journalctl@ceph.mds.j.smithi052.stdout:May 27 07:31:06 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-j[108554]: 3: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x55556422a23d] 2024-05-27T07:31:06.680 INFO:journalctl@ceph.mds.j.smithi052.stdout:May 27 07:31:06 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-j[108554]: 4: (SnapRealm::build_snap_set() const+0x323) [0x555564215971] 2024-05-27T07:31:06.680 INFO:journalctl@ceph.mds.j.smithi052.stdout:May 27 07:31:06 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-j[108554]: 5: (SnapRealm::check_cache() const+0x13e) [0x555564213fa0] 2024-05-27T07:31:06.680 INFO:journalctl@ceph.mds.j.smithi052.stdout:May 27 07:31:06 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-j[108554]: 6: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x555564214651] 2024-05-27T07:31:06.680 INFO:journalctl@ceph.mds.j.smithi052.stdout:May 27 07:31:06 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-j[108554]: 7: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x5555641a9347] 2024-05-27T07:31:06.680 INFO:journalctl@ceph.mds.j.smithi052.stdout:May 27 07:31:06 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-j[108554]: 8: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x555563ea587e] 2024-05-27T07:31:06.680 INFO:journalctl@ceph.mds.j.smithi052.stdout:May 27 07:31:06 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-j[108554]: 9: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x279d) [0x555563eaaf2f] 2024-05-27T07:31:06.680 INFO:journalctl@ceph.mds.j.smithi052.stdout:May 27 07:31:06 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-j[108554]: 10: (EUpdate::replay(MDSRank*)+0x63) [0x555563eb5969] 2024-05-27T07:31:06.680 INFO:journalctl@ceph.mds.j.smithi052.stdout:May 27 07:31:06 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-j[108554]: 11: (MDLog::_replay_thread()+0x1f38) [0x55556424e48c] 2024-05-27T07:31:06.680 INFO:journalctl@ceph.mds.j.smithi052.stdout:May 27 07:31:06 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-j[108554]: 12: (MDLog::ReplayThread::entry()+0x11) [0x5555642579cb] 2024-05-27T07:31:06.681 INFO:journalctl@ceph.mds.j.smithi052.stdout:May 27 07:31:06 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-j[108554]: 13: (Thread::entry_wrapper()+0x43) [0x7f04c51c372d] 2024-05-27T07:31:06.681 INFO:journalctl@ceph.mds.j.smithi052.stdout:May 27 07:31:06 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-j[108554]: 14: (Thread::_entry_func(void*)+0xd) [0x7f04c51c3749] 2024-05-27T07:31:06.681 INFO:journalctl@ceph.mds.j.smithi052.stdout:May 27 07:31:06 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-j[108554]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7f04c3d671ca] 2024-05-27T07:31:06.681 INFO:journalctl@ceph.mds.j.smithi052.stdout:May 27 07:31:06 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-j[108554]: 16: clone() 2024-05-27T07:31:06.681 INFO:journalctl@ceph.mds.j.smithi052.stdout:May 27 07:31:06 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-j[108554]: 2024-05-27T07:31:06.681 INFO:journalctl@ceph.mds.j.smithi052.stdout:May 27 07:31:06 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-j[108554]: 0> 2024-05-27T07:31:06.346+0000 7f04b29af700 -1 *** Caught signal (Aborted) ** 2024-05-27T07:31:06.681 INFO:journalctl@ceph.mds.j.smithi052.stdout:May 27 07:31:06 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-j[108554]: in thread 7f04b29af700 thread_name:md_log_replay 2024-05-27T07:31:06.681 INFO:journalctl@ceph.mds.j.smithi052.stdout:May 27 07:31:06 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-j[108554]: 2024-05-27T07:31:06.681 INFO:journalctl@ceph.mds.j.smithi052.stdout:May 27 07:31:06 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-j[108554]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:31:06.681 INFO:journalctl@ceph.mds.j.smithi052.stdout:May 27 07:31:06 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-j[108554]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f04c3d71d20] 2024-05-27T07:31:06.681 INFO:journalctl@ceph.mds.j.smithi052.stdout:May 27 07:31:06 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-j[108554]: 2: gsignal() 2024-05-27T07:31:06.681 INFO:journalctl@ceph.mds.j.smithi052.stdout:May 27 07:31:06 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-j[108554]: 3: abort() 2024-05-27T07:31:06.681 INFO:journalctl@ceph.mds.j.smithi052.stdout:May 27 07:31:06 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-j[108554]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x26e) [0x7f04c51f4ced] 2024-05-27T07:31:06.682 INFO:journalctl@ceph.mds.j.smithi052.stdout:May 27 07:31:06 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-j[108554]: 5: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7f04c51f4dd1] 2024-05-27T07:31:06.682 INFO:journalctl@ceph.mds.j.smithi052.stdout:May 27 07:31:06 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-j[108554]: 6: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x55556422a23d] 2024-05-27T07:31:06.682 INFO:journalctl@ceph.mds.j.smithi052.stdout:May 27 07:31:06 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-j[108554]: 7: (SnapRealm::build_snap_set() const+0x323) [0x555564215971] 2024-05-27T07:31:06.682 INFO:journalctl@ceph.mds.j.smithi052.stdout:May 27 07:31:06 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-j[108554]: 8: (SnapRealm::check_cache() const+0x13e) [0x555564213fa0] 2024-05-27T07:31:06.682 INFO:journalctl@ceph.mds.j.smithi052.stdout:May 27 07:31:06 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-j[108554]: 9: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x555564214651] 2024-05-27T07:31:06.682 INFO:journalctl@ceph.mds.j.smithi052.stdout:May 27 07:31:06 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-j[108554]: 10: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x5555641a9347] 2024-05-27T07:31:06.682 INFO:journalctl@ceph.mds.j.smithi052.stdout:May 27 07:31:06 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-j[108554]: 11: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x555563ea587e] 2024-05-27T07:31:06.682 INFO:journalctl@ceph.mds.j.smithi052.stdout:May 27 07:31:06 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-j[108554]: 12: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x279d) [0x555563eaaf2f] 2024-05-27T07:31:06.682 INFO:journalctl@ceph.mds.j.smithi052.stdout:May 27 07:31:06 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-j[108554]: 13: (EUpdate::replay(MDSRank*)+0x63) [0x555563eb5969] 2024-05-27T07:31:06.682 INFO:journalctl@ceph.mds.j.smithi052.stdout:May 27 07:31:06 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-j[108554]: 14: (MDLog::_replay_thread()+0x1f38) [0x55556424e48c] 2024-05-27T07:31:06.682 INFO:journalctl@ceph.mds.j.smithi052.stdout:May 27 07:31:06 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-j[108554]: 15: (MDLog::ReplayThread::entry()+0x11) [0x5555642579cb] 2024-05-27T07:31:06.683 INFO:journalctl@ceph.mds.j.smithi052.stdout:May 27 07:31:06 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-j[108554]: 16: (Thread::entry_wrapper()+0x43) [0x7f04c51c372d] 2024-05-27T07:31:06.683 INFO:journalctl@ceph.mds.j.smithi052.stdout:May 27 07:31:06 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-j[108554]: 17: (Thread::_entry_func(void*)+0xd) [0x7f04c51c3749] 2024-05-27T07:31:06.683 INFO:journalctl@ceph.mds.j.smithi052.stdout:May 27 07:31:06 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-j[108554]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7f04c3d671ca] 2024-05-27T07:31:06.683 INFO:journalctl@ceph.mds.j.smithi052.stdout:May 27 07:31:06 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-j[108554]: 19: clone() 2024-05-27T07:31:06.683 INFO:journalctl@ceph.mds.j.smithi052.stdout:May 27 07:31:06 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-j[108554]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-05-27T07:31:06.683 INFO:journalctl@ceph.mds.j.smithi052.stdout:May 27 07:31:06 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-j[108554]: 2024-05-27T07:31:06.683 INFO:journalctl@ceph.mds.j.smithi052.stdout:May 27 07:31:06 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-j[108554]: -1> 2024-05-27T07:31:06.344+0000 7f04b29af700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: In function 'std::set SnapClient::filter(const std::set&) const' thread 7f04b29af700 time 2024-05-27T07:31:06.342977+0000 2024-05-27T07:31:06.683 INFO:journalctl@ceph.mds.j.smithi052.stdout:May 27 07:31:06 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-j[108554]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: 197: FAILED ceph_assert(cached_version > 0) 2024-05-27T07:31:06.684 INFO:journalctl@ceph.mds.j.smithi052.stdout:May 27 07:31:06 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-j[108554]: 2024-05-27T07:31:06.684 INFO:journalctl@ceph.mds.j.smithi052.stdout:May 27 07:31:06 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-j[108554]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:31:06.684 INFO:journalctl@ceph.mds.j.smithi052.stdout:May 27 07:31:06 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-j[108554]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x127) [0x7f04c51f4ba6] 2024-05-27T07:31:06.684 INFO:journalctl@ceph.mds.j.smithi052.stdout:May 27 07:31:06 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-j[108554]: 2: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7f04c51f4dd1] 2024-05-27T07:31:06.684 INFO:journalctl@ceph.mds.j.smithi052.stdout:May 27 07:31:06 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-j[108554]: 3: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x55556422a23d] 2024-05-27T07:31:06.684 INFO:journalctl@ceph.mds.j.smithi052.stdout:May 27 07:31:06 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-j[108554]: 4: (SnapRealm::build_snap_set() const+0x323) [0x555564215971] 2024-05-27T07:31:06.684 INFO:journalctl@ceph.mds.j.smithi052.stdout:May 27 07:31:06 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-j[108554]: 5: (SnapRealm::check_cache() const+0x13e) [0x555564213fa0] 2024-05-27T07:31:06.684 INFO:journalctl@ceph.mds.j.smithi052.stdout:May 27 07:31:06 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-j[108554]: 6: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x555564214651] 2024-05-27T07:31:06.684 INFO:journalctl@ceph.mds.j.smithi052.stdout:May 27 07:31:06 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-j[108554]: 7: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x5555641a9347] 2024-05-27T07:31:06.684 INFO:journalctl@ceph.mds.j.smithi052.stdout:May 27 07:31:06 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-j[108554]: 8: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x555563ea587e] 2024-05-27T07:31:06.684 INFO:journalctl@ceph.mds.j.smithi052.stdout:May 27 07:31:06 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-j[108554]: 9: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x279d) [0x555563eaaf2f] 2024-05-27T07:31:06.684 INFO:journalctl@ceph.mds.j.smithi052.stdout:May 27 07:31:06 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-j[108554]: 10: (EUpdate::replay(MDSRank*)+0x63) [0x555563eb5969] 2024-05-27T07:31:06.685 INFO:journalctl@ceph.mds.j.smithi052.stdout:May 27 07:31:06 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-j[108554]: 11: (MDLog::_replay_thread()+0x1f38) [0x55556424e48c] 2024-05-27T07:31:06.685 INFO:journalctl@ceph.mds.j.smithi052.stdout:May 27 07:31:06 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-j[108554]: 12: (MDLog::ReplayThread::entry()+0x11) [0x5555642579cb] 2024-05-27T07:31:06.685 INFO:journalctl@ceph.mds.j.smithi052.stdout:May 27 07:31:06 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-j[108554]: 13: (Thread::entry_wrapper()+0x43) [0x7f04c51c372d] 2024-05-27T07:31:06.685 INFO:journalctl@ceph.mds.j.smithi052.stdout:May 27 07:31:06 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-j[108554]: 14: (Thread::_entry_func(void*)+0xd) [0x7f04c51c3749] 2024-05-27T07:31:06.685 INFO:journalctl@ceph.mds.j.smithi052.stdout:May 27 07:31:06 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-j[108554]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7f04c3d671ca] 2024-05-27T07:31:06.685 INFO:journalctl@ceph.mds.j.smithi052.stdout:May 27 07:31:06 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-j[108554]: 16: clone() 2024-05-27T07:31:06.685 INFO:journalctl@ceph.mds.j.smithi052.stdout:May 27 07:31:06 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-j[108554]: 2024-05-27T07:31:06.685 INFO:journalctl@ceph.mds.j.smithi052.stdout:May 27 07:31:06 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-j[108554]: 0> 2024-05-27T07:31:06.346+0000 7f04b29af700 -1 *** Caught signal (Aborted) ** 2024-05-27T07:31:06.685 INFO:journalctl@ceph.mds.j.smithi052.stdout:May 27 07:31:06 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-j[108554]: in thread 7f04b29af700 thread_name:md_log_replay 2024-05-27T07:31:06.685 INFO:journalctl@ceph.mds.j.smithi052.stdout:May 27 07:31:06 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-j[108554]: 2024-05-27T07:31:06.685 INFO:journalctl@ceph.mds.j.smithi052.stdout:May 27 07:31:06 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-j[108554]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:31:06.685 INFO:journalctl@ceph.mds.j.smithi052.stdout:May 27 07:31:06 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-j[108554]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f04c3d71d20] 2024-05-27T07:31:06.685 INFO:journalctl@ceph.mds.j.smithi052.stdout:May 27 07:31:06 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-j[108554]: 2: gsignal() 2024-05-27T07:31:06.686 INFO:journalctl@ceph.mds.j.smithi052.stdout:May 27 07:31:06 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-j[108554]: 3: abort() 2024-05-27T07:31:06.686 INFO:journalctl@ceph.mds.j.smithi052.stdout:May 27 07:31:06 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-j[108554]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x26e) [0x7f04c51f4ced] 2024-05-27T07:31:06.686 INFO:journalctl@ceph.mds.j.smithi052.stdout:May 27 07:31:06 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-j[108554]: 5: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7f04c51f4dd1] 2024-05-27T07:31:06.686 INFO:journalctl@ceph.mds.j.smithi052.stdout:May 27 07:31:06 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-j[108554]: 6: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x55556422a23d] 2024-05-27T07:31:06.686 INFO:journalctl@ceph.mds.j.smithi052.stdout:May 27 07:31:06 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-j[108554]: 7: (SnapRealm::build_snap_set() const+0x323) [0x555564215971] 2024-05-27T07:31:06.686 INFO:journalctl@ceph.mds.j.smithi052.stdout:May 27 07:31:06 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-j[108554]: 8: (SnapRealm::check_cache() const+0x13e) [0x555564213fa0] 2024-05-27T07:31:06.686 INFO:journalctl@ceph.mds.j.smithi052.stdout:May 27 07:31:06 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-j[108554]: 9: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x555564214651] 2024-05-27T07:31:06.686 INFO:journalctl@ceph.mds.j.smithi052.stdout:May 27 07:31:06 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-j[108554]: 10: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x5555641a9347] 2024-05-27T07:31:06.686 INFO:journalctl@ceph.mds.j.smithi052.stdout:May 27 07:31:06 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-j[108554]: 11: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x555563ea587e] 2024-05-27T07:31:06.686 INFO:journalctl@ceph.mds.j.smithi052.stdout:May 27 07:31:06 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-j[108554]: 12: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x279d) [0x555563eaaf2f] 2024-05-27T07:31:06.686 INFO:journalctl@ceph.mds.j.smithi052.stdout:May 27 07:31:06 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-j[108554]: 13: (EUpdate::replay(MDSRank*)+0x63) [0x555563eb5969] 2024-05-27T07:31:06.686 INFO:journalctl@ceph.mds.j.smithi052.stdout:May 27 07:31:06 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-j[108554]: 14: (MDLog::_replay_thread()+0x1f38) [0x55556424e48c] 2024-05-27T07:31:06.687 INFO:journalctl@ceph.mds.j.smithi052.stdout:May 27 07:31:06 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-j[108554]: 15: (MDLog::ReplayThread::entry()+0x11) [0x5555642579cb] 2024-05-27T07:31:06.687 INFO:journalctl@ceph.mds.j.smithi052.stdout:May 27 07:31:06 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-j[108554]: 16: (Thread::entry_wrapper()+0x43) [0x7f04c51c372d] 2024-05-27T07:31:06.687 INFO:journalctl@ceph.mds.j.smithi052.stdout:May 27 07:31:06 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-j[108554]: 17: (Thread::_entry_func(void*)+0xd) [0x7f04c51c3749] 2024-05-27T07:31:06.687 INFO:journalctl@ceph.mds.j.smithi052.stdout:May 27 07:31:06 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-j[108554]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7f04c3d671ca] 2024-05-27T07:31:06.687 INFO:journalctl@ceph.mds.j.smithi052.stdout:May 27 07:31:06 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-j[108554]: 19: clone() 2024-05-27T07:31:06.687 INFO:journalctl@ceph.mds.j.smithi052.stdout:May 27 07:31:06 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-j[108554]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-05-27T07:31:06.687 INFO:journalctl@ceph.mds.j.smithi052.stdout:May 27 07:31:06 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-j[108554]: 2024-05-27T07:31:06.687 INFO:journalctl@ceph.mds.j.smithi052.stdout:May 27 07:31:06 smithi052 podman[108589]: 2024-05-27 07:31:06.388203109 +0000 UTC m=+0.019137805 container died 3c3b023744ef806cc320db3e4bb680a307035d3e387dc1ca9704917e5d7bfdcf (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-j, org.label-schema.license=GPLv2, GIT_CLEAN=True, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, ceph=True, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, org.label-schema.build-date=20240521, CEPH_POINT_RELEASE=, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0) 2024-05-27T07:31:06.691 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:31:06 smithi105 ceph-mon[35977]: pgmap v181: 129 pgs: 129 active+clean; 416 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 3.4 MiB/s rd, 4.4 MiB/s wr, 1.36k op/s 2024-05-27T07:31:06.692 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:31:06 smithi105 ceph-mon[35977]: daemon mds.j restarted 2024-05-27T07:31:07.545 INFO:journalctl@ceph.mds.j.smithi052.stdout:May 27 07:31:07 smithi052 podman[108589]: 2024-05-27 07:31:07.492027505 +0000 UTC m=+1.122962199 container remove 3c3b023744ef806cc320db3e4bb680a307035d3e387dc1ca9704917e5d7bfdcf (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-j, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240521, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, GIT_CLEAN=True, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, ceph=True, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90) 2024-05-27T07:31:07.545 INFO:journalctl@ceph.mds.j.smithi052.stdout:May 27 07:31:07 smithi052 systemd[1]: ceph-73df2636-1bf8-11ef-bc9b-c7b262605968@mds.j.service: Main process exited, code=exited, status=134/n/a 2024-05-27T07:31:07.545 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:31:07 smithi052 ceph-mon[31955]: osdmap e116: 12 total, 12 up, 12 in 2024-05-27T07:31:07.546 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:31:07 smithi052 ceph-mon[31955]: mds.? [v2:172.21.15.52:6834/1343237258,v1:172.21.15.52:6835/1343237258] up:boot 2024-05-27T07:31:07.546 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:31:07 smithi052 ceph-mon[31955]: fsmap cephfs:5 1 up:standby 5 up:active 2024-05-27T07:31:07.546 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:31:07 smithi052 ceph-mon[31955]: from='mgr.15333 172.21.15.52:0/416389082' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "j"}]: dispatch 2024-05-27T07:31:07.546 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:31:07 smithi052 ceph-mon[31955]: fsmap cephfs:5 1 up:standby-replay 5 up:active 2024-05-27T07:31:07.604 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:31:07 smithi173 ceph-mon[37000]: osdmap e116: 12 total, 12 up, 12 in 2024-05-27T07:31:07.604 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:31:07 smithi173 ceph-mon[37000]: mds.? [v2:172.21.15.52:6834/1343237258,v1:172.21.15.52:6835/1343237258] up:boot 2024-05-27T07:31:07.604 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:31:07 smithi173 ceph-mon[37000]: fsmap cephfs:5 1 up:standby 5 up:active 2024-05-27T07:31:07.604 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:31:07 smithi173 ceph-mon[37000]: from='mgr.15333 172.21.15.52:0/416389082' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "j"}]: dispatch 2024-05-27T07:31:07.604 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:31:07 smithi173 ceph-mon[37000]: fsmap cephfs:5 1 up:standby-replay 5 up:active 2024-05-27T07:31:07.692 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:31:07 smithi105 ceph-mon[35977]: osdmap e116: 12 total, 12 up, 12 in 2024-05-27T07:31:07.692 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:31:07 smithi105 ceph-mon[35977]: mds.? [v2:172.21.15.52:6834/1343237258,v1:172.21.15.52:6835/1343237258] up:boot 2024-05-27T07:31:07.692 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:31:07 smithi105 ceph-mon[35977]: fsmap cephfs:5 1 up:standby 5 up:active 2024-05-27T07:31:07.692 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:31:07 smithi105 ceph-mon[35977]: from='mgr.15333 172.21.15.52:0/416389082' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "j"}]: dispatch 2024-05-27T07:31:07.692 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:31:07 smithi105 ceph-mon[35977]: fsmap cephfs:5 1 up:standby-replay 5 up:active 2024-05-27T07:31:07.981 INFO:journalctl@ceph.mds.j.smithi052.stdout:May 27 07:31:07 smithi052 systemd[1]: ceph-73df2636-1bf8-11ef-bc9b-c7b262605968@mds.j.service: Failed with result 'exit-code'. 2024-05-27T07:31:08.603 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:31:08 smithi173 ceph-mon[37000]: pgmap v183: 129 pgs: 129 active+clean; 428 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 6.2 MiB/s wr, 1.66k op/s 2024-05-27T07:31:08.691 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:31:08 smithi105 ceph-mon[35977]: pgmap v183: 129 pgs: 129 active+clean; 428 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 6.2 MiB/s wr, 1.66k op/s 2024-05-27T07:31:08.731 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:31:08 smithi052 ceph-mon[31955]: pgmap v183: 129 pgs: 129 active+clean; 428 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 6.2 MiB/s wr, 1.66k op/s 2024-05-27T07:31:09.603 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:31:09 smithi173 ceph-mon[37000]: scrub summary: idle+waiting paths [/] 2024-05-27T07:31:09.692 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:31:09 smithi105 ceph-mon[35977]: scrub summary: idle+waiting paths [/] 2024-05-27T07:31:09.731 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:31:09 smithi052 ceph-mon[31955]: scrub summary: idle+waiting paths [/] 2024-05-27T07:31:10.603 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:31:10 smithi173 ceph-mon[37000]: pgmap v184: 129 pgs: 129 active+clean; 452 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 8.3 MiB/s wr, 1.25k op/s 2024-05-27T07:31:10.692 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:31:10 smithi105 ceph-mon[35977]: pgmap v184: 129 pgs: 129 active+clean; 452 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 8.3 MiB/s wr, 1.25k op/s 2024-05-27T07:31:10.731 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:31:10 smithi052 ceph-mon[31955]: pgmap v184: 129 pgs: 129 active+clean; 452 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 8.3 MiB/s wr, 1.25k op/s 2024-05-27T07:31:11.604 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:31:11 smithi173 ceph-mon[37000]: pgmap v185: 129 pgs: 129 active+clean; 456 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 8.7 MiB/s wr, 786 op/s 2024-05-27T07:31:11.692 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:31:11 smithi105 ceph-mon[35977]: pgmap v185: 129 pgs: 129 active+clean; 456 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 8.7 MiB/s wr, 786 op/s 2024-05-27T07:31:11.731 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:31:11 smithi052 ceph-mon[31955]: pgmap v185: 129 pgs: 129 active+clean; 456 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 8.7 MiB/s wr, 786 op/s 2024-05-27T07:31:12.604 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:31:12 smithi173 ceph-mon[37000]: scrub summary: idle 2024-05-27T07:31:12.604 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:31:12 smithi173 ceph-mon[37000]: scrub summary: idle 2024-05-27T07:31:12.604 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:31:12 smithi173 ceph-mon[37000]: scrub summary: idle 2024-05-27T07:31:12.692 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:31:12 smithi105 ceph-mon[35977]: scrub summary: idle 2024-05-27T07:31:12.692 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:31:12 smithi105 ceph-mon[35977]: scrub summary: idle 2024-05-27T07:31:12.692 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:31:12 smithi105 ceph-mon[35977]: scrub summary: idle 2024-05-27T07:31:12.741 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:31:12 smithi052 ceph-mon[31955]: scrub summary: idle 2024-05-27T07:31:12.741 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:31:12 smithi052 ceph-mon[31955]: scrub summary: idle 2024-05-27T07:31:12.741 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:31:12 smithi052 ceph-mon[31955]: scrub summary: idle 2024-05-27T07:31:13.604 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:31:13 smithi173 ceph-mon[37000]: scrub summary: idle 2024-05-27T07:31:13.604 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:31:13 smithi173 ceph-mon[37000]: scrub summary: idle 2024-05-27T07:31:13.604 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:31:13 smithi173 ceph-mon[37000]: fsmap cephfs:5 1 up:standby-replay 5 up:active 2024-05-27T07:31:13.604 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:31:13 smithi173 ceph-mon[37000]: pgmap v186: 129 pgs: 129 active+clean; 458 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 8.0 MiB/s wr, 538 op/s 2024-05-27T07:31:13.692 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:31:13 smithi105 ceph-mon[35977]: scrub summary: idle 2024-05-27T07:31:13.692 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:31:13 smithi105 ceph-mon[35977]: scrub summary: idle 2024-05-27T07:31:13.692 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:31:13 smithi105 ceph-mon[35977]: fsmap cephfs:5 1 up:standby-replay 5 up:active 2024-05-27T07:31:13.692 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:31:13 smithi105 ceph-mon[35977]: pgmap v186: 129 pgs: 129 active+clean; 458 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 8.0 MiB/s wr, 538 op/s 2024-05-27T07:31:13.731 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:31:13 smithi052 ceph-mon[31955]: scrub summary: idle 2024-05-27T07:31:13.732 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:31:13 smithi052 ceph-mon[31955]: scrub summary: idle 2024-05-27T07:31:13.732 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:31:13 smithi052 ceph-mon[31955]: fsmap cephfs:5 1 up:standby-replay 5 up:active 2024-05-27T07:31:13.732 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:31:13 smithi052 ceph-mon[31955]: pgmap v186: 129 pgs: 129 active+clean; 458 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 8.0 MiB/s wr, 538 op/s 2024-05-27T07:31:13.967 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:31:13.974+0000 7f33bc93e640 1 -- 172.21.15.52:0/2981831600 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f33b00960c0 msgr2=0x7f33b00a24b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:31:13.967 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:31:13.974+0000 7f33bc93e640 1 --2- 172.21.15.52:0/2981831600 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f33b00960c0 0x7f33b00a24b0 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7f33b80687b0 tx=0x7f33b8097b00 comp rx=0 tx=0).stop 2024-05-27T07:31:13.967 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:31:13.974+0000 7f33bc93e640 1 -- 172.21.15.52:0/2981831600 shutdown_connections 2024-05-27T07:31:13.967 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:31:13.974+0000 7f33bc93e640 1 --2- 172.21.15.52:0/2981831600 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f33b00960c0 0x7f33b00a24b0 unknown :-1 s=CLOSED pgs=173 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:31:13.967 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:31:13.974+0000 7f33bc93e640 1 --2- 172.21.15.52:0/2981831600 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f33b009bec0 0x7f33b0095a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:31:13.967 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:31:13.974+0000 7f33bc93e640 1 --2- 172.21.15.52:0/2981831600 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f33b009ba20 0x7f33b0095510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:31:13.968 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:31:13.974+0000 7f33bc93e640 1 -- 172.21.15.52:0/2981831600 >> 172.21.15.52:0/2981831600 conn(0x7f33b000b010 msgr2=0x7f33b000b450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:31:13.968 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:31:13.974+0000 7f33bc93e640 1 -- 172.21.15.52:0/2981831600 shutdown_connections 2024-05-27T07:31:13.968 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:31:13.975+0000 7f33bc93e640 1 -- 172.21.15.52:0/2981831600 wait complete. 2024-05-27T07:31:13.970 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:31:13.976+0000 7f33bc93e640 1 Processor -- start 2024-05-27T07:31:13.971 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:31:13.976+0000 7f33bc93e640 1 -- start start 2024-05-27T07:31:13.971 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:31:13.977+0000 7f33bc93e640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f33b00960c0 0x7f33b012b4b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:31:13.971 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:31:13.977+0000 7f33bc93e640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f33b009ba20 0x7f33b012ba10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:31:13.971 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:31:13.977+0000 7f33bc93e640 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f33b009bec0 0x7f33b012fcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:31:13.971 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:31:13.977+0000 7f33bc93e640 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x7f33b00a4b30 con 0x7f33b009bec0 2024-05-27T07:31:13.971 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:31:13.977+0000 7f33bc93e640 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f33b00a49b0 con 0x7f33b00960c0 2024-05-27T07:31:13.971 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:31:13.977+0000 7f33bc93e640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f33b00a4cb0 con 0x7f33b009ba20 2024-05-27T07:31:13.971 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:31:13.977+0000 7f33b6d9d640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f33b00960c0 0x7f33b012b4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:31:13.971 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:31:13.977+0000 7f33b6d9d640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f33b00960c0 0x7f33b012b4b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.52:44404/0 (socket says 172.21.15.52:44404) 2024-05-27T07:31:13.972 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:31:13.977+0000 7f33b6d9d640 1 -- 172.21.15.52:0/2729144489 learned_addr learned my addr 172.21.15.52:0/2729144489 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-05-27T07:31:13.972 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:31:13.977+0000 7f33b759e640 1 --2- 172.21.15.52:0/2729144489 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f33b009bec0 0x7f33b012fcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:31:13.972 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:31:13.977+0000 7f33b6d9d640 1 -- 172.21.15.52:0/2729144489 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f33b009ba20 msgr2=0x7f33b012ba10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:31:13.972 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:31:13.977+0000 7f33b6d9d640 1 --2- 172.21.15.52:0/2729144489 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f33b009ba20 0x7f33b012ba10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:31:13.972 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:31:13.977+0000 7f33b6d9d640 1 -- 172.21.15.52:0/2729144489 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f33b009bec0 msgr2=0x7f33b012fcd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:31:13.972 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:31:13.977+0000 7f33b6d9d640 1 --2- 172.21.15.52:0/2729144489 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f33b009bec0 0x7f33b012fcd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:31:13.972 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:31:13.977+0000 7f33b6d9d640 1 -- 172.21.15.52:0/2729144489 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f33b0130310 con 0x7f33b00960c0 2024-05-27T07:31:13.972 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:31:13.978+0000 7f33b759e640 1 --2- 172.21.15.52:0/2729144489 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f33b009bec0 0x7f33b012fcd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-27T07:31:13.972 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:31:13.978+0000 7f33b6d9d640 1 --2- 172.21.15.52:0/2729144489 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f33b00960c0 0x7f33b012b4b0 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7f33a8009a00 tx=0x7f33a8009ec0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:31:13.972 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:31:13.978+0000 7f339ffff640 1 -- 172.21.15.52:0/2729144489 <== mon.2 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f33a8007ad0 con 0x7f33b00960c0 2024-05-27T07:31:13.972 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:31:13.978+0000 7f339ffff640 1 -- 172.21.15.52:0/2729144489 <== mon.2 v2:172.21.15.105:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f33a8007c70 con 0x7f33b00960c0 2024-05-27T07:31:13.973 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:31:13.978+0000 7f33bc93e640 1 -- 172.21.15.52:0/2729144489 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f33b0130600 con 0x7f33b00960c0 2024-05-27T07:31:13.973 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:31:13.978+0000 7f339ffff640 1 -- 172.21.15.52:0/2729144489 <== mon.2 v2:172.21.15.105:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f33a8003df0 con 0x7f33b00960c0 2024-05-27T07:31:13.973 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:31:13.978+0000 7f33bc93e640 1 -- 172.21.15.52:0/2729144489 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f33b0137f80 con 0x7f33b00960c0 2024-05-27T07:31:13.973 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:31:13.981+0000 7f33c3ff8740 1 Processor -- start 2024-05-27T07:31:13.973 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:31:13.981+0000 7f33c3ff8740 1 -- start start 2024-05-27T07:31:13.976 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:31:13.982+0000 7f33c3ff8740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x562a4a5c81b0 0x562a4a969460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:31:13.976 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:31:13.982+0000 7f33c3ff8740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x562a4a59f2f0 0x562a4a9699a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:31:13.976 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:31:13.982+0000 7f33c3ff8740 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x562a4a5a74d0 0x562a4a969ee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:31:13.976 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:31:13.982+0000 7f33c3ff8740 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x562a4a8dbc30 con 0x562a4a5a74d0 2024-05-27T07:31:13.976 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:31:13.982+0000 7f33c3ff8740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x562a4a8cb7d0 con 0x562a4a59f2f0 2024-05-27T07:31:13.976 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:31:13.982+0000 7f33c3ff8740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x562a4a8cf810 con 0x562a4a5c81b0 2024-05-27T07:31:13.976 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:31:13.982+0000 7f33b759e640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x562a4a5c81b0 0x562a4a969460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:31:13.976 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:31:13.982+0000 7f33b759e640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x562a4a5c81b0 0x562a4a969460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.173:3300/0 says I am v2:172.21.15.52:33476/0 (socket says 172.21.15.52:33476) 2024-05-27T07:31:13.976 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:31:13.982+0000 7f33b759e640 1 -- 172.21.15.52:0/2317135777 learned_addr learned my addr 172.21.15.52:0/2317135777 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-05-27T07:31:13.976 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:31:13.982+0000 7f33b659c640 1 --2- 172.21.15.52:0/2317135777 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x562a4a5a74d0 0x562a4a969ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:31:13.976 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:31:13.983+0000 7f339ffff640 1 -- 172.21.15.52:0/2729144489 <== mon.2 v2:172.21.15.105:3300/0 4 ==== mgrmap(e 26) ==== 149783+0+0 (secure 0 0 0) 0x7f33a801e030 con 0x7f33b00960c0 2024-05-27T07:31:13.977 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:31:13.983+0000 7f33b6d9d640 1 --2- 172.21.15.52:0/2317135777 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x562a4a59f2f0 0x562a4a9699a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:31:13.977 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:31:13.983+0000 7f33b759e640 1 -- 172.21.15.52:0/2317135777 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x562a4a59f2f0 msgr2=0x562a4a9699a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:31:13.977 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:31:13.983+0000 7f33b759e640 1 --2- 172.21.15.52:0/2317135777 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x562a4a59f2f0 0x562a4a9699a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:31:13.977 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:31:13.983+0000 7f33b759e640 1 -- 172.21.15.52:0/2317135777 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x562a4a5a74d0 msgr2=0x562a4a969ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:31:13.977 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:31:13.983+0000 7f33b759e640 1 --2- 172.21.15.52:0/2317135777 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x562a4a5a74d0 0x562a4a969ee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:31:13.977 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:31:13.983+0000 7f33b759e640 1 -- 172.21.15.52:0/2317135777 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x562a4a5e0210 con 0x562a4a5c81b0 2024-05-27T07:31:13.977 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:31:13.983+0000 7f33b659c640 1 --2- 172.21.15.52:0/2317135777 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x562a4a5a74d0 0x562a4a969ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-27T07:31:13.977 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:31:13.984+0000 7f33b759e640 1 --2- 172.21.15.52:0/2317135777 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x562a4a5c81b0 0x562a4a969460 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7f33b8068b80 tx=0x7f33b8068f80 comp rx=0 tx=0).ready entity=mon.1 client_cookie=ec213ad442b941a0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:31:13.980 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:31:13.984+0000 7f339d7fa640 1 -- 172.21.15.52:0/2317135777 <== mon.1 v2:172.21.15.173:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f33b80b2070 con 0x562a4a5c81b0 2024-05-27T07:31:13.980 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:31:13.984+0000 7f339ffff640 1 --2- 172.21.15.52:0/2729144489 >> [v2:172.21.15.52:6800/1822318384,v1:172.21.15.52:6801/1822318384] conn(0x7f33a00b29e0 0x7f33a00b4ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:31:13.980 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:31:13.984+0000 7f339d7fa640 1 -- 172.21.15.52:0/2317135777 <== mon.1 v2:172.21.15.173:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f33b8069cb0 con 0x562a4a5c81b0 2024-05-27T07:31:13.980 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:31:13.984+0000 7f339d7fa640 1 -- 172.21.15.52:0/2317135777 <== mon.1 v2:172.21.15.173:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f33b809e050 con 0x562a4a5c81b0 2024-05-27T07:31:13.980 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:31:13.984+0000 7f33b6d9d640 1 --2- 172.21.15.52:0/2729144489 >> [v2:172.21.15.52:6800/1822318384,v1:172.21.15.52:6801/1822318384] conn(0x7f33a00b29e0 0x7f33a00b4ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:31:13.980 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:31:13.985+0000 7f339ffff640 1 -- 172.21.15.52:0/2729144489 <== mon.2 v2:172.21.15.105:3300/0 5 ==== osd_map(116..116 src has 1..116) ==== 9379+0+0 (secure 0 0 0) 0x7f33a80e1b10 con 0x7f33b00960c0 2024-05-27T07:31:13.980 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:31:13.985+0000 7f33c3ff8740 1 -- 172.21.15.52:0/2317135777 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x562a4a5c81b0 msgr2=0x562a4a969460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:31:13.980 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:31:13.985+0000 7f33c3ff8740 1 --2- 172.21.15.52:0/2317135777 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x562a4a5c81b0 0x562a4a969460 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7f33b8068b80 tx=0x7f33b8068f80 comp rx=0 tx=0).stop 2024-05-27T07:31:13.981 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:31:13.985+0000 7f33b6d9d640 1 --2- 172.21.15.52:0/2729144489 >> [v2:172.21.15.52:6800/1822318384,v1:172.21.15.52:6801/1822318384] conn(0x7f33a00b29e0 0x7f33a00b4ef0 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7f33a80defb0 tx=0x7f33a80e1e10 comp rx=0 tx=0).ready entity=mgr.15333 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:31:13.981 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:31:13.985+0000 7f33c3ff8740 1 -- 172.21.15.52:0/2317135777 shutdown_connections 2024-05-27T07:31:13.981 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:31:13.985+0000 7f33c3ff8740 1 --2- 172.21.15.52:0/2317135777 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x562a4a5a74d0 0x562a4a969ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:31:13.981 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:31:13.985+0000 7f33c3ff8740 1 --2- 172.21.15.52:0/2317135777 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x562a4a59f2f0 0x562a4a9699a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:31:13.981 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:31:13.985+0000 7f33c3ff8740 1 --2- 172.21.15.52:0/2317135777 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x562a4a5c81b0 0x562a4a969460 unknown :-1 s=CLOSED pgs=136 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:31:13.981 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:31:13.985+0000 7f33c3ff8740 1 -- 172.21.15.52:0/2317135777 >> 172.21.15.52:0/2317135777 conn(0x562a4a5bc300 msgr2=0x562a4a965d40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:31:13.981 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:31:13.985+0000 7f33c3ff8740 1 -- 172.21.15.52:0/2317135777 shutdown_connections 2024-05-27T07:31:13.981 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:31:13.985+0000 7f33c3ff8740 1 -- 172.21.15.52:0/2317135777 wait complete. 2024-05-27T07:31:13.981 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:31:13.986+0000 7f33c3ff8740 1 -- start start 2024-05-27T07:31:13.981 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:31:13.986+0000 7f33c3ff8740 1 Processor -- start 2024-05-27T07:31:13.981 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:31:13.987+0000 7f33c3ff8740 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x562a4a59f2f0 0x562a4a978000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:31:13.981 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:31:13.987+0000 7f33c3ff8740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x562a4a5a74d0 0x562a4a978540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:31:13.982 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:31:13.987+0000 7f33c3ff8740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x562a4a5c81b0 0x562a4a97de50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:31:13.982 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:31:13.987+0000 7f33c3ff8740 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x562a4a5b1790 con 0x562a4a59f2f0 2024-05-27T07:31:13.982 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:31:13.987+0000 7f33c3ff8740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x562a4a887890 con 0x562a4a5c81b0 2024-05-27T07:31:13.982 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:31:13.987+0000 7f33c3ff8740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x562a4a97ab20 con 0x562a4a5a74d0 2024-05-27T07:31:13.982 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:31:13.987+0000 7f33c3ff8740 10 client.0 _ll_register_callbacks cb 0x562a4a5dfa00 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T07:31:13.982 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:31:13.987+0000 7f33b659c640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x562a4a5a74d0 0x562a4a978540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:31:13.982 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:31:13.987+0000 7f33b659c640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x562a4a5a74d0 0x562a4a978540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.173:3300/0 says I am v2:172.21.15.52:33486/0 (socket says 172.21.15.52:33486) 2024-05-27T07:31:13.982 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:31:13.987+0000 7f33b759e640 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x562a4a59f2f0 0x562a4a978000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:31:13.982 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:31:13.987+0000 7f33b659c640 1 -- 172.21.15.52:0/1800693004 learned_addr learned my addr 172.21.15.52:0/1800693004 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-05-27T07:31:13.982 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:31:13.987+0000 7f33b659c640 1 -- 172.21.15.52:0/1800693004 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x562a4a5c81b0 msgr2=0x562a4a97de50 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-05-27T07:31:13.982 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:31:13.987+0000 7f33b659c640 1 --2- 172.21.15.52:0/1800693004 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x562a4a5c81b0 0x562a4a97de50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:31:13.982 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:31:13.987+0000 7f33b659c640 1 -- 172.21.15.52:0/1800693004 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x562a4a59f2f0 msgr2=0x562a4a978000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:31:13.983 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:31:13.988+0000 7f33b659c640 1 --2- 172.21.15.52:0/1800693004 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x562a4a59f2f0 0x562a4a978000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:31:13.983 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:31:13.988+0000 7f33b659c640 1 -- 172.21.15.52:0/1800693004 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({osdmap=0}) -- 0x562a4a6ac620 con 0x562a4a5a74d0 2024-05-27T07:31:13.983 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:31:13.988+0000 7f33b759e640 1 --2- 172.21.15.52:0/1800693004 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x562a4a59f2f0 0x562a4a978000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-27T07:31:13.983 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:31:13.988+0000 7f33b659c640 1 --2- 172.21.15.52:0/1800693004 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x562a4a5a74d0 0x562a4a978540 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7f33ac00c200 tx=0x7f33ac00c6c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=e068333bb059fa05 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:31:13.983 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:31:13.988+0000 7f339aff5640 10 client.0 ms_handle_connect on v2:172.21.15.173:3300/0 2024-05-27T07:31:13.984 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:31:13.988+0000 7f339aff5640 1 -- 172.21.15.52:0/1800693004 <== mon.1 v2:172.21.15.173:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f33ac015070 con 0x562a4a5a74d0 2024-05-27T07:31:13.984 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:31:13.988+0000 7f339aff5640 1 -- 172.21.15.52:0/1800693004 <== mon.1 v2:172.21.15.173:3300/0 2 ==== osd_map(116..116 src has 1..116) ==== 9379+0+0 (secure 0 0 0) 0x7f33ac011800 con 0x562a4a5a74d0 2024-05-27T07:31:13.984 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:31:13.988+0000 7f33c3ff8740 1 -- 172.21.15.52:0/1800693004 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f33b8069cb0 con 0x562a4a5a74d0 2024-05-27T07:31:13.985 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:31:13.989+0000 7f339aff5640 1 -- 172.21.15.52:0/1800693004 <== mon.1 v2:172.21.15.173:3300/0 3 ==== mon_get_version_reply(handle=1 version=99) ==== 24+0+0 (secure 0 0 0) 0x7f33ac021050 con 0x562a4a5a74d0 2024-05-27T07:31:13.985 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:31:13.989+0000 7f33c3ff8740 10 client.26113 fetch_fsmap learned FSMap version 99 2024-05-27T07:31:13.985 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:31:13.989+0000 7f33c3ff8740 1 -- 172.21.15.52:0/1800693004 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({config=0+,fsmap=99,monmap=4+}) -- 0x562a4a986d50 con 0x562a4a5a74d0 2024-05-27T07:31:13.985 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:31:13.989+0000 7f339aff5640 1 -- 172.21.15.52:0/1800693004 <== mon.1 v2:172.21.15.173:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f33b8069cb0 con 0x562a4a5a74d0 2024-05-27T07:31:13.985 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:31:13.991+0000 7f339aff5640 1 -- 172.21.15.52:0/1800693004 <== mon.1 v2:172.21.15.173:3300/0 5 ==== fsmap(e 99) ==== 4792+0+0 (secure 0 0 0) 0x7f33ac019af0 con 0x562a4a5a74d0 2024-05-27T07:31:13.985 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:31:13.991+0000 7f33c3ff8740 10 client.26113 fetch_fsmap finished waiting for FSMap version 99 2024-05-27T07:31:13.985 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:31:13.991+0000 7f33c3ff8740 10 client.26113 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-05-27T07:31:13.985 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:31:13.991+0000 7f33c3ff8740 20 client.26113 populate_metadata read hostname 'smithi052' 2024-05-27T07:31:13.985 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:31:13.991+0000 7f33c3ff8740 1 --2- 172.21.15.52:0/1800693004 >> [v2:172.21.15.105:6838/500348151,v1:172.21.15.105:6839/500348151] conn(0x562a4a586d80 0x562a4a98a7d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:31:13.985 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:31:13.991+0000 7f33c3ff8740 4 client.26113 mds_command: new command op to 24527 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-27T07:31:13.985 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:31:13.991+0000 7f33c3ff8740 1 -- 172.21.15.52:0/1800693004 --> [v2:172.21.15.105:6838/500348151,v1:172.21.15.105:6839/500348151] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x562a4a8ddb80 con 0x562a4a586d80 2024-05-27T07:31:13.985 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:31:13.991+0000 7f33b759e640 1 --2- 172.21.15.52:0/1800693004 >> [v2:172.21.15.105:6838/500348151,v1:172.21.15.105:6839/500348151] conn(0x562a4a586d80 0x562a4a98a7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:31:13.986 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:31:13.991+0000 7f33b759e640 1 --2- 172.21.15.52:0/1800693004 >> [v2:172.21.15.105:6838/500348151,v1:172.21.15.105:6839/500348151] conn(0x562a4a586d80 0x562a4a98a7d0 crc :-1 s=READY pgs=113 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=e499dffe521d760b server_cookie=b7374aa1412f004f in_seq=0 out_seq=0 2024-05-27T07:31:13.986 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:31:13.991+0000 7f339aff5640 10 client.26113 ms_handle_connect on v2:172.21.15.105:6838/500348151 2024-05-27T07:31:13.987 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:31:13.994+0000 7f339aff5640 1 -- 172.21.15.52:0/1800693004 <== mds.0 v2:172.21.15.105:6838/500348151 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24041 (crc 0 0 0) 0x562a4a8ddb80 con 0x562a4a586d80 2024-05-27T07:31:13.987 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:31:13.994+0000 7f339aff5640 10 client.26113 handle_command_reply: tid=0 2024-05-27T07:31:13.988 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:31:13.994+0000 7f33c3ff8740 1 client.26113 shutdown 2024-05-27T07:31:13.988 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:31:13.995+0000 7f33c3ff8740 1 -- 172.21.15.52:0/1800693004 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x562a4a5a74d0 msgr2=0x562a4a978540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:31:13.988 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:31:13.995+0000 7f33c3ff8740 1 --2- 172.21.15.52:0/1800693004 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x562a4a5a74d0 0x562a4a978540 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7f33ac00c200 tx=0x7f33ac00c6c0 comp rx=0 tx=0).stop 2024-05-27T07:31:13.988 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:31:13.995+0000 7f33c3ff8740 1 -- 172.21.15.52:0/1800693004 shutdown_connections 2024-05-27T07:31:13.988 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:31:13.995+0000 7f33c3ff8740 1 --2- 172.21.15.52:0/1800693004 >> [v2:172.21.15.105:6838/500348151,v1:172.21.15.105:6839/500348151] conn(0x562a4a586d80 0x562a4a98a7d0 crc :-1 s=READY pgs=113 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:31:13.988 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:31:13.995+0000 7f339aff5640 0 client.26113 ms_handle_reset on v2:172.21.15.105:6838/500348151 2024-05-27T07:31:13.988 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:31:13.995+0000 7f33c3ff8740 1 --2- 172.21.15.52:0/1800693004 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x562a4a5c81b0 0x562a4a97de50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:31:13.988 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:31:13.995+0000 7f33c3ff8740 1 --2- 172.21.15.52:0/1800693004 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x562a4a5a74d0 0x562a4a978540 unknown :-1 s=CLOSED pgs=137 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:31:13.988 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:31:13.995+0000 7f33c3ff8740 1 --2- 172.21.15.52:0/1800693004 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x562a4a59f2f0 0x562a4a978000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:31:13.988 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:31:13.995+0000 7f33c3ff8740 1 -- 172.21.15.52:0/1800693004 >> 172.21.15.52:0/1800693004 conn(0x562a4a5bc300 msgr2=0x562a4a96bc00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:31:13.988 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:31:13.995+0000 7f33c3ff8740 1 -- 172.21.15.52:0/1800693004 shutdown_connections 2024-05-27T07:31:13.988 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:31:13.995+0000 7f33c3ff8740 1 -- 172.21.15.52:0/1800693004 wait complete. 2024-05-27T07:31:13.989 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:31:13.995+0000 7f33c3ff8740 20 client.26113 trim_cache size 0 max 16384 2024-05-27T07:31:14.011 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:31:14.019+0000 7f33c3ff8740 1 Processor -- start 2024-05-27T07:31:14.012 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:31:14.019+0000 7f33c3ff8740 1 -- start start 2024-05-27T07:31:14.012 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:31:14.020+0000 7f33c3ff8740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x562a4a586d80 0x562a4a9717d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:31:14.012 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:31:14.020+0000 7f33c3ff8740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x562a4a59f2f0 0x562a4a984af0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:31:14.012 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:31:14.020+0000 7f33c3ff8740 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x562a4a5a74d0 0x562a4a969a30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:31:14.012 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:31:14.020+0000 7f33c3ff8740 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x7f33b0142860 con 0x562a4a5a74d0 2024-05-27T07:31:14.012 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:31:14.020+0000 7f33c3ff8740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f33b01405c0 con 0x562a4a586d80 2024-05-27T07:31:14.012 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:31:14.020+0000 7f33c3ff8740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f33b0142190 con 0x562a4a59f2f0 2024-05-27T07:31:14.013 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:31:14.020+0000 7f33b659c640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x562a4a59f2f0 0x562a4a984af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:31:14.013 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:31:14.020+0000 7f33b759e640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x562a4a586d80 0x562a4a9717d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:31:14.013 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:31:14.020+0000 7f33b659c640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x562a4a59f2f0 0x562a4a984af0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.173:3300/0 says I am v2:172.21.15.52:33496/0 (socket says 172.21.15.52:33496) 2024-05-27T07:31:14.013 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:31:14.020+0000 7f33b759e640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x562a4a586d80 0x562a4a9717d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.52:44410/0 (socket says 172.21.15.52:44410) 2024-05-27T07:31:14.013 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:31:14.020+0000 7f33b659c640 1 -- 172.21.15.52:0/1944995352 learned_addr learned my addr 172.21.15.52:0/1944995352 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-05-27T07:31:14.013 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:31:14.020+0000 7f33b759e640 1 --2- 172.21.15.52:0/1944995352 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x562a4a5a74d0 0x562a4a969a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:31:14.014 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:31:14.020+0000 7f33b759e640 1 -- 172.21.15.52:0/1944995352 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x562a4a59f2f0 msgr2=0x562a4a984af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:31:14.014 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:31:14.020+0000 7f33b759e640 1 --2- 172.21.15.52:0/1944995352 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x562a4a59f2f0 0x562a4a984af0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:31:14.014 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:31:14.020+0000 7f33b759e640 1 -- 172.21.15.52:0/1944995352 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x562a4a5a74d0 msgr2=0x562a4a969a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:31:14.014 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:31:14.020+0000 7f33b759e640 1 --2- 172.21.15.52:0/1944995352 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x562a4a5a74d0 0x562a4a969a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:31:14.014 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:31:14.020+0000 7f33b759e640 1 -- 172.21.15.52:0/1944995352 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x562a4a976340 con 0x562a4a586d80 2024-05-27T07:31:14.014 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:31:14.020+0000 7f33b759e640 1 --2- 172.21.15.52:0/1944995352 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x562a4a586d80 0x562a4a9717d0 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7f33b8068640 tx=0x7f33b80693d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=80027871445af283 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:31:14.014 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:31:14.021+0000 7f339cff9640 1 -- 172.21.15.52:0/1944995352 <== mon.2 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f33b809e070 con 0x562a4a586d80 2024-05-27T07:31:14.014 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:31:14.021+0000 7f339cff9640 1 -- 172.21.15.52:0/1944995352 <== mon.2 v2:172.21.15.105:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f33b80bf040 con 0x562a4a586d80 2024-05-27T07:31:14.014 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:31:14.021+0000 7f339cff9640 1 -- 172.21.15.52:0/1944995352 <== mon.2 v2:172.21.15.105:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f33b80b2050 con 0x562a4a586d80 2024-05-27T07:31:14.018 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:31:14.022+0000 7f33c3ff8740 1 -- 172.21.15.52:0/1944995352 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x562a4a586d80 msgr2=0x562a4a9717d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:31:14.018 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:31:14.022+0000 7f33c3ff8740 1 --2- 172.21.15.52:0/1944995352 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x562a4a586d80 0x562a4a9717d0 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7f33b8068640 tx=0x7f33b80693d0 comp rx=0 tx=0).stop 2024-05-27T07:31:14.019 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:31:14.022+0000 7f33c3ff8740 1 -- 172.21.15.52:0/1944995352 shutdown_connections 2024-05-27T07:31:14.019 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:31:14.022+0000 7f33c3ff8740 1 --2- 172.21.15.52:0/1944995352 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x562a4a5a74d0 0x562a4a969a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:31:14.019 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:31:14.022+0000 7f33c3ff8740 1 --2- 172.21.15.52:0/1944995352 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x562a4a59f2f0 0x562a4a984af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:31:14.019 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:31:14.022+0000 7f33c3ff8740 1 --2- 172.21.15.52:0/1944995352 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x562a4a586d80 0x562a4a9717d0 unknown :-1 s=CLOSED pgs=175 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:31:14.019 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:31:14.022+0000 7f33c3ff8740 1 -- 172.21.15.52:0/1944995352 >> 172.21.15.52:0/1944995352 conn(0x562a4a5bc300 msgr2=0x562a4a969320 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:31:14.019 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:31:14.022+0000 7f33c3ff8740 1 -- 172.21.15.52:0/1944995352 shutdown_connections 2024-05-27T07:31:14.019 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:31:14.022+0000 7f33c3ff8740 1 -- 172.21.15.52:0/1944995352 wait complete. 2024-05-27T07:31:14.019 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:31:14.023+0000 7f33c3ff8740 1 -- start start 2024-05-27T07:31:14.019 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:31:14.023+0000 7f33c3ff8740 1 Processor -- start 2024-05-27T07:31:14.019 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:31:14.024+0000 7f33c3ff8740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x562a4a586d80 0x562a4a97d3d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:31:14.020 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:31:14.024+0000 7f33c3ff8740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x562a4a59f2f0 0x562a4a97d910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:31:14.020 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:31:14.024+0000 7f33c3ff8740 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x562a4a5a74d0 0x562a4a97de50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:31:14.020 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:31:14.024+0000 7f33c3ff8740 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x7f33b0141360 con 0x562a4a5a74d0 2024-05-27T07:31:14.020 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:31:14.024+0000 7f33c3ff8740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f33b0140c90 con 0x562a4a586d80 2024-05-27T07:31:14.020 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:31:14.024+0000 7f33c3ff8740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f33b0141a30 con 0x562a4a59f2f0 2024-05-27T07:31:14.020 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:31:14.024+0000 7f33c3ff8740 10 client.0 _ll_register_callbacks cb 0x7f33ac00a0f0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T07:31:14.020 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:31:14.024+0000 7f33b759e640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x562a4a586d80 0x562a4a97d3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:31:14.020 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:31:14.024+0000 7f33b759e640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x562a4a586d80 0x562a4a97d3d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.52:44422/0 (socket says 172.21.15.52:44422) 2024-05-27T07:31:14.021 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:31:14.024+0000 7f33b759e640 1 -- 172.21.15.52:0/1702810386 learned_addr learned my addr 172.21.15.52:0/1702810386 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-05-27T07:31:14.021 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:31:14.024+0000 7f33b759e640 1 --2- 172.21.15.52:0/1702810386 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x562a4a5a74d0 0x562a4a97de50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:31:14.021 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:31:14.024+0000 7f33b759e640 1 -- 172.21.15.52:0/1702810386 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x562a4a59f2f0 msgr2=0x562a4a97d910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:31:14.021 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:31:14.024+0000 7f33b759e640 1 --2- 172.21.15.52:0/1702810386 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x562a4a59f2f0 0x562a4a97d910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:31:14.021 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:31:14.024+0000 7f33b759e640 1 -- 172.21.15.52:0/1702810386 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x562a4a5a74d0 msgr2=0x562a4a97de50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:31:14.021 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:31:14.024+0000 7f33b759e640 1 --2- 172.21.15.52:0/1702810386 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x562a4a5a74d0 0x562a4a97de50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:31:14.021 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:31:14.024+0000 7f33b759e640 1 -- 172.21.15.52:0/1702810386 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({osdmap=0}) -- 0x562a4a9f1040 con 0x562a4a586d80 2024-05-27T07:31:14.021 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:31:14.025+0000 7f33b759e640 1 --2- 172.21.15.52:0/1702810386 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x562a4a586d80 0x562a4a97d3d0 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7f33b8068b80 tx=0x7f33b8065e80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=454d46de939ba580 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:31:14.021 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:31:14.025+0000 7f339aff5640 10 client.0 ms_handle_connect on v2:172.21.15.105:3300/0 2024-05-27T07:31:14.021 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:31:14.025+0000 7f339aff5640 1 -- 172.21.15.52:0/1702810386 <== mon.2 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f33b80b2070 con 0x562a4a586d80 2024-05-27T07:31:14.021 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:31:14.025+0000 7f33c3ff8740 1 -- 172.21.15.52:0/1702810386 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f33b80bf040 con 0x562a4a586d80 2024-05-27T07:31:14.021 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:31:14.025+0000 7f339aff5640 1 -- 172.21.15.52:0/1702810386 <== mon.2 v2:172.21.15.105:3300/0 2 ==== osd_map(116..116 src has 1..116) ==== 9379+0+0 (secure 0 0 0) 0x7f33b8099810 con 0x562a4a586d80 2024-05-27T07:31:14.022 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:31:14.025+0000 7f339aff5640 1 -- 172.21.15.52:0/1702810386 <== mon.2 v2:172.21.15.105:3300/0 3 ==== mon_get_version_reply(handle=1 version=99) ==== 24+0+0 (secure 0 0 0) 0x7f33b809e040 con 0x562a4a586d80 2024-05-27T07:31:14.022 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:31:14.025+0000 7f33c3ff8740 10 client.26234 fetch_fsmap learned FSMap version 99 2024-05-27T07:31:14.022 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:31:14.025+0000 7f33c3ff8740 1 -- 172.21.15.52:0/1702810386 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,fsmap=99,monmap=4+}) -- 0x562a4a9f2290 con 0x562a4a586d80 2024-05-27T07:31:14.022 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:31:14.025+0000 7f339aff5640 1 -- 172.21.15.52:0/1702810386 <== mon.2 v2:172.21.15.105:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f33b80bf040 con 0x562a4a586d80 2024-05-27T07:31:14.022 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:31:14.025+0000 7f339aff5640 1 -- 172.21.15.52:0/1702810386 <== mon.2 v2:172.21.15.105:3300/0 5 ==== fsmap(e 99) ==== 4792+0+0 (secure 0 0 0) 0x7f33b80ac070 con 0x562a4a586d80 2024-05-27T07:31:14.022 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:31:14.025+0000 7f33c3ff8740 10 client.26234 fetch_fsmap finished waiting for FSMap version 99 2024-05-27T07:31:14.022 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:31:14.025+0000 7f33c3ff8740 10 client.26234 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-05-27T07:31:14.022 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:31:14.025+0000 7f33c3ff8740 20 client.26234 populate_metadata read hostname 'smithi052' 2024-05-27T07:31:14.022 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:31:14.025+0000 7f33c3ff8740 1 --2- 172.21.15.52:0/1702810386 >> [v2:172.21.15.105:6838/500348151,v1:172.21.15.105:6839/500348151] conn(0x562a4a5c81b0 0x562a4a9f4810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:31:14.022 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:31:14.025+0000 7f33c3ff8740 4 client.26234 mds_command: new command op to 24527 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-05-27T07:31:14.022 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:31:14.025+0000 7f33c3ff8740 1 -- 172.21.15.52:0/1702810386 --> [v2:172.21.15.105:6838/500348151,v1:172.21.15.105:6839/500348151] -- command(tid 0: {"prefix": "flush journal"}) -- 0x562a4a8d7360 con 0x562a4a5c81b0 2024-05-27T07:31:14.022 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:31:14.026+0000 7f33b659c640 1 --2- 172.21.15.52:0/1702810386 >> [v2:172.21.15.105:6838/500348151,v1:172.21.15.105:6839/500348151] conn(0x562a4a5c81b0 0x562a4a9f4810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:31:14.023 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:31:14.026+0000 7f339aff5640 10 client.26234 ms_handle_connect on v2:172.21.15.105:6838/500348151 2024-05-27T07:31:14.023 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:31:14.026+0000 7f33b659c640 1 --2- 172.21.15.52:0/1702810386 >> [v2:172.21.15.105:6838/500348151,v1:172.21.15.105:6839/500348151] conn(0x562a4a5c81b0 0x562a4a9f4810 crc :-1 s=READY pgs=114 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=fe88db391bc1a259 server_cookie=6ac458b8d62a063d in_seq=0 out_seq=0 2024-05-27T07:31:15.218 INFO:tasks.workunit.client.0.smithi052.stdout: 122 721542 233 512945 438 406209 752 2024-05-27T07:31:16.395 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:31:16 smithi105 ceph-mon[35977]: pgmap v187: 129 pgs: 129 active+clean; 458 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 MiB/s rd, 5.6 MiB/s wr, 414 op/s 2024-05-27T07:31:16.482 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:31:16 smithi052 ceph-mon[31955]: pgmap v187: 129 pgs: 129 active+clean; 458 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 MiB/s rd, 5.6 MiB/s wr, 414 op/s 2024-05-27T07:31:16.604 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:31:16 smithi173 ceph-mon[37000]: pgmap v187: 129 pgs: 129 active+clean; 458 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 MiB/s rd, 5.6 MiB/s wr, 414 op/s 2024-05-27T07:31:17.421 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:31:17.428+0000 7f339aff5640 1 -- 172.21.15.52:0/1702810386 <== mds.0 v2:172.21.15.105:6838/500348151 1 ==== command_reply(tid 0: 0 ) ==== 8+0+25 (crc 0 0 0) 0x562a4a8d7360 con 0x562a4a5c81b0 2024-05-27T07:31:17.422 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:31:17.428+0000 7f339aff5640 10 client.26234 handle_command_reply: tid=0 2024-05-27T07:31:17.422 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:31:17.428+0000 7f33c3ff8740 1 client.26234 shutdown 2024-05-27T07:31:17.422 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:31:17.429+0000 7f33c3ff8740 1 -- 172.21.15.52:0/1702810386 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x562a4a586d80 msgr2=0x562a4a97d3d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:31:17.423 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:31:17.429+0000 7f33c3ff8740 1 --2- 172.21.15.52:0/1702810386 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x562a4a586d80 0x562a4a97d3d0 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7f33b8068b80 tx=0x7f33b8065e80 comp rx=0 tx=0).stop 2024-05-27T07:31:17.423 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:31:17.429+0000 7f33c3ff8740 1 -- 172.21.15.52:0/1702810386 shutdown_connections 2024-05-27T07:31:17.423 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:31:17.429+0000 7f33c3ff8740 1 --2- 172.21.15.52:0/1702810386 >> [v2:172.21.15.105:6838/500348151,v1:172.21.15.105:6839/500348151] conn(0x562a4a5c81b0 0x562a4a9f4810 crc :-1 s=READY pgs=114 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:31:17.423 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:31:17.429+0000 7f33c3ff8740 1 --2- 172.21.15.52:0/1702810386 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x562a4a5a74d0 0x562a4a97de50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:31:17.423 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:31:17.429+0000 7f33c3ff8740 1 --2- 172.21.15.52:0/1702810386 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x562a4a59f2f0 0x562a4a97d910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:31:17.423 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:31:17.429+0000 7f33c3ff8740 1 --2- 172.21.15.52:0/1702810386 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x562a4a586d80 0x562a4a97d3d0 unknown :-1 s=CLOSED pgs=176 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:31:17.423 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:31:17.429+0000 7f339aff5640 0 client.26234 ms_handle_reset on v2:172.21.15.105:6838/500348151 2024-05-27T07:31:17.423 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:31:17.429+0000 7f33c3ff8740 1 -- 172.21.15.52:0/1702810386 >> 172.21.15.52:0/1702810386 conn(0x562a4a5bc300 msgr2=0x562a4a978770 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:31:17.424 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:31:17.430+0000 7f33c3ff8740 1 -- 172.21.15.52:0/1702810386 shutdown_connections 2024-05-27T07:31:17.424 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:31:17.430+0000 7f33c3ff8740 1 -- 172.21.15.52:0/1702810386 wait complete. 2024-05-27T07:31:17.424 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:31:17.431+0000 7f33c3ff8740 20 client.26234 trim_cache size 0 max 16384 2024-05-27T07:31:17.425 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stdout:{ 2024-05-27T07:31:17.425 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stdout: "return_code": 0 2024-05-27T07:31:17.425 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stdout:} 2024-05-27T07:31:17.426 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:31:17.432+0000 7f33bc93e640 1 -- 172.21.15.52:0/2729144489 >> [v2:172.21.15.52:6800/1822318384,v1:172.21.15.52:6801/1822318384] conn(0x7f33a00b29e0 msgr2=0x7f33a00b4ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:31:17.426 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:31:17.432+0000 7f33bc93e640 1 --2- 172.21.15.52:0/2729144489 >> [v2:172.21.15.52:6800/1822318384,v1:172.21.15.52:6801/1822318384] conn(0x7f33a00b29e0 0x7f33a00b4ef0 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7f33a80defb0 tx=0x7f33a80e1e10 comp rx=0 tx=0).stop 2024-05-27T07:31:17.426 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:31:17.432+0000 7f33bc93e640 1 -- 172.21.15.52:0/2729144489 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f33b00960c0 msgr2=0x7f33b012b4b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:31:17.426 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:31:17.432+0000 7f33bc93e640 1 --2- 172.21.15.52:0/2729144489 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f33b00960c0 0x7f33b012b4b0 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7f33a8009a00 tx=0x7f33a8009ec0 comp rx=0 tx=0).stop 2024-05-27T07:31:17.427 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:31:17.433+0000 7f33bc93e640 1 -- 172.21.15.52:0/2729144489 shutdown_connections 2024-05-27T07:31:17.427 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:31:17.433+0000 7f33bc93e640 1 --2- 172.21.15.52:0/2729144489 >> [v2:172.21.15.52:6800/1822318384,v1:172.21.15.52:6801/1822318384] conn(0x7f33a00b29e0 0x7f33a00b4ef0 unknown :-1 s=CLOSED pgs=144 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:31:17.427 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:31:17.433+0000 7f33bc93e640 1 --2- 172.21.15.52:0/2729144489 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f33b009bec0 0x7f33b012fcd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:31:17.427 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:31:17.433+0000 7f33bc93e640 1 --2- 172.21.15.52:0/2729144489 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f33b009ba20 0x7f33b012ba10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:31:17.427 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:31:17.433+0000 7f33bc93e640 1 --2- 172.21.15.52:0/2729144489 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f33b00960c0 0x7f33b012b4b0 unknown :-1 s=CLOSED pgs=174 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:31:17.427 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:31:17.433+0000 7f33bc93e640 1 -- 172.21.15.52:0/2729144489 >> 172.21.15.52:0/2729144489 conn(0x7f33b000b010 msgr2=0x7f33b0093b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:31:17.428 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:31:17.434+0000 7f33bc93e640 1 -- 172.21.15.52:0/2729144489 shutdown_connections 2024-05-27T07:31:17.428 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:31:17.434+0000 7f33bc93e640 1 -- 172.21.15.52:0/2729144489 wait complete. 2024-05-27T07:31:18.442 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:31:18 smithi105 ceph-mon[35977]: pgmap v188: 129 pgs: 129 active+clean; 462 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 MiB/s rd, 3.5 MiB/s wr, 462 op/s 2024-05-27T07:31:18.481 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:31:18 smithi052 ceph-mon[31955]: pgmap v188: 129 pgs: 129 active+clean; 462 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 MiB/s rd, 3.5 MiB/s wr, 462 op/s 2024-05-27T07:31:18.481 INFO:journalctl@ceph.mds.j.smithi052.stdout:May 27 07:31:17 smithi052 systemd[1]: ceph-73df2636-1bf8-11ef-bc9b-c7b262605968@mds.j.service: Scheduled restart job, restart counter is at 6. 2024-05-27T07:31:18.482 INFO:journalctl@ceph.mds.j.smithi052.stdout:May 27 07:31:17 smithi052 systemd[1]: Stopped Ceph mds.j for 73df2636-1bf8-11ef-bc9b-c7b262605968. 2024-05-27T07:31:18.482 INFO:journalctl@ceph.mds.j.smithi052.stdout:May 27 07:31:17 smithi052 systemd[1]: ceph-73df2636-1bf8-11ef-bc9b-c7b262605968@mds.j.service: Start request repeated too quickly. 2024-05-27T07:31:18.482 INFO:journalctl@ceph.mds.j.smithi052.stdout:May 27 07:31:17 smithi052 systemd[1]: ceph-73df2636-1bf8-11ef-bc9b-c7b262605968@mds.j.service: Failed with result 'exit-code'. 2024-05-27T07:31:18.482 INFO:journalctl@ceph.mds.j.smithi052.stdout:May 27 07:31:17 smithi052 systemd[1]: Failed to start Ceph mds.j for 73df2636-1bf8-11ef-bc9b-c7b262605968. 2024-05-27T07:31:18.604 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:31:18 smithi173 ceph-mon[37000]: pgmap v188: 129 pgs: 129 active+clean; 462 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 MiB/s rd, 3.5 MiB/s wr, 462 op/s 2024-05-27T07:31:19.442 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:31:19 smithi105 ceph-mon[35977]: from='mgr.15333 172.21.15.52:0/416389082' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-27T07:31:19.490 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:31:19 smithi052 ceph-mon[31955]: from='mgr.15333 172.21.15.52:0/416389082' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-27T07:31:19.604 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:31:19 smithi173 ceph-mon[37000]: from='mgr.15333 172.21.15.52:0/416389082' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-27T07:31:20.442 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:31:20 smithi105 ceph-mon[35977]: pgmap v189: 129 pgs: 129 active+clean; 445 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.7 MiB/s rd, 3.8 MiB/s wr, 908 op/s 2024-05-27T07:31:20.481 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:31:20 smithi052 ceph-mon[31955]: pgmap v189: 129 pgs: 129 active+clean; 445 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.7 MiB/s rd, 3.8 MiB/s wr, 908 op/s 2024-05-27T07:31:20.604 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:31:20 smithi173 ceph-mon[37000]: pgmap v189: 129 pgs: 129 active+clean; 445 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.7 MiB/s rd, 3.8 MiB/s wr, 908 op/s 2024-05-27T07:31:21.777 DEBUG:teuthology.orchestra.run.smithi052:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:38f9d900c83c63799fbdbe61acc9a11b0d3554a6 shell --fsid 73df2636-1bf8-11ef-bc9b-c7b262605968 -- ceph tell mds.1:0 scrub status 2024-05-27T07:31:22.175 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:31:22 smithi052 ceph-mon[31955]: pgmap v190: 129 pgs: 129 active+clean; 437 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.7 MiB/s rd, 2.1 MiB/s wr, 985 op/s 2024-05-27T07:31:22.175 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:31:22 smithi052 ceph-mon[31955]: from='mgr.15333 172.21.15.52:0/416389082' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T07:31:22.175 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:31:22 smithi052 ceph-mon[31955]: from='mgr.15333 172.21.15.52:0/416389082' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-27T07:31:22.175 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:31:22 smithi052 ceph-mon[31955]: from='mgr.15333 172.21.15.52:0/416389082' entity='mgr.x' 2024-05-27T07:31:22.430 INFO:teuthology.orchestra.run.smithi052.stderr:Inferring config /var/lib/ceph/73df2636-1bf8-11ef-bc9b-c7b262605968/mon.a/config 2024-05-27T07:31:22.442 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:31:22 smithi105 ceph-mon[35977]: pgmap v190: 129 pgs: 129 active+clean; 437 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.7 MiB/s rd, 2.1 MiB/s wr, 985 op/s 2024-05-27T07:31:22.442 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:31:22 smithi105 ceph-mon[35977]: from='mgr.15333 172.21.15.52:0/416389082' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T07:31:22.442 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:31:22 smithi105 ceph-mon[35977]: from='mgr.15333 172.21.15.52:0/416389082' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-27T07:31:22.442 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:31:22 smithi105 ceph-mon[35977]: from='mgr.15333 172.21.15.52:0/416389082' entity='mgr.x' 2024-05-27T07:31:22.605 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:31:22 smithi173 ceph-mon[37000]: pgmap v190: 129 pgs: 129 active+clean; 437 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.7 MiB/s rd, 2.1 MiB/s wr, 985 op/s 2024-05-27T07:31:22.605 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:31:22 smithi173 ceph-mon[37000]: from='mgr.15333 172.21.15.52:0/416389082' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T07:31:22.606 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:31:22 smithi173 ceph-mon[37000]: from='mgr.15333 172.21.15.52:0/416389082' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-27T07:31:22.606 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:31:22 smithi173 ceph-mon[37000]: from='mgr.15333 172.21.15.52:0/416389082' entity='mgr.x' 2024-05-27T07:31:23.854 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:31:23 smithi173 ceph-mon[37000]: MDS daemon mds.j is removed because it is dead or otherwise unavailable. 2024-05-27T07:31:23.854 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:31:23 smithi173 ceph-mon[37000]: fsmap cephfs:5 1 up:standby-replay 5 up:active 2024-05-27T07:31:23.854 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:31:23 smithi173 ceph-mon[37000]: fsmap cephfs:5 5 up:active 2024-05-27T07:31:23.854 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:31:23 smithi173 ceph-mon[37000]: pgmap v191: 129 pgs: 129 active+clean; 437 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.4 MiB/s rd, 2.0 MiB/s wr, 995 op/s 2024-05-27T07:31:23.942 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:31:23 smithi105 ceph-mon[35977]: MDS daemon mds.j is removed because it is dead or otherwise unavailable. 2024-05-27T07:31:23.942 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:31:23 smithi105 ceph-mon[35977]: fsmap cephfs:5 1 up:standby-replay 5 up:active 2024-05-27T07:31:23.942 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:31:23 smithi105 ceph-mon[35977]: fsmap cephfs:5 5 up:active 2024-05-27T07:31:23.942 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:31:23 smithi105 ceph-mon[35977]: pgmap v191: 129 pgs: 129 active+clean; 437 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.4 MiB/s rd, 2.0 MiB/s wr, 995 op/s 2024-05-27T07:31:23.982 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:31:23 smithi052 ceph-mon[31955]: MDS daemon mds.j is removed because it is dead or otherwise unavailable. 2024-05-27T07:31:23.982 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:31:23 smithi052 ceph-mon[31955]: fsmap cephfs:5 1 up:standby-replay 5 up:active 2024-05-27T07:31:23.982 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:31:23 smithi052 ceph-mon[31955]: fsmap cephfs:5 5 up:active 2024-05-27T07:31:23.982 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:31:23 smithi052 ceph-mon[31955]: pgmap v191: 129 pgs: 129 active+clean; 437 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.4 MiB/s rd, 2.0 MiB/s wr, 995 op/s 2024-05-27T07:31:25.221 INFO:tasks.workunit.client.0.smithi052.stdout: 128 1458828 566 1042852 371 812479 1169 2024-05-27T07:31:25.371 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:31:25.378+0000 7fef478bb700 1 -- 172.21.15.52:0/3896405872 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fef4010a960 msgr2=0x7fef4010ad70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:31:25.372 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:31:25.378+0000 7fef478bb700 1 --2- 172.21.15.52:0/3896405872 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fef4010a960 0x7fef4010ad70 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7fef3800b0a0 tx=0x7fef3800b3b0 comp rx=0 tx=0).stop 2024-05-27T07:31:25.375 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:31:25.382+0000 7fef478bb700 1 -- 172.21.15.52:0/3896405872 shutdown_connections 2024-05-27T07:31:25.375 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:31:25.382+0000 7fef478bb700 1 --2- 172.21.15.52:0/3896405872 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fef4010b2b0 0x7fef40117830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:31:25.375 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:31:25.382+0000 7fef478bb700 1 --2- 172.21.15.52:0/3896405872 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fef40110990 0x7fef40110da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:31:25.375 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:31:25.382+0000 7fef478bb700 1 --2- 172.21.15.52:0/3896405872 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fef4010a960 0x7fef4010ad70 unknown :-1 s=CLOSED pgs=177 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:31:25.376 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:31:25.382+0000 7fef478bb700 1 -- 172.21.15.52:0/3896405872 >> 172.21.15.52:0/3896405872 conn(0x7fef40076cb0 msgr2=0x7fef400770c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:31:25.380 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:31:25.387+0000 7fef478bb700 1 -- 172.21.15.52:0/3896405872 shutdown_connections 2024-05-27T07:31:25.385 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:31:25.390+0000 7fef478bb700 1 -- 172.21.15.52:0/3896405872 wait complete. 2024-05-27T07:31:25.386 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:31:25.391+0000 7fef478bb700 1 Processor -- start 2024-05-27T07:31:25.386 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:31:25.391+0000 7fef478bb700 1 -- start start 2024-05-27T07:31:25.386 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:31:25.391+0000 7fef478bb700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fef4010a960 0x7fef40084170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:31:25.386 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:31:25.392+0000 7fef478bb700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fef4010b2b0 0x7fef4007d2a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:31:25.386 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:31:25.392+0000 7fef478bb700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fef40110990 0x7fef4007d860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:31:25.386 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:31:25.392+0000 7fef478bb700 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x7fef4011ab30 con 0x7fef4010a960 2024-05-27T07:31:25.386 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:31:25.392+0000 7fef478bb700 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7fef4011a9b0 con 0x7fef4010b2b0 2024-05-27T07:31:25.386 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:31:25.392+0000 7fef478bb700 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7fef4011acb0 con 0x7fef40110990 2024-05-27T07:31:25.391 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:31:25.395+0000 7fef45e58700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fef40110990 0x7fef4007d860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:31:25.391 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:31:25.395+0000 7fef45657700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fef4010a960 0x7fef40084170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:31:25.391 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:31:25.395+0000 7fef45657700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fef4010a960 0x7fef40084170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.52:46998/0 (socket says 172.21.15.52:46998) 2024-05-27T07:31:25.391 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:31:25.396+0000 7fef44e56700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fef4010b2b0 0x7fef4007d2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:31:25.391 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:31:25.396+0000 7fef45657700 1 -- 172.21.15.52:0/628693250 learned_addr learned my addr 172.21.15.52:0/628693250 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-05-27T07:31:25.391 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:31:25.395+0000 7fef45e58700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fef40110990 0x7fef4007d860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.173:3300/0 says I am v2:172.21.15.52:39638/0 (socket says 172.21.15.52:39638) 2024-05-27T07:31:25.391 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:31:25.397+0000 7fef45e58700 1 -- 172.21.15.52:0/628693250 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fef4010b2b0 msgr2=0x7fef4007d2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:31:25.392 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:31:25.397+0000 7fef45e58700 1 --2- 172.21.15.52:0/628693250 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fef4010b2b0 0x7fef4007d2a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:31:25.392 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:31:25.397+0000 7fef45e58700 1 -- 172.21.15.52:0/628693250 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fef4010a960 msgr2=0x7fef40084170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:31:25.392 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:31:25.397+0000 7fef45e58700 1 --2- 172.21.15.52:0/628693250 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fef4010a960 0x7fef40084170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:31:25.392 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:31:25.397+0000 7fef45e58700 1 -- 172.21.15.52:0/628693250 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fef4007e0c0 con 0x7fef40110990 2024-05-27T07:31:25.392 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:31:25.397+0000 7fef45657700 1 --2- 172.21.15.52:0/628693250 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fef4010a960 0x7fef40084170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-27T07:31:25.392 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:31:25.398+0000 7fef45e58700 1 --2- 172.21.15.52:0/628693250 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fef40110990 0x7fef4007d860 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7fef3000d720 tx=0x7fef3000da30 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:31:25.393 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:31:25.398+0000 7fef367fc700 1 -- 172.21.15.52:0/628693250 <== mon.1 v2:172.21.15.173:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fef30016070 con 0x7fef40110990 2024-05-27T07:31:25.393 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:31:25.398+0000 7fef478bb700 1 -- 172.21.15.52:0/628693250 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fef40081dc0 con 0x7fef40110990 2024-05-27T07:31:25.393 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:31:25.398+0000 7fef478bb700 1 -- 172.21.15.52:0/628693250 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fef40082300 con 0x7fef40110990 2024-05-27T07:31:25.393 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:31:25.399+0000 7fef367fc700 1 -- 172.21.15.52:0/628693250 <== mon.1 v2:172.21.15.173:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fef3000f690 con 0x7fef40110990 2024-05-27T07:31:25.393 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:31:25.399+0000 7fef367fc700 1 -- 172.21.15.52:0/628693250 <== mon.1 v2:172.21.15.173:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fef30003ae0 con 0x7fef40110990 2024-05-27T07:31:25.406 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:31:25.407+0000 7fef562d0b80 1 Processor -- start 2024-05-27T07:31:25.406 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:31:25.407+0000 7fef562d0b80 1 -- start start 2024-05-27T07:31:25.406 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:31:25.408+0000 7fef562d0b80 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55bb6af2ae40 0x55bb6b0ae710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:31:25.406 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:31:25.408+0000 7fef562d0b80 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55bb6b0aec50 0x55bb6b0a8a40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:31:25.406 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:31:25.408+0000 7fef562d0b80 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55bb6b0a8f80 0x55bb6b0a9390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:31:25.406 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:31:25.409+0000 7fef562d0b80 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x55bb6afd7c90 con 0x55bb6af2ae40 2024-05-27T07:31:25.406 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:31:25.409+0000 7fef562d0b80 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x55bb6afc7830 con 0x55bb6b0a8f80 2024-05-27T07:31:25.406 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:31:25.409+0000 7fef562d0b80 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x55bb6afcb870 con 0x55bb6b0aec50 2024-05-27T07:31:25.407 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:31:25.409+0000 7fef45657700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55bb6b0aec50 0x55bb6b0a8a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:31:25.407 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:31:25.409+0000 7fef45657700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55bb6b0aec50 0x55bb6b0a8a40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.173:3300/0 says I am v2:172.21.15.52:39650/0 (socket says 172.21.15.52:39650) 2024-05-27T07:31:25.407 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:31:25.409+0000 7fef45657700 1 -- 172.21.15.52:0/3668104637 learned_addr learned my addr 172.21.15.52:0/3668104637 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-05-27T07:31:25.407 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:31:25.412+0000 7fef45657700 1 -- 172.21.15.52:0/3668104637 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55bb6b0a8f80 msgr2=0x55bb6b0a9390 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-05-27T07:31:25.407 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:31:25.412+0000 7fef45657700 1 --2- 172.21.15.52:0/3668104637 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55bb6b0a8f80 0x55bb6b0a9390 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:31:25.407 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:31:25.412+0000 7fef45657700 1 -- 172.21.15.52:0/3668104637 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55bb6af2ae40 msgr2=0x55bb6b0ae710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:31:25.407 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:31:25.412+0000 7fef45657700 1 --2- 172.21.15.52:0/3668104637 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55bb6af2ae40 0x55bb6b0ae710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:31:25.407 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:31:25.412+0000 7fef45657700 1 -- 172.21.15.52:0/3668104637 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55bb6af178d0 con 0x55bb6b0aec50 2024-05-27T07:31:25.410 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:31:25.413+0000 7fef45657700 1 --2- 172.21.15.52:0/3668104637 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55bb6b0aec50 0x55bb6b0a8a40 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7fef3c010550 tx=0x7fef3c010860 comp rx=0 tx=0).ready entity=mon.1 client_cookie=fc05ab06c02bd9ac server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:31:25.410 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:31:25.414+0000 7fef2b7fe700 1 -- 172.21.15.52:0/3668104637 <== mon.1 v2:172.21.15.173:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fef3c011640 con 0x55bb6b0aec50 2024-05-27T07:31:25.410 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:31:25.416+0000 7fef367fc700 1 -- 172.21.15.52:0/628693250 <== mon.1 v2:172.21.15.173:3300/0 4 ==== mgrmap(e 26) ==== 149783+0+0 (secure 0 0 0) 0x7fef30003c80 con 0x7fef40110990 2024-05-27T07:31:25.422 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:31:25.425+0000 7fef367fc700 1 --2- 172.21.15.52:0/628693250 >> [v2:172.21.15.52:6800/1822318384,v1:172.21.15.52:6801/1822318384] conn(0x7fef2c0b2ab0 0x7fef2c0b4fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:31:25.422 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:31:25.425+0000 7fef367fc700 1 -- 172.21.15.52:0/628693250 <== mon.1 v2:172.21.15.173:3300/0 5 ==== osd_map(116..116 src has 1..116) ==== 9379+0+0 (secure 0 0 0) 0x7fef300a5560 con 0x7fef40110990 2024-05-27T07:31:25.422 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:31:25.425+0000 7fef2b7fe700 1 -- 172.21.15.52:0/3668104637 <== mon.1 v2:172.21.15.173:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fef3c011c90 con 0x55bb6b0aec50 2024-05-27T07:31:25.422 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:31:25.427+0000 7fef562d0b80 1 -- 172.21.15.52:0/3668104637 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55bb6b0aec50 msgr2=0x55bb6b0a8a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:31:25.422 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:31:25.427+0000 7fef562d0b80 1 --2- 172.21.15.52:0/3668104637 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55bb6b0aec50 0x55bb6b0a8a40 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7fef3c010550 tx=0x7fef3c010860 comp rx=0 tx=0).stop 2024-05-27T07:31:25.422 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:31:25.427+0000 7fef562d0b80 1 -- 172.21.15.52:0/3668104637 shutdown_connections 2024-05-27T07:31:25.422 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:31:25.427+0000 7fef562d0b80 1 --2- 172.21.15.52:0/3668104637 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55bb6b0a8f80 0x55bb6b0a9390 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:31:25.423 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:31:25.427+0000 7fef562d0b80 1 --2- 172.21.15.52:0/3668104637 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55bb6af2ae40 0x55bb6b0ae710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:31:25.423 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:31:25.427+0000 7fef562d0b80 1 --2- 172.21.15.52:0/3668104637 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55bb6b0aec50 0x55bb6b0a8a40 unknown :-1 s=CLOSED pgs=139 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:31:25.423 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:31:25.427+0000 7fef562d0b80 1 -- 172.21.15.52:0/3668104637 >> 172.21.15.52:0/3668104637 conn(0x55bb6af85af0 msgr2=0x55bb6b0a3640 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:31:25.423 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:31:25.430+0000 7fef44e56700 1 --2- 172.21.15.52:0/628693250 >> [v2:172.21.15.52:6800/1822318384,v1:172.21.15.52:6801/1822318384] conn(0x7fef2c0b2ab0 0x7fef2c0b4fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:31:25.426 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:31:25.433+0000 7fef44e56700 1 --2- 172.21.15.52:0/628693250 >> [v2:172.21.15.52:6800/1822318384,v1:172.21.15.52:6801/1822318384] conn(0x7fef2c0b2ab0 0x7fef2c0b4fc0 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7fef3800b830 tx=0x7fef38006000 comp rx=0 tx=0).ready entity=mgr.15333 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:31:25.434 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:31:25.436+0000 7fef562d0b80 1 -- 172.21.15.52:0/3668104637 shutdown_connections 2024-05-27T07:31:25.434 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:31:25.437+0000 7fef562d0b80 1 -- 172.21.15.52:0/3668104637 wait complete. 2024-05-27T07:31:25.435 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:31:25.437+0000 7fef562d0b80 1 -- start start 2024-05-27T07:31:25.435 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:31:25.437+0000 7fef562d0b80 1 Processor -- start 2024-05-27T07:31:25.435 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:31:25.439+0000 7fef562d0b80 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55bb6af2ae40 0x55bb6b0b22b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:31:25.435 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:31:25.439+0000 7fef562d0b80 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55bb6b0a8f80 0x55bb6b0b27f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:31:25.435 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:31:25.439+0000 7fef562d0b80 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55bb6b0aec50 0x55bb6b0b2d30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:31:25.435 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:31:25.439+0000 7fef562d0b80 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x55bb6aecdfa0 con 0x55bb6af2ae40 2024-05-27T07:31:25.435 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:31:25.439+0000 7fef562d0b80 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x55bb6af697c0 con 0x55bb6b0a8f80 2024-05-27T07:31:25.435 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:31:25.439+0000 7fef562d0b80 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x55bb6b0b5790 con 0x55bb6b0aec50 2024-05-27T07:31:25.435 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:31:25.439+0000 7fef562d0b80 10 client.0 _ll_register_callbacks cb 0x55bb6af62920 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T07:31:25.435 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:31:25.439+0000 7fef45657700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55bb6af2ae40 0x55bb6b0b22b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:31:25.435 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:31:25.440+0000 7fef45657700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55bb6af2ae40 0x55bb6b0b22b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.52:47030/0 (socket says 172.21.15.52:47030) 2024-05-27T07:31:25.435 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:31:25.440+0000 7fef45657700 1 -- 172.21.15.52:0/3905552139 learned_addr learned my addr 172.21.15.52:0/3905552139 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-05-27T07:31:25.436 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:31:25.439+0000 7fef44e56700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55bb6b0a8f80 0x55bb6b0b27f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:31:25.436 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:31:25.440+0000 7fef45657700 1 -- 172.21.15.52:0/3905552139 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55bb6b0aec50 msgr2=0x55bb6b0b2d30 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-05-27T07:31:25.436 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:31:25.440+0000 7fef45657700 1 --2- 172.21.15.52:0/3905552139 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55bb6b0aec50 0x55bb6b0b2d30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:31:25.436 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:31:25.440+0000 7fef45657700 1 -- 172.21.15.52:0/3905552139 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55bb6b0a8f80 msgr2=0x55bb6b0b27f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:31:25.436 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:31:25.440+0000 7fef45657700 1 --2- 172.21.15.52:0/3905552139 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55bb6b0a8f80 0x55bb6b0b27f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:31:25.436 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:31:25.440+0000 7fef45657700 1 -- 172.21.15.52:0/3905552139 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55bb6af66690 con 0x55bb6af2ae40 2024-05-27T07:31:25.436 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:31:25.441+0000 7fef45657700 1 --2- 172.21.15.52:0/3905552139 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55bb6af2ae40 0x55bb6b0b22b0 secure :-1 s=READY pgs=654 cs=0 l=1 rev1=1 crypto rx=0x7fef3c01b900 tx=0x7fef3c01b710 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8f9003323145e141 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:31:25.437 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:31:25.444+0000 7fef28ff9700 10 client.0 ms_handle_connect on v2:172.21.15.52:3300/0 2024-05-27T07:31:25.437 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:31:25.444+0000 7fef28ff9700 1 -- 172.21.15.52:0/3905552139 <== mon.0 v2:172.21.15.52:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fef3c005300 con 0x55bb6af2ae40 2024-05-27T07:31:25.437 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:31:25.444+0000 7fef28ff9700 1 -- 172.21.15.52:0/3905552139 <== mon.0 v2:172.21.15.52:3300/0 2 ==== osd_map(116..116 src has 1..116) ==== 9379+0+0 (secure 0 0 0) 0x7fef3c0054a0 con 0x55bb6af2ae40 2024-05-27T07:31:25.438 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:31:25.444+0000 7fef562d0b80 1 -- 172.21.15.52:0/3905552139 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fef3c011c90 con 0x55bb6af2ae40 2024-05-27T07:31:25.439 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:31:25.445+0000 7fef28ff9700 1 -- 172.21.15.52:0/3905552139 <== mon.0 v2:172.21.15.52:3300/0 3 ==== mon_get_version_reply(handle=1 version=101) ==== 24+0+0 (secure 0 0 0) 0x7fef3c0056a0 con 0x55bb6af2ae40 2024-05-27T07:31:25.439 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:31:25.446+0000 7fef562d0b80 10 client.16602 fetch_fsmap learned FSMap version 101 2024-05-27T07:31:25.439 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:31:25.446+0000 7fef562d0b80 1 -- 172.21.15.52:0/3905552139 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({config=0+,fsmap=101,monmap=4+}) -- 0x55bb6b0e2250 con 0x55bb6af2ae40 2024-05-27T07:31:25.441 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:31:25.447+0000 7fef28ff9700 1 -- 172.21.15.52:0/3905552139 <== mon.0 v2:172.21.15.52:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fef3c011c90 con 0x55bb6af2ae40 2024-05-27T07:31:25.441 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:31:25.447+0000 7fef28ff9700 1 -- 172.21.15.52:0/3905552139 <== mon.0 v2:172.21.15.52:3300/0 5 ==== fsmap(e 101) ==== 4203+0+0 (secure 0 0 0) 0x7fef3c003c60 con 0x55bb6af2ae40 2024-05-27T07:31:25.442 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:31:25.448+0000 7fef562d0b80 10 client.16602 fetch_fsmap finished waiting for FSMap version 101 2024-05-27T07:31:25.442 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:31:25.448+0000 7fef562d0b80 10 client.16602 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-05-27T07:31:25.442 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:31:25.448+0000 7fef562d0b80 20 client.16602 populate_metadata read hostname 'smithi052' 2024-05-27T07:31:25.442 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:31:25.448+0000 7fef562d0b80 1 --2- 172.21.15.52:0/3905552139 >> [v2:172.21.15.105:6838/500348151,v1:172.21.15.105:6839/500348151] conn(0x55bb6b0e4b60 0x55bb6b0e7010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:31:25.442 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:31:25.448+0000 7fef562d0b80 4 client.16602 mds_command: new command op to 24527 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-27T07:31:25.442 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:31:25.448+0000 7fef562d0b80 1 -- 172.21.15.52:0/3905552139 --> [v2:172.21.15.105:6838/500348151,v1:172.21.15.105:6839/500348151] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55bb6b0749f0 con 0x55bb6b0e4b60 2024-05-27T07:31:25.445 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:31:25.450+0000 7fef44e56700 1 --2- 172.21.15.52:0/3905552139 >> [v2:172.21.15.105:6838/500348151,v1:172.21.15.105:6839/500348151] conn(0x55bb6b0e4b60 0x55bb6b0e7010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:31:25.463 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:31:25.452+0000 7fef44e56700 1 --2- 172.21.15.52:0/3905552139 >> [v2:172.21.15.105:6838/500348151,v1:172.21.15.105:6839/500348151] conn(0x55bb6b0e4b60 0x55bb6b0e7010 crc :-1 s=READY pgs=115 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=1987ab58cde66e91 server_cookie=fac22858d6c90814 in_seq=0 out_seq=0 2024-05-27T07:31:25.463 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:31:25.454+0000 7fef28ff9700 10 client.16602 ms_handle_connect on v2:172.21.15.105:6838/500348151 2024-05-27T07:31:25.463 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:31:25.457+0000 7fef28ff9700 1 -- 172.21.15.52:0/3905552139 <== mds.0 v2:172.21.15.105:6838/500348151 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24041 (crc 0 0 0) 0x55bb6b0749f0 con 0x55bb6b0e4b60 2024-05-27T07:31:25.463 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:31:25.457+0000 7fef28ff9700 10 client.16602 handle_command_reply: tid=0 2024-05-27T07:31:25.463 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:31:25.457+0000 7fef562d0b80 1 client.16602 shutdown 2024-05-27T07:31:25.463 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:31:25.459+0000 7fef562d0b80 1 -- 172.21.15.52:0/3905552139 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55bb6af2ae40 msgr2=0x55bb6b0b22b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:31:25.463 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:31:25.459+0000 7fef562d0b80 1 --2- 172.21.15.52:0/3905552139 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55bb6af2ae40 0x55bb6b0b22b0 secure :-1 s=READY pgs=654 cs=0 l=1 rev1=1 crypto rx=0x7fef3c01b900 tx=0x7fef3c01b710 comp rx=0 tx=0).stop 2024-05-27T07:31:25.463 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:31:25.460+0000 7fef562d0b80 1 -- 172.21.15.52:0/3905552139 shutdown_connections 2024-05-27T07:31:25.463 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:31:25.460+0000 7fef562d0b80 1 --2- 172.21.15.52:0/3905552139 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55bb6b0aec50 0x55bb6b0b2d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:31:25.464 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:31:25.460+0000 7fef562d0b80 1 --2- 172.21.15.52:0/3905552139 >> [v2:172.21.15.105:6838/500348151,v1:172.21.15.105:6839/500348151] conn(0x55bb6b0e4b60 0x55bb6b0e7010 crc :-1 s=READY pgs=115 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:31:25.464 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:31:25.460+0000 7fef562d0b80 1 --2- 172.21.15.52:0/3905552139 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55bb6af2ae40 0x55bb6b0b22b0 secure :-1 s=CLOSED pgs=654 cs=0 l=1 rev1=1 crypto rx=0x7fef3c01b900 tx=0x7fef3c01b710 comp rx=0 tx=0).stop 2024-05-27T07:31:25.464 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:31:25.460+0000 7fef562d0b80 1 --2- 172.21.15.52:0/3905552139 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55bb6b0a8f80 0x55bb6b0b27f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:31:25.464 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:31:25.460+0000 7fef562d0b80 1 -- 172.21.15.52:0/3905552139 >> 172.21.15.52:0/3905552139 conn(0x55bb6af85af0 msgr2=0x55bb6b0a7a30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:31:25.464 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:31:25.461+0000 7fef28ff9700 0 client.16602 ms_handle_reset on v2:172.21.15.105:6838/500348151 2024-05-27T07:31:25.464 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:31:25.464+0000 7fef562d0b80 1 -- 172.21.15.52:0/3905552139 shutdown_connections 2024-05-27T07:31:25.464 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:31:25.467+0000 7fef562d0b80 1 -- 172.21.15.52:0/3905552139 wait complete. 2024-05-27T07:31:25.464 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:31:25.470+0000 7fef562d0b80 20 client.16602 trim_cache size 0 max 16384 2024-05-27T07:31:25.525 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:31:25.531+0000 7fef562d0b80 1 Processor -- start 2024-05-27T07:31:25.525 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:31:25.531+0000 7fef562d0b80 1 -- start start 2024-05-27T07:31:25.526 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:31:25.531+0000 7fef562d0b80 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55bb6b0a8f80 0x55bb6b0a8a40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:31:25.526 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:31:25.531+0000 7fef562d0b80 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55bb6b0aec50 0x55bb6b0b2b70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:31:25.526 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:31:25.532+0000 7fef562d0b80 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55bb6b0aca40 0x55bb6b0be910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:31:25.526 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:31:25.532+0000 7fef44e56700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55bb6b0aec50 0x55bb6b0b2b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:31:25.526 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:31:25.532+0000 7fef44e56700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55bb6b0aec50 0x55bb6b0b2b70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.52:59054/0 (socket says 172.21.15.52:59054) 2024-05-27T07:31:25.526 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:31:25.532+0000 7fef44e56700 1 -- 172.21.15.52:0/57848168 learned_addr learned my addr 172.21.15.52:0/57848168 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-05-27T07:31:25.527 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:31:25.532+0000 7fef45657700 1 --2- 172.21.15.52:0/57848168 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55bb6b0a8f80 0x55bb6b0a8a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:31:25.527 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:31:25.532+0000 7fef45e58700 1 --2- 172.21.15.52:0/57848168 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55bb6b0aca40 0x55bb6b0be910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:31:25.527 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:31:25.532+0000 7fef562d0b80 1 -- 172.21.15.52:0/57848168 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x55bb6b0a3bd0 con 0x55bb6b0a8f80 2024-05-27T07:31:25.527 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:31:25.533+0000 7fef562d0b80 1 -- 172.21.15.52:0/57848168 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7fef2400f1d0 con 0x55bb6b0aec50 2024-05-27T07:31:25.528 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:31:25.533+0000 7fef562d0b80 1 -- 172.21.15.52:0/57848168 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7fef2400d5d0 con 0x55bb6b0aca40 2024-05-27T07:31:25.528 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:31:25.534+0000 7fef45e58700 1 -- 172.21.15.52:0/57848168 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55bb6b0aec50 msgr2=0x55bb6b0b2b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:31:25.528 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:31:25.534+0000 7fef45e58700 1 --2- 172.21.15.52:0/57848168 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55bb6b0aec50 0x55bb6b0b2b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:31:25.528 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:31:25.534+0000 7fef45e58700 1 -- 172.21.15.52:0/57848168 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55bb6b0a8f80 msgr2=0x55bb6b0a8a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:31:25.528 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:31:25.534+0000 7fef45e58700 1 --2- 172.21.15.52:0/57848168 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55bb6b0a8f80 0x55bb6b0a8a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:31:25.528 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:31:25.534+0000 7fef45e58700 1 -- 172.21.15.52:0/57848168 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55bb6b0bee50 con 0x55bb6b0aca40 2024-05-27T07:31:25.528 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:31:25.534+0000 7fef45e58700 1 --2- 172.21.15.52:0/57848168 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55bb6b0aca40 0x55bb6b0be910 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7fef300df3e0 tx=0x7fef300ae440 comp rx=0 tx=0).ready entity=mon.1 client_cookie=ad0ccbd51203a944 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:31:25.531 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:31:25.538+0000 7fef2affd700 1 -- 172.21.15.52:0/57848168 <== mon.1 v2:172.21.15.173:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fef300a5760 con 0x55bb6b0aca40 2024-05-27T07:31:25.531 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:31:25.538+0000 7fef2affd700 1 -- 172.21.15.52:0/57848168 <== mon.1 v2:172.21.15.173:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fef3000ee50 con 0x55bb6b0aca40 2024-05-27T07:31:25.531 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:31:25.538+0000 7fef2affd700 1 -- 172.21.15.52:0/57848168 <== mon.1 v2:172.21.15.173:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fef300a5900 con 0x55bb6b0aca40 2024-05-27T07:31:25.535 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:31:25.541+0000 7fef562d0b80 1 -- 172.21.15.52:0/57848168 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55bb6b0aca40 msgr2=0x55bb6b0be910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:31:25.535 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:31:25.541+0000 7fef562d0b80 1 --2- 172.21.15.52:0/57848168 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55bb6b0aca40 0x55bb6b0be910 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7fef300df3e0 tx=0x7fef300ae440 comp rx=0 tx=0).stop 2024-05-27T07:31:25.535 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:31:25.541+0000 7fef562d0b80 1 -- 172.21.15.52:0/57848168 shutdown_connections 2024-05-27T07:31:25.535 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:31:25.541+0000 7fef562d0b80 1 --2- 172.21.15.52:0/57848168 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55bb6b0aca40 0x55bb6b0be910 unknown :-1 s=CLOSED pgs=140 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:31:25.535 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:31:25.542+0000 7fef562d0b80 1 --2- 172.21.15.52:0/57848168 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55bb6b0a8f80 0x55bb6b0a8a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:31:25.535 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:31:25.542+0000 7fef562d0b80 1 --2- 172.21.15.52:0/57848168 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55bb6b0aec50 0x55bb6b0b2b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:31:25.535 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:31:25.542+0000 7fef562d0b80 1 -- 172.21.15.52:0/57848168 >> 172.21.15.52:0/57848168 conn(0x55bb6af85af0 msgr2=0x55bb6b09d6f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:31:25.539 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:31:25.546+0000 7fef562d0b80 1 -- 172.21.15.52:0/57848168 shutdown_connections 2024-05-27T07:31:25.539 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:31:25.546+0000 7fef562d0b80 1 -- 172.21.15.52:0/57848168 wait complete. 2024-05-27T07:31:25.541 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:31:25.547+0000 7fef562d0b80 1 -- start start 2024-05-27T07:31:25.541 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:31:25.548+0000 7fef562d0b80 1 Processor -- start 2024-05-27T07:31:25.542 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:31:25.549+0000 7fef562d0b80 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55bb6b0a8f80 0x55bb6b0bb150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:31:25.543 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:31:25.550+0000 7fef45657700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55bb6b0a8f80 0x55bb6b0bb150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:31:25.543 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:31:25.550+0000 7fef45657700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55bb6b0a8f80 0x55bb6b0bb150 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.52:47050/0 (socket says 172.21.15.52:47050) 2024-05-27T07:31:25.543 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:31:25.550+0000 7fef45657700 1 -- 172.21.15.52:0/3412739305 learned_addr learned my addr 172.21.15.52:0/3412739305 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-05-27T07:31:25.543 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:31:25.550+0000 7fef562d0b80 1 --2- 172.21.15.52:0/3412739305 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55bb6b0aec50 0x55bb6b0bd900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:31:25.544 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:31:25.550+0000 7fef562d0b80 1 --2- 172.21.15.52:0/3412739305 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55bb6b0bde40 0x55bb6b0be250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:31:25.544 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:31:25.550+0000 7fef44e56700 1 --2- 172.21.15.52:0/3412739305 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55bb6b0aec50 0x55bb6b0bd900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:31:25.544 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:31:25.551+0000 7fef562d0b80 1 -- 172.21.15.52:0/3412739305 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x7fef2400dca0 con 0x55bb6b0a8f80 2024-05-27T07:31:25.544 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:31:25.551+0000 7fef562d0b80 1 -- 172.21.15.52:0/3412739305 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7fef2400eb00 con 0x55bb6b0aec50 2024-05-27T07:31:25.545 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:31:25.551+0000 7fef45e58700 1 --2- 172.21.15.52:0/3412739305 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55bb6b0bde40 0x55bb6b0be250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:31:25.545 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:31:25.551+0000 7fef562d0b80 1 -- 172.21.15.52:0/3412739305 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7fef2400e370 con 0x55bb6b0bde40 2024-05-27T07:31:25.547 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:31:25.552+0000 7fef45e58700 1 -- 172.21.15.52:0/3412739305 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55bb6b0aec50 msgr2=0x55bb6b0bd900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:31:25.547 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:31:25.552+0000 7fef45e58700 1 --2- 172.21.15.52:0/3412739305 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55bb6b0aec50 0x55bb6b0bd900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:31:25.547 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:31:25.552+0000 7fef45e58700 1 -- 172.21.15.52:0/3412739305 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55bb6b0a8f80 msgr2=0x55bb6b0bb150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:31:25.547 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:31:25.552+0000 7fef45e58700 1 --2- 172.21.15.52:0/3412739305 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55bb6b0a8f80 0x55bb6b0bb150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:31:25.547 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:31:25.552+0000 7fef45e58700 1 -- 172.21.15.52:0/3412739305 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55bb6b14e760 con 0x55bb6b0bde40 2024-05-27T07:31:25.547 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:31:25.552+0000 7fef45657700 1 --2- 172.21.15.52:0/3412739305 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55bb6b0a8f80 0x55bb6b0bb150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-27T07:31:25.547 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:31:25.553+0000 7fef45e58700 1 --2- 172.21.15.52:0/3412739305 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55bb6b0bde40 0x55bb6b0be250 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7fef300a5e10 tx=0x7fef300a8af0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=901c81683e8284 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:31:25.547 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:31:25.553+0000 7fef28ff9700 10 client.0 ms_handle_connect on v2:172.21.15.173:3300/0 2024-05-27T07:31:25.547 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:31:25.553+0000 7fef562d0b80 10 client.0 _ll_register_callbacks cb 0x55bb6af62920 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T07:31:25.547 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:31:25.554+0000 7fef562d0b80 1 -- 172.21.15.52:0/3412739305 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fef3000ee50 con 0x55bb6b0bde40 2024-05-27T07:31:25.549 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:31:25.554+0000 7fef28ff9700 1 -- 172.21.15.52:0/3412739305 <== mon.1 v2:172.21.15.173:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fef300a5ad0 con 0x55bb6b0bde40 2024-05-27T07:31:25.549 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:31:25.554+0000 7fef28ff9700 1 -- 172.21.15.52:0/3412739305 <== mon.1 v2:172.21.15.173:3300/0 2 ==== osd_map(116..116 src has 1..116) ==== 9379+0+0 (secure 0 0 0) 0x7fef300a81d0 con 0x55bb6b0bde40 2024-05-27T07:31:25.550 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:31:25.557+0000 7fef28ff9700 1 -- 172.21.15.52:0/3412739305 <== mon.1 v2:172.21.15.173:3300/0 3 ==== mon_get_version_reply(handle=1 version=101) ==== 24+0+0 (secure 0 0 0) 0x7fef300a5c70 con 0x55bb6b0bde40 2024-05-27T07:31:25.550 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:31:25.557+0000 7fef562d0b80 10 client.26131 fetch_fsmap learned FSMap version 101 2024-05-27T07:31:25.550 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:31:25.557+0000 7fef562d0b80 1 -- 172.21.15.52:0/3412739305 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({fsmap=101}) -- 0x55bb6b14f810 con 0x55bb6b0bde40 2024-05-27T07:31:25.556 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:31:25.562+0000 7fef28ff9700 1 -- 172.21.15.52:0/3412739305 <== mon.1 v2:172.21.15.173:3300/0 4 ==== fsmap(e 101) ==== 4203+0+0 (secure 0 0 0) 0x7fef300afb10 con 0x55bb6b0bde40 2024-05-27T07:31:25.556 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:31:25.562+0000 7fef562d0b80 10 client.26131 fetch_fsmap finished waiting for FSMap version 101 2024-05-27T07:31:25.556 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:31:25.562+0000 7fef562d0b80 10 client.26131 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-05-27T07:31:25.556 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:31:25.562+0000 7fef562d0b80 20 client.26131 populate_metadata read hostname 'smithi052' 2024-05-27T07:31:25.556 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:31:25.562+0000 7fef562d0b80 1 --2- 172.21.15.52:0/3412739305 >> [v2:172.21.15.105:6838/500348151,v1:172.21.15.105:6839/500348151] conn(0x55bb6b14fe00 0x55bb6b152220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:31:25.556 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:31:25.562+0000 7fef562d0b80 4 client.26131 mds_command: new command op to 24527 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-05-27T07:31:25.556 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:31:25.562+0000 7fef562d0b80 1 -- 172.21.15.52:0/3412739305 --> [v2:172.21.15.105:6838/500348151,v1:172.21.15.105:6839/500348151] -- command(tid 0: {"prefix": "scrub status"}) -- 0x7fef3000ee50 con 0x55bb6b14fe00 2024-05-27T07:31:25.557 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:31:25.564+0000 7fef45657700 1 --2- 172.21.15.52:0/3412739305 >> [v2:172.21.15.105:6838/500348151,v1:172.21.15.105:6839/500348151] conn(0x55bb6b14fe00 0x55bb6b152220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:31:25.560 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:31:25.567+0000 7fef45657700 1 --2- 172.21.15.52:0/3412739305 >> [v2:172.21.15.105:6838/500348151,v1:172.21.15.105:6839/500348151] conn(0x55bb6b14fe00 0x55bb6b152220 crc :-1 s=READY pgs=116 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=ba9124df33d5d3ab server_cookie=4abaa8941e39ca08 in_seq=0 out_seq=0 2024-05-27T07:31:25.560 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:31:25.567+0000 7fef28ff9700 10 client.26131 ms_handle_connect on v2:172.21.15.105:6838/500348151 2024-05-27T07:31:25.563 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:31:25.569+0000 7fef28ff9700 1 -- 172.21.15.52:0/3412739305 <== mds.0 v2:172.21.15.105:6838/500348151 1 ==== command_reply(tid 0: 0 ) ==== 8+0+63 (crc 0 0 0) 0x7fef3000ee50 con 0x55bb6b14fe00 2024-05-27T07:31:25.563 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:31:25.569+0000 7fef28ff9700 10 client.26131 handle_command_reply: tid=0 2024-05-27T07:31:25.563 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:31:25.570+0000 7fef562d0b80 1 client.26131 shutdown 2024-05-27T07:31:25.563 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:31:25.570+0000 7fef562d0b80 1 -- 172.21.15.52:0/3412739305 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55bb6b0bde40 msgr2=0x55bb6b0be250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:31:25.563 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:31:25.570+0000 7fef562d0b80 1 --2- 172.21.15.52:0/3412739305 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55bb6b0bde40 0x55bb6b0be250 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7fef300a5e10 tx=0x7fef300a8af0 comp rx=0 tx=0).stop 2024-05-27T07:31:25.564 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:31:25.570+0000 7fef562d0b80 1 -- 172.21.15.52:0/3412739305 shutdown_connections 2024-05-27T07:31:25.564 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:31:25.570+0000 7fef562d0b80 1 --2- 172.21.15.52:0/3412739305 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55bb6b0bde40 0x55bb6b0be250 unknown :-1 s=CLOSED pgs=141 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:31:25.564 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:31:25.570+0000 7fef562d0b80 1 --2- 172.21.15.52:0/3412739305 >> [v2:172.21.15.105:6838/500348151,v1:172.21.15.105:6839/500348151] conn(0x55bb6b14fe00 0x55bb6b152220 crc :-1 s=READY pgs=116 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:31:25.564 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:31:25.570+0000 7fef562d0b80 1 --2- 172.21.15.52:0/3412739305 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55bb6b0a8f80 0x55bb6b0bb150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:31:25.564 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:31:25.571+0000 7fef562d0b80 1 --2- 172.21.15.52:0/3412739305 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55bb6b0aec50 0x55bb6b0bd900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:31:25.564 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:31:25.571+0000 7fef562d0b80 1 -- 172.21.15.52:0/3412739305 >> 172.21.15.52:0/3412739305 conn(0x55bb6af85af0 msgr2=0x55bb6b0b0ba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:31:25.564 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:31:25.571+0000 7fef28ff9700 0 client.26131 ms_handle_reset on v2:172.21.15.105:6838/500348151 2024-05-27T07:31:25.568 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:31:25.575+0000 7fef562d0b80 1 -- 172.21.15.52:0/3412739305 shutdown_connections 2024-05-27T07:31:25.568 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:31:25.575+0000 7fef562d0b80 1 -- 172.21.15.52:0/3412739305 wait complete. 2024-05-27T07:31:25.569 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:31:25.576+0000 7fef562d0b80 20 client.26131 trim_cache size 0 max 16384 2024-05-27T07:31:25.571 INFO:teuthology.orchestra.run.smithi052.stdout:{ 2024-05-27T07:31:25.571 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "no active scrubs running", 2024-05-27T07:31:25.571 INFO:teuthology.orchestra.run.smithi052.stdout: "scrubs": {} 2024-05-27T07:31:25.572 INFO:teuthology.orchestra.run.smithi052.stdout:} 2024-05-27T07:31:25.573 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:31:25.579+0000 7fef478bb700 1 -- 172.21.15.52:0/628693250 >> [v2:172.21.15.52:6800/1822318384,v1:172.21.15.52:6801/1822318384] conn(0x7fef2c0b2ab0 msgr2=0x7fef2c0b4fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:31:25.573 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:31:25.580+0000 7fef478bb700 1 --2- 172.21.15.52:0/628693250 >> [v2:172.21.15.52:6800/1822318384,v1:172.21.15.52:6801/1822318384] conn(0x7fef2c0b2ab0 0x7fef2c0b4fc0 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7fef3800b830 tx=0x7fef38006000 comp rx=0 tx=0).stop 2024-05-27T07:31:25.573 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:31:25.580+0000 7fef478bb700 1 -- 172.21.15.52:0/628693250 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fef40110990 msgr2=0x7fef4007d860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:31:25.573 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:31:25.580+0000 7fef478bb700 1 --2- 172.21.15.52:0/628693250 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fef40110990 0x7fef4007d860 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7fef3000d720 tx=0x7fef3000da30 comp rx=0 tx=0).stop 2024-05-27T07:31:25.578 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:31:25.584+0000 7fef478bb700 1 -- 172.21.15.52:0/628693250 shutdown_connections 2024-05-27T07:31:25.578 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:31:25.584+0000 7fef478bb700 1 --2- 172.21.15.52:0/628693250 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fef40110990 0x7fef4007d860 unknown :-1 s=CLOSED pgs=138 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:31:25.578 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:31:25.584+0000 7fef478bb700 1 --2- 172.21.15.52:0/628693250 >> [v2:172.21.15.52:6800/1822318384,v1:172.21.15.52:6801/1822318384] conn(0x7fef2c0b2ab0 0x7fef2c0b4fc0 unknown :-1 s=CLOSED pgs=145 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:31:25.578 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:31:25.584+0000 7fef478bb700 1 --2- 172.21.15.52:0/628693250 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fef4010a960 0x7fef40084170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:31:25.578 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:31:25.584+0000 7fef478bb700 1 --2- 172.21.15.52:0/628693250 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fef4010b2b0 0x7fef4007d2a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:31:25.578 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:31:25.584+0000 7fef478bb700 1 -- 172.21.15.52:0/628693250 >> 172.21.15.52:0/628693250 conn(0x7fef40076cb0 msgr2=0x7fef401087b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:31:25.578 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:31:25.584+0000 7fef478bb700 1 -- 172.21.15.52:0/628693250 shutdown_connections 2024-05-27T07:31:25.578 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:31:25.584+0000 7fef478bb700 1 -- 172.21.15.52:0/628693250 wait complete. 2024-05-27T07:31:26.442 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:31:26 smithi105 ceph-mon[35977]: pgmap v192: 129 pgs: 129 active+clean; 443 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.5 MiB/s rd, 2.3 MiB/s wr, 999 op/s 2024-05-27T07:31:26.453 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:31:26 smithi052 ceph-mon[31955]: pgmap v192: 129 pgs: 129 active+clean; 443 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.5 MiB/s rd, 2.3 MiB/s wr, 999 op/s 2024-05-27T07:31:26.604 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:31:26 smithi173 ceph-mon[37000]: pgmap v192: 129 pgs: 129 active+clean; 443 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.5 MiB/s rd, 2.3 MiB/s wr, 999 op/s 2024-05-27T07:31:26.651 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-05-27T07:31:26.652 DEBUG:teuthology.orchestra.run.smithi052:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:38f9d900c83c63799fbdbe61acc9a11b0d3554a6 shell --fsid 73df2636-1bf8-11ef-bc9b-c7b262605968 -- ceph fs dump --format=json 2024-05-27T07:31:27.511 INFO:teuthology.orchestra.run.smithi052.stderr:Inferring config /var/lib/ceph/73df2636-1bf8-11ef-bc9b-c7b262605968/mon.a/config 2024-05-27T07:31:28.232 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:31:28 smithi052 ceph-mon[31955]: pgmap v193: 129 pgs: 129 active+clean; 443 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.5 MiB/s rd, 2.2 MiB/s wr, 981 op/s 2024-05-27T07:31:28.442 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:31:28 smithi105 ceph-mon[35977]: pgmap v193: 129 pgs: 129 active+clean; 443 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.5 MiB/s rd, 2.2 MiB/s wr, 981 op/s 2024-05-27T07:31:28.604 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:31:28 smithi173 ceph-mon[37000]: pgmap v193: 129 pgs: 129 active+clean; 443 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.5 MiB/s rd, 2.2 MiB/s wr, 981 op/s 2024-05-27T07:31:30.442 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:31:30 smithi105 ceph-mon[35977]: pgmap v194: 129 pgs: 129 active+clean; 462 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 5.3 MiB/s rd, 3.5 MiB/s wr, 974 op/s 2024-05-27T07:31:30.481 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:31:30 smithi052 ceph-mon[31955]: pgmap v194: 129 pgs: 129 active+clean; 462 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 5.3 MiB/s rd, 3.5 MiB/s wr, 974 op/s 2024-05-27T07:31:30.604 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:31:30 smithi173 ceph-mon[37000]: pgmap v194: 129 pgs: 129 active+clean; 462 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 5.3 MiB/s rd, 3.5 MiB/s wr, 974 op/s 2024-05-27T07:31:30.653 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:31:30.659+0000 7f0809a8e640 1 -- 172.21.15.52:0/1304854864 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f0804109f50 msgr2=0x7f0804103b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:31:30.653 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:31:30.659+0000 7f0809a8e640 1 --2- 172.21.15.52:0/1304854864 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f0804109f50 0x7f0804103b90 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7f07f0009900 tx=0x7f07f002a410 comp rx=0 tx=0).stop 2024-05-27T07:31:30.653 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:31:30.659+0000 7f0809a8e640 1 -- 172.21.15.52:0/1304854864 shutdown_connections 2024-05-27T07:31:30.653 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:31:30.659+0000 7f0809a8e640 1 --2- 172.21.15.52:0/1304854864 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f0804104200 0x7f08041105d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:31:30.653 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:31:30.659+0000 7f0809a8e640 1 --2- 172.21.15.52:0/1304854864 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f0804109f50 0x7f0804103b90 unknown :-1 s=CLOSED pgs=142 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:31:30.653 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:31:30.659+0000 7f0809a8e640 1 --2- 172.21.15.52:0/1304854864 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f0804109ab0 0x7f0804103650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:31:30.653 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:31:30.659+0000 7f0809a8e640 1 -- 172.21.15.52:0/1304854864 >> 172.21.15.52:0/1304854864 conn(0x7f0804079170 msgr2=0x7f08040795b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:31:30.653 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:31:30.659+0000 7f0809a8e640 1 -- 172.21.15.52:0/1304854864 shutdown_connections 2024-05-27T07:31:30.653 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:31:30.659+0000 7f0809a8e640 1 -- 172.21.15.52:0/1304854864 wait complete. 2024-05-27T07:31:30.654 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:31:30.660+0000 7f0809a8e640 1 Processor -- start 2024-05-27T07:31:30.655 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:31:30.660+0000 7f0809a8e640 1 -- start start 2024-05-27T07:31:30.655 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:31:30.660+0000 7f0809a8e640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f0804104200 0x7f080419a6d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:31:30.655 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:31:30.660+0000 7f0809a8e640 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f0804109ab0 0x7f080419ac30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:31:30.655 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:31:30.660+0000 7f0809a8e640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f0804109f50 0x7f080419eef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:31:30.655 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:31:30.661+0000 7f0809a8e640 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x7f0804113cf0 con 0x7f0804109ab0 2024-05-27T07:31:30.655 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:31:30.661+0000 7f0809a8e640 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f0804113b70 con 0x7f0804104200 2024-05-27T07:31:30.655 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:31:30.661+0000 7f0809a8e640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f0804113e70 con 0x7f0804109f50 2024-05-27T07:31:30.655 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:31:30.661+0000 7f0803fff640 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f0804109ab0 0x7f080419ac30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:31:30.655 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:31:30.661+0000 7f0803fff640 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f0804109ab0 0x7f080419ac30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.52:47064/0 (socket says 172.21.15.52:47064) 2024-05-27T07:31:30.655 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:31:30.661+0000 7f0803fff640 1 -- 172.21.15.52:0/2557948725 learned_addr learned my addr 172.21.15.52:0/2557948725 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-05-27T07:31:30.655 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:31:30.661+0000 7f0808a8c640 1 --2- 172.21.15.52:0/2557948725 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f0804104200 0x7f080419a6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:31:30.656 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:31:30.661+0000 7f080928d640 1 --2- 172.21.15.52:0/2557948725 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f0804109f50 0x7f080419eef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:31:30.656 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:31:30.661+0000 7f0803fff640 1 -- 172.21.15.52:0/2557948725 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f0804109f50 msgr2=0x7f080419eef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:31:30.656 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:31:30.661+0000 7f0803fff640 1 --2- 172.21.15.52:0/2557948725 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f0804109f50 0x7f080419eef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:31:30.656 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:31:30.661+0000 7f0803fff640 1 -- 172.21.15.52:0/2557948725 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f0804104200 msgr2=0x7f080419a6d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:31:30.656 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:31:30.661+0000 7f0803fff640 1 --2- 172.21.15.52:0/2557948725 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f0804104200 0x7f080419a6d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:31:30.656 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:31:30.661+0000 7f0803fff640 1 -- 172.21.15.52:0/2557948725 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f080419f430 con 0x7f0804109ab0 2024-05-27T07:31:30.656 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:31:30.661+0000 7f0808a8c640 1 --2- 172.21.15.52:0/2557948725 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f0804104200 0x7f080419a6d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-27T07:31:30.657 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:31:30.662+0000 7f0803fff640 1 --2- 172.21.15.52:0/2557948725 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f0804109ab0 0x7f080419ac30 secure :-1 s=READY pgs=655 cs=0 l=1 rev1=1 crypto rx=0x7f07f0031440 tx=0x7f07f0003b20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:31:30.657 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:31:30.662+0000 7f0801ffb640 1 -- 172.21.15.52:0/2557948725 <== mon.0 v2:172.21.15.52:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f07f003f070 con 0x7f0804109ab0 2024-05-27T07:31:30.657 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:31:30.662+0000 7f0801ffb640 1 -- 172.21.15.52:0/2557948725 <== mon.0 v2:172.21.15.52:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f07f0031b50 con 0x7f0804109ab0 2024-05-27T07:31:30.657 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:31:30.662+0000 7f0801ffb640 1 -- 172.21.15.52:0/2557948725 <== mon.0 v2:172.21.15.52:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f07f0035030 con 0x7f0804109ab0 2024-05-27T07:31:30.657 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:31:30.662+0000 7f0809a8e640 1 -- 172.21.15.52:0/2557948725 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f080419f6c0 con 0x7f0804109ab0 2024-05-27T07:31:30.657 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:31:30.663+0000 7f0809a8e640 1 -- 172.21.15.52:0/2557948725 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f080419fb60 con 0x7f0804109ab0 2024-05-27T07:31:30.660 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:31:30.667+0000 7f0811b9f740 1 Processor -- start 2024-05-27T07:31:30.660 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:31:30.667+0000 7f0811b9f740 1 -- start start 2024-05-27T07:31:30.661 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:31:30.668+0000 7f0811b9f740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55c343e191b0 0x55c3441ba590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:31:30.661 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:31:30.668+0000 7f0801ffb640 1 -- 172.21.15.52:0/2557948725 <== mon.0 v2:172.21.15.52:3300/0 4 ==== mgrmap(e 26) ==== 149783+0+0 (secure 0 0 0) 0x7f07f0042070 con 0x7f0804109ab0 2024-05-27T07:31:30.661 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:31:30.668+0000 7f0811b9f740 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55c343df02f0 0x55c3441baad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:31:30.661 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:31:30.668+0000 7f0811b9f740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55c343df84d0 0x55c3441bb010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:31:30.661 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:31:30.668+0000 7f0811b9f740 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x55c34412cc30 con 0x55c343df02f0 2024-05-27T07:31:30.661 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:31:30.668+0000 7f0811b9f740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x55c34411c7d0 con 0x55c343df84d0 2024-05-27T07:31:30.661 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:31:30.668+0000 7f0811b9f740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x55c344120810 con 0x55c343e191b0 2024-05-27T07:31:30.663 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:31:30.668+0000 7f0808a8c640 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55c343df02f0 0x55c3441baad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:31:30.663 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:31:30.668+0000 7f0808a8c640 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55c343df02f0 0x55c3441baad0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.52:47066/0 (socket says 172.21.15.52:47066) 2024-05-27T07:31:30.663 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:31:30.668+0000 7f0808a8c640 1 -- 172.21.15.52:0/2031071316 learned_addr learned my addr 172.21.15.52:0/2031071316 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-05-27T07:31:30.663 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:31:30.669+0000 7f0808a8c640 1 -- 172.21.15.52:0/2031071316 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55c343e191b0 msgr2=0x55c3441ba590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:31:30.663 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:31:30.669+0000 7f0803fff640 1 --2- 172.21.15.52:0/2031071316 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55c343df84d0 0x55c3441bb010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:31:30.663 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:31:30.669+0000 7f0808a8c640 1 --2- 172.21.15.52:0/2031071316 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55c343e191b0 0x55c3441ba590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:31:30.663 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:31:30.669+0000 7f0808a8c640 1 -- 172.21.15.52:0/2031071316 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55c343df84d0 msgr2=0x55c3441bb010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:31:30.663 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:31:30.669+0000 7f0808a8c640 1 --2- 172.21.15.52:0/2031071316 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55c343df84d0 0x55c3441bb010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:31:30.664 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:31:30.669+0000 7f0808a8c640 1 -- 172.21.15.52:0/2031071316 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55c343e31210 con 0x55c343df02f0 2024-05-27T07:31:30.664 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:31:30.669+0000 7f0803fff640 1 --2- 172.21.15.52:0/2031071316 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55c343df84d0 0x55c3441bb010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-27T07:31:30.664 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:31:30.669+0000 7f0808a8c640 1 --2- 172.21.15.52:0/2031071316 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55c343df02f0 0x55c3441baad0 secure :-1 s=READY pgs=656 cs=0 l=1 rev1=1 crypto rx=0x7f07f8011470 tx=0x7f07f8011930 comp rx=0 tx=0).ready entity=mon.0 client_cookie=31ea3caecac3ce16 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:31:30.664 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:31:30.669+0000 7f0801ffb640 1 --2- 172.21.15.52:0/2557948725 >> [v2:172.21.15.52:6800/1822318384,v1:172.21.15.52:6801/1822318384] conn(0x7f07f40b2a20 0x7f07f40b4f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:31:30.664 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:31:30.669+0000 7f080928d640 1 --2- 172.21.15.52:0/2557948725 >> [v2:172.21.15.52:6800/1822318384,v1:172.21.15.52:6801/1822318384] conn(0x7f07f40b2a20 0x7f07f40b4f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:31:30.664 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:31:30.670+0000 7f07eeffd640 1 -- 172.21.15.52:0/2031071316 <== mon.0 v2:172.21.15.52:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f07f801e070 con 0x55c343df02f0 2024-05-27T07:31:30.664 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:31:30.670+0000 7f0801ffb640 1 -- 172.21.15.52:0/2557948725 <== mon.0 v2:172.21.15.52:3300/0 5 ==== osd_map(116..116 src has 1..116) ==== 9379+0+0 (secure 0 0 0) 0x7f07f0102c70 con 0x7f0804109ab0 2024-05-27T07:31:30.664 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:31:30.670+0000 7f07eeffd640 1 -- 172.21.15.52:0/2031071316 <== mon.0 v2:172.21.15.52:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f07f8004310 con 0x55c343df02f0 2024-05-27T07:31:30.665 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:31:30.670+0000 7f07eeffd640 1 -- 172.21.15.52:0/2031071316 <== mon.0 v2:172.21.15.52:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f07f800fa00 con 0x55c343df02f0 2024-05-27T07:31:30.666 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:31:30.671+0000 7f0811b9f740 1 -- 172.21.15.52:0/2031071316 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55c343df02f0 msgr2=0x55c3441baad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:31:30.666 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:31:30.671+0000 7f080928d640 1 --2- 172.21.15.52:0/2557948725 >> [v2:172.21.15.52:6800/1822318384,v1:172.21.15.52:6801/1822318384] conn(0x7f07f40b2a20 0x7f07f40b4f30 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x55c3441b7700 tx=0x7f07fc009b10 comp rx=0 tx=0).ready entity=mgr.15333 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:31:30.666 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:31:30.671+0000 7f0811b9f740 1 --2- 172.21.15.52:0/2031071316 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55c343df02f0 0x55c3441baad0 secure :-1 s=READY pgs=656 cs=0 l=1 rev1=1 crypto rx=0x7f07f8011470 tx=0x7f07f8011930 comp rx=0 tx=0).stop 2024-05-27T07:31:30.666 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:31:30.671+0000 7f0811b9f740 1 -- 172.21.15.52:0/2031071316 shutdown_connections 2024-05-27T07:31:30.666 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:31:30.671+0000 7f0811b9f740 1 --2- 172.21.15.52:0/2031071316 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55c343df84d0 0x55c3441bb010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:31:30.666 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:31:30.671+0000 7f0811b9f740 1 --2- 172.21.15.52:0/2031071316 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55c343df02f0 0x55c3441baad0 unknown :-1 s=CLOSED pgs=656 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:31:30.666 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:31:30.671+0000 7f0811b9f740 1 --2- 172.21.15.52:0/2031071316 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55c343e191b0 0x55c3441ba590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:31:30.666 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:31:30.671+0000 7f0811b9f740 1 -- 172.21.15.52:0/2031071316 >> 172.21.15.52:0/2031071316 conn(0x55c343e0d300 msgr2=0x55c3441b6d40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:31:30.666 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:31:30.671+0000 7f0811b9f740 1 -- 172.21.15.52:0/2031071316 shutdown_connections 2024-05-27T07:31:30.666 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:31:30.671+0000 7f0811b9f740 1 -- 172.21.15.52:0/2031071316 wait complete. 2024-05-27T07:31:30.666 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:31:30.672+0000 7f0811b9f740 1 -- start start 2024-05-27T07:31:30.667 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:31:30.672+0000 7f0811b9f740 1 Processor -- start 2024-05-27T07:31:30.668 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:31:30.673+0000 7f0811b9f740 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55c343df02f0 0x55c3441c6640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:31:30.668 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:31:30.674+0000 7f0811b9f740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55c343df84d0 0x55c3441c6ba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:31:30.668 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:31:30.674+0000 7f0808a8c640 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55c343df02f0 0x55c3441c6640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:31:30.668 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:31:30.674+0000 7f0808a8c640 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55c343df02f0 0x55c3441c6640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.52:47076/0 (socket says 172.21.15.52:47076) 2024-05-27T07:31:30.668 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:31:30.674+0000 7f0803fff640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55c343df84d0 0x55c3441c6ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:31:30.668 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:31:30.674+0000 7f0803fff640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55c343df84d0 0x55c3441c6ba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.52:59120/0 (socket says 172.21.15.52:59120) 2024-05-27T07:31:30.668 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:31:30.674+0000 7f0811b9f740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55c343e191b0 0x55c3441c4ac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:31:30.668 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:31:30.674+0000 7f0808a8c640 1 -- 172.21.15.52:0/3995294932 learned_addr learned my addr 172.21.15.52:0/3995294932 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-05-27T07:31:30.669 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:31:30.674+0000 7f0811b9f740 1 -- 172.21.15.52:0/3995294932 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x55c343e02790 con 0x55c343df02f0 2024-05-27T07:31:30.669 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:31:30.674+0000 7f080928d640 1 --2- 172.21.15.52:0/3995294932 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55c343e191b0 0x55c3441c4ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:31:30.669 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:31:30.674+0000 7f0811b9f740 1 -- 172.21.15.52:0/3995294932 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x55c3440d8890 con 0x55c343df84d0 2024-05-27T07:31:30.669 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:31:30.675+0000 7f0811b9f740 1 -- 172.21.15.52:0/3995294932 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x55c3441c7760 con 0x55c343e191b0 2024-05-27T07:31:30.671 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:31:30.675+0000 7f0803fff640 1 -- 172.21.15.52:0/3995294932 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55c343e191b0 msgr2=0x55c3441c4ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:31:30.671 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:31:30.675+0000 7f0803fff640 1 --2- 172.21.15.52:0/3995294932 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55c343e191b0 0x55c3441c4ac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:31:30.671 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:31:30.675+0000 7f0803fff640 1 -- 172.21.15.52:0/3995294932 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55c343df02f0 msgr2=0x55c3441c6640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:31:30.671 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:31:30.675+0000 7f0803fff640 1 --2- 172.21.15.52:0/3995294932 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55c343df02f0 0x55c3441c6640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:31:30.671 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:31:30.675+0000 7f0803fff640 1 -- 172.21.15.52:0/3995294932 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55c343efd620 con 0x55c343df84d0 2024-05-27T07:31:30.671 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:31:30.675+0000 7f0803fff640 1 --2- 172.21.15.52:0/3995294932 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55c343df84d0 0x55c3441c6ba0 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7f07f0100000 tx=0x7f07f00c5c30 comp rx=0 tx=0).ready entity=mon.2 client_cookie=79dd5c76804f2300 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:31:30.671 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:31:30.675+0000 7f07e3fff640 10 client.0 ms_handle_connect on v2:172.21.15.105:3300/0 2024-05-27T07:31:30.671 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:31:30.676+0000 7f0811b9f740 10 client.0 _ll_register_callbacks cb 0x55c343e30a00 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T07:31:30.671 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:31:30.676+0000 7f0811b9f740 1 -- 172.21.15.52:0/3995294932 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f07f8004310 con 0x55c343df84d0 2024-05-27T07:31:30.671 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:31:30.676+0000 7f07e3fff640 1 -- 172.21.15.52:0/3995294932 <== mon.2 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f07f0049070 con 0x55c343df84d0 2024-05-27T07:31:30.671 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:31:30.676+0000 7f07e3fff640 1 -- 172.21.15.52:0/3995294932 <== mon.2 v2:172.21.15.105:3300/0 2 ==== osd_map(116..116 src has 1..116) ==== 9379+0+0 (secure 0 0 0) 0x7f07f0100280 con 0x55c343df84d0 2024-05-27T07:31:30.671 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:31:30.676+0000 7f07e3fff640 1 -- 172.21.15.52:0/3995294932 <== mon.2 v2:172.21.15.105:3300/0 3 ==== mon_get_version_reply(handle=1 version=101) ==== 24+0+0 (secure 0 0 0) 0x7f07f0100480 con 0x55c343df84d0 2024-05-27T07:31:30.672 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:31:30.676+0000 7f0811b9f740 10 client.26255 fetch_fsmap learned FSMap version 101 2024-05-27T07:31:30.672 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:31:30.676+0000 7f0811b9f740 1 -- 172.21.15.52:0/3995294932 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({fsmap=101}) -- 0x55c3441d3780 con 0x55c343df84d0 2024-05-27T07:31:30.672 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:31:30.676+0000 7f07e3fff640 1 -- 172.21.15.52:0/3995294932 <== mon.2 v2:172.21.15.105:3300/0 4 ==== fsmap(e 101) ==== 4203+0+0 (secure 0 0 0) 0x7f07f004c060 con 0x55c343df84d0 2024-05-27T07:31:30.672 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:31:30.676+0000 7f0811b9f740 10 client.26255 fetch_fsmap finished waiting for FSMap version 101 2024-05-27T07:31:30.672 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:31:30.676+0000 7f0811b9f740 10 client.26255 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-05-27T07:31:30.672 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:31:30.676+0000 7f0811b9f740 20 client.26255 populate_metadata read hostname 'smithi052' 2024-05-27T07:31:30.672 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:31:30.676+0000 7f0811b9f740 1 --2- 172.21.15.52:0/3995294932 >> [v2:172.21.15.105:6838/500348151,v1:172.21.15.105:6839/500348151] conn(0x55c343dd7d80 0x55c3441d6f20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:31:30.672 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:31:30.676+0000 7f0811b9f740 4 client.26255 mds_command: new command op to 24527 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-27T07:31:30.672 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:31:30.676+0000 7f0811b9f740 1 -- 172.21.15.52:0/3995294932 --> [v2:172.21.15.105:6838/500348151,v1:172.21.15.105:6839/500348151] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55c34412eb80 con 0x55c343dd7d80 2024-05-27T07:31:30.673 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:31:30.677+0000 7f0808a8c640 1 --2- 172.21.15.52:0/3995294932 >> [v2:172.21.15.105:6838/500348151,v1:172.21.15.105:6839/500348151] conn(0x55c343dd7d80 0x55c3441d6f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:31:30.673 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:31:30.677+0000 7f0808a8c640 1 --2- 172.21.15.52:0/3995294932 >> [v2:172.21.15.105:6838/500348151,v1:172.21.15.105:6839/500348151] conn(0x55c343dd7d80 0x55c3441d6f20 crc :-1 s=READY pgs=117 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=2dbb3311033f281f server_cookie=65d3064feb3c73d1 in_seq=0 out_seq=0 2024-05-27T07:31:30.673 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:31:30.677+0000 7f07e3fff640 10 client.26255 ms_handle_connect on v2:172.21.15.105:6838/500348151 2024-05-27T07:31:30.674 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:31:30.680+0000 7f07e3fff640 1 -- 172.21.15.52:0/3995294932 <== mds.0 v2:172.21.15.105:6838/500348151 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24041 (crc 0 0 0) 0x55c34412eb80 con 0x55c343dd7d80 2024-05-27T07:31:30.674 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:31:30.680+0000 7f07e3fff640 10 client.26255 handle_command_reply: tid=0 2024-05-27T07:31:30.674 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:31:30.680+0000 7f0811b9f740 1 client.26255 shutdown 2024-05-27T07:31:30.674 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:31:30.680+0000 7f0811b9f740 1 -- 172.21.15.52:0/3995294932 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55c343df84d0 msgr2=0x55c3441c6ba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:31:30.674 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:31:30.680+0000 7f0811b9f740 1 --2- 172.21.15.52:0/3995294932 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55c343df84d0 0x55c3441c6ba0 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7f07f0100000 tx=0x7f07f00c5c30 comp rx=0 tx=0).stop 2024-05-27T07:31:30.675 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:31:30.680+0000 7f0811b9f740 1 -- 172.21.15.52:0/3995294932 shutdown_connections 2024-05-27T07:31:30.675 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:31:30.680+0000 7f0811b9f740 1 --2- 172.21.15.52:0/3995294932 >> [v2:172.21.15.105:6838/500348151,v1:172.21.15.105:6839/500348151] conn(0x55c343dd7d80 0x55c3441d6f20 crc :-1 s=READY pgs=117 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:31:30.675 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:31:30.680+0000 7f0811b9f740 1 --2- 172.21.15.52:0/3995294932 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55c343e191b0 0x55c3441c4ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:31:30.675 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:31:30.680+0000 7f07e3fff640 0 client.26255 ms_handle_reset on v2:172.21.15.105:6838/500348151 2024-05-27T07:31:30.675 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:31:30.680+0000 7f0811b9f740 1 --2- 172.21.15.52:0/3995294932 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55c343df84d0 0x55c3441c6ba0 unknown :-1 s=CLOSED pgs=178 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:31:30.675 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:31:30.680+0000 7f0811b9f740 1 --2- 172.21.15.52:0/3995294932 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55c343df02f0 0x55c3441c6640 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:31:30.675 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:31:30.680+0000 7f0811b9f740 1 -- 172.21.15.52:0/3995294932 >> 172.21.15.52:0/3995294932 conn(0x55c343e0d300 msgr2=0x55c3441ba3c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:31:30.675 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:31:30.680+0000 7f0811b9f740 1 -- 172.21.15.52:0/3995294932 shutdown_connections 2024-05-27T07:31:30.675 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:31:30.680+0000 7f0811b9f740 1 -- 172.21.15.52:0/3995294932 wait complete. 2024-05-27T07:31:30.675 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:31:30.681+0000 7f0811b9f740 20 client.26255 trim_cache size 0 max 16384 2024-05-27T07:31:30.695 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:31:30.700+0000 7f0811b9f740 1 Processor -- start 2024-05-27T07:31:30.695 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:31:30.700+0000 7f0811b9f740 1 -- start start 2024-05-27T07:31:30.695 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:31:30.700+0000 7f0811b9f740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55c343dd7d80 0x55c3441c4a60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:31:30.695 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:31:30.701+0000 7f0811b9f740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55c343df02f0 0x55c3441dc6e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:31:30.695 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:31:30.701+0000 7f0811b9f740 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55c343df84d0 0x55c3441dcc20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:31:30.695 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:31:30.701+0000 7f0811b9f740 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x7f08041b7aa0 con 0x55c343df84d0 2024-05-27T07:31:30.695 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:31:30.701+0000 7f0811b9f740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f08041b5ea0 con 0x55c343df02f0 2024-05-27T07:31:30.695 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:31:30.701+0000 7f0811b9f740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f08041b73d0 con 0x55c343dd7d80 2024-05-27T07:31:30.695 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:31:30.701+0000 7f0803fff640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55c343df02f0 0x55c3441dc6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:31:30.695 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:31:30.701+0000 7f0803fff640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55c343df02f0 0x55c3441dc6e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.52:37358/0 (socket says 172.21.15.52:37358) 2024-05-27T07:31:30.695 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:31:30.701+0000 7f0803fff640 1 -- 172.21.15.52:0/2295243660 learned_addr learned my addr 172.21.15.52:0/2295243660 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-05-27T07:31:30.695 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:31:30.701+0000 7f0808a8c640 1 --2- 172.21.15.52:0/2295243660 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55c343dd7d80 0x55c3441c4a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:31:30.696 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:31:30.701+0000 7f080928d640 1 --2- 172.21.15.52:0/2295243660 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55c343df84d0 0x55c3441dcc20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:31:30.696 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:31:30.701+0000 7f0808a8c640 1 -- 172.21.15.52:0/2295243660 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55c343df02f0 msgr2=0x55c3441dc6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:31:30.696 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:31:30.701+0000 7f0808a8c640 1 --2- 172.21.15.52:0/2295243660 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55c343df02f0 0x55c3441dc6e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:31:30.696 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:31:30.701+0000 7f0808a8c640 1 -- 172.21.15.52:0/2295243660 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55c343df84d0 msgr2=0x55c3441dcc20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:31:30.696 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:31:30.701+0000 7f0808a8c640 1 --2- 172.21.15.52:0/2295243660 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55c343df84d0 0x55c3441dcc20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:31:30.696 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:31:30.701+0000 7f0808a8c640 1 -- 172.21.15.52:0/2295243660 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55c3441b7930 con 0x55c343dd7d80 2024-05-27T07:31:30.696 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:31:30.701+0000 7f080928d640 1 --2- 172.21.15.52:0/2295243660 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55c343df84d0 0x55c3441dcc20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-27T07:31:30.697 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:31:30.702+0000 7f0808a8c640 1 --2- 172.21.15.52:0/2295243660 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55c343dd7d80 0x55c3441c4a60 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7f07f800f580 tx=0x7f07f80096a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=d82ca025d2dd2b37 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:31:30.697 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:31:30.703+0000 7f07ee7fc640 1 -- 172.21.15.52:0/2295243660 <== mon.1 v2:172.21.15.173:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f07f801e040 con 0x55c343dd7d80 2024-05-27T07:31:30.697 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:31:30.703+0000 7f07ee7fc640 1 -- 172.21.15.52:0/2295243660 <== mon.1 v2:172.21.15.173:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f07f802b070 con 0x55c343dd7d80 2024-05-27T07:31:30.697 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:31:30.703+0000 7f07ee7fc640 1 -- 172.21.15.52:0/2295243660 <== mon.1 v2:172.21.15.173:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f07f80048c0 con 0x55c343dd7d80 2024-05-27T07:31:30.700 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:31:30.704+0000 7f0811b9f740 1 -- 172.21.15.52:0/2295243660 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55c343dd7d80 msgr2=0x55c3441c4a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:31:30.700 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:31:30.704+0000 7f0811b9f740 1 --2- 172.21.15.52:0/2295243660 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55c343dd7d80 0x55c3441c4a60 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7f07f800f580 tx=0x7f07f80096a0 comp rx=0 tx=0).stop 2024-05-27T07:31:30.701 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:31:30.704+0000 7f0811b9f740 1 -- 172.21.15.52:0/2295243660 shutdown_connections 2024-05-27T07:31:30.701 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:31:30.704+0000 7f0811b9f740 1 --2- 172.21.15.52:0/2295243660 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55c343df84d0 0x55c3441dcc20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:31:30.701 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:31:30.704+0000 7f0811b9f740 1 --2- 172.21.15.52:0/2295243660 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55c343df02f0 0x55c3441dc6e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:31:30.701 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:31:30.704+0000 7f0811b9f740 1 --2- 172.21.15.52:0/2295243660 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55c343dd7d80 0x55c3441c4a60 unknown :-1 s=CLOSED pgs=143 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:31:30.701 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:31:30.704+0000 7f0811b9f740 1 -- 172.21.15.52:0/2295243660 >> 172.21.15.52:0/2295243660 conn(0x55c343e0d300 msgr2=0x55c3441c4350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:31:30.704 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:31:30.704+0000 7f0811b9f740 1 -- 172.21.15.52:0/2295243660 shutdown_connections 2024-05-27T07:31:30.704 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:31:30.704+0000 7f0811b9f740 1 -- 172.21.15.52:0/2295243660 wait complete. 2024-05-27T07:31:30.704 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:31:30.705+0000 7f0811b9f740 1 -- start start 2024-05-27T07:31:30.704 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:31:30.705+0000 7f0811b9f740 1 Processor -- start 2024-05-27T07:31:30.704 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:31:30.705+0000 7f0811b9f740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55c343dd7d80 0x55c344237e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:31:30.704 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:31:30.705+0000 7f0811b9f740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55c343df02f0 0x55c344238350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:31:30.704 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:31:30.705+0000 7f0811b9f740 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55c343df84d0 0x55c3441cc390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:31:30.704 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:31:30.705+0000 7f0811b9f740 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x7f08041b6c40 con 0x55c343df84d0 2024-05-27T07:31:30.704 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:31:30.705+0000 7f0811b9f740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f08041b6570 con 0x55c343df02f0 2024-05-27T07:31:30.705 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:31:30.705+0000 7f0811b9f740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x55c3441bb340 con 0x55c343dd7d80 2024-05-27T07:31:30.705 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:31:30.705+0000 7f0811b9f740 10 client.0 _ll_register_callbacks cb 0x7f07f0009560 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T07:31:30.705 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:31:30.706+0000 7f080928d640 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55c343df84d0 0x55c3441cc390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:31:30.705 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:31:30.706+0000 7f0803fff640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55c343df02f0 0x55c344238350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:31:30.705 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:31:30.706+0000 7f0808a8c640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55c343dd7d80 0x55c344237e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:31:30.705 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:31:30.706+0000 7f080928d640 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55c343df84d0 0x55c3441cc390 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.52:50722/0 (socket says 172.21.15.52:50722) 2024-05-27T07:31:30.705 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:31:30.706+0000 7f080928d640 1 -- 172.21.15.52:0/140781730 learned_addr learned my addr 172.21.15.52:0/140781730 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-05-27T07:31:30.705 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:31:30.706+0000 7f0803fff640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55c343df02f0 0x55c344238350 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.52:37374/0 (socket says 172.21.15.52:37374) 2024-05-27T07:31:30.705 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:31:30.706+0000 7f0808a8c640 1 -- 172.21.15.52:0/140781730 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55c343df02f0 msgr2=0x55c344238350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:31:30.705 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:31:30.706+0000 7f0808a8c640 1 --2- 172.21.15.52:0/140781730 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55c343df02f0 0x55c344238350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:31:30.705 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:31:30.706+0000 7f0808a8c640 1 -- 172.21.15.52:0/140781730 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55c343df84d0 msgr2=0x55c3441cc390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:31:30.706 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:31:30.706+0000 7f0808a8c640 1 --2- 172.21.15.52:0/140781730 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55c343df84d0 0x55c3441cc390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:31:30.706 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:31:30.706+0000 7f0808a8c640 1 -- 172.21.15.52:0/140781730 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55c3441cc980 con 0x55c343dd7d80 2024-05-27T07:31:30.706 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:31:30.706+0000 7f0808a8c640 1 --2- 172.21.15.52:0/140781730 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55c343dd7d80 0x55c344237e10 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7f07f80058c0 tx=0x7f07f8009d60 comp rx=0 tx=0).ready entity=mon.1 client_cookie=592039fbc0fe18dd server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:31:30.706 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:31:30.706+0000 7f07e3fff640 10 client.0 ms_handle_connect on v2:172.21.15.173:3300/0 2024-05-27T07:31:30.706 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:31:30.706+0000 7f07e3fff640 1 -- 172.21.15.52:0/140781730 <== mon.1 v2:172.21.15.173:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f07f8039060 con 0x55c343dd7d80 2024-05-27T07:31:30.706 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:31:30.707+0000 7f07e3fff640 1 -- 172.21.15.52:0/140781730 <== mon.1 v2:172.21.15.173:3300/0 2 ==== osd_map(116..116 src has 1..116) ==== 9379+0+0 (secure 0 0 0) 0x7f07f80035f0 con 0x55c343dd7d80 2024-05-27T07:31:30.706 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:31:30.707+0000 7f0811b9f740 1 -- 172.21.15.52:0/140781730 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f07f802b070 con 0x55c343dd7d80 2024-05-27T07:31:30.706 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:31:30.707+0000 7f07e3fff640 1 -- 172.21.15.52:0/140781730 <== mon.1 v2:172.21.15.173:3300/0 3 ==== mon_get_version_reply(handle=1 version=101) ==== 24+0+0 (secure 0 0 0) 0x7f07f801e070 con 0x55c343dd7d80 2024-05-27T07:31:30.706 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:31:30.707+0000 7f0811b9f740 10 client.26146 fetch_fsmap learned FSMap version 101 2024-05-27T07:31:30.706 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:31:30.707+0000 7f0811b9f740 1 -- 172.21.15.52:0/140781730 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({config=0+,fsmap=101,monmap=4+}) -- 0x55c34423f3b0 con 0x55c343dd7d80 2024-05-27T07:31:30.706 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:31:30.707+0000 7f07e3fff640 1 -- 172.21.15.52:0/140781730 <== mon.1 v2:172.21.15.173:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f07f802b070 con 0x55c343dd7d80 2024-05-27T07:31:30.706 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:31:30.707+0000 7f07e3fff640 1 -- 172.21.15.52:0/140781730 <== mon.1 v2:172.21.15.173:3300/0 5 ==== fsmap(e 101) ==== 4203+0+0 (secure 0 0 0) 0x7f07f8030030 con 0x55c343dd7d80 2024-05-27T07:31:30.707 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:31:30.707+0000 7f0811b9f740 10 client.26146 fetch_fsmap finished waiting for FSMap version 101 2024-05-27T07:31:30.707 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:31:30.707+0000 7f0811b9f740 10 client.26146 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-05-27T07:31:30.707 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:31:30.707+0000 7f0811b9f740 20 client.26146 populate_metadata read hostname 'smithi052' 2024-05-27T07:31:30.707 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:31:30.707+0000 7f0811b9f740 1 --2- 172.21.15.52:0/140781730 >> [v2:172.21.15.105:6838/500348151,v1:172.21.15.105:6839/500348151] conn(0x55c343e191b0 0x55c344241840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:31:30.707 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:31:30.707+0000 7f0811b9f740 4 client.26146 mds_command: new command op to 24527 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-05-27T07:31:30.707 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:31:30.707+0000 7f0811b9f740 1 -- 172.21.15.52:0/140781730 --> [v2:172.21.15.105:6838/500348151,v1:172.21.15.105:6839/500348151] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55c344128360 con 0x55c343e191b0 2024-05-27T07:31:30.707 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:31:30.707+0000 7f0803fff640 1 --2- 172.21.15.52:0/140781730 >> [v2:172.21.15.105:6838/500348151,v1:172.21.15.105:6839/500348151] conn(0x55c343e191b0 0x55c344241840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:31:30.707 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:31:30.708+0000 7f07e3fff640 10 client.26146 ms_handle_connect on v2:172.21.15.105:6838/500348151 2024-05-27T07:31:30.707 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:31:30.708+0000 7f0803fff640 1 --2- 172.21.15.52:0/140781730 >> [v2:172.21.15.105:6838/500348151,v1:172.21.15.105:6839/500348151] conn(0x55c343e191b0 0x55c344241840 crc :-1 s=READY pgs=118 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=147900915644f536 server_cookie=da09abd979bf6983 in_seq=0 out_seq=0 2024-05-27T07:31:32.032 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:31:32.037+0000 7f12250ae700 1 -- 172.21.15.52:0/1406901812 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f122010a920 msgr2=0x7f122010ad30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:31:32.033 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:31:32.037+0000 7f12250ae700 1 --2- 172.21.15.52:0/1406901812 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f122010a920 0x7f122010ad30 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7f1210009990 tx=0x7f1210009ca0 comp rx=0 tx=0).stop 2024-05-27T07:31:32.033 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:31:32.038+0000 7f12250ae700 1 -- 172.21.15.52:0/1406901812 shutdown_connections 2024-05-27T07:31:32.033 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:31:32.038+0000 7f12250ae700 1 --2- 172.21.15.52:0/1406901812 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f122011d920 0x7f122011fd40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:31:32.033 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:31:32.038+0000 7f12250ae700 1 --2- 172.21.15.52:0/1406901812 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f122010a920 0x7f122010ad30 unknown :-1 s=CLOSED pgs=179 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:31:32.033 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:31:32.038+0000 7f12250ae700 1 --2- 172.21.15.52:0/1406901812 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f122010b270 0x7f122011d3e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:31:32.033 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:31:32.038+0000 7f12250ae700 1 -- 172.21.15.52:0/1406901812 >> 172.21.15.52:0/1406901812 conn(0x7f122006f4c0 msgr2=0x7f122006f900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:31:32.038 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:31:32.042+0000 7f12250ae700 1 -- 172.21.15.52:0/1406901812 shutdown_connections 2024-05-27T07:31:32.038 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:31:32.042+0000 7f12250ae700 1 -- 172.21.15.52:0/1406901812 wait complete. 2024-05-27T07:31:32.038 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:31:32.043+0000 7f12250ae700 1 Processor -- start 2024-05-27T07:31:32.039 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:31:32.044+0000 7f12250ae700 1 -- start start 2024-05-27T07:31:32.039 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:31:32.044+0000 7f12250ae700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f122010a920 0x7f122011cef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:31:32.039 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:31:32.044+0000 7f12250ae700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f122010b270 0x7f1220117010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:31:32.039 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:31:32.044+0000 7f12250ae700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f122011d920 0x7f1220117550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:31:32.039 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:31:32.044+0000 7f12250ae700 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x7f1220113090 con 0x7f122010a920 2024-05-27T07:31:32.039 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:31:32.044+0000 7f12250ae700 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f1220112f10 con 0x7f122011d920 2024-05-27T07:31:32.039 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:31:32.044+0000 7f12250ae700 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f1220113210 con 0x7f122010b270 2024-05-27T07:31:32.039 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:31:32.045+0000 7f121f59e700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f122011d920 0x7f1220117550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:31:32.039 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:31:32.045+0000 7f121f59e700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f122011d920 0x7f1220117550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.52:37386/0 (socket says 172.21.15.52:37386) 2024-05-27T07:31:32.039 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:31:32.045+0000 7f121f59e700 1 -- 172.21.15.52:0/2951848466 learned_addr learned my addr 172.21.15.52:0/2951848466 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-05-27T07:31:32.039 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:31:32.045+0000 7f121ed9d700 1 --2- 172.21.15.52:0/2951848466 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f122010a920 0x7f122011cef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:31:32.039 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:31:32.046+0000 7f121e59c700 1 --2- 172.21.15.52:0/2951848466 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f122010b270 0x7f1220117010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:31:32.041 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:31:32.047+0000 7f121ed9d700 1 -- 172.21.15.52:0/2951848466 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f122010b270 msgr2=0x7f1220117010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:31:32.041 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:31:32.047+0000 7f121ed9d700 1 --2- 172.21.15.52:0/2951848466 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f122010b270 0x7f1220117010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:31:32.041 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:31:32.047+0000 7f121ed9d700 1 -- 172.21.15.52:0/2951848466 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f122011d920 msgr2=0x7f1220117550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:31:32.041 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:31:32.047+0000 7f121ed9d700 1 --2- 172.21.15.52:0/2951848466 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f122011d920 0x7f1220117550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:31:32.042 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:31:32.047+0000 7f121ed9d700 1 -- 172.21.15.52:0/2951848466 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1220117b10 con 0x7f122010a920 2024-05-27T07:31:32.042 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:31:32.048+0000 7f121ed9d700 1 --2- 172.21.15.52:0/2951848466 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f122010a920 0x7f122011cef0 secure :-1 s=READY pgs=657 cs=0 l=1 rev1=1 crypto rx=0x7f12100098e0 tx=0x7f1210005b00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:31:32.047 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:31:32.051+0000 7f1207fff700 1 -- 172.21.15.52:0/2951848466 <== mon.0 v2:172.21.15.52:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1210003a50 con 0x7f122010a920 2024-05-27T07:31:32.047 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:31:32.051+0000 7f1207fff700 1 -- 172.21.15.52:0/2951848466 <== mon.0 v2:172.21.15.52:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f1210003bf0 con 0x7f122010a920 2024-05-27T07:31:32.047 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:31:32.051+0000 7f1207fff700 1 -- 172.21.15.52:0/2951848466 <== mon.0 v2:172.21.15.52:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1210017650 con 0x7f122010a920 2024-05-27T07:31:32.047 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:31:32.053+0000 7f12250ae700 1 -- 172.21.15.52:0/2951848466 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1220117da0 con 0x7f122010a920 2024-05-27T07:31:32.047 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:31:32.053+0000 7f12250ae700 1 -- 172.21.15.52:0/2951848466 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f122007bc40 con 0x7f122010a920 2024-05-27T07:31:32.048 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:31:32.055+0000 7f12250ae700 1 -- 172.21.15.52:0/2951848466 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f122010c060 con 0x7f122010a920 2024-05-27T07:31:32.052 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:31:32.058+0000 7f1207fff700 1 -- 172.21.15.52:0/2951848466 <== mon.0 v2:172.21.15.52:3300/0 4 ==== mgrmap(e 26) ==== 149783+0+0 (secure 0 0 0) 0x7f1210009d10 con 0x7f122010a920 2024-05-27T07:31:32.052 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:31:32.059+0000 7f1207fff700 1 --2- 172.21.15.52:0/2951848466 >> [v2:172.21.15.52:6800/1822318384,v1:172.21.15.52:6801/1822318384] conn(0x7f12080b2b80 0x7f12080b5090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:31:32.056 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:31:32.064+0000 7f121e59c700 1 --2- 172.21.15.52:0/2951848466 >> [v2:172.21.15.52:6800/1822318384,v1:172.21.15.52:6801/1822318384] conn(0x7f12080b2b80 0x7f12080b5090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:31:32.056 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:31:32.064+0000 7f1207fff700 1 -- 172.21.15.52:0/2951848466 <== mon.0 v2:172.21.15.52:3300/0 5 ==== osd_map(116..116 src has 1..116) ==== 9379+0+0 (secure 0 0 0) 0x7f12100a7560 con 0x7f122010a920 2024-05-27T07:31:32.069 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:31:32.077+0000 7f1207fff700 1 -- 172.21.15.52:0/2951848466 <== mon.0 v2:172.21.15.52:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191963 (secure 0 0 0) 0x7f1210013070 con 0x7f122010a920 2024-05-27T07:31:32.069 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:31:32.077+0000 7f121e59c700 1 --2- 172.21.15.52:0/2951848466 >> [v2:172.21.15.52:6800/1822318384,v1:172.21.15.52:6801/1822318384] conn(0x7f12080b2b80 0x7f12080b5090 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7f1220112040 tx=0x7f121800b040 comp rx=0 tx=0).ready entity=mgr.15333 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:31:32.442 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:31:32 smithi105 ceph-mon[35977]: pgmap v195: 129 pgs: 129 active+clean; 496 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.5 MiB/s rd, 5.9 MiB/s wr, 649 op/s 2024-05-27T07:31:32.495 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:31:32 smithi052 ceph-mon[31955]: pgmap v195: 129 pgs: 129 active+clean; 496 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.5 MiB/s rd, 5.9 MiB/s wr, 649 op/s 2024-05-27T07:31:32.604 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:31:32 smithi173 ceph-mon[37000]: pgmap v195: 129 pgs: 129 active+clean; 496 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.5 MiB/s rd, 5.9 MiB/s wr, 649 op/s 2024-05-27T07:31:32.658 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:31:32.665+0000 7f12250ae700 1 -- 172.21.15.52:0/2951848466 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f122002e5e0 con 0x7f122010a920 2024-05-27T07:31:32.666 INFO:teuthology.orchestra.run.smithi052.stdout: 2024-05-27T07:31:32.667 INFO:teuthology.orchestra.run.smithi052.stdout:{"epoch":101,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":101,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-05-27T07:21:50.521783+0000","modified":"2024-05-27T07:31:22.605010+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24527,"mds_1":24497,"mds_2":24475,"mds_3":24473,"mds_4":24466},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24466":{"gid":24466,"name":"c","rank":4,"incarnation":26,"state":"up:active","state_seq":25,"addr":"172.21.15.173:6837/1396363827","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6836","nonce":1396363827},{"type":"v1","addr":"172.21.15.173:6837","nonce":1396363827}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24473":{"gid":24473,"name":"e","rank":3,"incarnation":24,"state":"up:active","state_seq":26,"addr":"172.21.15.105:6835/3298498384","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6834","nonce":3298498384},{"type":"v1","addr":"172.21.15.105:6835","nonce":3298498384}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24475":{"gid":24475,"name":"i","rank":2,"incarnation":22,"state":"up:active","state_seq":23,"addr":"172.21.15.173:6839/564778293","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6838","nonce":564778293},{"type":"v1","addr":"172.21.15.173:6839","nonce":564778293}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24497":{"gid":24497,"name":"h","rank":1,"incarnation":20,"state":"up:active","state_seq":24,"addr":"172.21.15.105:6837/191393149","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6836","nonce":191393149},{"type":"v1","addr":"172.21.15.105:6837","nonce":191393149}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24527":{"gid":24527,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":5,"addr":"172.21.15.105:6839/500348151","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6838","nonce":500348151},{"type":"v1","addr":"172.21.15.105:6839","nonce":500348151}]},"join_fscid":-1,"export_targets":[1,2,3,4],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24527,"qdb_cluster":[24466,24473,24475,24497,24527]},"id":1}]} 2024-05-27T07:31:32.667 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:31:32.667+0000 7f1207fff700 1 -- 172.21.15.52:0/2951848466 <== mon.0 v2:172.21.15.52:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 101 v101) ==== 95+0+6127 (secure 0 0 0) 0x7f12100ab670 con 0x7f122010a920 2024-05-27T07:31:32.675 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:31:32.677+0000 7f12250ae700 1 -- 172.21.15.52:0/2951848466 >> [v2:172.21.15.52:6800/1822318384,v1:172.21.15.52:6801/1822318384] conn(0x7f12080b2b80 msgr2=0x7f12080b5090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:31:32.675 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:31:32.677+0000 7f12250ae700 1 --2- 172.21.15.52:0/2951848466 >> [v2:172.21.15.52:6800/1822318384,v1:172.21.15.52:6801/1822318384] conn(0x7f12080b2b80 0x7f12080b5090 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7f1220112040 tx=0x7f121800b040 comp rx=0 tx=0).stop 2024-05-27T07:31:32.675 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:31:32.677+0000 7f12250ae700 1 -- 172.21.15.52:0/2951848466 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f122010a920 msgr2=0x7f122011cef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:31:32.675 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:31:32.677+0000 7f12250ae700 1 --2- 172.21.15.52:0/2951848466 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f122010a920 0x7f122011cef0 secure :-1 s=READY pgs=657 cs=0 l=1 rev1=1 crypto rx=0x7f12100098e0 tx=0x7f1210005b00 comp rx=0 tx=0).stop 2024-05-27T07:31:32.676 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:31:32.679+0000 7f12250ae700 1 -- 172.21.15.52:0/2951848466 shutdown_connections 2024-05-27T07:31:32.676 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:31:32.679+0000 7f12250ae700 1 --2- 172.21.15.52:0/2951848466 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f122011d920 0x7f1220117550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:31:32.676 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:31:32.679+0000 7f12250ae700 1 --2- 172.21.15.52:0/2951848466 >> [v2:172.21.15.52:6800/1822318384,v1:172.21.15.52:6801/1822318384] conn(0x7f12080b2b80 0x7f12080b5090 secure :-1 s=CLOSED pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7f1220112040 tx=0x7f121800b040 comp rx=0 tx=0).stop 2024-05-27T07:31:32.676 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:31:32.679+0000 7f12250ae700 1 --2- 172.21.15.52:0/2951848466 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f122010a920 0x7f122011cef0 unknown :-1 s=CLOSED pgs=657 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:31:32.676 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:31:32.679+0000 7f12250ae700 1 --2- 172.21.15.52:0/2951848466 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f122010b270 0x7f1220117010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:31:32.676 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:31:32.679+0000 7f12250ae700 1 -- 172.21.15.52:0/2951848466 >> 172.21.15.52:0/2951848466 conn(0x7f122006f4c0 msgr2=0x7f122011df10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:31:32.676 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:31:32.682+0000 7f12250ae700 1 -- 172.21.15.52:0/2951848466 shutdown_connections 2024-05-27T07:31:32.676 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:31:32.683+0000 7f12250ae700 1 -- 172.21.15.52:0/2951848466 wait complete. 2024-05-27T07:31:32.678 INFO:teuthology.orchestra.run.smithi052.stderr:dumped fsmap epoch 101 2024-05-27T07:31:33.232 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:31:33 smithi052 ceph-mon[31955]: from='client.? 172.21.15.52:0/2951848466' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-27T07:31:33.440 DEBUG:teuthology.orchestra.run.smithi052:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:38f9d900c83c63799fbdbe61acc9a11b0d3554a6 shell --fsid 73df2636-1bf8-11ef-bc9b-c7b262605968 -- ceph tell mds.1:4 damage ls 2024-05-27T07:31:33.442 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:31:33 smithi105 ceph-mon[35977]: from='client.? 172.21.15.52:0/2951848466' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-27T07:31:33.604 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:31:33 smithi173 ceph-mon[37000]: from='client.? 172.21.15.52:0/2951848466' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-27T07:31:34.240 INFO:teuthology.orchestra.run.smithi052.stderr:Inferring config /var/lib/ceph/73df2636-1bf8-11ef-bc9b-c7b262605968/mon.a/config 2024-05-27T07:31:34.419 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:31:34 smithi052 ceph-mon[31955]: pgmap v196: 129 pgs: 129 active+clean; 505 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 6.3 MiB/s wr, 610 op/s 2024-05-27T07:31:34.442 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:31:34 smithi105 ceph-mon[35977]: pgmap v196: 129 pgs: 129 active+clean; 505 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 6.3 MiB/s wr, 610 op/s 2024-05-27T07:31:34.604 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:31:34 smithi173 ceph-mon[37000]: pgmap v196: 129 pgs: 129 active+clean; 505 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 6.3 MiB/s wr, 610 op/s 2024-05-27T07:31:35.219 INFO:tasks.workunit.client.0.smithi052.stdout: 132 814208 273 580614 179 448901 653 2024-05-27T07:31:36.345 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:31:36.353+0000 7f07e3fff640 1 -- 172.21.15.52:0/140781730 <== mds.0 v2:172.21.15.105:6838/500348151 1 ==== command_reply(tid 0: 0 ) ==== 8+0+25 (crc 0 0 0) 0x55c344128360 con 0x55c343e191b0 2024-05-27T07:31:36.346 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:31:36.353+0000 7f07e3fff640 10 client.26146 handle_command_reply: tid=0 2024-05-27T07:31:36.346 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:31:36.353+0000 7f0811b9f740 1 client.26146 shutdown 2024-05-27T07:31:36.346 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:31:36.353+0000 7f0811b9f740 1 -- 172.21.15.52:0/140781730 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55c343dd7d80 msgr2=0x55c344237e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:31:36.347 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:31:36.353+0000 7f0811b9f740 1 --2- 172.21.15.52:0/140781730 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55c343dd7d80 0x55c344237e10 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7f07f80058c0 tx=0x7f07f8009d60 comp rx=0 tx=0).stop 2024-05-27T07:31:36.347 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:31:36.354+0000 7f0811b9f740 1 -- 172.21.15.52:0/140781730 shutdown_connections 2024-05-27T07:31:36.347 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:31:36.354+0000 7f0811b9f740 1 --2- 172.21.15.52:0/140781730 >> [v2:172.21.15.105:6838/500348151,v1:172.21.15.105:6839/500348151] conn(0x55c343e191b0 0x55c344241840 crc :-1 s=READY pgs=118 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:31:36.347 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:31:36.354+0000 7f0811b9f740 1 --2- 172.21.15.52:0/140781730 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55c343df84d0 0x55c3441cc390 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:31:36.347 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:31:36.354+0000 7f0811b9f740 1 --2- 172.21.15.52:0/140781730 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55c343df02f0 0x55c344238350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:31:36.347 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:31:36.354+0000 7f0811b9f740 1 --2- 172.21.15.52:0/140781730 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55c343dd7d80 0x55c344237e10 unknown :-1 s=CLOSED pgs=144 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:31:36.347 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:31:36.354+0000 7f0811b9f740 1 -- 172.21.15.52:0/140781730 >> 172.21.15.52:0/140781730 conn(0x55c343e0d300 msgr2=0x55c3441ba320 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:31:36.347 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:31:36.354+0000 7f07e3fff640 0 client.26146 ms_handle_reset on v2:172.21.15.105:6838/500348151 2024-05-27T07:31:36.347 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:31:36.354+0000 7f0811b9f740 1 -- 172.21.15.52:0/140781730 shutdown_connections 2024-05-27T07:31:36.347 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:31:36.354+0000 7f0811b9f740 1 -- 172.21.15.52:0/140781730 wait complete. 2024-05-27T07:31:36.348 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:31:36.354+0000 7f0811b9f740 20 client.26146 trim_cache size 0 max 16384 2024-05-27T07:31:36.348 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stdout:{ 2024-05-27T07:31:36.348 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stdout: "return_code": 0 2024-05-27T07:31:36.348 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stdout:} 2024-05-27T07:31:36.354 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:31:36.362+0000 7f0809a8e640 1 -- 172.21.15.52:0/2557948725 >> [v2:172.21.15.52:6800/1822318384,v1:172.21.15.52:6801/1822318384] conn(0x7f07f40b2a20 msgr2=0x7f07f40b4f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:31:36.354 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:31:36.362+0000 7f0809a8e640 1 --2- 172.21.15.52:0/2557948725 >> [v2:172.21.15.52:6800/1822318384,v1:172.21.15.52:6801/1822318384] conn(0x7f07f40b2a20 0x7f07f40b4f30 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x55c3441b7700 tx=0x7f07fc009b10 comp rx=0 tx=0).stop 2024-05-27T07:31:36.355 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:31:36.362+0000 7f0809a8e640 1 -- 172.21.15.52:0/2557948725 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f0804109ab0 msgr2=0x7f080419ac30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:31:36.355 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:31:36.362+0000 7f0809a8e640 1 --2- 172.21.15.52:0/2557948725 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f0804109ab0 0x7f080419ac30 secure :-1 s=READY pgs=655 cs=0 l=1 rev1=1 crypto rx=0x7f07f0031440 tx=0x7f07f0003b20 comp rx=0 tx=0).stop 2024-05-27T07:31:36.359 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:31:36.366+0000 7f0809a8e640 1 -- 172.21.15.52:0/2557948725 shutdown_connections 2024-05-27T07:31:36.359 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:31:36.366+0000 7f0809a8e640 1 --2- 172.21.15.52:0/2557948725 >> [v2:172.21.15.52:6800/1822318384,v1:172.21.15.52:6801/1822318384] conn(0x7f07f40b2a20 0x7f07f40b4f30 unknown :-1 s=CLOSED pgs=146 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:31:36.359 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:31:36.366+0000 7f0809a8e640 1 --2- 172.21.15.52:0/2557948725 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f0804109f50 0x7f080419eef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:31:36.359 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:31:36.366+0000 7f0809a8e640 1 --2- 172.21.15.52:0/2557948725 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f0804109ab0 0x7f080419ac30 unknown :-1 s=CLOSED pgs=655 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:31:36.360 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:31:36.366+0000 7f0809a8e640 1 --2- 172.21.15.52:0/2557948725 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f0804104200 0x7f080419a6d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:31:36.360 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:31:36.366+0000 7f0809a8e640 1 -- 172.21.15.52:0/2557948725 >> 172.21.15.52:0/2557948725 conn(0x7f0804079170 msgr2=0x7f0804102230 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:31:36.360 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:31:36.367+0000 7f0809a8e640 1 -- 172.21.15.52:0/2557948725 shutdown_connections 2024-05-27T07:31:36.360 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:31:36.367+0000 7f0809a8e640 1 -- 172.21.15.52:0/2557948725 wait complete. 2024-05-27T07:31:36.442 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:31:36 smithi105 ceph-mon[35977]: pgmap v197: 129 pgs: 129 active+clean; 517 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 7.2 MiB/s wr, 895 op/s 2024-05-27T07:31:36.482 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:31:36 smithi052 ceph-mon[31955]: pgmap v197: 129 pgs: 129 active+clean; 517 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 7.2 MiB/s wr, 895 op/s 2024-05-27T07:31:36.604 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:31:36 smithi173 ceph-mon[37000]: pgmap v197: 129 pgs: 129 active+clean; 517 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 7.2 MiB/s wr, 895 op/s 2024-05-27T07:31:38.483 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:31:38 smithi052 ceph-mon[31955]: pgmap v198: 129 pgs: 129 active+clean; 517 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 6.7 MiB/s wr, 1.07k op/s 2024-05-27T07:31:38.604 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:31:38 smithi173 ceph-mon[37000]: pgmap v198: 129 pgs: 129 active+clean; 517 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 6.7 MiB/s wr, 1.07k op/s 2024-05-27T07:31:38.692 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:31:38 smithi105 ceph-mon[35977]: pgmap v198: 129 pgs: 129 active+clean; 517 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 6.7 MiB/s wr, 1.07k op/s 2024-05-27T07:31:38.780 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:31:38.787+0000 7f1b858fd700 1 -- 172.21.15.52:0/1689112995 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f1b801105e0 msgr2=0x7f1b801109f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:31:38.780 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:31:38.787+0000 7f1b858fd700 1 --2- 172.21.15.52:0/1689112995 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f1b801105e0 0x7f1b801109f0 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7f1b78009f90 tx=0x7f1b7800b330 comp rx=0 tx=0).stop 2024-05-27T07:31:38.782 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:31:38.790+0000 7f1b858fd700 1 -- 172.21.15.52:0/1689112995 shutdown_connections 2024-05-27T07:31:38.782 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:31:38.790+0000 7f1b858fd700 1 --2- 172.21.15.52:0/1689112995 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f1b80110f30 0x7f1b801117c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:31:38.783 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:31:38.790+0000 7f1b858fd700 1 --2- 172.21.15.52:0/1689112995 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f1b8010a970 0x7f1b8010ae30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:31:38.783 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:31:38.790+0000 7f1b858fd700 1 --2- 172.21.15.52:0/1689112995 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f1b801105e0 0x7f1b801109f0 unknown :-1 s=CLOSED pgs=145 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:31:38.783 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:31:38.790+0000 7f1b858fd700 1 -- 172.21.15.52:0/1689112995 >> 172.21.15.52:0/1689112995 conn(0x7f1b8006f4c0 msgr2=0x7f1b8006f900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:31:38.786 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:31:38.791+0000 7f1b858fd700 1 -- 172.21.15.52:0/1689112995 shutdown_connections 2024-05-27T07:31:38.786 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:31:38.794+0000 7f1b858fd700 1 -- 172.21.15.52:0/1689112995 wait complete. 2024-05-27T07:31:38.794 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:31:38.795+0000 7f1b858fd700 1 Processor -- start 2024-05-27T07:31:38.794 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:31:38.795+0000 7f1b858fd700 1 -- start start 2024-05-27T07:31:38.794 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:31:38.796+0000 7f1b858fd700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f1b8010a970 0x7f1b801ab540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:31:38.794 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:31:38.796+0000 7f1b858fd700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f1b801105e0 0x7f1b801aba80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:31:38.794 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:31:38.796+0000 7f1b858fd700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f1b80110f30 0x7f1b801a56b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:31:38.795 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:31:38.796+0000 7f1b7effd700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f1b8010a970 0x7f1b801ab540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:31:38.795 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:31:38.796+0000 7f1b7effd700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f1b8010a970 0x7f1b801ab540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.52:37404/0 (socket says 172.21.15.52:37404) 2024-05-27T07:31:38.795 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:31:38.796+0000 7f1b7effd700 1 -- 172.21.15.52:0/2694532192 learned_addr learned my addr 172.21.15.52:0/2694532192 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-05-27T07:31:38.795 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:31:38.796+0000 7f1b7e7fc700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f1b801105e0 0x7f1b801aba80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:31:38.795 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:31:38.796+0000 7f1b858fd700 1 -- 172.21.15.52:0/2694532192 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x7f1b8011ef20 con 0x7f1b80110f30 2024-05-27T07:31:38.795 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:31:38.796+0000 7f1b858fd700 1 -- 172.21.15.52:0/2694532192 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f1b801148a0 con 0x7f1b8010a970 2024-05-27T07:31:38.795 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:31:38.796+0000 7f1b858fd700 1 -- 172.21.15.52:0/2694532192 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f1b8011f0a0 con 0x7f1b801105e0 2024-05-27T07:31:38.795 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:31:38.798+0000 7f1b7e7fc700 1 -- 172.21.15.52:0/2694532192 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f1b8010a970 msgr2=0x7f1b801ab540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:31:38.795 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:31:38.798+0000 7f1b7e7fc700 1 --2- 172.21.15.52:0/2694532192 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f1b8010a970 0x7f1b801ab540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:31:38.795 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:31:38.798+0000 7f1b7e7fc700 1 -- 172.21.15.52:0/2694532192 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f1b80110f30 msgr2=0x7f1b801a56b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:31:38.795 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:31:38.798+0000 7f1b7e7fc700 1 --2- 172.21.15.52:0/2694532192 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f1b80110f30 0x7f1b801a56b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:31:38.795 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:31:38.798+0000 7f1b7e7fc700 1 -- 172.21.15.52:0/2694532192 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1b801a5e20 con 0x7f1b801105e0 2024-05-27T07:31:38.795 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:31:38.798+0000 7f1b7e7fc700 1 --2- 172.21.15.52:0/2694532192 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f1b801105e0 0x7f1b801aba80 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7f1b7800b6c0 tx=0x7f1b78007760 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:31:38.796 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:31:38.800+0000 7f1b848fb700 1 -- 172.21.15.52:0/2694532192 <== mon.1 v2:172.21.15.173:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1b78020030 con 0x7f1b801105e0 2024-05-27T07:31:38.796 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:31:38.800+0000 7f1b848fb700 1 -- 172.21.15.52:0/2694532192 <== mon.1 v2:172.21.15.173:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f1b78007bd0 con 0x7f1b801105e0 2024-05-27T07:31:38.796 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:31:38.800+0000 7f1b848fb700 1 -- 172.21.15.52:0/2694532192 <== mon.1 v2:172.21.15.173:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1b780092e0 con 0x7f1b801105e0 2024-05-27T07:31:38.796 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:31:38.802+0000 7f1b858fd700 1 -- 172.21.15.52:0/2694532192 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1b801a60b0 con 0x7f1b801105e0 2024-05-27T07:31:38.796 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:31:38.802+0000 7f1b858fd700 1 -- 172.21.15.52:0/2694532192 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1b801a6510 con 0x7f1b801105e0 2024-05-27T07:31:38.801 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:31:38.808+0000 7f1b848fb700 1 -- 172.21.15.52:0/2694532192 <== mon.1 v2:172.21.15.173:3300/0 4 ==== mgrmap(e 26) ==== 149783+0+0 (secure 0 0 0) 0x7f1b7801a040 con 0x7f1b801105e0 2024-05-27T07:31:38.806 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:31:38.812+0000 7f1b94312b80 1 Processor -- start 2024-05-27T07:31:38.806 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:31:38.812+0000 7f1b94312b80 1 -- start start 2024-05-27T07:31:38.806 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:31:38.812+0000 7f1b94312b80 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x558aaffd9e40 0x558ab015d710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:31:38.807 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:31:38.812+0000 7f1b94312b80 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x558ab015dc50 0x558ab0157a40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:31:38.807 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:31:38.813+0000 7f1b94312b80 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x558ab0157f80 0x558ab0158390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:31:38.807 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:31:38.813+0000 7f1b94312b80 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x558ab0086c90 con 0x558ab0157f80 2024-05-27T07:31:38.807 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:31:38.813+0000 7f1b94312b80 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x558ab0076830 con 0x558ab015dc50 2024-05-27T07:31:38.807 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:31:38.813+0000 7f1b94312b80 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x558ab007a870 con 0x558aaffd9e40 2024-05-27T07:31:38.808 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:31:38.815+0000 7f1b7e7fc700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x558ab0157f80 0x558ab0158390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:31:38.808 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:31:38.815+0000 7f1b7e7fc700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x558ab0157f80 0x558ab0158390 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.52:50784/0 (socket says 172.21.15.52:50784) 2024-05-27T07:31:38.808 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:31:38.815+0000 7f1b7e7fc700 1 -- 172.21.15.52:0/3226850007 learned_addr learned my addr 172.21.15.52:0/3226850007 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-05-27T07:31:38.808 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:31:38.815+0000 7f1b7effd700 1 --2- 172.21.15.52:0/3226850007 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x558ab015dc50 0x558ab0157a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:31:38.809 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:31:38.816+0000 7f1b7e7fc700 1 -- 172.21.15.52:0/3226850007 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x558aaffd9e40 msgr2=0x558ab015d710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:31:38.809 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:31:38.816+0000 7f1b7e7fc700 1 --2- 172.21.15.52:0/3226850007 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x558aaffd9e40 0x558ab015d710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:31:38.809 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:31:38.816+0000 7f1b7e7fc700 1 -- 172.21.15.52:0/3226850007 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x558ab015dc50 msgr2=0x558ab0157a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:31:38.809 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:31:38.816+0000 7f1b7e7fc700 1 --2- 172.21.15.52:0/3226850007 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x558ab015dc50 0x558ab0157a40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:31:38.809 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:31:38.816+0000 7f1b7e7fc700 1 -- 172.21.15.52:0/3226850007 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x558aaffc68d0 con 0x558ab0157f80 2024-05-27T07:31:38.809 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:31:38.816+0000 7f1b848fb700 1 --2- 172.21.15.52:0/2694532192 >> [v2:172.21.15.52:6800/1822318384,v1:172.21.15.52:6801/1822318384] conn(0x7f1b680b2bc0 0x7f1b680b50d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:31:38.809 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:31:38.816+0000 7f1b7e7fc700 1 --2- 172.21.15.52:0/3226850007 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x558ab0157f80 0x558ab0158390 secure :-1 s=READY pgs=658 cs=0 l=1 rev1=1 crypto rx=0x7f1b780e0a30 tx=0x7f1b780e9a60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b95a0beca82484d3 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:31:38.810 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:31:38.818+0000 7f1b848fb700 1 -- 172.21.15.52:0/2694532192 <== mon.1 v2:172.21.15.173:3300/0 5 ==== osd_map(116..116 src has 1..116) ==== 9379+0+0 (secure 0 0 0) 0x7f1b780e3760 con 0x7f1b801105e0 2024-05-27T07:31:38.814 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:31:38.819+0000 7f1b7f7fe700 1 --2- 172.21.15.52:0/2694532192 >> [v2:172.21.15.52:6800/1822318384,v1:172.21.15.52:6801/1822318384] conn(0x7f1b680b2bc0 0x7f1b680b50d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:31:38.814 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:31:38.820+0000 7f1b65ffb700 1 -- 172.21.15.52:0/3226850007 <== mon.0 v2:172.21.15.52:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1b780a65d0 con 0x558ab0157f80 2024-05-27T07:31:38.814 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:31:38.820+0000 7f1b65ffb700 1 -- 172.21.15.52:0/3226850007 <== mon.0 v2:172.21.15.52:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f1b780aa220 con 0x558ab0157f80 2024-05-27T07:31:38.814 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:31:38.820+0000 7f1b65ffb700 1 -- 172.21.15.52:0/3226850007 <== mon.0 v2:172.21.15.52:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1b780a9e80 con 0x558ab0157f80 2024-05-27T07:31:38.814 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:31:38.821+0000 7f1b7f7fe700 1 --2- 172.21.15.52:0/2694532192 >> [v2:172.21.15.52:6800/1822318384,v1:172.21.15.52:6801/1822318384] conn(0x7f1b680b2bc0 0x7f1b680b50d0 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7f1b74005b40 tx=0x7f1b74009db0 comp rx=0 tx=0).ready entity=mgr.15333 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:31:38.817 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:31:38.823+0000 7f1b94312b80 1 -- 172.21.15.52:0/3226850007 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x558ab0157f80 msgr2=0x558ab0158390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:31:38.817 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:31:38.823+0000 7f1b94312b80 1 --2- 172.21.15.52:0/3226850007 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x558ab0157f80 0x558ab0158390 secure :-1 s=READY pgs=658 cs=0 l=1 rev1=1 crypto rx=0x7f1b780e0a30 tx=0x7f1b780e9a60 comp rx=0 tx=0).stop 2024-05-27T07:31:38.821 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:31:38.828+0000 7f1b94312b80 1 -- 172.21.15.52:0/3226850007 shutdown_connections 2024-05-27T07:31:38.821 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:31:38.828+0000 7f1b94312b80 1 --2- 172.21.15.52:0/3226850007 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x558ab0157f80 0x558ab0158390 unknown :-1 s=CLOSED pgs=658 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:31:38.821 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:31:38.828+0000 7f1b94312b80 1 --2- 172.21.15.52:0/3226850007 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x558aaffd9e40 0x558ab015d710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:31:38.821 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:31:38.828+0000 7f1b94312b80 1 --2- 172.21.15.52:0/3226850007 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x558ab015dc50 0x558ab0157a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:31:38.821 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:31:38.828+0000 7f1b94312b80 1 -- 172.21.15.52:0/3226850007 >> 172.21.15.52:0/3226850007 conn(0x558ab0034af0 msgr2=0x558ab0152640 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:31:38.821 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:31:38.829+0000 7f1b94312b80 1 -- 172.21.15.52:0/3226850007 shutdown_connections 2024-05-27T07:31:38.827 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:31:38.832+0000 7f1b94312b80 1 -- 172.21.15.52:0/3226850007 wait complete. 2024-05-27T07:31:38.827 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:31:38.832+0000 7f1b94312b80 1 -- start start 2024-05-27T07:31:38.827 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:31:38.833+0000 7f1b94312b80 1 Processor -- start 2024-05-27T07:31:38.827 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:31:38.833+0000 7f1b94312b80 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x558aaffd9e40 0x558ab0161460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:31:38.827 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:31:38.834+0000 7f1b94312b80 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x558ab0157f80 0x558ab01619a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:31:38.827 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:31:38.834+0000 7f1b94312b80 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x558ab015dc50 0x558ab0161ee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:31:38.827 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:31:38.834+0000 7f1b7effd700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x558aaffd9e40 0x558ab0161460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:31:38.828 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:31:38.834+0000 7f1b7effd700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x558aaffd9e40 0x558ab0161460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.173:3300/0 says I am v2:172.21.15.52:39672/0 (socket says 172.21.15.52:39672) 2024-05-27T07:31:38.828 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:31:38.834+0000 7f1b7effd700 1 -- 172.21.15.52:0/4085848961 learned_addr learned my addr 172.21.15.52:0/4085848961 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-05-27T07:31:38.828 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:31:38.834+0000 7f1b7e7fc700 1 --2- 172.21.15.52:0/4085848961 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x558ab0157f80 0x558ab01619a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:31:38.828 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:31:38.835+0000 7f1b94312b80 1 -- 172.21.15.52:0/4085848961 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x558aaff7cfa0 con 0x558ab015dc50 2024-05-27T07:31:38.828 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:31:38.835+0000 7f1b94312b80 1 -- 172.21.15.52:0/4085848961 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x558ab00187c0 con 0x558ab0157f80 2024-05-27T07:31:38.828 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:31:38.835+0000 7f1b94312b80 1 -- 172.21.15.52:0/4085848961 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x558ab0164560 con 0x558aaffd9e40 2024-05-27T07:31:38.828 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:31:38.835+0000 7f1b94312b80 10 client.0 _ll_register_callbacks cb 0x558ab0011920 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T07:31:38.831 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:31:38.838+0000 7f1b7f7fe700 1 --2- 172.21.15.52:0/4085848961 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x558ab015dc50 0x558ab0161ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:31:38.832 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:31:38.839+0000 7f1b7f7fe700 1 -- 172.21.15.52:0/4085848961 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x558aaffd9e40 msgr2=0x558ab0161460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:31:38.832 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:31:38.839+0000 7f1b7f7fe700 1 --2- 172.21.15.52:0/4085848961 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x558aaffd9e40 0x558ab0161460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:31:38.832 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:31:38.839+0000 7f1b7f7fe700 1 -- 172.21.15.52:0/4085848961 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x558ab0157f80 msgr2=0x558ab01619a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:31:38.832 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:31:38.839+0000 7f1b7f7fe700 1 --2- 172.21.15.52:0/4085848961 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x558ab0157f80 0x558ab01619a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:31:38.833 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:31:38.839+0000 7f1b7f7fe700 1 -- 172.21.15.52:0/4085848961 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({osdmap=0}) -- 0x558ab0015690 con 0x558ab015dc50 2024-05-27T07:31:38.833 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:31:38.841+0000 7f1b7f7fe700 1 --2- 172.21.15.52:0/4085848961 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x558ab015dc50 0x558ab0161ee0 secure :-1 s=READY pgs=659 cs=0 l=1 rev1=1 crypto rx=0x7f1b74009620 tx=0x7f1b740058e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=393fe35bbdd08a9e server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:31:38.833 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:31:38.841+0000 7f1b5effd700 10 client.0 ms_handle_connect on v2:172.21.15.52:3300/0 2024-05-27T07:31:38.834 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:31:38.841+0000 7f1b5effd700 1 -- 172.21.15.52:0/4085848961 <== mon.0 v2:172.21.15.52:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1b74021020 con 0x558ab015dc50 2024-05-27T07:31:38.834 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:31:38.842+0000 7f1b94312b80 1 -- 172.21.15.52:0/4085848961 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f1b780aa220 con 0x558ab015dc50 2024-05-27T07:31:38.834 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:31:38.842+0000 7f1b5effd700 1 -- 172.21.15.52:0/4085848961 <== mon.0 v2:172.21.15.52:3300/0 2 ==== osd_map(116..116 src has 1..116) ==== 9379+0+0 (secure 0 0 0) 0x7f1b74010ac0 con 0x558ab015dc50 2024-05-27T07:31:38.835 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:31:38.843+0000 7f1b5effd700 1 -- 172.21.15.52:0/4085848961 <== mon.0 v2:172.21.15.52:3300/0 3 ==== mon_get_version_reply(handle=1 version=101) ==== 24+0+0 (secure 0 0 0) 0x7f1b7402c050 con 0x558ab015dc50 2024-05-27T07:31:38.835 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:31:38.843+0000 7f1b94312b80 10 client.16641 fetch_fsmap learned FSMap version 101 2024-05-27T07:31:38.835 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:31:38.843+0000 7f1b94312b80 1 -- 172.21.15.52:0/4085848961 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({config=0+,fsmap=101,monmap=4+}) -- 0x558ab0177950 con 0x558ab015dc50 2024-05-27T07:31:38.837 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:31:38.844+0000 7f1b5effd700 1 -- 172.21.15.52:0/4085848961 <== mon.0 v2:172.21.15.52:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f1b780aa220 con 0x558ab015dc50 2024-05-27T07:31:38.839 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:31:38.846+0000 7f1b5effd700 1 -- 172.21.15.52:0/4085848961 <== mon.0 v2:172.21.15.52:3300/0 5 ==== fsmap(e 101) ==== 4203+0+0 (secure 0 0 0) 0x7f1b740274c0 con 0x558ab015dc50 2024-05-27T07:31:38.839 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:31:38.846+0000 7f1b94312b80 10 client.16641 fetch_fsmap finished waiting for FSMap version 101 2024-05-27T07:31:38.839 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:31:38.846+0000 7f1b94312b80 10 client.16641 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.c 2024-05-27T07:31:38.839 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:31:38.846+0000 7f1b94312b80 20 client.16641 populate_metadata read hostname 'smithi052' 2024-05-27T07:31:38.839 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:31:38.846+0000 7f1b94312b80 1 --2- 172.21.15.52:0/4085848961 >> [v2:172.21.15.173:6836/1396363827,v1:172.21.15.173:6837/1396363827] conn(0x558ab017a2a0 0x558ab017c770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:31:38.839 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:31:38.846+0000 7f1b94312b80 4 client.16641 mds_command: new command op to 24466 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-27T07:31:38.839 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:31:38.846+0000 7f1b94312b80 1 -- 172.21.15.52:0/4085848961 --> [v2:172.21.15.173:6836/1396363827,v1:172.21.15.173:6837/1396363827] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x558ab01239f0 con 0x558ab017a2a0 2024-05-27T07:31:38.839 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:31:38.847+0000 7f1b7effd700 1 --2- 172.21.15.52:0/4085848961 >> [v2:172.21.15.173:6836/1396363827,v1:172.21.15.173:6837/1396363827] conn(0x558ab017a2a0 0x558ab017c770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:31:38.843 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:31:38.850+0000 7f1b7effd700 1 --2- 172.21.15.52:0/4085848961 >> [v2:172.21.15.173:6836/1396363827,v1:172.21.15.173:6837/1396363827] conn(0x558ab017a2a0 0x558ab017c770 crc :-1 s=READY pgs=40 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=1f24bb3d4c6ef9e0 server_cookie=c4bbdf5101985ab4 in_seq=0 out_seq=0 2024-05-27T07:31:38.843 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:31:38.851+0000 7f1b5effd700 10 client.16641 ms_handle_connect on v2:172.21.15.173:6836/1396363827 2024-05-27T07:31:38.847 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:31:38.855+0000 7f1b5effd700 1 -- 172.21.15.52:0/4085848961 <== mds.4 v2:172.21.15.173:6836/1396363827 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24041 (crc 0 0 0) 0x558ab01239f0 con 0x558ab017a2a0 2024-05-27T07:31:38.847 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:31:38.855+0000 7f1b5effd700 10 client.16641 handle_command_reply: tid=0 2024-05-27T07:31:38.847 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:31:38.855+0000 7f1b94312b80 1 client.16641 shutdown 2024-05-27T07:31:38.853 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:31:38.859+0000 7f1b94312b80 1 -- 172.21.15.52:0/4085848961 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x558ab015dc50 msgr2=0x558ab0161ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:31:38.853 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:31:38.859+0000 7f1b94312b80 1 --2- 172.21.15.52:0/4085848961 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x558ab015dc50 0x558ab0161ee0 secure :-1 s=READY pgs=659 cs=0 l=1 rev1=1 crypto rx=0x7f1b74009620 tx=0x7f1b740058e0 comp rx=0 tx=0).stop 2024-05-27T07:31:38.853 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:31:38.860+0000 7f1b94312b80 1 -- 172.21.15.52:0/4085848961 shutdown_connections 2024-05-27T07:31:38.853 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:31:38.860+0000 7f1b94312b80 1 --2- 172.21.15.52:0/4085848961 >> [v2:172.21.15.173:6836/1396363827,v1:172.21.15.173:6837/1396363827] conn(0x558ab017a2a0 0x558ab017c770 crc :-1 s=READY pgs=40 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:31:38.853 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:31:38.860+0000 7f1b94312b80 1 --2- 172.21.15.52:0/4085848961 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x558ab015dc50 0x558ab0161ee0 unknown :-1 s=CLOSED pgs=659 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:31:38.854 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:31:38.860+0000 7f1b94312b80 1 --2- 172.21.15.52:0/4085848961 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x558aaffd9e40 0x558ab0161460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:31:38.854 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:31:38.860+0000 7f1b94312b80 1 --2- 172.21.15.52:0/4085848961 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x558ab0157f80 0x558ab01619a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:31:38.854 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:31:38.860+0000 7f1b94312b80 1 -- 172.21.15.52:0/4085848961 >> 172.21.15.52:0/4085848961 conn(0x558ab0034af0 msgr2=0x558ab0156a30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:31:38.854 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:31:38.860+0000 7f1b5effd700 0 client.16641 ms_handle_reset on v2:172.21.15.173:6836/1396363827 2024-05-27T07:31:38.854 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:31:38.862+0000 7f1b94312b80 1 -- 172.21.15.52:0/4085848961 shutdown_connections 2024-05-27T07:31:38.857 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:31:38.865+0000 7f1b94312b80 1 -- 172.21.15.52:0/4085848961 wait complete. 2024-05-27T07:31:38.862 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:31:38.868+0000 7f1b94312b80 20 client.16641 trim_cache size 0 max 16384 2024-05-27T07:31:38.901 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:31:38.908+0000 7f1b94312b80 1 Processor -- start 2024-05-27T07:31:38.901 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:31:38.909+0000 7f1b94312b80 1 -- start start 2024-05-27T07:31:38.902 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:31:38.910+0000 7f1b94312b80 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x558aaffd9e40 0x558ab0157a40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:31:38.902 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:31:38.910+0000 7f1b94312b80 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x558ab0157f80 0x558ab0160780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:31:38.902 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:31:38.910+0000 7f1b94312b80 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x558ab015dc50 0x558ab0161d20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:31:38.903 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:31:38.910+0000 7f1b94312b80 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x7f1b800806a0 con 0x558ab0157f80 2024-05-27T07:31:38.903 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:31:38.910+0000 7f1b94312b80 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f1b8007eaa0 con 0x558ab015dc50 2024-05-27T07:31:38.903 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:31:38.910+0000 7f1b94312b80 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f1b8007ffd0 con 0x558aaffd9e40 2024-05-27T07:31:38.903 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:31:38.910+0000 7f1b7e7fc700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x558ab0157f80 0x558ab0160780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:31:38.903 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:31:38.910+0000 7f1b7e7fc700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x558ab0157f80 0x558ab0160780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.52:50790/0 (socket says 172.21.15.52:50790) 2024-05-27T07:31:38.903 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:31:38.910+0000 7f1b7e7fc700 1 -- 172.21.15.52:0/3279125058 learned_addr learned my addr 172.21.15.52:0/3279125058 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-05-27T07:31:38.903 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:31:38.910+0000 7f1b7f7fe700 1 --2- 172.21.15.52:0/3279125058 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x558ab015dc50 0x558ab0161d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:31:38.904 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:31:38.911+0000 7f1b7e7fc700 1 -- 172.21.15.52:0/3279125058 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x558aaffd9e40 msgr2=0x558ab0157a40 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-05-27T07:31:38.904 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:31:38.911+0000 7f1b7e7fc700 1 --2- 172.21.15.52:0/3279125058 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x558aaffd9e40 0x558ab0157a40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:31:38.904 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:31:38.911+0000 7f1b7e7fc700 1 -- 172.21.15.52:0/3279125058 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x558ab015dc50 msgr2=0x558ab0161d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:31:38.904 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:31:38.911+0000 7f1b7e7fc700 1 --2- 172.21.15.52:0/3279125058 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x558ab015dc50 0x558ab0161d20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:31:38.904 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:31:38.911+0000 7f1b7e7fc700 1 -- 172.21.15.52:0/3279125058 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x558ab015f9d0 con 0x558ab0157f80 2024-05-27T07:31:38.905 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:31:38.911+0000 7f1b7e7fc700 1 --2- 172.21.15.52:0/3279125058 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x558ab0157f80 0x558ab0160780 secure :-1 s=READY pgs=660 cs=0 l=1 rev1=1 crypto rx=0x7f1b780e99b0 tx=0x7f1b7801cd20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3359601a94040f21 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:31:38.905 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:31:38.912+0000 7f1b657fa700 1 -- 172.21.15.52:0/3279125058 <== mon.0 v2:172.21.15.52:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1b780aa060 con 0x558ab0157f80 2024-05-27T07:31:38.905 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:31:38.912+0000 7f1b657fa700 1 -- 172.21.15.52:0/3279125058 <== mon.0 v2:172.21.15.52:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f1b780aa220 con 0x558ab0157f80 2024-05-27T07:31:38.905 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:31:38.912+0000 7f1b657fa700 1 -- 172.21.15.52:0/3279125058 <== mon.0 v2:172.21.15.52:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1b780aa4e0 con 0x558ab0157f80 2024-05-27T07:31:38.906 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:31:38.913+0000 7f1b94312b80 1 -- 172.21.15.52:0/3279125058 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x558ab0157f80 msgr2=0x558ab0160780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:31:38.906 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:31:38.913+0000 7f1b94312b80 1 --2- 172.21.15.52:0/3279125058 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x558ab0157f80 0x558ab0160780 secure :-1 s=READY pgs=660 cs=0 l=1 rev1=1 crypto rx=0x7f1b780e99b0 tx=0x7f1b7801cd20 comp rx=0 tx=0).stop 2024-05-27T07:31:38.906 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:31:38.914+0000 7f1b94312b80 1 -- 172.21.15.52:0/3279125058 shutdown_connections 2024-05-27T07:31:38.907 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:31:38.914+0000 7f1b94312b80 1 --2- 172.21.15.52:0/3279125058 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x558ab015dc50 0x558ab0161d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:31:38.907 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:31:38.914+0000 7f1b94312b80 1 --2- 172.21.15.52:0/3279125058 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x558aaffd9e40 0x558ab0157a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:31:38.907 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:31:38.914+0000 7f1b94312b80 1 --2- 172.21.15.52:0/3279125058 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x558ab0157f80 0x558ab0160780 unknown :-1 s=CLOSED pgs=660 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:31:38.907 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:31:38.914+0000 7f1b94312b80 1 -- 172.21.15.52:0/3279125058 >> 172.21.15.52:0/3279125058 conn(0x558ab0034af0 msgr2=0x558ab015c440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:31:38.907 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:31:38.914+0000 7f1b94312b80 1 -- 172.21.15.52:0/3279125058 shutdown_connections 2024-05-27T07:31:38.907 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:31:38.914+0000 7f1b94312b80 1 -- 172.21.15.52:0/3279125058 wait complete. 2024-05-27T07:31:38.908 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:31:38.915+0000 7f1b94312b80 1 -- start start 2024-05-27T07:31:38.908 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:31:38.916+0000 7f1b94312b80 1 Processor -- start 2024-05-27T07:31:38.909 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:31:38.916+0000 7f1b94312b80 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x558aaffd9e40 0x558ab0172730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:31:38.910 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:31:38.917+0000 7f1b7effd700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x558aaffd9e40 0x558ab0172730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:31:38.910 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:31:38.917+0000 7f1b7effd700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x558aaffd9e40 0x558ab0172730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.52:50800/0 (socket says 172.21.15.52:50800) 2024-05-27T07:31:38.910 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:31:38.917+0000 7f1b7effd700 1 -- 172.21.15.52:0/980726534 learned_addr learned my addr 172.21.15.52:0/980726534 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-05-27T07:31:38.910 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:31:38.917+0000 7f1b94312b80 1 --2- 172.21.15.52:0/980726534 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x558ab0157f80 0x558ab01816d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:31:38.910 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:31:38.917+0000 7f1b94312b80 1 --2- 172.21.15.52:0/980726534 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x558ab015dc50 0x558ab0181c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:31:38.910 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:31:38.917+0000 7f1b7e7fc700 1 --2- 172.21.15.52:0/980726534 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x558ab0157f80 0x558ab01816d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:31:38.911 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:31:38.918+0000 7f1b94312b80 1 -- 172.21.15.52:0/980726534 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x7f1b8007f840 con 0x558aaffd9e40 2024-05-27T07:31:38.911 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:31:38.918+0000 7f1b7f7fe700 1 --2- 172.21.15.52:0/980726534 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x558ab015dc50 0x558ab0181c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:31:38.911 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:31:38.918+0000 7f1b94312b80 1 -- 172.21.15.52:0/980726534 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f1b8007f170 con 0x558ab0157f80 2024-05-27T07:31:38.911 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:31:38.918+0000 7f1b94312b80 1 -- 172.21.15.52:0/980726534 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x558ab0154540 con 0x558ab015dc50 2024-05-27T07:31:38.911 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:31:38.919+0000 7f1b7e7fc700 1 -- 172.21.15.52:0/980726534 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x558ab015dc50 msgr2=0x558ab0181c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:31:38.912 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:31:38.919+0000 7f1b7e7fc700 1 --2- 172.21.15.52:0/980726534 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x558ab015dc50 0x558ab0181c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:31:38.912 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:31:38.919+0000 7f1b7e7fc700 1 -- 172.21.15.52:0/980726534 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x558aaffd9e40 msgr2=0x558ab0172730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:31:38.912 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:31:38.919+0000 7f1b7e7fc700 1 --2- 172.21.15.52:0/980726534 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x558aaffd9e40 0x558ab0172730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:31:38.913 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:31:38.919+0000 7f1b7e7fc700 1 -- 172.21.15.52:0/980726534 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({osdmap=0}) -- 0x558ab01822b0 con 0x558ab0157f80 2024-05-27T07:31:38.913 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:31:38.919+0000 7f1b94312b80 10 client.0 _ll_register_callbacks cb 0x558ab0011920 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T07:31:38.913 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:31:38.919+0000 7f1b94312b80 1 -- 172.21.15.52:0/980726534 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f1b780aa220 con 0x558ab0157f80 2024-05-27T07:31:38.914 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:31:38.919+0000 7f1b7effd700 1 --2- 172.21.15.52:0/980726534 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x558aaffd9e40 0x558ab0172730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-27T07:31:38.914 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:31:38.920+0000 7f1b7e7fc700 1 --2- 172.21.15.52:0/980726534 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x558ab0157f80 0x558ab01816d0 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7f1b780aa410 tx=0x7f1b780e3610 comp rx=0 tx=0).ready entity=mon.2 client_cookie=78dcd76bf215127 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:31:38.914 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:31:38.920+0000 7f1b5effd700 10 client.26282 ms_handle_connect on v2:172.21.15.105:3300/0 2024-05-27T07:31:38.914 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:31:38.920+0000 7f1b5effd700 1 -- 172.21.15.52:0/980726534 <== mon.2 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1b780aa4e0 con 0x558ab0157f80 2024-05-27T07:31:38.915 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:31:38.920+0000 7f1b5effd700 1 -- 172.21.15.52:0/980726534 <== mon.2 v2:172.21.15.105:3300/0 2 ==== osd_map(116..116 src has 1..116) ==== 9379+0+0 (secure 0 0 0) 0x7f1b780aa970 con 0x558ab0157f80 2024-05-27T07:31:38.915 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:31:38.920+0000 7f1b5effd700 1 -- 172.21.15.52:0/980726534 <== mon.2 v2:172.21.15.105:3300/0 3 ==== mon_get_version_reply(handle=1 version=101) ==== 24+0+0 (secure 0 0 0) 0x7f1b780e0f80 con 0x558ab0157f80 2024-05-27T07:31:38.915 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:31:38.920+0000 7f1b94312b80 10 client.26282 fetch_fsmap learned FSMap version 101 2024-05-27T07:31:38.915 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:31:38.920+0000 7f1b94312b80 1 -- 172.21.15.52:0/980726534 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({fsmap=101}) -- 0x558ab01822b0 con 0x558ab0157f80 2024-05-27T07:31:38.915 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:31:38.921+0000 7f1b5effd700 1 -- 172.21.15.52:0/980726534 <== mon.2 v2:172.21.15.105:3300/0 4 ==== fsmap(e 101) ==== 4203+0+0 (secure 0 0 0) 0x7f1b78011070 con 0x558ab0157f80 2024-05-27T07:31:38.916 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:31:38.921+0000 7f1b94312b80 10 client.26282 fetch_fsmap finished waiting for FSMap version 101 2024-05-27T07:31:38.916 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:31:38.921+0000 7f1b94312b80 10 client.26282 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.c 2024-05-27T07:31:38.916 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:31:38.921+0000 7f1b94312b80 20 client.26282 populate_metadata read hostname 'smithi052' 2024-05-27T07:31:38.916 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:31:38.921+0000 7f1b94312b80 1 --2- 172.21.15.52:0/980726534 >> [v2:172.21.15.173:6836/1396363827,v1:172.21.15.173:6837/1396363827] conn(0x558ab01e49a0 0x558ab01e6dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:31:38.916 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:31:38.921+0000 7f1b94312b80 4 client.26282 mds_command: new command op to 24466 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-05-27T07:31:38.916 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:31:38.921+0000 7f1b94312b80 1 -- 172.21.15.52:0/980726534 --> [v2:172.21.15.173:6836/1396363827,v1:172.21.15.173:6837/1396363827] -- command(tid 0: {"prefix": "damage ls"}) -- 0x558ab0088be0 con 0x558ab01e49a0 2024-05-27T07:31:38.917 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:31:38.922+0000 7f1b7effd700 1 --2- 172.21.15.52:0/980726534 >> [v2:172.21.15.173:6836/1396363827,v1:172.21.15.173:6837/1396363827] conn(0x558ab01e49a0 0x558ab01e6dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:31:38.917 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:31:38.925+0000 7f1b7effd700 1 --2- 172.21.15.52:0/980726534 >> [v2:172.21.15.173:6836/1396363827,v1:172.21.15.173:6837/1396363827] conn(0x558ab01e49a0 0x558ab01e6dc0 crc :-1 s=READY pgs=41 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=ce6ac852cf63c537 server_cookie=f86f4da344eaecbb in_seq=0 out_seq=0 2024-05-27T07:31:38.917 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:31:38.925+0000 7f1b5effd700 10 client.26282 ms_handle_connect on v2:172.21.15.173:6836/1396363827 2024-05-27T07:31:38.918 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:31:38.925+0000 7f1b5effd700 1 -- 172.21.15.52:0/980726534 <== mds.4 v2:172.21.15.173:6836/1396363827 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x558ab0088be0 con 0x558ab01e49a0 2024-05-27T07:31:38.918 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:31:38.925+0000 7f1b5effd700 10 client.26282 handle_command_reply: tid=0 2024-05-27T07:31:38.918 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:31:38.925+0000 7f1b94312b80 1 client.26282 shutdown 2024-05-27T07:31:38.919 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:31:38.926+0000 7f1b94312b80 1 -- 172.21.15.52:0/980726534 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x558ab0157f80 msgr2=0x558ab01816d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:31:38.919 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:31:38.926+0000 7f1b94312b80 1 --2- 172.21.15.52:0/980726534 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x558ab0157f80 0x558ab01816d0 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7f1b780aa410 tx=0x7f1b780e3610 comp rx=0 tx=0).stop 2024-05-27T07:31:38.919 INFO:teuthology.orchestra.run.smithi052.stdout:[] 2024-05-27T07:31:38.920 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:31:38.926+0000 7f1b94312b80 1 -- 172.21.15.52:0/980726534 shutdown_connections 2024-05-27T07:31:38.920 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:31:38.926+0000 7f1b94312b80 1 --2- 172.21.15.52:0/980726534 >> [v2:172.21.15.173:6836/1396363827,v1:172.21.15.173:6837/1396363827] conn(0x558ab01e49a0 0x558ab01e6dc0 crc :-1 s=READY pgs=41 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:31:38.920 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:31:38.926+0000 7f1b5effd700 0 client.26282 ms_handle_reset on v2:172.21.15.173:6836/1396363827 2024-05-27T07:31:38.920 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:31:38.926+0000 7f1b94312b80 1 --2- 172.21.15.52:0/980726534 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x558ab015dc50 0x558ab0181c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:31:38.920 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:31:38.926+0000 7f1b94312b80 1 --2- 172.21.15.52:0/980726534 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x558aaffd9e40 0x558ab0172730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:31:38.921 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:31:38.926+0000 7f1b94312b80 1 --2- 172.21.15.52:0/980726534 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x558ab0157f80 0x558ab01816d0 unknown :-1 s=CLOSED pgs=180 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:31:38.921 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:31:38.926+0000 7f1b94312b80 1 -- 172.21.15.52:0/980726534 >> 172.21.15.52:0/980726534 conn(0x558ab0034af0 msgr2=0x558ab015c440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:31:38.921 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:31:38.926+0000 7f1b94312b80 1 -- 172.21.15.52:0/980726534 shutdown_connections 2024-05-27T07:31:38.921 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:31:38.926+0000 7f1b94312b80 1 -- 172.21.15.52:0/980726534 wait complete. 2024-05-27T07:31:38.921 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:31:38.926+0000 7f1b94312b80 20 client.26282 trim_cache size 0 max 16384 2024-05-27T07:31:38.922 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:31:38.927+0000 7f1b858fd700 1 -- 172.21.15.52:0/2694532192 >> [v2:172.21.15.52:6800/1822318384,v1:172.21.15.52:6801/1822318384] conn(0x7f1b680b2bc0 msgr2=0x7f1b680b50d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:31:38.922 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:31:38.927+0000 7f1b858fd700 1 --2- 172.21.15.52:0/2694532192 >> [v2:172.21.15.52:6800/1822318384,v1:172.21.15.52:6801/1822318384] conn(0x7f1b680b2bc0 0x7f1b680b50d0 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7f1b74005b40 tx=0x7f1b74009db0 comp rx=0 tx=0).stop 2024-05-27T07:31:38.922 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:31:38.927+0000 7f1b858fd700 1 -- 172.21.15.52:0/2694532192 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f1b801105e0 msgr2=0x7f1b801aba80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:31:38.922 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:31:38.927+0000 7f1b858fd700 1 --2- 172.21.15.52:0/2694532192 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f1b801105e0 0x7f1b801aba80 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7f1b7800b6c0 tx=0x7f1b78007760 comp rx=0 tx=0).stop 2024-05-27T07:31:38.922 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:31:38.927+0000 7f1b858fd700 1 -- 172.21.15.52:0/2694532192 shutdown_connections 2024-05-27T07:31:38.923 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:31:38.927+0000 7f1b858fd700 1 --2- 172.21.15.52:0/2694532192 >> [v2:172.21.15.52:6800/1822318384,v1:172.21.15.52:6801/1822318384] conn(0x7f1b680b2bc0 0x7f1b680b50d0 unknown :-1 s=CLOSED pgs=148 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:31:38.923 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:31:38.927+0000 7f1b858fd700 1 --2- 172.21.15.52:0/2694532192 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f1b80110f30 0x7f1b801a56b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:31:38.923 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:31:38.927+0000 7f1b858fd700 1 --2- 172.21.15.52:0/2694532192 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f1b8010a970 0x7f1b801ab540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:31:38.923 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:31:38.927+0000 7f1b858fd700 1 --2- 172.21.15.52:0/2694532192 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f1b801105e0 0x7f1b801aba80 unknown :-1 s=CLOSED pgs=146 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:31:38.923 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:31:38.927+0000 7f1b858fd700 1 -- 172.21.15.52:0/2694532192 >> 172.21.15.52:0/2694532192 conn(0x7f1b8006f4c0 msgr2=0x7f1b8010d2b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:31:38.923 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:31:38.927+0000 7f1b858fd700 1 -- 172.21.15.52:0/2694532192 shutdown_connections 2024-05-27T07:31:38.924 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:31:38.928+0000 7f1b858fd700 1 -- 172.21.15.52:0/2694532192 wait complete. 2024-05-27T07:31:39.654 DEBUG:teuthology.orchestra.run.smithi052:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:38f9d900c83c63799fbdbe61acc9a11b0d3554a6 shell --fsid 73df2636-1bf8-11ef-bc9b-c7b262605968 -- ceph tell mds.1:3 damage ls 2024-05-27T07:31:39.906 INFO:teuthology.orchestra.run.smithi052.stderr:Inferring config /var/lib/ceph/73df2636-1bf8-11ef-bc9b-c7b262605968/mon.a/config 2024-05-27T07:31:40.281 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:31:40 smithi052 ceph-mon[31955]: pgmap v199: 129 pgs: 129 active+clean; 505 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 7.5 MiB/s wr, 1.23k op/s 2024-05-27T07:31:40.604 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:31:40 smithi173 ceph-mon[37000]: pgmap v199: 129 pgs: 129 active+clean; 505 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 7.5 MiB/s wr, 1.23k op/s 2024-05-27T07:31:40.692 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:31:40 smithi105 ceph-mon[35977]: pgmap v199: 129 pgs: 129 active+clean; 505 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 7.5 MiB/s wr, 1.23k op/s 2024-05-27T07:31:42.446 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:31:42 smithi052 ceph-mon[31955]: pgmap v200: 129 pgs: 129 active+clean; 492 MiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 371 KiB/s rd, 5.9 MiB/s wr, 1.10k op/s 2024-05-27T07:31:42.604 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:31:42 smithi173 ceph-mon[37000]: pgmap v200: 129 pgs: 129 active+clean; 492 MiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 371 KiB/s rd, 5.9 MiB/s wr, 1.10k op/s 2024-05-27T07:31:42.692 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:31:42 smithi105 ceph-mon[35977]: pgmap v200: 129 pgs: 129 active+clean; 492 MiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 371 KiB/s rd, 5.9 MiB/s wr, 1.10k op/s 2024-05-27T07:31:42.892 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:31:42.899+0000 7f5136250700 1 -- 172.21.15.52:0/1358853120 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f5130107de0 msgr2=0x7f51301081f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:31:42.892 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:31:42.899+0000 7f5136250700 1 --2- 172.21.15.52:0/1358853120 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f5130107de0 0x7f51301081f0 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7f5118009fc0 tx=0x7f511800b330 comp rx=0 tx=0).stop 2024-05-27T07:31:42.892 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:31:42.899+0000 7f5136250700 1 -- 172.21.15.52:0/1358853120 shutdown_connections 2024-05-27T07:31:42.892 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:31:42.899+0000 7f5136250700 1 --2- 172.21.15.52:0/1358853120 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f5130108730 0x7f5130110090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:31:42.893 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:31:42.899+0000 7f5136250700 1 --2- 172.21.15.52:0/1358853120 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f51301034b0 0x7f51301038c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:31:42.893 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:31:42.899+0000 7f5136250700 1 --2- 172.21.15.52:0/1358853120 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f5130107de0 0x7f51301081f0 unknown :-1 s=CLOSED pgs=147 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:31:42.893 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:31:42.899+0000 7f5136250700 1 -- 172.21.15.52:0/1358853120 >> 172.21.15.52:0/1358853120 conn(0x7f51300fedb0 msgr2=0x7f51301011d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:31:42.893 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:31:42.899+0000 7f5136250700 1 -- 172.21.15.52:0/1358853120 shutdown_connections 2024-05-27T07:31:42.893 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:31:42.899+0000 7f5136250700 1 -- 172.21.15.52:0/1358853120 wait complete. 2024-05-27T07:31:42.893 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:31:42.901+0000 7f5136250700 1 Processor -- start 2024-05-27T07:31:42.895 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:31:42.901+0000 7f5136250700 1 -- start start 2024-05-27T07:31:42.895 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:31:42.902+0000 7f5136250700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f51301034b0 0x7f51301a0a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:31:42.896 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:31:42.902+0000 7f5136250700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f5130107de0 0x7f51301a0f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:31:42.896 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:31:42.902+0000 7f512f7fe700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f51301034b0 0x7f51301a0a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:31:42.896 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:31:42.902+0000 7f512f7fe700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f51301034b0 0x7f51301a0a40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.52:34508/0 (socket says 172.21.15.52:34508) 2024-05-27T07:31:42.896 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:31:42.902+0000 7f512f7fe700 1 -- 172.21.15.52:0/2141627318 learned_addr learned my addr 172.21.15.52:0/2141627318 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-05-27T07:31:42.896 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:31:42.902+0000 7f5126ffd700 1 --2- 172.21.15.52:0/2141627318 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f5130107de0 0x7f51301a0f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:31:42.896 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:31:42.903+0000 7f5136250700 1 --2- 172.21.15.52:0/2141627318 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f5130108730 0x7f513019abf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:31:42.896 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:31:42.903+0000 7f5136250700 1 -- 172.21.15.52:0/2141627318 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x7f5130113340 con 0x7f5130107de0 2024-05-27T07:31:42.896 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:31:42.903+0000 7f5136250700 1 -- 172.21.15.52:0/2141627318 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f51301131c0 con 0x7f51301034b0 2024-05-27T07:31:42.896 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:31:42.903+0000 7f5136250700 1 -- 172.21.15.52:0/2141627318 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f51301134c0 con 0x7f5130108730 2024-05-27T07:31:42.896 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:31:42.903+0000 7f512ffff700 1 --2- 172.21.15.52:0/2141627318 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f5130108730 0x7f513019abf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:31:42.897 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:31:42.903+0000 7f512f7fe700 1 -- 172.21.15.52:0/2141627318 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f5130108730 msgr2=0x7f513019abf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:31:42.897 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:31:42.903+0000 7f512f7fe700 1 --2- 172.21.15.52:0/2141627318 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f5130108730 0x7f513019abf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:31:42.897 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:31:42.903+0000 7f512f7fe700 1 -- 172.21.15.52:0/2141627318 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f5130107de0 msgr2=0x7f51301a0f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:31:42.897 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:31:42.903+0000 7f512f7fe700 1 --2- 172.21.15.52:0/2141627318 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f5130107de0 0x7f51301a0f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:31:42.897 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:31:42.903+0000 7f512f7fe700 1 -- 172.21.15.52:0/2141627318 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f513019b360 con 0x7f51301034b0 2024-05-27T07:31:42.898 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:31:42.904+0000 7f512f7fe700 1 --2- 172.21.15.52:0/2141627318 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f51301034b0 0x7f51301a0a40 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7f512000e9a0 tx=0x7f512000ecb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:31:42.898 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:31:42.904+0000 7f512d7fa700 1 -- 172.21.15.52:0/2141627318 <== mon.2 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5120020070 con 0x7f51301034b0 2024-05-27T07:31:42.898 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:31:42.904+0000 7f512d7fa700 1 -- 172.21.15.52:0/2141627318 <== mon.2 v2:172.21.15.105:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f512000c7c0 con 0x7f51301034b0 2024-05-27T07:31:42.898 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:31:42.904+0000 7f512d7fa700 1 -- 172.21.15.52:0/2141627318 <== mon.2 v2:172.21.15.105:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f512001b770 con 0x7f51301034b0 2024-05-27T07:31:42.898 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:31:42.904+0000 7f5136250700 1 -- 172.21.15.52:0/2141627318 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f513019b650 con 0x7f51301034b0 2024-05-27T07:31:42.898 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:31:42.905+0000 7f5136250700 1 -- 172.21.15.52:0/2141627318 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5130113b00 con 0x7f51301034b0 2024-05-27T07:31:42.900 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:31:42.908+0000 7f512d7fa700 1 -- 172.21.15.52:0/2141627318 <== mon.2 v2:172.21.15.105:3300/0 4 ==== mgrmap(e 26) ==== 149783+0+0 (secure 0 0 0) 0x7f5120023070 con 0x7f51301034b0 2024-05-27T07:31:42.901 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:31:42.909+0000 7f512d7fa700 1 --2- 172.21.15.52:0/2141627318 >> [v2:172.21.15.52:6800/1822318384,v1:172.21.15.52:6801/1822318384] conn(0x7f511c0b2a70 0x7f511c0b4f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:31:42.901 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:31:42.909+0000 7f5126ffd700 1 --2- 172.21.15.52:0/2141627318 >> [v2:172.21.15.52:6800/1822318384,v1:172.21.15.52:6801/1822318384] conn(0x7f511c0b2a70 0x7f511c0b4f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:31:42.901 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:31:42.909+0000 7f512d7fa700 1 -- 172.21.15.52:0/2141627318 <== mon.2 v2:172.21.15.105:3300/0 5 ==== osd_map(116..116 src has 1..116) ==== 9379+0+0 (secure 0 0 0) 0x7f51200e6730 con 0x7f51301034b0 2024-05-27T07:31:42.902 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:31:42.910+0000 7f5126ffd700 1 --2- 172.21.15.52:0/2141627318 >> [v2:172.21.15.52:6800/1822318384,v1:172.21.15.52:6801/1822318384] conn(0x7f511c0b2a70 0x7f511c0b4f80 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7f5130105a00 tx=0x7f5118009380 comp rx=0 tx=0).ready entity=mgr.15333 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:31:42.905 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:31:42.913+0000 7f5144c65b80 1 Processor -- start 2024-05-27T07:31:42.905 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:31:42.913+0000 7f5144c65b80 1 -- start start 2024-05-27T07:31:42.906 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:31:42.913+0000 7f5144c65b80 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x558d7d4f6e40 0x558d7d67a710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:31:42.907 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:31:42.913+0000 7f5144c65b80 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x558d7d67ac50 0x558d7d674a40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:31:42.907 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:31:42.914+0000 7f5144c65b80 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x558d7d674f80 0x558d7d675390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:31:42.907 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:31:42.914+0000 7f5126ffd700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x558d7d67ac50 0x558d7d674a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:31:42.907 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:31:42.914+0000 7f512f7fe700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x558d7d4f6e40 0x558d7d67a710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:31:42.907 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:31:42.914+0000 7f5126ffd700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x558d7d67ac50 0x558d7d674a40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.52:58782/0 (socket says 172.21.15.52:58782) 2024-05-27T07:31:42.907 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:31:42.914+0000 7f512f7fe700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x558d7d4f6e40 0x558d7d67a710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.52:34524/0 (socket says 172.21.15.52:34524) 2024-05-27T07:31:42.907 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:31:42.914+0000 7f5126ffd700 1 -- 172.21.15.52:0/1158982879 learned_addr learned my addr 172.21.15.52:0/1158982879 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-05-27T07:31:42.907 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:31:42.914+0000 7f5144c65b80 1 -- 172.21.15.52:0/1158982879 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x558d7d5a3c90 con 0x558d7d67ac50 2024-05-27T07:31:42.907 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:31:42.914+0000 7f5144c65b80 1 -- 172.21.15.52:0/1158982879 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x558d7d593830 con 0x558d7d4f6e40 2024-05-27T07:31:42.907 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:31:42.914+0000 7f5144c65b80 1 -- 172.21.15.52:0/1158982879 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x558d7d597870 con 0x558d7d674f80 2024-05-27T07:31:42.908 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:31:42.915+0000 7f512ffff700 1 --2- 172.21.15.52:0/1158982879 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x558d7d674f80 0x558d7d675390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:31:42.908 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:31:42.915+0000 7f5126ffd700 1 -- 172.21.15.52:0/1158982879 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x558d7d674f80 msgr2=0x558d7d675390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:31:42.908 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:31:42.915+0000 7f5126ffd700 1 --2- 172.21.15.52:0/1158982879 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x558d7d674f80 0x558d7d675390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:31:42.908 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:31:42.915+0000 7f5126ffd700 1 -- 172.21.15.52:0/1158982879 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x558d7d4f6e40 msgr2=0x558d7d67a710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:31:42.908 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:31:42.915+0000 7f5126ffd700 1 --2- 172.21.15.52:0/1158982879 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x558d7d4f6e40 0x558d7d67a710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:31:42.909 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:31:42.915+0000 7f5126ffd700 1 -- 172.21.15.52:0/1158982879 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x558d7d4e38d0 con 0x558d7d67ac50 2024-05-27T07:31:42.909 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:31:42.916+0000 7f5126ffd700 1 --2- 172.21.15.52:0/1158982879 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x558d7d67ac50 0x558d7d674a40 secure :-1 s=READY pgs=661 cs=0 l=1 rev1=1 crypto rx=0x7f5130076aa0 tx=0x7f511801d000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cfb1aa8973a30dc7 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:31:42.909 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:31:42.917+0000 7f5125ffb700 1 -- 172.21.15.52:0/1158982879 <== mon.0 v2:172.21.15.52:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f511802d070 con 0x558d7d67ac50 2024-05-27T07:31:42.910 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:31:42.917+0000 7f5125ffb700 1 -- 172.21.15.52:0/1158982879 <== mon.0 v2:172.21.15.52:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f511800bd30 con 0x558d7d67ac50 2024-05-27T07:31:42.911 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:31:42.917+0000 7f5125ffb700 1 -- 172.21.15.52:0/1158982879 <== mon.0 v2:172.21.15.52:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5118029850 con 0x558d7d67ac50 2024-05-27T07:31:42.912 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:31:42.919+0000 7f5144c65b80 1 -- 172.21.15.52:0/1158982879 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x558d7d67ac50 msgr2=0x558d7d674a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:31:42.912 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:31:42.919+0000 7f5144c65b80 1 --2- 172.21.15.52:0/1158982879 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x558d7d67ac50 0x558d7d674a40 secure :-1 s=READY pgs=661 cs=0 l=1 rev1=1 crypto rx=0x7f5130076aa0 tx=0x7f511801d000 comp rx=0 tx=0).stop 2024-05-27T07:31:42.912 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:31:42.919+0000 7f5144c65b80 1 -- 172.21.15.52:0/1158982879 shutdown_connections 2024-05-27T07:31:42.912 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:31:42.920+0000 7f5144c65b80 1 --2- 172.21.15.52:0/1158982879 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x558d7d674f80 0x558d7d675390 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:31:42.912 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:31:42.920+0000 7f5144c65b80 1 --2- 172.21.15.52:0/1158982879 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x558d7d4f6e40 0x558d7d67a710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:31:42.913 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:31:42.920+0000 7f5144c65b80 1 --2- 172.21.15.52:0/1158982879 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x558d7d67ac50 0x558d7d674a40 unknown :-1 s=CLOSED pgs=661 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:31:42.913 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:31:42.920+0000 7f5144c65b80 1 -- 172.21.15.52:0/1158982879 >> 172.21.15.52:0/1158982879 conn(0x558d7d551af0 msgr2=0x558d7d66f640 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:31:42.913 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:31:42.920+0000 7f5144c65b80 1 -- 172.21.15.52:0/1158982879 shutdown_connections 2024-05-27T07:31:42.913 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:31:42.920+0000 7f5144c65b80 1 -- 172.21.15.52:0/1158982879 wait complete. 2024-05-27T07:31:42.913 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:31:42.921+0000 7f5144c65b80 1 -- start start 2024-05-27T07:31:42.913 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:31:42.921+0000 7f5144c65b80 1 Processor -- start 2024-05-27T07:31:42.915 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:31:42.921+0000 7f5144c65b80 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x558d7d4f6e40 0x558d7d67e2a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:31:42.915 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:31:42.921+0000 7f5144c65b80 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x558d7d674f80 0x558d7d67e7e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:31:42.915 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:31:42.921+0000 7f5144c65b80 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x558d7d67ac50 0x558d7d67ed20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:31:42.915 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:31:42.921+0000 7f5144c65b80 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x558d7d499fa0 con 0x558d7d67ac50 2024-05-27T07:31:42.915 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:31:42.921+0000 7f5144c65b80 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x558d7d5357c0 con 0x558d7d4f6e40 2024-05-27T07:31:42.915 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:31:42.921+0000 7f5144c65b80 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x558d7d681780 con 0x558d7d674f80 2024-05-27T07:31:42.915 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:31:42.921+0000 7f5144c65b80 10 client.0 _ll_register_callbacks cb 0x558d7d52e920 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T07:31:42.915 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:31:42.922+0000 7f5126ffd700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x558d7d674f80 0x558d7d67e7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:31:42.915 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:31:42.922+0000 7f5126ffd700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x558d7d674f80 0x558d7d67e7e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.173:3300/0 says I am v2:172.21.15.52:39174/0 (socket says 172.21.15.52:39174) 2024-05-27T07:31:42.915 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:31:42.922+0000 7f5126ffd700 1 -- 172.21.15.52:0/2419887021 learned_addr learned my addr 172.21.15.52:0/2419887021 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-05-27T07:31:42.915 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:31:42.922+0000 7f5126ffd700 1 -- 172.21.15.52:0/2419887021 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x558d7d4f6e40 msgr2=0x558d7d67e2a0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-05-27T07:31:42.915 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:31:42.922+0000 7f5126ffd700 1 --2- 172.21.15.52:0/2419887021 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x558d7d4f6e40 0x558d7d67e2a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:31:42.916 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:31:42.922+0000 7f5126ffd700 1 -- 172.21.15.52:0/2419887021 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x558d7d67ac50 msgr2=0x558d7d67ed20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:31:42.916 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:31:42.922+0000 7f5126ffd700 1 --2- 172.21.15.52:0/2419887021 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x558d7d67ac50 0x558d7d67ed20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:31:42.916 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:31:42.922+0000 7f5126ffd700 1 -- 172.21.15.52:0/2419887021 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({osdmap=0}) -- 0x558d7d532690 con 0x558d7d674f80 2024-05-27T07:31:42.917 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:31:42.922+0000 7f5126ffd700 1 --2- 172.21.15.52:0/2419887021 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x558d7d674f80 0x558d7d67e7e0 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7f5118000f80 tx=0x7f511800b330 comp rx=0 tx=0).ready entity=mon.1 client_cookie=d0db94efbcb3441a server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:31:42.917 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:31:42.922+0000 7f510effd700 10 client.0 ms_handle_connect on v2:172.21.15.173:3300/0 2024-05-27T07:31:42.917 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:31:42.923+0000 7f510effd700 1 -- 172.21.15.52:0/2419887021 <== mon.1 v2:172.21.15.173:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f511803a070 con 0x558d7d674f80 2024-05-27T07:31:42.917 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:31:42.923+0000 7f510effd700 1 -- 172.21.15.52:0/2419887021 <== mon.1 v2:172.21.15.173:3300/0 2 ==== osd_map(116..116 src has 1..116) ==== 9379+0+0 (secure 0 0 0) 0x7f511802ad10 con 0x558d7d674f80 2024-05-27T07:31:42.917 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:31:42.923+0000 7f5144c65b80 1 -- 172.21.15.52:0/2419887021 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f511800bd30 con 0x558d7d674f80 2024-05-27T07:31:42.917 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:31:42.923+0000 7f510effd700 1 -- 172.21.15.52:0/2419887021 <== mon.1 v2:172.21.15.173:3300/0 3 ==== mon_get_version_reply(handle=1 version=101) ==== 24+0+0 (secure 0 0 0) 0x7f5118046050 con 0x558d7d674f80 2024-05-27T07:31:42.918 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:31:42.923+0000 7f5144c65b80 10 client.26176 fetch_fsmap learned FSMap version 101 2024-05-27T07:31:42.918 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:31:42.923+0000 7f5144c65b80 1 -- 172.21.15.52:0/2419887021 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({config=0+,fsmap=101,monmap=4+}) -- 0x558d7d694790 con 0x558d7d674f80 2024-05-27T07:31:42.918 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:31:42.924+0000 7f510effd700 1 -- 172.21.15.52:0/2419887021 <== mon.1 v2:172.21.15.173:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f5118007b10 con 0x558d7d674f80 2024-05-27T07:31:42.918 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:31:42.924+0000 7f510effd700 1 -- 172.21.15.52:0/2419887021 <== mon.1 v2:172.21.15.173:3300/0 5 ==== fsmap(e 101) ==== 4203+0+0 (secure 0 0 0) 0x7f5118011070 con 0x558d7d674f80 2024-05-27T07:31:42.918 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:31:42.924+0000 7f5144c65b80 10 client.26176 fetch_fsmap finished waiting for FSMap version 101 2024-05-27T07:31:42.918 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:31:42.924+0000 7f5144c65b80 10 client.26176 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.e 2024-05-27T07:31:42.918 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:31:42.924+0000 7f5144c65b80 20 client.26176 populate_metadata read hostname 'smithi052' 2024-05-27T07:31:42.919 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:31:42.924+0000 7f5144c65b80 1 --2- 172.21.15.52:0/2419887021 >> [v2:172.21.15.105:6834/3298498384,v1:172.21.15.105:6835/3298498384] conn(0x558d7d696d50 0x558d7d6991d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:31:42.919 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:31:42.924+0000 7f5144c65b80 4 client.26176 mds_command: new command op to 24473 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-27T07:31:42.919 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:31:42.924+0000 7f5144c65b80 1 -- 172.21.15.52:0/2419887021 --> [v2:172.21.15.105:6834/3298498384,v1:172.21.15.105:6835/3298498384] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x7f511800bd30 con 0x558d7d696d50 2024-05-27T07:31:42.919 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:31:42.925+0000 7f512f7fe700 1 --2- 172.21.15.52:0/2419887021 >> [v2:172.21.15.105:6834/3298498384,v1:172.21.15.105:6835/3298498384] conn(0x558d7d696d50 0x558d7d6991d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:31:42.921 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:31:42.926+0000 7f510effd700 10 client.26176 ms_handle_connect on v2:172.21.15.105:6834/3298498384 2024-05-27T07:31:42.921 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:31:42.926+0000 7f512f7fe700 1 --2- 172.21.15.52:0/2419887021 >> [v2:172.21.15.105:6834/3298498384,v1:172.21.15.105:6835/3298498384] conn(0x558d7d696d50 0x558d7d6991d0 crc :-1 s=READY pgs=41 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=f65f7546909ec8bb server_cookie=a7d0083af65b4b61 in_seq=0 out_seq=0 2024-05-27T07:31:42.921 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:31:42.927+0000 7f510effd700 1 -- 172.21.15.52:0/2419887021 <== mds.3 v2:172.21.15.105:6834/3298498384 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24041 (crc 0 0 0) 0x7f511800bd30 con 0x558d7d696d50 2024-05-27T07:31:42.921 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:31:42.927+0000 7f510effd700 10 client.26176 handle_command_reply: tid=0 2024-05-27T07:31:42.922 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:31:42.927+0000 7f5144c65b80 1 client.26176 shutdown 2024-05-27T07:31:42.922 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:31:42.927+0000 7f5144c65b80 1 -- 172.21.15.52:0/2419887021 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x558d7d674f80 msgr2=0x558d7d67e7e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:31:42.922 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:31:42.927+0000 7f5144c65b80 1 --2- 172.21.15.52:0/2419887021 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x558d7d674f80 0x558d7d67e7e0 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7f5118000f80 tx=0x7f511800b330 comp rx=0 tx=0).stop 2024-05-27T07:31:42.922 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:31:42.927+0000 7f5144c65b80 1 -- 172.21.15.52:0/2419887021 shutdown_connections 2024-05-27T07:31:42.922 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:31:42.927+0000 7f5144c65b80 1 --2- 172.21.15.52:0/2419887021 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x558d7d67ac50 0x558d7d67ed20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:31:42.922 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:31:42.927+0000 7f5144c65b80 1 --2- 172.21.15.52:0/2419887021 >> [v2:172.21.15.105:6834/3298498384,v1:172.21.15.105:6835/3298498384] conn(0x558d7d696d50 0x558d7d6991d0 crc :-1 s=READY pgs=41 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:31:42.923 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:31:42.927+0000 7f5144c65b80 1 --2- 172.21.15.52:0/2419887021 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x558d7d4f6e40 0x558d7d67e2a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:31:42.923 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:31:42.927+0000 7f510effd700 0 client.26176 ms_handle_reset on v2:172.21.15.105:6834/3298498384 2024-05-27T07:31:42.923 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:31:42.927+0000 7f5144c65b80 1 --2- 172.21.15.52:0/2419887021 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x558d7d674f80 0x558d7d67e7e0 unknown :-1 s=CLOSED pgs=148 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:31:42.923 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:31:42.927+0000 7f5144c65b80 1 -- 172.21.15.52:0/2419887021 >> 172.21.15.52:0/2419887021 conn(0x558d7d551af0 msgr2=0x558d7d673a30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:31:42.923 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:31:42.928+0000 7f5144c65b80 1 -- 172.21.15.52:0/2419887021 shutdown_connections 2024-05-27T07:31:42.923 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:31:42.928+0000 7f5144c65b80 1 -- 172.21.15.52:0/2419887021 wait complete. 2024-05-27T07:31:42.924 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:31:42.928+0000 7f5144c65b80 20 client.26176 trim_cache size 0 max 16384 2024-05-27T07:31:42.947 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:31:42.954+0000 7f5144c65b80 1 Processor -- start 2024-05-27T07:31:42.947 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:31:42.954+0000 7f5144c65b80 1 -- start start 2024-05-27T07:31:42.947 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:31:42.955+0000 7f5144c65b80 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x558d7d4f6e40 0x558d7d67eb60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:31:42.948 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:31:42.955+0000 7f5144c65b80 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x558d7d674f80 0x558d7d68a870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:31:42.948 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:31:42.956+0000 7f512f7fe700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x558d7d4f6e40 0x558d7d67eb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:31:42.949 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:31:42.956+0000 7f512f7fe700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x558d7d4f6e40 0x558d7d67eb60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.52:34542/0 (socket says 172.21.15.52:34542) 2024-05-27T07:31:42.949 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:31:42.956+0000 7f512f7fe700 1 -- 172.21.15.52:0/2337748515 learned_addr learned my addr 172.21.15.52:0/2337748515 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-05-27T07:31:42.949 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:31:42.956+0000 7f5144c65b80 1 --2- 172.21.15.52:0/2337748515 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x558d7d67ac50 0x558d7d678720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:31:42.949 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:31:42.956+0000 7f5144c65b80 1 -- 172.21.15.52:0/2337748515 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x7f51301ae6a0 con 0x558d7d674f80 2024-05-27T07:31:42.949 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:31:42.956+0000 7f5144c65b80 1 -- 172.21.15.52:0/2337748515 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f51301acaa0 con 0x558d7d4f6e40 2024-05-27T07:31:42.949 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:31:42.956+0000 7f5144c65b80 1 -- 172.21.15.52:0/2337748515 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f51301adfd0 con 0x558d7d67ac50 2024-05-27T07:31:42.950 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:31:42.956+0000 7f5126ffd700 1 --2- 172.21.15.52:0/2337748515 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x558d7d674f80 0x558d7d68a870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:31:42.950 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:31:42.957+0000 7f512f7fe700 1 -- 172.21.15.52:0/2337748515 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x558d7d67ac50 msgr2=0x558d7d678720 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-05-27T07:31:42.950 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:31:42.957+0000 7f512f7fe700 1 --2- 172.21.15.52:0/2337748515 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x558d7d67ac50 0x558d7d678720 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:31:42.950 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:31:42.957+0000 7f512f7fe700 1 -- 172.21.15.52:0/2337748515 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x558d7d674f80 msgr2=0x558d7d68a870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:31:42.950 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:31:42.957+0000 7f512f7fe700 1 --2- 172.21.15.52:0/2337748515 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x558d7d674f80 0x558d7d68a870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:31:42.950 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:31:42.957+0000 7f512f7fe700 1 -- 172.21.15.52:0/2337748515 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x558d7d66c090 con 0x558d7d4f6e40 2024-05-27T07:31:42.950 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:31:42.957+0000 7f512f7fe700 1 --2- 172.21.15.52:0/2337748515 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x558d7d4f6e40 0x558d7d67eb60 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7f51200e4120 tx=0x7f5120009820 comp rx=0 tx=0).ready entity=mon.2 client_cookie=3783e97e8c9df64f server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:31:42.951 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:31:42.957+0000 7f51257fa700 1 -- 172.21.15.52:0/2337748515 <== mon.2 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f51200b1b70 con 0x558d7d4f6e40 2024-05-27T07:31:42.951 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:31:42.958+0000 7f51257fa700 1 -- 172.21.15.52:0/2337748515 <== mon.2 v2:172.21.15.105:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f51200b6620 con 0x558d7d4f6e40 2024-05-27T07:31:42.951 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:31:42.958+0000 7f51257fa700 1 -- 172.21.15.52:0/2337748515 <== mon.2 v2:172.21.15.105:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f51200ada20 con 0x558d7d4f6e40 2024-05-27T07:31:42.952 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:31:42.959+0000 7f5144c65b80 1 -- 172.21.15.52:0/2337748515 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x558d7d4f6e40 msgr2=0x558d7d67eb60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:31:42.952 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:31:42.959+0000 7f5144c65b80 1 --2- 172.21.15.52:0/2337748515 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x558d7d4f6e40 0x558d7d67eb60 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7f51200e4120 tx=0x7f5120009820 comp rx=0 tx=0).stop 2024-05-27T07:31:42.953 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:31:42.960+0000 7f5144c65b80 1 -- 172.21.15.52:0/2337748515 shutdown_connections 2024-05-27T07:31:42.953 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:31:42.960+0000 7f5144c65b80 1 --2- 172.21.15.52:0/2337748515 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x558d7d67ac50 0x558d7d678720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:31:42.953 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:31:42.960+0000 7f5144c65b80 1 --2- 172.21.15.52:0/2337748515 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x558d7d4f6e40 0x558d7d67eb60 unknown :-1 s=CLOSED pgs=182 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:31:42.953 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:31:42.960+0000 7f5144c65b80 1 --2- 172.21.15.52:0/2337748515 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x558d7d674f80 0x558d7d68a870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:31:42.954 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:31:42.960+0000 7f5144c65b80 1 -- 172.21.15.52:0/2337748515 >> 172.21.15.52:0/2337748515 conn(0x558d7d551af0 msgr2=0x558d7d678010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:31:42.954 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:31:42.960+0000 7f5144c65b80 1 -- 172.21.15.52:0/2337748515 shutdown_connections 2024-05-27T07:31:42.954 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:31:42.960+0000 7f5144c65b80 1 -- 172.21.15.52:0/2337748515 wait complete. 2024-05-27T07:31:42.958 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:31:42.961+0000 7f5144c65b80 1 -- start start 2024-05-27T07:31:42.958 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:31:42.961+0000 7f5144c65b80 1 Processor -- start 2024-05-27T07:31:42.958 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:31:42.962+0000 7f5144c65b80 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x558d7d4f6e40 0x558d7d6f9c00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:31:42.958 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:31:42.962+0000 7f5144c65b80 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x558d7d674f80 0x558d7d6fa140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:31:42.958 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:31:42.962+0000 7f5144c65b80 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x558d7d67ac50 0x558d7d681fc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:31:42.958 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:31:42.962+0000 7f5144c65b80 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x7f51301ad840 con 0x558d7d674f80 2024-05-27T07:31:42.959 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:31:42.962+0000 7f5144c65b80 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f51301ad170 con 0x558d7d4f6e40 2024-05-27T07:31:42.959 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:31:42.962+0000 7f5144c65b80 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x558d7d667a00 con 0x558d7d67ac50 2024-05-27T07:31:42.959 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:31:42.962+0000 7f5144c65b80 10 client.0 _ll_register_callbacks cb 0x558d7d52e920 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T07:31:42.959 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:31:42.962+0000 7f512f7fe700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x558d7d4f6e40 0x558d7d6f9c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:31:42.959 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:31:42.962+0000 7f5126ffd700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x558d7d674f80 0x558d7d6fa140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:31:42.959 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:31:42.962+0000 7f5126ffd700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x558d7d674f80 0x558d7d6fa140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.52:58810/0 (socket says 172.21.15.52:58810) 2024-05-27T07:31:42.959 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:31:42.962+0000 7f5126ffd700 1 -- 172.21.15.52:0/1332984982 learned_addr learned my addr 172.21.15.52:0/1332984982 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-05-27T07:31:42.959 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:31:42.962+0000 7f512ffff700 1 --2- 172.21.15.52:0/1332984982 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x558d7d67ac50 0x558d7d681fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:31:42.959 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:31:42.962+0000 7f5126ffd700 1 -- 172.21.15.52:0/1332984982 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x558d7d67ac50 msgr2=0x558d7d681fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:31:42.959 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:31:42.962+0000 7f5126ffd700 1 --2- 172.21.15.52:0/1332984982 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x558d7d67ac50 0x558d7d681fc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:31:42.959 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:31:42.962+0000 7f5126ffd700 1 -- 172.21.15.52:0/1332984982 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x558d7d4f6e40 msgr2=0x558d7d6f9c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:31:42.959 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:31:42.962+0000 7f5126ffd700 1 --2- 172.21.15.52:0/1332984982 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x558d7d4f6e40 0x558d7d6f9c00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:31:42.959 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:31:42.962+0000 7f5126ffd700 1 -- 172.21.15.52:0/1332984982 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({osdmap=0}) -- 0x558d7d682600 con 0x558d7d674f80 2024-05-27T07:31:42.960 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:31:42.963+0000 7f5126ffd700 1 --2- 172.21.15.52:0/1332984982 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x558d7d674f80 0x558d7d6fa140 secure :-1 s=READY pgs=662 cs=0 l=1 rev1=1 crypto rx=0x7f51180406d0 tx=0x7f511802a910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5fde39426d8afda8 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:31:42.960 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:31:42.963+0000 7f510effd700 10 client.0 ms_handle_connect on v2:172.21.15.52:3300/0 2024-05-27T07:31:42.960 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:31:42.963+0000 7f510effd700 1 -- 172.21.15.52:0/1332984982 <== mon.0 v2:172.21.15.52:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5118020070 con 0x558d7d674f80 2024-05-27T07:31:42.960 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:31:42.963+0000 7f510effd700 1 -- 172.21.15.52:0/1332984982 <== mon.0 v2:172.21.15.52:3300/0 2 ==== osd_map(116..116 src has 1..116) ==== 9379+0+0 (secure 0 0 0) 0x7f511800bb30 con 0x558d7d674f80 2024-05-27T07:31:42.960 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:31:42.963+0000 7f5144c65b80 1 -- 172.21.15.52:0/1332984982 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f51200b6620 con 0x558d7d674f80 2024-05-27T07:31:42.961 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:31:42.963+0000 7f510effd700 1 -- 172.21.15.52:0/1332984982 <== mon.0 v2:172.21.15.52:3300/0 3 ==== mon_get_version_reply(handle=1 version=101) ==== 24+0+0 (secure 0 0 0) 0x7f511803a040 con 0x558d7d674f80 2024-05-27T07:31:42.961 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:31:42.963+0000 7f5144c65b80 10 client.16662 fetch_fsmap learned FSMap version 101 2024-05-27T07:31:42.961 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:31:42.963+0000 7f5144c65b80 1 -- 172.21.15.52:0/1332984982 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({config=0+,fsmap=101,monmap=4+}) -- 0x558d7d701270 con 0x558d7d674f80 2024-05-27T07:31:42.961 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:31:42.964+0000 7f510effd700 1 -- 172.21.15.52:0/1332984982 <== mon.0 v2:172.21.15.52:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f511800bd30 con 0x558d7d674f80 2024-05-27T07:31:42.961 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:31:42.964+0000 7f510effd700 1 -- 172.21.15.52:0/1332984982 <== mon.0 v2:172.21.15.52:3300/0 5 ==== fsmap(e 101) ==== 4203+0+0 (secure 0 0 0) 0x7f5118011030 con 0x558d7d674f80 2024-05-27T07:31:42.961 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:31:42.964+0000 7f5144c65b80 10 client.16662 fetch_fsmap finished waiting for FSMap version 101 2024-05-27T07:31:42.961 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:31:42.964+0000 7f5144c65b80 10 client.16662 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.e 2024-05-27T07:31:42.961 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:31:42.964+0000 7f5144c65b80 20 client.16662 populate_metadata read hostname 'smithi052' 2024-05-27T07:31:42.961 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:31:42.964+0000 7f5144c65b80 1 --2- 172.21.15.52:0/1332984982 >> [v2:172.21.15.105:6834/3298498384,v1:172.21.15.105:6835/3298498384] conn(0x558d7d696d50 0x558d7d7038a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:31:42.961 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:31:42.964+0000 7f5144c65b80 4 client.16662 mds_command: new command op to 24473 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-05-27T07:31:42.961 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:31:42.964+0000 7f5144c65b80 1 -- 172.21.15.52:0/1332984982 --> [v2:172.21.15.105:6834/3298498384,v1:172.21.15.105:6835/3298498384] -- command(tid 0: {"prefix": "damage ls"}) -- 0x7f51200b6620 con 0x558d7d696d50 2024-05-27T07:31:42.961 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:31:42.964+0000 7f512f7fe700 1 --2- 172.21.15.52:0/1332984982 >> [v2:172.21.15.105:6834/3298498384,v1:172.21.15.105:6835/3298498384] conn(0x558d7d696d50 0x558d7d7038a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:31:42.961 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:31:42.965+0000 7f510effd700 10 client.16662 ms_handle_connect on v2:172.21.15.105:6834/3298498384 2024-05-27T07:31:42.962 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:31:42.965+0000 7f512f7fe700 1 --2- 172.21.15.52:0/1332984982 >> [v2:172.21.15.105:6834/3298498384,v1:172.21.15.105:6835/3298498384] conn(0x558d7d696d50 0x558d7d7038a0 crc :-1 s=READY pgs=42 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=eb09d2baf7b05e48 server_cookie=8b7259ebd3145e8 in_seq=0 out_seq=0 2024-05-27T07:31:42.962 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:31:42.965+0000 7f510effd700 1 -- 172.21.15.52:0/1332984982 <== mds.3 v2:172.21.15.105:6834/3298498384 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x7f51200b6620 con 0x558d7d696d50 2024-05-27T07:31:42.962 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:31:42.965+0000 7f510effd700 10 client.16662 handle_command_reply: tid=0 2024-05-27T07:31:42.962 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:31:42.965+0000 7f5144c65b80 1 client.16662 shutdown 2024-05-27T07:31:42.962 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:31:42.966+0000 7f5144c65b80 1 -- 172.21.15.52:0/1332984982 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x558d7d674f80 msgr2=0x558d7d6fa140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:31:42.962 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:31:42.966+0000 7f5144c65b80 1 --2- 172.21.15.52:0/1332984982 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x558d7d674f80 0x558d7d6fa140 secure :-1 s=READY pgs=662 cs=0 l=1 rev1=1 crypto rx=0x7f51180406d0 tx=0x7f511802a910 comp rx=0 tx=0).stop 2024-05-27T07:31:42.962 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:31:42.966+0000 7f5144c65b80 1 -- 172.21.15.52:0/1332984982 shutdown_connections 2024-05-27T07:31:42.962 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:31:42.966+0000 7f5144c65b80 1 --2- 172.21.15.52:0/1332984982 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x558d7d67ac50 0x558d7d681fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:31:42.962 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:31:42.966+0000 7f5144c65b80 1 --2- 172.21.15.52:0/1332984982 >> [v2:172.21.15.105:6834/3298498384,v1:172.21.15.105:6835/3298498384] conn(0x558d7d696d50 0x558d7d7038a0 crc :-1 s=READY pgs=42 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:31:42.962 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:31:42.966+0000 7f510effd700 0 client.16662 ms_handle_reset on v2:172.21.15.105:6834/3298498384 2024-05-27T07:31:42.962 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:31:42.966+0000 7f5144c65b80 1 --2- 172.21.15.52:0/1332984982 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x558d7d4f6e40 0x558d7d6f9c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:31:42.963 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:31:42.966+0000 7f5144c65b80 1 --2- 172.21.15.52:0/1332984982 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x558d7d674f80 0x558d7d6fa140 unknown :-1 s=CLOSED pgs=662 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:31:42.963 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:31:42.966+0000 7f5144c65b80 1 -- 172.21.15.52:0/1332984982 >> 172.21.15.52:0/1332984982 conn(0x558d7d551af0 msgr2=0x558d7d67b060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:31:42.963 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:31:42.966+0000 7f5144c65b80 1 -- 172.21.15.52:0/1332984982 shutdown_connections 2024-05-27T07:31:42.963 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:31:42.967+0000 7f5144c65b80 1 -- 172.21.15.52:0/1332984982 wait complete. 2024-05-27T07:31:42.963 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:31:42.967+0000 7f5144c65b80 20 client.16662 trim_cache size 0 max 16384 2024-05-27T07:31:42.963 INFO:teuthology.orchestra.run.smithi052.stdout:[] 2024-05-27T07:31:42.964 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:31:42.969+0000 7f5136250700 1 -- 172.21.15.52:0/2141627318 >> [v2:172.21.15.52:6800/1822318384,v1:172.21.15.52:6801/1822318384] conn(0x7f511c0b2a70 msgr2=0x7f511c0b4f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:31:42.964 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:31:42.969+0000 7f5136250700 1 --2- 172.21.15.52:0/2141627318 >> [v2:172.21.15.52:6800/1822318384,v1:172.21.15.52:6801/1822318384] conn(0x7f511c0b2a70 0x7f511c0b4f80 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7f5130105a00 tx=0x7f5118009380 comp rx=0 tx=0).stop 2024-05-27T07:31:42.964 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:31:42.969+0000 7f5136250700 1 -- 172.21.15.52:0/2141627318 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f51301034b0 msgr2=0x7f51301a0a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:31:42.964 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:31:42.969+0000 7f5136250700 1 --2- 172.21.15.52:0/2141627318 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f51301034b0 0x7f51301a0a40 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7f512000e9a0 tx=0x7f512000ecb0 comp rx=0 tx=0).stop 2024-05-27T07:31:42.964 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:31:42.970+0000 7f5136250700 1 -- 172.21.15.52:0/2141627318 shutdown_connections 2024-05-27T07:31:42.964 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:31:42.970+0000 7f5136250700 1 --2- 172.21.15.52:0/2141627318 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f5130108730 0x7f513019abf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:31:42.965 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:31:42.970+0000 7f5136250700 1 --2- 172.21.15.52:0/2141627318 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f51301034b0 0x7f51301a0a40 unknown :-1 s=CLOSED pgs=181 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:31:42.965 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:31:42.970+0000 7f5136250700 1 --2- 172.21.15.52:0/2141627318 >> [v2:172.21.15.52:6800/1822318384,v1:172.21.15.52:6801/1822318384] conn(0x7f511c0b2a70 0x7f511c0b4f80 unknown :-1 s=CLOSED pgs=149 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:31:42.965 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:31:42.970+0000 7f5136250700 1 --2- 172.21.15.52:0/2141627318 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f5130107de0 0x7f51301a0f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:31:42.965 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:31:42.970+0000 7f5136250700 1 -- 172.21.15.52:0/2141627318 >> 172.21.15.52:0/2141627318 conn(0x7f51300fedb0 msgr2=0x7f51301011d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:31:42.965 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:31:42.970+0000 7f5136250700 1 -- 172.21.15.52:0/2141627318 shutdown_connections 2024-05-27T07:31:42.965 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:31:42.971+0000 7f5136250700 1 -- 172.21.15.52:0/2141627318 wait complete. 2024-05-27T07:31:44.288 DEBUG:teuthology.orchestra.run.smithi052:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:38f9d900c83c63799fbdbe61acc9a11b0d3554a6 shell --fsid 73df2636-1bf8-11ef-bc9b-c7b262605968 -- ceph tell mds.1:2 damage ls 2024-05-27T07:31:44.485 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:31:44 smithi052 ceph-mon[31955]: pgmap v201: 129 pgs: 129 active+clean; 496 MiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 354 KiB/s rd, 3.2 MiB/s wr, 958 op/s 2024-05-27T07:31:44.605 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:31:44 smithi173 ceph-mon[37000]: pgmap v201: 129 pgs: 129 active+clean; 496 MiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 354 KiB/s rd, 3.2 MiB/s wr, 958 op/s 2024-05-27T07:31:44.692 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:31:44 smithi105 ceph-mon[35977]: pgmap v201: 129 pgs: 129 active+clean; 496 MiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 354 KiB/s rd, 3.2 MiB/s wr, 958 op/s 2024-05-27T07:31:45.219 INFO:tasks.workunit.client.0.smithi052.stdout: 137 1510895 389 1067804 314 845729 1319 2024-05-27T07:31:46.482 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:31:46 smithi052 ceph-mon[31955]: pgmap v202: 129 pgs: 129 active+clean; 505 MiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 635 KiB/s rd, 3.2 MiB/s wr, 861 op/s 2024-05-27T07:31:46.494 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:31:46 smithi105 ceph-mon[35977]: pgmap v202: 129 pgs: 129 active+clean; 505 MiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 635 KiB/s rd, 3.2 MiB/s wr, 861 op/s 2024-05-27T07:31:46.505 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:31:46 smithi173 ceph-mon[37000]: pgmap v202: 129 pgs: 129 active+clean; 505 MiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 635 KiB/s rd, 3.2 MiB/s wr, 861 op/s 2024-05-27T07:31:47.033 INFO:teuthology.orchestra.run.smithi052.stderr:Inferring config /var/lib/ceph/73df2636-1bf8-11ef-bc9b-c7b262605968/mon.a/config 2024-05-27T07:31:48.485 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:31:48 smithi052 ceph-mon[31955]: pgmap v203: 129 pgs: 129 active+clean; 504 MiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 605 KiB/s rd, 2.0 MiB/s wr, 558 op/s 2024-05-27T07:31:48.604 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:31:48 smithi173 ceph-mon[37000]: pgmap v203: 129 pgs: 129 active+clean; 504 MiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 605 KiB/s rd, 2.0 MiB/s wr, 558 op/s 2024-05-27T07:31:48.692 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:31:48 smithi105 ceph-mon[35977]: pgmap v203: 129 pgs: 129 active+clean; 504 MiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 605 KiB/s rd, 2.0 MiB/s wr, 558 op/s 2024-05-27T07:31:49.778 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:31:49.784+0000 7f02f0d6c640 1 -- 172.21.15.52:0/1038820532 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f02ec10ee80 msgr2=0x7f02ec10f2b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:31:49.778 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:31:49.784+0000 7f02f0d6c640 1 --2- 172.21.15.52:0/1038820532 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f02ec10ee80 0x7f02ec10f2b0 secure :-1 s=READY pgs=663 cs=0 l=1 rev1=1 crypto rx=0x7f02dc009900 tx=0x7f02dc02a410 comp rx=0 tx=0).stop 2024-05-27T07:31:49.781 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:31:49.789+0000 7f02f0d6c640 1 -- 172.21.15.52:0/1038820532 shutdown_connections 2024-05-27T07:31:49.781 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:31:49.789+0000 7f02f0d6c640 1 --2- 172.21.15.52:0/1038820532 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f02ec10f940 0x7f02ec1170e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:31:49.782 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:31:49.789+0000 7f02f0d6c640 1 --2- 172.21.15.52:0/1038820532 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f02ec10ee80 0x7f02ec10f2b0 secure :-1 s=CLOSED pgs=663 cs=0 l=1 rev1=1 crypto rx=0x7f02dc009900 tx=0x7f02dc02a410 comp rx=0 tx=0).stop 2024-05-27T07:31:49.782 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:31:49.789+0000 7f02f0d6c640 1 --2- 172.21.15.52:0/1038820532 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f02ec10a8f0 0x7f02ec10ad00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:31:49.782 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:31:49.789+0000 7f02f0d6c640 1 -- 172.21.15.52:0/1038820532 >> 172.21.15.52:0/1038820532 conn(0x7f02ec06eda0 msgr2=0x7f02ec06f1b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:31:49.798 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:31:49.805+0000 7f02f0d6c640 1 -- 172.21.15.52:0/1038820532 shutdown_connections 2024-05-27T07:31:49.799 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:31:49.807+0000 7f02f0d6c640 1 -- 172.21.15.52:0/1038820532 wait complete. 2024-05-27T07:31:49.800 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:31:49.808+0000 7f02f0d6c640 1 Processor -- start 2024-05-27T07:31:49.801 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:31:49.808+0000 7f02f0d6c640 1 -- start start 2024-05-27T07:31:49.801 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:31:49.808+0000 7f02f0d6c640 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f02ec10a8f0 0x7f02ec11c230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:31:49.801 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:31:49.808+0000 7f02f0d6c640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f02ec10f940 0x7f02ec11c770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:31:49.801 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:31:49.808+0000 7f02f0d6c640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f02ec1b3470 0x7f02ec1b5890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:31:49.801 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:31:49.808+0000 7f02f0d6c640 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x7f02ec11a0e0 con 0x7f02ec10a8f0 2024-05-27T07:31:49.801 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:31:49.808+0000 7f02f0d6c640 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f02ec119f60 con 0x7f02ec10f940 2024-05-27T07:31:49.801 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:31:49.808+0000 7f02f0d6c640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f02ec11a260 con 0x7f02ec1b3470 2024-05-27T07:31:49.802 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:31:49.809+0000 7f02e9d9b640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f02ec10f940 0x7f02ec11c770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:31:49.802 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:31:49.809+0000 7f02ead9d640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f02ec1b3470 0x7f02ec1b5890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:31:49.802 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:31:49.809+0000 7f02ead9d640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f02ec1b3470 0x7f02ec1b5890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.173:3300/0 says I am v2:172.21.15.52:39230/0 (socket says 172.21.15.52:39230) 2024-05-27T07:31:49.802 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:31:49.809+0000 7f02ead9d640 1 -- 172.21.15.52:0/62428442 learned_addr learned my addr 172.21.15.52:0/62428442 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-05-27T07:31:49.802 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:31:49.809+0000 7f02e9d9b640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f02ec10f940 0x7f02ec11c770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.52:34574/0 (socket says 172.21.15.52:34574) 2024-05-27T07:31:49.802 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:31:49.809+0000 7f02ead9d640 1 -- 172.21.15.52:0/62428442 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f02ec10f940 msgr2=0x7f02ec11c770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:31:49.802 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:31:49.809+0000 7f02ead9d640 1 --2- 172.21.15.52:0/62428442 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f02ec10f940 0x7f02ec11c770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:31:49.802 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:31:49.809+0000 7f02ead9d640 1 -- 172.21.15.52:0/62428442 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f02ec10a8f0 msgr2=0x7f02ec11c230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:31:49.802 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:31:49.809+0000 7f02ead9d640 1 --2- 172.21.15.52:0/62428442 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f02ec10a8f0 0x7f02ec11c230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:31:49.802 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:31:49.809+0000 7f02ead9d640 1 -- 172.21.15.52:0/62428442 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f02ec11ef00 con 0x7f02ec1b3470 2024-05-27T07:31:49.803 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:31:49.809+0000 7f02ead9d640 1 --2- 172.21.15.52:0/62428442 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f02ec1b3470 0x7f02ec1b5890 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7f02e0007860 tx=0x7f02e0007d20 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:31:49.804 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:31:49.811+0000 7f02db7fe640 1 -- 172.21.15.52:0/62428442 <== mon.1 v2:172.21.15.173:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f02e0017030 con 0x7f02ec1b3470 2024-05-27T07:31:49.804 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:31:49.811+0000 7f02f0d6c640 1 -- 172.21.15.52:0/62428442 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f02ec11f1f0 con 0x7f02ec1b3470 2024-05-27T07:31:49.804 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:31:49.811+0000 7f02f0d6c640 1 -- 172.21.15.52:0/62428442 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f02ec11f710 con 0x7f02ec1b3470 2024-05-27T07:31:49.804 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:31:49.812+0000 7f02db7fe640 1 -- 172.21.15.52:0/62428442 <== mon.1 v2:172.21.15.173:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f02e000dd60 con 0x7f02ec1b3470 2024-05-27T07:31:49.804 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:31:49.812+0000 7f02db7fe640 1 -- 172.21.15.52:0/62428442 <== mon.1 v2:172.21.15.173:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f02e0004a00 con 0x7f02ec1b3470 2024-05-27T07:31:49.810 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:31:49.813+0000 7f46ac5e0700 1 -- 172.21.15.52:0/885286167 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f46a41045e0 msgr2=0x7f46a41049f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:31:49.810 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:31:49.813+0000 7f46ac5e0700 1 --2- 172.21.15.52:0/885286167 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f46a41045e0 0x7f46a41049f0 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7f46a0009990 tx=0x7f46a0009ca0 comp rx=0 tx=0).stop 2024-05-27T07:31:49.810 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:31:49.817+0000 7f46ac5e0700 1 -- 172.21.15.52:0/885286167 shutdown_connections 2024-05-27T07:31:49.810 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:31:49.817+0000 7f46ac5e0700 1 --2- 172.21.15.52:0/885286167 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f46a4109860 0x7f46a41111c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:31:49.810 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:31:49.817+0000 7f46ac5e0700 1 --2- 172.21.15.52:0/885286167 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f46a41045e0 0x7f46a41049f0 unknown :-1 s=CLOSED pgs=183 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:31:49.812 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:31:49.817+0000 7f46ac5e0700 1 --2- 172.21.15.52:0/885286167 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f46a4108f10 0x7f46a4109320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:31:49.812 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:31:49.817+0000 7f46ac5e0700 1 -- 172.21.15.52:0/885286167 >> 172.21.15.52:0/885286167 conn(0x7f46a4100030 msgr2=0x7f46a4102450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:31:49.815 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:31:49.820+0000 7f46ac5e0700 1 -- 172.21.15.52:0/885286167 shutdown_connections 2024-05-27T07:31:49.817 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:31:49.824+0000 7f02db7fe640 1 -- 172.21.15.52:0/62428442 <== mon.1 v2:172.21.15.173:3300/0 4 ==== mgrmap(e 26) ==== 149783+0+0 (secure 0 0 0) 0x7f02e001a070 con 0x7f02ec1b3470 2024-05-27T07:31:49.817 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:31:49.824+0000 7f02f7c1a740 1 Processor -- start 2024-05-27T07:31:49.817 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:31:49.824+0000 7f02f7c1a740 1 -- start start 2024-05-27T07:31:49.818 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:31:49.823+0000 7f46ac5e0700 1 -- 172.21.15.52:0/885286167 wait complete. 2024-05-27T07:31:49.818 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:31:49.824+0000 7f46ac5e0700 1 Processor -- start 2024-05-27T07:31:49.818 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:31:49.824+0000 7f46ac5e0700 1 -- start start 2024-05-27T07:31:49.818 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:31:49.825+0000 7f46ac5e0700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f46a41045e0 0x7f46a4119f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:31:49.821 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:31:49.825+0000 7f02f7c1a740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x556f935491b0 0x556f938ea3d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:31:49.821 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:31:49.825+0000 7f02f7c1a740 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x556f935202f0 0x556f938ea910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:31:49.821 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:31:49.825+0000 7f02f7c1a740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x556f935284d0 0x556f938eae50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:31:49.821 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:31:49.825+0000 7f02f7c1a740 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x556f9385cc30 con 0x556f935202f0 2024-05-27T07:31:49.821 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:31:49.825+0000 7f02f7c1a740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x556f9384c7d0 con 0x556f935491b0 2024-05-27T07:31:49.821 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:31:49.825+0000 7f02f7c1a740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x556f93850810 con 0x556f935284d0 2024-05-27T07:31:49.822 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:31:49.826+0000 7f02db7fe640 1 --2- 172.21.15.52:0/62428442 >> [v2:172.21.15.52:6800/1822318384,v1:172.21.15.52:6801/1822318384] conn(0x7f02d40b2ae0 0x7f02d40b4ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:31:49.822 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:31:49.826+0000 7f02e9d9b640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x556f935491b0 0x556f938ea3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:31:49.822 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:31:49.826+0000 7f02e9d9b640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x556f935491b0 0x556f938ea3d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.52:34592/0 (socket says 172.21.15.52:34592) 2024-05-27T07:31:49.822 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:31:49.826+0000 7f02e9d9b640 1 -- 172.21.15.52:0/688573814 learned_addr learned my addr 172.21.15.52:0/688573814 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-05-27T07:31:49.822 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:31:49.826+0000 7f02db7fe640 1 -- 172.21.15.52:0/62428442 <== mon.1 v2:172.21.15.173:3300/0 5 ==== osd_map(116..116 src has 1..116) ==== 9379+0+0 (secure 0 0 0) 0x7f02e00e26a0 con 0x7f02ec1b3470 2024-05-27T07:31:49.822 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:31:49.825+0000 7f46ac5e0700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f46a4108f10 0x7f46a41130b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:31:49.822 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:31:49.825+0000 7f46ac5e0700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f46a4109860 0x7f46a4113700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:31:49.822 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:31:49.825+0000 7f46ac5e0700 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x7f46a411ac60 con 0x7f46a41045e0 2024-05-27T07:31:49.822 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:31:49.825+0000 7f46ac5e0700 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f46a411aae0 con 0x7f46a4109860 2024-05-27T07:31:49.822 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:31:49.825+0000 7f46aab7d700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f46a4109860 0x7f46a4113700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:31:49.823 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:31:49.826+0000 7f46aab7d700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f46a4109860 0x7f46a4113700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.52:34578/0 (socket says 172.21.15.52:34578) 2024-05-27T07:31:49.823 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:31:49.826+0000 7f46aab7d700 1 -- 172.21.15.52:0/3544192187 learned_addr learned my addr 172.21.15.52:0/3544192187 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-05-27T07:31:49.823 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:31:49.826+0000 7f46ac5e0700 1 -- 172.21.15.52:0/3544192187 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f46a411ade0 con 0x7f46a4108f10 2024-05-27T07:31:49.823 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:31:49.826+0000 7f46aa37c700 1 --2- 172.21.15.52:0/3544192187 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f46a41045e0 0x7f46a4119f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:31:49.823 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:31:49.827+0000 7f02e9d9b640 1 --2- 172.21.15.52:0/688573814 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x556f935284d0 0x556f938eae50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:31:49.823 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:31:49.827+0000 7f02e9d9b640 1 -- 172.21.15.52:0/688573814 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x556f935284d0 msgr2=0x556f938eae50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:31:49.823 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:31:49.827+0000 7f02e9d9b640 1 --2- 172.21.15.52:0/688573814 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x556f935284d0 0x556f938eae50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:31:49.823 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:31:49.827+0000 7f02e9d9b640 1 -- 172.21.15.52:0/688573814 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x556f935202f0 msgr2=0x556f938ea910 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-05-27T07:31:49.823 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:31:49.827+0000 7f02e9d9b640 1 --2- 172.21.15.52:0/688573814 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x556f935202f0 0x556f938ea910 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:31:49.824 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:31:49.827+0000 7f02e9d9b640 1 -- 172.21.15.52:0/688573814 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x556f93561210 con 0x556f935491b0 2024-05-27T07:31:49.824 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:31:49.829+0000 7f02e9d9b640 1 --2- 172.21.15.52:0/688573814 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x556f935491b0 0x556f938ea3d0 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7f02dc009850 tx=0x7f02dc005fd0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=1ca70821c1721edf server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:31:49.825 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:31:49.833+0000 7f02d8ff9640 1 -- 172.21.15.52:0/688573814 <== mon.2 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f02dc035040 con 0x556f935491b0 2024-05-27T07:31:49.825 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:31:49.833+0000 7f02d8ff9640 1 -- 172.21.15.52:0/688573814 <== mon.2 v2:172.21.15.105:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f02dc005570 con 0x556f935491b0 2024-05-27T07:31:49.826 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:31:49.833+0000 7f02d8ff9640 1 -- 172.21.15.52:0/688573814 <== mon.2 v2:172.21.15.105:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f02dc003a10 con 0x556f935491b0 2024-05-27T07:31:49.826 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:31:49.830+0000 7f46aab7d700 1 -- 172.21.15.52:0/3544192187 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f46a4108f10 msgr2=0x7f46a41130b0 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-05-27T07:31:49.826 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:31:49.830+0000 7f46aab7d700 1 --2- 172.21.15.52:0/3544192187 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f46a4108f10 0x7f46a41130b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:31:49.826 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:31:49.830+0000 7f46aab7d700 1 -- 172.21.15.52:0/3544192187 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f46a41045e0 msgr2=0x7f46a4119f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:31:49.826 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:31:49.830+0000 7f46aab7d700 1 --2- 172.21.15.52:0/3544192187 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f46a41045e0 0x7f46a4119f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:31:49.826 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:31:49.830+0000 7f46aab7d700 1 -- 172.21.15.52:0/3544192187 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f46a4117a60 con 0x7f46a4109860 2024-05-27T07:31:49.826 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:31:49.830+0000 7f46aab7d700 1 --2- 172.21.15.52:0/3544192187 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f46a4109860 0x7f46a4113700 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7f469c00b970 tx=0x7f469c00bc80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:31:49.826 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:31:49.831+0000 7f469b7fe700 1 -- 172.21.15.52:0/3544192187 <== mon.2 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f469c019070 con 0x7f46a4109860 2024-05-27T07:31:49.827 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:31:49.831+0000 7f46ac5e0700 1 -- 172.21.15.52:0/3544192187 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f46a4117d50 con 0x7f46a4109860 2024-05-27T07:31:49.827 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:31:49.831+0000 7f46ac5e0700 1 -- 172.21.15.52:0/3544192187 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f46a4118290 con 0x7f46a4109860 2024-05-27T07:31:49.827 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:31:49.831+0000 7f469b7fe700 1 -- 172.21.15.52:0/3544192187 <== mon.2 v2:172.21.15.105:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f469c00cde0 con 0x7f46a4109860 2024-05-27T07:31:49.827 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:31:49.831+0000 7f469b7fe700 1 -- 172.21.15.52:0/3544192187 <== mon.2 v2:172.21.15.105:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f469c007370 con 0x7f46a4109860 2024-05-27T07:31:49.828 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:31:49.835+0000 7f02ead9d640 1 --2- 172.21.15.52:0/62428442 >> [v2:172.21.15.52:6800/1822318384,v1:172.21.15.52:6801/1822318384] conn(0x7f02d40b2ae0 0x7f02d40b4ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:31:49.828 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:31:49.835+0000 7f02f7c1a740 1 -- 172.21.15.52:0/688573814 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x556f935491b0 msgr2=0x556f938ea3d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:31:49.828 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:31:49.835+0000 7f02f7c1a740 1 --2- 172.21.15.52:0/688573814 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x556f935491b0 0x556f938ea3d0 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7f02dc009850 tx=0x7f02dc005fd0 comp rx=0 tx=0).stop 2024-05-27T07:31:49.832 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:31:49.839+0000 7f02f7c1a740 1 -- 172.21.15.52:0/688573814 shutdown_connections 2024-05-27T07:31:49.832 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:31:49.839+0000 7f02f7c1a740 1 --2- 172.21.15.52:0/688573814 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x556f935284d0 0x556f938eae50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:31:49.832 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:31:49.839+0000 7f02f7c1a740 1 --2- 172.21.15.52:0/688573814 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x556f935202f0 0x556f938ea910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:31:49.832 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:31:49.839+0000 7f02f7c1a740 1 --2- 172.21.15.52:0/688573814 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x556f935491b0 0x556f938ea3d0 unknown :-1 s=CLOSED pgs=184 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:31:49.832 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:31:49.839+0000 7f02f7c1a740 1 -- 172.21.15.52:0/688573814 >> 172.21.15.52:0/688573814 conn(0x556f9353d300 msgr2=0x556f938e6b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:31:49.838 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:31:49.846+0000 7f02ead9d640 1 --2- 172.21.15.52:0/62428442 >> [v2:172.21.15.52:6800/1822318384,v1:172.21.15.52:6801/1822318384] conn(0x7f02d40b2ae0 0x7f02d40b4ff0 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7f02e00a95b0 tx=0x7f02e0005b90 comp rx=0 tx=0).ready entity=mgr.15333 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:31:49.842 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:31:49.849+0000 7f02f7c1a740 1 -- 172.21.15.52:0/688573814 shutdown_connections 2024-05-27T07:31:49.844 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:31:49.845+0000 7f46baff5b80 1 Processor -- start 2024-05-27T07:31:49.844 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:31:49.852+0000 7f02f7c1a740 1 -- 172.21.15.52:0/688573814 wait complete. 2024-05-27T07:31:49.845 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:31:49.852+0000 7f02f7c1a740 1 -- start start 2024-05-27T07:31:49.846 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:31:49.852+0000 7f02f7c1a740 1 Processor -- start 2024-05-27T07:31:49.847 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:31:49.854+0000 7f02f7c1a740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x556f935202f0 0x556f938f64c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:31:49.847 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:31:49.854+0000 7f02f7c1a740 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x556f935284d0 0x556f938f6a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:31:49.847 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:31:49.854+0000 7f02f7c1a740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x556f935491b0 0x556f938f4940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:31:49.847 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:31:49.854+0000 7f02f7c1a740 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x556f93532790 con 0x556f935284d0 2024-05-27T07:31:49.847 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:31:49.854+0000 7f02f7c1a740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x556f93808890 con 0x556f935202f0 2024-05-27T07:31:49.847 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:31:49.854+0000 7f02f7c1a740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x556f938f74a0 con 0x556f935491b0 2024-05-27T07:31:49.847 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:31:49.854+0000 7f02f7c1a740 10 client.0 _ll_register_callbacks cb 0x556f93560a00 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T07:31:49.848 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:31:49.851+0000 7f469b7fe700 1 -- 172.21.15.52:0/3544192187 <== mon.2 v2:172.21.15.105:3300/0 4 ==== mgrmap(e 26) ==== 149783+0+0 (secure 0 0 0) 0x7f469c01c070 con 0x7f46a4109860 2024-05-27T07:31:49.848 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:31:49.852+0000 7f469b7fe700 1 --2- 172.21.15.52:0/3544192187 >> [v2:172.21.15.52:6800/1822318384,v1:172.21.15.52:6801/1822318384] conn(0x7f46900b2ab0 0x7f46900b4fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:31:49.848 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:31:49.853+0000 7f469b7fe700 1 -- 172.21.15.52:0/3544192187 <== mon.2 v2:172.21.15.105:3300/0 5 ==== osd_map(116..116 src has 1..116) ==== 9379+0+0 (secure 0 0 0) 0x7f469c0e1c10 con 0x7f46a4109860 2024-05-27T07:31:49.848 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:31:49.853+0000 7f46a9b7b700 1 --2- 172.21.15.52:0/3544192187 >> [v2:172.21.15.52:6800/1822318384,v1:172.21.15.52:6801/1822318384] conn(0x7f46900b2ab0 0x7f46900b4fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:31:49.848 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:31:49.854+0000 7f46baff5b80 1 -- start start 2024-05-27T07:31:49.849 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:31:49.854+0000 7f46baff5b80 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x559611afde40 0x559611c81710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:31:49.849 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:31:49.854+0000 7f46baff5b80 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x559611c81c50 0x559611c7ba40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:31:49.849 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:31:49.854+0000 7f46baff5b80 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x559611c7bf80 0x559611c7c390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:31:49.849 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:31:49.854+0000 7f46baff5b80 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x559611baac90 con 0x559611afde40 2024-05-27T07:31:49.849 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:31:49.854+0000 7f46baff5b80 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x559611b9a830 con 0x559611c7bf80 2024-05-27T07:31:49.849 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:31:49.854+0000 7f46baff5b80 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x559611b9e870 con 0x559611c81c50 2024-05-27T07:31:49.849 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:31:49.855+0000 7f46aa37c700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x559611afde40 0x559611c81710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:31:49.849 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:31:49.855+0000 7f46aa37c700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x559611afde40 0x559611c81710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.52:58856/0 (socket says 172.21.15.52:58856) 2024-05-27T07:31:49.849 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:31:49.855+0000 7f46aa37c700 1 -- 172.21.15.52:0/609888276 learned_addr learned my addr 172.21.15.52:0/609888276 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-05-27T07:31:49.849 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:31:49.856+0000 7f02ea59c640 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x556f935284d0 0x556f938f6a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:31:49.849 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:31:49.856+0000 7f02ea59c640 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x556f935284d0 0x556f938f6a00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.52:58872/0 (socket says 172.21.15.52:58872) 2024-05-27T07:31:49.850 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:31:49.856+0000 7f02ea59c640 1 -- 172.21.15.52:0/3097825534 learned_addr learned my addr 172.21.15.52:0/3097825534 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-05-27T07:31:49.850 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:31:49.856+0000 7f46aab7d700 1 --2- 172.21.15.52:0/609888276 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x559611c7bf80 0x559611c7c390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:31:49.850 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:31:49.857+0000 7f46a9b7b700 1 --2- 172.21.15.52:0/609888276 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x559611c81c50 0x559611c7ba40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:31:49.850 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:31:49.857+0000 7f46aab7d700 1 -- 172.21.15.52:0/609888276 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x559611c81c50 msgr2=0x559611c7ba40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:31:49.851 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:31:49.857+0000 7f46aab7d700 1 --2- 172.21.15.52:0/609888276 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x559611c81c50 0x559611c7ba40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:31:49.851 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:31:49.857+0000 7f46aab7d700 1 -- 172.21.15.52:0/609888276 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x559611afde40 msgr2=0x559611c81710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:31:49.851 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:31:49.857+0000 7f46aab7d700 1 --2- 172.21.15.52:0/609888276 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x559611afde40 0x559611c81710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:31:49.851 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:31:49.857+0000 7f46aab7d700 1 -- 172.21.15.52:0/609888276 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x559611aea8d0 con 0x559611c7bf80 2024-05-27T07:31:49.851 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:31:49.857+0000 7f46aab7d700 1 --2- 172.21.15.52:0/609888276 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x559611c7bf80 0x559611c7c390 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7f469c0df110 tx=0x7f469c0a8700 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f232bba5f8b6497 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:31:49.851 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:31:49.857+0000 7f4698ff9700 1 -- 172.21.15.52:0/609888276 <== mon.2 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f469c0a93a0 con 0x559611c7bf80 2024-05-27T07:31:49.851 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:31:49.858+0000 7f02ea59c640 1 -- 172.21.15.52:0/3097825534 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x556f935491b0 msgr2=0x556f938f4940 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-05-27T07:31:49.851 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:31:49.858+0000 7f02ea59c640 1 --2- 172.21.15.52:0/3097825534 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x556f935491b0 0x556f938f4940 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:31:49.851 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:31:49.858+0000 7f02ea59c640 1 -- 172.21.15.52:0/3097825534 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x556f935202f0 msgr2=0x556f938f64c0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-05-27T07:31:49.851 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:31:49.858+0000 7f02ea59c640 1 --2- 172.21.15.52:0/3097825534 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x556f935202f0 0x556f938f64c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:31:49.852 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:31:49.858+0000 7f02ea59c640 1 -- 172.21.15.52:0/3097825534 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({osdmap=0}) -- 0x556f9362d620 con 0x556f935284d0 2024-05-27T07:31:49.852 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:31:49.858+0000 7f4698ff9700 1 -- 172.21.15.52:0/609888276 <== mon.2 v2:172.21.15.105:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f469c0a8ca0 con 0x559611c7bf80 2024-05-27T07:31:49.852 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:31:49.858+0000 7f4698ff9700 1 -- 172.21.15.52:0/609888276 <== mon.2 v2:172.21.15.105:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f469c0a8160 con 0x559611c7bf80 2024-05-27T07:31:49.852 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:31:49.858+0000 7f02ea59c640 1 --2- 172.21.15.52:0/3097825534 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x556f935284d0 0x556f938f6a00 secure :-1 s=READY pgs=664 cs=0 l=1 rev1=1 crypto rx=0x7f02e400c5b0 tx=0x7f02e400ca70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5e64f0d8f06e1126 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:31:49.853 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:31:49.860+0000 7f02d1ffb640 10 client.0 ms_handle_connect on v2:172.21.15.52:3300/0 2024-05-27T07:31:49.853 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:31:49.860+0000 7f02d1ffb640 1 -- 172.21.15.52:0/3097825534 <== mon.0 v2:172.21.15.52:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f02e400a900 con 0x556f935284d0 2024-05-27T07:31:49.854 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:31:49.860+0000 7f02f7c1a740 1 -- 172.21.15.52:0/3097825534 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f02dc005570 con 0x556f935284d0 2024-05-27T07:31:49.854 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:31:49.860+0000 7f02d1ffb640 1 -- 172.21.15.52:0/3097825534 <== mon.0 v2:172.21.15.52:3300/0 2 ==== osd_map(116..116 src has 1..116) ==== 9379+0+0 (secure 0 0 0) 0x7f02e400aaa0 con 0x556f935284d0 2024-05-27T07:31:49.854 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:31:49.859+0000 7f46baff5b80 1 -- 172.21.15.52:0/609888276 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x559611c7bf80 msgr2=0x559611c7c390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:31:49.854 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:31:49.859+0000 7f46baff5b80 1 --2- 172.21.15.52:0/609888276 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x559611c7bf80 0x559611c7c390 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7f469c0df110 tx=0x7f469c0a8700 comp rx=0 tx=0).stop 2024-05-27T07:31:49.854 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:31:49.861+0000 7f02d1ffb640 1 -- 172.21.15.52:0/3097825534 <== mon.0 v2:172.21.15.52:3300/0 3 ==== mon_get_version_reply(handle=1 version=101) ==== 24+0+0 (secure 0 0 0) 0x7f02e4021050 con 0x556f935284d0 2024-05-27T07:31:49.855 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:31:49.862+0000 7f02f7c1a740 10 client.16674 fetch_fsmap learned FSMap version 101 2024-05-27T07:31:49.855 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:31:49.862+0000 7f02f7c1a740 1 -- 172.21.15.52:0/3097825534 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({config=0+,fsmap=101,monmap=4+}) -- 0x556f93903910 con 0x556f935284d0 2024-05-27T07:31:49.857 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:31:49.862+0000 7f46baff5b80 1 -- 172.21.15.52:0/609888276 shutdown_connections 2024-05-27T07:31:49.857 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:31:49.862+0000 7f46baff5b80 1 --2- 172.21.15.52:0/609888276 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x559611c7bf80 0x559611c7c390 unknown :-1 s=CLOSED pgs=186 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:31:49.857 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:31:49.862+0000 7f46baff5b80 1 --2- 172.21.15.52:0/609888276 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x559611afde40 0x559611c81710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:31:49.857 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:31:49.862+0000 7f46baff5b80 1 --2- 172.21.15.52:0/609888276 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x559611c81c50 0x559611c7ba40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:31:49.857 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:31:49.862+0000 7f46baff5b80 1 -- 172.21.15.52:0/609888276 >> 172.21.15.52:0/609888276 conn(0x559611b58af0 msgr2=0x559611c76640 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:31:49.858 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:31:49.862+0000 7f46a9b7b700 1 --2- 172.21.15.52:0/3544192187 >> [v2:172.21.15.52:6800/1822318384,v1:172.21.15.52:6801/1822318384] conn(0x7f46900b2ab0 0x7f46900b4fc0 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x559611c780a0 tx=0x7f46940091f0 comp rx=0 tx=0).ready entity=mgr.15333 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:31:49.858 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:31:49.864+0000 7f46baff5b80 1 -- 172.21.15.52:0/609888276 shutdown_connections 2024-05-27T07:31:49.858 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:31:49.864+0000 7f46baff5b80 1 -- 172.21.15.52:0/609888276 wait complete. 2024-05-27T07:31:49.861 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:31:49.865+0000 7f46baff5b80 1 -- start start 2024-05-27T07:31:49.861 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:31:49.865+0000 7f46baff5b80 1 Processor -- start 2024-05-27T07:31:49.861 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:31:49.866+0000 7f46baff5b80 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x559611afde40 0x559611c853d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:31:49.861 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:31:49.866+0000 7f46baff5b80 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x559611c7bf80 0x559611c85910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:31:49.861 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:31:49.866+0000 7f46aa37c700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x559611afde40 0x559611c853d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:31:49.862 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:31:49.866+0000 7f46aa37c700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x559611afde40 0x559611c853d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.173:3300/0 says I am v2:172.21.15.52:39266/0 (socket says 172.21.15.52:39266) 2024-05-27T07:31:49.862 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:31:49.867+0000 7f46aa37c700 1 -- 172.21.15.52:0/2100872114 learned_addr learned my addr 172.21.15.52:0/2100872114 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-05-27T07:31:49.862 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:31:49.867+0000 7f46baff5b80 1 --2- 172.21.15.52:0/2100872114 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x559611c81c50 0x559611c85e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:31:49.862 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:31:49.867+0000 7f46baff5b80 1 -- 172.21.15.52:0/2100872114 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x559611aa0fa0 con 0x559611c7bf80 2024-05-27T07:31:49.862 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:31:49.867+0000 7f46aab7d700 1 --2- 172.21.15.52:0/2100872114 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x559611c81c50 0x559611c85e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:31:49.862 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:31:49.867+0000 7f46baff5b80 1 -- 172.21.15.52:0/2100872114 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x559611b3c7c0 con 0x559611c81c50 2024-05-27T07:31:49.862 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:31:49.867+0000 7f46baff5b80 1 -- 172.21.15.52:0/2100872114 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x559611c88790 con 0x559611afde40 2024-05-27T07:31:49.862 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:31:49.868+0000 7f46aa37c700 1 -- 172.21.15.52:0/2100872114 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x559611c81c50 msgr2=0x559611c85e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:31:49.862 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:31:49.868+0000 7f46aa37c700 1 --2- 172.21.15.52:0/2100872114 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x559611c81c50 0x559611c85e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:31:49.862 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:31:49.868+0000 7f46aa37c700 1 -- 172.21.15.52:0/2100872114 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x559611c7bf80 msgr2=0x559611c85910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:31:49.862 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:31:49.868+0000 7f46aa37c700 1 --2- 172.21.15.52:0/2100872114 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x559611c7bf80 0x559611c85910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:31:49.862 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:31:49.868+0000 7f46aa37c700 1 -- 172.21.15.52:0/2100872114 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({osdmap=0}) -- 0x559611b39690 con 0x559611afde40 2024-05-27T07:31:49.862 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:31:49.868+0000 7f46aa37c700 1 --2- 172.21.15.52:0/2100872114 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x559611afde40 0x559611c853d0 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7f46a0015000 tx=0x7f46a000bf20 comp rx=0 tx=0).ready entity=mon.1 client_cookie=554935d81db87684 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:31:49.863 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:31:49.868+0000 7f46baff5b80 10 client.0 _ll_register_callbacks cb 0x559611b35920 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T07:31:49.863 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:31:49.869+0000 7f46baff5b80 1 -- 172.21.15.52:0/2100872114 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f469c0a8ca0 con 0x559611afde40 2024-05-27T07:31:49.863 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:31:49.869+0000 7f02d1ffb640 1 -- 172.21.15.52:0/3097825534 <== mon.0 v2:172.21.15.52:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f02dc005570 con 0x556f935284d0 2024-05-27T07:31:49.863 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:31:49.869+0000 7f02d1ffb640 1 -- 172.21.15.52:0/3097825534 <== mon.0 v2:172.21.15.52:3300/0 5 ==== fsmap(e 101) ==== 4203+0+0 (secure 0 0 0) 0x7f02e4018400 con 0x556f935284d0 2024-05-27T07:31:49.863 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:31:49.871+0000 7f02f7c1a740 10 client.16674 fetch_fsmap finished waiting for FSMap version 101 2024-05-27T07:31:49.863 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:31:49.871+0000 7f02f7c1a740 10 client.16674 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-05-27T07:31:49.863 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:31:49.871+0000 7f02f7c1a740 20 client.16674 populate_metadata read hostname 'smithi052' 2024-05-27T07:31:49.864 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:31:49.871+0000 7f02f7c1a740 1 --2- 172.21.15.52:0/3097825534 >> [v2:172.21.15.105:6838/500348151,v1:172.21.15.105:6839/500348151] conn(0x556f93507d80 0x556f939073b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:31:49.864 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:31:49.871+0000 7f02f7c1a740 4 client.16674 mds_command: new command op to 24527 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-27T07:31:49.864 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:31:49.871+0000 7f02f7c1a740 1 -- 172.21.15.52:0/3097825534 --> [v2:172.21.15.105:6838/500348151,v1:172.21.15.105:6839/500348151] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x556f9385eb80 con 0x556f93507d80 2024-05-27T07:31:49.864 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:31:49.870+0000 7f468dffb700 10 client.26188 ms_handle_connect on v2:172.21.15.173:3300/0 2024-05-27T07:31:49.864 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:31:49.870+0000 7f468dffb700 1 -- 172.21.15.52:0/2100872114 <== mon.1 v2:172.21.15.173:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f46a0003d00 con 0x559611afde40 2024-05-27T07:31:49.864 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:31:49.870+0000 7f468dffb700 1 -- 172.21.15.52:0/2100872114 <== mon.1 v2:172.21.15.173:3300/0 2 ==== osd_map(116..116 src has 1..116) ==== 9379+0+0 (secure 0 0 0) 0x7f46a00050d0 con 0x559611afde40 2024-05-27T07:31:49.867 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:31:49.872+0000 7f468dffb700 1 -- 172.21.15.52:0/2100872114 <== mon.1 v2:172.21.15.173:3300/0 3 ==== mon_get_version_reply(handle=1 version=101) ==== 24+0+0 (secure 0 0 0) 0x7f46a002f050 con 0x559611afde40 2024-05-27T07:31:49.867 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:31:49.872+0000 7f46baff5b80 10 client.26188 fetch_fsmap learned FSMap version 101 2024-05-27T07:31:49.867 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:31:49.872+0000 7f46baff5b80 1 -- 172.21.15.52:0/2100872114 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({fsmap=101}) -- 0x559611c931c0 con 0x559611afde40 2024-05-27T07:31:49.869 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:31:49.874+0000 7f468dffb700 1 -- 172.21.15.52:0/2100872114 <== mon.1 v2:172.21.15.173:3300/0 4 ==== fsmap(e 101) ==== 4203+0+0 (secure 0 0 0) 0x7f46a00052d0 con 0x559611afde40 2024-05-27T07:31:49.869 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:31:49.874+0000 7f46baff5b80 10 client.26188 fetch_fsmap finished waiting for FSMap version 101 2024-05-27T07:31:49.869 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:31:49.874+0000 7f46baff5b80 10 client.26188 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.i 2024-05-27T07:31:49.869 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:31:49.874+0000 7f46baff5b80 20 client.26188 populate_metadata read hostname 'smithi052' 2024-05-27T07:31:49.869 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:31:49.874+0000 7f46baff5b80 1 --2- 172.21.15.52:0/2100872114 >> [v2:172.21.15.173:6838/564778293,v1:172.21.15.173:6839/564778293] conn(0x559611c95bc0 0x559611c980d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:31:49.869 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:31:49.874+0000 7f46baff5b80 4 client.26188 mds_command: new command op to 24475 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-27T07:31:49.870 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:31:49.874+0000 7f46baff5b80 1 -- 172.21.15.52:0/2100872114 --> [v2:172.21.15.173:6838/564778293,v1:172.21.15.173:6839/564778293] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x7f469c0a8ca0 con 0x559611c95bc0 2024-05-27T07:31:49.870 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:31:49.875+0000 7f46a9b7b700 1 --2- 172.21.15.52:0/2100872114 >> [v2:172.21.15.173:6838/564778293,v1:172.21.15.173:6839/564778293] conn(0x559611c95bc0 0x559611c980d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:31:49.870 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:31:49.875+0000 7f46a9b7b700 1 --2- 172.21.15.52:0/2100872114 >> [v2:172.21.15.173:6838/564778293,v1:172.21.15.173:6839/564778293] conn(0x559611c95bc0 0x559611c980d0 crc :-1 s=READY pgs=45 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=e81297e4191d5417 server_cookie=dd44b2321d0ecd8e in_seq=0 out_seq=0 2024-05-27T07:31:49.870 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:31:49.875+0000 7f468dffb700 10 client.26188 ms_handle_connect on v2:172.21.15.173:6838/564778293 2024-05-27T07:31:49.870 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:31:49.876+0000 7f468dffb700 1 -- 172.21.15.52:0/2100872114 <== mds.2 v2:172.21.15.173:6838/564778293 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24041 (crc 0 0 0) 0x7f469c0a8ca0 con 0x559611c95bc0 2024-05-27T07:31:49.870 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:31:49.876+0000 7f468dffb700 10 client.26188 handle_command_reply: tid=0 2024-05-27T07:31:49.870 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:31:49.876+0000 7f46baff5b80 1 client.26188 shutdown 2024-05-27T07:31:49.872 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:31:49.880+0000 7f46baff5b80 1 -- 172.21.15.52:0/2100872114 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x559611afde40 msgr2=0x559611c853d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:31:49.872 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:31:49.880+0000 7f46baff5b80 1 --2- 172.21.15.52:0/2100872114 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x559611afde40 0x559611c853d0 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7f46a0015000 tx=0x7f46a000bf20 comp rx=0 tx=0).stop 2024-05-27T07:31:49.873 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:31:49.880+0000 7f46baff5b80 1 -- 172.21.15.52:0/2100872114 shutdown_connections 2024-05-27T07:31:49.873 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:31:49.880+0000 7f46baff5b80 1 --2- 172.21.15.52:0/2100872114 >> [v2:172.21.15.173:6838/564778293,v1:172.21.15.173:6839/564778293] conn(0x559611c95bc0 0x559611c980d0 crc :-1 s=READY pgs=45 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:31:49.873 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:31:49.880+0000 7f46baff5b80 1 --2- 172.21.15.52:0/2100872114 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x559611c81c50 0x559611c85e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:31:49.873 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:31:49.880+0000 7f46baff5b80 1 --2- 172.21.15.52:0/2100872114 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x559611afde40 0x559611c853d0 unknown :-1 s=CLOSED pgs=150 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:31:49.874 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:31:49.880+0000 7f46baff5b80 1 --2- 172.21.15.52:0/2100872114 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x559611c7bf80 0x559611c85910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:31:49.874 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:31:49.880+0000 7f46baff5b80 1 -- 172.21.15.52:0/2100872114 >> 172.21.15.52:0/2100872114 conn(0x559611b58af0 msgr2=0x559611c7aa30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:31:49.874 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:31:49.880+0000 7f468dffb700 0 client.26188 ms_handle_reset on v2:172.21.15.173:6838/564778293 2024-05-27T07:31:49.874 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:31:49.881+0000 7f46baff5b80 1 -- 172.21.15.52:0/2100872114 shutdown_connections 2024-05-27T07:31:49.874 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:31:49.881+0000 7f46baff5b80 1 -- 172.21.15.52:0/2100872114 wait complete. 2024-05-27T07:31:49.874 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:31:49.881+0000 7f46baff5b80 20 client.26188 trim_cache size 0 max 16384 2024-05-27T07:31:49.875 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:31:49.883+0000 7f02e9d9b640 1 --2- 172.21.15.52:0/3097825534 >> [v2:172.21.15.105:6838/500348151,v1:172.21.15.105:6839/500348151] conn(0x556f93507d80 0x556f939073b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:31:49.879 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:31:49.887+0000 7f02e9d9b640 1 --2- 172.21.15.52:0/3097825534 >> [v2:172.21.15.105:6838/500348151,v1:172.21.15.105:6839/500348151] conn(0x556f93507d80 0x556f939073b0 crc :-1 s=READY pgs=119 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=195cf6d6672f792c server_cookie=8279cec08c049686 in_seq=0 out_seq=0 2024-05-27T07:31:49.879 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:31:49.887+0000 7f02d1ffb640 10 client.16674 ms_handle_connect on v2:172.21.15.105:6838/500348151 2024-05-27T07:31:49.883 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:31:49.890+0000 7f02d1ffb640 1 -- 172.21.15.52:0/3097825534 <== mds.0 v2:172.21.15.105:6838/500348151 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24041 (crc 0 0 0) 0x556f9385eb80 con 0x556f93507d80 2024-05-27T07:31:49.883 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:31:49.890+0000 7f02d1ffb640 10 client.16674 handle_command_reply: tid=0 2024-05-27T07:31:49.883 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:31:49.890+0000 7f02f7c1a740 1 client.16674 shutdown 2024-05-27T07:31:49.890 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:31:49.896+0000 7f02f7c1a740 1 -- 172.21.15.52:0/3097825534 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x556f935284d0 msgr2=0x556f938f6a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:31:49.890 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:31:49.896+0000 7f02f7c1a740 1 --2- 172.21.15.52:0/3097825534 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x556f935284d0 0x556f938f6a00 secure :-1 s=READY pgs=664 cs=0 l=1 rev1=1 crypto rx=0x7f02e400c5b0 tx=0x7f02e400ca70 comp rx=0 tx=0).stop 2024-05-27T07:31:49.894 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:31:49.901+0000 7f02f7c1a740 1 -- 172.21.15.52:0/3097825534 shutdown_connections 2024-05-27T07:31:49.894 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:31:49.901+0000 7f02f7c1a740 1 --2- 172.21.15.52:0/3097825534 >> [v2:172.21.15.105:6838/500348151,v1:172.21.15.105:6839/500348151] conn(0x556f93507d80 0x556f939073b0 crc :-1 s=READY pgs=119 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:31:49.894 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:31:49.901+0000 7f02f7c1a740 1 --2- 172.21.15.52:0/3097825534 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x556f935491b0 0x556f938f4940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:31:49.894 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:31:49.901+0000 7f02f7c1a740 1 --2- 172.21.15.52:0/3097825534 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x556f935284d0 0x556f938f6a00 unknown :-1 s=CLOSED pgs=664 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:31:49.894 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:31:49.901+0000 7f02f7c1a740 1 --2- 172.21.15.52:0/3097825534 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x556f935202f0 0x556f938f64c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:31:49.894 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:31:49.901+0000 7f02f7c1a740 1 -- 172.21.15.52:0/3097825534 >> 172.21.15.52:0/3097825534 conn(0x556f9353d300 msgr2=0x556f938ea200 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:31:49.894 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:31:49.901+0000 7f02d1ffb640 0 client.16674 ms_handle_reset on v2:172.21.15.105:6838/500348151 2024-05-27T07:31:49.895 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:31:49.903+0000 7f02f7c1a740 1 -- 172.21.15.52:0/3097825534 shutdown_connections 2024-05-27T07:31:49.898 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:31:49.906+0000 7f02f7c1a740 1 -- 172.21.15.52:0/3097825534 wait complete. 2024-05-27T07:31:49.898 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:31:49.906+0000 7f02f7c1a740 20 client.16674 trim_cache size 0 max 16384 2024-05-27T07:31:49.904 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:31:49.911+0000 7f46baff5b80 1 Processor -- start 2024-05-27T07:31:49.904 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:31:49.911+0000 7f46baff5b80 1 -- start start 2024-05-27T07:31:49.908 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:31:49.912+0000 7f46baff5b80 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x559611afde40 0x559611c84330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:31:49.908 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:31:49.912+0000 7f46baff5b80 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x559611c7bf80 0x559611c85c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:31:49.908 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:31:49.912+0000 7f46aa37c700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x559611afde40 0x559611c84330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:31:49.908 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:31:49.912+0000 7f46aa37c700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x559611afde40 0x559611c84330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.173:3300/0 says I am v2:172.21.15.52:39278/0 (socket says 172.21.15.52:39278) 2024-05-27T07:31:49.908 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:31:49.912+0000 7f46aa37c700 1 -- 172.21.15.52:0/2696059956 learned_addr learned my addr 172.21.15.52:0/2696059956 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-05-27T07:31:49.908 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:31:49.912+0000 7f46baff5b80 1 --2- 172.21.15.52:0/2696059956 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x559611c81c50 0x559611c878f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:31:49.908 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:31:49.913+0000 7f46a9b7b700 1 --2- 172.21.15.52:0/2696059956 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x559611c7bf80 0x559611c85c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:31:49.908 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:31:49.913+0000 7f46baff5b80 1 -- 172.21.15.52:0/2696059956 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x559611c98300 con 0x559611c81c50 2024-05-27T07:31:49.909 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:31:49.913+0000 7f46baff5b80 1 -- 172.21.15.52:0/2696059956 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f468800f1c0 con 0x559611c7bf80 2024-05-27T07:31:49.909 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:31:49.913+0000 7f46baff5b80 1 -- 172.21.15.52:0/2696059956 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f468800d5c0 con 0x559611afde40 2024-05-27T07:31:49.909 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:31:49.913+0000 7f46aab7d700 1 --2- 172.21.15.52:0/2696059956 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x559611c81c50 0x559611c878f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:31:49.909 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:31:49.914+0000 7f46a9b7b700 1 -- 172.21.15.52:0/2696059956 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x559611afde40 msgr2=0x559611c84330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:31:49.909 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:31:49.914+0000 7f46a9b7b700 1 --2- 172.21.15.52:0/2696059956 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x559611afde40 0x559611c84330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:31:49.909 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:31:49.914+0000 7f46a9b7b700 1 -- 172.21.15.52:0/2696059956 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x559611c81c50 msgr2=0x559611c878f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:31:49.909 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:31:49.914+0000 7f46a9b7b700 1 --2- 172.21.15.52:0/2696059956 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x559611c81c50 0x559611c878f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:31:49.909 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:31:49.914+0000 7f46a9b7b700 1 -- 172.21.15.52:0/2696059956 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x559611c83940 con 0x559611c7bf80 2024-05-27T07:31:49.909 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:31:49.915+0000 7f46a9b7b700 1 --2- 172.21.15.52:0/2696059956 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x559611c7bf80 0x559611c85c90 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x559611c98870 tx=0x7f4694004800 comp rx=0 tx=0).ready entity=mon.2 client_cookie=9c35309bdf424cde server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:31:49.911 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:31:49.917+0000 7f468ffff700 1 -- 172.21.15.52:0/2696059956 <== mon.2 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f469400f040 con 0x559611c7bf80 2024-05-27T07:31:49.911 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:31:49.917+0000 7f468ffff700 1 -- 172.21.15.52:0/2696059956 <== mon.2 v2:172.21.15.105:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f4694029070 con 0x559611c7bf80 2024-05-27T07:31:49.911 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:31:49.917+0000 7f468ffff700 1 -- 172.21.15.52:0/2696059956 <== mon.2 v2:172.21.15.105:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4694016b30 con 0x559611c7bf80 2024-05-27T07:31:49.911 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:31:49.918+0000 7f46baff5b80 1 -- 172.21.15.52:0/2696059956 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x559611c7bf80 msgr2=0x559611c85c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:31:49.912 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:31:49.918+0000 7f46baff5b80 1 --2- 172.21.15.52:0/2696059956 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x559611c7bf80 0x559611c85c90 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x559611c98870 tx=0x7f4694004800 comp rx=0 tx=0).stop 2024-05-27T07:31:49.916 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:31:49.923+0000 7f46baff5b80 1 -- 172.21.15.52:0/2696059956 shutdown_connections 2024-05-27T07:31:49.916 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:31:49.923+0000 7f46baff5b80 1 --2- 172.21.15.52:0/2696059956 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x559611c81c50 0x559611c878f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:31:49.916 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:31:49.923+0000 7f46baff5b80 1 --2- 172.21.15.52:0/2696059956 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x559611afde40 0x559611c84330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:31:49.916 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:31:49.923+0000 7f46baff5b80 1 --2- 172.21.15.52:0/2696059956 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x559611c7bf80 0x559611c85c90 unknown :-1 s=CLOSED pgs=187 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:31:49.916 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:31:49.923+0000 7f46baff5b80 1 -- 172.21.15.52:0/2696059956 >> 172.21.15.52:0/2696059956 conn(0x559611b58af0 msgr2=0x559611c80350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:31:49.928 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:31:49.934+0000 7f46baff5b80 1 -- 172.21.15.52:0/2696059956 shutdown_connections 2024-05-27T07:31:49.928 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:31:49.936+0000 7f46baff5b80 1 -- 172.21.15.52:0/2696059956 wait complete. 2024-05-27T07:31:49.930 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:31:49.936+0000 7f46baff5b80 1 -- start start 2024-05-27T07:31:49.930 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:31:49.937+0000 7f46baff5b80 1 Processor -- start 2024-05-27T07:31:49.931 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:31:49.937+0000 7f46baff5b80 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x559611afde40 0x559611c8dfb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:31:49.931 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:31:49.937+0000 7f46baff5b80 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x559611c7bf80 0x559611c9cec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:31:49.931 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:31:49.938+0000 7f46baff5b80 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x559611c81c50 0x559611c9d400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:31:49.931 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:31:49.938+0000 7f46baff5b80 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x7f468800dc90 con 0x559611c81c50 2024-05-27T07:31:49.931 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:31:49.938+0000 7f46baff5b80 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f468800eaf0 con 0x559611c7bf80 2024-05-27T07:31:49.931 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:31:49.938+0000 7f46baff5b80 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f468800e360 con 0x559611afde40 2024-05-27T07:31:49.931 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:31:49.938+0000 7f46baff5b80 10 client.0 _ll_register_callbacks cb 0x559611b35920 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T07:31:49.934 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:31:49.941+0000 7f46a9b7b700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x559611c7bf80 0x559611c9cec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:31:49.934 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:31:49.941+0000 7f46aa37c700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x559611afde40 0x559611c8dfb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:31:49.934 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:31:49.941+0000 7f46a9b7b700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x559611c7bf80 0x559611c9cec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.52:34624/0 (socket says 172.21.15.52:34624) 2024-05-27T07:31:49.934 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:31:49.941+0000 7f46a9b7b700 1 -- 172.21.15.52:0/808371415 learned_addr learned my addr 172.21.15.52:0/808371415 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-05-27T07:31:49.940 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:31:49.948+0000 7f02f7c1a740 1 Processor -- start 2024-05-27T07:31:49.940 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:31:49.948+0000 7f02f7c1a740 1 -- start start 2024-05-27T07:31:49.940 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:31:49.942+0000 7f46aab7d700 1 --2- 172.21.15.52:0/808371415 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x559611c81c50 0x559611c9d400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:31:49.941 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:31:49.945+0000 7f46a9b7b700 1 -- 172.21.15.52:0/808371415 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x559611afde40 msgr2=0x559611c8dfb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:31:49.941 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:31:49.945+0000 7f46a9b7b700 1 --2- 172.21.15.52:0/808371415 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x559611afde40 0x559611c8dfb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:31:49.941 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:31:49.945+0000 7f46a9b7b700 1 -- 172.21.15.52:0/808371415 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x559611c81c50 msgr2=0x559611c9d400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:31:49.941 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:31:49.945+0000 7f46a9b7b700 1 --2- 172.21.15.52:0/808371415 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x559611c81c50 0x559611c9d400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:31:49.941 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:31:49.945+0000 7f46a9b7b700 1 -- 172.21.15.52:0/808371415 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({osdmap=0}) -- 0x559611c9d9c0 con 0x559611c7bf80 2024-05-27T07:31:49.941 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:31:49.947+0000 7f46a9b7b700 1 --2- 172.21.15.52:0/808371415 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x559611c7bf80 0x559611c9cec0 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7f4694016960 tx=0x7f4694016990 comp rx=0 tx=0).ready entity=mon.2 client_cookie=4ce6cfb957054d9a server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:31:49.941 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:31:49.948+0000 7f02f7c1a740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x556f935202f0 0x556f938e6c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:31:49.944 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:31:49.948+0000 7f02f7c1a740 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x556f935284d0 0x556f938ee020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:31:49.944 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:31:49.948+0000 7f02f7c1a740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x556f935491b0 0x556f938ee560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:31:49.944 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:31:49.948+0000 7f02f7c1a740 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x7f02cc010450 con 0x556f935284d0 2024-05-27T07:31:49.944 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:31:49.948+0000 7f02f7c1a740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f02cc00e850 con 0x556f935202f0 2024-05-27T07:31:49.944 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:31:49.949+0000 7f02ea59c640 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x556f935284d0 0x556f938ee020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:31:49.944 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:31:49.951+0000 7f02ea59c640 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x556f935284d0 0x556f938ee020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.52:58900/0 (socket says 172.21.15.52:58900) 2024-05-27T07:31:49.945 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:31:49.951+0000 7f02ea59c640 1 -- 172.21.15.52:0/2600302524 learned_addr learned my addr 172.21.15.52:0/2600302524 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-05-27T07:31:49.945 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:31:49.952+0000 7f02f7c1a740 1 -- 172.21.15.52:0/2600302524 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f02cc00fd80 con 0x556f935491b0 2024-05-27T07:31:49.945 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:31:49.950+0000 7f468dffb700 10 client.0 ms_handle_connect on v2:172.21.15.105:3300/0 2024-05-27T07:31:49.945 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:31:49.950+0000 7f468dffb700 1 -- 172.21.15.52:0/808371415 <== mon.2 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f469400f040 con 0x559611c7bf80 2024-05-27T07:31:49.945 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:31:49.950+0000 7f46baff5b80 1 -- 172.21.15.52:0/808371415 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f4694029070 con 0x559611c7bf80 2024-05-27T07:31:49.945 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:31:49.950+0000 7f468dffb700 1 -- 172.21.15.52:0/808371415 <== mon.2 v2:172.21.15.105:3300/0 2 ==== osd_map(116..116 src has 1..116) ==== 9379+0+0 (secure 0 0 0) 0x7f4694023670 con 0x559611c7bf80 2024-05-27T07:31:49.947 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:31:49.954+0000 7f02e9d9b640 1 --2- 172.21.15.52:0/2600302524 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x556f935202f0 0x556f938e6c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:31:49.947 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:31:49.954+0000 7f02e9d9b640 1 --2- 172.21.15.52:0/2600302524 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x556f935491b0 0x556f938ee560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:31:49.947 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:31:49.955+0000 7f02e9d9b640 1 -- 172.21.15.52:0/2600302524 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x556f935491b0 msgr2=0x556f938ee560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:31:49.947 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:31:49.955+0000 7f02e9d9b640 1 --2- 172.21.15.52:0/2600302524 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x556f935491b0 0x556f938ee560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:31:49.947 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:31:49.955+0000 7f02e9d9b640 1 -- 172.21.15.52:0/2600302524 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x556f935284d0 msgr2=0x556f938ee020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:31:49.947 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:31:49.955+0000 7f02e9d9b640 1 --2- 172.21.15.52:0/2600302524 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x556f935284d0 0x556f938ee020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:31:49.947 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:31:49.955+0000 7f02e9d9b640 1 -- 172.21.15.52:0/2600302524 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x556f938e7750 con 0x556f935202f0 2024-05-27T07:31:49.948 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:31:49.955+0000 7f02e9d9b640 1 --2- 172.21.15.52:0/2600302524 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x556f935202f0 0x556f938e6c50 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7f02dc02a450 tx=0x7f02dc000bc0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=2f608935181e7c37 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:31:49.949 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:31:49.955+0000 7f468dffb700 1 -- 172.21.15.52:0/808371415 <== mon.2 v2:172.21.15.105:3300/0 3 ==== mon_get_version_reply(handle=1 version=101) ==== 24+0+0 (secure 0 0 0) 0x7f4694016e30 con 0x559611c7bf80 2024-05-27T07:31:49.950 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:31:49.957+0000 7f46baff5b80 10 client.26324 fetch_fsmap learned FSMap version 101 2024-05-27T07:31:49.950 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:31:49.957+0000 7f46baff5b80 1 -- 172.21.15.52:0/808371415 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,fsmap=101,monmap=4+}) -- 0x559611d004b0 con 0x559611c7bf80 2024-05-27T07:31:49.951 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:31:49.957+0000 7f02d3fff640 1 -- 172.21.15.52:0/2600302524 <== mon.2 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f02dc035040 con 0x556f935202f0 2024-05-27T07:31:49.951 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:31:49.957+0000 7f02d3fff640 1 -- 172.21.15.52:0/2600302524 <== mon.2 v2:172.21.15.105:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f02dc041070 con 0x556f935202f0 2024-05-27T07:31:49.951 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:31:49.957+0000 7f02d3fff640 1 -- 172.21.15.52:0/2600302524 <== mon.2 v2:172.21.15.105:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f02dc004ab0 con 0x556f935202f0 2024-05-27T07:31:49.952 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:31:49.960+0000 7f02f7c1a740 1 -- 172.21.15.52:0/2600302524 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x556f935202f0 msgr2=0x556f938e6c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:31:49.952 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:31:49.960+0000 7f02f7c1a740 1 --2- 172.21.15.52:0/2600302524 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x556f935202f0 0x556f938e6c50 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7f02dc02a450 tx=0x7f02dc000bc0 comp rx=0 tx=0).stop 2024-05-27T07:31:49.952 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:31:49.960+0000 7f02ea59c640 1 --2- 172.21.15.52:0/2600302524 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x556f935284d0 0x556f938ee020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-27T07:31:49.954 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:31:49.960+0000 7f468dffb700 1 -- 172.21.15.52:0/808371415 <== mon.2 v2:172.21.15.105:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f4694029070 con 0x559611c7bf80 2024-05-27T07:31:49.954 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:31:49.960+0000 7f468dffb700 1 -- 172.21.15.52:0/808371415 <== mon.2 v2:172.21.15.105:3300/0 5 ==== fsmap(e 101) ==== 4203+0+0 (secure 0 0 0) 0x7f4694040080 con 0x559611c7bf80 2024-05-27T07:31:49.954 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:31:49.960+0000 7f46baff5b80 10 client.26324 fetch_fsmap finished waiting for FSMap version 101 2024-05-27T07:31:49.954 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:31:49.960+0000 7f46baff5b80 10 client.26324 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.i 2024-05-27T07:31:49.954 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:31:49.960+0000 7f46baff5b80 20 client.26324 populate_metadata read hostname 'smithi052' 2024-05-27T07:31:49.954 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:31:49.960+0000 7f46baff5b80 1 --2- 172.21.15.52:0/808371415 >> [v2:172.21.15.173:6838/564778293,v1:172.21.15.173:6839/564778293] conn(0x559611d00b40 0x559611d02f60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:31:49.954 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:31:49.960+0000 7f46baff5b80 4 client.26324 mds_command: new command op to 24475 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-05-27T07:31:49.954 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:31:49.960+0000 7f46baff5b80 1 -- 172.21.15.52:0/808371415 --> [v2:172.21.15.173:6838/564778293,v1:172.21.15.173:6839/564778293] -- command(tid 0: {"prefix": "damage ls"}) -- 0x559611c479f0 con 0x559611d00b40 2024-05-27T07:31:49.954 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:31:49.960+0000 7f46aa37c700 1 --2- 172.21.15.52:0/808371415 >> [v2:172.21.15.173:6838/564778293,v1:172.21.15.173:6839/564778293] conn(0x559611d00b40 0x559611d02f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:31:49.959 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:31:49.967+0000 7f02f7c1a740 1 -- 172.21.15.52:0/2600302524 shutdown_connections 2024-05-27T07:31:49.959 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:31:49.967+0000 7f02f7c1a740 1 --2- 172.21.15.52:0/2600302524 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x556f935491b0 0x556f938ee560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:31:49.960 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:31:49.967+0000 7f02f7c1a740 1 --2- 172.21.15.52:0/2600302524 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x556f935284d0 0x556f938ee020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:31:49.960 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:31:49.967+0000 7f02f7c1a740 1 --2- 172.21.15.52:0/2600302524 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x556f935202f0 0x556f938e6c50 unknown :-1 s=CLOSED pgs=189 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:31:49.960 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:31:49.967+0000 7f02f7c1a740 1 -- 172.21.15.52:0/2600302524 >> 172.21.15.52:0/2600302524 conn(0x556f9353d300 msgr2=0x556f938e6540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:31:49.960 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:31:49.965+0000 7f46aa37c700 1 --2- 172.21.15.52:0/808371415 >> [v2:172.21.15.173:6838/564778293,v1:172.21.15.173:6839/564778293] conn(0x559611d00b40 0x559611d02f60 crc :-1 s=READY pgs=46 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=56e956b406d28bc7 server_cookie=d3d5e3b22f5b513b in_seq=0 out_seq=0 2024-05-27T07:31:49.964 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:31:49.969+0000 7f468dffb700 10 client.26324 ms_handle_connect on v2:172.21.15.173:6838/564778293 2024-05-27T07:31:49.964 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:31:49.969+0000 7f468dffb700 1 -- 172.21.15.52:0/808371415 <== mds.2 v2:172.21.15.173:6838/564778293 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x559611c479f0 con 0x559611d00b40 2024-05-27T07:31:49.965 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:31:49.969+0000 7f468dffb700 10 client.26324 handle_command_reply: tid=0 2024-05-27T07:31:49.965 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:31:49.969+0000 7f46baff5b80 1 client.26324 shutdown 2024-05-27T07:31:49.966 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:31:49.973+0000 7f46baff5b80 1 -- 172.21.15.52:0/808371415 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x559611c7bf80 msgr2=0x559611c9cec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:31:49.966 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:31:49.974+0000 7f46baff5b80 1 --2- 172.21.15.52:0/808371415 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x559611c7bf80 0x559611c9cec0 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7f4694016960 tx=0x7f4694016990 comp rx=0 tx=0).stop 2024-05-27T07:31:49.968 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:31:49.975+0000 7f46baff5b80 1 -- 172.21.15.52:0/808371415 shutdown_connections 2024-05-27T07:31:49.969 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:31:49.975+0000 7f46baff5b80 1 --2- 172.21.15.52:0/808371415 >> [v2:172.21.15.173:6838/564778293,v1:172.21.15.173:6839/564778293] conn(0x559611d00b40 0x559611d02f60 crc :-1 s=READY pgs=46 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:31:49.969 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:31:49.975+0000 7f46baff5b80 1 --2- 172.21.15.52:0/808371415 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x559611c81c50 0x559611c9d400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:31:49.969 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:31:49.975+0000 7f46baff5b80 1 --2- 172.21.15.52:0/808371415 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x559611afde40 0x559611c8dfb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:31:49.969 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:31:49.975+0000 7f46baff5b80 1 --2- 172.21.15.52:0/808371415 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x559611c7bf80 0x559611c9cec0 unknown :-1 s=CLOSED pgs=188 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:31:49.969 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:31:49.975+0000 7f46baff5b80 1 -- 172.21.15.52:0/808371415 >> 172.21.15.52:0/808371415 conn(0x559611b58af0 msgr2=0x559611c7f7a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:31:49.969 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:31:49.976+0000 7f468dffb700 0 client.26324 ms_handle_reset on v2:172.21.15.173:6838/564778293 2024-05-27T07:31:49.969 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:31:49.977+0000 7f02f7c1a740 1 -- 172.21.15.52:0/2600302524 shutdown_connections 2024-05-27T07:31:49.969 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:31:49.977+0000 7f02f7c1a740 1 -- 172.21.15.52:0/2600302524 wait complete. 2024-05-27T07:31:49.970 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:31:49.977+0000 7f02f7c1a740 1 -- start start 2024-05-27T07:31:49.970 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:31:49.978+0000 7f02f7c1a740 1 Processor -- start 2024-05-27T07:31:49.971 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:31:49.978+0000 7f02f7c1a740 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x556f935202f0 0x556f938fcbf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:31:49.971 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:31:49.978+0000 7f02f7c1a740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x556f935284d0 0x556f938fa000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:31:49.974 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:31:49.979+0000 7f02f7c1a740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x556f935491b0 0x556f938fa540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:31:49.974 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:31:49.979+0000 7f02f7c1a740 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x7f02cc00f5f0 con 0x556f935202f0 2024-05-27T07:31:49.974 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:31:49.979+0000 7f02f7c1a740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f02cc00ef20 con 0x556f935284d0 2024-05-27T07:31:49.974 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:31:49.979+0000 7f02f7c1a740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x556f938f2e90 con 0x556f935491b0 2024-05-27T07:31:49.975 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:31:49.979+0000 7f02ea59c640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x556f935284d0 0x556f938fa000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:31:49.975 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:31:49.979+0000 7f02ea59c640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x556f935284d0 0x556f938fa000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.52:34648/0 (socket says 172.21.15.52:34648) 2024-05-27T07:31:49.975 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:31:49.979+0000 7f02ea59c640 1 -- 172.21.15.52:0/1478820666 learned_addr learned my addr 172.21.15.52:0/1478820666 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-05-27T07:31:49.975 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:31:49.979+0000 7f02f7c1a740 10 client.0 _ll_register_callbacks cb 0x7f02e4007380 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T07:31:49.975 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:31:49.980+0000 7f02e9d9b640 1 --2- 172.21.15.52:0/1478820666 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x556f935202f0 0x556f938fcbf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:31:49.975 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:31:49.981+0000 7f02e9d9b640 1 --2- 172.21.15.52:0/1478820666 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x556f935491b0 0x556f938fa540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:31:49.975 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:31:49.981+0000 7f02e9d9b640 1 -- 172.21.15.52:0/1478820666 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x556f935284d0 msgr2=0x556f938fa000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:31:49.975 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:31:49.981+0000 7f02e9d9b640 1 --2- 172.21.15.52:0/1478820666 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x556f935284d0 0x556f938fa000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:31:49.975 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:31:49.981+0000 7f02e9d9b640 1 -- 172.21.15.52:0/1478820666 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x556f935202f0 msgr2=0x556f938fcbf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:31:49.975 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:31:49.981+0000 7f02e9d9b640 1 --2- 172.21.15.52:0/1478820666 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x556f935202f0 0x556f938fcbf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:31:49.975 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:31:49.981+0000 7f02e9d9b640 1 -- 172.21.15.52:0/1478820666 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({osdmap=0}) -- 0x556f938faa80 con 0x556f935491b0 2024-05-27T07:31:49.975 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:31:49.982+0000 7f02e9d9b640 1 --2- 172.21.15.52:0/1478820666 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x556f935491b0 0x556f938fa540 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x556f93904f90 tx=0x7f02dc0035f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=e1e5b20e56865be5 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:31:49.976 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:31:49.982+0000 7f02d1ffb640 10 client.0 ms_handle_connect on v2:172.21.15.173:3300/0 2024-05-27T07:31:49.976 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:31:49.980+0000 7f46baff5b80 1 -- 172.21.15.52:0/808371415 shutdown_connections 2024-05-27T07:31:49.976 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:31:49.982+0000 7f02d1ffb640 1 -- 172.21.15.52:0/1478820666 <== mon.1 v2:172.21.15.173:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f02dc035070 con 0x556f935491b0 2024-05-27T07:31:49.976 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:31:49.982+0000 7f02d1ffb640 1 -- 172.21.15.52:0/1478820666 <== mon.1 v2:172.21.15.173:3300/0 2 ==== osd_map(116..116 src has 1..116) ==== 9379+0+0 (secure 0 0 0) 0x7f02dc004b00 con 0x556f935491b0 2024-05-27T07:31:49.977 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:31:49.984+0000 7f46baff5b80 1 -- 172.21.15.52:0/808371415 wait complete. 2024-05-27T07:31:49.977 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:31:49.984+0000 7f46baff5b80 20 client.26324 trim_cache size 0 max 16384 2024-05-27T07:31:49.977 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:31:49.984+0000 7f02f7c1a740 1 -- 172.21.15.52:0/1478820666 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f02dc041070 con 0x556f935491b0 2024-05-27T07:31:49.977 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:31:49.985+0000 7f02d1ffb640 1 -- 172.21.15.52:0/1478820666 <== mon.1 v2:172.21.15.173:3300/0 3 ==== mon_get_version_reply(handle=1 version=101) ==== 24+0+0 (secure 0 0 0) 0x7f02dc068050 con 0x556f935491b0 2024-05-27T07:31:49.978 INFO:teuthology.orchestra.run.smithi052.stdout:[] 2024-05-27T07:31:49.983 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:31:49.990+0000 7f46ac5e0700 1 -- 172.21.15.52:0/3544192187 >> [v2:172.21.15.52:6800/1822318384,v1:172.21.15.52:6801/1822318384] conn(0x7f46900b2ab0 msgr2=0x7f46900b4fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:31:49.983 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:31:49.990+0000 7f46ac5e0700 1 --2- 172.21.15.52:0/3544192187 >> [v2:172.21.15.52:6800/1822318384,v1:172.21.15.52:6801/1822318384] conn(0x7f46900b2ab0 0x7f46900b4fc0 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x559611c780a0 tx=0x7f46940091f0 comp rx=0 tx=0).stop 2024-05-27T07:31:49.983 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:31:49.990+0000 7f46ac5e0700 1 -- 172.21.15.52:0/3544192187 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f46a4109860 msgr2=0x7f46a4113700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:31:49.983 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:31:49.990+0000 7f46ac5e0700 1 --2- 172.21.15.52:0/3544192187 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f46a4109860 0x7f46a4113700 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7f469c00b970 tx=0x7f469c00bc80 comp rx=0 tx=0).stop 2024-05-27T07:31:49.983 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:31:49.990+0000 7f46ac5e0700 1 -- 172.21.15.52:0/3544192187 shutdown_connections 2024-05-27T07:31:49.983 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:31:49.990+0000 7f46ac5e0700 1 --2- 172.21.15.52:0/3544192187 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f46a4109860 0x7f46a4113700 unknown :-1 s=CLOSED pgs=185 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:31:49.983 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:31:49.990+0000 7f46ac5e0700 1 --2- 172.21.15.52:0/3544192187 >> [v2:172.21.15.52:6800/1822318384,v1:172.21.15.52:6801/1822318384] conn(0x7f46900b2ab0 0x7f46900b4fc0 unknown :-1 s=CLOSED pgs=151 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:31:49.983 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:31:49.990+0000 7f46ac5e0700 1 --2- 172.21.15.52:0/3544192187 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f46a41045e0 0x7f46a4119f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:31:49.983 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:31:49.990+0000 7f46ac5e0700 1 --2- 172.21.15.52:0/3544192187 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f46a4108f10 0x7f46a41130b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:31:49.983 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:31:49.990+0000 7f46ac5e0700 1 -- 172.21.15.52:0/3544192187 >> 172.21.15.52:0/3544192187 conn(0x7f46a4100030 msgr2=0x7f46a410a7c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:31:49.985 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:31:49.992+0000 7f02f7c1a740 10 client.26200 fetch_fsmap learned FSMap version 101 2024-05-27T07:31:49.985 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:31:49.992+0000 7f02f7c1a740 1 -- 172.21.15.52:0/1478820666 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({config=0+,fsmap=101,monmap=4+}) -- 0x556f9396f650 con 0x556f935491b0 2024-05-27T07:31:49.985 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:31:49.992+0000 7f46ac5e0700 1 -- 172.21.15.52:0/3544192187 shutdown_connections 2024-05-27T07:31:49.986 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:31:49.993+0000 7f02d1ffb640 1 -- 172.21.15.52:0/1478820666 <== mon.1 v2:172.21.15.173:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f02dc041070 con 0x556f935491b0 2024-05-27T07:31:49.986 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:31:49.993+0000 7f02d1ffb640 1 -- 172.21.15.52:0/1478820666 <== mon.1 v2:172.21.15.173:3300/0 5 ==== fsmap(e 101) ==== 4203+0+0 (secure 0 0 0) 0x7f02dc003db0 con 0x556f935491b0 2024-05-27T07:31:49.986 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:31:49.993+0000 7f02f7c1a740 10 client.26200 fetch_fsmap finished waiting for FSMap version 101 2024-05-27T07:31:49.986 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:31:49.993+0000 7f02f7c1a740 10 client.26200 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-05-27T07:31:49.986 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:31:49.993+0000 7f02f7c1a740 20 client.26200 populate_metadata read hostname 'smithi052' 2024-05-27T07:31:49.987 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:31:49.993+0000 7f02f7c1a740 1 --2- 172.21.15.52:0/1478820666 >> [v2:172.21.15.105:6838/500348151,v1:172.21.15.105:6839/500348151] conn(0x556f934dd1c0 0x556f93971c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:31:49.987 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:31:49.994+0000 7f46ac5e0700 1 -- 172.21.15.52:0/3544192187 wait complete. 2024-05-27T07:31:49.987 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:31:49.994+0000 7f02f7c1a740 4 client.26200 mds_command: new command op to 24527 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-05-27T07:31:49.987 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:31:49.994+0000 7f02ea59c640 1 --2- 172.21.15.52:0/1478820666 >> [v2:172.21.15.105:6838/500348151,v1:172.21.15.105:6839/500348151] conn(0x556f934dd1c0 0x556f93971c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:31:49.988 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:31:49.994+0000 7f02f7c1a740 1 -- 172.21.15.52:0/1478820666 --> [v2:172.21.15.105:6838/500348151,v1:172.21.15.105:6839/500348151] -- command(tid 0: {"prefix": "flush journal"}) -- 0x556f93858360 con 0x556f934dd1c0 2024-05-27T07:31:49.988 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:31:49.994+0000 7f02ea59c640 1 --2- 172.21.15.52:0/1478820666 >> [v2:172.21.15.105:6838/500348151,v1:172.21.15.105:6839/500348151] conn(0x556f934dd1c0 0x556f93971c30 crc :-1 s=READY pgs=120 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=61c3cbd7f3e9df51 server_cookie=60de898b60b78ec9 in_seq=0 out_seq=0 2024-05-27T07:31:49.988 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:31:49.995+0000 7f02d1ffb640 10 client.26200 ms_handle_connect on v2:172.21.15.105:6838/500348151 2024-05-27T07:31:50.231 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:31:50 smithi052 ceph-mon[31955]: pgmap v204: 129 pgs: 129 active+clean; 516 MiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 610 KiB/s rd, 3.0 MiB/s wr, 460 op/s 2024-05-27T07:31:50.604 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:31:50 smithi173 ceph-mon[37000]: pgmap v204: 129 pgs: 129 active+clean; 516 MiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 610 KiB/s rd, 3.0 MiB/s wr, 460 op/s 2024-05-27T07:31:50.692 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:31:50 smithi105 ceph-mon[35977]: pgmap v204: 129 pgs: 129 active+clean; 516 MiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 610 KiB/s rd, 3.0 MiB/s wr, 460 op/s 2024-05-27T07:31:50.980 DEBUG:teuthology.orchestra.run.smithi052:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:38f9d900c83c63799fbdbe61acc9a11b0d3554a6 shell --fsid 73df2636-1bf8-11ef-bc9b-c7b262605968 -- ceph tell mds.1:1 damage ls 2024-05-27T07:31:51.996 INFO:teuthology.orchestra.run.smithi052.stderr:Inferring config /var/lib/ceph/73df2636-1bf8-11ef-bc9b-c7b262605968/mon.a/config 2024-05-27T07:31:52.404 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:31:52 smithi052 ceph-mon[31955]: pgmap v205: 129 pgs: 129 active+clean; 544 MiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 586 KiB/s rd, 4.5 MiB/s wr, 436 op/s 2024-05-27T07:31:52.604 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:31:52 smithi173 ceph-mon[37000]: pgmap v205: 129 pgs: 129 active+clean; 544 MiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 586 KiB/s rd, 4.5 MiB/s wr, 436 op/s 2024-05-27T07:31:52.692 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:31:52 smithi105 ceph-mon[35977]: pgmap v205: 129 pgs: 129 active+clean; 544 MiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 586 KiB/s rd, 4.5 MiB/s wr, 436 op/s 2024-05-27T07:31:54.564 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:31:54 smithi052 ceph-mon[31955]: pgmap v206: 129 pgs: 129 active+clean; 548 MiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 354 KiB/s rd, 4.9 MiB/s wr, 410 op/s 2024-05-27T07:31:54.604 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:31:54 smithi173 ceph-mon[37000]: pgmap v206: 129 pgs: 129 active+clean; 548 MiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 354 KiB/s rd, 4.9 MiB/s wr, 410 op/s 2024-05-27T07:31:54.692 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:31:54 smithi105 ceph-mon[35977]: pgmap v206: 129 pgs: 129 active+clean; 548 MiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 354 KiB/s rd, 4.9 MiB/s wr, 410 op/s 2024-05-27T07:31:54.696 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:31:54.702+0000 7fb8220e2700 1 -- 172.21.15.52:0/3233087502 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fb81c102780 msgr2=0x7fb81c102b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:31:54.696 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:31:54.702+0000 7fb8220e2700 1 --2- 172.21.15.52:0/3233087502 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fb81c102780 0x7fb81c102b90 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7fb810009990 tx=0x7fb810009ca0 comp rx=0 tx=0).stop 2024-05-27T07:31:54.696 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:31:54.702+0000 7fb8220e2700 1 -- 172.21.15.52:0/3233087502 shutdown_connections 2024-05-27T07:31:54.696 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:31:54.702+0000 7fb8220e2700 1 --2- 172.21.15.52:0/3233087502 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fb81c1030d0 0x7fb81c111160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:31:54.696 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:31:54.702+0000 7fb8220e2700 1 --2- 172.21.15.52:0/3233087502 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fb81c1092f0 0x7fb81c109700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:31:54.696 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:31:54.702+0000 7fb8220e2700 1 --2- 172.21.15.52:0/3233087502 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fb81c102780 0x7fb81c102b90 unknown :-1 s=CLOSED pgs=190 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:31:54.696 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:31:54.702+0000 7fb8220e2700 1 -- 172.21.15.52:0/3233087502 >> 172.21.15.52:0/3233087502 conn(0x7fb81c0fe510 msgr2=0x7fb81c100960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:31:54.696 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:31:54.702+0000 7fb8220e2700 1 -- 172.21.15.52:0/3233087502 shutdown_connections 2024-05-27T07:31:54.696 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:31:54.702+0000 7fb8220e2700 1 -- 172.21.15.52:0/3233087502 wait complete. 2024-05-27T07:31:54.697 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:31:54.703+0000 7fb8220e2700 1 Processor -- start 2024-05-27T07:31:54.699 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:31:54.704+0000 7fb8220e2700 1 -- start start 2024-05-27T07:31:54.699 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:31:54.704+0000 7fb8220e2700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fb81c102780 0x7fb81c19af50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:31:54.700 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:31:54.704+0000 7fb8220e2700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fb81c1030d0 0x7fb81c19b4f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:31:54.700 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:31:54.704+0000 7fb8220e2700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fb81c1092f0 0x7fb81c19f780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:31:54.700 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:31:54.704+0000 7fb8220e2700 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x7fb81c1145e0 con 0x7fb81c102780 2024-05-27T07:31:54.700 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:31:54.704+0000 7fb81b7fe700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fb81c102780 0x7fb81c19af50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:31:54.700 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:31:54.704+0000 7fb81bfff700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fb81c1092f0 0x7fb81c19f780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:31:54.700 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:31:54.705+0000 7fb81bfff700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fb81c1092f0 0x7fb81c19f780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.52:35866/0 (socket says 172.21.15.52:35866) 2024-05-27T07:31:54.700 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:31:54.705+0000 7fb81bfff700 1 -- 172.21.15.52:0/691031440 learned_addr learned my addr 172.21.15.52:0/691031440 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-05-27T07:31:54.700 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:31:54.705+0000 7fb81affd700 1 --2- 172.21.15.52:0/691031440 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fb81c1030d0 0x7fb81c19b4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:31:54.700 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:31:54.705+0000 7fb8220e2700 1 -- 172.21.15.52:0/691031440 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7fb81c114460 con 0x7fb81c1092f0 2024-05-27T07:31:54.700 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:31:54.705+0000 7fb8220e2700 1 -- 172.21.15.52:0/691031440 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7fb81c114760 con 0x7fb81c1030d0 2024-05-27T07:31:54.700 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:31:54.705+0000 7fb81bfff700 1 -- 172.21.15.52:0/691031440 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fb81c1030d0 msgr2=0x7fb81c19b4f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:31:54.701 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:31:54.705+0000 7fb81bfff700 1 --2- 172.21.15.52:0/691031440 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fb81c1030d0 0x7fb81c19b4f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:31:54.701 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:31:54.705+0000 7fb81bfff700 1 -- 172.21.15.52:0/691031440 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fb81c102780 msgr2=0x7fb81c19af50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:31:54.701 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:31:54.705+0000 7fb81bfff700 1 --2- 172.21.15.52:0/691031440 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fb81c102780 0x7fb81c19af50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:31:54.701 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:31:54.705+0000 7fb81bfff700 1 -- 172.21.15.52:0/691031440 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb81c19fe20 con 0x7fb81c1092f0 2024-05-27T07:31:54.701 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:31:54.705+0000 7fb81affd700 1 --2- 172.21.15.52:0/691031440 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fb81c1030d0 0x7fb81c19b4f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-27T07:31:54.701 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:31:54.705+0000 7fb81bfff700 1 --2- 172.21.15.52:0/691031440 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fb81c1092f0 0x7fb81c19f780 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7fb81400ef30 tx=0x7fb81400c3a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:31:54.701 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:31:54.706+0000 7fb818ff9700 1 -- 172.21.15.52:0/691031440 <== mon.2 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb8140092e0 con 0x7fb81c1092f0 2024-05-27T07:31:54.701 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:31:54.706+0000 7fb818ff9700 1 -- 172.21.15.52:0/691031440 <== mon.2 v2:172.21.15.105:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fb814004bb0 con 0x7fb81c1092f0 2024-05-27T07:31:54.701 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:31:54.706+0000 7fb818ff9700 1 -- 172.21.15.52:0/691031440 <== mon.2 v2:172.21.15.105:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb814004280 con 0x7fb81c1092f0 2024-05-27T07:31:54.701 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:31:54.706+0000 7fb8220e2700 1 -- 172.21.15.52:0/691031440 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb81c1a00b0 con 0x7fb81c1092f0 2024-05-27T07:31:54.701 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:31:54.706+0000 7fb8220e2700 1 -- 172.21.15.52:0/691031440 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb81c1a05a0 con 0x7fb81c1092f0 2024-05-27T07:31:54.702 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:31:54.710+0000 7fb818ff9700 1 -- 172.21.15.52:0/691031440 <== mon.2 v2:172.21.15.105:3300/0 4 ==== mgrmap(e 26) ==== 149783+0+0 (secure 0 0 0) 0x7fb81401b030 con 0x7fb81c1092f0 2024-05-27T07:31:54.704 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:31:54.711+0000 7fb830af7b80 1 Processor -- start 2024-05-27T07:31:54.704 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:31:54.711+0000 7fb830af7b80 1 -- start start 2024-05-27T07:31:54.704 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:31:54.711+0000 7fb830af7b80 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x564134fcde40 0x564135151290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:31:54.704 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:31:54.711+0000 7fb830af7b80 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x5641351517d0 0x564135155a70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:31:54.704 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:31:54.711+0000 7fb830af7b80 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x564135155fb0 0x5641351563c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:31:54.704 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:31:54.711+0000 7fb830af7b80 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x56413507ad70 con 0x5641351517d0 2024-05-27T07:31:54.705 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:31:54.711+0000 7fb830af7b80 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x56413506a910 con 0x564135155fb0 2024-05-27T07:31:54.705 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:31:54.711+0000 7fb830af7b80 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x56413506e950 con 0x564134fcde40 2024-05-27T07:31:54.705 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:31:54.711+0000 7fb818ff9700 1 --2- 172.21.15.52:0/691031440 >> [v2:172.21.15.52:6800/1822318384,v1:172.21.15.52:6801/1822318384] conn(0x7fb8040b2a70 0x7fb8040b4f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:31:54.705 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:31:54.711+0000 7fb81b7fe700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x5641351517d0 0x564135155a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:31:54.705 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:31:54.711+0000 7fb81b7fe700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x5641351517d0 0x564135155a70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.52:39120/0 (socket says 172.21.15.52:39120) 2024-05-27T07:31:54.705 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:31:54.711+0000 7fb81b7fe700 1 -- 172.21.15.52:0/958777746 learned_addr learned my addr 172.21.15.52:0/958777746 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-05-27T07:31:54.719 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:31:54.711+0000 7fb81affd700 1 --2- 172.21.15.52:0/958777746 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x564134fcde40 0x564135151290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:31:54.719 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:31:54.711+0000 7fb81affd700 1 --2- 172.21.15.52:0/958777746 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x564135155fb0 0x5641351563c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:31:54.719 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:31:54.712+0000 7fb81bfff700 1 --2- 172.21.15.52:0/691031440 >> [v2:172.21.15.52:6800/1822318384,v1:172.21.15.52:6801/1822318384] conn(0x7fb8040b2a70 0x7fb8040b4f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:31:54.719 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:31:54.712+0000 7fb81affd700 1 -- 172.21.15.52:0/958777746 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x564135155fb0 msgr2=0x5641351563c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:31:54.719 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:31:54.712+0000 7fb81affd700 1 --2- 172.21.15.52:0/958777746 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x564135155fb0 0x5641351563c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:31:54.720 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:31:54.712+0000 7fb81affd700 1 -- 172.21.15.52:0/958777746 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x5641351517d0 msgr2=0x564135155a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:31:54.720 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:31:54.712+0000 7fb81affd700 1 --2- 172.21.15.52:0/958777746 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x5641351517d0 0x564135155a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:31:54.720 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:31:54.712+0000 7fb81affd700 1 -- 172.21.15.52:0/958777746 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x564135013fb0 con 0x564134fcde40 2024-05-27T07:31:54.720 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:31:54.712+0000 7fb81affd700 1 --2- 172.21.15.52:0/958777746 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x564134fcde40 0x564135151290 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7fb81000b2e0 tx=0x7fb81000b610 comp rx=0 tx=0).ready entity=mon.1 client_cookie=1d60506fc700f029 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:31:54.720 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:31:54.712+0000 7fb801ffb700 1 -- 172.21.15.52:0/958777746 <== mon.1 v2:172.21.15.173:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb8100299e0 con 0x564134fcde40 2024-05-27T07:31:54.721 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:31:54.712+0000 7fb818ff9700 1 -- 172.21.15.52:0/691031440 <== mon.2 v2:172.21.15.105:3300/0 5 ==== osd_map(116..116 src has 1..116) ==== 9379+0+0 (secure 0 0 0) 0x7fb8140e2400 con 0x7fb81c1092f0 2024-05-27T07:31:54.721 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:31:54.712+0000 7fb801ffb700 1 -- 172.21.15.52:0/958777746 <== mon.1 v2:172.21.15.173:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fb8100055d0 con 0x564134fcde40 2024-05-27T07:31:54.721 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:31:54.712+0000 7fb801ffb700 1 -- 172.21.15.52:0/958777746 <== mon.1 v2:172.21.15.173:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb810003680 con 0x564134fcde40 2024-05-27T07:31:54.721 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:31:54.713+0000 7fb830af7b80 1 -- 172.21.15.52:0/958777746 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x564134fcde40 msgr2=0x564135151290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:31:54.721 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:31:54.713+0000 7fb81bfff700 1 --2- 172.21.15.52:0/691031440 >> [v2:172.21.15.52:6800/1822318384,v1:172.21.15.52:6801/1822318384] conn(0x7fb8040b2a70 0x7fb8040b4f80 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7fb8140a6680 tx=0x7fb814009000 comp rx=0 tx=0).ready entity=mgr.15333 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:31:54.721 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:31:54.713+0000 7fb830af7b80 1 --2- 172.21.15.52:0/958777746 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x564134fcde40 0x564135151290 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7fb81000b2e0 tx=0x7fb81000b610 comp rx=0 tx=0).stop 2024-05-27T07:31:54.722 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:31:54.713+0000 7fb830af7b80 1 -- 172.21.15.52:0/958777746 shutdown_connections 2024-05-27T07:31:54.722 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:31:54.714+0000 7fb830af7b80 1 --2- 172.21.15.52:0/958777746 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x564135155fb0 0x5641351563c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:31:54.722 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:31:54.714+0000 7fb830af7b80 1 --2- 172.21.15.52:0/958777746 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x564134fcde40 0x564135151290 unknown :-1 s=CLOSED pgs=152 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:31:54.722 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:31:54.714+0000 7fb830af7b80 1 --2- 172.21.15.52:0/958777746 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x5641351517d0 0x564135155a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:31:54.722 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:31:54.714+0000 7fb830af7b80 1 -- 172.21.15.52:0/958777746 >> 172.21.15.52:0/958777746 conn(0x564134fd0770 msgr2=0x564135147d90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:31:54.722 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:31:54.714+0000 7fb830af7b80 1 -- 172.21.15.52:0/958777746 shutdown_connections 2024-05-27T07:31:54.722 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:31:54.714+0000 7fb830af7b80 1 -- 172.21.15.52:0/958777746 wait complete. 2024-05-27T07:31:54.723 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:31:54.714+0000 7fb830af7b80 1 -- start start 2024-05-27T07:31:54.723 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:31:54.714+0000 7fb830af7b80 1 Processor -- start 2024-05-27T07:31:54.723 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:31:54.715+0000 7fb830af7b80 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x564134fcde40 0x564135156db0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:31:54.723 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:31:54.715+0000 7fb830af7b80 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x5641351517d0 0x5641351572f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:31:54.723 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:31:54.715+0000 7fb830af7b80 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x564135155fb0 0x564135157830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:31:54.723 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:31:54.715+0000 7fb830af7b80 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x564134f70fa0 con 0x5641351517d0 2024-05-27T07:31:54.723 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:31:54.715+0000 7fb830af7b80 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x564134fd2da0 con 0x564134fcde40 2024-05-27T07:31:54.723 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:31:54.715+0000 7fb830af7b80 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x56413514cd60 con 0x564135155fb0 2024-05-27T07:31:54.723 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:31:54.715+0000 7fb830af7b80 10 client.0 _ll_register_callbacks cb 0x5641350015b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T07:31:54.723 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:31:54.715+0000 7fb81affd700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x564134fcde40 0x564135156db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:31:54.724 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:31:54.715+0000 7fb81affd700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x564134fcde40 0x564135156db0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.52:35876/0 (socket says 172.21.15.52:35876) 2024-05-27T07:31:54.724 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:31:54.715+0000 7fb81affd700 1 -- 172.21.15.52:0/742337642 learned_addr learned my addr 172.21.15.52:0/742337642 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-05-27T07:31:54.724 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:31:54.715+0000 7fb81affd700 1 --2- 172.21.15.52:0/742337642 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x564135155fb0 0x564135157830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:31:54.724 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:31:54.716+0000 7fb81b7fe700 1 --2- 172.21.15.52:0/742337642 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x5641351517d0 0x5641351572f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:31:54.724 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:31:54.716+0000 7fb81affd700 1 -- 172.21.15.52:0/742337642 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x564135155fb0 msgr2=0x564135157830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:31:54.724 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:31:54.716+0000 7fb81affd700 1 --2- 172.21.15.52:0/742337642 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x564135155fb0 0x564135157830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:31:54.724 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:31:54.716+0000 7fb81affd700 1 -- 172.21.15.52:0/742337642 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x5641351517d0 msgr2=0x5641351572f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:31:54.724 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:31:54.716+0000 7fb81affd700 1 --2- 172.21.15.52:0/742337642 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x5641351517d0 0x5641351572f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:31:54.724 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:31:54.716+0000 7fb81affd700 1 -- 172.21.15.52:0/742337642 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({osdmap=0}) -- 0x564134fba8d0 con 0x564134fcde40 2024-05-27T07:31:54.724 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:31:54.716+0000 7fb81b7fe700 1 --2- 172.21.15.52:0/742337642 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x5641351517d0 0x5641351572f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-27T07:31:54.724 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:31:54.716+0000 7fb81affd700 1 --2- 172.21.15.52:0/742337642 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x564134fcde40 0x564135156db0 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7fb81002aa40 tx=0x7fb810009ca0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=98b7baa9a9a35b56 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:31:54.725 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:31:54.716+0000 7fb7faffd700 10 client.0 ms_handle_connect on v2:172.21.15.105:3300/0 2024-05-27T07:31:54.725 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:31:54.716+0000 7fb7faffd700 1 -- 172.21.15.52:0/742337642 <== mon.2 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb81000ba10 con 0x564134fcde40 2024-05-27T07:31:54.725 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:31:54.716+0000 7fb7faffd700 1 -- 172.21.15.52:0/742337642 <== mon.2 v2:172.21.15.105:3300/0 2 ==== osd_map(116..116 src has 1..116) ==== 9379+0+0 (secure 0 0 0) 0x7fb81000bbb0 con 0x564134fcde40 2024-05-27T07:31:54.725 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:31:54.716+0000 7fb830af7b80 1 -- 172.21.15.52:0/742337642 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fb8100055d0 con 0x564134fcde40 2024-05-27T07:31:54.725 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:31:54.717+0000 7fb7faffd700 1 -- 172.21.15.52:0/742337642 <== mon.2 v2:172.21.15.105:3300/0 3 ==== mon_get_version_reply(handle=1 version=101) ==== 24+0+0 (secure 0 0 0) 0x7fb81000bdb0 con 0x564134fcde40 2024-05-27T07:31:54.725 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:31:54.717+0000 7fb830af7b80 10 client.26342 fetch_fsmap learned FSMap version 101 2024-05-27T07:31:54.725 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:31:54.717+0000 7fb830af7b80 1 -- 172.21.15.52:0/742337642 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,fsmap=101,monmap=4+}) -- 0x564135024760 con 0x564134fcde40 2024-05-27T07:31:54.725 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:31:54.717+0000 7fb7faffd700 1 -- 172.21.15.52:0/742337642 <== mon.2 v2:172.21.15.105:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fb8100055d0 con 0x564134fcde40 2024-05-27T07:31:54.725 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:31:54.717+0000 7fb7faffd700 1 -- 172.21.15.52:0/742337642 <== mon.2 v2:172.21.15.105:3300/0 5 ==== fsmap(e 101) ==== 4203+0+0 (secure 0 0 0) 0x7fb81003a610 con 0x564134fcde40 2024-05-27T07:31:54.725 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:31:54.718+0000 7fb830af7b80 10 client.26342 fetch_fsmap finished waiting for FSMap version 101 2024-05-27T07:31:54.725 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:31:54.718+0000 7fb830af7b80 10 client.26342 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.h 2024-05-27T07:31:54.726 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:31:54.718+0000 7fb830af7b80 20 client.26342 populate_metadata read hostname 'smithi052' 2024-05-27T07:31:54.726 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:31:54.718+0000 7fb830af7b80 1 --2- 172.21.15.52:0/742337642 >> [v2:172.21.15.105:6836/191393149,v1:172.21.15.105:6837/191393149] conn(0x564135166960 0x564135168d80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:31:54.726 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:31:54.718+0000 7fb830af7b80 4 client.26342 mds_command: new command op to 24497 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-27T07:31:54.726 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:31:54.718+0000 7fb830af7b80 1 -- 172.21.15.52:0/742337642 --> [v2:172.21.15.105:6836/191393149,v1:172.21.15.105:6837/191393149] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x56413507ccc0 con 0x564135166960 2024-05-27T07:31:54.726 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:31:54.718+0000 7fb81b7fe700 1 --2- 172.21.15.52:0/742337642 >> [v2:172.21.15.105:6836/191393149,v1:172.21.15.105:6837/191393149] conn(0x564135166960 0x564135168d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:31:54.726 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:31:54.718+0000 7fb7faffd700 10 client.26342 ms_handle_connect on v2:172.21.15.105:6836/191393149 2024-05-27T07:31:54.726 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:31:54.718+0000 7fb81b7fe700 1 --2- 172.21.15.52:0/742337642 >> [v2:172.21.15.105:6836/191393149,v1:172.21.15.105:6837/191393149] conn(0x564135166960 0x564135168d80 crc :-1 s=READY pgs=45 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=95289395e142cfe7 server_cookie=b9560d131b938d70 in_seq=0 out_seq=0 2024-05-27T07:31:54.726 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:31:54.720+0000 7fb7faffd700 1 -- 172.21.15.52:0/742337642 <== mds.1 v2:172.21.15.105:6836/191393149 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24041 (crc 0 0 0) 0x56413507ccc0 con 0x564135166960 2024-05-27T07:31:54.726 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:31:54.720+0000 7fb7faffd700 10 client.26342 handle_command_reply: tid=0 2024-05-27T07:31:54.726 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:31:54.720+0000 7fb830af7b80 1 client.26342 shutdown 2024-05-27T07:31:54.727 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:31:54.720+0000 7fb830af7b80 1 -- 172.21.15.52:0/742337642 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x564134fcde40 msgr2=0x564135156db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:31:54.727 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:31:54.720+0000 7fb830af7b80 1 --2- 172.21.15.52:0/742337642 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x564134fcde40 0x564135156db0 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7fb81002aa40 tx=0x7fb810009ca0 comp rx=0 tx=0).stop 2024-05-27T07:31:54.727 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:31:54.720+0000 7fb830af7b80 1 -- 172.21.15.52:0/742337642 shutdown_connections 2024-05-27T07:31:54.727 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:31:54.720+0000 7fb830af7b80 1 --2- 172.21.15.52:0/742337642 >> [v2:172.21.15.105:6836/191393149,v1:172.21.15.105:6837/191393149] conn(0x564135166960 0x564135168d80 crc :-1 s=READY pgs=45 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:31:54.727 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:31:54.720+0000 7fb830af7b80 1 --2- 172.21.15.52:0/742337642 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x564135155fb0 0x564135157830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:31:54.727 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:31:54.720+0000 7fb830af7b80 1 --2- 172.21.15.52:0/742337642 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x564134fcde40 0x564135156db0 unknown :-1 s=CLOSED pgs=192 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:31:54.727 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:31:54.720+0000 7fb830af7b80 1 --2- 172.21.15.52:0/742337642 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x5641351517d0 0x5641351572f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:31:54.727 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:31:54.720+0000 7fb830af7b80 1 -- 172.21.15.52:0/742337642 >> 172.21.15.52:0/742337642 conn(0x564134fd0770 msgr2=0x56413514bbd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:31:54.727 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:31:54.720+0000 7fb7faffd700 0 client.26342 ms_handle_reset on v2:172.21.15.105:6836/191393149 2024-05-27T07:31:54.727 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:31:54.720+0000 7fb830af7b80 1 -- 172.21.15.52:0/742337642 shutdown_connections 2024-05-27T07:31:54.727 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:31:54.720+0000 7fb830af7b80 1 -- 172.21.15.52:0/742337642 wait complete. 2024-05-27T07:31:54.728 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:31:54.720+0000 7fb830af7b80 20 client.26342 trim_cache size 0 max 16384 2024-05-27T07:31:54.729 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:31:54.736+0000 7fb830af7b80 1 Processor -- start 2024-05-27T07:31:54.729 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:31:54.736+0000 7fb830af7b80 1 -- start start 2024-05-27T07:31:54.730 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:31:54.737+0000 7fb830af7b80 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x564134fcde40 0x5641351504c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:31:54.730 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:31:54.737+0000 7fb830af7b80 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5641351517d0 0x564135155950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:31:54.731 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:31:54.737+0000 7fb830af7b80 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x564135155fb0 0x564135141c00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:31:54.731 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:31:54.737+0000 7fb830af7b80 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x7fb81c1ae7d0 con 0x564135155fb0 2024-05-27T07:31:54.731 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:31:54.737+0000 7fb830af7b80 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7fb81c1acbd0 con 0x564134fcde40 2024-05-27T07:31:54.731 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:31:54.737+0000 7fb830af7b80 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7fb81c1ae100 con 0x5641351517d0 2024-05-27T07:31:54.731 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:31:54.737+0000 7fb81affd700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x564134fcde40 0x5641351504c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:31:54.731 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:31:54.737+0000 7fb81affd700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x564135155fb0 0x564135141c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:31:54.731 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:31:54.737+0000 7fb81affd700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x564135155fb0 0x564135141c00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.52:39136/0 (socket says 172.21.15.52:39136) 2024-05-27T07:31:54.731 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:31:54.737+0000 7fb81affd700 1 -- 172.21.15.52:0/1823756262 learned_addr learned my addr 172.21.15.52:0/1823756262 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-05-27T07:31:54.731 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:31:54.737+0000 7fb81b7fe700 1 --2- 172.21.15.52:0/1823756262 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5641351517d0 0x564135155950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:31:54.731 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:31:54.737+0000 7fb81affd700 1 -- 172.21.15.52:0/1823756262 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5641351517d0 msgr2=0x564135155950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:31:54.731 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:31:54.737+0000 7fb81affd700 1 --2- 172.21.15.52:0/1823756262 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5641351517d0 0x564135155950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:31:54.732 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:31:54.737+0000 7fb81affd700 1 -- 172.21.15.52:0/1823756262 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x564134fcde40 msgr2=0x5641351504c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:31:54.732 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:31:54.737+0000 7fb81affd700 1 --2- 172.21.15.52:0/1823756262 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x564134fcde40 0x5641351504c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:31:54.732 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:31:54.737+0000 7fb81affd700 1 -- 172.21.15.52:0/1823756262 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56413515d660 con 0x564135155fb0 2024-05-27T07:31:54.732 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:31:54.738+0000 7fb81affd700 1 --2- 172.21.15.52:0/1823756262 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x564135155fb0 0x564135141c00 secure :-1 s=READY pgs=665 cs=0 l=1 rev1=1 crypto rx=0x7fb81003afd0 tx=0x7fb810000bc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d6c9cc7ed9fd8685 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:31:54.732 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:31:54.738+0000 7fb8017fa700 1 -- 172.21.15.52:0/1823756262 <== mon.0 v2:172.21.15.52:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb81000b2b0 con 0x564135155fb0 2024-05-27T07:31:54.732 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:31:54.738+0000 7fb8017fa700 1 -- 172.21.15.52:0/1823756262 <== mon.0 v2:172.21.15.52:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fb8100055a0 con 0x564135155fb0 2024-05-27T07:31:54.732 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:31:54.738+0000 7fb8017fa700 1 -- 172.21.15.52:0/1823756262 <== mon.0 v2:172.21.15.52:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb8100038e0 con 0x564135155fb0 2024-05-27T07:31:54.734 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:31:54.739+0000 7fb830af7b80 1 -- 172.21.15.52:0/1823756262 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x564135155fb0 msgr2=0x564135141c00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:31:54.734 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:31:54.739+0000 7fb830af7b80 1 --2- 172.21.15.52:0/1823756262 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x564135155fb0 0x564135141c00 secure :-1 s=READY pgs=665 cs=0 l=1 rev1=1 crypto rx=0x7fb81003afd0 tx=0x7fb810000bc0 comp rx=0 tx=0).stop 2024-05-27T07:31:54.735 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:31:54.739+0000 7fb830af7b80 1 -- 172.21.15.52:0/1823756262 shutdown_connections 2024-05-27T07:31:54.735 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:31:54.739+0000 7fb830af7b80 1 --2- 172.21.15.52:0/1823756262 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x564135155fb0 0x564135141c00 unknown :-1 s=CLOSED pgs=665 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:31:54.735 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:31:54.739+0000 7fb830af7b80 1 --2- 172.21.15.52:0/1823756262 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x564134fcde40 0x5641351504c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:31:54.735 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:31:54.739+0000 7fb830af7b80 1 --2- 172.21.15.52:0/1823756262 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5641351517d0 0x564135155950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:31:54.735 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:31:54.739+0000 7fb830af7b80 1 -- 172.21.15.52:0/1823756262 >> 172.21.15.52:0/1823756262 conn(0x564134fd0770 msgr2=0x5641351414f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:31:54.735 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:31:54.739+0000 7fb830af7b80 1 -- 172.21.15.52:0/1823756262 shutdown_connections 2024-05-27T07:31:54.735 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:31:54.739+0000 7fb830af7b80 1 -- 172.21.15.52:0/1823756262 wait complete. 2024-05-27T07:31:54.735 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:31:54.739+0000 7fb830af7b80 1 -- start start 2024-05-27T07:31:54.735 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:31:54.740+0000 7fb830af7b80 1 Processor -- start 2024-05-27T07:31:54.735 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:31:54.740+0000 7fb830af7b80 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x564134fcde40 0x56413515dd60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:31:54.735 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:31:54.740+0000 7fb830af7b80 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5641351517d0 0x56413515e2a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:31:54.736 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:31:54.740+0000 7fb830af7b80 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x564135155fb0 0x56413515e7e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:31:54.736 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:31:54.740+0000 7fb830af7b80 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x7fb81c1ad970 con 0x564134fcde40 2024-05-27T07:31:54.736 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:31:54.740+0000 7fb830af7b80 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7fb81c1ad2a0 con 0x564135155fb0 2024-05-27T07:31:54.736 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:31:54.740+0000 7fb830af7b80 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x564135148860 con 0x5641351517d0 2024-05-27T07:31:54.736 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:31:54.740+0000 7fb830af7b80 10 client.0 _ll_register_callbacks cb 0x56413515bb70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T07:31:54.736 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:31:54.740+0000 7fb81affd700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x564134fcde40 0x56413515dd60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:31:54.736 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:31:54.740+0000 7fb81affd700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x564134fcde40 0x56413515dd60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.52:39138/0 (socket says 172.21.15.52:39138) 2024-05-27T07:31:54.736 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:31:54.741+0000 7fb81affd700 1 -- 172.21.15.52:0/1426964625 learned_addr learned my addr 172.21.15.52:0/1426964625 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-05-27T07:31:54.736 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:31:54.741+0000 7fb81b7fe700 1 --2- 172.21.15.52:0/1426964625 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5641351517d0 0x56413515e2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:31:54.736 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:31:54.741+0000 7fb81affd700 1 --2- 172.21.15.52:0/1426964625 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x564135155fb0 0x56413515e7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:31:54.736 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:31:54.741+0000 7fb81affd700 1 -- 172.21.15.52:0/1426964625 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5641351517d0 msgr2=0x56413515e2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:31:54.736 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:31:54.741+0000 7fb81affd700 1 --2- 172.21.15.52:0/1426964625 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5641351517d0 0x56413515e2a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:31:54.737 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:31:54.741+0000 7fb81affd700 1 -- 172.21.15.52:0/1426964625 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x564135155fb0 msgr2=0x56413515e7e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:31:54.737 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:31:54.741+0000 7fb81affd700 1 --2- 172.21.15.52:0/1426964625 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x564135155fb0 0x56413515e7e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:31:54.737 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:31:54.741+0000 7fb81affd700 1 -- 172.21.15.52:0/1426964625 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5641351cf3b0 con 0x564134fcde40 2024-05-27T07:31:54.737 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:31:54.741+0000 7fb81b7fe700 1 --2- 172.21.15.52:0/1426964625 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5641351517d0 0x56413515e2a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-27T07:31:54.737 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:31:54.741+0000 7fb81affd700 1 --2- 172.21.15.52:0/1426964625 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x564134fcde40 0x56413515dd60 secure :-1 s=READY pgs=666 cs=0 l=1 rev1=1 crypto rx=0x7fb81002afa0 tx=0x7fb81000bcb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c2f6f962bd04d672 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:31:54.737 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:31:54.741+0000 7fb7faffd700 10 client.0 ms_handle_connect on v2:172.21.15.52:3300/0 2024-05-27T07:31:54.737 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:31:54.742+0000 7fb7faffd700 1 -- 172.21.15.52:0/1426964625 <== mon.0 v2:172.21.15.52:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb8100299e0 con 0x564134fcde40 2024-05-27T07:31:54.737 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:31:54.742+0000 7fb830af7b80 1 -- 172.21.15.52:0/1426964625 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fb8100055a0 con 0x564134fcde40 2024-05-27T07:31:54.737 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:31:54.742+0000 7fb7faffd700 1 -- 172.21.15.52:0/1426964625 <== mon.0 v2:172.21.15.52:3300/0 2 ==== osd_map(116..116 src has 1..116) ==== 9379+0+0 (secure 0 0 0) 0x7fb810029b80 con 0x564134fcde40 2024-05-27T07:31:54.737 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:31:54.742+0000 7fb7faffd700 1 -- 172.21.15.52:0/1426964625 <== mon.0 v2:172.21.15.52:3300/0 3 ==== mon_get_version_reply(handle=1 version=101) ==== 24+0+0 (secure 0 0 0) 0x7fb810029d80 con 0x564134fcde40 2024-05-27T07:31:54.739 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:31:54.742+0000 7fb830af7b80 10 client.16698 fetch_fsmap learned FSMap version 101 2024-05-27T07:31:54.739 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:31:54.742+0000 7fb830af7b80 1 -- 172.21.15.52:0/1426964625 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({config=0+,fsmap=101,monmap=4+}) -- 0x5641351d05e0 con 0x564134fcde40 2024-05-27T07:31:54.739 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:31:54.742+0000 7fb7faffd700 1 -- 172.21.15.52:0/1426964625 <== mon.0 v2:172.21.15.52:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fb8100055a0 con 0x564134fcde40 2024-05-27T07:31:54.739 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:31:54.742+0000 7fb7faffd700 1 -- 172.21.15.52:0/1426964625 <== mon.0 v2:172.21.15.52:3300/0 5 ==== fsmap(e 101) ==== 4203+0+0 (secure 0 0 0) 0x7fb8100175f0 con 0x564134fcde40 2024-05-27T07:31:54.740 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:31:54.743+0000 7fb830af7b80 10 client.16698 fetch_fsmap finished waiting for FSMap version 101 2024-05-27T07:31:54.740 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:31:54.743+0000 7fb830af7b80 10 client.16698 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.h 2024-05-27T07:31:54.740 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:31:54.743+0000 7fb830af7b80 20 client.16698 populate_metadata read hostname 'smithi052' 2024-05-27T07:31:54.740 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:31:54.743+0000 7fb830af7b80 1 --2- 172.21.15.52:0/1426964625 >> [v2:172.21.15.105:6836/191393149,v1:172.21.15.105:6837/191393149] conn(0x5641351d0ad0 0x5641351d2ef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:31:54.740 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:31:54.743+0000 7fb830af7b80 4 client.16698 mds_command: new command op to 24497 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-05-27T07:31:54.740 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:31:54.743+0000 7fb830af7b80 1 -- 172.21.15.52:0/1426964625 --> [v2:172.21.15.105:6836/191393149,v1:172.21.15.105:6837/191393149] -- command(tid 0: {"prefix": "damage ls"}) -- 0x5641350764a0 con 0x5641351d0ad0 2024-05-27T07:31:54.740 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:31:54.743+0000 7fb81b7fe700 1 --2- 172.21.15.52:0/1426964625 >> [v2:172.21.15.105:6836/191393149,v1:172.21.15.105:6837/191393149] conn(0x5641351d0ad0 0x5641351d2ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:31:54.740 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:31:54.743+0000 7fb7faffd700 10 client.16698 ms_handle_connect on v2:172.21.15.105:6836/191393149 2024-05-27T07:31:54.740 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:31:54.744+0000 7fb81b7fe700 1 --2- 172.21.15.52:0/1426964625 >> [v2:172.21.15.105:6836/191393149,v1:172.21.15.105:6837/191393149] conn(0x5641351d0ad0 0x5641351d2ef0 crc :-1 s=READY pgs=46 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=cdbef4ed308b770a server_cookie=834ba60629d561f4 in_seq=0 out_seq=0 2024-05-27T07:31:54.740 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:31:54.744+0000 7fb7faffd700 1 -- 172.21.15.52:0/1426964625 <== mds.1 v2:172.21.15.105:6836/191393149 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x5641350764a0 con 0x5641351d0ad0 2024-05-27T07:31:54.740 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:31:54.744+0000 7fb7faffd700 10 client.16698 handle_command_reply: tid=0 2024-05-27T07:31:54.740 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:31:54.745+0000 7fb830af7b80 1 client.16698 shutdown 2024-05-27T07:31:54.741 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:31:54.745+0000 7fb830af7b80 1 -- 172.21.15.52:0/1426964625 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x564134fcde40 msgr2=0x56413515dd60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:31:54.741 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:31:54.745+0000 7fb830af7b80 1 --2- 172.21.15.52:0/1426964625 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x564134fcde40 0x56413515dd60 secure :-1 s=READY pgs=666 cs=0 l=1 rev1=1 crypto rx=0x7fb81002afa0 tx=0x7fb81000bcb0 comp rx=0 tx=0).stop 2024-05-27T07:31:54.741 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:31:54.745+0000 7fb830af7b80 1 -- 172.21.15.52:0/1426964625 shutdown_connections 2024-05-27T07:31:54.741 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:31:54.745+0000 7fb830af7b80 1 --2- 172.21.15.52:0/1426964625 >> [v2:172.21.15.105:6836/191393149,v1:172.21.15.105:6837/191393149] conn(0x5641351d0ad0 0x5641351d2ef0 crc :-1 s=READY pgs=46 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:31:54.741 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:31:54.745+0000 7fb830af7b80 1 --2- 172.21.15.52:0/1426964625 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x564135155fb0 0x56413515e7e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:31:54.741 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:31:54.745+0000 7fb830af7b80 1 --2- 172.21.15.52:0/1426964625 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x564134fcde40 0x56413515dd60 unknown :-1 s=CLOSED pgs=666 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:31:54.741 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:31:54.745+0000 7fb830af7b80 1 --2- 172.21.15.52:0/1426964625 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5641351517d0 0x56413515e2a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:31:54.741 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:31:54.745+0000 7fb830af7b80 1 -- 172.21.15.52:0/1426964625 >> 172.21.15.52:0/1426964625 conn(0x564134fd0770 msgr2=0x564135150220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:31:54.741 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:31:54.745+0000 7fb7faffd700 0 client.16698 ms_handle_reset on v2:172.21.15.105:6836/191393149 2024-05-27T07:31:54.741 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:31:54.745+0000 7fb830af7b80 1 -- 172.21.15.52:0/1426964625 shutdown_connections 2024-05-27T07:31:54.741 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:31:54.745+0000 7fb830af7b80 1 -- 172.21.15.52:0/1426964625 wait complete. 2024-05-27T07:31:54.742 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:31:54.745+0000 7fb830af7b80 20 client.16698 trim_cache size 0 max 16384 2024-05-27T07:31:54.742 INFO:teuthology.orchestra.run.smithi052.stdout:[] 2024-05-27T07:31:54.743 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:31:54.746+0000 7fb8220e2700 1 -- 172.21.15.52:0/691031440 >> [v2:172.21.15.52:6800/1822318384,v1:172.21.15.52:6801/1822318384] conn(0x7fb8040b2a70 msgr2=0x7fb8040b4f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:31:54.743 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:31:54.746+0000 7fb8220e2700 1 --2- 172.21.15.52:0/691031440 >> [v2:172.21.15.52:6800/1822318384,v1:172.21.15.52:6801/1822318384] conn(0x7fb8040b2a70 0x7fb8040b4f80 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7fb8140a6680 tx=0x7fb814009000 comp rx=0 tx=0).stop 2024-05-27T07:31:54.743 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:31:54.746+0000 7fb8220e2700 1 -- 172.21.15.52:0/691031440 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fb81c1092f0 msgr2=0x7fb81c19f780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:31:54.743 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:31:54.746+0000 7fb8220e2700 1 --2- 172.21.15.52:0/691031440 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fb81c1092f0 0x7fb81c19f780 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7fb81400ef30 tx=0x7fb81400c3a0 comp rx=0 tx=0).stop 2024-05-27T07:31:54.743 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:31:54.746+0000 7fb8220e2700 1 -- 172.21.15.52:0/691031440 shutdown_connections 2024-05-27T07:31:54.743 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:31:54.746+0000 7fb8220e2700 1 --2- 172.21.15.52:0/691031440 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fb81c1092f0 0x7fb81c19f780 unknown :-1 s=CLOSED pgs=191 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:31:54.743 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:31:54.746+0000 7fb8220e2700 1 --2- 172.21.15.52:0/691031440 >> [v2:172.21.15.52:6800/1822318384,v1:172.21.15.52:6801/1822318384] conn(0x7fb8040b2a70 0x7fb8040b4f80 unknown :-1 s=CLOSED pgs=152 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:31:54.743 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:31:54.746+0000 7fb8220e2700 1 --2- 172.21.15.52:0/691031440 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fb81c102780 0x7fb81c19af50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:31:54.743 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:31:54.746+0000 7fb8220e2700 1 --2- 172.21.15.52:0/691031440 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fb81c1030d0 0x7fb81c19b4f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:31:54.744 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:31:54.746+0000 7fb8220e2700 1 -- 172.21.15.52:0/691031440 >> 172.21.15.52:0/691031440 conn(0x7fb81c0fe510 msgr2=0x7fb81c1071f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:31:54.744 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:31:54.746+0000 7fb8220e2700 1 -- 172.21.15.52:0/691031440 shutdown_connections 2024-05-27T07:31:54.744 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:31:54.746+0000 7fb8220e2700 1 -- 172.21.15.52:0/691031440 wait complete. 2024-05-27T07:31:55.219 INFO:tasks.workunit.client.0.smithi052.stdout: 140 1225045 197 855698 268 715801 1156 2024-05-27T07:31:55.672 DEBUG:teuthology.orchestra.run.smithi052:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:38f9d900c83c63799fbdbe61acc9a11b0d3554a6 shell --fsid 73df2636-1bf8-11ef-bc9b-c7b262605968 -- ceph tell mds.1:0 damage ls 2024-05-27T07:31:56.113 INFO:teuthology.orchestra.run.smithi052.stderr:Inferring config /var/lib/ceph/73df2636-1bf8-11ef-bc9b-c7b262605968/mon.a/config 2024-05-27T07:31:56.354 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:31:56 smithi173 ceph-mon[37000]: pgmap v207: 129 pgs: 129 active+clean; 566 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 350 KiB/s rd, 6.2 MiB/s wr, 458 op/s 2024-05-27T07:31:56.381 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:31:56 smithi052 ceph-mon[31955]: pgmap v207: 129 pgs: 129 active+clean; 566 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 350 KiB/s rd, 6.2 MiB/s wr, 458 op/s 2024-05-27T07:31:56.442 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:31:56 smithi105 ceph-mon[35977]: pgmap v207: 129 pgs: 129 active+clean; 566 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 350 KiB/s rd, 6.2 MiB/s wr, 458 op/s 2024-05-27T07:31:58.442 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:31:58 smithi105 ceph-mon[35977]: pgmap v208: 129 pgs: 129 active+clean; 573 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 65 KiB/s rd, 6.0 MiB/s wr, 427 op/s 2024-05-27T07:31:58.482 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:31:58 smithi052 ceph-mon[31955]: pgmap v208: 129 pgs: 129 active+clean; 573 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 65 KiB/s rd, 6.0 MiB/s wr, 427 op/s 2024-05-27T07:31:58.604 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:31:58 smithi173 ceph-mon[37000]: pgmap v208: 129 pgs: 129 active+clean; 573 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 65 KiB/s rd, 6.0 MiB/s wr, 427 op/s 2024-05-27T07:32:00.169 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:32:00.175+0000 7f5568498700 1 -- 172.21.15.52:0/3641621879 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f5560105e00 msgr2=0x7f556010e350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:32:00.169 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:32:00.175+0000 7f5568498700 1 --2- 172.21.15.52:0/3641621879 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f5560105e00 0x7f556010e350 secure :-1 s=READY pgs=667 cs=0 l=1 rev1=1 crypto rx=0x7f5550009990 tx=0x7f5550009ca0 comp rx=0 tx=0).stop 2024-05-27T07:32:00.169 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:32:00.175+0000 7f5568498700 1 -- 172.21.15.52:0/3641621879 shutdown_connections 2024-05-27T07:32:00.169 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:32:00.175+0000 7f5568498700 1 --2- 172.21.15.52:0/3641621879 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f556010e8e0 0x7f5560110dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:32:00.169 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:32:00.175+0000 7f5568498700 1 --2- 172.21.15.52:0/3641621879 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f55601054b0 0x7f55601058c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:32:00.169 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:32:00.175+0000 7f5568498700 1 --2- 172.21.15.52:0/3641621879 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f5560105e00 0x7f556010e350 unknown :-1 s=CLOSED pgs=667 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:32:00.170 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:32:00.175+0000 7f5568498700 1 -- 172.21.15.52:0/3641621879 >> 172.21.15.52:0/3641621879 conn(0x7f55600fedb0 msgr2=0x7f55601011d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:32:00.170 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:32:00.175+0000 7f5568498700 1 -- 172.21.15.52:0/3641621879 shutdown_connections 2024-05-27T07:32:00.170 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:32:00.175+0000 7f5568498700 1 -- 172.21.15.52:0/3641621879 wait complete. 2024-05-27T07:32:00.170 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:32:00.176+0000 7f5568498700 1 Processor -- start 2024-05-27T07:32:00.170 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:32:00.176+0000 7f5568498700 1 -- start start 2024-05-27T07:32:00.170 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:32:00.176+0000 7f5568498700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f55601054b0 0x7f556019adf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:32:00.170 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:32:00.176+0000 7f5568498700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f5560105e00 0x7f556019b390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:32:00.171 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:32:00.176+0000 7f5568498700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f556010e8e0 0x7f556019f620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:32:00.171 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:32:00.176+0000 7f5568498700 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x7f5560114460 con 0x7f55601054b0 2024-05-27T07:32:00.172 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:32:00.176+0000 7f5568498700 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f55601142e0 con 0x7f556010e8e0 2024-05-27T07:32:00.172 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:32:00.176+0000 7f5568498700 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f55601145e0 con 0x7f5560105e00 2024-05-27T07:32:00.172 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:32:00.176+0000 7f5566234700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f55601054b0 0x7f556019adf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:32:00.172 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:32:00.176+0000 7f5566234700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f55601054b0 0x7f556019adf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.52:39152/0 (socket says 172.21.15.52:39152) 2024-05-27T07:32:00.172 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:32:00.177+0000 7f5566234700 1 -- 172.21.15.52:0/320908648 learned_addr learned my addr 172.21.15.52:0/320908648 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-05-27T07:32:00.172 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:32:00.177+0000 7f5565a33700 1 --2- 172.21.15.52:0/320908648 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f5560105e00 0x7f556019b390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:32:00.172 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:32:00.177+0000 7f5566234700 1 -- 172.21.15.52:0/320908648 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f5560105e00 msgr2=0x7f556019b390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:32:00.172 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:32:00.177+0000 7f5566234700 1 --2- 172.21.15.52:0/320908648 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f5560105e00 0x7f556019b390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:32:00.172 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:32:00.177+0000 7f5566234700 1 -- 172.21.15.52:0/320908648 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f556010e8e0 msgr2=0x7f556019f620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:32:00.172 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:32:00.177+0000 7f5566234700 1 --2- 172.21.15.52:0/320908648 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f556010e8e0 0x7f556019f620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:32:00.172 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:32:00.177+0000 7f5566234700 1 -- 172.21.15.52:0/320908648 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f556019fcc0 con 0x7f55601054b0 2024-05-27T07:32:00.172 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:32:00.177+0000 7f5565a33700 1 --2- 172.21.15.52:0/320908648 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f5560105e00 0x7f556019b390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-27T07:32:00.173 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:32:00.178+0000 7f5566234700 1 --2- 172.21.15.52:0/320908648 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f55601054b0 0x7f556019adf0 secure :-1 s=READY pgs=668 cs=0 l=1 rev1=1 crypto rx=0x7f555c00ef30 tx=0x7f555c00c3a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:32:00.173 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:32:00.178+0000 7f55577fe700 1 -- 172.21.15.52:0/320908648 <== mon.0 v2:172.21.15.52:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f555c0092e0 con 0x7f55601054b0 2024-05-27T07:32:00.173 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:32:00.178+0000 7f55577fe700 1 -- 172.21.15.52:0/320908648 <== mon.0 v2:172.21.15.52:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f555c004bb0 con 0x7f55601054b0 2024-05-27T07:32:00.173 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:32:00.178+0000 7f55577fe700 1 -- 172.21.15.52:0/320908648 <== mon.0 v2:172.21.15.52:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f555c004280 con 0x7f55601054b0 2024-05-27T07:32:00.173 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:32:00.179+0000 7f5568498700 1 -- 172.21.15.52:0/320908648 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f556019ff50 con 0x7f55601054b0 2024-05-27T07:32:00.173 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:32:00.179+0000 7f5568498700 1 -- 172.21.15.52:0/320908648 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f55601a0410 con 0x7f55601054b0 2024-05-27T07:32:00.174 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:32:00.181+0000 7f55577fe700 1 -- 172.21.15.52:0/320908648 <== mon.0 v2:172.21.15.52:3300/0 4 ==== mgrmap(e 26) ==== 149783+0+0 (secure 0 0 0) 0x7f555c01b030 con 0x7f55601054b0 2024-05-27T07:32:00.175 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:32:00.182+0000 7f55577fe700 1 --2- 172.21.15.52:0/320908648 >> [v2:172.21.15.52:6800/1822318384,v1:172.21.15.52:6801/1822318384] conn(0x7f554c0b2b40 0x7f554c0b5050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:32:00.175 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:32:00.182+0000 7f5565a33700 1 --2- 172.21.15.52:0/320908648 >> [v2:172.21.15.52:6800/1822318384,v1:172.21.15.52:6801/1822318384] conn(0x7f554c0b2b40 0x7f554c0b5050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:32:00.175 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:32:00.182+0000 7f55577fe700 1 -- 172.21.15.52:0/320908648 <== mon.0 v2:172.21.15.52:3300/0 5 ==== osd_map(116..116 src has 1..116) ==== 9379+0+0 (secure 0 0 0) 0x7f555c0e2480 con 0x7f55601054b0 2024-05-27T07:32:00.175 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:32:00.182+0000 7f5576eadb80 1 Processor -- start 2024-05-27T07:32:00.177 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:32:00.183+0000 7f5576eadb80 1 -- start start 2024-05-27T07:32:00.177 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:32:00.183+0000 7f5565a33700 1 --2- 172.21.15.52:0/320908648 >> [v2:172.21.15.52:6800/1822318384,v1:172.21.15.52:6801/1822318384] conn(0x7f554c0b2b40 0x7f554c0b5050 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7f555000b2e0 tx=0x7f5550005960 comp rx=0 tx=0).ready entity=mgr.15333 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:32:00.177 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:32:00.183+0000 7f5576eadb80 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x562ffebdee40 0x562ffed66b20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:32:00.178 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:32:00.183+0000 7f5576eadb80 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x562ffed67060 0x562ffed60e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:32:00.178 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:32:00.183+0000 7f5576eadb80 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x562ffed61390 0x562ffed617a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:32:00.178 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:32:00.183+0000 7f5576eadb80 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x562ffec8bc90 con 0x562ffed61390 2024-05-27T07:32:00.178 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:32:00.183+0000 7f5576eadb80 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x562ffec7b830 con 0x562ffed67060 2024-05-27T07:32:00.178 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:32:00.183+0000 7f5576eadb80 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x562ffec7f870 con 0x562ffebdee40 2024-05-27T07:32:00.178 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:32:00.183+0000 7f5566a35700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x562ffed61390 0x562ffed617a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:32:00.178 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:32:00.183+0000 7f5565a33700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x562ffed67060 0x562ffed60e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:32:00.178 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:32:00.183+0000 7f5566234700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x562ffebdee40 0x562ffed66b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:32:00.178 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:32:00.183+0000 7f5565a33700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x562ffed67060 0x562ffed60e50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.52:35924/0 (socket says 172.21.15.52:35924) 2024-05-27T07:32:00.178 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:32:00.183+0000 7f5566a35700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x562ffed61390 0x562ffed617a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.52:39166/0 (socket says 172.21.15.52:39166) 2024-05-27T07:32:00.179 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:32:00.183+0000 7f5565a33700 1 -- 172.21.15.52:0/2339214910 learned_addr learned my addr 172.21.15.52:0/2339214910 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-05-27T07:32:00.179 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:32:00.183+0000 7f5565a33700 1 -- 172.21.15.52:0/2339214910 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x562ffebdee40 msgr2=0x562ffed66b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:32:00.179 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:32:00.184+0000 7f5565a33700 1 --2- 172.21.15.52:0/2339214910 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x562ffebdee40 0x562ffed66b20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:32:00.179 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:32:00.184+0000 7f5565a33700 1 -- 172.21.15.52:0/2339214910 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x562ffed61390 msgr2=0x562ffed617a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:32:00.179 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:32:00.184+0000 7f5565a33700 1 --2- 172.21.15.52:0/2339214910 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x562ffed61390 0x562ffed617a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:32:00.179 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:32:00.184+0000 7f5565a33700 1 -- 172.21.15.52:0/2339214910 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x562ffebcb8d0 con 0x562ffed67060 2024-05-27T07:32:00.179 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:32:00.184+0000 7f5566234700 1 --2- 172.21.15.52:0/2339214910 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x562ffebdee40 0x562ffed66b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-27T07:32:00.179 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:32:00.184+0000 7f5565a33700 1 --2- 172.21.15.52:0/2339214910 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x562ffed67060 0x562ffed60e50 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7f5550015000 tx=0x7f555001e000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f4cdbad00a143d5e server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:32:00.179 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:32:00.184+0000 7f5554d47700 1 -- 172.21.15.52:0/2339214910 <== mon.2 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5550004a30 con 0x562ffed67060 2024-05-27T07:32:00.180 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:32:00.184+0000 7f5554d47700 1 -- 172.21.15.52:0/2339214910 <== mon.2 v2:172.21.15.105:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f5550004bd0 con 0x562ffed67060 2024-05-27T07:32:00.180 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:32:00.184+0000 7f5554d47700 1 -- 172.21.15.52:0/2339214910 <== mon.2 v2:172.21.15.105:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f555000f6c0 con 0x562ffed67060 2024-05-27T07:32:00.183 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:32:00.185+0000 7f5576eadb80 1 -- 172.21.15.52:0/2339214910 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x562ffed67060 msgr2=0x562ffed60e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:32:00.184 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:32:00.185+0000 7f5576eadb80 1 --2- 172.21.15.52:0/2339214910 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x562ffed67060 0x562ffed60e50 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7f5550015000 tx=0x7f555001e000 comp rx=0 tx=0).stop 2024-05-27T07:32:00.184 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:32:00.185+0000 7f5576eadb80 1 -- 172.21.15.52:0/2339214910 shutdown_connections 2024-05-27T07:32:00.184 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:32:00.185+0000 7f5576eadb80 1 --2- 172.21.15.52:0/2339214910 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x562ffed61390 0x562ffed617a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:32:00.184 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:32:00.185+0000 7f5576eadb80 1 --2- 172.21.15.52:0/2339214910 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x562ffebdee40 0x562ffed66b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:32:00.184 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:32:00.185+0000 7f5576eadb80 1 --2- 172.21.15.52:0/2339214910 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x562ffed67060 0x562ffed60e50 unknown :-1 s=CLOSED pgs=193 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:32:00.184 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:32:00.185+0000 7f5576eadb80 1 -- 172.21.15.52:0/2339214910 >> 172.21.15.52:0/2339214910 conn(0x562ffec39af0 msgr2=0x562ffed57640 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:32:00.184 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:32:00.185+0000 7f5576eadb80 1 -- 172.21.15.52:0/2339214910 shutdown_connections 2024-05-27T07:32:00.184 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:32:00.185+0000 7f5576eadb80 1 -- 172.21.15.52:0/2339214910 wait complete. 2024-05-27T07:32:00.184 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:32:00.186+0000 7f5576eadb80 1 -- start start 2024-05-27T07:32:00.185 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:32:00.186+0000 7f5576eadb80 1 Processor -- start 2024-05-27T07:32:00.185 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:32:00.186+0000 7f5576eadb80 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x562ffebdee40 0x562ffed6a730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:32:00.185 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:32:00.186+0000 7f5576eadb80 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x562ffed61390 0x562ffed6ac70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:32:00.185 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:32:00.186+0000 7f5576eadb80 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x562ffed67060 0x562ffed6b1b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:32:00.185 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:32:00.186+0000 7f5576eadb80 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x562ffeb81fa0 con 0x562ffed67060 2024-05-27T07:32:00.185 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:32:00.186+0000 7f5576eadb80 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x562ffec1d7c0 con 0x562ffed61390 2024-05-27T07:32:00.185 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:32:00.186+0000 7f5576eadb80 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x562ffed6daf0 con 0x562ffebdee40 2024-05-27T07:32:00.185 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:32:00.186+0000 7f5576eadb80 10 client.0 _ll_register_callbacks cb 0x562ffec16920 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T07:32:00.185 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:32:00.186+0000 7f5566a35700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x562ffed67060 0x562ffed6b1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:32:00.185 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:32:00.186+0000 7f5565a33700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x562ffed61390 0x562ffed6ac70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:32:00.185 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:32:00.186+0000 7f5566234700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x562ffebdee40 0x562ffed6a730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:32:00.185 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:32:00.186+0000 7f5566a35700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x562ffed67060 0x562ffed6b1b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.52:39176/0 (socket says 172.21.15.52:39176) 2024-05-27T07:32:00.185 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:32:00.186+0000 7f5566a35700 1 -- 172.21.15.52:0/3097352409 learned_addr learned my addr 172.21.15.52:0/3097352409 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-05-27T07:32:00.186 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:32:00.187+0000 7f5566234700 1 -- 172.21.15.52:0/3097352409 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x562ffed61390 msgr2=0x562ffed6ac70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:32:00.186 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:32:00.187+0000 7f5566234700 1 --2- 172.21.15.52:0/3097352409 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x562ffed61390 0x562ffed6ac70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:32:00.186 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:32:00.187+0000 7f5566234700 1 -- 172.21.15.52:0/3097352409 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x562ffed67060 msgr2=0x562ffed6b1b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:32:00.186 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:32:00.187+0000 7f5566234700 1 --2- 172.21.15.52:0/3097352409 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x562ffed67060 0x562ffed6b1b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:32:00.186 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:32:00.187+0000 7f5566234700 1 -- 172.21.15.52:0/3097352409 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({osdmap=0}) -- 0x562ffec1a690 con 0x562ffebdee40 2024-05-27T07:32:00.186 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:32:00.187+0000 7f5566a35700 1 --2- 172.21.15.52:0/3097352409 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x562ffed67060 0x562ffed6b1b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-27T07:32:00.186 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:32:00.187+0000 7f5565a33700 1 --2- 172.21.15.52:0/3097352409 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x562ffed61390 0x562ffed6ac70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-27T07:32:00.186 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:32:00.187+0000 7f5566234700 1 --2- 172.21.15.52:0/3097352409 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x562ffebdee40 0x562ffed6a730 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7f555c0df870 tx=0x7f555c0e23a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=d59dcd307fa08d20 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:32:00.186 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:32:00.187+0000 7f5549ffb700 10 client.0 ms_handle_connect on v2:172.21.15.173:3300/0 2024-05-27T07:32:00.186 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:32:00.187+0000 7f5549ffb700 1 -- 172.21.15.52:0/3097352409 <== mon.1 v2:172.21.15.173:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f555c0a9b90 con 0x562ffebdee40 2024-05-27T07:32:00.186 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:32:00.187+0000 7f5576eadb80 1 -- 172.21.15.52:0/3097352409 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f5550004bd0 con 0x562ffebdee40 2024-05-27T07:32:00.187 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:32:00.187+0000 7f5549ffb700 1 -- 172.21.15.52:0/3097352409 <== mon.1 v2:172.21.15.173:3300/0 2 ==== osd_map(116..116 src has 1..116) ==== 9379+0+0 (secure 0 0 0) 0x7f555c0b1fe0 con 0x562ffebdee40 2024-05-27T07:32:00.187 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:32:00.187+0000 7f5549ffb700 1 -- 172.21.15.52:0/3097352409 <== mon.1 v2:172.21.15.173:3300/0 3 ==== mon_get_version_reply(handle=1 version=101) ==== 24+0+0 (secure 0 0 0) 0x7f555c0a9490 con 0x562ffebdee40 2024-05-27T07:32:00.187 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:32:00.187+0000 7f5576eadb80 10 client.26227 fetch_fsmap learned FSMap version 101 2024-05-27T07:32:00.187 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:32:00.187+0000 7f5576eadb80 1 -- 172.21.15.52:0/3097352409 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({config=0+,fsmap=101,monmap=4+}) -- 0x562ffed785f0 con 0x562ffebdee40 2024-05-27T07:32:00.187 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:32:00.187+0000 7f5549ffb700 1 -- 172.21.15.52:0/3097352409 <== mon.1 v2:172.21.15.173:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f5550004bd0 con 0x562ffebdee40 2024-05-27T07:32:00.187 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:32:00.188+0000 7f5549ffb700 1 -- 172.21.15.52:0/3097352409 <== mon.1 v2:172.21.15.173:3300/0 5 ==== fsmap(e 101) ==== 4203+0+0 (secure 0 0 0) 0x7f555c004420 con 0x562ffebdee40 2024-05-27T07:32:00.187 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:32:00.188+0000 7f5576eadb80 10 client.26227 fetch_fsmap finished waiting for FSMap version 101 2024-05-27T07:32:00.187 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:32:00.188+0000 7f5576eadb80 10 client.26227 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-05-27T07:32:00.187 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:32:00.188+0000 7f5576eadb80 20 client.26227 populate_metadata read hostname 'smithi052' 2024-05-27T07:32:00.187 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:32:00.188+0000 7f5576eadb80 1 --2- 172.21.15.52:0/3097352409 >> [v2:172.21.15.105:6838/500348151,v1:172.21.15.105:6839/500348151] conn(0x562ffed7b0b0 0x562ffed7d560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:32:00.187 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:32:00.188+0000 7f5576eadb80 4 client.26227 mds_command: new command op to 24527 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-27T07:32:00.187 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:32:00.188+0000 7f5576eadb80 1 -- 172.21.15.52:0/3097352409 --> [v2:172.21.15.105:6838/500348151,v1:172.21.15.105:6839/500348151] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x562ffed289f0 con 0x562ffed7b0b0 2024-05-27T07:32:00.187 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:32:00.188+0000 7f5565a33700 1 --2- 172.21.15.52:0/3097352409 >> [v2:172.21.15.105:6838/500348151,v1:172.21.15.105:6839/500348151] conn(0x562ffed7b0b0 0x562ffed7d560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:32:00.188 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:32:00.188+0000 7f5549ffb700 10 client.26227 ms_handle_connect on v2:172.21.15.105:6838/500348151 2024-05-27T07:32:00.188 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:32:00.188+0000 7f5565a33700 1 --2- 172.21.15.52:0/3097352409 >> [v2:172.21.15.105:6838/500348151,v1:172.21.15.105:6839/500348151] conn(0x562ffed7b0b0 0x562ffed7d560 crc :-1 s=READY pgs=121 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=6ce37bb96213680f server_cookie=6307597ce2ec414f in_seq=0 out_seq=0 2024-05-27T07:32:00.227 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:32:00.205+0000 7f5549ffb700 1 -- 172.21.15.52:0/3097352409 <== mds.0 v2:172.21.15.105:6838/500348151 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24041 (crc 0 0 0) 0x562ffed289f0 con 0x562ffed7b0b0 2024-05-27T07:32:00.227 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:32:00.205+0000 7f5549ffb700 10 client.26227 handle_command_reply: tid=0 2024-05-27T07:32:00.227 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:32:00.205+0000 7f5576eadb80 1 client.26227 shutdown 2024-05-27T07:32:00.228 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:32:00.210+0000 7f5576eadb80 1 -- 172.21.15.52:0/3097352409 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x562ffebdee40 msgr2=0x562ffed6a730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:32:00.228 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:32:00.210+0000 7f5576eadb80 1 --2- 172.21.15.52:0/3097352409 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x562ffebdee40 0x562ffed6a730 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7f555c0df870 tx=0x7f555c0e23a0 comp rx=0 tx=0).stop 2024-05-27T07:32:00.228 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:32:00.212+0000 7f5576eadb80 1 -- 172.21.15.52:0/3097352409 shutdown_connections 2024-05-27T07:32:00.228 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:32:00.212+0000 7f5576eadb80 1 --2- 172.21.15.52:0/3097352409 >> [v2:172.21.15.105:6838/500348151,v1:172.21.15.105:6839/500348151] conn(0x562ffed7b0b0 0x562ffed7d560 crc :-1 s=READY pgs=121 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:32:00.228 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:32:00.212+0000 7f5576eadb80 1 --2- 172.21.15.52:0/3097352409 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x562ffed67060 0x562ffed6b1b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:32:00.228 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:32:00.212+0000 7f5576eadb80 1 --2- 172.21.15.52:0/3097352409 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x562ffebdee40 0x562ffed6a730 unknown :-1 s=CLOSED pgs=153 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:32:00.228 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:32:00.212+0000 7f5576eadb80 1 --2- 172.21.15.52:0/3097352409 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x562ffed61390 0x562ffed6ac70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:32:00.228 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:32:00.212+0000 7f5576eadb80 1 -- 172.21.15.52:0/3097352409 >> 172.21.15.52:0/3097352409 conn(0x562ffec39af0 msgr2=0x562ffed5fe40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:32:00.228 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:32:00.214+0000 7f5549ffb700 0 client.26227 ms_handle_reset on v2:172.21.15.105:6838/500348151 2024-05-27T07:32:00.228 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:32:00.219+0000 7f5576eadb80 1 -- 172.21.15.52:0/3097352409 shutdown_connections 2024-05-27T07:32:00.228 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:32:00.220+0000 7f5576eadb80 1 -- 172.21.15.52:0/3097352409 wait complete. 2024-05-27T07:32:00.228 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:32:00.223+0000 7f5576eadb80 20 client.26227 trim_cache size 0 max 16384 2024-05-27T07:32:00.260 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:32:00.260+0000 7f5576eadb80 1 Processor -- start 2024-05-27T07:32:00.260 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:32:00.262+0000 7f5576eadb80 1 -- start start 2024-05-27T07:32:00.260 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:32:00.262+0000 7f5576eadb80 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x562ffebdee40 0x562ffed60e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:32:00.260 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:32:00.263+0000 7f5576eadb80 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x562ffed61390 0x562ffed76b90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:32:00.260 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:32:00.263+0000 7f5576eadb80 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x562ffed67060 0x562ffed64b30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:32:00.260 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:32:00.263+0000 7f5576eadb80 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x562ffed57bd0 con 0x562ffebdee40 2024-05-27T07:32:00.260 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:32:00.263+0000 7f5576eadb80 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f55601ae740 con 0x562ffed67060 2024-05-27T07:32:00.260 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:32:00.263+0000 7f5576eadb80 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f55601acb40 con 0x562ffed61390 2024-05-27T07:32:00.261 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:32:00.263+0000 7f5566a35700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x562ffed67060 0x562ffed64b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:32:00.261 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:32:00.263+0000 7f5566a35700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x562ffed67060 0x562ffed64b30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.52:35946/0 (socket says 172.21.15.52:35946) 2024-05-27T07:32:00.261 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:32:00.263+0000 7f5566a35700 1 -- 172.21.15.52:0/2000565805 learned_addr learned my addr 172.21.15.52:0/2000565805 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-05-27T07:32:00.261 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:32:00.264+0000 7f5566234700 1 --2- 172.21.15.52:0/2000565805 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x562ffebdee40 0x562ffed60e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:32:00.261 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:32:00.266+0000 7f5566234700 1 -- 172.21.15.52:0/2000565805 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x562ffed61390 msgr2=0x562ffed76b90 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-05-27T07:32:00.261 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:32:00.266+0000 7f5566234700 1 --2- 172.21.15.52:0/2000565805 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x562ffed61390 0x562ffed76b90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:32:00.261 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:32:00.266+0000 7f5566234700 1 -- 172.21.15.52:0/2000565805 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x562ffed67060 msgr2=0x562ffed64b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:32:00.261 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:32:00.266+0000 7f5566234700 1 --2- 172.21.15.52:0/2000565805 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x562ffed67060 0x562ffed64b30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:32:00.261 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:32:00.266+0000 7f5566234700 1 -- 172.21.15.52:0/2000565805 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x562ffed68cc0 con 0x562ffebdee40 2024-05-27T07:32:00.263 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:32:00.271+0000 7f5566234700 1 --2- 172.21.15.52:0/2000565805 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x562ffebdee40 0x562ffed60e50 secure :-1 s=READY pgs=669 cs=0 l=1 rev1=1 crypto rx=0x7f555c0be880 tx=0x7f555c009ed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=59a83f71831acddf server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:32:00.275 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:32:00.277+0000 7f554bfff700 1 -- 172.21.15.52:0/2000565805 <== mon.0 v2:172.21.15.52:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f555c0ad840 con 0x562ffebdee40 2024-05-27T07:32:00.275 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:32:00.277+0000 7f554bfff700 1 -- 172.21.15.52:0/2000565805 <== mon.0 v2:172.21.15.52:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f555c0b2260 con 0x562ffebdee40 2024-05-27T07:32:00.275 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:32:00.278+0000 7f5576eadb80 1 -- 172.21.15.52:0/2000565805 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x562ffebdee40 msgr2=0x562ffed60e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:32:00.275 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:32:00.278+0000 7f5576eadb80 1 --2- 172.21.15.52:0/2000565805 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x562ffebdee40 0x562ffed60e50 secure :-1 s=READY pgs=669 cs=0 l=1 rev1=1 crypto rx=0x7f555c0be880 tx=0x7f555c009ed0 comp rx=0 tx=0).stop 2024-05-27T07:32:00.276 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:32:00.282+0000 7f5576eadb80 1 -- 172.21.15.52:0/2000565805 shutdown_connections 2024-05-27T07:32:00.276 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:32:00.282+0000 7f5576eadb80 1 --2- 172.21.15.52:0/2000565805 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x562ffed67060 0x562ffed64b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:32:00.276 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:32:00.282+0000 7f5576eadb80 1 --2- 172.21.15.52:0/2000565805 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x562ffebdee40 0x562ffed60e50 unknown :-1 s=CLOSED pgs=669 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:32:00.276 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:32:00.282+0000 7f5576eadb80 1 --2- 172.21.15.52:0/2000565805 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x562ffed61390 0x562ffed76b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:32:00.276 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:32:00.282+0000 7f5576eadb80 1 -- 172.21.15.52:0/2000565805 >> 172.21.15.52:0/2000565805 conn(0x562ffec39af0 msgr2=0x562ffed64420 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:32:00.286 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:32:00.294+0000 7f5576eadb80 1 -- 172.21.15.52:0/2000565805 shutdown_connections 2024-05-27T07:32:00.290 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:32:00.297+0000 7f5576eadb80 1 -- 172.21.15.52:0/2000565805 wait complete. 2024-05-27T07:32:00.290 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:32:00.297+0000 7f5576eadb80 1 -- start start 2024-05-27T07:32:00.290 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:32:00.297+0000 7f5576eadb80 1 Processor -- start 2024-05-27T07:32:00.293 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:32:00.298+0000 7f5576eadb80 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x562ffebdee40 0x562ffed5be50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:32:00.293 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:32:00.298+0000 7f5576eadb80 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x562ffed61390 0x562ffed5c390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:32:00.293 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:32:00.298+0000 7f5576eadb80 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x562ffed67060 0x562ffed5c8d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:32:00.293 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:32:00.298+0000 7f5576eadb80 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x7f55601ad210 con 0x562ffed67060 2024-05-27T07:32:00.293 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:32:00.298+0000 7f5576eadb80 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f55601ae070 con 0x562ffed61390 2024-05-27T07:32:00.293 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:32:00.298+0000 7f5576eadb80 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f55601ad8e0 con 0x562ffebdee40 2024-05-27T07:32:00.293 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:32:00.298+0000 7f5576eadb80 10 client.0 _ll_register_callbacks cb 0x562ffec16920 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T07:32:00.293 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:32:00.299+0000 7f5565a33700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x562ffed61390 0x562ffed5c390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:32:00.293 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:32:00.299+0000 7f5565a33700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x562ffed61390 0x562ffed5c390 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.52:35960/0 (socket says 172.21.15.52:35960) 2024-05-27T07:32:00.293 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:32:00.299+0000 7f5565a33700 1 -- 172.21.15.52:0/2942382808 learned_addr learned my addr 172.21.15.52:0/2942382808 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-05-27T07:32:00.294 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:32:00.299+0000 7f5565a33700 1 -- 172.21.15.52:0/2942382808 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x562ffebdee40 msgr2=0x562ffed5be50 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-05-27T07:32:00.294 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:32:00.299+0000 7f5565a33700 1 --2- 172.21.15.52:0/2942382808 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x562ffebdee40 0x562ffed5be50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:32:00.294 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:32:00.299+0000 7f5565a33700 1 -- 172.21.15.52:0/2942382808 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x562ffed67060 msgr2=0x562ffed5c8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:32:00.294 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:32:00.299+0000 7f5565a33700 1 --2- 172.21.15.52:0/2942382808 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x562ffed67060 0x562ffed5c8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:32:00.294 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:32:00.299+0000 7f5565a33700 1 -- 172.21.15.52:0/2942382808 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({osdmap=0}) -- 0x562ffed71d80 con 0x562ffed61390 2024-05-27T07:32:00.294 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:32:00.299+0000 7f5565a33700 1 --2- 172.21.15.52:0/2942382808 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x562ffed61390 0x562ffed5c390 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7f555002fe10 tx=0x7f555002fe40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=17565ab7ef62bac7 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:32:00.294 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:32:00.300+0000 7f5549ffb700 10 client.0 ms_handle_connect on v2:172.21.15.105:3300/0 2024-05-27T07:32:00.294 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:32:00.300+0000 7f5549ffb700 1 -- 172.21.15.52:0/2942382808 <== mon.2 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f555002f680 con 0x562ffed61390 2024-05-27T07:32:00.294 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:32:00.300+0000 7f5576eadb80 1 -- 172.21.15.52:0/2942382808 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f555c0b2260 con 0x562ffed61390 2024-05-27T07:32:00.294 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:32:00.300+0000 7f5549ffb700 1 -- 172.21.15.52:0/2942382808 <== mon.2 v2:172.21.15.105:3300/0 2 ==== osd_map(116..116 src has 1..116) ==== 9379+0+0 (secure 0 0 0) 0x7f555002f820 con 0x562ffed61390 2024-05-27T07:32:00.296 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:32:00.304+0000 7f5549ffb700 1 -- 172.21.15.52:0/2942382808 <== mon.2 v2:172.21.15.105:3300/0 3 ==== mon_get_version_reply(handle=1 version=101) ==== 24+0+0 (secure 0 0 0) 0x7f555004a050 con 0x562ffed61390 2024-05-27T07:32:00.297 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:32:00 smithi052 ceph-mon[31955]: pgmap v209: 129 pgs: 129 active+clean; 572 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 6.2 MiB/s wr, 845 op/s 2024-05-27T07:32:00.299 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:32:00.306+0000 7f5576eadb80 10 client.26363 fetch_fsmap learned FSMap version 101 2024-05-27T07:32:00.299 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:32:00.306+0000 7f5576eadb80 1 -- 172.21.15.52:0/2942382808 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,fsmap=101,monmap=4+}) -- 0x562ffed72fe0 con 0x562ffed61390 2024-05-27T07:32:00.303 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:32:00.310+0000 7f5549ffb700 1 -- 172.21.15.52:0/2942382808 <== mon.2 v2:172.21.15.105:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f555c0b2260 con 0x562ffed61390 2024-05-27T07:32:00.303 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:32:00.310+0000 7f5549ffb700 1 -- 172.21.15.52:0/2942382808 <== mon.2 v2:172.21.15.105:3300/0 5 ==== fsmap(e 101) ==== 4203+0+0 (secure 0 0 0) 0x7f5550022070 con 0x562ffed61390 2024-05-27T07:32:00.303 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:32:00.310+0000 7f5576eadb80 10 client.26363 fetch_fsmap finished waiting for FSMap version 101 2024-05-27T07:32:00.303 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:32:00.310+0000 7f5576eadb80 10 client.26363 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-05-27T07:32:00.303 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:32:00.310+0000 7f5576eadb80 20 client.26363 populate_metadata read hostname 'smithi052' 2024-05-27T07:32:00.304 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:32:00.310+0000 7f5576eadb80 1 --2- 172.21.15.52:0/2942382808 >> [v2:172.21.15.105:6838/500348151,v1:172.21.15.105:6839/500348151] conn(0x562ffede1bd0 0x562ffedf53c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:32:00.304 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:32:00.310+0000 7f5576eadb80 4 client.26363 mds_command: new command op to 24527 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-05-27T07:32:00.304 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:32:00.310+0000 7f5576eadb80 1 -- 172.21.15.52:0/2942382808 --> [v2:172.21.15.105:6838/500348151,v1:172.21.15.105:6839/500348151] -- command(tid 0: {"prefix": "damage ls"}) -- 0x562ffec8dbe0 con 0x562ffede1bd0 2024-05-27T07:32:00.304 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:32:00.311+0000 7f5566234700 1 --2- 172.21.15.52:0/2942382808 >> [v2:172.21.15.105:6838/500348151,v1:172.21.15.105:6839/500348151] conn(0x562ffede1bd0 0x562ffedf53c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:32:00.321 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:32:00.325+0000 7f5566234700 1 --2- 172.21.15.52:0/2942382808 >> [v2:172.21.15.105:6838/500348151,v1:172.21.15.105:6839/500348151] conn(0x562ffede1bd0 0x562ffedf53c0 crc :-1 s=READY pgs=122 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=798eba17ac7dd7df server_cookie=c5f8cdfc0b56da67 in_seq=0 out_seq=0 2024-05-27T07:32:00.321 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:32:00.325+0000 7f5549ffb700 10 client.26363 ms_handle_connect on v2:172.21.15.105:6838/500348151 2024-05-27T07:32:00.340 INFO:teuthology.orchestra.run.smithi052.stdout:[] 2024-05-27T07:32:00.340 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:32:00.345+0000 7f5549ffb700 1 -- 172.21.15.52:0/2942382808 <== mds.0 v2:172.21.15.105:6838/500348151 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x562ffec8dbe0 con 0x562ffede1bd0 2024-05-27T07:32:00.340 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:32:00.345+0000 7f5549ffb700 10 client.26363 handle_command_reply: tid=0 2024-05-27T07:32:00.340 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:32:00.345+0000 7f5576eadb80 1 client.26363 shutdown 2024-05-27T07:32:00.340 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:32:00.345+0000 7f5576eadb80 1 -- 172.21.15.52:0/2942382808 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x562ffed61390 msgr2=0x562ffed5c390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:32:00.341 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:32:00.345+0000 7f5576eadb80 1 --2- 172.21.15.52:0/2942382808 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x562ffed61390 0x562ffed5c390 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7f555002fe10 tx=0x7f555002fe40 comp rx=0 tx=0).stop 2024-05-27T07:32:00.341 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:32:00.345+0000 7f5576eadb80 1 -- 172.21.15.52:0/2942382808 shutdown_connections 2024-05-27T07:32:00.341 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:32:00.345+0000 7f5576eadb80 1 --2- 172.21.15.52:0/2942382808 >> [v2:172.21.15.105:6838/500348151,v1:172.21.15.105:6839/500348151] conn(0x562ffede1bd0 0x562ffedf53c0 crc :-1 s=READY pgs=122 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:32:00.341 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:32:00.345+0000 7f5576eadb80 1 --2- 172.21.15.52:0/2942382808 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x562ffed67060 0x562ffed5c8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:32:00.341 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:32:00.345+0000 7f5576eadb80 1 --2- 172.21.15.52:0/2942382808 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x562ffebdee40 0x562ffed5be50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:32:00.341 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:32:00.345+0000 7f5549ffb700 0 client.26363 ms_handle_reset on v2:172.21.15.105:6838/500348151 2024-05-27T07:32:00.341 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:32:00.345+0000 7f5576eadb80 1 --2- 172.21.15.52:0/2942382808 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x562ffed61390 0x562ffed5c390 unknown :-1 s=CLOSED pgs=194 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:32:00.341 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:32:00.345+0000 7f5576eadb80 1 -- 172.21.15.52:0/2942382808 >> 172.21.15.52:0/2942382808 conn(0x562ffec39af0 msgr2=0x562ffed51e80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:32:00.341 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:32:00.345+0000 7f5576eadb80 1 -- 172.21.15.52:0/2942382808 shutdown_connections 2024-05-27T07:32:00.341 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:32:00.345+0000 7f5576eadb80 1 -- 172.21.15.52:0/2942382808 wait complete. 2024-05-27T07:32:00.341 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:32:00.346+0000 7f5576eadb80 20 client.26363 trim_cache size 0 max 16384 2024-05-27T07:32:00.341 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:32:00.347+0000 7f5568498700 1 -- 172.21.15.52:0/320908648 >> [v2:172.21.15.52:6800/1822318384,v1:172.21.15.52:6801/1822318384] conn(0x7f554c0b2b40 msgr2=0x7f554c0b5050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:32:00.341 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:32:00.347+0000 7f5568498700 1 --2- 172.21.15.52:0/320908648 >> [v2:172.21.15.52:6800/1822318384,v1:172.21.15.52:6801/1822318384] conn(0x7f554c0b2b40 0x7f554c0b5050 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7f555000b2e0 tx=0x7f5550005960 comp rx=0 tx=0).stop 2024-05-27T07:32:00.342 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:32:00.347+0000 7f5568498700 1 -- 172.21.15.52:0/320908648 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f55601054b0 msgr2=0x7f556019adf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:32:00.342 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:32:00.347+0000 7f5568498700 1 --2- 172.21.15.52:0/320908648 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f55601054b0 0x7f556019adf0 secure :-1 s=READY pgs=668 cs=0 l=1 rev1=1 crypto rx=0x7f555c00ef30 tx=0x7f555c00c3a0 comp rx=0 tx=0).stop 2024-05-27T07:32:00.342 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:32:00.347+0000 7f5568498700 1 -- 172.21.15.52:0/320908648 shutdown_connections 2024-05-27T07:32:00.342 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:32:00.347+0000 7f5568498700 1 --2- 172.21.15.52:0/320908648 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f556010e8e0 0x7f556019f620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:32:00.342 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:32:00.347+0000 7f5568498700 1 --2- 172.21.15.52:0/320908648 >> [v2:172.21.15.52:6800/1822318384,v1:172.21.15.52:6801/1822318384] conn(0x7f554c0b2b40 0x7f554c0b5050 unknown :-1 s=CLOSED pgs=153 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:32:00.342 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:32:00.347+0000 7f5568498700 1 --2- 172.21.15.52:0/320908648 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f55601054b0 0x7f556019adf0 unknown :-1 s=CLOSED pgs=668 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:32:00.342 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:32:00.347+0000 7f5568498700 1 --2- 172.21.15.52:0/320908648 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f5560105e00 0x7f556019b390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:32:00.342 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:32:00.347+0000 7f5568498700 1 -- 172.21.15.52:0/320908648 >> 172.21.15.52:0/320908648 conn(0x7f55600fedb0 msgr2=0x7f55600ff440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:32:00.342 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:32:00.347+0000 7f5568498700 1 -- 172.21.15.52:0/320908648 shutdown_connections 2024-05-27T07:32:00.342 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:32:00.347+0000 7f5568498700 1 -- 172.21.15.52:0/320908648 wait complete. 2024-05-27T07:32:00.442 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:32:00 smithi105 ceph-mon[35977]: pgmap v209: 129 pgs: 129 active+clean; 572 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 6.2 MiB/s wr, 845 op/s 2024-05-27T07:32:00.604 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:32:00 smithi173 ceph-mon[37000]: pgmap v209: 129 pgs: 129 active+clean; 572 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 6.2 MiB/s wr, 845 op/s 2024-05-27T07:32:00.978 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:32:00.985+0000 7f02d1ffb640 1 -- 172.21.15.52:0/1478820666 <== mds.0 v2:172.21.15.105:6838/500348151 1 ==== command_reply(tid 0: 0 ) ==== 8+0+25 (crc 0 0 0) 0x556f93858360 con 0x556f934dd1c0 2024-05-27T07:32:00.978 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:32:00.985+0000 7f02d1ffb640 10 client.26200 handle_command_reply: tid=0 2024-05-27T07:32:00.980 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:32:00.987+0000 7f02f7c1a740 1 client.26200 shutdown 2024-05-27T07:32:00.980 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:32:00.987+0000 7f02f7c1a740 1 -- 172.21.15.52:0/1478820666 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x556f935491b0 msgr2=0x556f938fa540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:32:00.980 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:32:00.987+0000 7f02f7c1a740 1 --2- 172.21.15.52:0/1478820666 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x556f935491b0 0x556f938fa540 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x556f93904f90 tx=0x7f02dc0035f0 comp rx=0 tx=0).stop 2024-05-27T07:32:00.980 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:32:00.987+0000 7f02f7c1a740 1 -- 172.21.15.52:0/1478820666 shutdown_connections 2024-05-27T07:32:00.980 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:32:00.987+0000 7f02f7c1a740 1 --2- 172.21.15.52:0/1478820666 >> [v2:172.21.15.105:6838/500348151,v1:172.21.15.105:6839/500348151] conn(0x556f934dd1c0 0x556f93971c30 crc :-1 s=READY pgs=120 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:32:00.980 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:32:00.987+0000 7f02f7c1a740 1 --2- 172.21.15.52:0/1478820666 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x556f935491b0 0x556f938fa540 unknown :-1 s=CLOSED pgs=151 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:32:00.980 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:32:00.987+0000 7f02f7c1a740 1 --2- 172.21.15.52:0/1478820666 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x556f935284d0 0x556f938fa000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:32:00.980 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:32:00.987+0000 7f02f7c1a740 1 --2- 172.21.15.52:0/1478820666 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x556f935202f0 0x556f938fcbf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:32:00.981 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:32:00.987+0000 7f02f7c1a740 1 -- 172.21.15.52:0/1478820666 >> 172.21.15.52:0/1478820666 conn(0x556f9353d300 msgr2=0x556f938ee020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:32:00.981 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:32:00.987+0000 7f02d1ffb640 0 client.26200 ms_handle_reset on v2:172.21.15.105:6838/500348151 2024-05-27T07:32:00.981 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:32:00.987+0000 7f02f7c1a740 1 -- 172.21.15.52:0/1478820666 shutdown_connections 2024-05-27T07:32:00.981 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:32:00.988+0000 7f02f7c1a740 1 -- 172.21.15.52:0/1478820666 wait complete. 2024-05-27T07:32:00.981 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:32:00.988+0000 7f02f7c1a740 20 client.26200 trim_cache size 0 max 16384 2024-05-27T07:32:00.982 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stdout:{ 2024-05-27T07:32:00.982 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stdout: "return_code": 0 2024-05-27T07:32:00.982 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stdout:} 2024-05-27T07:32:00.983 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:32:00.990+0000 7f02f0d6c640 1 -- 172.21.15.52:0/62428442 >> [v2:172.21.15.52:6800/1822318384,v1:172.21.15.52:6801/1822318384] conn(0x7f02d40b2ae0 msgr2=0x7f02d40b4ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:32:00.983 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:32:00.990+0000 7f02f0d6c640 1 --2- 172.21.15.52:0/62428442 >> [v2:172.21.15.52:6800/1822318384,v1:172.21.15.52:6801/1822318384] conn(0x7f02d40b2ae0 0x7f02d40b4ff0 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7f02e00a95b0 tx=0x7f02e0005b90 comp rx=0 tx=0).stop 2024-05-27T07:32:00.983 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:32:00.990+0000 7f02f0d6c640 1 -- 172.21.15.52:0/62428442 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f02ec1b3470 msgr2=0x7f02ec1b5890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:32:00.983 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:32:00.990+0000 7f02f0d6c640 1 --2- 172.21.15.52:0/62428442 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f02ec1b3470 0x7f02ec1b5890 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7f02e0007860 tx=0x7f02e0007d20 comp rx=0 tx=0).stop 2024-05-27T07:32:00.983 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:32:00.990+0000 7f02f0d6c640 1 -- 172.21.15.52:0/62428442 shutdown_connections 2024-05-27T07:32:00.983 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:32:00.990+0000 7f02f0d6c640 1 --2- 172.21.15.52:0/62428442 >> [v2:172.21.15.52:6800/1822318384,v1:172.21.15.52:6801/1822318384] conn(0x7f02d40b2ae0 0x7f02d40b4ff0 unknown :-1 s=CLOSED pgs=150 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:32:00.983 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:32:00.990+0000 7f02f0d6c640 1 --2- 172.21.15.52:0/62428442 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f02ec1b3470 0x7f02ec1b5890 unknown :-1 s=CLOSED pgs=149 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:32:00.983 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:32:00.990+0000 7f02f0d6c640 1 --2- 172.21.15.52:0/62428442 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f02ec10f940 0x7f02ec11c770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:32:00.983 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:32:00.990+0000 7f02f0d6c640 1 --2- 172.21.15.52:0/62428442 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f02ec10a8f0 0x7f02ec11c230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:32:00.984 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:32:00.990+0000 7f02f0d6c640 1 -- 172.21.15.52:0/62428442 >> 172.21.15.52:0/62428442 conn(0x7f02ec06eda0 msgr2=0x7f02ec10ddd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:32:00.984 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:32:00.990+0000 7f02f0d6c640 1 -- 172.21.15.52:0/62428442 shutdown_connections 2024-05-27T07:32:00.984 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:32:00.990+0000 7f02f0d6c640 1 -- 172.21.15.52:0/62428442 wait complete. 2024-05-27T07:32:01.998 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-05-27T07:32:01.999 DEBUG:teuthology.orchestra.run.smithi052:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:38f9d900c83c63799fbdbe61acc9a11b0d3554a6 shell --fsid 73df2636-1bf8-11ef-bc9b-c7b262605968 -- ceph tell mds.1:0 scrub start / force,recursive 2024-05-27T07:32:02.256 INFO:teuthology.orchestra.run.smithi052.stderr:Inferring config /var/lib/ceph/73df2636-1bf8-11ef-bc9b-c7b262605968/mon.a/config 2024-05-27T07:32:02.379 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:32:02 smithi052 ceph-mon[31955]: pgmap v210: 129 pgs: 129 active+clean; 574 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 8.7 KiB/s rd, 5.5 MiB/s wr, 1.25k op/s 2024-05-27T07:32:02.442 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:32:02 smithi105 ceph-mon[35977]: pgmap v210: 129 pgs: 129 active+clean; 574 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 8.7 KiB/s rd, 5.5 MiB/s wr, 1.25k op/s 2024-05-27T07:32:02.605 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:32:02 smithi173 ceph-mon[37000]: pgmap v210: 129 pgs: 129 active+clean; 574 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 8.7 KiB/s rd, 5.5 MiB/s wr, 1.25k op/s 2024-05-27T07:32:04.443 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:32:04 smithi105 ceph-mon[35977]: pgmap v211: 129 pgs: 129 active+clean; 562 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 3.2 MiB/s wr, 1.19k op/s 2024-05-27T07:32:04.450 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:32:04 smithi052 ceph-mon[31955]: pgmap v211: 129 pgs: 129 active+clean; 562 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 3.2 MiB/s wr, 1.19k op/s 2024-05-27T07:32:04.576 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:32:04.583+0000 7f4d05857700 1 -- 172.21.15.52:0/2120709027 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f4d001042a0 msgr2=0x7f4d00110060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:32:04.576 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:32:04.583+0000 7f4d05857700 1 --2- 172.21.15.52:0/2120709027 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f4d001042a0 0x7f4d00110060 secure :-1 s=READY pgs=670 cs=0 l=1 rev1=1 crypto rx=0x7f4cf800b0a0 tx=0x7f4cf800b3b0 comp rx=0 tx=0).stop 2024-05-27T07:32:04.576 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:32:04.583+0000 7f4d05857700 1 -- 172.21.15.52:0/2120709027 shutdown_connections 2024-05-27T07:32:04.576 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:32:04.583+0000 7f4d05857700 1 --2- 172.21.15.52:0/2120709027 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f4d001042a0 0x7f4d00110060 unknown :-1 s=CLOSED pgs=670 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:32:04.576 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:32:04.583+0000 7f4d05857700 1 --2- 172.21.15.52:0/2120709027 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f4d001081f0 0x7f4d00108600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:32:04.576 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:32:04.583+0000 7f4d05857700 1 --2- 172.21.15.52:0/2120709027 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f4d00103950 0x7f4d00103d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:32:04.577 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:32:04.583+0000 7f4d05857700 1 -- 172.21.15.52:0/2120709027 >> 172.21.15.52:0/2120709027 conn(0x7f4d000fe4d0 msgr2=0x7f4d00100920 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:32:04.577 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:32:04.583+0000 7f4d05857700 1 -- 172.21.15.52:0/2120709027 shutdown_connections 2024-05-27T07:32:04.577 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:32:04.583+0000 7f4d05857700 1 -- 172.21.15.52:0/2120709027 wait complete. 2024-05-27T07:32:04.578 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:32:04.584+0000 7f4d05857700 1 Processor -- start 2024-05-27T07:32:04.578 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:32:04.584+0000 7f4d05857700 1 -- start start 2024-05-27T07:32:04.579 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:32:04.584+0000 7f4d05857700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f4d00103950 0x7f4d0019bc00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:32:04.579 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:32:04.584+0000 7f4d05857700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f4d001042a0 0x7f4d0019c140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:32:04.579 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:32:04.584+0000 7f4d05857700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f4d001081f0 0x7f4d001a6c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:32:04.579 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:32:04.584+0000 7f4d05857700 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x7f4d00113460 con 0x7f4d00103950 2024-05-27T07:32:04.579 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:32:04.584+0000 7f4d05857700 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f4d001132e0 con 0x7f4d001081f0 2024-05-27T07:32:04.579 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:32:04.584+0000 7f4d05857700 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f4d001135e0 con 0x7f4d001042a0 2024-05-27T07:32:04.579 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:32:04.584+0000 7f4cfeffd700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f4d00103950 0x7f4d0019bc00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:32:04.579 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:32:04.584+0000 7f4cfeffd700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f4d00103950 0x7f4d0019bc00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.52:47372/0 (socket says 172.21.15.52:47372) 2024-05-27T07:32:04.579 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:32:04.584+0000 7f4cfeffd700 1 -- 172.21.15.52:0/631726525 learned_addr learned my addr 172.21.15.52:0/631726525 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-05-27T07:32:04.579 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:32:04.584+0000 7f4cff7fe700 1 --2- 172.21.15.52:0/631726525 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f4d001081f0 0x7f4d001a6c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:32:04.579 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:32:04.585+0000 7f4cfeffd700 1 -- 172.21.15.52:0/631726525 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f4d001042a0 msgr2=0x7f4d0019c140 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-05-27T07:32:04.579 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:32:04.585+0000 7f4cfeffd700 1 --2- 172.21.15.52:0/631726525 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f4d001042a0 0x7f4d0019c140 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:32:04.579 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:32:04.585+0000 7f4cfeffd700 1 -- 172.21.15.52:0/631726525 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f4d001081f0 msgr2=0x7f4d001a6c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:32:04.580 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:32:04.585+0000 7f4cfeffd700 1 --2- 172.21.15.52:0/631726525 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f4d001081f0 0x7f4d001a6c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:32:04.580 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:32:04.585+0000 7f4cfeffd700 1 -- 172.21.15.52:0/631726525 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4d0019c960 con 0x7f4d00103950 2024-05-27T07:32:04.580 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:32:04.585+0000 7f4cfeffd700 1 --2- 172.21.15.52:0/631726525 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f4d00103950 0x7f4d0019bc00 secure :-1 s=READY pgs=671 cs=0 l=1 rev1=1 crypto rx=0x7f4ce800c970 tx=0x7f4ce800cd30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:32:04.580 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:32:04.585+0000 7f4d04855700 1 -- 172.21.15.52:0/631726525 <== mon.0 v2:172.21.15.52:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4ce8007b00 con 0x7f4d00103950 2024-05-27T07:32:04.580 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:32:04.585+0000 7f4d04855700 1 -- 172.21.15.52:0/631726525 <== mon.0 v2:172.21.15.52:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f4ce8007ca0 con 0x7f4d00103950 2024-05-27T07:32:04.580 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:32:04.585+0000 7f4d04855700 1 -- 172.21.15.52:0/631726525 <== mon.0 v2:172.21.15.52:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4ce8018660 con 0x7f4d00103950 2024-05-27T07:32:04.580 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:32:04.585+0000 7f4d05857700 1 -- 172.21.15.52:0/631726525 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4d001a7220 con 0x7f4d00103950 2024-05-27T07:32:04.580 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:32:04.585+0000 7f4d05857700 1 -- 172.21.15.52:0/631726525 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4d001a7760 con 0x7f4d00103950 2024-05-27T07:32:04.581 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:32:04.588+0000 7f4d04855700 1 -- 172.21.15.52:0/631726525 <== mon.0 v2:172.21.15.52:3300/0 4 ==== mgrmap(e 26) ==== 149783+0+0 (secure 0 0 0) 0x7f4ce800f450 con 0x7f4d00103950 2024-05-27T07:32:04.583 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:32:04.588+0000 7f4d04855700 1 --2- 172.21.15.52:0/631726525 >> [v2:172.21.15.52:6800/1822318384,v1:172.21.15.52:6801/1822318384] conn(0x7f4cec0b29f0 0x7f4cec0b4f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:32:04.583 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:32:04.589+0000 7f4cfe7fc700 1 --2- 172.21.15.52:0/631726525 >> [v2:172.21.15.52:6800/1822318384,v1:172.21.15.52:6801/1822318384] conn(0x7f4cec0b29f0 0x7f4cec0b4f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:32:04.583 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:32:04.589+0000 7f4d04855700 1 -- 172.21.15.52:0/631726525 <== mon.0 v2:172.21.15.52:3300/0 5 ==== osd_map(116..116 src has 1..116) ==== 9379+0+0 (secure 0 0 0) 0x7f4ce80e2980 con 0x7f4d00103950 2024-05-27T07:32:04.584 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:32:04.589+0000 7f4d1426cb80 1 Processor -- start 2024-05-27T07:32:04.584 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:32:04.589+0000 7f4d1426cb80 1 -- start start 2024-05-27T07:32:04.584 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:32:04.589+0000 7f4cfe7fc700 1 --2- 172.21.15.52:0/631726525 >> [v2:172.21.15.52:6800/1822318384,v1:172.21.15.52:6801/1822318384] conn(0x7f4cec0b29f0 0x7f4cec0b4f00 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7f4d00106780 tx=0x7f4cf0006c60 comp rx=0 tx=0).ready entity=mgr.15333 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:32:04.584 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:32:04.589+0000 7f4d1426cb80 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x558b3bff2ec0 0x558b3c176790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:32:04.584 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:32:04.589+0000 7f4d1426cb80 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x558b3c176cd0 0x558b3c170ac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:32:04.584 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:32:04.589+0000 7f4d1426cb80 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x558b3c171000 0x558b3c171410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:32:04.584 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:32:04.589+0000 7f4d1426cb80 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x558b3c09fd10 con 0x558b3bff2ec0 2024-05-27T07:32:04.584 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:32:04.589+0000 7f4d1426cb80 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x558b3c08f8b0 con 0x558b3c171000 2024-05-27T07:32:04.584 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:32:04.589+0000 7f4d1426cb80 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x558b3c0938f0 con 0x558b3c176cd0 2024-05-27T07:32:04.584 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:32:04.589+0000 7f4cfeffd700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x558b3bff2ec0 0x558b3c176790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:32:04.585 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:32:04.589+0000 7f4cfeffd700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x558b3bff2ec0 0x558b3c176790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.52:47380/0 (socket says 172.21.15.52:47380) 2024-05-27T07:32:04.585 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:32:04.589+0000 7f4cfeffd700 1 -- 172.21.15.52:0/912668490 learned_addr learned my addr 172.21.15.52:0/912668490 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-05-27T07:32:04.585 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:32:04.590+0000 7f4cff7fe700 1 --2- 172.21.15.52:0/912668490 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x558b3c171000 0x558b3c171410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:32:04.585 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:32:04.590+0000 7f4cfeffd700 1 -- 172.21.15.52:0/912668490 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x558b3c176cd0 msgr2=0x558b3c170ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:32:04.585 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:32:04.590+0000 7f4cfeffd700 1 --2- 172.21.15.52:0/912668490 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x558b3c176cd0 0x558b3c170ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:32:04.585 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:32:04.590+0000 7f4cfeffd700 1 -- 172.21.15.52:0/912668490 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x558b3c171000 msgr2=0x558b3c171410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:32:04.585 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:32:04.590+0000 7f4cfeffd700 1 --2- 172.21.15.52:0/912668490 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x558b3c171000 0x558b3c171410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:32:04.585 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:32:04.590+0000 7f4cfeffd700 1 -- 172.21.15.52:0/912668490 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x558b3bfdf950 con 0x558b3bff2ec0 2024-05-27T07:32:04.586 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:32:04.590+0000 7f4cff7fe700 1 --2- 172.21.15.52:0/912668490 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x558b3c171000 0x558b3c171410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-27T07:32:04.586 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:32:04.590+0000 7f4cfeffd700 1 --2- 172.21.15.52:0/912668490 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x558b3bff2ec0 0x558b3c176790 secure :-1 s=READY pgs=672 cs=0 l=1 rev1=1 crypto rx=0x7f4ce80a9600 tx=0x7f4ce80a58f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f3d0d3dc56cadf78 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:32:04.586 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:32:04.590+0000 7f4cf5ffb700 1 -- 172.21.15.52:0/912668490 <== mon.0 v2:172.21.15.52:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4ce80aa000 con 0x558b3bff2ec0 2024-05-27T07:32:04.586 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:32:04.590+0000 7f4cf5ffb700 1 -- 172.21.15.52:0/912668490 <== mon.0 v2:172.21.15.52:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f4ce80a9900 con 0x558b3bff2ec0 2024-05-27T07:32:04.586 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:32:04.590+0000 7f4cf5ffb700 1 -- 172.21.15.52:0/912668490 <== mon.0 v2:172.21.15.52:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4ce80a9dc0 con 0x558b3bff2ec0 2024-05-27T07:32:04.591 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:32:04.591+0000 7f4d1426cb80 1 -- 172.21.15.52:0/912668490 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x558b3bff2ec0 msgr2=0x558b3c176790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:32:04.591 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:32:04.591+0000 7f4d1426cb80 1 --2- 172.21.15.52:0/912668490 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x558b3bff2ec0 0x558b3c176790 secure :-1 s=READY pgs=672 cs=0 l=1 rev1=1 crypto rx=0x7f4ce80a9600 tx=0x7f4ce80a58f0 comp rx=0 tx=0).stop 2024-05-27T07:32:04.591 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:32:04.591+0000 7f4d1426cb80 1 -- 172.21.15.52:0/912668490 shutdown_connections 2024-05-27T07:32:04.591 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:32:04.591+0000 7f4d1426cb80 1 --2- 172.21.15.52:0/912668490 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x558b3c171000 0x558b3c171410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:32:04.591 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:32:04.591+0000 7f4d1426cb80 1 --2- 172.21.15.52:0/912668490 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x558b3bff2ec0 0x558b3c176790 unknown :-1 s=CLOSED pgs=672 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:32:04.592 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:32:04.591+0000 7f4d1426cb80 1 --2- 172.21.15.52:0/912668490 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x558b3c176cd0 0x558b3c170ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:32:04.592 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:32:04.591+0000 7f4d1426cb80 1 -- 172.21.15.52:0/912668490 >> 172.21.15.52:0/912668490 conn(0x558b3c04db70 msgr2=0x558b3c16b6c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:32:04.592 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:32:04.591+0000 7f4d1426cb80 1 -- 172.21.15.52:0/912668490 shutdown_connections 2024-05-27T07:32:04.592 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:32:04.591+0000 7f4d1426cb80 1 -- 172.21.15.52:0/912668490 wait complete. 2024-05-27T07:32:04.592 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:32:04.592+0000 7f4d1426cb80 1 -- start start 2024-05-27T07:32:04.592 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:32:04.592+0000 7f4d1426cb80 1 Processor -- start 2024-05-27T07:32:04.592 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:32:04.592+0000 7f4d1426cb80 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x558b3bff2ec0 0x558b3c17a4e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:32:04.592 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:32:04.592+0000 7f4d1426cb80 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x558b3c171000 0x558b3c17aa20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:32:04.593 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:32:04.592+0000 7f4d1426cb80 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x558b3c176cd0 0x558b3c17af60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:32:04.593 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:32:04.592+0000 7f4d1426cb80 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x558b3bf96020 con 0x558b3bff2ec0 2024-05-27T07:32:04.593 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:32:04.592+0000 7f4d1426cb80 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x558b3c031840 con 0x558b3c171000 2024-05-27T07:32:04.593 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:32:04.592+0000 7f4d1426cb80 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x558b3c17d7c0 con 0x558b3c176cd0 2024-05-27T07:32:04.593 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:32:04.592+0000 7f4d1426cb80 10 client.0 _ll_register_callbacks cb 0x558b3c02a9a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T07:32:04.593 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:32:04.592+0000 7f4cfeffd700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x558b3bff2ec0 0x558b3c17a4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:32:04.593 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:32:04.592+0000 7f4cfeffd700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x558b3bff2ec0 0x558b3c17a4e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.52:47390/0 (socket says 172.21.15.52:47390) 2024-05-27T07:32:04.593 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:32:04.592+0000 7f4cfeffd700 1 -- 172.21.15.52:0/4149998873 learned_addr learned my addr 172.21.15.52:0/4149998873 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-05-27T07:32:04.594 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:32:04.593+0000 7f4cfe7fc700 1 --2- 172.21.15.52:0/4149998873 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x558b3c171000 0x558b3c17aa20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:32:04.594 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:32:04.593+0000 7f4cfeffd700 1 -- 172.21.15.52:0/4149998873 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x558b3c176cd0 msgr2=0x558b3c17af60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:32:04.594 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:32:04.593+0000 7f4cfeffd700 1 --2- 172.21.15.52:0/4149998873 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x558b3c176cd0 0x558b3c17af60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:32:04.594 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:32:04.593+0000 7f4cfeffd700 1 -- 172.21.15.52:0/4149998873 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x558b3c171000 msgr2=0x558b3c17aa20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:32:04.594 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:32:04.593+0000 7f4cfeffd700 1 --2- 172.21.15.52:0/4149998873 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x558b3c171000 0x558b3c17aa20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:32:04.594 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:32:04.593+0000 7f4cfeffd700 1 -- 172.21.15.52:0/4149998873 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({osdmap=0}) -- 0x558b3c02e710 con 0x558b3bff2ec0 2024-05-27T07:32:04.595 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:32:04.593+0000 7f4cfe7fc700 1 --2- 172.21.15.52:0/4149998873 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x558b3c171000 0x558b3c17aa20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-27T07:32:04.595 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:32:04.593+0000 7f4cfeffd700 1 --2- 172.21.15.52:0/4149998873 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x558b3bff2ec0 0x558b3c17a4e0 secure :-1 s=READY pgs=673 cs=0 l=1 rev1=1 crypto rx=0x7f4ce80e0370 tx=0x7f4ce80a9e20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4a1ba482112d6631 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:32:04.595 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:32:04.593+0000 7f4cdeffd700 10 client.0 ms_handle_connect on v2:172.21.15.52:3300/0 2024-05-27T07:32:04.595 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:32:04.593+0000 7f4cdeffd700 1 -- 172.21.15.52:0/4149998873 <== mon.0 v2:172.21.15.52:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4ce80adcb0 con 0x558b3bff2ec0 2024-05-27T07:32:04.595 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:32:04.593+0000 7f4cdeffd700 1 -- 172.21.15.52:0/4149998873 <== mon.0 v2:172.21.15.52:3300/0 2 ==== osd_map(116..116 src has 1..116) ==== 9379+0+0 (secure 0 0 0) 0x7f4ce80b24e0 con 0x558b3bff2ec0 2024-05-27T07:32:04.595 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:32:04.593+0000 7f4d1426cb80 1 -- 172.21.15.52:0/4149998873 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f4ce80a9900 con 0x558b3bff2ec0 2024-05-27T07:32:04.595 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:32:04.593+0000 7f4cdeffd700 1 -- 172.21.15.52:0/4149998873 <== mon.0 v2:172.21.15.52:3300/0 3 ==== mon_get_version_reply(handle=1 version=101) ==== 24+0+0 (secure 0 0 0) 0x7f4ce80aa000 con 0x558b3bff2ec0 2024-05-27T07:32:04.595 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:32:04.593+0000 7f4d1426cb80 10 client.16725 fetch_fsmap learned FSMap version 101 2024-05-27T07:32:04.595 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:32:04.593+0000 7f4d1426cb80 1 -- 172.21.15.52:0/4149998873 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({config=0+,fsmap=101,monmap=4+}) -- 0x558b3c188340 con 0x558b3bff2ec0 2024-05-27T07:32:04.595 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:32:04.593+0000 7f4cdeffd700 1 -- 172.21.15.52:0/4149998873 <== mon.0 v2:172.21.15.52:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f4ce80a9900 con 0x558b3bff2ec0 2024-05-27T07:32:04.595 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:32:04.594+0000 7f4cdeffd700 1 -- 172.21.15.52:0/4149998873 <== mon.0 v2:172.21.15.52:3300/0 5 ==== fsmap(e 101) ==== 4203+0+0 (secure 0 0 0) 0x7f4ce80c7020 con 0x558b3bff2ec0 2024-05-27T07:32:04.595 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:32:04.594+0000 7f4d1426cb80 10 client.16725 fetch_fsmap finished waiting for FSMap version 101 2024-05-27T07:32:04.596 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:32:04.594+0000 7f4d1426cb80 10 client.16725 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-05-27T07:32:04.596 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:32:04.594+0000 7f4d1426cb80 20 client.16725 populate_metadata read hostname 'smithi052' 2024-05-27T07:32:04.596 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:32:04.594+0000 7f4d1426cb80 1 --2- 172.21.15.52:0/4149998873 >> [v2:172.21.15.105:6838/500348151,v1:172.21.15.105:6839/500348151] conn(0x558b3c18aad0 0x558b3c18cfa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:32:04.596 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:32:04.594+0000 7f4d1426cb80 4 client.16725 mds_command: new command op to 24527 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-27T07:32:04.596 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:32:04.594+0000 7f4d1426cb80 1 -- 172.21.15.52:0/4149998873 --> [v2:172.21.15.105:6838/500348151,v1:172.21.15.105:6839/500348151] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x558b3c13ca70 con 0x558b3c18aad0 2024-05-27T07:32:04.596 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:32:04.594+0000 7f4cfe7fc700 1 --2- 172.21.15.52:0/4149998873 >> [v2:172.21.15.105:6838/500348151,v1:172.21.15.105:6839/500348151] conn(0x558b3c18aad0 0x558b3c18cfa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:32:04.596 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:32:04.594+0000 7f4cfe7fc700 1 --2- 172.21.15.52:0/4149998873 >> [v2:172.21.15.105:6838/500348151,v1:172.21.15.105:6839/500348151] conn(0x558b3c18aad0 0x558b3c18cfa0 crc :-1 s=READY pgs=123 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=771b815281225bc4 server_cookie=b29cfa61f5834abb in_seq=0 out_seq=0 2024-05-27T07:32:04.596 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:32:04.594+0000 7f4cdeffd700 10 client.16725 ms_handle_connect on v2:172.21.15.105:6838/500348151 2024-05-27T07:32:04.596 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:32:04.595+0000 7f4cdeffd700 1 -- 172.21.15.52:0/4149998873 <== mds.0 v2:172.21.15.105:6838/500348151 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24041 (crc 0 0 0) 0x558b3c13ca70 con 0x558b3c18aad0 2024-05-27T07:32:04.596 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:32:04.595+0000 7f4cdeffd700 10 client.16725 handle_command_reply: tid=0 2024-05-27T07:32:04.596 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:32:04.595+0000 7f4d1426cb80 1 client.16725 shutdown 2024-05-27T07:32:04.596 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:32:04.596+0000 7f4d1426cb80 1 -- 172.21.15.52:0/4149998873 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x558b3bff2ec0 msgr2=0x558b3c17a4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:32:04.597 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:32:04.596+0000 7f4d1426cb80 1 --2- 172.21.15.52:0/4149998873 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x558b3bff2ec0 0x558b3c17a4e0 secure :-1 s=READY pgs=673 cs=0 l=1 rev1=1 crypto rx=0x7f4ce80e0370 tx=0x7f4ce80a9e20 comp rx=0 tx=0).stop 2024-05-27T07:32:04.597 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:32:04.596+0000 7f4d1426cb80 1 -- 172.21.15.52:0/4149998873 shutdown_connections 2024-05-27T07:32:04.597 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:32:04.596+0000 7f4d1426cb80 1 --2- 172.21.15.52:0/4149998873 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x558b3c176cd0 0x558b3c17af60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:32:04.597 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:32:04.596+0000 7f4d1426cb80 1 --2- 172.21.15.52:0/4149998873 >> [v2:172.21.15.105:6838/500348151,v1:172.21.15.105:6839/500348151] conn(0x558b3c18aad0 0x558b3c18cfa0 crc :-1 s=READY pgs=123 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:32:04.597 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:32:04.596+0000 7f4d1426cb80 1 --2- 172.21.15.52:0/4149998873 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x558b3bff2ec0 0x558b3c17a4e0 unknown :-1 s=CLOSED pgs=673 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:32:04.597 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:32:04.596+0000 7f4d1426cb80 1 --2- 172.21.15.52:0/4149998873 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x558b3c171000 0x558b3c17aa20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:32:04.597 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:32:04.596+0000 7f4d1426cb80 1 -- 172.21.15.52:0/4149998873 >> 172.21.15.52:0/4149998873 conn(0x558b3c04db70 msgr2=0x558b3c16fab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:32:04.597 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:32:04.596+0000 7f4cdeffd700 0 client.16725 ms_handle_reset on v2:172.21.15.105:6838/500348151 2024-05-27T07:32:04.597 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:32:04.596+0000 7f4d1426cb80 1 -- 172.21.15.52:0/4149998873 shutdown_connections 2024-05-27T07:32:04.597 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:32:04.596+0000 7f4d1426cb80 1 -- 172.21.15.52:0/4149998873 wait complete. 2024-05-27T07:32:04.597 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:32:04.596+0000 7f4d1426cb80 20 client.16725 trim_cache size 0 max 16384 2024-05-27T07:32:04.604 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:32:04 smithi173 ceph-mon[37000]: pgmap v211: 129 pgs: 129 active+clean; 562 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 3.2 MiB/s wr, 1.19k op/s 2024-05-27T07:32:04.611 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:32:04.617+0000 7f4d1426cb80 1 Processor -- start 2024-05-27T07:32:04.611 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:32:04.618+0000 7f4d1426cb80 1 -- start start 2024-05-27T07:32:04.612 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:32:04.618+0000 7f4d1426cb80 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x558b3bff2ec0 0x558b3c170ac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:32:04.612 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:32:04.618+0000 7f4d1426cb80 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x558b3c171000 0x558b3c1794e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:32:04.612 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:32:04.618+0000 7f4d1426cb80 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x558b3c176cd0 0x558b3c192760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:32:04.612 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:32:04.618+0000 7f4d1426cb80 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x7f4d001b0a00 con 0x558b3bff2ec0 2024-05-27T07:32:04.612 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:32:04.618+0000 7f4d1426cb80 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f4d001aee00 con 0x558b3c171000 2024-05-27T07:32:04.612 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:32:04.618+0000 7f4d1426cb80 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f4d001b0330 con 0x558b3c176cd0 2024-05-27T07:32:04.612 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:32:04.618+0000 7f4cfeffd700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x558b3bff2ec0 0x558b3c170ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:32:04.612 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:32:04.618+0000 7f4cff7fe700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x558b3c176cd0 0x558b3c192760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:32:04.612 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:32:04.618+0000 7f4cfeffd700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x558b3bff2ec0 0x558b3c170ac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.52:47402/0 (socket says 172.21.15.52:47402) 2024-05-27T07:32:04.612 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:32:04.618+0000 7f4cfeffd700 1 -- 172.21.15.52:0/1116194347 learned_addr learned my addr 172.21.15.52:0/1116194347 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-05-27T07:32:04.612 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:32:04.618+0000 7f4cfe7fc700 1 --2- 172.21.15.52:0/1116194347 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x558b3c171000 0x558b3c1794e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:32:04.612 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:32:04.618+0000 7f4cfeffd700 1 -- 172.21.15.52:0/1116194347 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x558b3c176cd0 msgr2=0x558b3c192760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:32:04.612 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:32:04.618+0000 7f4cfeffd700 1 --2- 172.21.15.52:0/1116194347 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x558b3c176cd0 0x558b3c192760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:32:04.613 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:32:04.618+0000 7f4cfeffd700 1 -- 172.21.15.52:0/1116194347 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x558b3c171000 msgr2=0x558b3c1794e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:32:04.613 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:32:04.618+0000 7f4cfeffd700 1 --2- 172.21.15.52:0/1116194347 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x558b3c171000 0x558b3c1794e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:32:04.613 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:32:04.618+0000 7f4cfeffd700 1 -- 172.21.15.52:0/1116194347 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x558b3c178a10 con 0x558b3bff2ec0 2024-05-27T07:32:04.613 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:32:04.618+0000 7f4cfe7fc700 1 --2- 172.21.15.52:0/1116194347 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x558b3c171000 0x558b3c1794e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-27T07:32:04.613 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:32:04.619+0000 7f4cfeffd700 1 --2- 172.21.15.52:0/1116194347 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x558b3bff2ec0 0x558b3c170ac0 secure :-1 s=READY pgs=674 cs=0 l=1 rev1=1 crypto rx=0x7f4ce800c940 tx=0x7f4ce80a5030 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4b02f311313d4974 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:32:04.613 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:32:04.619+0000 7f4cf57fa700 1 -- 172.21.15.52:0/1116194347 <== mon.0 v2:172.21.15.52:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4ce80a9900 con 0x558b3bff2ec0 2024-05-27T07:32:04.613 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:32:04.619+0000 7f4cf57fa700 1 -- 172.21.15.52:0/1116194347 <== mon.0 v2:172.21.15.52:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f4ce80b2760 con 0x558b3bff2ec0 2024-05-27T07:32:04.613 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:32:04.619+0000 7f4cf57fa700 1 -- 172.21.15.52:0/1116194347 <== mon.0 v2:172.21.15.52:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4ce8004870 con 0x558b3bff2ec0 2024-05-27T07:32:04.616 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:32:04.620+0000 7f4d1426cb80 1 -- 172.21.15.52:0/1116194347 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x558b3bff2ec0 msgr2=0x558b3c170ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:32:04.616 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:32:04.620+0000 7f4d1426cb80 1 --2- 172.21.15.52:0/1116194347 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x558b3bff2ec0 0x558b3c170ac0 secure :-1 s=READY pgs=674 cs=0 l=1 rev1=1 crypto rx=0x7f4ce800c940 tx=0x7f4ce80a5030 comp rx=0 tx=0).stop 2024-05-27T07:32:04.616 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:32:04.620+0000 7f4d1426cb80 1 -- 172.21.15.52:0/1116194347 shutdown_connections 2024-05-27T07:32:04.616 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:32:04.620+0000 7f4d1426cb80 1 --2- 172.21.15.52:0/1116194347 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x558b3c176cd0 0x558b3c192760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:32:04.616 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:32:04.620+0000 7f4d1426cb80 1 --2- 172.21.15.52:0/1116194347 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x558b3bff2ec0 0x558b3c170ac0 unknown :-1 s=CLOSED pgs=674 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:32:04.616 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:32:04.620+0000 7f4d1426cb80 1 --2- 172.21.15.52:0/1116194347 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x558b3c171000 0x558b3c1794e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:32:04.616 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:32:04.620+0000 7f4d1426cb80 1 -- 172.21.15.52:0/1116194347 >> 172.21.15.52:0/1116194347 conn(0x558b3c04db70 msgr2=0x558b3c178dd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:32:04.617 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:32:04.620+0000 7f4d1426cb80 1 -- 172.21.15.52:0/1116194347 shutdown_connections 2024-05-27T07:32:04.617 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:32:04.620+0000 7f4d1426cb80 1 -- 172.21.15.52:0/1116194347 wait complete. 2024-05-27T07:32:04.617 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:32:04.620+0000 7f4d1426cb80 1 -- start start 2024-05-27T07:32:04.617 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:32:04.620+0000 7f4d1426cb80 1 Processor -- start 2024-05-27T07:32:04.617 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:32:04.621+0000 7f4d1426cb80 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x558b3bff2ec0 0x558b3c182a50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:32:04.617 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:32:04.621+0000 7f4d1426cb80 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x558b3c171000 0x558b3c182f90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:32:04.617 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:32:04.621+0000 7f4d1426cb80 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x558b3c176cd0 0x558b3c17fa20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:32:04.617 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:32:04.621+0000 7f4d1426cb80 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x7f4d001afba0 con 0x558b3bff2ec0 2024-05-27T07:32:04.617 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:32:04.621+0000 7f4d1426cb80 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f4d001af4d0 con 0x558b3c176cd0 2024-05-27T07:32:04.618 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:32:04.621+0000 7f4d1426cb80 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x558b3c13ca70 con 0x558b3c171000 2024-05-27T07:32:04.618 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:32:04.621+0000 7f4d1426cb80 10 client.0 _ll_register_callbacks cb 0x558b3c02a9a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T07:32:04.618 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:32:04.621+0000 7f4cfeffd700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x558b3bff2ec0 0x558b3c182a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:32:04.618 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:32:04.621+0000 7f4cfeffd700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x558b3bff2ec0 0x558b3c182a50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.52:47404/0 (socket says 172.21.15.52:47404) 2024-05-27T07:32:04.618 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:32:04.621+0000 7f4cfeffd700 1 -- 172.21.15.52:0/3476207404 learned_addr learned my addr 172.21.15.52:0/3476207404 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-05-27T07:32:04.618 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:32:04.621+0000 7f4cfe7fc700 1 --2- 172.21.15.52:0/3476207404 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x558b3c171000 0x558b3c182f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:32:04.618 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:32:04.621+0000 7f4cff7fe700 1 --2- 172.21.15.52:0/3476207404 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x558b3c176cd0 0x558b3c17fa20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:32:04.618 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:32:04.621+0000 7f4cfeffd700 1 -- 172.21.15.52:0/3476207404 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x558b3c171000 msgr2=0x558b3c182f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:32:04.618 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:32:04.621+0000 7f4cfeffd700 1 --2- 172.21.15.52:0/3476207404 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x558b3c171000 0x558b3c182f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:32:04.618 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:32:04.621+0000 7f4cfeffd700 1 -- 172.21.15.52:0/3476207404 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x558b3c176cd0 msgr2=0x558b3c17fa20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:32:04.618 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:32:04.621+0000 7f4cfeffd700 1 --2- 172.21.15.52:0/3476207404 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x558b3c176cd0 0x558b3c17fa20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:32:04.619 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:32:04.621+0000 7f4cfeffd700 1 -- 172.21.15.52:0/3476207404 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({osdmap=0}) -- 0x558b3c1835a0 con 0x558b3bff2ec0 2024-05-27T07:32:04.619 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:32:04.622+0000 7f4cfeffd700 1 --2- 172.21.15.52:0/3476207404 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x558b3bff2ec0 0x558b3c182a50 secure :-1 s=READY pgs=675 cs=0 l=1 rev1=1 crypto rx=0x7f4ce80a9890 tx=0x7f4ce80cc2e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e31d7cd4bd6a8e06 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:32:04.619 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:32:04.622+0000 7f4cdeffd700 10 client.0 ms_handle_connect on v2:172.21.15.52:3300/0 2024-05-27T07:32:04.619 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:32:04.622+0000 7f4cdeffd700 1 -- 172.21.15.52:0/3476207404 <== mon.0 v2:172.21.15.52:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4ce80a9900 con 0x558b3bff2ec0 2024-05-27T07:32:04.619 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:32:04.622+0000 7f4cdeffd700 1 -- 172.21.15.52:0/3476207404 <== mon.0 v2:172.21.15.52:3300/0 2 ==== osd_map(116..116 src has 1..116) ==== 9379+0+0 (secure 0 0 0) 0x7f4ce80cabb0 con 0x558b3bff2ec0 2024-05-27T07:32:04.619 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:32:04.622+0000 7f4d1426cb80 1 -- 172.21.15.52:0/3476207404 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f4ce80b2760 con 0x558b3bff2ec0 2024-05-27T07:32:04.619 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:32:04.622+0000 7f4cdeffd700 1 -- 172.21.15.52:0/3476207404 <== mon.0 v2:172.21.15.52:3300/0 3 ==== mon_get_version_reply(handle=1 version=101) ==== 24+0+0 (secure 0 0 0) 0x7f4ce8004870 con 0x558b3bff2ec0 2024-05-27T07:32:04.619 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:32:04.622+0000 7f4d1426cb80 10 client.16731 fetch_fsmap learned FSMap version 101 2024-05-27T07:32:04.620 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:32:04.622+0000 7f4d1426cb80 1 -- 172.21.15.52:0/3476207404 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({config=0+,fsmap=101,monmap=4+}) -- 0x558b3c1835a0 con 0x558b3bff2ec0 2024-05-27T07:32:04.620 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:32:04.622+0000 7f4cdeffd700 1 -- 172.21.15.52:0/3476207404 <== mon.0 v2:172.21.15.52:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f4ce80b2760 con 0x558b3bff2ec0 2024-05-27T07:32:04.620 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:32:04.622+0000 7f4cdeffd700 1 -- 172.21.15.52:0/3476207404 <== mon.0 v2:172.21.15.52:3300/0 5 ==== fsmap(e 101) ==== 4203+0+0 (secure 0 0 0) 0x7f4ce80c7070 con 0x558b3bff2ec0 2024-05-27T07:32:04.620 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:32:04.622+0000 7f4d1426cb80 10 client.16731 fetch_fsmap finished waiting for FSMap version 101 2024-05-27T07:32:04.620 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:32:04.622+0000 7f4d1426cb80 10 client.16731 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-05-27T07:32:04.620 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:32:04.622+0000 7f4d1426cb80 20 client.16731 populate_metadata read hostname 'smithi052' 2024-05-27T07:32:04.620 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:32:04.622+0000 7f4d1426cb80 1 --2- 172.21.15.52:0/3476207404 >> [v2:172.21.15.105:6838/500348151,v1:172.21.15.105:6839/500348151] conn(0x558b3c1f5ea0 0x558b3c1f82c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:32:04.620 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:32:04.622+0000 7f4d1426cb80 4 client.16731 mds_command: new command op to 24527 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-05-27T07:32:04.620 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:32:04.622+0000 7f4d1426cb80 1 -- 172.21.15.52:0/3476207404 --> [v2:172.21.15.105:6838/500348151,v1:172.21.15.105:6839/500348151] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) -- 0x558b3c0a1c60 con 0x558b3c1f5ea0 2024-05-27T07:32:04.620 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:32:04.623+0000 7f4cfe7fc700 1 --2- 172.21.15.52:0/3476207404 >> [v2:172.21.15.105:6838/500348151,v1:172.21.15.105:6839/500348151] conn(0x558b3c1f5ea0 0x558b3c1f82c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:32:04.621 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:32:04.623+0000 7f4cfe7fc700 1 --2- 172.21.15.52:0/3476207404 >> [v2:172.21.15.105:6838/500348151,v1:172.21.15.105:6839/500348151] conn(0x558b3c1f5ea0 0x558b3c1f82c0 crc :-1 s=READY pgs=124 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=34baeabbe426c7bb server_cookie=8f64d6f80d296a51 in_seq=0 out_seq=0 2024-05-27T07:32:04.621 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:32:04.623+0000 7f4cdeffd700 10 client.16731 ms_handle_connect on v2:172.21.15.105:6838/500348151 2024-05-27T07:32:04.622 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:32:04.624+0000 7f4cdeffd700 1 -- 172.21.15.52:0/3476207404 <== mds.0 v2:172.21.15.105:6838/500348151 1 ==== command_reply(tid 0: 0 ) ==== 8+0+110 (crc 0 0 0) 0x558b3c0a1c60 con 0x558b3c1f5ea0 2024-05-27T07:32:04.622 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:32:04.624+0000 7f4cdeffd700 10 client.16731 handle_command_reply: tid=0 2024-05-27T07:32:04.622 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:32:04.624+0000 7f4d1426cb80 1 client.16731 shutdown 2024-05-27T07:32:04.622 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:32:04.625+0000 7f4d1426cb80 1 -- 172.21.15.52:0/3476207404 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x558b3bff2ec0 msgr2=0x558b3c182a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:32:04.622 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:32:04.625+0000 7f4d1426cb80 1 --2- 172.21.15.52:0/3476207404 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x558b3bff2ec0 0x558b3c182a50 secure :-1 s=READY pgs=675 cs=0 l=1 rev1=1 crypto rx=0x7f4ce80a9890 tx=0x7f4ce80cc2e0 comp rx=0 tx=0).stop 2024-05-27T07:32:04.622 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:32:04.625+0000 7f4d1426cb80 1 -- 172.21.15.52:0/3476207404 shutdown_connections 2024-05-27T07:32:04.623 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:32:04.625+0000 7f4d1426cb80 1 --2- 172.21.15.52:0/3476207404 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x558b3c176cd0 0x558b3c17fa20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:32:04.623 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:32:04.625+0000 7f4d1426cb80 1 --2- 172.21.15.52:0/3476207404 >> [v2:172.21.15.105:6838/500348151,v1:172.21.15.105:6839/500348151] conn(0x558b3c1f5ea0 0x558b3c1f82c0 crc :-1 s=READY pgs=124 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:32:04.623 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:32:04.625+0000 7f4d1426cb80 1 --2- 172.21.15.52:0/3476207404 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x558b3bff2ec0 0x558b3c182a50 unknown :-1 s=CLOSED pgs=675 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:32:04.623 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:32:04.625+0000 7f4d1426cb80 1 --2- 172.21.15.52:0/3476207404 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x558b3c171000 0x558b3c182f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:32:04.623 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:32:04.625+0000 7f4cdeffd700 0 client.16731 ms_handle_reset on v2:172.21.15.105:6838/500348151 2024-05-27T07:32:04.623 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:32:04.625+0000 7f4d1426cb80 1 -- 172.21.15.52:0/3476207404 >> 172.21.15.52:0/3476207404 conn(0x558b3c04db70 msgr2=0x558b3c174090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:32:04.623 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:32:04.625+0000 7f4d1426cb80 1 -- 172.21.15.52:0/3476207404 shutdown_connections 2024-05-27T07:32:04.623 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:32:04.625+0000 7f4d1426cb80 1 -- 172.21.15.52:0/3476207404 wait complete. 2024-05-27T07:32:04.623 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:32:04.625+0000 7f4d1426cb80 20 client.16731 trim_cache size 0 max 16384 2024-05-27T07:32:04.624 INFO:teuthology.orchestra.run.smithi052.stdout:{ 2024-05-27T07:32:04.624 INFO:teuthology.orchestra.run.smithi052.stdout: "return_code": 0, 2024-05-27T07:32:04.624 INFO:teuthology.orchestra.run.smithi052.stdout: "scrub_tag": "36f21956-1be9-436b-b6db-8bc2e943eb70", 2024-05-27T07:32:04.624 INFO:teuthology.orchestra.run.smithi052.stdout: "mode": "asynchronous" 2024-05-27T07:32:04.624 INFO:teuthology.orchestra.run.smithi052.stdout:} 2024-05-27T07:32:04.625 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:32:04.626+0000 7f4d05857700 1 -- 172.21.15.52:0/631726525 >> [v2:172.21.15.52:6800/1822318384,v1:172.21.15.52:6801/1822318384] conn(0x7f4cec0b29f0 msgr2=0x7f4cec0b4f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:32:04.625 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:32:04.626+0000 7f4d05857700 1 --2- 172.21.15.52:0/631726525 >> [v2:172.21.15.52:6800/1822318384,v1:172.21.15.52:6801/1822318384] conn(0x7f4cec0b29f0 0x7f4cec0b4f00 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7f4d00106780 tx=0x7f4cf0006c60 comp rx=0 tx=0).stop 2024-05-27T07:32:04.625 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:32:04.626+0000 7f4d05857700 1 -- 172.21.15.52:0/631726525 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f4d00103950 msgr2=0x7f4d0019bc00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:32:04.625 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:32:04.626+0000 7f4d05857700 1 --2- 172.21.15.52:0/631726525 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f4d00103950 0x7f4d0019bc00 secure :-1 s=READY pgs=671 cs=0 l=1 rev1=1 crypto rx=0x7f4ce800c970 tx=0x7f4ce800cd30 comp rx=0 tx=0).stop 2024-05-27T07:32:04.626 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:32:04.626+0000 7f4d05857700 1 -- 172.21.15.52:0/631726525 shutdown_connections 2024-05-27T07:32:04.626 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:32:04.626+0000 7f4d05857700 1 --2- 172.21.15.52:0/631726525 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f4d001081f0 0x7f4d001a6c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:32:04.626 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:32:04.626+0000 7f4d05857700 1 --2- 172.21.15.52:0/631726525 >> [v2:172.21.15.52:6800/1822318384,v1:172.21.15.52:6801/1822318384] conn(0x7f4cec0b29f0 0x7f4cec0b4f00 unknown :-1 s=CLOSED pgs=154 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:32:04.626 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:32:04.626+0000 7f4d05857700 1 --2- 172.21.15.52:0/631726525 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f4d00103950 0x7f4d0019bc00 unknown :-1 s=CLOSED pgs=671 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:32:04.626 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:32:04.626+0000 7f4d05857700 1 --2- 172.21.15.52:0/631726525 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f4d001042a0 0x7f4d0019c140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:32:04.626 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:32:04.626+0000 7f4d05857700 1 -- 172.21.15.52:0/631726525 >> 172.21.15.52:0/631726525 conn(0x7f4d000fe4d0 msgr2=0x7f4d001075c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:32:04.626 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:32:04.626+0000 7f4d05857700 1 -- 172.21.15.52:0/631726525 shutdown_connections 2024-05-27T07:32:04.626 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:32:04.626+0000 7f4d05857700 1 -- 172.21.15.52:0/631726525 wait complete. 2024-05-27T07:32:05.220 INFO:tasks.workunit.client.0.smithi052.stdout: 142 221485 146 157809 88 133026 510 2024-05-27T07:32:05.443 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:32:05 smithi105 ceph-mon[35977]: scrub queued for path: / 2024-05-27T07:32:05.443 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:32:05 smithi105 ceph-mon[35977]: scrub summary: idle+waiting paths [/] 2024-05-27T07:32:05.443 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:32:05 smithi105 ceph-mon[35977]: scrub summary: active paths [/] 2024-05-27T07:32:05.443 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:32:05 smithi105 ceph-mon[35977]: scrub summary: active paths [/] 2024-05-27T07:32:05.443 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:32:05 smithi105 ceph-mon[35977]: scrub summary: idle+waiting paths [/] 2024-05-27T07:32:05.443 INFO:journalctl@ceph.mds.b.smithi105.stdout:May 27 07:32:05 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-b[65543]: 2024-05-27T07:32:05.061+0000 7f33fced0700 -1 mds.0.scrubstack _validate_inode_done scrub error on inode [inode 0x10000001cb9 [...7,head] /volumes/qa/sv_1/95e9ac73-825f-48e3-998b-133d94d96559/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-31/ auth{2=1} fragtree_t(*^3) v235369 ap=2 RANDEPHEMERALPIN f(v13 m2024-05-27T07:31:22.044938+0000 260=260+0) n(v18 rc2024-05-27T07:31:22.044938+0000 b4665310 261=260+1) old_inodes=3 (inest sync->mix g=2) caps={25060=pAsLsXsFs/pAsLsXsFsx@236} | dirtyscattered=0 request=0 lock=0 dirfrag=8 caps=1 dirtyrstat=0 dirtyparent=0 replicated=1 dirty=0 waiter=0 authpin=1 discoverbase=0 scrubqueue=0 randepin 0x55ee51210580]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":true,"read_ret_val":0,"ondisk_value":"(2)0x10000001cb9:[<0x10000000399/blog-31 v55383>,<0x10000000208/blogtest_in v437>,<0x10000000201/src v2033>,<0x10000000006/blogbench-1.0 v1794>,<0x10000000005/tmp v93>,<0x100000001fb/client.0 v85>,<0x100000001fa/95e9ac73-825f-48e3-998b-133d94d96559 v98>,<0x10000000001/sv_1 v113>,<0x10000000000/qa v105>,<0x1/volumes v102>]//[]","memoryvalue":"(2)0x10000001cb9:[<0x10000000399/blog-31 v235369>,<0x10000000208/blogtest_in v1568>,<0x10000000201/src v2432>,<0x10000000006/blogbench-1.0 v2163>,<0x10000000005/tmp v453>,<0x100000001fb/client.0 v419>,<0x100000001fa/95e9ac73-825f-48e3-998b-133d94d96559 v424>,<0x10000000001/sv_1 v447>,<0x10000000000/qa v421>,<0x1/volumes v430>]//[]","error_str":""},"raw_stats":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f(v0 m2024-05-27T07:31:21.250947+0000 259=259+0)","ondisk_value.rstat":"n(v0 rc2024-05-27T07:31:22.044938+0000 b4665310 261=260+1)","memory_value.dirstat":"f(v13 m2024-05-27T07:31:22.044938+0000 260=260+0)","memory_value.rstat":"n(v18 rc2024-05-27T07:31:22.044938+0000 b4665310 261=260+1)","error_str":"freshly-calculated rstats don't match existing ones"},"return_code":0} 2024-05-27T07:32:05.443 INFO:journalctl@ceph.mds.b.smithi105.stdout:May 27 07:32:05 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-b[65543]: 2024-05-27T07:32:05.070+0000 7f33fced0700 -1 mds.0.scrubstack _validate_inode_done scrub error on inode [inode 0x1000000803f [...7,head] /volumes/qa/sv_1/95e9ac73-825f-48e3-998b-133d94d96559/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-125/ auth{1=1} v239937 ap=2 RANDEPHEMERALPIN f(v2 m2024-05-27T07:31:34.999788+0000 66=66+0) n(v4 rc2024-05-27T07:31:34.999788+0000 b1912722 67=66+1) (inest sync->mix g=1) caps={25060=pAsLsXsFs/pAsLsXsFsx@52} | dirtyscattered=0 request=0 lock=0 dirfrag=1 caps=1 dirtyrstat=0 dirtyparent=0 replicated=1 dirty=0 waiter=0 authpin=1 discoverbase=0 scrubqueue=0 randepin 0x55ee58461600]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":true,"read_ret_val":0,"ondisk_value":"(2)0x1000000803f:[<0x10000000399/blog-125 v234544>,<0x10000000208/blogtest_in v1385>,<0x10000000201/src v2380>,<0x10000000006/blogbench-1.0 v2115>,<0x10000000005/tmp v403>,<0x100000001fb/client.0 v370>,<0x100000001fa/95e9ac73-825f-48e3-998b-133d94d96559 v380>,<0x10000000001/sv_1 v398>,<0x10000000000/qa v377>,<0x1/volumes v381>]//[]","memoryvalue":"(2)0x1000000803f:[<0x10000000399/blog-125 v239937>,<0x10000000208/blogtest_in v1568>,<0x10000000201/src v2432>,<0x10000000006/blogbench-1.0 v2163>,<0x10000000005/tmp v453>,<0x100000001fb/client.0 v419>,<0x100000001fa/95e9ac73-825f-48e3-998b-133d94d96559 v424>,<0x10000000001/sv_1 v447>,<0x10000000000/qa v421>,<0x1/volumes v430>]//[]","error_str":""},"raw_stats":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f(v0 m2024-05-27T07:31:20.045961+0000 54=54+0)","ondisk_value.rstat":"n(v0 rc2024-05-27T07:31:34.999788+0000 b1912722 67=66+1)","memory_value.dirstat":"f(v2 m2024-05-27T07:31:34.999788+0000 66=66+0)","memory_value.rstat":"n(v4 rc2024-05-27T07:31:34.999788+0000 b1912722 67=66+1)","error_str":"freshly-calculated rstats don't match existing ones"},"return_code":0} 2024-05-27T07:32:05.482 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:32:05 smithi052 ceph-mon[31955]: scrub queued for path: / 2024-05-27T07:32:05.482 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:32:05 smithi052 ceph-mon[31955]: scrub summary: idle+waiting paths [/] 2024-05-27T07:32:05.482 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:32:05 smithi052 ceph-mon[31955]: scrub summary: active paths [/] 2024-05-27T07:32:05.482 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:32:05 smithi052 ceph-mon[31955]: scrub summary: active paths [/] 2024-05-27T07:32:05.482 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:32:05 smithi052 ceph-mon[31955]: scrub summary: idle+waiting paths [/] 2024-05-27T07:32:05.605 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:32:05 smithi173 ceph-mon[37000]: scrub queued for path: / 2024-05-27T07:32:05.605 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:32:05 smithi173 ceph-mon[37000]: scrub summary: idle+waiting paths [/] 2024-05-27T07:32:05.605 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:32:05 smithi173 ceph-mon[37000]: scrub summary: active paths [/] 2024-05-27T07:32:05.605 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:32:05 smithi173 ceph-mon[37000]: scrub summary: active paths [/] 2024-05-27T07:32:05.605 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:32:05 smithi173 ceph-mon[37000]: scrub summary: idle+waiting paths [/] 2024-05-27T07:32:05.634 DEBUG:teuthology.orchestra.run.smithi052:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:38f9d900c83c63799fbdbe61acc9a11b0d3554a6 shell --fsid 73df2636-1bf8-11ef-bc9b-c7b262605968 -- ceph tell mds.1:0 scrub status 2024-05-27T07:32:05.965 INFO:teuthology.orchestra.run.smithi052.stderr:Inferring config /var/lib/ceph/73df2636-1bf8-11ef-bc9b-c7b262605968/mon.a/config 2024-05-27T07:32:06.372 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:32:06 smithi052 ceph-mon[31955]: scrub summary: active paths [/] 2024-05-27T07:32:06.372 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:32:06 smithi052 ceph-mon[31955]: scrub summary: idle+waiting paths [/] 2024-05-27T07:32:06.372 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:32:06 smithi052 ceph-mon[31955]: scrub summary: active paths [/] 2024-05-27T07:32:06.372 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:32:06 smithi052 ceph-mon[31955]: scrub summary: idle+waiting paths [/] 2024-05-27T07:32:06.372 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:32:06 smithi052 ceph-mon[31955]: scrub summary: active paths [/] 2024-05-27T07:32:06.372 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:32:06 smithi052 ceph-mon[31955]: scrub summary: idle+waiting paths [/] 2024-05-27T07:32:06.372 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:32:06 smithi052 ceph-mon[31955]: scrub summary: active paths [/] 2024-05-27T07:32:06.372 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:32:06 smithi052 ceph-mon[31955]: scrub summary: idle+waiting paths [/] 2024-05-27T07:32:06.372 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:32:06 smithi052 ceph-mon[31955]: scrub summary: active paths [/] 2024-05-27T07:32:06.373 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:32:06 smithi052 ceph-mon[31955]: scrub summary: idle+waiting paths [/] 2024-05-27T07:32:06.373 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:32:06 smithi052 ceph-mon[31955]: scrub summary: active paths [/] 2024-05-27T07:32:06.373 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:32:06 smithi052 ceph-mon[31955]: scrub summary: active paths [/] 2024-05-27T07:32:06.373 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:32:06 smithi052 ceph-mon[31955]: scrub summary: idle+waiting paths [/] 2024-05-27T07:32:06.373 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:32:06 smithi052 ceph-mon[31955]: Scrub error on inode 0x10000001cb9 (/volumes/qa/sv_1/95e9ac73-825f-48e3-998b-133d94d96559/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-31) see mds.b log and `damage ls` output for details 2024-05-27T07:32:06.373 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:32:06 smithi052 ceph-mon[31955]: scrub summary: active paths [/] 2024-05-27T07:32:06.373 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:32:06 smithi052 ceph-mon[31955]: scrub summary: idle+waiting paths [/] 2024-05-27T07:32:06.373 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:32:06 smithi052 ceph-mon[31955]: Scrub error on inode 0x1000000803f (/volumes/qa/sv_1/95e9ac73-825f-48e3-998b-133d94d96559/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-125) see mds.b log and `damage ls` output for details 2024-05-27T07:32:06.373 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:32:06 smithi052 ceph-mon[31955]: scrub summary: active paths [/] 2024-05-27T07:32:06.373 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:32:06 smithi052 ceph-mon[31955]: pgmap v212: 129 pgs: 129 active+clean; 545 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 3.1 MiB/s wr, 1.30k op/s 2024-05-27T07:32:06.373 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:32:06 smithi052 ceph-mon[31955]: scrub summary: active paths [/] 2024-05-27T07:32:06.373 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:32:06 smithi052 ceph-mon[31955]: scrub summary: idle+waiting paths [/] 2024-05-27T07:32:06.373 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:32:06 smithi052 ceph-mon[31955]: scrub summary: idle+waiting paths [/] 2024-05-27T07:32:06.374 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:32:06 smithi052 ceph-mon[31955]: scrub summary: idle+waiting paths [/] 2024-05-27T07:32:06.374 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:32:06 smithi052 ceph-mon[31955]: scrub summary: active paths [/] 2024-05-27T07:32:06.374 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:32:06 smithi052 ceph-mon[31955]: scrub summary: active paths [/] 2024-05-27T07:32:06.374 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:32:06 smithi052 ceph-mon[31955]: scrub summary: idle+waiting paths [/] 2024-05-27T07:32:06.374 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:32:06 smithi052 ceph-mon[31955]: scrub summary: idle+waiting paths [/] 2024-05-27T07:32:06.374 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:32:06 smithi052 ceph-mon[31955]: scrub summary: active paths [/] 2024-05-27T07:32:06.374 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:32:06 smithi052 ceph-mon[31955]: scrub summary: active paths [/] 2024-05-27T07:32:06.374 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:32:06 smithi052 ceph-mon[31955]: scrub summary: idle+waiting paths [/] 2024-05-27T07:32:06.374 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:32:06 smithi052 ceph-mon[31955]: scrub summary: idle+waiting paths [/] 2024-05-27T07:32:06.443 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:32:06 smithi105 ceph-mon[35977]: scrub summary: active paths [/] 2024-05-27T07:32:06.443 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:32:06 smithi105 ceph-mon[35977]: scrub summary: idle+waiting paths [/] 2024-05-27T07:32:06.443 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:32:06 smithi105 ceph-mon[35977]: scrub summary: active paths [/] 2024-05-27T07:32:06.443 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:32:06 smithi105 ceph-mon[35977]: scrub summary: idle+waiting paths [/] 2024-05-27T07:32:06.443 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:32:06 smithi105 ceph-mon[35977]: scrub summary: active paths [/] 2024-05-27T07:32:06.443 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:32:06 smithi105 ceph-mon[35977]: scrub summary: idle+waiting paths [/] 2024-05-27T07:32:06.444 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:32:06 smithi105 ceph-mon[35977]: scrub summary: active paths [/] 2024-05-27T07:32:06.444 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:32:06 smithi105 ceph-mon[35977]: scrub summary: idle+waiting paths [/] 2024-05-27T07:32:06.444 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:32:06 smithi105 ceph-mon[35977]: scrub summary: active paths [/] 2024-05-27T07:32:06.444 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:32:06 smithi105 ceph-mon[35977]: scrub summary: idle+waiting paths [/] 2024-05-27T07:32:06.444 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:32:06 smithi105 ceph-mon[35977]: scrub summary: active paths [/] 2024-05-27T07:32:06.444 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:32:06 smithi105 ceph-mon[35977]: scrub summary: active paths [/] 2024-05-27T07:32:06.444 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:32:06 smithi105 ceph-mon[35977]: scrub summary: idle+waiting paths [/] 2024-05-27T07:32:06.444 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:32:06 smithi105 ceph-mon[35977]: Scrub error on inode 0x10000001cb9 (/volumes/qa/sv_1/95e9ac73-825f-48e3-998b-133d94d96559/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-31) see mds.b log and `damage ls` output for details 2024-05-27T07:32:06.444 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:32:06 smithi105 ceph-mon[35977]: scrub summary: active paths [/] 2024-05-27T07:32:06.444 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:32:06 smithi105 ceph-mon[35977]: scrub summary: idle+waiting paths [/] 2024-05-27T07:32:06.444 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:32:06 smithi105 ceph-mon[35977]: Scrub error on inode 0x1000000803f (/volumes/qa/sv_1/95e9ac73-825f-48e3-998b-133d94d96559/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-125) see mds.b log and `damage ls` output for details 2024-05-27T07:32:06.444 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:32:06 smithi105 ceph-mon[35977]: scrub summary: active paths [/] 2024-05-27T07:32:06.444 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:32:06 smithi105 ceph-mon[35977]: pgmap v212: 129 pgs: 129 active+clean; 545 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 3.1 MiB/s wr, 1.30k op/s 2024-05-27T07:32:06.445 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:32:06 smithi105 ceph-mon[35977]: scrub summary: active paths [/] 2024-05-27T07:32:06.445 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:32:06 smithi105 ceph-mon[35977]: scrub summary: idle+waiting paths [/] 2024-05-27T07:32:06.445 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:32:06 smithi105 ceph-mon[35977]: scrub summary: idle+waiting paths [/] 2024-05-27T07:32:06.445 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:32:06 smithi105 ceph-mon[35977]: scrub summary: idle+waiting paths [/] 2024-05-27T07:32:06.445 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:32:06 smithi105 ceph-mon[35977]: scrub summary: active paths [/] 2024-05-27T07:32:06.445 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:32:06 smithi105 ceph-mon[35977]: scrub summary: active paths [/] 2024-05-27T07:32:06.445 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:32:06 smithi105 ceph-mon[35977]: scrub summary: idle+waiting paths [/] 2024-05-27T07:32:06.445 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:32:06 smithi105 ceph-mon[35977]: scrub summary: idle+waiting paths [/] 2024-05-27T07:32:06.445 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:32:06 smithi105 ceph-mon[35977]: scrub summary: active paths [/] 2024-05-27T07:32:06.445 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:32:06 smithi105 ceph-mon[35977]: scrub summary: active paths [/] 2024-05-27T07:32:06.445 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:32:06 smithi105 ceph-mon[35977]: scrub summary: idle+waiting paths [/] 2024-05-27T07:32:06.445 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:32:06 smithi105 ceph-mon[35977]: scrub summary: idle+waiting paths [/] 2024-05-27T07:32:06.605 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:32:06 smithi173 ceph-mon[37000]: scrub summary: active paths [/] 2024-05-27T07:32:06.605 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:32:06 smithi173 ceph-mon[37000]: scrub summary: idle+waiting paths [/] 2024-05-27T07:32:06.605 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:32:06 smithi173 ceph-mon[37000]: scrub summary: active paths [/] 2024-05-27T07:32:06.605 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:32:06 smithi173 ceph-mon[37000]: scrub summary: idle+waiting paths [/] 2024-05-27T07:32:06.605 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:32:06 smithi173 ceph-mon[37000]: scrub summary: active paths [/] 2024-05-27T07:32:06.605 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:32:06 smithi173 ceph-mon[37000]: scrub summary: idle+waiting paths [/] 2024-05-27T07:32:06.605 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:32:06 smithi173 ceph-mon[37000]: scrub summary: active paths [/] 2024-05-27T07:32:06.605 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:32:06 smithi173 ceph-mon[37000]: scrub summary: idle+waiting paths [/] 2024-05-27T07:32:06.605 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:32:06 smithi173 ceph-mon[37000]: scrub summary: active paths [/] 2024-05-27T07:32:06.606 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:32:06 smithi173 ceph-mon[37000]: scrub summary: idle+waiting paths [/] 2024-05-27T07:32:06.606 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:32:06 smithi173 ceph-mon[37000]: scrub summary: active paths [/] 2024-05-27T07:32:06.606 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:32:06 smithi173 ceph-mon[37000]: scrub summary: active paths [/] 2024-05-27T07:32:06.606 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:32:06 smithi173 ceph-mon[37000]: scrub summary: idle+waiting paths [/] 2024-05-27T07:32:06.606 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:32:06 smithi173 ceph-mon[37000]: Scrub error on inode 0x10000001cb9 (/volumes/qa/sv_1/95e9ac73-825f-48e3-998b-133d94d96559/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-31) see mds.b log and `damage ls` output for details 2024-05-27T07:32:06.606 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:32:06 smithi173 ceph-mon[37000]: scrub summary: active paths [/] 2024-05-27T07:32:06.606 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:32:06 smithi173 ceph-mon[37000]: scrub summary: idle+waiting paths [/] 2024-05-27T07:32:06.606 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:32:06 smithi173 ceph-mon[37000]: Scrub error on inode 0x1000000803f (/volumes/qa/sv_1/95e9ac73-825f-48e3-998b-133d94d96559/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-125) see mds.b log and `damage ls` output for details 2024-05-27T07:32:06.606 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:32:06 smithi173 ceph-mon[37000]: scrub summary: active paths [/] 2024-05-27T07:32:06.606 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:32:06 smithi173 ceph-mon[37000]: pgmap v212: 129 pgs: 129 active+clean; 545 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 3.1 MiB/s wr, 1.30k op/s 2024-05-27T07:32:06.606 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:32:06 smithi173 ceph-mon[37000]: scrub summary: active paths [/] 2024-05-27T07:32:06.606 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:32:06 smithi173 ceph-mon[37000]: scrub summary: idle+waiting paths [/] 2024-05-27T07:32:06.606 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:32:06 smithi173 ceph-mon[37000]: scrub summary: idle+waiting paths [/] 2024-05-27T07:32:06.607 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:32:06 smithi173 ceph-mon[37000]: scrub summary: idle+waiting paths [/] 2024-05-27T07:32:06.607 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:32:06 smithi173 ceph-mon[37000]: scrub summary: active paths [/] 2024-05-27T07:32:06.607 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:32:06 smithi173 ceph-mon[37000]: scrub summary: active paths [/] 2024-05-27T07:32:06.607 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:32:06 smithi173 ceph-mon[37000]: scrub summary: idle+waiting paths [/] 2024-05-27T07:32:06.607 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:32:06 smithi173 ceph-mon[37000]: scrub summary: idle+waiting paths [/] 2024-05-27T07:32:06.607 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:32:06 smithi173 ceph-mon[37000]: scrub summary: active paths [/] 2024-05-27T07:32:06.607 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:32:06 smithi173 ceph-mon[37000]: scrub summary: active paths [/] 2024-05-27T07:32:06.607 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:32:06 smithi173 ceph-mon[37000]: scrub summary: idle+waiting paths [/] 2024-05-27T07:32:06.607 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:32:06 smithi173 ceph-mon[37000]: scrub summary: idle+waiting paths [/] 2024-05-27T07:32:07.711 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:32:07.717+0000 7f63ff939700 1 -- 172.21.15.52:0/3339488800 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f63f8107de0 msgr2=0x7f63f81081f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:32:07.711 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:32:07.717+0000 7f63ff939700 1 --2- 172.21.15.52:0/3339488800 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f63f8107de0 0x7f63f81081f0 secure :-1 s=READY pgs=676 cs=0 l=1 rev1=1 crypto rx=0x7f63e80099c0 tx=0x7f63e8009cd0 comp rx=0 tx=0).stop 2024-05-27T07:32:07.711 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:32:07.717+0000 7f63ff939700 1 -- 172.21.15.52:0/3339488800 shutdown_connections 2024-05-27T07:32:07.712 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:32:07.717+0000 7f63ff939700 1 --2- 172.21.15.52:0/3339488800 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f63f8108730 0x7f63f8110090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:32:07.712 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:32:07.717+0000 7f63ff939700 1 --2- 172.21.15.52:0/3339488800 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f63f81034b0 0x7f63f81038c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:32:07.712 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:32:07.717+0000 7f63ff939700 1 --2- 172.21.15.52:0/3339488800 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f63f8107de0 0x7f63f81081f0 unknown :-1 s=CLOSED pgs=676 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:32:07.712 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:32:07.717+0000 7f63ff939700 1 -- 172.21.15.52:0/3339488800 >> 172.21.15.52:0/3339488800 conn(0x7f63f80fedb0 msgr2=0x7f63f81011d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:32:07.712 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:32:07.717+0000 7f63ff939700 1 -- 172.21.15.52:0/3339488800 shutdown_connections 2024-05-27T07:32:07.712 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:32:07.717+0000 7f63ff939700 1 -- 172.21.15.52:0/3339488800 wait complete. 2024-05-27T07:32:07.712 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:32:07.718+0000 7f63ff939700 1 Processor -- start 2024-05-27T07:32:07.712 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:32:07.718+0000 7f63ff939700 1 -- start start 2024-05-27T07:32:07.714 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:32:07.719+0000 7f63ff939700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f63f81034b0 0x7f63f80ff290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:32:07.714 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:32:07.719+0000 7f63ff939700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f63f8107de0 0x7f63f80ff7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:32:07.714 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:32:07.719+0000 7f63ff939700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f63f8108730 0x7f63f80ffd10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:32:07.714 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:32:07.719+0000 7f63ff939700 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x7f63f8113390 con 0x7f63f8108730 2024-05-27T07:32:07.714 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:32:07.719+0000 7f63ff939700 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f63f8113210 con 0x7f63f8107de0 2024-05-27T07:32:07.715 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:32:07.719+0000 7f63ff939700 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f63f8113510 con 0x7f63f81034b0 2024-05-27T07:32:07.715 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:32:07.719+0000 7f63fced4700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f63f8107de0 0x7f63f80ff7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:32:07.715 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:32:07.719+0000 7f63fced4700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f63f8107de0 0x7f63f80ff7d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.52:57262/0 (socket says 172.21.15.52:57262) 2024-05-27T07:32:07.715 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:32:07.719+0000 7f63fced4700 1 -- 172.21.15.52:0/3057778159 learned_addr learned my addr 172.21.15.52:0/3057778159 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-05-27T07:32:07.715 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:32:07.719+0000 7f63fded6700 1 --2- 172.21.15.52:0/3057778159 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f63f8108730 0x7f63f80ffd10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:32:07.715 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:32:07.719+0000 7f63fd6d5700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f63f81034b0 0x7f63f80ff290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:32:07.715 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:32:07.719+0000 7f63fced4700 1 -- 172.21.15.52:0/3057778159 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f63f81034b0 msgr2=0x7f63f80ff290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:32:07.715 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:32:07.719+0000 7f63fced4700 1 --2- 172.21.15.52:0/3057778159 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f63f81034b0 0x7f63f80ff290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:32:07.716 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:32:07.719+0000 7f63fced4700 1 -- 172.21.15.52:0/3057778159 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f63f8108730 msgr2=0x7f63f80ffd10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:32:07.716 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:32:07.719+0000 7f63fced4700 1 --2- 172.21.15.52:0/3057778159 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f63f8108730 0x7f63f80ffd10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:32:07.716 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:32:07.719+0000 7f63fced4700 1 -- 172.21.15.52:0/3057778159 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f63f81a91f0 con 0x7f63f8107de0 2024-05-27T07:32:07.716 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:32:07.719+0000 7f63fced4700 1 --2- 172.21.15.52:0/3057778159 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f63f8107de0 0x7f63f80ff7d0 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7f63e800b580 tx=0x7f63e80045e0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:32:07.716 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:32:07.720+0000 7f63ee7fc700 1 -- 172.21.15.52:0/3057778159 <== mon.2 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f63e8009e60 con 0x7f63f8107de0 2024-05-27T07:32:07.716 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:32:07.720+0000 7f63ee7fc700 1 -- 172.21.15.52:0/3057778159 <== mon.2 v2:172.21.15.105:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f63e80047c0 con 0x7f63f8107de0 2024-05-27T07:32:07.716 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:32:07.720+0000 7f63ff939700 1 -- 172.21.15.52:0/3057778159 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f63f81a9480 con 0x7f63f8107de0 2024-05-27T07:32:07.716 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:32:07.720+0000 7f63ee7fc700 1 -- 172.21.15.52:0/3057778159 <== mon.2 v2:172.21.15.105:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f63e8005270 con 0x7f63f8107de0 2024-05-27T07:32:07.716 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:32:07.720+0000 7f63ff939700 1 -- 172.21.15.52:0/3057778159 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f63f81a9870 con 0x7f63f8107de0 2024-05-27T07:32:07.719 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:32:07.723+0000 7f63ee7fc700 1 -- 172.21.15.52:0/3057778159 <== mon.2 v2:172.21.15.105:3300/0 4 ==== mgrmap(e 26) ==== 149783+0+0 (secure 0 0 0) 0x7f63e8005410 con 0x7f63f8107de0 2024-05-27T07:32:07.719 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:32:07.724+0000 7f63ee7fc700 1 --2- 172.21.15.52:0/3057778159 >> [v2:172.21.15.52:6800/1822318384,v1:172.21.15.52:6801/1822318384] conn(0x7f63e40b29f0 0x7f63e40b4f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:32:07.719 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:32:07.724+0000 7f640e34eb80 1 Processor -- start 2024-05-27T07:32:07.719 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:32:07.725+0000 7f640e34eb80 1 -- start start 2024-05-27T07:32:07.719 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:32:07.725+0000 7f63fd6d5700 1 --2- 172.21.15.52:0/3057778159 >> [v2:172.21.15.52:6800/1822318384,v1:172.21.15.52:6801/1822318384] conn(0x7f63e40b29f0 0x7f63e40b4f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:32:07.719 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:32:07.725+0000 7f63ee7fc700 1 -- 172.21.15.52:0/3057778159 <== mon.2 v2:172.21.15.105:3300/0 5 ==== osd_map(116..116 src has 1..116) ==== 9379+0+0 (secure 0 0 0) 0x7f63e80e3330 con 0x7f63f8107de0 2024-05-27T07:32:07.719 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:32:07.725+0000 7f640e34eb80 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5623f36d7e40 0x5623f385cb30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:32:07.719 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:32:07.725+0000 7f640e34eb80 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x5623f385d070 0x5623f3856be0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:32:07.719 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:32:07.725+0000 7f640e34eb80 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5623f3857120 0x5623f3857530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:32:07.719 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:32:07.725+0000 7f640e34eb80 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x5623f3784d70 con 0x5623f385d070 2024-05-27T07:32:07.719 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:32:07.725+0000 7f640e34eb80 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x5623f3774910 con 0x5623f3857120 2024-05-27T07:32:07.720 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:32:07.725+0000 7f640e34eb80 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x5623f3778950 con 0x5623f36d7e40 2024-05-27T07:32:07.720 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:32:07.725+0000 7f63fced4700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x5623f385d070 0x5623f3856be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:32:07.720 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:32:07.725+0000 7f63fced4700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x5623f385d070 0x5623f3856be0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.52:47434/0 (socket says 172.21.15.52:47434) 2024-05-27T07:32:07.720 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:32:07.725+0000 7f63fced4700 1 -- 172.21.15.52:0/2606498786 learned_addr learned my addr 172.21.15.52:0/2606498786 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-05-27T07:32:07.720 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:32:07.725+0000 7f63fd6d5700 1 --2- 172.21.15.52:0/2606498786 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5623f36d7e40 0x5623f385cb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:32:07.720 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:32:07.725+0000 7f63fded6700 1 --2- 172.21.15.52:0/2606498786 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5623f3857120 0x5623f3857530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:32:07.720 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:32:07.725+0000 7f63fd6d5700 1 --2- 172.21.15.52:0/3057778159 >> [v2:172.21.15.52:6800/1822318384,v1:172.21.15.52:6801/1822318384] conn(0x7f63e40b29f0 0x7f63e40b4f00 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7f63f4009cb0 tx=0x7f63f4009260 comp rx=0 tx=0).ready entity=mgr.15333 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:32:07.720 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:32:07.725+0000 7f63fced4700 1 -- 172.21.15.52:0/2606498786 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5623f36d7e40 msgr2=0x5623f385cb30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:32:07.720 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:32:07.725+0000 7f63fced4700 1 --2- 172.21.15.52:0/2606498786 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5623f36d7e40 0x5623f385cb30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:32:07.720 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:32:07.725+0000 7f63fced4700 1 -- 172.21.15.52:0/2606498786 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5623f3857120 msgr2=0x5623f3857530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:32:07.720 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:32:07.725+0000 7f63fced4700 1 --2- 172.21.15.52:0/2606498786 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5623f3857120 0x5623f3857530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:32:07.720 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:32:07.726+0000 7f63fced4700 1 -- 172.21.15.52:0/2606498786 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5623f371dfb0 con 0x5623f385d070 2024-05-27T07:32:07.720 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:32:07.726+0000 7f63fced4700 1 --2- 172.21.15.52:0/2606498786 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x5623f385d070 0x5623f3856be0 secure :-1 s=READY pgs=677 cs=0 l=1 rev1=1 crypto rx=0x7f63e80a6360 tx=0x7f63e8009310 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ae31b93f4c2dcaba server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:32:07.721 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:32:07.726+0000 7f63e37fe700 1 -- 172.21.15.52:0/2606498786 <== mon.0 v2:172.21.15.52:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f63e80ae690 con 0x5623f385d070 2024-05-27T07:32:07.721 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:32:07.726+0000 7f63e37fe700 1 -- 172.21.15.52:0/2606498786 <== mon.0 v2:172.21.15.52:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f63e80b30b0 con 0x5623f385d070 2024-05-27T07:32:07.721 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:32:07.726+0000 7f63e37fe700 1 -- 172.21.15.52:0/2606498786 <== mon.0 v2:172.21.15.52:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f63e80aa540 con 0x5623f385d070 2024-05-27T07:32:07.724 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:32:07.727+0000 7f640e34eb80 1 -- 172.21.15.52:0/2606498786 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x5623f385d070 msgr2=0x5623f3856be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:32:07.724 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:32:07.727+0000 7f640e34eb80 1 --2- 172.21.15.52:0/2606498786 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x5623f385d070 0x5623f3856be0 secure :-1 s=READY pgs=677 cs=0 l=1 rev1=1 crypto rx=0x7f63e80a6360 tx=0x7f63e8009310 comp rx=0 tx=0).stop 2024-05-27T07:32:07.724 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:32:07.727+0000 7f640e34eb80 1 -- 172.21.15.52:0/2606498786 shutdown_connections 2024-05-27T07:32:07.724 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:32:07.727+0000 7f640e34eb80 1 --2- 172.21.15.52:0/2606498786 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5623f3857120 0x5623f3857530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:32:07.724 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:32:07.727+0000 7f640e34eb80 1 --2- 172.21.15.52:0/2606498786 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5623f36d7e40 0x5623f385cb30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:32:07.724 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:32:07.727+0000 7f640e34eb80 1 --2- 172.21.15.52:0/2606498786 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x5623f385d070 0x5623f3856be0 unknown :-1 s=CLOSED pgs=677 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:32:07.724 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:32:07.727+0000 7f640e34eb80 1 -- 172.21.15.52:0/2606498786 >> 172.21.15.52:0/2606498786 conn(0x5623f36da770 msgr2=0x5623f3851d90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:32:07.724 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:32:07.727+0000 7f640e34eb80 1 -- 172.21.15.52:0/2606498786 shutdown_connections 2024-05-27T07:32:07.724 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:32:07.727+0000 7f640e34eb80 1 -- 172.21.15.52:0/2606498786 wait complete. 2024-05-27T07:32:07.724 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:32:07.727+0000 7f640e34eb80 1 -- start start 2024-05-27T07:32:07.725 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:32:07.727+0000 7f640e34eb80 1 Processor -- start 2024-05-27T07:32:07.725 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:32:07.728+0000 7f640e34eb80 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5623f36d7e40 0x5623f38608d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:32:07.725 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:32:07.728+0000 7f640e34eb80 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x5623f3857120 0x5623f3860e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:32:07.725 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:32:07.728+0000 7f640e34eb80 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5623f385d070 0x5623f3861350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:32:07.725 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:32:07.728+0000 7f640e34eb80 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x5623f367afa0 con 0x5623f3857120 2024-05-27T07:32:07.725 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:32:07.728+0000 7f640e34eb80 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x5623f36dcda0 con 0x5623f385d070 2024-05-27T07:32:07.725 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:32:07.728+0000 7f640e34eb80 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x5623f3863920 con 0x5623f36d7e40 2024-05-27T07:32:07.725 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:32:07.728+0000 7f640e34eb80 10 client.0 _ll_register_callbacks cb 0x5623f370b5b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T07:32:07.725 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:32:07.728+0000 7f63fced4700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x5623f3857120 0x5623f3860e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:32:07.725 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:32:07.728+0000 7f63fded6700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5623f385d070 0x5623f3861350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:32:07.725 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:32:07.728+0000 7f63fced4700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x5623f3857120 0x5623f3860e10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.52:47450/0 (socket says 172.21.15.52:47450) 2024-05-27T07:32:07.725 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:32:07.728+0000 7f63fced4700 1 -- 172.21.15.52:0/1368254761 learned_addr learned my addr 172.21.15.52:0/1368254761 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-05-27T07:32:07.725 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:32:07.728+0000 7f63fded6700 1 -- 172.21.15.52:0/1368254761 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5623f36d7e40 msgr2=0x5623f38608d0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-05-27T07:32:07.726 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:32:07.728+0000 7f63fded6700 1 --2- 172.21.15.52:0/1368254761 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5623f36d7e40 0x5623f38608d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:32:07.726 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:32:07.728+0000 7f63fded6700 1 -- 172.21.15.52:0/1368254761 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x5623f3857120 msgr2=0x5623f3860e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:32:07.726 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:32:07.728+0000 7f63fded6700 1 --2- 172.21.15.52:0/1368254761 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x5623f3857120 0x5623f3860e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:32:07.726 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:32:07.728+0000 7f63fded6700 1 -- 172.21.15.52:0/1368254761 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5623f36c48d0 con 0x5623f385d070 2024-05-27T07:32:07.726 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:32:07.728+0000 7f63fced4700 1 --2- 172.21.15.52:0/1368254761 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x5623f3857120 0x5623f3860e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-27T07:32:07.726 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:32:07.729+0000 7f63fded6700 1 --2- 172.21.15.52:0/1368254761 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5623f385d070 0x5623f3861350 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7f63f0009fb0 tx=0x7f63f0004040 comp rx=0 tx=0).ready entity=mon.2 client_cookie=9c5c0446c8cbfd74 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:32:07.726 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:32:07.729+0000 7f63e0ff9700 10 client.0 ms_handle_connect on v2:172.21.15.105:3300/0 2024-05-27T07:32:07.726 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:32:07.729+0000 7f63e0ff9700 1 -- 172.21.15.52:0/1368254761 <== mon.2 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f63f001a070 con 0x5623f385d070 2024-05-27T07:32:07.726 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:32:07.729+0000 7f63e0ff9700 1 -- 172.21.15.52:0/1368254761 <== mon.2 v2:172.21.15.105:3300/0 2 ==== osd_map(116..116 src has 1..116) ==== 9379+0+0 (secure 0 0 0) 0x7f63f0004db0 con 0x5623f385d070 2024-05-27T07:32:07.726 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:32:07.729+0000 7f640e34eb80 1 -- 172.21.15.52:0/1368254761 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f63e80b30b0 con 0x5623f385d070 2024-05-27T07:32:07.726 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:32:07.729+0000 7f63e0ff9700 1 -- 172.21.15.52:0/1368254761 <== mon.2 v2:172.21.15.105:3300/0 3 ==== mon_get_version_reply(handle=1 version=101) ==== 24+0+0 (secure 0 0 0) 0x7f63f0025050 con 0x5623f385d070 2024-05-27T07:32:07.726 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:32:07.729+0000 7f640e34eb80 10 client.26387 fetch_fsmap learned FSMap version 101 2024-05-27T07:32:07.727 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:32:07.729+0000 7f640e34eb80 1 -- 172.21.15.52:0/1368254761 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,fsmap=101,monmap=4+}) -- 0x5623f372e760 con 0x5623f385d070 2024-05-27T07:32:07.727 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:32:07.729+0000 7f63e0ff9700 1 -- 172.21.15.52:0/1368254761 <== mon.2 v2:172.21.15.105:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f63e80b30b0 con 0x5623f385d070 2024-05-27T07:32:07.727 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:32:07.729+0000 7f63e0ff9700 1 -- 172.21.15.52:0/1368254761 <== mon.2 v2:172.21.15.105:3300/0 5 ==== fsmap(e 101) ==== 4203+0+0 (secure 0 0 0) 0x7f63f00165f0 con 0x5623f385d070 2024-05-27T07:32:07.727 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:32:07.729+0000 7f640e34eb80 10 client.26387 fetch_fsmap finished waiting for FSMap version 101 2024-05-27T07:32:07.727 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:32:07.729+0000 7f640e34eb80 10 client.26387 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-05-27T07:32:07.727 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:32:07.729+0000 7f640e34eb80 20 client.26387 populate_metadata read hostname 'smithi052' 2024-05-27T07:32:07.727 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:32:07.729+0000 7f640e34eb80 1 --2- 172.21.15.52:0/1368254761 >> [v2:172.21.15.105:6838/500348151,v1:172.21.15.105:6839/500348151] conn(0x5623f3870c50 0x5623f3873100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:32:07.727 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:32:07.730+0000 7f640e34eb80 4 client.26387 mds_command: new command op to 24527 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-27T07:32:07.727 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:32:07.730+0000 7f640e34eb80 1 -- 172.21.15.52:0/1368254761 --> [v2:172.21.15.105:6838/500348151,v1:172.21.15.105:6839/500348151] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5623f3786cc0 con 0x5623f3870c50 2024-05-27T07:32:07.727 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:32:07.730+0000 7f63fd6d5700 1 --2- 172.21.15.52:0/1368254761 >> [v2:172.21.15.105:6838/500348151,v1:172.21.15.105:6839/500348151] conn(0x5623f3870c50 0x5623f3873100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:32:07.727 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:32:07.730+0000 7f63fd6d5700 1 --2- 172.21.15.52:0/1368254761 >> [v2:172.21.15.105:6838/500348151,v1:172.21.15.105:6839/500348151] conn(0x5623f3870c50 0x5623f3873100 crc :-1 s=READY pgs=125 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=2aa8acb9a525892d server_cookie=770d1a379e35bce7 in_seq=0 out_seq=0 2024-05-27T07:32:07.727 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:32:07.730+0000 7f63e0ff9700 10 client.26387 ms_handle_connect on v2:172.21.15.105:6838/500348151 2024-05-27T07:32:07.730 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:32:07.737+0000 7f63e0ff9700 1 -- 172.21.15.52:0/1368254761 <== mds.0 v2:172.21.15.105:6838/500348151 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24041 (crc 0 0 0) 0x5623f3786cc0 con 0x5623f3870c50 2024-05-27T07:32:07.731 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:32:07.737+0000 7f63e0ff9700 10 client.26387 handle_command_reply: tid=0 2024-05-27T07:32:07.731 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:32:07.737+0000 7f640e34eb80 1 client.26387 shutdown 2024-05-27T07:32:07.731 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:32:07.737+0000 7f640e34eb80 1 -- 172.21.15.52:0/1368254761 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5623f385d070 msgr2=0x5623f3861350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:32:07.731 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:32:07.737+0000 7f640e34eb80 1 --2- 172.21.15.52:0/1368254761 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5623f385d070 0x5623f3861350 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7f63f0009fb0 tx=0x7f63f0004040 comp rx=0 tx=0).stop 2024-05-27T07:32:07.731 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:32:07.737+0000 7f640e34eb80 1 -- 172.21.15.52:0/1368254761 shutdown_connections 2024-05-27T07:32:07.731 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:32:07.737+0000 7f640e34eb80 1 --2- 172.21.15.52:0/1368254761 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5623f385d070 0x5623f3861350 unknown :-1 s=CLOSED pgs=196 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:32:07.731 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:32:07.737+0000 7f640e34eb80 1 --2- 172.21.15.52:0/1368254761 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5623f36d7e40 0x5623f38608d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:32:07.731 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:32:07.737+0000 7f640e34eb80 1 --2- 172.21.15.52:0/1368254761 >> [v2:172.21.15.105:6838/500348151,v1:172.21.15.105:6839/500348151] conn(0x5623f3870c50 0x5623f3873100 crc :-1 s=READY pgs=125 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:32:07.731 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:32:07.737+0000 7f640e34eb80 1 --2- 172.21.15.52:0/1368254761 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x5623f3857120 0x5623f3860e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:32:07.731 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:32:07.737+0000 7f640e34eb80 1 -- 172.21.15.52:0/1368254761 >> 172.21.15.52:0/1368254761 conn(0x5623f36da770 msgr2=0x5623f3855bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:32:07.731 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:32:07.737+0000 7f63e0ff9700 0 client.26387 ms_handle_reset on v2:172.21.15.105:6838/500348151 2024-05-27T07:32:07.732 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:32:07.738+0000 7f640e34eb80 1 -- 172.21.15.52:0/1368254761 shutdown_connections 2024-05-27T07:32:07.732 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:32:07.738+0000 7f640e34eb80 1 -- 172.21.15.52:0/1368254761 wait complete. 2024-05-27T07:32:07.732 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:32:07.738+0000 7f640e34eb80 20 client.26387 trim_cache size 0 max 16384 2024-05-27T07:32:07.752 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:32:07.759+0000 7f640e34eb80 1 Processor -- start 2024-05-27T07:32:07.752 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:32:07.759+0000 7f640e34eb80 1 -- start start 2024-05-27T07:32:07.753 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:32:07.759+0000 7f640e34eb80 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5623f36d7e40 0x5623f385bd60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:32:07.753 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:32:07.759+0000 7f640e34eb80 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x5623f3857120 0x5623f3861190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:32:07.753 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:32:07.759+0000 7f640e34eb80 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5623f385d070 0x5623f386c980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:32:07.753 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:32:07.759+0000 7f640e34eb80 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x7f63f81b2ac0 con 0x5623f3857120 2024-05-27T07:32:07.753 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:32:07.759+0000 7f640e34eb80 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f63f81b0ec0 con 0x5623f385d070 2024-05-27T07:32:07.753 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:32:07.759+0000 7f640e34eb80 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f63f81b23f0 con 0x5623f36d7e40 2024-05-27T07:32:07.753 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:32:07.759+0000 7f63fd6d5700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5623f36d7e40 0x5623f385bd60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:32:07.753 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:32:07.760+0000 7f63fd6d5700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5623f36d7e40 0x5623f385bd60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.173:3300/0 says I am v2:172.21.15.52:60858/0 (socket says 172.21.15.52:60858) 2024-05-27T07:32:07.753 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:32:07.760+0000 7f63fd6d5700 1 -- 172.21.15.52:0/3511729618 learned_addr learned my addr 172.21.15.52:0/3511729618 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-05-27T07:32:07.753 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:32:07.760+0000 7f63fded6700 1 --2- 172.21.15.52:0/3511729618 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5623f385d070 0x5623f386c980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:32:07.753 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:32:07.760+0000 7f63fced4700 1 --2- 172.21.15.52:0/3511729618 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x5623f3857120 0x5623f3861190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:32:07.755 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:32:07.760+0000 7f63fd6d5700 1 -- 172.21.15.52:0/3511729618 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5623f385d070 msgr2=0x5623f386c980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:32:07.755 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:32:07.760+0000 7f63fd6d5700 1 --2- 172.21.15.52:0/3511729618 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5623f385d070 0x5623f386c980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:32:07.755 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:32:07.760+0000 7f63fd6d5700 1 -- 172.21.15.52:0/3511729618 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x5623f3857120 msgr2=0x5623f3861190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:32:07.755 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:32:07.760+0000 7f63fd6d5700 1 --2- 172.21.15.52:0/3511729618 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x5623f3857120 0x5623f3861190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:32:07.755 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:32:07.760+0000 7f63fd6d5700 1 -- 172.21.15.52:0/3511729618 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5623f38616d0 con 0x5623f36d7e40 2024-05-27T07:32:07.755 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:32:07.760+0000 7f63fced4700 1 --2- 172.21.15.52:0/3511729618 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x5623f3857120 0x5623f3861190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-27T07:32:07.755 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:32:07.760+0000 7f63fded6700 1 --2- 172.21.15.52:0/3511729618 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5623f385d070 0x5623f386c980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-27T07:32:07.755 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:32:07.760+0000 7f63fd6d5700 1 --2- 172.21.15.52:0/3511729618 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5623f36d7e40 0x5623f385bd60 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x5623f38738a0 tx=0x7f63f4014000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=f293f147533f2021 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:32:07.756 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:32:07.760+0000 7f63e2ffd700 1 -- 172.21.15.52:0/3511729618 <== mon.1 v2:172.21.15.173:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f63f4003900 con 0x5623f36d7e40 2024-05-27T07:32:07.756 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:32:07.761+0000 7f63e2ffd700 1 -- 172.21.15.52:0/3511729618 <== mon.1 v2:172.21.15.173:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f63f401c5d0 con 0x5623f36d7e40 2024-05-27T07:32:07.756 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:32:07.761+0000 7f63e2ffd700 1 -- 172.21.15.52:0/3511729618 <== mon.1 v2:172.21.15.173:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f63f4027670 con 0x5623f36d7e40 2024-05-27T07:32:07.756 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:32:07.761+0000 7f640e34eb80 1 -- 172.21.15.52:0/3511729618 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5623f36d7e40 msgr2=0x5623f385bd60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:32:07.756 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:32:07.761+0000 7f640e34eb80 1 --2- 172.21.15.52:0/3511729618 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5623f36d7e40 0x5623f385bd60 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x5623f38738a0 tx=0x7f63f4014000 comp rx=0 tx=0).stop 2024-05-27T07:32:07.756 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:32:07.761+0000 7f640e34eb80 1 -- 172.21.15.52:0/3511729618 shutdown_connections 2024-05-27T07:32:07.756 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:32:07.761+0000 7f640e34eb80 1 --2- 172.21.15.52:0/3511729618 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5623f385d070 0x5623f386c980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:32:07.756 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:32:07.761+0000 7f640e34eb80 1 --2- 172.21.15.52:0/3511729618 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5623f36d7e40 0x5623f385bd60 unknown :-1 s=CLOSED pgs=154 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:32:07.756 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:32:07.761+0000 7f640e34eb80 1 --2- 172.21.15.52:0/3511729618 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x5623f3857120 0x5623f3861190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:32:07.756 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:32:07.762+0000 7f640e34eb80 1 -- 172.21.15.52:0/3511729618 >> 172.21.15.52:0/3511729618 conn(0x5623f36da770 msgr2=0x5623f385f1e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:32:07.756 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:32:07.762+0000 7f640e34eb80 1 -- 172.21.15.52:0/3511729618 shutdown_connections 2024-05-27T07:32:07.756 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:32:07.762+0000 7f640e34eb80 1 -- 172.21.15.52:0/3511729618 wait complete. 2024-05-27T07:32:07.757 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:32:07.762+0000 7f640e34eb80 1 -- start start 2024-05-27T07:32:07.757 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:32:07.762+0000 7f640e34eb80 1 Processor -- start 2024-05-27T07:32:07.759 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:32:07.763+0000 7f640e34eb80 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5623f36d7e40 0x5623f38691a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:32:07.759 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:32:07.763+0000 7f640e34eb80 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5623f3857120 0x5623f3878090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:32:07.759 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:32:07.763+0000 7f640e34eb80 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x5623f385d070 0x5623f38785d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:32:07.759 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:32:07.763+0000 7f640e34eb80 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x7f63f81b1c60 con 0x5623f385d070 2024-05-27T07:32:07.759 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:32:07.763+0000 7f640e34eb80 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f63f81b1590 con 0x5623f3857120 2024-05-27T07:32:07.759 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:32:07.763+0000 7f640e34eb80 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x5623f386e270 con 0x5623f36d7e40 2024-05-27T07:32:07.759 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:32:07.763+0000 7f640e34eb80 10 client.0 _ll_register_callbacks cb 0x5623f370b5b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T07:32:07.760 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:32:07.763+0000 7f63fced4700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5623f3857120 0x5623f3878090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:32:07.760 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:32:07.763+0000 7f63fced4700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5623f3857120 0x5623f3878090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.52:57312/0 (socket says 172.21.15.52:57312) 2024-05-27T07:32:07.760 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:32:07.763+0000 7f63fced4700 1 -- 172.21.15.52:0/914588339 learned_addr learned my addr 172.21.15.52:0/914588339 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-05-27T07:32:07.760 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:32:07.763+0000 7f63fded6700 1 --2- 172.21.15.52:0/914588339 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x5623f385d070 0x5623f38785d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:32:07.760 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:32:07.763+0000 7f63fd6d5700 1 --2- 172.21.15.52:0/914588339 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5623f36d7e40 0x5623f38691a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:32:07.760 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:32:07.763+0000 7f63fced4700 1 -- 172.21.15.52:0/914588339 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5623f36d7e40 msgr2=0x5623f38691a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:32:07.760 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:32:07.763+0000 7f63fced4700 1 --2- 172.21.15.52:0/914588339 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5623f36d7e40 0x5623f38691a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:32:07.760 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:32:07.763+0000 7f63fced4700 1 -- 172.21.15.52:0/914588339 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x5623f385d070 msgr2=0x5623f38785d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:32:07.760 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:32:07.763+0000 7f63fced4700 1 --2- 172.21.15.52:0/914588339 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x5623f385d070 0x5623f38785d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:32:07.760 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:32:07.763+0000 7f63fced4700 1 -- 172.21.15.52:0/914588339 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5623f3878c70 con 0x5623f3857120 2024-05-27T07:32:07.760 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:32:07.764+0000 7f63fced4700 1 --2- 172.21.15.52:0/914588339 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5623f3857120 0x5623f3878090 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7f63e80a9fe0 tx=0x7f63e80aaa60 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f66dbc54631b7cd6 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:32:07.760 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:32:07.764+0000 7f63e0ff9700 10 client.0 ms_handle_connect on v2:172.21.15.105:3300/0 2024-05-27T07:32:07.760 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:32:07.764+0000 7f63e0ff9700 1 -- 172.21.15.52:0/914588339 <== mon.2 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f63e806f060 con 0x5623f3857120 2024-05-27T07:32:07.761 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:32:07.764+0000 7f63e0ff9700 1 -- 172.21.15.52:0/914588339 <== mon.2 v2:172.21.15.105:3300/0 2 ==== osd_map(116..116 src has 1..116) ==== 9379+0+0 (secure 0 0 0) 0x7f63e80aa750 con 0x5623f3857120 2024-05-27T07:32:07.761 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:32:07.764+0000 7f640e34eb80 1 -- 172.21.15.52:0/914588339 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f63f401c5d0 con 0x5623f3857120 2024-05-27T07:32:07.761 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:32:07.764+0000 7f63e0ff9700 1 -- 172.21.15.52:0/914588339 <== mon.2 v2:172.21.15.105:3300/0 3 ==== mon_get_version_reply(handle=1 version=101) ==== 24+0+0 (secure 0 0 0) 0x7f63e80b30b0 con 0x5623f3857120 2024-05-27T07:32:07.761 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:32:07.764+0000 7f640e34eb80 10 client.26393 fetch_fsmap learned FSMap version 101 2024-05-27T07:32:07.761 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:32:07.764+0000 7f640e34eb80 1 -- 172.21.15.52:0/914588339 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,fsmap=101,monmap=4+}) -- 0x5623f38db610 con 0x5623f3857120 2024-05-27T07:32:07.761 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:32:07.764+0000 7f63e0ff9700 1 -- 172.21.15.52:0/914588339 <== mon.2 v2:172.21.15.105:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f63f401c5d0 con 0x5623f3857120 2024-05-27T07:32:07.761 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:32:07.764+0000 7f63e0ff9700 1 -- 172.21.15.52:0/914588339 <== mon.2 v2:172.21.15.105:3300/0 5 ==== fsmap(e 101) ==== 4203+0+0 (secure 0 0 0) 0x7f63e80ae270 con 0x5623f3857120 2024-05-27T07:32:07.761 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:32:07.764+0000 7f640e34eb80 10 client.26393 fetch_fsmap finished waiting for FSMap version 101 2024-05-27T07:32:07.761 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:32:07.764+0000 7f640e34eb80 10 client.26393 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-05-27T07:32:07.761 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:32:07.765+0000 7f640e34eb80 20 client.26393 populate_metadata read hostname 'smithi052' 2024-05-27T07:32:07.761 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:32:07.765+0000 7f640e34eb80 1 --2- 172.21.15.52:0/914588339 >> [v2:172.21.15.105:6838/500348151,v1:172.21.15.105:6839/500348151] conn(0x5623f38dbbf0 0x5623f38de010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:32:07.762 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:32:07.765+0000 7f640e34eb80 4 client.26393 mds_command: new command op to 24527 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-05-27T07:32:07.762 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:32:07.765+0000 7f640e34eb80 1 -- 172.21.15.52:0/914588339 --> [v2:172.21.15.105:6838/500348151,v1:172.21.15.105:6839/500348151] -- command(tid 0: {"prefix": "scrub status"}) -- 0x5623f37804a0 con 0x5623f38dbbf0 2024-05-27T07:32:07.762 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:32:07.765+0000 7f63fd6d5700 1 --2- 172.21.15.52:0/914588339 >> [v2:172.21.15.105:6838/500348151,v1:172.21.15.105:6839/500348151] conn(0x5623f38dbbf0 0x5623f38de010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:32:07.762 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:32:07.765+0000 7f63e0ff9700 10 client.26393 ms_handle_connect on v2:172.21.15.105:6838/500348151 2024-05-27T07:32:07.762 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:32:07.765+0000 7f63fd6d5700 1 --2- 172.21.15.52:0/914588339 >> [v2:172.21.15.105:6838/500348151,v1:172.21.15.105:6839/500348151] conn(0x5623f38dbbf0 0x5623f38de010 crc :-1 s=READY pgs=126 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=1add5254851b6722 server_cookie=d124dc4e51674136 in_seq=0 out_seq=0 2024-05-27T07:32:07.765 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:32:07.772+0000 7f63e0ff9700 1 -- 172.21.15.52:0/914588339 <== mds.0 v2:172.21.15.105:6838/500348151 1 ==== command_reply(tid 0: 0 ) ==== 8+0+269 (crc 0 0 0) 0x5623f37804a0 con 0x5623f38dbbf0 2024-05-27T07:32:07.765 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:32:07.772+0000 7f63e0ff9700 10 client.26393 handle_command_reply: tid=0 2024-05-27T07:32:07.765 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:32:07.772+0000 7f640e34eb80 1 client.26393 shutdown 2024-05-27T07:32:07.765 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:32:07.772+0000 7f640e34eb80 1 -- 172.21.15.52:0/914588339 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5623f3857120 msgr2=0x5623f3878090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:32:07.765 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:32:07.772+0000 7f640e34eb80 1 --2- 172.21.15.52:0/914588339 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5623f3857120 0x5623f3878090 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7f63e80a9fe0 tx=0x7f63e80aaa60 comp rx=0 tx=0).stop 2024-05-27T07:32:07.766 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:32:07.772+0000 7f640e34eb80 1 -- 172.21.15.52:0/914588339 shutdown_connections 2024-05-27T07:32:07.766 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:32:07.772+0000 7f640e34eb80 1 --2- 172.21.15.52:0/914588339 >> [v2:172.21.15.105:6838/500348151,v1:172.21.15.105:6839/500348151] conn(0x5623f38dbbf0 0x5623f38de010 crc :-1 s=READY pgs=126 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:32:07.766 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:32:07.772+0000 7f640e34eb80 1 --2- 172.21.15.52:0/914588339 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x5623f385d070 0x5623f38785d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:32:07.766 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:32:07.772+0000 7f640e34eb80 1 --2- 172.21.15.52:0/914588339 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5623f36d7e40 0x5623f38691a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:32:07.766 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:32:07.772+0000 7f640e34eb80 1 --2- 172.21.15.52:0/914588339 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5623f3857120 0x5623f3878090 unknown :-1 s=CLOSED pgs=197 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:32:07.766 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:32:07.772+0000 7f63e0ff9700 0 client.26393 ms_handle_reset on v2:172.21.15.105:6838/500348151 2024-05-27T07:32:07.766 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:32:07.772+0000 7f640e34eb80 1 -- 172.21.15.52:0/914588339 >> 172.21.15.52:0/914588339 conn(0x5623f36da770 msgr2=0x5623f384b4f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:32:07.766 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:32:07.772+0000 7f640e34eb80 1 -- 172.21.15.52:0/914588339 shutdown_connections 2024-05-27T07:32:07.766 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:32:07.772+0000 7f640e34eb80 1 -- 172.21.15.52:0/914588339 wait complete. 2024-05-27T07:32:07.766 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:32:07.772+0000 7f640e34eb80 20 client.26393 trim_cache size 0 max 16384 2024-05-27T07:32:07.767 INFO:teuthology.orchestra.run.smithi052.stdout:{ 2024-05-27T07:32:07.767 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "scrub active (20200 inodes in the stack)", 2024-05-27T07:32:07.767 INFO:teuthology.orchestra.run.smithi052.stdout: "scrubs": { 2024-05-27T07:32:07.767 INFO:teuthology.orchestra.run.smithi052.stdout: "36f21956-1be9-436b-b6db-8bc2e943eb70": { 2024-05-27T07:32:07.767 INFO:teuthology.orchestra.run.smithi052.stdout: "path": "/", 2024-05-27T07:32:07.767 INFO:teuthology.orchestra.run.smithi052.stdout: "tag": "36f21956-1be9-436b-b6db-8bc2e943eb70", 2024-05-27T07:32:07.767 INFO:teuthology.orchestra.run.smithi052.stdout: "options": "recursive,force" 2024-05-27T07:32:07.767 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-05-27T07:32:07.767 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-05-27T07:32:07.767 INFO:teuthology.orchestra.run.smithi052.stdout:} 2024-05-27T07:32:07.768 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:32:07.773+0000 7f63ff939700 1 -- 172.21.15.52:0/3057778159 >> [v2:172.21.15.52:6800/1822318384,v1:172.21.15.52:6801/1822318384] conn(0x7f63e40b29f0 msgr2=0x7f63e40b4f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:32:07.768 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:32:07.773+0000 7f63ff939700 1 --2- 172.21.15.52:0/3057778159 >> [v2:172.21.15.52:6800/1822318384,v1:172.21.15.52:6801/1822318384] conn(0x7f63e40b29f0 0x7f63e40b4f00 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7f63f4009cb0 tx=0x7f63f4009260 comp rx=0 tx=0).stop 2024-05-27T07:32:07.768 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:32:07.773+0000 7f63ff939700 1 -- 172.21.15.52:0/3057778159 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f63f8107de0 msgr2=0x7f63f80ff7d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:32:07.768 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:32:07.773+0000 7f63ff939700 1 --2- 172.21.15.52:0/3057778159 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f63f8107de0 0x7f63f80ff7d0 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7f63e800b580 tx=0x7f63e80045e0 comp rx=0 tx=0).stop 2024-05-27T07:32:07.768 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:32:07.773+0000 7f63ff939700 1 -- 172.21.15.52:0/3057778159 shutdown_connections 2024-05-27T07:32:07.768 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:32:07.773+0000 7f63ff939700 1 --2- 172.21.15.52:0/3057778159 >> [v2:172.21.15.52:6800/1822318384,v1:172.21.15.52:6801/1822318384] conn(0x7f63e40b29f0 0x7f63e40b4f00 unknown :-1 s=CLOSED pgs=155 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:32:07.769 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:32:07.773+0000 7f63ff939700 1 --2- 172.21.15.52:0/3057778159 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f63f8108730 0x7f63f80ffd10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:32:07.769 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:32:07.773+0000 7f63ff939700 1 --2- 172.21.15.52:0/3057778159 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f63f81034b0 0x7f63f80ff290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:32:07.769 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:32:07.773+0000 7f63ff939700 1 --2- 172.21.15.52:0/3057778159 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f63f8107de0 0x7f63f80ff7d0 unknown :-1 s=CLOSED pgs=195 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:32:07.769 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:32:07.773+0000 7f63ff939700 1 -- 172.21.15.52:0/3057778159 >> 172.21.15.52:0/3057778159 conn(0x7f63f80fedb0 msgr2=0x7f63f806b940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:32:07.769 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:32:07.773+0000 7f63ff939700 1 -- 172.21.15.52:0/3057778159 shutdown_connections 2024-05-27T07:32:07.769 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:32:07.773+0000 7f63ff939700 1 -- 172.21.15.52:0/3057778159 wait complete. 2024-05-27T07:32:08.442 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:32:08 smithi105 ceph-mon[35977]: pgmap v213: 129 pgs: 129 active+clean; 540 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 36 KiB/s rd, 1.5 MiB/s wr, 1.27k op/s 2024-05-27T07:32:08.482 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:32:08 smithi052 ceph-mon[31955]: pgmap v213: 129 pgs: 129 active+clean; 540 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 36 KiB/s rd, 1.5 MiB/s wr, 1.27k op/s 2024-05-27T07:32:08.586 INFO:tasks.cephfs.filesystem:scrub status for tag:36f21956-1be9-436b-b6db-8bc2e943eb70 - {'path': '/', 'tag': '36f21956-1be9-436b-b6db-8bc2e943eb70', 'options': 'recursive,force'} 2024-05-27T07:32:08.604 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:32:08 smithi173 ceph-mon[37000]: pgmap v213: 129 pgs: 129 active+clean; 540 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 36 KiB/s rd, 1.5 MiB/s wr, 1.27k op/s 2024-05-27T07:32:10.443 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:32:10 smithi105 ceph-mon[35977]: pgmap v214: 129 pgs: 129 active+clean; 546 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 102 KiB/s rd, 1.4 MiB/s wr, 1.33k op/s 2024-05-27T07:32:10.482 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:32:10 smithi052 ceph-mon[31955]: pgmap v214: 129 pgs: 129 active+clean; 546 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 102 KiB/s rd, 1.4 MiB/s wr, 1.33k op/s 2024-05-27T07:32:10.605 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:32:10 smithi173 ceph-mon[37000]: pgmap v214: 129 pgs: 129 active+clean; 546 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 102 KiB/s rd, 1.4 MiB/s wr, 1.33k op/s 2024-05-27T07:32:12.443 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:32:12 smithi105 ceph-mon[35977]: pgmap v215: 129 pgs: 129 active+clean; 569 MiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 189 KiB/s rd, 3.2 MiB/s wr, 1.10k op/s 2024-05-27T07:32:12.482 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:32:12 smithi052 ceph-mon[31955]: pgmap v215: 129 pgs: 129 active+clean; 569 MiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 189 KiB/s rd, 3.2 MiB/s wr, 1.10k op/s 2024-05-27T07:32:12.605 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:32:12 smithi173 ceph-mon[37000]: pgmap v215: 129 pgs: 129 active+clean; 569 MiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 189 KiB/s rd, 3.2 MiB/s wr, 1.10k op/s 2024-05-27T07:32:14.443 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:32:14 smithi105 ceph-mon[35977]: pgmap v216: 129 pgs: 129 active+clean; 576 MiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 201 KiB/s rd, 3.5 MiB/s wr, 622 op/s 2024-05-27T07:32:14.482 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:32:14 smithi052 ceph-mon[31955]: pgmap v216: 129 pgs: 129 active+clean; 576 MiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 201 KiB/s rd, 3.5 MiB/s wr, 622 op/s 2024-05-27T07:32:14.605 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:32:14 smithi173 ceph-mon[37000]: pgmap v216: 129 pgs: 129 active+clean; 576 MiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 201 KiB/s rd, 3.5 MiB/s wr, 622 op/s 2024-05-27T07:32:15.039 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:32:15.045+0000 7f3b690fc640 1 -- 172.21.15.52:0/2801411311 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f3b6410f320 msgr2=0x7f3b6407af80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:32:15.039 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:32:15.045+0000 7f3b690fc640 1 --2- 172.21.15.52:0/2801411311 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f3b6410f320 0x7f3b6407af80 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7f3b5c009ca0 tx=0x7f3b5c02a8f0 comp rx=0 tx=0).stop 2024-05-27T07:32:15.039 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:32:15.045+0000 7f3b690fc640 1 -- 172.21.15.52:0/2801411311 shutdown_connections 2024-05-27T07:32:15.039 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:32:15.045+0000 7f3b690fc640 1 --2- 172.21.15.52:0/2801411311 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f3b6410f320 0x7f3b6407af80 unknown :-1 s=CLOSED pgs=198 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:32:15.039 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:32:15.045+0000 7f3b690fc640 1 --2- 172.21.15.52:0/2801411311 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f3b6410e860 0x7f3b6410ec90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:32:15.039 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:32:15.045+0000 7f3b690fc640 1 --2- 172.21.15.52:0/2801411311 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f3b6410a380 0x7f3b6410a790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:32:15.039 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:32:15.046+0000 7f3b690fc640 1 -- 172.21.15.52:0/2801411311 >> 172.21.15.52:0/2801411311 conn(0x7f3b6406f800 msgr2=0x7f3b6406fc10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:32:15.039 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:32:15.046+0000 7f3b690fc640 1 -- 172.21.15.52:0/2801411311 shutdown_connections 2024-05-27T07:32:15.042 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:32:15.046+0000 7f3b690fc640 1 -- 172.21.15.52:0/2801411311 wait complete. 2024-05-27T07:32:15.042 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:32:15.047+0000 7f3b690fc640 1 Processor -- start 2024-05-27T07:32:15.042 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:32:15.047+0000 7f3b690fc640 1 -- start start 2024-05-27T07:32:15.042 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:32:15.047+0000 7f3b690fc640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f3b6410a380 0x7f3b6407feb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:32:15.042 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:32:15.047+0000 7f3b690fc640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f3b6410e860 0x7f3b640803f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:32:15.042 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:32:15.047+0000 7f3b690fc640 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f3b64080a10 0x7f3b641bd780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:32:15.042 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:32:15.047+0000 7f3b690fc640 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x7f3b6407e7a0 con 0x7f3b64080a10 2024-05-27T07:32:15.043 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:32:15.047+0000 7f3b690fc640 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f3b6407e620 con 0x7f3b6410e860 2024-05-27T07:32:15.043 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:32:15.047+0000 7f3b690fc640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f3b6407e920 con 0x7f3b6410a380 2024-05-27T07:32:15.043 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:32:15.048+0000 7f3b637fe640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f3b6410e860 0x7f3b640803f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:32:15.043 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:32:15.048+0000 7f3b637fe640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f3b6410e860 0x7f3b640803f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.52:37530/0 (socket says 172.21.15.52:37530) 2024-05-27T07:32:15.043 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:32:15.048+0000 7f3b637fe640 1 -- 172.21.15.52:0/3731799994 learned_addr learned my addr 172.21.15.52:0/3731799994 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-05-27T07:32:15.043 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:32:15.048+0000 7f3b688fb640 1 --2- 172.21.15.52:0/3731799994 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f3b64080a10 0x7f3b641bd780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:32:15.043 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:32:15.048+0000 7f3b63fff640 1 --2- 172.21.15.52:0/3731799994 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f3b6410a380 0x7f3b6407feb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:32:15.043 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:32:15.048+0000 7f3b637fe640 1 -- 172.21.15.52:0/3731799994 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f3b6410a380 msgr2=0x7f3b6407feb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:32:15.043 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:32:15.048+0000 7f3b637fe640 1 --2- 172.21.15.52:0/3731799994 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f3b6410a380 0x7f3b6407feb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:32:15.043 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:32:15.048+0000 7f3b637fe640 1 -- 172.21.15.52:0/3731799994 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f3b64080a10 msgr2=0x7f3b641bd780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:32:15.043 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:32:15.048+0000 7f3b637fe640 1 --2- 172.21.15.52:0/3731799994 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f3b64080a10 0x7f3b641bd780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:32:15.043 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:32:15.049+0000 7f3b637fe640 1 -- 172.21.15.52:0/3731799994 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3b641bdf80 con 0x7f3b6410e860 2024-05-27T07:32:15.051 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:32:15.049+0000 7f3b637fe640 1 --2- 172.21.15.52:0/3731799994 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f3b6410e860 0x7f3b640803f0 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7f3b5800b450 tx=0x7f3b5800b910 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:32:15.053 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:32:15.060+0000 7f3b617fa640 1 -- 172.21.15.52:0/3731799994 <== mon.2 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3b58015020 con 0x7f3b6410e860 2024-05-27T07:32:15.053 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:32:15.060+0000 7f3b617fa640 1 -- 172.21.15.52:0/3731799994 <== mon.2 v2:172.21.15.105:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f3b58011d40 con 0x7f3b6410e860 2024-05-27T07:32:15.053 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:32:15.060+0000 7f3b690fc640 1 -- 172.21.15.52:0/3731799994 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3b641be270 con 0x7f3b6410e860 2024-05-27T07:32:15.053 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:32:15.060+0000 7f3b617fa640 1 -- 172.21.15.52:0/3731799994 <== mon.2 v2:172.21.15.105:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3b58010e20 con 0x7f3b6410e860 2024-05-27T07:32:15.053 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:32:15.060+0000 7f3b690fc640 1 -- 172.21.15.52:0/3731799994 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3b641be7c0 con 0x7f3b6410e860 2024-05-27T07:32:15.061 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:32:15.068+0000 7f3b617fa640 1 -- 172.21.15.52:0/3731799994 <== mon.2 v2:172.21.15.105:3300/0 4 ==== mgrmap(e 26) ==== 149783+0+0 (secure 0 0 0) 0x7f3b58018020 con 0x7f3b6410e860 2024-05-27T07:32:15.069 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:32:15.075+0000 7f3b7120f740 1 Processor -- start 2024-05-27T07:32:15.069 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:32:15.076+0000 7f3b617fa640 1 --2- 172.21.15.52:0/3731799994 >> [v2:172.21.15.52:6800/1822318384,v1:172.21.15.52:6801/1822318384] conn(0x7f3b4c0b2ae0 0x7f3b4c0b4ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:32:15.069 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:32:15.076+0000 7f3b688fb640 1 --2- 172.21.15.52:0/3731799994 >> [v2:172.21.15.52:6800/1822318384,v1:172.21.15.52:6801/1822318384] conn(0x7f3b4c0b2ae0 0x7f3b4c0b4ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:32:15.069 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:32:15.076+0000 7f3b617fa640 1 -- 172.21.15.52:0/3731799994 <== mon.2 v2:172.21.15.105:3300/0 5 ==== osd_map(116..116 src has 1..116) ==== 9379+0+0 (secure 0 0 0) 0x7f3b580e2150 con 0x7f3b6410e860 2024-05-27T07:32:15.069 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:32:15.076+0000 7f3b7120f740 1 -- start start 2024-05-27T07:32:15.070 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:32:15.077+0000 7f3b688fb640 1 --2- 172.21.15.52:0/3731799994 >> [v2:172.21.15.52:6800/1822318384,v1:172.21.15.52:6801/1822318384] conn(0x7f3b4c0b2ae0 0x7f3b4c0b4ff0 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7f3b5c004af0 tx=0x7f3b5c004a80 comp rx=0 tx=0).ready entity=mgr.15333 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:32:15.072 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:32:15.077+0000 7f3b7120f740 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x5629777e51b0 0x562977b86590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:32:15.072 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:32:15.078+0000 7f3b7120f740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5629777bc2f0 0x562977b86ad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:32:15.072 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:32:15.078+0000 7f3b7120f740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5629777c44d0 0x562977b87010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:32:15.072 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:32:15.078+0000 7f3b63fff640 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x5629777e51b0 0x562977b86590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:32:15.072 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:32:15.078+0000 7f3b63fff640 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x5629777e51b0 0x562977b86590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.52:60124/0 (socket says 172.21.15.52:60124) 2024-05-27T07:32:15.072 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:32:15.078+0000 7f3b637fe640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5629777bc2f0 0x562977b86ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:32:15.072 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:32:15.078+0000 7f3b637fe640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5629777bc2f0 0x562977b86ad0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.52:37538/0 (socket says 172.21.15.52:37538) 2024-05-27T07:32:15.072 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:32:15.078+0000 7f3b63fff640 1 -- 172.21.15.52:0/1572989695 learned_addr learned my addr 172.21.15.52:0/1572989695 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-05-27T07:32:15.072 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:32:15.078+0000 7f3b7120f740 1 -- 172.21.15.52:0/1572989695 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x562977af8c30 con 0x5629777e51b0 2024-05-27T07:32:15.072 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:32:15.078+0000 7f3b7120f740 1 -- 172.21.15.52:0/1572989695 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x562977ae87d0 con 0x5629777bc2f0 2024-05-27T07:32:15.072 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:32:15.078+0000 7f3b7120f740 1 -- 172.21.15.52:0/1572989695 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x562977aec810 con 0x5629777c44d0 2024-05-27T07:32:15.072 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:32:15.078+0000 7f3b688fb640 1 --2- 172.21.15.52:0/1572989695 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5629777c44d0 0x562977b87010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:32:15.073 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:32:15.079+0000 7f3b63fff640 1 -- 172.21.15.52:0/1572989695 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5629777c44d0 msgr2=0x562977b87010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:32:15.073 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:32:15.079+0000 7f3b63fff640 1 --2- 172.21.15.52:0/1572989695 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5629777c44d0 0x562977b87010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:32:15.073 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:32:15.079+0000 7f3b63fff640 1 -- 172.21.15.52:0/1572989695 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5629777bc2f0 msgr2=0x562977b86ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:32:15.073 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:32:15.079+0000 7f3b63fff640 1 --2- 172.21.15.52:0/1572989695 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5629777bc2f0 0x562977b86ad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:32:15.073 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:32:15.079+0000 7f3b63fff640 1 -- 172.21.15.52:0/1572989695 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5629777fd210 con 0x5629777e51b0 2024-05-27T07:32:15.073 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:32:15.079+0000 7f3b63fff640 1 --2- 172.21.15.52:0/1572989695 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x5629777e51b0 0x562977b86590 secure :-1 s=READY pgs=678 cs=0 l=1 rev1=1 crypto rx=0x7f3b5400bcd0 tx=0x7f3b540107a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fe1afb90c28bc1b3 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:32:15.077 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:32:15.084+0000 7f3b4a7fc640 1 -- 172.21.15.52:0/1572989695 <== mon.0 v2:172.21.15.52:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3b54005220 con 0x5629777e51b0 2024-05-27T07:32:15.078 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:32:15.085+0000 7f3b4a7fc640 1 -- 172.21.15.52:0/1572989695 <== mon.0 v2:172.21.15.52:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f3b54004c50 con 0x5629777e51b0 2024-05-27T07:32:15.079 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:32:15.086+0000 7f3b7120f740 1 -- 172.21.15.52:0/1572989695 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x5629777e51b0 msgr2=0x562977b86590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:32:15.079 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:32:15.086+0000 7f3b7120f740 1 --2- 172.21.15.52:0/1572989695 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x5629777e51b0 0x562977b86590 secure :-1 s=READY pgs=678 cs=0 l=1 rev1=1 crypto rx=0x7f3b5400bcd0 tx=0x7f3b540107a0 comp rx=0 tx=0).stop 2024-05-27T07:32:15.081 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:32:15.088+0000 7f3b7120f740 1 -- 172.21.15.52:0/1572989695 shutdown_connections 2024-05-27T07:32:15.081 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:32:15.088+0000 7f3b7120f740 1 --2- 172.21.15.52:0/1572989695 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5629777c44d0 0x562977b87010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:32:15.081 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:32:15.088+0000 7f3b7120f740 1 --2- 172.21.15.52:0/1572989695 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5629777bc2f0 0x562977b86ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:32:15.081 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:32:15.088+0000 7f3b7120f740 1 --2- 172.21.15.52:0/1572989695 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x5629777e51b0 0x562977b86590 unknown :-1 s=CLOSED pgs=678 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:32:15.081 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:32:15.088+0000 7f3b7120f740 1 -- 172.21.15.52:0/1572989695 >> 172.21.15.52:0/1572989695 conn(0x5629777d9300 msgr2=0x562977b82d40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:32:15.081 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:32:15.088+0000 7f3b7120f740 1 -- 172.21.15.52:0/1572989695 shutdown_connections 2024-05-27T07:32:15.081 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:32:15.088+0000 7f3b7120f740 1 -- 172.21.15.52:0/1572989695 wait complete. 2024-05-27T07:32:15.082 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:32:15.089+0000 7f3b7120f740 1 -- start start 2024-05-27T07:32:15.082 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:32:15.089+0000 7f3b7120f740 1 Processor -- start 2024-05-27T07:32:15.083 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:32:15.089+0000 7f3b7120f740 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x5629777bc2f0 0x562977b90bd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:32:15.083 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:32:15.090+0000 7f3b7120f740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5629777c44d0 0x562977b91110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:32:15.083 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:32:15.090+0000 7f3b63fff640 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x5629777bc2f0 0x562977b90bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:32:15.083 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:32:15.090+0000 7f3b7120f740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5629777e51b0 0x562977b96a60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:32:15.083 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:32:15.090+0000 7f3b63fff640 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x5629777bc2f0 0x562977b90bd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.52:60136/0 (socket says 172.21.15.52:60136) 2024-05-27T07:32:15.083 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:32:15.090+0000 7f3b637fe640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5629777c44d0 0x562977b91110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:32:15.083 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:32:15.090+0000 7f3b637fe640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5629777c44d0 0x562977b91110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.52:37542/0 (socket says 172.21.15.52:37542) 2024-05-27T07:32:15.083 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:32:15.090+0000 7f3b63fff640 1 -- 172.21.15.52:0/71137716 learned_addr learned my addr 172.21.15.52:0/71137716 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-05-27T07:32:15.084 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:32:15.090+0000 7f3b7120f740 1 -- 172.21.15.52:0/71137716 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x5629777ce790 con 0x5629777bc2f0 2024-05-27T07:32:15.084 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:32:15.090+0000 7f3b7120f740 1 -- 172.21.15.52:0/71137716 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x562977aa4890 con 0x5629777c44d0 2024-05-27T07:32:15.084 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:32:15.090+0000 7f3b7120f740 1 -- 172.21.15.52:0/71137716 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x562977b93710 con 0x5629777e51b0 2024-05-27T07:32:15.085 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:32:15.090+0000 7f3b688fb640 1 --2- 172.21.15.52:0/71137716 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5629777e51b0 0x562977b96a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:32:15.085 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:32:15.090+0000 7f3b7120f740 10 client.0 _ll_register_callbacks cb 0x5629777fca00 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T07:32:15.085 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:32:15.091+0000 7f3b637fe640 1 -- 172.21.15.52:0/71137716 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5629777e51b0 msgr2=0x562977b96a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:32:15.086 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:32:15.091+0000 7f3b637fe640 1 --2- 172.21.15.52:0/71137716 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5629777e51b0 0x562977b96a60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:32:15.086 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:32:15.091+0000 7f3b637fe640 1 -- 172.21.15.52:0/71137716 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x5629777bc2f0 msgr2=0x562977b90bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:32:15.086 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:32:15.091+0000 7f3b637fe640 1 --2- 172.21.15.52:0/71137716 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x5629777bc2f0 0x562977b90bd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:32:15.086 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:32:15.091+0000 7f3b637fe640 1 -- 172.21.15.52:0/71137716 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5629778c9620 con 0x5629777c44d0 2024-05-27T07:32:15.086 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:32:15.091+0000 7f3b63fff640 1 --2- 172.21.15.52:0/71137716 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x5629777bc2f0 0x562977b90bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-27T07:32:15.086 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:32:15.091+0000 7f3b637fe640 1 --2- 172.21.15.52:0/71137716 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5629777c44d0 0x562977b91110 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7f3b580df650 tx=0x7f3b580a8a20 comp rx=0 tx=0).ready entity=mon.2 client_cookie=39a7128128c45c5f server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:32:15.086 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:32:15.092+0000 7f3b437fe640 10 client.0 ms_handle_connect on v2:172.21.15.105:3300/0 2024-05-27T07:32:15.086 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:32:15.092+0000 7f3b437fe640 1 -- 172.21.15.52:0/71137716 <== mon.2 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3b580c2070 con 0x5629777c44d0 2024-05-27T07:32:15.086 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:32:15.092+0000 7f3b7120f740 1 -- 172.21.15.52:0/71137716 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f3b54004c50 con 0x5629777c44d0 2024-05-27T07:32:15.087 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:32:15.092+0000 7f3b437fe640 1 -- 172.21.15.52:0/71137716 <== mon.2 v2:172.21.15.105:3300/0 2 ==== osd_map(116..116 src has 1..116) ==== 9379+0+0 (secure 0 0 0) 0x7f3b58011890 con 0x5629777c44d0 2024-05-27T07:32:15.087 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:32:15.093+0000 7f3b437fe640 1 -- 172.21.15.52:0/71137716 <== mon.2 v2:172.21.15.105:3300/0 3 ==== mon_get_version_reply(handle=1 version=101) ==== 24+0+0 (secure 0 0 0) 0x7f3b580ad380 con 0x5629777c44d0 2024-05-27T07:32:15.087 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:32:15.093+0000 7f3b7120f740 10 client.26405 fetch_fsmap learned FSMap version 101 2024-05-27T07:32:15.087 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:32:15.093+0000 7f3b7120f740 1 -- 172.21.15.52:0/71137716 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,fsmap=101,monmap=4+}) -- 0x562977b9f790 con 0x5629777c44d0 2024-05-27T07:32:15.088 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:32:15.093+0000 7f3b437fe640 1 -- 172.21.15.52:0/71137716 <== mon.2 v2:172.21.15.105:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f3b54004c50 con 0x5629777c44d0 2024-05-27T07:32:15.088 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:32:15.093+0000 7f3b437fe640 1 -- 172.21.15.52:0/71137716 <== mon.2 v2:172.21.15.105:3300/0 5 ==== fsmap(e 101) ==== 4203+0+0 (secure 0 0 0) 0x7f3b580a4b50 con 0x5629777c44d0 2024-05-27T07:32:15.088 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:32:15.093+0000 7f3b7120f740 10 client.26405 fetch_fsmap finished waiting for FSMap version 101 2024-05-27T07:32:15.088 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:32:15.093+0000 7f3b7120f740 10 client.26405 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-05-27T07:32:15.088 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:32:15.093+0000 7f3b7120f740 20 client.26405 populate_metadata read hostname 'smithi052' 2024-05-27T07:32:15.088 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:32:15.093+0000 7f3b7120f740 1 --2- 172.21.15.52:0/71137716 >> [v2:172.21.15.105:6838/500348151,v1:172.21.15.105:6839/500348151] conn(0x5629777a3d80 0x562977ba3290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:32:15.089 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:32:15.093+0000 7f3b7120f740 4 client.26405 mds_command: new command op to 24527 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-27T07:32:15.089 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:32:15.093+0000 7f3b7120f740 1 -- 172.21.15.52:0/71137716 --> [v2:172.21.15.105:6838/500348151,v1:172.21.15.105:6839/500348151] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x562977afab80 con 0x5629777a3d80 2024-05-27T07:32:15.089 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:32:15.094+0000 7f3b63fff640 1 --2- 172.21.15.52:0/71137716 >> [v2:172.21.15.105:6838/500348151,v1:172.21.15.105:6839/500348151] conn(0x5629777a3d80 0x562977ba3290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:32:15.089 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:32:15.094+0000 7f3b437fe640 10 client.26405 ms_handle_connect on v2:172.21.15.105:6838/500348151 2024-05-27T07:32:15.089 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:32:15.095+0000 7f3b63fff640 1 --2- 172.21.15.52:0/71137716 >> [v2:172.21.15.105:6838/500348151,v1:172.21.15.105:6839/500348151] conn(0x5629777a3d80 0x562977ba3290 crc :-1 s=READY pgs=127 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=9abf3af49674d419 server_cookie=979fe3e2a5d76fb1 in_seq=0 out_seq=0 2024-05-27T07:32:15.091 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:32:15.099+0000 7f3b437fe640 1 -- 172.21.15.52:0/71137716 <== mds.0 v2:172.21.15.105:6838/500348151 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24041 (crc 0 0 0) 0x562977afab80 con 0x5629777a3d80 2024-05-27T07:32:15.092 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:32:15.099+0000 7f3b437fe640 10 client.26405 handle_command_reply: tid=0 2024-05-27T07:32:15.092 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:32:15.099+0000 7f3b7120f740 1 client.26405 shutdown 2024-05-27T07:32:15.093 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:32:15.099+0000 7f3b7120f740 1 -- 172.21.15.52:0/71137716 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5629777c44d0 msgr2=0x562977b91110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:32:15.093 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:32:15.099+0000 7f3b7120f740 1 --2- 172.21.15.52:0/71137716 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5629777c44d0 0x562977b91110 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7f3b580df650 tx=0x7f3b580a8a20 comp rx=0 tx=0).stop 2024-05-27T07:32:15.093 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:32:15.099+0000 7f3b7120f740 1 -- 172.21.15.52:0/71137716 shutdown_connections 2024-05-27T07:32:15.093 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:32:15.099+0000 7f3b7120f740 1 --2- 172.21.15.52:0/71137716 >> [v2:172.21.15.105:6838/500348151,v1:172.21.15.105:6839/500348151] conn(0x5629777a3d80 0x562977ba3290 crc :-1 s=READY pgs=127 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:32:15.093 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:32:15.099+0000 7f3b7120f740 1 --2- 172.21.15.52:0/71137716 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5629777e51b0 0x562977b96a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:32:15.093 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:32:15.099+0000 7f3b7120f740 1 --2- 172.21.15.52:0/71137716 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5629777c44d0 0x562977b91110 unknown :-1 s=CLOSED pgs=200 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:32:15.093 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:32:15.099+0000 7f3b7120f740 1 --2- 172.21.15.52:0/71137716 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x5629777bc2f0 0x562977b90bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:32:15.093 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:32:15.099+0000 7f3b7120f740 1 -- 172.21.15.52:0/71137716 >> 172.21.15.52:0/71137716 conn(0x5629777d9300 msgr2=0x562977b863c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:32:15.093 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:32:15.100+0000 7f3b437fe640 0 client.26405 ms_handle_reset on v2:172.21.15.105:6838/500348151 2024-05-27T07:32:15.093 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:32:15.100+0000 7f3b7120f740 1 -- 172.21.15.52:0/71137716 shutdown_connections 2024-05-27T07:32:15.093 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:32:15.100+0000 7f3b7120f740 1 -- 172.21.15.52:0/71137716 wait complete. 2024-05-27T07:32:15.094 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:32:15.100+0000 7f3b7120f740 20 client.26405 trim_cache size 0 max 16384 2024-05-27T07:32:15.129 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:32:15.136+0000 7f3b7120f740 1 Processor -- start 2024-05-27T07:32:15.130 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:32:15.136+0000 7f3b7120f740 1 -- start start 2024-05-27T07:32:15.130 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:32:15.137+0000 7f3b7120f740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5629777bc2f0 0x562977b8b1f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:32:15.130 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:32:15.137+0000 7f3b7120f740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5629777c44d0 0x562977b86a30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:32:15.131 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:32:15.138+0000 7f3b637fe640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5629777c44d0 0x562977b86a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:32:15.131 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:32:15.138+0000 7f3b63fff640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5629777bc2f0 0x562977b8b1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:32:15.131 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:32:15.138+0000 7f3b637fe640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5629777c44d0 0x562977b86a30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.52:37546/0 (socket says 172.21.15.52:37546) 2024-05-27T07:32:15.131 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:32:15.138+0000 7f3b63fff640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5629777bc2f0 0x562977b8b1f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.173:3300/0 says I am v2:172.21.15.52:57370/0 (socket says 172.21.15.52:57370) 2024-05-27T07:32:15.131 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:32:15.138+0000 7f3b7120f740 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x5629777e51b0 0x562977b969d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:32:15.132 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:32:15.138+0000 7f3b637fe640 1 -- 172.21.15.52:0/134847905 learned_addr learned my addr 172.21.15.52:0/134847905 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-05-27T07:32:15.132 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:32:15.138+0000 7f3b7120f740 1 -- 172.21.15.52:0/134847905 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x7f3b500103b0 con 0x5629777e51b0 2024-05-27T07:32:15.132 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:32:15.138+0000 7f3b7120f740 1 -- 172.21.15.52:0/134847905 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f3b5000e7b0 con 0x5629777c44d0 2024-05-27T07:32:15.132 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:32:15.138+0000 7f3b7120f740 1 -- 172.21.15.52:0/134847905 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f3b5000fce0 con 0x5629777bc2f0 2024-05-27T07:32:15.132 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:32:15.139+0000 7f3b688fb640 1 --2- 172.21.15.52:0/134847905 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x5629777e51b0 0x562977b969d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:32:15.133 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:32:15.139+0000 7f3b63fff640 1 -- 172.21.15.52:0/134847905 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5629777c44d0 msgr2=0x562977b86a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:32:15.133 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:32:15.139+0000 7f3b63fff640 1 --2- 172.21.15.52:0/134847905 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5629777c44d0 0x562977b86a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:32:15.133 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:32:15.139+0000 7f3b63fff640 1 -- 172.21.15.52:0/134847905 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x5629777e51b0 msgr2=0x562977b969d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:32:15.133 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:32:15.139+0000 7f3b63fff640 1 --2- 172.21.15.52:0/134847905 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x5629777e51b0 0x562977b969d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:32:15.133 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:32:15.139+0000 7f3b63fff640 1 -- 172.21.15.52:0/134847905 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x562977b91760 con 0x5629777bc2f0 2024-05-27T07:32:15.133 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:32:15.139+0000 7f3b63fff640 1 --2- 172.21.15.52:0/134847905 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5629777bc2f0 0x562977b8b1f0 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x562977ba39f0 tx=0x7f3b54013040 comp rx=0 tx=0).ready entity=mon.1 client_cookie=af52cd7b8d18593a server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:32:15.133 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:32:15.140+0000 7f3b49ffb640 1 -- 172.21.15.52:0/134847905 <== mon.1 v2:172.21.15.173:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3b54011a80 con 0x5629777bc2f0 2024-05-27T07:32:15.133 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:32:15.140+0000 7f3b49ffb640 1 -- 172.21.15.52:0/134847905 <== mon.1 v2:172.21.15.173:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f3b54011c20 con 0x5629777bc2f0 2024-05-27T07:32:15.133 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:32:15.140+0000 7f3b49ffb640 1 -- 172.21.15.52:0/134847905 <== mon.1 v2:172.21.15.173:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3b5401b430 con 0x5629777bc2f0 2024-05-27T07:32:15.135 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:32:15.141+0000 7f3b7120f740 1 -- 172.21.15.52:0/134847905 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5629777bc2f0 msgr2=0x562977b8b1f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:32:15.135 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:32:15.141+0000 7f3b7120f740 1 --2- 172.21.15.52:0/134847905 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5629777bc2f0 0x562977b8b1f0 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x562977ba39f0 tx=0x7f3b54013040 comp rx=0 tx=0).stop 2024-05-27T07:32:15.135 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:32:15.142+0000 7f3b7120f740 1 -- 172.21.15.52:0/134847905 shutdown_connections 2024-05-27T07:32:15.135 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:32:15.142+0000 7f3b7120f740 1 --2- 172.21.15.52:0/134847905 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x5629777e51b0 0x562977b969d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:32:15.135 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:32:15.142+0000 7f3b7120f740 1 --2- 172.21.15.52:0/134847905 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5629777c44d0 0x562977b86a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:32:15.135 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:32:15.142+0000 7f3b7120f740 1 --2- 172.21.15.52:0/134847905 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5629777bc2f0 0x562977b8b1f0 unknown :-1 s=CLOSED pgs=155 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:32:15.135 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:32:15.142+0000 7f3b7120f740 1 -- 172.21.15.52:0/134847905 >> 172.21.15.52:0/134847905 conn(0x5629777d9300 msgr2=0x562977b86320 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:32:15.135 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:32:15.142+0000 7f3b7120f740 1 -- 172.21.15.52:0/134847905 shutdown_connections 2024-05-27T07:32:15.135 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:32:15.142+0000 7f3b7120f740 1 -- 172.21.15.52:0/134847905 wait complete. 2024-05-27T07:32:15.136 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:32:15.143+0000 7f3b7120f740 1 -- start start 2024-05-27T07:32:15.136 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:32:15.144+0000 7f3b7120f740 1 Processor -- start 2024-05-27T07:32:15.137 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:32:15.144+0000 7f3b7120f740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5629777bc2f0 0x562977b979f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:32:15.137 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:32:15.144+0000 7f3b7120f740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5629777c44d0 0x562977b97f30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:32:15.137 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:32:15.144+0000 7f3b7120f740 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x5629777e51b0 0x562977b98470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:32:15.138 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:32:15.144+0000 7f3b7120f740 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x7f3b5000f550 con 0x5629777e51b0 2024-05-27T07:32:15.138 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:32:15.144+0000 7f3b7120f740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f3b5000ee80 con 0x5629777c44d0 2024-05-27T07:32:15.138 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:32:15.144+0000 7f3b7120f740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x562977b8ef10 con 0x5629777bc2f0 2024-05-27T07:32:15.138 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:32:15.144+0000 7f3b7120f740 10 client.0 _ll_register_callbacks cb 0x7f3b580a8b80 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T07:32:15.138 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:32:15.144+0000 7f3b63fff640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5629777bc2f0 0x562977b979f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:32:15.138 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:32:15.145+0000 7f3b63fff640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5629777bc2f0 0x562977b979f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.173:3300/0 says I am v2:172.21.15.52:57382/0 (socket says 172.21.15.52:57382) 2024-05-27T07:32:15.138 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:32:15.145+0000 7f3b63fff640 1 -- 172.21.15.52:0/2563321497 learned_addr learned my addr 172.21.15.52:0/2563321497 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-05-27T07:32:15.139 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:32:15.145+0000 7f3b637fe640 1 --2- 172.21.15.52:0/2563321497 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5629777c44d0 0x562977b97f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:32:15.139 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:32:15.145+0000 7f3b688fb640 1 --2- 172.21.15.52:0/2563321497 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x5629777e51b0 0x562977b98470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:32:15.140 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:32:15.145+0000 7f3b688fb640 1 -- 172.21.15.52:0/2563321497 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5629777bc2f0 msgr2=0x562977b979f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:32:15.141 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:32:15.145+0000 7f3b688fb640 1 --2- 172.21.15.52:0/2563321497 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5629777bc2f0 0x562977b979f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:32:15.141 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:32:15.145+0000 7f3b688fb640 1 -- 172.21.15.52:0/2563321497 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5629777c44d0 msgr2=0x562977b97f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:32:15.141 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:32:15.145+0000 7f3b688fb640 1 --2- 172.21.15.52:0/2563321497 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5629777c44d0 0x562977b97f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:32:15.141 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:32:15.145+0000 7f3b688fb640 1 -- 172.21.15.52:0/2563321497 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({osdmap=0}) -- 0x562977c09910 con 0x5629777e51b0 2024-05-27T07:32:15.142 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:32:15.148+0000 7f3b688fb640 1 --2- 172.21.15.52:0/2563321497 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x5629777e51b0 0x562977b98470 secure :-1 s=READY pgs=679 cs=0 l=1 rev1=1 crypto rx=0x7f3b5c006010 tx=0x7f3b5c02a8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3fc725edd7fa8584 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:32:15.142 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:32:15.149+0000 7f3b437fe640 10 client.0 ms_handle_connect on v2:172.21.15.52:3300/0 2024-05-27T07:32:15.143 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:32:15.149+0000 7f3b437fe640 1 -- 172.21.15.52:0/2563321497 <== mon.0 v2:172.21.15.52:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3b5c048820 con 0x5629777e51b0 2024-05-27T07:32:15.143 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:32:15.149+0000 7f3b437fe640 1 -- 172.21.15.52:0/2563321497 <== mon.0 v2:172.21.15.52:3300/0 2 ==== osd_map(116..116 src has 1..116) ==== 9379+0+0 (secure 0 0 0) 0x7f3b5c0489c0 con 0x5629777e51b0 2024-05-27T07:32:15.143 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:32:15.149+0000 7f3b7120f740 1 -- 172.21.15.52:0/2563321497 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f3b54011c20 con 0x5629777e51b0 2024-05-27T07:32:15.143 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:32:15.150+0000 7f3b437fe640 1 -- 172.21.15.52:0/2563321497 <== mon.0 v2:172.21.15.52:3300/0 3 ==== mon_get_version_reply(handle=1 version=101) ==== 24+0+0 (secure 0 0 0) 0x7f3b5c031dc0 con 0x5629777e51b0 2024-05-27T07:32:15.143 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:32:15.150+0000 7f3b7120f740 10 client.16767 fetch_fsmap learned FSMap version 101 2024-05-27T07:32:15.143 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:32:15.150+0000 7f3b7120f740 1 -- 172.21.15.52:0/2563321497 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({config=0+,fsmap=101,monmap=4+}) -- 0x562977c0aad0 con 0x5629777e51b0 2024-05-27T07:32:15.143 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:32:15.150+0000 7f3b437fe640 1 -- 172.21.15.52:0/2563321497 <== mon.0 v2:172.21.15.52:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f3b5c059070 con 0x5629777e51b0 2024-05-27T07:32:15.144 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:32:15.150+0000 7f3b437fe640 1 -- 172.21.15.52:0/2563321497 <== mon.0 v2:172.21.15.52:3300/0 5 ==== fsmap(e 101) ==== 4203+0+0 (secure 0 0 0) 0x7f3b5c03c410 con 0x5629777e51b0 2024-05-27T07:32:15.145 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:32:15.152+0000 7f3b7120f740 10 client.16767 fetch_fsmap finished waiting for FSMap version 101 2024-05-27T07:32:15.145 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:32:15.152+0000 7f3b7120f740 10 client.16767 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-05-27T07:32:15.145 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:32:15.152+0000 7f3b7120f740 20 client.16767 populate_metadata read hostname 'smithi052' 2024-05-27T07:32:15.145 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:32:15.152+0000 7f3b7120f740 1 --2- 172.21.15.52:0/2563321497 >> [v2:172.21.15.105:6838/500348151,v1:172.21.15.105:6839/500348151] conn(0x5629777791c0 0x562977c0d320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:32:15.145 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:32:15.152+0000 7f3b7120f740 4 client.16767 mds_command: new command op to 24527 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-05-27T07:32:15.145 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:32:15.152+0000 7f3b7120f740 1 -- 172.21.15.52:0/2563321497 --> [v2:172.21.15.105:6838/500348151,v1:172.21.15.105:6839/500348151] -- command(tid 0: {"prefix": "flush journal"}) -- 0x7f3b54011c20 con 0x5629777791c0 2024-05-27T07:32:15.146 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:32:15.152+0000 7f3b63fff640 1 --2- 172.21.15.52:0/2563321497 >> [v2:172.21.15.105:6838/500348151,v1:172.21.15.105:6839/500348151] conn(0x5629777791c0 0x562977c0d320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:32:15.146 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:32:15.154+0000 7f3b63fff640 1 --2- 172.21.15.52:0/2563321497 >> [v2:172.21.15.105:6838/500348151,v1:172.21.15.105:6839/500348151] conn(0x5629777791c0 0x562977c0d320 crc :-1 s=READY pgs=128 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=f9e00a4ee00d3a48 server_cookie=f4b99e81ba05f3 in_seq=0 out_seq=0 2024-05-27T07:32:15.146 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:32:15.154+0000 7f3b437fe640 10 client.16767 ms_handle_connect on v2:172.21.15.105:6838/500348151 2024-05-27T07:32:15.221 INFO:tasks.workunit.client.0.smithi052.stdout: 144 580396 300 408319 221 351698 813 2024-05-27T07:32:16.443 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:32:16 smithi105 ceph-mon[35977]: pgmap v217: 129 pgs: 129 active+clean; 588 MiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 281 KiB/s rd, 4.5 MiB/s wr, 649 op/s 2024-05-27T07:32:16.482 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:32:16 smithi052 ceph-mon[31955]: pgmap v217: 129 pgs: 129 active+clean; 588 MiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 281 KiB/s rd, 4.5 MiB/s wr, 649 op/s 2024-05-27T07:32:16.605 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:32:16 smithi173 ceph-mon[37000]: pgmap v217: 129 pgs: 129 active+clean; 588 MiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 281 KiB/s rd, 4.5 MiB/s wr, 649 op/s 2024-05-27T07:32:18.140 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:32:18.146+0000 7f3b437fe640 1 -- 172.21.15.52:0/2563321497 <== mds.0 v2:172.21.15.105:6838/500348151 1 ==== command_reply(tid 0: 0 ) ==== 8+0+25 (crc 0 0 0) 0x7f3b54011c20 con 0x5629777791c0 2024-05-27T07:32:18.140 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:32:18.146+0000 7f3b437fe640 10 client.16767 handle_command_reply: tid=0 2024-05-27T07:32:18.140 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:32:18.146+0000 7f3b7120f740 1 client.16767 shutdown 2024-05-27T07:32:18.140 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:32:18.146+0000 7f3b7120f740 1 -- 172.21.15.52:0/2563321497 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x5629777e51b0 msgr2=0x562977b98470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:32:18.140 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:32:18.146+0000 7f3b7120f740 1 --2- 172.21.15.52:0/2563321497 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x5629777e51b0 0x562977b98470 secure :-1 s=READY pgs=679 cs=0 l=1 rev1=1 crypto rx=0x7f3b5c006010 tx=0x7f3b5c02a8f0 comp rx=0 tx=0).stop 2024-05-27T07:32:18.140 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:32:18.146+0000 7f3b7120f740 1 -- 172.21.15.52:0/2563321497 shutdown_connections 2024-05-27T07:32:18.140 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:32:18.146+0000 7f3b7120f740 1 --2- 172.21.15.52:0/2563321497 >> [v2:172.21.15.105:6838/500348151,v1:172.21.15.105:6839/500348151] conn(0x5629777791c0 0x562977c0d320 crc :-1 s=READY pgs=128 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:32:18.141 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:32:18.146+0000 7f3b7120f740 1 --2- 172.21.15.52:0/2563321497 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x5629777e51b0 0x562977b98470 unknown :-1 s=CLOSED pgs=679 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:32:18.141 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:32:18.146+0000 7f3b437fe640 0 client.16767 ms_handle_reset on v2:172.21.15.105:6838/500348151 2024-05-27T07:32:18.141 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:32:18.146+0000 7f3b7120f740 1 --2- 172.21.15.52:0/2563321497 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5629777c44d0 0x562977b97f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:32:18.141 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:32:18.146+0000 7f3b7120f740 1 --2- 172.21.15.52:0/2563321497 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5629777bc2f0 0x562977b979f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:32:18.141 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:32:18.146+0000 7f3b7120f740 1 -- 172.21.15.52:0/2563321497 >> 172.21.15.52:0/2563321497 conn(0x5629777d9300 msgr2=0x562977b8f290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:32:18.141 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:32:18.146+0000 7f3b7120f740 1 -- 172.21.15.52:0/2563321497 shutdown_connections 2024-05-27T07:32:18.141 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:32:18.146+0000 7f3b7120f740 1 -- 172.21.15.52:0/2563321497 wait complete. 2024-05-27T07:32:18.141 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:32:18.147+0000 7f3b7120f740 20 client.16767 trim_cache size 0 max 16384 2024-05-27T07:32:18.141 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stdout:{ 2024-05-27T07:32:18.142 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stdout: "return_code": 0 2024-05-27T07:32:18.142 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stdout:} 2024-05-27T07:32:18.142 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:32:18.147+0000 7f3b690fc640 1 -- 172.21.15.52:0/3731799994 >> [v2:172.21.15.52:6800/1822318384,v1:172.21.15.52:6801/1822318384] conn(0x7f3b4c0b2ae0 msgr2=0x7f3b4c0b4ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:32:18.143 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:32:18.147+0000 7f3b690fc640 1 --2- 172.21.15.52:0/3731799994 >> [v2:172.21.15.52:6800/1822318384,v1:172.21.15.52:6801/1822318384] conn(0x7f3b4c0b2ae0 0x7f3b4c0b4ff0 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7f3b5c004af0 tx=0x7f3b5c004a80 comp rx=0 tx=0).stop 2024-05-27T07:32:18.143 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:32:18.147+0000 7f3b690fc640 1 -- 172.21.15.52:0/3731799994 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f3b6410e860 msgr2=0x7f3b640803f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:32:18.143 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:32:18.147+0000 7f3b690fc640 1 --2- 172.21.15.52:0/3731799994 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f3b6410e860 0x7f3b640803f0 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7f3b5800b450 tx=0x7f3b5800b910 comp rx=0 tx=0).stop 2024-05-27T07:32:18.143 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:32:18.147+0000 7f3b690fc640 1 -- 172.21.15.52:0/3731799994 shutdown_connections 2024-05-27T07:32:18.143 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:32:18.147+0000 7f3b690fc640 1 --2- 172.21.15.52:0/3731799994 >> [v2:172.21.15.52:6800/1822318384,v1:172.21.15.52:6801/1822318384] conn(0x7f3b4c0b2ae0 0x7f3b4c0b4ff0 unknown :-1 s=CLOSED pgs=156 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:32:18.143 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:32:18.147+0000 7f3b690fc640 1 --2- 172.21.15.52:0/3731799994 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f3b64080a10 0x7f3b641bd780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:32:18.143 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:32:18.147+0000 7f3b690fc640 1 --2- 172.21.15.52:0/3731799994 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f3b6410e860 0x7f3b640803f0 unknown :-1 s=CLOSED pgs=199 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:32:18.143 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:32:18.147+0000 7f3b690fc640 1 --2- 172.21.15.52:0/3731799994 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f3b6410a380 0x7f3b6407feb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:32:18.143 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:32:18.147+0000 7f3b690fc640 1 -- 172.21.15.52:0/3731799994 >> 172.21.15.52:0/3731799994 conn(0x7f3b6406f800 msgr2=0x7f3b6410dbd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:32:18.143 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:32:18.147+0000 7f3b690fc640 1 -- 172.21.15.52:0/3731799994 shutdown_connections 2024-05-27T07:32:18.143 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:32:18.147+0000 7f3b690fc640 1 -- 172.21.15.52:0/3731799994 wait complete. 2024-05-27T07:32:18.482 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:32:18 smithi052 ceph-mon[31955]: pgmap v218: 129 pgs: 129 active+clean; 591 MiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 368 KiB/s rd, 4.5 MiB/s wr, 586 op/s 2024-05-27T07:32:18.605 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:32:18 smithi173 ceph-mon[37000]: pgmap v218: 129 pgs: 129 active+clean; 591 MiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 368 KiB/s rd, 4.5 MiB/s wr, 586 op/s 2024-05-27T07:32:18.693 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:32:18 smithi105 ceph-mon[35977]: pgmap v218: 129 pgs: 129 active+clean; 591 MiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 368 KiB/s rd, 4.5 MiB/s wr, 586 op/s 2024-05-27T07:32:20.482 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:32:20 smithi052 ceph-mon[31955]: scrub summary: active paths [/] 2024-05-27T07:32:20.482 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:32:20 smithi052 ceph-mon[31955]: scrub summary: idle+waiting paths [/] 2024-05-27T07:32:20.482 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:32:20 smithi052 ceph-mon[31955]: pgmap v219: 129 pgs: 129 active+clean; 587 MiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 417 KiB/s rd, 4.9 MiB/s wr, 894 op/s 2024-05-27T07:32:20.605 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:32:20 smithi173 ceph-mon[37000]: scrub summary: active paths [/] 2024-05-27T07:32:20.605 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:32:20 smithi173 ceph-mon[37000]: scrub summary: idle+waiting paths [/] 2024-05-27T07:32:20.605 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:32:20 smithi173 ceph-mon[37000]: pgmap v219: 129 pgs: 129 active+clean; 587 MiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 417 KiB/s rd, 4.9 MiB/s wr, 894 op/s 2024-05-27T07:32:20.693 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:32:20 smithi105 ceph-mon[35977]: scrub summary: active paths [/] 2024-05-27T07:32:20.693 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:32:20 smithi105 ceph-mon[35977]: scrub summary: idle+waiting paths [/] 2024-05-27T07:32:20.693 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:32:20 smithi105 ceph-mon[35977]: pgmap v219: 129 pgs: 129 active+clean; 587 MiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 417 KiB/s rd, 4.9 MiB/s wr, 894 op/s 2024-05-27T07:32:22.482 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:32:22 smithi052 ceph-mon[31955]: pgmap v220: 129 pgs: 129 active+clean; 563 MiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 496 KiB/s rd, 4.9 MiB/s wr, 1.35k op/s 2024-05-27T07:32:22.483 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:32:22 smithi052 ceph-mon[31955]: from='mgr.15333 172.21.15.52:0/416389082' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-27T07:32:22.605 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:32:22 smithi173 ceph-mon[37000]: pgmap v220: 129 pgs: 129 active+clean; 563 MiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 496 KiB/s rd, 4.9 MiB/s wr, 1.35k op/s 2024-05-27T07:32:22.605 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:32:22 smithi173 ceph-mon[37000]: from='mgr.15333 172.21.15.52:0/416389082' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-27T07:32:22.693 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:32:22 smithi105 ceph-mon[35977]: pgmap v220: 129 pgs: 129 active+clean; 563 MiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 496 KiB/s rd, 4.9 MiB/s wr, 1.35k op/s 2024-05-27T07:32:22.693 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:32:22 smithi105 ceph-mon[35977]: from='mgr.15333 172.21.15.52:0/416389082' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-27T07:32:24.487 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:32:24 smithi052 ceph-mon[31955]: pgmap v221: 129 pgs: 129 active+clean; 562 MiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 433 KiB/s rd, 3.5 MiB/s wr, 1.19k op/s 2024-05-27T07:32:24.605 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:32:24 smithi173 ceph-mon[37000]: pgmap v221: 129 pgs: 129 active+clean; 562 MiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 433 KiB/s rd, 3.5 MiB/s wr, 1.19k op/s 2024-05-27T07:32:24.693 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:32:24 smithi105 ceph-mon[35977]: pgmap v221: 129 pgs: 129 active+clean; 562 MiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 433 KiB/s rd, 3.5 MiB/s wr, 1.19k op/s 2024-05-27T07:32:25.220 INFO:tasks.workunit.client.0.smithi052.stdout: 149 1186067 493 830557 201 696956 1005 2024-05-27T07:32:26.482 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:32:26 smithi052 ceph-mon[31955]: pgmap v222: 129 pgs: 129 active+clean; 566 MiB data, 3.0 GiB used, 1.0 TiB / 1.0 TiB avail; 501 KiB/s rd, 3.3 MiB/s wr, 1.25k op/s 2024-05-27T07:32:26.605 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:32:26 smithi173 ceph-mon[37000]: pgmap v222: 129 pgs: 129 active+clean; 566 MiB data, 3.0 GiB used, 1.0 TiB / 1.0 TiB avail; 501 KiB/s rd, 3.3 MiB/s wr, 1.25k op/s 2024-05-27T07:32:26.693 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:32:26 smithi105 ceph-mon[35977]: pgmap v222: 129 pgs: 129 active+clean; 566 MiB data, 3.0 GiB used, 1.0 TiB / 1.0 TiB avail; 501 KiB/s rd, 3.3 MiB/s wr, 1.25k op/s 2024-05-27T07:32:27.482 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:32:27 smithi052 ceph-mon[31955]: from='mgr.15333 172.21.15.52:0/416389082' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T07:32:27.483 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:32:27 smithi052 ceph-mon[31955]: from='mgr.15333 172.21.15.52:0/416389082' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-27T07:32:27.483 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:32:27 smithi052 ceph-mon[31955]: from='mgr.15333 172.21.15.52:0/416389082' entity='mgr.x' 2024-05-27T07:32:27.605 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:32:27 smithi173 ceph-mon[37000]: from='mgr.15333 172.21.15.52:0/416389082' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T07:32:27.605 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:32:27 smithi173 ceph-mon[37000]: from='mgr.15333 172.21.15.52:0/416389082' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-27T07:32:27.605 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:32:27 smithi173 ceph-mon[37000]: from='mgr.15333 172.21.15.52:0/416389082' entity='mgr.x' 2024-05-27T07:32:27.693 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:32:27 smithi105 ceph-mon[35977]: from='mgr.15333 172.21.15.52:0/416389082' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T07:32:27.693 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:32:27 smithi105 ceph-mon[35977]: from='mgr.15333 172.21.15.52:0/416389082' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-27T07:32:27.693 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:32:27 smithi105 ceph-mon[35977]: from='mgr.15333 172.21.15.52:0/416389082' entity='mgr.x' 2024-05-27T07:32:28.482 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:32:28 smithi052 ceph-mon[31955]: pgmap v223: 129 pgs: 129 active+clean; 566 MiB data, 3.0 GiB used, 1.0 TiB / 1.0 TiB avail; 434 KiB/s rd, 2.2 MiB/s wr, 1.14k op/s 2024-05-27T07:32:28.605 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:32:28 smithi173 ceph-mon[37000]: pgmap v223: 129 pgs: 129 active+clean; 566 MiB data, 3.0 GiB used, 1.0 TiB / 1.0 TiB avail; 434 KiB/s rd, 2.2 MiB/s wr, 1.14k op/s 2024-05-27T07:32:28.693 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:32:28 smithi105 ceph-mon[35977]: pgmap v223: 129 pgs: 129 active+clean; 566 MiB data, 3.0 GiB used, 1.0 TiB / 1.0 TiB avail; 434 KiB/s rd, 2.2 MiB/s wr, 1.14k op/s 2024-05-27T07:32:30.483 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:32:30 smithi052 ceph-mon[31955]: pgmap v224: 129 pgs: 129 active+clean; 568 MiB data, 3.0 GiB used, 1.0 TiB / 1.0 TiB avail; 383 KiB/s rd, 2.1 MiB/s wr, 1.12k op/s 2024-05-27T07:32:30.605 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:32:30 smithi173 ceph-mon[37000]: pgmap v224: 129 pgs: 129 active+clean; 568 MiB data, 3.0 GiB used, 1.0 TiB / 1.0 TiB avail; 383 KiB/s rd, 2.1 MiB/s wr, 1.12k op/s 2024-05-27T07:32:30.693 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:32:30 smithi105 ceph-mon[35977]: pgmap v224: 129 pgs: 129 active+clean; 568 MiB data, 3.0 GiB used, 1.0 TiB / 1.0 TiB avail; 383 KiB/s rd, 2.1 MiB/s wr, 1.12k op/s 2024-05-27T07:32:31.303 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:32:31.309+0000 7f9de0100640 1 -- 172.21.15.52:0/3770707813 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f9dd8112900 msgr2=0x7f9dd8118eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:32:31.303 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:32:31.309+0000 7f9de0100640 1 --2- 172.21.15.52:0/3770707813 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f9dd8112900 0x7f9dd8118eb0 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7f9dc8009860 tx=0x7f9dc802a3d0 comp rx=0 tx=0).stop 2024-05-27T07:32:31.303 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:32:31.309+0000 7f9de0100640 1 -- 172.21.15.52:0/3770707813 shutdown_connections 2024-05-27T07:32:31.303 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:32:31.309+0000 7f9de0100640 1 --2- 172.21.15.52:0/3770707813 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f9dd8112900 0x7f9dd8118eb0 unknown :-1 s=CLOSED pgs=201 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:32:31.303 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:32:31.309+0000 7f9de0100640 1 --2- 172.21.15.52:0/3770707813 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f9dd8111de0 0x7f9dd8112270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:32:31.303 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:32:31.309+0000 7f9de0100640 1 --2- 172.21.15.52:0/3770707813 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f9dd810a490 0x7f9dd810a8a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:32:31.303 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:32:31.309+0000 7f9de0100640 1 -- 172.21.15.52:0/3770707813 >> 172.21.15.52:0/3770707813 conn(0x7f9dd806e8f0 msgr2=0x7f9dd806ed30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:32:31.306 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:32:31.313+0000 7f9de0100640 1 -- 172.21.15.52:0/3770707813 shutdown_connections 2024-05-27T07:32:31.307 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:32:31.314+0000 7f9de0100640 1 -- 172.21.15.52:0/3770707813 wait complete. 2024-05-27T07:32:31.308 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:32:31.315+0000 7f9de0100640 1 Processor -- start 2024-05-27T07:32:31.308 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:32:31.315+0000 7f9de0100640 1 -- start start 2024-05-27T07:32:31.309 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:32:31.315+0000 7f9de0100640 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f9dd810a490 0x7f9dd81a50b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:32:31.309 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:32:31.315+0000 7f9de0100640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f9dd8111de0 0x7f9dd81a5610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:32:31.309 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:32:31.315+0000 7f9de0100640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f9dd8112900 0x7f9dd81a98d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:32:31.309 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:32:31.316+0000 7f9de0100640 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x7f9dd811c6b0 con 0x7f9dd810a490 2024-05-27T07:32:31.309 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:32:31.316+0000 7f9de0100640 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f9dd811c530 con 0x7f9dd8111de0 2024-05-27T07:32:31.309 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:32:31.316+0000 7f9ddd69b640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f9dd8111de0 0x7f9dd81a5610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:32:31.309 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:32:31.316+0000 7f9ddd69b640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f9dd8111de0 0x7f9dd81a5610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.52:60964/0 (socket says 172.21.15.52:60964) 2024-05-27T07:32:31.310 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:32:31.316+0000 7f9ddd69b640 1 -- 172.21.15.52:0/3198512478 learned_addr learned my addr 172.21.15.52:0/3198512478 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-05-27T07:32:31.310 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:32:31.316+0000 7f9dde69d640 1 --2- 172.21.15.52:0/3198512478 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f9dd8112900 0x7f9dd81a98d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:32:31.310 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:32:31.316+0000 7f9de0100640 1 -- 172.21.15.52:0/3198512478 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f9dd811c830 con 0x7f9dd8112900 2024-05-27T07:32:31.311 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:32:31.316+0000 7f9ddde9c640 1 --2- 172.21.15.52:0/3198512478 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f9dd810a490 0x7f9dd81a50b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:32:31.311 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:32:31.316+0000 7f9ddd69b640 1 -- 172.21.15.52:0/3198512478 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f9dd8112900 msgr2=0x7f9dd81a98d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:32:31.311 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:32:31.317+0000 7f9ddd69b640 1 --2- 172.21.15.52:0/3198512478 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f9dd8112900 0x7f9dd81a98d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:32:31.311 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:32:31.317+0000 7f9ddd69b640 1 -- 172.21.15.52:0/3198512478 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f9dd810a490 msgr2=0x7f9dd81a50b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:32:31.311 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:32:31.317+0000 7f9ddd69b640 1 --2- 172.21.15.52:0/3198512478 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f9dd810a490 0x7f9dd81a50b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:32:31.311 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:32:31.317+0000 7f9ddd69b640 1 -- 172.21.15.52:0/3198512478 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9dd81a9f10 con 0x7f9dd8111de0 2024-05-27T07:32:31.311 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:32:31.317+0000 7f9dde69d640 1 --2- 172.21.15.52:0/3198512478 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f9dd8112900 0x7f9dd81a98d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-27T07:32:31.311 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:32:31.317+0000 7f9ddde9c640 1 --2- 172.21.15.52:0/3198512478 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f9dd810a490 0x7f9dd81a50b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-27T07:32:31.312 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:32:31.317+0000 7f9ddd69b640 1 --2- 172.21.15.52:0/3198512478 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f9dd8111de0 0x7f9dd81a5610 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7f9dd400b4a0 tx=0x7f9dd400b960 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:32:31.312 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:32:31.318+0000 7f9dceffd640 1 -- 172.21.15.52:0/3198512478 <== mon.2 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9dd4014070 con 0x7f9dd8111de0 2024-05-27T07:32:31.312 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:32:31.318+0000 7f9dceffd640 1 -- 172.21.15.52:0/3198512478 <== mon.2 v2:172.21.15.105:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f9dd4010ac0 con 0x7f9dd8111de0 2024-05-27T07:32:31.312 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:32:31.318+0000 7f9dceffd640 1 -- 172.21.15.52:0/3198512478 <== mon.2 v2:172.21.15.105:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9dd4017c30 con 0x7f9dd8111de0 2024-05-27T07:32:31.312 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:32:31.319+0000 7f9de0100640 1 -- 172.21.15.52:0/3198512478 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9dd81aa200 con 0x7f9dd8111de0 2024-05-27T07:32:31.312 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:32:31.319+0000 7f9de0100640 1 -- 172.21.15.52:0/3198512478 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9dd81aa700 con 0x7f9dd8111de0 2024-05-27T07:32:31.319 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:32:31.322+0000 7f9de6fba740 1 Processor -- start 2024-05-27T07:32:31.320 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:32:31.323+0000 7f9de6fba740 1 -- start start 2024-05-27T07:32:31.320 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:32:31.323+0000 7f9de6fba740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55a9780b31b0 0x55a9784542c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:32:31.320 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:32:31.323+0000 7f9de6fba740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55a97808a2f0 0x55a978454800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:32:31.320 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:32:31.323+0000 7f9de6fba740 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55a9780924d0 0x55a978454d40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:32:31.320 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:32:31.323+0000 7f9de6fba740 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x55a9783c6c30 con 0x55a9780924d0 2024-05-27T07:32:31.320 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:32:31.323+0000 7f9de6fba740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x55a9783b67d0 con 0x55a97808a2f0 2024-05-27T07:32:31.320 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:32:31.323+0000 7f9de6fba740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x55a9783ba810 con 0x55a9780b31b0 2024-05-27T07:32:31.320 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:32:31.324+0000 7f9dde69d640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55a9780b31b0 0x55a9784542c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:32:31.320 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:32:31.324+0000 7f9dde69d640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55a9780b31b0 0x55a9784542c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.173:3300/0 says I am v2:172.21.15.52:55228/0 (socket says 172.21.15.52:55228) 2024-05-27T07:32:31.320 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:32:31.324+0000 7f9dde69d640 1 -- 172.21.15.52:0/2783913365 learned_addr learned my addr 172.21.15.52:0/2783913365 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-05-27T07:32:31.320 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:32:31.324+0000 7f9ddde9c640 1 --2- 172.21.15.52:0/2783913365 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55a97808a2f0 0x55a978454800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:32:31.321 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:32:31.325+0000 7f9dde69d640 1 -- 172.21.15.52:0/2783913365 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55a97808a2f0 msgr2=0x55a978454800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:32:31.321 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:32:31.325+0000 7f9dde69d640 1 --2- 172.21.15.52:0/2783913365 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55a97808a2f0 0x55a978454800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:32:31.321 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:32:31.325+0000 7f9dde69d640 1 -- 172.21.15.52:0/2783913365 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55a9780924d0 msgr2=0x55a978454d40 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-05-27T07:32:31.321 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:32:31.325+0000 7f9dde69d640 1 --2- 172.21.15.52:0/2783913365 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55a9780924d0 0x55a978454d40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:32:31.321 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:32:31.325+0000 7f9dde69d640 1 -- 172.21.15.52:0/2783913365 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55a9780cb210 con 0x55a9780b31b0 2024-05-27T07:32:31.321 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:32:31.325+0000 7f9dde69d640 1 --2- 172.21.15.52:0/2783913365 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55a9780b31b0 0x55a9784542c0 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7f9dc802a8f0 tx=0x7f9dc80305c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=17b56878aee9cd7 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:32:31.321 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:32:31.326+0000 7f9dc3fff640 1 -- 172.21.15.52:0/2783913365 <== mon.1 v2:172.21.15.173:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9dc8041070 con 0x55a9780b31b0 2024-05-27T07:32:31.321 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:32:31.326+0000 7f9dc3fff640 1 -- 172.21.15.52:0/2783913365 <== mon.1 v2:172.21.15.173:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f9dc8005d20 con 0x55a9780b31b0 2024-05-27T07:32:31.321 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:32:31.326+0000 7f9dc3fff640 1 -- 172.21.15.52:0/2783913365 <== mon.1 v2:172.21.15.173:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9dc80046c0 con 0x55a9780b31b0 2024-05-27T07:32:31.322 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:32:31.327+0000 7f9de6fba740 1 -- 172.21.15.52:0/2783913365 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55a9780b31b0 msgr2=0x55a9784542c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:32:31.322 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:32:31.327+0000 7f9de6fba740 1 --2- 172.21.15.52:0/2783913365 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55a9780b31b0 0x55a9784542c0 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7f9dc802a8f0 tx=0x7f9dc80305c0 comp rx=0 tx=0).stop 2024-05-27T07:32:31.323 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:32:31.327+0000 7f9dceffd640 1 -- 172.21.15.52:0/3198512478 <== mon.2 v2:172.21.15.105:3300/0 4 ==== mgrmap(e 26) ==== 149783+0+0 (secure 0 0 0) 0x7f9dd4017070 con 0x7f9dd8111de0 2024-05-27T07:32:31.323 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:32:31.328+0000 7f9de6fba740 1 -- 172.21.15.52:0/2783913365 shutdown_connections 2024-05-27T07:32:31.323 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:32:31.328+0000 7f9de6fba740 1 --2- 172.21.15.52:0/2783913365 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55a9780924d0 0x55a978454d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:32:31.323 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:32:31.328+0000 7f9de6fba740 1 --2- 172.21.15.52:0/2783913365 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55a97808a2f0 0x55a978454800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:32:31.323 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:32:31.328+0000 7f9de6fba740 1 --2- 172.21.15.52:0/2783913365 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55a9780b31b0 0x55a9784542c0 unknown :-1 s=CLOSED pgs=156 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:32:31.323 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:32:31.328+0000 7f9de6fba740 1 -- 172.21.15.52:0/2783913365 >> 172.21.15.52:0/2783913365 conn(0x55a9780a7300 msgr2=0x55a978450b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:32:31.323 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:32:31.328+0000 7f9de6fba740 1 -- 172.21.15.52:0/2783913365 shutdown_connections 2024-05-27T07:32:31.323 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:32:31.328+0000 7f9de6fba740 1 -- 172.21.15.52:0/2783913365 wait complete. 2024-05-27T07:32:31.323 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:32:31.329+0000 7f9de6fba740 1 -- start start 2024-05-27T07:32:31.323 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:32:31.329+0000 7f9de6fba740 1 Processor -- start 2024-05-27T07:32:31.325 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:32:31.329+0000 7f9dceffd640 1 --2- 172.21.15.52:0/3198512478 >> [v2:172.21.15.52:6800/1822318384,v1:172.21.15.52:6801/1822318384] conn(0x7f9dc40b2a30 0x7f9dc40b4f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:32:31.325 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:32:31.330+0000 7f9dceffd640 1 -- 172.21.15.52:0/3198512478 <== mon.2 v2:172.21.15.105:3300/0 5 ==== osd_map(116..116 src has 1..116) ==== 9379+0+0 (secure 0 0 0) 0x7f9dd40e1f80 con 0x7f9dd8111de0 2024-05-27T07:32:31.325 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:32:31.331+0000 7f9de6fba740 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55a97808a2f0 0x55a978463b60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:32:31.325 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:32:31.331+0000 7f9de6fba740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55a9780924d0 0x55a9784640c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:32:31.325 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:32:31.331+0000 7f9de6fba740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55a9780b31b0 0x55a97845ef40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:32:31.325 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:32:31.331+0000 7f9de6fba740 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x55a97809c790 con 0x55a97808a2f0 2024-05-27T07:32:31.325 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:32:31.331+0000 7f9de6fba740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x55a978372890 con 0x55a9780924d0 2024-05-27T07:32:31.325 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:32:31.331+0000 7f9de6fba740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x55a978457430 con 0x55a9780b31b0 2024-05-27T07:32:31.326 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:32:31.331+0000 7f9dde69d640 1 --2- 172.21.15.52:0/3198512478 >> [v2:172.21.15.52:6800/1822318384,v1:172.21.15.52:6801/1822318384] conn(0x7f9dc40b2a30 0x7f9dc40b4f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:32:31.326 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:32:31.332+0000 7f9dde69d640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55a9780b31b0 0x55a97845ef40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:32:31.326 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:32:31.332+0000 7f9dde69d640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55a9780b31b0 0x55a97845ef40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.173:3300/0 says I am v2:172.21.15.52:55232/0 (socket says 172.21.15.52:55232) 2024-05-27T07:32:31.326 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:32:31.332+0000 7f9dde69d640 1 -- 172.21.15.52:0/1792390534 learned_addr learned my addr 172.21.15.52:0/1792390534 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-05-27T07:32:31.326 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:32:31.332+0000 7f9de6fba740 10 client.0 _ll_register_callbacks cb 0x55a9780caa00 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T07:32:31.326 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:32:31.332+0000 7f9ddd69b640 1 --2- 172.21.15.52:0/1792390534 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55a9780924d0 0x55a9784640c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:32:31.326 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:32:31.332+0000 7f9dde69d640 1 -- 172.21.15.52:0/1792390534 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55a9780924d0 msgr2=0x55a9784640c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:32:31.327 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:32:31.332+0000 7f9dde69d640 1 --2- 172.21.15.52:0/1792390534 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55a9780924d0 0x55a9784640c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:32:31.327 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:32:31.332+0000 7f9dde69d640 1 -- 172.21.15.52:0/1792390534 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55a97808a2f0 msgr2=0x55a978463b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:32:31.328 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:32:31.332+0000 7f9dde69d640 1 --2- 172.21.15.52:0/1792390534 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55a97808a2f0 0x55a978463b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:32:31.328 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:32:31.332+0000 7f9dde69d640 1 -- 172.21.15.52:0/1792390534 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55a978197620 con 0x55a9780b31b0 2024-05-27T07:32:31.328 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:32:31.333+0000 7f9dde69d640 1 --2- 172.21.15.52:0/1792390534 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55a9780b31b0 0x55a97845ef40 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7f9dc8030500 tx=0x7f9dc8030db0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=978e7979375ab0a0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:32:31.328 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:32:31.333+0000 7f9ddd69b640 1 --2- 172.21.15.52:0/1792390534 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55a9780924d0 0x55a9784640c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-27T07:32:31.328 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:32:31.333+0000 7f9dc17fa640 10 client.0 ms_handle_connect on v2:172.21.15.173:3300/0 2024-05-27T07:32:31.328 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:32:31.333+0000 7f9dc17fa640 1 -- 172.21.15.52:0/1792390534 <== mon.1 v2:172.21.15.173:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9dc8041050 con 0x55a9780b31b0 2024-05-27T07:32:31.328 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:32:31.333+0000 7f9de6fba740 1 -- 172.21.15.52:0/1792390534 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x55a9783c8b80 con 0x55a9780b31b0 2024-05-27T07:32:31.328 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:32:31.333+0000 7f9dde69d640 1 --2- 172.21.15.52:0/3198512478 >> [v2:172.21.15.52:6800/1822318384,v1:172.21.15.52:6801/1822318384] conn(0x7f9dc40b2a30 0x7f9dc40b4f40 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7f9dc8030e20 tx=0x7f9dc803e000 comp rx=0 tx=0).ready entity=mgr.15333 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:32:31.328 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:32:31.333+0000 7f9dc17fa640 1 -- 172.21.15.52:0/1792390534 <== mon.1 v2:172.21.15.173:3300/0 2 ==== osd_map(116..116 src has 1..116) ==== 9379+0+0 (secure 0 0 0) 0x7f9dc80035f0 con 0x55a9780b31b0 2024-05-27T07:32:31.330 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:32:31.336+0000 7f9dc17fa640 1 -- 172.21.15.52:0/1792390534 <== mon.1 v2:172.21.15.173:3300/0 3 ==== mon_get_version_reply(handle=1 version=101) ==== 24+0+0 (secure 0 0 0) 0x7f9dc8062050 con 0x55a9780b31b0 2024-05-27T07:32:31.330 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:32:31.337+0000 7f9de6fba740 10 client.26272 fetch_fsmap learned FSMap version 101 2024-05-27T07:32:31.330 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:32:31.337+0000 7f9de6fba740 1 -- 172.21.15.52:0/1792390534 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({config=0+,fsmap=101,monmap=4+}) -- 0x55a97847e920 con 0x55a9780b31b0 2024-05-27T07:32:31.331 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:32:31.337+0000 7f9dc17fa640 1 -- 172.21.15.52:0/1792390534 <== mon.1 v2:172.21.15.173:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x55a9783c8b80 con 0x55a9780b31b0 2024-05-27T07:32:31.331 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:32:31.337+0000 7f9dc17fa640 1 -- 172.21.15.52:0/1792390534 <== mon.1 v2:172.21.15.173:3300/0 5 ==== fsmap(e 101) ==== 4203+0+0 (secure 0 0 0) 0x7f9dc803d430 con 0x55a9780b31b0 2024-05-27T07:32:31.331 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:32:31.337+0000 7f9de6fba740 10 client.26272 fetch_fsmap finished waiting for FSMap version 101 2024-05-27T07:32:31.331 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:32:31.337+0000 7f9de6fba740 10 client.26272 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-05-27T07:32:31.331 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:32:31.337+0000 7f9de6fba740 20 client.26272 populate_metadata read hostname 'smithi052' 2024-05-27T07:32:31.331 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:32:31.337+0000 7f9de6fba740 1 --2- 172.21.15.52:0/1792390534 >> [v2:172.21.15.105:6838/500348151,v1:172.21.15.105:6839/500348151] conn(0x55a978071d80 0x55a9784820a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:32:31.331 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:32:31.337+0000 7f9de6fba740 4 client.26272 mds_command: new command op to 24527 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-27T07:32:31.332 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:32:31.337+0000 7f9de6fba740 1 -- 172.21.15.52:0/1792390534 --> [v2:172.21.15.105:6838/500348151,v1:172.21.15.105:6839/500348151] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55a9783c2360 con 0x55a978071d80 2024-05-27T07:32:31.332 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:32:31.338+0000 7f9ddde9c640 1 --2- 172.21.15.52:0/1792390534 >> [v2:172.21.15.105:6838/500348151,v1:172.21.15.105:6839/500348151] conn(0x55a978071d80 0x55a9784820a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:32:31.332 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:32:31.338+0000 7f9ddde9c640 1 --2- 172.21.15.52:0/1792390534 >> [v2:172.21.15.105:6838/500348151,v1:172.21.15.105:6839/500348151] conn(0x55a978071d80 0x55a9784820a0 crc :-1 s=READY pgs=129 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=c0e8df56ed9f77c server_cookie=b0e84d6302c2ca45 in_seq=0 out_seq=0 2024-05-27T07:32:31.332 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:32:31.339+0000 7f9dc17fa640 10 client.26272 ms_handle_connect on v2:172.21.15.105:6838/500348151 2024-05-27T07:32:31.338 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:32:31.342+0000 7f9dc17fa640 1 -- 172.21.15.52:0/1792390534 <== mds.0 v2:172.21.15.105:6838/500348151 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24041 (crc 0 0 0) 0x55a9783c2360 con 0x55a978071d80 2024-05-27T07:32:31.338 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:32:31.342+0000 7f9dc17fa640 10 client.26272 handle_command_reply: tid=0 2024-05-27T07:32:31.338 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:32:31.342+0000 7f9de6fba740 1 client.26272 shutdown 2024-05-27T07:32:31.338 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:32:31.345+0000 7f9de6fba740 1 -- 172.21.15.52:0/1792390534 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55a9780b31b0 msgr2=0x55a97845ef40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:32:31.338 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:32:31.345+0000 7f9de6fba740 1 --2- 172.21.15.52:0/1792390534 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55a9780b31b0 0x55a97845ef40 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7f9dc8030500 tx=0x7f9dc8030db0 comp rx=0 tx=0).stop 2024-05-27T07:32:31.341 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:32:31.348+0000 7f9de6fba740 1 -- 172.21.15.52:0/1792390534 shutdown_connections 2024-05-27T07:32:31.341 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:32:31.348+0000 7f9de6fba740 1 --2- 172.21.15.52:0/1792390534 >> [v2:172.21.15.105:6838/500348151,v1:172.21.15.105:6839/500348151] conn(0x55a978071d80 0x55a9784820a0 crc :-1 s=READY pgs=129 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:32:31.341 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:32:31.348+0000 7f9de6fba740 1 --2- 172.21.15.52:0/1792390534 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55a9780b31b0 0x55a97845ef40 unknown :-1 s=CLOSED pgs=157 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:32:31.341 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:32:31.348+0000 7f9de6fba740 1 --2- 172.21.15.52:0/1792390534 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55a9780924d0 0x55a9784640c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:32:31.341 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:32:31.348+0000 7f9de6fba740 1 --2- 172.21.15.52:0/1792390534 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55a97808a2f0 0x55a978463b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:32:31.341 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:32:31.348+0000 7f9de6fba740 1 -- 172.21.15.52:0/1792390534 >> 172.21.15.52:0/1792390534 conn(0x55a9780a7300 msgr2=0x55a978454160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:32:31.342 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:32:31.349+0000 7f9dc17fa640 0 client.26272 ms_handle_reset on v2:172.21.15.105:6838/500348151 2024-05-27T07:32:31.349 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:32:31.356+0000 7f9de6fba740 1 -- 172.21.15.52:0/1792390534 shutdown_connections 2024-05-27T07:32:31.351 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:32:31.356+0000 7f9de6fba740 1 -- 172.21.15.52:0/1792390534 wait complete. 2024-05-27T07:32:31.351 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:32:31.357+0000 7f9de6fba740 20 client.26272 trim_cache size 0 max 16384 2024-05-27T07:32:31.388 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:32:31.395+0000 7f9de6fba740 1 Processor -- start 2024-05-27T07:32:31.389 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:32:31.396+0000 7f9de6fba740 1 -- start start 2024-05-27T07:32:31.389 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:32:31.396+0000 7f9de6fba740 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55a97808a2f0 0x55a978454870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:32:31.390 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:32:31.396+0000 7f9de6fba740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55a9780924d0 0x55a97845a720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:32:31.390 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:32:31.396+0000 7f9de6fba740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55a9780b31b0 0x55a978463530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:32:31.390 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:32:31.396+0000 7f9de6fba740 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x7f9dd81b7ae0 con 0x55a97808a2f0 2024-05-27T07:32:31.390 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:32:31.396+0000 7f9de6fba740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f9dd81b5ee0 con 0x55a9780924d0 2024-05-27T07:32:31.390 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:32:31.396+0000 7f9de6fba740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f9dd81b7410 con 0x55a9780b31b0 2024-05-27T07:32:31.391 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:32:31.397+0000 7f9ddd69b640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55a9780924d0 0x55a97845a720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:32:31.391 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:32:31.397+0000 7f9ddd69b640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55a9780924d0 0x55a97845a720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.52:32776/0 (socket says 172.21.15.52:32776) 2024-05-27T07:32:31.391 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:32:31.397+0000 7f9ddd69b640 1 -- 172.21.15.52:0/1450800642 learned_addr learned my addr 172.21.15.52:0/1450800642 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-05-27T07:32:31.391 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:32:31.398+0000 7f9dde69d640 1 --2- 172.21.15.52:0/1450800642 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55a9780b31b0 0x55a978463530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:32:31.393 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:32:31.398+0000 7f9ddd69b640 1 -- 172.21.15.52:0/1450800642 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55a9780b31b0 msgr2=0x55a978463530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:32:31.393 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:32:31.398+0000 7f9ddd69b640 1 --2- 172.21.15.52:0/1450800642 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55a9780b31b0 0x55a978463530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:32:31.393 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:32:31.398+0000 7f9ddd69b640 1 -- 172.21.15.52:0/1450800642 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55a97808a2f0 msgr2=0x55a978454870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:32:31.393 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:32:31.398+0000 7f9ddd69b640 1 --2- 172.21.15.52:0/1450800642 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55a97808a2f0 0x55a978454870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:32:31.393 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:32:31.398+0000 7f9ddd69b640 1 -- 172.21.15.52:0/1450800642 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55a978464430 con 0x55a9780924d0 2024-05-27T07:32:31.393 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:32:31.399+0000 7f9ddd69b640 1 --2- 172.21.15.52:0/1450800642 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55a9780924d0 0x55a97845a720 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7f9dd40a6220 tx=0x7f9dd40047d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d2aa90bf2d62fcca server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:32:31.393 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:32:31.400+0000 7f9dc37fe640 1 -- 172.21.15.52:0/1450800642 <== mon.2 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9dd401e050 con 0x55a9780924d0 2024-05-27T07:32:31.393 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:32:31.400+0000 7f9dc37fe640 1 -- 172.21.15.52:0/1450800642 <== mon.2 v2:172.21.15.105:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f9dd40b1e10 con 0x55a9780924d0 2024-05-27T07:32:31.393 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:32:31.400+0000 7f9dc37fe640 1 -- 172.21.15.52:0/1450800642 <== mon.2 v2:172.21.15.105:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9dd40b1980 con 0x55a9780924d0 2024-05-27T07:32:31.394 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:32:31.401+0000 7f9de6fba740 1 -- 172.21.15.52:0/1450800642 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55a9780924d0 msgr2=0x55a97845a720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:32:31.394 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:32:31.401+0000 7f9de6fba740 1 --2- 172.21.15.52:0/1450800642 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55a9780924d0 0x55a97845a720 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7f9dd40a6220 tx=0x7f9dd40047d0 comp rx=0 tx=0).stop 2024-05-27T07:32:31.395 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:32:31.402+0000 7f9de6fba740 1 -- 172.21.15.52:0/1450800642 shutdown_connections 2024-05-27T07:32:31.395 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:32:31.402+0000 7f9de6fba740 1 --2- 172.21.15.52:0/1450800642 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55a9780b31b0 0x55a978463530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:32:31.395 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:32:31.402+0000 7f9de6fba740 1 --2- 172.21.15.52:0/1450800642 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55a9780924d0 0x55a97845a720 unknown :-1 s=CLOSED pgs=203 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:32:31.395 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:32:31.402+0000 7f9de6fba740 1 --2- 172.21.15.52:0/1450800642 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55a97808a2f0 0x55a978454870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:32:31.395 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:32:31.402+0000 7f9de6fba740 1 -- 172.21.15.52:0/1450800642 >> 172.21.15.52:0/1450800642 conn(0x55a9780a7300 msgr2=0x55a978454160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:32:31.396 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:32:31.403+0000 7f9de6fba740 1 -- 172.21.15.52:0/1450800642 shutdown_connections 2024-05-27T07:32:31.396 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:32:31.403+0000 7f9de6fba740 1 -- 172.21.15.52:0/1450800642 wait complete. 2024-05-27T07:32:31.399 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:32:31.403+0000 7f9de6fba740 1 -- start start 2024-05-27T07:32:31.399 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:32:31.404+0000 7f9de6fba740 1 Processor -- start 2024-05-27T07:32:31.399 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:32:31.404+0000 7f9de6fba740 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55a97808a2f0 0x55a978493a70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:32:31.399 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:32:31.404+0000 7f9de6fba740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55a9780924d0 0x55a97848e450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:32:31.400 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:32:31.404+0000 7f9de6fba740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55a9780b31b0 0x55a97848e990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:32:31.400 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:32:31.404+0000 7f9de6fba740 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x7f9dd81b6c80 con 0x55a97808a2f0 2024-05-27T07:32:31.400 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:32:31.404+0000 7f9de6fba740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f9dd81b65b0 con 0x55a9780b31b0 2024-05-27T07:32:31.400 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:32:31.404+0000 7f9de6fba740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x55a9783c2360 con 0x55a9780924d0 2024-05-27T07:32:31.400 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:32:31.404+0000 7f9de6fba740 10 client.0 _ll_register_callbacks cb 0x7f9dd0009ac0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T07:32:31.400 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:32:31.405+0000 7f9dde69d640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55a9780b31b0 0x55a97848e990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:32:31.400 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:32:31.405+0000 7f9dde69d640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55a9780b31b0 0x55a97848e990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.52:32790/0 (socket says 172.21.15.52:32790) 2024-05-27T07:32:31.400 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:32:31.405+0000 7f9dde69d640 1 -- 172.21.15.52:0/536728758 learned_addr learned my addr 172.21.15.52:0/536728758 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-05-27T07:32:31.400 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:32:31.405+0000 7f9dde69d640 1 -- 172.21.15.52:0/536728758 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55a9780924d0 msgr2=0x55a97848e450 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-05-27T07:32:31.400 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:32:31.405+0000 7f9dde69d640 1 --2- 172.21.15.52:0/536728758 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55a9780924d0 0x55a97848e450 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:32:31.400 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:32:31.405+0000 7f9dde69d640 1 -- 172.21.15.52:0/536728758 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55a97808a2f0 msgr2=0x55a978493a70 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-05-27T07:32:31.400 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:32:31.405+0000 7f9dde69d640 1 --2- 172.21.15.52:0/536728758 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55a97808a2f0 0x55a978493a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:32:31.401 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:32:31.405+0000 7f9dde69d640 1 -- 172.21.15.52:0/536728758 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55a97848eed0 con 0x55a9780b31b0 2024-05-27T07:32:31.401 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:32:31.405+0000 7f9dde69d640 1 --2- 172.21.15.52:0/536728758 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55a9780b31b0 0x55a97848e990 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7f9dc802a990 tx=0x7f9dc8030ec0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=2c97a5e2fdb9cda8 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:32:31.401 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:32:31.405+0000 7f9dc17fa640 10 client.0 ms_handle_connect on v2:172.21.15.105:3300/0 2024-05-27T07:32:31.401 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:32:31.406+0000 7f9dc17fa640 1 -- 172.21.15.52:0/536728758 <== mon.2 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9dc8041070 con 0x55a9780b31b0 2024-05-27T07:32:31.401 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:32:31.406+0000 7f9dc17fa640 1 -- 172.21.15.52:0/536728758 <== mon.2 v2:172.21.15.105:3300/0 2 ==== osd_map(116..116 src has 1..116) ==== 9379+0+0 (secure 0 0 0) 0x7f9dc803d430 con 0x55a9780b31b0 2024-05-27T07:32:31.401 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:32:31.406+0000 7f9de6fba740 1 -- 172.21.15.52:0/536728758 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f9dd40b1e10 con 0x55a9780b31b0 2024-05-27T07:32:31.404 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:32:31.411+0000 7f9dc17fa640 1 -- 172.21.15.52:0/536728758 <== mon.2 v2:172.21.15.105:3300/0 3 ==== mon_get_version_reply(handle=1 version=101) ==== 24+0+0 (secure 0 0 0) 0x7f9dc804f040 con 0x55a9780b31b0 2024-05-27T07:32:31.404 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:32:31.411+0000 7f9de6fba740 10 client.26429 fetch_fsmap learned FSMap version 101 2024-05-27T07:32:31.404 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:32:31.411+0000 7f9de6fba740 1 -- 172.21.15.52:0/536728758 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,fsmap=101,monmap=4+}) -- 0x55a9784e9220 con 0x55a9780b31b0 2024-05-27T07:32:31.406 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:32:31.413+0000 7f9dc17fa640 1 -- 172.21.15.52:0/536728758 <== mon.2 v2:172.21.15.105:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f9dd40b1e10 con 0x55a9780b31b0 2024-05-27T07:32:31.406 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:32:31.413+0000 7f9dc17fa640 1 -- 172.21.15.52:0/536728758 <== mon.2 v2:172.21.15.105:3300/0 5 ==== fsmap(e 101) ==== 4203+0+0 (secure 0 0 0) 0x7f9dc805da10 con 0x55a9780b31b0 2024-05-27T07:32:31.406 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:32:31.413+0000 7f9de6fba740 10 client.26429 fetch_fsmap finished waiting for FSMap version 101 2024-05-27T07:32:31.406 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:32:31.413+0000 7f9de6fba740 10 client.26429 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-05-27T07:32:31.407 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:32:31.413+0000 7f9de6fba740 20 client.26429 populate_metadata read hostname 'smithi052' 2024-05-27T07:32:31.407 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:32:31.413+0000 7f9de6fba740 1 --2- 172.21.15.52:0/536728758 >> [v2:172.21.15.105:6838/500348151,v1:172.21.15.105:6839/500348151] conn(0x55a9780471c0 0x55a9784ebab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:32:31.407 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:32:31.413+0000 7f9de6fba740 4 client.26429 mds_command: new command op to 24527 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-05-27T07:32:31.407 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:32:31.413+0000 7f9de6fba740 1 -- 172.21.15.52:0/536728758 --> [v2:172.21.15.105:6838/500348151,v1:172.21.15.105:6839/500348151] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55a97833c5f0 con 0x55a9780471c0 2024-05-27T07:32:31.407 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:32:31.414+0000 7f9ddde9c640 1 --2- 172.21.15.52:0/536728758 >> [v2:172.21.15.105:6838/500348151,v1:172.21.15.105:6839/500348151] conn(0x55a9780471c0 0x55a9784ebab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:32:31.408 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:32:31.415+0000 7f9ddde9c640 1 --2- 172.21.15.52:0/536728758 >> [v2:172.21.15.105:6838/500348151,v1:172.21.15.105:6839/500348151] conn(0x55a9780471c0 0x55a9784ebab0 crc :-1 s=READY pgs=130 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=c3add50f04499c8b server_cookie=c337e5598888711b in_seq=0 out_seq=0 2024-05-27T07:32:31.408 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:32:31.415+0000 7f9dc17fa640 10 client.26429 ms_handle_connect on v2:172.21.15.105:6838/500348151 2024-05-27T07:32:32.482 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:32:32 smithi052 ceph-mon[31955]: pgmap v225: 129 pgs: 129 active+clean; 595 MiB data, 3.1 GiB used, 1.0 TiB / 1.0 TiB avail; 426 KiB/s rd, 3.9 MiB/s wr, 959 op/s 2024-05-27T07:32:32.605 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:32:32 smithi173 ceph-mon[37000]: pgmap v225: 129 pgs: 129 active+clean; 595 MiB data, 3.1 GiB used, 1.0 TiB / 1.0 TiB avail; 426 KiB/s rd, 3.9 MiB/s wr, 959 op/s 2024-05-27T07:32:32.693 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:32:32 smithi105 ceph-mon[35977]: pgmap v225: 129 pgs: 129 active+clean; 595 MiB data, 3.1 GiB used, 1.0 TiB / 1.0 TiB avail; 426 KiB/s rd, 3.9 MiB/s wr, 959 op/s 2024-05-27T07:32:34.605 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:32:34 smithi173 ceph-mon[37000]: pgmap v226: 129 pgs: 129 active+clean; 606 MiB data, 3.1 GiB used, 1.0 TiB / 1.0 TiB avail; 287 KiB/s rd, 4.5 MiB/s wr, 560 op/s 2024-05-27T07:32:34.693 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:32:34 smithi105 ceph-mon[35977]: pgmap v226: 129 pgs: 129 active+clean; 606 MiB data, 3.1 GiB used, 1.0 TiB / 1.0 TiB avail; 287 KiB/s rd, 4.5 MiB/s wr, 560 op/s 2024-05-27T07:32:34.732 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:32:34 smithi052 ceph-mon[31955]: pgmap v226: 129 pgs: 129 active+clean; 606 MiB data, 3.1 GiB used, 1.0 TiB / 1.0 TiB avail; 287 KiB/s rd, 4.5 MiB/s wr, 560 op/s 2024-05-27T07:32:35.221 INFO:tasks.workunit.client.0.smithi052.stdout: 151 684296 195 471046 201 415622 604 2024-05-27T07:32:35.826 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:32:35.831+0000 7f9dc17fa640 1 -- 172.21.15.52:0/536728758 <== mds.0 v2:172.21.15.105:6838/500348151 1 ==== command_reply(tid 0: 0 ) ==== 8+0+25 (crc 0 0 0) 0x55a97833c5f0 con 0x55a9780471c0 2024-05-27T07:32:35.826 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:32:35.831+0000 7f9dc17fa640 10 client.26429 handle_command_reply: tid=0 2024-05-27T07:32:35.826 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:32:35.831+0000 7f9de6fba740 1 client.26429 shutdown 2024-05-27T07:32:35.826 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:32:35.832+0000 7f9de6fba740 1 -- 172.21.15.52:0/536728758 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55a9780b31b0 msgr2=0x55a97848e990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:32:35.826 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:32:35.832+0000 7f9de6fba740 1 --2- 172.21.15.52:0/536728758 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55a9780b31b0 0x55a97848e990 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7f9dc802a990 tx=0x7f9dc8030ec0 comp rx=0 tx=0).stop 2024-05-27T07:32:35.826 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:32:35.832+0000 7f9de6fba740 1 -- 172.21.15.52:0/536728758 shutdown_connections 2024-05-27T07:32:35.826 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:32:35.832+0000 7f9de6fba740 1 --2- 172.21.15.52:0/536728758 >> [v2:172.21.15.105:6838/500348151,v1:172.21.15.105:6839/500348151] conn(0x55a9780471c0 0x55a9784ebab0 crc :-1 s=READY pgs=130 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:32:35.826 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:32:35.832+0000 7f9de6fba740 1 --2- 172.21.15.52:0/536728758 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55a9780b31b0 0x55a97848e990 unknown :-1 s=CLOSED pgs=204 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:32:35.827 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:32:35.832+0000 7f9de6fba740 1 --2- 172.21.15.52:0/536728758 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55a9780924d0 0x55a97848e450 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:32:35.827 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:32:35.832+0000 7f9de6fba740 1 --2- 172.21.15.52:0/536728758 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55a97808a2f0 0x55a978493a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:32:35.827 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:32:35.832+0000 7f9dc17fa640 0 client.26429 ms_handle_reset on v2:172.21.15.105:6838/500348151 2024-05-27T07:32:35.827 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:32:35.832+0000 7f9de6fba740 1 -- 172.21.15.52:0/536728758 >> 172.21.15.52:0/536728758 conn(0x55a9780a7300 msgr2=0x55a97845d810 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:32:35.827 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:32:35.832+0000 7f9de6fba740 1 -- 172.21.15.52:0/536728758 shutdown_connections 2024-05-27T07:32:35.827 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:32:35.832+0000 7f9de6fba740 1 -- 172.21.15.52:0/536728758 wait complete. 2024-05-27T07:32:35.827 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:32:35.832+0000 7f9de6fba740 20 client.26429 trim_cache size 0 max 16384 2024-05-27T07:32:35.827 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stdout:{ 2024-05-27T07:32:35.827 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stdout: "return_code": 0 2024-05-27T07:32:35.827 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stdout:} 2024-05-27T07:32:35.828 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:32:35.833+0000 7f9de0100640 1 -- 172.21.15.52:0/3198512478 >> [v2:172.21.15.52:6800/1822318384,v1:172.21.15.52:6801/1822318384] conn(0x7f9dc40b2a30 msgr2=0x7f9dc40b4f40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:32:35.828 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:32:35.833+0000 7f9de0100640 1 --2- 172.21.15.52:0/3198512478 >> [v2:172.21.15.52:6800/1822318384,v1:172.21.15.52:6801/1822318384] conn(0x7f9dc40b2a30 0x7f9dc40b4f40 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7f9dc8030e20 tx=0x7f9dc803e000 comp rx=0 tx=0).stop 2024-05-27T07:32:35.828 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:32:35.833+0000 7f9de0100640 1 -- 172.21.15.52:0/3198512478 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f9dd8111de0 msgr2=0x7f9dd81a5610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:32:35.828 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:32:35.833+0000 7f9de0100640 1 --2- 172.21.15.52:0/3198512478 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f9dd8111de0 0x7f9dd81a5610 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7f9dd400b4a0 tx=0x7f9dd400b960 comp rx=0 tx=0).stop 2024-05-27T07:32:35.829 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:32:35.833+0000 7f9de0100640 1 -- 172.21.15.52:0/3198512478 shutdown_connections 2024-05-27T07:32:35.829 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:32:35.833+0000 7f9de0100640 1 --2- 172.21.15.52:0/3198512478 >> [v2:172.21.15.52:6800/1822318384,v1:172.21.15.52:6801/1822318384] conn(0x7f9dc40b2a30 0x7f9dc40b4f40 unknown :-1 s=CLOSED pgs=157 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:32:35.829 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:32:35.833+0000 7f9de0100640 1 --2- 172.21.15.52:0/3198512478 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f9dd8112900 0x7f9dd81a98d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:32:35.829 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:32:35.833+0000 7f9de0100640 1 --2- 172.21.15.52:0/3198512478 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f9dd8111de0 0x7f9dd81a5610 unknown :-1 s=CLOSED pgs=202 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:32:35.829 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:32:35.833+0000 7f9de0100640 1 --2- 172.21.15.52:0/3198512478 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f9dd810a490 0x7f9dd81a50b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:32:35.829 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:32:35.833+0000 7f9de0100640 1 -- 172.21.15.52:0/3198512478 >> 172.21.15.52:0/3198512478 conn(0x7f9dd806e8f0 msgr2=0x7f9dd810ba90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:32:35.829 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:32:35.833+0000 7f9de0100640 1 -- 172.21.15.52:0/3198512478 shutdown_connections 2024-05-27T07:32:35.829 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:32:35.833+0000 7f9de0100640 1 -- 172.21.15.52:0/3198512478 wait complete. 2024-05-27T07:32:36.605 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:32:36 smithi173 ceph-mon[37000]: pgmap v227: 129 pgs: 129 active+clean; 625 MiB data, 3.2 GiB used, 1.0 TiB / 1.0 TiB avail; 310 KiB/s rd, 5.6 MiB/s wr, 1.03k op/s 2024-05-27T07:32:36.693 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:32:36 smithi105 ceph-mon[35977]: pgmap v227: 129 pgs: 129 active+clean; 625 MiB data, 3.2 GiB used, 1.0 TiB / 1.0 TiB avail; 310 KiB/s rd, 5.6 MiB/s wr, 1.03k op/s 2024-05-27T07:32:36.733 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:32:36 smithi052 ceph-mon[31955]: pgmap v227: 129 pgs: 129 active+clean; 625 MiB data, 3.2 GiB used, 1.0 TiB / 1.0 TiB avail; 310 KiB/s rd, 5.6 MiB/s wr, 1.03k op/s 2024-05-27T07:32:38.587 DEBUG:teuthology.orchestra.run.smithi052:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:38f9d900c83c63799fbdbe61acc9a11b0d3554a6 shell --fsid 73df2636-1bf8-11ef-bc9b-c7b262605968 -- ceph tell mds.1:0 scrub status 2024-05-27T07:32:38.605 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:32:38 smithi173 ceph-mon[37000]: pgmap v228: 129 pgs: 129 active+clean; 626 MiB data, 3.2 GiB used, 1.0 TiB / 1.0 TiB avail; 234 KiB/s rd, 5.4 MiB/s wr, 1.19k op/s 2024-05-27T07:32:38.693 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:32:38 smithi105 ceph-mon[35977]: pgmap v228: 129 pgs: 129 active+clean; 626 MiB data, 3.2 GiB used, 1.0 TiB / 1.0 TiB avail; 234 KiB/s rd, 5.4 MiB/s wr, 1.19k op/s 2024-05-27T07:32:38.732 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:32:38 smithi052 ceph-mon[31955]: pgmap v228: 129 pgs: 129 active+clean; 626 MiB data, 3.2 GiB used, 1.0 TiB / 1.0 TiB avail; 234 KiB/s rd, 5.4 MiB/s wr, 1.19k op/s 2024-05-27T07:32:40.605 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:32:40 smithi173 ceph-mon[37000]: pgmap v229: 129 pgs: 129 active+clean; 616 MiB data, 3.2 GiB used, 1.0 TiB / 1.0 TiB avail; 225 KiB/s rd, 5.6 MiB/s wr, 1.29k op/s 2024-05-27T07:32:40.693 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:32:40 smithi105 ceph-mon[35977]: pgmap v229: 129 pgs: 129 active+clean; 616 MiB data, 3.2 GiB used, 1.0 TiB / 1.0 TiB avail; 225 KiB/s rd, 5.6 MiB/s wr, 1.29k op/s 2024-05-27T07:32:40.734 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:32:40 smithi052 ceph-mon[31955]: pgmap v229: 129 pgs: 129 active+clean; 616 MiB data, 3.2 GiB used, 1.0 TiB / 1.0 TiB avail; 225 KiB/s rd, 5.6 MiB/s wr, 1.29k op/s 2024-05-27T07:32:42.605 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:32:42 smithi173 ceph-mon[37000]: pgmap v230: 129 pgs: 129 active+clean; 609 MiB data, 3.2 GiB used, 1.0 TiB / 1.0 TiB avail; 366 KiB/s rd, 6.3 MiB/s wr, 1.32k op/s 2024-05-27T07:32:42.693 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:32:42 smithi105 ceph-mon[35977]: pgmap v230: 129 pgs: 129 active+clean; 609 MiB data, 3.2 GiB used, 1.0 TiB / 1.0 TiB avail; 366 KiB/s rd, 6.3 MiB/s wr, 1.32k op/s 2024-05-27T07:32:42.732 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:32:42 smithi052 ceph-mon[31955]: pgmap v230: 129 pgs: 129 active+clean; 609 MiB data, 3.2 GiB used, 1.0 TiB / 1.0 TiB avail; 366 KiB/s rd, 6.3 MiB/s wr, 1.32k op/s 2024-05-27T07:32:42.845 INFO:teuthology.orchestra.run.smithi052.stderr:Inferring config /var/lib/ceph/73df2636-1bf8-11ef-bc9b-c7b262605968/mon.a/config 2024-05-27T07:32:44.483 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:32:44 smithi052 ceph-mon[31955]: pgmap v231: 129 pgs: 129 active+clean; 612 MiB data, 3.2 GiB used, 1.0 TiB / 1.0 TiB avail; 263 KiB/s rd, 4.3 MiB/s wr, 1.16k op/s 2024-05-27T07:32:44.605 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:32:44 smithi173 ceph-mon[37000]: pgmap v231: 129 pgs: 129 active+clean; 612 MiB data, 3.2 GiB used, 1.0 TiB / 1.0 TiB avail; 263 KiB/s rd, 4.3 MiB/s wr, 1.16k op/s 2024-05-27T07:32:44.693 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:32:44 smithi105 ceph-mon[35977]: pgmap v231: 129 pgs: 129 active+clean; 612 MiB data, 3.2 GiB used, 1.0 TiB / 1.0 TiB avail; 263 KiB/s rd, 4.3 MiB/s wr, 1.16k op/s 2024-05-27T07:32:45.221 INFO:tasks.workunit.client.0.smithi052.stdout: 157 1504939 463 1035299 334 899444 1242 2024-05-27T07:32:46.458 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:32:46 smithi052 ceph-mon[31955]: pgmap v232: 129 pgs: 129 active+clean; 616 MiB data, 3.2 GiB used, 1.0 TiB / 1.0 TiB avail; 299 KiB/s rd, 3.6 MiB/s wr, 1.09k op/s 2024-05-27T07:32:46.458 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:32:46.459+0000 7fe234efd700 1 -- 172.21.15.52:0/116825083 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fe23010b270 msgr2=0x7fe23011d3e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:32:46.458 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:32:46.459+0000 7fe234efd700 1 --2- 172.21.15.52:0/116825083 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fe23010b270 0x7fe23011d3e0 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7fe22800b0a0 tx=0x7fe22800b3b0 comp rx=0 tx=0).stop 2024-05-27T07:32:46.458 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:32:46.460+0000 7fe234efd700 1 -- 172.21.15.52:0/116825083 shutdown_connections 2024-05-27T07:32:46.458 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:32:46.460+0000 7fe234efd700 1 --2- 172.21.15.52:0/116825083 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fe23011d920 0x7fe23011fd40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:32:46.458 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:32:46.460+0000 7fe234efd700 1 --2- 172.21.15.52:0/116825083 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fe23010a920 0x7fe23010ad30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:32:46.458 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:32:46.460+0000 7fe234efd700 1 --2- 172.21.15.52:0/116825083 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fe23010b270 0x7fe23011d3e0 unknown :-1 s=CLOSED pgs=205 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:32:46.459 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:32:46.460+0000 7fe234efd700 1 -- 172.21.15.52:0/116825083 >> 172.21.15.52:0/116825083 conn(0x7fe23006f4c0 msgr2=0x7fe23006f900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:32:46.459 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:32:46.460+0000 7fe234efd700 1 -- 172.21.15.52:0/116825083 shutdown_connections 2024-05-27T07:32:46.459 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:32:46.460+0000 7fe234efd700 1 -- 172.21.15.52:0/116825083 wait complete. 2024-05-27T07:32:46.459 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:32:46.461+0000 7fe234efd700 1 Processor -- start 2024-05-27T07:32:46.460 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:32:46.465+0000 7fe234efd700 1 -- start start 2024-05-27T07:32:46.460 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:32:46.465+0000 7fe234efd700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fe23010a920 0x7fe2301af7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:32:46.461 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:32:46.465+0000 7fe234efd700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fe23010b270 0x7fe2301afd10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:32:46.461 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:32:46.465+0000 7fe234efd700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fe23011d920 0x7fe2301a99f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:32:46.461 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:32:46.465+0000 7fe234efd700 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x7fe230123330 con 0x7fe23010a920 2024-05-27T07:32:46.461 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:32:46.465+0000 7fe234efd700 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7fe2301231b0 con 0x7fe23011d920 2024-05-27T07:32:46.461 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:32:46.465+0000 7fe234efd700 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7fe2301234b0 con 0x7fe23010b270 2024-05-27T07:32:46.462 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:32:46.466+0000 7fe22dd9b700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fe23010b270 0x7fe2301afd10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:32:46.462 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:32:46.466+0000 7fe22dd9b700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fe23010b270 0x7fe2301afd10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.173:3300/0 says I am v2:172.21.15.52:42726/0 (socket says 172.21.15.52:42726) 2024-05-27T07:32:46.462 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:32:46.466+0000 7fe22dd9b700 1 -- 172.21.15.52:0/1427696319 learned_addr learned my addr 172.21.15.52:0/1427696319 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-05-27T07:32:46.462 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:32:46.466+0000 7fe22ed9d700 1 --2- 172.21.15.52:0/1427696319 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fe23011d920 0x7fe2301a99f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:32:46.462 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:32:46.467+0000 7fe22e59c700 1 --2- 172.21.15.52:0/1427696319 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fe23010a920 0x7fe2301af7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:32:46.462 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:32:46.467+0000 7fe22dd9b700 1 -- 172.21.15.52:0/1427696319 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fe23011d920 msgr2=0x7fe2301a99f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:32:46.462 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:32:46.467+0000 7fe22dd9b700 1 --2- 172.21.15.52:0/1427696319 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fe23011d920 0x7fe2301a99f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:32:46.463 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:32:46.467+0000 7fe22dd9b700 1 -- 172.21.15.52:0/1427696319 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fe23010a920 msgr2=0x7fe2301af7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:32:46.463 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:32:46.467+0000 7fe22dd9b700 1 --2- 172.21.15.52:0/1427696319 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fe23010a920 0x7fe2301af7d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:32:46.463 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:32:46.467+0000 7fe22dd9b700 1 -- 172.21.15.52:0/1427696319 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe2301aa030 con 0x7fe23010b270 2024-05-27T07:32:46.463 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:32:46.467+0000 7fe22e59c700 1 --2- 172.21.15.52:0/1427696319 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fe23010a920 0x7fe2301af7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-27T07:32:46.463 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:32:46.469+0000 7fe22dd9b700 1 --2- 172.21.15.52:0/1427696319 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fe23010b270 0x7fe2301afd10 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7fe2280095c0 tx=0x7fe228013940 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:32:46.470 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:32:46.474+0000 7fe21f7fe700 1 -- 172.21.15.52:0/1427696319 <== mon.1 v2:172.21.15.173:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe22801f030 con 0x7fe23010b270 2024-05-27T07:32:46.470 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:32:46.474+0000 7fe21f7fe700 1 -- 172.21.15.52:0/1427696319 <== mon.1 v2:172.21.15.173:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fe228013e20 con 0x7fe23010b270 2024-05-27T07:32:46.470 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:32:46.474+0000 7fe21f7fe700 1 -- 172.21.15.52:0/1427696319 <== mon.1 v2:172.21.15.173:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe228005860 con 0x7fe23010b270 2024-05-27T07:32:46.470 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:32:46.474+0000 7fe234efd700 1 -- 172.21.15.52:0/1427696319 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe2301aa2c0 con 0x7fe23010b270 2024-05-27T07:32:46.470 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:32:46.474+0000 7fe234efd700 1 -- 172.21.15.52:0/1427696319 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe2301aa750 con 0x7fe23010b270 2024-05-27T07:32:46.491 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:32:46.496+0000 7fe243912b80 1 Processor -- start 2024-05-27T07:32:46.494 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:32:46.497+0000 7fe21f7fe700 1 -- 172.21.15.52:0/1427696319 <== mon.1 v2:172.21.15.173:3300/0 4 ==== mgrmap(e 26) ==== 149783+0+0 (secure 0 0 0) 0x7fe228022030 con 0x7fe23010b270 2024-05-27T07:32:46.495 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:32:46.499+0000 7fe21f7fe700 1 --2- 172.21.15.52:0/1427696319 >> [v2:172.21.15.52:6800/1822318384,v1:172.21.15.52:6801/1822318384] conn(0x7fe2180b2bf0 0x7fe2180b5100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:32:46.495 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:32:46.499+0000 7fe21f7fe700 1 -- 172.21.15.52:0/1427696319 <== mon.1 v2:172.21.15.173:3300/0 5 ==== osd_map(116..116 src has 1..116) ==== 9379+0+0 (secure 0 0 0) 0x7fe22800f020 con 0x7fe23010b270 2024-05-27T07:32:46.495 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:32:46.500+0000 7fe22ed9d700 1 --2- 172.21.15.52:0/1427696319 >> [v2:172.21.15.52:6800/1822318384,v1:172.21.15.52:6801/1822318384] conn(0x7fe2180b2bf0 0x7fe2180b5100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:32:46.495 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:32:46.500+0000 7fe243912b80 1 -- start start 2024-05-27T07:32:46.496 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:32:46.501+0000 7fe243912b80 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x563104fdee40 0x563105163b30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:32:46.497 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:32:46.501+0000 7fe22ed9d700 1 --2- 172.21.15.52:0/1427696319 >> [v2:172.21.15.52:6800/1822318384,v1:172.21.15.52:6801/1822318384] conn(0x7fe2180b2bf0 0x7fe2180b5100 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7fe230112b90 tx=0x7fe22400a2d0 comp rx=0 tx=0).ready entity=mgr.15333 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:32:46.497 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:32:46.501+0000 7fe243912b80 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x563105164070 0x56310515dbe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:32:46.497 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:32:46.501+0000 7fe243912b80 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x56310515e120 0x56310515e530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:32:46.497 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:32:46.501+0000 7fe243912b80 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x56310508bd70 con 0x563104fdee40 2024-05-27T07:32:46.497 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:32:46.502+0000 7fe243912b80 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x56310507b910 con 0x563105164070 2024-05-27T07:32:46.497 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:32:46.502+0000 7fe243912b80 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x56310507f950 con 0x56310515e120 2024-05-27T07:32:46.497 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:32:46.502+0000 7fe22e59c700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x563104fdee40 0x563105163b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:32:46.497 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:32:46.502+0000 7fe22e59c700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x563104fdee40 0x563105163b30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.52:55838/0 (socket says 172.21.15.52:55838) 2024-05-27T07:32:46.497 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:32:46.502+0000 7fe22e59c700 1 -- 172.21.15.52:0/3947039082 learned_addr learned my addr 172.21.15.52:0/3947039082 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-05-27T07:32:46.497 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:32:46.502+0000 7fe22dd9b700 1 --2- 172.21.15.52:0/3947039082 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x563105164070 0x56310515dbe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:32:46.499 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:32:46.503+0000 7fe22e59c700 1 -- 172.21.15.52:0/3947039082 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x56310515e120 msgr2=0x56310515e530 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-05-27T07:32:46.500 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:32:46.503+0000 7fe22e59c700 1 --2- 172.21.15.52:0/3947039082 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x56310515e120 0x56310515e530 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:32:46.500 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:32:46.503+0000 7fe22e59c700 1 -- 172.21.15.52:0/3947039082 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x563105164070 msgr2=0x56310515dbe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:32:46.500 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:32:46.503+0000 7fe22e59c700 1 --2- 172.21.15.52:0/3947039082 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x563105164070 0x56310515dbe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:32:46.500 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:32:46.503+0000 7fe22e59c700 1 -- 172.21.15.52:0/3947039082 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x563105024fb0 con 0x563104fdee40 2024-05-27T07:32:46.500 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:32:46.503+0000 7fe22dd9b700 1 --2- 172.21.15.52:0/3947039082 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x563105164070 0x56310515dbe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-27T07:32:46.500 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:32:46.504+0000 7fe22e59c700 1 --2- 172.21.15.52:0/3947039082 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x563104fdee40 0x563105163b30 secure :-1 s=READY pgs=680 cs=0 l=1 rev1=1 crypto rx=0x7fe22000bd00 tx=0x7fe22000f640 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6b40105848e377b1 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:32:46.500 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:32:46.504+0000 7fe21cff9700 1 -- 172.21.15.52:0/3947039082 <== mon.0 v2:172.21.15.52:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe220015070 con 0x563104fdee40 2024-05-27T07:32:46.500 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:32:46.505+0000 7fe21cff9700 1 -- 172.21.15.52:0/3947039082 <== mon.0 v2:172.21.15.52:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fe22000be50 con 0x563104fdee40 2024-05-27T07:32:46.500 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:32:46.505+0000 7fe21cff9700 1 -- 172.21.15.52:0/3947039082 <== mon.0 v2:172.21.15.52:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe220011710 con 0x563104fdee40 2024-05-27T07:32:46.502 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:32:46.507+0000 7fe243912b80 1 -- 172.21.15.52:0/3947039082 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x563104fdee40 msgr2=0x563105163b30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:32:46.502 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:32:46.507+0000 7fe243912b80 1 --2- 172.21.15.52:0/3947039082 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x563104fdee40 0x563105163b30 secure :-1 s=READY pgs=680 cs=0 l=1 rev1=1 crypto rx=0x7fe22000bd00 tx=0x7fe22000f640 comp rx=0 tx=0).stop 2024-05-27T07:32:46.514 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:32:46.519+0000 7fe243912b80 1 -- 172.21.15.52:0/3947039082 shutdown_connections 2024-05-27T07:32:46.514 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:32:46.519+0000 7fe243912b80 1 --2- 172.21.15.52:0/3947039082 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x56310515e120 0x56310515e530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:32:46.514 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:32:46.519+0000 7fe243912b80 1 --2- 172.21.15.52:0/3947039082 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x563104fdee40 0x563105163b30 unknown :-1 s=CLOSED pgs=680 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:32:46.514 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:32:46.519+0000 7fe243912b80 1 --2- 172.21.15.52:0/3947039082 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x563105164070 0x56310515dbe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:32:46.514 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:32:46.519+0000 7fe243912b80 1 -- 172.21.15.52:0/3947039082 >> 172.21.15.52:0/3947039082 conn(0x563104fe1770 msgr2=0x563105158d90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:32:46.521 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:32:46.527+0000 7fe243912b80 1 -- 172.21.15.52:0/3947039082 shutdown_connections 2024-05-27T07:32:46.524 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:32:46.528+0000 7fe243912b80 1 -- 172.21.15.52:0/3947039082 wait complete. 2024-05-27T07:32:46.524 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:32:46.528+0000 7fe243912b80 1 -- start start 2024-05-27T07:32:46.524 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:32:46.529+0000 7fe243912b80 1 Processor -- start 2024-05-27T07:32:46.529 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:32:46.530+0000 7fe243912b80 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x563104fdee40 0x563105167880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:32:46.529 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:32:46.530+0000 7fe243912b80 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x56310515e120 0x563105167dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:32:46.529 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:32:46.530+0000 7fe243912b80 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x563105164070 0x563105168300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:32:46.529 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:32:46.530+0000 7fe22dd9b700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x56310515e120 0x563105167dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:32:46.529 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:32:46.530+0000 7fe22dd9b700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x56310515e120 0x563105167dc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.52:56488/0 (socket says 172.21.15.52:56488) 2024-05-27T07:32:46.529 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:32:46.530+0000 7fe22dd9b700 1 -- 172.21.15.52:0/533814802 learned_addr learned my addr 172.21.15.52:0/533814802 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-05-27T07:32:46.529 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:32:46.530+0000 7fe243912b80 1 -- 172.21.15.52:0/533814802 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x563104f81fa0 con 0x563105164070 2024-05-27T07:32:46.529 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:32:46.530+0000 7fe243912b80 1 -- 172.21.15.52:0/533814802 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x563104fe3da0 con 0x56310515e120 2024-05-27T07:32:46.530 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:32:46.530+0000 7fe243912b80 1 -- 172.21.15.52:0/533814802 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x56310516ab60 con 0x563104fdee40 2024-05-27T07:32:46.530 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:32:46.531+0000 7fe243912b80 10 client.0 _ll_register_callbacks cb 0x5631050125b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T07:32:46.530 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:32:46.531+0000 7fe22ed9d700 1 --2- 172.21.15.52:0/533814802 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x563105164070 0x563105168300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:32:46.530 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:32:46.531+0000 7fe22e59c700 1 --2- 172.21.15.52:0/533814802 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x563104fdee40 0x563105167880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:32:46.530 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:32:46.533+0000 7fe22ed9d700 1 -- 172.21.15.52:0/533814802 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x563104fdee40 msgr2=0x563105167880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:32:46.530 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:32:46.533+0000 7fe22ed9d700 1 --2- 172.21.15.52:0/533814802 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x563104fdee40 0x563105167880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:32:46.530 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:32:46.533+0000 7fe22ed9d700 1 -- 172.21.15.52:0/533814802 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x56310515e120 msgr2=0x563105167dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:32:46.530 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:32:46.533+0000 7fe22ed9d700 1 --2- 172.21.15.52:0/533814802 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x56310515e120 0x563105167dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:32:46.530 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:32:46.533+0000 7fe22ed9d700 1 -- 172.21.15.52:0/533814802 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({osdmap=0}) -- 0x563104fcb8d0 con 0x563105164070 2024-05-27T07:32:46.530 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:32:46.533+0000 7fe22ed9d700 1 --2- 172.21.15.52:0/533814802 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x563105164070 0x563105168300 secure :-1 s=READY pgs=681 cs=0 l=1 rev1=1 crypto rx=0x7fe22400f6a0 tx=0x7fe2240047c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=db9ace8a32740549 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:32:46.530 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:32:46.533+0000 7fe215ffb700 10 client.0 ms_handle_connect on v2:172.21.15.52:3300/0 2024-05-27T07:32:46.534 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:32:46.538+0000 7fe215ffb700 1 -- 172.21.15.52:0/533814802 <== mon.0 v2:172.21.15.52:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe224016070 con 0x563105164070 2024-05-27T07:32:46.534 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:32:46.538+0000 7fe243912b80 1 -- 172.21.15.52:0/533814802 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fe22000be50 con 0x563105164070 2024-05-27T07:32:46.534 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:32:46.538+0000 7fe215ffb700 1 -- 172.21.15.52:0/533814802 <== mon.0 v2:172.21.15.52:3300/0 2 ==== osd_map(116..116 src has 1..116) ==== 9379+0+0 (secure 0 0 0) 0x7fe22401b420 con 0x563105164070 2024-05-27T07:32:46.535 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:32:46.540+0000 7fe215ffb700 1 -- 172.21.15.52:0/533814802 <== mon.0 v2:172.21.15.52:3300/0 3 ==== mon_get_version_reply(handle=1 version=101) ==== 24+0+0 (secure 0 0 0) 0x7fe22401b620 con 0x563105164070 2024-05-27T07:32:46.537 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:32:46.541+0000 7fe243912b80 10 client.16779 fetch_fsmap learned FSMap version 101 2024-05-27T07:32:46.537 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:32:46.541+0000 7fe243912b80 1 -- 172.21.15.52:0/533814802 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({config=0+,fsmap=101,monmap=4+}) -- 0x563105035760 con 0x563105164070 2024-05-27T07:32:46.537 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:32:46.542+0000 7fe215ffb700 1 -- 172.21.15.52:0/533814802 <== mon.0 v2:172.21.15.52:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fe22000be50 con 0x563105164070 2024-05-27T07:32:46.537 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:32:46.542+0000 7fe215ffb700 1 -- 172.21.15.52:0/533814802 <== mon.0 v2:172.21.15.52:3300/0 5 ==== fsmap(e 101) ==== 4203+0+0 (secure 0 0 0) 0x7fe22402f990 con 0x563105164070 2024-05-27T07:32:46.537 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:32:46.542+0000 7fe243912b80 10 client.16779 fetch_fsmap finished waiting for FSMap version 101 2024-05-27T07:32:46.537 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:32:46.542+0000 7fe243912b80 10 client.16779 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-05-27T07:32:46.537 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:32:46.542+0000 7fe243912b80 20 client.16779 populate_metadata read hostname 'smithi052' 2024-05-27T07:32:46.538 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:32:46.542+0000 7fe243912b80 1 --2- 172.21.15.52:0/533814802 >> [v2:172.21.15.105:6838/500348151,v1:172.21.15.105:6839/500348151] conn(0x563105177ec0 0x56310517a330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:32:46.538 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:32:46.542+0000 7fe243912b80 4 client.16779 mds_command: new command op to 24527 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-27T07:32:46.538 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:32:46.542+0000 7fe243912b80 1 -- 172.21.15.52:0/533814802 --> [v2:172.21.15.105:6838/500348151,v1:172.21.15.105:6839/500348151] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x56310508dcc0 con 0x563105177ec0 2024-05-27T07:32:46.540 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:32:46.545+0000 7fe22e59c700 1 --2- 172.21.15.52:0/533814802 >> [v2:172.21.15.105:6838/500348151,v1:172.21.15.105:6839/500348151] conn(0x563105177ec0 0x56310517a330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:32:46.545 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:32:46.551+0000 7fe22e59c700 1 --2- 172.21.15.52:0/533814802 >> [v2:172.21.15.105:6838/500348151,v1:172.21.15.105:6839/500348151] conn(0x563105177ec0 0x56310517a330 crc :-1 s=READY pgs=131 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=a6061d611ecfd887 server_cookie=c28c78a41e5a2374 in_seq=0 out_seq=0 2024-05-27T07:32:46.545 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:32:46.551+0000 7fe215ffb700 10 client.16779 ms_handle_connect on v2:172.21.15.105:6838/500348151 2024-05-27T07:32:46.549 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:32:46.554+0000 7fe215ffb700 1 -- 172.21.15.52:0/533814802 <== mds.0 v2:172.21.15.105:6838/500348151 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24041 (crc 0 0 0) 0x56310508dcc0 con 0x563105177ec0 2024-05-27T07:32:46.549 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:32:46.554+0000 7fe215ffb700 10 client.16779 handle_command_reply: tid=0 2024-05-27T07:32:46.549 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:32:46.554+0000 7fe243912b80 1 client.16779 shutdown 2024-05-27T07:32:46.554 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:32:46.558+0000 7fe243912b80 1 -- 172.21.15.52:0/533814802 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x563105164070 msgr2=0x563105168300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:32:46.554 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:32:46.558+0000 7fe243912b80 1 --2- 172.21.15.52:0/533814802 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x563105164070 0x563105168300 secure :-1 s=READY pgs=681 cs=0 l=1 rev1=1 crypto rx=0x7fe22400f6a0 tx=0x7fe2240047c0 comp rx=0 tx=0).stop 2024-05-27T07:32:46.554 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:32:46.559+0000 7fe243912b80 1 -- 172.21.15.52:0/533814802 shutdown_connections 2024-05-27T07:32:46.554 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:32:46.559+0000 7fe243912b80 1 --2- 172.21.15.52:0/533814802 >> [v2:172.21.15.105:6838/500348151,v1:172.21.15.105:6839/500348151] conn(0x563105177ec0 0x56310517a330 crc :-1 s=READY pgs=131 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:32:46.554 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:32:46.559+0000 7fe243912b80 1 --2- 172.21.15.52:0/533814802 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x563105164070 0x563105168300 unknown :-1 s=CLOSED pgs=681 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:32:46.554 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:32:46.559+0000 7fe243912b80 1 --2- 172.21.15.52:0/533814802 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x563104fdee40 0x563105167880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:32:46.554 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:32:46.559+0000 7fe243912b80 1 --2- 172.21.15.52:0/533814802 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x56310515e120 0x563105167dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:32:46.554 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:32:46.559+0000 7fe243912b80 1 -- 172.21.15.52:0/533814802 >> 172.21.15.52:0/533814802 conn(0x563104fe1770 msgr2=0x56310515cbd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:32:46.557 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:32:46.563+0000 7fe215ffb700 0 client.16779 ms_handle_reset on v2:172.21.15.105:6838/500348151 2024-05-27T07:32:46.561 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:32:46.564+0000 7fe243912b80 1 -- 172.21.15.52:0/533814802 shutdown_connections 2024-05-27T07:32:46.561 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:32:46.564+0000 7fe243912b80 1 -- 172.21.15.52:0/533814802 wait complete. 2024-05-27T07:32:46.562 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:32:46.568+0000 7fe243912b80 20 client.16779 trim_cache size 0 max 16384 2024-05-27T07:32:46.605 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:32:46 smithi173 ceph-mon[37000]: pgmap v232: 129 pgs: 129 active+clean; 616 MiB data, 3.2 GiB used, 1.0 TiB / 1.0 TiB avail; 299 KiB/s rd, 3.6 MiB/s wr, 1.09k op/s 2024-05-27T07:32:46.618 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:32:46.621+0000 7fe243912b80 1 Processor -- start 2024-05-27T07:32:46.618 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:32:46.621+0000 7fe243912b80 1 -- start start 2024-05-27T07:32:46.618 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:32:46.621+0000 7fe243912b80 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x563104fdee40 0x56310515dbe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:32:46.618 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:32:46.621+0000 7fe243912b80 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x56310515e120 0x563105166500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:32:46.618 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:32:46.621+0000 7fe243912b80 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x563105164070 0x563105173c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:32:46.618 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:32:46.621+0000 7fe22e59c700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x563104fdee40 0x56310515dbe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:32:46.618 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:32:46.621+0000 7fe22e59c700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x563104fdee40 0x56310515dbe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.173:3300/0 says I am v2:172.21.15.52:42734/0 (socket says 172.21.15.52:42734) 2024-05-27T07:32:46.618 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:32:46.622+0000 7fe22dd9b700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x56310515e120 0x563105166500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:32:46.618 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:32:46.622+0000 7fe22dd9b700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x56310515e120 0x563105166500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.52:56496/0 (socket says 172.21.15.52:56496) 2024-05-27T07:32:46.618 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:32:46.622+0000 7fe22e59c700 1 -- 172.21.15.52:0/4131878671 learned_addr learned my addr 172.21.15.52:0/4131878671 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-05-27T07:32:46.619 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:32:46.622+0000 7fe243912b80 1 -- 172.21.15.52:0/4131878671 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x7fe21000f290 con 0x563105164070 2024-05-27T07:32:46.619 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:32:46.622+0000 7fe243912b80 1 -- 172.21.15.52:0/4131878671 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7fe21000d690 con 0x56310515e120 2024-05-27T07:32:46.619 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:32:46.622+0000 7fe243912b80 1 -- 172.21.15.52:0/4131878671 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7fe21000ebc0 con 0x563104fdee40 2024-05-27T07:32:46.619 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:32:46.622+0000 7fe22dd9b700 1 -- 172.21.15.52:0/4131878671 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x563104fdee40 msgr2=0x56310515dbe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:32:46.619 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:32:46.622+0000 7fe22dd9b700 1 --2- 172.21.15.52:0/4131878671 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x563104fdee40 0x56310515dbe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:32:46.619 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:32:46.622+0000 7fe22dd9b700 1 -- 172.21.15.52:0/4131878671 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x563105164070 msgr2=0x563105173c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:32:46.619 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:32:46.622+0000 7fe22dd9b700 1 --2- 172.21.15.52:0/4131878671 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x563105164070 0x563105173c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:32:46.619 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:32:46.622+0000 7fe22dd9b700 1 -- 172.21.15.52:0/4131878671 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x563105165e10 con 0x56310515e120 2024-05-27T07:32:46.619 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:32:46.624+0000 7fe22dd9b700 1 --2- 172.21.15.52:0/4131878671 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x56310515e120 0x563105166500 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7fe22800b830 tx=0x7fe2280aa7a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=59b19ee7d1aa09aa server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:32:46.622 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:32:46.627+0000 7fe217fff700 1 -- 172.21.15.52:0/4131878671 <== mon.2 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe2280aa460 con 0x56310515e120 2024-05-27T07:32:46.622 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:32:46.627+0000 7fe217fff700 1 -- 172.21.15.52:0/4131878671 <== mon.2 v2:172.21.15.105:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fe2280ab050 con 0x56310515e120 2024-05-27T07:32:46.622 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:32:46.628+0000 7fe217fff700 1 -- 172.21.15.52:0/4131878671 <== mon.2 v2:172.21.15.105:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe2280cd0e0 con 0x56310515e120 2024-05-27T07:32:46.624 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:32:46.629+0000 7fe243912b80 1 -- 172.21.15.52:0/4131878671 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x56310515e120 msgr2=0x563105166500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:32:46.624 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:32:46.629+0000 7fe243912b80 1 --2- 172.21.15.52:0/4131878671 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x56310515e120 0x563105166500 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7fe22800b830 tx=0x7fe2280aa7a0 comp rx=0 tx=0).stop 2024-05-27T07:32:46.625 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:32:46.630+0000 7fe243912b80 1 -- 172.21.15.52:0/4131878671 shutdown_connections 2024-05-27T07:32:46.625 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:32:46.630+0000 7fe243912b80 1 --2- 172.21.15.52:0/4131878671 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x563105164070 0x563105173c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:32:46.625 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:32:46.630+0000 7fe243912b80 1 --2- 172.21.15.52:0/4131878671 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x563104fdee40 0x56310515dbe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:32:46.625 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:32:46.630+0000 7fe243912b80 1 --2- 172.21.15.52:0/4131878671 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x56310515e120 0x563105166500 unknown :-1 s=CLOSED pgs=206 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:32:46.625 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:32:46.630+0000 7fe243912b80 1 -- 172.21.15.52:0/4131878671 >> 172.21.15.52:0/4131878671 conn(0x563104fe1770 msgr2=0x563105161490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:32:46.629 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:32:46.634+0000 7fe243912b80 1 -- 172.21.15.52:0/4131878671 shutdown_connections 2024-05-27T07:32:46.629 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:32:46.634+0000 7fe243912b80 1 -- 172.21.15.52:0/4131878671 wait complete. 2024-05-27T07:32:46.630 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:32:46.635+0000 7fe243912b80 1 -- start start 2024-05-27T07:32:46.630 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:32:46.635+0000 7fe243912b80 1 Processor -- start 2024-05-27T07:32:46.630 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:32:46.635+0000 7fe243912b80 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x563104fdee40 0x563105170020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:32:46.631 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:32:46.635+0000 7fe243912b80 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x56310515e120 0x563105170560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:32:46.631 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:32:46.636+0000 7fe243912b80 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x563105164070 0x56310517ef40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:32:46.631 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:32:46.636+0000 7fe243912b80 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x7fe21000e430 con 0x56310515e120 2024-05-27T07:32:46.631 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:32:46.636+0000 7fe243912b80 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7fe21000dd60 con 0x563105164070 2024-05-27T07:32:46.631 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:32:46.636+0000 7fe243912b80 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x56310517a540 con 0x563104fdee40 2024-05-27T07:32:46.631 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:32:46.636+0000 7fe243912b80 10 client.0 _ll_register_callbacks cb 0x5631050125b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T07:32:46.632 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:32:46.636+0000 7fe22ed9d700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x563105164070 0x56310517ef40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:32:46.632 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:32:46.636+0000 7fe22ed9d700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x563105164070 0x56310517ef40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.52:56508/0 (socket says 172.21.15.52:56508) 2024-05-27T07:32:46.632 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:32:46.636+0000 7fe22ed9d700 1 -- 172.21.15.52:0/820458023 learned_addr learned my addr 172.21.15.52:0/820458023 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-05-27T07:32:46.632 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:32:46.636+0000 7fe22e59c700 1 --2- 172.21.15.52:0/820458023 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x563104fdee40 0x563105170020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:32:46.632 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:32:46.637+0000 7fe22e59c700 1 -- 172.21.15.52:0/820458023 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x563105164070 msgr2=0x56310517ef40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:32:46.632 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:32:46.637+0000 7fe22e59c700 1 --2- 172.21.15.52:0/820458023 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x563105164070 0x56310517ef40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:32:46.632 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:32:46.637+0000 7fe22e59c700 1 -- 172.21.15.52:0/820458023 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x56310515e120 msgr2=0x563105170560 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-05-27T07:32:46.633 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:32:46.637+0000 7fe22e59c700 1 --2- 172.21.15.52:0/820458023 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x56310515e120 0x563105170560 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:32:46.633 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:32:46.637+0000 7fe22e59c700 1 -- 172.21.15.52:0/820458023 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56310517f710 con 0x563104fdee40 2024-05-27T07:32:46.633 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:32:46.638+0000 7fe22e59c700 1 --2- 172.21.15.52:0/820458023 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x563104fdee40 0x563105170020 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7fe22000fa40 tx=0x7fe220005ef0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=d06f122c166b2b02 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:32:46.633 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:32:46.638+0000 7fe215ffb700 10 client.0 ms_handle_connect on v2:172.21.15.173:3300/0 2024-05-27T07:32:46.633 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:32:46.638+0000 7fe215ffb700 1 -- 172.21.15.52:0/820458023 <== mon.1 v2:172.21.15.173:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe220015040 con 0x563104fdee40 2024-05-27T07:32:46.634 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:32:46.639+0000 7fe243912b80 1 -- 172.21.15.52:0/820458023 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fe2280ab050 con 0x563104fdee40 2024-05-27T07:32:46.634 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:32:46.639+0000 7fe215ffb700 1 -- 172.21.15.52:0/820458023 <== mon.1 v2:172.21.15.173:3300/0 2 ==== osd_map(116..116 src has 1..116) ==== 9379+0+0 (secure 0 0 0) 0x7fe220004240 con 0x563104fdee40 2024-05-27T07:32:46.634 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:32:46.639+0000 7fe215ffb700 1 -- 172.21.15.52:0/820458023 <== mon.1 v2:172.21.15.173:3300/0 3 ==== mon_get_version_reply(handle=1 version=101) ==== 24+0+0 (secure 0 0 0) 0x7fe220004440 con 0x563104fdee40 2024-05-27T07:32:46.634 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:32:46.639+0000 7fe243912b80 10 client.26284 fetch_fsmap learned FSMap version 101 2024-05-27T07:32:46.634 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:32:46.639+0000 7fe243912b80 1 -- 172.21.15.52:0/820458023 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({config=0+,fsmap=101,monmap=4+}) -- 0x5631051e2450 con 0x563104fdee40 2024-05-27T07:32:46.636 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:32:46.641+0000 7fe215ffb700 1 -- 172.21.15.52:0/820458023 <== mon.1 v2:172.21.15.173:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fe2280ab050 con 0x563104fdee40 2024-05-27T07:32:46.636 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:32:46.642+0000 7fe215ffb700 1 -- 172.21.15.52:0/820458023 <== mon.1 v2:172.21.15.173:3300/0 5 ==== fsmap(e 101) ==== 4203+0+0 (secure 0 0 0) 0x7fe220003960 con 0x563104fdee40 2024-05-27T07:32:46.638 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:32:46.643+0000 7fe243912b80 10 client.26284 fetch_fsmap finished waiting for FSMap version 101 2024-05-27T07:32:46.639 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:32:46.643+0000 7fe243912b80 10 client.26284 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-05-27T07:32:46.639 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:32:46.643+0000 7fe243912b80 20 client.26284 populate_metadata read hostname 'smithi052' 2024-05-27T07:32:46.639 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:32:46.643+0000 7fe243912b80 1 --2- 172.21.15.52:0/820458023 >> [v2:172.21.15.105:6838/500348151,v1:172.21.15.105:6839/500348151] conn(0x5631051e2a30 0x5631051e4e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:32:46.639 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:32:46.644+0000 7fe243912b80 4 client.26284 mds_command: new command op to 24527 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-05-27T07:32:46.639 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:32:46.644+0000 7fe243912b80 1 -- 172.21.15.52:0/820458023 --> [v2:172.21.15.105:6838/500348151,v1:172.21.15.105:6839/500348151] -- command(tid 0: {"prefix": "scrub status"}) -- 0x5631050874a0 con 0x5631051e2a30 2024-05-27T07:32:46.639 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:32:46.644+0000 7fe22dd9b700 1 --2- 172.21.15.52:0/820458023 >> [v2:172.21.15.105:6838/500348151,v1:172.21.15.105:6839/500348151] conn(0x5631051e2a30 0x5631051e4e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:32:46.645 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:32:46.651+0000 7fe22dd9b700 1 --2- 172.21.15.52:0/820458023 >> [v2:172.21.15.105:6838/500348151,v1:172.21.15.105:6839/500348151] conn(0x5631051e2a30 0x5631051e4e50 crc :-1 s=READY pgs=132 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=fb60d89238d565d6 server_cookie=3df3594a6f09b1e1 in_seq=0 out_seq=0 2024-05-27T07:32:46.645 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:32:46.651+0000 7fe215ffb700 10 client.26284 ms_handle_connect on v2:172.21.15.105:6838/500348151 2024-05-27T07:32:46.650 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:32:46.656+0000 7fe215ffb700 1 -- 172.21.15.52:0/820458023 <== mds.0 v2:172.21.15.105:6838/500348151 1 ==== command_reply(tid 0: 0 ) ==== 8+0+269 (crc 0 0 0) 0x5631050874a0 con 0x5631051e2a30 2024-05-27T07:32:46.650 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:32:46.656+0000 7fe215ffb700 10 client.26284 handle_command_reply: tid=0 2024-05-27T07:32:46.650 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:32:46.656+0000 7fe243912b80 1 client.26284 shutdown 2024-05-27T07:32:46.663 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:32:46.669+0000 7fe243912b80 1 -- 172.21.15.52:0/820458023 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x563104fdee40 msgr2=0x563105170020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:32:46.664 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:32:46.669+0000 7fe243912b80 1 --2- 172.21.15.52:0/820458023 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x563104fdee40 0x563105170020 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7fe22000fa40 tx=0x7fe220005ef0 comp rx=0 tx=0).stop 2024-05-27T07:32:46.666 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:32:46.671+0000 7fe243912b80 1 -- 172.21.15.52:0/820458023 shutdown_connections 2024-05-27T07:32:46.666 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:32:46.671+0000 7fe243912b80 1 --2- 172.21.15.52:0/820458023 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x563105164070 0x56310517ef40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:32:46.666 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:32:46.671+0000 7fe243912b80 1 --2- 172.21.15.52:0/820458023 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x563104fdee40 0x563105170020 unknown :-1 s=CLOSED pgs=159 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:32:46.666 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:32:46.671+0000 7fe243912b80 1 --2- 172.21.15.52:0/820458023 >> [v2:172.21.15.105:6838/500348151,v1:172.21.15.105:6839/500348151] conn(0x5631051e2a30 0x5631051e4e50 crc :-1 s=READY pgs=132 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:32:46.666 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:32:46.671+0000 7fe243912b80 1 --2- 172.21.15.52:0/820458023 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x56310515e120 0x563105170560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:32:46.666 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:32:46.671+0000 7fe243912b80 1 -- 172.21.15.52:0/820458023 >> 172.21.15.52:0/820458023 conn(0x563104fe1770 msgr2=0x563105167820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:32:46.667 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:32:46.672+0000 7fe215ffb700 0 client.26284 ms_handle_reset on v2:172.21.15.105:6838/500348151 2024-05-27T07:32:46.669 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:32:46.675+0000 7fe243912b80 1 -- 172.21.15.52:0/820458023 shutdown_connections 2024-05-27T07:32:46.669 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:32:46.675+0000 7fe243912b80 1 -- 172.21.15.52:0/820458023 wait complete. 2024-05-27T07:32:46.670 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:32:46.676+0000 7fe243912b80 20 client.26284 trim_cache size 0 max 16384 2024-05-27T07:32:46.675 INFO:teuthology.orchestra.run.smithi052.stdout:{ 2024-05-27T07:32:46.675 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "scrub active (11439 inodes in the stack)", 2024-05-27T07:32:46.675 INFO:teuthology.orchestra.run.smithi052.stdout: "scrubs": { 2024-05-27T07:32:46.675 INFO:teuthology.orchestra.run.smithi052.stdout: "36f21956-1be9-436b-b6db-8bc2e943eb70": { 2024-05-27T07:32:46.675 INFO:teuthology.orchestra.run.smithi052.stdout: "path": "/", 2024-05-27T07:32:46.675 INFO:teuthology.orchestra.run.smithi052.stdout: "tag": "36f21956-1be9-436b-b6db-8bc2e943eb70", 2024-05-27T07:32:46.675 INFO:teuthology.orchestra.run.smithi052.stdout: "options": "recursive,force" 2024-05-27T07:32:46.675 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-05-27T07:32:46.675 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-05-27T07:32:46.676 INFO:teuthology.orchestra.run.smithi052.stdout:} 2024-05-27T07:32:46.676 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:32:46.678+0000 7fe234efd700 1 -- 172.21.15.52:0/1427696319 >> [v2:172.21.15.52:6800/1822318384,v1:172.21.15.52:6801/1822318384] conn(0x7fe2180b2bf0 msgr2=0x7fe2180b5100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:32:46.676 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:32:46.678+0000 7fe234efd700 1 --2- 172.21.15.52:0/1427696319 >> [v2:172.21.15.52:6800/1822318384,v1:172.21.15.52:6801/1822318384] conn(0x7fe2180b2bf0 0x7fe2180b5100 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7fe230112b90 tx=0x7fe22400a2d0 comp rx=0 tx=0).stop 2024-05-27T07:32:46.676 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:32:46.678+0000 7fe234efd700 1 -- 172.21.15.52:0/1427696319 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fe23010b270 msgr2=0x7fe2301afd10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:32:46.676 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:32:46.678+0000 7fe234efd700 1 --2- 172.21.15.52:0/1427696319 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fe23010b270 0x7fe2301afd10 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7fe2280095c0 tx=0x7fe228013940 comp rx=0 tx=0).stop 2024-05-27T07:32:46.676 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:32:46.679+0000 7fe234efd700 1 -- 172.21.15.52:0/1427696319 shutdown_connections 2024-05-27T07:32:46.676 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:32:46.679+0000 7fe234efd700 1 --2- 172.21.15.52:0/1427696319 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fe23011d920 0x7fe2301a99f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:32:46.676 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:32:46.679+0000 7fe234efd700 1 --2- 172.21.15.52:0/1427696319 >> [v2:172.21.15.52:6800/1822318384,v1:172.21.15.52:6801/1822318384] conn(0x7fe2180b2bf0 0x7fe2180b5100 unknown :-1 s=CLOSED pgs=158 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:32:46.676 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:32:46.679+0000 7fe234efd700 1 --2- 172.21.15.52:0/1427696319 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fe23010a920 0x7fe2301af7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:32:46.676 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:32:46.679+0000 7fe234efd700 1 --2- 172.21.15.52:0/1427696319 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fe23010b270 0x7fe2301afd10 unknown :-1 s=CLOSED pgs=158 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:32:46.677 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:32:46.679+0000 7fe234efd700 1 -- 172.21.15.52:0/1427696319 >> 172.21.15.52:0/1427696319 conn(0x7fe23006f4c0 msgr2=0x7fe23011b3d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:32:46.677 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:32:46.679+0000 7fe234efd700 1 -- 172.21.15.52:0/1427696319 shutdown_connections 2024-05-27T07:32:46.677 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:32:46.679+0000 7fe234efd700 1 -- 172.21.15.52:0/1427696319 wait complete. 2024-05-27T07:32:46.693 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:32:46 smithi105 ceph-mon[35977]: pgmap v232: 129 pgs: 129 active+clean; 616 MiB data, 3.2 GiB used, 1.0 TiB / 1.0 TiB avail; 299 KiB/s rd, 3.6 MiB/s wr, 1.09k op/s 2024-05-27T07:32:48.396 INFO:tasks.cephfs.filesystem:scrub status for tag:36f21956-1be9-436b-b6db-8bc2e943eb70 - {'path': '/', 'tag': '36f21956-1be9-436b-b6db-8bc2e943eb70', 'options': 'recursive,force'} 2024-05-27T07:32:48.605 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:32:48 smithi173 ceph-mon[37000]: pgmap v233: 129 pgs: 129 active+clean; 618 MiB data, 3.2 GiB used, 1.0 TiB / 1.0 TiB avail; 354 KiB/s rd, 2.1 MiB/s wr, 620 op/s 2024-05-27T07:32:48.693 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:32:48 smithi105 ceph-mon[35977]: pgmap v233: 129 pgs: 129 active+clean; 618 MiB data, 3.2 GiB used, 1.0 TiB / 1.0 TiB avail; 354 KiB/s rd, 2.1 MiB/s wr, 620 op/s 2024-05-27T07:32:48.734 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:32:48 smithi052 ceph-mon[31955]: pgmap v233: 129 pgs: 129 active+clean; 618 MiB data, 3.2 GiB used, 1.0 TiB / 1.0 TiB avail; 354 KiB/s rd, 2.1 MiB/s wr, 620 op/s 2024-05-27T07:32:49.782 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:32:49.787+0000 7f6786d57640 1 -- 172.21.15.52:0/2532985060 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f678011d240 msgr2=0x7f678011f660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:32:49.782 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:32:49.787+0000 7f6786d57640 1 --2- 172.21.15.52:0/2532985060 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f678011d240 0x7f678011f660 secure :-1 s=READY pgs=682 cs=0 l=1 rev1=1 crypto rx=0x7f6778009ca0 tx=0x7f677802a980 comp rx=0 tx=0).stop 2024-05-27T07:32:49.782 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:32:49.787+0000 7f6786d57640 1 -- 172.21.15.52:0/2532985060 shutdown_connections 2024-05-27T07:32:49.782 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:32:49.787+0000 7f6786d57640 1 --2- 172.21.15.52:0/2532985060 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f678011d240 0x7f678011f660 unknown :-1 s=CLOSED pgs=682 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:32:49.782 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:32:49.787+0000 7f6786d57640 1 --2- 172.21.15.52:0/2532985060 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f678011a8e0 0x7f678011cd00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:32:49.782 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:32:49.787+0000 7f6786d57640 1 --2- 172.21.15.52:0/2532985060 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f678010a0e0 0x7f678010a4f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:32:49.782 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:32:49.787+0000 7f6786d57640 1 -- 172.21.15.52:0/2532985060 >> 172.21.15.52:0/2532985060 conn(0x7f678006f770 msgr2=0x7f678006fba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:32:49.783 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:32:49.789+0000 7f6786d57640 1 -- 172.21.15.52:0/2532985060 shutdown_connections 2024-05-27T07:32:49.784 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:32:49.789+0000 7f6786d57640 1 -- 172.21.15.52:0/2532985060 wait complete. 2024-05-27T07:32:49.784 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:32:49.790+0000 7f6786d57640 1 Processor -- start 2024-05-27T07:32:49.785 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:32:49.790+0000 7f6786d57640 1 -- start start 2024-05-27T07:32:49.785 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:32:49.790+0000 7f6786d57640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f678010a0e0 0x7f67801a9140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:32:49.785 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:32:49.790+0000 7f6786d57640 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f678011a8e0 0x7f67801a96a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:32:49.785 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:32:49.790+0000 7f6786d57640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f67801ad8a0 0x7f67801adcb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:32:49.785 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:32:49.790+0000 7f6786d57640 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x7f6780122b20 con 0x7f678011a8e0 2024-05-27T07:32:49.785 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:32:49.790+0000 7f6786d57640 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f67801229a0 con 0x7f67801ad8a0 2024-05-27T07:32:49.786 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:32:49.790+0000 7f6786d57640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f6780122ca0 con 0x7f678010a0e0 2024-05-27T07:32:49.786 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:32:49.791+0000 7f67852f4640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f67801ad8a0 0x7f67801adcb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:32:49.786 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:32:49.791+0000 7f67852f4640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f67801ad8a0 0x7f67801adcb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.52:56532/0 (socket says 172.21.15.52:56532) 2024-05-27T07:32:49.786 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:32:49.791+0000 7f67852f4640 1 -- 172.21.15.52:0/1713758666 learned_addr learned my addr 172.21.15.52:0/1713758666 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-05-27T07:32:49.786 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:32:49.792+0000 7f67852f4640 1 -- 172.21.15.52:0/1713758666 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f678010a0e0 msgr2=0x7f67801a9140 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-05-27T07:32:49.787 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:32:49.792+0000 7f67852f4640 1 --2- 172.21.15.52:0/1713758666 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f678010a0e0 0x7f67801a9140 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:32:49.787 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:32:49.792+0000 7f67852f4640 1 -- 172.21.15.52:0/1713758666 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f678011a8e0 msgr2=0x7f67801a96a0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-05-27T07:32:49.787 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:32:49.792+0000 7f677ffff640 1 --2- 172.21.15.52:0/1713758666 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f678011a8e0 0x7f67801a96a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:32:49.787 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:32:49.792+0000 7f67852f4640 1 --2- 172.21.15.52:0/1713758666 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f678011a8e0 0x7f67801a96a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:32:49.787 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:32:49.792+0000 7f67852f4640 1 -- 172.21.15.52:0/1713758666 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f67801ae480 con 0x7f67801ad8a0 2024-05-27T07:32:49.787 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:32:49.792+0000 7f67852f4640 1 --2- 172.21.15.52:0/1713758666 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f67801ad8a0 0x7f67801adcb0 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7f677802fe10 tx=0x7f6778031c30 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:32:49.800 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:32:49.806+0000 7f677dffb640 1 -- 172.21.15.52:0/1713758666 <== mon.2 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6778034040 con 0x7f67801ad8a0 2024-05-27T07:32:49.801 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:32:49.806+0000 7f677dffb640 1 -- 172.21.15.52:0/1713758666 <== mon.2 v2:172.21.15.105:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f6778039ce0 con 0x7f67801ad8a0 2024-05-27T07:32:49.801 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:32:49.806+0000 7f677dffb640 1 -- 172.21.15.52:0/1713758666 <== mon.2 v2:172.21.15.105:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6778044ab0 con 0x7f67801ad8a0 2024-05-27T07:32:49.801 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:32:49.806+0000 7f6786d57640 1 -- 172.21.15.52:0/1713758666 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f67801b5b50 con 0x7f67801ad8a0 2024-05-27T07:32:49.801 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:32:49.806+0000 7f6786d57640 1 -- 172.21.15.52:0/1713758666 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f67801b5f00 con 0x7f67801ad8a0 2024-05-27T07:32:49.807 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:32:49.812+0000 7f678dc08740 1 Processor -- start 2024-05-27T07:32:49.807 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:32:49.813+0000 7f678dc08740 1 -- start start 2024-05-27T07:32:49.808 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:32:49.813+0000 7f678dc08740 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x5557991971b0 0x555799538590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:32:49.808 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:32:49.813+0000 7f678dc08740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55579916e2f0 0x555799538ad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:32:49.808 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:32:49.813+0000 7f678dc08740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5557991764d0 0x555799539010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:32:49.808 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:32:49.813+0000 7f678dc08740 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x5557994aac30 con 0x5557991971b0 2024-05-27T07:32:49.808 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:32:49.813+0000 7f678dc08740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x55579949a7d0 con 0x5557991764d0 2024-05-27T07:32:49.808 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:32:49.813+0000 7f678dc08740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x55579949e810 con 0x55579916e2f0 2024-05-27T07:32:49.808 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:32:49.813+0000 7f677dffb640 1 -- 172.21.15.52:0/1713758666 <== mon.2 v2:172.21.15.105:3300/0 4 ==== mgrmap(e 26) ==== 149783+0+0 (secure 0 0 0) 0x7f6778037020 con 0x7f67801ad8a0 2024-05-27T07:32:49.809 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:32:49.814+0000 7f677ffff640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5557991764d0 0x555799539010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:32:49.809 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:32:49.814+0000 7f6784af3640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55579916e2f0 0x555799538ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:32:49.809 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:32:49.814+0000 7f6784af3640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55579916e2f0 0x555799538ad0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.173:3300/0 says I am v2:172.21.15.52:42766/0 (socket says 172.21.15.52:42766) 2024-05-27T07:32:49.809 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:32:49.814+0000 7f6784af3640 1 -- 172.21.15.52:0/2246354344 learned_addr learned my addr 172.21.15.52:0/2246354344 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-05-27T07:32:49.809 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:32:49.814+0000 7f6784af3640 1 -- 172.21.15.52:0/2246354344 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5557991764d0 msgr2=0x555799539010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:32:49.809 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:32:49.814+0000 7f6784af3640 1 --2- 172.21.15.52:0/2246354344 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5557991764d0 0x555799539010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:32:49.809 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:32:49.814+0000 7f6784af3640 1 -- 172.21.15.52:0/2246354344 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x5557991971b0 msgr2=0x555799538590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:32:49.809 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:32:49.814+0000 7f6784af3640 1 --2- 172.21.15.52:0/2246354344 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x5557991971b0 0x555799538590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:32:49.809 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:32:49.814+0000 7f6784af3640 1 -- 172.21.15.52:0/2246354344 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5557991af210 con 0x55579916e2f0 2024-05-27T07:32:49.813 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:32:49.815+0000 7f677dffb640 1 --2- 172.21.15.52:0/1713758666 >> [v2:172.21.15.52:6800/1822318384,v1:172.21.15.52:6801/1822318384] conn(0x7f67680b2b70 0x7f67680b5080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:32:49.813 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:32:49.815+0000 7f6784af3640 1 --2- 172.21.15.52:0/2246354344 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55579916e2f0 0x555799538ad0 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7f677400ba80 tx=0x7f677400bf40 comp rx=0 tx=0).ready entity=mon.1 client_cookie=9b7b68c7984d0cd8 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:32:49.819 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:32:49.820+0000 7f677ffff640 1 --2- 172.21.15.52:0/1713758666 >> [v2:172.21.15.52:6800/1822318384,v1:172.21.15.52:6801/1822318384] conn(0x7f67680b2b70 0x7f67680b5080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:32:49.820 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:32:49.825+0000 7f677dffb640 1 -- 172.21.15.52:0/1713758666 <== mon.2 v2:172.21.15.105:3300/0 5 ==== osd_map(116..116 src has 1..116) ==== 9379+0+0 (secure 0 0 0) 0x7f67781038a0 con 0x7f67801ad8a0 2024-05-27T07:32:49.820 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:32:49.826+0000 7f677ffff640 1 --2- 172.21.15.52:0/1713758666 >> [v2:172.21.15.52:6800/1822318384,v1:172.21.15.52:6801/1822318384] conn(0x7f67680b2b70 0x7f67680b5080 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x555799535700 tx=0x7f6770008040 comp rx=0 tx=0).ready entity=mgr.15333 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:32:49.830 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:32:49.829+0000 7f6766ffd640 1 -- 172.21.15.52:0/2246354344 <== mon.1 v2:172.21.15.173:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6774016070 con 0x55579916e2f0 2024-05-27T07:32:49.830 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:32:49.836+0000 7f6766ffd640 1 -- 172.21.15.52:0/2246354344 <== mon.1 v2:172.21.15.173:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f67740054d0 con 0x55579916e2f0 2024-05-27T07:32:49.836 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:32:49.842+0000 7f678dc08740 1 -- 172.21.15.52:0/2246354344 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55579916e2f0 msgr2=0x555799538ad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:32:49.837 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:32:49.842+0000 7f678dc08740 1 --2- 172.21.15.52:0/2246354344 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55579916e2f0 0x555799538ad0 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7f677400ba80 tx=0x7f677400bf40 comp rx=0 tx=0).stop 2024-05-27T07:32:49.837 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:32:49.843+0000 7f678dc08740 1 -- 172.21.15.52:0/2246354344 shutdown_connections 2024-05-27T07:32:49.837 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:32:49.843+0000 7f678dc08740 1 --2- 172.21.15.52:0/2246354344 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5557991764d0 0x555799539010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:32:49.837 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:32:49.843+0000 7f678dc08740 1 --2- 172.21.15.52:0/2246354344 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55579916e2f0 0x555799538ad0 unknown :-1 s=CLOSED pgs=160 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:32:49.837 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:32:49.843+0000 7f678dc08740 1 --2- 172.21.15.52:0/2246354344 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x5557991971b0 0x555799538590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:32:49.837 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:32:49.843+0000 7f678dc08740 1 -- 172.21.15.52:0/2246354344 >> 172.21.15.52:0/2246354344 conn(0x55579918b300 msgr2=0x555799534d40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:32:49.838 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:32:49.843+0000 7f678dc08740 1 -- 172.21.15.52:0/2246354344 shutdown_connections 2024-05-27T07:32:49.838 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:32:49.843+0000 7f678dc08740 1 -- 172.21.15.52:0/2246354344 wait complete. 2024-05-27T07:32:49.838 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:32:49.843+0000 7f678dc08740 1 -- start start 2024-05-27T07:32:49.838 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:32:49.844+0000 7f678dc08740 1 Processor -- start 2024-05-27T07:32:49.840 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:32:49.844+0000 7f678dc08740 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55579916e2f0 0x555799542b70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:32:49.840 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:32:49.844+0000 7f678dc08740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5557991764d0 0x5557995430b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:32:49.840 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:32:49.844+0000 7f678dc08740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5557991971b0 0x55579954ea90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:32:49.840 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:32:49.844+0000 7f678dc08740 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x555799180790 con 0x55579916e2f0 2024-05-27T07:32:49.840 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:32:49.844+0000 7f678dc08740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x555799456890 con 0x5557991764d0 2024-05-27T07:32:49.840 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:32:49.845+0000 7f678dc08740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x555799545710 con 0x5557991971b0 2024-05-27T07:32:49.840 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:32:49.845+0000 7f678dc08740 10 client.0 _ll_register_callbacks cb 0x5557991aea00 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T07:32:49.840 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:32:49.845+0000 7f677ffff640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5557991764d0 0x5557995430b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:32:49.841 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:32:49.845+0000 7f677ffff640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5557991764d0 0x5557995430b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.52:56556/0 (socket says 172.21.15.52:56556) 2024-05-27T07:32:49.841 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:32:49.845+0000 7f677ffff640 1 -- 172.21.15.52:0/2853362764 learned_addr learned my addr 172.21.15.52:0/2853362764 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-05-27T07:32:49.841 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:32:49.846+0000 7f677ffff640 1 -- 172.21.15.52:0/2853362764 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5557991971b0 msgr2=0x55579954ea90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:32:49.841 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:32:49.846+0000 7f677ffff640 1 --2- 172.21.15.52:0/2853362764 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5557991971b0 0x55579954ea90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:32:49.841 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:32:49.846+0000 7f677ffff640 1 -- 172.21.15.52:0/2853362764 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55579916e2f0 msgr2=0x555799542b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:32:49.841 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:32:49.846+0000 7f677ffff640 1 --2- 172.21.15.52:0/2853362764 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55579916e2f0 0x555799542b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:32:49.841 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:32:49.846+0000 7f677ffff640 1 -- 172.21.15.52:0/2853362764 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55579927b620 con 0x5557991764d0 2024-05-27T07:32:49.842 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:32:49.846+0000 7f677ffff640 1 --2- 172.21.15.52:0/2853362764 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5557991764d0 0x5557995430b0 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7f677000ff90 tx=0x7f6770006ad0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=18589ea5e2cd289c server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:32:49.849 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:32:49.855+0000 7f675ffff640 10 client.0 ms_handle_connect on v2:172.21.15.105:3300/0 2024-05-27T07:32:49.849 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:32:49.855+0000 7f675ffff640 1 -- 172.21.15.52:0/2853362764 <== mon.2 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6770023070 con 0x5557991764d0 2024-05-27T07:32:49.849 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:32:49.855+0000 7f678dc08740 1 -- 172.21.15.52:0/2853362764 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f67740054d0 con 0x5557991764d0 2024-05-27T07:32:49.850 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:32:49.855+0000 7f675ffff640 1 -- 172.21.15.52:0/2853362764 <== mon.2 v2:172.21.15.105:3300/0 2 ==== osd_map(116..116 src has 1..116) ==== 9379+0+0 (secure 0 0 0) 0x7f677001f5f0 con 0x5557991764d0 2024-05-27T07:32:49.850 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:32:49.856+0000 7f675ffff640 1 -- 172.21.15.52:0/2853362764 <== mon.2 v2:172.21.15.105:3300/0 3 ==== mon_get_version_reply(handle=1 version=101) ==== 24+0+0 (secure 0 0 0) 0x7f677002f050 con 0x5557991764d0 2024-05-27T07:32:49.850 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:32:49.856+0000 7f678dc08740 10 client.26456 fetch_fsmap learned FSMap version 101 2024-05-27T07:32:49.850 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:32:49.856+0000 7f678dc08740 1 -- 172.21.15.52:0/2853362764 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,fsmap=101,monmap=4+}) -- 0x5557995629d0 con 0x5557991764d0 2024-05-27T07:32:49.851 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:32:49.856+0000 7f675ffff640 1 -- 172.21.15.52:0/2853362764 <== mon.2 v2:172.21.15.105:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f67740054d0 con 0x5557991764d0 2024-05-27T07:32:49.852 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:32:49.858+0000 7f675ffff640 1 -- 172.21.15.52:0/2853362764 <== mon.2 v2:172.21.15.105:3300/0 5 ==== fsmap(e 101) ==== 4203+0+0 (secure 0 0 0) 0x7f677001e070 con 0x5557991764d0 2024-05-27T07:32:49.852 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:32:49.858+0000 7f678dc08740 10 client.26456 fetch_fsmap finished waiting for FSMap version 101 2024-05-27T07:32:49.853 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:32:49.858+0000 7f678dc08740 10 client.26456 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-05-27T07:32:49.853 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:32:49.858+0000 7f678dc08740 20 client.26456 populate_metadata read hostname 'smithi052' 2024-05-27T07:32:49.853 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:32:49.858+0000 7f678dc08740 1 --2- 172.21.15.52:0/2853362764 >> [v2:172.21.15.105:6838/500348151,v1:172.21.15.105:6839/500348151] conn(0x555799155d80 0x555799566510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:32:49.853 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:32:49.858+0000 7f678dc08740 4 client.26456 mds_command: new command op to 24527 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-27T07:32:49.853 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:32:49.858+0000 7f678dc08740 1 -- 172.21.15.52:0/2853362764 --> [v2:172.21.15.105:6838/500348151,v1:172.21.15.105:6839/500348151] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5557994acb80 con 0x555799155d80 2024-05-27T07:32:49.853 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:32:49.858+0000 7f6784af3640 1 --2- 172.21.15.52:0/2853362764 >> [v2:172.21.15.105:6838/500348151,v1:172.21.15.105:6839/500348151] conn(0x555799155d80 0x555799566510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:32:49.853 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:32:49.859+0000 7f6784af3640 1 --2- 172.21.15.52:0/2853362764 >> [v2:172.21.15.105:6838/500348151,v1:172.21.15.105:6839/500348151] conn(0x555799155d80 0x555799566510 crc :-1 s=READY pgs=133 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=1c5e23a95f766a3f server_cookie=570d53877b99db8f in_seq=0 out_seq=0 2024-05-27T07:32:49.854 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:32:49.859+0000 7f675ffff640 10 client.26456 ms_handle_connect on v2:172.21.15.105:6838/500348151 2024-05-27T07:32:49.858 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:32:49.864+0000 7f675ffff640 1 -- 172.21.15.52:0/2853362764 <== mds.0 v2:172.21.15.105:6838/500348151 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24041 (crc 0 0 0) 0x5557994acb80 con 0x555799155d80 2024-05-27T07:32:49.858 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:32:49.864+0000 7f675ffff640 10 client.26456 handle_command_reply: tid=0 2024-05-27T07:32:49.858 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:32:49.864+0000 7f678dc08740 1 client.26456 shutdown 2024-05-27T07:32:49.859 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:32:49.864+0000 7f678dc08740 1 -- 172.21.15.52:0/2853362764 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5557991764d0 msgr2=0x5557995430b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:32:49.859 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:32:49.864+0000 7f678dc08740 1 --2- 172.21.15.52:0/2853362764 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5557991764d0 0x5557995430b0 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7f677000ff90 tx=0x7f6770006ad0 comp rx=0 tx=0).stop 2024-05-27T07:32:49.859 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:32:49.865+0000 7f678dc08740 1 -- 172.21.15.52:0/2853362764 shutdown_connections 2024-05-27T07:32:49.859 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:32:49.865+0000 7f678dc08740 1 --2- 172.21.15.52:0/2853362764 >> [v2:172.21.15.105:6838/500348151,v1:172.21.15.105:6839/500348151] conn(0x555799155d80 0x555799566510 crc :-1 s=READY pgs=133 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:32:49.859 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:32:49.865+0000 7f678dc08740 1 --2- 172.21.15.52:0/2853362764 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5557991971b0 0x55579954ea90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:32:49.859 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:32:49.865+0000 7f678dc08740 1 --2- 172.21.15.52:0/2853362764 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5557991764d0 0x5557995430b0 unknown :-1 s=CLOSED pgs=208 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:32:49.859 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:32:49.865+0000 7f678dc08740 1 --2- 172.21.15.52:0/2853362764 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55579916e2f0 0x555799542b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:32:49.860 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:32:49.865+0000 7f678dc08740 1 -- 172.21.15.52:0/2853362764 >> 172.21.15.52:0/2853362764 conn(0x55579918b300 msgr2=0x55579953a2b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:32:49.860 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:32:49.865+0000 7f675ffff640 0 client.26456 ms_handle_reset on v2:172.21.15.105:6838/500348151 2024-05-27T07:32:49.868 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:32:49.874+0000 7f678dc08740 1 -- 172.21.15.52:0/2853362764 shutdown_connections 2024-05-27T07:32:49.868 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:32:49.874+0000 7f678dc08740 1 -- 172.21.15.52:0/2853362764 wait complete. 2024-05-27T07:32:49.868 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:32:49.874+0000 7f678dc08740 20 client.26456 trim_cache size 0 max 16384 2024-05-27T07:32:49.969 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:32:49.975+0000 7f678dc08740 1 Processor -- start 2024-05-27T07:32:49.970 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:32:49.975+0000 7f678dc08740 1 -- start start 2024-05-27T07:32:49.973 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:32:49.979+0000 7f678dc08740 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55579916e2f0 0x5557995393a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:32:49.974 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:32:49.979+0000 7f678dc08740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5557991764d0 0x55579953e140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:32:49.975 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:32:49.980+0000 7f6784af3640 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55579916e2f0 0x5557995393a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:32:49.975 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:32:49.980+0000 7f6784af3640 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55579916e2f0 0x5557995393a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.52:55894/0 (socket says 172.21.15.52:55894) 2024-05-27T07:32:49.975 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:32:49.980+0000 7f6784af3640 1 -- 172.21.15.52:0/2261420644 learned_addr learned my addr 172.21.15.52:0/2261420644 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-05-27T07:32:49.975 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:32:49.980+0000 7f677ffff640 1 --2- 172.21.15.52:0/2261420644 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5557991764d0 0x55579953e140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:32:49.975 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:32:49.980+0000 7f678dc08740 1 --2- 172.21.15.52:0/2261420644 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5557991971b0 0x55579953a8e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:32:49.975 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:32:49.980+0000 7f678dc08740 1 -- 172.21.15.52:0/2261420644 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x7f67801b7c50 con 0x55579916e2f0 2024-05-27T07:32:49.975 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:32:49.981+0000 7f67852f4640 1 --2- 172.21.15.52:0/2261420644 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5557991971b0 0x55579953a8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:32:49.976 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:32:49.981+0000 7f678dc08740 1 -- 172.21.15.52:0/2261420644 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f67801b6190 con 0x5557991764d0 2024-05-27T07:32:49.976 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:32:49.981+0000 7f678dc08740 1 -- 172.21.15.52:0/2261420644 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f67801b7580 con 0x5557991971b0 2024-05-27T07:32:49.976 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:32:49.981+0000 7f677ffff640 1 -- 172.21.15.52:0/2261420644 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5557991971b0 msgr2=0x55579953a8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:32:49.976 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:32:49.981+0000 7f677ffff640 1 --2- 172.21.15.52:0/2261420644 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5557991971b0 0x55579953a8e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:32:49.976 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:32:49.981+0000 7f677ffff640 1 -- 172.21.15.52:0/2261420644 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55579916e2f0 msgr2=0x5557995393a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:32:49.977 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:32:49.981+0000 7f677ffff640 1 --2- 172.21.15.52:0/2261420644 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55579916e2f0 0x5557995393a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:32:49.977 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:32:49.981+0000 7f677ffff640 1 -- 172.21.15.52:0/2261420644 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x555799543a30 con 0x5557991764d0 2024-05-27T07:32:49.977 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:32:49.982+0000 7f677ffff640 1 --2- 172.21.15.52:0/2261420644 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5557991764d0 0x55579953e140 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7f6770027f80 tx=0x7f6770004750 comp rx=0 tx=0).ready entity=mon.2 client_cookie=af916b87aa122cf2 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:32:49.980 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:32:49.986+0000 7f67667fc640 1 -- 172.21.15.52:0/2261420644 <== mon.2 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6770037070 con 0x5557991764d0 2024-05-27T07:32:49.981 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:32:49.986+0000 7f67667fc640 1 -- 172.21.15.52:0/2261420644 <== mon.2 v2:172.21.15.105:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f6770019070 con 0x5557991764d0 2024-05-27T07:32:49.981 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:32:49.986+0000 7f67667fc640 1 -- 172.21.15.52:0/2261420644 <== mon.2 v2:172.21.15.105:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6770023040 con 0x5557991764d0 2024-05-27T07:32:49.983 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:32:49.989+0000 7f678dc08740 1 -- 172.21.15.52:0/2261420644 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5557991764d0 msgr2=0x55579953e140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:32:49.983 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:32:49.989+0000 7f678dc08740 1 --2- 172.21.15.52:0/2261420644 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5557991764d0 0x55579953e140 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7f6770027f80 tx=0x7f6770004750 comp rx=0 tx=0).stop 2024-05-27T07:32:49.987 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:32:49.993+0000 7f678dc08740 1 -- 172.21.15.52:0/2261420644 shutdown_connections 2024-05-27T07:32:49.987 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:32:49.993+0000 7f678dc08740 1 --2- 172.21.15.52:0/2261420644 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5557991971b0 0x55579953a8e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:32:49.988 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:32:49.993+0000 7f678dc08740 1 --2- 172.21.15.52:0/2261420644 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5557991764d0 0x55579953e140 unknown :-1 s=CLOSED pgs=209 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:32:49.988 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:32:49.993+0000 7f678dc08740 1 --2- 172.21.15.52:0/2261420644 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55579916e2f0 0x5557995393a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:32:49.988 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:32:49.993+0000 7f678dc08740 1 -- 172.21.15.52:0/2261420644 >> 172.21.15.52:0/2261420644 conn(0x55579918b300 msgr2=0x55579953a1d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:32:50.002 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:32:50.007+0000 7f678dc08740 1 -- 172.21.15.52:0/2261420644 shutdown_connections 2024-05-27T07:32:50.002 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:32:50.008+0000 7f678dc08740 1 -- 172.21.15.52:0/2261420644 wait complete. 2024-05-27T07:32:50.003 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:32:50.009+0000 7f678dc08740 1 -- start start 2024-05-27T07:32:50.003 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:32:50.009+0000 7f678dc08740 1 Processor -- start 2024-05-27T07:32:50.005 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:32:50.010+0000 7f678dc08740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55579916e2f0 0x5557995459b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:32:50.005 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:32:50.010+0000 7f678dc08740 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x5557991764d0 0x555799545ef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:32:50.006 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:32:50.011+0000 7f677ffff640 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x5557991764d0 0x555799545ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:32:50.006 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:32:50.011+0000 7f6784af3640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55579916e2f0 0x5557995459b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:32:50.006 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:32:50.011+0000 7f6784af3640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55579916e2f0 0x5557995459b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.173:3300/0 says I am v2:172.21.15.52:42792/0 (socket says 172.21.15.52:42792) 2024-05-27T07:32:50.006 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:32:50.011+0000 7f6784af3640 1 -- 172.21.15.52:0/4224494927 learned_addr learned my addr 172.21.15.52:0/4224494927 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-05-27T07:32:50.006 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:32:50.011+0000 7f678dc08740 1 --2- 172.21.15.52:0/4224494927 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5557991971b0 0x555799546430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:32:50.006 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:32:50.011+0000 7f678dc08740 1 -- 172.21.15.52:0/4224494927 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x7f67801b6df0 con 0x5557991764d0 2024-05-27T07:32:50.006 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:32:50.012+0000 7f67852f4640 1 --2- 172.21.15.52:0/4224494927 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5557991971b0 0x555799546430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:32:50.007 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:32:50.012+0000 7f678dc08740 1 -- 172.21.15.52:0/4224494927 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f67801b6720 con 0x5557991971b0 2024-05-27T07:32:50.007 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:32:50.012+0000 7f678dc08740 1 -- 172.21.15.52:0/4224494927 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x555799540eb0 con 0x55579916e2f0 2024-05-27T07:32:50.008 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:32:50.012+0000 7f67852f4640 1 -- 172.21.15.52:0/4224494927 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55579916e2f0 msgr2=0x5557995459b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:32:50.008 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:32:50.012+0000 7f67852f4640 1 --2- 172.21.15.52:0/4224494927 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55579916e2f0 0x5557995459b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:32:50.008 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:32:50.012+0000 7f67852f4640 1 -- 172.21.15.52:0/4224494927 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x5557991764d0 msgr2=0x555799545ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:32:50.008 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:32:50.012+0000 7f67852f4640 1 --2- 172.21.15.52:0/4224494927 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x5557991764d0 0x555799545ef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:32:50.008 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:32:50.012+0000 7f67852f4640 1 -- 172.21.15.52:0/4224494927 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5557995ccbd0 con 0x5557991971b0 2024-05-27T07:32:50.008 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:32:50.013+0000 7f67852f4640 1 --2- 172.21.15.52:0/4224494927 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5557991971b0 0x555799546430 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7f6778100d40 tx=0x7f67780caea0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=41d5c6526c4893d4 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:32:50.008 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:32:50.013+0000 7f678dc08740 10 client.0 _ll_register_callbacks cb 0x7f6770006b40 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T07:32:50.008 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:32:50.013+0000 7f678dc08740 1 -- 172.21.15.52:0/4224494927 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f6770019070 con 0x5557991971b0 2024-05-27T07:32:50.029 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:32:50.035+0000 7f675ffff640 10 client.26462 ms_handle_connect on v2:172.21.15.105:3300/0 2024-05-27T07:32:50.029 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:32:50.035+0000 7f675ffff640 1 -- 172.21.15.52:0/4224494927 <== mon.2 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f67780ca6a0 con 0x5557991971b0 2024-05-27T07:32:50.029 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:32:50.035+0000 7f675ffff640 1 -- 172.21.15.52:0/4224494927 <== mon.2 v2:172.21.15.105:3300/0 2 ==== osd_map(116..116 src has 1..116) ==== 9379+0+0 (secure 0 0 0) 0x7f6778039830 con 0x5557991971b0 2024-05-27T07:32:50.030 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:32:50.035+0000 7f675ffff640 1 -- 172.21.15.52:0/4224494927 <== mon.2 v2:172.21.15.105:3300/0 3 ==== mon_get_version_reply(handle=1 version=101) ==== 24+0+0 (secure 0 0 0) 0x7f67780c6060 con 0x5557991971b0 2024-05-27T07:32:50.030 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:32:50.035+0000 7f678dc08740 10 client.26462 fetch_fsmap learned FSMap version 101 2024-05-27T07:32:50.030 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:32:50.035+0000 7f678dc08740 1 -- 172.21.15.52:0/4224494927 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({fsmap=101}) -- 0x5557995cdd60 con 0x5557991971b0 2024-05-27T07:32:50.030 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:32:50.035+0000 7f675ffff640 1 -- 172.21.15.52:0/4224494927 <== mon.2 v2:172.21.15.105:3300/0 4 ==== fsmap(e 101) ==== 4203+0+0 (secure 0 0 0) 0x7f677804c030 con 0x5557991971b0 2024-05-27T07:32:50.030 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:32:50.036+0000 7f678dc08740 10 client.26462 fetch_fsmap finished waiting for FSMap version 101 2024-05-27T07:32:50.030 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:32:50.036+0000 7f678dc08740 10 client.26462 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-05-27T07:32:50.030 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:32:50.036+0000 7f678dc08740 20 client.26462 populate_metadata read hostname 'smithi052' 2024-05-27T07:32:50.031 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:32:50.036+0000 7f678dc08740 1 --2- 172.21.15.52:0/4224494927 >> [v2:172.21.15.105:6838/500348151,v1:172.21.15.105:6839/500348151] conn(0x55579912b1c0 0x5557995d04d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:32:50.031 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:32:50.036+0000 7f678dc08740 4 client.26462 mds_command: new command op to 24527 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-05-27T07:32:50.031 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:32:50.036+0000 7f678dc08740 1 -- 172.21.15.52:0/4224494927 --> [v2:172.21.15.105:6838/500348151,v1:172.21.15.105:6839/500348151] -- command(tid 0: {"prefix": "flush journal"}) -- 0x7f6770019070 con 0x55579912b1c0 2024-05-27T07:32:50.031 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:32:50.036+0000 7f6784af3640 1 --2- 172.21.15.52:0/4224494927 >> [v2:172.21.15.105:6838/500348151,v1:172.21.15.105:6839/500348151] conn(0x55579912b1c0 0x5557995d04d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:32:50.031 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:32:50.036+0000 7f6784af3640 1 --2- 172.21.15.52:0/4224494927 >> [v2:172.21.15.105:6838/500348151,v1:172.21.15.105:6839/500348151] conn(0x55579912b1c0 0x5557995d04d0 crc :-1 s=READY pgs=134 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=4afc5f42d1b3e43c server_cookie=c77d9ac9708737dd in_seq=0 out_seq=0 2024-05-27T07:32:50.031 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:32:50.037+0000 7f675ffff640 10 client.26462 ms_handle_connect on v2:172.21.15.105:6838/500348151 2024-05-27T07:32:50.605 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:32:50 smithi173 ceph-mon[37000]: pgmap v234: 129 pgs: 129 active+clean; 626 MiB data, 3.2 GiB used, 1.0 TiB / 1.0 TiB avail; 370 KiB/s rd, 2.6 MiB/s wr, 412 op/s 2024-05-27T07:32:50.693 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:32:50 smithi105 ceph-mon[35977]: pgmap v234: 129 pgs: 129 active+clean; 626 MiB data, 3.2 GiB used, 1.0 TiB / 1.0 TiB avail; 370 KiB/s rd, 2.6 MiB/s wr, 412 op/s 2024-05-27T07:32:50.733 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:32:50 smithi052 ceph-mon[31955]: pgmap v234: 129 pgs: 129 active+clean; 626 MiB data, 3.2 GiB used, 1.0 TiB / 1.0 TiB avail; 370 KiB/s rd, 2.6 MiB/s wr, 412 op/s 2024-05-27T07:32:52.605 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:32:52 smithi173 ceph-mon[37000]: pgmap v235: 129 pgs: 129 active+clean; 625 MiB data, 3.2 GiB used, 1.0 TiB / 1.0 TiB avail; 388 KiB/s rd, 2.4 MiB/s wr, 368 op/s 2024-05-27T07:32:52.693 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:32:52 smithi105 ceph-mon[35977]: pgmap v235: 129 pgs: 129 active+clean; 625 MiB data, 3.2 GiB used, 1.0 TiB / 1.0 TiB avail; 388 KiB/s rd, 2.4 MiB/s wr, 368 op/s 2024-05-27T07:32:52.733 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:32:52 smithi052 ceph-mon[31955]: pgmap v235: 129 pgs: 129 active+clean; 625 MiB data, 3.2 GiB used, 1.0 TiB / 1.0 TiB avail; 388 KiB/s rd, 2.4 MiB/s wr, 368 op/s 2024-05-27T07:32:54.605 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:32:54 smithi173 ceph-mon[37000]: pgmap v236: 129 pgs: 129 active+clean; 625 MiB data, 3.2 GiB used, 1.0 TiB / 1.0 TiB avail; 218 KiB/s rd, 1.7 MiB/s wr, 457 op/s 2024-05-27T07:32:54.693 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:32:54 smithi105 ceph-mon[35977]: pgmap v236: 129 pgs: 129 active+clean; 625 MiB data, 3.2 GiB used, 1.0 TiB / 1.0 TiB avail; 218 KiB/s rd, 1.7 MiB/s wr, 457 op/s 2024-05-27T07:32:54.733 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:32:54 smithi052 ceph-mon[31955]: pgmap v236: 129 pgs: 129 active+clean; 625 MiB data, 3.2 GiB used, 1.0 TiB / 1.0 TiB avail; 218 KiB/s rd, 1.7 MiB/s wr, 457 op/s 2024-05-27T07:32:55.221 INFO:tasks.workunit.client.0.smithi052.stdout: 162 1141337 402 777908 181 671545 700 2024-05-27T07:32:56.605 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:32:56 smithi173 ceph-mon[37000]: pgmap v237: 129 pgs: 129 active+clean; 633 MiB data, 3.3 GiB used, 1.0 TiB / 1.0 TiB avail; 222 KiB/s rd, 2.1 MiB/s wr, 798 op/s 2024-05-27T07:32:56.693 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:32:56 smithi105 ceph-mon[35977]: pgmap v237: 129 pgs: 129 active+clean; 633 MiB data, 3.3 GiB used, 1.0 TiB / 1.0 TiB avail; 222 KiB/s rd, 2.1 MiB/s wr, 798 op/s 2024-05-27T07:32:56.733 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:32:56 smithi052 ceph-mon[31955]: pgmap v237: 129 pgs: 129 active+clean; 633 MiB data, 3.3 GiB used, 1.0 TiB / 1.0 TiB avail; 222 KiB/s rd, 2.1 MiB/s wr, 798 op/s 2024-05-27T07:32:58.483 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:32:58 smithi052 ceph-mon[31955]: pgmap v238: 129 pgs: 129 active+clean; 643 MiB data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail; 193 KiB/s rd, 2.7 MiB/s wr, 1.30k op/s 2024-05-27T07:32:58.605 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:32:58 smithi173 ceph-mon[37000]: pgmap v238: 129 pgs: 129 active+clean; 643 MiB data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail; 193 KiB/s rd, 2.7 MiB/s wr, 1.30k op/s 2024-05-27T07:32:58.693 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:32:58 smithi105 ceph-mon[35977]: pgmap v238: 129 pgs: 129 active+clean; 643 MiB data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail; 193 KiB/s rd, 2.7 MiB/s wr, 1.30k op/s 2024-05-27T07:32:59.476 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:32:59.482+0000 7f675ffff640 1 -- 172.21.15.52:0/4224494927 <== mds.0 v2:172.21.15.105:6838/500348151 1 ==== command_reply(tid 0: 0 ) ==== 8+0+25 (crc 0 0 0) 0x7f6770019070 con 0x55579912b1c0 2024-05-27T07:32:59.476 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:32:59.482+0000 7f675ffff640 10 client.26462 handle_command_reply: tid=0 2024-05-27T07:32:59.477 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:32:59.482+0000 7f678dc08740 1 client.26462 shutdown 2024-05-27T07:32:59.477 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:32:59.482+0000 7f678dc08740 1 -- 172.21.15.52:0/4224494927 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5557991971b0 msgr2=0x555799546430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:32:59.477 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:32:59.482+0000 7f678dc08740 1 --2- 172.21.15.52:0/4224494927 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5557991971b0 0x555799546430 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7f6778100d40 tx=0x7f67780caea0 comp rx=0 tx=0).stop 2024-05-27T07:32:59.477 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:32:59.482+0000 7f678dc08740 1 -- 172.21.15.52:0/4224494927 shutdown_connections 2024-05-27T07:32:59.477 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:32:59.482+0000 7f678dc08740 1 --2- 172.21.15.52:0/4224494927 >> [v2:172.21.15.105:6838/500348151,v1:172.21.15.105:6839/500348151] conn(0x55579912b1c0 0x5557995d04d0 crc :-1 s=READY pgs=134 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:32:59.477 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:32:59.482+0000 7f678dc08740 1 --2- 172.21.15.52:0/4224494927 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5557991971b0 0x555799546430 unknown :-1 s=CLOSED pgs=210 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:32:59.477 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:32:59.482+0000 7f675ffff640 0 client.26462 ms_handle_reset on v2:172.21.15.105:6838/500348151 2024-05-27T07:32:59.477 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:32:59.482+0000 7f678dc08740 1 --2- 172.21.15.52:0/4224494927 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x5557991764d0 0x555799545ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:32:59.477 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:32:59.482+0000 7f678dc08740 1 --2- 172.21.15.52:0/4224494927 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55579916e2f0 0x5557995459b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:32:59.477 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:32:59.482+0000 7f678dc08740 1 -- 172.21.15.52:0/4224494927 >> 172.21.15.52:0/4224494927 conn(0x55579918b300 msgr2=0x555799544010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:32:59.477 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:32:59.482+0000 7f678dc08740 1 -- 172.21.15.52:0/4224494927 shutdown_connections 2024-05-27T07:32:59.478 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:32:59.482+0000 7f678dc08740 1 -- 172.21.15.52:0/4224494927 wait complete. 2024-05-27T07:32:59.478 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:32:59.482+0000 7f678dc08740 20 client.26462 trim_cache size 0 max 16384 2024-05-27T07:32:59.478 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stdout:{ 2024-05-27T07:32:59.478 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stdout: "return_code": 0 2024-05-27T07:32:59.478 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stdout:} 2024-05-27T07:32:59.479 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:32:59.483+0000 7f6786d57640 1 -- 172.21.15.52:0/1713758666 >> [v2:172.21.15.52:6800/1822318384,v1:172.21.15.52:6801/1822318384] conn(0x7f67680b2b70 msgr2=0x7f67680b5080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:32:59.479 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:32:59.483+0000 7f6786d57640 1 --2- 172.21.15.52:0/1713758666 >> [v2:172.21.15.52:6800/1822318384,v1:172.21.15.52:6801/1822318384] conn(0x7f67680b2b70 0x7f67680b5080 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x555799535700 tx=0x7f6770008040 comp rx=0 tx=0).stop 2024-05-27T07:32:59.479 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:32:59.483+0000 7f6786d57640 1 -- 172.21.15.52:0/1713758666 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f67801ad8a0 msgr2=0x7f67801adcb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:32:59.479 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:32:59.483+0000 7f6786d57640 1 --2- 172.21.15.52:0/1713758666 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f67801ad8a0 0x7f67801adcb0 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7f677802fe10 tx=0x7f6778031c30 comp rx=0 tx=0).stop 2024-05-27T07:32:59.479 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:32:59.483+0000 7f6786d57640 1 -- 172.21.15.52:0/1713758666 shutdown_connections 2024-05-27T07:32:59.479 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:32:59.483+0000 7f6786d57640 1 --2- 172.21.15.52:0/1713758666 >> [v2:172.21.15.52:6800/1822318384,v1:172.21.15.52:6801/1822318384] conn(0x7f67680b2b70 0x7f67680b5080 secure :-1 s=CLOSED pgs=159 cs=0 l=1 rev1=1 crypto rx=0x555799535700 tx=0x7f6770008040 comp rx=0 tx=0).stop 2024-05-27T07:32:59.479 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:32:59.483+0000 7f6786d57640 1 --2- 172.21.15.52:0/1713758666 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f67801ad8a0 0x7f67801adcb0 unknown :-1 s=CLOSED pgs=207 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:32:59.480 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:32:59.483+0000 7f6786d57640 1 --2- 172.21.15.52:0/1713758666 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f678011a8e0 0x7f67801a96a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:32:59.480 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:32:59.483+0000 7f6786d57640 1 --2- 172.21.15.52:0/1713758666 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f678010a0e0 0x7f67801a9140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:32:59.480 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:32:59.483+0000 7f6786d57640 1 -- 172.21.15.52:0/1713758666 >> 172.21.15.52:0/1713758666 conn(0x7f678006f770 msgr2=0x7f678010eed0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:32:59.480 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:32:59.483+0000 7f6786d57640 1 -- 172.21.15.52:0/1713758666 shutdown_connections 2024-05-27T07:32:59.480 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:32:59.484+0000 7f6786d57640 1 -- 172.21.15.52:0/1713758666 wait complete. 2024-05-27T07:33:00.028 INFO:tasks.workunit.client.0.smithi052.stdout: 2024-05-27T07:33:00.028 INFO:tasks.workunit.client.0.smithi052.stdout:Final score for writes: 162 2024-05-27T07:33:00.028 INFO:tasks.workunit.client.0.smithi052.stdout:Final score for reads : 541884 2024-05-27T07:33:00.028 INFO:tasks.workunit.client.0.smithi052.stdout: 2024-05-27T07:33:00.029 INFO:teuthology.orchestra.run:Running command with timeout 3600 2024-05-27T07:33:00.029 DEBUG:teuthology.orchestra.run.smithi052:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0/tmp 2024-05-27T07:33:00.605 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:33:00 smithi173 ceph-mon[37000]: pgmap v239: 129 pgs: 129 active+clean; 664 MiB data, 3.5 GiB used, 1.0 TiB / 1.0 TiB avail; 86 KiB/s rd, 4.4 MiB/s wr, 1.42k op/s 2024-05-27T07:33:00.693 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:33:00 smithi105 ceph-mon[35977]: pgmap v239: 129 pgs: 129 active+clean; 664 MiB data, 3.5 GiB used, 1.0 TiB / 1.0 TiB avail; 86 KiB/s rd, 4.4 MiB/s wr, 1.42k op/s 2024-05-27T07:33:00.733 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:33:00 smithi052 ceph-mon[31955]: pgmap v239: 129 pgs: 129 active+clean; 664 MiB data, 3.5 GiB used, 1.0 TiB / 1.0 TiB avail; 86 KiB/s rd, 4.4 MiB/s wr, 1.42k op/s 2024-05-27T07:33:02.605 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:33:02 smithi173 ceph-mon[37000]: pgmap v240: 129 pgs: 129 active+clean; 671 MiB data, 3.5 GiB used, 1.0 TiB / 1.0 TiB avail; 218 KiB/s rd, 5.7 MiB/s wr, 1.73k op/s 2024-05-27T07:33:02.693 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:33:02 smithi105 ceph-mon[35977]: pgmap v240: 129 pgs: 129 active+clean; 671 MiB data, 3.5 GiB used, 1.0 TiB / 1.0 TiB avail; 218 KiB/s rd, 5.7 MiB/s wr, 1.73k op/s 2024-05-27T07:33:02.733 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:33:02 smithi052 ceph-mon[31955]: pgmap v240: 129 pgs: 129 active+clean; 671 MiB data, 3.5 GiB used, 1.0 TiB / 1.0 TiB avail; 218 KiB/s rd, 5.7 MiB/s wr, 1.73k op/s 2024-05-27T07:33:03.605 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:33:03 smithi173 ceph-mon[37000]: osdmap e117: 12 total, 12 up, 12 in 2024-05-27T07:33:03.693 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:33:03 smithi105 ceph-mon[35977]: osdmap e117: 12 total, 12 up, 12 in 2024-05-27T07:33:03.733 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:33:03 smithi052 ceph-mon[31955]: osdmap e117: 12 total, 12 up, 12 in 2024-05-27T07:33:04.605 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:33:04 smithi173 ceph-mon[37000]: pgmap v242: 129 pgs: 3 active+clean+snaptrim, 119 active+clean, 7 active+clean+snaptrim_wait; 656 MiB data, 3.5 GiB used, 1.0 TiB / 1.0 TiB avail; 354 KiB/s rd, 7.2 MiB/s wr, 1.92k op/s 2024-05-27T07:33:04.694 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:33:04 smithi105 ceph-mon[35977]: pgmap v242: 129 pgs: 3 active+clean+snaptrim, 119 active+clean, 7 active+clean+snaptrim_wait; 656 MiB data, 3.5 GiB used, 1.0 TiB / 1.0 TiB avail; 354 KiB/s rd, 7.2 MiB/s wr, 1.92k op/s 2024-05-27T07:33:04.733 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:33:04 smithi052 ceph-mon[31955]: pgmap v242: 129 pgs: 3 active+clean+snaptrim, 119 active+clean, 7 active+clean+snaptrim_wait; 656 MiB data, 3.5 GiB used, 1.0 TiB / 1.0 TiB avail; 354 KiB/s rd, 7.2 MiB/s wr, 1.92k op/s 2024-05-27T07:33:06.693 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:33:06 smithi105 ceph-mon[35977]: pgmap v243: 129 pgs: 9 active+clean+snaptrim, 100 active+clean, 20 active+clean+snaptrim_wait; 656 MiB data, 3.5 GiB used, 1.0 TiB / 1.0 TiB avail; 600 KiB/s rd, 6.8 MiB/s wr, 1.80k op/s 2024-05-27T07:33:06.733 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:33:06 smithi052 ceph-mon[31955]: pgmap v243: 129 pgs: 9 active+clean+snaptrim, 100 active+clean, 20 active+clean+snaptrim_wait; 656 MiB data, 3.5 GiB used, 1.0 TiB / 1.0 TiB avail; 600 KiB/s rd, 6.8 MiB/s wr, 1.80k op/s 2024-05-27T07:33:06.855 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:33:06 smithi173 ceph-mon[37000]: pgmap v243: 129 pgs: 9 active+clean+snaptrim, 100 active+clean, 20 active+clean+snaptrim_wait; 656 MiB data, 3.5 GiB used, 1.0 TiB / 1.0 TiB avail; 600 KiB/s rd, 6.8 MiB/s wr, 1.80k op/s 2024-05-27T07:33:07.694 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:33:07 smithi105 ceph-mon[35977]: fsmap cephfs:5 5 up:active 2024-05-27T07:33:07.694 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:33:07 smithi105 ceph-mon[35977]: scrub summary: idle+waiting paths [/] 2024-05-27T07:33:07.694 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:33:07 smithi105 ceph-mon[35977]: scrub summary: idle 2024-05-27T07:33:07.694 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:33:07 smithi105 ceph-mon[35977]: scrub summary: idle 2024-05-27T07:33:07.694 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:33:07 smithi105 ceph-mon[35977]: scrub summary: idle 2024-05-27T07:33:07.694 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:33:07 smithi105 ceph-mon[35977]: pgmap v244: 129 pgs: 19 active+clean+snaptrim, 61 active+clean, 49 active+clean+snaptrim_wait; 657 MiB data, 3.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 5.8 MiB/s wr, 1.61k op/s 2024-05-27T07:33:07.733 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:33:07 smithi052 ceph-mon[31955]: fsmap cephfs:5 5 up:active 2024-05-27T07:33:07.733 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:33:07 smithi052 ceph-mon[31955]: scrub summary: idle+waiting paths [/] 2024-05-27T07:33:07.733 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:33:07 smithi052 ceph-mon[31955]: scrub summary: idle 2024-05-27T07:33:07.733 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:33:07 smithi052 ceph-mon[31955]: scrub summary: idle 2024-05-27T07:33:07.733 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:33:07 smithi052 ceph-mon[31955]: scrub summary: idle 2024-05-27T07:33:07.733 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:33:07 smithi052 ceph-mon[31955]: pgmap v244: 129 pgs: 19 active+clean+snaptrim, 61 active+clean, 49 active+clean+snaptrim_wait; 657 MiB data, 3.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 5.8 MiB/s wr, 1.61k op/s 2024-05-27T07:33:07.855 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:33:07 smithi173 ceph-mon[37000]: fsmap cephfs:5 5 up:active 2024-05-27T07:33:07.855 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:33:07 smithi173 ceph-mon[37000]: scrub summary: idle+waiting paths [/] 2024-05-27T07:33:07.856 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:33:07 smithi173 ceph-mon[37000]: scrub summary: idle 2024-05-27T07:33:07.856 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:33:07 smithi173 ceph-mon[37000]: scrub summary: idle 2024-05-27T07:33:07.856 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:33:07 smithi173 ceph-mon[37000]: scrub summary: idle 2024-05-27T07:33:07.856 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:33:07 smithi173 ceph-mon[37000]: pgmap v244: 129 pgs: 19 active+clean+snaptrim, 61 active+clean, 49 active+clean+snaptrim_wait; 657 MiB data, 3.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 5.8 MiB/s wr, 1.61k op/s 2024-05-27T07:33:08.693 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:33:08 smithi105 ceph-mon[35977]: scrub summary: idle 2024-05-27T07:33:08.694 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:33:08 smithi105 ceph-mon[35977]: scrub summary: idle 2024-05-27T07:33:08.733 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:33:08 smithi052 ceph-mon[31955]: scrub summary: idle 2024-05-27T07:33:08.733 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:33:08 smithi052 ceph-mon[31955]: scrub summary: idle 2024-05-27T07:33:08.855 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:33:08 smithi173 ceph-mon[37000]: scrub summary: idle 2024-05-27T07:33:08.855 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:33:08 smithi173 ceph-mon[37000]: scrub summary: idle 2024-05-27T07:33:09.693 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:33:09 smithi105 ceph-mon[35977]: pgmap v245: 129 pgs: 19 active+clean+snaptrim, 61 active+clean, 49 active+clean+snaptrim_wait; 660 MiB data, 3.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 3.9 MiB/s wr, 1.51k op/s 2024-05-27T07:33:09.734 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:33:09 smithi052 ceph-mon[31955]: pgmap v245: 129 pgs: 19 active+clean+snaptrim, 61 active+clean, 49 active+clean+snaptrim_wait; 660 MiB data, 3.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 3.9 MiB/s wr, 1.51k op/s 2024-05-27T07:33:09.855 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:33:09 smithi173 ceph-mon[37000]: pgmap v245: 129 pgs: 19 active+clean+snaptrim, 61 active+clean, 49 active+clean+snaptrim_wait; 660 MiB data, 3.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 3.9 MiB/s wr, 1.51k op/s 2024-05-27T07:33:12.443 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:33:12 smithi105 ceph-mon[35977]: pgmap v246: 129 pgs: 19 active+clean+snaptrim, 61 active+clean, 49 active+clean+snaptrim_wait; 662 MiB data, 3.6 GiB used, 1.0 TiB / 1.0 TiB avail; 924 KiB/s rd, 1.7 MiB/s wr, 1.14k op/s 2024-05-27T07:33:12.483 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:33:12 smithi052 ceph-mon[31955]: pgmap v246: 129 pgs: 19 active+clean+snaptrim, 61 active+clean, 49 active+clean+snaptrim_wait; 662 MiB data, 3.6 GiB used, 1.0 TiB / 1.0 TiB avail; 924 KiB/s rd, 1.7 MiB/s wr, 1.14k op/s 2024-05-27T07:33:12.605 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:33:12 smithi173 ceph-mon[37000]: pgmap v246: 129 pgs: 19 active+clean+snaptrim, 61 active+clean, 49 active+clean+snaptrim_wait; 662 MiB data, 3.6 GiB used, 1.0 TiB / 1.0 TiB avail; 924 KiB/s rd, 1.7 MiB/s wr, 1.14k op/s 2024-05-27T07:33:12.646 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:33:12.651+0000 7fe42cbf3640 1 -- 172.21.15.52:0/218857513 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fe428111270 msgr2=0x7fe428111680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:33:12.646 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:33:12.651+0000 7fe42cbf3640 1 --2- 172.21.15.52:0/218857513 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fe428111270 0x7fe428111680 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7fe410009900 tx=0x7fe41002a400 comp rx=0 tx=0).stop 2024-05-27T07:33:12.646 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:33:12.652+0000 7fe42cbf3640 1 -- 172.21.15.52:0/218857513 shutdown_connections 2024-05-27T07:33:12.647 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:33:12.652+0000 7fe42cbf3640 1 --2- 172.21.15.52:0/218857513 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fe428110640 0x7fe42810a9e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:33:12.647 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:33:12.653+0000 7fe42cbf3640 1 --2- 172.21.15.52:0/218857513 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fe42810fb80 0x7fe42810ffb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:33:12.647 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:33:12.653+0000 7fe42cbf3640 1 --2- 172.21.15.52:0/218857513 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fe428111270 0x7fe428111680 unknown :-1 s=CLOSED pgs=211 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:33:12.647 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:33:12.653+0000 7fe42cbf3640 1 -- 172.21.15.52:0/218857513 >> 172.21.15.52:0/218857513 conn(0x7fe42806e8f0 msgr2=0x7fe42806ed30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:33:12.648 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:33:12.654+0000 7fe42cbf3640 1 -- 172.21.15.52:0/218857513 shutdown_connections 2024-05-27T07:33:12.648 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:33:12.654+0000 7fe42cbf3640 1 -- 172.21.15.52:0/218857513 wait complete. 2024-05-27T07:33:12.649 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:33:12.656+0000 7fe42cbf3640 1 Processor -- start 2024-05-27T07:33:12.653 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:33:12.658+0000 7fe42cbf3640 1 -- start start 2024-05-27T07:33:12.653 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:33:12.658+0000 7fe42cbf3640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fe42810fb80 0x7fe42811b1e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:33:12.653 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:33:12.658+0000 7fe42cbf3640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fe428110640 0x7fe42811b720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:33:12.653 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:33:12.658+0000 7fe42cbf3640 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fe428111270 0x7fe42811bc60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:33:12.653 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:33:12.658+0000 7fe42cbf3640 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x7fe42811e900 con 0x7fe428111270 2024-05-27T07:33:12.653 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:33:12.658+0000 7fe42cbf3640 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7fe42811e780 con 0x7fe428110640 2024-05-27T07:33:12.653 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:33:12.658+0000 7fe42cbf3640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7fe42811ea80 con 0x7fe42810fb80 2024-05-27T07:33:12.655 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:33:12.661+0000 7fe425d9b640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fe428110640 0x7fe42811b720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:33:12.655 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:33:12.661+0000 7fe425d9b640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fe428110640 0x7fe42811b720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.52:35686/0 (socket says 172.21.15.52:35686) 2024-05-27T07:33:12.655 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:33:12.661+0000 7fe425d9b640 1 -- 172.21.15.52:0/1867373087 learned_addr learned my addr 172.21.15.52:0/1867373087 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-05-27T07:33:12.657 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:33:12.663+0000 7fe42659c640 1 --2- 172.21.15.52:0/1867373087 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fe42810fb80 0x7fe42811b1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:33:12.657 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:33:12.663+0000 7fe426d9d640 1 --2- 172.21.15.52:0/1867373087 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fe428111270 0x7fe42811bc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:33:12.658 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:33:12.664+0000 7fe42659c640 1 -- 172.21.15.52:0/1867373087 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fe428110640 msgr2=0x7fe42811b720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:33:12.658 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:33:12.664+0000 7fe42659c640 1 --2- 172.21.15.52:0/1867373087 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fe428110640 0x7fe42811b720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:33:12.658 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:33:12.664+0000 7fe42659c640 1 -- 172.21.15.52:0/1867373087 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fe428111270 msgr2=0x7fe42811bc60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:33:12.658 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:33:12.664+0000 7fe42659c640 1 --2- 172.21.15.52:0/1867373087 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fe428111270 0x7fe42811bc60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:33:12.658 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:33:12.664+0000 7fe42659c640 1 -- 172.21.15.52:0/1867373087 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe4281aae90 con 0x7fe42810fb80 2024-05-27T07:33:12.658 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:33:12.664+0000 7fe42659c640 1 --2- 172.21.15.52:0/1867373087 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fe42810fb80 0x7fe42811b1e0 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7fe41002a9f0 tx=0x7fe410030640 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:33:12.658 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:33:12.664+0000 7fe425d9b640 1 --2- 172.21.15.52:0/1867373087 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fe428110640 0x7fe42811b720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-27T07:33:12.661 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:33:12.665+0000 7fe41f7fe640 1 -- 172.21.15.52:0/1867373087 <== mon.1 v2:172.21.15.173:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe41003f030 con 0x7fe42810fb80 2024-05-27T07:33:12.661 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:33:12.665+0000 7fe41f7fe640 1 -- 172.21.15.52:0/1867373087 <== mon.1 v2:172.21.15.173:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fe41003b460 con 0x7fe42810fb80 2024-05-27T07:33:12.662 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:33:12.665+0000 7fe41f7fe640 1 -- 172.21.15.52:0/1867373087 <== mon.1 v2:172.21.15.173:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe410035070 con 0x7fe42810fb80 2024-05-27T07:33:12.662 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:33:12.665+0000 7fe42cbf3640 1 -- 172.21.15.52:0/1867373087 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe4281ab120 con 0x7fe42810fb80 2024-05-27T07:33:12.662 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:33:12.665+0000 7fe42cbf3640 1 -- 172.21.15.52:0/1867373087 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe4281ab5c0 con 0x7fe42810fb80 2024-05-27T07:33:12.670 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:33:12.674+0000 7fe433b7d740 1 Processor -- start 2024-05-27T07:33:12.670 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:33:12.674+0000 7fe41f7fe640 1 -- 172.21.15.52:0/1867373087 <== mon.1 v2:172.21.15.173:3300/0 4 ==== mgrmap(e 26) ==== 149783+0+0 (secure 0 0 0) 0x7fe410042030 con 0x7fe42810fb80 2024-05-27T07:33:12.671 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:33:12.676+0000 7fe41f7fe640 1 --2- 172.21.15.52:0/1867373087 >> [v2:172.21.15.52:6800/1822318384,v1:172.21.15.52:6801/1822318384] conn(0x7fe4140b2b60 0x7fe4140b5070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:33:12.671 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:33:12.676+0000 7fe41f7fe640 1 -- 172.21.15.52:0/1867373087 <== mon.1 v2:172.21.15.173:3300/0 5 ==== osd_map(117..117 src has 1..117) ==== 9555+0+0 (secure 0 0 0) 0x7fe4101032c0 con 0x7fe42810fb80 2024-05-27T07:33:12.671 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:33:12.677+0000 7fe433b7d740 1 -- start start 2024-05-27T07:33:12.672 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:33:12.677+0000 7fe433b7d740 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x5624d0f551b0 0x5624d12f63d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:33:12.672 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:33:12.677+0000 7fe433b7d740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5624d0f2c2f0 0x5624d12f6910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:33:12.672 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:33:12.677+0000 7fe433b7d740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5624d0f344d0 0x5624d12f6e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:33:12.672 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:33:12.677+0000 7fe433b7d740 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x5624d1268c30 con 0x5624d0f551b0 2024-05-27T07:33:12.672 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:33:12.677+0000 7fe433b7d740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x5624d12587d0 con 0x5624d0f344d0 2024-05-27T07:33:12.672 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:33:12.677+0000 7fe433b7d740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x5624d125c810 con 0x5624d0f2c2f0 2024-05-27T07:33:12.672 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:33:12.677+0000 7fe426d9d640 1 --2- 172.21.15.52:0/1867373087 >> [v2:172.21.15.52:6800/1822318384,v1:172.21.15.52:6801/1822318384] conn(0x7fe4140b2b60 0x7fe4140b5070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:33:12.672 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:33:12.678+0000 7fe426d9d640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5624d0f344d0 0x5624d12f6e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:33:12.672 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:33:12.678+0000 7fe426d9d640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5624d0f344d0 0x5624d12f6e50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.52:35702/0 (socket says 172.21.15.52:35702) 2024-05-27T07:33:12.672 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:33:12.678+0000 7fe426d9d640 1 -- 172.21.15.52:0/3601574803 learned_addr learned my addr 172.21.15.52:0/3601574803 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-05-27T07:33:12.673 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:33:12.679+0000 7fe425d9b640 1 --2- 172.21.15.52:0/3601574803 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5624d0f2c2f0 0x5624d12f6910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:33:12.673 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:33:12.679+0000 7fe42659c640 1 --2- 172.21.15.52:0/3601574803 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x5624d0f551b0 0x5624d12f63d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:33:12.679 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:33:12.683+0000 7fe42659c640 1 -- 172.21.15.52:0/3601574803 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5624d0f2c2f0 msgr2=0x5624d12f6910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:33:12.679 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:33:12.683+0000 7fe42659c640 1 --2- 172.21.15.52:0/3601574803 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5624d0f2c2f0 0x5624d12f6910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:33:12.679 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:33:12.683+0000 7fe42659c640 1 -- 172.21.15.52:0/3601574803 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5624d0f344d0 msgr2=0x5624d12f6e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:33:12.679 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:33:12.683+0000 7fe42659c640 1 --2- 172.21.15.52:0/3601574803 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5624d0f344d0 0x5624d12f6e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:33:12.679 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:33:12.683+0000 7fe42659c640 1 -- 172.21.15.52:0/3601574803 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5624d0f6d210 con 0x5624d0f551b0 2024-05-27T07:33:12.679 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:33:12.684+0000 7fe42659c640 1 --2- 172.21.15.52:0/3601574803 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x5624d0f551b0 0x5624d12f63d0 secure :-1 s=READY pgs=683 cs=0 l=1 rev1=1 crypto rx=0x7fe4101005a0 tx=0x7fe4100c6090 comp rx=0 tx=0).ready entity=mon.0 client_cookie=deb2040bf78ac463 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:33:12.682 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:33:12.685+0000 7fe426d9d640 1 --2- 172.21.15.52:0/1867373087 >> [v2:172.21.15.52:6800/1822318384,v1:172.21.15.52:6801/1822318384] conn(0x7fe4140b2b60 0x7fe4140b5070 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7fe420012780 tx=0x7fe42000c5a0 comp rx=0 tx=0).ready entity=mgr.15333 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:33:12.682 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:33:12.685+0000 7fe41cff9640 1 -- 172.21.15.52:0/3601574803 <== mon.0 v2:172.21.15.52:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe4100e4070 con 0x5624d0f551b0 2024-05-27T07:33:12.682 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:33:12.685+0000 7fe41cff9640 1 -- 172.21.15.52:0/3601574803 <== mon.0 v2:172.21.15.52:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fe4100ca130 con 0x5624d0f551b0 2024-05-27T07:33:12.682 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:33:12.686+0000 7fe41cff9640 1 -- 172.21.15.52:0/3601574803 <== mon.0 v2:172.21.15.52:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe4100d2b00 con 0x5624d0f551b0 2024-05-27T07:33:12.682 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:33:12.687+0000 7fe433b7d740 1 -- 172.21.15.52:0/3601574803 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x5624d0f551b0 msgr2=0x5624d12f63d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:33:12.682 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:33:12.687+0000 7fe433b7d740 1 --2- 172.21.15.52:0/3601574803 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x5624d0f551b0 0x5624d12f63d0 secure :-1 s=READY pgs=683 cs=0 l=1 rev1=1 crypto rx=0x7fe4101005a0 tx=0x7fe4100c6090 comp rx=0 tx=0).stop 2024-05-27T07:33:12.682 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:33:12.687+0000 7fe433b7d740 1 -- 172.21.15.52:0/3601574803 shutdown_connections 2024-05-27T07:33:12.682 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:33:12.687+0000 7fe433b7d740 1 --2- 172.21.15.52:0/3601574803 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5624d0f344d0 0x5624d12f6e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:33:12.682 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:33:12.687+0000 7fe433b7d740 1 --2- 172.21.15.52:0/3601574803 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5624d0f2c2f0 0x5624d12f6910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:33:12.683 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:33:12.687+0000 7fe433b7d740 1 --2- 172.21.15.52:0/3601574803 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x5624d0f551b0 0x5624d12f63d0 unknown :-1 s=CLOSED pgs=683 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:33:12.683 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:33:12.687+0000 7fe433b7d740 1 -- 172.21.15.52:0/3601574803 >> 172.21.15.52:0/3601574803 conn(0x5624d0f49300 msgr2=0x5624d12f2b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:33:12.688 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:33:12.691+0000 7fe433b7d740 1 -- 172.21.15.52:0/3601574803 shutdown_connections 2024-05-27T07:33:12.691 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:33:12.695+0000 7fe433b7d740 1 -- 172.21.15.52:0/3601574803 wait complete. 2024-05-27T07:33:12.691 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:33:12.695+0000 7fe433b7d740 1 -- start start 2024-05-27T07:33:12.691 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:33:12.695+0000 7fe433b7d740 1 Processor -- start 2024-05-27T07:33:12.693 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:33:12.699+0000 7fe433b7d740 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x5624d0f2c2f0 0x5624d13009e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:33:12.693 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:33:12.699+0000 7fe433b7d740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5624d0f344d0 0x5624d1300f20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:33:12.693 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:33:12.699+0000 7fe433b7d740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5624d0f551b0 0x5624d130c600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:33:12.693 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:33:12.699+0000 7fe433b7d740 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x5624d0f3e790 con 0x5624d0f2c2f0 2024-05-27T07:33:12.693 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:33:12.699+0000 7fe433b7d740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x5624d1214890 con 0x5624d0f551b0 2024-05-27T07:33:12.693 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:33:12.699+0000 7fe433b7d740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x5624d13032c0 con 0x5624d0f344d0 2024-05-27T07:33:12.693 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:33:12.699+0000 7fe433b7d740 10 client.0 _ll_register_callbacks cb 0x5624d0f6ca00 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T07:33:12.694 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:33:12.700+0000 7fe42659c640 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x5624d0f2c2f0 0x5624d13009e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:33:12.694 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:33:12.700+0000 7fe42659c640 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x5624d0f2c2f0 0x5624d13009e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.52:32970/0 (socket says 172.21.15.52:32970) 2024-05-27T07:33:12.694 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:33:12.700+0000 7fe42659c640 1 -- 172.21.15.52:0/1841269016 learned_addr learned my addr 172.21.15.52:0/1841269016 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-05-27T07:33:12.694 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:33:12.700+0000 7fe425d9b640 1 --2- 172.21.15.52:0/1841269016 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5624d0f344d0 0x5624d1300f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:33:12.694 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:33:12.700+0000 7fe42659c640 1 -- 172.21.15.52:0/1841269016 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5624d0f344d0 msgr2=0x5624d1300f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:33:12.694 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:33:12.700+0000 7fe42659c640 1 --2- 172.21.15.52:0/1841269016 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5624d0f344d0 0x5624d1300f20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:33:12.694 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:33:12.700+0000 7fe42659c640 1 -- 172.21.15.52:0/1841269016 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5624d0f551b0 msgr2=0x5624d130c600 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-05-27T07:33:12.695 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:33:12.700+0000 7fe42659c640 1 --2- 172.21.15.52:0/1841269016 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5624d0f551b0 0x5624d130c600 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:33:12.695 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:33:12.700+0000 7fe42659c640 1 -- 172.21.15.52:0/1841269016 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5624d1039620 con 0x5624d0f2c2f0 2024-05-27T07:33:12.695 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:33:12.701+0000 7fe42659c640 1 --2- 172.21.15.52:0/1841269016 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x5624d0f2c2f0 0x5624d13009e0 secure :-1 s=READY pgs=684 cs=0 l=1 rev1=1 crypto rx=0x7fe4100c9e30 tx=0x7fe4100ce3a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=77b28523f3bdab6e server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:33:12.695 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:33:12.701+0000 7fe40dffb640 10 client.0 ms_handle_connect on v2:172.21.15.52:3300/0 2024-05-27T07:33:12.695 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:33:12.702+0000 7fe40dffb640 1 -- 172.21.15.52:0/1841269016 <== mon.0 v2:172.21.15.52:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe4100e4040 con 0x5624d0f2c2f0 2024-05-27T07:33:12.696 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:33:12.702+0000 7fe433b7d740 1 -- 172.21.15.52:0/1841269016 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fe4100ca130 con 0x5624d0f2c2f0 2024-05-27T07:33:12.696 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:33:12.702+0000 7fe40dffb640 1 -- 172.21.15.52:0/1841269016 <== mon.0 v2:172.21.15.52:3300/0 2 ==== osd_map(117..117 src has 1..117) ==== 9555+0+0 (secure 0 0 0) 0x7fe4100c6690 con 0x5624d0f2c2f0 2024-05-27T07:33:12.696 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:33:12.702+0000 7fe40dffb640 1 -- 172.21.15.52:0/1841269016 <== mon.0 v2:172.21.15.52:3300/0 3 ==== mon_get_version_reply(handle=1 version=102) ==== 24+0+0 (secure 0 0 0) 0x7fe410100a30 con 0x5624d0f2c2f0 2024-05-27T07:33:12.696 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:33:12.703+0000 7fe433b7d740 10 client.16800 fetch_fsmap learned FSMap version 102 2024-05-27T07:33:12.696 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:33:12.703+0000 7fe433b7d740 1 -- 172.21.15.52:0/1841269016 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({config=0+,fsmap=102,monmap=4+}) -- 0x5624d130f4c0 con 0x5624d0f2c2f0 2024-05-27T07:33:12.697 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:33:12.703+0000 7fe40dffb640 1 -- 172.21.15.52:0/1841269016 <== mon.0 v2:172.21.15.52:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fe4100ca130 con 0x5624d0f2c2f0 2024-05-27T07:33:12.697 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:33:12.703+0000 7fe40dffb640 1 -- 172.21.15.52:0/1841269016 <== mon.0 v2:172.21.15.52:3300/0 5 ==== fsmap(e 102) ==== 4211+0+0 (secure 0 0 0) 0x7fe4100c9390 con 0x5624d0f2c2f0 2024-05-27T07:33:12.697 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:33:12.703+0000 7fe433b7d740 10 client.16800 fetch_fsmap finished waiting for FSMap version 102 2024-05-27T07:33:12.697 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:33:12.703+0000 7fe433b7d740 10 client.16800 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-05-27T07:33:12.697 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:33:12.703+0000 7fe433b7d740 20 client.16800 populate_metadata read hostname 'smithi052' 2024-05-27T07:33:12.697 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:33:12.703+0000 7fe433b7d740 1 --2- 172.21.15.52:0/1841269016 >> [v2:172.21.15.105:6838/500348151,v1:172.21.15.105:6839/500348151] conn(0x5624d0f13d80 0x5624d1312be0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:33:12.697 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:33:12.703+0000 7fe433b7d740 4 client.16800 mds_command: new command op to 24527 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-27T07:33:12.697 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:33:12.703+0000 7fe433b7d740 1 -- 172.21.15.52:0/1841269016 --> [v2:172.21.15.105:6838/500348151,v1:172.21.15.105:6839/500348151] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5624d126ab80 con 0x5624d0f13d80 2024-05-27T07:33:12.699 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:33:12.705+0000 7fe425d9b640 1 --2- 172.21.15.52:0/1841269016 >> [v2:172.21.15.105:6838/500348151,v1:172.21.15.105:6839/500348151] conn(0x5624d0f13d80 0x5624d1312be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:33:12.703 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:33:12.709+0000 7fe425d9b640 1 --2- 172.21.15.52:0/1841269016 >> [v2:172.21.15.105:6838/500348151,v1:172.21.15.105:6839/500348151] conn(0x5624d0f13d80 0x5624d1312be0 crc :-1 s=READY pgs=135 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=887b2eaa7c7f1453 server_cookie=edd4ee15b56598ce in_seq=0 out_seq=0 2024-05-27T07:33:12.703 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:33:12.709+0000 7fe40dffb640 10 client.16800 ms_handle_connect on v2:172.21.15.105:6838/500348151 2024-05-27T07:33:12.704 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:33:12.710+0000 7fe40dffb640 1 -- 172.21.15.52:0/1841269016 <== mds.0 v2:172.21.15.105:6838/500348151 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24041 (crc 0 0 0) 0x5624d126ab80 con 0x5624d0f13d80 2024-05-27T07:33:12.704 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:33:12.710+0000 7fe40dffb640 10 client.16800 handle_command_reply: tid=0 2024-05-27T07:33:12.704 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:33:12.710+0000 7fe433b7d740 1 client.16800 shutdown 2024-05-27T07:33:12.705 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:33:12.712+0000 7fe433b7d740 1 -- 172.21.15.52:0/1841269016 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x5624d0f2c2f0 msgr2=0x5624d13009e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:33:12.705 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:33:12.712+0000 7fe433b7d740 1 --2- 172.21.15.52:0/1841269016 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x5624d0f2c2f0 0x5624d13009e0 secure :-1 s=READY pgs=684 cs=0 l=1 rev1=1 crypto rx=0x7fe4100c9e30 tx=0x7fe4100ce3a0 comp rx=0 tx=0).stop 2024-05-27T07:33:12.708 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:33:12.714+0000 7fe433b7d740 1 -- 172.21.15.52:0/1841269016 shutdown_connections 2024-05-27T07:33:12.708 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:33:12.714+0000 7fe433b7d740 1 --2- 172.21.15.52:0/1841269016 >> [v2:172.21.15.105:6838/500348151,v1:172.21.15.105:6839/500348151] conn(0x5624d0f13d80 0x5624d1312be0 crc :-1 s=READY pgs=135 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:33:12.708 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:33:12.714+0000 7fe433b7d740 1 --2- 172.21.15.52:0/1841269016 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5624d0f551b0 0x5624d130c600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:33:12.708 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:33:12.714+0000 7fe433b7d740 1 --2- 172.21.15.52:0/1841269016 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5624d0f344d0 0x5624d1300f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:33:12.708 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:33:12.714+0000 7fe433b7d740 1 --2- 172.21.15.52:0/1841269016 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x5624d0f2c2f0 0x5624d13009e0 unknown :-1 s=CLOSED pgs=684 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:33:12.708 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:33:12.714+0000 7fe433b7d740 1 -- 172.21.15.52:0/1841269016 >> 172.21.15.52:0/1841269016 conn(0x5624d0f49300 msgr2=0x5624d12f6200 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:33:12.708 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:33:12.714+0000 7fe40dffb640 0 client.16800 ms_handle_reset on v2:172.21.15.105:6838/500348151 2024-05-27T07:33:12.709 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:33:12.714+0000 7fe433b7d740 1 -- 172.21.15.52:0/1841269016 shutdown_connections 2024-05-27T07:33:12.709 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:33:12.714+0000 7fe433b7d740 1 -- 172.21.15.52:0/1841269016 wait complete. 2024-05-27T07:33:12.709 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:33:12.715+0000 7fe433b7d740 20 client.16800 trim_cache size 0 max 16384 2024-05-27T07:33:12.741 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:33:12.746+0000 7fe433b7d740 1 Processor -- start 2024-05-27T07:33:12.741 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:33:12.746+0000 7fe433b7d740 1 -- start start 2024-05-27T07:33:12.741 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:33:12.746+0000 7fe433b7d740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5624d0f2c2f0 0x5624d12f7170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:33:12.741 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:33:12.746+0000 7fe433b7d740 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x5624d0f344d0 0x5624d12f1b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:33:12.741 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:33:12.746+0000 7fe433b7d740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5624d0f551b0 0x5624d13183e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:33:12.741 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:33:12.746+0000 7fe433b7d740 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x7fe408010660 con 0x5624d0f344d0 2024-05-27T07:33:12.741 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:33:12.746+0000 7fe433b7d740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7fe40800ea30 con 0x5624d0f2c2f0 2024-05-27T07:33:12.741 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:33:12.747+0000 7fe433b7d740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7fe40800ff60 con 0x5624d0f551b0 2024-05-27T07:33:12.741 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:33:12.747+0000 7fe42659c640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5624d0f2c2f0 0x5624d12f7170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:33:12.742 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:33:12.747+0000 7fe425d9b640 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x5624d0f344d0 0x5624d12f1b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:33:12.742 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:33:12.747+0000 7fe425d9b640 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x5624d0f344d0 0x5624d12f1b80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.52:32972/0 (socket says 172.21.15.52:32972) 2024-05-27T07:33:12.742 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:33:12.747+0000 7fe425d9b640 1 -- 172.21.15.52:0/2752996497 learned_addr learned my addr 172.21.15.52:0/2752996497 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-05-27T07:33:12.742 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:33:12.747+0000 7fe426d9d640 1 --2- 172.21.15.52:0/2752996497 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5624d0f551b0 0x5624d13183e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:33:12.742 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:33:12.747+0000 7fe426d9d640 1 -- 172.21.15.52:0/2752996497 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5624d0f2c2f0 msgr2=0x5624d12f7170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:33:12.742 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:33:12.747+0000 7fe426d9d640 1 --2- 172.21.15.52:0/2752996497 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5624d0f2c2f0 0x5624d12f7170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:33:12.742 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:33:12.747+0000 7fe426d9d640 1 -- 172.21.15.52:0/2752996497 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x5624d0f344d0 msgr2=0x5624d12f1b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:33:12.742 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:33:12.747+0000 7fe426d9d640 1 --2- 172.21.15.52:0/2752996497 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x5624d0f344d0 0x5624d12f1b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:33:12.743 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:33:12.747+0000 7fe426d9d640 1 -- 172.21.15.52:0/2752996497 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5624d12f77b0 con 0x5624d0f551b0 2024-05-27T07:33:12.743 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:33:12.748+0000 7fe426d9d640 1 --2- 172.21.15.52:0/2752996497 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5624d0f551b0 0x5624d13183e0 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7fe420005fd0 tx=0x7fe42001e000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=670a8ea3300c560e server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:33:12.745 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:33:12.749+0000 7fe40ffff640 1 -- 172.21.15.52:0/2752996497 <== mon.1 v2:172.21.15.173:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe42001ca70 con 0x5624d0f551b0 2024-05-27T07:33:12.745 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:33:12.749+0000 7fe40ffff640 1 -- 172.21.15.52:0/2752996497 <== mon.1 v2:172.21.15.173:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fe420023070 con 0x5624d0f551b0 2024-05-27T07:33:12.745 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:33:12.749+0000 7fe40ffff640 1 -- 172.21.15.52:0/2752996497 <== mon.1 v2:172.21.15.173:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe4200049e0 con 0x5624d0f551b0 2024-05-27T07:33:12.745 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:33:12.750+0000 7fe433b7d740 1 -- 172.21.15.52:0/2752996497 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5624d0f551b0 msgr2=0x5624d13183e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:33:12.745 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:33:12.750+0000 7fe433b7d740 1 --2- 172.21.15.52:0/2752996497 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5624d0f551b0 0x5624d13183e0 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7fe420005fd0 tx=0x7fe42001e000 comp rx=0 tx=0).stop 2024-05-27T07:33:12.745 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:33:12.751+0000 7fe433b7d740 1 -- 172.21.15.52:0/2752996497 shutdown_connections 2024-05-27T07:33:12.746 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:33:12.751+0000 7fe433b7d740 1 --2- 172.21.15.52:0/2752996497 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5624d0f551b0 0x5624d13183e0 unknown :-1 s=CLOSED pgs=162 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:33:12.746 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:33:12.751+0000 7fe433b7d740 1 --2- 172.21.15.52:0/2752996497 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x5624d0f344d0 0x5624d12f1b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:33:12.746 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:33:12.751+0000 7fe433b7d740 1 --2- 172.21.15.52:0/2752996497 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5624d0f2c2f0 0x5624d12f7170 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:33:12.746 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:33:12.751+0000 7fe433b7d740 1 -- 172.21.15.52:0/2752996497 >> 172.21.15.52:0/2752996497 conn(0x5624d0f49300 msgr2=0x5624d13029c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:33:12.746 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:33:12.753+0000 7fe433b7d740 1 -- 172.21.15.52:0/2752996497 shutdown_connections 2024-05-27T07:33:12.748 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:33:12.754+0000 7fe433b7d740 1 -- 172.21.15.52:0/2752996497 wait complete. 2024-05-27T07:33:12.749 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:33:12.755+0000 7fe433b7d740 1 -- start start 2024-05-27T07:33:12.749 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:33:12.755+0000 7fe433b7d740 1 Processor -- start 2024-05-27T07:33:12.750 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:33:12.756+0000 7fe433b7d740 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x5624d0f2c2f0 0x5624d1303d70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:33:12.750 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:33:12.756+0000 7fe433b7d740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5624d0f344d0 0x5624d13042b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:33:12.751 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:33:12.756+0000 7fe433b7d740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5624d0f551b0 0x5624d1373270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:33:12.751 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:33:12.757+0000 7fe425d9b640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5624d0f344d0 0x5624d13042b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:33:12.751 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:33:12.757+0000 7fe425d9b640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5624d0f344d0 0x5624d13042b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.52:35732/0 (socket says 172.21.15.52:35732) 2024-05-27T07:33:12.751 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:33:12.757+0000 7fe425d9b640 1 -- 172.21.15.52:0/3401293905 learned_addr learned my addr 172.21.15.52:0/3401293905 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-05-27T07:33:12.751 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:33:12.757+0000 7fe426d9d640 1 --2- 172.21.15.52:0/3401293905 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5624d0f551b0 0x5624d1373270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:33:12.751 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:33:12.757+0000 7fe433b7d740 1 -- 172.21.15.52:0/3401293905 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x7fe40800f7d0 con 0x5624d0f2c2f0 2024-05-27T07:33:12.751 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:33:12.757+0000 7fe433b7d740 1 -- 172.21.15.52:0/3401293905 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7fe40800f100 con 0x5624d0f344d0 2024-05-27T07:33:12.751 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:33:12.757+0000 7fe433b7d740 1 -- 172.21.15.52:0/3401293905 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x5624d126ab80 con 0x5624d0f551b0 2024-05-27T07:33:12.752 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:33:12.758+0000 7fe433b7d740 10 client.0 _ll_register_callbacks cb 0x7fe4100c9a80 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T07:33:12.752 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:33:12.758+0000 7fe426d9d640 1 -- 172.21.15.52:0/3401293905 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5624d0f344d0 msgr2=0x5624d13042b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:33:12.752 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:33:12.758+0000 7fe426d9d640 1 --2- 172.21.15.52:0/3401293905 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5624d0f344d0 0x5624d13042b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:33:12.752 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:33:12.758+0000 7fe426d9d640 1 -- 172.21.15.52:0/3401293905 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x5624d0f2c2f0 msgr2=0x5624d1303d70 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-05-27T07:33:12.752 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:33:12.758+0000 7fe426d9d640 1 --2- 172.21.15.52:0/3401293905 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x5624d0f2c2f0 0x5624d1303d70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:33:12.752 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:33:12.758+0000 7fe426d9d640 1 -- 172.21.15.52:0/3401293905 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5624d1373830 con 0x5624d0f551b0 2024-05-27T07:33:12.753 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:33:12.758+0000 7fe426d9d640 1 --2- 172.21.15.52:0/3401293905 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5624d0f551b0 0x5624d1373270 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7fe420025000 tx=0x7fe420005e90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=86293419dd947107 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:33:12.753 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:33:12.758+0000 7fe40dffb640 10 client.26302 ms_handle_connect on v2:172.21.15.173:3300/0 2024-05-27T07:33:12.753 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:33:12.758+0000 7fe433b7d740 1 -- 172.21.15.52:0/3401293905 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fe420023070 con 0x5624d0f551b0 2024-05-27T07:33:12.753 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:33:12.760+0000 7fe40dffb640 1 -- 172.21.15.52:0/3401293905 <== mon.1 v2:172.21.15.173:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe420027820 con 0x5624d0f551b0 2024-05-27T07:33:12.754 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:33:12.760+0000 7fe40dffb640 1 -- 172.21.15.52:0/3401293905 <== mon.1 v2:172.21.15.173:3300/0 2 ==== osd_map(117..117 src has 1..117) ==== 9555+0+0 (secure 0 0 0) 0x7fe4200279c0 con 0x5624d0f551b0 2024-05-27T07:33:12.754 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:33:12.760+0000 7fe40dffb640 1 -- 172.21.15.52:0/3401293905 <== mon.1 v2:172.21.15.173:3300/0 3 ==== mon_get_version_reply(handle=1 version=102) ==== 24+0+0 (secure 0 0 0) 0x7fe420037050 con 0x5624d0f551b0 2024-05-27T07:33:12.754 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:33:12.760+0000 7fe433b7d740 10 client.26302 fetch_fsmap learned FSMap version 102 2024-05-27T07:33:12.754 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:33:12.760+0000 7fe433b7d740 1 -- 172.21.15.52:0/3401293905 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({fsmap=102}) -- 0x5624d137aa80 con 0x5624d0f551b0 2024-05-27T07:33:12.755 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:33:12.762+0000 7fe40dffb640 1 -- 172.21.15.52:0/3401293905 <== mon.1 v2:172.21.15.173:3300/0 4 ==== fsmap(e 102) ==== 4211+0+0 (secure 0 0 0) 0x7fe42001e070 con 0x5624d0f551b0 2024-05-27T07:33:12.756 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:33:12.763+0000 7fe433b7d740 10 client.26302 fetch_fsmap finished waiting for FSMap version 102 2024-05-27T07:33:12.756 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:33:12.763+0000 7fe433b7d740 10 client.26302 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-05-27T07:33:12.756 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:33:12.763+0000 7fe433b7d740 20 client.26302 populate_metadata read hostname 'smithi052' 2024-05-27T07:33:12.756 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:33:12.763+0000 7fe433b7d740 1 --2- 172.21.15.52:0/3401293905 >> [v2:172.21.15.105:6838/500348151,v1:172.21.15.105:6839/500348151] conn(0x5624d0ee91c0 0x5624d137d150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:33:12.757 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:33:12.763+0000 7fe433b7d740 4 client.26302 mds_command: new command op to 24527 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-05-27T07:33:12.757 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:33:12.763+0000 7fe433b7d740 1 -- 172.21.15.52:0/3401293905 --> [v2:172.21.15.105:6838/500348151,v1:172.21.15.105:6839/500348151] -- command(tid 0: {"prefix": "flush journal"}) -- 0x7fe420023070 con 0x5624d0ee91c0 2024-05-27T07:33:12.760 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:33:12.766+0000 7fe42659c640 1 --2- 172.21.15.52:0/3401293905 >> [v2:172.21.15.105:6838/500348151,v1:172.21.15.105:6839/500348151] conn(0x5624d0ee91c0 0x5624d137d150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:33:12.774 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:33:12.778+0000 7fe42659c640 1 --2- 172.21.15.52:0/3401293905 >> [v2:172.21.15.105:6838/500348151,v1:172.21.15.105:6839/500348151] conn(0x5624d0ee91c0 0x5624d137d150 crc :-1 s=READY pgs=136 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=e0bfe256284d2788 server_cookie=ba36b5aad2c9d0a1 in_seq=0 out_seq=0 2024-05-27T07:33:12.774 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:33:12.778+0000 7fe40dffb640 10 client.26302 ms_handle_connect on v2:172.21.15.105:6838/500348151 2024-05-27T07:33:14.371 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:33:14.376+0000 7fe40dffb640 1 -- 172.21.15.52:0/3401293905 <== mds.0 v2:172.21.15.105:6838/500348151 1 ==== command_reply(tid 0: 0 ) ==== 8+0+25 (crc 0 0 0) 0x7fe420023070 con 0x5624d0ee91c0 2024-05-27T07:33:14.371 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:33:14.376+0000 7fe40dffb640 10 client.26302 handle_command_reply: tid=0 2024-05-27T07:33:14.371 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:33:14.376+0000 7fe433b7d740 1 client.26302 shutdown 2024-05-27T07:33:14.371 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:33:14.376+0000 7fe433b7d740 1 -- 172.21.15.52:0/3401293905 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5624d0f551b0 msgr2=0x5624d1373270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:33:14.371 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:33:14.376+0000 7fe433b7d740 1 --2- 172.21.15.52:0/3401293905 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5624d0f551b0 0x5624d1373270 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7fe420025000 tx=0x7fe420005e90 comp rx=0 tx=0).stop 2024-05-27T07:33:14.371 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:33:14.377+0000 7fe433b7d740 1 -- 172.21.15.52:0/3401293905 shutdown_connections 2024-05-27T07:33:14.371 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:33:14.377+0000 7fe433b7d740 1 --2- 172.21.15.52:0/3401293905 >> [v2:172.21.15.105:6838/500348151,v1:172.21.15.105:6839/500348151] conn(0x5624d0ee91c0 0x5624d137d150 crc :-1 s=READY pgs=136 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:33:14.371 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:33:14.377+0000 7fe40dffb640 0 client.26302 ms_handle_reset on v2:172.21.15.105:6838/500348151 2024-05-27T07:33:14.372 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:33:14.377+0000 7fe433b7d740 1 --2- 172.21.15.52:0/3401293905 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5624d0f551b0 0x5624d1373270 unknown :-1 s=CLOSED pgs=163 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:33:14.372 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:33:14.377+0000 7fe433b7d740 1 --2- 172.21.15.52:0/3401293905 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5624d0f344d0 0x5624d13042b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:33:14.372 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:33:14.377+0000 7fe433b7d740 1 --2- 172.21.15.52:0/3401293905 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x5624d0f2c2f0 0x5624d1303d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:33:14.372 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:33:14.377+0000 7fe433b7d740 1 -- 172.21.15.52:0/3401293905 >> 172.21.15.52:0/3401293905 conn(0x5624d0f49300 msgr2=0x5624d12f1b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:33:14.372 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:33:14.377+0000 7fe433b7d740 1 -- 172.21.15.52:0/3401293905 shutdown_connections 2024-05-27T07:33:14.372 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:33:14.377+0000 7fe433b7d740 1 -- 172.21.15.52:0/3401293905 wait complete. 2024-05-27T07:33:14.372 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:33:14.378+0000 7fe433b7d740 20 client.26302 trim_cache size 0 max 16384 2024-05-27T07:33:14.373 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stdout:{ 2024-05-27T07:33:14.373 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stdout: "return_code": 0 2024-05-27T07:33:14.373 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stdout:} 2024-05-27T07:33:14.373 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:33:14.379+0000 7fe42cbf3640 1 -- 172.21.15.52:0/1867373087 >> [v2:172.21.15.52:6800/1822318384,v1:172.21.15.52:6801/1822318384] conn(0x7fe4140b2b60 msgr2=0x7fe4140b5070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:33:14.374 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:33:14.379+0000 7fe42cbf3640 1 --2- 172.21.15.52:0/1867373087 >> [v2:172.21.15.52:6800/1822318384,v1:172.21.15.52:6801/1822318384] conn(0x7fe4140b2b60 0x7fe4140b5070 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7fe420012780 tx=0x7fe42000c5a0 comp rx=0 tx=0).stop 2024-05-27T07:33:14.374 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:33:14.379+0000 7fe42cbf3640 1 -- 172.21.15.52:0/1867373087 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fe42810fb80 msgr2=0x7fe42811b1e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:33:14.374 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:33:14.379+0000 7fe42cbf3640 1 --2- 172.21.15.52:0/1867373087 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fe42810fb80 0x7fe42811b1e0 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7fe41002a9f0 tx=0x7fe410030640 comp rx=0 tx=0).stop 2024-05-27T07:33:14.374 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:33:14.379+0000 7fe42cbf3640 1 -- 172.21.15.52:0/1867373087 shutdown_connections 2024-05-27T07:33:14.374 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:33:14.379+0000 7fe42cbf3640 1 --2- 172.21.15.52:0/1867373087 >> [v2:172.21.15.52:6800/1822318384,v1:172.21.15.52:6801/1822318384] conn(0x7fe4140b2b60 0x7fe4140b5070 unknown :-1 s=CLOSED pgs=160 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:33:14.374 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:33:14.379+0000 7fe42cbf3640 1 --2- 172.21.15.52:0/1867373087 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fe428111270 0x7fe42811bc60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:33:14.374 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:33:14.379+0000 7fe42cbf3640 1 --2- 172.21.15.52:0/1867373087 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fe428110640 0x7fe42811b720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:33:14.374 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:33:14.379+0000 7fe42cbf3640 1 --2- 172.21.15.52:0/1867373087 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fe42810fb80 0x7fe42811b1e0 unknown :-1 s=CLOSED pgs=161 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:33:14.374 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:33:14.379+0000 7fe42cbf3640 1 -- 172.21.15.52:0/1867373087 >> 172.21.15.52:0/1867373087 conn(0x7fe42806e8f0 msgr2=0x7fe428111d90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:33:14.374 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:33:14.379+0000 7fe42cbf3640 1 -- 172.21.15.52:0/1867373087 shutdown_connections 2024-05-27T07:33:14.374 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:33:14.379+0000 7fe42cbf3640 1 -- 172.21.15.52:0/1867373087 wait complete. 2024-05-27T07:33:14.445 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:33:14 smithi105 ceph-mon[35977]: pgmap v247: 129 pgs: 19 active+clean+snaptrim, 61 active+clean, 49 active+clean+snaptrim_wait; 663 MiB data, 3.7 GiB used, 1.0 TiB / 1.0 TiB avail; 865 KiB/s rd, 1.5 MiB/s wr, 1.08k op/s 2024-05-27T07:33:14.483 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:33:14 smithi052 ceph-mon[31955]: pgmap v247: 129 pgs: 19 active+clean+snaptrim, 61 active+clean, 49 active+clean+snaptrim_wait; 663 MiB data, 3.7 GiB used, 1.0 TiB / 1.0 TiB avail; 865 KiB/s rd, 1.5 MiB/s wr, 1.08k op/s 2024-05-27T07:33:14.605 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:33:14 smithi173 ceph-mon[37000]: pgmap v247: 129 pgs: 19 active+clean+snaptrim, 61 active+clean, 49 active+clean+snaptrim_wait; 663 MiB data, 3.7 GiB used, 1.0 TiB / 1.0 TiB avail; 865 KiB/s rd, 1.5 MiB/s wr, 1.08k op/s 2024-05-27T07:33:16.444 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:33:16 smithi105 ceph-mon[35977]: pgmap v248: 129 pgs: 19 active+clean+snaptrim, 61 active+clean, 49 active+clean+snaptrim_wait; 665 MiB data, 3.7 GiB used, 1.0 TiB / 1.0 TiB avail; 666 KiB/s rd, 1.2 MiB/s wr, 946 op/s 2024-05-27T07:33:16.483 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:33:16 smithi052 ceph-mon[31955]: pgmap v248: 129 pgs: 19 active+clean+snaptrim, 61 active+clean, 49 active+clean+snaptrim_wait; 665 MiB data, 3.7 GiB used, 1.0 TiB / 1.0 TiB avail; 666 KiB/s rd, 1.2 MiB/s wr, 946 op/s 2024-05-27T07:33:16.605 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:33:16 smithi173 ceph-mon[37000]: pgmap v248: 129 pgs: 19 active+clean+snaptrim, 61 active+clean, 49 active+clean+snaptrim_wait; 665 MiB data, 3.7 GiB used, 1.0 TiB / 1.0 TiB avail; 666 KiB/s rd, 1.2 MiB/s wr, 946 op/s 2024-05-27T07:33:18.397 DEBUG:teuthology.orchestra.run.smithi052:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:38f9d900c83c63799fbdbe61acc9a11b0d3554a6 shell --fsid 73df2636-1bf8-11ef-bc9b-c7b262605968 -- ceph tell mds.1:0 scrub status 2024-05-27T07:33:18.483 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:33:18 smithi052 ceph-mon[31955]: pgmap v249: 129 pgs: 17 active+clean+snaptrim, 69 active+clean, 43 active+clean+snaptrim_wait; 663 MiB data, 3.8 GiB used, 1.0 TiB / 1.0 TiB avail; 445 KiB/s rd, 1.2 MiB/s wr, 902 op/s 2024-05-27T07:33:18.605 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:33:18 smithi173 ceph-mon[37000]: pgmap v249: 129 pgs: 17 active+clean+snaptrim, 69 active+clean, 43 active+clean+snaptrim_wait; 663 MiB data, 3.8 GiB used, 1.0 TiB / 1.0 TiB avail; 445 KiB/s rd, 1.2 MiB/s wr, 902 op/s 2024-05-27T07:33:18.693 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:33:18 smithi105 ceph-mon[35977]: pgmap v249: 129 pgs: 17 active+clean+snaptrim, 69 active+clean, 43 active+clean+snaptrim_wait; 663 MiB data, 3.8 GiB used, 1.0 TiB / 1.0 TiB avail; 445 KiB/s rd, 1.2 MiB/s wr, 902 op/s 2024-05-27T07:33:18.867 INFO:teuthology.orchestra.run.smithi052.stderr:Inferring config /var/lib/ceph/73df2636-1bf8-11ef-bc9b-c7b262605968/mon.a/config 2024-05-27T07:33:20.455 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:33:20 smithi052 ceph-mon[31955]: pgmap v250: 129 pgs: 15 active+clean+snaptrim, 73 active+clean, 41 active+clean+snaptrim_wait; 648 MiB data, 3.8 GiB used, 1.0 TiB / 1.0 TiB avail; 83 KiB/s rd, 1.2 MiB/s wr, 622 op/s 2024-05-27T07:33:20.456 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:33:20 smithi052 ceph-mon[31955]: fsmap cephfs:5 5 up:active 2024-05-27T07:33:20.606 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:33:20 smithi173 ceph-mon[37000]: pgmap v250: 129 pgs: 15 active+clean+snaptrim, 73 active+clean, 41 active+clean+snaptrim_wait; 648 MiB data, 3.8 GiB used, 1.0 TiB / 1.0 TiB avail; 83 KiB/s rd, 1.2 MiB/s wr, 622 op/s 2024-05-27T07:33:20.606 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:33:20 smithi173 ceph-mon[37000]: fsmap cephfs:5 5 up:active 2024-05-27T07:33:20.694 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:33:20 smithi105 ceph-mon[35977]: pgmap v250: 129 pgs: 15 active+clean+snaptrim, 73 active+clean, 41 active+clean+snaptrim_wait; 648 MiB data, 3.8 GiB used, 1.0 TiB / 1.0 TiB avail; 83 KiB/s rd, 1.2 MiB/s wr, 622 op/s 2024-05-27T07:33:20.694 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:33:20 smithi105 ceph-mon[35977]: fsmap cephfs:5 5 up:active 2024-05-27T07:33:21.618 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:21.621+0000 7f51a2d6b700 1 -- 172.21.15.52:0/2528173767 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f519c10a920 msgr2=0x7f519c10ad30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:33:21.619 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:21.621+0000 7f51a2d6b700 1 --2- 172.21.15.52:0/2528173767 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f519c10a920 0x7f519c10ad30 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7f5194008980 tx=0x7f5194008c90 comp rx=0 tx=0).stop 2024-05-27T07:33:21.619 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:21.621+0000 7f51a2d6b700 1 -- 172.21.15.52:0/2528173767 shutdown_connections 2024-05-27T07:33:21.619 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:21.621+0000 7f51a2d6b700 1 --2- 172.21.15.52:0/2528173767 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f519c10b270 0x7f519c1198a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:33:21.619 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:21.621+0000 7f51a2d6b700 1 --2- 172.21.15.52:0/2528173767 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f519c111bc0 0x7f519c111fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:33:21.620 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:21.621+0000 7f51a2d6b700 1 --2- 172.21.15.52:0/2528173767 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f519c10a920 0x7f519c10ad30 unknown :-1 s=CLOSED pgs=164 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:33:21.620 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:21.621+0000 7f51a2d6b700 1 -- 172.21.15.52:0/2528173767 >> 172.21.15.52:0/2528173767 conn(0x7f519c06f4c0 msgr2=0x7f519c06f900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:33:21.620 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:21.621+0000 7f51a2d6b700 1 -- 172.21.15.52:0/2528173767 shutdown_connections 2024-05-27T07:33:21.620 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:21.621+0000 7f51a2d6b700 1 -- 172.21.15.52:0/2528173767 wait complete. 2024-05-27T07:33:21.620 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:21.622+0000 7f51a2d6b700 1 Processor -- start 2024-05-27T07:33:21.620 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:21.623+0000 7f51a2d6b700 1 -- start start 2024-05-27T07:33:21.621 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:21.623+0000 7f51a2d6b700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f519c10a920 0x7f519c1b2400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:33:21.621 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:21.623+0000 7f51a2d6b700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f519c10b270 0x7f519c1b29a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:33:21.621 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:21.623+0000 7f51a2d6b700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f519c111bc0 0x7f519c1b6c30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:33:21.621 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:21.623+0000 7f51a2d6b700 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x7f519c12bb70 con 0x7f519c10a920 2024-05-27T07:33:21.621 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:21.623+0000 7f51a2d6b700 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f519c12b9f0 con 0x7f519c111bc0 2024-05-27T07:33:21.622 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:21.623+0000 7f51a2d6b700 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f519c12bcf0 con 0x7f519c10b270 2024-05-27T07:33:21.622 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:21.624+0000 7f51a1308700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f519c111bc0 0x7f519c1b6c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:33:21.622 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:21.624+0000 7f51a1308700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f519c111bc0 0x7f519c1b6c30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.52:54218/0 (socket says 172.21.15.52:54218) 2024-05-27T07:33:21.622 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:21.624+0000 7f51a1308700 1 -- 172.21.15.52:0/955600229 learned_addr learned my addr 172.21.15.52:0/955600229 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-05-27T07:33:21.622 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:21.624+0000 7f51a0b07700 1 --2- 172.21.15.52:0/955600229 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f519c10a920 0x7f519c1b2400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:33:21.624 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:21.625+0000 7f51a0b07700 1 -- 172.21.15.52:0/955600229 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f519c10b270 msgr2=0x7f519c1b29a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:33:21.624 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:21.625+0000 7f51a0b07700 1 --2- 172.21.15.52:0/955600229 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f519c10b270 0x7f519c1b29a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:33:21.625 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:21.625+0000 7f51a0b07700 1 -- 172.21.15.52:0/955600229 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f519c111bc0 msgr2=0x7f519c1b6c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:33:21.625 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:21.625+0000 7f51a0b07700 1 --2- 172.21.15.52:0/955600229 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f519c111bc0 0x7f519c1b6c30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:33:21.625 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:21.625+0000 7f51a0b07700 1 -- 172.21.15.52:0/955600229 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f519c1b72d0 con 0x7f519c10a920 2024-05-27T07:33:21.625 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:21.627+0000 7f51a0b07700 1 --2- 172.21.15.52:0/955600229 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f519c10a920 0x7f519c1b2400 secure :-1 s=READY pgs=685 cs=0 l=1 rev1=1 crypto rx=0x7f519000d740 tx=0x7f519000db00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:33:21.625 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:21.628+0000 7f5199ffb700 1 -- 172.21.15.52:0/955600229 <== mon.0 v2:172.21.15.52:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5190017070 con 0x7f519c10a920 2024-05-27T07:33:21.625 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:21.630+0000 7f51a2d6b700 1 -- 172.21.15.52:0/955600229 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f519c1b75c0 con 0x7f519c10a920 2024-05-27T07:33:21.626 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:21.630+0000 7f51a2d6b700 1 -- 172.21.15.52:0/955600229 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f519c1beee0 con 0x7f519c10a920 2024-05-27T07:33:21.627 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:21.631+0000 7f5199ffb700 1 -- 172.21.15.52:0/955600229 <== mon.0 v2:172.21.15.52:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f51900096d0 con 0x7f519c10a920 2024-05-27T07:33:21.627 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:21.632+0000 7f5199ffb700 1 -- 172.21.15.52:0/955600229 <== mon.0 v2:172.21.15.52:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5190003b40 con 0x7f519c10a920 2024-05-27T07:33:21.630 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:21.636+0000 7f51b1780b80 1 Processor -- start 2024-05-27T07:33:21.631 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:21.637+0000 7f5199ffb700 1 -- 172.21.15.52:0/955600229 <== mon.0 v2:172.21.15.52:3300/0 4 ==== mgrmap(e 26) ==== 149783+0+0 (secure 0 0 0) 0x7f5190003ce0 con 0x7f519c10a920 2024-05-27T07:33:21.631 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:21.637+0000 7f51b1780b80 1 -- start start 2024-05-27T07:33:21.634 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:21.640+0000 7f51b1780b80 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55b64fecbe40 0x55b65004f710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:33:21.634 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:21.640+0000 7f51b1780b80 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55b65004fc50 0x55b650049a40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:33:21.634 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:21.640+0000 7f51b1780b80 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55b650049f80 0x55b65004a390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:33:21.634 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:21.640+0000 7f51b1780b80 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x55b64ff78c90 con 0x55b650049f80 2024-05-27T07:33:21.634 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:21.640+0000 7f51b1780b80 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x55b64ff68830 con 0x55b65004fc50 2024-05-27T07:33:21.634 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:21.640+0000 7f51b1780b80 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x55b64ff6c870 con 0x55b64fecbe40 2024-05-27T07:33:21.635 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:21.640+0000 7f51a0b07700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55b65004fc50 0x55b650049a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:33:21.635 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:21.640+0000 7f5199ffb700 1 --2- 172.21.15.52:0/955600229 >> [v2:172.21.15.52:6800/1822318384,v1:172.21.15.52:6801/1822318384] conn(0x7f51840b2aa0 0x7f51840b4fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:33:21.636 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:21.640+0000 7f51a0b07700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55b65004fc50 0x55b650049a40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.52:54224/0 (socket says 172.21.15.52:54224) 2024-05-27T07:33:21.636 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:21.640+0000 7f51a0b07700 1 -- 172.21.15.52:0/2774713930 learned_addr learned my addr 172.21.15.52:0/2774713930 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-05-27T07:33:21.636 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:21.641+0000 7f51a1308700 1 --2- 172.21.15.52:0/2774713930 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55b64fecbe40 0x55b65004f710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:33:21.636 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:21.641+0000 7f5199ffb700 1 -- 172.21.15.52:0/955600229 <== mon.0 v2:172.21.15.52:3300/0 5 ==== osd_map(117..117 src has 1..117) ==== 9555+0+0 (secure 0 0 0) 0x7f5190004280 con 0x7f519c10a920 2024-05-27T07:33:21.636 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:21.641+0000 7f51a1308700 1 --2- 172.21.15.52:0/955600229 >> [v2:172.21.15.52:6800/1822318384,v1:172.21.15.52:6801/1822318384] conn(0x7f51840b2aa0 0x7f51840b4fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:33:21.636 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:21.641+0000 7f51a1308700 1 -- 172.21.15.52:0/2774713930 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55b65004fc50 msgr2=0x55b650049a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:33:21.636 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:21.641+0000 7f51a1308700 1 --2- 172.21.15.52:0/2774713930 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55b65004fc50 0x55b650049a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:33:21.636 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:21.641+0000 7f51a1308700 1 -- 172.21.15.52:0/2774713930 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55b650049f80 msgr2=0x55b65004a390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:33:21.636 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:21.641+0000 7f51a1308700 1 --2- 172.21.15.52:0/2774713930 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55b650049f80 0x55b65004a390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:33:21.636 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:21.641+0000 7f51a1308700 1 -- 172.21.15.52:0/2774713930 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55b64feb88d0 con 0x55b64fecbe40 2024-05-27T07:33:21.637 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:21.642+0000 7f51a0b07700 1 --2- 172.21.15.52:0/2774713930 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55b65004fc50 0x55b650049a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-27T07:33:21.637 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:21.642+0000 7f51a1308700 1 --2- 172.21.15.52:0/2774713930 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55b64fecbe40 0x55b65004f710 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7f518c013850 tx=0x7f518c013b60 comp rx=0 tx=0).ready entity=mon.1 client_cookie=f3d88080c65af48d server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:33:21.637 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:21.643+0000 7f51a1308700 1 --2- 172.21.15.52:0/955600229 >> [v2:172.21.15.52:6800/1822318384,v1:172.21.15.52:6801/1822318384] conn(0x7f51840b2aa0 0x7f51840b4fb0 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7f518c009fd0 tx=0x7f518c020000 comp rx=0 tx=0).ready entity=mgr.15333 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:33:21.637 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:21.643+0000 7f5182ffd700 1 -- 172.21.15.52:0/2774713930 <== mon.1 v2:172.21.15.173:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f518c00fde0 con 0x55b64fecbe40 2024-05-27T07:33:21.637 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:21.643+0000 7f5182ffd700 1 -- 172.21.15.52:0/2774713930 <== mon.1 v2:172.21.15.173:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f518c023020 con 0x55b64fecbe40 2024-05-27T07:33:21.637 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:21.643+0000 7f5182ffd700 1 -- 172.21.15.52:0/2774713930 <== mon.1 v2:172.21.15.173:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f518c01fda0 con 0x55b64fecbe40 2024-05-27T07:33:21.640 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:21.646+0000 7f51b1780b80 1 -- 172.21.15.52:0/2774713930 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55b64fecbe40 msgr2=0x55b65004f710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:33:21.640 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:21.646+0000 7f51b1780b80 1 --2- 172.21.15.52:0/2774713930 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55b64fecbe40 0x55b65004f710 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7f518c013850 tx=0x7f518c013b60 comp rx=0 tx=0).stop 2024-05-27T07:33:21.641 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:21.647+0000 7f51b1780b80 1 -- 172.21.15.52:0/2774713930 shutdown_connections 2024-05-27T07:33:21.641 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:21.647+0000 7f51b1780b80 1 --2- 172.21.15.52:0/2774713930 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55b650049f80 0x55b65004a390 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:33:21.641 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:21.647+0000 7f51b1780b80 1 --2- 172.21.15.52:0/2774713930 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55b64fecbe40 0x55b65004f710 unknown :-1 s=CLOSED pgs=165 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:33:21.641 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:21.647+0000 7f51b1780b80 1 --2- 172.21.15.52:0/2774713930 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55b65004fc50 0x55b650049a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:33:21.641 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:21.647+0000 7f51b1780b80 1 -- 172.21.15.52:0/2774713930 >> 172.21.15.52:0/2774713930 conn(0x55b64ff26af0 msgr2=0x55b650044640 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:33:21.643 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:21.650+0000 7f51b1780b80 1 -- 172.21.15.52:0/2774713930 shutdown_connections 2024-05-27T07:33:21.646 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:21.652+0000 7f51b1780b80 1 -- 172.21.15.52:0/2774713930 wait complete. 2024-05-27T07:33:21.646 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:21.652+0000 7f51b1780b80 1 -- start start 2024-05-27T07:33:21.646 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:21.652+0000 7f51b1780b80 1 Processor -- start 2024-05-27T07:33:21.649 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:21.655+0000 7f51b1780b80 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55b64fecbe40 0x55b6500532e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:33:21.650 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:21.655+0000 7f51b1780b80 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55b650049f80 0x55b650053820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:33:21.650 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:21.655+0000 7f51b1780b80 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55b65004fc50 0x55b650053d60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:33:21.650 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:21.655+0000 7f51b1780b80 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x55b64fe6efa0 con 0x55b65004fc50 2024-05-27T07:33:21.650 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:21.655+0000 7f51b1780b80 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x55b64ff0a7c0 con 0x55b650049f80 2024-05-27T07:33:21.650 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:21.655+0000 7f51b1780b80 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x55b6500566e0 con 0x55b64fecbe40 2024-05-27T07:33:21.650 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:21.656+0000 7f51b1780b80 10 client.0 _ll_register_callbacks cb 0x55b64ff03920 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T07:33:21.650 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:21.657+0000 7f519bfff700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55b650049f80 0x55b650053820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:33:21.651 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:21.657+0000 7f51a0b07700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55b64fecbe40 0x55b6500532e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:33:21.651 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:21.657+0000 7f51a0b07700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55b64fecbe40 0x55b6500532e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.173:3300/0 says I am v2:172.21.15.52:54356/0 (socket says 172.21.15.52:54356) 2024-05-27T07:33:21.651 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:21.657+0000 7f51a0b07700 1 -- 172.21.15.52:0/1642907313 learned_addr learned my addr 172.21.15.52:0/1642907313 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-05-27T07:33:21.652 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:21.658+0000 7f51a0b07700 1 -- 172.21.15.52:0/1642907313 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55b650049f80 msgr2=0x55b650053820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:33:21.652 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:21.658+0000 7f51a0b07700 1 --2- 172.21.15.52:0/1642907313 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55b650049f80 0x55b650053820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:33:21.652 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:21.658+0000 7f51a0b07700 1 -- 172.21.15.52:0/1642907313 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55b65004fc50 msgr2=0x55b650053d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:33:21.652 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:21.658+0000 7f51a0b07700 1 --2- 172.21.15.52:0/1642907313 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55b65004fc50 0x55b650053d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:33:21.653 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:21.658+0000 7f51a0b07700 1 -- 172.21.15.52:0/1642907313 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55b64ff07690 con 0x55b64fecbe40 2024-05-27T07:33:21.653 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:21.658+0000 7f51a0b07700 1 --2- 172.21.15.52:0/1642907313 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55b64fecbe40 0x55b6500532e0 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7f51900a6ae0 tx=0x7f51900a96f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=94571f14f6191ec6 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:33:21.653 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:21.659+0000 7f517bfff700 10 client.0 ms_handle_connect on v2:172.21.15.173:3300/0 2024-05-27T07:33:21.654 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:21.659+0000 7f517bfff700 1 -- 172.21.15.52:0/1642907313 <== mon.1 v2:172.21.15.173:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5190020070 con 0x55b64fecbe40 2024-05-27T07:33:21.655 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:21.659+0000 7f517bfff700 1 -- 172.21.15.52:0/1642907313 <== mon.1 v2:172.21.15.173:3300/0 2 ==== osd_map(117..117 src has 1..117) ==== 9555+0+0 (secure 0 0 0) 0x7f51900a9ad0 con 0x55b64fecbe40 2024-05-27T07:33:21.656 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:21.663+0000 7f51b1780b80 1 -- 172.21.15.52:0/1642907313 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f518c023020 con 0x55b64fecbe40 2024-05-27T07:33:21.657 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:21.663+0000 7f517bfff700 1 -- 172.21.15.52:0/1642907313 <== mon.1 v2:172.21.15.173:3300/0 3 ==== mon_get_version_reply(handle=1 version=103) ==== 24+0+0 (secure 0 0 0) 0x7f51900a9380 con 0x55b64fecbe40 2024-05-27T07:33:21.659 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:21.665+0000 7f51b1780b80 10 client.26311 fetch_fsmap learned FSMap version 103 2024-05-27T07:33:21.659 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:21.665+0000 7f51b1780b80 1 -- 172.21.15.52:0/1642907313 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({config=0+,fsmap=103,monmap=4+}) -- 0x55b650061170 con 0x55b64fecbe40 2024-05-27T07:33:21.660 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:21.666+0000 7f517bfff700 1 -- 172.21.15.52:0/1642907313 <== mon.1 v2:172.21.15.173:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f518c023020 con 0x55b64fecbe40 2024-05-27T07:33:21.660 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:21.666+0000 7f517bfff700 1 -- 172.21.15.52:0/1642907313 <== mon.1 v2:172.21.15.173:3300/0 5 ==== fsmap(e 103) ==== 4215+0+0 (secure 0 0 0) 0x7f51900abc80 con 0x55b64fecbe40 2024-05-27T07:33:21.661 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:21.666+0000 7f51b1780b80 10 client.26311 fetch_fsmap finished waiting for FSMap version 103 2024-05-27T07:33:21.661 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:21.666+0000 7f51b1780b80 10 client.26311 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-05-27T07:33:21.661 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:21.666+0000 7f51b1780b80 20 client.26311 populate_metadata read hostname 'smithi052' 2024-05-27T07:33:21.661 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:21.666+0000 7f51b1780b80 1 --2- 172.21.15.52:0/1642907313 >> [v2:172.21.15.105:6838/500348151,v1:172.21.15.105:6839/500348151] conn(0x55b650063c60 0x55b650066130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:33:21.662 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:21.666+0000 7f51b1780b80 4 client.26311 mds_command: new command op to 24527 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-27T07:33:21.662 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:21.666+0000 7f51b1780b80 1 -- 172.21.15.52:0/1642907313 --> [v2:172.21.15.105:6838/500348151,v1:172.21.15.105:6839/500348151] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55b6500159f0 con 0x55b650063c60 2024-05-27T07:33:21.662 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:21.667+0000 7f519bfff700 1 --2- 172.21.15.52:0/1642907313 >> [v2:172.21.15.105:6838/500348151,v1:172.21.15.105:6839/500348151] conn(0x55b650063c60 0x55b650066130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:33:21.662 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:21.667+0000 7f519bfff700 1 --2- 172.21.15.52:0/1642907313 >> [v2:172.21.15.105:6838/500348151,v1:172.21.15.105:6839/500348151] conn(0x55b650063c60 0x55b650066130 crc :-1 s=READY pgs=137 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=2104f84c967f90c8 server_cookie=a73ed342fe47131c in_seq=0 out_seq=0 2024-05-27T07:33:21.662 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:21.667+0000 7f517bfff700 10 client.26311 ms_handle_connect on v2:172.21.15.105:6838/500348151 2024-05-27T07:33:21.663 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:21.669+0000 7f517bfff700 1 -- 172.21.15.52:0/1642907313 <== mds.0 v2:172.21.15.105:6838/500348151 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24041 (crc 0 0 0) 0x55b6500159f0 con 0x55b650063c60 2024-05-27T07:33:21.663 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:21.669+0000 7f517bfff700 10 client.26311 handle_command_reply: tid=0 2024-05-27T07:33:21.663 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:21.669+0000 7f51b1780b80 1 client.26311 shutdown 2024-05-27T07:33:21.667 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:21.672+0000 7f51b1780b80 1 -- 172.21.15.52:0/1642907313 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55b64fecbe40 msgr2=0x55b6500532e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:33:21.667 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:21.672+0000 7f51b1780b80 1 --2- 172.21.15.52:0/1642907313 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55b64fecbe40 0x55b6500532e0 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7f51900a6ae0 tx=0x7f51900a96f0 comp rx=0 tx=0).stop 2024-05-27T07:33:21.667 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:21.673+0000 7f51b1780b80 1 -- 172.21.15.52:0/1642907313 shutdown_connections 2024-05-27T07:33:21.667 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:21.673+0000 7f51b1780b80 1 --2- 172.21.15.52:0/1642907313 >> [v2:172.21.15.105:6838/500348151,v1:172.21.15.105:6839/500348151] conn(0x55b650063c60 0x55b650066130 crc :-1 s=READY pgs=137 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:33:21.667 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:21.673+0000 7f51b1780b80 1 --2- 172.21.15.52:0/1642907313 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55b65004fc50 0x55b650053d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:33:21.667 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:21.673+0000 7f51b1780b80 1 --2- 172.21.15.52:0/1642907313 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55b64fecbe40 0x55b6500532e0 unknown :-1 s=CLOSED pgs=166 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:33:21.667 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:21.673+0000 7f51b1780b80 1 --2- 172.21.15.52:0/1642907313 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55b650049f80 0x55b650053820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:33:21.667 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:21.673+0000 7f51b1780b80 1 -- 172.21.15.52:0/1642907313 >> 172.21.15.52:0/1642907313 conn(0x55b64ff26af0 msgr2=0x55b650048a30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:33:21.667 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:21.673+0000 7f517bfff700 0 client.26311 ms_handle_reset on v2:172.21.15.105:6838/500348151 2024-05-27T07:33:21.668 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:21.674+0000 7f51b1780b80 1 -- 172.21.15.52:0/1642907313 shutdown_connections 2024-05-27T07:33:21.668 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:21.674+0000 7f51b1780b80 1 -- 172.21.15.52:0/1642907313 wait complete. 2024-05-27T07:33:21.668 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:21.674+0000 7f51b1780b80 20 client.26311 trim_cache size 0 max 16384 2024-05-27T07:33:21.711 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:21.717+0000 7f51b1780b80 1 Processor -- start 2024-05-27T07:33:21.711 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:21.717+0000 7f51b1780b80 1 -- start start 2024-05-27T07:33:21.711 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:21.717+0000 7f51b1780b80 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55b64fecbe40 0x55b650049a40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:33:21.711 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:21.717+0000 7f51b1780b80 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55b650049f80 0x55b650052240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:33:21.713 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:21.717+0000 7f51a0b07700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55b64fecbe40 0x55b650049a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:33:21.713 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:21.717+0000 7f51a0b07700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55b64fecbe40 0x55b650049a40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.52:54242/0 (socket says 172.21.15.52:54242) 2024-05-27T07:33:21.714 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:21.718+0000 7f51a0b07700 1 -- 172.21.15.52:0/4289527271 learned_addr learned my addr 172.21.15.52:0/4289527271 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-05-27T07:33:21.714 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:21.718+0000 7f51b1780b80 1 --2- 172.21.15.52:0/4289527271 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55b65004fc50 0x55b65004d720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:33:21.714 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:21.718+0000 7f51b1780b80 1 -- 172.21.15.52:0/4289527271 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x7f518800f410 con 0x55b650049f80 2024-05-27T07:33:21.714 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:21.718+0000 7f51b1780b80 1 -- 172.21.15.52:0/4289527271 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f518800d7b0 con 0x55b64fecbe40 2024-05-27T07:33:21.714 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:21.718+0000 7f51b1780b80 1 -- 172.21.15.52:0/4289527271 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f518800ed10 con 0x55b65004fc50 2024-05-27T07:33:21.714 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:21.718+0000 7f519bfff700 1 --2- 172.21.15.52:0/4289527271 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55b650049f80 0x55b650052240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:33:21.714 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:21.718+0000 7f51a1308700 1 --2- 172.21.15.52:0/4289527271 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55b65004fc50 0x55b65004d720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:33:21.714 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:21.718+0000 7f51a0b07700 1 -- 172.21.15.52:0/4289527271 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55b65004fc50 msgr2=0x55b65004d720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:33:21.714 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:21.718+0000 7f51a0b07700 1 --2- 172.21.15.52:0/4289527271 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55b65004fc50 0x55b65004d720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:33:21.714 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:21.718+0000 7f51a0b07700 1 -- 172.21.15.52:0/4289527271 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55b650049f80 msgr2=0x55b650052240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:33:21.714 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:21.718+0000 7f51a0b07700 1 --2- 172.21.15.52:0/4289527271 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55b650049f80 0x55b650052240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:33:21.714 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:21.718+0000 7f51a0b07700 1 -- 172.21.15.52:0/4289527271 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55b6500457b0 con 0x55b64fecbe40 2024-05-27T07:33:21.715 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:21.719+0000 7f51a0b07700 1 --2- 172.21.15.52:0/4289527271 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55b64fecbe40 0x55b650049a40 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7f51900a9570 tx=0x7f51900cc300 comp rx=0 tx=0).ready entity=mon.2 client_cookie=9a21f7786f769bfa server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:33:21.716 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:21.722+0000 7f51827fc700 1 -- 172.21.15.52:0/4289527271 <== mon.2 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5190020070 con 0x55b64fecbe40 2024-05-27T07:33:21.716 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:21.722+0000 7f51827fc700 1 -- 172.21.15.52:0/4289527271 <== mon.2 v2:172.21.15.105:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f51900b24a0 con 0x55b64fecbe40 2024-05-27T07:33:21.716 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:21.722+0000 7f51827fc700 1 -- 172.21.15.52:0/4289527271 <== mon.2 v2:172.21.15.105:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f51900aac90 con 0x55b64fecbe40 2024-05-27T07:33:21.721 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:21.724+0000 7f51b1780b80 1 -- 172.21.15.52:0/4289527271 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55b64fecbe40 msgr2=0x55b650049a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:33:21.721 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:21.724+0000 7f51b1780b80 1 --2- 172.21.15.52:0/4289527271 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55b64fecbe40 0x55b650049a40 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7f51900a9570 tx=0x7f51900cc300 comp rx=0 tx=0).stop 2024-05-27T07:33:21.721 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:21.726+0000 7f51b1780b80 1 -- 172.21.15.52:0/4289527271 shutdown_connections 2024-05-27T07:33:21.721 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:21.726+0000 7f51b1780b80 1 --2- 172.21.15.52:0/4289527271 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55b65004fc50 0x55b65004d720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:33:21.721 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:21.726+0000 7f51b1780b80 1 --2- 172.21.15.52:0/4289527271 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55b64fecbe40 0x55b650049a40 unknown :-1 s=CLOSED pgs=212 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:33:21.721 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:21.726+0000 7f51b1780b80 1 --2- 172.21.15.52:0/4289527271 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55b650049f80 0x55b650052240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:33:21.721 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:21.726+0000 7f51b1780b80 1 -- 172.21.15.52:0/4289527271 >> 172.21.15.52:0/4289527271 conn(0x55b64ff26af0 msgr2=0x55b65004d010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:33:21.721 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:21.727+0000 7f51b1780b80 1 -- 172.21.15.52:0/4289527271 shutdown_connections 2024-05-27T07:33:21.726 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:21.727+0000 7f51b1780b80 1 -- 172.21.15.52:0/4289527271 wait complete. 2024-05-27T07:33:21.726 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:21.727+0000 7f51b1780b80 1 -- start start 2024-05-27T07:33:21.726 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:21.727+0000 7f51b1780b80 1 Processor -- start 2024-05-27T07:33:21.726 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:21.728+0000 7f51b1780b80 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55b64fecbe40 0x55b65005bfc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:33:21.726 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:21.728+0000 7f51b1780b80 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55b650049f80 0x55b65006b050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:33:21.726 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:21.728+0000 7f51b1780b80 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55b65004fc50 0x55b65006b590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:33:21.727 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:21.728+0000 7f51b1780b80 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x7f518800e580 con 0x55b65004fc50 2024-05-27T07:33:21.727 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:21.728+0000 7f51b1780b80 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f518800de80 con 0x55b650049f80 2024-05-27T07:33:21.727 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:21.728+0000 7f51b1780b80 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x55b65004a9f0 con 0x55b64fecbe40 2024-05-27T07:33:21.727 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:21.728+0000 7f51b1780b80 10 client.0 _ll_register_callbacks cb 0x55b64ff03920 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T07:33:21.727 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:21.729+0000 7f51a1308700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55b65004fc50 0x55b65006b590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:33:21.727 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:21.729+0000 7f51a1308700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55b65004fc50 0x55b65006b590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.52:50448/0 (socket says 172.21.15.52:50448) 2024-05-27T07:33:21.727 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:21.729+0000 7f51a1308700 1 -- 172.21.15.52:0/862259024 learned_addr learned my addr 172.21.15.52:0/862259024 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-05-27T07:33:21.727 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:21.729+0000 7f51a0b07700 1 --2- 172.21.15.52:0/862259024 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55b64fecbe40 0x55b65005bfc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:33:21.727 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:21.729+0000 7f519bfff700 1 --2- 172.21.15.52:0/862259024 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55b650049f80 0x55b65006b050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:33:21.727 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:21.729+0000 7f51a0b07700 1 -- 172.21.15.52:0/862259024 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55b650049f80 msgr2=0x55b65006b050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:33:21.727 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:21.729+0000 7f51a0b07700 1 --2- 172.21.15.52:0/862259024 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55b650049f80 0x55b65006b050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:33:21.727 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:21.729+0000 7f51a0b07700 1 -- 172.21.15.52:0/862259024 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55b65004fc50 msgr2=0x55b65006b590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:33:21.727 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:21.729+0000 7f51a0b07700 1 --2- 172.21.15.52:0/862259024 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55b65004fc50 0x55b65006b590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:33:21.728 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:21.729+0000 7f51a0b07700 1 -- 172.21.15.52:0/862259024 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55b65006bbd0 con 0x55b64fecbe40 2024-05-27T07:33:21.728 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:21.730+0000 7f51a0b07700 1 --2- 172.21.15.52:0/862259024 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55b64fecbe40 0x55b65005bfc0 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7f51900e0260 tx=0x7f51900aaf90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=d5f0a006f97542f6 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:33:21.728 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:21.730+0000 7f517bfff700 10 client.0 ms_handle_connect on v2:172.21.15.173:3300/0 2024-05-27T07:33:21.728 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:21.730+0000 7f517bfff700 1 -- 172.21.15.52:0/862259024 <== mon.1 v2:172.21.15.173:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5190020040 con 0x55b64fecbe40 2024-05-27T07:33:21.728 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:21.730+0000 7f51b1780b80 1 -- 172.21.15.52:0/862259024 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f51900b24a0 con 0x55b64fecbe40 2024-05-27T07:33:21.728 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:21.730+0000 7f517bfff700 1 -- 172.21.15.52:0/862259024 <== mon.1 v2:172.21.15.173:3300/0 2 ==== osd_map(117..117 src has 1..117) ==== 9555+0+0 (secure 0 0 0) 0x7f51900a9ad0 con 0x55b64fecbe40 2024-05-27T07:33:21.728 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:21.731+0000 7f517bfff700 1 -- 172.21.15.52:0/862259024 <== mon.1 v2:172.21.15.173:3300/0 3 ==== mon_get_version_reply(handle=1 version=103) ==== 24+0+0 (secure 0 0 0) 0x7f51900a9f70 con 0x55b64fecbe40 2024-05-27T07:33:21.728 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:21.731+0000 7f51b1780b80 10 client.26317 fetch_fsmap learned FSMap version 103 2024-05-27T07:33:21.728 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:21.731+0000 7f51b1780b80 1 -- 172.21.15.52:0/862259024 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({config=0+,fsmap=103,monmap=4+}) -- 0x55b6500ce5e0 con 0x55b64fecbe40 2024-05-27T07:33:21.728 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:21.731+0000 7f517bfff700 1 -- 172.21.15.52:0/862259024 <== mon.1 v2:172.21.15.173:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f51900b24a0 con 0x55b64fecbe40 2024-05-27T07:33:21.730 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:21.732+0000 7f517bfff700 1 -- 172.21.15.52:0/862259024 <== mon.1 v2:172.21.15.173:3300/0 5 ==== fsmap(e 103) ==== 4215+0+0 (secure 0 0 0) 0x7f51900910d0 con 0x55b64fecbe40 2024-05-27T07:33:21.731 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:21.733+0000 7f51b1780b80 10 client.26317 fetch_fsmap finished waiting for FSMap version 103 2024-05-27T07:33:21.731 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:21.733+0000 7f51b1780b80 10 client.26317 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-05-27T07:33:21.731 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:21.733+0000 7f51b1780b80 20 client.26317 populate_metadata read hostname 'smithi052' 2024-05-27T07:33:21.731 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:21.733+0000 7f51b1780b80 1 --2- 172.21.15.52:0/862259024 >> [v2:172.21.15.105:6838/500348151,v1:172.21.15.105:6839/500348151] conn(0x55b6500ceaa0 0x55b6500d0ec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:33:21.731 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:21.733+0000 7f51b1780b80 4 client.26317 mds_command: new command op to 24527 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-05-27T07:33:21.731 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:21.733+0000 7f51b1780b80 1 -- 172.21.15.52:0/862259024 --> [v2:172.21.15.105:6838/500348151,v1:172.21.15.105:6839/500348151] -- command(tid 0: {"prefix": "scrub status"}) -- 0x55b64ff7abe0 con 0x55b6500ceaa0 2024-05-27T07:33:21.731 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:21.734+0000 7f519bfff700 1 --2- 172.21.15.52:0/862259024 >> [v2:172.21.15.105:6838/500348151,v1:172.21.15.105:6839/500348151] conn(0x55b6500ceaa0 0x55b6500d0ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:33:21.731 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:21.735+0000 7f517bfff700 10 client.26317 ms_handle_connect on v2:172.21.15.105:6838/500348151 2024-05-27T07:33:21.731 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:21.735+0000 7f519bfff700 1 --2- 172.21.15.52:0/862259024 >> [v2:172.21.15.105:6838/500348151,v1:172.21.15.105:6839/500348151] conn(0x55b6500ceaa0 0x55b6500d0ec0 crc :-1 s=READY pgs=138 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=dee1c936773e825d server_cookie=6054e74c4ae86b06 in_seq=0 out_seq=0 2024-05-27T07:33:21.731 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:21.736+0000 7f517bfff700 1 -- 172.21.15.52:0/862259024 <== mds.0 v2:172.21.15.105:6838/500348151 1 ==== command_reply(tid 0: 0 ) ==== 8+0+63 (crc 0 0 0) 0x55b64ff7abe0 con 0x55b6500ceaa0 2024-05-27T07:33:21.731 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:21.736+0000 7f517bfff700 10 client.26317 handle_command_reply: tid=0 2024-05-27T07:33:21.732 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:21.736+0000 7f51b1780b80 1 client.26317 shutdown 2024-05-27T07:33:21.732 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:21.738+0000 7f51b1780b80 1 -- 172.21.15.52:0/862259024 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55b64fecbe40 msgr2=0x55b65005bfc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:33:21.732 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:21.738+0000 7f51b1780b80 1 --2- 172.21.15.52:0/862259024 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55b64fecbe40 0x55b65005bfc0 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7f51900e0260 tx=0x7f51900aaf90 comp rx=0 tx=0).stop 2024-05-27T07:33:21.733 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:21.738+0000 7f51b1780b80 1 -- 172.21.15.52:0/862259024 shutdown_connections 2024-05-27T07:33:21.733 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:21.738+0000 7f51b1780b80 1 --2- 172.21.15.52:0/862259024 >> [v2:172.21.15.105:6838/500348151,v1:172.21.15.105:6839/500348151] conn(0x55b6500ceaa0 0x55b6500d0ec0 crc :-1 s=READY pgs=138 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:33:21.733 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:21.738+0000 7f51b1780b80 1 --2- 172.21.15.52:0/862259024 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55b65004fc50 0x55b65006b590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:33:21.733 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:21.738+0000 7f51b1780b80 1 --2- 172.21.15.52:0/862259024 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55b64fecbe40 0x55b65005bfc0 unknown :-1 s=CLOSED pgs=167 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:33:21.733 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:21.738+0000 7f51b1780b80 1 --2- 172.21.15.52:0/862259024 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55b650049f80 0x55b65006b050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:33:21.733 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:21.739+0000 7f51b1780b80 1 -- 172.21.15.52:0/862259024 >> 172.21.15.52:0/862259024 conn(0x55b64ff26af0 msgr2=0x55b65004e320 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:33:21.733 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:21.739+0000 7f517bfff700 0 client.26317 ms_handle_reset on v2:172.21.15.105:6838/500348151 2024-05-27T07:33:21.733 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:21.739+0000 7f51b1780b80 1 -- 172.21.15.52:0/862259024 shutdown_connections 2024-05-27T07:33:21.734 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:21.740+0000 7f51b1780b80 1 -- 172.21.15.52:0/862259024 wait complete. 2024-05-27T07:33:21.737 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:21.744+0000 7f51b1780b80 20 client.26317 trim_cache size 0 max 16384 2024-05-27T07:33:21.739 INFO:teuthology.orchestra.run.smithi052.stdout:{ 2024-05-27T07:33:21.739 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "no active scrubs running", 2024-05-27T07:33:21.739 INFO:teuthology.orchestra.run.smithi052.stdout: "scrubs": {} 2024-05-27T07:33:21.739 INFO:teuthology.orchestra.run.smithi052.stdout:} 2024-05-27T07:33:21.740 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:21.746+0000 7f51a2d6b700 1 -- 172.21.15.52:0/955600229 >> [v2:172.21.15.52:6800/1822318384,v1:172.21.15.52:6801/1822318384] conn(0x7f51840b2aa0 msgr2=0x7f51840b4fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:33:21.740 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:21.746+0000 7f51a2d6b700 1 --2- 172.21.15.52:0/955600229 >> [v2:172.21.15.52:6800/1822318384,v1:172.21.15.52:6801/1822318384] conn(0x7f51840b2aa0 0x7f51840b4fb0 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7f518c009fd0 tx=0x7f518c020000 comp rx=0 tx=0).stop 2024-05-27T07:33:21.740 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:21.746+0000 7f51a2d6b700 1 -- 172.21.15.52:0/955600229 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f519c10a920 msgr2=0x7f519c1b2400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:33:21.740 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:21.746+0000 7f51a2d6b700 1 --2- 172.21.15.52:0/955600229 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f519c10a920 0x7f519c1b2400 secure :-1 s=READY pgs=685 cs=0 l=1 rev1=1 crypto rx=0x7f519000d740 tx=0x7f519000db00 comp rx=0 tx=0).stop 2024-05-27T07:33:21.740 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:21.746+0000 7f51a2d6b700 1 -- 172.21.15.52:0/955600229 shutdown_connections 2024-05-27T07:33:21.741 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:21.746+0000 7f51a2d6b700 1 --2- 172.21.15.52:0/955600229 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f519c111bc0 0x7f519c1b6c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:33:21.741 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:21.746+0000 7f51a2d6b700 1 --2- 172.21.15.52:0/955600229 >> [v2:172.21.15.52:6800/1822318384,v1:172.21.15.52:6801/1822318384] conn(0x7f51840b2aa0 0x7f51840b4fb0 unknown :-1 s=CLOSED pgs=161 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:33:21.741 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:21.746+0000 7f51a2d6b700 1 --2- 172.21.15.52:0/955600229 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f519c10a920 0x7f519c1b2400 unknown :-1 s=CLOSED pgs=685 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:33:21.741 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:21.746+0000 7f51a2d6b700 1 --2- 172.21.15.52:0/955600229 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f519c10b270 0x7f519c1b29a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:33:21.741 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:21.746+0000 7f51a2d6b700 1 -- 172.21.15.52:0/955600229 >> 172.21.15.52:0/955600229 conn(0x7f519c06f4c0 msgr2=0x7f519c110630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:33:21.741 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:21.747+0000 7f51a2d6b700 1 -- 172.21.15.52:0/955600229 shutdown_connections 2024-05-27T07:33:21.741 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:21.747+0000 7f51a2d6b700 1 -- 172.21.15.52:0/955600229 wait complete. 2024-05-27T07:33:22.483 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:33:22 smithi052 ceph-mon[31955]: pgmap v251: 129 pgs: 14 active+clean+snaptrim, 75 active+clean, 40 active+clean+snaptrim_wait; 649 MiB data, 3.9 GiB used, 1.0 TiB / 1.0 TiB avail; 35 KiB/s rd, 1.7 MiB/s wr, 597 op/s 2024-05-27T07:33:22.606 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:33:22 smithi173 ceph-mon[37000]: pgmap v251: 129 pgs: 14 active+clean+snaptrim, 75 active+clean, 40 active+clean+snaptrim_wait; 649 MiB data, 3.9 GiB used, 1.0 TiB / 1.0 TiB avail; 35 KiB/s rd, 1.7 MiB/s wr, 597 op/s 2024-05-27T07:33:22.689 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-05-27T07:33:22.690 DEBUG:teuthology.orchestra.run.smithi052:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:38f9d900c83c63799fbdbe61acc9a11b0d3554a6 shell --fsid 73df2636-1bf8-11ef-bc9b-c7b262605968 -- ceph fs dump --format=json 2024-05-27T07:33:22.694 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:33:22 smithi105 ceph-mon[35977]: pgmap v251: 129 pgs: 14 active+clean+snaptrim, 75 active+clean, 40 active+clean+snaptrim_wait; 649 MiB data, 3.9 GiB used, 1.0 TiB / 1.0 TiB avail; 35 KiB/s rd, 1.7 MiB/s wr, 597 op/s 2024-05-27T07:33:23.186 INFO:teuthology.orchestra.run.smithi052.stderr:Inferring config /var/lib/ceph/73df2636-1bf8-11ef-bc9b-c7b262605968/mon.a/config 2024-05-27T07:33:24.484 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:33:24 smithi052 ceph-mon[31955]: pgmap v252: 129 pgs: 12 active+clean+snaptrim, 84 active+clean, 33 active+clean+snaptrim_wait; 650 MiB data, 4.0 GiB used, 1.0 TiB / 1.0 TiB avail; 26 KiB/s rd, 1.7 MiB/s wr, 579 op/s 2024-05-27T07:33:24.606 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:33:24 smithi173 ceph-mon[37000]: pgmap v252: 129 pgs: 12 active+clean+snaptrim, 84 active+clean, 33 active+clean+snaptrim_wait; 650 MiB data, 4.0 GiB used, 1.0 TiB / 1.0 TiB avail; 26 KiB/s rd, 1.7 MiB/s wr, 579 op/s 2024-05-27T07:33:24.693 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:33:24 smithi105 ceph-mon[35977]: pgmap v252: 129 pgs: 12 active+clean+snaptrim, 84 active+clean, 33 active+clean+snaptrim_wait; 650 MiB data, 4.0 GiB used, 1.0 TiB / 1.0 TiB avail; 26 KiB/s rd, 1.7 MiB/s wr, 579 op/s 2024-05-27T07:33:25.563 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:25.567+0000 7f6aed53e700 1 -- 172.21.15.52:0/1388768744 <== mon.1 v2:172.21.15.173:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6ae0012650 con 0x7f6ae8108750 2024-05-27T07:33:25.563 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:25.567+0000 7f6af07a4700 1 -- 172.21.15.52:0/1388768744 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f6ae8108750 msgr2=0x7f6ae806c030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:33:25.563 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:25.567+0000 7f6af07a4700 1 --2- 172.21.15.52:0/1388768744 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f6ae8108750 0x7f6ae806c030 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7f6ae0009f90 tx=0x7f6ae000b330 comp rx=0 tx=0).stop 2024-05-27T07:33:25.563 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:25.567+0000 7f6af07a4700 1 -- 172.21.15.52:0/1388768744 shutdown_connections 2024-05-27T07:33:25.563 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:25.567+0000 7f6af07a4700 1 --2- 172.21.15.52:0/1388768744 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f6ae8108750 0x7f6ae806c030 unknown :-1 s=CLOSED pgs=168 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:33:25.563 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:25.567+0000 7f6af07a4700 1 --2- 172.21.15.52:0/1388768744 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f6ae81034d0 0x7f6ae81038e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:33:25.563 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:25.567+0000 7f6af07a4700 1 --2- 172.21.15.52:0/1388768744 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f6ae8107e00 0x7f6ae8108210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:33:25.563 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:25.567+0000 7f6af07a4700 1 -- 172.21.15.52:0/1388768744 >> 172.21.15.52:0/1388768744 conn(0x7f6ae80fef20 msgr2=0x7f6ae8101340 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:33:25.564 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:25.571+0000 7f6af07a4700 1 -- 172.21.15.52:0/1388768744 shutdown_connections 2024-05-27T07:33:25.565 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:25.572+0000 7f6af07a4700 1 -- 172.21.15.52:0/1388768744 wait complete. 2024-05-27T07:33:25.567 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:25.573+0000 7f6af07a4700 1 Processor -- start 2024-05-27T07:33:25.571 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:25.576+0000 7f6af07a4700 1 -- start start 2024-05-27T07:33:25.571 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:25.576+0000 7f6af07a4700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f6ae81034d0 0x7f6ae806ee70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:33:25.571 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:25.576+0000 7f6af07a4700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f6ae8107e00 0x7f6ae806f3b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:33:25.571 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:25.576+0000 7f6af07a4700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f6ae8108750 0x7f6ae80736a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:33:25.571 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:25.576+0000 7f6af07a4700 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x7f6ae806d350 con 0x7f6ae8108750 2024-05-27T07:33:25.571 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:25.576+0000 7f6af07a4700 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f6ae806d1d0 con 0x7f6ae8107e00 2024-05-27T07:33:25.571 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:25.576+0000 7f6af07a4700 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f6ae806d4d0 con 0x7f6ae81034d0 2024-05-27T07:33:25.572 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:25.577+0000 7f6aedd3f700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f6ae8107e00 0x7f6ae806f3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:33:25.572 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:25.577+0000 7f6aeed41700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f6ae8108750 0x7f6ae80736a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:33:25.573 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:25.577+0000 7f6aeed41700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f6ae8108750 0x7f6ae80736a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.52:50468/0 (socket says 172.21.15.52:50468) 2024-05-27T07:33:25.573 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:25.577+0000 7f6aeed41700 1 -- 172.21.15.52:0/3620737119 learned_addr learned my addr 172.21.15.52:0/3620737119 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-05-27T07:33:25.573 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:25.578+0000 7f6aeed41700 1 -- 172.21.15.52:0/3620737119 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f6ae81034d0 msgr2=0x7f6ae806ee70 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-05-27T07:33:25.573 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:25.578+0000 7f6aeed41700 1 --2- 172.21.15.52:0/3620737119 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f6ae81034d0 0x7f6ae806ee70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:33:25.573 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:25.578+0000 7f6aeed41700 1 -- 172.21.15.52:0/3620737119 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f6ae8107e00 msgr2=0x7f6ae806f3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:33:25.573 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:25.578+0000 7f6aeed41700 1 --2- 172.21.15.52:0/3620737119 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f6ae8107e00 0x7f6ae806f3b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:33:25.573 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:25.578+0000 7f6aeed41700 1 -- 172.21.15.52:0/3620737119 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6ae8073c90 con 0x7f6ae8108750 2024-05-27T07:33:25.573 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:25.578+0000 7f6aeed41700 1 --2- 172.21.15.52:0/3620737119 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f6ae8108750 0x7f6ae80736a0 secure :-1 s=READY pgs=686 cs=0 l=1 rev1=1 crypto rx=0x7f6ae00060b0 tx=0x7f6ae0009490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:33:25.577 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:25.584+0000 7f6adf7fe700 1 -- 172.21.15.52:0/3620737119 <== mon.0 v2:172.21.15.52:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6ae0021070 con 0x7f6ae8108750 2024-05-27T07:33:25.578 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:25.584+0000 7f6adf7fe700 1 -- 172.21.15.52:0/3620737119 <== mon.0 v2:172.21.15.52:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f6ae0007bf0 con 0x7f6ae8108750 2024-05-27T07:33:25.578 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:25.584+0000 7f6adf7fe700 1 -- 172.21.15.52:0/3620737119 <== mon.0 v2:172.21.15.52:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6ae00052a0 con 0x7f6ae8108750 2024-05-27T07:33:25.582 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:25.587+0000 7f6af07a4700 1 -- 172.21.15.52:0/3620737119 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6ae8073f20 con 0x7f6ae8108750 2024-05-27T07:33:25.586 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:25.591+0000 7f6af07a4700 1 -- 172.21.15.52:0/3620737119 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6ae8074380 con 0x7f6ae8108750 2024-05-27T07:33:25.589 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:25.595+0000 7f6adf7fe700 1 -- 172.21.15.52:0/3620737119 <== mon.0 v2:172.21.15.52:3300/0 4 ==== mgrmap(e 26) ==== 149783+0+0 (secure 0 0 0) 0x7f6ae0024070 con 0x7f6ae8108750 2024-05-27T07:33:25.593 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:25.596+0000 7f6af07a4700 1 -- 172.21.15.52:0/3620737119 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6ae8104980 con 0x7f6ae8108750 2024-05-27T07:33:25.594 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:25.600+0000 7f6adf7fe700 1 --2- 172.21.15.52:0/3620737119 >> [v2:172.21.15.52:6800/1822318384,v1:172.21.15.52:6801/1822318384] conn(0x7f6ad40b2b80 0x7f6ad40b5090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:33:25.594 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:25.601+0000 7f6adf7fe700 1 -- 172.21.15.52:0/3620737119 <== mon.0 v2:172.21.15.52:3300/0 5 ==== osd_map(117..117 src has 1..117) ==== 9555+0+0 (secure 0 0 0) 0x7f6ae000f070 con 0x7f6ae8108750 2024-05-27T07:33:25.596 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:25.602+0000 7f6aee540700 1 --2- 172.21.15.52:0/3620737119 >> [v2:172.21.15.52:6800/1822318384,v1:172.21.15.52:6801/1822318384] conn(0x7f6ad40b2b80 0x7f6ad40b5090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:33:25.597 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:25.603+0000 7f6aee540700 1 --2- 172.21.15.52:0/3620737119 >> [v2:172.21.15.52:6800/1822318384,v1:172.21.15.52:6801/1822318384] conn(0x7f6ad40b2b80 0x7f6ad40b5090 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7f6ae4005fd0 tx=0x7f6ae4013040 comp rx=0 tx=0).ready entity=mgr.15333 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:33:25.606 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:25.612+0000 7f6adf7fe700 1 -- 172.21.15.52:0/3620737119 <== mon.0 v2:172.21.15.52:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191963 (secure 0 0 0) 0x7f6ae00aa6e0 con 0x7f6ae8108750 2024-05-27T07:33:26.118 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:26.124+0000 7f6af07a4700 1 -- 172.21.15.52:0/3620737119 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f6ae80701d0 con 0x7f6ae8108750 2024-05-27T07:33:26.119 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:26.126+0000 7f6adf7fe700 1 -- 172.21.15.52:0/3620737119 <== mon.0 v2:172.21.15.52:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 103 v103) ==== 95+0+6130 (secure 0 0 0) 0x7f6ae00a9fe0 con 0x7f6ae8108750 2024-05-27T07:33:26.120 INFO:teuthology.orchestra.run.smithi052.stdout: 2024-05-27T07:33:26.120 INFO:teuthology.orchestra.run.smithi052.stdout:{"epoch":103,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":103,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-05-27T07:21:50.521783+0000","modified":"2024-05-27T07:33:18.617071+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24527,"mds_1":24497,"mds_2":24475,"mds_3":24473,"mds_4":24466},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24466":{"gid":24466,"name":"c","rank":4,"incarnation":26,"state":"up:active","state_seq":25,"addr":"172.21.15.173:6837/1396363827","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6836","nonce":1396363827},{"type":"v1","addr":"172.21.15.173:6837","nonce":1396363827}]},"join_fscid":-1,"export_targets":[0],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24473":{"gid":24473,"name":"e","rank":3,"incarnation":24,"state":"up:active","state_seq":26,"addr":"172.21.15.105:6835/3298498384","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6834","nonce":3298498384},{"type":"v1","addr":"172.21.15.105:6835","nonce":3298498384}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24475":{"gid":24475,"name":"i","rank":2,"incarnation":22,"state":"up:active","state_seq":23,"addr":"172.21.15.173:6839/564778293","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6838","nonce":564778293},{"type":"v1","addr":"172.21.15.173:6839","nonce":564778293}]},"join_fscid":-1,"export_targets":[0],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24497":{"gid":24497,"name":"h","rank":1,"incarnation":20,"state":"up:active","state_seq":24,"addr":"172.21.15.105:6837/191393149","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6836","nonce":191393149},{"type":"v1","addr":"172.21.15.105:6837","nonce":191393149}]},"join_fscid":-1,"export_targets":[0],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24527":{"gid":24527,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":5,"addr":"172.21.15.105:6839/500348151","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6838","nonce":500348151},{"type":"v1","addr":"172.21.15.105:6839","nonce":500348151}]},"join_fscid":-1,"export_targets":[1,2,3,4],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24527,"qdb_cluster":[24473,24466,24475,24527,24497]},"id":1}]} 2024-05-27T07:33:26.123 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:26.130+0000 7f6af07a4700 1 -- 172.21.15.52:0/3620737119 >> [v2:172.21.15.52:6800/1822318384,v1:172.21.15.52:6801/1822318384] conn(0x7f6ad40b2b80 msgr2=0x7f6ad40b5090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:33:26.124 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:26.130+0000 7f6af07a4700 1 --2- 172.21.15.52:0/3620737119 >> [v2:172.21.15.52:6800/1822318384,v1:172.21.15.52:6801/1822318384] conn(0x7f6ad40b2b80 0x7f6ad40b5090 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7f6ae4005fd0 tx=0x7f6ae4013040 comp rx=0 tx=0).stop 2024-05-27T07:33:26.124 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:26.130+0000 7f6af07a4700 1 -- 172.21.15.52:0/3620737119 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f6ae8108750 msgr2=0x7f6ae80736a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:33:26.124 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:26.130+0000 7f6af07a4700 1 --2- 172.21.15.52:0/3620737119 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f6ae8108750 0x7f6ae80736a0 secure :-1 s=READY pgs=686 cs=0 l=1 rev1=1 crypto rx=0x7f6ae00060b0 tx=0x7f6ae0009490 comp rx=0 tx=0).stop 2024-05-27T07:33:26.125 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:26.131+0000 7f6af07a4700 1 -- 172.21.15.52:0/3620737119 shutdown_connections 2024-05-27T07:33:26.125 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:26.131+0000 7f6af07a4700 1 --2- 172.21.15.52:0/3620737119 >> [v2:172.21.15.52:6800/1822318384,v1:172.21.15.52:6801/1822318384] conn(0x7f6ad40b2b80 0x7f6ad40b5090 unknown :-1 s=CLOSED pgs=162 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:33:26.125 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:26.131+0000 7f6af07a4700 1 --2- 172.21.15.52:0/3620737119 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f6ae8108750 0x7f6ae80736a0 unknown :-1 s=CLOSED pgs=686 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:33:26.125 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:26.131+0000 7f6af07a4700 1 --2- 172.21.15.52:0/3620737119 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f6ae81034d0 0x7f6ae806ee70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:33:26.125 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:26.131+0000 7f6af07a4700 1 --2- 172.21.15.52:0/3620737119 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f6ae8107e00 0x7f6ae806f3b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:33:26.125 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:26.131+0000 7f6af07a4700 1 -- 172.21.15.52:0/3620737119 >> 172.21.15.52:0/3620737119 conn(0x7f6ae80fef20 msgr2=0x7f6ae81056b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:33:26.125 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:26.132+0000 7f6af07a4700 1 -- 172.21.15.52:0/3620737119 shutdown_connections 2024-05-27T07:33:26.126 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:26.132+0000 7f6af07a4700 1 -- 172.21.15.52:0/3620737119 wait complete. 2024-05-27T07:33:26.127 INFO:teuthology.orchestra.run.smithi052.stderr:dumped fsmap epoch 103 2024-05-27T07:33:26.483 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:33:26 smithi052 ceph-mon[31955]: pgmap v253: 129 pgs: 10 active+clean+snaptrim, 94 active+clean, 25 active+clean+snaptrim_wait; 655 MiB data, 4.0 GiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 2.1 MiB/s wr, 597 op/s 2024-05-27T07:33:26.483 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:33:26 smithi052 ceph-mon[31955]: from='client.? 172.21.15.52:0/3620737119' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-27T07:33:26.605 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:33:26 smithi173 ceph-mon[37000]: pgmap v253: 129 pgs: 10 active+clean+snaptrim, 94 active+clean, 25 active+clean+snaptrim_wait; 655 MiB data, 4.0 GiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 2.1 MiB/s wr, 597 op/s 2024-05-27T07:33:26.606 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:33:26 smithi173 ceph-mon[37000]: from='client.? 172.21.15.52:0/3620737119' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-27T07:33:26.693 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:33:26 smithi105 ceph-mon[35977]: pgmap v253: 129 pgs: 10 active+clean+snaptrim, 94 active+clean, 25 active+clean+snaptrim_wait; 655 MiB data, 4.0 GiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 2.1 MiB/s wr, 597 op/s 2024-05-27T07:33:26.694 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:33:26 smithi105 ceph-mon[35977]: from='client.? 172.21.15.52:0/3620737119' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-27T07:33:27.523 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:33:27.528+0000 7f1c2535e640 1 -- 172.21.15.52:0/2585991067 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f1c20108010 msgr2=0x7f1c201084a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:33:27.523 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:33:27.528+0000 7f1c2535e640 1 --2- 172.21.15.52:0/2585991067 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f1c20108010 0x7f1c201084a0 secure :-1 s=READY pgs=687 cs=0 l=1 rev1=1 crypto rx=0x7f1c10009900 tx=0x7f1c1002a420 comp rx=0 tx=0).stop 2024-05-27T07:33:27.523 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:33:27.528+0000 7f1c2535e640 1 -- 172.21.15.52:0/2585991067 shutdown_connections 2024-05-27T07:33:27.523 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:33:27.528+0000 7f1c2535e640 1 --2- 172.21.15.52:0/2585991067 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f1c20108010 0x7f1c201084a0 unknown :-1 s=CLOSED pgs=687 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:33:27.523 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:33:27.528+0000 7f1c2535e640 1 --2- 172.21.15.52:0/2585991067 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f1c201075e0 0x7f1c201079f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:33:27.523 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:33:27.528+0000 7f1c2535e640 1 --2- 172.21.15.52:0/2585991067 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f1c20102b90 0x7f1c20102fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:33:27.523 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:33:27.528+0000 7f1c2535e640 1 -- 172.21.15.52:0/2585991067 >> 172.21.15.52:0/2585991067 conn(0x7f1c20078270 msgr2=0x7f1c200786b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:33:27.526 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:33:27.530+0000 7f1c2535e640 1 -- 172.21.15.52:0/2585991067 shutdown_connections 2024-05-27T07:33:27.527 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:33:27.532+0000 7f1c2535e640 1 -- 172.21.15.52:0/2585991067 wait complete. 2024-05-27T07:33:27.527 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:33:27.533+0000 7f1c2535e640 1 Processor -- start 2024-05-27T07:33:27.528 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:33:27.534+0000 7f1c2535e640 1 -- start start 2024-05-27T07:33:27.528 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:33:27.534+0000 7f1c2535e640 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f1c20102b90 0x7f1c20074980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:33:27.528 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:33:27.534+0000 7f1c2535e640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f1c201075e0 0x7f1c2006dac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:33:27.528 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:33:27.534+0000 7f1c2535e640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f1c20108010 0x7f1c2006e180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:33:27.528 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:33:27.534+0000 7f1c2535e640 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x7f1c2011a4b0 con 0x7f1c20102b90 2024-05-27T07:33:27.528 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:33:27.534+0000 7f1c2535e640 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f1c2011a330 con 0x7f1c201075e0 2024-05-27T07:33:27.529 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:33:27.534+0000 7f1c2535e640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f1c2011a630 con 0x7f1c20108010 2024-05-27T07:33:27.529 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:33:27.535+0000 7f1c1effd640 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f1c20102b90 0x7f1c20074980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:33:27.529 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:33:27.535+0000 7f1c1effd640 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f1c20102b90 0x7f1c20074980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.52:50488/0 (socket says 172.21.15.52:50488) 2024-05-27T07:33:27.530 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:33:27.535+0000 7f1c1effd640 1 -- 172.21.15.52:0/1603374091 learned_addr learned my addr 172.21.15.52:0/1603374091 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-05-27T07:33:27.530 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:33:27.535+0000 7f1c1f7fe640 1 --2- 172.21.15.52:0/1603374091 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f1c20108010 0x7f1c2006e180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:33:27.530 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:33:27.535+0000 7f1c1e7fc640 1 --2- 172.21.15.52:0/1603374091 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f1c201075e0 0x7f1c2006dac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:33:27.531 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:33:27.536+0000 7f1c1effd640 1 -- 172.21.15.52:0/1603374091 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f1c20108010 msgr2=0x7f1c2006e180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:33:27.531 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:33:27.536+0000 7f1c1effd640 1 --2- 172.21.15.52:0/1603374091 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f1c20108010 0x7f1c2006e180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:33:27.531 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:33:27.536+0000 7f1c1effd640 1 -- 172.21.15.52:0/1603374091 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f1c201075e0 msgr2=0x7f1c2006dac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:33:27.531 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:33:27.536+0000 7f1c1effd640 1 --2- 172.21.15.52:0/1603374091 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f1c201075e0 0x7f1c2006dac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:33:27.531 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:33:27.536+0000 7f1c1effd640 1 -- 172.21.15.52:0/1603374091 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1c200724a0 con 0x7f1c20102b90 2024-05-27T07:33:27.531 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:33:27.537+0000 7f1c1f7fe640 1 --2- 172.21.15.52:0/1603374091 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f1c20108010 0x7f1c2006e180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-27T07:33:27.531 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:33:27.537+0000 7f1c1effd640 1 --2- 172.21.15.52:0/1603374091 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f1c20102b90 0x7f1c20074980 secure :-1 s=READY pgs=688 cs=0 l=1 rev1=1 crypto rx=0x7f1c1800eab0 tx=0x7f1c1800ef70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:33:27.532 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:33:27.538+0000 7f1c07fff640 1 -- 172.21.15.52:0/1603374091 <== mon.0 v2:172.21.15.52:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1c1800ce40 con 0x7f1c20102b90 2024-05-27T07:33:27.532 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:33:27.538+0000 7f1c07fff640 1 -- 172.21.15.52:0/1603374091 <== mon.0 v2:172.21.15.52:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f1c18011920 con 0x7f1c20102b90 2024-05-27T07:33:27.532 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:33:27.538+0000 7f1c07fff640 1 -- 172.21.15.52:0/1603374091 <== mon.0 v2:172.21.15.52:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1c1801aa60 con 0x7f1c20102b90 2024-05-27T07:33:27.533 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:33:27.539+0000 7f1c2535e640 1 -- 172.21.15.52:0/1603374091 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1c20072790 con 0x7f1c20102b90 2024-05-27T07:33:27.534 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:33:27.539+0000 7f1c2535e640 1 -- 172.21.15.52:0/1603374091 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1c20072c90 con 0x7f1c20102b90 2024-05-27T07:33:27.536 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:33:27.543+0000 7f1c2c214740 1 Processor -- start 2024-05-27T07:33:27.539 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:33:27.545+0000 7f1c07fff640 1 -- 172.21.15.52:0/1603374091 <== mon.0 v2:172.21.15.52:3300/0 4 ==== mgrmap(e 26) ==== 149783+0+0 (secure 0 0 0) 0x7f1c1801ac00 con 0x7f1c20102b90 2024-05-27T07:33:27.541 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:33:27.547+0000 7f1c2c214740 1 -- start start 2024-05-27T07:33:27.546 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:33:27.547+0000 7f1c2c214740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55b73d9bb1b0 0x55b73dd5c3d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:33:27.546 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:33:27.547+0000 7f1c2c214740 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55b73d9922f0 0x55b73dd5c910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:33:27.546 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:33:27.547+0000 7f1c2c214740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55b73d99a4d0 0x55b73dd5ce50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:33:27.546 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:33:27.548+0000 7f1c2c214740 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x55b73dccec30 con 0x55b73d9922f0 2024-05-27T07:33:27.547 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:33:27.548+0000 7f1c2c214740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x55b73dcbe7d0 con 0x55b73d99a4d0 2024-05-27T07:33:27.547 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:33:27.548+0000 7f1c2c214740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x55b73dcc2810 con 0x55b73d9bb1b0 2024-05-27T07:33:27.547 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:33:27.548+0000 7f1c07fff640 1 --2- 172.21.15.52:0/1603374091 >> [v2:172.21.15.52:6800/1822318384,v1:172.21.15.52:6801/1822318384] conn(0x7f1c080b2a50 0x7f1c080b4f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:33:27.547 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:33:27.549+0000 7f1c1e7fc640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55b73d99a4d0 0x55b73dd5ce50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:33:27.547 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:33:27.549+0000 7f1c1e7fc640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55b73d99a4d0 0x55b73dd5ce50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.52:54290/0 (socket says 172.21.15.52:54290) 2024-05-27T07:33:27.547 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:33:27.549+0000 7f1c1e7fc640 1 -- 172.21.15.52:0/3450933874 learned_addr learned my addr 172.21.15.52:0/3450933874 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-05-27T07:33:27.547 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:33:27.549+0000 7f1c1f7fe640 1 --2- 172.21.15.52:0/3450933874 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55b73d9bb1b0 0x55b73dd5c3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:33:27.547 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:33:27.551+0000 7f1c1effd640 1 --2- 172.21.15.52:0/3450933874 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55b73d9922f0 0x55b73dd5c910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:33:27.547 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:33:27.552+0000 7f1c1e7fc640 1 -- 172.21.15.52:0/3450933874 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55b73d9bb1b0 msgr2=0x55b73dd5c3d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:33:27.547 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:33:27.552+0000 7f1c1e7fc640 1 --2- 172.21.15.52:0/3450933874 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55b73d9bb1b0 0x55b73dd5c3d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:33:27.547 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:33:27.552+0000 7f1c1e7fc640 1 -- 172.21.15.52:0/3450933874 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55b73d9922f0 msgr2=0x55b73dd5c910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:33:27.547 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:33:27.552+0000 7f1c1e7fc640 1 --2- 172.21.15.52:0/3450933874 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55b73d9922f0 0x55b73dd5c910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:33:27.548 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:33:27.552+0000 7f1c1e7fc640 1 -- 172.21.15.52:0/3450933874 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55b73d9d3210 con 0x55b73d99a4d0 2024-05-27T07:33:27.548 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:33:27.552+0000 7f1c07fff640 1 -- 172.21.15.52:0/1603374091 <== mon.0 v2:172.21.15.52:3300/0 5 ==== osd_map(117..117 src has 1..117) ==== 9555+0+0 (secure 0 0 0) 0x7f1c180e2c60 con 0x7f1c20102b90 2024-05-27T07:33:27.548 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:33:27.553+0000 7f1c1e7fc640 1 --2- 172.21.15.52:0/3450933874 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55b73d99a4d0 0x55b73dd5ce50 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7f1c1400b7c0 tx=0x7f1c1400bc80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=edb2bf802dada0f4 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:33:27.548 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:33:27.554+0000 7f1c1f7fe640 1 --2- 172.21.15.52:0/1603374091 >> [v2:172.21.15.52:6800/1822318384,v1:172.21.15.52:6801/1822318384] conn(0x7f1c080b2a50 0x7f1c080b4f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:33:27.551 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:33:27.556+0000 7f1c057fa640 1 -- 172.21.15.52:0/3450933874 <== mon.2 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1c1400d950 con 0x55b73d99a4d0 2024-05-27T07:33:27.551 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:33:27.556+0000 7f1c057fa640 1 -- 172.21.15.52:0/3450933874 <== mon.2 v2:172.21.15.105:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f1c14018070 con 0x55b73d99a4d0 2024-05-27T07:33:27.551 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:33:27.557+0000 7f1c2c214740 1 -- 172.21.15.52:0/3450933874 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55b73d99a4d0 msgr2=0x55b73dd5ce50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:33:27.551 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:33:27.557+0000 7f1c2c214740 1 --2- 172.21.15.52:0/3450933874 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55b73d99a4d0 0x55b73dd5ce50 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7f1c1400b7c0 tx=0x7f1c1400bc80 comp rx=0 tx=0).stop 2024-05-27T07:33:27.554 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:33:27.559+0000 7f1c1f7fe640 1 --2- 172.21.15.52:0/1603374091 >> [v2:172.21.15.52:6800/1822318384,v1:172.21.15.52:6801/1822318384] conn(0x7f1c080b2a50 0x7f1c080b4f60 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x55b73dd59540 tx=0x7f1c10009260 comp rx=0 tx=0).ready entity=mgr.15333 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:33:27.554 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:33:27.559+0000 7f1c2c214740 1 -- 172.21.15.52:0/3450933874 shutdown_connections 2024-05-27T07:33:27.554 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:33:27.559+0000 7f1c2c214740 1 --2- 172.21.15.52:0/3450933874 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55b73d99a4d0 0x55b73dd5ce50 unknown :-1 s=CLOSED pgs=213 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:33:27.554 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:33:27.559+0000 7f1c2c214740 1 --2- 172.21.15.52:0/3450933874 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55b73d9922f0 0x55b73dd5c910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:33:27.554 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:33:27.559+0000 7f1c2c214740 1 --2- 172.21.15.52:0/3450933874 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55b73d9bb1b0 0x55b73dd5c3d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:33:27.554 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:33:27.559+0000 7f1c2c214740 1 -- 172.21.15.52:0/3450933874 >> 172.21.15.52:0/3450933874 conn(0x55b73d9af300 msgr2=0x55b73dd58b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:33:27.558 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:33:27.565+0000 7f1c2c214740 1 -- 172.21.15.52:0/3450933874 shutdown_connections 2024-05-27T07:33:27.561 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:33:27.567+0000 7f1c2c214740 1 -- 172.21.15.52:0/3450933874 wait complete. 2024-05-27T07:33:27.561 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:33:27.567+0000 7f1c2c214740 1 -- start start 2024-05-27T07:33:27.561 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:33:27.567+0000 7f1c2c214740 1 Processor -- start 2024-05-27T07:33:27.562 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:33:27.568+0000 7f1c2c214740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55b73d9922f0 0x55b73dd669b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:33:27.563 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:33:27.568+0000 7f1c2c214740 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55b73d99a4d0 0x55b73dd66ef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:33:27.563 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:33:27.568+0000 7f1c2c214740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55b73d9bb1b0 0x55b73dd728d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:33:27.563 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:33:27.568+0000 7f1c2c214740 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x55b73d9a4790 con 0x55b73d99a4d0 2024-05-27T07:33:27.563 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:33:27.568+0000 7f1c2c214740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x55b73dc7a890 con 0x55b73d9bb1b0 2024-05-27T07:33:27.563 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:33:27.568+0000 7f1c2c214740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x55b73dd69550 con 0x55b73d9922f0 2024-05-27T07:33:27.563 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:33:27.568+0000 7f1c2c214740 10 client.0 _ll_register_callbacks cb 0x55b73d9d2a00 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T07:33:27.566 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:33:27.569+0000 7f1c1e7fc640 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55b73d99a4d0 0x55b73dd66ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:33:27.566 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:33:27.569+0000 7f1c1e7fc640 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55b73d99a4d0 0x55b73dd66ef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.52:50506/0 (socket says 172.21.15.52:50506) 2024-05-27T07:33:27.566 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:33:27.569+0000 7f1c1e7fc640 1 -- 172.21.15.52:0/962688880 learned_addr learned my addr 172.21.15.52:0/962688880 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-05-27T07:33:27.566 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:33:27.569+0000 7f1c1f7fe640 1 --2- 172.21.15.52:0/962688880 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55b73d9bb1b0 0x55b73dd728d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:33:27.566 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:33:27.570+0000 7f1c1effd640 1 --2- 172.21.15.52:0/962688880 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55b73d9922f0 0x55b73dd669b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:33:27.566 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:33:27.570+0000 7f1c1e7fc640 1 -- 172.21.15.52:0/962688880 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55b73d9922f0 msgr2=0x55b73dd669b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:33:27.566 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:33:27.570+0000 7f1c1e7fc640 1 --2- 172.21.15.52:0/962688880 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55b73d9922f0 0x55b73dd669b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:33:27.567 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:33:27.570+0000 7f1c1e7fc640 1 -- 172.21.15.52:0/962688880 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55b73d9bb1b0 msgr2=0x55b73dd728d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:33:27.567 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:33:27.570+0000 7f1c1e7fc640 1 --2- 172.21.15.52:0/962688880 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55b73d9bb1b0 0x55b73dd728d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:33:27.567 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:33:27.570+0000 7f1c1e7fc640 1 -- 172.21.15.52:0/962688880 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55b73da9f620 con 0x55b73d99a4d0 2024-05-27T07:33:27.567 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:33:27.570+0000 7f1c1effd640 1 --2- 172.21.15.52:0/962688880 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55b73d9922f0 0x55b73dd669b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-27T07:33:27.567 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:33:27.570+0000 7f1c1f7fe640 1 --2- 172.21.15.52:0/962688880 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55b73d9bb1b0 0x55b73dd728d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-27T07:33:27.567 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:33:27.571+0000 7f1c1e7fc640 1 --2- 172.21.15.52:0/962688880 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55b73d99a4d0 0x55b73dd66ef0 secure :-1 s=READY pgs=689 cs=0 l=1 rev1=1 crypto rx=0x7f1c140047b0 tx=0x7f1c140136b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=75ea23d6e6072258 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:33:27.567 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:33:27.571+0000 7f1c02ff5640 10 client.0 ms_handle_connect on v2:172.21.15.52:3300/0 2024-05-27T07:33:27.567 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:33:27.571+0000 7f1c02ff5640 1 -- 172.21.15.52:0/962688880 <== mon.0 v2:172.21.15.52:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1c1400dd60 con 0x55b73d99a4d0 2024-05-27T07:33:27.567 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:33:27.571+0000 7f1c02ff5640 1 -- 172.21.15.52:0/962688880 <== mon.0 v2:172.21.15.52:3300/0 2 ==== osd_map(117..117 src has 1..117) ==== 9555+0+0 (secure 0 0 0) 0x7f1c14004030 con 0x55b73d99a4d0 2024-05-27T07:33:27.567 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:33:27.571+0000 7f1c2c214740 1 -- 172.21.15.52:0/962688880 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f1c14018070 con 0x55b73d99a4d0 2024-05-27T07:33:27.567 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:33:27.571+0000 7f1c02ff5640 1 -- 172.21.15.52:0/962688880 <== mon.0 v2:172.21.15.52:3300/0 3 ==== mon_get_version_reply(handle=1 version=103) ==== 24+0+0 (secure 0 0 0) 0x7f1c1402e050 con 0x55b73d99a4d0 2024-05-27T07:33:27.567 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:33:27.571+0000 7f1c2c214740 10 client.16827 fetch_fsmap learned FSMap version 103 2024-05-27T07:33:27.568 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:33:27.571+0000 7f1c2c214740 1 -- 172.21.15.52:0/962688880 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({config=0+,fsmap=103,monmap=4+}) -- 0x55b73dd75780 con 0x55b73d99a4d0 2024-05-27T07:33:27.568 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:33:27.571+0000 7f1c02ff5640 1 -- 172.21.15.52:0/962688880 <== mon.0 v2:172.21.15.52:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f1c14018070 con 0x55b73d99a4d0 2024-05-27T07:33:27.568 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:33:27.571+0000 7f1c02ff5640 1 -- 172.21.15.52:0/962688880 <== mon.0 v2:172.21.15.52:3300/0 5 ==== fsmap(e 103) ==== 4215+0+0 (secure 0 0 0) 0x7f1c14003c80 con 0x55b73d99a4d0 2024-05-27T07:33:27.568 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:33:27.571+0000 7f1c2c214740 10 client.16827 fetch_fsmap finished waiting for FSMap version 103 2024-05-27T07:33:27.568 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:33:27.571+0000 7f1c2c214740 10 client.16827 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-05-27T07:33:27.568 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:33:27.572+0000 7f1c2c214740 20 client.16827 populate_metadata read hostname 'smithi052' 2024-05-27T07:33:27.568 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:33:27.572+0000 7f1c2c214740 1 --2- 172.21.15.52:0/962688880 >> [v2:172.21.15.105:6838/500348151,v1:172.21.15.105:6839/500348151] conn(0x55b73d979d80 0x55b73dd79110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:33:27.568 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:33:27.572+0000 7f1c2c214740 4 client.16827 mds_command: new command op to 24527 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-27T07:33:27.568 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:33:27.572+0000 7f1c2c214740 1 -- 172.21.15.52:0/962688880 --> [v2:172.21.15.105:6838/500348151,v1:172.21.15.105:6839/500348151] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55b73dcd0b80 con 0x55b73d979d80 2024-05-27T07:33:27.568 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:33:27.572+0000 7f1c1effd640 1 --2- 172.21.15.52:0/962688880 >> [v2:172.21.15.105:6838/500348151,v1:172.21.15.105:6839/500348151] conn(0x55b73d979d80 0x55b73dd79110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:33:27.568 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:33:27.572+0000 7f1c02ff5640 10 client.16827 ms_handle_connect on v2:172.21.15.105:6838/500348151 2024-05-27T07:33:27.568 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:33:27.572+0000 7f1c1effd640 1 --2- 172.21.15.52:0/962688880 >> [v2:172.21.15.105:6838/500348151,v1:172.21.15.105:6839/500348151] conn(0x55b73d979d80 0x55b73dd79110 crc :-1 s=READY pgs=139 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=26e82cab0df915a3 server_cookie=2fcf98d98d3a2fb6 in_seq=0 out_seq=0 2024-05-27T07:33:27.569 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:33:27.574+0000 7f1c02ff5640 1 -- 172.21.15.52:0/962688880 <== mds.0 v2:172.21.15.105:6838/500348151 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24041 (crc 0 0 0) 0x55b73dcd0b80 con 0x55b73d979d80 2024-05-27T07:33:27.570 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:33:27.574+0000 7f1c02ff5640 10 client.16827 handle_command_reply: tid=0 2024-05-27T07:33:27.570 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:33:27.574+0000 7f1c2c214740 1 client.16827 shutdown 2024-05-27T07:33:27.570 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:33:27.574+0000 7f1c2c214740 1 -- 172.21.15.52:0/962688880 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55b73d99a4d0 msgr2=0x55b73dd66ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:33:27.570 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:33:27.574+0000 7f1c2c214740 1 --2- 172.21.15.52:0/962688880 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55b73d99a4d0 0x55b73dd66ef0 secure :-1 s=READY pgs=689 cs=0 l=1 rev1=1 crypto rx=0x7f1c140047b0 tx=0x7f1c140136b0 comp rx=0 tx=0).stop 2024-05-27T07:33:27.570 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:33:27.574+0000 7f1c2c214740 1 -- 172.21.15.52:0/962688880 shutdown_connections 2024-05-27T07:33:27.570 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:33:27.574+0000 7f1c2c214740 1 --2- 172.21.15.52:0/962688880 >> [v2:172.21.15.105:6838/500348151,v1:172.21.15.105:6839/500348151] conn(0x55b73d979d80 0x55b73dd79110 crc :-1 s=READY pgs=139 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:33:27.570 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:33:27.574+0000 7f1c2c214740 1 --2- 172.21.15.52:0/962688880 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55b73d9bb1b0 0x55b73dd728d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:33:27.570 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:33:27.574+0000 7f1c02ff5640 0 client.16827 ms_handle_reset on v2:172.21.15.105:6838/500348151 2024-05-27T07:33:27.570 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:33:27.574+0000 7f1c2c214740 1 --2- 172.21.15.52:0/962688880 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55b73d99a4d0 0x55b73dd66ef0 unknown :-1 s=CLOSED pgs=689 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:33:27.570 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:33:27.574+0000 7f1c2c214740 1 --2- 172.21.15.52:0/962688880 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55b73d9922f0 0x55b73dd669b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:33:27.570 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:33:27.574+0000 7f1c2c214740 1 -- 172.21.15.52:0/962688880 >> 172.21.15.52:0/962688880 conn(0x55b73d9af300 msgr2=0x55b73dd5e0f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:33:27.570 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:33:27.574+0000 7f1c2c214740 1 -- 172.21.15.52:0/962688880 shutdown_connections 2024-05-27T07:33:27.571 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:33:27.574+0000 7f1c2c214740 1 -- 172.21.15.52:0/962688880 wait complete. 2024-05-27T07:33:27.571 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:33:27.574+0000 7f1c2c214740 20 client.16827 trim_cache size 0 max 16384 2024-05-27T07:33:27.585 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:33:27.590+0000 7f1c2c214740 1 Processor -- start 2024-05-27T07:33:27.585 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:33:27.590+0000 7f1c2c214740 1 -- start start 2024-05-27T07:33:27.585 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:33:27.590+0000 7f1c2c214740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55b73d979d80 0x55b73dd58c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:33:27.585 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:33:27.590+0000 7f1c2c214740 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55b73d9922f0 0x55b73dd60130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:33:27.585 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:33:27.590+0000 7f1c2c214740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55b73d99a4d0 0x55b73dd60670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:33:27.585 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:33:27.590+0000 7f1c2c214740 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x7f1c201c90a0 con 0x55b73d9922f0 2024-05-27T07:33:27.585 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:33:27.590+0000 7f1c2c214740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f1c201c89a0 con 0x55b73d99a4d0 2024-05-27T07:33:27.585 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:33:27.590+0000 7f1c2c214740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f1c201c7b10 con 0x55b73d979d80 2024-05-27T07:33:27.585 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:33:27.590+0000 7f1c1e7fc640 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55b73d9922f0 0x55b73dd60130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:33:27.585 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:33:27.590+0000 7f1c1e7fc640 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55b73d9922f0 0x55b73dd60130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.52:50508/0 (socket says 172.21.15.52:50508) 2024-05-27T07:33:27.585 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:33:27.590+0000 7f1c1e7fc640 1 -- 172.21.15.52:0/3333427302 learned_addr learned my addr 172.21.15.52:0/3333427302 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-05-27T07:33:27.586 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:33:27.590+0000 7f1c1f7fe640 1 --2- 172.21.15.52:0/3333427302 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55b73d99a4d0 0x55b73dd60670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:33:27.586 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:33:27.590+0000 7f1c1effd640 1 --2- 172.21.15.52:0/3333427302 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55b73d979d80 0x55b73dd58c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:33:27.586 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:33:27.590+0000 7f1c1e7fc640 1 -- 172.21.15.52:0/3333427302 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55b73d979d80 msgr2=0x55b73dd58c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:33:27.586 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:33:27.590+0000 7f1c1e7fc640 1 --2- 172.21.15.52:0/3333427302 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55b73d979d80 0x55b73dd58c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:33:27.586 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:33:27.590+0000 7f1c1e7fc640 1 -- 172.21.15.52:0/3333427302 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55b73d99a4d0 msgr2=0x55b73dd60670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:33:27.586 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:33:27.590+0000 7f1c1e7fc640 1 --2- 172.21.15.52:0/3333427302 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55b73d99a4d0 0x55b73dd60670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:33:27.586 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:33:27.590+0000 7f1c1e7fc640 1 -- 172.21.15.52:0/3333427302 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55b73dd65ac0 con 0x55b73d9922f0 2024-05-27T07:33:27.586 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:33:27.591+0000 7f1c1e7fc640 1 --2- 172.21.15.52:0/3333427302 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55b73d9922f0 0x55b73dd60130 secure :-1 s=READY pgs=690 cs=0 l=1 rev1=1 crypto rx=0x7f1c1400bcc0 tx=0x7f1c140138b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bc5dc142ed01f4c6 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:33:27.586 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:33:27.591+0000 7f1c04ff9640 1 -- 172.21.15.52:0/3333427302 <== mon.0 v2:172.21.15.52:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1c14023580 con 0x55b73d9922f0 2024-05-27T07:33:27.586 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:33:27.591+0000 7f1c04ff9640 1 -- 172.21.15.52:0/3333427302 <== mon.0 v2:172.21.15.52:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f1c14018070 con 0x55b73d9922f0 2024-05-27T07:33:27.586 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:33:27.591+0000 7f1c04ff9640 1 -- 172.21.15.52:0/3333427302 <== mon.0 v2:172.21.15.52:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1c14012e40 con 0x55b73d9922f0 2024-05-27T07:33:27.589 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:33:27.592+0000 7f1c2c214740 1 -- 172.21.15.52:0/3333427302 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55b73d9922f0 msgr2=0x55b73dd60130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:33:27.589 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:33:27.592+0000 7f1c2c214740 1 --2- 172.21.15.52:0/3333427302 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55b73d9922f0 0x55b73dd60130 secure :-1 s=READY pgs=690 cs=0 l=1 rev1=1 crypto rx=0x7f1c1400bcc0 tx=0x7f1c140138b0 comp rx=0 tx=0).stop 2024-05-27T07:33:27.590 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:33:27.592+0000 7f1c2c214740 1 -- 172.21.15.52:0/3333427302 shutdown_connections 2024-05-27T07:33:27.590 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:33:27.592+0000 7f1c2c214740 1 --2- 172.21.15.52:0/3333427302 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55b73d99a4d0 0x55b73dd60670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:33:27.590 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:33:27.592+0000 7f1c2c214740 1 --2- 172.21.15.52:0/3333427302 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55b73d9922f0 0x55b73dd60130 unknown :-1 s=CLOSED pgs=690 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:33:27.590 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:33:27.592+0000 7f1c2c214740 1 --2- 172.21.15.52:0/3333427302 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55b73d979d80 0x55b73dd58c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:33:27.590 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:33:27.592+0000 7f1c2c214740 1 -- 172.21.15.52:0/3333427302 >> 172.21.15.52:0/3333427302 conn(0x55b73d9af300 msgr2=0x55b73dd58540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:33:27.590 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:33:27.592+0000 7f1c2c214740 1 -- 172.21.15.52:0/3333427302 shutdown_connections 2024-05-27T07:33:27.590 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:33:27.592+0000 7f1c2c214740 1 -- 172.21.15.52:0/3333427302 wait complete. 2024-05-27T07:33:27.590 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:33:27.592+0000 7f1c2c214740 1 -- start start 2024-05-27T07:33:27.590 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:33:27.592+0000 7f1c2c214740 1 Processor -- start 2024-05-27T07:33:27.590 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:33:27.592+0000 7f1c2c214740 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55b73d979d80 0x55b73dd571f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:33:27.590 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:33:27.592+0000 7f1c2c214740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55b73d9922f0 0x55b73dd57730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:33:27.590 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:33:27.592+0000 7f1c2c214740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55b73d99a4d0 0x55b73dd57c70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:33:27.590 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:33:27.592+0000 7f1c2c214740 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x55b73dd64d70 con 0x55b73d979d80 2024-05-27T07:33:27.591 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:33:27.592+0000 7f1c2c214740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f1c201c8210 con 0x55b73d99a4d0 2024-05-27T07:33:27.591 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:33:27.593+0000 7f1c2c214740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x55b73dd5d170 con 0x55b73d9922f0 2024-05-27T07:33:27.591 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:33:27.593+0000 7f1c1effd640 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55b73d979d80 0x55b73dd571f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:33:27.591 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:33:27.593+0000 7f1c1effd640 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55b73d979d80 0x55b73dd571f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.52:50510/0 (socket says 172.21.15.52:50510) 2024-05-27T07:33:27.591 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:33:27.593+0000 7f1c2c214740 10 client.0 _ll_register_callbacks cb 0x7f1c10009780 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T07:33:27.591 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:33:27.593+0000 7f1c1effd640 1 -- 172.21.15.52:0/2737213505 learned_addr learned my addr 172.21.15.52:0/2737213505 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-05-27T07:33:27.591 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:33:27.593+0000 7f1c1f7fe640 1 --2- 172.21.15.52:0/2737213505 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55b73d99a4d0 0x55b73dd57c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:33:27.591 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:33:27.593+0000 7f1c1e7fc640 1 --2- 172.21.15.52:0/2737213505 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55b73d9922f0 0x55b73dd57730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:33:27.591 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:33:27.593+0000 7f1c1effd640 1 -- 172.21.15.52:0/2737213505 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55b73d9922f0 msgr2=0x55b73dd57730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:33:27.591 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:33:27.593+0000 7f1c1effd640 1 --2- 172.21.15.52:0/2737213505 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55b73d9922f0 0x55b73dd57730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:33:27.591 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:33:27.593+0000 7f1c1effd640 1 -- 172.21.15.52:0/2737213505 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55b73d99a4d0 msgr2=0x55b73dd57c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:33:27.591 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:33:27.593+0000 7f1c1effd640 1 --2- 172.21.15.52:0/2737213505 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55b73d99a4d0 0x55b73dd57c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:33:27.592 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:33:27.593+0000 7f1c1effd640 1 -- 172.21.15.52:0/2737213505 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55b73dd6d8b0 con 0x55b73d979d80 2024-05-27T07:33:27.592 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:33:27.593+0000 7f1c1effd640 1 --2- 172.21.15.52:0/2737213505 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55b73d979d80 0x55b73dd571f0 secure :-1 s=READY pgs=691 cs=0 l=1 rev1=1 crypto rx=0x7f1c180a9a60 tx=0x7f1c1801a500 comp rx=0 tx=0).ready entity=mon.0 client_cookie=98a92e253386ef7a server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:33:27.592 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:33:27.593+0000 7f1c02ff5640 10 client.0 ms_handle_connect on v2:172.21.15.52:3300/0 2024-05-27T07:33:27.592 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:33:27.593+0000 7f1c02ff5640 1 -- 172.21.15.52:0/2737213505 <== mon.0 v2:172.21.15.52:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1c180a9ad0 con 0x55b73d979d80 2024-05-27T07:33:27.592 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:33:27.593+0000 7f1c02ff5640 1 -- 172.21.15.52:0/2737213505 <== mon.0 v2:172.21.15.52:3300/0 2 ==== osd_map(117..117 src has 1..117) ==== 9555+0+0 (secure 0 0 0) 0x7f1c180a60c0 con 0x55b73d979d80 2024-05-27T07:33:27.592 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:33:27.593+0000 7f1c2c214740 1 -- 172.21.15.52:0/2737213505 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f1c14018070 con 0x55b73d979d80 2024-05-27T07:33:27.592 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:33:27.593+0000 7f1c02ff5640 1 -- 172.21.15.52:0/2737213505 <== mon.0 v2:172.21.15.52:3300/0 3 ==== mon_get_version_reply(handle=1 version=103) ==== 24+0+0 (secure 0 0 0) 0x7f1c180ade80 con 0x55b73d979d80 2024-05-27T07:33:27.592 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:33:27.593+0000 7f1c2c214740 10 client.16833 fetch_fsmap learned FSMap version 103 2024-05-27T07:33:27.592 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:33:27.593+0000 7f1c2c214740 1 -- 172.21.15.52:0/2737213505 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({config=0+,fsmap=103,monmap=4+}) -- 0x55b73dd6eb80 con 0x55b73d979d80 2024-05-27T07:33:27.592 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:33:27.594+0000 7f1c02ff5640 1 -- 172.21.15.52:0/2737213505 <== mon.0 v2:172.21.15.52:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f1c14018070 con 0x55b73d979d80 2024-05-27T07:33:27.592 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:33:27.594+0000 7f1c02ff5640 1 -- 172.21.15.52:0/2737213505 <== mon.0 v2:172.21.15.52:3300/0 5 ==== fsmap(e 103) ==== 4215+0+0 (secure 0 0 0) 0x7f1c18022060 con 0x55b73d979d80 2024-05-27T07:33:27.592 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:33:27.594+0000 7f1c2c214740 10 client.16833 fetch_fsmap finished waiting for FSMap version 103 2024-05-27T07:33:27.592 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:33:27.594+0000 7f1c2c214740 10 client.16833 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-05-27T07:33:27.593 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:33:27.594+0000 7f1c2c214740 20 client.16833 populate_metadata read hostname 'smithi052' 2024-05-27T07:33:27.593 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:33:27.594+0000 7f1c2c214740 1 --2- 172.21.15.52:0/2737213505 >> [v2:172.21.15.105:6838/500348151,v1:172.21.15.105:6839/500348151] conn(0x55b73d9bb1b0 0x55b73dde37f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:33:27.593 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:33:27.594+0000 7f1c2c214740 4 client.16833 mds_command: new command op to 24527 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-05-27T07:33:27.593 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:33:27.594+0000 7f1c2c214740 1 -- 172.21.15.52:0/2737213505 --> [v2:172.21.15.105:6838/500348151,v1:172.21.15.105:6839/500348151] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55b73dcca360 con 0x55b73d9bb1b0 2024-05-27T07:33:27.593 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:33:27.594+0000 7f1c1e7fc640 1 --2- 172.21.15.52:0/2737213505 >> [v2:172.21.15.105:6838/500348151,v1:172.21.15.105:6839/500348151] conn(0x55b73d9bb1b0 0x55b73dde37f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:33:27.593 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:33:27.594+0000 7f1c02ff5640 10 client.16833 ms_handle_connect on v2:172.21.15.105:6838/500348151 2024-05-27T07:33:27.593 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:33:27.594+0000 7f1c1e7fc640 1 --2- 172.21.15.52:0/2737213505 >> [v2:172.21.15.105:6838/500348151,v1:172.21.15.105:6839/500348151] conn(0x55b73d9bb1b0 0x55b73dde37f0 crc :-1 s=READY pgs=140 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=c16a2686ba694d43 server_cookie=19fd2d3691525190 in_seq=0 out_seq=0 2024-05-27T07:33:28.483 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:33:28 smithi052 ceph-mon[31955]: pgmap v254: 129 pgs: 9 active+clean+snaptrim, 101 active+clean, 19 active+clean+snaptrim_wait; 659 MiB data, 4.0 GiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 2.1 MiB/s wr, 557 op/s 2024-05-27T07:33:28.483 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:33:28 smithi052 ceph-mon[31955]: from='mgr.15333 172.21.15.52:0/416389082' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-27T07:33:28.605 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:33:28 smithi173 ceph-mon[37000]: pgmap v254: 129 pgs: 9 active+clean+snaptrim, 101 active+clean, 19 active+clean+snaptrim_wait; 659 MiB data, 4.0 GiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 2.1 MiB/s wr, 557 op/s 2024-05-27T07:33:28.606 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:33:28 smithi173 ceph-mon[37000]: from='mgr.15333 172.21.15.52:0/416389082' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-27T07:33:28.694 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:33:28 smithi105 ceph-mon[35977]: pgmap v254: 129 pgs: 9 active+clean+snaptrim, 101 active+clean, 19 active+clean+snaptrim_wait; 659 MiB data, 4.0 GiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 2.1 MiB/s wr, 557 op/s 2024-05-27T07:33:28.694 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:33:28 smithi105 ceph-mon[35977]: from='mgr.15333 172.21.15.52:0/416389082' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-27T07:33:28.959 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:33:28.964+0000 7f1c02ff5640 1 -- 172.21.15.52:0/2737213505 <== mds.0 v2:172.21.15.105:6838/500348151 1 ==== command_reply(tid 0: 0 ) ==== 8+0+25 (crc 0 0 0) 0x55b73dcca360 con 0x55b73d9bb1b0 2024-05-27T07:33:28.959 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:33:28.964+0000 7f1c02ff5640 10 client.16833 handle_command_reply: tid=0 2024-05-27T07:33:28.959 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:33:28.965+0000 7f1c2c214740 1 client.16833 shutdown 2024-05-27T07:33:28.960 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:33:28.965+0000 7f1c2c214740 1 -- 172.21.15.52:0/2737213505 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55b73d979d80 msgr2=0x55b73dd571f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:33:28.960 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:33:28.965+0000 7f1c2c214740 1 --2- 172.21.15.52:0/2737213505 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55b73d979d80 0x55b73dd571f0 secure :-1 s=READY pgs=691 cs=0 l=1 rev1=1 crypto rx=0x7f1c180a9a60 tx=0x7f1c1801a500 comp rx=0 tx=0).stop 2024-05-27T07:33:28.960 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:33:28.965+0000 7f1c2c214740 1 -- 172.21.15.52:0/2737213505 shutdown_connections 2024-05-27T07:33:28.960 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:33:28.965+0000 7f1c2c214740 1 --2- 172.21.15.52:0/2737213505 >> [v2:172.21.15.105:6838/500348151,v1:172.21.15.105:6839/500348151] conn(0x55b73d9bb1b0 0x55b73dde37f0 crc :-1 s=READY pgs=140 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:33:28.960 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:33:28.965+0000 7f1c2c214740 1 --2- 172.21.15.52:0/2737213505 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55b73d99a4d0 0x55b73dd57c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:33:28.960 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:33:28.965+0000 7f1c2c214740 1 --2- 172.21.15.52:0/2737213505 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55b73d9922f0 0x55b73dd57730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:33:28.960 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:33:28.965+0000 7f1c02ff5640 0 client.16833 ms_handle_reset on v2:172.21.15.105:6838/500348151 2024-05-27T07:33:28.960 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:33:28.965+0000 7f1c2c214740 1 --2- 172.21.15.52:0/2737213505 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55b73d979d80 0x55b73dd571f0 unknown :-1 s=CLOSED pgs=691 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:33:28.960 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:33:28.965+0000 7f1c2c214740 1 -- 172.21.15.52:0/2737213505 >> 172.21.15.52:0/2737213505 conn(0x55b73d9af300 msgr2=0x55b73dd5e010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:33:28.960 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:33:28.965+0000 7f1c2c214740 1 -- 172.21.15.52:0/2737213505 shutdown_connections 2024-05-27T07:33:28.960 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:33:28.965+0000 7f1c2c214740 1 -- 172.21.15.52:0/2737213505 wait complete. 2024-05-27T07:33:28.960 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:33:28.965+0000 7f1c2c214740 20 client.16833 trim_cache size 0 max 16384 2024-05-27T07:33:28.961 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stdout:{ 2024-05-27T07:33:28.961 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stdout: "return_code": 0 2024-05-27T07:33:28.961 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stdout:} 2024-05-27T07:33:28.962 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:33:28.966+0000 7f1c2535e640 1 -- 172.21.15.52:0/1603374091 >> [v2:172.21.15.52:6800/1822318384,v1:172.21.15.52:6801/1822318384] conn(0x7f1c080b2a50 msgr2=0x7f1c080b4f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:33:28.962 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:33:28.966+0000 7f1c2535e640 1 --2- 172.21.15.52:0/1603374091 >> [v2:172.21.15.52:6800/1822318384,v1:172.21.15.52:6801/1822318384] conn(0x7f1c080b2a50 0x7f1c080b4f60 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x55b73dd59540 tx=0x7f1c10009260 comp rx=0 tx=0).stop 2024-05-27T07:33:28.962 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:33:28.966+0000 7f1c2535e640 1 -- 172.21.15.52:0/1603374091 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f1c20102b90 msgr2=0x7f1c20074980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:33:28.962 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:33:28.966+0000 7f1c2535e640 1 --2- 172.21.15.52:0/1603374091 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f1c20102b90 0x7f1c20074980 secure :-1 s=READY pgs=688 cs=0 l=1 rev1=1 crypto rx=0x7f1c1800eab0 tx=0x7f1c1800ef70 comp rx=0 tx=0).stop 2024-05-27T07:33:28.962 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:33:28.966+0000 7f1c2535e640 1 -- 172.21.15.52:0/1603374091 shutdown_connections 2024-05-27T07:33:28.962 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:33:28.966+0000 7f1c2535e640 1 --2- 172.21.15.52:0/1603374091 >> [v2:172.21.15.52:6800/1822318384,v1:172.21.15.52:6801/1822318384] conn(0x7f1c080b2a50 0x7f1c080b4f60 unknown :-1 s=CLOSED pgs=163 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:33:28.962 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:33:28.966+0000 7f1c2535e640 1 --2- 172.21.15.52:0/1603374091 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f1c20108010 0x7f1c2006e180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:33:28.963 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:33:28.966+0000 7f1c2535e640 1 --2- 172.21.15.52:0/1603374091 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f1c201075e0 0x7f1c2006dac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:33:28.963 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:33:28.966+0000 7f1c2535e640 1 --2- 172.21.15.52:0/1603374091 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f1c20102b90 0x7f1c20074980 unknown :-1 s=CLOSED pgs=688 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:33:28.963 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:33:28.966+0000 7f1c2535e640 1 -- 172.21.15.52:0/1603374091 >> 172.21.15.52:0/1603374091 conn(0x7f1c20078270 msgr2=0x7f1c20105100 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:33:28.963 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:33:28.966+0000 7f1c2535e640 1 -- 172.21.15.52:0/1603374091 shutdown_connections 2024-05-27T07:33:28.963 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:33:28.966+0000 7f1c2535e640 1 -- 172.21.15.52:0/1603374091 wait complete. 2024-05-27T07:33:30.061 DEBUG:teuthology.orchestra.run.smithi052:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:38f9d900c83c63799fbdbe61acc9a11b0d3554a6 shell --fsid 73df2636-1bf8-11ef-bc9b-c7b262605968 -- ceph tell mds.1:4 damage ls 2024-05-27T07:33:30.484 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:33:30 smithi052 ceph-mon[31955]: pgmap v255: 129 pgs: 6 active+clean+snaptrim, 114 active+clean, 9 active+clean+snaptrim_wait; 664 MiB data, 4.1 GiB used, 1.0 TiB / 1.0 TiB avail; 30 KiB/s rd, 2.3 MiB/s wr, 451 op/s 2024-05-27T07:33:30.525 INFO:teuthology.orchestra.run.smithi052.stderr:Inferring config /var/lib/ceph/73df2636-1bf8-11ef-bc9b-c7b262605968/mon.a/config 2024-05-27T07:33:30.606 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:33:30 smithi173 ceph-mon[37000]: pgmap v255: 129 pgs: 6 active+clean+snaptrim, 114 active+clean, 9 active+clean+snaptrim_wait; 664 MiB data, 4.1 GiB used, 1.0 TiB / 1.0 TiB avail; 30 KiB/s rd, 2.3 MiB/s wr, 451 op/s 2024-05-27T07:33:30.694 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:33:30 smithi105 ceph-mon[35977]: pgmap v255: 129 pgs: 6 active+clean+snaptrim, 114 active+clean, 9 active+clean+snaptrim_wait; 664 MiB data, 4.1 GiB used, 1.0 TiB / 1.0 TiB avail; 30 KiB/s rd, 2.3 MiB/s wr, 451 op/s 2024-05-27T07:33:32.483 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:33:32 smithi052 ceph-mon[31955]: pgmap v256: 129 pgs: 5 active+clean+snaptrim, 118 active+clean, 6 active+clean+snaptrim_wait; 657 MiB data, 4.1 GiB used, 1.0 TiB / 1.0 TiB avail; 22 KiB/s rd, 2.7 MiB/s wr, 419 op/s 2024-05-27T07:33:32.606 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:33:32 smithi173 ceph-mon[37000]: pgmap v256: 129 pgs: 5 active+clean+snaptrim, 118 active+clean, 6 active+clean+snaptrim_wait; 657 MiB data, 4.1 GiB used, 1.0 TiB / 1.0 TiB avail; 22 KiB/s rd, 2.7 MiB/s wr, 419 op/s 2024-05-27T07:33:32.694 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:33:32 smithi105 ceph-mon[35977]: pgmap v256: 129 pgs: 5 active+clean+snaptrim, 118 active+clean, 6 active+clean+snaptrim_wait; 657 MiB data, 4.1 GiB used, 1.0 TiB / 1.0 TiB avail; 22 KiB/s rd, 2.7 MiB/s wr, 419 op/s 2024-05-27T07:33:33.720 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:33.725+0000 7fb31cfed700 1 -- 172.21.15.52:0/3185042216 <== mon.0 v2:172.21.15.52:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb30c004230 con 0x7fb31810c5d0 2024-05-27T07:33:33.720 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:33.726+0000 7fb320253700 1 -- 172.21.15.52:0/3185042216 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fb31810c5d0 msgr2=0x7fb31810c9e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:33:33.720 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:33.726+0000 7fb320253700 1 --2- 172.21.15.52:0/3185042216 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fb31810c5d0 0x7fb31810c9e0 secure :-1 s=READY pgs=692 cs=0 l=1 rev1=1 crypto rx=0x7fb30c009990 tx=0x7fb30c009ca0 comp rx=0 tx=0).stop 2024-05-27T07:33:33.720 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:33.726+0000 7fb320253700 1 -- 172.21.15.52:0/3185042216 shutdown_connections 2024-05-27T07:33:33.721 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:33.726+0000 7fb320253700 1 --2- 172.21.15.52:0/3185042216 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fb318106ee0 0x7fb318113370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:33:33.721 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:33.726+0000 7fb320253700 1 --2- 172.21.15.52:0/3185042216 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fb31810c5d0 0x7fb31810c9e0 unknown :-1 s=CLOSED pgs=692 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:33:33.721 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:33.726+0000 7fb320253700 1 --2- 172.21.15.52:0/3185042216 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fb318106590 0x7fb3181069a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:33:33.721 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:33.726+0000 7fb320253700 1 -- 172.21.15.52:0/3185042216 >> 172.21.15.52:0/3185042216 conn(0x7fb318102070 msgr2=0x7fb3181044c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:33:33.723 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:33.729+0000 7fb320253700 1 -- 172.21.15.52:0/3185042216 shutdown_connections 2024-05-27T07:33:33.724 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:33.729+0000 7fb320253700 1 -- 172.21.15.52:0/3185042216 wait complete. 2024-05-27T07:33:33.724 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:33.730+0000 7fb320253700 1 Processor -- start 2024-05-27T07:33:33.725 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:33.730+0000 7fb320253700 1 -- start start 2024-05-27T07:33:33.726 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:33.731+0000 7fb320253700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fb318106590 0x7fb318111ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:33:33.726 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:33.731+0000 7fb320253700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fb318106ee0 0x7fb318112410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:33:33.726 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:33.731+0000 7fb320253700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fb31810c5d0 0x7fb318112950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:33:33.726 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:33.731+0000 7fb320253700 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x7fb318079db0 con 0x7fb31810c5d0 2024-05-27T07:33:33.726 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:33.731+0000 7fb320253700 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7fb318079c30 con 0x7fb318106ee0 2024-05-27T07:33:33.726 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:33.731+0000 7fb320253700 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7fb318079f30 con 0x7fb318106590 2024-05-27T07:33:33.726 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:33.731+0000 7fb31dfef700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fb318106590 0x7fb318111ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:33:33.726 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:33.731+0000 7fb31dfef700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fb318106590 0x7fb318111ed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.173:3300/0 says I am v2:172.21.15.52:56344/0 (socket says 172.21.15.52:56344) 2024-05-27T07:33:33.727 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:33.731+0000 7fb31dfef700 1 -- 172.21.15.52:0/2008278956 learned_addr learned my addr 172.21.15.52:0/2008278956 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-05-27T07:33:33.727 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:33.731+0000 7fb31e7f0700 1 --2- 172.21.15.52:0/2008278956 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fb31810c5d0 0x7fb318112950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:33:33.728 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:33.732+0000 7fb31d7ee700 1 --2- 172.21.15.52:0/2008278956 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fb318106ee0 0x7fb318112410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:33:33.728 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:33.732+0000 7fb31dfef700 1 -- 172.21.15.52:0/2008278956 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fb318106ee0 msgr2=0x7fb318112410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:33:33.728 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:33.732+0000 7fb31dfef700 1 --2- 172.21.15.52:0/2008278956 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fb318106ee0 0x7fb318112410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:33:33.728 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:33.732+0000 7fb31dfef700 1 -- 172.21.15.52:0/2008278956 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fb31810c5d0 msgr2=0x7fb318112950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:33:33.728 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:33.732+0000 7fb31dfef700 1 --2- 172.21.15.52:0/2008278956 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fb31810c5d0 0x7fb318112950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:33:33.728 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:33.732+0000 7fb31dfef700 1 -- 172.21.15.52:0/2008278956 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb318115820 con 0x7fb318106590 2024-05-27T07:33:33.728 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:33.732+0000 7fb31dfef700 1 --2- 172.21.15.52:0/2008278956 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fb318106590 0x7fb318111ed0 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7fb30c0098e0 tx=0x7fb30c0038b0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:33:33.728 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:33.733+0000 7fb316ffd700 1 -- 172.21.15.52:0/2008278956 <== mon.1 v2:172.21.15.173:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb30c027070 con 0x7fb318106590 2024-05-27T07:33:33.728 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:33.733+0000 7fb316ffd700 1 -- 172.21.15.52:0/2008278956 <== mon.1 v2:172.21.15.173:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fb30c00be40 con 0x7fb318106590 2024-05-27T07:33:33.729 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:33.734+0000 7fb316ffd700 1 -- 172.21.15.52:0/2008278956 <== mon.1 v2:172.21.15.173:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb30c004230 con 0x7fb318106590 2024-05-27T07:33:33.729 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:33.735+0000 7fb320253700 1 -- 172.21.15.52:0/2008278956 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb318115ab0 con 0x7fb318106590 2024-05-27T07:33:33.730 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:33.735+0000 7fb320253700 1 -- 172.21.15.52:0/2008278956 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb318115f90 con 0x7fb318106590 2024-05-27T07:33:33.738 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:33.742+0000 7fb316ffd700 1 -- 172.21.15.52:0/2008278956 <== mon.1 v2:172.21.15.173:3300/0 4 ==== mgrmap(e 26) ==== 149783+0+0 (secure 0 0 0) 0x7fb30c009d10 con 0x7fb318106590 2024-05-27T07:33:33.738 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:33.744+0000 7fb316ffd700 1 --2- 172.21.15.52:0/2008278956 >> [v2:172.21.15.52:6800/1822318384,v1:172.21.15.52:6801/1822318384] conn(0x7fb3000b29a0 0x7fb3000b4eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:33:33.738 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:33.744+0000 7fb32ec68b80 1 Processor -- start 2024-05-27T07:33:33.739 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:33.744+0000 7fb32ec68b80 1 -- start start 2024-05-27T07:33:33.739 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:33.745+0000 7fb31d7ee700 1 --2- 172.21.15.52:0/2008278956 >> [v2:172.21.15.52:6800/1822318384,v1:172.21.15.52:6801/1822318384] conn(0x7fb3000b29a0 0x7fb3000b4eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:33:33.739 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:33.745+0000 7fb316ffd700 1 -- 172.21.15.52:0/2008278956 <== mon.1 v2:172.21.15.173:3300/0 5 ==== osd_map(117..117 src has 1..117) ==== 9555+0+0 (secure 0 0 0) 0x7fb30c0e35d0 con 0x7fb318106590 2024-05-27T07:33:33.739 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:33.745+0000 7fb32ec68b80 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55d4946f1e40 0x55d494876b30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:33:33.741 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:33.746+0000 7fb32ec68b80 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55d494877070 0x55d494870be0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:33:33.741 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:33.746+0000 7fb31d7ee700 1 --2- 172.21.15.52:0/2008278956 >> [v2:172.21.15.52:6800/1822318384,v1:172.21.15.52:6801/1822318384] conn(0x7fb3000b29a0 0x7fb3000b4eb0 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7fb31806ace0 tx=0x7fb308009260 comp rx=0 tx=0).ready entity=mgr.15333 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:33:33.742 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:33.746+0000 7fb32ec68b80 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55d494871120 0x55d494871530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:33:33.742 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:33.746+0000 7fb31dfef700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55d4946f1e40 0x55d494876b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:33:33.742 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:33.746+0000 7fb31dfef700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55d4946f1e40 0x55d494876b30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.52:44564/0 (socket says 172.21.15.52:44564) 2024-05-27T07:33:33.742 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:33.746+0000 7fb31dfef700 1 -- 172.21.15.52:0/834663385 learned_addr learned my addr 172.21.15.52:0/834663385 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-05-27T07:33:33.742 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:33.746+0000 7fb32ec68b80 1 -- 172.21.15.52:0/834663385 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x55d49479ed70 con 0x55d4946f1e40 2024-05-27T07:33:33.742 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:33.746+0000 7fb32ec68b80 1 -- 172.21.15.52:0/834663385 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x55d49478e910 con 0x55d494871120 2024-05-27T07:33:33.742 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:33.746+0000 7fb32ec68b80 1 -- 172.21.15.52:0/834663385 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x55d494792950 con 0x55d494877070 2024-05-27T07:33:33.742 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:33.746+0000 7fb31e7f0700 1 --2- 172.21.15.52:0/834663385 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55d494871120 0x55d494871530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:33:33.742 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:33.746+0000 7fb31d7ee700 1 --2- 172.21.15.52:0/834663385 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55d494877070 0x55d494870be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:33:33.742 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:33.746+0000 7fb31e7f0700 1 -- 172.21.15.52:0/834663385 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55d494877070 msgr2=0x55d494870be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:33:33.742 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:33.746+0000 7fb31e7f0700 1 --2- 172.21.15.52:0/834663385 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55d494877070 0x55d494870be0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:33:33.742 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:33.746+0000 7fb31e7f0700 1 -- 172.21.15.52:0/834663385 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55d4946f1e40 msgr2=0x55d494876b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:33:33.743 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:33.746+0000 7fb31e7f0700 1 --2- 172.21.15.52:0/834663385 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55d4946f1e40 0x55d494876b30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:33:33.743 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:33.746+0000 7fb31e7f0700 1 -- 172.21.15.52:0/834663385 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55d494737fb0 con 0x55d494871120 2024-05-27T07:33:33.743 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:33.747+0000 7fb31e7f0700 1 --2- 172.21.15.52:0/834663385 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55d494871120 0x55d494871530 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7fb310009f70 tx=0x7fb310009650 comp rx=0 tx=0).ready entity=mon.2 client_cookie=2d0a183a2f516043 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:33:33.743 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:33.747+0000 7fb2fffff700 1 -- 172.21.15.52:0/834663385 <== mon.2 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb310014620 con 0x55d494871120 2024-05-27T07:33:33.743 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:33.747+0000 7fb2fffff700 1 -- 172.21.15.52:0/834663385 <== mon.2 v2:172.21.15.105:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fb310019070 con 0x55d494871120 2024-05-27T07:33:33.743 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:33.747+0000 7fb2fffff700 1 -- 172.21.15.52:0/834663385 <== mon.2 v2:172.21.15.105:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb310013980 con 0x55d494871120 2024-05-27T07:33:33.744 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:33.749+0000 7fb32ec68b80 1 -- 172.21.15.52:0/834663385 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55d494871120 msgr2=0x55d494871530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:33:33.744 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:33.749+0000 7fb32ec68b80 1 --2- 172.21.15.52:0/834663385 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55d494871120 0x55d494871530 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7fb310009f70 tx=0x7fb310009650 comp rx=0 tx=0).stop 2024-05-27T07:33:33.744 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:33.749+0000 7fb32ec68b80 1 -- 172.21.15.52:0/834663385 shutdown_connections 2024-05-27T07:33:33.744 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:33.749+0000 7fb32ec68b80 1 --2- 172.21.15.52:0/834663385 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55d494871120 0x55d494871530 unknown :-1 s=CLOSED pgs=214 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:33:33.744 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:33.749+0000 7fb32ec68b80 1 --2- 172.21.15.52:0/834663385 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55d4946f1e40 0x55d494876b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:33:33.744 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:33.749+0000 7fb32ec68b80 1 --2- 172.21.15.52:0/834663385 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55d494877070 0x55d494870be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:33:33.744 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:33.749+0000 7fb32ec68b80 1 -- 172.21.15.52:0/834663385 >> 172.21.15.52:0/834663385 conn(0x55d4946f4770 msgr2=0x55d49486bd90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:33:33.744 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:33.749+0000 7fb32ec68b80 1 -- 172.21.15.52:0/834663385 shutdown_connections 2024-05-27T07:33:33.744 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:33.749+0000 7fb32ec68b80 1 -- 172.21.15.52:0/834663385 wait complete. 2024-05-27T07:33:33.745 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:33.750+0000 7fb32ec68b80 1 -- start start 2024-05-27T07:33:33.745 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:33.750+0000 7fb32ec68b80 1 Processor -- start 2024-05-27T07:33:33.746 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:33.750+0000 7fb32ec68b80 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55d4946f1e40 0x55d49487a800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:33:33.746 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:33.751+0000 7fb32ec68b80 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55d494871120 0x55d49487ad40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:33:33.746 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:33.751+0000 7fb31dfef700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55d4946f1e40 0x55d49487a800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:33:33.746 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:33.751+0000 7fb31dfef700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55d4946f1e40 0x55d49487a800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.52:44572/0 (socket says 172.21.15.52:44572) 2024-05-27T07:33:33.746 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:33.751+0000 7fb32ec68b80 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55d494877070 0x55d49487b280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:33:33.746 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:33.751+0000 7fb31dfef700 1 -- 172.21.15.52:0/888392793 learned_addr learned my addr 172.21.15.52:0/888392793 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-05-27T07:33:33.746 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:33.751+0000 7fb32ec68b80 1 -- 172.21.15.52:0/888392793 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x55d494694fa0 con 0x55d4946f1e40 2024-05-27T07:33:33.746 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:33.751+0000 7fb32ec68b80 1 -- 172.21.15.52:0/888392793 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x55d4946f6da0 con 0x55d494877070 2024-05-27T07:33:33.746 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:33.751+0000 7fb32ec68b80 1 -- 172.21.15.52:0/888392793 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x55d49487dbb0 con 0x55d494871120 2024-05-27T07:33:33.746 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:33.751+0000 7fb32ec68b80 10 client.0 _ll_register_callbacks cb 0x55d4947255b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T07:33:33.746 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:33.751+0000 7fb31e7f0700 1 --2- 172.21.15.52:0/888392793 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55d494877070 0x55d49487b280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:33:33.746 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:33.751+0000 7fb31d7ee700 1 --2- 172.21.15.52:0/888392793 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55d494871120 0x55d49487ad40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:33:33.747 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:33.752+0000 7fb31e7f0700 1 -- 172.21.15.52:0/888392793 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55d494871120 msgr2=0x55d49487ad40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:33:33.747 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:33.752+0000 7fb31e7f0700 1 --2- 172.21.15.52:0/888392793 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55d494871120 0x55d49487ad40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:33:33.747 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:33.752+0000 7fb31e7f0700 1 -- 172.21.15.52:0/888392793 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55d4946f1e40 msgr2=0x55d49487a800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:33:33.747 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:33.752+0000 7fb31e7f0700 1 --2- 172.21.15.52:0/888392793 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55d4946f1e40 0x55d49487a800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:33:33.747 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:33.752+0000 7fb31e7f0700 1 -- 172.21.15.52:0/888392793 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55d4946de8d0 con 0x55d494877070 2024-05-27T07:33:33.747 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:33.752+0000 7fb31e7f0700 1 --2- 172.21.15.52:0/888392793 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55d494877070 0x55d49487b280 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7fb310009f70 tx=0x7fb310013ad0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=514f4eb616353c9c server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:33:33.748 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:33.752+0000 7fb2fd7fa700 10 client.0 ms_handle_connect on v2:172.21.15.105:3300/0 2024-05-27T07:33:33.748 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:33.753+0000 7fb2fd7fa700 1 -- 172.21.15.52:0/888392793 <== mon.2 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb310014620 con 0x55d494877070 2024-05-27T07:33:33.748 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:33.753+0000 7fb32ec68b80 1 -- 172.21.15.52:0/888392793 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fb310019070 con 0x55d494877070 2024-05-27T07:33:33.748 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:33.754+0000 7fb2fd7fa700 1 -- 172.21.15.52:0/888392793 <== mon.2 v2:172.21.15.105:3300/0 2 ==== osd_map(117..117 src has 1..117) ==== 9555+0+0 (secure 0 0 0) 0x7fb3100147c0 con 0x55d494877070 2024-05-27T07:33:33.748 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:33.754+0000 7fb2fd7fa700 1 -- 172.21.15.52:0/888392793 <== mon.2 v2:172.21.15.105:3300/0 3 ==== mon_get_version_reply(handle=1 version=103) ==== 24+0+0 (secure 0 0 0) 0x7fb310030050 con 0x55d494877070 2024-05-27T07:33:33.749 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:33.755+0000 7fb32ec68b80 10 client.26525 fetch_fsmap learned FSMap version 103 2024-05-27T07:33:33.749 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:33.755+0000 7fb32ec68b80 1 -- 172.21.15.52:0/888392793 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,fsmap=103,monmap=4+}) -- 0x55d494748760 con 0x55d494877070 2024-05-27T07:33:33.750 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:33.756+0000 7fb2fd7fa700 1 -- 172.21.15.52:0/888392793 <== mon.2 v2:172.21.15.105:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fb310019070 con 0x55d494877070 2024-05-27T07:33:33.750 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:33.756+0000 7fb2fd7fa700 1 -- 172.21.15.52:0/888392793 <== mon.2 v2:172.21.15.105:3300/0 5 ==== fsmap(e 103) ==== 4215+0+0 (secure 0 0 0) 0x7fb31001dab0 con 0x55d494877070 2024-05-27T07:33:33.751 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:33.756+0000 7fb32ec68b80 10 client.26525 fetch_fsmap finished waiting for FSMap version 103 2024-05-27T07:33:33.751 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:33.756+0000 7fb32ec68b80 10 client.26525 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.c 2024-05-27T07:33:33.751 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:33.756+0000 7fb32ec68b80 20 client.26525 populate_metadata read hostname 'smithi052' 2024-05-27T07:33:33.751 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:33.756+0000 7fb32ec68b80 1 --2- 172.21.15.52:0/888392793 >> [v2:172.21.15.173:6836/1396363827,v1:172.21.15.173:6837/1396363827] conn(0x55d49488ab30 0x55d49488cf80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:33:33.751 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:33.756+0000 7fb32ec68b80 4 client.26525 mds_command: new command op to 24466 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-27T07:33:33.751 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:33.756+0000 7fb32ec68b80 1 -- 172.21.15.52:0/888392793 --> [v2:172.21.15.173:6836/1396363827,v1:172.21.15.173:6837/1396363827] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55d4947a0cc0 con 0x55d49488ab30 2024-05-27T07:33:33.751 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:33.757+0000 7fb31dfef700 1 --2- 172.21.15.52:0/888392793 >> [v2:172.21.15.173:6836/1396363827,v1:172.21.15.173:6837/1396363827] conn(0x55d49488ab30 0x55d49488cf80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:33:33.752 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:33.758+0000 7fb31dfef700 1 --2- 172.21.15.52:0/888392793 >> [v2:172.21.15.173:6836/1396363827,v1:172.21.15.173:6837/1396363827] conn(0x55d49488ab30 0x55d49488cf80 crc :-1 s=READY pgs=44 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=d4b1933459abd556 server_cookie=6382df75bc2f66c8 in_seq=0 out_seq=0 2024-05-27T07:33:33.752 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:33.758+0000 7fb2fd7fa700 10 client.26525 ms_handle_connect on v2:172.21.15.173:6836/1396363827 2024-05-27T07:33:33.753 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:33.759+0000 7fb2fd7fa700 1 -- 172.21.15.52:0/888392793 <== mds.4 v2:172.21.15.173:6836/1396363827 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24041 (crc 0 0 0) 0x55d4947a0cc0 con 0x55d49488ab30 2024-05-27T07:33:33.753 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:33.759+0000 7fb2fd7fa700 10 client.26525 handle_command_reply: tid=0 2024-05-27T07:33:33.753 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:33.759+0000 7fb32ec68b80 1 client.26525 shutdown 2024-05-27T07:33:33.753 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:33.759+0000 7fb32ec68b80 1 -- 172.21.15.52:0/888392793 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55d494877070 msgr2=0x55d49487b280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:33:33.753 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:33.759+0000 7fb32ec68b80 1 --2- 172.21.15.52:0/888392793 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55d494877070 0x55d49487b280 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7fb310009f70 tx=0x7fb310013ad0 comp rx=0 tx=0).stop 2024-05-27T07:33:33.754 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:33.760+0000 7fb32ec68b80 1 -- 172.21.15.52:0/888392793 shutdown_connections 2024-05-27T07:33:33.754 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:33.760+0000 7fb32ec68b80 1 --2- 172.21.15.52:0/888392793 >> [v2:172.21.15.173:6836/1396363827,v1:172.21.15.173:6837/1396363827] conn(0x55d49488ab30 0x55d49488cf80 crc :-1 s=READY pgs=44 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:33:33.754 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:33.760+0000 7fb32ec68b80 1 --2- 172.21.15.52:0/888392793 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55d494877070 0x55d49487b280 unknown :-1 s=CLOSED pgs=215 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:33:33.754 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:33.760+0000 7fb2fd7fa700 0 client.26525 ms_handle_reset on v2:172.21.15.173:6836/1396363827 2024-05-27T07:33:33.755 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:33.760+0000 7fb32ec68b80 1 --2- 172.21.15.52:0/888392793 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55d4946f1e40 0x55d49487a800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:33:33.755 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:33.760+0000 7fb32ec68b80 1 --2- 172.21.15.52:0/888392793 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55d494871120 0x55d49487ad40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:33:33.755 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:33.760+0000 7fb32ec68b80 1 -- 172.21.15.52:0/888392793 >> 172.21.15.52:0/888392793 conn(0x55d4946f4770 msgr2=0x55d49486fbd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:33:33.756 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:33.762+0000 7fb32ec68b80 1 -- 172.21.15.52:0/888392793 shutdown_connections 2024-05-27T07:33:33.756 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:33.762+0000 7fb32ec68b80 1 -- 172.21.15.52:0/888392793 wait complete. 2024-05-27T07:33:33.757 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:33.763+0000 7fb32ec68b80 20 client.26525 trim_cache size 0 max 16384 2024-05-27T07:33:33.787 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:33.793+0000 7fb32ec68b80 1 Processor -- start 2024-05-27T07:33:33.787 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:33.793+0000 7fb32ec68b80 1 -- start start 2024-05-27T07:33:33.788 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:33.794+0000 7fb32ec68b80 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55d4946f1e40 0x55d494870be0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:33:33.789 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:33.794+0000 7fb32ec68b80 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55d494871120 0x55d494886c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:33:33.789 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:33.794+0000 7fb32ec68b80 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55d494877070 0x55d49487ad70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:33:33.789 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:33.794+0000 7fb32ec68b80 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x7fb3181c8030 con 0x55d4946f1e40 2024-05-27T07:33:33.789 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:33.794+0000 7fb32ec68b80 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7fb31811d680 con 0x55d494877070 2024-05-27T07:33:33.789 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:33.794+0000 7fb32ec68b80 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7fb3181c7930 con 0x55d494871120 2024-05-27T07:33:33.789 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:33.794+0000 7fb31d7ee700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55d494871120 0x55d494886c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:33:33.789 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:33.794+0000 7fb31d7ee700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55d494871120 0x55d494886c50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.173:3300/0 says I am v2:172.21.15.52:56376/0 (socket says 172.21.15.52:56376) 2024-05-27T07:33:33.789 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:33.794+0000 7fb31d7ee700 1 -- 172.21.15.52:0/695128645 learned_addr learned my addr 172.21.15.52:0/695128645 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-05-27T07:33:33.790 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:33.795+0000 7fb31d7ee700 1 -- 172.21.15.52:0/695128645 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55d494877070 msgr2=0x55d49487ad70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:33:33.791 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:33.795+0000 7fb31d7ee700 1 --2- 172.21.15.52:0/695128645 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55d494877070 0x55d49487ad70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:33:33.791 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:33.795+0000 7fb31d7ee700 1 -- 172.21.15.52:0/695128645 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55d4946f1e40 msgr2=0x55d494870be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:33:33.791 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:33.795+0000 7fb31d7ee700 1 --2- 172.21.15.52:0/695128645 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55d4946f1e40 0x55d494870be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:33:33.791 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:33.795+0000 7fb31d7ee700 1 -- 172.21.15.52:0/695128645 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55d494871880 con 0x55d494871120 2024-05-27T07:33:33.791 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:33.796+0000 7fb31d7ee700 1 --2- 172.21.15.52:0/695128645 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55d494871120 0x55d494886c50 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7fb318075b70 tx=0x7fb308014000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=804b4a441cc692cc server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:33:33.791 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:33.796+0000 7fb2ff7fe700 1 -- 172.21.15.52:0/695128645 <== mon.1 v2:172.21.15.173:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb308028440 con 0x55d494871120 2024-05-27T07:33:33.791 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:33.796+0000 7fb2ff7fe700 1 -- 172.21.15.52:0/695128645 <== mon.1 v2:172.21.15.173:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fb30801e8c0 con 0x55d494871120 2024-05-27T07:33:33.791 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:33.796+0000 7fb2ff7fe700 1 -- 172.21.15.52:0/695128645 <== mon.1 v2:172.21.15.173:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb308027710 con 0x55d494871120 2024-05-27T07:33:33.791 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:33.797+0000 7fb32ec68b80 1 -- 172.21.15.52:0/695128645 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55d494871120 msgr2=0x55d494886c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:33:33.792 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:33.798+0000 7fb32ec68b80 1 --2- 172.21.15.52:0/695128645 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55d494871120 0x55d494886c50 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7fb318075b70 tx=0x7fb308014000 comp rx=0 tx=0).stop 2024-05-27T07:33:33.792 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:33.798+0000 7fb32ec68b80 1 -- 172.21.15.52:0/695128645 shutdown_connections 2024-05-27T07:33:33.792 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:33.798+0000 7fb32ec68b80 1 --2- 172.21.15.52:0/695128645 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55d494877070 0x55d49487ad70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:33:33.792 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:33.798+0000 7fb32ec68b80 1 --2- 172.21.15.52:0/695128645 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55d4946f1e40 0x55d494870be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:33:33.792 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:33.798+0000 7fb32ec68b80 1 --2- 172.21.15.52:0/695128645 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55d494871120 0x55d494886c50 unknown :-1 s=CLOSED pgs=170 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:33:33.792 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:33.798+0000 7fb32ec68b80 1 -- 172.21.15.52:0/695128645 >> 172.21.15.52:0/695128645 conn(0x55d4946f4770 msgr2=0x55d494875dd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:33:33.793 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:33.798+0000 7fb32ec68b80 1 -- 172.21.15.52:0/695128645 shutdown_connections 2024-05-27T07:33:33.793 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:33.798+0000 7fb32ec68b80 1 -- 172.21.15.52:0/695128645 wait complete. 2024-05-27T07:33:33.793 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:33.799+0000 7fb32ec68b80 1 -- start start 2024-05-27T07:33:33.793 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:33.799+0000 7fb32ec68b80 1 Processor -- start 2024-05-27T07:33:33.795 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:33.800+0000 7fb32ec68b80 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55d4946f1e40 0x55d494883280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:33:33.795 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:33.800+0000 7fb32ec68b80 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55d494871120 0x55d494891d10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:33:33.795 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:33.801+0000 7fb31dfef700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55d4946f1e40 0x55d494883280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:33:33.795 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:33.801+0000 7fb31dfef700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55d4946f1e40 0x55d494883280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.52:53732/0 (socket says 172.21.15.52:53732) 2024-05-27T07:33:33.795 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:33.801+0000 7fb31dfef700 1 -- 172.21.15.52:0/2046827291 learned_addr learned my addr 172.21.15.52:0/2046827291 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-05-27T07:33:33.795 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:33.801+0000 7fb31d7ee700 1 --2- 172.21.15.52:0/2046827291 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55d494871120 0x55d494891d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:33:33.795 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:33.801+0000 7fb32ec68b80 1 --2- 172.21.15.52:0/2046827291 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55d494877070 0x55d494892250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:33:33.797 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:33.801+0000 7fb32ec68b80 1 -- 172.21.15.52:0/2046827291 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x7fb3181c71a0 con 0x55d494877070 2024-05-27T07:33:33.797 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:33.801+0000 7fb31e7f0700 1 --2- 172.21.15.52:0/2046827291 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55d494877070 0x55d494892250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:33:33.797 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:33.801+0000 7fb32ec68b80 1 -- 172.21.15.52:0/2046827291 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7fb31811dd50 con 0x55d4946f1e40 2024-05-27T07:33:33.797 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:33.801+0000 7fb32ec68b80 1 -- 172.21.15.52:0/2046827291 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x55d494883ab0 con 0x55d494871120 2024-05-27T07:33:33.797 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:33.802+0000 7fb32ec68b80 10 client.0 _ll_register_callbacks cb 0x55d4947255b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T07:33:33.797 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:33.802+0000 7fb31d7ee700 1 -- 172.21.15.52:0/2046827291 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55d4946f1e40 msgr2=0x55d494883280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:33:33.797 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:33.802+0000 7fb31d7ee700 1 --2- 172.21.15.52:0/2046827291 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55d4946f1e40 0x55d494883280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:33:33.797 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:33.802+0000 7fb31d7ee700 1 -- 172.21.15.52:0/2046827291 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55d494877070 msgr2=0x55d494892250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:33:33.797 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:33.802+0000 7fb31d7ee700 1 --2- 172.21.15.52:0/2046827291 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55d494877070 0x55d494892250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:33:33.797 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:33.802+0000 7fb31d7ee700 1 -- 172.21.15.52:0/2046827291 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55d494892810 con 0x55d494871120 2024-05-27T07:33:33.798 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:33.802+0000 7fb31d7ee700 1 --2- 172.21.15.52:0/2046827291 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55d494871120 0x55d494891d10 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7fb3080097d0 tx=0x7fb30801e820 comp rx=0 tx=0).ready entity=mon.1 client_cookie=fe5ddd088f4156a8 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:33:33.798 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:33.802+0000 7fb2fd7fa700 10 client.0 ms_handle_connect on v2:172.21.15.173:3300/0 2024-05-27T07:33:33.798 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:33.803+0000 7fb2fd7fa700 1 -- 172.21.15.52:0/2046827291 <== mon.1 v2:172.21.15.173:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb30801ed80 con 0x55d494871120 2024-05-27T07:33:33.798 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:33.803+0000 7fb2fd7fa700 1 -- 172.21.15.52:0/2046827291 <== mon.1 v2:172.21.15.173:3300/0 2 ==== osd_map(117..117 src has 1..117) ==== 9555+0+0 (secure 0 0 0) 0x7fb30802d070 con 0x55d494871120 2024-05-27T07:33:33.798 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:33.803+0000 7fb32ec68b80 1 -- 172.21.15.52:0/2046827291 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fb30801e8c0 con 0x55d494871120 2024-05-27T07:33:33.798 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:33.803+0000 7fb2fd7fa700 1 -- 172.21.15.52:0/2046827291 <== mon.1 v2:172.21.15.173:3300/0 3 ==== mon_get_version_reply(handle=1 version=103) ==== 24+0+0 (secure 0 0 0) 0x7fb30803d050 con 0x55d494871120 2024-05-27T07:33:33.798 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:33.804+0000 7fb32ec68b80 10 client.26350 fetch_fsmap learned FSMap version 103 2024-05-27T07:33:33.798 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:33.804+0000 7fb32ec68b80 1 -- 172.21.15.52:0/2046827291 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({config=0+,fsmap=103,monmap=4+}) -- 0x55d4948f5300 con 0x55d494871120 2024-05-27T07:33:33.800 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:33.806+0000 7fb2fd7fa700 1 -- 172.21.15.52:0/2046827291 <== mon.1 v2:172.21.15.173:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fb30801e8c0 con 0x55d494871120 2024-05-27T07:33:33.801 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:33.807+0000 7fb2fd7fa700 1 -- 172.21.15.52:0/2046827291 <== mon.1 v2:172.21.15.173:3300/0 5 ==== fsmap(e 103) ==== 4215+0+0 (secure 0 0 0) 0x7fb308026070 con 0x55d494871120 2024-05-27T07:33:33.801 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:33.807+0000 7fb32ec68b80 10 client.26350 fetch_fsmap finished waiting for FSMap version 103 2024-05-27T07:33:33.801 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:33.807+0000 7fb32ec68b80 10 client.26350 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.c 2024-05-27T07:33:33.801 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:33.807+0000 7fb32ec68b80 20 client.26350 populate_metadata read hostname 'smithi052' 2024-05-27T07:33:33.801 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:33.807+0000 7fb32ec68b80 1 --2- 172.21.15.52:0/2046827291 >> [v2:172.21.15.173:6836/1396363827,v1:172.21.15.173:6837/1396363827] conn(0x55d4948f5950 0x55d4948f7d70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:33:33.802 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:33.807+0000 7fb32ec68b80 4 client.26350 mds_command: new command op to 24466 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-05-27T07:33:33.802 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:33.807+0000 7fb32ec68b80 1 -- 172.21.15.52:0/2046827291 --> [v2:172.21.15.173:6836/1396363827,v1:172.21.15.173:6837/1396363827] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55d49479a4a0 con 0x55d4948f5950 2024-05-27T07:33:33.802 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:33.807+0000 7fb31dfef700 1 --2- 172.21.15.52:0/2046827291 >> [v2:172.21.15.173:6836/1396363827,v1:172.21.15.173:6837/1396363827] conn(0x55d4948f5950 0x55d4948f7d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:33:33.802 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:33.808+0000 7fb2fd7fa700 10 client.26350 ms_handle_connect on v2:172.21.15.173:6836/1396363827 2024-05-27T07:33:33.802 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:33.808+0000 7fb31dfef700 1 --2- 172.21.15.52:0/2046827291 >> [v2:172.21.15.173:6836/1396363827,v1:172.21.15.173:6837/1396363827] conn(0x55d4948f5950 0x55d4948f7d70 crc :-1 s=READY pgs=45 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=91d9cfd21e50e9da server_cookie=d20cffc7dea8afc3 in_seq=0 out_seq=0 2024-05-27T07:33:33.803 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:33.809+0000 7fb2fd7fa700 1 -- 172.21.15.52:0/2046827291 <== mds.4 v2:172.21.15.173:6836/1396363827 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55d49479a4a0 con 0x55d4948f5950 2024-05-27T07:33:33.803 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:33.809+0000 7fb2fd7fa700 10 client.26350 handle_command_reply: tid=0 2024-05-27T07:33:33.803 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:33.809+0000 7fb32ec68b80 1 client.26350 shutdown 2024-05-27T07:33:33.807 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:33.811+0000 7fb32ec68b80 1 -- 172.21.15.52:0/2046827291 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55d494871120 msgr2=0x55d494891d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:33:33.807 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:33.811+0000 7fb32ec68b80 1 --2- 172.21.15.52:0/2046827291 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55d494871120 0x55d494891d10 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7fb3080097d0 tx=0x7fb30801e820 comp rx=0 tx=0).stop 2024-05-27T07:33:33.809 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:33.815+0000 7fb32ec68b80 1 -- 172.21.15.52:0/2046827291 shutdown_connections 2024-05-27T07:33:33.809 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:33.815+0000 7fb32ec68b80 1 --2- 172.21.15.52:0/2046827291 >> [v2:172.21.15.173:6836/1396363827,v1:172.21.15.173:6837/1396363827] conn(0x55d4948f5950 0x55d4948f7d70 crc :-1 s=READY pgs=45 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:33:33.809 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:33.815+0000 7fb32ec68b80 1 --2- 172.21.15.52:0/2046827291 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55d494877070 0x55d494892250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:33:33.809 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:33.815+0000 7fb32ec68b80 1 --2- 172.21.15.52:0/2046827291 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55d4946f1e40 0x55d494883280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:33:33.809 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:33.815+0000 7fb32ec68b80 1 --2- 172.21.15.52:0/2046827291 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55d494871120 0x55d494891d10 unknown :-1 s=CLOSED pgs=171 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:33:33.809 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:33.815+0000 7fb32ec68b80 1 -- 172.21.15.52:0/2046827291 >> 172.21.15.52:0/2046827291 conn(0x55d4946f4770 msgr2=0x55d4948654f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:33:33.810 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:33.815+0000 7fb2fd7fa700 0 client.26350 ms_handle_reset on v2:172.21.15.173:6836/1396363827 2024-05-27T07:33:33.810 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:33.815+0000 7fb32ec68b80 1 -- 172.21.15.52:0/2046827291 shutdown_connections 2024-05-27T07:33:33.810 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:33.816+0000 7fb32ec68b80 1 -- 172.21.15.52:0/2046827291 wait complete. 2024-05-27T07:33:33.810 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:33.816+0000 7fb32ec68b80 20 client.26350 trim_cache size 0 max 16384 2024-05-27T07:33:33.811 INFO:teuthology.orchestra.run.smithi052.stdout:[] 2024-05-27T07:33:33.812 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:33.818+0000 7fb320253700 1 -- 172.21.15.52:0/2008278956 >> [v2:172.21.15.52:6800/1822318384,v1:172.21.15.52:6801/1822318384] conn(0x7fb3000b29a0 msgr2=0x7fb3000b4eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:33:33.812 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:33.818+0000 7fb320253700 1 --2- 172.21.15.52:0/2008278956 >> [v2:172.21.15.52:6800/1822318384,v1:172.21.15.52:6801/1822318384] conn(0x7fb3000b29a0 0x7fb3000b4eb0 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7fb31806ace0 tx=0x7fb308009260 comp rx=0 tx=0).stop 2024-05-27T07:33:33.812 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:33.818+0000 7fb320253700 1 -- 172.21.15.52:0/2008278956 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fb318106590 msgr2=0x7fb318111ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:33:33.812 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:33.818+0000 7fb320253700 1 --2- 172.21.15.52:0/2008278956 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fb318106590 0x7fb318111ed0 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7fb30c0098e0 tx=0x7fb30c0038b0 comp rx=0 tx=0).stop 2024-05-27T07:33:33.813 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:33.818+0000 7fb320253700 1 -- 172.21.15.52:0/2008278956 shutdown_connections 2024-05-27T07:33:33.813 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:33.818+0000 7fb320253700 1 --2- 172.21.15.52:0/2008278956 >> [v2:172.21.15.52:6800/1822318384,v1:172.21.15.52:6801/1822318384] conn(0x7fb3000b29a0 0x7fb3000b4eb0 unknown :-1 s=CLOSED pgs=164 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:33:33.813 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:33.818+0000 7fb320253700 1 --2- 172.21.15.52:0/2008278956 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fb31810c5d0 0x7fb318112950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:33:33.813 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:33.819+0000 7fb320253700 1 --2- 172.21.15.52:0/2008278956 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fb318106590 0x7fb318111ed0 unknown :-1 s=CLOSED pgs=169 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:33:33.813 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:33.819+0000 7fb320253700 1 --2- 172.21.15.52:0/2008278956 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fb318106ee0 0x7fb318112410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:33:33.813 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:33.819+0000 7fb320253700 1 -- 172.21.15.52:0/2008278956 >> 172.21.15.52:0/2008278956 conn(0x7fb318102070 msgr2=0x7fb31807c830 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:33:33.815 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:33.821+0000 7fb320253700 1 -- 172.21.15.52:0/2008278956 shutdown_connections 2024-05-27T07:33:33.815 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:33.821+0000 7fb320253700 1 -- 172.21.15.52:0/2008278956 wait complete. 2024-05-27T07:33:34.483 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:33:34 smithi052 ceph-mon[31955]: pgmap v257: 129 pgs: 5 active+clean+snaptrim, 118 active+clean, 6 active+clean+snaptrim_wait; 662 MiB data, 4.2 GiB used, 1.0 TiB / 1.0 TiB avail; 22 KiB/s rd, 2.4 MiB/s wr, 417 op/s 2024-05-27T07:33:34.606 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:33:34 smithi173 ceph-mon[37000]: pgmap v257: 129 pgs: 5 active+clean+snaptrim, 118 active+clean, 6 active+clean+snaptrim_wait; 662 MiB data, 4.2 GiB used, 1.0 TiB / 1.0 TiB avail; 22 KiB/s rd, 2.4 MiB/s wr, 417 op/s 2024-05-27T07:33:34.622 DEBUG:teuthology.orchestra.run.smithi052:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:38f9d900c83c63799fbdbe61acc9a11b0d3554a6 shell --fsid 73df2636-1bf8-11ef-bc9b-c7b262605968 -- ceph tell mds.1:3 damage ls 2024-05-27T07:33:34.694 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:33:34 smithi105 ceph-mon[35977]: pgmap v257: 129 pgs: 5 active+clean+snaptrim, 118 active+clean, 6 active+clean+snaptrim_wait; 662 MiB data, 4.2 GiB used, 1.0 TiB / 1.0 TiB avail; 22 KiB/s rd, 2.4 MiB/s wr, 417 op/s 2024-05-27T07:33:35.036 INFO:teuthology.orchestra.run.smithi052.stderr:Inferring config /var/lib/ceph/73df2636-1bf8-11ef-bc9b-c7b262605968/mon.a/config 2024-05-27T07:33:36.338 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:33:36 smithi052 ceph-mon[31955]: osdmap e118: 12 total, 12 up, 12 in 2024-05-27T07:33:36.338 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:33:36 smithi052 ceph-mon[31955]: pgmap v259: 129 pgs: 4 active+clean+snaptrim, 119 active+clean, 6 active+clean+snaptrim_wait; 654 MiB data, 4.2 GiB used, 1.0 TiB / 1.0 TiB avail; 27 KiB/s rd, 2.2 MiB/s wr, 521 op/s 2024-05-27T07:33:36.356 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:33:36 smithi173 ceph-mon[37000]: osdmap e118: 12 total, 12 up, 12 in 2024-05-27T07:33:36.356 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:33:36 smithi173 ceph-mon[37000]: pgmap v259: 129 pgs: 4 active+clean+snaptrim, 119 active+clean, 6 active+clean+snaptrim_wait; 654 MiB data, 4.2 GiB used, 1.0 TiB / 1.0 TiB avail; 27 KiB/s rd, 2.2 MiB/s wr, 521 op/s 2024-05-27T07:33:36.444 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:33:36 smithi105 ceph-mon[35977]: osdmap e118: 12 total, 12 up, 12 in 2024-05-27T07:33:36.444 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:33:36 smithi105 ceph-mon[35977]: pgmap v259: 129 pgs: 4 active+clean+snaptrim, 119 active+clean, 6 active+clean+snaptrim_wait; 654 MiB data, 4.2 GiB used, 1.0 TiB / 1.0 TiB avail; 27 KiB/s rd, 2.2 MiB/s wr, 521 op/s 2024-05-27T07:33:36.967 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:36.972+0000 7fc209ffb700 1 -- 172.21.15.52:0/2449230252 <== mon.0 v2:172.21.15.52:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc204011400 con 0x7fc20c10b010 2024-05-27T07:33:36.967 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:36.973+0000 7fc21186e700 1 -- 172.21.15.52:0/2449230252 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fc20c10b010 msgr2=0x7fc20c10b420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:33:36.967 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:36.973+0000 7fc21186e700 1 --2- 172.21.15.52:0/2449230252 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fc20c10b010 0x7fc20c10b420 secure :-1 s=READY pgs=693 cs=0 l=1 rev1=1 crypto rx=0x7fc20400b0d0 tx=0x7fc20400b3e0 comp rx=0 tx=0).stop 2024-05-27T07:33:36.967 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:36.973+0000 7fc21186e700 1 -- 172.21.15.52:0/2449230252 shutdown_connections 2024-05-27T07:33:36.968 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:36.973+0000 7fc21186e700 1 --2- 172.21.15.52:0/2449230252 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fc20c110290 0x7fc20c117bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:33:36.968 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:36.973+0000 7fc21186e700 1 --2- 172.21.15.52:0/2449230252 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fc20c10b010 0x7fc20c10b420 unknown :-1 s=CLOSED pgs=693 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:33:36.968 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:36.973+0000 7fc21186e700 1 --2- 172.21.15.52:0/2449230252 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fc20c10f940 0x7fc20c10fd50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:33:36.968 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:36.973+0000 7fc21186e700 1 -- 172.21.15.52:0/2449230252 >> 172.21.15.52:0/2449230252 conn(0x7fc20c06f980 msgr2=0x7fc20c06fd90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:33:36.968 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:36.973+0000 7fc21186e700 1 -- 172.21.15.52:0/2449230252 shutdown_connections 2024-05-27T07:33:36.968 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:36.973+0000 7fc21186e700 1 -- 172.21.15.52:0/2449230252 wait complete. 2024-05-27T07:33:36.968 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:36.974+0000 7fc21186e700 1 Processor -- start 2024-05-27T07:33:36.969 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:36.974+0000 7fc21186e700 1 -- start start 2024-05-27T07:33:36.969 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:36.975+0000 7fc21186e700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fc20c10b010 0x7fc20c1a8010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:33:36.969 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:36.975+0000 7fc21186e700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fc20c10f940 0x7fc20c1a0120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:33:36.969 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:36.975+0000 7fc21186e700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fc20c110290 0x7fc20c1a0660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:33:36.969 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:36.975+0000 7fc21186e700 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x7fc20c11ab80 con 0x7fc20c10b010 2024-05-27T07:33:36.969 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:36.975+0000 7fc21186e700 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7fc20c11aa00 con 0x7fc20c110290 2024-05-27T07:33:36.969 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:36.975+0000 7fc21186e700 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7fc20c11ad00 con 0x7fc20c10f940 2024-05-27T07:33:36.970 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:36.975+0000 7fc20b7fe700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fc20c110290 0x7fc20c1a0660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:33:36.970 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:36.975+0000 7fc20b7fe700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fc20c110290 0x7fc20c1a0660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.52:53754/0 (socket says 172.21.15.52:53754) 2024-05-27T07:33:36.970 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:36.975+0000 7fc20b7fe700 1 -- 172.21.15.52:0/1117936826 learned_addr learned my addr 172.21.15.52:0/1117936826 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-05-27T07:33:36.970 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:36.976+0000 7fc20a7fc700 1 --2- 172.21.15.52:0/1117936826 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fc20c10f940 0x7fc20c1a0120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:33:36.970 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:36.976+0000 7fc20b7fe700 1 -- 172.21.15.52:0/1117936826 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fc20c10f940 msgr2=0x7fc20c1a0120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:33:36.971 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:36.976+0000 7fc20b7fe700 1 --2- 172.21.15.52:0/1117936826 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fc20c10f940 0x7fc20c1a0120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:33:36.971 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:36.976+0000 7fc20b7fe700 1 -- 172.21.15.52:0/1117936826 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fc20c10b010 msgr2=0x7fc20c1a8010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:33:36.971 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:36.976+0000 7fc20b7fe700 1 --2- 172.21.15.52:0/1117936826 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fc20c10b010 0x7fc20c1a8010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:33:36.971 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:36.976+0000 7fc20b7fe700 1 -- 172.21.15.52:0/1117936826 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc20c1a0cd0 con 0x7fc20c110290 2024-05-27T07:33:36.971 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:36.976+0000 7fc20b7fe700 1 --2- 172.21.15.52:0/1117936826 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fc20c110290 0x7fc20c1a0660 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7fc1fc00d980 tx=0x7fc1fc00dd40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:33:36.973 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:36.978+0000 7fc21086c700 1 -- 172.21.15.52:0/1117936826 <== mon.2 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc1fc017070 con 0x7fc20c110290 2024-05-27T07:33:36.973 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:36.978+0000 7fc21186e700 1 -- 172.21.15.52:0/1117936826 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc20c1bead0 con 0x7fc20c110290 2024-05-27T07:33:36.975 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:36.979+0000 7fc21186e700 1 -- 172.21.15.52:0/1117936826 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc20c1beec0 con 0x7fc20c110290 2024-05-27T07:33:36.976 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:36.979+0000 7fc21086c700 1 -- 172.21.15.52:0/1117936826 <== mon.2 v2:172.21.15.105:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fc1fc009df0 con 0x7fc20c110290 2024-05-27T07:33:36.976 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:36.979+0000 7fc21086c700 1 -- 172.21.15.52:0/1117936826 <== mon.2 v2:172.21.15.105:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc1fc0053a0 con 0x7fc20c110290 2024-05-27T07:33:36.981 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:36.987+0000 7fc21086c700 1 -- 172.21.15.52:0/1117936826 <== mon.2 v2:172.21.15.105:3300/0 4 ==== mgrmap(e 26) ==== 149783+0+0 (secure 0 0 0) 0x7fc1fc005540 con 0x7fc20c110290 2024-05-27T07:33:36.981 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:36.987+0000 7fc220283b80 1 Processor -- start 2024-05-27T07:33:36.982 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:36.987+0000 7fc220283b80 1 -- start start 2024-05-27T07:33:36.982 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:36.988+0000 7fc220283b80 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5638038bee40 0x563803a43b30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:33:36.982 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:36.988+0000 7fc220283b80 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x563803a44070 0x563803a3dbe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:33:36.983 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:36.988+0000 7fc220283b80 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x563803a3e120 0x563803a3e530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:33:36.983 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:36.988+0000 7fc220283b80 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x56380396bd70 con 0x563803a44070 2024-05-27T07:33:36.983 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:36.988+0000 7fc220283b80 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x56380395b910 con 0x563803a3e120 2024-05-27T07:33:36.983 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:36.988+0000 7fc220283b80 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x56380395f950 con 0x5638038bee40 2024-05-27T07:33:36.983 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:36.988+0000 7fc20a7fc700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5638038bee40 0x563803a43b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:33:36.983 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:36.989+0000 7fc20affd700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x563803a44070 0x563803a3dbe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:33:36.984 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:36.989+0000 7fc20a7fc700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5638038bee40 0x563803a43b30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.173:3300/0 says I am v2:172.21.15.52:56408/0 (socket says 172.21.15.52:56408) 2024-05-27T07:33:36.984 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:36.989+0000 7fc20a7fc700 1 -- 172.21.15.52:0/2396684832 learned_addr learned my addr 172.21.15.52:0/2396684832 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-05-27T07:33:36.984 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:36.989+0000 7fc20a7fc700 1 --2- 172.21.15.52:0/2396684832 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x563803a3e120 0x563803a3e530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:33:36.986 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:36.990+0000 7fc20a7fc700 1 -- 172.21.15.52:0/2396684832 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x563803a3e120 msgr2=0x563803a3e530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:33:36.986 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:36.990+0000 7fc20a7fc700 1 --2- 172.21.15.52:0/2396684832 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x563803a3e120 0x563803a3e530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:33:36.986 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:36.990+0000 7fc20a7fc700 1 -- 172.21.15.52:0/2396684832 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x563803a44070 msgr2=0x563803a3dbe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:33:36.986 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:36.990+0000 7fc20a7fc700 1 --2- 172.21.15.52:0/2396684832 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x563803a44070 0x563803a3dbe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:33:36.986 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:36.990+0000 7fc20a7fc700 1 -- 172.21.15.52:0/2396684832 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x563803904fb0 con 0x5638038bee40 2024-05-27T07:33:36.986 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:36.991+0000 7fc21086c700 1 --2- 172.21.15.52:0/1117936826 >> [v2:172.21.15.52:6800/1822318384,v1:172.21.15.52:6801/1822318384] conn(0x7fc1f40b2b80 0x7fc1f40b5090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:33:36.986 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:36.991+0000 7fc20a7fc700 1 --2- 172.21.15.52:0/2396684832 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5638038bee40 0x563803a43b30 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7fc20000af30 tx=0x7fc20000a150 comp rx=0 tx=0).ready entity=mon.1 client_cookie=14d3f6397c44f023 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:33:36.986 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:36.991+0000 7fc21086c700 1 -- 172.21.15.52:0/1117936826 <== mon.2 v2:172.21.15.105:3300/0 5 ==== osd_map(118..118 src has 1..118) ==== 9479+0+0 (secure 0 0 0) 0x7fc1fc0e2890 con 0x7fc20c110290 2024-05-27T07:33:36.986 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:36.991+0000 7fc1f1ffb700 1 -- 172.21.15.52:0/2396684832 <== mon.1 v2:172.21.15.173:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc20000f040 con 0x5638038bee40 2024-05-27T07:33:36.987 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:36.991+0000 7fc1f1ffb700 1 -- 172.21.15.52:0/2396684832 <== mon.1 v2:172.21.15.173:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fc200015e10 con 0x5638038bee40 2024-05-27T07:33:36.987 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:36.991+0000 7fc1f1ffb700 1 -- 172.21.15.52:0/2396684832 <== mon.1 v2:172.21.15.173:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc20000f040 con 0x5638038bee40 2024-05-27T07:33:36.987 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:36.992+0000 7fc20affd700 1 --2- 172.21.15.52:0/1117936826 >> [v2:172.21.15.52:6800/1822318384,v1:172.21.15.52:6801/1822318384] conn(0x7fc1f40b2b80 0x7fc1f40b5090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:33:36.987 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:36.993+0000 7fc220283b80 1 -- 172.21.15.52:0/2396684832 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5638038bee40 msgr2=0x563803a43b30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:33:36.987 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:36.993+0000 7fc220283b80 1 --2- 172.21.15.52:0/2396684832 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5638038bee40 0x563803a43b30 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7fc20000af30 tx=0x7fc20000a150 comp rx=0 tx=0).stop 2024-05-27T07:33:36.987 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:36.993+0000 7fc20affd700 1 --2- 172.21.15.52:0/1117936826 >> [v2:172.21.15.52:6800/1822318384,v1:172.21.15.52:6801/1822318384] conn(0x7fc1f40b2b80 0x7fc1f40b5090 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7fc204009e60 tx=0x7fc20400b040 comp rx=0 tx=0).ready entity=mgr.15333 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:33:36.995 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:36.998+0000 7fc220283b80 1 -- 172.21.15.52:0/2396684832 shutdown_connections 2024-05-27T07:33:36.996 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:36.998+0000 7fc220283b80 1 --2- 172.21.15.52:0/2396684832 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x563803a3e120 0x563803a3e530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:33:36.996 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:36.998+0000 7fc220283b80 1 --2- 172.21.15.52:0/2396684832 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5638038bee40 0x563803a43b30 unknown :-1 s=CLOSED pgs=172 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:33:36.996 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:36.998+0000 7fc220283b80 1 --2- 172.21.15.52:0/2396684832 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x563803a44070 0x563803a3dbe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:33:36.996 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:36.998+0000 7fc220283b80 1 -- 172.21.15.52:0/2396684832 >> 172.21.15.52:0/2396684832 conn(0x5638038c1770 msgr2=0x563803a38d90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:33:36.996 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:37.002+0000 7fc220283b80 1 -- 172.21.15.52:0/2396684832 shutdown_connections 2024-05-27T07:33:37.001 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:37.007+0000 7fc220283b80 1 -- 172.21.15.52:0/2396684832 wait complete. 2024-05-27T07:33:37.001 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:37.007+0000 7fc220283b80 1 -- start start 2024-05-27T07:33:37.001 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:37.008+0000 7fc220283b80 1 Processor -- start 2024-05-27T07:33:37.002 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:37.008+0000 7fc220283b80 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x5638038bee40 0x563803a47830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:33:37.003 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:37.008+0000 7fc220283b80 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x563803a3e120 0x563803a47d70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:33:37.003 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:37.008+0000 7fc220283b80 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x563803a44070 0x563803a482b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:33:37.003 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:37.008+0000 7fc220283b80 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x563803861fa0 con 0x5638038bee40 2024-05-27T07:33:37.003 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:37.008+0000 7fc220283b80 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x5638038c3da0 con 0x563803a3e120 2024-05-27T07:33:37.003 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:37.008+0000 7fc220283b80 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x563803a4ab00 con 0x563803a44070 2024-05-27T07:33:37.003 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:37.008+0000 7fc220283b80 10 client.0 _ll_register_callbacks cb 0x5638038f25b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T07:33:37.003 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:37.009+0000 7fc20b7fe700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x563803a44070 0x563803a482b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:33:37.003 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:37.009+0000 7fc20b7fe700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x563803a44070 0x563803a482b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.173:3300/0 says I am v2:172.21.15.52:56412/0 (socket says 172.21.15.52:56412) 2024-05-27T07:33:37.003 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:37.009+0000 7fc20b7fe700 1 -- 172.21.15.52:0/2778816339 learned_addr learned my addr 172.21.15.52:0/2778816339 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-05-27T07:33:37.004 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:37.009+0000 7fc20a7fc700 1 --2- 172.21.15.52:0/2778816339 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x563803a3e120 0x563803a47d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:33:37.004 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:37.009+0000 7fc20affd700 1 --2- 172.21.15.52:0/2778816339 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x5638038bee40 0x563803a47830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:33:37.004 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:37.009+0000 7fc20b7fe700 1 -- 172.21.15.52:0/2778816339 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x563803a3e120 msgr2=0x563803a47d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:33:37.004 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:37.009+0000 7fc20b7fe700 1 --2- 172.21.15.52:0/2778816339 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x563803a3e120 0x563803a47d70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:33:37.004 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:37.009+0000 7fc20b7fe700 1 -- 172.21.15.52:0/2778816339 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x5638038bee40 msgr2=0x563803a47830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:33:37.004 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:37.009+0000 7fc20b7fe700 1 --2- 172.21.15.52:0/2778816339 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x5638038bee40 0x563803a47830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:33:37.005 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:37.009+0000 7fc20b7fe700 1 -- 172.21.15.52:0/2778816339 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5638038ab8d0 con 0x563803a44070 2024-05-27T07:33:37.005 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:37.009+0000 7fc20b7fe700 1 --2- 172.21.15.52:0/2778816339 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x563803a44070 0x563803a482b0 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7fc1fc0a94b0 tx=0x7fc1fc0a9400 comp rx=0 tx=0).ready entity=mon.1 client_cookie=b33252d98cf4045f server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:33:37.005 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:37.010+0000 7fc1eaffd700 10 client.0 ms_handle_connect on v2:172.21.15.173:3300/0 2024-05-27T07:33:37.006 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:37.010+0000 7fc1eaffd700 1 -- 172.21.15.52:0/2778816339 <== mon.1 v2:172.21.15.173:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc1fc0a97b0 con 0x563803a44070 2024-05-27T07:33:37.006 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:37.010+0000 7fc1eaffd700 1 -- 172.21.15.52:0/2778816339 <== mon.1 v2:172.21.15.173:3300/0 2 ==== osd_map(118..118 src has 1..118) ==== 9479+0+0 (secure 0 0 0) 0x7fc1fc0b2390 con 0x563803a44070 2024-05-27T07:33:37.006 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:37.011+0000 7fc220283b80 1 -- 172.21.15.52:0/2778816339 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fc200015e10 con 0x563803a44070 2024-05-27T07:33:37.006 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:37.011+0000 7fc1eaffd700 1 -- 172.21.15.52:0/2778816339 <== mon.1 v2:172.21.15.173:3300/0 3 ==== mon_get_version_reply(handle=1 version=103) ==== 24+0+0 (secure 0 0 0) 0x7fc1fc0adb60 con 0x563803a44070 2024-05-27T07:33:37.006 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:37.012+0000 7fc220283b80 10 client.26359 fetch_fsmap learned FSMap version 103 2024-05-27T07:33:37.006 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:37.012+0000 7fc220283b80 1 -- 172.21.15.52:0/2778816339 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({config=0+,fsmap=103,monmap=4+}) -- 0x563803915760 con 0x563803a44070 2024-05-27T07:33:37.008 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:37.012+0000 7fc1eaffd700 1 -- 172.21.15.52:0/2778816339 <== mon.1 v2:172.21.15.173:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fc1fc0b2610 con 0x563803a44070 2024-05-27T07:33:37.008 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:37.012+0000 7fc1eaffd700 1 -- 172.21.15.52:0/2778816339 <== mon.1 v2:172.21.15.173:3300/0 5 ==== fsmap(e 103) ==== 4215+0+0 (secure 0 0 0) 0x7fc1fc022c90 con 0x563803a44070 2024-05-27T07:33:37.008 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:37.012+0000 7fc220283b80 10 client.26359 fetch_fsmap finished waiting for FSMap version 103 2024-05-27T07:33:37.008 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:37.012+0000 7fc220283b80 10 client.26359 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.e 2024-05-27T07:33:37.008 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:37.012+0000 7fc220283b80 20 client.26359 populate_metadata read hostname 'smithi052' 2024-05-27T07:33:37.008 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:37.012+0000 7fc220283b80 1 --2- 172.21.15.52:0/2778816339 >> [v2:172.21.15.105:6834/3298498384,v1:172.21.15.105:6835/3298498384] conn(0x563803a57e30 0x563803a5a2a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:33:37.008 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:37.012+0000 7fc220283b80 4 client.26359 mds_command: new command op to 24473 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-27T07:33:37.008 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:37.012+0000 7fc220283b80 1 -- 172.21.15.52:0/2778816339 --> [v2:172.21.15.105:6834/3298498384,v1:172.21.15.105:6835/3298498384] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x7fc200015e10 con 0x563803a57e30 2024-05-27T07:33:37.008 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:37.014+0000 7fc20affd700 1 --2- 172.21.15.52:0/2778816339 >> [v2:172.21.15.105:6834/3298498384,v1:172.21.15.105:6835/3298498384] conn(0x563803a57e30 0x563803a5a2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:33:37.011 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:37.017+0000 7fc20affd700 1 --2- 172.21.15.52:0/2778816339 >> [v2:172.21.15.105:6834/3298498384,v1:172.21.15.105:6835/3298498384] conn(0x563803a57e30 0x563803a5a2a0 crc :-1 s=READY pgs=43 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=6e6c70fee92ce039 server_cookie=7bc495975f618b7e in_seq=0 out_seq=0 2024-05-27T07:33:37.011 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:37.018+0000 7fc1eaffd700 10 client.26359 ms_handle_connect on v2:172.21.15.105:6834/3298498384 2024-05-27T07:33:37.015 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:37.021+0000 7fc1eaffd700 1 -- 172.21.15.52:0/2778816339 <== mds.3 v2:172.21.15.105:6834/3298498384 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24041 (crc 0 0 0) 0x7fc200015e10 con 0x563803a57e30 2024-05-27T07:33:37.016 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:37.021+0000 7fc1eaffd700 10 client.26359 handle_command_reply: tid=0 2024-05-27T07:33:37.016 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:37.021+0000 7fc220283b80 1 client.26359 shutdown 2024-05-27T07:33:37.016 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:37.021+0000 7fc220283b80 1 -- 172.21.15.52:0/2778816339 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x563803a44070 msgr2=0x563803a482b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:33:37.016 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:37.021+0000 7fc220283b80 1 --2- 172.21.15.52:0/2778816339 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x563803a44070 0x563803a482b0 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7fc1fc0a94b0 tx=0x7fc1fc0a9400 comp rx=0 tx=0).stop 2024-05-27T07:33:37.016 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:37.021+0000 7fc220283b80 1 -- 172.21.15.52:0/2778816339 shutdown_connections 2024-05-27T07:33:37.016 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:37.021+0000 7fc220283b80 1 --2- 172.21.15.52:0/2778816339 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x563803a44070 0x563803a482b0 unknown :-1 s=CLOSED pgs=173 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:33:37.016 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:37.021+0000 7fc220283b80 1 --2- 172.21.15.52:0/2778816339 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x5638038bee40 0x563803a47830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:33:37.016 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:37.021+0000 7fc220283b80 1 --2- 172.21.15.52:0/2778816339 >> [v2:172.21.15.105:6834/3298498384,v1:172.21.15.105:6835/3298498384] conn(0x563803a57e30 0x563803a5a2a0 crc :-1 s=READY pgs=43 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:33:37.016 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:37.021+0000 7fc1eaffd700 0 client.26359 ms_handle_reset on v2:172.21.15.105:6834/3298498384 2024-05-27T07:33:37.016 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:37.021+0000 7fc220283b80 1 --2- 172.21.15.52:0/2778816339 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x563803a3e120 0x563803a47d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:33:37.016 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:37.021+0000 7fc220283b80 1 -- 172.21.15.52:0/2778816339 >> 172.21.15.52:0/2778816339 conn(0x5638038c1770 msgr2=0x563803a3cbd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:33:37.016 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:37.022+0000 7fc220283b80 1 -- 172.21.15.52:0/2778816339 shutdown_connections 2024-05-27T07:33:37.016 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:37.022+0000 7fc220283b80 1 -- 172.21.15.52:0/2778816339 wait complete. 2024-05-27T07:33:37.016 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:37.022+0000 7fc220283b80 20 client.26359 trim_cache size 0 max 16384 2024-05-27T07:33:37.044 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:37.050+0000 7fc220283b80 1 Processor -- start 2024-05-27T07:33:37.044 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:37.050+0000 7fc220283b80 1 -- start start 2024-05-27T07:33:37.045 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:37.051+0000 7fc220283b80 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x5638038bee40 0x563803a480f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:33:37.045 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:37.051+0000 7fc220283b80 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x563803a3e120 0x563803a43060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:33:37.045 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:37.051+0000 7fc220283b80 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x563803a44070 0x563803a53c00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:33:37.046 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:37.051+0000 7fc220283b80 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x7fc1f800f3e0 con 0x5638038bee40 2024-05-27T07:33:37.046 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:37.051+0000 7fc220283b80 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7fc1f800d780 con 0x563803a3e120 2024-05-27T07:33:37.046 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:37.051+0000 7fc220283b80 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7fc1f800ece0 con 0x563803a44070 2024-05-27T07:33:37.047 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:37.053+0000 7fc20affd700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x5638038bee40 0x563803a480f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:33:37.047 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:37.053+0000 7fc20affd700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x5638038bee40 0x563803a480f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.52:44626/0 (socket says 172.21.15.52:44626) 2024-05-27T07:33:37.047 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:37.053+0000 7fc20affd700 1 -- 172.21.15.52:0/3168520314 learned_addr learned my addr 172.21.15.52:0/3168520314 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-05-27T07:33:37.047 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:37.053+0000 7fc20a7fc700 1 --2- 172.21.15.52:0/3168520314 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x563803a3e120 0x563803a43060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:33:37.047 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:37.053+0000 7fc20b7fe700 1 --2- 172.21.15.52:0/3168520314 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x563803a44070 0x563803a53c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:33:37.048 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:37.054+0000 7fc20affd700 1 -- 172.21.15.52:0/3168520314 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x563803a44070 msgr2=0x563803a53c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:33:37.049 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:37.054+0000 7fc20affd700 1 --2- 172.21.15.52:0/3168520314 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x563803a44070 0x563803a53c00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:33:37.049 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:37.054+0000 7fc20affd700 1 -- 172.21.15.52:0/3168520314 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x563803a3e120 msgr2=0x563803a43060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:33:37.049 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:37.054+0000 7fc20affd700 1 --2- 172.21.15.52:0/3168520314 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x563803a3e120 0x563803a43060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:33:37.049 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:37.054+0000 7fc20affd700 1 -- 172.21.15.52:0/3168520314 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x563803a46b50 con 0x5638038bee40 2024-05-27T07:33:37.049 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:37.054+0000 7fc20affd700 1 --2- 172.21.15.52:0/3168520314 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x5638038bee40 0x563803a480f0 secure :-1 s=READY pgs=694 cs=0 l=1 rev1=1 crypto rx=0x7fc20400b0d0 tx=0x7fc20401c070 comp rx=0 tx=0).ready entity=mon.0 client_cookie=852a32b29626533c server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:33:37.049 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:37.055+0000 7fc1f17fa700 1 -- 172.21.15.52:0/3168520314 <== mon.0 v2:172.21.15.52:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc204036070 con 0x5638038bee40 2024-05-27T07:33:37.049 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:37.055+0000 7fc1f17fa700 1 -- 172.21.15.52:0/3168520314 <== mon.0 v2:172.21.15.52:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fc204003e10 con 0x5638038bee40 2024-05-27T07:33:37.050 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:37.055+0000 7fc1f17fa700 1 -- 172.21.15.52:0/3168520314 <== mon.0 v2:172.21.15.52:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc204015040 con 0x5638038bee40 2024-05-27T07:33:37.053 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:37.056+0000 7fc220283b80 1 -- 172.21.15.52:0/3168520314 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x5638038bee40 msgr2=0x563803a480f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:33:37.053 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:37.056+0000 7fc220283b80 1 --2- 172.21.15.52:0/3168520314 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x5638038bee40 0x563803a480f0 secure :-1 s=READY pgs=694 cs=0 l=1 rev1=1 crypto rx=0x7fc20400b0d0 tx=0x7fc20401c070 comp rx=0 tx=0).stop 2024-05-27T07:33:37.054 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:37.057+0000 7fc220283b80 1 -- 172.21.15.52:0/3168520314 shutdown_connections 2024-05-27T07:33:37.054 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:37.057+0000 7fc220283b80 1 --2- 172.21.15.52:0/3168520314 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x563803a44070 0x563803a53c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:33:37.054 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:37.057+0000 7fc220283b80 1 --2- 172.21.15.52:0/3168520314 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x5638038bee40 0x563803a480f0 secure :-1 s=CLOSED pgs=694 cs=0 l=1 rev1=1 crypto rx=0x7fc20400b0d0 tx=0x7fc20401c070 comp rx=0 tx=0).stop 2024-05-27T07:33:37.054 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:37.057+0000 7fc220283b80 1 --2- 172.21.15.52:0/3168520314 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x563803a3e120 0x563803a43060 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:33:37.054 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:37.057+0000 7fc220283b80 1 -- 172.21.15.52:0/3168520314 >> 172.21.15.52:0/3168520314 conn(0x5638038c1770 msgr2=0x563803a42950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:33:37.054 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:37.060+0000 7fc220283b80 1 -- 172.21.15.52:0/3168520314 shutdown_connections 2024-05-27T07:33:37.054 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:37.060+0000 7fc220283b80 1 -- 172.21.15.52:0/3168520314 wait complete. 2024-05-27T07:33:37.055 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:37.061+0000 7fc220283b80 1 -- start start 2024-05-27T07:33:37.055 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:37.061+0000 7fc220283b80 1 Processor -- start 2024-05-27T07:33:37.056 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:37.062+0000 7fc220283b80 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x563803a3e120 0x563803a50320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:33:37.056 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:37.062+0000 7fc220283b80 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x563803a44070 0x563803a5f150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:33:37.057 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:37.063+0000 7fc220283b80 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x563803a5f690 0x563803a5faa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:33:37.057 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:37.063+0000 7fc20affd700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x563803a3e120 0x563803a50320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:33:37.058 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:37.063+0000 7fc220283b80 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x7fc1f800e550 con 0x563803a44070 2024-05-27T07:33:37.058 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:37.063+0000 7fc220283b80 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7fc1f800de50 con 0x563803a3e120 2024-05-27T07:33:37.058 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:37.063+0000 7fc220283b80 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x563803a48630 con 0x563803a5f690 2024-05-27T07:33:37.058 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:37.063+0000 7fc20affd700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x563803a3e120 0x563803a50320 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.52:53798/0 (socket says 172.21.15.52:53798) 2024-05-27T07:33:37.058 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:37.063+0000 7fc20affd700 1 -- 172.21.15.52:0/3532070101 learned_addr learned my addr 172.21.15.52:0/3532070101 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-05-27T07:33:37.058 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:37.063+0000 7fc220283b80 10 client.0 _ll_register_callbacks cb 0x5638038f25b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T07:33:37.058 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:37.063+0000 7fc20b7fe700 1 --2- 172.21.15.52:0/3532070101 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x563803a5f690 0x563803a5faa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:33:37.058 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:37.063+0000 7fc20a7fc700 1 --2- 172.21.15.52:0/3532070101 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x563803a44070 0x563803a5f150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:33:37.060 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:37.064+0000 7fc20b7fe700 1 -- 172.21.15.52:0/3532070101 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x563803a3e120 msgr2=0x563803a50320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:33:37.060 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:37.064+0000 7fc20b7fe700 1 --2- 172.21.15.52:0/3532070101 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x563803a3e120 0x563803a50320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:33:37.060 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:37.064+0000 7fc20b7fe700 1 -- 172.21.15.52:0/3532070101 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x563803a44070 msgr2=0x563803a5f150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:33:37.060 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:37.064+0000 7fc20b7fe700 1 --2- 172.21.15.52:0/3532070101 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x563803a44070 0x563803a5f150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:33:37.060 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:37.064+0000 7fc20b7fe700 1 -- 172.21.15.52:0/3532070101 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({osdmap=0}) -- 0x563803ac1970 con 0x563803a5f690 2024-05-27T07:33:37.060 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:37.064+0000 7fc20b7fe700 1 --2- 172.21.15.52:0/3532070101 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x563803a5f690 0x563803a5faa0 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7fc1fc0224c0 tx=0x7fc1fc0a7a30 comp rx=0 tx=0).ready entity=mon.1 client_cookie=2c4ef6d13720fc29 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:33:37.060 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:37.064+0000 7fc1eaffd700 10 client.0 ms_handle_connect on v2:172.21.15.173:3300/0 2024-05-27T07:33:37.061 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:37.065+0000 7fc1eaffd700 1 -- 172.21.15.52:0/3532070101 <== mon.1 v2:172.21.15.173:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc1fc0c7030 con 0x563803a5f690 2024-05-27T07:33:37.061 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:37.065+0000 7fc1eaffd700 1 -- 172.21.15.52:0/3532070101 <== mon.1 v2:172.21.15.173:3300/0 2 ==== osd_map(118..118 src has 1..118) ==== 9479+0+0 (secure 0 0 0) 0x7fc1fc0a5d10 con 0x563803a5f690 2024-05-27T07:33:37.061 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:37.065+0000 7fc220283b80 1 -- 172.21.15.52:0/3532070101 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fc204003e10 con 0x563803a5f690 2024-05-27T07:33:37.061 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:37.065+0000 7fc1eaffd700 1 -- 172.21.15.52:0/3532070101 <== mon.1 v2:172.21.15.173:3300/0 3 ==== mon_get_version_reply(handle=1 version=103) ==== 24+0+0 (secure 0 0 0) 0x7fc1fc0a97b0 con 0x563803a5f690 2024-05-27T07:33:37.061 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:37.065+0000 7fc220283b80 10 client.26365 fetch_fsmap learned FSMap version 103 2024-05-27T07:33:37.061 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:37.065+0000 7fc220283b80 1 -- 172.21.15.52:0/3532070101 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({config=0+,fsmap=103,monmap=4+}) -- 0x563803ac2a50 con 0x563803a5f690 2024-05-27T07:33:37.061 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:37.065+0000 7fc1eaffd700 1 -- 172.21.15.52:0/3532070101 <== mon.1 v2:172.21.15.173:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fc1fc022300 con 0x563803a5f690 2024-05-27T07:33:37.061 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:37.065+0000 7fc1eaffd700 1 -- 172.21.15.52:0/3532070101 <== mon.1 v2:172.21.15.173:3300/0 5 ==== fsmap(e 103) ==== 4215+0+0 (secure 0 0 0) 0x7fc1fc0225b0 con 0x563803a5f690 2024-05-27T07:33:37.061 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:37.065+0000 7fc220283b80 10 client.26365 fetch_fsmap finished waiting for FSMap version 103 2024-05-27T07:33:37.061 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:37.066+0000 7fc220283b80 10 client.26365 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.e 2024-05-27T07:33:37.061 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:37.066+0000 7fc220283b80 20 client.26365 populate_metadata read hostname 'smithi052' 2024-05-27T07:33:37.061 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:37.066+0000 7fc220283b80 1 --2- 172.21.15.52:0/3532070101 >> [v2:172.21.15.105:6834/3298498384,v1:172.21.15.105:6835/3298498384] conn(0x563803ac3090 0x563803ac54b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:33:37.061 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:37.066+0000 7fc220283b80 4 client.26365 mds_command: new command op to 24473 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-05-27T07:33:37.061 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:37.066+0000 7fc220283b80 1 -- 172.21.15.52:0/3532070101 --> [v2:172.21.15.105:6834/3298498384,v1:172.21.15.105:6835/3298498384] -- command(tid 0: {"prefix": "damage ls"}) -- 0x7fc204003e10 con 0x563803ac3090 2024-05-27T07:33:37.062 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:37.066+0000 7fc20affd700 1 --2- 172.21.15.52:0/3532070101 >> [v2:172.21.15.105:6834/3298498384,v1:172.21.15.105:6835/3298498384] conn(0x563803ac3090 0x563803ac54b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:33:37.065 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:37.070+0000 7fc20affd700 1 --2- 172.21.15.52:0/3532070101 >> [v2:172.21.15.105:6834/3298498384,v1:172.21.15.105:6835/3298498384] conn(0x563803ac3090 0x563803ac54b0 crc :-1 s=READY pgs=44 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=1f9000754fdca47e server_cookie=4cc151eb9cce85e7 in_seq=0 out_seq=0 2024-05-27T07:33:37.065 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:37.070+0000 7fc1eaffd700 10 client.26365 ms_handle_connect on v2:172.21.15.105:6834/3298498384 2024-05-27T07:33:37.066 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:37.071+0000 7fc1eaffd700 1 -- 172.21.15.52:0/3532070101 <== mds.3 v2:172.21.15.105:6834/3298498384 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x7fc204003e10 con 0x563803ac3090 2024-05-27T07:33:37.066 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:37.071+0000 7fc1eaffd700 10 client.26365 handle_command_reply: tid=0 2024-05-27T07:33:37.066 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:37.071+0000 7fc220283b80 1 client.26365 shutdown 2024-05-27T07:33:37.067 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:37.072+0000 7fc220283b80 1 -- 172.21.15.52:0/3532070101 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x563803a5f690 msgr2=0x563803a5faa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:33:37.067 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:37.072+0000 7fc220283b80 1 --2- 172.21.15.52:0/3532070101 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x563803a5f690 0x563803a5faa0 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7fc1fc0224c0 tx=0x7fc1fc0a7a30 comp rx=0 tx=0).stop 2024-05-27T07:33:37.067 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:37.072+0000 7fc220283b80 1 -- 172.21.15.52:0/3532070101 shutdown_connections 2024-05-27T07:33:37.067 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:37.072+0000 7fc220283b80 1 --2- 172.21.15.52:0/3532070101 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x563803a5f690 0x563803a5faa0 unknown :-1 s=CLOSED pgs=174 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:33:37.067 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:37.072+0000 7fc220283b80 1 --2- 172.21.15.52:0/3532070101 >> [v2:172.21.15.105:6834/3298498384,v1:172.21.15.105:6835/3298498384] conn(0x563803ac3090 0x563803ac54b0 crc :-1 s=READY pgs=44 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:33:37.067 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:37.072+0000 7fc220283b80 1 --2- 172.21.15.52:0/3532070101 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x563803a3e120 0x563803a50320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:33:37.068 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:37.072+0000 7fc220283b80 1 --2- 172.21.15.52:0/3532070101 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x563803a44070 0x563803a5f150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:33:37.068 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:37.072+0000 7fc220283b80 1 -- 172.21.15.52:0/3532070101 >> 172.21.15.52:0/3532070101 conn(0x5638038c1770 msgr2=0x563803a324f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:33:37.068 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:37.072+0000 7fc1eaffd700 0 client.26365 ms_handle_reset on v2:172.21.15.105:6834/3298498384 2024-05-27T07:33:37.068 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:37.072+0000 7fc220283b80 1 -- 172.21.15.52:0/3532070101 shutdown_connections 2024-05-27T07:33:37.068 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:37.072+0000 7fc220283b80 1 -- 172.21.15.52:0/3532070101 wait complete. 2024-05-27T07:33:37.068 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:37.072+0000 7fc220283b80 20 client.26365 trim_cache size 0 max 16384 2024-05-27T07:33:37.068 INFO:teuthology.orchestra.run.smithi052.stdout:[] 2024-05-27T07:33:37.069 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:37.073+0000 7fc21186e700 1 -- 172.21.15.52:0/1117936826 >> [v2:172.21.15.52:6800/1822318384,v1:172.21.15.52:6801/1822318384] conn(0x7fc1f40b2b80 msgr2=0x7fc1f40b5090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:33:37.069 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:37.073+0000 7fc21186e700 1 --2- 172.21.15.52:0/1117936826 >> [v2:172.21.15.52:6800/1822318384,v1:172.21.15.52:6801/1822318384] conn(0x7fc1f40b2b80 0x7fc1f40b5090 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7fc204009e60 tx=0x7fc20400b040 comp rx=0 tx=0).stop 2024-05-27T07:33:37.069 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:37.073+0000 7fc21186e700 1 -- 172.21.15.52:0/1117936826 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fc20c110290 msgr2=0x7fc20c1a0660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:33:37.069 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:37.073+0000 7fc21186e700 1 --2- 172.21.15.52:0/1117936826 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fc20c110290 0x7fc20c1a0660 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7fc1fc00d980 tx=0x7fc1fc00dd40 comp rx=0 tx=0).stop 2024-05-27T07:33:37.069 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:37.074+0000 7fc21186e700 1 -- 172.21.15.52:0/1117936826 shutdown_connections 2024-05-27T07:33:37.069 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:37.074+0000 7fc21186e700 1 --2- 172.21.15.52:0/1117936826 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fc20c110290 0x7fc20c1a0660 unknown :-1 s=CLOSED pgs=216 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:33:37.069 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:37.074+0000 7fc21186e700 1 --2- 172.21.15.52:0/1117936826 >> [v2:172.21.15.52:6800/1822318384,v1:172.21.15.52:6801/1822318384] conn(0x7fc1f40b2b80 0x7fc1f40b5090 unknown :-1 s=CLOSED pgs=165 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:33:37.069 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:37.074+0000 7fc21186e700 1 --2- 172.21.15.52:0/1117936826 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fc20c10b010 0x7fc20c1a8010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:33:37.070 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:37.074+0000 7fc21186e700 1 --2- 172.21.15.52:0/1117936826 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fc20c10f940 0x7fc20c1a0120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:33:37.070 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:37.074+0000 7fc21186e700 1 -- 172.21.15.52:0/1117936826 >> 172.21.15.52:0/1117936826 conn(0x7fc20c06f980 msgr2=0x7fc20c10d380 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:33:37.070 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:37.075+0000 7fc21186e700 1 -- 172.21.15.52:0/1117936826 shutdown_connections 2024-05-27T07:33:37.070 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:37.075+0000 7fc21186e700 1 -- 172.21.15.52:0/1117936826 wait complete. 2024-05-27T07:33:37.802 DEBUG:teuthology.orchestra.run.smithi052:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:38f9d900c83c63799fbdbe61acc9a11b0d3554a6 shell --fsid 73df2636-1bf8-11ef-bc9b-c7b262605968 -- ceph tell mds.1:2 damage ls 2024-05-27T07:33:38.198 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:33:38 smithi052 ceph-mon[31955]: pgmap v260: 129 pgs: 2 active+clean+snaptrim, 121 active+clean, 6 active+clean+snaptrim_wait; 655 MiB data, 4.3 GiB used, 1.0 TiB / 1.0 TiB avail; 27 KiB/s rd, 2.0 MiB/s wr, 600 op/s 2024-05-27T07:33:38.225 INFO:teuthology.orchestra.run.smithi052.stderr:Inferring config /var/lib/ceph/73df2636-1bf8-11ef-bc9b-c7b262605968/mon.a/config 2024-05-27T07:33:38.606 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:33:38 smithi173 ceph-mon[37000]: pgmap v260: 129 pgs: 2 active+clean+snaptrim, 121 active+clean, 6 active+clean+snaptrim_wait; 655 MiB data, 4.3 GiB used, 1.0 TiB / 1.0 TiB avail; 27 KiB/s rd, 2.0 MiB/s wr, 600 op/s 2024-05-27T07:33:38.694 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:33:38 smithi105 ceph-mon[35977]: pgmap v260: 129 pgs: 2 active+clean+snaptrim, 121 active+clean, 6 active+clean+snaptrim_wait; 655 MiB data, 4.3 GiB used, 1.0 TiB / 1.0 TiB avail; 27 KiB/s rd, 2.0 MiB/s wr, 600 op/s 2024-05-27T07:33:39.483 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:33:39 smithi052 ceph-mon[31955]: from='mgr.15333 172.21.15.52:0/416389082' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T07:33:39.483 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:33:39 smithi052 ceph-mon[31955]: from='mgr.15333 172.21.15.52:0/416389082' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-27T07:33:39.483 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:33:39 smithi052 ceph-mon[31955]: from='mgr.15333 172.21.15.52:0/416389082' entity='mgr.x' 2024-05-27T07:33:39.606 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:33:39 smithi173 ceph-mon[37000]: from='mgr.15333 172.21.15.52:0/416389082' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T07:33:39.606 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:33:39 smithi173 ceph-mon[37000]: from='mgr.15333 172.21.15.52:0/416389082' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-27T07:33:39.606 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:33:39 smithi173 ceph-mon[37000]: from='mgr.15333 172.21.15.52:0/416389082' entity='mgr.x' 2024-05-27T07:33:39.694 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:33:39 smithi105 ceph-mon[35977]: from='mgr.15333 172.21.15.52:0/416389082' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T07:33:39.694 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:33:39 smithi105 ceph-mon[35977]: from='mgr.15333 172.21.15.52:0/416389082' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-27T07:33:39.694 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:33:39 smithi105 ceph-mon[35977]: from='mgr.15333 172.21.15.52:0/416389082' entity='mgr.x' 2024-05-27T07:33:40.483 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:33:40 smithi052 ceph-mon[31955]: pgmap v261: 129 pgs: 2 active+clean+snaptrim, 121 active+clean, 6 active+clean+snaptrim_wait; 659 MiB data, 4.3 GiB used, 1.0 TiB / 1.0 TiB avail; 22 KiB/s rd, 1.7 MiB/s wr, 608 op/s 2024-05-27T07:33:40.606 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:33:40 smithi173 ceph-mon[37000]: pgmap v261: 129 pgs: 2 active+clean+snaptrim, 121 active+clean, 6 active+clean+snaptrim_wait; 659 MiB data, 4.3 GiB used, 1.0 TiB / 1.0 TiB avail; 22 KiB/s rd, 1.7 MiB/s wr, 608 op/s 2024-05-27T07:33:40.694 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:33:40 smithi105 ceph-mon[35977]: pgmap v261: 129 pgs: 2 active+clean+snaptrim, 121 active+clean, 6 active+clean+snaptrim_wait; 659 MiB data, 4.3 GiB used, 1.0 TiB / 1.0 TiB avail; 22 KiB/s rd, 1.7 MiB/s wr, 608 op/s 2024-05-27T07:33:41.756 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:41.761+0000 7fddaa878700 1 -- 172.21.15.52:0/2653945870 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fdda410b720 msgr2=0x7fdda410bb30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:33:41.756 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:41.761+0000 7fddaa878700 1 --2- 172.21.15.52:0/2653945870 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fdda410b720 0x7fdda410bb30 secure :-1 s=READY pgs=695 cs=0 l=1 rev1=1 crypto rx=0x7fdd9800bb50 tx=0x7fdd9800be60 comp rx=0 tx=0).stop 2024-05-27T07:33:41.756 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:41.761+0000 7fddaa878700 1 -- 172.21.15.52:0/2653945870 shutdown_connections 2024-05-27T07:33:41.756 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:41.761+0000 7fddaa878700 1 --2- 172.21.15.52:0/2653945870 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fdda4110700 0x7fdda4118060 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:33:41.756 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:41.761+0000 7fddaa878700 1 --2- 172.21.15.52:0/2653945870 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fdda410b720 0x7fdda410bb30 unknown :-1 s=CLOSED pgs=695 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:33:41.756 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:41.761+0000 7fddaa878700 1 --2- 172.21.15.52:0/2653945870 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fdda410fdb0 0x7fdda41101c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:33:41.756 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:41.761+0000 7fddaa878700 1 -- 172.21.15.52:0/2653945870 >> 172.21.15.52:0/2653945870 conn(0x7fdda406f4c0 msgr2=0x7fdda406f900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:33:41.756 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:41.762+0000 7fddaa878700 1 -- 172.21.15.52:0/2653945870 shutdown_connections 2024-05-27T07:33:41.757 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:41.762+0000 7fddaa878700 1 -- 172.21.15.52:0/2653945870 wait complete. 2024-05-27T07:33:41.757 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:41.763+0000 7fddaa878700 1 Processor -- start 2024-05-27T07:33:41.757 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:41.763+0000 7fddaa878700 1 -- start start 2024-05-27T07:33:41.758 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:41.764+0000 7fddaa878700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fdda410b720 0x7fdda41a1420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:33:41.758 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:41.764+0000 7fddaa878700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fdda410fdb0 0x7fdda41a19c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:33:41.758 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:41.764+0000 7fddaa878700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fdda4110700 0x7fdda41a5c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:33:41.758 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:41.764+0000 7fddaa878700 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x7fdda411aaf0 con 0x7fdda4110700 2024-05-27T07:33:41.758 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:41.764+0000 7fddaa878700 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7fdda411a970 con 0x7fdda410fdb0 2024-05-27T07:33:41.759 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:41.764+0000 7fddaa878700 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7fdda411ac70 con 0x7fdda410b720 2024-05-27T07:33:41.759 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:41.764+0000 7fdda37fe700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fdda410fdb0 0x7fdda41a19c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:33:41.759 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:41.764+0000 7fdda8e15700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fdda4110700 0x7fdda41a5c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:33:41.759 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:41.764+0000 7fdda37fe700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fdda410fdb0 0x7fdda41a19c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.52:52212/0 (socket says 172.21.15.52:52212) 2024-05-27T07:33:41.759 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:41.764+0000 7fdda37fe700 1 -- 172.21.15.52:0/3507859883 learned_addr learned my addr 172.21.15.52:0/3507859883 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-05-27T07:33:41.759 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:41.764+0000 7fdda3fff700 1 --2- 172.21.15.52:0/3507859883 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fdda410b720 0x7fdda41a1420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:33:41.760 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:41.765+0000 7fdda37fe700 1 -- 172.21.15.52:0/3507859883 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fdda410b720 msgr2=0x7fdda41a1420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:33:41.760 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:41.765+0000 7fdda37fe700 1 --2- 172.21.15.52:0/3507859883 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fdda410b720 0x7fdda41a1420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:33:41.760 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:41.765+0000 7fdda37fe700 1 -- 172.21.15.52:0/3507859883 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fdda4110700 msgr2=0x7fdda41a5c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:33:41.760 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:41.765+0000 7fdda37fe700 1 --2- 172.21.15.52:0/3507859883 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fdda4110700 0x7fdda41a5c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:33:41.760 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:41.765+0000 7fdda37fe700 1 -- 172.21.15.52:0/3507859883 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdda41a62f0 con 0x7fdda410fdb0 2024-05-27T07:33:41.760 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:41.765+0000 7fdda8e15700 1 --2- 172.21.15.52:0/3507859883 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fdda4110700 0x7fdda41a5c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-27T07:33:41.761 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:41.765+0000 7fdda3fff700 1 --2- 172.21.15.52:0/3507859883 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fdda410b720 0x7fdda41a1420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-27T07:33:41.761 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:41.765+0000 7fdda37fe700 1 --2- 172.21.15.52:0/3507859883 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fdda410fdb0 0x7fdda41a19c0 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7fdd9c007ee0 tx=0x7fdd9c00d3b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:33:41.761 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:41.765+0000 7fdda17fa700 1 -- 172.21.15.52:0/3507859883 <== mon.2 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdd9c00ddd0 con 0x7fdda410fdb0 2024-05-27T07:33:41.761 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:41.765+0000 7fddaa878700 1 -- 172.21.15.52:0/3507859883 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fdda41a65e0 con 0x7fdda410fdb0 2024-05-27T07:33:41.761 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:41.766+0000 7fddaa878700 1 -- 172.21.15.52:0/3507859883 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fdda41b2310 con 0x7fdda410fdb0 2024-05-27T07:33:41.761 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:41.766+0000 7fdda17fa700 1 -- 172.21.15.52:0/3507859883 <== mon.2 v2:172.21.15.105:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fdd9c004bb0 con 0x7fdda410fdb0 2024-05-27T07:33:41.761 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:41.766+0000 7fdda17fa700 1 -- 172.21.15.52:0/3507859883 <== mon.2 v2:172.21.15.105:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdd9c004650 con 0x7fdda410fdb0 2024-05-27T07:33:41.767 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:41.771+0000 7fddb928db80 1 Processor -- start 2024-05-27T07:33:41.767 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:41.772+0000 7fdda17fa700 1 -- 172.21.15.52:0/3507859883 <== mon.2 v2:172.21.15.105:3300/0 4 ==== mgrmap(e 26) ==== 149783+0+0 (secure 0 0 0) 0x7fdd9c01c020 con 0x7fdda410fdb0 2024-05-27T07:33:41.767 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:41.773+0000 7fddb928db80 1 -- start start 2024-05-27T07:33:41.768 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:41.773+0000 7fddb928db80 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5586cfe89e40 0x5586d0015f20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:33:41.768 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:41.773+0000 7fddb928db80 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x5586d0016460 0x5586d0010250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:33:41.768 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:41.774+0000 7fddb928db80 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5586d0010790 0x5586d0010ba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:33:41.768 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:41.774+0000 7fddb928db80 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x5586cff36c90 con 0x5586d0016460 2024-05-27T07:33:41.768 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:41.774+0000 7fddb928db80 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x5586cff26830 con 0x5586cfe89e40 2024-05-27T07:33:41.768 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:41.774+0000 7fddb928db80 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x5586cff2a870 con 0x5586d0010790 2024-05-27T07:33:41.768 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:41.774+0000 7fdda3fff700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x5586d0016460 0x5586d0010250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:33:41.769 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:41.774+0000 7fdda8e15700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5586cfe89e40 0x5586d0015f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:33:41.769 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:41.774+0000 7fdda3fff700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x5586d0016460 0x5586d0010250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.52:56674/0 (socket says 172.21.15.52:56674) 2024-05-27T07:33:41.769 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:41.774+0000 7fdda3fff700 1 -- 172.21.15.52:0/2275565648 learned_addr learned my addr 172.21.15.52:0/2275565648 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-05-27T07:33:41.769 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:41.774+0000 7fdda37fe700 1 --2- 172.21.15.52:0/2275565648 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5586d0010790 0x5586d0010ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:33:41.770 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:41.775+0000 7fdda3fff700 1 -- 172.21.15.52:0/2275565648 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5586d0010790 msgr2=0x5586d0010ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:33:41.770 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:41.775+0000 7fdda3fff700 1 --2- 172.21.15.52:0/2275565648 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5586d0010790 0x5586d0010ba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:33:41.770 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:41.775+0000 7fdda3fff700 1 -- 172.21.15.52:0/2275565648 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5586cfe89e40 msgr2=0x5586d0015f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:33:41.770 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:41.775+0000 7fdda3fff700 1 --2- 172.21.15.52:0/2275565648 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5586cfe89e40 0x5586d0015f20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:33:41.770 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:41.775+0000 7fdda3fff700 1 -- 172.21.15.52:0/2275565648 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5586cfe768d0 con 0x5586d0016460 2024-05-27T07:33:41.770 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:41.775+0000 7fdda3fff700 1 --2- 172.21.15.52:0/2275565648 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x5586d0016460 0x5586d0010250 secure :-1 s=READY pgs=696 cs=0 l=1 rev1=1 crypto rx=0x7fdd9800bb50 tx=0x7fdd9800b140 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6ca4ca55bb2df4e6 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:33:41.771 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:41.776+0000 7fdda17fa700 1 --2- 172.21.15.52:0/3507859883 >> [v2:172.21.15.52:6800/1822318384,v1:172.21.15.52:6801/1822318384] conn(0x7fdd8c0b2af0 0x7fdd8c0b5000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:33:41.771 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:41.776+0000 7fdd8a7fc700 1 -- 172.21.15.52:0/2275565648 <== mon.0 v2:172.21.15.52:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdd98021070 con 0x5586d0016460 2024-05-27T07:33:41.771 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:41.776+0000 7fdd8a7fc700 1 -- 172.21.15.52:0/2275565648 <== mon.0 v2:172.21.15.52:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fdd98005040 con 0x5586d0016460 2024-05-27T07:33:41.771 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:41.777+0000 7fdd8a7fc700 1 -- 172.21.15.52:0/2275565648 <== mon.0 v2:172.21.15.52:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdd9800d9f0 con 0x5586d0016460 2024-05-27T07:33:41.771 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:41.777+0000 7fdda8e15700 1 --2- 172.21.15.52:0/3507859883 >> [v2:172.21.15.52:6800/1822318384,v1:172.21.15.52:6801/1822318384] conn(0x7fdd8c0b2af0 0x7fdd8c0b5000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:33:41.771 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:41.777+0000 7fdda17fa700 1 -- 172.21.15.52:0/3507859883 <== mon.2 v2:172.21.15.105:3300/0 5 ==== osd_map(118..118 src has 1..118) ==== 9479+0+0 (secure 0 0 0) 0x7fdd9c0e2330 con 0x7fdda410fdb0 2024-05-27T07:33:41.773 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:41.778+0000 7fdda8e15700 1 --2- 172.21.15.52:0/3507859883 >> [v2:172.21.15.52:6800/1822318384,v1:172.21.15.52:6801/1822318384] conn(0x7fdd8c0b2af0 0x7fdd8c0b5000 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x5586d00113c0 tx=0x7fdd94009260 comp rx=0 tx=0).ready entity=mgr.15333 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:33:41.774 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:41.778+0000 7fddb928db80 1 -- 172.21.15.52:0/2275565648 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x5586d0016460 msgr2=0x5586d0010250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:33:41.774 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:41.778+0000 7fddb928db80 1 --2- 172.21.15.52:0/2275565648 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x5586d0016460 0x5586d0010250 secure :-1 s=READY pgs=696 cs=0 l=1 rev1=1 crypto rx=0x7fdd9800bb50 tx=0x7fdd9800b140 comp rx=0 tx=0).stop 2024-05-27T07:33:41.774 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:41.779+0000 7fddb928db80 1 -- 172.21.15.52:0/2275565648 shutdown_connections 2024-05-27T07:33:41.774 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:41.779+0000 7fddb928db80 1 --2- 172.21.15.52:0/2275565648 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5586d0010790 0x5586d0010ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:33:41.774 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:41.779+0000 7fddb928db80 1 --2- 172.21.15.52:0/2275565648 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5586cfe89e40 0x5586d0015f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:33:41.774 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:41.779+0000 7fddb928db80 1 --2- 172.21.15.52:0/2275565648 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x5586d0016460 0x5586d0010250 unknown :-1 s=CLOSED pgs=696 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:33:41.774 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:41.779+0000 7fddb928db80 1 -- 172.21.15.52:0/2275565648 >> 172.21.15.52:0/2275565648 conn(0x5586cfee4af0 msgr2=0x5586d0002640 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:33:41.774 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:41.780+0000 7fddb928db80 1 -- 172.21.15.52:0/2275565648 shutdown_connections 2024-05-27T07:33:41.775 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:41.780+0000 7fddb928db80 1 -- 172.21.15.52:0/2275565648 wait complete. 2024-05-27T07:33:41.775 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:41.781+0000 7fddb928db80 1 -- start start 2024-05-27T07:33:41.775 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:41.781+0000 7fddb928db80 1 Processor -- start 2024-05-27T07:33:41.776 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:41.782+0000 7fddb928db80 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x5586cfe89e40 0x5586d0019b50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:33:41.777 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:41.782+0000 7fddb928db80 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5586d0010790 0x5586d001a090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:33:41.777 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:41.782+0000 7fddb928db80 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5586d0016460 0x5586d001a5d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:33:41.777 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:41.782+0000 7fdda37fe700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5586d0010790 0x5586d001a090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:33:41.777 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:41.782+0000 7fdda3fff700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x5586cfe89e40 0x5586d0019b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:33:41.777 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:41.782+0000 7fdda37fe700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5586d0010790 0x5586d001a090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.52:52238/0 (socket says 172.21.15.52:52238) 2024-05-27T07:33:41.777 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:41.782+0000 7fdda37fe700 1 -- 172.21.15.52:0/1370930526 learned_addr learned my addr 172.21.15.52:0/1370930526 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-05-27T07:33:41.777 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:41.783+0000 7fddb928db80 1 -- 172.21.15.52:0/1370930526 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x5586cfe2cfa0 con 0x5586cfe89e40 2024-05-27T07:33:41.777 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:41.783+0000 7fdda8e15700 1 --2- 172.21.15.52:0/1370930526 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5586d0016460 0x5586d001a5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:33:41.778 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:41.783+0000 7fddb928db80 1 -- 172.21.15.52:0/1370930526 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x5586cfec87c0 con 0x5586d0010790 2024-05-27T07:33:41.778 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:41.783+0000 7fddb928db80 1 -- 172.21.15.52:0/1370930526 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x5586d001cef0 con 0x5586d0016460 2024-05-27T07:33:41.778 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:41.783+0000 7fddb928db80 10 client.0 _ll_register_callbacks cb 0x5586cfec1920 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T07:33:41.778 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:41.783+0000 7fdda37fe700 1 -- 172.21.15.52:0/1370930526 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5586d0016460 msgr2=0x5586d001a5d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:33:41.779 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:41.783+0000 7fdda37fe700 1 --2- 172.21.15.52:0/1370930526 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5586d0016460 0x5586d001a5d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:33:41.779 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:41.783+0000 7fdda37fe700 1 -- 172.21.15.52:0/1370930526 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x5586cfe89e40 msgr2=0x5586d0019b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:33:41.779 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:41.783+0000 7fdda37fe700 1 --2- 172.21.15.52:0/1370930526 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x5586cfe89e40 0x5586d0019b50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:33:41.779 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:41.783+0000 7fdda37fe700 1 -- 172.21.15.52:0/1370930526 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5586cfec5690 con 0x5586d0010790 2024-05-27T07:33:41.779 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:41.783+0000 7fdda8e15700 1 --2- 172.21.15.52:0/1370930526 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5586d0016460 0x5586d001a5d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-27T07:33:41.779 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:41.784+0000 7fddb928db80 1 -- 172.21.15.52:0/1370930526 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fdd98005040 con 0x5586d0010790 2024-05-27T07:33:41.779 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:41.784+0000 7fdda37fe700 1 --2- 172.21.15.52:0/1370930526 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5586d0010790 0x5586d001a090 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7fdd9c007fd0 tx=0x7fdd9c0a51e0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=45e0001cd2cfec02 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:33:41.779 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:41.784+0000 7fdd837fe700 10 client.26549 ms_handle_connect on v2:172.21.15.105:3300/0 2024-05-27T07:33:41.783 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:41.789+0000 7fdd837fe700 1 -- 172.21.15.52:0/1370930526 <== mon.2 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdd9c019040 con 0x5586d0010790 2024-05-27T07:33:41.783 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:41.789+0000 7fdd837fe700 1 -- 172.21.15.52:0/1370930526 <== mon.2 v2:172.21.15.105:3300/0 2 ==== osd_map(118..118 src has 1..118) ==== 9479+0+0 (secure 0 0 0) 0x7fdd9c0ad140 con 0x5586d0010790 2024-05-27T07:33:41.783 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:41.789+0000 7fdd837fe700 1 -- 172.21.15.52:0/1370930526 <== mon.2 v2:172.21.15.105:3300/0 3 ==== mon_get_version_reply(handle=1 version=103) ==== 24+0+0 (secure 0 0 0) 0x7fdd9c0ad340 con 0x5586d0010790 2024-05-27T07:33:41.785 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:41.791+0000 7fddb928db80 10 client.26549 fetch_fsmap learned FSMap version 103 2024-05-27T07:33:41.785 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:41.791+0000 7fddb928db80 1 -- 172.21.15.52:0/1370930526 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({fsmap=103}) -- 0x5586d0027930 con 0x5586d0010790 2024-05-27T07:33:41.789 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:41.795+0000 7fdd837fe700 1 -- 172.21.15.52:0/1370930526 <== mon.2 v2:172.21.15.105:3300/0 4 ==== fsmap(e 103) ==== 4215+0+0 (secure 0 0 0) 0x7fdd9c0a9450 con 0x5586d0010790 2024-05-27T07:33:41.790 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:41.795+0000 7fddb928db80 10 client.26549 fetch_fsmap finished waiting for FSMap version 103 2024-05-27T07:33:41.790 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:41.795+0000 7fddb928db80 10 client.26549 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.i 2024-05-27T07:33:41.790 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:41.795+0000 7fddb928db80 20 client.26549 populate_metadata read hostname 'smithi052' 2024-05-27T07:33:41.790 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:41.795+0000 7fddb928db80 1 --2- 172.21.15.52:0/1370930526 >> [v2:172.21.15.173:6838/564778293,v1:172.21.15.173:6839/564778293] conn(0x5586d002a200 0x5586d002c6f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:33:41.790 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:41.796+0000 7fddb928db80 4 client.26549 mds_command: new command op to 24475 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-27T07:33:41.791 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:41.796+0000 7fddb928db80 1 -- 172.21.15.52:0/1370930526 --> [v2:172.21.15.173:6838/564778293,v1:172.21.15.173:6839/564778293] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5586cffd39f0 con 0x5586d002a200 2024-05-27T07:33:41.791 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:41.796+0000 7fdda3fff700 1 --2- 172.21.15.52:0/1370930526 >> [v2:172.21.15.173:6838/564778293,v1:172.21.15.173:6839/564778293] conn(0x5586d002a200 0x5586d002c6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:33:41.793 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:41.798+0000 7fdda3fff700 1 --2- 172.21.15.52:0/1370930526 >> [v2:172.21.15.173:6838/564778293,v1:172.21.15.173:6839/564778293] conn(0x5586d002a200 0x5586d002c6f0 crc :-1 s=READY pgs=47 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=46a222a7629ac852 server_cookie=d2e5ec309e57c8a in_seq=0 out_seq=0 2024-05-27T07:33:41.793 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:41.798+0000 7fdd837fe700 10 client.26549 ms_handle_connect on v2:172.21.15.173:6838/564778293 2024-05-27T07:33:41.793 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:41.799+0000 7fdd837fe700 1 -- 172.21.15.52:0/1370930526 <== mds.2 v2:172.21.15.173:6838/564778293 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24041 (crc 0 0 0) 0x5586cffd39f0 con 0x5586d002a200 2024-05-27T07:33:41.793 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:41.799+0000 7fdd837fe700 10 client.26549 handle_command_reply: tid=0 2024-05-27T07:33:41.793 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:41.799+0000 7fddb928db80 1 client.26549 shutdown 2024-05-27T07:33:41.795 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:41.801+0000 7fddb928db80 1 -- 172.21.15.52:0/1370930526 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5586d0010790 msgr2=0x5586d001a090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:33:41.795 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:41.801+0000 7fddb928db80 1 --2- 172.21.15.52:0/1370930526 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5586d0010790 0x5586d001a090 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7fdd9c007fd0 tx=0x7fdd9c0a51e0 comp rx=0 tx=0).stop 2024-05-27T07:33:41.796 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:41.801+0000 7fddb928db80 1 -- 172.21.15.52:0/1370930526 shutdown_connections 2024-05-27T07:33:41.796 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:41.801+0000 7fddb928db80 1 --2- 172.21.15.52:0/1370930526 >> [v2:172.21.15.173:6838/564778293,v1:172.21.15.173:6839/564778293] conn(0x5586d002a200 0x5586d002c6f0 crc :-1 s=READY pgs=47 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:33:41.796 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:41.801+0000 7fddb928db80 1 --2- 172.21.15.52:0/1370930526 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5586d0016460 0x5586d001a5d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:33:41.797 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:41.801+0000 7fddb928db80 1 --2- 172.21.15.52:0/1370930526 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x5586cfe89e40 0x5586d0019b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:33:41.797 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:41.801+0000 7fddb928db80 1 --2- 172.21.15.52:0/1370930526 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5586d0010790 0x5586d001a090 unknown :-1 s=CLOSED pgs=218 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:33:41.797 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:41.801+0000 7fddb928db80 1 -- 172.21.15.52:0/1370930526 >> 172.21.15.52:0/1370930526 conn(0x5586cfee4af0 msgr2=0x5586d000f240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:33:41.797 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:41.802+0000 7fdd837fe700 0 client.26549 ms_handle_reset on v2:172.21.15.173:6838/564778293 2024-05-27T07:33:41.797 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:41.802+0000 7fddb928db80 1 -- 172.21.15.52:0/1370930526 shutdown_connections 2024-05-27T07:33:41.797 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:41.802+0000 7fddb928db80 1 -- 172.21.15.52:0/1370930526 wait complete. 2024-05-27T07:33:41.797 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:41.802+0000 7fddb928db80 20 client.26549 trim_cache size 0 max 16384 2024-05-27T07:33:41.827 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:41.833+0000 7fddb928db80 1 Processor -- start 2024-05-27T07:33:41.827 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:41.833+0000 7fddb928db80 1 -- start start 2024-05-27T07:33:41.829 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:41.834+0000 7fddb928db80 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5586cfe89e40 0x5586d0018e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:33:41.829 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:41.834+0000 7fddb928db80 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x5586d0010790 0x5586d001c0c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:33:41.829 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:41.834+0000 7fddb928db80 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5586d0016460 0x5586d001a410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:33:41.829 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:41.834+0000 7fddb928db80 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x7fdda41b7310 con 0x5586d0010790 2024-05-27T07:33:41.829 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:41.834+0000 7fddb928db80 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7fdda41b56b0 con 0x5586cfe89e40 2024-05-27T07:33:41.829 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:41.834+0000 7fddb928db80 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7fdda41b6c10 con 0x5586d0016460 2024-05-27T07:33:41.830 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:41.834+0000 7fdda3fff700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5586cfe89e40 0x5586d0018e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:33:41.830 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:41.834+0000 7fdda3fff700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5586cfe89e40 0x5586d0018e90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.52:52248/0 (socket says 172.21.15.52:52248) 2024-05-27T07:33:41.830 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:41.834+0000 7fdda3fff700 1 -- 172.21.15.52:0/583266103 learned_addr learned my addr 172.21.15.52:0/583266103 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-05-27T07:33:41.830 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:41.834+0000 7fdda8e15700 1 --2- 172.21.15.52:0/583266103 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5586d0016460 0x5586d001a410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:33:41.830 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:41.834+0000 7fdda3fff700 1 -- 172.21.15.52:0/583266103 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5586d0016460 msgr2=0x5586d001a410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:33:41.830 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:41.834+0000 7fdda3fff700 1 --2- 172.21.15.52:0/583266103 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5586d0016460 0x5586d001a410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:33:41.830 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:41.834+0000 7fdda3fff700 1 -- 172.21.15.52:0/583266103 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x5586d0010790 msgr2=0x5586d001c0c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:33:41.830 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:41.834+0000 7fdda3fff700 1 --2- 172.21.15.52:0/583266103 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x5586d0010790 0x5586d001c0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:33:41.830 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:41.834+0000 7fdda3fff700 1 -- 172.21.15.52:0/583266103 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5586d0003eb0 con 0x5586cfe89e40 2024-05-27T07:33:41.830 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:41.835+0000 7fdda3fff700 1 --2- 172.21.15.52:0/583266103 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5586cfe89e40 0x5586d0018e90 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x5586d00040a0 tx=0x7fdd980054a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=db9c3dea6edc570a server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:33:41.830 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:41.835+0000 7fdd89ffb700 1 -- 172.21.15.52:0/583266103 <== mon.2 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdd98021070 con 0x5586cfe89e40 2024-05-27T07:33:41.830 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:41.835+0000 7fdd89ffb700 1 -- 172.21.15.52:0/583266103 <== mon.2 v2:172.21.15.105:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fdd98005610 con 0x5586cfe89e40 2024-05-27T07:33:41.831 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:41.835+0000 7fdd89ffb700 1 -- 172.21.15.52:0/583266103 <== mon.2 v2:172.21.15.105:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdd9800dc30 con 0x5586cfe89e40 2024-05-27T07:33:41.831 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:41.836+0000 7fddb928db80 1 -- 172.21.15.52:0/583266103 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5586cfe89e40 msgr2=0x5586d0018e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:33:41.832 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:41.836+0000 7fddb928db80 1 --2- 172.21.15.52:0/583266103 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5586cfe89e40 0x5586d0018e90 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x5586d00040a0 tx=0x7fdd980054a0 comp rx=0 tx=0).stop 2024-05-27T07:33:41.832 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:41.837+0000 7fddb928db80 1 -- 172.21.15.52:0/583266103 shutdown_connections 2024-05-27T07:33:41.832 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:41.837+0000 7fddb928db80 1 --2- 172.21.15.52:0/583266103 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5586d0016460 0x5586d001a410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:33:41.832 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:41.837+0000 7fddb928db80 1 --2- 172.21.15.52:0/583266103 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5586cfe89e40 0x5586d0018e90 unknown :-1 s=CLOSED pgs=219 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:33:41.832 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:41.837+0000 7fddb928db80 1 --2- 172.21.15.52:0/583266103 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x5586d0010790 0x5586d001c0c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:33:41.832 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:41.837+0000 7fddb928db80 1 -- 172.21.15.52:0/583266103 >> 172.21.15.52:0/583266103 conn(0x5586cfee4af0 msgr2=0x5586d0015270 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:33:41.835 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:41.841+0000 7fddb928db80 1 -- 172.21.15.52:0/583266103 shutdown_connections 2024-05-27T07:33:41.836 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:41.841+0000 7fddb928db80 1 -- 172.21.15.52:0/583266103 wait complete. 2024-05-27T07:33:41.836 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:41.841+0000 7fddb928db80 1 -- start start 2024-05-27T07:33:41.836 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:41.842+0000 7fddb928db80 1 Processor -- start 2024-05-27T07:33:41.837 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:41.842+0000 7fddb928db80 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x5586cfe89e40 0x5586d0022860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:33:41.837 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:41.842+0000 7fddb928db80 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5586d0010790 0x5586d001dbc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:33:41.837 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:41.842+0000 7fddb928db80 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5586d0016460 0x5586d001e100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:33:41.837 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:41.842+0000 7fddb928db80 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x7fdda41b6480 con 0x5586cfe89e40 2024-05-27T07:33:41.837 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:41.842+0000 7fddb928db80 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7fdda41b5d80 con 0x5586d0010790 2024-05-27T07:33:41.837 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:41.842+0000 7fddb928db80 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x5586cffff090 con 0x5586d0016460 2024-05-27T07:33:41.837 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:41.842+0000 7fddb928db80 10 client.0 _ll_register_callbacks cb 0x5586cfec1920 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T07:33:41.841 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:41.846+0000 7fdda37fe700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5586d0010790 0x5586d001dbc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:33:41.841 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:41.846+0000 7fdda37fe700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5586d0010790 0x5586d001dbc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.52:52254/0 (socket says 172.21.15.52:52254) 2024-05-27T07:33:41.841 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:41.846+0000 7fdda37fe700 1 -- 172.21.15.52:0/2804478949 learned_addr learned my addr 172.21.15.52:0/2804478949 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-05-27T07:33:41.841 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:41.846+0000 7fdda3fff700 1 --2- 172.21.15.52:0/2804478949 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x5586cfe89e40 0x5586d0022860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:33:41.841 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:41.847+0000 7fdda8e15700 1 --2- 172.21.15.52:0/2804478949 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5586d0016460 0x5586d001e100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:33:41.844 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:41.849+0000 7fdda37fe700 1 -- 172.21.15.52:0/2804478949 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5586d0016460 msgr2=0x5586d001e100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:33:41.844 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:41.849+0000 7fdda37fe700 1 --2- 172.21.15.52:0/2804478949 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5586d0016460 0x5586d001e100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:33:41.844 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:41.849+0000 7fdda37fe700 1 -- 172.21.15.52:0/2804478949 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x5586cfe89e40 msgr2=0x5586d0022860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:33:41.844 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:41.849+0000 7fdda37fe700 1 --2- 172.21.15.52:0/2804478949 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x5586cfe89e40 0x5586d0022860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:33:41.844 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:41.849+0000 7fdda37fe700 1 -- 172.21.15.52:0/2804478949 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5586d001e7f0 con 0x5586d0010790 2024-05-27T07:33:41.844 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:41.849+0000 7fdda37fe700 1 --2- 172.21.15.52:0/2804478949 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5586d0010790 0x5586d001dbc0 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7fdd9c0a9180 tx=0x7fdd9c0e4770 comp rx=0 tx=0).ready entity=mon.2 client_cookie=bd1dcec3eb6edfd5 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:33:41.844 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:41.849+0000 7fdd837fe700 10 client.0 ms_handle_connect on v2:172.21.15.105:3300/0 2024-05-27T07:33:41.844 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:41.850+0000 7fdd837fe700 1 -- 172.21.15.52:0/2804478949 <== mon.2 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdd9c019070 con 0x5586d0010790 2024-05-27T07:33:41.844 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:41.850+0000 7fdd837fe700 1 -- 172.21.15.52:0/2804478949 <== mon.2 v2:172.21.15.105:3300/0 2 ==== osd_map(118..118 src has 1..118) ==== 9479+0+0 (secure 0 0 0) 0x7fdd9c0a57e0 con 0x5586d0010790 2024-05-27T07:33:41.844 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:41.850+0000 7fddb928db80 1 -- 172.21.15.52:0/2804478949 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fdd98005610 con 0x5586d0010790 2024-05-27T07:33:41.845 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:41.850+0000 7fdd837fe700 1 -- 172.21.15.52:0/2804478949 <== mon.2 v2:172.21.15.105:3300/0 3 ==== mon_get_version_reply(handle=1 version=103) ==== 24+0+0 (secure 0 0 0) 0x7fdd9c0a8d00 con 0x5586d0010790 2024-05-27T07:33:41.845 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:41.850+0000 7fddb928db80 10 client.26555 fetch_fsmap learned FSMap version 103 2024-05-27T07:33:41.845 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:41.850+0000 7fddb928db80 1 -- 172.21.15.52:0/2804478949 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,fsmap=103,monmap=4+}) -- 0x5586d0094c50 con 0x5586d0010790 2024-05-27T07:33:41.847 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:41.852+0000 7fdd837fe700 1 -- 172.21.15.52:0/2804478949 <== mon.2 v2:172.21.15.105:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fdd98005610 con 0x5586d0010790 2024-05-27T07:33:41.847 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:41.852+0000 7fdd837fe700 1 -- 172.21.15.52:0/2804478949 <== mon.2 v2:172.21.15.105:3300/0 5 ==== fsmap(e 103) ==== 4215+0+0 (secure 0 0 0) 0x7fdd9c0c4bc0 con 0x5586d0010790 2024-05-27T07:33:41.847 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:41.852+0000 7fddb928db80 10 client.26555 fetch_fsmap finished waiting for FSMap version 103 2024-05-27T07:33:41.847 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:41.852+0000 7fddb928db80 10 client.26555 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.i 2024-05-27T07:33:41.847 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:41.852+0000 7fddb928db80 20 client.26555 populate_metadata read hostname 'smithi052' 2024-05-27T07:33:41.847 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:41.852+0000 7fddb928db80 1 --2- 172.21.15.52:0/2804478949 >> [v2:172.21.15.173:6838/564778293,v1:172.21.15.173:6839/564778293] conn(0x5586d0095350 0x5586d0097770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:33:41.847 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:41.852+0000 7fddb928db80 4 client.26555 mds_command: new command op to 24475 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-05-27T07:33:41.847 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:41.852+0000 7fddb928db80 1 -- 172.21.15.52:0/2804478949 --> [v2:172.21.15.173:6838/564778293,v1:172.21.15.173:6839/564778293] -- command(tid 0: {"prefix": "damage ls"}) -- 0x5586cff38be0 con 0x5586d0095350 2024-05-27T07:33:41.847 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:41.852+0000 7fdda3fff700 1 --2- 172.21.15.52:0/2804478949 >> [v2:172.21.15.173:6838/564778293,v1:172.21.15.173:6839/564778293] conn(0x5586d0095350 0x5586d0097770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:33:41.849 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:41.853+0000 7fdd837fe700 10 client.26555 ms_handle_connect on v2:172.21.15.173:6838/564778293 2024-05-27T07:33:41.849 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:41.853+0000 7fdda3fff700 1 --2- 172.21.15.52:0/2804478949 >> [v2:172.21.15.173:6838/564778293,v1:172.21.15.173:6839/564778293] conn(0x5586d0095350 0x5586d0097770 crc :-1 s=READY pgs=48 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=237575ae5b2cb5f7 server_cookie=abafdedd1fe4402e in_seq=0 out_seq=0 2024-05-27T07:33:41.849 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:41.853+0000 7fdd837fe700 1 -- 172.21.15.52:0/2804478949 <== mds.2 v2:172.21.15.173:6838/564778293 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x5586cff38be0 con 0x5586d0095350 2024-05-27T07:33:41.849 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:41.854+0000 7fdd837fe700 10 client.26555 handle_command_reply: tid=0 2024-05-27T07:33:41.849 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:41.854+0000 7fddb928db80 1 client.26555 shutdown 2024-05-27T07:33:41.850 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:41.854+0000 7fddb928db80 1 -- 172.21.15.52:0/2804478949 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5586d0010790 msgr2=0x5586d001dbc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:33:41.850 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:41.854+0000 7fddb928db80 1 --2- 172.21.15.52:0/2804478949 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5586d0010790 0x5586d001dbc0 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7fdd9c0a9180 tx=0x7fdd9c0e4770 comp rx=0 tx=0).stop 2024-05-27T07:33:41.850 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:41.854+0000 7fddb928db80 1 -- 172.21.15.52:0/2804478949 shutdown_connections 2024-05-27T07:33:41.850 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:41.854+0000 7fddb928db80 1 --2- 172.21.15.52:0/2804478949 >> [v2:172.21.15.173:6838/564778293,v1:172.21.15.173:6839/564778293] conn(0x5586d0095350 0x5586d0097770 crc :-1 s=READY pgs=48 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:33:41.850 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:41.854+0000 7fddb928db80 1 --2- 172.21.15.52:0/2804478949 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5586d0016460 0x5586d001e100 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:33:41.850 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:41.854+0000 7fddb928db80 1 --2- 172.21.15.52:0/2804478949 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x5586cfe89e40 0x5586d0022860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:33:41.850 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:41.854+0000 7fddb928db80 1 --2- 172.21.15.52:0/2804478949 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5586d0010790 0x5586d001dbc0 unknown :-1 s=CLOSED pgs=220 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:33:41.850 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:41.854+0000 7fddb928db80 1 -- 172.21.15.52:0/2804478949 >> 172.21.15.52:0/2804478949 conn(0x5586cfee4af0 msgr2=0x5586d0015270 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:33:41.850 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:41.854+0000 7fdd837fe700 0 client.26555 ms_handle_reset on v2:172.21.15.173:6838/564778293 2024-05-27T07:33:41.850 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:41.855+0000 7fddb928db80 1 -- 172.21.15.52:0/2804478949 shutdown_connections 2024-05-27T07:33:41.850 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:41.855+0000 7fddb928db80 1 -- 172.21.15.52:0/2804478949 wait complete. 2024-05-27T07:33:41.850 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:41.855+0000 7fddb928db80 20 client.26555 trim_cache size 0 max 16384 2024-05-27T07:33:41.851 INFO:teuthology.orchestra.run.smithi052.stdout:[] 2024-05-27T07:33:41.851 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:41.856+0000 7fddaa878700 1 -- 172.21.15.52:0/3507859883 >> [v2:172.21.15.52:6800/1822318384,v1:172.21.15.52:6801/1822318384] conn(0x7fdd8c0b2af0 msgr2=0x7fdd8c0b5000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:33:41.851 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:41.856+0000 7fddaa878700 1 --2- 172.21.15.52:0/3507859883 >> [v2:172.21.15.52:6800/1822318384,v1:172.21.15.52:6801/1822318384] conn(0x7fdd8c0b2af0 0x7fdd8c0b5000 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x5586d00113c0 tx=0x7fdd94009260 comp rx=0 tx=0).stop 2024-05-27T07:33:41.851 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:41.856+0000 7fddaa878700 1 -- 172.21.15.52:0/3507859883 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fdda410fdb0 msgr2=0x7fdda41a19c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:33:41.852 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:41.856+0000 7fddaa878700 1 --2- 172.21.15.52:0/3507859883 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fdda410fdb0 0x7fdda41a19c0 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7fdd9c007ee0 tx=0x7fdd9c00d3b0 comp rx=0 tx=0).stop 2024-05-27T07:33:41.852 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:41.856+0000 7fddaa878700 1 -- 172.21.15.52:0/3507859883 shutdown_connections 2024-05-27T07:33:41.852 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:41.856+0000 7fddaa878700 1 --2- 172.21.15.52:0/3507859883 >> [v2:172.21.15.52:6800/1822318384,v1:172.21.15.52:6801/1822318384] conn(0x7fdd8c0b2af0 0x7fdd8c0b5000 unknown :-1 s=CLOSED pgs=166 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:33:41.852 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:41.856+0000 7fddaa878700 1 --2- 172.21.15.52:0/3507859883 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fdda4110700 0x7fdda41a5c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:33:41.852 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:41.856+0000 7fddaa878700 1 --2- 172.21.15.52:0/3507859883 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fdda410b720 0x7fdda41a1420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:33:41.852 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:41.856+0000 7fddaa878700 1 --2- 172.21.15.52:0/3507859883 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fdda410fdb0 0x7fdda41a19c0 unknown :-1 s=CLOSED pgs=217 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:33:41.852 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:41.856+0000 7fddaa878700 1 -- 172.21.15.52:0/3507859883 >> 172.21.15.52:0/3507859883 conn(0x7fdda406f4c0 msgr2=0x7fdda410de60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:33:41.852 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:41.856+0000 7fddaa878700 1 -- 172.21.15.52:0/3507859883 shutdown_connections 2024-05-27T07:33:41.852 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:41.856+0000 7fddaa878700 1 -- 172.21.15.52:0/3507859883 wait complete. 2024-05-27T07:33:42.123 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:33:42.128+0000 7fce6d0bd640 1 -- 172.21.15.52:0/2600100806 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fce6009e650 msgr2=0x7fce600a0a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:33:42.123 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:33:42.128+0000 7fce6d0bd640 1 --2- 172.21.15.52:0/2600100806 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fce6009e650 0x7fce600a0a70 secure :-1 s=READY pgs=697 cs=0 l=1 rev1=1 crypto rx=0x7fce54009900 tx=0x7fce5402a4b0 comp rx=0 tx=0).stop 2024-05-27T07:33:42.123 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:33:42.128+0000 7fce6d0bd640 1 -- 172.21.15.52:0/2600100806 shutdown_connections 2024-05-27T07:33:42.124 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:33:42.128+0000 7fce6d0bd640 1 --2- 172.21.15.52:0/2600100806 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fce600a0fb0 0x7fce600a3450 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:33:42.124 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:33:42.128+0000 7fce6d0bd640 1 --2- 172.21.15.52:0/2600100806 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fce6009e650 0x7fce600a0a70 unknown :-1 s=CLOSED pgs=697 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:33:42.124 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:33:42.128+0000 7fce6d0bd640 1 --2- 172.21.15.52:0/2600100806 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fce60097bf0 0x7fce60098000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:33:42.124 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:33:42.128+0000 7fce6d0bd640 1 -- 172.21.15.52:0/2600100806 >> 172.21.15.52:0/2600100806 conn(0x7fce6000aff0 msgr2=0x7fce6000b430 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:33:42.124 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:33:42.129+0000 7fce6d0bd640 1 -- 172.21.15.52:0/2600100806 shutdown_connections 2024-05-27T07:33:42.124 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:33:42.129+0000 7fce6d0bd640 1 -- 172.21.15.52:0/2600100806 wait complete. 2024-05-27T07:33:42.127 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:33:42.130+0000 7fce6d0bd640 1 Processor -- start 2024-05-27T07:33:42.127 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:33:42.130+0000 7fce6d0bd640 1 -- start start 2024-05-27T07:33:42.127 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:33:42.131+0000 7fce6d0bd640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fce60097bf0 0x7fce6009c5c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:33:42.128 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:33:42.131+0000 7fce6d0bd640 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fce6009e650 0x7fce6009cb00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:33:42.128 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:33:42.132+0000 7fce67fff640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fce60097bf0 0x7fce6009c5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:33:42.128 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:33:42.132+0000 7fce67fff640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fce60097bf0 0x7fce6009c5c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.52:52268/0 (socket says 172.21.15.52:52268) 2024-05-27T07:33:42.128 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:33:42.132+0000 7fce6d0bd640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fce600a0fb0 0x7fce6009d040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:33:42.128 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:33:42.132+0000 7fce6d0bd640 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x7fce600a4ad0 con 0x7fce6009e650 2024-05-27T07:33:42.128 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:33:42.132+0000 7fce6d0bd640 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7fce600a4950 con 0x7fce60097bf0 2024-05-27T07:33:42.129 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:33:42.132+0000 7fce6d0bd640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7fce600a4c50 con 0x7fce600a0fb0 2024-05-27T07:33:42.129 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:33:42.132+0000 7fce67fff640 1 -- 172.21.15.52:0/456024616 learned_addr learned my addr 172.21.15.52:0/456024616 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-05-27T07:33:42.129 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:33:42.132+0000 7fce677fe640 1 --2- 172.21.15.52:0/456024616 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fce6009e650 0x7fce6009cb00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:33:42.129 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:33:42.132+0000 7fce6c8bc640 1 --2- 172.21.15.52:0/456024616 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fce600a0fb0 0x7fce6009d040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:33:42.129 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:33:42.132+0000 7fce67fff640 1 -- 172.21.15.52:0/456024616 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fce600a0fb0 msgr2=0x7fce6009d040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:33:42.129 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:33:42.132+0000 7fce67fff640 1 --2- 172.21.15.52:0/456024616 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fce600a0fb0 0x7fce6009d040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:33:42.130 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:33:42.132+0000 7fce67fff640 1 -- 172.21.15.52:0/456024616 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fce6009e650 msgr2=0x7fce6009cb00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:33:42.130 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:33:42.132+0000 7fce67fff640 1 --2- 172.21.15.52:0/456024616 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fce6009e650 0x7fce6009cb00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:33:42.130 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:33:42.133+0000 7fce67fff640 1 -- 172.21.15.52:0/456024616 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fce6013fcd0 con 0x7fce60097bf0 2024-05-27T07:33:42.130 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:33:42.133+0000 7fce677fe640 1 --2- 172.21.15.52:0/456024616 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fce6009e650 0x7fce6009cb00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-27T07:33:42.130 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:33:42.133+0000 7fce67fff640 1 --2- 172.21.15.52:0/456024616 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fce60097bf0 0x7fce6009c5c0 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7fce5c00b450 tx=0x7fce5c00b910 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:33:42.132 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:33:42.133+0000 7fce657fa640 1 -- 172.21.15.52:0/456024616 <== mon.2 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fce5c015020 con 0x7fce60097bf0 2024-05-27T07:33:42.132 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:33:42.134+0000 7fce657fa640 1 -- 172.21.15.52:0/456024616 <== mon.2 v2:172.21.15.105:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fce5c011d40 con 0x7fce60097bf0 2024-05-27T07:33:42.132 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:33:42.134+0000 7fce657fa640 1 -- 172.21.15.52:0/456024616 <== mon.2 v2:172.21.15.105:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fce5c00f340 con 0x7fce60097bf0 2024-05-27T07:33:42.132 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:33:42.134+0000 7fce6d0bd640 1 -- 172.21.15.52:0/456024616 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fce6013ffc0 con 0x7fce60097bf0 2024-05-27T07:33:42.132 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:33:42.134+0000 7fce6d0bd640 1 -- 172.21.15.52:0/456024616 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fce601403d0 con 0x7fce60097bf0 2024-05-27T07:33:42.133 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:33:42.138+0000 7fce751d2740 1 Processor -- start 2024-05-27T07:33:42.134 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:33:42.138+0000 7fce751d2740 1 -- start start 2024-05-27T07:33:42.134 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:33:42.138+0000 7fce751d2740 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x5588fb7fe1b0 0x5588fbb9f590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:33:42.134 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:33:42.138+0000 7fce751d2740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5588fb7d52f0 0x5588fbb9fad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:33:42.134 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:33:42.138+0000 7fce751d2740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5588fb7dd4d0 0x5588fbba0010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:33:42.135 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:33:42.138+0000 7fce751d2740 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x5588fbb11c30 con 0x5588fb7fe1b0 2024-05-27T07:33:42.135 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:33:42.138+0000 7fce751d2740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x5588fbb017d0 con 0x5588fb7d52f0 2024-05-27T07:33:42.135 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:33:42.138+0000 7fce751d2740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x5588fbb05810 con 0x5588fb7dd4d0 2024-05-27T07:33:42.135 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:33:42.139+0000 7fce677fe640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5588fb7dd4d0 0x5588fbba0010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:33:42.135 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:33:42.139+0000 7fce677fe640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5588fb7dd4d0 0x5588fbba0010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.173:3300/0 says I am v2:172.21.15.52:50820/0 (socket says 172.21.15.52:50820) 2024-05-27T07:33:42.136 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:33:42.139+0000 7fce677fe640 1 -- 172.21.15.52:0/3648725576 learned_addr learned my addr 172.21.15.52:0/3648725576 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-05-27T07:33:42.136 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:33:42.139+0000 7fce6c8bc640 1 --2- 172.21.15.52:0/3648725576 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x5588fb7fe1b0 0x5588fbb9f590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:33:42.138 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:33:42.139+0000 7fce6c8bc640 1 -- 172.21.15.52:0/3648725576 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5588fb7dd4d0 msgr2=0x5588fbba0010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:33:42.138 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:33:42.139+0000 7fce6c8bc640 1 --2- 172.21.15.52:0/3648725576 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5588fb7dd4d0 0x5588fbba0010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:33:42.138 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:33:42.139+0000 7fce657fa640 1 -- 172.21.15.52:0/456024616 <== mon.2 v2:172.21.15.105:3300/0 4 ==== mgrmap(e 26) ==== 149783+0+0 (secure 0 0 0) 0x7fce5c018020 con 0x7fce60097bf0 2024-05-27T07:33:42.138 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:33:42.139+0000 7fce6c8bc640 1 -- 172.21.15.52:0/3648725576 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5588fb7d52f0 msgr2=0x5588fbb9fad0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-05-27T07:33:42.139 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:33:42.139+0000 7fce6c8bc640 1 --2- 172.21.15.52:0/3648725576 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5588fb7d52f0 0x5588fbb9fad0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:33:42.139 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:33:42.139+0000 7fce6c8bc640 1 -- 172.21.15.52:0/3648725576 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5588fb816210 con 0x5588fb7fe1b0 2024-05-27T07:33:42.139 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:33:42.141+0000 7fce657fa640 1 --2- 172.21.15.52:0/456024616 >> [v2:172.21.15.52:6800/1822318384,v1:172.21.15.52:6801/1822318384] conn(0x7fce580b2a10 0x7fce580b4f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:33:42.139 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:33:42.141+0000 7fce657fa640 1 -- 172.21.15.52:0/456024616 <== mon.2 v2:172.21.15.105:3300/0 5 ==== osd_map(118..118 src has 1..118) ==== 9479+0+0 (secure 0 0 0) 0x7fce5c0e23e0 con 0x7fce60097bf0 2024-05-27T07:33:42.139 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:33:42.141+0000 7fce67fff640 1 --2- 172.21.15.52:0/456024616 >> [v2:172.21.15.52:6800/1822318384,v1:172.21.15.52:6801/1822318384] conn(0x7fce580b2a10 0x7fce580b4f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:33:42.140 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:33:42.142+0000 7fce6c8bc640 1 --2- 172.21.15.52:0/3648725576 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x5588fb7fe1b0 0x5588fbb9f590 secure :-1 s=READY pgs=698 cs=0 l=1 rev1=1 crypto rx=0x7fce68052160 tx=0x7fce6806bd60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c2679f65b8885bbd server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:33:42.140 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:33:42.142+0000 7fce67fff640 1 --2- 172.21.15.52:0/456024616 >> [v2:172.21.15.52:6800/1822318384,v1:172.21.15.52:6801/1822318384] conn(0x7fce580b2a10 0x7fce580b4f20 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x5588fbb9d080 tx=0x7fce5c006df0 comp rx=0 tx=0).ready entity=mgr.15333 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:33:42.140 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:33:42.143+0000 7fce527fc640 1 -- 172.21.15.52:0/3648725576 <== mon.0 v2:172.21.15.52:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fce6806c440 con 0x5588fb7fe1b0 2024-05-27T07:33:42.140 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:33:42.143+0000 7fce527fc640 1 -- 172.21.15.52:0/3648725576 <== mon.0 v2:172.21.15.52:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fce680818d0 con 0x5588fb7fe1b0 2024-05-27T07:33:42.140 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:33:42.143+0000 7fce527fc640 1 -- 172.21.15.52:0/3648725576 <== mon.0 v2:172.21.15.52:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fce68080670 con 0x5588fb7fe1b0 2024-05-27T07:33:42.144 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:33:42.144+0000 7fce751d2740 1 -- 172.21.15.52:0/3648725576 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x5588fb7fe1b0 msgr2=0x5588fbb9f590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:33:42.144 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:33:42.144+0000 7fce751d2740 1 --2- 172.21.15.52:0/3648725576 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x5588fb7fe1b0 0x5588fbb9f590 secure :-1 s=READY pgs=698 cs=0 l=1 rev1=1 crypto rx=0x7fce68052160 tx=0x7fce6806bd60 comp rx=0 tx=0).stop 2024-05-27T07:33:42.144 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:33:42.145+0000 7fce751d2740 1 -- 172.21.15.52:0/3648725576 shutdown_connections 2024-05-27T07:33:42.144 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:33:42.145+0000 7fce751d2740 1 --2- 172.21.15.52:0/3648725576 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5588fb7dd4d0 0x5588fbba0010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:33:42.145 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:33:42.145+0000 7fce751d2740 1 --2- 172.21.15.52:0/3648725576 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5588fb7d52f0 0x5588fbb9fad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:33:42.145 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:33:42.145+0000 7fce751d2740 1 --2- 172.21.15.52:0/3648725576 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x5588fb7fe1b0 0x5588fbb9f590 unknown :-1 s=CLOSED pgs=698 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:33:42.145 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:33:42.145+0000 7fce751d2740 1 -- 172.21.15.52:0/3648725576 >> 172.21.15.52:0/3648725576 conn(0x5588fb7f2300 msgr2=0x5588fbb9bd40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:33:42.145 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:33:42.146+0000 7fce751d2740 1 -- 172.21.15.52:0/3648725576 shutdown_connections 2024-05-27T07:33:42.145 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:33:42.146+0000 7fce751d2740 1 -- 172.21.15.52:0/3648725576 wait complete. 2024-05-27T07:33:42.145 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:33:42.146+0000 7fce751d2740 1 -- start start 2024-05-27T07:33:42.146 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:33:42.147+0000 7fce751d2740 1 Processor -- start 2024-05-27T07:33:42.146 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:33:42.148+0000 7fce751d2740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5588fb7d52f0 0x5588fbba9c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:33:42.146 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:33:42.148+0000 7fce751d2740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5588fb7dd4d0 0x5588fbbaa1c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:33:42.146 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:33:42.148+0000 7fce751d2740 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x5588fb7fe1b0 0x5588fbbafaa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:33:42.146 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:33:42.148+0000 7fce677fe640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5588fb7dd4d0 0x5588fbbaa1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:33:42.147 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:33:42.148+0000 7fce677fe640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5588fb7dd4d0 0x5588fbbaa1c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.173:3300/0 says I am v2:172.21.15.52:50828/0 (socket says 172.21.15.52:50828) 2024-05-27T07:33:42.147 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:33:42.148+0000 7fce677fe640 1 -- 172.21.15.52:0/832094796 learned_addr learned my addr 172.21.15.52:0/832094796 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-05-27T07:33:42.147 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:33:42.148+0000 7fce751d2740 1 -- 172.21.15.52:0/832094796 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x5588fb7e7790 con 0x5588fb7fe1b0 2024-05-27T07:33:42.147 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:33:42.148+0000 7fce751d2740 1 -- 172.21.15.52:0/832094796 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x5588fbabd890 con 0x5588fb7d52f0 2024-05-27T07:33:42.147 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:33:42.148+0000 7fce6c8bc640 1 --2- 172.21.15.52:0/832094796 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5588fb7d52f0 0x5588fbba9c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:33:42.147 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:33:42.148+0000 7fce751d2740 1 -- 172.21.15.52:0/832094796 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x5588fbbac6c0 con 0x5588fb7dd4d0 2024-05-27T07:33:42.147 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:33:42.149+0000 7fce6c8bc640 1 --2- 172.21.15.52:0/832094796 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x5588fb7fe1b0 0x5588fbbafaa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:33:42.148 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:33:42.149+0000 7fce751d2740 10 client.0 _ll_register_callbacks cb 0x5588fb815a00 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T07:33:42.148 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:33:42.149+0000 7fce677fe640 1 -- 172.21.15.52:0/832094796 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5588fb7d52f0 msgr2=0x5588fbba9c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:33:42.148 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:33:42.149+0000 7fce677fe640 1 --2- 172.21.15.52:0/832094796 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5588fb7d52f0 0x5588fbba9c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:33:42.148 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:33:42.149+0000 7fce677fe640 1 -- 172.21.15.52:0/832094796 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x5588fb7fe1b0 msgr2=0x5588fbbafaa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:33:42.148 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:33:42.149+0000 7fce677fe640 1 --2- 172.21.15.52:0/832094796 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x5588fb7fe1b0 0x5588fbbafaa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:33:42.148 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:33:42.149+0000 7fce677fe640 1 -- 172.21.15.52:0/832094796 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5588fb8e2620 con 0x5588fb7dd4d0 2024-05-27T07:33:42.149 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:33:42.149+0000 7fce677fe640 1 --2- 172.21.15.52:0/832094796 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5588fb7dd4d0 0x5588fbbaa1c0 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7fce54003bc0 tx=0x7fce54003bf0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=a2bca2d711ddcb44 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:33:42.149 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:33:42.149+0000 7fce477fe640 10 client.0 ms_handle_connect on v2:172.21.15.173:3300/0 2024-05-27T07:33:42.150 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:33:42.151+0000 7fce477fe640 1 -- 172.21.15.52:0/832094796 <== mon.1 v2:172.21.15.173:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fce54035070 con 0x5588fb7dd4d0 2024-05-27T07:33:42.151 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:33:42.151+0000 7fce477fe640 1 -- 172.21.15.52:0/832094796 <== mon.1 v2:172.21.15.173:3300/0 2 ==== osd_map(118..118 src has 1..118) ==== 9479+0+0 (secure 0 0 0) 0x7fce54005020 con 0x5588fb7dd4d0 2024-05-27T07:33:42.151 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:33:42.152+0000 7fce751d2740 1 -- 172.21.15.52:0/832094796 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fce680818d0 con 0x5588fb7dd4d0 2024-05-27T07:33:42.151 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:33:42.153+0000 7fce477fe640 1 -- 172.21.15.52:0/832094796 <== mon.1 v2:172.21.15.173:3300/0 3 ==== mon_get_version_reply(handle=1 version=103) ==== 24+0+0 (secure 0 0 0) 0x7fce54005220 con 0x5588fb7dd4d0 2024-05-27T07:33:42.151 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:33:42.153+0000 7fce751d2740 10 client.26392 fetch_fsmap learned FSMap version 103 2024-05-27T07:33:42.151 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:33:42.153+0000 7fce751d2740 1 -- 172.21.15.52:0/832094796 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({config=0+,fsmap=103,monmap=4+}) -- 0x5588fbbb89d0 con 0x5588fb7dd4d0 2024-05-27T07:33:42.151 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:33:42.155+0000 7fce477fe640 1 -- 172.21.15.52:0/832094796 <== mon.1 v2:172.21.15.173:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fce680818d0 con 0x5588fb7dd4d0 2024-05-27T07:33:42.151 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:33:42.155+0000 7fce477fe640 1 -- 172.21.15.52:0/832094796 <== mon.1 v2:172.21.15.173:3300/0 5 ==== fsmap(e 103) ==== 4215+0+0 (secure 0 0 0) 0x7fce54043030 con 0x5588fb7dd4d0 2024-05-27T07:33:42.151 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:33:42.156+0000 7fce751d2740 10 client.26392 fetch_fsmap finished waiting for FSMap version 103 2024-05-27T07:33:42.151 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:33:42.156+0000 7fce751d2740 10 client.26392 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-05-27T07:33:42.151 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:33:42.156+0000 7fce751d2740 20 client.26392 populate_metadata read hostname 'smithi052' 2024-05-27T07:33:42.151 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:33:42.156+0000 7fce751d2740 1 --2- 172.21.15.52:0/832094796 >> [v2:172.21.15.105:6838/500348151,v1:172.21.15.105:6839/500348151] conn(0x5588fb7bcd80 0x5588fbbbc400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:33:42.151 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:33:42.156+0000 7fce751d2740 4 client.26392 mds_command: new command op to 24527 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-27T07:33:42.151 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:33:42.156+0000 7fce751d2740 1 -- 172.21.15.52:0/832094796 --> [v2:172.21.15.105:6838/500348151,v1:172.21.15.105:6839/500348151] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5588fbb13b80 con 0x5588fb7bcd80 2024-05-27T07:33:42.152 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:33:42.158+0000 7fce6c8bc640 1 --2- 172.21.15.52:0/832094796 >> [v2:172.21.15.105:6838/500348151,v1:172.21.15.105:6839/500348151] conn(0x5588fb7bcd80 0x5588fbbbc400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:33:42.154 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:33:42.160+0000 7fce6c8bc640 1 --2- 172.21.15.52:0/832094796 >> [v2:172.21.15.105:6838/500348151,v1:172.21.15.105:6839/500348151] conn(0x5588fb7bcd80 0x5588fbbbc400 crc :-1 s=READY pgs=141 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=c489bdf64c092532 server_cookie=67f07494172b21a4 in_seq=0 out_seq=0 2024-05-27T07:33:42.154 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:33:42.160+0000 7fce477fe640 10 client.26392 ms_handle_connect on v2:172.21.15.105:6838/500348151 2024-05-27T07:33:42.156 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:33:42.162+0000 7fce477fe640 1 -- 172.21.15.52:0/832094796 <== mds.0 v2:172.21.15.105:6838/500348151 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24041 (crc 0 0 0) 0x5588fbb13b80 con 0x5588fb7bcd80 2024-05-27T07:33:42.156 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:33:42.162+0000 7fce477fe640 10 client.26392 handle_command_reply: tid=0 2024-05-27T07:33:42.157 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:33:42.163+0000 7fce751d2740 1 client.26392 shutdown 2024-05-27T07:33:42.158 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:33:42.163+0000 7fce751d2740 1 -- 172.21.15.52:0/832094796 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5588fb7dd4d0 msgr2=0x5588fbbaa1c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:33:42.158 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:33:42.163+0000 7fce751d2740 1 --2- 172.21.15.52:0/832094796 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5588fb7dd4d0 0x5588fbbaa1c0 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7fce54003bc0 tx=0x7fce54003bf0 comp rx=0 tx=0).stop 2024-05-27T07:33:42.158 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:33:42.163+0000 7fce751d2740 1 -- 172.21.15.52:0/832094796 shutdown_connections 2024-05-27T07:33:42.158 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:33:42.163+0000 7fce751d2740 1 --2- 172.21.15.52:0/832094796 >> [v2:172.21.15.105:6838/500348151,v1:172.21.15.105:6839/500348151] conn(0x5588fb7bcd80 0x5588fbbbc400 crc :-1 s=READY pgs=141 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:33:42.158 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:33:42.163+0000 7fce751d2740 1 --2- 172.21.15.52:0/832094796 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x5588fb7fe1b0 0x5588fbbafaa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:33:42.158 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:33:42.163+0000 7fce751d2740 1 --2- 172.21.15.52:0/832094796 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5588fb7dd4d0 0x5588fbbaa1c0 unknown :-1 s=CLOSED pgs=175 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:33:42.158 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:33:42.163+0000 7fce751d2740 1 --2- 172.21.15.52:0/832094796 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5588fb7d52f0 0x5588fbba9c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:33:42.158 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:33:42.163+0000 7fce751d2740 1 -- 172.21.15.52:0/832094796 >> 172.21.15.52:0/832094796 conn(0x5588fb7f2300 msgr2=0x5588fbb9f3c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:33:42.158 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:33:42.163+0000 7fce477fe640 0 client.26392 ms_handle_reset on v2:172.21.15.105:6838/500348151 2024-05-27T07:33:42.159 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:33:42.164+0000 7fce751d2740 1 -- 172.21.15.52:0/832094796 shutdown_connections 2024-05-27T07:33:42.159 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:33:42.164+0000 7fce751d2740 1 -- 172.21.15.52:0/832094796 wait complete. 2024-05-27T07:33:42.159 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:33:42.164+0000 7fce751d2740 20 client.26392 trim_cache size 0 max 16384 2024-05-27T07:33:42.184 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:33:42.190+0000 7fce751d2740 1 Processor -- start 2024-05-27T07:33:42.185 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:33:42.192+0000 7fce751d2740 1 -- start start 2024-05-27T07:33:42.186 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:33:42.192+0000 7fce751d2740 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x5588fb7d52f0 0x5588fbba31e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:33:42.187 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:33:42.192+0000 7fce751d2740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5588fb7dd4d0 0x5588fbba3720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:33:42.187 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:33:42.193+0000 7fce751d2740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5588fb7fe1b0 0x5588fbbc1c00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:33:42.187 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:33:42.193+0000 7fce751d2740 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x7fce6014ac60 con 0x5588fb7d52f0 2024-05-27T07:33:42.187 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:33:42.193+0000 7fce751d2740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7fce60149000 con 0x5588fb7fe1b0 2024-05-27T07:33:42.187 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:33:42.193+0000 7fce751d2740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7fce6014a560 con 0x5588fb7dd4d0 2024-05-27T07:33:42.187 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:33:42.193+0000 7fce6c8bc640 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x5588fb7d52f0 0x5588fbba31e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:33:42.187 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:33:42.193+0000 7fce6c8bc640 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x5588fb7d52f0 0x5588fbba31e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.52:56746/0 (socket says 172.21.15.52:56746) 2024-05-27T07:33:42.187 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:33:42.193+0000 7fce6c8bc640 1 -- 172.21.15.52:0/1348496359 learned_addr learned my addr 172.21.15.52:0/1348496359 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-05-27T07:33:42.188 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:33:42.193+0000 7fce6c8bc640 1 --2- 172.21.15.52:0/1348496359 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5588fb7fe1b0 0x5588fbbc1c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:33:42.188 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:33:42.193+0000 7fce677fe640 1 --2- 172.21.15.52:0/1348496359 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5588fb7dd4d0 0x5588fbba3720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:33:42.189 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:33:42.194+0000 7fce6c8bc640 1 -- 172.21.15.52:0/1348496359 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5588fb7dd4d0 msgr2=0x5588fbba3720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:33:42.189 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:33:42.194+0000 7fce6c8bc640 1 --2- 172.21.15.52:0/1348496359 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5588fb7dd4d0 0x5588fbba3720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:33:42.189 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:33:42.194+0000 7fce6c8bc640 1 -- 172.21.15.52:0/1348496359 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5588fb7fe1b0 msgr2=0x5588fbbc1c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:33:42.189 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:33:42.194+0000 7fce6c8bc640 1 --2- 172.21.15.52:0/1348496359 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5588fb7fe1b0 0x5588fbbc1c00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:33:42.189 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:33:42.194+0000 7fce6c8bc640 1 -- 172.21.15.52:0/1348496359 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5588fbb960a0 con 0x5588fb7d52f0 2024-05-27T07:33:42.189 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:33:42.194+0000 7fce6c8bc640 1 --2- 172.21.15.52:0/1348496359 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x5588fb7d52f0 0x5588fbba31e0 secure :-1 s=READY pgs=699 cs=0 l=1 rev1=1 crypto rx=0x7fce68052210 tx=0x7fce68068f80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a14b8b7048335103 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:33:42.189 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:33:42.194+0000 7fce51ffb640 1 -- 172.21.15.52:0/1348496359 <== mon.0 v2:172.21.15.52:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fce6806a4c0 con 0x5588fb7d52f0 2024-05-27T07:33:42.189 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:33:42.194+0000 7fce51ffb640 1 -- 172.21.15.52:0/1348496359 <== mon.0 v2:172.21.15.52:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fce6806a660 con 0x5588fb7d52f0 2024-05-27T07:33:42.189 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:33:42.194+0000 7fce51ffb640 1 -- 172.21.15.52:0/1348496359 <== mon.0 v2:172.21.15.52:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fce68081950 con 0x5588fb7d52f0 2024-05-27T07:33:42.190 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:33:42.196+0000 7fce751d2740 1 -- 172.21.15.52:0/1348496359 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x5588fb7d52f0 msgr2=0x5588fbba31e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:33:42.190 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:33:42.196+0000 7fce751d2740 1 --2- 172.21.15.52:0/1348496359 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x5588fb7d52f0 0x5588fbba31e0 secure :-1 s=READY pgs=699 cs=0 l=1 rev1=1 crypto rx=0x7fce68052210 tx=0x7fce68068f80 comp rx=0 tx=0).stop 2024-05-27T07:33:42.190 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:33:42.196+0000 7fce751d2740 1 -- 172.21.15.52:0/1348496359 shutdown_connections 2024-05-27T07:33:42.191 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:33:42.196+0000 7fce751d2740 1 --2- 172.21.15.52:0/1348496359 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5588fb7fe1b0 0x5588fbbc1c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:33:42.191 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:33:42.196+0000 7fce751d2740 1 --2- 172.21.15.52:0/1348496359 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5588fb7dd4d0 0x5588fbba3720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:33:42.191 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:33:42.196+0000 7fce751d2740 1 --2- 172.21.15.52:0/1348496359 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x5588fb7d52f0 0x5588fbba31e0 unknown :-1 s=CLOSED pgs=699 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:33:42.191 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:33:42.196+0000 7fce751d2740 1 -- 172.21.15.52:0/1348496359 >> 172.21.15.52:0/1348496359 conn(0x5588fb7f2300 msgr2=0x5588fbba8340 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:33:42.191 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:33:42.196+0000 7fce751d2740 1 -- 172.21.15.52:0/1348496359 shutdown_connections 2024-05-27T07:33:42.191 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:33:42.196+0000 7fce751d2740 1 -- 172.21.15.52:0/1348496359 wait complete. 2024-05-27T07:33:42.191 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:33:42.197+0000 7fce751d2740 1 -- start start 2024-05-27T07:33:42.192 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:33:42.197+0000 7fce751d2740 1 Processor -- start 2024-05-27T07:33:42.192 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:33:42.198+0000 7fce751d2740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5588fb7d52f0 0x5588fbbb3320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:33:42.192 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:33:42.198+0000 7fce751d2740 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x5588fb7dd4d0 0x5588fbbb3860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:33:42.192 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:33:42.198+0000 7fce751d2740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5588fb7fe1b0 0x5588fbc1ba90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:33:42.193 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:33:42.198+0000 7fce751d2740 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x7fce60149dd0 con 0x5588fb7dd4d0 2024-05-27T07:33:42.193 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:33:42.198+0000 7fce751d2740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7fce601496d0 con 0x5588fb7fe1b0 2024-05-27T07:33:42.193 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:33:42.198+0000 7fce751d2740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x5588fbb9c930 con 0x5588fb7d52f0 2024-05-27T07:33:42.193 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:33:42.198+0000 7fce751d2740 10 client.0 _ll_register_callbacks cb 0x7fce54009640 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T07:33:42.193 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:33:42.198+0000 7fce6c8bc640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5588fb7d52f0 0x5588fbbb3320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:33:42.193 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:33:42.198+0000 7fce6c8bc640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5588fb7d52f0 0x5588fbbb3320 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.173:3300/0 says I am v2:172.21.15.52:50842/0 (socket says 172.21.15.52:50842) 2024-05-27T07:33:42.193 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:33:42.198+0000 7fce6c8bc640 1 -- 172.21.15.52:0/1093854406 learned_addr learned my addr 172.21.15.52:0/1093854406 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-05-27T07:33:42.193 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:33:42.198+0000 7fce6c8bc640 1 --2- 172.21.15.52:0/1093854406 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5588fb7fe1b0 0x5588fbc1ba90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:33:42.193 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:33:42.198+0000 7fce677fe640 1 --2- 172.21.15.52:0/1093854406 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x5588fb7dd4d0 0x5588fbbb3860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:33:42.194 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:33:42.199+0000 7fce6c8bc640 1 -- 172.21.15.52:0/1093854406 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5588fb7fe1b0 msgr2=0x5588fbc1ba90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:33:42.195 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:33:42.199+0000 7fce6c8bc640 1 --2- 172.21.15.52:0/1093854406 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5588fb7fe1b0 0x5588fbc1ba90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:33:42.195 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:33:42.199+0000 7fce6c8bc640 1 -- 172.21.15.52:0/1093854406 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x5588fb7dd4d0 msgr2=0x5588fbbb3860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:33:42.195 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:33:42.199+0000 7fce6c8bc640 1 --2- 172.21.15.52:0/1093854406 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x5588fb7dd4d0 0x5588fbbb3860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:33:42.195 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:33:42.199+0000 7fce6c8bc640 1 -- 172.21.15.52:0/1093854406 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5588fbc1c050 con 0x5588fb7d52f0 2024-05-27T07:33:42.195 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:33:42.199+0000 7fce677fe640 1 --2- 172.21.15.52:0/1093854406 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x5588fb7dd4d0 0x5588fbbb3860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-27T07:33:42.195 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:33:42.199+0000 7fce6c8bc640 1 --2- 172.21.15.52:0/1093854406 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5588fb7d52f0 0x5588fbbb3320 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7fce68065d80 tx=0x7fce68081420 comp rx=0 tx=0).ready entity=mon.1 client_cookie=5a0cfe04adbf5f6a server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:33:42.195 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:33:42.199+0000 7fce477fe640 10 client.0 ms_handle_connect on v2:172.21.15.173:3300/0 2024-05-27T07:33:42.195 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:33:42.200+0000 7fce477fe640 1 -- 172.21.15.52:0/1093854406 <== mon.1 v2:172.21.15.173:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fce680815f0 con 0x5588fb7d52f0 2024-05-27T07:33:42.195 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:33:42.200+0000 7fce477fe640 1 -- 172.21.15.52:0/1093854406 <== mon.1 v2:172.21.15.173:3300/0 2 ==== osd_map(118..118 src has 1..118) ==== 9479+0+0 (secure 0 0 0) 0x7fce6809e070 con 0x5588fb7d52f0 2024-05-27T07:33:42.195 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:33:42.200+0000 7fce751d2740 1 -- 172.21.15.52:0/1093854406 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fce6806a660 con 0x5588fb7d52f0 2024-05-27T07:33:42.195 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:33:42.201+0000 7fce477fe640 1 -- 172.21.15.52:0/1093854406 <== mon.1 v2:172.21.15.173:3300/0 3 ==== mon_get_version_reply(handle=1 version=103) ==== 24+0+0 (secure 0 0 0) 0x7fce68081b70 con 0x5588fb7d52f0 2024-05-27T07:33:42.195 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:33:42.201+0000 7fce751d2740 10 client.26398 fetch_fsmap learned FSMap version 103 2024-05-27T07:33:42.196 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:33:42.201+0000 7fce751d2740 1 -- 172.21.15.52:0/1093854406 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({config=0+,fsmap=103,monmap=4+}) -- 0x5588fbc24550 con 0x5588fb7d52f0 2024-05-27T07:33:42.196 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:33:42.201+0000 7fce477fe640 1 -- 172.21.15.52:0/1093854406 <== mon.1 v2:172.21.15.173:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fce68081d10 con 0x5588fb7d52f0 2024-05-27T07:33:42.197 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:33:42.203+0000 7fce477fe640 1 -- 172.21.15.52:0/1093854406 <== mon.1 v2:172.21.15.173:3300/0 5 ==== fsmap(e 103) ==== 4215+0+0 (secure 0 0 0) 0x7fce680a4040 con 0x5588fb7d52f0 2024-05-27T07:33:42.197 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:33:42.203+0000 7fce751d2740 10 client.26398 fetch_fsmap finished waiting for FSMap version 103 2024-05-27T07:33:42.197 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:33:42.203+0000 7fce751d2740 10 client.26398 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-05-27T07:33:42.197 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:33:42.203+0000 7fce751d2740 20 client.26398 populate_metadata read hostname 'smithi052' 2024-05-27T07:33:42.197 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:33:42.203+0000 7fce751d2740 1 --2- 172.21.15.52:0/1093854406 >> [v2:172.21.15.105:6838/500348151,v1:172.21.15.105:6839/500348151] conn(0x5588fb7921c0 0x5588fbc26910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:33:42.197 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:33:42.203+0000 7fce751d2740 4 client.26398 mds_command: new command op to 24527 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-05-27T07:33:42.197 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:33:42.203+0000 7fce751d2740 1 -- 172.21.15.52:0/1093854406 --> [v2:172.21.15.105:6838/500348151,v1:172.21.15.105:6839/500348151] -- command(tid 0: {"prefix": "flush journal"}) -- 0x7fce6806a660 con 0x5588fb7921c0 2024-05-27T07:33:42.198 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:33:42.203+0000 7fce677fe640 1 --2- 172.21.15.52:0/1093854406 >> [v2:172.21.15.105:6838/500348151,v1:172.21.15.105:6839/500348151] conn(0x5588fb7921c0 0x5588fbc26910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:33:42.199 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:33:42.204+0000 7fce677fe640 1 --2- 172.21.15.52:0/1093854406 >> [v2:172.21.15.105:6838/500348151,v1:172.21.15.105:6839/500348151] conn(0x5588fb7921c0 0x5588fbc26910 crc :-1 s=READY pgs=142 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=eed1b3afd794c7fb server_cookie=2da166ec48533652 in_seq=0 out_seq=0 2024-05-27T07:33:42.199 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:33:42.205+0000 7fce477fe640 10 client.26398 ms_handle_connect on v2:172.21.15.105:6838/500348151 2024-05-27T07:33:42.483 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:33:42 smithi052 ceph-mon[31955]: pgmap v262: 129 pgs: 2 active+clean+snaptrim, 121 active+clean, 6 active+clean+snaptrim_wait; 665 MiB data, 4.4 GiB used, 1.0 TiB / 1.0 TiB avail; 22 KiB/s rd, 1.6 MiB/s wr, 560 op/s 2024-05-27T07:33:42.606 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:33:42 smithi173 ceph-mon[37000]: pgmap v262: 129 pgs: 2 active+clean+snaptrim, 121 active+clean, 6 active+clean+snaptrim_wait; 665 MiB data, 4.4 GiB used, 1.0 TiB / 1.0 TiB avail; 22 KiB/s rd, 1.6 MiB/s wr, 560 op/s 2024-05-27T07:33:42.655 DEBUG:teuthology.orchestra.run.smithi052:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:38f9d900c83c63799fbdbe61acc9a11b0d3554a6 shell --fsid 73df2636-1bf8-11ef-bc9b-c7b262605968 -- ceph tell mds.1:1 damage ls 2024-05-27T07:33:42.694 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:33:42 smithi105 ceph-mon[35977]: pgmap v262: 129 pgs: 2 active+clean+snaptrim, 121 active+clean, 6 active+clean+snaptrim_wait; 665 MiB data, 4.4 GiB used, 1.0 TiB / 1.0 TiB avail; 22 KiB/s rd, 1.6 MiB/s wr, 560 op/s 2024-05-27T07:33:43.087 INFO:teuthology.orchestra.run.smithi052.stderr:Inferring config /var/lib/ceph/73df2636-1bf8-11ef-bc9b-c7b262605968/mon.a/config 2024-05-27T07:33:43.440 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:33:43.445+0000 7fce477fe640 1 -- 172.21.15.52:0/1093854406 <== mds.0 v2:172.21.15.105:6838/500348151 1 ==== command_reply(tid 0: 0 ) ==== 8+0+25 (crc 0 0 0) 0x7fce6806a660 con 0x5588fb7921c0 2024-05-27T07:33:43.440 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:33:43.445+0000 7fce477fe640 10 client.26398 handle_command_reply: tid=0 2024-05-27T07:33:43.440 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:33:43.445+0000 7fce751d2740 1 client.26398 shutdown 2024-05-27T07:33:43.440 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:33:43.446+0000 7fce751d2740 1 -- 172.21.15.52:0/1093854406 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5588fb7d52f0 msgr2=0x5588fbbb3320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:33:43.441 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:33:43.446+0000 7fce751d2740 1 --2- 172.21.15.52:0/1093854406 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5588fb7d52f0 0x5588fbbb3320 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7fce68065d80 tx=0x7fce68081420 comp rx=0 tx=0).stop 2024-05-27T07:33:43.441 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:33:43.446+0000 7fce751d2740 1 -- 172.21.15.52:0/1093854406 shutdown_connections 2024-05-27T07:33:43.441 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:33:43.446+0000 7fce751d2740 1 --2- 172.21.15.52:0/1093854406 >> [v2:172.21.15.105:6838/500348151,v1:172.21.15.105:6839/500348151] conn(0x5588fb7921c0 0x5588fbc26910 crc :-1 s=READY pgs=142 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:33:43.441 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:33:43.446+0000 7fce751d2740 1 --2- 172.21.15.52:0/1093854406 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5588fb7fe1b0 0x5588fbc1ba90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:33:43.441 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:33:43.446+0000 7fce751d2740 1 --2- 172.21.15.52:0/1093854406 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x5588fb7dd4d0 0x5588fbbb3860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:33:43.441 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:33:43.446+0000 7fce751d2740 1 --2- 172.21.15.52:0/1093854406 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5588fb7d52f0 0x5588fbbb3320 unknown :-1 s=CLOSED pgs=176 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:33:43.441 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:33:43.446+0000 7fce751d2740 1 -- 172.21.15.52:0/1093854406 >> 172.21.15.52:0/1093854406 conn(0x5588fb7f2300 msgr2=0x5588fbba31e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:33:43.441 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:33:43.446+0000 7fce477fe640 0 client.26398 ms_handle_reset on v2:172.21.15.105:6838/500348151 2024-05-27T07:33:43.441 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:33:43.446+0000 7fce751d2740 1 -- 172.21.15.52:0/1093854406 shutdown_connections 2024-05-27T07:33:43.441 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:33:43.446+0000 7fce751d2740 1 -- 172.21.15.52:0/1093854406 wait complete. 2024-05-27T07:33:43.441 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:33:43.446+0000 7fce751d2740 20 client.26398 trim_cache size 0 max 16384 2024-05-27T07:33:43.442 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stdout:{ 2024-05-27T07:33:43.442 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stdout: "return_code": 0 2024-05-27T07:33:43.442 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stdout:} 2024-05-27T07:33:43.443 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:33:43.447+0000 7fce6d0bd640 1 -- 172.21.15.52:0/456024616 >> [v2:172.21.15.52:6800/1822318384,v1:172.21.15.52:6801/1822318384] conn(0x7fce580b2a10 msgr2=0x7fce580b4f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:33:43.443 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:33:43.447+0000 7fce6d0bd640 1 --2- 172.21.15.52:0/456024616 >> [v2:172.21.15.52:6800/1822318384,v1:172.21.15.52:6801/1822318384] conn(0x7fce580b2a10 0x7fce580b4f20 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x5588fbb9d080 tx=0x7fce5c006df0 comp rx=0 tx=0).stop 2024-05-27T07:33:43.443 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:33:43.447+0000 7fce6d0bd640 1 -- 172.21.15.52:0/456024616 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fce60097bf0 msgr2=0x7fce6009c5c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:33:43.443 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:33:43.447+0000 7fce6d0bd640 1 --2- 172.21.15.52:0/456024616 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fce60097bf0 0x7fce6009c5c0 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7fce5c00b450 tx=0x7fce5c00b910 comp rx=0 tx=0).stop 2024-05-27T07:33:43.443 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:33:43.447+0000 7fce6d0bd640 1 -- 172.21.15.52:0/456024616 shutdown_connections 2024-05-27T07:33:43.443 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:33:43.447+0000 7fce6d0bd640 1 --2- 172.21.15.52:0/456024616 >> [v2:172.21.15.52:6800/1822318384,v1:172.21.15.52:6801/1822318384] conn(0x7fce580b2a10 0x7fce580b4f20 unknown :-1 s=CLOSED pgs=167 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:33:43.444 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:33:43.447+0000 7fce6d0bd640 1 --2- 172.21.15.52:0/456024616 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fce600a0fb0 0x7fce6009d040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:33:43.444 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:33:43.447+0000 7fce6d0bd640 1 --2- 172.21.15.52:0/456024616 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fce6009e650 0x7fce6009cb00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:33:43.444 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:33:43.447+0000 7fce6d0bd640 1 --2- 172.21.15.52:0/456024616 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fce60097bf0 0x7fce6009c5c0 unknown :-1 s=CLOSED pgs=221 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:33:43.444 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:33:43.447+0000 7fce6d0bd640 1 -- 172.21.15.52:0/456024616 >> 172.21.15.52:0/456024616 conn(0x7fce6000aff0 msgr2=0x7fce60094070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:33:43.444 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:33:43.447+0000 7fce6d0bd640 1 -- 172.21.15.52:0/456024616 shutdown_connections 2024-05-27T07:33:43.444 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:33:43.447+0000 7fce6d0bd640 1 -- 172.21.15.52:0/456024616 wait complete. 2024-05-27T07:33:44.483 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:33:44 smithi052 ceph-mon[31955]: pgmap v263: 129 pgs: 129 active+clean; 665 MiB data, 4.4 GiB used, 1.0 TiB / 1.0 TiB avail; 22 KiB/s rd, 1.1 MiB/s wr, 620 op/s 2024-05-27T07:33:44.606 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:33:44 smithi173 ceph-mon[37000]: pgmap v263: 129 pgs: 129 active+clean; 665 MiB data, 4.4 GiB used, 1.0 TiB / 1.0 TiB avail; 22 KiB/s rd, 1.1 MiB/s wr, 620 op/s 2024-05-27T07:33:44.694 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:33:44 smithi105 ceph-mon[35977]: pgmap v263: 129 pgs: 129 active+clean; 665 MiB data, 4.4 GiB used, 1.0 TiB / 1.0 TiB avail; 22 KiB/s rd, 1.1 MiB/s wr, 620 op/s 2024-05-27T07:33:46.233 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:33:46 smithi052 ceph-mon[31955]: osdmap e119: 12 total, 12 up, 12 in 2024-05-27T07:33:46.234 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:33:46 smithi052 ceph-mon[31955]: pgmap v265: 129 pgs: 129 active+clean; 662 MiB data, 4.4 GiB used, 1.0 TiB / 1.0 TiB avail; 25 KiB/s rd, 1.3 MiB/s wr, 680 op/s 2024-05-27T07:33:46.356 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:33:46 smithi173 ceph-mon[37000]: osdmap e119: 12 total, 12 up, 12 in 2024-05-27T07:33:46.356 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:33:46 smithi173 ceph-mon[37000]: pgmap v265: 129 pgs: 129 active+clean; 662 MiB data, 4.4 GiB used, 1.0 TiB / 1.0 TiB avail; 25 KiB/s rd, 1.3 MiB/s wr, 680 op/s 2024-05-27T07:33:46.444 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:33:46 smithi105 ceph-mon[35977]: osdmap e119: 12 total, 12 up, 12 in 2024-05-27T07:33:46.444 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:33:46 smithi105 ceph-mon[35977]: pgmap v265: 129 pgs: 129 active+clean; 662 MiB data, 4.4 GiB used, 1.0 TiB / 1.0 TiB avail; 25 KiB/s rd, 1.3 MiB/s wr, 680 op/s 2024-05-27T07:33:46.907 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:46.912+0000 7f68c9173700 1 -- 172.21.15.52:0/512562077 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f68c411d920 msgr2=0x7f68c411fd40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:33:46.908 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:46.912+0000 7f68c9173700 1 --2- 172.21.15.52:0/512562077 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f68c411d920 0x7f68c411fd40 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7f68bc00b0a0 tx=0x7f68bc00b3b0 comp rx=0 tx=0).stop 2024-05-27T07:33:46.908 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:46.912+0000 7f68c9173700 1 -- 172.21.15.52:0/512562077 shutdown_connections 2024-05-27T07:33:46.908 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:46.912+0000 7f68c9173700 1 --2- 172.21.15.52:0/512562077 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f68c411d920 0x7f68c411fd40 unknown :-1 s=CLOSED pgs=222 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:33:46.908 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:46.912+0000 7f68c9173700 1 --2- 172.21.15.52:0/512562077 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f68c410a7b0 0x7f68c410abc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:33:46.908 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:46.912+0000 7f68c9173700 1 --2- 172.21.15.52:0/512562077 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f68c410b100 0x7f68c411d3e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:33:46.908 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:46.912+0000 7f68c9173700 1 -- 172.21.15.52:0/512562077 >> 172.21.15.52:0/512562077 conn(0x7f68c406f2b0 msgr2=0x7f68c406f6f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:33:46.909 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:46.914+0000 7f68c9173700 1 -- 172.21.15.52:0/512562077 shutdown_connections 2024-05-27T07:33:46.909 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:46.914+0000 7f68c9173700 1 -- 172.21.15.52:0/512562077 wait complete. 2024-05-27T07:33:46.910 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:46.916+0000 7f68c9173700 1 Processor -- start 2024-05-27T07:33:46.911 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:46.917+0000 7f68c9173700 1 -- start start 2024-05-27T07:33:47.010 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:46.918+0000 7f68c9173700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f68c410a7b0 0x7f68c41af650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:33:47.010 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:46.918+0000 7f68c9173700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f68c410b100 0x7f68c41afb90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:33:47.010 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:46.918+0000 7f68c9173700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f68c411d920 0x7f68c41a9870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:33:47.010 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:46.918+0000 7f68c9173700 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x7f68c41230f0 con 0x7f68c411d920 2024-05-27T07:33:47.011 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:46.918+0000 7f68c9173700 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f68c4122f70 con 0x7f68c410a7b0 2024-05-27T07:33:47.011 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:46.918+0000 7f68c9173700 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f68c4123270 con 0x7f68c410b100 2024-05-27T07:33:47.011 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:46.919+0000 7f68c359e700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f68c411d920 0x7f68c41a9870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:33:47.011 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:46.919+0000 7f68c359e700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f68c411d920 0x7f68c41a9870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.52:56786/0 (socket says 172.21.15.52:56786) 2024-05-27T07:33:47.011 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:46.919+0000 7f68c359e700 1 -- 172.21.15.52:0/658831325 learned_addr learned my addr 172.21.15.52:0/658831325 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-05-27T07:33:47.011 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:46.920+0000 7f68c359e700 1 -- 172.21.15.52:0/658831325 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f68c410b100 msgr2=0x7f68c41afb90 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-05-27T07:33:47.011 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:46.920+0000 7f68c359e700 1 --2- 172.21.15.52:0/658831325 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f68c410b100 0x7f68c41afb90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:33:47.011 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:46.920+0000 7f68c359e700 1 -- 172.21.15.52:0/658831325 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f68c410a7b0 msgr2=0x7f68c41af650 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-05-27T07:33:47.011 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:46.920+0000 7f68c359e700 1 --2- 172.21.15.52:0/658831325 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f68c410a7b0 0x7f68c41af650 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:33:47.011 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:46.920+0000 7f68c359e700 1 -- 172.21.15.52:0/658831325 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f68c41a9f20 con 0x7f68c411d920 2024-05-27T07:33:47.011 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:46.921+0000 7f68c359e700 1 --2- 172.21.15.52:0/658831325 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f68c411d920 0x7f68c41a9870 secure :-1 s=READY pgs=700 cs=0 l=1 rev1=1 crypto rx=0x7f68bc00b070 tx=0x7f68bc009c90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:33:47.012 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:46.922+0000 7f68abfff700 1 -- 172.21.15.52:0/658831325 <== mon.0 v2:172.21.15.52:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f68bc021070 con 0x7f68c411d920 2024-05-27T07:33:47.012 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:46.922+0000 7f68abfff700 1 -- 172.21.15.52:0/658831325 <== mon.0 v2:172.21.15.52:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f68bc007e30 con 0x7f68c411d920 2024-05-27T07:33:47.012 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:46.922+0000 7f68abfff700 1 -- 172.21.15.52:0/658831325 <== mon.0 v2:172.21.15.52:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f68bc004780 con 0x7f68c411d920 2024-05-27T07:33:47.012 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:46.923+0000 7f68c9173700 1 -- 172.21.15.52:0/658831325 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f68c41aa1b0 con 0x7f68c411d920 2024-05-27T07:33:47.012 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:46.923+0000 7f68c9173700 1 -- 172.21.15.52:0/658831325 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f68c41aa6a0 con 0x7f68c411d920 2024-05-27T07:33:47.012 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:46.929+0000 7f68abfff700 1 -- 172.21.15.52:0/658831325 <== mon.0 v2:172.21.15.52:3300/0 4 ==== mgrmap(e 26) ==== 149783+0+0 (secure 0 0 0) 0x7f68bc01d450 con 0x7f68c411d920 2024-05-27T07:33:47.012 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:46.930+0000 7f68d7b88b80 1 Processor -- start 2024-05-27T07:33:47.012 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:46.930+0000 7f68d7b88b80 1 -- start start 2024-05-27T07:33:47.012 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:46.931+0000 7f68d7b88b80 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55ed2128ee40 0x55ed2141dd10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:33:47.012 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:46.931+0000 7f68d7b88b80 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55ed2141e250 0x55ed214223a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:33:47.012 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:46.931+0000 7f68d7b88b80 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55ed214228e0 0x55ed21422cf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:33:47.012 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:46.931+0000 7f68d7b88b80 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x55ed2133bc90 con 0x55ed2141e250 2024-05-27T07:33:47.013 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:46.931+0000 7f68d7b88b80 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x55ed2132b830 con 0x55ed2128ee40 2024-05-27T07:33:47.013 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:46.931+0000 7f68d7b88b80 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x55ed2132f870 con 0x55ed214228e0 2024-05-27T07:33:47.013 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:46.931+0000 7f68c259c700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55ed2128ee40 0x55ed2141dd10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:33:47.013 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:46.931+0000 7f68c259c700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55ed2128ee40 0x55ed2141dd10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.52:52300/0 (socket says 172.21.15.52:52300) 2024-05-27T07:33:47.013 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:46.931+0000 7f68c259c700 1 -- 172.21.15.52:0/3407253221 learned_addr learned my addr 172.21.15.52:0/3407253221 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-05-27T07:33:47.013 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:46.932+0000 7f68c259c700 1 --2- 172.21.15.52:0/3407253221 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55ed214228e0 0x55ed21422cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:33:47.013 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:46.932+0000 7f68abfff700 1 --2- 172.21.15.52:0/658831325 >> [v2:172.21.15.52:6800/1822318384,v1:172.21.15.52:6801/1822318384] conn(0x7f68ac0b2bc0 0x7f68ac0b50d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:33:47.013 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:46.933+0000 7f68c259c700 1 -- 172.21.15.52:0/3407253221 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55ed214228e0 msgr2=0x55ed21422cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:33:47.013 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:46.933+0000 7f68c259c700 1 --2- 172.21.15.52:0/3407253221 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55ed214228e0 0x55ed21422cf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:33:47.013 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:46.933+0000 7f68c259c700 1 -- 172.21.15.52:0/3407253221 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55ed2141e250 msgr2=0x55ed214223a0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-05-27T07:33:47.013 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:46.933+0000 7f68c259c700 1 --2- 172.21.15.52:0/3407253221 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55ed2141e250 0x55ed214223a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:33:47.013 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:46.933+0000 7f68abfff700 1 -- 172.21.15.52:0/658831325 <== mon.0 v2:172.21.15.52:3300/0 5 ==== osd_map(119..119 src has 1..119) ==== 9451+0+0 (secure 0 0 0) 0x7f68bc00f020 con 0x7f68c411d920 2024-05-27T07:33:47.014 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:46.933+0000 7f68c259c700 1 -- 172.21.15.52:0/3407253221 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55ed2127b8d0 con 0x55ed2128ee40 2024-05-27T07:33:47.014 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:46.933+0000 7f68c259c700 1 --2- 172.21.15.52:0/3407253221 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55ed2128ee40 0x55ed2141dd10 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7f68b8034720 tx=0x7f68b80094a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f67201df93edb808 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:33:47.014 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:46.934+0000 7f68c359e700 1 --2- 172.21.15.52:0/658831325 >> [v2:172.21.15.52:6800/1822318384,v1:172.21.15.52:6801/1822318384] conn(0x7f68ac0b2bc0 0x7f68ac0b50d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:33:47.014 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:46.937+0000 7f68a97fa700 1 -- 172.21.15.52:0/3407253221 <== mon.2 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f68b8004030 con 0x55ed2128ee40 2024-05-27T07:33:47.014 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:46.937+0000 7f68a97fa700 1 -- 172.21.15.52:0/3407253221 <== mon.2 v2:172.21.15.105:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f68b8004830 con 0x55ed2128ee40 2024-05-27T07:33:47.014 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:46.937+0000 7f68a97fa700 1 -- 172.21.15.52:0/3407253221 <== mon.2 v2:172.21.15.105:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f68b8005280 con 0x55ed2128ee40 2024-05-27T07:33:47.014 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:46.938+0000 7f68d7b88b80 1 -- 172.21.15.52:0/3407253221 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55ed2128ee40 msgr2=0x55ed2141dd10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:33:47.014 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:46.938+0000 7f68d7b88b80 1 --2- 172.21.15.52:0/3407253221 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55ed2128ee40 0x55ed2141dd10 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7f68b8034720 tx=0x7f68b80094a0 comp rx=0 tx=0).stop 2024-05-27T07:33:47.014 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:46.939+0000 7f68d7b88b80 1 -- 172.21.15.52:0/3407253221 shutdown_connections 2024-05-27T07:33:47.014 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:46.939+0000 7f68d7b88b80 1 --2- 172.21.15.52:0/3407253221 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55ed214228e0 0x55ed21422cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:33:47.014 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:46.939+0000 7f68d7b88b80 1 --2- 172.21.15.52:0/3407253221 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55ed2128ee40 0x55ed2141dd10 unknown :-1 s=CLOSED pgs=223 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:33:47.014 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:46.939+0000 7f68d7b88b80 1 --2- 172.21.15.52:0/3407253221 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55ed2141e250 0x55ed214223a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:33:47.015 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:46.939+0000 7f68d7b88b80 1 -- 172.21.15.52:0/3407253221 >> 172.21.15.52:0/3407253221 conn(0x55ed212e9af0 msgr2=0x55ed21407740 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:33:47.015 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:46.939+0000 7f68c359e700 1 --2- 172.21.15.52:0/658831325 >> [v2:172.21.15.52:6800/1822318384,v1:172.21.15.52:6801/1822318384] conn(0x7f68ac0b2bc0 0x7f68ac0b50d0 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7f68bc00b780 tx=0x7f68bc009040 comp rx=0 tx=0).ready entity=mgr.15333 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:33:47.015 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:46.939+0000 7f68d7b88b80 1 -- 172.21.15.52:0/3407253221 shutdown_connections 2024-05-27T07:33:47.015 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:46.939+0000 7f68d7b88b80 1 -- 172.21.15.52:0/3407253221 wait complete. 2024-05-27T07:33:47.015 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:46.940+0000 7f68d7b88b80 1 -- start start 2024-05-27T07:33:47.015 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:46.940+0000 7f68d7b88b80 1 Processor -- start 2024-05-27T07:33:47.015 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:46.941+0000 7f68d7b88b80 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55ed2128ee40 0x55ed21414c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:33:47.015 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:46.941+0000 7f68d7b88b80 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55ed2141e250 0x55ed214151f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:33:47.015 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:46.941+0000 7f68d7b88b80 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55ed214228e0 0x55ed21410060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:33:47.015 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:46.941+0000 7f68d7b88b80 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x55ed21231fa0 con 0x55ed2141e250 2024-05-27T07:33:47.015 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:46.941+0000 7f68d7b88b80 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x55ed212cd7c0 con 0x55ed214228e0 2024-05-27T07:33:47.015 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:46.941+0000 7f68d7b88b80 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x55ed2140cbc0 con 0x55ed2128ee40 2024-05-27T07:33:47.016 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:46.941+0000 7f68d7b88b80 10 client.0 _ll_register_callbacks cb 0x55ed212c6920 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T07:33:47.016 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:46.941+0000 7f68c259c700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55ed2128ee40 0x55ed21414c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:33:47.016 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:46.941+0000 7f68c259c700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55ed2128ee40 0x55ed21414c50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.173:3300/0 says I am v2:172.21.15.52:50858/0 (socket says 172.21.15.52:50858) 2024-05-27T07:33:47.016 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:46.941+0000 7f68c259c700 1 -- 172.21.15.52:0/1116025608 learned_addr learned my addr 172.21.15.52:0/1116025608 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-05-27T07:33:47.016 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:46.941+0000 7f68c259c700 1 --2- 172.21.15.52:0/1116025608 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55ed214228e0 0x55ed21410060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:33:47.016 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:46.942+0000 7f68c259c700 1 -- 172.21.15.52:0/1116025608 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55ed214228e0 msgr2=0x55ed21410060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:33:47.016 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:46.942+0000 7f68c259c700 1 --2- 172.21.15.52:0/1116025608 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55ed214228e0 0x55ed21410060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:33:47.016 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:46.942+0000 7f68c259c700 1 -- 172.21.15.52:0/1116025608 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55ed2141e250 msgr2=0x55ed214151f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:33:47.016 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:46.942+0000 7f68c259c700 1 --2- 172.21.15.52:0/1116025608 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55ed2141e250 0x55ed214151f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:33:47.016 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:46.942+0000 7f68c259c700 1 -- 172.21.15.52:0/1116025608 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55ed212ca690 con 0x55ed2128ee40 2024-05-27T07:33:47.016 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:46.942+0000 7f68c259c700 1 --2- 172.21.15.52:0/1116025608 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55ed2128ee40 0x55ed21414c50 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7f68b8004e60 tx=0x7f68b8009400 comp rx=0 tx=0).ready entity=mon.1 client_cookie=6c029fe29094d602 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:33:47.016 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:46.942+0000 7f68a6ff5700 10 client.0 ms_handle_connect on v2:172.21.15.173:3300/0 2024-05-27T07:33:47.016 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:46.942+0000 7f68a6ff5700 1 -- 172.21.15.52:0/1116025608 <== mon.1 v2:172.21.15.173:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f68b80054d0 con 0x55ed2128ee40 2024-05-27T07:33:47.017 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:46.943+0000 7f68a6ff5700 1 -- 172.21.15.52:0/1116025608 <== mon.1 v2:172.21.15.173:3300/0 2 ==== osd_map(119..119 src has 1..119) ==== 9451+0+0 (secure 0 0 0) 0x7f68b8005670 con 0x55ed2128ee40 2024-05-27T07:33:47.017 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:46.943+0000 7f68d7b88b80 1 -- 172.21.15.52:0/1116025608 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f68b8004830 con 0x55ed2128ee40 2024-05-27T07:33:47.017 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:46.943+0000 7f68a6ff5700 1 -- 172.21.15.52:0/1116025608 <== mon.1 v2:172.21.15.173:3300/0 3 ==== mon_get_version_reply(handle=1 version=103) ==== 24+0+0 (secure 0 0 0) 0x7f68b8004050 con 0x55ed2128ee40 2024-05-27T07:33:47.017 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:46.944+0000 7f68d7b88b80 10 client.26407 fetch_fsmap learned FSMap version 103 2024-05-27T07:33:47.017 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:46.944+0000 7f68d7b88b80 1 -- 172.21.15.52:0/1116025608 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({config=0+,fsmap=103,monmap=4+}) -- 0x55ed214244e0 con 0x55ed2128ee40 2024-05-27T07:33:47.017 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:46.946+0000 7f68a6ff5700 1 -- 172.21.15.52:0/1116025608 <== mon.1 v2:172.21.15.173:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f68b8004830 con 0x55ed2128ee40 2024-05-27T07:33:47.017 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:46.946+0000 7f68a6ff5700 1 -- 172.21.15.52:0/1116025608 <== mon.1 v2:172.21.15.173:3300/0 5 ==== fsmap(e 103) ==== 4215+0+0 (secure 0 0 0) 0x7f68b804e450 con 0x55ed2128ee40 2024-05-27T07:33:47.017 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:46.946+0000 7f68d7b88b80 10 client.26407 fetch_fsmap finished waiting for FSMap version 103 2024-05-27T07:33:47.017 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:46.946+0000 7f68d7b88b80 10 client.26407 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.h 2024-05-27T07:33:47.017 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:46.946+0000 7f68d7b88b80 20 client.26407 populate_metadata read hostname 'smithi052' 2024-05-27T07:33:47.017 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:46.946+0000 7f68d7b88b80 1 --2- 172.21.15.52:0/1116025608 >> [v2:172.21.15.105:6836/191393149,v1:172.21.15.105:6837/191393149] conn(0x55ed21426cb0 0x55ed21429140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:33:47.017 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:46.946+0000 7f68d7b88b80 4 client.26407 mds_command: new command op to 24497 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-27T07:33:47.018 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:46.946+0000 7f68d7b88b80 1 -- 172.21.15.52:0/1116025608 --> [v2:172.21.15.105:6836/191393149,v1:172.21.15.105:6837/191393149] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55ed213d89f0 con 0x55ed21426cb0 2024-05-27T07:33:47.018 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:46.946+0000 7f68c2d9d700 1 --2- 172.21.15.52:0/1116025608 >> [v2:172.21.15.105:6836/191393149,v1:172.21.15.105:6837/191393149] conn(0x55ed21426cb0 0x55ed21429140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:33:47.018 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:46.947+0000 7f68c2d9d700 1 --2- 172.21.15.52:0/1116025608 >> [v2:172.21.15.105:6836/191393149,v1:172.21.15.105:6837/191393149] conn(0x55ed21426cb0 0x55ed21429140 crc :-1 s=READY pgs=47 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=8e79c2117dba4d server_cookie=c2adc402b18b4867 in_seq=0 out_seq=0 2024-05-27T07:33:47.018 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:46.948+0000 7f68a6ff5700 10 client.26407 ms_handle_connect on v2:172.21.15.105:6836/191393149 2024-05-27T07:33:47.018 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:46.948+0000 7f68a6ff5700 1 -- 172.21.15.52:0/1116025608 <== mds.1 v2:172.21.15.105:6836/191393149 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24041 (crc 0 0 0) 0x55ed213d89f0 con 0x55ed21426cb0 2024-05-27T07:33:47.018 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:46.948+0000 7f68a6ff5700 10 client.26407 handle_command_reply: tid=0 2024-05-27T07:33:47.018 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:46.948+0000 7f68d7b88b80 1 client.26407 shutdown 2024-05-27T07:33:47.018 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:46.949+0000 7f68d7b88b80 1 -- 172.21.15.52:0/1116025608 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55ed2128ee40 msgr2=0x55ed21414c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:33:47.018 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:46.949+0000 7f68d7b88b80 1 --2- 172.21.15.52:0/1116025608 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55ed2128ee40 0x55ed21414c50 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7f68b8004e60 tx=0x7f68b8009400 comp rx=0 tx=0).stop 2024-05-27T07:33:47.018 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:46.950+0000 7f68d7b88b80 1 -- 172.21.15.52:0/1116025608 shutdown_connections 2024-05-27T07:33:47.018 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:46.950+0000 7f68d7b88b80 1 --2- 172.21.15.52:0/1116025608 >> [v2:172.21.15.105:6836/191393149,v1:172.21.15.105:6837/191393149] conn(0x55ed21426cb0 0x55ed21429140 crc :-1 s=READY pgs=47 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:33:47.018 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:46.950+0000 7f68d7b88b80 1 --2- 172.21.15.52:0/1116025608 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55ed214228e0 0x55ed21410060 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:33:47.019 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:46.950+0000 7f68d7b88b80 1 --2- 172.21.15.52:0/1116025608 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55ed2128ee40 0x55ed21414c50 unknown :-1 s=CLOSED pgs=177 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:33:47.019 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:46.950+0000 7f68d7b88b80 1 --2- 172.21.15.52:0/1116025608 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55ed2141e250 0x55ed214151f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:33:47.019 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:46.950+0000 7f68d7b88b80 1 -- 172.21.15.52:0/1116025608 >> 172.21.15.52:0/1116025608 conn(0x55ed212e9af0 msgr2=0x55ed2140ba30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:33:47.019 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:46.950+0000 7f68a6ff5700 0 client.26407 ms_handle_reset on v2:172.21.15.105:6836/191393149 2024-05-27T07:33:47.019 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:46.951+0000 7f68d7b88b80 1 -- 172.21.15.52:0/1116025608 shutdown_connections 2024-05-27T07:33:47.019 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:46.952+0000 7f68d7b88b80 1 -- 172.21.15.52:0/1116025608 wait complete. 2024-05-27T07:33:47.019 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:46.954+0000 7f68d7b88b80 20 client.26407 trim_cache size 0 max 16384 2024-05-27T07:33:47.019 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:46.982+0000 7f68d7b88b80 1 Processor -- start 2024-05-27T07:33:47.019 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:46.982+0000 7f68d7b88b80 1 -- start start 2024-05-27T07:33:47.019 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:46.983+0000 7f68d7b88b80 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55ed2141e250 0x55ed21423d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:33:47.019 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:46.983+0000 7f68d7b88b80 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55ed214228e0 0x55ed21414710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:33:47.019 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:46.983+0000 7f68d7b88b80 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55ed21408ff0 0x55ed21414c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:33:47.020 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:46.983+0000 7f68d7b88b80 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x7f68c41b6f60 con 0x55ed214228e0 2024-05-27T07:33:47.020 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:46.983+0000 7f68d7b88b80 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f68c411a090 con 0x55ed2141e250 2024-05-27T07:33:47.020 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:46.983+0000 7f68d7b88b80 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f68c41b6860 con 0x55ed21408ff0 2024-05-27T07:33:47.020 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:46.983+0000 7f68c259c700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55ed2141e250 0x55ed21423d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:33:47.020 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:46.983+0000 7f68c259c700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55ed2141e250 0x55ed21423d00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.52:52320/0 (socket says 172.21.15.52:52320) 2024-05-27T07:33:47.020 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:46.983+0000 7f68c259c700 1 -- 172.21.15.52:0/1287496355 learned_addr learned my addr 172.21.15.52:0/1287496355 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-05-27T07:33:47.020 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:46.983+0000 7f68c259c700 1 --2- 172.21.15.52:0/1287496355 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55ed21408ff0 0x55ed21414c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:33:47.020 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:46.984+0000 7f68c259c700 1 -- 172.21.15.52:0/1287496355 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55ed2141e250 msgr2=0x55ed21423d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:33:47.020 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:46.984+0000 7f68c259c700 1 --2- 172.21.15.52:0/1287496355 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55ed2141e250 0x55ed21423d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:33:47.020 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:46.984+0000 7f68c259c700 1 -- 172.21.15.52:0/1287496355 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55ed214228e0 msgr2=0x55ed21414710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:33:47.020 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:46.984+0000 7f68c259c700 1 --2- 172.21.15.52:0/1287496355 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55ed214228e0 0x55ed21414710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:33:47.020 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:46.984+0000 7f68c259c700 1 -- 172.21.15.52:0/1287496355 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55ed2141e970 con 0x55ed21408ff0 2024-05-27T07:33:47.021 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:46.984+0000 7f68c259c700 1 --2- 172.21.15.52:0/1287496355 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55ed21408ff0 0x55ed21414c50 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7f68b804e8a0 tx=0x7f68b8000bc0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=3dc8dc38a4c19627 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:33:47.021 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:46.985+0000 7f68a8ff9700 1 -- 172.21.15.52:0/1287496355 <== mon.1 v2:172.21.15.173:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f68b8056030 con 0x55ed21408ff0 2024-05-27T07:33:47.021 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:46.985+0000 7f68a8ff9700 1 -- 172.21.15.52:0/1287496355 <== mon.1 v2:172.21.15.173:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f68b8038070 con 0x55ed21408ff0 2024-05-27T07:33:47.021 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:46.985+0000 7f68a8ff9700 1 -- 172.21.15.52:0/1287496355 <== mon.1 v2:172.21.15.173:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f68b8005410 con 0x55ed21408ff0 2024-05-27T07:33:47.021 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:46.986+0000 7f68d7b88b80 1 -- 172.21.15.52:0/1287496355 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55ed21408ff0 msgr2=0x55ed21414c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:33:47.021 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:46.986+0000 7f68d7b88b80 1 --2- 172.21.15.52:0/1287496355 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55ed21408ff0 0x55ed21414c50 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7f68b804e8a0 tx=0x7f68b8000bc0 comp rx=0 tx=0).stop 2024-05-27T07:33:47.021 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:46.986+0000 7f68d7b88b80 1 -- 172.21.15.52:0/1287496355 shutdown_connections 2024-05-27T07:33:47.021 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:46.986+0000 7f68d7b88b80 1 --2- 172.21.15.52:0/1287496355 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55ed21408ff0 0x55ed21414c50 unknown :-1 s=CLOSED pgs=178 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:33:47.021 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:46.986+0000 7f68d7b88b80 1 --2- 172.21.15.52:0/1287496355 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55ed2141e250 0x55ed21423d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:33:47.021 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:46.986+0000 7f68d7b88b80 1 --2- 172.21.15.52:0/1287496355 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55ed214228e0 0x55ed21414710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:33:47.021 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:46.986+0000 7f68d7b88b80 1 -- 172.21.15.52:0/1287496355 >> 172.21.15.52:0/1287496355 conn(0x55ed212e9af0 msgr2=0x55ed21421ce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:33:47.021 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:46.986+0000 7f68d7b88b80 1 -- 172.21.15.52:0/1287496355 shutdown_connections 2024-05-27T07:33:47.022 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:46.986+0000 7f68d7b88b80 1 -- 172.21.15.52:0/1287496355 wait complete. 2024-05-27T07:33:47.022 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:46.987+0000 7f68d7b88b80 1 -- start start 2024-05-27T07:33:47.022 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:46.987+0000 7f68d7b88b80 1 Processor -- start 2024-05-27T07:33:47.022 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:46.988+0000 7f68d7b88b80 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55ed21408ff0 0x55ed21489b00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:33:47.022 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:46.988+0000 7f68d7b88b80 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55ed2141e250 0x55ed2148a040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:33:47.022 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:46.988+0000 7f68d7b88b80 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55ed214228e0 0x55ed21417880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:33:47.022 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:46.988+0000 7f68d7b88b80 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x7f68c411ae60 con 0x55ed21408ff0 2024-05-27T07:33:47.022 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:46.988+0000 7f68d7b88b80 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f68c411a760 con 0x55ed214228e0 2024-05-27T07:33:47.022 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:46.988+0000 7f68d7b88b80 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x55ed2141ebf0 con 0x55ed2141e250 2024-05-27T07:33:47.022 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:46.988+0000 7f68d7b88b80 10 client.0 _ll_register_callbacks cb 0x55ed212c6920 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T07:33:47.022 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:46.988+0000 7f68c2d9d700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55ed2141e250 0x55ed2148a040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:33:47.022 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:46.988+0000 7f68c2d9d700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55ed2141e250 0x55ed2148a040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.173:3300/0 says I am v2:172.21.15.52:50866/0 (socket says 172.21.15.52:50866) 2024-05-27T07:33:47.023 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:46.988+0000 7f68c2d9d700 1 -- 172.21.15.52:0/3459754956 learned_addr learned my addr 172.21.15.52:0/3459754956 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-05-27T07:33:47.023 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:46.988+0000 7f68c259c700 1 --2- 172.21.15.52:0/3459754956 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55ed21408ff0 0x55ed21489b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:33:47.023 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:46.989+0000 7f68c259c700 1 --2- 172.21.15.52:0/3459754956 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55ed214228e0 0x55ed21417880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:33:47.023 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:46.989+0000 7f68c2d9d700 1 -- 172.21.15.52:0/3459754956 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55ed214228e0 msgr2=0x55ed21417880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:33:47.023 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:46.989+0000 7f68c2d9d700 1 --2- 172.21.15.52:0/3459754956 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55ed214228e0 0x55ed21417880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:33:47.023 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:46.989+0000 7f68c2d9d700 1 -- 172.21.15.52:0/3459754956 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55ed21408ff0 msgr2=0x55ed21489b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:33:47.023 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:46.989+0000 7f68c2d9d700 1 --2- 172.21.15.52:0/3459754956 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55ed21408ff0 0x55ed21489b00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:33:47.023 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:46.989+0000 7f68c2d9d700 1 -- 172.21.15.52:0/3459754956 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55ed21417ec0 con 0x55ed2141e250 2024-05-27T07:33:47.023 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:46.989+0000 7f68c259c700 1 --2- 172.21.15.52:0/3459754956 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55ed214228e0 0x55ed21417880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-27T07:33:47.023 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:46.989+0000 7f68c2d9d700 1 --2- 172.21.15.52:0/3459754956 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55ed2141e250 0x55ed2148a040 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x55ed21408fc0 tx=0x7f68b4009f60 comp rx=0 tx=0).ready entity=mon.1 client_cookie=d201de3429423eef server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:33:47.023 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:46.990+0000 7f68a6ff5700 10 client.0 ms_handle_connect on v2:172.21.15.173:3300/0 2024-05-27T07:33:47.023 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:46.990+0000 7f68a6ff5700 1 -- 172.21.15.52:0/3459754956 <== mon.1 v2:172.21.15.173:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f68b4010d90 con 0x55ed2141e250 2024-05-27T07:33:47.023 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:46.990+0000 7f68d7b88b80 1 -- 172.21.15.52:0/3459754956 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f68b8038070 con 0x55ed2141e250 2024-05-27T07:33:47.024 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:46.990+0000 7f68a6ff5700 1 -- 172.21.15.52:0/3459754956 <== mon.1 v2:172.21.15.173:3300/0 2 ==== osd_map(119..119 src has 1..119) ==== 9451+0+0 (secure 0 0 0) 0x7f68b4004030 con 0x55ed2141e250 2024-05-27T07:33:47.024 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:46.990+0000 7f68a6ff5700 1 -- 172.21.15.52:0/3459754956 <== mon.1 v2:172.21.15.173:3300/0 3 ==== mon_get_version_reply(handle=1 version=103) ==== 24+0+0 (secure 0 0 0) 0x7f68b4004270 con 0x55ed2141e250 2024-05-27T07:33:47.024 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:46.990+0000 7f68d7b88b80 10 client.26413 fetch_fsmap learned FSMap version 103 2024-05-27T07:33:47.024 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:46.991+0000 7f68d7b88b80 1 -- 172.21.15.52:0/3459754956 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({config=0+,fsmap=103,monmap=4+}) -- 0x55ed214911f0 con 0x55ed2141e250 2024-05-27T07:33:47.024 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:46.994+0000 7f68a6ff5700 1 -- 172.21.15.52:0/3459754956 <== mon.1 v2:172.21.15.173:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f68b4004410 con 0x55ed2141e250 2024-05-27T07:33:47.024 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:46.994+0000 7f68a6ff5700 1 -- 172.21.15.52:0/3459754956 <== mon.1 v2:172.21.15.173:3300/0 5 ==== fsmap(e 103) ==== 4215+0+0 (secure 0 0 0) 0x7f68b40185e0 con 0x55ed2141e250 2024-05-27T07:33:47.024 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:46.994+0000 7f68d7b88b80 10 client.26413 fetch_fsmap finished waiting for FSMap version 103 2024-05-27T07:33:47.024 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:46.994+0000 7f68d7b88b80 10 client.26413 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.h 2024-05-27T07:33:47.024 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:46.994+0000 7f68d7b88b80 20 client.26413 populate_metadata read hostname 'smithi052' 2024-05-27T07:33:47.024 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:46.994+0000 7f68d7b88b80 1 --2- 172.21.15.52:0/3459754956 >> [v2:172.21.15.105:6836/191393149,v1:172.21.15.105:6837/191393149] conn(0x55ed21491950 0x55ed21493d70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:33:47.024 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:46.994+0000 7f68d7b88b80 4 client.26413 mds_command: new command op to 24497 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-05-27T07:33:47.024 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:46.994+0000 7f68d7b88b80 1 -- 172.21.15.52:0/3459754956 --> [v2:172.21.15.105:6836/191393149,v1:172.21.15.105:6837/191393149] -- command(tid 0: {"prefix": "damage ls"}) -- 0x7f68b8038070 con 0x55ed21491950 2024-05-27T07:33:47.024 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:46.994+0000 7f68c259c700 1 --2- 172.21.15.52:0/3459754956 >> [v2:172.21.15.105:6836/191393149,v1:172.21.15.105:6837/191393149] conn(0x55ed21491950 0x55ed21493d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:33:47.025 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:46.995+0000 7f68c259c700 1 --2- 172.21.15.52:0/3459754956 >> [v2:172.21.15.105:6836/191393149,v1:172.21.15.105:6837/191393149] conn(0x55ed21491950 0x55ed21493d70 crc :-1 s=READY pgs=48 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=9f58b07cf1f56a20 server_cookie=497662b1d0baadc2 in_seq=0 out_seq=0 2024-05-27T07:33:47.025 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:46.995+0000 7f68a6ff5700 10 client.26413 ms_handle_connect on v2:172.21.15.105:6836/191393149 2024-05-27T07:33:47.025 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:46.996+0000 7f68a6ff5700 1 -- 172.21.15.52:0/3459754956 <== mds.1 v2:172.21.15.105:6836/191393149 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x7f68b8038070 con 0x55ed21491950 2024-05-27T07:33:47.025 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:46.996+0000 7f68a6ff5700 10 client.26413 handle_command_reply: tid=0 2024-05-27T07:33:47.025 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:46.996+0000 7f68d7b88b80 1 client.26413 shutdown 2024-05-27T07:33:47.025 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:46.996+0000 7f68d7b88b80 1 -- 172.21.15.52:0/3459754956 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55ed2141e250 msgr2=0x55ed2148a040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:33:47.025 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:46.996+0000 7f68d7b88b80 1 --2- 172.21.15.52:0/3459754956 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55ed2141e250 0x55ed2148a040 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x55ed21408fc0 tx=0x7f68b4009f60 comp rx=0 tx=0).stop 2024-05-27T07:33:47.025 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:46.996+0000 7f68d7b88b80 1 -- 172.21.15.52:0/3459754956 shutdown_connections 2024-05-27T07:33:47.025 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:46.996+0000 7f68d7b88b80 1 --2- 172.21.15.52:0/3459754956 >> [v2:172.21.15.105:6836/191393149,v1:172.21.15.105:6837/191393149] conn(0x55ed21491950 0x55ed21493d70 crc :-1 s=READY pgs=48 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:33:47.025 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:46.996+0000 7f68a6ff5700 0 client.26413 ms_handle_reset on v2:172.21.15.105:6836/191393149 2024-05-27T07:33:47.025 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:46.996+0000 7f68d7b88b80 1 --2- 172.21.15.52:0/3459754956 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55ed214228e0 0x55ed21417880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:33:47.025 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:46.996+0000 7f68d7b88b80 1 --2- 172.21.15.52:0/3459754956 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55ed21408ff0 0x55ed21489b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:33:47.025 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:46.996+0000 7f68d7b88b80 1 --2- 172.21.15.52:0/3459754956 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55ed2141e250 0x55ed2148a040 unknown :-1 s=CLOSED pgs=179 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:33:47.025 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:46.996+0000 7f68d7b88b80 1 -- 172.21.15.52:0/3459754956 >> 172.21.15.52:0/3459754956 conn(0x55ed212e9af0 msgr2=0x55ed214071c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:33:47.026 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:46.997+0000 7f68d7b88b80 1 -- 172.21.15.52:0/3459754956 shutdown_connections 2024-05-27T07:33:47.026 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:46.997+0000 7f68d7b88b80 1 -- 172.21.15.52:0/3459754956 wait complete. 2024-05-27T07:33:47.026 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:46.997+0000 7f68d7b88b80 20 client.26413 trim_cache size 0 max 16384 2024-05-27T07:33:47.026 INFO:teuthology.orchestra.run.smithi052.stdout:[] 2024-05-27T07:33:47.027 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:46.999+0000 7f68c9173700 1 -- 172.21.15.52:0/658831325 >> [v2:172.21.15.52:6800/1822318384,v1:172.21.15.52:6801/1822318384] conn(0x7f68ac0b2bc0 msgr2=0x7f68ac0b50d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:33:47.027 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:46.999+0000 7f68c9173700 1 --2- 172.21.15.52:0/658831325 >> [v2:172.21.15.52:6800/1822318384,v1:172.21.15.52:6801/1822318384] conn(0x7f68ac0b2bc0 0x7f68ac0b50d0 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7f68bc00b780 tx=0x7f68bc009040 comp rx=0 tx=0).stop 2024-05-27T07:33:47.027 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:46.999+0000 7f68c9173700 1 -- 172.21.15.52:0/658831325 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f68c411d920 msgr2=0x7f68c41a9870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:33:47.028 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:46.999+0000 7f68c9173700 1 --2- 172.21.15.52:0/658831325 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f68c411d920 0x7f68c41a9870 secure :-1 s=READY pgs=700 cs=0 l=1 rev1=1 crypto rx=0x7f68bc00b070 tx=0x7f68bc009c90 comp rx=0 tx=0).stop 2024-05-27T07:33:47.028 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:47.000+0000 7f68c9173700 1 -- 172.21.15.52:0/658831325 shutdown_connections 2024-05-27T07:33:47.028 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:47.000+0000 7f68c9173700 1 --2- 172.21.15.52:0/658831325 >> [v2:172.21.15.52:6800/1822318384,v1:172.21.15.52:6801/1822318384] conn(0x7f68ac0b2bc0 0x7f68ac0b50d0 unknown :-1 s=CLOSED pgs=168 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:33:47.028 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:47.000+0000 7f68c9173700 1 --2- 172.21.15.52:0/658831325 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f68c411d920 0x7f68c41a9870 unknown :-1 s=CLOSED pgs=700 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:33:47.028 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:47.000+0000 7f68c9173700 1 --2- 172.21.15.52:0/658831325 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f68c410a7b0 0x7f68c41af650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:33:47.028 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:47.000+0000 7f68c9173700 1 --2- 172.21.15.52:0/658831325 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f68c410b100 0x7f68c41afb90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:33:47.028 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:47.000+0000 7f68c9173700 1 -- 172.21.15.52:0/658831325 >> 172.21.15.52:0/658831325 conn(0x7f68c406f2b0 msgr2=0x7f68c411b3d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:33:47.028 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:47.003+0000 7f68c9173700 1 -- 172.21.15.52:0/658831325 shutdown_connections 2024-05-27T07:33:47.028 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:47.003+0000 7f68c9173700 1 -- 172.21.15.52:0/658831325 wait complete. 2024-05-27T07:33:48.138 DEBUG:teuthology.orchestra.run.smithi052:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:38f9d900c83c63799fbdbe61acc9a11b0d3554a6 shell --fsid 73df2636-1bf8-11ef-bc9b-c7b262605968 -- ceph tell mds.1:0 damage ls 2024-05-27T07:33:48.460 INFO:teuthology.orchestra.run.smithi052.stderr:Inferring config /var/lib/ceph/73df2636-1bf8-11ef-bc9b-c7b262605968/mon.a/config 2024-05-27T07:33:48.483 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:33:48 smithi052 ceph-mon[31955]: pgmap v266: 129 pgs: 129 active+clean; 662 MiB data, 4.5 GiB used, 1.0 TiB / 1.0 TiB avail; 29 KiB/s rd, 1.6 MiB/s wr, 718 op/s 2024-05-27T07:33:48.606 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:33:48 smithi173 ceph-mon[37000]: pgmap v266: 129 pgs: 129 active+clean; 662 MiB data, 4.5 GiB used, 1.0 TiB / 1.0 TiB avail; 29 KiB/s rd, 1.6 MiB/s wr, 718 op/s 2024-05-27T07:33:48.694 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:33:48 smithi105 ceph-mon[35977]: pgmap v266: 129 pgs: 129 active+clean; 662 MiB data, 4.5 GiB used, 1.0 TiB / 1.0 TiB avail; 29 KiB/s rd, 1.6 MiB/s wr, 718 op/s 2024-05-27T07:33:50.245 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:50.250+0000 7fe9de50b700 1 -- 172.21.15.52:0/4133664920 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fe9d811d920 msgr2=0x7fe9d811fd40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:33:50.245 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:50.250+0000 7fe9de50b700 1 --2- 172.21.15.52:0/4133664920 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fe9d811d920 0x7fe9d811fd40 secure :-1 s=READY pgs=701 cs=0 l=1 rev1=1 crypto rx=0x7fe9d000b0a0 tx=0x7fe9d000b3b0 comp rx=0 tx=0).stop 2024-05-27T07:33:50.245 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:50.251+0000 7fe9de50b700 1 -- 172.21.15.52:0/4133664920 shutdown_connections 2024-05-27T07:33:50.245 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:50.251+0000 7fe9de50b700 1 --2- 172.21.15.52:0/4133664920 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fe9d811d920 0x7fe9d811fd40 unknown :-1 s=CLOSED pgs=701 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:33:50.245 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:50.251+0000 7fe9de50b700 1 --2- 172.21.15.52:0/4133664920 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fe9d810a920 0x7fe9d810ad30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:33:50.245 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:50.251+0000 7fe9de50b700 1 --2- 172.21.15.52:0/4133664920 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fe9d810b270 0x7fe9d811d3e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:33:50.246 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:50.251+0000 7fe9de50b700 1 -- 172.21.15.52:0/4133664920 >> 172.21.15.52:0/4133664920 conn(0x7fe9d806f4c0 msgr2=0x7fe9d806f900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:33:50.246 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:50.251+0000 7fe9de50b700 1 -- 172.21.15.52:0/4133664920 shutdown_connections 2024-05-27T07:33:50.246 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:50.252+0000 7fe9de50b700 1 -- 172.21.15.52:0/4133664920 wait complete. 2024-05-27T07:33:50.247 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:50.253+0000 7fe9de50b700 1 Processor -- start 2024-05-27T07:33:50.247 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:50.253+0000 7fe9de50b700 1 -- start start 2024-05-27T07:33:50.248 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:50.253+0000 7fe9de50b700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fe9d810a920 0x7fe9d81a9d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:33:50.248 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:50.253+0000 7fe9de50b700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fe9d810b270 0x7fe9d81aa2d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:33:50.248 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:50.253+0000 7fe9de50b700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fe9d811d920 0x7fe9d81ae560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:33:50.248 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:50.253+0000 7fe9de50b700 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x7fe9d8123320 con 0x7fe9d811d920 2024-05-27T07:33:50.248 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:50.253+0000 7fe9d7fff700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fe9d810a920 0x7fe9d81a9d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:33:50.248 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:50.253+0000 7fe9d7fff700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fe9d810a920 0x7fe9d81a9d30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.173:3300/0 says I am v2:172.21.15.52:50880/0 (socket says 172.21.15.52:50880) 2024-05-27T07:33:50.248 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:50.253+0000 7fe9d7fff700 1 -- 172.21.15.52:0/1456130657 learned_addr learned my addr 172.21.15.52:0/1456130657 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-05-27T07:33:50.248 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:50.254+0000 7fe9de50b700 1 -- 172.21.15.52:0/1456130657 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7fe9d81231a0 con 0x7fe9d810b270 2024-05-27T07:33:50.249 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:50.254+0000 7fe9de50b700 1 -- 172.21.15.52:0/1456130657 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7fe9d81234a0 con 0x7fe9d810a920 2024-05-27T07:33:50.249 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:50.254+0000 7fe9cf7fe700 1 --2- 172.21.15.52:0/1456130657 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fe9d810b270 0x7fe9d81aa2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:33:50.249 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:50.254+0000 7fe9dcaa8700 1 --2- 172.21.15.52:0/1456130657 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fe9d811d920 0x7fe9d81ae560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:33:50.249 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:50.254+0000 7fe9dcaa8700 1 -- 172.21.15.52:0/1456130657 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fe9d810a920 msgr2=0x7fe9d81a9d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:33:50.250 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:50.254+0000 7fe9dcaa8700 1 --2- 172.21.15.52:0/1456130657 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fe9d810a920 0x7fe9d81a9d30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:33:50.250 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:50.254+0000 7fe9dcaa8700 1 -- 172.21.15.52:0/1456130657 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fe9d810b270 msgr2=0x7fe9d81aa2d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:33:50.250 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:50.254+0000 7fe9dcaa8700 1 --2- 172.21.15.52:0/1456130657 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fe9d810b270 0x7fe9d81aa2d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:33:50.250 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:50.254+0000 7fe9dcaa8700 1 -- 172.21.15.52:0/1456130657 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe9d81aec00 con 0x7fe9d811d920 2024-05-27T07:33:50.250 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:50.254+0000 7fe9cf7fe700 1 --2- 172.21.15.52:0/1456130657 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fe9d810b270 0x7fe9d81aa2d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-27T07:33:50.250 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:50.254+0000 7fe9dcaa8700 1 --2- 172.21.15.52:0/1456130657 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fe9d811d920 0x7fe9d81ae560 secure :-1 s=READY pgs=702 cs=0 l=1 rev1=1 crypto rx=0x7fe9d000b070 tx=0x7fe9d0004c80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:33:50.250 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:50.255+0000 7fe9d5ffb700 1 -- 172.21.15.52:0/1456130657 <== mon.0 v2:172.21.15.52:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe9d0021070 con 0x7fe9d811d920 2024-05-27T07:33:50.250 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:50.255+0000 7fe9d5ffb700 1 -- 172.21.15.52:0/1456130657 <== mon.0 v2:172.21.15.52:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fe9d0004e30 con 0x7fe9d811d920 2024-05-27T07:33:50.250 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:50.255+0000 7fe9d5ffb700 1 -- 172.21.15.52:0/1456130657 <== mon.0 v2:172.21.15.52:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe9d0004620 con 0x7fe9d811d920 2024-05-27T07:33:50.250 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:50.255+0000 7fe9de50b700 1 -- 172.21.15.52:0/1456130657 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe9d81aee90 con 0x7fe9d811d920 2024-05-27T07:33:50.251 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:50.255+0000 7fe9de50b700 1 -- 172.21.15.52:0/1456130657 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe9d81af350 con 0x7fe9d811d920 2024-05-27T07:33:50.252 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:50.258+0000 7fe9d5ffb700 1 -- 172.21.15.52:0/1456130657 <== mon.0 v2:172.21.15.52:3300/0 4 ==== mgrmap(e 26) ==== 149783+0+0 (secure 0 0 0) 0x7fe9d0007980 con 0x7fe9d811d920 2024-05-27T07:33:50.253 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:50.259+0000 7fe9d5ffb700 1 --2- 172.21.15.52:0/1456130657 >> [v2:172.21.15.52:6800/1822318384,v1:172.21.15.52:6801/1822318384] conn(0x7fe9c40b2a70 0x7fe9c40b4f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:33:50.253 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:50.259+0000 7fe9ecf20b80 1 Processor -- start 2024-05-27T07:33:50.254 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:50.259+0000 7fe9d5ffb700 1 -- 172.21.15.52:0/1456130657 <== mon.0 v2:172.21.15.52:3300/0 5 ==== osd_map(119..119 src has 1..119) ==== 9451+0+0 (secure 0 0 0) 0x7fe9d000f020 con 0x7fe9d811d920 2024-05-27T07:33:50.254 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:50.259+0000 7fe9d7fff700 1 --2- 172.21.15.52:0/1456130657 >> [v2:172.21.15.52:6800/1822318384,v1:172.21.15.52:6801/1822318384] conn(0x7fe9c40b2a70 0x7fe9c40b4f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:33:50.254 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:50.259+0000 7fe9ecf20b80 1 -- start start 2024-05-27T07:33:50.255 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:50.260+0000 7fe9ecf20b80 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x564c20bbae40 0x564c20d3e710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:33:50.255 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:50.260+0000 7fe9ecf20b80 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x564c20d3ec50 0x564c20d38a40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:33:50.256 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:50.260+0000 7fe9ecf20b80 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x564c20d38f80 0x564c20d39390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:33:50.256 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:50.260+0000 7fe9ecf20b80 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x564c20c67c90 con 0x564c20d3ec50 2024-05-27T07:33:50.256 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:50.260+0000 7fe9ecf20b80 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x564c20c57830 con 0x564c20d38f80 2024-05-27T07:33:50.256 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:50.260+0000 7fe9ecf20b80 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x564c20c5b870 con 0x564c20bbae40 2024-05-27T07:33:50.256 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:50.260+0000 7fe9d7fff700 1 --2- 172.21.15.52:0/1456130657 >> [v2:172.21.15.52:6800/1822318384,v1:172.21.15.52:6801/1822318384] conn(0x7fe9c40b2a70 0x7fe9c40b4f80 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7fe9c8009ad0 tx=0x7fe9c8009290 comp rx=0 tx=0).ready entity=mgr.15333 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:33:50.256 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:50.260+0000 7fe9cf7fe700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x564c20d3ec50 0x564c20d38a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:33:50.256 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:50.260+0000 7fe9dcaa8700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x564c20d38f80 0x564c20d39390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:33:50.256 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:50.260+0000 7fe9dcaa8700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x564c20d38f80 0x564c20d39390 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.52:52356/0 (socket says 172.21.15.52:52356) 2024-05-27T07:33:50.256 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:50.260+0000 7fe9cf7fe700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x564c20d3ec50 0x564c20d38a40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.52:56854/0 (socket says 172.21.15.52:56854) 2024-05-27T07:33:50.256 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:50.260+0000 7fe9dcaa8700 1 -- 172.21.15.52:0/3489234923 learned_addr learned my addr 172.21.15.52:0/3489234923 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-05-27T07:33:50.256 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:50.260+0000 7fe9d7fff700 1 --2- 172.21.15.52:0/3489234923 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x564c20bbae40 0x564c20d3e710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:33:50.257 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:50.260+0000 7fe9dcaa8700 1 -- 172.21.15.52:0/3489234923 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x564c20bbae40 msgr2=0x564c20d3e710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:33:50.257 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:50.260+0000 7fe9dcaa8700 1 --2- 172.21.15.52:0/3489234923 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x564c20bbae40 0x564c20d3e710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:33:50.257 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:50.260+0000 7fe9dcaa8700 1 -- 172.21.15.52:0/3489234923 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x564c20d3ec50 msgr2=0x564c20d38a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:33:50.257 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:50.260+0000 7fe9dcaa8700 1 --2- 172.21.15.52:0/3489234923 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x564c20d3ec50 0x564c20d38a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:33:50.257 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:50.260+0000 7fe9dcaa8700 1 -- 172.21.15.52:0/3489234923 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x564c20ba78d0 con 0x564c20d38f80 2024-05-27T07:33:50.257 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:50.261+0000 7fe9dcaa8700 1 --2- 172.21.15.52:0/3489234923 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x564c20d38f80 0x564c20d39390 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7fe9d000d010 tx=0x7fe9d000b3b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d97eef71d33bfc8a server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:33:50.257 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:50.261+0000 7fe9ce7fc700 1 -- 172.21.15.52:0/3489234923 <== mon.2 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe9d00aa7b0 con 0x564c20d38f80 2024-05-27T07:33:50.257 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:50.261+0000 7fe9ce7fc700 1 -- 172.21.15.52:0/3489234923 <== mon.2 v2:172.21.15.105:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fe9d00ab3a0 con 0x564c20d38f80 2024-05-27T07:33:50.257 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:50.261+0000 7fe9ce7fc700 1 -- 172.21.15.52:0/3489234923 <== mon.2 v2:172.21.15.105:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe9d0004870 con 0x564c20d38f80 2024-05-27T07:33:50.262 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:50.262+0000 7fe9ecf20b80 1 -- 172.21.15.52:0/3489234923 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x564c20d38f80 msgr2=0x564c20d39390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:33:50.262 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:50.262+0000 7fe9ecf20b80 1 --2- 172.21.15.52:0/3489234923 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x564c20d38f80 0x564c20d39390 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7fe9d000d010 tx=0x7fe9d000b3b0 comp rx=0 tx=0).stop 2024-05-27T07:33:50.263 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:50.262+0000 7fe9ecf20b80 1 -- 172.21.15.52:0/3489234923 shutdown_connections 2024-05-27T07:33:50.263 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:50.262+0000 7fe9ecf20b80 1 --2- 172.21.15.52:0/3489234923 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x564c20d38f80 0x564c20d39390 unknown :-1 s=CLOSED pgs=224 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:33:50.263 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:50.262+0000 7fe9ecf20b80 1 --2- 172.21.15.52:0/3489234923 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x564c20bbae40 0x564c20d3e710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:33:50.263 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:50.262+0000 7fe9ecf20b80 1 --2- 172.21.15.52:0/3489234923 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x564c20d3ec50 0x564c20d38a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:33:50.263 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:50.262+0000 7fe9ecf20b80 1 -- 172.21.15.52:0/3489234923 >> 172.21.15.52:0/3489234923 conn(0x564c20c15af0 msgr2=0x564c20d33640 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:33:50.263 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:50.262+0000 7fe9ecf20b80 1 -- 172.21.15.52:0/3489234923 shutdown_connections 2024-05-27T07:33:50.263 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:50.262+0000 7fe9ecf20b80 1 -- 172.21.15.52:0/3489234923 wait complete. 2024-05-27T07:33:50.263 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:50.262+0000 7fe9ecf20b80 1 -- start start 2024-05-27T07:33:50.263 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:50.263+0000 7fe9ecf20b80 1 Processor -- start 2024-05-27T07:33:50.263 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:50.263+0000 7fe9ecf20b80 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x564c20bbae40 0x564c20d42390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:33:50.263 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:50.263+0000 7fe9ecf20b80 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x564c20d38f80 0x564c20d428d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:33:50.264 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:50.263+0000 7fe9ecf20b80 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x564c20d3ec50 0x564c20d42e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:33:50.264 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:50.263+0000 7fe9ecf20b80 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x564c20b5dfa0 con 0x564c20d3ec50 2024-05-27T07:33:50.264 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:50.263+0000 7fe9ecf20b80 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x564c20bf97c0 con 0x564c20d38f80 2024-05-27T07:33:50.264 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:50.263+0000 7fe9ecf20b80 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x564c20d45500 con 0x564c20bbae40 2024-05-27T07:33:50.264 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:50.263+0000 7fe9ecf20b80 10 client.0 _ll_register_callbacks cb 0x564c20bf2920 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T07:33:50.264 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:50.263+0000 7fe9cf7fe700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x564c20d38f80 0x564c20d428d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:33:50.264 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:50.263+0000 7fe9dcaa8700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x564c20d3ec50 0x564c20d42e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:33:50.264 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:50.263+0000 7fe9cf7fe700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x564c20d38f80 0x564c20d428d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.52:52358/0 (socket says 172.21.15.52:52358) 2024-05-27T07:33:50.264 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:50.263+0000 7fe9dcaa8700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x564c20d3ec50 0x564c20d42e10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.52:56868/0 (socket says 172.21.15.52:56868) 2024-05-27T07:33:50.264 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:50.263+0000 7fe9cf7fe700 1 -- 172.21.15.52:0/2496953060 learned_addr learned my addr 172.21.15.52:0/2496953060 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-05-27T07:33:50.264 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:50.263+0000 7fe9d7fff700 1 --2- 172.21.15.52:0/2496953060 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x564c20bbae40 0x564c20d42390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:33:50.265 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:50.264+0000 7fe9dcaa8700 1 -- 172.21.15.52:0/2496953060 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x564c20bbae40 msgr2=0x564c20d42390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:33:50.265 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:50.264+0000 7fe9dcaa8700 1 --2- 172.21.15.52:0/2496953060 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x564c20bbae40 0x564c20d42390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:33:50.265 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:50.264+0000 7fe9dcaa8700 1 -- 172.21.15.52:0/2496953060 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x564c20d38f80 msgr2=0x564c20d428d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:33:50.265 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:50.264+0000 7fe9dcaa8700 1 --2- 172.21.15.52:0/2496953060 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x564c20d38f80 0x564c20d428d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:33:50.265 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:50.264+0000 7fe9dcaa8700 1 -- 172.21.15.52:0/2496953060 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({osdmap=0}) -- 0x564c20bf6690 con 0x564c20d3ec50 2024-05-27T07:33:50.265 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:50.264+0000 7fe9cf7fe700 1 --2- 172.21.15.52:0/2496953060 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x564c20d38f80 0x564c20d428d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-27T07:33:50.265 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:50.264+0000 7fe9dcaa8700 1 --2- 172.21.15.52:0/2496953060 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x564c20d3ec50 0x564c20d42e10 secure :-1 s=READY pgs=703 cs=0 l=1 rev1=1 crypto rx=0x7fe9d00aa9a0 tx=0x7fe9d0009e10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c5763e0395ed128c server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:33:50.265 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:50.264+0000 7fe9b77fe700 10 client.0 ms_handle_connect on v2:172.21.15.52:3300/0 2024-05-27T07:33:50.266 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:50.264+0000 7fe9b77fe700 1 -- 172.21.15.52:0/2496953060 <== mon.0 v2:172.21.15.52:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe9d00c9020 con 0x564c20d3ec50 2024-05-27T07:33:50.266 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:50.264+0000 7fe9b77fe700 1 -- 172.21.15.52:0/2496953060 <== mon.0 v2:172.21.15.52:3300/0 2 ==== osd_map(119..119 src has 1..119) ==== 9451+0+0 (secure 0 0 0) 0x7fe9d0003d20 con 0x564c20d3ec50 2024-05-27T07:33:50.266 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:50.264+0000 7fe9ecf20b80 1 -- 172.21.15.52:0/2496953060 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fe9d00ab3a0 con 0x564c20d3ec50 2024-05-27T07:33:50.266 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:50.264+0000 7fe9b77fe700 1 -- 172.21.15.52:0/2496953060 <== mon.0 v2:172.21.15.52:3300/0 3 ==== mon_get_version_reply(handle=1 version=103) ==== 24+0+0 (secure 0 0 0) 0x7fe9d008e050 con 0x564c20d3ec50 2024-05-27T07:33:50.266 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:50.264+0000 7fe9ecf20b80 10 client.16917 fetch_fsmap learned FSMap version 103 2024-05-27T07:33:50.266 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:50.264+0000 7fe9ecf20b80 1 -- 172.21.15.52:0/2496953060 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({config=0+,fsmap=103,monmap=4+}) -- 0x564c20d4fea0 con 0x564c20d3ec50 2024-05-27T07:33:50.266 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:50.264+0000 7fe9b77fe700 1 -- 172.21.15.52:0/2496953060 <== mon.0 v2:172.21.15.52:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fe9d00ab3a0 con 0x564c20d3ec50 2024-05-27T07:33:50.266 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:50.265+0000 7fe9b77fe700 1 -- 172.21.15.52:0/2496953060 <== mon.0 v2:172.21.15.52:3300/0 5 ==== fsmap(e 103) ==== 4215+0+0 (secure 0 0 0) 0x7fe9d0011070 con 0x564c20d3ec50 2024-05-27T07:33:50.266 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:50.265+0000 7fe9ecf20b80 10 client.16917 fetch_fsmap finished waiting for FSMap version 103 2024-05-27T07:33:50.266 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:50.265+0000 7fe9ecf20b80 10 client.16917 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-05-27T07:33:50.266 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:50.265+0000 7fe9ecf20b80 20 client.16917 populate_metadata read hostname 'smithi052' 2024-05-27T07:33:50.267 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:50.265+0000 7fe9ecf20b80 1 --2- 172.21.15.52:0/2496953060 >> [v2:172.21.15.105:6838/500348151,v1:172.21.15.105:6839/500348151] conn(0x564c20d524e0 0x564c20d54930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:33:50.267 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:50.265+0000 7fe9ecf20b80 4 client.16917 mds_command: new command op to 24527 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-27T07:33:50.267 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:50.265+0000 7fe9ecf20b80 1 -- 172.21.15.52:0/2496953060 --> [v2:172.21.15.105:6838/500348151,v1:172.21.15.105:6839/500348151] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x564c20d049f0 con 0x564c20d524e0 2024-05-27T07:33:50.267 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:50.265+0000 7fe9d7fff700 1 --2- 172.21.15.52:0/2496953060 >> [v2:172.21.15.105:6838/500348151,v1:172.21.15.105:6839/500348151] conn(0x564c20d524e0 0x564c20d54930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:33:50.267 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:50.265+0000 7fe9b77fe700 10 client.16917 ms_handle_connect on v2:172.21.15.105:6838/500348151 2024-05-27T07:33:50.267 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:50.265+0000 7fe9d7fff700 1 --2- 172.21.15.52:0/2496953060 >> [v2:172.21.15.105:6838/500348151,v1:172.21.15.105:6839/500348151] conn(0x564c20d524e0 0x564c20d54930 crc :-1 s=READY pgs=143 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=5b187f4dde3b1a96 server_cookie=b4dd0270e0254e60 in_seq=0 out_seq=0 2024-05-27T07:33:50.267 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:50.266+0000 7fe9b77fe700 1 -- 172.21.15.52:0/2496953060 <== mds.0 v2:172.21.15.105:6838/500348151 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24041 (crc 0 0 0) 0x564c20d049f0 con 0x564c20d524e0 2024-05-27T07:33:50.267 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:50.266+0000 7fe9b77fe700 10 client.16917 handle_command_reply: tid=0 2024-05-27T07:33:50.267 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:50.266+0000 7fe9ecf20b80 1 client.16917 shutdown 2024-05-27T07:33:50.267 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:50.267+0000 7fe9ecf20b80 1 -- 172.21.15.52:0/2496953060 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x564c20d3ec50 msgr2=0x564c20d42e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:33:50.267 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:50.267+0000 7fe9ecf20b80 1 --2- 172.21.15.52:0/2496953060 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x564c20d3ec50 0x564c20d42e10 secure :-1 s=READY pgs=703 cs=0 l=1 rev1=1 crypto rx=0x7fe9d00aa9a0 tx=0x7fe9d0009e10 comp rx=0 tx=0).stop 2024-05-27T07:33:50.268 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:50.267+0000 7fe9ecf20b80 1 -- 172.21.15.52:0/2496953060 shutdown_connections 2024-05-27T07:33:50.268 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:50.267+0000 7fe9ecf20b80 1 --2- 172.21.15.52:0/2496953060 >> [v2:172.21.15.105:6838/500348151,v1:172.21.15.105:6839/500348151] conn(0x564c20d524e0 0x564c20d54930 crc :-1 s=READY pgs=143 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:33:50.268 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:50.267+0000 7fe9b77fe700 0 client.16917 ms_handle_reset on v2:172.21.15.105:6838/500348151 2024-05-27T07:33:50.268 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:50.267+0000 7fe9ecf20b80 1 --2- 172.21.15.52:0/2496953060 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x564c20d3ec50 0x564c20d42e10 unknown :-1 s=CLOSED pgs=703 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:33:50.268 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:50.267+0000 7fe9ecf20b80 1 --2- 172.21.15.52:0/2496953060 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x564c20bbae40 0x564c20d42390 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:33:50.268 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:50.267+0000 7fe9ecf20b80 1 --2- 172.21.15.52:0/2496953060 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x564c20d38f80 0x564c20d428d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:33:50.268 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:50.267+0000 7fe9ecf20b80 1 -- 172.21.15.52:0/2496953060 >> 172.21.15.52:0/2496953060 conn(0x564c20c15af0 msgr2=0x564c20d37a30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:33:50.268 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:50.267+0000 7fe9ecf20b80 1 -- 172.21.15.52:0/2496953060 shutdown_connections 2024-05-27T07:33:50.268 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:50.267+0000 7fe9ecf20b80 1 -- 172.21.15.52:0/2496953060 wait complete. 2024-05-27T07:33:50.268 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:50.267+0000 7fe9ecf20b80 20 client.16917 trim_cache size 0 max 16384 2024-05-27T07:33:50.282 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:50.282+0000 7fe9ecf20b80 1 Processor -- start 2024-05-27T07:33:50.282 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:50.282+0000 7fe9ecf20b80 1 -- start start 2024-05-27T07:33:50.282 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:50.282+0000 7fe9ecf20b80 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x564c20bbae40 0x564c20d44830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:33:50.282 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:50.282+0000 7fe9ecf20b80 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x564c20d38f80 0x564c20d4e4b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:33:50.282 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:50.282+0000 7fe9d7fff700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x564c20bbae40 0x564c20d44830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:33:50.282 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:50.282+0000 7fe9d7fff700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x564c20bbae40 0x564c20d44830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.52:56874/0 (socket says 172.21.15.52:56874) 2024-05-27T07:33:50.282 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:50.282+0000 7fe9ecf20b80 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x564c20d3ec50 0x564c20d42900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:33:50.282 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:50.283+0000 7fe9d7fff700 1 -- 172.21.15.52:0/4071550599 learned_addr learned my addr 172.21.15.52:0/4071550599 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-05-27T07:33:50.283 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:50.283+0000 7fe9ecf20b80 1 -- 172.21.15.52:0/4071550599 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x7fe9d81bf9b0 con 0x564c20bbae40 2024-05-27T07:33:50.283 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:50.283+0000 7fe9ecf20b80 1 -- 172.21.15.52:0/4071550599 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7fe9d81bdd50 con 0x564c20d38f80 2024-05-27T07:33:50.283 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:50.283+0000 7fe9ecf20b80 1 -- 172.21.15.52:0/4071550599 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7fe9d81bf2b0 con 0x564c20d3ec50 2024-05-27T07:33:50.283 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:50.283+0000 7fe9cf7fe700 1 --2- 172.21.15.52:0/4071550599 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x564c20d38f80 0x564c20d4e4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:33:50.283 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:50.283+0000 7fe9dcaa8700 1 --2- 172.21.15.52:0/4071550599 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x564c20d3ec50 0x564c20d42900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:33:50.283 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:50.283+0000 7fe9d7fff700 1 -- 172.21.15.52:0/4071550599 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x564c20d3ec50 msgr2=0x564c20d42900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:33:50.283 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:50.283+0000 7fe9d7fff700 1 --2- 172.21.15.52:0/4071550599 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x564c20d3ec50 0x564c20d42900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:33:50.283 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:50.283+0000 7fe9d7fff700 1 -- 172.21.15.52:0/4071550599 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x564c20d38f80 msgr2=0x564c20d4e4b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:33:50.283 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:50.283+0000 7fe9d7fff700 1 --2- 172.21.15.52:0/4071550599 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x564c20d38f80 0x564c20d4e4b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:33:50.283 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:50.283+0000 7fe9d7fff700 1 -- 172.21.15.52:0/4071550599 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x564c20d40900 con 0x564c20bbae40 2024-05-27T07:33:50.283 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:50.283+0000 7fe9dcaa8700 1 --2- 172.21.15.52:0/4071550599 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x564c20d3ec50 0x564c20d42900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-27T07:33:50.284 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:50.283+0000 7fe9d7fff700 1 --2- 172.21.15.52:0/4071550599 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x564c20bbae40 0x564c20d44830 secure :-1 s=READY pgs=704 cs=0 l=1 rev1=1 crypto rx=0x7fe9c800e4b0 tx=0x7fe9c8015000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=575fd330cac67f63 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:33:50.284 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:50.283+0000 7fe9cdffb700 1 -- 172.21.15.52:0/4071550599 <== mon.0 v2:172.21.15.52:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe9c802f070 con 0x564c20bbae40 2024-05-27T07:33:50.284 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:50.283+0000 7fe9cdffb700 1 -- 172.21.15.52:0/4071550599 <== mon.0 v2:172.21.15.52:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fe9c8016a50 con 0x564c20bbae40 2024-05-27T07:33:50.284 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:50.283+0000 7fe9cdffb700 1 -- 172.21.15.52:0/4071550599 <== mon.0 v2:172.21.15.52:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe9c801b040 con 0x564c20bbae40 2024-05-27T07:33:50.284 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:50.284+0000 7fe9ecf20b80 1 -- 172.21.15.52:0/4071550599 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x564c20bbae40 msgr2=0x564c20d44830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:33:50.284 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:50.284+0000 7fe9ecf20b80 1 --2- 172.21.15.52:0/4071550599 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x564c20bbae40 0x564c20d44830 secure :-1 s=READY pgs=704 cs=0 l=1 rev1=1 crypto rx=0x7fe9c800e4b0 tx=0x7fe9c8015000 comp rx=0 tx=0).stop 2024-05-27T07:33:50.284 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:50.284+0000 7fe9ecf20b80 1 -- 172.21.15.52:0/4071550599 shutdown_connections 2024-05-27T07:33:50.284 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:50.284+0000 7fe9ecf20b80 1 --2- 172.21.15.52:0/4071550599 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x564c20d3ec50 0x564c20d42900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:33:50.284 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:50.284+0000 7fe9ecf20b80 1 --2- 172.21.15.52:0/4071550599 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x564c20bbae40 0x564c20d44830 unknown :-1 s=CLOSED pgs=704 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:33:50.284 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:50.284+0000 7fe9ecf20b80 1 --2- 172.21.15.52:0/4071550599 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x564c20d38f80 0x564c20d4e4b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:33:50.284 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:50.284+0000 7fe9ecf20b80 1 -- 172.21.15.52:0/4071550599 >> 172.21.15.52:0/4071550599 conn(0x564c20c15af0 msgr2=0x564c20d40c80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:33:50.284 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:50.284+0000 7fe9ecf20b80 1 -- 172.21.15.52:0/4071550599 shutdown_connections 2024-05-27T07:33:50.285 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:50.284+0000 7fe9ecf20b80 1 -- 172.21.15.52:0/4071550599 wait complete. 2024-05-27T07:33:50.285 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:50.285+0000 7fe9ecf20b80 1 -- start start 2024-05-27T07:33:50.285 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:50.285+0000 7fe9ecf20b80 1 Processor -- start 2024-05-27T07:33:50.285 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:50.285+0000 7fe9ecf20b80 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x564c20bbae40 0x564c20d4aae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:33:50.285 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:50.285+0000 7fe9ecf20b80 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x564c20d38f80 0x564c20d597a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:33:50.285 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:50.285+0000 7fe9ecf20b80 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x564c20d3ec50 0x564c20d59ce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:33:50.285 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:50.285+0000 7fe9ecf20b80 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x7fe9d81beb20 con 0x564c20d38f80 2024-05-27T07:33:50.285 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:50.285+0000 7fe9ecf20b80 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7fe9d81be420 con 0x564c20d3ec50 2024-05-27T07:33:50.285 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:50.285+0000 7fe9ecf20b80 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x564c20d44d70 con 0x564c20bbae40 2024-05-27T07:33:50.285 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:50.285+0000 7fe9ecf20b80 10 client.0 _ll_register_callbacks cb 0x564c20bf2920 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T07:33:50.286 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:50.285+0000 7fe9cf7fe700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x564c20d38f80 0x564c20d597a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:33:50.286 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:50.285+0000 7fe9cf7fe700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x564c20d38f80 0x564c20d597a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.52:56876/0 (socket says 172.21.15.52:56876) 2024-05-27T07:33:50.286 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:50.285+0000 7fe9cf7fe700 1 -- 172.21.15.52:0/4035373004 learned_addr learned my addr 172.21.15.52:0/4035373004 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-05-27T07:33:50.286 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:50.285+0000 7fe9dcaa8700 1 --2- 172.21.15.52:0/4035373004 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x564c20d3ec50 0x564c20d59ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:33:50.286 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:50.286+0000 7fe9d7fff700 1 --2- 172.21.15.52:0/4035373004 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x564c20bbae40 0x564c20d4aae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:33:50.286 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:50.286+0000 7fe9dcaa8700 1 -- 172.21.15.52:0/4035373004 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x564c20bbae40 msgr2=0x564c20d4aae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:33:50.286 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:50.286+0000 7fe9dcaa8700 1 --2- 172.21.15.52:0/4035373004 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x564c20bbae40 0x564c20d4aae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:33:50.286 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:50.286+0000 7fe9dcaa8700 1 -- 172.21.15.52:0/4035373004 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x564c20d38f80 msgr2=0x564c20d597a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:33:50.286 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:50.286+0000 7fe9dcaa8700 1 --2- 172.21.15.52:0/4035373004 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x564c20d38f80 0x564c20d597a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:33:50.286 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:50.286+0000 7fe9dcaa8700 1 -- 172.21.15.52:0/4035373004 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({osdmap=0}) -- 0x564c20d5a220 con 0x564c20d3ec50 2024-05-27T07:33:50.287 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:50.286+0000 7fe9d7fff700 1 --2- 172.21.15.52:0/4035373004 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x564c20bbae40 0x564c20d4aae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-27T07:33:50.287 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:50.286+0000 7fe9dcaa8700 1 --2- 172.21.15.52:0/4035373004 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x564c20d3ec50 0x564c20d59ce0 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7fe9d00e1800 tx=0x7fe9d00099e0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=42a32440beb6c718 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:33:50.287 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:50.286+0000 7fe9b77fe700 10 client.0 ms_handle_connect on v2:172.21.15.105:3300/0 2024-05-27T07:33:50.287 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:50.286+0000 7fe9b77fe700 1 -- 172.21.15.52:0/4035373004 <== mon.2 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe9d00c9030 con 0x564c20d3ec50 2024-05-27T07:33:50.287 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:50.286+0000 7fe9ecf20b80 1 -- 172.21.15.52:0/4035373004 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fe9c8016a50 con 0x564c20d3ec50 2024-05-27T07:33:50.287 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:50.286+0000 7fe9b77fe700 1 -- 172.21.15.52:0/4035373004 <== mon.2 v2:172.21.15.105:3300/0 2 ==== osd_map(119..119 src has 1..119) ==== 9451+0+0 (secure 0 0 0) 0x7fe9d00a9f00 con 0x564c20d3ec50 2024-05-27T07:33:50.287 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:50.286+0000 7fe9b77fe700 1 -- 172.21.15.52:0/4035373004 <== mon.2 v2:172.21.15.105:3300/0 3 ==== mon_get_version_reply(handle=1 version=103) ==== 24+0+0 (secure 0 0 0) 0x7fe9d00a6b60 con 0x564c20d3ec50 2024-05-27T07:33:50.287 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:50.286+0000 7fe9ecf20b80 10 client.26597 fetch_fsmap learned FSMap version 103 2024-05-27T07:33:50.287 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:50.286+0000 7fe9ecf20b80 1 -- 172.21.15.52:0/4035373004 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,fsmap=103,monmap=4+}) -- 0x564c20d5a220 con 0x564c20d3ec50 2024-05-27T07:33:50.287 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:50.287+0000 7fe9b77fe700 1 -- 172.21.15.52:0/4035373004 <== mon.2 v2:172.21.15.105:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fe9c8016a50 con 0x564c20d3ec50 2024-05-27T07:33:50.287 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:50.287+0000 7fe9b77fe700 1 -- 172.21.15.52:0/4035373004 <== mon.2 v2:172.21.15.105:3300/0 5 ==== fsmap(e 103) ==== 4215+0+0 (secure 0 0 0) 0x7fe9d0011070 con 0x564c20d3ec50 2024-05-27T07:33:50.288 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:50.287+0000 7fe9ecf20b80 10 client.26597 fetch_fsmap finished waiting for FSMap version 103 2024-05-27T07:33:50.288 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:50.287+0000 7fe9ecf20b80 10 client.26597 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-05-27T07:33:50.288 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:50.287+0000 7fe9ecf20b80 20 client.26597 populate_metadata read hostname 'smithi052' 2024-05-27T07:33:50.288 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:50.287+0000 7fe9ecf20b80 1 --2- 172.21.15.52:0/4035373004 >> [v2:172.21.15.105:6838/500348151,v1:172.21.15.105:6839/500348151] conn(0x564c20d524e0 0x564c20dbef20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:33:50.288 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:50.287+0000 7fe9ecf20b80 4 client.26597 mds_command: new command op to 24527 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-05-27T07:33:50.288 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:50.287+0000 7fe9ecf20b80 1 -- 172.21.15.52:0/4035373004 --> [v2:172.21.15.105:6838/500348151,v1:172.21.15.105:6839/500348151] -- command(tid 0: {"prefix": "damage ls"}) -- 0x564c20c69be0 con 0x564c20d524e0 2024-05-27T07:33:50.288 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:50.287+0000 7fe9d7fff700 1 --2- 172.21.15.52:0/4035373004 >> [v2:172.21.15.105:6838/500348151,v1:172.21.15.105:6839/500348151] conn(0x564c20d524e0 0x564c20dbef20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:33:50.288 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:50.287+0000 7fe9b77fe700 10 client.26597 ms_handle_connect on v2:172.21.15.105:6838/500348151 2024-05-27T07:33:50.288 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:50.288+0000 7fe9d7fff700 1 --2- 172.21.15.52:0/4035373004 >> [v2:172.21.15.105:6838/500348151,v1:172.21.15.105:6839/500348151] conn(0x564c20d524e0 0x564c20dbef20 crc :-1 s=READY pgs=144 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=85f28d07d649ef9b server_cookie=3749fb3956f1c722 in_seq=0 out_seq=0 2024-05-27T07:33:50.289 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:50.288+0000 7fe9b77fe700 1 -- 172.21.15.52:0/4035373004 <== mds.0 v2:172.21.15.105:6838/500348151 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x564c20c69be0 con 0x564c20d524e0 2024-05-27T07:33:50.290 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:50.288+0000 7fe9b77fe700 10 client.26597 handle_command_reply: tid=0 2024-05-27T07:33:50.290 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:50.288+0000 7fe9ecf20b80 1 client.26597 shutdown 2024-05-27T07:33:50.290 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:50.288+0000 7fe9ecf20b80 1 -- 172.21.15.52:0/4035373004 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x564c20d3ec50 msgr2=0x564c20d59ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:33:50.290 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:50.288+0000 7fe9ecf20b80 1 --2- 172.21.15.52:0/4035373004 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x564c20d3ec50 0x564c20d59ce0 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7fe9d00e1800 tx=0x7fe9d00099e0 comp rx=0 tx=0).stop 2024-05-27T07:33:50.290 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:50.288+0000 7fe9ecf20b80 1 -- 172.21.15.52:0/4035373004 shutdown_connections 2024-05-27T07:33:50.290 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:50.288+0000 7fe9ecf20b80 1 --2- 172.21.15.52:0/4035373004 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x564c20d3ec50 0x564c20d59ce0 unknown :-1 s=CLOSED pgs=225 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:33:50.290 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:50.288+0000 7fe9ecf20b80 1 --2- 172.21.15.52:0/4035373004 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x564c20bbae40 0x564c20d4aae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:33:50.290 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:50.288+0000 7fe9ecf20b80 1 --2- 172.21.15.52:0/4035373004 >> [v2:172.21.15.105:6838/500348151,v1:172.21.15.105:6839/500348151] conn(0x564c20d524e0 0x564c20dbef20 crc :-1 s=READY pgs=144 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:33:50.290 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:50.288+0000 7fe9b77fe700 0 client.26597 ms_handle_reset on v2:172.21.15.105:6838/500348151 2024-05-27T07:33:50.290 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:50.288+0000 7fe9ecf20b80 1 --2- 172.21.15.52:0/4035373004 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x564c20d38f80 0x564c20d597a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:33:50.291 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:50.288+0000 7fe9ecf20b80 1 -- 172.21.15.52:0/4035373004 >> 172.21.15.52:0/4035373004 conn(0x564c20c15af0 msgr2=0x564c20d3c010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:33:50.291 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:50.289+0000 7fe9ecf20b80 1 -- 172.21.15.52:0/4035373004 shutdown_connections 2024-05-27T07:33:50.291 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:50.289+0000 7fe9ecf20b80 1 -- 172.21.15.52:0/4035373004 wait complete. 2024-05-27T07:33:50.291 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:50.289+0000 7fe9ecf20b80 20 client.26597 trim_cache size 0 max 16384 2024-05-27T07:33:50.291 INFO:teuthology.orchestra.run.smithi052.stdout:[] 2024-05-27T07:33:50.292 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:50.289+0000 7fe9de50b700 1 -- 172.21.15.52:0/1456130657 >> [v2:172.21.15.52:6800/1822318384,v1:172.21.15.52:6801/1822318384] conn(0x7fe9c40b2a70 msgr2=0x7fe9c40b4f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:33:50.292 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:50.289+0000 7fe9de50b700 1 --2- 172.21.15.52:0/1456130657 >> [v2:172.21.15.52:6800/1822318384,v1:172.21.15.52:6801/1822318384] conn(0x7fe9c40b2a70 0x7fe9c40b4f80 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7fe9c8009ad0 tx=0x7fe9c8009290 comp rx=0 tx=0).stop 2024-05-27T07:33:50.292 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:50.289+0000 7fe9de50b700 1 -- 172.21.15.52:0/1456130657 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fe9d811d920 msgr2=0x7fe9d81ae560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:33:50.292 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:50.289+0000 7fe9de50b700 1 --2- 172.21.15.52:0/1456130657 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fe9d811d920 0x7fe9d81ae560 secure :-1 s=READY pgs=702 cs=0 l=1 rev1=1 crypto rx=0x7fe9d000b070 tx=0x7fe9d0004c80 comp rx=0 tx=0).stop 2024-05-27T07:33:50.292 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:50.289+0000 7fe9de50b700 1 -- 172.21.15.52:0/1456130657 shutdown_connections 2024-05-27T07:33:50.292 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:50.289+0000 7fe9de50b700 1 --2- 172.21.15.52:0/1456130657 >> [v2:172.21.15.52:6800/1822318384,v1:172.21.15.52:6801/1822318384] conn(0x7fe9c40b2a70 0x7fe9c40b4f80 unknown :-1 s=CLOSED pgs=169 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:33:50.292 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:50.289+0000 7fe9de50b700 1 --2- 172.21.15.52:0/1456130657 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fe9d811d920 0x7fe9d81ae560 unknown :-1 s=CLOSED pgs=702 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:33:50.293 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:50.289+0000 7fe9de50b700 1 --2- 172.21.15.52:0/1456130657 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fe9d810a920 0x7fe9d81a9d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:33:50.293 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:50.289+0000 7fe9de50b700 1 --2- 172.21.15.52:0/1456130657 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fe9d810b270 0x7fe9d81aa2d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:33:50.293 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:50.289+0000 7fe9de50b700 1 -- 172.21.15.52:0/1456130657 >> 172.21.15.52:0/1456130657 conn(0x7fe9d806f4c0 msgr2=0x7fe9d811b3d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:33:50.293 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:50.290+0000 7fe9de50b700 1 -- 172.21.15.52:0/1456130657 shutdown_connections 2024-05-27T07:33:50.293 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:50.290+0000 7fe9de50b700 1 -- 172.21.15.52:0/1456130657 wait complete. 2024-05-27T07:33:50.483 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:33:50 smithi052 ceph-mon[31955]: pgmap v267: 129 pgs: 129 active+clean; 654 MiB data, 4.5 GiB used, 1.0 TiB / 1.0 TiB avail; 37 KiB/s rd, 1.3 MiB/s wr, 710 op/s 2024-05-27T07:33:50.606 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:33:50 smithi173 ceph-mon[37000]: pgmap v267: 129 pgs: 129 active+clean; 654 MiB data, 4.5 GiB used, 1.0 TiB / 1.0 TiB avail; 37 KiB/s rd, 1.3 MiB/s wr, 710 op/s 2024-05-27T07:33:50.694 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:33:50 smithi105 ceph-mon[35977]: pgmap v267: 129 pgs: 129 active+clean; 654 MiB data, 4.5 GiB used, 1.0 TiB / 1.0 TiB avail; 37 KiB/s rd, 1.3 MiB/s wr, 710 op/s 2024-05-27T07:33:52.060 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-05-27T07:33:52.060 DEBUG:teuthology.orchestra.run.smithi052:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:38f9d900c83c63799fbdbe61acc9a11b0d3554a6 shell --fsid 73df2636-1bf8-11ef-bc9b-c7b262605968 -- ceph tell mds.1:0 scrub start / force,recursive 2024-05-27T07:33:52.334 INFO:teuthology.orchestra.run.smithi052.stderr:Inferring config /var/lib/ceph/73df2636-1bf8-11ef-bc9b-c7b262605968/mon.a/config 2024-05-27T07:33:52.456 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:33:52 smithi052 ceph-mon[31955]: pgmap v268: 129 pgs: 129 active+clean; 662 MiB data, 4.5 GiB used, 1.0 TiB / 1.0 TiB avail; 37 KiB/s rd, 1.5 MiB/s wr, 677 op/s 2024-05-27T07:33:52.566 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:33:52 smithi105 ceph-mon[35977]: pgmap v268: 129 pgs: 129 active+clean; 662 MiB data, 4.5 GiB used, 1.0 TiB / 1.0 TiB avail; 37 KiB/s rd, 1.5 MiB/s wr, 677 op/s 2024-05-27T07:33:52.606 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:33:52 smithi173 ceph-mon[37000]: pgmap v268: 129 pgs: 129 active+clean; 662 MiB data, 4.5 GiB used, 1.0 TiB / 1.0 TiB avail; 37 KiB/s rd, 1.5 MiB/s wr, 677 op/s 2024-05-27T07:33:53.483 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:33:53 smithi052 ceph-mon[31955]: fsmap cephfs:5 5 up:active 2024-05-27T07:33:53.606 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:33:53 smithi173 ceph-mon[37000]: fsmap cephfs:5 5 up:active 2024-05-27T07:33:53.694 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:33:53 smithi105 ceph-mon[35977]: fsmap cephfs:5 5 up:active 2024-05-27T07:33:54.483 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:33:54 smithi052 ceph-mon[31955]: pgmap v269: 129 pgs: 129 active+clean; 662 MiB data, 4.5 GiB used, 1.0 TiB / 1.0 TiB avail; 37 KiB/s rd, 1.5 MiB/s wr, 631 op/s 2024-05-27T07:33:54.606 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:33:54 smithi173 ceph-mon[37000]: pgmap v269: 129 pgs: 129 active+clean; 662 MiB data, 4.5 GiB used, 1.0 TiB / 1.0 TiB avail; 37 KiB/s rd, 1.5 MiB/s wr, 631 op/s 2024-05-27T07:33:54.694 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:33:54 smithi105 ceph-mon[35977]: pgmap v269: 129 pgs: 129 active+clean; 662 MiB data, 4.5 GiB used, 1.0 TiB / 1.0 TiB avail; 37 KiB/s rd, 1.5 MiB/s wr, 631 op/s 2024-05-27T07:33:55.508 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:55.259+0000 7fc4f9875700 1 -- 172.21.15.52:0/4235686098 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fc4f4105470 msgr2=0x7fc4f4105880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:33:55.508 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:55.259+0000 7fc4f9875700 1 --2- 172.21.15.52:0/4235686098 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fc4f4105470 0x7fc4f4105880 secure :-1 s=READY pgs=705 cs=0 l=1 rev1=1 crypto rx=0x7fc4e4009990 tx=0x7fc4e4009ca0 comp rx=0 tx=0).stop 2024-05-27T07:33:55.508 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:55.259+0000 7fc4f9875700 1 -- 172.21.15.52:0/4235686098 shutdown_connections 2024-05-27T07:33:55.508 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:55.259+0000 7fc4f9875700 1 --2- 172.21.15.52:0/4235686098 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fc4f410e8e0 0x7fc4f4110dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:33:55.508 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:55.260+0000 7fc4f9875700 1 --2- 172.21.15.52:0/4235686098 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fc4f4105470 0x7fc4f4105880 unknown :-1 s=CLOSED pgs=705 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:33:55.508 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:55.260+0000 7fc4f9875700 1 --2- 172.21.15.52:0/4235686098 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fc4f4105dc0 0x7fc4f410e310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:33:55.508 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:55.260+0000 7fc4f9875700 1 -- 172.21.15.52:0/4235686098 >> 172.21.15.52:0/4235686098 conn(0x7fc4f40fef20 msgr2=0x7fc4f4101340 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:33:55.508 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:55.260+0000 7fc4f9875700 1 -- 172.21.15.52:0/4235686098 shutdown_connections 2024-05-27T07:33:55.508 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:55.260+0000 7fc4f9875700 1 -- 172.21.15.52:0/4235686098 wait complete. 2024-05-27T07:33:55.508 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:55.260+0000 7fc4f9875700 1 Processor -- start 2024-05-27T07:33:55.509 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:55.260+0000 7fc4f9875700 1 -- start start 2024-05-27T07:33:55.509 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:55.261+0000 7fc4f9875700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fc4f4105470 0x7fc4f419ae40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:33:55.509 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:55.261+0000 7fc4f9875700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fc4f4105dc0 0x7fc4f419b3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:33:55.509 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:55.261+0000 7fc4f9875700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fc4f410e8e0 0x7fc4f419f670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:33:55.509 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:55.261+0000 7fc4f9875700 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x7fc4f4114430 con 0x7fc4f410e8e0 2024-05-27T07:33:55.509 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:55.261+0000 7fc4f9875700 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7fc4f41142b0 con 0x7fc4f4105470 2024-05-27T07:33:55.509 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:55.261+0000 7fc4f9875700 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7fc4f41145b0 con 0x7fc4f4105dc0 2024-05-27T07:33:55.509 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:55.261+0000 7fc4f2ffd700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fc4f4105470 0x7fc4f419ae40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:33:55.509 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:55.261+0000 7fc4f37fe700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fc4f410e8e0 0x7fc4f419f670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:33:55.509 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:55.261+0000 7fc4f2ffd700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fc4f4105470 0x7fc4f419ae40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.52:37150/0 (socket says 172.21.15.52:37150) 2024-05-27T07:33:55.509 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:55.261+0000 7fc4f2ffd700 1 -- 172.21.15.52:0/3963227716 learned_addr learned my addr 172.21.15.52:0/3963227716 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-05-27T07:33:55.509 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:55.261+0000 7fc4f27fc700 1 --2- 172.21.15.52:0/3963227716 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fc4f4105dc0 0x7fc4f419b3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:33:55.510 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:55.261+0000 7fc4f37fe700 1 -- 172.21.15.52:0/3963227716 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fc4f4105dc0 msgr2=0x7fc4f419b3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:33:55.510 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:55.261+0000 7fc4f37fe700 1 --2- 172.21.15.52:0/3963227716 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fc4f4105dc0 0x7fc4f419b3e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:33:55.510 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:55.261+0000 7fc4f37fe700 1 -- 172.21.15.52:0/3963227716 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fc4f4105470 msgr2=0x7fc4f419ae40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:33:55.510 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:55.261+0000 7fc4f37fe700 1 --2- 172.21.15.52:0/3963227716 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fc4f4105470 0x7fc4f419ae40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:33:55.510 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:55.261+0000 7fc4f37fe700 1 -- 172.21.15.52:0/3963227716 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc4f419fd10 con 0x7fc4f410e8e0 2024-05-27T07:33:55.510 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:55.261+0000 7fc4f2ffd700 1 --2- 172.21.15.52:0/3963227716 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fc4f4105470 0x7fc4f419ae40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-27T07:33:55.510 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:55.261+0000 7fc4f37fe700 1 --2- 172.21.15.52:0/3963227716 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fc4f410e8e0 0x7fc4f419f670 secure :-1 s=READY pgs=706 cs=0 l=1 rev1=1 crypto rx=0x7fc4ec00efc0 tx=0x7fc4ec00c3a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:33:55.510 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:55.262+0000 7fc4f8873700 1 -- 172.21.15.52:0/3963227716 <== mon.0 v2:172.21.15.52:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc4ec0092e0 con 0x7fc4f410e8e0 2024-05-27T07:33:55.510 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:55.262+0000 7fc4f9875700 1 -- 172.21.15.52:0/3963227716 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc4f419ffa0 con 0x7fc4f410e8e0 2024-05-27T07:33:55.510 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:55.262+0000 7fc4f8873700 1 -- 172.21.15.52:0/3963227716 <== mon.0 v2:172.21.15.52:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fc4ec004bb0 con 0x7fc4f410e8e0 2024-05-27T07:33:55.510 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:55.262+0000 7fc4f8873700 1 -- 172.21.15.52:0/3963227716 <== mon.0 v2:172.21.15.52:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc4ec004020 con 0x7fc4f410e8e0 2024-05-27T07:33:55.510 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:55.262+0000 7fc4f9875700 1 -- 172.21.15.52:0/3963227716 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc4f41a7920 con 0x7fc4f410e8e0 2024-05-27T07:33:55.510 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:55.264+0000 7fc4f8873700 1 -- 172.21.15.52:0/3963227716 <== mon.0 v2:172.21.15.52:3300/0 4 ==== mgrmap(e 26) ==== 149783+0+0 (secure 0 0 0) 0x7fc4ec01b030 con 0x7fc4f410e8e0 2024-05-27T07:33:55.511 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:55.265+0000 7fc4f8873700 1 --2- 172.21.15.52:0/3963227716 >> [v2:172.21.15.52:6800/1822318384,v1:172.21.15.52:6801/1822318384] conn(0x7fc4dc0b29f0 0x7fc4dc0b4f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:33:55.511 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:55.265+0000 7fc4f2ffd700 1 --2- 172.21.15.52:0/3963227716 >> [v2:172.21.15.52:6800/1822318384,v1:172.21.15.52:6801/1822318384] conn(0x7fc4dc0b29f0 0x7fc4dc0b4f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:33:55.511 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:55.265+0000 7fc4f8873700 1 -- 172.21.15.52:0/3963227716 <== mon.0 v2:172.21.15.52:3300/0 5 ==== osd_map(119..119 src has 1..119) ==== 9451+0+0 (secure 0 0 0) 0x7fc4ec0e24a0 con 0x7fc4f410e8e0 2024-05-27T07:33:55.511 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:55.265+0000 7fc50828ab80 1 Processor -- start 2024-05-27T07:33:55.511 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:55.265+0000 7fc50828ab80 1 -- start start 2024-05-27T07:33:55.511 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:55.266+0000 7fc4f2ffd700 1 --2- 172.21.15.52:0/3963227716 >> [v2:172.21.15.52:6800/1822318384,v1:172.21.15.52:6801/1822318384] conn(0x7fc4dc0b29f0 0x7fc4dc0b4f00 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7fc4e4006010 tx=0x7fc4e4009590 comp rx=0 tx=0).ready entity=mgr.15333 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:33:55.511 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:55.266+0000 7fc50828ab80 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55aedea09ec0 0x55aedeb91ab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:33:55.511 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:55.266+0000 7fc50828ab80 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55aedeb91ff0 0x55aedeb8bed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:33:55.511 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:55.266+0000 7fc50828ab80 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55aedeb8c410 0x55aedeb8c820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:33:55.511 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:55.266+0000 7fc50828ab80 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x55aedeab6d10 con 0x55aedeb91ff0 2024-05-27T07:33:55.511 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:55.266+0000 7fc50828ab80 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x55aedeaa68b0 con 0x55aedea09ec0 2024-05-27T07:33:55.511 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:55.266+0000 7fc50828ab80 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x55aedeaaa8f0 con 0x55aedeb8c410 2024-05-27T07:33:55.511 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:55.266+0000 7fc4f27fc700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55aedeb91ff0 0x55aedeb8bed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:33:55.512 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:55.266+0000 7fc4f2ffd700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55aedea09ec0 0x55aedeb91ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:33:55.512 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:55.266+0000 7fc4f27fc700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55aedeb91ff0 0x55aedeb8bed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.52:40476/0 (socket says 172.21.15.52:40476) 2024-05-27T07:33:55.512 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:55.266+0000 7fc4f27fc700 1 -- 172.21.15.52:0/821906688 learned_addr learned my addr 172.21.15.52:0/821906688 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-05-27T07:33:55.512 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:55.266+0000 7fc4f37fe700 1 --2- 172.21.15.52:0/821906688 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55aedeb8c410 0x55aedeb8c820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:33:55.512 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:55.266+0000 7fc4f27fc700 1 -- 172.21.15.52:0/821906688 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55aedeb8c410 msgr2=0x55aedeb8c820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:33:55.512 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:55.266+0000 7fc4f27fc700 1 --2- 172.21.15.52:0/821906688 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55aedeb8c410 0x55aedeb8c820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:33:55.512 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:55.266+0000 7fc4f27fc700 1 -- 172.21.15.52:0/821906688 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55aedea09ec0 msgr2=0x55aedeb91ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:33:55.512 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:55.266+0000 7fc4f27fc700 1 --2- 172.21.15.52:0/821906688 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55aedea09ec0 0x55aedeb91ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:33:55.512 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:55.266+0000 7fc4f27fc700 1 -- 172.21.15.52:0/821906688 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55aede9f6950 con 0x55aedeb91ff0 2024-05-27T07:33:55.512 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:55.266+0000 7fc4f2ffd700 1 --2- 172.21.15.52:0/821906688 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55aedea09ec0 0x55aedeb91ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-27T07:33:55.512 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:55.266+0000 7fc4f27fc700 1 --2- 172.21.15.52:0/821906688 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55aedeb91ff0 0x55aedeb8bed0 secure :-1 s=READY pgs=707 cs=0 l=1 rev1=1 crypto rx=0x7fc4e800bd30 tx=0x7fc4e80106b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cac1b780bce92d1c server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:33:55.512 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:55.267+0000 7fc4d9ffb700 1 -- 172.21.15.52:0/821906688 <== mon.0 v2:172.21.15.52:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc4e8010db0 con 0x55aedeb91ff0 2024-05-27T07:33:55.512 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:55.267+0000 7fc4d9ffb700 1 -- 172.21.15.52:0/821906688 <== mon.0 v2:172.21.15.52:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fc4e80044e0 con 0x55aedeb91ff0 2024-05-27T07:33:55.513 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:55.267+0000 7fc4d9ffb700 1 -- 172.21.15.52:0/821906688 <== mon.0 v2:172.21.15.52:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc4e8011440 con 0x55aedeb91ff0 2024-05-27T07:33:55.513 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:55.267+0000 7fc50828ab80 1 -- 172.21.15.52:0/821906688 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55aedeb91ff0 msgr2=0x55aedeb8bed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:33:55.513 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:55.267+0000 7fc50828ab80 1 --2- 172.21.15.52:0/821906688 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55aedeb91ff0 0x55aedeb8bed0 secure :-1 s=READY pgs=707 cs=0 l=1 rev1=1 crypto rx=0x7fc4e800bd30 tx=0x7fc4e80106b0 comp rx=0 tx=0).stop 2024-05-27T07:33:55.513 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:55.267+0000 7fc50828ab80 1 -- 172.21.15.52:0/821906688 shutdown_connections 2024-05-27T07:33:55.513 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:55.267+0000 7fc50828ab80 1 --2- 172.21.15.52:0/821906688 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55aedeb8c410 0x55aedeb8c820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:33:55.513 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:55.267+0000 7fc50828ab80 1 --2- 172.21.15.52:0/821906688 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55aedea09ec0 0x55aedeb91ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:33:55.513 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:55.267+0000 7fc50828ab80 1 --2- 172.21.15.52:0/821906688 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55aedeb91ff0 0x55aedeb8bed0 unknown :-1 s=CLOSED pgs=707 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:33:55.513 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:55.267+0000 7fc50828ab80 1 -- 172.21.15.52:0/821906688 >> 172.21.15.52:0/821906688 conn(0x55aedea64b70 msgr2=0x55aedeb826c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:33:55.513 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:55.268+0000 7fc50828ab80 1 -- 172.21.15.52:0/821906688 shutdown_connections 2024-05-27T07:33:55.513 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:55.268+0000 7fc50828ab80 1 -- 172.21.15.52:0/821906688 wait complete. 2024-05-27T07:33:55.513 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:55.268+0000 7fc50828ab80 1 -- start start 2024-05-27T07:33:55.513 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:55.268+0000 7fc50828ab80 1 Processor -- start 2024-05-27T07:33:55.513 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:55.269+0000 7fc50828ab80 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55aedea09ec0 0x55aedeb95940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:33:55.514 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:55.269+0000 7fc50828ab80 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55aedeb8c410 0x55aedeb95e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:33:55.514 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:55.269+0000 7fc50828ab80 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55aedeb91ff0 0x55aedeb963c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:33:55.514 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:55.269+0000 7fc50828ab80 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x55aede9ad020 con 0x55aedeb8c410 2024-05-27T07:33:55.514 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:55.269+0000 7fc50828ab80 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x55aedea48840 con 0x55aedea09ec0 2024-05-27T07:33:55.514 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:55.269+0000 7fc50828ab80 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x55aedeb98b90 con 0x55aedeb91ff0 2024-05-27T07:33:55.514 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:55.269+0000 7fc50828ab80 10 client.0 _ll_register_callbacks cb 0x55aedea419a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T07:33:55.514 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:55.269+0000 7fc4f2ffd700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55aedea09ec0 0x55aedeb95940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:33:55.514 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:55.269+0000 7fc4f2ffd700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55aedea09ec0 0x55aedeb95940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.52:37162/0 (socket says 172.21.15.52:37162) 2024-05-27T07:33:55.514 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:55.269+0000 7fc4f27fc700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55aedeb8c410 0x55aedeb95e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:33:55.514 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:55.269+0000 7fc4f2ffd700 1 -- 172.21.15.52:0/96110 learned_addr learned my addr 172.21.15.52:0/96110 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-05-27T07:33:55.514 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:55.269+0000 7fc4f37fe700 1 --2- 172.21.15.52:0/96110 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55aedeb91ff0 0x55aedeb963c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:33:55.514 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:55.269+0000 7fc4f2ffd700 1 -- 172.21.15.52:0/96110 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55aedeb91ff0 msgr2=0x55aedeb963c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:33:55.514 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:55.269+0000 7fc4f2ffd700 1 --2- 172.21.15.52:0/96110 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55aedeb91ff0 0x55aedeb963c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:33:55.515 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:55.269+0000 7fc4f2ffd700 1 -- 172.21.15.52:0/96110 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55aedeb8c410 msgr2=0x55aedeb95e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:33:55.515 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:55.269+0000 7fc4f2ffd700 1 --2- 172.21.15.52:0/96110 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55aedeb8c410 0x55aedeb95e80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:33:55.515 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:55.269+0000 7fc4f2ffd700 1 -- 172.21.15.52:0/96110 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55aedea45710 con 0x55aedea09ec0 2024-05-27T07:33:55.515 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:55.269+0000 7fc4f2ffd700 1 --2- 172.21.15.52:0/96110 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55aedea09ec0 0x55aedeb95940 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7fc4e4003ad0 tx=0x7fc4e401a050 comp rx=0 tx=0).ready entity=mon.2 client_cookie=dd4de7f8d63cec78 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:33:55.515 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:55.269+0000 7fc4d2ffd700 10 client.0 ms_handle_connect on v2:172.21.15.105:3300/0 2024-05-27T07:33:55.515 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:55.269+0000 7fc4d2ffd700 1 -- 172.21.15.52:0/96110 <== mon.2 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc4e4003ea0 con 0x55aedea09ec0 2024-05-27T07:33:55.515 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:55.270+0000 7fc4d2ffd700 1 -- 172.21.15.52:0/96110 <== mon.2 v2:172.21.15.105:3300/0 2 ==== osd_map(119..119 src has 1..119) ==== 9451+0+0 (secure 0 0 0) 0x7fc4e400fc00 con 0x55aedea09ec0 2024-05-27T07:33:55.515 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:55.270+0000 7fc50828ab80 1 -- 172.21.15.52:0/96110 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fc4e80044e0 con 0x55aedea09ec0 2024-05-27T07:33:55.515 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:55.270+0000 7fc4d2ffd700 1 -- 172.21.15.52:0/96110 <== mon.2 v2:172.21.15.105:3300/0 3 ==== mon_get_version_reply(handle=1 version=104) ==== 24+0+0 (secure 0 0 0) 0x7fc4e403d050 con 0x55aedea09ec0 2024-05-27T07:33:55.515 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:55.270+0000 7fc50828ab80 10 client.26609 fetch_fsmap learned FSMap version 104 2024-05-27T07:33:55.515 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:55.270+0000 7fc50828ab80 1 -- 172.21.15.52:0/96110 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,fsmap=104,monmap=4+}) -- 0x55aedeba36e0 con 0x55aedea09ec0 2024-05-27T07:33:55.515 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:55.270+0000 7fc4d2ffd700 1 -- 172.21.15.52:0/96110 <== mon.2 v2:172.21.15.105:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fc4e80044e0 con 0x55aedea09ec0 2024-05-27T07:33:55.515 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:55.270+0000 7fc4d2ffd700 1 -- 172.21.15.52:0/96110 <== mon.2 v2:172.21.15.105:3300/0 5 ==== fsmap(e 104) ==== 4211+0+0 (secure 0 0 0) 0x7fc4e4022070 con 0x55aedea09ec0 2024-05-27T07:33:55.515 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:55.270+0000 7fc50828ab80 10 client.26609 fetch_fsmap finished waiting for FSMap version 104 2024-05-27T07:33:55.516 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:55.270+0000 7fc50828ab80 10 client.26609 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-05-27T07:33:55.516 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:55.270+0000 7fc50828ab80 20 client.26609 populate_metadata read hostname 'smithi052' 2024-05-27T07:33:55.516 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:55.270+0000 7fc50828ab80 1 --2- 172.21.15.52:0/96110 >> [v2:172.21.15.105:6838/500348151,v1:172.21.15.105:6839/500348151] conn(0x55aedeba61d0 0x55aedeba8660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:33:55.516 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:55.270+0000 7fc50828ab80 4 client.26609 mds_command: new command op to 24527 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-27T07:33:55.516 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:55.270+0000 7fc50828ab80 1 -- 172.21.15.52:0/96110 --> [v2:172.21.15.105:6838/500348151,v1:172.21.15.105:6839/500348151] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55aedeb53a70 con 0x55aedeba61d0 2024-05-27T07:33:55.516 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:55.270+0000 7fc4f27fc700 1 --2- 172.21.15.52:0/96110 >> [v2:172.21.15.105:6838/500348151,v1:172.21.15.105:6839/500348151] conn(0x55aedeba61d0 0x55aedeba8660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:33:55.516 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:55.271+0000 7fc4f27fc700 1 --2- 172.21.15.52:0/96110 >> [v2:172.21.15.105:6838/500348151,v1:172.21.15.105:6839/500348151] conn(0x55aedeba61d0 0x55aedeba8660 crc :-1 s=READY pgs=145 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=f1254b0d4a3363e5 server_cookie=31995707cde08da in_seq=0 out_seq=0 2024-05-27T07:33:55.516 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:55.271+0000 7fc4d2ffd700 10 client.26609 ms_handle_connect on v2:172.21.15.105:6838/500348151 2024-05-27T07:33:55.516 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:55.272+0000 7fc4d2ffd700 1 -- 172.21.15.52:0/96110 <== mds.0 v2:172.21.15.105:6838/500348151 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24041 (crc 0 0 0) 0x55aedeb53a70 con 0x55aedeba61d0 2024-05-27T07:33:55.516 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:55.272+0000 7fc4d2ffd700 10 client.26609 handle_command_reply: tid=0 2024-05-27T07:33:55.516 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:55.272+0000 7fc50828ab80 1 client.26609 shutdown 2024-05-27T07:33:55.516 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:55.273+0000 7fc50828ab80 1 -- 172.21.15.52:0/96110 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55aedea09ec0 msgr2=0x55aedeb95940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:33:55.516 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:55.273+0000 7fc50828ab80 1 --2- 172.21.15.52:0/96110 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55aedea09ec0 0x55aedeb95940 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7fc4e4003ad0 tx=0x7fc4e401a050 comp rx=0 tx=0).stop 2024-05-27T07:33:55.517 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:55.273+0000 7fc50828ab80 1 -- 172.21.15.52:0/96110 shutdown_connections 2024-05-27T07:33:55.517 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:55.273+0000 7fc50828ab80 1 --2- 172.21.15.52:0/96110 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55aedeb91ff0 0x55aedeb963c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:33:55.517 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:55.273+0000 7fc50828ab80 1 --2- 172.21.15.52:0/96110 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55aedea09ec0 0x55aedeb95940 unknown :-1 s=CLOSED pgs=226 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:33:55.517 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:55.273+0000 7fc50828ab80 1 --2- 172.21.15.52:0/96110 >> [v2:172.21.15.105:6838/500348151,v1:172.21.15.105:6839/500348151] conn(0x55aedeba61d0 0x55aedeba8660 crc :-1 s=READY pgs=145 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:33:55.517 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:55.273+0000 7fc50828ab80 1 --2- 172.21.15.52:0/96110 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55aedeb8c410 0x55aedeb95e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:33:55.517 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:55.273+0000 7fc50828ab80 1 -- 172.21.15.52:0/96110 >> 172.21.15.52:0/96110 conn(0x55aedea64b70 msgr2=0x55aedeb86ab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:33:55.517 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:55.273+0000 7fc4d2ffd700 0 client.26609 ms_handle_reset on v2:172.21.15.105:6838/500348151 2024-05-27T07:33:55.517 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:55.273+0000 7fc50828ab80 1 -- 172.21.15.52:0/96110 shutdown_connections 2024-05-27T07:33:55.517 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:55.273+0000 7fc50828ab80 1 -- 172.21.15.52:0/96110 wait complete. 2024-05-27T07:33:55.517 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:55.273+0000 7fc50828ab80 20 client.26609 trim_cache size 0 max 16384 2024-05-27T07:33:55.517 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:55.295+0000 7fc50828ab80 1 Processor -- start 2024-05-27T07:33:55.517 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:55.295+0000 7fc50828ab80 1 -- start start 2024-05-27T07:33:55.517 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:55.295+0000 7fc50828ab80 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55aedea09ec0 0x55aedeb874e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:33:55.518 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:55.295+0000 7fc50828ab80 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55aedeb8c410 0x55aedeb96200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:33:55.518 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:55.295+0000 7fc50828ab80 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55aedeb91ff0 0x55aedeb97c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:33:55.518 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:55.295+0000 7fc50828ab80 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x55aedeb94960 con 0x55aedeb91ff0 2024-05-27T07:33:55.518 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:55.295+0000 7fc50828ab80 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7fc4f41ae8d0 con 0x55aedeb8c410 2024-05-27T07:33:55.518 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:55.295+0000 7fc50828ab80 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7fc4f41acca0 con 0x55aedea09ec0 2024-05-27T07:33:55.518 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:55.295+0000 7fc4f37fe700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55aedeb91ff0 0x55aedeb97c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:33:55.518 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:55.295+0000 7fc4f37fe700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55aedeb91ff0 0x55aedeb97c50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.52:40500/0 (socket says 172.21.15.52:40500) 2024-05-27T07:33:55.518 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:55.295+0000 7fc4f37fe700 1 -- 172.21.15.52:0/3589439343 learned_addr learned my addr 172.21.15.52:0/3589439343 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-05-27T07:33:55.518 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:55.295+0000 7fc4f2ffd700 1 --2- 172.21.15.52:0/3589439343 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55aedea09ec0 0x55aedeb874e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:33:55.518 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:55.295+0000 7fc4f27fc700 1 --2- 172.21.15.52:0/3589439343 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55aedeb8c410 0x55aedeb96200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:33:55.518 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:55.296+0000 7fc4f37fe700 1 -- 172.21.15.52:0/3589439343 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55aedea09ec0 msgr2=0x55aedeb874e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:33:55.518 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:55.296+0000 7fc4f37fe700 1 --2- 172.21.15.52:0/3589439343 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55aedea09ec0 0x55aedeb874e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:33:55.519 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:55.296+0000 7fc4f37fe700 1 -- 172.21.15.52:0/3589439343 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55aedeb8c410 msgr2=0x55aedeb96200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:33:55.519 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:55.296+0000 7fc4f37fe700 1 --2- 172.21.15.52:0/3589439343 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55aedeb8c410 0x55aedeb96200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:33:55.519 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:55.296+0000 7fc4f37fe700 1 -- 172.21.15.52:0/3589439343 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55aedeb93eb0 con 0x55aedeb91ff0 2024-05-27T07:33:55.519 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:55.296+0000 7fc4f2ffd700 1 --2- 172.21.15.52:0/3589439343 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55aedea09ec0 0x55aedeb874e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-27T07:33:55.519 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:55.296+0000 7fc4f37fe700 1 --2- 172.21.15.52:0/3589439343 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55aedeb91ff0 0x55aedeb97c50 secure :-1 s=READY pgs=708 cs=0 l=1 rev1=1 crypto rx=0x7fc4ec00ef90 tx=0x7fc4ec004b40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=45d3a0147a1811a6 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:33:55.519 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:55.296+0000 7fc4d97fa700 1 -- 172.21.15.52:0/3589439343 <== mon.0 v2:172.21.15.52:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc4ec0e20c0 con 0x55aedeb91ff0 2024-05-27T07:33:55.519 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:55.296+0000 7fc4d97fa700 1 -- 172.21.15.52:0/3589439343 <== mon.0 v2:172.21.15.52:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fc4ec0e2260 con 0x55aedeb91ff0 2024-05-27T07:33:55.519 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:55.296+0000 7fc4d97fa700 1 -- 172.21.15.52:0/3589439343 <== mon.0 v2:172.21.15.52:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc4ec0a4f40 con 0x55aedeb91ff0 2024-05-27T07:33:55.519 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:55.297+0000 7fc50828ab80 1 -- 172.21.15.52:0/3589439343 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55aedeb91ff0 msgr2=0x55aedeb97c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:33:55.519 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:55.297+0000 7fc50828ab80 1 --2- 172.21.15.52:0/3589439343 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55aedeb91ff0 0x55aedeb97c50 secure :-1 s=READY pgs=708 cs=0 l=1 rev1=1 crypto rx=0x7fc4ec00ef90 tx=0x7fc4ec004b40 comp rx=0 tx=0).stop 2024-05-27T07:33:55.519 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:55.297+0000 7fc50828ab80 1 -- 172.21.15.52:0/3589439343 shutdown_connections 2024-05-27T07:33:55.519 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:55.297+0000 7fc50828ab80 1 --2- 172.21.15.52:0/3589439343 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55aedeb91ff0 0x55aedeb97c50 unknown :-1 s=CLOSED pgs=708 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:33:55.519 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:55.297+0000 7fc50828ab80 1 --2- 172.21.15.52:0/3589439343 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55aedea09ec0 0x55aedeb874e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:33:55.519 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:55.297+0000 7fc50828ab80 1 --2- 172.21.15.52:0/3589439343 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55aedeb8c410 0x55aedeb96200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:33:55.520 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:55.297+0000 7fc50828ab80 1 -- 172.21.15.52:0/3589439343 >> 172.21.15.52:0/3589439343 conn(0x55aedea64b70 msgr2=0x55aedeb7cde0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:33:55.520 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:55.297+0000 7fc50828ab80 1 -- 172.21.15.52:0/3589439343 shutdown_connections 2024-05-27T07:33:55.520 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:55.297+0000 7fc50828ab80 1 -- 172.21.15.52:0/3589439343 wait complete. 2024-05-27T07:33:55.520 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:55.297+0000 7fc50828ab80 1 -- start start 2024-05-27T07:33:55.520 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:55.297+0000 7fc50828ab80 1 Processor -- start 2024-05-27T07:33:55.520 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:55.298+0000 7fc50828ab80 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55aedea09ec0 0x55aedeb9e380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:33:55.520 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:55.298+0000 7fc50828ab80 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55aedeb8c410 0x55aedeb9adb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:33:55.520 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:55.298+0000 7fc50828ab80 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55aedeb91ff0 0x55aedeb9b2f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:33:55.520 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:55.298+0000 7fc50828ab80 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x7fc4f41ad370 con 0x55aedea09ec0 2024-05-27T07:33:55.520 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:55.298+0000 7fc50828ab80 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7fc4f41ae1d0 con 0x55aedeb91ff0 2024-05-27T07:33:55.520 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:55.298+0000 7fc50828ab80 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7fc4f41ada70 con 0x55aedeb8c410 2024-05-27T07:33:55.520 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:55.298+0000 7fc50828ab80 10 client.0 _ll_register_callbacks cb 0x55aedea419a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T07:33:55.521 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:55.298+0000 7fc4f2ffd700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55aedea09ec0 0x55aedeb9e380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:33:55.521 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:55.298+0000 7fc4f2ffd700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55aedea09ec0 0x55aedeb9e380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.52:40506/0 (socket says 172.21.15.52:40506) 2024-05-27T07:33:55.521 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:55.298+0000 7fc4f37fe700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55aedeb91ff0 0x55aedeb9b2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:33:55.521 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:55.298+0000 7fc4f2ffd700 1 -- 172.21.15.52:0/4247306327 learned_addr learned my addr 172.21.15.52:0/4247306327 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-05-27T07:33:55.521 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:55.298+0000 7fc4f27fc700 1 --2- 172.21.15.52:0/4247306327 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55aedeb8c410 0x55aedeb9adb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:33:55.521 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:55.298+0000 7fc4f2ffd700 1 -- 172.21.15.52:0/4247306327 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55aedeb8c410 msgr2=0x55aedeb9adb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:33:55.521 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:55.298+0000 7fc4f2ffd700 1 --2- 172.21.15.52:0/4247306327 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55aedeb8c410 0x55aedeb9adb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:33:55.521 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:55.298+0000 7fc4f2ffd700 1 -- 172.21.15.52:0/4247306327 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55aedeb91ff0 msgr2=0x55aedeb9b2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:33:55.521 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:55.298+0000 7fc4f2ffd700 1 --2- 172.21.15.52:0/4247306327 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55aedeb91ff0 0x55aedeb9b2f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:33:55.521 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:55.298+0000 7fc4f2ffd700 1 -- 172.21.15.52:0/4247306327 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55aedeb9b8e0 con 0x55aedea09ec0 2024-05-27T07:33:55.521 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:55.298+0000 7fc4f37fe700 1 --2- 172.21.15.52:0/4247306327 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55aedeb91ff0 0x55aedeb9b2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-27T07:33:55.521 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:55.299+0000 7fc4f2ffd700 1 --2- 172.21.15.52:0/4247306327 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55aedea09ec0 0x55aedeb9e380 secure :-1 s=READY pgs=709 cs=0 l=1 rev1=1 crypto rx=0x7fc4e4023e90 tx=0x7fc4e402f6b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=196a24f5a998ecd5 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:33:55.521 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:55.299+0000 7fc4d2ffd700 10 client.0 ms_handle_connect on v2:172.21.15.52:3300/0 2024-05-27T07:33:55.521 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:55.299+0000 7fc4d2ffd700 1 -- 172.21.15.52:0/4247306327 <== mon.0 v2:172.21.15.52:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc4e400fe10 con 0x55aedea09ec0 2024-05-27T07:33:55.522 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:55.299+0000 7fc4d2ffd700 1 -- 172.21.15.52:0/4247306327 <== mon.0 v2:172.21.15.52:3300/0 2 ==== osd_map(119..119 src has 1..119) ==== 9451+0+0 (secure 0 0 0) 0x7fc4e402f7d0 con 0x55aedea09ec0 2024-05-27T07:33:55.522 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:55.299+0000 7fc50828ab80 1 -- 172.21.15.52:0/4247306327 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fc4ec0e2260 con 0x55aedea09ec0 2024-05-27T07:33:55.522 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:55.299+0000 7fc4d2ffd700 1 -- 172.21.15.52:0/4247306327 <== mon.0 v2:172.21.15.52:3300/0 3 ==== mon_get_version_reply(handle=1 version=104) ==== 24+0+0 (secure 0 0 0) 0x7fc4e404e050 con 0x55aedea09ec0 2024-05-27T07:33:55.522 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:55.299+0000 7fc50828ab80 10 client.16941 fetch_fsmap learned FSMap version 104 2024-05-27T07:33:55.522 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:55.299+0000 7fc50828ab80 1 -- 172.21.15.52:0/4247306327 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({config=0+,fsmap=104,monmap=4+}) -- 0x55aedec11af0 con 0x55aedea09ec0 2024-05-27T07:33:55.522 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:55.299+0000 7fc4d2ffd700 1 -- 172.21.15.52:0/4247306327 <== mon.0 v2:172.21.15.52:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fc4ec0e2260 con 0x55aedea09ec0 2024-05-27T07:33:55.522 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:55.299+0000 7fc4d2ffd700 1 -- 172.21.15.52:0/4247306327 <== mon.0 v2:172.21.15.52:3300/0 5 ==== fsmap(e 104) ==== 4211+0+0 (secure 0 0 0) 0x7fc4e4022040 con 0x55aedea09ec0 2024-05-27T07:33:55.522 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:55.299+0000 7fc50828ab80 10 client.16941 fetch_fsmap finished waiting for FSMap version 104 2024-05-27T07:33:55.522 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:55.299+0000 7fc50828ab80 10 client.16941 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-05-27T07:33:55.522 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:55.299+0000 7fc50828ab80 20 client.16941 populate_metadata read hostname 'smithi052' 2024-05-27T07:33:55.522 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:55.299+0000 7fc50828ab80 1 --2- 172.21.15.52:0/4247306327 >> [v2:172.21.15.105:6838/500348151,v1:172.21.15.105:6839/500348151] conn(0x55aedec12140 0x55aedec14560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:33:55.522 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:55.299+0000 7fc50828ab80 4 client.16941 mds_command: new command op to 24527 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-05-27T07:33:55.523 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:55.299+0000 7fc50828ab80 1 -- 172.21.15.52:0/4247306327 --> [v2:172.21.15.105:6838/500348151,v1:172.21.15.105:6839/500348151] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) -- 0x55aedeab8c60 con 0x55aedec12140 2024-05-27T07:33:55.523 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:55.300+0000 7fc4f27fc700 1 --2- 172.21.15.52:0/4247306327 >> [v2:172.21.15.105:6838/500348151,v1:172.21.15.105:6839/500348151] conn(0x55aedec12140 0x55aedec14560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:33:55.523 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:55.300+0000 7fc4d2ffd700 10 client.16941 ms_handle_connect on v2:172.21.15.105:6838/500348151 2024-05-27T07:33:55.523 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:55.300+0000 7fc4f27fc700 1 --2- 172.21.15.52:0/4247306327 >> [v2:172.21.15.105:6838/500348151,v1:172.21.15.105:6839/500348151] conn(0x55aedec12140 0x55aedec14560 crc :-1 s=READY pgs=146 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=3e63573fde5dbbcb server_cookie=5fd51acbc0518bde in_seq=0 out_seq=0 2024-05-27T07:33:55.523 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:55.301+0000 7fc4d2ffd700 1 -- 172.21.15.52:0/4247306327 <== mds.0 v2:172.21.15.105:6838/500348151 1 ==== command_reply(tid 0: 0 ) ==== 8+0+110 (crc 0 0 0) 0x55aedeab8c60 con 0x55aedec12140 2024-05-27T07:33:55.523 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:55.301+0000 7fc4d2ffd700 10 client.16941 handle_command_reply: tid=0 2024-05-27T07:33:55.523 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:55.301+0000 7fc50828ab80 1 client.16941 shutdown 2024-05-27T07:33:55.523 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:55.301+0000 7fc50828ab80 1 -- 172.21.15.52:0/4247306327 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55aedea09ec0 msgr2=0x55aedeb9e380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:33:55.523 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:55.301+0000 7fc50828ab80 1 --2- 172.21.15.52:0/4247306327 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55aedea09ec0 0x55aedeb9e380 secure :-1 s=READY pgs=709 cs=0 l=1 rev1=1 crypto rx=0x7fc4e4023e90 tx=0x7fc4e402f6b0 comp rx=0 tx=0).stop 2024-05-27T07:33:55.523 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:55.301+0000 7fc50828ab80 1 -- 172.21.15.52:0/4247306327 shutdown_connections 2024-05-27T07:33:55.523 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:55.301+0000 7fc50828ab80 1 --2- 172.21.15.52:0/4247306327 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55aedeb91ff0 0x55aedeb9b2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:33:55.523 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:55.301+0000 7fc50828ab80 1 --2- 172.21.15.52:0/4247306327 >> [v2:172.21.15.105:6838/500348151,v1:172.21.15.105:6839/500348151] conn(0x55aedec12140 0x55aedec14560 crc :-1 s=READY pgs=146 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:33:55.524 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:55.301+0000 7fc4d2ffd700 0 client.16941 ms_handle_reset on v2:172.21.15.105:6838/500348151 2024-05-27T07:33:55.524 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:55.301+0000 7fc50828ab80 1 --2- 172.21.15.52:0/4247306327 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55aedea09ec0 0x55aedeb9e380 unknown :-1 s=CLOSED pgs=709 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:33:55.524 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:55.302+0000 7fc50828ab80 1 --2- 172.21.15.52:0/4247306327 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55aedeb8c410 0x55aedeb9adb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:33:55.524 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:55.302+0000 7fc50828ab80 1 -- 172.21.15.52:0/4247306327 >> 172.21.15.52:0/4247306327 conn(0x55aedea64b70 msgr2=0x55aedeb8f450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:33:55.524 INFO:teuthology.orchestra.run.smithi052.stdout:{ 2024-05-27T07:33:55.524 INFO:teuthology.orchestra.run.smithi052.stdout: "return_code": 0, 2024-05-27T07:33:55.524 INFO:teuthology.orchestra.run.smithi052.stdout: "scrub_tag": "b39c870d-d57b-4c7d-baaf-c256689c7e81", 2024-05-27T07:33:55.524 INFO:teuthology.orchestra.run.smithi052.stdout: "mode": "asynchronous" 2024-05-27T07:33:55.524 INFO:teuthology.orchestra.run.smithi052.stdout:} 2024-05-27T07:33:55.525 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:55.302+0000 7fc50828ab80 1 -- 172.21.15.52:0/4247306327 shutdown_connections 2024-05-27T07:33:55.525 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:55.302+0000 7fc50828ab80 1 -- 172.21.15.52:0/4247306327 wait complete. 2024-05-27T07:33:55.525 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:55.302+0000 7fc50828ab80 20 client.16941 trim_cache size 0 max 16384 2024-05-27T07:33:55.525 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:55.302+0000 7fc4f9875700 1 -- 172.21.15.52:0/3963227716 >> [v2:172.21.15.52:6800/1822318384,v1:172.21.15.52:6801/1822318384] conn(0x7fc4dc0b29f0 msgr2=0x7fc4dc0b4f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:33:55.525 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:55.302+0000 7fc4f9875700 1 --2- 172.21.15.52:0/3963227716 >> [v2:172.21.15.52:6800/1822318384,v1:172.21.15.52:6801/1822318384] conn(0x7fc4dc0b29f0 0x7fc4dc0b4f00 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7fc4e4006010 tx=0x7fc4e4009590 comp rx=0 tx=0).stop 2024-05-27T07:33:55.525 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:55.302+0000 7fc4f9875700 1 -- 172.21.15.52:0/3963227716 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fc4f410e8e0 msgr2=0x7fc4f419f670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:33:55.525 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:55.302+0000 7fc4f9875700 1 --2- 172.21.15.52:0/3963227716 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fc4f410e8e0 0x7fc4f419f670 secure :-1 s=READY pgs=706 cs=0 l=1 rev1=1 crypto rx=0x7fc4ec00efc0 tx=0x7fc4ec00c3a0 comp rx=0 tx=0).stop 2024-05-27T07:33:55.525 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:55.302+0000 7fc4f9875700 1 -- 172.21.15.52:0/3963227716 shutdown_connections 2024-05-27T07:33:55.525 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:55.302+0000 7fc4f9875700 1 --2- 172.21.15.52:0/3963227716 >> [v2:172.21.15.52:6800/1822318384,v1:172.21.15.52:6801/1822318384] conn(0x7fc4dc0b29f0 0x7fc4dc0b4f00 unknown :-1 s=CLOSED pgs=170 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:33:55.525 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:55.302+0000 7fc4f9875700 1 --2- 172.21.15.52:0/3963227716 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fc4f410e8e0 0x7fc4f419f670 unknown :-1 s=CLOSED pgs=706 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:33:55.526 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:55.302+0000 7fc4f9875700 1 --2- 172.21.15.52:0/3963227716 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fc4f4105470 0x7fc4f419ae40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:33:55.526 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:55.302+0000 7fc4f9875700 1 --2- 172.21.15.52:0/3963227716 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fc4f4105dc0 0x7fc4f419b3e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:33:55.526 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:55.302+0000 7fc4f9875700 1 -- 172.21.15.52:0/3963227716 >> 172.21.15.52:0/3963227716 conn(0x7fc4f40fef20 msgr2=0x7fc4f40ff570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:33:55.526 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:55.303+0000 7fc4f9875700 1 -- 172.21.15.52:0/3963227716 shutdown_connections 2024-05-27T07:33:55.526 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:55.303+0000 7fc4f9875700 1 -- 172.21.15.52:0/3963227716 wait complete. 2024-05-27T07:33:56.165 INFO:journalctl@ceph.mds.b.smithi105.stdout:May 27 07:33:55 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-b[65543]: 2024-05-27T07:33:55.845+0000 7f33fced0700 -1 mds.0.scrubstack _validate_inode_done scrub error on inode [inode 0x100000055d6 [...a,head] /volumes/qa/sv_1/95e9ac73-825f-48e3-998b-133d94d96559/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-78/ auth{4=1} fragtree_t(*^3) v251474 ap=2 RANDEPHEMERALPIN f(v13 m2024-05-27T07:32:49.932927+0000 160=160+0) n(v16 rc2024-05-27T07:32:49.932927+0000 b2428200 161=160+1) old_inodes=3 (inest sync->mix g=4) caps={25060=pAsLsXsFs/pAsLsXsFsx@168} | dirtyscattered=0 request=0 lock=0 dirfrag=8 caps=1 dirtyparent=0 replicated=1 dirty=0 waiter=0 authpin=1 discoverbase=0 scrubqueue=0 randepin 0x55ee504c8c00]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":true,"read_ret_val":0,"ondisk_value":"(2)0x100000055d6:[<0x10000000399/blog-78 v190478>,<0x10000000208/blogtest_in v917>,<0x10000000201/src v2221>,<0x10000000006/blogbench-1.0 v1969>,<0x10000000005/tmp v270>,<0x100000001fb/client.0 v241>,<0x100000001fa/95e9ac73-825f-48e3-998b-133d94d96559 v253>,<0x10000000001/sv_1 v264>,<0x10000000000/qa v255>,<0x1/volumes v248>]//[]","memoryvalue":"(2)0x100000055d6:[<0x10000000399/blog-78 v251474>,<0x10000000208/blogtest_in v2204>,<0x10000000201/src v2692>,<0x10000000006/blogbench-1.0 v2345>,<0x10000000005/tmp v624>,<0x100000001fb/client.0 v587>,<0x100000001fa/95e9ac73-825f-48e3-998b-133d94d96559 v592>,<0x10000000001/sv_1 v607>,<0x10000000000/qa v579>,<0x1/volumes v588>]//[]","error_str":""},"raw_stats":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f(v0 m2024-05-27T07:32:38.084063+0000 159=159+0)","ondisk_value.rstat":"n(v0 rc2024-05-27T07:32:49.932927+0000 b2428200 161=160+1)","memory_value.dirstat":"f(v13 m2024-05-27T07:32:49.932927+0000 160=160+0)","memory_value.rstat":"n(v16 rc2024-05-27T07:32:49.932927+0000 b2428200 161=160+1)","error_str":"freshly-calculated rstats don't match existing ones"},"return_code":0} 2024-05-27T07:33:56.165 INFO:journalctl@ceph.mds.b.smithi105.stdout:May 27 07:33:56 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-b[65543]: 2024-05-27T07:33:56.020+0000 7f33fced0700 -1 mds.0.scrubstack _validate_inode_done scrub error on inode [inode 0x10000000c2a [...a,head] /volumes/qa/sv_1/95e9ac73-825f-48e3-998b-133d94d96559/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-14/ auth{3=1} fragtree_t(*^3) v251472 ap=2 RANDEPHEMERALPIN f(v20 m2024-05-27T07:32:47.354957+0000 204=204+0) n(v26 rc2024-05-27T07:32:47.354957+0000 b2695830 205=204+1) old_inodes=5 (inest sync->mix g=3) caps={25060=pAsLsXsFs/pAsLsXsFsx@336} mcw={3=AsLsXsFs} | dirtyscattered=0 request=0 lock=0 dirfrag=8 caps=1 dirtyparent=0 replicated=1 dirty=0 waiter=0 authpin=1 discoverbase=0 scrubqueue=0 randepin 0x55ee4db18000]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":true,"read_ret_val":0,"ondisk_value":"(2)0x10000000c2a:[<0x10000000399/blog-14 v34369>,<0x10000000208/blogtest_in v368>,<0x10000000201/src v2008>,<0x10000000006/blogbench-1.0 v1768>,<0x10000000005/tmp v67>,<0x100000001fb/client.0 v58>,<0x100000001fa/95e9ac73-825f-48e3-998b-133d94d96559 v72>,<0x10000000001/sv_1 v89>,<0x10000000000/qa v79>,<0x1/volumes v78>]//[]","memoryvalue":"(2)0x10000000c2a:[<0x10000000399/blog-14 v251472>,<0x10000000208/blogtest_in v2208>,<0x10000000201/src v2694>,<0x10000000006/blogbench-1.0 v2347>,<0x10000000005/tmp v626>,<0x100000001fb/client.0 v589>,<0x100000001fa/95e9ac73-825f-48e3-998b-133d94d96559 v594>,<0x10000000001/sv_1 v609>,<0x10000000000/qa v581>,<0x1/volumes v591>]//[]","error_str":""},"raw_stats":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f(v0 m2024-05-27T07:31:57.488529+0000 203=203+0)","ondisk_value.rstat":"n(v0 rc2024-05-27T07:32:47.354957+0000 b2695830 205=204+1)","memory_value.dirstat":"f(v20 m2024-05-27T07:32:47.354957+0000 204=204+0)","memory_value.rstat":"n(v26 rc2024-05-27T07:32:47.354957+0000 b2695830 205=204+1)","error_str":"freshly-calculated rstats don't match existing ones"},"return_code":0} 2024-05-27T07:33:56.166 INFO:journalctl@ceph.mds.b.smithi105.stdout:May 27 07:33:56 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-b[65543]: 2024-05-27T07:33:56.170+0000 7f33fced0700 -1 mds.0.scrubstack _validate_inode_done scrub error on inode [inode 0x1000000819b [...a,head] /volumes/qa/sv_1/95e9ac73-825f-48e3-998b-133d94d96559/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-127/ auth{3=1} fragtree_t(*^3) v257294 ap=2 RANDEPHEMERALPIN f(v4 m2024-05-27T07:32:39.205050+0000 114=114+0) n(v7 rc2024-05-27T07:32:39.205050+0000 b3120010 115=114+1) old_inodes=2 (inest sync->mix g=3) caps={25060=pAsLsXsFs/pAsLsXsFsx@643} | dirtyscattered=0 request=0 lock=0 dirfrag=8 caps=1 dirtyparent=0 replicated=1 dirty=0 waiter=0 authpin=1 discoverbase=0 scrubqueue=0 randepin 0x55ee50df4100]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":true,"read_ret_val":0,"ondisk_value":"(2)0x1000000819b:[<0x10000000399/blog-127 v236640>,<0x10000000208/blogtest_in v1385>,<0x10000000201/src v2380>,<0x10000000006/blogbench-1.0 v2115>,<0x10000000005/tmp v403>,<0x100000001fb/client.0 v370>,<0x100000001fa/95e9ac73-825f-48e3-998b-133d94d96559 v380>,<0x10000000001/sv_1 v398>,<0x10000000000/qa v377>,<0x1/volumes v381>]//[]","memoryvalue":"(2)0x1000000819b:[<0x10000000399/blog-127 v257294>,<0x10000000208/blogtest_in v2210>,<0x10000000201/src v2694>,<0x10000000006/blogbench-1.0 v2347>,<0x10000000005/tmp v626>,<0x100000001fb/client.0 v589>,<0x100000001fa/95e9ac73-825f-48e3-998b-133d94d96559 v594>,<0x10000000001/sv_1 v609>,<0x10000000000/qa v581>,<0x1/volumes v591>]//[]","error_str":""},"raw_stats":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f(v0 m2024-05-27T07:32:07.347416+0000 113=113+0)","ondisk_value.rstat":"n(v0 rc2024-05-27T07:32:39.205050+0000 b3120010 115=114+1)","memory_value.dirstat":"f(v4 m2024-05-27T07:32:39.205050+0000 114=114+0)","memory_value.rstat":"n(v7 rc2024-05-27T07:32:39.205050+0000 b3120010 115=114+1)","error_str":"freshly-calculated rstats don't match existing ones"},"return_code":0} 2024-05-27T07:33:56.303 DEBUG:teuthology.orchestra.run.smithi052:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:38f9d900c83c63799fbdbe61acc9a11b0d3554a6 shell --fsid 73df2636-1bf8-11ef-bc9b-c7b262605968 -- ceph tell mds.1:0 scrub status 2024-05-27T07:33:56.444 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:33:56 smithi105 ceph-mon[35977]: pgmap v270: 129 pgs: 129 active+clean; 668 MiB data, 4.6 GiB used, 1.0 TiB / 1.0 TiB avail; 35 KiB/s rd, 1.9 MiB/s wr, 554 op/s 2024-05-27T07:33:56.445 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:33:56 smithi105 ceph-mon[35977]: scrub queued for path: / 2024-05-27T07:33:56.445 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:33:56 smithi105 ceph-mon[35977]: scrub summary: idle+waiting paths [/] 2024-05-27T07:33:56.445 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:33:56 smithi105 ceph-mon[35977]: scrub summary: active paths [/] 2024-05-27T07:33:56.445 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:33:56 smithi105 ceph-mon[35977]: scrub summary: active paths [/] 2024-05-27T07:33:56.445 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:33:56 smithi105 ceph-mon[35977]: scrub summary: idle+waiting paths [/] 2024-05-27T07:33:56.446 INFO:journalctl@ceph.mds.b.smithi105.stdout:May 27 07:33:56 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-b[65543]: 2024-05-27T07:33:56.179+0000 7f33fced0700 -1 mds.0.scrubstack _validate_inode_done scrub error on inode [inode 0x10000009421 [...a,head] /volumes/qa/sv_1/95e9ac73-825f-48e3-998b-133d94d96559/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-141/ auth{2=1} fragtree_t(*^3) v257291 ap=2 RANDEPHEMERALPIN f(v4 m2024-05-27T07:32:43.464001+0000 103=103+0) n(v7 rc2024-05-27T07:32:43.464001+0000 b3174278 104=103+1) old_inodes=2 (inest sync->mix g=2) caps={25060=pAsLsXsFs/pAsLsXsFsx@176} | dirtyscattered=0 request=0 lock=0 dirfrag=8 caps=1 dirtyparent=0 replicated=1 dirty=0 waiter=0 authpin=1 discoverbase=0 scrubqueue=0 randepin 0x55ee50f2a580]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":true,"read_ret_val":0,"ondisk_value":"(2)0x10000009421:[<0x10000000399/blog-141 v243309>,<0x10000000208/blogtest_in v1521>,<0x10000000201/src v2419>,<0x10000000006/blogbench-1.0 v2148>,<0x10000000005/tmp v436>,<0x100000001fb/client.0 v403>,<0x100000001fa/95e9ac73-825f-48e3-998b-133d94d96559 v410>,<0x10000000001/sv_1 v431>,<0x10000000000/qa v407>,<0x1/volumes v413>]//[]","memoryvalue":"(2)0x10000009421:[<0x10000000399/blog-141 v257291>,<0x10000000208/blogtest_in v2210>,<0x10000000201/src v2694>,<0x10000000006/blogbench-1.0 v2347>,<0x10000000005/tmp v626>,<0x100000001fb/client.0 v589>,<0x100000001fa/95e9ac73-825f-48e3-998b-133d94d96559 v594>,<0x10000000001/sv_1 v609>,<0x10000000000/qa v581>,<0x1/volumes v591>]//[]","error_str":""},"raw_stats":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f(v0 m2024-05-27T07:32:35.849089+0000 102=102+0)","ondisk_value.rstat":"n(v0 rc2024-05-27T07:32:43.464001+0000 b3174278 104=103+1)","memory_value.dirstat":"f(v4 m2024-05-27T07:32:43.464001+0000 103=103+0)","memory_value.rstat":"n(v7 rc2024-05-27T07:32:43.464001+0000 b3174278 104=103+1)","error_str":"freshly-calculated rstats don't match existing ones"},"return_code":0} 2024-05-27T07:33:56.446 INFO:journalctl@ceph.mds.b.smithi105.stdout:May 27 07:33:56 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-b[65543]: 2024-05-27T07:33:56.294+0000 7f33fced0700 -1 mds.0.scrubstack _validate_inode_done scrub error on inode [inode 0x10000003fad [...a,head] /volumes/qa/sv_1/95e9ac73-825f-48e3-998b-133d94d96559/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-58/ auth{4=1} fragtree_t(*^3) v239494 ap=2 RANDEPHEMERALPIN f(v9 m2024-05-27T07:32:27.400186+0000 105=105+0) n(v14 rc2024-05-27T07:32:27.400186+0000 b2409600 106=105+1) old_inodes=4 (inest sync->mix g=4) caps={25060=pAsLsXsFs/pAsLsXsFsx@226} | dirtyscattered=0 request=0 lock=0 dirfrag=8 caps=1 dirtyparent=0 replicated=1 dirty=0 waiter=0 authpin=1 discoverbase=0 scrubqueue=0 randepin 0x55ee54bf8b00]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":true,"read_ret_val":0,"ondisk_value":"(2)0x10000003fad:[<0x10000000399/blog-58 v152699>,<0x10000000208/blogtest_in v814>,<0x10000000201/src v2182>,<0x10000000006/blogbench-1.0 v1931>,<0x10000000005/tmp v231>,<0x100000001fb/client.0 v208>,<0x100000001fa/95e9ac73-825f-48e3-998b-133d94d96559 v218>,<0x10000000001/sv_1 v230>,<0x10000000000/qa v222>,<0x1/volumes v215>]//[]","memoryvalue":"(2)0x10000003fad:[<0x10000000399/blog-58 v239494>,<0x10000000208/blogtest_in v2210>,<0x10000000201/src v2694>,<0x10000000006/blogbench-1.0 v2347>,<0x10000000005/tmp v626>,<0x100000001fb/client.0 v589>,<0x100000001fa/95e9ac73-825f-48e3-998b-133d94d96559 v594>,<0x10000000001/sv_1 v609>,<0x10000000000/qa v581>,<0x1/volumes v591>]//[]","error_str":""},"raw_stats":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f(v0 m2024-05-27T07:31:35.406784+0000 104=104+0)","ondisk_value.rstat":"n(v0 rc2024-05-27T07:32:27.400186+0000 b2409600 106=105+1)","memory_value.dirstat":"f(v9 m2024-05-27T07:32:27.400186+0000 105=105+0)","memory_value.rstat":"n(v14 rc2024-05-27T07:32:27.400186+0000 b2409600 106=105+1)","error_str":"freshly-calculated rstats don't match existing ones"},"return_code":0} 2024-05-27T07:33:56.446 INFO:journalctl@ceph.mds.b.smithi105.stdout:May 27 07:33:56 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-b[65543]: 2024-05-27T07:33:56.355+0000 7f33fced0700 -1 mds.0.scrubstack _validate_inode_done scrub error on inode [inode 0x1000000918d [...a,head] /volumes/qa/sv_1/95e9ac73-825f-48e3-998b-133d94d96559/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-139/ auth{1=1,3=1,4=1} fragtree_t(*^3) v239498 ap=2 RANDEPHEMERALPIN f(v4 m2024-05-27T07:32:50.502921+0000 107=107+0) n(v8 rc2024-05-27T07:32:50.502921+0000 b3643698 108=107+1) old_inodes=2 (inest sync->mix g=1,3,4) caps={25060=pAsLsXsFs/pAsLsXsFsx@167} | dirtyscattered=0 request=0 lock=0 dirfrag=8 caps=1 dirtyrstat=0 dirtyparent=0 replicated=1 dirty=0 waiter=0 authpin=1 scrubqueue=0 randepin 0x55ee5187b700]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":true,"read_ret_val":0,"ondisk_value":"(2)0x1000000918d:[<0x10000000399/blog-139 v234197>,<0x10000000208/blogtest_in v1521>,<0x10000000201/src v2419>,<0x10000000006/blogbench-1.0 v2148>,<0x10000000005/tmp v436>,<0x100000001fb/client.0 v403>,<0x100000001fa/95e9ac73-825f-48e3-998b-133d94d96559 v410>,<0x10000000001/sv_1 v431>,<0x10000000000/qa v407>,<0x1/volumes v413>]//[]","memoryvalue":"(2)0x1000000918d:[<0x10000000399/blog-139 v239498>,<0x10000000208/blogtest_in v2210>,<0x10000000201/src v2694>,<0x10000000006/blogbench-1.0 v2347>,<0x10000000005/tmp v626>,<0x100000001fb/client.0 v589>,<0x100000001fa/95e9ac73-825f-48e3-998b-133d94d96559 v594>,<0x10000000001/sv_1 v609>,<0x10000000000/qa v581>,<0x1/volumes v591>]//[]","error_str":""},"raw_stats":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f(v0 m2024-05-27T07:32:27.669183+0000 106=106+0)","ondisk_value.rstat":"n(v0 rc2024-05-27T07:32:50.502921+0000 b3643698 108=107+1)","memory_value.dirstat":"f(v4 m2024-05-27T07:32:50.502921+0000 107=107+0)","memory_value.rstat":"n(v8 rc2024-05-27T07:32:50.502921+0000 b3643698 108=107+1)","error_str":"freshly-calculated rstats don't match existing ones"},"return_code":0} 2024-05-27T07:33:56.446 INFO:journalctl@ceph.mds.b.smithi105.stdout:May 27 07:33:56 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-b[65543]: 2024-05-27T07:33:56.357+0000 7f33fced0700 -1 mds.0.scrubstack _validate_inode_done scrub error on inode [inode 0x100000089f8 [...a,head] /volumes/qa/sv_1/95e9ac73-825f-48e3-998b-133d94d96559/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-134/ auth{3=1,4=1} fragtree_t(*^3) v239496 ap=2 RANDEPHEMERALPIN f(v6 m2024-05-27T07:32:47.574954+0000 192=192+0) n(v10 rc2024-05-27T07:32:47.574954+0000 b3668857 193=192+1) old_inodes=2 (inest sync->mix g=3,4) caps={25060=pAsLsXsFs/pAsLsXsFsx@164} mcw={4=AsLsXsFs} | dirtyscattered=0 request=0 lock=0 dirfrag=8 caps=1 dirtyrstat=0 dirtyparent=0 replicated=1 dirty=0 waiter=0 authpin=1 scrubqueue=0 randepin 0x55ee57608b00]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":true,"read_ret_val":0,"ondisk_value":"(2)0x100000089f8:[<0x10000000399/blog-134 v233203>,<0x10000000208/blogtest_in v1521>,<0x10000000201/src v2419>,<0x10000000006/blogbench-1.0 v2148>,<0x10000000005/tmp v436>,<0x100000001fb/client.0 v403>,<0x100000001fa/95e9ac73-825f-48e3-998b-133d94d96559 v410>,<0x10000000001/sv_1 v431>,<0x10000000000/qa v407>,<0x1/volumes v413>]//[]","memoryvalue":"(2)0x100000089f8:[<0x10000000399/blog-134 v239496>,<0x10000000208/blogtest_in v2210>,<0x10000000201/src v2694>,<0x10000000006/blogbench-1.0 v2347>,<0x10000000005/tmp v626>,<0x100000001fb/client.0 v589>,<0x100000001fa/95e9ac73-825f-48e3-998b-133d94d96559 v594>,<0x10000000001/sv_1 v609>,<0x10000000000/qa v581>,<0x1/volumes v591>]//[]","error_str":""},"raw_stats":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f(v0 m2024-05-27T07:32:43.410002+0000 191=191+0)","ondisk_value.rstat":"n(v0 rc2024-05-27T07:32:47.574954+0000 b3668857 193=192+1)","memory_value.dirstat":"f(v6 m2024-05-27T07:32:47.574954+0000 192=192+0)","memory_value.rstat":"n(v10 rc2024-05-27T07:32:47.574954+0000 b3668857 193=192+1)","error_str":"freshly-calculated rstats don't match existing ones"},"return_code":0} 2024-05-27T07:33:56.483 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:33:56 smithi052 ceph-mon[31955]: pgmap v270: 129 pgs: 129 active+clean; 668 MiB data, 4.6 GiB used, 1.0 TiB / 1.0 TiB avail; 35 KiB/s rd, 1.9 MiB/s wr, 554 op/s 2024-05-27T07:33:56.484 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:33:56 smithi052 ceph-mon[31955]: scrub queued for path: / 2024-05-27T07:33:56.484 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:33:56 smithi052 ceph-mon[31955]: scrub summary: idle+waiting paths [/] 2024-05-27T07:33:56.484 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:33:56 smithi052 ceph-mon[31955]: scrub summary: active paths [/] 2024-05-27T07:33:56.484 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:33:56 smithi052 ceph-mon[31955]: scrub summary: active paths [/] 2024-05-27T07:33:56.484 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:33:56 smithi052 ceph-mon[31955]: scrub summary: idle+waiting paths [/] 2024-05-27T07:33:56.606 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:33:56 smithi173 ceph-mon[37000]: pgmap v270: 129 pgs: 129 active+clean; 668 MiB data, 4.6 GiB used, 1.0 TiB / 1.0 TiB avail; 35 KiB/s rd, 1.9 MiB/s wr, 554 op/s 2024-05-27T07:33:56.606 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:33:56 smithi173 ceph-mon[37000]: scrub queued for path: / 2024-05-27T07:33:56.606 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:33:56 smithi173 ceph-mon[37000]: scrub summary: idle+waiting paths [/] 2024-05-27T07:33:56.607 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:33:56 smithi173 ceph-mon[37000]: scrub summary: active paths [/] 2024-05-27T07:33:56.607 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:33:56 smithi173 ceph-mon[37000]: scrub summary: active paths [/] 2024-05-27T07:33:56.607 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:33:56 smithi173 ceph-mon[37000]: scrub summary: idle+waiting paths [/] 2024-05-27T07:33:56.616 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:33:56.620+0000 7fb80f7b2640 1 -- 172.21.15.52:0/910347062 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fb808104410 msgr2=0x7fb8081107e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:33:56.616 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:33:56.620+0000 7fb80f7b2640 1 --2- 172.21.15.52:0/910347062 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fb808104410 0x7fb8081107e0 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7fb80001acb0 tx=0x7fb80003b900 comp rx=0 tx=0).stop 2024-05-27T07:33:56.618 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:33:56.624+0000 7fb80f7b2640 1 -- 172.21.15.52:0/910347062 shutdown_connections 2024-05-27T07:33:56.618 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:33:56.624+0000 7fb80f7b2640 1 --2- 172.21.15.52:0/910347062 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fb808104410 0x7fb8081107e0 unknown :-1 s=CLOSED pgs=180 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:33:56.618 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:33:56.624+0000 7fb80f7b2640 1 --2- 172.21.15.52:0/910347062 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fb80810a160 0x7fb808103da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:33:56.618 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:33:56.624+0000 7fb80f7b2640 1 --2- 172.21.15.52:0/910347062 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fb808109cc0 0x7fb808103860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:33:56.618 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:33:56.624+0000 7fb80f7b2640 1 -- 172.21.15.52:0/910347062 >> 172.21.15.52:0/910347062 conn(0x7fb8080793c0 msgr2=0x7fb808079800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:33:56.621 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:33:56.627+0000 7fb80f7b2640 1 -- 172.21.15.52:0/910347062 shutdown_connections 2024-05-27T07:33:56.623 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:33:56.628+0000 7fb80f7b2640 1 -- 172.21.15.52:0/910347062 wait complete. 2024-05-27T07:33:56.626 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:33:56.629+0000 7fb80f7b2640 1 Processor -- start 2024-05-27T07:33:56.627 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:33:56.630+0000 7fb80f7b2640 1 -- start start 2024-05-27T07:33:56.627 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:33:56.631+0000 7fb80f7b2640 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fb808104410 0x7fb808110300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:33:56.627 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:33:56.631+0000 7fb80f7b2640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fb808109cc0 0x7fb80810a6c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:33:56.627 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:33:56.631+0000 7fb80d54e640 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fb808104410 0x7fb808110300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:33:56.627 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:33:56.631+0000 7fb80d54e640 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fb808104410 0x7fb808110300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.52:40530/0 (socket says 172.21.15.52:40530) 2024-05-27T07:33:56.627 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:33:56.631+0000 7fb80d54e640 1 -- 172.21.15.52:0/625699404 learned_addr learned my addr 172.21.15.52:0/625699404 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-05-27T07:33:56.628 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:33:56.632+0000 7fb80f7b2640 1 --2- 172.21.15.52:0/625699404 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fb80810a160 0x7fb80810ac00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:33:56.628 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:33:56.632+0000 7fb80f7b2640 1 -- 172.21.15.52:0/625699404 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x7fb80806e560 con 0x7fb808104410 2024-05-27T07:33:56.628 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:33:56.632+0000 7fb80f7b2640 1 -- 172.21.15.52:0/625699404 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7fb80806e3e0 con 0x7fb80810a160 2024-05-27T07:33:56.628 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:33:56.632+0000 7fb80f7b2640 1 -- 172.21.15.52:0/625699404 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7fb80806e6e0 con 0x7fb808109cc0 2024-05-27T07:33:56.628 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:33:56.632+0000 7fb80dd4f640 1 --2- 172.21.15.52:0/625699404 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fb80810a160 0x7fb80810ac00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:33:56.629 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:33:56.633+0000 7fb80d54e640 1 -- 172.21.15.52:0/625699404 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fb808109cc0 msgr2=0x7fb80810a6c0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-05-27T07:33:56.629 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:33:56.633+0000 7fb80d54e640 1 --2- 172.21.15.52:0/625699404 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fb808109cc0 0x7fb80810a6c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:33:56.629 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:33:56.633+0000 7fb80d54e640 1 -- 172.21.15.52:0/625699404 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fb80810a160 msgr2=0x7fb80810ac00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:33:56.629 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:33:56.633+0000 7fb80d54e640 1 --2- 172.21.15.52:0/625699404 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fb80810a160 0x7fb80810ac00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:33:56.629 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:33:56.633+0000 7fb80d54e640 1 -- 172.21.15.52:0/625699404 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb80810b140 con 0x7fb808104410 2024-05-27T07:33:56.629 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:33:56.634+0000 7fb80d54e640 1 --2- 172.21.15.52:0/625699404 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fb808104410 0x7fb808110300 secure :-1 s=READY pgs=710 cs=0 l=1 rev1=1 crypto rx=0x7fb7f800e9c0 tx=0x7fb7f800ee80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:33:56.629 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:33:56.634+0000 7fb7fe7fc640 1 -- 172.21.15.52:0/625699404 <== mon.0 v2:172.21.15.52:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb7f800cd80 con 0x7fb808104410 2024-05-27T07:33:56.629 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:33:56.634+0000 7fb7fe7fc640 1 -- 172.21.15.52:0/625699404 <== mon.0 v2:172.21.15.52:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fb7f80054d0 con 0x7fb808104410 2024-05-27T07:33:56.629 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:33:56.634+0000 7fb7fe7fc640 1 -- 172.21.15.52:0/625699404 <== mon.0 v2:172.21.15.52:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb7f8010430 con 0x7fb808104410 2024-05-27T07:33:56.629 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:33:56.634+0000 7fb80f7b2640 1 -- 172.21.15.52:0/625699404 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb80810b430 con 0x7fb808104410 2024-05-27T07:33:56.629 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:33:56.634+0000 7fb80f7b2640 1 -- 172.21.15.52:0/625699404 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb8081a4e40 con 0x7fb808104410 2024-05-27T07:33:56.636 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:33:56.642+0000 7fb7fe7fc640 1 -- 172.21.15.52:0/625699404 <== mon.0 v2:172.21.15.52:3300/0 4 ==== mgrmap(e 26) ==== 149783+0+0 (secure 0 0 0) 0x7fb7f80105d0 con 0x7fb808104410 2024-05-27T07:33:56.637 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:33:56.642+0000 7fb816669740 1 Processor -- start 2024-05-27T07:33:56.638 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:33:56.644+0000 7fb7fe7fc640 1 --2- 172.21.15.52:0/625699404 >> [v2:172.21.15.52:6800/1822318384,v1:172.21.15.52:6801/1822318384] conn(0x7fb7f40b2a50 0x7fb7f40b4f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:33:56.638 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:33:56.644+0000 7fb7fe7fc640 1 -- 172.21.15.52:0/625699404 <== mon.0 v2:172.21.15.52:3300/0 5 ==== osd_map(119..119 src has 1..119) ==== 9451+0+0 (secure 0 0 0) 0x7fb7f80e3e80 con 0x7fb808104410 2024-05-27T07:33:56.639 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:33:56.644+0000 7fb816669740 1 -- start start 2024-05-27T07:33:56.639 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:33:56.645+0000 7fb816669740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55da09f5c1b0 0x55da0a2fd3d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:33:56.639 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:33:56.645+0000 7fb816669740 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55da09f332f0 0x55da0a2fd910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:33:56.639 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:33:56.645+0000 7fb816669740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55da09f3b4d0 0x55da0a2fde50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:33:56.640 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:33:56.645+0000 7fb816669740 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x55da0a26fc30 con 0x55da09f332f0 2024-05-27T07:33:56.640 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:33:56.645+0000 7fb816669740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x55da0a25f7d0 con 0x55da09f3b4d0 2024-05-27T07:33:56.640 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:33:56.645+0000 7fb816669740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x55da0a263810 con 0x55da09f5c1b0 2024-05-27T07:33:56.640 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:33:56.645+0000 7fb80d54e640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55da09f5c1b0 0x55da0a2fd3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:33:56.640 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:33:56.645+0000 7fb80d54e640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55da09f5c1b0 0x55da0a2fd3d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.173:3300/0 says I am v2:172.21.15.52:53236/0 (socket says 172.21.15.52:53236) 2024-05-27T07:33:56.640 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:33:56.645+0000 7fb80d54e640 1 -- 172.21.15.52:0/2883770515 learned_addr learned my addr 172.21.15.52:0/2883770515 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-05-27T07:33:56.640 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:33:56.645+0000 7fb80dd4f640 1 --2- 172.21.15.52:0/2883770515 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55da09f3b4d0 0x55da0a2fde50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:33:56.641 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:33:56.645+0000 7fb80d54e640 1 -- 172.21.15.52:0/2883770515 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55da09f3b4d0 msgr2=0x55da0a2fde50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:33:56.641 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:33:56.645+0000 7fb80d54e640 1 --2- 172.21.15.52:0/2883770515 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55da09f3b4d0 0x55da0a2fde50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:33:56.641 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:33:56.645+0000 7fb80d54e640 1 -- 172.21.15.52:0/2883770515 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55da09f332f0 msgr2=0x55da0a2fd910 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-05-27T07:33:56.641 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:33:56.645+0000 7fb80d54e640 1 --2- 172.21.15.52:0/2883770515 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55da09f332f0 0x55da0a2fd910 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:33:56.642 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:33:56.645+0000 7fb80d54e640 1 -- 172.21.15.52:0/2883770515 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55da09f74210 con 0x55da09f5c1b0 2024-05-27T07:33:56.642 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:33:56.646+0000 7fb80dd4f640 1 --2- 172.21.15.52:0/625699404 >> [v2:172.21.15.52:6800/1822318384,v1:172.21.15.52:6801/1822318384] conn(0x7fb7f40b2a50 0x7fb7f40b4f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:33:56.642 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:33:56.646+0000 7fb80d54e640 1 --2- 172.21.15.52:0/2883770515 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55da09f5c1b0 0x55da0a2fd3d0 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7fb7f800c6a0 tx=0x7fb7f80aa9b0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=d4b60e64a601013a server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:33:56.642 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:33:56.646+0000 7fb7f37fe640 1 -- 172.21.15.52:0/2883770515 <== mon.1 v2:172.21.15.173:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb7f80aad60 con 0x55da09f5c1b0 2024-05-27T07:33:56.642 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:33:56.646+0000 7fb7f37fe640 1 -- 172.21.15.52:0/2883770515 <== mon.1 v2:172.21.15.173:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fb7f80af110 con 0x55da09f5c1b0 2024-05-27T07:33:56.642 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:33:56.646+0000 7fb7f37fe640 1 -- 172.21.15.52:0/2883770515 <== mon.1 v2:172.21.15.173:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb7f80aa220 con 0x55da09f5c1b0 2024-05-27T07:33:56.643 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:33:56.648+0000 7fb816669740 1 -- 172.21.15.52:0/2883770515 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55da09f5c1b0 msgr2=0x55da0a2fd3d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:33:56.643 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:33:56.648+0000 7fb816669740 1 --2- 172.21.15.52:0/2883770515 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55da09f5c1b0 0x55da0a2fd3d0 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7fb7f800c6a0 tx=0x7fb7f80aa9b0 comp rx=0 tx=0).stop 2024-05-27T07:33:56.643 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:33:56.648+0000 7fb80dd4f640 1 --2- 172.21.15.52:0/625699404 >> [v2:172.21.15.52:6800/1822318384,v1:172.21.15.52:6801/1822318384] conn(0x7fb7f40b2a50 0x7fb7f40b4f60 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7fb80001a8e0 tx=0x7fb80003b900 comp rx=0 tx=0).ready entity=mgr.15333 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:33:56.644 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:33:56.648+0000 7fb816669740 1 -- 172.21.15.52:0/2883770515 shutdown_connections 2024-05-27T07:33:56.644 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:33:56.648+0000 7fb816669740 1 --2- 172.21.15.52:0/2883770515 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55da09f3b4d0 0x55da0a2fde50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:33:56.644 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:33:56.648+0000 7fb816669740 1 --2- 172.21.15.52:0/2883770515 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55da09f332f0 0x55da0a2fd910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:33:56.644 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:33:56.648+0000 7fb816669740 1 --2- 172.21.15.52:0/2883770515 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55da09f5c1b0 0x55da0a2fd3d0 unknown :-1 s=CLOSED pgs=181 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:33:56.644 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:33:56.648+0000 7fb816669740 1 -- 172.21.15.52:0/2883770515 >> 172.21.15.52:0/2883770515 conn(0x55da09f50300 msgr2=0x55da0a2f9b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:33:56.644 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:33:56.648+0000 7fb816669740 1 -- 172.21.15.52:0/2883770515 shutdown_connections 2024-05-27T07:33:56.644 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:33:56.648+0000 7fb816669740 1 -- 172.21.15.52:0/2883770515 wait complete. 2024-05-27T07:33:56.644 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:33:56.649+0000 7fb816669740 1 -- start start 2024-05-27T07:33:56.645 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:33:56.649+0000 7fb816669740 1 Processor -- start 2024-05-27T07:33:56.645 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:33:56.650+0000 7fb816669740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55da09f332f0 0x55da0a307a50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:33:56.645 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:33:56.650+0000 7fb816669740 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55da09f3b4d0 0x55da0a307f90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:33:56.645 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:33:56.650+0000 7fb816669740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55da09f5c1b0 0x55da0a31e930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:33:56.646 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:33:56.650+0000 7fb80d54e640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55da09f332f0 0x55da0a307a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:33:56.646 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:33:56.650+0000 7fb80d54e640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55da09f332f0 0x55da0a307a50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.173:3300/0 says I am v2:172.21.15.52:53252/0 (socket says 172.21.15.52:53252) 2024-05-27T07:33:56.646 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:33:56.650+0000 7fb80d54e640 1 -- 172.21.15.52:0/1694433472 learned_addr learned my addr 172.21.15.52:0/1694433472 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-05-27T07:33:56.646 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:33:56.650+0000 7fb80dd4f640 1 --2- 172.21.15.52:0/1694433472 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55da09f5c1b0 0x55da0a31e930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:33:56.646 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:33:56.650+0000 7fb816669740 1 -- 172.21.15.52:0/1694433472 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x55da09f45790 con 0x55da09f3b4d0 2024-05-27T07:33:56.646 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:33:56.650+0000 7fb816669740 1 -- 172.21.15.52:0/1694433472 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x55da0a21b890 con 0x55da09f5c1b0 2024-05-27T07:33:56.646 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:33:56.650+0000 7fb816669740 1 -- 172.21.15.52:0/1694433472 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x55da0a31b560 con 0x55da09f332f0 2024-05-27T07:33:56.646 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:33:56.651+0000 7fb816669740 10 client.0 _ll_register_callbacks cb 0x55da09f73a00 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T07:33:56.647 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:33:56.652+0000 7fb80dd4f640 1 -- 172.21.15.52:0/1694433472 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55da09f332f0 msgr2=0x55da0a307a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:33:56.647 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:33:56.652+0000 7fb80dd4f640 1 --2- 172.21.15.52:0/1694433472 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55da09f332f0 0x55da0a307a50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:33:56.647 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:33:56.652+0000 7fb80dd4f640 1 -- 172.21.15.52:0/1694433472 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55da09f3b4d0 msgr2=0x55da0a307f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:33:56.647 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:33:56.652+0000 7fb80dd4f640 1 --2- 172.21.15.52:0/1694433472 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55da09f3b4d0 0x55da0a307f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:33:56.647 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:33:56.652+0000 7fb80dd4f640 1 -- 172.21.15.52:0/1694433472 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55da0a040620 con 0x55da09f5c1b0 2024-05-27T07:33:56.647 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:33:56.652+0000 7fb80dd4f640 1 --2- 172.21.15.52:0/1694433472 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55da09f5c1b0 0x55da0a31e930 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7fb80003be20 tx=0x7fb800005fd0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=79720e2678378f72 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:33:56.648 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:33:56.653+0000 7fb7f0ff9640 10 client.0 ms_handle_connect on v2:172.21.15.105:3300/0 2024-05-27T07:33:56.648 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:33:56.653+0000 7fb7f0ff9640 1 -- 172.21.15.52:0/1694433472 <== mon.2 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb800045040 con 0x55da09f5c1b0 2024-05-27T07:33:56.648 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:33:56.653+0000 7fb816669740 1 -- 172.21.15.52:0/1694433472 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fb7f80af110 con 0x55da09f5c1b0 2024-05-27T07:33:56.649 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:33:56.653+0000 7fb7f0ff9640 1 -- 172.21.15.52:0/1694433472 <== mon.2 v2:172.21.15.105:3300/0 2 ==== osd_map(119..119 src has 1..119) ==== 9451+0+0 (secure 0 0 0) 0x7fb800019040 con 0x55da09f5c1b0 2024-05-27T07:33:56.649 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:33:56.654+0000 7fb7f0ff9640 1 -- 172.21.15.52:0/1694433472 <== mon.2 v2:172.21.15.105:3300/0 3 ==== mon_get_version_reply(handle=1 version=104) ==== 24+0+0 (secure 0 0 0) 0x7fb800004390 con 0x55da09f5c1b0 2024-05-27T07:33:56.649 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:33:56.654+0000 7fb816669740 10 client.26627 fetch_fsmap learned FSMap version 104 2024-05-27T07:33:56.649 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:33:56.654+0000 7fb816669740 1 -- 172.21.15.52:0/1694433472 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,fsmap=104,monmap=4+}) -- 0x55da0a327870 con 0x55da09f5c1b0 2024-05-27T07:33:56.649 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:33:56.654+0000 7fb7f0ff9640 1 -- 172.21.15.52:0/1694433472 <== mon.2 v2:172.21.15.105:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fb7f80af110 con 0x55da09f5c1b0 2024-05-27T07:33:56.649 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:33:56.655+0000 7fb7f0ff9640 1 -- 172.21.15.52:0/1694433472 <== mon.2 v2:172.21.15.105:3300/0 5 ==== fsmap(e 104) ==== 4211+0+0 (secure 0 0 0) 0x7fb800062550 con 0x55da09f5c1b0 2024-05-27T07:33:56.650 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:33:56.655+0000 7fb816669740 10 client.26627 fetch_fsmap finished waiting for FSMap version 104 2024-05-27T07:33:56.650 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:33:56.655+0000 7fb816669740 10 client.26627 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-05-27T07:33:56.650 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:33:56.655+0000 7fb816669740 20 client.26627 populate_metadata read hostname 'smithi052' 2024-05-27T07:33:56.650 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:33:56.655+0000 7fb816669740 1 --2- 172.21.15.52:0/1694433472 >> [v2:172.21.15.105:6838/500348151,v1:172.21.15.105:6839/500348151] conn(0x55da09f1ad80 0x55da0a32b2d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:33:56.650 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:33:56.655+0000 7fb816669740 4 client.26627 mds_command: new command op to 24527 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-27T07:33:56.650 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:33:56.655+0000 7fb816669740 1 -- 172.21.15.52:0/1694433472 --> [v2:172.21.15.105:6838/500348151,v1:172.21.15.105:6839/500348151] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55da0a271b80 con 0x55da09f1ad80 2024-05-27T07:33:56.651 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:33:56.657+0000 7fb80d54e640 1 --2- 172.21.15.52:0/1694433472 >> [v2:172.21.15.105:6838/500348151,v1:172.21.15.105:6839/500348151] conn(0x55da09f1ad80 0x55da0a32b2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:33:56.654 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:33:56.660+0000 7fb80d54e640 1 --2- 172.21.15.52:0/1694433472 >> [v2:172.21.15.105:6838/500348151,v1:172.21.15.105:6839/500348151] conn(0x55da09f1ad80 0x55da0a32b2d0 crc :-1 s=READY pgs=147 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=e245925a820457e5 server_cookie=5b5178cb165e7198 in_seq=0 out_seq=0 2024-05-27T07:33:56.654 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:33:56.660+0000 7fb7f0ff9640 10 client.26627 ms_handle_connect on v2:172.21.15.105:6838/500348151 2024-05-27T07:33:56.658 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:33:56.663+0000 7fb7f0ff9640 1 -- 172.21.15.52:0/1694433472 <== mds.0 v2:172.21.15.105:6838/500348151 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24041 (crc 0 0 0) 0x55da0a271b80 con 0x55da09f1ad80 2024-05-27T07:33:56.658 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:33:56.663+0000 7fb7f0ff9640 10 client.26627 handle_command_reply: tid=0 2024-05-27T07:33:56.658 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:33:56.663+0000 7fb816669740 1 client.26627 shutdown 2024-05-27T07:33:56.658 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:33:56.663+0000 7fb816669740 1 -- 172.21.15.52:0/1694433472 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55da09f5c1b0 msgr2=0x55da0a31e930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:33:56.658 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:33:56.663+0000 7fb816669740 1 --2- 172.21.15.52:0/1694433472 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55da09f5c1b0 0x55da0a31e930 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7fb80003be20 tx=0x7fb800005fd0 comp rx=0 tx=0).stop 2024-05-27T07:33:56.658 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:33:56.663+0000 7fb816669740 1 -- 172.21.15.52:0/1694433472 shutdown_connections 2024-05-27T07:33:56.658 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:33:56.663+0000 7fb816669740 1 --2- 172.21.15.52:0/1694433472 >> [v2:172.21.15.105:6838/500348151,v1:172.21.15.105:6839/500348151] conn(0x55da09f1ad80 0x55da0a32b2d0 crc :-1 s=READY pgs=147 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:33:56.658 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:33:56.663+0000 7fb816669740 1 --2- 172.21.15.52:0/1694433472 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55da09f5c1b0 0x55da0a31e930 unknown :-1 s=CLOSED pgs=227 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:33:56.659 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:33:56.663+0000 7fb816669740 1 --2- 172.21.15.52:0/1694433472 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55da09f3b4d0 0x55da0a307f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:33:56.659 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:33:56.663+0000 7fb7f0ff9640 0 client.26627 ms_handle_reset on v2:172.21.15.105:6838/500348151 2024-05-27T07:33:56.659 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:33:56.663+0000 7fb816669740 1 --2- 172.21.15.52:0/1694433472 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55da09f332f0 0x55da0a307a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:33:56.659 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:33:56.663+0000 7fb816669740 1 -- 172.21.15.52:0/1694433472 >> 172.21.15.52:0/1694433472 conn(0x55da09f50300 msgr2=0x55da0a2fd200 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:33:56.659 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:33:56.663+0000 7fb816669740 1 -- 172.21.15.52:0/1694433472 shutdown_connections 2024-05-27T07:33:56.659 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:33:56.663+0000 7fb816669740 1 -- 172.21.15.52:0/1694433472 wait complete. 2024-05-27T07:33:56.659 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:33:56.664+0000 7fb816669740 20 client.26627 trim_cache size 0 max 16384 2024-05-27T07:33:56.686 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:33:56.691+0000 7fb816669740 1 Processor -- start 2024-05-27T07:33:56.687 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:33:56.693+0000 7fb816669740 1 -- start start 2024-05-27T07:33:56.687 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:33:56.693+0000 7fb816669740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55da09f50300 0x55da0a2fd870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:33:56.687 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:33:56.693+0000 7fb816669740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55da09f1ad80 0x55da0a330ad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:33:56.688 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:33:56.693+0000 7fb816669740 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55da09f332f0 0x55da0a331010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:33:56.688 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:33:56.693+0000 7fb816669740 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x7fb7ec010560 con 0x55da09f332f0 2024-05-27T07:33:56.688 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:33:56.693+0000 7fb816669740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7fb7ec00e930 con 0x55da09f1ad80 2024-05-27T07:33:56.688 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:33:56.693+0000 7fb816669740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7fb7ec00fe60 con 0x55da09f50300 2024-05-27T07:33:56.689 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:33:56.693+0000 7fb80d54e640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55da09f50300 0x55da0a2fd870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:33:56.689 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:33:56.694+0000 7fb80d54e640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55da09f50300 0x55da0a2fd870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.173:3300/0 says I am v2:172.21.15.52:53264/0 (socket says 172.21.15.52:53264) 2024-05-27T07:33:56.689 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:33:56.694+0000 7fb80d54e640 1 -- 172.21.15.52:0/1876317549 learned_addr learned my addr 172.21.15.52:0/1876317549 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-05-27T07:33:56.689 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:33:56.694+0000 7fb80dd4f640 1 --2- 172.21.15.52:0/1876317549 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55da09f332f0 0x55da0a331010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:33:56.689 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:33:56.694+0000 7fb80cd4d640 1 --2- 172.21.15.52:0/1876317549 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55da09f1ad80 0x55da0a330ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:33:56.689 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:33:56.694+0000 7fb80d54e640 1 -- 172.21.15.52:0/1876317549 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55da09f1ad80 msgr2=0x55da0a330ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:33:56.689 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:33:56.694+0000 7fb80d54e640 1 --2- 172.21.15.52:0/1876317549 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55da09f1ad80 0x55da0a330ad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:33:56.690 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:33:56.694+0000 7fb80d54e640 1 -- 172.21.15.52:0/1876317549 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55da09f332f0 msgr2=0x55da0a331010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:33:56.690 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:33:56.694+0000 7fb80d54e640 1 --2- 172.21.15.52:0/1876317549 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55da09f332f0 0x55da0a331010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:33:56.690 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:33:56.694+0000 7fb80d54e640 1 -- 172.21.15.52:0/1876317549 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55da0a2fe4b0 con 0x55da09f50300 2024-05-27T07:33:56.690 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:33:56.694+0000 7fb80d54e640 1 --2- 172.21.15.52:0/1876317549 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55da09f50300 0x55da0a2fd870 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7fb7f8004fd0 tx=0x7fb7f80aac60 comp rx=0 tx=0).ready entity=mon.1 client_cookie=a4cb71033da5ea18 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:33:56.690 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:33:56.695+0000 7fb7f2ffd640 1 -- 172.21.15.52:0/1876317549 <== mon.1 v2:172.21.15.173:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb7f80c9020 con 0x55da09f50300 2024-05-27T07:33:56.690 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:33:56.695+0000 7fb7f2ffd640 1 -- 172.21.15.52:0/1876317549 <== mon.1 v2:172.21.15.173:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fb7f80af110 con 0x55da09f50300 2024-05-27T07:33:56.690 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:33:56.695+0000 7fb7f2ffd640 1 -- 172.21.15.52:0/1876317549 <== mon.1 v2:172.21.15.173:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb7f80aa1c0 con 0x55da09f50300 2024-05-27T07:33:56.694 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:33:56.696+0000 7fb816669740 1 -- 172.21.15.52:0/1876317549 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55da09f50300 msgr2=0x55da0a2fd870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:33:56.694 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:33:56.696+0000 7fb816669740 1 --2- 172.21.15.52:0/1876317549 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55da09f50300 0x55da0a2fd870 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7fb7f8004fd0 tx=0x7fb7f80aac60 comp rx=0 tx=0).stop 2024-05-27T07:33:56.694 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:33:56.696+0000 7fb816669740 1 -- 172.21.15.52:0/1876317549 shutdown_connections 2024-05-27T07:33:56.694 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:33:56.696+0000 7fb816669740 1 --2- 172.21.15.52:0/1876317549 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55da09f332f0 0x55da0a331010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:33:56.694 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:33:56.696+0000 7fb816669740 1 --2- 172.21.15.52:0/1876317549 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55da09f1ad80 0x55da0a330ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:33:56.694 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:33:56.696+0000 7fb816669740 1 --2- 172.21.15.52:0/1876317549 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55da09f50300 0x55da0a2fd870 unknown :-1 s=CLOSED pgs=182 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:33:56.694 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:33:56.696+0000 7fb816669740 1 -- 172.21.15.52:0/1876317549 >> 172.21.15.52:0/1876317549 conn(0x55da0a3031c0 msgr2=0x55da0a2fd160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:33:56.694 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:33:56.696+0000 7fb816669740 1 -- 172.21.15.52:0/1876317549 shutdown_connections 2024-05-27T07:33:56.694 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:33:56.696+0000 7fb816669740 1 -- 172.21.15.52:0/1876317549 wait complete. 2024-05-27T07:33:56.695 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:33:56.697+0000 7fb816669740 1 -- start start 2024-05-27T07:33:56.695 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:33:56.697+0000 7fb816669740 1 Processor -- start 2024-05-27T07:33:56.695 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:33:56.697+0000 7fb816669740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55da09f1ad80 0x55da0a324de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:33:56.695 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:33:56.697+0000 7fb816669740 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55da09f332f0 0x55da0a325320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:33:56.695 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:33:56.697+0000 7fb816669740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55da09f50300 0x55da0a320250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:33:56.695 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:33:56.697+0000 7fb816669740 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x7fb7ec00f700 con 0x55da09f332f0 2024-05-27T07:33:56.695 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:33:56.698+0000 7fb816669740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7fb7ec00f000 con 0x55da09f1ad80 2024-05-27T07:33:56.695 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:33:56.698+0000 7fb816669740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x55da0a308320 con 0x55da09f50300 2024-05-27T07:33:56.695 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:33:56.698+0000 7fb816669740 10 client.0 _ll_register_callbacks cb 0x7fb80001ab30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T07:33:56.695 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:33:56.698+0000 7fb80d54e640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55da09f1ad80 0x55da0a324de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:33:56.695 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:33:56.698+0000 7fb80d54e640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55da09f1ad80 0x55da0a324de0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.52:37218/0 (socket says 172.21.15.52:37218) 2024-05-27T07:33:56.695 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:33:56.698+0000 7fb80d54e640 1 -- 172.21.15.52:0/3524298557 learned_addr learned my addr 172.21.15.52:0/3524298557 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-05-27T07:33:56.695 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:33:56.698+0000 7fb80dd4f640 1 --2- 172.21.15.52:0/3524298557 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55da09f50300 0x55da0a320250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:33:56.696 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:33:56.698+0000 7fb80d54e640 1 -- 172.21.15.52:0/3524298557 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55da09f50300 msgr2=0x55da0a320250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:33:56.696 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:33:56.698+0000 7fb80d54e640 1 --2- 172.21.15.52:0/3524298557 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55da09f50300 0x55da0a320250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:33:56.696 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:33:56.699+0000 7fb80d54e640 1 -- 172.21.15.52:0/3524298557 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55da09f332f0 msgr2=0x55da0a325320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:33:56.696 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:33:56.699+0000 7fb80d54e640 1 --2- 172.21.15.52:0/3524298557 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55da09f332f0 0x55da0a325320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:33:56.696 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:33:56.699+0000 7fb80d54e640 1 -- 172.21.15.52:0/3524298557 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55da0a320910 con 0x55da09f1ad80 2024-05-27T07:33:56.696 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:33:56.699+0000 7fb80dd4f640 1 --2- 172.21.15.52:0/3524298557 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55da09f50300 0x55da0a320250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-27T07:33:56.696 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:33:56.699+0000 7fb80d54e640 1 --2- 172.21.15.52:0/3524298557 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55da09f1ad80 0x55da0a324de0 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7fb7f80b3c60 tx=0x7fb7f80aae80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=df5f4b8655376ce server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:33:56.698 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:33:56.701+0000 7fb7f0ff9640 10 client.0 ms_handle_connect on v2:172.21.15.105:3300/0 2024-05-27T07:33:56.698 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:33:56.701+0000 7fb7f0ff9640 1 -- 172.21.15.52:0/3524298557 <== mon.2 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb7f80c9040 con 0x55da09f1ad80 2024-05-27T07:33:56.698 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:33:56.701+0000 7fb7f0ff9640 1 -- 172.21.15.52:0/3524298557 <== mon.2 v2:172.21.15.105:3300/0 2 ==== osd_map(119..119 src has 1..119) ==== 9451+0+0 (secure 0 0 0) 0x7fb7f8005220 con 0x55da09f1ad80 2024-05-27T07:33:56.698 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:33:56.701+0000 7fb816669740 1 -- 172.21.15.52:0/3524298557 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fb7f80af110 con 0x55da09f1ad80 2024-05-27T07:33:56.698 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:33:56.702+0000 7fb7f0ff9640 1 -- 172.21.15.52:0/3524298557 <== mon.2 v2:172.21.15.105:3300/0 3 ==== mon_get_version_reply(handle=1 version=104) ==== 24+0+0 (secure 0 0 0) 0x7fb7f80a9fc0 con 0x55da09f1ad80 2024-05-27T07:33:56.698 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:33:56.702+0000 7fb816669740 10 client.26633 fetch_fsmap learned FSMap version 104 2024-05-27T07:33:56.698 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:33:56.702+0000 7fb816669740 1 -- 172.21.15.52:0/3524298557 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,fsmap=104,monmap=4+}) -- 0x55da0a3b5300 con 0x55da09f1ad80 2024-05-27T07:33:56.698 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:33:56.702+0000 7fb7f0ff9640 1 -- 172.21.15.52:0/3524298557 <== mon.2 v2:172.21.15.105:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fb7f80af110 con 0x55da09f1ad80 2024-05-27T07:33:56.698 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:33:56.703+0000 7fb7f0ff9640 1 -- 172.21.15.52:0/3524298557 <== mon.2 v2:172.21.15.105:3300/0 5 ==== fsmap(e 104) ==== 4211+0+0 (secure 0 0 0) 0x7fb7f80a9a30 con 0x55da09f1ad80 2024-05-27T07:33:56.698 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:33:56.703+0000 7fb816669740 10 client.26633 fetch_fsmap finished waiting for FSMap version 104 2024-05-27T07:33:56.699 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:33:56.703+0000 7fb816669740 10 client.26633 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-05-27T07:33:56.699 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:33:56.703+0000 7fb816669740 20 client.26633 populate_metadata read hostname 'smithi052' 2024-05-27T07:33:56.699 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:33:56.703+0000 7fb816669740 1 --2- 172.21.15.52:0/3524298557 >> [v2:172.21.15.105:6838/500348151,v1:172.21.15.105:6839/500348151] conn(0x55da09f3b4d0 0x55da0a3b7860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:33:56.699 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:33:56.703+0000 7fb816669740 4 client.26633 mds_command: new command op to 24527 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-05-27T07:33:56.699 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:33:56.703+0000 7fb816669740 1 -- 172.21.15.52:0/3524298557 --> [v2:172.21.15.105:6838/500348151,v1:172.21.15.105:6839/500348151] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55da0a26b360 con 0x55da09f3b4d0 2024-05-27T07:33:56.699 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:33:56.703+0000 7fb80cd4d640 1 --2- 172.21.15.52:0/3524298557 >> [v2:172.21.15.105:6838/500348151,v1:172.21.15.105:6839/500348151] conn(0x55da09f3b4d0 0x55da0a3b7860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:33:56.699 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:33:56.704+0000 7fb80cd4d640 1 --2- 172.21.15.52:0/3524298557 >> [v2:172.21.15.105:6838/500348151,v1:172.21.15.105:6839/500348151] conn(0x55da09f3b4d0 0x55da0a3b7860 crc :-1 s=READY pgs=148 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=bed134b4b674ac3e server_cookie=55201c9489e99e57 in_seq=0 out_seq=0 2024-05-27T07:33:56.699 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:33:56.704+0000 7fb7f0ff9640 10 client.26633 ms_handle_connect on v2:172.21.15.105:6838/500348151 2024-05-27T07:33:56.736 INFO:teuthology.orchestra.run.smithi052.stderr:Inferring config /var/lib/ceph/73df2636-1bf8-11ef-bc9b-c7b262605968/mon.a/config 2024-05-27T07:33:57.484 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:33:57 smithi052 ceph-mon[31955]: scrub summary: active paths [/] 2024-05-27T07:33:57.484 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:33:57 smithi052 ceph-mon[31955]: Scrub error on inode 0x100000055d6 (/volumes/qa/sv_1/95e9ac73-825f-48e3-998b-133d94d96559/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-78) see mds.b log and `damage ls` output for details 2024-05-27T07:33:57.484 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:33:57 smithi052 ceph-mon[31955]: scrub summary: active paths [/] 2024-05-27T07:33:57.485 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:33:57 smithi052 ceph-mon[31955]: scrub summary: idle+waiting paths [/] 2024-05-27T07:33:57.485 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:33:57 smithi052 ceph-mon[31955]: scrub summary: idle+waiting paths [/] 2024-05-27T07:33:57.485 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:33:57 smithi052 ceph-mon[31955]: scrub summary: active paths [/] 2024-05-27T07:33:57.485 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:33:57 smithi052 ceph-mon[31955]: scrub summary: active paths [/] 2024-05-27T07:33:57.485 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:33:57 smithi052 ceph-mon[31955]: scrub summary: idle+waiting paths [/] 2024-05-27T07:33:57.485 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:33:57 smithi052 ceph-mon[31955]: scrub summary: idle+waiting paths [/] 2024-05-27T07:33:57.485 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:33:57 smithi052 ceph-mon[31955]: Scrub error on inode 0x10000000c2a (/volumes/qa/sv_1/95e9ac73-825f-48e3-998b-133d94d96559/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-14) see mds.b log and `damage ls` output for details 2024-05-27T07:33:57.485 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:33:57 smithi052 ceph-mon[31955]: scrub summary: active paths [/] 2024-05-27T07:33:57.485 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:33:57 smithi052 ceph-mon[31955]: scrub summary: active paths [/] 2024-05-27T07:33:57.485 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:33:57 smithi052 ceph-mon[31955]: scrub summary: idle+waiting paths [/] 2024-05-27T07:33:57.485 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:33:57 smithi052 ceph-mon[31955]: scrub summary: idle+waiting paths [/] 2024-05-27T07:33:57.485 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:33:57 smithi052 ceph-mon[31955]: Scrub error on inode 0x1000000819b (/volumes/qa/sv_1/95e9ac73-825f-48e3-998b-133d94d96559/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-127) see mds.b log and `damage ls` output for details 2024-05-27T07:33:57.485 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:33:57 smithi052 ceph-mon[31955]: Scrub error on inode 0x10000009421 (/volumes/qa/sv_1/95e9ac73-825f-48e3-998b-133d94d96559/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-141) see mds.b log and `damage ls` output for details 2024-05-27T07:33:57.486 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:33:57 smithi052 ceph-mon[31955]: scrub summary: active paths [/] 2024-05-27T07:33:57.486 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:33:57 smithi052 ceph-mon[31955]: scrub summary: active paths [/] 2024-05-27T07:33:57.486 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:33:57 smithi052 ceph-mon[31955]: scrub summary: idle+waiting paths [/] 2024-05-27T07:33:57.486 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:33:57 smithi052 ceph-mon[31955]: Scrub error on inode 0x10000003fad (/volumes/qa/sv_1/95e9ac73-825f-48e3-998b-133d94d96559/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-58) see mds.b log and `damage ls` output for details 2024-05-27T07:33:57.486 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:33:57 smithi052 ceph-mon[31955]: scrub summary: active paths [/] 2024-05-27T07:33:57.486 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:33:57 smithi052 ceph-mon[31955]: Scrub error on inode 0x1000000918d (/volumes/qa/sv_1/95e9ac73-825f-48e3-998b-133d94d96559/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-139) see mds.b log and `damage ls` output for details 2024-05-27T07:33:57.486 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:33:57 smithi052 ceph-mon[31955]: Scrub error on inode 0x100000089f8 (/volumes/qa/sv_1/95e9ac73-825f-48e3-998b-133d94d96559/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-134) see mds.b log and `damage ls` output for details 2024-05-27T07:33:57.486 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:33:57 smithi052 ceph-mon[31955]: scrub summary: idle+waiting paths [/] 2024-05-27T07:33:57.486 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:33:57 smithi052 ceph-mon[31955]: scrub summary: idle+waiting paths [/] 2024-05-27T07:33:57.567 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:33:57.571+0000 7fb7f0ff9640 1 -- 172.21.15.52:0/3524298557 <== mds.0 v2:172.21.15.105:6838/500348151 1 ==== command_reply(tid 0: 0 ) ==== 8+0+25 (crc 0 0 0) 0x55da0a26b360 con 0x55da09f3b4d0 2024-05-27T07:33:57.567 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:33:57.571+0000 7fb7f0ff9640 10 client.26633 handle_command_reply: tid=0 2024-05-27T07:33:57.567 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:33:57.571+0000 7fb816669740 1 client.26633 shutdown 2024-05-27T07:33:57.567 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:33:57.571+0000 7fb816669740 1 -- 172.21.15.52:0/3524298557 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55da09f1ad80 msgr2=0x55da0a324de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:33:57.567 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:33:57.571+0000 7fb816669740 1 --2- 172.21.15.52:0/3524298557 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55da09f1ad80 0x55da0a324de0 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7fb7f80b3c60 tx=0x7fb7f80aae80 comp rx=0 tx=0).stop 2024-05-27T07:33:57.567 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:33:57.571+0000 7fb816669740 1 -- 172.21.15.52:0/3524298557 shutdown_connections 2024-05-27T07:33:57.568 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:33:57.571+0000 7fb816669740 1 --2- 172.21.15.52:0/3524298557 >> [v2:172.21.15.105:6838/500348151,v1:172.21.15.105:6839/500348151] conn(0x55da09f3b4d0 0x55da0a3b7860 crc :-1 s=READY pgs=148 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:33:57.568 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:33:57.571+0000 7fb816669740 1 --2- 172.21.15.52:0/3524298557 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55da09f50300 0x55da0a320250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:33:57.568 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:33:57.571+0000 7fb7f0ff9640 0 client.26633 ms_handle_reset on v2:172.21.15.105:6838/500348151 2024-05-27T07:33:57.568 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:33:57.571+0000 7fb816669740 1 --2- 172.21.15.52:0/3524298557 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55da09f332f0 0x55da0a325320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:33:57.568 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:33:57.571+0000 7fb816669740 1 --2- 172.21.15.52:0/3524298557 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55da09f1ad80 0x55da0a324de0 unknown :-1 s=CLOSED pgs=228 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:33:57.568 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:33:57.572+0000 7fb816669740 1 -- 172.21.15.52:0/3524298557 >> 172.21.15.52:0/3524298557 conn(0x55da0a3031c0 msgr2=0x55da0a308d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:33:57.568 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:33:57.572+0000 7fb816669740 1 -- 172.21.15.52:0/3524298557 shutdown_connections 2024-05-27T07:33:57.568 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:33:57.572+0000 7fb816669740 1 -- 172.21.15.52:0/3524298557 wait complete. 2024-05-27T07:33:57.568 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:33:57.572+0000 7fb816669740 20 client.26633 trim_cache size 0 max 16384 2024-05-27T07:33:57.569 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stdout:{ 2024-05-27T07:33:57.569 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stdout: "return_code": 0 2024-05-27T07:33:57.569 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stdout:} 2024-05-27T07:33:57.570 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:33:57.572+0000 7fb80f7b2640 1 -- 172.21.15.52:0/625699404 >> [v2:172.21.15.52:6800/1822318384,v1:172.21.15.52:6801/1822318384] conn(0x7fb7f40b2a50 msgr2=0x7fb7f40b4f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:33:57.570 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:33:57.572+0000 7fb80f7b2640 1 --2- 172.21.15.52:0/625699404 >> [v2:172.21.15.52:6800/1822318384,v1:172.21.15.52:6801/1822318384] conn(0x7fb7f40b2a50 0x7fb7f40b4f60 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7fb80001a8e0 tx=0x7fb80003b900 comp rx=0 tx=0).stop 2024-05-27T07:33:57.570 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:33:57.572+0000 7fb80f7b2640 1 -- 172.21.15.52:0/625699404 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fb808104410 msgr2=0x7fb808110300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:33:57.570 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:33:57.572+0000 7fb80f7b2640 1 --2- 172.21.15.52:0/625699404 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fb808104410 0x7fb808110300 secure :-1 s=READY pgs=710 cs=0 l=1 rev1=1 crypto rx=0x7fb7f800e9c0 tx=0x7fb7f800ee80 comp rx=0 tx=0).stop 2024-05-27T07:33:57.570 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:33:57.572+0000 7fb80f7b2640 1 -- 172.21.15.52:0/625699404 shutdown_connections 2024-05-27T07:33:57.570 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:33:57.572+0000 7fb80f7b2640 1 --2- 172.21.15.52:0/625699404 >> [v2:172.21.15.52:6800/1822318384,v1:172.21.15.52:6801/1822318384] conn(0x7fb7f40b2a50 0x7fb7f40b4f60 unknown :-1 s=CLOSED pgs=171 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:33:57.570 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:33:57.572+0000 7fb80f7b2640 1 --2- 172.21.15.52:0/625699404 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fb80810a160 0x7fb80810ac00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:33:57.570 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:33:57.572+0000 7fb80f7b2640 1 --2- 172.21.15.52:0/625699404 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fb808109cc0 0x7fb80810a6c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:33:57.570 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:33:57.572+0000 7fb80f7b2640 1 --2- 172.21.15.52:0/625699404 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fb808104410 0x7fb808110300 unknown :-1 s=CLOSED pgs=710 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:33:57.570 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:33:57.572+0000 7fb80f7b2640 1 -- 172.21.15.52:0/625699404 >> 172.21.15.52:0/625699404 conn(0x7fb8080793c0 msgr2=0x7fb808102af0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:33:57.570 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:33:57.573+0000 7fb80f7b2640 1 -- 172.21.15.52:0/625699404 shutdown_connections 2024-05-27T07:33:57.570 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:33:57.573+0000 7fb80f7b2640 1 -- 172.21.15.52:0/625699404 wait complete. 2024-05-27T07:33:57.606 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:33:57 smithi173 ceph-mon[37000]: scrub summary: active paths [/] 2024-05-27T07:33:57.607 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:33:57 smithi173 ceph-mon[37000]: Scrub error on inode 0x100000055d6 (/volumes/qa/sv_1/95e9ac73-825f-48e3-998b-133d94d96559/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-78) see mds.b log and `damage ls` output for details 2024-05-27T07:33:57.607 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:33:57 smithi173 ceph-mon[37000]: scrub summary: active paths [/] 2024-05-27T07:33:57.607 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:33:57 smithi173 ceph-mon[37000]: scrub summary: idle+waiting paths [/] 2024-05-27T07:33:57.607 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:33:57 smithi173 ceph-mon[37000]: scrub summary: idle+waiting paths [/] 2024-05-27T07:33:57.607 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:33:57 smithi173 ceph-mon[37000]: scrub summary: active paths [/] 2024-05-27T07:33:57.607 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:33:57 smithi173 ceph-mon[37000]: scrub summary: active paths [/] 2024-05-27T07:33:57.607 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:33:57 smithi173 ceph-mon[37000]: scrub summary: idle+waiting paths [/] 2024-05-27T07:33:57.607 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:33:57 smithi173 ceph-mon[37000]: scrub summary: idle+waiting paths [/] 2024-05-27T07:33:57.607 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:33:57 smithi173 ceph-mon[37000]: Scrub error on inode 0x10000000c2a (/volumes/qa/sv_1/95e9ac73-825f-48e3-998b-133d94d96559/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-14) see mds.b log and `damage ls` output for details 2024-05-27T07:33:57.607 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:33:57 smithi173 ceph-mon[37000]: scrub summary: active paths [/] 2024-05-27T07:33:57.607 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:33:57 smithi173 ceph-mon[37000]: scrub summary: active paths [/] 2024-05-27T07:33:57.607 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:33:57 smithi173 ceph-mon[37000]: scrub summary: idle+waiting paths [/] 2024-05-27T07:33:57.608 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:33:57 smithi173 ceph-mon[37000]: scrub summary: idle+waiting paths [/] 2024-05-27T07:33:57.608 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:33:57 smithi173 ceph-mon[37000]: Scrub error on inode 0x1000000819b (/volumes/qa/sv_1/95e9ac73-825f-48e3-998b-133d94d96559/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-127) see mds.b log and `damage ls` output for details 2024-05-27T07:33:57.608 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:33:57 smithi173 ceph-mon[37000]: Scrub error on inode 0x10000009421 (/volumes/qa/sv_1/95e9ac73-825f-48e3-998b-133d94d96559/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-141) see mds.b log and `damage ls` output for details 2024-05-27T07:33:57.608 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:33:57 smithi173 ceph-mon[37000]: scrub summary: active paths [/] 2024-05-27T07:33:57.608 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:33:57 smithi173 ceph-mon[37000]: scrub summary: active paths [/] 2024-05-27T07:33:57.608 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:33:57 smithi173 ceph-mon[37000]: scrub summary: idle+waiting paths [/] 2024-05-27T07:33:57.608 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:33:57 smithi173 ceph-mon[37000]: Scrub error on inode 0x10000003fad (/volumes/qa/sv_1/95e9ac73-825f-48e3-998b-133d94d96559/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-58) see mds.b log and `damage ls` output for details 2024-05-27T07:33:57.608 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:33:57 smithi173 ceph-mon[37000]: scrub summary: active paths [/] 2024-05-27T07:33:57.608 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:33:57 smithi173 ceph-mon[37000]: Scrub error on inode 0x1000000918d (/volumes/qa/sv_1/95e9ac73-825f-48e3-998b-133d94d96559/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-139) see mds.b log and `damage ls` output for details 2024-05-27T07:33:57.608 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:33:57 smithi173 ceph-mon[37000]: Scrub error on inode 0x100000089f8 (/volumes/qa/sv_1/95e9ac73-825f-48e3-998b-133d94d96559/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-134) see mds.b log and `damage ls` output for details 2024-05-27T07:33:57.608 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:33:57 smithi173 ceph-mon[37000]: scrub summary: idle+waiting paths [/] 2024-05-27T07:33:57.608 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:33:57 smithi173 ceph-mon[37000]: scrub summary: idle+waiting paths [/] 2024-05-27T07:33:57.694 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:33:57 smithi105 ceph-mon[35977]: scrub summary: active paths [/] 2024-05-27T07:33:57.695 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:33:57 smithi105 ceph-mon[35977]: Scrub error on inode 0x100000055d6 (/volumes/qa/sv_1/95e9ac73-825f-48e3-998b-133d94d96559/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-78) see mds.b log and `damage ls` output for details 2024-05-27T07:33:57.695 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:33:57 smithi105 ceph-mon[35977]: scrub summary: active paths [/] 2024-05-27T07:33:57.695 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:33:57 smithi105 ceph-mon[35977]: scrub summary: idle+waiting paths [/] 2024-05-27T07:33:57.695 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:33:57 smithi105 ceph-mon[35977]: scrub summary: idle+waiting paths [/] 2024-05-27T07:33:57.695 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:33:57 smithi105 ceph-mon[35977]: scrub summary: active paths [/] 2024-05-27T07:33:57.695 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:33:57 smithi105 ceph-mon[35977]: scrub summary: active paths [/] 2024-05-27T07:33:57.695 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:33:57 smithi105 ceph-mon[35977]: scrub summary: idle+waiting paths [/] 2024-05-27T07:33:57.695 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:33:57 smithi105 ceph-mon[35977]: scrub summary: idle+waiting paths [/] 2024-05-27T07:33:57.695 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:33:57 smithi105 ceph-mon[35977]: Scrub error on inode 0x10000000c2a (/volumes/qa/sv_1/95e9ac73-825f-48e3-998b-133d94d96559/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-14) see mds.b log and `damage ls` output for details 2024-05-27T07:33:57.695 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:33:57 smithi105 ceph-mon[35977]: scrub summary: active paths [/] 2024-05-27T07:33:57.696 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:33:57 smithi105 ceph-mon[35977]: scrub summary: active paths [/] 2024-05-27T07:33:57.696 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:33:57 smithi105 ceph-mon[35977]: scrub summary: idle+waiting paths [/] 2024-05-27T07:33:57.696 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:33:57 smithi105 ceph-mon[35977]: scrub summary: idle+waiting paths [/] 2024-05-27T07:33:57.696 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:33:57 smithi105 ceph-mon[35977]: Scrub error on inode 0x1000000819b (/volumes/qa/sv_1/95e9ac73-825f-48e3-998b-133d94d96559/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-127) see mds.b log and `damage ls` output for details 2024-05-27T07:33:57.696 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:33:57 smithi105 ceph-mon[35977]: Scrub error on inode 0x10000009421 (/volumes/qa/sv_1/95e9ac73-825f-48e3-998b-133d94d96559/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-141) see mds.b log and `damage ls` output for details 2024-05-27T07:33:57.696 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:33:57 smithi105 ceph-mon[35977]: scrub summary: active paths [/] 2024-05-27T07:33:57.696 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:33:57 smithi105 ceph-mon[35977]: scrub summary: active paths [/] 2024-05-27T07:33:57.696 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:33:57 smithi105 ceph-mon[35977]: scrub summary: idle+waiting paths [/] 2024-05-27T07:33:57.696 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:33:57 smithi105 ceph-mon[35977]: Scrub error on inode 0x10000003fad (/volumes/qa/sv_1/95e9ac73-825f-48e3-998b-133d94d96559/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-58) see mds.b log and `damage ls` output for details 2024-05-27T07:33:57.696 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:33:57 smithi105 ceph-mon[35977]: scrub summary: active paths [/] 2024-05-27T07:33:57.696 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:33:57 smithi105 ceph-mon[35977]: Scrub error on inode 0x1000000918d (/volumes/qa/sv_1/95e9ac73-825f-48e3-998b-133d94d96559/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-139) see mds.b log and `damage ls` output for details 2024-05-27T07:33:57.696 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:33:57 smithi105 ceph-mon[35977]: Scrub error on inode 0x100000089f8 (/volumes/qa/sv_1/95e9ac73-825f-48e3-998b-133d94d96559/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-134) see mds.b log and `damage ls` output for details 2024-05-27T07:33:57.696 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:33:57 smithi105 ceph-mon[35977]: scrub summary: idle+waiting paths [/] 2024-05-27T07:33:57.696 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:33:57 smithi105 ceph-mon[35977]: scrub summary: idle+waiting paths [/] 2024-05-27T07:33:58.542 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:33:58 smithi052 ceph-mon[31955]: pgmap v271: 129 pgs: 129 active+clean; 669 MiB data, 4.7 GiB used, 1.0 TiB / 1.0 TiB avail; 35 KiB/s rd, 1.7 MiB/s wr, 554 op/s 2024-05-27T07:33:58.606 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:33:58 smithi173 ceph-mon[37000]: pgmap v271: 129 pgs: 129 active+clean; 669 MiB data, 4.7 GiB used, 1.0 TiB / 1.0 TiB avail; 35 KiB/s rd, 1.7 MiB/s wr, 554 op/s 2024-05-27T07:33:58.694 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:33:58 smithi105 ceph-mon[35977]: pgmap v271: 129 pgs: 129 active+clean; 669 MiB data, 4.7 GiB used, 1.0 TiB / 1.0 TiB avail; 35 KiB/s rd, 1.7 MiB/s wr, 554 op/s 2024-05-27T07:33:58.800 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:58.806+0000 7fb6c87d6700 1 -- 172.21.15.52:0/241023090 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fb6c0108730 msgr2=0x7fb6c0110090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:33:58.800 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:58.806+0000 7fb6c87d6700 1 --2- 172.21.15.52:0/241023090 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fb6c0108730 0x7fb6c0110090 secure :-1 s=READY pgs=711 cs=0 l=1 rev1=1 crypto rx=0x7fb6b800b0d0 tx=0x7fb6b800b3e0 comp rx=0 tx=0).stop 2024-05-27T07:33:58.800 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:58.806+0000 7fb6c87d6700 1 -- 172.21.15.52:0/241023090 shutdown_connections 2024-05-27T07:33:58.800 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:58.806+0000 7fb6c87d6700 1 --2- 172.21.15.52:0/241023090 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fb6c0108730 0x7fb6c0110090 unknown :-1 s=CLOSED pgs=711 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:33:58.801 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:58.806+0000 7fb6c87d6700 1 --2- 172.21.15.52:0/241023090 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fb6c01034b0 0x7fb6c01038c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:33:58.801 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:58.806+0000 7fb6c87d6700 1 --2- 172.21.15.52:0/241023090 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fb6c0107de0 0x7fb6c01081f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:33:58.801 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:58.806+0000 7fb6c87d6700 1 -- 172.21.15.52:0/241023090 >> 172.21.15.52:0/241023090 conn(0x7fb6c00fedb0 msgr2=0x7fb6c01011d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:33:58.801 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:58.806+0000 7fb6c87d6700 1 -- 172.21.15.52:0/241023090 shutdown_connections 2024-05-27T07:33:58.801 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:58.806+0000 7fb6c87d6700 1 -- 172.21.15.52:0/241023090 wait complete. 2024-05-27T07:33:58.803 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:58.806+0000 7fb6c87d6700 1 Processor -- start 2024-05-27T07:33:58.803 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:58.807+0000 7fb6c87d6700 1 -- start start 2024-05-27T07:33:58.803 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:58.807+0000 7fb6c87d6700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fb6c01034b0 0x7fb6c00ff290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:33:58.803 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:58.807+0000 7fb6c87d6700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fb6c0107de0 0x7fb6c00ff7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:33:58.803 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:58.807+0000 7fb6c87d6700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fb6c0108730 0x7fb6c00ffd10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:33:58.803 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:58.807+0000 7fb6c87d6700 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x7fb6c0113360 con 0x7fb6c0108730 2024-05-27T07:33:58.803 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:58.807+0000 7fb6c87d6700 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7fb6c01131e0 con 0x7fb6c01034b0 2024-05-27T07:33:58.803 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:58.807+0000 7fb6c87d6700 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7fb6c01134e0 con 0x7fb6c0107de0 2024-05-27T07:33:58.803 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:58.807+0000 7fb6c6572700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fb6c01034b0 0x7fb6c00ff290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:33:58.803 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:58.807+0000 7fb6c6572700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fb6c01034b0 0x7fb6c00ff290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.52:37230/0 (socket says 172.21.15.52:37230) 2024-05-27T07:33:58.803 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:58.807+0000 7fb6c6d73700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fb6c0108730 0x7fb6c00ffd10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:33:58.804 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:58.807+0000 7fb6c6572700 1 -- 172.21.15.52:0/2183860932 learned_addr learned my addr 172.21.15.52:0/2183860932 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-05-27T07:33:58.804 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:58.807+0000 7fb6c5d71700 1 --2- 172.21.15.52:0/2183860932 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fb6c0107de0 0x7fb6c00ff7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:33:58.804 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:58.807+0000 7fb6c6572700 1 -- 172.21.15.52:0/2183860932 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fb6c0107de0 msgr2=0x7fb6c00ff7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:33:58.804 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:58.807+0000 7fb6c6572700 1 --2- 172.21.15.52:0/2183860932 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fb6c0107de0 0x7fb6c00ff7d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:33:58.804 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:58.807+0000 7fb6c6572700 1 -- 172.21.15.52:0/2183860932 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fb6c0108730 msgr2=0x7fb6c00ffd10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:33:58.804 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:58.807+0000 7fb6c6572700 1 --2- 172.21.15.52:0/2183860932 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fb6c0108730 0x7fb6c00ffd10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:33:58.804 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:58.807+0000 7fb6c6572700 1 -- 172.21.15.52:0/2183860932 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb6c01a90e0 con 0x7fb6c01034b0 2024-05-27T07:33:58.804 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:58.807+0000 7fb6c6d73700 1 --2- 172.21.15.52:0/2183860932 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fb6c0108730 0x7fb6c00ffd10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-27T07:33:58.804 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:58.808+0000 7fb6c6572700 1 --2- 172.21.15.52:0/2183860932 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fb6c01034b0 0x7fb6c00ff290 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7fb6bc00e9a0 tx=0x7fb6bc00ecb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:33:58.804 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:58.808+0000 7fb6b77fe700 1 -- 172.21.15.52:0/2183860932 <== mon.2 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb6bc00ca10 con 0x7fb6c01034b0 2024-05-27T07:33:58.804 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:58.808+0000 7fb6b77fe700 1 -- 172.21.15.52:0/2183860932 <== mon.2 v2:172.21.15.105:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fb6bc00cbb0 con 0x7fb6c01034b0 2024-05-27T07:33:58.804 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:58.808+0000 7fb6c87d6700 1 -- 172.21.15.52:0/2183860932 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb6c01a93d0 con 0x7fb6c01034b0 2024-05-27T07:33:58.804 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:58.808+0000 7fb6b77fe700 1 -- 172.21.15.52:0/2183860932 <== mon.2 v2:172.21.15.105:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb6bc018680 con 0x7fb6c01034b0 2024-05-27T07:33:58.805 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:58.808+0000 7fb6c87d6700 1 -- 172.21.15.52:0/2183860932 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb6c01a9960 con 0x7fb6c01034b0 2024-05-27T07:33:58.805 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:58.811+0000 7fb6b77fe700 1 -- 172.21.15.52:0/2183860932 <== mon.2 v2:172.21.15.105:3300/0 4 ==== mgrmap(e 26) ==== 149783+0+0 (secure 0 0 0) 0x7fb6bc010430 con 0x7fb6c01034b0 2024-05-27T07:33:58.806 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:58.812+0000 7fb6b77fe700 1 --2- 172.21.15.52:0/2183860932 >> [v2:172.21.15.52:6800/1822318384,v1:172.21.15.52:6801/1822318384] conn(0x7fb6ac0b2a70 0x7fb6ac0b4f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:33:58.806 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:58.812+0000 7fb6d71ebb80 1 Processor -- start 2024-05-27T07:33:58.806 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:58.812+0000 7fb6c5d71700 1 --2- 172.21.15.52:0/2183860932 >> [v2:172.21.15.52:6800/1822318384,v1:172.21.15.52:6801/1822318384] conn(0x7fb6ac0b2a70 0x7fb6ac0b4f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:33:58.806 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:58.812+0000 7fb6d71ebb80 1 -- start start 2024-05-27T07:33:58.806 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:58.812+0000 7fb6b77fe700 1 -- 172.21.15.52:0/2183860932 <== mon.2 v2:172.21.15.105:3300/0 5 ==== osd_map(119..119 src has 1..119) ==== 9451+0+0 (secure 0 0 0) 0x7fb6bc0e2ad0 con 0x7fb6c01034b0 2024-05-27T07:33:58.808 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:58.812+0000 7fb6d71ebb80 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5575a129ae40 0x5575a141e710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:33:58.808 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:58.812+0000 7fb6d71ebb80 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x5575a141ec50 0x5575a1418a40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:33:58.808 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:58.812+0000 7fb6d71ebb80 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5575a1418f80 0x5575a1419390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:33:58.808 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:58.812+0000 7fb6d71ebb80 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x5575a1347c90 con 0x5575a141ec50 2024-05-27T07:33:58.808 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:58.812+0000 7fb6d71ebb80 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x5575a1337830 con 0x5575a129ae40 2024-05-27T07:33:58.808 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:58.812+0000 7fb6d71ebb80 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x5575a133b870 con 0x5575a1418f80 2024-05-27T07:33:58.808 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:58.812+0000 7fb6c6572700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5575a129ae40 0x5575a141e710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:33:58.808 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:58.812+0000 7fb6c6572700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5575a129ae40 0x5575a141e710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.52:37242/0 (socket says 172.21.15.52:37242) 2024-05-27T07:33:58.808 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:58.812+0000 7fb6c6572700 1 -- 172.21.15.52:0/1083939545 learned_addr learned my addr 172.21.15.52:0/1083939545 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-05-27T07:33:58.808 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:58.812+0000 7fb6c5d71700 1 --2- 172.21.15.52:0/2183860932 >> [v2:172.21.15.52:6800/1822318384,v1:172.21.15.52:6801/1822318384] conn(0x7fb6ac0b2a70 0x7fb6ac0b4f80 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7fb6c01099f0 tx=0x7fb6b0009260 comp rx=0 tx=0).ready entity=mgr.15333 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:33:58.809 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:58.813+0000 7fb6c6d73700 1 --2- 172.21.15.52:0/1083939545 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5575a1418f80 0x5575a1419390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:33:58.809 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:58.813+0000 7fb6c5d71700 1 --2- 172.21.15.52:0/1083939545 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x5575a141ec50 0x5575a1418a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:33:58.809 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:58.813+0000 7fb6c6572700 1 -- 172.21.15.52:0/1083939545 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5575a1418f80 msgr2=0x5575a1419390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:33:58.809 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:58.813+0000 7fb6c6572700 1 --2- 172.21.15.52:0/1083939545 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5575a1418f80 0x5575a1419390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:33:58.809 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:58.813+0000 7fb6c6572700 1 -- 172.21.15.52:0/1083939545 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x5575a141ec50 msgr2=0x5575a1418a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:33:58.809 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:58.813+0000 7fb6c6572700 1 --2- 172.21.15.52:0/1083939545 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x5575a141ec50 0x5575a1418a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:33:58.809 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:58.813+0000 7fb6c6572700 1 -- 172.21.15.52:0/1083939545 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5575a12878d0 con 0x5575a129ae40 2024-05-27T07:33:58.809 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:58.813+0000 7fb6c6572700 1 --2- 172.21.15.52:0/1083939545 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5575a129ae40 0x5575a141e710 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7fb6bc0a5a70 tx=0x7fb6bc0a9500 comp rx=0 tx=0).ready entity=mon.2 client_cookie=e08a9ebc574fcecd server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:33:58.809 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:58.813+0000 7fb6b4ff9700 1 -- 172.21.15.52:0/1083939545 <== mon.2 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb6bc0adda0 con 0x5575a129ae40 2024-05-27T07:33:58.809 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:58.813+0000 7fb6b4ff9700 1 -- 172.21.15.52:0/1083939545 <== mon.2 v2:172.21.15.105:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fb6bc0b2850 con 0x5575a129ae40 2024-05-27T07:33:58.809 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:58.813+0000 7fb6b4ff9700 1 -- 172.21.15.52:0/1083939545 <== mon.2 v2:172.21.15.105:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb6bc0b23c0 con 0x5575a129ae40 2024-05-27T07:33:58.812 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:58.814+0000 7fb6d71ebb80 1 -- 172.21.15.52:0/1083939545 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5575a129ae40 msgr2=0x5575a141e710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:33:58.812 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:58.814+0000 7fb6d71ebb80 1 --2- 172.21.15.52:0/1083939545 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5575a129ae40 0x5575a141e710 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7fb6bc0a5a70 tx=0x7fb6bc0a9500 comp rx=0 tx=0).stop 2024-05-27T07:33:58.812 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:58.814+0000 7fb6d71ebb80 1 -- 172.21.15.52:0/1083939545 shutdown_connections 2024-05-27T07:33:58.812 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:58.814+0000 7fb6d71ebb80 1 --2- 172.21.15.52:0/1083939545 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5575a1418f80 0x5575a1419390 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:33:58.813 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:58.814+0000 7fb6d71ebb80 1 --2- 172.21.15.52:0/1083939545 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5575a129ae40 0x5575a141e710 unknown :-1 s=CLOSED pgs=230 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:33:58.813 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:58.814+0000 7fb6d71ebb80 1 --2- 172.21.15.52:0/1083939545 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x5575a141ec50 0x5575a1418a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:33:58.813 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:58.814+0000 7fb6d71ebb80 1 -- 172.21.15.52:0/1083939545 >> 172.21.15.52:0/1083939545 conn(0x5575a12f5af0 msgr2=0x5575a1413640 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:33:58.813 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:58.814+0000 7fb6d71ebb80 1 -- 172.21.15.52:0/1083939545 shutdown_connections 2024-05-27T07:33:58.813 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:58.814+0000 7fb6d71ebb80 1 -- 172.21.15.52:0/1083939545 wait complete. 2024-05-27T07:33:58.813 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:58.815+0000 7fb6d71ebb80 1 -- start start 2024-05-27T07:33:58.813 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:58.815+0000 7fb6d71ebb80 1 Processor -- start 2024-05-27T07:33:58.813 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:58.815+0000 7fb6d71ebb80 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5575a129ae40 0x5575a1422420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:33:58.813 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:58.815+0000 7fb6d71ebb80 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5575a1418f80 0x5575a1422960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:33:58.813 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:58.815+0000 7fb6d71ebb80 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x5575a141ec50 0x5575a1422ea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:33:58.813 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:58.815+0000 7fb6d71ebb80 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x5575a123dfa0 con 0x5575a141ec50 2024-05-27T07:33:58.813 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:58.815+0000 7fb6d71ebb80 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x5575a12d97c0 con 0x5575a129ae40 2024-05-27T07:33:58.814 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:58.815+0000 7fb6d71ebb80 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x5575a1425740 con 0x5575a1418f80 2024-05-27T07:33:58.814 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:58.815+0000 7fb6d71ebb80 10 client.0 _ll_register_callbacks cb 0x5575a12d2920 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T07:33:58.814 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:58.815+0000 7fb6c6572700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5575a129ae40 0x5575a1422420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:33:58.814 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:58.815+0000 7fb6c6d73700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x5575a141ec50 0x5575a1422ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:33:58.814 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:58.815+0000 7fb6c6572700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5575a129ae40 0x5575a1422420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.52:37246/0 (socket says 172.21.15.52:37246) 2024-05-27T07:33:58.814 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:58.815+0000 7fb6c6572700 1 -- 172.21.15.52:0/3448924853 learned_addr learned my addr 172.21.15.52:0/3448924853 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-05-27T07:33:58.814 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:58.815+0000 7fb6c5d71700 1 --2- 172.21.15.52:0/3448924853 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5575a1418f80 0x5575a1422960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:33:58.814 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:58.815+0000 7fb6c6572700 1 -- 172.21.15.52:0/3448924853 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5575a1418f80 msgr2=0x5575a1422960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:33:58.814 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:58.815+0000 7fb6c6572700 1 --2- 172.21.15.52:0/3448924853 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5575a1418f80 0x5575a1422960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:33:58.814 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:58.816+0000 7fb6c6572700 1 -- 172.21.15.52:0/3448924853 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x5575a141ec50 msgr2=0x5575a1422ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:33:58.814 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:58.816+0000 7fb6c6572700 1 --2- 172.21.15.52:0/3448924853 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x5575a141ec50 0x5575a1422ea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:33:58.814 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:58.816+0000 7fb6c6572700 1 -- 172.21.15.52:0/3448924853 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5575a12d6690 con 0x5575a129ae40 2024-05-27T07:33:58.815 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:58.816+0000 7fb6c6572700 1 --2- 172.21.15.52:0/3448924853 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5575a129ae40 0x5575a1422420 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7fb6bc0a96a0 tx=0x7fb6bc0dfe80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=5a4bdaeeeafa907 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:33:58.815 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:58.816+0000 7fb6a9ffb700 10 client.0 ms_handle_connect on v2:172.21.15.105:3300/0 2024-05-27T07:33:58.815 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:58.816+0000 7fb6a9ffb700 1 -- 172.21.15.52:0/3448924853 <== mon.2 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb6bc0c3070 con 0x5575a129ae40 2024-05-27T07:33:58.815 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:58.816+0000 7fb6a9ffb700 1 -- 172.21.15.52:0/3448924853 <== mon.2 v2:172.21.15.105:3300/0 2 ==== osd_map(119..119 src has 1..119) ==== 9451+0+0 (secure 0 0 0) 0x7fb6bc0b25d0 con 0x5575a129ae40 2024-05-27T07:33:58.815 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:58.816+0000 7fb6d71ebb80 1 -- 172.21.15.52:0/3448924853 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fb6bc0b2850 con 0x5575a129ae40 2024-05-27T07:33:58.815 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:58.816+0000 7fb6a9ffb700 1 -- 172.21.15.52:0/3448924853 <== mon.2 v2:172.21.15.105:3300/0 3 ==== mon_get_version_reply(handle=1 version=104) ==== 24+0+0 (secure 0 0 0) 0x7fb6bc0b7050 con 0x5575a129ae40 2024-05-27T07:33:58.815 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:58.816+0000 7fb6d71ebb80 10 client.26645 fetch_fsmap learned FSMap version 104 2024-05-27T07:33:58.815 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:58.816+0000 7fb6d71ebb80 1 -- 172.21.15.52:0/3448924853 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,fsmap=104,monmap=4+}) -- 0x5575a1430290 con 0x5575a129ae40 2024-05-27T07:33:58.815 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:58.816+0000 7fb6a9ffb700 1 -- 172.21.15.52:0/3448924853 <== mon.2 v2:172.21.15.105:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fb6bc0b2850 con 0x5575a129ae40 2024-05-27T07:33:58.815 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:58.816+0000 7fb6a9ffb700 1 -- 172.21.15.52:0/3448924853 <== mon.2 v2:172.21.15.105:3300/0 5 ==== fsmap(e 104) ==== 4211+0+0 (secure 0 0 0) 0x7fb6bc0a5360 con 0x5575a129ae40 2024-05-27T07:33:58.815 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:58.816+0000 7fb6d71ebb80 10 client.26645 fetch_fsmap finished waiting for FSMap version 104 2024-05-27T07:33:58.815 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:58.816+0000 7fb6d71ebb80 10 client.26645 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-05-27T07:33:58.815 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:58.817+0000 7fb6d71ebb80 20 client.26645 populate_metadata read hostname 'smithi052' 2024-05-27T07:33:58.816 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:58.817+0000 7fb6d71ebb80 1 --2- 172.21.15.52:0/3448924853 >> [v2:172.21.15.105:6838/500348151,v1:172.21.15.105:6839/500348151] conn(0x5575a14328a0 0x5575a1434d10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:33:58.816 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:58.817+0000 7fb6d71ebb80 4 client.26645 mds_command: new command op to 24527 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-27T07:33:58.816 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:58.817+0000 7fb6d71ebb80 1 -- 172.21.15.52:0/3448924853 --> [v2:172.21.15.105:6838/500348151,v1:172.21.15.105:6839/500348151] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5575a13e49f0 con 0x5575a14328a0 2024-05-27T07:33:58.816 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:58.817+0000 7fb6c5d71700 1 --2- 172.21.15.52:0/3448924853 >> [v2:172.21.15.105:6838/500348151,v1:172.21.15.105:6839/500348151] conn(0x5575a14328a0 0x5575a1434d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:33:58.816 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:58.817+0000 7fb6c5d71700 1 --2- 172.21.15.52:0/3448924853 >> [v2:172.21.15.105:6838/500348151,v1:172.21.15.105:6839/500348151] conn(0x5575a14328a0 0x5575a1434d10 crc :-1 s=READY pgs=149 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=856ee4cfd126a6e4 server_cookie=1b5515df70ca0ecb in_seq=0 out_seq=0 2024-05-27T07:33:58.816 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:58.817+0000 7fb6a9ffb700 10 client.26645 ms_handle_connect on v2:172.21.15.105:6838/500348151 2024-05-27T07:33:58.817 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:58.820+0000 7fb6a9ffb700 1 -- 172.21.15.52:0/3448924853 <== mds.0 v2:172.21.15.105:6838/500348151 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24041 (crc 0 0 0) 0x5575a13e49f0 con 0x5575a14328a0 2024-05-27T07:33:58.817 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:58.820+0000 7fb6a9ffb700 10 client.26645 handle_command_reply: tid=0 2024-05-27T07:33:58.817 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:58.820+0000 7fb6d71ebb80 1 client.26645 shutdown 2024-05-27T07:33:58.817 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:58.820+0000 7fb6d71ebb80 1 -- 172.21.15.52:0/3448924853 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5575a129ae40 msgr2=0x5575a1422420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:33:58.817 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:58.820+0000 7fb6d71ebb80 1 --2- 172.21.15.52:0/3448924853 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5575a129ae40 0x5575a1422420 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7fb6bc0a96a0 tx=0x7fb6bc0dfe80 comp rx=0 tx=0).stop 2024-05-27T07:33:58.817 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:58.820+0000 7fb6d71ebb80 1 -- 172.21.15.52:0/3448924853 shutdown_connections 2024-05-27T07:33:58.817 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:58.820+0000 7fb6d71ebb80 1 --2- 172.21.15.52:0/3448924853 >> [v2:172.21.15.105:6838/500348151,v1:172.21.15.105:6839/500348151] conn(0x5575a14328a0 0x5575a1434d10 crc :-1 s=READY pgs=149 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:33:58.818 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:58.820+0000 7fb6d71ebb80 1 --2- 172.21.15.52:0/3448924853 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x5575a141ec50 0x5575a1422ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:33:58.818 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:58.820+0000 7fb6d71ebb80 1 --2- 172.21.15.52:0/3448924853 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5575a129ae40 0x5575a1422420 unknown :-1 s=CLOSED pgs=231 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:33:58.818 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:58.820+0000 7fb6d71ebb80 1 --2- 172.21.15.52:0/3448924853 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5575a1418f80 0x5575a1422960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:33:58.818 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:58.820+0000 7fb6a9ffb700 0 client.26645 ms_handle_reset on v2:172.21.15.105:6838/500348151 2024-05-27T07:33:58.818 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:58.820+0000 7fb6d71ebb80 1 -- 172.21.15.52:0/3448924853 >> 172.21.15.52:0/3448924853 conn(0x5575a12f5af0 msgr2=0x5575a1417a30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:33:58.818 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:58.820+0000 7fb6d71ebb80 1 -- 172.21.15.52:0/3448924853 shutdown_connections 2024-05-27T07:33:58.818 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:58.820+0000 7fb6d71ebb80 1 -- 172.21.15.52:0/3448924853 wait complete. 2024-05-27T07:33:58.818 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:58.820+0000 7fb6d71ebb80 20 client.26645 trim_cache size 0 max 16384 2024-05-27T07:33:58.836 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:58.841+0000 7fb6d71ebb80 1 Processor -- start 2024-05-27T07:33:58.836 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:58.841+0000 7fb6d71ebb80 1 -- start start 2024-05-27T07:33:58.837 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:58.841+0000 7fb6d71ebb80 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5575a129ae40 0x5575a1420070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:33:58.837 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:58.841+0000 7fb6d71ebb80 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5575a1418f80 0x5575a1422e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:33:58.837 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:58.841+0000 7fb6d71ebb80 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x5575a141ec50 0x5575a1424990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:33:58.837 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:58.841+0000 7fb6d71ebb80 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x7fb6c01ae920 con 0x5575a141ec50 2024-05-27T07:33:58.837 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:58.841+0000 7fb6d71ebb80 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7fb6c01accf0 con 0x5575a129ae40 2024-05-27T07:33:58.837 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:58.841+0000 7fb6d71ebb80 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7fb6c01ae220 con 0x5575a1418f80 2024-05-27T07:33:58.837 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:58.841+0000 7fb6c6d73700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x5575a141ec50 0x5575a1424990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:33:58.837 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:58.841+0000 7fb6c6d73700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x5575a141ec50 0x5575a1424990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.52:40622/0 (socket says 172.21.15.52:40622) 2024-05-27T07:33:58.837 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:58.841+0000 7fb6c6d73700 1 -- 172.21.15.52:0/3196068474 learned_addr learned my addr 172.21.15.52:0/3196068474 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-05-27T07:33:58.837 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:58.841+0000 7fb6c6572700 1 --2- 172.21.15.52:0/3196068474 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5575a129ae40 0x5575a1420070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:33:58.837 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:58.842+0000 7fb6c5d71700 1 --2- 172.21.15.52:0/3196068474 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5575a1418f80 0x5575a1422e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:33:58.837 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:58.842+0000 7fb6c6d73700 1 -- 172.21.15.52:0/3196068474 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5575a1418f80 msgr2=0x5575a1422e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:33:58.837 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:58.842+0000 7fb6c6d73700 1 --2- 172.21.15.52:0/3196068474 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5575a1418f80 0x5575a1422e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:33:58.838 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:58.842+0000 7fb6c6d73700 1 -- 172.21.15.52:0/3196068474 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5575a129ae40 msgr2=0x5575a1420070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:33:58.838 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:58.842+0000 7fb6c6d73700 1 --2- 172.21.15.52:0/3196068474 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5575a129ae40 0x5575a1420070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:33:58.838 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:58.842+0000 7fb6c6d73700 1 -- 172.21.15.52:0/3196068474 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5575a1414110 con 0x5575a141ec50 2024-05-27T07:33:58.838 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:58.842+0000 7fb6c6d73700 1 --2- 172.21.15.52:0/3196068474 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x5575a141ec50 0x5575a1424990 secure :-1 s=READY pgs=712 cs=0 l=1 rev1=1 crypto rx=0x7fb6b8009d10 tx=0x7fb6b800b040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=50e7a7aa6dc4105f server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:33:58.838 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:58.842+0000 7fb6abfff700 1 -- 172.21.15.52:0/3196068474 <== mon.0 v2:172.21.15.52:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb6b80092e0 con 0x5575a141ec50 2024-05-27T07:33:58.838 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:58.842+0000 7fb6abfff700 1 -- 172.21.15.52:0/3196068474 <== mon.0 v2:172.21.15.52:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fb6b8009480 con 0x5575a141ec50 2024-05-27T07:33:58.838 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:58.842+0000 7fb6abfff700 1 -- 172.21.15.52:0/3196068474 <== mon.0 v2:172.21.15.52:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb6b80050b0 con 0x5575a141ec50 2024-05-27T07:33:58.841 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:58.843+0000 7fb6d71ebb80 1 -- 172.21.15.52:0/3196068474 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x5575a141ec50 msgr2=0x5575a1424990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:33:58.841 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:58.843+0000 7fb6d71ebb80 1 --2- 172.21.15.52:0/3196068474 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x5575a141ec50 0x5575a1424990 secure :-1 s=READY pgs=712 cs=0 l=1 rev1=1 crypto rx=0x7fb6b8009d10 tx=0x7fb6b800b040 comp rx=0 tx=0).stop 2024-05-27T07:33:58.841 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:58.843+0000 7fb6d71ebb80 1 -- 172.21.15.52:0/3196068474 shutdown_connections 2024-05-27T07:33:58.841 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:58.843+0000 7fb6d71ebb80 1 --2- 172.21.15.52:0/3196068474 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x5575a141ec50 0x5575a1424990 unknown :-1 s=CLOSED pgs=712 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:33:58.841 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:58.843+0000 7fb6d71ebb80 1 --2- 172.21.15.52:0/3196068474 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5575a129ae40 0x5575a1420070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:33:58.841 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:58.843+0000 7fb6d71ebb80 1 --2- 172.21.15.52:0/3196068474 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5575a1418f80 0x5575a1422e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:33:58.842 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:58.843+0000 7fb6d71ebb80 1 -- 172.21.15.52:0/3196068474 >> 172.21.15.52:0/3196068474 conn(0x5575a12f5af0 msgr2=0x5575a1421380 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:33:58.842 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:58.843+0000 7fb6d71ebb80 1 -- 172.21.15.52:0/3196068474 shutdown_connections 2024-05-27T07:33:58.842 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:58.843+0000 7fb6d71ebb80 1 -- 172.21.15.52:0/3196068474 wait complete. 2024-05-27T07:33:58.842 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:58.843+0000 7fb6d71ebb80 1 -- start start 2024-05-27T07:33:58.842 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:58.844+0000 7fb6d71ebb80 1 Processor -- start 2024-05-27T07:33:58.842 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:58.844+0000 7fb6d71ebb80 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x5575a129ae40 0x5575a142abd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:33:58.842 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:58.844+0000 7fb6d71ebb80 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5575a1418f80 0x5575a142b110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:33:58.842 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:58.844+0000 7fb6d71ebb80 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5575a141ec50 0x5575a14279a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:33:58.842 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:58.844+0000 7fb6d71ebb80 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x7fb6c01adac0 con 0x5575a129ae40 2024-05-27T07:33:58.842 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:58.844+0000 7fb6d71ebb80 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7fb6c01ad3c0 con 0x5575a1418f80 2024-05-27T07:33:58.842 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:58.844+0000 7fb6d71ebb80 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x5575a1410090 con 0x5575a141ec50 2024-05-27T07:33:58.842 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:58.844+0000 7fb6d71ebb80 10 client.0 _ll_register_callbacks cb 0x5575a12d2920 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T07:33:58.843 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:58.844+0000 7fb6c6572700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x5575a129ae40 0x5575a142abd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:33:58.843 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:58.844+0000 7fb6c6572700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x5575a129ae40 0x5575a142abd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.52:40628/0 (socket says 172.21.15.52:40628) 2024-05-27T07:33:58.843 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:58.844+0000 7fb6c6572700 1 -- 172.21.15.52:0/4266823048 learned_addr learned my addr 172.21.15.52:0/4266823048 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-05-27T07:33:58.843 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:58.844+0000 7fb6c5d71700 1 --2- 172.21.15.52:0/4266823048 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5575a1418f80 0x5575a142b110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:33:58.843 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:58.844+0000 7fb6c6d73700 1 --2- 172.21.15.52:0/4266823048 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5575a141ec50 0x5575a14279a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:33:58.843 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:58.844+0000 7fb6c6572700 1 -- 172.21.15.52:0/4266823048 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5575a141ec50 msgr2=0x5575a14279a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:33:58.843 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:58.844+0000 7fb6c6572700 1 --2- 172.21.15.52:0/4266823048 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5575a141ec50 0x5575a14279a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:33:58.843 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:58.844+0000 7fb6c6572700 1 -- 172.21.15.52:0/4266823048 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5575a1418f80 msgr2=0x5575a142b110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:33:58.843 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:58.844+0000 7fb6c6572700 1 --2- 172.21.15.52:0/4266823048 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5575a1418f80 0x5575a142b110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:33:58.843 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:58.844+0000 7fb6c6572700 1 -- 172.21.15.52:0/4266823048 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5575a1427fe0 con 0x5575a129ae40 2024-05-27T07:33:58.843 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:58.844+0000 7fb6c5d71700 1 --2- 172.21.15.52:0/4266823048 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5575a1418f80 0x5575a142b110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-27T07:33:58.843 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:58.845+0000 7fb6c6572700 1 --2- 172.21.15.52:0/4266823048 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x5575a129ae40 0x5575a142abd0 secure :-1 s=READY pgs=713 cs=0 l=1 rev1=1 crypto rx=0x7fb6bc0a5760 tx=0x7fb6bc009820 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3f5cc5b449fd47f8 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:33:58.843 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:58.845+0000 7fb6a9ffb700 10 client.0 ms_handle_connect on v2:172.21.15.52:3300/0 2024-05-27T07:33:58.844 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:58.845+0000 7fb6a9ffb700 1 -- 172.21.15.52:0/4266823048 <== mon.0 v2:172.21.15.52:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb6bc0c3070 con 0x5575a129ae40 2024-05-27T07:33:58.844 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:58.845+0000 7fb6a9ffb700 1 -- 172.21.15.52:0/4266823048 <== mon.0 v2:172.21.15.52:3300/0 2 ==== osd_map(119..119 src has 1..119) ==== 9451+0+0 (secure 0 0 0) 0x7fb6bc0a5360 con 0x5575a129ae40 2024-05-27T07:33:58.844 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:58.845+0000 7fb6d71ebb80 1 -- 172.21.15.52:0/4266823048 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x5575a1349be0 con 0x5575a129ae40 2024-05-27T07:33:58.844 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:58.845+0000 7fb6a9ffb700 1 -- 172.21.15.52:0/4266823048 <== mon.0 v2:172.21.15.52:3300/0 3 ==== mon_get_version_reply(handle=1 version=104) ==== 24+0+0 (secure 0 0 0) 0x7fb6bc0a5560 con 0x5575a129ae40 2024-05-27T07:33:58.844 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:58.845+0000 7fb6d71ebb80 10 client.16965 fetch_fsmap learned FSMap version 104 2024-05-27T07:33:58.844 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:58.845+0000 7fb6d71ebb80 1 -- 172.21.15.52:0/4266823048 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({config=0+,fsmap=104,monmap=4+}) -- 0x5575a149dca0 con 0x5575a129ae40 2024-05-27T07:33:58.844 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:58.845+0000 7fb6a9ffb700 1 -- 172.21.15.52:0/4266823048 <== mon.0 v2:172.21.15.52:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x5575a1349be0 con 0x5575a129ae40 2024-05-27T07:33:58.844 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:58.845+0000 7fb6a9ffb700 1 -- 172.21.15.52:0/4266823048 <== mon.0 v2:172.21.15.52:3300/0 5 ==== fsmap(e 104) ==== 4211+0+0 (secure 0 0 0) 0x7fb6bc010720 con 0x5575a129ae40 2024-05-27T07:33:58.844 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:58.845+0000 7fb6d71ebb80 10 client.16965 fetch_fsmap finished waiting for FSMap version 104 2024-05-27T07:33:58.844 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:58.845+0000 7fb6d71ebb80 10 client.16965 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-05-27T07:33:58.844 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:58.845+0000 7fb6d71ebb80 20 client.16965 populate_metadata read hostname 'smithi052' 2024-05-27T07:33:58.844 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:58.845+0000 7fb6d71ebb80 1 --2- 172.21.15.52:0/4266823048 >> [v2:172.21.15.105:6838/500348151,v1:172.21.15.105:6839/500348151] conn(0x5575a149e190 0x5575a14a05b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:33:58.844 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:58.845+0000 7fb6d71ebb80 4 client.16965 mds_command: new command op to 24527 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-05-27T07:33:58.845 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:58.845+0000 7fb6d71ebb80 1 -- 172.21.15.52:0/4266823048 --> [v2:172.21.15.105:6838/500348151,v1:172.21.15.105:6839/500348151] -- command(tid 0: {"prefix": "scrub status"}) -- 0x5575a13433c0 con 0x5575a149e190 2024-05-27T07:33:58.845 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:58.846+0000 7fb6c5d71700 1 --2- 172.21.15.52:0/4266823048 >> [v2:172.21.15.105:6838/500348151,v1:172.21.15.105:6839/500348151] conn(0x5575a149e190 0x5575a14a05b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:33:58.845 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:58.846+0000 7fb6a9ffb700 10 client.16965 ms_handle_connect on v2:172.21.15.105:6838/500348151 2024-05-27T07:33:58.845 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:58.846+0000 7fb6c5d71700 1 --2- 172.21.15.52:0/4266823048 >> [v2:172.21.15.105:6838/500348151,v1:172.21.15.105:6839/500348151] conn(0x5575a149e190 0x5575a14a05b0 crc :-1 s=READY pgs=150 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=fba7da22a328cb64 server_cookie=4d9913d3c3664bd4 in_seq=0 out_seq=0 2024-05-27T07:33:58.846 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:58.849+0000 7fb6a9ffb700 1 -- 172.21.15.52:0/4266823048 <== mds.0 v2:172.21.15.105:6838/500348151 1 ==== command_reply(tid 0: 0 ) ==== 8+0+269 (crc 0 0 0) 0x5575a13433c0 con 0x5575a149e190 2024-05-27T07:33:58.846 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:58.849+0000 7fb6a9ffb700 10 client.16965 handle_command_reply: tid=0 2024-05-27T07:33:58.846 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:58.849+0000 7fb6d71ebb80 1 client.16965 shutdown 2024-05-27T07:33:58.846 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:58.849+0000 7fb6d71ebb80 1 -- 172.21.15.52:0/4266823048 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x5575a129ae40 msgr2=0x5575a142abd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:33:58.846 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:58.849+0000 7fb6d71ebb80 1 --2- 172.21.15.52:0/4266823048 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x5575a129ae40 0x5575a142abd0 secure :-1 s=READY pgs=713 cs=0 l=1 rev1=1 crypto rx=0x7fb6bc0a5760 tx=0x7fb6bc009820 comp rx=0 tx=0).stop 2024-05-27T07:33:58.846 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:58.849+0000 7fb6d71ebb80 1 -- 172.21.15.52:0/4266823048 shutdown_connections 2024-05-27T07:33:58.846 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:58.849+0000 7fb6d71ebb80 1 --2- 172.21.15.52:0/4266823048 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5575a141ec50 0x5575a14279a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:33:58.846 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:58.849+0000 7fb6d71ebb80 1 --2- 172.21.15.52:0/4266823048 >> [v2:172.21.15.105:6838/500348151,v1:172.21.15.105:6839/500348151] conn(0x5575a149e190 0x5575a14a05b0 crc :-1 s=READY pgs=150 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:33:58.846 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:58.849+0000 7fb6d71ebb80 1 --2- 172.21.15.52:0/4266823048 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x5575a129ae40 0x5575a142abd0 unknown :-1 s=CLOSED pgs=713 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:33:58.847 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:58.849+0000 7fb6d71ebb80 1 --2- 172.21.15.52:0/4266823048 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5575a1418f80 0x5575a142b110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:33:58.847 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:58.849+0000 7fb6d71ebb80 1 -- 172.21.15.52:0/4266823048 >> 172.21.15.52:0/4266823048 conn(0x5575a12f5af0 msgr2=0x5575a141c010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:33:58.847 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:58.849+0000 7fb6a9ffb700 0 client.16965 ms_handle_reset on v2:172.21.15.105:6838/500348151 2024-05-27T07:33:58.847 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:58.849+0000 7fb6d71ebb80 1 -- 172.21.15.52:0/4266823048 shutdown_connections 2024-05-27T07:33:58.847 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:58.849+0000 7fb6d71ebb80 1 -- 172.21.15.52:0/4266823048 wait complete. 2024-05-27T07:33:58.847 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:58.849+0000 7fb6d71ebb80 20 client.16965 trim_cache size 0 max 16384 2024-05-27T07:33:58.847 INFO:teuthology.orchestra.run.smithi052.stdout:{ 2024-05-27T07:33:58.847 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "scrub active (14925 inodes in the stack)", 2024-05-27T07:33:58.847 INFO:teuthology.orchestra.run.smithi052.stdout: "scrubs": { 2024-05-27T07:33:58.847 INFO:teuthology.orchestra.run.smithi052.stdout: "b39c870d-d57b-4c7d-baaf-c256689c7e81": { 2024-05-27T07:33:58.848 INFO:teuthology.orchestra.run.smithi052.stdout: "path": "/", 2024-05-27T07:33:58.848 INFO:teuthology.orchestra.run.smithi052.stdout: "tag": "b39c870d-d57b-4c7d-baaf-c256689c7e81", 2024-05-27T07:33:58.848 INFO:teuthology.orchestra.run.smithi052.stdout: "options": "recursive,force" 2024-05-27T07:33:58.848 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-05-27T07:33:58.848 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-05-27T07:33:58.848 INFO:teuthology.orchestra.run.smithi052.stdout:} 2024-05-27T07:33:58.848 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:58.850+0000 7fb6c87d6700 1 -- 172.21.15.52:0/2183860932 >> [v2:172.21.15.52:6800/1822318384,v1:172.21.15.52:6801/1822318384] conn(0x7fb6ac0b2a70 msgr2=0x7fb6ac0b4f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:33:58.849 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:58.850+0000 7fb6c87d6700 1 --2- 172.21.15.52:0/2183860932 >> [v2:172.21.15.52:6800/1822318384,v1:172.21.15.52:6801/1822318384] conn(0x7fb6ac0b2a70 0x7fb6ac0b4f80 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7fb6c01099f0 tx=0x7fb6b0009260 comp rx=0 tx=0).stop 2024-05-27T07:33:58.849 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:58.850+0000 7fb6c87d6700 1 -- 172.21.15.52:0/2183860932 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fb6c01034b0 msgr2=0x7fb6c00ff290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:33:58.849 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:58.850+0000 7fb6c87d6700 1 --2- 172.21.15.52:0/2183860932 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fb6c01034b0 0x7fb6c00ff290 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7fb6bc00e9a0 tx=0x7fb6bc00ecb0 comp rx=0 tx=0).stop 2024-05-27T07:33:58.849 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:58.850+0000 7fb6c87d6700 1 -- 172.21.15.52:0/2183860932 shutdown_connections 2024-05-27T07:33:58.849 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:58.850+0000 7fb6c87d6700 1 --2- 172.21.15.52:0/2183860932 >> [v2:172.21.15.52:6800/1822318384,v1:172.21.15.52:6801/1822318384] conn(0x7fb6ac0b2a70 0x7fb6ac0b4f80 unknown :-1 s=CLOSED pgs=172 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:33:58.849 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:58.850+0000 7fb6c87d6700 1 --2- 172.21.15.52:0/2183860932 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fb6c0108730 0x7fb6c00ffd10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:33:58.849 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:58.850+0000 7fb6c87d6700 1 --2- 172.21.15.52:0/2183860932 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fb6c01034b0 0x7fb6c00ff290 unknown :-1 s=CLOSED pgs=229 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:33:58.849 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:58.850+0000 7fb6c87d6700 1 --2- 172.21.15.52:0/2183860932 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fb6c0107de0 0x7fb6c00ff7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:33:58.849 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:58.850+0000 7fb6c87d6700 1 -- 172.21.15.52:0/2183860932 >> 172.21.15.52:0/2183860932 conn(0x7fb6c00fedb0 msgr2=0x7fb6c006b8e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:33:58.849 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:58.850+0000 7fb6c87d6700 1 -- 172.21.15.52:0/2183860932 shutdown_connections 2024-05-27T07:33:58.849 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:33:58.850+0000 7fb6c87d6700 1 -- 172.21.15.52:0/2183860932 wait complete. 2024-05-27T07:33:59.560 INFO:tasks.cephfs.filesystem:scrub status for tag:b39c870d-d57b-4c7d-baaf-c256689c7e81 - {'path': '/', 'tag': 'b39c870d-d57b-4c7d-baaf-c256689c7e81', 'options': 'recursive,force'} 2024-05-27T07:34:00.606 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:34:00 smithi173 ceph-mon[37000]: pgmap v272: 129 pgs: 129 active+clean; 672 MiB data, 4.7 GiB used, 1.0 TiB / 1.0 TiB avail; 102 KiB/s rd, 2.0 MiB/s wr, 551 op/s 2024-05-27T07:34:00.694 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:34:00 smithi105 ceph-mon[35977]: pgmap v272: 129 pgs: 129 active+clean; 672 MiB data, 4.7 GiB used, 1.0 TiB / 1.0 TiB avail; 102 KiB/s rd, 2.0 MiB/s wr, 551 op/s 2024-05-27T07:34:00.733 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:34:00 smithi052 ceph-mon[31955]: pgmap v272: 129 pgs: 129 active+clean; 672 MiB data, 4.7 GiB used, 1.0 TiB / 1.0 TiB avail; 102 KiB/s rd, 2.0 MiB/s wr, 551 op/s 2024-05-27T07:34:02.606 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:34:02 smithi173 ceph-mon[37000]: pgmap v273: 129 pgs: 129 active+clean; 657 MiB data, 4.7 GiB used, 1.0 TiB / 1.0 TiB avail; 172 KiB/s rd, 2.1 MiB/s wr, 599 op/s 2024-05-27T07:34:02.687 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:34:02 smithi105 ceph-mon[35977]: pgmap v273: 129 pgs: 129 active+clean; 657 MiB data, 4.7 GiB used, 1.0 TiB / 1.0 TiB avail; 172 KiB/s rd, 2.1 MiB/s wr, 599 op/s 2024-05-27T07:34:02.734 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:34:02 smithi052 ceph-mon[31955]: pgmap v273: 129 pgs: 129 active+clean; 657 MiB data, 4.7 GiB used, 1.0 TiB / 1.0 TiB avail; 172 KiB/s rd, 2.1 MiB/s wr, 599 op/s 2024-05-27T07:34:03.606 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:34:03 smithi173 ceph-mon[37000]: fsmap cephfs:5 5 up:active 2024-05-27T07:34:03.607 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:34:03 smithi173 ceph-mon[37000]: osdmap e120: 12 total, 12 up, 12 in 2024-05-27T07:34:03.695 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:34:03 smithi105 ceph-mon[35977]: fsmap cephfs:5 5 up:active 2024-05-27T07:34:03.695 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:34:03 smithi105 ceph-mon[35977]: osdmap e120: 12 total, 12 up, 12 in 2024-05-27T07:34:03.734 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:34:03 smithi052 ceph-mon[31955]: fsmap cephfs:5 5 up:active 2024-05-27T07:34:03.734 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:34:03 smithi052 ceph-mon[31955]: osdmap e120: 12 total, 12 up, 12 in 2024-05-27T07:34:04.606 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:34:04 smithi173 ceph-mon[37000]: pgmap v275: 129 pgs: 1 active+clean+snaptrim, 128 active+clean; 657 MiB data, 4.7 GiB used, 1.0 TiB / 1.0 TiB avail; 338 KiB/s rd, 1.8 MiB/s wr, 822 op/s 2024-05-27T07:34:04.694 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:34:04 smithi105 ceph-mon[35977]: pgmap v275: 129 pgs: 1 active+clean+snaptrim, 128 active+clean; 657 MiB data, 4.7 GiB used, 1.0 TiB / 1.0 TiB avail; 338 KiB/s rd, 1.8 MiB/s wr, 822 op/s 2024-05-27T07:34:04.733 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:34:04 smithi052 ceph-mon[31955]: pgmap v275: 129 pgs: 1 active+clean+snaptrim, 128 active+clean; 657 MiB data, 4.7 GiB used, 1.0 TiB / 1.0 TiB avail; 338 KiB/s rd, 1.8 MiB/s wr, 822 op/s 2024-05-27T07:34:06.606 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:34:06 smithi173 ceph-mon[37000]: pgmap v276: 129 pgs: 21 active+clean+snaptrim_wait, 9 active+clean+snaptrim, 99 active+clean; 660 MiB data, 4.7 GiB used, 1.0 TiB / 1.0 TiB avail; 457 KiB/s rd, 1.5 MiB/s wr, 932 op/s 2024-05-27T07:34:06.694 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:34:06 smithi105 ceph-mon[35977]: pgmap v276: 129 pgs: 21 active+clean+snaptrim_wait, 9 active+clean+snaptrim, 99 active+clean; 660 MiB data, 4.7 GiB used, 1.0 TiB / 1.0 TiB avail; 457 KiB/s rd, 1.5 MiB/s wr, 932 op/s 2024-05-27T07:34:06.734 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:34:06 smithi052 ceph-mon[31955]: pgmap v276: 129 pgs: 21 active+clean+snaptrim_wait, 9 active+clean+snaptrim, 99 active+clean; 660 MiB data, 4.7 GiB used, 1.0 TiB / 1.0 TiB avail; 457 KiB/s rd, 1.5 MiB/s wr, 932 op/s 2024-05-27T07:34:08.606 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:34:08 smithi173 ceph-mon[37000]: pgmap v277: 129 pgs: 55 active+clean+snaptrim_wait, 19 active+clean+snaptrim, 55 active+clean; 662 MiB data, 4.8 GiB used, 1.0 TiB / 1.0 TiB avail; 581 KiB/s rd, 1.6 MiB/s wr, 1.03k op/s 2024-05-27T07:34:08.694 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:34:08 smithi105 ceph-mon[35977]: pgmap v277: 129 pgs: 55 active+clean+snaptrim_wait, 19 active+clean+snaptrim, 55 active+clean; 662 MiB data, 4.8 GiB used, 1.0 TiB / 1.0 TiB avail; 581 KiB/s rd, 1.6 MiB/s wr, 1.03k op/s 2024-05-27T07:34:08.734 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:34:08 smithi052 ceph-mon[31955]: pgmap v277: 129 pgs: 55 active+clean+snaptrim_wait, 19 active+clean+snaptrim, 55 active+clean; 662 MiB data, 4.8 GiB used, 1.0 TiB / 1.0 TiB avail; 581 KiB/s rd, 1.6 MiB/s wr, 1.03k op/s 2024-05-27T07:34:10.606 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:34:10 smithi173 ceph-mon[37000]: pgmap v278: 129 pgs: 55 active+clean+snaptrim_wait, 19 active+clean+snaptrim, 55 active+clean; 665 MiB data, 4.8 GiB used, 1.0 TiB / 1.0 TiB avail; 554 KiB/s rd, 1.2 MiB/s wr, 933 op/s 2024-05-27T07:34:10.694 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:34:10 smithi105 ceph-mon[35977]: pgmap v278: 129 pgs: 55 active+clean+snaptrim_wait, 19 active+clean+snaptrim, 55 active+clean; 665 MiB data, 4.8 GiB used, 1.0 TiB / 1.0 TiB avail; 554 KiB/s rd, 1.2 MiB/s wr, 933 op/s 2024-05-27T07:34:10.735 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:34:10 smithi052 ceph-mon[31955]: pgmap v278: 129 pgs: 55 active+clean+snaptrim_wait, 19 active+clean+snaptrim, 55 active+clean; 665 MiB data, 4.8 GiB used, 1.0 TiB / 1.0 TiB avail; 554 KiB/s rd, 1.2 MiB/s wr, 933 op/s 2024-05-27T07:34:10.747 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:34:10.749+0000 7f924f33b640 1 -- 172.21.15.52:0/3747865057 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f9240095940 msgr2=0x7f9240095d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:34:10.748 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:34:10.749+0000 7f924f33b640 1 --2- 172.21.15.52:0/3747865057 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f9240095940 0x7f9240095d50 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7f9238009900 tx=0x7f923802a4d0 comp rx=0 tx=0).stop 2024-05-27T07:34:10.748 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:34:10.753+0000 7f924f33b640 1 -- 172.21.15.52:0/3747865057 shutdown_connections 2024-05-27T07:34:10.748 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:34:10.753+0000 7f924f33b640 1 --2- 172.21.15.52:0/3747865057 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f924009ad30 0x7f924009b210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:34:10.748 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:34:10.753+0000 7f924f33b640 1 --2- 172.21.15.52:0/3747865057 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f924009a390 0x7f924009a7a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:34:10.748 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:34:10.753+0000 7f924f33b640 1 --2- 172.21.15.52:0/3747865057 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f9240095940 0x7f9240095d50 unknown :-1 s=CLOSED pgs=232 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:34:10.748 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:34:10.753+0000 7f924f33b640 1 -- 172.21.15.52:0/3747865057 >> 172.21.15.52:0/3747865057 conn(0x7f924000aff0 msgr2=0x7f924000b430 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:34:10.753 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:34:10.758+0000 7f924f33b640 1 -- 172.21.15.52:0/3747865057 shutdown_connections 2024-05-27T07:34:10.757 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:34:10.762+0000 7f924f33b640 1 -- 172.21.15.52:0/3747865057 wait complete. 2024-05-27T07:34:10.760 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:34:10.763+0000 7f924f33b640 1 Processor -- start 2024-05-27T07:34:10.763 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:34:10.769+0000 7f924f33b640 1 -- start start 2024-05-27T07:34:10.766 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:34:10.771+0000 7f924f33b640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f9240095940 0x7f92400a9a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:34:10.766 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:34:10.771+0000 7f924f33b640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f924009a390 0x7f92400a2b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:34:10.766 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:34:10.771+0000 7f924f33b640 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f924009ad30 0x7f92400a3060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:34:10.766 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:34:10.771+0000 7f924f33b640 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x7f92400be280 con 0x7f924009ad30 2024-05-27T07:34:10.766 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:34:10.771+0000 7f924f33b640 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f92400be100 con 0x7f9240095940 2024-05-27T07:34:10.766 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:34:10.771+0000 7f924f33b640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f92400be400 con 0x7f924009a390 2024-05-27T07:34:10.774 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:34:10.776+0000 7f924db38640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f924009a390 0x7f92400a2b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:34:10.775 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:34:10.776+0000 7f924db38640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f924009a390 0x7f92400a2b20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.173:3300/0 says I am v2:172.21.15.52:46028/0 (socket says 172.21.15.52:46028) 2024-05-27T07:34:10.775 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:34:10.776+0000 7f924db38640 1 -- 172.21.15.52:0/1393189295 learned_addr learned my addr 172.21.15.52:0/1393189295 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-05-27T07:34:10.775 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:34:10.778+0000 7f924eb3a640 1 --2- 172.21.15.52:0/1393189295 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f924009ad30 0x7f92400a3060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:34:10.775 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:34:10.779+0000 7f924e339640 1 --2- 172.21.15.52:0/1393189295 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f9240095940 0x7f92400a9a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:34:10.775 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:34:10.779+0000 7f924eb3a640 1 -- 172.21.15.52:0/1393189295 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f924009a390 msgr2=0x7f92400a2b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:34:10.775 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:34:10.779+0000 7f924eb3a640 1 --2- 172.21.15.52:0/1393189295 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f924009a390 0x7f92400a2b20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:34:10.775 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:34:10.779+0000 7f924eb3a640 1 -- 172.21.15.52:0/1393189295 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f9240095940 msgr2=0x7f92400a9a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:34:10.775 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:34:10.779+0000 7f924eb3a640 1 --2- 172.21.15.52:0/1393189295 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f9240095940 0x7f92400a9a00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:34:10.775 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:34:10.779+0000 7f924eb3a640 1 -- 172.21.15.52:0/1393189295 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f92400a3830 con 0x7f924009ad30 2024-05-27T07:34:10.775 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:34:10.779+0000 7f924eb3a640 1 --2- 172.21.15.52:0/1393189295 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f924009ad30 0x7f92400a3060 secure :-1 s=READY pgs=714 cs=0 l=1 rev1=1 crypto rx=0x7f9248069170 tx=0x7f9248078670 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:34:10.780 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:34:10.786+0000 7f923f7fe640 1 -- 172.21.15.52:0/1393189295 <== mon.0 v2:172.21.15.52:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9248074600 con 0x7f924009ad30 2024-05-27T07:34:10.780 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:34:10.786+0000 7f923f7fe640 1 -- 172.21.15.52:0/1393189295 <== mon.0 v2:172.21.15.52:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f9248074c50 con 0x7f924009ad30 2024-05-27T07:34:10.780 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:34:10.786+0000 7f923f7fe640 1 -- 172.21.15.52:0/1393189295 <== mon.0 v2:172.21.15.52:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9248071d70 con 0x7f924009ad30 2024-05-27T07:34:10.782 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:34:10.787+0000 7f924f33b640 1 -- 172.21.15.52:0/1393189295 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f92400a7520 con 0x7f924009ad30 2024-05-27T07:34:10.782 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:34:10.787+0000 7f924f33b640 1 -- 172.21.15.52:0/1393189295 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f92400a79f0 con 0x7f924009ad30 2024-05-27T07:34:10.790 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:34:10.796+0000 7f925744d740 1 Processor -- start 2024-05-27T07:34:10.791 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:34:10.796+0000 7f925744d740 1 -- start start 2024-05-27T07:34:10.791 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:34:10.796+0000 7f925744d740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x555afcba01b0 0x555afcf41590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:34:10.791 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:34:10.796+0000 7f925744d740 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x555afcb772f0 0x555afcf41ad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:34:10.791 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:34:10.797+0000 7f925744d740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x555afcb7f4d0 0x555afcf42010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:34:10.791 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:34:10.797+0000 7f925744d740 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x555afceb3c30 con 0x555afcb772f0 2024-05-27T07:34:10.791 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:34:10.797+0000 7f925744d740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x555afcea37d0 con 0x555afcb7f4d0 2024-05-27T07:34:10.791 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:34:10.797+0000 7f925744d740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x555afcea7810 con 0x555afcba01b0 2024-05-27T07:34:10.794 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:34:10.799+0000 7f923f7fe640 1 -- 172.21.15.52:0/1393189295 <== mon.0 v2:172.21.15.52:3300/0 4 ==== mgrmap(e 26) ==== 149783+0+0 (secure 0 0 0) 0x7f9248071470 con 0x7f924009ad30 2024-05-27T07:34:10.795 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:34:10.801+0000 7f923f7fe640 1 --2- 172.21.15.52:0/1393189295 >> [v2:172.21.15.52:6800/1822318384,v1:172.21.15.52:6801/1822318384] conn(0x7f92340b2b70 0x7f92340b5080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:34:10.796 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:34:10.801+0000 7f923f7fe640 1 -- 172.21.15.52:0/1393189295 <== mon.0 v2:172.21.15.52:3300/0 5 ==== osd_map(120..120 src has 1..120) ==== 9555+0+0 (secure 0 0 0) 0x7f92481473b0 con 0x7f924009ad30 2024-05-27T07:34:10.796 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:34:10.802+0000 7f924eb3a640 1 --2- 172.21.15.52:0/1393189295 >> [v2:172.21.15.52:6800/1822318384,v1:172.21.15.52:6801/1822318384] conn(0x7f92340b2b70 0x7f92340b5080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:34:10.797 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:34:10.802+0000 7f924eb3a640 1 --2- 172.21.15.52:0/1393189295 >> [v2:172.21.15.52:6800/1822318384,v1:172.21.15.52:6801/1822318384] conn(0x7f92340b2b70 0x7f92340b5080 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7f924810a3b0 tx=0x7f9248068790 comp rx=0 tx=0).ready entity=mgr.15333 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:34:10.797 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:34:10.803+0000 7f924e339640 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x555afcb772f0 0x555afcf41ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:34:10.797 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:34:10.803+0000 7f924e339640 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x555afcb772f0 0x555afcf41ad0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.52:35030/0 (socket says 172.21.15.52:35030) 2024-05-27T07:34:10.797 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:34:10.803+0000 7f924e339640 1 -- 172.21.15.52:0/2755500482 learned_addr learned my addr 172.21.15.52:0/2755500482 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-05-27T07:34:10.802 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:34:10.808+0000 7f924db38640 1 --2- 172.21.15.52:0/2755500482 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x555afcba01b0 0x555afcf41590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:34:10.802 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:34:10.808+0000 7f924db38640 1 --2- 172.21.15.52:0/2755500482 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x555afcb7f4d0 0x555afcf42010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:34:10.803 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:34:10.808+0000 7f924db38640 1 -- 172.21.15.52:0/2755500482 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x555afcb7f4d0 msgr2=0x555afcf42010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:34:10.803 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:34:10.808+0000 7f924db38640 1 --2- 172.21.15.52:0/2755500482 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x555afcb7f4d0 0x555afcf42010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:34:10.803 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:34:10.809+0000 7f924db38640 1 -- 172.21.15.52:0/2755500482 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x555afcb772f0 msgr2=0x555afcf41ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:34:10.803 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:34:10.809+0000 7f924db38640 1 --2- 172.21.15.52:0/2755500482 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x555afcb772f0 0x555afcf41ad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:34:10.803 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:34:10.809+0000 7f924db38640 1 -- 172.21.15.52:0/2755500482 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x555afcbb8210 con 0x555afcba01b0 2024-05-27T07:34:10.804 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:34:10.809+0000 7f924db38640 1 --2- 172.21.15.52:0/2755500482 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x555afcba01b0 0x555afcf41590 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7f9244007b60 tx=0x7f9244007160 comp rx=0 tx=0).ready entity=mon.1 client_cookie=2214bfe27833542 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:34:10.808 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:34:10.814+0000 7f923cbfe640 1 -- 172.21.15.52:0/2755500482 <== mon.1 v2:172.21.15.173:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f924401a070 con 0x555afcba01b0 2024-05-27T07:34:10.808 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:34:10.814+0000 7f923cbfe640 1 -- 172.21.15.52:0/2755500482 <== mon.1 v2:172.21.15.173:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f9244010040 con 0x555afcba01b0 2024-05-27T07:34:10.809 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:34:10.815+0000 7f925744d740 1 -- 172.21.15.52:0/2755500482 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x555afcba01b0 msgr2=0x555afcf41590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:34:10.809 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:34:10.815+0000 7f925744d740 1 --2- 172.21.15.52:0/2755500482 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x555afcba01b0 0x555afcf41590 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7f9244007b60 tx=0x7f9244007160 comp rx=0 tx=0).stop 2024-05-27T07:34:10.812 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:34:10.818+0000 7f925744d740 1 -- 172.21.15.52:0/2755500482 shutdown_connections 2024-05-27T07:34:10.812 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:34:10.818+0000 7f925744d740 1 --2- 172.21.15.52:0/2755500482 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x555afcb7f4d0 0x555afcf42010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:34:10.813 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:34:10.818+0000 7f925744d740 1 --2- 172.21.15.52:0/2755500482 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x555afcb772f0 0x555afcf41ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:34:10.813 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:34:10.818+0000 7f925744d740 1 --2- 172.21.15.52:0/2755500482 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x555afcba01b0 0x555afcf41590 unknown :-1 s=CLOSED pgs=183 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:34:10.813 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:34:10.818+0000 7f925744d740 1 -- 172.21.15.52:0/2755500482 >> 172.21.15.52:0/2755500482 conn(0x555afcb94300 msgr2=0x555afcf3dd40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:34:10.819 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:34:10.824+0000 7f925744d740 1 -- 172.21.15.52:0/2755500482 shutdown_connections 2024-05-27T07:34:10.822 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:34:10.825+0000 7f925744d740 1 -- 172.21.15.52:0/2755500482 wait complete. 2024-05-27T07:34:10.822 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:34:10.825+0000 7f925744d740 1 -- start start 2024-05-27T07:34:10.822 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:34:10.825+0000 7f925744d740 1 Processor -- start 2024-05-27T07:34:10.825 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:34:10.829+0000 7f925744d740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x555afcb772f0 0x555afcf4d630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:34:10.825 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:34:10.829+0000 7f925744d740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x555afcb7f4d0 0x555afcf4db70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:34:10.825 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:34:10.829+0000 7f925744d740 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x555afcba01b0 0x555afcf4bab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:34:10.825 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:34:10.829+0000 7f925744d740 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x555afcb89790 con 0x555afcba01b0 2024-05-27T07:34:10.825 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:34:10.829+0000 7f925744d740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x555afce5f890 con 0x555afcb772f0 2024-05-27T07:34:10.825 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:34:10.829+0000 7f925744d740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x555afcf4e710 con 0x555afcb7f4d0 2024-05-27T07:34:10.825 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:34:10.829+0000 7f925744d740 10 client.0 _ll_register_callbacks cb 0x555afcbb7a00 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T07:34:10.826 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:34:10.830+0000 7f924e339640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x555afcb7f4d0 0x555afcf4db70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:34:10.826 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:34:10.830+0000 7f924e339640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x555afcb7f4d0 0x555afcf4db70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.173:3300/0 says I am v2:172.21.15.52:46050/0 (socket says 172.21.15.52:46050) 2024-05-27T07:34:10.826 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:34:10.830+0000 7f924e339640 1 -- 172.21.15.52:0/3194936130 learned_addr learned my addr 172.21.15.52:0/3194936130 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-05-27T07:34:10.826 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:34:10.830+0000 7f924db38640 1 --2- 172.21.15.52:0/3194936130 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x555afcb772f0 0x555afcf4d630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:34:10.826 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:34:10.830+0000 7f924db38640 1 --2- 172.21.15.52:0/3194936130 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x555afcba01b0 0x555afcf4bab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:34:10.826 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:34:10.831+0000 7f924e339640 1 -- 172.21.15.52:0/3194936130 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x555afcb772f0 msgr2=0x555afcf4d630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:34:10.826 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:34:10.831+0000 7f924e339640 1 --2- 172.21.15.52:0/3194936130 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x555afcb772f0 0x555afcf4d630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:34:10.826 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:34:10.831+0000 7f924e339640 1 -- 172.21.15.52:0/3194936130 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x555afcba01b0 msgr2=0x555afcf4bab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:34:10.826 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:34:10.831+0000 7f924e339640 1 --2- 172.21.15.52:0/3194936130 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x555afcba01b0 0x555afcf4bab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:34:10.826 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:34:10.831+0000 7f924e339640 1 -- 172.21.15.52:0/3194936130 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({osdmap=0}) -- 0x555afcc84620 con 0x555afcb7f4d0 2024-05-27T07:34:10.827 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:34:10.831+0000 7f924e339640 1 --2- 172.21.15.52:0/3194936130 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x555afcb7f4d0 0x555afcf4db70 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7f9238009900 tx=0x7f9238005650 comp rx=0 tx=0).ready entity=mon.1 client_cookie=3f09fe84562223c3 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:34:10.827 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:34:10.831+0000 7f9231ffb640 10 client.0 ms_handle_connect on v2:172.21.15.173:3300/0 2024-05-27T07:34:10.827 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:34:10.832+0000 7f9231ffb640 1 -- 172.21.15.52:0/3194936130 <== mon.1 v2:172.21.15.173:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9238035070 con 0x555afcb7f4d0 2024-05-27T07:34:10.827 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:34:10.832+0000 7f925744d740 1 -- 172.21.15.52:0/3194936130 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f9244010040 con 0x555afcb7f4d0 2024-05-27T07:34:10.828 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:34:10.832+0000 7f9231ffb640 1 -- 172.21.15.52:0/3194936130 <== mon.1 v2:172.21.15.173:3300/0 2 ==== osd_map(120..120 src has 1..120) ==== 9555+0+0 (secure 0 0 0) 0x7f9238004440 con 0x555afcb7f4d0 2024-05-27T07:34:10.828 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:34:10.832+0000 7f9231ffb640 1 -- 172.21.15.52:0/3194936130 <== mon.1 v2:172.21.15.173:3300/0 3 ==== mon_get_version_reply(handle=1 version=105) ==== 24+0+0 (secure 0 0 0) 0x7f923804d050 con 0x555afcb7f4d0 2024-05-27T07:34:10.830 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:34:10.836+0000 7f925744d740 10 client.26485 fetch_fsmap learned FSMap version 105 2024-05-27T07:34:10.830 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:34:10.836+0000 7f925744d740 1 -- 172.21.15.52:0/3194936130 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({config=0+,fsmap=105,monmap=4+}) -- 0x555afcf5ab20 con 0x555afcb7f4d0 2024-05-27T07:34:10.832 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:34:10.837+0000 7f9231ffb640 1 -- 172.21.15.52:0/3194936130 <== mon.1 v2:172.21.15.173:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f9244010040 con 0x555afcb7f4d0 2024-05-27T07:34:10.832 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:34:10.837+0000 7f9231ffb640 1 -- 172.21.15.52:0/3194936130 <== mon.1 v2:172.21.15.173:3300/0 5 ==== fsmap(e 105) ==== 4207+0+0 (secure 0 0 0) 0x7f9238003b70 con 0x555afcb7f4d0 2024-05-27T07:34:10.832 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:34:10.837+0000 7f925744d740 10 client.26485 fetch_fsmap finished waiting for FSMap version 105 2024-05-27T07:34:10.832 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:34:10.837+0000 7f925744d740 10 client.26485 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-05-27T07:34:10.832 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:34:10.837+0000 7f925744d740 20 client.26485 populate_metadata read hostname 'smithi052' 2024-05-27T07:34:10.832 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:34:10.838+0000 7f925744d740 1 --2- 172.21.15.52:0/3194936130 >> [v2:172.21.15.105:6838/500348151,v1:172.21.15.105:6839/500348151] conn(0x555afcb5ed80 0x555afcf5e590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:34:10.832 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:34:10.838+0000 7f925744d740 4 client.26485 mds_command: new command op to 24527 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-27T07:34:10.832 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:34:10.838+0000 7f925744d740 1 -- 172.21.15.52:0/3194936130 --> [v2:172.21.15.105:6838/500348151,v1:172.21.15.105:6839/500348151] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x555afceb5b80 con 0x555afcb5ed80 2024-05-27T07:34:10.838 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:34:10.843+0000 7f924db38640 1 --2- 172.21.15.52:0/3194936130 >> [v2:172.21.15.105:6838/500348151,v1:172.21.15.105:6839/500348151] conn(0x555afcb5ed80 0x555afcf5e590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:34:10.843 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:34:10.849+0000 7f924db38640 1 --2- 172.21.15.52:0/3194936130 >> [v2:172.21.15.105:6838/500348151,v1:172.21.15.105:6839/500348151] conn(0x555afcb5ed80 0x555afcf5e590 crc :-1 s=READY pgs=151 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=67bb84eb713aa139 server_cookie=d855ea76ffd419cb in_seq=0 out_seq=0 2024-05-27T07:34:10.843 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:34:10.849+0000 7f9231ffb640 10 client.26485 ms_handle_connect on v2:172.21.15.105:6838/500348151 2024-05-27T07:34:10.853 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:34:10.858+0000 7f9231ffb640 1 -- 172.21.15.52:0/3194936130 <== mds.0 v2:172.21.15.105:6838/500348151 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24041 (crc 0 0 0) 0x555afceb5b80 con 0x555afcb5ed80 2024-05-27T07:34:10.853 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:34:10.858+0000 7f9231ffb640 10 client.26485 handle_command_reply: tid=0 2024-05-27T07:34:10.853 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:34:10.858+0000 7f925744d740 1 client.26485 shutdown 2024-05-27T07:34:10.858 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:34:10.863+0000 7f925744d740 1 -- 172.21.15.52:0/3194936130 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x555afcb7f4d0 msgr2=0x555afcf4db70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:34:10.858 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:34:10.863+0000 7f925744d740 1 --2- 172.21.15.52:0/3194936130 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x555afcb7f4d0 0x555afcf4db70 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7f9238009900 tx=0x7f9238005650 comp rx=0 tx=0).stop 2024-05-27T07:34:10.860 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:34:10.865+0000 7f925744d740 1 -- 172.21.15.52:0/3194936130 shutdown_connections 2024-05-27T07:34:10.860 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:34:10.865+0000 7f925744d740 1 --2- 172.21.15.52:0/3194936130 >> [v2:172.21.15.105:6838/500348151,v1:172.21.15.105:6839/500348151] conn(0x555afcb5ed80 0x555afcf5e590 crc :-1 s=READY pgs=151 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:34:10.860 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:34:10.865+0000 7f925744d740 1 --2- 172.21.15.52:0/3194936130 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x555afcba01b0 0x555afcf4bab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:34:10.860 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:34:10.865+0000 7f925744d740 1 --2- 172.21.15.52:0/3194936130 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x555afcb7f4d0 0x555afcf4db70 unknown :-1 s=CLOSED pgs=184 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:34:10.860 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:34:10.865+0000 7f925744d740 1 --2- 172.21.15.52:0/3194936130 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x555afcb772f0 0x555afcf4d630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:34:10.860 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:34:10.865+0000 7f925744d740 1 -- 172.21.15.52:0/3194936130 >> 172.21.15.52:0/3194936130 conn(0x555afcb94300 msgr2=0x555afcf41320 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:34:10.860 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:34:10.866+0000 7f9231ffb640 0 client.26485 ms_handle_reset on v2:172.21.15.105:6838/500348151 2024-05-27T07:34:10.861 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:34:10.866+0000 7f925744d740 1 -- 172.21.15.52:0/3194936130 shutdown_connections 2024-05-27T07:34:10.862 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:34:10.868+0000 7f925744d740 1 -- 172.21.15.52:0/3194936130 wait complete. 2024-05-27T07:34:10.862 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:34:10.868+0000 7f925744d740 20 client.26485 trim_cache size 0 max 16384 2024-05-27T07:34:10.895 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:34:10.900+0000 7f925744d740 1 Processor -- start 2024-05-27T07:34:10.895 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:34:10.900+0000 7f925744d740 1 -- start start 2024-05-27T07:34:10.896 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:34:10.901+0000 7f925744d740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x555afcb772f0 0x555afcf3dee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:34:10.896 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:34:10.901+0000 7f925744d740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x555afcb7f4d0 0x555afcf451e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:34:10.896 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:34:10.901+0000 7f925744d740 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x555afcba01b0 0x555afcf45720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:34:10.896 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:34:10.901+0000 7f924db38640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x555afcb772f0 0x555afcf3dee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:34:10.896 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:34:10.901+0000 7f924db38640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x555afcb772f0 0x555afcf3dee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.52:60130/0 (socket says 172.21.15.52:60130) 2024-05-27T07:34:10.896 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:34:10.901+0000 7f924db38640 1 -- 172.21.15.52:0/3413519805 learned_addr learned my addr 172.21.15.52:0/3413519805 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-05-27T07:34:10.896 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:34:10.902+0000 7f925744d740 1 -- 172.21.15.52:0/3413519805 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x7f92400b6bc0 con 0x555afcba01b0 2024-05-27T07:34:10.897 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:34:10.902+0000 7f925744d740 1 -- 172.21.15.52:0/3413519805 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f92400b4fc0 con 0x555afcb772f0 2024-05-27T07:34:10.897 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:34:10.902+0000 7f925744d740 1 -- 172.21.15.52:0/3413519805 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f92400b64c0 con 0x555afcb7f4d0 2024-05-27T07:34:10.899 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:34:10.904+0000 7f924db38640 1 --2- 172.21.15.52:0/3413519805 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x555afcba01b0 0x555afcf45720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:34:10.900 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:34:10.904+0000 7f924db38640 1 -- 172.21.15.52:0/3413519805 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x555afcb7f4d0 msgr2=0x555afcf451e0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-05-27T07:34:10.900 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:34:10.904+0000 7f924db38640 1 --2- 172.21.15.52:0/3413519805 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x555afcb7f4d0 0x555afcf451e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:34:10.900 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:34:10.904+0000 7f924db38640 1 -- 172.21.15.52:0/3413519805 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x555afcba01b0 msgr2=0x555afcf45720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:34:10.900 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:34:10.904+0000 7f924db38640 1 --2- 172.21.15.52:0/3413519805 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x555afcba01b0 0x555afcf45720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:34:10.900 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:34:10.904+0000 7f924db38640 1 -- 172.21.15.52:0/3413519805 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x555afcf4e7d0 con 0x555afcb772f0 2024-05-27T07:34:10.900 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:34:10.906+0000 7f924db38640 1 --2- 172.21.15.52:0/3413519805 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x555afcb772f0 0x555afcf3dee0 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7f9244007b30 tx=0x7f9244007570 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b5fd19169f028bde server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:34:10.901 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:34:10.907+0000 7f9233fff640 1 -- 172.21.15.52:0/3413519805 <== mon.2 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f924401a040 con 0x555afcb772f0 2024-05-27T07:34:10.903 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:34:10.909+0000 7f9233fff640 1 -- 172.21.15.52:0/3413519805 <== mon.2 v2:172.21.15.105:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f9244050040 con 0x555afcb772f0 2024-05-27T07:34:10.904 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:34:10.909+0000 7f9233fff640 1 -- 172.21.15.52:0/3413519805 <== mon.2 v2:172.21.15.105:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f92440147b0 con 0x555afcb772f0 2024-05-27T07:34:10.904 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:34:10.910+0000 7f925744d740 1 -- 172.21.15.52:0/3413519805 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x555afcb772f0 msgr2=0x555afcf3dee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:34:10.904 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:34:10.910+0000 7f925744d740 1 --2- 172.21.15.52:0/3413519805 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x555afcb772f0 0x555afcf3dee0 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7f9244007b30 tx=0x7f9244007570 comp rx=0 tx=0).stop 2024-05-27T07:34:10.904 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:34:10.910+0000 7f925744d740 1 -- 172.21.15.52:0/3413519805 shutdown_connections 2024-05-27T07:34:10.904 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:34:10.910+0000 7f925744d740 1 --2- 172.21.15.52:0/3413519805 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x555afcba01b0 0x555afcf45720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:34:10.904 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:34:10.910+0000 7f925744d740 1 --2- 172.21.15.52:0/3413519805 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x555afcb7f4d0 0x555afcf451e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:34:10.905 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:34:10.910+0000 7f925744d740 1 --2- 172.21.15.52:0/3413519805 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x555afcb772f0 0x555afcf3dee0 unknown :-1 s=CLOSED pgs=233 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:34:10.905 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:34:10.910+0000 7f925744d740 1 -- 172.21.15.52:0/3413519805 >> 172.21.15.52:0/3413519805 conn(0x555afcb94300 msgr2=0x555afcf3d7d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:34:10.911 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:34:10.917+0000 7f925744d740 1 -- 172.21.15.52:0/3413519805 shutdown_connections 2024-05-27T07:34:10.912 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:34:10.917+0000 7f925744d740 1 -- 172.21.15.52:0/3413519805 wait complete. 2024-05-27T07:34:10.912 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:34:10.918+0000 7f925744d740 1 -- start start 2024-05-27T07:34:10.912 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:34:10.918+0000 7f925744d740 1 Processor -- start 2024-05-27T07:34:10.917 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:34:10.923+0000 7f925744d740 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x555afcb772f0 0x555afcf50960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:34:10.917 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:34:10.923+0000 7f925744d740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x555afcb7f4d0 0x555afcf50ea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:34:10.917 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:34:10.923+0000 7f925744d740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x555afcba01b0 0x555afcf513e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:34:10.917 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:34:10.923+0000 7f925744d740 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x7f92400b5d90 con 0x555afcb772f0 2024-05-27T07:34:10.917 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:34:10.923+0000 7f925744d740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f92400b5690 con 0x555afcba01b0 2024-05-27T07:34:10.917 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:34:10.923+0000 7f925744d740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x555afcb5f010 con 0x555afcb7f4d0 2024-05-27T07:34:10.918 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:34:10.923+0000 7f925744d740 10 client.0 _ll_register_callbacks cb 0x7f92380052e0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T07:34:10.919 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:34:10.924+0000 7f924e339640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x555afcb7f4d0 0x555afcf50ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:34:10.919 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:34:10.925+0000 7f924e339640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x555afcb7f4d0 0x555afcf50ea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.173:3300/0 says I am v2:172.21.15.52:46056/0 (socket says 172.21.15.52:46056) 2024-05-27T07:34:10.919 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:34:10.925+0000 7f924e339640 1 -- 172.21.15.52:0/2118421407 learned_addr learned my addr 172.21.15.52:0/2118421407 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-05-27T07:34:10.922 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:34:10.928+0000 7f924e339640 1 -- 172.21.15.52:0/2118421407 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x555afcba01b0 msgr2=0x555afcf513e0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-05-27T07:34:10.922 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:34:10.928+0000 7f924e339640 1 --2- 172.21.15.52:0/2118421407 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x555afcba01b0 0x555afcf513e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:34:10.922 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:34:10.928+0000 7f924e339640 1 -- 172.21.15.52:0/2118421407 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x555afcb772f0 msgr2=0x555afcf50960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:34:10.922 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:34:10.928+0000 7f924e339640 1 --2- 172.21.15.52:0/2118421407 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x555afcb772f0 0x555afcf50960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:34:10.922 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:34:10.928+0000 7f924e339640 1 -- 172.21.15.52:0/2118421407 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({osdmap=0}) -- 0x555afcfc4d20 con 0x555afcb7f4d0 2024-05-27T07:34:10.923 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:34:10.929+0000 7f924e339640 1 --2- 172.21.15.52:0/2118421407 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x555afcb7f4d0 0x555afcf50ea0 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7f9238005ab0 tx=0x7f9238005df0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=57143dc187753e5a server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:34:10.928 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:34:10.934+0000 7f9231ffb640 10 client.0 ms_handle_connect on v2:172.21.15.173:3300/0 2024-05-27T07:34:10.929 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:34:10.934+0000 7f9231ffb640 1 -- 172.21.15.52:0/2118421407 <== mon.1 v2:172.21.15.173:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9238055070 con 0x555afcb7f4d0 2024-05-27T07:34:10.929 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:34:10.934+0000 7f9231ffb640 1 -- 172.21.15.52:0/2118421407 <== mon.1 v2:172.21.15.173:3300/0 2 ==== osd_map(120..120 src has 1..120) ==== 9555+0+0 (secure 0 0 0) 0x7f92380468b0 con 0x555afcb7f4d0 2024-05-27T07:34:10.929 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:34:10.934+0000 7f925744d740 1 -- 172.21.15.52:0/2118421407 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f9244050040 con 0x555afcb7f4d0 2024-05-27T07:34:10.934 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:34:10.939+0000 7f9231ffb640 1 -- 172.21.15.52:0/2118421407 <== mon.1 v2:172.21.15.173:3300/0 3 ==== mon_get_version_reply(handle=1 version=105) ==== 24+0+0 (secure 0 0 0) 0x7f923805e050 con 0x555afcb7f4d0 2024-05-27T07:34:10.934 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:34:10.940+0000 7f925744d740 10 client.26488 fetch_fsmap learned FSMap version 105 2024-05-27T07:34:10.934 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:34:10.940+0000 7f925744d740 1 -- 172.21.15.52:0/2118421407 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({config=0+,fsmap=105,monmap=4+}) -- 0x555afcfc5f90 con 0x555afcb7f4d0 2024-05-27T07:34:10.938 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:34:10.944+0000 7f9231ffb640 1 -- 172.21.15.52:0/2118421407 <== mon.1 v2:172.21.15.173:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f9244050040 con 0x555afcb7f4d0 2024-05-27T07:34:10.938 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:34:10.944+0000 7f9231ffb640 1 -- 172.21.15.52:0/2118421407 <== mon.1 v2:172.21.15.173:3300/0 5 ==== fsmap(e 105) ==== 4207+0+0 (secure 0 0 0) 0x7f9238058020 con 0x555afcb7f4d0 2024-05-27T07:34:10.938 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:34:10.944+0000 7f925744d740 10 client.26488 fetch_fsmap finished waiting for FSMap version 105 2024-05-27T07:34:10.938 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:34:10.944+0000 7f925744d740 10 client.26488 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-05-27T07:34:10.938 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:34:10.944+0000 7f925744d740 20 client.26488 populate_metadata read hostname 'smithi052' 2024-05-27T07:34:10.939 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:34:10.944+0000 7f925744d740 1 --2- 172.21.15.52:0/2118421407 >> [v2:172.21.15.105:6838/500348151,v1:172.21.15.105:6839/500348151] conn(0x555afcb341c0 0x555afcfc8820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:34:10.939 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:34:10.944+0000 7f925744d740 4 client.26488 mds_command: new command op to 24527 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-05-27T07:34:10.939 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:34:10.944+0000 7f925744d740 1 -- 172.21.15.52:0/2118421407 --> [v2:172.21.15.105:6838/500348151,v1:172.21.15.105:6839/500348151] -- command(tid 0: {"prefix": "flush journal"}) -- 0x555afceaf360 con 0x555afcb341c0 2024-05-27T07:34:10.947 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:34:10.953+0000 7f924db38640 1 --2- 172.21.15.52:0/2118421407 >> [v2:172.21.15.105:6838/500348151,v1:172.21.15.105:6839/500348151] conn(0x555afcb341c0 0x555afcfc8820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:34:10.958 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:34:10.963+0000 7f924db38640 1 --2- 172.21.15.52:0/2118421407 >> [v2:172.21.15.105:6838/500348151,v1:172.21.15.105:6839/500348151] conn(0x555afcb341c0 0x555afcfc8820 crc :-1 s=READY pgs=152 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=fbdc8dde2a5db24d server_cookie=1c79f1f0a20ea187 in_seq=0 out_seq=0 2024-05-27T07:34:10.958 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:34:10.964+0000 7f9231ffb640 10 client.26488 ms_handle_connect on v2:172.21.15.105:6838/500348151 2024-05-27T07:34:12.611 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:34:12 smithi173 ceph-mon[37000]: pgmap v279: 129 pgs: 55 active+clean+snaptrim_wait, 19 active+clean+snaptrim, 55 active+clean; 669 MiB data, 4.9 GiB used, 1.0 TiB / 1.0 TiB avail; 485 KiB/s rd, 1.4 MiB/s wr, 791 op/s 2024-05-27T07:34:12.694 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:34:12 smithi105 ceph-mon[35977]: pgmap v279: 129 pgs: 55 active+clean+snaptrim_wait, 19 active+clean+snaptrim, 55 active+clean; 669 MiB data, 4.9 GiB used, 1.0 TiB / 1.0 TiB avail; 485 KiB/s rd, 1.4 MiB/s wr, 791 op/s 2024-05-27T07:34:12.734 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:34:12 smithi052 ceph-mon[31955]: pgmap v279: 129 pgs: 55 active+clean+snaptrim_wait, 19 active+clean+snaptrim, 55 active+clean; 669 MiB data, 4.9 GiB used, 1.0 TiB / 1.0 TiB avail; 485 KiB/s rd, 1.4 MiB/s wr, 791 op/s 2024-05-27T07:34:13.395 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:34:13.399+0000 7f9231ffb640 1 -- 172.21.15.52:0/2118421407 <== mds.0 v2:172.21.15.105:6838/500348151 1 ==== command_reply(tid 0: 0 ) ==== 8+0+25 (crc 0 0 0) 0x555afceaf360 con 0x555afcb341c0 2024-05-27T07:34:13.395 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:34:13.399+0000 7f9231ffb640 10 client.26488 handle_command_reply: tid=0 2024-05-27T07:34:13.396 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:34:13.399+0000 7f925744d740 1 client.26488 shutdown 2024-05-27T07:34:13.396 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:34:13.399+0000 7f925744d740 1 -- 172.21.15.52:0/2118421407 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x555afcb7f4d0 msgr2=0x555afcf50ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:34:13.396 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:34:13.399+0000 7f925744d740 1 --2- 172.21.15.52:0/2118421407 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x555afcb7f4d0 0x555afcf50ea0 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7f9238005ab0 tx=0x7f9238005df0 comp rx=0 tx=0).stop 2024-05-27T07:34:13.396 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:34:13.399+0000 7f925744d740 1 -- 172.21.15.52:0/2118421407 shutdown_connections 2024-05-27T07:34:13.396 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:34:13.399+0000 7f925744d740 1 --2- 172.21.15.52:0/2118421407 >> [v2:172.21.15.105:6838/500348151,v1:172.21.15.105:6839/500348151] conn(0x555afcb341c0 0x555afcfc8820 crc :-1 s=READY pgs=152 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:34:13.396 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:34:13.400+0000 7f925744d740 1 --2- 172.21.15.52:0/2118421407 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x555afcba01b0 0x555afcf513e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:34:13.396 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:34:13.400+0000 7f925744d740 1 --2- 172.21.15.52:0/2118421407 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x555afcb7f4d0 0x555afcf50ea0 secure :-1 s=CLOSED pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7f9238005ab0 tx=0x7f9238005df0 comp rx=0 tx=0).stop 2024-05-27T07:34:13.396 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:34:13.400+0000 7f925744d740 1 --2- 172.21.15.52:0/2118421407 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x555afcb772f0 0x555afcf50960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:34:13.396 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:34:13.400+0000 7f925744d740 1 -- 172.21.15.52:0/2118421407 >> 172.21.15.52:0/2118421407 conn(0x555afcb94300 msgr2=0x555afcf41320 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:34:13.396 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:34:13.400+0000 7f9231ffb640 0 client.26488 ms_handle_reset on v2:172.21.15.105:6838/500348151 2024-05-27T07:34:13.396 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:34:13.400+0000 7f925744d740 1 -- 172.21.15.52:0/2118421407 shutdown_connections 2024-05-27T07:34:13.397 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:34:13.400+0000 7f925744d740 1 -- 172.21.15.52:0/2118421407 wait complete. 2024-05-27T07:34:13.397 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:34:13.400+0000 7f925744d740 20 client.26488 trim_cache size 0 max 16384 2024-05-27T07:34:13.397 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stdout:{ 2024-05-27T07:34:13.397 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stdout: "return_code": 0 2024-05-27T07:34:13.397 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stdout:} 2024-05-27T07:34:13.398 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:34:13.401+0000 7f924f33b640 1 -- 172.21.15.52:0/1393189295 >> [v2:172.21.15.52:6800/1822318384,v1:172.21.15.52:6801/1822318384] conn(0x7f92340b2b70 msgr2=0x7f92340b5080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:34:13.398 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:34:13.401+0000 7f924f33b640 1 --2- 172.21.15.52:0/1393189295 >> [v2:172.21.15.52:6800/1822318384,v1:172.21.15.52:6801/1822318384] conn(0x7f92340b2b70 0x7f92340b5080 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7f924810a3b0 tx=0x7f9248068790 comp rx=0 tx=0).stop 2024-05-27T07:34:13.398 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:34:13.401+0000 7f924f33b640 1 -- 172.21.15.52:0/1393189295 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f924009ad30 msgr2=0x7f92400a3060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:34:13.398 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:34:13.401+0000 7f924f33b640 1 --2- 172.21.15.52:0/1393189295 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f924009ad30 0x7f92400a3060 secure :-1 s=READY pgs=714 cs=0 l=1 rev1=1 crypto rx=0x7f9248069170 tx=0x7f9248078670 comp rx=0 tx=0).stop 2024-05-27T07:34:13.398 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:34:13.401+0000 7f924f33b640 1 -- 172.21.15.52:0/1393189295 shutdown_connections 2024-05-27T07:34:13.398 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:34:13.401+0000 7f924f33b640 1 --2- 172.21.15.52:0/1393189295 >> [v2:172.21.15.52:6800/1822318384,v1:172.21.15.52:6801/1822318384] conn(0x7f92340b2b70 0x7f92340b5080 unknown :-1 s=CLOSED pgs=173 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:34:13.398 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:34:13.401+0000 7f924f33b640 1 --2- 172.21.15.52:0/1393189295 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f924009ad30 0x7f92400a3060 unknown :-1 s=CLOSED pgs=714 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:34:13.398 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:34:13.401+0000 7f924f33b640 1 --2- 172.21.15.52:0/1393189295 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f924009a390 0x7f92400a2b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:34:13.399 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:34:13.401+0000 7f924f33b640 1 --2- 172.21.15.52:0/1393189295 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f9240095940 0x7f92400a9a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:34:13.399 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:34:13.401+0000 7f924f33b640 1 -- 172.21.15.52:0/1393189295 >> 172.21.15.52:0/1393189295 conn(0x7f924000aff0 msgr2=0x7f92400930f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:34:13.399 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:34:13.401+0000 7f924f33b640 1 -- 172.21.15.52:0/1393189295 shutdown_connections 2024-05-27T07:34:13.399 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:34:13.401+0000 7f924f33b640 1 -- 172.21.15.52:0/1393189295 wait complete. 2024-05-27T07:34:14.606 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:34:14 smithi173 ceph-mon[37000]: pgmap v280: 129 pgs: 55 active+clean+snaptrim_wait, 19 active+clean+snaptrim, 55 active+clean; 668 MiB data, 4.9 GiB used, 1.0 TiB / 1.0 TiB avail; 375 KiB/s rd, 1.3 MiB/s wr, 609 op/s 2024-05-27T07:34:14.695 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:34:14 smithi105 ceph-mon[35977]: pgmap v280: 129 pgs: 55 active+clean+snaptrim_wait, 19 active+clean+snaptrim, 55 active+clean; 668 MiB data, 4.9 GiB used, 1.0 TiB / 1.0 TiB avail; 375 KiB/s rd, 1.3 MiB/s wr, 609 op/s 2024-05-27T07:34:14.734 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:34:14 smithi052 ceph-mon[31955]: pgmap v280: 129 pgs: 55 active+clean+snaptrim_wait, 19 active+clean+snaptrim, 55 active+clean; 668 MiB data, 4.9 GiB used, 1.0 TiB / 1.0 TiB avail; 375 KiB/s rd, 1.3 MiB/s wr, 609 op/s 2024-05-27T07:34:16.606 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:34:16 smithi173 ceph-mon[37000]: pgmap v281: 129 pgs: 55 active+clean+snaptrim_wait, 19 active+clean+snaptrim, 55 active+clean; 672 MiB data, 5.0 GiB used, 1.0 TiB / 1.0 TiB avail; 357 KiB/s rd, 2.3 MiB/s wr, 625 op/s 2024-05-27T07:34:16.694 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:34:16 smithi105 ceph-mon[35977]: pgmap v281: 129 pgs: 55 active+clean+snaptrim_wait, 19 active+clean+snaptrim, 55 active+clean; 672 MiB data, 5.0 GiB used, 1.0 TiB / 1.0 TiB avail; 357 KiB/s rd, 2.3 MiB/s wr, 625 op/s 2024-05-27T07:34:16.734 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:34:16 smithi052 ceph-mon[31955]: pgmap v281: 129 pgs: 55 active+clean+snaptrim_wait, 19 active+clean+snaptrim, 55 active+clean; 672 MiB data, 5.0 GiB used, 1.0 TiB / 1.0 TiB avail; 357 KiB/s rd, 2.3 MiB/s wr, 625 op/s 2024-05-27T07:34:18.606 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:34:18 smithi173 ceph-mon[37000]: pgmap v282: 129 pgs: 55 active+clean+snaptrim_wait, 19 active+clean+snaptrim, 55 active+clean; 668 MiB data, 5.0 GiB used, 1.0 TiB / 1.0 TiB avail; 284 KiB/s rd, 2.5 MiB/s wr, 557 op/s 2024-05-27T07:34:18.606 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:34:18 smithi173 ceph-mon[37000]: fsmap cephfs:5 5 up:active 2024-05-27T07:34:18.695 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:34:18 smithi105 ceph-mon[35977]: pgmap v282: 129 pgs: 55 active+clean+snaptrim_wait, 19 active+clean+snaptrim, 55 active+clean; 668 MiB data, 5.0 GiB used, 1.0 TiB / 1.0 TiB avail; 284 KiB/s rd, 2.5 MiB/s wr, 557 op/s 2024-05-27T07:34:18.695 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:34:18 smithi105 ceph-mon[35977]: fsmap cephfs:5 5 up:active 2024-05-27T07:34:18.734 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:34:18 smithi052 ceph-mon[31955]: pgmap v282: 129 pgs: 55 active+clean+snaptrim_wait, 19 active+clean+snaptrim, 55 active+clean; 668 MiB data, 5.0 GiB used, 1.0 TiB / 1.0 TiB avail; 284 KiB/s rd, 2.5 MiB/s wr, 557 op/s 2024-05-27T07:34:18.734 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:34:18 smithi052 ceph-mon[31955]: fsmap cephfs:5 5 up:active 2024-05-27T07:34:20.606 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:34:20 smithi173 ceph-mon[37000]: pgmap v283: 129 pgs: 55 active+clean+snaptrim_wait, 18 active+clean+snaptrim, 56 active+clean; 655 MiB data, 5.0 GiB used, 1.0 TiB / 1.0 TiB avail; 214 KiB/s rd, 2.3 MiB/s wr, 428 op/s 2024-05-27T07:34:20.694 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:34:20 smithi105 ceph-mon[35977]: pgmap v283: 129 pgs: 55 active+clean+snaptrim_wait, 18 active+clean+snaptrim, 56 active+clean; 655 MiB data, 5.0 GiB used, 1.0 TiB / 1.0 TiB avail; 214 KiB/s rd, 2.3 MiB/s wr, 428 op/s 2024-05-27T07:34:20.734 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:34:20 smithi052 ceph-mon[31955]: pgmap v283: 129 pgs: 55 active+clean+snaptrim_wait, 18 active+clean+snaptrim, 56 active+clean; 655 MiB data, 5.0 GiB used, 1.0 TiB / 1.0 TiB avail; 214 KiB/s rd, 2.3 MiB/s wr, 428 op/s 2024-05-27T07:34:22.606 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:34:22 smithi173 ceph-mon[37000]: pgmap v284: 129 pgs: 55 active+clean+snaptrim_wait, 18 active+clean+snaptrim, 56 active+clean; 659 MiB data, 5.0 GiB used, 1.0 TiB / 1.0 TiB avail; 186 KiB/s rd, 2.3 MiB/s wr, 374 op/s 2024-05-27T07:34:22.694 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:34:22 smithi105 ceph-mon[35977]: pgmap v284: 129 pgs: 55 active+clean+snaptrim_wait, 18 active+clean+snaptrim, 56 active+clean; 659 MiB data, 5.0 GiB used, 1.0 TiB / 1.0 TiB avail; 186 KiB/s rd, 2.3 MiB/s wr, 374 op/s 2024-05-27T07:34:22.734 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:34:22 smithi052 ceph-mon[31955]: pgmap v284: 129 pgs: 55 active+clean+snaptrim_wait, 18 active+clean+snaptrim, 56 active+clean; 659 MiB data, 5.0 GiB used, 1.0 TiB / 1.0 TiB avail; 186 KiB/s rd, 2.3 MiB/s wr, 374 op/s 2024-05-27T07:34:24.606 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:34:24 smithi173 ceph-mon[37000]: pgmap v285: 129 pgs: 55 active+clean+snaptrim_wait, 18 active+clean+snaptrim, 56 active+clean; 671 MiB data, 5.0 GiB used, 1.0 TiB / 1.0 TiB avail; 189 KiB/s rd, 3.0 MiB/s wr, 364 op/s 2024-05-27T07:34:24.695 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:34:24 smithi105 ceph-mon[35977]: pgmap v285: 129 pgs: 55 active+clean+snaptrim_wait, 18 active+clean+snaptrim, 56 active+clean; 671 MiB data, 5.0 GiB used, 1.0 TiB / 1.0 TiB avail; 189 KiB/s rd, 3.0 MiB/s wr, 364 op/s 2024-05-27T07:34:24.734 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:34:24 smithi052 ceph-mon[31955]: pgmap v285: 129 pgs: 55 active+clean+snaptrim_wait, 18 active+clean+snaptrim, 56 active+clean; 671 MiB data, 5.0 GiB used, 1.0 TiB / 1.0 TiB avail; 189 KiB/s rd, 3.0 MiB/s wr, 364 op/s 2024-05-27T07:34:26.575 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:34:26.578+0000 7fe9c4314640 1 -- 172.21.15.52:0/3527742554 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fe9bc11d240 msgr2=0x7fe9bc11f660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:34:26.576 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:34:26.578+0000 7fe9c4314640 1 --2- 172.21.15.52:0/3527742554 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fe9bc11d240 0x7fe9bc11f660 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7fe9ac01a910 tx=0x7fe9ac03b4c0 comp rx=0 tx=0).stop 2024-05-27T07:34:26.581 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:34:26.586+0000 7fe9c4314640 1 -- 172.21.15.52:0/3527742554 shutdown_connections 2024-05-27T07:34:26.581 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:34:26.586+0000 7fe9c4314640 1 --2- 172.21.15.52:0/3527742554 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fe9bc11d240 0x7fe9bc11f660 unknown :-1 s=CLOSED pgs=234 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:34:26.581 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:34:26.586+0000 7fe9c4314640 1 --2- 172.21.15.52:0/3527742554 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fe9bc11a8e0 0x7fe9bc11cd00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:34:26.581 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:34:26.586+0000 7fe9c4314640 1 --2- 172.21.15.52:0/3527742554 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fe9bc10a1f0 0x7fe9bc10a600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:34:26.581 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:34:26.586+0000 7fe9c4314640 1 -- 172.21.15.52:0/3527742554 >> 172.21.15.52:0/3527742554 conn(0x7fe9bc06e8f0 msgr2=0x7fe9bc06ed30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:34:26.583 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:34:26.588+0000 7fe9c4314640 1 -- 172.21.15.52:0/3527742554 shutdown_connections 2024-05-27T07:34:26.583 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:34:26.588+0000 7fe9c4314640 1 -- 172.21.15.52:0/3527742554 wait complete. 2024-05-27T07:34:26.584 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:34:26.589+0000 7fe9c4314640 1 Processor -- start 2024-05-27T07:34:26.584 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:34:26.589+0000 7fe9c4314640 1 -- start start 2024-05-27T07:34:26.584 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:34:26.590+0000 7fe9c4314640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fe9bc10a1f0 0x7fe9bc11a300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:34:26.584 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:34:26.590+0000 7fe9c4314640 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fe9bc11a8e0 0x7fe9bc116430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:34:26.584 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:34:26.590+0000 7fe9c4314640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fe9bc11d240 0x7fe9bc116a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:34:26.584 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:34:26.590+0000 7fe9c4314640 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x7fe9bc114910 con 0x7fe9bc11a8e0 2024-05-27T07:34:26.585 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:34:26.590+0000 7fe9c4314640 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7fe9bc114790 con 0x7fe9bc11d240 2024-05-27T07:34:26.585 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:34:26.590+0000 7fe9c4314640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7fe9bc114a90 con 0x7fe9bc10a1f0 2024-05-27T07:34:26.587 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:34:26.592+0000 7fe9c28b1640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fe9bc11d240 0x7fe9bc116a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:34:26.587 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:34:26.593+0000 7fe9c28b1640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fe9bc11d240 0x7fe9bc116a30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.52:57914/0 (socket says 172.21.15.52:57914) 2024-05-27T07:34:26.587 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:34:26.593+0000 7fe9c28b1640 1 -- 172.21.15.52:0/1197919848 learned_addr learned my addr 172.21.15.52:0/1197919848 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-05-27T07:34:26.588 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:34:26.594+0000 7fe9c20b0640 1 --2- 172.21.15.52:0/1197919848 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fe9bc10a1f0 0x7fe9bc11a300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:34:26.588 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:34:26.594+0000 7fe9c18af640 1 --2- 172.21.15.52:0/1197919848 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fe9bc11a8e0 0x7fe9bc116430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:34:26.590 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:34:26.595+0000 7fe9c28b1640 1 -- 172.21.15.52:0/1197919848 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fe9bc10a1f0 msgr2=0x7fe9bc11a300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:34:26.590 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:34:26.595+0000 7fe9c28b1640 1 --2- 172.21.15.52:0/1197919848 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fe9bc10a1f0 0x7fe9bc11a300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:34:26.590 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:34:26.595+0000 7fe9c28b1640 1 -- 172.21.15.52:0/1197919848 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fe9bc11a8e0 msgr2=0x7fe9bc116430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:34:26.590 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:34:26.595+0000 7fe9c28b1640 1 --2- 172.21.15.52:0/1197919848 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fe9bc11a8e0 0x7fe9bc116430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:34:26.591 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:34:26.596+0000 7fe9c28b1640 1 -- 172.21.15.52:0/1197919848 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe9bc117220 con 0x7fe9bc11d240 2024-05-27T07:34:26.592 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:34:26.598+0000 7fe9c20b0640 1 --2- 172.21.15.52:0/1197919848 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fe9bc10a1f0 0x7fe9bc11a300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-27T07:34:26.592 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:34:26.598+0000 7fe9c28b1640 1 --2- 172.21.15.52:0/1197919848 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fe9bc11d240 0x7fe9bc116a30 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7fe9ac004980 tx=0x7fe9ac0049b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:34:26.595 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:34:26.600+0000 7fe9b37fe640 1 -- 172.21.15.52:0/1197919848 <== mon.2 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe9ac04f070 con 0x7fe9bc11d240 2024-05-27T07:34:26.595 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:34:26.600+0000 7fe9b37fe640 1 -- 172.21.15.52:0/1197919848 <== mon.2 v2:172.21.15.105:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fe9ac0409c0 con 0x7fe9bc11d240 2024-05-27T07:34:26.596 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:34:26.602+0000 7fe9b37fe640 1 -- 172.21.15.52:0/1197919848 <== mon.2 v2:172.21.15.105:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe9ac046030 con 0x7fe9bc11d240 2024-05-27T07:34:26.596 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:34:26.602+0000 7fe9c4314640 1 -- 172.21.15.52:0/1197919848 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe9bc1ad200 con 0x7fe9bc11d240 2024-05-27T07:34:26.597 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:34:26.602+0000 7fe9c4314640 1 -- 172.21.15.52:0/1197919848 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe9bc1ad670 con 0x7fe9bc11d240 2024-05-27T07:34:26.606 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:34:26.611+0000 7fe9cb1c7740 1 Processor -- start 2024-05-27T07:34:26.606 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:34:26 smithi173 ceph-mon[37000]: pgmap v286: 129 pgs: 40 active+clean+snaptrim_wait, 14 active+clean+snaptrim, 75 active+clean; 684 MiB data, 5.1 GiB used, 1.0 TiB / 1.0 TiB avail; 179 KiB/s rd, 4.0 MiB/s wr, 321 op/s 2024-05-27T07:34:26.608 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:34:26.614+0000 7fe9b37fe640 1 -- 172.21.15.52:0/1197919848 <== mon.2 v2:172.21.15.105:3300/0 4 ==== mgrmap(e 26) ==== 149783+0+0 (secure 0 0 0) 0x7fe9ac052070 con 0x7fe9bc11d240 2024-05-27T07:34:26.611 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:34:26.616+0000 7fe9cb1c7740 1 -- start start 2024-05-27T07:34:26.612 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:34:26.617+0000 7fe9cb1c7740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55e9a15621b0 0x55e9a19033d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:34:26.612 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:34:26.617+0000 7fe9cb1c7740 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55e9a15392f0 0x55e9a1903910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:34:26.612 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:34:26.617+0000 7fe9cb1c7740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55e9a15414d0 0x55e9a1903e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:34:26.612 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:34:26.617+0000 7fe9cb1c7740 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x55e9a1875c30 con 0x55e9a15392f0 2024-05-27T07:34:26.612 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:34:26.617+0000 7fe9cb1c7740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x55e9a18657d0 con 0x55e9a15414d0 2024-05-27T07:34:26.612 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:34:26.617+0000 7fe9cb1c7740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x55e9a1869810 con 0x55e9a15621b0 2024-05-27T07:34:26.612 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:34:26.618+0000 7fe9b37fe640 1 --2- 172.21.15.52:0/1197919848 >> [v2:172.21.15.52:6800/1822318384,v1:172.21.15.52:6801/1822318384] conn(0x7fe9a80b29d0 0x7fe9a80b4ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:34:26.613 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:34:26.618+0000 7fe9b37fe640 1 -- 172.21.15.52:0/1197919848 <== mon.2 v2:172.21.15.105:3300/0 5 ==== osd_map(120..120 src has 1..120) ==== 9555+0+0 (secure 0 0 0) 0x7fe9ac113b40 con 0x7fe9bc11d240 2024-05-27T07:34:26.614 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:34:26.618+0000 7fe9c20b0640 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55e9a15392f0 0x55e9a1903910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:34:26.614 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:34:26.619+0000 7fe9c20b0640 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55e9a15392f0 0x55e9a1903910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.52:54508/0 (socket says 172.21.15.52:54508) 2024-05-27T07:34:26.614 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:34:26.619+0000 7fe9c20b0640 1 -- 172.21.15.52:0/2105469290 learned_addr learned my addr 172.21.15.52:0/2105469290 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-05-27T07:34:26.614 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:34:26.619+0000 7fe9c18af640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55e9a15414d0 0x55e9a1903e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:34:26.614 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:34:26.619+0000 7fe9c28b1640 1 --2- 172.21.15.52:0/1197919848 >> [v2:172.21.15.52:6800/1822318384,v1:172.21.15.52:6801/1822318384] conn(0x7fe9a80b29d0 0x7fe9a80b4ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:34:26.614 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:34:26.619+0000 7fe9c18af640 1 --2- 172.21.15.52:0/2105469290 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55e9a15621b0 0x55e9a19033d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:34:26.615 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:34:26.619+0000 7fe9c28b1640 1 --2- 172.21.15.52:0/1197919848 >> [v2:172.21.15.52:6800/1822318384,v1:172.21.15.52:6801/1822318384] conn(0x7fe9a80b29d0 0x7fe9a80b4ee0 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7fe9ac110e80 tx=0x7fe9ac03b4c0 comp rx=0 tx=0).ready entity=mgr.15333 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:34:26.615 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:34:26.620+0000 7fe9c18af640 1 -- 172.21.15.52:0/2105469290 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55e9a15621b0 msgr2=0x55e9a19033d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:34:26.615 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:34:26.620+0000 7fe9c18af640 1 --2- 172.21.15.52:0/2105469290 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55e9a15621b0 0x55e9a19033d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:34:26.615 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:34:26.620+0000 7fe9c18af640 1 -- 172.21.15.52:0/2105469290 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55e9a15392f0 msgr2=0x55e9a1903910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:34:26.616 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:34:26.620+0000 7fe9c18af640 1 --2- 172.21.15.52:0/2105469290 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55e9a15392f0 0x55e9a1903910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:34:26.616 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:34:26.620+0000 7fe9c18af640 1 -- 172.21.15.52:0/2105469290 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55e9a157a210 con 0x55e9a15414d0 2024-05-27T07:34:26.616 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:34:26.620+0000 7fe9c20b0640 1 --2- 172.21.15.52:0/2105469290 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55e9a15392f0 0x55e9a1903910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-27T07:34:26.616 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:34:26.620+0000 7fe9c18af640 1 --2- 172.21.15.52:0/2105469290 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55e9a15414d0 0x55e9a1903e50 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7fe9bc11c880 tx=0x7fe9b40099c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=28d23f13630de34f server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:34:26.616 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:34:26.621+0000 7fe9b0bfe640 1 -- 172.21.15.52:0/2105469290 <== mon.2 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe9b4012110 con 0x55e9a15414d0 2024-05-27T07:34:26.616 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:34:26.621+0000 7fe9b0bfe640 1 -- 172.21.15.52:0/2105469290 <== mon.2 v2:172.21.15.105:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fe9b4003e20 con 0x55e9a15414d0 2024-05-27T07:34:26.617 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:34:26.622+0000 7fe9b0bfe640 1 -- 172.21.15.52:0/2105469290 <== mon.2 v2:172.21.15.105:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe9b4004030 con 0x55e9a15414d0 2024-05-27T07:34:26.617 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:34:26.622+0000 7fe9cb1c7740 1 -- 172.21.15.52:0/2105469290 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55e9a15414d0 msgr2=0x55e9a1903e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:34:26.617 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:34:26.622+0000 7fe9cb1c7740 1 --2- 172.21.15.52:0/2105469290 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55e9a15414d0 0x55e9a1903e50 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7fe9bc11c880 tx=0x7fe9b40099c0 comp rx=0 tx=0).stop 2024-05-27T07:34:26.620 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:34:26.626+0000 7fe9cb1c7740 1 -- 172.21.15.52:0/2105469290 shutdown_connections 2024-05-27T07:34:26.620 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:34:26.626+0000 7fe9cb1c7740 1 --2- 172.21.15.52:0/2105469290 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55e9a15414d0 0x55e9a1903e50 unknown :-1 s=CLOSED pgs=236 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:34:26.620 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:34:26.626+0000 7fe9cb1c7740 1 --2- 172.21.15.52:0/2105469290 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55e9a15392f0 0x55e9a1903910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:34:26.620 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:34:26.626+0000 7fe9cb1c7740 1 --2- 172.21.15.52:0/2105469290 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55e9a15621b0 0x55e9a19033d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:34:26.621 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:34:26.626+0000 7fe9cb1c7740 1 -- 172.21.15.52:0/2105469290 >> 172.21.15.52:0/2105469290 conn(0x55e9a1556300 msgr2=0x55e9a18ffb80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:34:26.621 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:34:26.627+0000 7fe9cb1c7740 1 -- 172.21.15.52:0/2105469290 shutdown_connections 2024-05-27T07:34:26.622 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:34:26.627+0000 7fe9cb1c7740 1 -- 172.21.15.52:0/2105469290 wait complete. 2024-05-27T07:34:26.622 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:34:26.628+0000 7fe9cb1c7740 1 -- start start 2024-05-27T07:34:26.622 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:34:26.628+0000 7fe9cb1c7740 1 Processor -- start 2024-05-27T07:34:26.623 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:34:26.628+0000 7fe9cb1c7740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55e9a15392f0 0x55e9a190da30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:34:26.623 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:34:26.628+0000 7fe9cb1c7740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55e9a15414d0 0x55e9a190df70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:34:26.623 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:34:26.628+0000 7fe9cb1c7740 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55e9a15621b0 0x55e9a1913950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:34:26.623 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:34:26.628+0000 7fe9cb1c7740 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x55e9a154b790 con 0x55e9a15621b0 2024-05-27T07:34:26.623 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:34:26.628+0000 7fe9cb1c7740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x55e9a1821890 con 0x55e9a15414d0 2024-05-27T07:34:26.623 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:34:26.629+0000 7fe9cb1c7740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x55e9a19105a0 con 0x55e9a15392f0 2024-05-27T07:34:26.623 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:34:26.629+0000 7fe9cb1c7740 10 client.0 _ll_register_callbacks cb 0x55e9a1579a00 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T07:34:26.625 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:34:26.629+0000 7fe9c20b0640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55e9a15414d0 0x55e9a190df70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:34:26.625 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:34:26.629+0000 7fe9c20b0640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55e9a15414d0 0x55e9a190df70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.52:57934/0 (socket says 172.21.15.52:57934) 2024-05-27T07:34:26.625 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:34:26.629+0000 7fe9c20b0640 1 -- 172.21.15.52:0/1354460984 learned_addr learned my addr 172.21.15.52:0/1354460984 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-05-27T07:34:26.625 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:34:26.630+0000 7fe9c18af640 1 --2- 172.21.15.52:0/1354460984 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55e9a15621b0 0x55e9a1913950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:34:26.625 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:34:26.630+0000 7fe9c20b0640 1 -- 172.21.15.52:0/1354460984 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55e9a15392f0 msgr2=0x55e9a190da30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:34:26.625 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:34:26.630+0000 7fe9c20b0640 1 --2- 172.21.15.52:0/1354460984 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55e9a15392f0 0x55e9a190da30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:34:26.625 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:34:26.630+0000 7fe9c20b0640 1 -- 172.21.15.52:0/1354460984 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55e9a15621b0 msgr2=0x55e9a1913950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:34:26.625 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:34:26.630+0000 7fe9c20b0640 1 --2- 172.21.15.52:0/1354460984 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55e9a15621b0 0x55e9a1913950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:34:26.625 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:34:26.630+0000 7fe9c20b0640 1 -- 172.21.15.52:0/1354460984 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55e9a1646620 con 0x55e9a15414d0 2024-05-27T07:34:26.626 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:34:26.631+0000 7fe9c20b0640 1 --2- 172.21.15.52:0/1354460984 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55e9a15414d0 0x55e9a190df70 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7fe9b80106a0 tx=0x7fe9b8010b60 comp rx=0 tx=0).ready entity=mon.2 client_cookie=54f374ffa33fee17 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:34:26.626 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:34:26.631+0000 7fe9a5ffb640 10 client.0 ms_handle_connect on v2:172.21.15.105:3300/0 2024-05-27T07:34:26.629 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:34:26.635+0000 7fe9a5ffb640 1 -- 172.21.15.52:0/1354460984 <== mon.2 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe9b8010ea0 con 0x55e9a15414d0 2024-05-27T07:34:26.629 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:34:26.635+0000 7fe9cb1c7740 1 -- 172.21.15.52:0/1354460984 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fe9b4003e20 con 0x55e9a15414d0 2024-05-27T07:34:26.630 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:34:26.635+0000 7fe9a5ffb640 1 -- 172.21.15.52:0/1354460984 <== mon.2 v2:172.21.15.105:3300/0 2 ==== osd_map(120..120 src has 1..120) ==== 9555+0+0 (secure 0 0 0) 0x7fe9b8004220 con 0x55e9a15414d0 2024-05-27T07:34:26.630 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:34:26.635+0000 7fe9a5ffb640 1 -- 172.21.15.52:0/1354460984 <== mon.2 v2:172.21.15.105:3300/0 3 ==== mon_get_version_reply(handle=1 version=106) ==== 24+0+0 (secure 0 0 0) 0x7fe9b8024050 con 0x55e9a15414d0 2024-05-27T07:34:26.630 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:34:26.636+0000 7fe9cb1c7740 10 client.26672 fetch_fsmap learned FSMap version 106 2024-05-27T07:34:26.630 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:34:26.636+0000 7fe9cb1c7740 1 -- 172.21.15.52:0/1354460984 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,fsmap=106,monmap=4+}) -- 0x55e9a191c8e0 con 0x55e9a15414d0 2024-05-27T07:34:26.635 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:34:26.641+0000 7fe9a5ffb640 1 -- 172.21.15.52:0/1354460984 <== mon.2 v2:172.21.15.105:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fe9b4003e20 con 0x55e9a15414d0 2024-05-27T07:34:26.635 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:34:26.641+0000 7fe9a5ffb640 1 -- 172.21.15.52:0/1354460984 <== mon.2 v2:172.21.15.105:3300/0 5 ==== fsmap(e 106) ==== 4203+0+0 (secure 0 0 0) 0x7fe9b800f040 con 0x55e9a15414d0 2024-05-27T07:34:26.637 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:34:26.642+0000 7fe9cb1c7740 10 client.26672 fetch_fsmap finished waiting for FSMap version 106 2024-05-27T07:34:26.637 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:34:26.642+0000 7fe9cb1c7740 10 client.26672 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-05-27T07:34:26.637 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:34:26.642+0000 7fe9cb1c7740 20 client.26672 populate_metadata read hostname 'smithi052' 2024-05-27T07:34:26.637 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:34:26.642+0000 7fe9cb1c7740 1 --2- 172.21.15.52:0/1354460984 >> [v2:172.21.15.105:6838/500348151,v1:172.21.15.105:6839/500348151] conn(0x55e9a1520d80 0x55e9a19202f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:34:26.637 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:34:26.642+0000 7fe9cb1c7740 4 client.26672 mds_command: new command op to 24527 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-27T07:34:26.637 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:34:26.642+0000 7fe9cb1c7740 1 -- 172.21.15.52:0/1354460984 --> [v2:172.21.15.105:6838/500348151,v1:172.21.15.105:6839/500348151] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55e9a1877b80 con 0x55e9a1520d80 2024-05-27T07:34:26.641 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:34:26.646+0000 7fe9c18af640 1 --2- 172.21.15.52:0/1354460984 >> [v2:172.21.15.105:6838/500348151,v1:172.21.15.105:6839/500348151] conn(0x55e9a1520d80 0x55e9a19202f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:34:26.647 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:34:26.651+0000 7fe9c18af640 1 --2- 172.21.15.52:0/1354460984 >> [v2:172.21.15.105:6838/500348151,v1:172.21.15.105:6839/500348151] conn(0x55e9a1520d80 0x55e9a19202f0 crc :-1 s=READY pgs=153 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=43190163b0f19caf server_cookie=2b1329f5c20370f2 in_seq=0 out_seq=0 2024-05-27T07:34:26.647 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:34:26.651+0000 7fe9a5ffb640 10 client.26672 ms_handle_connect on v2:172.21.15.105:6838/500348151 2024-05-27T07:34:26.647 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:34:26.652+0000 7fe9a5ffb640 1 -- 172.21.15.52:0/1354460984 <== mds.0 v2:172.21.15.105:6838/500348151 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24041 (crc 0 0 0) 0x55e9a1877b80 con 0x55e9a1520d80 2024-05-27T07:34:26.647 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:34:26.652+0000 7fe9a5ffb640 10 client.26672 handle_command_reply: tid=0 2024-05-27T07:34:26.647 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:34:26.652+0000 7fe9cb1c7740 1 client.26672 shutdown 2024-05-27T07:34:26.648 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:34:26.654+0000 7fe9cb1c7740 1 -- 172.21.15.52:0/1354460984 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55e9a15414d0 msgr2=0x55e9a190df70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:34:26.648 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:34:26.654+0000 7fe9cb1c7740 1 --2- 172.21.15.52:0/1354460984 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55e9a15414d0 0x55e9a190df70 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7fe9b80106a0 tx=0x7fe9b8010b60 comp rx=0 tx=0).stop 2024-05-27T07:34:26.649 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:34:26.654+0000 7fe9cb1c7740 1 -- 172.21.15.52:0/1354460984 shutdown_connections 2024-05-27T07:34:26.649 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:34:26.654+0000 7fe9cb1c7740 1 --2- 172.21.15.52:0/1354460984 >> [v2:172.21.15.105:6838/500348151,v1:172.21.15.105:6839/500348151] conn(0x55e9a1520d80 0x55e9a19202f0 crc :-1 s=READY pgs=153 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:34:26.649 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:34:26.654+0000 7fe9cb1c7740 1 --2- 172.21.15.52:0/1354460984 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55e9a15621b0 0x55e9a1913950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:34:26.649 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:34:26.654+0000 7fe9cb1c7740 1 --2- 172.21.15.52:0/1354460984 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55e9a15414d0 0x55e9a190df70 secure :-1 s=CLOSED pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7fe9b80106a0 tx=0x7fe9b8010b60 comp rx=0 tx=0).stop 2024-05-27T07:34:26.649 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:34:26.654+0000 7fe9cb1c7740 1 --2- 172.21.15.52:0/1354460984 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55e9a15392f0 0x55e9a190da30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:34:26.649 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:34:26.654+0000 7fe9cb1c7740 1 -- 172.21.15.52:0/1354460984 >> 172.21.15.52:0/1354460984 conn(0x55e9a1556300 msgr2=0x55e9a19050f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:34:26.652 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:34:26.656+0000 7fe9a5ffb640 0 client.26672 ms_handle_reset on v2:172.21.15.105:6838/500348151 2024-05-27T07:34:26.655 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:34:26.661+0000 7fe9cb1c7740 1 -- 172.21.15.52:0/1354460984 shutdown_connections 2024-05-27T07:34:26.659 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:34:26.665+0000 7fe9cb1c7740 1 -- 172.21.15.52:0/1354460984 wait complete. 2024-05-27T07:34:26.661 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:34:26.667+0000 7fe9cb1c7740 20 client.26672 trim_cache size 0 max 16384 2024-05-27T07:34:26.694 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:34:26 smithi105 ceph-mon[35977]: pgmap v286: 129 pgs: 40 active+clean+snaptrim_wait, 14 active+clean+snaptrim, 75 active+clean; 684 MiB data, 5.1 GiB used, 1.0 TiB / 1.0 TiB avail; 179 KiB/s rd, 4.0 MiB/s wr, 321 op/s 2024-05-27T07:34:26.695 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:34:26.700+0000 7fe9cb1c7740 1 Processor -- start 2024-05-27T07:34:26.695 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:34:26.701+0000 7fe9cb1c7740 1 -- start start 2024-05-27T07:34:26.695 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:34:26.701+0000 7fe9cb1c7740 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55e9a15392f0 0x55e9a1907080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:34:26.695 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:34:26.701+0000 7fe9cb1c7740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55e9a15621b0 0x55e9a191a5f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:34:26.695 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:34:26.701+0000 7fe9cb1c7740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55e9a14f61c0 0x55e9a19075c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:34:26.696 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:34:26.701+0000 7fe9cb1c7740 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x7fe9bc1b7d80 con 0x55e9a15392f0 2024-05-27T07:34:26.696 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:34:26.701+0000 7fe9cb1c7740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7fe9bc1b61b0 con 0x55e9a14f61c0 2024-05-27T07:34:26.696 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:34:26.701+0000 7fe9cb1c7740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7fe9bc1b7680 con 0x55e9a15621b0 2024-05-27T07:34:26.699 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:34:26.703+0000 7fe9c18af640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55e9a14f61c0 0x55e9a19075c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:34:26.699 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:34:26.703+0000 7fe9c18af640 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55e9a15392f0 0x55e9a1907080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:34:26.699 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:34:26.703+0000 7fe9c18af640 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55e9a15392f0 0x55e9a1907080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.52:54520/0 (socket says 172.21.15.52:54520) 2024-05-27T07:34:26.699 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:34:26.703+0000 7fe9c18af640 1 -- 172.21.15.52:0/4137523442 learned_addr learned my addr 172.21.15.52:0/4137523442 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-05-27T07:34:26.701 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:34:26.706+0000 7fe9c20b0640 1 --2- 172.21.15.52:0/4137523442 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55e9a15621b0 0x55e9a191a5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:34:26.701 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:34:26.706+0000 7fe9c18af640 1 -- 172.21.15.52:0/4137523442 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55e9a15621b0 msgr2=0x55e9a191a5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:34:26.701 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:34:26.706+0000 7fe9c18af640 1 --2- 172.21.15.52:0/4137523442 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55e9a15621b0 0x55e9a191a5f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:34:26.701 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:34:26.706+0000 7fe9c18af640 1 -- 172.21.15.52:0/4137523442 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55e9a15392f0 msgr2=0x55e9a1907080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:34:26.701 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:34:26.706+0000 7fe9c18af640 1 --2- 172.21.15.52:0/4137523442 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55e9a15392f0 0x55e9a1907080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:34:26.701 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:34:26.707+0000 7fe9c18af640 1 -- 172.21.15.52:0/4137523442 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55e9a1910610 con 0x55e9a14f61c0 2024-05-27T07:34:26.707 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:34:26.711+0000 7fe9c18af640 1 --2- 172.21.15.52:0/4137523442 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55e9a14f61c0 0x55e9a19075c0 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7fe9b4000f80 tx=0x7fe9b4017040 comp rx=0 tx=0).ready entity=mon.2 client_cookie=67cb9672aed7c74f server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:34:26.709 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:34:26.713+0000 7fe9a7fff640 1 -- 172.21.15.52:0/4137523442 <== mon.2 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe9b4003bf0 con 0x55e9a14f61c0 2024-05-27T07:34:26.709 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:34:26.713+0000 7fe9a7fff640 1 -- 172.21.15.52:0/4137523442 <== mon.2 v2:172.21.15.105:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fe9b4003d90 con 0x55e9a14f61c0 2024-05-27T07:34:26.709 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:34:26.713+0000 7fe9a7fff640 1 -- 172.21.15.52:0/4137523442 <== mon.2 v2:172.21.15.105:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe9b40056b0 con 0x55e9a14f61c0 2024-05-27T07:34:26.709 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:34:26.714+0000 7fe9cb1c7740 1 -- 172.21.15.52:0/4137523442 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55e9a14f61c0 msgr2=0x55e9a19075c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:34:26.709 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:34:26.714+0000 7fe9cb1c7740 1 --2- 172.21.15.52:0/4137523442 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55e9a14f61c0 0x55e9a19075c0 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7fe9b4000f80 tx=0x7fe9b4017040 comp rx=0 tx=0).stop 2024-05-27T07:34:26.710 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:34:26.714+0000 7fe9cb1c7740 1 -- 172.21.15.52:0/4137523442 shutdown_connections 2024-05-27T07:34:26.710 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:34:26.714+0000 7fe9cb1c7740 1 --2- 172.21.15.52:0/4137523442 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55e9a14f61c0 0x55e9a19075c0 unknown :-1 s=CLOSED pgs=238 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:34:26.710 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:34:26.714+0000 7fe9cb1c7740 1 --2- 172.21.15.52:0/4137523442 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55e9a15621b0 0x55e9a191a5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:34:26.710 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:34:26.714+0000 7fe9cb1c7740 1 --2- 172.21.15.52:0/4137523442 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55e9a15392f0 0x55e9a1907080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:34:26.710 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:34:26.714+0000 7fe9cb1c7740 1 -- 172.21.15.52:0/4137523442 >> 172.21.15.52:0/4137523442 conn(0x55e9a1556300 msgr2=0x55e9a190c0f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:34:26.710 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:34:26.715+0000 7fe9cb1c7740 1 -- 172.21.15.52:0/4137523442 shutdown_connections 2024-05-27T07:34:26.710 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:34:26.715+0000 7fe9cb1c7740 1 -- 172.21.15.52:0/4137523442 wait complete. 2024-05-27T07:34:26.711 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:34:26.715+0000 7fe9cb1c7740 1 -- start start 2024-05-27T07:34:26.711 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:34:26.715+0000 7fe9cb1c7740 1 Processor -- start 2024-05-27T07:34:26.711 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:34:26.716+0000 7fe9cb1c7740 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55e9a14f61c0 0x55e9a191a120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:34:26.711 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:34:26.716+0000 7fe9cb1c7740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55e9a15392f0 0x55e9a19134a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:34:26.711 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:34:26.716+0000 7fe9cb1c7740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55e9a15621b0 0x55e9a19139e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:34:26.711 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:34:26.716+0000 7fe9cb1c7740 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x7fe9bc1b6f80 con 0x55e9a14f61c0 2024-05-27T07:34:26.711 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:34:26.716+0000 7fe9cb1c7740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7fe9bc1b6880 con 0x55e9a15621b0 2024-05-27T07:34:26.711 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:34:26.716+0000 7fe9cb1c7740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x55e9a190bd70 con 0x55e9a15392f0 2024-05-27T07:34:26.711 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:34:26.716+0000 7fe9cb1c7740 10 client.0 _ll_register_callbacks cb 0x7fe9b80093a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T07:34:26.712 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:34:26.716+0000 7fe9c20b0640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55e9a15392f0 0x55e9a19134a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:34:26.712 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:34:26.716+0000 7fe9c20b0640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55e9a15392f0 0x55e9a19134a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.173:3300/0 says I am v2:172.21.15.52:49874/0 (socket says 172.21.15.52:49874) 2024-05-27T07:34:26.712 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:34:26.716+0000 7fe9c20b0640 1 -- 172.21.15.52:0/938601324 learned_addr learned my addr 172.21.15.52:0/938601324 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-05-27T07:34:26.712 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:34:26.717+0000 7fe9c18af640 1 --2- 172.21.15.52:0/938601324 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55e9a15621b0 0x55e9a19139e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:34:26.712 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:34:26.717+0000 7fe9c20b0640 1 -- 172.21.15.52:0/938601324 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55e9a15621b0 msgr2=0x55e9a19139e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:34:26.713 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:34:26.717+0000 7fe9c20b0640 1 --2- 172.21.15.52:0/938601324 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55e9a15621b0 0x55e9a19139e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:34:26.713 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:34:26.717+0000 7fe9c20b0640 1 -- 172.21.15.52:0/938601324 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55e9a14f61c0 msgr2=0x55e9a191a120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:34:26.713 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:34:26.717+0000 7fe9c20b0640 1 --2- 172.21.15.52:0/938601324 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55e9a14f61c0 0x55e9a191a120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:34:26.713 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:34:26.717+0000 7fe9c20b0640 1 -- 172.21.15.52:0/938601324 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55e9a1913f20 con 0x55e9a15392f0 2024-05-27T07:34:26.713 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:34:26.717+0000 7fe9c18af640 1 --2- 172.21.15.52:0/938601324 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55e9a15621b0 0x55e9a19139e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-27T07:34:26.713 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:34:26.717+0000 7fe9c20b0640 1 --2- 172.21.15.52:0/938601324 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55e9a15392f0 0x55e9a19134a0 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7fe9b8003be0 tx=0x7fe9b8009da0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=aa797e5bf2ca56a6 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:34:26.717 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:34:26.721+0000 7fe9a5ffb640 10 client.0 ms_handle_connect on v2:172.21.15.173:3300/0 2024-05-27T07:34:26.717 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:34:26.721+0000 7fe9a5ffb640 1 -- 172.21.15.52:0/938601324 <== mon.1 v2:172.21.15.173:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe9b80037f0 con 0x55e9a15392f0 2024-05-27T07:34:26.718 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:34:26.721+0000 7fe9a5ffb640 1 -- 172.21.15.52:0/938601324 <== mon.1 v2:172.21.15.173:3300/0 2 ==== osd_map(120..120 src has 1..120) ==== 9555+0+0 (secure 0 0 0) 0x7fe9b800e8d0 con 0x55e9a15392f0 2024-05-27T07:34:26.718 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:34:26.722+0000 7fe9cb1c7740 1 -- 172.21.15.52:0/938601324 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fe9b4003d90 con 0x55e9a15392f0 2024-05-27T07:34:26.721 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:34:26.727+0000 7fe9a5ffb640 1 -- 172.21.15.52:0/938601324 <== mon.1 v2:172.21.15.173:3300/0 3 ==== mon_get_version_reply(handle=1 version=106) ==== 24+0+0 (secure 0 0 0) 0x7fe9b8038050 con 0x55e9a15392f0 2024-05-27T07:34:26.726 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:34:26.732+0000 7fe9cb1c7740 10 client.26500 fetch_fsmap learned FSMap version 106 2024-05-27T07:34:26.726 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:34:26.732+0000 7fe9cb1c7740 1 -- 172.21.15.52:0/938601324 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({config=0+,fsmap=106,monmap=4+}) -- 0x55e9a1918830 con 0x55e9a15392f0 2024-05-27T07:34:26.730 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:34:26.736+0000 7fe9a5ffb640 1 -- 172.21.15.52:0/938601324 <== mon.1 v2:172.21.15.173:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fe9b4003d90 con 0x55e9a15392f0 2024-05-27T07:34:26.730 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:34:26.736+0000 7fe9a5ffb640 1 -- 172.21.15.52:0/938601324 <== mon.1 v2:172.21.15.173:3300/0 5 ==== fsmap(e 106) ==== 4203+0+0 (secure 0 0 0) 0x7fe9b80042e0 con 0x55e9a15392f0 2024-05-27T07:34:26.730 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:34:26.736+0000 7fe9cb1c7740 10 client.26500 fetch_fsmap finished waiting for FSMap version 106 2024-05-27T07:34:26.730 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:34:26.736+0000 7fe9cb1c7740 10 client.26500 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-05-27T07:34:26.731 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:34:26.736+0000 7fe9cb1c7740 20 client.26500 populate_metadata read hostname 'smithi052' 2024-05-27T07:34:26.731 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:34:26.736+0000 7fe9cb1c7740 1 --2- 172.21.15.52:0/938601324 >> [v2:172.21.15.105:6838/500348151,v1:172.21.15.105:6839/500348151] conn(0x55e9a19aada0 0x55e9a19ad1c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:34:26.731 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:34:26.736+0000 7fe9cb1c7740 4 client.26500 mds_command: new command op to 24527 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-05-27T07:34:26.731 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:34:26.736+0000 7fe9cb1c7740 1 -- 172.21.15.52:0/938601324 --> [v2:172.21.15.105:6838/500348151,v1:172.21.15.105:6839/500348151] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55e9a1871360 con 0x55e9a19aada0 2024-05-27T07:34:26.734 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:34:26 smithi052 ceph-mon[31955]: pgmap v286: 129 pgs: 40 active+clean+snaptrim_wait, 14 active+clean+snaptrim, 75 active+clean; 684 MiB data, 5.1 GiB used, 1.0 TiB / 1.0 TiB avail; 179 KiB/s rd, 4.0 MiB/s wr, 321 op/s 2024-05-27T07:34:26.738 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:34:26.743+0000 7fe9c18af640 1 --2- 172.21.15.52:0/938601324 >> [v2:172.21.15.105:6838/500348151,v1:172.21.15.105:6839/500348151] conn(0x55e9a19aada0 0x55e9a19ad1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:34:26.747 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:34:26.752+0000 7fe9a5ffb640 10 client.26500 ms_handle_connect on v2:172.21.15.105:6838/500348151 2024-05-27T07:34:26.747 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:34:26.752+0000 7fe9c18af640 1 --2- 172.21.15.52:0/938601324 >> [v2:172.21.15.105:6838/500348151,v1:172.21.15.105:6839/500348151] conn(0x55e9a19aada0 0x55e9a19ad1c0 crc :-1 s=READY pgs=154 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=95a18311e6c47e3b server_cookie=b331def6d1859d66 in_seq=0 out_seq=0 2024-05-27T07:34:28.231 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:34:28.235+0000 7fe9a5ffb640 1 -- 172.21.15.52:0/938601324 <== mds.0 v2:172.21.15.105:6838/500348151 1 ==== command_reply(tid 0: 0 ) ==== 8+0+25 (crc 0 0 0) 0x55e9a1871360 con 0x55e9a19aada0 2024-05-27T07:34:28.231 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:34:28.235+0000 7fe9a5ffb640 10 client.26500 handle_command_reply: tid=0 2024-05-27T07:34:28.231 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:34:28.235+0000 7fe9cb1c7740 1 client.26500 shutdown 2024-05-27T07:34:28.231 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:34:28.236+0000 7fe9cb1c7740 1 -- 172.21.15.52:0/938601324 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55e9a15392f0 msgr2=0x55e9a19134a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:34:28.232 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:34:28.236+0000 7fe9cb1c7740 1 --2- 172.21.15.52:0/938601324 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55e9a15392f0 0x55e9a19134a0 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7fe9b8003be0 tx=0x7fe9b8009da0 comp rx=0 tx=0).stop 2024-05-27T07:34:28.232 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:34:28.236+0000 7fe9cb1c7740 1 -- 172.21.15.52:0/938601324 shutdown_connections 2024-05-27T07:34:28.232 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:34:28.236+0000 7fe9cb1c7740 1 --2- 172.21.15.52:0/938601324 >> [v2:172.21.15.105:6838/500348151,v1:172.21.15.105:6839/500348151] conn(0x55e9a19aada0 0x55e9a19ad1c0 crc :-1 s=READY pgs=154 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:34:28.232 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:34:28.236+0000 7fe9cb1c7740 1 --2- 172.21.15.52:0/938601324 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55e9a15621b0 0x55e9a19139e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:34:28.232 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:34:28.236+0000 7fe9cb1c7740 1 --2- 172.21.15.52:0/938601324 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55e9a15392f0 0x55e9a19134a0 unknown :-1 s=CLOSED pgs=186 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:34:28.232 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:34:28.236+0000 7fe9cb1c7740 1 --2- 172.21.15.52:0/938601324 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55e9a14f61c0 0x55e9a191a120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:34:28.232 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:34:28.236+0000 7fe9cb1c7740 1 -- 172.21.15.52:0/938601324 >> 172.21.15.52:0/938601324 conn(0x55e9a1556300 msgr2=0x55e9a1903160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:34:28.232 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:34:28.236+0000 7fe9a5ffb640 0 client.26500 ms_handle_reset on v2:172.21.15.105:6838/500348151 2024-05-27T07:34:28.232 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:34:28.236+0000 7fe9cb1c7740 1 -- 172.21.15.52:0/938601324 shutdown_connections 2024-05-27T07:34:28.233 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:34:28.236+0000 7fe9cb1c7740 1 -- 172.21.15.52:0/938601324 wait complete. 2024-05-27T07:34:28.233 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:34:28.237+0000 7fe9cb1c7740 20 client.26500 trim_cache size 0 max 16384 2024-05-27T07:34:28.233 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stdout:{ 2024-05-27T07:34:28.233 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stdout: "return_code": 0 2024-05-27T07:34:28.233 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stdout:} 2024-05-27T07:34:28.234 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:34:28.238+0000 7fe9c4314640 1 -- 172.21.15.52:0/1197919848 >> [v2:172.21.15.52:6800/1822318384,v1:172.21.15.52:6801/1822318384] conn(0x7fe9a80b29d0 msgr2=0x7fe9a80b4ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:34:28.235 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:34:28.238+0000 7fe9c4314640 1 --2- 172.21.15.52:0/1197919848 >> [v2:172.21.15.52:6800/1822318384,v1:172.21.15.52:6801/1822318384] conn(0x7fe9a80b29d0 0x7fe9a80b4ee0 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7fe9ac110e80 tx=0x7fe9ac03b4c0 comp rx=0 tx=0).stop 2024-05-27T07:34:28.235 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:34:28.238+0000 7fe9c4314640 1 -- 172.21.15.52:0/1197919848 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fe9bc11d240 msgr2=0x7fe9bc116a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:34:28.235 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:34:28.238+0000 7fe9c4314640 1 --2- 172.21.15.52:0/1197919848 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fe9bc11d240 0x7fe9bc116a30 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7fe9ac004980 tx=0x7fe9ac0049b0 comp rx=0 tx=0).stop 2024-05-27T07:34:28.235 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:34:28.238+0000 7fe9c4314640 1 -- 172.21.15.52:0/1197919848 shutdown_connections 2024-05-27T07:34:28.235 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:34:28.238+0000 7fe9c4314640 1 --2- 172.21.15.52:0/1197919848 >> [v2:172.21.15.52:6800/1822318384,v1:172.21.15.52:6801/1822318384] conn(0x7fe9a80b29d0 0x7fe9a80b4ee0 unknown :-1 s=CLOSED pgs=174 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:34:28.235 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:34:28.238+0000 7fe9c4314640 1 --2- 172.21.15.52:0/1197919848 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fe9bc11d240 0x7fe9bc116a30 unknown :-1 s=CLOSED pgs=235 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:34:28.235 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:34:28.238+0000 7fe9c4314640 1 --2- 172.21.15.52:0/1197919848 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fe9bc11a8e0 0x7fe9bc116430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:34:28.235 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:34:28.238+0000 7fe9c4314640 1 --2- 172.21.15.52:0/1197919848 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fe9bc10a1f0 0x7fe9bc11a300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:34:28.235 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:34:28.238+0000 7fe9c4314640 1 -- 172.21.15.52:0/1197919848 >> 172.21.15.52:0/1197919848 conn(0x7fe9bc06e8f0 msgr2=0x7fe9bc11acf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:34:28.235 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:34:28.239+0000 7fe9c4314640 1 -- 172.21.15.52:0/1197919848 shutdown_connections 2024-05-27T07:34:28.235 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:34:28.239+0000 7fe9c4314640 1 -- 172.21.15.52:0/1197919848 wait complete. 2024-05-27T07:34:28.606 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:34:28 smithi173 ceph-mon[37000]: pgmap v287: 129 pgs: 24 active+clean+snaptrim_wait, 10 active+clean+snaptrim, 95 active+clean; 683 MiB data, 5.1 GiB used, 1.0 TiB / 1.0 TiB avail; 236 KiB/s rd, 2.9 MiB/s wr, 332 op/s 2024-05-27T07:34:28.694 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:34:28 smithi105 ceph-mon[35977]: pgmap v287: 129 pgs: 24 active+clean+snaptrim_wait, 10 active+clean+snaptrim, 95 active+clean; 683 MiB data, 5.1 GiB used, 1.0 TiB / 1.0 TiB avail; 236 KiB/s rd, 2.9 MiB/s wr, 332 op/s 2024-05-27T07:34:28.734 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:34:28 smithi052 ceph-mon[31955]: pgmap v287: 129 pgs: 24 active+clean+snaptrim_wait, 10 active+clean+snaptrim, 95 active+clean; 683 MiB data, 5.1 GiB used, 1.0 TiB / 1.0 TiB avail; 236 KiB/s rd, 2.9 MiB/s wr, 332 op/s 2024-05-27T07:34:29.561 DEBUG:teuthology.orchestra.run.smithi052:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:38f9d900c83c63799fbdbe61acc9a11b0d3554a6 shell --fsid 73df2636-1bf8-11ef-bc9b-c7b262605968 -- ceph tell mds.1:0 scrub status 2024-05-27T07:34:30.026 INFO:teuthology.orchestra.run.smithi052.stderr:Inferring config /var/lib/ceph/73df2636-1bf8-11ef-bc9b-c7b262605968/mon.a/config 2024-05-27T07:34:30.475 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:34:30 smithi052 ceph-mon[31955]: pgmap v288: 129 pgs: 24 active+clean+snaptrim_wait, 10 active+clean+snaptrim, 95 active+clean; 674 MiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 258 KiB/s rd, 3.2 MiB/s wr, 392 op/s 2024-05-27T07:34:30.606 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:34:30 smithi173 ceph-mon[37000]: pgmap v288: 129 pgs: 24 active+clean+snaptrim_wait, 10 active+clean+snaptrim, 95 active+clean; 674 MiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 258 KiB/s rd, 3.2 MiB/s wr, 392 op/s 2024-05-27T07:34:30.694 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:34:30 smithi105 ceph-mon[35977]: pgmap v288: 129 pgs: 24 active+clean+snaptrim_wait, 10 active+clean+snaptrim, 95 active+clean; 674 MiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 258 KiB/s rd, 3.2 MiB/s wr, 392 op/s 2024-05-27T07:34:32.342 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:34:32.345+0000 7f1209d5f700 1 -- 172.21.15.52:0/625073721 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f11fc09cc10 msgr2=0x7f11fc09d020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:34:32.342 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:34:32.345+0000 7f1209d5f700 1 --2- 172.21.15.52:0/625073721 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f11fc09cc10 0x7f11fc09d020 secure :-1 s=READY pgs=715 cs=0 l=1 rev1=1 crypto rx=0x7f11f801a9a0 tx=0x7f11f801acb0 comp rx=0 tx=0).stop 2024-05-27T07:34:32.343 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:34:32.345+0000 7f1209d5f700 1 -- 172.21.15.52:0/625073721 shutdown_connections 2024-05-27T07:34:32.343 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:34:32.345+0000 7f1209d5f700 1 --2- 172.21.15.52:0/625073721 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f11fc09d560 0x7f11fc0a99d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:34:32.343 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:34:32.345+0000 7f1209d5f700 1 --2- 172.21.15.52:0/625073721 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f11fc0a2c50 0x7f11fc0a3060 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:34:32.343 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:34:32.345+0000 7f1209d5f700 1 --2- 172.21.15.52:0/625073721 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f11fc09cc10 0x7f11fc09d020 unknown :-1 s=CLOSED pgs=715 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:34:32.343 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:34:32.345+0000 7f1209d5f700 1 -- 172.21.15.52:0/625073721 >> 172.21.15.52:0/625073721 conn(0x7f11fc009e40 msgr2=0x7f11fc00a250 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:34:32.343 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:34:32.346+0000 7f1209d5f700 1 -- 172.21.15.52:0/625073721 shutdown_connections 2024-05-27T07:34:32.343 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:34:32.346+0000 7f1209d5f700 1 -- 172.21.15.52:0/625073721 wait complete. 2024-05-27T07:34:32.343 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:34:32.347+0000 7f1209d5f700 1 Processor -- start 2024-05-27T07:34:32.343 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:34:32.347+0000 7f1209d5f700 1 -- start start 2024-05-27T07:34:32.351 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:34:32.347+0000 7f1209d5f700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f11fc09d560 0x7f11fc00f1d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:34:32.351 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:34:32.347+0000 7f1209d5f700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f11fc0a2c50 0x7f11fc00f770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:34:32.351 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:34:32.347+0000 7f1209d5f700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f11fc013a00 0x7f11fc013e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:34:32.351 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:34:32.347+0000 7f1209d5f700 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x7f11fc00da90 con 0x7f11fc013a00 2024-05-27T07:34:32.352 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:34:32.347+0000 7f1209d5f700 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f11fc00d910 con 0x7f11fc0a2c50 2024-05-27T07:34:32.352 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:34:32.347+0000 7f1209d5f700 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f11fc00dc10 con 0x7f11fc09d560 2024-05-27T07:34:32.352 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:34:32.348+0000 7f12037fe700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f11fc09d560 0x7f11fc00f1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:34:32.352 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:34:32.348+0000 7f12037fe700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f11fc09d560 0x7f11fc00f1d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.173:3300/0 says I am v2:172.21.15.52:50240/0 (socket says 172.21.15.52:50240) 2024-05-27T07:34:32.352 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:34:32.348+0000 7f12037fe700 1 -- 172.21.15.52:0/3379767108 learned_addr learned my addr 172.21.15.52:0/3379767108 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-05-27T07:34:32.352 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:34:32.348+0000 7f1202ffd700 1 --2- 172.21.15.52:0/3379767108 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f11fc0a2c50 0x7f11fc00f770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:34:32.352 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:34:32.348+0000 7f12037fe700 1 -- 172.21.15.52:0/3379767108 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f11fc0a2c50 msgr2=0x7f11fc00f770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:34:32.352 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:34:32.348+0000 7f12037fe700 1 --2- 172.21.15.52:0/3379767108 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f11fc0a2c50 0x7f11fc00f770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:34:32.352 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:34:32.348+0000 7f12037fe700 1 -- 172.21.15.52:0/3379767108 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f11fc013a00 msgr2=0x7f11fc013e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:34:32.352 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:34:32.348+0000 7f12037fe700 1 --2- 172.21.15.52:0/3379767108 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f11fc013a00 0x7f11fc013e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:34:32.352 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:34:32.348+0000 7f12037fe700 1 -- 172.21.15.52:0/3379767108 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f11fc014540 con 0x7f11fc09d560 2024-05-27T07:34:32.352 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:34:32.349+0000 7f12037fe700 1 --2- 172.21.15.52:0/3379767108 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f11fc09d560 0x7f11fc00f1d0 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7f1204052c50 tx=0x7f12040756c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:34:32.353 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:34:32.352+0000 7f1200ff9700 1 -- 172.21.15.52:0/3379767108 <== mon.1 v2:172.21.15.173:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f120406c310 con 0x7f11fc09d560 2024-05-27T07:34:32.353 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:34:32.352+0000 7f1209d5f700 1 -- 172.21.15.52:0/3379767108 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f11fc014830 con 0x7f11fc09d560 2024-05-27T07:34:32.353 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:34:32.352+0000 7f1209d5f700 1 -- 172.21.15.52:0/3379767108 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f11fc0b2bc0 con 0x7f11fc09d560 2024-05-27T07:34:32.353 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:34:32.352+0000 7f1200ff9700 1 -- 172.21.15.52:0/3379767108 <== mon.1 v2:172.21.15.173:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f120406c4b0 con 0x7f11fc09d560 2024-05-27T07:34:32.353 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:34:32.354+0000 7f1200ff9700 1 -- 172.21.15.52:0/3379767108 <== mon.1 v2:172.21.15.173:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f120407cdc0 con 0x7f11fc09d560 2024-05-27T07:34:32.359 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:34:32.362+0000 7f1218774b80 1 Processor -- start 2024-05-27T07:34:32.359 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:34:32.362+0000 7f1200ff9700 1 -- 172.21.15.52:0/3379767108 <== mon.1 v2:172.21.15.173:3300/0 4 ==== mgrmap(e 26) ==== 149783+0+0 (secure 0 0 0) 0x7f1204086350 con 0x7f11fc09d560 2024-05-27T07:34:32.360 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:34:32.363+0000 7f1200ff9700 1 --2- 172.21.15.52:0/3379767108 >> [v2:172.21.15.52:6800/1822318384,v1:172.21.15.52:6801/1822318384] conn(0x7f11f40b2bf0 0x7f11f40b5100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:34:32.360 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:34:32.363+0000 7f1200ff9700 1 -- 172.21.15.52:0/3379767108 <== mon.1 v2:172.21.15.173:3300/0 5 ==== osd_map(120..120 src has 1..120) ==== 9555+0+0 (secure 0 0 0) 0x7f12041475f0 con 0x7f11fc09d560 2024-05-27T07:34:32.365 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:34:32.367+0000 7f1218774b80 1 -- start start 2024-05-27T07:34:32.365 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:34:32.367+0000 7f1203fff700 1 --2- 172.21.15.52:0/3379767108 >> [v2:172.21.15.52:6800/1822318384,v1:172.21.15.52:6801/1822318384] conn(0x7f11f40b2bf0 0x7f11f40b5100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:34:32.365 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:34:32.367+0000 7f1218774b80 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x5634ea8cde40 0x5634eaa4c010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:34:32.365 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:34:32.367+0000 7f1218774b80 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5634eaa4c550 0x5634eaa4c960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:34:32.365 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:34:32.367+0000 7f1218774b80 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5634eaa56de0 0x5634eaa59200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:34:32.365 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:34:32.367+0000 7f1218774b80 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x5634ea97ad70 con 0x5634ea8cde40 2024-05-27T07:34:32.366 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:34:32.367+0000 7f1218774b80 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x5634ea96a910 con 0x5634eaa4c550 2024-05-27T07:34:32.366 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:34:32.367+0000 7f1218774b80 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x5634ea96e950 con 0x5634eaa56de0 2024-05-27T07:34:32.366 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:34:32.368+0000 7f12037fe700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x5634ea8cde40 0x5634eaa4c010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:34:32.366 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:34:32.368+0000 7f12037fe700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x5634ea8cde40 0x5634eaa4c010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.52:46744/0 (socket says 172.21.15.52:46744) 2024-05-27T07:34:32.366 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:34:32.368+0000 7f12037fe700 1 -- 172.21.15.52:0/4169664340 learned_addr learned my addr 172.21.15.52:0/4169664340 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-05-27T07:34:32.366 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:34:32.368+0000 7f12037fe700 1 -- 172.21.15.52:0/4169664340 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5634eaa56de0 msgr2=0x5634eaa59200 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-05-27T07:34:32.366 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:34:32.368+0000 7f12037fe700 1 --2- 172.21.15.52:0/4169664340 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5634eaa56de0 0x5634eaa59200 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:34:32.366 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:34:32.368+0000 7f12037fe700 1 -- 172.21.15.52:0/4169664340 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5634eaa4c550 msgr2=0x5634eaa4c960 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-05-27T07:34:32.366 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:34:32.368+0000 7f12037fe700 1 --2- 172.21.15.52:0/4169664340 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5634eaa4c550 0x5634eaa4c960 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:34:32.366 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:34:32.368+0000 7f12037fe700 1 -- 172.21.15.52:0/4169664340 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5634ea913fb0 con 0x5634ea8cde40 2024-05-27T07:34:32.366 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:34:32.369+0000 7f1203fff700 1 --2- 172.21.15.52:0/3379767108 >> [v2:172.21.15.52:6800/1822318384,v1:172.21.15.52:6801/1822318384] conn(0x7f11f40b2bf0 0x7f11f40b5100 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7f11f0005950 tx=0x7f11f00058e0 comp rx=0 tx=0).ready entity=mgr.15333 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:34:32.367 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:34:32.372+0000 7f12037fe700 1 --2- 172.21.15.52:0/4169664340 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x5634ea8cde40 0x5634eaa4c010 secure :-1 s=READY pgs=716 cs=0 l=1 rev1=1 crypto rx=0x7f1204052100 tx=0x7f120406df50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4224bf2bec7517d server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:34:32.373 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:34:32.373+0000 7f11edffb700 1 -- 172.21.15.52:0/4169664340 <== mon.0 v2:172.21.15.52:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1204083070 con 0x5634ea8cde40 2024-05-27T07:34:32.373 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:34:32.373+0000 7f11edffb700 1 -- 172.21.15.52:0/4169664340 <== mon.0 v2:172.21.15.52:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f1204113330 con 0x5634ea8cde40 2024-05-27T07:34:32.373 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:34:32.374+0000 7f1218774b80 1 -- 172.21.15.52:0/4169664340 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x5634ea8cde40 msgr2=0x5634eaa4c010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:34:32.373 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:34:32.374+0000 7f1218774b80 1 --2- 172.21.15.52:0/4169664340 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x5634ea8cde40 0x5634eaa4c010 secure :-1 s=READY pgs=716 cs=0 l=1 rev1=1 crypto rx=0x7f1204052100 tx=0x7f120406df50 comp rx=0 tx=0).stop 2024-05-27T07:34:32.373 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:34:32.376+0000 7f1218774b80 1 -- 172.21.15.52:0/4169664340 shutdown_connections 2024-05-27T07:34:32.373 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:34:32.376+0000 7f1218774b80 1 --2- 172.21.15.52:0/4169664340 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5634eaa56de0 0x5634eaa59200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:34:32.374 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:34:32.376+0000 7f1218774b80 1 --2- 172.21.15.52:0/4169664340 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x5634ea8cde40 0x5634eaa4c010 unknown :-1 s=CLOSED pgs=716 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:34:32.374 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:34:32.376+0000 7f1218774b80 1 --2- 172.21.15.52:0/4169664340 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5634eaa4c550 0x5634eaa4c960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:34:32.374 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:34:32.376+0000 7f1218774b80 1 -- 172.21.15.52:0/4169664340 >> 172.21.15.52:0/4169664340 conn(0x5634ea8d0770 msgr2=0x5634eaa48040 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:34:32.374 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:34:32.377+0000 7f1218774b80 1 -- 172.21.15.52:0/4169664340 shutdown_connections 2024-05-27T07:34:32.374 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:34:32.379+0000 7f1218774b80 1 -- 172.21.15.52:0/4169664340 wait complete. 2024-05-27T07:34:32.375 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:34:32.379+0000 7f1218774b80 1 -- start start 2024-05-27T07:34:32.375 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:34:32.380+0000 7f1218774b80 1 Processor -- start 2024-05-27T07:34:32.378 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:34:32.380+0000 7f1218774b80 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5634ea8cde40 0x5634eaa53ce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:34:32.378 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:34:32.380+0000 7f1218774b80 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5634eaa4c550 0x5634eaa54220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:34:32.378 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:34:32.380+0000 7f1218774b80 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x5634eaa56de0 0x5634eaa54760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:34:32.378 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:34:32.380+0000 7f1218774b80 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x5634ea870fa0 con 0x5634eaa56de0 2024-05-27T07:34:32.378 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:34:32.380+0000 7f1218774b80 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x5634ea8d2da0 con 0x5634eaa4c550 2024-05-27T07:34:32.378 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:34:32.380+0000 7f1218774b80 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x5634eaa59d00 con 0x5634ea8cde40 2024-05-27T07:34:32.378 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:34:32.380+0000 7f1218774b80 10 client.0 _ll_register_callbacks cb 0x5634ea9015b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T07:34:32.379 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:34:32.381+0000 7f12037fe700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5634ea8cde40 0x5634eaa53ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:34:32.379 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:34:32.381+0000 7f12037fe700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5634ea8cde40 0x5634eaa53ce0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.173:3300/0 says I am v2:172.21.15.52:50248/0 (socket says 172.21.15.52:50248) 2024-05-27T07:34:32.379 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:34:32.381+0000 7f12037fe700 1 -- 172.21.15.52:0/1057348071 learned_addr learned my addr 172.21.15.52:0/1057348071 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-05-27T07:34:32.379 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:34:32.381+0000 7f1203fff700 1 --2- 172.21.15.52:0/1057348071 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x5634eaa56de0 0x5634eaa54760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:34:32.379 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:34:32.381+0000 7f12037fe700 1 -- 172.21.15.52:0/1057348071 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5634eaa4c550 msgr2=0x5634eaa54220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:34:32.379 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:34:32.381+0000 7f12037fe700 1 --2- 172.21.15.52:0/1057348071 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5634eaa4c550 0x5634eaa54220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:34:32.379 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:34:32.381+0000 7f12037fe700 1 -- 172.21.15.52:0/1057348071 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x5634eaa56de0 msgr2=0x5634eaa54760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:34:32.379 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:34:32.381+0000 7f12037fe700 1 --2- 172.21.15.52:0/1057348071 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x5634eaa56de0 0x5634eaa54760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:34:32.379 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:34:32.381+0000 7f12037fe700 1 -- 172.21.15.52:0/1057348071 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5634ea8ba8d0 con 0x5634ea8cde40 2024-05-27T07:34:32.380 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:34:32.385+0000 7f12037fe700 1 --2- 172.21.15.52:0/1057348071 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5634ea8cde40 0x5634eaa53ce0 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7f1204117d50 tx=0x7f1204051c90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=e3e056c8ab456b56 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:34:32.380 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:34:32.385+0000 7f11e2ffd700 10 client.0 ms_handle_connect on v2:172.21.15.173:3300/0 2024-05-27T07:34:32.385 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:34:32.390+0000 7f11e2ffd700 1 -- 172.21.15.52:0/1057348071 <== mon.1 v2:172.21.15.173:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1204083040 con 0x5634ea8cde40 2024-05-27T07:34:32.385 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:34:32.390+0000 7f1218774b80 1 -- 172.21.15.52:0/1057348071 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f1204113330 con 0x5634ea8cde40 2024-05-27T07:34:32.385 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:34:32.390+0000 7f11e2ffd700 1 -- 172.21.15.52:0/1057348071 <== mon.1 v2:172.21.15.173:3300/0 2 ==== osd_map(120..120 src has 1..120) ==== 9555+0+0 (secure 0 0 0) 0x7f1204147310 con 0x5634ea8cde40 2024-05-27T07:34:32.387 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:34:32.393+0000 7f11e2ffd700 1 -- 172.21.15.52:0/1057348071 <== mon.1 v2:172.21.15.173:3300/0 3 ==== mon_get_version_reply(handle=1 version=107) ==== 24+0+0 (secure 0 0 0) 0x7f120411c070 con 0x5634ea8cde40 2024-05-27T07:34:32.388 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:34:32.393+0000 7f1218774b80 10 client.26509 fetch_fsmap learned FSMap version 107 2024-05-27T07:34:32.388 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:34:32.393+0000 7f1218774b80 1 -- 172.21.15.52:0/1057348071 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({config=0+,fsmap=107,monmap=4+}) -- 0x5634ea924760 con 0x5634ea8cde40 2024-05-27T07:34:32.389 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:34:32.394+0000 7f11e2ffd700 1 -- 172.21.15.52:0/1057348071 <== mon.1 v2:172.21.15.173:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f1204113330 con 0x5634ea8cde40 2024-05-27T07:34:32.389 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:34:32.394+0000 7f11e2ffd700 1 -- 172.21.15.52:0/1057348071 <== mon.1 v2:172.21.15.173:3300/0 5 ==== fsmap(e 107) ==== 4199+0+0 (secure 0 0 0) 0x7f120410aa00 con 0x5634ea8cde40 2024-05-27T07:34:32.389 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:34:32.394+0000 7f1218774b80 10 client.26509 fetch_fsmap finished waiting for FSMap version 107 2024-05-27T07:34:32.389 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:34:32.394+0000 7f1218774b80 10 client.26509 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-05-27T07:34:32.389 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:34:32.394+0000 7f1218774b80 20 client.26509 populate_metadata read hostname 'smithi052' 2024-05-27T07:34:32.389 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:34:32.394+0000 7f1218774b80 1 --2- 172.21.15.52:0/1057348071 >> [v2:172.21.15.105:6838/500348151,v1:172.21.15.105:6839/500348151] conn(0x5634eaa66b10 0x5634eaa68f70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:34:32.389 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:34:32.394+0000 7f1218774b80 4 client.26509 mds_command: new command op to 24527 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-27T07:34:32.389 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:34:32.394+0000 7f1218774b80 1 -- 172.21.15.52:0/1057348071 --> [v2:172.21.15.105:6838/500348151,v1:172.21.15.105:6839/500348151] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5634ea97ccc0 con 0x5634eaa66b10 2024-05-27T07:34:32.389 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:34:32.394+0000 7f1202ffd700 1 --2- 172.21.15.52:0/1057348071 >> [v2:172.21.15.105:6838/500348151,v1:172.21.15.105:6839/500348151] conn(0x5634eaa66b10 0x5634eaa68f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:34:32.394 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:34:32.398+0000 7f1202ffd700 1 --2- 172.21.15.52:0/1057348071 >> [v2:172.21.15.105:6838/500348151,v1:172.21.15.105:6839/500348151] conn(0x5634eaa66b10 0x5634eaa68f70 crc :-1 s=READY pgs=155 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=8a693718d99e11bf server_cookie=f3ea314058ea0a7a in_seq=0 out_seq=0 2024-05-27T07:34:32.394 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:34:32.398+0000 7f11e2ffd700 10 client.26509 ms_handle_connect on v2:172.21.15.105:6838/500348151 2024-05-27T07:34:32.395 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:34:32 smithi052 ceph-mon[31955]: pgmap v289: 129 pgs: 24 active+clean+snaptrim_wait, 10 active+clean+snaptrim, 95 active+clean; 661 MiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 241 KiB/s rd, 3.3 MiB/s wr, 434 op/s 2024-05-27T07:34:32.404 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:34:32.409+0000 7f11e2ffd700 1 -- 172.21.15.52:0/1057348071 <== mds.0 v2:172.21.15.105:6838/500348151 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24041 (crc 0 0 0) 0x5634ea97ccc0 con 0x5634eaa66b10 2024-05-27T07:34:32.404 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:34:32.409+0000 7f11e2ffd700 10 client.26509 handle_command_reply: tid=0 2024-05-27T07:34:32.404 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:34:32.409+0000 7f1218774b80 1 client.26509 shutdown 2024-05-27T07:34:32.408 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:34:32.412+0000 7f1218774b80 1 -- 172.21.15.52:0/1057348071 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5634ea8cde40 msgr2=0x5634eaa53ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:34:32.408 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:34:32.412+0000 7f1218774b80 1 --2- 172.21.15.52:0/1057348071 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5634ea8cde40 0x5634eaa53ce0 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7f1204117d50 tx=0x7f1204051c90 comp rx=0 tx=0).stop 2024-05-27T07:34:32.408 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:34:32.412+0000 7f1218774b80 1 -- 172.21.15.52:0/1057348071 shutdown_connections 2024-05-27T07:34:32.409 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:34:32.412+0000 7f1218774b80 1 --2- 172.21.15.52:0/1057348071 >> [v2:172.21.15.105:6838/500348151,v1:172.21.15.105:6839/500348151] conn(0x5634eaa66b10 0x5634eaa68f70 crc :-1 s=READY pgs=155 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:34:32.409 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:34:32.412+0000 7f1218774b80 1 --2- 172.21.15.52:0/1057348071 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x5634eaa56de0 0x5634eaa54760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:34:32.409 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:34:32.412+0000 7f1218774b80 1 --2- 172.21.15.52:0/1057348071 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5634ea8cde40 0x5634eaa53ce0 secure :-1 s=CLOSED pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7f1204117d50 tx=0x7f1204051c90 comp rx=0 tx=0).stop 2024-05-27T07:34:32.409 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:34:32.412+0000 7f1218774b80 1 --2- 172.21.15.52:0/1057348071 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5634eaa4c550 0x5634eaa54220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:34:32.409 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:34:32.412+0000 7f1218774b80 1 -- 172.21.15.52:0/1057348071 >> 172.21.15.52:0/1057348071 conn(0x5634ea8d0770 msgr2=0x5634eaa465a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:34:32.409 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:34:32.412+0000 7f11e2ffd700 0 client.26509 ms_handle_reset on v2:172.21.15.105:6838/500348151 2024-05-27T07:34:32.410 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:34:32.414+0000 7f1218774b80 1 -- 172.21.15.52:0/1057348071 shutdown_connections 2024-05-27T07:34:32.410 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:34:32.415+0000 7f1218774b80 1 -- 172.21.15.52:0/1057348071 wait complete. 2024-05-27T07:34:32.410 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:34:32.416+0000 7f1218774b80 20 client.26509 trim_cache size 0 max 16384 2024-05-27T07:34:32.453 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:34:32.458+0000 7f1218774b80 1 Processor -- start 2024-05-27T07:34:32.455 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:34:32.458+0000 7f1218774b80 1 -- start start 2024-05-27T07:34:32.455 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:34:32.458+0000 7f1218774b80 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x5634eaa4c550 0x5634eaa48140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:34:32.455 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:34:32.458+0000 7f1218774b80 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5634eaa56de0 0x5634eaa4e630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:34:32.455 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:34:32.458+0000 7f1218774b80 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5634eaa56040 0x5634eaa62e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:34:32.456 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:34:32.458+0000 7f1218774b80 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x7f11e800f4c0 con 0x5634eaa4c550 2024-05-27T07:34:32.456 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:34:32.458+0000 7f1218774b80 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f11e800d920 con 0x5634eaa56040 2024-05-27T07:34:32.456 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:34:32.458+0000 7f1218774b80 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f11e800edc0 con 0x5634eaa56de0 2024-05-27T07:34:32.456 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:34:32.458+0000 7f1202ffd700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5634eaa56de0 0x5634eaa4e630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:34:32.456 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:34:32.458+0000 7f1202ffd700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5634eaa56de0 0x5634eaa4e630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.173:3300/0 says I am v2:172.21.15.52:50256/0 (socket says 172.21.15.52:50256) 2024-05-27T07:34:32.456 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:34:32.458+0000 7f1202ffd700 1 -- 172.21.15.52:0/2324061045 learned_addr learned my addr 172.21.15.52:0/2324061045 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-05-27T07:34:32.456 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:34:32.459+0000 7f12037fe700 1 --2- 172.21.15.52:0/2324061045 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x5634eaa4c550 0x5634eaa48140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:34:32.456 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:34:32.459+0000 7f1203fff700 1 --2- 172.21.15.52:0/2324061045 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5634eaa56040 0x5634eaa62e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:34:32.456 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:34:32.459+0000 7f1202ffd700 1 -- 172.21.15.52:0/2324061045 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5634eaa56040 msgr2=0x5634eaa62e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:34:32.456 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:34:32.459+0000 7f1202ffd700 1 --2- 172.21.15.52:0/2324061045 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5634eaa56040 0x5634eaa62e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:34:32.456 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:34:32.459+0000 7f1202ffd700 1 -- 172.21.15.52:0/2324061045 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x5634eaa4c550 msgr2=0x5634eaa48140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:34:32.456 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:34:32.459+0000 7f1202ffd700 1 --2- 172.21.15.52:0/2324061045 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x5634eaa4c550 0x5634eaa48140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:34:32.456 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:34:32.459+0000 7f1202ffd700 1 -- 172.21.15.52:0/2324061045 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5634eaa52ca0 con 0x5634eaa56de0 2024-05-27T07:34:32.457 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:34:32.459+0000 7f1202ffd700 1 --2- 172.21.15.52:0/2324061045 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5634eaa56de0 0x5634eaa4e630 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7f11f801a9a0 tx=0x7f11f801a250 comp rx=0 tx=0).ready entity=mon.1 client_cookie=c21ac0ce4c687d23 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:34:32.457 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:34:32.460+0000 7f11ed7fa700 1 -- 172.21.15.52:0/2324061045 <== mon.1 v2:172.21.15.173:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f11f8011070 con 0x5634eaa56de0 2024-05-27T07:34:32.457 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:34:32.460+0000 7f11ed7fa700 1 -- 172.21.15.52:0/2324061045 <== mon.1 v2:172.21.15.173:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f11f8003b30 con 0x5634eaa56de0 2024-05-27T07:34:32.457 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:34:32.461+0000 7f11ed7fa700 1 -- 172.21.15.52:0/2324061045 <== mon.1 v2:172.21.15.173:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f11f801ca50 con 0x5634eaa56de0 2024-05-27T07:34:32.457 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:34:32.461+0000 7f1218774b80 1 -- 172.21.15.52:0/2324061045 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5634eaa56de0 msgr2=0x5634eaa4e630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:34:32.457 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:34:32.461+0000 7f1218774b80 1 --2- 172.21.15.52:0/2324061045 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5634eaa56de0 0x5634eaa4e630 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7f11f801a9a0 tx=0x7f11f801a250 comp rx=0 tx=0).stop 2024-05-27T07:34:32.458 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:34:32.462+0000 7f1218774b80 1 -- 172.21.15.52:0/2324061045 shutdown_connections 2024-05-27T07:34:32.458 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:34:32.462+0000 7f1218774b80 1 --2- 172.21.15.52:0/2324061045 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5634eaa56040 0x5634eaa62e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:34:32.458 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:34:32.462+0000 7f1218774b80 1 --2- 172.21.15.52:0/2324061045 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x5634eaa4c550 0x5634eaa48140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:34:32.458 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:34:32.462+0000 7f1218774b80 1 --2- 172.21.15.52:0/2324061045 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5634eaa56de0 0x5634eaa4e630 unknown :-1 s=CLOSED pgs=189 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:34:32.458 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:34:32.462+0000 7f1218774b80 1 -- 172.21.15.52:0/2324061045 >> 172.21.15.52:0/2324061045 conn(0x5634ea8d0770 msgr2=0x5634eaa50d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:34:32.462 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:34:32.467+0000 7f1218774b80 1 -- 172.21.15.52:0/2324061045 shutdown_connections 2024-05-27T07:34:32.462 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:34:32.467+0000 7f1218774b80 1 -- 172.21.15.52:0/2324061045 wait complete. 2024-05-27T07:34:32.466 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:34:32.468+0000 7f1218774b80 1 -- start start 2024-05-27T07:34:32.466 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:34:32.468+0000 7f1218774b80 1 Processor -- start 2024-05-27T07:34:32.466 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:34:32.469+0000 7f1218774b80 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5634eaa56040 0x5634eaa6e3e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:34:32.466 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:34:32.469+0000 7f1218774b80 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5634eaa56de0 0x5634eaa6e920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:34:32.466 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:34:32.469+0000 7f1218774b80 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x5634eaa6ee60 0x5634eaad0600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:34:32.466 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:34:32.469+0000 7f1218774b80 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x7f11e800e6f0 con 0x5634eaa6ee60 2024-05-27T07:34:32.466 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:34:32.469+0000 7f1218774b80 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f11e800dff0 con 0x5634eaa56040 2024-05-27T07:34:32.466 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:34:32.469+0000 7f1218774b80 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x5634eaa63360 con 0x5634eaa56de0 2024-05-27T07:34:32.466 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:34:32.469+0000 7f1218774b80 10 client.0 _ll_register_callbacks cb 0x5634ea9015b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T07:34:32.466 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:34:32.469+0000 7f1202ffd700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5634eaa56de0 0x5634eaa6e920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:34:32.467 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:34:32.469+0000 7f1203fff700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x5634eaa6ee60 0x5634eaad0600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:34:32.467 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:34:32.469+0000 7f1203fff700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x5634eaa6ee60 0x5634eaad0600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.52:46768/0 (socket says 172.21.15.52:46768) 2024-05-27T07:34:32.467 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:34:32.469+0000 7f1203fff700 1 -- 172.21.15.52:0/2570409030 learned_addr learned my addr 172.21.15.52:0/2570409030 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-05-27T07:34:32.467 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:34:32.470+0000 7f1202ffd700 1 -- 172.21.15.52:0/2570409030 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5634eaa56040 msgr2=0x5634eaa6e3e0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-05-27T07:34:32.467 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:34:32.470+0000 7f1202ffd700 1 --2- 172.21.15.52:0/2570409030 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5634eaa56040 0x5634eaa6e3e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:34:32.467 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:34:32.470+0000 7f1202ffd700 1 -- 172.21.15.52:0/2570409030 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x5634eaa6ee60 msgr2=0x5634eaad0600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:34:32.467 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:34:32.470+0000 7f1202ffd700 1 --2- 172.21.15.52:0/2570409030 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x5634eaa6ee60 0x5634eaad0600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:34:32.467 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:34:32.470+0000 7f1202ffd700 1 -- 172.21.15.52:0/2570409030 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5634eaad0b40 con 0x5634eaa56de0 2024-05-27T07:34:32.467 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:34:32.471+0000 7f1202ffd700 1 --2- 172.21.15.52:0/2570409030 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5634eaa56de0 0x5634eaa6e920 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7f11f8000c00 tx=0x7f11f80058e0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=708c9208f75e8d4b server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:34:32.467 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:34:32.471+0000 7f11e2ffd700 10 client.0 ms_handle_connect on v2:172.21.15.173:3300/0 2024-05-27T07:34:32.469 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:34:32.475+0000 7f11e2ffd700 1 -- 172.21.15.52:0/2570409030 <== mon.1 v2:172.21.15.173:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f11f8011070 con 0x5634eaa56de0 2024-05-27T07:34:32.470 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:34:32.475+0000 7f11e2ffd700 1 -- 172.21.15.52:0/2570409030 <== mon.1 v2:172.21.15.173:3300/0 2 ==== osd_map(120..120 src has 1..120) ==== 9555+0+0 (secure 0 0 0) 0x7f11f8005050 con 0x5634eaa56de0 2024-05-27T07:34:32.470 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:34:32.475+0000 7f1218774b80 1 -- 172.21.15.52:0/2570409030 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f11f8003b30 con 0x5634eaa56de0 2024-05-27T07:34:32.474 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:34:32.479+0000 7f11e2ffd700 1 -- 172.21.15.52:0/2570409030 <== mon.1 v2:172.21.15.173:3300/0 3 ==== mon_get_version_reply(handle=1 version=107) ==== 24+0+0 (secure 0 0 0) 0x7f11f8037050 con 0x5634eaa56de0 2024-05-27T07:34:32.474 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:34:32.479+0000 7f1218774b80 10 client.26515 fetch_fsmap learned FSMap version 107 2024-05-27T07:34:32.474 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:34:32.479+0000 7f1218774b80 1 -- 172.21.15.52:0/2570409030 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({config=0+,fsmap=107,monmap=4+}) -- 0x5634eaad1d30 con 0x5634eaa56de0 2024-05-27T07:34:32.483 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:34:32.488+0000 7f11e2ffd700 1 -- 172.21.15.52:0/2570409030 <== mon.1 v2:172.21.15.173:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f11f8003b30 con 0x5634eaa56de0 2024-05-27T07:34:32.483 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:34:32.488+0000 7f11e2ffd700 1 -- 172.21.15.52:0/2570409030 <== mon.1 v2:172.21.15.173:3300/0 5 ==== fsmap(e 107) ==== 4199+0+0 (secure 0 0 0) 0x7f11f8017530 con 0x5634eaa56de0 2024-05-27T07:34:32.483 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:34:32.488+0000 7f1218774b80 10 client.26515 fetch_fsmap finished waiting for FSMap version 107 2024-05-27T07:34:32.483 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:34:32.488+0000 7f1218774b80 10 client.26515 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-05-27T07:34:32.483 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:34:32.488+0000 7f1218774b80 20 client.26515 populate_metadata read hostname 'smithi052' 2024-05-27T07:34:32.483 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:34:32.488+0000 7f1218774b80 1 --2- 172.21.15.52:0/2570409030 >> [v2:172.21.15.105:6838/500348151,v1:172.21.15.105:6839/500348151] conn(0x5634eaad2090 0x5634eaad44b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:34:32.483 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:34:32.488+0000 7f1218774b80 4 client.26515 mds_command: new command op to 24527 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-05-27T07:34:32.483 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:34:32.488+0000 7f1218774b80 1 -- 172.21.15.52:0/2570409030 --> [v2:172.21.15.105:6838/500348151,v1:172.21.15.105:6839/500348151] -- command(tid 0: {"prefix": "scrub status"}) -- 0x5634ea9764a0 con 0x5634eaad2090 2024-05-27T07:34:32.483 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:34:32.488+0000 7f12037fe700 1 --2- 172.21.15.52:0/2570409030 >> [v2:172.21.15.105:6838/500348151,v1:172.21.15.105:6839/500348151] conn(0x5634eaad2090 0x5634eaad44b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:34:32.484 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:34:32.489+0000 7f12037fe700 1 --2- 172.21.15.52:0/2570409030 >> [v2:172.21.15.105:6838/500348151,v1:172.21.15.105:6839/500348151] conn(0x5634eaad2090 0x5634eaad44b0 crc :-1 s=READY pgs=156 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=dced3c26c25ecb61 server_cookie=8bf6737a12572a60 in_seq=0 out_seq=0 2024-05-27T07:34:32.484 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:34:32.489+0000 7f11e2ffd700 10 client.26515 ms_handle_connect on v2:172.21.15.105:6838/500348151 2024-05-27T07:34:32.488 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:34:32.494+0000 7f11e2ffd700 1 -- 172.21.15.52:0/2570409030 <== mds.0 v2:172.21.15.105:6838/500348151 1 ==== command_reply(tid 0: 0 ) ==== 8+0+268 (crc 0 0 0) 0x5634ea9764a0 con 0x5634eaad2090 2024-05-27T07:34:32.489 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:34:32.494+0000 7f11e2ffd700 10 client.26515 handle_command_reply: tid=0 2024-05-27T07:34:32.489 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:34:32.494+0000 7f1218774b80 1 client.26515 shutdown 2024-05-27T07:34:32.490 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:34:32.495+0000 7f1218774b80 1 -- 172.21.15.52:0/2570409030 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5634eaa56de0 msgr2=0x5634eaa6e920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:34:32.490 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:34:32.495+0000 7f1218774b80 1 --2- 172.21.15.52:0/2570409030 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5634eaa56de0 0x5634eaa6e920 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7f11f8000c00 tx=0x7f11f80058e0 comp rx=0 tx=0).stop 2024-05-27T07:34:32.490 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:34:32.496+0000 7f1218774b80 1 -- 172.21.15.52:0/2570409030 shutdown_connections 2024-05-27T07:34:32.491 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:34:32.496+0000 7f1218774b80 1 --2- 172.21.15.52:0/2570409030 >> [v2:172.21.15.105:6838/500348151,v1:172.21.15.105:6839/500348151] conn(0x5634eaad2090 0x5634eaad44b0 crc :-1 s=READY pgs=156 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:34:32.491 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:34:32.496+0000 7f1218774b80 1 --2- 172.21.15.52:0/2570409030 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x5634eaa6ee60 0x5634eaad0600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:34:32.491 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:34:32.496+0000 7f1218774b80 1 --2- 172.21.15.52:0/2570409030 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5634eaa56040 0x5634eaa6e3e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:34:32.491 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:34:32.496+0000 7f1218774b80 1 --2- 172.21.15.52:0/2570409030 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5634eaa56de0 0x5634eaa6e920 secure :-1 s=CLOSED pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7f11f8000c00 tx=0x7f11f80058e0 comp rx=0 tx=0).stop 2024-05-27T07:34:32.491 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:34:32.496+0000 7f1218774b80 1 -- 172.21.15.52:0/2570409030 >> 172.21.15.52:0/2570409030 conn(0x5634ea8d0770 msgr2=0x5634eaa4e370 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:34:32.491 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:34:32.496+0000 7f11e2ffd700 0 client.26515 ms_handle_reset on v2:172.21.15.105:6838/500348151 2024-05-27T07:34:32.491 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:34:32.497+0000 7f1218774b80 1 -- 172.21.15.52:0/2570409030 shutdown_connections 2024-05-27T07:34:32.493 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:34:32.499+0000 7f1218774b80 1 -- 172.21.15.52:0/2570409030 wait complete. 2024-05-27T07:34:32.494 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:34:32.499+0000 7f1218774b80 20 client.26515 trim_cache size 0 max 16384 2024-05-27T07:34:32.494 INFO:teuthology.orchestra.run.smithi052.stdout:{ 2024-05-27T07:34:32.495 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "scrub active (4361 inodes in the stack)", 2024-05-27T07:34:32.495 INFO:teuthology.orchestra.run.smithi052.stdout: "scrubs": { 2024-05-27T07:34:32.495 INFO:teuthology.orchestra.run.smithi052.stdout: "b39c870d-d57b-4c7d-baaf-c256689c7e81": { 2024-05-27T07:34:32.495 INFO:teuthology.orchestra.run.smithi052.stdout: "path": "/", 2024-05-27T07:34:32.495 INFO:teuthology.orchestra.run.smithi052.stdout: "tag": "b39c870d-d57b-4c7d-baaf-c256689c7e81", 2024-05-27T07:34:32.495 INFO:teuthology.orchestra.run.smithi052.stdout: "options": "recursive,force" 2024-05-27T07:34:32.495 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-05-27T07:34:32.495 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-05-27T07:34:32.495 INFO:teuthology.orchestra.run.smithi052.stdout:} 2024-05-27T07:34:32.498 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:34:32.502+0000 7f1209d5f700 1 -- 172.21.15.52:0/3379767108 >> [v2:172.21.15.52:6800/1822318384,v1:172.21.15.52:6801/1822318384] conn(0x7f11f40b2bf0 msgr2=0x7f11f40b5100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:34:32.498 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:34:32.502+0000 7f1209d5f700 1 --2- 172.21.15.52:0/3379767108 >> [v2:172.21.15.52:6800/1822318384,v1:172.21.15.52:6801/1822318384] conn(0x7f11f40b2bf0 0x7f11f40b5100 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7f11f0005950 tx=0x7f11f00058e0 comp rx=0 tx=0).stop 2024-05-27T07:34:32.498 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:34:32.503+0000 7f1209d5f700 1 -- 172.21.15.52:0/3379767108 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f11fc09d560 msgr2=0x7f11fc00f1d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:34:32.498 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:34:32.503+0000 7f1209d5f700 1 --2- 172.21.15.52:0/3379767108 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f11fc09d560 0x7f11fc00f1d0 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7f1204052c50 tx=0x7f12040756c0 comp rx=0 tx=0).stop 2024-05-27T07:34:32.498 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:34:32.503+0000 7f1209d5f700 1 -- 172.21.15.52:0/3379767108 shutdown_connections 2024-05-27T07:34:32.498 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:34:32.503+0000 7f1209d5f700 1 --2- 172.21.15.52:0/3379767108 >> [v2:172.21.15.52:6800/1822318384,v1:172.21.15.52:6801/1822318384] conn(0x7f11f40b2bf0 0x7f11f40b5100 unknown :-1 s=CLOSED pgs=175 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:34:32.498 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:34:32.503+0000 7f1209d5f700 1 --2- 172.21.15.52:0/3379767108 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f11fc013a00 0x7f11fc013e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:34:32.498 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:34:32.503+0000 7f1209d5f700 1 --2- 172.21.15.52:0/3379767108 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f11fc09d560 0x7f11fc00f1d0 unknown :-1 s=CLOSED pgs=187 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:34:32.499 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:34:32.503+0000 7f1209d5f700 1 --2- 172.21.15.52:0/3379767108 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f11fc0a2c50 0x7f11fc00f770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:34:32.499 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:34:32.503+0000 7f1209d5f700 1 -- 172.21.15.52:0/3379767108 >> 172.21.15.52:0/3379767108 conn(0x7f11fc009e40 msgr2=0x7f11fc0a3790 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:34:32.499 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:34:32.505+0000 7f1209d5f700 1 -- 172.21.15.52:0/3379767108 shutdown_connections 2024-05-27T07:34:32.499 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:34:32.505+0000 7f1209d5f700 1 -- 172.21.15.52:0/3379767108 wait complete. 2024-05-27T07:34:32.694 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:34:32 smithi105 ceph-mon[35977]: pgmap v289: 129 pgs: 24 active+clean+snaptrim_wait, 10 active+clean+snaptrim, 95 active+clean; 661 MiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 241 KiB/s rd, 3.3 MiB/s wr, 434 op/s 2024-05-27T07:34:32.856 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:34:32 smithi173 ceph-mon[37000]: pgmap v289: 129 pgs: 24 active+clean+snaptrim_wait, 10 active+clean+snaptrim, 95 active+clean; 661 MiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 241 KiB/s rd, 3.3 MiB/s wr, 434 op/s 2024-05-27T07:34:33.197 INFO:tasks.cephfs.filesystem:scrub status for tag:b39c870d-d57b-4c7d-baaf-c256689c7e81 - {'path': '/', 'tag': 'b39c870d-d57b-4c7d-baaf-c256689c7e81', 'options': 'recursive,force'} 2024-05-27T07:34:33.484 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:34:33 smithi052 ceph-mon[31955]: fsmap cephfs:5 5 up:active 2024-05-27T07:34:33.695 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:34:33 smithi105 ceph-mon[35977]: fsmap cephfs:5 5 up:active 2024-05-27T07:34:33.859 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:34:33 smithi173 ceph-mon[37000]: fsmap cephfs:5 5 up:active 2024-05-27T07:34:34.695 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:34:34 smithi105 ceph-mon[35977]: pgmap v290: 129 pgs: 24 active+clean+snaptrim_wait, 10 active+clean+snaptrim, 95 active+clean; 657 MiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 287 KiB/s rd, 3.7 MiB/s wr, 587 op/s 2024-05-27T07:34:34.734 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:34:34 smithi052 ceph-mon[31955]: pgmap v290: 129 pgs: 24 active+clean+snaptrim_wait, 10 active+clean+snaptrim, 95 active+clean; 657 MiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 287 KiB/s rd, 3.7 MiB/s wr, 587 op/s 2024-05-27T07:34:34.856 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:34:34 smithi173 ceph-mon[37000]: pgmap v290: 129 pgs: 24 active+clean+snaptrim_wait, 10 active+clean+snaptrim, 95 active+clean; 657 MiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 287 KiB/s rd, 3.7 MiB/s wr, 587 op/s 2024-05-27T07:34:36.694 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:34:36 smithi105 ceph-mon[35977]: pgmap v291: 129 pgs: 18 active+clean+snaptrim_wait, 7 active+clean+snaptrim, 104 active+clean; 657 MiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 348 KiB/s rd, 2.7 MiB/s wr, 683 op/s 2024-05-27T07:34:36.734 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:34:36 smithi052 ceph-mon[31955]: pgmap v291: 129 pgs: 18 active+clean+snaptrim_wait, 7 active+clean+snaptrim, 104 active+clean; 657 MiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 348 KiB/s rd, 2.7 MiB/s wr, 683 op/s 2024-05-27T07:34:36.856 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:34:36 smithi173 ceph-mon[37000]: pgmap v291: 129 pgs: 18 active+clean+snaptrim_wait, 7 active+clean+snaptrim, 104 active+clean; 657 MiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 348 KiB/s rd, 2.7 MiB/s wr, 683 op/s 2024-05-27T07:34:38.694 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:34:38 smithi105 ceph-mon[35977]: pgmap v292: 129 pgs: 6 active+clean+snaptrim_wait, 3 active+clean+snaptrim, 120 active+clean; 660 MiB data, 5.3 GiB used, 1.0 TiB / 1.0 TiB avail; 471 KiB/s rd, 1.9 MiB/s wr, 876 op/s 2024-05-27T07:34:38.734 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:34:38 smithi052 ceph-mon[31955]: pgmap v292: 129 pgs: 6 active+clean+snaptrim_wait, 3 active+clean+snaptrim, 120 active+clean; 660 MiB data, 5.3 GiB used, 1.0 TiB / 1.0 TiB avail; 471 KiB/s rd, 1.9 MiB/s wr, 876 op/s 2024-05-27T07:34:38.856 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:34:38 smithi173 ceph-mon[37000]: pgmap v292: 129 pgs: 6 active+clean+snaptrim_wait, 3 active+clean+snaptrim, 120 active+clean; 660 MiB data, 5.3 GiB used, 1.0 TiB / 1.0 TiB avail; 471 KiB/s rd, 1.9 MiB/s wr, 876 op/s 2024-05-27T07:34:39.699 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:34:39 smithi105 ceph-mon[35977]: from='mgr.15333 172.21.15.52:0/416389082' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-27T07:34:39.699 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:34:39 smithi105 ceph-mon[35977]: pgmap v293: 129 pgs: 6 active+clean+snaptrim_wait, 2 active+clean+snaptrim, 121 active+clean; 662 MiB data, 5.3 GiB used, 1.0 TiB / 1.0 TiB avail; 504 KiB/s rd, 2.1 MiB/s wr, 974 op/s 2024-05-27T07:34:39.734 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:34:39 smithi052 ceph-mon[31955]: from='mgr.15333 172.21.15.52:0/416389082' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-27T07:34:39.734 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:34:39 smithi052 ceph-mon[31955]: pgmap v293: 129 pgs: 6 active+clean+snaptrim_wait, 2 active+clean+snaptrim, 121 active+clean; 662 MiB data, 5.3 GiB used, 1.0 TiB / 1.0 TiB avail; 504 KiB/s rd, 2.1 MiB/s wr, 974 op/s 2024-05-27T07:34:39.856 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:34:39 smithi173 ceph-mon[37000]: from='mgr.15333 172.21.15.52:0/416389082' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-27T07:34:39.857 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:34:39 smithi173 ceph-mon[37000]: pgmap v293: 129 pgs: 6 active+clean+snaptrim_wait, 2 active+clean+snaptrim, 121 active+clean; 662 MiB data, 5.3 GiB used, 1.0 TiB / 1.0 TiB avail; 504 KiB/s rd, 2.1 MiB/s wr, 974 op/s 2024-05-27T07:34:40.694 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:34:40 smithi105 ceph-mon[35977]: scrub summary: idle+waiting paths [/] 2024-05-27T07:34:40.695 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:34:40 smithi105 ceph-mon[35977]: from='mgr.15333 172.21.15.52:0/416389082' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T07:34:40.695 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:34:40 smithi105 ceph-mon[35977]: from='mgr.15333 172.21.15.52:0/416389082' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-27T07:34:40.695 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:34:40 smithi105 ceph-mon[35977]: from='mgr.15333 172.21.15.52:0/416389082' entity='mgr.x' 2024-05-27T07:34:40.734 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:34:40 smithi052 ceph-mon[31955]: scrub summary: idle+waiting paths [/] 2024-05-27T07:34:40.734 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:34:40 smithi052 ceph-mon[31955]: from='mgr.15333 172.21.15.52:0/416389082' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T07:34:40.735 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:34:40 smithi052 ceph-mon[31955]: from='mgr.15333 172.21.15.52:0/416389082' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-27T07:34:40.735 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:34:40 smithi052 ceph-mon[31955]: from='mgr.15333 172.21.15.52:0/416389082' entity='mgr.x' 2024-05-27T07:34:40.856 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:34:40 smithi173 ceph-mon[37000]: scrub summary: idle+waiting paths [/] 2024-05-27T07:34:40.857 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:34:40 smithi173 ceph-mon[37000]: from='mgr.15333 172.21.15.52:0/416389082' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T07:34:40.857 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:34:40 smithi173 ceph-mon[37000]: from='mgr.15333 172.21.15.52:0/416389082' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-27T07:34:40.857 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:34:40 smithi173 ceph-mon[37000]: from='mgr.15333 172.21.15.52:0/416389082' entity='mgr.x' 2024-05-27T07:34:41.392 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:34:41.397+0000 7f3943b72640 1 -- 172.21.15.52:0/164867001 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f393c10a1f0 msgr2=0x7f393c10a600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:34:41.392 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:34:41.397+0000 7f3943b72640 1 --2- 172.21.15.52:0/164867001 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f393c10a1f0 0x7f393c10a600 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7f3934009be0 tx=0x7f393402a750 comp rx=0 tx=0).stop 2024-05-27T07:34:41.397 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:34:41.401+0000 7f3943b72640 1 -- 172.21.15.52:0/164867001 shutdown_connections 2024-05-27T07:34:41.397 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:34:41.401+0000 7f3943b72640 1 --2- 172.21.15.52:0/164867001 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f393c11d240 0x7f393c11f660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:34:41.398 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:34:41.401+0000 7f3943b72640 1 --2- 172.21.15.52:0/164867001 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f393c11a8e0 0x7f393c11cd00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:34:41.398 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:34:41.401+0000 7f3943b72640 1 --2- 172.21.15.52:0/164867001 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f393c10a1f0 0x7f393c10a600 unknown :-1 s=CLOSED pgs=239 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:34:41.398 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:34:41.401+0000 7f3943b72640 1 -- 172.21.15.52:0/164867001 >> 172.21.15.52:0/164867001 conn(0x7f393c06e8f0 msgr2=0x7f393c06ed30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:34:41.401 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:34:41.406+0000 7f3943b72640 1 -- 172.21.15.52:0/164867001 shutdown_connections 2024-05-27T07:34:41.401 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:34:41.407+0000 7f3943b72640 1 -- 172.21.15.52:0/164867001 wait complete. 2024-05-27T07:34:41.403 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:34:41.408+0000 7f3943b72640 1 Processor -- start 2024-05-27T07:34:41.403 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:34:41.408+0000 7f3943b72640 1 -- start start 2024-05-27T07:34:41.403 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:34:41.409+0000 7f3943b72640 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f393c10a1f0 0x7f393c1a9630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:34:41.404 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:34:41.409+0000 7f3943b72640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f393c11a8e0 0x7f393c1a9b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:34:41.404 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:34:41.409+0000 7f3943b72640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f393c11d240 0x7f393c1ade00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:34:41.404 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:34:41.409+0000 7f394110d640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f393c11a8e0 0x7f393c1a9b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:34:41.404 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:34:41.409+0000 7f394110d640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f393c11a8e0 0x7f393c1a9b90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.173:3300/0 says I am v2:172.21.15.52:33186/0 (socket says 172.21.15.52:33186) 2024-05-27T07:34:41.405 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:34:41.409+0000 7f394110d640 1 -- 172.21.15.52:0/1881844083 learned_addr learned my addr 172.21.15.52:0/1881844083 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-05-27T07:34:41.405 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:34:41.410+0000 7f3943b72640 1 -- 172.21.15.52:0/1881844083 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x7f393c122c70 con 0x7f393c10a1f0 2024-05-27T07:34:41.405 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:34:41.410+0000 7f3943b72640 1 -- 172.21.15.52:0/1881844083 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f393c122af0 con 0x7f393c11d240 2024-05-27T07:34:41.405 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:34:41.410+0000 7f3943b72640 1 -- 172.21.15.52:0/1881844083 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f393c122df0 con 0x7f393c11a8e0 2024-05-27T07:34:41.406 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:34:41.411+0000 7f394110d640 1 -- 172.21.15.52:0/1881844083 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f393c11d240 msgr2=0x7f393c1ade00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:34:41.406 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:34:41.411+0000 7f394110d640 1 --2- 172.21.15.52:0/1881844083 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f393c11d240 0x7f393c1ade00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:34:41.406 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:34:41.411+0000 7f394110d640 1 -- 172.21.15.52:0/1881844083 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f393c10a1f0 msgr2=0x7f393c1a9630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:34:41.406 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:34:41.411+0000 7f394110d640 1 --2- 172.21.15.52:0/1881844083 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f393c10a1f0 0x7f393c1a9630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:34:41.407 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:34:41.411+0000 7f394110d640 1 -- 172.21.15.52:0/1881844083 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f393c1ae440 con 0x7f393c11a8e0 2024-05-27T07:34:41.410 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:34:41.416+0000 7f394110d640 1 --2- 172.21.15.52:0/1881844083 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f393c11a8e0 0x7f393c1a9b90 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7f392c00e9f0 tx=0x7f392c00edf0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:34:41.414 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:34:41.419+0000 7f3932ffd640 1 -- 172.21.15.52:0/1881844083 <== mon.1 v2:172.21.15.173:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f392c00cd80 con 0x7f393c11a8e0 2024-05-27T07:34:41.414 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:34:41.419+0000 7f3932ffd640 1 -- 172.21.15.52:0/1881844083 <== mon.1 v2:172.21.15.173:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f392c0054d0 con 0x7f393c11a8e0 2024-05-27T07:34:41.414 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:34:41.420+0000 7f3932ffd640 1 -- 172.21.15.52:0/1881844083 <== mon.1 v2:172.21.15.173:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f392c010690 con 0x7f393c11a8e0 2024-05-27T07:34:41.415 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:34:41.421+0000 7f3943b72640 1 -- 172.21.15.52:0/1881844083 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f393c1ae730 con 0x7f393c11a8e0 2024-05-27T07:34:41.415 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:34:41.421+0000 7f3943b72640 1 -- 172.21.15.52:0/1881844083 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f393c1aec00 con 0x7f393c11a8e0 2024-05-27T07:34:41.420 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:34:41.425+0000 7f394aa27740 1 Processor -- start 2024-05-27T07:34:41.423 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:34:41.428+0000 7f3932ffd640 1 -- 172.21.15.52:0/1881844083 <== mon.1 v2:172.21.15.173:3300/0 4 ==== mgrmap(e 26) ==== 149783+0+0 (secure 0 0 0) 0x7f392c004e10 con 0x7f393c11a8e0 2024-05-27T07:34:41.423 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:34:41.429+0000 7f3932ffd640 1 --2- 172.21.15.52:0/1881844083 >> [v2:172.21.15.52:6800/1822318384,v1:172.21.15.52:6801/1822318384] conn(0x7f39280b2af0 0x7f39280b5000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:34:41.424 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:34:41.429+0000 7f3932ffd640 1 -- 172.21.15.52:0/1881844083 <== mon.1 v2:172.21.15.173:3300/0 5 ==== osd_map(120..120 src has 1..120) ==== 9555+0+0 (secure 0 0 0) 0x7f392c0a63e0 con 0x7f393c11a8e0 2024-05-27T07:34:41.424 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:34:41.430+0000 7f394aa27740 1 -- start start 2024-05-27T07:34:41.426 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:34:41.431+0000 7f394aa27740 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55b0f82e11b0 0x55b0f86823d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:34:41.426 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:34:41.431+0000 7f394aa27740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55b0f82b82f0 0x55b0f8682910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:34:41.426 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:34:41.431+0000 7f394aa27740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55b0f82c04d0 0x55b0f8682e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:34:41.426 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:34:41.431+0000 7f394110d640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55b0f82b82f0 0x55b0f8682910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:34:41.426 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:34:41.431+0000 7f394190e640 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55b0f82e11b0 0x55b0f86823d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:34:41.426 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:34:41.431+0000 7f394190e640 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55b0f82e11b0 0x55b0f86823d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.52:38026/0 (socket says 172.21.15.52:38026) 2024-05-27T07:34:41.426 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:34:41.431+0000 7f394190e640 1 -- 172.21.15.52:0/4283248626 learned_addr learned my addr 172.21.15.52:0/4283248626 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-05-27T07:34:41.427 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:34:41.431+0000 7f394aa27740 1 -- 172.21.15.52:0/4283248626 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x55b0f85f4c30 con 0x55b0f82e11b0 2024-05-27T07:34:41.427 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:34:41.431+0000 7f394aa27740 1 -- 172.21.15.52:0/4283248626 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x55b0f85e47d0 con 0x55b0f82c04d0 2024-05-27T07:34:41.427 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:34:41.431+0000 7f394aa27740 1 -- 172.21.15.52:0/4283248626 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x55b0f85e8810 con 0x55b0f82b82f0 2024-05-27T07:34:41.428 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:34:41.432+0000 7f394110d640 1 -- 172.21.15.52:0/4283248626 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55b0f82c04d0 msgr2=0x55b0f8682e50 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-05-27T07:34:41.428 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:34:41.432+0000 7f394110d640 1 --2- 172.21.15.52:0/4283248626 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55b0f82c04d0 0x55b0f8682e50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:34:41.428 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:34:41.432+0000 7f394110d640 1 -- 172.21.15.52:0/4283248626 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55b0f82e11b0 msgr2=0x55b0f86823d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:34:41.428 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:34:41.432+0000 7f394110d640 1 --2- 172.21.15.52:0/4283248626 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55b0f82e11b0 0x55b0f86823d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:34:41.428 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:34:41.432+0000 7f394110d640 1 -- 172.21.15.52:0/4283248626 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55b0f82f9210 con 0x55b0f82b82f0 2024-05-27T07:34:41.428 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:34:41.432+0000 7f394110d640 1 --2- 172.21.15.52:0/4283248626 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55b0f82b82f0 0x55b0f8682910 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7f392c0e0260 tx=0x7f392c0af2c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=b6b8ab38d1027218 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:34:41.428 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:34:41.432+0000 7f394210f640 1 --2- 172.21.15.52:0/1881844083 >> [v2:172.21.15.52:6800/1822318384,v1:172.21.15.52:6801/1822318384] conn(0x7f39280b2af0 0x7f39280b5000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:34:41.428 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:34:41.433+0000 7f3927fff640 1 -- 172.21.15.52:0/4283248626 <== mon.1 v2:172.21.15.173:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f392c0a65e0 con 0x55b0f82b82f0 2024-05-27T07:34:41.429 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:34:41.433+0000 7f3927fff640 1 -- 172.21.15.52:0/4283248626 <== mon.1 v2:172.21.15.173:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f392c005680 con 0x55b0f82b82f0 2024-05-27T07:34:41.429 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:34:41.433+0000 7f3927fff640 1 -- 172.21.15.52:0/4283248626 <== mon.1 v2:172.21.15.173:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f392c0a6780 con 0x55b0f82b82f0 2024-05-27T07:34:41.430 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:34:41.434+0000 7f394aa27740 1 -- 172.21.15.52:0/4283248626 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55b0f82b82f0 msgr2=0x55b0f8682910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:34:41.430 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:34:41.434+0000 7f394aa27740 1 --2- 172.21.15.52:0/4283248626 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55b0f82b82f0 0x55b0f8682910 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7f392c0e0260 tx=0x7f392c0af2c0 comp rx=0 tx=0).stop 2024-05-27T07:34:41.430 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:34:41.434+0000 7f394210f640 1 --2- 172.21.15.52:0/1881844083 >> [v2:172.21.15.52:6800/1822318384,v1:172.21.15.52:6801/1822318384] conn(0x7f39280b2af0 0x7f39280b5000 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7f3938005fd0 tx=0x7f39380095a0 comp rx=0 tx=0).ready entity=mgr.15333 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:34:41.430 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:34:41.434+0000 7f394aa27740 1 -- 172.21.15.52:0/4283248626 shutdown_connections 2024-05-27T07:34:41.430 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:34:41.434+0000 7f394aa27740 1 --2- 172.21.15.52:0/4283248626 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55b0f82c04d0 0x55b0f8682e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:34:41.430 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:34:41.434+0000 7f394aa27740 1 --2- 172.21.15.52:0/4283248626 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55b0f82b82f0 0x55b0f8682910 unknown :-1 s=CLOSED pgs=192 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:34:41.430 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:34:41.434+0000 7f394aa27740 1 --2- 172.21.15.52:0/4283248626 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55b0f82e11b0 0x55b0f86823d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:34:41.430 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:34:41.434+0000 7f394aa27740 1 -- 172.21.15.52:0/4283248626 >> 172.21.15.52:0/4283248626 conn(0x55b0f82d5300 msgr2=0x55b0f867eb80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:34:41.431 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:34:41.434+0000 7f394aa27740 1 -- 172.21.15.52:0/4283248626 shutdown_connections 2024-05-27T07:34:41.431 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:34:41.434+0000 7f394aa27740 1 -- 172.21.15.52:0/4283248626 wait complete. 2024-05-27T07:34:41.431 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:34:41.435+0000 7f394aa27740 1 -- start start 2024-05-27T07:34:41.431 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:34:41.435+0000 7f394aa27740 1 Processor -- start 2024-05-27T07:34:41.432 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:34:41.436+0000 7f394aa27740 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55b0f82b82f0 0x55b0f868e4c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:34:41.432 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:34:41.436+0000 7f394aa27740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55b0f82c04d0 0x55b0f868ea00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:34:41.432 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:34:41.436+0000 7f394aa27740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55b0f82e11b0 0x55b0f868c960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:34:41.432 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:34:41.436+0000 7f394aa27740 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x55b0f82ca790 con 0x55b0f82b82f0 2024-05-27T07:34:41.432 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:34:41.436+0000 7f394aa27740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x55b0f85a0890 con 0x55b0f82e11b0 2024-05-27T07:34:41.433 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:34:41.436+0000 7f394aa27740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x55b0f868f410 con 0x55b0f82c04d0 2024-05-27T07:34:41.433 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:34:41.436+0000 7f394aa27740 10 client.0 _ll_register_callbacks cb 0x55b0f82f8a00 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T07:34:41.433 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:34:41.436+0000 7f394190e640 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55b0f82b82f0 0x55b0f868e4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:34:41.433 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:34:41.436+0000 7f394190e640 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55b0f82b82f0 0x55b0f868e4c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.52:38028/0 (socket says 172.21.15.52:38028) 2024-05-27T07:34:41.433 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:34:41.436+0000 7f394190e640 1 -- 172.21.15.52:0/3867459394 learned_addr learned my addr 172.21.15.52:0/3867459394 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-05-27T07:34:41.433 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:34:41.436+0000 7f394210f640 1 --2- 172.21.15.52:0/3867459394 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55b0f82e11b0 0x55b0f868c960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:34:41.433 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:34:41.436+0000 7f394190e640 1 -- 172.21.15.52:0/3867459394 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55b0f82c04d0 msgr2=0x55b0f868ea00 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-05-27T07:34:41.433 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:34:41.436+0000 7f394190e640 1 --2- 172.21.15.52:0/3867459394 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55b0f82c04d0 0x55b0f868ea00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:34:41.433 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:34:41.436+0000 7f394190e640 1 -- 172.21.15.52:0/3867459394 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55b0f82e11b0 msgr2=0x55b0f868c960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:34:41.433 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:34:41.436+0000 7f394190e640 1 --2- 172.21.15.52:0/3867459394 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55b0f82e11b0 0x55b0f868c960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:34:41.433 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:34:41.436+0000 7f394190e640 1 -- 172.21.15.52:0/3867459394 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55b0f83c5620 con 0x55b0f82b82f0 2024-05-27T07:34:41.433 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:34:41.437+0000 7f394190e640 1 --2- 172.21.15.52:0/3867459394 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55b0f82b82f0 0x55b0f868e4c0 secure :-1 s=READY pgs=717 cs=0 l=1 rev1=1 crypto rx=0x7f3934009920 tx=0x7f3934009950 comp rx=0 tx=0).ready entity=mon.0 client_cookie=71aa04a2db867cf4 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:34:41.433 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:34:41.437+0000 7f39257fa640 10 client.0 ms_handle_connect on v2:172.21.15.52:3300/0 2024-05-27T07:34:41.434 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:34:41.438+0000 7f39257fa640 1 -- 172.21.15.52:0/3867459394 <== mon.0 v2:172.21.15.52:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3934036040 con 0x55b0f82b82f0 2024-05-27T07:34:41.434 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:34:41.438+0000 7f39257fa640 1 -- 172.21.15.52:0/3867459394 <== mon.0 v2:172.21.15.52:3300/0 2 ==== osd_map(120..120 src has 1..120) ==== 9555+0+0 (secure 0 0 0) 0x7f39340303a0 con 0x55b0f82b82f0 2024-05-27T07:34:41.434 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:34:41.438+0000 7f394aa27740 1 -- 172.21.15.52:0/3867459394 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f392c005680 con 0x55b0f82b82f0 2024-05-27T07:34:41.435 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:34:41.438+0000 7f39257fa640 1 -- 172.21.15.52:0/3867459394 <== mon.0 v2:172.21.15.52:3300/0 3 ==== mon_get_version_reply(handle=1 version=107) ==== 24+0+0 (secure 0 0 0) 0x7f393404c050 con 0x55b0f82b82f0 2024-05-27T07:34:41.438 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:34:41.438+0000 7f394aa27740 10 client.17019 fetch_fsmap learned FSMap version 107 2024-05-27T07:34:41.438 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:34:41.438+0000 7f394aa27740 1 -- 172.21.15.52:0/3867459394 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({config=0+,fsmap=107,monmap=4+}) -- 0x55b0f869b770 con 0x55b0f82b82f0 2024-05-27T07:34:41.438 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:34:41.438+0000 7f39257fa640 1 -- 172.21.15.52:0/3867459394 <== mon.0 v2:172.21.15.52:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f392c005680 con 0x55b0f82b82f0 2024-05-27T07:34:41.438 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:34:41.438+0000 7f39257fa640 1 -- 172.21.15.52:0/3867459394 <== mon.0 v2:172.21.15.52:3300/0 5 ==== fsmap(e 107) ==== 4199+0+0 (secure 0 0 0) 0x7f39340043a0 con 0x55b0f82b82f0 2024-05-27T07:34:41.439 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:34:41.439+0000 7f394aa27740 10 client.17019 fetch_fsmap finished waiting for FSMap version 107 2024-05-27T07:34:41.439 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:34:41.439+0000 7f394aa27740 10 client.17019 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-05-27T07:34:41.439 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:34:41.439+0000 7f394aa27740 20 client.17019 populate_metadata read hostname 'smithi052' 2024-05-27T07:34:41.439 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:34:41.439+0000 7f394aa27740 1 --2- 172.21.15.52:0/3867459394 >> [v2:172.21.15.105:6838/500348151,v1:172.21.15.105:6839/500348151] conn(0x55b0f829fd80 0x55b0f869f150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:34:41.439 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:34:41.439+0000 7f394aa27740 4 client.17019 mds_command: new command op to 24527 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-27T07:34:41.439 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:34:41.439+0000 7f394aa27740 1 -- 172.21.15.52:0/3867459394 --> [v2:172.21.15.105:6838/500348151,v1:172.21.15.105:6839/500348151] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55b0f85f6b80 con 0x55b0f829fd80 2024-05-27T07:34:41.439 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:34:41.439+0000 7f394110d640 1 --2- 172.21.15.52:0/3867459394 >> [v2:172.21.15.105:6838/500348151,v1:172.21.15.105:6839/500348151] conn(0x55b0f829fd80 0x55b0f869f150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:34:41.441 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:34:41.441+0000 7f394110d640 1 --2- 172.21.15.52:0/3867459394 >> [v2:172.21.15.105:6838/500348151,v1:172.21.15.105:6839/500348151] conn(0x55b0f829fd80 0x55b0f869f150 crc :-1 s=READY pgs=157 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=c841d5e0137cd0a7 server_cookie=41f86557adaa5c0f in_seq=0 out_seq=0 2024-05-27T07:34:41.441 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:34:41.441+0000 7f39257fa640 10 client.17019 ms_handle_connect on v2:172.21.15.105:6838/500348151 2024-05-27T07:34:41.441 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:34:41.445+0000 7f39257fa640 1 -- 172.21.15.52:0/3867459394 <== mds.0 v2:172.21.15.105:6838/500348151 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24041 (crc 0 0 0) 0x55b0f85f6b80 con 0x55b0f829fd80 2024-05-27T07:34:41.441 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:34:41.445+0000 7f39257fa640 10 client.17019 handle_command_reply: tid=0 2024-05-27T07:34:41.441 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:34:41.445+0000 7f394aa27740 1 client.17019 shutdown 2024-05-27T07:34:41.441 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:34:41.445+0000 7f394aa27740 1 -- 172.21.15.52:0/3867459394 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55b0f82b82f0 msgr2=0x55b0f868e4c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:34:41.441 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:34:41.445+0000 7f394aa27740 1 --2- 172.21.15.52:0/3867459394 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55b0f82b82f0 0x55b0f868e4c0 secure :-1 s=READY pgs=717 cs=0 l=1 rev1=1 crypto rx=0x7f3934009920 tx=0x7f3934009950 comp rx=0 tx=0).stop 2024-05-27T07:34:41.441 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:34:41.446+0000 7f394aa27740 1 -- 172.21.15.52:0/3867459394 shutdown_connections 2024-05-27T07:34:41.441 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:34:41.446+0000 7f394aa27740 1 --2- 172.21.15.52:0/3867459394 >> [v2:172.21.15.105:6838/500348151,v1:172.21.15.105:6839/500348151] conn(0x55b0f829fd80 0x55b0f869f150 crc :-1 s=READY pgs=157 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:34:41.441 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:34:41.446+0000 7f394aa27740 1 --2- 172.21.15.52:0/3867459394 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55b0f82e11b0 0x55b0f868c960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:34:41.441 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:34:41.446+0000 7f394aa27740 1 --2- 172.21.15.52:0/3867459394 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55b0f82c04d0 0x55b0f868ea00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:34:41.442 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:34:41.446+0000 7f394aa27740 1 --2- 172.21.15.52:0/3867459394 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55b0f82b82f0 0x55b0f868e4c0 unknown :-1 s=CLOSED pgs=717 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:34:41.442 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:34:41.446+0000 7f394aa27740 1 -- 172.21.15.52:0/3867459394 >> 172.21.15.52:0/3867459394 conn(0x55b0f82d5300 msgr2=0x55b0f86840f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:34:41.442 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:34:41.446+0000 7f39257fa640 0 client.17019 ms_handle_reset on v2:172.21.15.105:6838/500348151 2024-05-27T07:34:41.442 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:34:41.446+0000 7f394aa27740 1 -- 172.21.15.52:0/3867459394 shutdown_connections 2024-05-27T07:34:41.442 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:34:41.446+0000 7f394aa27740 1 -- 172.21.15.52:0/3867459394 wait complete. 2024-05-27T07:34:41.442 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:34:41.446+0000 7f394aa27740 20 client.17019 trim_cache size 0 max 16384 2024-05-27T07:34:41.459 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:34:41.464+0000 7f394aa27740 1 Processor -- start 2024-05-27T07:34:41.459 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:34:41.464+0000 7f394aa27740 1 -- start start 2024-05-27T07:34:41.460 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:34:41.464+0000 7f394aa27740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55b0f82b82f0 0x55b0f8684720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:34:41.460 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:34:41.465+0000 7f394aa27740 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55b0f82c04d0 0x55b0f8686150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:34:41.460 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:34:41.465+0000 7f394aa27740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55b0f82e11b0 0x55b0f8686690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:34:41.460 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:34:41.465+0000 7f394aa27740 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x7f393c1c0710 con 0x55b0f82c04d0 2024-05-27T07:34:41.460 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:34:41.465+0000 7f394aa27740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f393c1beb70 con 0x55b0f82e11b0 2024-05-27T07:34:41.460 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:34:41.465+0000 7f394aa27740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f393c1c0010 con 0x55b0f82b82f0 2024-05-27T07:34:41.460 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:34:41.465+0000 7f394110d640 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55b0f82c04d0 0x55b0f8686150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:34:41.460 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:34:41.465+0000 7f394190e640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55b0f82b82f0 0x55b0f8684720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:34:41.460 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:34:41.465+0000 7f394110d640 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55b0f82c04d0 0x55b0f8686150 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.52:38030/0 (socket says 172.21.15.52:38030) 2024-05-27T07:34:41.460 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:34:41.465+0000 7f394110d640 1 -- 172.21.15.52:0/981736470 learned_addr learned my addr 172.21.15.52:0/981736470 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-05-27T07:34:41.461 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:34:41.465+0000 7f394110d640 1 -- 172.21.15.52:0/981736470 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55b0f82b82f0 msgr2=0x55b0f8684720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:34:41.461 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:34:41.465+0000 7f394110d640 1 --2- 172.21.15.52:0/981736470 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55b0f82b82f0 0x55b0f8684720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:34:41.461 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:34:41.465+0000 7f394110d640 1 -- 172.21.15.52:0/981736470 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55b0f82e11b0 msgr2=0x55b0f8686690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:34:41.461 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:34:41.465+0000 7f394110d640 1 --2- 172.21.15.52:0/981736470 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55b0f82e11b0 0x55b0f8686690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:34:41.462 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:34:41.465+0000 7f394110d640 1 -- 172.21.15.52:0/981736470 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55b0f8684c60 con 0x55b0f82c04d0 2024-05-27T07:34:41.462 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:34:41.465+0000 7f394110d640 1 --2- 172.21.15.52:0/981736470 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55b0f82c04d0 0x55b0f8686150 secure :-1 s=READY pgs=718 cs=0 l=1 rev1=1 crypto rx=0x7f392c0a6e60 tx=0x7f392c0af2c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fecb3d09a6ff9a62 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:34:41.462 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:34:41.466+0000 7f39277fe640 1 -- 172.21.15.52:0/981736470 <== mon.0 v2:172.21.15.52:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f392c0a6950 con 0x55b0f82c04d0 2024-05-27T07:34:41.462 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:34:41.466+0000 7f39277fe640 1 -- 172.21.15.52:0/981736470 <== mon.0 v2:172.21.15.52:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f392c00c6a0 con 0x55b0f82c04d0 2024-05-27T07:34:41.462 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:34:41.466+0000 7f39277fe640 1 -- 172.21.15.52:0/981736470 <== mon.0 v2:172.21.15.52:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f392c0a6af0 con 0x55b0f82c04d0 2024-05-27T07:34:41.464 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:34:41.467+0000 7f394aa27740 1 -- 172.21.15.52:0/981736470 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55b0f82c04d0 msgr2=0x55b0f8686150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:34:41.464 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:34:41.467+0000 7f394aa27740 1 --2- 172.21.15.52:0/981736470 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55b0f82c04d0 0x55b0f8686150 secure :-1 s=READY pgs=718 cs=0 l=1 rev1=1 crypto rx=0x7f392c0a6e60 tx=0x7f392c0af2c0 comp rx=0 tx=0).stop 2024-05-27T07:34:41.464 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:34:41.467+0000 7f394aa27740 1 -- 172.21.15.52:0/981736470 shutdown_connections 2024-05-27T07:34:41.464 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:34:41.467+0000 7f394aa27740 1 --2- 172.21.15.52:0/981736470 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55b0f82e11b0 0x55b0f8686690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:34:41.464 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:34:41.467+0000 7f394aa27740 1 --2- 172.21.15.52:0/981736470 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55b0f82c04d0 0x55b0f8686150 unknown :-1 s=CLOSED pgs=718 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:34:41.464 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:34:41.467+0000 7f394aa27740 1 --2- 172.21.15.52:0/981736470 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55b0f82b82f0 0x55b0f8684720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:34:41.464 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:34:41.467+0000 7f394aa27740 1 -- 172.21.15.52:0/981736470 >> 172.21.15.52:0/981736470 conn(0x55b0f82d5300 msgr2=0x55b0f8684010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:34:41.464 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:34:41.468+0000 7f394aa27740 1 -- 172.21.15.52:0/981736470 shutdown_connections 2024-05-27T07:34:41.464 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:34:41.468+0000 7f394aa27740 1 -- 172.21.15.52:0/981736470 wait complete. 2024-05-27T07:34:41.464 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:34:41.468+0000 7f394aa27740 1 -- start start 2024-05-27T07:34:41.464 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:34:41.468+0000 7f394aa27740 1 Processor -- start 2024-05-27T07:34:41.464 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:34:41.469+0000 7f394aa27740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55b0f82b82f0 0x55b0f8694b60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:34:41.465 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:34:41.469+0000 7f394aa27740 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55b0f82c04d0 0x55b0f868f660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:34:41.465 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:34:41.469+0000 7f394aa27740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55b0f82e11b0 0x55b0f868fba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:34:41.465 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:34:41.469+0000 7f394aa27740 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x7f393c1bf940 con 0x55b0f82c04d0 2024-05-27T07:34:41.465 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:34:41.469+0000 7f394aa27740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f393c1bf240 con 0x55b0f82e11b0 2024-05-27T07:34:41.465 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:34:41.469+0000 7f394aa27740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x55b0f86996e0 con 0x55b0f82b82f0 2024-05-27T07:34:41.465 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:34:41.469+0000 7f394aa27740 10 client.0 _ll_register_callbacks cb 0x7f3938005ef0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T07:34:41.465 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:34:41.469+0000 7f394110d640 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55b0f82c04d0 0x55b0f868f660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:34:41.465 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:34:41.469+0000 7f394110d640 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55b0f82c04d0 0x55b0f868f660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.52:38032/0 (socket says 172.21.15.52:38032) 2024-05-27T07:34:41.465 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:34:41.469+0000 7f394110d640 1 -- 172.21.15.52:0/1700960650 learned_addr learned my addr 172.21.15.52:0/1700960650 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-05-27T07:34:41.467 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:34:41.470+0000 7f394110d640 1 -- 172.21.15.52:0/1700960650 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55b0f82b82f0 msgr2=0x55b0f8694b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:34:41.467 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:34:41.470+0000 7f394110d640 1 --2- 172.21.15.52:0/1700960650 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55b0f82b82f0 0x55b0f8694b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:34:41.467 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:34:41.470+0000 7f394110d640 1 -- 172.21.15.52:0/1700960650 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55b0f82e11b0 msgr2=0x55b0f868fba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:34:41.467 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:34:41.470+0000 7f394110d640 1 --2- 172.21.15.52:0/1700960650 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55b0f82e11b0 0x55b0f868fba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:34:41.467 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:34:41.470+0000 7f394110d640 1 -- 172.21.15.52:0/1700960650 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55b0f86900e0 con 0x55b0f82c04d0 2024-05-27T07:34:41.467 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:34:41.470+0000 7f394110d640 1 --2- 172.21.15.52:0/1700960650 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55b0f82c04d0 0x55b0f868f660 secure :-1 s=READY pgs=719 cs=0 l=1 rev1=1 crypto rx=0x7f392c0a7060 tx=0x7f392c0ae270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=38ee017d3f906479 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:34:41.468 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:34:41.470+0000 7f39257fa640 10 client.0 ms_handle_connect on v2:172.21.15.52:3300/0 2024-05-27T07:34:41.468 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:34:41.470+0000 7f39257fa640 1 -- 172.21.15.52:0/1700960650 <== mon.0 v2:172.21.15.52:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f392c0b6070 con 0x55b0f82c04d0 2024-05-27T07:34:41.468 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:34:41.470+0000 7f39257fa640 1 -- 172.21.15.52:0/1700960650 <== mon.0 v2:172.21.15.52:3300/0 2 ==== osd_map(120..120 src has 1..120) ==== 9555+0+0 (secure 0 0 0) 0x7f392c0a9bc0 con 0x55b0f82c04d0 2024-05-27T07:34:41.468 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:34:41.470+0000 7f394aa27740 1 -- 172.21.15.52:0/1700960650 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f392c00c6a0 con 0x55b0f82c04d0 2024-05-27T07:34:41.468 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:34:41.470+0000 7f39257fa640 1 -- 172.21.15.52:0/1700960650 <== mon.0 v2:172.21.15.52:3300/0 3 ==== mon_get_version_reply(handle=1 version=107) ==== 24+0+0 (secure 0 0 0) 0x7f392c0a6cc0 con 0x55b0f82c04d0 2024-05-27T07:34:41.468 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:34:41.470+0000 7f394aa27740 10 client.17025 fetch_fsmap learned FSMap version 107 2024-05-27T07:34:41.468 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:34:41.471+0000 7f394aa27740 1 -- 172.21.15.52:0/1700960650 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({config=0+,fsmap=107,monmap=4+}) -- 0x55b0f8706400 con 0x55b0f82c04d0 2024-05-27T07:34:41.468 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:34:41.471+0000 7f39257fa640 1 -- 172.21.15.52:0/1700960650 <== mon.0 v2:172.21.15.52:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f392c0e06e0 con 0x55b0f82c04d0 2024-05-27T07:34:41.468 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:34:41.471+0000 7f39257fa640 1 -- 172.21.15.52:0/1700960650 <== mon.0 v2:172.21.15.52:3300/0 5 ==== fsmap(e 107) ==== 4199+0+0 (secure 0 0 0) 0x7f392c0b1a60 con 0x55b0f82c04d0 2024-05-27T07:34:41.468 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:34:41.471+0000 7f394aa27740 10 client.17025 fetch_fsmap finished waiting for FSMap version 107 2024-05-27T07:34:41.468 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:34:41.471+0000 7f394aa27740 10 client.17025 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-05-27T07:34:41.468 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:34:41.471+0000 7f394aa27740 20 client.17025 populate_metadata read hostname 'smithi052' 2024-05-27T07:34:41.468 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:34:41.472+0000 7f394aa27740 1 --2- 172.21.15.52:0/1700960650 >> [v2:172.21.15.105:6838/500348151,v1:172.21.15.105:6839/500348151] conn(0x55b0f82751c0 0x55b0f87087f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:34:41.469 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:34:41.472+0000 7f394aa27740 4 client.17025 mds_command: new command op to 24527 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-05-27T07:34:41.469 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:34:41.472+0000 7f394aa27740 1 -- 172.21.15.52:0/1700960650 --> [v2:172.21.15.105:6838/500348151,v1:172.21.15.105:6839/500348151] -- command(tid 0: {"prefix": "flush journal"}) -- 0x7f392c00c6a0 con 0x55b0f82751c0 2024-05-27T07:34:41.469 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:34:41.473+0000 7f394190e640 1 --2- 172.21.15.52:0/1700960650 >> [v2:172.21.15.105:6838/500348151,v1:172.21.15.105:6839/500348151] conn(0x55b0f82751c0 0x55b0f87087f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:34:41.471 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:34:41.476+0000 7f394190e640 1 --2- 172.21.15.52:0/1700960650 >> [v2:172.21.15.105:6838/500348151,v1:172.21.15.105:6839/500348151] conn(0x55b0f82751c0 0x55b0f87087f0 crc :-1 s=READY pgs=158 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=638cd78cd904e278 server_cookie=1fdd8774925a6929 in_seq=0 out_seq=0 2024-05-27T07:34:41.471 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:34:41.476+0000 7f39257fa640 10 client.17025 ms_handle_connect on v2:172.21.15.105:6838/500348151 2024-05-27T07:34:41.694 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:34:41 smithi105 ceph-mon[35977]: pgmap v294: 129 pgs: 6 active+clean+snaptrim_wait, 2 active+clean+snaptrim, 121 active+clean; 662 MiB data, 5.3 GiB used, 1.0 TiB / 1.0 TiB avail; 483 KiB/s rd, 1.3 MiB/s wr, 878 op/s 2024-05-27T07:34:41.734 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:34:41 smithi052 ceph-mon[31955]: pgmap v294: 129 pgs: 6 active+clean+snaptrim_wait, 2 active+clean+snaptrim, 121 active+clean; 662 MiB data, 5.3 GiB used, 1.0 TiB / 1.0 TiB avail; 483 KiB/s rd, 1.3 MiB/s wr, 878 op/s 2024-05-27T07:34:41.856 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:34:41 smithi173 ceph-mon[37000]: pgmap v294: 129 pgs: 6 active+clean+snaptrim_wait, 2 active+clean+snaptrim, 121 active+clean; 662 MiB data, 5.3 GiB used, 1.0 TiB / 1.0 TiB avail; 483 KiB/s rd, 1.3 MiB/s wr, 878 op/s 2024-05-27T07:34:42.695 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:34:42 smithi105 ceph-mon[35977]: scrub summary: idle 2024-05-27T07:34:42.695 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:34:42 smithi105 ceph-mon[35977]: scrub summary: idle 2024-05-27T07:34:42.695 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:34:42 smithi105 ceph-mon[35977]: scrub summary: idle 2024-05-27T07:34:42.695 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:34:42 smithi105 ceph-mon[35977]: scrub summary: idle 2024-05-27T07:34:42.734 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:34:42 smithi052 ceph-mon[31955]: scrub summary: idle 2024-05-27T07:34:42.734 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:34:42 smithi052 ceph-mon[31955]: scrub summary: idle 2024-05-27T07:34:42.735 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:34:42 smithi052 ceph-mon[31955]: scrub summary: idle 2024-05-27T07:34:42.735 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:34:42 smithi052 ceph-mon[31955]: scrub summary: idle 2024-05-27T07:34:42.855 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:34:42.859+0000 7f39257fa640 1 -- 172.21.15.52:0/1700960650 <== mds.0 v2:172.21.15.105:6838/500348151 1 ==== command_reply(tid 0: 0 ) ==== 8+0+25 (crc 0 0 0) 0x7f392c00c6a0 con 0x55b0f82751c0 2024-05-27T07:34:42.856 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:34:42.859+0000 7f39257fa640 10 client.17025 handle_command_reply: tid=0 2024-05-27T07:34:42.856 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:34:42.859+0000 7f394aa27740 1 client.17025 shutdown 2024-05-27T07:34:42.856 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:34:42.859+0000 7f394aa27740 1 -- 172.21.15.52:0/1700960650 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55b0f82c04d0 msgr2=0x55b0f868f660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:34:42.856 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:34:42.859+0000 7f394aa27740 1 --2- 172.21.15.52:0/1700960650 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55b0f82c04d0 0x55b0f868f660 secure :-1 s=READY pgs=719 cs=0 l=1 rev1=1 crypto rx=0x7f392c0a7060 tx=0x7f392c0ae270 comp rx=0 tx=0).stop 2024-05-27T07:34:42.856 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:34:42.860+0000 7f394aa27740 1 -- 172.21.15.52:0/1700960650 shutdown_connections 2024-05-27T07:34:42.856 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:34:42.860+0000 7f394aa27740 1 --2- 172.21.15.52:0/1700960650 >> [v2:172.21.15.105:6838/500348151,v1:172.21.15.105:6839/500348151] conn(0x55b0f82751c0 0x55b0f87087f0 crc :-1 s=READY pgs=158 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:34:42.856 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:34:42.860+0000 7f39257fa640 0 client.17025 ms_handle_reset on v2:172.21.15.105:6838/500348151 2024-05-27T07:34:42.856 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:34:42.860+0000 7f394aa27740 1 --2- 172.21.15.52:0/1700960650 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55b0f82e11b0 0x55b0f868fba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:34:42.857 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:34:42.860+0000 7f394aa27740 1 --2- 172.21.15.52:0/1700960650 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55b0f82c04d0 0x55b0f868f660 unknown :-1 s=CLOSED pgs=719 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:34:42.857 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:34:42.860+0000 7f394aa27740 1 --2- 172.21.15.52:0/1700960650 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55b0f82b82f0 0x55b0f8694b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:34:42.857 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:34:42.860+0000 7f394aa27740 1 -- 172.21.15.52:0/1700960650 >> 172.21.15.52:0/1700960650 conn(0x55b0f82d5300 msgr2=0x55b0f86822b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:34:42.857 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:34:42.860+0000 7f394aa27740 1 -- 172.21.15.52:0/1700960650 shutdown_connections 2024-05-27T07:34:42.857 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:34:42.860+0000 7f394aa27740 1 -- 172.21.15.52:0/1700960650 wait complete. 2024-05-27T07:34:42.857 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:34:42.860+0000 7f394aa27740 20 client.17025 trim_cache size 0 max 16384 2024-05-27T07:34:42.858 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stdout:{ 2024-05-27T07:34:42.858 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stdout: "return_code": 0 2024-05-27T07:34:42.858 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stdout:} 2024-05-27T07:34:42.860 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:34:42 smithi173 ceph-mon[37000]: scrub summary: idle 2024-05-27T07:34:42.860 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:34:42 smithi173 ceph-mon[37000]: scrub summary: idle 2024-05-27T07:34:42.860 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:34:42 smithi173 ceph-mon[37000]: scrub summary: idle 2024-05-27T07:34:42.861 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:34:42 smithi173 ceph-mon[37000]: scrub summary: idle 2024-05-27T07:34:42.861 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:34:42.861+0000 7f3943b72640 1 -- 172.21.15.52:0/1881844083 >> [v2:172.21.15.52:6800/1822318384,v1:172.21.15.52:6801/1822318384] conn(0x7f39280b2af0 msgr2=0x7f39280b5000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:34:42.861 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:34:42.861+0000 7f3943b72640 1 --2- 172.21.15.52:0/1881844083 >> [v2:172.21.15.52:6800/1822318384,v1:172.21.15.52:6801/1822318384] conn(0x7f39280b2af0 0x7f39280b5000 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7f3938005fd0 tx=0x7f39380095a0 comp rx=0 tx=0).stop 2024-05-27T07:34:42.861 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:34:42.862+0000 7f3943b72640 1 -- 172.21.15.52:0/1881844083 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f393c11a8e0 msgr2=0x7f393c1a9b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:34:42.861 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:34:42.862+0000 7f3943b72640 1 --2- 172.21.15.52:0/1881844083 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f393c11a8e0 0x7f393c1a9b90 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7f392c00e9f0 tx=0x7f392c00edf0 comp rx=0 tx=0).stop 2024-05-27T07:34:42.861 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:34:42.862+0000 7f3943b72640 1 -- 172.21.15.52:0/1881844083 shutdown_connections 2024-05-27T07:34:42.861 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:34:42.862+0000 7f3943b72640 1 --2- 172.21.15.52:0/1881844083 >> [v2:172.21.15.52:6800/1822318384,v1:172.21.15.52:6801/1822318384] conn(0x7f39280b2af0 0x7f39280b5000 unknown :-1 s=CLOSED pgs=176 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:34:42.861 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:34:42.862+0000 7f3943b72640 1 --2- 172.21.15.52:0/1881844083 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f393c11d240 0x7f393c1ade00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:34:42.861 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:34:42.862+0000 7f3943b72640 1 --2- 172.21.15.52:0/1881844083 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f393c11a8e0 0x7f393c1a9b90 unknown :-1 s=CLOSED pgs=191 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:34:42.862 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:34:42.862+0000 7f3943b72640 1 --2- 172.21.15.52:0/1881844083 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f393c10a1f0 0x7f393c1a9630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:34:42.862 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:34:42.862+0000 7f3943b72640 1 -- 172.21.15.52:0/1881844083 >> 172.21.15.52:0/1881844083 conn(0x7f393c06e8f0 msgr2=0x7f393c11acf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:34:42.862 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:34:42.862+0000 7f3943b72640 1 -- 172.21.15.52:0/1881844083 shutdown_connections 2024-05-27T07:34:42.862 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:34:42.862+0000 7f3943b72640 1 -- 172.21.15.52:0/1881844083 wait complete. 2024-05-27T07:34:43.695 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:34:43 smithi105 ceph-mon[35977]: scrub summary: idle 2024-05-27T07:34:43.695 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:34:43 smithi105 ceph-mon[35977]: pgmap v295: 129 pgs: 6 active+clean+snaptrim_wait, 2 active+clean+snaptrim, 121 active+clean; 653 MiB data, 5.4 GiB used, 1.0 TiB / 1.0 TiB avail; 546 KiB/s rd, 1.5 MiB/s wr, 989 op/s 2024-05-27T07:34:43.734 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:34:43 smithi052 ceph-mon[31955]: scrub summary: idle 2024-05-27T07:34:43.734 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:34:43 smithi052 ceph-mon[31955]: pgmap v295: 129 pgs: 6 active+clean+snaptrim_wait, 2 active+clean+snaptrim, 121 active+clean; 653 MiB data, 5.4 GiB used, 1.0 TiB / 1.0 TiB avail; 546 KiB/s rd, 1.5 MiB/s wr, 989 op/s 2024-05-27T07:34:43.857 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:34:43 smithi173 ceph-mon[37000]: scrub summary: idle 2024-05-27T07:34:43.857 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:34:43 smithi173 ceph-mon[37000]: pgmap v295: 129 pgs: 6 active+clean+snaptrim_wait, 2 active+clean+snaptrim, 121 active+clean; 653 MiB data, 5.4 GiB used, 1.0 TiB / 1.0 TiB avail; 546 KiB/s rd, 1.5 MiB/s wr, 989 op/s 2024-05-27T07:34:46.484 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:34:46 smithi052 ceph-mon[31955]: pgmap v296: 129 pgs: 6 active+clean+snaptrim_wait, 2 active+clean+snaptrim, 121 active+clean; 654 MiB data, 5.4 GiB used, 1.0 TiB / 1.0 TiB avail; 494 KiB/s rd, 1.2 MiB/s wr, 928 op/s 2024-05-27T07:34:46.559 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:34:46 smithi173 ceph-mon[37000]: pgmap v296: 129 pgs: 6 active+clean+snaptrim_wait, 2 active+clean+snaptrim, 121 active+clean; 654 MiB data, 5.4 GiB used, 1.0 TiB / 1.0 TiB avail; 494 KiB/s rd, 1.2 MiB/s wr, 928 op/s 2024-05-27T07:34:46.694 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:34:46 smithi105 ceph-mon[35977]: pgmap v296: 129 pgs: 6 active+clean+snaptrim_wait, 2 active+clean+snaptrim, 121 active+clean; 654 MiB data, 5.4 GiB used, 1.0 TiB / 1.0 TiB avail; 494 KiB/s rd, 1.2 MiB/s wr, 928 op/s 2024-05-27T07:34:48.484 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:34:48 smithi052 ceph-mon[31955]: pgmap v297: 129 pgs: 1 active+clean+snaptrim, 128 active+clean; 650 MiB data, 5.4 GiB used, 1.0 TiB / 1.0 TiB avail; 404 KiB/s rd, 1.2 MiB/s wr, 897 op/s 2024-05-27T07:34:48.607 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:34:48 smithi173 ceph-mon[37000]: pgmap v297: 129 pgs: 1 active+clean+snaptrim, 128 active+clean; 650 MiB data, 5.4 GiB used, 1.0 TiB / 1.0 TiB avail; 404 KiB/s rd, 1.2 MiB/s wr, 897 op/s 2024-05-27T07:34:48.695 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:34:48 smithi105 ceph-mon[35977]: pgmap v297: 129 pgs: 1 active+clean+snaptrim, 128 active+clean; 650 MiB data, 5.4 GiB used, 1.0 TiB / 1.0 TiB avail; 404 KiB/s rd, 1.2 MiB/s wr, 897 op/s 2024-05-27T07:34:50.484 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:34:50 smithi052 ceph-mon[31955]: pgmap v298: 129 pgs: 1 active+clean+snaptrim, 128 active+clean; 655 MiB data, 5.5 GiB used, 1.0 TiB / 1.0 TiB avail; 269 KiB/s rd, 1.5 MiB/s wr, 755 op/s 2024-05-27T07:34:50.484 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:34:50 smithi052 ceph-mon[31955]: osdmap e121: 12 total, 12 up, 12 in 2024-05-27T07:34:50.607 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:34:50 smithi173 ceph-mon[37000]: pgmap v298: 129 pgs: 1 active+clean+snaptrim, 128 active+clean; 655 MiB data, 5.5 GiB used, 1.0 TiB / 1.0 TiB avail; 269 KiB/s rd, 1.5 MiB/s wr, 755 op/s 2024-05-27T07:34:50.607 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:34:50 smithi173 ceph-mon[37000]: osdmap e121: 12 total, 12 up, 12 in 2024-05-27T07:34:50.695 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:34:50 smithi105 ceph-mon[35977]: pgmap v298: 129 pgs: 1 active+clean+snaptrim, 128 active+clean; 655 MiB data, 5.5 GiB used, 1.0 TiB / 1.0 TiB avail; 269 KiB/s rd, 1.5 MiB/s wr, 755 op/s 2024-05-27T07:34:50.695 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:34:50 smithi105 ceph-mon[35977]: osdmap e121: 12 total, 12 up, 12 in 2024-05-27T07:34:52.607 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:34:52 smithi173 ceph-mon[37000]: pgmap v300: 129 pgs: 1 active+clean+snaptrim, 128 active+clean; 658 MiB data, 5.5 GiB used, 1.0 TiB / 1.0 TiB avail; 134 KiB/s rd, 1.8 MiB/s wr, 622 op/s 2024-05-27T07:34:52.695 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:34:52 smithi105 ceph-mon[35977]: pgmap v300: 129 pgs: 1 active+clean+snaptrim, 128 active+clean; 658 MiB data, 5.5 GiB used, 1.0 TiB / 1.0 TiB avail; 134 KiB/s rd, 1.8 MiB/s wr, 622 op/s 2024-05-27T07:34:52.734 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:34:52 smithi052 ceph-mon[31955]: pgmap v300: 129 pgs: 1 active+clean+snaptrim, 128 active+clean; 658 MiB data, 5.5 GiB used, 1.0 TiB / 1.0 TiB avail; 134 KiB/s rd, 1.8 MiB/s wr, 622 op/s 2024-05-27T07:34:54.607 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:34:54 smithi173 ceph-mon[37000]: pgmap v301: 129 pgs: 129 active+clean; 662 MiB data, 5.5 GiB used, 1.0 TiB / 1.0 TiB avail; 32 KiB/s rd, 1.8 MiB/s wr, 498 op/s 2024-05-27T07:34:54.695 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:34:54 smithi105 ceph-mon[35977]: pgmap v301: 129 pgs: 129 active+clean; 662 MiB data, 5.5 GiB used, 1.0 TiB / 1.0 TiB avail; 32 KiB/s rd, 1.8 MiB/s wr, 498 op/s 2024-05-27T07:34:54.734 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:34:54 smithi052 ceph-mon[31955]: pgmap v301: 129 pgs: 129 active+clean; 662 MiB data, 5.5 GiB used, 1.0 TiB / 1.0 TiB avail; 32 KiB/s rd, 1.8 MiB/s wr, 498 op/s 2024-05-27T07:34:55.979 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:34:55.971+0000 7fc0a4842640 1 -- 172.21.15.52:0/1051949975 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fc0a0109c70 msgr2=0x7fc0a0103810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:34:55.979 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:34:55.971+0000 7fc0a4842640 1 --2- 172.21.15.52:0/1051949975 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fc0a0109c70 0x7fc0a0103810 secure :-1 s=READY pgs=720 cs=0 l=1 rev1=1 crypto rx=0x7fc090009900 tx=0x7fc09002a410 comp rx=0 tx=0).stop 2024-05-27T07:34:55.979 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:34:55.971+0000 7fc0a4842640 1 -- 172.21.15.52:0/1051949975 shutdown_connections 2024-05-27T07:34:55.979 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:34:55.971+0000 7fc0a4842640 1 --2- 172.21.15.52:0/1051949975 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fc0a01043c0 0x7fc0a0110790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:34:55.979 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:34:55.971+0000 7fc0a4842640 1 --2- 172.21.15.52:0/1051949975 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fc0a010a110 0x7fc0a0103d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:34:55.979 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:34:55.971+0000 7fc0a4842640 1 --2- 172.21.15.52:0/1051949975 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fc0a0109c70 0x7fc0a0103810 unknown :-1 s=CLOSED pgs=720 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:34:55.979 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:34:55.971+0000 7fc0a4842640 1 -- 172.21.15.52:0/1051949975 >> 172.21.15.52:0/1051949975 conn(0x7fc0a00ff350 msgr2=0x7fc0a01017a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:34:55.980 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:34:55.971+0000 7fc0a4842640 1 -- 172.21.15.52:0/1051949975 shutdown_connections 2024-05-27T07:34:55.980 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:34:55.971+0000 7fc0a4842640 1 -- 172.21.15.52:0/1051949975 wait complete. 2024-05-27T07:34:55.980 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:34:55.972+0000 7fc0a4842640 1 Processor -- start 2024-05-27T07:34:55.980 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:34:55.972+0000 7fc0a4842640 1 -- start start 2024-05-27T07:34:55.980 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:34:55.972+0000 7fc0a4842640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fc0a01043c0 0x7fc0a01a0590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:34:55.980 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:34:55.972+0000 7fc0a4842640 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fc0a0109c70 0x7fc0a01a0ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:34:55.980 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:34:55.972+0000 7fc0a4842640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fc0a010a110 0x7fc0a019a710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:34:55.980 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:34:55.972+0000 7fc0a4842640 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x7fc0a0113eb0 con 0x7fc0a0109c70 2024-05-27T07:34:55.980 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:34:55.972+0000 7fc0a4842640 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7fc0a0113d30 con 0x7fc0a01043c0 2024-05-27T07:34:55.980 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:34:55.972+0000 7fc0a4842640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7fc0a0114030 con 0x7fc0a010a110 2024-05-27T07:34:55.980 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:34:55.972+0000 7fc097fff640 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fc0a0109c70 0x7fc0a01a0ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:34:55.981 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:34:55.972+0000 7fc097fff640 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fc0a0109c70 0x7fc0a01a0ad0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.52:60498/0 (socket says 172.21.15.52:60498) 2024-05-27T07:34:55.981 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:34:55.972+0000 7fc09e59c640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fc0a01043c0 0x7fc0a01a0590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:34:55.981 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:34:55.973+0000 7fc097fff640 1 -- 172.21.15.52:0/2661018427 learned_addr learned my addr 172.21.15.52:0/2661018427 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-05-27T07:34:55.981 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:34:55.973+0000 7fc09ed9d640 1 --2- 172.21.15.52:0/2661018427 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fc0a010a110 0x7fc0a019a710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:34:55.981 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:34:55.973+0000 7fc097fff640 1 -- 172.21.15.52:0/2661018427 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fc0a010a110 msgr2=0x7fc0a019a710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:34:55.981 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:34:55.973+0000 7fc097fff640 1 --2- 172.21.15.52:0/2661018427 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fc0a010a110 0x7fc0a019a710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:34:55.981 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:34:55.973+0000 7fc097fff640 1 -- 172.21.15.52:0/2661018427 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fc0a01043c0 msgr2=0x7fc0a01a0590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:34:55.981 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:34:55.973+0000 7fc097fff640 1 --2- 172.21.15.52:0/2661018427 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fc0a01043c0 0x7fc0a01a0590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:34:55.981 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:34:55.973+0000 7fc097fff640 1 -- 172.21.15.52:0/2661018427 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc0a019ae40 con 0x7fc0a0109c70 2024-05-27T07:34:55.982 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:34:55.973+0000 7fc09ed9d640 1 --2- 172.21.15.52:0/2661018427 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fc0a010a110 0x7fc0a019a710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-27T07:34:55.982 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:34:55.973+0000 7fc09e59c640 1 --2- 172.21.15.52:0/2661018427 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fc0a01043c0 0x7fc0a01a0590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-27T07:34:55.982 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:34:55.973+0000 7fc097fff640 1 --2- 172.21.15.52:0/2661018427 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fc0a0109c70 0x7fc0a01a0ad0 secure :-1 s=READY pgs=721 cs=0 l=1 rev1=1 crypto rx=0x7fc08800e9f0 tx=0x7fc08800eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:34:55.982 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:34:55.973+0000 7fc0977fe640 1 -- 172.21.15.52:0/2661018427 <== mon.0 v2:172.21.15.52:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc08800ccc0 con 0x7fc0a0109c70 2024-05-27T07:34:55.982 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:34:55.973+0000 7fc0977fe640 1 -- 172.21.15.52:0/2661018427 <== mon.0 v2:172.21.15.52:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fc0880035f0 con 0x7fc0a0109c70 2024-05-27T07:34:55.982 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:34:55.973+0000 7fc0a4842640 1 -- 172.21.15.52:0/2661018427 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc0a019b0d0 con 0x7fc0a0109c70 2024-05-27T07:34:55.982 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:34:55.973+0000 7fc0977fe640 1 -- 172.21.15.52:0/2661018427 <== mon.0 v2:172.21.15.52:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc08800ce60 con 0x7fc0a0109c70 2024-05-27T07:34:55.982 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:34:55.973+0000 7fc0a4842640 1 -- 172.21.15.52:0/2661018427 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc0a01a7330 con 0x7fc0a0109c70 2024-05-27T07:34:55.982 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:34:55.976+0000 7fc0ab7cf740 1 Processor -- start 2024-05-27T07:34:55.983 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:34:55.976+0000 7fc0ab7cf740 1 -- start start 2024-05-27T07:34:55.983 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:34:55.976+0000 7fc0ab7cf740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x557a130331b0 0x557a133d43d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:34:55.983 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:34:55.976+0000 7fc0ab7cf740 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x557a1300a2f0 0x557a133d4910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:34:55.983 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:34:55.976+0000 7fc0ab7cf740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x557a130124d0 0x557a133d4e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:34:55.983 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:34:55.976+0000 7fc0ab7cf740 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x557a13346c30 con 0x557a1300a2f0 2024-05-27T07:34:55.983 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:34:55.976+0000 7fc0ab7cf740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x557a133367d0 con 0x557a130331b0 2024-05-27T07:34:55.983 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:34:55.976+0000 7fc0ab7cf740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x557a1333a810 con 0x557a130124d0 2024-05-27T07:34:55.983 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:34:55.976+0000 7fc09e59c640 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x557a1300a2f0 0x557a133d4910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:34:55.983 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:34:55.976+0000 7fc09ed9d640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x557a130331b0 0x557a133d43d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:34:55.983 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:34:55.976+0000 7fc09e59c640 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x557a1300a2f0 0x557a133d4910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.52:60500/0 (socket says 172.21.15.52:60500) 2024-05-27T07:34:55.983 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:34:55.976+0000 7fc09e59c640 1 -- 172.21.15.52:0/4227537322 learned_addr learned my addr 172.21.15.52:0/4227537322 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-05-27T07:34:55.983 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:34:55.976+0000 7fc0977fe640 1 -- 172.21.15.52:0/2661018427 <== mon.0 v2:172.21.15.52:3300/0 4 ==== mgrmap(e 26) ==== 149783+0+0 (secure 0 0 0) 0x7fc088004020 con 0x7fc0a0109c70 2024-05-27T07:34:55.983 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:34:55.977+0000 7fc09ed9d640 1 -- 172.21.15.52:0/4227537322 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x557a130124d0 msgr2=0x557a133d4e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:34:55.984 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:34:55.977+0000 7fc09ed9d640 1 --2- 172.21.15.52:0/4227537322 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x557a130124d0 0x557a133d4e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:34:55.984 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:34:55.977+0000 7fc09ed9d640 1 -- 172.21.15.52:0/4227537322 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x557a1300a2f0 msgr2=0x557a133d4910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:34:55.984 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:34:55.977+0000 7fc09ed9d640 1 --2- 172.21.15.52:0/4227537322 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x557a1300a2f0 0x557a133d4910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:34:55.984 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:34:55.977+0000 7fc09ed9d640 1 -- 172.21.15.52:0/4227537322 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x557a1304b210 con 0x557a130331b0 2024-05-27T07:34:55.984 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:34:55.977+0000 7fc09ed9d640 1 --2- 172.21.15.52:0/4227537322 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x557a130331b0 0x557a133d43d0 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7fc098009de0 tx=0x7fc09800f730 comp rx=0 tx=0).ready entity=mon.2 client_cookie=5514b14aaeb5fe8d server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:34:55.984 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:34:55.977+0000 7fc094ff9640 1 -- 172.21.15.52:0/4227537322 <== mon.2 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc098004b20 con 0x557a130331b0 2024-05-27T07:34:55.984 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:34:55.977+0000 7fc094ff9640 1 -- 172.21.15.52:0/4227537322 <== mon.2 v2:172.21.15.105:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fc098004cc0 con 0x557a130331b0 2024-05-27T07:34:55.984 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:34:55.977+0000 7fc094ff9640 1 -- 172.21.15.52:0/4227537322 <== mon.2 v2:172.21.15.105:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc098011860 con 0x557a130331b0 2024-05-27T07:34:55.984 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:34:55.977+0000 7fc0977fe640 1 --2- 172.21.15.52:0/2661018427 >> [v2:172.21.15.52:6800/1822318384,v1:172.21.15.52:6801/1822318384] conn(0x7fc08c0b2950 0x7fc08c0b4e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:34:55.984 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:34:55.977+0000 7fc09e59c640 1 --2- 172.21.15.52:0/2661018427 >> [v2:172.21.15.52:6800/1822318384,v1:172.21.15.52:6801/1822318384] conn(0x7fc08c0b2950 0x7fc08c0b4e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:34:55.984 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:34:55.978+0000 7fc0977fe640 1 -- 172.21.15.52:0/2661018427 <== mon.0 v2:172.21.15.52:3300/0 5 ==== osd_map(122..122 src has 1..122) ==== 9423+0+0 (secure 0 0 0) 0x7fc0880e2ce0 con 0x7fc0a0109c70 2024-05-27T07:34:55.984 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:34:55.978+0000 7fc0ab7cf740 1 -- 172.21.15.52:0/4227537322 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x557a130331b0 msgr2=0x557a133d43d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:34:55.985 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:34:55.978+0000 7fc09e59c640 1 --2- 172.21.15.52:0/2661018427 >> [v2:172.21.15.52:6800/1822318384,v1:172.21.15.52:6801/1822318384] conn(0x7fc08c0b2950 0x7fc08c0b4e60 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7fc090006010 tx=0x7fc090005a20 comp rx=0 tx=0).ready entity=mgr.15333 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:34:55.985 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:34:55.978+0000 7fc0ab7cf740 1 --2- 172.21.15.52:0/4227537322 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x557a130331b0 0x557a133d43d0 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7fc098009de0 tx=0x7fc09800f730 comp rx=0 tx=0).stop 2024-05-27T07:34:55.985 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:34:55.978+0000 7fc0ab7cf740 1 -- 172.21.15.52:0/4227537322 shutdown_connections 2024-05-27T07:34:55.985 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:34:55.978+0000 7fc0ab7cf740 1 --2- 172.21.15.52:0/4227537322 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x557a130124d0 0x557a133d4e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:34:55.985 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:34:55.978+0000 7fc0ab7cf740 1 --2- 172.21.15.52:0/4227537322 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x557a1300a2f0 0x557a133d4910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:34:55.985 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:34:55.978+0000 7fc0ab7cf740 1 --2- 172.21.15.52:0/4227537322 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x557a130331b0 0x557a133d43d0 unknown :-1 s=CLOSED pgs=240 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:34:55.985 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:34:55.978+0000 7fc0ab7cf740 1 -- 172.21.15.52:0/4227537322 >> 172.21.15.52:0/4227537322 conn(0x557a13027300 msgr2=0x557a133d0b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:34:55.985 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:34:55.978+0000 7fc0ab7cf740 1 -- 172.21.15.52:0/4227537322 shutdown_connections 2024-05-27T07:34:55.985 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:34:55.978+0000 7fc0ab7cf740 1 -- 172.21.15.52:0/4227537322 wait complete. 2024-05-27T07:34:55.985 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:34:55.979+0000 7fc0ab7cf740 1 -- start start 2024-05-27T07:34:55.985 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:34:55.979+0000 7fc0ab7cf740 1 Processor -- start 2024-05-27T07:34:55.985 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:34:55.979+0000 7fc0ab7cf740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x557a1300a2f0 0x557a133e0470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:34:55.985 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:34:55.979+0000 7fc0ab7cf740 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x557a130124d0 0x557a133e09d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:34:55.986 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:34:55.979+0000 7fc0ab7cf740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x557a130331b0 0x557a133de910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:34:55.986 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:34:55.979+0000 7fc0ab7cf740 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x557a1301c790 con 0x557a130124d0 2024-05-27T07:34:55.986 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:34:55.979+0000 7fc0ab7cf740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x557a132f2890 con 0x557a130331b0 2024-05-27T07:34:55.986 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:34:55.979+0000 7fc0ab7cf740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x557a133e1500 con 0x557a1300a2f0 2024-05-27T07:34:55.986 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:34:55.979+0000 7fc0ab7cf740 10 client.0 _ll_register_callbacks cb 0x557a1304aa00 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T07:34:55.986 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:34:55.979+0000 7fc097fff640 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x557a130124d0 0x557a133e09d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:34:55.986 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:34:55.979+0000 7fc097fff640 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x557a130124d0 0x557a133e09d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.52:60508/0 (socket says 172.21.15.52:60508) 2024-05-27T07:34:55.986 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:34:55.979+0000 7fc097fff640 1 -- 172.21.15.52:0/2130519988 learned_addr learned my addr 172.21.15.52:0/2130519988 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-05-27T07:34:55.986 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:34:55.979+0000 7fc09ed9d640 1 --2- 172.21.15.52:0/2130519988 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x557a130331b0 0x557a133de910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:34:55.986 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:34:55.979+0000 7fc09e59c640 1 --2- 172.21.15.52:0/2130519988 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x557a1300a2f0 0x557a133e0470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:34:55.986 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:34:55.980+0000 7fc097fff640 1 -- 172.21.15.52:0/2130519988 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x557a1300a2f0 msgr2=0x557a133e0470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:34:55.987 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:34:55.980+0000 7fc097fff640 1 --2- 172.21.15.52:0/2130519988 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x557a1300a2f0 0x557a133e0470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:34:55.987 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:34:55.980+0000 7fc097fff640 1 -- 172.21.15.52:0/2130519988 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x557a130331b0 msgr2=0x557a133de910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:34:55.987 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:34:55.980+0000 7fc097fff640 1 --2- 172.21.15.52:0/2130519988 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x557a130331b0 0x557a133de910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:34:55.987 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:34:55.980+0000 7fc097fff640 1 -- 172.21.15.52:0/2130519988 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({osdmap=0}) -- 0x557a13117620 con 0x557a130124d0 2024-05-27T07:34:55.987 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:34:55.980+0000 7fc097fff640 1 --2- 172.21.15.52:0/2130519988 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x557a130124d0 0x557a133e09d0 secure :-1 s=READY pgs=722 cs=0 l=1 rev1=1 crypto rx=0x7fc0880a9940 tx=0x7fc088004310 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c63832e00e4c3a8 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:34:55.987 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:34:55.980+0000 7fc085ffb640 10 client.0 ms_handle_connect on v2:172.21.15.52:3300/0 2024-05-27T07:34:55.987 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:34:55.980+0000 7fc085ffb640 1 -- 172.21.15.52:0/2130519988 <== mon.0 v2:172.21.15.52:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc0880aa340 con 0x557a130124d0 2024-05-27T07:34:55.987 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:34:55.980+0000 7fc085ffb640 1 -- 172.21.15.52:0/2130519988 <== mon.0 v2:172.21.15.52:3300/0 2 ==== osd_map(122..122 src has 1..122) ==== 9423+0+0 (secure 0 0 0) 0x7fc088003aa0 con 0x557a130124d0 2024-05-27T07:34:55.987 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:34:55.980+0000 7fc0ab7cf740 1 -- 172.21.15.52:0/2130519988 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fc098004cc0 con 0x557a130124d0 2024-05-27T07:34:55.987 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:34:55.980+0000 7fc085ffb640 1 -- 172.21.15.52:0/2130519988 <== mon.0 v2:172.21.15.52:3300/0 3 ==== mon_get_version_reply(handle=1 version=107) ==== 24+0+0 (secure 0 0 0) 0x7fc08806e050 con 0x557a130124d0 2024-05-27T07:34:55.988 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:34:55.980+0000 7fc0ab7cf740 10 client.17037 fetch_fsmap learned FSMap version 107 2024-05-27T07:34:55.988 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:34:55.980+0000 7fc0ab7cf740 1 -- 172.21.15.52:0/2130519988 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({config=0+,fsmap=107,monmap=4+}) -- 0x557a133ed900 con 0x557a130124d0 2024-05-27T07:34:55.988 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:34:55.980+0000 7fc085ffb640 1 -- 172.21.15.52:0/2130519988 <== mon.0 v2:172.21.15.52:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fc098004cc0 con 0x557a130124d0 2024-05-27T07:34:55.988 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:34:55.980+0000 7fc085ffb640 1 -- 172.21.15.52:0/2130519988 <== mon.0 v2:172.21.15.52:3300/0 5 ==== fsmap(e 107) ==== 4199+0+0 (secure 0 0 0) 0x7fc0880b2610 con 0x557a130124d0 2024-05-27T07:34:55.988 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:34:55.980+0000 7fc0ab7cf740 10 client.17037 fetch_fsmap finished waiting for FSMap version 107 2024-05-27T07:34:55.988 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:34:55.980+0000 7fc0ab7cf740 10 client.17037 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-05-27T07:34:55.988 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:34:55.980+0000 7fc0ab7cf740 20 client.17037 populate_metadata read hostname 'smithi052' 2024-05-27T07:34:55.988 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:34:55.981+0000 7fc0ab7cf740 1 --2- 172.21.15.52:0/2130519988 >> [v2:172.21.15.105:6838/500348151,v1:172.21.15.105:6839/500348151] conn(0x557a12ff1d80 0x557a133f1190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:34:55.989 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:34:55.981+0000 7fc0ab7cf740 4 client.17037 mds_command: new command op to 24527 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-27T07:34:55.989 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:34:55.981+0000 7fc0ab7cf740 1 -- 172.21.15.52:0/2130519988 --> [v2:172.21.15.105:6838/500348151,v1:172.21.15.105:6839/500348151] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x557a13348b80 con 0x557a12ff1d80 2024-05-27T07:34:55.989 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:34:55.981+0000 7fc09e59c640 1 --2- 172.21.15.52:0/2130519988 >> [v2:172.21.15.105:6838/500348151,v1:172.21.15.105:6839/500348151] conn(0x557a12ff1d80 0x557a133f1190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:34:55.989 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:34:55.981+0000 7fc085ffb640 10 client.17037 ms_handle_connect on v2:172.21.15.105:6838/500348151 2024-05-27T07:34:55.989 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:34:55.981+0000 7fc09e59c640 1 --2- 172.21.15.52:0/2130519988 >> [v2:172.21.15.105:6838/500348151,v1:172.21.15.105:6839/500348151] conn(0x557a12ff1d80 0x557a133f1190 crc :-1 s=READY pgs=159 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=b4442f6d7ffd8440 server_cookie=16dd23f04550ce20 in_seq=0 out_seq=0 2024-05-27T07:34:55.989 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:34:55.982+0000 7fc085ffb640 1 -- 172.21.15.52:0/2130519988 <== mds.0 v2:172.21.15.105:6838/500348151 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24041 (crc 0 0 0) 0x557a13348b80 con 0x557a12ff1d80 2024-05-27T07:34:55.989 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:34:55.982+0000 7fc085ffb640 10 client.17037 handle_command_reply: tid=0 2024-05-27T07:34:55.989 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:34:55.982+0000 7fc0ab7cf740 1 client.17037 shutdown 2024-05-27T07:34:55.989 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:34:55.982+0000 7fc0ab7cf740 1 -- 172.21.15.52:0/2130519988 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x557a130124d0 msgr2=0x557a133e09d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:34:55.990 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:34:55.982+0000 7fc0ab7cf740 1 --2- 172.21.15.52:0/2130519988 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x557a130124d0 0x557a133e09d0 secure :-1 s=READY pgs=722 cs=0 l=1 rev1=1 crypto rx=0x7fc0880a9940 tx=0x7fc088004310 comp rx=0 tx=0).stop 2024-05-27T07:34:55.990 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:34:55.982+0000 7fc0ab7cf740 1 -- 172.21.15.52:0/2130519988 shutdown_connections 2024-05-27T07:34:55.990 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:34:55.982+0000 7fc0ab7cf740 1 --2- 172.21.15.52:0/2130519988 >> [v2:172.21.15.105:6838/500348151,v1:172.21.15.105:6839/500348151] conn(0x557a12ff1d80 0x557a133f1190 crc :-1 s=READY pgs=159 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:34:55.990 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:34:55.982+0000 7fc0ab7cf740 1 --2- 172.21.15.52:0/2130519988 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x557a130331b0 0x557a133de910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:34:55.990 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:34:55.982+0000 7fc0ab7cf740 1 --2- 172.21.15.52:0/2130519988 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x557a130124d0 0x557a133e09d0 unknown :-1 s=CLOSED pgs=722 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:34:55.990 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:34:55.982+0000 7fc085ffb640 0 client.17037 ms_handle_reset on v2:172.21.15.105:6838/500348151 2024-05-27T07:34:55.990 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:34:55.982+0000 7fc0ab7cf740 1 --2- 172.21.15.52:0/2130519988 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x557a1300a2f0 0x557a133e0470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:34:55.990 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:34:55.982+0000 7fc0ab7cf740 1 -- 172.21.15.52:0/2130519988 >> 172.21.15.52:0/2130519988 conn(0x557a13027300 msgr2=0x557a133d60f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:34:55.991 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:34:55.982+0000 7fc0ab7cf740 1 -- 172.21.15.52:0/2130519988 shutdown_connections 2024-05-27T07:34:55.991 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:34:55.982+0000 7fc0ab7cf740 1 -- 172.21.15.52:0/2130519988 wait complete. 2024-05-27T07:34:55.991 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:34:55.982+0000 7fc0ab7cf740 20 client.17037 trim_cache size 0 max 16384 2024-05-27T07:34:55.999 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:34:55.998+0000 7fc0ab7cf740 1 Processor -- start 2024-05-27T07:34:55.999 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:34:55.998+0000 7fc0ab7cf740 1 -- start start 2024-05-27T07:34:55.999 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:34:55.998+0000 7fc0ab7cf740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x557a1300a2f0 0x557a133de8b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:34:55.999 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:34:55.998+0000 7fc0ab7cf740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x557a130124d0 0x557a133d6080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:34:55.999 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:34:55.998+0000 7fc0ab7cf740 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x557a130331b0 0x557a133d65c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:34:55.999 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:34:55.998+0000 7fc0ab7cf740 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x7fc0a01b17e0 con 0x557a130331b0 2024-05-27T07:34:55.999 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:34:55.998+0000 7fc0ab7cf740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7fc0a01afc40 con 0x557a130124d0 2024-05-27T07:34:55.999 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:34:55.998+0000 7fc0ab7cf740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7fc0a01b10e0 con 0x557a1300a2f0 2024-05-27T07:34:55.999 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:34:55.998+0000 7fc09ed9d640 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x557a130331b0 0x557a133d65c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:34:56.000 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:34:55.998+0000 7fc09ed9d640 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x557a130331b0 0x557a133d65c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.52:60522/0 (socket says 172.21.15.52:60522) 2024-05-27T07:34:56.000 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:34:55.998+0000 7fc09ed9d640 1 -- 172.21.15.52:0/3573498228 learned_addr learned my addr 172.21.15.52:0/3573498228 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-05-27T07:34:56.000 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:34:55.998+0000 7fc097fff640 1 --2- 172.21.15.52:0/3573498228 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x557a130124d0 0x557a133d6080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:34:56.000 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:34:55.998+0000 7fc09e59c640 1 --2- 172.21.15.52:0/3573498228 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x557a1300a2f0 0x557a133de8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:34:56.000 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:34:55.998+0000 7fc09ed9d640 1 -- 172.21.15.52:0/3573498228 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x557a1300a2f0 msgr2=0x557a133de8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:34:56.000 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:34:55.999+0000 7fc09ed9d640 1 --2- 172.21.15.52:0/3573498228 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x557a1300a2f0 0x557a133de8b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:34:56.000 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:34:55.999+0000 7fc09ed9d640 1 -- 172.21.15.52:0/3573498228 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x557a130124d0 msgr2=0x557a133d6080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:34:56.000 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:34:55.999+0000 7fc09ed9d640 1 --2- 172.21.15.52:0/3573498228 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x557a130124d0 0x557a133d6080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:34:56.000 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:34:55.999+0000 7fc09ed9d640 1 -- 172.21.15.52:0/3573498228 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x557a133ddc10 con 0x557a130331b0 2024-05-27T07:34:56.000 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:34:55.999+0000 7fc09e59c640 1 --2- 172.21.15.52:0/3573498228 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x557a1300a2f0 0x557a133de8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-27T07:34:56.000 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:34:55.999+0000 7fc09ed9d640 1 --2- 172.21.15.52:0/3573498228 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x557a130331b0 0x557a133d65c0 secure :-1 s=READY pgs=723 cs=0 l=1 rev1=1 crypto rx=0x7fc09800fc30 tx=0x7fc09800ffd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=74da15f660493cdc server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:34:56.000 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:34:55.999+0000 7fc087fff640 1 -- 172.21.15.52:0/3573498228 <== mon.0 v2:172.21.15.52:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc09802a070 con 0x557a130331b0 2024-05-27T07:34:56.000 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:34:55.999+0000 7fc087fff640 1 -- 172.21.15.52:0/3573498228 <== mon.0 v2:172.21.15.52:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fc098004c40 con 0x557a130331b0 2024-05-27T07:34:56.000 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:34:55.999+0000 7fc087fff640 1 -- 172.21.15.52:0/3573498228 <== mon.0 v2:172.21.15.52:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc098011940 con 0x557a130331b0 2024-05-27T07:34:56.001 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:34:56.000+0000 7fc0ab7cf740 1 -- 172.21.15.52:0/3573498228 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x557a130331b0 msgr2=0x557a133d65c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:34:56.001 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:34:56.000+0000 7fc0ab7cf740 1 --2- 172.21.15.52:0/3573498228 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x557a130331b0 0x557a133d65c0 secure :-1 s=READY pgs=723 cs=0 l=1 rev1=1 crypto rx=0x7fc09800fc30 tx=0x7fc09800ffd0 comp rx=0 tx=0).stop 2024-05-27T07:34:56.001 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:34:56.000+0000 7fc0ab7cf740 1 -- 172.21.15.52:0/3573498228 shutdown_connections 2024-05-27T07:34:56.001 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:34:56.000+0000 7fc0ab7cf740 1 --2- 172.21.15.52:0/3573498228 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x557a130331b0 0x557a133d65c0 unknown :-1 s=CLOSED pgs=723 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:34:56.001 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:34:56.000+0000 7fc0ab7cf740 1 --2- 172.21.15.52:0/3573498228 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x557a130124d0 0x557a133d6080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:34:56.001 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:34:56.000+0000 7fc0ab7cf740 1 --2- 172.21.15.52:0/3573498228 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x557a1300a2f0 0x557a133de8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:34:56.001 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:34:56.000+0000 7fc0ab7cf740 1 -- 172.21.15.52:0/3573498228 >> 172.21.15.52:0/3573498228 conn(0x557a13027300 msgr2=0x557a133de1a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:34:56.001 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:34:56.000+0000 7fc0ab7cf740 1 -- 172.21.15.52:0/3573498228 shutdown_connections 2024-05-27T07:34:56.001 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:34:56.000+0000 7fc0ab7cf740 1 -- 172.21.15.52:0/3573498228 wait complete. 2024-05-27T07:34:56.001 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:34:56.000+0000 7fc0ab7cf740 1 -- start start 2024-05-27T07:34:56.001 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:34:56.000+0000 7fc0ab7cf740 1 Processor -- start 2024-05-27T07:34:56.001 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:34:56.000+0000 7fc0ab7cf740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x557a1300a2f0 0x557a133e3c00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:34:56.002 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:34:56.000+0000 7fc0ab7cf740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x557a130124d0 0x557a133e4140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:34:56.002 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:34:56.000+0000 7fc0ab7cf740 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x557a130331b0 0x557a13457830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:34:56.002 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:34:56.000+0000 7fc0ab7cf740 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x7fc0a01b0a10 con 0x557a130331b0 2024-05-27T07:34:56.002 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:34:56.001+0000 7fc0ab7cf740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7fc0a01b0310 con 0x557a1300a2f0 2024-05-27T07:34:56.002 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:34:56.001+0000 7fc0ab7cf740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x557a13457d70 con 0x557a130124d0 2024-05-27T07:34:56.002 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:34:56.001+0000 7fc0ab7cf740 10 client.0 _ll_register_callbacks cb 0x7fc0880046c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T07:34:56.002 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:34:56.001+0000 7fc09ed9d640 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x557a130331b0 0x557a13457830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:34:56.002 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:34:56.001+0000 7fc09ed9d640 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x557a130331b0 0x557a13457830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.52:60534/0 (socket says 172.21.15.52:60534) 2024-05-27T07:34:56.002 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:34:56.001+0000 7fc09ed9d640 1 -- 172.21.15.52:0/1330713506 learned_addr learned my addr 172.21.15.52:0/1330713506 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-05-27T07:34:56.003 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:34:56.001+0000 7fc09e59c640 1 --2- 172.21.15.52:0/1330713506 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x557a1300a2f0 0x557a133e3c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:34:56.003 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:34:56.001+0000 7fc097fff640 1 --2- 172.21.15.52:0/1330713506 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x557a130124d0 0x557a133e4140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:34:56.003 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:34:56.001+0000 7fc09ed9d640 1 -- 172.21.15.52:0/1330713506 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x557a130124d0 msgr2=0x557a133e4140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:34:56.003 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:34:56.001+0000 7fc09ed9d640 1 --2- 172.21.15.52:0/1330713506 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x557a130124d0 0x557a133e4140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:34:56.003 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:34:56.001+0000 7fc09ed9d640 1 -- 172.21.15.52:0/1330713506 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x557a1300a2f0 msgr2=0x557a133e3c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:34:56.003 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:34:56.001+0000 7fc09ed9d640 1 --2- 172.21.15.52:0/1330713506 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x557a1300a2f0 0x557a133e3c00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:34:56.003 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:34:56.001+0000 7fc09ed9d640 1 -- 172.21.15.52:0/1330713506 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({osdmap=0}) -- 0x557a13457ef0 con 0x557a130331b0 2024-05-27T07:34:56.003 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:34:56.001+0000 7fc09ed9d640 1 --2- 172.21.15.52:0/1330713506 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x557a130331b0 0x557a13457830 secure :-1 s=READY pgs=724 cs=0 l=1 rev1=1 crypto rx=0x7fc098009de0 tx=0x7fc098011800 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5a832b46c9113af7 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:34:56.003 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:34:56.001+0000 7fc085ffb640 10 client.0 ms_handle_connect on v2:172.21.15.52:3300/0 2024-05-27T07:34:56.003 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:34:56.001+0000 7fc085ffb640 1 -- 172.21.15.52:0/1330713506 <== mon.0 v2:172.21.15.52:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc09802a050 con 0x557a130331b0 2024-05-27T07:34:56.003 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:34:56.001+0000 7fc085ffb640 1 -- 172.21.15.52:0/1330713506 <== mon.0 v2:172.21.15.52:3300/0 2 ==== osd_map(122..122 src has 1..122) ==== 9423+0+0 (secure 0 0 0) 0x7fc098011c00 con 0x557a130331b0 2024-05-27T07:34:56.003 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:34:56.001+0000 7fc0ab7cf740 1 -- 172.21.15.52:0/1330713506 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fc098004c40 con 0x557a130331b0 2024-05-27T07:34:56.004 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:34:56.001+0000 7fc085ffb640 1 -- 172.21.15.52:0/1330713506 <== mon.0 v2:172.21.15.52:3300/0 3 ==== mon_get_version_reply(handle=1 version=107) ==== 24+0+0 (secure 0 0 0) 0x7fc09803b050 con 0x557a130331b0 2024-05-27T07:34:56.004 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:34:56.001+0000 7fc0ab7cf740 10 client.17043 fetch_fsmap learned FSMap version 107 2024-05-27T07:34:56.004 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:34:56.002+0000 7fc0ab7cf740 1 -- 172.21.15.52:0/1330713506 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({config=0+,fsmap=107,monmap=4+}) -- 0x557a13459090 con 0x557a130331b0 2024-05-27T07:34:56.004 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:34:56.002+0000 7fc085ffb640 1 -- 172.21.15.52:0/1330713506 <== mon.0 v2:172.21.15.52:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fc098004c40 con 0x557a130331b0 2024-05-27T07:34:56.004 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:34:56.002+0000 7fc085ffb640 1 -- 172.21.15.52:0/1330713506 <== mon.0 v2:172.21.15.52:3300/0 5 ==== fsmap(e 107) ==== 4199+0+0 (secure 0 0 0) 0x7fc098036030 con 0x557a130331b0 2024-05-27T07:34:56.004 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:34:56.002+0000 7fc0ab7cf740 10 client.17043 fetch_fsmap finished waiting for FSMap version 107 2024-05-27T07:34:56.004 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:34:56.002+0000 7fc0ab7cf740 10 client.17043 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-05-27T07:34:56.004 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:34:56.002+0000 7fc0ab7cf740 20 client.17043 populate_metadata read hostname 'smithi052' 2024-05-27T07:34:56.004 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:34:56.002+0000 7fc0ab7cf740 1 --2- 172.21.15.52:0/1330713506 >> [v2:172.21.15.105:6838/500348151,v1:172.21.15.105:6839/500348151] conn(0x557a12fc71c0 0x557a1345b6d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:34:56.004 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:34:56.002+0000 7fc0ab7cf740 4 client.17043 mds_command: new command op to 24527 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-05-27T07:34:56.004 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:34:56.002+0000 7fc0ab7cf740 1 -- 172.21.15.52:0/1330713506 --> [v2:172.21.15.105:6838/500348151,v1:172.21.15.105:6839/500348151] -- command(tid 0: {"prefix": "flush journal"}) -- 0x557a13342360 con 0x557a12fc71c0 2024-05-27T07:34:56.004 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:34:56.002+0000 7fc09e59c640 1 --2- 172.21.15.52:0/1330713506 >> [v2:172.21.15.105:6838/500348151,v1:172.21.15.105:6839/500348151] conn(0x557a12fc71c0 0x557a1345b6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:34:56.004 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:34:56.002+0000 7fc085ffb640 10 client.17043 ms_handle_connect on v2:172.21.15.105:6838/500348151 2024-05-27T07:34:56.005 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:34:56.002+0000 7fc09e59c640 1 --2- 172.21.15.52:0/1330713506 >> [v2:172.21.15.105:6838/500348151,v1:172.21.15.105:6839/500348151] conn(0x557a12fc71c0 0x557a1345b6d0 crc :-1 s=READY pgs=160 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=d39f00cf89659fa server_cookie=e24ce3e242da3f97 in_seq=0 out_seq=0 2024-05-27T07:34:56.357 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:34:56 smithi173 ceph-mon[37000]: osdmap e122: 12 total, 12 up, 12 in 2024-05-27T07:34:56.357 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:34:56 smithi173 ceph-mon[37000]: pgmap v303: 129 pgs: 129 active+clean; 658 MiB data, 5.5 GiB used, 1.0 TiB / 1.0 TiB avail; 38 KiB/s rd, 1.5 MiB/s wr, 441 op/s 2024-05-27T07:34:56.445 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:34:56 smithi105 ceph-mon[35977]: osdmap e122: 12 total, 12 up, 12 in 2024-05-27T07:34:56.445 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:34:56 smithi105 ceph-mon[35977]: pgmap v303: 129 pgs: 129 active+clean; 658 MiB data, 5.5 GiB used, 1.0 TiB / 1.0 TiB avail; 38 KiB/s rd, 1.5 MiB/s wr, 441 op/s 2024-05-27T07:34:56.484 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:34:56 smithi052 ceph-mon[31955]: osdmap e122: 12 total, 12 up, 12 in 2024-05-27T07:34:56.484 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:34:56 smithi052 ceph-mon[31955]: pgmap v303: 129 pgs: 129 active+clean; 658 MiB data, 5.5 GiB used, 1.0 TiB / 1.0 TiB avail; 38 KiB/s rd, 1.5 MiB/s wr, 441 op/s 2024-05-27T07:34:56.948 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:34:56.952+0000 7fc085ffb640 1 -- 172.21.15.52:0/1330713506 <== mds.0 v2:172.21.15.105:6838/500348151 1 ==== command_reply(tid 0: 0 ) ==== 8+0+25 (crc 0 0 0) 0x557a13342360 con 0x557a12fc71c0 2024-05-27T07:34:56.948 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:34:56.952+0000 7fc085ffb640 10 client.17043 handle_command_reply: tid=0 2024-05-27T07:34:56.948 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:34:56.952+0000 7fc0ab7cf740 1 client.17043 shutdown 2024-05-27T07:34:56.948 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:34:56.952+0000 7fc0ab7cf740 1 -- 172.21.15.52:0/1330713506 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x557a130331b0 msgr2=0x557a13457830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:34:56.949 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:34:56.952+0000 7fc0ab7cf740 1 --2- 172.21.15.52:0/1330713506 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x557a130331b0 0x557a13457830 secure :-1 s=READY pgs=724 cs=0 l=1 rev1=1 crypto rx=0x7fc098009de0 tx=0x7fc098011800 comp rx=0 tx=0).stop 2024-05-27T07:34:56.949 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:34:56.952+0000 7fc0ab7cf740 1 -- 172.21.15.52:0/1330713506 shutdown_connections 2024-05-27T07:34:56.949 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:34:56.952+0000 7fc0ab7cf740 1 --2- 172.21.15.52:0/1330713506 >> [v2:172.21.15.105:6838/500348151,v1:172.21.15.105:6839/500348151] conn(0x557a12fc71c0 0x557a1345b6d0 crc :-1 s=READY pgs=160 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:34:56.949 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:34:56.952+0000 7fc0ab7cf740 1 --2- 172.21.15.52:0/1330713506 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x557a130331b0 0x557a13457830 unknown :-1 s=CLOSED pgs=724 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:34:56.949 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:34:56.952+0000 7fc0ab7cf740 1 --2- 172.21.15.52:0/1330713506 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x557a130124d0 0x557a133e4140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:34:56.949 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:34:56.952+0000 7fc085ffb640 0 client.17043 ms_handle_reset on v2:172.21.15.105:6838/500348151 2024-05-27T07:34:56.949 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:34:56.952+0000 7fc0ab7cf740 1 --2- 172.21.15.52:0/1330713506 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x557a1300a2f0 0x557a133e3c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:34:56.949 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:34:56.952+0000 7fc0ab7cf740 1 -- 172.21.15.52:0/1330713506 >> 172.21.15.52:0/1330713506 conn(0x557a13027300 msgr2=0x557a133dcde0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:34:56.949 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:34:56.952+0000 7fc0ab7cf740 1 -- 172.21.15.52:0/1330713506 shutdown_connections 2024-05-27T07:34:56.949 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:34:56.952+0000 7fc0ab7cf740 1 -- 172.21.15.52:0/1330713506 wait complete. 2024-05-27T07:34:56.949 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:34:56.953+0000 7fc0ab7cf740 20 client.17043 trim_cache size 0 max 16384 2024-05-27T07:34:56.950 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stdout:{ 2024-05-27T07:34:56.950 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stdout: "return_code": 0 2024-05-27T07:34:56.950 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stdout:} 2024-05-27T07:34:56.951 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:34:56.953+0000 7fc0a4842640 1 -- 172.21.15.52:0/2661018427 >> [v2:172.21.15.52:6800/1822318384,v1:172.21.15.52:6801/1822318384] conn(0x7fc08c0b2950 msgr2=0x7fc08c0b4e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:34:56.951 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:34:56.953+0000 7fc0a4842640 1 --2- 172.21.15.52:0/2661018427 >> [v2:172.21.15.52:6800/1822318384,v1:172.21.15.52:6801/1822318384] conn(0x7fc08c0b2950 0x7fc08c0b4e60 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7fc090006010 tx=0x7fc090005a20 comp rx=0 tx=0).stop 2024-05-27T07:34:56.951 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:34:56.953+0000 7fc0a4842640 1 -- 172.21.15.52:0/2661018427 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fc0a0109c70 msgr2=0x7fc0a01a0ad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:34:56.951 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:34:56.953+0000 7fc0a4842640 1 --2- 172.21.15.52:0/2661018427 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fc0a0109c70 0x7fc0a01a0ad0 secure :-1 s=READY pgs=721 cs=0 l=1 rev1=1 crypto rx=0x7fc08800e9f0 tx=0x7fc08800eeb0 comp rx=0 tx=0).stop 2024-05-27T07:34:56.951 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:34:56.953+0000 7fc0a4842640 1 -- 172.21.15.52:0/2661018427 shutdown_connections 2024-05-27T07:34:56.951 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:34:56.953+0000 7fc0a4842640 1 --2- 172.21.15.52:0/2661018427 >> [v2:172.21.15.52:6800/1822318384,v1:172.21.15.52:6801/1822318384] conn(0x7fc08c0b2950 0x7fc08c0b4e60 unknown :-1 s=CLOSED pgs=177 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:34:56.951 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:34:56.953+0000 7fc0a4842640 1 --2- 172.21.15.52:0/2661018427 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fc0a010a110 0x7fc0a019a710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:34:56.951 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:34:56.953+0000 7fc0a4842640 1 --2- 172.21.15.52:0/2661018427 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fc0a0109c70 0x7fc0a01a0ad0 unknown :-1 s=CLOSED pgs=721 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:34:56.951 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:34:56.953+0000 7fc0a4842640 1 --2- 172.21.15.52:0/2661018427 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fc0a01043c0 0x7fc0a01a0590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:34:56.951 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:34:56.953+0000 7fc0a4842640 1 -- 172.21.15.52:0/2661018427 >> 172.21.15.52:0/2661018427 conn(0x7fc0a00ff350 msgr2=0x7fc0a00ff760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:34:56.951 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:34:56.953+0000 7fc0a4842640 1 -- 172.21.15.52:0/2661018427 shutdown_connections 2024-05-27T07:34:56.951 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:34:56.953+0000 7fc0a4842640 1 -- 172.21.15.52:0/2661018427 wait complete. 2024-05-27T07:34:58.484 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:34:58 smithi052 ceph-mon[31955]: pgmap v304: 129 pgs: 129 active+clean; 668 MiB data, 5.6 GiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 2.1 MiB/s wr, 454 op/s 2024-05-27T07:34:58.607 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:34:58 smithi173 ceph-mon[37000]: pgmap v304: 129 pgs: 129 active+clean; 668 MiB data, 5.6 GiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 2.1 MiB/s wr, 454 op/s 2024-05-27T07:34:58.695 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:34:58 smithi105 ceph-mon[35977]: pgmap v304: 129 pgs: 129 active+clean; 668 MiB data, 5.6 GiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 2.1 MiB/s wr, 454 op/s 2024-05-27T07:35:00.484 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:35:00 smithi052 ceph-mon[31955]: pgmap v305: 129 pgs: 129 active+clean; 660 MiB data, 5.6 GiB used, 1.0 TiB / 1.0 TiB avail; 34 KiB/s rd, 1.8 MiB/s wr, 528 op/s 2024-05-27T07:35:00.607 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:35:00 smithi173 ceph-mon[37000]: pgmap v305: 129 pgs: 129 active+clean; 660 MiB data, 5.6 GiB used, 1.0 TiB / 1.0 TiB avail; 34 KiB/s rd, 1.8 MiB/s wr, 528 op/s 2024-05-27T07:35:00.695 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:35:00 smithi105 ceph-mon[35977]: pgmap v305: 129 pgs: 129 active+clean; 660 MiB data, 5.6 GiB used, 1.0 TiB / 1.0 TiB avail; 34 KiB/s rd, 1.8 MiB/s wr, 528 op/s 2024-05-27T07:35:02.484 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:35:02 smithi052 ceph-mon[31955]: pgmap v306: 129 pgs: 129 active+clean; 649 MiB data, 5.6 GiB used, 1.0 TiB / 1.0 TiB avail; 33 KiB/s rd, 1.7 MiB/s wr, 519 op/s 2024-05-27T07:35:02.607 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:35:02 smithi173 ceph-mon[37000]: pgmap v306: 129 pgs: 129 active+clean; 649 MiB data, 5.6 GiB used, 1.0 TiB / 1.0 TiB avail; 33 KiB/s rd, 1.7 MiB/s wr, 519 op/s 2024-05-27T07:35:02.695 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:35:02 smithi105 ceph-mon[35977]: pgmap v306: 129 pgs: 129 active+clean; 649 MiB data, 5.6 GiB used, 1.0 TiB / 1.0 TiB avail; 33 KiB/s rd, 1.7 MiB/s wr, 519 op/s 2024-05-27T07:35:03.198 DEBUG:teuthology.orchestra.run.smithi052:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:38f9d900c83c63799fbdbe61acc9a11b0d3554a6 shell --fsid 73df2636-1bf8-11ef-bc9b-c7b262605968 -- ceph tell mds.1:0 scrub status 2024-05-27T07:35:03.448 INFO:teuthology.orchestra.run.smithi052.stderr:Inferring config /var/lib/ceph/73df2636-1bf8-11ef-bc9b-c7b262605968/mon.a/config 2024-05-27T07:35:04.484 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:35:04 smithi052 ceph-mon[31955]: pgmap v307: 129 pgs: 129 active+clean; 652 MiB data, 5.7 GiB used, 1.0 TiB / 1.0 TiB avail; 34 KiB/s rd, 2.1 MiB/s wr, 567 op/s 2024-05-27T07:35:04.607 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:35:04 smithi173 ceph-mon[37000]: pgmap v307: 129 pgs: 129 active+clean; 652 MiB data, 5.7 GiB used, 1.0 TiB / 1.0 TiB avail; 34 KiB/s rd, 2.1 MiB/s wr, 567 op/s 2024-05-27T07:35:04.695 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:35:04 smithi105 ceph-mon[35977]: pgmap v307: 129 pgs: 129 active+clean; 652 MiB data, 5.7 GiB used, 1.0 TiB / 1.0 TiB avail; 34 KiB/s rd, 2.1 MiB/s wr, 567 op/s 2024-05-27T07:35:05.687 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:05.691+0000 7f638ebfe700 1 -- 172.21.15.52:0/3476617554 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f63800a5f00 msgr2=0x7f63800a6310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:35:05.687 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:05.691+0000 7f638ebfe700 1 --2- 172.21.15.52:0/3476617554 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f63800a5f00 0x7f63800a6310 secure :-1 s=READY pgs=725 cs=0 l=1 rev1=1 crypto rx=0x7f6388069810 tx=0x7f638806a480 comp rx=0 tx=0).stop 2024-05-27T07:35:05.687 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:05.691+0000 7f638ebfe700 1 -- 172.21.15.52:0/3476617554 shutdown_connections 2024-05-27T07:35:05.687 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:05.691+0000 7f638ebfe700 1 --2- 172.21.15.52:0/3476617554 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f63800a6850 0x7f63800b2a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:35:05.687 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:05.691+0000 7f638ebfe700 1 --2- 172.21.15.52:0/3476617554 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f63800abc50 0x7f63800ac060 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:35:05.687 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:05.691+0000 7f638ebfe700 1 --2- 172.21.15.52:0/3476617554 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f63800a5f00 0x7f63800a6310 unknown :-1 s=CLOSED pgs=725 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:35:05.687 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:05.691+0000 7f638ebfe700 1 -- 172.21.15.52:0/3476617554 >> 172.21.15.52:0/3476617554 conn(0x7f638001b3a0 msgr2=0x7f638001b7b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:35:05.687 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:05.692+0000 7f638ebfe700 1 -- 172.21.15.52:0/3476617554 shutdown_connections 2024-05-27T07:35:05.688 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:05.692+0000 7f638ebfe700 1 -- 172.21.15.52:0/3476617554 wait complete. 2024-05-27T07:35:05.688 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:05.693+0000 7f638ebfe700 1 Processor -- start 2024-05-27T07:35:05.688 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:05.693+0000 7f638ebfe700 1 -- start start 2024-05-27T07:35:05.688 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:05.693+0000 7f638ebfe700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f63800a5f00 0x7f63801429f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:35:05.689 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:05.693+0000 7f638ebfe700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f63800a6850 0x7f638013aa60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:35:05.689 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:05.693+0000 7f638ebfe700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f63800abc50 0x7f638013afa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:35:05.689 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:05.693+0000 7f638ebfe700 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x7f63800b5520 con 0x7f63800abc50 2024-05-27T07:35:05.689 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:05.693+0000 7f638ebfe700 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f63800b53a0 con 0x7f63800a5f00 2024-05-27T07:35:05.689 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:05.693+0000 7f638ebfe700 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f63800b56a0 con 0x7f63800a6850 2024-05-27T07:35:05.691 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:05.694+0000 7f638d3fb700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f63800a6850 0x7f638013aa60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:35:05.691 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:05.694+0000 7f638d3fb700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f63800a6850 0x7f638013aa60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.173:3300/0 says I am v2:172.21.15.52:60052/0 (socket says 172.21.15.52:60052) 2024-05-27T07:35:05.691 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:05.694+0000 7f638d3fb700 1 -- 172.21.15.52:0/2086955673 learned_addr learned my addr 172.21.15.52:0/2086955673 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-05-27T07:35:05.691 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:05.694+0000 7f638dbfc700 1 --2- 172.21.15.52:0/2086955673 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f63800a5f00 0x7f63801429f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:35:05.691 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:05.694+0000 7f638e3fd700 1 --2- 172.21.15.52:0/2086955673 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f63800abc50 0x7f638013afa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:35:05.691 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:05.694+0000 7f638d3fb700 1 -- 172.21.15.52:0/2086955673 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f63800a5f00 msgr2=0x7f63801429f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:35:05.692 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:05.694+0000 7f638d3fb700 1 --2- 172.21.15.52:0/2086955673 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f63800a5f00 0x7f63801429f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:35:05.692 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:05.694+0000 7f638d3fb700 1 -- 172.21.15.52:0/2086955673 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f63800abc50 msgr2=0x7f638013afa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:35:05.692 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:05.694+0000 7f638d3fb700 1 --2- 172.21.15.52:0/2086955673 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f63800abc50 0x7f638013afa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:35:05.692 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:05.694+0000 7f638d3fb700 1 -- 172.21.15.52:0/2086955673 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f638013b6a0 con 0x7f63800a6850 2024-05-27T07:35:05.692 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:05.695+0000 7f638d3fb700 1 --2- 172.21.15.52:0/2086955673 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f63800a6850 0x7f638013aa60 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7f6388069870 tx=0x7f638806c340 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:35:05.692 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:05.695+0000 7f637effd700 1 -- 172.21.15.52:0/2086955673 <== mon.1 v2:172.21.15.173:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f638807c040 con 0x7f63800a6850 2024-05-27T07:35:05.692 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:05.695+0000 7f638ebfe700 1 -- 172.21.15.52:0/2086955673 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6380159430 con 0x7f63800a6850 2024-05-27T07:35:05.692 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:05.695+0000 7f637effd700 1 -- 172.21.15.52:0/2086955673 <== mon.1 v2:172.21.15.173:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f6388087020 con 0x7f63800a6850 2024-05-27T07:35:05.692 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:05.696+0000 7f637effd700 1 -- 172.21.15.52:0/2086955673 <== mon.1 v2:172.21.15.173:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6388082700 con 0x7f63800a6850 2024-05-27T07:35:05.692 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:05.696+0000 7f638ebfe700 1 -- 172.21.15.52:0/2086955673 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f63801597d0 con 0x7f63800a6850 2024-05-27T07:35:05.695 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:05.700+0000 7f637effd700 1 -- 172.21.15.52:0/2086955673 <== mon.1 v2:172.21.15.173:3300/0 4 ==== mgrmap(e 26) ==== 149783+0+0 (secure 0 0 0) 0x7f6388069d80 con 0x7f63800a6850 2024-05-27T07:35:05.696 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:05.702+0000 7f639e875b80 1 Processor -- start 2024-05-27T07:35:05.697 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:05.702+0000 7f639e875b80 1 -- start start 2024-05-27T07:35:05.698 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:05.703+0000 7f639e875b80 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55b652f9de40 0x55b65311c010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:35:05.698 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:05.703+0000 7f639e875b80 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55b65311c550 0x55b65311c960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:35:05.698 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:05.703+0000 7f639e875b80 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55b653126de0 0x55b653129200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:35:05.699 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:05.703+0000 7f639e875b80 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x55b65304ad70 con 0x55b652f9de40 2024-05-27T07:35:05.699 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:05.703+0000 7f639e875b80 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x55b65303a910 con 0x55b653126de0 2024-05-27T07:35:05.699 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:05.703+0000 7f639e875b80 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x55b65303e950 con 0x55b65311c550 2024-05-27T07:35:05.699 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:05.703+0000 7f638dbfc700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55b65311c550 0x55b65311c960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:35:05.699 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:05.703+0000 7f638dbfc700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55b65311c550 0x55b65311c960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.173:3300/0 says I am v2:172.21.15.52:60064/0 (socket says 172.21.15.52:60064) 2024-05-27T07:35:05.699 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:05.703+0000 7f638dbfc700 1 -- 172.21.15.52:0/447493751 learned_addr learned my addr 172.21.15.52:0/447493751 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-05-27T07:35:05.699 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:05.703+0000 7f638d3fb700 1 --2- 172.21.15.52:0/447493751 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55b653126de0 0x55b653129200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:35:05.700 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:05.704+0000 7f638dbfc700 1 -- 172.21.15.52:0/447493751 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55b653126de0 msgr2=0x55b653129200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:35:05.700 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:05.704+0000 7f638dbfc700 1 --2- 172.21.15.52:0/447493751 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55b653126de0 0x55b653129200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:35:05.700 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:05.704+0000 7f638dbfc700 1 -- 172.21.15.52:0/447493751 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55b652f9de40 msgr2=0x55b65311c010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:35:05.700 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:05.704+0000 7f638dbfc700 1 --2- 172.21.15.52:0/447493751 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55b652f9de40 0x55b65311c010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:35:05.700 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:05.704+0000 7f638dbfc700 1 -- 172.21.15.52:0/447493751 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55b652fe3fb0 con 0x55b65311c550 2024-05-27T07:35:05.700 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:05.704+0000 7f637effd700 1 --2- 172.21.15.52:0/2086955673 >> [v2:172.21.15.52:6800/1822318384,v1:172.21.15.52:6801/1822318384] conn(0x7f63740b2c00 0x7f63740b5110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:35:05.701 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:05.704+0000 7f637effd700 1 -- 172.21.15.52:0/2086955673 <== mon.1 v2:172.21.15.173:3300/0 5 ==== osd_map(122..122 src has 1..122) ==== 9423+0+0 (secure 0 0 0) 0x7f638814a1d0 con 0x7f63800a6850 2024-05-27T07:35:05.701 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:05.705+0000 7f638e3fd700 1 --2- 172.21.15.52:0/2086955673 >> [v2:172.21.15.52:6800/1822318384,v1:172.21.15.52:6801/1822318384] conn(0x7f63740b2c00 0x7f63740b5110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:35:05.701 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:05.705+0000 7f638dbfc700 1 --2- 172.21.15.52:0/447493751 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55b65311c550 0x55b65311c960 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7f638400bcf0 tx=0x7f63840103d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=81d0917ac0be0d9c server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:35:05.701 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:05.706+0000 7f6373fff700 1 -- 172.21.15.52:0/447493751 <== mon.1 v2:172.21.15.173:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6384010c50 con 0x55b65311c550 2024-05-27T07:35:05.701 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:05.706+0000 7f6373fff700 1 -- 172.21.15.52:0/447493751 <== mon.1 v2:172.21.15.173:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f63840044e0 con 0x55b65311c550 2024-05-27T07:35:05.701 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:05.706+0000 7f6373fff700 1 -- 172.21.15.52:0/447493751 <== mon.1 v2:172.21.15.173:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6384005280 con 0x55b65311c550 2024-05-27T07:35:05.703 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:05.707+0000 7f639e875b80 1 -- 172.21.15.52:0/447493751 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55b65311c550 msgr2=0x55b65311c960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:35:05.703 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:05.707+0000 7f639e875b80 1 --2- 172.21.15.52:0/447493751 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55b65311c550 0x55b65311c960 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7f638400bcf0 tx=0x7f63840103d0 comp rx=0 tx=0).stop 2024-05-27T07:35:05.703 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:05.708+0000 7f638e3fd700 1 --2- 172.21.15.52:0/2086955673 >> [v2:172.21.15.52:6800/1822318384,v1:172.21.15.52:6801/1822318384] conn(0x7f63740b2c00 0x7f63740b5110 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7f6378005950 tx=0x7f63780058e0 comp rx=0 tx=0).ready entity=mgr.15333 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:35:05.703 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:05.708+0000 7f639e875b80 1 -- 172.21.15.52:0/447493751 shutdown_connections 2024-05-27T07:35:05.703 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:05.708+0000 7f639e875b80 1 --2- 172.21.15.52:0/447493751 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55b653126de0 0x55b653129200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:35:05.704 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:05.708+0000 7f639e875b80 1 --2- 172.21.15.52:0/447493751 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55b652f9de40 0x55b65311c010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:35:05.704 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:05.708+0000 7f639e875b80 1 --2- 172.21.15.52:0/447493751 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55b65311c550 0x55b65311c960 unknown :-1 s=CLOSED pgs=194 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:35:05.704 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:05.708+0000 7f639e875b80 1 -- 172.21.15.52:0/447493751 >> 172.21.15.52:0/447493751 conn(0x55b652fa0770 msgr2=0x55b653118040 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:35:05.704 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:05.708+0000 7f639e875b80 1 -- 172.21.15.52:0/447493751 shutdown_connections 2024-05-27T07:35:05.704 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:05.708+0000 7f639e875b80 1 -- 172.21.15.52:0/447493751 wait complete. 2024-05-27T07:35:05.704 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:05.709+0000 7f639e875b80 1 -- start start 2024-05-27T07:35:05.704 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:05.709+0000 7f639e875b80 1 Processor -- start 2024-05-27T07:35:05.705 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:05.710+0000 7f639e875b80 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55b652f9de40 0x55b653123d60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:35:05.707 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:05.710+0000 7f639e875b80 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55b65311c550 0x55b6531242a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:35:05.707 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:05.711+0000 7f639e875b80 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55b653126de0 0x55b6531247e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:35:05.707 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:05.711+0000 7f639e875b80 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x55b652f40fa0 con 0x55b65311c550 2024-05-27T07:35:05.707 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:05.711+0000 7f639e875b80 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x55b652fa2da0 con 0x55b652f9de40 2024-05-27T07:35:05.707 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:05.711+0000 7f639e875b80 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x55b653129d00 con 0x55b653126de0 2024-05-27T07:35:05.707 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:05.711+0000 7f638e3fd700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55b653126de0 0x55b6531247e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:35:05.707 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:05.711+0000 7f638e3fd700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55b653126de0 0x55b6531247e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.173:3300/0 says I am v2:172.21.15.52:60076/0 (socket says 172.21.15.52:60076) 2024-05-27T07:35:05.707 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:05.711+0000 7f638e3fd700 1 -- 172.21.15.52:0/271855036 learned_addr learned my addr 172.21.15.52:0/271855036 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-05-27T07:35:05.707 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:05.711+0000 7f638dbfc700 1 --2- 172.21.15.52:0/271855036 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55b652f9de40 0x55b653123d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:35:05.708 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:05.711+0000 7f638e3fd700 1 -- 172.21.15.52:0/271855036 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55b652f9de40 msgr2=0x55b653123d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:35:05.708 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:05.711+0000 7f638e3fd700 1 --2- 172.21.15.52:0/271855036 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55b652f9de40 0x55b653123d60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:35:05.708 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:05.711+0000 7f638e3fd700 1 -- 172.21.15.52:0/271855036 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55b65311c550 msgr2=0x55b6531242a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:35:05.708 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:05.711+0000 7f638e3fd700 1 --2- 172.21.15.52:0/271855036 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55b65311c550 0x55b6531242a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:35:05.708 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:05.711+0000 7f639e875b80 10 client.0 _ll_register_callbacks cb 0x55b652fd15b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T07:35:05.708 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:05.711+0000 7f638e3fd700 1 -- 172.21.15.52:0/271855036 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55b652f8a8d0 con 0x55b653126de0 2024-05-27T07:35:05.708 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:05.711+0000 7f639e875b80 1 -- 172.21.15.52:0/271855036 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f63840044e0 con 0x55b653126de0 2024-05-27T07:35:05.708 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:05.712+0000 7f638e3fd700 1 --2- 172.21.15.52:0/271855036 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55b653126de0 0x55b6531247e0 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7f63780196b0 tx=0x7f63780047c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=43d30bf99a236eca server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:35:05.708 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:05.713+0000 7f63717fa700 10 client.26551 ms_handle_connect on v2:172.21.15.173:3300/0 2024-05-27T07:35:05.709 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:05.713+0000 7f63717fa700 1 -- 172.21.15.52:0/271855036 <== mon.1 v2:172.21.15.173:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6378027060 con 0x55b653126de0 2024-05-27T07:35:05.709 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:05.713+0000 7f63717fa700 1 -- 172.21.15.52:0/271855036 <== mon.1 v2:172.21.15.173:3300/0 2 ==== osd_map(122..122 src has 1..122) ==== 9423+0+0 (secure 0 0 0) 0x7f6378019ba0 con 0x55b653126de0 2024-05-27T07:35:05.709 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:05.713+0000 7f63717fa700 1 -- 172.21.15.52:0/271855036 <== mon.1 v2:172.21.15.173:3300/0 3 ==== mon_get_version_reply(handle=1 version=107) ==== 24+0+0 (secure 0 0 0) 0x7f6378019da0 con 0x55b653126de0 2024-05-27T07:35:05.709 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:05.715+0000 7f639e875b80 10 client.26551 fetch_fsmap learned FSMap version 107 2024-05-27T07:35:05.710 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:05.715+0000 7f639e875b80 1 -- 172.21.15.52:0/271855036 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({fsmap=107}) -- 0x55b652ff4760 con 0x55b653126de0 2024-05-27T07:35:05.711 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:05.716+0000 7f63717fa700 1 -- 172.21.15.52:0/271855036 <== mon.1 v2:172.21.15.173:3300/0 4 ==== fsmap(e 107) ==== 4199+0+0 (secure 0 0 0) 0x7f637802d4f0 con 0x55b653126de0 2024-05-27T07:35:05.711 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:05.716+0000 7f639e875b80 10 client.26551 fetch_fsmap finished waiting for FSMap version 107 2024-05-27T07:35:05.711 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:05.716+0000 7f639e875b80 10 client.26551 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-05-27T07:35:05.711 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:05.716+0000 7f639e875b80 20 client.26551 populate_metadata read hostname 'smithi052' 2024-05-27T07:35:05.711 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:05.716+0000 7f639e875b80 1 --2- 172.21.15.52:0/271855036 >> [v2:172.21.15.105:6838/500348151,v1:172.21.15.105:6839/500348151] conn(0x55b653136e50 0x55b653139300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:35:05.711 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:05.716+0000 7f639e875b80 4 client.26551 mds_command: new command op to 24527 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-27T07:35:05.711 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:05.716+0000 7f639e875b80 1 -- 172.21.15.52:0/271855036 --> [v2:172.21.15.105:6838/500348151,v1:172.21.15.105:6839/500348151] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55b65304ccc0 con 0x55b653136e50 2024-05-27T07:35:05.712 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:05.716+0000 7f638dbfc700 1 --2- 172.21.15.52:0/271855036 >> [v2:172.21.15.105:6838/500348151,v1:172.21.15.105:6839/500348151] conn(0x55b653136e50 0x55b653139300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:35:05.712 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:05.717+0000 7f638dbfc700 1 --2- 172.21.15.52:0/271855036 >> [v2:172.21.15.105:6838/500348151,v1:172.21.15.105:6839/500348151] conn(0x55b653136e50 0x55b653139300 crc :-1 s=READY pgs=161 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=943789e7c3a1f8a4 server_cookie=61c79fac6a6ad842 in_seq=0 out_seq=0 2024-05-27T07:35:05.712 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:05.717+0000 7f63717fa700 10 client.26551 ms_handle_connect on v2:172.21.15.105:6838/500348151 2024-05-27T07:35:05.716 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:05.721+0000 7f63717fa700 1 -- 172.21.15.52:0/271855036 <== mds.0 v2:172.21.15.105:6838/500348151 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24041 (crc 0 0 0) 0x55b65304ccc0 con 0x55b653136e50 2024-05-27T07:35:05.716 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:05.721+0000 7f63717fa700 10 client.26551 handle_command_reply: tid=0 2024-05-27T07:35:05.716 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:05.722+0000 7f639e875b80 1 client.26551 shutdown 2024-05-27T07:35:05.717 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:05.722+0000 7f639e875b80 1 -- 172.21.15.52:0/271855036 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55b653126de0 msgr2=0x55b6531247e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:35:05.717 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:05.722+0000 7f639e875b80 1 --2- 172.21.15.52:0/271855036 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55b653126de0 0x55b6531247e0 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7f63780196b0 tx=0x7f63780047c0 comp rx=0 tx=0).stop 2024-05-27T07:35:05.718 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:05.723+0000 7f639e875b80 1 -- 172.21.15.52:0/271855036 shutdown_connections 2024-05-27T07:35:05.718 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:05.723+0000 7f639e875b80 1 --2- 172.21.15.52:0/271855036 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55b653126de0 0x55b6531247e0 unknown :-1 s=CLOSED pgs=195 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:35:05.718 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:05.723+0000 7f639e875b80 1 --2- 172.21.15.52:0/271855036 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55b652f9de40 0x55b653123d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:35:05.718 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:05.723+0000 7f639e875b80 1 --2- 172.21.15.52:0/271855036 >> [v2:172.21.15.105:6838/500348151,v1:172.21.15.105:6839/500348151] conn(0x55b653136e50 0x55b653139300 crc :-1 s=READY pgs=161 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:35:05.718 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:05.723+0000 7f639e875b80 1 --2- 172.21.15.52:0/271855036 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55b65311c550 0x55b6531242a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:35:05.718 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:05.723+0000 7f639e875b80 1 -- 172.21.15.52:0/271855036 >> 172.21.15.52:0/271855036 conn(0x55b652fa0770 msgr2=0x55b6531165a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:35:05.719 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:05.724+0000 7f63717fa700 0 client.26551 ms_handle_reset on v2:172.21.15.105:6838/500348151 2024-05-27T07:35:05.719 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:05.724+0000 7f639e875b80 1 -- 172.21.15.52:0/271855036 shutdown_connections 2024-05-27T07:35:05.720 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:05.724+0000 7f639e875b80 1 -- 172.21.15.52:0/271855036 wait complete. 2024-05-27T07:35:05.720 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:05.725+0000 7f639e875b80 20 client.26551 trim_cache size 0 max 16384 2024-05-27T07:35:05.759 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:05.763+0000 7f639e875b80 1 Processor -- start 2024-05-27T07:35:05.759 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:05.763+0000 7f639e875b80 1 -- start start 2024-05-27T07:35:05.759 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:05.764+0000 7f639e875b80 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55b652f9de40 0x55b65311e610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:35:05.759 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:05.764+0000 7f639e875b80 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55b65311c550 0x55b6531260c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:35:05.759 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:05.764+0000 7f639e875b80 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55b653126de0 0x55b653132e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:35:05.759 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:05.764+0000 7f639e875b80 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x7f636c00f270 con 0x55b653126de0 2024-05-27T07:35:05.759 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:05.764+0000 7f639e875b80 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f636c00d6d0 con 0x55b652f9de40 2024-05-27T07:35:05.759 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:05.764+0000 7f639e875b80 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f636c00eb70 con 0x55b65311c550 2024-05-27T07:35:05.763 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:05.764+0000 7f638d3fb700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55b65311c550 0x55b6531260c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:35:05.763 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:05.764+0000 7f638d3fb700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55b65311c550 0x55b6531260c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.173:3300/0 says I am v2:172.21.15.52:60092/0 (socket says 172.21.15.52:60092) 2024-05-27T07:35:05.763 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:05.764+0000 7f638d3fb700 1 -- 172.21.15.52:0/2531096040 learned_addr learned my addr 172.21.15.52:0/2531096040 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-05-27T07:35:05.763 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:05.764+0000 7f638dbfc700 1 --2- 172.21.15.52:0/2531096040 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55b652f9de40 0x55b65311e610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:35:05.763 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:05.765+0000 7f638d3fb700 1 -- 172.21.15.52:0/2531096040 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55b652f9de40 msgr2=0x55b65311e610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:35:05.763 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:05.765+0000 7f638d3fb700 1 --2- 172.21.15.52:0/2531096040 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55b652f9de40 0x55b65311e610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:35:05.763 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:05.765+0000 7f638d3fb700 1 -- 172.21.15.52:0/2531096040 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55b653126de0 msgr2=0x55b653132e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:35:05.764 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:05.765+0000 7f638d3fb700 1 --2- 172.21.15.52:0/2531096040 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55b653126de0 0x55b653132e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:35:05.764 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:05.765+0000 7f638d3fb700 1 -- 172.21.15.52:0/2531096040 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55b653126600 con 0x55b65311c550 2024-05-27T07:35:05.764 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:05.765+0000 7f638d3fb700 1 --2- 172.21.15.52:0/2531096040 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55b65311c550 0x55b6531260c0 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7f6388052c50 tx=0x7f6388083000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=c0303e6b3de6218e server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:35:05.764 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:05.765+0000 7f63737fe700 1 -- 172.21.15.52:0/2531096040 <== mon.1 v2:172.21.15.173:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6388086030 con 0x55b65311c550 2024-05-27T07:35:05.764 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:05.765+0000 7f63737fe700 1 -- 172.21.15.52:0/2531096040 <== mon.1 v2:172.21.15.173:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f6388110c70 con 0x55b65311c550 2024-05-27T07:35:05.764 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:05.766+0000 7f63737fe700 1 -- 172.21.15.52:0/2531096040 <== mon.1 v2:172.21.15.173:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6388119f30 con 0x55b65311c550 2024-05-27T07:35:05.764 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:05.766+0000 7f639e875b80 1 -- 172.21.15.52:0/2531096040 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55b65311c550 msgr2=0x55b6531260c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:35:05.764 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:05.766+0000 7f639e875b80 1 --2- 172.21.15.52:0/2531096040 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55b65311c550 0x55b6531260c0 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7f6388052c50 tx=0x7f6388083000 comp rx=0 tx=0).stop 2024-05-27T07:35:05.765 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:05.770+0000 7f639e875b80 1 -- 172.21.15.52:0/2531096040 shutdown_connections 2024-05-27T07:35:05.765 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:05.770+0000 7f639e875b80 1 --2- 172.21.15.52:0/2531096040 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55b653126de0 0x55b653132e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:35:05.765 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:05.770+0000 7f639e875b80 1 --2- 172.21.15.52:0/2531096040 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55b652f9de40 0x55b65311e610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:35:05.765 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:05.770+0000 7f639e875b80 1 --2- 172.21.15.52:0/2531096040 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55b65311c550 0x55b6531260c0 unknown :-1 s=CLOSED pgs=196 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:35:05.766 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:05.770+0000 7f639e875b80 1 -- 172.21.15.52:0/2531096040 >> 172.21.15.52:0/2531096040 conn(0x55b652fa0770 msgr2=0x55b653120ce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:35:05.770 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:05.773+0000 7f639e875b80 1 -- 172.21.15.52:0/2531096040 shutdown_connections 2024-05-27T07:35:05.770 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:05.775+0000 7f639e875b80 1 -- 172.21.15.52:0/2531096040 wait complete. 2024-05-27T07:35:05.771 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:05.775+0000 7f639e875b80 1 -- start start 2024-05-27T07:35:05.771 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:05.776+0000 7f639e875b80 1 Processor -- start 2024-05-27T07:35:05.772 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:05.777+0000 7f639e875b80 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55b652f9de40 0x55b65312f220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:35:05.772 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:05.777+0000 7f639e875b80 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55b65311c550 0x55b65312f760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:35:05.772 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:05.777+0000 7f639e875b80 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55b653126de0 0x55b65313df10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:35:05.772 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:05.777+0000 7f639e875b80 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x7f636c00e4a0 con 0x55b652f9de40 2024-05-27T07:35:05.774 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:05.777+0000 7f639e875b80 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f636c00dda0 con 0x55b653126de0 2024-05-27T07:35:05.774 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:05.777+0000 7f639e875b80 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x55b653133370 con 0x55b65311c550 2024-05-27T07:35:05.774 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:05.777+0000 7f639e875b80 10 client.0 _ll_register_callbacks cb 0x55b6531399f0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T07:35:05.774 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:05.777+0000 7f638d3fb700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55b65311c550 0x55b65312f760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:35:05.774 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:05.777+0000 7f638e3fd700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55b653126de0 0x55b65313df10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:35:05.774 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:05.777+0000 7f638e3fd700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55b653126de0 0x55b65313df10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.52:59842/0 (socket says 172.21.15.52:59842) 2024-05-27T07:35:05.774 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:05.777+0000 7f638e3fd700 1 -- 172.21.15.52:0/2694489583 learned_addr learned my addr 172.21.15.52:0/2694489583 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-05-27T07:35:05.775 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:05.777+0000 7f638d3fb700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55b65311c550 0x55b65312f760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.173:3300/0 says I am v2:172.21.15.52:60106/0 (socket says 172.21.15.52:60106) 2024-05-27T07:35:05.775 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:05.778+0000 7f638d3fb700 1 -- 172.21.15.52:0/2694489583 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55b653126de0 msgr2=0x55b65313df10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:35:05.775 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:05.778+0000 7f638d3fb700 1 --2- 172.21.15.52:0/2694489583 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55b653126de0 0x55b65313df10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:35:05.775 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:05.778+0000 7f638d3fb700 1 -- 172.21.15.52:0/2694489583 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55b652f9de40 msgr2=0x55b65312f220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:35:05.775 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:05.778+0000 7f638d3fb700 1 --2- 172.21.15.52:0/2694489583 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55b652f9de40 0x55b65312f220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:35:05.775 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:05.778+0000 7f638d3fb700 1 -- 172.21.15.52:0/2694489583 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55b65313e6b0 con 0x55b65311c550 2024-05-27T07:35:05.775 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:05.779+0000 7f63717fa700 10 client.0 ms_handle_connect on v2:172.21.15.173:3300/0 2024-05-27T07:35:05.775 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:05.779+0000 7f638d3fb700 1 --2- 172.21.15.52:0/2694489583 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55b65311c550 0x55b65312f760 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x55b652fa2e50 tx=0x7f6388110b60 comp rx=0 tx=0).ready entity=mon.1 client_cookie=db90ffe32aded3f9 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:35:05.776 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:05.779+0000 7f63717fa700 1 -- 172.21.15.52:0/2694489583 <== mon.1 v2:172.21.15.173:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6388086070 con 0x55b65311c550 2024-05-27T07:35:05.776 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:05.779+0000 7f63717fa700 1 -- 172.21.15.52:0/2694489583 <== mon.1 v2:172.21.15.173:3300/0 2 ==== osd_map(122..122 src has 1..122) ==== 9423+0+0 (secure 0 0 0) 0x7f6388071c40 con 0x55b65311c550 2024-05-27T07:35:05.776 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:05.779+0000 7f639e875b80 1 -- 172.21.15.52:0/2694489583 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f6388110c70 con 0x55b65311c550 2024-05-27T07:35:05.776 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:05.780+0000 7f63717fa700 1 -- 172.21.15.52:0/2694489583 <== mon.1 v2:172.21.15.173:3300/0 3 ==== mon_get_version_reply(handle=1 version=107) ==== 24+0+0 (secure 0 0 0) 0x7f6388111860 con 0x55b65311c550 2024-05-27T07:35:05.776 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:05.780+0000 7f639e875b80 10 client.26557 fetch_fsmap learned FSMap version 107 2024-05-27T07:35:05.777 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:05.780+0000 7f639e875b80 1 -- 172.21.15.52:0/2694489583 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({config=0+,fsmap=107,monmap=4+}) -- 0x55b6531a1400 con 0x55b65311c550 2024-05-27T07:35:05.777 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:05.780+0000 7f63717fa700 1 -- 172.21.15.52:0/2694489583 <== mon.1 v2:172.21.15.173:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f6388110c70 con 0x55b65311c550 2024-05-27T07:35:05.777 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:05.780+0000 7f63717fa700 1 -- 172.21.15.52:0/2694489583 <== mon.1 v2:172.21.15.173:3300/0 5 ==== fsmap(e 107) ==== 4199+0+0 (secure 0 0 0) 0x7f6388072040 con 0x55b65311c550 2024-05-27T07:35:05.782 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:05.781+0000 7f639e875b80 10 client.26557 fetch_fsmap finished waiting for FSMap version 107 2024-05-27T07:35:05.782 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:05.781+0000 7f639e875b80 10 client.26557 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-05-27T07:35:05.782 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:05.781+0000 7f639e875b80 20 client.26557 populate_metadata read hostname 'smithi052' 2024-05-27T07:35:05.782 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:05.781+0000 7f639e875b80 1 --2- 172.21.15.52:0/2694489583 >> [v2:172.21.15.105:6838/500348151,v1:172.21.15.105:6839/500348151] conn(0x55b6531a1850 0x55b6531a3c70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:35:05.782 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:05.781+0000 7f639e875b80 4 client.26557 mds_command: new command op to 24527 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-05-27T07:35:05.782 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:05.781+0000 7f639e875b80 1 -- 172.21.15.52:0/2694489583 --> [v2:172.21.15.105:6838/500348151,v1:172.21.15.105:6839/500348151] -- command(tid 0: {"prefix": "scrub status"}) -- 0x55b6530464a0 con 0x55b6531a1850 2024-05-27T07:35:05.782 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:05.784+0000 7f638dbfc700 1 --2- 172.21.15.52:0/2694489583 >> [v2:172.21.15.105:6838/500348151,v1:172.21.15.105:6839/500348151] conn(0x55b6531a1850 0x55b6531a3c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:35:05.785 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:05.790+0000 7f63717fa700 10 client.26557 ms_handle_connect on v2:172.21.15.105:6838/500348151 2024-05-27T07:35:05.785 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:05.790+0000 7f638dbfc700 1 --2- 172.21.15.52:0/2694489583 >> [v2:172.21.15.105:6838/500348151,v1:172.21.15.105:6839/500348151] conn(0x55b6531a1850 0x55b6531a3c70 crc :-1 s=READY pgs=162 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=a43c26bdc138e0ee server_cookie=39035b8a32d11351 in_seq=0 out_seq=0 2024-05-27T07:35:05.798 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:05.800+0000 7f63717fa700 1 -- 172.21.15.52:0/2694489583 <== mds.0 v2:172.21.15.105:6838/500348151 1 ==== command_reply(tid 0: 0 ) ==== 8+0+63 (crc 0 0 0) 0x55b6530464a0 con 0x55b6531a1850 2024-05-27T07:35:05.798 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:05.800+0000 7f63717fa700 10 client.26557 handle_command_reply: tid=0 2024-05-27T07:35:05.798 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:05.800+0000 7f639e875b80 1 client.26557 shutdown 2024-05-27T07:35:05.798 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:05.802+0000 7f639e875b80 1 -- 172.21.15.52:0/2694489583 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55b65311c550 msgr2=0x55b65312f760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:35:05.798 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:05.802+0000 7f639e875b80 1 --2- 172.21.15.52:0/2694489583 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55b65311c550 0x55b65312f760 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x55b652fa2e50 tx=0x7f6388110b60 comp rx=0 tx=0).stop 2024-05-27T07:35:05.798 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:05.802+0000 7f639e875b80 1 -- 172.21.15.52:0/2694489583 shutdown_connections 2024-05-27T07:35:05.798 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:05.802+0000 7f639e875b80 1 --2- 172.21.15.52:0/2694489583 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55b653126de0 0x55b65313df10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:35:05.798 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:05.802+0000 7f639e875b80 1 --2- 172.21.15.52:0/2694489583 >> [v2:172.21.15.105:6838/500348151,v1:172.21.15.105:6839/500348151] conn(0x55b6531a1850 0x55b6531a3c70 crc :-1 s=READY pgs=162 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:35:05.799 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:05.802+0000 7f639e875b80 1 --2- 172.21.15.52:0/2694489583 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55b652f9de40 0x55b65312f220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:35:05.799 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:05.802+0000 7f639e875b80 1 --2- 172.21.15.52:0/2694489583 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55b65311c550 0x55b65312f760 unknown :-1 s=CLOSED pgs=197 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:35:05.799 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:05.802+0000 7f63717fa700 0 client.26557 ms_handle_reset on v2:172.21.15.105:6838/500348151 2024-05-27T07:35:05.799 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:05.802+0000 7f639e875b80 1 -- 172.21.15.52:0/2694489583 >> 172.21.15.52:0/2694489583 conn(0x55b652fa0770 msgr2=0x55b653121710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:35:05.799 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:05.802+0000 7f639e875b80 1 -- 172.21.15.52:0/2694489583 shutdown_connections 2024-05-27T07:35:05.799 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:05.802+0000 7f639e875b80 1 -- 172.21.15.52:0/2694489583 wait complete. 2024-05-27T07:35:05.799 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:05.803+0000 7f639e875b80 20 client.26557 trim_cache size 0 max 16384 2024-05-27T07:35:05.799 INFO:teuthology.orchestra.run.smithi052.stdout:{ 2024-05-27T07:35:05.800 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "no active scrubs running", 2024-05-27T07:35:05.800 INFO:teuthology.orchestra.run.smithi052.stdout: "scrubs": {} 2024-05-27T07:35:05.800 INFO:teuthology.orchestra.run.smithi052.stdout:} 2024-05-27T07:35:05.800 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:05.804+0000 7f638ebfe700 1 -- 172.21.15.52:0/2086955673 >> [v2:172.21.15.52:6800/1822318384,v1:172.21.15.52:6801/1822318384] conn(0x7f63740b2c00 msgr2=0x7f63740b5110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:35:05.801 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:05.804+0000 7f638ebfe700 1 --2- 172.21.15.52:0/2086955673 >> [v2:172.21.15.52:6800/1822318384,v1:172.21.15.52:6801/1822318384] conn(0x7f63740b2c00 0x7f63740b5110 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7f6378005950 tx=0x7f63780058e0 comp rx=0 tx=0).stop 2024-05-27T07:35:05.801 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:05.804+0000 7f638ebfe700 1 -- 172.21.15.52:0/2086955673 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f63800a6850 msgr2=0x7f638013aa60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:35:05.801 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:05.804+0000 7f638ebfe700 1 --2- 172.21.15.52:0/2086955673 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f63800a6850 0x7f638013aa60 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7f6388069870 tx=0x7f638806c340 comp rx=0 tx=0).stop 2024-05-27T07:35:05.801 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:05.804+0000 7f638ebfe700 1 -- 172.21.15.52:0/2086955673 shutdown_connections 2024-05-27T07:35:05.801 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:05.804+0000 7f638ebfe700 1 --2- 172.21.15.52:0/2086955673 >> [v2:172.21.15.52:6800/1822318384,v1:172.21.15.52:6801/1822318384] conn(0x7f63740b2c00 0x7f63740b5110 unknown :-1 s=CLOSED pgs=178 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:35:05.801 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:05.805+0000 7f638ebfe700 1 --2- 172.21.15.52:0/2086955673 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f63800abc50 0x7f638013afa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:35:05.801 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:05.805+0000 7f638ebfe700 1 --2- 172.21.15.52:0/2086955673 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f63800a5f00 0x7f63801429f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:35:05.801 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:05.805+0000 7f638ebfe700 1 --2- 172.21.15.52:0/2086955673 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f63800a6850 0x7f638013aa60 unknown :-1 s=CLOSED pgs=193 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:35:05.801 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:05.805+0000 7f638ebfe700 1 -- 172.21.15.52:0/2086955673 >> 172.21.15.52:0/2086955673 conn(0x7f638001b3a0 msgr2=0x7f63800a3c50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:35:05.801 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:05.805+0000 7f638ebfe700 1 -- 172.21.15.52:0/2086955673 shutdown_connections 2024-05-27T07:35:05.801 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:05.805+0000 7f638ebfe700 1 -- 172.21.15.52:0/2086955673 wait complete. 2024-05-27T07:35:06.484 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:35:06 smithi052 ceph-mon[31955]: pgmap v308: 129 pgs: 129 active+clean; 652 MiB data, 5.7 GiB used, 1.0 TiB / 1.0 TiB avail; 34 KiB/s rd, 2.1 MiB/s wr, 558 op/s 2024-05-27T07:35:06.607 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:35:06 smithi173 ceph-mon[37000]: pgmap v308: 129 pgs: 129 active+clean; 652 MiB data, 5.7 GiB used, 1.0 TiB / 1.0 TiB avail; 34 KiB/s rd, 2.1 MiB/s wr, 558 op/s 2024-05-27T07:35:06.661 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-05-27T07:35:06.661 DEBUG:teuthology.orchestra.run.smithi052:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:38f9d900c83c63799fbdbe61acc9a11b0d3554a6 shell --fsid 73df2636-1bf8-11ef-bc9b-c7b262605968 -- ceph fs dump --format=json 2024-05-27T07:35:06.695 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:35:06 smithi105 ceph-mon[35977]: pgmap v308: 129 pgs: 129 active+clean; 652 MiB data, 5.7 GiB used, 1.0 TiB / 1.0 TiB avail; 34 KiB/s rd, 2.1 MiB/s wr, 558 op/s 2024-05-27T07:35:06.960 INFO:teuthology.orchestra.run.smithi052.stderr:Inferring config /var/lib/ceph/73df2636-1bf8-11ef-bc9b-c7b262605968/mon.a/config 2024-05-27T07:35:08.565 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:35:08 smithi052 ceph-mon[31955]: pgmap v309: 129 pgs: 129 active+clean; 659 MiB data, 5.7 GiB used, 1.0 TiB / 1.0 TiB avail; 38 KiB/s rd, 2.3 MiB/s wr, 561 op/s 2024-05-27T07:35:08.566 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:35:08 smithi052 ceph-mon[31955]: osdmap e123: 12 total, 12 up, 12 in 2024-05-27T07:35:08.607 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:35:08 smithi173 ceph-mon[37000]: pgmap v309: 129 pgs: 129 active+clean; 659 MiB data, 5.7 GiB used, 1.0 TiB / 1.0 TiB avail; 38 KiB/s rd, 2.3 MiB/s wr, 561 op/s 2024-05-27T07:35:08.607 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:35:08 smithi173 ceph-mon[37000]: osdmap e123: 12 total, 12 up, 12 in 2024-05-27T07:35:08.695 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:35:08 smithi105 ceph-mon[35977]: pgmap v309: 129 pgs: 129 active+clean; 659 MiB data, 5.7 GiB used, 1.0 TiB / 1.0 TiB avail; 38 KiB/s rd, 2.3 MiB/s wr, 561 op/s 2024-05-27T07:35:08.695 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:35:08 smithi105 ceph-mon[35977]: osdmap e123: 12 total, 12 up, 12 in 2024-05-27T07:35:10.062 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:10.066+0000 7fc2fa745700 1 -- 172.21.15.52:0/284071520 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fc2f41117e0 msgr2=0x7fc2f4111bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:35:10.062 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:10.066+0000 7fc2fa745700 1 --2- 172.21.15.52:0/284071520 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fc2f41117e0 0x7fc2f4111bf0 secure :-1 s=READY pgs=726 cs=0 l=1 rev1=1 crypto rx=0x7fc2e4009990 tx=0x7fc2e4009ca0 comp rx=0 tx=0).stop 2024-05-27T07:35:10.062 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:10.067+0000 7fc2fa745700 1 -- 172.21.15.52:0/284071520 shutdown_connections 2024-05-27T07:35:10.062 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:10.067+0000 7fc2fa745700 1 --2- 172.21.15.52:0/284071520 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fc2f410b710 0x7fc2f4110470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:35:10.062 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:10.067+0000 7fc2fa745700 1 --2- 172.21.15.52:0/284071520 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fc2f41117e0 0x7fc2f4111bf0 unknown :-1 s=CLOSED pgs=726 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:35:10.062 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:10.067+0000 7fc2fa745700 1 --2- 172.21.15.52:0/284071520 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fc2f410adc0 0x7fc2f410b1d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:35:10.062 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:10.067+0000 7fc2fa745700 1 -- 172.21.15.52:0/284071520 >> 172.21.15.52:0/284071520 conn(0x7fc2f4074d80 msgr2=0x7fc2f40751c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:35:10.064 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:10.069+0000 7fc2fa745700 1 -- 172.21.15.52:0/284071520 shutdown_connections 2024-05-27T07:35:10.064 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:10.069+0000 7fc2fa745700 1 -- 172.21.15.52:0/284071520 wait complete. 2024-05-27T07:35:10.067 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:10.070+0000 7fc2fa745700 1 Processor -- start 2024-05-27T07:35:10.070 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:10.071+0000 7fc2fa745700 1 -- start start 2024-05-27T07:35:10.070 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:10.071+0000 7fc2fa745700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fc2f410adc0 0x7fc2f4120690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:35:10.070 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:10.071+0000 7fc2fa745700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fc2f410b710 0x7fc2f4119810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:35:10.070 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:10.071+0000 7fc2fa745700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fc2f41117e0 0x7fc2f4119dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:35:10.070 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:10.071+0000 7fc2fa745700 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x7fc2f41230e0 con 0x7fc2f410adc0 2024-05-27T07:35:10.070 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:10.071+0000 7fc2fa745700 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7fc2f4122f60 con 0x7fc2f41117e0 2024-05-27T07:35:10.070 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:10.071+0000 7fc2fa745700 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7fc2f4123260 con 0x7fc2f410b710 2024-05-27T07:35:10.070 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:10.071+0000 7fc2f9f44700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fc2f41117e0 0x7fc2f4119dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:35:10.072 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:10.071+0000 7fc2f9f44700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fc2f41117e0 0x7fc2f4119dd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.52:59862/0 (socket says 172.21.15.52:59862) 2024-05-27T07:35:10.072 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:10.071+0000 7fc2f9f44700 1 -- 172.21.15.52:0/3474641898 learned_addr learned my addr 172.21.15.52:0/3474641898 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-05-27T07:35:10.072 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:10.073+0000 7fc2f9f44700 1 -- 172.21.15.52:0/3474641898 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fc2f410b710 msgr2=0x7fc2f4119810 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-05-27T07:35:10.073 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:10.073+0000 7fc2f9f44700 1 --2- 172.21.15.52:0/3474641898 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fc2f410b710 0x7fc2f4119810 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:35:10.073 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:10.073+0000 7fc2f9f44700 1 -- 172.21.15.52:0/3474641898 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fc2f410adc0 msgr2=0x7fc2f4120690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:35:10.073 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:10.073+0000 7fc2f9f44700 1 --2- 172.21.15.52:0/3474641898 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fc2f410adc0 0x7fc2f4120690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:35:10.073 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:10.073+0000 7fc2f9f44700 1 -- 172.21.15.52:0/3474641898 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc2f411a630 con 0x7fc2f41117e0 2024-05-27T07:35:10.073 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:10.074+0000 7fc2f9f44700 1 --2- 172.21.15.52:0/3474641898 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fc2f41117e0 0x7fc2f4119dd0 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7fc2ec007ee0 tx=0x7fc2ec00d3b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:35:10.073 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:10.074+0000 7fc2ea7fc700 1 -- 172.21.15.52:0/3474641898 <== mon.2 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc2ec00dbd0 con 0x7fc2f41117e0 2024-05-27T07:35:10.073 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:10.074+0000 7fc2fa745700 1 -- 172.21.15.52:0/3474641898 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc2f411e2e0 con 0x7fc2f41117e0 2024-05-27T07:35:10.073 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:10.074+0000 7fc2fa745700 1 -- 172.21.15.52:0/3474641898 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc2f411e870 con 0x7fc2f41117e0 2024-05-27T07:35:10.073 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:10.075+0000 7fc2ea7fc700 1 -- 172.21.15.52:0/3474641898 <== mon.2 v2:172.21.15.105:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fc2ec00dd70 con 0x7fc2f41117e0 2024-05-27T07:35:10.073 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:10.075+0000 7fc2ea7fc700 1 -- 172.21.15.52:0/3474641898 <== mon.2 v2:172.21.15.105:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc2ec005a00 con 0x7fc2f41117e0 2024-05-27T07:35:10.073 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:10.076+0000 7fc2fa745700 1 -- 172.21.15.52:0/3474641898 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc2d8005330 con 0x7fc2f41117e0 2024-05-27T07:35:10.076 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:10.079+0000 7fc2ea7fc700 1 -- 172.21.15.52:0/3474641898 <== mon.2 v2:172.21.15.105:3300/0 4 ==== mgrmap(e 26) ==== 149783+0+0 (secure 0 0 0) 0x7fc2ec01a020 con 0x7fc2f41117e0 2024-05-27T07:35:10.076 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:10.080+0000 7fc2ea7fc700 1 --2- 172.21.15.52:0/3474641898 >> [v2:172.21.15.52:6800/1822318384,v1:172.21.15.52:6801/1822318384] conn(0x7fc2e00b2af0 0x7fc2e00b5000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:35:10.076 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:10.081+0000 7fc2ea7fc700 1 -- 172.21.15.52:0/3474641898 <== mon.2 v2:172.21.15.105:3300/0 5 ==== osd_map(123..123 src has 1..123) ==== 9555+0+0 (secure 0 0 0) 0x7fc2ec0e27f0 con 0x7fc2f41117e0 2024-05-27T07:35:10.077 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:10.082+0000 7fc2f9743700 1 --2- 172.21.15.52:0/3474641898 >> [v2:172.21.15.52:6800/1822318384,v1:172.21.15.52:6801/1822318384] conn(0x7fc2e00b2af0 0x7fc2e00b5000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:35:10.082 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:10.087+0000 7fc2f9743700 1 --2- 172.21.15.52:0/3474641898 >> [v2:172.21.15.52:6800/1822318384,v1:172.21.15.52:6801/1822318384] conn(0x7fc2e00b2af0 0x7fc2e00b5000 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7fc2e40098e0 tx=0x7fc2e4000bc0 comp rx=0 tx=0).ready entity=mgr.15333 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:35:10.083 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:10.088+0000 7fc2ea7fc700 1 -- 172.21.15.52:0/3474641898 <== mon.2 v2:172.21.15.105:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191963 (secure 0 0 0) 0x7fc2ec010070 con 0x7fc2f41117e0 2024-05-27T07:35:10.127 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:35:10.132+0000 7f97159cf640 1 -- 172.21.15.52:0/169228672 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f9710107410 msgr2=0x7f9710107820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:35:10.128 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:35:10.132+0000 7f97159cf640 1 --2- 172.21.15.52:0/169228672 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f9710107410 0x7f9710107820 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7f9708009ca0 tx=0x7f970802a960 comp rx=0 tx=0).stop 2024-05-27T07:35:10.128 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:35:10.132+0000 7f97159cf640 1 -- 172.21.15.52:0/169228672 shutdown_connections 2024-05-27T07:35:10.128 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:35:10.132+0000 7f97159cf640 1 --2- 172.21.15.52:0/169228672 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f9710107410 0x7f9710107820 unknown :-1 s=CLOSED pgs=198 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:35:10.128 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:35:10.132+0000 7f97159cf640 1 --2- 172.21.15.52:0/169228672 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f9710108f80 0x7f9710102200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:35:10.128 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:35:10.132+0000 7f97159cf640 1 --2- 172.21.15.52:0/169228672 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f9710108b70 0x7f9710101c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:35:10.128 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:35:10.132+0000 7f97159cf640 1 -- 172.21.15.52:0/169228672 >> 172.21.15.52:0/169228672 conn(0x7f97100779d0 msgr2=0x7f9710077e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:35:10.134 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:35:10.136+0000 7f97159cf640 1 -- 172.21.15.52:0/169228672 shutdown_connections 2024-05-27T07:35:10.134 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:35:10.138+0000 7f97159cf640 1 -- 172.21.15.52:0/169228672 wait complete. 2024-05-27T07:35:10.135 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:35:10.139+0000 7f97159cf640 1 Processor -- start 2024-05-27T07:35:10.137 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:35:10.139+0000 7f97159cf640 1 -- start start 2024-05-27T07:35:10.137 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:35:10.139+0000 7f97159cf640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f9710107410 0x7f97101a0fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:35:10.138 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:35:10.139+0000 7f97159cf640 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f9710108b70 0x7f97101a1520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:35:10.138 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:35:10.139+0000 7f97159cf640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f9710108f80 0x7f97101a57e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:35:10.138 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:35:10.139+0000 7f97159cf640 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x7f971011a560 con 0x7f9710108b70 2024-05-27T07:35:10.138 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:35:10.140+0000 7f97159cf640 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f971011a3e0 con 0x7f9710107410 2024-05-27T07:35:10.138 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:35:10.140+0000 7f97159cf640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f971011a6e0 con 0x7f9710108f80 2024-05-27T07:35:10.138 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:35:10.141+0000 7f970e7fc640 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f9710108b70 0x7f97101a1520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:35:10.138 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:35:10.141+0000 7f970e7fc640 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f9710108b70 0x7f97101a1520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.52:48886/0 (socket says 172.21.15.52:48886) 2024-05-27T07:35:10.138 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:35:10.141+0000 7f970e7fc640 1 -- 172.21.15.52:0/876878597 learned_addr learned my addr 172.21.15.52:0/876878597 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-05-27T07:35:10.138 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:35:10.141+0000 7f970f7fe640 1 --2- 172.21.15.52:0/876878597 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f9710108f80 0x7f97101a57e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:35:10.142 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:35:10.143+0000 7f970e7fc640 1 -- 172.21.15.52:0/876878597 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f9710108f80 msgr2=0x7f97101a57e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:35:10.142 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:35:10.143+0000 7f970e7fc640 1 --2- 172.21.15.52:0/876878597 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f9710108f80 0x7f97101a57e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:35:10.143 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:35:10.143+0000 7f970e7fc640 1 -- 172.21.15.52:0/876878597 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f9710107410 msgr2=0x7f97101a0fc0 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-05-27T07:35:10.143 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:35:10.143+0000 7f970e7fc640 1 --2- 172.21.15.52:0/876878597 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f9710107410 0x7f97101a0fc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:35:10.143 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:35:10.143+0000 7f970e7fc640 1 -- 172.21.15.52:0/876878597 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f97101a5e20 con 0x7f9710108b70 2024-05-27T07:35:10.144 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:35:10.148+0000 7f970e7fc640 1 --2- 172.21.15.52:0/876878597 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f9710108b70 0x7f97101a1520 secure :-1 s=READY pgs=727 cs=0 l=1 rev1=1 crypto rx=0x7f970400b450 tx=0x7f970400b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:35:10.145 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:35:10.149+0000 7f97149cd640 1 -- 172.21.15.52:0/876878597 <== mon.0 v2:172.21.15.52:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9704015020 con 0x7f9710108b70 2024-05-27T07:35:10.145 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:35:10.149+0000 7f97149cd640 1 -- 172.21.15.52:0/876878597 <== mon.0 v2:172.21.15.52:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f9704011d40 con 0x7f9710108b70 2024-05-27T07:35:10.145 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:35:10.149+0000 7f97149cd640 1 -- 172.21.15.52:0/876878597 <== mon.0 v2:172.21.15.52:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f970400f340 con 0x7f9710108b70 2024-05-27T07:35:10.145 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:35:10.149+0000 7f97159cf640 1 -- 172.21.15.52:0/876878597 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f97101a6110 con 0x7f9710108b70 2024-05-27T07:35:10.145 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:35:10.149+0000 7f97159cf640 1 -- 172.21.15.52:0/876878597 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f97101a6610 con 0x7f9710108b70 2024-05-27T07:35:10.154 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:35:10.159+0000 7f97149cd640 1 -- 172.21.15.52:0/876878597 <== mon.0 v2:172.21.15.52:3300/0 4 ==== mgrmap(e 26) ==== 149783+0+0 (secure 0 0 0) 0x7f9704018020 con 0x7f9710108b70 2024-05-27T07:35:10.156 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:35:10.162+0000 7f971c87d740 1 Processor -- start 2024-05-27T07:35:10.157 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:35:10.162+0000 7f971c87d740 1 -- start start 2024-05-27T07:35:10.163 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:35:10.162+0000 7f971c87d740 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x5625b24c01b0 0x5625b28613d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:35:10.164 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:35:10.162+0000 7f971c87d740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5625b24972f0 0x5625b2861910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:35:10.164 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:35:10.162+0000 7f971c87d740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5625b249f4d0 0x5625b2861e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:35:10.164 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:35:10.162+0000 7f971c87d740 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x5625b27d3c30 con 0x5625b24c01b0 2024-05-27T07:35:10.164 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:35:10.162+0000 7f971c87d740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x5625b27c37d0 con 0x5625b24972f0 2024-05-27T07:35:10.164 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:35:10.162+0000 7f971c87d740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x5625b27c7810 con 0x5625b249f4d0 2024-05-27T07:35:10.164 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:35:10.163+0000 7f970e7fc640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5625b249f4d0 0x5625b2861e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:35:10.164 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:35:10.163+0000 7f970e7fc640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5625b249f4d0 0x5625b2861e50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.173:3300/0 says I am v2:172.21.15.52:60160/0 (socket says 172.21.15.52:60160) 2024-05-27T07:35:10.164 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:35:10.163+0000 7f970e7fc640 1 -- 172.21.15.52:0/2928818532 learned_addr learned my addr 172.21.15.52:0/2928818532 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-05-27T07:35:10.164 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:35:10.163+0000 7f970f7fe640 1 --2- 172.21.15.52:0/2928818532 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x5625b24c01b0 0x5625b28613d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:35:10.164 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:35:10.163+0000 7f970e7fc640 1 -- 172.21.15.52:0/2928818532 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5625b24972f0 msgr2=0x5625b2861910 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-05-27T07:35:10.164 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:35:10.163+0000 7f970e7fc640 1 --2- 172.21.15.52:0/2928818532 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5625b24972f0 0x5625b2861910 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:35:10.164 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:35:10.163+0000 7f970e7fc640 1 -- 172.21.15.52:0/2928818532 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x5625b24c01b0 msgr2=0x5625b28613d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:35:10.165 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:35:10.163+0000 7f970e7fc640 1 --2- 172.21.15.52:0/2928818532 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x5625b24c01b0 0x5625b28613d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:35:10.165 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:35:10.163+0000 7f970e7fc640 1 -- 172.21.15.52:0/2928818532 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5625b24d8210 con 0x5625b249f4d0 2024-05-27T07:35:10.165 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:35:10.164+0000 7f970e7fc640 1 --2- 172.21.15.52:0/2928818532 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5625b249f4d0 0x5625b2861e50 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7f97040df750 tx=0x7f97040ea9f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=5fb4f9c84623363a server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:35:10.165 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:35:10.171+0000 7f97149cd640 1 --2- 172.21.15.52:0/876878597 >> [v2:172.21.15.52:6800/1822318384,v1:172.21.15.52:6801/1822318384] conn(0x7f96f80b2aa0 0x7f96f80b4fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:35:10.166 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:35:10.171+0000 7f97149cd640 1 -- 172.21.15.52:0/876878597 <== mon.0 v2:172.21.15.52:3300/0 5 ==== osd_map(123..123 src has 1..123) ==== 9555+0+0 (secure 0 0 0) 0x7f97040e2420 con 0x7f9710108b70 2024-05-27T07:35:10.166 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:35:10.166+0000 7f96f5ffb640 1 -- 172.21.15.52:0/2928818532 <== mon.1 v2:172.21.15.173:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f97040f6070 con 0x5625b249f4d0 2024-05-27T07:35:10.167 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:35:10.172+0000 7f96f5ffb640 1 -- 172.21.15.52:0/2928818532 <== mon.1 v2:172.21.15.173:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f97040e44e0 con 0x5625b249f4d0 2024-05-27T07:35:10.167 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:35:10.172+0000 7f96f5ffb640 1 -- 172.21.15.52:0/2928818532 <== mon.1 v2:172.21.15.173:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f97040ee9b0 con 0x5625b249f4d0 2024-05-27T07:35:10.169 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:35:10.174+0000 7f971c87d740 1 -- 172.21.15.52:0/2928818532 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5625b249f4d0 msgr2=0x5625b2861e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:35:10.169 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:35:10.174+0000 7f971c87d740 1 --2- 172.21.15.52:0/2928818532 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5625b249f4d0 0x5625b2861e50 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7f97040df750 tx=0x7f97040ea9f0 comp rx=0 tx=0).stop 2024-05-27T07:35:10.169 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:35:10.174+0000 7f970f7fe640 1 --2- 172.21.15.52:0/876878597 >> [v2:172.21.15.52:6800/1822318384,v1:172.21.15.52:6801/1822318384] conn(0x7f96f80b2aa0 0x7f96f80b4fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:35:10.170 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:35:10.175+0000 7f970f7fe640 1 --2- 172.21.15.52:0/876878597 >> [v2:172.21.15.52:6800/1822318384,v1:172.21.15.52:6801/1822318384] conn(0x7f96f80b2aa0 0x7f96f80b4fb0 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x5625b285e540 tx=0x7f97080093e0 comp rx=0 tx=0).ready entity=mgr.15333 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:35:10.173 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:35:10.177+0000 7f971c87d740 1 -- 172.21.15.52:0/2928818532 shutdown_connections 2024-05-27T07:35:10.173 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:35:10.177+0000 7f971c87d740 1 --2- 172.21.15.52:0/2928818532 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5625b249f4d0 0x5625b2861e50 unknown :-1 s=CLOSED pgs=199 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:35:10.174 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:35:10.177+0000 7f971c87d740 1 --2- 172.21.15.52:0/2928818532 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5625b24972f0 0x5625b2861910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:35:10.174 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:35:10.177+0000 7f971c87d740 1 --2- 172.21.15.52:0/2928818532 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x5625b24c01b0 0x5625b28613d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:35:10.174 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:35:10.177+0000 7f971c87d740 1 -- 172.21.15.52:0/2928818532 >> 172.21.15.52:0/2928818532 conn(0x5625b24b4300 msgr2=0x5625b285db80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:35:10.182 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:35:10.187+0000 7f971c87d740 1 -- 172.21.15.52:0/2928818532 shutdown_connections 2024-05-27T07:35:10.190 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:35:10.195+0000 7f971c87d740 1 -- 172.21.15.52:0/2928818532 wait complete. 2024-05-27T07:35:10.190 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:35:10.195+0000 7f971c87d740 1 -- start start 2024-05-27T07:35:10.191 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:35:10.195+0000 7f971c87d740 1 Processor -- start 2024-05-27T07:35:10.191 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:35:10.196+0000 7f971c87d740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5625b24972f0 0x5625b286d490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:35:10.191 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:35:10.196+0000 7f971c87d740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5625b249f4d0 0x5625b286d9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:35:10.192 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:35:10.196+0000 7f971c87d740 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x5625b24c01b0 0x5625b286b9a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:35:10.192 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:35:10.196+0000 7f971c87d740 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x5625b24a9790 con 0x5625b24c01b0 2024-05-27T07:35:10.192 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:35:10.196+0000 7f971c87d740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x5625b277f890 con 0x5625b249f4d0 2024-05-27T07:35:10.192 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:35:10.196+0000 7f971c87d740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x5625b286e500 con 0x5625b24972f0 2024-05-27T07:35:10.192 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:35:10.196+0000 7f971c87d740 10 client.0 _ll_register_callbacks cb 0x5625b24d7a00 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T07:35:10.193 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:35:10.198+0000 7f970effd640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5625b24972f0 0x5625b286d490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:35:10.193 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:35:10.198+0000 7f970effd640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5625b24972f0 0x5625b286d490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.173:3300/0 says I am v2:172.21.15.52:60166/0 (socket says 172.21.15.52:60166) 2024-05-27T07:35:10.193 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:35:10.198+0000 7f970effd640 1 -- 172.21.15.52:0/2696870670 learned_addr learned my addr 172.21.15.52:0/2696870670 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-05-27T07:35:10.193 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:35:10.198+0000 7f970f7fe640 1 --2- 172.21.15.52:0/2696870670 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x5625b24c01b0 0x5625b286b9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:35:10.193 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:35:10.198+0000 7f970effd640 1 -- 172.21.15.52:0/2696870670 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5625b249f4d0 msgr2=0x5625b286d9d0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-05-27T07:35:10.193 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:35:10.198+0000 7f970effd640 1 --2- 172.21.15.52:0/2696870670 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5625b249f4d0 0x5625b286d9d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:35:10.193 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:35:10.198+0000 7f970effd640 1 -- 172.21.15.52:0/2696870670 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x5625b24c01b0 msgr2=0x5625b286b9a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:35:10.194 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:35:10.198+0000 7f970effd640 1 --2- 172.21.15.52:0/2696870670 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x5625b24c01b0 0x5625b286b9a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:35:10.194 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:35:10.198+0000 7f970effd640 1 -- 172.21.15.52:0/2696870670 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5625b25a4620 con 0x5625b24972f0 2024-05-27T07:35:10.198 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:35:10.203+0000 7f970effd640 1 --2- 172.21.15.52:0/2696870670 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5625b24972f0 0x5625b286d490 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7f970000bd60 tx=0x7f9700009490 comp rx=0 tx=0).ready entity=mon.1 client_cookie=ad0364cabf9245b server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:35:10.199 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:35:10.204+0000 7f96eeffd640 10 client.0 ms_handle_connect on v2:172.21.15.173:3300/0 2024-05-27T07:35:10.204 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:35:10.209+0000 7f96eeffd640 1 -- 172.21.15.52:0/2696870670 <== mon.1 v2:172.21.15.173:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9700011420 con 0x5625b24972f0 2024-05-27T07:35:10.204 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:35:10.209+0000 7f96eeffd640 1 -- 172.21.15.52:0/2696870670 <== mon.1 v2:172.21.15.173:3300/0 2 ==== osd_map(123..123 src has 1..123) ==== 9555+0+0 (secure 0 0 0) 0x7f97000115c0 con 0x5625b24972f0 2024-05-27T07:35:10.204 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:35:10.210+0000 7f971c87d740 1 -- 172.21.15.52:0/2696870670 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f97040e44e0 con 0x5625b24972f0 2024-05-27T07:35:10.210 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:35:10.215+0000 7f96eeffd640 1 -- 172.21.15.52:0/2696870670 <== mon.1 v2:172.21.15.173:3300/0 3 ==== mon_get_version_reply(handle=1 version=107) ==== 24+0+0 (secure 0 0 0) 0x7f9700011800 con 0x5625b24972f0 2024-05-27T07:35:10.210 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:35:10.215+0000 7f971c87d740 10 client.26572 fetch_fsmap learned FSMap version 107 2024-05-27T07:35:10.210 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:35:10.216+0000 7f971c87d740 1 -- 172.21.15.52:0/2696870670 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({config=0+,fsmap=107,monmap=4+}) -- 0x5625b287a930 con 0x5625b24972f0 2024-05-27T07:35:10.217 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:35:10.222+0000 7f96eeffd640 1 -- 172.21.15.52:0/2696870670 <== mon.1 v2:172.21.15.173:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f97040e44e0 con 0x5625b24972f0 2024-05-27T07:35:10.217 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:35:10.222+0000 7f96eeffd640 1 -- 172.21.15.52:0/2696870670 <== mon.1 v2:172.21.15.173:3300/0 5 ==== fsmap(e 107) ==== 4199+0+0 (secure 0 0 0) 0x7f9700019ab0 con 0x5625b24972f0 2024-05-27T07:35:10.217 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:35:10.223+0000 7f971c87d740 10 client.26572 fetch_fsmap finished waiting for FSMap version 107 2024-05-27T07:35:10.218 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:35:10.223+0000 7f971c87d740 10 client.26572 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-05-27T07:35:10.218 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:35:10.223+0000 7f971c87d740 20 client.26572 populate_metadata read hostname 'smithi052' 2024-05-27T07:35:10.218 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:35:10.223+0000 7f971c87d740 1 --2- 172.21.15.52:0/2696870670 >> [v2:172.21.15.105:6838/500348151,v1:172.21.15.105:6839/500348151] conn(0x5625b247ed80 0x5625b287e370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:35:10.218 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:35:10.223+0000 7f971c87d740 4 client.26572 mds_command: new command op to 24527 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-27T07:35:10.218 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:35:10.223+0000 7f971c87d740 1 -- 172.21.15.52:0/2696870670 --> [v2:172.21.15.105:6838/500348151,v1:172.21.15.105:6839/500348151] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5625b27d5b80 con 0x5625b247ed80 2024-05-27T07:35:10.218 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:35:10.224+0000 7f970e7fc640 1 --2- 172.21.15.52:0/2696870670 >> [v2:172.21.15.105:6838/500348151,v1:172.21.15.105:6839/500348151] conn(0x5625b247ed80 0x5625b287e370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:35:10.234 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:35:10.234+0000 7f970e7fc640 1 --2- 172.21.15.52:0/2696870670 >> [v2:172.21.15.105:6838/500348151,v1:172.21.15.105:6839/500348151] conn(0x5625b247ed80 0x5625b287e370 crc :-1 s=READY pgs=163 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=804e5bf52fb742f9 server_cookie=cc38a2f883c7a74d in_seq=0 out_seq=0 2024-05-27T07:35:10.234 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:35:10.234+0000 7f96eeffd640 10 client.26572 ms_handle_connect on v2:172.21.15.105:6838/500348151 2024-05-27T07:35:10.234 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:35:10.238+0000 7f96eeffd640 1 -- 172.21.15.52:0/2696870670 <== mds.0 v2:172.21.15.105:6838/500348151 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24041 (crc 0 0 0) 0x5625b27d5b80 con 0x5625b247ed80 2024-05-27T07:35:10.234 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:35:10.238+0000 7f96eeffd640 10 client.26572 handle_command_reply: tid=0 2024-05-27T07:35:10.234 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:35:10.239+0000 7f971c87d740 1 client.26572 shutdown 2024-05-27T07:35:10.238 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:35:10.243+0000 7f971c87d740 1 -- 172.21.15.52:0/2696870670 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5625b24972f0 msgr2=0x5625b286d490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:35:10.238 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:35:10.243+0000 7f971c87d740 1 --2- 172.21.15.52:0/2696870670 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5625b24972f0 0x5625b286d490 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7f970000bd60 tx=0x7f9700009490 comp rx=0 tx=0).stop 2024-05-27T07:35:10.239 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:35:10.245+0000 7f971c87d740 1 -- 172.21.15.52:0/2696870670 shutdown_connections 2024-05-27T07:35:10.239 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:35:10.245+0000 7f971c87d740 1 --2- 172.21.15.52:0/2696870670 >> [v2:172.21.15.105:6838/500348151,v1:172.21.15.105:6839/500348151] conn(0x5625b247ed80 0x5625b287e370 crc :-1 s=READY pgs=163 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:35:10.240 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:35:10.245+0000 7f971c87d740 1 --2- 172.21.15.52:0/2696870670 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x5625b24c01b0 0x5625b286b9a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:35:10.240 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:35:10.245+0000 7f971c87d740 1 --2- 172.21.15.52:0/2696870670 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5625b249f4d0 0x5625b286d9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:35:10.240 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:35:10.245+0000 7f971c87d740 1 --2- 172.21.15.52:0/2696870670 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5625b24972f0 0x5625b286d490 unknown :-1 s=CLOSED pgs=200 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:35:10.240 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:35:10.245+0000 7f971c87d740 1 -- 172.21.15.52:0/2696870670 >> 172.21.15.52:0/2696870670 conn(0x5625b24b4300 msgr2=0x5625b28630f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:35:10.240 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:35:10.245+0000 7f96eeffd640 0 client.26572 ms_handle_reset on v2:172.21.15.105:6838/500348151 2024-05-27T07:35:10.245 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:35:10.250+0000 7f971c87d740 1 -- 172.21.15.52:0/2696870670 shutdown_connections 2024-05-27T07:35:10.246 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:35:10.251+0000 7f971c87d740 1 -- 172.21.15.52:0/2696870670 wait complete. 2024-05-27T07:35:10.247 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:35:10.252+0000 7f971c87d740 20 client.26572 trim_cache size 0 max 16384 2024-05-27T07:35:10.285 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:35:10.289+0000 7f971c87d740 1 Processor -- start 2024-05-27T07:35:10.286 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:35:10.290+0000 7f971c87d740 1 -- start start 2024-05-27T07:35:10.286 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:35:10.290+0000 7f971c87d740 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x5625b24972f0 0x5625b286d7d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:35:10.286 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:35:10.290+0000 7f971c87d740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5625b249f4d0 0x5625b2862170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:35:10.286 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:35:10.290+0000 7f971c87d740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5625b24c01b0 0x5625b2865150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:35:10.286 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:35:10.290+0000 7f971c87d740 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x7f97101c09e0 con 0x5625b24972f0 2024-05-27T07:35:10.286 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:35:10.290+0000 7f971c87d740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f97101bee40 con 0x5625b249f4d0 2024-05-27T07:35:10.286 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:35:10.290+0000 7f971c87d740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f97101c02e0 con 0x5625b24c01b0 2024-05-27T07:35:10.291 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:35:10.294+0000 7f970f7fe640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5625b24c01b0 0x5625b2865150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:35:10.291 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:35:10.294+0000 7f970f7fe640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5625b24c01b0 0x5625b2865150 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.173:3300/0 says I am v2:172.21.15.52:60178/0 (socket says 172.21.15.52:60178) 2024-05-27T07:35:10.291 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:35:10.294+0000 7f970f7fe640 1 -- 172.21.15.52:0/962655498 learned_addr learned my addr 172.21.15.52:0/962655498 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-05-27T07:35:10.291 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:35:10.295+0000 7f970e7fc640 1 --2- 172.21.15.52:0/962655498 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5625b249f4d0 0x5625b2862170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:35:10.291 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:35:10.295+0000 7f970effd640 1 --2- 172.21.15.52:0/962655498 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x5625b24972f0 0x5625b286d7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:35:10.291 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:35:10.296+0000 7f970f7fe640 1 -- 172.21.15.52:0/962655498 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5625b249f4d0 msgr2=0x5625b2862170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:35:10.291 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:35:10.296+0000 7f970f7fe640 1 --2- 172.21.15.52:0/962655498 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5625b249f4d0 0x5625b2862170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:35:10.292 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:35:10.296+0000 7f970f7fe640 1 -- 172.21.15.52:0/962655498 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x5625b24972f0 msgr2=0x5625b286d7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:35:10.292 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:35:10.296+0000 7f970f7fe640 1 --2- 172.21.15.52:0/962655498 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x5625b24972f0 0x5625b286d7d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:35:10.292 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:35:10.296+0000 7f970f7fe640 1 -- 172.21.15.52:0/962655498 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5625b286c910 con 0x5625b24c01b0 2024-05-27T07:35:10.297 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:35:10.301+0000 7f970f7fe640 1 --2- 172.21.15.52:0/962655498 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5625b24c01b0 0x5625b2865150 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7f9708035000 tx=0x7f9708044000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=c8bdc7e5a429973d server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:35:10.302 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:35:10.307+0000 7f96f57fa640 1 -- 172.21.15.52:0/962655498 <== mon.1 v2:172.21.15.173:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9708055070 con 0x5625b24c01b0 2024-05-27T07:35:10.302 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:35:10.307+0000 7f96f57fa640 1 -- 172.21.15.52:0/962655498 <== mon.1 v2:172.21.15.173:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f9708030b60 con 0x5625b24c01b0 2024-05-27T07:35:10.302 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:35:10.307+0000 7f96f57fa640 1 -- 172.21.15.52:0/962655498 <== mon.1 v2:172.21.15.173:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9708034040 con 0x5625b24c01b0 2024-05-27T07:35:10.304 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:35:10.309+0000 7f971c87d740 1 -- 172.21.15.52:0/962655498 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5625b24c01b0 msgr2=0x5625b2865150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:35:10.304 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:35:10.309+0000 7f971c87d740 1 --2- 172.21.15.52:0/962655498 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5625b24c01b0 0x5625b2865150 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7f9708035000 tx=0x7f9708044000 comp rx=0 tx=0).stop 2024-05-27T07:35:10.307 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:35:10.312+0000 7f971c87d740 1 -- 172.21.15.52:0/962655498 shutdown_connections 2024-05-27T07:35:10.307 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:35:10.312+0000 7f971c87d740 1 --2- 172.21.15.52:0/962655498 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5625b24c01b0 0x5625b2865150 unknown :-1 s=CLOSED pgs=201 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:35:10.307 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:35:10.312+0000 7f971c87d740 1 --2- 172.21.15.52:0/962655498 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5625b249f4d0 0x5625b2862170 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:35:10.308 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:35:10.312+0000 7f971c87d740 1 --2- 172.21.15.52:0/962655498 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x5625b24972f0 0x5625b286d7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:35:10.308 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:35:10.312+0000 7f971c87d740 1 -- 172.21.15.52:0/962655498 >> 172.21.15.52:0/962655498 conn(0x5625b24b4300 msgr2=0x5625b286d0c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:35:10.312 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:35:10.316+0000 7f971c87d740 1 -- 172.21.15.52:0/962655498 shutdown_connections 2024-05-27T07:35:10.316 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:35:10.320+0000 7f971c87d740 1 -- 172.21.15.52:0/962655498 wait complete. 2024-05-27T07:35:10.316 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:35:10.321+0000 7f971c87d740 1 -- start start 2024-05-27T07:35:10.316 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:35:10.321+0000 7f971c87d740 1 Processor -- start 2024-05-27T07:35:10.318 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:35:10.322+0000 7f971c87d740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5625b24972f0 0x5625b28737c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:35:10.318 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:35:10.322+0000 7f971c87d740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5625b249f4d0 0x5625b2873d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:35:10.318 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:35:10.322+0000 7f971c87d740 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x5625b24c01b0 0x5625b28e1a20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:35:10.318 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:35:10.323+0000 7f971c87d740 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x7f97101bfc10 con 0x5625b24c01b0 2024-05-27T07:35:10.318 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:35:10.323+0000 7f971c87d740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f97101bf510 con 0x5625b249f4d0 2024-05-27T07:35:10.318 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:35:10.323+0000 7f970f7fe640 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x5625b24c01b0 0x5625b28e1a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:35:10.318 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:35:10.323+0000 7f971c87d740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x5625b247efb0 con 0x5625b24972f0 2024-05-27T07:35:10.318 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:35:10.323+0000 7f971c87d740 10 client.0 _ll_register_callbacks cb 0x7f9700006df0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T07:35:10.318 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:35:10.323+0000 7f970effd640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5625b24972f0 0x5625b28737c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:35:10.318 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:35:10.323+0000 7f970effd640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5625b24972f0 0x5625b28737c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.173:3300/0 says I am v2:172.21.15.52:60182/0 (socket says 172.21.15.52:60182) 2024-05-27T07:35:10.319 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:35:10.323+0000 7f970effd640 1 -- 172.21.15.52:0/3545053736 learned_addr learned my addr 172.21.15.52:0/3545053736 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-05-27T07:35:10.319 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:35:10.323+0000 7f970e7fc640 1 --2- 172.21.15.52:0/3545053736 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5625b249f4d0 0x5625b2873d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:35:10.320 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:35:10.325+0000 7f970effd640 1 -- 172.21.15.52:0/3545053736 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5625b249f4d0 msgr2=0x5625b2873d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:35:10.320 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:35:10.325+0000 7f970effd640 1 --2- 172.21.15.52:0/3545053736 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5625b249f4d0 0x5625b2873d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:35:10.320 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:35:10.325+0000 7f970effd640 1 -- 172.21.15.52:0/3545053736 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x5625b24c01b0 msgr2=0x5625b28e1a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:35:10.320 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:35:10.325+0000 7f970effd640 1 --2- 172.21.15.52:0/3545053736 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x5625b24c01b0 0x5625b28e1a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:35:10.320 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:35:10.325+0000 7f970effd640 1 -- 172.21.15.52:0/3545053736 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5625b28697d0 con 0x5625b24972f0 2024-05-27T07:35:10.323 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:35:10.328+0000 7f970effd640 1 --2- 172.21.15.52:0/3545053736 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5625b24972f0 0x5625b28737c0 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7f970000bd30 tx=0x7f9700011650 comp rx=0 tx=0).ready entity=mon.1 client_cookie=f5141e3fa34f0a61 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:35:10.323 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:35:10.329+0000 7f96eeffd640 10 client.0 ms_handle_connect on v2:172.21.15.173:3300/0 2024-05-27T07:35:10.329 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:35:10.334+0000 7f96eeffd640 1 -- 172.21.15.52:0/3545053736 <== mon.1 v2:172.21.15.173:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f970001b220 con 0x5625b24972f0 2024-05-27T07:35:10.329 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:35:10.334+0000 7f96eeffd640 1 -- 172.21.15.52:0/3545053736 <== mon.1 v2:172.21.15.173:3300/0 2 ==== osd_map(123..123 src has 1..123) ==== 9555+0+0 (secure 0 0 0) 0x7f9700011bb0 con 0x5625b24972f0 2024-05-27T07:35:10.329 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:35:10.334+0000 7f971c87d740 1 -- 172.21.15.52:0/3545053736 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f9708030b60 con 0x5625b24972f0 2024-05-27T07:35:10.329 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:35:10.334+0000 7f96eeffd640 1 -- 172.21.15.52:0/3545053736 <== mon.1 v2:172.21.15.173:3300/0 3 ==== mon_get_version_reply(handle=1 version=107) ==== 24+0+0 (secure 0 0 0) 0x7f9700011dd0 con 0x5625b24972f0 2024-05-27T07:35:10.330 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:35:10.334+0000 7f971c87d740 10 client.26578 fetch_fsmap learned FSMap version 107 2024-05-27T07:35:10.330 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:35:10.334+0000 7f971c87d740 1 -- 172.21.15.52:0/3545053736 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({config=0+,fsmap=107,monmap=4+}) -- 0x5625b2867f80 con 0x5625b24972f0 2024-05-27T07:35:10.330 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:35:10.335+0000 7f96eeffd640 1 -- 172.21.15.52:0/3545053736 <== mon.1 v2:172.21.15.173:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f9708030b60 con 0x5625b24972f0 2024-05-27T07:35:10.330 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:35:10.335+0000 7f96eeffd640 1 -- 172.21.15.52:0/3545053736 <== mon.1 v2:172.21.15.173:3300/0 5 ==== fsmap(e 107) ==== 4199+0+0 (secure 0 0 0) 0x7f9700003ab0 con 0x5625b24972f0 2024-05-27T07:35:10.330 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:35:10.335+0000 7f971c87d740 10 client.26578 fetch_fsmap finished waiting for FSMap version 107 2024-05-27T07:35:10.330 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:35:10.335+0000 7f971c87d740 10 client.26578 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-05-27T07:35:10.331 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:35:10.335+0000 7f971c87d740 20 client.26578 populate_metadata read hostname 'smithi052' 2024-05-27T07:35:10.331 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:35:10.335+0000 7f971c87d740 1 --2- 172.21.15.52:0/3545053736 >> [v2:172.21.15.105:6838/500348151,v1:172.21.15.105:6839/500348151] conn(0x5625b24541c0 0x5625b28f9a60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:35:10.331 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:35:10.335+0000 7f971c87d740 4 client.26578 mds_command: new command op to 24527 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-05-27T07:35:10.331 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:35:10.335+0000 7f971c87d740 1 -- 172.21.15.52:0/3545053736 --> [v2:172.21.15.105:6838/500348151,v1:172.21.15.105:6839/500348151] -- command(tid 0: {"prefix": "flush journal"}) -- 0x5625b27cf360 con 0x5625b24541c0 2024-05-27T07:35:10.332 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:35:10.337+0000 7f970e7fc640 1 --2- 172.21.15.52:0/3545053736 >> [v2:172.21.15.105:6838/500348151,v1:172.21.15.105:6839/500348151] conn(0x5625b24541c0 0x5625b28f9a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:35:10.333 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:35:10.338+0000 7f970e7fc640 1 --2- 172.21.15.52:0/3545053736 >> [v2:172.21.15.105:6838/500348151,v1:172.21.15.105:6839/500348151] conn(0x5625b24541c0 0x5625b28f9a60 crc :-1 s=READY pgs=164 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=29747a3b96387585 server_cookie=170a81df897f4264 in_seq=0 out_seq=0 2024-05-27T07:35:10.333 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:35:10.338+0000 7f96eeffd640 10 client.26578 ms_handle_connect on v2:172.21.15.105:6838/500348151 2024-05-27T07:35:10.596 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:10.601+0000 7fc2fa745700 1 -- 172.21.15.52:0/3474641898 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fc2d8005620 con 0x7fc2f41117e0 2024-05-27T07:35:10.596 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:35:10 smithi052 ceph-mon[31955]: pgmap v311: 129 pgs: 12 active+clean+snaptrim, 31 active+clean+snaptrim_wait, 86 active+clean; 659 MiB data, 5.8 GiB used, 1.0 TiB / 1.0 TiB avail; 47 KiB/s rd, 1.7 MiB/s wr, 505 op/s 2024-05-27T07:35:10.601 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:10.606+0000 7fc2ea7fc700 1 -- 172.21.15.52:0/3474641898 <== mon.2 v2:172.21.15.105:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 107 v107) ==== 95+0+6123 (secure 0 0 0) 0x7fc2ec0a9d60 con 0x7fc2f41117e0 2024-05-27T07:35:10.603 INFO:teuthology.orchestra.run.smithi052.stdout: 2024-05-27T07:35:10.603 INFO:teuthology.orchestra.run.smithi052.stdout:{"epoch":107,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":107,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-05-27T07:21:50.521783+0000","modified":"2024-05-27T07:34:31.780374+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24527,"mds_1":24497,"mds_2":24475,"mds_3":24473,"mds_4":24466},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24466":{"gid":24466,"name":"c","rank":4,"incarnation":26,"state":"up:active","state_seq":25,"addr":"172.21.15.173:6837/1396363827","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6836","nonce":1396363827},{"type":"v1","addr":"172.21.15.173:6837","nonce":1396363827}]},"join_fscid":-1,"export_targets":[0],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24473":{"gid":24473,"name":"e","rank":3,"incarnation":24,"state":"up:active","state_seq":26,"addr":"172.21.15.105:6835/3298498384","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6834","nonce":3298498384},{"type":"v1","addr":"172.21.15.105:6835","nonce":3298498384}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24475":{"gid":24475,"name":"i","rank":2,"incarnation":22,"state":"up:active","state_seq":23,"addr":"172.21.15.173:6839/564778293","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6838","nonce":564778293},{"type":"v1","addr":"172.21.15.173:6839","nonce":564778293}]},"join_fscid":-1,"export_targets":[0],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24497":{"gid":24497,"name":"h","rank":1,"incarnation":20,"state":"up:active","state_seq":24,"addr":"172.21.15.105:6837/191393149","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6836","nonce":191393149},{"type":"v1","addr":"172.21.15.105:6837","nonce":191393149}]},"join_fscid":-1,"export_targets":[0],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24527":{"gid":24527,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":5,"addr":"172.21.15.105:6839/500348151","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6838","nonce":500348151},{"type":"v1","addr":"172.21.15.105:6839","nonce":500348151}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24527,"qdb_cluster":[24473,24466,24475,24527,24497]},"id":1}]} 2024-05-27T07:35:10.607 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:35:10 smithi173 ceph-mon[37000]: pgmap v311: 129 pgs: 12 active+clean+snaptrim, 31 active+clean+snaptrim_wait, 86 active+clean; 659 MiB data, 5.8 GiB used, 1.0 TiB / 1.0 TiB avail; 47 KiB/s rd, 1.7 MiB/s wr, 505 op/s 2024-05-27T07:35:10.608 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:10.612+0000 7fc2fa745700 1 -- 172.21.15.52:0/3474641898 >> [v2:172.21.15.52:6800/1822318384,v1:172.21.15.52:6801/1822318384] conn(0x7fc2e00b2af0 msgr2=0x7fc2e00b5000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:35:10.608 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:10.612+0000 7fc2fa745700 1 --2- 172.21.15.52:0/3474641898 >> [v2:172.21.15.52:6800/1822318384,v1:172.21.15.52:6801/1822318384] conn(0x7fc2e00b2af0 0x7fc2e00b5000 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7fc2e40098e0 tx=0x7fc2e4000bc0 comp rx=0 tx=0).stop 2024-05-27T07:35:10.608 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:10.612+0000 7fc2fa745700 1 -- 172.21.15.52:0/3474641898 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fc2f41117e0 msgr2=0x7fc2f4119dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:35:10.609 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:10.612+0000 7fc2fa745700 1 --2- 172.21.15.52:0/3474641898 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fc2f41117e0 0x7fc2f4119dd0 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7fc2ec007ee0 tx=0x7fc2ec00d3b0 comp rx=0 tx=0).stop 2024-05-27T07:35:10.609 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:10.613+0000 7fc2fa745700 1 -- 172.21.15.52:0/3474641898 shutdown_connections 2024-05-27T07:35:10.609 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:10.613+0000 7fc2fa745700 1 --2- 172.21.15.52:0/3474641898 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fc2f41117e0 0x7fc2f4119dd0 unknown :-1 s=CLOSED pgs=241 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:35:10.609 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:10.613+0000 7fc2fa745700 1 --2- 172.21.15.52:0/3474641898 >> [v2:172.21.15.52:6800/1822318384,v1:172.21.15.52:6801/1822318384] conn(0x7fc2e00b2af0 0x7fc2e00b5000 unknown :-1 s=CLOSED pgs=179 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:35:10.609 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:10.613+0000 7fc2fa745700 1 --2- 172.21.15.52:0/3474641898 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fc2f410adc0 0x7fc2f4120690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:35:10.609 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:10.613+0000 7fc2fa745700 1 --2- 172.21.15.52:0/3474641898 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fc2f410b710 0x7fc2f4119810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:35:10.609 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:10.613+0000 7fc2fa745700 1 -- 172.21.15.52:0/3474641898 >> 172.21.15.52:0/3474641898 conn(0x7fc2f4074d80 msgr2=0x7fc2f410e440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:35:10.611 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:10.615+0000 7fc2fa745700 1 -- 172.21.15.52:0/3474641898 shutdown_connections 2024-05-27T07:35:10.611 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:10.616+0000 7fc2fa745700 1 -- 172.21.15.52:0/3474641898 wait complete. 2024-05-27T07:35:10.614 INFO:teuthology.orchestra.run.smithi052.stderr:dumped fsmap epoch 107 2024-05-27T07:35:10.695 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:35:10 smithi105 ceph-mon[35977]: pgmap v311: 129 pgs: 12 active+clean+snaptrim, 31 active+clean+snaptrim_wait, 86 active+clean; 659 MiB data, 5.8 GiB used, 1.0 TiB / 1.0 TiB avail; 47 KiB/s rd, 1.7 MiB/s wr, 505 op/s 2024-05-27T07:35:11.484 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:35:11 smithi052 ceph-mon[31955]: from='client.? 172.21.15.52:0/3474641898' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-27T07:35:11.573 DEBUG:teuthology.orchestra.run.smithi052:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:38f9d900c83c63799fbdbe61acc9a11b0d3554a6 shell --fsid 73df2636-1bf8-11ef-bc9b-c7b262605968 -- ceph tell mds.1:4 damage ls 2024-05-27T07:35:11.607 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:35:11 smithi173 ceph-mon[37000]: from='client.? 172.21.15.52:0/3474641898' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-27T07:35:11.696 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:35:11 smithi105 ceph-mon[35977]: from='client.? 172.21.15.52:0/3474641898' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-27T07:35:12.039 INFO:teuthology.orchestra.run.smithi052.stderr:Inferring config /var/lib/ceph/73df2636-1bf8-11ef-bc9b-c7b262605968/mon.a/config 2024-05-27T07:35:12.476 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:35:12 smithi052 ceph-mon[31955]: pgmap v312: 129 pgs: 16 active+clean+snaptrim, 44 active+clean+snaptrim_wait, 69 active+clean; 665 MiB data, 5.8 GiB used, 1.0 TiB / 1.0 TiB avail; 44 KiB/s rd, 2.1 MiB/s wr, 467 op/s 2024-05-27T07:35:12.548 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:35:12.552+0000 7f96eeffd640 1 -- 172.21.15.52:0/3545053736 <== mds.0 v2:172.21.15.105:6838/500348151 1 ==== command_reply(tid 0: 0 ) ==== 8+0+25 (crc 0 0 0) 0x5625b27cf360 con 0x5625b24541c0 2024-05-27T07:35:12.548 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:35:12.552+0000 7f96eeffd640 10 client.26578 handle_command_reply: tid=0 2024-05-27T07:35:12.548 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:35:12.552+0000 7f971c87d740 1 client.26578 shutdown 2024-05-27T07:35:12.549 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:35:12.553+0000 7f971c87d740 1 -- 172.21.15.52:0/3545053736 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5625b24972f0 msgr2=0x5625b28737c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:35:12.549 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:35:12.553+0000 7f971c87d740 1 --2- 172.21.15.52:0/3545053736 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5625b24972f0 0x5625b28737c0 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7f970000bd30 tx=0x7f9700011650 comp rx=0 tx=0).stop 2024-05-27T07:35:12.549 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:35:12.553+0000 7f971c87d740 1 -- 172.21.15.52:0/3545053736 shutdown_connections 2024-05-27T07:35:12.549 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:35:12.553+0000 7f971c87d740 1 --2- 172.21.15.52:0/3545053736 >> [v2:172.21.15.105:6838/500348151,v1:172.21.15.105:6839/500348151] conn(0x5625b24541c0 0x5625b28f9a60 crc :-1 s=READY pgs=164 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:35:12.549 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:35:12.553+0000 7f971c87d740 1 --2- 172.21.15.52:0/3545053736 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x5625b24c01b0 0x5625b28e1a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:35:12.549 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:35:12.553+0000 7f971c87d740 1 --2- 172.21.15.52:0/3545053736 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5625b249f4d0 0x5625b2873d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:35:12.549 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:35:12.553+0000 7f971c87d740 1 --2- 172.21.15.52:0/3545053736 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5625b24972f0 0x5625b28737c0 unknown :-1 s=CLOSED pgs=202 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:35:12.549 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:35:12.553+0000 7f971c87d740 1 -- 172.21.15.52:0/3545053736 >> 172.21.15.52:0/3545053736 conn(0x5625b24b4300 msgr2=0x5625b286d8b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:35:12.549 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:35:12.553+0000 7f96eeffd640 0 client.26578 ms_handle_reset on v2:172.21.15.105:6838/500348151 2024-05-27T07:35:12.549 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:35:12.553+0000 7f971c87d740 1 -- 172.21.15.52:0/3545053736 shutdown_connections 2024-05-27T07:35:12.549 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:35:12.553+0000 7f971c87d740 1 -- 172.21.15.52:0/3545053736 wait complete. 2024-05-27T07:35:12.549 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:35:12.554+0000 7f971c87d740 20 client.26578 trim_cache size 0 max 16384 2024-05-27T07:35:12.550 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stdout:{ 2024-05-27T07:35:12.550 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stdout: "return_code": 0 2024-05-27T07:35:12.550 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stdout:} 2024-05-27T07:35:12.551 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:35:12.554+0000 7f97159cf640 1 -- 172.21.15.52:0/876878597 >> [v2:172.21.15.52:6800/1822318384,v1:172.21.15.52:6801/1822318384] conn(0x7f96f80b2aa0 msgr2=0x7f96f80b4fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:35:12.551 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:35:12.554+0000 7f97159cf640 1 --2- 172.21.15.52:0/876878597 >> [v2:172.21.15.52:6800/1822318384,v1:172.21.15.52:6801/1822318384] conn(0x7f96f80b2aa0 0x7f96f80b4fb0 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x5625b285e540 tx=0x7f97080093e0 comp rx=0 tx=0).stop 2024-05-27T07:35:12.551 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:35:12.555+0000 7f97159cf640 1 -- 172.21.15.52:0/876878597 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f9710108b70 msgr2=0x7f97101a1520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:35:12.552 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:35:12.555+0000 7f97159cf640 1 --2- 172.21.15.52:0/876878597 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f9710108b70 0x7f97101a1520 secure :-1 s=READY pgs=727 cs=0 l=1 rev1=1 crypto rx=0x7f970400b450 tx=0x7f970400b910 comp rx=0 tx=0).stop 2024-05-27T07:35:12.552 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:35:12.555+0000 7f97159cf640 1 -- 172.21.15.52:0/876878597 shutdown_connections 2024-05-27T07:35:12.552 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:35:12.555+0000 7f97159cf640 1 --2- 172.21.15.52:0/876878597 >> [v2:172.21.15.52:6800/1822318384,v1:172.21.15.52:6801/1822318384] conn(0x7f96f80b2aa0 0x7f96f80b4fb0 unknown :-1 s=CLOSED pgs=180 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:35:12.552 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:35:12.555+0000 7f97159cf640 1 --2- 172.21.15.52:0/876878597 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f9710108f80 0x7f97101a57e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:35:12.552 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:35:12.555+0000 7f97159cf640 1 --2- 172.21.15.52:0/876878597 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f9710108b70 0x7f97101a1520 unknown :-1 s=CLOSED pgs=727 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:35:12.552 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:35:12.555+0000 7f97159cf640 1 --2- 172.21.15.52:0/876878597 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f9710107410 0x7f97101a0fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:35:12.552 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:35:12.555+0000 7f97159cf640 1 -- 172.21.15.52:0/876878597 >> 172.21.15.52:0/876878597 conn(0x7f97100779d0 msgr2=0x7f97101053e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:35:12.552 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:35:12.556+0000 7f97159cf640 1 -- 172.21.15.52:0/876878597 shutdown_connections 2024-05-27T07:35:12.552 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:35:12.556+0000 7f97159cf640 1 -- 172.21.15.52:0/876878597 wait complete. 2024-05-27T07:35:12.607 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:35:12 smithi173 ceph-mon[37000]: pgmap v312: 129 pgs: 16 active+clean+snaptrim, 44 active+clean+snaptrim_wait, 69 active+clean; 665 MiB data, 5.8 GiB used, 1.0 TiB / 1.0 TiB avail; 44 KiB/s rd, 2.1 MiB/s wr, 467 op/s 2024-05-27T07:35:12.695 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:35:12 smithi105 ceph-mon[35977]: pgmap v312: 129 pgs: 16 active+clean+snaptrim, 44 active+clean+snaptrim_wait, 69 active+clean; 665 MiB data, 5.8 GiB used, 1.0 TiB / 1.0 TiB avail; 44 KiB/s rd, 2.1 MiB/s wr, 467 op/s 2024-05-27T07:35:14.474 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:35:14 smithi052 ceph-mon[31955]: pgmap v313: 129 pgs: 23 active+clean+snaptrim, 69 active+clean+snaptrim_wait, 37 active+clean; 660 MiB data, 5.8 GiB used, 1.0 TiB / 1.0 TiB avail; 43 KiB/s rd, 1.4 MiB/s wr, 366 op/s 2024-05-27T07:35:14.607 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:35:14 smithi173 ceph-mon[37000]: pgmap v313: 129 pgs: 23 active+clean+snaptrim, 69 active+clean+snaptrim_wait, 37 active+clean; 660 MiB data, 5.8 GiB used, 1.0 TiB / 1.0 TiB avail; 43 KiB/s rd, 1.4 MiB/s wr, 366 op/s 2024-05-27T07:35:14.695 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:35:14 smithi105 ceph-mon[35977]: pgmap v313: 129 pgs: 23 active+clean+snaptrim, 69 active+clean+snaptrim_wait, 37 active+clean; 660 MiB data, 5.8 GiB used, 1.0 TiB / 1.0 TiB avail; 43 KiB/s rd, 1.4 MiB/s wr, 366 op/s 2024-05-27T07:35:16.062 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:16.064+0000 7f1666881700 1 -- 172.21.15.52:0/2934434361 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f1660070030 msgr2=0x7f166011d490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:35:16.062 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:16.064+0000 7f1666881700 1 --2- 172.21.15.52:0/2934434361 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f1660070030 0x7f166011d490 secure :-1 s=READY pgs=728 cs=0 l=1 rev1=1 crypto rx=0x7f1650009990 tx=0x7f1650009ca0 comp rx=0 tx=0).stop 2024-05-27T07:35:16.066 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:16.069+0000 7f1666881700 1 -- 172.21.15.52:0/2934434361 shutdown_connections 2024-05-27T07:35:16.066 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:16.069+0000 7f1666881700 1 --2- 172.21.15.52:0/2934434361 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f166011d9d0 0x7f166011fdf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:35:16.066 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:16.069+0000 7f1666881700 1 --2- 172.21.15.52:0/2934434361 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f166006f6e0 0x7f166006faf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:35:16.066 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:16.069+0000 7f1666881700 1 --2- 172.21.15.52:0/2934434361 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f1660070030 0x7f166011d490 unknown :-1 s=CLOSED pgs=728 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:35:16.066 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:16.069+0000 7f1666881700 1 -- 172.21.15.52:0/2934434361 >> 172.21.15.52:0/2934434361 conn(0x7f166010ab50 msgr2=0x7f166010cf70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:35:16.070 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:16.075+0000 7f1666881700 1 -- 172.21.15.52:0/2934434361 shutdown_connections 2024-05-27T07:35:16.074 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:16.077+0000 7f1666881700 1 -- 172.21.15.52:0/2934434361 wait complete. 2024-05-27T07:35:16.074 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:16.078+0000 7f1666881700 1 Processor -- start 2024-05-27T07:35:16.079 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:16.079+0000 7f1666881700 1 -- start start 2024-05-27T07:35:16.079 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:16.079+0000 7f1666881700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f166006f6e0 0x7f16601af890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:35:16.079 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:16.079+0000 7f1666881700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f1660070030 0x7f16601afdd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:35:16.079 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:16.079+0000 7f1666881700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f166011d9d0 0x7f16601a9ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:35:16.079 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:16.079+0000 7f1666881700 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x7f16601232f0 con 0x7f166006f6e0 2024-05-27T07:35:16.079 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:16.080+0000 7f1666881700 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f1660123170 con 0x7f1660070030 2024-05-27T07:35:16.080 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:16.080+0000 7f1666881700 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f1660123470 con 0x7f166011d9d0 2024-05-27T07:35:16.080 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:16.080+0000 7f165f7fe700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f1660070030 0x7f16601afdd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:35:16.080 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:16.080+0000 7f165ffff700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f166006f6e0 0x7f16601af890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:35:16.080 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:16.080+0000 7f165ffff700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f166006f6e0 0x7f16601af890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.52:47128/0 (socket says 172.21.15.52:47128) 2024-05-27T07:35:16.080 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:16.080+0000 7f165ffff700 1 -- 172.21.15.52:0/1629379318 learned_addr learned my addr 172.21.15.52:0/1629379318 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-05-27T07:35:16.080 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:16.082+0000 7f165ffff700 1 -- 172.21.15.52:0/1629379318 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f166011d9d0 msgr2=0x7f16601a9ab0 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-05-27T07:35:16.080 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:16.082+0000 7f165ffff700 1 --2- 172.21.15.52:0/1629379318 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f166011d9d0 0x7f16601a9ab0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:35:16.080 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:16.082+0000 7f165ffff700 1 -- 172.21.15.52:0/1629379318 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f1660070030 msgr2=0x7f16601afdd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:35:16.080 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:16.082+0000 7f165ffff700 1 --2- 172.21.15.52:0/1629379318 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f1660070030 0x7f16601afdd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:35:16.080 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:16.083+0000 7f165ffff700 1 -- 172.21.15.52:0/1629379318 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f16601aa180 con 0x7f166006f6e0 2024-05-27T07:35:16.084 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:16.087+0000 7f165ffff700 1 --2- 172.21.15.52:0/1629379318 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f166006f6e0 0x7f16601af890 secure :-1 s=READY pgs=729 cs=0 l=1 rev1=1 crypto rx=0x7f165800ef10 tx=0x7f165800c3a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:35:16.085 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:16.090+0000 7f165d7fa700 1 -- 172.21.15.52:0/1629379318 <== mon.0 v2:172.21.15.52:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f165800cad0 con 0x7f166006f6e0 2024-05-27T07:35:16.085 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:16.090+0000 7f1666881700 1 -- 172.21.15.52:0/1629379318 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f16601aa410 con 0x7f166006f6e0 2024-05-27T07:35:16.086 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:16.090+0000 7f1666881700 1 -- 172.21.15.52:0/1629379318 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f166010ec40 con 0x7f166006f6e0 2024-05-27T07:35:16.086 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:16.090+0000 7f165d7fa700 1 -- 172.21.15.52:0/1629379318 <== mon.0 v2:172.21.15.52:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f1658019070 con 0x7f166006f6e0 2024-05-27T07:35:16.086 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:16.090+0000 7f165d7fa700 1 -- 172.21.15.52:0/1629379318 <== mon.0 v2:172.21.15.52:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f16580146e0 con 0x7f166006f6e0 2024-05-27T07:35:16.094 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:16.099+0000 7f1675296b80 1 Processor -- start 2024-05-27T07:35:16.094 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:16.099+0000 7f1675296b80 1 -- start start 2024-05-27T07:35:16.094 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:16.099+0000 7f1675296b80 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55e6cc489e40 0x55e6cc60eb30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:35:16.095 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:16.100+0000 7f1675296b80 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55e6cc60f070 0x55e6cc608be0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:35:16.095 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:16.100+0000 7f1675296b80 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55e6cc609120 0x55e6cc609530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:35:16.096 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:16.100+0000 7f1675296b80 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x55e6cc536d70 con 0x55e6cc609120 2024-05-27T07:35:16.096 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:16.100+0000 7f1664e1e700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55e6cc489e40 0x55e6cc60eb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:35:16.096 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:16.100+0000 7f1664e1e700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55e6cc489e40 0x55e6cc60eb30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.52:53312/0 (socket says 172.21.15.52:53312) 2024-05-27T07:35:16.096 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:16.100+0000 7f1664e1e700 1 -- 172.21.15.52:0/1206848616 learned_addr learned my addr 172.21.15.52:0/1206848616 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-05-27T07:35:16.096 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:16.100+0000 7f1675296b80 1 -- 172.21.15.52:0/1206848616 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x55e6cc526910 con 0x55e6cc489e40 2024-05-27T07:35:16.096 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:16.100+0000 7f1675296b80 1 -- 172.21.15.52:0/1206848616 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x55e6cc52a950 con 0x55e6cc60f070 2024-05-27T07:35:16.096 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:16.100+0000 7f165f7fe700 1 --2- 172.21.15.52:0/1206848616 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55e6cc609120 0x55e6cc609530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:35:16.096 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:16.101+0000 7f165f7fe700 1 -- 172.21.15.52:0/1206848616 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55e6cc60f070 msgr2=0x55e6cc608be0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-05-27T07:35:16.097 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:16.101+0000 7f165f7fe700 1 --2- 172.21.15.52:0/1206848616 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55e6cc60f070 0x55e6cc608be0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:35:16.097 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:16.101+0000 7f165f7fe700 1 -- 172.21.15.52:0/1206848616 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55e6cc489e40 msgr2=0x55e6cc60eb30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:35:16.097 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:16.101+0000 7f165f7fe700 1 --2- 172.21.15.52:0/1206848616 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55e6cc489e40 0x55e6cc60eb30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:35:16.098 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:16.101+0000 7f165f7fe700 1 -- 172.21.15.52:0/1206848616 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55e6cc4cffb0 con 0x55e6cc609120 2024-05-27T07:35:16.098 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:16.101+0000 7f165f7fe700 1 --2- 172.21.15.52:0/1206848616 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55e6cc609120 0x55e6cc609530 secure :-1 s=READY pgs=730 cs=0 l=1 rev1=1 crypto rx=0x7f16500098e0 tx=0x7f16500096a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bd0fbb0eb3478e75 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:35:16.098 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:16.101+0000 7f16467fc700 1 -- 172.21.15.52:0/1206848616 <== mon.0 v2:172.21.15.52:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f16500054d0 con 0x55e6cc609120 2024-05-27T07:35:16.098 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:16.102+0000 7f16467fc700 1 -- 172.21.15.52:0/1206848616 <== mon.0 v2:172.21.15.52:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f1650005670 con 0x55e6cc609120 2024-05-27T07:35:16.098 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:16.103+0000 7f1675296b80 1 -- 172.21.15.52:0/1206848616 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55e6cc609120 msgr2=0x55e6cc609530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:35:16.099 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:16.104+0000 7f1675296b80 1 --2- 172.21.15.52:0/1206848616 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55e6cc609120 0x55e6cc609530 secure :-1 s=READY pgs=730 cs=0 l=1 rev1=1 crypto rx=0x7f16500098e0 tx=0x7f16500096a0 comp rx=0 tx=0).stop 2024-05-27T07:35:16.102 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:16.106+0000 7f1675296b80 1 -- 172.21.15.52:0/1206848616 shutdown_connections 2024-05-27T07:35:16.102 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:16.106+0000 7f1675296b80 1 --2- 172.21.15.52:0/1206848616 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55e6cc609120 0x55e6cc609530 unknown :-1 s=CLOSED pgs=730 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:35:16.102 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:16.106+0000 7f1675296b80 1 --2- 172.21.15.52:0/1206848616 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55e6cc489e40 0x55e6cc60eb30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:35:16.103 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:16.106+0000 7f1675296b80 1 --2- 172.21.15.52:0/1206848616 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55e6cc60f070 0x55e6cc608be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:35:16.103 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:16.106+0000 7f1675296b80 1 -- 172.21.15.52:0/1206848616 >> 172.21.15.52:0/1206848616 conn(0x55e6cc48c770 msgr2=0x55e6cc603d90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:35:16.103 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:16.107+0000 7f165d7fa700 1 -- 172.21.15.52:0/1629379318 <== mon.0 v2:172.21.15.52:3300/0 4 ==== mgrmap(e 26) ==== 149783+0+0 (secure 0 0 0) 0x7f1658012050 con 0x7f166006f6e0 2024-05-27T07:35:16.103 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:16.108+0000 7f1675296b80 1 -- 172.21.15.52:0/1206848616 shutdown_connections 2024-05-27T07:35:16.103 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:16.108+0000 7f1675296b80 1 -- 172.21.15.52:0/1206848616 wait complete. 2024-05-27T07:35:16.104 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:16.108+0000 7f165d7fa700 1 --2- 172.21.15.52:0/1629379318 >> [v2:172.21.15.52:6800/1822318384,v1:172.21.15.52:6801/1822318384] conn(0x7f16480b2ac0 0x7f16480b4fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:35:16.104 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:16.109+0000 7f165d7fa700 1 -- 172.21.15.52:0/1629379318 <== mon.0 v2:172.21.15.52:3300/0 5 ==== osd_map(123..123 src has 1..123) ==== 9555+0+0 (secure 0 0 0) 0x7f16580e2a40 con 0x7f166006f6e0 2024-05-27T07:35:16.104 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:16.109+0000 7f1675296b80 1 -- start start 2024-05-27T07:35:16.105 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:16.110+0000 7f1675296b80 1 Processor -- start 2024-05-27T07:35:16.105 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:16.110+0000 7f1675296b80 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55e6cc489e40 0x55e6cc6127a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:35:16.105 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:16.110+0000 7f1675296b80 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55e6cc609120 0x55e6cc612ce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:35:16.106 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:16.111+0000 7f1675296b80 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55e6cc60f070 0x55e6cc613220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:35:16.107 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:16.111+0000 7f1675296b80 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x55e6cc42cfa0 con 0x55e6cc60f070 2024-05-27T07:35:16.107 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:16.111+0000 7f1675296b80 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x55e6cc48eda0 con 0x55e6cc489e40 2024-05-27T07:35:16.107 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:16.111+0000 7f165f7fe700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55e6cc609120 0x55e6cc612ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:35:16.107 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:16.111+0000 7f165f7fe700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55e6cc609120 0x55e6cc612ce0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.173:3300/0 says I am v2:172.21.15.52:44868/0 (socket says 172.21.15.52:44868) 2024-05-27T07:35:16.107 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:16.111+0000 7f165f7fe700 1 -- 172.21.15.52:0/1765444784 learned_addr learned my addr 172.21.15.52:0/1765444784 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-05-27T07:35:16.107 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:16.111+0000 7f1675296b80 1 -- 172.21.15.52:0/1765444784 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x55e6cc615ba0 con 0x55e6cc609120 2024-05-27T07:35:16.107 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:16.111+0000 7f1675296b80 10 client.0 _ll_register_callbacks cb 0x55e6cc4bd5b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T07:35:16.107 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:16.111+0000 7f1664e1e700 1 --2- 172.21.15.52:0/1629379318 >> [v2:172.21.15.52:6800/1822318384,v1:172.21.15.52:6801/1822318384] conn(0x7f16480b2ac0 0x7f16480b4fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:35:16.107 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:16.112+0000 7f1664e1e700 1 --2- 172.21.15.52:0/1765444784 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55e6cc60f070 0x55e6cc613220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:35:16.108 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:16.112+0000 7f1664e1e700 1 -- 172.21.15.52:0/1765444784 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55e6cc609120 msgr2=0x55e6cc612ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:35:16.108 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:16.112+0000 7f1664e1e700 1 --2- 172.21.15.52:0/1765444784 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55e6cc609120 0x55e6cc612ce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:35:16.109 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:16.112+0000 7f1664e1e700 1 -- 172.21.15.52:0/1765444784 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55e6cc489e40 msgr2=0x55e6cc6127a0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-05-27T07:35:16.109 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:16.112+0000 7f1664e1e700 1 --2- 172.21.15.52:0/1765444784 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55e6cc489e40 0x55e6cc6127a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:35:16.109 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:16.112+0000 7f1664e1e700 1 -- 172.21.15.52:0/1765444784 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55e6cc4768d0 con 0x55e6cc60f070 2024-05-27T07:35:16.109 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:16.113+0000 7f1664e1e700 1 --2- 172.21.15.52:0/1629379318 >> [v2:172.21.15.52:6800/1822318384,v1:172.21.15.52:6801/1822318384] conn(0x7f16480b2ac0 0x7f16480b4fd0 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7f1654010ec0 tx=0x7f165401b000 comp rx=0 tx=0).ready entity=mgr.15333 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:35:16.109 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:16.113+0000 7f1664e1e700 1 --2- 172.21.15.52:0/1765444784 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55e6cc60f070 0x55e6cc613220 secure :-1 s=READY pgs=731 cs=0 l=1 rev1=1 crypto rx=0x7f1654005c20 tx=0x7f1654010ac0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9d6945e45f348e8f server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:35:16.109 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:16.113+0000 7f163f7fe700 10 client.0 ms_handle_connect on v2:172.21.15.52:3300/0 2024-05-27T07:35:16.109 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:16.114+0000 7f163f7fe700 1 -- 172.21.15.52:0/1765444784 <== mon.0 v2:172.21.15.52:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1654013320 con 0x55e6cc60f070 2024-05-27T07:35:16.109 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:16.114+0000 7f163f7fe700 1 -- 172.21.15.52:0/1765444784 <== mon.0 v2:172.21.15.52:3300/0 2 ==== osd_map(123..123 src has 1..123) ==== 9555+0+0 (secure 0 0 0) 0x7f165400bad0 con 0x55e6cc60f070 2024-05-27T07:35:16.111 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:16.115+0000 7f1675296b80 1 -- 172.21.15.52:0/1765444784 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f1650005670 con 0x55e6cc60f070 2024-05-27T07:35:16.114 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:16.119+0000 7f163f7fe700 1 -- 172.21.15.52:0/1765444784 <== mon.0 v2:172.21.15.52:3300/0 3 ==== mon_get_version_reply(handle=1 version=108) ==== 24+0+0 (secure 0 0 0) 0x7f165400bcd0 con 0x55e6cc60f070 2024-05-27T07:35:16.120 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:16.123+0000 7f1675296b80 10 client.17079 fetch_fsmap learned FSMap version 108 2024-05-27T07:35:16.120 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:16.123+0000 7f1675296b80 1 -- 172.21.15.52:0/1765444784 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({config=0+,fsmap=108,monmap=4+}) -- 0x55e6cc4e0760 con 0x55e6cc60f070 2024-05-27T07:35:16.120 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:16.125+0000 7f163f7fe700 1 -- 172.21.15.52:0/1765444784 <== mon.0 v2:172.21.15.52:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f1650005670 con 0x55e6cc60f070 2024-05-27T07:35:16.120 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:16.125+0000 7f163f7fe700 1 -- 172.21.15.52:0/1765444784 <== mon.0 v2:172.21.15.52:3300/0 5 ==== fsmap(e 108) ==== 4203+0+0 (secure 0 0 0) 0x7f165401c070 con 0x55e6cc60f070 2024-05-27T07:35:16.124 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:16.129+0000 7f1675296b80 10 client.17079 fetch_fsmap finished waiting for FSMap version 108 2024-05-27T07:35:16.124 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:16.129+0000 7f1675296b80 10 client.17079 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.c 2024-05-27T07:35:16.124 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:16.129+0000 7f1675296b80 20 client.17079 populate_metadata read hostname 'smithi052' 2024-05-27T07:35:16.124 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:16.129+0000 7f1675296b80 1 --2- 172.21.15.52:0/1765444784 >> [v2:172.21.15.173:6836/1396363827,v1:172.21.15.173:6837/1396363827] conn(0x55e6cc622d50 0x55e6cc6251e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:35:16.124 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:16.129+0000 7f1675296b80 4 client.17079 mds_command: new command op to 24466 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-27T07:35:16.124 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:16.129+0000 7f1675296b80 1 -- 172.21.15.52:0/1765444784 --> [v2:172.21.15.173:6836/1396363827,v1:172.21.15.173:6837/1396363827] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55e6cc538cc0 con 0x55e6cc622d50 2024-05-27T07:35:16.130 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:16.135+0000 7f165ffff700 1 --2- 172.21.15.52:0/1765444784 >> [v2:172.21.15.173:6836/1396363827,v1:172.21.15.173:6837/1396363827] conn(0x55e6cc622d50 0x55e6cc6251e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:35:16.142 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:16.147+0000 7f163f7fe700 10 client.17079 ms_handle_connect on v2:172.21.15.173:6836/1396363827 2024-05-27T07:35:16.142 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:16.147+0000 7f165ffff700 1 --2- 172.21.15.52:0/1765444784 >> [v2:172.21.15.173:6836/1396363827,v1:172.21.15.173:6837/1396363827] conn(0x55e6cc622d50 0x55e6cc6251e0 crc :-1 s=READY pgs=46 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=42f04b1d4fe6d8a4 server_cookie=e324bad3f43c3869 in_seq=0 out_seq=0 2024-05-27T07:35:16.145 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:16.150+0000 7f163f7fe700 1 -- 172.21.15.52:0/1765444784 <== mds.4 v2:172.21.15.173:6836/1396363827 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24041 (crc 0 0 0) 0x55e6cc538cc0 con 0x55e6cc622d50 2024-05-27T07:35:16.145 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:16.150+0000 7f163f7fe700 10 client.17079 handle_command_reply: tid=0 2024-05-27T07:35:16.148 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:16.151+0000 7f1675296b80 1 client.17079 shutdown 2024-05-27T07:35:16.150 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:16.155+0000 7f1675296b80 1 -- 172.21.15.52:0/1765444784 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55e6cc60f070 msgr2=0x55e6cc613220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:35:16.150 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:16.155+0000 7f1675296b80 1 --2- 172.21.15.52:0/1765444784 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55e6cc60f070 0x55e6cc613220 secure :-1 s=READY pgs=731 cs=0 l=1 rev1=1 crypto rx=0x7f1654005c20 tx=0x7f1654010ac0 comp rx=0 tx=0).stop 2024-05-27T07:35:16.151 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:16.155+0000 7f1675296b80 1 -- 172.21.15.52:0/1765444784 shutdown_connections 2024-05-27T07:35:16.151 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:16.156+0000 7f1675296b80 1 --2- 172.21.15.52:0/1765444784 >> [v2:172.21.15.173:6836/1396363827,v1:172.21.15.173:6837/1396363827] conn(0x55e6cc622d50 0x55e6cc6251e0 crc :-1 s=READY pgs=46 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:35:16.151 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:16.156+0000 7f1675296b80 1 --2- 172.21.15.52:0/1765444784 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55e6cc60f070 0x55e6cc613220 unknown :-1 s=CLOSED pgs=731 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:35:16.151 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:16.156+0000 7f1675296b80 1 --2- 172.21.15.52:0/1765444784 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55e6cc489e40 0x55e6cc6127a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:35:16.151 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:16.156+0000 7f1675296b80 1 --2- 172.21.15.52:0/1765444784 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55e6cc609120 0x55e6cc612ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:35:16.151 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:16.156+0000 7f163f7fe700 0 client.17079 ms_handle_reset on v2:172.21.15.173:6836/1396363827 2024-05-27T07:35:16.151 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:16.156+0000 7f1675296b80 1 -- 172.21.15.52:0/1765444784 >> 172.21.15.52:0/1765444784 conn(0x55e6cc48c770 msgr2=0x55e6cc607bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:35:16.152 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:16.156+0000 7f1675296b80 1 -- 172.21.15.52:0/1765444784 shutdown_connections 2024-05-27T07:35:16.152 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:16.157+0000 7f1675296b80 1 -- 172.21.15.52:0/1765444784 wait complete. 2024-05-27T07:35:16.152 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:16.157+0000 7f1675296b80 20 client.17079 trim_cache size 0 max 16384 2024-05-27T07:35:16.184 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:16.188+0000 7f1675296b80 1 Processor -- start 2024-05-27T07:35:16.185 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:16.190+0000 7f1675296b80 1 -- start start 2024-05-27T07:35:16.186 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:16.191+0000 7f1675296b80 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55e6cc48c770 0x55e6cc608be0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:35:16.186 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:16.191+0000 7f1675296b80 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55e6cc489e40 0x55e6cc611700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:35:16.186 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:16.191+0000 7f1675296b80 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55e6cc609120 0x55e6cc613060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:35:16.186 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:16.191+0000 7f1675296b80 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x7f164c00f500 con 0x55e6cc48c770 2024-05-27T07:35:16.186 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:16.191+0000 7f1675296b80 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f164c00d930 con 0x55e6cc609120 2024-05-27T07:35:16.186 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:16.191+0000 7f1675296b80 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f164c00ee00 con 0x55e6cc489e40 2024-05-27T07:35:16.189 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:16.194+0000 7f165f7fe700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55e6cc489e40 0x55e6cc611700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:35:16.189 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:16.194+0000 7f1664e1e700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55e6cc609120 0x55e6cc613060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:35:16.189 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:16.194+0000 7f1664e1e700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55e6cc609120 0x55e6cc613060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.52:53326/0 (socket says 172.21.15.52:53326) 2024-05-27T07:35:16.189 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:16.194+0000 7f1664e1e700 1 -- 172.21.15.52:0/1544296895 learned_addr learned my addr 172.21.15.52:0/1544296895 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-05-27T07:35:16.190 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:16.194+0000 7f165f7fe700 1 -- 172.21.15.52:0/1544296895 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55e6cc609120 msgr2=0x55e6cc613060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:35:16.191 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:16.194+0000 7f165f7fe700 1 --2- 172.21.15.52:0/1544296895 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55e6cc609120 0x55e6cc613060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:35:16.191 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:16.194+0000 7f165f7fe700 1 -- 172.21.15.52:0/1544296895 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55e6cc48c770 msgr2=0x55e6cc608be0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-05-27T07:35:16.191 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:16.194+0000 7f165f7fe700 1 --2- 172.21.15.52:0/1544296895 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55e6cc48c770 0x55e6cc608be0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:35:16.191 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:16.194+0000 7f165f7fe700 1 -- 172.21.15.52:0/1544296895 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55e6cc615660 con 0x55e6cc489e40 2024-05-27T07:35:16.192 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:16.197+0000 7f165f7fe700 1 --2- 172.21.15.52:0/1544296895 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55e6cc489e40 0x55e6cc611700 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x55e6cc5ffae0 tx=0x7f1650009710 comp rx=0 tx=0).ready entity=mon.1 client_cookie=9d87266db1bdd2ed server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:35:16.193 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:16.198+0000 7f1645ffb700 1 -- 172.21.15.52:0/1544296895 <== mon.1 v2:172.21.15.173:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f16500056e0 con 0x55e6cc489e40 2024-05-27T07:35:16.193 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:16.198+0000 7f1645ffb700 1 -- 172.21.15.52:0/1544296895 <== mon.1 v2:172.21.15.173:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f165000bc70 con 0x55e6cc489e40 2024-05-27T07:35:16.193 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:16.198+0000 7f1645ffb700 1 -- 172.21.15.52:0/1544296895 <== mon.1 v2:172.21.15.173:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1650022bc0 con 0x55e6cc489e40 2024-05-27T07:35:16.194 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:16.199+0000 7f1675296b80 1 -- 172.21.15.52:0/1544296895 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55e6cc489e40 msgr2=0x55e6cc611700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:35:16.194 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:16.199+0000 7f1675296b80 1 --2- 172.21.15.52:0/1544296895 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55e6cc489e40 0x55e6cc611700 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x55e6cc5ffae0 tx=0x7f1650009710 comp rx=0 tx=0).stop 2024-05-27T07:35:16.195 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:16.200+0000 7f1675296b80 1 -- 172.21.15.52:0/1544296895 shutdown_connections 2024-05-27T07:35:16.196 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:16.200+0000 7f1675296b80 1 --2- 172.21.15.52:0/1544296895 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55e6cc609120 0x55e6cc613060 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:35:16.196 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:16.200+0000 7f1675296b80 1 --2- 172.21.15.52:0/1544296895 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55e6cc48c770 0x55e6cc608be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:35:16.196 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:16.200+0000 7f1675296b80 1 --2- 172.21.15.52:0/1544296895 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55e6cc489e40 0x55e6cc611700 unknown :-1 s=CLOSED pgs=203 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:35:16.196 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:16.200+0000 7f1675296b80 1 -- 172.21.15.52:0/1544296895 >> 172.21.15.52:0/1544296895 conn(0x55e6cc605480 msgr2=0x55e6cc60d6b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:35:16.202 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:16.207+0000 7f1675296b80 1 -- 172.21.15.52:0/1544296895 shutdown_connections 2024-05-27T07:35:16.203 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:16.207+0000 7f1675296b80 1 -- 172.21.15.52:0/1544296895 wait complete. 2024-05-27T07:35:16.204 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:16.208+0000 7f1675296b80 1 -- start start 2024-05-27T07:35:16.204 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:16.209+0000 7f1675296b80 1 Processor -- start 2024-05-27T07:35:16.204 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:16.209+0000 7f1675296b80 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55e6cc489e40 0x55e6cc61b270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:35:16.205 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:16.210+0000 7f1675296b80 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55e6cc48c770 0x55e6cc629fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:35:16.206 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:16.210+0000 7f1675296b80 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55e6cc609120 0x55e6cc62a4f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:35:16.206 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:16.210+0000 7f165f7fe700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55e6cc48c770 0x55e6cc629fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:35:16.206 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:16.210+0000 7f165f7fe700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55e6cc48c770 0x55e6cc629fb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.52:53336/0 (socket says 172.21.15.52:53336) 2024-05-27T07:35:16.206 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:16.210+0000 7f1664e1e700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55e6cc609120 0x55e6cc62a4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:35:16.206 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:16.210+0000 7f1664e1e700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55e6cc609120 0x55e6cc62a4f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.52:47162/0 (socket says 172.21.15.52:47162) 2024-05-27T07:35:16.206 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:16.211+0000 7f1664e1e700 1 -- 172.21.15.52:0/334275331 learned_addr learned my addr 172.21.15.52:0/334275331 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-05-27T07:35:16.206 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:16.211+0000 7f1675296b80 1 -- 172.21.15.52:0/334275331 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x7f164c00e730 con 0x55e6cc609120 2024-05-27T07:35:16.206 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:16.211+0000 7f1675296b80 1 -- 172.21.15.52:0/334275331 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f164c00e030 con 0x55e6cc48c770 2024-05-27T07:35:16.207 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:16.211+0000 7f1675296b80 1 -- 172.21.15.52:0/334275331 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x55e6cc609eb0 con 0x55e6cc489e40 2024-05-27T07:35:16.207 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:16.211+0000 7f1675296b80 10 client.0 _ll_register_callbacks cb 0x55e6cc4bd5b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T07:35:16.207 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:16.212+0000 7f165ffff700 1 --2- 172.21.15.52:0/334275331 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55e6cc489e40 0x55e6cc61b270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:35:16.210 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:16.214+0000 7f165ffff700 1 -- 172.21.15.52:0/334275331 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55e6cc48c770 msgr2=0x55e6cc629fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:35:16.210 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:16.214+0000 7f165ffff700 1 --2- 172.21.15.52:0/334275331 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55e6cc48c770 0x55e6cc629fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:35:16.210 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:16.214+0000 7f165ffff700 1 -- 172.21.15.52:0/334275331 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55e6cc609120 msgr2=0x55e6cc62a4f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:35:16.210 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:16.214+0000 7f165ffff700 1 --2- 172.21.15.52:0/334275331 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55e6cc609120 0x55e6cc62a4f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:35:16.210 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:16.214+0000 7f165ffff700 1 -- 172.21.15.52:0/334275331 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55e6cc62aa30 con 0x55e6cc489e40 2024-05-27T07:35:16.211 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:16.216+0000 7f165ffff700 1 --2- 172.21.15.52:0/334275331 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55e6cc489e40 0x55e6cc61b270 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7f16580dfd20 tx=0x7f1658009ac0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=2ae0404925b89f61 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:35:16.211 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:16.216+0000 7f163f7fe700 10 client.0 ms_handle_connect on v2:172.21.15.173:3300/0 2024-05-27T07:35:16.211 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:16.216+0000 7f163f7fe700 1 -- 172.21.15.52:0/334275331 <== mon.1 v2:172.21.15.173:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f16580e01b0 con 0x55e6cc489e40 2024-05-27T07:35:16.211 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:16.216+0000 7f1675296b80 1 -- 172.21.15.52:0/334275331 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f165000bc70 con 0x55e6cc489e40 2024-05-27T07:35:16.211 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:16.216+0000 7f163f7fe700 1 -- 172.21.15.52:0/334275331 <== mon.1 v2:172.21.15.173:3300/0 2 ==== osd_map(123..123 src has 1..123) ==== 9555+0+0 (secure 0 0 0) 0x7f16580a5ea0 con 0x55e6cc489e40 2024-05-27T07:35:16.215 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:16.220+0000 7f163f7fe700 1 -- 172.21.15.52:0/334275331 <== mon.1 v2:172.21.15.173:3300/0 3 ==== mon_get_version_reply(handle=1 version=108) ==== 24+0+0 (secure 0 0 0) 0x7f16580a9fb0 con 0x55e6cc489e40 2024-05-27T07:35:16.215 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:16.220+0000 7f1675296b80 10 client.26587 fetch_fsmap learned FSMap version 108 2024-05-27T07:35:16.215 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:16.220+0000 7f1675296b80 1 -- 172.21.15.52:0/334275331 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({config=0+,fsmap=108,monmap=4+}) -- 0x55e6cc68d490 con 0x55e6cc489e40 2024-05-27T07:35:16.218 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:16.222+0000 7f163f7fe700 1 -- 172.21.15.52:0/334275331 <== mon.1 v2:172.21.15.173:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f165000bc70 con 0x55e6cc489e40 2024-05-27T07:35:16.218 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:16.223+0000 7f163f7fe700 1 -- 172.21.15.52:0/334275331 <== mon.1 v2:172.21.15.173:3300/0 5 ==== fsmap(e 108) ==== 4203+0+0 (secure 0 0 0) 0x7f16580a9b10 con 0x55e6cc489e40 2024-05-27T07:35:16.218 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:16.223+0000 7f1675296b80 10 client.26587 fetch_fsmap finished waiting for FSMap version 108 2024-05-27T07:35:16.218 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:16.223+0000 7f1675296b80 10 client.26587 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.c 2024-05-27T07:35:16.219 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:16.223+0000 7f1675296b80 20 client.26587 populate_metadata read hostname 'smithi052' 2024-05-27T07:35:16.219 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:16.223+0000 7f1675296b80 1 --2- 172.21.15.52:0/334275331 >> [v2:172.21.15.173:6836/1396363827,v1:172.21.15.173:6837/1396363827] conn(0x55e6cc60f070 0x55e6cc68fb40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:35:16.219 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:16.223+0000 7f1675296b80 4 client.26587 mds_command: new command op to 24466 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-05-27T07:35:16.219 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:16.223+0000 7f1675296b80 1 -- 172.21.15.52:0/334275331 --> [v2:172.21.15.173:6836/1396363827,v1:172.21.15.173:6837/1396363827] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55e6cc5324a0 con 0x55e6cc60f070 2024-05-27T07:35:16.227 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:16.232+0000 7f165f7fe700 1 --2- 172.21.15.52:0/334275331 >> [v2:172.21.15.173:6836/1396363827,v1:172.21.15.173:6837/1396363827] conn(0x55e6cc60f070 0x55e6cc68fb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:35:16.230 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:16.234+0000 7f163f7fe700 10 client.26587 ms_handle_connect on v2:172.21.15.173:6836/1396363827 2024-05-27T07:35:16.230 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:16.234+0000 7f165f7fe700 1 --2- 172.21.15.52:0/334275331 >> [v2:172.21.15.173:6836/1396363827,v1:172.21.15.173:6837/1396363827] conn(0x55e6cc60f070 0x55e6cc68fb40 crc :-1 s=READY pgs=47 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=b0c3241e73e6ced2 server_cookie=44c881bcfc5a84c9 in_seq=0 out_seq=0 2024-05-27T07:35:16.230 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:16.235+0000 7f163f7fe700 1 -- 172.21.15.52:0/334275331 <== mds.4 v2:172.21.15.173:6836/1396363827 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55e6cc5324a0 con 0x55e6cc60f070 2024-05-27T07:35:16.230 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:16.235+0000 7f163f7fe700 10 client.26587 handle_command_reply: tid=0 2024-05-27T07:35:16.230 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:16.235+0000 7f1675296b80 1 client.26587 shutdown 2024-05-27T07:35:16.231 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:16.236+0000 7f1675296b80 1 -- 172.21.15.52:0/334275331 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55e6cc489e40 msgr2=0x55e6cc61b270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:35:16.231 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:16.236+0000 7f1675296b80 1 --2- 172.21.15.52:0/334275331 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55e6cc489e40 0x55e6cc61b270 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7f16580dfd20 tx=0x7f1658009ac0 comp rx=0 tx=0).stop 2024-05-27T07:35:16.232 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:16.236+0000 7f1675296b80 1 -- 172.21.15.52:0/334275331 shutdown_connections 2024-05-27T07:35:16.232 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:16.236+0000 7f1675296b80 1 --2- 172.21.15.52:0/334275331 >> [v2:172.21.15.173:6836/1396363827,v1:172.21.15.173:6837/1396363827] conn(0x55e6cc60f070 0x55e6cc68fb40 crc :-1 s=READY pgs=47 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:35:16.232 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:16.236+0000 7f1675296b80 1 --2- 172.21.15.52:0/334275331 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55e6cc609120 0x55e6cc62a4f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:35:16.232 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:16.236+0000 7f1675296b80 1 --2- 172.21.15.52:0/334275331 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55e6cc489e40 0x55e6cc61b270 unknown :-1 s=CLOSED pgs=204 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:35:16.232 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:16.236+0000 7f1675296b80 1 --2- 172.21.15.52:0/334275331 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55e6cc48c770 0x55e6cc629fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:35:16.232 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:16.236+0000 7f1675296b80 1 -- 172.21.15.52:0/334275331 >> 172.21.15.52:0/334275331 conn(0x55e6cc605480 msgr2=0x55e6cc60f7f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:35:16.232 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:16.236+0000 7f163f7fe700 0 client.26587 ms_handle_reset on v2:172.21.15.173:6836/1396363827 2024-05-27T07:35:16.232 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:16.236+0000 7f1675296b80 1 -- 172.21.15.52:0/334275331 shutdown_connections 2024-05-27T07:35:16.233 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:16.237+0000 7f1675296b80 1 -- 172.21.15.52:0/334275331 wait complete. 2024-05-27T07:35:16.233 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:16.237+0000 7f1675296b80 20 client.26587 trim_cache size 0 max 16384 2024-05-27T07:35:16.233 INFO:teuthology.orchestra.run.smithi052.stdout:[] 2024-05-27T07:35:16.235 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:16.239+0000 7f1666881700 1 -- 172.21.15.52:0/1629379318 >> [v2:172.21.15.52:6800/1822318384,v1:172.21.15.52:6801/1822318384] conn(0x7f16480b2ac0 msgr2=0x7f16480b4fd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:35:16.235 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:16.239+0000 7f1666881700 1 --2- 172.21.15.52:0/1629379318 >> [v2:172.21.15.52:6800/1822318384,v1:172.21.15.52:6801/1822318384] conn(0x7f16480b2ac0 0x7f16480b4fd0 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7f1654010ec0 tx=0x7f165401b000 comp rx=0 tx=0).stop 2024-05-27T07:35:16.235 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:16.239+0000 7f1666881700 1 -- 172.21.15.52:0/1629379318 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f166006f6e0 msgr2=0x7f16601af890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:35:16.235 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:16.239+0000 7f1666881700 1 --2- 172.21.15.52:0/1629379318 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f166006f6e0 0x7f16601af890 secure :-1 s=READY pgs=729 cs=0 l=1 rev1=1 crypto rx=0x7f165800ef10 tx=0x7f165800c3a0 comp rx=0 tx=0).stop 2024-05-27T07:35:16.235 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:16.239+0000 7f1666881700 1 -- 172.21.15.52:0/1629379318 shutdown_connections 2024-05-27T07:35:16.235 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:16.239+0000 7f1666881700 1 --2- 172.21.15.52:0/1629379318 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f166011d9d0 0x7f16601a9ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:35:16.235 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:16.239+0000 7f1666881700 1 --2- 172.21.15.52:0/1629379318 >> [v2:172.21.15.52:6800/1822318384,v1:172.21.15.52:6801/1822318384] conn(0x7f16480b2ac0 0x7f16480b4fd0 unknown :-1 s=CLOSED pgs=181 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:35:16.235 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:16.239+0000 7f1666881700 1 --2- 172.21.15.52:0/1629379318 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f166006f6e0 0x7f16601af890 unknown :-1 s=CLOSED pgs=729 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:35:16.235 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:16.239+0000 7f1666881700 1 --2- 172.21.15.52:0/1629379318 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f1660070030 0x7f16601afdd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:35:16.235 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:16.239+0000 7f1666881700 1 -- 172.21.15.52:0/1629379318 >> 172.21.15.52:0/1629379318 conn(0x7f166010ab50 msgr2=0x7f1660072880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:35:16.236 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:16.239+0000 7f1666881700 1 -- 172.21.15.52:0/1629379318 shutdown_connections 2024-05-27T07:35:16.236 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:16.240+0000 7f1666881700 1 -- 172.21.15.52:0/1629379318 wait complete. 2024-05-27T07:35:16.484 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:35:16 smithi052 ceph-mon[31955]: pgmap v314: 129 pgs: 20 active+clean+snaptrim, 62 active+clean+snaptrim_wait, 47 active+clean; 659 MiB data, 5.8 GiB used, 1.0 TiB / 1.0 TiB avail; 35 KiB/s rd, 1.4 MiB/s wr, 319 op/s 2024-05-27T07:35:16.485 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:35:16 smithi052 ceph-mon[31955]: fsmap cephfs:5 5 up:active 2024-05-27T07:35:16.607 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:35:16 smithi173 ceph-mon[37000]: pgmap v314: 129 pgs: 20 active+clean+snaptrim, 62 active+clean+snaptrim_wait, 47 active+clean; 659 MiB data, 5.8 GiB used, 1.0 TiB / 1.0 TiB avail; 35 KiB/s rd, 1.4 MiB/s wr, 319 op/s 2024-05-27T07:35:16.607 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:35:16 smithi173 ceph-mon[37000]: fsmap cephfs:5 5 up:active 2024-05-27T07:35:16.695 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:35:16 smithi105 ceph-mon[35977]: pgmap v314: 129 pgs: 20 active+clean+snaptrim, 62 active+clean+snaptrim_wait, 47 active+clean; 659 MiB data, 5.8 GiB used, 1.0 TiB / 1.0 TiB avail; 35 KiB/s rd, 1.4 MiB/s wr, 319 op/s 2024-05-27T07:35:16.695 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:35:16 smithi105 ceph-mon[35977]: fsmap cephfs:5 5 up:active 2024-05-27T07:35:17.053 DEBUG:teuthology.orchestra.run.smithi052:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:38f9d900c83c63799fbdbe61acc9a11b0d3554a6 shell --fsid 73df2636-1bf8-11ef-bc9b-c7b262605968 -- ceph tell mds.1:3 damage ls 2024-05-27T07:35:17.589 INFO:teuthology.orchestra.run.smithi052.stderr:Inferring config /var/lib/ceph/73df2636-1bf8-11ef-bc9b-c7b262605968/mon.a/config 2024-05-27T07:35:18.607 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:35:18 smithi173 ceph-mon[37000]: pgmap v315: 129 pgs: 20 active+clean+snaptrim, 62 active+clean+snaptrim_wait, 47 active+clean; 646 MiB data, 5.9 GiB used, 1.0 TiB / 1.0 TiB avail; 24 KiB/s rd, 1.2 MiB/s wr, 278 op/s 2024-05-27T07:35:18.695 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:35:18 smithi105 ceph-mon[35977]: pgmap v315: 129 pgs: 20 active+clean+snaptrim, 62 active+clean+snaptrim_wait, 47 active+clean; 646 MiB data, 5.9 GiB used, 1.0 TiB / 1.0 TiB avail; 24 KiB/s rd, 1.2 MiB/s wr, 278 op/s 2024-05-27T07:35:18.734 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:35:18 smithi052 ceph-mon[31955]: pgmap v315: 129 pgs: 20 active+clean+snaptrim, 62 active+clean+snaptrim_wait, 47 active+clean; 646 MiB data, 5.9 GiB used, 1.0 TiB / 1.0 TiB avail; 24 KiB/s rd, 1.2 MiB/s wr, 278 op/s 2024-05-27T07:35:19.797 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:19.801+0000 7f9ab36c7700 1 -- 172.21.15.52:0/4286176443 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f9aac0764a0 msgr2=0x7f9aac114e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:35:19.797 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:19.801+0000 7f9ab36c7700 1 --2- 172.21.15.52:0/4286176443 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f9aac0764a0 0x7f9aac114e10 secure :-1 s=READY pgs=732 cs=0 l=1 rev1=1 crypto rx=0x7f9aa80098f0 tx=0x7f9aa8009c00 comp rx=0 tx=0).stop 2024-05-27T07:35:19.800 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:19.803+0000 7f9ab36c7700 1 -- 172.21.15.52:0/4286176443 shutdown_connections 2024-05-27T07:35:19.800 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:19.803+0000 7f9ab36c7700 1 --2- 172.21.15.52:0/4286176443 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f9aac115350 0x7f9aac117770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:35:19.800 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:19.803+0000 7f9ab36c7700 1 --2- 172.21.15.52:0/4286176443 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f9aac075b50 0x7f9aac075f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:35:19.800 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:19.803+0000 7f9ab36c7700 1 --2- 172.21.15.52:0/4286176443 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f9aac0764a0 0x7f9aac114e10 unknown :-1 s=CLOSED pgs=732 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:35:19.800 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:19.803+0000 7f9ab36c7700 1 -- 172.21.15.52:0/4286176443 >> 172.21.15.52:0/4286176443 conn(0x7f9aac0ffe60 msgr2=0x7f9aac1022b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:35:19.804 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:19.807+0000 7f9ab36c7700 1 -- 172.21.15.52:0/4286176443 shutdown_connections 2024-05-27T07:35:19.804 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:19.809+0000 7f9ab36c7700 1 -- 172.21.15.52:0/4286176443 wait complete. 2024-05-27T07:35:19.805 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:19.810+0000 7f9ab36c7700 1 Processor -- start 2024-05-27T07:35:19.808 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:19.813+0000 7f9ab36c7700 1 -- start start 2024-05-27T07:35:19.809 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:19.814+0000 7f9ab36c7700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f9aac075b50 0x7f9aac072b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:35:19.809 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:19.814+0000 7f9ab36c7700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f9aac0764a0 0x7f9aac073060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:35:19.810 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:19.814+0000 7f9ab36c7700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f9aac115350 0x7f9aac0735c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:35:19.810 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:19.814+0000 7f9ab0c62700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f9aac0764a0 0x7f9aac073060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:35:19.810 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:19.814+0000 7f9ab0c62700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f9aac0764a0 0x7f9aac073060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.52:47184/0 (socket says 172.21.15.52:47184) 2024-05-27T07:35:19.810 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:19.814+0000 7f9ab0c62700 1 -- 172.21.15.52:0/2475906691 learned_addr learned my addr 172.21.15.52:0/2475906691 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-05-27T07:35:19.810 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:19.814+0000 7f9ab36c7700 1 -- 172.21.15.52:0/2475906691 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x7f9aac10ae00 con 0x7f9aac0764a0 2024-05-27T07:35:19.810 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:19.814+0000 7f9ab36c7700 1 -- 172.21.15.52:0/2475906691 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f9aac10ac80 con 0x7f9aac115350 2024-05-27T07:35:19.810 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:19.814+0000 7f9ab36c7700 1 -- 172.21.15.52:0/2475906691 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f9aac10af80 con 0x7f9aac075b50 2024-05-27T07:35:19.810 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:19.815+0000 7f9ab0c62700 1 -- 172.21.15.52:0/2475906691 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f9aac075b50 msgr2=0x7f9aac072b20 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-05-27T07:35:19.811 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:19.815+0000 7f9ab0c62700 1 --2- 172.21.15.52:0/2475906691 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f9aac075b50 0x7f9aac072b20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:35:19.811 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:19.815+0000 7f9ab0c62700 1 -- 172.21.15.52:0/2475906691 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f9aac115350 msgr2=0x7f9aac0735c0 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-05-27T07:35:19.812 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:19.815+0000 7f9ab0c62700 1 --2- 172.21.15.52:0/2475906691 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f9aac115350 0x7f9aac0735c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:35:19.812 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:19.815+0000 7f9ab0c62700 1 -- 172.21.15.52:0/2475906691 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9aac1148f0 con 0x7f9aac0764a0 2024-05-27T07:35:19.812 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:19.817+0000 7f9ab0c62700 1 --2- 172.21.15.52:0/2475906691 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f9aac0764a0 0x7f9aac073060 secure :-1 s=READY pgs=733 cs=0 l=1 rev1=1 crypto rx=0x7f9aa8009810 tx=0x7f9aa8005690 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:35:19.817 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:19.822+0000 7f9aa27fc700 1 -- 172.21.15.52:0/2475906691 <== mon.0 v2:172.21.15.52:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9aa800ba30 con 0x7f9aac0764a0 2024-05-27T07:35:19.817 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:19.822+0000 7f9aa27fc700 1 -- 172.21.15.52:0/2475906691 <== mon.0 v2:172.21.15.52:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f9aa800bbd0 con 0x7f9aac0764a0 2024-05-27T07:35:19.817 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:19.822+0000 7f9aa27fc700 1 -- 172.21.15.52:0/2475906691 <== mon.0 v2:172.21.15.52:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9aa8017960 con 0x7f9aac0764a0 2024-05-27T07:35:19.817 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:19.822+0000 7f9ab36c7700 1 -- 172.21.15.52:0/2475906691 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9aac114b80 con 0x7f9aac0764a0 2024-05-27T07:35:19.818 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:19.822+0000 7f9ab36c7700 1 -- 172.21.15.52:0/2475906691 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9aac115010 con 0x7f9aac0764a0 2024-05-27T07:35:19.829 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:19.833+0000 7f9aa27fc700 1 -- 172.21.15.52:0/2475906691 <== mon.0 v2:172.21.15.52:3300/0 4 ==== mgrmap(e 26) ==== 149783+0+0 (secure 0 0 0) 0x7f9aa8003680 con 0x7f9aac0764a0 2024-05-27T07:35:19.833 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:19.836+0000 7f9ac20dcb80 1 Processor -- start 2024-05-27T07:35:19.833 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:19.837+0000 7f9aa27fc700 1 --2- 172.21.15.52:0/2475906691 >> [v2:172.21.15.52:6800/1822318384,v1:172.21.15.52:6801/1822318384] conn(0x7f9a980b2c00 0x7f9a980b5110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:35:19.833 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:19.837+0000 7f9aa27fc700 1 -- 172.21.15.52:0/2475906691 <== mon.0 v2:172.21.15.52:3300/0 5 ==== osd_map(123..123 src has 1..123) ==== 9555+0+0 (secure 0 0 0) 0x7f9aa80e39d0 con 0x7f9aac0764a0 2024-05-27T07:35:19.833 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:19.838+0000 7f9ab1c64700 1 --2- 172.21.15.52:0/2475906691 >> [v2:172.21.15.52:6800/1822318384,v1:172.21.15.52:6801/1822318384] conn(0x7f9a980b2c00 0x7f9a980b5110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:35:19.835 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:19.840+0000 7f9ab1c64700 1 --2- 172.21.15.52:0/2475906691 >> [v2:172.21.15.52:6800/1822318384,v1:172.21.15.52:6801/1822318384] conn(0x7f9a980b2c00 0x7f9a980b5110 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7f9aac073b00 tx=0x7f9a9c006c60 comp rx=0 tx=0).ready entity=mgr.15333 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:35:19.839 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:19.841+0000 7f9ac20dcb80 1 -- start start 2024-05-27T07:35:19.839 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:19.841+0000 7f9ac20dcb80 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5564854d0e40 0x556485655b30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:35:19.839 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:19.841+0000 7f9ac20dcb80 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x556485656070 0x55648564fbe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:35:19.839 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:19.841+0000 7f9ac20dcb80 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x556485650120 0x556485650530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:35:19.839 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:19.841+0000 7f9ac20dcb80 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x55648557dd70 con 0x556485656070 2024-05-27T07:35:19.840 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:19.841+0000 7f9ac20dcb80 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x55648556d910 con 0x556485650120 2024-05-27T07:35:19.840 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:19.841+0000 7f9ac20dcb80 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x556485571950 con 0x5564854d0e40 2024-05-27T07:35:19.840 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:19.842+0000 7f9ab1463700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5564854d0e40 0x556485655b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:35:19.840 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:19.843+0000 7f9ab0c62700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x556485656070 0x55648564fbe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:35:19.840 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:19.843+0000 7f9ab1463700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5564854d0e40 0x556485655b30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.173:3300/0 says I am v2:172.21.15.52:44896/0 (socket says 172.21.15.52:44896) 2024-05-27T07:35:19.840 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:19.843+0000 7f9ab1463700 1 -- 172.21.15.52:0/3957227594 learned_addr learned my addr 172.21.15.52:0/3957227594 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-05-27T07:35:19.848 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:19.846+0000 7f9ab1c64700 1 --2- 172.21.15.52:0/3957227594 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x556485650120 0x556485650530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:35:19.848 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:19.847+0000 7f9ab0c62700 1 -- 172.21.15.52:0/3957227594 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5564854d0e40 msgr2=0x556485655b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:35:19.848 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:19.847+0000 7f9ab0c62700 1 --2- 172.21.15.52:0/3957227594 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5564854d0e40 0x556485655b30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:35:19.848 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:19.847+0000 7f9ab0c62700 1 -- 172.21.15.52:0/3957227594 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x556485650120 msgr2=0x556485650530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:35:19.848 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:19.847+0000 7f9ab0c62700 1 --2- 172.21.15.52:0/3957227594 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x556485650120 0x556485650530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:35:19.848 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:19.847+0000 7f9ab0c62700 1 -- 172.21.15.52:0/3957227594 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x556485516fb0 con 0x556485656070 2024-05-27T07:35:19.848 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:19.848+0000 7f9ab0c62700 1 --2- 172.21.15.52:0/3957227594 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x556485656070 0x55648564fbe0 secure :-1 s=READY pgs=734 cs=0 l=1 rev1=1 crypto rx=0x7f9aa80e0cb0 tx=0x7f9aa80aa040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2dad1afcc8dafcf server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:35:19.849 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:19.851+0000 7f9a977fe700 1 -- 172.21.15.52:0/3957227594 <== mon.0 v2:172.21.15.52:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9aa8026030 con 0x556485656070 2024-05-27T07:35:19.849 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:19.852+0000 7f9a977fe700 1 -- 172.21.15.52:0/3957227594 <== mon.0 v2:172.21.15.52:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f9aa80aafd0 con 0x556485656070 2024-05-27T07:35:19.849 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:19.852+0000 7f9a977fe700 1 -- 172.21.15.52:0/3957227594 <== mon.0 v2:172.21.15.52:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9aa80aab30 con 0x556485656070 2024-05-27T07:35:19.849 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:19.853+0000 7f9ac20dcb80 1 -- 172.21.15.52:0/3957227594 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x556485656070 msgr2=0x55648564fbe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:35:19.849 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:19.853+0000 7f9ac20dcb80 1 --2- 172.21.15.52:0/3957227594 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x556485656070 0x55648564fbe0 secure :-1 s=READY pgs=734 cs=0 l=1 rev1=1 crypto rx=0x7f9aa80e0cb0 tx=0x7f9aa80aa040 comp rx=0 tx=0).stop 2024-05-27T07:35:19.850 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:19.855+0000 7f9ac20dcb80 1 -- 172.21.15.52:0/3957227594 shutdown_connections 2024-05-27T07:35:19.850 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:19.855+0000 7f9ac20dcb80 1 --2- 172.21.15.52:0/3957227594 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x556485650120 0x556485650530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:35:19.850 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:19.855+0000 7f9ac20dcb80 1 --2- 172.21.15.52:0/3957227594 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5564854d0e40 0x556485655b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:35:19.850 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:19.855+0000 7f9ac20dcb80 1 --2- 172.21.15.52:0/3957227594 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x556485656070 0x55648564fbe0 unknown :-1 s=CLOSED pgs=734 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:35:19.850 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:19.855+0000 7f9ac20dcb80 1 -- 172.21.15.52:0/3957227594 >> 172.21.15.52:0/3957227594 conn(0x5564854d3770 msgr2=0x55648564ad90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:35:19.851 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:19.855+0000 7f9ac20dcb80 1 -- 172.21.15.52:0/3957227594 shutdown_connections 2024-05-27T07:35:19.851 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:19.856+0000 7f9ac20dcb80 1 -- 172.21.15.52:0/3957227594 wait complete. 2024-05-27T07:35:19.852 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:19.856+0000 7f9ac20dcb80 1 -- start start 2024-05-27T07:35:19.852 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:19.857+0000 7f9ac20dcb80 1 Processor -- start 2024-05-27T07:35:19.854 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:19.858+0000 7f9ac20dcb80 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x5564854d0e40 0x556485659900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:35:19.854 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:19.858+0000 7f9ac20dcb80 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x556485650120 0x556485659e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:35:19.854 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:19.858+0000 7f9ac20dcb80 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x556485656070 0x55648565a380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:35:19.854 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:19.858+0000 7f9ac20dcb80 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x556485473fa0 con 0x5564854d0e40 2024-05-27T07:35:19.854 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:19.858+0000 7f9ac20dcb80 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x5564854d5da0 con 0x556485656070 2024-05-27T07:35:19.854 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:19.858+0000 7f9ac20dcb80 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x55648557dd70 con 0x556485650120 2024-05-27T07:35:19.854 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:19.859+0000 7f9ac20dcb80 10 client.0 _ll_register_callbacks cb 0x5564855045b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T07:35:19.855 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:19.859+0000 7f9ab1463700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x5564854d0e40 0x556485659900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:35:19.855 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:19.859+0000 7f9ab1463700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x5564854d0e40 0x556485659900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.52:47210/0 (socket says 172.21.15.52:47210) 2024-05-27T07:35:19.855 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:19.859+0000 7f9ab1463700 1 -- 172.21.15.52:0/4012721223 learned_addr learned my addr 172.21.15.52:0/4012721223 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-05-27T07:35:19.855 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:19.860+0000 7f9ab1463700 1 -- 172.21.15.52:0/4012721223 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x556485650120 msgr2=0x556485659e40 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-05-27T07:35:19.856 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:19.860+0000 7f9ab1463700 1 --2- 172.21.15.52:0/4012721223 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x556485650120 0x556485659e40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:35:19.856 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:19.860+0000 7f9ab1463700 1 -- 172.21.15.52:0/4012721223 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x556485656070 msgr2=0x55648565a380 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-05-27T07:35:19.856 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:19.860+0000 7f9ab1463700 1 --2- 172.21.15.52:0/4012721223 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x556485656070 0x55648565a380 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:35:19.856 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:19.860+0000 7f9ab1463700 1 -- 172.21.15.52:0/4012721223 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5564854bd8d0 con 0x5564854d0e40 2024-05-27T07:35:19.856 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:19.860+0000 7f9ab1463700 1 --2- 172.21.15.52:0/4012721223 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x5564854d0e40 0x556485659900 secure :-1 s=READY pgs=735 cs=0 l=1 rev1=1 crypto rx=0x7f9aa4022cc0 tx=0x7f9aa4020490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=978e3174926b502d server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:35:19.856 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:19.861+0000 7f9a94ff9700 10 client.0 ms_handle_connect on v2:172.21.15.52:3300/0 2024-05-27T07:35:19.857 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:19.862+0000 7f9a94ff9700 1 -- 172.21.15.52:0/4012721223 <== mon.0 v2:172.21.15.52:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9aa402a070 con 0x5564854d0e40 2024-05-27T07:35:19.857 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:19.862+0000 7f9a94ff9700 1 -- 172.21.15.52:0/4012721223 <== mon.0 v2:172.21.15.52:3300/0 2 ==== osd_map(123..123 src has 1..123) ==== 9555+0+0 (secure 0 0 0) 0x7f9aa4003a00 con 0x5564854d0e40 2024-05-27T07:35:19.857 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:19.862+0000 7f9ac20dcb80 1 -- 172.21.15.52:0/4012721223 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f9aa80aafd0 con 0x5564854d0e40 2024-05-27T07:35:19.860 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:19.865+0000 7f9a94ff9700 1 -- 172.21.15.52:0/4012721223 <== mon.0 v2:172.21.15.52:3300/0 3 ==== mon_get_version_reply(handle=1 version=108) ==== 24+0+0 (secure 0 0 0) 0x7f9aa4033050 con 0x5564854d0e40 2024-05-27T07:35:19.860 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:19.865+0000 7f9ac20dcb80 10 client.17094 fetch_fsmap learned FSMap version 108 2024-05-27T07:35:19.860 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:19.865+0000 7f9ac20dcb80 1 -- 172.21.15.52:0/4012721223 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({config=0+,fsmap=108,monmap=4+}) -- 0x556485527760 con 0x5564854d0e40 2024-05-27T07:35:19.861 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:19.866+0000 7f9a94ff9700 1 -- 172.21.15.52:0/4012721223 <== mon.0 v2:172.21.15.52:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f9aa4003c00 con 0x5564854d0e40 2024-05-27T07:35:19.861 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:19.866+0000 7f9a94ff9700 1 -- 172.21.15.52:0/4012721223 <== mon.0 v2:172.21.15.52:3300/0 5 ==== fsmap(e 108) ==== 4203+0+0 (secure 0 0 0) 0x7f9aa40043f0 con 0x5564854d0e40 2024-05-27T07:35:19.862 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:19.866+0000 7f9ac20dcb80 10 client.17094 fetch_fsmap finished waiting for FSMap version 108 2024-05-27T07:35:19.862 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:19.866+0000 7f9ac20dcb80 10 client.17094 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.e 2024-05-27T07:35:19.862 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:19.866+0000 7f9ac20dcb80 20 client.17094 populate_metadata read hostname 'smithi052' 2024-05-27T07:35:19.863 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:19.866+0000 7f9ac20dcb80 1 --2- 172.21.15.52:0/4012721223 >> [v2:172.21.15.105:6834/3298498384,v1:172.21.15.105:6835/3298498384] conn(0x556485669900 0x55648566bdb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:35:19.863 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:19.866+0000 7f9ac20dcb80 4 client.17094 mds_command: new command op to 24473 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-27T07:35:19.863 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:19.866+0000 7f9ac20dcb80 1 -- 172.21.15.52:0/4012721223 --> [v2:172.21.15.105:6834/3298498384,v1:172.21.15.105:6835/3298498384] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x7f9aa80aafd0 con 0x556485669900 2024-05-27T07:35:19.869 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:19.872+0000 7f9ab0c62700 1 --2- 172.21.15.52:0/4012721223 >> [v2:172.21.15.105:6834/3298498384,v1:172.21.15.105:6835/3298498384] conn(0x556485669900 0x55648566bdb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:35:19.874 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:19.878+0000 7f9ab0c62700 1 --2- 172.21.15.52:0/4012721223 >> [v2:172.21.15.105:6834/3298498384,v1:172.21.15.105:6835/3298498384] conn(0x556485669900 0x55648566bdb0 crc :-1 s=READY pgs=45 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=57563423db9ade97 server_cookie=cdafa5c6569de25a in_seq=0 out_seq=0 2024-05-27T07:35:19.874 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:19.879+0000 7f9a94ff9700 10 client.17094 ms_handle_connect on v2:172.21.15.105:6834/3298498384 2024-05-27T07:35:19.878 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:19.883+0000 7f9a94ff9700 1 -- 172.21.15.52:0/4012721223 <== mds.3 v2:172.21.15.105:6834/3298498384 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24041 (crc 0 0 0) 0x7f9aa80aafd0 con 0x556485669900 2024-05-27T07:35:19.879 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:19.883+0000 7f9a94ff9700 10 client.17094 handle_command_reply: tid=0 2024-05-27T07:35:19.879 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:19.883+0000 7f9ac20dcb80 1 client.17094 shutdown 2024-05-27T07:35:19.882 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:19.886+0000 7f9ac20dcb80 1 -- 172.21.15.52:0/4012721223 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x5564854d0e40 msgr2=0x556485659900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:35:19.882 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:19.886+0000 7f9ac20dcb80 1 --2- 172.21.15.52:0/4012721223 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x5564854d0e40 0x556485659900 secure :-1 s=READY pgs=735 cs=0 l=1 rev1=1 crypto rx=0x7f9aa4022cc0 tx=0x7f9aa4020490 comp rx=0 tx=0).stop 2024-05-27T07:35:19.884 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:19.889+0000 7f9ac20dcb80 1 -- 172.21.15.52:0/4012721223 shutdown_connections 2024-05-27T07:35:19.884 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:19.889+0000 7f9ac20dcb80 1 --2- 172.21.15.52:0/4012721223 >> [v2:172.21.15.105:6834/3298498384,v1:172.21.15.105:6835/3298498384] conn(0x556485669900 0x55648566bdb0 crc :-1 s=READY pgs=45 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:35:19.884 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:19.889+0000 7f9ac20dcb80 1 --2- 172.21.15.52:0/4012721223 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x556485656070 0x55648565a380 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:35:19.884 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:19.889+0000 7f9ac20dcb80 1 --2- 172.21.15.52:0/4012721223 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x5564854d0e40 0x556485659900 unknown :-1 s=CLOSED pgs=735 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:35:19.884 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:19.889+0000 7f9ac20dcb80 1 --2- 172.21.15.52:0/4012721223 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x556485650120 0x556485659e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:35:19.884 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:19.889+0000 7f9ac20dcb80 1 -- 172.21.15.52:0/4012721223 >> 172.21.15.52:0/4012721223 conn(0x5564854d3770 msgr2=0x55648564ebd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:35:19.884 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:19.889+0000 7f9a94ff9700 0 client.17094 ms_handle_reset on v2:172.21.15.105:6834/3298498384 2024-05-27T07:35:19.884 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:19.889+0000 7f9ac20dcb80 1 -- 172.21.15.52:0/4012721223 shutdown_connections 2024-05-27T07:35:19.885 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:19.889+0000 7f9ac20dcb80 1 -- 172.21.15.52:0/4012721223 wait complete. 2024-05-27T07:35:19.885 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:19.889+0000 7f9ac20dcb80 20 client.17094 trim_cache size 0 max 16384 2024-05-27T07:35:19.918 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:19.920+0000 7f9ac20dcb80 1 Processor -- start 2024-05-27T07:35:19.918 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:19.921+0000 7f9ac20dcb80 1 -- start start 2024-05-27T07:35:19.918 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:19.921+0000 7f9ac20dcb80 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x5564854d0e40 0x55648564fbe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:35:19.918 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:19.921+0000 7f9ac20dcb80 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x556485650120 0x556485658880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:35:19.918 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:19.922+0000 7f9ac20dcb80 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x556485656070 0x55648565a1c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:35:19.918 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:19.922+0000 7f9ac20dcb80 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x7f9a9000f4a0 con 0x5564854d0e40 2024-05-27T07:35:19.918 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:19.922+0000 7f9ab0c62700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x556485650120 0x556485658880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:35:19.918 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:19.922+0000 7f9ab0c62700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x556485650120 0x556485658880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.173:3300/0 says I am v2:172.21.15.52:44924/0 (socket says 172.21.15.52:44924) 2024-05-27T07:35:19.918 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:19.922+0000 7f9ab0c62700 1 -- 172.21.15.52:0/3256656615 learned_addr learned my addr 172.21.15.52:0/3256656615 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-05-27T07:35:19.918 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:19.922+0000 7f9ac20dcb80 1 -- 172.21.15.52:0/3256656615 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f9a9000d8d0 con 0x556485656070 2024-05-27T07:35:19.919 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:19.923+0000 7f9ac20dcb80 1 -- 172.21.15.52:0/3256656615 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f9a9000eda0 con 0x556485650120 2024-05-27T07:35:19.919 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:19.923+0000 7f9ab1c64700 1 --2- 172.21.15.52:0/3256656615 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x556485656070 0x55648565a1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:35:19.923 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:19.928+0000 7f9ab1c64700 1 -- 172.21.15.52:0/3256656615 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x556485650120 msgr2=0x556485658880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:35:19.923 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:19.928+0000 7f9ab1c64700 1 --2- 172.21.15.52:0/3256656615 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x556485650120 0x556485658880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:35:19.924 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:19.928+0000 7f9ab1c64700 1 -- 172.21.15.52:0/3256656615 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x5564854d0e40 msgr2=0x55648564fbe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:35:19.924 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:19.928+0000 7f9ab1c64700 1 --2- 172.21.15.52:0/3256656615 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x5564854d0e40 0x55648564fbe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:35:19.924 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:19.928+0000 7f9ab1c64700 1 -- 172.21.15.52:0/3256656615 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x556485642950 con 0x556485656070 2024-05-27T07:35:19.925 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:19.930+0000 7f9ab1c64700 1 --2- 172.21.15.52:0/3256656615 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x556485656070 0x55648565a1c0 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7f9a9c009db0 tx=0x7f9a9c012000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f6b532e2096e2a86 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:35:19.925 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:19.931+0000 7f9a96ffd700 1 -- 172.21.15.52:0/3256656615 <== mon.2 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9a9c012920 con 0x556485656070 2024-05-27T07:35:19.927 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:19.931+0000 7f9a96ffd700 1 -- 172.21.15.52:0/3256656615 <== mon.2 v2:172.21.15.105:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f9a9c024070 con 0x556485656070 2024-05-27T07:35:19.927 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:19.931+0000 7f9a96ffd700 1 -- 172.21.15.52:0/3256656615 <== mon.2 v2:172.21.15.105:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9a9c010360 con 0x556485656070 2024-05-27T07:35:19.928 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:19.932+0000 7f9ac20dcb80 1 -- 172.21.15.52:0/3256656615 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x556485656070 msgr2=0x55648565a1c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:35:19.928 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:19.932+0000 7f9ac20dcb80 1 --2- 172.21.15.52:0/3256656615 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x556485656070 0x55648565a1c0 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7f9a9c009db0 tx=0x7f9a9c012000 comp rx=0 tx=0).stop 2024-05-27T07:35:19.931 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:19.936+0000 7f9ac20dcb80 1 -- 172.21.15.52:0/3256656615 shutdown_connections 2024-05-27T07:35:19.931 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:19.936+0000 7f9ac20dcb80 1 --2- 172.21.15.52:0/3256656615 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x556485656070 0x55648565a1c0 unknown :-1 s=CLOSED pgs=242 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:35:19.931 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:19.936+0000 7f9ac20dcb80 1 --2- 172.21.15.52:0/3256656615 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x5564854d0e40 0x55648564fbe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:35:19.931 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:19.936+0000 7f9ac20dcb80 1 --2- 172.21.15.52:0/3256656615 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x556485650120 0x556485658880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:35:19.931 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:19.936+0000 7f9ac20dcb80 1 -- 172.21.15.52:0/3256656615 >> 172.21.15.52:0/3256656615 conn(0x5564854d3770 msgr2=0x5564856444f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:35:19.937 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:19.940+0000 7f9ac20dcb80 1 -- 172.21.15.52:0/3256656615 shutdown_connections 2024-05-27T07:35:19.937 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:19.941+0000 7f9ac20dcb80 1 -- 172.21.15.52:0/3256656615 wait complete. 2024-05-27T07:35:19.937 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:19.941+0000 7f9ac20dcb80 1 -- start start 2024-05-27T07:35:19.937 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:19.942+0000 7f9ac20dcb80 1 Processor -- start 2024-05-27T07:35:19.938 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:19.943+0000 7f9ac20dcb80 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x5564854d0e40 0x556485671cb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:35:19.938 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:19.943+0000 7f9ac20dcb80 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x556485650120 0x5564856619f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:35:19.938 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:19.943+0000 7f9ab1463700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x5564854d0e40 0x556485671cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:35:19.938 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:19.943+0000 7f9ac20dcb80 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x556485656070 0x556485661f30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:35:19.939 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:19.943+0000 7f9ab1463700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x5564854d0e40 0x556485671cb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.52:47232/0 (socket says 172.21.15.52:47232) 2024-05-27T07:35:19.939 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:19.943+0000 7f9ab1463700 1 -- 172.21.15.52:0/691552761 learned_addr learned my addr 172.21.15.52:0/691552761 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-05-27T07:35:19.939 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:19.943+0000 7f9ac20dcb80 1 -- 172.21.15.52:0/691552761 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x7f9a9000e6d0 con 0x5564854d0e40 2024-05-27T07:35:19.939 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:19.943+0000 7f9ac20dcb80 1 -- 172.21.15.52:0/691552761 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f9a9000dfd0 con 0x556485650120 2024-05-27T07:35:19.939 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:19.943+0000 7f9ac20dcb80 1 -- 172.21.15.52:0/691552761 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x5564856625c0 con 0x556485656070 2024-05-27T07:35:19.939 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:19.944+0000 7f9ab0c62700 1 --2- 172.21.15.52:0/691552761 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x556485650120 0x5564856619f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:35:19.940 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:19.944+0000 7f9ac20dcb80 10 client.0 _ll_register_callbacks cb 0x5564855045b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T07:35:19.940 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:19.944+0000 7f9ab1c64700 1 --2- 172.21.15.52:0/691552761 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x556485656070 0x556485661f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:35:19.943 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:19.947+0000 7f9ab0c62700 1 -- 172.21.15.52:0/691552761 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x556485656070 msgr2=0x556485661f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:35:19.943 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:19.947+0000 7f9ab0c62700 1 --2- 172.21.15.52:0/691552761 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x556485656070 0x556485661f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:35:19.943 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:19.947+0000 7f9ab0c62700 1 -- 172.21.15.52:0/691552761 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x5564854d0e40 msgr2=0x556485671cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:35:19.943 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:19.947+0000 7f9ab0c62700 1 --2- 172.21.15.52:0/691552761 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x5564854d0e40 0x556485671cb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:35:19.943 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:19.947+0000 7f9ab0c62700 1 -- 172.21.15.52:0/691552761 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5564856705a0 con 0x556485650120 2024-05-27T07:35:19.943 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:19.948+0000 7f9ab1c64700 1 --2- 172.21.15.52:0/691552761 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x556485656070 0x556485661f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-27T07:35:19.944 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:19.948+0000 7f9ab0c62700 1 --2- 172.21.15.52:0/691552761 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x556485650120 0x5564856619f0 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7f9aa8005d90 tx=0x7f9aa80b1300 comp rx=0 tx=0).ready entity=mon.2 client_cookie=2ce7f174ab9b1a89 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:35:19.944 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:19.948+0000 7f9a94ff9700 10 client.0 ms_handle_connect on v2:172.21.15.105:3300/0 2024-05-27T07:35:19.947 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:19.952+0000 7f9a94ff9700 1 -- 172.21.15.52:0/691552761 <== mon.2 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9aa80c4070 con 0x556485650120 2024-05-27T07:35:19.947 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:19.952+0000 7f9a94ff9700 1 -- 172.21.15.52:0/691552761 <== mon.2 v2:172.21.15.105:3300/0 2 ==== osd_map(123..123 src has 1..123) ==== 9555+0+0 (secure 0 0 0) 0x7f9aa80aab30 con 0x556485650120 2024-05-27T07:35:19.947 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:19.952+0000 7f9ac20dcb80 1 -- 172.21.15.52:0/691552761 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f9a9c024070 con 0x556485650120 2024-05-27T07:35:19.948 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:19.953+0000 7f9a94ff9700 1 -- 172.21.15.52:0/691552761 <== mon.2 v2:172.21.15.105:3300/0 3 ==== mon_get_version_reply(handle=1 version=108) ==== 24+0+0 (secure 0 0 0) 0x7f9aa8026040 con 0x556485650120 2024-05-27T07:35:19.952 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:19.955+0000 7f9ac20dcb80 10 client.26762 fetch_fsmap learned FSMap version 108 2024-05-27T07:35:19.952 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:19.955+0000 7f9ac20dcb80 1 -- 172.21.15.52:0/691552761 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,fsmap=108,monmap=4+}) -- 0x5564856d3690 con 0x556485650120 2024-05-27T07:35:19.955 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:19.960+0000 7f9a94ff9700 1 -- 172.21.15.52:0/691552761 <== mon.2 v2:172.21.15.105:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f9aa80aafd0 con 0x556485650120 2024-05-27T07:35:19.955 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:19.960+0000 7f9a94ff9700 1 -- 172.21.15.52:0/691552761 <== mon.2 v2:172.21.15.105:3300/0 5 ==== fsmap(e 108) ==== 4203+0+0 (secure 0 0 0) 0x7f9aa80b3210 con 0x556485650120 2024-05-27T07:35:19.956 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:19.960+0000 7f9ac20dcb80 10 client.26762 fetch_fsmap finished waiting for FSMap version 108 2024-05-27T07:35:19.956 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:19.960+0000 7f9ac20dcb80 10 client.26762 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.e 2024-05-27T07:35:19.956 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:19.960+0000 7f9ac20dcb80 20 client.26762 populate_metadata read hostname 'smithi052' 2024-05-27T07:35:19.956 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:19.960+0000 7f9ac20dcb80 1 --2- 172.21.15.52:0/691552761 >> [v2:172.21.15.105:6834/3298498384,v1:172.21.15.105:6835/3298498384] conn(0x5564856d3ac0 0x5564856d5ee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:35:19.956 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:19.960+0000 7f9ac20dcb80 4 client.26762 mds_command: new command op to 24473 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-05-27T07:35:19.956 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:19.960+0000 7f9ac20dcb80 1 -- 172.21.15.52:0/691552761 --> [v2:172.21.15.105:6834/3298498384,v1:172.21.15.105:6835/3298498384] -- command(tid 0: {"prefix": "damage ls"}) -- 0x7f9a9c024070 con 0x5564856d3ac0 2024-05-27T07:35:19.956 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:19.961+0000 7f9ab1463700 1 --2- 172.21.15.52:0/691552761 >> [v2:172.21.15.105:6834/3298498384,v1:172.21.15.105:6835/3298498384] conn(0x5564856d3ac0 0x5564856d5ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:35:19.959 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:19.963+0000 7f9ab1463700 1 --2- 172.21.15.52:0/691552761 >> [v2:172.21.15.105:6834/3298498384,v1:172.21.15.105:6835/3298498384] conn(0x5564856d3ac0 0x5564856d5ee0 crc :-1 s=READY pgs=46 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=27360e467eeedf66 server_cookie=4d2ba6b6a4eeea02 in_seq=0 out_seq=0 2024-05-27T07:35:19.959 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:19.963+0000 7f9a94ff9700 10 client.26762 ms_handle_connect on v2:172.21.15.105:6834/3298498384 2024-05-27T07:35:19.959 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:19.963+0000 7f9a94ff9700 1 -- 172.21.15.52:0/691552761 <== mds.3 v2:172.21.15.105:6834/3298498384 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x7f9a9c024070 con 0x5564856d3ac0 2024-05-27T07:35:19.959 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:19.963+0000 7f9a94ff9700 10 client.26762 handle_command_reply: tid=0 2024-05-27T07:35:19.959 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:19.963+0000 7f9ac20dcb80 1 client.26762 shutdown 2024-05-27T07:35:19.959 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:19.963+0000 7f9ac20dcb80 1 -- 172.21.15.52:0/691552761 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x556485650120 msgr2=0x5564856619f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:35:19.959 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:19.963+0000 7f9ac20dcb80 1 --2- 172.21.15.52:0/691552761 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x556485650120 0x5564856619f0 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7f9aa8005d90 tx=0x7f9aa80b1300 comp rx=0 tx=0).stop 2024-05-27T07:35:19.960 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:19.963+0000 7f9ac20dcb80 1 -- 172.21.15.52:0/691552761 shutdown_connections 2024-05-27T07:35:19.960 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:19.963+0000 7f9ac20dcb80 1 --2- 172.21.15.52:0/691552761 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x556485656070 0x556485661f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:35:19.960 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:19.963+0000 7f9ac20dcb80 1 --2- 172.21.15.52:0/691552761 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x5564854d0e40 0x556485671cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:35:19.960 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:19.963+0000 7f9ac20dcb80 1 --2- 172.21.15.52:0/691552761 >> [v2:172.21.15.105:6834/3298498384,v1:172.21.15.105:6835/3298498384] conn(0x5564856d3ac0 0x5564856d5ee0 crc :-1 s=READY pgs=46 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:35:19.960 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:19.963+0000 7f9ac20dcb80 1 --2- 172.21.15.52:0/691552761 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x556485650120 0x5564856619f0 unknown :-1 s=CLOSED pgs=243 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:35:19.960 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:19.963+0000 7f9ac20dcb80 1 -- 172.21.15.52:0/691552761 >> 172.21.15.52:0/691552761 conn(0x5564854d3770 msgr2=0x5564856444f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:35:19.960 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:19.963+0000 7f9a94ff9700 0 client.26762 ms_handle_reset on v2:172.21.15.105:6834/3298498384 2024-05-27T07:35:19.960 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:19.964+0000 7f9ac20dcb80 1 -- 172.21.15.52:0/691552761 shutdown_connections 2024-05-27T07:35:19.960 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:19.964+0000 7f9ac20dcb80 1 -- 172.21.15.52:0/691552761 wait complete. 2024-05-27T07:35:19.961 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:19.964+0000 7f9ac20dcb80 20 client.26762 trim_cache size 0 max 16384 2024-05-27T07:35:19.961 INFO:teuthology.orchestra.run.smithi052.stdout:[] 2024-05-27T07:35:19.962 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:19.967+0000 7f9ab36c7700 1 -- 172.21.15.52:0/2475906691 >> [v2:172.21.15.52:6800/1822318384,v1:172.21.15.52:6801/1822318384] conn(0x7f9a980b2c00 msgr2=0x7f9a980b5110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:35:19.962 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:19.967+0000 7f9ab36c7700 1 --2- 172.21.15.52:0/2475906691 >> [v2:172.21.15.52:6800/1822318384,v1:172.21.15.52:6801/1822318384] conn(0x7f9a980b2c00 0x7f9a980b5110 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7f9aac073b00 tx=0x7f9a9c006c60 comp rx=0 tx=0).stop 2024-05-27T07:35:19.962 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:19.967+0000 7f9ab36c7700 1 -- 172.21.15.52:0/2475906691 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f9aac0764a0 msgr2=0x7f9aac073060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:35:19.962 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:19.967+0000 7f9ab36c7700 1 --2- 172.21.15.52:0/2475906691 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f9aac0764a0 0x7f9aac073060 secure :-1 s=READY pgs=733 cs=0 l=1 rev1=1 crypto rx=0x7f9aa8009810 tx=0x7f9aa8005690 comp rx=0 tx=0).stop 2024-05-27T07:35:19.963 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:19.967+0000 7f9ab36c7700 1 -- 172.21.15.52:0/2475906691 shutdown_connections 2024-05-27T07:35:19.963 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:19.967+0000 7f9ab36c7700 1 --2- 172.21.15.52:0/2475906691 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f9aac115350 0x7f9aac0735c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:35:19.963 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:19.967+0000 7f9ab36c7700 1 --2- 172.21.15.52:0/2475906691 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f9aac075b50 0x7f9aac072b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:35:19.963 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:19.967+0000 7f9ab36c7700 1 --2- 172.21.15.52:0/2475906691 >> [v2:172.21.15.52:6800/1822318384,v1:172.21.15.52:6801/1822318384] conn(0x7f9a980b2c00 0x7f9a980b5110 unknown :-1 s=CLOSED pgs=182 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:35:19.963 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:19.967+0000 7f9ab36c7700 1 --2- 172.21.15.52:0/2475906691 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f9aac0764a0 0x7f9aac073060 unknown :-1 s=CLOSED pgs=733 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:35:19.963 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:19.967+0000 7f9ab36c7700 1 -- 172.21.15.52:0/2475906691 >> 172.21.15.52:0/2475906691 conn(0x7f9aac0ffe60 msgr2=0x7f9aac100850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:35:19.964 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:19.967+0000 7f9ab36c7700 1 -- 172.21.15.52:0/2475906691 shutdown_connections 2024-05-27T07:35:19.964 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:19.967+0000 7f9ab36c7700 1 -- 172.21.15.52:0/2475906691 wait complete. 2024-05-27T07:35:20.607 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:35:20 smithi173 ceph-mon[37000]: pgmap v316: 129 pgs: 20 active+clean+snaptrim, 62 active+clean+snaptrim_wait, 47 active+clean; 646 MiB data, 6.0 GiB used, 1.0 TiB / 1.0 TiB avail; 29 KiB/s rd, 1.1 MiB/s wr, 261 op/s 2024-05-27T07:35:20.665 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:35:20 smithi052 ceph-mon[31955]: pgmap v316: 129 pgs: 20 active+clean+snaptrim, 62 active+clean+snaptrim_wait, 47 active+clean; 646 MiB data, 6.0 GiB used, 1.0 TiB / 1.0 TiB avail; 29 KiB/s rd, 1.1 MiB/s wr, 261 op/s 2024-05-27T07:35:20.695 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:35:20 smithi105 ceph-mon[35977]: pgmap v316: 129 pgs: 20 active+clean+snaptrim, 62 active+clean+snaptrim_wait, 47 active+clean; 646 MiB data, 6.0 GiB used, 1.0 TiB / 1.0 TiB avail; 29 KiB/s rd, 1.1 MiB/s wr, 261 op/s 2024-05-27T07:35:20.714 DEBUG:teuthology.orchestra.run.smithi052:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:38f9d900c83c63799fbdbe61acc9a11b0d3554a6 shell --fsid 73df2636-1bf8-11ef-bc9b-c7b262605968 -- ceph tell mds.1:2 damage ls 2024-05-27T07:35:21.248 INFO:teuthology.orchestra.run.smithi052.stderr:Inferring config /var/lib/ceph/73df2636-1bf8-11ef-bc9b-c7b262605968/mon.a/config 2024-05-27T07:35:22.607 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:35:22 smithi173 ceph-mon[37000]: pgmap v317: 129 pgs: 20 active+clean+snaptrim, 58 active+clean+snaptrim_wait, 51 active+clean; 649 MiB data, 6.1 GiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 1.4 MiB/s wr, 215 op/s 2024-05-27T07:35:22.695 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:35:22 smithi105 ceph-mon[35977]: pgmap v317: 129 pgs: 20 active+clean+snaptrim, 58 active+clean+snaptrim_wait, 51 active+clean; 649 MiB data, 6.1 GiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 1.4 MiB/s wr, 215 op/s 2024-05-27T07:35:22.734 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:35:22 smithi052 ceph-mon[31955]: pgmap v317: 129 pgs: 20 active+clean+snaptrim, 58 active+clean+snaptrim_wait, 51 active+clean; 649 MiB data, 6.1 GiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 1.4 MiB/s wr, 215 op/s 2024-05-27T07:35:23.355 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:23.359+0000 7f0755637700 1 -- 172.21.15.52:0/2865592493 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f075010b270 msgr2=0x7f075011d3e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:35:23.355 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:23.359+0000 7f0755637700 1 --2- 172.21.15.52:0/2865592493 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f075010b270 0x7f075011d3e0 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7f0748009f90 tx=0x7f074800b330 comp rx=0 tx=0).stop 2024-05-27T07:35:23.359 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:23.364+0000 7f0755637700 1 -- 172.21.15.52:0/2865592493 shutdown_connections 2024-05-27T07:35:23.359 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:23.364+0000 7f0755637700 1 --2- 172.21.15.52:0/2865592493 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f075011d920 0x7f075011fd40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:35:23.359 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:23.364+0000 7f0755637700 1 --2- 172.21.15.52:0/2865592493 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f075010a920 0x7f075010ad30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:35:23.360 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:23.364+0000 7f0755637700 1 --2- 172.21.15.52:0/2865592493 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f075010b270 0x7f075011d3e0 unknown :-1 s=CLOSED pgs=244 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:35:23.360 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:23.364+0000 7f0755637700 1 -- 172.21.15.52:0/2865592493 >> 172.21.15.52:0/2865592493 conn(0x7f075006f4c0 msgr2=0x7f075006f900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:35:23.361 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:23.365+0000 7f0755637700 1 -- 172.21.15.52:0/2865592493 shutdown_connections 2024-05-27T07:35:23.361 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:23.365+0000 7f0755637700 1 -- 172.21.15.52:0/2865592493 wait complete. 2024-05-27T07:35:23.365 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:23.366+0000 7f0755637700 1 Processor -- start 2024-05-27T07:35:23.368 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:23.372+0000 7f0755637700 1 -- start start 2024-05-27T07:35:23.368 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:23.372+0000 7f0755637700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f075010a920 0x7f075011d220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:35:23.368 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:23.373+0000 7f0755637700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f075010b270 0x7f0750119360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:35:23.368 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:23.373+0000 7f0755637700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f075011d920 0x7f07501198a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:35:23.368 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:23.373+0000 7f0755637700 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x7f0750115400 con 0x7f075010b270 2024-05-27T07:35:23.368 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:23.373+0000 7f0755637700 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f0750115280 con 0x7f075010a920 2024-05-27T07:35:23.369 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:23.373+0000 7f0755637700 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f0750115580 con 0x7f075011d920 2024-05-27T07:35:23.373 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:23.378+0000 7f074e7fc700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f075010b270 0x7f0750119360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:35:23.373 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:23.378+0000 7f074effd700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f075010a920 0x7f075011d220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:35:23.373 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:23.378+0000 7f074e7fc700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f075010b270 0x7f0750119360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.52:57304/0 (socket says 172.21.15.52:57304) 2024-05-27T07:35:23.373 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:23.378+0000 7f074e7fc700 1 -- 172.21.15.52:0/2999777564 learned_addr learned my addr 172.21.15.52:0/2999777564 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-05-27T07:35:23.375 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:23.380+0000 7f074f7fe700 1 --2- 172.21.15.52:0/2999777564 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f075011d920 0x7f07501198a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:35:23.383 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:23.385+0000 7f074e7fc700 1 -- 172.21.15.52:0/2999777564 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f075011d920 msgr2=0x7f07501198a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:35:23.383 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:23.385+0000 7f074e7fc700 1 --2- 172.21.15.52:0/2999777564 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f075011d920 0x7f07501198a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:35:23.383 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:23.385+0000 7f074e7fc700 1 -- 172.21.15.52:0/2999777564 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f075010a920 msgr2=0x7f075011d220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:35:23.383 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:23.385+0000 7f074e7fc700 1 --2- 172.21.15.52:0/2999777564 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f075010a920 0x7f075011d220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:35:23.383 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:23.385+0000 7f074e7fc700 1 -- 172.21.15.52:0/2999777564 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f075011a080 con 0x7f075010b270 2024-05-27T07:35:23.386 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:23.391+0000 7f074e7fc700 1 --2- 172.21.15.52:0/2999777564 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f075010b270 0x7f0750119360 secure :-1 s=READY pgs=736 cs=0 l=1 rev1=1 crypto rx=0x7f0748000f80 tx=0x7f074800cdf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:35:23.392 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:23.395+0000 7f0737fff700 1 -- 172.21.15.52:0/2999777564 <== mon.0 v2:172.21.15.52:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0748021020 con 0x7f075010b270 2024-05-27T07:35:23.392 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:23.395+0000 7f0737fff700 1 -- 172.21.15.52:0/2999777564 <== mon.0 v2:172.21.15.52:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f0748012d00 con 0x7f075010b270 2024-05-27T07:35:23.392 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:23.395+0000 7f0737fff700 1 -- 172.21.15.52:0/2999777564 <== mon.0 v2:172.21.15.52:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f074801da20 con 0x7f075010b270 2024-05-27T07:35:23.392 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:23.395+0000 7f0755637700 1 -- 172.21.15.52:0/2999777564 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f07501cf8c0 con 0x7f075010b270 2024-05-27T07:35:23.392 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:23.396+0000 7f0755637700 1 -- 172.21.15.52:0/2999777564 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f07501cfd50 con 0x7f075010b270 2024-05-27T07:35:23.399 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:23.404+0000 7f0737fff700 1 -- 172.21.15.52:0/2999777564 <== mon.0 v2:172.21.15.52:3300/0 4 ==== mgrmap(e 26) ==== 149783+0+0 (secure 0 0 0) 0x7f0748024020 con 0x7f075010b270 2024-05-27T07:35:23.402 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:23.407+0000 7f076404cb80 1 Processor -- start 2024-05-27T07:35:23.402 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:23.407+0000 7f076404cb80 1 -- start start 2024-05-27T07:35:23.404 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:23.408+0000 7f076404cb80 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5603096c7e40 0x56030984cb30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:35:23.404 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:23.408+0000 7f076404cb80 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x56030984d070 0x560309846be0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:35:23.404 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:23.408+0000 7f076404cb80 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x560309847120 0x560309847530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:35:23.404 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:23.408+0000 7f076404cb80 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x560309774d70 con 0x560309847120 2024-05-27T07:35:23.404 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:23.408+0000 7f076404cb80 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x560309764910 con 0x5603096c7e40 2024-05-27T07:35:23.405 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:23.408+0000 7f076404cb80 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x560309768950 con 0x56030984d070 2024-05-27T07:35:23.405 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:23.408+0000 7f074effd700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x56030984d070 0x560309846be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:35:23.405 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:23.410+0000 7f074effd700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x56030984d070 0x560309846be0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.173:3300/0 says I am v2:172.21.15.52:42068/0 (socket says 172.21.15.52:42068) 2024-05-27T07:35:23.405 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:23.410+0000 7f074effd700 1 -- 172.21.15.52:0/2631891534 learned_addr learned my addr 172.21.15.52:0/2631891534 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-05-27T07:35:23.405 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:23.410+0000 7f074f7fe700 1 --2- 172.21.15.52:0/2631891534 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5603096c7e40 0x56030984cb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:35:23.406 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:23.410+0000 7f074effd700 1 -- 172.21.15.52:0/2631891534 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5603096c7e40 msgr2=0x56030984cb30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:35:23.406 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:23.410+0000 7f074effd700 1 --2- 172.21.15.52:0/2631891534 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5603096c7e40 0x56030984cb30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:35:23.406 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:23.410+0000 7f074effd700 1 -- 172.21.15.52:0/2631891534 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x560309847120 msgr2=0x560309847530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:35:23.406 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:23.410+0000 7f074effd700 1 --2- 172.21.15.52:0/2631891534 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x560309847120 0x560309847530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:35:23.406 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:23.411+0000 7f074effd700 1 -- 172.21.15.52:0/2631891534 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56030970dfb0 con 0x56030984d070 2024-05-27T07:35:23.406 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:23.411+0000 7f074effd700 1 --2- 172.21.15.52:0/2631891534 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x56030984d070 0x560309846be0 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7f074001ad70 tx=0x7f074000e630 comp rx=0 tx=0).ready entity=mon.1 client_cookie=27c0f7af3c86d1a9 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:35:23.407 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:23.411+0000 7f07357fa700 1 -- 172.21.15.52:0/2631891534 <== mon.1 v2:172.21.15.173:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f074000ed90 con 0x56030984d070 2024-05-27T07:35:23.407 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:23.411+0000 7f07357fa700 1 -- 172.21.15.52:0/2631891534 <== mon.1 v2:172.21.15.173:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f0740004ce0 con 0x56030984d070 2024-05-27T07:35:23.407 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:23.412+0000 7f07357fa700 1 -- 172.21.15.52:0/2631891534 <== mon.1 v2:172.21.15.173:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f07400056b0 con 0x56030984d070 2024-05-27T07:35:23.408 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:23.413+0000 7f076404cb80 1 -- 172.21.15.52:0/2631891534 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x56030984d070 msgr2=0x560309846be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:35:23.408 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:23.413+0000 7f076404cb80 1 --2- 172.21.15.52:0/2631891534 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x56030984d070 0x560309846be0 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7f074001ad70 tx=0x7f074000e630 comp rx=0 tx=0).stop 2024-05-27T07:35:23.409 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:23.413+0000 7f0737fff700 1 --2- 172.21.15.52:0/2999777564 >> [v2:172.21.15.52:6800/1822318384,v1:172.21.15.52:6801/1822318384] conn(0x7f07380b2bf0 0x7f07380b5100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:35:23.409 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:23.413+0000 7f0737fff700 1 -- 172.21.15.52:0/2999777564 <== mon.0 v2:172.21.15.52:3300/0 5 ==== osd_map(123..123 src has 1..123) ==== 9555+0+0 (secure 0 0 0) 0x7f074800f020 con 0x7f075010b270 2024-05-27T07:35:23.409 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:23.413+0000 7f074f7fe700 1 --2- 172.21.15.52:0/2999777564 >> [v2:172.21.15.52:6800/1822318384,v1:172.21.15.52:6801/1822318384] conn(0x7f07380b2bf0 0x7f07380b5100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:35:23.409 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:23.414+0000 7f076404cb80 1 -- 172.21.15.52:0/2631891534 shutdown_connections 2024-05-27T07:35:23.409 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:23.414+0000 7f076404cb80 1 --2- 172.21.15.52:0/2631891534 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x560309847120 0x560309847530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:35:23.409 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:23.414+0000 7f076404cb80 1 --2- 172.21.15.52:0/2631891534 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5603096c7e40 0x56030984cb30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:35:23.409 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:23.414+0000 7f076404cb80 1 --2- 172.21.15.52:0/2631891534 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x56030984d070 0x560309846be0 unknown :-1 s=CLOSED pgs=205 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:35:23.410 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:23.414+0000 7f076404cb80 1 -- 172.21.15.52:0/2631891534 >> 172.21.15.52:0/2631891534 conn(0x5603096ca770 msgr2=0x560309841d90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:35:23.412 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:23.417+0000 7f074f7fe700 1 --2- 172.21.15.52:0/2999777564 >> [v2:172.21.15.52:6800/1822318384,v1:172.21.15.52:6801/1822318384] conn(0x7f07380b2bf0 0x7f07380b5100 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7f0750112b90 tx=0x7f07440090e0 comp rx=0 tx=0).ready entity=mgr.15333 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:35:23.414 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:23.418+0000 7f076404cb80 1 -- 172.21.15.52:0/2631891534 shutdown_connections 2024-05-27T07:35:23.414 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:23.418+0000 7f076404cb80 1 -- 172.21.15.52:0/2631891534 wait complete. 2024-05-27T07:35:23.414 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:23.419+0000 7f076404cb80 1 -- start start 2024-05-27T07:35:23.414 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:23.419+0000 7f076404cb80 1 Processor -- start 2024-05-27T07:35:23.415 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:23.420+0000 7f076404cb80 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5603096c7e40 0x5603098508d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:35:23.415 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:23.420+0000 7f076404cb80 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x560309847120 0x560309850e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:35:23.415 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:23.420+0000 7f076404cb80 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x56030984d070 0x560309851350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:35:23.416 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:23.421+0000 7f076404cb80 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x56030966afa0 con 0x56030984d070 2024-05-27T07:35:23.416 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:23.421+0000 7f076404cb80 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x5603096ccda0 con 0x560309847120 2024-05-27T07:35:23.416 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:23.421+0000 7f076404cb80 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x560309853b60 con 0x5603096c7e40 2024-05-27T07:35:23.416 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:23.421+0000 7f074f7fe700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x56030984d070 0x560309851350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:35:23.417 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:23.421+0000 7f074f7fe700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x56030984d070 0x560309851350 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.52:57328/0 (socket says 172.21.15.52:57328) 2024-05-27T07:35:23.417 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:23.421+0000 7f074f7fe700 1 -- 172.21.15.52:0/2665484610 learned_addr learned my addr 172.21.15.52:0/2665484610 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-05-27T07:35:23.417 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:23.421+0000 7f076404cb80 10 client.0 _ll_register_callbacks cb 0x5603096fb5b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T07:35:23.420 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:23.422+0000 7f074f7fe700 1 -- 172.21.15.52:0/2665484610 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5603096c7e40 msgr2=0x5603098508d0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-05-27T07:35:23.420 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:23.422+0000 7f074f7fe700 1 --2- 172.21.15.52:0/2665484610 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5603096c7e40 0x5603098508d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:35:23.420 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:23.422+0000 7f074f7fe700 1 -- 172.21.15.52:0/2665484610 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x560309847120 msgr2=0x560309850e10 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-05-27T07:35:23.420 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:23.422+0000 7f074f7fe700 1 --2- 172.21.15.52:0/2665484610 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x560309847120 0x560309850e10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:35:23.421 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:23.422+0000 7f074f7fe700 1 -- 172.21.15.52:0/2665484610 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5603096b48d0 con 0x56030984d070 2024-05-27T07:35:23.421 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:23.423+0000 7f074f7fe700 1 --2- 172.21.15.52:0/2665484610 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x56030984d070 0x560309851350 secure :-1 s=READY pgs=737 cs=0 l=1 rev1=1 crypto rx=0x7f0744009300 tx=0x7f0744004770 comp rx=0 tx=0).ready entity=mon.0 client_cookie=772425f86fe2e5c9 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:35:23.421 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:23.423+0000 7f0732ff5700 10 client.0 ms_handle_connect on v2:172.21.15.52:3300/0 2024-05-27T07:35:23.421 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:23.423+0000 7f0732ff5700 1 -- 172.21.15.52:0/2665484610 <== mon.0 v2:172.21.15.52:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0744025070 con 0x56030984d070 2024-05-27T07:35:23.421 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:23.423+0000 7f0732ff5700 1 -- 172.21.15.52:0/2665484610 <== mon.0 v2:172.21.15.52:3300/0 2 ==== osd_map(123..123 src has 1..123) ==== 9555+0+0 (secure 0 0 0) 0x7f0744021600 con 0x56030984d070 2024-05-27T07:35:23.421 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:23.424+0000 7f076404cb80 1 -- 172.21.15.52:0/2665484610 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f0740004ce0 con 0x56030984d070 2024-05-27T07:35:23.424 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:23.429+0000 7f0732ff5700 1 -- 172.21.15.52:0/2665484610 <== mon.0 v2:172.21.15.52:3300/0 3 ==== mon_get_version_reply(handle=1 version=108) ==== 24+0+0 (secure 0 0 0) 0x7f0744021800 con 0x56030984d070 2024-05-27T07:35:23.424 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:23.429+0000 7f076404cb80 10 client.17106 fetch_fsmap learned FSMap version 108 2024-05-27T07:35:23.425 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:23.429+0000 7f076404cb80 1 -- 172.21.15.52:0/2665484610 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({fsmap=108}) -- 0x56030971e760 con 0x56030984d070 2024-05-27T07:35:23.429 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:23.434+0000 7f0732ff5700 1 -- 172.21.15.52:0/2665484610 <== mon.0 v2:172.21.15.52:3300/0 4 ==== fsmap(e 108) ==== 4203+0+0 (secure 0 0 0) 0x7f0744032020 con 0x56030984d070 2024-05-27T07:35:23.429 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:23.434+0000 7f076404cb80 10 client.17106 fetch_fsmap finished waiting for FSMap version 108 2024-05-27T07:35:23.429 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:23.434+0000 7f076404cb80 10 client.17106 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.i 2024-05-27T07:35:23.429 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:23.434+0000 7f076404cb80 20 client.17106 populate_metadata read hostname 'smithi052' 2024-05-27T07:35:23.430 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:23.434+0000 7f076404cb80 1 --2- 172.21.15.52:0/2665484610 >> [v2:172.21.15.173:6838/564778293,v1:172.21.15.173:6839/564778293] conn(0x560309860dd0 0x5603098632e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:35:23.430 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:23.434+0000 7f076404cb80 4 client.17106 mds_command: new command op to 24475 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-27T07:35:23.430 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:23.434+0000 7f076404cb80 1 -- 172.21.15.52:0/2665484610 --> [v2:172.21.15.173:6838/564778293,v1:172.21.15.173:6839/564778293] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x560309776cc0 con 0x560309860dd0 2024-05-27T07:35:23.430 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:23.435+0000 7f074effd700 1 --2- 172.21.15.52:0/2665484610 >> [v2:172.21.15.173:6838/564778293,v1:172.21.15.173:6839/564778293] conn(0x560309860dd0 0x5603098632e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:35:23.444 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:23.449+0000 7f0732ff5700 10 client.17106 ms_handle_connect on v2:172.21.15.173:6838/564778293 2024-05-27T07:35:23.445 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:23.449+0000 7f074effd700 1 --2- 172.21.15.52:0/2665484610 >> [v2:172.21.15.173:6838/564778293,v1:172.21.15.173:6839/564778293] conn(0x560309860dd0 0x5603098632e0 crc :-1 s=READY pgs=49 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=9eea8bcdf5594219 server_cookie=e6645d2374afd539 in_seq=0 out_seq=0 2024-05-27T07:35:23.449 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:23.453+0000 7f0732ff5700 1 -- 172.21.15.52:0/2665484610 <== mds.2 v2:172.21.15.173:6838/564778293 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24041 (crc 0 0 0) 0x560309776cc0 con 0x560309860dd0 2024-05-27T07:35:23.449 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:23.453+0000 7f0732ff5700 10 client.17106 handle_command_reply: tid=0 2024-05-27T07:35:23.449 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:23.453+0000 7f076404cb80 1 client.17106 shutdown 2024-05-27T07:35:23.454 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:23.459+0000 7f076404cb80 1 -- 172.21.15.52:0/2665484610 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x56030984d070 msgr2=0x560309851350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:35:23.455 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:23.459+0000 7f076404cb80 1 --2- 172.21.15.52:0/2665484610 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x56030984d070 0x560309851350 secure :-1 s=READY pgs=737 cs=0 l=1 rev1=1 crypto rx=0x7f0744009300 tx=0x7f0744004770 comp rx=0 tx=0).stop 2024-05-27T07:35:23.455 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:23.459+0000 7f076404cb80 1 -- 172.21.15.52:0/2665484610 shutdown_connections 2024-05-27T07:35:23.455 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:23.459+0000 7f076404cb80 1 --2- 172.21.15.52:0/2665484610 >> [v2:172.21.15.173:6838/564778293,v1:172.21.15.173:6839/564778293] conn(0x560309860dd0 0x5603098632e0 crc :-1 s=READY pgs=49 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:35:23.455 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:23.459+0000 7f076404cb80 1 --2- 172.21.15.52:0/2665484610 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x56030984d070 0x560309851350 unknown :-1 s=CLOSED pgs=737 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:35:23.455 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:23.459+0000 7f076404cb80 1 --2- 172.21.15.52:0/2665484610 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5603096c7e40 0x5603098508d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:35:23.455 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:23.459+0000 7f076404cb80 1 --2- 172.21.15.52:0/2665484610 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x560309847120 0x560309850e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:35:23.455 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:23.459+0000 7f076404cb80 1 -- 172.21.15.52:0/2665484610 >> 172.21.15.52:0/2665484610 conn(0x5603096ca770 msgr2=0x560309845bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:35:23.455 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:23.459+0000 7f0732ff5700 0 client.17106 ms_handle_reset on v2:172.21.15.173:6838/564778293 2024-05-27T07:35:23.457 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:23.462+0000 7f076404cb80 1 -- 172.21.15.52:0/2665484610 shutdown_connections 2024-05-27T07:35:23.457 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:23.462+0000 7f076404cb80 1 -- 172.21.15.52:0/2665484610 wait complete. 2024-05-27T07:35:23.457 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:23.462+0000 7f076404cb80 20 client.17106 trim_cache size 0 max 16384 2024-05-27T07:35:23.490 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:23.495+0000 7f076404cb80 1 Processor -- start 2024-05-27T07:35:23.491 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:23.496+0000 7f076404cb80 1 -- start start 2024-05-27T07:35:23.493 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:23.497+0000 7f076404cb80 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5603096c7e40 0x560309846be0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:35:23.493 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:23.497+0000 7f076404cb80 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x560309847120 0x56030984f850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:35:23.493 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:23.497+0000 7f074effd700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5603096c7e40 0x560309846be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:35:23.493 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:23.497+0000 7f074effd700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5603096c7e40 0x560309846be0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.52:56394/0 (socket says 172.21.15.52:56394) 2024-05-27T07:35:23.493 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:23.497+0000 7f074effd700 1 -- 172.21.15.52:0/549144281 learned_addr learned my addr 172.21.15.52:0/549144281 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-05-27T07:35:23.493 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:23.497+0000 7f074e7fc700 1 --2- 172.21.15.52:0/549144281 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x560309847120 0x56030984f850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:35:23.493 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:23.498+0000 7f076404cb80 1 --2- 172.21.15.52:0/549144281 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x56030984d070 0x560309851190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:35:23.495 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:23.498+0000 7f076404cb80 1 -- 172.21.15.52:0/549144281 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x7f07501d9190 con 0x56030984d070 2024-05-27T07:35:23.495 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:23.498+0000 7f076404cb80 1 -- 172.21.15.52:0/549144281 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f07501d75c0 con 0x5603096c7e40 2024-05-27T07:35:23.495 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:23.498+0000 7f076404cb80 1 -- 172.21.15.52:0/549144281 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f07501d8a90 con 0x560309847120 2024-05-27T07:35:23.495 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:23.498+0000 7f074f7fe700 1 --2- 172.21.15.52:0/549144281 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x56030984d070 0x560309851190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:35:23.495 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:23.499+0000 7f074effd700 1 -- 172.21.15.52:0/549144281 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x560309847120 msgr2=0x56030984f850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:35:23.495 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:23.499+0000 7f074effd700 1 --2- 172.21.15.52:0/549144281 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x560309847120 0x56030984f850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:35:23.495 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:23.499+0000 7f074effd700 1 -- 172.21.15.52:0/549144281 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x56030984d070 msgr2=0x560309851190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:35:23.496 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:23.499+0000 7f074effd700 1 --2- 172.21.15.52:0/549144281 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x56030984d070 0x560309851190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:35:23.496 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:23.499+0000 7f074effd700 1 -- 172.21.15.52:0/549144281 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56030984ee60 con 0x5603096c7e40 2024-05-27T07:35:23.496 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:23.499+0000 7f074effd700 1 --2- 172.21.15.52:0/549144281 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5603096c7e40 0x560309846be0 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x5603098432a0 tx=0x7f074001a930 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b60273dade015631 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:35:23.496 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:23.499+0000 7f0734ff9700 1 -- 172.21.15.52:0/549144281 <== mon.2 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0740004e90 con 0x5603096c7e40 2024-05-27T07:35:23.496 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:23.500+0000 7f0734ff9700 1 -- 172.21.15.52:0/549144281 <== mon.2 v2:172.21.15.105:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f074000ec10 con 0x5603096c7e40 2024-05-27T07:35:23.496 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:23.500+0000 7f0734ff9700 1 -- 172.21.15.52:0/549144281 <== mon.2 v2:172.21.15.105:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f07400048f0 con 0x5603096c7e40 2024-05-27T07:35:23.497 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:23.501+0000 7f076404cb80 1 -- 172.21.15.52:0/549144281 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5603096c7e40 msgr2=0x560309846be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:35:23.497 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:23.501+0000 7f076404cb80 1 --2- 172.21.15.52:0/549144281 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5603096c7e40 0x560309846be0 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x5603098432a0 tx=0x7f074001a930 comp rx=0 tx=0).stop 2024-05-27T07:35:23.497 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:23.501+0000 7f076404cb80 1 -- 172.21.15.52:0/549144281 shutdown_connections 2024-05-27T07:35:23.497 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:23.501+0000 7f076404cb80 1 --2- 172.21.15.52:0/549144281 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x56030984d070 0x560309851190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:35:23.497 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:23.501+0000 7f076404cb80 1 --2- 172.21.15.52:0/549144281 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5603096c7e40 0x560309846be0 unknown :-1 s=CLOSED pgs=245 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:35:23.497 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:23.501+0000 7f076404cb80 1 --2- 172.21.15.52:0/549144281 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x560309847120 0x56030984f850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:35:23.498 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:23.501+0000 7f076404cb80 1 -- 172.21.15.52:0/549144281 >> 172.21.15.52:0/549144281 conn(0x5603096ca770 msgr2=0x56030984a490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:35:23.500 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:23.505+0000 7f076404cb80 1 -- 172.21.15.52:0/549144281 shutdown_connections 2024-05-27T07:35:23.503 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:23.507+0000 7f076404cb80 1 -- 172.21.15.52:0/549144281 wait complete. 2024-05-27T07:35:23.503 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:23.507+0000 7f076404cb80 1 -- start start 2024-05-27T07:35:23.503 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:23.507+0000 7f076404cb80 1 Processor -- start 2024-05-27T07:35:23.504 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:23.508+0000 7f076404cb80 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5603096c7e40 0x560309858ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:35:23.504 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:23.508+0000 7f076404cb80 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x560309847120 0x560309859530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:35:23.504 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:23.508+0000 7f076404cb80 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x56030984d070 0x560309867cb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:35:23.504 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:23.509+0000 7f076404cb80 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x7f07501d83c0 con 0x56030984d070 2024-05-27T07:35:23.505 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:23.509+0000 7f076404cb80 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f07501d7cc0 con 0x560309847120 2024-05-27T07:35:23.505 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:23.509+0000 7f076404cb80 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x5603098478c0 con 0x5603096c7e40 2024-05-27T07:35:23.505 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:23.509+0000 7f076404cb80 10 client.0 _ll_register_callbacks cb 0x5603096fb5b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T07:35:23.506 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:23.510+0000 7f074f7fe700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x56030984d070 0x560309867cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:35:23.506 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:23.510+0000 7f074f7fe700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x56030984d070 0x560309867cb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.52:57348/0 (socket says 172.21.15.52:57348) 2024-05-27T07:35:23.506 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:23.510+0000 7f074f7fe700 1 -- 172.21.15.52:0/1810184685 learned_addr learned my addr 172.21.15.52:0/1810184685 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-05-27T07:35:23.506 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:23.511+0000 7f074effd700 1 --2- 172.21.15.52:0/1810184685 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5603096c7e40 0x560309858ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:35:23.507 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:23.511+0000 7f074f7fe700 1 -- 172.21.15.52:0/1810184685 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5603096c7e40 msgr2=0x560309858ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:35:23.507 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:23.511+0000 7f074f7fe700 1 --2- 172.21.15.52:0/1810184685 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5603096c7e40 0x560309858ff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:35:23.508 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:23.511+0000 7f074f7fe700 1 -- 172.21.15.52:0/1810184685 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x560309847120 msgr2=0x560309859530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:35:23.508 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:23.511+0000 7f074f7fe700 1 --2- 172.21.15.52:0/1810184685 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x560309847120 0x560309859530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:35:23.508 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:23.511+0000 7f074f7fe700 1 -- 172.21.15.52:0/1810184685 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({osdmap=0}) -- 0x560309868350 con 0x56030984d070 2024-05-27T07:35:23.508 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:23.511+0000 7f074f7fe700 1 --2- 172.21.15.52:0/1810184685 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x56030984d070 0x560309867cb0 secure :-1 s=READY pgs=738 cs=0 l=1 rev1=1 crypto rx=0x7f0744014de0 tx=0x7f0744016000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e8db3bc37ef78ec server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:35:23.508 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:23.512+0000 7f0732ff5700 10 client.0 ms_handle_connect on v2:172.21.15.52:3300/0 2024-05-27T07:35:23.508 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:23.512+0000 7f0732ff5700 1 -- 172.21.15.52:0/1810184685 <== mon.0 v2:172.21.15.52:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0744021b30 con 0x56030984d070 2024-05-27T07:35:23.508 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:23.512+0000 7f0732ff5700 1 -- 172.21.15.52:0/1810184685 <== mon.0 v2:172.21.15.52:3300/0 2 ==== osd_map(123..123 src has 1..123) ==== 9555+0+0 (secure 0 0 0) 0x7f07440133a0 con 0x56030984d070 2024-05-27T07:35:23.511 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:23.516+0000 7f076404cb80 1 -- 172.21.15.52:0/1810184685 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f074000ec10 con 0x56030984d070 2024-05-27T07:35:23.516 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:23.521+0000 7f0732ff5700 1 -- 172.21.15.52:0/1810184685 <== mon.0 v2:172.21.15.52:3300/0 3 ==== mon_get_version_reply(handle=1 version=108) ==== 24+0+0 (secure 0 0 0) 0x7f07440135a0 con 0x56030984d070 2024-05-27T07:35:23.518 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:23.522+0000 7f076404cb80 10 client.17112 fetch_fsmap learned FSMap version 108 2024-05-27T07:35:23.518 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:23.522+0000 7f076404cb80 1 -- 172.21.15.52:0/1810184685 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({config=0+,fsmap=108,monmap=4+}) -- 0x5603098cb2b0 con 0x56030984d070 2024-05-27T07:35:23.521 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:23.526+0000 7f0732ff5700 1 -- 172.21.15.52:0/1810184685 <== mon.0 v2:172.21.15.52:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f0740004ce0 con 0x56030984d070 2024-05-27T07:35:23.521 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:23.526+0000 7f0732ff5700 1 -- 172.21.15.52:0/1810184685 <== mon.0 v2:172.21.15.52:3300/0 5 ==== fsmap(e 108) ==== 4203+0+0 (secure 0 0 0) 0x7f0744032070 con 0x56030984d070 2024-05-27T07:35:23.525 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:23.529+0000 7f076404cb80 10 client.17112 fetch_fsmap finished waiting for FSMap version 108 2024-05-27T07:35:23.525 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:23.529+0000 7f076404cb80 10 client.17112 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.i 2024-05-27T07:35:23.525 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:23.529+0000 7f076404cb80 20 client.17112 populate_metadata read hostname 'smithi052' 2024-05-27T07:35:23.525 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:23.529+0000 7f076404cb80 1 --2- 172.21.15.52:0/1810184685 >> [v2:172.21.15.173:6838/564778293,v1:172.21.15.173:6839/564778293] conn(0x5603098cb8b0 0x5603098cdcd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:35:23.525 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:23.529+0000 7f076404cb80 4 client.17112 mds_command: new command op to 24475 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-05-27T07:35:23.525 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:23.529+0000 7f076404cb80 1 -- 172.21.15.52:0/1810184685 --> [v2:172.21.15.173:6838/564778293,v1:172.21.15.173:6839/564778293] -- command(tid 0: {"prefix": "damage ls"}) -- 0x7f074000ec10 con 0x5603098cb8b0 2024-05-27T07:35:23.529 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:23.533+0000 7f074effd700 1 --2- 172.21.15.52:0/1810184685 >> [v2:172.21.15.173:6838/564778293,v1:172.21.15.173:6839/564778293] conn(0x5603098cb8b0 0x5603098cdcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:35:23.529 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:23.534+0000 7f074effd700 1 --2- 172.21.15.52:0/1810184685 >> [v2:172.21.15.173:6838/564778293,v1:172.21.15.173:6839/564778293] conn(0x5603098cb8b0 0x5603098cdcd0 crc :-1 s=READY pgs=50 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=2bde1037d07dc6d6 server_cookie=e00f6434a4bd8ca4 in_seq=0 out_seq=0 2024-05-27T07:35:23.530 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:23.534+0000 7f0732ff5700 10 client.17112 ms_handle_connect on v2:172.21.15.173:6838/564778293 2024-05-27T07:35:23.530 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:23.534+0000 7f0732ff5700 1 -- 172.21.15.52:0/1810184685 <== mds.2 v2:172.21.15.173:6838/564778293 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x7f074000ec10 con 0x5603098cb8b0 2024-05-27T07:35:23.530 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:23.534+0000 7f0732ff5700 10 client.17112 handle_command_reply: tid=0 2024-05-27T07:35:23.530 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:23.534+0000 7f076404cb80 1 client.17112 shutdown 2024-05-27T07:35:23.539 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:23.543+0000 7f076404cb80 1 -- 172.21.15.52:0/1810184685 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x56030984d070 msgr2=0x560309867cb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:35:23.539 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:23.543+0000 7f076404cb80 1 --2- 172.21.15.52:0/1810184685 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x56030984d070 0x560309867cb0 secure :-1 s=READY pgs=738 cs=0 l=1 rev1=1 crypto rx=0x7f0744014de0 tx=0x7f0744016000 comp rx=0 tx=0).stop 2024-05-27T07:35:23.539 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:23.543+0000 7f076404cb80 1 -- 172.21.15.52:0/1810184685 shutdown_connections 2024-05-27T07:35:23.539 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:23.543+0000 7f076404cb80 1 --2- 172.21.15.52:0/1810184685 >> [v2:172.21.15.173:6838/564778293,v1:172.21.15.173:6839/564778293] conn(0x5603098cb8b0 0x5603098cdcd0 crc :-1 s=READY pgs=50 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:35:23.539 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:23.543+0000 7f076404cb80 1 --2- 172.21.15.52:0/1810184685 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x56030984d070 0x560309867cb0 unknown :-1 s=CLOSED pgs=738 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:35:23.539 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:23.543+0000 7f0732ff5700 0 client.17112 ms_handle_reset on v2:172.21.15.173:6838/564778293 2024-05-27T07:35:23.539 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:23.543+0000 7f076404cb80 1 --2- 172.21.15.52:0/1810184685 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5603096c7e40 0x560309858ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:35:23.539 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:23.543+0000 7f076404cb80 1 --2- 172.21.15.52:0/1810184685 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x560309847120 0x560309859530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:35:23.540 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:23.543+0000 7f076404cb80 1 -- 172.21.15.52:0/1810184685 >> 172.21.15.52:0/1810184685 conn(0x5603096ca770 msgr2=0x56030984ac20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:35:23.540 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:23.544+0000 7f076404cb80 1 -- 172.21.15.52:0/1810184685 shutdown_connections 2024-05-27T07:35:23.540 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:23.544+0000 7f076404cb80 1 -- 172.21.15.52:0/1810184685 wait complete. 2024-05-27T07:35:23.540 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:23.545+0000 7f076404cb80 20 client.17112 trim_cache size 0 max 16384 2024-05-27T07:35:23.540 INFO:teuthology.orchestra.run.smithi052.stdout:[] 2024-05-27T07:35:23.541 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:23.545+0000 7f0755637700 1 -- 172.21.15.52:0/2999777564 >> [v2:172.21.15.52:6800/1822318384,v1:172.21.15.52:6801/1822318384] conn(0x7f07380b2bf0 msgr2=0x7f07380b5100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:35:23.541 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:23.545+0000 7f0755637700 1 --2- 172.21.15.52:0/2999777564 >> [v2:172.21.15.52:6800/1822318384,v1:172.21.15.52:6801/1822318384] conn(0x7f07380b2bf0 0x7f07380b5100 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7f0750112b90 tx=0x7f07440090e0 comp rx=0 tx=0).stop 2024-05-27T07:35:23.541 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:23.546+0000 7f0755637700 1 -- 172.21.15.52:0/2999777564 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f075010b270 msgr2=0x7f0750119360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:35:23.541 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:23.546+0000 7f0755637700 1 --2- 172.21.15.52:0/2999777564 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f075010b270 0x7f0750119360 secure :-1 s=READY pgs=736 cs=0 l=1 rev1=1 crypto rx=0x7f0748000f80 tx=0x7f074800cdf0 comp rx=0 tx=0).stop 2024-05-27T07:35:23.542 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:23.546+0000 7f0755637700 1 -- 172.21.15.52:0/2999777564 shutdown_connections 2024-05-27T07:35:23.542 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:23.546+0000 7f0755637700 1 --2- 172.21.15.52:0/2999777564 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f075011d920 0x7f07501198a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:35:23.542 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:23.546+0000 7f0755637700 1 --2- 172.21.15.52:0/2999777564 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f075010a920 0x7f075011d220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:35:23.542 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:23.546+0000 7f0755637700 1 --2- 172.21.15.52:0/2999777564 >> [v2:172.21.15.52:6800/1822318384,v1:172.21.15.52:6801/1822318384] conn(0x7f07380b2bf0 0x7f07380b5100 unknown :-1 s=CLOSED pgs=183 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:35:23.542 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:23.546+0000 7f0755637700 1 --2- 172.21.15.52:0/2999777564 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f075010b270 0x7f0750119360 unknown :-1 s=CLOSED pgs=736 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:35:23.542 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:23.546+0000 7f0755637700 1 -- 172.21.15.52:0/2999777564 >> 172.21.15.52:0/2999777564 conn(0x7f075006f4c0 msgr2=0x7f075011df10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:35:23.542 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:23.546+0000 7f0755637700 1 -- 172.21.15.52:0/2999777564 shutdown_connections 2024-05-27T07:35:23.542 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:23.546+0000 7f0755637700 1 -- 172.21.15.52:0/2999777564 wait complete. 2024-05-27T07:35:24.607 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:35:24 smithi173 ceph-mon[37000]: pgmap v318: 129 pgs: 20 active+clean+snaptrim, 58 active+clean+snaptrim_wait, 51 active+clean; 649 MiB data, 6.1 GiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 1.1 MiB/s wr, 222 op/s 2024-05-27T07:35:24.695 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:35:24 smithi105 ceph-mon[35977]: pgmap v318: 129 pgs: 20 active+clean+snaptrim, 58 active+clean+snaptrim_wait, 51 active+clean; 649 MiB data, 6.1 GiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 1.1 MiB/s wr, 222 op/s 2024-05-27T07:35:24.734 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:35:24 smithi052 ceph-mon[31955]: pgmap v318: 129 pgs: 20 active+clean+snaptrim, 58 active+clean+snaptrim_wait, 51 active+clean; 649 MiB data, 6.1 GiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 1.1 MiB/s wr, 222 op/s 2024-05-27T07:35:25.701 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:35:25.703+0000 7efc9f822640 1 -- 172.21.15.52:0/3220006377 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7efc98108b90 msgr2=0x7efc98102730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:35:25.701 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:35:25.703+0000 7efc9f822640 1 --2- 172.21.15.52:0/3220006377 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7efc98108b90 0x7efc98102730 secure :-1 s=READY pgs=739 cs=0 l=1 rev1=1 crypto rx=0x7efc94009ca0 tx=0x7efc9402a8c0 comp rx=0 tx=0).stop 2024-05-27T07:35:25.701 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:35:25.703+0000 7efc9f822640 1 -- 172.21.15.52:0/3220006377 shutdown_connections 2024-05-27T07:35:25.702 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:35:25.703+0000 7efc9f822640 1 --2- 172.21.15.52:0/3220006377 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7efc981032e0 0x7efc9810f6b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:35:25.702 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:35:25.703+0000 7efc9f822640 1 --2- 172.21.15.52:0/3220006377 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7efc98109030 0x7efc98102c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:35:25.702 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:35:25.703+0000 7efc9f822640 1 --2- 172.21.15.52:0/3220006377 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7efc98108b90 0x7efc98102730 unknown :-1 s=CLOSED pgs=739 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:35:25.702 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:35:25.703+0000 7efc9f822640 1 -- 172.21.15.52:0/3220006377 >> 172.21.15.52:0/3220006377 conn(0x7efc980782d0 msgr2=0x7efc98078710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:35:25.702 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:35:25.703+0000 7efc9f822640 1 -- 172.21.15.52:0/3220006377 shutdown_connections 2024-05-27T07:35:25.703 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:35:25.703+0000 7efc9f822640 1 -- 172.21.15.52:0/3220006377 wait complete. 2024-05-27T07:35:25.703 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:35:25.704+0000 7efc9f822640 1 Processor -- start 2024-05-27T07:35:25.703 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:35:25.704+0000 7efc9f822640 1 -- start start 2024-05-27T07:35:25.703 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:35:25.705+0000 7efc9f822640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7efc981032e0 0x7efc981986e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:35:25.703 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:35:25.705+0000 7efc9f822640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7efc98108b90 0x7efc98198c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:35:25.703 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:35:25.705+0000 7efc9f822640 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7efc98109030 0x7efc9819cf00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:35:25.703 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:35:25.705+0000 7efc9f822640 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x7efc98111c80 con 0x7efc98109030 2024-05-27T07:35:25.703 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:35:25.705+0000 7efc9f822640 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7efc98111b00 con 0x7efc981032e0 2024-05-27T07:35:25.704 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:35:25.705+0000 7efc9f822640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7efc98111e00 con 0x7efc98108b90 2024-05-27T07:35:25.704 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:35:25.705+0000 7efc9ddbf640 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7efc98109030 0x7efc9819cf00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:35:25.704 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:35:25.705+0000 7efc9cdbd640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7efc98108b90 0x7efc98198c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:35:25.704 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:35:25.705+0000 7efc9d5be640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7efc981032e0 0x7efc981986e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:35:25.704 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:35:25.705+0000 7efc9cdbd640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7efc98108b90 0x7efc98198c40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.173:3300/0 says I am v2:172.21.15.52:42096/0 (socket says 172.21.15.52:42096) 2024-05-27T07:35:25.704 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:35:25.705+0000 7efc9cdbd640 1 -- 172.21.15.52:0/3546543659 learned_addr learned my addr 172.21.15.52:0/3546543659 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-05-27T07:35:25.704 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:35:25.705+0000 7efc9cdbd640 1 -- 172.21.15.52:0/3546543659 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7efc981032e0 msgr2=0x7efc981986e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:35:25.705 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:35:25.705+0000 7efc9cdbd640 1 --2- 172.21.15.52:0/3546543659 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7efc981032e0 0x7efc981986e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:35:25.705 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:35:25.705+0000 7efc9cdbd640 1 -- 172.21.15.52:0/3546543659 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7efc98109030 msgr2=0x7efc9819cf00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:35:25.705 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:35:25.705+0000 7efc9cdbd640 1 --2- 172.21.15.52:0/3546543659 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7efc98109030 0x7efc9819cf00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:35:25.705 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:35:25.705+0000 7efc9cdbd640 1 -- 172.21.15.52:0/3546543659 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7efc9819d540 con 0x7efc98108b90 2024-05-27T07:35:25.705 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:35:25.705+0000 7efc9ddbf640 1 --2- 172.21.15.52:0/3546543659 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7efc98109030 0x7efc9819cf00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-27T07:35:25.705 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:35:25.705+0000 7efc9cdbd640 1 --2- 172.21.15.52:0/3546543659 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7efc98108b90 0x7efc98198c40 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7efc8c010ad0 tx=0x7efc8c010f90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:35:25.705 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:35:25.706+0000 7efc8a7fc640 1 -- 172.21.15.52:0/3546543659 <== mon.1 v2:172.21.15.173:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efc8c01a030 con 0x7efc98108b90 2024-05-27T07:35:25.706 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:35:25.706+0000 7efc8a7fc640 1 -- 172.21.15.52:0/3546543659 <== mon.1 v2:172.21.15.173:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7efc8c004020 con 0x7efc98108b90 2024-05-27T07:35:25.706 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:35:25.706+0000 7efc8a7fc640 1 -- 172.21.15.52:0/3546543659 <== mon.1 v2:172.21.15.173:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efc8c003b50 con 0x7efc98108b90 2024-05-27T07:35:25.706 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:35:25.706+0000 7efc9f822640 1 -- 172.21.15.52:0/3546543659 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7efc9819d830 con 0x7efc98108b90 2024-05-27T07:35:25.706 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:35:25.706+0000 7efc9f822640 1 -- 172.21.15.52:0/3546543659 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7efc9819dcd0 con 0x7efc98108b90 2024-05-27T07:35:25.708 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:35:25.708+0000 7efca67ae740 1 Processor -- start 2024-05-27T07:35:25.708 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:35:25.708+0000 7efc8a7fc640 1 -- 172.21.15.52:0/3546543659 <== mon.1 v2:172.21.15.173:3300/0 4 ==== mgrmap(e 26) ==== 149783+0+0 (secure 0 0 0) 0x7efc8c01d070 con 0x7efc98108b90 2024-05-27T07:35:25.708 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:35:25.709+0000 7efca67ae740 1 -- start start 2024-05-27T07:35:25.709 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:35:25.709+0000 7efca67ae740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55cf057ab1b0 0x55cf05b4c3d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:35:25.709 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:35:25.709+0000 7efca67ae740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55cf057822f0 0x55cf05b4c910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:35:25.709 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:35:25.709+0000 7efca67ae740 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55cf0578a4d0 0x55cf05b4ce50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:35:25.709 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:35:25.709+0000 7efca67ae740 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x55cf05abec30 con 0x55cf0578a4d0 2024-05-27T07:35:25.709 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:35:25.709+0000 7efca67ae740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x55cf05aae7d0 con 0x55cf057ab1b0 2024-05-27T07:35:25.709 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:35:25.709+0000 7efca67ae740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x55cf05ab2810 con 0x55cf057822f0 2024-05-27T07:35:25.709 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:35:25.709+0000 7efc9cdbd640 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55cf0578a4d0 0x55cf05b4ce50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:35:25.710 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:35:25.709+0000 7efc9ddbf640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55cf057ab1b0 0x55cf05b4c3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:35:25.710 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:35:25.709+0000 7efc9cdbd640 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55cf0578a4d0 0x55cf05b4ce50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.52:57370/0 (socket says 172.21.15.52:57370) 2024-05-27T07:35:25.710 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:35:25.709+0000 7efc9ddbf640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55cf057ab1b0 0x55cf05b4c3d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.52:56414/0 (socket says 172.21.15.52:56414) 2024-05-27T07:35:25.710 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:35:25.709+0000 7efc9d5be640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55cf057822f0 0x55cf05b4c910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:35:25.710 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:35:25.709+0000 7efc9cdbd640 1 -- 172.21.15.52:0/971336509 learned_addr learned my addr 172.21.15.52:0/971336509 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-05-27T07:35:25.710 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:35:25.709+0000 7efc8a7fc640 1 --2- 172.21.15.52:0/3546543659 >> [v2:172.21.15.52:6800/1822318384,v1:172.21.15.52:6801/1822318384] conn(0x7efc840b29d0 0x7efc840b4ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:35:25.710 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:35:25.709+0000 7efc9ddbf640 1 --2- 172.21.15.52:0/3546543659 >> [v2:172.21.15.52:6800/1822318384,v1:172.21.15.52:6801/1822318384] conn(0x7efc840b29d0 0x7efc840b4ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:35:25.710 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:35:25.710+0000 7efc9d5be640 1 -- 172.21.15.52:0/971336509 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55cf057ab1b0 msgr2=0x55cf05b4c3d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:35:25.710 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:35:25.710+0000 7efc9d5be640 1 --2- 172.21.15.52:0/971336509 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55cf057ab1b0 0x55cf05b4c3d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:35:25.711 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:35:25.710+0000 7efc9d5be640 1 -- 172.21.15.52:0/971336509 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55cf0578a4d0 msgr2=0x55cf05b4ce50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:35:25.711 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:35:25.710+0000 7efc9d5be640 1 --2- 172.21.15.52:0/971336509 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55cf0578a4d0 0x55cf05b4ce50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:35:25.711 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:35:25.710+0000 7efc9d5be640 1 -- 172.21.15.52:0/971336509 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55cf057c3210 con 0x55cf057822f0 2024-05-27T07:35:25.711 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:35:25.710+0000 7efc8a7fc640 1 -- 172.21.15.52:0/3546543659 <== mon.1 v2:172.21.15.173:3300/0 5 ==== osd_map(123..123 src has 1..123) ==== 9555+0+0 (secure 0 0 0) 0x7efc8c0e4700 con 0x7efc98108b90 2024-05-27T07:35:25.711 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:35:25.710+0000 7efc9d5be640 1 --2- 172.21.15.52:0/971336509 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55cf057822f0 0x55cf05b4c910 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7efc94006010 tx=0x7efc940093f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=5cfe4aec81da13d9 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:35:25.711 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:35:25.710+0000 7efc9ddbf640 1 --2- 172.21.15.52:0/3546543659 >> [v2:172.21.15.52:6800/1822318384,v1:172.21.15.52:6801/1822318384] conn(0x7efc840b29d0 0x7efc840b4ee0 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x55cf05b49540 tx=0x7efc90004730 comp rx=0 tx=0).ready entity=mgr.15333 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:35:25.711 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:35:25.710+0000 7efc837fe640 1 -- 172.21.15.52:0/971336509 <== mon.1 v2:172.21.15.173:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efc94031cf0 con 0x55cf057822f0 2024-05-27T07:35:25.711 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:35:25.710+0000 7efc837fe640 1 -- 172.21.15.52:0/971336509 <== mon.1 v2:172.21.15.173:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7efc94036030 con 0x55cf057822f0 2024-05-27T07:35:25.711 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:35:25.710+0000 7efc837fe640 1 -- 172.21.15.52:0/971336509 <== mon.1 v2:172.21.15.173:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efc94004260 con 0x55cf057822f0 2024-05-27T07:35:25.711 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:35:25.711+0000 7efca67ae740 1 -- 172.21.15.52:0/971336509 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55cf057822f0 msgr2=0x55cf05b4c910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:35:25.711 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:35:25.711+0000 7efca67ae740 1 --2- 172.21.15.52:0/971336509 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55cf057822f0 0x55cf05b4c910 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7efc94006010 tx=0x7efc940093f0 comp rx=0 tx=0).stop 2024-05-27T07:35:25.712 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:35:25.711+0000 7efca67ae740 1 -- 172.21.15.52:0/971336509 shutdown_connections 2024-05-27T07:35:25.712 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:35:25.711+0000 7efca67ae740 1 --2- 172.21.15.52:0/971336509 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55cf0578a4d0 0x55cf05b4ce50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:35:25.712 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:35:25.711+0000 7efca67ae740 1 --2- 172.21.15.52:0/971336509 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55cf057822f0 0x55cf05b4c910 unknown :-1 s=CLOSED pgs=207 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:35:25.712 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:35:25.711+0000 7efca67ae740 1 --2- 172.21.15.52:0/971336509 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55cf057ab1b0 0x55cf05b4c3d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:35:25.712 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:35:25.711+0000 7efca67ae740 1 -- 172.21.15.52:0/971336509 >> 172.21.15.52:0/971336509 conn(0x55cf0579f300 msgr2=0x55cf05b48b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:35:25.712 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:35:25.711+0000 7efca67ae740 1 -- 172.21.15.52:0/971336509 shutdown_connections 2024-05-27T07:35:25.712 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:35:25.711+0000 7efca67ae740 1 -- 172.21.15.52:0/971336509 wait complete. 2024-05-27T07:35:25.712 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:35:25.711+0000 7efca67ae740 1 -- start start 2024-05-27T07:35:25.712 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:35:25.711+0000 7efca67ae740 1 Processor -- start 2024-05-27T07:35:25.713 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:35:25.712+0000 7efca67ae740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55cf057822f0 0x55cf05b584c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:35:25.713 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:35:25.712+0000 7efca67ae740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55cf0578a4d0 0x55cf05b58a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:35:25.713 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:35:25.712+0000 7efca67ae740 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55cf057ab1b0 0x55cf05b56940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:35:25.713 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:35:25.712+0000 7efca67ae740 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x55cf05794790 con 0x55cf057ab1b0 2024-05-27T07:35:25.713 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:35:25.712+0000 7efca67ae740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x55cf05a6a890 con 0x55cf0578a4d0 2024-05-27T07:35:25.713 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:35:25.712+0000 7efca67ae740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x55cf05b5d730 con 0x55cf057822f0 2024-05-27T07:35:25.713 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:35:25.712+0000 7efca67ae740 10 client.0 _ll_register_callbacks cb 0x55cf057c2a00 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T07:35:25.713 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:35:25.713+0000 7efc9cdbd640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55cf0578a4d0 0x55cf05b58a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:35:25.713 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:35:25.713+0000 7efc9cdbd640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55cf0578a4d0 0x55cf05b58a00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.52:56430/0 (socket says 172.21.15.52:56430) 2024-05-27T07:35:25.713 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:35:25.713+0000 7efc9cdbd640 1 -- 172.21.15.52:0/567513974 learned_addr learned my addr 172.21.15.52:0/567513974 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-05-27T07:35:25.713 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:35:25.714+0000 7efc9ddbf640 1 --2- 172.21.15.52:0/567513974 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55cf057ab1b0 0x55cf05b56940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:35:25.713 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:35:25.715+0000 7efc9ddbf640 1 -- 172.21.15.52:0/567513974 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55cf057822f0 msgr2=0x55cf05b584c0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-05-27T07:35:25.713 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:35:25.715+0000 7efc9ddbf640 1 --2- 172.21.15.52:0/567513974 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55cf057822f0 0x55cf05b584c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:35:25.714 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:35:25.715+0000 7efc9ddbf640 1 -- 172.21.15.52:0/567513974 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55cf0578a4d0 msgr2=0x55cf05b58a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:35:25.714 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:35:25.715+0000 7efc9ddbf640 1 --2- 172.21.15.52:0/567513974 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55cf0578a4d0 0x55cf05b58a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:35:25.714 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:35:25.715+0000 7efc9ddbf640 1 -- 172.21.15.52:0/567513974 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55cf0588f620 con 0x55cf057ab1b0 2024-05-27T07:35:25.714 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:35:25.715+0000 7efc9ddbf640 1 --2- 172.21.15.52:0/567513974 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55cf057ab1b0 0x55cf05b56940 secure :-1 s=READY pgs=740 cs=0 l=1 rev1=1 crypto rx=0x7efc90009710 tx=0x7efc900046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8fea616d099d4fac server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:35:25.721 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:35:25.725+0000 7efc80ff9640 10 client.0 ms_handle_connect on v2:172.21.15.52:3300/0 2024-05-27T07:35:25.721 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:35:25.725+0000 7efc80ff9640 1 -- 172.21.15.52:0/567513974 <== mon.0 v2:172.21.15.52:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efc90048070 con 0x55cf057ab1b0 2024-05-27T07:35:25.721 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:35:25.725+0000 7efc80ff9640 1 -- 172.21.15.52:0/567513974 <== mon.0 v2:172.21.15.52:3300/0 2 ==== osd_map(123..123 src has 1..123) ==== 9555+0+0 (secure 0 0 0) 0x7efc90010c90 con 0x55cf057ab1b0 2024-05-27T07:35:25.723 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:35:25.728+0000 7efca67ae740 1 -- 172.21.15.52:0/567513974 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7efc94036030 con 0x55cf057ab1b0 2024-05-27T07:35:25.730 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:35:25.734+0000 7efc80ff9640 1 -- 172.21.15.52:0/567513974 <== mon.0 v2:172.21.15.52:3300/0 3 ==== mon_get_version_reply(handle=1 version=108) ==== 24+0+0 (secure 0 0 0) 0x7efc90052050 con 0x55cf057ab1b0 2024-05-27T07:35:25.730 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:35:25.735+0000 7efca67ae740 10 client.17124 fetch_fsmap learned FSMap version 108 2024-05-27T07:35:25.730 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:35:25.735+0000 7efca67ae740 1 -- 172.21.15.52:0/567513974 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({config=0+,fsmap=108,monmap=4+}) -- 0x55cf05b67ad0 con 0x55cf057ab1b0 2024-05-27T07:35:25.733 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:35:25.735+0000 7efc80ff9640 1 -- 172.21.15.52:0/567513974 <== mon.0 v2:172.21.15.52:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7efc90003e90 con 0x55cf057ab1b0 2024-05-27T07:35:25.733 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:35:25.735+0000 7efc80ff9640 1 -- 172.21.15.52:0/567513974 <== mon.0 v2:172.21.15.52:3300/0 5 ==== fsmap(e 108) ==== 4203+0+0 (secure 0 0 0) 0x7efc90044400 con 0x55cf057ab1b0 2024-05-27T07:35:25.733 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:35:25.735+0000 7efca67ae740 10 client.17124 fetch_fsmap finished waiting for FSMap version 108 2024-05-27T07:35:25.733 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:35:25.735+0000 7efca67ae740 10 client.17124 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-05-27T07:35:25.734 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:35:25.735+0000 7efca67ae740 20 client.17124 populate_metadata read hostname 'smithi052' 2024-05-27T07:35:25.734 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:35:25.735+0000 7efca67ae740 1 --2- 172.21.15.52:0/567513974 >> [v2:172.21.15.105:6838/500348151,v1:172.21.15.105:6839/500348151] conn(0x55cf05769d80 0x55cf05b6b4b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:35:25.734 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:35:25.735+0000 7efca67ae740 4 client.17124 mds_command: new command op to 24527 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-27T07:35:25.734 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:35:25.735+0000 7efca67ae740 1 -- 172.21.15.52:0/567513974 --> [v2:172.21.15.105:6838/500348151,v1:172.21.15.105:6839/500348151] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x7efc94036030 con 0x55cf05769d80 2024-05-27T07:35:25.737 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:35:25.742+0000 7efc9d5be640 1 --2- 172.21.15.52:0/567513974 >> [v2:172.21.15.105:6838/500348151,v1:172.21.15.105:6839/500348151] conn(0x55cf05769d80 0x55cf05b6b4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:35:25.751 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:35:25.756+0000 7efc9d5be640 1 --2- 172.21.15.52:0/567513974 >> [v2:172.21.15.105:6838/500348151,v1:172.21.15.105:6839/500348151] conn(0x55cf05769d80 0x55cf05b6b4b0 crc :-1 s=READY pgs=165 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=a3e44111ad4978dc server_cookie=ffb3886348b42faf in_seq=0 out_seq=0 2024-05-27T07:35:25.756 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:35:25.758+0000 7efc80ff9640 10 client.17124 ms_handle_connect on v2:172.21.15.105:6838/500348151 2024-05-27T07:35:25.770 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:35:25.775+0000 7efc80ff9640 1 -- 172.21.15.52:0/567513974 <== mds.0 v2:172.21.15.105:6838/500348151 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24041 (crc 0 0 0) 0x7efc94036030 con 0x55cf05769d80 2024-05-27T07:35:25.770 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:35:25.775+0000 7efc80ff9640 10 client.17124 handle_command_reply: tid=0 2024-05-27T07:35:25.770 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:35:25.775+0000 7efca67ae740 1 client.17124 shutdown 2024-05-27T07:35:25.772 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:35:25.777+0000 7efca67ae740 1 -- 172.21.15.52:0/567513974 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55cf057ab1b0 msgr2=0x55cf05b56940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:35:25.772 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:35:25.777+0000 7efca67ae740 1 --2- 172.21.15.52:0/567513974 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55cf057ab1b0 0x55cf05b56940 secure :-1 s=READY pgs=740 cs=0 l=1 rev1=1 crypto rx=0x7efc90009710 tx=0x7efc900046c0 comp rx=0 tx=0).stop 2024-05-27T07:35:25.776 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:35:25.781+0000 7efca67ae740 1 -- 172.21.15.52:0/567513974 shutdown_connections 2024-05-27T07:35:25.776 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:35:25.781+0000 7efca67ae740 1 --2- 172.21.15.52:0/567513974 >> [v2:172.21.15.105:6838/500348151,v1:172.21.15.105:6839/500348151] conn(0x55cf05769d80 0x55cf05b6b4b0 crc :-1 s=READY pgs=165 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:35:25.776 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:35:25.781+0000 7efca67ae740 1 --2- 172.21.15.52:0/567513974 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55cf057ab1b0 0x55cf05b56940 unknown :-1 s=CLOSED pgs=740 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:35:25.776 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:35:25.781+0000 7efca67ae740 1 --2- 172.21.15.52:0/567513974 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55cf0578a4d0 0x55cf05b58a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:35:25.776 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:35:25.781+0000 7efca67ae740 1 --2- 172.21.15.52:0/567513974 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55cf057822f0 0x55cf05b584c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:35:25.776 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:35:25.781+0000 7efca67ae740 1 -- 172.21.15.52:0/567513974 >> 172.21.15.52:0/567513974 conn(0x55cf0579f300 msgr2=0x55cf05b4c200 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:35:25.776 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:35:25.781+0000 7efc80ff9640 0 client.17124 ms_handle_reset on v2:172.21.15.105:6838/500348151 2024-05-27T07:35:25.777 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:35:25.782+0000 7efca67ae740 1 -- 172.21.15.52:0/567513974 shutdown_connections 2024-05-27T07:35:25.780 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:35:25.784+0000 7efca67ae740 1 -- 172.21.15.52:0/567513974 wait complete. 2024-05-27T07:35:25.782 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:35:25.786+0000 7efca67ae740 20 client.17124 trim_cache size 0 max 16384 2024-05-27T07:35:25.818 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:35:25.823+0000 7efca67ae740 1 Processor -- start 2024-05-27T07:35:25.821 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:35:25.824+0000 7efca67ae740 1 -- start start 2024-05-27T07:35:25.822 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:35:25.824+0000 7efca67ae740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55cf057822f0 0x55cf05b5d110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:35:25.822 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:35:25.824+0000 7efca67ae740 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55cf0578a4d0 0x55cf05b55880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:35:25.822 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:35:25.824+0000 7efca67ae740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55cf057ab1b0 0x55cf05b4c870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:35:25.822 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:35:25.824+0000 7efca67ae740 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x7efc981a7090 con 0x55cf0578a4d0 2024-05-27T07:35:25.822 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:35:25.824+0000 7efca67ae740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7efc981a54c0 con 0x55cf057ab1b0 2024-05-27T07:35:25.822 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:35:25.824+0000 7efca67ae740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7efc981a6990 con 0x55cf057822f0 2024-05-27T07:35:25.822 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:35:25.824+0000 7efc9d5be640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55cf057822f0 0x55cf05b5d110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:35:25.822 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:35:25.824+0000 7efc9d5be640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55cf057822f0 0x55cf05b5d110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.173:3300/0 says I am v2:172.21.15.52:42118/0 (socket says 172.21.15.52:42118) 2024-05-27T07:35:25.822 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:35:25.824+0000 7efc9d5be640 1 -- 172.21.15.52:0/1279855598 learned_addr learned my addr 172.21.15.52:0/1279855598 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-05-27T07:35:25.822 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:35:25.824+0000 7efc9cdbd640 1 --2- 172.21.15.52:0/1279855598 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55cf0578a4d0 0x55cf05b55880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:35:25.822 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:35:25.825+0000 7efc9ddbf640 1 --2- 172.21.15.52:0/1279855598 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55cf057ab1b0 0x55cf05b4c870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:35:25.823 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:35:25.825+0000 7efc9cdbd640 1 -- 172.21.15.52:0/1279855598 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55cf057822f0 msgr2=0x55cf05b5d110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:35:25.823 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:35:25.825+0000 7efc9cdbd640 1 --2- 172.21.15.52:0/1279855598 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55cf057822f0 0x55cf05b5d110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:35:25.823 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:35:25.825+0000 7efc9cdbd640 1 -- 172.21.15.52:0/1279855598 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55cf057ab1b0 msgr2=0x55cf05b4c870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:35:25.823 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:35:25.825+0000 7efc9cdbd640 1 --2- 172.21.15.52:0/1279855598 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55cf057ab1b0 0x55cf05b4c870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:35:25.823 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:35:25.825+0000 7efc9cdbd640 1 -- 172.21.15.52:0/1279855598 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55cf05b6ba70 con 0x55cf0578a4d0 2024-05-27T07:35:25.824 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:35:25.827+0000 7efc9cdbd640 1 --2- 172.21.15.52:0/1279855598 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55cf0578a4d0 0x55cf05b55880 secure :-1 s=READY pgs=741 cs=0 l=1 rev1=1 crypto rx=0x7efc8c0ab2d0 tx=0x7efc8c0a7510 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8c2ed622fd202a74 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:35:25.824 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:35:25.827+0000 7efc82ffd640 1 -- 172.21.15.52:0/1279855598 <== mon.0 v2:172.21.15.52:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efc8c0aaf20 con 0x55cf0578a4d0 2024-05-27T07:35:25.824 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:35:25.827+0000 7efc82ffd640 1 -- 172.21.15.52:0/1279855598 <== mon.0 v2:172.21.15.52:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7efc8c0ab0e0 con 0x55cf0578a4d0 2024-05-27T07:35:25.824 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:35:25.828+0000 7efc82ffd640 1 -- 172.21.15.52:0/1279855598 <== mon.0 v2:172.21.15.52:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efc8c0aad40 con 0x55cf0578a4d0 2024-05-27T07:35:25.828 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:35:25.833+0000 7efca67ae740 1 -- 172.21.15.52:0/1279855598 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55cf0578a4d0 msgr2=0x55cf05b55880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:35:25.828 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:35:25.833+0000 7efca67ae740 1 --2- 172.21.15.52:0/1279855598 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55cf0578a4d0 0x55cf05b55880 secure :-1 s=READY pgs=741 cs=0 l=1 rev1=1 crypto rx=0x7efc8c0ab2d0 tx=0x7efc8c0a7510 comp rx=0 tx=0).stop 2024-05-27T07:35:25.828 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:35:25.833+0000 7efca67ae740 1 -- 172.21.15.52:0/1279855598 shutdown_connections 2024-05-27T07:35:25.828 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:35:25.833+0000 7efca67ae740 1 --2- 172.21.15.52:0/1279855598 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55cf057ab1b0 0x55cf05b4c870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:35:25.829 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:35:25.833+0000 7efca67ae740 1 --2- 172.21.15.52:0/1279855598 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55cf0578a4d0 0x55cf05b55880 unknown :-1 s=CLOSED pgs=741 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:35:25.829 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:35:25.833+0000 7efca67ae740 1 --2- 172.21.15.52:0/1279855598 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55cf057822f0 0x55cf05b5d110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:35:25.829 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:35:25.833+0000 7efca67ae740 1 -- 172.21.15.52:0/1279855598 >> 172.21.15.52:0/1279855598 conn(0x55cf0579f300 msgr2=0x55cf05b4c160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:35:25.832 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:35:25.837+0000 7efca67ae740 1 -- 172.21.15.52:0/1279855598 shutdown_connections 2024-05-27T07:35:25.832 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:35:25.837+0000 7efca67ae740 1 -- 172.21.15.52:0/1279855598 wait complete. 2024-05-27T07:35:25.832 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:35:25.837+0000 7efca67ae740 1 -- start start 2024-05-27T07:35:25.832 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:35:25.837+0000 7efca67ae740 1 Processor -- start 2024-05-27T07:35:25.833 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:35:25.838+0000 7efca67ae740 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55cf057822f0 0x55cf05b5c270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:35:25.834 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:35:25.838+0000 7efca67ae740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55cf0578a4d0 0x55cf05b5c7b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:35:25.834 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:35:25.838+0000 7efca67ae740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55cf057ab1b0 0x55cf05b5ccf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:35:25.834 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:35:25.838+0000 7efca67ae740 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x7efc981a62c0 con 0x55cf057822f0 2024-05-27T07:35:25.834 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:35:25.838+0000 7efca67ae740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7efc981a5bc0 con 0x55cf0578a4d0 2024-05-27T07:35:25.834 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:35:25.838+0000 7efca67ae740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x55cf05b63500 con 0x55cf057ab1b0 2024-05-27T07:35:25.834 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:35:25.838+0000 7efca67ae740 10 client.0 _ll_register_callbacks cb 0x7efc90013df0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T07:35:25.834 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:35:25.838+0000 7efc9d5be640 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55cf057822f0 0x55cf05b5c270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:35:25.834 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:35:25.838+0000 7efc9d5be640 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55cf057822f0 0x55cf05b5c270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.52:57404/0 (socket says 172.21.15.52:57404) 2024-05-27T07:35:25.834 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:35:25.838+0000 7efc9d5be640 1 -- 172.21.15.52:0/4104769853 learned_addr learned my addr 172.21.15.52:0/4104769853 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-05-27T07:35:25.835 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:35:25.839+0000 7efc9d5be640 1 -- 172.21.15.52:0/4104769853 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55cf057ab1b0 msgr2=0x55cf05b5ccf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:35:25.835 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:35:25.839+0000 7efc9d5be640 1 --2- 172.21.15.52:0/4104769853 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55cf057ab1b0 0x55cf05b5ccf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:35:25.835 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:35:25.839+0000 7efc9d5be640 1 -- 172.21.15.52:0/4104769853 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55cf0578a4d0 msgr2=0x55cf05b5c7b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:35:25.835 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:35:25.839+0000 7efc9d5be640 1 --2- 172.21.15.52:0/4104769853 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55cf0578a4d0 0x55cf05b5c7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:35:25.835 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:35:25.839+0000 7efc9d5be640 1 -- 172.21.15.52:0/4104769853 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55cf05b60cf0 con 0x55cf057822f0 2024-05-27T07:35:25.835 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:35:25.840+0000 7efc9d5be640 1 --2- 172.21.15.52:0/4104769853 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55cf057822f0 0x55cf05b5c270 secure :-1 s=READY pgs=742 cs=0 l=1 rev1=1 crypto rx=0x55cf05b49ec0 tx=0x7efc94009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3c6d36b15a8f6517 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:35:25.835 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:35:25.840+0000 7efc80ff9640 10 client.0 ms_handle_connect on v2:172.21.15.52:3300/0 2024-05-27T07:35:25.836 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:35:25.840+0000 7efc80ff9640 1 -- 172.21.15.52:0/4104769853 <== mon.0 v2:172.21.15.52:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efc9404a040 con 0x55cf057822f0 2024-05-27T07:35:25.836 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:35:25.840+0000 7efca67ae740 1 -- 172.21.15.52:0/4104769853 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7efc8c0ab0e0 con 0x55cf057822f0 2024-05-27T07:35:25.836 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:35:25.841+0000 7efc80ff9640 1 -- 172.21.15.52:0/4104769853 <== mon.0 v2:172.21.15.52:3300/0 2 ==== osd_map(123..123 src has 1..123) ==== 9555+0+0 (secure 0 0 0) 0x7efc94031af0 con 0x55cf057822f0 2024-05-27T07:35:25.836 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:35:25.841+0000 7efc80ff9640 1 -- 172.21.15.52:0/4104769853 <== mon.0 v2:172.21.15.52:3300/0 3 ==== mon_get_version_reply(handle=1 version=108) ==== 24+0+0 (secure 0 0 0) 0x7efc94031cf0 con 0x55cf057822f0 2024-05-27T07:35:25.836 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:35:25.841+0000 7efca67ae740 10 client.17130 fetch_fsmap learned FSMap version 108 2024-05-27T07:35:25.837 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:35:25.841+0000 7efca67ae740 1 -- 172.21.15.52:0/4104769853 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({config=0+,fsmap=108,monmap=4+}) -- 0x55cf05bd0e70 con 0x55cf057822f0 2024-05-27T07:35:25.837 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:35:25.842+0000 7efc80ff9640 1 -- 172.21.15.52:0/4104769853 <== mon.0 v2:172.21.15.52:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7efc8c0ab0e0 con 0x55cf057822f0 2024-05-27T07:35:25.837 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:35:25.842+0000 7efc80ff9640 1 -- 172.21.15.52:0/4104769853 <== mon.0 v2:172.21.15.52:3300/0 5 ==== fsmap(e 108) ==== 4203+0+0 (secure 0 0 0) 0x7efc94043070 con 0x55cf057822f0 2024-05-27T07:35:25.837 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:35:25.842+0000 7efca67ae740 10 client.17130 fetch_fsmap finished waiting for FSMap version 108 2024-05-27T07:35:25.838 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:35:25.842+0000 7efca67ae740 10 client.17130 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-05-27T07:35:25.838 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:35:25.842+0000 7efca67ae740 20 client.17130 populate_metadata read hostname 'smithi052' 2024-05-27T07:35:25.838 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:35:25.842+0000 7efca67ae740 1 --2- 172.21.15.52:0/4104769853 >> [v2:172.21.15.105:6838/500348151,v1:172.21.15.105:6839/500348151] conn(0x55cf0573f1c0 0x55cf05bd3580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:35:25.838 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:35:25.842+0000 7efca67ae740 4 client.17130 mds_command: new command op to 24527 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-05-27T07:35:25.838 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:35:25.842+0000 7efca67ae740 1 -- 172.21.15.52:0/4104769853 --> [v2:172.21.15.105:6838/500348151,v1:172.21.15.105:6839/500348151] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55cf05b58d90 con 0x55cf0573f1c0 2024-05-27T07:35:25.840 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:35:25.844+0000 7efc9cdbd640 1 --2- 172.21.15.52:0/4104769853 >> [v2:172.21.15.105:6838/500348151,v1:172.21.15.105:6839/500348151] conn(0x55cf0573f1c0 0x55cf05bd3580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:35:25.841 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:35:25.846+0000 7efc9cdbd640 1 --2- 172.21.15.52:0/4104769853 >> [v2:172.21.15.105:6838/500348151,v1:172.21.15.105:6839/500348151] conn(0x55cf0573f1c0 0x55cf05bd3580 crc :-1 s=READY pgs=166 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=7b072d7f3c5d2bcc server_cookie=8319bf449253a952 in_seq=0 out_seq=0 2024-05-27T07:35:25.841 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:35:25.846+0000 7efc80ff9640 10 client.17130 ms_handle_connect on v2:172.21.15.105:6838/500348151 2024-05-27T07:35:26.607 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:35:26 smithi173 ceph-mon[37000]: pgmap v319: 129 pgs: 19 active+clean+snaptrim, 55 active+clean+snaptrim_wait, 55 active+clean; 649 MiB data, 6.2 GiB used, 1.0 TiB / 1.0 TiB avail; 24 KiB/s rd, 1.1 MiB/s wr, 204 op/s 2024-05-27T07:35:26.695 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:35:26 smithi105 ceph-mon[35977]: pgmap v319: 129 pgs: 19 active+clean+snaptrim, 55 active+clean+snaptrim_wait, 55 active+clean; 649 MiB data, 6.2 GiB used, 1.0 TiB / 1.0 TiB avail; 24 KiB/s rd, 1.1 MiB/s wr, 204 op/s 2024-05-27T07:35:26.700 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:35:26 smithi052 ceph-mon[31955]: pgmap v319: 129 pgs: 19 active+clean+snaptrim, 55 active+clean+snaptrim_wait, 55 active+clean; 649 MiB data, 6.2 GiB used, 1.0 TiB / 1.0 TiB avail; 24 KiB/s rd, 1.1 MiB/s wr, 204 op/s 2024-05-27T07:35:26.749 DEBUG:teuthology.orchestra.run.smithi052:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:38f9d900c83c63799fbdbe61acc9a11b0d3554a6 shell --fsid 73df2636-1bf8-11ef-bc9b-c7b262605968 -- ceph tell mds.1:1 damage ls 2024-05-27T07:35:27.181 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:35:27.186+0000 7efc80ff9640 1 -- 172.21.15.52:0/4104769853 <== mds.0 v2:172.21.15.105:6838/500348151 1 ==== command_reply(tid 0: 0 ) ==== 8+0+25 (crc 0 0 0) 0x55cf05b58d90 con 0x55cf0573f1c0 2024-05-27T07:35:27.181 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:35:27.186+0000 7efc80ff9640 10 client.17130 handle_command_reply: tid=0 2024-05-27T07:35:27.182 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:35:27.186+0000 7efca67ae740 1 client.17130 shutdown 2024-05-27T07:35:27.194 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:35:27.199+0000 7efca67ae740 1 -- 172.21.15.52:0/4104769853 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55cf057822f0 msgr2=0x55cf05b5c270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:35:27.194 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:35:27.199+0000 7efca67ae740 1 --2- 172.21.15.52:0/4104769853 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55cf057822f0 0x55cf05b5c270 secure :-1 s=READY pgs=742 cs=0 l=1 rev1=1 crypto rx=0x55cf05b49ec0 tx=0x7efc94009bd0 comp rx=0 tx=0).stop 2024-05-27T07:35:27.195 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:35:27.200+0000 7efca67ae740 1 -- 172.21.15.52:0/4104769853 shutdown_connections 2024-05-27T07:35:27.195 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:35:27.200+0000 7efca67ae740 1 --2- 172.21.15.52:0/4104769853 >> [v2:172.21.15.105:6838/500348151,v1:172.21.15.105:6839/500348151] conn(0x55cf0573f1c0 0x55cf05bd3580 crc :-1 s=READY pgs=166 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:35:27.195 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:35:27.200+0000 7efca67ae740 1 --2- 172.21.15.52:0/4104769853 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55cf057ab1b0 0x55cf05b5ccf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:35:27.196 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:35:27.200+0000 7efca67ae740 1 --2- 172.21.15.52:0/4104769853 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55cf0578a4d0 0x55cf05b5c7b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:35:27.196 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:35:27.200+0000 7efca67ae740 1 --2- 172.21.15.52:0/4104769853 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55cf057822f0 0x55cf05b5c270 unknown :-1 s=CLOSED pgs=742 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:35:27.196 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:35:27.200+0000 7efc80ff9640 0 client.17130 ms_handle_reset on v2:172.21.15.105:6838/500348151 2024-05-27T07:35:27.196 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:35:27.200+0000 7efca67ae740 1 -- 172.21.15.52:0/4104769853 >> 172.21.15.52:0/4104769853 conn(0x55cf0579f300 msgr2=0x55cf05b778a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:35:27.196 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:35:27.200+0000 7efca67ae740 1 -- 172.21.15.52:0/4104769853 shutdown_connections 2024-05-27T07:35:27.196 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:35:27.200+0000 7efca67ae740 1 -- 172.21.15.52:0/4104769853 wait complete. 2024-05-27T07:35:27.196 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:35:27.200+0000 7efca67ae740 20 client.17130 trim_cache size 0 max 16384 2024-05-27T07:35:27.196 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stdout:{ 2024-05-27T07:35:27.196 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stdout: "return_code": 0 2024-05-27T07:35:27.197 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stdout:} 2024-05-27T07:35:27.197 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:35:27.201+0000 7efc9f822640 1 -- 172.21.15.52:0/3546543659 >> [v2:172.21.15.52:6800/1822318384,v1:172.21.15.52:6801/1822318384] conn(0x7efc840b29d0 msgr2=0x7efc840b4ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:35:27.197 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:35:27.201+0000 7efc9f822640 1 --2- 172.21.15.52:0/3546543659 >> [v2:172.21.15.52:6800/1822318384,v1:172.21.15.52:6801/1822318384] conn(0x7efc840b29d0 0x7efc840b4ee0 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x55cf05b49540 tx=0x7efc90004730 comp rx=0 tx=0).stop 2024-05-27T07:35:27.197 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:35:27.201+0000 7efc9f822640 1 -- 172.21.15.52:0/3546543659 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7efc98108b90 msgr2=0x7efc98198c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:35:27.197 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:35:27.201+0000 7efc9f822640 1 --2- 172.21.15.52:0/3546543659 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7efc98108b90 0x7efc98198c40 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7efc8c010ad0 tx=0x7efc8c010f90 comp rx=0 tx=0).stop 2024-05-27T07:35:27.197 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:35:27.202+0000 7efc9f822640 1 -- 172.21.15.52:0/3546543659 shutdown_connections 2024-05-27T07:35:27.197 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:35:27.202+0000 7efc9f822640 1 --2- 172.21.15.52:0/3546543659 >> [v2:172.21.15.52:6800/1822318384,v1:172.21.15.52:6801/1822318384] conn(0x7efc840b29d0 0x7efc840b4ee0 unknown :-1 s=CLOSED pgs=184 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:35:27.198 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:35:27.202+0000 7efc9f822640 1 --2- 172.21.15.52:0/3546543659 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7efc98109030 0x7efc9819cf00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:35:27.198 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:35:27.202+0000 7efc9f822640 1 --2- 172.21.15.52:0/3546543659 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7efc98108b90 0x7efc98198c40 unknown :-1 s=CLOSED pgs=206 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:35:27.198 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:35:27.202+0000 7efc9f822640 1 --2- 172.21.15.52:0/3546543659 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7efc981032e0 0x7efc981986e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:35:27.198 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:35:27.202+0000 7efc9f822640 1 -- 172.21.15.52:0/3546543659 >> 172.21.15.52:0/3546543659 conn(0x7efc980782d0 msgr2=0x7efc980ff2a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:35:27.198 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:35:27.202+0000 7efc9f822640 1 -- 172.21.15.52:0/3546543659 shutdown_connections 2024-05-27T07:35:27.198 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:35:27.202+0000 7efc9f822640 1 -- 172.21.15.52:0/3546543659 wait complete. 2024-05-27T07:35:27.290 INFO:teuthology.orchestra.run.smithi052.stderr:Inferring config /var/lib/ceph/73df2636-1bf8-11ef-bc9b-c7b262605968/mon.a/config 2024-05-27T07:35:28.485 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:35:28 smithi052 ceph-mon[31955]: pgmap v320: 129 pgs: 19 active+clean+snaptrim, 55 active+clean+snaptrim_wait, 55 active+clean; 650 MiB data, 6.2 GiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 1.3 MiB/s wr, 199 op/s 2024-05-27T07:35:28.607 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:35:28 smithi173 ceph-mon[37000]: pgmap v320: 129 pgs: 19 active+clean+snaptrim, 55 active+clean+snaptrim_wait, 55 active+clean; 650 MiB data, 6.2 GiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 1.3 MiB/s wr, 199 op/s 2024-05-27T07:35:28.695 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:35:28 smithi105 ceph-mon[35977]: pgmap v320: 129 pgs: 19 active+clean+snaptrim, 55 active+clean+snaptrim_wait, 55 active+clean; 650 MiB data, 6.2 GiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 1.3 MiB/s wr, 199 op/s 2024-05-27T07:35:29.496 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:29.499+0000 7fcd5a7fc700 1 -- 172.21.15.52:0/2106765740 <== mon.0 v2:172.21.15.52:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcd4c00bb20 con 0x7fcd5c104410 2024-05-27T07:35:29.497 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:29.500+0000 7fcd60be7700 1 -- 172.21.15.52:0/2106765740 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fcd5c104410 msgr2=0x7fcd5c104820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:35:29.497 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:29.500+0000 7fcd60be7700 1 --2- 172.21.15.52:0/2106765740 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fcd5c104410 0x7fcd5c104820 secure :-1 s=READY pgs=743 cs=0 l=1 rev1=1 crypto rx=0x7fcd4c009990 tx=0x7fcd4c009ca0 comp rx=0 tx=0).stop 2024-05-27T07:35:29.502 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:29.505+0000 7fcd60be7700 1 -- 172.21.15.52:0/2106765740 shutdown_connections 2024-05-27T07:35:29.502 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:29.505+0000 7fcd60be7700 1 --2- 172.21.15.52:0/2106765740 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fcd5c109690 0x7fcd5c110ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:35:29.502 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:29.505+0000 7fcd60be7700 1 --2- 172.21.15.52:0/2106765740 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fcd5c104410 0x7fcd5c104820 unknown :-1 s=CLOSED pgs=743 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:35:29.502 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:29.505+0000 7fcd60be7700 1 --2- 172.21.15.52:0/2106765740 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fcd5c108d40 0x7fcd5c109150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:35:29.502 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:29.506+0000 7fcd60be7700 1 -- 172.21.15.52:0/2106765740 >> 172.21.15.52:0/2106765740 conn(0x7fcd5c07aca0 msgr2=0x7fcd5c07b100 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:35:29.505 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:29.508+0000 7fcd60be7700 1 -- 172.21.15.52:0/2106765740 shutdown_connections 2024-05-27T07:35:29.505 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:29.508+0000 7fcd60be7700 1 -- 172.21.15.52:0/2106765740 wait complete. 2024-05-27T07:35:29.505 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:29.509+0000 7fcd60be7700 1 Processor -- start 2024-05-27T07:35:29.505 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:29.509+0000 7fcd60be7700 1 -- start start 2024-05-27T07:35:29.505 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:29.509+0000 7fcd60be7700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fcd5c104410 0x7fcd5c075330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:35:29.506 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:29.509+0000 7fcd60be7700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fcd5c108d40 0x7fcd5c06e4b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:35:29.506 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:29.509+0000 7fcd60be7700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fcd5c109690 0x7fcd5c06ea70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:35:29.506 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:29.509+0000 7fcd60be7700 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x7fcd5c112140 con 0x7fcd5c109690 2024-05-27T07:35:29.506 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:29.509+0000 7fcd60be7700 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7fcd5c111fc0 con 0x7fcd5c104410 2024-05-27T07:35:29.506 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:29.509+0000 7fcd60be7700 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7fcd5c1122c0 con 0x7fcd5c108d40 2024-05-27T07:35:29.512 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:29.511+0000 7fcd5bfff700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fcd5c109690 0x7fcd5c06ea70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:35:29.512 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:29.511+0000 7fcd5bfff700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fcd5c109690 0x7fcd5c06ea70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.52:57422/0 (socket says 172.21.15.52:57422) 2024-05-27T07:35:29.512 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:29.511+0000 7fcd5bfff700 1 -- 172.21.15.52:0/442605786 learned_addr learned my addr 172.21.15.52:0/442605786 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-05-27T07:35:29.512 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:29.511+0000 7fcd5affd700 1 --2- 172.21.15.52:0/442605786 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fcd5c108d40 0x7fcd5c06e4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:35:29.512 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:29.511+0000 7fcd5bfff700 1 -- 172.21.15.52:0/442605786 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fcd5c108d40 msgr2=0x7fcd5c06e4b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:35:29.512 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:29.511+0000 7fcd5bfff700 1 --2- 172.21.15.52:0/442605786 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fcd5c108d40 0x7fcd5c06e4b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:35:29.513 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:29.511+0000 7fcd5bfff700 1 -- 172.21.15.52:0/442605786 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fcd5c104410 msgr2=0x7fcd5c075330 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-05-27T07:35:29.513 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:29.511+0000 7fcd5bfff700 1 --2- 172.21.15.52:0/442605786 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fcd5c104410 0x7fcd5c075330 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:35:29.513 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:29.511+0000 7fcd5bfff700 1 -- 172.21.15.52:0/442605786 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcd5c06f2d0 con 0x7fcd5c109690 2024-05-27T07:35:29.513 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:29.513+0000 7fcd5bfff700 1 --2- 172.21.15.52:0/442605786 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fcd5c109690 0x7fcd5c06ea70 secure :-1 s=READY pgs=744 cs=0 l=1 rev1=1 crypto rx=0x7fcd5400ada0 tx=0x7fcd5400c330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:35:29.513 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:29.518+0000 7fcd58ff9700 1 -- 172.21.15.52:0/442605786 <== mon.0 v2:172.21.15.52:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcd54017020 con 0x7fcd5c109690 2024-05-27T07:35:29.513 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:29.518+0000 7fcd58ff9700 1 -- 172.21.15.52:0/442605786 <== mon.0 v2:172.21.15.52:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fcd5400ce30 con 0x7fcd5c109690 2024-05-27T07:35:29.514 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:29.518+0000 7fcd58ff9700 1 -- 172.21.15.52:0/442605786 <== mon.0 v2:172.21.15.52:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcd54005520 con 0x7fcd5c109690 2024-05-27T07:35:29.518 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:29.519+0000 7fcd60be7700 1 -- 172.21.15.52:0/442605786 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fcd5c072f80 con 0x7fcd5c109690 2024-05-27T07:35:29.518 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:29.519+0000 7fcd60be7700 1 -- 172.21.15.52:0/442605786 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fcd5c073470 con 0x7fcd5c109690 2024-05-27T07:35:29.520 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:29.525+0000 7fcd7085eb80 1 Processor -- start 2024-05-27T07:35:29.523 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:29.527+0000 7fcd58ff9700 1 -- 172.21.15.52:0/442605786 <== mon.0 v2:172.21.15.52:3300/0 4 ==== mgrmap(e 26) ==== 149783+0+0 (secure 0 0 0) 0x7fcd54004ad0 con 0x7fcd5c109690 2024-05-27T07:35:29.523 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:29.527+0000 7fcd7085eb80 1 -- start start 2024-05-27T07:35:29.525 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:29.528+0000 7fcd7085eb80 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55a3e513de40 0x55a3e52c1880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:35:29.525 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:29.528+0000 7fcd7085eb80 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55a3e52c1dc0 0x55a3e52bba40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:35:29.525 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:29.528+0000 7fcd7085eb80 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55a3e52bbf80 0x55a3e52bc390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:35:29.525 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:29.528+0000 7fcd7085eb80 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x55a3e51eac90 con 0x55a3e52bbf80 2024-05-27T07:35:29.526 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:29.528+0000 7fcd7085eb80 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x55a3e51da830 con 0x55a3e52c1dc0 2024-05-27T07:35:29.526 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:29.528+0000 7fcd7085eb80 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x55a3e51de870 con 0x55a3e513de40 2024-05-27T07:35:29.526 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:29.529+0000 7fcd58ff9700 1 --2- 172.21.15.52:0/442605786 >> [v2:172.21.15.52:6800/1822318384,v1:172.21.15.52:6801/1822318384] conn(0x7fcd440b2c00 0x7fcd440b5110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:35:29.526 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:29.529+0000 7fcd58ff9700 1 -- 172.21.15.52:0/442605786 <== mon.0 v2:172.21.15.52:3300/0 5 ==== osd_map(123..123 src has 1..123) ==== 9555+0+0 (secure 0 0 0) 0x7fcd540e2130 con 0x7fcd5c109690 2024-05-27T07:35:29.526 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:29.530+0000 7fcd5affd700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55a3e52bbf80 0x55a3e52bc390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:35:29.526 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:29.530+0000 7fcd5affd700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55a3e52bbf80 0x55a3e52bc390 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.52:57432/0 (socket says 172.21.15.52:57432) 2024-05-27T07:35:29.526 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:29.530+0000 7fcd5affd700 1 -- 172.21.15.52:0/640840429 learned_addr learned my addr 172.21.15.52:0/640840429 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-05-27T07:35:29.526 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:29.530+0000 7fcd5affd700 1 --2- 172.21.15.52:0/640840429 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55a3e513de40 0x55a3e52c1880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:35:29.526 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:29.530+0000 7fcd5bfff700 1 --2- 172.21.15.52:0/442605786 >> [v2:172.21.15.52:6800/1822318384,v1:172.21.15.52:6801/1822318384] conn(0x7fcd440b2c00 0x7fcd440b5110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:35:29.527 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:29.531+0000 7fcd5bfff700 1 --2- 172.21.15.52:0/442605786 >> [v2:172.21.15.52:6800/1822318384,v1:172.21.15.52:6801/1822318384] conn(0x7fcd440b2c00 0x7fcd440b5110 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7fcd540a8fc0 tx=0x7fcd5400a2e0 comp rx=0 tx=0).ready entity=mgr.15333 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:35:29.528 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:29.533+0000 7fcd5affd700 1 -- 172.21.15.52:0/640840429 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55a3e52c1dc0 msgr2=0x55a3e52bba40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:35:29.529 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:29.533+0000 7fcd5affd700 1 --2- 172.21.15.52:0/640840429 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55a3e52c1dc0 0x55a3e52bba40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:35:29.529 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:29.533+0000 7fcd5affd700 1 -- 172.21.15.52:0/640840429 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55a3e52bbf80 msgr2=0x55a3e52bc390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:35:29.529 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:29.533+0000 7fcd5affd700 1 --2- 172.21.15.52:0/640840429 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55a3e52bbf80 0x55a3e52bc390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:35:29.529 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:29.533+0000 7fcd5affd700 1 -- 172.21.15.52:0/640840429 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55a3e512a8d0 con 0x55a3e513de40 2024-05-27T07:35:29.529 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:29.533+0000 7fcd5affd700 1 --2- 172.21.15.52:0/640840429 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55a3e513de40 0x55a3e52c1880 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7fcd50009e80 tx=0x7fcd50009260 comp rx=0 tx=0).ready entity=mon.1 client_cookie=df619f6406cbf62d server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:35:29.533 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:29.536+0000 7fcd41ffb700 1 -- 172.21.15.52:0/640840429 <== mon.1 v2:172.21.15.173:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcd5001e640 con 0x55a3e513de40 2024-05-27T07:35:29.533 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:29.536+0000 7fcd41ffb700 1 -- 172.21.15.52:0/640840429 <== mon.1 v2:172.21.15.173:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fcd5001ec90 con 0x55a3e513de40 2024-05-27T07:35:29.533 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:29.537+0000 7fcd7085eb80 1 -- 172.21.15.52:0/640840429 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55a3e513de40 msgr2=0x55a3e52c1880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:35:29.533 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:29.537+0000 7fcd7085eb80 1 --2- 172.21.15.52:0/640840429 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55a3e513de40 0x55a3e52c1880 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7fcd50009e80 tx=0x7fcd50009260 comp rx=0 tx=0).stop 2024-05-27T07:35:29.534 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:29.539+0000 7fcd7085eb80 1 -- 172.21.15.52:0/640840429 shutdown_connections 2024-05-27T07:35:29.534 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:29.539+0000 7fcd7085eb80 1 --2- 172.21.15.52:0/640840429 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55a3e52bbf80 0x55a3e52bc390 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:35:29.534 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:29.539+0000 7fcd7085eb80 1 --2- 172.21.15.52:0/640840429 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55a3e513de40 0x55a3e52c1880 unknown :-1 s=CLOSED pgs=208 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:35:29.534 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:29.539+0000 7fcd7085eb80 1 --2- 172.21.15.52:0/640840429 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55a3e52c1dc0 0x55a3e52bba40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:35:29.534 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:29.539+0000 7fcd7085eb80 1 -- 172.21.15.52:0/640840429 >> 172.21.15.52:0/640840429 conn(0x55a3e5198af0 msgr2=0x55a3e52b6740 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:35:29.534 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:29.539+0000 7fcd7085eb80 1 -- 172.21.15.52:0/640840429 shutdown_connections 2024-05-27T07:35:29.537 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:29.540+0000 7fcd7085eb80 1 -- 172.21.15.52:0/640840429 wait complete. 2024-05-27T07:35:29.537 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:29.541+0000 7fcd7085eb80 1 -- start start 2024-05-27T07:35:29.537 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:29.541+0000 7fcd7085eb80 1 Processor -- start 2024-05-27T07:35:29.537 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:29.542+0000 7fcd7085eb80 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55a3e513de40 0x55a3e52c5640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:35:29.538 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:29.542+0000 7fcd7085eb80 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55a3e52bbf80 0x55a3e52c5b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:35:29.538 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:29.542+0000 7fcd7085eb80 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55a3e52c1dc0 0x55a3e52c60c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:35:29.538 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:29.542+0000 7fcd7085eb80 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x55a3e50e0fa0 con 0x55a3e513de40 2024-05-27T07:35:29.538 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:29.542+0000 7fcd7085eb80 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x55a3e517c7c0 con 0x55a3e52bbf80 2024-05-27T07:35:29.538 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:29.542+0000 7fcd7085eb80 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x55a3e52c88e0 con 0x55a3e52c1dc0 2024-05-27T07:35:29.538 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:29.542+0000 7fcd7085eb80 10 client.0 _ll_register_callbacks cb 0x55a3e5175920 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T07:35:29.538 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:29.542+0000 7fcd5affd700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55a3e52c1dc0 0x55a3e52c60c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:35:29.539 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:29.543+0000 7fcd5affd700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55a3e52c1dc0 0x55a3e52c60c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.173:3300/0 says I am v2:172.21.15.52:42168/0 (socket says 172.21.15.52:42168) 2024-05-27T07:35:29.539 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:29.543+0000 7fcd5affd700 1 -- 172.21.15.52:0/160079867 learned_addr learned my addr 172.21.15.52:0/160079867 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-05-27T07:35:29.542 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:29.544+0000 7fcd5b7fe700 1 --2- 172.21.15.52:0/160079867 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55a3e52bbf80 0x55a3e52c5b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:35:29.542 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:29.545+0000 7fcd5affd700 1 --2- 172.21.15.52:0/160079867 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55a3e513de40 0x55a3e52c5640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:35:29.542 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:29.546+0000 7fcd5affd700 1 -- 172.21.15.52:0/160079867 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55a3e52bbf80 msgr2=0x55a3e52c5b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:35:29.542 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:29.546+0000 7fcd5affd700 1 --2- 172.21.15.52:0/160079867 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55a3e52bbf80 0x55a3e52c5b80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:35:29.542 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:29.546+0000 7fcd5affd700 1 -- 172.21.15.52:0/160079867 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55a3e513de40 msgr2=0x55a3e52c5640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:35:29.542 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:29.546+0000 7fcd5affd700 1 --2- 172.21.15.52:0/160079867 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55a3e513de40 0x55a3e52c5640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:35:29.543 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:29.546+0000 7fcd5affd700 1 -- 172.21.15.52:0/160079867 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55a3e5179690 con 0x55a3e52c1dc0 2024-05-27T07:35:29.543 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:29.546+0000 7fcd5affd700 1 --2- 172.21.15.52:0/160079867 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55a3e52c1dc0 0x55a3e52c60c0 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7fcd50009a90 tx=0x7fcd50009a00 comp rx=0 tx=0).ready entity=mon.1 client_cookie=337bad5f2b787291 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:35:29.543 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:29.546+0000 7fcd3affd700 10 client.0 ms_handle_connect on v2:172.21.15.173:3300/0 2024-05-27T07:35:29.543 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:29.547+0000 7fcd3affd700 1 -- 172.21.15.52:0/160079867 <== mon.1 v2:172.21.15.173:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcd50005510 con 0x55a3e52c1dc0 2024-05-27T07:35:29.543 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:29.547+0000 7fcd7085eb80 1 -- 172.21.15.52:0/160079867 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x55a3e52879f0 con 0x55a3e52c1dc0 2024-05-27T07:35:29.543 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:29.547+0000 7fcd3affd700 1 -- 172.21.15.52:0/160079867 <== mon.1 v2:172.21.15.173:3300/0 2 ==== osd_map(123..123 src has 1..123) ==== 9555+0+0 (secure 0 0 0) 0x7fcd50003680 con 0x55a3e52c1dc0 2024-05-27T07:35:29.546 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:29.547+0000 7fcd3affd700 1 -- 172.21.15.52:0/160079867 <== mon.1 v2:172.21.15.173:3300/0 3 ==== mon_get_version_reply(handle=1 version=108) ==== 24+0+0 (secure 0 0 0) 0x7fcd50030050 con 0x55a3e52c1dc0 2024-05-27T07:35:29.546 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:29.548+0000 7fcd7085eb80 10 client.26623 fetch_fsmap learned FSMap version 108 2024-05-27T07:35:29.546 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:29.548+0000 7fcd7085eb80 1 -- 172.21.15.52:0/160079867 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({config=0+,fsmap=108,monmap=4+}) -- 0x55a3e52d3520 con 0x55a3e52c1dc0 2024-05-27T07:35:29.546 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:29.549+0000 7fcd3affd700 1 -- 172.21.15.52:0/160079867 <== mon.1 v2:172.21.15.173:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x55a3e52879f0 con 0x55a3e52c1dc0 2024-05-27T07:35:29.546 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:29.549+0000 7fcd3affd700 1 -- 172.21.15.52:0/160079867 <== mon.1 v2:172.21.15.173:3300/0 5 ==== fsmap(e 108) ==== 4203+0+0 (secure 0 0 0) 0x7fcd5001c020 con 0x55a3e52c1dc0 2024-05-27T07:35:29.546 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:29.549+0000 7fcd7085eb80 10 client.26623 fetch_fsmap finished waiting for FSMap version 108 2024-05-27T07:35:29.546 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:29.549+0000 7fcd7085eb80 10 client.26623 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.h 2024-05-27T07:35:29.546 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:29.550+0000 7fcd7085eb80 20 client.26623 populate_metadata read hostname 'smithi052' 2024-05-27T07:35:29.546 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:29.550+0000 7fcd7085eb80 1 --2- 172.21.15.52:0/160079867 >> [v2:172.21.15.105:6836/191393149,v1:172.21.15.105:6837/191393149] conn(0x55a3e52d5d30 0x55a3e52d8200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:35:29.546 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:29.550+0000 7fcd7085eb80 4 client.26623 mds_command: new command op to 24497 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-27T07:35:29.547 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:29.550+0000 7fcd7085eb80 1 -- 172.21.15.52:0/160079867 --> [v2:172.21.15.105:6836/191393149,v1:172.21.15.105:6837/191393149] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55a3e51ecbe0 con 0x55a3e52d5d30 2024-05-27T07:35:29.552 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:29.556+0000 7fcd5b7fe700 1 --2- 172.21.15.52:0/160079867 >> [v2:172.21.15.105:6836/191393149,v1:172.21.15.105:6837/191393149] conn(0x55a3e52d5d30 0x55a3e52d8200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:35:29.556 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:29.559+0000 7fcd5b7fe700 1 --2- 172.21.15.52:0/160079867 >> [v2:172.21.15.105:6836/191393149,v1:172.21.15.105:6837/191393149] conn(0x55a3e52d5d30 0x55a3e52d8200 crc :-1 s=READY pgs=49 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=edc3c8e56a6948dd server_cookie=e82b5cb34cfc2506 in_seq=0 out_seq=0 2024-05-27T07:35:29.556 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:29.559+0000 7fcd3affd700 10 client.26623 ms_handle_connect on v2:172.21.15.105:6836/191393149 2024-05-27T07:35:29.557 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:29.562+0000 7fcd3affd700 1 -- 172.21.15.52:0/160079867 <== mds.1 v2:172.21.15.105:6836/191393149 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24041 (crc 0 0 0) 0x55a3e51ecbe0 con 0x55a3e52d5d30 2024-05-27T07:35:29.557 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:29.562+0000 7fcd3affd700 10 client.26623 handle_command_reply: tid=0 2024-05-27T07:35:29.560 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:29.564+0000 7fcd7085eb80 1 client.26623 shutdown 2024-05-27T07:35:29.567 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:29.572+0000 7fcd7085eb80 1 -- 172.21.15.52:0/160079867 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55a3e52c1dc0 msgr2=0x55a3e52c60c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:35:29.567 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:29.572+0000 7fcd7085eb80 1 --2- 172.21.15.52:0/160079867 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55a3e52c1dc0 0x55a3e52c60c0 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7fcd50009a90 tx=0x7fcd50009a00 comp rx=0 tx=0).stop 2024-05-27T07:35:29.571 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:29.574+0000 7fcd7085eb80 1 -- 172.21.15.52:0/160079867 shutdown_connections 2024-05-27T07:35:29.571 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:29.574+0000 7fcd7085eb80 1 --2- 172.21.15.52:0/160079867 >> [v2:172.21.15.105:6836/191393149,v1:172.21.15.105:6837/191393149] conn(0x55a3e52d5d30 0x55a3e52d8200 crc :-1 s=READY pgs=49 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:35:29.571 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:29.574+0000 7fcd7085eb80 1 --2- 172.21.15.52:0/160079867 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55a3e52c1dc0 0x55a3e52c60c0 unknown :-1 s=CLOSED pgs=209 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:35:29.571 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:29.574+0000 7fcd7085eb80 1 --2- 172.21.15.52:0/160079867 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55a3e513de40 0x55a3e52c5640 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:35:29.571 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:29.574+0000 7fcd7085eb80 1 --2- 172.21.15.52:0/160079867 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55a3e52bbf80 0x55a3e52c5b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:35:29.572 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:29.574+0000 7fcd7085eb80 1 -- 172.21.15.52:0/160079867 >> 172.21.15.52:0/160079867 conn(0x55a3e5198af0 msgr2=0x55a3e52baa30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:35:29.572 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:29.575+0000 7fcd3affd700 0 client.26623 ms_handle_reset on v2:172.21.15.105:6836/191393149 2024-05-27T07:35:29.574 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:29.578+0000 7fcd7085eb80 1 -- 172.21.15.52:0/160079867 shutdown_connections 2024-05-27T07:35:29.574 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:29.578+0000 7fcd7085eb80 1 -- 172.21.15.52:0/160079867 wait complete. 2024-05-27T07:35:29.574 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:29.578+0000 7fcd7085eb80 20 client.26623 trim_cache size 0 max 16384 2024-05-27T07:35:29.604 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:29.608+0000 7fcd7085eb80 1 Processor -- start 2024-05-27T07:35:29.604 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:29.608+0000 7fcd7085eb80 1 -- start start 2024-05-27T07:35:29.605 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:29.609+0000 7fcd7085eb80 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55a3e513de40 0x55a3e52bba40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:35:29.605 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:29.609+0000 7fcd7085eb80 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55a3e52bbf80 0x55a3e52b6830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:35:29.605 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:29.609+0000 7fcd7085eb80 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55a3e52c1dc0 0x55a3e52b0e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:35:29.605 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:29.609+0000 7fcd7085eb80 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x55a3e52d8470 con 0x55a3e52bbf80 2024-05-27T07:35:29.606 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:29.609+0000 7fcd7085eb80 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7fcd5c1b7240 con 0x55a3e52c1dc0 2024-05-27T07:35:29.606 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:29.609+0000 7fcd7085eb80 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7fcd5c1b5670 con 0x55a3e513de40 2024-05-27T07:35:29.606 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:29.609+0000 7fcd5b7fe700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55a3e52bbf80 0x55a3e52b6830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:35:29.606 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:29.609+0000 7fcd5b7fe700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55a3e52bbf80 0x55a3e52b6830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.52:57452/0 (socket says 172.21.15.52:57452) 2024-05-27T07:35:29.606 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:29.609+0000 7fcd5b7fe700 1 -- 172.21.15.52:0/1462490412 learned_addr learned my addr 172.21.15.52:0/1462490412 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-05-27T07:35:29.606 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:29.610+0000 7fcd5affd700 1 --2- 172.21.15.52:0/1462490412 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55a3e513de40 0x55a3e52bba40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:35:29.606 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:29.610+0000 7fcd5affd700 1 --2- 172.21.15.52:0/1462490412 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55a3e52c1dc0 0x55a3e52b0e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:35:29.606 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:29.610+0000 7fcd5b7fe700 1 -- 172.21.15.52:0/1462490412 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55a3e513de40 msgr2=0x55a3e52bba40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:35:29.606 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:29.610+0000 7fcd5b7fe700 1 --2- 172.21.15.52:0/1462490412 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55a3e513de40 0x55a3e52bba40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:35:29.606 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:29.610+0000 7fcd5b7fe700 1 -- 172.21.15.52:0/1462490412 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55a3e52c1dc0 msgr2=0x55a3e52b0e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:35:29.606 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:29.610+0000 7fcd5b7fe700 1 --2- 172.21.15.52:0/1462490412 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55a3e52c1dc0 0x55a3e52b0e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:35:29.607 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:29.610+0000 7fcd5b7fe700 1 -- 172.21.15.52:0/1462490412 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55a3e52b8080 con 0x55a3e52bbf80 2024-05-27T07:35:29.607 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:29.611+0000 7fcd5b7fe700 1 --2- 172.21.15.52:0/1462490412 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55a3e52bbf80 0x55a3e52b6830 secure :-1 s=READY pgs=745 cs=0 l=1 rev1=1 crypto rx=0x7fcd5c06fcb0 tx=0x7fcd4c0096a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d7f199f78da39609 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:35:29.617 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:29.620+0000 7fcd417fa700 1 -- 172.21.15.52:0/1462490412 <== mon.0 v2:172.21.15.52:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcd4c02a070 con 0x55a3e52bbf80 2024-05-27T07:35:29.618 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:29.620+0000 7fcd417fa700 1 -- 172.21.15.52:0/1462490412 <== mon.0 v2:172.21.15.52:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fcd4c00bc30 con 0x55a3e52bbf80 2024-05-27T07:35:29.618 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:29.620+0000 7fcd417fa700 1 -- 172.21.15.52:0/1462490412 <== mon.0 v2:172.21.15.52:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcd4c003d50 con 0x55a3e52bbf80 2024-05-27T07:35:29.618 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:29.622+0000 7fcd7085eb80 1 -- 172.21.15.52:0/1462490412 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55a3e52bbf80 msgr2=0x55a3e52b6830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:35:29.618 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:29.622+0000 7fcd7085eb80 1 --2- 172.21.15.52:0/1462490412 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55a3e52bbf80 0x55a3e52b6830 secure :-1 s=READY pgs=745 cs=0 l=1 rev1=1 crypto rx=0x7fcd5c06fcb0 tx=0x7fcd4c0096a0 comp rx=0 tx=0).stop 2024-05-27T07:35:29.618 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:29.622+0000 7fcd7085eb80 1 -- 172.21.15.52:0/1462490412 shutdown_connections 2024-05-27T07:35:29.618 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:29.622+0000 7fcd7085eb80 1 --2- 172.21.15.52:0/1462490412 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55a3e52c1dc0 0x55a3e52b0e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:35:29.618 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:29.622+0000 7fcd7085eb80 1 --2- 172.21.15.52:0/1462490412 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55a3e513de40 0x55a3e52bba40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:35:29.618 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:29.622+0000 7fcd7085eb80 1 --2- 172.21.15.52:0/1462490412 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55a3e52bbf80 0x55a3e52b6830 unknown :-1 s=CLOSED pgs=745 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:35:29.619 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:29.622+0000 7fcd7085eb80 1 -- 172.21.15.52:0/1462490412 >> 172.21.15.52:0/1462490412 conn(0x55a3e5198af0 msgr2=0x55a3e52c5600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:35:29.625 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:29.630+0000 7fcd7085eb80 1 -- 172.21.15.52:0/1462490412 shutdown_connections 2024-05-27T07:35:29.626 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:29.630+0000 7fcd7085eb80 1 -- 172.21.15.52:0/1462490412 wait complete. 2024-05-27T07:35:29.626 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:29.630+0000 7fcd7085eb80 1 -- start start 2024-05-27T07:35:29.626 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:29.631+0000 7fcd7085eb80 1 Processor -- start 2024-05-27T07:35:29.627 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:29.631+0000 7fcd7085eb80 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55a3e513de40 0x55a3e52de170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:35:29.628 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:29.631+0000 7fcd7085eb80 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55a3e52bbf80 0x55a3e52cdb80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:35:29.629 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:29.632+0000 7fcd7085eb80 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55a3e52c1dc0 0x55a3e52ce180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:35:29.629 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:29.632+0000 7fcd7085eb80 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x7fcd5c1b5d70 con 0x55a3e52bbf80 2024-05-27T07:35:29.629 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:29.632+0000 7fcd7085eb80 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7fcd5c1b6b40 con 0x55a3e52c1dc0 2024-05-27T07:35:29.629 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:29.632+0000 7fcd7085eb80 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7fcd5c1b6470 con 0x55a3e513de40 2024-05-27T07:35:29.629 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:29.632+0000 7fcd7085eb80 10 client.0 _ll_register_callbacks cb 0x55a3e5175920 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T07:35:29.629 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:29.632+0000 7fcd5affd700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55a3e513de40 0x55a3e52de170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:35:29.629 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:29.632+0000 7fcd5affd700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55a3e52c1dc0 0x55a3e52ce180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:35:29.629 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:29.633+0000 7fcd5affd700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55a3e513de40 0x55a3e52de170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.173:3300/0 says I am v2:172.21.15.52:42180/0 (socket says 172.21.15.52:42180) 2024-05-27T07:35:29.630 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:29.633+0000 7fcd5affd700 1 -- 172.21.15.52:0/3172887125 learned_addr learned my addr 172.21.15.52:0/3172887125 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-05-27T07:35:29.630 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:29.634+0000 7fcd5b7fe700 1 --2- 172.21.15.52:0/3172887125 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55a3e52bbf80 0x55a3e52cdb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:35:29.630 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:29.634+0000 7fcd5affd700 1 -- 172.21.15.52:0/3172887125 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55a3e52c1dc0 msgr2=0x55a3e52ce180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:35:29.630 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:29.634+0000 7fcd5affd700 1 --2- 172.21.15.52:0/3172887125 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55a3e52c1dc0 0x55a3e52ce180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:35:29.630 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:29.634+0000 7fcd5affd700 1 -- 172.21.15.52:0/3172887125 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55a3e52bbf80 msgr2=0x55a3e52cdb80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:35:29.630 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:29.634+0000 7fcd5affd700 1 --2- 172.21.15.52:0/3172887125 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55a3e52bbf80 0x55a3e52cdb80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:35:29.631 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:29.634+0000 7fcd5affd700 1 -- 172.21.15.52:0/3172887125 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55a3e52dc9f0 con 0x55a3e513de40 2024-05-27T07:35:29.631 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:29.635+0000 7fcd5affd700 1 --2- 172.21.15.52:0/3172887125 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55a3e513de40 0x55a3e52de170 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7fcd50012fd0 tx=0x7fcd50009a00 comp rx=0 tx=0).ready entity=mon.1 client_cookie=a9f0ce5459339d6f server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:35:29.631 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:29.635+0000 7fcd3affd700 10 client.0 ms_handle_connect on v2:172.21.15.173:3300/0 2024-05-27T07:35:29.631 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:29.635+0000 7fcd3affd700 1 -- 172.21.15.52:0/3172887125 <== mon.1 v2:172.21.15.173:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcd5002bbb0 con 0x55a3e513de40 2024-05-27T07:35:29.631 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:29.635+0000 7fcd7085eb80 1 -- 172.21.15.52:0/3172887125 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fcd4c00bc30 con 0x55a3e513de40 2024-05-27T07:35:29.631 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:29.635+0000 7fcd3affd700 1 -- 172.21.15.52:0/3172887125 <== mon.1 v2:172.21.15.173:3300/0 2 ==== osd_map(123..123 src has 1..123) ==== 9555+0+0 (secure 0 0 0) 0x7fcd5002bd50 con 0x55a3e513de40 2024-05-27T07:35:29.632 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:29.636+0000 7fcd3affd700 1 -- 172.21.15.52:0/3172887125 <== mon.1 v2:172.21.15.173:3300/0 3 ==== mon_get_version_reply(handle=1 version=108) ==== 24+0+0 (secure 0 0 0) 0x7fcd5002a3b0 con 0x55a3e513de40 2024-05-27T07:35:29.632 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:29.637+0000 7fcd7085eb80 10 client.26629 fetch_fsmap learned FSMap version 108 2024-05-27T07:35:29.632 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:29.637+0000 7fcd7085eb80 1 -- 172.21.15.52:0/3172887125 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({config=0+,fsmap=108,monmap=4+}) -- 0x55a3e533fae0 con 0x55a3e513de40 2024-05-27T07:35:29.633 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:29.638+0000 7fcd3affd700 1 -- 172.21.15.52:0/3172887125 <== mon.1 v2:172.21.15.173:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fcd4c00bc30 con 0x55a3e513de40 2024-05-27T07:35:29.634 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:29.638+0000 7fcd3affd700 1 -- 172.21.15.52:0/3172887125 <== mon.1 v2:172.21.15.173:3300/0 5 ==== fsmap(e 108) ==== 4203+0+0 (secure 0 0 0) 0x7fcd5001c040 con 0x55a3e513de40 2024-05-27T07:35:29.634 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:29.639+0000 7fcd7085eb80 10 client.26629 fetch_fsmap finished waiting for FSMap version 108 2024-05-27T07:35:29.634 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:29.639+0000 7fcd7085eb80 10 client.26629 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.h 2024-05-27T07:35:29.635 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:29.639+0000 7fcd7085eb80 20 client.26629 populate_metadata read hostname 'smithi052' 2024-05-27T07:35:29.635 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:29.639+0000 7fcd7085eb80 1 --2- 172.21.15.52:0/3172887125 >> [v2:172.21.15.105:6836/191393149,v1:172.21.15.105:6837/191393149] conn(0x55a3e5340080 0x55a3e53424a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:35:29.635 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:29.639+0000 7fcd7085eb80 4 client.26629 mds_command: new command op to 24497 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-05-27T07:35:29.635 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:29.639+0000 7fcd7085eb80 1 -- 172.21.15.52:0/3172887125 --> [v2:172.21.15.105:6836/191393149,v1:172.21.15.105:6837/191393149] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55a3e51e63c0 con 0x55a3e5340080 2024-05-27T07:35:29.635 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:29.639+0000 7fcd5b7fe700 1 --2- 172.21.15.52:0/3172887125 >> [v2:172.21.15.105:6836/191393149,v1:172.21.15.105:6837/191393149] conn(0x55a3e5340080 0x55a3e53424a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:35:29.637 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:29.640+0000 7fcd5b7fe700 1 --2- 172.21.15.52:0/3172887125 >> [v2:172.21.15.105:6836/191393149,v1:172.21.15.105:6837/191393149] conn(0x55a3e5340080 0x55a3e53424a0 crc :-1 s=READY pgs=50 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=85ebbf87d92a0894 server_cookie=165ebb71f14ddd8d in_seq=0 out_seq=0 2024-05-27T07:35:29.637 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:29.640+0000 7fcd3affd700 10 client.26629 ms_handle_connect on v2:172.21.15.105:6836/191393149 2024-05-27T07:35:29.637 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:29.640+0000 7fcd3affd700 1 -- 172.21.15.52:0/3172887125 <== mds.1 v2:172.21.15.105:6836/191393149 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55a3e51e63c0 con 0x55a3e5340080 2024-05-27T07:35:29.637 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:29.640+0000 7fcd3affd700 10 client.26629 handle_command_reply: tid=0 2024-05-27T07:35:29.638 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:29.640+0000 7fcd7085eb80 1 client.26629 shutdown 2024-05-27T07:35:29.638 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:29.641+0000 7fcd7085eb80 1 -- 172.21.15.52:0/3172887125 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55a3e513de40 msgr2=0x55a3e52de170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:35:29.638 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:29.641+0000 7fcd7085eb80 1 --2- 172.21.15.52:0/3172887125 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55a3e513de40 0x55a3e52de170 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7fcd50012fd0 tx=0x7fcd50009a00 comp rx=0 tx=0).stop 2024-05-27T07:35:29.638 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:29.641+0000 7fcd7085eb80 1 -- 172.21.15.52:0/3172887125 shutdown_connections 2024-05-27T07:35:29.638 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:29.641+0000 7fcd7085eb80 1 --2- 172.21.15.52:0/3172887125 >> [v2:172.21.15.105:6836/191393149,v1:172.21.15.105:6837/191393149] conn(0x55a3e5340080 0x55a3e53424a0 crc :-1 s=READY pgs=50 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:35:29.639 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:29.641+0000 7fcd7085eb80 1 --2- 172.21.15.52:0/3172887125 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55a3e52c1dc0 0x55a3e52ce180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:35:29.639 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:29.641+0000 7fcd7085eb80 1 --2- 172.21.15.52:0/3172887125 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55a3e513de40 0x55a3e52de170 unknown :-1 s=CLOSED pgs=210 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:35:29.639 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:29.641+0000 7fcd7085eb80 1 --2- 172.21.15.52:0/3172887125 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55a3e52bbf80 0x55a3e52cdb80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:35:29.639 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:29.641+0000 7fcd3affd700 0 client.26629 ms_handle_reset on v2:172.21.15.105:6836/191393149 2024-05-27T07:35:29.639 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:29.641+0000 7fcd7085eb80 1 -- 172.21.15.52:0/3172887125 >> 172.21.15.52:0/3172887125 conn(0x55a3e5198af0 msgr2=0x55a3e52c0e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:35:29.639 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:29.641+0000 7fcd7085eb80 1 -- 172.21.15.52:0/3172887125 shutdown_connections 2024-05-27T07:35:29.640 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:29.641+0000 7fcd7085eb80 1 -- 172.21.15.52:0/3172887125 wait complete. 2024-05-27T07:35:29.640 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:29.642+0000 7fcd7085eb80 20 client.26629 trim_cache size 0 max 16384 2024-05-27T07:35:29.640 INFO:teuthology.orchestra.run.smithi052.stdout:[] 2024-05-27T07:35:29.642 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:29.646+0000 7fcd60be7700 1 -- 172.21.15.52:0/442605786 >> [v2:172.21.15.52:6800/1822318384,v1:172.21.15.52:6801/1822318384] conn(0x7fcd440b2c00 msgr2=0x7fcd440b5110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:35:29.642 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:29.646+0000 7fcd60be7700 1 --2- 172.21.15.52:0/442605786 >> [v2:172.21.15.52:6800/1822318384,v1:172.21.15.52:6801/1822318384] conn(0x7fcd440b2c00 0x7fcd440b5110 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7fcd540a8fc0 tx=0x7fcd5400a2e0 comp rx=0 tx=0).stop 2024-05-27T07:35:29.643 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:29.646+0000 7fcd60be7700 1 -- 172.21.15.52:0/442605786 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fcd5c109690 msgr2=0x7fcd5c06ea70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:35:29.643 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:29.646+0000 7fcd60be7700 1 --2- 172.21.15.52:0/442605786 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fcd5c109690 0x7fcd5c06ea70 secure :-1 s=READY pgs=744 cs=0 l=1 rev1=1 crypto rx=0x7fcd5400ada0 tx=0x7fcd5400c330 comp rx=0 tx=0).stop 2024-05-27T07:35:29.643 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:29.647+0000 7fcd60be7700 1 -- 172.21.15.52:0/442605786 shutdown_connections 2024-05-27T07:35:29.643 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:29.647+0000 7fcd60be7700 1 --2- 172.21.15.52:0/442605786 >> [v2:172.21.15.52:6800/1822318384,v1:172.21.15.52:6801/1822318384] conn(0x7fcd440b2c00 0x7fcd440b5110 unknown :-1 s=CLOSED pgs=185 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:35:29.643 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:29.647+0000 7fcd60be7700 1 --2- 172.21.15.52:0/442605786 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fcd5c109690 0x7fcd5c06ea70 unknown :-1 s=CLOSED pgs=744 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:35:29.644 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:29.647+0000 7fcd60be7700 1 --2- 172.21.15.52:0/442605786 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fcd5c104410 0x7fcd5c075330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:35:29.644 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:29.647+0000 7fcd60be7700 1 --2- 172.21.15.52:0/442605786 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fcd5c108d40 0x7fcd5c06e4b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:35:29.644 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:29.647+0000 7fcd60be7700 1 -- 172.21.15.52:0/442605786 >> 172.21.15.52:0/442605786 conn(0x7fcd5c07aca0 msgr2=0x7fcd5c102810 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:35:29.644 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:29.647+0000 7fcd60be7700 1 -- 172.21.15.52:0/442605786 shutdown_connections 2024-05-27T07:35:29.644 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:29.647+0000 7fcd60be7700 1 -- 172.21.15.52:0/442605786 wait complete. 2024-05-27T07:35:30.366 DEBUG:teuthology.orchestra.run.smithi052:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:38f9d900c83c63799fbdbe61acc9a11b0d3554a6 shell --fsid 73df2636-1bf8-11ef-bc9b-c7b262605968 -- ceph tell mds.1:0 damage ls 2024-05-27T07:35:30.607 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:35:30 smithi173 ceph-mon[37000]: pgmap v321: 129 pgs: 19 active+clean+snaptrim, 55 active+clean+snaptrim_wait, 55 active+clean; 652 MiB data, 6.2 GiB used, 1.0 TiB / 1.0 TiB avail; 26 KiB/s rd, 1.1 MiB/s wr, 192 op/s 2024-05-27T07:35:30.695 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:35:30 smithi105 ceph-mon[35977]: pgmap v321: 129 pgs: 19 active+clean+snaptrim, 55 active+clean+snaptrim_wait, 55 active+clean; 652 MiB data, 6.2 GiB used, 1.0 TiB / 1.0 TiB avail; 26 KiB/s rd, 1.1 MiB/s wr, 192 op/s 2024-05-27T07:35:30.737 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:35:30 smithi052 ceph-mon[31955]: pgmap v321: 129 pgs: 19 active+clean+snaptrim, 55 active+clean+snaptrim_wait, 55 active+clean; 652 MiB data, 6.2 GiB used, 1.0 TiB / 1.0 TiB avail; 26 KiB/s rd, 1.1 MiB/s wr, 192 op/s 2024-05-27T07:35:30.887 INFO:teuthology.orchestra.run.smithi052.stderr:Inferring config /var/lib/ceph/73df2636-1bf8-11ef-bc9b-c7b262605968/mon.a/config 2024-05-27T07:35:32.611 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:35:32 smithi052 ceph-mon[31955]: pgmap v322: 129 pgs: 18 active+clean+snaptrim, 55 active+clean+snaptrim_wait, 56 active+clean; 644 MiB data, 6.2 GiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 1.1 MiB/s wr, 190 op/s 2024-05-27T07:35:32.695 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:35:32 smithi105 ceph-mon[35977]: pgmap v322: 129 pgs: 18 active+clean+snaptrim, 55 active+clean+snaptrim_wait, 56 active+clean; 644 MiB data, 6.2 GiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 1.1 MiB/s wr, 190 op/s 2024-05-27T07:35:32.857 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:35:32 smithi173 ceph-mon[37000]: pgmap v322: 129 pgs: 18 active+clean+snaptrim, 55 active+clean+snaptrim_wait, 56 active+clean; 644 MiB data, 6.2 GiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 1.1 MiB/s wr, 190 op/s 2024-05-27T07:35:33.145 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:33.149+0000 7fababb6b700 1 -- 172.21.15.52:0/3551454934 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7faba410a920 msgr2=0x7faba410ad30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:35:33.145 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:33.149+0000 7fababb6b700 1 --2- 172.21.15.52:0/3551454934 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7faba410a920 0x7faba410ad30 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7faba0009990 tx=0x7faba0009ca0 comp rx=0 tx=0).stop 2024-05-27T07:35:33.150 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:33.155+0000 7fababb6b700 1 -- 172.21.15.52:0/3551454934 shutdown_connections 2024-05-27T07:35:33.150 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:33.155+0000 7fababb6b700 1 --2- 172.21.15.52:0/3551454934 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7faba411d920 0x7faba411fd40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:35:33.150 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:33.155+0000 7fababb6b700 1 --2- 172.21.15.52:0/3551454934 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7faba410a920 0x7faba410ad30 unknown :-1 s=CLOSED pgs=246 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:35:33.150 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:33.155+0000 7fababb6b700 1 --2- 172.21.15.52:0/3551454934 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7faba410b270 0x7faba411d3e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:35:33.150 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:33.155+0000 7fababb6b700 1 -- 172.21.15.52:0/3551454934 >> 172.21.15.52:0/3551454934 conn(0x7faba406f4c0 msgr2=0x7faba406f900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:35:33.155 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:33.160+0000 7fababb6b700 1 -- 172.21.15.52:0/3551454934 shutdown_connections 2024-05-27T07:35:33.160 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:33.164+0000 7fababb6b700 1 -- 172.21.15.52:0/3551454934 wait complete. 2024-05-27T07:35:33.160 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:33.165+0000 7fababb6b700 1 Processor -- start 2024-05-27T07:35:33.161 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:33.165+0000 7fababb6b700 1 -- start start 2024-05-27T07:35:33.161 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:33.165+0000 7fababb6b700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7faba410a920 0x7faba41af860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:35:33.161 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:33.165+0000 7fababb6b700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7faba410b270 0x7faba41afda0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:35:33.161 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:33.165+0000 7fababb6b700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7faba411d920 0x7faba41a9a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:35:33.161 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:33.165+0000 7fababb6b700 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x7faba4123320 con 0x7faba410b270 2024-05-27T07:35:33.161 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:33.165+0000 7fababb6b700 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7faba41231a0 con 0x7faba411d920 2024-05-27T07:35:33.161 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:33.165+0000 7fababb6b700 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7faba41234a0 con 0x7faba410a920 2024-05-27T07:35:33.162 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:33.165+0000 7faba9907700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7faba410a920 0x7faba41af860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:35:33.162 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:33.166+0000 7faba9907700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7faba410a920 0x7faba41af860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.173:3300/0 says I am v2:172.21.15.52:51462/0 (socket says 172.21.15.52:51462) 2024-05-27T07:35:33.162 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:33.166+0000 7faba9907700 1 -- 172.21.15.52:0/761394454 learned_addr learned my addr 172.21.15.52:0/761394454 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-05-27T07:35:33.162 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:33.166+0000 7fabaa108700 1 --2- 172.21.15.52:0/761394454 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7faba411d920 0x7faba41a9a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:35:33.162 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:33.166+0000 7fabaa108700 1 -- 172.21.15.52:0/761394454 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7faba410a920 msgr2=0x7faba41af860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:35:33.162 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:33.166+0000 7fabaa108700 1 --2- 172.21.15.52:0/761394454 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7faba410a920 0x7faba41af860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:35:33.162 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:33.166+0000 7fabaa108700 1 -- 172.21.15.52:0/761394454 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7faba410b270 msgr2=0x7faba41afda0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-05-27T07:35:33.162 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:33.166+0000 7fabaa108700 1 --2- 172.21.15.52:0/761394454 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7faba410b270 0x7faba41afda0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:35:33.162 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:33.166+0000 7fabaa108700 1 -- 172.21.15.52:0/761394454 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7faba41aa150 con 0x7faba411d920 2024-05-27T07:35:33.165 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:33.168+0000 7fabaa108700 1 --2- 172.21.15.52:0/761394454 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7faba411d920 0x7faba41a9a80 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7fab9c007ee0 tx=0x7fab9c00d3b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:35:33.169 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:33.172+0000 7fab9affd700 1 -- 172.21.15.52:0/761394454 <== mon.2 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fab9c00ddd0 con 0x7faba411d920 2024-05-27T07:35:33.169 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:33.172+0000 7fab9affd700 1 -- 172.21.15.52:0/761394454 <== mon.2 v2:172.21.15.105:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fab9c009e20 con 0x7faba411d920 2024-05-27T07:35:33.169 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:33.172+0000 7fab9affd700 1 -- 172.21.15.52:0/761394454 <== mon.2 v2:172.21.15.105:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fab9c015400 con 0x7faba411d920 2024-05-27T07:35:33.169 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:33.174+0000 7fababb6b700 1 -- 172.21.15.52:0/761394454 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7faba41aa440 con 0x7faba411d920 2024-05-27T07:35:33.169 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:33.174+0000 7fababb6b700 1 -- 172.21.15.52:0/761394454 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7faba4112bc0 con 0x7faba411d920 2024-05-27T07:35:33.178 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:33.180+0000 7fabba580b80 1 Processor -- start 2024-05-27T07:35:33.179 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:33.184+0000 7fab9affd700 1 -- 172.21.15.52:0/761394454 <== mon.2 v2:172.21.15.105:3300/0 4 ==== mgrmap(e 26) ==== 149783+0+0 (secure 0 0 0) 0x7fab9c004c40 con 0x7faba411d920 2024-05-27T07:35:33.179 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:33.184+0000 7fabba580b80 1 -- start start 2024-05-27T07:35:33.183 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:33.186+0000 7fab9affd700 1 --2- 172.21.15.52:0/761394454 >> [v2:172.21.15.52:6800/1822318384,v1:172.21.15.52:6801/1822318384] conn(0x7fab900b2c00 0x7fab900b5110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:35:33.184 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:33.187+0000 7fab9affd700 1 -- 172.21.15.52:0/761394454 <== mon.2 v2:172.21.15.105:3300/0 5 ==== osd_map(123..123 src has 1..123) ==== 9555+0+0 (secure 0 0 0) 0x7fab9c0e22b0 con 0x7faba411d920 2024-05-27T07:35:33.184 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:33.188+0000 7fabba580b80 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55ef3eea5e40 0x55ef3f02ab30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:35:33.184 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:33.188+0000 7fabba580b80 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55ef3f02b070 0x55ef3f024be0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:35:33.184 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:33.188+0000 7fabba580b80 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55ef3f025120 0x55ef3f025530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:35:33.184 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:33.188+0000 7faba9907700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55ef3eea5e40 0x55ef3f02ab30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:35:33.184 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:33.188+0000 7faba9907700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55ef3eea5e40 0x55ef3f02ab30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.173:3300/0 says I am v2:172.21.15.52:51470/0 (socket says 172.21.15.52:51470) 2024-05-27T07:35:33.184 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:33.188+0000 7faba9907700 1 -- 172.21.15.52:0/2268106578 learned_addr learned my addr 172.21.15.52:0/2268106578 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-05-27T07:35:33.184 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:33.188+0000 7fabba580b80 1 -- 172.21.15.52:0/2268106578 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x55ef3ef52d70 con 0x55ef3f02b070 2024-05-27T07:35:33.184 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:33.188+0000 7fabaa108700 1 --2- 172.21.15.52:0/2268106578 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55ef3f025120 0x55ef3f025530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:35:33.184 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:33.188+0000 7fabba580b80 1 -- 172.21.15.52:0/2268106578 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x55ef3ef42910 con 0x55ef3f025120 2024-05-27T07:35:33.184 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:33.188+0000 7fabba580b80 1 -- 172.21.15.52:0/2268106578 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x55ef3ef46950 con 0x55ef3eea5e40 2024-05-27T07:35:33.185 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:33.189+0000 7faba9106700 1 --2- 172.21.15.52:0/761394454 >> [v2:172.21.15.52:6800/1822318384,v1:172.21.15.52:6801/1822318384] conn(0x7fab900b2c00 0x7fab900b5110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:35:33.185 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:33.189+0000 7faba9907700 1 -- 172.21.15.52:0/2268106578 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55ef3f025120 msgr2=0x55ef3f025530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:35:33.185 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:33.189+0000 7faba9907700 1 --2- 172.21.15.52:0/2268106578 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55ef3f025120 0x55ef3f025530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:35:33.185 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:33.189+0000 7faba9907700 1 -- 172.21.15.52:0/2268106578 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55ef3f02b070 msgr2=0x55ef3f024be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:35:33.185 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:33.189+0000 7faba9907700 1 --2- 172.21.15.52:0/2268106578 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55ef3f02b070 0x55ef3f024be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:35:33.185 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:33.189+0000 7faba9907700 1 -- 172.21.15.52:0/2268106578 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55ef3eeebfb0 con 0x55ef3eea5e40 2024-05-27T07:35:33.186 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:33.191+0000 7faba9907700 1 --2- 172.21.15.52:0/2268106578 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55ef3eea5e40 0x55ef3f02ab30 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7faba0005f50 tx=0x7faba0005050 comp rx=0 tx=0).ready entity=mon.1 client_cookie=8a753991025a307b server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:35:33.191 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:33.196+0000 7faba9106700 1 --2- 172.21.15.52:0/761394454 >> [v2:172.21.15.52:6800/1822318384,v1:172.21.15.52:6801/1822318384] conn(0x7fab900b2c00 0x7fab900b5110 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7faba411d8f0 tx=0x7fab94009670 comp rx=0 tx=0).ready entity=mgr.15333 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:35:33.192 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:33.197+0000 7fab8ffff700 1 -- 172.21.15.52:0/2268106578 <== mon.1 v2:172.21.15.173:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faba000b7c0 con 0x55ef3eea5e40 2024-05-27T07:35:33.192 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:33.197+0000 7fab8ffff700 1 -- 172.21.15.52:0/2268106578 <== mon.1 v2:172.21.15.173:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7faba000be10 con 0x55ef3eea5e40 2024-05-27T07:35:33.192 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:33.197+0000 7fab8ffff700 1 -- 172.21.15.52:0/2268106578 <== mon.1 v2:172.21.15.173:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faba000f460 con 0x55ef3eea5e40 2024-05-27T07:35:33.197 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:33.202+0000 7fabba580b80 1 -- 172.21.15.52:0/2268106578 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55ef3eea5e40 msgr2=0x55ef3f02ab30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:35:33.197 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:33.202+0000 7fabba580b80 1 --2- 172.21.15.52:0/2268106578 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55ef3eea5e40 0x55ef3f02ab30 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7faba0005f50 tx=0x7faba0005050 comp rx=0 tx=0).stop 2024-05-27T07:35:33.199 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:33.204+0000 7fabba580b80 1 -- 172.21.15.52:0/2268106578 shutdown_connections 2024-05-27T07:35:33.199 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:33.204+0000 7fabba580b80 1 --2- 172.21.15.52:0/2268106578 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55ef3f025120 0x55ef3f025530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:35:33.199 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:33.204+0000 7fabba580b80 1 --2- 172.21.15.52:0/2268106578 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55ef3eea5e40 0x55ef3f02ab30 unknown :-1 s=CLOSED pgs=211 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:35:33.199 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:33.204+0000 7fabba580b80 1 --2- 172.21.15.52:0/2268106578 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55ef3f02b070 0x55ef3f024be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:35:33.199 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:33.204+0000 7fabba580b80 1 -- 172.21.15.52:0/2268106578 >> 172.21.15.52:0/2268106578 conn(0x55ef3eea8770 msgr2=0x55ef3f01fd90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:35:33.201 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:33.206+0000 7fabba580b80 1 -- 172.21.15.52:0/2268106578 shutdown_connections 2024-05-27T07:35:33.201 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:33.206+0000 7fabba580b80 1 -- 172.21.15.52:0/2268106578 wait complete. 2024-05-27T07:35:33.202 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:33.207+0000 7fabba580b80 1 -- start start 2024-05-27T07:35:33.202 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:33.207+0000 7fabba580b80 1 Processor -- start 2024-05-27T07:35:33.203 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:33.208+0000 7fabba580b80 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55ef3eea5e40 0x55ef3f02e920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:35:33.203 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:33.208+0000 7fabba580b80 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55ef3f025120 0x55ef3f02ee60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:35:33.204 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:33.208+0000 7fabba580b80 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55ef3f02b070 0x55ef3f02f3a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:35:33.204 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:33.209+0000 7faba9106700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55ef3f025120 0x55ef3f02ee60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:35:33.205 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:33.209+0000 7faba9106700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55ef3f025120 0x55ef3f02ee60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.52:53694/0 (socket says 172.21.15.52:53694) 2024-05-27T07:35:33.205 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:33.209+0000 7faba9106700 1 -- 172.21.15.52:0/758334476 learned_addr learned my addr 172.21.15.52:0/758334476 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-05-27T07:35:33.205 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:33.209+0000 7fabba580b80 1 -- 172.21.15.52:0/758334476 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x55ef3ee48fa0 con 0x55ef3f025120 2024-05-27T07:35:33.205 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:33.209+0000 7fabba580b80 1 -- 172.21.15.52:0/758334476 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x55ef3eeaada0 con 0x55ef3f02b070 2024-05-27T07:35:33.205 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:33.209+0000 7fabba580b80 1 -- 172.21.15.52:0/758334476 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x55ef3f031aa0 con 0x55ef3eea5e40 2024-05-27T07:35:33.205 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:33.209+0000 7fabba580b80 10 client.0 _ll_register_callbacks cb 0x55ef3eed95b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T07:35:33.208 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:33.211+0000 7faba9907700 1 --2- 172.21.15.52:0/758334476 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55ef3eea5e40 0x55ef3f02e920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:35:33.208 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:33.211+0000 7fabaa108700 1 --2- 172.21.15.52:0/758334476 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55ef3f02b070 0x55ef3f02f3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:35:33.208 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:33.213+0000 7faba9106700 1 -- 172.21.15.52:0/758334476 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55ef3eea5e40 msgr2=0x55ef3f02e920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:35:33.208 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:33.213+0000 7faba9106700 1 --2- 172.21.15.52:0/758334476 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55ef3eea5e40 0x55ef3f02e920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:35:33.208 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:33.213+0000 7faba9106700 1 -- 172.21.15.52:0/758334476 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55ef3f02b070 msgr2=0x55ef3f02f3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:35:33.209 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:33.213+0000 7faba9106700 1 --2- 172.21.15.52:0/758334476 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55ef3f02b070 0x55ef3f02f3a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:35:33.209 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:33.213+0000 7faba9106700 1 -- 172.21.15.52:0/758334476 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55ef3ee928d0 con 0x55ef3f025120 2024-05-27T07:35:33.211 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:33.215+0000 7faba9106700 1 --2- 172.21.15.52:0/758334476 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55ef3f025120 0x55ef3f02ee60 secure :-1 s=READY pgs=746 cs=0 l=1 rev1=1 crypto rx=0x7fab94012480 tx=0x7fab940047c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4678b3a4a2d269ef server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:35:33.211 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:33.215+0000 7fab8d7fa700 10 client.0 ms_handle_connect on v2:172.21.15.52:3300/0 2024-05-27T07:35:33.211 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:33.216+0000 7fab8d7fa700 1 -- 172.21.15.52:0/758334476 <== mon.0 v2:172.21.15.52:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fab94022030 con 0x55ef3f025120 2024-05-27T07:35:33.211 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:33.216+0000 7fabba580b80 1 -- 172.21.15.52:0/758334476 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7faba000be10 con 0x55ef3f025120 2024-05-27T07:35:33.211 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:33.216+0000 7fab8d7fa700 1 -- 172.21.15.52:0/758334476 <== mon.0 v2:172.21.15.52:3300/0 2 ==== osd_map(123..123 src has 1..123) ==== 9555+0+0 (secure 0 0 0) 0x7fab9401e420 con 0x55ef3f025120 2024-05-27T07:35:33.212 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:33.217+0000 7fab8d7fa700 1 -- 172.21.15.52:0/758334476 <== mon.0 v2:172.21.15.52:3300/0 3 ==== mon_get_version_reply(handle=1 version=108) ==== 24+0+0 (secure 0 0 0) 0x7fab9402f050 con 0x55ef3f025120 2024-05-27T07:35:33.215 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:33.219+0000 7fabba580b80 10 client.17154 fetch_fsmap learned FSMap version 108 2024-05-27T07:35:33.215 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:33.220+0000 7fabba580b80 1 -- 172.21.15.52:0/758334476 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({config=0+,fsmap=108,monmap=4+}) -- 0x55ef3eefc760 con 0x55ef3f025120 2024-05-27T07:35:33.217 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:33.221+0000 7fab8d7fa700 1 -- 172.21.15.52:0/758334476 <== mon.0 v2:172.21.15.52:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7faba000be10 con 0x55ef3f025120 2024-05-27T07:35:33.217 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:33.221+0000 7fab8d7fa700 1 -- 172.21.15.52:0/758334476 <== mon.0 v2:172.21.15.52:3300/0 5 ==== fsmap(e 108) ==== 4203+0+0 (secure 0 0 0) 0x7fab94027650 con 0x55ef3f025120 2024-05-27T07:35:33.217 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:33.221+0000 7fabba580b80 10 client.17154 fetch_fsmap finished waiting for FSMap version 108 2024-05-27T07:35:33.217 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:33.221+0000 7fabba580b80 10 client.17154 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-05-27T07:35:33.218 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:33.221+0000 7fabba580b80 20 client.17154 populate_metadata read hostname 'smithi052' 2024-05-27T07:35:33.218 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:33.221+0000 7fabba580b80 1 --2- 172.21.15.52:0/758334476 >> [v2:172.21.15.105:6838/500348151,v1:172.21.15.105:6839/500348151] conn(0x55ef3f03ed30 0x55ef3f0411c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:35:33.218 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:33.221+0000 7fabba580b80 4 client.17154 mds_command: new command op to 24527 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-27T07:35:33.218 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:33.221+0000 7fabba580b80 1 -- 172.21.15.52:0/758334476 --> [v2:172.21.15.105:6838/500348151,v1:172.21.15.105:6839/500348151] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55ef3ef54cc0 con 0x55ef3f03ed30 2024-05-27T07:35:33.219 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:33.222+0000 7faba9907700 1 --2- 172.21.15.52:0/758334476 >> [v2:172.21.15.105:6838/500348151,v1:172.21.15.105:6839/500348151] conn(0x55ef3f03ed30 0x55ef3f0411c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:35:33.219 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:33.223+0000 7faba9907700 1 --2- 172.21.15.52:0/758334476 >> [v2:172.21.15.105:6838/500348151,v1:172.21.15.105:6839/500348151] conn(0x55ef3f03ed30 0x55ef3f0411c0 crc :-1 s=READY pgs=167 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=b4697fbe22dfed2b server_cookie=3d7033fc2d09fecf in_seq=0 out_seq=0 2024-05-27T07:35:33.219 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:33.223+0000 7fab8d7fa700 10 client.17154 ms_handle_connect on v2:172.21.15.105:6838/500348151 2024-05-27T07:35:33.223 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:33.225+0000 7fab8d7fa700 1 -- 172.21.15.52:0/758334476 <== mds.0 v2:172.21.15.105:6838/500348151 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24041 (crc 0 0 0) 0x55ef3ef54cc0 con 0x55ef3f03ed30 2024-05-27T07:35:33.223 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:33.225+0000 7fab8d7fa700 10 client.17154 handle_command_reply: tid=0 2024-05-27T07:35:33.223 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:33.225+0000 7fabba580b80 1 client.17154 shutdown 2024-05-27T07:35:33.227 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:33.232+0000 7fabba580b80 1 -- 172.21.15.52:0/758334476 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55ef3f025120 msgr2=0x55ef3f02ee60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:35:33.227 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:33.232+0000 7fabba580b80 1 --2- 172.21.15.52:0/758334476 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55ef3f025120 0x55ef3f02ee60 secure :-1 s=READY pgs=746 cs=0 l=1 rev1=1 crypto rx=0x7fab94012480 tx=0x7fab940047c0 comp rx=0 tx=0).stop 2024-05-27T07:35:33.230 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:33.234+0000 7fabba580b80 1 -- 172.21.15.52:0/758334476 shutdown_connections 2024-05-27T07:35:33.230 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:33.234+0000 7fabba580b80 1 --2- 172.21.15.52:0/758334476 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55ef3f02b070 0x55ef3f02f3a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:35:33.230 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:33.234+0000 7fabba580b80 1 --2- 172.21.15.52:0/758334476 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55ef3eea5e40 0x55ef3f02e920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:35:33.230 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:33.234+0000 7fabba580b80 1 --2- 172.21.15.52:0/758334476 >> [v2:172.21.15.105:6838/500348151,v1:172.21.15.105:6839/500348151] conn(0x55ef3f03ed30 0x55ef3f0411c0 crc :-1 s=READY pgs=167 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:35:33.231 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:33.234+0000 7fabba580b80 1 --2- 172.21.15.52:0/758334476 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55ef3f025120 0x55ef3f02ee60 unknown :-1 s=CLOSED pgs=746 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:35:33.231 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:33.234+0000 7fabba580b80 1 -- 172.21.15.52:0/758334476 >> 172.21.15.52:0/758334476 conn(0x55ef3eea8770 msgr2=0x55ef3f023bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:35:33.231 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:33.234+0000 7fab8d7fa700 0 client.17154 ms_handle_reset on v2:172.21.15.105:6838/500348151 2024-05-27T07:35:33.232 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:33.236+0000 7fabba580b80 1 -- 172.21.15.52:0/758334476 shutdown_connections 2024-05-27T07:35:33.236 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:33.240+0000 7fabba580b80 1 -- 172.21.15.52:0/758334476 wait complete. 2024-05-27T07:35:33.236 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:33.241+0000 7fabba580b80 20 client.17154 trim_cache size 0 max 16384 2024-05-27T07:35:33.266 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:33.270+0000 7fabba580b80 1 Processor -- start 2024-05-27T07:35:33.266 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:33.270+0000 7fabba580b80 1 -- start start 2024-05-27T07:35:33.266 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:33.270+0000 7fabba580b80 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55ef3eea5e40 0x55ef3f024be0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:35:33.267 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:33.270+0000 7fabba580b80 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55ef3f025120 0x55ef3f02a060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:35:33.267 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:33.270+0000 7fabba580b80 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55ef3f02b070 0x55ef3f0194f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:35:33.267 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:33.270+0000 7fabba580b80 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x7faba41c8400 con 0x55ef3f025120 2024-05-27T07:35:33.267 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:33.270+0000 7fabba580b80 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7faba411a4f0 con 0x55ef3f02b070 2024-05-27T07:35:33.267 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:33.270+0000 7fabba580b80 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7faba41c7d00 con 0x55ef3eea5e40 2024-05-27T07:35:33.267 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:33.271+0000 7fabaa108700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55ef3f02b070 0x55ef3f0194f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:35:33.267 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:33.271+0000 7fabaa108700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55ef3f02b070 0x55ef3f0194f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.52:56626/0 (socket says 172.21.15.52:56626) 2024-05-27T07:35:33.267 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:33.271+0000 7fabaa108700 1 -- 172.21.15.52:0/320165677 learned_addr learned my addr 172.21.15.52:0/320165677 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-05-27T07:35:33.267 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:33.271+0000 7faba9907700 1 --2- 172.21.15.52:0/320165677 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55ef3eea5e40 0x55ef3f024be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:35:33.268 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:33.271+0000 7fabaa108700 1 -- 172.21.15.52:0/320165677 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55ef3eea5e40 msgr2=0x55ef3f024be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:35:33.268 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:33.271+0000 7fabaa108700 1 --2- 172.21.15.52:0/320165677 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55ef3eea5e40 0x55ef3f024be0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:35:33.268 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:33.271+0000 7fabaa108700 1 -- 172.21.15.52:0/320165677 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55ef3f025120 msgr2=0x55ef3f02a060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:35:33.268 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:33.271+0000 7fabaa108700 1 --2- 172.21.15.52:0/320165677 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55ef3f025120 0x55ef3f02a060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:35:33.268 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:33.271+0000 7fabaa108700 1 -- 172.21.15.52:0/320165677 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55ef3f031710 con 0x55ef3f02b070 2024-05-27T07:35:33.269 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:33.271+0000 7faba9907700 1 --2- 172.21.15.52:0/320165677 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55ef3eea5e40 0x55ef3f024be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-27T07:35:33.269 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:33.271+0000 7fabaa108700 1 --2- 172.21.15.52:0/320165677 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55ef3f02b070 0x55ef3f0194f0 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7fab9c0a90b0 tx=0x7fab9c0a8c30 comp rx=0 tx=0).ready entity=mon.2 client_cookie=47d19221805cb273 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:35:33.269 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:33.272+0000 7fab8f7fe700 1 -- 172.21.15.52:0/320165677 <== mon.2 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fab9c0c2020 con 0x55ef3f02b070 2024-05-27T07:35:33.270 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:33.274+0000 7fab8f7fe700 1 -- 172.21.15.52:0/320165677 <== mon.2 v2:172.21.15.105:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fab9c0a9820 con 0x55ef3f02b070 2024-05-27T07:35:33.271 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:33.274+0000 7fab8f7fe700 1 -- 172.21.15.52:0/320165677 <== mon.2 v2:172.21.15.105:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fab9c0a9380 con 0x55ef3f02b070 2024-05-27T07:35:33.271 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:33.276+0000 7fabba580b80 1 -- 172.21.15.52:0/320165677 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55ef3f02b070 msgr2=0x55ef3f0194f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:35:33.271 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:33.276+0000 7fabba580b80 1 --2- 172.21.15.52:0/320165677 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55ef3f02b070 0x55ef3f0194f0 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7fab9c0a90b0 tx=0x7fab9c0a8c30 comp rx=0 tx=0).stop 2024-05-27T07:35:33.275 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:33.277+0000 7fabba580b80 1 -- 172.21.15.52:0/320165677 shutdown_connections 2024-05-27T07:35:33.275 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:33.277+0000 7fabba580b80 1 --2- 172.21.15.52:0/320165677 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55ef3f02b070 0x55ef3f0194f0 unknown :-1 s=CLOSED pgs=248 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:35:33.275 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:33.277+0000 7fabba580b80 1 --2- 172.21.15.52:0/320165677 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55ef3eea5e40 0x55ef3f024be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:35:33.275 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:33.277+0000 7fabba580b80 1 --2- 172.21.15.52:0/320165677 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55ef3f025120 0x55ef3f02a060 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:35:33.276 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:33.277+0000 7fabba580b80 1 -- 172.21.15.52:0/320165677 >> 172.21.15.52:0/320165677 conn(0x55ef3eea8770 msgr2=0x55ef3f029950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:35:33.276 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:33.279+0000 7fabba580b80 1 -- 172.21.15.52:0/320165677 shutdown_connections 2024-05-27T07:35:33.276 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:33.280+0000 7fabba580b80 1 -- 172.21.15.52:0/320165677 wait complete. 2024-05-27T07:35:33.278 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:33.280+0000 7fabba580b80 1 -- start start 2024-05-27T07:35:33.278 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:33.280+0000 7fabba580b80 1 Processor -- start 2024-05-27T07:35:33.278 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:33.281+0000 7fabba580b80 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55ef3eea5e40 0x55ef3f036d30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:35:33.279 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:33.281+0000 7fabba580b80 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55ef3f025120 0x55ef3f037270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:35:33.279 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:33.281+0000 7fabba580b80 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55ef3f02b070 0x55ef3f045970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:35:33.279 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:33.281+0000 7fabba580b80 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x7faba41c7700 con 0x55ef3eea5e40 2024-05-27T07:35:33.279 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:33.281+0000 7fabba580b80 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7faba411abf0 con 0x55ef3f02b070 2024-05-27T07:35:33.279 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:33.281+0000 7fabba580b80 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x55ef3f020ab0 con 0x55ef3f025120 2024-05-27T07:35:33.279 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:33.281+0000 7fabba580b80 10 client.0 _ll_register_callbacks cb 0x55ef3eed95b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T07:35:33.279 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:33.283+0000 7fabaa108700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55ef3f02b070 0x55ef3f045970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:35:33.281 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:33.283+0000 7faba9907700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55ef3eea5e40 0x55ef3f036d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:35:33.281 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:33.284+0000 7faba9907700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55ef3eea5e40 0x55ef3f036d30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.52:53716/0 (socket says 172.21.15.52:53716) 2024-05-27T07:35:33.281 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:33.284+0000 7faba9907700 1 -- 172.21.15.52:0/2033455342 learned_addr learned my addr 172.21.15.52:0/2033455342 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-05-27T07:35:33.281 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:33.284+0000 7faba9907700 1 -- 172.21.15.52:0/2033455342 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55ef3f025120 msgr2=0x55ef3f037270 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-05-27T07:35:33.281 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:33.284+0000 7faba9907700 1 --2- 172.21.15.52:0/2033455342 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55ef3f025120 0x55ef3f037270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:35:33.281 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:33.284+0000 7faba9907700 1 -- 172.21.15.52:0/2033455342 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55ef3f02b070 msgr2=0x55ef3f045970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:35:33.282 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:33.284+0000 7faba9907700 1 --2- 172.21.15.52:0/2033455342 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55ef3f02b070 0x55ef3f045970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:35:33.282 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:33.284+0000 7faba9907700 1 -- 172.21.15.52:0/2033455342 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55ef3f047330 con 0x55ef3eea5e40 2024-05-27T07:35:33.282 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:33.285+0000 7faba9907700 1 --2- 172.21.15.52:0/2033455342 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55ef3eea5e40 0x55ef3f036d30 secure :-1 s=READY pgs=747 cs=0 l=1 rev1=1 crypto rx=0x7faba0003b30 tx=0x7faba0005a90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a743c0cd05415b05 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:35:33.282 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:33.285+0000 7fab8d7fa700 10 client.0 ms_handle_connect on v2:172.21.15.52:3300/0 2024-05-27T07:35:33.282 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:33.285+0000 7fab8d7fa700 1 -- 172.21.15.52:0/2033455342 <== mon.0 v2:172.21.15.52:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faba000bd60 con 0x55ef3eea5e40 2024-05-27T07:35:33.282 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:33.285+0000 7fab8d7fa700 1 -- 172.21.15.52:0/2033455342 <== mon.0 v2:172.21.15.52:3300/0 2 ==== osd_map(123..123 src has 1..123) ==== 9555+0+0 (secure 0 0 0) 0x7faba000f460 con 0x55ef3eea5e40 2024-05-27T07:35:33.282 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:33.286+0000 7fabba580b80 1 -- 172.21.15.52:0/2033455342 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fab9c0a9820 con 0x55ef3eea5e40 2024-05-27T07:35:33.283 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:33.286+0000 7fab8d7fa700 1 -- 172.21.15.52:0/2033455342 <== mon.0 v2:172.21.15.52:3300/0 3 ==== mon_get_version_reply(handle=1 version=108) ==== 24+0+0 (secure 0 0 0) 0x7faba000f660 con 0x55ef3eea5e40 2024-05-27T07:35:33.283 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:33.286+0000 7fabba580b80 10 client.17157 fetch_fsmap learned FSMap version 108 2024-05-27T07:35:33.283 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:33.286+0000 7fabba580b80 1 -- 172.21.15.52:0/2033455342 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({config=0+,fsmap=108,monmap=4+}) -- 0x55ef3f0a8bc0 con 0x55ef3eea5e40 2024-05-27T07:35:33.285 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:33.290+0000 7fab8d7fa700 1 -- 172.21.15.52:0/2033455342 <== mon.0 v2:172.21.15.52:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fab9c0a9820 con 0x55ef3eea5e40 2024-05-27T07:35:33.285 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:33.290+0000 7fab8d7fa700 1 -- 172.21.15.52:0/2033455342 <== mon.0 v2:172.21.15.52:3300/0 5 ==== fsmap(e 108) ==== 4203+0+0 (secure 0 0 0) 0x7faba0022770 con 0x55ef3eea5e40 2024-05-27T07:35:33.285 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:33.290+0000 7fabba580b80 10 client.17157 fetch_fsmap finished waiting for FSMap version 108 2024-05-27T07:35:33.285 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:33.290+0000 7fabba580b80 10 client.17157 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-05-27T07:35:33.285 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:33.290+0000 7fabba580b80 20 client.17157 populate_metadata read hostname 'smithi052' 2024-05-27T07:35:33.286 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:33.290+0000 7fabba580b80 1 --2- 172.21.15.52:0/2033455342 >> [v2:172.21.15.105:6838/500348151,v1:172.21.15.105:6839/500348151] conn(0x55ef3f0a91a0 0x55ef3f0ab5c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:35:33.286 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:33.290+0000 7fabba580b80 4 client.17157 mds_command: new command op to 24527 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-05-27T07:35:33.286 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:33.290+0000 7fabba580b80 1 -- 172.21.15.52:0/2033455342 --> [v2:172.21.15.105:6838/500348151,v1:172.21.15.105:6839/500348151] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55ef3ef4e4a0 con 0x55ef3f0a91a0 2024-05-27T07:35:33.287 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:33.292+0000 7faba9106700 1 --2- 172.21.15.52:0/2033455342 >> [v2:172.21.15.105:6838/500348151,v1:172.21.15.105:6839/500348151] conn(0x55ef3f0a91a0 0x55ef3f0ab5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:35:33.291 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:33.296+0000 7faba9106700 1 --2- 172.21.15.52:0/2033455342 >> [v2:172.21.15.105:6838/500348151,v1:172.21.15.105:6839/500348151] conn(0x55ef3f0a91a0 0x55ef3f0ab5c0 crc :-1 s=READY pgs=168 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=66494c97136319d0 server_cookie=5fad343e49ce7e4e in_seq=0 out_seq=0 2024-05-27T07:35:33.291 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:33.296+0000 7fab8d7fa700 10 client.17157 ms_handle_connect on v2:172.21.15.105:6838/500348151 2024-05-27T07:35:33.295 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:33.300+0000 7fab8d7fa700 1 -- 172.21.15.52:0/2033455342 <== mds.0 v2:172.21.15.105:6838/500348151 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55ef3ef4e4a0 con 0x55ef3f0a91a0 2024-05-27T07:35:33.295 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:33.300+0000 7fab8d7fa700 10 client.17157 handle_command_reply: tid=0 2024-05-27T07:35:33.295 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:33.300+0000 7fabba580b80 1 client.17157 shutdown 2024-05-27T07:35:33.296 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:33.301+0000 7fabba580b80 1 -- 172.21.15.52:0/2033455342 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55ef3eea5e40 msgr2=0x55ef3f036d30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:35:33.296 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:33.301+0000 7fabba580b80 1 --2- 172.21.15.52:0/2033455342 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55ef3eea5e40 0x55ef3f036d30 secure :-1 s=READY pgs=747 cs=0 l=1 rev1=1 crypto rx=0x7faba0003b30 tx=0x7faba0005a90 comp rx=0 tx=0).stop 2024-05-27T07:35:33.300 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:33.304+0000 7fabba580b80 1 -- 172.21.15.52:0/2033455342 shutdown_connections 2024-05-27T07:35:33.300 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:33.304+0000 7fabba580b80 1 --2- 172.21.15.52:0/2033455342 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55ef3f02b070 0x55ef3f045970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:35:33.300 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:33.304+0000 7fabba580b80 1 --2- 172.21.15.52:0/2033455342 >> [v2:172.21.15.105:6838/500348151,v1:172.21.15.105:6839/500348151] conn(0x55ef3f0a91a0 0x55ef3f0ab5c0 crc :-1 s=READY pgs=168 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:35:33.300 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:33.304+0000 7fabba580b80 1 --2- 172.21.15.52:0/2033455342 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55ef3eea5e40 0x55ef3f036d30 unknown :-1 s=CLOSED pgs=747 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:35:33.300 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:33.304+0000 7fabba580b80 1 --2- 172.21.15.52:0/2033455342 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55ef3f025120 0x55ef3f037270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:35:33.300 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:33.304+0000 7fabba580b80 1 -- 172.21.15.52:0/2033455342 >> 172.21.15.52:0/2033455342 conn(0x55ef3eea8770 msgr2=0x55ef3f028490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:35:33.300 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:33.304+0000 7fab8d7fa700 0 client.17157 ms_handle_reset on v2:172.21.15.105:6838/500348151 2024-05-27T07:35:33.301 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:33.305+0000 7fabba580b80 1 -- 172.21.15.52:0/2033455342 shutdown_connections 2024-05-27T07:35:33.301 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:33.305+0000 7fabba580b80 1 -- 172.21.15.52:0/2033455342 wait complete. 2024-05-27T07:35:33.304 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:33.307+0000 7fabba580b80 20 client.17157 trim_cache size 0 max 16384 2024-05-27T07:35:33.304 INFO:teuthology.orchestra.run.smithi052.stdout:[] 2024-05-27T07:35:33.305 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:33.310+0000 7fababb6b700 1 -- 172.21.15.52:0/761394454 >> [v2:172.21.15.52:6800/1822318384,v1:172.21.15.52:6801/1822318384] conn(0x7fab900b2c00 msgr2=0x7fab900b5110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:35:33.305 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:33.310+0000 7fababb6b700 1 --2- 172.21.15.52:0/761394454 >> [v2:172.21.15.52:6800/1822318384,v1:172.21.15.52:6801/1822318384] conn(0x7fab900b2c00 0x7fab900b5110 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7faba411d8f0 tx=0x7fab94009670 comp rx=0 tx=0).stop 2024-05-27T07:35:33.305 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:33.310+0000 7fababb6b700 1 -- 172.21.15.52:0/761394454 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7faba411d920 msgr2=0x7faba41a9a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:35:33.306 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:33.310+0000 7fababb6b700 1 --2- 172.21.15.52:0/761394454 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7faba411d920 0x7faba41a9a80 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7fab9c007ee0 tx=0x7fab9c00d3b0 comp rx=0 tx=0).stop 2024-05-27T07:35:33.306 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:33.310+0000 7fababb6b700 1 -- 172.21.15.52:0/761394454 shutdown_connections 2024-05-27T07:35:33.306 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:33.310+0000 7fababb6b700 1 --2- 172.21.15.52:0/761394454 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7faba411d920 0x7faba41a9a80 unknown :-1 s=CLOSED pgs=247 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:35:33.306 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:33.310+0000 7fababb6b700 1 --2- 172.21.15.52:0/761394454 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7faba410a920 0x7faba41af860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:35:33.306 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:33.310+0000 7fababb6b700 1 --2- 172.21.15.52:0/761394454 >> [v2:172.21.15.52:6800/1822318384,v1:172.21.15.52:6801/1822318384] conn(0x7fab900b2c00 0x7fab900b5110 unknown :-1 s=CLOSED pgs=186 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:35:33.306 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:33.310+0000 7fababb6b700 1 --2- 172.21.15.52:0/761394454 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7faba410b270 0x7faba41afda0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:35:33.306 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:33.310+0000 7fababb6b700 1 -- 172.21.15.52:0/761394454 >> 172.21.15.52:0/761394454 conn(0x7faba406f4c0 msgr2=0x7faba411b3d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:35:33.306 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:33.310+0000 7fababb6b700 1 -- 172.21.15.52:0/761394454 shutdown_connections 2024-05-27T07:35:33.306 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:33.310+0000 7fababb6b700 1 -- 172.21.15.52:0/761394454 wait complete. 2024-05-27T07:35:34.695 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:35:34 smithi105 ceph-mon[35977]: pgmap v323: 129 pgs: 17 active+clean+snaptrim, 55 active+clean+snaptrim_wait, 57 active+clean; 642 MiB data, 6.2 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 1.1 MiB/s wr, 221 op/s 2024-05-27T07:35:34.734 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:35:34 smithi052 ceph-mon[31955]: pgmap v323: 129 pgs: 17 active+clean+snaptrim, 55 active+clean+snaptrim_wait, 57 active+clean; 642 MiB data, 6.2 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 1.1 MiB/s wr, 221 op/s 2024-05-27T07:35:34.857 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:35:34 smithi173 ceph-mon[37000]: pgmap v323: 129 pgs: 17 active+clean+snaptrim, 55 active+clean+snaptrim_wait, 57 active+clean; 642 MiB data, 6.2 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 1.1 MiB/s wr, 221 op/s 2024-05-27T07:35:35.031 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-05-27T07:35:35.032 DEBUG:teuthology.orchestra.run.smithi052:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:38f9d900c83c63799fbdbe61acc9a11b0d3554a6 shell --fsid 73df2636-1bf8-11ef-bc9b-c7b262605968 -- ceph tell mds.1:0 scrub start / force,recursive 2024-05-27T07:35:35.523 INFO:teuthology.orchestra.run.smithi052.stderr:Inferring config /var/lib/ceph/73df2636-1bf8-11ef-bc9b-c7b262605968/mon.a/config 2024-05-27T07:35:36.357 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:35:36 smithi173 ceph-mon[37000]: pgmap v324: 129 pgs: 16 active+clean+snaptrim, 55 active+clean+snaptrim_wait, 58 active+clean; 645 MiB data, 6.3 GiB used, 1.0 TiB / 1.0 TiB avail; 21 KiB/s rd, 1.3 MiB/s wr, 225 op/s 2024-05-27T07:35:36.405 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:35:36 smithi052 ceph-mon[31955]: pgmap v324: 129 pgs: 16 active+clean+snaptrim, 55 active+clean+snaptrim_wait, 58 active+clean; 645 MiB data, 6.3 GiB used, 1.0 TiB / 1.0 TiB avail; 21 KiB/s rd, 1.3 MiB/s wr, 225 op/s 2024-05-27T07:35:36.445 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:35:36 smithi105 ceph-mon[35977]: pgmap v324: 129 pgs: 16 active+clean+snaptrim, 55 active+clean+snaptrim_wait, 58 active+clean; 645 MiB data, 6.3 GiB used, 1.0 TiB / 1.0 TiB avail; 21 KiB/s rd, 1.3 MiB/s wr, 225 op/s 2024-05-27T07:35:38.017 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:38.021+0000 7fdc6a9d7700 1 -- 172.21.15.52:0/413211463 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fdc641034b0 msgr2=0x7fdc641038c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:35:38.017 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:38.021+0000 7fdc6a9d7700 1 --2- 172.21.15.52:0/413211463 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fdc641034b0 0x7fdc641038c0 secure :-1 s=READY pgs=748 cs=0 l=1 rev1=1 crypto rx=0x7fdc540099c0 tx=0x7fdc54009cd0 comp rx=0 tx=0).stop 2024-05-27T07:35:38.018 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:38.023+0000 7fdc6a9d7700 1 -- 172.21.15.52:0/413211463 shutdown_connections 2024-05-27T07:35:38.018 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:38.023+0000 7fdc6a9d7700 1 --2- 172.21.15.52:0/413211463 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fdc64108730 0x7fdc64110090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:35:38.018 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:38.023+0000 7fdc6a9d7700 1 --2- 172.21.15.52:0/413211463 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fdc641034b0 0x7fdc641038c0 unknown :-1 s=CLOSED pgs=748 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:35:38.018 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:38.023+0000 7fdc6a9d7700 1 --2- 172.21.15.52:0/413211463 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fdc64107de0 0x7fdc641081f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:35:38.018 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:38.023+0000 7fdc6a9d7700 1 -- 172.21.15.52:0/413211463 >> 172.21.15.52:0/413211463 conn(0x7fdc640fedb0 msgr2=0x7fdc641011d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:35:38.019 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:38.024+0000 7fdc6a9d7700 1 -- 172.21.15.52:0/413211463 shutdown_connections 2024-05-27T07:35:38.020 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:38.024+0000 7fdc6a9d7700 1 -- 172.21.15.52:0/413211463 wait complete. 2024-05-27T07:35:38.020 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:38.025+0000 7fdc6a9d7700 1 Processor -- start 2024-05-27T07:35:38.022 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:38.026+0000 7fdc6a9d7700 1 -- start start 2024-05-27T07:35:38.022 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:38.027+0000 7fdc6a9d7700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fdc641034b0 0x7fdc64075330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:35:38.022 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:38.027+0000 7fdc6a9d7700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fdc64107de0 0x7fdc6406e460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:35:38.022 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:38.027+0000 7fdc6a9d7700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fdc64108730 0x7fdc6406ea20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:35:38.023 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:38.027+0000 7fdc6a9d7700 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x7fdc6411ab70 con 0x7fdc641034b0 2024-05-27T07:35:38.023 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:38.027+0000 7fdc6a9d7700 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7fdc6411a9f0 con 0x7fdc64107de0 2024-05-27T07:35:38.023 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:38.027+0000 7fdc6a9d7700 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7fdc6411acf0 con 0x7fdc64108730 2024-05-27T07:35:38.025 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:38.027+0000 7fdc68f74700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fdc64108730 0x7fdc6406ea20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:35:38.025 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:38.027+0000 7fdc637fe700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fdc64107de0 0x7fdc6406e460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:35:38.025 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:38.027+0000 7fdc68f74700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fdc64108730 0x7fdc6406ea20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.173:3300/0 says I am v2:172.21.15.52:51506/0 (socket says 172.21.15.52:51506) 2024-05-27T07:35:38.025 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:38.027+0000 7fdc637fe700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fdc64107de0 0x7fdc6406e460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.52:56648/0 (socket says 172.21.15.52:56648) 2024-05-27T07:35:38.025 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:38.027+0000 7fdc68f74700 1 -- 172.21.15.52:0/120949598 learned_addr learned my addr 172.21.15.52:0/120949598 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-05-27T07:35:38.025 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:38.028+0000 7fdc68f74700 1 -- 172.21.15.52:0/120949598 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fdc64107de0 msgr2=0x7fdc6406e460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:35:38.025 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:38.028+0000 7fdc68f74700 1 --2- 172.21.15.52:0/120949598 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fdc64107de0 0x7fdc6406e460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:35:38.025 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:38.028+0000 7fdc68f74700 1 -- 172.21.15.52:0/120949598 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fdc641034b0 msgr2=0x7fdc64075330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:35:38.025 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:38.028+0000 7fdc68f74700 1 --2- 172.21.15.52:0/120949598 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fdc641034b0 0x7fdc64075330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:35:38.026 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:38.028+0000 7fdc68f74700 1 -- 172.21.15.52:0/120949598 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdc6406f280 con 0x7fdc64108730 2024-05-27T07:35:38.026 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:38.029+0000 7fdc68f74700 1 --2- 172.21.15.52:0/120949598 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fdc64108730 0x7fdc6406ea20 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7fdc5c009c10 tx=0x7fdc5c00e5b0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:35:38.026 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:38.029+0000 7fdc617fa700 1 -- 172.21.15.52:0/120949598 <== mon.1 v2:172.21.15.173:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdc5c015030 con 0x7fdc64108730 2024-05-27T07:35:38.026 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:38.029+0000 7fdc6a9d7700 1 -- 172.21.15.52:0/120949598 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fdc64072f80 con 0x7fdc64108730 2024-05-27T07:35:38.026 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:38.029+0000 7fdc6a9d7700 1 -- 172.21.15.52:0/120949598 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fdc640734c0 con 0x7fdc64108730 2024-05-27T07:35:38.026 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:38.030+0000 7fdc617fa700 1 -- 172.21.15.52:0/120949598 <== mon.1 v2:172.21.15.173:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fdc5c00ee10 con 0x7fdc64108730 2024-05-27T07:35:38.026 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:38.030+0000 7fdc617fa700 1 -- 172.21.15.52:0/120949598 <== mon.1 v2:172.21.15.173:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdc5c0045e0 con 0x7fdc64108730 2024-05-27T07:35:38.032 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:38.034+0000 7fdc617fa700 1 -- 172.21.15.52:0/120949598 <== mon.1 v2:172.21.15.173:3300/0 4 ==== mgrmap(e 26) ==== 149783+0+0 (secure 0 0 0) 0x7fdc5c004ad0 con 0x7fdc64108730 2024-05-27T07:35:38.032 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:38.036+0000 7fdc617fa700 1 --2- 172.21.15.52:0/120949598 >> [v2:172.21.15.52:6800/1822318384,v1:172.21.15.52:6801/1822318384] conn(0x7fdc4c0b2c00 0x7fdc4c0b5110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:35:38.032 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:38.036+0000 7fdc617fa700 1 -- 172.21.15.52:0/120949598 <== mon.1 v2:172.21.15.173:3300/0 5 ==== osd_map(123..123 src has 1..123) ==== 9555+0+0 (secure 0 0 0) 0x7fdc5c0e2520 con 0x7fdc64108730 2024-05-27T07:35:38.033 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:38.036+0000 7fdc63fff700 1 --2- 172.21.15.52:0/120949598 >> [v2:172.21.15.52:6800/1822318384,v1:172.21.15.52:6801/1822318384] conn(0x7fdc4c0b2c00 0x7fdc4c0b5110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:35:38.033 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:38.037+0000 7fdc63fff700 1 --2- 172.21.15.52:0/120949598 >> [v2:172.21.15.52:6800/1822318384,v1:172.21.15.52:6801/1822318384] conn(0x7fdc4c0b2c00 0x7fdc4c0b5110 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7fdc540046b0 tx=0x7fdc54000bc0 comp rx=0 tx=0).ready entity=mgr.15333 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:35:38.039 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:38.044+0000 7fdc793ecb80 1 Processor -- start 2024-05-27T07:35:38.043 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:38.048+0000 7fdc793ecb80 1 -- start start 2024-05-27T07:35:38.044 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:38.049+0000 7fdc793ecb80 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55c9f23adec0 0x55c9f2532bb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:35:38.044 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:38.049+0000 7fdc793ecb80 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55c9f25330f0 0x55c9f252cc60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:35:38.044 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:38.049+0000 7fdc793ecb80 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55c9f252d1a0 0x55c9f252d5b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:35:38.045 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:38.049+0000 7fdc793ecb80 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x55c9f245adf0 con 0x55c9f25330f0 2024-05-27T07:35:38.045 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:38.049+0000 7fdc793ecb80 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x55c9f244a990 con 0x55c9f252d1a0 2024-05-27T07:35:38.045 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:38.049+0000 7fdc793ecb80 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x55c9f244e9d0 con 0x55c9f23adec0 2024-05-27T07:35:38.045 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:38.049+0000 7fdc63fff700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55c9f23adec0 0x55c9f2532bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:35:38.045 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:38.049+0000 7fdc63fff700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55c9f23adec0 0x55c9f2532bb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.173:3300/0 says I am v2:172.21.15.52:51518/0 (socket says 172.21.15.52:51518) 2024-05-27T07:35:38.045 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:38.049+0000 7fdc63fff700 1 -- 172.21.15.52:0/3994507872 learned_addr learned my addr 172.21.15.52:0/3994507872 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-05-27T07:35:38.046 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:38.050+0000 7fdc637fe700 1 --2- 172.21.15.52:0/3994507872 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55c9f25330f0 0x55c9f252cc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:35:38.046 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:38.050+0000 7fdc63fff700 1 -- 172.21.15.52:0/3994507872 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55c9f252d1a0 msgr2=0x55c9f252d5b0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-05-27T07:35:38.047 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:38.050+0000 7fdc63fff700 1 --2- 172.21.15.52:0/3994507872 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55c9f252d1a0 0x55c9f252d5b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:35:38.047 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:38.050+0000 7fdc63fff700 1 -- 172.21.15.52:0/3994507872 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55c9f25330f0 msgr2=0x55c9f252cc60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:35:38.047 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:38.050+0000 7fdc63fff700 1 --2- 172.21.15.52:0/3994507872 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55c9f25330f0 0x55c9f252cc60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:35:38.047 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:38.050+0000 7fdc63fff700 1 -- 172.21.15.52:0/3994507872 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55c9f23f4030 con 0x55c9f23adec0 2024-05-27T07:35:38.047 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:38.050+0000 7fdc637fe700 1 --2- 172.21.15.52:0/3994507872 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55c9f25330f0 0x55c9f252cc60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-27T07:35:38.047 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:38.050+0000 7fdc63fff700 1 --2- 172.21.15.52:0/3994507872 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55c9f23adec0 0x55c9f2532bb0 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7fdc540046e0 tx=0x7fdc5400f000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=3ba89055809639b5 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:35:38.047 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:38.050+0000 7fdc4a7fc700 1 -- 172.21.15.52:0/3994507872 <== mon.1 v2:172.21.15.173:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdc54010430 con 0x55c9f23adec0 2024-05-27T07:35:38.047 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:38.051+0000 7fdc4a7fc700 1 -- 172.21.15.52:0/3994507872 <== mon.1 v2:172.21.15.173:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fdc540105d0 con 0x55c9f23adec0 2024-05-27T07:35:38.048 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:38.052+0000 7fdc4a7fc700 1 -- 172.21.15.52:0/3994507872 <== mon.1 v2:172.21.15.173:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdc5400fb30 con 0x55c9f23adec0 2024-05-27T07:35:38.049 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:38.052+0000 7fdc793ecb80 1 -- 172.21.15.52:0/3994507872 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55c9f23adec0 msgr2=0x55c9f2532bb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:35:38.049 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:38.052+0000 7fdc793ecb80 1 --2- 172.21.15.52:0/3994507872 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55c9f23adec0 0x55c9f2532bb0 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7fdc540046e0 tx=0x7fdc5400f000 comp rx=0 tx=0).stop 2024-05-27T07:35:38.049 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:38.052+0000 7fdc793ecb80 1 -- 172.21.15.52:0/3994507872 shutdown_connections 2024-05-27T07:35:38.049 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:38.052+0000 7fdc793ecb80 1 --2- 172.21.15.52:0/3994507872 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55c9f252d1a0 0x55c9f252d5b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:35:38.049 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:38.052+0000 7fdc793ecb80 1 --2- 172.21.15.52:0/3994507872 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55c9f23adec0 0x55c9f2532bb0 unknown :-1 s=CLOSED pgs=213 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:35:38.049 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:38.052+0000 7fdc793ecb80 1 --2- 172.21.15.52:0/3994507872 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55c9f25330f0 0x55c9f252cc60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:35:38.049 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:38.052+0000 7fdc793ecb80 1 -- 172.21.15.52:0/3994507872 >> 172.21.15.52:0/3994507872 conn(0x55c9f23b07f0 msgr2=0x55c9f2527e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:35:38.049 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:38.052+0000 7fdc793ecb80 1 -- 172.21.15.52:0/3994507872 shutdown_connections 2024-05-27T07:35:38.049 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:38.052+0000 7fdc793ecb80 1 -- 172.21.15.52:0/3994507872 wait complete. 2024-05-27T07:35:38.049 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:38.053+0000 7fdc793ecb80 1 -- start start 2024-05-27T07:35:38.050 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:38.053+0000 7fdc793ecb80 1 Processor -- start 2024-05-27T07:35:38.053 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:38.054+0000 7fdc793ecb80 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55c9f23adec0 0x55c9f25367e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:35:38.053 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:38.054+0000 7fdc793ecb80 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55c9f252d1a0 0x55c9f2536d20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:35:38.053 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:38.054+0000 7fdc793ecb80 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55c9f25330f0 0x55c9f2537260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:35:38.053 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:38.054+0000 7fdc793ecb80 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x55c9f2351020 con 0x55c9f25330f0 2024-05-27T07:35:38.053 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:38.054+0000 7fdc793ecb80 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x55c9f23b2e20 con 0x55c9f23adec0 2024-05-27T07:35:38.053 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:38.054+0000 7fdc793ecb80 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x55c9f2539bc0 con 0x55c9f252d1a0 2024-05-27T07:35:38.053 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:38.054+0000 7fdc793ecb80 10 client.0 _ll_register_callbacks cb 0x55c9f23e1630 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T07:35:38.053 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:38.054+0000 7fdc63fff700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55c9f23adec0 0x55c9f25367e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:35:38.053 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:38.054+0000 7fdc63fff700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55c9f23adec0 0x55c9f25367e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.52:56654/0 (socket says 172.21.15.52:56654) 2024-05-27T07:35:38.053 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:38.054+0000 7fdc63fff700 1 -- 172.21.15.52:0/3555232007 learned_addr learned my addr 172.21.15.52:0/3555232007 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-05-27T07:35:38.054 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:38.055+0000 7fdc63fff700 1 -- 172.21.15.52:0/3555232007 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55c9f252d1a0 msgr2=0x55c9f2536d20 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-05-27T07:35:38.054 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:38.055+0000 7fdc63fff700 1 --2- 172.21.15.52:0/3555232007 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55c9f252d1a0 0x55c9f2536d20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:35:38.054 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:38.055+0000 7fdc63fff700 1 -- 172.21.15.52:0/3555232007 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55c9f25330f0 msgr2=0x55c9f2537260 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-05-27T07:35:38.054 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:38.055+0000 7fdc63fff700 1 --2- 172.21.15.52:0/3555232007 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55c9f25330f0 0x55c9f2537260 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:35:38.054 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:38.055+0000 7fdc63fff700 1 -- 172.21.15.52:0/3555232007 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55c9f239a950 con 0x55c9f23adec0 2024-05-27T07:35:38.054 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:38.055+0000 7fdc63fff700 1 --2- 172.21.15.52:0/3555232007 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55c9f23adec0 0x55c9f25367e0 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7fdc5400fd60 tx=0x7fdc5400f450 comp rx=0 tx=0).ready entity=mon.2 client_cookie=bb17e05b4cb8bbc server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:35:38.054 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:38.055+0000 7fdc437fe700 10 client.0 ms_handle_connect on v2:172.21.15.105:3300/0 2024-05-27T07:35:38.054 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:38.055+0000 7fdc437fe700 1 -- 172.21.15.52:0/3555232007 <== mon.2 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdc54039020 con 0x55c9f23adec0 2024-05-27T07:35:38.054 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:38.056+0000 7fdc793ecb80 1 -- 172.21.15.52:0/3555232007 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fdc540105d0 con 0x55c9f23adec0 2024-05-27T07:35:38.054 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:38.056+0000 7fdc437fe700 1 -- 172.21.15.52:0/3555232007 <== mon.2 v2:172.21.15.105:3300/0 2 ==== osd_map(123..123 src has 1..123) ==== 9555+0+0 (secure 0 0 0) 0x7fdc54031040 con 0x55c9f23adec0 2024-05-27T07:35:38.055 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:38.056+0000 7fdc437fe700 1 -- 172.21.15.52:0/3555232007 <== mon.2 v2:172.21.15.105:3300/0 3 ==== mon_get_version_reply(handle=1 version=108) ==== 24+0+0 (secure 0 0 0) 0x7fdc5403f910 con 0x55c9f23adec0 2024-05-27T07:35:38.055 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:38.056+0000 7fdc793ecb80 10 client.26819 fetch_fsmap learned FSMap version 108 2024-05-27T07:35:38.055 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:38.056+0000 7fdc793ecb80 1 -- 172.21.15.52:0/3555232007 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,fsmap=108,monmap=4+}) -- 0x55c9f24047e0 con 0x55c9f23adec0 2024-05-27T07:35:38.055 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:38.057+0000 7fdc437fe700 1 -- 172.21.15.52:0/3555232007 <== mon.2 v2:172.21.15.105:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fdc540105d0 con 0x55c9f23adec0 2024-05-27T07:35:38.055 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:38.057+0000 7fdc437fe700 1 -- 172.21.15.52:0/3555232007 <== mon.2 v2:172.21.15.105:3300/0 5 ==== fsmap(e 108) ==== 4203+0+0 (secure 0 0 0) 0x7fdc5403f4d0 con 0x55c9f23adec0 2024-05-27T07:35:38.055 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:38.057+0000 7fdc793ecb80 10 client.26819 fetch_fsmap finished waiting for FSMap version 108 2024-05-27T07:35:38.055 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:38.057+0000 7fdc793ecb80 10 client.26819 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-05-27T07:35:38.055 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:38.057+0000 7fdc793ecb80 20 client.26819 populate_metadata read hostname 'smithi052' 2024-05-27T07:35:38.055 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:38.057+0000 7fdc793ecb80 1 --2- 172.21.15.52:0/3555232007 >> [v2:172.21.15.105:6838/500348151,v1:172.21.15.105:6839/500348151] conn(0x55c9f2546b00 0x55c9f2548f90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:35:38.055 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:38.057+0000 7fdc793ecb80 4 client.26819 mds_command: new command op to 24527 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-27T07:35:38.055 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:38.057+0000 7fdc793ecb80 1 -- 172.21.15.52:0/3555232007 --> [v2:172.21.15.105:6838/500348151,v1:172.21.15.105:6839/500348151] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55c9f245cd40 con 0x55c9f2546b00 2024-05-27T07:35:38.055 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:38.057+0000 7fdc637fe700 1 --2- 172.21.15.52:0/3555232007 >> [v2:172.21.15.105:6838/500348151,v1:172.21.15.105:6839/500348151] conn(0x55c9f2546b00 0x55c9f2548f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:35:38.056 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:38.059+0000 7fdc637fe700 1 --2- 172.21.15.52:0/3555232007 >> [v2:172.21.15.105:6838/500348151,v1:172.21.15.105:6839/500348151] conn(0x55c9f2546b00 0x55c9f2548f90 crc :-1 s=READY pgs=169 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=44f81f3ee7b70f5 server_cookie=4b5352ae696685ce in_seq=0 out_seq=0 2024-05-27T07:35:38.057 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:38.059+0000 7fdc437fe700 10 client.26819 ms_handle_connect on v2:172.21.15.105:6838/500348151 2024-05-27T07:35:38.057 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:38.060+0000 7fdc437fe700 1 -- 172.21.15.52:0/3555232007 <== mds.0 v2:172.21.15.105:6838/500348151 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24041 (crc 0 0 0) 0x55c9f245cd40 con 0x55c9f2546b00 2024-05-27T07:35:38.057 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:38.060+0000 7fdc437fe700 10 client.26819 handle_command_reply: tid=0 2024-05-27T07:35:38.057 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:38.060+0000 7fdc793ecb80 1 client.26819 shutdown 2024-05-27T07:35:38.057 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:38.061+0000 7fdc793ecb80 1 -- 172.21.15.52:0/3555232007 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55c9f23adec0 msgr2=0x55c9f25367e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:35:38.057 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:38.061+0000 7fdc793ecb80 1 --2- 172.21.15.52:0/3555232007 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55c9f23adec0 0x55c9f25367e0 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7fdc5400fd60 tx=0x7fdc5400f450 comp rx=0 tx=0).stop 2024-05-27T07:35:38.058 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:38.062+0000 7fdc793ecb80 1 -- 172.21.15.52:0/3555232007 shutdown_connections 2024-05-27T07:35:38.058 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:38.062+0000 7fdc793ecb80 1 --2- 172.21.15.52:0/3555232007 >> [v2:172.21.15.105:6838/500348151,v1:172.21.15.105:6839/500348151] conn(0x55c9f2546b00 0x55c9f2548f90 crc :-1 s=READY pgs=169 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:35:38.059 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:38.062+0000 7fdc793ecb80 1 --2- 172.21.15.52:0/3555232007 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55c9f25330f0 0x55c9f2537260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:35:38.059 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:38.062+0000 7fdc793ecb80 1 --2- 172.21.15.52:0/3555232007 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55c9f23adec0 0x55c9f25367e0 unknown :-1 s=CLOSED pgs=249 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:35:38.059 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:38.062+0000 7fdc793ecb80 1 --2- 172.21.15.52:0/3555232007 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55c9f252d1a0 0x55c9f2536d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:35:38.059 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:38.062+0000 7fdc793ecb80 1 -- 172.21.15.52:0/3555232007 >> 172.21.15.52:0/3555232007 conn(0x55c9f23b07f0 msgr2=0x55c9f252bc50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:35:38.059 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:38.062+0000 7fdc437fe700 0 client.26819 ms_handle_reset on v2:172.21.15.105:6838/500348151 2024-05-27T07:35:38.059 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:38.063+0000 7fdc793ecb80 1 -- 172.21.15.52:0/3555232007 shutdown_connections 2024-05-27T07:35:38.059 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:38.063+0000 7fdc793ecb80 1 -- 172.21.15.52:0/3555232007 wait complete. 2024-05-27T07:35:38.060 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:38.064+0000 7fdc793ecb80 20 client.26819 trim_cache size 0 max 16384 2024-05-27T07:35:38.099 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:38.104+0000 7fdc793ecb80 1 Processor -- start 2024-05-27T07:35:38.100 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:38.105+0000 7fdc793ecb80 1 -- start start 2024-05-27T07:35:38.101 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:38.105+0000 7fdc793ecb80 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55c9f23adec0 0x55c9f252cc60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:35:38.101 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:38.106+0000 7fdc793ecb80 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55c9f252d1a0 0x55c9f2549180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:35:38.101 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:38.106+0000 7fdc793ecb80 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55c9f25330f0 0x55c9f2531e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:35:38.101 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:38.106+0000 7fdc793ecb80 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x7fdc5000f3d0 con 0x55c9f252d1a0 2024-05-27T07:35:38.101 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:38.106+0000 7fdc793ecb80 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7fdc5000d800 con 0x55c9f25330f0 2024-05-27T07:35:38.102 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:38.106+0000 7fdc793ecb80 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7fdc5000ecd0 con 0x55c9f23adec0 2024-05-27T07:35:38.105 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:38.110+0000 7fdc637fe700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55c9f252d1a0 0x55c9f2549180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:35:38.105 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:38.110+0000 7fdc637fe700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55c9f252d1a0 0x55c9f2549180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.52:53754/0 (socket says 172.21.15.52:53754) 2024-05-27T07:35:38.105 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:38.110+0000 7fdc637fe700 1 -- 172.21.15.52:0/3947273319 learned_addr learned my addr 172.21.15.52:0/3947273319 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-05-27T07:35:38.105 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:38.110+0000 7fdc63fff700 1 --2- 172.21.15.52:0/3947273319 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55c9f23adec0 0x55c9f252cc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:35:38.107 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:38.111+0000 7fdc637fe700 1 -- 172.21.15.52:0/3947273319 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55c9f23adec0 msgr2=0x55c9f252cc60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:35:38.107 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:38.111+0000 7fdc637fe700 1 --2- 172.21.15.52:0/3947273319 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55c9f23adec0 0x55c9f252cc60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:35:38.107 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:38.111+0000 7fdc637fe700 1 -- 172.21.15.52:0/3947273319 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55c9f25330f0 msgr2=0x55c9f2531e50 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-05-27T07:35:38.107 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:38.111+0000 7fdc637fe700 1 --2- 172.21.15.52:0/3947273319 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55c9f25330f0 0x55c9f2531e50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:35:38.107 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:38.111+0000 7fdc637fe700 1 -- 172.21.15.52:0/3947273319 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55c9f2532390 con 0x55c9f252d1a0 2024-05-27T07:35:38.108 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:38.112+0000 7fdc637fe700 1 --2- 172.21.15.52:0/3947273319 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55c9f252d1a0 0x55c9f2549180 secure :-1 s=READY pgs=749 cs=0 l=1 rev1=1 crypto rx=0x7fdc64102430 tx=0x7fdc5800bd60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7a495734e0d51a00 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:35:38.112 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:38.115+0000 7fdc49ffb700 1 -- 172.21.15.52:0/3947273319 <== mon.0 v2:172.21.15.52:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdc58024070 con 0x55c9f252d1a0 2024-05-27T07:35:38.112 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:38.115+0000 7fdc49ffb700 1 -- 172.21.15.52:0/3947273319 <== mon.0 v2:172.21.15.52:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fdc58016d20 con 0x55c9f252d1a0 2024-05-27T07:35:38.112 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:38.116+0000 7fdc793ecb80 1 -- 172.21.15.52:0/3947273319 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55c9f252d1a0 msgr2=0x55c9f2549180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:35:38.112 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:38.116+0000 7fdc793ecb80 1 --2- 172.21.15.52:0/3947273319 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55c9f252d1a0 0x55c9f2549180 secure :-1 s=READY pgs=749 cs=0 l=1 rev1=1 crypto rx=0x7fdc64102430 tx=0x7fdc5800bd60 comp rx=0 tx=0).stop 2024-05-27T07:35:38.117 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:38.121+0000 7fdc793ecb80 1 -- 172.21.15.52:0/3947273319 shutdown_connections 2024-05-27T07:35:38.117 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:38.121+0000 7fdc793ecb80 1 --2- 172.21.15.52:0/3947273319 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55c9f25330f0 0x55c9f2531e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:35:38.117 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:38.121+0000 7fdc793ecb80 1 --2- 172.21.15.52:0/3947273319 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55c9f23adec0 0x55c9f252cc60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:35:38.118 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:38.121+0000 7fdc793ecb80 1 --2- 172.21.15.52:0/3947273319 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55c9f252d1a0 0x55c9f2549180 unknown :-1 s=CLOSED pgs=749 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:35:38.118 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:38.121+0000 7fdc793ecb80 1 -- 172.21.15.52:0/3947273319 >> 172.21.15.52:0/3947273319 conn(0x55c9f23b07f0 msgr2=0x55c9f2531740 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:35:38.124 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:38.129+0000 7fdc793ecb80 1 -- 172.21.15.52:0/3947273319 shutdown_connections 2024-05-27T07:35:38.126 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:38.130+0000 7fdc793ecb80 1 -- 172.21.15.52:0/3947273319 wait complete. 2024-05-27T07:35:38.127 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:38.130+0000 7fdc793ecb80 1 -- start start 2024-05-27T07:35:38.127 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:38.130+0000 7fdc793ecb80 1 Processor -- start 2024-05-27T07:35:38.129 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:38.131+0000 7fdc793ecb80 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55c9f23adec0 0x55c9f253f2b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:35:38.129 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:38.131+0000 7fdc793ecb80 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55c9f252d1a0 0x55c9f254dc20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:35:38.130 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:38.131+0000 7fdc793ecb80 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55c9f25330f0 0x55c9f254e160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:35:38.130 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:38.131+0000 7fdc793ecb80 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x7fdc5000e600 con 0x55c9f252d1a0 2024-05-27T07:35:38.130 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:38.131+0000 7fdc793ecb80 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7fdc5000df00 con 0x55c9f23adec0 2024-05-27T07:35:38.130 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:38.131+0000 7fdc793ecb80 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x55c9f245cd40 con 0x55c9f25330f0 2024-05-27T07:35:38.130 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:38.131+0000 7fdc793ecb80 10 client.0 _ll_register_callbacks cb 0x55c9f23e1630 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T07:35:38.130 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:38.132+0000 7fdc637fe700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55c9f252d1a0 0x55c9f254dc20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:35:38.130 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:38.132+0000 7fdc637fe700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55c9f252d1a0 0x55c9f254dc20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.52:53766/0 (socket says 172.21.15.52:53766) 2024-05-27T07:35:38.130 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:38.132+0000 7fdc637fe700 1 -- 172.21.15.52:0/4139466519 learned_addr learned my addr 172.21.15.52:0/4139466519 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-05-27T07:35:38.130 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:38.133+0000 7fdc637fe700 1 -- 172.21.15.52:0/4139466519 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55c9f25330f0 msgr2=0x55c9f254e160 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-05-27T07:35:38.130 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:38.133+0000 7fdc637fe700 1 --2- 172.21.15.52:0/4139466519 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55c9f25330f0 0x55c9f254e160 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:35:38.131 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:38.133+0000 7fdc637fe700 1 -- 172.21.15.52:0/4139466519 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55c9f23adec0 msgr2=0x55c9f253f2b0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-05-27T07:35:38.131 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:38.133+0000 7fdc637fe700 1 --2- 172.21.15.52:0/4139466519 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55c9f23adec0 0x55c9f253f2b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:35:38.131 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:38.133+0000 7fdc637fe700 1 -- 172.21.15.52:0/4139466519 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55c9f254e6a0 con 0x55c9f252d1a0 2024-05-27T07:35:38.131 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:38.134+0000 7fdc637fe700 1 --2- 172.21.15.52:0/4139466519 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55c9f252d1a0 0x55c9f254dc20 secure :-1 s=READY pgs=750 cs=0 l=1 rev1=1 crypto rx=0x7fdc5801f970 tx=0x7fdc5800bf40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a57f13e7ddedee5d server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:35:38.131 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:38.134+0000 7fdc437fe700 10 client.0 ms_handle_connect on v2:172.21.15.52:3300/0 2024-05-27T07:35:38.131 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:38.135+0000 7fdc437fe700 1 -- 172.21.15.52:0/4139466519 <== mon.0 v2:172.21.15.52:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdc5802c020 con 0x55c9f252d1a0 2024-05-27T07:35:38.132 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:38.135+0000 7fdc437fe700 1 -- 172.21.15.52:0/4139466519 <== mon.0 v2:172.21.15.52:3300/0 2 ==== osd_map(123..123 src has 1..123) ==== 9555+0+0 (secure 0 0 0) 0x7fdc5801f680 con 0x55c9f252d1a0 2024-05-27T07:35:38.132 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:38.135+0000 7fdc793ecb80 1 -- 172.21.15.52:0/4139466519 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fdc58016d20 con 0x55c9f252d1a0 2024-05-27T07:35:38.132 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:38.136+0000 7fdc437fe700 1 -- 172.21.15.52:0/4139466519 <== mon.0 v2:172.21.15.52:3300/0 3 ==== mon_get_version_reply(handle=1 version=108) ==== 24+0+0 (secure 0 0 0) 0x7fdc58024040 con 0x55c9f252d1a0 2024-05-27T07:35:38.133 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:38.137+0000 7fdc793ecb80 10 client.17169 fetch_fsmap learned FSMap version 108 2024-05-27T07:35:38.133 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:38.137+0000 7fdc793ecb80 1 -- 172.21.15.52:0/4139466519 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({config=0+,fsmap=108,monmap=4+}) -- 0x55c9f25c21b0 con 0x55c9f252d1a0 2024-05-27T07:35:38.135 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:38.139+0000 7fdc437fe700 1 -- 172.21.15.52:0/4139466519 <== mon.0 v2:172.21.15.52:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fdc58016d20 con 0x55c9f252d1a0 2024-05-27T07:35:38.135 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:38.139+0000 7fdc437fe700 1 -- 172.21.15.52:0/4139466519 <== mon.0 v2:172.21.15.52:3300/0 5 ==== fsmap(e 108) ==== 4203+0+0 (secure 0 0 0) 0x7fdc580038c0 con 0x55c9f252d1a0 2024-05-27T07:35:38.135 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:38.139+0000 7fdc793ecb80 10 client.17169 fetch_fsmap finished waiting for FSMap version 108 2024-05-27T07:35:38.135 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:38.139+0000 7fdc793ecb80 10 client.17169 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-05-27T07:35:38.135 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:38.139+0000 7fdc793ecb80 20 client.17169 populate_metadata read hostname 'smithi052' 2024-05-27T07:35:38.135 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:38.139+0000 7fdc793ecb80 1 --2- 172.21.15.52:0/4139466519 >> [v2:172.21.15.105:6838/500348151,v1:172.21.15.105:6839/500348151] conn(0x55c9f25c27f0 0x55c9f25c4c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:35:38.135 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:38.139+0000 7fdc793ecb80 4 client.17169 mds_command: new command op to 24527 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-05-27T07:35:38.135 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:38.139+0000 7fdc793ecb80 1 -- 172.21.15.52:0/4139466519 --> [v2:172.21.15.105:6838/500348151,v1:172.21.15.105:6839/500348151] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) -- 0x55c9f2456520 con 0x55c9f25c27f0 2024-05-27T07:35:38.136 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:38.140+0000 7fdc63fff700 1 --2- 172.21.15.52:0/4139466519 >> [v2:172.21.15.105:6838/500348151,v1:172.21.15.105:6839/500348151] conn(0x55c9f25c27f0 0x55c9f25c4c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:35:38.136 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:38.141+0000 7fdc63fff700 1 --2- 172.21.15.52:0/4139466519 >> [v2:172.21.15.105:6838/500348151,v1:172.21.15.105:6839/500348151] conn(0x55c9f25c27f0 0x55c9f25c4c10 crc :-1 s=READY pgs=170 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=66af048cbdece7af server_cookie=ae562eecfeef9786 in_seq=0 out_seq=0 2024-05-27T07:35:38.136 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:38.141+0000 7fdc437fe700 10 client.17169 ms_handle_connect on v2:172.21.15.105:6838/500348151 2024-05-27T07:35:38.139 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:38.143+0000 7fdc437fe700 1 -- 172.21.15.52:0/4139466519 <== mds.0 v2:172.21.15.105:6838/500348151 1 ==== command_reply(tid 0: 0 ) ==== 8+0+110 (crc 0 0 0) 0x55c9f2456520 con 0x55c9f25c27f0 2024-05-27T07:35:38.139 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:38.143+0000 7fdc437fe700 10 client.17169 handle_command_reply: tid=0 2024-05-27T07:35:38.139 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:38.144+0000 7fdc793ecb80 1 client.17169 shutdown 2024-05-27T07:35:38.145 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:38.148+0000 7fdc793ecb80 1 -- 172.21.15.52:0/4139466519 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55c9f252d1a0 msgr2=0x55c9f254dc20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:35:38.145 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:38.148+0000 7fdc793ecb80 1 --2- 172.21.15.52:0/4139466519 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55c9f252d1a0 0x55c9f254dc20 secure :-1 s=READY pgs=750 cs=0 l=1 rev1=1 crypto rx=0x7fdc5801f970 tx=0x7fdc5800bf40 comp rx=0 tx=0).stop 2024-05-27T07:35:38.146 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:38.148+0000 7fdc793ecb80 1 -- 172.21.15.52:0/4139466519 shutdown_connections 2024-05-27T07:35:38.146 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:38.148+0000 7fdc793ecb80 1 --2- 172.21.15.52:0/4139466519 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55c9f25330f0 0x55c9f254e160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:35:38.146 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:38.148+0000 7fdc793ecb80 1 --2- 172.21.15.52:0/4139466519 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55c9f23adec0 0x55c9f253f2b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:35:38.146 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:38.148+0000 7fdc793ecb80 1 --2- 172.21.15.52:0/4139466519 >> [v2:172.21.15.105:6838/500348151,v1:172.21.15.105:6839/500348151] conn(0x55c9f25c27f0 0x55c9f25c4c10 crc :-1 s=READY pgs=170 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:35:38.146 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:38.149+0000 7fdc793ecb80 1 --2- 172.21.15.52:0/4139466519 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55c9f252d1a0 0x55c9f254dc20 unknown :-1 s=CLOSED pgs=750 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:35:38.146 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:38.149+0000 7fdc793ecb80 1 -- 172.21.15.52:0/4139466519 >> 172.21.15.52:0/4139466519 conn(0x55c9f23b07f0 msgr2=0x55c9f2521570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:35:38.146 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:38.149+0000 7fdc437fe700 0 client.17169 ms_handle_reset on v2:172.21.15.105:6838/500348151 2024-05-27T07:35:38.146 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:38.150+0000 7fdc793ecb80 1 -- 172.21.15.52:0/4139466519 shutdown_connections 2024-05-27T07:35:38.146 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:38.150+0000 7fdc793ecb80 1 -- 172.21.15.52:0/4139466519 wait complete. 2024-05-27T07:35:38.146 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:38.150+0000 7fdc793ecb80 20 client.17169 trim_cache size 0 max 16384 2024-05-27T07:35:38.147 INFO:teuthology.orchestra.run.smithi052.stdout:{ 2024-05-27T07:35:38.148 INFO:teuthology.orchestra.run.smithi052.stdout: "return_code": 0, 2024-05-27T07:35:38.148 INFO:teuthology.orchestra.run.smithi052.stdout: "scrub_tag": "6e2d3879-5958-41b8-8d1b-86e2464004b8", 2024-05-27T07:35:38.148 INFO:teuthology.orchestra.run.smithi052.stdout: "mode": "asynchronous" 2024-05-27T07:35:38.148 INFO:teuthology.orchestra.run.smithi052.stdout:} 2024-05-27T07:35:38.149 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:38.153+0000 7fdc6a9d7700 1 -- 172.21.15.52:0/120949598 >> [v2:172.21.15.52:6800/1822318384,v1:172.21.15.52:6801/1822318384] conn(0x7fdc4c0b2c00 msgr2=0x7fdc4c0b5110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:35:38.149 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:38.153+0000 7fdc6a9d7700 1 --2- 172.21.15.52:0/120949598 >> [v2:172.21.15.52:6800/1822318384,v1:172.21.15.52:6801/1822318384] conn(0x7fdc4c0b2c00 0x7fdc4c0b5110 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7fdc540046b0 tx=0x7fdc54000bc0 comp rx=0 tx=0).stop 2024-05-27T07:35:38.149 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:38.153+0000 7fdc6a9d7700 1 -- 172.21.15.52:0/120949598 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fdc64108730 msgr2=0x7fdc6406ea20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:35:38.149 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:38.153+0000 7fdc6a9d7700 1 --2- 172.21.15.52:0/120949598 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fdc64108730 0x7fdc6406ea20 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7fdc5c009c10 tx=0x7fdc5c00e5b0 comp rx=0 tx=0).stop 2024-05-27T07:35:38.149 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:38.153+0000 7fdc6a9d7700 1 -- 172.21.15.52:0/120949598 shutdown_connections 2024-05-27T07:35:38.149 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:38.153+0000 7fdc6a9d7700 1 --2- 172.21.15.52:0/120949598 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fdc64108730 0x7fdc6406ea20 unknown :-1 s=CLOSED pgs=212 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:35:38.149 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:38.153+0000 7fdc6a9d7700 1 --2- 172.21.15.52:0/120949598 >> [v2:172.21.15.52:6800/1822318384,v1:172.21.15.52:6801/1822318384] conn(0x7fdc4c0b2c00 0x7fdc4c0b5110 unknown :-1 s=CLOSED pgs=187 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:35:38.149 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:38.153+0000 7fdc6a9d7700 1 --2- 172.21.15.52:0/120949598 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fdc641034b0 0x7fdc64075330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:35:38.149 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:38.153+0000 7fdc6a9d7700 1 --2- 172.21.15.52:0/120949598 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fdc64107de0 0x7fdc6406e460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:35:38.149 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:38.153+0000 7fdc6a9d7700 1 -- 172.21.15.52:0/120949598 >> 172.21.15.52:0/120949598 conn(0x7fdc640fedb0 msgr2=0x7fdc64107580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:35:38.150 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:38.155+0000 7fdc6a9d7700 1 -- 172.21.15.52:0/120949598 shutdown_connections 2024-05-27T07:35:38.150 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:38.155+0000 7fdc6a9d7700 1 -- 172.21.15.52:0/120949598 wait complete. 2024-05-27T07:35:38.484 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:35:38 smithi052 ceph-mon[31955]: pgmap v325: 129 pgs: 12 active+clean+snaptrim, 39 active+clean+snaptrim_wait, 78 active+clean; 645 MiB data, 6.3 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 1.3 MiB/s wr, 235 op/s 2024-05-27T07:35:38.607 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:35:38 smithi173 ceph-mon[37000]: pgmap v325: 129 pgs: 12 active+clean+snaptrim, 39 active+clean+snaptrim_wait, 78 active+clean; 645 MiB data, 6.3 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 1.3 MiB/s wr, 235 op/s 2024-05-27T07:35:38.695 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:35:38 smithi105 ceph-mon[35977]: pgmap v325: 129 pgs: 12 active+clean+snaptrim, 39 active+clean+snaptrim_wait, 78 active+clean; 645 MiB data, 6.3 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 1.3 MiB/s wr, 235 op/s 2024-05-27T07:35:38.696 INFO:journalctl@ceph.mds.b.smithi105.stdout:May 27 07:35:38 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-b[65543]: 2024-05-27T07:35:38.326+0000 7f33fced0700 -1 mds.0.scrubstack _validate_inode_done scrub error on inode [inode 0x10000003fad [...a,head] /volumes/qa/sv_1/95e9ac73-825f-48e3-998b-133d94d96559/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-58/ auth{4=1} fragtree_t(*^3) v239494 ap=2 RANDEPHEMERALPIN f(v9 m2024-05-27T07:32:27.400186+0000 105=105+0) n(v14 rc2024-05-27T07:32:27.400186+0000 b2409600 106=105+1) old_inodes=4 (inest sync->mix g=4) caps={25060=pAsLsXsFs/pAsLsXsFsx@226} | dirtyscattered=0 request=0 lock=0 dirfrag=8 caps=1 dirtyparent=0 replicated=1 dirty=0 waiter=0 authpin=1 discoverbase=0 scrubqueue=0 randepin 0x55ee54bf8b00]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":true,"read_ret_val":0,"ondisk_value":"(2)0x10000003fad:[<0x10000000399/blog-58 v152699>,<0x10000000208/blogtest_in v814>,<0x10000000201/src v2182>,<0x10000000006/blogbench-1.0 v1931>,<0x10000000005/tmp v231>,<0x100000001fb/client.0 v208>,<0x100000001fa/95e9ac73-825f-48e3-998b-133d94d96559 v218>,<0x10000000001/sv_1 v230>,<0x10000000000/qa v222>,<0x1/volumes v215>]//[]","memoryvalue":"(2)0x10000003fad:[<0x10000000399/blog-58 v239494>,<0x10000000208/blogtest_in v2501>,<0x10000000201/src v2932>,<0x10000000006/blogbench-1.0 v2530>,<0x10000000005/tmp v807>,<0x100000001fb/client.0 v770>,<0x100000001fa/95e9ac73-825f-48e3-998b-133d94d96559 v772>,<0x10000000001/sv_1 v783>,<0x10000000000/qa v754>,<0x1/volumes v761>]//[]","error_str":""},"raw_stats":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f(v0 m2024-05-27T07:31:35.406784+0000 104=104+0)","ondisk_value.rstat":"n(v0 rc2024-05-27T07:32:27.400186+0000 b2409600 106=105+1)","memory_value.dirstat":"f(v9 m2024-05-27T07:32:27.400186+0000 105=105+0)","memory_value.rstat":"n(v14 rc2024-05-27T07:32:27.400186+0000 b2409600 106=105+1)","error_str":"freshly-calculated rstats don't match existing ones"},"return_code":0} 2024-05-27T07:35:38.696 INFO:journalctl@ceph.mds.b.smithi105.stdout:May 27 07:35:38 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-b[65543]: 2024-05-27T07:35:38.354+0000 7f33fced0700 -1 mds.0.scrubstack _validate_inode_done scrub error on inode [inode 0x100000089f8 [...a,head] /volumes/qa/sv_1/95e9ac73-825f-48e3-998b-133d94d96559/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-134/ auth{4=1} fragtree_t(*^3) v239496 ap=2 RANDEPHEMERALPIN f(v6 m2024-05-27T07:32:47.574954+0000 192=192+0) n(v10 rc2024-05-27T07:32:47.574954+0000 b3668857 193=192+1) old_inodes=2 (inest sync->mix g=4) caps={25060=pAsLsXsFs/pAsLsXsFsx@164} mcw={4=AsLsXsFs} | dirtyscattered=0 request=0 lock=0 dirfrag=8 caps=1 dirtyrstat=0 dirtyparent=0 replicated=1 dirty=0 waiter=0 authpin=1 scrubqueue=0 randepin 0x55ee57608b00]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":true,"read_ret_val":0,"ondisk_value":"(2)0x100000089f8:[<0x10000000399/blog-134 v233203>,<0x10000000208/blogtest_in v1521>,<0x10000000201/src v2419>,<0x10000000006/blogbench-1.0 v2148>,<0x10000000005/tmp v436>,<0x100000001fb/client.0 v403>,<0x100000001fa/95e9ac73-825f-48e3-998b-133d94d96559 v410>,<0x10000000001/sv_1 v431>,<0x10000000000/qa v407>,<0x1/volumes v413>]//[]","memoryvalue":"(2)0x100000089f8:[<0x10000000399/blog-134 v239496>,<0x10000000208/blogtest_in v2501>,<0x10000000201/src v2932>,<0x10000000006/blogbench-1.0 v2530>,<0x10000000005/tmp v807>,<0x100000001fb/client.0 v770>,<0x100000001fa/95e9ac73-825f-48e3-998b-133d94d96559 v772>,<0x10000000001/sv_1 v783>,<0x10000000000/qa v754>,<0x1/volumes v761>]//[]","error_str":""},"raw_stats":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f(v0 m2024-05-27T07:32:43.410002+0000 191=191+0)","ondisk_value.rstat":"n(v0 rc2024-05-27T07:32:47.574954+0000 b3668857 193=192+1)","memory_value.dirstat":"f(v6 m2024-05-27T07:32:47.574954+0000 192=192+0)","memory_value.rstat":"n(v10 rc2024-05-27T07:32:47.574954+0000 b3668857 193=192+1)","error_str":"freshly-calculated rstats don't match existing ones"},"return_code":0} 2024-05-27T07:35:38.696 INFO:journalctl@ceph.mds.b.smithi105.stdout:May 27 07:35:38 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-b[65543]: 2024-05-27T07:35:38.356+0000 7f33fced0700 -1 mds.0.scrubstack _validate_inode_done scrub error on inode [inode 0x1000000918d [...a,head] /volumes/qa/sv_1/95e9ac73-825f-48e3-998b-133d94d96559/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-139/ auth{1=1} fragtree_t(*^3) v239498 ap=2 RANDEPHEMERALPIN f(v4 m2024-05-27T07:32:50.502921+0000 107=107+0) n(v8 rc2024-05-27T07:32:50.502921+0000 b3643698 108=107+1) old_inodes=2 (inest sync->mix g=1) caps={25060=pAsLsXsFs/pAsLsXsFsx@167} | dirtyscattered=0 request=0 lock=0 dirfrag=8 caps=1 dirtyrstat=0 dirtyparent=0 replicated=1 dirty=0 waiter=0 authpin=1 scrubqueue=0 randepin 0x55ee5187b700]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":true,"read_ret_val":0,"ondisk_value":"(2)0x1000000918d:[<0x10000000399/blog-139 v234197>,<0x10000000208/blogtest_in v1521>,<0x10000000201/src v2419>,<0x10000000006/blogbench-1.0 v2148>,<0x10000000005/tmp v436>,<0x100000001fb/client.0 v403>,<0x100000001fa/95e9ac73-825f-48e3-998b-133d94d96559 v410>,<0x10000000001/sv_1 v431>,<0x10000000000/qa v407>,<0x1/volumes v413>]//[]","memoryvalue":"(2)0x1000000918d:[<0x10000000399/blog-139 v239498>,<0x10000000208/blogtest_in v2501>,<0x10000000201/src v2932>,<0x10000000006/blogbench-1.0 v2530>,<0x10000000005/tmp v807>,<0x100000001fb/client.0 v770>,<0x100000001fa/95e9ac73-825f-48e3-998b-133d94d96559 v772>,<0x10000000001/sv_1 v783>,<0x10000000000/qa v754>,<0x1/volumes v761>]//[]","error_str":""},"raw_stats":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f(v0 m2024-05-27T07:32:27.669183+0000 106=106+0)","ondisk_value.rstat":"n(v0 rc2024-05-27T07:32:50.502921+0000 b3643698 108=107+1)","memory_value.dirstat":"f(v4 m2024-05-27T07:32:50.502921+0000 107=107+0)","memory_value.rstat":"n(v8 rc2024-05-27T07:32:50.502921+0000 b3643698 108=107+1)","error_str":"freshly-calculated rstats don't match existing ones"},"return_code":0} 2024-05-27T07:35:38.981 DEBUG:teuthology.orchestra.run.smithi052:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:38f9d900c83c63799fbdbe61acc9a11b0d3554a6 shell --fsid 73df2636-1bf8-11ef-bc9b-c7b262605968 -- ceph tell mds.1:0 scrub status 2024-05-27T07:35:39.493 INFO:teuthology.orchestra.run.smithi052.stderr:Inferring config /var/lib/ceph/73df2636-1bf8-11ef-bc9b-c7b262605968/mon.a/config 2024-05-27T07:35:39.607 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:35:39 smithi173 ceph-mon[37000]: scrub queued for path: / 2024-05-27T07:35:39.607 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:35:39 smithi173 ceph-mon[37000]: scrub summary: idle+waiting paths [/] 2024-05-27T07:35:39.607 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:35:39 smithi173 ceph-mon[37000]: scrub summary: active paths [/] 2024-05-27T07:35:39.607 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:35:39 smithi173 ceph-mon[37000]: scrub summary: active paths [/] 2024-05-27T07:35:39.608 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:35:39 smithi173 ceph-mon[37000]: scrub summary: active paths [/] 2024-05-27T07:35:39.608 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:35:39 smithi173 ceph-mon[37000]: Scrub error on inode 0x10000003fad (/volumes/qa/sv_1/95e9ac73-825f-48e3-998b-133d94d96559/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-58) see mds.b log and `damage ls` output for details 2024-05-27T07:35:39.608 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:35:39 smithi173 ceph-mon[37000]: Scrub error on inode 0x100000089f8 (/volumes/qa/sv_1/95e9ac73-825f-48e3-998b-133d94d96559/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-134) see mds.b log and `damage ls` output for details 2024-05-27T07:35:39.608 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:35:39 smithi173 ceph-mon[37000]: Scrub error on inode 0x1000000918d (/volumes/qa/sv_1/95e9ac73-825f-48e3-998b-133d94d96559/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-139) see mds.b log and `damage ls` output for details 2024-05-27T07:35:39.608 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:35:39 smithi173 ceph-mon[37000]: scrub summary: idle+waiting paths [/] 2024-05-27T07:35:39.608 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:35:39 smithi173 ceph-mon[37000]: scrub summary: idle+waiting paths [/] 2024-05-27T07:35:39.644 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:35:39 smithi052 ceph-mon[31955]: scrub queued for path: / 2024-05-27T07:35:39.645 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:35:39 smithi052 ceph-mon[31955]: scrub summary: idle+waiting paths [/] 2024-05-27T07:35:39.645 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:35:39 smithi052 ceph-mon[31955]: scrub summary: active paths [/] 2024-05-27T07:35:39.645 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:35:39 smithi052 ceph-mon[31955]: scrub summary: active paths [/] 2024-05-27T07:35:39.645 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:35:39 smithi052 ceph-mon[31955]: scrub summary: active paths [/] 2024-05-27T07:35:39.645 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:35:39 smithi052 ceph-mon[31955]: Scrub error on inode 0x10000003fad (/volumes/qa/sv_1/95e9ac73-825f-48e3-998b-133d94d96559/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-58) see mds.b log and `damage ls` output for details 2024-05-27T07:35:39.645 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:35:39 smithi052 ceph-mon[31955]: Scrub error on inode 0x100000089f8 (/volumes/qa/sv_1/95e9ac73-825f-48e3-998b-133d94d96559/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-134) see mds.b log and `damage ls` output for details 2024-05-27T07:35:39.645 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:35:39 smithi052 ceph-mon[31955]: Scrub error on inode 0x1000000918d (/volumes/qa/sv_1/95e9ac73-825f-48e3-998b-133d94d96559/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-139) see mds.b log and `damage ls` output for details 2024-05-27T07:35:39.645 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:35:39 smithi052 ceph-mon[31955]: scrub summary: idle+waiting paths [/] 2024-05-27T07:35:39.645 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:35:39 smithi052 ceph-mon[31955]: scrub summary: idle+waiting paths [/] 2024-05-27T07:35:39.695 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:35:39 smithi105 ceph-mon[35977]: scrub queued for path: / 2024-05-27T07:35:39.695 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:35:39 smithi105 ceph-mon[35977]: scrub summary: idle+waiting paths [/] 2024-05-27T07:35:39.696 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:35:39 smithi105 ceph-mon[35977]: scrub summary: active paths [/] 2024-05-27T07:35:39.696 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:35:39 smithi105 ceph-mon[35977]: scrub summary: active paths [/] 2024-05-27T07:35:39.696 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:35:39 smithi105 ceph-mon[35977]: scrub summary: active paths [/] 2024-05-27T07:35:39.696 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:35:39 smithi105 ceph-mon[35977]: Scrub error on inode 0x10000003fad (/volumes/qa/sv_1/95e9ac73-825f-48e3-998b-133d94d96559/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-58) see mds.b log and `damage ls` output for details 2024-05-27T07:35:39.696 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:35:39 smithi105 ceph-mon[35977]: Scrub error on inode 0x100000089f8 (/volumes/qa/sv_1/95e9ac73-825f-48e3-998b-133d94d96559/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-134) see mds.b log and `damage ls` output for details 2024-05-27T07:35:39.696 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:35:39 smithi105 ceph-mon[35977]: Scrub error on inode 0x1000000918d (/volumes/qa/sv_1/95e9ac73-825f-48e3-998b-133d94d96559/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-139) see mds.b log and `damage ls` output for details 2024-05-27T07:35:39.696 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:35:39 smithi105 ceph-mon[35977]: scrub summary: idle+waiting paths [/] 2024-05-27T07:35:39.696 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:35:39 smithi105 ceph-mon[35977]: scrub summary: idle+waiting paths [/] 2024-05-27T07:35:40.403 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:35:40.406+0000 7fc66299d640 1 -- 172.21.15.52:0/299822698 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fc65c10fa00 msgr2=0x7fc65c10fe10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:35:40.403 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:35:40.406+0000 7fc66299d640 1 --2- 172.21.15.52:0/299822698 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fc65c10fa00 0x7fc65c10fe10 secure :-1 s=READY pgs=751 cs=0 l=1 rev1=1 crypto rx=0x7fc654009be0 tx=0x7fc65402a710 comp rx=0 tx=0).stop 2024-05-27T07:35:40.403 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:35:40.406+0000 7fc66299d640 1 -- 172.21.15.52:0/299822698 shutdown_connections 2024-05-27T07:35:40.403 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:35:40.406+0000 7fc66299d640 1 --2- 172.21.15.52:0/299822698 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fc65c10a200 0x7fc65c10a690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:35:40.404 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:35:40.406+0000 7fc66299d640 1 --2- 172.21.15.52:0/299822698 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fc65c1103e0 0x7fc65c109cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:35:40.404 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:35:40.406+0000 7fc66299d640 1 --2- 172.21.15.52:0/299822698 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fc65c10fa00 0x7fc65c10fe10 secure :-1 s=CLOSED pgs=751 cs=0 l=1 rev1=1 crypto rx=0x7fc654009be0 tx=0x7fc65402a710 comp rx=0 tx=0).stop 2024-05-27T07:35:40.404 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:35:40.406+0000 7fc66299d640 1 -- 172.21.15.52:0/299822698 >> 172.21.15.52:0/299822698 conn(0x7fc65c06f800 msgr2=0x7fc65c06fc10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:35:40.404 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:35:40.408+0000 7fc66299d640 1 -- 172.21.15.52:0/299822698 shutdown_connections 2024-05-27T07:35:40.405 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:35:40.408+0000 7fc66299d640 1 -- 172.21.15.52:0/299822698 wait complete. 2024-05-27T07:35:40.405 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:35:40.409+0000 7fc66299d640 1 Processor -- start 2024-05-27T07:35:40.410 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:35:40.413+0000 7fc66299d640 1 -- start start 2024-05-27T07:35:40.410 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:35:40.414+0000 7fc66299d640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fc65c10a200 0x7fc65c1a9720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:35:40.410 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:35:40.414+0000 7fc66299d640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fc65c1a9c60 0x7fc65c1ade80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:35:40.411 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:35:40.414+0000 7fc66299d640 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fc65c1ae3c0 0x7fc65c1ae7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:35:40.411 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:35:40.414+0000 7fc66299d640 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x7fc65c122aa0 con 0x7fc65c1ae3c0 2024-05-27T07:35:40.411 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:35:40.414+0000 7fc66299d640 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7fc65c122920 con 0x7fc65c1a9c60 2024-05-27T07:35:40.411 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:35:40.414+0000 7fc66299d640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7fc65c122c20 con 0x7fc65c10a200 2024-05-27T07:35:40.411 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:35:40.415+0000 7fc66199b640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fc65c10a200 0x7fc65c1a9720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:35:40.411 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:35:40.415+0000 7fc66199b640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fc65c10a200 0x7fc65c1a9720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.173:3300/0 says I am v2:172.21.15.52:51564/0 (socket says 172.21.15.52:51564) 2024-05-27T07:35:40.412 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:35:40.415+0000 7fc66199b640 1 -- 172.21.15.52:0/2134933493 learned_addr learned my addr 172.21.15.52:0/2134933493 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-05-27T07:35:40.413 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:35:40.416+0000 7fc66119a640 1 --2- 172.21.15.52:0/2134933493 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fc65c1a9c60 0x7fc65c1ade80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:35:40.413 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:35:40.416+0000 7fc66219c640 1 --2- 172.21.15.52:0/2134933493 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fc65c1ae3c0 0x7fc65c1ae7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:35:40.416 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:35:40.419+0000 7fc66219c640 1 -- 172.21.15.52:0/2134933493 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fc65c10a200 msgr2=0x7fc65c1a9720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:35:40.416 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:35:40.419+0000 7fc66219c640 1 --2- 172.21.15.52:0/2134933493 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fc65c10a200 0x7fc65c1a9720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:35:40.416 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:35:40.419+0000 7fc66219c640 1 -- 172.21.15.52:0/2134933493 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fc65c1a9c60 msgr2=0x7fc65c1ade80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:35:40.416 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:35:40.419+0000 7fc66219c640 1 --2- 172.21.15.52:0/2134933493 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fc65c1a9c60 0x7fc65c1ade80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:35:40.416 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:35:40.419+0000 7fc66219c640 1 -- 172.21.15.52:0/2134933493 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc65c1120d0 con 0x7fc65c1ae3c0 2024-05-27T07:35:40.417 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:35:40.420+0000 7fc66219c640 1 --2- 172.21.15.52:0/2134933493 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fc65c1ae3c0 0x7fc65c1ae7d0 secure :-1 s=READY pgs=752 cs=0 l=1 rev1=1 crypto rx=0x7fc64c00d750 tx=0x7fc64c00dc10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:35:40.418 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:35:40.421+0000 7fc652ffd640 1 -- 172.21.15.52:0/2134933493 <== mon.0 v2:172.21.15.52:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc64c017070 con 0x7fc65c1ae3c0 2024-05-27T07:35:40.418 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:35:40.422+0000 7fc652ffd640 1 -- 172.21.15.52:0/2134933493 <== mon.0 v2:172.21.15.52:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fc64c004020 con 0x7fc65c1ae3c0 2024-05-27T07:35:40.418 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:35:40.422+0000 7fc652ffd640 1 -- 172.21.15.52:0/2134933493 <== mon.0 v2:172.21.15.52:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc64c003cc0 con 0x7fc65c1ae3c0 2024-05-27T07:35:40.422 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:35:40.422+0000 7fc66299d640 1 -- 172.21.15.52:0/2134933493 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc65c1123c0 con 0x7fc65c1ae3c0 2024-05-27T07:35:40.422 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:35:40.422+0000 7fc66299d640 1 -- 172.21.15.52:0/2134933493 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc65c1128c0 con 0x7fc65c1ae3c0 2024-05-27T07:35:40.423 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:35:40.427+0000 7fc66ab8e740 1 Processor -- start 2024-05-27T07:35:40.428 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:35:40.432+0000 7fc66ab8e740 1 -- start start 2024-05-27T07:35:40.429 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:35:40.432+0000 7fc66ab8e740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x556ddf3a91b0 0x556ddf74a590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:35:40.429 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:35:40.432+0000 7fc66ab8e740 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x556ddf3802f0 0x556ddf74aad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:35:40.429 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:35:40.432+0000 7fc66ab8e740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x556ddf3884d0 0x556ddf74b010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:35:40.429 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:35:40.432+0000 7fc66ab8e740 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x556ddf6bcc30 con 0x556ddf3802f0 2024-05-27T07:35:40.429 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:35:40.432+0000 7fc66ab8e740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x556ddf6ac7d0 con 0x556ddf3884d0 2024-05-27T07:35:40.429 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:35:40.432+0000 7fc66ab8e740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x556ddf6b0810 con 0x556ddf3a91b0 2024-05-27T07:35:40.430 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:35:40.432+0000 7fc66199b640 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x556ddf3802f0 0x556ddf74aad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:35:40.430 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:35:40.432+0000 7fc66199b640 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x556ddf3802f0 0x556ddf74aad0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.52:53788/0 (socket says 172.21.15.52:53788) 2024-05-27T07:35:40.430 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:35:40.432+0000 7fc66199b640 1 -- 172.21.15.52:0/151568821 learned_addr learned my addr 172.21.15.52:0/151568821 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-05-27T07:35:40.430 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:35:40.433+0000 7fc66119a640 1 --2- 172.21.15.52:0/151568821 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x556ddf3a91b0 0x556ddf74a590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:35:40.430 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:35:40.433+0000 7fc66119a640 1 --2- 172.21.15.52:0/151568821 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x556ddf3884d0 0x556ddf74b010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:35:40.431 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:35:40.434+0000 7fc66119a640 1 -- 172.21.15.52:0/151568821 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x556ddf3884d0 msgr2=0x556ddf74b010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:35:40.431 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:35:40.434+0000 7fc66119a640 1 --2- 172.21.15.52:0/151568821 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x556ddf3884d0 0x556ddf74b010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:35:40.431 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:35:40.434+0000 7fc66119a640 1 -- 172.21.15.52:0/151568821 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x556ddf3802f0 msgr2=0x556ddf74aad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:35:40.431 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:35:40.434+0000 7fc66119a640 1 --2- 172.21.15.52:0/151568821 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x556ddf3802f0 0x556ddf74aad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:35:40.431 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:35:40.434+0000 7fc66119a640 1 -- 172.21.15.52:0/151568821 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x556ddf3c1210 con 0x556ddf3a91b0 2024-05-27T07:35:40.431 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:35:40.435+0000 7fc66119a640 1 --2- 172.21.15.52:0/151568821 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x556ddf3a91b0 0x556ddf74a590 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7fc65c10c7a0 tx=0x7fc6580092b0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=89ff8d24291a9180 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:35:40.437 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:35:40.438+0000 7fc652ffd640 1 -- 172.21.15.52:0/2134933493 <== mon.0 v2:172.21.15.52:3300/0 4 ==== mgrmap(e 26) ==== 149783+0+0 (secure 0 0 0) 0x7fc64c0041d0 con 0x7fc65c1ae3c0 2024-05-27T07:35:40.437 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:35:40.439+0000 7fc647fff640 1 -- 172.21.15.52:0/151568821 <== mon.1 v2:172.21.15.173:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc6580183f0 con 0x556ddf3a91b0 2024-05-27T07:35:40.437 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:35:40.439+0000 7fc647fff640 1 -- 172.21.15.52:0/151568821 <== mon.1 v2:172.21.15.173:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fc65801d070 con 0x556ddf3a91b0 2024-05-27T07:35:40.437 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:35:40.439+0000 7fc647fff640 1 -- 172.21.15.52:0/151568821 <== mon.1 v2:172.21.15.173:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc6580156e0 con 0x556ddf3a91b0 2024-05-27T07:35:40.437 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:35:40.440+0000 7fc652ffd640 1 --2- 172.21.15.52:0/2134933493 >> [v2:172.21.15.52:6800/1822318384,v1:172.21.15.52:6801/1822318384] conn(0x7fc6480b2b20 0x7fc6480b5030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:35:40.437 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:35:40.440+0000 7fc652ffd640 1 -- 172.21.15.52:0/2134933493 <== mon.0 v2:172.21.15.52:3300/0 5 ==== osd_map(123..123 src has 1..123) ==== 9555+0+0 (secure 0 0 0) 0x7fc64c0e20a0 con 0x7fc65c1ae3c0 2024-05-27T07:35:40.440 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:35:40.443+0000 7fc66199b640 1 --2- 172.21.15.52:0/2134933493 >> [v2:172.21.15.52:6800/1822318384,v1:172.21.15.52:6801/1822318384] conn(0x7fc6480b2b20 0x7fc6480b5030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:35:40.441 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:35:40.445+0000 7fc66ab8e740 1 -- 172.21.15.52:0/151568821 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x556ddf3a91b0 msgr2=0x556ddf74a590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:35:40.441 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:35:40.445+0000 7fc66ab8e740 1 --2- 172.21.15.52:0/151568821 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x556ddf3a91b0 0x556ddf74a590 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7fc65c10c7a0 tx=0x7fc6580092b0 comp rx=0 tx=0).stop 2024-05-27T07:35:40.443 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:35:40.446+0000 7fc66ab8e740 1 -- 172.21.15.52:0/151568821 shutdown_connections 2024-05-27T07:35:40.443 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:35:40.446+0000 7fc66ab8e740 1 --2- 172.21.15.52:0/151568821 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x556ddf3884d0 0x556ddf74b010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:35:40.443 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:35:40.447+0000 7fc66ab8e740 1 --2- 172.21.15.52:0/151568821 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x556ddf3802f0 0x556ddf74aad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:35:40.443 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:35:40.447+0000 7fc66ab8e740 1 --2- 172.21.15.52:0/151568821 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x556ddf3a91b0 0x556ddf74a590 unknown :-1 s=CLOSED pgs=214 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:35:40.443 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:35:40.447+0000 7fc66ab8e740 1 -- 172.21.15.52:0/151568821 >> 172.21.15.52:0/151568821 conn(0x556ddf39d300 msgr2=0x556ddf746d40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:35:40.445 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:35:40.449+0000 7fc66199b640 1 --2- 172.21.15.52:0/2134933493 >> [v2:172.21.15.52:6800/1822318384,v1:172.21.15.52:6801/1822318384] conn(0x7fc6480b2b20 0x7fc6480b5030 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7fc654009420 tx=0x7fc654030b70 comp rx=0 tx=0).ready entity=mgr.15333 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:35:40.445 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:35:40.449+0000 7fc66ab8e740 1 -- 172.21.15.52:0/151568821 shutdown_connections 2024-05-27T07:35:40.446 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:35:40.449+0000 7fc66ab8e740 1 -- 172.21.15.52:0/151568821 wait complete. 2024-05-27T07:35:40.446 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:35:40.449+0000 7fc66ab8e740 1 -- start start 2024-05-27T07:35:40.446 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:35:40.450+0000 7fc66ab8e740 1 Processor -- start 2024-05-27T07:35:40.447 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:35:40.451+0000 7fc66ab8e740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x556ddf3802f0 0x556ddf7566e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:35:40.447 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:35:40.451+0000 7fc66ab8e740 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x556ddf3884d0 0x556ddf756c20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:35:40.447 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:35:40.451+0000 7fc66ab8e740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x556ddf3a91b0 0x556ddf754b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:35:40.447 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:35:40.451+0000 7fc66ab8e740 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x556ddf392790 con 0x556ddf3884d0 2024-05-27T07:35:40.447 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:35:40.451+0000 7fc66ab8e740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x556ddf668890 con 0x556ddf3a91b0 2024-05-27T07:35:40.447 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:35:40.451+0000 7fc66ab8e740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x556ddf757660 con 0x556ddf3802f0 2024-05-27T07:35:40.447 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:35:40.451+0000 7fc66ab8e740 10 client.0 _ll_register_callbacks cb 0x556ddf3c0a00 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T07:35:40.448 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:35:40.451+0000 7fc66219c640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x556ddf3a91b0 0x556ddf754b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:35:40.448 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:35:40.451+0000 7fc66219c640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x556ddf3a91b0 0x556ddf754b80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.52:56714/0 (socket says 172.21.15.52:56714) 2024-05-27T07:35:40.448 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:35:40.451+0000 7fc66219c640 1 -- 172.21.15.52:0/2959561877 learned_addr learned my addr 172.21.15.52:0/2959561877 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-05-27T07:35:40.448 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:35:40.451+0000 7fc66119a640 1 --2- 172.21.15.52:0/2959561877 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x556ddf3884d0 0x556ddf756c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:35:40.448 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:35:40.451+0000 7fc66199b640 1 --2- 172.21.15.52:0/2959561877 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x556ddf3802f0 0x556ddf7566e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:35:40.449 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:35:40.452+0000 7fc66119a640 1 -- 172.21.15.52:0/2959561877 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x556ddf3802f0 msgr2=0x556ddf7566e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:35:40.449 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:35:40.452+0000 7fc66119a640 1 --2- 172.21.15.52:0/2959561877 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x556ddf3802f0 0x556ddf7566e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:35:40.449 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:35:40.452+0000 7fc66119a640 1 -- 172.21.15.52:0/2959561877 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x556ddf3a91b0 msgr2=0x556ddf754b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:35:40.449 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:35:40.452+0000 7fc66119a640 1 --2- 172.21.15.52:0/2959561877 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x556ddf3a91b0 0x556ddf754b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:35:40.449 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:35:40.452+0000 7fc66119a640 1 -- 172.21.15.52:0/2959561877 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({osdmap=0}) -- 0x556ddf48d620 con 0x556ddf3884d0 2024-05-27T07:35:40.449 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:35:40.453+0000 7fc66119a640 1 --2- 172.21.15.52:0/2959561877 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x556ddf3884d0 0x556ddf756c20 secure :-1 s=READY pgs=753 cs=0 l=1 rev1=1 crypto rx=0x7fc658006a40 tx=0x7fc658018620 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6cfe349316642389 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:35:40.450 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:35:40.453+0000 7fc6457fa640 10 client.0 ms_handle_connect on v2:172.21.15.52:3300/0 2024-05-27T07:35:40.450 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:35:40.453+0000 7fc6457fa640 1 -- 172.21.15.52:0/2959561877 <== mon.0 v2:172.21.15.52:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc658025020 con 0x556ddf3884d0 2024-05-27T07:35:40.450 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:35:40.453+0000 7fc6457fa640 1 -- 172.21.15.52:0/2959561877 <== mon.0 v2:172.21.15.52:3300/0 2 ==== osd_map(123..123 src has 1..123) ==== 9555+0+0 (secure 0 0 0) 0x7fc658004440 con 0x556ddf3884d0 2024-05-27T07:35:40.451 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:35:40.455+0000 7fc66ab8e740 1 -- 172.21.15.52:0/2959561877 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fc65801d070 con 0x556ddf3884d0 2024-05-27T07:35:40.453 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:35:40.456+0000 7fc6457fa640 1 -- 172.21.15.52:0/2959561877 <== mon.0 v2:172.21.15.52:3300/0 3 ==== mon_get_version_reply(handle=1 version=108) ==== 24+0+0 (secure 0 0 0) 0x7fc658004640 con 0x556ddf3884d0 2024-05-27T07:35:40.453 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:35:40.457+0000 7fc66ab8e740 10 client.17181 fetch_fsmap learned FSMap version 108 2024-05-27T07:35:40.453 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:35:40.457+0000 7fc66ab8e740 1 -- 172.21.15.52:0/2959561877 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({config=0+,fsmap=108,monmap=4+}) -- 0x556ddf763ac0 con 0x556ddf3884d0 2024-05-27T07:35:40.454 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:35:40.457+0000 7fc6457fa640 1 -- 172.21.15.52:0/2959561877 <== mon.0 v2:172.21.15.52:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fc65801d070 con 0x556ddf3884d0 2024-05-27T07:35:40.454 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:35:40.457+0000 7fc6457fa640 1 -- 172.21.15.52:0/2959561877 <== mon.0 v2:172.21.15.52:3300/0 5 ==== fsmap(e 108) ==== 4203+0+0 (secure 0 0 0) 0x7fc658015040 con 0x556ddf3884d0 2024-05-27T07:35:40.454 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:35:40.457+0000 7fc66ab8e740 10 client.17181 fetch_fsmap finished waiting for FSMap version 108 2024-05-27T07:35:40.454 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:35:40.457+0000 7fc66ab8e740 10 client.17181 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-05-27T07:35:40.454 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:35:40.457+0000 7fc66ab8e740 20 client.17181 populate_metadata read hostname 'smithi052' 2024-05-27T07:35:40.454 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:35:40.458+0000 7fc66ab8e740 1 --2- 172.21.15.52:0/2959561877 >> [v2:172.21.15.105:6838/500348151,v1:172.21.15.105:6839/500348151] conn(0x556ddf367d80 0x556ddf7672b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:35:40.454 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:35:40.458+0000 7fc66ab8e740 4 client.17181 mds_command: new command op to 24527 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-27T07:35:40.454 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:35:40.458+0000 7fc66ab8e740 1 -- 172.21.15.52:0/2959561877 --> [v2:172.21.15.105:6838/500348151,v1:172.21.15.105:6839/500348151] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x556ddf6beb80 con 0x556ddf367d80 2024-05-27T07:35:40.455 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:35:40.459+0000 7fc66199b640 1 --2- 172.21.15.52:0/2959561877 >> [v2:172.21.15.105:6838/500348151,v1:172.21.15.105:6839/500348151] conn(0x556ddf367d80 0x556ddf7672b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:35:40.458 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:35:40.462+0000 7fc6457fa640 10 client.17181 ms_handle_connect on v2:172.21.15.105:6838/500348151 2024-05-27T07:35:40.459 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:35:40.462+0000 7fc66199b640 1 --2- 172.21.15.52:0/2959561877 >> [v2:172.21.15.105:6838/500348151,v1:172.21.15.105:6839/500348151] conn(0x556ddf367d80 0x556ddf7672b0 crc :-1 s=READY pgs=171 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=26aaa206fe3d15fc server_cookie=209984207afd35fc in_seq=0 out_seq=0 2024-05-27T07:35:40.460 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:35:40.464+0000 7fc6457fa640 1 -- 172.21.15.52:0/2959561877 <== mds.0 v2:172.21.15.105:6838/500348151 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24041 (crc 0 0 0) 0x556ddf6beb80 con 0x556ddf367d80 2024-05-27T07:35:40.460 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:35:40.464+0000 7fc6457fa640 10 client.17181 handle_command_reply: tid=0 2024-05-27T07:35:40.460 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:35:40.464+0000 7fc66ab8e740 1 client.17181 shutdown 2024-05-27T07:35:40.465 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:35:40.469+0000 7fc66ab8e740 1 -- 172.21.15.52:0/2959561877 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x556ddf3884d0 msgr2=0x556ddf756c20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:35:40.465 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:35:40.469+0000 7fc66ab8e740 1 --2- 172.21.15.52:0/2959561877 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x556ddf3884d0 0x556ddf756c20 secure :-1 s=READY pgs=753 cs=0 l=1 rev1=1 crypto rx=0x7fc658006a40 tx=0x7fc658018620 comp rx=0 tx=0).stop 2024-05-27T07:35:40.470 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:35:40.471+0000 7fc66ab8e740 1 -- 172.21.15.52:0/2959561877 shutdown_connections 2024-05-27T07:35:40.470 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:35:40.471+0000 7fc66ab8e740 1 --2- 172.21.15.52:0/2959561877 >> [v2:172.21.15.105:6838/500348151,v1:172.21.15.105:6839/500348151] conn(0x556ddf367d80 0x556ddf7672b0 crc :-1 s=READY pgs=171 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:35:40.471 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:35:40.471+0000 7fc66ab8e740 1 --2- 172.21.15.52:0/2959561877 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x556ddf3a91b0 0x556ddf754b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:35:40.471 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:35:40.471+0000 7fc66ab8e740 1 --2- 172.21.15.52:0/2959561877 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x556ddf3884d0 0x556ddf756c20 unknown :-1 s=CLOSED pgs=753 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:35:40.471 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:35:40.471+0000 7fc66ab8e740 1 --2- 172.21.15.52:0/2959561877 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x556ddf3802f0 0x556ddf7566e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:35:40.471 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:35:40.471+0000 7fc66ab8e740 1 -- 172.21.15.52:0/2959561877 >> 172.21.15.52:0/2959561877 conn(0x556ddf39d300 msgr2=0x556ddf74a360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:35:40.475 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:35:40.475+0000 7fc6457fa640 0 client.17181 ms_handle_reset on v2:172.21.15.105:6838/500348151 2024-05-27T07:35:40.477 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:35:40.479+0000 7fc66ab8e740 1 -- 172.21.15.52:0/2959561877 shutdown_connections 2024-05-27T07:35:40.477 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:35:40.481+0000 7fc66ab8e740 1 -- 172.21.15.52:0/2959561877 wait complete. 2024-05-27T07:35:40.480 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:35:40.484+0000 7fc66ab8e740 20 client.17181 trim_cache size 0 max 16384 2024-05-27T07:35:40.513 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:35:40.516+0000 7fc66ab8e740 1 Processor -- start 2024-05-27T07:35:40.513 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:35:40.517+0000 7fc66ab8e740 1 -- start start 2024-05-27T07:35:40.514 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:35:40.517+0000 7fc66ab8e740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x556ddf3802f0 0x556ddf746ee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:35:40.514 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:35:40.517+0000 7fc66ab8e740 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x556ddf3884d0 0x556ddf750360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:35:40.514 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:35:40.517+0000 7fc66ab8e740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x556ddf3a91b0 0x556ddf74e1e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:35:40.514 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:35:40.517+0000 7fc66ab8e740 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x7fc640010620 con 0x556ddf3884d0 2024-05-27T07:35:40.514 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:35:40.517+0000 7fc66ab8e740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7fc64000ea50 con 0x556ddf3a91b0 2024-05-27T07:35:40.514 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:35:40.517+0000 7fc66ab8e740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7fc64000ff20 con 0x556ddf3802f0 2024-05-27T07:35:40.515 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:35:40.517+0000 7fc66219c640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x556ddf3a91b0 0x556ddf74e1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:35:40.515 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:35:40.517+0000 7fc66219c640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x556ddf3a91b0 0x556ddf74e1e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.52:56728/0 (socket says 172.21.15.52:56728) 2024-05-27T07:35:40.515 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:35:40.517+0000 7fc66219c640 1 -- 172.21.15.52:0/2920376622 learned_addr learned my addr 172.21.15.52:0/2920376622 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-05-27T07:35:40.515 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:35:40.517+0000 7fc66199b640 1 --2- 172.21.15.52:0/2920376622 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x556ddf3802f0 0x556ddf746ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:35:40.515 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:35:40.518+0000 7fc66219c640 1 -- 172.21.15.52:0/2920376622 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x556ddf3802f0 msgr2=0x556ddf746ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:35:40.515 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:35:40.518+0000 7fc66219c640 1 --2- 172.21.15.52:0/2920376622 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x556ddf3802f0 0x556ddf746ee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:35:40.515 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:35:40.518+0000 7fc66219c640 1 -- 172.21.15.52:0/2920376622 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x556ddf3884d0 msgr2=0x556ddf750360 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-05-27T07:35:40.515 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:35:40.518+0000 7fc66219c640 1 --2- 172.21.15.52:0/2920376622 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x556ddf3884d0 0x556ddf750360 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:35:40.515 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:35:40.518+0000 7fc66219c640 1 -- 172.21.15.52:0/2920376622 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x556ddf74b970 con 0x556ddf3a91b0 2024-05-27T07:35:40.516 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:35:40.520+0000 7fc66219c640 1 --2- 172.21.15.52:0/2920376622 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x556ddf3a91b0 0x556ddf74e1e0 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7fc64c0a8a80 tx=0x7fc64c0a8fc0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=2251092be31e8f92 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:35:40.518 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:35:40.522+0000 7fc6477fe640 1 -- 172.21.15.52:0/2920376622 <== mon.2 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc64c020040 con 0x556ddf3a91b0 2024-05-27T07:35:40.518 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:35:40.522+0000 7fc6477fe640 1 -- 172.21.15.52:0/2920376622 <== mon.2 v2:172.21.15.105:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fc64c0a5250 con 0x556ddf3a91b0 2024-05-27T07:35:40.519 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:35:40.523+0000 7fc66ab8e740 1 -- 172.21.15.52:0/2920376622 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x556ddf3a91b0 msgr2=0x556ddf74e1e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:35:40.520 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:35:40.523+0000 7fc66ab8e740 1 --2- 172.21.15.52:0/2920376622 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x556ddf3a91b0 0x556ddf74e1e0 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7fc64c0a8a80 tx=0x7fc64c0a8fc0 comp rx=0 tx=0).stop 2024-05-27T07:35:40.520 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:35:40.523+0000 7fc66ab8e740 1 -- 172.21.15.52:0/2920376622 shutdown_connections 2024-05-27T07:35:40.520 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:35:40.523+0000 7fc66ab8e740 1 --2- 172.21.15.52:0/2920376622 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x556ddf3a91b0 0x556ddf74e1e0 unknown :-1 s=CLOSED pgs=250 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:35:40.520 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:35:40.523+0000 7fc66ab8e740 1 --2- 172.21.15.52:0/2920376622 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x556ddf3884d0 0x556ddf750360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:35:40.520 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:35:40.523+0000 7fc66ab8e740 1 --2- 172.21.15.52:0/2920376622 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x556ddf3802f0 0x556ddf746ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:35:40.520 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:35:40.523+0000 7fc66ab8e740 1 -- 172.21.15.52:0/2920376622 >> 172.21.15.52:0/2920376622 conn(0x556ddf39d300 msgr2=0x556ddf7467d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:35:40.528 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:35:40.532+0000 7fc66ab8e740 1 -- 172.21.15.52:0/2920376622 shutdown_connections 2024-05-27T07:35:40.531 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:35:40.534+0000 7fc66ab8e740 1 -- 172.21.15.52:0/2920376622 wait complete. 2024-05-27T07:35:40.531 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:35:40.535+0000 7fc66ab8e740 1 -- start start 2024-05-27T07:35:40.531 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:35:40.535+0000 7fc66ab8e740 1 Processor -- start 2024-05-27T07:35:40.533 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:35:40.536+0000 7fc66ab8e740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x556ddf3802f0 0x556ddf75d200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:35:40.533 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:35:40.536+0000 7fc66ab8e740 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x556ddf3884d0 0x556ddf759560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:35:40.533 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:35:40.536+0000 7fc66ab8e740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x556ddf3a91b0 0x556ddf759c00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:35:40.533 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:35:40.536+0000 7fc66ab8e740 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x7fc64000f850 con 0x556ddf3884d0 2024-05-27T07:35:40.533 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:35:40.536+0000 7fc66ab8e740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7fc64000f150 con 0x556ddf3a91b0 2024-05-27T07:35:40.534 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:35:40.536+0000 7fc66ab8e740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x556ddf6beb80 con 0x556ddf3802f0 2024-05-27T07:35:40.534 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:35:40.536+0000 7fc66ab8e740 10 client.0 _ll_register_callbacks cb 0x7fc658009790 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T07:35:40.534 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:35:40.537+0000 7fc66219c640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x556ddf3a91b0 0x556ddf759c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:35:40.534 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:35:40.537+0000 7fc66219c640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x556ddf3a91b0 0x556ddf759c00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.52:56744/0 (socket says 172.21.15.52:56744) 2024-05-27T07:35:40.534 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:35:40.537+0000 7fc66219c640 1 -- 172.21.15.52:0/3537509979 learned_addr learned my addr 172.21.15.52:0/3537509979 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-05-27T07:35:40.534 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:35:40.537+0000 7fc66199b640 1 --2- 172.21.15.52:0/3537509979 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x556ddf3802f0 0x556ddf75d200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:35:40.537 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:35:40.537+0000 7fc66219c640 1 -- 172.21.15.52:0/3537509979 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x556ddf3802f0 msgr2=0x556ddf75d200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:35:40.537 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:35:40.537+0000 7fc66219c640 1 --2- 172.21.15.52:0/3537509979 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x556ddf3802f0 0x556ddf75d200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:35:40.537 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:35:40.537+0000 7fc66219c640 1 -- 172.21.15.52:0/3537509979 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x556ddf3884d0 msgr2=0x556ddf759560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:35:40.537 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:35:40.537+0000 7fc66219c640 1 --2- 172.21.15.52:0/3537509979 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x556ddf3884d0 0x556ddf759560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:35:40.537 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:35:40.537+0000 7fc66219c640 1 -- 172.21.15.52:0/3537509979 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({osdmap=0}) -- 0x556ddf75a340 con 0x556ddf3a91b0 2024-05-27T07:35:40.537 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:35:40.538+0000 7fc66219c640 1 --2- 172.21.15.52:0/3537509979 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x556ddf3a91b0 0x556ddf759c00 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7fc64c0a8cc0 tx=0x7fc64c0df810 comp rx=0 tx=0).ready entity=mon.2 client_cookie=2dada0388d2debfc server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:35:40.537 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:35:40.538+0000 7fc6457fa640 10 client.0 ms_handle_connect on v2:172.21.15.105:3300/0 2024-05-27T07:35:40.537 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:35:40.538+0000 7fc6457fa640 1 -- 172.21.15.52:0/3537509979 <== mon.2 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc64c0c7020 con 0x556ddf3a91b0 2024-05-27T07:35:40.537 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:35:40.538+0000 7fc66ab8e740 1 -- 172.21.15.52:0/3537509979 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fc64c0a5250 con 0x556ddf3a91b0 2024-05-27T07:35:40.537 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:35:40.538+0000 7fc6457fa640 1 -- 172.21.15.52:0/3537509979 <== mon.2 v2:172.21.15.105:3300/0 2 ==== osd_map(123..123 src has 1..123) ==== 9555+0+0 (secure 0 0 0) 0x7fc64c0a8220 con 0x556ddf3a91b0 2024-05-27T07:35:40.538 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:35:40.538+0000 7fc6457fa640 1 -- 172.21.15.52:0/3537509979 <== mon.2 v2:172.21.15.105:3300/0 3 ==== mon_get_version_reply(handle=1 version=108) ==== 24+0+0 (secure 0 0 0) 0x7fc64c020040 con 0x556ddf3a91b0 2024-05-27T07:35:40.538 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:35:40.541+0000 7fc66ab8e740 10 client.26837 fetch_fsmap learned FSMap version 108 2024-05-27T07:35:40.538 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:35:40.541+0000 7fc66ab8e740 1 -- 172.21.15.52:0/3537509979 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,fsmap=108,monmap=4+}) -- 0x556ddf7ce9e0 con 0x556ddf3a91b0 2024-05-27T07:35:40.538 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:35:40.542+0000 7fc6457fa640 1 -- 172.21.15.52:0/3537509979 <== mon.2 v2:172.21.15.105:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fc64c0a5250 con 0x556ddf3a91b0 2024-05-27T07:35:40.538 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:35:40.542+0000 7fc6457fa640 1 -- 172.21.15.52:0/3537509979 <== mon.2 v2:172.21.15.105:3300/0 5 ==== fsmap(e 108) ==== 4203+0+0 (secure 0 0 0) 0x7fc64c0ae450 con 0x556ddf3a91b0 2024-05-27T07:35:40.539 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:35:40.542+0000 7fc66ab8e740 10 client.26837 fetch_fsmap finished waiting for FSMap version 108 2024-05-27T07:35:40.540 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:35:40.542+0000 7fc66ab8e740 10 client.26837 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-05-27T07:35:40.540 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:35:40.542+0000 7fc66ab8e740 20 client.26837 populate_metadata read hostname 'smithi052' 2024-05-27T07:35:40.540 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:35:40.542+0000 7fc66ab8e740 1 --2- 172.21.15.52:0/3537509979 >> [v2:172.21.15.105:6838/500348151,v1:172.21.15.105:6839/500348151] conn(0x556ddf33d1c0 0x556ddf7d1100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:35:40.540 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:35:40.542+0000 7fc66ab8e740 4 client.26837 mds_command: new command op to 24527 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-05-27T07:35:40.540 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:35:40.542+0000 7fc66ab8e740 1 -- 172.21.15.52:0/3537509979 --> [v2:172.21.15.105:6838/500348151,v1:172.21.15.105:6839/500348151] -- command(tid 0: {"prefix": "flush journal"}) -- 0x556ddf6b8360 con 0x556ddf33d1c0 2024-05-27T07:35:40.540 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:35:40.542+0000 7fc66199b640 1 --2- 172.21.15.52:0/3537509979 >> [v2:172.21.15.105:6838/500348151,v1:172.21.15.105:6839/500348151] conn(0x556ddf33d1c0 0x556ddf7d1100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:35:40.543 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:35:40.545+0000 7fc66199b640 1 --2- 172.21.15.52:0/3537509979 >> [v2:172.21.15.105:6838/500348151,v1:172.21.15.105:6839/500348151] conn(0x556ddf33d1c0 0x556ddf7d1100 crc :-1 s=READY pgs=172 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=4a43fb29c12e2945 server_cookie=3320169132d370a7 in_seq=0 out_seq=0 2024-05-27T07:35:40.543 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:35:40.545+0000 7fc6457fa640 10 client.26837 ms_handle_connect on v2:172.21.15.105:6838/500348151 2024-05-27T07:35:40.607 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:35:40 smithi173 ceph-mon[37000]: pgmap v326: 129 pgs: 10 active+clean+snaptrim, 26 active+clean+snaptrim_wait, 93 active+clean; 647 MiB data, 6.3 GiB used, 1.0 TiB / 1.0 TiB avail; 39 KiB/s rd, 1.2 MiB/s wr, 236 op/s 2024-05-27T07:35:40.607 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:35:40 smithi173 ceph-mon[37000]: from='mgr.15333 172.21.15.52:0/416389082' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-27T07:35:40.688 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:35:40 smithi052 ceph-mon[31955]: pgmap v326: 129 pgs: 10 active+clean+snaptrim, 26 active+clean+snaptrim_wait, 93 active+clean; 647 MiB data, 6.3 GiB used, 1.0 TiB / 1.0 TiB avail; 39 KiB/s rd, 1.2 MiB/s wr, 236 op/s 2024-05-27T07:35:40.689 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:35:40 smithi052 ceph-mon[31955]: from='mgr.15333 172.21.15.52:0/416389082' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-27T07:35:40.695 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:35:40 smithi105 ceph-mon[35977]: pgmap v326: 129 pgs: 10 active+clean+snaptrim, 26 active+clean+snaptrim_wait, 93 active+clean; 647 MiB data, 6.3 GiB used, 1.0 TiB / 1.0 TiB avail; 39 KiB/s rd, 1.2 MiB/s wr, 236 op/s 2024-05-27T07:35:40.695 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:35:40 smithi105 ceph-mon[35977]: from='mgr.15333 172.21.15.52:0/416389082' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-27T07:35:42.128 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:35:42.131+0000 7fc6457fa640 1 -- 172.21.15.52:0/3537509979 <== mds.0 v2:172.21.15.105:6838/500348151 1 ==== command_reply(tid 0: 0 ) ==== 8+0+25 (crc 0 0 0) 0x556ddf6b8360 con 0x556ddf33d1c0 2024-05-27T07:35:42.128 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:35:42.131+0000 7fc6457fa640 10 client.26837 handle_command_reply: tid=0 2024-05-27T07:35:42.128 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:35:42.131+0000 7fc66ab8e740 1 client.26837 shutdown 2024-05-27T07:35:42.128 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:35:42.131+0000 7fc66ab8e740 1 -- 172.21.15.52:0/3537509979 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x556ddf3a91b0 msgr2=0x556ddf759c00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:35:42.128 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:35:42.131+0000 7fc66ab8e740 1 --2- 172.21.15.52:0/3537509979 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x556ddf3a91b0 0x556ddf759c00 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7fc64c0a8cc0 tx=0x7fc64c0df810 comp rx=0 tx=0).stop 2024-05-27T07:35:42.128 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:35:42.131+0000 7fc66ab8e740 1 -- 172.21.15.52:0/3537509979 shutdown_connections 2024-05-27T07:35:42.128 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:35:42.131+0000 7fc66ab8e740 1 --2- 172.21.15.52:0/3537509979 >> [v2:172.21.15.105:6838/500348151,v1:172.21.15.105:6839/500348151] conn(0x556ddf33d1c0 0x556ddf7d1100 crc :-1 s=READY pgs=172 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:35:42.129 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:35:42.131+0000 7fc66ab8e740 1 --2- 172.21.15.52:0/3537509979 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x556ddf3a91b0 0x556ddf759c00 unknown :-1 s=CLOSED pgs=251 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:35:42.129 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:35:42.131+0000 7fc66ab8e740 1 --2- 172.21.15.52:0/3537509979 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x556ddf3884d0 0x556ddf759560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:35:42.129 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:35:42.131+0000 7fc66ab8e740 1 --2- 172.21.15.52:0/3537509979 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x556ddf3802f0 0x556ddf75d200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:35:42.129 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:35:42.131+0000 7fc6457fa640 0 client.26837 ms_handle_reset on v2:172.21.15.105:6838/500348151 2024-05-27T07:35:42.129 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:35:42.131+0000 7fc66ab8e740 1 -- 172.21.15.52:0/3537509979 >> 172.21.15.52:0/3537509979 conn(0x556ddf39d300 msgr2=0x556ddf74e1e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:35:42.129 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:35:42.131+0000 7fc66ab8e740 1 -- 172.21.15.52:0/3537509979 shutdown_connections 2024-05-27T07:35:42.129 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:35:42.131+0000 7fc66ab8e740 1 -- 172.21.15.52:0/3537509979 wait complete. 2024-05-27T07:35:42.129 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:35:42.131+0000 7fc66ab8e740 20 client.26837 trim_cache size 0 max 16384 2024-05-27T07:35:42.130 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stdout:{ 2024-05-27T07:35:42.130 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stdout: "return_code": 0 2024-05-27T07:35:42.130 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stdout:} 2024-05-27T07:35:42.131 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:35:42.132+0000 7fc66299d640 1 -- 172.21.15.52:0/2134933493 >> [v2:172.21.15.52:6800/1822318384,v1:172.21.15.52:6801/1822318384] conn(0x7fc6480b2b20 msgr2=0x7fc6480b5030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:35:42.131 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:35:42.132+0000 7fc66299d640 1 --2- 172.21.15.52:0/2134933493 >> [v2:172.21.15.52:6800/1822318384,v1:172.21.15.52:6801/1822318384] conn(0x7fc6480b2b20 0x7fc6480b5030 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7fc654009420 tx=0x7fc654030b70 comp rx=0 tx=0).stop 2024-05-27T07:35:42.131 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:35:42.132+0000 7fc66299d640 1 -- 172.21.15.52:0/2134933493 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fc65c1ae3c0 msgr2=0x7fc65c1ae7d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:35:42.131 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:35:42.132+0000 7fc66299d640 1 --2- 172.21.15.52:0/2134933493 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fc65c1ae3c0 0x7fc65c1ae7d0 secure :-1 s=READY pgs=752 cs=0 l=1 rev1=1 crypto rx=0x7fc64c00d750 tx=0x7fc64c00dc10 comp rx=0 tx=0).stop 2024-05-27T07:35:42.131 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:35:42.132+0000 7fc66299d640 1 -- 172.21.15.52:0/2134933493 shutdown_connections 2024-05-27T07:35:42.131 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:35:42.132+0000 7fc66299d640 1 --2- 172.21.15.52:0/2134933493 >> [v2:172.21.15.52:6800/1822318384,v1:172.21.15.52:6801/1822318384] conn(0x7fc6480b2b20 0x7fc6480b5030 unknown :-1 s=CLOSED pgs=188 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:35:42.131 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:35:42.132+0000 7fc66299d640 1 --2- 172.21.15.52:0/2134933493 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fc65c1ae3c0 0x7fc65c1ae7d0 unknown :-1 s=CLOSED pgs=752 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:35:42.131 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:35:42.132+0000 7fc66299d640 1 --2- 172.21.15.52:0/2134933493 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fc65c1a9c60 0x7fc65c1ade80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:35:42.131 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:35:42.132+0000 7fc66299d640 1 --2- 172.21.15.52:0/2134933493 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fc65c10a200 0x7fc65c1a9720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:35:42.131 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:35:42.132+0000 7fc66299d640 1 -- 172.21.15.52:0/2134933493 >> 172.21.15.52:0/2134933493 conn(0x7fc65c06f800 msgr2=0x7fc65c10d450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:35:42.131 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:35:42.132+0000 7fc66299d640 1 -- 172.21.15.52:0/2134933493 shutdown_connections 2024-05-27T07:35:42.131 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:35:42.132+0000 7fc66299d640 1 -- 172.21.15.52:0/2134933493 wait complete. 2024-05-27T07:35:42.195 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:35:41 smithi105 ceph-mon[35977]: from='mgr.15333 172.21.15.52:0/416389082' entity='mgr.x' 2024-05-27T07:35:42.195 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:35:41 smithi105 ceph-mon[35977]: from='mgr.15333 172.21.15.52:0/416389082' entity='mgr.x' 2024-05-27T07:35:42.195 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:35:41 smithi105 ceph-mon[35977]: pgmap v327: 129 pgs: 10 active+clean+snaptrim, 26 active+clean+snaptrim_wait, 93 active+clean; 650 MiB data, 6.4 GiB used, 1.0 TiB / 1.0 TiB avail; 46 KiB/s rd, 1.3 MiB/s wr, 218 op/s 2024-05-27T07:35:42.196 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:35:41 smithi105 ceph-mon[35977]: from='mgr.15333 172.21.15.52:0/416389082' entity='mgr.x' 2024-05-27T07:35:42.196 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:35:41 smithi105 ceph-mon[35977]: from='mgr.15333 172.21.15.52:0/416389082' entity='mgr.x' 2024-05-27T07:35:42.196 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:35:41 smithi105 ceph-mon[35977]: from='mgr.15333 172.21.15.52:0/416389082' entity='mgr.x' 2024-05-27T07:35:42.196 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:35:41 smithi105 ceph-mon[35977]: from='mgr.15333 172.21.15.52:0/416389082' entity='mgr.x' 2024-05-27T07:35:42.234 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:35:41 smithi052 ceph-mon[31955]: from='mgr.15333 172.21.15.52:0/416389082' entity='mgr.x' 2024-05-27T07:35:42.235 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:35:41 smithi052 ceph-mon[31955]: from='mgr.15333 172.21.15.52:0/416389082' entity='mgr.x' 2024-05-27T07:35:42.235 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:35:41 smithi052 ceph-mon[31955]: pgmap v327: 129 pgs: 10 active+clean+snaptrim, 26 active+clean+snaptrim_wait, 93 active+clean; 650 MiB data, 6.4 GiB used, 1.0 TiB / 1.0 TiB avail; 46 KiB/s rd, 1.3 MiB/s wr, 218 op/s 2024-05-27T07:35:42.235 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:35:41 smithi052 ceph-mon[31955]: from='mgr.15333 172.21.15.52:0/416389082' entity='mgr.x' 2024-05-27T07:35:42.235 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:35:41 smithi052 ceph-mon[31955]: from='mgr.15333 172.21.15.52:0/416389082' entity='mgr.x' 2024-05-27T07:35:42.235 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:35:41 smithi052 ceph-mon[31955]: from='mgr.15333 172.21.15.52:0/416389082' entity='mgr.x' 2024-05-27T07:35:42.235 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:35:41 smithi052 ceph-mon[31955]: from='mgr.15333 172.21.15.52:0/416389082' entity='mgr.x' 2024-05-27T07:35:42.357 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:35:41 smithi173 ceph-mon[37000]: from='mgr.15333 172.21.15.52:0/416389082' entity='mgr.x' 2024-05-27T07:35:42.357 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:35:41 smithi173 ceph-mon[37000]: from='mgr.15333 172.21.15.52:0/416389082' entity='mgr.x' 2024-05-27T07:35:42.358 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:35:41 smithi173 ceph-mon[37000]: pgmap v327: 129 pgs: 10 active+clean+snaptrim, 26 active+clean+snaptrim_wait, 93 active+clean; 650 MiB data, 6.4 GiB used, 1.0 TiB / 1.0 TiB avail; 46 KiB/s rd, 1.3 MiB/s wr, 218 op/s 2024-05-27T07:35:42.358 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:35:41 smithi173 ceph-mon[37000]: from='mgr.15333 172.21.15.52:0/416389082' entity='mgr.x' 2024-05-27T07:35:42.358 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:35:41 smithi173 ceph-mon[37000]: from='mgr.15333 172.21.15.52:0/416389082' entity='mgr.x' 2024-05-27T07:35:42.358 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:35:41 smithi173 ceph-mon[37000]: from='mgr.15333 172.21.15.52:0/416389082' entity='mgr.x' 2024-05-27T07:35:42.358 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:35:41 smithi173 ceph-mon[37000]: from='mgr.15333 172.21.15.52:0/416389082' entity='mgr.x' 2024-05-27T07:35:42.668 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:42.529+0000 7f4f4edf8700 1 -- 172.21.15.52:0/284885850 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f4f4810b270 msgr2=0x7f4f4811d3e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:35:42.668 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:42.529+0000 7f4f4edf8700 1 --2- 172.21.15.52:0/284885850 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f4f4810b270 0x7f4f4811d3e0 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7f4f4000b0a0 tx=0x7f4f4000b3b0 comp rx=0 tx=0).stop 2024-05-27T07:35:42.668 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:42.531+0000 7f4f4edf8700 1 -- 172.21.15.52:0/284885850 shutdown_connections 2024-05-27T07:35:42.668 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:42.531+0000 7f4f4edf8700 1 --2- 172.21.15.52:0/284885850 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f4f4811d920 0x7f4f4811fd40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:35:42.668 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:42.531+0000 7f4f4edf8700 1 --2- 172.21.15.52:0/284885850 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f4f4810a920 0x7f4f4810ad30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:35:42.668 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:42.531+0000 7f4f4edf8700 1 --2- 172.21.15.52:0/284885850 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f4f4810b270 0x7f4f4811d3e0 unknown :-1 s=CLOSED pgs=252 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:35:42.668 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:42.531+0000 7f4f4edf8700 1 -- 172.21.15.52:0/284885850 >> 172.21.15.52:0/284885850 conn(0x7f4f4806f4c0 msgr2=0x7f4f4806f900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:35:42.668 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:42.537+0000 7f4f4edf8700 1 -- 172.21.15.52:0/284885850 shutdown_connections 2024-05-27T07:35:42.668 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:42.537+0000 7f4f4edf8700 1 -- 172.21.15.52:0/284885850 wait complete. 2024-05-27T07:35:42.668 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:42.538+0000 7f4f4edf8700 1 Processor -- start 2024-05-27T07:35:42.669 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:42.538+0000 7f4f4edf8700 1 -- start start 2024-05-27T07:35:42.669 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:42.538+0000 7f4f4edf8700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f4f4810a920 0x7f4f4811bde0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:35:42.669 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:42.538+0000 7f4f4edf8700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f4f4810b270 0x7f4f48114f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:35:42.669 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:42.538+0000 7f4f4edf8700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f4f4811d920 0x7f4f48115600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:35:42.669 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:42.538+0000 7f4f4edf8700 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x7f4f48123320 con 0x7f4f4810a920 2024-05-27T07:35:42.669 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:42.538+0000 7f4f4edf8700 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f4f481231a0 con 0x7f4f4810b270 2024-05-27T07:35:42.669 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:42.538+0000 7f4f4edf8700 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f4f481234a0 con 0x7f4f4811d920 2024-05-27T07:35:42.669 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:42.541+0000 7f4f47fff700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f4f4810b270 0x7f4f48114f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:35:42.669 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:42.541+0000 7f4f4d395700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f4f4811d920 0x7f4f48115600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:35:42.669 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:42.541+0000 7f4f47fff700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f4f4810b270 0x7f4f48114f60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.52:54294/0 (socket says 172.21.15.52:54294) 2024-05-27T07:35:42.669 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:42.541+0000 7f4f4cb94700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f4f4810a920 0x7f4f4811bde0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:35:42.669 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:42.541+0000 7f4f4cb94700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f4f4810a920 0x7f4f4811bde0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.52:34580/0 (socket says 172.21.15.52:34580) 2024-05-27T07:35:42.670 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:42.541+0000 7f4f47fff700 1 -- 172.21.15.52:0/2613021927 learned_addr learned my addr 172.21.15.52:0/2613021927 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-05-27T07:35:42.670 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:42.542+0000 7f4f47fff700 1 -- 172.21.15.52:0/2613021927 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f4f4811d920 msgr2=0x7f4f48115600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:35:42.670 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:42.542+0000 7f4f47fff700 1 --2- 172.21.15.52:0/2613021927 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f4f4811d920 0x7f4f48115600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:35:42.670 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:42.542+0000 7f4f47fff700 1 -- 172.21.15.52:0/2613021927 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f4f4810a920 msgr2=0x7f4f4811bde0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:35:42.670 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:42.546+0000 7f4f47fff700 1 --2- 172.21.15.52:0/2613021927 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f4f4810a920 0x7f4f4811bde0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:35:42.670 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:42.546+0000 7f4f47fff700 1 -- 172.21.15.52:0/2613021927 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4f48119910 con 0x7f4f4810b270 2024-05-27T07:35:42.670 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:42.546+0000 7f4f4cb94700 1 --2- 172.21.15.52:0/2613021927 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f4f4810a920 0x7f4f4811bde0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-27T07:35:42.670 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:42.547+0000 7f4f47fff700 1 --2- 172.21.15.52:0/2613021927 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f4f4810b270 0x7f4f48114f60 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7f4f4000b070 tx=0x7f4f40004e60 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:35:42.670 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:42.551+0000 7f4f45ffb700 1 -- 172.21.15.52:0/2613021927 <== mon.2 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4f40021070 con 0x7f4f4810b270 2024-05-27T07:35:42.670 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:42.551+0000 7f4f45ffb700 1 -- 172.21.15.52:0/2613021927 <== mon.2 v2:172.21.15.105:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f4f40007c00 con 0x7f4f4810b270 2024-05-27T07:35:42.670 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:42.551+0000 7f4f45ffb700 1 -- 172.21.15.52:0/2613021927 <== mon.2 v2:172.21.15.105:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4f400052a0 con 0x7f4f4810b270 2024-05-27T07:35:42.670 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:42.554+0000 7f4f4edf8700 1 -- 172.21.15.52:0/2613021927 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4f48119ba0 con 0x7f4f4810b270 2024-05-27T07:35:42.671 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:42.554+0000 7f4f4edf8700 1 -- 172.21.15.52:0/2613021927 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4f4811a060 con 0x7f4f4810b270 2024-05-27T07:35:42.671 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:42.562+0000 7f4f45ffb700 1 -- 172.21.15.52:0/2613021927 <== mon.2 v2:172.21.15.105:3300/0 4 ==== mgrmap(e 26) ==== 149783+0+0 (secure 0 0 0) 0x7f4f40024070 con 0x7f4f4810b270 2024-05-27T07:35:42.671 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:42.563+0000 7f4f5d80db80 1 Processor -- start 2024-05-27T07:35:42.671 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:42.563+0000 7f4f5d80db80 1 -- start start 2024-05-27T07:35:42.671 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:42.563+0000 7f4f45ffb700 1 --2- 172.21.15.52:0/2613021927 >> [v2:172.21.15.52:6800/1822318384,v1:172.21.15.52:6801/1822318384] conn(0x7f4f300b2bc0 0x7f4f300b50d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:35:42.671 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:42.564+0000 7f4f45ffb700 1 -- 172.21.15.52:0/2613021927 <== mon.2 v2:172.21.15.105:3300/0 5 ==== osd_map(123..123 src has 1..123) ==== 9555+0+0 (secure 0 0 0) 0x7f4f4000f070 con 0x7f4f4810b270 2024-05-27T07:35:42.671 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:42.564+0000 7f4f4d395700 1 --2- 172.21.15.52:0/2613021927 >> [v2:172.21.15.52:6800/1822318384,v1:172.21.15.52:6801/1822318384] conn(0x7f4f300b2bc0 0x7f4f300b50d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:35:42.671 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:42.564+0000 7f4f5d80db80 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x556727a72e40 0x556727bf7b30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:35:42.671 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:42.564+0000 7f4f5d80db80 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x556727bf8070 0x556727bf1be0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:35:42.671 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:42.564+0000 7f4f5d80db80 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x556727bf2120 0x556727bf2530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:35:42.671 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:42.564+0000 7f4f5d80db80 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x556727b1fd70 con 0x556727a72e40 2024-05-27T07:35:42.671 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:42.564+0000 7f4f5d80db80 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x556727b0f910 con 0x556727bf8070 2024-05-27T07:35:42.672 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:42.564+0000 7f4f5d80db80 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x556727b13950 con 0x556727bf2120 2024-05-27T07:35:42.672 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:42.564+0000 7f4f47fff700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x556727bf8070 0x556727bf1be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:35:42.672 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:42.564+0000 7f4f47fff700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x556727bf8070 0x556727bf1be0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.52:54306/0 (socket says 172.21.15.52:54306) 2024-05-27T07:35:42.672 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:42.564+0000 7f4f47fff700 1 -- 172.21.15.52:0/1279884665 learned_addr learned my addr 172.21.15.52:0/1279884665 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-05-27T07:35:42.672 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:42.564+0000 7f4f4cb94700 1 --2- 172.21.15.52:0/1279884665 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x556727a72e40 0x556727bf7b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:35:42.672 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:42.565+0000 7f4f4d395700 1 --2- 172.21.15.52:0/1279884665 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x556727bf2120 0x556727bf2530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:35:42.672 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:42.565+0000 7f4f4d395700 1 --2- 172.21.15.52:0/2613021927 >> [v2:172.21.15.52:6800/1822318384,v1:172.21.15.52:6801/1822318384] conn(0x7f4f300b2bc0 0x7f4f300b50d0 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7f4f38005950 tx=0x7f4f380046a0 comp rx=0 tx=0).ready entity=mgr.15333 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:35:42.672 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:42.565+0000 7f4f47fff700 1 -- 172.21.15.52:0/1279884665 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x556727bf2120 msgr2=0x556727bf2530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:35:42.672 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:42.565+0000 7f4f47fff700 1 --2- 172.21.15.52:0/1279884665 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x556727bf2120 0x556727bf2530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:35:42.672 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:42.565+0000 7f4f47fff700 1 -- 172.21.15.52:0/1279884665 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x556727a72e40 msgr2=0x556727bf7b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:35:42.672 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:42.565+0000 7f4f47fff700 1 --2- 172.21.15.52:0/1279884665 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x556727a72e40 0x556727bf7b30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:35:42.672 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:42.565+0000 7f4f47fff700 1 -- 172.21.15.52:0/1279884665 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x556727ab8fb0 con 0x556727bf8070 2024-05-27T07:35:42.672 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:42.565+0000 7f4f47fff700 1 --2- 172.21.15.52:0/1279884665 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x556727bf8070 0x556727bf1be0 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7f4f400095c0 tx=0x7f4f40009c10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b91e25f0f02415bd server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:35:42.673 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:42.566+0000 7f4f2effd700 1 -- 172.21.15.52:0/1279884665 <== mon.2 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4f400aeab0 con 0x556727bf8070 2024-05-27T07:35:42.673 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:42.566+0000 7f4f2effd700 1 -- 172.21.15.52:0/1279884665 <== mon.2 v2:172.21.15.105:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f4f400b34d0 con 0x556727bf8070 2024-05-27T07:35:42.673 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:42.566+0000 7f4f2effd700 1 -- 172.21.15.52:0/1279884665 <== mon.2 v2:172.21.15.105:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4f400aa960 con 0x556727bf8070 2024-05-27T07:35:42.673 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:42.567+0000 7f4f5d80db80 1 -- 172.21.15.52:0/1279884665 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x556727bf8070 msgr2=0x556727bf1be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:35:42.673 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:42.567+0000 7f4f5d80db80 1 --2- 172.21.15.52:0/1279884665 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x556727bf8070 0x556727bf1be0 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7f4f400095c0 tx=0x7f4f40009c10 comp rx=0 tx=0).stop 2024-05-27T07:35:42.673 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:42.567+0000 7f4f5d80db80 1 -- 172.21.15.52:0/1279884665 shutdown_connections 2024-05-27T07:35:42.673 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:42.567+0000 7f4f5d80db80 1 --2- 172.21.15.52:0/1279884665 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x556727bf2120 0x556727bf2530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:35:42.673 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:42.567+0000 7f4f5d80db80 1 --2- 172.21.15.52:0/1279884665 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x556727a72e40 0x556727bf7b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:35:42.673 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:42.567+0000 7f4f5d80db80 1 --2- 172.21.15.52:0/1279884665 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x556727bf8070 0x556727bf1be0 unknown :-1 s=CLOSED pgs=254 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:35:42.673 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:42.567+0000 7f4f5d80db80 1 -- 172.21.15.52:0/1279884665 >> 172.21.15.52:0/1279884665 conn(0x556727a75770 msgr2=0x556727becd90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:35:42.673 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:42.572+0000 7f4f5d80db80 1 -- 172.21.15.52:0/1279884665 shutdown_connections 2024-05-27T07:35:42.673 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:42.572+0000 7f4f5d80db80 1 -- 172.21.15.52:0/1279884665 wait complete. 2024-05-27T07:35:42.673 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:42.573+0000 7f4f5d80db80 1 -- start start 2024-05-27T07:35:42.674 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:42.573+0000 7f4f5d80db80 1 Processor -- start 2024-05-27T07:35:42.674 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:42.574+0000 7f4f5d80db80 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x556727bf2120 0x556727bfb9a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:35:42.674 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:42.575+0000 7f4f5d80db80 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x556727bf8070 0x556727bfbee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:35:42.674 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:42.575+0000 7f4f4cb94700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x556727bf2120 0x556727bfb9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:35:42.674 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:42.575+0000 7f4f4cb94700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x556727bf2120 0x556727bfb9a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.52:54318/0 (socket says 172.21.15.52:54318) 2024-05-27T07:35:42.674 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:42.575+0000 7f4f4cb94700 1 -- 172.21.15.52:0/817267662 learned_addr learned my addr 172.21.15.52:0/817267662 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-05-27T07:35:42.674 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:42.575+0000 7f4f5d80db80 1 --2- 172.21.15.52:0/817267662 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x556727bfc420 0x556727c07b90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:35:42.674 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:42.575+0000 7f4f47fff700 1 --2- 172.21.15.52:0/817267662 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x556727bf8070 0x556727bfbee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:35:42.674 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:42.576+0000 7f4f5d80db80 1 -- 172.21.15.52:0/817267662 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x556727a15fa0 con 0x556727bfc420 2024-05-27T07:35:42.674 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:42.576+0000 7f4f5d80db80 1 -- 172.21.15.52:0/817267662 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x556727a77da0 con 0x556727bf2120 2024-05-27T07:35:42.674 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:42.576+0000 7f4f5d80db80 1 -- 172.21.15.52:0/817267662 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x556727bfe920 con 0x556727bf8070 2024-05-27T07:35:42.674 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:42.576+0000 7f4f5d80db80 10 client.0 _ll_register_callbacks cb 0x556727aa65b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T07:35:42.675 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:42.576+0000 7f4f4d395700 1 --2- 172.21.15.52:0/817267662 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x556727bfc420 0x556727c07b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:35:42.675 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:42.577+0000 7f4f4d395700 1 -- 172.21.15.52:0/817267662 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x556727bf8070 msgr2=0x556727bfbee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:35:42.675 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:42.577+0000 7f4f4d395700 1 --2- 172.21.15.52:0/817267662 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x556727bf8070 0x556727bfbee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:35:42.675 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:42.577+0000 7f4f4d395700 1 -- 172.21.15.52:0/817267662 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x556727bf2120 msgr2=0x556727bfb9a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:35:42.675 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:42.577+0000 7f4f4d395700 1 --2- 172.21.15.52:0/817267662 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x556727bf2120 0x556727bfb9a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:35:42.675 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:42.578+0000 7f4f4d395700 1 -- 172.21.15.52:0/817267662 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({osdmap=0}) -- 0x556727a5f8d0 con 0x556727bfc420 2024-05-27T07:35:42.675 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:42.580+0000 7f4f4d395700 1 --2- 172.21.15.52:0/817267662 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x556727bfc420 0x556727c07b90 secure :-1 s=READY pgs=754 cs=0 l=1 rev1=1 crypto rx=0x7f4f48112b90 tx=0x7f4f380047c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=249c8093ab094dfd server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:35:42.675 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:42.580+0000 7f4f27fff700 10 client.0 ms_handle_connect on v2:172.21.15.52:3300/0 2024-05-27T07:35:42.675 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:42.581+0000 7f4f27fff700 1 -- 172.21.15.52:0/817267662 <== mon.0 v2:172.21.15.52:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4f38004830 con 0x556727bfc420 2024-05-27T07:35:42.675 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:42.581+0000 7f4f27fff700 1 -- 172.21.15.52:0/817267662 <== mon.0 v2:172.21.15.52:3300/0 2 ==== osd_map(123..123 src has 1..123) ==== 9555+0+0 (secure 0 0 0) 0x7f4f380049d0 con 0x556727bfc420 2024-05-27T07:35:42.675 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:42.581+0000 7f4f5d80db80 1 -- 172.21.15.52:0/817267662 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f4f400b34d0 con 0x556727bfc420 2024-05-27T07:35:42.675 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:42.583+0000 7f4f27fff700 1 -- 172.21.15.52:0/817267662 <== mon.0 v2:172.21.15.52:3300/0 3 ==== mon_get_version_reply(handle=1 version=108) ==== 24+0+0 (secure 0 0 0) 0x7f4f38030050 con 0x556727bfc420 2024-05-27T07:35:42.675 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:42.583+0000 7f4f5d80db80 10 client.17193 fetch_fsmap learned FSMap version 108 2024-05-27T07:35:42.676 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:42.584+0000 7f4f5d80db80 1 -- 172.21.15.52:0/817267662 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({config=0+,fsmap=108,monmap=4+}) -- 0x556727ac9760 con 0x556727bfc420 2024-05-27T07:35:42.676 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:42.584+0000 7f4f27fff700 1 -- 172.21.15.52:0/817267662 <== mon.0 v2:172.21.15.52:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f4f38004c10 con 0x556727bfc420 2024-05-27T07:35:42.676 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:42.584+0000 7f4f27fff700 1 -- 172.21.15.52:0/817267662 <== mon.0 v2:172.21.15.52:3300/0 5 ==== fsmap(e 108) ==== 4203+0+0 (secure 0 0 0) 0x7f4f38026030 con 0x556727bfc420 2024-05-27T07:35:42.676 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:42.585+0000 7f4f5d80db80 10 client.17193 fetch_fsmap finished waiting for FSMap version 108 2024-05-27T07:35:42.676 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:42.585+0000 7f4f5d80db80 10 client.17193 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-05-27T07:35:42.676 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:42.585+0000 7f4f5d80db80 20 client.17193 populate_metadata read hostname 'smithi052' 2024-05-27T07:35:42.676 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:42.585+0000 7f4f5d80db80 1 --2- 172.21.15.52:0/817267662 >> [v2:172.21.15.105:6838/500348151,v1:172.21.15.105:6839/500348151] conn(0x556727c0bf00 0x556727c0e350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:35:42.676 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:42.585+0000 7f4f5d80db80 4 client.17193 mds_command: new command op to 24527 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-27T07:35:42.676 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:42.585+0000 7f4f5d80db80 1 -- 172.21.15.52:0/817267662 --> [v2:172.21.15.105:6838/500348151,v1:172.21.15.105:6839/500348151] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x7f4f400b34d0 con 0x556727c0bf00 2024-05-27T07:35:42.676 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:42.586+0000 7f4f4cb94700 1 --2- 172.21.15.52:0/817267662 >> [v2:172.21.15.105:6838/500348151,v1:172.21.15.105:6839/500348151] conn(0x556727c0bf00 0x556727c0e350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:35:42.676 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:42.589+0000 7f4f4cb94700 1 --2- 172.21.15.52:0/817267662 >> [v2:172.21.15.105:6838/500348151,v1:172.21.15.105:6839/500348151] conn(0x556727c0bf00 0x556727c0e350 crc :-1 s=READY pgs=173 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=a41e051ccd4dc545 server_cookie=ee5ca792508c2e7c in_seq=0 out_seq=0 2024-05-27T07:35:42.676 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:42.589+0000 7f4f27fff700 10 client.17193 ms_handle_connect on v2:172.21.15.105:6838/500348151 2024-05-27T07:35:42.676 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:42.593+0000 7f4f27fff700 1 -- 172.21.15.52:0/817267662 <== mds.0 v2:172.21.15.105:6838/500348151 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24041 (crc 0 0 0) 0x7f4f400b34d0 con 0x556727c0bf00 2024-05-27T07:35:42.677 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:42.593+0000 7f4f27fff700 10 client.17193 handle_command_reply: tid=0 2024-05-27T07:35:42.677 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:42.593+0000 7f4f5d80db80 1 client.17193 shutdown 2024-05-27T07:35:42.677 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:42.593+0000 7f4f5d80db80 1 -- 172.21.15.52:0/817267662 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x556727bfc420 msgr2=0x556727c07b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:35:42.677 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:42.593+0000 7f4f5d80db80 1 --2- 172.21.15.52:0/817267662 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x556727bfc420 0x556727c07b90 secure :-1 s=READY pgs=754 cs=0 l=1 rev1=1 crypto rx=0x7f4f48112b90 tx=0x7f4f380047c0 comp rx=0 tx=0).stop 2024-05-27T07:35:42.677 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:42.593+0000 7f4f5d80db80 1 -- 172.21.15.52:0/817267662 shutdown_connections 2024-05-27T07:35:42.677 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:42.593+0000 7f4f5d80db80 1 --2- 172.21.15.52:0/817267662 >> [v2:172.21.15.105:6838/500348151,v1:172.21.15.105:6839/500348151] conn(0x556727c0bf00 0x556727c0e350 crc :-1 s=READY pgs=173 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:35:42.677 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:42.594+0000 7f4f5d80db80 1 --2- 172.21.15.52:0/817267662 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x556727bfc420 0x556727c07b90 unknown :-1 s=CLOSED pgs=754 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:35:42.677 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:42.594+0000 7f4f5d80db80 1 --2- 172.21.15.52:0/817267662 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x556727bf2120 0x556727bfb9a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:35:42.677 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:42.594+0000 7f4f5d80db80 1 --2- 172.21.15.52:0/817267662 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x556727bf8070 0x556727bfbee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:35:42.677 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:42.594+0000 7f4f5d80db80 1 -- 172.21.15.52:0/817267662 >> 172.21.15.52:0/817267662 conn(0x556727a75770 msgr2=0x556727bf0bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:35:42.677 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:42.594+0000 7f4f27fff700 0 client.17193 ms_handle_reset on v2:172.21.15.105:6838/500348151 2024-05-27T07:35:42.677 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:42.594+0000 7f4f5d80db80 1 -- 172.21.15.52:0/817267662 shutdown_connections 2024-05-27T07:35:42.678 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:42.594+0000 7f4f5d80db80 1 -- 172.21.15.52:0/817267662 wait complete. 2024-05-27T07:35:42.678 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:42.595+0000 7f4f5d80db80 20 client.17193 trim_cache size 0 max 16384 2024-05-27T07:35:42.678 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:42.636+0000 7f4f5d80db80 1 Processor -- start 2024-05-27T07:35:42.678 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:42.637+0000 7f4f5d80db80 1 -- start start 2024-05-27T07:35:42.678 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:42.637+0000 7f4f5d80db80 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x556727bf2120 0x556727c07980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:35:42.678 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:42.637+0000 7f4f5d80db80 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x556727bf8070 0x556727c07ec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:35:42.678 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:42.638+0000 7f4f5d80db80 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x556727bfc420 0x556727bec720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:35:42.678 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:42.638+0000 7f4f47fff700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x556727bf8070 0x556727c07ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:35:42.678 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:42.638+0000 7f4f4cb94700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x556727bf2120 0x556727c07980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:35:42.678 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:42.638+0000 7f4f47fff700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x556727bf8070 0x556727c07ec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.52:54326/0 (socket says 172.21.15.52:54326) 2024-05-27T07:35:42.678 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:42.638+0000 7f4f4cb94700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x556727bf2120 0x556727c07980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.52:34610/0 (socket says 172.21.15.52:34610) 2024-05-27T07:35:42.679 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:42.638+0000 7f4f5d80db80 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x556727bf1180 con 0x556727bf2120 2024-05-27T07:35:42.679 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:42.638+0000 7f4f5d80db80 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f4f481c8340 con 0x556727bf8070 2024-05-27T07:35:42.679 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:42.638+0000 7f4f5d80db80 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f4f481c6770 con 0x556727bfc420 2024-05-27T07:35:42.679 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:42.638+0000 7f4f47fff700 1 -- 172.21.15.52:0/2098682594 learned_addr learned my addr 172.21.15.52:0/2098682594 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-05-27T07:35:42.679 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:42.638+0000 7f4f4d395700 1 --2- 172.21.15.52:0/2098682594 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x556727bfc420 0x556727bec720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:35:42.679 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:42.638+0000 7f4f4cb94700 1 -- 172.21.15.52:0/2098682594 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x556727bfc420 msgr2=0x556727bec720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:35:42.679 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:42.638+0000 7f4f4cb94700 1 --2- 172.21.15.52:0/2098682594 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x556727bfc420 0x556727bec720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:35:42.679 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:42.638+0000 7f4f4cb94700 1 -- 172.21.15.52:0/2098682594 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x556727bf8070 msgr2=0x556727c07ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:35:42.679 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:42.638+0000 7f4f4cb94700 1 --2- 172.21.15.52:0/2098682594 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x556727bf8070 0x556727c07ec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:35:42.679 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:42.639+0000 7f4f4cb94700 1 -- 172.21.15.52:0/2098682594 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x556727be62f0 con 0x556727bf2120 2024-05-27T07:35:42.679 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:42.639+0000 7f4f4cb94700 1 --2- 172.21.15.52:0/2098682594 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x556727bf2120 0x556727c07980 secure :-1 s=READY pgs=755 cs=0 l=1 rev1=1 crypto rx=0x556727c0ea90 tx=0x7f4f3c008040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=147390b60c754d3d server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:35:42.679 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:42.639+0000 7f4f2e7fc700 1 -- 172.21.15.52:0/2098682594 <== mon.0 v2:172.21.15.52:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4f3c0168b0 con 0x556727bf2120 2024-05-27T07:35:42.679 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:42.640+0000 7f4f2e7fc700 1 -- 172.21.15.52:0/2098682594 <== mon.0 v2:172.21.15.52:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f4f3c020030 con 0x556727bf2120 2024-05-27T07:35:42.680 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:42.640+0000 7f4f2e7fc700 1 -- 172.21.15.52:0/2098682594 <== mon.0 v2:172.21.15.52:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4f3c01c9a0 con 0x556727bf2120 2024-05-27T07:35:42.680 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:42.641+0000 7f4f5d80db80 1 -- 172.21.15.52:0/2098682594 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x556727bf2120 msgr2=0x556727c07980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:35:42.680 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:42.641+0000 7f4f5d80db80 1 --2- 172.21.15.52:0/2098682594 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x556727bf2120 0x556727c07980 secure :-1 s=READY pgs=755 cs=0 l=1 rev1=1 crypto rx=0x556727c0ea90 tx=0x7f4f3c008040 comp rx=0 tx=0).stop 2024-05-27T07:35:42.680 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:42.641+0000 7f4f5d80db80 1 -- 172.21.15.52:0/2098682594 shutdown_connections 2024-05-27T07:35:42.680 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:42.641+0000 7f4f5d80db80 1 --2- 172.21.15.52:0/2098682594 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x556727bfc420 0x556727bec720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:35:42.680 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:42.641+0000 7f4f5d80db80 1 --2- 172.21.15.52:0/2098682594 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x556727bf2120 0x556727c07980 unknown :-1 s=CLOSED pgs=755 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:35:42.680 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:42.641+0000 7f4f5d80db80 1 --2- 172.21.15.52:0/2098682594 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x556727bf8070 0x556727c07ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:35:42.680 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:42.642+0000 7f4f5d80db80 1 -- 172.21.15.52:0/2098682594 >> 172.21.15.52:0/2098682594 conn(0x556727a75770 msgr2=0x556727bfa2b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:35:42.680 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:42.642+0000 7f4f5d80db80 1 -- 172.21.15.52:0/2098682594 shutdown_connections 2024-05-27T07:35:42.680 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:42.642+0000 7f4f5d80db80 1 -- 172.21.15.52:0/2098682594 wait complete. 2024-05-27T07:35:42.680 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:42.643+0000 7f4f5d80db80 1 -- start start 2024-05-27T07:35:42.680 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:42.643+0000 7f4f5d80db80 1 Processor -- start 2024-05-27T07:35:42.680 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:42.644+0000 7f4f5d80db80 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x556727bf2120 0x556727c04190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:35:42.681 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:42.644+0000 7f4f5d80db80 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x556727bf8070 0x556727c07170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:35:42.681 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:42.645+0000 7f4f4cb94700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x556727bf2120 0x556727c04190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:35:42.681 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:42.645+0000 7f4f4cb94700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x556727bf2120 0x556727c04190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.173:3300/0 says I am v2:172.21.15.52:50950/0 (socket says 172.21.15.52:50950) 2024-05-27T07:35:42.681 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:42.645+0000 7f4f47fff700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x556727bf8070 0x556727c07170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:35:42.681 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:42.645+0000 7f4f47fff700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x556727bf8070 0x556727c07170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.52:54338/0 (socket says 172.21.15.52:54338) 2024-05-27T07:35:42.681 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:42.645+0000 7f4f4cb94700 1 -- 172.21.15.52:0/2771109195 learned_addr learned my addr 172.21.15.52:0/2771109195 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-05-27T07:35:42.681 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:42.645+0000 7f4f5d80db80 1 --2- 172.21.15.52:0/2771109195 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x556727bfc420 0x556727c076b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:35:42.681 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:42.645+0000 7f4f5d80db80 1 -- 172.21.15.52:0/2771109195 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x7f4f481c6e70 con 0x556727bfc420 2024-05-27T07:35:42.681 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:42.645+0000 7f4f5d80db80 1 -- 172.21.15.52:0/2771109195 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f4f481c7c40 con 0x556727bf8070 2024-05-27T07:35:42.681 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:42.645+0000 7f4f5d80db80 1 -- 172.21.15.52:0/2771109195 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f4f481c7570 con 0x556727bf2120 2024-05-27T07:35:42.681 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:42.645+0000 7f4f5d80db80 10 client.0 _ll_register_callbacks cb 0x556727bf2070 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T07:35:42.681 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:42.646+0000 7f4f4d395700 1 --2- 172.21.15.52:0/2771109195 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x556727bfc420 0x556727c076b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:35:42.682 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:42.646+0000 7f4f47fff700 1 -- 172.21.15.52:0/2771109195 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x556727bf2120 msgr2=0x556727c04190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:35:42.682 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:42.646+0000 7f4f47fff700 1 --2- 172.21.15.52:0/2771109195 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x556727bf2120 0x556727c04190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:35:42.682 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:42.646+0000 7f4f47fff700 1 -- 172.21.15.52:0/2771109195 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x556727bfc420 msgr2=0x556727c076b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:35:42.682 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:42.646+0000 7f4f47fff700 1 --2- 172.21.15.52:0/2771109195 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x556727bfc420 0x556727c076b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:35:42.682 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:42.646+0000 7f4f47fff700 1 -- 172.21.15.52:0/2771109195 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({osdmap=0}) -- 0x556727c07cc0 con 0x556727bf8070 2024-05-27T07:35:42.682 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:42.648+0000 7f4f47fff700 1 --2- 172.21.15.52:0/2771109195 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x556727bf8070 0x556727c07170 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7f4f400a7970 tx=0x7f4f400099e0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=513c8bb6f70061d3 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:35:42.682 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:42.649+0000 7f4f27fff700 10 client.0 ms_handle_connect on v2:172.21.15.105:3300/0 2024-05-27T07:35:42.682 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:42.649+0000 7f4f27fff700 1 -- 172.21.15.52:0/2771109195 <== mon.2 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4f400c3070 con 0x556727bf8070 2024-05-27T07:35:42.682 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:42.649+0000 7f4f27fff700 1 -- 172.21.15.52:0/2771109195 <== mon.2 v2:172.21.15.105:3300/0 2 ==== osd_map(123..123 src has 1..123) ==== 9555+0+0 (secure 0 0 0) 0x7f4f400aa960 con 0x556727bf8070 2024-05-27T07:35:42.682 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:42.649+0000 7f4f5d80db80 1 -- 172.21.15.52:0/2771109195 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f4f3c020030 con 0x556727bf8070 2024-05-27T07:35:42.682 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:42.650+0000 7f4f27fff700 1 -- 172.21.15.52:0/2771109195 <== mon.2 v2:172.21.15.105:3300/0 3 ==== mon_get_version_reply(handle=1 version=108) ==== 24+0+0 (secure 0 0 0) 0x7f4f400b34d0 con 0x556727bf8070 2024-05-27T07:35:42.682 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:42.650+0000 7f4f5d80db80 10 client.26855 fetch_fsmap learned FSMap version 108 2024-05-27T07:35:42.682 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:42.650+0000 7f4f5d80db80 1 -- 172.21.15.52:0/2771109195 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,fsmap=108,monmap=4+}) -- 0x556727c767d0 con 0x556727bf8070 2024-05-27T07:35:42.683 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:42.654+0000 7f4f27fff700 1 -- 172.21.15.52:0/2771109195 <== mon.2 v2:172.21.15.105:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f4f400115a0 con 0x556727bf8070 2024-05-27T07:35:42.683 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:42.654+0000 7f4f27fff700 1 -- 172.21.15.52:0/2771109195 <== mon.2 v2:172.21.15.105:3300/0 5 ==== fsmap(e 108) ==== 4203+0+0 (secure 0 0 0) 0x7f4f40011070 con 0x556727bf8070 2024-05-27T07:35:42.683 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:42.656+0000 7f4f5d80db80 10 client.26855 fetch_fsmap finished waiting for FSMap version 108 2024-05-27T07:35:42.683 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:42.656+0000 7f4f5d80db80 10 client.26855 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-05-27T07:35:42.683 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:42.656+0000 7f4f5d80db80 20 client.26855 populate_metadata read hostname 'smithi052' 2024-05-27T07:35:42.683 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:42.656+0000 7f4f5d80db80 1 --2- 172.21.15.52:0/2771109195 >> [v2:172.21.15.105:6838/500348151,v1:172.21.15.105:6839/500348151] conn(0x556727c76b50 0x556727c78f70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:35:42.683 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:42.656+0000 7f4f5d80db80 4 client.26855 mds_command: new command op to 24527 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-05-27T07:35:42.683 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:42.656+0000 7f4f5d80db80 1 -- 172.21.15.52:0/2771109195 --> [v2:172.21.15.105:6838/500348151,v1:172.21.15.105:6839/500348151] -- command(tid 0: {"prefix": "scrub status"}) -- 0x7f4f3c020030 con 0x556727c76b50 2024-05-27T07:35:42.683 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:42.657+0000 7f4f4cb94700 1 --2- 172.21.15.52:0/2771109195 >> [v2:172.21.15.105:6838/500348151,v1:172.21.15.105:6839/500348151] conn(0x556727c76b50 0x556727c78f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:35:42.683 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:42.665+0000 7f4f4cb94700 1 --2- 172.21.15.52:0/2771109195 >> [v2:172.21.15.105:6838/500348151,v1:172.21.15.105:6839/500348151] conn(0x556727c76b50 0x556727c78f70 crc :-1 s=READY pgs=174 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=128d062f06ee2b44 server_cookie=8ee7b7dee214a197 in_seq=0 out_seq=0 2024-05-27T07:35:42.683 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:42.665+0000 7f4f27fff700 10 client.26855 ms_handle_connect on v2:172.21.15.105:6838/500348151 2024-05-27T07:35:42.683 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:42.667+0000 7f4f27fff700 1 -- 172.21.15.52:0/2771109195 <== mds.0 v2:172.21.15.105:6838/500348151 1 ==== command_reply(tid 0: 0 ) ==== 8+0+268 (crc 0 0 0) 0x7f4f3c020030 con 0x556727c76b50 2024-05-27T07:35:42.683 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:42.667+0000 7f4f27fff700 10 client.26855 handle_command_reply: tid=0 2024-05-27T07:35:42.684 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:42.668+0000 7f4f5d80db80 1 client.26855 shutdown 2024-05-27T07:35:42.684 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:42.669+0000 7f4f5d80db80 1 -- 172.21.15.52:0/2771109195 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x556727bf8070 msgr2=0x556727c07170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:35:42.684 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:42.669+0000 7f4f5d80db80 1 --2- 172.21.15.52:0/2771109195 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x556727bf8070 0x556727c07170 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7f4f400a7970 tx=0x7f4f400099e0 comp rx=0 tx=0).stop 2024-05-27T07:35:42.684 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:42.670+0000 7f4f5d80db80 1 -- 172.21.15.52:0/2771109195 shutdown_connections 2024-05-27T07:35:42.684 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:42.670+0000 7f4f5d80db80 1 --2- 172.21.15.52:0/2771109195 >> [v2:172.21.15.105:6838/500348151,v1:172.21.15.105:6839/500348151] conn(0x556727c76b50 0x556727c78f70 crc :-1 s=READY pgs=174 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:35:42.684 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:42.670+0000 7f4f5d80db80 1 --2- 172.21.15.52:0/2771109195 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x556727bfc420 0x556727c076b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:35:42.684 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:42.670+0000 7f4f5d80db80 1 --2- 172.21.15.52:0/2771109195 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x556727bf2120 0x556727c04190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:35:42.684 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:42.670+0000 7f4f5d80db80 1 --2- 172.21.15.52:0/2771109195 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x556727bf8070 0x556727c07170 unknown :-1 s=CLOSED pgs=255 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:35:42.684 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:42.670+0000 7f4f5d80db80 1 -- 172.21.15.52:0/2771109195 >> 172.21.15.52:0/2771109195 conn(0x556727a75770 msgr2=0x556727bec720 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:35:42.684 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:42.670+0000 7f4f27fff700 0 client.26855 ms_handle_reset on v2:172.21.15.105:6838/500348151 2024-05-27T07:35:42.684 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:42.671+0000 7f4f5d80db80 1 -- 172.21.15.52:0/2771109195 shutdown_connections 2024-05-27T07:35:42.685 INFO:teuthology.orchestra.run.smithi052.stdout:{ 2024-05-27T07:35:42.685 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "scrub active (1574 inodes in the stack)", 2024-05-27T07:35:42.685 INFO:teuthology.orchestra.run.smithi052.stdout: "scrubs": { 2024-05-27T07:35:42.685 INFO:teuthology.orchestra.run.smithi052.stdout: "6e2d3879-5958-41b8-8d1b-86e2464004b8": { 2024-05-27T07:35:42.685 INFO:teuthology.orchestra.run.smithi052.stdout: "path": "/", 2024-05-27T07:35:42.685 INFO:teuthology.orchestra.run.smithi052.stdout: "tag": "6e2d3879-5958-41b8-8d1b-86e2464004b8", 2024-05-27T07:35:42.685 INFO:teuthology.orchestra.run.smithi052.stdout: "options": "recursive,force" 2024-05-27T07:35:42.685 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-05-27T07:35:42.685 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-05-27T07:35:42.685 INFO:teuthology.orchestra.run.smithi052.stdout:} 2024-05-27T07:35:42.686 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:42.671+0000 7f4f5d80db80 1 -- 172.21.15.52:0/2771109195 wait complete. 2024-05-27T07:35:42.686 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:42.672+0000 7f4f5d80db80 20 client.26855 trim_cache size 0 max 16384 2024-05-27T07:35:42.686 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:42.673+0000 7f4f4edf8700 1 -- 172.21.15.52:0/2613021927 >> [v2:172.21.15.52:6800/1822318384,v1:172.21.15.52:6801/1822318384] conn(0x7f4f300b2bc0 msgr2=0x7f4f300b50d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:35:42.686 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:42.673+0000 7f4f4edf8700 1 --2- 172.21.15.52:0/2613021927 >> [v2:172.21.15.52:6800/1822318384,v1:172.21.15.52:6801/1822318384] conn(0x7f4f300b2bc0 0x7f4f300b50d0 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7f4f38005950 tx=0x7f4f380046a0 comp rx=0 tx=0).stop 2024-05-27T07:35:42.686 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:42.674+0000 7f4f4edf8700 1 -- 172.21.15.52:0/2613021927 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f4f4810b270 msgr2=0x7f4f48114f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:35:42.686 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:42.674+0000 7f4f4edf8700 1 --2- 172.21.15.52:0/2613021927 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f4f4810b270 0x7f4f48114f60 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7f4f4000b070 tx=0x7f4f40004e60 comp rx=0 tx=0).stop 2024-05-27T07:35:42.686 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:42.674+0000 7f4f4edf8700 1 -- 172.21.15.52:0/2613021927 shutdown_connections 2024-05-27T07:35:42.687 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:42.674+0000 7f4f4edf8700 1 --2- 172.21.15.52:0/2613021927 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f4f4811d920 0x7f4f48115600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:35:42.687 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:42.674+0000 7f4f4edf8700 1 --2- 172.21.15.52:0/2613021927 >> [v2:172.21.15.52:6800/1822318384,v1:172.21.15.52:6801/1822318384] conn(0x7f4f300b2bc0 0x7f4f300b50d0 unknown :-1 s=CLOSED pgs=189 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:35:42.687 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:42.674+0000 7f4f4edf8700 1 --2- 172.21.15.52:0/2613021927 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f4f4810a920 0x7f4f4811bde0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:35:42.687 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:42.674+0000 7f4f4edf8700 1 --2- 172.21.15.52:0/2613021927 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f4f4810b270 0x7f4f48114f60 secure :-1 s=CLOSED pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7f4f4000b070 tx=0x7f4f40004e60 comp rx=0 tx=0).stop 2024-05-27T07:35:42.687 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:42.674+0000 7f4f4edf8700 1 -- 172.21.15.52:0/2613021927 >> 172.21.15.52:0/2613021927 conn(0x7f4f4806f4c0 msgr2=0x7f4f48072780 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:35:42.687 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:42.679+0000 7f4f4edf8700 1 -- 172.21.15.52:0/2613021927 shutdown_connections 2024-05-27T07:35:42.687 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:35:42.680+0000 7f4f4edf8700 1 -- 172.21.15.52:0/2613021927 wait complete. 2024-05-27T07:35:43.195 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:35:42 smithi105 ceph-mon[35977]: from='mgr.15333 172.21.15.52:0/416389082' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T07:35:43.195 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:35:43 smithi105 ceph-mon[35977]: from='mgr.15333 172.21.15.52:0/416389082' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-27T07:35:43.195 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:35:43 smithi105 ceph-mon[35977]: from='mgr.15333 172.21.15.52:0/416389082' entity='mgr.x' 2024-05-27T07:35:43.235 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:35:42 smithi052 ceph-mon[31955]: from='mgr.15333 172.21.15.52:0/416389082' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T07:35:43.235 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:35:42 smithi052 ceph-mon[31955]: from='mgr.15333 172.21.15.52:0/416389082' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-27T07:35:43.235 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:35:42 smithi052 ceph-mon[31955]: from='mgr.15333 172.21.15.52:0/416389082' entity='mgr.x' 2024-05-27T07:35:43.357 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:35:42 smithi173 ceph-mon[37000]: from='mgr.15333 172.21.15.52:0/416389082' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T07:35:43.357 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:35:42 smithi173 ceph-mon[37000]: from='mgr.15333 172.21.15.52:0/416389082' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-27T07:35:43.358 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:35:42 smithi173 ceph-mon[37000]: from='mgr.15333 172.21.15.52:0/416389082' entity='mgr.x' 2024-05-27T07:35:43.388 INFO:tasks.cephfs.filesystem:scrub status for tag:6e2d3879-5958-41b8-8d1b-86e2464004b8 - {'path': '/', 'tag': '6e2d3879-5958-41b8-8d1b-86e2464004b8', 'options': 'recursive,force'} 2024-05-27T07:35:44.196 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:35:43 smithi105 ceph-mon[35977]: pgmap v328: 129 pgs: 10 active+clean+snaptrim, 26 active+clean+snaptrim_wait, 93 active+clean; 658 MiB data, 6.4 GiB used, 1.0 TiB / 1.0 TiB avail; 108 KiB/s rd, 1.9 MiB/s wr, 313 op/s 2024-05-27T07:35:44.235 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:35:43 smithi052 ceph-mon[31955]: pgmap v328: 129 pgs: 10 active+clean+snaptrim, 26 active+clean+snaptrim_wait, 93 active+clean; 658 MiB data, 6.4 GiB used, 1.0 TiB / 1.0 TiB avail; 108 KiB/s rd, 1.9 MiB/s wr, 313 op/s 2024-05-27T07:35:44.357 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:35:43 smithi173 ceph-mon[37000]: pgmap v328: 129 pgs: 10 active+clean+snaptrim, 26 active+clean+snaptrim_wait, 93 active+clean; 658 MiB data, 6.4 GiB used, 1.0 TiB / 1.0 TiB avail; 108 KiB/s rd, 1.9 MiB/s wr, 313 op/s 2024-05-27T07:35:46.607 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:35:46 smithi173 ceph-mon[37000]: pgmap v329: 129 pgs: 8 active+clean+snaptrim, 18 active+clean+snaptrim_wait, 103 active+clean; 650 MiB data, 6.4 GiB used, 1.0 TiB / 1.0 TiB avail; 182 KiB/s rd, 2.1 MiB/s wr, 386 op/s 2024-05-27T07:35:46.607 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:35:46 smithi173 ceph-mon[37000]: scrub summary: idle+waiting paths [/] 2024-05-27T07:35:46.695 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:35:46 smithi105 ceph-mon[35977]: pgmap v329: 129 pgs: 8 active+clean+snaptrim, 18 active+clean+snaptrim_wait, 103 active+clean; 650 MiB data, 6.4 GiB used, 1.0 TiB / 1.0 TiB avail; 182 KiB/s rd, 2.1 MiB/s wr, 386 op/s 2024-05-27T07:35:46.695 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:35:46 smithi105 ceph-mon[35977]: scrub summary: idle+waiting paths [/] 2024-05-27T07:35:46.735 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:35:46 smithi052 ceph-mon[31955]: pgmap v329: 129 pgs: 8 active+clean+snaptrim, 18 active+clean+snaptrim_wait, 103 active+clean; 650 MiB data, 6.4 GiB used, 1.0 TiB / 1.0 TiB avail; 182 KiB/s rd, 2.1 MiB/s wr, 386 op/s 2024-05-27T07:35:46.735 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:35:46 smithi052 ceph-mon[31955]: scrub summary: idle+waiting paths [/] 2024-05-27T07:35:47.607 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:35:47 smithi173 ceph-mon[37000]: scrub summary: idle 2024-05-27T07:35:47.607 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:35:47 smithi173 ceph-mon[37000]: scrub summary: idle 2024-05-27T07:35:47.695 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:35:47 smithi105 ceph-mon[35977]: scrub summary: idle 2024-05-27T07:35:47.695 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:35:47 smithi105 ceph-mon[35977]: scrub summary: idle 2024-05-27T07:35:47.735 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:35:47 smithi052 ceph-mon[31955]: scrub summary: idle 2024-05-27T07:35:47.735 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:35:47 smithi052 ceph-mon[31955]: scrub summary: idle 2024-05-27T07:35:48.607 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:35:48 smithi173 ceph-mon[37000]: scrub summary: idle 2024-05-27T07:35:48.607 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:35:48 smithi173 ceph-mon[37000]: pgmap v330: 129 pgs: 6 active+clean+snaptrim, 12 active+clean+snaptrim_wait, 111 active+clean; 641 MiB data, 6.4 GiB used, 1.0 TiB / 1.0 TiB avail; 267 KiB/s rd, 1.8 MiB/s wr, 520 op/s 2024-05-27T07:35:48.695 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:35:48 smithi105 ceph-mon[35977]: scrub summary: idle 2024-05-27T07:35:48.695 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:35:48 smithi105 ceph-mon[35977]: pgmap v330: 129 pgs: 6 active+clean+snaptrim, 12 active+clean+snaptrim_wait, 111 active+clean; 641 MiB data, 6.4 GiB used, 1.0 TiB / 1.0 TiB avail; 267 KiB/s rd, 1.8 MiB/s wr, 520 op/s 2024-05-27T07:35:48.735 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:35:48 smithi052 ceph-mon[31955]: scrub summary: idle 2024-05-27T07:35:48.735 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:35:48 smithi052 ceph-mon[31955]: pgmap v330: 129 pgs: 6 active+clean+snaptrim, 12 active+clean+snaptrim_wait, 111 active+clean; 641 MiB data, 6.4 GiB used, 1.0 TiB / 1.0 TiB avail; 267 KiB/s rd, 1.8 MiB/s wr, 520 op/s 2024-05-27T07:35:50.607 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:35:50 smithi173 ceph-mon[37000]: pgmap v331: 129 pgs: 4 active+clean+snaptrim, 12 active+clean+snaptrim_wait, 113 active+clean; 645 MiB data, 6.4 GiB used, 1.0 TiB / 1.0 TiB avail; 309 KiB/s rd, 2.0 MiB/s wr, 576 op/s 2024-05-27T07:35:50.608 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:35:50 smithi173 ceph-mon[37000]: osdmap e124: 12 total, 12 up, 12 in 2024-05-27T07:35:50.695 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:35:50 smithi105 ceph-mon[35977]: pgmap v331: 129 pgs: 4 active+clean+snaptrim, 12 active+clean+snaptrim_wait, 113 active+clean; 645 MiB data, 6.4 GiB used, 1.0 TiB / 1.0 TiB avail; 309 KiB/s rd, 2.0 MiB/s wr, 576 op/s 2024-05-27T07:35:50.695 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:35:50 smithi105 ceph-mon[35977]: osdmap e124: 12 total, 12 up, 12 in 2024-05-27T07:35:50.734 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:35:50 smithi052 ceph-mon[31955]: pgmap v331: 129 pgs: 4 active+clean+snaptrim, 12 active+clean+snaptrim_wait, 113 active+clean; 645 MiB data, 6.4 GiB used, 1.0 TiB / 1.0 TiB avail; 309 KiB/s rd, 2.0 MiB/s wr, 576 op/s 2024-05-27T07:35:50.735 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:35:50 smithi052 ceph-mon[31955]: osdmap e124: 12 total, 12 up, 12 in 2024-05-27T07:35:52.607 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:35:52 smithi173 ceph-mon[37000]: pgmap v333: 129 pgs: 4 active+clean+snaptrim, 12 active+clean+snaptrim_wait, 113 active+clean; 647 MiB data, 6.5 GiB used, 1.0 TiB / 1.0 TiB avail; 335 KiB/s rd, 2.1 MiB/s wr, 664 op/s 2024-05-27T07:35:52.695 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:35:52 smithi105 ceph-mon[35977]: pgmap v333: 129 pgs: 4 active+clean+snaptrim, 12 active+clean+snaptrim_wait, 113 active+clean; 647 MiB data, 6.5 GiB used, 1.0 TiB / 1.0 TiB avail; 335 KiB/s rd, 2.1 MiB/s wr, 664 op/s 2024-05-27T07:35:52.735 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:35:52 smithi052 ceph-mon[31955]: pgmap v333: 129 pgs: 4 active+clean+snaptrim, 12 active+clean+snaptrim_wait, 113 active+clean; 647 MiB data, 6.5 GiB used, 1.0 TiB / 1.0 TiB avail; 335 KiB/s rd, 2.1 MiB/s wr, 664 op/s 2024-05-27T07:35:54.607 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:35:54 smithi173 ceph-mon[37000]: pgmap v334: 129 pgs: 4 active+clean+snaptrim, 6 active+clean+snaptrim_wait, 119 active+clean; 649 MiB data, 6.5 GiB used, 1.0 TiB / 1.0 TiB avail; 260 KiB/s rd, 1.5 MiB/s wr, 626 op/s 2024-05-27T07:35:54.695 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:35:54 smithi105 ceph-mon[35977]: pgmap v334: 129 pgs: 4 active+clean+snaptrim, 6 active+clean+snaptrim_wait, 119 active+clean; 649 MiB data, 6.5 GiB used, 1.0 TiB / 1.0 TiB avail; 260 KiB/s rd, 1.5 MiB/s wr, 626 op/s 2024-05-27T07:35:54.735 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:35:54 smithi052 ceph-mon[31955]: pgmap v334: 129 pgs: 4 active+clean+snaptrim, 6 active+clean+snaptrim_wait, 119 active+clean; 649 MiB data, 6.5 GiB used, 1.0 TiB / 1.0 TiB avail; 260 KiB/s rd, 1.5 MiB/s wr, 626 op/s 2024-05-27T07:35:55.280 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:35:55.284+0000 7fb33eb86640 1 -- 172.21.15.52:0/2362184152 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fb33810a500 msgr2=0x7fb33810a910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:35:55.280 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:35:55.284+0000 7fb33eb86640 1 --2- 172.21.15.52:0/2362184152 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fb33810a500 0x7fb33810a910 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7fb32c009860 tx=0x7fb32c02a340 comp rx=0 tx=0).stop 2024-05-27T07:35:55.281 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:35:55.284+0000 7fb33eb86640 1 -- 172.21.15.52:0/2362184152 shutdown_connections 2024-05-27T07:35:55.281 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:35:55.284+0000 7fb33eb86640 1 --2- 172.21.15.52:0/2362184152 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fb33810f550 0x7fb338116cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:35:55.281 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:35:55.284+0000 7fb33eb86640 1 --2- 172.21.15.52:0/2362184152 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fb33810ea90 0x7fb33810eec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:35:55.281 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:35:55.284+0000 7fb33eb86640 1 --2- 172.21.15.52:0/2362184152 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fb33810a500 0x7fb33810a910 unknown :-1 s=CLOSED pgs=256 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:35:55.281 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:35:55.284+0000 7fb33eb86640 1 -- 172.21.15.52:0/2362184152 >> 172.21.15.52:0/2362184152 conn(0x7fb338077190 msgr2=0x7fb3380775d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:35:55.281 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:35:55.284+0000 7fb33eb86640 1 -- 172.21.15.52:0/2362184152 shutdown_connections 2024-05-27T07:35:55.281 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:35:55.284+0000 7fb33eb86640 1 -- 172.21.15.52:0/2362184152 wait complete. 2024-05-27T07:35:55.283 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:35:55.285+0000 7fb33eb86640 1 Processor -- start 2024-05-27T07:35:55.283 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:35:55.285+0000 7fb33eb86640 1 -- start start 2024-05-27T07:35:55.283 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:35:55.286+0000 7fb33eb86640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fb33810a500 0x7fb3381a0e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:35:55.283 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:35:55.286+0000 7fb33eb86640 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fb33810ea90 0x7fb3381a13c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:35:55.284 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:35:55.286+0000 7fb33eb86640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fb33810f550 0x7fb3381a5680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:35:55.284 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:35:55.286+0000 7fb33eb86640 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x7fb33811a480 con 0x7fb33810ea90 2024-05-27T07:35:55.284 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:35:55.286+0000 7fb33eb86640 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7fb33811a300 con 0x7fb33810f550 2024-05-27T07:35:55.284 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:35:55.286+0000 7fb33eb86640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7fb33811a600 con 0x7fb33810a500 2024-05-27T07:35:55.284 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:35:55.286+0000 7fb33d123640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fb33810f550 0x7fb3381a5680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:35:55.284 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:35:55.286+0000 7fb33d123640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fb33810f550 0x7fb3381a5680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.52:34954/0 (socket says 172.21.15.52:34954) 2024-05-27T07:35:55.284 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:35:55.286+0000 7fb33d123640 1 -- 172.21.15.52:0/4223507819 learned_addr learned my addr 172.21.15.52:0/4223507819 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-05-27T07:35:55.284 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:35:55.286+0000 7fb33c922640 1 --2- 172.21.15.52:0/4223507819 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fb33810a500 0x7fb3381a0e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:35:55.284 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:35:55.287+0000 7fb337fff640 1 --2- 172.21.15.52:0/4223507819 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fb33810ea90 0x7fb3381a13c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:35:55.284 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:35:55.287+0000 7fb33c922640 1 -- 172.21.15.52:0/4223507819 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fb33810f550 msgr2=0x7fb3381a5680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:35:55.285 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:35:55.287+0000 7fb33c922640 1 --2- 172.21.15.52:0/4223507819 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fb33810f550 0x7fb3381a5680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:35:55.285 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:35:55.287+0000 7fb33c922640 1 -- 172.21.15.52:0/4223507819 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fb33810ea90 msgr2=0x7fb3381a13c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:35:55.285 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:35:55.287+0000 7fb33c922640 1 --2- 172.21.15.52:0/4223507819 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fb33810ea90 0x7fb3381a13c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:35:55.285 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:35:55.287+0000 7fb33c922640 1 -- 172.21.15.52:0/4223507819 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb3381a5cc0 con 0x7fb33810a500 2024-05-27T07:35:55.285 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:35:55.287+0000 7fb337fff640 1 --2- 172.21.15.52:0/4223507819 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fb33810ea90 0x7fb3381a13c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-27T07:35:55.285 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:35:55.287+0000 7fb33c922640 1 --2- 172.21.15.52:0/4223507819 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fb33810a500 0x7fb3381a0e60 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7fb32c035040 tx=0x7fb32c032dc0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:35:55.285 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:35:55.287+0000 7fb335ffb640 1 -- 172.21.15.52:0/4223507819 <== mon.1 v2:172.21.15.173:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb32c040020 con 0x7fb33810a500 2024-05-27T07:35:55.285 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:35:55.287+0000 7fb335ffb640 1 -- 172.21.15.52:0/4223507819 <== mon.1 v2:172.21.15.173:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fb32c030450 con 0x7fb33810a500 2024-05-27T07:35:55.285 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:35:55.288+0000 7fb33eb86640 1 -- 172.21.15.52:0/4223507819 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb3381a5f50 con 0x7fb33810a500 2024-05-27T07:35:55.285 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:35:55.288+0000 7fb335ffb640 1 -- 172.21.15.52:0/4223507819 <== mon.1 v2:172.21.15.173:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb32c037040 con 0x7fb33810a500 2024-05-27T07:35:55.285 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:35:55.288+0000 7fb33eb86640 1 -- 172.21.15.52:0/4223507819 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb3381a6490 con 0x7fb33810a500 2024-05-27T07:35:55.287 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:35:55.291+0000 7fb335ffb640 1 -- 172.21.15.52:0/4223507819 <== mon.1 v2:172.21.15.173:3300/0 4 ==== mgrmap(e 26) ==== 149783+0+0 (secure 0 0 0) 0x7fb32c043020 con 0x7fb33810a500 2024-05-27T07:35:55.287 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:35:55.292+0000 7fb345a40740 1 Processor -- start 2024-05-27T07:35:55.288 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:35:55.292+0000 7fb345a40740 1 -- start start 2024-05-27T07:35:55.288 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:35:55.292+0000 7fb335ffb640 1 --2- 172.21.15.52:0/4223507819 >> [v2:172.21.15.52:6800/1822318384,v1:172.21.15.52:6801/1822318384] conn(0x7fb3280b29d0 0x7fb3280b4ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:35:55.288 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:35:55.293+0000 7fb335ffb640 1 -- 172.21.15.52:0/4223507819 <== mon.1 v2:172.21.15.173:3300/0 5 ==== osd_map(124..124 src has 1..124) ==== 9479+0+0 (secure 0 0 0) 0x7fb32c102470 con 0x7fb33810a500 2024-05-27T07:35:55.290 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:35:55.293+0000 7fb345a40740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55da059311b0 0x55da05cd23d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:35:55.290 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:35:55.293+0000 7fb33d123640 1 --2- 172.21.15.52:0/4223507819 >> [v2:172.21.15.52:6800/1822318384,v1:172.21.15.52:6801/1822318384] conn(0x7fb3280b29d0 0x7fb3280b4ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:35:55.290 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:35:55.293+0000 7fb345a40740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55da059082f0 0x55da05cd2910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:35:55.290 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:35:55.293+0000 7fb33c922640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55da059311b0 0x55da05cd23d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:35:55.290 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:35:55.294+0000 7fb33c922640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55da059311b0 0x55da05cd23d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.52:34962/0 (socket says 172.21.15.52:34962) 2024-05-27T07:35:55.290 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:35:55.294+0000 7fb33c922640 1 -- 172.21.15.52:0/4215610992 learned_addr learned my addr 172.21.15.52:0/4215610992 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-05-27T07:35:55.290 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:35:55.294+0000 7fb345a40740 1 --2- 172.21.15.52:0/4215610992 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55da059104d0 0x55da05cd2e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:35:55.290 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:35:55.294+0000 7fb345a40740 1 -- 172.21.15.52:0/4215610992 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x55da05c44c30 con 0x55da059104d0 2024-05-27T07:35:55.290 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:35:55.294+0000 7fb337fff640 1 --2- 172.21.15.52:0/4215610992 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55da059082f0 0x55da05cd2910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:35:55.290 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:35:55.294+0000 7fb345a40740 1 -- 172.21.15.52:0/4215610992 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x55da05c347d0 con 0x55da059311b0 2024-05-27T07:35:55.291 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:35:55.294+0000 7fb345a40740 1 -- 172.21.15.52:0/4215610992 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x55da05c38810 con 0x55da059082f0 2024-05-27T07:35:55.291 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:35:55.294+0000 7fb33c922640 1 -- 172.21.15.52:0/4215610992 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55da059082f0 msgr2=0x55da05cd2910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:35:55.291 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:35:55.294+0000 7fb33c922640 1 --2- 172.21.15.52:0/4215610992 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55da059082f0 0x55da05cd2910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:35:55.291 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:35:55.294+0000 7fb33c922640 1 -- 172.21.15.52:0/4215610992 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55da059104d0 msgr2=0x55da05cd2e50 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-05-27T07:35:55.291 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:35:55.294+0000 7fb33c922640 1 --2- 172.21.15.52:0/4215610992 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55da059104d0 0x55da05cd2e50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:35:55.291 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:35:55.294+0000 7fb33c922640 1 -- 172.21.15.52:0/4215610992 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55da05949210 con 0x55da059311b0 2024-05-27T07:35:55.291 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:35:55.294+0000 7fb337fff640 1 --2- 172.21.15.52:0/4215610992 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55da059082f0 0x55da05cd2910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-27T07:35:55.291 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:35:55.295+0000 7fb33c922640 1 --2- 172.21.15.52:0/4215610992 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55da059311b0 0x55da05cd23d0 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7fb32c0c54c0 tx=0x7fb32c005c90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=3116462a9658bd6f server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:35:55.292 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:35:55.295+0000 7fb322ffd640 1 -- 172.21.15.52:0/4215610992 <== mon.2 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb32c0c9440 con 0x55da059311b0 2024-05-27T07:35:55.292 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:35:55.295+0000 7fb322ffd640 1 -- 172.21.15.52:0/4215610992 <== mon.2 v2:172.21.15.105:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fb32c0cd7f0 con 0x55da059311b0 2024-05-27T07:35:55.293 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:35:55.295+0000 7fb322ffd640 1 -- 172.21.15.52:0/4215610992 <== mon.2 v2:172.21.15.105:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb32c0d1e10 con 0x55da059311b0 2024-05-27T07:35:55.293 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:35:55.296+0000 7fb345a40740 1 -- 172.21.15.52:0/4215610992 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55da059311b0 msgr2=0x55da05cd23d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:35:55.293 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:35:55.296+0000 7fb345a40740 1 --2- 172.21.15.52:0/4215610992 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55da059311b0 0x55da05cd23d0 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7fb32c0c54c0 tx=0x7fb32c005c90 comp rx=0 tx=0).stop 2024-05-27T07:35:55.293 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:35:55.296+0000 7fb345a40740 1 -- 172.21.15.52:0/4215610992 shutdown_connections 2024-05-27T07:35:55.293 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:35:55.296+0000 7fb33d123640 1 --2- 172.21.15.52:0/4223507819 >> [v2:172.21.15.52:6800/1822318384,v1:172.21.15.52:6801/1822318384] conn(0x7fb3280b29d0 0x7fb3280b4ee0 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7fb330009270 tx=0x7fb330009680 comp rx=0 tx=0).ready entity=mgr.15333 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:35:55.293 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:35:55.296+0000 7fb345a40740 1 --2- 172.21.15.52:0/4215610992 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55da059104d0 0x55da05cd2e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:35:55.293 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:35:55.296+0000 7fb345a40740 1 --2- 172.21.15.52:0/4215610992 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55da059082f0 0x55da05cd2910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:35:55.293 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:35:55.296+0000 7fb345a40740 1 --2- 172.21.15.52:0/4215610992 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55da059311b0 0x55da05cd23d0 unknown :-1 s=CLOSED pgs=257 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:35:55.293 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:35:55.297+0000 7fb345a40740 1 -- 172.21.15.52:0/4215610992 >> 172.21.15.52:0/4215610992 conn(0x55da05925300 msgr2=0x55da05cceb80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:35:55.293 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:35:55.297+0000 7fb345a40740 1 -- 172.21.15.52:0/4215610992 shutdown_connections 2024-05-27T07:35:55.293 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:35:55.297+0000 7fb345a40740 1 -- 172.21.15.52:0/4215610992 wait complete. 2024-05-27T07:35:55.293 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:35:55.297+0000 7fb345a40740 1 -- start start 2024-05-27T07:35:55.293 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:35:55.297+0000 7fb345a40740 1 Processor -- start 2024-05-27T07:35:55.295 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:35:55.298+0000 7fb345a40740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55da059082f0 0x55da05cde530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:35:55.295 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:35:55.298+0000 7fb345a40740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55da059104d0 0x55da05cdea70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:35:55.295 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:35:55.298+0000 7fb345a40740 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55da059311b0 0x55da05cdc9b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:35:55.295 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:35:55.298+0000 7fb345a40740 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x55da0591a790 con 0x55da059311b0 2024-05-27T07:35:55.295 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:35:55.298+0000 7fb345a40740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x55da05bf0890 con 0x55da059082f0 2024-05-27T07:35:55.295 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:35:55.298+0000 7fb345a40740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x55da05c347d0 con 0x55da059104d0 2024-05-27T07:35:55.295 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:35:55.298+0000 7fb33c922640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55da059082f0 0x55da05cde530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:35:55.296 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:35:55.298+0000 7fb33c922640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55da059082f0 0x55da05cde530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.52:34968/0 (socket says 172.21.15.52:34968) 2024-05-27T07:35:55.296 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:35:55.298+0000 7fb33c922640 1 -- 172.21.15.52:0/3624820127 learned_addr learned my addr 172.21.15.52:0/3624820127 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-05-27T07:35:55.296 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:35:55.298+0000 7fb345a40740 10 client.0 _ll_register_callbacks cb 0x55da05948a00 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T07:35:55.296 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:35:55.298+0000 7fb337fff640 1 --2- 172.21.15.52:0/3624820127 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55da059104d0 0x55da05cdea70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:35:55.296 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:35:55.298+0000 7fb33c922640 1 -- 172.21.15.52:0/3624820127 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55da059104d0 msgr2=0x55da05cdea70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:35:55.296 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:35:55.298+0000 7fb33c922640 1 --2- 172.21.15.52:0/3624820127 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55da059104d0 0x55da05cdea70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:35:55.296 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:35:55.298+0000 7fb33c922640 1 -- 172.21.15.52:0/3624820127 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55da059311b0 msgr2=0x55da05cdc9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:35:55.296 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:35:55.298+0000 7fb33c922640 1 --2- 172.21.15.52:0/3624820127 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55da059311b0 0x55da05cdc9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:35:55.296 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:35:55.298+0000 7fb33c922640 1 -- 172.21.15.52:0/3624820127 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55da05a15620 con 0x55da059082f0 2024-05-27T07:35:55.296 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:35:55.299+0000 7fb33c922640 1 --2- 172.21.15.52:0/3624820127 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55da059082f0 0x55da05cde530 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7fb32c0c5470 tx=0x7fb32c0c5960 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c2ec5d414c0e9038 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:35:55.296 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:35:55.299+0000 7fb317fff640 10 client.0 ms_handle_connect on v2:172.21.15.105:3300/0 2024-05-27T07:35:55.296 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:35:55.299+0000 7fb317fff640 1 -- 172.21.15.52:0/3624820127 <== mon.2 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb32c0e3060 con 0x55da059082f0 2024-05-27T07:35:55.297 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:35:55.299+0000 7fb317fff640 1 -- 172.21.15.52:0/3624820127 <== mon.2 v2:172.21.15.105:3300/0 2 ==== osd_map(124..124 src has 1..124) ==== 9479+0+0 (secure 0 0 0) 0x7fb32c0d2020 con 0x55da059082f0 2024-05-27T07:35:55.297 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:35:55.299+0000 7fb345a40740 1 -- 172.21.15.52:0/3624820127 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fb32c0cd7f0 con 0x55da059082f0 2024-05-27T07:35:55.297 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:35:55.299+0000 7fb317fff640 1 -- 172.21.15.52:0/3624820127 <== mon.2 v2:172.21.15.105:3300/0 3 ==== mon_get_version_reply(handle=1 version=108) ==== 24+0+0 (secure 0 0 0) 0x7fb32c0c9440 con 0x55da059082f0 2024-05-27T07:35:55.297 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:35:55.299+0000 7fb345a40740 10 client.26867 fetch_fsmap learned FSMap version 108 2024-05-27T07:35:55.297 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:35:55.299+0000 7fb345a40740 1 -- 172.21.15.52:0/3624820127 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,fsmap=108,monmap=4+}) -- 0x55da05ceb350 con 0x55da059082f0 2024-05-27T07:35:55.299 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:35:55.300+0000 7fb317fff640 1 -- 172.21.15.52:0/3624820127 <== mon.2 v2:172.21.15.105:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fb32c0cd7f0 con 0x55da059082f0 2024-05-27T07:35:55.299 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:35:55.300+0000 7fb317fff640 1 -- 172.21.15.52:0/3624820127 <== mon.2 v2:172.21.15.105:3300/0 5 ==== fsmap(e 108) ==== 4203+0+0 (secure 0 0 0) 0x7fb32c0a3b00 con 0x55da059082f0 2024-05-27T07:35:55.299 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:35:55.300+0000 7fb345a40740 10 client.26867 fetch_fsmap finished waiting for FSMap version 108 2024-05-27T07:35:55.299 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:35:55.300+0000 7fb345a40740 10 client.26867 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-05-27T07:35:55.299 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:35:55.300+0000 7fb345a40740 20 client.26867 populate_metadata read hostname 'smithi052' 2024-05-27T07:35:55.299 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:35:55.300+0000 7fb345a40740 1 --2- 172.21.15.52:0/3624820127 >> [v2:172.21.15.105:6838/500348151,v1:172.21.15.105:6839/500348151] conn(0x55da058efd80 0x55da05cee7d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:35:55.299 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:35:55.300+0000 7fb345a40740 4 client.26867 mds_command: new command op to 24527 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-27T07:35:55.299 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:35:55.300+0000 7fb345a40740 1 -- 172.21.15.52:0/3624820127 --> [v2:172.21.15.105:6838/500348151,v1:172.21.15.105:6839/500348151] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55da05c46b80 con 0x55da058efd80 2024-05-27T07:35:55.300 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:35:55.301+0000 7fb337fff640 1 --2- 172.21.15.52:0/3624820127 >> [v2:172.21.15.105:6838/500348151,v1:172.21.15.105:6839/500348151] conn(0x55da058efd80 0x55da05cee7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:35:55.300 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:35:55.302+0000 7fb337fff640 1 --2- 172.21.15.52:0/3624820127 >> [v2:172.21.15.105:6838/500348151,v1:172.21.15.105:6839/500348151] conn(0x55da058efd80 0x55da05cee7d0 crc :-1 s=READY pgs=175 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=457a728b3e038b0b server_cookie=9e9b65c5de022b78 in_seq=0 out_seq=0 2024-05-27T07:35:55.300 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:35:55.302+0000 7fb317fff640 10 client.26867 ms_handle_connect on v2:172.21.15.105:6838/500348151 2024-05-27T07:35:55.300 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:35:55.302+0000 7fb317fff640 1 -- 172.21.15.52:0/3624820127 <== mds.0 v2:172.21.15.105:6838/500348151 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24041 (crc 0 0 0) 0x55da05c46b80 con 0x55da058efd80 2024-05-27T07:35:55.300 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:35:55.302+0000 7fb317fff640 10 client.26867 handle_command_reply: tid=0 2024-05-27T07:35:55.300 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:35:55.302+0000 7fb345a40740 1 client.26867 shutdown 2024-05-27T07:35:55.300 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:35:55.303+0000 7fb345a40740 1 -- 172.21.15.52:0/3624820127 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55da059082f0 msgr2=0x55da05cde530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:35:55.300 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:35:55.303+0000 7fb345a40740 1 --2- 172.21.15.52:0/3624820127 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55da059082f0 0x55da05cde530 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7fb32c0c5470 tx=0x7fb32c0c5960 comp rx=0 tx=0).stop 2024-05-27T07:35:55.300 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:35:55.303+0000 7fb345a40740 1 -- 172.21.15.52:0/3624820127 shutdown_connections 2024-05-27T07:35:55.300 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:35:55.303+0000 7fb345a40740 1 --2- 172.21.15.52:0/3624820127 >> [v2:172.21.15.105:6838/500348151,v1:172.21.15.105:6839/500348151] conn(0x55da058efd80 0x55da05cee7d0 crc :-1 s=READY pgs=175 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:35:55.300 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:35:55.303+0000 7fb345a40740 1 --2- 172.21.15.52:0/3624820127 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55da059311b0 0x55da05cdc9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:35:55.300 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:35:55.303+0000 7fb317fff640 0 client.26867 ms_handle_reset on v2:172.21.15.105:6838/500348151 2024-05-27T07:35:55.301 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:35:55.303+0000 7fb345a40740 1 --2- 172.21.15.52:0/3624820127 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55da059104d0 0x55da05cdea70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:35:55.301 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:35:55.303+0000 7fb345a40740 1 --2- 172.21.15.52:0/3624820127 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55da059082f0 0x55da05cde530 unknown :-1 s=CLOSED pgs=258 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:35:55.301 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:35:55.303+0000 7fb345a40740 1 -- 172.21.15.52:0/3624820127 >> 172.21.15.52:0/3624820127 conn(0x55da05925300 msgr2=0x55da05cd40f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:35:55.301 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:35:55.303+0000 7fb345a40740 1 -- 172.21.15.52:0/3624820127 shutdown_connections 2024-05-27T07:35:55.302 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:35:55.305+0000 7fb345a40740 1 -- 172.21.15.52:0/3624820127 wait complete. 2024-05-27T07:35:55.302 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:35:55.305+0000 7fb345a40740 20 client.26867 trim_cache size 0 max 16384 2024-05-27T07:35:55.327 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:35:55.331+0000 7fb345a40740 1 Processor -- start 2024-05-27T07:35:55.327 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:35:55.331+0000 7fb345a40740 1 -- start start 2024-05-27T07:35:55.327 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:35:55.332+0000 7fb345a40740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55da058efd80 0x55da05cd4690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:35:55.328 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:35:55.332+0000 7fb345a40740 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55da059082f0 0x55da05cd6080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:35:55.328 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:35:55.332+0000 7fb345a40740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55da059104d0 0x55da05cd65c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:35:55.328 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:35:55.332+0000 7fb345a40740 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x7fb3381b7ee0 con 0x55da059082f0 2024-05-27T07:35:55.328 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:35:55.332+0000 7fb345a40740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7fb3381b6310 con 0x55da058efd80 2024-05-27T07:35:55.328 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:35:55.332+0000 7fb345a40740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7fb3381b77e0 con 0x55da059104d0 2024-05-27T07:35:55.328 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:35:55.332+0000 7fb33c922640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55da058efd80 0x55da05cd4690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:35:55.328 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:35:55.332+0000 7fb337fff640 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55da059082f0 0x55da05cd6080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:35:55.328 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:35:55.332+0000 7fb33c922640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55da058efd80 0x55da05cd4690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.52:34974/0 (socket says 172.21.15.52:34974) 2024-05-27T07:35:55.328 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:35:55.332+0000 7fb33c922640 1 -- 172.21.15.52:0/3541668089 learned_addr learned my addr 172.21.15.52:0/3541668089 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-05-27T07:35:55.328 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:35:55.332+0000 7fb33d123640 1 --2- 172.21.15.52:0/3541668089 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55da059104d0 0x55da05cd65c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:35:55.329 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:35:55.333+0000 7fb33c922640 1 -- 172.21.15.52:0/3541668089 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55da059104d0 msgr2=0x55da05cd65c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:35:55.329 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:35:55.333+0000 7fb33c922640 1 --2- 172.21.15.52:0/3541668089 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55da059104d0 0x55da05cd65c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:35:55.329 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:35:55.333+0000 7fb33c922640 1 -- 172.21.15.52:0/3541668089 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55da059082f0 msgr2=0x55da05cd6080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:35:55.329 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:35:55.333+0000 7fb33c922640 1 --2- 172.21.15.52:0/3541668089 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55da059082f0 0x55da05cd6080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:35:55.330 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:35:55.333+0000 7fb33c922640 1 -- 172.21.15.52:0/3541668089 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55da05ccf480 con 0x55da058efd80 2024-05-27T07:35:55.330 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:35:55.333+0000 7fb337fff640 1 --2- 172.21.15.52:0/3541668089 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55da059082f0 0x55da05cd6080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-27T07:35:55.330 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:35:55.333+0000 7fb33c922640 1 --2- 172.21.15.52:0/3541668089 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55da058efd80 0x55da05cd4690 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7fb32c0c93d0 tx=0x7fb32c005c90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=3d7e712ffa790a40 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:35:55.330 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:35:55.333+0000 7fb3227fc640 1 -- 172.21.15.52:0/3541668089 <== mon.2 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb32c0e3040 con 0x55da058efd80 2024-05-27T07:35:55.330 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:35:55.333+0000 7fb3227fc640 1 -- 172.21.15.52:0/3541668089 <== mon.2 v2:172.21.15.105:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fb32c0cd7f0 con 0x55da058efd80 2024-05-27T07:35:55.330 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:35:55.333+0000 7fb3227fc640 1 -- 172.21.15.52:0/3541668089 <== mon.2 v2:172.21.15.105:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb32c0c4ff0 con 0x55da058efd80 2024-05-27T07:35:55.331 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:35:55.335+0000 7fb345a40740 1 -- 172.21.15.52:0/3541668089 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55da058efd80 msgr2=0x55da05cd4690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:35:55.331 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:35:55.335+0000 7fb345a40740 1 --2- 172.21.15.52:0/3541668089 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55da058efd80 0x55da05cd4690 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7fb32c0c93d0 tx=0x7fb32c005c90 comp rx=0 tx=0).stop 2024-05-27T07:35:55.331 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:35:55.335+0000 7fb345a40740 1 -- 172.21.15.52:0/3541668089 shutdown_connections 2024-05-27T07:35:55.331 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:35:55.335+0000 7fb345a40740 1 --2- 172.21.15.52:0/3541668089 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55da059104d0 0x55da05cd65c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:35:55.331 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:35:55.335+0000 7fb345a40740 1 --2- 172.21.15.52:0/3541668089 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55da059082f0 0x55da05cd6080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:35:55.331 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:35:55.335+0000 7fb345a40740 1 --2- 172.21.15.52:0/3541668089 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55da058efd80 0x55da05cd4690 unknown :-1 s=CLOSED pgs=259 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:35:55.331 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:35:55.335+0000 7fb345a40740 1 -- 172.21.15.52:0/3541668089 >> 172.21.15.52:0/3541668089 conn(0x55da05925300 msgr2=0x55da05cd3f80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:35:55.332 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:35:55.336+0000 7fb345a40740 1 -- 172.21.15.52:0/3541668089 shutdown_connections 2024-05-27T07:35:55.332 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:35:55.336+0000 7fb345a40740 1 -- 172.21.15.52:0/3541668089 wait complete. 2024-05-27T07:35:55.332 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:35:55.337+0000 7fb345a40740 1 -- start start 2024-05-27T07:35:55.332 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:35:55.337+0000 7fb345a40740 1 Processor -- start 2024-05-27T07:35:55.333 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:35:55.338+0000 7fb345a40740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55da058efd80 0x55da05ce5880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:35:55.333 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:35:55.338+0000 7fb345a40740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55da059082f0 0x55da05ce5dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:35:55.334 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:35:55.338+0000 7fb33c922640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55da058efd80 0x55da05ce5880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:35:55.334 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:35:55.338+0000 7fb33c922640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55da058efd80 0x55da05ce5880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.173:3300/0 says I am v2:172.21.15.52:38706/0 (socket says 172.21.15.52:38706) 2024-05-27T07:35:55.334 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:35:55.338+0000 7fb33c922640 1 -- 172.21.15.52:0/397400605 learned_addr learned my addr 172.21.15.52:0/397400605 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-05-27T07:35:55.334 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:35:55.338+0000 7fb337fff640 1 --2- 172.21.15.52:0/397400605 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55da059082f0 0x55da05ce5dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:35:55.335 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:35:55.338+0000 7fb345a40740 1 --2- 172.21.15.52:0/397400605 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55da059104d0 0x55da05ce6300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:35:55.335 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:35:55.339+0000 7fb345a40740 1 -- 172.21.15.52:0/397400605 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x7fb3381b7110 con 0x55da059104d0 2024-05-27T07:35:55.335 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:35:55.339+0000 7fb345a40740 1 -- 172.21.15.52:0/397400605 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7fb3381b6a10 con 0x55da059082f0 2024-05-27T07:35:55.335 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:35:55.339+0000 7fb345a40740 1 -- 172.21.15.52:0/397400605 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x55da05cd37b0 con 0x55da058efd80 2024-05-27T07:35:55.336 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:35:55.339+0000 7fb345a40740 10 client.0 _ll_register_callbacks cb 0x7fb32c0c8d90 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T07:35:55.336 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:35:55.339+0000 7fb33c922640 1 -- 172.21.15.52:0/397400605 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55da059082f0 msgr2=0x55da05ce5dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:35:55.336 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:35:55.339+0000 7fb33c922640 1 --2- 172.21.15.52:0/397400605 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55da059082f0 0x55da05ce5dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:35:55.336 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:35:55.339+0000 7fb33c922640 1 -- 172.21.15.52:0/397400605 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55da059104d0 msgr2=0x55da05ce6300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:35:55.336 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:35:55.339+0000 7fb33c922640 1 --2- 172.21.15.52:0/397400605 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55da059104d0 0x55da05ce6300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:35:55.337 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:35:55.339+0000 7fb33c922640 1 -- 172.21.15.52:0/397400605 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55da05d550a0 con 0x55da058efd80 2024-05-27T07:35:55.337 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:35:55.340+0000 7fb33c922640 1 --2- 172.21.15.52:0/397400605 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55da058efd80 0x55da05ce5880 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7fb32c0c9140 tx=0x7fb32c030870 comp rx=0 tx=0).ready entity=mon.1 client_cookie=c7c91c0d2980202d server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:35:55.337 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:35:55.340+0000 7fb345a40740 1 -- 172.21.15.52:0/397400605 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fb32c0cd7f0 con 0x55da058efd80 2024-05-27T07:35:55.337 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:35:55.340+0000 7fb317fff640 10 client.26701 ms_handle_connect on v2:172.21.15.173:3300/0 2024-05-27T07:35:55.337 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:35:55.340+0000 7fb317fff640 1 -- 172.21.15.52:0/397400605 <== mon.1 v2:172.21.15.173:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb32c0c5300 con 0x55da058efd80 2024-05-27T07:35:55.337 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:35:55.340+0000 7fb317fff640 1 -- 172.21.15.52:0/397400605 <== mon.1 v2:172.21.15.173:3300/0 2 ==== osd_map(124..124 src has 1..124) ==== 9479+0+0 (secure 0 0 0) 0x7fb32c0d0b70 con 0x55da058efd80 2024-05-27T07:35:55.337 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:35:55.341+0000 7fb317fff640 1 -- 172.21.15.52:0/397400605 <== mon.1 v2:172.21.15.173:3300/0 3 ==== mon_get_version_reply(handle=1 version=108) ==== 24+0+0 (secure 0 0 0) 0x7fb32c0d0d70 con 0x55da058efd80 2024-05-27T07:35:55.337 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:35:55.341+0000 7fb345a40740 10 client.26701 fetch_fsmap learned FSMap version 108 2024-05-27T07:35:55.337 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:35:55.341+0000 7fb345a40740 1 -- 172.21.15.52:0/397400605 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({fsmap=108}) -- 0x55da05d550a0 con 0x55da058efd80 2024-05-27T07:35:55.338 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:35:55.342+0000 7fb317fff640 1 -- 172.21.15.52:0/397400605 <== mon.1 v2:172.21.15.173:3300/0 4 ==== fsmap(e 108) ==== 4203+0+0 (secure 0 0 0) 0x7fb32c0c86a0 con 0x55da058efd80 2024-05-27T07:35:55.338 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:35:55.342+0000 7fb345a40740 10 client.26701 fetch_fsmap finished waiting for FSMap version 108 2024-05-27T07:35:55.338 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:35:55.342+0000 7fb345a40740 10 client.26701 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-05-27T07:35:55.338 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:35:55.342+0000 7fb345a40740 20 client.26701 populate_metadata read hostname 'smithi052' 2024-05-27T07:35:55.339 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:35:55.342+0000 7fb345a40740 1 --2- 172.21.15.52:0/397400605 >> [v2:172.21.15.105:6838/500348151,v1:172.21.15.105:6839/500348151] conn(0x55da059311b0 0x55da05d58210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:35:55.339 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:35:55.342+0000 7fb345a40740 4 client.26701 mds_command: new command op to 24527 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-05-27T07:35:55.339 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:35:55.342+0000 7fb345a40740 1 -- 172.21.15.52:0/397400605 --> [v2:172.21.15.105:6838/500348151,v1:172.21.15.105:6839/500348151] -- command(tid 0: {"prefix": "flush journal"}) -- 0x7fb32c0cd7f0 con 0x55da059311b0 2024-05-27T07:35:55.339 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:35:55.342+0000 7fb337fff640 1 --2- 172.21.15.52:0/397400605 >> [v2:172.21.15.105:6838/500348151,v1:172.21.15.105:6839/500348151] conn(0x55da059311b0 0x55da05d58210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:35:55.346 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:35:55.350+0000 7fb337fff640 1 --2- 172.21.15.52:0/397400605 >> [v2:172.21.15.105:6838/500348151,v1:172.21.15.105:6839/500348151] conn(0x55da059311b0 0x55da05d58210 crc :-1 s=READY pgs=176 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=331d22f9af528059 server_cookie=24ff4e7dd6f79206 in_seq=0 out_seq=0 2024-05-27T07:35:55.346 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:35:55.350+0000 7fb317fff640 10 client.26701 ms_handle_connect on v2:172.21.15.105:6838/500348151 2024-05-27T07:35:55.816 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:35:55.818+0000 7fb317fff640 1 -- 172.21.15.52:0/397400605 <== mds.0 v2:172.21.15.105:6838/500348151 1 ==== command_reply(tid 0: 0 ) ==== 8+0+25 (crc 0 0 0) 0x7fb32c0cd7f0 con 0x55da059311b0 2024-05-27T07:35:55.816 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:35:55.819+0000 7fb317fff640 10 client.26701 handle_command_reply: tid=0 2024-05-27T07:35:55.816 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:35:55.819+0000 7fb345a40740 1 client.26701 shutdown 2024-05-27T07:35:55.817 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:35:55.819+0000 7fb345a40740 1 -- 172.21.15.52:0/397400605 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55da058efd80 msgr2=0x55da05ce5880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:35:55.817 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:35:55.819+0000 7fb345a40740 1 --2- 172.21.15.52:0/397400605 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55da058efd80 0x55da05ce5880 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7fb32c0c9140 tx=0x7fb32c030870 comp rx=0 tx=0).stop 2024-05-27T07:35:55.817 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:35:55.819+0000 7fb345a40740 1 -- 172.21.15.52:0/397400605 shutdown_connections 2024-05-27T07:35:55.817 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:35:55.819+0000 7fb345a40740 1 --2- 172.21.15.52:0/397400605 >> [v2:172.21.15.105:6838/500348151,v1:172.21.15.105:6839/500348151] conn(0x55da059311b0 0x55da05d58210 crc :-1 s=READY pgs=176 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:35:55.817 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:35:55.819+0000 7fb345a40740 1 --2- 172.21.15.52:0/397400605 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55da059104d0 0x55da05ce6300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:35:55.817 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:35:55.819+0000 7fb317fff640 0 client.26701 ms_handle_reset on v2:172.21.15.105:6838/500348151 2024-05-27T07:35:55.817 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:35:55.820+0000 7fb345a40740 1 --2- 172.21.15.52:0/397400605 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55da059082f0 0x55da05ce5dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:35:55.817 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:35:55.820+0000 7fb345a40740 1 --2- 172.21.15.52:0/397400605 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55da058efd80 0x55da05ce5880 unknown :-1 s=CLOSED pgs=216 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:35:55.817 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:35:55.820+0000 7fb345a40740 1 -- 172.21.15.52:0/397400605 >> 172.21.15.52:0/397400605 conn(0x55da05925300 msgr2=0x55da05cde060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:35:55.817 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:35:55.820+0000 7fb345a40740 1 -- 172.21.15.52:0/397400605 shutdown_connections 2024-05-27T07:35:55.817 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:35:55.820+0000 7fb345a40740 1 -- 172.21.15.52:0/397400605 wait complete. 2024-05-27T07:35:55.817 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:35:55.820+0000 7fb345a40740 20 client.26701 trim_cache size 0 max 16384 2024-05-27T07:35:55.818 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stdout:{ 2024-05-27T07:35:55.818 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stdout: "return_code": 0 2024-05-27T07:35:55.818 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stdout:} 2024-05-27T07:35:55.819 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:35:55.821+0000 7fb33eb86640 1 -- 172.21.15.52:0/4223507819 >> [v2:172.21.15.52:6800/1822318384,v1:172.21.15.52:6801/1822318384] conn(0x7fb3280b29d0 msgr2=0x7fb3280b4ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:35:55.819 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:35:55.821+0000 7fb33eb86640 1 --2- 172.21.15.52:0/4223507819 >> [v2:172.21.15.52:6800/1822318384,v1:172.21.15.52:6801/1822318384] conn(0x7fb3280b29d0 0x7fb3280b4ee0 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7fb330009270 tx=0x7fb330009680 comp rx=0 tx=0).stop 2024-05-27T07:35:55.819 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:35:55.821+0000 7fb33eb86640 1 -- 172.21.15.52:0/4223507819 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fb33810a500 msgr2=0x7fb3381a0e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:35:55.819 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:35:55.821+0000 7fb33eb86640 1 --2- 172.21.15.52:0/4223507819 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fb33810a500 0x7fb3381a0e60 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7fb32c035040 tx=0x7fb32c032dc0 comp rx=0 tx=0).stop 2024-05-27T07:35:55.819 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:35:55.821+0000 7fb33eb86640 1 -- 172.21.15.52:0/4223507819 shutdown_connections 2024-05-27T07:35:55.819 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:35:55.821+0000 7fb33eb86640 1 --2- 172.21.15.52:0/4223507819 >> [v2:172.21.15.52:6800/1822318384,v1:172.21.15.52:6801/1822318384] conn(0x7fb3280b29d0 0x7fb3280b4ee0 unknown :-1 s=CLOSED pgs=190 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:35:55.819 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:35:55.821+0000 7fb33eb86640 1 --2- 172.21.15.52:0/4223507819 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fb33810f550 0x7fb3381a5680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:35:55.819 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:35:55.821+0000 7fb33eb86640 1 --2- 172.21.15.52:0/4223507819 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fb33810ea90 0x7fb3381a13c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:35:55.819 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:35:55.821+0000 7fb33eb86640 1 --2- 172.21.15.52:0/4223507819 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fb33810a500 0x7fb3381a0e60 unknown :-1 s=CLOSED pgs=215 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:35:55.819 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:35:55.821+0000 7fb33eb86640 1 -- 172.21.15.52:0/4223507819 >> 172.21.15.52:0/4223507819 conn(0x7fb338077190 msgr2=0x7fb338107ae0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:35:55.819 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:35:55.822+0000 7fb33eb86640 1 -- 172.21.15.52:0/4223507819 shutdown_connections 2024-05-27T07:35:55.819 INFO:teuthology.task.background_exec.ubuntu@smithi052.front.sepia.ceph.com.smithi052.stderr:2024-05-27T07:35:55.822+0000 7fb33eb86640 1 -- 172.21.15.52:0/4223507819 wait complete. 2024-05-27T07:35:56.607 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:35:56 smithi173 ceph-mon[37000]: pgmap v335: 129 pgs: 4 active+clean+snaptrim, 6 active+clean+snaptrim_wait, 119 active+clean; 649 MiB data, 6.6 GiB used, 1.0 TiB / 1.0 TiB avail; 185 KiB/s rd, 1.0 MiB/s wr, 585 op/s 2024-05-27T07:35:56.695 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:35:56 smithi105 ceph-mon[35977]: pgmap v335: 129 pgs: 4 active+clean+snaptrim, 6 active+clean+snaptrim_wait, 119 active+clean; 649 MiB data, 6.6 GiB used, 1.0 TiB / 1.0 TiB avail; 185 KiB/s rd, 1.0 MiB/s wr, 585 op/s 2024-05-27T07:35:56.735 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:35:56 smithi052 ceph-mon[31955]: pgmap v335: 129 pgs: 4 active+clean+snaptrim, 6 active+clean+snaptrim_wait, 119 active+clean; 649 MiB data, 6.6 GiB used, 1.0 TiB / 1.0 TiB avail; 185 KiB/s rd, 1.0 MiB/s wr, 585 op/s 2024-05-27T07:35:58.536 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:35:58 smithi052 ceph-mon[31955]: pgmap v336: 129 pgs: 3 active+clean+snaptrim, 126 active+clean; 653 MiB data, 6.6 GiB used, 1.0 TiB / 1.0 TiB avail; 73 KiB/s rd, 1.4 MiB/s wr, 501 op/s 2024-05-27T07:35:58.607 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:35:58 smithi173 ceph-mon[37000]: pgmap v336: 129 pgs: 3 active+clean+snaptrim, 126 active+clean; 653 MiB data, 6.6 GiB used, 1.0 TiB / 1.0 TiB avail; 73 KiB/s rd, 1.4 MiB/s wr, 501 op/s 2024-05-27T07:35:58.695 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:35:58 smithi105 ceph-mon[35977]: pgmap v336: 129 pgs: 3 active+clean+snaptrim, 126 active+clean; 653 MiB data, 6.6 GiB used, 1.0 TiB / 1.0 TiB avail; 73 KiB/s rd, 1.4 MiB/s wr, 501 op/s 2024-05-27T07:35:59.198 INFO:tasks.workunit:Stopping ['suites/blogbench.sh'] on client.0... 2024-05-27T07:35:59.198 DEBUG:teuthology.orchestra.run.smithi052:> sudo rm -rf -- /home/ubuntu/cephtest/workunits.list.client.0 /home/ubuntu/cephtest/clone.client.0 2024-05-27T07:35:59.667 DEBUG:teuthology.parallel:result is None 2024-05-27T07:35:59.668 DEBUG:teuthology.orchestra.run.smithi052:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0 2024-05-27T07:35:59.735 INFO:tasks.workunit:Deleted dir /home/ubuntu/cephtest/mnt.0/client.0 2024-05-27T07:35:59.736 DEBUG:teuthology.run_tasks:Unwinding manager background_exec 2024-05-27T07:35:59.748 INFO:teuthology.task.background_exec:Stopping background command on ubuntu@smithi052.front.sepia.ceph.com 2024-05-27T07:35:59.760 DEBUG:teuthology.run_tasks:Unwinding manager full_sequential_finally 2024-05-27T07:35:59.774 INFO:teuthology.task.full_sequential_finally:In full_sequential_finally, running task exec... 2024-05-27T07:35:59.774 INFO:teuthology.task.exec:Executing custom commands... 2024-05-27T07:35:59.774 INFO:teuthology.task.exec:Running commands on role mon.a host ubuntu@smithi052.front.sepia.ceph.com 2024-05-27T07:35:59.775 DEBUG:teuthology.orchestra.run.smithi052:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'date +%s > END_TIME' 2024-05-27T07:35:59.850 DEBUG:teuthology.orchestra.run.smithi052:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'START_TIME=$(cat START_TIME); END_TIME=$(cat END_TIME); DIFF_TIME=$((600-(END_TIME-START_TIME))); if [ "$DIFF_TIME" -gt 0 ]; then sleep "$DIFF_TIME"; fi' 2024-05-27T07:36:00.607 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:36:00 smithi173 ceph-mon[37000]: pgmap v337: 129 pgs: 1 active+clean+snaptrim, 128 active+clean; 644 MiB data, 6.6 GiB used, 1.0 TiB / 1.0 TiB avail; 31 KiB/s rd, 1.5 MiB/s wr, 565 op/s 2024-05-27T07:36:00.608 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:36:00 smithi173 ceph-mon[37000]: osdmap e125: 12 total, 12 up, 12 in 2024-05-27T07:36:00.695 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:36:00 smithi105 ceph-mon[35977]: pgmap v337: 129 pgs: 1 active+clean+snaptrim, 128 active+clean; 644 MiB data, 6.6 GiB used, 1.0 TiB / 1.0 TiB avail; 31 KiB/s rd, 1.5 MiB/s wr, 565 op/s 2024-05-27T07:36:00.696 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:36:00 smithi105 ceph-mon[35977]: osdmap e125: 12 total, 12 up, 12 in 2024-05-27T07:36:00.735 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:36:00 smithi052 ceph-mon[31955]: pgmap v337: 129 pgs: 1 active+clean+snaptrim, 128 active+clean; 644 MiB data, 6.6 GiB used, 1.0 TiB / 1.0 TiB avail; 31 KiB/s rd, 1.5 MiB/s wr, 565 op/s 2024-05-27T07:36:00.735 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:36:00 smithi052 ceph-mon[31955]: osdmap e125: 12 total, 12 up, 12 in 2024-05-27T07:36:02.607 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:36:02 smithi173 ceph-mon[37000]: pgmap v339: 129 pgs: 1 active+clean+snaptrim, 128 active+clean; 637 MiB data, 6.6 GiB used, 1.0 TiB / 1.0 TiB avail; 25 KiB/s rd, 1.4 MiB/s wr, 588 op/s 2024-05-27T07:36:02.696 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:36:02 smithi105 ceph-mon[35977]: pgmap v339: 129 pgs: 1 active+clean+snaptrim, 128 active+clean; 637 MiB data, 6.6 GiB used, 1.0 TiB / 1.0 TiB avail; 25 KiB/s rd, 1.4 MiB/s wr, 588 op/s 2024-05-27T07:36:02.735 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:36:02 smithi052 ceph-mon[31955]: pgmap v339: 129 pgs: 1 active+clean+snaptrim, 128 active+clean; 637 MiB data, 6.6 GiB used, 1.0 TiB / 1.0 TiB avail; 25 KiB/s rd, 1.4 MiB/s wr, 588 op/s 2024-05-27T07:36:04.608 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:36:04 smithi173 ceph-mon[37000]: pgmap v340: 129 pgs: 129 active+clean; 640 MiB data, 6.7 GiB used, 1.0 TiB / 1.0 TiB avail; 36 KiB/s rd, 1.4 MiB/s wr, 637 op/s 2024-05-27T07:36:04.696 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:36:04 smithi105 ceph-mon[35977]: pgmap v340: 129 pgs: 129 active+clean; 640 MiB data, 6.7 GiB used, 1.0 TiB / 1.0 TiB avail; 36 KiB/s rd, 1.4 MiB/s wr, 637 op/s 2024-05-27T07:36:04.735 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:36:04 smithi052 ceph-mon[31955]: pgmap v340: 129 pgs: 129 active+clean; 640 MiB data, 6.7 GiB used, 1.0 TiB / 1.0 TiB avail; 36 KiB/s rd, 1.4 MiB/s wr, 637 op/s 2024-05-27T07:36:06.445 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:36:06 smithi105 ceph-mon[35977]: osdmap e126: 12 total, 12 up, 12 in 2024-05-27T07:36:06.446 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:36:06 smithi105 ceph-mon[35977]: pgmap v342: 129 pgs: 129 active+clean; 640 MiB data, 6.7 GiB used, 1.0 TiB / 1.0 TiB avail; 33 KiB/s rd, 1.1 MiB/s wr, 683 op/s 2024-05-27T07:36:06.485 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:36:06 smithi052 ceph-mon[31955]: osdmap e126: 12 total, 12 up, 12 in 2024-05-27T07:36:06.485 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:36:06 smithi052 ceph-mon[31955]: pgmap v342: 129 pgs: 129 active+clean; 640 MiB data, 6.7 GiB used, 1.0 TiB / 1.0 TiB avail; 33 KiB/s rd, 1.1 MiB/s wr, 683 op/s 2024-05-27T07:36:06.607 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:36:06 smithi173 ceph-mon[37000]: osdmap e126: 12 total, 12 up, 12 in 2024-05-27T07:36:06.608 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:36:06 smithi173 ceph-mon[37000]: pgmap v342: 129 pgs: 129 active+clean; 640 MiB data, 6.7 GiB used, 1.0 TiB / 1.0 TiB avail; 33 KiB/s rd, 1.1 MiB/s wr, 683 op/s 2024-05-27T07:36:07.445 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:36:07 smithi105 ceph-mon[35977]: osdmap e127: 12 total, 12 up, 12 in 2024-05-27T07:36:07.485 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:36:07 smithi052 ceph-mon[31955]: osdmap e127: 12 total, 12 up, 12 in 2024-05-27T07:36:07.608 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:36:07 smithi173 ceph-mon[37000]: osdmap e127: 12 total, 12 up, 12 in 2024-05-27T07:36:08.445 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:36:08 smithi105 ceph-mon[35977]: pgmap v344: 129 pgs: 6 active+clean+snaptrim, 22 active+clean+snaptrim_wait, 101 active+clean; 640 MiB data, 6.7 GiB used, 1.0 TiB / 1.0 TiB avail; 23 KiB/s rd, 645 KiB/s wr, 555 op/s 2024-05-27T07:36:08.485 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:36:08 smithi052 ceph-mon[31955]: pgmap v344: 129 pgs: 6 active+clean+snaptrim, 22 active+clean+snaptrim_wait, 101 active+clean; 640 MiB data, 6.7 GiB used, 1.0 TiB / 1.0 TiB avail; 23 KiB/s rd, 645 KiB/s wr, 555 op/s 2024-05-27T07:36:08.607 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:36:08 smithi173 ceph-mon[37000]: pgmap v344: 129 pgs: 6 active+clean+snaptrim, 22 active+clean+snaptrim_wait, 101 active+clean; 640 MiB data, 6.7 GiB used, 1.0 TiB / 1.0 TiB avail; 23 KiB/s rd, 645 KiB/s wr, 555 op/s 2024-05-27T07:36:10.607 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:36:10 smithi173 ceph-mon[37000]: pgmap v345: 129 pgs: 18 active+clean+snaptrim, 54 active+clean+snaptrim_wait, 57 active+clean; 640 MiB data, 6.7 GiB used, 1.0 TiB / 1.0 TiB avail; 21 KiB/s rd, 446 KiB/s wr, 456 op/s 2024-05-27T07:36:10.695 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:36:10 smithi105 ceph-mon[35977]: pgmap v345: 129 pgs: 18 active+clean+snaptrim, 54 active+clean+snaptrim_wait, 57 active+clean; 640 MiB data, 6.7 GiB used, 1.0 TiB / 1.0 TiB avail; 21 KiB/s rd, 446 KiB/s wr, 456 op/s 2024-05-27T07:36:10.735 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:36:10 smithi052 ceph-mon[31955]: pgmap v345: 129 pgs: 18 active+clean+snaptrim, 54 active+clean+snaptrim_wait, 57 active+clean; 640 MiB data, 6.7 GiB used, 1.0 TiB / 1.0 TiB avail; 21 KiB/s rd, 446 KiB/s wr, 456 op/s 2024-05-27T07:36:12.608 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:36:12 smithi173 ceph-mon[37000]: pgmap v346: 129 pgs: 24 active+clean+snaptrim, 72 active+clean+snaptrim_wait, 33 active+clean; 639 MiB data, 6.7 GiB used, 1.0 TiB / 1.0 TiB avail; 7.1 KiB/s rd, 53 KiB/s wr, 260 op/s 2024-05-27T07:36:12.699 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:36:12 smithi105 ceph-mon[35977]: pgmap v346: 129 pgs: 24 active+clean+snaptrim, 72 active+clean+snaptrim_wait, 33 active+clean; 639 MiB data, 6.7 GiB used, 1.0 TiB / 1.0 TiB avail; 7.1 KiB/s rd, 53 KiB/s wr, 260 op/s 2024-05-27T07:36:12.735 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:36:12 smithi052 ceph-mon[31955]: pgmap v346: 129 pgs: 24 active+clean+snaptrim, 72 active+clean+snaptrim_wait, 33 active+clean; 639 MiB data, 6.7 GiB used, 1.0 TiB / 1.0 TiB avail; 7.1 KiB/s rd, 53 KiB/s wr, 260 op/s 2024-05-27T07:36:12.901 DEBUG:teuthology.orchestra.run.smithi052:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'ceph fs snap-schedule status --fs=cephfs --path=/' 2024-05-27T07:36:13.102 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:13.105+0000 7f6f73fff640 1 -- 172.21.15.52:0/3250096540 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f6f6c09a360 msgr2=0x7f6f6c09a770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:36:13.102 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:13.105+0000 7f6f73fff640 1 --2- 172.21.15.52:0/3250096540 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f6f6c09a360 0x7f6f6c09a770 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7f6f68009860 tx=0x7f6f6802a410 comp rx=0 tx=0).stop 2024-05-27T07:36:13.111 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:13.110+0000 7f6f73fff640 1 -- 172.21.15.52:0/3250096540 shutdown_connections 2024-05-27T07:36:13.111 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:13.110+0000 7f6f73fff640 1 --2- 172.21.15.52:0/3250096540 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f6f6c09ad00 0x7f6f6c09b1e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:36:13.112 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:13.111+0000 7f6f73fff640 1 --2- 172.21.15.52:0/3250096540 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f6f6c09a360 0x7f6f6c09a770 unknown :-1 s=CLOSED pgs=260 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:36:13.112 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:13.111+0000 7f6f73fff640 1 --2- 172.21.15.52:0/3250096540 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f6f6c095910 0x7f6f6c095d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:36:13.112 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:13.111+0000 7f6f73fff640 1 -- 172.21.15.52:0/3250096540 >> 172.21.15.52:0/3250096540 conn(0x7f6f6c00afc0 msgr2=0x7f6f6c00b400 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:36:13.112 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:13.113+0000 7f6f73fff640 1 -- 172.21.15.52:0/3250096540 shutdown_connections 2024-05-27T07:36:13.112 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:13.113+0000 7f6f73fff640 1 -- 172.21.15.52:0/3250096540 wait complete. 2024-05-27T07:36:13.114 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:13.114+0000 7f6f73fff640 1 Processor -- start 2024-05-27T07:36:13.114 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:13.114+0000 7f6f73fff640 1 -- start start 2024-05-27T07:36:13.114 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:13.114+0000 7f6f73fff640 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f6f6c095910 0x7f6f6c0a9a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:36:13.114 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:13.114+0000 7f6f73fff640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f6f6c09a360 0x7f6f6c0a3b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:36:13.114 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:13.115+0000 7f6f73fff640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f6f6c09ad00 0x7f6f6c0a40b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:36:13.115 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:13.115+0000 7f6f73fff640 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x7f6f6c09c230 con 0x7f6f6c095910 2024-05-27T07:36:13.115 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:13.115+0000 7f6f73fff640 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f6f6c09c0b0 con 0x7f6f6c09a360 2024-05-27T07:36:13.115 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:13.115+0000 7f6f73fff640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f6f6c09c3b0 con 0x7f6f6c09ad00 2024-05-27T07:36:13.115 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:13.115+0000 7f6f727fc640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f6f6c09a360 0x7f6f6c0a3b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:36:13.115 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:13.115+0000 7f6f727fc640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f6f6c09a360 0x7f6f6c0a3b70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.52:35152/0 (socket says 172.21.15.52:35152) 2024-05-27T07:36:13.115 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:13.115+0000 7f6f727fc640 1 -- 172.21.15.52:0/2061593221 learned_addr learned my addr 172.21.15.52:0/2061593221 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-05-27T07:36:13.115 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:13.115+0000 7f6f737fe640 1 --2- 172.21.15.52:0/2061593221 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f6f6c09ad00 0x7f6f6c0a40b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:36:13.115 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:13.116+0000 7f6f727fc640 1 -- 172.21.15.52:0/2061593221 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f6f6c09ad00 msgr2=0x7f6f6c0a40b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:36:13.115 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:13.116+0000 7f6f727fc640 1 --2- 172.21.15.52:0/2061593221 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f6f6c09ad00 0x7f6f6c0a40b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:36:13.115 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:13.116+0000 7f6f727fc640 1 -- 172.21.15.52:0/2061593221 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f6f6c095910 msgr2=0x7f6f6c0a9a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:36:13.115 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:13.116+0000 7f6f727fc640 1 --2- 172.21.15.52:0/2061593221 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f6f6c095910 0x7f6f6c0a9a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:36:13.115 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:13.116+0000 7f6f727fc640 1 -- 172.21.15.52:0/2061593221 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6f6c0a48b0 con 0x7f6f6c09a360 2024-05-27T07:36:13.115 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:13.116+0000 7f6f727fc640 1 --2- 172.21.15.52:0/2061593221 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f6f6c09a360 0x7f6f6c0a3b70 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7f6f68005e00 tx=0x7f6f680050c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:36:13.116 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:13.117+0000 7f6f5bfff640 1 -- 172.21.15.52:0/2061593221 <== mon.2 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6f68062070 con 0x7f6f6c09a360 2024-05-27T07:36:13.116 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:13.117+0000 7f6f5bfff640 1 -- 172.21.15.52:0/2061593221 <== mon.2 v2:172.21.15.105:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f6f680318b0 con 0x7f6f6c09a360 2024-05-27T07:36:13.116 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:13.117+0000 7f6f73fff640 1 -- 172.21.15.52:0/2061593221 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6f6c142660 con 0x7f6f6c09a360 2024-05-27T07:36:13.117 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:13.117+0000 7f6f5bfff640 1 -- 172.21.15.52:0/2061593221 <== mon.2 v2:172.21.15.105:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6f68067460 con 0x7f6f6c09a360 2024-05-27T07:36:13.117 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:13.118+0000 7f6f73fff640 1 -- 172.21.15.52:0/2061593221 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6f6c142b50 con 0x7f6f6c09a360 2024-05-27T07:36:13.118 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:13.122+0000 7f6f59ffb640 1 -- 172.21.15.52:0/2061593221 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6f54005290 con 0x7f6f6c09a360 2024-05-27T07:36:13.124 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:13.129+0000 7f6f5bfff640 1 -- 172.21.15.52:0/2061593221 <== mon.2 v2:172.21.15.105:3300/0 4 ==== mgrmap(e 26) ==== 149783+0+0 (secure 0 0 0) 0x7f6f68039040 con 0x7f6f6c09a360 2024-05-27T07:36:13.130 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:13.134+0000 7f6f5bfff640 1 --2- 172.21.15.52:0/2061593221 >> [v2:172.21.15.52:6800/1822318384,v1:172.21.15.52:6801/1822318384] conn(0x7f6f5c0b2b30 0x7f6f5c0b5040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:36:13.130 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:13.134+0000 7f6f5bfff640 1 -- 172.21.15.52:0/2061593221 <== mon.2 v2:172.21.15.105:3300/0 5 ==== osd_map(127..127 src has 1..127) ==== 9555+0+0 (secure 0 0 0) 0x7f6f68125ee0 con 0x7f6f6c09a360 2024-05-27T07:36:13.130 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:13.135+0000 7f6f5bfff640 1 -- 172.21.15.52:0/2061593221 <== mon.2 v2:172.21.15.105:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191963 (secure 0 0 0) 0x7f6f68126300 con 0x7f6f6c09a360 2024-05-27T07:36:13.135 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:13.138+0000 7f6f72ffd640 1 --2- 172.21.15.52:0/2061593221 >> [v2:172.21.15.52:6800/1822318384,v1:172.21.15.52:6801/1822318384] conn(0x7f6f5c0b2b30 0x7f6f5c0b5040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:36:13.138 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:13.142+0000 7f6f72ffd640 1 --2- 172.21.15.52:0/2061593221 >> [v2:172.21.15.52:6800/1822318384,v1:172.21.15.52:6801/1822318384] conn(0x7f6f5c0b2b30 0x7f6f5c0b5040 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7f6f74066570 tx=0x7f6f74068d40 comp rx=0 tx=0).ready entity=mgr.15333 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:36:13.390 DEBUG:teuthology.orchestra.run.smithi052:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:38f9d900c83c63799fbdbe61acc9a11b0d3554a6 shell --fsid 73df2636-1bf8-11ef-bc9b-c7b262605968 -- ceph tell mds.1:0 scrub status 2024-05-27T07:36:13.549 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:13.553+0000 7f6f59ffb640 1 -- 172.21.15.52:0/2061593221 --> [v2:172.21.15.52:6800/1822318384,v1:172.21.15.52:6801/1822318384] -- mgr_command(tid 0: {"prefix": "fs snap-schedule status", "fs": "cephfs", "path": "/", "target": ["mon-mgr", ""]}) -- 0x7f6f54000bc0 con 0x7f6f5c0b2b30 2024-05-27T07:36:13.639 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:13.643+0000 7f6f5bfff640 1 -- 172.21.15.52:0/2061593221 <== mgr.15333 v2:172.21.15.52:6800/1822318384 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+350 (secure 0 0 0) 0x7f6f54000bc0 con 0x7f6f5c0b2b30 2024-05-27T07:36:13.639 INFO:teuthology.orchestra.run.smithi052.stdout:{"fs": "cephfs", "subvol": null, "group": null, "path": "/", "rel_path": "/", "schedule": "1m", "retention": {"m": 6, "h": 3}, "start": "2024-05-27T00:00:00", "created": "2024-05-27T07:26:11", "first": "2024-05-27T07:27:00", "last": "2024-05-27T07:36:01", "last_pruned": "2024-05-27T07:36:01", "created_count": 10, "pruned_count": 4, "active": true} 2024-05-27T07:36:13.642 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:13.647+0000 7f6f59ffb640 1 -- 172.21.15.52:0/2061593221 >> [v2:172.21.15.52:6800/1822318384,v1:172.21.15.52:6801/1822318384] conn(0x7f6f5c0b2b30 msgr2=0x7f6f5c0b5040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:36:13.643 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:13.647+0000 7f6f59ffb640 1 --2- 172.21.15.52:0/2061593221 >> [v2:172.21.15.52:6800/1822318384,v1:172.21.15.52:6801/1822318384] conn(0x7f6f5c0b2b30 0x7f6f5c0b5040 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7f6f74066570 tx=0x7f6f74068d40 comp rx=0 tx=0).stop 2024-05-27T07:36:13.643 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:13.647+0000 7f6f59ffb640 1 -- 172.21.15.52:0/2061593221 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f6f6c09a360 msgr2=0x7f6f6c0a3b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:36:13.643 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:13.647+0000 7f6f59ffb640 1 --2- 172.21.15.52:0/2061593221 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f6f6c09a360 0x7f6f6c0a3b70 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7f6f68005e00 tx=0x7f6f680050c0 comp rx=0 tx=0).stop 2024-05-27T07:36:13.643 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:13.647+0000 7f6f59ffb640 1 -- 172.21.15.52:0/2061593221 shutdown_connections 2024-05-27T07:36:13.643 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:13.647+0000 7f6f59ffb640 1 --2- 172.21.15.52:0/2061593221 >> [v2:172.21.15.52:6800/1822318384,v1:172.21.15.52:6801/1822318384] conn(0x7f6f5c0b2b30 0x7f6f5c0b5040 unknown :-1 s=CLOSED pgs=191 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:36:13.643 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:13.647+0000 7f6f59ffb640 1 --2- 172.21.15.52:0/2061593221 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f6f6c09ad00 0x7f6f6c0a40b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:36:13.643 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:13.647+0000 7f6f59ffb640 1 --2- 172.21.15.52:0/2061593221 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f6f6c09a360 0x7f6f6c0a3b70 unknown :-1 s=CLOSED pgs=261 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:36:13.643 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:13.647+0000 7f6f59ffb640 1 --2- 172.21.15.52:0/2061593221 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f6f6c095910 0x7f6f6c0a9a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:36:13.643 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:13.647+0000 7f6f59ffb640 1 -- 172.21.15.52:0/2061593221 >> 172.21.15.52:0/2061593221 conn(0x7f6f6c00afc0 msgr2=0x7f6f6c093110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:36:13.644 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:13.647+0000 7f6f59ffb640 1 -- 172.21.15.52:0/2061593221 shutdown_connections 2024-05-27T07:36:13.644 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:13.647+0000 7f6f59ffb640 1 -- 172.21.15.52:0/2061593221 wait complete. 2024-05-27T07:36:13.667 DEBUG:teuthology.orchestra.run.smithi052:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'ceph fs snap-schedule list --fs=cephfs --path=/ --recursive=true' 2024-05-27T07:36:13.919 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:13.923+0000 7fbf2189f640 1 -- 172.21.15.52:0/952264376 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fbf1c11a8e0 msgr2=0x7fbf1c11cd00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:36:13.919 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:13.923+0000 7fbf2189f640 1 --2- 172.21.15.52:0/952264376 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fbf1c11a8e0 0x7fbf1c11cd00 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7fbf0c009860 tx=0x7fbf0c02a410 comp rx=0 tx=0).stop 2024-05-27T07:36:13.927 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:13.928+0000 7fbf2189f640 1 -- 172.21.15.52:0/952264376 shutdown_connections 2024-05-27T07:36:13.927 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:13.928+0000 7fbf2189f640 1 --2- 172.21.15.52:0/952264376 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fbf1c11d240 0x7fbf1c11f660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:36:13.927 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:13.928+0000 7fbf2189f640 1 --2- 172.21.15.52:0/952264376 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fbf1c11a8e0 0x7fbf1c11cd00 unknown :-1 s=CLOSED pgs=217 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:36:13.927 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:13.928+0000 7fbf2189f640 1 --2- 172.21.15.52:0/952264376 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fbf1c10a1f0 0x7fbf1c10a600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:36:13.927 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:13.928+0000 7fbf2189f640 1 -- 172.21.15.52:0/952264376 >> 172.21.15.52:0/952264376 conn(0x7fbf1c06e8f0 msgr2=0x7fbf1c06ed30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:36:13.927 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:13.931+0000 7fbf2189f640 1 -- 172.21.15.52:0/952264376 shutdown_connections 2024-05-27T07:36:13.929 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:13.933+0000 7fbf2189f640 1 -- 172.21.15.52:0/952264376 wait complete. 2024-05-27T07:36:13.930 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:13.934+0000 7fbf2189f640 1 Processor -- start 2024-05-27T07:36:13.931 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:13.935+0000 7fbf2189f640 1 -- start start 2024-05-27T07:36:13.932 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:13.936+0000 7fbf2189f640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fbf1c10a1f0 0x7fbf1c1a96b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:36:13.932 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:13.936+0000 7fbf2189f640 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fbf1c11a8e0 0x7fbf1c1a9c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:36:13.932 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:13.936+0000 7fbf2189f640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fbf1c11d240 0x7fbf1c1aded0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:36:13.932 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:13.936+0000 7fbf2189f640 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x7fbf1c122cb0 con 0x7fbf1c11a8e0 2024-05-27T07:36:13.932 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:13.936+0000 7fbf2189f640 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7fbf1c122b30 con 0x7fbf1c11d240 2024-05-27T07:36:13.932 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:13.936+0000 7fbf2189f640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7fbf1c122e30 con 0x7fbf1c10a1f0 2024-05-27T07:36:13.933 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:13.936+0000 7fbf1b7fe640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fbf1c11d240 0x7fbf1c1aded0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:36:13.933 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:13.937+0000 7fbf1b7fe640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fbf1c11d240 0x7fbf1c1aded0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.52:35168/0 (socket says 172.21.15.52:35168) 2024-05-27T07:36:13.933 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:13.937+0000 7fbf1b7fe640 1 -- 172.21.15.52:0/3955416573 learned_addr learned my addr 172.21.15.52:0/3955416573 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-05-27T07:36:13.933 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:13.937+0000 7fbf1b7fe640 1 -- 172.21.15.52:0/3955416573 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fbf1c10a1f0 msgr2=0x7fbf1c1a96b0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-05-27T07:36:13.933 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:13.937+0000 7fbf1b7fe640 1 --2- 172.21.15.52:0/3955416573 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fbf1c10a1f0 0x7fbf1c1a96b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:36:13.933 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:13.937+0000 7fbf1b7fe640 1 -- 172.21.15.52:0/3955416573 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fbf1c11a8e0 msgr2=0x7fbf1c1a9c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:36:13.933 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:13.937+0000 7fbf1b7fe640 1 --2- 172.21.15.52:0/3955416573 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fbf1c11a8e0 0x7fbf1c1a9c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:36:13.934 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:13.937+0000 7fbf1b7fe640 1 -- 172.21.15.52:0/3955416573 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbf1c1ae510 con 0x7fbf1c11d240 2024-05-27T07:36:13.934 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:13.937+0000 7fbf1b7fe640 1 --2- 172.21.15.52:0/3955416573 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fbf1c11d240 0x7fbf1c1aded0 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7fbf1400eda0 tx=0x7fbf1400c6e0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:36:13.935 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:13.940+0000 7fbf2089d640 1 -- 172.21.15.52:0/3955416573 <== mon.2 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbf14003b70 con 0x7fbf1c11d240 2024-05-27T07:36:13.937 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:13.940+0000 7fbf2189f640 1 -- 172.21.15.52:0/3955416573 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbf1c1ae800 con 0x7fbf1c11d240 2024-05-27T07:36:13.937 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:13.940+0000 7fbf2189f640 1 -- 172.21.15.52:0/3955416573 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbf1c1b6180 con 0x7fbf1c11d240 2024-05-27T07:36:13.937 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:13.941+0000 7fbf2089d640 1 -- 172.21.15.52:0/3955416573 <== mon.2 v2:172.21.15.105:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fbf14003d10 con 0x7fbf1c11d240 2024-05-27T07:36:13.938 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:13.943+0000 7fbf2089d640 1 -- 172.21.15.52:0/3955416573 <== mon.2 v2:172.21.15.105:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbf14017900 con 0x7fbf1c11d240 2024-05-27T07:36:13.939 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:13.943+0000 7fbf2189f640 1 -- 172.21.15.52:0/3955416573 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fbefc0052c0 con 0x7fbf1c11d240 2024-05-27T07:36:13.944 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:13.949+0000 7fbf2089d640 1 -- 172.21.15.52:0/3955416573 <== mon.2 v2:172.21.15.105:3300/0 4 ==== mgrmap(e 26) ==== 149783+0+0 (secure 0 0 0) 0x7fbf14017aa0 con 0x7fbf1c11d240 2024-05-27T07:36:13.946 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:13.950+0000 7fbf2089d640 1 --2- 172.21.15.52:0/3955416573 >> [v2:172.21.15.52:6800/1822318384,v1:172.21.15.52:6801/1822318384] conn(0x7fbf080b2a50 0x7fbf080b4f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:36:13.946 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:13.951+0000 7fbf2089d640 1 -- 172.21.15.52:0/3955416573 <== mon.2 v2:172.21.15.105:3300/0 5 ==== osd_map(127..127 src has 1..127) ==== 9555+0+0 (secure 0 0 0) 0x7fbf140e2dc0 con 0x7fbf1c11d240 2024-05-27T07:36:13.953 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:13.955+0000 7fbf1affd640 1 --2- 172.21.15.52:0/3955416573 >> [v2:172.21.15.52:6800/1822318384,v1:172.21.15.52:6801/1822318384] conn(0x7fbf080b2a50 0x7fbf080b4f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:36:13.953 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:13.957+0000 7fbf2089d640 1 -- 172.21.15.52:0/3955416573 <== mon.2 v2:172.21.15.105:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191963 (secure 0 0 0) 0x7fbf140aa4c0 con 0x7fbf1c11d240 2024-05-27T07:36:13.967 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:13.963+0000 7fbf1affd640 1 --2- 172.21.15.52:0/3955416573 >> [v2:172.21.15.52:6800/1822318384,v1:172.21.15.52:6801/1822318384] conn(0x7fbf080b2a50 0x7fbf080b4f60 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7fbf04009670 tx=0x7fbf04006bd0 comp rx=0 tx=0).ready entity=mgr.15333 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:36:13.979 INFO:teuthology.orchestra.run.smithi052.stderr:Inferring config /var/lib/ceph/73df2636-1bf8-11ef-bc9b-c7b262605968/mon.a/config 2024-05-27T07:36:14.375 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:14.377+0000 7fbf2189f640 1 -- 172.21.15.52:0/3955416573 --> [v2:172.21.15.52:6800/1822318384,v1:172.21.15.52:6801/1822318384] -- mgr_command(tid 0: {"prefix": "fs snap-schedule list", "fs": "cephfs", "path": "/", "recursive": true, "target": ["mon-mgr", ""]}) -- 0x7fbefc000bf0 con 0x7fbf080b2a50 2024-05-27T07:36:14.485 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:36:14 smithi052 ceph-mon[31955]: pgmap v347: 129 pgs: 24 active+clean+snaptrim, 72 active+clean+snaptrim_wait, 33 active+clean; 638 MiB data, 6.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 KiB/s rd, 46 KiB/s wr, 95 op/s 2024-05-27T07:36:14.545 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:14.549+0000 7fbf2089d640 1 -- 172.21.15.52:0/3955416573 <== mgr.15333 v2:172.21.15.52:6800/1822318384 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+10 (secure 0 0 0) 0x7fbefc000bf0 con 0x7fbf080b2a50 2024-05-27T07:36:14.545 INFO:teuthology.orchestra.run.smithi052.stdout:/ 1m 6m3h 2024-05-27T07:36:14.550 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:14.554+0000 7fbf2189f640 1 -- 172.21.15.52:0/3955416573 >> [v2:172.21.15.52:6800/1822318384,v1:172.21.15.52:6801/1822318384] conn(0x7fbf080b2a50 msgr2=0x7fbf080b4f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:36:14.550 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:14.554+0000 7fbf2189f640 1 --2- 172.21.15.52:0/3955416573 >> [v2:172.21.15.52:6800/1822318384,v1:172.21.15.52:6801/1822318384] conn(0x7fbf080b2a50 0x7fbf080b4f60 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7fbf04009670 tx=0x7fbf04006bd0 comp rx=0 tx=0).stop 2024-05-27T07:36:14.550 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:14.554+0000 7fbf2189f640 1 -- 172.21.15.52:0/3955416573 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fbf1c11d240 msgr2=0x7fbf1c1aded0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:36:14.550 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:14.554+0000 7fbf2189f640 1 --2- 172.21.15.52:0/3955416573 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fbf1c11d240 0x7fbf1c1aded0 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7fbf1400eda0 tx=0x7fbf1400c6e0 comp rx=0 tx=0).stop 2024-05-27T07:36:14.550 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:14.554+0000 7fbf2189f640 1 -- 172.21.15.52:0/3955416573 shutdown_connections 2024-05-27T07:36:14.550 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:14.554+0000 7fbf2189f640 1 --2- 172.21.15.52:0/3955416573 >> [v2:172.21.15.52:6800/1822318384,v1:172.21.15.52:6801/1822318384] conn(0x7fbf080b2a50 0x7fbf080b4f60 unknown :-1 s=CLOSED pgs=192 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:36:14.550 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:14.554+0000 7fbf2189f640 1 --2- 172.21.15.52:0/3955416573 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fbf1c11d240 0x7fbf1c1aded0 unknown :-1 s=CLOSED pgs=262 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:36:14.550 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:14.554+0000 7fbf2189f640 1 --2- 172.21.15.52:0/3955416573 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fbf1c11a8e0 0x7fbf1c1a9c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:36:14.551 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:14.554+0000 7fbf2189f640 1 --2- 172.21.15.52:0/3955416573 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fbf1c10a1f0 0x7fbf1c1a96b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:36:14.551 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:14.554+0000 7fbf2189f640 1 -- 172.21.15.52:0/3955416573 >> 172.21.15.52:0/3955416573 conn(0x7fbf1c06e8f0 msgr2=0x7fbf1c11acf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:36:14.551 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:14.556+0000 7fbf2189f640 1 -- 172.21.15.52:0/3955416573 shutdown_connections 2024-05-27T07:36:14.551 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:14.556+0000 7fbf2189f640 1 -- 172.21.15.52:0/3955416573 wait complete. 2024-05-27T07:36:14.570 DEBUG:teuthology.run_tasks:Unwinding manager fwd_scrub 2024-05-27T07:36:14.581 INFO:tasks.fwd_scrub:joining ForwardScrubbers 2024-05-27T07:36:14.607 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:36:14 smithi173 ceph-mon[37000]: pgmap v347: 129 pgs: 24 active+clean+snaptrim, 72 active+clean+snaptrim_wait, 33 active+clean; 638 MiB data, 6.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 KiB/s rd, 46 KiB/s wr, 95 op/s 2024-05-27T07:36:14.695 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:36:14 smithi105 ceph-mon[35977]: pgmap v347: 129 pgs: 24 active+clean+snaptrim, 72 active+clean+snaptrim_wait, 33 active+clean; 638 MiB data, 6.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 KiB/s rd, 46 KiB/s wr, 95 op/s 2024-05-27T07:36:15.581 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:36:15 smithi052 ceph-mon[31955]: from='client.26879 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule status", "fs": "cephfs", "path": "/", "target": ["mon-mgr", ""]}]: dispatch 2024-05-27T07:36:15.581 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:36:15 smithi052 ceph-mon[31955]: from='client.26882 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule list", "fs": "cephfs", "path": "/", "recursive": true, "target": ["mon-mgr", ""]}]: dispatch 2024-05-27T07:36:15.607 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:36:15 smithi173 ceph-mon[37000]: from='client.26879 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule status", "fs": "cephfs", "path": "/", "target": ["mon-mgr", ""]}]: dispatch 2024-05-27T07:36:15.608 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:36:15 smithi173 ceph-mon[37000]: from='client.26882 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule list", "fs": "cephfs", "path": "/", "recursive": true, "target": ["mon-mgr", ""]}]: dispatch 2024-05-27T07:36:15.695 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:36:15 smithi105 ceph-mon[35977]: from='client.26879 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule status", "fs": "cephfs", "path": "/", "target": ["mon-mgr", ""]}]: dispatch 2024-05-27T07:36:15.696 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:36:15 smithi105 ceph-mon[35977]: from='client.26882 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule list", "fs": "cephfs", "path": "/", "recursive": true, "target": ["mon-mgr", ""]}]: dispatch 2024-05-27T07:36:16.597 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:36:16 smithi105 ceph-mon[35977]: pgmap v348: 129 pgs: 22 active+clean+snaptrim, 70 active+clean+snaptrim_wait, 37 active+clean; 637 MiB data, 6.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 38 KiB/s wr, 77 op/s 2024-05-27T07:36:16.607 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:36:16 smithi173 ceph-mon[37000]: pgmap v348: 129 pgs: 22 active+clean+snaptrim, 70 active+clean+snaptrim_wait, 37 active+clean; 637 MiB data, 6.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 38 KiB/s wr, 77 op/s 2024-05-27T07:36:16.617 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:36:16 smithi052 ceph-mon[31955]: pgmap v348: 129 pgs: 22 active+clean+snaptrim, 70 active+clean+snaptrim_wait, 37 active+clean; 637 MiB data, 6.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 38 KiB/s wr, 77 op/s 2024-05-27T07:36:17.239 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:17.243+0000 7fb2b439c700 1 -- 172.21.15.52:0/670065315 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fb2ac10b270 msgr2=0x7fb2ac1198a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:36:17.239 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:17.243+0000 7fb2b439c700 1 --2- 172.21.15.52:0/670065315 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fb2ac10b270 0x7fb2ac1198a0 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7fb29c009990 tx=0x7fb29c009ca0 comp rx=0 tx=0).stop 2024-05-27T07:36:17.239 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:17.243+0000 7fb2b439c700 1 -- 172.21.15.52:0/670065315 shutdown_connections 2024-05-27T07:36:17.240 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:17.243+0000 7fb2b439c700 1 --2- 172.21.15.52:0/670065315 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fb2ac10b270 0x7fb2ac1198a0 unknown :-1 s=CLOSED pgs=218 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:36:17.240 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:17.243+0000 7fb2b439c700 1 --2- 172.21.15.52:0/670065315 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fb2ac111bc0 0x7fb2ac111fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:36:17.240 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:17.243+0000 7fb2b439c700 1 --2- 172.21.15.52:0/670065315 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fb2ac10a920 0x7fb2ac10ad30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:36:17.240 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:17.243+0000 7fb2b439c700 1 -- 172.21.15.52:0/670065315 >> 172.21.15.52:0/670065315 conn(0x7fb2ac06f4c0 msgr2=0x7fb2ac06f900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:36:17.243 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:17.248+0000 7fb2b439c700 1 -- 172.21.15.52:0/670065315 shutdown_connections 2024-05-27T07:36:17.244 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:17.248+0000 7fb2b439c700 1 -- 172.21.15.52:0/670065315 wait complete. 2024-05-27T07:36:17.245 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:17.249+0000 7fb2b439c700 1 Processor -- start 2024-05-27T07:36:17.250 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:17.249+0000 7fb2b439c700 1 -- start start 2024-05-27T07:36:17.250 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:17.250+0000 7fb2b439c700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fb2ac10a920 0x7fb2ac116de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:36:17.250 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:17.250+0000 7fb2b439c700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fb2ac10b270 0x7fb2ac117320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:36:17.250 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:17.250+0000 7fb2b439c700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fb2ac111bc0 0x7fb2ac117880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:36:17.250 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:17.250+0000 7fb2b439c700 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x7fb2ac10e140 con 0x7fb2ac10b270 2024-05-27T07:36:17.250 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:17.250+0000 7fb2b439c700 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7fb2ac10dfc0 con 0x7fb2ac111bc0 2024-05-27T07:36:17.250 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:17.250+0000 7fb2b439c700 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7fb2ac10e2c0 con 0x7fb2ac10a920 2024-05-27T07:36:17.250 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:17.251+0000 7fb2b1937700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fb2ac10b270 0x7fb2ac117320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:36:17.250 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:17.251+0000 7fb2b1937700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fb2ac10b270 0x7fb2ac117320 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.52:42574/0 (socket says 172.21.15.52:42574) 2024-05-27T07:36:17.251 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:17.251+0000 7fb2b1937700 1 -- 172.21.15.52:0/1404061159 learned_addr learned my addr 172.21.15.52:0/1404061159 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-05-27T07:36:17.251 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:17.253+0000 7fb2b2138700 1 --2- 172.21.15.52:0/1404061159 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fb2ac10a920 0x7fb2ac116de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:36:17.251 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:17.254+0000 7fb2b2138700 1 -- 172.21.15.52:0/1404061159 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fb2ac111bc0 msgr2=0x7fb2ac117880 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-05-27T07:36:17.251 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:17.254+0000 7fb2b2138700 1 --2- 172.21.15.52:0/1404061159 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fb2ac111bc0 0x7fb2ac117880 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:36:17.251 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:17.254+0000 7fb2b2138700 1 -- 172.21.15.52:0/1404061159 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fb2ac10b270 msgr2=0x7fb2ac117320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:36:17.251 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:17.254+0000 7fb2b2138700 1 --2- 172.21.15.52:0/1404061159 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fb2ac10b270 0x7fb2ac117320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:36:17.251 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:17.254+0000 7fb2b2138700 1 -- 172.21.15.52:0/1404061159 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb2ac11f750 con 0x7fb2ac10a920 2024-05-27T07:36:17.251 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:17.254+0000 7fb2b2138700 1 --2- 172.21.15.52:0/1404061159 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fb2ac10a920 0x7fb2ac116de0 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7fb2a400bd40 tx=0x7fb2a400be20 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:36:17.252 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:17.257+0000 7fb2a37fe700 1 -- 172.21.15.52:0/1404061159 <== mon.1 v2:172.21.15.173:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb2a4019070 con 0x7fb2ac10a920 2024-05-27T07:36:17.253 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:17.257+0000 7fb2a37fe700 1 -- 172.21.15.52:0/1404061159 <== mon.1 v2:172.21.15.173:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fb2a400cd10 con 0x7fb2ac10a920 2024-05-27T07:36:17.253 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:17.257+0000 7fb2a37fe700 1 -- 172.21.15.52:0/1404061159 <== mon.1 v2:172.21.15.173:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb2a4012630 con 0x7fb2ac10a920 2024-05-27T07:36:17.254 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:17.259+0000 7fb2b439c700 1 -- 172.21.15.52:0/1404061159 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb2ac11fa40 con 0x7fb2ac10a920 2024-05-27T07:36:17.254 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:17.259+0000 7fb2b439c700 1 -- 172.21.15.52:0/1404061159 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb2ac11ffb0 con 0x7fb2ac10a920 2024-05-27T07:36:17.260 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:17.264+0000 7fb2c2db1b80 1 Processor -- start 2024-05-27T07:36:17.263 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:17.265+0000 7fb2c2db1b80 1 -- start start 2024-05-27T07:36:17.263 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:17.265+0000 7fb2c2db1b80 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5570ad2ece40 0x5570ad470720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:36:17.263 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:17.265+0000 7fb2c2db1b80 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x5570ad470c60 0x5570ad46aa40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:36:17.263 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:17.265+0000 7fb2c2db1b80 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5570ad46af80 0x5570ad46b390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:36:17.263 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:17.265+0000 7fb2c2db1b80 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x5570ad399c90 con 0x5570ad470c60 2024-05-27T07:36:17.263 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:17.265+0000 7fb2c2db1b80 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x5570ad389830 con 0x5570ad46af80 2024-05-27T07:36:17.263 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:17.265+0000 7fb2c2db1b80 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x5570ad38d870 con 0x5570ad2ece40 2024-05-27T07:36:17.263 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:17.266+0000 7fb2b1937700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5570ad46af80 0x5570ad46b390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:36:17.263 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:17.266+0000 7fb2b1937700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5570ad46af80 0x5570ad46b390 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.52:35186/0 (socket says 172.21.15.52:35186) 2024-05-27T07:36:17.263 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:17.266+0000 7fb2b1937700 1 -- 172.21.15.52:0/2998939539 learned_addr learned my addr 172.21.15.52:0/2998939539 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-05-27T07:36:17.264 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:17.267+0000 7fb2b2939700 1 --2- 172.21.15.52:0/2998939539 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5570ad2ece40 0x5570ad470720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:36:17.264 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:17.267+0000 7fb2b1937700 1 -- 172.21.15.52:0/2998939539 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5570ad2ece40 msgr2=0x5570ad470720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:36:17.264 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:17.267+0000 7fb2b1937700 1 --2- 172.21.15.52:0/2998939539 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5570ad2ece40 0x5570ad470720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:36:17.264 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:17.267+0000 7fb2b1937700 1 -- 172.21.15.52:0/2998939539 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x5570ad470c60 msgr2=0x5570ad46aa40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:36:17.265 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:17.267+0000 7fb2b1937700 1 --2- 172.21.15.52:0/2998939539 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x5570ad470c60 0x5570ad46aa40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:36:17.265 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:17.267+0000 7fb2b1937700 1 -- 172.21.15.52:0/2998939539 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5570ad2d98d0 con 0x5570ad46af80 2024-05-27T07:36:17.265 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:17.270+0000 7fb2b1937700 1 --2- 172.21.15.52:0/2998939539 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5570ad46af80 0x5570ad46b390 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7fb2a8009a50 tx=0x7fb2a8009d60 comp rx=0 tx=0).ready entity=mon.2 client_cookie=2c45b6bbf8a9b70e server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:36:17.266 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:17.270+0000 7fb2b439c700 1 -- 172.21.15.52:0/2998939539 <== mon.2 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb2a8017020 con 0x5570ad46af80 2024-05-27T07:36:17.266 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:17.271+0000 7fb2b439c700 1 -- 172.21.15.52:0/2998939539 <== mon.2 v2:172.21.15.105:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fb2a800ed10 con 0x5570ad46af80 2024-05-27T07:36:17.268 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:17.272+0000 7fb2c2db1b80 1 -- 172.21.15.52:0/2998939539 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5570ad46af80 msgr2=0x5570ad46b390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:36:17.268 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:17.272+0000 7fb2c2db1b80 1 --2- 172.21.15.52:0/2998939539 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5570ad46af80 0x5570ad46b390 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7fb2a8009a50 tx=0x7fb2a8009d60 comp rx=0 tx=0).stop 2024-05-27T07:36:17.270 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:17.275+0000 7fb2c2db1b80 1 -- 172.21.15.52:0/2998939539 shutdown_connections 2024-05-27T07:36:17.271 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:17.275+0000 7fb2c2db1b80 1 --2- 172.21.15.52:0/2998939539 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5570ad46af80 0x5570ad46b390 unknown :-1 s=CLOSED pgs=263 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:36:17.271 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:17.275+0000 7fb2c2db1b80 1 --2- 172.21.15.52:0/2998939539 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5570ad2ece40 0x5570ad470720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:36:17.271 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:17.275+0000 7fb2c2db1b80 1 --2- 172.21.15.52:0/2998939539 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x5570ad470c60 0x5570ad46aa40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:36:17.271 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:17.275+0000 7fb2c2db1b80 1 -- 172.21.15.52:0/2998939539 >> 172.21.15.52:0/2998939539 conn(0x5570ad347af0 msgr2=0x5570ad4659c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:36:17.275 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:17.280+0000 7fb2a37fe700 1 -- 172.21.15.52:0/1404061159 <== mon.1 v2:172.21.15.173:3300/0 4 ==== mgrmap(e 26) ==== 149783+0+0 (secure 0 0 0) 0x7fb2a4014a10 con 0x7fb2ac10a920 2024-05-27T07:36:17.276 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:17.280+0000 7fb2c2db1b80 1 -- 172.21.15.52:0/2998939539 shutdown_connections 2024-05-27T07:36:17.277 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:17.281+0000 7fb2a37fe700 1 --2- 172.21.15.52:0/1404061159 >> [v2:172.21.15.52:6800/1822318384,v1:172.21.15.52:6801/1822318384] conn(0x7fb2980b2ac0 0x7fb2980b4fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:36:17.277 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:17.282+0000 7fb2a37fe700 1 -- 172.21.15.52:0/1404061159 <== mon.1 v2:172.21.15.173:3300/0 5 ==== osd_map(127..127 src has 1..127) ==== 9555+0+0 (secure 0 0 0) 0x7fb2a40e1e70 con 0x7fb2ac10a920 2024-05-27T07:36:17.277 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:17.282+0000 7fb2c2db1b80 1 -- 172.21.15.52:0/2998939539 wait complete. 2024-05-27T07:36:17.278 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:17.283+0000 7fb2c2db1b80 1 -- start start 2024-05-27T07:36:17.278 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:17.283+0000 7fb2c2db1b80 1 Processor -- start 2024-05-27T07:36:17.279 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:17.284+0000 7fb2c2db1b80 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5570ad2ece40 0x5570ad4742f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:36:17.280 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:17.284+0000 7fb2c2db1b80 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5570ad46af80 0x5570ad474830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:36:17.280 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:17.284+0000 7fb2c2db1b80 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x5570ad470c60 0x5570ad474d70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:36:17.280 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:17.284+0000 7fb2c2db1b80 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x5570ad28ffa0 con 0x5570ad470c60 2024-05-27T07:36:17.280 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:17.284+0000 7fb2c2db1b80 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x5570ad32b7c0 con 0x5570ad2ece40 2024-05-27T07:36:17.280 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:17.284+0000 7fb2c2db1b80 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x5570ad477850 con 0x5570ad46af80 2024-05-27T07:36:17.280 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:17.284+0000 7fb2c2db1b80 10 client.0 _ll_register_callbacks cb 0x5570ad324920 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T07:36:17.283 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:17.288+0000 7fb2b1937700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5570ad46af80 0x5570ad474830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:36:17.284 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:17.288+0000 7fb2b1937700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5570ad46af80 0x5570ad474830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.173:3300/0 says I am v2:172.21.15.52:33578/0 (socket says 172.21.15.52:33578) 2024-05-27T07:36:17.284 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:17.288+0000 7fb2b1937700 1 -- 172.21.15.52:0/66668424 learned_addr learned my addr 172.21.15.52:0/66668424 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-05-27T07:36:17.284 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:17.288+0000 7fb2b2138700 1 --2- 172.21.15.52:0/66668424 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5570ad2ece40 0x5570ad4742f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:36:17.284 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:17.288+0000 7fb2b1937700 1 --2- 172.21.15.52:0/1404061159 >> [v2:172.21.15.52:6800/1822318384,v1:172.21.15.52:6801/1822318384] conn(0x7fb2980b2ac0 0x7fb2980b4fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:36:17.286 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:17.291+0000 7fb2b2939700 1 --2- 172.21.15.52:0/66668424 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x5570ad470c60 0x5570ad474d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:36:17.287 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:17.291+0000 7fb2b1937700 1 -- 172.21.15.52:0/66668424 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5570ad2ece40 msgr2=0x5570ad4742f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:36:17.287 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:17.291+0000 7fb2b1937700 1 --2- 172.21.15.52:0/66668424 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5570ad2ece40 0x5570ad4742f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:36:17.287 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:17.291+0000 7fb2b1937700 1 -- 172.21.15.52:0/66668424 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x5570ad470c60 msgr2=0x5570ad474d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:36:17.287 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:17.291+0000 7fb2b1937700 1 --2- 172.21.15.52:0/66668424 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x5570ad470c60 0x5570ad474d70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:36:17.287 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:17.291+0000 7fb2b1937700 1 -- 172.21.15.52:0/66668424 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5570ad328690 con 0x5570ad46af80 2024-05-27T07:36:17.291 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:17.295+0000 7fb2b1937700 1 --2- 172.21.15.52:0/66668424 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5570ad46af80 0x5570ad474830 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7fb2a8008010 tx=0x7fb2a8009d60 comp rx=0 tx=0).ready entity=mon.1 client_cookie=7233548e471d1a14 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:36:17.291 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:17.295+0000 7fb295ffb700 10 client.0 ms_handle_connect on v2:172.21.15.173:3300/0 2024-05-27T07:36:17.291 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:17.295+0000 7fb2b1937700 1 --2- 172.21.15.52:0/1404061159 >> [v2:172.21.15.52:6800/1822318384,v1:172.21.15.52:6801/1822318384] conn(0x7fb2980b2ac0 0x7fb2980b4fd0 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7fb2a8004790 tx=0x7fb2a8013a00 comp rx=0 tx=0).ready entity=mgr.15333 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:36:17.292 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:17.296+0000 7fb295ffb700 1 -- 172.21.15.52:0/66668424 <== mon.1 v2:172.21.15.173:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb2a8017030 con 0x5570ad46af80 2024-05-27T07:36:17.292 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:17.296+0000 7fb295ffb700 1 -- 172.21.15.52:0/66668424 <== mon.1 v2:172.21.15.173:3300/0 2 ==== osd_map(127..127 src has 1..127) ==== 9555+0+0 (secure 0 0 0) 0x7fb2a8005a50 con 0x5570ad46af80 2024-05-27T07:36:17.292 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:17.296+0000 7fb2c2db1b80 1 -- 172.21.15.52:0/66668424 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fb2a800ed10 con 0x5570ad46af80 2024-05-27T07:36:17.296 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:17.301+0000 7fb295ffb700 1 -- 172.21.15.52:0/66668424 <== mon.1 v2:172.21.15.173:3300/0 3 ==== mon_get_version_reply(handle=1 version=108) ==== 24+0+0 (secure 0 0 0) 0x7fb2a800eb50 con 0x5570ad46af80 2024-05-27T07:36:17.297 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:17.302+0000 7fb2c2db1b80 10 client.26719 fetch_fsmap learned FSMap version 108 2024-05-27T07:36:17.297 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:17.302+0000 7fb2c2db1b80 1 -- 172.21.15.52:0/66668424 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({config=0+,fsmap=108,monmap=4+}) -- 0x5570ad482290 con 0x5570ad46af80 2024-05-27T07:36:17.303 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:17.306+0000 7fb295ffb700 1 -- 172.21.15.52:0/66668424 <== mon.1 v2:172.21.15.173:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fb2a800ed10 con 0x5570ad46af80 2024-05-27T07:36:17.304 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:17.306+0000 7fb295ffb700 1 -- 172.21.15.52:0/66668424 <== mon.1 v2:172.21.15.173:3300/0 5 ==== fsmap(e 108) ==== 4203+0+0 (secure 0 0 0) 0x7fb2a8012050 con 0x5570ad46af80 2024-05-27T07:36:17.304 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:17.308+0000 7fb2c2db1b80 10 client.26719 fetch_fsmap finished waiting for FSMap version 108 2024-05-27T07:36:17.304 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:17.308+0000 7fb2c2db1b80 10 client.26719 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-05-27T07:36:17.304 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:17.308+0000 7fb2c2db1b80 20 client.26719 populate_metadata read hostname 'smithi052' 2024-05-27T07:36:17.304 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:17.308+0000 7fb2c2db1b80 1 --2- 172.21.15.52:0/66668424 >> [v2:172.21.15.105:6838/500348151,v1:172.21.15.105:6839/500348151] conn(0x5570ad484c20 0x5570ad4870a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:36:17.304 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:17.308+0000 7fb2c2db1b80 4 client.26719 mds_command: new command op to 24527 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-27T07:36:17.304 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:17.308+0000 7fb2c2db1b80 1 -- 172.21.15.52:0/66668424 --> [v2:172.21.15.105:6838/500348151,v1:172.21.15.105:6839/500348151] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5570ad4369f0 con 0x5570ad484c20 2024-05-27T07:36:17.310 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:17.313+0000 7fb2b2138700 1 --2- 172.21.15.52:0/66668424 >> [v2:172.21.15.105:6838/500348151,v1:172.21.15.105:6839/500348151] conn(0x5570ad484c20 0x5570ad4870a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:36:17.321 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:17.325+0000 7fb2b2138700 1 --2- 172.21.15.52:0/66668424 >> [v2:172.21.15.105:6838/500348151,v1:172.21.15.105:6839/500348151] conn(0x5570ad484c20 0x5570ad4870a0 crc :-1 s=READY pgs=177 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=449c2b80bf68a19e server_cookie=3eaeca3510009790 in_seq=0 out_seq=0 2024-05-27T07:36:17.321 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:17.326+0000 7fb295ffb700 10 client.26719 ms_handle_connect on v2:172.21.15.105:6838/500348151 2024-05-27T07:36:17.326 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:17.330+0000 7fb295ffb700 1 -- 172.21.15.52:0/66668424 <== mds.0 v2:172.21.15.105:6838/500348151 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24041 (crc 0 0 0) 0x5570ad4369f0 con 0x5570ad484c20 2024-05-27T07:36:17.326 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:17.330+0000 7fb295ffb700 10 client.26719 handle_command_reply: tid=0 2024-05-27T07:36:17.326 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:17.330+0000 7fb2c2db1b80 1 client.26719 shutdown 2024-05-27T07:36:17.328 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:17.331+0000 7fb2c2db1b80 1 -- 172.21.15.52:0/66668424 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5570ad46af80 msgr2=0x5570ad474830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:36:17.328 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:17.331+0000 7fb2c2db1b80 1 --2- 172.21.15.52:0/66668424 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5570ad46af80 0x5570ad474830 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7fb2a8008010 tx=0x7fb2a8009d60 comp rx=0 tx=0).stop 2024-05-27T07:36:17.328 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:17.331+0000 7fb2c2db1b80 1 -- 172.21.15.52:0/66668424 shutdown_connections 2024-05-27T07:36:17.328 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:17.331+0000 7fb2c2db1b80 1 --2- 172.21.15.52:0/66668424 >> [v2:172.21.15.105:6838/500348151,v1:172.21.15.105:6839/500348151] conn(0x5570ad484c20 0x5570ad4870a0 crc :-1 s=READY pgs=177 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:36:17.328 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:17.331+0000 7fb2c2db1b80 1 --2- 172.21.15.52:0/66668424 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x5570ad470c60 0x5570ad474d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:36:17.328 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:17.331+0000 7fb2c2db1b80 1 --2- 172.21.15.52:0/66668424 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5570ad2ece40 0x5570ad4742f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:36:17.328 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:17.331+0000 7fb2c2db1b80 1 --2- 172.21.15.52:0/66668424 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5570ad46af80 0x5570ad474830 unknown :-1 s=CLOSED pgs=220 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:36:17.328 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:17.331+0000 7fb2c2db1b80 1 -- 172.21.15.52:0/66668424 >> 172.21.15.52:0/66668424 conn(0x5570ad347af0 msgr2=0x5570ad469a30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:36:17.328 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:17.331+0000 7fb295ffb700 0 client.26719 ms_handle_reset on v2:172.21.15.105:6838/500348151 2024-05-27T07:36:17.328 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:17.331+0000 7fb2c2db1b80 1 -- 172.21.15.52:0/66668424 shutdown_connections 2024-05-27T07:36:17.328 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:17.331+0000 7fb2c2db1b80 1 -- 172.21.15.52:0/66668424 wait complete. 2024-05-27T07:36:17.330 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:17.334+0000 7fb2c2db1b80 20 client.26719 trim_cache size 0 max 16384 2024-05-27T07:36:17.376 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:17.380+0000 7fb2c2db1b80 1 Processor -- start 2024-05-27T07:36:17.380 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:17.380+0000 7fb2c2db1b80 1 -- start start 2024-05-27T07:36:17.380 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:17.381+0000 7fb2c2db1b80 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5570ad2ece40 0x5570ad46aa40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:36:17.381 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:17.381+0000 7fb2c2db1b80 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x5570ad470c60 0x5570ad480940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:36:17.381 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:17.381+0000 7fb2c2db1b80 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5570ad460120 0x5570ad48c8a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:36:17.381 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:17.381+0000 7fb2c2db1b80 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x7fb2ac1c8430 con 0x5570ad470c60 2024-05-27T07:36:17.381 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:17.381+0000 7fb2c2db1b80 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7fb2ac127940 con 0x5570ad460120 2024-05-27T07:36:17.381 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:17.381+0000 7fb2c2db1b80 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7fb2ac1c7d30 con 0x5570ad2ece40 2024-05-27T07:36:17.381 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:17.384+0000 7fb2b2939700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5570ad460120 0x5570ad48c8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:36:17.381 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:17.384+0000 7fb2b2939700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5570ad460120 0x5570ad48c8a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.52:35198/0 (socket says 172.21.15.52:35198) 2024-05-27T07:36:17.381 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:17.384+0000 7fb2b2939700 1 -- 172.21.15.52:0/517865184 learned_addr learned my addr 172.21.15.52:0/517865184 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-05-27T07:36:17.381 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:17.384+0000 7fb2b2138700 1 --2- 172.21.15.52:0/517865184 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5570ad2ece40 0x5570ad46aa40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:36:17.382 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:17.385+0000 7fb2b2939700 1 -- 172.21.15.52:0/517865184 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5570ad2ece40 msgr2=0x5570ad46aa40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:36:17.382 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:17.385+0000 7fb2b2939700 1 --2- 172.21.15.52:0/517865184 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5570ad2ece40 0x5570ad46aa40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:36:17.382 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:17.385+0000 7fb2b1937700 1 --2- 172.21.15.52:0/517865184 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x5570ad470c60 0x5570ad480940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:36:17.382 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:17.385+0000 7fb2b2939700 1 -- 172.21.15.52:0/517865184 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x5570ad470c60 msgr2=0x5570ad480940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:36:17.382 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:17.385+0000 7fb2b2939700 1 --2- 172.21.15.52:0/517865184 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x5570ad470c60 0x5570ad480940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:36:17.382 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:17.385+0000 7fb2b2939700 1 -- 172.21.15.52:0/517865184 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5570ad480e80 con 0x5570ad460120 2024-05-27T07:36:17.383 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:17.386+0000 7fb2b1937700 1 --2- 172.21.15.52:0/517865184 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x5570ad470c60 0x5570ad480940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-27T07:36:17.383 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:17.388+0000 7fb2b2939700 1 --2- 172.21.15.52:0/517865184 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5570ad460120 0x5570ad48c8a0 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7fb2ac117dc0 tx=0x7fb29c019040 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f0fb8b0860df5348 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:36:17.388 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:17.392+0000 7fb297fff700 1 -- 172.21.15.52:0/517865184 <== mon.2 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb29c003a60 con 0x5570ad460120 2024-05-27T07:36:17.388 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:17.392+0000 7fb297fff700 1 -- 172.21.15.52:0/517865184 <== mon.2 v2:172.21.15.105:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fb29c027070 con 0x5570ad460120 2024-05-27T07:36:17.388 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:17.392+0000 7fb297fff700 1 -- 172.21.15.52:0/517865184 <== mon.2 v2:172.21.15.105:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb29c00bbb0 con 0x5570ad460120 2024-05-27T07:36:17.389 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:17.393+0000 7fb2c2db1b80 1 -- 172.21.15.52:0/517865184 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5570ad460120 msgr2=0x5570ad48c8a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:36:17.389 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:17.393+0000 7fb2c2db1b80 1 --2- 172.21.15.52:0/517865184 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5570ad460120 0x5570ad48c8a0 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7fb2ac117dc0 tx=0x7fb29c019040 comp rx=0 tx=0).stop 2024-05-27T07:36:17.389 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:17.394+0000 7fb2c2db1b80 1 -- 172.21.15.52:0/517865184 shutdown_connections 2024-05-27T07:36:17.390 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:17.394+0000 7fb2c2db1b80 1 --2- 172.21.15.52:0/517865184 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5570ad460120 0x5570ad48c8a0 unknown :-1 s=CLOSED pgs=264 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:36:17.390 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:17.394+0000 7fb2c2db1b80 1 --2- 172.21.15.52:0/517865184 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5570ad2ece40 0x5570ad46aa40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:36:17.390 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:17.394+0000 7fb2c2db1b80 1 --2- 172.21.15.52:0/517865184 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x5570ad470c60 0x5570ad480940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:36:17.390 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:17.394+0000 7fb2c2db1b80 1 -- 172.21.15.52:0/517865184 >> 172.21.15.52:0/517865184 conn(0x5570ad347af0 msgr2=0x5570ad46f9f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:36:17.394 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:17.399+0000 7fb2c2db1b80 1 -- 172.21.15.52:0/517865184 shutdown_connections 2024-05-27T07:36:17.394 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:17.399+0000 7fb2c2db1b80 1 -- 172.21.15.52:0/517865184 wait complete. 2024-05-27T07:36:17.395 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:17.399+0000 7fb2c2db1b80 1 -- start start 2024-05-27T07:36:17.395 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:17.400+0000 7fb2c2db1b80 1 Processor -- start 2024-05-27T07:36:17.396 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:17.400+0000 7fb2c2db1b80 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x5570ad2ece40 0x5570ad47fd50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:36:17.396 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:17.401+0000 7fb2c2db1b80 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5570ad460120 0x5570ad480290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:36:17.397 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:17.401+0000 7fb2b2138700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x5570ad2ece40 0x5570ad47fd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:36:17.397 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:17.401+0000 7fb2b2138700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x5570ad2ece40 0x5570ad47fd50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.52:42620/0 (socket says 172.21.15.52:42620) 2024-05-27T07:36:17.397 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:17.401+0000 7fb2b2138700 1 -- 172.21.15.52:0/2370719569 learned_addr learned my addr 172.21.15.52:0/2370719569 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-05-27T07:36:17.398 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:17.401+0000 7fb2c2db1b80 1 --2- 172.21.15.52:0/2370719569 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5570ad470c60 0x5570ad4807d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:36:17.398 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:17.401+0000 7fb2b1937700 1 --2- 172.21.15.52:0/2370719569 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5570ad460120 0x5570ad480290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:36:17.398 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:17.401+0000 7fb2c2db1b80 1 -- 172.21.15.52:0/2370719569 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x7fb2ac1c7660 con 0x5570ad2ece40 2024-05-27T07:36:17.398 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:17.401+0000 7fb2b2939700 1 --2- 172.21.15.52:0/2370719569 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5570ad470c60 0x5570ad4807d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:36:17.398 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:17.401+0000 7fb2c2db1b80 1 -- 172.21.15.52:0/2370719569 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7fb2ac1c6f60 con 0x5570ad460120 2024-05-27T07:36:17.398 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:17.402+0000 7fb2c2db1b80 1 -- 172.21.15.52:0/2370719569 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x5570ad4369f0 con 0x5570ad470c60 2024-05-27T07:36:17.401 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:17.402+0000 7fb2c2db1b80 10 client.0 _ll_register_callbacks cb 0x5570ad324920 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T07:36:17.401 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:17.404+0000 7fb2b2939700 1 -- 172.21.15.52:0/2370719569 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5570ad460120 msgr2=0x5570ad480290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:36:17.401 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:17.404+0000 7fb2b2939700 1 --2- 172.21.15.52:0/2370719569 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5570ad460120 0x5570ad480290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:36:17.401 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:17.404+0000 7fb2b2939700 1 -- 172.21.15.52:0/2370719569 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x5570ad2ece40 msgr2=0x5570ad47fd50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:36:17.401 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:17.404+0000 7fb2b2939700 1 --2- 172.21.15.52:0/2370719569 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x5570ad2ece40 0x5570ad47fd50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:36:17.401 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:17.404+0000 7fb2b2939700 1 -- 172.21.15.52:0/2370719569 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5570ad4ee7a0 con 0x5570ad470c60 2024-05-27T07:36:17.401 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:17.405+0000 7fb2b2939700 1 --2- 172.21.15.52:0/2370719569 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5570ad470c60 0x5570ad4807d0 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7fb29c009960 tx=0x7fb29c005f10 comp rx=0 tx=0).ready entity=mon.1 client_cookie=38067d0983dcbe03 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:36:17.401 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:17.405+0000 7fb295ffb700 10 client.0 ms_handle_connect on v2:172.21.15.173:3300/0 2024-05-27T07:36:17.402 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:17.406+0000 7fb295ffb700 1 -- 172.21.15.52:0/2370719569 <== mon.1 v2:172.21.15.173:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb29c00b9d0 con 0x5570ad470c60 2024-05-27T07:36:17.402 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:17.406+0000 7fb295ffb700 1 -- 172.21.15.52:0/2370719569 <== mon.1 v2:172.21.15.173:3300/0 2 ==== osd_map(127..127 src has 1..127) ==== 9555+0+0 (secure 0 0 0) 0x7fb29c00bb70 con 0x5570ad470c60 2024-05-27T07:36:17.402 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:17.406+0000 7fb2c2db1b80 1 -- 172.21.15.52:0/2370719569 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fb29c027070 con 0x5570ad470c60 2024-05-27T07:36:17.403 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:17.408+0000 7fb295ffb700 1 -- 172.21.15.52:0/2370719569 <== mon.1 v2:172.21.15.173:3300/0 3 ==== mon_get_version_reply(handle=1 version=108) ==== 24+0+0 (secure 0 0 0) 0x7fb29c03d050 con 0x5570ad470c60 2024-05-27T07:36:17.405 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:17.408+0000 7fb2c2db1b80 10 client.26725 fetch_fsmap learned FSMap version 108 2024-05-27T07:36:17.405 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:17.408+0000 7fb2c2db1b80 1 -- 172.21.15.52:0/2370719569 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({config=0+,fsmap=108,monmap=4+}) -- 0x5570ad4ef980 con 0x5570ad470c60 2024-05-27T07:36:17.407 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:17.411+0000 7fb295ffb700 1 -- 172.21.15.52:0/2370719569 <== mon.1 v2:172.21.15.173:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fb29c031070 con 0x5570ad470c60 2024-05-27T07:36:17.407 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:17.411+0000 7fb295ffb700 1 -- 172.21.15.52:0/2370719569 <== mon.1 v2:172.21.15.173:3300/0 5 ==== fsmap(e 108) ==== 4203+0+0 (secure 0 0 0) 0x7fb29c037520 con 0x5570ad470c60 2024-05-27T07:36:17.408 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:17.411+0000 7fb2c2db1b80 10 client.26725 fetch_fsmap finished waiting for FSMap version 108 2024-05-27T07:36:17.408 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:17.411+0000 7fb2c2db1b80 10 client.26725 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-05-27T07:36:17.409 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:17.411+0000 7fb2c2db1b80 20 client.26725 populate_metadata read hostname 'smithi052' 2024-05-27T07:36:17.409 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:17.411+0000 7fb2c2db1b80 1 --2- 172.21.15.52:0/2370719569 >> [v2:172.21.15.105:6838/500348151,v1:172.21.15.105:6839/500348151] conn(0x5570ad4efe50 0x5570ad4f2270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:36:17.409 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:17.411+0000 7fb2c2db1b80 4 client.26725 mds_command: new command op to 24527 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-05-27T07:36:17.409 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:17.411+0000 7fb2c2db1b80 1 -- 172.21.15.52:0/2370719569 --> [v2:172.21.15.105:6838/500348151,v1:172.21.15.105:6839/500348151] -- command(tid 0: {"prefix": "scrub status"}) -- 0x7fb29c027070 con 0x5570ad4efe50 2024-05-27T07:36:17.409 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:17.412+0000 7fb2b2138700 1 --2- 172.21.15.52:0/2370719569 >> [v2:172.21.15.105:6838/500348151,v1:172.21.15.105:6839/500348151] conn(0x5570ad4efe50 0x5570ad4f2270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:36:17.416 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:17.421+0000 7fb2b2138700 1 --2- 172.21.15.52:0/2370719569 >> [v2:172.21.15.105:6838/500348151,v1:172.21.15.105:6839/500348151] conn(0x5570ad4efe50 0x5570ad4f2270 crc :-1 s=READY pgs=178 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=b123f50a9acbf76d server_cookie=7e589062577dedd1 in_seq=0 out_seq=0 2024-05-27T07:36:17.416 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:17.421+0000 7fb295ffb700 10 client.26725 ms_handle_connect on v2:172.21.15.105:6838/500348151 2024-05-27T07:36:17.421 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:17.424+0000 7fb295ffb700 1 -- 172.21.15.52:0/2370719569 <== mds.0 v2:172.21.15.105:6838/500348151 1 ==== command_reply(tid 0: 0 ) ==== 8+0+63 (crc 0 0 0) 0x7fb29c027070 con 0x5570ad4efe50 2024-05-27T07:36:17.421 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:17.424+0000 7fb295ffb700 10 client.26725 handle_command_reply: tid=0 2024-05-27T07:36:17.421 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:17.424+0000 7fb2c2db1b80 1 client.26725 shutdown 2024-05-27T07:36:17.421 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:17.425+0000 7fb2c2db1b80 1 -- 172.21.15.52:0/2370719569 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5570ad470c60 msgr2=0x5570ad4807d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:36:17.421 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:17.425+0000 7fb2c2db1b80 1 --2- 172.21.15.52:0/2370719569 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5570ad470c60 0x5570ad4807d0 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7fb29c009960 tx=0x7fb29c005f10 comp rx=0 tx=0).stop 2024-05-27T07:36:17.424 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:17.427+0000 7fb2c2db1b80 1 -- 172.21.15.52:0/2370719569 shutdown_connections 2024-05-27T07:36:17.424 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:17.427+0000 7fb2c2db1b80 1 --2- 172.21.15.52:0/2370719569 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5570ad470c60 0x5570ad4807d0 unknown :-1 s=CLOSED pgs=221 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:36:17.424 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:17.427+0000 7fb2c2db1b80 1 --2- 172.21.15.52:0/2370719569 >> [v2:172.21.15.105:6838/500348151,v1:172.21.15.105:6839/500348151] conn(0x5570ad4efe50 0x5570ad4f2270 crc :-1 s=READY pgs=178 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:36:17.424 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:17.427+0000 7fb2c2db1b80 1 --2- 172.21.15.52:0/2370719569 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x5570ad2ece40 0x5570ad47fd50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:36:17.424 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:17.427+0000 7fb2c2db1b80 1 --2- 172.21.15.52:0/2370719569 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5570ad460120 0x5570ad480290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:36:17.425 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:17.427+0000 7fb2c2db1b80 1 -- 172.21.15.52:0/2370719569 >> 172.21.15.52:0/2370719569 conn(0x5570ad347af0 msgr2=0x5570ad46f9f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:36:17.425 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:17.428+0000 7fb295ffb700 0 client.26725 ms_handle_reset on v2:172.21.15.105:6838/500348151 2024-05-27T07:36:17.426 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:17.431+0000 7fb2c2db1b80 1 -- 172.21.15.52:0/2370719569 shutdown_connections 2024-05-27T07:36:17.427 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:17.432+0000 7fb2c2db1b80 1 -- 172.21.15.52:0/2370719569 wait complete. 2024-05-27T07:36:17.429 INFO:teuthology.orchestra.run.smithi052.stdout:{ 2024-05-27T07:36:17.429 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "no active scrubs running", 2024-05-27T07:36:17.429 INFO:teuthology.orchestra.run.smithi052.stdout: "scrubs": {} 2024-05-27T07:36:17.429 INFO:teuthology.orchestra.run.smithi052.stdout:} 2024-05-27T07:36:17.430 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:17.432+0000 7fb2c2db1b80 20 client.26725 trim_cache size 0 max 16384 2024-05-27T07:36:17.430 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:17.433+0000 7fb2a1548700 1 -- 172.21.15.52:0/1404061159 >> [v2:172.21.15.52:6800/1822318384,v1:172.21.15.52:6801/1822318384] conn(0x7fb2980b2ac0 msgr2=0x7fb2980b4fd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:36:17.430 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:17.433+0000 7fb2a1548700 1 --2- 172.21.15.52:0/1404061159 >> [v2:172.21.15.52:6800/1822318384,v1:172.21.15.52:6801/1822318384] conn(0x7fb2980b2ac0 0x7fb2980b4fd0 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7fb2a8004790 tx=0x7fb2a8013a00 comp rx=0 tx=0).stop 2024-05-27T07:36:17.430 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:17.433+0000 7fb2a1548700 1 -- 172.21.15.52:0/1404061159 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fb2ac10a920 msgr2=0x7fb2ac116de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:36:17.430 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:17.433+0000 7fb2a1548700 1 --2- 172.21.15.52:0/1404061159 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fb2ac10a920 0x7fb2ac116de0 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7fb2a400bd40 tx=0x7fb2a400be20 comp rx=0 tx=0).stop 2024-05-27T07:36:17.431 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:17.434+0000 7fb2a1548700 1 -- 172.21.15.52:0/1404061159 shutdown_connections 2024-05-27T07:36:17.431 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:17.434+0000 7fb2a1548700 1 --2- 172.21.15.52:0/1404061159 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fb2ac111bc0 0x7fb2ac117880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:36:17.431 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:17.434+0000 7fb2a1548700 1 --2- 172.21.15.52:0/1404061159 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fb2ac10a920 0x7fb2ac116de0 unknown :-1 s=CLOSED pgs=219 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:36:17.431 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:17.434+0000 7fb2a1548700 1 --2- 172.21.15.52:0/1404061159 >> [v2:172.21.15.52:6800/1822318384,v1:172.21.15.52:6801/1822318384] conn(0x7fb2980b2ac0 0x7fb2980b4fd0 unknown :-1 s=CLOSED pgs=193 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:36:17.431 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:17.434+0000 7fb2a1548700 1 --2- 172.21.15.52:0/1404061159 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fb2ac10b270 0x7fb2ac117320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:36:17.431 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:17.434+0000 7fb2a1548700 1 -- 172.21.15.52:0/1404061159 >> 172.21.15.52:0/1404061159 conn(0x7fb2ac06f4c0 msgr2=0x7fb2ac1104f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:36:17.431 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:17.434+0000 7fb2a1548700 1 -- 172.21.15.52:0/1404061159 shutdown_connections 2024-05-27T07:36:17.431 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:17.434+0000 7fb2a1548700 1 -- 172.21.15.52:0/1404061159 wait complete. 2024-05-27T07:36:18.186 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-05-27T07:36:18.187 DEBUG:teuthology.orchestra.run.smithi052:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:38f9d900c83c63799fbdbe61acc9a11b0d3554a6 shell --fsid 73df2636-1bf8-11ef-bc9b-c7b262605968 -- ceph fs dump --format=json 2024-05-27T07:36:18.487 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:36:18 smithi052 ceph-mon[31955]: pgmap v349: 129 pgs: 22 active+clean+snaptrim, 70 active+clean+snaptrim_wait, 37 active+clean; 636 MiB data, 6.8 GiB used, 1.0 TiB / 1.0 TiB avail; 184 B/s rd, 8.4 KiB/s wr, 47 op/s 2024-05-27T07:36:18.608 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:36:18 smithi173 ceph-mon[37000]: pgmap v349: 129 pgs: 22 active+clean+snaptrim, 70 active+clean+snaptrim_wait, 37 active+clean; 636 MiB data, 6.8 GiB used, 1.0 TiB / 1.0 TiB avail; 184 B/s rd, 8.4 KiB/s wr, 47 op/s 2024-05-27T07:36:18.695 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:36:18 smithi105 ceph-mon[35977]: pgmap v349: 129 pgs: 22 active+clean+snaptrim, 70 active+clean+snaptrim_wait, 37 active+clean; 636 MiB data, 6.8 GiB used, 1.0 TiB / 1.0 TiB avail; 184 B/s rd, 8.4 KiB/s wr, 47 op/s 2024-05-27T07:36:18.745 INFO:teuthology.orchestra.run.smithi052.stderr:Inferring config /var/lib/ceph/73df2636-1bf8-11ef-bc9b-c7b262605968/mon.a/config 2024-05-27T07:36:20.608 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:36:20 smithi173 ceph-mon[37000]: pgmap v350: 129 pgs: 21 active+clean+snaptrim, 67 active+clean+snaptrim_wait, 41 active+clean; 635 MiB data, 6.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 11 KiB/s wr, 12 op/s 2024-05-27T07:36:20.696 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:36:20 smithi105 ceph-mon[35977]: pgmap v350: 129 pgs: 21 active+clean+snaptrim, 67 active+clean+snaptrim_wait, 41 active+clean; 635 MiB data, 6.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 11 KiB/s wr, 12 op/s 2024-05-27T07:36:20.735 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:36:20 smithi052 ceph-mon[31955]: pgmap v350: 129 pgs: 21 active+clean+snaptrim, 67 active+clean+snaptrim_wait, 41 active+clean; 635 MiB data, 6.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 11 KiB/s wr, 12 op/s 2024-05-27T07:36:22.110 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:22.114+0000 7fa742285700 1 -- 172.21.15.52:0/1907246750 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fa73c10a920 msgr2=0x7fa73c10ad30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:36:22.111 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:22.114+0000 7fa742285700 1 --2- 172.21.15.52:0/1907246750 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fa73c10a920 0x7fa73c10ad30 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7fa72c01a9a0 tx=0x7fa72c01acb0 comp rx=0 tx=0).stop 2024-05-27T07:36:22.113 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:22.118+0000 7fa742285700 1 -- 172.21.15.52:0/1907246750 shutdown_connections 2024-05-27T07:36:22.114 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:22.118+0000 7fa742285700 1 --2- 172.21.15.52:0/1907246750 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fa73c11d920 0x7fa73c11fd40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:36:22.114 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:22.118+0000 7fa742285700 1 --2- 172.21.15.52:0/1907246750 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fa73c10a920 0x7fa73c10ad30 unknown :-1 s=CLOSED pgs=222 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:36:22.114 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:22.118+0000 7fa742285700 1 --2- 172.21.15.52:0/1907246750 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fa73c10b270 0x7fa73c11d3e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:36:22.114 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:22.118+0000 7fa742285700 1 -- 172.21.15.52:0/1907246750 >> 172.21.15.52:0/1907246750 conn(0x7fa73c06f4c0 msgr2=0x7fa73c06f900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:36:22.115 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:22.120+0000 7fa742285700 1 -- 172.21.15.52:0/1907246750 shutdown_connections 2024-05-27T07:36:22.116 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:22.120+0000 7fa742285700 1 -- 172.21.15.52:0/1907246750 wait complete. 2024-05-27T07:36:22.117 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:22.121+0000 7fa742285700 1 Processor -- start 2024-05-27T07:36:22.117 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:22.121+0000 7fa742285700 1 -- start start 2024-05-27T07:36:22.117 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:22.121+0000 7fa742285700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fa73c10a920 0x7fa73c11d1c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:36:22.118 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:22.122+0000 7fa742285700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fa73c10b270 0x7fa73c119300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:36:22.118 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:22.122+0000 7fa73bfff700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fa73c10a920 0x7fa73c11d1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:36:22.118 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:22.122+0000 7fa73bfff700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fa73c10a920 0x7fa73c11d1c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.52:54990/0 (socket says 172.21.15.52:54990) 2024-05-27T07:36:22.118 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:22.122+0000 7fa742285700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fa73c11d920 0x7fa73c119840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:36:22.118 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:22.122+0000 7fa73bfff700 1 -- 172.21.15.52:0/608971972 learned_addr learned my addr 172.21.15.52:0/608971972 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-05-27T07:36:22.119 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:22.122+0000 7fa73b7fe700 1 --2- 172.21.15.52:0/608971972 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fa73c10b270 0x7fa73c119300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:36:22.119 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:22.122+0000 7fa740822700 1 --2- 172.21.15.52:0/608971972 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fa73c11d920 0x7fa73c119840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:36:22.119 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:22.122+0000 7fa742285700 1 -- 172.21.15.52:0/608971972 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x7fa73c1153e0 con 0x7fa73c10a920 2024-05-27T07:36:22.119 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:22.122+0000 7fa742285700 1 -- 172.21.15.52:0/608971972 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7fa73c115260 con 0x7fa73c10b270 2024-05-27T07:36:22.119 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:22.122+0000 7fa742285700 1 -- 172.21.15.52:0/608971972 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7fa73c115560 con 0x7fa73c11d920 2024-05-27T07:36:22.120 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:22.123+0000 7fa73bfff700 1 -- 172.21.15.52:0/608971972 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fa73c11d920 msgr2=0x7fa73c119840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:36:22.120 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:22.123+0000 7fa73bfff700 1 --2- 172.21.15.52:0/608971972 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fa73c11d920 0x7fa73c119840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:36:22.120 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:22.123+0000 7fa73bfff700 1 -- 172.21.15.52:0/608971972 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fa73c10b270 msgr2=0x7fa73c119300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:36:22.120 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:22.123+0000 7fa73bfff700 1 --2- 172.21.15.52:0/608971972 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fa73c10b270 0x7fa73c119300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:36:22.120 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:22.123+0000 7fa73bfff700 1 -- 172.21.15.52:0/608971972 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa73c119fd0 con 0x7fa73c10a920 2024-05-27T07:36:22.120 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:22.123+0000 7fa740822700 1 --2- 172.21.15.52:0/608971972 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fa73c11d920 0x7fa73c119840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-27T07:36:22.120 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:22.123+0000 7fa73bfff700 1 --2- 172.21.15.52:0/608971972 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fa73c10a920 0x7fa73c11d1c0 secure :-1 s=READY pgs=756 cs=0 l=1 rev1=1 crypto rx=0x7fa72c005f50 tx=0x7fa72c0286d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:36:22.122 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:22.127+0000 7fa7397fa700 1 -- 172.21.15.52:0/608971972 <== mon.0 v2:172.21.15.52:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa72c028ab0 con 0x7fa73c10a920 2024-05-27T07:36:22.122 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:22.127+0000 7fa7397fa700 1 -- 172.21.15.52:0/608971972 <== mon.0 v2:172.21.15.52:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fa72c0054e0 con 0x7fa73c10a920 2024-05-27T07:36:22.123 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:22.127+0000 7fa7397fa700 1 -- 172.21.15.52:0/608971972 <== mon.0 v2:172.21.15.52:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa72c004b50 con 0x7fa73c10a920 2024-05-27T07:36:22.124 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:22.128+0000 7fa742285700 1 -- 172.21.15.52:0/608971972 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa73c1ad8b0 con 0x7fa73c10a920 2024-05-27T07:36:22.125 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:22.128+0000 7fa742285700 1 -- 172.21.15.52:0/608971972 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa73c1adcb0 con 0x7fa73c10a920 2024-05-27T07:36:22.135 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:22.137+0000 7fa7397fa700 1 -- 172.21.15.52:0/608971972 <== mon.0 v2:172.21.15.52:3300/0 4 ==== mgrmap(e 26) ==== 149783+0+0 (secure 0 0 0) 0x7fa72c004cf0 con 0x7fa73c10a920 2024-05-27T07:36:22.135 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:22.138+0000 7fa7397fa700 1 --2- 172.21.15.52:0/608971972 >> [v2:172.21.15.52:6800/1822318384,v1:172.21.15.52:6801/1822318384] conn(0x7fa7240b2c00 0x7fa7240b5110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:36:22.135 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:22.138+0000 7fa722ffd700 1 -- 172.21.15.52:0/608971972 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa71c005330 con 0x7fa73c10a920 2024-05-27T07:36:22.135 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:22.139+0000 7fa7397fa700 1 -- 172.21.15.52:0/608971972 <== mon.0 v2:172.21.15.52:3300/0 5 ==== osd_map(127..127 src has 1..127) ==== 9555+0+0 (secure 0 0 0) 0x7fa72c0f5530 con 0x7fa73c10a920 2024-05-27T07:36:22.145 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:22.144+0000 7fa73b7fe700 1 --2- 172.21.15.52:0/608971972 >> [v2:172.21.15.52:6800/1822318384,v1:172.21.15.52:6801/1822318384] conn(0x7fa7240b2c00 0x7fa7240b5110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:36:22.147 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:22.150+0000 7fa7397fa700 1 -- 172.21.15.52:0/608971972 <== mon.0 v2:172.21.15.52:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191963 (secure 0 0 0) 0x7fa72c024070 con 0x7fa73c10a920 2024-05-27T07:36:22.156 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:22.159+0000 7fa73b7fe700 1 --2- 172.21.15.52:0/608971972 >> [v2:172.21.15.52:6800/1822318384,v1:172.21.15.52:6801/1822318384] conn(0x7fa7240b2c00 0x7fa7240b5110 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7fa730005950 tx=0x7fa7300058e0 comp rx=0 tx=0).ready entity=mgr.15333 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:36:22.485 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:36:22 smithi052 ceph-mon[31955]: pgmap v351: 129 pgs: 20 active+clean+snaptrim, 60 active+clean+snaptrim_wait, 49 active+clean; 634 MiB data, 6.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3.7 KiB/s wr, 2 op/s 2024-05-27T07:36:22.608 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:36:22 smithi173 ceph-mon[37000]: pgmap v351: 129 pgs: 20 active+clean+snaptrim, 60 active+clean+snaptrim_wait, 49 active+clean; 634 MiB data, 6.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3.7 KiB/s wr, 2 op/s 2024-05-27T07:36:22.655 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:22.659+0000 7fa722ffd700 1 -- 172.21.15.52:0/608971972 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fa71c005620 con 0x7fa73c10a920 2024-05-27T07:36:22.656 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:22.660+0000 7fa7397fa700 1 -- 172.21.15.52:0/608971972 <== mon.0 v2:172.21.15.52:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 108 v108) ==== 95+0+6124 (secure 0 0 0) 0x7fa72c0bcf00 con 0x7fa73c10a920 2024-05-27T07:36:22.656 INFO:teuthology.orchestra.run.smithi052.stdout: 2024-05-27T07:36:22.656 INFO:teuthology.orchestra.run.smithi052.stdout:{"epoch":108,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":108,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-05-27T07:21:50.521783+0000","modified":"2024-05-27T07:35:15.151893+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24527,"mds_1":24497,"mds_2":24475,"mds_3":24473,"mds_4":24466},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24466":{"gid":24466,"name":"c","rank":4,"incarnation":26,"state":"up:active","state_seq":25,"addr":"172.21.15.173:6837/1396363827","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6836","nonce":1396363827},{"type":"v1","addr":"172.21.15.173:6837","nonce":1396363827}]},"join_fscid":-1,"export_targets":[0],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24473":{"gid":24473,"name":"e","rank":3,"incarnation":24,"state":"up:active","state_seq":26,"addr":"172.21.15.105:6835/3298498384","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6834","nonce":3298498384},{"type":"v1","addr":"172.21.15.105:6835","nonce":3298498384}]},"join_fscid":-1,"export_targets":[0],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24475":{"gid":24475,"name":"i","rank":2,"incarnation":22,"state":"up:active","state_seq":23,"addr":"172.21.15.173:6839/564778293","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6838","nonce":564778293},{"type":"v1","addr":"172.21.15.173:6839","nonce":564778293}]},"join_fscid":-1,"export_targets":[0],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24497":{"gid":24497,"name":"h","rank":1,"incarnation":20,"state":"up:active","state_seq":24,"addr":"172.21.15.105:6837/191393149","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6836","nonce":191393149},{"type":"v1","addr":"172.21.15.105:6837","nonce":191393149}]},"join_fscid":-1,"export_targets":[0],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24527":{"gid":24527,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":5,"addr":"172.21.15.105:6839/500348151","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6838","nonce":500348151},{"type":"v1","addr":"172.21.15.105:6839","nonce":500348151}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24527,"qdb_cluster":[24497,24527,24475,24473,24466]},"id":1}]} 2024-05-27T07:36:22.662 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:22.665+0000 7fa722ffd700 1 -- 172.21.15.52:0/608971972 >> [v2:172.21.15.52:6800/1822318384,v1:172.21.15.52:6801/1822318384] conn(0x7fa7240b2c00 msgr2=0x7fa7240b5110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:36:22.662 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:22.665+0000 7fa722ffd700 1 --2- 172.21.15.52:0/608971972 >> [v2:172.21.15.52:6800/1822318384,v1:172.21.15.52:6801/1822318384] conn(0x7fa7240b2c00 0x7fa7240b5110 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7fa730005950 tx=0x7fa7300058e0 comp rx=0 tx=0).stop 2024-05-27T07:36:22.663 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:22.665+0000 7fa722ffd700 1 -- 172.21.15.52:0/608971972 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fa73c10a920 msgr2=0x7fa73c11d1c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:36:22.663 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:22.665+0000 7fa722ffd700 1 --2- 172.21.15.52:0/608971972 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fa73c10a920 0x7fa73c11d1c0 secure :-1 s=READY pgs=756 cs=0 l=1 rev1=1 crypto rx=0x7fa72c005f50 tx=0x7fa72c0286d0 comp rx=0 tx=0).stop 2024-05-27T07:36:22.663 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:22.666+0000 7fa722ffd700 1 -- 172.21.15.52:0/608971972 shutdown_connections 2024-05-27T07:36:22.663 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:22.666+0000 7fa722ffd700 1 --2- 172.21.15.52:0/608971972 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fa73c11d920 0x7fa73c119840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:36:22.663 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:22.666+0000 7fa722ffd700 1 --2- 172.21.15.52:0/608971972 >> [v2:172.21.15.52:6800/1822318384,v1:172.21.15.52:6801/1822318384] conn(0x7fa7240b2c00 0x7fa7240b5110 unknown :-1 s=CLOSED pgs=194 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:36:22.663 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:22.666+0000 7fa722ffd700 1 --2- 172.21.15.52:0/608971972 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fa73c10a920 0x7fa73c11d1c0 unknown :-1 s=CLOSED pgs=756 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:36:22.663 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:22.666+0000 7fa722ffd700 1 --2- 172.21.15.52:0/608971972 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fa73c10b270 0x7fa73c119300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:36:22.663 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:22.666+0000 7fa722ffd700 1 -- 172.21.15.52:0/608971972 >> 172.21.15.52:0/608971972 conn(0x7fa73c06f4c0 msgr2=0x7fa73c11df70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:36:22.666 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:22.670+0000 7fa722ffd700 1 -- 172.21.15.52:0/608971972 shutdown_connections 2024-05-27T07:36:22.670 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:22.672+0000 7fa722ffd700 1 -- 172.21.15.52:0/608971972 wait complete. 2024-05-27T07:36:22.675 INFO:teuthology.orchestra.run.smithi052.stderr:dumped fsmap epoch 108 2024-05-27T07:36:22.695 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:36:22 smithi105 ceph-mon[35977]: pgmap v351: 129 pgs: 20 active+clean+snaptrim, 60 active+clean+snaptrim_wait, 49 active+clean; 634 MiB data, 6.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3.7 KiB/s wr, 2 op/s 2024-05-27T07:36:23.532 DEBUG:teuthology.orchestra.run.smithi052:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:38f9d900c83c63799fbdbe61acc9a11b0d3554a6 shell --fsid 73df2636-1bf8-11ef-bc9b-c7b262605968 -- ceph tell mds.1:4 damage ls 2024-05-27T07:36:23.584 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:36:23 smithi052 ceph-mon[31955]: from='client.? 172.21.15.52:0/608971972' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-27T07:36:23.607 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:36:23 smithi173 ceph-mon[37000]: from='client.? 172.21.15.52:0/608971972' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-27T07:36:23.696 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:36:23 smithi105 ceph-mon[35977]: from='client.? 172.21.15.52:0/608971972' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-27T07:36:24.061 INFO:teuthology.orchestra.run.smithi052.stderr:Inferring config /var/lib/ceph/73df2636-1bf8-11ef-bc9b-c7b262605968/mon.a/config 2024-05-27T07:36:24.485 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:36:24 smithi052 ceph-mon[31955]: pgmap v352: 129 pgs: 20 active+clean+snaptrim, 60 active+clean+snaptrim_wait, 49 active+clean; 634 MiB data, 7.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3.2 KiB/s wr, 2 op/s 2024-05-27T07:36:24.608 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:36:24 smithi173 ceph-mon[37000]: pgmap v352: 129 pgs: 20 active+clean+snaptrim, 60 active+clean+snaptrim_wait, 49 active+clean; 634 MiB data, 7.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3.2 KiB/s wr, 2 op/s 2024-05-27T07:36:24.695 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:36:24 smithi105 ceph-mon[35977]: pgmap v352: 129 pgs: 20 active+clean+snaptrim, 60 active+clean+snaptrim_wait, 49 active+clean; 634 MiB data, 7.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3.2 KiB/s wr, 2 op/s 2024-05-27T07:36:26.608 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:36:26 smithi173 ceph-mon[37000]: pgmap v353: 129 pgs: 18 active+clean+snaptrim, 55 active+clean+snaptrim_wait, 56 active+clean; 633 MiB data, 7.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3.2 KiB/s wr, 2 op/s 2024-05-27T07:36:26.696 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:36:26 smithi105 ceph-mon[35977]: pgmap v353: 129 pgs: 18 active+clean+snaptrim, 55 active+clean+snaptrim_wait, 56 active+clean; 633 MiB data, 7.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3.2 KiB/s wr, 2 op/s 2024-05-27T07:36:26.712 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:36:26 smithi052 ceph-mon[31955]: pgmap v353: 129 pgs: 18 active+clean+snaptrim, 55 active+clean+snaptrim_wait, 56 active+clean; 633 MiB data, 7.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3.2 KiB/s wr, 2 op/s 2024-05-27T07:36:27.340 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:27.344+0000 7f9301bfd700 1 --2- 172.21.15.52:0/3736081906 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f92fc10b100 0x7f92fc11d3e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-27T07:36:27.342 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:27.345+0000 7f9304662700 1 -- 172.21.15.52:0/3736081906 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f92fc10a7b0 msgr2=0x7f92fc10abc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:36:27.342 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:27.345+0000 7f9304662700 1 --2- 172.21.15.52:0/3736081906 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f92fc10a7b0 0x7f92fc10abc0 secure :-1 s=READY pgs=757 cs=0 l=1 rev1=1 crypto rx=0x7f92f401a9a0 tx=0x7f92f401acb0 comp rx=0 tx=0).stop 2024-05-27T07:36:27.349 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:27.353+0000 7f9304662700 1 -- 172.21.15.52:0/3736081906 shutdown_connections 2024-05-27T07:36:27.350 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:27.353+0000 7f9304662700 1 --2- 172.21.15.52:0/3736081906 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f92fc11d920 0x7f92fc11fd40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:36:27.350 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:27.353+0000 7f9304662700 1 --2- 172.21.15.52:0/3736081906 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f92fc10a7b0 0x7f92fc10abc0 unknown :-1 s=CLOSED pgs=757 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:36:27.350 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:27.353+0000 7f9304662700 1 --2- 172.21.15.52:0/3736081906 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f92fc10b100 0x7f92fc11d3e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:36:27.350 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:27.353+0000 7f9304662700 1 -- 172.21.15.52:0/3736081906 >> 172.21.15.52:0/3736081906 conn(0x7f92fc06f280 msgr2=0x7f92fc06f6c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:36:27.355 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:27.359+0000 7f9304662700 1 -- 172.21.15.52:0/3736081906 shutdown_connections 2024-05-27T07:36:27.355 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:27.359+0000 7f9304662700 1 -- 172.21.15.52:0/3736081906 wait complete. 2024-05-27T07:36:27.356 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:27.360+0000 7f9304662700 1 Processor -- start 2024-05-27T07:36:27.360 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:27.363+0000 7f9304662700 1 -- start start 2024-05-27T07:36:27.360 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:27.364+0000 7f9304662700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f92fc10a7b0 0x7f92fc11d2c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:36:27.360 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:27.364+0000 7f9304662700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f92fc10b100 0x7f92fc119400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:36:27.360 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:27.364+0000 7f9304662700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f92fc11d920 0x7f92fc119980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:36:27.361 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:27.364+0000 7f9304662700 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x7f92fc1154a0 con 0x7f92fc10b100 2024-05-27T07:36:27.361 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:27.364+0000 7f9304662700 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f92fc115320 con 0x7f92fc11d920 2024-05-27T07:36:27.361 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:27.364+0000 7f9304662700 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f92fc115620 con 0x7f92fc10a7b0 2024-05-27T07:36:27.362 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:27.366+0000 7f9301bfd700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f92fc10b100 0x7f92fc119400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:36:27.362 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:27.366+0000 7f9301bfd700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f92fc10b100 0x7f92fc119400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.52:55008/0 (socket says 172.21.15.52:55008) 2024-05-27T07:36:27.363 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:27.366+0000 7f9301bfd700 1 -- 172.21.15.52:0/3761724207 learned_addr learned my addr 172.21.15.52:0/3761724207 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-05-27T07:36:27.363 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:27.366+0000 7f9302bff700 1 --2- 172.21.15.52:0/3761724207 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f92fc11d920 0x7f92fc119980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:36:27.363 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:27.366+0000 7f93023fe700 1 --2- 172.21.15.52:0/3761724207 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f92fc10a7b0 0x7f92fc11d2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:36:27.365 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:27.368+0000 7f9302bff700 1 -- 172.21.15.52:0/3761724207 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f92fc10a7b0 msgr2=0x7f92fc11d2c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:36:27.365 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:27.368+0000 7f9302bff700 1 --2- 172.21.15.52:0/3761724207 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f92fc10a7b0 0x7f92fc11d2c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:36:27.365 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:27.368+0000 7f9302bff700 1 -- 172.21.15.52:0/3761724207 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f92fc10b100 msgr2=0x7f92fc119400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:36:27.365 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:27.368+0000 7f9302bff700 1 --2- 172.21.15.52:0/3761724207 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f92fc10b100 0x7f92fc119400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:36:27.365 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:27.368+0000 7f9302bff700 1 -- 172.21.15.52:0/3761724207 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f92fc11a1b0 con 0x7f92fc11d920 2024-05-27T07:36:27.365 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:27.369+0000 7f9302bff700 1 --2- 172.21.15.52:0/3761724207 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f92fc11d920 0x7f92fc119980 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0x7f92ec00be00 tx=0x7f92ec009ed0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:36:27.368 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:27.372+0000 7f92fb7fe700 1 -- 172.21.15.52:0/3761724207 <== mon.2 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f92ec019070 con 0x7f92fc11d920 2024-05-27T07:36:27.368 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:27.372+0000 7f92fb7fe700 1 -- 172.21.15.52:0/3761724207 <== mon.2 v2:172.21.15.105:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f92ec00cb90 con 0x7f92fc11d920 2024-05-27T07:36:27.368 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:27.372+0000 7f92fb7fe700 1 -- 172.21.15.52:0/3761724207 <== mon.2 v2:172.21.15.105:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f92ec0125a0 con 0x7f92fc11d920 2024-05-27T07:36:27.370 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:27.375+0000 7f9304662700 1 -- 172.21.15.52:0/3761724207 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f92fc1ad760 con 0x7f92fc11d920 2024-05-27T07:36:27.370 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:27.375+0000 7f9304662700 1 -- 172.21.15.52:0/3761724207 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f92fc1adc20 con 0x7f92fc11d920 2024-05-27T07:36:27.378 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:27.380+0000 7f9313077b80 1 Processor -- start 2024-05-27T07:36:27.378 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:27.380+0000 7f9313077b80 1 -- start start 2024-05-27T07:36:27.378 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:27.381+0000 7f9313077b80 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x555894edce40 0x55589505aeb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:36:27.378 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:27.381+0000 7f9313077b80 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55589505b3f0 0x555895066010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:36:27.378 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:27.381+0000 7f9313077b80 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55589505b800 0x555895068560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:36:27.378 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:27.381+0000 7f9313077b80 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x555894f89d70 con 0x55589505b3f0 2024-05-27T07:36:27.378 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:27.381+0000 7f9313077b80 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x555894f79910 con 0x555894edce40 2024-05-27T07:36:27.378 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:27.381+0000 7f9313077b80 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x555894f7d950 con 0x55589505b800 2024-05-27T07:36:27.378 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:27.381+0000 7f9301bfd700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x555894edce40 0x55589505aeb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:36:27.378 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:27.381+0000 7f9301bfd700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x555894edce40 0x55589505aeb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.52:53160/0 (socket says 172.21.15.52:53160) 2024-05-27T07:36:27.378 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:27.381+0000 7f9301bfd700 1 -- 172.21.15.52:0/219309907 learned_addr learned my addr 172.21.15.52:0/219309907 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-05-27T07:36:27.379 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:27.381+0000 7f9301bfd700 1 --2- 172.21.15.52:0/219309907 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55589505b800 0x555895068560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:36:27.381 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:27.384+0000 7f9301bfd700 1 -- 172.21.15.52:0/219309907 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x555894edce40 msgr2=0x55589505aeb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:36:27.381 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:27.384+0000 7f9301bfd700 1 --2- 172.21.15.52:0/219309907 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x555894edce40 0x55589505aeb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:36:27.382 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:27.384+0000 7f9301bfd700 1 -- 172.21.15.52:0/219309907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55589505b3f0 msgr2=0x555895066010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:36:27.382 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:27.384+0000 7f9301bfd700 1 --2- 172.21.15.52:0/219309907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55589505b3f0 0x555895066010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:36:27.382 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:27.384+0000 7f9301bfd700 1 -- 172.21.15.52:0/219309907 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x555894f22fb0 con 0x55589505b800 2024-05-27T07:36:27.382 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:27.385+0000 7f92fb7fe700 1 -- 172.21.15.52:0/3761724207 <== mon.2 v2:172.21.15.105:3300/0 4 ==== mgrmap(e 26) ==== 149783+0+0 (secure 0 0 0) 0x7f92ec01c070 con 0x7f92fc11d920 2024-05-27T07:36:27.382 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:27.386+0000 7f9301bfd700 1 --2- 172.21.15.52:0/219309907 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55589505b800 0x555895068560 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7f92f0033980 tx=0x7f92f0009260 comp rx=0 tx=0).ready entity=mon.1 client_cookie=92f1985c1ae02d server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:36:27.385 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:27.386+0000 7f92e3fff700 1 -- 172.21.15.52:0/219309907 <== mon.1 v2:172.21.15.173:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f92f000f480 con 0x55589505b800 2024-05-27T07:36:27.385 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:27.386+0000 7f92e3fff700 1 -- 172.21.15.52:0/219309907 <== mon.1 v2:172.21.15.173:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f92f0033640 con 0x55589505b800 2024-05-27T07:36:27.385 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:27.386+0000 7f92e3fff700 1 -- 172.21.15.52:0/219309907 <== mon.1 v2:172.21.15.173:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f92f0040830 con 0x55589505b800 2024-05-27T07:36:27.385 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:27.388+0000 7f9313077b80 1 -- 172.21.15.52:0/219309907 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55589505b800 msgr2=0x555895068560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:36:27.385 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:27.388+0000 7f9313077b80 1 --2- 172.21.15.52:0/219309907 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55589505b800 0x555895068560 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7f92f0033980 tx=0x7f92f0009260 comp rx=0 tx=0).stop 2024-05-27T07:36:27.385 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:27.388+0000 7f92fb7fe700 1 --2- 172.21.15.52:0/3761724207 >> [v2:172.21.15.52:6800/1822318384,v1:172.21.15.52:6801/1822318384] conn(0x7f92e40b2c00 0x7f92e40b5110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:36:27.385 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:27.388+0000 7f92fb7fe700 1 -- 172.21.15.52:0/3761724207 <== mon.2 v2:172.21.15.105:3300/0 5 ==== osd_map(127..127 src has 1..127) ==== 9555+0+0 (secure 0 0 0) 0x7f92ec0e1e20 con 0x7f92fc11d920 2024-05-27T07:36:27.386 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:27.389+0000 7f9301bfd700 1 --2- 172.21.15.52:0/3761724207 >> [v2:172.21.15.52:6800/1822318384,v1:172.21.15.52:6801/1822318384] conn(0x7f92e40b2c00 0x7f92e40b5110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:36:27.387 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:27.389+0000 7f9313077b80 1 -- 172.21.15.52:0/219309907 shutdown_connections 2024-05-27T07:36:27.387 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:27.389+0000 7f9313077b80 1 --2- 172.21.15.52:0/219309907 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55589505b800 0x555895068560 unknown :-1 s=CLOSED pgs=223 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:36:27.387 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:27.389+0000 7f9313077b80 1 --2- 172.21.15.52:0/219309907 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x555894edce40 0x55589505aeb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:36:27.387 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:27.389+0000 7f9313077b80 1 --2- 172.21.15.52:0/219309907 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55589505b3f0 0x555895066010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:36:27.387 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:27.389+0000 7f9313077b80 1 -- 172.21.15.52:0/219309907 >> 172.21.15.52:0/219309907 conn(0x555894edf770 msgr2=0x5558950571f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:36:27.387 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:27.390+0000 7f9301bfd700 1 --2- 172.21.15.52:0/3761724207 >> [v2:172.21.15.52:6800/1822318384,v1:172.21.15.52:6801/1822318384] conn(0x7f92e40b2c00 0x7f92e40b5110 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7f92f0033980 tx=0x7f92f00409d0 comp rx=0 tx=0).ready entity=mgr.15333 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:36:27.387 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:27.390+0000 7f9313077b80 1 -- 172.21.15.52:0/219309907 shutdown_connections 2024-05-27T07:36:27.388 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:27.390+0000 7f9313077b80 1 -- 172.21.15.52:0/219309907 wait complete. 2024-05-27T07:36:27.388 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:27.391+0000 7f9313077b80 1 -- start start 2024-05-27T07:36:27.388 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:27.391+0000 7f9313077b80 1 Processor -- start 2024-05-27T07:36:27.388 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:27.391+0000 7f9313077b80 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x555894edce40 0x555895068600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:36:27.388 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:27.391+0000 7f9313077b80 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55589505b3f0 0x555895065a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:36:27.392 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:27.391+0000 7f9313077b80 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55589505b800 0x5558950660d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:36:27.393 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:27.392+0000 7f9313077b80 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x555894e7ffa0 con 0x55589505b800 2024-05-27T07:36:27.393 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:27.392+0000 7f9313077b80 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x555894ee1da0 con 0x55589505b3f0 2024-05-27T07:36:27.393 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:27.392+0000 7f9313077b80 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x555895069040 con 0x555894edce40 2024-05-27T07:36:27.393 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:27.392+0000 7f9301bfd700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55589505b3f0 0x555895065a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:36:27.393 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:27.392+0000 7f9301bfd700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55589505b3f0 0x555895065a10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.52:53166/0 (socket says 172.21.15.52:53166) 2024-05-27T07:36:27.393 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:27.392+0000 7f9301bfd700 1 -- 172.21.15.52:0/2445661483 learned_addr learned my addr 172.21.15.52:0/2445661483 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-05-27T07:36:27.393 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:27.392+0000 7f93023fe700 1 --2- 172.21.15.52:0/2445661483 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x555894edce40 0x555895068600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:36:27.393 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:27.392+0000 7f9313077b80 10 client.0 _ll_register_callbacks cb 0x555894f105b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T07:36:27.393 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:27.392+0000 7f9301bfd700 1 -- 172.21.15.52:0/2445661483 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x555894edce40 msgr2=0x555895068600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:36:27.393 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:27.392+0000 7f9301bfd700 1 --2- 172.21.15.52:0/2445661483 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x555894edce40 0x555895068600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:36:27.394 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:27.392+0000 7f9301bfd700 1 -- 172.21.15.52:0/2445661483 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55589505b800 msgr2=0x5558950660d0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-05-27T07:36:27.394 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:27.392+0000 7f9302bff700 1 --2- 172.21.15.52:0/2445661483 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55589505b800 0x5558950660d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:36:27.394 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:27.392+0000 7f9301bfd700 1 --2- 172.21.15.52:0/2445661483 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55589505b800 0x5558950660d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:36:27.394 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:27.392+0000 7f9301bfd700 1 -- 172.21.15.52:0/2445661483 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({osdmap=0}) -- 0x555894ec98d0 con 0x55589505b3f0 2024-05-27T07:36:27.394 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:27.393+0000 7f9301bfd700 1 --2- 172.21.15.52:0/2445661483 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55589505b3f0 0x555895065a10 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x7f92f00046d0 tx=0x7f92f0009260 comp rx=0 tx=0).ready entity=mon.2 client_cookie=6c3bc8d861108f59 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:36:27.394 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:27.393+0000 7f92e1ffb700 10 client.0 ms_handle_connect on v2:172.21.15.105:3300/0 2024-05-27T07:36:27.394 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:27.395+0000 7f92e1ffb700 1 -- 172.21.15.52:0/2445661483 <== mon.2 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f92f0046040 con 0x55589505b3f0 2024-05-27T07:36:27.394 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:27.395+0000 7f92e1ffb700 1 -- 172.21.15.52:0/2445661483 <== mon.2 v2:172.21.15.105:3300/0 2 ==== osd_map(127..127 src has 1..127) ==== 9555+0+0 (secure 0 0 0) 0x7f92f003c040 con 0x55589505b3f0 2024-05-27T07:36:27.394 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:27.395+0000 7f9313077b80 1 -- 172.21.15.52:0/2445661483 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f92f0033640 con 0x55589505b3f0 2024-05-27T07:36:27.395 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:27.397+0000 7f92e1ffb700 1 -- 172.21.15.52:0/2445661483 <== mon.2 v2:172.21.15.105:3300/0 3 ==== mon_get_version_reply(handle=1 version=108) ==== 24+0+0 (secure 0 0 0) 0x7f92f00405d0 con 0x55589505b3f0 2024-05-27T07:36:27.396 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:27.399+0000 7f9313077b80 10 client.26912 fetch_fsmap learned FSMap version 108 2024-05-27T07:36:27.396 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:27.399+0000 7f9313077b80 1 -- 172.21.15.52:0/2445661483 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,fsmap=108,monmap=4+}) -- 0x555894f33760 con 0x55589505b3f0 2024-05-27T07:36:27.399 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:27.402+0000 7f92e1ffb700 1 -- 172.21.15.52:0/2445661483 <== mon.2 v2:172.21.15.105:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f92f0033640 con 0x55589505b3f0 2024-05-27T07:36:27.399 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:27.402+0000 7f92e1ffb700 1 -- 172.21.15.52:0/2445661483 <== mon.2 v2:172.21.15.105:3300/0 5 ==== fsmap(e 108) ==== 4203+0+0 (secure 0 0 0) 0x7f92f0063020 con 0x55589505b3f0 2024-05-27T07:36:27.399 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:27.402+0000 7f9313077b80 10 client.26912 fetch_fsmap finished waiting for FSMap version 108 2024-05-27T07:36:27.399 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:27.402+0000 7f9313077b80 10 client.26912 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.c 2024-05-27T07:36:27.399 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:27.402+0000 7f9313077b80 20 client.26912 populate_metadata read hostname 'smithi052' 2024-05-27T07:36:27.400 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:27.402+0000 7f9313077b80 1 --2- 172.21.15.52:0/2445661483 >> [v2:172.21.15.173:6836/1396363827,v1:172.21.15.173:6837/1396363827] conn(0x555895075f90 0x555895078420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:36:27.400 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:27.402+0000 7f9313077b80 4 client.26912 mds_command: new command op to 24466 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-27T07:36:27.400 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:27.402+0000 7f9313077b80 1 -- 172.21.15.52:0/2445661483 --> [v2:172.21.15.173:6836/1396363827,v1:172.21.15.173:6837/1396363827] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x555894f8bcc0 con 0x555895075f90 2024-05-27T07:36:27.400 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:27.403+0000 7f93023fe700 1 --2- 172.21.15.52:0/2445661483 >> [v2:172.21.15.173:6836/1396363827,v1:172.21.15.173:6837/1396363827] conn(0x555895075f90 0x555895078420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:36:27.406 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:27.409+0000 7f93023fe700 1 --2- 172.21.15.52:0/2445661483 >> [v2:172.21.15.173:6836/1396363827,v1:172.21.15.173:6837/1396363827] conn(0x555895075f90 0x555895078420 crc :-1 s=READY pgs=48 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=b5a6da1fe3dc34c5 server_cookie=98a463bd3e913565 in_seq=0 out_seq=0 2024-05-27T07:36:27.406 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:27.409+0000 7f92e1ffb700 10 client.26912 ms_handle_connect on v2:172.21.15.173:6836/1396363827 2024-05-27T07:36:27.406 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:27.410+0000 7f92e1ffb700 1 -- 172.21.15.52:0/2445661483 <== mds.4 v2:172.21.15.173:6836/1396363827 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24041 (crc 0 0 0) 0x555894f8bcc0 con 0x555895075f90 2024-05-27T07:36:27.406 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:27.410+0000 7f92e1ffb700 10 client.26912 handle_command_reply: tid=0 2024-05-27T07:36:27.406 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:27.410+0000 7f9313077b80 1 client.26912 shutdown 2024-05-27T07:36:27.406 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:27.411+0000 7f9313077b80 1 -- 172.21.15.52:0/2445661483 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55589505b3f0 msgr2=0x555895065a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:36:27.407 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:27.411+0000 7f9313077b80 1 --2- 172.21.15.52:0/2445661483 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55589505b3f0 0x555895065a10 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x7f92f00046d0 tx=0x7f92f0009260 comp rx=0 tx=0).stop 2024-05-27T07:36:27.408 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:27.412+0000 7f9313077b80 1 -- 172.21.15.52:0/2445661483 shutdown_connections 2024-05-27T07:36:27.408 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:27.412+0000 7f9313077b80 1 --2- 172.21.15.52:0/2445661483 >> [v2:172.21.15.173:6836/1396363827,v1:172.21.15.173:6837/1396363827] conn(0x555895075f90 0x555895078420 crc :-1 s=READY pgs=48 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:36:27.408 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:27.412+0000 7f9313077b80 1 --2- 172.21.15.52:0/2445661483 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55589505b800 0x5558950660d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:36:27.408 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:27.412+0000 7f9313077b80 1 --2- 172.21.15.52:0/2445661483 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x555894edce40 0x555895068600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:36:27.408 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:27.412+0000 7f9313077b80 1 --2- 172.21.15.52:0/2445661483 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55589505b3f0 0x555895065a10 unknown :-1 s=CLOSED pgs=266 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:36:27.408 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:27.412+0000 7f9313077b80 1 -- 172.21.15.52:0/2445661483 >> 172.21.15.52:0/2445661483 conn(0x555894edf770 msgr2=0x55589505bd50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:36:27.409 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:27.412+0000 7f92e1ffb700 0 client.26912 ms_handle_reset on v2:172.21.15.173:6836/1396363827 2024-05-27T07:36:27.409 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:27.414+0000 7f9313077b80 1 -- 172.21.15.52:0/2445661483 shutdown_connections 2024-05-27T07:36:27.409 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:27.414+0000 7f9313077b80 1 -- 172.21.15.52:0/2445661483 wait complete. 2024-05-27T07:36:27.410 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:27.414+0000 7f9313077b80 20 client.26912 trim_cache size 0 max 16384 2024-05-27T07:36:27.439 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:27.443+0000 7f9313077b80 1 Processor -- start 2024-05-27T07:36:27.555 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:27.446+0000 7f9313077b80 1 -- start start 2024-05-27T07:36:27.555 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:27.446+0000 7f9313077b80 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x555894edce40 0x555895057390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:36:27.555 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:27.446+0000 7f9313077b80 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55589505b3f0 0x555895072140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:36:27.555 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:27.446+0000 7f9313077b80 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55589505b800 0x555895060f00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:36:27.555 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:27.446+0000 7f9313077b80 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x7f92fc1b71d0 con 0x55589505b3f0 2024-05-27T07:36:27.556 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:27.446+0000 7f9313077b80 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f92fc1b5600 con 0x555894edce40 2024-05-27T07:36:27.556 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:27.446+0000 7f9313077b80 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f92fc1b6ad0 con 0x55589505b800 2024-05-27T07:36:27.556 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:27.448+0000 7f9302bff700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55589505b800 0x555895060f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:36:27.556 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:27.448+0000 7f9301bfd700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55589505b3f0 0x555895072140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:36:27.556 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:27.448+0000 7f9301bfd700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55589505b3f0 0x555895072140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.52:55018/0 (socket says 172.21.15.52:55018) 2024-05-27T07:36:27.556 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:27.448+0000 7f9301bfd700 1 -- 172.21.15.52:0/3891648206 learned_addr learned my addr 172.21.15.52:0/3891648206 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-05-27T07:36:27.556 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:27.448+0000 7f93023fe700 1 --2- 172.21.15.52:0/3891648206 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x555894edce40 0x555895057390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:36:27.556 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:27.451+0000 7f93023fe700 1 -- 172.21.15.52:0/3891648206 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55589505b800 msgr2=0x555895060f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:36:27.556 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:27.451+0000 7f93023fe700 1 --2- 172.21.15.52:0/3891648206 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55589505b800 0x555895060f00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:36:27.556 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:27.451+0000 7f93023fe700 1 -- 172.21.15.52:0/3891648206 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55589505b3f0 msgr2=0x555895072140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:36:27.556 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:27.451+0000 7f93023fe700 1 --2- 172.21.15.52:0/3891648206 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55589505b3f0 0x555895072140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:36:27.556 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:27.451+0000 7f93023fe700 1 -- 172.21.15.52:0/3891648206 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x555895038cb0 con 0x555894edce40 2024-05-27T07:36:27.557 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:27.451+0000 7f93023fe700 1 --2- 172.21.15.52:0/3891648206 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x555894edce40 0x555895057390 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 crypto rx=0x7f92f401a8f0 tx=0x7f92f4005040 comp rx=0 tx=0).ready entity=mon.2 client_cookie=63720f819ce96e2f server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:36:27.557 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:27.452+0000 7f92e37fe700 1 -- 172.21.15.52:0/3891648206 <== mon.2 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f92f4005390 con 0x555894edce40 2024-05-27T07:36:27.557 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:27.452+0000 7f92e37fe700 1 -- 172.21.15.52:0/3891648206 <== mon.2 v2:172.21.15.105:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f92f4005530 con 0x555894edce40 2024-05-27T07:36:27.557 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:27.453+0000 7f9313077b80 1 -- 172.21.15.52:0/3891648206 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x555894edce40 msgr2=0x555895057390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:36:27.557 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:27.453+0000 7f9313077b80 1 --2- 172.21.15.52:0/3891648206 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x555894edce40 0x555895057390 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 crypto rx=0x7f92f401a8f0 tx=0x7f92f4005040 comp rx=0 tx=0).stop 2024-05-27T07:36:27.557 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:27.454+0000 7f9313077b80 1 -- 172.21.15.52:0/3891648206 shutdown_connections 2024-05-27T07:36:27.557 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:27.454+0000 7f9313077b80 1 --2- 172.21.15.52:0/3891648206 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55589505b800 0x555895060f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:36:27.557 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:27.454+0000 7f9313077b80 1 --2- 172.21.15.52:0/3891648206 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x555894edce40 0x555895057390 unknown :-1 s=CLOSED pgs=267 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:36:27.557 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:27.454+0000 7f9313077b80 1 --2- 172.21.15.52:0/3891648206 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55589505b3f0 0x555895072140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:36:27.557 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:27.454+0000 7f9313077b80 1 -- 172.21.15.52:0/3891648206 >> 172.21.15.52:0/3891648206 conn(0x555894edf770 msgr2=0x555895056c80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:36:27.557 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:27.454+0000 7f9313077b80 1 -- 172.21.15.52:0/3891648206 shutdown_connections 2024-05-27T07:36:27.557 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:27.454+0000 7f9313077b80 1 -- 172.21.15.52:0/3891648206 wait complete. 2024-05-27T07:36:27.557 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:27.454+0000 7f9313077b80 1 -- start start 2024-05-27T07:36:27.557 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:27.455+0000 7f9313077b80 1 Processor -- start 2024-05-27T07:36:27.558 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:27.455+0000 7f9313077b80 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x555894edce40 0x55589506e540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:36:27.558 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:27.455+0000 7f9313077b80 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55589505b3f0 0x55589506ea80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:36:27.558 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:27.455+0000 7f9313077b80 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55589505b800 0x555895069310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:36:27.558 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:27.455+0000 7f9313077b80 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x7f92fc1b6400 con 0x55589505b3f0 2024-05-27T07:36:27.558 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:27.455+0000 7f9313077b80 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f92fc1b5d00 con 0x55589505b800 2024-05-27T07:36:27.558 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:27.455+0000 7f9313077b80 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x55589505d1d0 con 0x555894edce40 2024-05-27T07:36:27.558 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:27.455+0000 7f9313077b80 10 client.0 _ll_register_callbacks cb 0x555894f105b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T07:36:27.558 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:27.456+0000 7f9302bff700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55589505b800 0x555895069310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:36:27.558 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:27.456+0000 7f9302bff700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55589505b800 0x555895069310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.52:53178/0 (socket says 172.21.15.52:53178) 2024-05-27T07:36:27.558 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:27.456+0000 7f9302bff700 1 -- 172.21.15.52:0/3214974532 learned_addr learned my addr 172.21.15.52:0/3214974532 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-05-27T07:36:27.558 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:27.456+0000 7f9302bff700 1 -- 172.21.15.52:0/3214974532 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x555894edce40 msgr2=0x55589506e540 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-05-27T07:36:27.558 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:27.456+0000 7f9302bff700 1 --2- 172.21.15.52:0/3214974532 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x555894edce40 0x55589506e540 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:36:27.559 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:27.456+0000 7f9302bff700 1 -- 172.21.15.52:0/3214974532 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55589505b3f0 msgr2=0x55589506ea80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:36:27.559 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:27.456+0000 7f9302bff700 1 --2- 172.21.15.52:0/3214974532 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55589505b3f0 0x55589506ea80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:36:27.559 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:27.456+0000 7f9302bff700 1 -- 172.21.15.52:0/3214974532 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({osdmap=0}) -- 0x555895069ab0 con 0x55589505b800 2024-05-27T07:36:27.559 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:27.457+0000 7f9302bff700 1 --2- 172.21.15.52:0/3214974532 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55589505b800 0x555895069310 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 crypto rx=0x7f92ec0a6010 tx=0x7f92ec0a92e0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=442b41a5d71a389 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:36:27.559 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:27.457+0000 7f92e1ffb700 10 client.0 ms_handle_connect on v2:172.21.15.105:3300/0 2024-05-27T07:36:27.559 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:27.457+0000 7f92e1ffb700 1 -- 172.21.15.52:0/3214974532 <== mon.2 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f92ec0a94a0 con 0x55589505b800 2024-05-27T07:36:27.559 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:27.457+0000 7f9313077b80 1 -- 172.21.15.52:0/3214974532 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f92f4005530 con 0x55589505b800 2024-05-27T07:36:27.559 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:27.458+0000 7f92e1ffb700 1 -- 172.21.15.52:0/3214974532 <== mon.2 v2:172.21.15.105:3300/0 2 ==== osd_map(127..127 src has 1..127) ==== 9555+0+0 (secure 0 0 0) 0x7f92ec0a8210 con 0x55589505b800 2024-05-27T07:36:27.559 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:27.458+0000 7f92e1ffb700 1 -- 172.21.15.52:0/3214974532 <== mon.2 v2:172.21.15.105:3300/0 3 ==== mon_get_version_reply(handle=1 version=108) ==== 24+0+0 (secure 0 0 0) 0x7f92ec0a8da0 con 0x55589505b800 2024-05-27T07:36:27.559 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:27.458+0000 7f9313077b80 10 client.26918 fetch_fsmap learned FSMap version 108 2024-05-27T07:36:27.559 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:27.458+0000 7f9313077b80 1 -- 172.21.15.52:0/3214974532 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,fsmap=108,monmap=4+}) -- 0x5558950df520 con 0x55589505b800 2024-05-27T07:36:27.560 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:27.461+0000 7f92e1ffb700 1 -- 172.21.15.52:0/3214974532 <== mon.2 v2:172.21.15.105:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f92f4005530 con 0x55589505b800 2024-05-27T07:36:27.560 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:27.461+0000 7f92e1ffb700 1 -- 172.21.15.52:0/3214974532 <== mon.2 v2:172.21.15.105:3300/0 5 ==== fsmap(e 108) ==== 4203+0+0 (secure 0 0 0) 0x7f92ec0c1070 con 0x55589505b800 2024-05-27T07:36:27.560 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:27.461+0000 7f9313077b80 10 client.26918 fetch_fsmap finished waiting for FSMap version 108 2024-05-27T07:36:27.560 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:27.461+0000 7f9313077b80 10 client.26918 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.c 2024-05-27T07:36:27.560 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:27.461+0000 7f9313077b80 20 client.26918 populate_metadata read hostname 'smithi052' 2024-05-27T07:36:27.560 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:27.461+0000 7f9313077b80 1 --2- 172.21.15.52:0/3214974532 >> [v2:172.21.15.173:6836/1396363827,v1:172.21.15.173:6837/1396363827] conn(0x5558950dfc30 0x5558950e20d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:36:27.560 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:27.461+0000 7f9313077b80 4 client.26918 mds_command: new command op to 24466 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-05-27T07:36:27.560 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:27.461+0000 7f9313077b80 1 -- 172.21.15.52:0/3214974532 --> [v2:172.21.15.173:6836/1396363827,v1:172.21.15.173:6837/1396363827] -- command(tid 0: {"prefix": "damage ls"}) -- 0x555894f854a0 con 0x5558950dfc30 2024-05-27T07:36:27.560 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:27.461+0000 7f93023fe700 1 --2- 172.21.15.52:0/3214974532 >> [v2:172.21.15.173:6836/1396363827,v1:172.21.15.173:6837/1396363827] conn(0x5558950dfc30 0x5558950e20d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:36:27.560 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:27.466+0000 7f93023fe700 1 --2- 172.21.15.52:0/3214974532 >> [v2:172.21.15.173:6836/1396363827,v1:172.21.15.173:6837/1396363827] conn(0x5558950dfc30 0x5558950e20d0 crc :-1 s=READY pgs=49 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=cd8cdd33ffecf05d server_cookie=340d3dc3d26fdf59 in_seq=0 out_seq=0 2024-05-27T07:36:27.560 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:27.466+0000 7f92e1ffb700 10 client.26918 ms_handle_connect on v2:172.21.15.173:6836/1396363827 2024-05-27T07:36:27.561 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:27.468+0000 7f92e1ffb700 1 -- 172.21.15.52:0/3214974532 <== mds.4 v2:172.21.15.173:6836/1396363827 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x555894f854a0 con 0x5558950dfc30 2024-05-27T07:36:27.561 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:27.468+0000 7f92e1ffb700 10 client.26918 handle_command_reply: tid=0 2024-05-27T07:36:27.561 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:27.468+0000 7f9313077b80 1 client.26918 shutdown 2024-05-27T07:36:27.561 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:27.468+0000 7f9313077b80 1 -- 172.21.15.52:0/3214974532 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55589505b800 msgr2=0x555895069310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:36:27.561 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:27.468+0000 7f9313077b80 1 --2- 172.21.15.52:0/3214974532 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55589505b800 0x555895069310 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 crypto rx=0x7f92ec0a6010 tx=0x7f92ec0a92e0 comp rx=0 tx=0).stop 2024-05-27T07:36:27.561 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:27.470+0000 7f9313077b80 1 -- 172.21.15.52:0/3214974532 shutdown_connections 2024-05-27T07:36:27.561 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:27.470+0000 7f9313077b80 1 --2- 172.21.15.52:0/3214974532 >> [v2:172.21.15.173:6836/1396363827,v1:172.21.15.173:6837/1396363827] conn(0x5558950dfc30 0x5558950e20d0 crc :-1 s=READY pgs=49 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:36:27.561 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:27.470+0000 7f9313077b80 1 --2- 172.21.15.52:0/3214974532 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55589505b800 0x555895069310 unknown :-1 s=CLOSED pgs=268 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:36:27.561 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:27.470+0000 7f9313077b80 1 --2- 172.21.15.52:0/3214974532 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x555894edce40 0x55589506e540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:36:27.561 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:27.470+0000 7f9313077b80 1 --2- 172.21.15.52:0/3214974532 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55589505b3f0 0x55589506ea80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:36:27.561 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:27.470+0000 7f9313077b80 1 -- 172.21.15.52:0/3214974532 >> 172.21.15.52:0/3214974532 conn(0x555894edf770 msgr2=0x555895050e00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:36:27.561 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:27.475+0000 7f92e1ffb700 0 client.26918 ms_handle_reset on v2:172.21.15.173:6836/1396363827 2024-05-27T07:36:27.561 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:27.477+0000 7f9313077b80 1 -- 172.21.15.52:0/3214974532 shutdown_connections 2024-05-27T07:36:27.561 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:27.477+0000 7f9313077b80 1 -- 172.21.15.52:0/3214974532 wait complete. 2024-05-27T07:36:27.562 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:27.477+0000 7f9313077b80 20 client.26918 trim_cache size 0 max 16384 2024-05-27T07:36:27.562 INFO:teuthology.orchestra.run.smithi052.stdout:[] 2024-05-27T07:36:27.563 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:27.479+0000 7f92f97fa700 1 -- 172.21.15.52:0/3761724207 >> [v2:172.21.15.52:6800/1822318384,v1:172.21.15.52:6801/1822318384] conn(0x7f92e40b2c00 msgr2=0x7f92e40b5110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:36:27.563 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:27.479+0000 7f92f97fa700 1 --2- 172.21.15.52:0/3761724207 >> [v2:172.21.15.52:6800/1822318384,v1:172.21.15.52:6801/1822318384] conn(0x7f92e40b2c00 0x7f92e40b5110 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7f92f0033980 tx=0x7f92f00409d0 comp rx=0 tx=0).stop 2024-05-27T07:36:27.563 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:27.479+0000 7f92f97fa700 1 -- 172.21.15.52:0/3761724207 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f92fc11d920 msgr2=0x7f92fc119980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:36:27.563 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:27.479+0000 7f92f97fa700 1 --2- 172.21.15.52:0/3761724207 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f92fc11d920 0x7f92fc119980 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0x7f92ec00be00 tx=0x7f92ec009ed0 comp rx=0 tx=0).stop 2024-05-27T07:36:27.563 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:27.480+0000 7f92f97fa700 1 -- 172.21.15.52:0/3761724207 shutdown_connections 2024-05-27T07:36:27.563 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:27.480+0000 7f92f97fa700 1 --2- 172.21.15.52:0/3761724207 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f92fc11d920 0x7f92fc119980 unknown :-1 s=CLOSED pgs=265 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:36:27.564 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:27.480+0000 7f92f97fa700 1 --2- 172.21.15.52:0/3761724207 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f92fc10a7b0 0x7f92fc11d2c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:36:27.564 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:27.480+0000 7f92f97fa700 1 --2- 172.21.15.52:0/3761724207 >> [v2:172.21.15.52:6800/1822318384,v1:172.21.15.52:6801/1822318384] conn(0x7f92e40b2c00 0x7f92e40b5110 unknown :-1 s=CLOSED pgs=195 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:36:27.564 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:27.480+0000 7f92f97fa700 1 --2- 172.21.15.52:0/3761724207 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f92fc10b100 0x7f92fc119400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:36:27.564 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:27.480+0000 7f92f97fa700 1 -- 172.21.15.52:0/3761724207 >> 172.21.15.52:0/3761724207 conn(0x7f92fc06f280 msgr2=0x7f92fc11e010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:36:27.564 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:27.480+0000 7f92f97fa700 1 -- 172.21.15.52:0/3761724207 shutdown_connections 2024-05-27T07:36:27.564 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:27.480+0000 7f92f97fa700 1 -- 172.21.15.52:0/3761724207 wait complete. 2024-05-27T07:36:28.249 DEBUG:teuthology.orchestra.run.smithi052:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:38f9d900c83c63799fbdbe61acc9a11b0d3554a6 shell --fsid 73df2636-1bf8-11ef-bc9b-c7b262605968 -- ceph tell mds.1:3 damage ls 2024-05-27T07:36:28.485 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:36:28 smithi052 ceph-mon[31955]: pgmap v354: 129 pgs: 18 active+clean+snaptrim, 55 active+clean+snaptrim_wait, 56 active+clean; 632 MiB data, 7.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3.2 KiB/s wr, 2 op/s 2024-05-27T07:36:28.608 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:36:28 smithi173 ceph-mon[37000]: pgmap v354: 129 pgs: 18 active+clean+snaptrim, 55 active+clean+snaptrim_wait, 56 active+clean; 632 MiB data, 7.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3.2 KiB/s wr, 2 op/s 2024-05-27T07:36:28.695 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:36:28 smithi105 ceph-mon[35977]: pgmap v354: 129 pgs: 18 active+clean+snaptrim, 55 active+clean+snaptrim_wait, 56 active+clean; 632 MiB data, 7.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3.2 KiB/s wr, 2 op/s 2024-05-27T07:36:28.753 INFO:teuthology.orchestra.run.smithi052.stderr:Inferring config /var/lib/ceph/73df2636-1bf8-11ef-bc9b-c7b262605968/mon.a/config 2024-05-27T07:36:30.608 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:36:30 smithi173 ceph-mon[37000]: pgmap v355: 129 pgs: 18 active+clean+snaptrim, 55 active+clean+snaptrim_wait, 56 active+clean; 631 MiB data, 7.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 KiB/s rd, 3.2 KiB/s wr, 2 op/s 2024-05-27T07:36:30.627 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:36:30 smithi052 ceph-mon[31955]: pgmap v355: 129 pgs: 18 active+clean+snaptrim, 55 active+clean+snaptrim_wait, 56 active+clean; 631 MiB data, 7.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 KiB/s rd, 3.2 KiB/s wr, 2 op/s 2024-05-27T07:36:30.696 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:36:30 smithi105 ceph-mon[35977]: pgmap v355: 129 pgs: 18 active+clean+snaptrim, 55 active+clean+snaptrim_wait, 56 active+clean; 631 MiB data, 7.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 KiB/s rd, 3.2 KiB/s wr, 2 op/s 2024-05-27T07:36:30.831 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:30.834+0000 7f5ee2fbc700 1 -- 172.21.15.52:0/1003582382 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f5edc10b000 msgr2=0x7f5edc10b410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:36:30.831 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:30.834+0000 7f5ee2fbc700 1 --2- 172.21.15.52:0/1003582382 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f5edc10b000 0x7f5edc10b410 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7f5ed000b630 tx=0x7f5ed000b940 comp rx=0 tx=0).stop 2024-05-27T07:36:30.831 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:30.834+0000 7f5ee2fbc700 1 -- 172.21.15.52:0/1003582382 shutdown_connections 2024-05-27T07:36:30.831 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:30.834+0000 7f5ee2fbc700 1 --2- 172.21.15.52:0/1003582382 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f5edc110280 0x7f5edc117be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:36:30.831 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:30.834+0000 7f5ee2fbc700 1 --2- 172.21.15.52:0/1003582382 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f5edc10b000 0x7f5edc10b410 unknown :-1 s=CLOSED pgs=224 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:36:30.831 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:30.834+0000 7f5ee2fbc700 1 --2- 172.21.15.52:0/1003582382 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f5edc10f930 0x7f5edc10fd40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:36:30.831 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:30.834+0000 7f5ee2fbc700 1 -- 172.21.15.52:0/1003582382 >> 172.21.15.52:0/1003582382 conn(0x7f5edc06f4c0 msgr2=0x7f5edc06f900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:36:30.831 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:30.835+0000 7f5ee2fbc700 1 -- 172.21.15.52:0/1003582382 shutdown_connections 2024-05-27T07:36:30.832 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:30.835+0000 7f5ee2fbc700 1 -- 172.21.15.52:0/1003582382 wait complete. 2024-05-27T07:36:30.832 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:30.836+0000 7f5ee2fbc700 1 Processor -- start 2024-05-27T07:36:30.833 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:30.836+0000 7f5ee2fbc700 1 -- start start 2024-05-27T07:36:30.833 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:30.837+0000 7f5ee2fbc700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f5edc10b000 0x7f5edc1a2500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:36:30.833 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:30.837+0000 7f5ee2fbc700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f5edc10f930 0x7f5edc1a2aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:36:30.834 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:30.838+0000 7f5ee0d58700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f5edc10b000 0x7f5edc1a2500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:36:30.834 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:30.838+0000 7f5ee0d58700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f5edc10b000 0x7f5edc1a2500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.52:50076/0 (socket says 172.21.15.52:50076) 2024-05-27T07:36:30.835 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:30.838+0000 7f5ee0d58700 1 -- 172.21.15.52:0/3841226278 learned_addr learned my addr 172.21.15.52:0/3841226278 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-05-27T07:36:30.835 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:30.838+0000 7f5edbfff700 1 --2- 172.21.15.52:0/3841226278 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f5edc10f930 0x7f5edc1a2aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:36:30.835 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:30.838+0000 7f5ee2fbc700 1 --2- 172.21.15.52:0/3841226278 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f5edc110280 0x7f5edc1a6d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:36:30.835 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:30.839+0000 7f5ee2fbc700 1 -- 172.21.15.52:0/3841226278 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x7f5edc11aaa0 con 0x7f5edc10f930 2024-05-27T07:36:30.835 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:30.839+0000 7f5ee1559700 1 --2- 172.21.15.52:0/3841226278 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f5edc110280 0x7f5edc1a6d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:36:30.835 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:30.839+0000 7f5ee2fbc700 1 -- 172.21.15.52:0/3841226278 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f5edc11a920 con 0x7f5edc10b000 2024-05-27T07:36:30.835 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:30.839+0000 7f5ee2fbc700 1 -- 172.21.15.52:0/3841226278 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f5edc11ac20 con 0x7f5edc110280 2024-05-27T07:36:30.836 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:30.840+0000 7f5edbfff700 1 -- 172.21.15.52:0/3841226278 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f5edc110280 msgr2=0x7f5edc1a6d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:36:30.836 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:30.840+0000 7f5edbfff700 1 --2- 172.21.15.52:0/3841226278 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f5edc110280 0x7f5edc1a6d30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:36:30.836 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:30.840+0000 7f5edbfff700 1 -- 172.21.15.52:0/3841226278 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f5edc10b000 msgr2=0x7f5edc1a2500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:36:30.836 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:30.840+0000 7f5edbfff700 1 --2- 172.21.15.52:0/3841226278 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f5edc10b000 0x7f5edc1a2500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:36:30.837 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:30.840+0000 7f5edbfff700 1 -- 172.21.15.52:0/3841226278 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5edc1a73d0 con 0x7f5edc10f930 2024-05-27T07:36:30.837 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:30.841+0000 7f5edbfff700 1 --2- 172.21.15.52:0/3841226278 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f5edc10f930 0x7f5edc1a2aa0 secure :-1 s=READY pgs=758 cs=0 l=1 rev1=1 crypto rx=0x7f5ec800e9c0 tx=0x7f5ec800ecd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:36:30.840 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:30.844+0000 7f5ed9ffb700 1 -- 172.21.15.52:0/3841226278 <== mon.0 v2:172.21.15.52:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5ec800cb70 con 0x7f5edc10f930 2024-05-27T07:36:30.840 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:30.844+0000 7f5ed9ffb700 1 -- 172.21.15.52:0/3841226278 <== mon.0 v2:172.21.15.52:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f5ec800ee60 con 0x7f5edc10f930 2024-05-27T07:36:30.841 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:30.844+0000 7f5ed9ffb700 1 -- 172.21.15.52:0/3841226278 <== mon.0 v2:172.21.15.52:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5ec8003d00 con 0x7f5edc10f930 2024-05-27T07:36:30.841 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:30.844+0000 7f5ee2fbc700 1 -- 172.21.15.52:0/3841226278 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5edc1a76c0 con 0x7f5edc10f930 2024-05-27T07:36:30.841 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:30.845+0000 7f5ee2fbc700 1 -- 172.21.15.52:0/3841226278 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5edc1aefe0 con 0x7f5edc10f930 2024-05-27T07:36:30.848 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:30.852+0000 7f5ef19d1b80 1 Processor -- start 2024-05-27T07:36:30.848 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:30.853+0000 7f5ef19d1b80 1 -- start start 2024-05-27T07:36:30.849 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:30.853+0000 7f5ef19d1b80 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x558244525e40 0x5582446aab30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:36:30.850 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:30.853+0000 7f5ef19d1b80 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5582446ab070 0x5582446a4be0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:36:30.850 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:30.854+0000 7f5ee1559700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x558244525e40 0x5582446aab30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:36:30.850 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:30.854+0000 7f5ee1559700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x558244525e40 0x5582446aab30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.173:3300/0 says I am v2:172.21.15.52:35342/0 (socket says 172.21.15.52:35342) 2024-05-27T07:36:30.850 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:30.854+0000 7f5ee1559700 1 -- 172.21.15.52:0/835143962 learned_addr learned my addr 172.21.15.52:0/835143962 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-05-27T07:36:30.850 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:30.854+0000 7f5ef19d1b80 1 --2- 172.21.15.52:0/835143962 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x5582446a5120 0x5582446a5530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:36:30.850 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:30.854+0000 7f5ef19d1b80 1 -- 172.21.15.52:0/835143962 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x5582445d2d70 con 0x5582446a5120 2024-05-27T07:36:30.850 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:30.854+0000 7f5ef19d1b80 1 -- 172.21.15.52:0/835143962 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x5582445c2910 con 0x5582446ab070 2024-05-27T07:36:30.850 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:30.854+0000 7f5ef19d1b80 1 -- 172.21.15.52:0/835143962 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x5582445c6950 con 0x558244525e40 2024-05-27T07:36:30.851 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:30.854+0000 7f5ed9ffb700 1 -- 172.21.15.52:0/3841226278 <== mon.0 v2:172.21.15.52:3300/0 4 ==== mgrmap(e 26) ==== 149783+0+0 (secure 0 0 0) 0x7f5ec8003ee0 con 0x7f5edc10f930 2024-05-27T07:36:30.851 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:30.856+0000 7f5ed9ffb700 1 --2- 172.21.15.52:0/3841226278 >> [v2:172.21.15.52:6800/1822318384,v1:172.21.15.52:6801/1822318384] conn(0x7f5ecc0b2b40 0x7f5ecc0b5050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:36:30.851 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:30.856+0000 7f5ee0d58700 1 --2- 172.21.15.52:0/835143962 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5582446ab070 0x5582446a4be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:36:30.853 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:30.856+0000 7f5ee1559700 1 --2- 172.21.15.52:0/3841226278 >> [v2:172.21.15.52:6800/1822318384,v1:172.21.15.52:6801/1822318384] conn(0x7f5ecc0b2b40 0x7f5ecc0b5050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:36:30.853 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:30.857+0000 7f5ee0d58700 1 -- 172.21.15.52:0/835143962 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x558244525e40 msgr2=0x5582446aab30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:36:30.853 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:30.857+0000 7f5ee0d58700 1 --2- 172.21.15.52:0/835143962 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x558244525e40 0x5582446aab30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:36:30.853 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:30.857+0000 7f5ee0d58700 1 -- 172.21.15.52:0/835143962 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x5582446a5120 msgr2=0x5582446a5530 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-05-27T07:36:30.853 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:30.857+0000 7f5ee0d58700 1 --2- 172.21.15.52:0/835143962 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x5582446a5120 0x5582446a5530 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:36:30.853 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:30.857+0000 7f5ee0d58700 1 -- 172.21.15.52:0/835143962 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55824456bfb0 con 0x5582446ab070 2024-05-27T07:36:30.853 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:30.857+0000 7f5ee1559700 1 --2- 172.21.15.52:0/835143962 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x558244525e40 0x5582446aab30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-27T07:36:30.854 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:30.858+0000 7f5ed9ffb700 1 -- 172.21.15.52:0/3841226278 <== mon.0 v2:172.21.15.52:3300/0 5 ==== osd_map(127..127 src has 1..127) ==== 9555+0+0 (secure 0 0 0) 0x7f5ec80a5e90 con 0x7f5edc10f930 2024-05-27T07:36:30.854 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:30.858+0000 7f5ee1559700 1 --2- 172.21.15.52:0/3841226278 >> [v2:172.21.15.52:6800/1822318384,v1:172.21.15.52:6801/1822318384] conn(0x7f5ecc0b2b40 0x7f5ecc0b5050 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7f5ed4003fd0 tx=0x7f5ed4004b40 comp rx=0 tx=0).ready entity=mgr.15333 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:36:30.854 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:30.858+0000 7f5ee0d58700 1 --2- 172.21.15.52:0/835143962 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5582446ab070 0x5582446a4be0 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 crypto rx=0x7f5ed000b600 tx=0x7f5ed0005fb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d6410ff139fb619 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:36:30.858 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:30.861+0000 7f5ec6ffd700 1 -- 172.21.15.52:0/835143962 <== mon.2 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5ed0005070 con 0x5582446ab070 2024-05-27T07:36:30.858 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:30.861+0000 7f5ec6ffd700 1 -- 172.21.15.52:0/835143962 <== mon.2 v2:172.21.15.105:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f5ed0009740 con 0x5582446ab070 2024-05-27T07:36:30.858 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:30.861+0000 7f5ec6ffd700 1 -- 172.21.15.52:0/835143962 <== mon.2 v2:172.21.15.105:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5ed00038c0 con 0x5582446ab070 2024-05-27T07:36:30.858 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:30.862+0000 7f5ef19d1b80 1 -- 172.21.15.52:0/835143962 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5582446ab070 msgr2=0x5582446a4be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:36:30.858 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:30.862+0000 7f5ef19d1b80 1 --2- 172.21.15.52:0/835143962 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5582446ab070 0x5582446a4be0 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 crypto rx=0x7f5ed000b600 tx=0x7f5ed0005fb0 comp rx=0 tx=0).stop 2024-05-27T07:36:30.862 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:30.865+0000 7f5ef19d1b80 1 -- 172.21.15.52:0/835143962 shutdown_connections 2024-05-27T07:36:30.862 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:30.865+0000 7f5ef19d1b80 1 --2- 172.21.15.52:0/835143962 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x5582446a5120 0x5582446a5530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:36:30.862 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:30.865+0000 7f5ef19d1b80 1 --2- 172.21.15.52:0/835143962 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x558244525e40 0x5582446aab30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:36:30.862 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:30.865+0000 7f5ef19d1b80 1 --2- 172.21.15.52:0/835143962 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5582446ab070 0x5582446a4be0 unknown :-1 s=CLOSED pgs=269 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:36:30.862 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:30.865+0000 7f5ef19d1b80 1 -- 172.21.15.52:0/835143962 >> 172.21.15.52:0/835143962 conn(0x558244528770 msgr2=0x55824469fd90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:36:30.862 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:30.866+0000 7f5ef19d1b80 1 -- 172.21.15.52:0/835143962 shutdown_connections 2024-05-27T07:36:30.862 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:30.866+0000 7f5ef19d1b80 1 -- 172.21.15.52:0/835143962 wait complete. 2024-05-27T07:36:30.863 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:30.867+0000 7f5ef19d1b80 1 -- start start 2024-05-27T07:36:30.863 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:30.867+0000 7f5ef19d1b80 1 Processor -- start 2024-05-27T07:36:30.863 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:30.868+0000 7f5ef19d1b80 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x558244525e40 0x5582446ae8b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:36:30.865 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:30.868+0000 7f5ef19d1b80 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5582446a5120 0x5582446aedf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:36:30.865 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:30.868+0000 7f5ef19d1b80 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5582446ab070 0x5582446af330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:36:30.865 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:30.868+0000 7f5edbfff700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5582446a5120 0x5582446aedf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:36:30.865 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:30.868+0000 7f5edbfff700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5582446a5120 0x5582446aedf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.173:3300/0 says I am v2:172.21.15.52:35358/0 (socket says 172.21.15.52:35358) 2024-05-27T07:36:30.865 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:30.868+0000 7f5ee1559700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5582446ab070 0x5582446af330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:36:30.865 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:30.869+0000 7f5ee1559700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5582446ab070 0x5582446af330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.52:50090/0 (socket says 172.21.15.52:50090) 2024-05-27T07:36:30.865 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:30.869+0000 7f5ee1559700 1 -- 172.21.15.52:0/4096909346 learned_addr learned my addr 172.21.15.52:0/4096909346 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-05-27T07:36:30.866 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:30.869+0000 7f5ef19d1b80 1 -- 172.21.15.52:0/4096909346 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x5582444c8fa0 con 0x558244525e40 2024-05-27T07:36:30.866 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:30.869+0000 7f5ee0d58700 1 --2- 172.21.15.52:0/4096909346 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x558244525e40 0x5582446ae8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:36:30.866 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:30.869+0000 7f5ef19d1b80 1 -- 172.21.15.52:0/4096909346 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x55824452ada0 con 0x5582446ab070 2024-05-27T07:36:30.866 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:30.869+0000 7f5ef19d1b80 1 -- 172.21.15.52:0/4096909346 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x5582446b1b00 con 0x5582446a5120 2024-05-27T07:36:30.866 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:30.869+0000 7f5ef19d1b80 10 client.0 _ll_register_callbacks cb 0x5582445595b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T07:36:30.867 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:30.870+0000 7f5ee1559700 1 -- 172.21.15.52:0/4096909346 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5582446a5120 msgr2=0x5582446aedf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:36:30.867 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:30.870+0000 7f5ee1559700 1 --2- 172.21.15.52:0/4096909346 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5582446a5120 0x5582446aedf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:36:30.867 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:30.870+0000 7f5ee1559700 1 -- 172.21.15.52:0/4096909346 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x558244525e40 msgr2=0x5582446ae8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:36:30.867 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:30.870+0000 7f5ee1559700 1 --2- 172.21.15.52:0/4096909346 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x558244525e40 0x5582446ae8b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:36:30.867 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:30.870+0000 7f5ee1559700 1 -- 172.21.15.52:0/4096909346 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5582445128d0 con 0x5582446ab070 2024-05-27T07:36:30.868 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:30.870+0000 7f5ee1559700 1 --2- 172.21.15.52:0/4096909346 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5582446ab070 0x5582446af330 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 crypto rx=0x7f5ed4004090 tx=0x7f5ed4004ad0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=1ea137a55a142769 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:36:30.868 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:30.870+0000 7f5ebbfff700 10 client.0 ms_handle_connect on v2:172.21.15.105:3300/0 2024-05-27T07:36:30.869 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:30.871+0000 7f5ebbfff700 1 -- 172.21.15.52:0/4096909346 <== mon.2 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5ed401d0d0 con 0x5582446ab070 2024-05-27T07:36:30.869 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:30.872+0000 7f5ef19d1b80 1 -- 172.21.15.52:0/4096909346 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f5ed0009740 con 0x5582446ab070 2024-05-27T07:36:30.869 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:30.872+0000 7f5ebbfff700 1 -- 172.21.15.52:0/4096909346 <== mon.2 v2:172.21.15.105:3300/0 2 ==== osd_map(127..127 src has 1..127) ==== 9555+0+0 (secure 0 0 0) 0x7f5ed4012040 con 0x5582446ab070 2024-05-27T07:36:30.869 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:30.872+0000 7f5ebbfff700 1 -- 172.21.15.52:0/4096909346 <== mon.2 v2:172.21.15.105:3300/0 3 ==== mon_get_version_reply(handle=1 version=108) ==== 24+0+0 (secure 0 0 0) 0x7f5ed4031050 con 0x5582446ab070 2024-05-27T07:36:30.869 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:30.872+0000 7f5ef19d1b80 10 client.26930 fetch_fsmap learned FSMap version 108 2024-05-27T07:36:30.869 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:30.872+0000 7f5ef19d1b80 1 -- 172.21.15.52:0/4096909346 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,fsmap=108,monmap=4+}) -- 0x55824457c760 con 0x5582446ab070 2024-05-27T07:36:30.869 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:30.873+0000 7f5ebbfff700 1 -- 172.21.15.52:0/4096909346 <== mon.2 v2:172.21.15.105:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f5ed0009740 con 0x5582446ab070 2024-05-27T07:36:30.871 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:30.874+0000 7f5ebbfff700 1 -- 172.21.15.52:0/4096909346 <== mon.2 v2:172.21.15.105:3300/0 5 ==== fsmap(e 108) ==== 4203+0+0 (secure 0 0 0) 0x7f5ed4018040 con 0x5582446ab070 2024-05-27T07:36:30.871 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:30.874+0000 7f5ef19d1b80 10 client.26930 fetch_fsmap finished waiting for FSMap version 108 2024-05-27T07:36:30.871 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:30.874+0000 7f5ef19d1b80 10 client.26930 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.e 2024-05-27T07:36:30.871 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:30.874+0000 7f5ef19d1b80 20 client.26930 populate_metadata read hostname 'smithi052' 2024-05-27T07:36:30.871 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:30.874+0000 7f5ef19d1b80 1 --2- 172.21.15.52:0/4096909346 >> [v2:172.21.15.105:6834/3298498384,v1:172.21.15.105:6835/3298498384] conn(0x5582446bee50 0x5582446c12e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:36:30.871 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:30.874+0000 7f5ef19d1b80 4 client.26930 mds_command: new command op to 24473 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-27T07:36:30.871 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:30.874+0000 7f5ef19d1b80 1 -- 172.21.15.52:0/4096909346 --> [v2:172.21.15.105:6834/3298498384,v1:172.21.15.105:6835/3298498384] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5582445d4cc0 con 0x5582446bee50 2024-05-27T07:36:30.871 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:30.875+0000 7f5ee0d58700 1 --2- 172.21.15.52:0/4096909346 >> [v2:172.21.15.105:6834/3298498384,v1:172.21.15.105:6835/3298498384] conn(0x5582446bee50 0x5582446c12e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:36:30.887 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:30.891+0000 7f5ee0d58700 1 --2- 172.21.15.52:0/4096909346 >> [v2:172.21.15.105:6834/3298498384,v1:172.21.15.105:6835/3298498384] conn(0x5582446bee50 0x5582446c12e0 crc :-1 s=READY pgs=47 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=a0868a79bfc2e578 server_cookie=38c6371a577f3ab2 in_seq=0 out_seq=0 2024-05-27T07:36:30.887 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:30.891+0000 7f5ebbfff700 10 client.26930 ms_handle_connect on v2:172.21.15.105:6834/3298498384 2024-05-27T07:36:30.888 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:30.892+0000 7f5ebbfff700 1 -- 172.21.15.52:0/4096909346 <== mds.3 v2:172.21.15.105:6834/3298498384 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24041 (crc 0 0 0) 0x5582445d4cc0 con 0x5582446bee50 2024-05-27T07:36:30.888 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:30.892+0000 7f5ebbfff700 10 client.26930 handle_command_reply: tid=0 2024-05-27T07:36:30.891 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:30.895+0000 7f5ef19d1b80 1 client.26930 shutdown 2024-05-27T07:36:30.895 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:30.899+0000 7f5ef19d1b80 1 -- 172.21.15.52:0/4096909346 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5582446ab070 msgr2=0x5582446af330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:36:30.895 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:30.899+0000 7f5ef19d1b80 1 --2- 172.21.15.52:0/4096909346 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5582446ab070 0x5582446af330 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 crypto rx=0x7f5ed4004090 tx=0x7f5ed4004ad0 comp rx=0 tx=0).stop 2024-05-27T07:36:30.896 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:30.901+0000 7f5ef19d1b80 1 -- 172.21.15.52:0/4096909346 shutdown_connections 2024-05-27T07:36:30.896 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:30.901+0000 7f5ef19d1b80 1 --2- 172.21.15.52:0/4096909346 >> [v2:172.21.15.105:6834/3298498384,v1:172.21.15.105:6835/3298498384] conn(0x5582446bee50 0x5582446c12e0 crc :-1 s=READY pgs=47 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:36:30.897 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:30.901+0000 7f5ef19d1b80 1 --2- 172.21.15.52:0/4096909346 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5582446ab070 0x5582446af330 unknown :-1 s=CLOSED pgs=270 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:36:30.897 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:30.901+0000 7f5ef19d1b80 1 --2- 172.21.15.52:0/4096909346 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x558244525e40 0x5582446ae8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:36:30.897 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:30.901+0000 7f5ef19d1b80 1 --2- 172.21.15.52:0/4096909346 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5582446a5120 0x5582446aedf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:36:30.897 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:30.901+0000 7f5ebbfff700 0 client.26930 ms_handle_reset on v2:172.21.15.105:6834/3298498384 2024-05-27T07:36:30.897 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:30.901+0000 7f5ef19d1b80 1 -- 172.21.15.52:0/4096909346 >> 172.21.15.52:0/4096909346 conn(0x558244528770 msgr2=0x5582446a3bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:36:30.898 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:30.903+0000 7f5ef19d1b80 1 -- 172.21.15.52:0/4096909346 shutdown_connections 2024-05-27T07:36:30.899 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:30.903+0000 7f5ef19d1b80 1 -- 172.21.15.52:0/4096909346 wait complete. 2024-05-27T07:36:30.901 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:30.905+0000 7f5ef19d1b80 20 client.26930 trim_cache size 0 max 16384 2024-05-27T07:36:30.932 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:30.936+0000 7f5ef19d1b80 1 Processor -- start 2024-05-27T07:36:30.934 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:30.939+0000 7f5ef19d1b80 1 -- start start 2024-05-27T07:36:30.936 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:30.939+0000 7f5ef19d1b80 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x558244525e40 0x5582446a4be0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:36:30.936 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:30.939+0000 7f5ef19d1b80 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5582446a5120 0x5582446a9fd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:36:30.936 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:30.940+0000 7f5ef19d1b80 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5582446ab070 0x5582446c6ae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:36:30.936 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:30.940+0000 7f5ef19d1b80 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x7f5edc1bb700 con 0x558244525e40 2024-05-27T07:36:30.936 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:30.940+0000 7f5ee0d58700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x558244525e40 0x5582446a4be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:36:30.936 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:30.940+0000 7f5ee0d58700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x558244525e40 0x5582446a4be0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.52:39534/0 (socket says 172.21.15.52:39534) 2024-05-27T07:36:30.936 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:30.940+0000 7f5ee0d58700 1 -- 172.21.15.52:0/2836630485 learned_addr learned my addr 172.21.15.52:0/2836630485 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-05-27T07:36:30.936 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:30.940+0000 7f5edbfff700 1 --2- 172.21.15.52:0/2836630485 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5582446a5120 0x5582446a9fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:36:30.936 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:30.940+0000 7f5ef19d1b80 1 -- 172.21.15.52:0/2836630485 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f5edc1b9b30 con 0x5582446a5120 2024-05-27T07:36:30.937 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:30.940+0000 7f5ef19d1b80 1 -- 172.21.15.52:0/2836630485 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f5edc1bb000 con 0x5582446ab070 2024-05-27T07:36:30.938 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:30.942+0000 7f5edbfff700 1 -- 172.21.15.52:0/2836630485 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5582446ab070 msgr2=0x5582446c6ae0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-05-27T07:36:30.938 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:30.942+0000 7f5edbfff700 1 --2- 172.21.15.52:0/2836630485 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5582446ab070 0x5582446c6ae0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:36:30.938 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:30.942+0000 7f5edbfff700 1 -- 172.21.15.52:0/2836630485 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x558244525e40 msgr2=0x5582446a4be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:36:30.938 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:30.942+0000 7f5edbfff700 1 --2- 172.21.15.52:0/2836630485 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x558244525e40 0x5582446a4be0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:36:30.938 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:30.942+0000 7f5edbfff700 1 -- 172.21.15.52:0/2836630485 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5582446b1150 con 0x5582446a5120 2024-05-27T07:36:30.939 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:30.943+0000 7f5edbfff700 1 --2- 172.21.15.52:0/2836630485 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5582446a5120 0x5582446a9fd0 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 crypto rx=0x7f5ec80dfd10 tx=0x7f5ec80aed70 comp rx=0 tx=0).ready entity=mon.2 client_cookie=3a6ab22d7c064de server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:36:30.942 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:30.947+0000 7f5ec67fc700 1 -- 172.21.15.52:0/2836630485 <== mon.2 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5ec80a6090 con 0x5582446a5120 2024-05-27T07:36:30.942 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:30.947+0000 7f5ec67fc700 1 -- 172.21.15.52:0/2836630485 <== mon.2 v2:172.21.15.105:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f5ec8021e50 con 0x5582446a5120 2024-05-27T07:36:30.943 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:30.947+0000 7f5ec67fc700 1 -- 172.21.15.52:0/2836630485 <== mon.2 v2:172.21.15.105:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5ec80a6230 con 0x5582446a5120 2024-05-27T07:36:30.944 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:30.948+0000 7f5ef19d1b80 1 -- 172.21.15.52:0/2836630485 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5582446a5120 msgr2=0x5582446a9fd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:36:30.945 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:30.948+0000 7f5ef19d1b80 1 --2- 172.21.15.52:0/2836630485 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5582446a5120 0x5582446a9fd0 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 crypto rx=0x7f5ec80dfd10 tx=0x7f5ec80aed70 comp rx=0 tx=0).stop 2024-05-27T07:36:30.945 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:30.948+0000 7f5ef19d1b80 1 -- 172.21.15.52:0/2836630485 shutdown_connections 2024-05-27T07:36:30.945 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:30.948+0000 7f5ef19d1b80 1 --2- 172.21.15.52:0/2836630485 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5582446ab070 0x5582446c6ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:36:30.945 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:30.948+0000 7f5ef19d1b80 1 --2- 172.21.15.52:0/2836630485 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x558244525e40 0x5582446a4be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:36:30.945 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:30.948+0000 7f5ef19d1b80 1 --2- 172.21.15.52:0/2836630485 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5582446a5120 0x5582446a9fd0 unknown :-1 s=CLOSED pgs=271 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:36:30.945 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:30.948+0000 7f5ef19d1b80 1 -- 172.21.15.52:0/2836630485 >> 172.21.15.52:0/2836630485 conn(0x558244528770 msgr2=0x5582446a98c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:36:30.945 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:30.949+0000 7f5ef19d1b80 1 -- 172.21.15.52:0/2836630485 shutdown_connections 2024-05-27T07:36:30.946 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:30.949+0000 7f5ef19d1b80 1 -- 172.21.15.52:0/2836630485 wait complete. 2024-05-27T07:36:30.946 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:30.949+0000 7f5ef19d1b80 1 -- start start 2024-05-27T07:36:30.946 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:30.950+0000 7f5ef19d1b80 1 Processor -- start 2024-05-27T07:36:30.946 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:30.950+0000 7f5ef19d1b80 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x558244525e40 0x5582446b6d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:36:30.947 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:30.951+0000 7f5ef19d1b80 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x5582446a5120 0x5582446b72d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:36:30.948 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:30.951+0000 7f5ee0d58700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x558244525e40 0x5582446b6d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:36:30.948 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:30.951+0000 7f5ee0d58700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x558244525e40 0x5582446b6d90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.173:3300/0 says I am v2:172.21.15.52:35374/0 (socket says 172.21.15.52:35374) 2024-05-27T07:36:30.948 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:30.951+0000 7f5edbfff700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x5582446a5120 0x5582446b72d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:36:30.948 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:30.951+0000 7f5edbfff700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x5582446a5120 0x5582446b72d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.52:39542/0 (socket says 172.21.15.52:39542) 2024-05-27T07:36:30.948 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:30.951+0000 7f5ee0d58700 1 -- 172.21.15.52:0/2646985209 learned_addr learned my addr 172.21.15.52:0/2646985209 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-05-27T07:36:30.948 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:30.951+0000 7f5ef19d1b80 1 --2- 172.21.15.52:0/2646985209 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5582446ab070 0x5582446a3bd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:36:30.948 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:30.952+0000 7f5ef19d1b80 1 -- 172.21.15.52:0/2646985209 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x7f5edc1ba930 con 0x5582446a5120 2024-05-27T07:36:30.949 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:30.952+0000 7f5ef19d1b80 1 -- 172.21.15.52:0/2646985209 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f5edc1ba230 con 0x5582446ab070 2024-05-27T07:36:30.949 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:30.952+0000 7f5ef19d1b80 1 -- 172.21.15.52:0/2646985209 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x5582446c18c0 con 0x558244525e40 2024-05-27T07:36:30.950 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:30.953+0000 7f5ef19d1b80 10 client.0 _ll_register_callbacks cb 0x5582445595b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T07:36:30.950 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:30.953+0000 7f5ee0d58700 1 -- 172.21.15.52:0/2646985209 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5582446ab070 msgr2=0x5582446a3bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:36:30.950 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:30.953+0000 7f5ee0d58700 1 --2- 172.21.15.52:0/2646985209 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5582446ab070 0x5582446a3bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:36:30.950 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:30.953+0000 7f5ee0d58700 1 -- 172.21.15.52:0/2646985209 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x5582446a5120 msgr2=0x5582446b72d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:36:30.950 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:30.953+0000 7f5ee0d58700 1 --2- 172.21.15.52:0/2646985209 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x5582446a5120 0x5582446b72d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:36:30.950 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:30.953+0000 7f5ee0d58700 1 -- 172.21.15.52:0/2646985209 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5582446b7910 con 0x558244525e40 2024-05-27T07:36:30.950 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:30.954+0000 7f5ee0d58700 1 --2- 172.21.15.52:0/2646985209 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x558244525e40 0x5582446b6d90 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7f5ed000f4a0 tx=0x7f5ed0009ba0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=1bf59605c0c2ce11 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:36:30.951 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:30.954+0000 7f5ebbfff700 10 client.0 ms_handle_connect on v2:172.21.15.173:3300/0 2024-05-27T07:36:30.953 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:30.958+0000 7f5ebbfff700 1 -- 172.21.15.52:0/2646985209 <== mon.1 v2:172.21.15.173:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5ed00096a0 con 0x558244525e40 2024-05-27T07:36:30.954 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:30.958+0000 7f5ef19d1b80 1 -- 172.21.15.52:0/2646985209 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f5ec8021e50 con 0x558244525e40 2024-05-27T07:36:30.954 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:30.958+0000 7f5ebbfff700 1 -- 172.21.15.52:0/2646985209 <== mon.1 v2:172.21.15.173:3300/0 2 ==== osd_map(127..127 src has 1..127) ==== 9555+0+0 (secure 0 0 0) 0x7f5ed00050d0 con 0x558244525e40 2024-05-27T07:36:30.955 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:30.960+0000 7f5ebbfff700 1 -- 172.21.15.52:0/2646985209 <== mon.1 v2:172.21.15.173:3300/0 3 ==== mon_get_version_reply(handle=1 version=108) ==== 24+0+0 (secure 0 0 0) 0x7f5ed003c050 con 0x558244525e40 2024-05-27T07:36:30.956 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:30.960+0000 7f5ef19d1b80 10 client.26752 fetch_fsmap learned FSMap version 108 2024-05-27T07:36:30.956 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:30.960+0000 7f5ef19d1b80 1 -- 172.21.15.52:0/2646985209 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({config=0+,fsmap=108,monmap=4+}) -- 0x5582447316a0 con 0x558244525e40 2024-05-27T07:36:30.956 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:30.961+0000 7f5ebbfff700 1 -- 172.21.15.52:0/2646985209 <== mon.1 v2:172.21.15.173:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f5ed00052d0 con 0x558244525e40 2024-05-27T07:36:30.957 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:30.961+0000 7f5ebbfff700 1 -- 172.21.15.52:0/2646985209 <== mon.1 v2:172.21.15.173:3300/0 5 ==== fsmap(e 108) ==== 4203+0+0 (secure 0 0 0) 0x7f5ed000f8a0 con 0x558244525e40 2024-05-27T07:36:30.957 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:30.961+0000 7f5ef19d1b80 10 client.26752 fetch_fsmap finished waiting for FSMap version 108 2024-05-27T07:36:30.957 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:30.961+0000 7f5ef19d1b80 10 client.26752 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.e 2024-05-27T07:36:30.957 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:30.961+0000 7f5ef19d1b80 20 client.26752 populate_metadata read hostname 'smithi052' 2024-05-27T07:36:30.957 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:30.961+0000 7f5ef19d1b80 1 --2- 172.21.15.52:0/2646985209 >> [v2:172.21.15.105:6834/3298498384,v1:172.21.15.105:6835/3298498384] conn(0x558244731b80 0x558244733fa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:36:30.957 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:30.961+0000 7f5ef19d1b80 4 client.26752 mds_command: new command op to 24473 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-05-27T07:36:30.958 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:30.961+0000 7f5ef19d1b80 1 -- 172.21.15.52:0/2646985209 --> [v2:172.21.15.105:6834/3298498384,v1:172.21.15.105:6835/3298498384] -- command(tid 0: {"prefix": "damage ls"}) -- 0x7f5ec8021e50 con 0x558244731b80 2024-05-27T07:36:30.958 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:30.962+0000 7f5edbfff700 1 --2- 172.21.15.52:0/2646985209 >> [v2:172.21.15.105:6834/3298498384,v1:172.21.15.105:6835/3298498384] conn(0x558244731b80 0x558244733fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:36:30.958 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:30.962+0000 7f5ebbfff700 10 client.26752 ms_handle_connect on v2:172.21.15.105:6834/3298498384 2024-05-27T07:36:30.958 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:30.963+0000 7f5edbfff700 1 --2- 172.21.15.52:0/2646985209 >> [v2:172.21.15.105:6834/3298498384,v1:172.21.15.105:6835/3298498384] conn(0x558244731b80 0x558244733fa0 crc :-1 s=READY pgs=48 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=cc09d235816b85bf server_cookie=ec99d1aa9ac49f9d in_seq=0 out_seq=0 2024-05-27T07:36:30.959 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:30.964+0000 7f5ebbfff700 1 -- 172.21.15.52:0/2646985209 <== mds.3 v2:172.21.15.105:6834/3298498384 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x7f5ec8021e50 con 0x558244731b80 2024-05-27T07:36:30.959 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:30.964+0000 7f5ebbfff700 10 client.26752 handle_command_reply: tid=0 2024-05-27T07:36:30.960 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:30.964+0000 7f5ef19d1b80 1 client.26752 shutdown 2024-05-27T07:36:30.964 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:30.968+0000 7f5ef19d1b80 1 -- 172.21.15.52:0/2646985209 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x558244525e40 msgr2=0x5582446b6d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:36:30.964 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:30.968+0000 7f5ef19d1b80 1 --2- 172.21.15.52:0/2646985209 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x558244525e40 0x5582446b6d90 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7f5ed000f4a0 tx=0x7f5ed0009ba0 comp rx=0 tx=0).stop 2024-05-27T07:36:30.964 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:30.968+0000 7f5ef19d1b80 1 -- 172.21.15.52:0/2646985209 shutdown_connections 2024-05-27T07:36:30.964 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:30.968+0000 7f5ef19d1b80 1 --2- 172.21.15.52:0/2646985209 >> [v2:172.21.15.105:6834/3298498384,v1:172.21.15.105:6835/3298498384] conn(0x558244731b80 0x558244733fa0 crc :-1 s=READY pgs=48 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:36:30.964 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:30.968+0000 7f5ef19d1b80 1 --2- 172.21.15.52:0/2646985209 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5582446ab070 0x5582446a3bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:36:30.964 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:30.968+0000 7f5ef19d1b80 1 --2- 172.21.15.52:0/2646985209 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x558244525e40 0x5582446b6d90 unknown :-1 s=CLOSED pgs=225 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:36:30.965 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:30.968+0000 7f5ef19d1b80 1 --2- 172.21.15.52:0/2646985209 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x5582446a5120 0x5582446b72d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:36:30.965 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:30.968+0000 7f5ef19d1b80 1 -- 172.21.15.52:0/2646985209 >> 172.21.15.52:0/2646985209 conn(0x558244528770 msgr2=0x5582446a8490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:36:30.965 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:30.968+0000 7f5ebbfff700 0 client.26752 ms_handle_reset on v2:172.21.15.105:6834/3298498384 2024-05-27T07:36:30.968 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:30.971+0000 7f5ef19d1b80 1 -- 172.21.15.52:0/2646985209 shutdown_connections 2024-05-27T07:36:30.968 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:30.971+0000 7f5ef19d1b80 1 -- 172.21.15.52:0/2646985209 wait complete. 2024-05-27T07:36:30.968 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:30.971+0000 7f5ef19d1b80 20 client.26752 trim_cache size 0 max 16384 2024-05-27T07:36:30.968 INFO:teuthology.orchestra.run.smithi052.stdout:[] 2024-05-27T07:36:30.970 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:30.974+0000 7f5ee2fbc700 1 -- 172.21.15.52:0/3841226278 >> [v2:172.21.15.52:6800/1822318384,v1:172.21.15.52:6801/1822318384] conn(0x7f5ecc0b2b40 msgr2=0x7f5ecc0b5050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:36:30.971 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:30.974+0000 7f5ee2fbc700 1 --2- 172.21.15.52:0/3841226278 >> [v2:172.21.15.52:6800/1822318384,v1:172.21.15.52:6801/1822318384] conn(0x7f5ecc0b2b40 0x7f5ecc0b5050 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7f5ed4003fd0 tx=0x7f5ed4004b40 comp rx=0 tx=0).stop 2024-05-27T07:36:30.971 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:30.974+0000 7f5ee2fbc700 1 -- 172.21.15.52:0/3841226278 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f5edc10f930 msgr2=0x7f5edc1a2aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:36:30.971 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:30.974+0000 7f5ee2fbc700 1 --2- 172.21.15.52:0/3841226278 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f5edc10f930 0x7f5edc1a2aa0 secure :-1 s=READY pgs=758 cs=0 l=1 rev1=1 crypto rx=0x7f5ec800e9c0 tx=0x7f5ec800ecd0 comp rx=0 tx=0).stop 2024-05-27T07:36:30.972 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:30.974+0000 7f5ee2fbc700 1 -- 172.21.15.52:0/3841226278 shutdown_connections 2024-05-27T07:36:30.972 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:30.974+0000 7f5ee2fbc700 1 --2- 172.21.15.52:0/3841226278 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f5edc110280 0x7f5edc1a6d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:36:30.972 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:30.974+0000 7f5ee2fbc700 1 --2- 172.21.15.52:0/3841226278 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f5edc10b000 0x7f5edc1a2500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:36:30.972 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:30.974+0000 7f5ee2fbc700 1 --2- 172.21.15.52:0/3841226278 >> [v2:172.21.15.52:6800/1822318384,v1:172.21.15.52:6801/1822318384] conn(0x7f5ecc0b2b40 0x7f5ecc0b5050 unknown :-1 s=CLOSED pgs=196 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:36:30.972 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:30.974+0000 7f5ee2fbc700 1 --2- 172.21.15.52:0/3841226278 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f5edc10f930 0x7f5edc1a2aa0 unknown :-1 s=CLOSED pgs=758 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:36:30.972 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:30.974+0000 7f5ee2fbc700 1 -- 172.21.15.52:0/3841226278 >> 172.21.15.52:0/3841226278 conn(0x7f5edc06f4c0 msgr2=0x7f5edc10e6a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:36:30.972 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:30.975+0000 7f5ee2fbc700 1 -- 172.21.15.52:0/3841226278 shutdown_connections 2024-05-27T07:36:30.972 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:30.975+0000 7f5ee2fbc700 1 -- 172.21.15.52:0/3841226278 wait complete. 2024-05-27T07:36:31.942 DEBUG:teuthology.orchestra.run.smithi052:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:38f9d900c83c63799fbdbe61acc9a11b0d3554a6 shell --fsid 73df2636-1bf8-11ef-bc9b-c7b262605968 -- ceph tell mds.1:2 damage ls 2024-05-27T07:36:32.427 INFO:teuthology.orchestra.run.smithi052.stderr:Inferring config /var/lib/ceph/73df2636-1bf8-11ef-bc9b-c7b262605968/mon.a/config 2024-05-27T07:36:32.608 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:36:32 smithi173 ceph-mon[37000]: pgmap v356: 129 pgs: 18 active+clean+snaptrim, 55 active+clean+snaptrim_wait, 56 active+clean; 631 MiB data, 7.0 GiB used, 1.0 TiB / 1.0 TiB avail 2024-05-27T07:36:32.696 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:36:32 smithi105 ceph-mon[35977]: pgmap v356: 129 pgs: 18 active+clean+snaptrim, 55 active+clean+snaptrim_wait, 56 active+clean; 631 MiB data, 7.0 GiB used, 1.0 TiB / 1.0 TiB avail 2024-05-27T07:36:32.738 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:36:32 smithi052 ceph-mon[31955]: pgmap v356: 129 pgs: 18 active+clean+snaptrim, 55 active+clean+snaptrim_wait, 56 active+clean; 631 MiB data, 7.0 GiB used, 1.0 TiB / 1.0 TiB avail 2024-05-27T07:36:34.608 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:36:34 smithi173 ceph-mon[37000]: pgmap v357: 129 pgs: 18 active+clean+snaptrim, 55 active+clean+snaptrim_wait, 56 active+clean; 630 MiB data, 7.1 GiB used, 1.0 TiB / 1.0 TiB avail 2024-05-27T07:36:34.696 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:36:34 smithi105 ceph-mon[35977]: pgmap v357: 129 pgs: 18 active+clean+snaptrim, 55 active+clean+snaptrim_wait, 56 active+clean; 630 MiB data, 7.1 GiB used, 1.0 TiB / 1.0 TiB avail 2024-05-27T07:36:34.735 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:36:34 smithi052 ceph-mon[31955]: pgmap v357: 129 pgs: 18 active+clean+snaptrim, 55 active+clean+snaptrim_wait, 56 active+clean; 630 MiB data, 7.1 GiB used, 1.0 TiB / 1.0 TiB avail 2024-05-27T07:36:35.313 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:35.317+0000 7f7d74301700 1 -- 172.21.15.52:0/94164606 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f7d6c109860 msgr2=0x7f7d6c1111c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:36:35.313 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:35.317+0000 7f7d74301700 1 --2- 172.21.15.52:0/94164606 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f7d6c109860 0x7f7d6c1111c0 secure :-1 s=READY pgs=759 cs=0 l=1 rev1=1 crypto rx=0x7f7d5c01a9d0 tx=0x7f7d5c01ace0 comp rx=0 tx=0).stop 2024-05-27T07:36:35.317 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:35.319+0000 7f7d74301700 1 -- 172.21.15.52:0/94164606 shutdown_connections 2024-05-27T07:36:35.318 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:35.319+0000 7f7d74301700 1 --2- 172.21.15.52:0/94164606 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f7d6c109860 0x7f7d6c1111c0 unknown :-1 s=CLOSED pgs=759 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:36:35.318 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:35.319+0000 7f7d74301700 1 --2- 172.21.15.52:0/94164606 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f7d6c1045e0 0x7f7d6c1049f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:36:35.318 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:35.319+0000 7f7d74301700 1 --2- 172.21.15.52:0/94164606 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f7d6c108f10 0x7f7d6c109320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:36:35.318 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:35.319+0000 7f7d74301700 1 -- 172.21.15.52:0/94164606 >> 172.21.15.52:0/94164606 conn(0x7f7d6c100030 msgr2=0x7f7d6c102450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:36:35.318 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:35.320+0000 7f7d74301700 1 -- 172.21.15.52:0/94164606 shutdown_connections 2024-05-27T07:36:35.318 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:35.320+0000 7f7d74301700 1 -- 172.21.15.52:0/94164606 wait complete. 2024-05-27T07:36:35.318 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:35.321+0000 7f7d74301700 1 Processor -- start 2024-05-27T07:36:35.318 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:35.321+0000 7f7d74301700 1 -- start start 2024-05-27T07:36:35.319 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:35.322+0000 7f7d74301700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f7d6c1045e0 0x7f7d6c110f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:36:35.319 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:35.322+0000 7f7d74301700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f7d6c108f10 0x7f7d6c10b170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:36:35.320 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:35.322+0000 7f7d74301700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f7d6c109860 0x7f7d6c10b6b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:36:35.320 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:35.322+0000 7f7d74301700 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x7f7d6c06ebc0 con 0x7f7d6c108f10 2024-05-27T07:36:35.320 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:35.322+0000 7f7d74301700 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f7d6c06ea40 con 0x7f7d6c109860 2024-05-27T07:36:35.320 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:35.322+0000 7f7d74301700 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f7d6c06ed40 con 0x7f7d6c1045e0 2024-05-27T07:36:35.320 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:35.323+0000 7f7d7209d700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f7d6c1045e0 0x7f7d6c110f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:36:35.320 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:35.323+0000 7f7d7209d700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f7d6c1045e0 0x7f7d6c110f70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.173:3300/0 says I am v2:172.21.15.52:35384/0 (socket says 172.21.15.52:35384) 2024-05-27T07:36:35.320 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:35.323+0000 7f7d7209d700 1 -- 172.21.15.52:0/3603416900 learned_addr learned my addr 172.21.15.52:0/3603416900 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-05-27T07:36:35.320 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:35.323+0000 7f7d7189c700 1 --2- 172.21.15.52:0/3603416900 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f7d6c108f10 0x7f7d6c10b170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:36:35.321 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:35.324+0000 7f7d7209d700 1 -- 172.21.15.52:0/3603416900 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f7d6c109860 msgr2=0x7f7d6c10b6b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:36:35.321 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:35.324+0000 7f7d7209d700 1 --2- 172.21.15.52:0/3603416900 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f7d6c109860 0x7f7d6c10b6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:36:35.321 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:35.324+0000 7f7d7209d700 1 -- 172.21.15.52:0/3603416900 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f7d6c108f10 msgr2=0x7f7d6c10b170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:36:35.321 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:35.324+0000 7f7d7209d700 1 --2- 172.21.15.52:0/3603416900 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f7d6c108f10 0x7f7d6c10b170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:36:35.321 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:35.324+0000 7f7d7209d700 1 -- 172.21.15.52:0/3603416900 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7d6c10be60 con 0x7f7d6c1045e0 2024-05-27T07:36:35.321 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:35.324+0000 7f7d7189c700 1 --2- 172.21.15.52:0/3603416900 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f7d6c108f10 0x7f7d6c10b170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-27T07:36:35.322 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:35.324+0000 7f7d7209d700 1 --2- 172.21.15.52:0/3603416900 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f7d6c1045e0 0x7f7d6c110f70 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7f7d6400b840 tx=0x7f7d6400bb50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:36:35.322 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:35.325+0000 7f7d6b7fe700 1 -- 172.21.15.52:0/3603416900 <== mon.1 v2:172.21.15.173:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7d6400f840 con 0x7f7d6c1045e0 2024-05-27T07:36:35.322 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:35.325+0000 7f7d6b7fe700 1 -- 172.21.15.52:0/3603416900 <== mon.1 v2:172.21.15.173:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f7d64017400 con 0x7f7d6c1045e0 2024-05-27T07:36:35.322 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:35.325+0000 7f7d6b7fe700 1 -- 172.21.15.52:0/3603416900 <== mon.1 v2:172.21.15.173:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7d6400d630 con 0x7f7d6c1045e0 2024-05-27T07:36:35.322 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:35.325+0000 7f7d74301700 1 -- 172.21.15.52:0/3603416900 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7d6c1c72c0 con 0x7f7d6c1045e0 2024-05-27T07:36:35.322 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:35.325+0000 7f7d74301700 1 -- 172.21.15.52:0/3603416900 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7d6c1c77d0 con 0x7f7d6c1045e0 2024-05-27T07:36:35.328 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:35.331+0000 7f7d82d16b80 1 Processor -- start 2024-05-27T07:36:35.331 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:35.335+0000 7f7d82d16b80 1 -- start start 2024-05-27T07:36:35.333 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:35.335+0000 7f7d82d16b80 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x56202ce2ae40 0x56202cfafb30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:36:35.333 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:35.335+0000 7f7d82d16b80 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x56202cfb0070 0x56202cfa9be0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:36:35.334 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:35.335+0000 7f7d82d16b80 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x56202cfaa120 0x56202cfaa530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:36:35.334 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:35.335+0000 7f7d82d16b80 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x56202ced7d70 con 0x56202ce2ae40 2024-05-27T07:36:35.334 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:35.335+0000 7f7d82d16b80 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x56202cec7910 con 0x56202cfaa120 2024-05-27T07:36:35.334 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:35.335+0000 7f7d82d16b80 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x56202cecb950 con 0x56202cfb0070 2024-05-27T07:36:35.334 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:35.335+0000 7f7d6b7fe700 1 -- 172.21.15.52:0/3603416900 <== mon.1 v2:172.21.15.173:3300/0 4 ==== mgrmap(e 26) ==== 149783+0+0 (secure 0 0 0) 0x7f7d64041bd0 con 0x7f7d6c1045e0 2024-05-27T07:36:35.334 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:35.336+0000 7f7d7189c700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x56202cfaa120 0x56202cfaa530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:36:35.334 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:35.336+0000 7f7d7189c700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x56202cfaa120 0x56202cfaa530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.52:50118/0 (socket says 172.21.15.52:50118) 2024-05-27T07:36:35.334 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:35.336+0000 7f7d7189c700 1 -- 172.21.15.52:0/1200186992 learned_addr learned my addr 172.21.15.52:0/1200186992 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-05-27T07:36:35.334 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:35.337+0000 7f7d6b7fe700 1 --2- 172.21.15.52:0/3603416900 >> [v2:172.21.15.52:6800/1822318384,v1:172.21.15.52:6801/1822318384] conn(0x7f7d540b2ab0 0x7f7d540b4fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:36:35.334 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:35.338+0000 7f7d6b7fe700 1 -- 172.21.15.52:0/3603416900 <== mon.1 v2:172.21.15.173:3300/0 5 ==== osd_map(127..127 src has 1..127) ==== 9555+0+0 (secure 0 0 0) 0x7f7d640e2f10 con 0x7f7d6c1045e0 2024-05-27T07:36:35.334 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:35.338+0000 7f7d7209d700 1 --2- 172.21.15.52:0/1200186992 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x56202cfb0070 0x56202cfa9be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:36:35.336 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:35.338+0000 7f7d7289e700 1 --2- 172.21.15.52:0/1200186992 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x56202ce2ae40 0x56202cfafb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:36:35.336 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:35.339+0000 7f7d7209d700 1 -- 172.21.15.52:0/1200186992 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x56202cfaa120 msgr2=0x56202cfaa530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:36:35.336 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:35.339+0000 7f7d7209d700 1 --2- 172.21.15.52:0/1200186992 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x56202cfaa120 0x56202cfaa530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:36:35.336 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:35.339+0000 7f7d7209d700 1 -- 172.21.15.52:0/1200186992 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x56202ce2ae40 msgr2=0x56202cfafb30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:36:35.336 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:35.339+0000 7f7d7209d700 1 --2- 172.21.15.52:0/1200186992 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x56202ce2ae40 0x56202cfafb30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:36:35.336 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:35.339+0000 7f7d7209d700 1 -- 172.21.15.52:0/1200186992 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56202ce70fb0 con 0x56202cfb0070 2024-05-27T07:36:35.336 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:35.339+0000 7f7d7209d700 1 --2- 172.21.15.52:0/1200186992 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x56202cfb0070 0x56202cfa9be0 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7f7d640e01f0 tx=0x7f7d640aa480 comp rx=0 tx=0).ready entity=mon.1 client_cookie=5b3fa29270975a0b server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:36:35.336 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:35.339+0000 7f7d7289e700 1 --2- 172.21.15.52:0/1200186992 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x56202ce2ae40 0x56202cfafb30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-27T07:36:35.337 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:35.340+0000 7f7d68d47700 1 -- 172.21.15.52:0/1200186992 <== mon.1 v2:172.21.15.173:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7d640e0680 con 0x56202cfb0070 2024-05-27T07:36:35.337 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:35.340+0000 7f7d68d47700 1 -- 172.21.15.52:0/1200186992 <== mon.1 v2:172.21.15.173:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f7d640b2be0 con 0x56202cfb0070 2024-05-27T07:36:35.337 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:35.340+0000 7f7d68d47700 1 -- 172.21.15.52:0/1200186992 <== mon.1 v2:172.21.15.173:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7d640b29b0 con 0x56202cfb0070 2024-05-27T07:36:35.338 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:35.342+0000 7f7d82d16b80 1 -- 172.21.15.52:0/1200186992 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x56202cfb0070 msgr2=0x56202cfa9be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:36:35.338 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:35.342+0000 7f7d82d16b80 1 --2- 172.21.15.52:0/1200186992 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x56202cfb0070 0x56202cfa9be0 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7f7d640e01f0 tx=0x7f7d640aa480 comp rx=0 tx=0).stop 2024-05-27T07:36:35.339 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:35.343+0000 7f7d7289e700 1 --2- 172.21.15.52:0/3603416900 >> [v2:172.21.15.52:6800/1822318384,v1:172.21.15.52:6801/1822318384] conn(0x7f7d540b2ab0 0x7f7d540b4fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:36:35.339 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:35.343+0000 7f7d82d16b80 1 -- 172.21.15.52:0/1200186992 shutdown_connections 2024-05-27T07:36:35.339 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:35.343+0000 7f7d82d16b80 1 --2- 172.21.15.52:0/1200186992 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x56202cfaa120 0x56202cfaa530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:36:35.339 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:35.343+0000 7f7d82d16b80 1 --2- 172.21.15.52:0/1200186992 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x56202ce2ae40 0x56202cfafb30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:36:35.339 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:35.343+0000 7f7d82d16b80 1 --2- 172.21.15.52:0/1200186992 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x56202cfb0070 0x56202cfa9be0 unknown :-1 s=CLOSED pgs=227 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:36:35.339 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:35.343+0000 7f7d82d16b80 1 -- 172.21.15.52:0/1200186992 >> 172.21.15.52:0/1200186992 conn(0x56202ce2d770 msgr2=0x56202cfa4d90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:36:35.346 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:35.345+0000 7f7d7289e700 1 --2- 172.21.15.52:0/3603416900 >> [v2:172.21.15.52:6800/1822318384,v1:172.21.15.52:6801/1822318384] conn(0x7f7d540b2ab0 0x7f7d540b4fc0 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7f7d5c01a9a0 tx=0x7f7d5c01a320 comp rx=0 tx=0).ready entity=mgr.15333 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:36:35.346 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:35.347+0000 7f7d82d16b80 1 -- 172.21.15.52:0/1200186992 shutdown_connections 2024-05-27T07:36:35.347 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:35.350+0000 7f7d82d16b80 1 -- 172.21.15.52:0/1200186992 wait complete. 2024-05-27T07:36:35.347 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:35.351+0000 7f7d82d16b80 1 -- start start 2024-05-27T07:36:35.347 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:35.351+0000 7f7d82d16b80 1 Processor -- start 2024-05-27T07:36:35.349 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:35.353+0000 7f7d82d16b80 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x56202ce2ae40 0x56202cfb38a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:36:35.349 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:35.353+0000 7f7d82d16b80 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x56202cfaa120 0x56202cfb3de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:36:35.349 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:35.353+0000 7f7d82d16b80 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x56202cfb0070 0x56202cfb4320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:36:35.349 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:35.353+0000 7f7d82d16b80 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x56202cdcdfa0 con 0x56202ce2ae40 2024-05-27T07:36:35.349 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:35.353+0000 7f7d82d16b80 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x56202ce2fda0 con 0x56202cfb0070 2024-05-27T07:36:35.349 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:35.353+0000 7f7d82d16b80 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x56202cfc7b70 con 0x56202cfaa120 2024-05-27T07:36:35.349 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:35.353+0000 7f7d82d16b80 10 client.0 _ll_register_callbacks cb 0x56202ce5e5b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T07:36:35.350 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:35.354+0000 7f7d7289e700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x56202cfb0070 0x56202cfb4320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:36:35.350 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:35.354+0000 7f7d7289e700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x56202cfb0070 0x56202cfb4320 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.52:50130/0 (socket says 172.21.15.52:50130) 2024-05-27T07:36:35.350 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:35.354+0000 7f7d7289e700 1 -- 172.21.15.52:0/3690642405 learned_addr learned my addr 172.21.15.52:0/3690642405 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-05-27T07:36:35.350 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:35.355+0000 7f7d7189c700 1 --2- 172.21.15.52:0/3690642405 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x56202cfaa120 0x56202cfb3de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:36:35.351 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:35.355+0000 7f7d7289e700 1 -- 172.21.15.52:0/3690642405 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x56202cfaa120 msgr2=0x56202cfb3de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:36:35.351 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:35.355+0000 7f7d7289e700 1 --2- 172.21.15.52:0/3690642405 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x56202cfaa120 0x56202cfb3de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:36:35.351 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:35.355+0000 7f7d7289e700 1 -- 172.21.15.52:0/3690642405 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x56202ce2ae40 msgr2=0x56202cfb38a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:36:35.351 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:35.355+0000 7f7d7289e700 1 --2- 172.21.15.52:0/3690642405 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x56202ce2ae40 0x56202cfb38a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:36:35.351 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:35.356+0000 7f7d7289e700 1 -- 172.21.15.52:0/3690642405 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56202ce178d0 con 0x56202cfb0070 2024-05-27T07:36:35.353 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:35.357+0000 7f7d7289e700 1 --2- 172.21.15.52:0/3690642405 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x56202cfb0070 0x56202cfb4320 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 crypto rx=0x7f7d5c003890 tx=0x7f7d5c01c550 comp rx=0 tx=0).ready entity=mon.2 client_cookie=3aa23c01f79b1e05 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:36:35.354 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:35.358+0000 7f7d51ffb700 10 client.0 ms_handle_connect on v2:172.21.15.105:3300/0 2024-05-27T07:36:35.354 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:35.358+0000 7f7d51ffb700 1 -- 172.21.15.52:0/3690642405 <== mon.2 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7d5c02c020 con 0x56202cfb0070 2024-05-27T07:36:35.354 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:35.358+0000 7f7d82d16b80 1 -- 172.21.15.52:0/3690642405 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f7d640b2be0 con 0x56202cfb0070 2024-05-27T07:36:35.354 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:35.358+0000 7f7d51ffb700 1 -- 172.21.15.52:0/3690642405 <== mon.2 v2:172.21.15.105:3300/0 2 ==== osd_map(127..127 src has 1..127) ==== 9555+0+0 (secure 0 0 0) 0x7f7d5c005240 con 0x56202cfb0070 2024-05-27T07:36:35.354 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:35.358+0000 7f7d51ffb700 1 -- 172.21.15.52:0/3690642405 <== mon.2 v2:172.21.15.105:3300/0 3 ==== mon_get_version_reply(handle=1 version=108) ==== 24+0+0 (secure 0 0 0) 0x7f7d5c037050 con 0x56202cfb0070 2024-05-27T07:36:35.355 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:35.358+0000 7f7d82d16b80 10 client.26939 fetch_fsmap learned FSMap version 108 2024-05-27T07:36:35.355 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:35.359+0000 7f7d82d16b80 1 -- 172.21.15.52:0/3690642405 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,fsmap=108,monmap=4+}) -- 0x56202ce81760 con 0x56202cfb0070 2024-05-27T07:36:35.355 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:35.359+0000 7f7d51ffb700 1 -- 172.21.15.52:0/3690642405 <== mon.2 v2:172.21.15.105:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f7d640b2be0 con 0x56202cfb0070 2024-05-27T07:36:35.355 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:35.359+0000 7f7d51ffb700 1 -- 172.21.15.52:0/3690642405 <== mon.2 v2:172.21.15.105:3300/0 5 ==== fsmap(e 108) ==== 4203+0+0 (secure 0 0 0) 0x7f7d5c0324c0 con 0x56202cfb0070 2024-05-27T07:36:35.356 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:35.359+0000 7f7d82d16b80 10 client.26939 fetch_fsmap finished waiting for FSMap version 108 2024-05-27T07:36:35.356 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:35.359+0000 7f7d82d16b80 10 client.26939 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.i 2024-05-27T07:36:35.356 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:35.359+0000 7f7d82d16b80 20 client.26939 populate_metadata read hostname 'smithi052' 2024-05-27T07:36:35.356 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:35.359+0000 7f7d82d16b80 1 --2- 172.21.15.52:0/3690642405 >> [v2:172.21.15.173:6838/564778293,v1:172.21.15.173:6839/564778293] conn(0x56202cfd4f00 0x56202cfd73f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:36:35.356 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:35.359+0000 7f7d82d16b80 4 client.26939 mds_command: new command op to 24475 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-27T07:36:35.356 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:35.359+0000 7f7d82d16b80 1 -- 172.21.15.52:0/3690642405 --> [v2:172.21.15.173:6838/564778293,v1:172.21.15.173:6839/564778293] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x56202ced9cc0 con 0x56202cfd4f00 2024-05-27T07:36:35.367 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:35.371+0000 7f7d7209d700 1 --2- 172.21.15.52:0/3690642405 >> [v2:172.21.15.173:6838/564778293,v1:172.21.15.173:6839/564778293] conn(0x56202cfd4f00 0x56202cfd73f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:36:35.374 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:35.378+0000 7f7d7209d700 1 --2- 172.21.15.52:0/3690642405 >> [v2:172.21.15.173:6838/564778293,v1:172.21.15.173:6839/564778293] conn(0x56202cfd4f00 0x56202cfd73f0 crc :-1 s=READY pgs=51 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=4746c000ceb5115a server_cookie=d325fce1f9662867 in_seq=0 out_seq=0 2024-05-27T07:36:35.375 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:35.378+0000 7f7d51ffb700 10 client.26939 ms_handle_connect on v2:172.21.15.173:6838/564778293 2024-05-27T07:36:35.375 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:35.380+0000 7f7d51ffb700 1 -- 172.21.15.52:0/3690642405 <== mds.2 v2:172.21.15.173:6838/564778293 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24041 (crc 0 0 0) 0x56202ced9cc0 con 0x56202cfd4f00 2024-05-27T07:36:35.375 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:35.380+0000 7f7d51ffb700 10 client.26939 handle_command_reply: tid=0 2024-05-27T07:36:35.379 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:35.382+0000 7f7d82d16b80 1 client.26939 shutdown 2024-05-27T07:36:35.381 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:35.385+0000 7f7d82d16b80 1 -- 172.21.15.52:0/3690642405 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x56202cfb0070 msgr2=0x56202cfb4320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:36:35.381 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:35.385+0000 7f7d82d16b80 1 --2- 172.21.15.52:0/3690642405 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x56202cfb0070 0x56202cfb4320 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 crypto rx=0x7f7d5c003890 tx=0x7f7d5c01c550 comp rx=0 tx=0).stop 2024-05-27T07:36:35.382 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:35.385+0000 7f7d82d16b80 1 -- 172.21.15.52:0/3690642405 shutdown_connections 2024-05-27T07:36:35.382 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:35.385+0000 7f7d82d16b80 1 --2- 172.21.15.52:0/3690642405 >> [v2:172.21.15.173:6838/564778293,v1:172.21.15.173:6839/564778293] conn(0x56202cfd4f00 0x56202cfd73f0 crc :-1 s=READY pgs=51 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:36:35.382 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:35.385+0000 7f7d82d16b80 1 --2- 172.21.15.52:0/3690642405 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x56202cfb0070 0x56202cfb4320 unknown :-1 s=CLOSED pgs=272 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:36:35.382 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:35.385+0000 7f7d82d16b80 1 --2- 172.21.15.52:0/3690642405 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x56202ce2ae40 0x56202cfb38a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:36:35.382 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:35.385+0000 7f7d82d16b80 1 --2- 172.21.15.52:0/3690642405 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x56202cfaa120 0x56202cfb3de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:36:35.382 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:35.385+0000 7f7d82d16b80 1 -- 172.21.15.52:0/3690642405 >> 172.21.15.52:0/3690642405 conn(0x56202ce2d770 msgr2=0x56202cfa8bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:36:35.382 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:35.385+0000 7f7d51ffb700 0 client.26939 ms_handle_reset on v2:172.21.15.173:6838/564778293 2024-05-27T07:36:35.382 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:35.386+0000 7f7d82d16b80 1 -- 172.21.15.52:0/3690642405 shutdown_connections 2024-05-27T07:36:35.383 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:35.386+0000 7f7d82d16b80 1 -- 172.21.15.52:0/3690642405 wait complete. 2024-05-27T07:36:35.383 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:35.386+0000 7f7d82d16b80 20 client.26939 trim_cache size 0 max 16384 2024-05-27T07:36:35.414 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:35.417+0000 7f7d82d16b80 1 Processor -- start 2024-05-27T07:36:35.414 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:35.417+0000 7f7d82d16b80 1 -- start start 2024-05-27T07:36:35.416 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:35.418+0000 7f7d82d16b80 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x56202ce2ae40 0x56202cfb5d40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:36:35.416 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:35.418+0000 7f7d82d16b80 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x56202cfaa120 0x56202cfa9af0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:36:35.416 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:35.419+0000 7f7d82d16b80 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x56202cfb0070 0x56202cfb4160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:36:35.416 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:35.419+0000 7f7d82d16b80 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x7f7d5800f400 con 0x56202cfb0070 2024-05-27T07:36:35.416 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:35.419+0000 7f7d7289e700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x56202cfb0070 0x56202cfb4160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:36:35.416 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:35.419+0000 7f7d7289e700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x56202cfb0070 0x56202cfb4160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.52:39586/0 (socket says 172.21.15.52:39586) 2024-05-27T07:36:35.416 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:35.419+0000 7f7d7289e700 1 -- 172.21.15.52:0/2999058904 learned_addr learned my addr 172.21.15.52:0/2999058904 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-05-27T07:36:35.416 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:35.420+0000 7f7d7189c700 1 --2- 172.21.15.52:0/2999058904 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x56202cfaa120 0x56202cfa9af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:36:35.417 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:35.420+0000 7f7d82d16b80 1 -- 172.21.15.52:0/2999058904 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f7d5800d830 con 0x56202cfaa120 2024-05-27T07:36:35.417 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:35.420+0000 7f7d82d16b80 1 -- 172.21.15.52:0/2999058904 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f7d5800ed00 con 0x56202ce2ae40 2024-05-27T07:36:35.420 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:35.424+0000 7f7d7189c700 1 -- 172.21.15.52:0/2999058904 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x56202ce2ae40 msgr2=0x56202cfb5d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:36:35.420 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:35.424+0000 7f7d7189c700 1 --2- 172.21.15.52:0/2999058904 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x56202ce2ae40 0x56202cfb5d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:36:35.420 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:35.424+0000 7f7d7189c700 1 -- 172.21.15.52:0/2999058904 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x56202cfb0070 msgr2=0x56202cfb4160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:36:35.420 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:35.424+0000 7f7d7189c700 1 --2- 172.21.15.52:0/2999058904 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x56202cfb0070 0x56202cfb4160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:36:35.420 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:35.424+0000 7f7d7189c700 1 -- 172.21.15.52:0/2999058904 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56202cfb1e30 con 0x56202cfaa120 2024-05-27T07:36:35.422 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:35.427+0000 7f7d7189c700 1 --2- 172.21.15.52:0/2999058904 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x56202cfaa120 0x56202cfa9af0 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 crypto rx=0x56202cfa0fa0 tx=0x7f7d60009190 comp rx=0 tx=0).ready entity=mon.2 client_cookie=70cfdb7979665995 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:36:35.424 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:35.428+0000 7f7d53fff700 1 -- 172.21.15.52:0/2999058904 <== mon.2 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7d6001a030 con 0x56202cfaa120 2024-05-27T07:36:35.424 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:35.428+0000 7f7d53fff700 1 -- 172.21.15.52:0/2999058904 <== mon.2 v2:172.21.15.105:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f7d60013a10 con 0x56202cfaa120 2024-05-27T07:36:35.424 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:35.428+0000 7f7d53fff700 1 -- 172.21.15.52:0/2999058904 <== mon.2 v2:172.21.15.105:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7d6000f040 con 0x56202cfaa120 2024-05-27T07:36:35.425 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:35.430+0000 7f7d82d16b80 1 -- 172.21.15.52:0/2999058904 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x56202cfaa120 msgr2=0x56202cfa9af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:36:35.425 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:35.430+0000 7f7d82d16b80 1 --2- 172.21.15.52:0/2999058904 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x56202cfaa120 0x56202cfa9af0 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 crypto rx=0x56202cfa0fa0 tx=0x7f7d60009190 comp rx=0 tx=0).stop 2024-05-27T07:36:35.427 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:35.431+0000 7f7d82d16b80 1 -- 172.21.15.52:0/2999058904 shutdown_connections 2024-05-27T07:36:35.427 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:35.431+0000 7f7d82d16b80 1 --2- 172.21.15.52:0/2999058904 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x56202cfb0070 0x56202cfb4160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:36:35.427 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:35.431+0000 7f7d82d16b80 1 --2- 172.21.15.52:0/2999058904 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x56202ce2ae40 0x56202cfb5d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:36:35.427 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:35.431+0000 7f7d82d16b80 1 --2- 172.21.15.52:0/2999058904 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x56202cfaa120 0x56202cfa9af0 unknown :-1 s=CLOSED pgs=273 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:36:35.427 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:35.431+0000 7f7d82d16b80 1 -- 172.21.15.52:0/2999058904 >> 172.21.15.52:0/2999058904 conn(0x56202ce2d770 msgr2=0x56202cfa93e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:36:35.429 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:35.434+0000 7f7d82d16b80 1 -- 172.21.15.52:0/2999058904 shutdown_connections 2024-05-27T07:36:35.429 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:35.434+0000 7f7d82d16b80 1 -- 172.21.15.52:0/2999058904 wait complete. 2024-05-27T07:36:35.430 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:35.434+0000 7f7d82d16b80 1 -- start start 2024-05-27T07:36:35.430 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:35.435+0000 7f7d82d16b80 1 Processor -- start 2024-05-27T07:36:35.432 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:35.435+0000 7f7d82d16b80 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x56202ce2ae40 0x56202cfcd210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:36:35.432 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:35.435+0000 7f7d82d16b80 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x56202cfaa120 0x56202cfdc1c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:36:35.432 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:35.436+0000 7f7d82d16b80 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x56202cfb0070 0x56202cfdc700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:36:35.432 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:35.436+0000 7f7d82d16b80 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x7f7d5800e630 con 0x56202cfb0070 2024-05-27T07:36:35.432 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:35.436+0000 7f7d82d16b80 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f7d5800df30 con 0x56202ce2ae40 2024-05-27T07:36:35.432 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:35.436+0000 7f7d82d16b80 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x56202cfa8bd0 con 0x56202cfaa120 2024-05-27T07:36:35.432 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:35.436+0000 7f7d82d16b80 10 client.0 _ll_register_callbacks cb 0x56202ce5e5b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T07:36:35.433 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:35.436+0000 7f7d7289e700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x56202cfb0070 0x56202cfdc700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:36:35.433 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:35.436+0000 7f7d7289e700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x56202cfb0070 0x56202cfdc700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.52:39598/0 (socket says 172.21.15.52:39598) 2024-05-27T07:36:35.433 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:35.437+0000 7f7d7289e700 1 -- 172.21.15.52:0/2468674134 learned_addr learned my addr 172.21.15.52:0/2468674134 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-05-27T07:36:35.433 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:35.437+0000 7f7d7209d700 1 --2- 172.21.15.52:0/2468674134 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x56202ce2ae40 0x56202cfcd210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:36:35.434 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:35.437+0000 7f7d7189c700 1 --2- 172.21.15.52:0/2468674134 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x56202cfaa120 0x56202cfdc1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:36:35.434 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:35.437+0000 7f7d7209d700 1 -- 172.21.15.52:0/2468674134 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x56202cfaa120 msgr2=0x56202cfdc1c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:36:35.434 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:35.437+0000 7f7d7209d700 1 --2- 172.21.15.52:0/2468674134 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x56202cfaa120 0x56202cfdc1c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:36:35.434 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:35.437+0000 7f7d7209d700 1 -- 172.21.15.52:0/2468674134 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x56202cfb0070 msgr2=0x56202cfdc700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:36:35.434 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:35.437+0000 7f7d7209d700 1 --2- 172.21.15.52:0/2468674134 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x56202cfb0070 0x56202cfdc700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:36:35.434 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:35.437+0000 7f7d7209d700 1 -- 172.21.15.52:0/2468674134 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56202cfdcc40 con 0x56202ce2ae40 2024-05-27T07:36:35.434 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:35.437+0000 7f7d7189c700 1 --2- 172.21.15.52:0/2468674134 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x56202cfaa120 0x56202cfdc1c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-27T07:36:35.434 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:35.437+0000 7f7d7209d700 1 --2- 172.21.15.52:0/2468674134 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x56202ce2ae40 0x56202cfcd210 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 crypto rx=0x7f7d640e0220 tx=0x7f7d640a6500 comp rx=0 tx=0).ready entity=mon.2 client_cookie=44726e17576ae474 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:36:35.435 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:35.438+0000 7f7d51ffb700 10 client.0 ms_handle_connect on v2:172.21.15.105:3300/0 2024-05-27T07:36:35.435 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:35.438+0000 7f7d51ffb700 1 -- 172.21.15.52:0/2468674134 <== mon.2 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7d640c9060 con 0x56202ce2ae40 2024-05-27T07:36:35.435 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:35.438+0000 7f7d51ffb700 1 -- 172.21.15.52:0/2468674134 <== mon.2 v2:172.21.15.105:3300/0 2 ==== osd_map(127..127 src has 1..127) ==== 9555+0+0 (secure 0 0 0) 0x7f7d640c4410 con 0x56202ce2ae40 2024-05-27T07:36:35.435 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:35.438+0000 7f7d82d16b80 1 -- 172.21.15.52:0/2468674134 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f7d60013a10 con 0x56202ce2ae40 2024-05-27T07:36:35.435 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:35.438+0000 7f7d51ffb700 1 -- 172.21.15.52:0/2468674134 <== mon.2 v2:172.21.15.105:3300/0 3 ==== mon_get_version_reply(handle=1 version=108) ==== 24+0+0 (secure 0 0 0) 0x7f7d640aa1f0 con 0x56202ce2ae40 2024-05-27T07:36:35.435 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:35.438+0000 7f7d82d16b80 10 client.26945 fetch_fsmap learned FSMap version 108 2024-05-27T07:36:35.435 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:35.438+0000 7f7d82d16b80 1 -- 172.21.15.52:0/2468674134 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,fsmap=108,monmap=4+}) -- 0x56202d03f6c0 con 0x56202ce2ae40 2024-05-27T07:36:35.435 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:35.439+0000 7f7d51ffb700 1 -- 172.21.15.52:0/2468674134 <== mon.2 v2:172.21.15.105:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f7d6406d030 con 0x56202ce2ae40 2024-05-27T07:36:35.435 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:35.439+0000 7f7d51ffb700 1 -- 172.21.15.52:0/2468674134 <== mon.2 v2:172.21.15.105:3300/0 5 ==== fsmap(e 108) ==== 4203+0+0 (secure 0 0 0) 0x7f7d640a58f0 con 0x56202ce2ae40 2024-05-27T07:36:35.436 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:35.439+0000 7f7d82d16b80 10 client.26945 fetch_fsmap finished waiting for FSMap version 108 2024-05-27T07:36:35.436 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:35.439+0000 7f7d82d16b80 10 client.26945 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.i 2024-05-27T07:36:35.436 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:35.439+0000 7f7d82d16b80 20 client.26945 populate_metadata read hostname 'smithi052' 2024-05-27T07:36:35.436 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:35.439+0000 7f7d82d16b80 1 --2- 172.21.15.52:0/2468674134 >> [v2:172.21.15.173:6838/564778293,v1:172.21.15.173:6839/564778293] conn(0x56202d03fcc0 0x56202d0420e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:36:35.436 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:35.439+0000 7f7d82d16b80 4 client.26945 mds_command: new command op to 24475 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-05-27T07:36:35.436 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:35.439+0000 7f7d82d16b80 1 -- 172.21.15.52:0/2468674134 --> [v2:172.21.15.173:6838/564778293,v1:172.21.15.173:6839/564778293] -- command(tid 0: {"prefix": "damage ls"}) -- 0x7f7d60013a10 con 0x56202d03fcc0 2024-05-27T07:36:35.442 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:35.446+0000 7f7d7189c700 1 --2- 172.21.15.52:0/2468674134 >> [v2:172.21.15.173:6838/564778293,v1:172.21.15.173:6839/564778293] conn(0x56202d03fcc0 0x56202d0420e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:36:35.445 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:35.449+0000 7f7d7189c700 1 --2- 172.21.15.52:0/2468674134 >> [v2:172.21.15.173:6838/564778293,v1:172.21.15.173:6839/564778293] conn(0x56202d03fcc0 0x56202d0420e0 crc :-1 s=READY pgs=52 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=428a2239eb729bf8 server_cookie=49cb0eebbd5624dd in_seq=0 out_seq=0 2024-05-27T07:36:35.445 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:35.449+0000 7f7d51ffb700 10 client.26945 ms_handle_connect on v2:172.21.15.173:6838/564778293 2024-05-27T07:36:35.455 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:35.460+0000 7f7d51ffb700 1 -- 172.21.15.52:0/2468674134 <== mds.2 v2:172.21.15.173:6838/564778293 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x7f7d60013a10 con 0x56202d03fcc0 2024-05-27T07:36:35.455 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:35.460+0000 7f7d51ffb700 10 client.26945 handle_command_reply: tid=0 2024-05-27T07:36:35.456 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:35.460+0000 7f7d82d16b80 1 client.26945 shutdown 2024-05-27T07:36:35.457 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:35.462+0000 7f7d82d16b80 1 -- 172.21.15.52:0/2468674134 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x56202ce2ae40 msgr2=0x56202cfcd210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:36:35.458 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:35.462+0000 7f7d82d16b80 1 --2- 172.21.15.52:0/2468674134 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x56202ce2ae40 0x56202cfcd210 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 crypto rx=0x7f7d640e0220 tx=0x7f7d640a6500 comp rx=0 tx=0).stop 2024-05-27T07:36:35.460 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:35.464+0000 7f7d82d16b80 1 -- 172.21.15.52:0/2468674134 shutdown_connections 2024-05-27T07:36:35.460 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:35.464+0000 7f7d82d16b80 1 --2- 172.21.15.52:0/2468674134 >> [v2:172.21.15.173:6838/564778293,v1:172.21.15.173:6839/564778293] conn(0x56202d03fcc0 0x56202d0420e0 crc :-1 s=READY pgs=52 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:36:35.460 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:35.464+0000 7f7d82d16b80 1 --2- 172.21.15.52:0/2468674134 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x56202cfb0070 0x56202cfdc700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:36:35.460 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:35.464+0000 7f7d51ffb700 0 client.26945 ms_handle_reset on v2:172.21.15.173:6838/564778293 2024-05-27T07:36:35.460 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:35.464+0000 7f7d82d16b80 1 --2- 172.21.15.52:0/2468674134 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x56202ce2ae40 0x56202cfcd210 unknown :-1 s=CLOSED pgs=274 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:36:35.460 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:35.464+0000 7f7d82d16b80 1 --2- 172.21.15.52:0/2468674134 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x56202cfaa120 0x56202cfdc1c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:36:35.460 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:35.464+0000 7f7d82d16b80 1 -- 172.21.15.52:0/2468674134 >> 172.21.15.52:0/2468674134 conn(0x56202ce2d770 msgr2=0x56202cfae800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:36:35.464 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:35.468+0000 7f7d82d16b80 1 -- 172.21.15.52:0/2468674134 shutdown_connections 2024-05-27T07:36:35.464 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:35.468+0000 7f7d82d16b80 1 -- 172.21.15.52:0/2468674134 wait complete. 2024-05-27T07:36:35.464 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:35.469+0000 7f7d82d16b80 20 client.26945 trim_cache size 0 max 16384 2024-05-27T07:36:35.468 INFO:teuthology.orchestra.run.smithi052.stdout:[] 2024-05-27T07:36:35.470 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:35.475+0000 7f7d74301700 1 -- 172.21.15.52:0/3603416900 >> [v2:172.21.15.52:6800/1822318384,v1:172.21.15.52:6801/1822318384] conn(0x7f7d540b2ab0 msgr2=0x7f7d540b4fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:36:35.471 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:35.475+0000 7f7d74301700 1 --2- 172.21.15.52:0/3603416900 >> [v2:172.21.15.52:6800/1822318384,v1:172.21.15.52:6801/1822318384] conn(0x7f7d540b2ab0 0x7f7d540b4fc0 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7f7d5c01a9a0 tx=0x7f7d5c01a320 comp rx=0 tx=0).stop 2024-05-27T07:36:35.471 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:35.475+0000 7f7d74301700 1 -- 172.21.15.52:0/3603416900 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f7d6c1045e0 msgr2=0x7f7d6c110f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:36:35.471 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:35.475+0000 7f7d74301700 1 --2- 172.21.15.52:0/3603416900 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f7d6c1045e0 0x7f7d6c110f70 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7f7d6400b840 tx=0x7f7d6400bb50 comp rx=0 tx=0).stop 2024-05-27T07:36:35.471 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:35.475+0000 7f7d74301700 1 -- 172.21.15.52:0/3603416900 shutdown_connections 2024-05-27T07:36:35.472 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:35.475+0000 7f7d74301700 1 --2- 172.21.15.52:0/3603416900 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f7d6c109860 0x7f7d6c10b6b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:36:35.472 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:35.475+0000 7f7d74301700 1 --2- 172.21.15.52:0/3603416900 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f7d6c1045e0 0x7f7d6c110f70 unknown :-1 s=CLOSED pgs=226 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:36:35.472 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:35.475+0000 7f7d74301700 1 --2- 172.21.15.52:0/3603416900 >> [v2:172.21.15.52:6800/1822318384,v1:172.21.15.52:6801/1822318384] conn(0x7f7d540b2ab0 0x7f7d540b4fc0 unknown :-1 s=CLOSED pgs=197 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:36:35.472 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:35.475+0000 7f7d74301700 1 --2- 172.21.15.52:0/3603416900 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f7d6c108f10 0x7f7d6c10b170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:36:35.472 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:35.475+0000 7f7d74301700 1 -- 172.21.15.52:0/3603416900 >> 172.21.15.52:0/3603416900 conn(0x7f7d6c100030 msgr2=0x7f7d6c108560 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:36:35.472 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:35.477+0000 7f7d74301700 1 -- 172.21.15.52:0/3603416900 shutdown_connections 2024-05-27T07:36:35.473 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:35.477+0000 7f7d74301700 1 -- 172.21.15.52:0/3603416900 wait complete. 2024-05-27T07:36:36.345 DEBUG:teuthology.orchestra.run.smithi052:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:38f9d900c83c63799fbdbe61acc9a11b0d3554a6 shell --fsid 73df2636-1bf8-11ef-bc9b-c7b262605968 -- ceph tell mds.1:1 damage ls 2024-05-27T07:36:36.696 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:36:36 smithi105 ceph-mon[35977]: pgmap v358: 129 pgs: 16 active+clean+snaptrim, 55 active+clean+snaptrim_wait, 58 active+clean; 629 MiB data, 7.1 GiB used, 1.0 TiB / 1.0 TiB avail 2024-05-27T07:36:36.736 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:36:36 smithi052 ceph-mon[31955]: pgmap v358: 129 pgs: 16 active+clean+snaptrim, 55 active+clean+snaptrim_wait, 58 active+clean; 629 MiB data, 7.1 GiB used, 1.0 TiB / 1.0 TiB avail 2024-05-27T07:36:36.826 INFO:teuthology.orchestra.run.smithi052.stderr:Inferring config /var/lib/ceph/73df2636-1bf8-11ef-bc9b-c7b262605968/mon.a/config 2024-05-27T07:36:36.858 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:36:36 smithi173 ceph-mon[37000]: pgmap v358: 129 pgs: 16 active+clean+snaptrim, 55 active+clean+snaptrim_wait, 58 active+clean; 629 MiB data, 7.1 GiB used, 1.0 TiB / 1.0 TiB avail 2024-05-27T07:36:38.696 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:36:38 smithi105 ceph-mon[35977]: pgmap v359: 129 pgs: 16 active+clean+snaptrim, 55 active+clean+snaptrim_wait, 58 active+clean; 628 MiB data, 7.1 GiB used, 1.0 TiB / 1.0 TiB avail 2024-05-27T07:36:38.720 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:36:38 smithi052 ceph-mon[31955]: pgmap v359: 129 pgs: 16 active+clean+snaptrim, 55 active+clean+snaptrim_wait, 58 active+clean; 628 MiB data, 7.1 GiB used, 1.0 TiB / 1.0 TiB avail 2024-05-27T07:36:38.858 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:36:38 smithi173 ceph-mon[37000]: pgmap v359: 129 pgs: 16 active+clean+snaptrim, 55 active+clean+snaptrim_wait, 58 active+clean; 628 MiB data, 7.1 GiB used, 1.0 TiB / 1.0 TiB avail 2024-05-27T07:36:40.466 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:40.470+0000 7f26cb7fe700 1 -- 172.21.15.52:0/1802526557 <== mon.1 v2:172.21.15.173:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f26b4003c70 con 0x7f26cc115140 2024-05-27T07:36:40.467 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:40.470+0000 7f26d1a42700 1 -- 172.21.15.52:0/1802526557 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f26cc115140 msgr2=0x7f26cc117560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:36:40.467 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:40.470+0000 7f26d1a42700 1 --2- 172.21.15.52:0/1802526557 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f26cc115140 0x7f26cc117560 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7f26b4009990 tx=0x7f26b4009ca0 comp rx=0 tx=0).stop 2024-05-27T07:36:40.467 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:40.470+0000 7f26d1a42700 1 -- 172.21.15.52:0/1802526557 shutdown_connections 2024-05-27T07:36:40.467 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:40.470+0000 7f26d1a42700 1 --2- 172.21.15.52:0/1802526557 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f26cc115140 0x7f26cc117560 unknown :-1 s=CLOSED pgs=228 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:36:40.467 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:40.470+0000 7f26d1a42700 1 --2- 172.21.15.52:0/1802526557 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f26cc075b50 0x7f26cc075f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:36:40.467 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:40.470+0000 7f26d1a42700 1 --2- 172.21.15.52:0/1802526557 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f26cc0764a0 0x7f26cc114c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:36:40.467 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:40.470+0000 7f26d1a42700 1 -- 172.21.15.52:0/1802526557 >> 172.21.15.52:0/1802526557 conn(0x7f26cc07acb0 msgr2=0x7f26cc07b0c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:36:40.467 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:40.471+0000 7f26d1a42700 1 -- 172.21.15.52:0/1802526557 shutdown_connections 2024-05-27T07:36:40.471 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:40.473+0000 7f26d1a42700 1 -- 172.21.15.52:0/1802526557 wait complete. 2024-05-27T07:36:40.471 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:40.474+0000 7f26d1a42700 1 Processor -- start 2024-05-27T07:36:40.471 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:40.474+0000 7f26d1a42700 1 -- start start 2024-05-27T07:36:40.471 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:40.474+0000 7f26d1a42700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f26cc075b50 0x7f26cc114b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:36:40.471 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:40.474+0000 7f26d1a42700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f26cc115140 0x7f26cc10dc90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:36:40.471 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:40.474+0000 7f26d1a42700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f26cc10e270 0x7f26cc10e720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:36:40.471 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:40.474+0000 7f26d1a42700 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x7f26cc11aa00 con 0x7f26cc115140 2024-05-27T07:36:40.472 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:40.474+0000 7f26d1a42700 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f26cc11a880 con 0x7f26cc10e270 2024-05-27T07:36:40.472 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:40.474+0000 7f26d1a42700 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f26cc11ab80 con 0x7f26cc075b50 2024-05-27T07:36:40.472 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:40.477+0000 7f26cbfff700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f26cc115140 0x7f26cc10dc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:36:40.473 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:40.477+0000 7f26cbfff700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f26cc115140 0x7f26cc10dc90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.52:39608/0 (socket says 172.21.15.52:39608) 2024-05-27T07:36:40.473 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:40.477+0000 7f26cbfff700 1 -- 172.21.15.52:0/3436560025 learned_addr learned my addr 172.21.15.52:0/3436560025 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-05-27T07:36:40.474 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:40.478+0000 7f26cbfff700 1 -- 172.21.15.52:0/3436560025 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f26cc075b50 msgr2=0x7f26cc114b60 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-05-27T07:36:40.475 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:40.478+0000 7f26cbfff700 1 --2- 172.21.15.52:0/3436560025 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f26cc075b50 0x7f26cc114b60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:36:40.475 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:40.478+0000 7f26cbfff700 1 -- 172.21.15.52:0/3436560025 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f26cc10e270 msgr2=0x7f26cc10e720 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-05-27T07:36:40.475 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:40.478+0000 7f26cbfff700 1 --2- 172.21.15.52:0/3436560025 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f26cc10e270 0x7f26cc10e720 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:36:40.475 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:40.478+0000 7f26cbfff700 1 -- 172.21.15.52:0/3436560025 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f26cc112990 con 0x7f26cc115140 2024-05-27T07:36:40.476 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:40.479+0000 7f26cbfff700 1 --2- 172.21.15.52:0/3436560025 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f26cc115140 0x7f26cc10dc90 secure :-1 s=READY pgs=760 cs=0 l=1 rev1=1 crypto rx=0x7f26c400b9b0 tx=0x7f26c400bd70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:36:40.476 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:40.479+0000 7f26c9ffb700 1 -- 172.21.15.52:0/3436560025 <== mon.0 v2:172.21.15.52:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f26c400ca30 con 0x7f26cc115140 2024-05-27T07:36:40.476 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:40.480+0000 7f26c9ffb700 1 -- 172.21.15.52:0/3436560025 <== mon.0 v2:172.21.15.52:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f26c400cbd0 con 0x7f26cc115140 2024-05-27T07:36:40.476 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:40.480+0000 7f26c9ffb700 1 -- 172.21.15.52:0/3436560025 <== mon.0 v2:172.21.15.52:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f26c40125a0 con 0x7f26cc115140 2024-05-27T07:36:40.476 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:40.480+0000 7f26d1a42700 1 -- 172.21.15.52:0/3436560025 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f26cc112c80 con 0x7f26cc115140 2024-05-27T07:36:40.476 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:40.480+0000 7f26d1a42700 1 -- 172.21.15.52:0/3436560025 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f26cc113210 con 0x7f26cc115140 2024-05-27T07:36:40.484 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:40.486+0000 7f26e16b9b80 1 Processor -- start 2024-05-27T07:36:40.485 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:40.488+0000 7f26e16b9b80 1 -- start start 2024-05-27T07:36:40.486 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:40.488+0000 7f26e16b9b80 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5566852d8e40 0x556685457010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:36:40.486 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:40.488+0000 7f26e16b9b80 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x556685457550 0x556685457960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:36:40.486 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:40.489+0000 7f26e16b9b80 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x556685461de0 0x556685464200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:36:40.486 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:40.489+0000 7f26e16b9b80 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x556685385d70 con 0x556685457550 2024-05-27T07:36:40.486 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:40.489+0000 7f26e16b9b80 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x556685375910 con 0x556685461de0 2024-05-27T07:36:40.486 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:40.489+0000 7f26e16b9b80 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x556685379950 con 0x5566852d8e40 2024-05-27T07:36:40.486 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:40.489+0000 7f26c9ffb700 1 -- 172.21.15.52:0/3436560025 <== mon.0 v2:172.21.15.52:3300/0 4 ==== mgrmap(e 26) ==== 149783+0+0 (secure 0 0 0) 0x7f26c4014440 con 0x7f26cc115140 2024-05-27T07:36:40.487 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:40.490+0000 7f26d0a40700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x556685457550 0x556685457960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:36:40.487 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:40.491+0000 7f26d0a40700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x556685457550 0x556685457960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.52:39618/0 (socket says 172.21.15.52:39618) 2024-05-27T07:36:40.487 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:40.491+0000 7f26d0a40700 1 -- 172.21.15.52:0/2161422024 learned_addr learned my addr 172.21.15.52:0/2161422024 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-05-27T07:36:40.487 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:40.491+0000 7f26c9ffb700 1 --2- 172.21.15.52:0/3436560025 >> [v2:172.21.15.52:6800/1822318384,v1:172.21.15.52:6801/1822318384] conn(0x7f26b80b2af0 0x7f26b80b5000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:36:40.488 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:40.491+0000 7f26c9ffb700 1 -- 172.21.15.52:0/3436560025 <== mon.0 v2:172.21.15.52:3300/0 5 ==== osd_map(127..127 src has 1..127) ==== 9555+0+0 (secure 0 0 0) 0x7f26c40e22d0 con 0x7f26cc115140 2024-05-27T07:36:40.489 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:40.492+0000 7f26d0a40700 1 -- 172.21.15.52:0/2161422024 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5566852d8e40 msgr2=0x556685457010 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-05-27T07:36:40.489 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:40.492+0000 7f26d0a40700 1 --2- 172.21.15.52:0/2161422024 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5566852d8e40 0x556685457010 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:36:40.489 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:40.492+0000 7f26cbfff700 1 --2- 172.21.15.52:0/2161422024 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x556685461de0 0x556685464200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:36:40.489 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:40.492+0000 7f26d0a40700 1 -- 172.21.15.52:0/2161422024 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x556685461de0 msgr2=0x556685464200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:36:40.489 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:40.493+0000 7f26d1241700 1 --2- 172.21.15.52:0/3436560025 >> [v2:172.21.15.52:6800/1822318384,v1:172.21.15.52:6801/1822318384] conn(0x7f26b80b2af0 0x7f26b80b5000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:36:40.489 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:40.493+0000 7f26d0a40700 1 --2- 172.21.15.52:0/2161422024 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x556685461de0 0x556685464200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:36:40.489 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:40.493+0000 7f26d0a40700 1 -- 172.21.15.52:0/2161422024 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55668531efb0 con 0x556685457550 2024-05-27T07:36:40.489 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:40.494+0000 7f26d0a40700 1 --2- 172.21.15.52:0/2161422024 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x556685457550 0x556685457960 secure :-1 s=READY pgs=761 cs=0 l=1 rev1=1 crypto rx=0x7f26bc009780 tx=0x7f26bc009a90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=891d55982549de20 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:36:40.490 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:40.494+0000 7f26c2ffd700 1 -- 172.21.15.52:0/2161422024 <== mon.0 v2:172.21.15.52:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f26bc015020 con 0x556685457550 2024-05-27T07:36:40.490 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:40.494+0000 7f26c2ffd700 1 -- 172.21.15.52:0/2161422024 <== mon.0 v2:172.21.15.52:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f26bc011ce0 con 0x556685457550 2024-05-27T07:36:40.492 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:40.496+0000 7f26e16b9b80 1 -- 172.21.15.52:0/2161422024 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x556685457550 msgr2=0x556685457960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:36:40.492 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:40.496+0000 7f26e16b9b80 1 --2- 172.21.15.52:0/2161422024 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x556685457550 0x556685457960 secure :-1 s=READY pgs=761 cs=0 l=1 rev1=1 crypto rx=0x7f26bc009780 tx=0x7f26bc009a90 comp rx=0 tx=0).stop 2024-05-27T07:36:40.497 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:40.501+0000 7f26e16b9b80 1 -- 172.21.15.52:0/2161422024 shutdown_connections 2024-05-27T07:36:40.497 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:40.501+0000 7f26e16b9b80 1 --2- 172.21.15.52:0/2161422024 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x556685461de0 0x556685464200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:36:40.497 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:40.501+0000 7f26e16b9b80 1 --2- 172.21.15.52:0/2161422024 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5566852d8e40 0x556685457010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:36:40.498 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:40.501+0000 7f26e16b9b80 1 --2- 172.21.15.52:0/2161422024 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x556685457550 0x556685457960 unknown :-1 s=CLOSED pgs=761 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:36:40.498 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:40.501+0000 7f26e16b9b80 1 -- 172.21.15.52:0/2161422024 >> 172.21.15.52:0/2161422024 conn(0x5566852db770 msgr2=0x556685453040 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:36:40.504 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:40.508+0000 7f26e16b9b80 1 -- 172.21.15.52:0/2161422024 shutdown_connections 2024-05-27T07:36:40.508 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:40.509+0000 7f26e16b9b80 1 -- 172.21.15.52:0/2161422024 wait complete. 2024-05-27T07:36:40.509 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:40.509+0000 7f26e16b9b80 1 -- start start 2024-05-27T07:36:40.509 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:40.509+0000 7f26d1241700 1 --2- 172.21.15.52:0/3436560025 >> [v2:172.21.15.52:6800/1822318384,v1:172.21.15.52:6801/1822318384] conn(0x7f26b80b2af0 0x7f26b80b5000 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7f26b4009750 tx=0x7f26b4005ab0 comp rx=0 tx=0).ready entity=mgr.15333 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:36:40.509 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:40.509+0000 7f26e16b9b80 1 Processor -- start 2024-05-27T07:36:40.511 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:40.514+0000 7f26e16b9b80 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5566852d8e40 0x55668545ed20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:36:40.511 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:40.514+0000 7f26e16b9b80 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x556685457550 0x55668545f260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:36:40.512 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:40.514+0000 7f26e16b9b80 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x556685461de0 0x55668545f7a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:36:40.512 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:40.514+0000 7f26e16b9b80 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x55668527bfa0 con 0x556685457550 2024-05-27T07:36:40.512 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:40.514+0000 7f26e16b9b80 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x5566852ddda0 con 0x5566852d8e40 2024-05-27T07:36:40.512 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:40.514+0000 7f26e16b9b80 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x556685475d60 con 0x556685461de0 2024-05-27T07:36:40.512 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:40.514+0000 7f26e16b9b80 10 client.0 _ll_register_callbacks cb 0x55668530c5b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T07:36:40.512 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:40.515+0000 7f26cbfff700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x556685457550 0x55668545f260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:36:40.512 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:40.515+0000 7f26cbfff700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x556685457550 0x55668545f260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.52:39630/0 (socket says 172.21.15.52:39630) 2024-05-27T07:36:40.512 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:40.515+0000 7f26cbfff700 1 -- 172.21.15.52:0/618276914 learned_addr learned my addr 172.21.15.52:0/618276914 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-05-27T07:36:40.512 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:40.515+0000 7f26d0a40700 1 --2- 172.21.15.52:0/618276914 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5566852d8e40 0x55668545ed20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:36:40.516 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:40.520+0000 7f26cbfff700 1 -- 172.21.15.52:0/618276914 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x556685461de0 msgr2=0x55668545f7a0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-05-27T07:36:40.516 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:40.520+0000 7f26cbfff700 1 --2- 172.21.15.52:0/618276914 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x556685461de0 0x55668545f7a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:36:40.516 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:40.520+0000 7f26cbfff700 1 -- 172.21.15.52:0/618276914 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5566852d8e40 msgr2=0x55668545ed20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:36:40.516 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:40.520+0000 7f26cbfff700 1 --2- 172.21.15.52:0/618276914 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5566852d8e40 0x55668545ed20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:36:40.519 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:40.520+0000 7f26cbfff700 1 -- 172.21.15.52:0/618276914 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5566852c58d0 con 0x556685457550 2024-05-27T07:36:40.520 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:40.522+0000 7f26cbfff700 1 --2- 172.21.15.52:0/618276914 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x556685457550 0x55668545f260 secure :-1 s=READY pgs=762 cs=0 l=1 rev1=1 crypto rx=0x7f26c4000f80 tx=0x7f26c40b1f00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e9c56c2a996ec5b6 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:36:40.520 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:40.523+0000 7f26abfff700 10 client.0 ms_handle_connect on v2:172.21.15.52:3300/0 2024-05-27T07:36:40.520 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:40.523+0000 7f26abfff700 1 -- 172.21.15.52:0/618276914 <== mon.0 v2:172.21.15.52:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f26c40a8d60 con 0x556685457550 2024-05-27T07:36:40.520 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:40.523+0000 7f26e16b9b80 1 -- 172.21.15.52:0/618276914 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f26bc011ce0 con 0x556685457550 2024-05-27T07:36:40.520 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:40.523+0000 7f26abfff700 1 -- 172.21.15.52:0/618276914 <== mon.0 v2:172.21.15.52:3300/0 2 ==== osd_map(127..127 src has 1..127) ==== 9555+0+0 (secure 0 0 0) 0x7f26c400cd80 con 0x556685457550 2024-05-27T07:36:40.520 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:40.523+0000 7f26d0a40700 1 --2- 172.21.15.52:0/618276914 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5566852d8e40 0x55668545ed20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-27T07:36:40.520 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:36:40 smithi052 ceph-mon[31955]: pgmap v360: 129 pgs: 14 active+clean+snaptrim, 39 active+clean+snaptrim_wait, 76 active+clean; 627 MiB data, 7.2 GiB used, 1.0 TiB / 1.0 TiB avail 2024-05-27T07:36:40.521 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:40.524+0000 7f26abfff700 1 -- 172.21.15.52:0/618276914 <== mon.0 v2:172.21.15.52:3300/0 3 ==== mon_get_version_reply(handle=1 version=108) ==== 24+0+0 (secure 0 0 0) 0x7f26c40a9950 con 0x556685457550 2024-05-27T07:36:40.521 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:40.524+0000 7f26e16b9b80 10 client.17271 fetch_fsmap learned FSMap version 108 2024-05-27T07:36:40.521 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:40.524+0000 7f26e16b9b80 1 -- 172.21.15.52:0/618276914 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({config=0+,fsmap=108,monmap=4+}) -- 0x55668532f760 con 0x556685457550 2024-05-27T07:36:40.523 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:40.526+0000 7f26abfff700 1 -- 172.21.15.52:0/618276914 <== mon.0 v2:172.21.15.52:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f26bc011ce0 con 0x556685457550 2024-05-27T07:36:40.523 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:40.527+0000 7f26abfff700 1 -- 172.21.15.52:0/618276914 <== mon.0 v2:172.21.15.52:3300/0 5 ==== fsmap(e 108) ==== 4203+0+0 (secure 0 0 0) 0x7f26c40ad280 con 0x556685457550 2024-05-27T07:36:40.527 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:40.531+0000 7f26e16b9b80 10 client.17271 fetch_fsmap finished waiting for FSMap version 108 2024-05-27T07:36:40.527 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:40.531+0000 7f26e16b9b80 10 client.17271 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.h 2024-05-27T07:36:40.527 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:40.531+0000 7f26e16b9b80 20 client.17271 populate_metadata read hostname 'smithi052' 2024-05-27T07:36:40.527 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:40.531+0000 7f26e16b9b80 1 --2- 172.21.15.52:0/618276914 >> [v2:172.21.15.105:6836/191393149,v1:172.21.15.105:6837/191393149] conn(0x556685483050 0x5566854854e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:36:40.527 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:40.531+0000 7f26e16b9b80 4 client.17271 mds_command: new command op to 24497 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-27T07:36:40.527 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:40.531+0000 7f26e16b9b80 1 -- 172.21.15.52:0/618276914 --> [v2:172.21.15.105:6836/191393149,v1:172.21.15.105:6837/191393149] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x556685387cc0 con 0x556685483050 2024-05-27T07:36:40.529 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:40.533+0000 7f26d0a40700 1 --2- 172.21.15.52:0/618276914 >> [v2:172.21.15.105:6836/191393149,v1:172.21.15.105:6837/191393149] conn(0x556685483050 0x5566854854e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:36:40.538 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:40.538+0000 7f26abfff700 10 client.17271 ms_handle_connect on v2:172.21.15.105:6836/191393149 2024-05-27T07:36:40.538 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:40.538+0000 7f26d0a40700 1 --2- 172.21.15.52:0/618276914 >> [v2:172.21.15.105:6836/191393149,v1:172.21.15.105:6837/191393149] conn(0x556685483050 0x5566854854e0 crc :-1 s=READY pgs=51 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=7b7c0eeead8264ff server_cookie=9bfe7844cf1c7e80 in_seq=0 out_seq=0 2024-05-27T07:36:40.538 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:40.539+0000 7f26abfff700 1 -- 172.21.15.52:0/618276914 <== mds.1 v2:172.21.15.105:6836/191393149 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24041 (crc 0 0 0) 0x556685387cc0 con 0x556685483050 2024-05-27T07:36:40.538 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:40.539+0000 7f26abfff700 10 client.17271 handle_command_reply: tid=0 2024-05-27T07:36:40.538 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:40.539+0000 7f26e16b9b80 1 client.17271 shutdown 2024-05-27T07:36:40.538 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:40.540+0000 7f26e16b9b80 1 -- 172.21.15.52:0/618276914 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x556685457550 msgr2=0x55668545f260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:36:40.538 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:40.540+0000 7f26e16b9b80 1 --2- 172.21.15.52:0/618276914 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x556685457550 0x55668545f260 secure :-1 s=READY pgs=762 cs=0 l=1 rev1=1 crypto rx=0x7f26c4000f80 tx=0x7f26c40b1f00 comp rx=0 tx=0).stop 2024-05-27T07:36:40.538 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:40.541+0000 7f26e16b9b80 1 -- 172.21.15.52:0/618276914 shutdown_connections 2024-05-27T07:36:40.538 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:40.541+0000 7f26e16b9b80 1 --2- 172.21.15.52:0/618276914 >> [v2:172.21.15.105:6836/191393149,v1:172.21.15.105:6837/191393149] conn(0x556685483050 0x5566854854e0 crc :-1 s=READY pgs=51 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:36:40.538 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:40.541+0000 7f26e16b9b80 1 --2- 172.21.15.52:0/618276914 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x556685461de0 0x55668545f7a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:36:40.538 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:40.541+0000 7f26e16b9b80 1 --2- 172.21.15.52:0/618276914 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5566852d8e40 0x55668545ed20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:36:40.538 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:40.541+0000 7f26e16b9b80 1 --2- 172.21.15.52:0/618276914 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x556685457550 0x55668545f260 unknown :-1 s=CLOSED pgs=762 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:36:40.539 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:40.541+0000 7f26e16b9b80 1 -- 172.21.15.52:0/618276914 >> 172.21.15.52:0/618276914 conn(0x5566852db770 msgr2=0x5566854515a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:36:40.539 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:40.541+0000 7f26abfff700 0 client.17271 ms_handle_reset on v2:172.21.15.105:6836/191393149 2024-05-27T07:36:40.542 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:40.544+0000 7f26e16b9b80 1 -- 172.21.15.52:0/618276914 shutdown_connections 2024-05-27T07:36:40.543 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:40.547+0000 7f26e16b9b80 1 -- 172.21.15.52:0/618276914 wait complete. 2024-05-27T07:36:40.543 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:40.547+0000 7f26e16b9b80 20 client.17271 trim_cache size 0 max 16384 2024-05-27T07:36:40.575 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:40.579+0000 7f26e16b9b80 1 Processor -- start 2024-05-27T07:36:40.575 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:40.579+0000 7f26e16b9b80 1 -- start start 2024-05-27T07:36:40.577 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:40.580+0000 7f26e16b9b80 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x5566852d8e40 0x5566854856d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:36:40.577 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:40.580+0000 7f26e16b9b80 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x556685457550 0x556685463200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:36:40.577 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:40.580+0000 7f26e16b9b80 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x556685461de0 0x556685463740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:36:40.577 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:40.580+0000 7f26e16b9b80 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x7f26cc1a6710 con 0x5566852d8e40 2024-05-27T07:36:40.577 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:40.580+0000 7f26e16b9b80 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f26cc074620 con 0x556685461de0 2024-05-27T07:36:40.577 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:40.580+0000 7f26d1241700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x556685461de0 0x556685463740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:36:40.578 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:40.580+0000 7f26d1241700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x556685461de0 0x556685463740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.52:50214/0 (socket says 172.21.15.52:50214) 2024-05-27T07:36:40.578 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:40.580+0000 7f26d1241700 1 -- 172.21.15.52:0/1322669685 learned_addr learned my addr 172.21.15.52:0/1322669685 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-05-27T07:36:40.578 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:40.581+0000 7f26e16b9b80 1 -- 172.21.15.52:0/1322669685 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f26cc1a6010 con 0x556685457550 2024-05-27T07:36:40.581 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:40.585+0000 7f26cbfff700 1 --2- 172.21.15.52:0/1322669685 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x556685457550 0x556685463200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:36:40.581 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:40.585+0000 7f26d1241700 1 -- 172.21.15.52:0/1322669685 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x556685457550 msgr2=0x556685463200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:36:40.581 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:40.585+0000 7f26d1241700 1 --2- 172.21.15.52:0/1322669685 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x556685457550 0x556685463200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:36:40.582 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:40.585+0000 7f26d1241700 1 -- 172.21.15.52:0/1322669685 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x5566852d8e40 msgr2=0x5566854856d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:36:40.582 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:40.585+0000 7f26d0a40700 1 --2- 172.21.15.52:0/1322669685 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x5566852d8e40 0x5566854856d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:36:40.582 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:40.585+0000 7f26d1241700 1 --2- 172.21.15.52:0/1322669685 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x5566852d8e40 0x5566854856d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:36:40.582 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:40.585+0000 7f26d1241700 1 -- 172.21.15.52:0/1322669685 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x556685460950 con 0x556685461de0 2024-05-27T07:36:40.582 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:40.585+0000 7f26d0a40700 1 --2- 172.21.15.52:0/1322669685 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x5566852d8e40 0x5566854856d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-27T07:36:40.584 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:40.589+0000 7f26d1241700 1 --2- 172.21.15.52:0/1322669685 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x556685461de0 0x556685463740 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 crypto rx=0x7f26b4003890 tx=0x7f26b401d000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c3902eb6cf983d57 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:36:40.585 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:40.589+0000 7f26c27fc700 1 -- 172.21.15.52:0/1322669685 <== mon.2 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f26b4005670 con 0x556685461de0 2024-05-27T07:36:40.585 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:40.589+0000 7f26c27fc700 1 -- 172.21.15.52:0/1322669685 <== mon.2 v2:172.21.15.105:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f26b4004710 con 0x556685461de0 2024-05-27T07:36:40.587 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:40.591+0000 7f26e16b9b80 1 -- 172.21.15.52:0/1322669685 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x556685461de0 msgr2=0x556685463740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:36:40.587 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:40.591+0000 7f26e16b9b80 1 --2- 172.21.15.52:0/1322669685 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x556685461de0 0x556685463740 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 crypto rx=0x7f26b4003890 tx=0x7f26b401d000 comp rx=0 tx=0).stop 2024-05-27T07:36:40.589 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:40.593+0000 7f26e16b9b80 1 -- 172.21.15.52:0/1322669685 shutdown_connections 2024-05-27T07:36:40.589 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:40.593+0000 7f26e16b9b80 1 --2- 172.21.15.52:0/1322669685 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x556685461de0 0x556685463740 unknown :-1 s=CLOSED pgs=275 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:36:40.589 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:40.593+0000 7f26e16b9b80 1 --2- 172.21.15.52:0/1322669685 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x5566852d8e40 0x5566854856d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:36:40.589 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:40.593+0000 7f26e16b9b80 1 --2- 172.21.15.52:0/1322669685 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x556685457550 0x556685463200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:36:40.589 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:40.593+0000 7f26e16b9b80 1 -- 172.21.15.52:0/1322669685 >> 172.21.15.52:0/1322669685 conn(0x5566852db770 msgr2=0x55668547dea0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:36:40.594 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:40.599+0000 7f26e16b9b80 1 -- 172.21.15.52:0/1322669685 shutdown_connections 2024-05-27T07:36:40.596 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:40.599+0000 7f26e16b9b80 1 -- 172.21.15.52:0/1322669685 wait complete. 2024-05-27T07:36:40.596 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:40.599+0000 7f26e16b9b80 1 -- start start 2024-05-27T07:36:40.596 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:40.599+0000 7f26e16b9b80 1 Processor -- start 2024-05-27T07:36:40.597 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:40.600+0000 7f26e16b9b80 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x5566852d8e40 0x55668547b4a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:36:40.597 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:40.600+0000 7f26e16b9b80 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x556685457550 0x556685479830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:36:40.597 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:40.600+0000 7f26e16b9b80 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x556685461de0 0x556685479d70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:36:40.597 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:40.600+0000 7f26e16b9b80 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x7f26cc075420 con 0x5566852d8e40 2024-05-27T07:36:40.597 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:40.600+0000 7f26e16b9b80 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f26cc074d20 con 0x556685457550 2024-05-27T07:36:40.598 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:40.600+0000 7f26e16b9b80 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x556685480740 con 0x556685461de0 2024-05-27T07:36:40.598 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:40.600+0000 7f26e16b9b80 10 client.0 _ll_register_callbacks cb 0x55668530c5b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T07:36:40.599 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:40.603+0000 7f26d1241700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x556685461de0 0x556685479d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:36:40.600 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:40.603+0000 7f26d1241700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x556685461de0 0x556685479d70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.173:3300/0 says I am v2:172.21.15.52:35464/0 (socket says 172.21.15.52:35464) 2024-05-27T07:36:40.600 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:40.603+0000 7f26d1241700 1 -- 172.21.15.52:0/1910553191 learned_addr learned my addr 172.21.15.52:0/1910553191 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-05-27T07:36:40.600 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:40.604+0000 7f26d0a40700 1 --2- 172.21.15.52:0/1910553191 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x5566852d8e40 0x55668547b4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:36:40.600 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:40.604+0000 7f26cbfff700 1 --2- 172.21.15.52:0/1910553191 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x556685457550 0x556685479830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:36:40.602 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:40.606+0000 7f26d0a40700 1 -- 172.21.15.52:0/1910553191 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x556685461de0 msgr2=0x556685479d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:36:40.602 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:40.606+0000 7f26d0a40700 1 --2- 172.21.15.52:0/1910553191 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x556685461de0 0x556685479d70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:36:40.602 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:40.606+0000 7f26d0a40700 1 -- 172.21.15.52:0/1910553191 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x556685457550 msgr2=0x556685479830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:36:40.602 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:40.606+0000 7f26d0a40700 1 --2- 172.21.15.52:0/1910553191 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x556685457550 0x556685479830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:36:40.603 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:40.606+0000 7f26d0a40700 1 -- 172.21.15.52:0/1910553191 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55668547a2b0 con 0x5566852d8e40 2024-05-27T07:36:40.605 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:40.609+0000 7f26d0a40700 1 --2- 172.21.15.52:0/1910553191 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x5566852d8e40 0x55668547b4a0 secure :-1 s=READY pgs=763 cs=0 l=1 rev1=1 crypto rx=0x7f26bc009750 tx=0x7f26bc010980 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fc5813ca839a2d03 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:36:40.605 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:40.610+0000 7f26abfff700 10 client.0 ms_handle_connect on v2:172.21.15.52:3300/0 2024-05-27T07:36:40.609 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:40.614+0000 7f26abfff700 1 -- 172.21.15.52:0/1910553191 <== mon.0 v2:172.21.15.52:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f26bc015040 con 0x5566852d8e40 2024-05-27T07:36:40.609 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:40.614+0000 7f26abfff700 1 -- 172.21.15.52:0/1910553191 <== mon.0 v2:172.21.15.52:3300/0 2 ==== osd_map(127..127 src has 1..127) ==== 9555+0+0 (secure 0 0 0) 0x7f26bc011630 con 0x5566852d8e40 2024-05-27T07:36:40.615 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:40.620+0000 7f26e16b9b80 1 -- 172.21.15.52:0/1910553191 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f26b4004710 con 0x5566852d8e40 2024-05-27T07:36:40.618 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:40.621+0000 7f26abfff700 1 -- 172.21.15.52:0/1910553191 <== mon.0 v2:172.21.15.52:3300/0 3 ==== mon_get_version_reply(handle=1 version=108) ==== 24+0+0 (secure 0 0 0) 0x7f26bc035050 con 0x5566852d8e40 2024-05-27T07:36:40.618 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:40.622+0000 7f26e16b9b80 10 client.17274 fetch_fsmap learned FSMap version 108 2024-05-27T07:36:40.618 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:40.622+0000 7f26e16b9b80 1 -- 172.21.15.52:0/1910553191 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({config=0+,fsmap=108,monmap=4+}) -- 0x5566854ec7a0 con 0x5566852d8e40 2024-05-27T07:36:40.623 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:40.625+0000 7f26abfff700 1 -- 172.21.15.52:0/1910553191 <== mon.0 v2:172.21.15.52:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f26b4004710 con 0x5566852d8e40 2024-05-27T07:36:40.624 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:40.626+0000 7f26abfff700 1 -- 172.21.15.52:0/1910553191 <== mon.0 v2:172.21.15.52:3300/0 5 ==== fsmap(e 108) ==== 4203+0+0 (secure 0 0 0) 0x7f26bc023040 con 0x5566852d8e40 2024-05-27T07:36:40.624 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:40.626+0000 7f26e16b9b80 10 client.17274 fetch_fsmap finished waiting for FSMap version 108 2024-05-27T07:36:40.624 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:40.626+0000 7f26e16b9b80 10 client.17274 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.h 2024-05-27T07:36:40.624 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:40.626+0000 7f26e16b9b80 20 client.17274 populate_metadata read hostname 'smithi052' 2024-05-27T07:36:40.624 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:40.626+0000 7f26e16b9b80 1 --2- 172.21.15.52:0/1910553191 >> [v2:172.21.15.105:6836/191393149,v1:172.21.15.105:6837/191393149] conn(0x5566854ece30 0x5566854ef250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:36:40.624 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:40.626+0000 7f26e16b9b80 4 client.17274 mds_command: new command op to 24497 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-05-27T07:36:40.624 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:40.626+0000 7f26e16b9b80 1 -- 172.21.15.52:0/1910553191 --> [v2:172.21.15.105:6836/191393149,v1:172.21.15.105:6837/191393149] -- command(tid 0: {"prefix": "damage ls"}) -- 0x5566853814a0 con 0x5566854ece30 2024-05-27T07:36:40.624 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:40.626+0000 7f26cbfff700 1 --2- 172.21.15.52:0/1910553191 >> [v2:172.21.15.105:6836/191393149,v1:172.21.15.105:6837/191393149] conn(0x5566854ece30 0x5566854ef250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:36:40.624 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:40.627+0000 7f26abfff700 10 client.17274 ms_handle_connect on v2:172.21.15.105:6836/191393149 2024-05-27T07:36:40.624 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:40.627+0000 7f26cbfff700 1 --2- 172.21.15.52:0/1910553191 >> [v2:172.21.15.105:6836/191393149,v1:172.21.15.105:6837/191393149] conn(0x5566854ece30 0x5566854ef250 crc :-1 s=READY pgs=52 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=3cad19fb9e563862 server_cookie=d536de1b3587b1a5 in_seq=0 out_seq=0 2024-05-27T07:36:40.624 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:40.627+0000 7f26abfff700 1 -- 172.21.15.52:0/1910553191 <== mds.1 v2:172.21.15.105:6836/191393149 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x5566853814a0 con 0x5566854ece30 2024-05-27T07:36:40.624 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:40.627+0000 7f26abfff700 10 client.17274 handle_command_reply: tid=0 2024-05-27T07:36:40.624 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:40.627+0000 7f26e16b9b80 1 client.17274 shutdown 2024-05-27T07:36:40.626 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:40.629+0000 7f26e16b9b80 1 -- 172.21.15.52:0/1910553191 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x5566852d8e40 msgr2=0x55668547b4a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:36:40.627 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:40.629+0000 7f26e16b9b80 1 --2- 172.21.15.52:0/1910553191 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x5566852d8e40 0x55668547b4a0 secure :-1 s=READY pgs=763 cs=0 l=1 rev1=1 crypto rx=0x7f26bc009750 tx=0x7f26bc010980 comp rx=0 tx=0).stop 2024-05-27T07:36:40.627 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:40.630+0000 7f26e16b9b80 1 -- 172.21.15.52:0/1910553191 shutdown_connections 2024-05-27T07:36:40.627 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:40.630+0000 7f26e16b9b80 1 --2- 172.21.15.52:0/1910553191 >> [v2:172.21.15.105:6836/191393149,v1:172.21.15.105:6837/191393149] conn(0x5566854ece30 0x5566854ef250 crc :-1 s=READY pgs=52 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:36:40.627 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:40.630+0000 7f26e16b9b80 1 --2- 172.21.15.52:0/1910553191 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x556685461de0 0x556685479d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:36:40.627 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:40.630+0000 7f26e16b9b80 1 --2- 172.21.15.52:0/1910553191 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x5566852d8e40 0x55668547b4a0 secure :-1 s=CLOSED pgs=763 cs=0 l=1 rev1=1 crypto rx=0x7f26bc009750 tx=0x7f26bc010980 comp rx=0 tx=0).stop 2024-05-27T07:36:40.627 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:40.630+0000 7f26e16b9b80 1 --2- 172.21.15.52:0/1910553191 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x556685457550 0x556685479830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:36:40.627 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:40.630+0000 7f26e16b9b80 1 -- 172.21.15.52:0/1910553191 >> 172.21.15.52:0/1910553191 conn(0x5566852db770 msgr2=0x55668545d9c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:36:40.627 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:40.630+0000 7f26abfff700 0 client.17274 ms_handle_reset on v2:172.21.15.105:6836/191393149 2024-05-27T07:36:40.627 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:40.630+0000 7f26e16b9b80 1 -- 172.21.15.52:0/1910553191 shutdown_connections 2024-05-27T07:36:40.627 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:40.630+0000 7f26e16b9b80 1 -- 172.21.15.52:0/1910553191 wait complete. 2024-05-27T07:36:40.627 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:40.630+0000 7f26e16b9b80 20 client.17274 trim_cache size 0 max 16384 2024-05-27T07:36:40.628 INFO:teuthology.orchestra.run.smithi052.stdout:[] 2024-05-27T07:36:40.628 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:40.632+0000 7f26d1a42700 1 -- 172.21.15.52:0/3436560025 >> [v2:172.21.15.52:6800/1822318384,v1:172.21.15.52:6801/1822318384] conn(0x7f26b80b2af0 msgr2=0x7f26b80b5000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:36:40.628 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:40.632+0000 7f26d1a42700 1 --2- 172.21.15.52:0/3436560025 >> [v2:172.21.15.52:6800/1822318384,v1:172.21.15.52:6801/1822318384] conn(0x7f26b80b2af0 0x7f26b80b5000 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7f26b4009750 tx=0x7f26b4005ab0 comp rx=0 tx=0).stop 2024-05-27T07:36:40.629 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:40.632+0000 7f26d1a42700 1 -- 172.21.15.52:0/3436560025 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f26cc115140 msgr2=0x7f26cc10dc90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:36:40.629 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:40.632+0000 7f26d1a42700 1 --2- 172.21.15.52:0/3436560025 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f26cc115140 0x7f26cc10dc90 secure :-1 s=READY pgs=760 cs=0 l=1 rev1=1 crypto rx=0x7f26c400b9b0 tx=0x7f26c400bd70 comp rx=0 tx=0).stop 2024-05-27T07:36:40.631 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:40.634+0000 7f26d1a42700 1 -- 172.21.15.52:0/3436560025 shutdown_connections 2024-05-27T07:36:40.631 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:40.635+0000 7f26d1a42700 1 --2- 172.21.15.52:0/3436560025 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f26cc10e270 0x7f26cc10e720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:36:40.631 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:40.635+0000 7f26d1a42700 1 --2- 172.21.15.52:0/3436560025 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f26cc075b50 0x7f26cc114b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:36:40.631 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:40.635+0000 7f26d1a42700 1 --2- 172.21.15.52:0/3436560025 >> [v2:172.21.15.52:6800/1822318384,v1:172.21.15.52:6801/1822318384] conn(0x7f26b80b2af0 0x7f26b80b5000 unknown :-1 s=CLOSED pgs=198 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:36:40.631 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:40.635+0000 7f26d1a42700 1 --2- 172.21.15.52:0/3436560025 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f26cc115140 0x7f26cc10dc90 unknown :-1 s=CLOSED pgs=760 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:36:40.631 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:40.635+0000 7f26d1a42700 1 -- 172.21.15.52:0/3436560025 >> 172.21.15.52:0/3436560025 conn(0x7f26cc07acb0 msgr2=0x7f26cc100f50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:36:40.635 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:40.639+0000 7f26d1a42700 1 -- 172.21.15.52:0/3436560025 shutdown_connections 2024-05-27T07:36:40.635 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:40.639+0000 7f26d1a42700 1 -- 172.21.15.52:0/3436560025 wait complete. 2024-05-27T07:36:40.696 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:36:40 smithi105 ceph-mon[35977]: pgmap v360: 129 pgs: 14 active+clean+snaptrim, 39 active+clean+snaptrim_wait, 76 active+clean; 627 MiB data, 7.2 GiB used, 1.0 TiB / 1.0 TiB avail 2024-05-27T07:36:40.858 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:36:40 smithi173 ceph-mon[37000]: pgmap v360: 129 pgs: 14 active+clean+snaptrim, 39 active+clean+snaptrim_wait, 76 active+clean; 627 MiB data, 7.2 GiB used, 1.0 TiB / 1.0 TiB avail 2024-05-27T07:36:41.513 DEBUG:teuthology.orchestra.run.smithi052:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:38f9d900c83c63799fbdbe61acc9a11b0d3554a6 shell --fsid 73df2636-1bf8-11ef-bc9b-c7b262605968 -- ceph tell mds.1:0 damage ls 2024-05-27T07:36:42.001 INFO:teuthology.orchestra.run.smithi052.stderr:Inferring config /var/lib/ceph/73df2636-1bf8-11ef-bc9b-c7b262605968/mon.a/config 2024-05-27T07:36:42.471 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:36:42 smithi052 ceph-mon[31955]: pgmap v361: 129 pgs: 13 active+clean+snaptrim, 33 active+clean+snaptrim_wait, 83 active+clean; 626 MiB data, 7.2 GiB used, 1.0 TiB / 1.0 TiB avail 2024-05-27T07:36:42.696 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:36:42 smithi105 ceph-mon[35977]: pgmap v361: 129 pgs: 13 active+clean+snaptrim, 33 active+clean+snaptrim_wait, 83 active+clean; 626 MiB data, 7.2 GiB used, 1.0 TiB / 1.0 TiB avail 2024-05-27T07:36:42.858 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:36:42 smithi173 ceph-mon[37000]: pgmap v361: 129 pgs: 13 active+clean+snaptrim, 33 active+clean+snaptrim_wait, 83 active+clean; 626 MiB data, 7.2 GiB used, 1.0 TiB / 1.0 TiB avail 2024-05-27T07:36:43.485 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:36:43 smithi052 ceph-mon[31955]: from='mgr.15333 172.21.15.52:0/416389082' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-27T07:36:43.697 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:36:43 smithi105 ceph-mon[35977]: from='mgr.15333 172.21.15.52:0/416389082' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-27T07:36:43.858 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:36:43 smithi173 ceph-mon[37000]: from='mgr.15333 172.21.15.52:0/416389082' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-27T07:36:44.696 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:36:44 smithi105 ceph-mon[35977]: pgmap v362: 129 pgs: 12 active+clean+snaptrim, 33 active+clean+snaptrim_wait, 84 active+clean; 625 MiB data, 7.2 GiB used, 1.0 TiB / 1.0 TiB avail 2024-05-27T07:36:44.735 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:36:44 smithi052 ceph-mon[31955]: pgmap v362: 129 pgs: 12 active+clean+snaptrim, 33 active+clean+snaptrim_wait, 84 active+clean; 625 MiB data, 7.2 GiB used, 1.0 TiB / 1.0 TiB avail 2024-05-27T07:36:44.858 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:36:44 smithi173 ceph-mon[37000]: pgmap v362: 129 pgs: 12 active+clean+snaptrim, 33 active+clean+snaptrim_wait, 84 active+clean; 625 MiB data, 7.2 GiB used, 1.0 TiB / 1.0 TiB avail 2024-05-27T07:36:45.869 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:45.871+0000 7fd099af5700 1 -- 172.21.15.52:0/644190196 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fd09410a7b0 msgr2=0x7fd09410abc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:36:45.869 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:45.871+0000 7fd099af5700 1 --2- 172.21.15.52:0/644190196 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fd09410a7b0 0x7fd09410abc0 secure :-1 s=READY pgs=764 cs=0 l=1 rev1=1 crypto rx=0x7fd08c00b0a0 tx=0x7fd08c00b3b0 comp rx=0 tx=0).stop 2024-05-27T07:36:45.869 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:45.874+0000 7fd099af5700 1 -- 172.21.15.52:0/644190196 shutdown_connections 2024-05-27T07:36:45.870 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:45.874+0000 7fd099af5700 1 --2- 172.21.15.52:0/644190196 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fd09411d920 0x7fd09411fd40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:36:45.870 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:45.874+0000 7fd099af5700 1 --2- 172.21.15.52:0/644190196 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fd09410a7b0 0x7fd09410abc0 unknown :-1 s=CLOSED pgs=764 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:36:45.870 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:45.874+0000 7fd099af5700 1 --2- 172.21.15.52:0/644190196 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fd09410b100 0x7fd09411d3e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:36:45.870 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:45.874+0000 7fd099af5700 1 -- 172.21.15.52:0/644190196 >> 172.21.15.52:0/644190196 conn(0x7fd09406f280 msgr2=0x7fd09406f6c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:36:45.871 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:45.876+0000 7fd099af5700 1 -- 172.21.15.52:0/644190196 shutdown_connections 2024-05-27T07:36:45.873 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:45.876+0000 7fd099af5700 1 -- 172.21.15.52:0/644190196 wait complete. 2024-05-27T07:36:45.873 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:45.877+0000 7fd099af5700 1 Processor -- start 2024-05-27T07:36:45.874 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:45.877+0000 7fd099af5700 1 -- start start 2024-05-27T07:36:45.874 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:45.877+0000 7fd099af5700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fd09410a7b0 0x7fd0941af620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:36:45.874 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:45.877+0000 7fd099af5700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fd09410b100 0x7fd0941afb80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:36:45.874 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:45.877+0000 7fd099af5700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fd09411d920 0x7fd0941a9860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:36:45.874 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:45.877+0000 7fd099af5700 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x7fd094123180 con 0x7fd09410b100 2024-05-27T07:36:45.874 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:45.877+0000 7fd099af5700 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7fd094123000 con 0x7fd09410a7b0 2024-05-27T07:36:45.874 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:45.877+0000 7fd099af5700 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7fd094123300 con 0x7fd09411d920 2024-05-27T07:36:45.874 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:45.878+0000 7fd0992f4700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fd09411d920 0x7fd0941a9860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:36:45.875 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:45.878+0000 7fd0992f4700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fd09411d920 0x7fd0941a9860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.173:3300/0 says I am v2:172.21.15.52:52020/0 (socket says 172.21.15.52:52020) 2024-05-27T07:36:45.875 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:45.878+0000 7fd0992f4700 1 -- 172.21.15.52:0/2201143029 learned_addr learned my addr 172.21.15.52:0/2201143029 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-05-27T07:36:45.875 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:45.879+0000 7fd0992f4700 1 -- 172.21.15.52:0/2201143029 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fd09410a7b0 msgr2=0x7fd0941af620 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-05-27T07:36:45.875 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:45.879+0000 7fd0992f4700 1 --2- 172.21.15.52:0/2201143029 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fd09410a7b0 0x7fd0941af620 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:36:45.875 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:45.879+0000 7fd0992f4700 1 -- 172.21.15.52:0/2201143029 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fd09410b100 msgr2=0x7fd0941afb80 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-05-27T07:36:45.875 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:45.879+0000 7fd0992f4700 1 --2- 172.21.15.52:0/2201143029 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fd09410b100 0x7fd0941afb80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:36:45.875 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:45.879+0000 7fd0992f4700 1 -- 172.21.15.52:0/2201143029 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd0941aa020 con 0x7fd09411d920 2024-05-27T07:36:45.875 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:45.879+0000 7fd0992f4700 1 --2- 172.21.15.52:0/2201143029 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fd09411d920 0x7fd0941a9860 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7fd08800b4a0 tx=0x7fd08800b860 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:36:45.878 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:45.880+0000 7fd091ffb700 1 -- 172.21.15.52:0/2201143029 <== mon.1 v2:172.21.15.173:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd088015020 con 0x7fd09411d920 2024-05-27T07:36:45.878 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:45.880+0000 7fd099af5700 1 -- 172.21.15.52:0/2201143029 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd0941aa310 con 0x7fd09411d920 2024-05-27T07:36:45.878 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:45.880+0000 7fd099af5700 1 -- 172.21.15.52:0/2201143029 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd094112bc0 con 0x7fd09411d920 2024-05-27T07:36:45.878 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:45.880+0000 7fd091ffb700 1 -- 172.21.15.52:0/2201143029 <== mon.1 v2:172.21.15.173:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fd088011ce0 con 0x7fd09411d920 2024-05-27T07:36:45.878 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:45.880+0000 7fd091ffb700 1 -- 172.21.15.52:0/2201143029 <== mon.1 v2:172.21.15.173:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd088010d40 con 0x7fd09411d920 2024-05-27T07:36:45.881 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:45.885+0000 7fd091ffb700 1 -- 172.21.15.52:0/2201143029 <== mon.1 v2:172.21.15.173:3300/0 4 ==== mgrmap(e 26) ==== 149783+0+0 (secure 0 0 0) 0x7fd088018020 con 0x7fd09411d920 2024-05-27T07:36:45.882 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:45.886+0000 7fd0a976cb80 1 Processor -- start 2024-05-27T07:36:45.882 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:45.886+0000 7fd0a976cb80 1 -- start start 2024-05-27T07:36:45.884 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:45.887+0000 7fd0a976cb80 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x558623452e40 0x5586235d6880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:36:45.884 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:45.887+0000 7fd0a976cb80 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x5586235d6dc0 0x5586235d0a40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:36:45.884 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:45.887+0000 7fd0a976cb80 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5586235d0f80 0x5586235d1390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:36:45.884 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:45.887+0000 7fd0a976cb80 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x5586234ffc90 con 0x5586235d6dc0 2024-05-27T07:36:45.884 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:45.887+0000 7fd0a976cb80 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x5586234ef830 con 0x5586235d0f80 2024-05-27T07:36:45.885 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:45.887+0000 7fd0a976cb80 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x5586234f3870 con 0x558623452e40 2024-05-27T07:36:45.885 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:45.887+0000 7fd091ffb700 1 --2- 172.21.15.52:0/2201143029 >> [v2:172.21.15.52:6800/1822318384,v1:172.21.15.52:6801/1822318384] conn(0x7fd07c0b2b30 0x7fd07c0b5040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:36:45.885 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:45.888+0000 7fd098af3700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x5586235d6dc0 0x5586235d0a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:36:45.885 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:45.888+0000 7fd098af3700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x5586235d6dc0 0x5586235d0a40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.52:46168/0 (socket says 172.21.15.52:46168) 2024-05-27T07:36:45.885 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:45.888+0000 7fd098af3700 1 -- 172.21.15.52:0/3559401711 learned_addr learned my addr 172.21.15.52:0/3559401711 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-05-27T07:36:45.885 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:45.888+0000 7fd091ffb700 1 -- 172.21.15.52:0/2201143029 <== mon.1 v2:172.21.15.173:3300/0 5 ==== osd_map(127..127 src has 1..127) ==== 9555+0+0 (secure 0 0 0) 0x7fd0880e3080 con 0x7fd09411d920 2024-05-27T07:36:45.887 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:45.888+0000 7fd098af3700 1 -- 172.21.15.52:0/3559401711 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x558623452e40 msgr2=0x5586235d6880 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-05-27T07:36:45.888 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:45.888+0000 7fd098af3700 1 --2- 172.21.15.52:0/3559401711 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x558623452e40 0x5586235d6880 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:36:45.888 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:45.888+0000 7fd098af3700 1 -- 172.21.15.52:0/3559401711 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5586235d0f80 msgr2=0x5586235d1390 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-05-27T07:36:45.888 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:45.888+0000 7fd098af3700 1 --2- 172.21.15.52:0/3559401711 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5586235d0f80 0x5586235d1390 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:36:45.888 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:45.889+0000 7fd098af3700 1 -- 172.21.15.52:0/3559401711 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55862343f8d0 con 0x5586235d6dc0 2024-05-27T07:36:45.888 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:45.889+0000 7fd098af3700 1 --2- 172.21.15.52:0/3559401711 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x5586235d6dc0 0x5586235d0a40 secure :-1 s=READY pgs=765 cs=0 l=1 rev1=1 crypto rx=0x7fd08c00b070 tx=0x7fd08c00ce40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=534945c791e49e32 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:36:45.888 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:45.891+0000 7fd07affd700 1 -- 172.21.15.52:0/3559401711 <== mon.0 v2:172.21.15.52:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd08c01f070 con 0x5586235d6dc0 2024-05-27T07:36:45.888 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:45.891+0000 7fd07affd700 1 -- 172.21.15.52:0/3559401711 <== mon.0 v2:172.21.15.52:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fd08c0123f0 con 0x5586235d6dc0 2024-05-27T07:36:45.888 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:45.892+0000 7fd0992f4700 1 --2- 172.21.15.52:0/2201143029 >> [v2:172.21.15.52:6800/1822318384,v1:172.21.15.52:6801/1822318384] conn(0x7fd07c0b2b30 0x7fd07c0b5040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:36:45.889 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:45.893+0000 7fd0a976cb80 1 -- 172.21.15.52:0/3559401711 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x5586235d6dc0 msgr2=0x5586235d0a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:36:45.889 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:45.893+0000 7fd0a976cb80 1 --2- 172.21.15.52:0/3559401711 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x5586235d6dc0 0x5586235d0a40 secure :-1 s=READY pgs=765 cs=0 l=1 rev1=1 crypto rx=0x7fd08c00b070 tx=0x7fd08c00ce40 comp rx=0 tx=0).stop 2024-05-27T07:36:45.890 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:45.894+0000 7fd0a976cb80 1 -- 172.21.15.52:0/3559401711 shutdown_connections 2024-05-27T07:36:45.890 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:45.894+0000 7fd0a976cb80 1 --2- 172.21.15.52:0/3559401711 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5586235d0f80 0x5586235d1390 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:36:45.890 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:45.894+0000 7fd0a976cb80 1 --2- 172.21.15.52:0/3559401711 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x558623452e40 0x5586235d6880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:36:45.890 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:45.894+0000 7fd0a976cb80 1 --2- 172.21.15.52:0/3559401711 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x5586235d6dc0 0x5586235d0a40 unknown :-1 s=CLOSED pgs=765 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:36:45.890 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:45.894+0000 7fd0a976cb80 1 -- 172.21.15.52:0/3559401711 >> 172.21.15.52:0/3559401711 conn(0x5586234adaf0 msgr2=0x5586235cb740 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:36:45.895 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:45.898+0000 7fd0a976cb80 1 -- 172.21.15.52:0/3559401711 shutdown_connections 2024-05-27T07:36:45.895 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:45.898+0000 7fd0992f4700 1 --2- 172.21.15.52:0/2201143029 >> [v2:172.21.15.52:6800/1822318384,v1:172.21.15.52:6801/1822318384] conn(0x7fd07c0b2b30 0x7fd07c0b5040 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7fd088010f70 tx=0x7fd088004770 comp rx=0 tx=0).ready entity=mgr.15333 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:36:45.895 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:45.898+0000 7fd0a976cb80 1 -- 172.21.15.52:0/3559401711 wait complete. 2024-05-27T07:36:45.896 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:45.899+0000 7fd0a976cb80 1 -- start start 2024-05-27T07:36:45.896 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:45.899+0000 7fd0a976cb80 1 Processor -- start 2024-05-27T07:36:45.896 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:45.900+0000 7fd0a976cb80 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5586235d6dc0 0x5586235da610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:36:45.896 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:45.900+0000 7fd0a976cb80 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5586235dab50 0x5586235daf60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:36:45.896 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:45.900+0000 7fd0a976cb80 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x5586235e4c60 0x5586235e7080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:36:45.896 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:45.900+0000 7fd0a976cb80 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x5586233f5fa0 con 0x5586235e4c60 2024-05-27T07:36:45.896 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:45.900+0000 7fd0a976cb80 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x5586234917c0 con 0x5586235dab50 2024-05-27T07:36:45.896 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:45.900+0000 7fd0a976cb80 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x5586235dd880 con 0x5586235d6dc0 2024-05-27T07:36:45.897 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:45.900+0000 7fd0a976cb80 10 client.0 _ll_register_callbacks cb 0x55862348a920 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T07:36:45.897 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:45.900+0000 7fd093fff700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5586235d6dc0 0x5586235da610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:36:45.897 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:45.900+0000 7fd093fff700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5586235d6dc0 0x5586235da610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.173:3300/0 says I am v2:172.21.15.52:52034/0 (socket says 172.21.15.52:52034) 2024-05-27T07:36:45.897 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:45.900+0000 7fd093fff700 1 -- 172.21.15.52:0/2947357512 learned_addr learned my addr 172.21.15.52:0/2947357512 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-05-27T07:36:45.897 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:45.900+0000 7fd093fff700 1 --2- 172.21.15.52:0/2947357512 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x5586235e4c60 0x5586235e7080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:36:45.898 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:45.901+0000 7fd093fff700 1 -- 172.21.15.52:0/2947357512 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5586235dab50 msgr2=0x5586235daf60 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-05-27T07:36:45.898 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:45.901+0000 7fd093fff700 1 --2- 172.21.15.52:0/2947357512 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5586235dab50 0x5586235daf60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:36:45.898 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:45.901+0000 7fd093fff700 1 -- 172.21.15.52:0/2947357512 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x5586235e4c60 msgr2=0x5586235e7080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:36:45.899 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:45.901+0000 7fd093fff700 1 --2- 172.21.15.52:0/2947357512 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x5586235e4c60 0x5586235e7080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:36:45.899 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:45.901+0000 7fd093fff700 1 -- 172.21.15.52:0/2947357512 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55862348e690 con 0x5586235d6dc0 2024-05-27T07:36:45.899 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:45.902+0000 7fd093fff700 1 --2- 172.21.15.52:0/2947357512 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5586235d6dc0 0x5586235da610 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7fd09411cc70 tx=0x7fd0840047c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=4be8919cf3bec2d8 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:36:45.899 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:45.902+0000 7fd073fff700 10 client.0 ms_handle_connect on v2:172.21.15.173:3300/0 2024-05-27T07:36:45.899 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:45.902+0000 7fd073fff700 1 -- 172.21.15.52:0/2947357512 <== mon.1 v2:172.21.15.173:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd084004020 con 0x5586235d6dc0 2024-05-27T07:36:45.899 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:45.902+0000 7fd073fff700 1 -- 172.21.15.52:0/2947357512 <== mon.1 v2:172.21.15.173:3300/0 2 ==== osd_map(127..127 src has 1..127) ==== 9555+0+0 (secure 0 0 0) 0x7fd0840041c0 con 0x5586235d6dc0 2024-05-27T07:36:45.899 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:45.903+0000 7fd0a976cb80 1 -- 172.21.15.52:0/2947357512 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x55862359c9f0 con 0x5586235d6dc0 2024-05-27T07:36:45.899 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:45.903+0000 7fd073fff700 1 -- 172.21.15.52:0/2947357512 <== mon.1 v2:172.21.15.173:3300/0 3 ==== mon_get_version_reply(handle=1 version=108) ==== 24+0+0 (secure 0 0 0) 0x7fd0840043c0 con 0x5586235d6dc0 2024-05-27T07:36:45.902 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:45.906+0000 7fd0a976cb80 10 client.26776 fetch_fsmap learned FSMap version 108 2024-05-27T07:36:45.902 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:45.906+0000 7fd0a976cb80 1 -- 172.21.15.52:0/2947357512 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({config=0+,fsmap=108,monmap=4+}) -- 0x5586235e8de0 con 0x5586235d6dc0 2024-05-27T07:36:45.903 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:45.907+0000 7fd073fff700 1 -- 172.21.15.52:0/2947357512 <== mon.1 v2:172.21.15.173:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x55862359c9f0 con 0x5586235d6dc0 2024-05-27T07:36:45.903 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:45.907+0000 7fd073fff700 1 -- 172.21.15.52:0/2947357512 <== mon.1 v2:172.21.15.173:3300/0 5 ==== fsmap(e 108) ==== 4203+0+0 (secure 0 0 0) 0x7fd084025020 con 0x5586235d6dc0 2024-05-27T07:36:45.903 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:45.907+0000 7fd0a976cb80 10 client.26776 fetch_fsmap finished waiting for FSMap version 108 2024-05-27T07:36:45.903 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:45.907+0000 7fd0a976cb80 10 client.26776 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-05-27T07:36:45.903 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:45.907+0000 7fd0a976cb80 20 client.26776 populate_metadata read hostname 'smithi052' 2024-05-27T07:36:45.903 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:45.907+0000 7fd0a976cb80 1 --2- 172.21.15.52:0/2947357512 >> [v2:172.21.15.105:6838/500348151,v1:172.21.15.105:6839/500348151] conn(0x5586235eb880 0x5586235edd50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:36:45.903 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:45.907+0000 7fd0a976cb80 4 client.26776 mds_command: new command op to 24527 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-27T07:36:45.903 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:45.907+0000 7fd0a976cb80 1 -- 172.21.15.52:0/2947357512 --> [v2:172.21.15.105:6838/500348151,v1:172.21.15.105:6839/500348151] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x558623501be0 con 0x5586235eb880 2024-05-27T07:36:45.908 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:45.912+0000 7fd098af3700 1 --2- 172.21.15.52:0/2947357512 >> [v2:172.21.15.105:6838/500348151,v1:172.21.15.105:6839/500348151] conn(0x5586235eb880 0x5586235edd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:36:45.909 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:45.913+0000 7fd098af3700 1 --2- 172.21.15.52:0/2947357512 >> [v2:172.21.15.105:6838/500348151,v1:172.21.15.105:6839/500348151] conn(0x5586235eb880 0x5586235edd50 crc :-1 s=READY pgs=179 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=67d538f81fcb1270 server_cookie=51c0fb22624c99ee in_seq=0 out_seq=0 2024-05-27T07:36:45.909 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:45.913+0000 7fd073fff700 10 client.26776 ms_handle_connect on v2:172.21.15.105:6838/500348151 2024-05-27T07:36:45.911 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:45.915+0000 7fd073fff700 1 -- 172.21.15.52:0/2947357512 <== mds.0 v2:172.21.15.105:6838/500348151 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24041 (crc 0 0 0) 0x558623501be0 con 0x5586235eb880 2024-05-27T07:36:45.911 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:45.915+0000 7fd073fff700 10 client.26776 handle_command_reply: tid=0 2024-05-27T07:36:45.911 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:45.915+0000 7fd0a976cb80 1 client.26776 shutdown 2024-05-27T07:36:45.912 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:45.915+0000 7fd0a976cb80 1 -- 172.21.15.52:0/2947357512 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5586235d6dc0 msgr2=0x5586235da610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:36:45.912 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:45.915+0000 7fd0a976cb80 1 --2- 172.21.15.52:0/2947357512 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5586235d6dc0 0x5586235da610 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7fd09411cc70 tx=0x7fd0840047c0 comp rx=0 tx=0).stop 2024-05-27T07:36:45.912 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:45.916+0000 7fd0a976cb80 1 -- 172.21.15.52:0/2947357512 shutdown_connections 2024-05-27T07:36:45.912 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:45.916+0000 7fd0a976cb80 1 --2- 172.21.15.52:0/2947357512 >> [v2:172.21.15.105:6838/500348151,v1:172.21.15.105:6839/500348151] conn(0x5586235eb880 0x5586235edd50 crc :-1 s=READY pgs=179 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:36:45.912 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:45.916+0000 7fd0a976cb80 1 --2- 172.21.15.52:0/2947357512 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x5586235e4c60 0x5586235e7080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:36:45.912 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:45.916+0000 7fd0a976cb80 1 --2- 172.21.15.52:0/2947357512 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5586235d6dc0 0x5586235da610 unknown :-1 s=CLOSED pgs=230 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:36:45.912 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:45.916+0000 7fd0a976cb80 1 --2- 172.21.15.52:0/2947357512 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5586235dab50 0x5586235daf60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:36:45.912 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:45.916+0000 7fd0a976cb80 1 -- 172.21.15.52:0/2947357512 >> 172.21.15.52:0/2947357512 conn(0x5586234adaf0 msgr2=0x5586235cfa30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:36:45.912 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:45.916+0000 7fd073fff700 0 client.26776 ms_handle_reset on v2:172.21.15.105:6838/500348151 2024-05-27T07:36:45.912 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:45.916+0000 7fd0a976cb80 1 -- 172.21.15.52:0/2947357512 shutdown_connections 2024-05-27T07:36:45.913 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:45.916+0000 7fd0a976cb80 1 -- 172.21.15.52:0/2947357512 wait complete. 2024-05-27T07:36:45.913 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:45.916+0000 7fd0a976cb80 20 client.26776 trim_cache size 0 max 16384 2024-05-27T07:36:45.943 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:45.946+0000 7fd0a976cb80 1 Processor -- start 2024-05-27T07:36:45.944 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:45.946+0000 7fd0a976cb80 1 -- start start 2024-05-27T07:36:45.944 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:45.947+0000 7fd0a976cb80 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x5586235d6dc0 0x5586235d0a40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:36:45.944 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:45.947+0000 7fd0a976cb80 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5586235dab50 0x5586235cb8d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:36:45.944 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:45.947+0000 7fd0a976cb80 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5586235e4c60 0x5586235c5e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:36:45.944 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:45.947+0000 7fd098af3700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5586235dab50 0x5586235cb8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:36:45.944 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:45.947+0000 7fd098af3700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5586235dab50 0x5586235cb8d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.52:56108/0 (socket says 172.21.15.52:56108) 2024-05-27T07:36:45.944 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:45.947+0000 7fd093fff700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5586235e4c60 0x5586235c5e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:36:45.944 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:45.947+0000 7fd093fff700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5586235e4c60 0x5586235c5e70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.173:3300/0 says I am v2:172.21.15.52:52042/0 (socket says 172.21.15.52:52042) 2024-05-27T07:36:45.944 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:45.947+0000 7fd098af3700 1 -- 172.21.15.52:0/2161596963 learned_addr learned my addr 172.21.15.52:0/2161596963 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-05-27T07:36:45.945 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:45.948+0000 7fd0a976cb80 1 -- 172.21.15.52:0/2161596963 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x7fd08000f530 con 0x5586235d6dc0 2024-05-27T07:36:45.945 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:45.948+0000 7fd0a976cb80 1 -- 172.21.15.52:0/2161596963 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7fd08000d960 con 0x5586235dab50 2024-05-27T07:36:45.945 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:45.948+0000 7fd0a976cb80 1 -- 172.21.15.52:0/2161596963 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7fd08000ee30 con 0x5586235e4c60 2024-05-27T07:36:45.946 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:45.950+0000 7fd093fff700 1 --2- 172.21.15.52:0/2161596963 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x5586235d6dc0 0x5586235d0a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:36:45.947 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:45.950+0000 7fd093fff700 1 -- 172.21.15.52:0/2161596963 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5586235dab50 msgr2=0x5586235cb8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:36:45.947 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:45.950+0000 7fd093fff700 1 --2- 172.21.15.52:0/2161596963 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5586235dab50 0x5586235cb8d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:36:45.947 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:45.950+0000 7fd093fff700 1 -- 172.21.15.52:0/2161596963 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x5586235d6dc0 msgr2=0x5586235d0a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:36:45.947 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:45.950+0000 7fd093fff700 1 --2- 172.21.15.52:0/2161596963 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x5586235d6dc0 0x5586235d0a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:36:45.947 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:45.950+0000 7fd093fff700 1 -- 172.21.15.52:0/2161596963 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5586235dd3c0 con 0x5586235e4c60 2024-05-27T07:36:45.947 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:45.951+0000 7fd093fff700 1 --2- 172.21.15.52:0/2161596963 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5586235e4c60 0x5586235c5e70 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7fd08400fcd0 tx=0x7fd084000bc0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=780c1ab4e52c1ab4 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:36:45.951 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:45.956+0000 7fd07a7fc700 1 -- 172.21.15.52:0/2161596963 <== mon.1 v2:172.21.15.173:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd08400e890 con 0x5586235e4c60 2024-05-27T07:36:45.952 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:45.956+0000 7fd07a7fc700 1 -- 172.21.15.52:0/2161596963 <== mon.1 v2:172.21.15.173:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fd08400ee00 con 0x5586235e4c60 2024-05-27T07:36:45.952 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:45.956+0000 7fd07a7fc700 1 -- 172.21.15.52:0/2161596963 <== mon.1 v2:172.21.15.173:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd08400ca50 con 0x5586235e4c60 2024-05-27T07:36:45.953 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:45.957+0000 7fd0a976cb80 1 -- 172.21.15.52:0/2161596963 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5586235e4c60 msgr2=0x5586235c5e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:36:45.953 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:45.957+0000 7fd0a976cb80 1 --2- 172.21.15.52:0/2161596963 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5586235e4c60 0x5586235c5e70 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7fd08400fcd0 tx=0x7fd084000bc0 comp rx=0 tx=0).stop 2024-05-27T07:36:45.954 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:45.959+0000 7fd0a976cb80 1 -- 172.21.15.52:0/2161596963 shutdown_connections 2024-05-27T07:36:45.954 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:45.959+0000 7fd0a976cb80 1 --2- 172.21.15.52:0/2161596963 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5586235e4c60 0x5586235c5e70 unknown :-1 s=CLOSED pgs=231 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:36:45.955 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:45.959+0000 7fd0a976cb80 1 --2- 172.21.15.52:0/2161596963 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x5586235d6dc0 0x5586235d0a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:36:45.955 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:45.959+0000 7fd0a976cb80 1 --2- 172.21.15.52:0/2161596963 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5586235dab50 0x5586235cb8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:36:45.955 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:45.959+0000 7fd0a976cb80 1 -- 172.21.15.52:0/2161596963 >> 172.21.15.52:0/2161596963 conn(0x5586234adaf0 msgr2=0x5586235cb1c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:36:45.959 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:45.964+0000 7fd0a976cb80 1 -- 172.21.15.52:0/2161596963 shutdown_connections 2024-05-27T07:36:45.960 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:45.964+0000 7fd0a976cb80 1 -- 172.21.15.52:0/2161596963 wait complete. 2024-05-27T07:36:45.960 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:45.964+0000 7fd0a976cb80 1 -- start start 2024-05-27T07:36:45.961 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:45.964+0000 7fd0a976cb80 1 Processor -- start 2024-05-27T07:36:45.962 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:45.965+0000 7fd0a976cb80 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x5586235d6dc0 0x5586235e46d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:36:45.962 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:45.965+0000 7fd0a976cb80 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5586235dab50 0x5586235f3320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:36:45.962 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:45.965+0000 7fd0a976cb80 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5586235e4c60 0x5586235f39e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:36:45.963 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:45.965+0000 7fd0a976cb80 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x7fd08000e760 con 0x5586235d6dc0 2024-05-27T07:36:45.964 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:45.965+0000 7fd0a976cb80 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7fd08000e060 con 0x5586235dab50 2024-05-27T07:36:45.964 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:45.965+0000 7fd0a976cb80 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x5586235d8b80 con 0x5586235e4c60 2024-05-27T07:36:45.964 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:45.965+0000 7fd0a976cb80 10 client.0 _ll_register_callbacks cb 0x55862348a920 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T07:36:45.966 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:45.970+0000 7fd098af3700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5586235dab50 0x5586235f3320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:36:45.967 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:45.970+0000 7fd098af3700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5586235dab50 0x5586235f3320 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.52:56120/0 (socket says 172.21.15.52:56120) 2024-05-27T07:36:45.967 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:45.970+0000 7fd098af3700 1 -- 172.21.15.52:0/2372853065 learned_addr learned my addr 172.21.15.52:0/2372853065 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-05-27T07:36:45.967 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:45.970+0000 7fd093fff700 1 --2- 172.21.15.52:0/2372853065 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5586235e4c60 0x5586235f39e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:36:45.968 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:45.972+0000 7fd098af3700 1 -- 172.21.15.52:0/2372853065 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5586235e4c60 msgr2=0x5586235f39e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:36:45.968 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:45.972+0000 7fd098af3700 1 --2- 172.21.15.52:0/2372853065 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5586235e4c60 0x5586235f39e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:36:45.968 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:45.972+0000 7fd098af3700 1 -- 172.21.15.52:0/2372853065 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x5586235d6dc0 msgr2=0x5586235e46d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:36:45.968 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:45.972+0000 7fd098af3700 1 --2- 172.21.15.52:0/2372853065 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x5586235d6dc0 0x5586235e46d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:36:45.968 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:45.972+0000 7fd098af3700 1 -- 172.21.15.52:0/2372853065 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5586235cd080 con 0x5586235dab50 2024-05-27T07:36:45.969 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:45.973+0000 7fd098af3700 1 --2- 172.21.15.52:0/2372853065 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5586235dab50 0x5586235f3320 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 crypto rx=0x7fd08c00b070 tx=0x7fd08c000f40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d593ee726fcaf02a server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:36:45.969 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:45.973+0000 7fd073fff700 10 client.0 ms_handle_connect on v2:172.21.15.105:3300/0 2024-05-27T07:36:45.969 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:45.973+0000 7fd073fff700 1 -- 172.21.15.52:0/2372853065 <== mon.2 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd08c031060 con 0x5586235dab50 2024-05-27T07:36:45.969 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:45.973+0000 7fd0a976cb80 1 -- 172.21.15.52:0/2372853065 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fd08400ee00 con 0x5586235dab50 2024-05-27T07:36:45.970 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:45.973+0000 7fd073fff700 1 -- 172.21.15.52:0/2372853065 <== mon.2 v2:172.21.15.105:3300/0 2 ==== osd_map(127..127 src has 1..127) ==== 9555+0+0 (secure 0 0 0) 0x7fd08c0090d0 con 0x5586235dab50 2024-05-27T07:36:45.970 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:45.974+0000 7fd073fff700 1 -- 172.21.15.52:0/2372853065 <== mon.2 v2:172.21.15.105:3300/0 3 ==== mon_get_version_reply(handle=1 version=108) ==== 24+0+0 (secure 0 0 0) 0x7fd08c038050 con 0x5586235dab50 2024-05-27T07:36:45.970 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:45.974+0000 7fd0a976cb80 10 client.26960 fetch_fsmap learned FSMap version 108 2024-05-27T07:36:45.970 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:45.974+0000 7fd0a976cb80 1 -- 172.21.15.52:0/2372853065 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,fsmap=108,monmap=4+}) -- 0x5586236553b0 con 0x5586235dab50 2024-05-27T07:36:45.974 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:45.978+0000 7fd073fff700 1 -- 172.21.15.52:0/2372853065 <== mon.2 v2:172.21.15.105:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fd08400ee00 con 0x5586235dab50 2024-05-27T07:36:45.974 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:45.978+0000 7fd073fff700 1 -- 172.21.15.52:0/2372853065 <== mon.2 v2:172.21.15.105:3300/0 5 ==== fsmap(e 108) ==== 4203+0+0 (secure 0 0 0) 0x7fd08c011070 con 0x5586235dab50 2024-05-27T07:36:45.977 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:45.979+0000 7fd0a976cb80 10 client.26960 fetch_fsmap finished waiting for FSMap version 108 2024-05-27T07:36:45.977 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:45.979+0000 7fd0a976cb80 10 client.26960 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-05-27T07:36:45.977 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:45.979+0000 7fd0a976cb80 20 client.26960 populate_metadata read hostname 'smithi052' 2024-05-27T07:36:45.977 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:45.980+0000 7fd0a976cb80 1 --2- 172.21.15.52:0/2372853065 >> [v2:172.21.15.105:6838/500348151,v1:172.21.15.105:6839/500348151] conn(0x5586236559e0 0x558623657e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:36:45.977 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:45.980+0000 7fd0a976cb80 4 client.26960 mds_command: new command op to 24527 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-05-27T07:36:45.977 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:45.980+0000 7fd0a976cb80 1 -- 172.21.15.52:0/2372853065 --> [v2:172.21.15.105:6838/500348151,v1:172.21.15.105:6839/500348151] -- command(tid 0: {"prefix": "damage ls"}) -- 0x5586234fb3c0 con 0x5586236559e0 2024-05-27T07:36:45.981 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:45.985+0000 7fd093fff700 1 --2- 172.21.15.52:0/2372853065 >> [v2:172.21.15.105:6838/500348151,v1:172.21.15.105:6839/500348151] conn(0x5586236559e0 0x558623657e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:36:45.982 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:45.986+0000 7fd093fff700 1 --2- 172.21.15.52:0/2372853065 >> [v2:172.21.15.105:6838/500348151,v1:172.21.15.105:6839/500348151] conn(0x5586236559e0 0x558623657e00 crc :-1 s=READY pgs=180 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=a817c6fac1b46d85 server_cookie=ea2ef49238bd2a8d in_seq=0 out_seq=0 2024-05-27T07:36:45.982 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:45.986+0000 7fd073fff700 10 client.26960 ms_handle_connect on v2:172.21.15.105:6838/500348151 2024-05-27T07:36:45.982 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:45.986+0000 7fd073fff700 1 -- 172.21.15.52:0/2372853065 <== mds.0 v2:172.21.15.105:6838/500348151 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x5586234fb3c0 con 0x5586236559e0 2024-05-27T07:36:45.983 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:45.987+0000 7fd073fff700 10 client.26960 handle_command_reply: tid=0 2024-05-27T07:36:45.983 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:45.987+0000 7fd0a976cb80 1 client.26960 shutdown 2024-05-27T07:36:45.989 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:45.993+0000 7fd0a976cb80 1 -- 172.21.15.52:0/2372853065 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5586235dab50 msgr2=0x5586235f3320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:36:45.989 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:45.993+0000 7fd0a976cb80 1 --2- 172.21.15.52:0/2372853065 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5586235dab50 0x5586235f3320 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 crypto rx=0x7fd08c00b070 tx=0x7fd08c000f40 comp rx=0 tx=0).stop 2024-05-27T07:36:45.989 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:45.993+0000 7fd0a976cb80 1 -- 172.21.15.52:0/2372853065 shutdown_connections 2024-05-27T07:36:45.989 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:45.993+0000 7fd0a976cb80 1 --2- 172.21.15.52:0/2372853065 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5586235e4c60 0x5586235f39e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:36:45.989 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:45.993+0000 7fd0a976cb80 1 --2- 172.21.15.52:0/2372853065 >> [v2:172.21.15.105:6838/500348151,v1:172.21.15.105:6839/500348151] conn(0x5586236559e0 0x558623657e00 crc :-1 s=READY pgs=180 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:36:45.989 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:45.993+0000 7fd0a976cb80 1 --2- 172.21.15.52:0/2372853065 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x5586235d6dc0 0x5586235e46d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:36:45.989 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:45.993+0000 7fd0a976cb80 1 --2- 172.21.15.52:0/2372853065 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5586235dab50 0x5586235f3320 secure :-1 s=CLOSED pgs=276 cs=0 l=1 rev1=1 crypto rx=0x7fd08c00b070 tx=0x7fd08c000f40 comp rx=0 tx=0).stop 2024-05-27T07:36:45.989 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:45.993+0000 7fd0a976cb80 1 -- 172.21.15.52:0/2372853065 >> 172.21.15.52:0/2372853065 conn(0x5586234adaf0 msgr2=0x5586235d8f00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:36:45.989 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:45.993+0000 7fd073fff700 0 client.26960 ms_handle_reset on v2:172.21.15.105:6838/500348151 2024-05-27T07:36:45.992 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:45.997+0000 7fd0a976cb80 1 -- 172.21.15.52:0/2372853065 shutdown_connections 2024-05-27T07:36:45.993 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:45.997+0000 7fd0a976cb80 1 -- 172.21.15.52:0/2372853065 wait complete. 2024-05-27T07:36:45.994 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:45.997+0000 7fd0a976cb80 20 client.26960 trim_cache size 0 max 16384 2024-05-27T07:36:45.995 INFO:teuthology.orchestra.run.smithi052.stdout:[] 2024-05-27T07:36:45.997 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:46.000+0000 7fd099af5700 1 -- 172.21.15.52:0/2201143029 >> [v2:172.21.15.52:6800/1822318384,v1:172.21.15.52:6801/1822318384] conn(0x7fd07c0b2b30 msgr2=0x7fd07c0b5040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:36:45.997 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:46.000+0000 7fd099af5700 1 --2- 172.21.15.52:0/2201143029 >> [v2:172.21.15.52:6800/1822318384,v1:172.21.15.52:6801/1822318384] conn(0x7fd07c0b2b30 0x7fd07c0b5040 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7fd088010f70 tx=0x7fd088004770 comp rx=0 tx=0).stop 2024-05-27T07:36:45.997 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:46.000+0000 7fd099af5700 1 -- 172.21.15.52:0/2201143029 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fd09411d920 msgr2=0x7fd0941a9860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:36:45.997 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:46.000+0000 7fd099af5700 1 --2- 172.21.15.52:0/2201143029 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fd09411d920 0x7fd0941a9860 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7fd08800b4a0 tx=0x7fd08800b860 comp rx=0 tx=0).stop 2024-05-27T07:36:45.997 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:46.000+0000 7fd099af5700 1 -- 172.21.15.52:0/2201143029 shutdown_connections 2024-05-27T07:36:45.997 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:46.000+0000 7fd099af5700 1 --2- 172.21.15.52:0/2201143029 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fd09411d920 0x7fd0941a9860 unknown :-1 s=CLOSED pgs=229 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:36:45.997 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:46.000+0000 7fd099af5700 1 --2- 172.21.15.52:0/2201143029 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fd09410a7b0 0x7fd0941af620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:36:45.997 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:46.000+0000 7fd099af5700 1 --2- 172.21.15.52:0/2201143029 >> [v2:172.21.15.52:6800/1822318384,v1:172.21.15.52:6801/1822318384] conn(0x7fd07c0b2b30 0x7fd07c0b5040 unknown :-1 s=CLOSED pgs=199 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:36:45.997 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:46.000+0000 7fd099af5700 1 --2- 172.21.15.52:0/2201143029 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fd09410b100 0x7fd0941afb80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:36:45.997 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:46.000+0000 7fd099af5700 1 -- 172.21.15.52:0/2201143029 >> 172.21.15.52:0/2201143029 conn(0x7fd09406f280 msgr2=0x7fd09411b3d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:36:45.998 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:46.001+0000 7fd099af5700 1 -- 172.21.15.52:0/2201143029 shutdown_connections 2024-05-27T07:36:45.998 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:46.001+0000 7fd099af5700 1 -- 172.21.15.52:0/2201143029 wait complete. 2024-05-27T07:36:46.660 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:36:46 smithi173 ceph-mon[37000]: pgmap v363: 129 pgs: 11 active+clean+snaptrim, 30 active+clean+snaptrim_wait, 88 active+clean; 624 MiB data, 7.3 GiB used, 1.0 TiB / 1.0 TiB avail 2024-05-27T07:36:46.660 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:36:46 smithi173 ceph-mon[37000]: from='mgr.15333 172.21.15.52:0/416389082' entity='mgr.x' 2024-05-27T07:36:46.661 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:36:46 smithi173 ceph-mon[37000]: from='mgr.15333 172.21.15.52:0/416389082' entity='mgr.x' 2024-05-27T07:36:46.661 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:36:46 smithi173 ceph-mon[37000]: from='mgr.15333 172.21.15.52:0/416389082' entity='mgr.x' 2024-05-27T07:36:46.661 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:36:46 smithi173 ceph-mon[37000]: from='mgr.15333 172.21.15.52:0/416389082' entity='mgr.x' 2024-05-27T07:36:46.696 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:36:46 smithi105 ceph-mon[35977]: pgmap v363: 129 pgs: 11 active+clean+snaptrim, 30 active+clean+snaptrim_wait, 88 active+clean; 624 MiB data, 7.3 GiB used, 1.0 TiB / 1.0 TiB avail 2024-05-27T07:36:46.696 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:36:46 smithi105 ceph-mon[35977]: from='mgr.15333 172.21.15.52:0/416389082' entity='mgr.x' 2024-05-27T07:36:46.696 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:36:46 smithi105 ceph-mon[35977]: from='mgr.15333 172.21.15.52:0/416389082' entity='mgr.x' 2024-05-27T07:36:46.696 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:36:46 smithi105 ceph-mon[35977]: from='mgr.15333 172.21.15.52:0/416389082' entity='mgr.x' 2024-05-27T07:36:46.696 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:36:46 smithi105 ceph-mon[35977]: from='mgr.15333 172.21.15.52:0/416389082' entity='mgr.x' 2024-05-27T07:36:46.735 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:36:46 smithi052 ceph-mon[31955]: pgmap v363: 129 pgs: 11 active+clean+snaptrim, 30 active+clean+snaptrim_wait, 88 active+clean; 624 MiB data, 7.3 GiB used, 1.0 TiB / 1.0 TiB avail 2024-05-27T07:36:46.735 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:36:46 smithi052 ceph-mon[31955]: from='mgr.15333 172.21.15.52:0/416389082' entity='mgr.x' 2024-05-27T07:36:46.735 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:36:46 smithi052 ceph-mon[31955]: from='mgr.15333 172.21.15.52:0/416389082' entity='mgr.x' 2024-05-27T07:36:46.736 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:36:46 smithi052 ceph-mon[31955]: from='mgr.15333 172.21.15.52:0/416389082' entity='mgr.x' 2024-05-27T07:36:46.736 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:36:46 smithi052 ceph-mon[31955]: from='mgr.15333 172.21.15.52:0/416389082' entity='mgr.x' 2024-05-27T07:36:47.502 ERROR:tasks.fwd_scrub.fs.[cephfs]:exception: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_ceph-c_38f9d900c83c63799fbdbe61acc9a11b0d3554a6/qa/tasks/fwd_scrub.py", line 35, in _run self.do_scrub() File "/home/teuthworker/src/git.ceph.com_ceph-c_38f9d900c83c63799fbdbe61acc9a11b0d3554a6/qa/tasks/fwd_scrub.py", line 49, in do_scrub self.sleep_unless_stopped(self.sleep_between_iterations) File "/home/teuthworker/src/git.ceph.com_ceph-c_38f9d900c83c63799fbdbe61acc9a11b0d3554a6/qa/tasks/thrasher.py", line 47, in sleep_unless_stopped raise self.Stopped() tasks.thrasher.ThrasherGreenlet.Stopped 2024-05-27T07:36:47.503 INFO:tasks.fwd_scrub:done joining 2024-05-27T07:36:47.503 DEBUG:teuthology.run_tasks:Unwinding manager check-counter 2024-05-27T07:36:47.515 DEBUG:teuthology.orchestra.run.smithi052:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-05-27T07:36:47.696 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:36:47 smithi105 ceph-mon[35977]: from='mgr.15333 172.21.15.52:0/416389082' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-05-27T07:36:47.696 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:36:47 smithi105 ceph-mon[35977]: from='mgr.15333 172.21.15.52:0/416389082' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-05-27T07:36:47.696 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:36:47 smithi105 ceph-mon[35977]: from='mgr.15333 172.21.15.52:0/416389082' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-05-27T07:36:47.697 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:36:47 smithi105 ceph-mon[35977]: from='mgr.15333 172.21.15.52:0/416389082' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-05-27T07:36:47.697 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:36:47 smithi105 ceph-mon[35977]: from='mgr.15333 172.21.15.52:0/416389082' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-05-27T07:36:47.697 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:36:47 smithi105 ceph-mon[35977]: from='mgr.15333 172.21.15.52:0/416389082' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-05-27T07:36:47.697 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:36:47 smithi105 ceph-mon[35977]: from='mgr.15333 172.21.15.52:0/416389082' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-05-27T07:36:47.697 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:36:47 smithi105 ceph-mon[35977]: from='mgr.15333 172.21.15.52:0/416389082' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-05-27T07:36:47.724 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:47.726+0000 7f0145c69640 1 -- 172.21.15.52:0/1349092147 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f014010fa00 msgr2=0x7f014010fe10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:36:47.724 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:47.726+0000 7f0145c69640 1 --2- 172.21.15.52:0/1349092147 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f014010fa00 0x7f014010fe10 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 crypto rx=0x7f0130009900 tx=0x7f013002a410 comp rx=0 tx=0).stop 2024-05-27T07:36:47.724 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:47.726+0000 7f0145c69640 1 -- 172.21.15.52:0/1349092147 shutdown_connections 2024-05-27T07:36:47.724 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:47.726+0000 7f0145c69640 1 --2- 172.21.15.52:0/1349092147 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f014010a200 0x7f014010a690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:36:47.724 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:47.726+0000 7f0145c69640 1 --2- 172.21.15.52:0/1349092147 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f01401103e0 0x7f0140109cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:36:47.724 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:47.726+0000 7f0145c69640 1 --2- 172.21.15.52:0/1349092147 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f014010fa00 0x7f014010fe10 unknown :-1 s=CLOSED pgs=277 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:36:47.724 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:47.726+0000 7f0145c69640 1 -- 172.21.15.52:0/1349092147 >> 172.21.15.52:0/1349092147 conn(0x7f014006e6e0 msgr2=0x7f014006eb60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:36:47.724 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:47.727+0000 7f0145c69640 1 -- 172.21.15.52:0/1349092147 shutdown_connections 2024-05-27T07:36:47.726 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:47.727+0000 7f0145c69640 1 -- 172.21.15.52:0/1349092147 wait complete. 2024-05-27T07:36:47.726 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:47.728+0000 7f0145c69640 1 Processor -- start 2024-05-27T07:36:47.726 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:47.728+0000 7f0145c69640 1 -- start start 2024-05-27T07:36:47.726 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:47.728+0000 7f0145c69640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f014010a200 0x7f01401a9130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:36:47.726 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:47.728+0000 7f0145c69640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f01401103e0 0x7f01401a9690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:36:47.726 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:47.728+0000 7f0145c69640 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f01401ad890 0x7f01401adca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:36:47.726 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:47.729+0000 7f0145c69640 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x7f0140122ab0 con 0x7f01401ad890 2024-05-27T07:36:47.727 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:47.729+0000 7f0145c69640 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f0140122930 con 0x7f014010a200 2024-05-27T07:36:47.727 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:47.729+0000 7f0145c69640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f0140122c30 con 0x7f01401103e0 2024-05-27T07:36:47.727 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:47.729+0000 7f013f7fe640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f014010a200 0x7f01401a9130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:36:47.727 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:47.729+0000 7f013f7fe640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f014010a200 0x7f01401a9130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.52:56144/0 (socket says 172.21.15.52:56144) 2024-05-27T07:36:47.727 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:47.729+0000 7f013f7fe640 1 -- 172.21.15.52:0/296155525 learned_addr learned my addr 172.21.15.52:0/296155525 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-05-27T07:36:47.727 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:47.729+0000 7f013ffff640 1 --2- 172.21.15.52:0/296155525 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f01401ad890 0x7f01401adca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:36:47.727 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:47.730+0000 7f013f7fe640 1 -- 172.21.15.52:0/296155525 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f01401103e0 msgr2=0x7f01401a9690 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-05-27T07:36:47.727 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:47.730+0000 7f013f7fe640 1 --2- 172.21.15.52:0/296155525 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f01401103e0 0x7f01401a9690 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:36:47.727 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:47.730+0000 7f013f7fe640 1 -- 172.21.15.52:0/296155525 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f01401ad890 msgr2=0x7f01401adca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:36:47.727 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:47.730+0000 7f013f7fe640 1 --2- 172.21.15.52:0/296155525 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f01401ad890 0x7f01401adca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:36:47.727 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:47.730+0000 7f013f7fe640 1 -- 172.21.15.52:0/296155525 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f01401ae470 con 0x7f014010a200 2024-05-27T07:36:47.728 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:47.730+0000 7f013f7fe640 1 --2- 172.21.15.52:0/296155525 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f014010a200 0x7f01401a9130 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 crypto rx=0x7f0130006010 tx=0x7f0130004530 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:36:47.728 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:47.731+0000 7f013d7fa640 1 -- 172.21.15.52:0/296155525 <== mon.2 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0130040070 con 0x7f014010a200 2024-05-27T07:36:47.728 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:47.731+0000 7f0145c69640 1 -- 172.21.15.52:0/296155525 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f014007af20 con 0x7f014010a200 2024-05-27T07:36:47.728 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:47.731+0000 7f0145c69640 1 -- 172.21.15.52:0/296155525 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f014007b410 con 0x7f014010a200 2024-05-27T07:36:47.728 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:47.731+0000 7f013d7fa640 1 -- 172.21.15.52:0/296155525 <== mon.2 v2:172.21.15.105:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f013003a3f0 con 0x7f014010a200 2024-05-27T07:36:47.729 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:47.733+0000 7f013d7fa640 1 -- 172.21.15.52:0/296155525 <== mon.2 v2:172.21.15.105:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0130035040 con 0x7f014010a200 2024-05-27T07:36:47.732 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:47.734+0000 7f0145c69640 1 -- 172.21.15.52:0/296155525 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0124005290 con 0x7f014010a200 2024-05-27T07:36:47.735 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:36:47 smithi052 ceph-mon[31955]: from='mgr.15333 172.21.15.52:0/416389082' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-05-27T07:36:47.735 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:36:47 smithi052 ceph-mon[31955]: from='mgr.15333 172.21.15.52:0/416389082' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-05-27T07:36:47.736 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:36:47 smithi052 ceph-mon[31955]: from='mgr.15333 172.21.15.52:0/416389082' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-05-27T07:36:47.736 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:36:47 smithi052 ceph-mon[31955]: from='mgr.15333 172.21.15.52:0/416389082' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-05-27T07:36:47.736 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:36:47 smithi052 ceph-mon[31955]: from='mgr.15333 172.21.15.52:0/416389082' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-05-27T07:36:47.736 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:36:47 smithi052 ceph-mon[31955]: from='mgr.15333 172.21.15.52:0/416389082' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-05-27T07:36:47.736 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:36:47 smithi052 ceph-mon[31955]: from='mgr.15333 172.21.15.52:0/416389082' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-05-27T07:36:47.736 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:36:47 smithi052 ceph-mon[31955]: from='mgr.15333 172.21.15.52:0/416389082' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-05-27T07:36:47.736 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:47.740+0000 7f013d7fa640 1 -- 172.21.15.52:0/296155525 <== mon.2 v2:172.21.15.105:3300/0 4 ==== mgrmap(e 26) ==== 149783+0+0 (secure 0 0 0) 0x7f0130043070 con 0x7f014010a200 2024-05-27T07:36:47.737 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:47.741+0000 7f013d7fa640 1 --2- 172.21.15.52:0/296155525 >> [v2:172.21.15.52:6800/1822318384,v1:172.21.15.52:6801/1822318384] conn(0x7f01200b2a40 0x7f01200b4f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:36:47.737 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:47.742+0000 7f013d7fa640 1 -- 172.21.15.52:0/296155525 <== mon.2 v2:172.21.15.105:3300/0 5 ==== osd_map(127..127 src has 1..127) ==== 9555+0+0 (secure 0 0 0) 0x7f0130102ad0 con 0x7f014010a200 2024-05-27T07:36:47.742 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:47.746+0000 7f013d7fa640 1 -- 172.21.15.52:0/296155525 <== mon.2 v2:172.21.15.105:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191963 (secure 0 0 0) 0x7f01300ca040 con 0x7f014010a200 2024-05-27T07:36:47.743 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:47.748+0000 7f0137fff640 1 --2- 172.21.15.52:0/296155525 >> [v2:172.21.15.52:6800/1822318384,v1:172.21.15.52:6801/1822318384] conn(0x7f01200b2a40 0x7f01200b4f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:36:47.748 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:47.752+0000 7f0137fff640 1 --2- 172.21.15.52:0/296155525 >> [v2:172.21.15.52:6800/1822318384,v1:172.21.15.52:6801/1822318384] conn(0x7f01200b2a40 0x7f01200b4f50 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7f014010e4f0 tx=0x7f0128005b00 comp rx=0 tx=0).ready entity=mgr.15333 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:36:47.858 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:36:47 smithi173 ceph-mon[37000]: from='mgr.15333 172.21.15.52:0/416389082' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-05-27T07:36:47.858 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:36:47 smithi173 ceph-mon[37000]: from='mgr.15333 172.21.15.52:0/416389082' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-05-27T07:36:47.858 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:36:47 smithi173 ceph-mon[37000]: from='mgr.15333 172.21.15.52:0/416389082' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-05-27T07:36:47.858 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:36:47 smithi173 ceph-mon[37000]: from='mgr.15333 172.21.15.52:0/416389082' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-05-27T07:36:47.858 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:36:47 smithi173 ceph-mon[37000]: from='mgr.15333 172.21.15.52:0/416389082' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-05-27T07:36:47.858 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:36:47 smithi173 ceph-mon[37000]: from='mgr.15333 172.21.15.52:0/416389082' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-05-27T07:36:47.859 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:36:47 smithi173 ceph-mon[37000]: from='mgr.15333 172.21.15.52:0/416389082' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-05-27T07:36:47.859 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:36:47 smithi173 ceph-mon[37000]: from='mgr.15333 172.21.15.52:0/416389082' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-05-27T07:36:48.134 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:48.138+0000 7f0145c69640 1 -- 172.21.15.52:0/296155525 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f0124005850 con 0x7f014010a200 2024-05-27T07:36:48.137 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:48.141+0000 7f013d7fa640 1 -- 172.21.15.52:0/296155525 <== mon.2 v2:172.21.15.105:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v127) ==== 74+0+25399 (secure 0 0 0) 0x7f01300c9940 con 0x7f014010a200 2024-05-27T07:36:48.140 INFO:teuthology.orchestra.run.smithi052.stdout: 2024-05-27T07:36:48.140 INFO:teuthology.orchestra.run.smithi052.stdout:{"epoch":127,"fsid":"73df2636-1bf8-11ef-bc9b-c7b262605968","created":"2024-05-27T07:13:34.738032+0000","modified":"2024-05-27T07:36:06.113142+0000","last_up_change":"2024-05-27T07:20:45.777597+0000","last_in_change":"2024-05-27T07:20:25.314823+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-05-27T07:16:57.644034+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-05-27T07:21:43.917748+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"79","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-05-27T07:21:47.341876+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"127","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":15,"snap_epoch":127,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-05-27T07:21:57.707611+0000","flags":8197,"flags_names":"hashpspool,ec_overwrites,selfmanaged_snaps","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"127","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":15,"snap_epoch":127,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"347b8a89-9dbe-4848-83b0-fd0e9374f33d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":82,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.52:6802","nonce":1626508899},{"type":"v1","addr":"172.21.15.52:6803","nonce":1626508899}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.52:6804","nonce":1626508899},{"type":"v1","addr":"172.21.15.52:6805","nonce":1626508899}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.52:6808","nonce":1626508899},{"type":"v1","addr":"172.21.15.52:6809","nonce":1626508899}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.52:6806","nonce":1626508899},{"type":"v1","addr":"172.21.15.52:6807","nonce":1626508899}]},"public_addr":"172.21.15.52:6803/1626508899","cluster_addr":"172.21.15.52:6805/1626508899","heartbeat_back_addr":"172.21.15.52:6809/1626508899","heartbeat_front_addr":"172.21.15.52:6807/1626508899","state":["exists","up"]},{"osd":1,"uuid":"7a3623d4-e087-40cb-9336-d4a5c52203d4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6800","nonce":2900093195},{"type":"v1","addr":"172.21.15.105:6801","nonce":2900093195}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6802","nonce":2900093195},{"type":"v1","addr":"172.21.15.105:6803","nonce":2900093195}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6806","nonce":2900093195},{"type":"v1","addr":"172.21.15.105:6807","nonce":2900093195}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6804","nonce":2900093195},{"type":"v1","addr":"172.21.15.105:6805","nonce":2900093195}]},"public_addr":"172.21.15.105:6801/2900093195","cluster_addr":"172.21.15.105:6803/2900093195","heartbeat_back_addr":"172.21.15.105:6807/2900093195","heartbeat_front_addr":"172.21.15.105:6805/2900093195","state":["exists","up"]},{"osd":2,"uuid":"41a9df5b-882a-47da-bb26-7f4d26ccf6c1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6800","nonce":2251189314},{"type":"v1","addr":"172.21.15.173:6801","nonce":2251189314}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6802","nonce":2251189314},{"type":"v1","addr":"172.21.15.173:6803","nonce":2251189314}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6806","nonce":2251189314},{"type":"v1","addr":"172.21.15.173:6807","nonce":2251189314}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6804","nonce":2251189314},{"type":"v1","addr":"172.21.15.173:6805","nonce":2251189314}]},"public_addr":"172.21.15.173:6801/2251189314","cluster_addr":"172.21.15.173:6803/2251189314","heartbeat_back_addr":"172.21.15.173:6807/2251189314","heartbeat_front_addr":"172.21.15.173:6805/2251189314","state":["exists","up"]},{"osd":3,"uuid":"f1245439-42a5-46c5-a998-e0a78f9df3fb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":26,"up_thru":82,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.52:6810","nonce":57920794},{"type":"v1","addr":"172.21.15.52:6811","nonce":57920794}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.52:6812","nonce":57920794},{"type":"v1","addr":"172.21.15.52:6813","nonce":57920794}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.52:6816","nonce":57920794},{"type":"v1","addr":"172.21.15.52:6817","nonce":57920794}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.52:6814","nonce":57920794},{"type":"v1","addr":"172.21.15.52:6815","nonce":57920794}]},"public_addr":"172.21.15.52:6811/57920794","cluster_addr":"172.21.15.52:6813/57920794","heartbeat_back_addr":"172.21.15.52:6817/57920794","heartbeat_front_addr":"172.21.15.52:6815/57920794","state":["exists","up"]},{"osd":4,"uuid":"2592d1cc-1726-4359-83d8-a2e6314b54ec","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":32,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6808","nonce":3441134818},{"type":"v1","addr":"172.21.15.105:6809","nonce":3441134818}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6810","nonce":3441134818},{"type":"v1","addr":"172.21.15.105:6811","nonce":3441134818}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6814","nonce":3441134818},{"type":"v1","addr":"172.21.15.105:6815","nonce":3441134818}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6812","nonce":3441134818},{"type":"v1","addr":"172.21.15.105:6813","nonce":3441134818}]},"public_addr":"172.21.15.105:6809/3441134818","cluster_addr":"172.21.15.105:6811/3441134818","heartbeat_back_addr":"172.21.15.105:6815/3441134818","heartbeat_front_addr":"172.21.15.105:6813/3441134818","state":["exists","up"]},{"osd":5,"uuid":"27e98ee2-4a92-4972-b2a3-959d0ed90981","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":37,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6808","nonce":2989239729},{"type":"v1","addr":"172.21.15.173:6809","nonce":2989239729}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6810","nonce":2989239729},{"type":"v1","addr":"172.21.15.173:6811","nonce":2989239729}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6814","nonce":2989239729},{"type":"v1","addr":"172.21.15.173:6815","nonce":2989239729}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6812","nonce":2989239729},{"type":"v1","addr":"172.21.15.173:6813","nonce":2989239729}]},"public_addr":"172.21.15.173:6809/2989239729","cluster_addr":"172.21.15.173:6811/2989239729","heartbeat_back_addr":"172.21.15.173:6815/2989239729","heartbeat_front_addr":"172.21.15.173:6813/2989239729","state":["exists","up"]},{"osd":6,"uuid":"2b0ec0cb-f438-4b2c-8642-0778ed374e9a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":43,"up_thru":82,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.52:6818","nonce":4244423065},{"type":"v1","addr":"172.21.15.52:6819","nonce":4244423065}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.52:6820","nonce":4244423065},{"type":"v1","addr":"172.21.15.52:6821","nonce":4244423065}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.52:6824","nonce":4244423065},{"type":"v1","addr":"172.21.15.52:6825","nonce":4244423065}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.52:6822","nonce":4244423065},{"type":"v1","addr":"172.21.15.52:6823","nonce":4244423065}]},"public_addr":"172.21.15.52:6819/4244423065","cluster_addr":"172.21.15.52:6821/4244423065","heartbeat_back_addr":"172.21.15.52:6825/4244423065","heartbeat_front_addr":"172.21.15.52:6823/4244423065","state":["exists","up"]},{"osd":7,"uuid":"cb2a7a6b-8015-43ea-865a-be144e45c51a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":48,"up_thru":82,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6816","nonce":2063611552},{"type":"v1","addr":"172.21.15.105:6817","nonce":2063611552}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6818","nonce":2063611552},{"type":"v1","addr":"172.21.15.105:6819","nonce":2063611552}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6822","nonce":2063611552},{"type":"v1","addr":"172.21.15.105:6823","nonce":2063611552}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6820","nonce":2063611552},{"type":"v1","addr":"172.21.15.105:6821","nonce":2063611552}]},"public_addr":"172.21.15.105:6817/2063611552","cluster_addr":"172.21.15.105:6819/2063611552","heartbeat_back_addr":"172.21.15.105:6823/2063611552","heartbeat_front_addr":"172.21.15.105:6821/2063611552","state":["exists","up"]},{"osd":8,"uuid":"c2fc565d-a0b9-4396-a209-bd884cddcfe3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":54,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6816","nonce":4131399852},{"type":"v1","addr":"172.21.15.173:6817","nonce":4131399852}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6818","nonce":4131399852},{"type":"v1","addr":"172.21.15.173:6819","nonce":4131399852}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6822","nonce":4131399852},{"type":"v1","addr":"172.21.15.173:6823","nonce":4131399852}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6820","nonce":4131399852},{"type":"v1","addr":"172.21.15.173:6821","nonce":4131399852}]},"public_addr":"172.21.15.173:6817/4131399852","cluster_addr":"172.21.15.173:6819/4131399852","heartbeat_back_addr":"172.21.15.173:6823/4131399852","heartbeat_front_addr":"172.21.15.173:6821/4131399852","state":["exists","up"]},{"osd":9,"uuid":"78fa7f17-1dad-4262-be40-31fa82e75520","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":60,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.52:6826","nonce":479119778},{"type":"v1","addr":"172.21.15.52:6827","nonce":479119778}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.52:6828","nonce":479119778},{"type":"v1","addr":"172.21.15.52:6829","nonce":479119778}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.52:6832","nonce":479119778},{"type":"v1","addr":"172.21.15.52:6833","nonce":479119778}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.52:6830","nonce":479119778},{"type":"v1","addr":"172.21.15.52:6831","nonce":479119778}]},"public_addr":"172.21.15.52:6827/479119778","cluster_addr":"172.21.15.52:6829/479119778","heartbeat_back_addr":"172.21.15.52:6833/479119778","heartbeat_front_addr":"172.21.15.52:6831/479119778","state":["exists","up"]},{"osd":10,"uuid":"e6f8e753-e15c-45b5-8ee9-d50f384559f9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":65,"up_thru":76,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6824","nonce":4032386628},{"type":"v1","addr":"172.21.15.105:6825","nonce":4032386628}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6826","nonce":4032386628},{"type":"v1","addr":"172.21.15.105:6827","nonce":4032386628}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6830","nonce":4032386628},{"type":"v1","addr":"172.21.15.105:6831","nonce":4032386628}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6828","nonce":4032386628},{"type":"v1","addr":"172.21.15.105:6829","nonce":4032386628}]},"public_addr":"172.21.15.105:6825/4032386628","cluster_addr":"172.21.15.105:6827/4032386628","heartbeat_back_addr":"172.21.15.105:6831/4032386628","heartbeat_front_addr":"172.21.15.105:6829/4032386628","state":["exists","up"]},{"osd":11,"uuid":"164572bf-67ed-4e8e-b32d-d7a2e8da607b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":71,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6824","nonce":1816308401},{"type":"v1","addr":"172.21.15.173:6825","nonce":1816308401}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6826","nonce":1816308401},{"type":"v1","addr":"172.21.15.173:6827","nonce":1816308401}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6830","nonce":1816308401},{"type":"v1","addr":"172.21.15.173:6831","nonce":1816308401}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6828","nonce":1816308401},{"type":"v1","addr":"172.21.15.173:6829","nonce":1816308401}]},"public_addr":"172.21.15.173:6825/1816308401","cluster_addr":"172.21.15.173:6827/1816308401","heartbeat_back_addr":"172.21.15.173:6831/1816308401","heartbeat_front_addr":"172.21.15.173:6829/1816308401","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-27T07:16:06.279220+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-27T07:16:29.668167+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-27T07:16:53.903808+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-27T07:17:18.787372+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-27T07:17:43.480784+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-27T07:18:08.816030+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-27T07:18:34.076148+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-27T07:19:00.283468+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-27T07:19:25.403023+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-27T07:19:52.139173+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-27T07:20:17.530304+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-27T07:20:43.665632+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.8","mappings":[{"from":2,"to":11}]},{"pgid":"2.f","mappings":[{"from":8,"to":11}]},{"pgid":"2.10","mappings":[{"from":2,"to":4}]},{"pgid":"3.0","mappings":[{"from":5,"to":9}]},{"pgid":"3.1b","mappings":[{"from":4,"to":8}]},{"pgid":"3.2c","mappings":[{"from":4,"to":9}]},{"pgid":"3.3c","mappings":[{"from":4,"to":9}]},{"pgid":"4.e","mappings":[{"from":9,"to":11}]},{"pgid":"4.12","mappings":[{"from":5,"to":10}]},{"pgid":"4.19","mappings":[{"from":9,"to":8}]},{"pgid":"4.1f","mappings":[{"from":9,"to":8}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.52:0/3430810618":"2024-05-28T07:25:56.925056+0000","172.21.15.52:0/2723769517":"2024-05-28T07:25:56.925056+0000","172.21.15.52:0/262308791":"2024-05-28T07:25:56.925056+0000","172.21.15.52:6801/1631683163":"2024-05-28T07:14:29.544633+0000","172.21.15.52:6801/3513075805":"2024-05-28T07:25:56.925056+0000","172.21.15.52:0/2776144659":"2024-05-28T07:14:29.544633+0000","172.21.15.52:0/3577886285":"2024-05-28T07:25:56.925056+0000","172.21.15.52:6800/1631683163":"2024-05-28T07:14:29.544633+0000","172.21.15.52:0/3675529097":"2024-05-28T07:14:01.960550+0000","172.21.15.52:0/727808107":"2024-05-28T07:14:01.960550+0000","172.21.15.52:0/3839436833":"2024-05-28T07:14:29.544633+0000","172.21.15.52:6800/587662512":"2024-05-28T07:14:01.960550+0000","172.21.15.52:6800/3513075805":"2024-05-28T07:25:56.925056+0000","172.21.15.52:6801/587662512":"2024-05-28T07:14:01.960550+0000","172.21.15.52:0/853867465":"2024-05-28T07:14:01.960550+0000","172.21.15.52:0/3480918279":"2024-05-28T07:14:29.544633+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":5,"length":1},{"begin":15,"length":1}]},{"pool":4,"snaps":[{"begin":5,"length":1},{"begin":15,"length":1}]}],"new_removed_snaps":[{"pool":3,"snaps":[{"begin":5,"length":1},{"begin":15,"length":1}]},{"pool":4,"snaps":[{"begin":5,"length":1},{"begin":15,"length":1}]}],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-05-27T07:36:48.141 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:48.145+0000 7f0145c69640 1 -- 172.21.15.52:0/296155525 >> [v2:172.21.15.52:6800/1822318384,v1:172.21.15.52:6801/1822318384] conn(0x7f01200b2a40 msgr2=0x7f01200b4f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:36:48.141 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:48.145+0000 7f0145c69640 1 --2- 172.21.15.52:0/296155525 >> [v2:172.21.15.52:6800/1822318384,v1:172.21.15.52:6801/1822318384] conn(0x7f01200b2a40 0x7f01200b4f50 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7f014010e4f0 tx=0x7f0128005b00 comp rx=0 tx=0).stop 2024-05-27T07:36:48.141 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:48.145+0000 7f0145c69640 1 -- 172.21.15.52:0/296155525 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f014010a200 msgr2=0x7f01401a9130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:36:48.141 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:48.145+0000 7f0145c69640 1 --2- 172.21.15.52:0/296155525 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f014010a200 0x7f01401a9130 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 crypto rx=0x7f0130006010 tx=0x7f0130004530 comp rx=0 tx=0).stop 2024-05-27T07:36:48.144 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:48.148+0000 7f0145c69640 1 -- 172.21.15.52:0/296155525 shutdown_connections 2024-05-27T07:36:48.144 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:48.148+0000 7f0145c69640 1 --2- 172.21.15.52:0/296155525 >> [v2:172.21.15.52:6800/1822318384,v1:172.21.15.52:6801/1822318384] conn(0x7f01200b2a40 0x7f01200b4f50 unknown :-1 s=CLOSED pgs=200 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:36:48.144 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:48.148+0000 7f0145c69640 1 --2- 172.21.15.52:0/296155525 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f01401ad890 0x7f01401adca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:36:48.145 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:48.148+0000 7f0145c69640 1 --2- 172.21.15.52:0/296155525 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f01401103e0 0x7f01401a9690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:36:48.145 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:48.148+0000 7f0145c69640 1 --2- 172.21.15.52:0/296155525 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f014010a200 0x7f01401a9130 unknown :-1 s=CLOSED pgs=278 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:36:48.145 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:48.148+0000 7f0145c69640 1 -- 172.21.15.52:0/296155525 >> 172.21.15.52:0/296155525 conn(0x7f014006e6e0 msgr2=0x7f01400694b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:36:48.146 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:48.150+0000 7f0145c69640 1 -- 172.21.15.52:0/296155525 shutdown_connections 2024-05-27T07:36:48.146 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:48.151+0000 7f0145c69640 1 -- 172.21.15.52:0/296155525 wait complete. 2024-05-27T07:36:48.181 INFO:tasks.check_counter.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-05-27T07:16:57.644034+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-05-27T07:21:43.917748+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '79', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.8799999952316284, 'score_stable': 1.8799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.8799999952316284, 'raw_score_stable': 1.8799999952316284, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-05-27T07:21:47.341876+0000', 'flags': 8193, 'flags_names': 'hashpspool,selfmanaged_snaps', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '127', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 15, 'snap_epoch': 127, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2024-05-27T07:21:57.707611+0000', 'flags': 8197, 'flags_names': 'hashpspool,ec_overwrites,selfmanaged_snaps', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '127', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 15, 'snap_epoch': 127, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2024-05-27T07:36:48.181 DEBUG:teuthology.orchestra.run.smithi052:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-05-27T07:36:48.461 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:48.464+0000 7f654f59e640 1 -- 172.21.15.52:0/4068841152 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f655011d240 msgr2=0x7f655011f660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:36:48.461 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:48.464+0000 7f654f59e640 1 --2- 172.21.15.52:0/4068841152 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f655011d240 0x7f655011f660 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7f6548009890 tx=0x7f654802a380 comp rx=0 tx=0).stop 2024-05-27T07:36:48.466 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:48.469+0000 7f654f59e640 1 -- 172.21.15.52:0/4068841152 shutdown_connections 2024-05-27T07:36:48.466 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:48.469+0000 7f654f59e640 1 --2- 172.21.15.52:0/4068841152 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f655011d240 0x7f655011f660 unknown :-1 s=CLOSED pgs=232 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:36:48.467 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:48.469+0000 7f654f59e640 1 --2- 172.21.15.52:0/4068841152 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f655011a8e0 0x7f655011cd00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:36:48.467 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:48.469+0000 7f654f59e640 1 --2- 172.21.15.52:0/4068841152 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f655010a0e0 0x7f655010a4f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:36:48.467 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:48.469+0000 7f654f59e640 1 -- 172.21.15.52:0/4068841152 >> 172.21.15.52:0/4068841152 conn(0x7f655006f770 msgr2=0x7f655006fb80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:36:48.472 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:48.476+0000 7f654f59e640 1 -- 172.21.15.52:0/4068841152 shutdown_connections 2024-05-27T07:36:48.473 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:48.477+0000 7f654f59e640 1 -- 172.21.15.52:0/4068841152 wait complete. 2024-05-27T07:36:48.474 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:48.478+0000 7f654f59e640 1 Processor -- start 2024-05-27T07:36:48.478 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:48.481+0000 7f654f59e640 1 -- start start 2024-05-27T07:36:48.478 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:48.481+0000 7f654f59e640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f655010a0e0 0x7f65501a9510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:36:48.478 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:48.481+0000 7f654f59e640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f655011a8e0 0x7f65501a9a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:36:48.479 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:48.481+0000 7f654f59e640 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f655011d240 0x7f65501add30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:36:48.479 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:48.481+0000 7f654f59e640 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x7f6550122a90 con 0x7f655011d240 2024-05-27T07:36:48.479 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:48.481+0000 7f654f59e640 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f6550122910 con 0x7f655011a8e0 2024-05-27T07:36:48.479 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:48.481+0000 7f654f59e640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f6550122c10 con 0x7f655010a0e0 2024-05-27T07:36:48.479 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:48.482+0000 7f654e59c640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f655010a0e0 0x7f65501a9510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:36:48.479 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:48.482+0000 7f654e59c640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f655010a0e0 0x7f65501a9510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.173:3300/0 says I am v2:172.21.15.52:52080/0 (socket says 172.21.15.52:52080) 2024-05-27T07:36:48.479 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:48.482+0000 7f654e59c640 1 -- 172.21.15.52:0/2375459484 learned_addr learned my addr 172.21.15.52:0/2375459484 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-05-27T07:36:48.481 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:48.485+0000 7f654ed9d640 1 --2- 172.21.15.52:0/2375459484 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f655011d240 0x7f65501add30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:36:48.481 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:48.486+0000 7f654dd9b640 1 --2- 172.21.15.52:0/2375459484 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f655011a8e0 0x7f65501a9a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:36:48.485 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:36:48 smithi052 ceph-mon[31955]: Adjusting osd_memory_target on smithi173 to 176.2M 2024-05-27T07:36:48.486 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:36:48 smithi052 ceph-mon[31955]: Unable to set osd_memory_target on smithi173 to 184791040: error parsing value: Value '184791040' is below minimum 939524096 2024-05-27T07:36:48.486 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:36:48 smithi052 ceph-mon[31955]: Adjusting osd_memory_target on smithi105 to 176.2M 2024-05-27T07:36:48.486 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:36:48 smithi052 ceph-mon[31955]: Unable to set osd_memory_target on smithi105 to 184784588: error parsing value: Value '184784588' is below minimum 939524096 2024-05-27T07:36:48.486 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:36:48 smithi052 ceph-mon[31955]: pgmap v364: 129 pgs: 11 active+clean+snaptrim, 30 active+clean+snaptrim_wait, 88 active+clean; 623 MiB data, 7.3 GiB used, 1.0 TiB / 1.0 TiB avail 2024-05-27T07:36:48.486 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:36:48 smithi052 ceph-mon[31955]: from='client.? 172.21.15.52:0/296155525' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-05-27T07:36:48.486 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:48.487+0000 7f654e59c640 1 -- 172.21.15.52:0/2375459484 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f655011a8e0 msgr2=0x7f65501a9a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:36:48.486 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:48.487+0000 7f654e59c640 1 --2- 172.21.15.52:0/2375459484 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f655011a8e0 0x7f65501a9a70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:36:48.486 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:48.487+0000 7f654e59c640 1 -- 172.21.15.52:0/2375459484 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f655011d240 msgr2=0x7f65501add30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:36:48.486 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:48.487+0000 7f654e59c640 1 --2- 172.21.15.52:0/2375459484 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f655011d240 0x7f65501add30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:36:48.487 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:48.487+0000 7f654e59c640 1 -- 172.21.15.52:0/2375459484 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f65501ae370 con 0x7f655010a0e0 2024-05-27T07:36:48.487 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:48.490+0000 7f654ed9d640 1 --2- 172.21.15.52:0/2375459484 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f655011d240 0x7f65501add30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-27T07:36:48.490 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:48.490+0000 7f654e59c640 1 --2- 172.21.15.52:0/2375459484 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f655010a0e0 0x7f65501a9510 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7f654000c3f0 tx=0x7f654000c8b0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:36:48.490 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:48.492+0000 7f653f7fe640 1 -- 172.21.15.52:0/2375459484 <== mon.1 v2:172.21.15.173:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6540004b20 con 0x7f655010a0e0 2024-05-27T07:36:48.491 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:48.492+0000 7f653f7fe640 1 -- 172.21.15.52:0/2375459484 <== mon.1 v2:172.21.15.173:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f6540003e20 con 0x7f655010a0e0 2024-05-27T07:36:48.491 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:48.492+0000 7f653f7fe640 1 -- 172.21.15.52:0/2375459484 <== mon.1 v2:172.21.15.173:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6540004800 con 0x7f655010a0e0 2024-05-27T07:36:48.491 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:48.495+0000 7f654f59e640 1 -- 172.21.15.52:0/2375459484 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f65501ae660 con 0x7f655010a0e0 2024-05-27T07:36:48.492 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:48.495+0000 7f654f59e640 1 -- 172.21.15.52:0/2375459484 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f65501aeb60 con 0x7f655010a0e0 2024-05-27T07:36:48.495 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:48.499+0000 7f653f7fe640 1 -- 172.21.15.52:0/2375459484 <== mon.1 v2:172.21.15.173:3300/0 4 ==== mgrmap(e 26) ==== 149783+0+0 (secure 0 0 0) 0x7f654001d020 con 0x7f655010a0e0 2024-05-27T07:36:48.495 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:48.500+0000 7f653f7fe640 1 --2- 172.21.15.52:0/2375459484 >> [v2:172.21.15.52:6800/1822318384,v1:172.21.15.52:6801/1822318384] conn(0x7f65380b2b70 0x7f65380b5080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:36:48.496 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:48.500+0000 7f653f7fe640 1 -- 172.21.15.52:0/2375459484 <== mon.1 v2:172.21.15.173:3300/0 5 ==== osd_map(127..127 src has 1..127) ==== 9555+0+0 (secure 0 0 0) 0x7f65400e3a10 con 0x7f655010a0e0 2024-05-27T07:36:48.497 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:48.501+0000 7f654f59e640 1 -- 172.21.15.52:0/2375459484 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6530005290 con 0x7f655010a0e0 2024-05-27T07:36:48.498 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:48.502+0000 7f654dd9b640 1 --2- 172.21.15.52:0/2375459484 >> [v2:172.21.15.52:6800/1822318384,v1:172.21.15.52:6801/1822318384] conn(0x7f65380b2b70 0x7f65380b5080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:36:48.498 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:48.503+0000 7f654dd9b640 1 --2- 172.21.15.52:0/2375459484 >> [v2:172.21.15.52:6800/1822318384,v1:172.21.15.52:6801/1822318384] conn(0x7f65380b2b70 0x7f65380b5080 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7f65501101c0 tx=0x7f6544005f90 comp rx=0 tx=0).ready entity=mgr.15333 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:36:48.506 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:48.510+0000 7f653f7fe640 1 -- 172.21.15.52:0/2375459484 <== mon.1 v2:172.21.15.173:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191963 (secure 0 0 0) 0x7f6540052df0 con 0x7f655010a0e0 2024-05-27T07:36:48.696 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:36:48 smithi105 ceph-mon[35977]: Adjusting osd_memory_target on smithi173 to 176.2M 2024-05-27T07:36:48.696 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:36:48 smithi105 ceph-mon[35977]: Unable to set osd_memory_target on smithi173 to 184791040: error parsing value: Value '184791040' is below minimum 939524096 2024-05-27T07:36:48.696 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:36:48 smithi105 ceph-mon[35977]: Adjusting osd_memory_target on smithi105 to 176.2M 2024-05-27T07:36:48.696 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:36:48 smithi105 ceph-mon[35977]: Unable to set osd_memory_target on smithi105 to 184784588: error parsing value: Value '184784588' is below minimum 939524096 2024-05-27T07:36:48.696 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:36:48 smithi105 ceph-mon[35977]: pgmap v364: 129 pgs: 11 active+clean+snaptrim, 30 active+clean+snaptrim_wait, 88 active+clean; 623 MiB data, 7.3 GiB used, 1.0 TiB / 1.0 TiB avail 2024-05-27T07:36:48.696 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:36:48 smithi105 ceph-mon[35977]: from='client.? 172.21.15.52:0/296155525' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-05-27T07:36:48.858 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:36:48 smithi173 ceph-mon[37000]: Adjusting osd_memory_target on smithi173 to 176.2M 2024-05-27T07:36:48.858 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:36:48 smithi173 ceph-mon[37000]: Unable to set osd_memory_target on smithi173 to 184791040: error parsing value: Value '184791040' is below minimum 939524096 2024-05-27T07:36:48.858 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:36:48 smithi173 ceph-mon[37000]: Adjusting osd_memory_target on smithi105 to 176.2M 2024-05-27T07:36:48.858 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:36:48 smithi173 ceph-mon[37000]: Unable to set osd_memory_target on smithi105 to 184784588: error parsing value: Value '184784588' is below minimum 939524096 2024-05-27T07:36:48.858 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:36:48 smithi173 ceph-mon[37000]: pgmap v364: 129 pgs: 11 active+clean+snaptrim, 30 active+clean+snaptrim_wait, 88 active+clean; 623 MiB data, 7.3 GiB used, 1.0 TiB / 1.0 TiB avail 2024-05-27T07:36:48.858 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:36:48 smithi173 ceph-mon[37000]: from='client.? 172.21.15.52:0/296155525' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-05-27T07:36:48.897 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:48.901+0000 7f654f59e640 1 -- 172.21.15.52:0/2375459484 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f6530005850 con 0x7f655010a0e0 2024-05-27T07:36:48.902 INFO:teuthology.orchestra.run.smithi052.stdout:pg_num: 1 2024-05-27T07:36:48.902 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:48.905+0000 7f653f7fe640 1 -- 172.21.15.52:0/2375459484 <== mon.1 v2:172.21.15.173:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v127) ==== 93+0+10 (secure 0 0 0) 0x7f65501aeb60 con 0x7f655010a0e0 2024-05-27T07:36:48.913 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:48.917+0000 7f654f59e640 1 -- 172.21.15.52:0/2375459484 >> [v2:172.21.15.52:6800/1822318384,v1:172.21.15.52:6801/1822318384] conn(0x7f65380b2b70 msgr2=0x7f65380b5080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:36:48.913 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:48.917+0000 7f654f59e640 1 --2- 172.21.15.52:0/2375459484 >> [v2:172.21.15.52:6800/1822318384,v1:172.21.15.52:6801/1822318384] conn(0x7f65380b2b70 0x7f65380b5080 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7f65501101c0 tx=0x7f6544005f90 comp rx=0 tx=0).stop 2024-05-27T07:36:48.913 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:48.917+0000 7f654f59e640 1 -- 172.21.15.52:0/2375459484 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f655010a0e0 msgr2=0x7f65501a9510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:36:48.913 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:48.917+0000 7f654f59e640 1 --2- 172.21.15.52:0/2375459484 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f655010a0e0 0x7f65501a9510 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7f654000c3f0 tx=0x7f654000c8b0 comp rx=0 tx=0).stop 2024-05-27T07:36:48.918 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:48.922+0000 7f654f59e640 1 -- 172.21.15.52:0/2375459484 shutdown_connections 2024-05-27T07:36:48.918 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:48.922+0000 7f654f59e640 1 --2- 172.21.15.52:0/2375459484 >> [v2:172.21.15.52:6800/1822318384,v1:172.21.15.52:6801/1822318384] conn(0x7f65380b2b70 0x7f65380b5080 unknown :-1 s=CLOSED pgs=201 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:36:48.918 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:48.922+0000 7f654f59e640 1 --2- 172.21.15.52:0/2375459484 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f655011d240 0x7f65501add30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:36:48.918 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:48.922+0000 7f654f59e640 1 --2- 172.21.15.52:0/2375459484 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f655011a8e0 0x7f65501a9a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:36:48.918 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:48.922+0000 7f654f59e640 1 --2- 172.21.15.52:0/2375459484 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f655010a0e0 0x7f65501a9510 unknown :-1 s=CLOSED pgs=233 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:36:48.919 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:48.922+0000 7f654f59e640 1 -- 172.21.15.52:0/2375459484 >> 172.21.15.52:0/2375459484 conn(0x7f655006f770 msgr2=0x7f655011acf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:36:48.922 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:48.927+0000 7f654f59e640 1 -- 172.21.15.52:0/2375459484 shutdown_connections 2024-05-27T07:36:48.925 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:48.930+0000 7f654f59e640 1 -- 172.21.15.52:0/2375459484 wait complete. 2024-05-27T07:36:48.954 DEBUG:teuthology.orchestra.run.smithi052:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-05-27T07:36:49.179 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:49.183+0000 7f95df022640 1 -- 172.21.15.52:0/797685782 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f95d81167c0 msgr2=0x7f95d8118cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:36:49.180 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:49.183+0000 7f95df022640 1 --2- 172.21.15.52:0/797685782 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f95d81167c0 0x7f95d8118cd0 secure :-1 s=READY pgs=766 cs=0 l=1 rev1=1 crypto rx=0x7f95d0009ca0 tx=0x7f95d002a8f0 comp rx=0 tx=0).stop 2024-05-27T07:36:49.180 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:49.183+0000 7f95df022640 1 -- 172.21.15.52:0/797685782 shutdown_connections 2024-05-27T07:36:49.180 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:49.183+0000 7f95df022640 1 --2- 172.21.15.52:0/797685782 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f95d81167c0 0x7f95d8118cd0 unknown :-1 s=CLOSED pgs=766 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:36:49.180 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:49.183+0000 7f95df022640 1 --2- 172.21.15.52:0/797685782 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f95d8113d10 0x7f95d8116130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:36:49.180 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:49.183+0000 7f95df022640 1 --2- 172.21.15.52:0/797685782 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f95d810a380 0x7f95d810a790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:36:49.180 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:49.183+0000 7f95df022640 1 -- 172.21.15.52:0/797685782 >> 172.21.15.52:0/797685782 conn(0x7f95d806e6f0 msgr2=0x7f95d806eb50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:36:49.180 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:49.184+0000 7f95df022640 1 -- 172.21.15.52:0/797685782 shutdown_connections 2024-05-27T07:36:49.183 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:49.187+0000 7f95df022640 1 -- 172.21.15.52:0/797685782 wait complete. 2024-05-27T07:36:49.184 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:49.188+0000 7f95df022640 1 Processor -- start 2024-05-27T07:36:49.187 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:49.190+0000 7f95df022640 1 -- start start 2024-05-27T07:36:49.187 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:49.191+0000 7f95df022640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f95d810a380 0x7f95d8121cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:36:49.187 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:49.191+0000 7f95df022640 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f95d8113d10 0x7f95d811be00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:36:49.188 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:49.191+0000 7f95df022640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f95d81167c0 0x7f95d811c340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:36:49.188 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:49.191+0000 7f95df022640 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x7f95d811a2d0 con 0x7f95d8113d10 2024-05-27T07:36:49.188 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:49.191+0000 7f95df022640 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f95d811a150 con 0x7f95d81167c0 2024-05-27T07:36:49.188 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:49.191+0000 7f95df022640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f95d811a450 con 0x7f95d810a380 2024-05-27T07:36:49.188 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:49.192+0000 7f95de821640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f95d81167c0 0x7f95d811c340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:36:49.188 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:49.193+0000 7f95de821640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f95d81167c0 0x7f95d811c340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.52:56168/0 (socket says 172.21.15.52:56168) 2024-05-27T07:36:49.188 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:49.193+0000 7f95de821640 1 -- 172.21.15.52:0/2162190679 learned_addr learned my addr 172.21.15.52:0/2162190679 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-05-27T07:36:49.189 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:49.193+0000 7f95de020640 1 --2- 172.21.15.52:0/2162190679 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f95d810a380 0x7f95d8121cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:36:49.189 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:49.193+0000 7f95de821640 1 -- 172.21.15.52:0/2162190679 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f95d810a380 msgr2=0x7f95d8121cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:36:49.190 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:49.193+0000 7f95dd81f640 1 --2- 172.21.15.52:0/2162190679 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f95d8113d10 0x7f95d811be00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:36:49.190 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:49.193+0000 7f95de821640 1 --2- 172.21.15.52:0/2162190679 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f95d810a380 0x7f95d8121cf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:36:49.190 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:49.193+0000 7f95de821640 1 -- 172.21.15.52:0/2162190679 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f95d8113d10 msgr2=0x7f95d811be00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:36:49.190 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:49.193+0000 7f95de821640 1 --2- 172.21.15.52:0/2162190679 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f95d8113d10 0x7f95d811be00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:36:49.190 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:49.193+0000 7f95de821640 1 -- 172.21.15.52:0/2162190679 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f95d811cb40 con 0x7f95d81167c0 2024-05-27T07:36:49.190 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:49.193+0000 7f95de020640 1 --2- 172.21.15.52:0/2162190679 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f95d810a380 0x7f95d8121cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-27T07:36:49.190 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:49.194+0000 7f95de821640 1 --2- 172.21.15.52:0/2162190679 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f95d81167c0 0x7f95d811c340 secure :-1 s=READY pgs=279 cs=0 l=1 rev1=1 crypto rx=0x7f95d0031a20 tx=0x7f95d002f3b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:36:49.190 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:49.193+0000 7f95dd81f640 1 --2- 172.21.15.52:0/2162190679 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f95d8113d10 0x7f95d811be00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-27T07:36:49.191 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:49.194+0000 7f95cf7fe640 1 -- 172.21.15.52:0/2162190679 <== mon.2 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f95d002fa50 con 0x7f95d81167c0 2024-05-27T07:36:49.193 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:49.197+0000 7f95df022640 1 -- 172.21.15.52:0/2162190679 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f95d81be160 con 0x7f95d81167c0 2024-05-27T07:36:49.193 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:49.197+0000 7f95df022640 1 -- 172.21.15.52:0/2162190679 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f95d81be6d0 con 0x7f95d81167c0 2024-05-27T07:36:49.193 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:49.197+0000 7f95cf7fe640 1 -- 172.21.15.52:0/2162190679 <== mon.2 v2:172.21.15.105:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f95d002d070 con 0x7f95d81167c0 2024-05-27T07:36:49.194 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:49.197+0000 7f95cf7fe640 1 -- 172.21.15.52:0/2162190679 <== mon.2 v2:172.21.15.105:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f95d0045420 con 0x7f95d81167c0 2024-05-27T07:36:49.195 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:49.199+0000 7f95df022640 1 -- 172.21.15.52:0/2162190679 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f95bc005290 con 0x7f95d81167c0 2024-05-27T07:36:49.202 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:49.206+0000 7f95cf7fe640 1 -- 172.21.15.52:0/2162190679 <== mon.2 v2:172.21.15.105:3300/0 4 ==== mgrmap(e 26) ==== 149783+0+0 (secure 0 0 0) 0x7f95d002fbf0 con 0x7f95d81167c0 2024-05-27T07:36:49.203 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:49.207+0000 7f95cf7fe640 1 --2- 172.21.15.52:0/2162190679 >> [v2:172.21.15.52:6800/1822318384,v1:172.21.15.52:6801/1822318384] conn(0x7f95c40b2aa0 0x7f95c40b4fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:36:49.204 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:49.208+0000 7f95cf7fe640 1 -- 172.21.15.52:0/2162190679 <== mon.2 v2:172.21.15.105:3300/0 5 ==== osd_map(127..127 src has 1..127) ==== 9555+0+0 (secure 0 0 0) 0x7f95d0104130 con 0x7f95d81167c0 2024-05-27T07:36:49.204 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:49.208+0000 7f95cf7fe640 1 -- 172.21.15.52:0/2162190679 <== mon.2 v2:172.21.15.105:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191963 (secure 0 0 0) 0x7f95d004a060 con 0x7f95d81167c0 2024-05-27T07:36:49.206 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:49.210+0000 7f95de020640 1 --2- 172.21.15.52:0/2162190679 >> [v2:172.21.15.52:6800/1822318384,v1:172.21.15.52:6801/1822318384] conn(0x7f95c40b2aa0 0x7f95c40b4fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:36:49.222 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:49.227+0000 7f95de020640 1 --2- 172.21.15.52:0/2162190679 >> [v2:172.21.15.52:6800/1822318384,v1:172.21.15.52:6801/1822318384] conn(0x7f95c40b2aa0 0x7f95c40b4fb0 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7f95d810ad50 tx=0x7f95c8008040 comp rx=0 tx=0).ready entity=mgr.15333 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:36:49.602 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:49.607+0000 7f95df022640 1 -- 172.21.15.52:0/2162190679 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7f95bc005850 con 0x7f95d81167c0 2024-05-27T07:36:49.604 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:49.608+0000 7f95cf7fe640 1 -- 172.21.15.52:0/2162190679 <== mon.2 v2:172.21.15.105:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v127) ==== 104+0+11 (secure 0 0 0) 0x7f95d00cb7b0 con 0x7f95d81167c0 2024-05-27T07:36:49.607 INFO:teuthology.orchestra.run.smithi052.stdout:pg_num: 32 2024-05-27T07:36:49.616 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:49.621+0000 7f95df022640 1 -- 172.21.15.52:0/2162190679 >> [v2:172.21.15.52:6800/1822318384,v1:172.21.15.52:6801/1822318384] conn(0x7f95c40b2aa0 msgr2=0x7f95c40b4fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:36:49.617 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:49.621+0000 7f95df022640 1 --2- 172.21.15.52:0/2162190679 >> [v2:172.21.15.52:6800/1822318384,v1:172.21.15.52:6801/1822318384] conn(0x7f95c40b2aa0 0x7f95c40b4fb0 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7f95d810ad50 tx=0x7f95c8008040 comp rx=0 tx=0).stop 2024-05-27T07:36:49.617 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:49.621+0000 7f95df022640 1 -- 172.21.15.52:0/2162190679 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f95d81167c0 msgr2=0x7f95d811c340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:36:49.617 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:49.621+0000 7f95df022640 1 --2- 172.21.15.52:0/2162190679 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f95d81167c0 0x7f95d811c340 secure :-1 s=READY pgs=279 cs=0 l=1 rev1=1 crypto rx=0x7f95d0031a20 tx=0x7f95d002f3b0 comp rx=0 tx=0).stop 2024-05-27T07:36:49.620 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:49.623+0000 7f95df022640 1 -- 172.21.15.52:0/2162190679 shutdown_connections 2024-05-27T07:36:49.620 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:49.623+0000 7f95df022640 1 --2- 172.21.15.52:0/2162190679 >> [v2:172.21.15.52:6800/1822318384,v1:172.21.15.52:6801/1822318384] conn(0x7f95c40b2aa0 0x7f95c40b4fb0 unknown :-1 s=CLOSED pgs=202 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:36:49.620 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:49.623+0000 7f95df022640 1 --2- 172.21.15.52:0/2162190679 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f95d81167c0 0x7f95d811c340 unknown :-1 s=CLOSED pgs=279 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:36:49.621 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:49.623+0000 7f95df022640 1 --2- 172.21.15.52:0/2162190679 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f95d8113d10 0x7f95d811be00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:36:49.621 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:49.623+0000 7f95df022640 1 --2- 172.21.15.52:0/2162190679 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f95d810a380 0x7f95d8121cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:36:49.621 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:49.623+0000 7f95df022640 1 -- 172.21.15.52:0/2162190679 >> 172.21.15.52:0/2162190679 conn(0x7f95d806e6f0 msgr2=0x7f95d8073a30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:36:49.622 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:49.626+0000 7f95df022640 1 -- 172.21.15.52:0/2162190679 shutdown_connections 2024-05-27T07:36:49.622 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:49.626+0000 7f95df022640 1 -- 172.21.15.52:0/2162190679 wait complete. 2024-05-27T07:36:49.645 DEBUG:teuthology.orchestra.run.smithi052:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-05-27T07:36:49.696 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:36:49 smithi105 ceph-mon[35977]: from='client.? 172.21.15.52:0/2375459484' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-05-27T07:36:49.703 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:36:49 smithi052 ceph-mon[31955]: from='client.? 172.21.15.52:0/2375459484' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-05-27T07:36:49.858 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:36:49 smithi173 ceph-mon[37000]: from='client.? 172.21.15.52:0/2375459484' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-05-27T07:36:49.879 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:49.880+0000 7fe4abce0640 1 -- 172.21.15.52:0/3160038633 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fe4a411d240 msgr2=0x7fe4a411f660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:36:49.879 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:49.880+0000 7fe4abce0640 1 --2- 172.21.15.52:0/3160038633 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fe4a411d240 0x7fe4a411f660 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7fe49c009cd0 tx=0x7fe49c02a930 comp rx=0 tx=0).stop 2024-05-27T07:36:49.879 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:49.882+0000 7fe4abce0640 1 -- 172.21.15.52:0/3160038633 shutdown_connections 2024-05-27T07:36:49.879 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:49.882+0000 7fe4abce0640 1 --2- 172.21.15.52:0/3160038633 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fe4a411d240 0x7fe4a411f660 unknown :-1 s=CLOSED pgs=234 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:36:49.879 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:49.882+0000 7fe4abce0640 1 --2- 172.21.15.52:0/3160038633 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fe4a411a8e0 0x7fe4a411cd00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:36:49.880 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:49.882+0000 7fe4abce0640 1 --2- 172.21.15.52:0/3160038633 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fe4a410a0e0 0x7fe4a410a4f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:36:49.880 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:49.882+0000 7fe4abce0640 1 -- 172.21.15.52:0/3160038633 >> 172.21.15.52:0/3160038633 conn(0x7fe4a406f740 msgr2=0x7fe4a406fb80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:36:49.884 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:49.888+0000 7fe4abce0640 1 -- 172.21.15.52:0/3160038633 shutdown_connections 2024-05-27T07:36:49.886 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:49.888+0000 7fe4abce0640 1 -- 172.21.15.52:0/3160038633 wait complete. 2024-05-27T07:36:49.886 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:49.889+0000 7fe4abce0640 1 Processor -- start 2024-05-27T07:36:49.886 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:49.889+0000 7fe4abce0640 1 -- start start 2024-05-27T07:36:49.886 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:49.889+0000 7fe4abce0640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fe4a410a0e0 0x7fe4a4119ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:36:49.886 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:49.889+0000 7fe4abce0640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fe4a411a8e0 0x7fe4a4113ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:36:49.887 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:49.889+0000 7fe4abce0640 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fe4a411d240 0x7fe4a4114530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:36:49.887 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:49.889+0000 7fe4abce0640 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x7fe4a41125a0 con 0x7fe4a411d240 2024-05-27T07:36:49.888 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:49.889+0000 7fe4abce0640 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7fe4a4112420 con 0x7fe4a410a0e0 2024-05-27T07:36:49.889 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:49.889+0000 7fe4abce0640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7fe4a4112720 con 0x7fe4a411a8e0 2024-05-27T07:36:49.889 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:49.890+0000 7fe4a9a7c640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fe4a410a0e0 0x7fe4a4119ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:36:49.889 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:49.890+0000 7fe4a9a7c640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fe4a410a0e0 0x7fe4a4119ee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.52:56176/0 (socket says 172.21.15.52:56176) 2024-05-27T07:36:49.889 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:49.890+0000 7fe4a9a7c640 1 -- 172.21.15.52:0/2892729639 learned_addr learned my addr 172.21.15.52:0/2892729639 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-05-27T07:36:49.889 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:49.890+0000 7fe4a927b640 1 --2- 172.21.15.52:0/2892729639 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fe4a411a8e0 0x7fe4a4113ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:36:49.889 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:49.890+0000 7fe4a9a7c640 1 -- 172.21.15.52:0/2892729639 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fe4a411a8e0 msgr2=0x7fe4a4113ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:36:49.889 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:49.890+0000 7fe4a9a7c640 1 --2- 172.21.15.52:0/2892729639 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fe4a411a8e0 0x7fe4a4113ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:36:49.889 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:49.891+0000 7fe4a9a7c640 1 -- 172.21.15.52:0/2892729639 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fe4a411d240 msgr2=0x7fe4a4114530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:36:49.889 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:49.891+0000 7fe4a9a7c640 1 --2- 172.21.15.52:0/2892729639 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fe4a411d240 0x7fe4a4114530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:36:49.889 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:49.891+0000 7fe4a9a7c640 1 -- 172.21.15.52:0/2892729639 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe4a4114a70 con 0x7fe4a410a0e0 2024-05-27T07:36:49.889 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:49.892+0000 7fe4a9a7c640 1 --2- 172.21.15.52:0/2892729639 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fe4a410a0e0 0x7fe4a4119ee0 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 crypto rx=0x7fe49400d770 tx=0x7fe49400dc30 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:36:49.890 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:49.892+0000 7fe49affd640 1 -- 172.21.15.52:0/2892729639 <== mon.2 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe494017070 con 0x7fe4a410a0e0 2024-05-27T07:36:49.890 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:49.892+0000 7fe4abce0640 1 -- 172.21.15.52:0/2892729639 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe4a4114d60 con 0x7fe4a410a0e0 2024-05-27T07:36:49.890 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:49.892+0000 7fe49affd640 1 -- 172.21.15.52:0/2892729639 <== mon.2 v2:172.21.15.105:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fe494009c30 con 0x7fe4a410a0e0 2024-05-27T07:36:49.890 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:49.892+0000 7fe49affd640 1 -- 172.21.15.52:0/2892729639 <== mon.2 v2:172.21.15.105:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe494017070 con 0x7fe4a410a0e0 2024-05-27T07:36:49.890 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:49.892+0000 7fe4abce0640 1 -- 172.21.15.52:0/2892729639 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe4a4128990 con 0x7fe4a410a0e0 2024-05-27T07:36:49.894 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:49.896+0000 7fe49affd640 1 -- 172.21.15.52:0/2892729639 <== mon.2 v2:172.21.15.105:3300/0 4 ==== mgrmap(e 26) ==== 149783+0+0 (secure 0 0 0) 0x7fe49401a070 con 0x7fe4a410a0e0 2024-05-27T07:36:49.894 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:49.897+0000 7fe49affd640 1 --2- 172.21.15.52:0/2892729639 >> [v2:172.21.15.52:6800/1822318384,v1:172.21.15.52:6801/1822318384] conn(0x7fe4900b2e20 0x7fe4900b5330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:36:49.894 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:49.898+0000 7fe49affd640 1 -- 172.21.15.52:0/2892729639 <== mon.2 v2:172.21.15.105:3300/0 5 ==== osd_map(127..127 src has 1..127) ==== 9555+0+0 (secure 0 0 0) 0x7fe494009a30 con 0x7fe4a410a0e0 2024-05-27T07:36:49.894 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:49.898+0000 7fe4abce0640 1 -- 172.21.15.52:0/2892729639 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe4880052c0 con 0x7fe4a410a0e0 2024-05-27T07:36:49.895 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:49.899+0000 7fe4a927b640 1 --2- 172.21.15.52:0/2892729639 >> [v2:172.21.15.52:6800/1822318384,v1:172.21.15.52:6801/1822318384] conn(0x7fe4900b2e20 0x7fe4900b5330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:36:49.900 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:49.905+0000 7fe49affd640 1 -- 172.21.15.52:0/2892729639 <== mon.2 v2:172.21.15.105:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191963 (secure 0 0 0) 0x7fe4940a9400 con 0x7fe4a410a0e0 2024-05-27T07:36:49.915 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:49.918+0000 7fe4a927b640 1 --2- 172.21.15.52:0/2892729639 >> [v2:172.21.15.52:6800/1822318384,v1:172.21.15.52:6801/1822318384] conn(0x7fe4900b2e20 0x7fe4900b5330 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7fe4a0006fd0 tx=0x7fe4a0008040 comp rx=0 tx=0).ready entity=mgr.15333 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:36:50.309 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:50.313+0000 7fe4abce0640 1 -- 172.21.15.52:0/2892729639 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7fe488005880 con 0x7fe4a410a0e0 2024-05-27T07:36:50.313 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:50.318+0000 7fe49affd640 1 -- 172.21.15.52:0/2892729639 <== mon.2 v2:172.21.15.105:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v127) ==== 100+0+11 (secure 0 0 0) 0x7fe494009de0 con 0x7fe4a410a0e0 2024-05-27T07:36:50.314 INFO:teuthology.orchestra.run.smithi052.stdout:pg_num: 64 2024-05-27T07:36:50.320 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:50.324+0000 7fe4abce0640 1 -- 172.21.15.52:0/2892729639 >> [v2:172.21.15.52:6800/1822318384,v1:172.21.15.52:6801/1822318384] conn(0x7fe4900b2e20 msgr2=0x7fe4900b5330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:36:50.320 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:50.324+0000 7fe4abce0640 1 --2- 172.21.15.52:0/2892729639 >> [v2:172.21.15.52:6800/1822318384,v1:172.21.15.52:6801/1822318384] conn(0x7fe4900b2e20 0x7fe4900b5330 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7fe4a0006fd0 tx=0x7fe4a0008040 comp rx=0 tx=0).stop 2024-05-27T07:36:50.320 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:50.324+0000 7fe4abce0640 1 -- 172.21.15.52:0/2892729639 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fe4a410a0e0 msgr2=0x7fe4a4119ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:36:50.320 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:50.324+0000 7fe4abce0640 1 --2- 172.21.15.52:0/2892729639 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fe4a410a0e0 0x7fe4a4119ee0 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 crypto rx=0x7fe49400d770 tx=0x7fe49400dc30 comp rx=0 tx=0).stop 2024-05-27T07:36:50.321 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:50.325+0000 7fe4abce0640 1 -- 172.21.15.52:0/2892729639 shutdown_connections 2024-05-27T07:36:50.321 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:50.325+0000 7fe4abce0640 1 --2- 172.21.15.52:0/2892729639 >> [v2:172.21.15.52:6800/1822318384,v1:172.21.15.52:6801/1822318384] conn(0x7fe4900b2e20 0x7fe4900b5330 unknown :-1 s=CLOSED pgs=203 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:36:50.321 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:50.325+0000 7fe4abce0640 1 --2- 172.21.15.52:0/2892729639 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fe4a411d240 0x7fe4a4114530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:36:50.321 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:50.325+0000 7fe4abce0640 1 --2- 172.21.15.52:0/2892729639 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fe4a411a8e0 0x7fe4a4113ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:36:50.321 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:50.325+0000 7fe4abce0640 1 --2- 172.21.15.52:0/2892729639 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fe4a410a0e0 0x7fe4a4119ee0 unknown :-1 s=CLOSED pgs=280 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:36:50.321 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:50.325+0000 7fe4abce0640 1 -- 172.21.15.52:0/2892729639 >> 172.21.15.52:0/2892729639 conn(0x7fe4a406f740 msgr2=0x7fe4a411acf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:36:50.322 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:50.326+0000 7fe4abce0640 1 -- 172.21.15.52:0/2892729639 shutdown_connections 2024-05-27T07:36:50.322 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:50.326+0000 7fe4abce0640 1 -- 172.21.15.52:0/2892729639 wait complete. 2024-05-27T07:36:50.342 DEBUG:teuthology.orchestra.run.smithi052:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2024-05-27T07:36:50.566 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:50.570+0000 7f80cca2a640 1 -- 172.21.15.52:0/2963730014 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f80c810a1d0 msgr2=0x7f80c810a5e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:36:50.567 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:50.570+0000 7f80cca2a640 1 --2- 172.21.15.52:0/2963730014 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f80c810a1d0 0x7f80c810a5e0 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 crypto rx=0x7f80b8009890 tx=0x7f80b802a440 comp rx=0 tx=0).stop 2024-05-27T07:36:50.570 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:50.574+0000 7f80cca2a640 1 -- 172.21.15.52:0/2963730014 shutdown_connections 2024-05-27T07:36:50.570 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:50.574+0000 7f80cca2a640 1 --2- 172.21.15.52:0/2963730014 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f80c811d240 0x7f80c811f660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:36:50.571 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:50.574+0000 7f80cca2a640 1 --2- 172.21.15.52:0/2963730014 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f80c811a8e0 0x7f80c811cd00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:36:50.571 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:50.574+0000 7f80cca2a640 1 --2- 172.21.15.52:0/2963730014 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f80c810a1d0 0x7f80c810a5e0 unknown :-1 s=CLOSED pgs=281 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:36:50.571 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:50.574+0000 7f80cca2a640 1 -- 172.21.15.52:0/2963730014 >> 172.21.15.52:0/2963730014 conn(0x7f80c806e8c0 msgr2=0x7f80c806ed20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:36:50.571 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:50.575+0000 7f80cca2a640 1 -- 172.21.15.52:0/2963730014 shutdown_connections 2024-05-27T07:36:50.572 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:50.575+0000 7f80cca2a640 1 -- 172.21.15.52:0/2963730014 wait complete. 2024-05-27T07:36:50.572 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:50.576+0000 7f80cca2a640 1 Processor -- start 2024-05-27T07:36:50.572 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:50.576+0000 7f80cca2a640 1 -- start start 2024-05-27T07:36:50.573 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:50.577+0000 7f80cca2a640 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f80c810a1d0 0x7f80c811a270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:36:50.573 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:50.577+0000 7f80cca2a640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f80c811a8e0 0x7f80c8113390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:36:50.573 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:50.577+0000 7f80cca2a640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f80c811d240 0x7f80c8113980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:36:50.573 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:50.577+0000 7f80cca2a640 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x7f80c8122c40 con 0x7f80c810a1d0 2024-05-27T07:36:50.574 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:50.577+0000 7f80cca2a640 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f80c8122ac0 con 0x7f80c811d240 2024-05-27T07:36:50.574 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:50.577+0000 7f80cca2a640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f80c8122dc0 con 0x7f80c811a8e0 2024-05-27T07:36:50.574 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:50.577+0000 7f80bffff640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f80c811a8e0 0x7f80c8113390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:36:50.574 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:50.577+0000 7f80bffff640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f80c811a8e0 0x7f80c8113390 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.173:3300/0 says I am v2:172.21.15.52:52130/0 (socket says 172.21.15.52:52130) 2024-05-27T07:36:50.574 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:50.577+0000 7f80bffff640 1 -- 172.21.15.52:0/3851470066 learned_addr learned my addr 172.21.15.52:0/3851470066 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-05-27T07:36:50.574 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:50.577+0000 7f80c6d9d640 1 --2- 172.21.15.52:0/3851470066 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f80c811d240 0x7f80c8113980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:36:50.574 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:50.578+0000 7f80bffff640 1 -- 172.21.15.52:0/3851470066 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f80c811d240 msgr2=0x7f80c8113980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:36:50.574 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:50.578+0000 7f80bffff640 1 --2- 172.21.15.52:0/3851470066 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f80c811d240 0x7f80c8113980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:36:50.574 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:50.578+0000 7f80bffff640 1 -- 172.21.15.52:0/3851470066 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f80c810a1d0 msgr2=0x7f80c811a270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:36:50.574 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:50.578+0000 7f80bffff640 1 --2- 172.21.15.52:0/3851470066 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f80c810a1d0 0x7f80c811a270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:36:50.574 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:50.578+0000 7f80bffff640 1 -- 172.21.15.52:0/3851470066 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f80c8114110 con 0x7f80c811a8e0 2024-05-27T07:36:50.575 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:50.578+0000 7f80bffff640 1 --2- 172.21.15.52:0/3851470066 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f80c811a8e0 0x7f80c8113390 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7f80b000c920 tx=0x7f80b000cde0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:36:50.575 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:50.579+0000 7f80bf7fe640 1 -- 172.21.15.52:0/3851470066 <== mon.1 v2:172.21.15.173:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f80b0007c70 con 0x7f80c811a8e0 2024-05-27T07:36:50.575 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:50.579+0000 7f80bf7fe640 1 -- 172.21.15.52:0/3851470066 <== mon.1 v2:172.21.15.173:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f80b0007e10 con 0x7f80c811a8e0 2024-05-27T07:36:50.575 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:50.579+0000 7f80bf7fe640 1 -- 172.21.15.52:0/3851470066 <== mon.1 v2:172.21.15.173:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f80b0018820 con 0x7f80c811a8e0 2024-05-27T07:36:50.576 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:50.580+0000 7f80cca2a640 1 -- 172.21.15.52:0/3851470066 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f80c8117df0 con 0x7f80c811a8e0 2024-05-27T07:36:50.576 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:50.580+0000 7f80cca2a640 1 -- 172.21.15.52:0/3851470066 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f80c81182f0 con 0x7f80c811a8e0 2024-05-27T07:36:50.581 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:50.585+0000 7f80cca2a640 1 -- 172.21.15.52:0/3851470066 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f80a8005290 con 0x7f80c811a8e0 2024-05-27T07:36:50.586 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:50.589+0000 7f80bf7fe640 1 -- 172.21.15.52:0/3851470066 <== mon.1 v2:172.21.15.173:3300/0 4 ==== mgrmap(e 26) ==== 149783+0+0 (secure 0 0 0) 0x7f80b000f450 con 0x7f80c811a8e0 2024-05-27T07:36:50.588 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:50.591+0000 7f80bf7fe640 1 --2- 172.21.15.52:0/3851470066 >> [v2:172.21.15.52:6800/1822318384,v1:172.21.15.52:6801/1822318384] conn(0x7f80b40b2b60 0x7f80b40b5070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:36:50.588 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:50.591+0000 7f80bf7fe640 1 -- 172.21.15.52:0/3851470066 <== mon.1 v2:172.21.15.173:3300/0 5 ==== osd_map(127..127 src has 1..127) ==== 9555+0+0 (secure 0 0 0) 0x7f80b00e29f0 con 0x7f80c811a8e0 2024-05-27T07:36:50.592 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:50.596+0000 7f80c659c640 1 --2- 172.21.15.52:0/3851470066 >> [v2:172.21.15.52:6800/1822318384,v1:172.21.15.52:6801/1822318384] conn(0x7f80b40b2b60 0x7f80b40b5070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:36:50.592 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:50.596+0000 7f80bf7fe640 1 -- 172.21.15.52:0/3851470066 <== mon.1 v2:172.21.15.173:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191963 (secure 0 0 0) 0x7f80b0014030 con 0x7f80c811a8e0 2024-05-27T07:36:50.601 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:50.605+0000 7f80c659c640 1 --2- 172.21.15.52:0/3851470066 >> [v2:172.21.15.52:6800/1822318384,v1:172.21.15.52:6801/1822318384] conn(0x7f80b40b2b60 0x7f80b40b5070 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7f80b8031000 tx=0x7f80b8005c70 comp rx=0 tx=0).ready entity=mgr.15333 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:36:50.696 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:36:50 smithi105 ceph-mon[35977]: pgmap v365: 129 pgs: 8 active+clean+snaptrim, 19 active+clean+snaptrim_wait, 102 active+clean; 622 MiB data, 7.3 GiB used, 1.0 TiB / 1.0 TiB avail 2024-05-27T07:36:50.696 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:36:50 smithi105 ceph-mon[35977]: from='client.? 172.21.15.52:0/2162190679' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-05-27T07:36:50.696 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:36:50 smithi105 ceph-mon[35977]: from='mgr.15333 172.21.15.52:0/416389082' entity='mgr.x' 2024-05-27T07:36:50.696 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:36:50 smithi105 ceph-mon[35977]: from='mgr.15333 172.21.15.52:0/416389082' entity='mgr.x' 2024-05-27T07:36:50.696 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:36:50 smithi105 ceph-mon[35977]: from='client.? 172.21.15.52:0/2892729639' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-05-27T07:36:50.735 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:36:50 smithi052 ceph-mon[31955]: pgmap v365: 129 pgs: 8 active+clean+snaptrim, 19 active+clean+snaptrim_wait, 102 active+clean; 622 MiB data, 7.3 GiB used, 1.0 TiB / 1.0 TiB avail 2024-05-27T07:36:50.736 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:36:50 smithi052 ceph-mon[31955]: from='client.? 172.21.15.52:0/2162190679' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-05-27T07:36:50.736 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:36:50 smithi052 ceph-mon[31955]: from='mgr.15333 172.21.15.52:0/416389082' entity='mgr.x' 2024-05-27T07:36:50.737 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:36:50 smithi052 ceph-mon[31955]: from='mgr.15333 172.21.15.52:0/416389082' entity='mgr.x' 2024-05-27T07:36:50.737 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:36:50 smithi052 ceph-mon[31955]: from='client.? 172.21.15.52:0/2892729639' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-05-27T07:36:50.858 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:36:50 smithi173 ceph-mon[37000]: pgmap v365: 129 pgs: 8 active+clean+snaptrim, 19 active+clean+snaptrim_wait, 102 active+clean; 622 MiB data, 7.3 GiB used, 1.0 TiB / 1.0 TiB avail 2024-05-27T07:36:50.858 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:36:50 smithi173 ceph-mon[37000]: from='client.? 172.21.15.52:0/2162190679' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-05-27T07:36:50.858 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:36:50 smithi173 ceph-mon[37000]: from='mgr.15333 172.21.15.52:0/416389082' entity='mgr.x' 2024-05-27T07:36:50.858 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:36:50 smithi173 ceph-mon[37000]: from='mgr.15333 172.21.15.52:0/416389082' entity='mgr.x' 2024-05-27T07:36:50.858 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:36:50 smithi173 ceph-mon[37000]: from='client.? 172.21.15.52:0/2892729639' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-05-27T07:36:50.977 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:50.981+0000 7f80cca2a640 1 -- 172.21.15.52:0/3851470066 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) -- 0x7f80a8005850 con 0x7f80c811a8e0 2024-05-27T07:36:50.979 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:50.983+0000 7f80bf7fe640 1 -- 172.21.15.52:0/3851470066 <== mon.1 v2:172.21.15.173:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v127) ==== 103+0+11 (secure 0 0 0) 0x7f80b00a9f60 con 0x7f80c811a8e0 2024-05-27T07:36:50.982 INFO:teuthology.orchestra.run.smithi052.stdout:pg_num: 32 2024-05-27T07:36:50.984 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:50.988+0000 7f80bd7fa640 1 -- 172.21.15.52:0/3851470066 >> [v2:172.21.15.52:6800/1822318384,v1:172.21.15.52:6801/1822318384] conn(0x7f80b40b2b60 msgr2=0x7f80b40b5070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:36:50.984 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:50.988+0000 7f80bd7fa640 1 --2- 172.21.15.52:0/3851470066 >> [v2:172.21.15.52:6800/1822318384,v1:172.21.15.52:6801/1822318384] conn(0x7f80b40b2b60 0x7f80b40b5070 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7f80b8031000 tx=0x7f80b8005c70 comp rx=0 tx=0).stop 2024-05-27T07:36:50.984 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:50.988+0000 7f80bd7fa640 1 -- 172.21.15.52:0/3851470066 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f80c811a8e0 msgr2=0x7f80c8113390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:36:50.984 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:50.988+0000 7f80bd7fa640 1 --2- 172.21.15.52:0/3851470066 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f80c811a8e0 0x7f80c8113390 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7f80b000c920 tx=0x7f80b000cde0 comp rx=0 tx=0).stop 2024-05-27T07:36:50.988 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:50.992+0000 7f80bd7fa640 1 -- 172.21.15.52:0/3851470066 shutdown_connections 2024-05-27T07:36:50.988 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:50.992+0000 7f80bd7fa640 1 --2- 172.21.15.52:0/3851470066 >> [v2:172.21.15.52:6800/1822318384,v1:172.21.15.52:6801/1822318384] conn(0x7f80b40b2b60 0x7f80b40b5070 unknown :-1 s=CLOSED pgs=204 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:36:50.988 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:50.992+0000 7f80bd7fa640 1 --2- 172.21.15.52:0/3851470066 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f80c811d240 0x7f80c8113980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:36:50.988 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:50.992+0000 7f80bd7fa640 1 --2- 172.21.15.52:0/3851470066 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f80c811a8e0 0x7f80c8113390 unknown :-1 s=CLOSED pgs=235 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:36:50.988 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:50.992+0000 7f80bd7fa640 1 --2- 172.21.15.52:0/3851470066 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f80c810a1d0 0x7f80c811a270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:36:50.988 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:50.992+0000 7f80bd7fa640 1 -- 172.21.15.52:0/3851470066 >> 172.21.15.52:0/3851470066 conn(0x7f80c806e8c0 msgr2=0x7f80c811acf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:36:50.993 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:50.997+0000 7f80bd7fa640 1 -- 172.21.15.52:0/3851470066 shutdown_connections 2024-05-27T07:36:50.997 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:51.002+0000 7f80bd7fa640 1 -- 172.21.15.52:0/3851470066 wait complete. 2024-05-27T07:36:51.021 DEBUG:teuthology.orchestra.run.smithi052:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mgr dump --format=json-pretty 2024-05-27T07:36:51.234 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:51.237+0000 7fee5759e640 1 -- 172.21.15.52:0/3287295113 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fee5811d150 msgr2=0x7fee5811f570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:36:51.234 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:51.237+0000 7fee5759e640 1 --2- 172.21.15.52:0/3287295113 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fee5811d150 0x7fee5811f570 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 crypto rx=0x7fee4c009930 tx=0x7fee4c02a480 comp rx=0 tx=0).stop 2024-05-27T07:36:51.237 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:51.241+0000 7fee5759e640 1 -- 172.21.15.52:0/3287295113 shutdown_connections 2024-05-27T07:36:51.237 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:51.241+0000 7fee5759e640 1 --2- 172.21.15.52:0/3287295113 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fee5811d150 0x7fee5811f570 unknown :-1 s=CLOSED pgs=282 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:36:51.237 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:51.241+0000 7fee5759e640 1 --2- 172.21.15.52:0/3287295113 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fee5811a7f0 0x7fee5811cc10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:36:51.237 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:51.241+0000 7fee5759e640 1 --2- 172.21.15.52:0/3287295113 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fee5806ecf0 0x7fee5806f100 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:36:51.237 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:51.241+0000 7fee5759e640 1 -- 172.21.15.52:0/3287295113 >> 172.21.15.52:0/3287295113 conn(0x7fee58109ea0 msgr2=0x7fee5810c2c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:36:51.247 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:51.251+0000 7fee5759e640 1 -- 172.21.15.52:0/3287295113 shutdown_connections 2024-05-27T07:36:51.250 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:51.253+0000 7fee5759e640 1 -- 172.21.15.52:0/3287295113 wait complete. 2024-05-27T07:36:51.250 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:51.254+0000 7fee5759e640 1 Processor -- start 2024-05-27T07:36:51.256 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:51.257+0000 7fee5759e640 1 -- start start 2024-05-27T07:36:51.256 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:51.257+0000 7fee5759e640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fee5806ecf0 0x7fee5811a200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:36:51.256 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:51.257+0000 7fee5759e640 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fee5811a7f0 0x7fee58113320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:36:51.256 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:51.257+0000 7fee5759e640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fee5811d150 0x7fee58113980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:36:51.256 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:51.257+0000 7fee5759e640 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x7fee58122ba0 con 0x7fee5811a7f0 2024-05-27T07:36:51.256 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:51.257+0000 7fee5759e640 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7fee58122a20 con 0x7fee5806ecf0 2024-05-27T07:36:51.256 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:51.257+0000 7fee5759e640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7fee58122d20 con 0x7fee5811d150 2024-05-27T07:36:51.258 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:51.261+0000 7fee5659c640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fee5806ecf0 0x7fee5811a200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:36:51.259 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:51.261+0000 7fee56d9d640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fee5811d150 0x7fee58113980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:36:51.259 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:51.261+0000 7fee56d9d640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fee5811d150 0x7fee58113980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.173:3300/0 says I am v2:172.21.15.52:51374/0 (socket says 172.21.15.52:51374) 2024-05-27T07:36:51.259 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:51.261+0000 7fee56d9d640 1 -- 172.21.15.52:0/923522457 learned_addr learned my addr 172.21.15.52:0/923522457 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-05-27T07:36:51.259 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:51.261+0000 7fee55d9b640 1 --2- 172.21.15.52:0/923522457 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fee5811a7f0 0x7fee58113320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:36:51.259 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:51.261+0000 7fee56d9d640 1 -- 172.21.15.52:0/923522457 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fee5806ecf0 msgr2=0x7fee5811a200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:36:51.259 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:51.261+0000 7fee56d9d640 1 --2- 172.21.15.52:0/923522457 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fee5806ecf0 0x7fee5811a200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:36:51.259 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:51.261+0000 7fee56d9d640 1 -- 172.21.15.52:0/923522457 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fee5811a7f0 msgr2=0x7fee58113320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:36:51.259 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:51.261+0000 7fee56d9d640 1 --2- 172.21.15.52:0/923522457 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fee5811a7f0 0x7fee58113320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:36:51.259 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:51.261+0000 7fee56d9d640 1 -- 172.21.15.52:0/923522457 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fee58117d20 con 0x7fee5811d150 2024-05-27T07:36:51.259 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:51.262+0000 7fee56d9d640 1 --2- 172.21.15.52:0/923522457 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fee5811d150 0x7fee58113980 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7fee4c005ab0 tx=0x7fee4c003820 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:36:51.259 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:51.263+0000 7fee477fe640 1 -- 172.21.15.52:0/923522457 <== mon.1 v2:172.21.15.173:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fee4c03f030 con 0x7fee5811d150 2024-05-27T07:36:51.260 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:51.263+0000 7fee477fe640 1 -- 172.21.15.52:0/923522457 <== mon.1 v2:172.21.15.173:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fee4c003ba0 con 0x7fee5811d150 2024-05-27T07:36:51.260 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:51.264+0000 7fee477fe640 1 -- 172.21.15.52:0/923522457 <== mon.1 v2:172.21.15.173:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fee4c035070 con 0x7fee5811d150 2024-05-27T07:36:51.261 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:51.265+0000 7fee5759e640 1 -- 172.21.15.52:0/923522457 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fee58117fb0 con 0x7fee5811d150 2024-05-27T07:36:51.261 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:51.265+0000 7fee5759e640 1 -- 172.21.15.52:0/923522457 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fee58118420 con 0x7fee5811d150 2024-05-27T07:36:51.262 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:51.266+0000 7fee5759e640 1 -- 172.21.15.52:0/923522457 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fee58070010 con 0x7fee5811d150 2024-05-27T07:36:51.269 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:51.272+0000 7fee477fe640 1 -- 172.21.15.52:0/923522457 <== mon.1 v2:172.21.15.173:3300/0 4 ==== mgrmap(e 26) ==== 149783+0+0 (secure 0 0 0) 0x7fee4c042030 con 0x7fee5811d150 2024-05-27T07:36:51.269 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:51.273+0000 7fee477fe640 1 --2- 172.21.15.52:0/923522457 >> [v2:172.21.15.52:6800/1822318384,v1:172.21.15.52:6801/1822318384] conn(0x7fee400b2970 0x7fee400b4e80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:36:51.270 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:51.273+0000 7fee477fe640 1 -- 172.21.15.52:0/923522457 <== mon.1 v2:172.21.15.173:3300/0 5 ==== osd_map(127..127 src has 1..127) ==== 9555+0+0 (secure 0 0 0) 0x7fee4c103300 con 0x7fee5811d150 2024-05-27T07:36:51.270 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:51.274+0000 7fee5659c640 1 --2- 172.21.15.52:0/923522457 >> [v2:172.21.15.52:6800/1822318384,v1:172.21.15.52:6801/1822318384] conn(0x7fee400b2970 0x7fee400b4e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:36:51.270 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:51.274+0000 7fee477fe640 1 -- 172.21.15.52:0/923522457 <== mon.1 v2:172.21.15.173:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191963 (secure 0 0 0) 0x7fee4c049070 con 0x7fee5811d150 2024-05-27T07:36:51.276 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:51.280+0000 7fee5659c640 1 --2- 172.21.15.52:0/923522457 >> [v2:172.21.15.52:6800/1822318384,v1:172.21.15.52:6801/1822318384] conn(0x7fee400b2970 0x7fee400b4e80 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7fee58114b70 tx=0x7fee50007310 comp rx=0 tx=0).ready entity=mgr.15333 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:36:51.736 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:36:51 smithi052 ceph-mon[31955]: from='client.? 172.21.15.52:0/3851470066' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-05-27T07:36:51.736 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:36:51 smithi052 ceph-mon[31955]: pgmap v366: 129 pgs: 8 active+clean+snaptrim, 19 active+clean+snaptrim_wait, 102 active+clean; 622 MiB data, 7.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-05-27T07:36:51.736 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:36:51 smithi052 ceph-mon[31955]: from='mgr.15333 172.21.15.52:0/416389082' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-05-27T07:36:51.736 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:36:51 smithi052 ceph-mon[31955]: from='mgr.15333 172.21.15.52:0/416389082' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-05-27T07:36:51.736 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:36:51 smithi052 ceph-mon[31955]: from='mgr.15333 172.21.15.52:0/416389082' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-05-27T07:36:51.737 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:36:51 smithi052 ceph-mon[31955]: from='mgr.15333 172.21.15.52:0/416389082' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-05-27T07:36:51.737 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:36:51 smithi052 ceph-mon[31955]: Adjusting osd_memory_target on smithi052 to 177.2M 2024-05-27T07:36:51.737 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:36:51 smithi052 ceph-mon[31955]: Unable to set osd_memory_target on smithi052 to 185879142: error parsing value: Value '185879142' is below minimum 939524096 2024-05-27T07:36:51.737 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:36:51 smithi052 ceph-mon[31955]: from='mgr.15333 172.21.15.52:0/416389082' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T07:36:51.737 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:36:51 smithi052 ceph-mon[31955]: from='mgr.15333 172.21.15.52:0/416389082' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-27T07:36:51.737 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:36:51 smithi052 ceph-mon[31955]: pgmap v367: 129 pgs: 8 active+clean+snaptrim, 19 active+clean+snaptrim_wait, 102 active+clean; 622 MiB data, 7.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-05-27T07:36:51.737 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:36:51 smithi052 ceph-mon[31955]: from='mgr.15333 172.21.15.52:0/416389082' entity='mgr.x' 2024-05-27T07:36:51.744 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:51.748+0000 7fee5759e640 1 -- 172.21.15.52:0/923522457 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_command({"prefix": "mgr dump", "format": "json-pretty"} v 0) -- 0x7fee58069d00 con 0x7fee5811d150 2024-05-27T07:36:51.758 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:51.762+0000 7fee477fe640 1 -- 172.21.15.52:0/923522457 <== mon.1 v2:172.21.15.173:3300/0 7 ==== mon_command_ack([{"prefix": "mgr dump", "format": "json-pretty"}]=0 v26) ==== 81+0+816173 (secure 0 0 0) 0x7fee4c0ca870 con 0x7fee5811d150 2024-05-27T07:36:51.769 INFO:teuthology.orchestra.run.smithi052.stdout: 2024-05-27T07:36:51.769 INFO:teuthology.orchestra.run.smithi052.stdout:{ 2024-05-27T07:36:51.769 INFO:teuthology.orchestra.run.smithi052.stdout: "epoch": 26, 2024-05-27T07:36:51.769 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 0, 2024-05-27T07:36:51.769 INFO:teuthology.orchestra.run.smithi052.stdout: "active_gid": 15333, 2024-05-27T07:36:51.770 INFO:teuthology.orchestra.run.smithi052.stdout: "active_name": "x", 2024-05-27T07:36:51.770 INFO:teuthology.orchestra.run.smithi052.stdout: "active_addrs": { 2024-05-27T07:36:51.770 INFO:teuthology.orchestra.run.smithi052.stdout: "addrvec": [ 2024-05-27T07:36:51.770 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-05-27T07:36:51.770 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "v2", 2024-05-27T07:36:51.770 INFO:teuthology.orchestra.run.smithi052.stdout: "addr": "172.21.15.52:6800", 2024-05-27T07:36:51.770 INFO:teuthology.orchestra.run.smithi052.stdout: "nonce": 1822318384 2024-05-27T07:36:51.770 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-05-27T07:36:51.770 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-05-27T07:36:51.770 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "v1", 2024-05-27T07:36:51.770 INFO:teuthology.orchestra.run.smithi052.stdout: "addr": "172.21.15.52:6801", 2024-05-27T07:36:51.770 INFO:teuthology.orchestra.run.smithi052.stdout: "nonce": 1822318384 2024-05-27T07:36:51.770 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-05-27T07:36:51.771 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2024-05-27T07:36:51.771 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-05-27T07:36:51.771 INFO:teuthology.orchestra.run.smithi052.stdout: "active_addr": "172.21.15.52:6801/1822318384", 2024-05-27T07:36:51.771 INFO:teuthology.orchestra.run.smithi052.stdout: "active_change": "2024-05-27T07:25:56.925417+0000", 2024-05-27T07:36:51.771 INFO:teuthology.orchestra.run.smithi052.stdout: "active_mgr_features": 4540701547738038271, 2024-05-27T07:36:51.771 INFO:teuthology.orchestra.run.smithi052.stdout: "available": true, 2024-05-27T07:36:51.771 INFO:teuthology.orchestra.run.smithi052.stdout: "standbys": [ 2024-05-27T07:36:51.771 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-05-27T07:36:51.771 INFO:teuthology.orchestra.run.smithi052.stdout: "gid": 25048, 2024-05-27T07:36:51.771 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "z", 2024-05-27T07:36:51.771 INFO:teuthology.orchestra.run.smithi052.stdout: "mgr_features": 4540701547738038271, 2024-05-27T07:36:51.771 INFO:teuthology.orchestra.run.smithi052.stdout: "available_modules": [ 2024-05-27T07:36:51.772 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-05-27T07:36:51.772 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "alerts", 2024-05-27T07:36:51.772 INFO:teuthology.orchestra.run.smithi052.stdout: "can_run": true, 2024-05-27T07:36:51.772 INFO:teuthology.orchestra.run.smithi052.stdout: "error_string": "", 2024-05-27T07:36:51.772 INFO:teuthology.orchestra.run.smithi052.stdout: "module_options": { 2024-05-27T07:36:51.772 INFO:teuthology.orchestra.run.smithi052.stdout: "interval": { 2024-05-27T07:36:51.772 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "interval", 2024-05-27T07:36:51.772 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "secs", 2024-05-27T07:36:51.772 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-05-27T07:36:51.772 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 1, 2024-05-27T07:36:51.773 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "60", 2024-05-27T07:36:51.773 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-05-27T07:36:51.773 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-05-27T07:36:51.773 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-05-27T07:36:51.773 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "How frequently to reexamine health status", 2024-05-27T07:36:51.773 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-05-27T07:36:51.773 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-05-27T07:36:51.773 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-05-27T07:36:51.773 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-05-27T07:36:51.773 INFO:teuthology.orchestra.run.smithi052.stdout: "log_level": { 2024-05-27T07:36:51.773 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "log_level", 2024-05-27T07:36:51.774 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "str", 2024-05-27T07:36:51.774 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-05-27T07:36:51.774 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 1, 2024-05-27T07:36:51.774 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "", 2024-05-27T07:36:51.774 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-05-27T07:36:51.774 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-05-27T07:36:51.774 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [ 2024-05-27T07:36:51.774 INFO:teuthology.orchestra.run.smithi052.stdout: "", 2024-05-27T07:36:51.774 INFO:teuthology.orchestra.run.smithi052.stdout: "critical", 2024-05-27T07:36:51.774 INFO:teuthology.orchestra.run.smithi052.stdout: "debug", 2024-05-27T07:36:51.774 INFO:teuthology.orchestra.run.smithi052.stdout: "error", 2024-05-27T07:36:51.774 INFO:teuthology.orchestra.run.smithi052.stdout: "info", 2024-05-27T07:36:51.775 INFO:teuthology.orchestra.run.smithi052.stdout: "warning" 2024-05-27T07:36:51.775 INFO:teuthology.orchestra.run.smithi052.stdout: ], 2024-05-27T07:36:51.775 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-05-27T07:36:51.775 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-05-27T07:36:51.775 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-05-27T07:36:51.775 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-05-27T07:36:51.775 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-05-27T07:36:51.775 INFO:teuthology.orchestra.run.smithi052.stdout: "log_to_cluster": { 2024-05-27T07:36:51.775 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "log_to_cluster", 2024-05-27T07:36:51.775 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "bool", 2024-05-27T07:36:51.775 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-05-27T07:36:51.775 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 1, 2024-05-27T07:36:51.776 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "False", 2024-05-27T07:36:51.776 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-05-27T07:36:51.776 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-05-27T07:36:51.776 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-05-27T07:36:51.776 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-05-27T07:36:51.776 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-05-27T07:36:51.776 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-05-27T07:36:51.776 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-05-27T07:36:51.776 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-05-27T07:36:51.776 INFO:teuthology.orchestra.run.smithi052.stdout: "log_to_cluster_level": { 2024-05-27T07:36:51.777 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "log_to_cluster_level", 2024-05-27T07:36:51.777 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "str", 2024-05-27T07:36:51.777 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-05-27T07:36:51.777 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 1, 2024-05-27T07:36:51.777 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "info", 2024-05-27T07:36:51.777 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-05-27T07:36:51.777 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-05-27T07:36:51.777 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [ 2024-05-27T07:36:51.777 INFO:teuthology.orchestra.run.smithi052.stdout: "", 2024-05-27T07:36:51.777 INFO:teuthology.orchestra.run.smithi052.stdout: "critical", 2024-05-27T07:36:51.777 INFO:teuthology.orchestra.run.smithi052.stdout: "debug", 2024-05-27T07:36:51.777 INFO:teuthology.orchestra.run.smithi052.stdout: "error", 2024-05-27T07:36:51.777 INFO:teuthology.orchestra.run.smithi052.stdout: "info", 2024-05-27T07:36:51.778 INFO:teuthology.orchestra.run.smithi052.stdout: "warning" 2024-05-27T07:36:51.778 INFO:teuthology.orchestra.run.smithi052.stdout: ], 2024-05-27T07:36:51.778 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-05-27T07:36:51.778 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-05-27T07:36:51.778 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-05-27T07:36:51.778 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-05-27T07:36:51.778 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-05-27T07:36:51.778 INFO:teuthology.orchestra.run.smithi052.stdout: "log_to_file": { 2024-05-27T07:36:51.778 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "log_to_file", 2024-05-27T07:36:51.778 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "bool", 2024-05-27T07:36:51.778 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-05-27T07:36:51.778 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 1, 2024-05-27T07:36:51.778 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "False", 2024-05-27T07:36:51.779 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-05-27T07:36:51.779 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-05-27T07:36:51.779 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-05-27T07:36:51.779 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-05-27T07:36:51.779 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-05-27T07:36:51.779 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-05-27T07:36:51.779 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-05-27T07:36:51.779 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-05-27T07:36:51.779 INFO:teuthology.orchestra.run.smithi052.stdout: "smtp_destination": { 2024-05-27T07:36:51.779 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "smtp_destination", 2024-05-27T07:36:51.779 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "str", 2024-05-27T07:36:51.779 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-05-27T07:36:51.779 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 1, 2024-05-27T07:36:51.780 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "", 2024-05-27T07:36:51.780 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-05-27T07:36:51.780 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-05-27T07:36:51.780 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-05-27T07:36:51.780 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "Email address to send alerts to", 2024-05-27T07:36:51.780 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-05-27T07:36:51.780 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-05-27T07:36:51.780 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-05-27T07:36:51.780 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-05-27T07:36:51.780 INFO:teuthology.orchestra.run.smithi052.stdout: "smtp_from_name": { 2024-05-27T07:36:51.780 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "smtp_from_name", 2024-05-27T07:36:51.780 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "str", 2024-05-27T07:36:51.780 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-05-27T07:36:51.781 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 1, 2024-05-27T07:36:51.781 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "Ceph", 2024-05-27T07:36:51.781 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-05-27T07:36:51.781 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-05-27T07:36:51.781 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-05-27T07:36:51.781 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "Email From: name", 2024-05-27T07:36:51.781 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-05-27T07:36:51.781 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-05-27T07:36:51.781 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-05-27T07:36:51.781 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-05-27T07:36:51.781 INFO:teuthology.orchestra.run.smithi052.stdout: "smtp_host": { 2024-05-27T07:36:51.781 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "smtp_host", 2024-05-27T07:36:51.781 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "str", 2024-05-27T07:36:51.782 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-05-27T07:36:51.782 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 1, 2024-05-27T07:36:51.782 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "", 2024-05-27T07:36:51.782 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-05-27T07:36:51.782 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-05-27T07:36:51.782 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-05-27T07:36:51.782 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "SMTP server", 2024-05-27T07:36:51.782 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-05-27T07:36:51.782 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-05-27T07:36:51.782 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-05-27T07:36:51.782 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-05-27T07:36:51.782 INFO:teuthology.orchestra.run.smithi052.stdout: "smtp_password": { 2024-05-27T07:36:51.782 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "smtp_password", 2024-05-27T07:36:51.783 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "str", 2024-05-27T07:36:51.783 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-05-27T07:36:51.783 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 1, 2024-05-27T07:36:51.783 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "", 2024-05-27T07:36:51.783 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-05-27T07:36:51.783 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-05-27T07:36:51.783 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-05-27T07:36:51.783 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "Password to authenticate with", 2024-05-27T07:36:51.783 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-05-27T07:36:51.783 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-05-27T07:36:51.783 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-05-27T07:36:51.783 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-05-27T07:36:51.783 INFO:teuthology.orchestra.run.smithi052.stdout: "smtp_port": { 2024-05-27T07:36:51.784 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "smtp_port", 2024-05-27T07:36:51.784 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "int", 2024-05-27T07:36:51.784 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-05-27T07:36:51.784 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 1, 2024-05-27T07:36:51.784 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "465", 2024-05-27T07:36:51.784 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-05-27T07:36:51.784 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-05-27T07:36:51.784 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-05-27T07:36:51.784 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "SMTP port", 2024-05-27T07:36:51.784 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-05-27T07:36:51.784 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-05-27T07:36:51.784 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-05-27T07:36:51.784 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-05-27T07:36:51.785 INFO:teuthology.orchestra.run.smithi052.stdout: "smtp_sender": { 2024-05-27T07:36:51.785 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "smtp_sender", 2024-05-27T07:36:51.785 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "str", 2024-05-27T07:36:51.785 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-05-27T07:36:51.785 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 1, 2024-05-27T07:36:51.785 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "", 2024-05-27T07:36:51.785 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-05-27T07:36:51.785 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-05-27T07:36:51.785 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-05-27T07:36:51.785 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "SMTP envelope sender", 2024-05-27T07:36:51.785 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-05-27T07:36:51.785 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-05-27T07:36:51.785 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-05-27T07:36:51.786 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-05-27T07:36:51.786 INFO:teuthology.orchestra.run.smithi052.stdout: "smtp_ssl": { 2024-05-27T07:36:51.786 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "smtp_ssl", 2024-05-27T07:36:51.786 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "bool", 2024-05-27T07:36:51.786 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-05-27T07:36:51.786 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 1, 2024-05-27T07:36:51.786 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "True", 2024-05-27T07:36:51.786 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-05-27T07:36:51.786 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-05-27T07:36:51.786 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-05-27T07:36:51.786 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "Use SSL to connect to SMTP server", 2024-05-27T07:36:51.786 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-05-27T07:36:51.787 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-05-27T07:36:51.787 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-05-27T07:36:51.787 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-05-27T07:36:51.787 INFO:teuthology.orchestra.run.smithi052.stdout: "smtp_user": { 2024-05-27T07:36:51.787 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "smtp_user", 2024-05-27T07:36:51.787 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "str", 2024-05-27T07:36:51.787 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-05-27T07:36:51.787 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 1, 2024-05-27T07:36:51.787 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "", 2024-05-27T07:36:51.787 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-05-27T07:36:51.787 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-05-27T07:36:51.787 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-05-27T07:36:51.787 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "User to authenticate as", 2024-05-27T07:36:51.788 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-05-27T07:36:51.788 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-05-27T07:36:51.788 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-05-27T07:36:51.788 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-05-27T07:36:51.788 INFO:teuthology.orchestra.run.smithi052.stdout: "sqlite3_killpoint": { 2024-05-27T07:36:51.788 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "sqlite3_killpoint", 2024-05-27T07:36:51.788 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "int", 2024-05-27T07:36:51.788 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "dev", 2024-05-27T07:36:51.788 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 1, 2024-05-27T07:36:51.788 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "0", 2024-05-27T07:36:51.788 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-05-27T07:36:51.788 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-05-27T07:36:51.788 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-05-27T07:36:51.789 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-05-27T07:36:51.789 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-05-27T07:36:51.789 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-05-27T07:36:51.789 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-05-27T07:36:51.789 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-05-27T07:36:51.789 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-05-27T07:36:51.789 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-05-27T07:36:51.789 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-05-27T07:36:51.789 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "balancer", 2024-05-27T07:36:51.789 INFO:teuthology.orchestra.run.smithi052.stdout: "can_run": true, 2024-05-27T07:36:51.789 INFO:teuthology.orchestra.run.smithi052.stdout: "error_string": "", 2024-05-27T07:36:51.789 INFO:teuthology.orchestra.run.smithi052.stdout: "module_options": { 2024-05-27T07:36:51.789 INFO:teuthology.orchestra.run.smithi052.stdout: "active": { 2024-05-27T07:36:51.790 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "active", 2024-05-27T07:36:51.790 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "bool", 2024-05-27T07:36:51.790 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-05-27T07:36:51.790 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 1, 2024-05-27T07:36:51.790 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "True", 2024-05-27T07:36:51.790 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-05-27T07:36:51.790 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-05-27T07:36:51.790 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-05-27T07:36:51.790 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "automatically balance PGs across cluster", 2024-05-27T07:36:51.790 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-05-27T07:36:51.790 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-05-27T07:36:51.790 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-05-27T07:36:51.791 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-05-27T07:36:51.791 INFO:teuthology.orchestra.run.smithi052.stdout: "begin_time": { 2024-05-27T07:36:51.791 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "begin_time", 2024-05-27T07:36:51.791 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "str", 2024-05-27T07:36:51.791 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-05-27T07:36:51.791 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 1, 2024-05-27T07:36:51.791 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "0000", 2024-05-27T07:36:51.791 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-05-27T07:36:51.791 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-05-27T07:36:51.791 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-05-27T07:36:51.791 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "beginning time of day to automatically balance", 2024-05-27T07:36:51.791 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "This is a time of day in the format HHMM.", 2024-05-27T07:36:51.792 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-05-27T07:36:51.792 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-05-27T07:36:51.792 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-05-27T07:36:51.792 INFO:teuthology.orchestra.run.smithi052.stdout: "begin_weekday": { 2024-05-27T07:36:51.792 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "begin_weekday", 2024-05-27T07:36:51.792 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "uint", 2024-05-27T07:36:51.792 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-05-27T07:36:51.792 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 1, 2024-05-27T07:36:51.792 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "0", 2024-05-27T07:36:51.792 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "0", 2024-05-27T07:36:51.792 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "6", 2024-05-27T07:36:51.792 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-05-27T07:36:51.792 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "Restrict automatic balancing to this day of the week or later", 2024-05-27T07:36:51.793 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "0 = Sunday, 1 = Monday, etc.", 2024-05-27T07:36:51.793 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-05-27T07:36:51.793 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-05-27T07:36:51.793 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-05-27T07:36:51.793 INFO:teuthology.orchestra.run.smithi052.stdout: "crush_compat_max_iterations": { 2024-05-27T07:36:51.793 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "crush_compat_max_iterations", 2024-05-27T07:36:51.793 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "uint", 2024-05-27T07:36:51.793 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-05-27T07:36:51.793 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 1, 2024-05-27T07:36:51.793 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "25", 2024-05-27T07:36:51.793 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "1", 2024-05-27T07:36:51.793 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "250", 2024-05-27T07:36:51.793 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-05-27T07:36:51.794 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "maximum number of iterations to attempt optimization", 2024-05-27T07:36:51.794 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-05-27T07:36:51.794 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-05-27T07:36:51.794 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-05-27T07:36:51.794 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-05-27T07:36:51.794 INFO:teuthology.orchestra.run.smithi052.stdout: "crush_compat_metrics": { 2024-05-27T07:36:51.794 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "crush_compat_metrics", 2024-05-27T07:36:51.794 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "str", 2024-05-27T07:36:51.794 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-05-27T07:36:51.794 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 1, 2024-05-27T07:36:51.794 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "pgs,objects,bytes", 2024-05-27T07:36:51.794 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-05-27T07:36:51.794 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-05-27T07:36:51.795 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-05-27T07:36:51.795 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "metrics with which to calculate OSD utilization", 2024-05-27T07:36:51.795 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.", 2024-05-27T07:36:51.795 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-05-27T07:36:51.795 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-05-27T07:36:51.795 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-05-27T07:36:51.795 INFO:teuthology.orchestra.run.smithi052.stdout: "crush_compat_step": { 2024-05-27T07:36:51.795 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "crush_compat_step", 2024-05-27T07:36:51.795 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "float", 2024-05-27T07:36:51.795 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-05-27T07:36:51.795 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 1, 2024-05-27T07:36:51.795 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "0.5", 2024-05-27T07:36:51.795 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "0.001", 2024-05-27T07:36:51.796 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "0.999", 2024-05-27T07:36:51.796 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-05-27T07:36:51.796 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "aggressiveness of optimization", 2024-05-27T07:36:51.796 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": ".99 is very aggressive, .01 is less aggressive", 2024-05-27T07:36:51.796 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-05-27T07:36:51.796 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-05-27T07:36:51.796 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-05-27T07:36:51.796 INFO:teuthology.orchestra.run.smithi052.stdout: "end_time": { 2024-05-27T07:36:51.796 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "end_time", 2024-05-27T07:36:51.796 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "str", 2024-05-27T07:36:51.796 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-05-27T07:36:51.796 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 1, 2024-05-27T07:36:51.796 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "2359", 2024-05-27T07:36:51.797 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-05-27T07:36:51.797 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-05-27T07:36:51.797 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-05-27T07:36:51.797 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "ending time of day to automatically balance", 2024-05-27T07:36:51.797 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "This is a time of day in the format HHMM.", 2024-05-27T07:36:51.797 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-05-27T07:36:51.797 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-05-27T07:36:51.797 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-05-27T07:36:51.797 INFO:teuthology.orchestra.run.smithi052.stdout: "end_weekday": { 2024-05-27T07:36:51.797 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "end_weekday", 2024-05-27T07:36:51.797 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "uint", 2024-05-27T07:36:51.797 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-05-27T07:36:51.797 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 1, 2024-05-27T07:36:51.798 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "0", 2024-05-27T07:36:51.798 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "0", 2024-05-27T07:36:51.798 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "6", 2024-05-27T07:36:51.798 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-05-27T07:36:51.798 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "Restrict automatic balancing to days of the week earlier than this", 2024-05-27T07:36:51.798 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "0 = Sunday, 1 = Monday, etc.", 2024-05-27T07:36:51.798 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-05-27T07:36:51.798 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-05-27T07:36:51.798 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-05-27T07:36:51.798 INFO:teuthology.orchestra.run.smithi052.stdout: "log_level": { 2024-05-27T07:36:51.798 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "log_level", 2024-05-27T07:36:51.798 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "str", 2024-05-27T07:36:51.798 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-05-27T07:36:51.799 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 1, 2024-05-27T07:36:51.799 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "", 2024-05-27T07:36:51.799 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-05-27T07:36:51.799 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-05-27T07:36:51.799 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [ 2024-05-27T07:36:51.799 INFO:teuthology.orchestra.run.smithi052.stdout: "", 2024-05-27T07:36:51.799 INFO:teuthology.orchestra.run.smithi052.stdout: "critical", 2024-05-27T07:36:51.799 INFO:teuthology.orchestra.run.smithi052.stdout: "debug", 2024-05-27T07:36:51.799 INFO:teuthology.orchestra.run.smithi052.stdout: "error", 2024-05-27T07:36:51.799 INFO:teuthology.orchestra.run.smithi052.stdout: "info", 2024-05-27T07:36:51.799 INFO:teuthology.orchestra.run.smithi052.stdout: "warning" 2024-05-27T07:36:51.799 INFO:teuthology.orchestra.run.smithi052.stdout: ], 2024-05-27T07:36:51.800 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-05-27T07:36:51.800 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-05-27T07:36:51.800 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-05-27T07:36:51.800 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-05-27T07:36:51.800 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-05-27T07:36:51.800 INFO:teuthology.orchestra.run.smithi052.stdout: "log_to_cluster": { 2024-05-27T07:36:51.800 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "log_to_cluster", 2024-05-27T07:36:51.800 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "bool", 2024-05-27T07:36:51.800 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-05-27T07:36:51.800 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 1, 2024-05-27T07:36:51.800 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "False", 2024-05-27T07:36:51.800 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-05-27T07:36:51.800 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-05-27T07:36:51.801 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-05-27T07:36:51.801 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-05-27T07:36:51.801 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-05-27T07:36:51.801 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-05-27T07:36:51.801 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-05-27T07:36:51.801 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-05-27T07:36:51.801 INFO:teuthology.orchestra.run.smithi052.stdout: "log_to_cluster_level": { 2024-05-27T07:36:51.801 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "log_to_cluster_level", 2024-05-27T07:36:51.801 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "str", 2024-05-27T07:36:51.801 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-05-27T07:36:51.801 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 1, 2024-05-27T07:36:51.801 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "info", 2024-05-27T07:36:51.801 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-05-27T07:36:51.802 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-05-27T07:36:51.802 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [ 2024-05-27T07:36:51.802 INFO:teuthology.orchestra.run.smithi052.stdout: "", 2024-05-27T07:36:51.802 INFO:teuthology.orchestra.run.smithi052.stdout: "critical", 2024-05-27T07:36:51.802 INFO:teuthology.orchestra.run.smithi052.stdout: "debug", 2024-05-27T07:36:51.802 INFO:teuthology.orchestra.run.smithi052.stdout: "error", 2024-05-27T07:36:51.802 INFO:teuthology.orchestra.run.smithi052.stdout: "info", 2024-05-27T07:36:51.802 INFO:teuthology.orchestra.run.smithi052.stdout: "warning" 2024-05-27T07:36:51.802 INFO:teuthology.orchestra.run.smithi052.stdout: ], 2024-05-27T07:36:51.802 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-05-27T07:36:51.802 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-05-27T07:36:51.802 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-05-27T07:36:51.803 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-05-27T07:36:51.803 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-05-27T07:36:51.803 INFO:teuthology.orchestra.run.smithi052.stdout: "log_to_file": { 2024-05-27T07:36:51.803 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "log_to_file", 2024-05-27T07:36:51.803 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "bool", 2024-05-27T07:36:51.803 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-05-27T07:36:51.803 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 1, 2024-05-27T07:36:51.803 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "False", 2024-05-27T07:36:51.803 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-05-27T07:36:51.803 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-05-27T07:36:51.803 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-05-27T07:36:51.803 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-05-27T07:36:51.803 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-05-27T07:36:51.804 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-05-27T07:36:51.804 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-05-27T07:36:51.804 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-05-27T07:36:51.804 INFO:teuthology.orchestra.run.smithi052.stdout: "min_score": { 2024-05-27T07:36:51.804 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "min_score", 2024-05-27T07:36:51.804 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "float", 2024-05-27T07:36:51.804 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-05-27T07:36:51.804 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 1, 2024-05-27T07:36:51.804 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "0", 2024-05-27T07:36:51.804 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-05-27T07:36:51.804 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-05-27T07:36:51.804 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-05-27T07:36:51.804 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "minimum score, below which no optimization is attempted", 2024-05-27T07:36:51.805 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-05-27T07:36:51.805 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-05-27T07:36:51.805 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-05-27T07:36:51.805 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-05-27T07:36:51.805 INFO:teuthology.orchestra.run.smithi052.stdout: "mode": { 2024-05-27T07:36:51.805 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "mode", 2024-05-27T07:36:51.805 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "str", 2024-05-27T07:36:51.805 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-05-27T07:36:51.805 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 1, 2024-05-27T07:36:51.805 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "upmap", 2024-05-27T07:36:51.805 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-05-27T07:36:51.805 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-05-27T07:36:51.806 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [ 2024-05-27T07:36:51.806 INFO:teuthology.orchestra.run.smithi052.stdout: "crush-compat", 2024-05-27T07:36:51.806 INFO:teuthology.orchestra.run.smithi052.stdout: "none", 2024-05-27T07:36:51.806 INFO:teuthology.orchestra.run.smithi052.stdout: "read", 2024-05-27T07:36:51.806 INFO:teuthology.orchestra.run.smithi052.stdout: "upmap", 2024-05-27T07:36:51.806 INFO:teuthology.orchestra.run.smithi052.stdout: "upmap-read" 2024-05-27T07:36:51.806 INFO:teuthology.orchestra.run.smithi052.stdout: ], 2024-05-27T07:36:51.806 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "Balancer mode", 2024-05-27T07:36:51.806 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-05-27T07:36:51.806 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-05-27T07:36:51.806 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-05-27T07:36:51.806 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-05-27T07:36:51.806 INFO:teuthology.orchestra.run.smithi052.stdout: "pool_ids": { 2024-05-27T07:36:51.807 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "pool_ids", 2024-05-27T07:36:51.807 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "str", 2024-05-27T07:36:51.807 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-05-27T07:36:51.807 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 1, 2024-05-27T07:36:51.807 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "", 2024-05-27T07:36:51.807 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-05-27T07:36:51.807 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-05-27T07:36:51.807 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-05-27T07:36:51.807 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "pools which the automatic balancing will be limited to", 2024-05-27T07:36:51.807 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-05-27T07:36:51.807 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-05-27T07:36:51.807 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-05-27T07:36:51.807 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-05-27T07:36:51.808 INFO:teuthology.orchestra.run.smithi052.stdout: "sleep_interval": { 2024-05-27T07:36:51.808 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "sleep_interval", 2024-05-27T07:36:51.808 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "secs", 2024-05-27T07:36:51.808 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-05-27T07:36:51.808 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 1, 2024-05-27T07:36:51.808 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "60", 2024-05-27T07:36:51.808 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-05-27T07:36:51.808 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-05-27T07:36:51.808 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-05-27T07:36:51.808 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "how frequently to wake up and attempt optimization", 2024-05-27T07:36:51.808 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-05-27T07:36:51.808 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-05-27T07:36:51.808 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-05-27T07:36:51.809 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-05-27T07:36:51.809 INFO:teuthology.orchestra.run.smithi052.stdout: "sqlite3_killpoint": { 2024-05-27T07:36:51.809 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "sqlite3_killpoint", 2024-05-27T07:36:51.809 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "int", 2024-05-27T07:36:51.809 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "dev", 2024-05-27T07:36:51.809 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 1, 2024-05-27T07:36:51.809 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "0", 2024-05-27T07:36:51.809 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-05-27T07:36:51.809 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-05-27T07:36:51.809 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-05-27T07:36:51.809 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-05-27T07:36:51.809 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-05-27T07:36:51.809 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-05-27T07:36:51.809 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-05-27T07:36:51.810 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-05-27T07:36:51.810 INFO:teuthology.orchestra.run.smithi052.stdout: "upmap_max_deviation": { 2024-05-27T07:36:51.810 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "upmap_max_deviation", 2024-05-27T07:36:51.810 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "int", 2024-05-27T07:36:51.810 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-05-27T07:36:51.810 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 1, 2024-05-27T07:36:51.810 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "5", 2024-05-27T07:36:51.810 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "1", 2024-05-27T07:36:51.810 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-05-27T07:36:51.810 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-05-27T07:36:51.810 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "deviation below which no optimization is attempted", 2024-05-27T07:36:51.810 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "If the number of PGs are within this count then no optimization is attempted", 2024-05-27T07:36:51.811 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-05-27T07:36:51.811 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-05-27T07:36:51.811 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-05-27T07:36:51.811 INFO:teuthology.orchestra.run.smithi052.stdout: "upmap_max_optimizations": { 2024-05-27T07:36:51.811 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "upmap_max_optimizations", 2024-05-27T07:36:51.811 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "uint", 2024-05-27T07:36:51.811 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-05-27T07:36:51.811 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 1, 2024-05-27T07:36:51.811 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "10", 2024-05-27T07:36:51.811 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-05-27T07:36:51.811 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-05-27T07:36:51.811 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-05-27T07:36:51.811 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "maximum upmap optimizations to make per attempt", 2024-05-27T07:36:51.812 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-05-27T07:36:51.812 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-05-27T07:36:51.812 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-05-27T07:36:51.812 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-05-27T07:36:51.812 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-05-27T07:36:51.812 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-05-27T07:36:51.812 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-05-27T07:36:51.812 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "cephadm", 2024-05-27T07:36:51.812 INFO:teuthology.orchestra.run.smithi052.stdout: "can_run": true, 2024-05-27T07:36:51.812 INFO:teuthology.orchestra.run.smithi052.stdout: "error_string": "", 2024-05-27T07:36:51.812 INFO:teuthology.orchestra.run.smithi052.stdout: "module_options": { 2024-05-27T07:36:51.812 INFO:teuthology.orchestra.run.smithi052.stdout: "agent_down_multiplier": { 2024-05-27T07:36:51.812 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "agent_down_multiplier", 2024-05-27T07:36:51.812 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "float", 2024-05-27T07:36:51.813 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-05-27T07:36:51.813 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 0, 2024-05-27T07:36:51.813 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "3.0", 2024-05-27T07:36:51.813 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-05-27T07:36:51.813 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-05-27T07:36:51.813 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-05-27T07:36:51.813 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "Multiplied by agent refresh rate to calculate how long agent must not report before being marked down", 2024-05-27T07:36:51.813 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-05-27T07:36:51.813 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-05-27T07:36:51.813 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-05-27T07:36:51.813 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-05-27T07:36:51.813 INFO:teuthology.orchestra.run.smithi052.stdout: "agent_refresh_rate": { 2024-05-27T07:36:51.814 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "agent_refresh_rate", 2024-05-27T07:36:51.814 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "secs", 2024-05-27T07:36:51.814 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-05-27T07:36:51.814 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 0, 2024-05-27T07:36:51.814 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "20", 2024-05-27T07:36:51.814 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-05-27T07:36:51.814 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-05-27T07:36:51.814 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-05-27T07:36:51.814 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "How often agent on each host will try to gather and send metadata", 2024-05-27T07:36:51.814 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-05-27T07:36:51.814 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-05-27T07:36:51.814 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-05-27T07:36:51.815 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-05-27T07:36:51.815 INFO:teuthology.orchestra.run.smithi052.stdout: "agent_starting_port": { 2024-05-27T07:36:51.815 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "agent_starting_port", 2024-05-27T07:36:51.815 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "int", 2024-05-27T07:36:51.815 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-05-27T07:36:51.815 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 0, 2024-05-27T07:36:51.815 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "4721", 2024-05-27T07:36:51.815 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-05-27T07:36:51.815 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-05-27T07:36:51.815 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-05-27T07:36:51.815 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)", 2024-05-27T07:36:51.815 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-05-27T07:36:51.816 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-05-27T07:36:51.816 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-05-27T07:36:51.816 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-05-27T07:36:51.816 INFO:teuthology.orchestra.run.smithi052.stdout: "allow_ptrace": { 2024-05-27T07:36:51.816 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "allow_ptrace", 2024-05-27T07:36:51.816 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "bool", 2024-05-27T07:36:51.816 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-05-27T07:36:51.816 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 0, 2024-05-27T07:36:51.816 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "False", 2024-05-27T07:36:51.816 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-05-27T07:36:51.816 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-05-27T07:36:51.816 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-05-27T07:36:51.816 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "allow SYS_PTRACE capability on ceph containers", 2024-05-27T07:36:51.817 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.", 2024-05-27T07:36:51.817 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-05-27T07:36:51.817 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-05-27T07:36:51.817 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-05-27T07:36:51.817 INFO:teuthology.orchestra.run.smithi052.stdout: "autotune_interval": { 2024-05-27T07:36:51.817 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "autotune_interval", 2024-05-27T07:36:51.817 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "secs", 2024-05-27T07:36:51.817 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-05-27T07:36:51.817 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 0, 2024-05-27T07:36:51.817 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "600", 2024-05-27T07:36:51.817 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-05-27T07:36:51.817 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-05-27T07:36:51.817 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-05-27T07:36:51.818 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "how frequently to autotune daemon memory", 2024-05-27T07:36:51.818 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-05-27T07:36:51.818 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-05-27T07:36:51.818 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-05-27T07:36:51.818 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-05-27T07:36:51.818 INFO:teuthology.orchestra.run.smithi052.stdout: "autotune_memory_target_ratio": { 2024-05-27T07:36:51.818 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "autotune_memory_target_ratio", 2024-05-27T07:36:51.818 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "float", 2024-05-27T07:36:51.818 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-05-27T07:36:51.818 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 0, 2024-05-27T07:36:51.818 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "0.7", 2024-05-27T07:36:51.818 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-05-27T07:36:51.819 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-05-27T07:36:51.819 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-05-27T07:36:51.819 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "ratio of total system memory to divide amongst autotuned daemons", 2024-05-27T07:36:51.819 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-05-27T07:36:51.819 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-05-27T07:36:51.819 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-05-27T07:36:51.819 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-05-27T07:36:51.819 INFO:teuthology.orchestra.run.smithi052.stdout: "cephadm_log_destination": { 2024-05-27T07:36:51.819 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "cephadm_log_destination", 2024-05-27T07:36:51.819 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "str", 2024-05-27T07:36:51.819 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-05-27T07:36:51.819 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 0, 2024-05-27T07:36:51.819 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "", 2024-05-27T07:36:51.820 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-05-27T07:36:51.820 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-05-27T07:36:51.820 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [ 2024-05-27T07:36:51.820 INFO:teuthology.orchestra.run.smithi052.stdout: "file", 2024-05-27T07:36:51.820 INFO:teuthology.orchestra.run.smithi052.stdout: "file,syslog", 2024-05-27T07:36:51.820 INFO:teuthology.orchestra.run.smithi052.stdout: "syslog" 2024-05-27T07:36:51.820 INFO:teuthology.orchestra.run.smithi052.stdout: ], 2024-05-27T07:36:51.820 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "Destination for cephadm command's persistent logging", 2024-05-27T07:36:51.820 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-05-27T07:36:51.820 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-05-27T07:36:51.820 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-05-27T07:36:51.820 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-05-27T07:36:51.820 INFO:teuthology.orchestra.run.smithi052.stdout: "cgroups_split": { 2024-05-27T07:36:51.821 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "cgroups_split", 2024-05-27T07:36:51.821 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "bool", 2024-05-27T07:36:51.821 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-05-27T07:36:51.821 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 0, 2024-05-27T07:36:51.821 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "True", 2024-05-27T07:36:51.821 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-05-27T07:36:51.821 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-05-27T07:36:51.821 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-05-27T07:36:51.821 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "Pass --cgroups=split when cephadm creates containers (currently podman only)", 2024-05-27T07:36:51.821 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-05-27T07:36:51.821 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-05-27T07:36:51.821 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-05-27T07:36:51.821 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-05-27T07:36:51.821 INFO:teuthology.orchestra.run.smithi052.stdout: "config_checks_enabled": { 2024-05-27T07:36:51.822 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "config_checks_enabled", 2024-05-27T07:36:51.822 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "bool", 2024-05-27T07:36:51.822 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-05-27T07:36:51.822 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 0, 2024-05-27T07:36:51.822 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "False", 2024-05-27T07:36:51.822 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-05-27T07:36:51.822 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-05-27T07:36:51.822 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-05-27T07:36:51.822 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "Enable or disable the cephadm configuration analysis", 2024-05-27T07:36:51.822 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-05-27T07:36:51.822 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-05-27T07:36:51.823 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-05-27T07:36:51.823 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-05-27T07:36:51.823 INFO:teuthology.orchestra.run.smithi052.stdout: "config_dashboard": { 2024-05-27T07:36:51.823 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "config_dashboard", 2024-05-27T07:36:51.823 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "bool", 2024-05-27T07:36:51.823 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-05-27T07:36:51.823 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 0, 2024-05-27T07:36:51.823 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "True", 2024-05-27T07:36:51.823 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-05-27T07:36:51.823 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-05-27T07:36:51.823 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-05-27T07:36:51.823 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "manage configs like API endpoints in Dashboard.", 2024-05-27T07:36:51.824 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-05-27T07:36:51.824 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-05-27T07:36:51.824 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-05-27T07:36:51.824 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-05-27T07:36:51.824 INFO:teuthology.orchestra.run.smithi052.stdout: "container_image_alertmanager": { 2024-05-27T07:36:51.824 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "container_image_alertmanager", 2024-05-27T07:36:51.824 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "str", 2024-05-27T07:36:51.824 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-05-27T07:36:51.824 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 0, 2024-05-27T07:36:51.825 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "quay.io/prometheus/alertmanager:v0.25.0", 2024-05-27T07:36:51.825 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-05-27T07:36:51.825 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-05-27T07:36:51.825 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-05-27T07:36:51.825 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "Prometheus container image", 2024-05-27T07:36:51.825 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-05-27T07:36:51.825 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-05-27T07:36:51.825 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-05-27T07:36:51.825 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-05-27T07:36:51.825 INFO:teuthology.orchestra.run.smithi052.stdout: "container_image_base": { 2024-05-27T07:36:51.825 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "container_image_base", 2024-05-27T07:36:51.825 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "str", 2024-05-27T07:36:51.825 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-05-27T07:36:51.826 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 1, 2024-05-27T07:36:51.826 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "quay.io/ceph/ceph", 2024-05-27T07:36:51.826 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-05-27T07:36:51.826 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-05-27T07:36:51.826 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-05-27T07:36:51.826 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "Container image name, without the tag", 2024-05-27T07:36:51.826 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-05-27T07:36:51.826 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-05-27T07:36:51.826 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-05-27T07:36:51.826 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-05-27T07:36:51.826 INFO:teuthology.orchestra.run.smithi052.stdout: "container_image_elasticsearch": { 2024-05-27T07:36:51.826 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "container_image_elasticsearch", 2024-05-27T07:36:51.826 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "str", 2024-05-27T07:36:51.827 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-05-27T07:36:51.827 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 0, 2024-05-27T07:36:51.827 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "quay.io/omrizeneva/elasticsearch:6.8.23", 2024-05-27T07:36:51.827 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-05-27T07:36:51.827 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-05-27T07:36:51.827 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-05-27T07:36:51.827 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "elasticsearch container image", 2024-05-27T07:36:51.827 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-05-27T07:36:51.827 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-05-27T07:36:51.827 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-05-27T07:36:51.827 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-05-27T07:36:51.827 INFO:teuthology.orchestra.run.smithi052.stdout: "container_image_grafana": { 2024-05-27T07:36:51.828 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "container_image_grafana", 2024-05-27T07:36:51.828 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "str", 2024-05-27T07:36:51.828 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-05-27T07:36:51.828 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 0, 2024-05-27T07:36:51.828 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "quay.io/ceph/grafana:9.4.12", 2024-05-27T07:36:51.828 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-05-27T07:36:51.828 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-05-27T07:36:51.828 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-05-27T07:36:51.828 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "Prometheus container image", 2024-05-27T07:36:51.828 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-05-27T07:36:51.828 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-05-27T07:36:51.828 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-05-27T07:36:51.828 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-05-27T07:36:51.829 INFO:teuthology.orchestra.run.smithi052.stdout: "container_image_haproxy": { 2024-05-27T07:36:51.829 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "container_image_haproxy", 2024-05-27T07:36:51.829 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "str", 2024-05-27T07:36:51.829 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-05-27T07:36:51.829 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 0, 2024-05-27T07:36:51.829 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "quay.io/ceph/haproxy:2.3", 2024-05-27T07:36:51.829 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-05-27T07:36:51.829 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-05-27T07:36:51.829 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-05-27T07:36:51.829 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "HAproxy container image", 2024-05-27T07:36:51.829 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-05-27T07:36:51.829 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-05-27T07:36:51.829 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-05-27T07:36:51.830 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-05-27T07:36:51.830 INFO:teuthology.orchestra.run.smithi052.stdout: "container_image_jaeger_agent": { 2024-05-27T07:36:51.830 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "container_image_jaeger_agent", 2024-05-27T07:36:51.830 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "str", 2024-05-27T07:36:51.830 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-05-27T07:36:51.830 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 0, 2024-05-27T07:36:51.830 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "quay.io/jaegertracing/jaeger-agent:1.29", 2024-05-27T07:36:51.830 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-05-27T07:36:51.830 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-05-27T07:36:51.830 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-05-27T07:36:51.830 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "Jaeger agent container image", 2024-05-27T07:36:51.830 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-05-27T07:36:51.830 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-05-27T07:36:51.831 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-05-27T07:36:51.831 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-05-27T07:36:51.831 INFO:teuthology.orchestra.run.smithi052.stdout: "container_image_jaeger_collector": { 2024-05-27T07:36:51.831 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "container_image_jaeger_collector", 2024-05-27T07:36:51.831 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "str", 2024-05-27T07:36:51.831 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-05-27T07:36:51.831 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 0, 2024-05-27T07:36:51.831 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "quay.io/jaegertracing/jaeger-collector:1.29", 2024-05-27T07:36:51.831 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-05-27T07:36:51.831 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-05-27T07:36:51.831 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-05-27T07:36:51.831 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "Jaeger collector container image", 2024-05-27T07:36:51.831 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-05-27T07:36:51.832 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-05-27T07:36:51.832 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-05-27T07:36:51.832 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-05-27T07:36:51.832 INFO:teuthology.orchestra.run.smithi052.stdout: "container_image_jaeger_query": { 2024-05-27T07:36:51.832 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "container_image_jaeger_query", 2024-05-27T07:36:51.832 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "str", 2024-05-27T07:36:51.832 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-05-27T07:36:51.832 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 0, 2024-05-27T07:36:51.832 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "quay.io/jaegertracing/jaeger-query:1.29", 2024-05-27T07:36:51.832 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-05-27T07:36:51.832 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-05-27T07:36:51.832 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-05-27T07:36:51.832 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "Jaeger query container image", 2024-05-27T07:36:51.833 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-05-27T07:36:51.833 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-05-27T07:36:51.833 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-05-27T07:36:51.833 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-05-27T07:36:51.833 INFO:teuthology.orchestra.run.smithi052.stdout: "container_image_keepalived": { 2024-05-27T07:36:51.833 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "container_image_keepalived", 2024-05-27T07:36:51.833 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "str", 2024-05-27T07:36:51.833 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-05-27T07:36:51.833 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 0, 2024-05-27T07:36:51.833 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "quay.io/ceph/keepalived:2.2.4", 2024-05-27T07:36:51.833 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-05-27T07:36:51.833 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-05-27T07:36:51.833 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-05-27T07:36:51.834 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "Keepalived container image", 2024-05-27T07:36:51.834 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-05-27T07:36:51.834 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-05-27T07:36:51.834 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-05-27T07:36:51.834 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-05-27T07:36:51.834 INFO:teuthology.orchestra.run.smithi052.stdout: "container_image_loki": { 2024-05-27T07:36:51.834 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "container_image_loki", 2024-05-27T07:36:51.834 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "str", 2024-05-27T07:36:51.834 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-05-27T07:36:51.834 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 0, 2024-05-27T07:36:51.834 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "docker.io/grafana/loki:3.0.0", 2024-05-27T07:36:51.834 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-05-27T07:36:51.834 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-05-27T07:36:51.835 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-05-27T07:36:51.835 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "Loki container image", 2024-05-27T07:36:51.835 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-05-27T07:36:51.835 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-05-27T07:36:51.835 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-05-27T07:36:51.835 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-05-27T07:36:51.835 INFO:teuthology.orchestra.run.smithi052.stdout: "container_image_node_exporter": { 2024-05-27T07:36:51.835 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "container_image_node_exporter", 2024-05-27T07:36:51.835 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "str", 2024-05-27T07:36:51.835 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-05-27T07:36:51.835 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 0, 2024-05-27T07:36:51.835 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "quay.io/prometheus/node-exporter:v1.5.0", 2024-05-27T07:36:51.835 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-05-27T07:36:51.835 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-05-27T07:36:51.836 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-05-27T07:36:51.836 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "Prometheus container image", 2024-05-27T07:36:51.836 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-05-27T07:36:51.836 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-05-27T07:36:51.836 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-05-27T07:36:51.836 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-05-27T07:36:51.836 INFO:teuthology.orchestra.run.smithi052.stdout: "container_image_nvmeof": { 2024-05-27T07:36:51.836 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "container_image_nvmeof", 2024-05-27T07:36:51.836 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "str", 2024-05-27T07:36:51.836 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-05-27T07:36:51.836 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 0, 2024-05-27T07:36:51.836 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "quay.io/ceph/nvmeof:1.2.5", 2024-05-27T07:36:51.836 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-05-27T07:36:51.837 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-05-27T07:36:51.837 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-05-27T07:36:51.837 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "Nvme-of container image", 2024-05-27T07:36:51.837 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-05-27T07:36:51.837 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-05-27T07:36:51.837 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-05-27T07:36:51.837 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-05-27T07:36:51.837 INFO:teuthology.orchestra.run.smithi052.stdout: "container_image_prometheus": { 2024-05-27T07:36:51.837 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "container_image_prometheus", 2024-05-27T07:36:51.837 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "str", 2024-05-27T07:36:51.837 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-05-27T07:36:51.837 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 0, 2024-05-27T07:36:51.837 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "quay.io/prometheus/prometheus:v2.43.0", 2024-05-27T07:36:51.838 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-05-27T07:36:51.838 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-05-27T07:36:51.838 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-05-27T07:36:51.838 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "Prometheus container image", 2024-05-27T07:36:51.838 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-05-27T07:36:51.838 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-05-27T07:36:51.838 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-05-27T07:36:51.838 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-05-27T07:36:51.838 INFO:teuthology.orchestra.run.smithi052.stdout: "container_image_promtail": { 2024-05-27T07:36:51.838 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "container_image_promtail", 2024-05-27T07:36:51.838 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "str", 2024-05-27T07:36:51.838 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-05-27T07:36:51.838 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 0, 2024-05-27T07:36:51.839 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "docker.io/grafana/promtail:3.0.0", 2024-05-27T07:36:51.839 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-05-27T07:36:51.839 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-05-27T07:36:51.839 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-05-27T07:36:51.839 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "Promtail container image", 2024-05-27T07:36:51.839 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-05-27T07:36:51.839 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-05-27T07:36:51.839 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-05-27T07:36:51.839 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-05-27T07:36:51.839 INFO:teuthology.orchestra.run.smithi052.stdout: "container_image_samba": { 2024-05-27T07:36:51.839 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "container_image_samba", 2024-05-27T07:36:51.839 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "str", 2024-05-27T07:36:51.839 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-05-27T07:36:51.839 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 0, 2024-05-27T07:36:51.840 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "quay.io/samba.org/samba-server:devbuilds-centos-amd64", 2024-05-27T07:36:51.840 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-05-27T07:36:51.840 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-05-27T07:36:51.840 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-05-27T07:36:51.840 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "Samba/SMB container image", 2024-05-27T07:36:51.840 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-05-27T07:36:51.840 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-05-27T07:36:51.840 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-05-27T07:36:51.840 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-05-27T07:36:51.840 INFO:teuthology.orchestra.run.smithi052.stdout: "container_image_snmp_gateway": { 2024-05-27T07:36:51.840 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "container_image_snmp_gateway", 2024-05-27T07:36:51.840 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "str", 2024-05-27T07:36:51.841 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-05-27T07:36:51.841 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 0, 2024-05-27T07:36:51.841 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "docker.io/maxwo/snmp-notifier:v1.2.1", 2024-05-27T07:36:51.841 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-05-27T07:36:51.841 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-05-27T07:36:51.841 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-05-27T07:36:51.841 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "SNMP Gateway container image", 2024-05-27T07:36:51.841 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-05-27T07:36:51.841 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-05-27T07:36:51.841 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-05-27T07:36:51.841 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-05-27T07:36:51.841 INFO:teuthology.orchestra.run.smithi052.stdout: "container_init": { 2024-05-27T07:36:51.842 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "container_init", 2024-05-27T07:36:51.842 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "bool", 2024-05-27T07:36:51.842 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-05-27T07:36:51.842 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 0, 2024-05-27T07:36:51.842 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "True", 2024-05-27T07:36:51.842 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-05-27T07:36:51.842 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-05-27T07:36:51.842 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-05-27T07:36:51.842 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "Run podman/docker with `--init`", 2024-05-27T07:36:51.842 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-05-27T07:36:51.842 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-05-27T07:36:51.842 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-05-27T07:36:51.843 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-05-27T07:36:51.843 INFO:teuthology.orchestra.run.smithi052.stdout: "daemon_cache_timeout": { 2024-05-27T07:36:51.843 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "daemon_cache_timeout", 2024-05-27T07:36:51.843 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "secs", 2024-05-27T07:36:51.843 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-05-27T07:36:51.843 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 0, 2024-05-27T07:36:51.843 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "600", 2024-05-27T07:36:51.843 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-05-27T07:36:51.843 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-05-27T07:36:51.843 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-05-27T07:36:51.843 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "seconds to cache service (daemon) inventory", 2024-05-27T07:36:51.843 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-05-27T07:36:51.843 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-05-27T07:36:51.844 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-05-27T07:36:51.844 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-05-27T07:36:51.844 INFO:teuthology.orchestra.run.smithi052.stdout: "default_cephadm_command_timeout": { 2024-05-27T07:36:51.844 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "default_cephadm_command_timeout", 2024-05-27T07:36:51.844 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "int", 2024-05-27T07:36:51.844 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-05-27T07:36:51.844 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 0, 2024-05-27T07:36:51.844 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "900", 2024-05-27T07:36:51.844 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-05-27T07:36:51.844 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-05-27T07:36:51.844 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-05-27T07:36:51.844 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "Default timeout applied to cephadm commands run directly on the host (in seconds)", 2024-05-27T07:36:51.844 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-05-27T07:36:51.845 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-05-27T07:36:51.845 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-05-27T07:36:51.845 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-05-27T07:36:51.845 INFO:teuthology.orchestra.run.smithi052.stdout: "default_registry": { 2024-05-27T07:36:51.845 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "default_registry", 2024-05-27T07:36:51.845 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "str", 2024-05-27T07:36:51.845 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-05-27T07:36:51.845 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 0, 2024-05-27T07:36:51.845 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "docker.io", 2024-05-27T07:36:51.845 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-05-27T07:36:51.845 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-05-27T07:36:51.845 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-05-27T07:36:51.845 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "Search-registry to which we should normalize unqualified image names. This is not the default registry", 2024-05-27T07:36:51.846 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-05-27T07:36:51.846 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-05-27T07:36:51.846 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-05-27T07:36:51.846 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-05-27T07:36:51.846 INFO:teuthology.orchestra.run.smithi052.stdout: "device_cache_timeout": { 2024-05-27T07:36:51.846 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "device_cache_timeout", 2024-05-27T07:36:51.846 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "secs", 2024-05-27T07:36:51.846 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-05-27T07:36:51.846 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 0, 2024-05-27T07:36:51.846 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "1800", 2024-05-27T07:36:51.846 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-05-27T07:36:51.846 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-05-27T07:36:51.846 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-05-27T07:36:51.847 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "seconds to cache device inventory", 2024-05-27T07:36:51.847 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-05-27T07:36:51.847 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-05-27T07:36:51.847 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-05-27T07:36:51.847 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-05-27T07:36:51.847 INFO:teuthology.orchestra.run.smithi052.stdout: "device_enhanced_scan": { 2024-05-27T07:36:51.847 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "device_enhanced_scan", 2024-05-27T07:36:51.847 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "bool", 2024-05-27T07:36:51.847 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-05-27T07:36:51.847 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 0, 2024-05-27T07:36:51.847 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "False", 2024-05-27T07:36:51.847 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-05-27T07:36:51.847 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-05-27T07:36:51.848 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-05-27T07:36:51.848 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "Use libstoragemgmt during device scans", 2024-05-27T07:36:51.848 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-05-27T07:36:51.848 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-05-27T07:36:51.848 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-05-27T07:36:51.848 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-05-27T07:36:51.848 INFO:teuthology.orchestra.run.smithi052.stdout: "facts_cache_timeout": { 2024-05-27T07:36:51.848 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "facts_cache_timeout", 2024-05-27T07:36:51.848 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "secs", 2024-05-27T07:36:51.848 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-05-27T07:36:51.848 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 0, 2024-05-27T07:36:51.848 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "60", 2024-05-27T07:36:51.848 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-05-27T07:36:51.849 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-05-27T07:36:51.849 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-05-27T07:36:51.849 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "seconds to cache host facts data", 2024-05-27T07:36:51.849 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-05-27T07:36:51.849 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-05-27T07:36:51.849 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-05-27T07:36:51.849 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-05-27T07:36:51.849 INFO:teuthology.orchestra.run.smithi052.stdout: "grafana_dashboards_path": { 2024-05-27T07:36:51.849 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "grafana_dashboards_path", 2024-05-27T07:36:51.849 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "str", 2024-05-27T07:36:51.849 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-05-27T07:36:51.849 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 0, 2024-05-27T07:36:51.849 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "/etc/grafana/dashboards/ceph-dashboard/", 2024-05-27T07:36:51.850 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-05-27T07:36:51.850 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-05-27T07:36:51.850 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-05-27T07:36:51.850 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "location of dashboards to include in grafana deployments", 2024-05-27T07:36:51.850 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-05-27T07:36:51.850 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-05-27T07:36:51.850 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-05-27T07:36:51.850 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-05-27T07:36:51.850 INFO:teuthology.orchestra.run.smithi052.stdout: "host_check_interval": { 2024-05-27T07:36:51.850 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "host_check_interval", 2024-05-27T07:36:51.850 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "secs", 2024-05-27T07:36:51.850 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-05-27T07:36:51.851 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 0, 2024-05-27T07:36:51.851 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "600", 2024-05-27T07:36:51.851 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-05-27T07:36:51.851 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-05-27T07:36:51.851 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-05-27T07:36:51.851 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "how frequently to perform a host check", 2024-05-27T07:36:51.851 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-05-27T07:36:51.851 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-05-27T07:36:51.851 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-05-27T07:36:51.851 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-05-27T07:36:51.851 INFO:teuthology.orchestra.run.smithi052.stdout: "hw_monitoring": { 2024-05-27T07:36:51.851 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "hw_monitoring", 2024-05-27T07:36:51.851 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "bool", 2024-05-27T07:36:51.852 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-05-27T07:36:51.852 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 0, 2024-05-27T07:36:51.852 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "False", 2024-05-27T07:36:51.852 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-05-27T07:36:51.852 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-05-27T07:36:51.852 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-05-27T07:36:51.852 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "Deploy hw monitoring daemon on every host.", 2024-05-27T07:36:51.852 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-05-27T07:36:51.852 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-05-27T07:36:51.852 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-05-27T07:36:51.852 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-05-27T07:36:51.852 INFO:teuthology.orchestra.run.smithi052.stdout: "inventory_list_all": { 2024-05-27T07:36:51.852 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "inventory_list_all", 2024-05-27T07:36:51.853 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "bool", 2024-05-27T07:36:51.853 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-05-27T07:36:51.853 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 0, 2024-05-27T07:36:51.853 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "False", 2024-05-27T07:36:51.853 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-05-27T07:36:51.853 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-05-27T07:36:51.853 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-05-27T07:36:51.853 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)", 2024-05-27T07:36:51.853 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-05-27T07:36:51.853 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-05-27T07:36:51.853 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-05-27T07:36:51.853 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-05-27T07:36:51.854 INFO:teuthology.orchestra.run.smithi052.stdout: "log_level": { 2024-05-27T07:36:51.854 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "log_level", 2024-05-27T07:36:51.854 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "str", 2024-05-27T07:36:51.854 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-05-27T07:36:51.854 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 1, 2024-05-27T07:36:51.854 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "", 2024-05-27T07:36:51.854 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-05-27T07:36:51.854 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-05-27T07:36:51.854 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [ 2024-05-27T07:36:51.854 INFO:teuthology.orchestra.run.smithi052.stdout: "", 2024-05-27T07:36:51.854 INFO:teuthology.orchestra.run.smithi052.stdout: "critical", 2024-05-27T07:36:51.854 INFO:teuthology.orchestra.run.smithi052.stdout: "debug", 2024-05-27T07:36:51.854 INFO:teuthology.orchestra.run.smithi052.stdout: "error", 2024-05-27T07:36:51.855 INFO:teuthology.orchestra.run.smithi052.stdout: "info", 2024-05-27T07:36:51.855 INFO:teuthology.orchestra.run.smithi052.stdout: "warning" 2024-05-27T07:36:51.855 INFO:teuthology.orchestra.run.smithi052.stdout: ], 2024-05-27T07:36:51.855 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-05-27T07:36:51.855 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-05-27T07:36:51.855 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-05-27T07:36:51.855 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-05-27T07:36:51.855 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-05-27T07:36:51.855 INFO:teuthology.orchestra.run.smithi052.stdout: "log_refresh_metadata": { 2024-05-27T07:36:51.855 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "log_refresh_metadata", 2024-05-27T07:36:51.855 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "bool", 2024-05-27T07:36:51.855 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-05-27T07:36:51.855 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 0, 2024-05-27T07:36:51.856 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "False", 2024-05-27T07:36:51.856 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-05-27T07:36:51.856 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-05-27T07:36:51.856 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-05-27T07:36:51.856 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level", 2024-05-27T07:36:51.856 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-05-27T07:36:51.856 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-05-27T07:36:51.856 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-05-27T07:36:51.856 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-05-27T07:36:51.856 INFO:teuthology.orchestra.run.smithi052.stdout: "log_to_cluster": { 2024-05-27T07:36:51.856 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "log_to_cluster", 2024-05-27T07:36:51.856 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "bool", 2024-05-27T07:36:51.856 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-05-27T07:36:51.857 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 0, 2024-05-27T07:36:51.857 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "True", 2024-05-27T07:36:51.857 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-05-27T07:36:51.857 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-05-27T07:36:51.857 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-05-27T07:36:51.857 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "log to the \"cephadm\" cluster log channel\"", 2024-05-27T07:36:51.857 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-05-27T07:36:51.857 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-05-27T07:36:51.857 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-05-27T07:36:51.857 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-05-27T07:36:51.857 INFO:teuthology.orchestra.run.smithi052.stdout: "log_to_cluster_level": { 2024-05-27T07:36:51.857 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "log_to_cluster_level", 2024-05-27T07:36:51.857 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "str", 2024-05-27T07:36:51.858 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-05-27T07:36:51.858 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 1, 2024-05-27T07:36:51.858 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "info", 2024-05-27T07:36:51.858 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-05-27T07:36:51.858 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-05-27T07:36:51.858 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [ 2024-05-27T07:36:51.858 INFO:teuthology.orchestra.run.smithi052.stdout: "", 2024-05-27T07:36:51.858 INFO:teuthology.orchestra.run.smithi052.stdout: "critical", 2024-05-27T07:36:51.858 INFO:teuthology.orchestra.run.smithi052.stdout: "debug", 2024-05-27T07:36:51.858 INFO:teuthology.orchestra.run.smithi052.stdout: "error", 2024-05-27T07:36:51.858 INFO:teuthology.orchestra.run.smithi052.stdout: "info", 2024-05-27T07:36:51.858 INFO:teuthology.orchestra.run.smithi052.stdout: "warning" 2024-05-27T07:36:51.858 INFO:teuthology.orchestra.run.smithi052.stdout: ], 2024-05-27T07:36:51.859 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-05-27T07:36:51.859 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-05-27T07:36:51.859 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-05-27T07:36:51.859 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-05-27T07:36:51.859 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-05-27T07:36:51.859 INFO:teuthology.orchestra.run.smithi052.stdout: "log_to_file": { 2024-05-27T07:36:51.859 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "log_to_file", 2024-05-27T07:36:51.859 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "bool", 2024-05-27T07:36:51.859 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-05-27T07:36:51.859 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 1, 2024-05-27T07:36:51.859 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "False", 2024-05-27T07:36:51.859 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-05-27T07:36:51.859 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-05-27T07:36:51.860 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-05-27T07:36:51.860 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-05-27T07:36:51.860 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-05-27T07:36:51.860 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-05-27T07:36:51.860 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-05-27T07:36:51.860 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-05-27T07:36:51.860 INFO:teuthology.orchestra.run.smithi052.stdout: "manage_etc_ceph_ceph_conf": { 2024-05-27T07:36:51.860 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "manage_etc_ceph_ceph_conf", 2024-05-27T07:36:51.860 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "bool", 2024-05-27T07:36:51.860 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-05-27T07:36:51.860 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 0, 2024-05-27T07:36:51.860 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "False", 2024-05-27T07:36:51.860 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-05-27T07:36:51.860 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-05-27T07:36:51.861 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-05-27T07:36:51.861 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "Manage and own /etc/ceph/ceph.conf on the hosts.", 2024-05-27T07:36:51.861 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-05-27T07:36:51.861 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-05-27T07:36:51.861 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-05-27T07:36:51.861 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-05-27T07:36:51.861 INFO:teuthology.orchestra.run.smithi052.stdout: "manage_etc_ceph_ceph_conf_hosts": { 2024-05-27T07:36:51.861 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "manage_etc_ceph_ceph_conf_hosts", 2024-05-27T07:36:51.861 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "str", 2024-05-27T07:36:51.861 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-05-27T07:36:51.861 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 0, 2024-05-27T07:36:51.861 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "*", 2024-05-27T07:36:51.861 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-05-27T07:36:51.862 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-05-27T07:36:51.862 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-05-27T07:36:51.862 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf", 2024-05-27T07:36:51.862 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-05-27T07:36:51.862 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-05-27T07:36:51.862 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-05-27T07:36:51.862 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-05-27T07:36:51.862 INFO:teuthology.orchestra.run.smithi052.stdout: "max_count_per_host": { 2024-05-27T07:36:51.862 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "max_count_per_host", 2024-05-27T07:36:51.862 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "int", 2024-05-27T07:36:51.862 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-05-27T07:36:51.862 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 0, 2024-05-27T07:36:51.862 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "10", 2024-05-27T07:36:51.863 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-05-27T07:36:51.863 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-05-27T07:36:51.863 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-05-27T07:36:51.863 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "max number of daemons per service per host", 2024-05-27T07:36:51.863 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-05-27T07:36:51.863 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-05-27T07:36:51.863 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-05-27T07:36:51.863 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-05-27T07:36:51.863 INFO:teuthology.orchestra.run.smithi052.stdout: "max_osd_draining_count": { 2024-05-27T07:36:51.863 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "max_osd_draining_count", 2024-05-27T07:36:51.863 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "int", 2024-05-27T07:36:51.863 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-05-27T07:36:51.863 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 0, 2024-05-27T07:36:51.863 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "10", 2024-05-27T07:36:51.864 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-05-27T07:36:51.864 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-05-27T07:36:51.864 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-05-27T07:36:51.864 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "max number of osds that will be drained simultaneously when osds are removed", 2024-05-27T07:36:51.864 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-05-27T07:36:51.864 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-05-27T07:36:51.864 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-05-27T07:36:51.864 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-05-27T07:36:51.864 INFO:teuthology.orchestra.run.smithi052.stdout: "migration_current": { 2024-05-27T07:36:51.864 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "migration_current", 2024-05-27T07:36:51.864 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "int", 2024-05-27T07:36:51.864 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-05-27T07:36:51.864 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 0, 2024-05-27T07:36:51.865 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "", 2024-05-27T07:36:51.865 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-05-27T07:36:51.865 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-05-27T07:36:51.865 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-05-27T07:36:51.865 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "internal - do not modify", 2024-05-27T07:36:51.865 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-05-27T07:36:51.865 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-05-27T07:36:51.865 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-05-27T07:36:51.865 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-05-27T07:36:51.865 INFO:teuthology.orchestra.run.smithi052.stdout: "mode": { 2024-05-27T07:36:51.865 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "mode", 2024-05-27T07:36:51.865 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "str", 2024-05-27T07:36:51.865 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-05-27T07:36:51.866 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 0, 2024-05-27T07:36:51.866 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "root", 2024-05-27T07:36:51.866 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-05-27T07:36:51.866 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-05-27T07:36:51.866 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [ 2024-05-27T07:36:51.866 INFO:teuthology.orchestra.run.smithi052.stdout: "cephadm-package", 2024-05-27T07:36:51.866 INFO:teuthology.orchestra.run.smithi052.stdout: "root" 2024-05-27T07:36:51.866 INFO:teuthology.orchestra.run.smithi052.stdout: ], 2024-05-27T07:36:51.866 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "mode for remote execution of cephadm", 2024-05-27T07:36:51.866 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-05-27T07:36:51.866 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-05-27T07:36:51.866 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-05-27T07:36:51.866 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-05-27T07:36:51.867 INFO:teuthology.orchestra.run.smithi052.stdout: "oob_default_addr": { 2024-05-27T07:36:51.867 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "oob_default_addr", 2024-05-27T07:36:51.867 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "str", 2024-05-27T07:36:51.867 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-05-27T07:36:51.867 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 0, 2024-05-27T07:36:51.867 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "169.254.1.1", 2024-05-27T07:36:51.867 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-05-27T07:36:51.867 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-05-27T07:36:51.867 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-05-27T07:36:51.867 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "Default address for RedFish API (oob management).", 2024-05-27T07:36:51.867 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-05-27T07:36:51.868 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-05-27T07:36:51.868 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-05-27T07:36:51.868 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-05-27T07:36:51.868 INFO:teuthology.orchestra.run.smithi052.stdout: "prometheus_alerts_path": { 2024-05-27T07:36:51.868 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "prometheus_alerts_path", 2024-05-27T07:36:51.868 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "str", 2024-05-27T07:36:51.868 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-05-27T07:36:51.868 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 0, 2024-05-27T07:36:51.868 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "/etc/prometheus/ceph/ceph_default_alerts.yml", 2024-05-27T07:36:51.868 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-05-27T07:36:51.868 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-05-27T07:36:51.868 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-05-27T07:36:51.868 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "location of alerts to include in prometheus deployments", 2024-05-27T07:36:51.869 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-05-27T07:36:51.869 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-05-27T07:36:51.869 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-05-27T07:36:51.869 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-05-27T07:36:51.869 INFO:teuthology.orchestra.run.smithi052.stdout: "registry_insecure": { 2024-05-27T07:36:51.869 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "registry_insecure", 2024-05-27T07:36:51.869 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "bool", 2024-05-27T07:36:51.869 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-05-27T07:36:51.869 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 0, 2024-05-27T07:36:51.869 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "False", 2024-05-27T07:36:51.869 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-05-27T07:36:51.869 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-05-27T07:36:51.869 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-05-27T07:36:51.869 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "Registry is to be considered insecure (no TLS available). Only for development purposes.", 2024-05-27T07:36:51.870 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-05-27T07:36:51.870 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-05-27T07:36:51.870 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-05-27T07:36:51.870 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-05-27T07:36:51.870 INFO:teuthology.orchestra.run.smithi052.stdout: "registry_password": { 2024-05-27T07:36:51.870 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "registry_password", 2024-05-27T07:36:51.870 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "str", 2024-05-27T07:36:51.870 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-05-27T07:36:51.870 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 0, 2024-05-27T07:36:51.870 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "", 2024-05-27T07:36:51.870 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-05-27T07:36:51.870 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-05-27T07:36:51.871 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-05-27T07:36:51.871 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "Custom repository password. Only used for logging into a registry.", 2024-05-27T07:36:51.871 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-05-27T07:36:51.871 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-05-27T07:36:51.871 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-05-27T07:36:51.871 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-05-27T07:36:51.871 INFO:teuthology.orchestra.run.smithi052.stdout: "registry_url": { 2024-05-27T07:36:51.871 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "registry_url", 2024-05-27T07:36:51.871 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "str", 2024-05-27T07:36:51.871 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-05-27T07:36:51.871 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 0, 2024-05-27T07:36:51.871 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "", 2024-05-27T07:36:51.871 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-05-27T07:36:51.872 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-05-27T07:36:51.872 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-05-27T07:36:51.872 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "Registry url for login purposes. This is not the default registry", 2024-05-27T07:36:51.872 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-05-27T07:36:51.872 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-05-27T07:36:51.872 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-05-27T07:36:51.872 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-05-27T07:36:51.872 INFO:teuthology.orchestra.run.smithi052.stdout: "registry_username": { 2024-05-27T07:36:51.872 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "registry_username", 2024-05-27T07:36:51.872 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "str", 2024-05-27T07:36:51.872 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-05-27T07:36:51.872 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 0, 2024-05-27T07:36:51.873 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "", 2024-05-27T07:36:51.873 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-05-27T07:36:51.873 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-05-27T07:36:51.873 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-05-27T07:36:51.873 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "Custom repository username. Only used for logging into a registry.", 2024-05-27T07:36:51.873 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-05-27T07:36:51.873 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-05-27T07:36:51.873 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-05-27T07:36:51.873 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-05-27T07:36:51.873 INFO:teuthology.orchestra.run.smithi052.stdout: "secure_monitoring_stack": { 2024-05-27T07:36:51.873 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "secure_monitoring_stack", 2024-05-27T07:36:51.873 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "bool", 2024-05-27T07:36:51.873 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-05-27T07:36:51.874 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 0, 2024-05-27T07:36:51.874 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "False", 2024-05-27T07:36:51.874 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-05-27T07:36:51.874 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-05-27T07:36:51.874 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-05-27T07:36:51.874 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "Enable TLS security for all the monitoring stack daemons", 2024-05-27T07:36:51.874 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-05-27T07:36:51.874 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-05-27T07:36:51.874 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-05-27T07:36:51.874 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-05-27T07:36:51.874 INFO:teuthology.orchestra.run.smithi052.stdout: "service_discovery_port": { 2024-05-27T07:36:51.874 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "service_discovery_port", 2024-05-27T07:36:51.874 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "int", 2024-05-27T07:36:51.875 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-05-27T07:36:51.875 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 0, 2024-05-27T07:36:51.875 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "8765", 2024-05-27T07:36:51.875 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-05-27T07:36:51.875 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-05-27T07:36:51.875 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-05-27T07:36:51.875 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "cephadm service discovery port", 2024-05-27T07:36:51.875 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-05-27T07:36:51.875 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-05-27T07:36:51.875 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-05-27T07:36:51.875 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-05-27T07:36:51.875 INFO:teuthology.orchestra.run.smithi052.stdout: "sqlite3_killpoint": { 2024-05-27T07:36:51.876 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "sqlite3_killpoint", 2024-05-27T07:36:51.876 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "int", 2024-05-27T07:36:51.876 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "dev", 2024-05-27T07:36:51.876 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 1, 2024-05-27T07:36:51.876 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "0", 2024-05-27T07:36:51.876 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-05-27T07:36:51.876 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-05-27T07:36:51.876 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-05-27T07:36:51.876 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-05-27T07:36:51.876 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-05-27T07:36:51.876 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-05-27T07:36:51.876 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-05-27T07:36:51.877 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-05-27T07:36:51.877 INFO:teuthology.orchestra.run.smithi052.stdout: "ssh_config_file": { 2024-05-27T07:36:51.877 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "ssh_config_file", 2024-05-27T07:36:51.877 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "str", 2024-05-27T07:36:51.877 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-05-27T07:36:51.877 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 0, 2024-05-27T07:36:51.877 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "", 2024-05-27T07:36:51.877 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-05-27T07:36:51.877 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-05-27T07:36:51.877 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-05-27T07:36:51.877 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "customized SSH config file to connect to managed hosts", 2024-05-27T07:36:51.877 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-05-27T07:36:51.877 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-05-27T07:36:51.878 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-05-27T07:36:51.878 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-05-27T07:36:51.878 INFO:teuthology.orchestra.run.smithi052.stdout: "use_agent": { 2024-05-27T07:36:51.878 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "use_agent", 2024-05-27T07:36:51.878 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "bool", 2024-05-27T07:36:51.878 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-05-27T07:36:51.878 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 0, 2024-05-27T07:36:51.878 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "False", 2024-05-27T07:36:51.878 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-05-27T07:36:51.878 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-05-27T07:36:51.878 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-05-27T07:36:51.878 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "Use cephadm agent on each host to gather and send metadata", 2024-05-27T07:36:51.879 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-05-27T07:36:51.879 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-05-27T07:36:51.879 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-05-27T07:36:51.879 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-05-27T07:36:51.879 INFO:teuthology.orchestra.run.smithi052.stdout: "use_repo_digest": { 2024-05-27T07:36:51.879 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "use_repo_digest", 2024-05-27T07:36:51.879 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "bool", 2024-05-27T07:36:51.879 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-05-27T07:36:51.879 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 0, 2024-05-27T07:36:51.879 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "True", 2024-05-27T07:36:51.879 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-05-27T07:36:51.879 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-05-27T07:36:51.879 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-05-27T07:36:51.880 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "Automatically convert image tags to image digest. Make sure all daemons use the same image", 2024-05-27T07:36:51.880 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-05-27T07:36:51.880 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-05-27T07:36:51.880 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-05-27T07:36:51.880 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-05-27T07:36:51.880 INFO:teuthology.orchestra.run.smithi052.stdout: "warn_on_failed_host_check": { 2024-05-27T07:36:51.880 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "warn_on_failed_host_check", 2024-05-27T07:36:51.880 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "bool", 2024-05-27T07:36:51.880 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-05-27T07:36:51.880 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 0, 2024-05-27T07:36:51.880 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "True", 2024-05-27T07:36:51.880 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-05-27T07:36:51.880 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-05-27T07:36:51.881 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-05-27T07:36:51.881 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "raise a health warning if the host check fails", 2024-05-27T07:36:51.881 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-05-27T07:36:51.881 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-05-27T07:36:51.881 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-05-27T07:36:51.881 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-05-27T07:36:51.881 INFO:teuthology.orchestra.run.smithi052.stdout: "warn_on_stray_daemons": { 2024-05-27T07:36:51.881 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "warn_on_stray_daemons", 2024-05-27T07:36:51.881 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "bool", 2024-05-27T07:36:51.881 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-05-27T07:36:51.881 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 0, 2024-05-27T07:36:51.881 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "True", 2024-05-27T07:36:51.882 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-05-27T07:36:51.882 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-05-27T07:36:51.882 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-05-27T07:36:51.882 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "raise a health warning if daemons are detected that are not managed by cephadm", 2024-05-27T07:36:51.882 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-05-27T07:36:51.882 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-05-27T07:36:51.882 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-05-27T07:36:51.882 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-05-27T07:36:51.882 INFO:teuthology.orchestra.run.smithi052.stdout: "warn_on_stray_hosts": { 2024-05-27T07:36:51.882 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "warn_on_stray_hosts", 2024-05-27T07:36:51.882 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "bool", 2024-05-27T07:36:51.882 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-05-27T07:36:51.883 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 0, 2024-05-27T07:36:51.883 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "True", 2024-05-27T07:36:51.883 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-05-27T07:36:51.883 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-05-27T07:36:51.883 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-05-27T07:36:51.883 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "raise a health warning if daemons are detected on a host that is not managed by cephadm", 2024-05-27T07:36:51.883 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-05-27T07:36:51.883 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-05-27T07:36:51.883 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-05-27T07:36:51.883 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-05-27T07:36:51.883 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-05-27T07:36:51.883 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-05-27T07:36:51.883 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-05-27T07:36:51.884 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "crash", 2024-05-27T07:36:51.884 INFO:teuthology.orchestra.run.smithi052.stdout: "can_run": true, 2024-05-27T07:36:51.884 INFO:teuthology.orchestra.run.smithi052.stdout: "error_string": "", 2024-05-27T07:36:51.884 INFO:teuthology.orchestra.run.smithi052.stdout: "module_options": { 2024-05-27T07:36:51.884 INFO:teuthology.orchestra.run.smithi052.stdout: "log_level": { 2024-05-27T07:36:51.884 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "log_level", 2024-05-27T07:36:51.884 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "str", 2024-05-27T07:36:51.884 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-05-27T07:36:51.884 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 1, 2024-05-27T07:36:51.884 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "", 2024-05-27T07:36:51.884 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-05-27T07:36:51.884 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-05-27T07:36:51.885 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [ 2024-05-27T07:36:51.885 INFO:teuthology.orchestra.run.smithi052.stdout: "", 2024-05-27T07:36:51.885 INFO:teuthology.orchestra.run.smithi052.stdout: "critical", 2024-05-27T07:36:51.885 INFO:teuthology.orchestra.run.smithi052.stdout: "debug", 2024-05-27T07:36:51.885 INFO:teuthology.orchestra.run.smithi052.stdout: "error", 2024-05-27T07:36:51.885 INFO:teuthology.orchestra.run.smithi052.stdout: "info", 2024-05-27T07:36:51.885 INFO:teuthology.orchestra.run.smithi052.stdout: "warning" 2024-05-27T07:36:51.885 INFO:teuthology.orchestra.run.smithi052.stdout: ], 2024-05-27T07:36:51.885 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-05-27T07:36:51.885 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-05-27T07:36:51.885 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-05-27T07:36:51.885 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-05-27T07:36:51.885 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-05-27T07:36:51.886 INFO:teuthology.orchestra.run.smithi052.stdout: "log_to_cluster": { 2024-05-27T07:36:51.886 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "log_to_cluster", 2024-05-27T07:36:51.886 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "bool", 2024-05-27T07:36:51.886 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-05-27T07:36:51.886 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 1, 2024-05-27T07:36:51.886 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "False", 2024-05-27T07:36:51.886 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-05-27T07:36:51.886 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-05-27T07:36:51.886 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-05-27T07:36:51.886 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-05-27T07:36:51.886 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-05-27T07:36:51.886 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-05-27T07:36:51.887 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-05-27T07:36:51.887 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-05-27T07:36:51.887 INFO:teuthology.orchestra.run.smithi052.stdout: "log_to_cluster_level": { 2024-05-27T07:36:51.887 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "log_to_cluster_level", 2024-05-27T07:36:51.887 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "str", 2024-05-27T07:36:51.887 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-05-27T07:36:51.887 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 1, 2024-05-27T07:36:51.887 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "info", 2024-05-27T07:36:51.887 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-05-27T07:36:51.887 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-05-27T07:36:51.887 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [ 2024-05-27T07:36:51.887 INFO:teuthology.orchestra.run.smithi052.stdout: "", 2024-05-27T07:36:51.887 INFO:teuthology.orchestra.run.smithi052.stdout: "critical", 2024-05-27T07:36:51.888 INFO:teuthology.orchestra.run.smithi052.stdout: "debug", 2024-05-27T07:36:51.888 INFO:teuthology.orchestra.run.smithi052.stdout: "error", 2024-05-27T07:36:51.888 INFO:teuthology.orchestra.run.smithi052.stdout: "info", 2024-05-27T07:36:51.888 INFO:teuthology.orchestra.run.smithi052.stdout: "warning" 2024-05-27T07:36:51.888 INFO:teuthology.orchestra.run.smithi052.stdout: ], 2024-05-27T07:36:51.888 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-05-27T07:36:51.888 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-05-27T07:36:51.888 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-05-27T07:36:51.888 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-05-27T07:36:51.888 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-05-27T07:36:51.888 INFO:teuthology.orchestra.run.smithi052.stdout: "log_to_file": { 2024-05-27T07:36:51.888 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "log_to_file", 2024-05-27T07:36:51.888 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "bool", 2024-05-27T07:36:51.889 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-05-27T07:36:51.889 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 1, 2024-05-27T07:36:51.889 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "False", 2024-05-27T07:36:51.889 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-05-27T07:36:51.889 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-05-27T07:36:51.889 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-05-27T07:36:51.889 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-05-27T07:36:51.889 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-05-27T07:36:51.889 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-05-27T07:36:51.889 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-05-27T07:36:51.889 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-05-27T07:36:51.889 INFO:teuthology.orchestra.run.smithi052.stdout: "retain_interval": { 2024-05-27T07:36:51.889 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "retain_interval", 2024-05-27T07:36:51.890 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "secs", 2024-05-27T07:36:51.890 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-05-27T07:36:51.890 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 1, 2024-05-27T07:36:51.890 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "31536000", 2024-05-27T07:36:51.890 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-05-27T07:36:51.890 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-05-27T07:36:51.890 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-05-27T07:36:51.890 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "how long to retain crashes before pruning them", 2024-05-27T07:36:51.890 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-05-27T07:36:51.890 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-05-27T07:36:51.890 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-05-27T07:36:51.890 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-05-27T07:36:51.891 INFO:teuthology.orchestra.run.smithi052.stdout: "sqlite3_killpoint": { 2024-05-27T07:36:51.891 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "sqlite3_killpoint", 2024-05-27T07:36:51.891 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "int", 2024-05-27T07:36:51.891 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "dev", 2024-05-27T07:36:51.891 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 1, 2024-05-27T07:36:51.891 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "0", 2024-05-27T07:36:51.891 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-05-27T07:36:51.891 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-05-27T07:36:51.891 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-05-27T07:36:51.891 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-05-27T07:36:51.891 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-05-27T07:36:51.891 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-05-27T07:36:51.891 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-05-27T07:36:51.892 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-05-27T07:36:51.892 INFO:teuthology.orchestra.run.smithi052.stdout: "warn_recent_interval": { 2024-05-27T07:36:51.892 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "warn_recent_interval", 2024-05-27T07:36:51.892 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "secs", 2024-05-27T07:36:51.892 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-05-27T07:36:51.892 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 1, 2024-05-27T07:36:51.892 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "1209600", 2024-05-27T07:36:51.892 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-05-27T07:36:51.892 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-05-27T07:36:51.892 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-05-27T07:36:51.892 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "time interval in which to warn about recent crashes", 2024-05-27T07:36:51.892 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-05-27T07:36:51.892 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-05-27T07:36:51.893 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-05-27T07:36:51.893 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-05-27T07:36:51.893 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-05-27T07:36:51.893 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-05-27T07:36:51.893 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-05-27T07:36:51.893 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "dashboard", 2024-05-27T07:36:51.893 INFO:teuthology.orchestra.run.smithi052.stdout: "can_run": true, 2024-05-27T07:36:51.893 INFO:teuthology.orchestra.run.smithi052.stdout: "error_string": "", 2024-05-27T07:36:51.893 INFO:teuthology.orchestra.run.smithi052.stdout: "module_options": { 2024-05-27T07:36:51.893 INFO:teuthology.orchestra.run.smithi052.stdout: "ACCOUNT_LOCKOUT_ATTEMPTS": { 2024-05-27T07:36:51.893 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "ACCOUNT_LOCKOUT_ATTEMPTS", 2024-05-27T07:36:51.893 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "int", 2024-05-27T07:36:51.893 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-05-27T07:36:51.894 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 0, 2024-05-27T07:36:51.894 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "10", 2024-05-27T07:36:51.894 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-05-27T07:36:51.894 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-05-27T07:36:51.894 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-05-27T07:36:51.894 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-05-27T07:36:51.894 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-05-27T07:36:51.894 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-05-27T07:36:51.894 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-05-27T07:36:51.894 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-05-27T07:36:51.894 INFO:teuthology.orchestra.run.smithi052.stdout: "ALERTMANAGER_API_HOST": { 2024-05-27T07:36:51.894 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "ALERTMANAGER_API_HOST", 2024-05-27T07:36:51.895 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "str", 2024-05-27T07:36:51.895 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-05-27T07:36:51.895 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 0, 2024-05-27T07:36:51.895 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "", 2024-05-27T07:36:51.895 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-05-27T07:36:51.895 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-05-27T07:36:51.895 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-05-27T07:36:51.895 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-05-27T07:36:51.895 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-05-27T07:36:51.895 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-05-27T07:36:51.895 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-05-27T07:36:51.895 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-05-27T07:36:51.895 INFO:teuthology.orchestra.run.smithi052.stdout: "ALERTMANAGER_API_SSL_VERIFY": { 2024-05-27T07:36:51.896 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "ALERTMANAGER_API_SSL_VERIFY", 2024-05-27T07:36:51.896 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "bool", 2024-05-27T07:36:51.896 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-05-27T07:36:51.896 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 0, 2024-05-27T07:36:51.896 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "True", 2024-05-27T07:36:51.896 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-05-27T07:36:51.896 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-05-27T07:36:51.896 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-05-27T07:36:51.896 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-05-27T07:36:51.896 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-05-27T07:36:51.896 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-05-27T07:36:51.896 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-05-27T07:36:51.896 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-05-27T07:36:51.897 INFO:teuthology.orchestra.run.smithi052.stdout: "AUDIT_API_ENABLED": { 2024-05-27T07:36:51.897 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "AUDIT_API_ENABLED", 2024-05-27T07:36:51.897 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "bool", 2024-05-27T07:36:51.897 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-05-27T07:36:51.897 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 0, 2024-05-27T07:36:51.897 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "False", 2024-05-27T07:36:51.897 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-05-27T07:36:51.897 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-05-27T07:36:51.897 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-05-27T07:36:51.897 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-05-27T07:36:51.897 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-05-27T07:36:51.897 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-05-27T07:36:51.898 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-05-27T07:36:51.898 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-05-27T07:36:51.898 INFO:teuthology.orchestra.run.smithi052.stdout: "AUDIT_API_LOG_PAYLOAD": { 2024-05-27T07:36:51.898 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "AUDIT_API_LOG_PAYLOAD", 2024-05-27T07:36:51.898 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "bool", 2024-05-27T07:36:51.898 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-05-27T07:36:51.898 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 0, 2024-05-27T07:36:51.898 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "True", 2024-05-27T07:36:51.898 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-05-27T07:36:51.898 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-05-27T07:36:51.898 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-05-27T07:36:51.898 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-05-27T07:36:51.898 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-05-27T07:36:51.899 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-05-27T07:36:51.899 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-05-27T07:36:51.899 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-05-27T07:36:51.899 INFO:teuthology.orchestra.run.smithi052.stdout: "ENABLE_BROWSABLE_API": { 2024-05-27T07:36:51.899 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "ENABLE_BROWSABLE_API", 2024-05-27T07:36:51.899 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "bool", 2024-05-27T07:36:51.899 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-05-27T07:36:51.899 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 0, 2024-05-27T07:36:51.899 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "True", 2024-05-27T07:36:51.899 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-05-27T07:36:51.899 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-05-27T07:36:51.899 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-05-27T07:36:51.899 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-05-27T07:36:51.900 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-05-27T07:36:51.900 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-05-27T07:36:51.900 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-05-27T07:36:51.900 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-05-27T07:36:51.900 INFO:teuthology.orchestra.run.smithi052.stdout: "FEATURE_TOGGLE_CEPHFS": { 2024-05-27T07:36:51.900 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "FEATURE_TOGGLE_CEPHFS", 2024-05-27T07:36:51.900 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "bool", 2024-05-27T07:36:51.900 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-05-27T07:36:51.900 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 0, 2024-05-27T07:36:51.900 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "True", 2024-05-27T07:36:51.900 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-05-27T07:36:51.900 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-05-27T07:36:51.900 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-05-27T07:36:51.901 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-05-27T07:36:51.901 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-05-27T07:36:51.901 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-05-27T07:36:51.901 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-05-27T07:36:51.901 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-05-27T07:36:51.901 INFO:teuthology.orchestra.run.smithi052.stdout: "FEATURE_TOGGLE_DASHBOARD": { 2024-05-27T07:36:51.901 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "FEATURE_TOGGLE_DASHBOARD", 2024-05-27T07:36:51.901 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "bool", 2024-05-27T07:36:51.901 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-05-27T07:36:51.901 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 0, 2024-05-27T07:36:51.901 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "True", 2024-05-27T07:36:51.901 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-05-27T07:36:51.901 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-05-27T07:36:51.902 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-05-27T07:36:51.902 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-05-27T07:36:51.902 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-05-27T07:36:51.902 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-05-27T07:36:51.902 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-05-27T07:36:51.902 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-05-27T07:36:51.902 INFO:teuthology.orchestra.run.smithi052.stdout: "FEATURE_TOGGLE_ISCSI": { 2024-05-27T07:36:51.902 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "FEATURE_TOGGLE_ISCSI", 2024-05-27T07:36:51.902 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "bool", 2024-05-27T07:36:51.902 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-05-27T07:36:51.902 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 0, 2024-05-27T07:36:51.902 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "True", 2024-05-27T07:36:51.902 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-05-27T07:36:51.903 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-05-27T07:36:51.903 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-05-27T07:36:51.903 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-05-27T07:36:51.903 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-05-27T07:36:51.903 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-05-27T07:36:51.903 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-05-27T07:36:51.903 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-05-27T07:36:51.903 INFO:teuthology.orchestra.run.smithi052.stdout: "FEATURE_TOGGLE_MIRRORING": { 2024-05-27T07:36:51.903 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "FEATURE_TOGGLE_MIRRORING", 2024-05-27T07:36:51.903 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "bool", 2024-05-27T07:36:51.903 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-05-27T07:36:51.903 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 0, 2024-05-27T07:36:51.903 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "True", 2024-05-27T07:36:51.903 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-05-27T07:36:51.904 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-05-27T07:36:51.904 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-05-27T07:36:51.904 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-05-27T07:36:51.904 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-05-27T07:36:51.904 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-05-27T07:36:51.904 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-05-27T07:36:51.904 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-05-27T07:36:51.904 INFO:teuthology.orchestra.run.smithi052.stdout: "FEATURE_TOGGLE_NFS": { 2024-05-27T07:36:51.904 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "FEATURE_TOGGLE_NFS", 2024-05-27T07:36:51.904 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "bool", 2024-05-27T07:36:51.904 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-05-27T07:36:51.904 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 0, 2024-05-27T07:36:51.904 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "True", 2024-05-27T07:36:51.905 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-05-27T07:36:51.905 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-05-27T07:36:51.905 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-05-27T07:36:51.905 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-05-27T07:36:51.905 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-05-27T07:36:51.905 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-05-27T07:36:51.905 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-05-27T07:36:51.905 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-05-27T07:36:51.905 INFO:teuthology.orchestra.run.smithi052.stdout: "FEATURE_TOGGLE_RBD": { 2024-05-27T07:36:51.905 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "FEATURE_TOGGLE_RBD", 2024-05-27T07:36:51.905 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "bool", 2024-05-27T07:36:51.905 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-05-27T07:36:51.905 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 0, 2024-05-27T07:36:51.906 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "True", 2024-05-27T07:36:51.906 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-05-27T07:36:51.906 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-05-27T07:36:51.906 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-05-27T07:36:51.906 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-05-27T07:36:51.906 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-05-27T07:36:51.906 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-05-27T07:36:51.906 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-05-27T07:36:51.906 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-05-27T07:36:51.906 INFO:teuthology.orchestra.run.smithi052.stdout: "FEATURE_TOGGLE_RGW": { 2024-05-27T07:36:51.906 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "FEATURE_TOGGLE_RGW", 2024-05-27T07:36:51.906 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "bool", 2024-05-27T07:36:51.906 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-05-27T07:36:51.907 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 0, 2024-05-27T07:36:51.907 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "True", 2024-05-27T07:36:51.907 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-05-27T07:36:51.907 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-05-27T07:36:51.907 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-05-27T07:36:51.907 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-05-27T07:36:51.907 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-05-27T07:36:51.907 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-05-27T07:36:51.907 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-05-27T07:36:51.907 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-05-27T07:36:51.907 INFO:teuthology.orchestra.run.smithi052.stdout: "GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE": { 2024-05-27T07:36:51.907 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE", 2024-05-27T07:36:51.907 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "str", 2024-05-27T07:36:51.907 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-05-27T07:36:51.908 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 0, 2024-05-27T07:36:51.908 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "", 2024-05-27T07:36:51.908 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-05-27T07:36:51.908 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-05-27T07:36:51.908 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-05-27T07:36:51.908 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-05-27T07:36:51.908 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-05-27T07:36:51.908 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-05-27T07:36:51.908 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-05-27T07:36:51.908 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-05-27T07:36:51.908 INFO:teuthology.orchestra.run.smithi052.stdout: "GRAFANA_API_PASSWORD": { 2024-05-27T07:36:51.908 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "GRAFANA_API_PASSWORD", 2024-05-27T07:36:51.908 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "str", 2024-05-27T07:36:51.909 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-05-27T07:36:51.909 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 0, 2024-05-27T07:36:51.909 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "admin", 2024-05-27T07:36:51.909 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-05-27T07:36:51.909 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-05-27T07:36:51.909 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-05-27T07:36:51.909 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-05-27T07:36:51.909 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-05-27T07:36:51.909 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-05-27T07:36:51.909 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-05-27T07:36:51.909 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-05-27T07:36:51.909 INFO:teuthology.orchestra.run.smithi052.stdout: "GRAFANA_API_SSL_VERIFY": { 2024-05-27T07:36:51.909 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "GRAFANA_API_SSL_VERIFY", 2024-05-27T07:36:51.910 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "bool", 2024-05-27T07:36:51.910 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-05-27T07:36:51.910 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 0, 2024-05-27T07:36:51.910 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "True", 2024-05-27T07:36:51.910 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-05-27T07:36:51.910 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-05-27T07:36:51.910 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-05-27T07:36:51.910 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-05-27T07:36:51.910 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-05-27T07:36:51.910 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-05-27T07:36:51.910 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-05-27T07:36:51.911 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-05-27T07:36:51.911 INFO:teuthology.orchestra.run.smithi052.stdout: "GRAFANA_API_URL": { 2024-05-27T07:36:51.911 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "GRAFANA_API_URL", 2024-05-27T07:36:51.911 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "str", 2024-05-27T07:36:51.911 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-05-27T07:36:51.911 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 0, 2024-05-27T07:36:51.911 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "", 2024-05-27T07:36:51.911 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-05-27T07:36:51.911 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-05-27T07:36:51.911 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-05-27T07:36:51.911 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-05-27T07:36:51.911 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-05-27T07:36:51.911 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-05-27T07:36:51.912 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-05-27T07:36:51.912 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-05-27T07:36:51.912 INFO:teuthology.orchestra.run.smithi052.stdout: "GRAFANA_API_USERNAME": { 2024-05-27T07:36:51.912 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "GRAFANA_API_USERNAME", 2024-05-27T07:36:51.912 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "str", 2024-05-27T07:36:51.912 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-05-27T07:36:51.912 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 0, 2024-05-27T07:36:51.912 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "admin", 2024-05-27T07:36:51.912 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-05-27T07:36:51.912 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-05-27T07:36:51.912 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-05-27T07:36:51.912 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-05-27T07:36:51.912 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-05-27T07:36:51.913 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-05-27T07:36:51.913 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-05-27T07:36:51.913 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-05-27T07:36:51.913 INFO:teuthology.orchestra.run.smithi052.stdout: "GRAFANA_FRONTEND_API_URL": { 2024-05-27T07:36:51.913 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "GRAFANA_FRONTEND_API_URL", 2024-05-27T07:36:51.913 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "str", 2024-05-27T07:36:51.913 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-05-27T07:36:51.913 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 0, 2024-05-27T07:36:51.913 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "", 2024-05-27T07:36:51.913 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-05-27T07:36:51.913 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-05-27T07:36:51.913 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-05-27T07:36:51.913 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-05-27T07:36:51.914 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-05-27T07:36:51.914 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-05-27T07:36:51.914 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-05-27T07:36:51.914 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-05-27T07:36:51.914 INFO:teuthology.orchestra.run.smithi052.stdout: "GRAFANA_UPDATE_DASHBOARDS": { 2024-05-27T07:36:51.914 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "GRAFANA_UPDATE_DASHBOARDS", 2024-05-27T07:36:51.914 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "bool", 2024-05-27T07:36:51.914 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-05-27T07:36:51.914 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 0, 2024-05-27T07:36:51.914 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "False", 2024-05-27T07:36:51.914 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-05-27T07:36:51.914 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-05-27T07:36:51.914 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-05-27T07:36:51.915 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-05-27T07:36:51.915 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-05-27T07:36:51.915 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-05-27T07:36:51.915 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-05-27T07:36:51.915 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-05-27T07:36:51.915 INFO:teuthology.orchestra.run.smithi052.stdout: "ISCSI_API_SSL_VERIFICATION": { 2024-05-27T07:36:51.915 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "ISCSI_API_SSL_VERIFICATION", 2024-05-27T07:36:51.915 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "bool", 2024-05-27T07:36:51.915 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-05-27T07:36:51.915 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 0, 2024-05-27T07:36:51.915 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "True", 2024-05-27T07:36:51.915 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-05-27T07:36:51.915 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-05-27T07:36:51.916 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-05-27T07:36:51.916 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-05-27T07:36:51.916 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-05-27T07:36:51.916 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-05-27T07:36:51.916 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-05-27T07:36:51.916 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-05-27T07:36:51.916 INFO:teuthology.orchestra.run.smithi052.stdout: "ISSUE_TRACKER_API_KEY": { 2024-05-27T07:36:51.916 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "ISSUE_TRACKER_API_KEY", 2024-05-27T07:36:51.916 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "str", 2024-05-27T07:36:51.916 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-05-27T07:36:51.916 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 0, 2024-05-27T07:36:51.916 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "", 2024-05-27T07:36:51.916 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-05-27T07:36:51.917 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-05-27T07:36:51.917 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-05-27T07:36:51.917 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-05-27T07:36:51.917 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-05-27T07:36:51.917 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-05-27T07:36:51.917 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-05-27T07:36:51.917 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-05-27T07:36:51.917 INFO:teuthology.orchestra.run.smithi052.stdout: "MANAGED_BY_CLUSTERS": { 2024-05-27T07:36:51.917 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "MANAGED_BY_CLUSTERS", 2024-05-27T07:36:51.917 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "str", 2024-05-27T07:36:51.917 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-05-27T07:36:51.917 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 0, 2024-05-27T07:36:51.917 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "[]", 2024-05-27T07:36:51.918 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-05-27T07:36:51.918 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-05-27T07:36:51.918 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-05-27T07:36:51.918 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-05-27T07:36:51.918 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-05-27T07:36:51.918 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-05-27T07:36:51.918 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-05-27T07:36:51.918 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-05-27T07:36:51.918 INFO:teuthology.orchestra.run.smithi052.stdout: "MULTICLUSTER_CONFIG": { 2024-05-27T07:36:51.918 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "MULTICLUSTER_CONFIG", 2024-05-27T07:36:51.918 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "str", 2024-05-27T07:36:51.918 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-05-27T07:36:51.919 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 0, 2024-05-27T07:36:51.919 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "{}", 2024-05-27T07:36:51.919 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-05-27T07:36:51.919 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-05-27T07:36:51.919 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-05-27T07:36:51.919 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-05-27T07:36:51.919 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-05-27T07:36:51.919 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-05-27T07:36:51.919 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-05-27T07:36:51.919 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-05-27T07:36:51.919 INFO:teuthology.orchestra.run.smithi052.stdout: "PROMETHEUS_API_HOST": { 2024-05-27T07:36:51.919 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "PROMETHEUS_API_HOST", 2024-05-27T07:36:51.919 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "str", 2024-05-27T07:36:51.919 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-05-27T07:36:51.920 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 0, 2024-05-27T07:36:51.920 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "", 2024-05-27T07:36:51.920 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-05-27T07:36:51.920 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-05-27T07:36:51.920 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-05-27T07:36:51.920 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-05-27T07:36:51.920 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-05-27T07:36:51.920 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-05-27T07:36:51.920 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-05-27T07:36:51.920 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-05-27T07:36:51.920 INFO:teuthology.orchestra.run.smithi052.stdout: "PROMETHEUS_API_SSL_VERIFY": { 2024-05-27T07:36:51.920 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "PROMETHEUS_API_SSL_VERIFY", 2024-05-27T07:36:51.920 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "bool", 2024-05-27T07:36:51.921 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-05-27T07:36:51.921 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 0, 2024-05-27T07:36:51.921 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "True", 2024-05-27T07:36:51.921 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-05-27T07:36:51.921 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-05-27T07:36:51.921 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-05-27T07:36:51.921 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-05-27T07:36:51.921 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-05-27T07:36:51.921 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-05-27T07:36:51.921 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-05-27T07:36:51.921 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-05-27T07:36:51.921 INFO:teuthology.orchestra.run.smithi052.stdout: "PWD_POLICY_CHECK_COMPLEXITY_ENABLED": { 2024-05-27T07:36:51.921 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "PWD_POLICY_CHECK_COMPLEXITY_ENABLED", 2024-05-27T07:36:51.922 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "bool", 2024-05-27T07:36:51.922 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-05-27T07:36:51.922 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 0, 2024-05-27T07:36:51.922 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "False", 2024-05-27T07:36:51.922 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-05-27T07:36:51.922 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-05-27T07:36:51.922 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-05-27T07:36:51.922 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-05-27T07:36:51.922 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-05-27T07:36:51.922 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-05-27T07:36:51.922 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-05-27T07:36:51.922 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-05-27T07:36:51.922 INFO:teuthology.orchestra.run.smithi052.stdout: "PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED": { 2024-05-27T07:36:51.923 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED", 2024-05-27T07:36:51.923 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "bool", 2024-05-27T07:36:51.923 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-05-27T07:36:51.923 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 0, 2024-05-27T07:36:51.923 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "False", 2024-05-27T07:36:51.923 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-05-27T07:36:51.923 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-05-27T07:36:51.923 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-05-27T07:36:51.923 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-05-27T07:36:51.923 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-05-27T07:36:51.923 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-05-27T07:36:51.923 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-05-27T07:36:51.923 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-05-27T07:36:51.924 INFO:teuthology.orchestra.run.smithi052.stdout: "PWD_POLICY_CHECK_LENGTH_ENABLED": { 2024-05-27T07:36:51.924 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "PWD_POLICY_CHECK_LENGTH_ENABLED", 2024-05-27T07:36:51.924 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "bool", 2024-05-27T07:36:51.924 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-05-27T07:36:51.924 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 0, 2024-05-27T07:36:51.924 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "True", 2024-05-27T07:36:51.924 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-05-27T07:36:51.924 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-05-27T07:36:51.924 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-05-27T07:36:51.924 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-05-27T07:36:51.924 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-05-27T07:36:51.924 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-05-27T07:36:51.925 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-05-27T07:36:51.925 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-05-27T07:36:51.925 INFO:teuthology.orchestra.run.smithi052.stdout: "PWD_POLICY_CHECK_OLDPWD_ENABLED": { 2024-05-27T07:36:51.925 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "PWD_POLICY_CHECK_OLDPWD_ENABLED", 2024-05-27T07:36:51.925 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "bool", 2024-05-27T07:36:51.925 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-05-27T07:36:51.925 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 0, 2024-05-27T07:36:51.925 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "True", 2024-05-27T07:36:51.925 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-05-27T07:36:51.925 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-05-27T07:36:51.925 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-05-27T07:36:51.925 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-05-27T07:36:51.925 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-05-27T07:36:51.926 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-05-27T07:36:51.926 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-05-27T07:36:51.926 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-05-27T07:36:51.926 INFO:teuthology.orchestra.run.smithi052.stdout: "PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED": { 2024-05-27T07:36:51.926 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED", 2024-05-27T07:36:51.926 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "bool", 2024-05-27T07:36:51.926 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-05-27T07:36:51.926 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 0, 2024-05-27T07:36:51.926 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "False", 2024-05-27T07:36:51.926 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-05-27T07:36:51.926 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-05-27T07:36:51.926 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-05-27T07:36:51.927 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-05-27T07:36:51.927 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-05-27T07:36:51.927 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-05-27T07:36:51.927 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-05-27T07:36:51.927 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-05-27T07:36:51.927 INFO:teuthology.orchestra.run.smithi052.stdout: "PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED": { 2024-05-27T07:36:51.927 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED", 2024-05-27T07:36:51.927 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "bool", 2024-05-27T07:36:51.927 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-05-27T07:36:51.927 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 0, 2024-05-27T07:36:51.927 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "False", 2024-05-27T07:36:51.927 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-05-27T07:36:51.927 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-05-27T07:36:51.928 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-05-27T07:36:51.928 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-05-27T07:36:51.928 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-05-27T07:36:51.928 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-05-27T07:36:51.928 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-05-27T07:36:51.928 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-05-27T07:36:51.928 INFO:teuthology.orchestra.run.smithi052.stdout: "PWD_POLICY_CHECK_USERNAME_ENABLED": { 2024-05-27T07:36:51.928 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "PWD_POLICY_CHECK_USERNAME_ENABLED", 2024-05-27T07:36:51.928 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "bool", 2024-05-27T07:36:51.928 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-05-27T07:36:51.928 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 0, 2024-05-27T07:36:51.928 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "False", 2024-05-27T07:36:51.928 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-05-27T07:36:51.929 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-05-27T07:36:51.929 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-05-27T07:36:51.929 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-05-27T07:36:51.929 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-05-27T07:36:51.929 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-05-27T07:36:51.929 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-05-27T07:36:51.929 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-05-27T07:36:51.929 INFO:teuthology.orchestra.run.smithi052.stdout: "PWD_POLICY_ENABLED": { 2024-05-27T07:36:51.929 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "PWD_POLICY_ENABLED", 2024-05-27T07:36:51.929 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "bool", 2024-05-27T07:36:51.929 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-05-27T07:36:51.929 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 0, 2024-05-27T07:36:51.929 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "True", 2024-05-27T07:36:51.930 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-05-27T07:36:51.930 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-05-27T07:36:51.930 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-05-27T07:36:51.930 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-05-27T07:36:51.930 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-05-27T07:36:51.930 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-05-27T07:36:51.930 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-05-27T07:36:51.930 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-05-27T07:36:51.930 INFO:teuthology.orchestra.run.smithi052.stdout: "PWD_POLICY_EXCLUSION_LIST": { 2024-05-27T07:36:51.930 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "PWD_POLICY_EXCLUSION_LIST", 2024-05-27T07:36:51.930 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "str", 2024-05-27T07:36:51.930 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-05-27T07:36:51.930 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 0, 2024-05-27T07:36:51.931 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps", 2024-05-27T07:36:51.931 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-05-27T07:36:51.931 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-05-27T07:36:51.931 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-05-27T07:36:51.931 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-05-27T07:36:51.931 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-05-27T07:36:51.931 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-05-27T07:36:51.931 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-05-27T07:36:51.931 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-05-27T07:36:51.931 INFO:teuthology.orchestra.run.smithi052.stdout: "PWD_POLICY_MIN_COMPLEXITY": { 2024-05-27T07:36:51.931 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "PWD_POLICY_MIN_COMPLEXITY", 2024-05-27T07:36:51.931 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "int", 2024-05-27T07:36:51.931 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-05-27T07:36:51.932 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 0, 2024-05-27T07:36:51.932 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "10", 2024-05-27T07:36:51.932 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-05-27T07:36:51.932 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-05-27T07:36:51.932 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-05-27T07:36:51.932 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-05-27T07:36:51.932 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-05-27T07:36:51.932 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-05-27T07:36:51.932 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-05-27T07:36:51.932 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-05-27T07:36:51.932 INFO:teuthology.orchestra.run.smithi052.stdout: "PWD_POLICY_MIN_LENGTH": { 2024-05-27T07:36:51.932 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "PWD_POLICY_MIN_LENGTH", 2024-05-27T07:36:51.932 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "int", 2024-05-27T07:36:51.932 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-05-27T07:36:51.933 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 0, 2024-05-27T07:36:51.933 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "8", 2024-05-27T07:36:51.933 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-05-27T07:36:51.933 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-05-27T07:36:51.933 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-05-27T07:36:51.933 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-05-27T07:36:51.933 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-05-27T07:36:51.933 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-05-27T07:36:51.933 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-05-27T07:36:51.933 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-05-27T07:36:51.933 INFO:teuthology.orchestra.run.smithi052.stdout: "REST_REQUESTS_TIMEOUT": { 2024-05-27T07:36:51.933 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "REST_REQUESTS_TIMEOUT", 2024-05-27T07:36:51.933 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "int", 2024-05-27T07:36:51.934 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-05-27T07:36:51.934 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 0, 2024-05-27T07:36:51.934 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "45", 2024-05-27T07:36:51.934 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-05-27T07:36:51.934 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-05-27T07:36:51.934 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-05-27T07:36:51.934 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-05-27T07:36:51.934 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-05-27T07:36:51.934 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-05-27T07:36:51.934 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-05-27T07:36:51.934 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-05-27T07:36:51.934 INFO:teuthology.orchestra.run.smithi052.stdout: "RGW_API_ACCESS_KEY": { 2024-05-27T07:36:51.934 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "RGW_API_ACCESS_KEY", 2024-05-27T07:36:51.935 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "str", 2024-05-27T07:36:51.935 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-05-27T07:36:51.935 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 0, 2024-05-27T07:36:51.935 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "", 2024-05-27T07:36:51.935 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-05-27T07:36:51.935 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-05-27T07:36:51.935 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-05-27T07:36:51.935 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-05-27T07:36:51.935 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-05-27T07:36:51.935 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-05-27T07:36:51.935 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-05-27T07:36:51.935 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-05-27T07:36:51.935 INFO:teuthology.orchestra.run.smithi052.stdout: "RGW_API_ADMIN_RESOURCE": { 2024-05-27T07:36:51.936 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "RGW_API_ADMIN_RESOURCE", 2024-05-27T07:36:51.936 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "str", 2024-05-27T07:36:51.936 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-05-27T07:36:51.936 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 0, 2024-05-27T07:36:51.936 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "admin", 2024-05-27T07:36:51.936 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-05-27T07:36:51.936 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-05-27T07:36:51.936 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-05-27T07:36:51.936 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-05-27T07:36:51.936 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-05-27T07:36:51.936 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-05-27T07:36:51.936 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-05-27T07:36:51.936 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-05-27T07:36:51.936 INFO:teuthology.orchestra.run.smithi052.stdout: "RGW_API_SECRET_KEY": { 2024-05-27T07:36:51.937 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "RGW_API_SECRET_KEY", 2024-05-27T07:36:51.937 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "str", 2024-05-27T07:36:51.937 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-05-27T07:36:51.937 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 0, 2024-05-27T07:36:51.937 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "", 2024-05-27T07:36:51.937 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-05-27T07:36:51.937 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-05-27T07:36:51.937 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-05-27T07:36:51.937 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-05-27T07:36:51.937 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-05-27T07:36:51.937 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-05-27T07:36:51.937 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-05-27T07:36:51.937 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-05-27T07:36:51.938 INFO:teuthology.orchestra.run.smithi052.stdout: "RGW_API_SSL_VERIFY": { 2024-05-27T07:36:51.938 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "RGW_API_SSL_VERIFY", 2024-05-27T07:36:51.938 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "bool", 2024-05-27T07:36:51.938 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-05-27T07:36:51.938 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 0, 2024-05-27T07:36:51.938 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "True", 2024-05-27T07:36:51.938 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-05-27T07:36:51.938 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-05-27T07:36:51.938 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-05-27T07:36:51.938 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-05-27T07:36:51.938 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-05-27T07:36:51.939 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-05-27T07:36:51.939 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-05-27T07:36:51.939 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-05-27T07:36:51.939 INFO:teuthology.orchestra.run.smithi052.stdout: "UNSAFE_TLS_v1_2": { 2024-05-27T07:36:51.939 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "UNSAFE_TLS_v1_2", 2024-05-27T07:36:51.939 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "bool", 2024-05-27T07:36:51.939 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-05-27T07:36:51.939 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 0, 2024-05-27T07:36:51.939 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "False", 2024-05-27T07:36:51.939 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-05-27T07:36:51.939 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-05-27T07:36:51.939 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-05-27T07:36:51.939 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-05-27T07:36:51.940 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-05-27T07:36:51.940 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-05-27T07:36:51.940 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-05-27T07:36:51.940 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-05-27T07:36:51.940 INFO:teuthology.orchestra.run.smithi052.stdout: "USER_PWD_EXPIRATION_SPAN": { 2024-05-27T07:36:51.940 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "USER_PWD_EXPIRATION_SPAN", 2024-05-27T07:36:51.940 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "int", 2024-05-27T07:36:51.940 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-05-27T07:36:51.940 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 0, 2024-05-27T07:36:51.940 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "0", 2024-05-27T07:36:51.940 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-05-27T07:36:51.940 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-05-27T07:36:51.940 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-05-27T07:36:51.941 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-05-27T07:36:51.941 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-05-27T07:36:51.941 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-05-27T07:36:51.941 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-05-27T07:36:51.941 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-05-27T07:36:51.941 INFO:teuthology.orchestra.run.smithi052.stdout: "USER_PWD_EXPIRATION_WARNING_1": { 2024-05-27T07:36:51.941 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "USER_PWD_EXPIRATION_WARNING_1", 2024-05-27T07:36:51.941 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "int", 2024-05-27T07:36:51.941 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-05-27T07:36:51.941 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 0, 2024-05-27T07:36:51.941 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "10", 2024-05-27T07:36:51.941 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-05-27T07:36:51.941 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-05-27T07:36:51.942 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-05-27T07:36:51.942 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-05-27T07:36:51.942 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-05-27T07:36:51.942 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-05-27T07:36:51.942 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-05-27T07:36:51.942 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-05-27T07:36:51.942 INFO:teuthology.orchestra.run.smithi052.stdout: "USER_PWD_EXPIRATION_WARNING_2": { 2024-05-27T07:36:51.942 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "USER_PWD_EXPIRATION_WARNING_2", 2024-05-27T07:36:51.942 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "int", 2024-05-27T07:36:51.942 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-05-27T07:36:51.942 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 0, 2024-05-27T07:36:51.942 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "5", 2024-05-27T07:36:51.942 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-05-27T07:36:51.943 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-05-27T07:36:51.943 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-05-27T07:36:51.943 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-05-27T07:36:51.943 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-05-27T07:36:51.943 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-05-27T07:36:51.943 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-05-27T07:36:51.943 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-05-27T07:36:51.943 INFO:teuthology.orchestra.run.smithi052.stdout: "cross_origin_url": { 2024-05-27T07:36:51.943 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "cross_origin_url", 2024-05-27T07:36:51.943 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "str", 2024-05-27T07:36:51.943 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-05-27T07:36:51.943 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 0, 2024-05-27T07:36:51.943 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "", 2024-05-27T07:36:51.944 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-05-27T07:36:51.944 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-05-27T07:36:51.944 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-05-27T07:36:51.944 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-05-27T07:36:51.944 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-05-27T07:36:51.944 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-05-27T07:36:51.944 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-05-27T07:36:51.944 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-05-27T07:36:51.944 INFO:teuthology.orchestra.run.smithi052.stdout: "crt_file": { 2024-05-27T07:36:51.944 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "crt_file", 2024-05-27T07:36:51.944 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "str", 2024-05-27T07:36:51.944 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-05-27T07:36:51.944 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 0, 2024-05-27T07:36:51.945 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "", 2024-05-27T07:36:51.945 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-05-27T07:36:51.945 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-05-27T07:36:51.945 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-05-27T07:36:51.945 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-05-27T07:36:51.945 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-05-27T07:36:51.945 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-05-27T07:36:51.945 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-05-27T07:36:51.945 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-05-27T07:36:51.945 INFO:teuthology.orchestra.run.smithi052.stdout: "debug": { 2024-05-27T07:36:51.945 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "debug", 2024-05-27T07:36:51.945 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "bool", 2024-05-27T07:36:51.945 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-05-27T07:36:51.945 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 0, 2024-05-27T07:36:51.946 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "False", 2024-05-27T07:36:51.946 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-05-27T07:36:51.946 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-05-27T07:36:51.946 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-05-27T07:36:51.946 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "Enable/disable debug options", 2024-05-27T07:36:51.946 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-05-27T07:36:51.946 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-05-27T07:36:51.946 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-05-27T07:36:51.946 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-05-27T07:36:51.946 INFO:teuthology.orchestra.run.smithi052.stdout: "jwt_token_ttl": { 2024-05-27T07:36:51.946 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "jwt_token_ttl", 2024-05-27T07:36:51.946 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "int", 2024-05-27T07:36:51.947 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-05-27T07:36:51.947 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 0, 2024-05-27T07:36:51.947 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "28800", 2024-05-27T07:36:51.947 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-05-27T07:36:51.947 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-05-27T07:36:51.947 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-05-27T07:36:51.947 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-05-27T07:36:51.947 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-05-27T07:36:51.947 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-05-27T07:36:51.947 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-05-27T07:36:51.947 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-05-27T07:36:51.947 INFO:teuthology.orchestra.run.smithi052.stdout: "key_file": { 2024-05-27T07:36:51.947 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "key_file", 2024-05-27T07:36:51.947 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "str", 2024-05-27T07:36:51.948 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-05-27T07:36:51.948 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 0, 2024-05-27T07:36:51.948 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "", 2024-05-27T07:36:51.948 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-05-27T07:36:51.948 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-05-27T07:36:51.948 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-05-27T07:36:51.948 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-05-27T07:36:51.948 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-05-27T07:36:51.948 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-05-27T07:36:51.948 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-05-27T07:36:51.948 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-05-27T07:36:51.948 INFO:teuthology.orchestra.run.smithi052.stdout: "log_level": { 2024-05-27T07:36:51.948 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "log_level", 2024-05-27T07:36:51.949 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "str", 2024-05-27T07:36:51.949 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-05-27T07:36:51.949 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 1, 2024-05-27T07:36:51.949 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "", 2024-05-27T07:36:51.949 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-05-27T07:36:51.949 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-05-27T07:36:51.949 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [ 2024-05-27T07:36:51.949 INFO:teuthology.orchestra.run.smithi052.stdout: "", 2024-05-27T07:36:51.949 INFO:teuthology.orchestra.run.smithi052.stdout: "critical", 2024-05-27T07:36:51.949 INFO:teuthology.orchestra.run.smithi052.stdout: "debug", 2024-05-27T07:36:51.949 INFO:teuthology.orchestra.run.smithi052.stdout: "error", 2024-05-27T07:36:51.949 INFO:teuthology.orchestra.run.smithi052.stdout: "info", 2024-05-27T07:36:51.949 INFO:teuthology.orchestra.run.smithi052.stdout: "warning" 2024-05-27T07:36:51.950 INFO:teuthology.orchestra.run.smithi052.stdout: ], 2024-05-27T07:36:51.950 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-05-27T07:36:51.950 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-05-27T07:36:51.950 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-05-27T07:36:51.950 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-05-27T07:36:51.950 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-05-27T07:36:51.950 INFO:teuthology.orchestra.run.smithi052.stdout: "log_to_cluster": { 2024-05-27T07:36:51.950 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "log_to_cluster", 2024-05-27T07:36:51.950 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "bool", 2024-05-27T07:36:51.950 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-05-27T07:36:51.950 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 1, 2024-05-27T07:36:51.950 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "False", 2024-05-27T07:36:51.950 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-05-27T07:36:51.951 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-05-27T07:36:51.951 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-05-27T07:36:51.951 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-05-27T07:36:51.951 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-05-27T07:36:51.951 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-05-27T07:36:51.951 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-05-27T07:36:51.951 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-05-27T07:36:51.951 INFO:teuthology.orchestra.run.smithi052.stdout: "log_to_cluster_level": { 2024-05-27T07:36:51.951 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "log_to_cluster_level", 2024-05-27T07:36:51.951 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "str", 2024-05-27T07:36:51.951 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-05-27T07:36:51.951 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 1, 2024-05-27T07:36:51.951 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "info", 2024-05-27T07:36:51.952 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-05-27T07:36:51.952 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-05-27T07:36:51.952 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [ 2024-05-27T07:36:51.952 INFO:teuthology.orchestra.run.smithi052.stdout: "", 2024-05-27T07:36:51.952 INFO:teuthology.orchestra.run.smithi052.stdout: "critical", 2024-05-27T07:36:51.952 INFO:teuthology.orchestra.run.smithi052.stdout: "debug", 2024-05-27T07:36:51.952 INFO:teuthology.orchestra.run.smithi052.stdout: "error", 2024-05-27T07:36:51.952 INFO:teuthology.orchestra.run.smithi052.stdout: "info", 2024-05-27T07:36:51.952 INFO:teuthology.orchestra.run.smithi052.stdout: "warning" 2024-05-27T07:36:51.952 INFO:teuthology.orchestra.run.smithi052.stdout: ], 2024-05-27T07:36:51.952 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-05-27T07:36:51.952 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-05-27T07:36:51.953 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-05-27T07:36:51.953 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-05-27T07:36:51.953 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-05-27T07:36:51.953 INFO:teuthology.orchestra.run.smithi052.stdout: "log_to_file": { 2024-05-27T07:36:51.953 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "log_to_file", 2024-05-27T07:36:51.953 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "bool", 2024-05-27T07:36:51.953 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-05-27T07:36:51.953 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 1, 2024-05-27T07:36:51.953 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "False", 2024-05-27T07:36:51.953 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-05-27T07:36:51.953 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-05-27T07:36:51.953 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-05-27T07:36:51.953 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-05-27T07:36:51.954 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-05-27T07:36:51.954 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-05-27T07:36:51.954 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-05-27T07:36:51.954 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-05-27T07:36:51.954 INFO:teuthology.orchestra.run.smithi052.stdout: "motd": { 2024-05-27T07:36:51.954 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "motd", 2024-05-27T07:36:51.954 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "str", 2024-05-27T07:36:51.954 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-05-27T07:36:51.954 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 0, 2024-05-27T07:36:51.954 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "", 2024-05-27T07:36:51.954 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-05-27T07:36:51.954 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-05-27T07:36:51.955 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-05-27T07:36:51.955 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "The message of the day", 2024-05-27T07:36:51.955 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-05-27T07:36:51.955 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-05-27T07:36:51.955 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-05-27T07:36:51.955 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-05-27T07:36:51.955 INFO:teuthology.orchestra.run.smithi052.stdout: "redirect_resolve_ip_addr": { 2024-05-27T07:36:51.955 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "redirect_resolve_ip_addr", 2024-05-27T07:36:51.955 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "bool", 2024-05-27T07:36:51.955 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-05-27T07:36:51.955 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 0, 2024-05-27T07:36:51.955 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "False", 2024-05-27T07:36:51.955 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-05-27T07:36:51.956 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-05-27T07:36:51.956 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-05-27T07:36:51.956 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-05-27T07:36:51.956 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-05-27T07:36:51.956 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-05-27T07:36:51.956 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-05-27T07:36:51.956 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-05-27T07:36:51.956 INFO:teuthology.orchestra.run.smithi052.stdout: "server_addr": { 2024-05-27T07:36:51.956 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "server_addr", 2024-05-27T07:36:51.956 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "str", 2024-05-27T07:36:51.956 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-05-27T07:36:51.956 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 0, 2024-05-27T07:36:51.956 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "::", 2024-05-27T07:36:51.957 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-05-27T07:36:51.957 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-05-27T07:36:51.957 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-05-27T07:36:51.957 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-05-27T07:36:51.957 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-05-27T07:36:51.957 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-05-27T07:36:51.957 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-05-27T07:36:51.957 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-05-27T07:36:51.957 INFO:teuthology.orchestra.run.smithi052.stdout: "server_port": { 2024-05-27T07:36:51.957 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "server_port", 2024-05-27T07:36:51.957 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "int", 2024-05-27T07:36:51.957 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-05-27T07:36:51.957 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 0, 2024-05-27T07:36:51.958 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "8080", 2024-05-27T07:36:51.958 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-05-27T07:36:51.958 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-05-27T07:36:51.958 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-05-27T07:36:51.958 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-05-27T07:36:51.958 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-05-27T07:36:51.958 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-05-27T07:36:51.958 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-05-27T07:36:51.958 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-05-27T07:36:51.958 INFO:teuthology.orchestra.run.smithi052.stdout: "sqlite3_killpoint": { 2024-05-27T07:36:51.958 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "sqlite3_killpoint", 2024-05-27T07:36:51.958 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "int", 2024-05-27T07:36:51.958 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "dev", 2024-05-27T07:36:51.959 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 1, 2024-05-27T07:36:51.959 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "0", 2024-05-27T07:36:51.959 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-05-27T07:36:51.959 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-05-27T07:36:51.959 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-05-27T07:36:51.959 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-05-27T07:36:51.959 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-05-27T07:36:51.959 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-05-27T07:36:51.959 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-05-27T07:36:51.959 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-05-27T07:36:51.959 INFO:teuthology.orchestra.run.smithi052.stdout: "ssl": { 2024-05-27T07:36:51.959 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "ssl", 2024-05-27T07:36:51.959 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "bool", 2024-05-27T07:36:51.960 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-05-27T07:36:51.960 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 0, 2024-05-27T07:36:51.962 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "True", 2024-05-27T07:36:51.962 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-05-27T07:36:51.962 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-05-27T07:36:51.962 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-05-27T07:36:51.963 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-05-27T07:36:51.963 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-05-27T07:36:51.963 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-05-27T07:36:51.963 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-05-27T07:36:51.963 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-05-27T07:36:51.963 INFO:teuthology.orchestra.run.smithi052.stdout: "ssl_server_port": { 2024-05-27T07:36:51.963 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "ssl_server_port", 2024-05-27T07:36:51.963 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "int", 2024-05-27T07:36:51.963 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-05-27T07:36:51.963 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 0, 2024-05-27T07:36:51.963 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "8443", 2024-05-27T07:36:51.963 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-05-27T07:36:51.963 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-05-27T07:36:51.964 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-05-27T07:36:51.964 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-05-27T07:36:51.964 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-05-27T07:36:51.964 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-05-27T07:36:51.964 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-05-27T07:36:51.964 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-05-27T07:36:51.964 INFO:teuthology.orchestra.run.smithi052.stdout: "standby_behaviour": { 2024-05-27T07:36:51.964 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "standby_behaviour", 2024-05-27T07:36:51.964 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "str", 2024-05-27T07:36:51.964 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-05-27T07:36:51.964 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 0, 2024-05-27T07:36:51.964 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "redirect", 2024-05-27T07:36:51.964 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-05-27T07:36:51.965 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-05-27T07:36:51.965 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [ 2024-05-27T07:36:51.965 INFO:teuthology.orchestra.run.smithi052.stdout: "error", 2024-05-27T07:36:51.965 INFO:teuthology.orchestra.run.smithi052.stdout: "redirect" 2024-05-27T07:36:51.965 INFO:teuthology.orchestra.run.smithi052.stdout: ], 2024-05-27T07:36:51.965 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-05-27T07:36:51.965 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-05-27T07:36:51.965 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-05-27T07:36:51.965 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-05-27T07:36:51.965 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-05-27T07:36:51.965 INFO:teuthology.orchestra.run.smithi052.stdout: "standby_error_status_code": { 2024-05-27T07:36:51.965 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "standby_error_status_code", 2024-05-27T07:36:51.965 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "int", 2024-05-27T07:36:51.966 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-05-27T07:36:51.966 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 0, 2024-05-27T07:36:51.966 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "500", 2024-05-27T07:36:51.966 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "400", 2024-05-27T07:36:51.966 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "599", 2024-05-27T07:36:51.966 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-05-27T07:36:51.966 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-05-27T07:36:51.966 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-05-27T07:36:51.966 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-05-27T07:36:51.966 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-05-27T07:36:51.966 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-05-27T07:36:51.966 INFO:teuthology.orchestra.run.smithi052.stdout: "url_prefix": { 2024-05-27T07:36:51.966 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "url_prefix", 2024-05-27T07:36:51.967 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "str", 2024-05-27T07:36:51.967 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-05-27T07:36:51.967 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 0, 2024-05-27T07:36:51.967 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "", 2024-05-27T07:36:51.967 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-05-27T07:36:51.967 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-05-27T07:36:51.967 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-05-27T07:36:51.967 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-05-27T07:36:51.967 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-05-27T07:36:51.967 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-05-27T07:36:51.967 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-05-27T07:36:51.967 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-05-27T07:36:51.967 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-05-27T07:36:51.968 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-05-27T07:36:51.968 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-05-27T07:36:51.968 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "devicehealth", 2024-05-27T07:36:51.968 INFO:teuthology.orchestra.run.smithi052.stdout: "can_run": true, 2024-05-27T07:36:51.968 INFO:teuthology.orchestra.run.smithi052.stdout: "error_string": "", 2024-05-27T07:36:51.968 INFO:teuthology.orchestra.run.smithi052.stdout: "module_options": { 2024-05-27T07:36:51.968 INFO:teuthology.orchestra.run.smithi052.stdout: "enable_monitoring": { 2024-05-27T07:36:51.968 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "enable_monitoring", 2024-05-27T07:36:51.968 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "bool", 2024-05-27T07:36:51.968 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-05-27T07:36:51.968 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 1, 2024-05-27T07:36:51.968 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "True", 2024-05-27T07:36:51.968 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-05-27T07:36:51.969 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-05-27T07:36:51.969 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-05-27T07:36:51.969 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "monitor device health metrics", 2024-05-27T07:36:51.969 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-05-27T07:36:51.969 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-05-27T07:36:51.969 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-05-27T07:36:51.969 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-05-27T07:36:51.969 INFO:teuthology.orchestra.run.smithi052.stdout: "log_level": { 2024-05-27T07:36:51.969 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "log_level", 2024-05-27T07:36:51.969 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "str", 2024-05-27T07:36:51.969 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-05-27T07:36:51.969 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 1, 2024-05-27T07:36:51.969 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "", 2024-05-27T07:36:51.969 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-05-27T07:36:51.970 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-05-27T07:36:51.970 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [ 2024-05-27T07:36:51.970 INFO:teuthology.orchestra.run.smithi052.stdout: "", 2024-05-27T07:36:51.970 INFO:teuthology.orchestra.run.smithi052.stdout: "critical", 2024-05-27T07:36:51.970 INFO:teuthology.orchestra.run.smithi052.stdout: "debug", 2024-05-27T07:36:51.970 INFO:teuthology.orchestra.run.smithi052.stdout: "error", 2024-05-27T07:36:51.970 INFO:teuthology.orchestra.run.smithi052.stdout: "info", 2024-05-27T07:36:51.970 INFO:teuthology.orchestra.run.smithi052.stdout: "warning" 2024-05-27T07:36:51.970 INFO:teuthology.orchestra.run.smithi052.stdout: ], 2024-05-27T07:36:51.970 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-05-27T07:36:51.970 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-05-27T07:36:51.971 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-05-27T07:36:51.971 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-05-27T07:36:51.971 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-05-27T07:36:51.971 INFO:teuthology.orchestra.run.smithi052.stdout: "log_to_cluster": { 2024-05-27T07:36:51.971 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "log_to_cluster", 2024-05-27T07:36:51.971 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "bool", 2024-05-27T07:36:51.971 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-05-27T07:36:51.971 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 1, 2024-05-27T07:36:51.971 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "False", 2024-05-27T07:36:51.971 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-05-27T07:36:51.971 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-05-27T07:36:51.971 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-05-27T07:36:51.971 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-05-27T07:36:51.972 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-05-27T07:36:51.972 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-05-27T07:36:51.972 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-05-27T07:36:51.972 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-05-27T07:36:51.972 INFO:teuthology.orchestra.run.smithi052.stdout: "log_to_cluster_level": { 2024-05-27T07:36:51.972 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "log_to_cluster_level", 2024-05-27T07:36:51.972 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "str", 2024-05-27T07:36:51.972 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-05-27T07:36:51.972 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 1, 2024-05-27T07:36:51.972 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "info", 2024-05-27T07:36:51.972 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-05-27T07:36:51.972 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-05-27T07:36:51.972 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [ 2024-05-27T07:36:51.973 INFO:teuthology.orchestra.run.smithi052.stdout: "", 2024-05-27T07:36:51.973 INFO:teuthology.orchestra.run.smithi052.stdout: "critical", 2024-05-27T07:36:51.973 INFO:teuthology.orchestra.run.smithi052.stdout: "debug", 2024-05-27T07:36:51.973 INFO:teuthology.orchestra.run.smithi052.stdout: "error", 2024-05-27T07:36:51.973 INFO:teuthology.orchestra.run.smithi052.stdout: "info", 2024-05-27T07:36:51.973 INFO:teuthology.orchestra.run.smithi052.stdout: "warning" 2024-05-27T07:36:51.973 INFO:teuthology.orchestra.run.smithi052.stdout: ], 2024-05-27T07:36:51.973 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-05-27T07:36:51.973 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-05-27T07:36:51.973 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-05-27T07:36:51.973 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-05-27T07:36:51.973 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-05-27T07:36:51.974 INFO:teuthology.orchestra.run.smithi052.stdout: "log_to_file": { 2024-05-27T07:36:51.974 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "log_to_file", 2024-05-27T07:36:51.974 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "bool", 2024-05-27T07:36:51.974 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-05-27T07:36:51.974 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 1, 2024-05-27T07:36:51.974 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "False", 2024-05-27T07:36:51.974 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-05-27T07:36:51.974 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-05-27T07:36:51.974 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-05-27T07:36:51.974 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-05-27T07:36:51.974 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-05-27T07:36:51.974 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-05-27T07:36:51.974 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-05-27T07:36:51.975 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-05-27T07:36:51.975 INFO:teuthology.orchestra.run.smithi052.stdout: "mark_out_threshold": { 2024-05-27T07:36:51.975 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "mark_out_threshold", 2024-05-27T07:36:51.975 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "secs", 2024-05-27T07:36:51.975 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-05-27T07:36:51.975 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 1, 2024-05-27T07:36:51.975 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "2419200", 2024-05-27T07:36:51.975 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-05-27T07:36:51.975 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-05-27T07:36:51.975 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-05-27T07:36:51.975 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "automatically mark OSD if it may fail before this long", 2024-05-27T07:36:51.975 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-05-27T07:36:51.976 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-05-27T07:36:51.976 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-05-27T07:36:51.976 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-05-27T07:36:51.976 INFO:teuthology.orchestra.run.smithi052.stdout: "pool_name": { 2024-05-27T07:36:51.976 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "pool_name", 2024-05-27T07:36:51.976 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "str", 2024-05-27T07:36:51.976 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-05-27T07:36:51.976 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 1, 2024-05-27T07:36:51.976 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "device_health_metrics", 2024-05-27T07:36:51.976 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-05-27T07:36:51.976 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-05-27T07:36:51.976 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-05-27T07:36:51.976 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "name of pool in which to store device health metrics", 2024-05-27T07:36:51.977 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-05-27T07:36:51.977 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-05-27T07:36:51.977 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-05-27T07:36:51.977 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-05-27T07:36:51.977 INFO:teuthology.orchestra.run.smithi052.stdout: "retention_period": { 2024-05-27T07:36:51.977 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "retention_period", 2024-05-27T07:36:51.977 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "secs", 2024-05-27T07:36:51.977 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-05-27T07:36:51.977 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 1, 2024-05-27T07:36:51.977 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "15552000", 2024-05-27T07:36:51.977 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-05-27T07:36:51.977 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-05-27T07:36:51.977 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-05-27T07:36:51.978 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "how long to retain device health metrics", 2024-05-27T07:36:51.978 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-05-27T07:36:51.978 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-05-27T07:36:51.978 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-05-27T07:36:51.978 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-05-27T07:36:51.978 INFO:teuthology.orchestra.run.smithi052.stdout: "scrape_frequency": { 2024-05-27T07:36:51.978 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "scrape_frequency", 2024-05-27T07:36:51.978 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "secs", 2024-05-27T07:36:51.978 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-05-27T07:36:51.978 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 1, 2024-05-27T07:36:51.978 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "86400", 2024-05-27T07:36:51.978 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-05-27T07:36:51.978 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-05-27T07:36:51.979 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-05-27T07:36:51.979 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "how frequently to scrape device health metrics", 2024-05-27T07:36:51.979 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-05-27T07:36:51.979 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-05-27T07:36:51.979 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-05-27T07:36:51.979 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-05-27T07:36:51.979 INFO:teuthology.orchestra.run.smithi052.stdout: "self_heal": { 2024-05-27T07:36:51.979 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "self_heal", 2024-05-27T07:36:51.979 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "bool", 2024-05-27T07:36:51.979 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-05-27T07:36:51.979 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 1, 2024-05-27T07:36:51.979 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "True", 2024-05-27T07:36:51.979 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-05-27T07:36:51.980 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-05-27T07:36:51.980 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-05-27T07:36:51.980 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "preemptively heal cluster around devices that may fail", 2024-05-27T07:36:51.980 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-05-27T07:36:51.980 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-05-27T07:36:51.980 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-05-27T07:36:51.980 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-05-27T07:36:51.980 INFO:teuthology.orchestra.run.smithi052.stdout: "sleep_interval": { 2024-05-27T07:36:51.980 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "sleep_interval", 2024-05-27T07:36:51.980 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "secs", 2024-05-27T07:36:51.980 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-05-27T07:36:51.980 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 1, 2024-05-27T07:36:51.980 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "600", 2024-05-27T07:36:51.980 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-05-27T07:36:51.981 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-05-27T07:36:51.981 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-05-27T07:36:51.981 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "how frequently to wake up and check device health", 2024-05-27T07:36:51.981 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-05-27T07:36:51.981 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-05-27T07:36:51.981 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-05-27T07:36:51.981 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-05-27T07:36:51.981 INFO:teuthology.orchestra.run.smithi052.stdout: "sqlite3_killpoint": { 2024-05-27T07:36:51.981 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "sqlite3_killpoint", 2024-05-27T07:36:51.981 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "int", 2024-05-27T07:36:51.981 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "dev", 2024-05-27T07:36:51.981 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 1, 2024-05-27T07:36:51.981 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "0", 2024-05-27T07:36:51.982 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-05-27T07:36:51.982 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-05-27T07:36:51.982 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-05-27T07:36:51.982 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-05-27T07:36:51.982 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-05-27T07:36:51.982 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-05-27T07:36:51.982 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-05-27T07:36:51.982 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-05-27T07:36:51.982 INFO:teuthology.orchestra.run.smithi052.stdout: "warn_threshold": { 2024-05-27T07:36:51.982 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "warn_threshold", 2024-05-27T07:36:51.982 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "secs", 2024-05-27T07:36:51.982 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-05-27T07:36:51.983 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 1, 2024-05-27T07:36:51.983 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "7257600", 2024-05-27T07:36:51.983 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-05-27T07:36:51.983 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-05-27T07:36:51.983 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-05-27T07:36:51.983 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "raise health warning if OSD may fail before this long", 2024-05-27T07:36:51.983 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-05-27T07:36:51.983 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-05-27T07:36:51.983 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-05-27T07:36:51.983 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-05-27T07:36:51.983 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-05-27T07:36:51.983 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-05-27T07:36:51.983 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-05-27T07:36:51.983 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "diskprediction_local", 2024-05-27T07:36:51.984 INFO:teuthology.orchestra.run.smithi052.stdout: "can_run": true, 2024-05-27T07:36:51.984 INFO:teuthology.orchestra.run.smithi052.stdout: "error_string": "", 2024-05-27T07:36:51.984 INFO:teuthology.orchestra.run.smithi052.stdout: "module_options": { 2024-05-27T07:36:51.984 INFO:teuthology.orchestra.run.smithi052.stdout: "log_level": { 2024-05-27T07:36:51.984 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "log_level", 2024-05-27T07:36:51.984 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "str", 2024-05-27T07:36:51.984 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-05-27T07:36:51.984 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 1, 2024-05-27T07:36:51.984 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "", 2024-05-27T07:36:51.984 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-05-27T07:36:51.984 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-05-27T07:36:51.984 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [ 2024-05-27T07:36:51.985 INFO:teuthology.orchestra.run.smithi052.stdout: "", 2024-05-27T07:36:51.985 INFO:teuthology.orchestra.run.smithi052.stdout: "critical", 2024-05-27T07:36:51.985 INFO:teuthology.orchestra.run.smithi052.stdout: "debug", 2024-05-27T07:36:51.985 INFO:teuthology.orchestra.run.smithi052.stdout: "error", 2024-05-27T07:36:51.985 INFO:teuthology.orchestra.run.smithi052.stdout: "info", 2024-05-27T07:36:51.985 INFO:teuthology.orchestra.run.smithi052.stdout: "warning" 2024-05-27T07:36:51.985 INFO:teuthology.orchestra.run.smithi052.stdout: ], 2024-05-27T07:36:51.985 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-05-27T07:36:51.985 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-05-27T07:36:51.985 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-05-27T07:36:51.985 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-05-27T07:36:51.985 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-05-27T07:36:51.985 INFO:teuthology.orchestra.run.smithi052.stdout: "log_to_cluster": { 2024-05-27T07:36:51.986 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "log_to_cluster", 2024-05-27T07:36:51.986 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "bool", 2024-05-27T07:36:51.986 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-05-27T07:36:51.986 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 1, 2024-05-27T07:36:51.986 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "False", 2024-05-27T07:36:51.986 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-05-27T07:36:51.986 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-05-27T07:36:51.986 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-05-27T07:36:51.986 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-05-27T07:36:51.986 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-05-27T07:36:51.986 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-05-27T07:36:51.986 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-05-27T07:36:51.987 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-05-27T07:36:51.987 INFO:teuthology.orchestra.run.smithi052.stdout: "log_to_cluster_level": { 2024-05-27T07:36:51.987 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "log_to_cluster_level", 2024-05-27T07:36:51.987 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "str", 2024-05-27T07:36:51.987 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-05-27T07:36:51.987 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 1, 2024-05-27T07:36:51.987 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "info", 2024-05-27T07:36:51.987 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-05-27T07:36:51.987 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-05-27T07:36:51.987 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [ 2024-05-27T07:36:51.987 INFO:teuthology.orchestra.run.smithi052.stdout: "", 2024-05-27T07:36:51.987 INFO:teuthology.orchestra.run.smithi052.stdout: "critical", 2024-05-27T07:36:51.987 INFO:teuthology.orchestra.run.smithi052.stdout: "debug", 2024-05-27T07:36:51.988 INFO:teuthology.orchestra.run.smithi052.stdout: "error", 2024-05-27T07:36:51.988 INFO:teuthology.orchestra.run.smithi052.stdout: "info", 2024-05-27T07:36:51.988 INFO:teuthology.orchestra.run.smithi052.stdout: "warning" 2024-05-27T07:36:51.988 INFO:teuthology.orchestra.run.smithi052.stdout: ], 2024-05-27T07:36:51.988 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-05-27T07:36:51.988 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-05-27T07:36:51.988 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-05-27T07:36:51.988 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-05-27T07:36:51.988 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-05-27T07:36:51.988 INFO:teuthology.orchestra.run.smithi052.stdout: "log_to_file": { 2024-05-27T07:36:51.988 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "log_to_file", 2024-05-27T07:36:51.988 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "bool", 2024-05-27T07:36:51.988 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-05-27T07:36:51.989 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 1, 2024-05-27T07:36:51.989 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "False", 2024-05-27T07:36:51.989 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-05-27T07:36:51.989 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-05-27T07:36:51.989 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-05-27T07:36:51.989 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-05-27T07:36:51.989 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-05-27T07:36:51.989 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-05-27T07:36:51.989 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-05-27T07:36:51.989 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-05-27T07:36:51.989 INFO:teuthology.orchestra.run.smithi052.stdout: "predict_interval": { 2024-05-27T07:36:51.989 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "predict_interval", 2024-05-27T07:36:51.989 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "str", 2024-05-27T07:36:51.990 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-05-27T07:36:51.990 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 0, 2024-05-27T07:36:51.990 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "86400", 2024-05-27T07:36:51.990 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-05-27T07:36:51.990 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-05-27T07:36:51.990 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-05-27T07:36:51.990 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-05-27T07:36:51.990 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-05-27T07:36:51.990 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-05-27T07:36:51.990 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-05-27T07:36:51.990 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-05-27T07:36:51.990 INFO:teuthology.orchestra.run.smithi052.stdout: "predictor_model": { 2024-05-27T07:36:51.990 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "predictor_model", 2024-05-27T07:36:51.991 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "str", 2024-05-27T07:36:51.991 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-05-27T07:36:51.991 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 0, 2024-05-27T07:36:51.991 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "prophetstor", 2024-05-27T07:36:51.991 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-05-27T07:36:51.991 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-05-27T07:36:51.991 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-05-27T07:36:51.991 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-05-27T07:36:51.991 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-05-27T07:36:51.991 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-05-27T07:36:51.991 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-05-27T07:36:51.991 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-05-27T07:36:51.991 INFO:teuthology.orchestra.run.smithi052.stdout: "sleep_interval": { 2024-05-27T07:36:51.992 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "sleep_interval", 2024-05-27T07:36:51.992 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "str", 2024-05-27T07:36:51.992 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-05-27T07:36:51.992 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 0, 2024-05-27T07:36:51.992 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "600", 2024-05-27T07:36:51.992 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-05-27T07:36:51.992 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-05-27T07:36:51.992 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-05-27T07:36:51.992 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-05-27T07:36:51.992 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-05-27T07:36:51.992 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-05-27T07:36:51.992 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-05-27T07:36:51.992 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-05-27T07:36:51.993 INFO:teuthology.orchestra.run.smithi052.stdout: "sqlite3_killpoint": { 2024-05-27T07:36:51.993 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "sqlite3_killpoint", 2024-05-27T07:36:51.993 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "int", 2024-05-27T07:36:51.993 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "dev", 2024-05-27T07:36:51.993 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 1, 2024-05-27T07:36:51.993 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "0", 2024-05-27T07:36:51.993 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-05-27T07:36:51.993 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-05-27T07:36:51.993 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-05-27T07:36:51.993 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-05-27T07:36:51.993 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-05-27T07:36:51.993 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-05-27T07:36:51.993 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-05-27T07:36:51.994 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-05-27T07:36:51.994 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-05-27T07:36:51.994 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-05-27T07:36:51.994 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-05-27T07:36:51.994 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "influx", 2024-05-27T07:36:51.994 INFO:teuthology.orchestra.run.smithi052.stdout: "can_run": false, 2024-05-27T07:36:51.994 INFO:teuthology.orchestra.run.smithi052.stdout: "error_string": "influxdb python module not found", 2024-05-27T07:36:51.994 INFO:teuthology.orchestra.run.smithi052.stdout: "module_options": { 2024-05-27T07:36:51.994 INFO:teuthology.orchestra.run.smithi052.stdout: "batch_size": { 2024-05-27T07:36:51.994 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "batch_size", 2024-05-27T07:36:51.994 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "int", 2024-05-27T07:36:51.994 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-05-27T07:36:51.994 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 0, 2024-05-27T07:36:51.995 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "5000", 2024-05-27T07:36:51.995 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-05-27T07:36:51.995 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-05-27T07:36:51.995 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-05-27T07:36:51.995 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "How big batches of data points should be when sending to InfluxDB.", 2024-05-27T07:36:51.995 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-05-27T07:36:51.995 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-05-27T07:36:51.995 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-05-27T07:36:51.995 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-05-27T07:36:51.995 INFO:teuthology.orchestra.run.smithi052.stdout: "database": { 2024-05-27T07:36:51.995 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "database", 2024-05-27T07:36:51.995 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "str", 2024-05-27T07:36:51.995 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-05-27T07:36:51.996 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 0, 2024-05-27T07:36:51.996 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "ceph", 2024-05-27T07:36:51.996 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-05-27T07:36:51.996 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-05-27T07:36:51.996 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-05-27T07:36:51.996 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.", 2024-05-27T07:36:51.996 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-05-27T07:36:51.996 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-05-27T07:36:51.996 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-05-27T07:36:51.996 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-05-27T07:36:51.996 INFO:teuthology.orchestra.run.smithi052.stdout: "hostname": { 2024-05-27T07:36:51.996 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "hostname", 2024-05-27T07:36:51.996 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "str", 2024-05-27T07:36:51.997 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-05-27T07:36:51.997 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 0, 2024-05-27T07:36:51.997 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "", 2024-05-27T07:36:51.997 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-05-27T07:36:51.997 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-05-27T07:36:51.997 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-05-27T07:36:51.997 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "InfluxDB server hostname", 2024-05-27T07:36:51.997 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-05-27T07:36:51.997 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-05-27T07:36:51.997 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-05-27T07:36:51.997 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-05-27T07:36:51.997 INFO:teuthology.orchestra.run.smithi052.stdout: "interval": { 2024-05-27T07:36:51.997 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "interval", 2024-05-27T07:36:51.997 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "secs", 2024-05-27T07:36:51.998 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-05-27T07:36:51.998 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 0, 2024-05-27T07:36:51.998 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "30", 2024-05-27T07:36:51.998 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "5", 2024-05-27T07:36:51.998 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-05-27T07:36:51.998 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-05-27T07:36:51.998 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "Time between reports to InfluxDB. Default 30 seconds.", 2024-05-27T07:36:51.998 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-05-27T07:36:51.998 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-05-27T07:36:51.998 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-05-27T07:36:51.998 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-05-27T07:36:51.998 INFO:teuthology.orchestra.run.smithi052.stdout: "log_level": { 2024-05-27T07:36:51.999 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "log_level", 2024-05-27T07:36:51.999 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "str", 2024-05-27T07:36:51.999 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-05-27T07:36:51.999 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 1, 2024-05-27T07:36:51.999 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "", 2024-05-27T07:36:51.999 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-05-27T07:36:51.999 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-05-27T07:36:51.999 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [ 2024-05-27T07:36:51.999 INFO:teuthology.orchestra.run.smithi052.stdout: "", 2024-05-27T07:36:51.999 INFO:teuthology.orchestra.run.smithi052.stdout: "critical", 2024-05-27T07:36:51.999 INFO:teuthology.orchestra.run.smithi052.stdout: "debug", 2024-05-27T07:36:51.999 INFO:teuthology.orchestra.run.smithi052.stdout: "error", 2024-05-27T07:36:52.000 INFO:teuthology.orchestra.run.smithi052.stdout: "info", 2024-05-27T07:36:52.000 INFO:teuthology.orchestra.run.smithi052.stdout: "warning" 2024-05-27T07:36:52.000 INFO:teuthology.orchestra.run.smithi052.stdout: ], 2024-05-27T07:36:52.000 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-05-27T07:36:52.000 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-05-27T07:36:52.000 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-05-27T07:36:52.000 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-05-27T07:36:52.000 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-05-27T07:36:52.000 INFO:teuthology.orchestra.run.smithi052.stdout: "log_to_cluster": { 2024-05-27T07:36:52.000 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "log_to_cluster", 2024-05-27T07:36:52.000 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "bool", 2024-05-27T07:36:52.000 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-05-27T07:36:52.000 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 1, 2024-05-27T07:36:52.001 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "False", 2024-05-27T07:36:52.001 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-05-27T07:36:52.001 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-05-27T07:36:52.001 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-05-27T07:36:52.001 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-05-27T07:36:52.001 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-05-27T07:36:52.001 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-05-27T07:36:52.001 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-05-27T07:36:52.001 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-05-27T07:36:52.001 INFO:teuthology.orchestra.run.smithi052.stdout: "log_to_cluster_level": { 2024-05-27T07:36:52.001 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "log_to_cluster_level", 2024-05-27T07:36:52.001 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "str", 2024-05-27T07:36:52.001 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-05-27T07:36:52.002 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 1, 2024-05-27T07:36:52.002 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "info", 2024-05-27T07:36:52.002 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-05-27T07:36:52.002 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-05-27T07:36:52.002 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [ 2024-05-27T07:36:52.002 INFO:teuthology.orchestra.run.smithi052.stdout: "", 2024-05-27T07:36:52.002 INFO:teuthology.orchestra.run.smithi052.stdout: "critical", 2024-05-27T07:36:52.002 INFO:teuthology.orchestra.run.smithi052.stdout: "debug", 2024-05-27T07:36:52.002 INFO:teuthology.orchestra.run.smithi052.stdout: "error", 2024-05-27T07:36:52.002 INFO:teuthology.orchestra.run.smithi052.stdout: "info", 2024-05-27T07:36:52.002 INFO:teuthology.orchestra.run.smithi052.stdout: "warning" 2024-05-27T07:36:52.002 INFO:teuthology.orchestra.run.smithi052.stdout: ], 2024-05-27T07:36:52.002 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-05-27T07:36:52.003 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-05-27T07:36:52.003 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-05-27T07:36:52.003 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-05-27T07:36:52.003 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-05-27T07:36:52.003 INFO:teuthology.orchestra.run.smithi052.stdout: "log_to_file": { 2024-05-27T07:36:52.003 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "log_to_file", 2024-05-27T07:36:52.003 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "bool", 2024-05-27T07:36:52.003 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-05-27T07:36:52.003 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 1, 2024-05-27T07:36:52.003 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "False", 2024-05-27T07:36:52.003 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-05-27T07:36:52.003 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-05-27T07:36:52.004 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-05-27T07:36:52.004 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-05-27T07:36:52.004 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-05-27T07:36:52.004 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-05-27T07:36:52.004 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-05-27T07:36:52.004 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-05-27T07:36:52.004 INFO:teuthology.orchestra.run.smithi052.stdout: "password": { 2024-05-27T07:36:52.004 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "password", 2024-05-27T07:36:52.004 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "str", 2024-05-27T07:36:52.004 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-05-27T07:36:52.004 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 0, 2024-05-27T07:36:52.004 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "", 2024-05-27T07:36:52.005 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-05-27T07:36:52.005 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-05-27T07:36:52.005 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-05-27T07:36:52.005 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "password of InfluxDB server user", 2024-05-27T07:36:52.005 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-05-27T07:36:52.005 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-05-27T07:36:52.005 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-05-27T07:36:52.005 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-05-27T07:36:52.005 INFO:teuthology.orchestra.run.smithi052.stdout: "port": { 2024-05-27T07:36:52.005 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "port", 2024-05-27T07:36:52.005 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "int", 2024-05-27T07:36:52.005 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-05-27T07:36:52.005 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 0, 2024-05-27T07:36:52.006 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "8086", 2024-05-27T07:36:52.006 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-05-27T07:36:52.006 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-05-27T07:36:52.006 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-05-27T07:36:52.006 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "InfluxDB server port", 2024-05-27T07:36:52.006 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-05-27T07:36:52.006 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-05-27T07:36:52.006 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-05-27T07:36:52.006 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-05-27T07:36:52.006 INFO:teuthology.orchestra.run.smithi052.stdout: "sqlite3_killpoint": { 2024-05-27T07:36:52.006 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "sqlite3_killpoint", 2024-05-27T07:36:52.006 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "int", 2024-05-27T07:36:52.007 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "dev", 2024-05-27T07:36:52.007 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 1, 2024-05-27T07:36:52.007 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "0", 2024-05-27T07:36:52.007 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-05-27T07:36:52.007 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-05-27T07:36:52.007 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-05-27T07:36:52.007 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-05-27T07:36:52.007 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-05-27T07:36:52.007 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-05-27T07:36:52.007 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-05-27T07:36:52.007 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-05-27T07:36:52.007 INFO:teuthology.orchestra.run.smithi052.stdout: "ssl": { 2024-05-27T07:36:52.007 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "ssl", 2024-05-27T07:36:52.008 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "str", 2024-05-27T07:36:52.008 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-05-27T07:36:52.008 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 0, 2024-05-27T07:36:52.008 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "false", 2024-05-27T07:36:52.008 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-05-27T07:36:52.008 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-05-27T07:36:52.008 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-05-27T07:36:52.008 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "Use https connection for InfluxDB server. Use \"true\" or \"false\".", 2024-05-27T07:36:52.008 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-05-27T07:36:52.008 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-05-27T07:36:52.008 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-05-27T07:36:52.008 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-05-27T07:36:52.008 INFO:teuthology.orchestra.run.smithi052.stdout: "threads": { 2024-05-27T07:36:52.009 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "threads", 2024-05-27T07:36:52.009 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "int", 2024-05-27T07:36:52.009 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-05-27T07:36:52.009 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 0, 2024-05-27T07:36:52.009 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "5", 2024-05-27T07:36:52.009 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "1", 2024-05-27T07:36:52.009 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "32", 2024-05-27T07:36:52.009 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-05-27T07:36:52.009 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "How many worker threads should be spawned for sending data to InfluxDB.", 2024-05-27T07:36:52.009 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-05-27T07:36:52.009 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-05-27T07:36:52.009 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-05-27T07:36:52.009 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-05-27T07:36:52.010 INFO:teuthology.orchestra.run.smithi052.stdout: "username": { 2024-05-27T07:36:52.010 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "username", 2024-05-27T07:36:52.010 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "str", 2024-05-27T07:36:52.010 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-05-27T07:36:52.010 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 0, 2024-05-27T07:36:52.010 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "", 2024-05-27T07:36:52.010 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-05-27T07:36:52.010 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-05-27T07:36:52.010 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-05-27T07:36:52.010 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "username of InfluxDB server user", 2024-05-27T07:36:52.010 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-05-27T07:36:52.010 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-05-27T07:36:52.011 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-05-27T07:36:52.011 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-05-27T07:36:52.011 INFO:teuthology.orchestra.run.smithi052.stdout: "verify_ssl": { 2024-05-27T07:36:52.011 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "verify_ssl", 2024-05-27T07:36:52.011 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "str", 2024-05-27T07:36:52.011 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-05-27T07:36:52.011 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 0, 2024-05-27T07:36:52.011 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "true", 2024-05-27T07:36:52.011 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-05-27T07:36:52.011 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-05-27T07:36:52.011 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-05-27T07:36:52.011 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "Verify https cert for InfluxDB server. Use \"true\" or \"false\".", 2024-05-27T07:36:52.011 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-05-27T07:36:52.012 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-05-27T07:36:52.012 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-05-27T07:36:52.012 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-05-27T07:36:52.012 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-05-27T07:36:52.012 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-05-27T07:36:52.012 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-05-27T07:36:52.012 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "insights", 2024-05-27T07:36:52.012 INFO:teuthology.orchestra.run.smithi052.stdout: "can_run": true, 2024-05-27T07:36:52.012 INFO:teuthology.orchestra.run.smithi052.stdout: "error_string": "", 2024-05-27T07:36:52.012 INFO:teuthology.orchestra.run.smithi052.stdout: "module_options": { 2024-05-27T07:36:52.012 INFO:teuthology.orchestra.run.smithi052.stdout: "log_level": { 2024-05-27T07:36:52.012 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "log_level", 2024-05-27T07:36:52.012 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "str", 2024-05-27T07:36:52.013 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-05-27T07:36:52.013 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 1, 2024-05-27T07:36:52.013 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "", 2024-05-27T07:36:52.013 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-05-27T07:36:52.013 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-05-27T07:36:52.013 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [ 2024-05-27T07:36:52.013 INFO:teuthology.orchestra.run.smithi052.stdout: "", 2024-05-27T07:36:52.013 INFO:teuthology.orchestra.run.smithi052.stdout: "critical", 2024-05-27T07:36:52.013 INFO:teuthology.orchestra.run.smithi052.stdout: "debug", 2024-05-27T07:36:52.013 INFO:teuthology.orchestra.run.smithi052.stdout: "error", 2024-05-27T07:36:52.013 INFO:teuthology.orchestra.run.smithi052.stdout: "info", 2024-05-27T07:36:52.013 INFO:teuthology.orchestra.run.smithi052.stdout: "warning" 2024-05-27T07:36:52.013 INFO:teuthology.orchestra.run.smithi052.stdout: ], 2024-05-27T07:36:52.013 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-05-27T07:36:52.014 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-05-27T07:36:52.014 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-05-27T07:36:52.014 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-05-27T07:36:52.014 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-05-27T07:36:52.014 INFO:teuthology.orchestra.run.smithi052.stdout: "log_to_cluster": { 2024-05-27T07:36:52.014 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "log_to_cluster", 2024-05-27T07:36:52.014 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "bool", 2024-05-27T07:36:52.014 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-05-27T07:36:52.014 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 1, 2024-05-27T07:36:52.014 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "False", 2024-05-27T07:36:52.014 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-05-27T07:36:52.015 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-05-27T07:36:52.015 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-05-27T07:36:52.015 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-05-27T07:36:52.015 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-05-27T07:36:52.015 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-05-27T07:36:52.015 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-05-27T07:36:52.015 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-05-27T07:36:52.015 INFO:teuthology.orchestra.run.smithi052.stdout: "log_to_cluster_level": { 2024-05-27T07:36:52.015 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "log_to_cluster_level", 2024-05-27T07:36:52.015 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "str", 2024-05-27T07:36:52.015 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-05-27T07:36:52.015 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 1, 2024-05-27T07:36:52.016 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "info", 2024-05-27T07:36:52.016 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-05-27T07:36:52.016 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-05-27T07:36:52.016 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [ 2024-05-27T07:36:52.016 INFO:teuthology.orchestra.run.smithi052.stdout: "", 2024-05-27T07:36:52.016 INFO:teuthology.orchestra.run.smithi052.stdout: "critical", 2024-05-27T07:36:52.016 INFO:teuthology.orchestra.run.smithi052.stdout: "debug", 2024-05-27T07:36:52.016 INFO:teuthology.orchestra.run.smithi052.stdout: "error", 2024-05-27T07:36:52.016 INFO:teuthology.orchestra.run.smithi052.stdout: "info", 2024-05-27T07:36:52.016 INFO:teuthology.orchestra.run.smithi052.stdout: "warning" 2024-05-27T07:36:52.016 INFO:teuthology.orchestra.run.smithi052.stdout: ], 2024-05-27T07:36:52.016 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-05-27T07:36:52.016 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-05-27T07:36:52.017 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-05-27T07:36:52.017 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-05-27T07:36:52.017 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-05-27T07:36:52.017 INFO:teuthology.orchestra.run.smithi052.stdout: "log_to_file": { 2024-05-27T07:36:52.017 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "log_to_file", 2024-05-27T07:36:52.017 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "bool", 2024-05-27T07:36:52.017 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-05-27T07:36:52.017 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 1, 2024-05-27T07:36:52.017 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "False", 2024-05-27T07:36:52.017 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-05-27T07:36:52.017 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-05-27T07:36:52.017 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-05-27T07:36:52.017 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-05-27T07:36:52.018 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-05-27T07:36:52.018 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-05-27T07:36:52.018 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-05-27T07:36:52.018 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-05-27T07:36:52.018 INFO:teuthology.orchestra.run.smithi052.stdout: "sqlite3_killpoint": { 2024-05-27T07:36:52.018 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "sqlite3_killpoint", 2024-05-27T07:36:52.018 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "int", 2024-05-27T07:36:52.018 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "dev", 2024-05-27T07:36:52.018 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 1, 2024-05-27T07:36:52.018 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "0", 2024-05-27T07:36:52.018 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-05-27T07:36:52.018 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-05-27T07:36:52.019 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-05-27T07:36:52.019 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-05-27T07:36:52.019 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-05-27T07:36:52.019 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-05-27T07:36:52.019 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-05-27T07:36:52.019 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-05-27T07:36:52.019 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-05-27T07:36:52.019 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-05-27T07:36:52.019 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-05-27T07:36:52.019 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "iostat", 2024-05-27T07:36:52.019 INFO:teuthology.orchestra.run.smithi052.stdout: "can_run": true, 2024-05-27T07:36:52.019 INFO:teuthology.orchestra.run.smithi052.stdout: "error_string": "", 2024-05-27T07:36:52.019 INFO:teuthology.orchestra.run.smithi052.stdout: "module_options": { 2024-05-27T07:36:52.020 INFO:teuthology.orchestra.run.smithi052.stdout: "log_level": { 2024-05-27T07:36:52.020 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "log_level", 2024-05-27T07:36:52.020 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "str", 2024-05-27T07:36:52.020 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-05-27T07:36:52.020 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 1, 2024-05-27T07:36:52.020 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "", 2024-05-27T07:36:52.020 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-05-27T07:36:52.020 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-05-27T07:36:52.020 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [ 2024-05-27T07:36:52.020 INFO:teuthology.orchestra.run.smithi052.stdout: "", 2024-05-27T07:36:52.020 INFO:teuthology.orchestra.run.smithi052.stdout: "critical", 2024-05-27T07:36:52.020 INFO:teuthology.orchestra.run.smithi052.stdout: "debug", 2024-05-27T07:36:52.020 INFO:teuthology.orchestra.run.smithi052.stdout: "error", 2024-05-27T07:36:52.021 INFO:teuthology.orchestra.run.smithi052.stdout: "info", 2024-05-27T07:36:52.021 INFO:teuthology.orchestra.run.smithi052.stdout: "warning" 2024-05-27T07:36:52.021 INFO:teuthology.orchestra.run.smithi052.stdout: ], 2024-05-27T07:36:52.021 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-05-27T07:36:52.021 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-05-27T07:36:52.021 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-05-27T07:36:52.021 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-05-27T07:36:52.021 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-05-27T07:36:52.021 INFO:teuthology.orchestra.run.smithi052.stdout: "log_to_cluster": { 2024-05-27T07:36:52.021 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "log_to_cluster", 2024-05-27T07:36:52.021 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "bool", 2024-05-27T07:36:52.021 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-05-27T07:36:52.021 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 1, 2024-05-27T07:36:52.021 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "False", 2024-05-27T07:36:52.022 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-05-27T07:36:52.022 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-05-27T07:36:52.022 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-05-27T07:36:52.022 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-05-27T07:36:52.022 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-05-27T07:36:52.022 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-05-27T07:36:52.022 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-05-27T07:36:52.022 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-05-27T07:36:52.022 INFO:teuthology.orchestra.run.smithi052.stdout: "log_to_cluster_level": { 2024-05-27T07:36:52.022 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "log_to_cluster_level", 2024-05-27T07:36:52.022 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "str", 2024-05-27T07:36:52.022 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-05-27T07:36:52.023 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 1, 2024-05-27T07:36:52.023 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "info", 2024-05-27T07:36:52.023 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-05-27T07:36:52.023 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-05-27T07:36:52.023 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [ 2024-05-27T07:36:52.023 INFO:teuthology.orchestra.run.smithi052.stdout: "", 2024-05-27T07:36:52.023 INFO:teuthology.orchestra.run.smithi052.stdout: "critical", 2024-05-27T07:36:52.023 INFO:teuthology.orchestra.run.smithi052.stdout: "debug", 2024-05-27T07:36:52.023 INFO:teuthology.orchestra.run.smithi052.stdout: "error", 2024-05-27T07:36:52.023 INFO:teuthology.orchestra.run.smithi052.stdout: "info", 2024-05-27T07:36:52.023 INFO:teuthology.orchestra.run.smithi052.stdout: "warning" 2024-05-27T07:36:52.023 INFO:teuthology.orchestra.run.smithi052.stdout: ], 2024-05-27T07:36:52.024 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-05-27T07:36:52.024 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-05-27T07:36:52.024 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-05-27T07:36:52.024 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-05-27T07:36:52.024 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-05-27T07:36:52.024 INFO:teuthology.orchestra.run.smithi052.stdout: "log_to_file": { 2024-05-27T07:36:52.024 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "log_to_file", 2024-05-27T07:36:52.024 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "bool", 2024-05-27T07:36:52.024 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-05-27T07:36:52.024 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 1, 2024-05-27T07:36:52.024 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "False", 2024-05-27T07:36:52.024 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-05-27T07:36:52.025 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-05-27T07:36:52.025 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-05-27T07:36:52.025 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-05-27T07:36:52.025 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-05-27T07:36:52.025 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-05-27T07:36:52.025 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-05-27T07:36:52.025 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-05-27T07:36:52.025 INFO:teuthology.orchestra.run.smithi052.stdout: "sqlite3_killpoint": { 2024-05-27T07:36:52.025 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "sqlite3_killpoint", 2024-05-27T07:36:52.025 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "int", 2024-05-27T07:36:52.025 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "dev", 2024-05-27T07:36:52.025 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 1, 2024-05-27T07:36:52.025 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "0", 2024-05-27T07:36:52.025 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-05-27T07:36:52.026 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-05-27T07:36:52.026 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-05-27T07:36:52.026 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-05-27T07:36:52.026 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-05-27T07:36:52.026 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-05-27T07:36:52.026 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-05-27T07:36:52.026 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-05-27T07:36:52.026 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-05-27T07:36:52.026 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-05-27T07:36:52.026 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-05-27T07:36:52.026 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "k8sevents", 2024-05-27T07:36:52.026 INFO:teuthology.orchestra.run.smithi052.stdout: "can_run": true, 2024-05-27T07:36:52.027 INFO:teuthology.orchestra.run.smithi052.stdout: "error_string": "", 2024-05-27T07:36:52.027 INFO:teuthology.orchestra.run.smithi052.stdout: "module_options": { 2024-05-27T07:36:52.027 INFO:teuthology.orchestra.run.smithi052.stdout: "ceph_event_retention_days": { 2024-05-27T07:36:52.027 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "ceph_event_retention_days", 2024-05-27T07:36:52.027 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "int", 2024-05-27T07:36:52.027 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-05-27T07:36:52.027 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 0, 2024-05-27T07:36:52.027 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "7", 2024-05-27T07:36:52.027 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-05-27T07:36:52.027 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-05-27T07:36:52.027 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-05-27T07:36:52.027 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "Days to hold ceph event information within local cache", 2024-05-27T07:36:52.027 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-05-27T07:36:52.028 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-05-27T07:36:52.028 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-05-27T07:36:52.028 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-05-27T07:36:52.028 INFO:teuthology.orchestra.run.smithi052.stdout: "config_check_secs": { 2024-05-27T07:36:52.028 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "config_check_secs", 2024-05-27T07:36:52.028 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "int", 2024-05-27T07:36:52.028 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-05-27T07:36:52.028 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 0, 2024-05-27T07:36:52.028 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "10", 2024-05-27T07:36:52.028 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "10", 2024-05-27T07:36:52.028 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-05-27T07:36:52.028 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-05-27T07:36:52.028 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "interval (secs) to check for cluster configuration changes", 2024-05-27T07:36:52.029 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-05-27T07:36:52.029 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-05-27T07:36:52.029 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-05-27T07:36:52.029 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-05-27T07:36:52.029 INFO:teuthology.orchestra.run.smithi052.stdout: "log_level": { 2024-05-27T07:36:52.029 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "log_level", 2024-05-27T07:36:52.029 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "str", 2024-05-27T07:36:52.029 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-05-27T07:36:52.029 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 1, 2024-05-27T07:36:52.029 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "", 2024-05-27T07:36:52.029 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-05-27T07:36:52.029 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-05-27T07:36:52.030 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [ 2024-05-27T07:36:52.030 INFO:teuthology.orchestra.run.smithi052.stdout: "", 2024-05-27T07:36:52.030 INFO:teuthology.orchestra.run.smithi052.stdout: "critical", 2024-05-27T07:36:52.030 INFO:teuthology.orchestra.run.smithi052.stdout: "debug", 2024-05-27T07:36:52.030 INFO:teuthology.orchestra.run.smithi052.stdout: "error", 2024-05-27T07:36:52.030 INFO:teuthology.orchestra.run.smithi052.stdout: "info", 2024-05-27T07:36:52.030 INFO:teuthology.orchestra.run.smithi052.stdout: "warning" 2024-05-27T07:36:52.030 INFO:teuthology.orchestra.run.smithi052.stdout: ], 2024-05-27T07:36:52.030 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-05-27T07:36:52.030 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-05-27T07:36:52.030 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-05-27T07:36:52.030 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-05-27T07:36:52.030 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-05-27T07:36:52.031 INFO:teuthology.orchestra.run.smithi052.stdout: "log_to_cluster": { 2024-05-27T07:36:52.031 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "log_to_cluster", 2024-05-27T07:36:52.031 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "bool", 2024-05-27T07:36:52.031 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-05-27T07:36:52.031 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 1, 2024-05-27T07:36:52.031 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "False", 2024-05-27T07:36:52.031 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-05-27T07:36:52.031 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-05-27T07:36:52.031 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-05-27T07:36:52.031 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-05-27T07:36:52.031 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-05-27T07:36:52.031 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-05-27T07:36:52.031 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-05-27T07:36:52.032 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-05-27T07:36:52.032 INFO:teuthology.orchestra.run.smithi052.stdout: "log_to_cluster_level": { 2024-05-27T07:36:52.032 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "log_to_cluster_level", 2024-05-27T07:36:52.032 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "str", 2024-05-27T07:36:52.032 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-05-27T07:36:52.032 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 1, 2024-05-27T07:36:52.032 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "info", 2024-05-27T07:36:52.032 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-05-27T07:36:52.032 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-05-27T07:36:52.032 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [ 2024-05-27T07:36:52.032 INFO:teuthology.orchestra.run.smithi052.stdout: "", 2024-05-27T07:36:52.032 INFO:teuthology.orchestra.run.smithi052.stdout: "critical", 2024-05-27T07:36:52.032 INFO:teuthology.orchestra.run.smithi052.stdout: "debug", 2024-05-27T07:36:52.033 INFO:teuthology.orchestra.run.smithi052.stdout: "error", 2024-05-27T07:36:52.033 INFO:teuthology.orchestra.run.smithi052.stdout: "info", 2024-05-27T07:36:52.033 INFO:teuthology.orchestra.run.smithi052.stdout: "warning" 2024-05-27T07:36:52.033 INFO:teuthology.orchestra.run.smithi052.stdout: ], 2024-05-27T07:36:52.033 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-05-27T07:36:52.033 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-05-27T07:36:52.033 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-05-27T07:36:52.033 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-05-27T07:36:52.033 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-05-27T07:36:52.033 INFO:teuthology.orchestra.run.smithi052.stdout: "log_to_file": { 2024-05-27T07:36:52.033 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "log_to_file", 2024-05-27T07:36:52.033 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "bool", 2024-05-27T07:36:52.033 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-05-27T07:36:52.033 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 1, 2024-05-27T07:36:52.034 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "False", 2024-05-27T07:36:52.034 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-05-27T07:36:52.034 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-05-27T07:36:52.034 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-05-27T07:36:52.034 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-05-27T07:36:52.034 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-05-27T07:36:52.034 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-05-27T07:36:52.034 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-05-27T07:36:52.034 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-05-27T07:36:52.034 INFO:teuthology.orchestra.run.smithi052.stdout: "sqlite3_killpoint": { 2024-05-27T07:36:52.034 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "sqlite3_killpoint", 2024-05-27T07:36:52.034 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "int", 2024-05-27T07:36:52.035 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "dev", 2024-05-27T07:36:52.035 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 1, 2024-05-27T07:36:52.035 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "0", 2024-05-27T07:36:52.035 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-05-27T07:36:52.035 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-05-27T07:36:52.035 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-05-27T07:36:52.035 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-05-27T07:36:52.035 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-05-27T07:36:52.035 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-05-27T07:36:52.035 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-05-27T07:36:52.035 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-05-27T07:36:52.035 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-05-27T07:36:52.035 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-05-27T07:36:52.036 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-05-27T07:36:52.036 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "localpool", 2024-05-27T07:36:52.036 INFO:teuthology.orchestra.run.smithi052.stdout: "can_run": true, 2024-05-27T07:36:52.036 INFO:teuthology.orchestra.run.smithi052.stdout: "error_string": "", 2024-05-27T07:36:52.036 INFO:teuthology.orchestra.run.smithi052.stdout: "module_options": { 2024-05-27T07:36:52.036 INFO:teuthology.orchestra.run.smithi052.stdout: "failure_domain": { 2024-05-27T07:36:52.036 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "failure_domain", 2024-05-27T07:36:52.036 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "str", 2024-05-27T07:36:52.036 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-05-27T07:36:52.036 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 1, 2024-05-27T07:36:52.036 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "host", 2024-05-27T07:36:52.036 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-05-27T07:36:52.036 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-05-27T07:36:52.036 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-05-27T07:36:52.037 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "failure domain for any created local pool", 2024-05-27T07:36:52.037 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "what failure domain we should separate data replicas across.", 2024-05-27T07:36:52.037 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-05-27T07:36:52.037 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-05-27T07:36:52.037 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-05-27T07:36:52.037 INFO:teuthology.orchestra.run.smithi052.stdout: "log_level": { 2024-05-27T07:36:52.037 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "log_level", 2024-05-27T07:36:52.037 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "str", 2024-05-27T07:36:52.037 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-05-27T07:36:52.037 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 1, 2024-05-27T07:36:52.037 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "", 2024-05-27T07:36:52.037 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-05-27T07:36:52.037 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-05-27T07:36:52.038 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [ 2024-05-27T07:36:52.038 INFO:teuthology.orchestra.run.smithi052.stdout: "", 2024-05-27T07:36:52.038 INFO:teuthology.orchestra.run.smithi052.stdout: "critical", 2024-05-27T07:36:52.038 INFO:teuthology.orchestra.run.smithi052.stdout: "debug", 2024-05-27T07:36:52.038 INFO:teuthology.orchestra.run.smithi052.stdout: "error", 2024-05-27T07:36:52.038 INFO:teuthology.orchestra.run.smithi052.stdout: "info", 2024-05-27T07:36:52.038 INFO:teuthology.orchestra.run.smithi052.stdout: "warning" 2024-05-27T07:36:52.038 INFO:teuthology.orchestra.run.smithi052.stdout: ], 2024-05-27T07:36:52.038 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-05-27T07:36:52.038 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-05-27T07:36:52.038 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-05-27T07:36:52.038 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-05-27T07:36:52.038 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-05-27T07:36:52.039 INFO:teuthology.orchestra.run.smithi052.stdout: "log_to_cluster": { 2024-05-27T07:36:52.039 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "log_to_cluster", 2024-05-27T07:36:52.039 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "bool", 2024-05-27T07:36:52.039 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-05-27T07:36:52.039 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 1, 2024-05-27T07:36:52.039 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "False", 2024-05-27T07:36:52.039 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-05-27T07:36:52.039 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-05-27T07:36:52.039 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-05-27T07:36:52.039 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-05-27T07:36:52.039 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-05-27T07:36:52.039 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-05-27T07:36:52.039 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-05-27T07:36:52.039 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-05-27T07:36:52.040 INFO:teuthology.orchestra.run.smithi052.stdout: "log_to_cluster_level": { 2024-05-27T07:36:52.040 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "log_to_cluster_level", 2024-05-27T07:36:52.040 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "str", 2024-05-27T07:36:52.040 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-05-27T07:36:52.040 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 1, 2024-05-27T07:36:52.040 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "info", 2024-05-27T07:36:52.040 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-05-27T07:36:52.040 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-05-27T07:36:52.040 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [ 2024-05-27T07:36:52.040 INFO:teuthology.orchestra.run.smithi052.stdout: "", 2024-05-27T07:36:52.040 INFO:teuthology.orchestra.run.smithi052.stdout: "critical", 2024-05-27T07:36:52.040 INFO:teuthology.orchestra.run.smithi052.stdout: "debug", 2024-05-27T07:36:52.040 INFO:teuthology.orchestra.run.smithi052.stdout: "error", 2024-05-27T07:36:52.040 INFO:teuthology.orchestra.run.smithi052.stdout: "info", 2024-05-27T07:36:52.041 INFO:teuthology.orchestra.run.smithi052.stdout: "warning" 2024-05-27T07:36:52.041 INFO:teuthology.orchestra.run.smithi052.stdout: ], 2024-05-27T07:36:52.041 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-05-27T07:36:52.041 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-05-27T07:36:52.041 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-05-27T07:36:52.041 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-05-27T07:36:52.041 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-05-27T07:36:52.041 INFO:teuthology.orchestra.run.smithi052.stdout: "log_to_file": { 2024-05-27T07:36:52.041 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "log_to_file", 2024-05-27T07:36:52.041 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "bool", 2024-05-27T07:36:52.041 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-05-27T07:36:52.041 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 1, 2024-05-27T07:36:52.042 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "False", 2024-05-27T07:36:52.042 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-05-27T07:36:52.042 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-05-27T07:36:52.042 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-05-27T07:36:52.042 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-05-27T07:36:52.042 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-05-27T07:36:52.042 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-05-27T07:36:52.042 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-05-27T07:36:52.042 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-05-27T07:36:52.042 INFO:teuthology.orchestra.run.smithi052.stdout: "min_size": { 2024-05-27T07:36:52.042 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "min_size", 2024-05-27T07:36:52.042 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "int", 2024-05-27T07:36:52.042 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-05-27T07:36:52.043 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 1, 2024-05-27T07:36:52.043 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "", 2024-05-27T07:36:52.043 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-05-27T07:36:52.043 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-05-27T07:36:52.043 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-05-27T07:36:52.043 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "default min_size for any created local pool", 2024-05-27T07:36:52.043 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "value to set min_size to (unchanged from Ceph's default if this option is not set)", 2024-05-27T07:36:52.043 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-05-27T07:36:52.043 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-05-27T07:36:52.043 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-05-27T07:36:52.043 INFO:teuthology.orchestra.run.smithi052.stdout: "num_rep": { 2024-05-27T07:36:52.043 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "num_rep", 2024-05-27T07:36:52.044 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "int", 2024-05-27T07:36:52.044 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-05-27T07:36:52.044 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 1, 2024-05-27T07:36:52.044 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "3", 2024-05-27T07:36:52.044 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-05-27T07:36:52.044 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-05-27T07:36:52.044 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-05-27T07:36:52.044 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "default replica count for any created local pool", 2024-05-27T07:36:52.044 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-05-27T07:36:52.044 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-05-27T07:36:52.044 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-05-27T07:36:52.044 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-05-27T07:36:52.045 INFO:teuthology.orchestra.run.smithi052.stdout: "pg_num": { 2024-05-27T07:36:52.045 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "pg_num", 2024-05-27T07:36:52.045 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "int", 2024-05-27T07:36:52.045 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-05-27T07:36:52.045 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 1, 2024-05-27T07:36:52.045 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "128", 2024-05-27T07:36:52.045 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-05-27T07:36:52.045 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-05-27T07:36:52.045 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-05-27T07:36:52.045 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "default pg_num for any created local pool", 2024-05-27T07:36:52.045 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-05-27T07:36:52.045 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-05-27T07:36:52.046 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-05-27T07:36:52.046 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-05-27T07:36:52.046 INFO:teuthology.orchestra.run.smithi052.stdout: "prefix": { 2024-05-27T07:36:52.046 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "prefix", 2024-05-27T07:36:52.046 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "str", 2024-05-27T07:36:52.046 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-05-27T07:36:52.046 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 1, 2024-05-27T07:36:52.046 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "", 2024-05-27T07:36:52.046 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-05-27T07:36:52.046 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-05-27T07:36:52.046 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-05-27T07:36:52.046 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "name prefix for any created local pool", 2024-05-27T07:36:52.046 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-05-27T07:36:52.047 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-05-27T07:36:52.047 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-05-27T07:36:52.047 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-05-27T07:36:52.047 INFO:teuthology.orchestra.run.smithi052.stdout: "sqlite3_killpoint": { 2024-05-27T07:36:52.047 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "sqlite3_killpoint", 2024-05-27T07:36:52.047 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "int", 2024-05-27T07:36:52.047 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "dev", 2024-05-27T07:36:52.047 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 1, 2024-05-27T07:36:52.047 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "0", 2024-05-27T07:36:52.047 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-05-27T07:36:52.047 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-05-27T07:36:52.047 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-05-27T07:36:52.047 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-05-27T07:36:52.048 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-05-27T07:36:52.048 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-05-27T07:36:52.048 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-05-27T07:36:52.048 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-05-27T07:36:52.048 INFO:teuthology.orchestra.run.smithi052.stdout: "subtree": { 2024-05-27T07:36:52.048 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "subtree", 2024-05-27T07:36:52.048 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "str", 2024-05-27T07:36:52.048 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-05-27T07:36:52.048 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 1, 2024-05-27T07:36:52.048 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "rack", 2024-05-27T07:36:52.048 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-05-27T07:36:52.048 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-05-27T07:36:52.049 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-05-27T07:36:52.049 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "CRUSH level for which to create a local pool", 2024-05-27T07:36:52.049 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "which CRUSH subtree type the module should create a pool for.", 2024-05-27T07:36:52.049 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-05-27T07:36:52.049 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-05-27T07:36:52.049 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-05-27T07:36:52.049 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-05-27T07:36:52.049 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-05-27T07:36:52.049 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-05-27T07:36:52.049 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "mds_autoscaler", 2024-05-27T07:36:52.049 INFO:teuthology.orchestra.run.smithi052.stdout: "can_run": true, 2024-05-27T07:36:52.049 INFO:teuthology.orchestra.run.smithi052.stdout: "error_string": "", 2024-05-27T07:36:52.049 INFO:teuthology.orchestra.run.smithi052.stdout: "module_options": { 2024-05-27T07:36:52.050 INFO:teuthology.orchestra.run.smithi052.stdout: "log_level": { 2024-05-27T07:36:52.050 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "log_level", 2024-05-27T07:36:52.050 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "str", 2024-05-27T07:36:52.050 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-05-27T07:36:52.050 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 1, 2024-05-27T07:36:52.050 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "", 2024-05-27T07:36:52.050 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-05-27T07:36:52.050 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-05-27T07:36:52.050 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [ 2024-05-27T07:36:52.050 INFO:teuthology.orchestra.run.smithi052.stdout: "", 2024-05-27T07:36:52.050 INFO:teuthology.orchestra.run.smithi052.stdout: "critical", 2024-05-27T07:36:52.050 INFO:teuthology.orchestra.run.smithi052.stdout: "debug", 2024-05-27T07:36:52.051 INFO:teuthology.orchestra.run.smithi052.stdout: "error", 2024-05-27T07:36:52.051 INFO:teuthology.orchestra.run.smithi052.stdout: "info", 2024-05-27T07:36:52.051 INFO:teuthology.orchestra.run.smithi052.stdout: "warning" 2024-05-27T07:36:52.051 INFO:teuthology.orchestra.run.smithi052.stdout: ], 2024-05-27T07:36:52.051 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-05-27T07:36:52.051 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-05-27T07:36:52.051 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-05-27T07:36:52.051 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-05-27T07:36:52.051 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-05-27T07:36:52.051 INFO:teuthology.orchestra.run.smithi052.stdout: "log_to_cluster": { 2024-05-27T07:36:52.051 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "log_to_cluster", 2024-05-27T07:36:52.051 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "bool", 2024-05-27T07:36:52.051 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-05-27T07:36:52.052 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 1, 2024-05-27T07:36:52.052 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "False", 2024-05-27T07:36:52.052 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-05-27T07:36:52.052 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-05-27T07:36:52.052 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-05-27T07:36:52.052 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-05-27T07:36:52.052 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-05-27T07:36:52.052 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-05-27T07:36:52.052 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-05-27T07:36:52.052 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-05-27T07:36:52.052 INFO:teuthology.orchestra.run.smithi052.stdout: "log_to_cluster_level": { 2024-05-27T07:36:52.052 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "log_to_cluster_level", 2024-05-27T07:36:52.052 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "str", 2024-05-27T07:36:52.053 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-05-27T07:36:52.053 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 1, 2024-05-27T07:36:52.053 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "info", 2024-05-27T07:36:52.053 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-05-27T07:36:52.053 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-05-27T07:36:52.053 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [ 2024-05-27T07:36:52.053 INFO:teuthology.orchestra.run.smithi052.stdout: "", 2024-05-27T07:36:52.053 INFO:teuthology.orchestra.run.smithi052.stdout: "critical", 2024-05-27T07:36:52.053 INFO:teuthology.orchestra.run.smithi052.stdout: "debug", 2024-05-27T07:36:52.053 INFO:teuthology.orchestra.run.smithi052.stdout: "error", 2024-05-27T07:36:52.053 INFO:teuthology.orchestra.run.smithi052.stdout: "info", 2024-05-27T07:36:52.053 INFO:teuthology.orchestra.run.smithi052.stdout: "warning" 2024-05-27T07:36:52.053 INFO:teuthology.orchestra.run.smithi052.stdout: ], 2024-05-27T07:36:52.054 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-05-27T07:36:52.054 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-05-27T07:36:52.054 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-05-27T07:36:52.054 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-05-27T07:36:52.054 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-05-27T07:36:52.054 INFO:teuthology.orchestra.run.smithi052.stdout: "log_to_file": { 2024-05-27T07:36:52.054 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "log_to_file", 2024-05-27T07:36:52.054 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "bool", 2024-05-27T07:36:52.054 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-05-27T07:36:52.054 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 1, 2024-05-27T07:36:52.054 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "False", 2024-05-27T07:36:52.054 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-05-27T07:36:52.055 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-05-27T07:36:52.055 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-05-27T07:36:52.055 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-05-27T07:36:52.055 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-05-27T07:36:52.055 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-05-27T07:36:52.055 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-05-27T07:36:52.055 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-05-27T07:36:52.055 INFO:teuthology.orchestra.run.smithi052.stdout: "sqlite3_killpoint": { 2024-05-27T07:36:52.055 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "sqlite3_killpoint", 2024-05-27T07:36:52.055 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "int", 2024-05-27T07:36:52.055 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "dev", 2024-05-27T07:36:52.055 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 1, 2024-05-27T07:36:52.055 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "0", 2024-05-27T07:36:52.056 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-05-27T07:36:52.056 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-05-27T07:36:52.056 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-05-27T07:36:52.056 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-05-27T07:36:52.056 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-05-27T07:36:52.056 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-05-27T07:36:52.056 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-05-27T07:36:52.056 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-05-27T07:36:52.056 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-05-27T07:36:52.056 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-05-27T07:36:52.056 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-05-27T07:36:52.056 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "mirroring", 2024-05-27T07:36:52.056 INFO:teuthology.orchestra.run.smithi052.stdout: "can_run": true, 2024-05-27T07:36:52.057 INFO:teuthology.orchestra.run.smithi052.stdout: "error_string": "", 2024-05-27T07:36:52.057 INFO:teuthology.orchestra.run.smithi052.stdout: "module_options": { 2024-05-27T07:36:52.057 INFO:teuthology.orchestra.run.smithi052.stdout: "log_level": { 2024-05-27T07:36:52.057 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "log_level", 2024-05-27T07:36:52.057 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "str", 2024-05-27T07:36:52.057 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-05-27T07:36:52.057 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 1, 2024-05-27T07:36:52.057 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "", 2024-05-27T07:36:52.057 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-05-27T07:36:52.057 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-05-27T07:36:52.057 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [ 2024-05-27T07:36:52.057 INFO:teuthology.orchestra.run.smithi052.stdout: "", 2024-05-27T07:36:52.057 INFO:teuthology.orchestra.run.smithi052.stdout: "critical", 2024-05-27T07:36:52.058 INFO:teuthology.orchestra.run.smithi052.stdout: "debug", 2024-05-27T07:36:52.058 INFO:teuthology.orchestra.run.smithi052.stdout: "error", 2024-05-27T07:36:52.058 INFO:teuthology.orchestra.run.smithi052.stdout: "info", 2024-05-27T07:36:52.058 INFO:teuthology.orchestra.run.smithi052.stdout: "warning" 2024-05-27T07:36:52.058 INFO:teuthology.orchestra.run.smithi052.stdout: ], 2024-05-27T07:36:52.058 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-05-27T07:36:52.058 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-05-27T07:36:52.058 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-05-27T07:36:52.058 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-05-27T07:36:52.058 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-05-27T07:36:52.058 INFO:teuthology.orchestra.run.smithi052.stdout: "log_to_cluster": { 2024-05-27T07:36:52.058 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "log_to_cluster", 2024-05-27T07:36:52.058 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "bool", 2024-05-27T07:36:52.059 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-05-27T07:36:52.059 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 1, 2024-05-27T07:36:52.059 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "False", 2024-05-27T07:36:52.059 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-05-27T07:36:52.059 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-05-27T07:36:52.059 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-05-27T07:36:52.059 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-05-27T07:36:52.059 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-05-27T07:36:52.059 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-05-27T07:36:52.059 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-05-27T07:36:52.059 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-05-27T07:36:52.059 INFO:teuthology.orchestra.run.smithi052.stdout: "log_to_cluster_level": { 2024-05-27T07:36:52.060 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "log_to_cluster_level", 2024-05-27T07:36:52.060 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "str", 2024-05-27T07:36:52.060 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-05-27T07:36:52.060 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 1, 2024-05-27T07:36:52.060 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "info", 2024-05-27T07:36:52.060 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-05-27T07:36:52.060 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-05-27T07:36:52.060 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [ 2024-05-27T07:36:52.060 INFO:teuthology.orchestra.run.smithi052.stdout: "", 2024-05-27T07:36:52.060 INFO:teuthology.orchestra.run.smithi052.stdout: "critical", 2024-05-27T07:36:52.060 INFO:teuthology.orchestra.run.smithi052.stdout: "debug", 2024-05-27T07:36:52.060 INFO:teuthology.orchestra.run.smithi052.stdout: "error", 2024-05-27T07:36:52.061 INFO:teuthology.orchestra.run.smithi052.stdout: "info", 2024-05-27T07:36:52.061 INFO:teuthology.orchestra.run.smithi052.stdout: "warning" 2024-05-27T07:36:52.061 INFO:teuthology.orchestra.run.smithi052.stdout: ], 2024-05-27T07:36:52.061 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-05-27T07:36:52.061 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-05-27T07:36:52.061 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-05-27T07:36:52.061 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-05-27T07:36:52.061 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-05-27T07:36:52.061 INFO:teuthology.orchestra.run.smithi052.stdout: "log_to_file": { 2024-05-27T07:36:52.061 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "log_to_file", 2024-05-27T07:36:52.061 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "bool", 2024-05-27T07:36:52.061 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-05-27T07:36:52.061 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 1, 2024-05-27T07:36:52.062 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "False", 2024-05-27T07:36:52.062 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-05-27T07:36:52.062 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-05-27T07:36:52.062 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-05-27T07:36:52.062 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-05-27T07:36:52.062 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-05-27T07:36:52.062 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-05-27T07:36:52.062 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-05-27T07:36:52.062 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-05-27T07:36:52.062 INFO:teuthology.orchestra.run.smithi052.stdout: "sqlite3_killpoint": { 2024-05-27T07:36:52.062 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "sqlite3_killpoint", 2024-05-27T07:36:52.062 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "int", 2024-05-27T07:36:52.063 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "dev", 2024-05-27T07:36:52.063 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 1, 2024-05-27T07:36:52.063 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "0", 2024-05-27T07:36:52.063 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-05-27T07:36:52.063 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-05-27T07:36:52.063 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-05-27T07:36:52.063 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-05-27T07:36:52.063 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-05-27T07:36:52.063 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-05-27T07:36:52.063 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-05-27T07:36:52.063 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-05-27T07:36:52.063 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-05-27T07:36:52.063 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-05-27T07:36:52.064 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-05-27T07:36:52.064 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "nfs", 2024-05-27T07:36:52.064 INFO:teuthology.orchestra.run.smithi052.stdout: "can_run": true, 2024-05-27T07:36:52.064 INFO:teuthology.orchestra.run.smithi052.stdout: "error_string": "", 2024-05-27T07:36:52.064 INFO:teuthology.orchestra.run.smithi052.stdout: "module_options": { 2024-05-27T07:36:52.064 INFO:teuthology.orchestra.run.smithi052.stdout: "log_level": { 2024-05-27T07:36:52.064 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "log_level", 2024-05-27T07:36:52.064 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "str", 2024-05-27T07:36:52.064 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-05-27T07:36:52.064 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 1, 2024-05-27T07:36:52.064 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "", 2024-05-27T07:36:52.064 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-05-27T07:36:52.064 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-05-27T07:36:52.065 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [ 2024-05-27T07:36:52.065 INFO:teuthology.orchestra.run.smithi052.stdout: "", 2024-05-27T07:36:52.065 INFO:teuthology.orchestra.run.smithi052.stdout: "critical", 2024-05-27T07:36:52.065 INFO:teuthology.orchestra.run.smithi052.stdout: "debug", 2024-05-27T07:36:52.065 INFO:teuthology.orchestra.run.smithi052.stdout: "error", 2024-05-27T07:36:52.065 INFO:teuthology.orchestra.run.smithi052.stdout: "info", 2024-05-27T07:36:52.065 INFO:teuthology.orchestra.run.smithi052.stdout: "warning" 2024-05-27T07:36:52.065 INFO:teuthology.orchestra.run.smithi052.stdout: ], 2024-05-27T07:36:52.065 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-05-27T07:36:52.065 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-05-27T07:36:52.065 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-05-27T07:36:52.065 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-05-27T07:36:52.066 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-05-27T07:36:52.066 INFO:teuthology.orchestra.run.smithi052.stdout: "log_to_cluster": { 2024-05-27T07:36:52.066 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "log_to_cluster", 2024-05-27T07:36:52.066 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "bool", 2024-05-27T07:36:52.066 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-05-27T07:36:52.066 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 1, 2024-05-27T07:36:52.066 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "False", 2024-05-27T07:36:52.066 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-05-27T07:36:52.066 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-05-27T07:36:52.066 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-05-27T07:36:52.066 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-05-27T07:36:52.066 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-05-27T07:36:52.066 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-05-27T07:36:52.067 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-05-27T07:36:52.067 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-05-27T07:36:52.067 INFO:teuthology.orchestra.run.smithi052.stdout: "log_to_cluster_level": { 2024-05-27T07:36:52.067 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "log_to_cluster_level", 2024-05-27T07:36:52.067 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "str", 2024-05-27T07:36:52.067 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-05-27T07:36:52.067 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 1, 2024-05-27T07:36:52.067 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "info", 2024-05-27T07:36:52.067 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-05-27T07:36:52.067 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-05-27T07:36:52.067 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [ 2024-05-27T07:36:52.067 INFO:teuthology.orchestra.run.smithi052.stdout: "", 2024-05-27T07:36:52.067 INFO:teuthology.orchestra.run.smithi052.stdout: "critical", 2024-05-27T07:36:52.068 INFO:teuthology.orchestra.run.smithi052.stdout: "debug", 2024-05-27T07:36:52.068 INFO:teuthology.orchestra.run.smithi052.stdout: "error", 2024-05-27T07:36:52.068 INFO:teuthology.orchestra.run.smithi052.stdout: "info", 2024-05-27T07:36:52.068 INFO:teuthology.orchestra.run.smithi052.stdout: "warning" 2024-05-27T07:36:52.068 INFO:teuthology.orchestra.run.smithi052.stdout: ], 2024-05-27T07:36:52.068 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-05-27T07:36:52.068 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-05-27T07:36:52.068 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-05-27T07:36:52.068 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-05-27T07:36:52.068 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-05-27T07:36:52.068 INFO:teuthology.orchestra.run.smithi052.stdout: "log_to_file": { 2024-05-27T07:36:52.068 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "log_to_file", 2024-05-27T07:36:52.068 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "bool", 2024-05-27T07:36:52.069 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-05-27T07:36:52.069 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 1, 2024-05-27T07:36:52.069 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "False", 2024-05-27T07:36:52.069 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-05-27T07:36:52.069 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-05-27T07:36:52.069 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-05-27T07:36:52.069 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-05-27T07:36:52.069 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-05-27T07:36:52.069 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-05-27T07:36:52.069 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-05-27T07:36:52.069 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-05-27T07:36:52.069 INFO:teuthology.orchestra.run.smithi052.stdout: "sqlite3_killpoint": { 2024-05-27T07:36:52.069 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "sqlite3_killpoint", 2024-05-27T07:36:52.070 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "int", 2024-05-27T07:36:52.070 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "dev", 2024-05-27T07:36:52.070 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 1, 2024-05-27T07:36:52.070 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "0", 2024-05-27T07:36:52.070 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-05-27T07:36:52.070 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-05-27T07:36:52.070 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-05-27T07:36:52.070 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-05-27T07:36:52.070 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-05-27T07:36:52.070 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-05-27T07:36:52.070 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-05-27T07:36:52.070 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-05-27T07:36:52.070 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-05-27T07:36:52.071 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-05-27T07:36:52.071 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-05-27T07:36:52.071 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "orchestrator", 2024-05-27T07:36:52.071 INFO:teuthology.orchestra.run.smithi052.stdout: "can_run": true, 2024-05-27T07:36:52.071 INFO:teuthology.orchestra.run.smithi052.stdout: "error_string": "", 2024-05-27T07:36:52.071 INFO:teuthology.orchestra.run.smithi052.stdout: "module_options": { 2024-05-27T07:36:52.071 INFO:teuthology.orchestra.run.smithi052.stdout: "fail_fs": { 2024-05-27T07:36:52.071 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "fail_fs", 2024-05-27T07:36:52.071 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "bool", 2024-05-27T07:36:52.071 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-05-27T07:36:52.071 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 0, 2024-05-27T07:36:52.071 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "False", 2024-05-27T07:36:52.071 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-05-27T07:36:52.072 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-05-27T07:36:52.072 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-05-27T07:36:52.072 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "Fail filesystem for rapid multi-rank mds upgrade", 2024-05-27T07:36:52.072 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-05-27T07:36:52.072 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-05-27T07:36:52.072 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-05-27T07:36:52.072 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-05-27T07:36:52.072 INFO:teuthology.orchestra.run.smithi052.stdout: "log_level": { 2024-05-27T07:36:52.072 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "log_level", 2024-05-27T07:36:52.072 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "str", 2024-05-27T07:36:52.072 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-05-27T07:36:52.072 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 1, 2024-05-27T07:36:52.072 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "", 2024-05-27T07:36:52.072 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-05-27T07:36:52.073 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-05-27T07:36:52.073 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [ 2024-05-27T07:36:52.073 INFO:teuthology.orchestra.run.smithi052.stdout: "", 2024-05-27T07:36:52.073 INFO:teuthology.orchestra.run.smithi052.stdout: "critical", 2024-05-27T07:36:52.073 INFO:teuthology.orchestra.run.smithi052.stdout: "debug", 2024-05-27T07:36:52.073 INFO:teuthology.orchestra.run.smithi052.stdout: "error", 2024-05-27T07:36:52.073 INFO:teuthology.orchestra.run.smithi052.stdout: "info", 2024-05-27T07:36:52.073 INFO:teuthology.orchestra.run.smithi052.stdout: "warning" 2024-05-27T07:36:52.073 INFO:teuthology.orchestra.run.smithi052.stdout: ], 2024-05-27T07:36:52.073 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-05-27T07:36:52.073 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-05-27T07:36:52.073 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-05-27T07:36:52.073 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-05-27T07:36:52.074 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-05-27T07:36:52.074 INFO:teuthology.orchestra.run.smithi052.stdout: "log_to_cluster": { 2024-05-27T07:36:52.074 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "log_to_cluster", 2024-05-27T07:36:52.074 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "bool", 2024-05-27T07:36:52.074 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-05-27T07:36:52.074 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 1, 2024-05-27T07:36:52.074 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "False", 2024-05-27T07:36:52.074 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-05-27T07:36:52.074 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-05-27T07:36:52.074 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-05-27T07:36:52.074 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-05-27T07:36:52.074 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-05-27T07:36:52.075 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-05-27T07:36:52.075 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-05-27T07:36:52.075 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-05-27T07:36:52.075 INFO:teuthology.orchestra.run.smithi052.stdout: "log_to_cluster_level": { 2024-05-27T07:36:52.075 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "log_to_cluster_level", 2024-05-27T07:36:52.075 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "str", 2024-05-27T07:36:52.075 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-05-27T07:36:52.075 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 1, 2024-05-27T07:36:52.075 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "info", 2024-05-27T07:36:52.075 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-05-27T07:36:52.075 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-05-27T07:36:52.075 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [ 2024-05-27T07:36:52.075 INFO:teuthology.orchestra.run.smithi052.stdout: "", 2024-05-27T07:36:52.076 INFO:teuthology.orchestra.run.smithi052.stdout: "critical", 2024-05-27T07:36:52.076 INFO:teuthology.orchestra.run.smithi052.stdout: "debug", 2024-05-27T07:36:52.076 INFO:teuthology.orchestra.run.smithi052.stdout: "error", 2024-05-27T07:36:52.076 INFO:teuthology.orchestra.run.smithi052.stdout: "info", 2024-05-27T07:36:52.076 INFO:teuthology.orchestra.run.smithi052.stdout: "warning" 2024-05-27T07:36:52.076 INFO:teuthology.orchestra.run.smithi052.stdout: ], 2024-05-27T07:36:52.076 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-05-27T07:36:52.076 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-05-27T07:36:52.076 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-05-27T07:36:52.076 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-05-27T07:36:52.076 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-05-27T07:36:52.076 INFO:teuthology.orchestra.run.smithi052.stdout: "log_to_file": { 2024-05-27T07:36:52.076 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "log_to_file", 2024-05-27T07:36:52.076 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "bool", 2024-05-27T07:36:52.077 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-05-27T07:36:52.077 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 1, 2024-05-27T07:36:52.077 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "False", 2024-05-27T07:36:52.077 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-05-27T07:36:52.077 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-05-27T07:36:52.077 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-05-27T07:36:52.077 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-05-27T07:36:52.077 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-05-27T07:36:52.077 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-05-27T07:36:52.077 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-05-27T07:36:52.077 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-05-27T07:36:52.077 INFO:teuthology.orchestra.run.smithi052.stdout: "orchestrator": { 2024-05-27T07:36:52.078 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "orchestrator", 2024-05-27T07:36:52.078 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "str", 2024-05-27T07:36:52.078 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-05-27T07:36:52.078 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 1, 2024-05-27T07:36:52.078 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "", 2024-05-27T07:36:52.078 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-05-27T07:36:52.078 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-05-27T07:36:52.078 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [ 2024-05-27T07:36:52.078 INFO:teuthology.orchestra.run.smithi052.stdout: "cephadm", 2024-05-27T07:36:52.078 INFO:teuthology.orchestra.run.smithi052.stdout: "rook", 2024-05-27T07:36:52.078 INFO:teuthology.orchestra.run.smithi052.stdout: "test_orchestrator" 2024-05-27T07:36:52.078 INFO:teuthology.orchestra.run.smithi052.stdout: ], 2024-05-27T07:36:52.078 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "Orchestrator backend", 2024-05-27T07:36:52.079 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-05-27T07:36:52.079 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-05-27T07:36:52.079 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-05-27T07:36:52.079 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-05-27T07:36:52.079 INFO:teuthology.orchestra.run.smithi052.stdout: "sqlite3_killpoint": { 2024-05-27T07:36:52.079 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "sqlite3_killpoint", 2024-05-27T07:36:52.079 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "int", 2024-05-27T07:36:52.079 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "dev", 2024-05-27T07:36:52.079 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 1, 2024-05-27T07:36:52.079 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "0", 2024-05-27T07:36:52.079 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-05-27T07:36:52.079 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-05-27T07:36:52.079 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-05-27T07:36:52.080 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-05-27T07:36:52.080 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-05-27T07:36:52.080 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-05-27T07:36:52.080 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-05-27T07:36:52.080 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-05-27T07:36:52.080 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-05-27T07:36:52.080 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-05-27T07:36:52.080 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-05-27T07:36:52.080 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "osd_perf_query", 2024-05-27T07:36:52.080 INFO:teuthology.orchestra.run.smithi052.stdout: "can_run": true, 2024-05-27T07:36:52.080 INFO:teuthology.orchestra.run.smithi052.stdout: "error_string": "", 2024-05-27T07:36:52.080 INFO:teuthology.orchestra.run.smithi052.stdout: "module_options": { 2024-05-27T07:36:52.080 INFO:teuthology.orchestra.run.smithi052.stdout: "log_level": { 2024-05-27T07:36:52.081 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "log_level", 2024-05-27T07:36:52.081 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "str", 2024-05-27T07:36:52.081 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-05-27T07:36:52.081 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 1, 2024-05-27T07:36:52.081 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "", 2024-05-27T07:36:52.081 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-05-27T07:36:52.081 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-05-27T07:36:52.081 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [ 2024-05-27T07:36:52.081 INFO:teuthology.orchestra.run.smithi052.stdout: "", 2024-05-27T07:36:52.081 INFO:teuthology.orchestra.run.smithi052.stdout: "critical", 2024-05-27T07:36:52.081 INFO:teuthology.orchestra.run.smithi052.stdout: "debug", 2024-05-27T07:36:52.081 INFO:teuthology.orchestra.run.smithi052.stdout: "error", 2024-05-27T07:36:52.081 INFO:teuthology.orchestra.run.smithi052.stdout: "info", 2024-05-27T07:36:52.082 INFO:teuthology.orchestra.run.smithi052.stdout: "warning" 2024-05-27T07:36:52.082 INFO:teuthology.orchestra.run.smithi052.stdout: ], 2024-05-27T07:36:52.082 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-05-27T07:36:52.082 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-05-27T07:36:52.082 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-05-27T07:36:52.082 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-05-27T07:36:52.082 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-05-27T07:36:52.082 INFO:teuthology.orchestra.run.smithi052.stdout: "log_to_cluster": { 2024-05-27T07:36:52.082 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "log_to_cluster", 2024-05-27T07:36:52.082 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "bool", 2024-05-27T07:36:52.082 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-05-27T07:36:52.082 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 1, 2024-05-27T07:36:52.082 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "False", 2024-05-27T07:36:52.083 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-05-27T07:36:52.083 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-05-27T07:36:52.083 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-05-27T07:36:52.083 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-05-27T07:36:52.083 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-05-27T07:36:52.083 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-05-27T07:36:52.083 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-05-27T07:36:52.083 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-05-27T07:36:52.083 INFO:teuthology.orchestra.run.smithi052.stdout: "log_to_cluster_level": { 2024-05-27T07:36:52.083 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "log_to_cluster_level", 2024-05-27T07:36:52.083 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "str", 2024-05-27T07:36:52.083 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-05-27T07:36:52.084 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 1, 2024-05-27T07:36:52.084 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "info", 2024-05-27T07:36:52.084 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-05-27T07:36:52.084 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-05-27T07:36:52.084 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [ 2024-05-27T07:36:52.084 INFO:teuthology.orchestra.run.smithi052.stdout: "", 2024-05-27T07:36:52.084 INFO:teuthology.orchestra.run.smithi052.stdout: "critical", 2024-05-27T07:36:52.084 INFO:teuthology.orchestra.run.smithi052.stdout: "debug", 2024-05-27T07:36:52.084 INFO:teuthology.orchestra.run.smithi052.stdout: "error", 2024-05-27T07:36:52.084 INFO:teuthology.orchestra.run.smithi052.stdout: "info", 2024-05-27T07:36:52.084 INFO:teuthology.orchestra.run.smithi052.stdout: "warning" 2024-05-27T07:36:52.084 INFO:teuthology.orchestra.run.smithi052.stdout: ], 2024-05-27T07:36:52.084 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-05-27T07:36:52.085 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-05-27T07:36:52.085 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-05-27T07:36:52.085 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-05-27T07:36:52.085 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-05-27T07:36:52.085 INFO:teuthology.orchestra.run.smithi052.stdout: "log_to_file": { 2024-05-27T07:36:52.085 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "log_to_file", 2024-05-27T07:36:52.085 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "bool", 2024-05-27T07:36:52.085 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-05-27T07:36:52.085 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 1, 2024-05-27T07:36:52.085 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "False", 2024-05-27T07:36:52.085 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-05-27T07:36:52.085 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-05-27T07:36:52.085 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-05-27T07:36:52.086 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-05-27T07:36:52.086 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-05-27T07:36:52.086 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-05-27T07:36:52.086 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-05-27T07:36:52.086 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-05-27T07:36:52.086 INFO:teuthology.orchestra.run.smithi052.stdout: "sqlite3_killpoint": { 2024-05-27T07:36:52.086 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "sqlite3_killpoint", 2024-05-27T07:36:52.086 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "int", 2024-05-27T07:36:52.086 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "dev", 2024-05-27T07:36:52.086 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 1, 2024-05-27T07:36:52.086 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "0", 2024-05-27T07:36:52.086 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-05-27T07:36:52.086 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-05-27T07:36:52.087 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-05-27T07:36:52.087 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-05-27T07:36:52.087 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-05-27T07:36:52.087 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-05-27T07:36:52.087 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-05-27T07:36:52.087 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-05-27T07:36:52.087 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-05-27T07:36:52.087 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-05-27T07:36:52.087 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-05-27T07:36:52.087 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "osd_support", 2024-05-27T07:36:52.087 INFO:teuthology.orchestra.run.smithi052.stdout: "can_run": true, 2024-05-27T07:36:52.087 INFO:teuthology.orchestra.run.smithi052.stdout: "error_string": "", 2024-05-27T07:36:52.087 INFO:teuthology.orchestra.run.smithi052.stdout: "module_options": { 2024-05-27T07:36:52.088 INFO:teuthology.orchestra.run.smithi052.stdout: "log_level": { 2024-05-27T07:36:52.088 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "log_level", 2024-05-27T07:36:52.088 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "str", 2024-05-27T07:36:52.088 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-05-27T07:36:52.088 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 1, 2024-05-27T07:36:52.088 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "", 2024-05-27T07:36:52.088 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-05-27T07:36:52.088 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-05-27T07:36:52.088 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [ 2024-05-27T07:36:52.088 INFO:teuthology.orchestra.run.smithi052.stdout: "", 2024-05-27T07:36:52.088 INFO:teuthology.orchestra.run.smithi052.stdout: "critical", 2024-05-27T07:36:52.088 INFO:teuthology.orchestra.run.smithi052.stdout: "debug", 2024-05-27T07:36:52.089 INFO:teuthology.orchestra.run.smithi052.stdout: "error", 2024-05-27T07:36:52.089 INFO:teuthology.orchestra.run.smithi052.stdout: "info", 2024-05-27T07:36:52.089 INFO:teuthology.orchestra.run.smithi052.stdout: "warning" 2024-05-27T07:36:52.089 INFO:teuthology.orchestra.run.smithi052.stdout: ], 2024-05-27T07:36:52.089 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-05-27T07:36:52.089 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-05-27T07:36:52.089 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-05-27T07:36:52.089 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-05-27T07:36:52.089 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-05-27T07:36:52.089 INFO:teuthology.orchestra.run.smithi052.stdout: "log_to_cluster": { 2024-05-27T07:36:52.089 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "log_to_cluster", 2024-05-27T07:36:52.089 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "bool", 2024-05-27T07:36:52.090 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-05-27T07:36:52.090 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 1, 2024-05-27T07:36:52.090 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "False", 2024-05-27T07:36:52.090 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-05-27T07:36:52.090 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-05-27T07:36:52.090 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-05-27T07:36:52.090 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-05-27T07:36:52.090 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-05-27T07:36:52.090 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-05-27T07:36:52.090 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-05-27T07:36:52.090 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-05-27T07:36:52.090 INFO:teuthology.orchestra.run.smithi052.stdout: "log_to_cluster_level": { 2024-05-27T07:36:52.091 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "log_to_cluster_level", 2024-05-27T07:36:52.091 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "str", 2024-05-27T07:36:52.091 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-05-27T07:36:52.091 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 1, 2024-05-27T07:36:52.091 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "info", 2024-05-27T07:36:52.091 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-05-27T07:36:52.091 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-05-27T07:36:52.091 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [ 2024-05-27T07:36:52.091 INFO:teuthology.orchestra.run.smithi052.stdout: "", 2024-05-27T07:36:52.091 INFO:teuthology.orchestra.run.smithi052.stdout: "critical", 2024-05-27T07:36:52.091 INFO:teuthology.orchestra.run.smithi052.stdout: "debug", 2024-05-27T07:36:52.091 INFO:teuthology.orchestra.run.smithi052.stdout: "error", 2024-05-27T07:36:52.091 INFO:teuthology.orchestra.run.smithi052.stdout: "info", 2024-05-27T07:36:52.092 INFO:teuthology.orchestra.run.smithi052.stdout: "warning" 2024-05-27T07:36:52.092 INFO:teuthology.orchestra.run.smithi052.stdout: ], 2024-05-27T07:36:52.092 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-05-27T07:36:52.092 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-05-27T07:36:52.092 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-05-27T07:36:52.092 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-05-27T07:36:52.092 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-05-27T07:36:52.092 INFO:teuthology.orchestra.run.smithi052.stdout: "log_to_file": { 2024-05-27T07:36:52.092 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "log_to_file", 2024-05-27T07:36:52.092 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "bool", 2024-05-27T07:36:52.092 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-05-27T07:36:52.092 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 1, 2024-05-27T07:36:52.093 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "False", 2024-05-27T07:36:52.093 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-05-27T07:36:52.093 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-05-27T07:36:52.093 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-05-27T07:36:52.093 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-05-27T07:36:52.093 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-05-27T07:36:52.093 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-05-27T07:36:52.093 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-05-27T07:36:52.093 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-05-27T07:36:52.093 INFO:teuthology.orchestra.run.smithi052.stdout: "sqlite3_killpoint": { 2024-05-27T07:36:52.093 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "sqlite3_killpoint", 2024-05-27T07:36:52.093 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "int", 2024-05-27T07:36:52.093 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "dev", 2024-05-27T07:36:52.094 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 1, 2024-05-27T07:36:52.094 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "0", 2024-05-27T07:36:52.094 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-05-27T07:36:52.094 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-05-27T07:36:52.094 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-05-27T07:36:52.094 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-05-27T07:36:52.094 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-05-27T07:36:52.094 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-05-27T07:36:52.094 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-05-27T07:36:52.094 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-05-27T07:36:52.094 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-05-27T07:36:52.094 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-05-27T07:36:52.094 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-05-27T07:36:52.095 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "pg_autoscaler", 2024-05-27T07:36:52.095 INFO:teuthology.orchestra.run.smithi052.stdout: "can_run": true, 2024-05-27T07:36:52.095 INFO:teuthology.orchestra.run.smithi052.stdout: "error_string": "", 2024-05-27T07:36:52.095 INFO:teuthology.orchestra.run.smithi052.stdout: "module_options": { 2024-05-27T07:36:52.095 INFO:teuthology.orchestra.run.smithi052.stdout: "log_level": { 2024-05-27T07:36:52.095 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "log_level", 2024-05-27T07:36:52.095 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "str", 2024-05-27T07:36:52.095 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-05-27T07:36:52.095 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 1, 2024-05-27T07:36:52.095 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "", 2024-05-27T07:36:52.095 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-05-27T07:36:52.095 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-05-27T07:36:52.096 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [ 2024-05-27T07:36:52.096 INFO:teuthology.orchestra.run.smithi052.stdout: "", 2024-05-27T07:36:52.096 INFO:teuthology.orchestra.run.smithi052.stdout: "critical", 2024-05-27T07:36:52.096 INFO:teuthology.orchestra.run.smithi052.stdout: "debug", 2024-05-27T07:36:52.096 INFO:teuthology.orchestra.run.smithi052.stdout: "error", 2024-05-27T07:36:52.096 INFO:teuthology.orchestra.run.smithi052.stdout: "info", 2024-05-27T07:36:52.096 INFO:teuthology.orchestra.run.smithi052.stdout: "warning" 2024-05-27T07:36:52.096 INFO:teuthology.orchestra.run.smithi052.stdout: ], 2024-05-27T07:36:52.096 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-05-27T07:36:52.096 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-05-27T07:36:52.096 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-05-27T07:36:52.096 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-05-27T07:36:52.096 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-05-27T07:36:52.097 INFO:teuthology.orchestra.run.smithi052.stdout: "log_to_cluster": { 2024-05-27T07:36:52.097 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "log_to_cluster", 2024-05-27T07:36:52.097 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "bool", 2024-05-27T07:36:52.097 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-05-27T07:36:52.097 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 1, 2024-05-27T07:36:52.097 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "False", 2024-05-27T07:36:52.097 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-05-27T07:36:52.097 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-05-27T07:36:52.097 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-05-27T07:36:52.097 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-05-27T07:36:52.097 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-05-27T07:36:52.097 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-05-27T07:36:52.097 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-05-27T07:36:52.098 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-05-27T07:36:52.098 INFO:teuthology.orchestra.run.smithi052.stdout: "log_to_cluster_level": { 2024-05-27T07:36:52.098 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "log_to_cluster_level", 2024-05-27T07:36:52.098 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "str", 2024-05-27T07:36:52.098 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-05-27T07:36:52.098 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 1, 2024-05-27T07:36:52.098 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "info", 2024-05-27T07:36:52.098 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-05-27T07:36:52.098 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-05-27T07:36:52.098 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [ 2024-05-27T07:36:52.098 INFO:teuthology.orchestra.run.smithi052.stdout: "", 2024-05-27T07:36:52.098 INFO:teuthology.orchestra.run.smithi052.stdout: "critical", 2024-05-27T07:36:52.099 INFO:teuthology.orchestra.run.smithi052.stdout: "debug", 2024-05-27T07:36:52.099 INFO:teuthology.orchestra.run.smithi052.stdout: "error", 2024-05-27T07:36:52.099 INFO:teuthology.orchestra.run.smithi052.stdout: "info", 2024-05-27T07:36:52.099 INFO:teuthology.orchestra.run.smithi052.stdout: "warning" 2024-05-27T07:36:52.099 INFO:teuthology.orchestra.run.smithi052.stdout: ], 2024-05-27T07:36:52.099 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-05-27T07:36:52.099 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-05-27T07:36:52.099 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-05-27T07:36:52.099 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-05-27T07:36:52.099 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-05-27T07:36:52.099 INFO:teuthology.orchestra.run.smithi052.stdout: "log_to_file": { 2024-05-27T07:36:52.099 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "log_to_file", 2024-05-27T07:36:52.099 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "bool", 2024-05-27T07:36:52.100 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-05-27T07:36:52.100 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 1, 2024-05-27T07:36:52.100 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "False", 2024-05-27T07:36:52.100 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-05-27T07:36:52.100 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-05-27T07:36:52.100 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-05-27T07:36:52.100 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-05-27T07:36:52.100 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-05-27T07:36:52.100 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-05-27T07:36:52.100 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-05-27T07:36:52.100 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-05-27T07:36:52.100 INFO:teuthology.orchestra.run.smithi052.stdout: "sleep_interval": { 2024-05-27T07:36:52.100 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "sleep_interval", 2024-05-27T07:36:52.100 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "secs", 2024-05-27T07:36:52.101 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-05-27T07:36:52.101 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 0, 2024-05-27T07:36:52.101 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "60", 2024-05-27T07:36:52.101 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-05-27T07:36:52.101 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-05-27T07:36:52.101 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-05-27T07:36:52.101 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-05-27T07:36:52.101 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-05-27T07:36:52.101 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-05-27T07:36:52.101 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-05-27T07:36:52.101 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-05-27T07:36:52.101 INFO:teuthology.orchestra.run.smithi052.stdout: "sqlite3_killpoint": { 2024-05-27T07:36:52.101 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "sqlite3_killpoint", 2024-05-27T07:36:52.102 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "int", 2024-05-27T07:36:52.102 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "dev", 2024-05-27T07:36:52.102 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 1, 2024-05-27T07:36:52.102 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "0", 2024-05-27T07:36:52.102 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-05-27T07:36:52.102 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-05-27T07:36:52.102 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-05-27T07:36:52.102 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-05-27T07:36:52.102 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-05-27T07:36:52.102 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-05-27T07:36:52.102 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-05-27T07:36:52.102 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-05-27T07:36:52.102 INFO:teuthology.orchestra.run.smithi052.stdout: "threshold": { 2024-05-27T07:36:52.103 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "threshold", 2024-05-27T07:36:52.103 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "float", 2024-05-27T07:36:52.103 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-05-27T07:36:52.103 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 0, 2024-05-27T07:36:52.103 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "3.0", 2024-05-27T07:36:52.103 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "1.0", 2024-05-27T07:36:52.103 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-05-27T07:36:52.103 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-05-27T07:36:52.103 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "scaling threshold", 2024-05-27T07:36:52.103 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0", 2024-05-27T07:36:52.103 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-05-27T07:36:52.103 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-05-27T07:36:52.103 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-05-27T07:36:52.104 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-05-27T07:36:52.104 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-05-27T07:36:52.104 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-05-27T07:36:52.104 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "progress", 2024-05-27T07:36:52.104 INFO:teuthology.orchestra.run.smithi052.stdout: "can_run": true, 2024-05-27T07:36:52.104 INFO:teuthology.orchestra.run.smithi052.stdout: "error_string": "", 2024-05-27T07:36:52.104 INFO:teuthology.orchestra.run.smithi052.stdout: "module_options": { 2024-05-27T07:36:52.104 INFO:teuthology.orchestra.run.smithi052.stdout: "allow_pg_recovery_event": { 2024-05-27T07:36:52.104 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "allow_pg_recovery_event", 2024-05-27T07:36:52.104 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "bool", 2024-05-27T07:36:52.104 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-05-27T07:36:52.104 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 1, 2024-05-27T07:36:52.105 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "False", 2024-05-27T07:36:52.105 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-05-27T07:36:52.105 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-05-27T07:36:52.105 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-05-27T07:36:52.105 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "allow the module to show pg recovery progress", 2024-05-27T07:36:52.105 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-05-27T07:36:52.105 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-05-27T07:36:52.105 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-05-27T07:36:52.105 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-05-27T07:36:52.105 INFO:teuthology.orchestra.run.smithi052.stdout: "enabled": { 2024-05-27T07:36:52.105 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "enabled", 2024-05-27T07:36:52.105 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "bool", 2024-05-27T07:36:52.106 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-05-27T07:36:52.106 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 0, 2024-05-27T07:36:52.106 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "True", 2024-05-27T07:36:52.106 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-05-27T07:36:52.106 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-05-27T07:36:52.106 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-05-27T07:36:52.106 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-05-27T07:36:52.106 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-05-27T07:36:52.106 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-05-27T07:36:52.106 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-05-27T07:36:52.106 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-05-27T07:36:52.106 INFO:teuthology.orchestra.run.smithi052.stdout: "log_level": { 2024-05-27T07:36:52.106 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "log_level", 2024-05-27T07:36:52.107 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "str", 2024-05-27T07:36:52.107 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-05-27T07:36:52.107 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 1, 2024-05-27T07:36:52.107 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "", 2024-05-27T07:36:52.107 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-05-27T07:36:52.107 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-05-27T07:36:52.107 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [ 2024-05-27T07:36:52.107 INFO:teuthology.orchestra.run.smithi052.stdout: "", 2024-05-27T07:36:52.107 INFO:teuthology.orchestra.run.smithi052.stdout: "critical", 2024-05-27T07:36:52.107 INFO:teuthology.orchestra.run.smithi052.stdout: "debug", 2024-05-27T07:36:52.107 INFO:teuthology.orchestra.run.smithi052.stdout: "error", 2024-05-27T07:36:52.107 INFO:teuthology.orchestra.run.smithi052.stdout: "info", 2024-05-27T07:36:52.107 INFO:teuthology.orchestra.run.smithi052.stdout: "warning" 2024-05-27T07:36:52.108 INFO:teuthology.orchestra.run.smithi052.stdout: ], 2024-05-27T07:36:52.108 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-05-27T07:36:52.108 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-05-27T07:36:52.108 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-05-27T07:36:52.108 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-05-27T07:36:52.108 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-05-27T07:36:52.108 INFO:teuthology.orchestra.run.smithi052.stdout: "log_to_cluster": { 2024-05-27T07:36:52.108 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "log_to_cluster", 2024-05-27T07:36:52.108 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "bool", 2024-05-27T07:36:52.108 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-05-27T07:36:52.108 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 1, 2024-05-27T07:36:52.108 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "False", 2024-05-27T07:36:52.109 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-05-27T07:36:52.109 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-05-27T07:36:52.109 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-05-27T07:36:52.109 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-05-27T07:36:52.109 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-05-27T07:36:52.109 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-05-27T07:36:52.109 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-05-27T07:36:52.109 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-05-27T07:36:52.109 INFO:teuthology.orchestra.run.smithi052.stdout: "log_to_cluster_level": { 2024-05-27T07:36:52.109 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "log_to_cluster_level", 2024-05-27T07:36:52.109 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "str", 2024-05-27T07:36:52.109 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-05-27T07:36:52.109 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 1, 2024-05-27T07:36:52.110 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "info", 2024-05-27T07:36:52.110 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-05-27T07:36:52.110 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-05-27T07:36:52.110 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [ 2024-05-27T07:36:52.110 INFO:teuthology.orchestra.run.smithi052.stdout: "", 2024-05-27T07:36:52.110 INFO:teuthology.orchestra.run.smithi052.stdout: "critical", 2024-05-27T07:36:52.110 INFO:teuthology.orchestra.run.smithi052.stdout: "debug", 2024-05-27T07:36:52.110 INFO:teuthology.orchestra.run.smithi052.stdout: "error", 2024-05-27T07:36:52.110 INFO:teuthology.orchestra.run.smithi052.stdout: "info", 2024-05-27T07:36:52.110 INFO:teuthology.orchestra.run.smithi052.stdout: "warning" 2024-05-27T07:36:52.110 INFO:teuthology.orchestra.run.smithi052.stdout: ], 2024-05-27T07:36:52.110 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-05-27T07:36:52.111 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-05-27T07:36:52.111 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-05-27T07:36:52.111 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-05-27T07:36:52.111 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-05-27T07:36:52.111 INFO:teuthology.orchestra.run.smithi052.stdout: "log_to_file": { 2024-05-27T07:36:52.111 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "log_to_file", 2024-05-27T07:36:52.111 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "bool", 2024-05-27T07:36:52.111 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-05-27T07:36:52.111 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 1, 2024-05-27T07:36:52.111 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "False", 2024-05-27T07:36:52.111 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-05-27T07:36:52.111 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-05-27T07:36:52.111 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-05-27T07:36:52.112 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-05-27T07:36:52.112 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-05-27T07:36:52.112 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-05-27T07:36:52.112 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-05-27T07:36:52.112 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-05-27T07:36:52.112 INFO:teuthology.orchestra.run.smithi052.stdout: "max_completed_events": { 2024-05-27T07:36:52.112 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "max_completed_events", 2024-05-27T07:36:52.112 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "int", 2024-05-27T07:36:52.112 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-05-27T07:36:52.112 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 1, 2024-05-27T07:36:52.112 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "50", 2024-05-27T07:36:52.112 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-05-27T07:36:52.112 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-05-27T07:36:52.113 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-05-27T07:36:52.113 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "number of past completed events to remember", 2024-05-27T07:36:52.113 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-05-27T07:36:52.113 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-05-27T07:36:52.113 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-05-27T07:36:52.113 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-05-27T07:36:52.113 INFO:teuthology.orchestra.run.smithi052.stdout: "sleep_interval": { 2024-05-27T07:36:52.113 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "sleep_interval", 2024-05-27T07:36:52.113 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "secs", 2024-05-27T07:36:52.113 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-05-27T07:36:52.113 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 1, 2024-05-27T07:36:52.113 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "5", 2024-05-27T07:36:52.114 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-05-27T07:36:52.114 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-05-27T07:36:52.114 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-05-27T07:36:52.114 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "how long the module is going to sleep", 2024-05-27T07:36:52.114 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-05-27T07:36:52.114 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-05-27T07:36:52.114 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-05-27T07:36:52.114 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-05-27T07:36:52.114 INFO:teuthology.orchestra.run.smithi052.stdout: "sqlite3_killpoint": { 2024-05-27T07:36:52.114 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "sqlite3_killpoint", 2024-05-27T07:36:52.114 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "int", 2024-05-27T07:36:52.114 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "dev", 2024-05-27T07:36:52.114 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 1, 2024-05-27T07:36:52.115 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "0", 2024-05-27T07:36:52.115 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-05-27T07:36:52.115 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-05-27T07:36:52.115 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-05-27T07:36:52.115 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-05-27T07:36:52.115 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-05-27T07:36:52.115 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-05-27T07:36:52.115 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-05-27T07:36:52.115 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-05-27T07:36:52.115 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-05-27T07:36:52.115 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-05-27T07:36:52.115 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-05-27T07:36:52.115 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "prometheus", 2024-05-27T07:36:52.116 INFO:teuthology.orchestra.run.smithi052.stdout: "can_run": true, 2024-05-27T07:36:52.116 INFO:teuthology.orchestra.run.smithi052.stdout: "error_string": "", 2024-05-27T07:36:52.116 INFO:teuthology.orchestra.run.smithi052.stdout: "module_options": { 2024-05-27T07:36:52.116 INFO:teuthology.orchestra.run.smithi052.stdout: "cache": { 2024-05-27T07:36:52.116 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "cache", 2024-05-27T07:36:52.116 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "bool", 2024-05-27T07:36:52.116 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-05-27T07:36:52.116 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 0, 2024-05-27T07:36:52.116 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "True", 2024-05-27T07:36:52.116 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-05-27T07:36:52.116 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-05-27T07:36:52.116 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-05-27T07:36:52.116 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-05-27T07:36:52.116 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-05-27T07:36:52.117 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-05-27T07:36:52.117 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-05-27T07:36:52.117 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-05-27T07:36:52.117 INFO:teuthology.orchestra.run.smithi052.stdout: "exclude_perf_counters": { 2024-05-27T07:36:52.117 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "exclude_perf_counters", 2024-05-27T07:36:52.117 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "bool", 2024-05-27T07:36:52.117 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-05-27T07:36:52.117 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 1, 2024-05-27T07:36:52.117 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "True", 2024-05-27T07:36:52.117 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-05-27T07:36:52.117 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-05-27T07:36:52.117 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-05-27T07:36:52.117 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "Do not include perf-counters in the metrics output", 2024-05-27T07:36:52.118 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.", 2024-05-27T07:36:52.118 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-05-27T07:36:52.118 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-05-27T07:36:52.118 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-05-27T07:36:52.118 INFO:teuthology.orchestra.run.smithi052.stdout: "log_level": { 2024-05-27T07:36:52.118 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "log_level", 2024-05-27T07:36:52.118 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "str", 2024-05-27T07:36:52.118 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-05-27T07:36:52.118 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 1, 2024-05-27T07:36:52.118 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "", 2024-05-27T07:36:52.118 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-05-27T07:36:52.118 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-05-27T07:36:52.119 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [ 2024-05-27T07:36:52.119 INFO:teuthology.orchestra.run.smithi052.stdout: "", 2024-05-27T07:36:52.119 INFO:teuthology.orchestra.run.smithi052.stdout: "critical", 2024-05-27T07:36:52.119 INFO:teuthology.orchestra.run.smithi052.stdout: "debug", 2024-05-27T07:36:52.119 INFO:teuthology.orchestra.run.smithi052.stdout: "error", 2024-05-27T07:36:52.119 INFO:teuthology.orchestra.run.smithi052.stdout: "info", 2024-05-27T07:36:52.119 INFO:teuthology.orchestra.run.smithi052.stdout: "warning" 2024-05-27T07:36:52.119 INFO:teuthology.orchestra.run.smithi052.stdout: ], 2024-05-27T07:36:52.119 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-05-27T07:36:52.120 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-05-27T07:36:52.120 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-05-27T07:36:52.120 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-05-27T07:36:52.120 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-05-27T07:36:52.120 INFO:teuthology.orchestra.run.smithi052.stdout: "log_to_cluster": { 2024-05-27T07:36:52.120 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "log_to_cluster", 2024-05-27T07:36:52.120 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "bool", 2024-05-27T07:36:52.120 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-05-27T07:36:52.120 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 1, 2024-05-27T07:36:52.120 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "False", 2024-05-27T07:36:52.120 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-05-27T07:36:52.120 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-05-27T07:36:52.121 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-05-27T07:36:52.121 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-05-27T07:36:52.121 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-05-27T07:36:52.121 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-05-27T07:36:52.121 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-05-27T07:36:52.121 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-05-27T07:36:52.121 INFO:teuthology.orchestra.run.smithi052.stdout: "log_to_cluster_level": { 2024-05-27T07:36:52.121 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "log_to_cluster_level", 2024-05-27T07:36:52.121 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "str", 2024-05-27T07:36:52.121 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-05-27T07:36:52.121 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 1, 2024-05-27T07:36:52.121 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "info", 2024-05-27T07:36:52.122 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-05-27T07:36:52.122 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-05-27T07:36:52.122 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [ 2024-05-27T07:36:52.122 INFO:teuthology.orchestra.run.smithi052.stdout: "", 2024-05-27T07:36:52.122 INFO:teuthology.orchestra.run.smithi052.stdout: "critical", 2024-05-27T07:36:52.122 INFO:teuthology.orchestra.run.smithi052.stdout: "debug", 2024-05-27T07:36:52.122 INFO:teuthology.orchestra.run.smithi052.stdout: "error", 2024-05-27T07:36:52.122 INFO:teuthology.orchestra.run.smithi052.stdout: "info", 2024-05-27T07:36:52.122 INFO:teuthology.orchestra.run.smithi052.stdout: "warning" 2024-05-27T07:36:52.122 INFO:teuthology.orchestra.run.smithi052.stdout: ], 2024-05-27T07:36:52.122 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-05-27T07:36:52.122 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-05-27T07:36:52.122 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-05-27T07:36:52.123 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-05-27T07:36:52.123 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-05-27T07:36:52.123 INFO:teuthology.orchestra.run.smithi052.stdout: "log_to_file": { 2024-05-27T07:36:52.123 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "log_to_file", 2024-05-27T07:36:52.123 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "bool", 2024-05-27T07:36:52.123 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-05-27T07:36:52.123 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 1, 2024-05-27T07:36:52.123 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "False", 2024-05-27T07:36:52.123 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-05-27T07:36:52.123 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-05-27T07:36:52.123 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-05-27T07:36:52.123 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-05-27T07:36:52.124 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-05-27T07:36:52.124 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-05-27T07:36:52.124 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-05-27T07:36:52.124 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-05-27T07:36:52.124 INFO:teuthology.orchestra.run.smithi052.stdout: "rbd_stats_pools": { 2024-05-27T07:36:52.124 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "rbd_stats_pools", 2024-05-27T07:36:52.124 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "str", 2024-05-27T07:36:52.124 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-05-27T07:36:52.124 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 0, 2024-05-27T07:36:52.124 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "", 2024-05-27T07:36:52.124 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-05-27T07:36:52.124 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-05-27T07:36:52.124 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-05-27T07:36:52.125 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-05-27T07:36:52.125 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-05-27T07:36:52.125 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-05-27T07:36:52.125 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-05-27T07:36:52.125 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-05-27T07:36:52.125 INFO:teuthology.orchestra.run.smithi052.stdout: "rbd_stats_pools_refresh_interval": { 2024-05-27T07:36:52.125 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "rbd_stats_pools_refresh_interval", 2024-05-27T07:36:52.125 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "int", 2024-05-27T07:36:52.125 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-05-27T07:36:52.125 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 0, 2024-05-27T07:36:52.125 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "300", 2024-05-27T07:36:52.125 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-05-27T07:36:52.126 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-05-27T07:36:52.126 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-05-27T07:36:52.126 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-05-27T07:36:52.126 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-05-27T07:36:52.126 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-05-27T07:36:52.126 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-05-27T07:36:52.126 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-05-27T07:36:52.126 INFO:teuthology.orchestra.run.smithi052.stdout: "scrape_interval": { 2024-05-27T07:36:52.126 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "scrape_interval", 2024-05-27T07:36:52.126 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "float", 2024-05-27T07:36:52.126 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-05-27T07:36:52.126 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 0, 2024-05-27T07:36:52.126 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "15.0", 2024-05-27T07:36:52.127 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-05-27T07:36:52.127 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-05-27T07:36:52.127 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-05-27T07:36:52.127 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-05-27T07:36:52.127 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-05-27T07:36:52.127 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-05-27T07:36:52.127 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-05-27T07:36:52.127 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-05-27T07:36:52.127 INFO:teuthology.orchestra.run.smithi052.stdout: "server_addr": { 2024-05-27T07:36:52.127 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "server_addr", 2024-05-27T07:36:52.127 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "str", 2024-05-27T07:36:52.127 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-05-27T07:36:52.128 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 0, 2024-05-27T07:36:52.128 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "::", 2024-05-27T07:36:52.128 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-05-27T07:36:52.128 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-05-27T07:36:52.128 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-05-27T07:36:52.128 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "the IPv4 or IPv6 address on which the module listens for HTTP requests", 2024-05-27T07:36:52.128 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-05-27T07:36:52.128 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-05-27T07:36:52.128 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-05-27T07:36:52.128 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-05-27T07:36:52.128 INFO:teuthology.orchestra.run.smithi052.stdout: "server_port": { 2024-05-27T07:36:52.128 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "server_port", 2024-05-27T07:36:52.129 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "int", 2024-05-27T07:36:52.129 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-05-27T07:36:52.129 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 1, 2024-05-27T07:36:52.129 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "9283", 2024-05-27T07:36:52.129 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-05-27T07:36:52.129 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-05-27T07:36:52.129 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-05-27T07:36:52.129 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "the port on which the module listens for HTTP requests", 2024-05-27T07:36:52.129 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-05-27T07:36:52.129 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-05-27T07:36:52.129 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-05-27T07:36:52.129 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-05-27T07:36:52.129 INFO:teuthology.orchestra.run.smithi052.stdout: "sqlite3_killpoint": { 2024-05-27T07:36:52.130 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "sqlite3_killpoint", 2024-05-27T07:36:52.130 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "int", 2024-05-27T07:36:52.130 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "dev", 2024-05-27T07:36:52.130 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 1, 2024-05-27T07:36:52.130 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "0", 2024-05-27T07:36:52.130 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-05-27T07:36:52.130 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-05-27T07:36:52.130 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-05-27T07:36:52.130 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-05-27T07:36:52.130 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-05-27T07:36:52.130 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-05-27T07:36:52.130 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-05-27T07:36:52.131 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-05-27T07:36:52.131 INFO:teuthology.orchestra.run.smithi052.stdout: "stale_cache_strategy": { 2024-05-27T07:36:52.131 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "stale_cache_strategy", 2024-05-27T07:36:52.131 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "str", 2024-05-27T07:36:52.131 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-05-27T07:36:52.131 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 0, 2024-05-27T07:36:52.131 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "log", 2024-05-27T07:36:52.131 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-05-27T07:36:52.131 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-05-27T07:36:52.131 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-05-27T07:36:52.131 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-05-27T07:36:52.131 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-05-27T07:36:52.131 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-05-27T07:36:52.132 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-05-27T07:36:52.132 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-05-27T07:36:52.132 INFO:teuthology.orchestra.run.smithi052.stdout: "standby_behaviour": { 2024-05-27T07:36:52.132 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "standby_behaviour", 2024-05-27T07:36:52.132 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "str", 2024-05-27T07:36:52.132 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-05-27T07:36:52.132 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 1, 2024-05-27T07:36:52.132 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "default", 2024-05-27T07:36:52.132 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-05-27T07:36:52.132 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-05-27T07:36:52.132 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [ 2024-05-27T07:36:52.132 INFO:teuthology.orchestra.run.smithi052.stdout: "default", 2024-05-27T07:36:52.132 INFO:teuthology.orchestra.run.smithi052.stdout: "error" 2024-05-27T07:36:52.133 INFO:teuthology.orchestra.run.smithi052.stdout: ], 2024-05-27T07:36:52.133 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-05-27T07:36:52.133 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-05-27T07:36:52.133 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-05-27T07:36:52.133 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-05-27T07:36:52.133 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-05-27T07:36:52.133 INFO:teuthology.orchestra.run.smithi052.stdout: "standby_error_status_code": { 2024-05-27T07:36:52.133 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "standby_error_status_code", 2024-05-27T07:36:52.133 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "int", 2024-05-27T07:36:52.133 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-05-27T07:36:52.133 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 1, 2024-05-27T07:36:52.133 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "500", 2024-05-27T07:36:52.133 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "400", 2024-05-27T07:36:52.134 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "599", 2024-05-27T07:36:52.134 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-05-27T07:36:52.134 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-05-27T07:36:52.134 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-05-27T07:36:52.134 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-05-27T07:36:52.134 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-05-27T07:36:52.134 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-05-27T07:36:52.134 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-05-27T07:36:52.134 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-05-27T07:36:52.134 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-05-27T07:36:52.134 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "rbd_support", 2024-05-27T07:36:52.135 INFO:teuthology.orchestra.run.smithi052.stdout: "can_run": true, 2024-05-27T07:36:52.135 INFO:teuthology.orchestra.run.smithi052.stdout: "error_string": "", 2024-05-27T07:36:52.135 INFO:teuthology.orchestra.run.smithi052.stdout: "module_options": { 2024-05-27T07:36:52.135 INFO:teuthology.orchestra.run.smithi052.stdout: "log_level": { 2024-05-27T07:36:52.135 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "log_level", 2024-05-27T07:36:52.135 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "str", 2024-05-27T07:36:52.135 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-05-27T07:36:52.135 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 1, 2024-05-27T07:36:52.135 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "", 2024-05-27T07:36:52.135 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-05-27T07:36:52.135 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-05-27T07:36:52.135 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [ 2024-05-27T07:36:52.135 INFO:teuthology.orchestra.run.smithi052.stdout: "", 2024-05-27T07:36:52.136 INFO:teuthology.orchestra.run.smithi052.stdout: "critical", 2024-05-27T07:36:52.136 INFO:teuthology.orchestra.run.smithi052.stdout: "debug", 2024-05-27T07:36:52.136 INFO:teuthology.orchestra.run.smithi052.stdout: "error", 2024-05-27T07:36:52.136 INFO:teuthology.orchestra.run.smithi052.stdout: "info", 2024-05-27T07:36:52.136 INFO:teuthology.orchestra.run.smithi052.stdout: "warning" 2024-05-27T07:36:52.136 INFO:teuthology.orchestra.run.smithi052.stdout: ], 2024-05-27T07:36:52.136 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-05-27T07:36:52.136 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-05-27T07:36:52.136 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-05-27T07:36:52.136 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-05-27T07:36:52.136 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-05-27T07:36:52.136 INFO:teuthology.orchestra.run.smithi052.stdout: "log_to_cluster": { 2024-05-27T07:36:52.137 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "log_to_cluster", 2024-05-27T07:36:52.137 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "bool", 2024-05-27T07:36:52.137 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-05-27T07:36:52.137 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 1, 2024-05-27T07:36:52.137 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "False", 2024-05-27T07:36:52.137 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-05-27T07:36:52.137 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-05-27T07:36:52.137 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-05-27T07:36:52.137 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-05-27T07:36:52.137 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-05-27T07:36:52.137 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-05-27T07:36:52.137 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-05-27T07:36:52.137 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-05-27T07:36:52.138 INFO:teuthology.orchestra.run.smithi052.stdout: "log_to_cluster_level": { 2024-05-27T07:36:52.138 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "log_to_cluster_level", 2024-05-27T07:36:52.138 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "str", 2024-05-27T07:36:52.138 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-05-27T07:36:52.138 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 1, 2024-05-27T07:36:52.138 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "info", 2024-05-27T07:36:52.138 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-05-27T07:36:52.138 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-05-27T07:36:52.138 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [ 2024-05-27T07:36:52.138 INFO:teuthology.orchestra.run.smithi052.stdout: "", 2024-05-27T07:36:52.138 INFO:teuthology.orchestra.run.smithi052.stdout: "critical", 2024-05-27T07:36:52.138 INFO:teuthology.orchestra.run.smithi052.stdout: "debug", 2024-05-27T07:36:52.139 INFO:teuthology.orchestra.run.smithi052.stdout: "error", 2024-05-27T07:36:52.139 INFO:teuthology.orchestra.run.smithi052.stdout: "info", 2024-05-27T07:36:52.139 INFO:teuthology.orchestra.run.smithi052.stdout: "warning" 2024-05-27T07:36:52.139 INFO:teuthology.orchestra.run.smithi052.stdout: ], 2024-05-27T07:36:52.139 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-05-27T07:36:52.139 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-05-27T07:36:52.139 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-05-27T07:36:52.139 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-05-27T07:36:52.139 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-05-27T07:36:52.139 INFO:teuthology.orchestra.run.smithi052.stdout: "log_to_file": { 2024-05-27T07:36:52.139 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "log_to_file", 2024-05-27T07:36:52.139 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "bool", 2024-05-27T07:36:52.139 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-05-27T07:36:52.140 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 1, 2024-05-27T07:36:52.140 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "False", 2024-05-27T07:36:52.140 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-05-27T07:36:52.140 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-05-27T07:36:52.140 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-05-27T07:36:52.140 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-05-27T07:36:52.140 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-05-27T07:36:52.140 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-05-27T07:36:52.140 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-05-27T07:36:52.140 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-05-27T07:36:52.140 INFO:teuthology.orchestra.run.smithi052.stdout: "max_concurrent_snap_create": { 2024-05-27T07:36:52.140 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "max_concurrent_snap_create", 2024-05-27T07:36:52.141 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "int", 2024-05-27T07:36:52.141 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-05-27T07:36:52.141 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 0, 2024-05-27T07:36:52.141 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "10", 2024-05-27T07:36:52.141 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-05-27T07:36:52.141 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-05-27T07:36:52.141 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-05-27T07:36:52.141 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-05-27T07:36:52.141 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-05-27T07:36:52.141 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-05-27T07:36:52.141 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-05-27T07:36:52.141 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-05-27T07:36:52.141 INFO:teuthology.orchestra.run.smithi052.stdout: "mirror_snapshot_schedule": { 2024-05-27T07:36:52.142 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "mirror_snapshot_schedule", 2024-05-27T07:36:52.142 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "str", 2024-05-27T07:36:52.142 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-05-27T07:36:52.142 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 0, 2024-05-27T07:36:52.142 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "", 2024-05-27T07:36:52.142 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-05-27T07:36:52.142 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-05-27T07:36:52.142 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-05-27T07:36:52.142 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-05-27T07:36:52.142 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-05-27T07:36:52.142 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-05-27T07:36:52.142 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-05-27T07:36:52.142 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-05-27T07:36:52.143 INFO:teuthology.orchestra.run.smithi052.stdout: "sqlite3_killpoint": { 2024-05-27T07:36:52.143 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "sqlite3_killpoint", 2024-05-27T07:36:52.143 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "int", 2024-05-27T07:36:52.143 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "dev", 2024-05-27T07:36:52.143 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 1, 2024-05-27T07:36:52.143 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "0", 2024-05-27T07:36:52.143 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-05-27T07:36:52.143 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-05-27T07:36:52.143 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-05-27T07:36:52.143 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-05-27T07:36:52.143 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-05-27T07:36:52.143 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-05-27T07:36:52.143 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-05-27T07:36:52.143 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-05-27T07:36:52.144 INFO:teuthology.orchestra.run.smithi052.stdout: "trash_purge_schedule": { 2024-05-27T07:36:52.144 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "trash_purge_schedule", 2024-05-27T07:36:52.144 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "str", 2024-05-27T07:36:52.144 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-05-27T07:36:52.144 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 0, 2024-05-27T07:36:52.144 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "", 2024-05-27T07:36:52.144 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-05-27T07:36:52.144 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-05-27T07:36:52.144 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-05-27T07:36:52.144 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-05-27T07:36:52.144 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-05-27T07:36:52.144 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-05-27T07:36:52.144 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-05-27T07:36:52.145 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-05-27T07:36:52.145 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-05-27T07:36:52.145 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-05-27T07:36:52.145 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-05-27T07:36:52.145 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "restful", 2024-05-27T07:36:52.145 INFO:teuthology.orchestra.run.smithi052.stdout: "can_run": true, 2024-05-27T07:36:52.145 INFO:teuthology.orchestra.run.smithi052.stdout: "error_string": "", 2024-05-27T07:36:52.145 INFO:teuthology.orchestra.run.smithi052.stdout: "module_options": { 2024-05-27T07:36:52.145 INFO:teuthology.orchestra.run.smithi052.stdout: "enable_auth": { 2024-05-27T07:36:52.145 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "enable_auth", 2024-05-27T07:36:52.145 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "bool", 2024-05-27T07:36:52.145 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-05-27T07:36:52.145 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 0, 2024-05-27T07:36:52.146 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "True", 2024-05-27T07:36:52.146 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-05-27T07:36:52.146 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-05-27T07:36:52.146 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-05-27T07:36:52.146 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-05-27T07:36:52.146 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-05-27T07:36:52.146 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-05-27T07:36:52.146 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-05-27T07:36:52.146 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-05-27T07:36:52.146 INFO:teuthology.orchestra.run.smithi052.stdout: "key_file": { 2024-05-27T07:36:52.146 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "key_file", 2024-05-27T07:36:52.146 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "str", 2024-05-27T07:36:52.146 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-05-27T07:36:52.147 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 0, 2024-05-27T07:36:52.147 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "", 2024-05-27T07:36:52.147 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-05-27T07:36:52.147 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-05-27T07:36:52.147 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-05-27T07:36:52.147 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-05-27T07:36:52.147 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-05-27T07:36:52.147 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-05-27T07:36:52.147 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-05-27T07:36:52.147 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-05-27T07:36:52.147 INFO:teuthology.orchestra.run.smithi052.stdout: "log_level": { 2024-05-27T07:36:52.147 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "log_level", 2024-05-27T07:36:52.147 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "str", 2024-05-27T07:36:52.148 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-05-27T07:36:52.148 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 1, 2024-05-27T07:36:52.148 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "", 2024-05-27T07:36:52.148 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-05-27T07:36:52.148 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-05-27T07:36:52.148 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [ 2024-05-27T07:36:52.148 INFO:teuthology.orchestra.run.smithi052.stdout: "", 2024-05-27T07:36:52.148 INFO:teuthology.orchestra.run.smithi052.stdout: "critical", 2024-05-27T07:36:52.148 INFO:teuthology.orchestra.run.smithi052.stdout: "debug", 2024-05-27T07:36:52.148 INFO:teuthology.orchestra.run.smithi052.stdout: "error", 2024-05-27T07:36:52.148 INFO:teuthology.orchestra.run.smithi052.stdout: "info", 2024-05-27T07:36:52.148 INFO:teuthology.orchestra.run.smithi052.stdout: "warning" 2024-05-27T07:36:52.149 INFO:teuthology.orchestra.run.smithi052.stdout: ], 2024-05-27T07:36:52.149 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-05-27T07:36:52.149 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-05-27T07:36:52.149 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-05-27T07:36:52.149 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-05-27T07:36:52.149 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-05-27T07:36:52.149 INFO:teuthology.orchestra.run.smithi052.stdout: "log_to_cluster": { 2024-05-27T07:36:52.149 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "log_to_cluster", 2024-05-27T07:36:52.149 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "bool", 2024-05-27T07:36:52.149 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-05-27T07:36:52.149 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 1, 2024-05-27T07:36:52.149 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "False", 2024-05-27T07:36:52.149 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-05-27T07:36:52.150 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-05-27T07:36:52.150 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-05-27T07:36:52.150 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-05-27T07:36:52.150 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-05-27T07:36:52.150 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-05-27T07:36:52.150 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-05-27T07:36:52.150 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-05-27T07:36:52.150 INFO:teuthology.orchestra.run.smithi052.stdout: "log_to_cluster_level": { 2024-05-27T07:36:52.150 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "log_to_cluster_level", 2024-05-27T07:36:52.150 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "str", 2024-05-27T07:36:52.150 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-05-27T07:36:52.150 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 1, 2024-05-27T07:36:52.151 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "info", 2024-05-27T07:36:52.151 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-05-27T07:36:52.151 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-05-27T07:36:52.151 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [ 2024-05-27T07:36:52.151 INFO:teuthology.orchestra.run.smithi052.stdout: "", 2024-05-27T07:36:52.151 INFO:teuthology.orchestra.run.smithi052.stdout: "critical", 2024-05-27T07:36:52.151 INFO:teuthology.orchestra.run.smithi052.stdout: "debug", 2024-05-27T07:36:52.151 INFO:teuthology.orchestra.run.smithi052.stdout: "error", 2024-05-27T07:36:52.151 INFO:teuthology.orchestra.run.smithi052.stdout: "info", 2024-05-27T07:36:52.151 INFO:teuthology.orchestra.run.smithi052.stdout: "warning" 2024-05-27T07:36:52.151 INFO:teuthology.orchestra.run.smithi052.stdout: ], 2024-05-27T07:36:52.151 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-05-27T07:36:52.151 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-05-27T07:36:52.152 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-05-27T07:36:52.152 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-05-27T07:36:52.152 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-05-27T07:36:52.152 INFO:teuthology.orchestra.run.smithi052.stdout: "log_to_file": { 2024-05-27T07:36:52.152 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "log_to_file", 2024-05-27T07:36:52.152 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "bool", 2024-05-27T07:36:52.152 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-05-27T07:36:52.152 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 1, 2024-05-27T07:36:52.152 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "False", 2024-05-27T07:36:52.152 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-05-27T07:36:52.152 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-05-27T07:36:52.152 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-05-27T07:36:52.152 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-05-27T07:36:52.153 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-05-27T07:36:52.153 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-05-27T07:36:52.153 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-05-27T07:36:52.153 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-05-27T07:36:52.153 INFO:teuthology.orchestra.run.smithi052.stdout: "server_addr": { 2024-05-27T07:36:52.153 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "server_addr", 2024-05-27T07:36:52.153 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "str", 2024-05-27T07:36:52.153 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-05-27T07:36:52.153 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 0, 2024-05-27T07:36:52.153 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "", 2024-05-27T07:36:52.153 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-05-27T07:36:52.153 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-05-27T07:36:52.153 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-05-27T07:36:52.154 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-05-27T07:36:52.154 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-05-27T07:36:52.154 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-05-27T07:36:52.154 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-05-27T07:36:52.154 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-05-27T07:36:52.154 INFO:teuthology.orchestra.run.smithi052.stdout: "server_port": { 2024-05-27T07:36:52.154 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "server_port", 2024-05-27T07:36:52.154 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "str", 2024-05-27T07:36:52.154 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-05-27T07:36:52.154 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 0, 2024-05-27T07:36:52.154 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "", 2024-05-27T07:36:52.154 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-05-27T07:36:52.155 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-05-27T07:36:52.155 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-05-27T07:36:52.155 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-05-27T07:36:52.155 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-05-27T07:36:52.155 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-05-27T07:36:52.155 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-05-27T07:36:52.155 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-05-27T07:36:52.155 INFO:teuthology.orchestra.run.smithi052.stdout: "sqlite3_killpoint": { 2024-05-27T07:36:52.155 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "sqlite3_killpoint", 2024-05-27T07:36:52.155 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "int", 2024-05-27T07:36:52.155 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "dev", 2024-05-27T07:36:52.155 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 1, 2024-05-27T07:36:52.156 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "0", 2024-05-27T07:36:52.156 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-05-27T07:36:52.156 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-05-27T07:36:52.156 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-05-27T07:36:52.156 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-05-27T07:36:52.156 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-05-27T07:36:52.156 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-05-27T07:36:52.156 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-05-27T07:36:52.156 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-05-27T07:36:52.156 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-05-27T07:36:52.156 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-05-27T07:36:52.156 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-05-27T07:36:52.156 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "rgw", 2024-05-27T07:36:52.157 INFO:teuthology.orchestra.run.smithi052.stdout: "can_run": true, 2024-05-27T07:36:52.157 INFO:teuthology.orchestra.run.smithi052.stdout: "error_string": "", 2024-05-27T07:36:52.157 INFO:teuthology.orchestra.run.smithi052.stdout: "module_options": { 2024-05-27T07:36:52.157 INFO:teuthology.orchestra.run.smithi052.stdout: "log_level": { 2024-05-27T07:36:52.157 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "log_level", 2024-05-27T07:36:52.157 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "str", 2024-05-27T07:36:52.157 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-05-27T07:36:52.157 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 1, 2024-05-27T07:36:52.157 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "", 2024-05-27T07:36:52.157 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-05-27T07:36:52.157 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-05-27T07:36:52.157 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [ 2024-05-27T07:36:52.157 INFO:teuthology.orchestra.run.smithi052.stdout: "", 2024-05-27T07:36:52.158 INFO:teuthology.orchestra.run.smithi052.stdout: "critical", 2024-05-27T07:36:52.158 INFO:teuthology.orchestra.run.smithi052.stdout: "debug", 2024-05-27T07:36:52.158 INFO:teuthology.orchestra.run.smithi052.stdout: "error", 2024-05-27T07:36:52.158 INFO:teuthology.orchestra.run.smithi052.stdout: "info", 2024-05-27T07:36:52.158 INFO:teuthology.orchestra.run.smithi052.stdout: "warning" 2024-05-27T07:36:52.158 INFO:teuthology.orchestra.run.smithi052.stdout: ], 2024-05-27T07:36:52.158 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-05-27T07:36:52.158 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-05-27T07:36:52.158 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-05-27T07:36:52.158 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-05-27T07:36:52.158 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-05-27T07:36:52.158 INFO:teuthology.orchestra.run.smithi052.stdout: "log_to_cluster": { 2024-05-27T07:36:52.159 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "log_to_cluster", 2024-05-27T07:36:52.159 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "bool", 2024-05-27T07:36:52.159 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-05-27T07:36:52.159 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 1, 2024-05-27T07:36:52.159 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "False", 2024-05-27T07:36:52.159 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-05-27T07:36:52.159 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-05-27T07:36:52.159 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-05-27T07:36:52.159 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-05-27T07:36:52.159 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-05-27T07:36:52.159 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-05-27T07:36:52.159 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-05-27T07:36:52.159 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-05-27T07:36:52.160 INFO:teuthology.orchestra.run.smithi052.stdout: "log_to_cluster_level": { 2024-05-27T07:36:52.160 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "log_to_cluster_level", 2024-05-27T07:36:52.160 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "str", 2024-05-27T07:36:52.160 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-05-27T07:36:52.160 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 1, 2024-05-27T07:36:52.160 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "info", 2024-05-27T07:36:52.160 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-05-27T07:36:52.160 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-05-27T07:36:52.160 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [ 2024-05-27T07:36:52.160 INFO:teuthology.orchestra.run.smithi052.stdout: "", 2024-05-27T07:36:52.160 INFO:teuthology.orchestra.run.smithi052.stdout: "critical", 2024-05-27T07:36:52.160 INFO:teuthology.orchestra.run.smithi052.stdout: "debug", 2024-05-27T07:36:52.160 INFO:teuthology.orchestra.run.smithi052.stdout: "error", 2024-05-27T07:36:52.161 INFO:teuthology.orchestra.run.smithi052.stdout: "info", 2024-05-27T07:36:52.161 INFO:teuthology.orchestra.run.smithi052.stdout: "warning" 2024-05-27T07:36:52.161 INFO:teuthology.orchestra.run.smithi052.stdout: ], 2024-05-27T07:36:52.161 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-05-27T07:36:52.161 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-05-27T07:36:52.161 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-05-27T07:36:52.161 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-05-27T07:36:52.161 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-05-27T07:36:52.161 INFO:teuthology.orchestra.run.smithi052.stdout: "log_to_file": { 2024-05-27T07:36:52.161 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "log_to_file", 2024-05-27T07:36:52.161 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "bool", 2024-05-27T07:36:52.161 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-05-27T07:36:52.161 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 1, 2024-05-27T07:36:52.162 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "False", 2024-05-27T07:36:52.162 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-05-27T07:36:52.162 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-05-27T07:36:52.162 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-05-27T07:36:52.162 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-05-27T07:36:52.162 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-05-27T07:36:52.162 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-05-27T07:36:52.162 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-05-27T07:36:52.162 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-05-27T07:36:52.162 INFO:teuthology.orchestra.run.smithi052.stdout: "sqlite3_killpoint": { 2024-05-27T07:36:52.162 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "sqlite3_killpoint", 2024-05-27T07:36:52.162 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "int", 2024-05-27T07:36:52.162 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "dev", 2024-05-27T07:36:52.163 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 1, 2024-05-27T07:36:52.165 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "0", 2024-05-27T07:36:52.165 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-05-27T07:36:52.165 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-05-27T07:36:52.165 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-05-27T07:36:52.165 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-05-27T07:36:52.165 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-05-27T07:36:52.166 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-05-27T07:36:52.166 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-05-27T07:36:52.166 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-05-27T07:36:52.166 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-05-27T07:36:52.166 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-05-27T07:36:52.166 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-05-27T07:36:52.166 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "rook", 2024-05-27T07:36:52.166 INFO:teuthology.orchestra.run.smithi052.stdout: "can_run": true, 2024-05-27T07:36:52.166 INFO:teuthology.orchestra.run.smithi052.stdout: "error_string": "", 2024-05-27T07:36:52.166 INFO:teuthology.orchestra.run.smithi052.stdout: "module_options": { 2024-05-27T07:36:52.166 INFO:teuthology.orchestra.run.smithi052.stdout: "log_level": { 2024-05-27T07:36:52.167 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "log_level", 2024-05-27T07:36:52.167 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "str", 2024-05-27T07:36:52.167 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-05-27T07:36:52.167 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 1, 2024-05-27T07:36:52.167 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "", 2024-05-27T07:36:52.167 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-05-27T07:36:52.167 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-05-27T07:36:52.167 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [ 2024-05-27T07:36:52.167 INFO:teuthology.orchestra.run.smithi052.stdout: "", 2024-05-27T07:36:52.167 INFO:teuthology.orchestra.run.smithi052.stdout: "critical", 2024-05-27T07:36:52.167 INFO:teuthology.orchestra.run.smithi052.stdout: "debug", 2024-05-27T07:36:52.167 INFO:teuthology.orchestra.run.smithi052.stdout: "error", 2024-05-27T07:36:52.167 INFO:teuthology.orchestra.run.smithi052.stdout: "info", 2024-05-27T07:36:52.168 INFO:teuthology.orchestra.run.smithi052.stdout: "warning" 2024-05-27T07:36:52.168 INFO:teuthology.orchestra.run.smithi052.stdout: ], 2024-05-27T07:36:52.168 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-05-27T07:36:52.168 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-05-27T07:36:52.168 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-05-27T07:36:52.168 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-05-27T07:36:52.168 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-05-27T07:36:52.168 INFO:teuthology.orchestra.run.smithi052.stdout: "log_to_cluster": { 2024-05-27T07:36:52.168 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "log_to_cluster", 2024-05-27T07:36:52.168 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "bool", 2024-05-27T07:36:52.168 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-05-27T07:36:52.169 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 1, 2024-05-27T07:36:52.169 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "False", 2024-05-27T07:36:52.169 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-05-27T07:36:52.169 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-05-27T07:36:52.169 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-05-27T07:36:52.169 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-05-27T07:36:52.169 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-05-27T07:36:52.169 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-05-27T07:36:52.169 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-05-27T07:36:52.169 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-05-27T07:36:52.169 INFO:teuthology.orchestra.run.smithi052.stdout: "log_to_cluster_level": { 2024-05-27T07:36:52.169 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "log_to_cluster_level", 2024-05-27T07:36:52.169 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "str", 2024-05-27T07:36:52.170 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-05-27T07:36:52.170 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 1, 2024-05-27T07:36:52.170 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "info", 2024-05-27T07:36:52.170 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-05-27T07:36:52.170 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-05-27T07:36:52.170 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [ 2024-05-27T07:36:52.170 INFO:teuthology.orchestra.run.smithi052.stdout: "", 2024-05-27T07:36:52.170 INFO:teuthology.orchestra.run.smithi052.stdout: "critical", 2024-05-27T07:36:52.170 INFO:teuthology.orchestra.run.smithi052.stdout: "debug", 2024-05-27T07:36:52.170 INFO:teuthology.orchestra.run.smithi052.stdout: "error", 2024-05-27T07:36:52.170 INFO:teuthology.orchestra.run.smithi052.stdout: "info", 2024-05-27T07:36:52.170 INFO:teuthology.orchestra.run.smithi052.stdout: "warning" 2024-05-27T07:36:52.170 INFO:teuthology.orchestra.run.smithi052.stdout: ], 2024-05-27T07:36:52.171 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-05-27T07:36:52.171 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-05-27T07:36:52.171 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-05-27T07:36:52.171 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-05-27T07:36:52.171 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-05-27T07:36:52.171 INFO:teuthology.orchestra.run.smithi052.stdout: "log_to_file": { 2024-05-27T07:36:52.171 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "log_to_file", 2024-05-27T07:36:52.171 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "bool", 2024-05-27T07:36:52.171 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-05-27T07:36:52.171 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 1, 2024-05-27T07:36:52.171 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "False", 2024-05-27T07:36:52.171 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-05-27T07:36:52.172 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-05-27T07:36:52.172 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-05-27T07:36:52.172 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-05-27T07:36:52.172 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-05-27T07:36:52.172 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-05-27T07:36:52.172 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-05-27T07:36:52.172 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-05-27T07:36:52.172 INFO:teuthology.orchestra.run.smithi052.stdout: "sqlite3_killpoint": { 2024-05-27T07:36:52.172 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "sqlite3_killpoint", 2024-05-27T07:36:52.172 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "int", 2024-05-27T07:36:52.172 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "dev", 2024-05-27T07:36:52.172 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 1, 2024-05-27T07:36:52.173 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "0", 2024-05-27T07:36:52.173 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-05-27T07:36:52.173 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-05-27T07:36:52.173 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-05-27T07:36:52.173 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-05-27T07:36:52.173 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-05-27T07:36:52.173 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-05-27T07:36:52.173 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-05-27T07:36:52.173 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-05-27T07:36:52.173 INFO:teuthology.orchestra.run.smithi052.stdout: "storage_class": { 2024-05-27T07:36:52.173 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "storage_class", 2024-05-27T07:36:52.173 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "str", 2024-05-27T07:36:52.173 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-05-27T07:36:52.174 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 0, 2024-05-27T07:36:52.174 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "local", 2024-05-27T07:36:52.174 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-05-27T07:36:52.174 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-05-27T07:36:52.174 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-05-27T07:36:52.174 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "storage class name for LSO-discovered PVs", 2024-05-27T07:36:52.174 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-05-27T07:36:52.174 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-05-27T07:36:52.174 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-05-27T07:36:52.174 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-05-27T07:36:52.174 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-05-27T07:36:52.174 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-05-27T07:36:52.174 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-05-27T07:36:52.175 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "selftest", 2024-05-27T07:36:52.175 INFO:teuthology.orchestra.run.smithi052.stdout: "can_run": true, 2024-05-27T07:36:52.175 INFO:teuthology.orchestra.run.smithi052.stdout: "error_string": "", 2024-05-27T07:36:52.175 INFO:teuthology.orchestra.run.smithi052.stdout: "module_options": { 2024-05-27T07:36:52.175 INFO:teuthology.orchestra.run.smithi052.stdout: "log_level": { 2024-05-27T07:36:52.175 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "log_level", 2024-05-27T07:36:52.175 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "str", 2024-05-27T07:36:52.175 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-05-27T07:36:52.175 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 1, 2024-05-27T07:36:52.175 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "", 2024-05-27T07:36:52.175 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-05-27T07:36:52.175 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-05-27T07:36:52.175 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [ 2024-05-27T07:36:52.176 INFO:teuthology.orchestra.run.smithi052.stdout: "", 2024-05-27T07:36:52.176 INFO:teuthology.orchestra.run.smithi052.stdout: "critical", 2024-05-27T07:36:52.176 INFO:teuthology.orchestra.run.smithi052.stdout: "debug", 2024-05-27T07:36:52.176 INFO:teuthology.orchestra.run.smithi052.stdout: "error", 2024-05-27T07:36:52.176 INFO:teuthology.orchestra.run.smithi052.stdout: "info", 2024-05-27T07:36:52.176 INFO:teuthology.orchestra.run.smithi052.stdout: "warning" 2024-05-27T07:36:52.176 INFO:teuthology.orchestra.run.smithi052.stdout: ], 2024-05-27T07:36:52.176 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-05-27T07:36:52.176 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-05-27T07:36:52.176 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-05-27T07:36:52.176 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-05-27T07:36:52.176 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-05-27T07:36:52.176 INFO:teuthology.orchestra.run.smithi052.stdout: "log_to_cluster": { 2024-05-27T07:36:52.177 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "log_to_cluster", 2024-05-27T07:36:52.177 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "bool", 2024-05-27T07:36:52.177 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-05-27T07:36:52.177 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 1, 2024-05-27T07:36:52.177 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "False", 2024-05-27T07:36:52.177 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-05-27T07:36:52.177 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-05-27T07:36:52.177 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-05-27T07:36:52.177 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-05-27T07:36:52.177 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-05-27T07:36:52.177 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-05-27T07:36:52.177 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-05-27T07:36:52.177 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-05-27T07:36:52.178 INFO:teuthology.orchestra.run.smithi052.stdout: "log_to_cluster_level": { 2024-05-27T07:36:52.178 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "log_to_cluster_level", 2024-05-27T07:36:52.178 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "str", 2024-05-27T07:36:52.178 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-05-27T07:36:52.178 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 1, 2024-05-27T07:36:52.178 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "info", 2024-05-27T07:36:52.178 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-05-27T07:36:52.178 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-05-27T07:36:52.178 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [ 2024-05-27T07:36:52.178 INFO:teuthology.orchestra.run.smithi052.stdout: "", 2024-05-27T07:36:52.178 INFO:teuthology.orchestra.run.smithi052.stdout: "critical", 2024-05-27T07:36:52.178 INFO:teuthology.orchestra.run.smithi052.stdout: "debug", 2024-05-27T07:36:52.179 INFO:teuthology.orchestra.run.smithi052.stdout: "error", 2024-05-27T07:36:52.179 INFO:teuthology.orchestra.run.smithi052.stdout: "info", 2024-05-27T07:36:52.179 INFO:teuthology.orchestra.run.smithi052.stdout: "warning" 2024-05-27T07:36:52.179 INFO:teuthology.orchestra.run.smithi052.stdout: ], 2024-05-27T07:36:52.179 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-05-27T07:36:52.179 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-05-27T07:36:52.179 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-05-27T07:36:52.179 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-05-27T07:36:52.179 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-05-27T07:36:52.179 INFO:teuthology.orchestra.run.smithi052.stdout: "log_to_file": { 2024-05-27T07:36:52.179 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "log_to_file", 2024-05-27T07:36:52.179 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "bool", 2024-05-27T07:36:52.179 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-05-27T07:36:52.179 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 1, 2024-05-27T07:36:52.180 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "False", 2024-05-27T07:36:52.180 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-05-27T07:36:52.180 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-05-27T07:36:52.180 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-05-27T07:36:52.180 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-05-27T07:36:52.180 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-05-27T07:36:52.180 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-05-27T07:36:52.180 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-05-27T07:36:52.180 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-05-27T07:36:52.180 INFO:teuthology.orchestra.run.smithi052.stdout: "roption1": { 2024-05-27T07:36:52.180 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "roption1", 2024-05-27T07:36:52.180 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "str", 2024-05-27T07:36:52.180 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-05-27T07:36:52.181 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 0, 2024-05-27T07:36:52.181 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "", 2024-05-27T07:36:52.181 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-05-27T07:36:52.181 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-05-27T07:36:52.181 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-05-27T07:36:52.181 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-05-27T07:36:52.181 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-05-27T07:36:52.181 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-05-27T07:36:52.181 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-05-27T07:36:52.181 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-05-27T07:36:52.181 INFO:teuthology.orchestra.run.smithi052.stdout: "roption2": { 2024-05-27T07:36:52.181 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "roption2", 2024-05-27T07:36:52.181 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "str", 2024-05-27T07:36:52.182 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-05-27T07:36:52.182 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 0, 2024-05-27T07:36:52.182 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "xyz", 2024-05-27T07:36:52.182 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-05-27T07:36:52.182 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-05-27T07:36:52.182 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-05-27T07:36:52.182 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-05-27T07:36:52.182 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-05-27T07:36:52.182 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-05-27T07:36:52.182 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-05-27T07:36:52.182 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-05-27T07:36:52.182 INFO:teuthology.orchestra.run.smithi052.stdout: "rwoption1": { 2024-05-27T07:36:52.182 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "rwoption1", 2024-05-27T07:36:52.183 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "str", 2024-05-27T07:36:52.183 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-05-27T07:36:52.183 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 0, 2024-05-27T07:36:52.183 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "", 2024-05-27T07:36:52.183 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-05-27T07:36:52.183 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-05-27T07:36:52.183 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-05-27T07:36:52.183 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-05-27T07:36:52.183 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-05-27T07:36:52.183 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-05-27T07:36:52.183 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-05-27T07:36:52.183 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-05-27T07:36:52.184 INFO:teuthology.orchestra.run.smithi052.stdout: "rwoption2": { 2024-05-27T07:36:52.184 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "rwoption2", 2024-05-27T07:36:52.184 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "int", 2024-05-27T07:36:52.184 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-05-27T07:36:52.184 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 0, 2024-05-27T07:36:52.184 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "", 2024-05-27T07:36:52.184 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-05-27T07:36:52.184 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-05-27T07:36:52.184 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-05-27T07:36:52.184 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-05-27T07:36:52.184 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-05-27T07:36:52.184 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-05-27T07:36:52.184 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-05-27T07:36:52.185 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-05-27T07:36:52.185 INFO:teuthology.orchestra.run.smithi052.stdout: "rwoption3": { 2024-05-27T07:36:52.185 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "rwoption3", 2024-05-27T07:36:52.185 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "float", 2024-05-27T07:36:52.185 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-05-27T07:36:52.185 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 0, 2024-05-27T07:36:52.185 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "", 2024-05-27T07:36:52.185 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-05-27T07:36:52.185 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-05-27T07:36:52.185 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-05-27T07:36:52.185 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-05-27T07:36:52.185 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-05-27T07:36:52.186 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-05-27T07:36:52.186 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-05-27T07:36:52.186 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-05-27T07:36:52.186 INFO:teuthology.orchestra.run.smithi052.stdout: "rwoption4": { 2024-05-27T07:36:52.186 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "rwoption4", 2024-05-27T07:36:52.186 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "str", 2024-05-27T07:36:52.186 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-05-27T07:36:52.186 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 0, 2024-05-27T07:36:52.186 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "", 2024-05-27T07:36:52.186 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-05-27T07:36:52.186 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-05-27T07:36:52.186 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-05-27T07:36:52.187 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-05-27T07:36:52.187 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-05-27T07:36:52.187 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-05-27T07:36:52.187 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-05-27T07:36:52.187 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-05-27T07:36:52.187 INFO:teuthology.orchestra.run.smithi052.stdout: "rwoption5": { 2024-05-27T07:36:52.187 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "rwoption5", 2024-05-27T07:36:52.187 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "bool", 2024-05-27T07:36:52.187 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-05-27T07:36:52.187 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 0, 2024-05-27T07:36:52.187 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "", 2024-05-27T07:36:52.187 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-05-27T07:36:52.188 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-05-27T07:36:52.188 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-05-27T07:36:52.188 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-05-27T07:36:52.188 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-05-27T07:36:52.188 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-05-27T07:36:52.188 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-05-27T07:36:52.188 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-05-27T07:36:52.188 INFO:teuthology.orchestra.run.smithi052.stdout: "rwoption6": { 2024-05-27T07:36:52.188 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "rwoption6", 2024-05-27T07:36:52.188 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "bool", 2024-05-27T07:36:52.188 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-05-27T07:36:52.188 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 0, 2024-05-27T07:36:52.188 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "True", 2024-05-27T07:36:52.189 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-05-27T07:36:52.189 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-05-27T07:36:52.189 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-05-27T07:36:52.189 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-05-27T07:36:52.189 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-05-27T07:36:52.189 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-05-27T07:36:52.189 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-05-27T07:36:52.189 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-05-27T07:36:52.189 INFO:teuthology.orchestra.run.smithi052.stdout: "rwoption7": { 2024-05-27T07:36:52.189 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "rwoption7", 2024-05-27T07:36:52.189 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "int", 2024-05-27T07:36:52.189 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-05-27T07:36:52.190 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 0, 2024-05-27T07:36:52.190 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "", 2024-05-27T07:36:52.190 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "1", 2024-05-27T07:36:52.190 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "42", 2024-05-27T07:36:52.190 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-05-27T07:36:52.190 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-05-27T07:36:52.190 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-05-27T07:36:52.190 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-05-27T07:36:52.190 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-05-27T07:36:52.190 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-05-27T07:36:52.190 INFO:teuthology.orchestra.run.smithi052.stdout: "sqlite3_killpoint": { 2024-05-27T07:36:52.190 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "sqlite3_killpoint", 2024-05-27T07:36:52.190 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "int", 2024-05-27T07:36:52.191 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "dev", 2024-05-27T07:36:52.191 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 1, 2024-05-27T07:36:52.191 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "0", 2024-05-27T07:36:52.191 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-05-27T07:36:52.191 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-05-27T07:36:52.191 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-05-27T07:36:52.191 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-05-27T07:36:52.191 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-05-27T07:36:52.191 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-05-27T07:36:52.191 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-05-27T07:36:52.191 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-05-27T07:36:52.191 INFO:teuthology.orchestra.run.smithi052.stdout: "testkey": { 2024-05-27T07:36:52.191 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "testkey", 2024-05-27T07:36:52.192 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "str", 2024-05-27T07:36:52.192 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-05-27T07:36:52.192 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 0, 2024-05-27T07:36:52.192 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "", 2024-05-27T07:36:52.192 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-05-27T07:36:52.192 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-05-27T07:36:52.192 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-05-27T07:36:52.192 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-05-27T07:36:52.192 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-05-27T07:36:52.192 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-05-27T07:36:52.192 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-05-27T07:36:52.192 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-05-27T07:36:52.192 INFO:teuthology.orchestra.run.smithi052.stdout: "testlkey": { 2024-05-27T07:36:52.193 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "testlkey", 2024-05-27T07:36:52.193 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "str", 2024-05-27T07:36:52.193 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-05-27T07:36:52.193 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 0, 2024-05-27T07:36:52.193 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "", 2024-05-27T07:36:52.193 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-05-27T07:36:52.193 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-05-27T07:36:52.193 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-05-27T07:36:52.193 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-05-27T07:36:52.193 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-05-27T07:36:52.193 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-05-27T07:36:52.193 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-05-27T07:36:52.193 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-05-27T07:36:52.194 INFO:teuthology.orchestra.run.smithi052.stdout: "testnewline": { 2024-05-27T07:36:52.194 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "testnewline", 2024-05-27T07:36:52.194 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "str", 2024-05-27T07:36:52.194 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-05-27T07:36:52.194 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 0, 2024-05-27T07:36:52.194 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "", 2024-05-27T07:36:52.194 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-05-27T07:36:52.194 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-05-27T07:36:52.194 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-05-27T07:36:52.194 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-05-27T07:36:52.194 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-05-27T07:36:52.194 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-05-27T07:36:52.194 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-05-27T07:36:52.195 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-05-27T07:36:52.195 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-05-27T07:36:52.195 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-05-27T07:36:52.195 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-05-27T07:36:52.195 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "smb", 2024-05-27T07:36:52.195 INFO:teuthology.orchestra.run.smithi052.stdout: "can_run": true, 2024-05-27T07:36:52.195 INFO:teuthology.orchestra.run.smithi052.stdout: "error_string": "", 2024-05-27T07:36:52.195 INFO:teuthology.orchestra.run.smithi052.stdout: "module_options": { 2024-05-27T07:36:52.195 INFO:teuthology.orchestra.run.smithi052.stdout: "log_level": { 2024-05-27T07:36:52.195 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "log_level", 2024-05-27T07:36:52.195 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "str", 2024-05-27T07:36:52.195 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-05-27T07:36:52.196 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 1, 2024-05-27T07:36:52.196 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "", 2024-05-27T07:36:52.196 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-05-27T07:36:52.196 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-05-27T07:36:52.196 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [ 2024-05-27T07:36:52.196 INFO:teuthology.orchestra.run.smithi052.stdout: "", 2024-05-27T07:36:52.196 INFO:teuthology.orchestra.run.smithi052.stdout: "critical", 2024-05-27T07:36:52.196 INFO:teuthology.orchestra.run.smithi052.stdout: "debug", 2024-05-27T07:36:52.196 INFO:teuthology.orchestra.run.smithi052.stdout: "error", 2024-05-27T07:36:52.196 INFO:teuthology.orchestra.run.smithi052.stdout: "info", 2024-05-27T07:36:52.196 INFO:teuthology.orchestra.run.smithi052.stdout: "warning" 2024-05-27T07:36:52.196 INFO:teuthology.orchestra.run.smithi052.stdout: ], 2024-05-27T07:36:52.196 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-05-27T07:36:52.196 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-05-27T07:36:52.197 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-05-27T07:36:52.197 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-05-27T07:36:52.197 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-05-27T07:36:52.197 INFO:teuthology.orchestra.run.smithi052.stdout: "log_to_cluster": { 2024-05-27T07:36:52.197 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "log_to_cluster", 2024-05-27T07:36:52.197 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "bool", 2024-05-27T07:36:52.197 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-05-27T07:36:52.197 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 1, 2024-05-27T07:36:52.197 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "False", 2024-05-27T07:36:52.197 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-05-27T07:36:52.197 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-05-27T07:36:52.197 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-05-27T07:36:52.197 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-05-27T07:36:52.198 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-05-27T07:36:52.198 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-05-27T07:36:52.198 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-05-27T07:36:52.198 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-05-27T07:36:52.198 INFO:teuthology.orchestra.run.smithi052.stdout: "log_to_cluster_level": { 2024-05-27T07:36:52.198 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "log_to_cluster_level", 2024-05-27T07:36:52.198 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "str", 2024-05-27T07:36:52.198 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-05-27T07:36:52.198 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 1, 2024-05-27T07:36:52.198 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "info", 2024-05-27T07:36:52.198 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-05-27T07:36:52.198 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-05-27T07:36:52.199 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [ 2024-05-27T07:36:52.199 INFO:teuthology.orchestra.run.smithi052.stdout: "", 2024-05-27T07:36:52.199 INFO:teuthology.orchestra.run.smithi052.stdout: "critical", 2024-05-27T07:36:52.199 INFO:teuthology.orchestra.run.smithi052.stdout: "debug", 2024-05-27T07:36:52.199 INFO:teuthology.orchestra.run.smithi052.stdout: "error", 2024-05-27T07:36:52.199 INFO:teuthology.orchestra.run.smithi052.stdout: "info", 2024-05-27T07:36:52.199 INFO:teuthology.orchestra.run.smithi052.stdout: "warning" 2024-05-27T07:36:52.199 INFO:teuthology.orchestra.run.smithi052.stdout: ], 2024-05-27T07:36:52.199 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-05-27T07:36:52.199 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-05-27T07:36:52.199 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-05-27T07:36:52.199 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-05-27T07:36:52.200 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-05-27T07:36:52.200 INFO:teuthology.orchestra.run.smithi052.stdout: "log_to_file": { 2024-05-27T07:36:52.200 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "log_to_file", 2024-05-27T07:36:52.200 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "bool", 2024-05-27T07:36:52.200 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-05-27T07:36:52.200 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 1, 2024-05-27T07:36:52.200 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "False", 2024-05-27T07:36:52.200 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-05-27T07:36:52.200 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-05-27T07:36:52.200 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-05-27T07:36:52.200 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-05-27T07:36:52.200 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-05-27T07:36:52.200 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-05-27T07:36:52.201 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-05-27T07:36:52.201 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-05-27T07:36:52.201 INFO:teuthology.orchestra.run.smithi052.stdout: "sqlite3_killpoint": { 2024-05-27T07:36:52.201 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "sqlite3_killpoint", 2024-05-27T07:36:52.201 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "int", 2024-05-27T07:36:52.201 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "dev", 2024-05-27T07:36:52.201 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 1, 2024-05-27T07:36:52.201 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "0", 2024-05-27T07:36:52.201 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-05-27T07:36:52.201 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-05-27T07:36:52.201 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-05-27T07:36:52.201 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-05-27T07:36:52.201 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-05-27T07:36:52.202 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-05-27T07:36:52.202 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-05-27T07:36:52.202 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-05-27T07:36:52.202 INFO:teuthology.orchestra.run.smithi052.stdout: "update_orchestration": { 2024-05-27T07:36:52.202 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "update_orchestration", 2024-05-27T07:36:52.202 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "bool", 2024-05-27T07:36:52.202 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-05-27T07:36:52.202 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 0, 2024-05-27T07:36:52.202 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "True", 2024-05-27T07:36:52.202 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-05-27T07:36:52.202 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-05-27T07:36:52.202 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-05-27T07:36:52.203 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "automatically update orchestration when smb resources are changed", 2024-05-27T07:36:52.203 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-05-27T07:36:52.203 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-05-27T07:36:52.203 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-05-27T07:36:52.203 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-05-27T07:36:52.203 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-05-27T07:36:52.203 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-05-27T07:36:52.203 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-05-27T07:36:52.203 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "snap_schedule", 2024-05-27T07:36:52.203 INFO:teuthology.orchestra.run.smithi052.stdout: "can_run": true, 2024-05-27T07:36:52.203 INFO:teuthology.orchestra.run.smithi052.stdout: "error_string": "", 2024-05-27T07:36:52.203 INFO:teuthology.orchestra.run.smithi052.stdout: "module_options": { 2024-05-27T07:36:52.203 INFO:teuthology.orchestra.run.smithi052.stdout: "allow_m_granularity": { 2024-05-27T07:36:52.204 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "allow_m_granularity", 2024-05-27T07:36:52.204 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "bool", 2024-05-27T07:36:52.204 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-05-27T07:36:52.204 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 1, 2024-05-27T07:36:52.204 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "False", 2024-05-27T07:36:52.204 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-05-27T07:36:52.204 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-05-27T07:36:52.204 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-05-27T07:36:52.204 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "allow minute scheduled snapshots", 2024-05-27T07:36:52.204 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-05-27T07:36:52.204 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-05-27T07:36:52.204 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-05-27T07:36:52.204 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-05-27T07:36:52.205 INFO:teuthology.orchestra.run.smithi052.stdout: "dump_on_update": { 2024-05-27T07:36:52.205 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "dump_on_update", 2024-05-27T07:36:52.205 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "bool", 2024-05-27T07:36:52.205 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-05-27T07:36:52.205 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 1, 2024-05-27T07:36:52.205 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "False", 2024-05-27T07:36:52.205 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-05-27T07:36:52.205 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-05-27T07:36:52.205 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-05-27T07:36:52.205 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "dump database to debug log on update", 2024-05-27T07:36:52.205 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-05-27T07:36:52.205 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-05-27T07:36:52.205 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-05-27T07:36:52.206 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-05-27T07:36:52.206 INFO:teuthology.orchestra.run.smithi052.stdout: "log_level": { 2024-05-27T07:36:52.206 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "log_level", 2024-05-27T07:36:52.206 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "str", 2024-05-27T07:36:52.206 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-05-27T07:36:52.206 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 1, 2024-05-27T07:36:52.206 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "", 2024-05-27T07:36:52.206 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-05-27T07:36:52.206 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-05-27T07:36:52.206 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [ 2024-05-27T07:36:52.206 INFO:teuthology.orchestra.run.smithi052.stdout: "", 2024-05-27T07:36:52.206 INFO:teuthology.orchestra.run.smithi052.stdout: "critical", 2024-05-27T07:36:52.207 INFO:teuthology.orchestra.run.smithi052.stdout: "debug", 2024-05-27T07:36:52.207 INFO:teuthology.orchestra.run.smithi052.stdout: "error", 2024-05-27T07:36:52.207 INFO:teuthology.orchestra.run.smithi052.stdout: "info", 2024-05-27T07:36:52.207 INFO:teuthology.orchestra.run.smithi052.stdout: "warning" 2024-05-27T07:36:52.207 INFO:teuthology.orchestra.run.smithi052.stdout: ], 2024-05-27T07:36:52.207 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-05-27T07:36:52.207 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-05-27T07:36:52.207 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-05-27T07:36:52.207 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-05-27T07:36:52.207 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-05-27T07:36:52.207 INFO:teuthology.orchestra.run.smithi052.stdout: "log_to_cluster": { 2024-05-27T07:36:52.207 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "log_to_cluster", 2024-05-27T07:36:52.207 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "bool", 2024-05-27T07:36:52.208 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-05-27T07:36:52.208 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 1, 2024-05-27T07:36:52.208 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "False", 2024-05-27T07:36:52.208 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-05-27T07:36:52.208 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-05-27T07:36:52.208 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-05-27T07:36:52.208 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-05-27T07:36:52.208 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-05-27T07:36:52.208 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-05-27T07:36:52.208 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-05-27T07:36:52.208 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-05-27T07:36:52.208 INFO:teuthology.orchestra.run.smithi052.stdout: "log_to_cluster_level": { 2024-05-27T07:36:52.208 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "log_to_cluster_level", 2024-05-27T07:36:52.208 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "str", 2024-05-27T07:36:52.209 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-05-27T07:36:52.209 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 1, 2024-05-27T07:36:52.209 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "info", 2024-05-27T07:36:52.209 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-05-27T07:36:52.209 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-05-27T07:36:52.209 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [ 2024-05-27T07:36:52.209 INFO:teuthology.orchestra.run.smithi052.stdout: "", 2024-05-27T07:36:52.209 INFO:teuthology.orchestra.run.smithi052.stdout: "critical", 2024-05-27T07:36:52.209 INFO:teuthology.orchestra.run.smithi052.stdout: "debug", 2024-05-27T07:36:52.209 INFO:teuthology.orchestra.run.smithi052.stdout: "error", 2024-05-27T07:36:52.209 INFO:teuthology.orchestra.run.smithi052.stdout: "info", 2024-05-27T07:36:52.209 INFO:teuthology.orchestra.run.smithi052.stdout: "warning" 2024-05-27T07:36:52.209 INFO:teuthology.orchestra.run.smithi052.stdout: ], 2024-05-27T07:36:52.210 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-05-27T07:36:52.210 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-05-27T07:36:52.210 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-05-27T07:36:52.210 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-05-27T07:36:52.210 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-05-27T07:36:52.210 INFO:teuthology.orchestra.run.smithi052.stdout: "log_to_file": { 2024-05-27T07:36:52.210 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "log_to_file", 2024-05-27T07:36:52.210 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "bool", 2024-05-27T07:36:52.210 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-05-27T07:36:52.210 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 1, 2024-05-27T07:36:52.210 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "False", 2024-05-27T07:36:52.210 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-05-27T07:36:52.211 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-05-27T07:36:52.211 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-05-27T07:36:52.211 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-05-27T07:36:52.211 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-05-27T07:36:52.211 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-05-27T07:36:52.211 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-05-27T07:36:52.211 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-05-27T07:36:52.211 INFO:teuthology.orchestra.run.smithi052.stdout: "sqlite3_killpoint": { 2024-05-27T07:36:52.211 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "sqlite3_killpoint", 2024-05-27T07:36:52.211 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "int", 2024-05-27T07:36:52.211 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "dev", 2024-05-27T07:36:52.211 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 1, 2024-05-27T07:36:52.211 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "0", 2024-05-27T07:36:52.212 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-05-27T07:36:52.212 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-05-27T07:36:52.212 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-05-27T07:36:52.212 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-05-27T07:36:52.212 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-05-27T07:36:52.212 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-05-27T07:36:52.212 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-05-27T07:36:52.212 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-05-27T07:36:52.212 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-05-27T07:36:52.212 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-05-27T07:36:52.212 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-05-27T07:36:52.212 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "stats", 2024-05-27T07:36:52.212 INFO:teuthology.orchestra.run.smithi052.stdout: "can_run": true, 2024-05-27T07:36:52.213 INFO:teuthology.orchestra.run.smithi052.stdout: "error_string": "", 2024-05-27T07:36:52.213 INFO:teuthology.orchestra.run.smithi052.stdout: "module_options": { 2024-05-27T07:36:52.213 INFO:teuthology.orchestra.run.smithi052.stdout: "log_level": { 2024-05-27T07:36:52.213 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "log_level", 2024-05-27T07:36:52.213 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "str", 2024-05-27T07:36:52.213 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-05-27T07:36:52.213 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 1, 2024-05-27T07:36:52.213 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "", 2024-05-27T07:36:52.213 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-05-27T07:36:52.213 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-05-27T07:36:52.213 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [ 2024-05-27T07:36:52.213 INFO:teuthology.orchestra.run.smithi052.stdout: "", 2024-05-27T07:36:52.214 INFO:teuthology.orchestra.run.smithi052.stdout: "critical", 2024-05-27T07:36:52.214 INFO:teuthology.orchestra.run.smithi052.stdout: "debug", 2024-05-27T07:36:52.214 INFO:teuthology.orchestra.run.smithi052.stdout: "error", 2024-05-27T07:36:52.214 INFO:teuthology.orchestra.run.smithi052.stdout: "info", 2024-05-27T07:36:52.214 INFO:teuthology.orchestra.run.smithi052.stdout: "warning" 2024-05-27T07:36:52.214 INFO:teuthology.orchestra.run.smithi052.stdout: ], 2024-05-27T07:36:52.214 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-05-27T07:36:52.214 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-05-27T07:36:52.214 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-05-27T07:36:52.214 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-05-27T07:36:52.214 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-05-27T07:36:52.214 INFO:teuthology.orchestra.run.smithi052.stdout: "log_to_cluster": { 2024-05-27T07:36:52.214 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "log_to_cluster", 2024-05-27T07:36:52.215 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "bool", 2024-05-27T07:36:52.215 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-05-27T07:36:52.215 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 1, 2024-05-27T07:36:52.215 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "False", 2024-05-27T07:36:52.215 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-05-27T07:36:52.215 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-05-27T07:36:52.215 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-05-27T07:36:52.215 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-05-27T07:36:52.215 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-05-27T07:36:52.215 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-05-27T07:36:52.215 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-05-27T07:36:52.215 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-05-27T07:36:52.216 INFO:teuthology.orchestra.run.smithi052.stdout: "log_to_cluster_level": { 2024-05-27T07:36:52.216 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "log_to_cluster_level", 2024-05-27T07:36:52.216 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "str", 2024-05-27T07:36:52.216 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-05-27T07:36:52.216 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 1, 2024-05-27T07:36:52.216 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "info", 2024-05-27T07:36:52.216 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-05-27T07:36:52.216 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-05-27T07:36:52.216 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [ 2024-05-27T07:36:52.216 INFO:teuthology.orchestra.run.smithi052.stdout: "", 2024-05-27T07:36:52.216 INFO:teuthology.orchestra.run.smithi052.stdout: "critical", 2024-05-27T07:36:52.216 INFO:teuthology.orchestra.run.smithi052.stdout: "debug", 2024-05-27T07:36:52.216 INFO:teuthology.orchestra.run.smithi052.stdout: "error", 2024-05-27T07:36:52.216 INFO:teuthology.orchestra.run.smithi052.stdout: "info", 2024-05-27T07:36:52.217 INFO:teuthology.orchestra.run.smithi052.stdout: "warning" 2024-05-27T07:36:52.217 INFO:teuthology.orchestra.run.smithi052.stdout: ], 2024-05-27T07:36:52.217 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-05-27T07:36:52.217 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-05-27T07:36:52.217 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-05-27T07:36:52.217 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-05-27T07:36:52.217 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-05-27T07:36:52.217 INFO:teuthology.orchestra.run.smithi052.stdout: "log_to_file": { 2024-05-27T07:36:52.217 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "log_to_file", 2024-05-27T07:36:52.217 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "bool", 2024-05-27T07:36:52.217 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-05-27T07:36:52.217 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 1, 2024-05-27T07:36:52.218 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "False", 2024-05-27T07:36:52.218 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-05-27T07:36:52.218 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-05-27T07:36:52.218 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-05-27T07:36:52.218 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-05-27T07:36:52.218 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-05-27T07:36:52.218 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-05-27T07:36:52.218 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-05-27T07:36:52.218 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-05-27T07:36:52.218 INFO:teuthology.orchestra.run.smithi052.stdout: "sqlite3_killpoint": { 2024-05-27T07:36:52.218 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "sqlite3_killpoint", 2024-05-27T07:36:52.218 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "int", 2024-05-27T07:36:52.219 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "dev", 2024-05-27T07:36:52.219 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 1, 2024-05-27T07:36:52.219 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "0", 2024-05-27T07:36:52.219 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-05-27T07:36:52.219 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-05-27T07:36:52.219 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-05-27T07:36:52.219 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-05-27T07:36:52.219 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-05-27T07:36:52.219 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-05-27T07:36:52.219 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-05-27T07:36:52.219 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-05-27T07:36:52.219 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-05-27T07:36:52.220 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-05-27T07:36:52.220 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-05-27T07:36:52.220 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "status", 2024-05-27T07:36:52.220 INFO:teuthology.orchestra.run.smithi052.stdout: "can_run": true, 2024-05-27T07:36:52.220 INFO:teuthology.orchestra.run.smithi052.stdout: "error_string": "", 2024-05-27T07:36:52.220 INFO:teuthology.orchestra.run.smithi052.stdout: "module_options": { 2024-05-27T07:36:52.220 INFO:teuthology.orchestra.run.smithi052.stdout: "log_level": { 2024-05-27T07:36:52.220 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "log_level", 2024-05-27T07:36:52.220 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "str", 2024-05-27T07:36:52.220 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-05-27T07:36:52.220 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 1, 2024-05-27T07:36:52.220 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "", 2024-05-27T07:36:52.220 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-05-27T07:36:52.221 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-05-27T07:36:52.221 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [ 2024-05-27T07:36:52.221 INFO:teuthology.orchestra.run.smithi052.stdout: "", 2024-05-27T07:36:52.221 INFO:teuthology.orchestra.run.smithi052.stdout: "critical", 2024-05-27T07:36:52.221 INFO:teuthology.orchestra.run.smithi052.stdout: "debug", 2024-05-27T07:36:52.221 INFO:teuthology.orchestra.run.smithi052.stdout: "error", 2024-05-27T07:36:52.221 INFO:teuthology.orchestra.run.smithi052.stdout: "info", 2024-05-27T07:36:52.221 INFO:teuthology.orchestra.run.smithi052.stdout: "warning" 2024-05-27T07:36:52.221 INFO:teuthology.orchestra.run.smithi052.stdout: ], 2024-05-27T07:36:52.221 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-05-27T07:36:52.221 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-05-27T07:36:52.221 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-05-27T07:36:52.221 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-05-27T07:36:52.222 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-05-27T07:36:52.222 INFO:teuthology.orchestra.run.smithi052.stdout: "log_to_cluster": { 2024-05-27T07:36:52.222 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "log_to_cluster", 2024-05-27T07:36:52.222 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "bool", 2024-05-27T07:36:52.222 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-05-27T07:36:52.222 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 1, 2024-05-27T07:36:52.222 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "False", 2024-05-27T07:36:52.222 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-05-27T07:36:52.222 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-05-27T07:36:52.222 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-05-27T07:36:52.222 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-05-27T07:36:52.222 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-05-27T07:36:52.223 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-05-27T07:36:52.223 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-05-27T07:36:52.223 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-05-27T07:36:52.223 INFO:teuthology.orchestra.run.smithi052.stdout: "log_to_cluster_level": { 2024-05-27T07:36:52.223 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "log_to_cluster_level", 2024-05-27T07:36:52.223 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "str", 2024-05-27T07:36:52.223 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-05-27T07:36:52.223 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 1, 2024-05-27T07:36:52.223 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "info", 2024-05-27T07:36:52.223 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-05-27T07:36:52.223 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-05-27T07:36:52.223 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [ 2024-05-27T07:36:52.223 INFO:teuthology.orchestra.run.smithi052.stdout: "", 2024-05-27T07:36:52.224 INFO:teuthology.orchestra.run.smithi052.stdout: "critical", 2024-05-27T07:36:52.224 INFO:teuthology.orchestra.run.smithi052.stdout: "debug", 2024-05-27T07:36:52.224 INFO:teuthology.orchestra.run.smithi052.stdout: "error", 2024-05-27T07:36:52.224 INFO:teuthology.orchestra.run.smithi052.stdout: "info", 2024-05-27T07:36:52.224 INFO:teuthology.orchestra.run.smithi052.stdout: "warning" 2024-05-27T07:36:52.224 INFO:teuthology.orchestra.run.smithi052.stdout: ], 2024-05-27T07:36:52.224 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-05-27T07:36:52.224 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-05-27T07:36:52.224 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-05-27T07:36:52.224 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-05-27T07:36:52.224 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-05-27T07:36:52.224 INFO:teuthology.orchestra.run.smithi052.stdout: "log_to_file": { 2024-05-27T07:36:52.224 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "log_to_file", 2024-05-27T07:36:52.225 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "bool", 2024-05-27T07:36:52.225 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-05-27T07:36:52.225 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 1, 2024-05-27T07:36:52.225 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "False", 2024-05-27T07:36:52.225 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-05-27T07:36:52.225 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-05-27T07:36:52.225 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-05-27T07:36:52.225 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-05-27T07:36:52.225 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-05-27T07:36:52.225 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-05-27T07:36:52.225 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-05-27T07:36:52.225 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-05-27T07:36:52.226 INFO:teuthology.orchestra.run.smithi052.stdout: "sqlite3_killpoint": { 2024-05-27T07:36:52.226 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "sqlite3_killpoint", 2024-05-27T07:36:52.226 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "int", 2024-05-27T07:36:52.226 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "dev", 2024-05-27T07:36:52.226 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 1, 2024-05-27T07:36:52.226 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "0", 2024-05-27T07:36:52.226 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-05-27T07:36:52.226 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-05-27T07:36:52.226 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-05-27T07:36:52.226 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-05-27T07:36:52.226 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-05-27T07:36:52.226 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-05-27T07:36:52.227 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-05-27T07:36:52.227 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-05-27T07:36:52.227 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-05-27T07:36:52.227 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-05-27T07:36:52.227 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-05-27T07:36:52.227 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "telegraf", 2024-05-27T07:36:52.227 INFO:teuthology.orchestra.run.smithi052.stdout: "can_run": true, 2024-05-27T07:36:52.227 INFO:teuthology.orchestra.run.smithi052.stdout: "error_string": "", 2024-05-27T07:36:52.227 INFO:teuthology.orchestra.run.smithi052.stdout: "module_options": { 2024-05-27T07:36:52.227 INFO:teuthology.orchestra.run.smithi052.stdout: "address": { 2024-05-27T07:36:52.227 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "address", 2024-05-27T07:36:52.227 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "str", 2024-05-27T07:36:52.227 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-05-27T07:36:52.228 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 0, 2024-05-27T07:36:52.228 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "unixgram:///tmp/telegraf.sock", 2024-05-27T07:36:52.228 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-05-27T07:36:52.228 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-05-27T07:36:52.228 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-05-27T07:36:52.228 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-05-27T07:36:52.228 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-05-27T07:36:52.228 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-05-27T07:36:52.228 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-05-27T07:36:52.228 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-05-27T07:36:52.228 INFO:teuthology.orchestra.run.smithi052.stdout: "interval": { 2024-05-27T07:36:52.228 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "interval", 2024-05-27T07:36:52.228 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "secs", 2024-05-27T07:36:52.229 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-05-27T07:36:52.229 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 0, 2024-05-27T07:36:52.229 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "15", 2024-05-27T07:36:52.229 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-05-27T07:36:52.229 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-05-27T07:36:52.229 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-05-27T07:36:52.229 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-05-27T07:36:52.229 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-05-27T07:36:52.229 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-05-27T07:36:52.229 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-05-27T07:36:52.229 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-05-27T07:36:52.229 INFO:teuthology.orchestra.run.smithi052.stdout: "log_level": { 2024-05-27T07:36:52.230 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "log_level", 2024-05-27T07:36:52.230 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "str", 2024-05-27T07:36:52.230 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-05-27T07:36:52.230 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 1, 2024-05-27T07:36:52.230 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "", 2024-05-27T07:36:52.230 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-05-27T07:36:52.230 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-05-27T07:36:52.230 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [ 2024-05-27T07:36:52.230 INFO:teuthology.orchestra.run.smithi052.stdout: "", 2024-05-27T07:36:52.230 INFO:teuthology.orchestra.run.smithi052.stdout: "critical", 2024-05-27T07:36:52.230 INFO:teuthology.orchestra.run.smithi052.stdout: "debug", 2024-05-27T07:36:52.230 INFO:teuthology.orchestra.run.smithi052.stdout: "error", 2024-05-27T07:36:52.231 INFO:teuthology.orchestra.run.smithi052.stdout: "info", 2024-05-27T07:36:52.231 INFO:teuthology.orchestra.run.smithi052.stdout: "warning" 2024-05-27T07:36:52.231 INFO:teuthology.orchestra.run.smithi052.stdout: ], 2024-05-27T07:36:52.231 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-05-27T07:36:52.231 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-05-27T07:36:52.231 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-05-27T07:36:52.231 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-05-27T07:36:52.231 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-05-27T07:36:52.231 INFO:teuthology.orchestra.run.smithi052.stdout: "log_to_cluster": { 2024-05-27T07:36:52.231 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "log_to_cluster", 2024-05-27T07:36:52.231 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "bool", 2024-05-27T07:36:52.231 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-05-27T07:36:52.232 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 1, 2024-05-27T07:36:52.232 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "False", 2024-05-27T07:36:52.232 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-05-27T07:36:52.232 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-05-27T07:36:52.232 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-05-27T07:36:52.232 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-05-27T07:36:52.232 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-05-27T07:36:52.232 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-05-27T07:36:52.232 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-05-27T07:36:52.232 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-05-27T07:36:52.232 INFO:teuthology.orchestra.run.smithi052.stdout: "log_to_cluster_level": { 2024-05-27T07:36:52.232 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "log_to_cluster_level", 2024-05-27T07:36:52.232 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "str", 2024-05-27T07:36:52.233 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-05-27T07:36:52.233 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 1, 2024-05-27T07:36:52.233 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "info", 2024-05-27T07:36:52.233 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-05-27T07:36:52.233 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-05-27T07:36:52.233 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [ 2024-05-27T07:36:52.233 INFO:teuthology.orchestra.run.smithi052.stdout: "", 2024-05-27T07:36:52.233 INFO:teuthology.orchestra.run.smithi052.stdout: "critical", 2024-05-27T07:36:52.233 INFO:teuthology.orchestra.run.smithi052.stdout: "debug", 2024-05-27T07:36:52.233 INFO:teuthology.orchestra.run.smithi052.stdout: "error", 2024-05-27T07:36:52.233 INFO:teuthology.orchestra.run.smithi052.stdout: "info", 2024-05-27T07:36:52.233 INFO:teuthology.orchestra.run.smithi052.stdout: "warning" 2024-05-27T07:36:52.233 INFO:teuthology.orchestra.run.smithi052.stdout: ], 2024-05-27T07:36:52.234 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-05-27T07:36:52.234 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-05-27T07:36:52.234 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-05-27T07:36:52.234 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-05-27T07:36:52.234 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-05-27T07:36:52.234 INFO:teuthology.orchestra.run.smithi052.stdout: "log_to_file": { 2024-05-27T07:36:52.234 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "log_to_file", 2024-05-27T07:36:52.234 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "bool", 2024-05-27T07:36:52.234 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-05-27T07:36:52.234 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 1, 2024-05-27T07:36:52.234 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "False", 2024-05-27T07:36:52.234 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-05-27T07:36:52.235 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-05-27T07:36:52.235 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-05-27T07:36:52.235 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-05-27T07:36:52.235 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-05-27T07:36:52.235 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-05-27T07:36:52.235 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-05-27T07:36:52.235 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-05-27T07:36:52.235 INFO:teuthology.orchestra.run.smithi052.stdout: "sqlite3_killpoint": { 2024-05-27T07:36:52.235 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "sqlite3_killpoint", 2024-05-27T07:36:52.235 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "int", 2024-05-27T07:36:52.235 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "dev", 2024-05-27T07:36:52.235 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 1, 2024-05-27T07:36:52.235 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "0", 2024-05-27T07:36:52.235 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-05-27T07:36:52.236 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-05-27T07:36:52.236 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-05-27T07:36:52.236 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-05-27T07:36:52.236 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-05-27T07:36:52.236 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-05-27T07:36:52.236 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-05-27T07:36:52.236 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-05-27T07:36:52.236 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-05-27T07:36:52.236 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-05-27T07:36:52.236 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-05-27T07:36:52.236 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "telemetry", 2024-05-27T07:36:52.236 INFO:teuthology.orchestra.run.smithi052.stdout: "can_run": true, 2024-05-27T07:36:52.236 INFO:teuthology.orchestra.run.smithi052.stdout: "error_string": "", 2024-05-27T07:36:52.237 INFO:teuthology.orchestra.run.smithi052.stdout: "module_options": { 2024-05-27T07:36:52.237 INFO:teuthology.orchestra.run.smithi052.stdout: "channel_basic": { 2024-05-27T07:36:52.237 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "channel_basic", 2024-05-27T07:36:52.237 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "bool", 2024-05-27T07:36:52.237 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-05-27T07:36:52.237 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 0, 2024-05-27T07:36:52.237 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "True", 2024-05-27T07:36:52.237 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-05-27T07:36:52.237 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-05-27T07:36:52.237 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-05-27T07:36:52.237 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "Share basic cluster information (size, version)", 2024-05-27T07:36:52.237 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-05-27T07:36:52.237 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-05-27T07:36:52.238 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-05-27T07:36:52.238 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-05-27T07:36:52.238 INFO:teuthology.orchestra.run.smithi052.stdout: "channel_crash": { 2024-05-27T07:36:52.238 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "channel_crash", 2024-05-27T07:36:52.238 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "bool", 2024-05-27T07:36:52.238 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-05-27T07:36:52.238 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 0, 2024-05-27T07:36:52.238 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "True", 2024-05-27T07:36:52.238 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-05-27T07:36:52.238 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-05-27T07:36:52.238 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-05-27T07:36:52.238 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "Share metadata about Ceph daemon crashes (version, stack straces, etc)", 2024-05-27T07:36:52.239 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-05-27T07:36:52.239 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-05-27T07:36:52.239 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-05-27T07:36:52.239 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-05-27T07:36:52.239 INFO:teuthology.orchestra.run.smithi052.stdout: "channel_device": { 2024-05-27T07:36:52.239 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "channel_device", 2024-05-27T07:36:52.239 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "bool", 2024-05-27T07:36:52.239 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-05-27T07:36:52.239 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 0, 2024-05-27T07:36:52.239 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "True", 2024-05-27T07:36:52.239 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-05-27T07:36:52.239 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-05-27T07:36:52.239 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-05-27T07:36:52.239 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)", 2024-05-27T07:36:52.240 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-05-27T07:36:52.240 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-05-27T07:36:52.240 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-05-27T07:36:52.240 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-05-27T07:36:52.240 INFO:teuthology.orchestra.run.smithi052.stdout: "channel_ident": { 2024-05-27T07:36:52.240 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "channel_ident", 2024-05-27T07:36:52.240 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "bool", 2024-05-27T07:36:52.240 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-05-27T07:36:52.240 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 0, 2024-05-27T07:36:52.240 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "False", 2024-05-27T07:36:52.240 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-05-27T07:36:52.240 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-05-27T07:36:52.240 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-05-27T07:36:52.241 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "Share a user-provided description and/or contact email for the cluster", 2024-05-27T07:36:52.241 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-05-27T07:36:52.241 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-05-27T07:36:52.241 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-05-27T07:36:52.241 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-05-27T07:36:52.241 INFO:teuthology.orchestra.run.smithi052.stdout: "channel_perf": { 2024-05-27T07:36:52.241 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "channel_perf", 2024-05-27T07:36:52.241 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "bool", 2024-05-27T07:36:52.241 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-05-27T07:36:52.241 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 0, 2024-05-27T07:36:52.241 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "False", 2024-05-27T07:36:52.241 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-05-27T07:36:52.241 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-05-27T07:36:52.241 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-05-27T07:36:52.242 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "Share various performance metrics of a cluster", 2024-05-27T07:36:52.242 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-05-27T07:36:52.242 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-05-27T07:36:52.242 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-05-27T07:36:52.242 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-05-27T07:36:52.242 INFO:teuthology.orchestra.run.smithi052.stdout: "contact": { 2024-05-27T07:36:52.242 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "contact", 2024-05-27T07:36:52.242 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "str", 2024-05-27T07:36:52.242 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-05-27T07:36:52.242 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 0, 2024-05-27T07:36:52.242 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "", 2024-05-27T07:36:52.242 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-05-27T07:36:52.243 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-05-27T07:36:52.243 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-05-27T07:36:52.243 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-05-27T07:36:52.243 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-05-27T07:36:52.243 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-05-27T07:36:52.243 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-05-27T07:36:52.243 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-05-27T07:36:52.243 INFO:teuthology.orchestra.run.smithi052.stdout: "description": { 2024-05-27T07:36:52.243 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "description", 2024-05-27T07:36:52.243 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "str", 2024-05-27T07:36:52.243 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-05-27T07:36:52.243 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 0, 2024-05-27T07:36:52.243 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "", 2024-05-27T07:36:52.244 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-05-27T07:36:52.244 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-05-27T07:36:52.244 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-05-27T07:36:52.244 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-05-27T07:36:52.244 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-05-27T07:36:52.244 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-05-27T07:36:52.244 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-05-27T07:36:52.244 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-05-27T07:36:52.244 INFO:teuthology.orchestra.run.smithi052.stdout: "device_url": { 2024-05-27T07:36:52.244 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "device_url", 2024-05-27T07:36:52.244 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "str", 2024-05-27T07:36:52.244 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-05-27T07:36:52.245 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 0, 2024-05-27T07:36:52.245 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "https://telemetry.ceph.com/device", 2024-05-27T07:36:52.245 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-05-27T07:36:52.245 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-05-27T07:36:52.245 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-05-27T07:36:52.245 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-05-27T07:36:52.245 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-05-27T07:36:52.245 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-05-27T07:36:52.245 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-05-27T07:36:52.245 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-05-27T07:36:52.245 INFO:teuthology.orchestra.run.smithi052.stdout: "enabled": { 2024-05-27T07:36:52.245 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "enabled", 2024-05-27T07:36:52.245 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "bool", 2024-05-27T07:36:52.246 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-05-27T07:36:52.246 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 0, 2024-05-27T07:36:52.246 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "False", 2024-05-27T07:36:52.246 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-05-27T07:36:52.246 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-05-27T07:36:52.246 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-05-27T07:36:52.246 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-05-27T07:36:52.246 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-05-27T07:36:52.246 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-05-27T07:36:52.246 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-05-27T07:36:52.246 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-05-27T07:36:52.246 INFO:teuthology.orchestra.run.smithi052.stdout: "interval": { 2024-05-27T07:36:52.247 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "interval", 2024-05-27T07:36:52.247 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "int", 2024-05-27T07:36:52.247 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-05-27T07:36:52.247 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 0, 2024-05-27T07:36:52.247 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "24", 2024-05-27T07:36:52.247 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "8", 2024-05-27T07:36:52.247 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-05-27T07:36:52.247 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-05-27T07:36:52.247 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-05-27T07:36:52.247 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-05-27T07:36:52.247 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-05-27T07:36:52.247 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-05-27T07:36:52.247 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-05-27T07:36:52.248 INFO:teuthology.orchestra.run.smithi052.stdout: "last_opt_revision": { 2024-05-27T07:36:52.248 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "last_opt_revision", 2024-05-27T07:36:52.248 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "int", 2024-05-27T07:36:52.248 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-05-27T07:36:52.248 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 0, 2024-05-27T07:36:52.248 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "1", 2024-05-27T07:36:52.248 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-05-27T07:36:52.248 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-05-27T07:36:52.248 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-05-27T07:36:52.248 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-05-27T07:36:52.248 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-05-27T07:36:52.248 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-05-27T07:36:52.248 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-05-27T07:36:52.248 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-05-27T07:36:52.249 INFO:teuthology.orchestra.run.smithi052.stdout: "leaderboard": { 2024-05-27T07:36:52.249 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "leaderboard", 2024-05-27T07:36:52.249 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "bool", 2024-05-27T07:36:52.249 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-05-27T07:36:52.249 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 0, 2024-05-27T07:36:52.249 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "False", 2024-05-27T07:36:52.249 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-05-27T07:36:52.249 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-05-27T07:36:52.249 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-05-27T07:36:52.249 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-05-27T07:36:52.249 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-05-27T07:36:52.249 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-05-27T07:36:52.250 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-05-27T07:36:52.250 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-05-27T07:36:52.250 INFO:teuthology.orchestra.run.smithi052.stdout: "leaderboard_description": { 2024-05-27T07:36:52.250 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "leaderboard_description", 2024-05-27T07:36:52.250 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "str", 2024-05-27T07:36:52.250 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-05-27T07:36:52.250 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 0, 2024-05-27T07:36:52.250 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "", 2024-05-27T07:36:52.250 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-05-27T07:36:52.250 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-05-27T07:36:52.250 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-05-27T07:36:52.250 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-05-27T07:36:52.250 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-05-27T07:36:52.251 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-05-27T07:36:52.251 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-05-27T07:36:52.251 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-05-27T07:36:52.251 INFO:teuthology.orchestra.run.smithi052.stdout: "log_level": { 2024-05-27T07:36:52.251 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "log_level", 2024-05-27T07:36:52.251 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "str", 2024-05-27T07:36:52.251 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-05-27T07:36:52.251 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 1, 2024-05-27T07:36:52.251 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "", 2024-05-27T07:36:52.251 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-05-27T07:36:52.251 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-05-27T07:36:52.251 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [ 2024-05-27T07:36:52.251 INFO:teuthology.orchestra.run.smithi052.stdout: "", 2024-05-27T07:36:52.252 INFO:teuthology.orchestra.run.smithi052.stdout: "critical", 2024-05-27T07:36:52.252 INFO:teuthology.orchestra.run.smithi052.stdout: "debug", 2024-05-27T07:36:52.252 INFO:teuthology.orchestra.run.smithi052.stdout: "error", 2024-05-27T07:36:52.252 INFO:teuthology.orchestra.run.smithi052.stdout: "info", 2024-05-27T07:36:52.252 INFO:teuthology.orchestra.run.smithi052.stdout: "warning" 2024-05-27T07:36:52.252 INFO:teuthology.orchestra.run.smithi052.stdout: ], 2024-05-27T07:36:52.252 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-05-27T07:36:52.252 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-05-27T07:36:52.252 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-05-27T07:36:52.252 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-05-27T07:36:52.252 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-05-27T07:36:52.252 INFO:teuthology.orchestra.run.smithi052.stdout: "log_to_cluster": { 2024-05-27T07:36:52.252 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "log_to_cluster", 2024-05-27T07:36:52.253 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "bool", 2024-05-27T07:36:52.253 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-05-27T07:36:52.253 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 1, 2024-05-27T07:36:52.253 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "False", 2024-05-27T07:36:52.253 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-05-27T07:36:52.253 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-05-27T07:36:52.253 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-05-27T07:36:52.253 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-05-27T07:36:52.253 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-05-27T07:36:52.253 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-05-27T07:36:52.253 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-05-27T07:36:52.253 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-05-27T07:36:52.253 INFO:teuthology.orchestra.run.smithi052.stdout: "log_to_cluster_level": { 2024-05-27T07:36:52.253 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "log_to_cluster_level", 2024-05-27T07:36:52.254 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "str", 2024-05-27T07:36:52.254 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-05-27T07:36:52.254 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 1, 2024-05-27T07:36:52.254 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "info", 2024-05-27T07:36:52.254 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-05-27T07:36:52.254 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-05-27T07:36:52.254 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [ 2024-05-27T07:36:52.254 INFO:teuthology.orchestra.run.smithi052.stdout: "", 2024-05-27T07:36:52.254 INFO:teuthology.orchestra.run.smithi052.stdout: "critical", 2024-05-27T07:36:52.254 INFO:teuthology.orchestra.run.smithi052.stdout: "debug", 2024-05-27T07:36:52.254 INFO:teuthology.orchestra.run.smithi052.stdout: "error", 2024-05-27T07:36:52.254 INFO:teuthology.orchestra.run.smithi052.stdout: "info", 2024-05-27T07:36:52.254 INFO:teuthology.orchestra.run.smithi052.stdout: "warning" 2024-05-27T07:36:52.255 INFO:teuthology.orchestra.run.smithi052.stdout: ], 2024-05-27T07:36:52.255 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-05-27T07:36:52.255 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-05-27T07:36:52.255 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-05-27T07:36:52.255 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-05-27T07:36:52.255 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-05-27T07:36:52.255 INFO:teuthology.orchestra.run.smithi052.stdout: "log_to_file": { 2024-05-27T07:36:52.255 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "log_to_file", 2024-05-27T07:36:52.255 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "bool", 2024-05-27T07:36:52.255 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-05-27T07:36:52.255 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 1, 2024-05-27T07:36:52.255 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "False", 2024-05-27T07:36:52.256 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-05-27T07:36:52.256 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-05-27T07:36:52.256 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-05-27T07:36:52.256 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-05-27T07:36:52.256 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-05-27T07:36:52.256 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-05-27T07:36:52.256 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-05-27T07:36:52.256 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-05-27T07:36:52.256 INFO:teuthology.orchestra.run.smithi052.stdout: "organization": { 2024-05-27T07:36:52.256 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "organization", 2024-05-27T07:36:52.256 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "str", 2024-05-27T07:36:52.256 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-05-27T07:36:52.256 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 0, 2024-05-27T07:36:52.256 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "", 2024-05-27T07:36:52.257 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-05-27T07:36:52.257 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-05-27T07:36:52.257 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-05-27T07:36:52.257 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-05-27T07:36:52.257 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-05-27T07:36:52.257 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-05-27T07:36:52.257 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-05-27T07:36:52.257 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-05-27T07:36:52.257 INFO:teuthology.orchestra.run.smithi052.stdout: "proxy": { 2024-05-27T07:36:52.257 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "proxy", 2024-05-27T07:36:52.257 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "str", 2024-05-27T07:36:52.257 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-05-27T07:36:52.257 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 0, 2024-05-27T07:36:52.258 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "", 2024-05-27T07:36:52.258 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-05-27T07:36:52.258 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-05-27T07:36:52.258 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-05-27T07:36:52.258 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-05-27T07:36:52.258 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-05-27T07:36:52.258 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-05-27T07:36:52.258 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-05-27T07:36:52.258 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-05-27T07:36:52.258 INFO:teuthology.orchestra.run.smithi052.stdout: "sqlite3_killpoint": { 2024-05-27T07:36:52.258 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "sqlite3_killpoint", 2024-05-27T07:36:52.258 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "int", 2024-05-27T07:36:52.258 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "dev", 2024-05-27T07:36:52.259 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 1, 2024-05-27T07:36:52.259 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "0", 2024-05-27T07:36:52.259 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-05-27T07:36:52.259 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-05-27T07:36:52.259 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-05-27T07:36:52.259 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-05-27T07:36:52.259 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-05-27T07:36:52.259 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-05-27T07:36:52.259 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-05-27T07:36:52.259 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-05-27T07:36:52.259 INFO:teuthology.orchestra.run.smithi052.stdout: "url": { 2024-05-27T07:36:52.259 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "url", 2024-05-27T07:36:52.260 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "str", 2024-05-27T07:36:52.260 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-05-27T07:36:52.260 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 0, 2024-05-27T07:36:52.260 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "https://telemetry.ceph.com/report", 2024-05-27T07:36:52.260 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-05-27T07:36:52.260 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-05-27T07:36:52.260 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-05-27T07:36:52.260 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-05-27T07:36:52.260 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-05-27T07:36:52.260 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-05-27T07:36:52.260 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-05-27T07:36:52.260 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-05-27T07:36:52.260 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-05-27T07:36:52.261 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-05-27T07:36:52.261 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-05-27T07:36:52.261 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "test_orchestrator", 2024-05-27T07:36:52.261 INFO:teuthology.orchestra.run.smithi052.stdout: "can_run": true, 2024-05-27T07:36:52.261 INFO:teuthology.orchestra.run.smithi052.stdout: "error_string": "", 2024-05-27T07:36:52.261 INFO:teuthology.orchestra.run.smithi052.stdout: "module_options": { 2024-05-27T07:36:52.261 INFO:teuthology.orchestra.run.smithi052.stdout: "log_level": { 2024-05-27T07:36:52.261 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "log_level", 2024-05-27T07:36:52.261 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "str", 2024-05-27T07:36:52.261 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-05-27T07:36:52.261 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 1, 2024-05-27T07:36:52.261 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "", 2024-05-27T07:36:52.261 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-05-27T07:36:52.262 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-05-27T07:36:52.262 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [ 2024-05-27T07:36:52.262 INFO:teuthology.orchestra.run.smithi052.stdout: "", 2024-05-27T07:36:52.262 INFO:teuthology.orchestra.run.smithi052.stdout: "critical", 2024-05-27T07:36:52.262 INFO:teuthology.orchestra.run.smithi052.stdout: "debug", 2024-05-27T07:36:52.262 INFO:teuthology.orchestra.run.smithi052.stdout: "error", 2024-05-27T07:36:52.262 INFO:teuthology.orchestra.run.smithi052.stdout: "info", 2024-05-27T07:36:52.262 INFO:teuthology.orchestra.run.smithi052.stdout: "warning" 2024-05-27T07:36:52.262 INFO:teuthology.orchestra.run.smithi052.stdout: ], 2024-05-27T07:36:52.262 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-05-27T07:36:52.262 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-05-27T07:36:52.262 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-05-27T07:36:52.263 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-05-27T07:36:52.263 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-05-27T07:36:52.263 INFO:teuthology.orchestra.run.smithi052.stdout: "log_to_cluster": { 2024-05-27T07:36:52.263 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "log_to_cluster", 2024-05-27T07:36:52.263 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "bool", 2024-05-27T07:36:52.263 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-05-27T07:36:52.263 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 1, 2024-05-27T07:36:52.263 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "False", 2024-05-27T07:36:52.263 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-05-27T07:36:52.263 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-05-27T07:36:52.263 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-05-27T07:36:52.263 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-05-27T07:36:52.263 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-05-27T07:36:52.264 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-05-27T07:36:52.264 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-05-27T07:36:52.264 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-05-27T07:36:52.264 INFO:teuthology.orchestra.run.smithi052.stdout: "log_to_cluster_level": { 2024-05-27T07:36:52.264 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "log_to_cluster_level", 2024-05-27T07:36:52.264 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "str", 2024-05-27T07:36:52.264 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-05-27T07:36:52.264 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 1, 2024-05-27T07:36:52.264 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "info", 2024-05-27T07:36:52.264 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-05-27T07:36:52.264 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-05-27T07:36:52.264 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [ 2024-05-27T07:36:52.264 INFO:teuthology.orchestra.run.smithi052.stdout: "", 2024-05-27T07:36:52.265 INFO:teuthology.orchestra.run.smithi052.stdout: "critical", 2024-05-27T07:36:52.265 INFO:teuthology.orchestra.run.smithi052.stdout: "debug", 2024-05-27T07:36:52.265 INFO:teuthology.orchestra.run.smithi052.stdout: "error", 2024-05-27T07:36:52.265 INFO:teuthology.orchestra.run.smithi052.stdout: "info", 2024-05-27T07:36:52.265 INFO:teuthology.orchestra.run.smithi052.stdout: "warning" 2024-05-27T07:36:52.265 INFO:teuthology.orchestra.run.smithi052.stdout: ], 2024-05-27T07:36:52.265 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-05-27T07:36:52.265 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-05-27T07:36:52.265 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-05-27T07:36:52.265 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-05-27T07:36:52.265 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-05-27T07:36:52.265 INFO:teuthology.orchestra.run.smithi052.stdout: "log_to_file": { 2024-05-27T07:36:52.265 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "log_to_file", 2024-05-27T07:36:52.265 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "bool", 2024-05-27T07:36:52.266 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-05-27T07:36:52.266 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 1, 2024-05-27T07:36:52.266 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "False", 2024-05-27T07:36:52.266 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-05-27T07:36:52.266 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-05-27T07:36:52.266 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-05-27T07:36:52.266 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-05-27T07:36:52.266 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-05-27T07:36:52.266 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-05-27T07:36:52.266 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-05-27T07:36:52.266 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-05-27T07:36:52.266 INFO:teuthology.orchestra.run.smithi052.stdout: "sqlite3_killpoint": { 2024-05-27T07:36:52.267 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "sqlite3_killpoint", 2024-05-27T07:36:52.267 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "int", 2024-05-27T07:36:52.267 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "dev", 2024-05-27T07:36:52.267 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 1, 2024-05-27T07:36:52.267 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "0", 2024-05-27T07:36:52.267 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-05-27T07:36:52.267 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-05-27T07:36:52.267 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-05-27T07:36:52.267 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-05-27T07:36:52.267 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-05-27T07:36:52.267 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-05-27T07:36:52.267 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-05-27T07:36:52.267 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-05-27T07:36:52.268 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-05-27T07:36:52.268 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-05-27T07:36:52.268 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-05-27T07:36:52.268 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "volumes", 2024-05-27T07:36:52.268 INFO:teuthology.orchestra.run.smithi052.stdout: "can_run": true, 2024-05-27T07:36:52.268 INFO:teuthology.orchestra.run.smithi052.stdout: "error_string": "", 2024-05-27T07:36:52.268 INFO:teuthology.orchestra.run.smithi052.stdout: "module_options": { 2024-05-27T07:36:52.268 INFO:teuthology.orchestra.run.smithi052.stdout: "log_level": { 2024-05-27T07:36:52.268 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "log_level", 2024-05-27T07:36:52.268 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "str", 2024-05-27T07:36:52.268 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-05-27T07:36:52.268 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 1, 2024-05-27T07:36:52.268 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "", 2024-05-27T07:36:52.269 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-05-27T07:36:52.269 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-05-27T07:36:52.269 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [ 2024-05-27T07:36:52.269 INFO:teuthology.orchestra.run.smithi052.stdout: "", 2024-05-27T07:36:52.269 INFO:teuthology.orchestra.run.smithi052.stdout: "critical", 2024-05-27T07:36:52.269 INFO:teuthology.orchestra.run.smithi052.stdout: "debug", 2024-05-27T07:36:52.269 INFO:teuthology.orchestra.run.smithi052.stdout: "error", 2024-05-27T07:36:52.269 INFO:teuthology.orchestra.run.smithi052.stdout: "info", 2024-05-27T07:36:52.269 INFO:teuthology.orchestra.run.smithi052.stdout: "warning" 2024-05-27T07:36:52.269 INFO:teuthology.orchestra.run.smithi052.stdout: ], 2024-05-27T07:36:52.269 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-05-27T07:36:52.269 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-05-27T07:36:52.269 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-05-27T07:36:52.270 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-05-27T07:36:52.270 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-05-27T07:36:52.270 INFO:teuthology.orchestra.run.smithi052.stdout: "log_to_cluster": { 2024-05-27T07:36:52.270 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "log_to_cluster", 2024-05-27T07:36:52.270 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "bool", 2024-05-27T07:36:52.270 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-05-27T07:36:52.270 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 1, 2024-05-27T07:36:52.270 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "False", 2024-05-27T07:36:52.270 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-05-27T07:36:52.270 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-05-27T07:36:52.270 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-05-27T07:36:52.270 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-05-27T07:36:52.271 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-05-27T07:36:52.271 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-05-27T07:36:52.271 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-05-27T07:36:52.271 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-05-27T07:36:52.271 INFO:teuthology.orchestra.run.smithi052.stdout: "log_to_cluster_level": { 2024-05-27T07:36:52.271 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "log_to_cluster_level", 2024-05-27T07:36:52.271 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "str", 2024-05-27T07:36:52.271 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-05-27T07:36:52.271 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 1, 2024-05-27T07:36:52.271 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "info", 2024-05-27T07:36:52.271 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-05-27T07:36:52.271 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-05-27T07:36:52.271 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [ 2024-05-27T07:36:52.272 INFO:teuthology.orchestra.run.smithi052.stdout: "", 2024-05-27T07:36:52.272 INFO:teuthology.orchestra.run.smithi052.stdout: "critical", 2024-05-27T07:36:52.272 INFO:teuthology.orchestra.run.smithi052.stdout: "debug", 2024-05-27T07:36:52.272 INFO:teuthology.orchestra.run.smithi052.stdout: "error", 2024-05-27T07:36:52.272 INFO:teuthology.orchestra.run.smithi052.stdout: "info", 2024-05-27T07:36:52.272 INFO:teuthology.orchestra.run.smithi052.stdout: "warning" 2024-05-27T07:36:52.272 INFO:teuthology.orchestra.run.smithi052.stdout: ], 2024-05-27T07:36:52.272 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-05-27T07:36:52.272 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-05-27T07:36:52.272 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-05-27T07:36:52.272 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-05-27T07:36:52.272 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-05-27T07:36:52.272 INFO:teuthology.orchestra.run.smithi052.stdout: "log_to_file": { 2024-05-27T07:36:52.273 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "log_to_file", 2024-05-27T07:36:52.273 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "bool", 2024-05-27T07:36:52.273 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-05-27T07:36:52.273 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 1, 2024-05-27T07:36:52.273 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "False", 2024-05-27T07:36:52.273 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-05-27T07:36:52.273 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-05-27T07:36:52.273 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-05-27T07:36:52.273 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-05-27T07:36:52.273 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-05-27T07:36:52.273 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-05-27T07:36:52.273 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-05-27T07:36:52.273 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-05-27T07:36:52.274 INFO:teuthology.orchestra.run.smithi052.stdout: "max_concurrent_clones": { 2024-05-27T07:36:52.274 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "max_concurrent_clones", 2024-05-27T07:36:52.274 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "int", 2024-05-27T07:36:52.274 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-05-27T07:36:52.274 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 0, 2024-05-27T07:36:52.274 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "4", 2024-05-27T07:36:52.274 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-05-27T07:36:52.274 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-05-27T07:36:52.274 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-05-27T07:36:52.274 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "Number of asynchronous cloner threads", 2024-05-27T07:36:52.274 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-05-27T07:36:52.274 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-05-27T07:36:52.275 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-05-27T07:36:52.275 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-05-27T07:36:52.275 INFO:teuthology.orchestra.run.smithi052.stdout: "periodic_async_work": { 2024-05-27T07:36:52.275 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "periodic_async_work", 2024-05-27T07:36:52.275 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "bool", 2024-05-27T07:36:52.275 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-05-27T07:36:52.275 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 0, 2024-05-27T07:36:52.275 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "False", 2024-05-27T07:36:52.275 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-05-27T07:36:52.275 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-05-27T07:36:52.275 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-05-27T07:36:52.275 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "Periodically check for async work", 2024-05-27T07:36:52.276 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-05-27T07:36:52.276 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-05-27T07:36:52.276 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-05-27T07:36:52.276 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-05-27T07:36:52.276 INFO:teuthology.orchestra.run.smithi052.stdout: "snapshot_clone_delay": { 2024-05-27T07:36:52.276 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "snapshot_clone_delay", 2024-05-27T07:36:52.276 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "int", 2024-05-27T07:36:52.276 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-05-27T07:36:52.276 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 0, 2024-05-27T07:36:52.276 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "0", 2024-05-27T07:36:52.276 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-05-27T07:36:52.276 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-05-27T07:36:52.276 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-05-27T07:36:52.277 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "Delay clone begin operation by snapshot_clone_delay seconds", 2024-05-27T07:36:52.277 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-05-27T07:36:52.277 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-05-27T07:36:52.277 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-05-27T07:36:52.277 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-05-27T07:36:52.277 INFO:teuthology.orchestra.run.smithi052.stdout: "snapshot_clone_no_wait": { 2024-05-27T07:36:52.277 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "snapshot_clone_no_wait", 2024-05-27T07:36:52.277 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "bool", 2024-05-27T07:36:52.277 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-05-27T07:36:52.277 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 0, 2024-05-27T07:36:52.277 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "True", 2024-05-27T07:36:52.277 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-05-27T07:36:52.278 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-05-27T07:36:52.278 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-05-27T07:36:52.278 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "Reject subvolume clone request when cloner threads are busy", 2024-05-27T07:36:52.278 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-05-27T07:36:52.278 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-05-27T07:36:52.278 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-05-27T07:36:52.278 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-05-27T07:36:52.278 INFO:teuthology.orchestra.run.smithi052.stdout: "sqlite3_killpoint": { 2024-05-27T07:36:52.278 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "sqlite3_killpoint", 2024-05-27T07:36:52.278 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "int", 2024-05-27T07:36:52.278 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "dev", 2024-05-27T07:36:52.278 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 1, 2024-05-27T07:36:52.278 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "0", 2024-05-27T07:36:52.279 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-05-27T07:36:52.279 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-05-27T07:36:52.279 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-05-27T07:36:52.279 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-05-27T07:36:52.279 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-05-27T07:36:52.279 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-05-27T07:36:52.279 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-05-27T07:36:52.279 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-05-27T07:36:52.279 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-05-27T07:36:52.279 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-05-27T07:36:52.279 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-05-27T07:36:52.279 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "zabbix", 2024-05-27T07:36:52.279 INFO:teuthology.orchestra.run.smithi052.stdout: "can_run": true, 2024-05-27T07:36:52.280 INFO:teuthology.orchestra.run.smithi052.stdout: "error_string": "", 2024-05-27T07:36:52.280 INFO:teuthology.orchestra.run.smithi052.stdout: "module_options": { 2024-05-27T07:36:52.280 INFO:teuthology.orchestra.run.smithi052.stdout: "discovery_interval": { 2024-05-27T07:36:52.280 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "discovery_interval", 2024-05-27T07:36:52.280 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "uint", 2024-05-27T07:36:52.280 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-05-27T07:36:52.280 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 0, 2024-05-27T07:36:52.280 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "100", 2024-05-27T07:36:52.280 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-05-27T07:36:52.280 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-05-27T07:36:52.280 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-05-27T07:36:52.280 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-05-27T07:36:52.281 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-05-27T07:36:52.281 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-05-27T07:36:52.281 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-05-27T07:36:52.281 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-05-27T07:36:52.281 INFO:teuthology.orchestra.run.smithi052.stdout: "identifier": { 2024-05-27T07:36:52.281 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "identifier", 2024-05-27T07:36:52.281 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "str", 2024-05-27T07:36:52.281 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-05-27T07:36:52.281 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 0, 2024-05-27T07:36:52.281 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "", 2024-05-27T07:36:52.281 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-05-27T07:36:52.281 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-05-27T07:36:52.281 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-05-27T07:36:52.282 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-05-27T07:36:52.282 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-05-27T07:36:52.282 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-05-27T07:36:52.282 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-05-27T07:36:52.282 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-05-27T07:36:52.282 INFO:teuthology.orchestra.run.smithi052.stdout: "interval": { 2024-05-27T07:36:52.282 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "interval", 2024-05-27T07:36:52.282 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "secs", 2024-05-27T07:36:52.282 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-05-27T07:36:52.282 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 0, 2024-05-27T07:36:52.282 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "60", 2024-05-27T07:36:52.282 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-05-27T07:36:52.283 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-05-27T07:36:52.283 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-05-27T07:36:52.283 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-05-27T07:36:52.283 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-05-27T07:36:52.283 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-05-27T07:36:52.283 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-05-27T07:36:52.283 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-05-27T07:36:52.283 INFO:teuthology.orchestra.run.smithi052.stdout: "log_level": { 2024-05-27T07:36:52.283 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "log_level", 2024-05-27T07:36:52.283 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "str", 2024-05-27T07:36:52.283 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-05-27T07:36:52.283 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 1, 2024-05-27T07:36:52.283 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "", 2024-05-27T07:36:52.284 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-05-27T07:36:52.284 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-05-27T07:36:52.284 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [ 2024-05-27T07:36:52.284 INFO:teuthology.orchestra.run.smithi052.stdout: "", 2024-05-27T07:36:52.284 INFO:teuthology.orchestra.run.smithi052.stdout: "critical", 2024-05-27T07:36:52.284 INFO:teuthology.orchestra.run.smithi052.stdout: "debug", 2024-05-27T07:36:52.284 INFO:teuthology.orchestra.run.smithi052.stdout: "error", 2024-05-27T07:36:52.284 INFO:teuthology.orchestra.run.smithi052.stdout: "info", 2024-05-27T07:36:52.284 INFO:teuthology.orchestra.run.smithi052.stdout: "warning" 2024-05-27T07:36:52.284 INFO:teuthology.orchestra.run.smithi052.stdout: ], 2024-05-27T07:36:52.284 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-05-27T07:36:52.284 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-05-27T07:36:52.284 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-05-27T07:36:52.285 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-05-27T07:36:52.285 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-05-27T07:36:52.285 INFO:teuthology.orchestra.run.smithi052.stdout: "log_to_cluster": { 2024-05-27T07:36:52.285 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "log_to_cluster", 2024-05-27T07:36:52.285 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "bool", 2024-05-27T07:36:52.285 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-05-27T07:36:52.285 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 1, 2024-05-27T07:36:52.285 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "False", 2024-05-27T07:36:52.285 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-05-27T07:36:52.285 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-05-27T07:36:52.285 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-05-27T07:36:52.285 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-05-27T07:36:52.286 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-05-27T07:36:52.286 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-05-27T07:36:52.286 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-05-27T07:36:52.286 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-05-27T07:36:52.286 INFO:teuthology.orchestra.run.smithi052.stdout: "log_to_cluster_level": { 2024-05-27T07:36:52.286 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "log_to_cluster_level", 2024-05-27T07:36:52.286 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "str", 2024-05-27T07:36:52.286 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-05-27T07:36:52.286 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 1, 2024-05-27T07:36:52.286 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "info", 2024-05-27T07:36:52.286 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-05-27T07:36:52.286 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-05-27T07:36:52.286 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [ 2024-05-27T07:36:52.287 INFO:teuthology.orchestra.run.smithi052.stdout: "", 2024-05-27T07:36:52.287 INFO:teuthology.orchestra.run.smithi052.stdout: "critical", 2024-05-27T07:36:52.287 INFO:teuthology.orchestra.run.smithi052.stdout: "debug", 2024-05-27T07:36:52.287 INFO:teuthology.orchestra.run.smithi052.stdout: "error", 2024-05-27T07:36:52.287 INFO:teuthology.orchestra.run.smithi052.stdout: "info", 2024-05-27T07:36:52.287 INFO:teuthology.orchestra.run.smithi052.stdout: "warning" 2024-05-27T07:36:52.287 INFO:teuthology.orchestra.run.smithi052.stdout: ], 2024-05-27T07:36:52.287 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-05-27T07:36:52.287 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-05-27T07:36:52.287 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-05-27T07:36:52.287 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-05-27T07:36:52.287 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-05-27T07:36:52.288 INFO:teuthology.orchestra.run.smithi052.stdout: "log_to_file": { 2024-05-27T07:36:52.288 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "log_to_file", 2024-05-27T07:36:52.288 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "bool", 2024-05-27T07:36:52.288 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-05-27T07:36:52.288 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 1, 2024-05-27T07:36:52.288 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "False", 2024-05-27T07:36:52.288 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-05-27T07:36:52.288 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-05-27T07:36:52.288 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-05-27T07:36:52.288 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-05-27T07:36:52.288 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-05-27T07:36:52.288 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-05-27T07:36:52.288 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-05-27T07:36:52.288 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-05-27T07:36:52.289 INFO:teuthology.orchestra.run.smithi052.stdout: "sqlite3_killpoint": { 2024-05-27T07:36:52.289 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "sqlite3_killpoint", 2024-05-27T07:36:52.289 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "int", 2024-05-27T07:36:52.289 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "dev", 2024-05-27T07:36:52.289 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 1, 2024-05-27T07:36:52.289 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "0", 2024-05-27T07:36:52.289 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-05-27T07:36:52.289 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-05-27T07:36:52.289 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-05-27T07:36:52.289 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-05-27T07:36:52.289 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-05-27T07:36:52.289 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-05-27T07:36:52.290 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-05-27T07:36:52.290 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-05-27T07:36:52.290 INFO:teuthology.orchestra.run.smithi052.stdout: "zabbix_host": { 2024-05-27T07:36:52.290 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "zabbix_host", 2024-05-27T07:36:52.290 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "str", 2024-05-27T07:36:52.290 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-05-27T07:36:52.290 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 0, 2024-05-27T07:36:52.290 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "", 2024-05-27T07:36:52.290 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-05-27T07:36:52.290 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-05-27T07:36:52.290 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-05-27T07:36:52.290 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-05-27T07:36:52.291 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-05-27T07:36:52.291 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-05-27T07:36:52.291 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-05-27T07:36:52.291 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-05-27T07:36:52.291 INFO:teuthology.orchestra.run.smithi052.stdout: "zabbix_port": { 2024-05-27T07:36:52.291 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "zabbix_port", 2024-05-27T07:36:52.291 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "int", 2024-05-27T07:36:52.291 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-05-27T07:36:52.291 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 0, 2024-05-27T07:36:52.291 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "10051", 2024-05-27T07:36:52.291 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-05-27T07:36:52.291 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-05-27T07:36:52.291 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-05-27T07:36:52.292 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-05-27T07:36:52.292 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-05-27T07:36:52.292 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-05-27T07:36:52.292 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-05-27T07:36:52.292 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-05-27T07:36:52.292 INFO:teuthology.orchestra.run.smithi052.stdout: "zabbix_sender": { 2024-05-27T07:36:52.292 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "zabbix_sender", 2024-05-27T07:36:52.292 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "str", 2024-05-27T07:36:52.292 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-05-27T07:36:52.292 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 0, 2024-05-27T07:36:52.292 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "/usr/bin/zabbix_sender", 2024-05-27T07:36:52.292 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-05-27T07:36:52.292 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-05-27T07:36:52.293 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-05-27T07:36:52.293 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-05-27T07:36:52.293 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-05-27T07:36:52.293 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-05-27T07:36:52.293 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-05-27T07:36:52.293 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-05-27T07:36:52.293 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-05-27T07:36:52.293 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-05-27T07:36:52.293 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2024-05-27T07:36:52.293 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-05-27T07:36:52.293 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-05-27T07:36:52.293 INFO:teuthology.orchestra.run.smithi052.stdout: "gid": 25073, 2024-05-27T07:36:52.293 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "y", 2024-05-27T07:36:52.294 INFO:teuthology.orchestra.run.smithi052.stdout: "mgr_features": 4540701547738038271, 2024-05-27T07:36:52.294 INFO:teuthology.orchestra.run.smithi052.stdout: "available_modules": [ 2024-05-27T07:36:52.294 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-05-27T07:36:52.294 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "alerts", 2024-05-27T07:36:52.294 INFO:teuthology.orchestra.run.smithi052.stdout: "can_run": true, 2024-05-27T07:36:52.294 INFO:teuthology.orchestra.run.smithi052.stdout: "error_string": "", 2024-05-27T07:36:52.294 INFO:teuthology.orchestra.run.smithi052.stdout: "module_options": { 2024-05-27T07:36:52.294 INFO:teuthology.orchestra.run.smithi052.stdout: "interval": { 2024-05-27T07:36:52.294 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "interval", 2024-05-27T07:36:52.294 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "secs", 2024-05-27T07:36:52.294 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-05-27T07:36:52.294 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 1, 2024-05-27T07:36:52.295 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "60", 2024-05-27T07:36:52.295 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-05-27T07:36:52.295 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-05-27T07:36:52.295 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-05-27T07:36:52.295 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "How frequently to reexamine health status", 2024-05-27T07:36:52.295 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-05-27T07:36:52.295 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-05-27T07:36:52.295 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-05-27T07:36:52.295 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-05-27T07:36:52.295 INFO:teuthology.orchestra.run.smithi052.stdout: "log_level": { 2024-05-27T07:36:52.295 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "log_level", 2024-05-27T07:36:52.295 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "str", 2024-05-27T07:36:52.295 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-05-27T07:36:52.295 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 1, 2024-05-27T07:36:52.296 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "", 2024-05-27T07:36:52.296 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-05-27T07:36:52.296 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-05-27T07:36:52.296 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [ 2024-05-27T07:36:52.296 INFO:teuthology.orchestra.run.smithi052.stdout: "", 2024-05-27T07:36:52.296 INFO:teuthology.orchestra.run.smithi052.stdout: "critical", 2024-05-27T07:36:52.296 INFO:teuthology.orchestra.run.smithi052.stdout: "debug", 2024-05-27T07:36:52.296 INFO:teuthology.orchestra.run.smithi052.stdout: "error", 2024-05-27T07:36:52.296 INFO:teuthology.orchestra.run.smithi052.stdout: "info", 2024-05-27T07:36:52.296 INFO:teuthology.orchestra.run.smithi052.stdout: "warning" 2024-05-27T07:36:52.296 INFO:teuthology.orchestra.run.smithi052.stdout: ], 2024-05-27T07:36:52.296 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-05-27T07:36:52.297 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-05-27T07:36:52.297 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-05-27T07:36:52.297 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-05-27T07:36:52.297 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-05-27T07:36:52.297 INFO:teuthology.orchestra.run.smithi052.stdout: "log_to_cluster": { 2024-05-27T07:36:52.297 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "log_to_cluster", 2024-05-27T07:36:52.297 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "bool", 2024-05-27T07:36:52.297 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-05-27T07:36:52.297 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 1, 2024-05-27T07:36:52.297 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "False", 2024-05-27T07:36:52.297 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-05-27T07:36:52.297 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-05-27T07:36:52.297 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-05-27T07:36:52.298 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-05-27T07:36:52.298 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-05-27T07:36:52.298 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-05-27T07:36:52.298 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-05-27T07:36:52.298 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-05-27T07:36:52.298 INFO:teuthology.orchestra.run.smithi052.stdout: "log_to_cluster_level": { 2024-05-27T07:36:52.298 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "log_to_cluster_level", 2024-05-27T07:36:52.298 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "str", 2024-05-27T07:36:52.298 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-05-27T07:36:52.298 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 1, 2024-05-27T07:36:52.298 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "info", 2024-05-27T07:36:52.298 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-05-27T07:36:52.299 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-05-27T07:36:52.299 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [ 2024-05-27T07:36:52.299 INFO:teuthology.orchestra.run.smithi052.stdout: "", 2024-05-27T07:36:52.299 INFO:teuthology.orchestra.run.smithi052.stdout: "critical", 2024-05-27T07:36:52.299 INFO:teuthology.orchestra.run.smithi052.stdout: "debug", 2024-05-27T07:36:52.299 INFO:teuthology.orchestra.run.smithi052.stdout: "error", 2024-05-27T07:36:52.299 INFO:teuthology.orchestra.run.smithi052.stdout: "info", 2024-05-27T07:36:52.299 INFO:teuthology.orchestra.run.smithi052.stdout: "warning" 2024-05-27T07:36:52.299 INFO:teuthology.orchestra.run.smithi052.stdout: ], 2024-05-27T07:36:52.299 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-05-27T07:36:52.299 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-05-27T07:36:52.299 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-05-27T07:36:52.299 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-05-27T07:36:52.300 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-05-27T07:36:52.300 INFO:teuthology.orchestra.run.smithi052.stdout: "log_to_file": { 2024-05-27T07:36:52.300 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "log_to_file", 2024-05-27T07:36:52.300 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "bool", 2024-05-27T07:36:52.300 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-05-27T07:36:52.300 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 1, 2024-05-27T07:36:52.300 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "False", 2024-05-27T07:36:52.300 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-05-27T07:36:52.300 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-05-27T07:36:52.300 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-05-27T07:36:52.300 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-05-27T07:36:52.300 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-05-27T07:36:52.300 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-05-27T07:36:52.301 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-05-27T07:36:52.301 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-05-27T07:36:52.301 INFO:teuthology.orchestra.run.smithi052.stdout: "smtp_destination": { 2024-05-27T07:36:52.301 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "smtp_destination", 2024-05-27T07:36:52.301 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "str", 2024-05-27T07:36:52.301 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-05-27T07:36:52.301 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 1, 2024-05-27T07:36:52.301 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "", 2024-05-27T07:36:52.301 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-05-27T07:36:52.301 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-05-27T07:36:52.301 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-05-27T07:36:52.301 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "Email address to send alerts to", 2024-05-27T07:36:52.301 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-05-27T07:36:52.302 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-05-27T07:36:52.302 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-05-27T07:36:52.302 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-05-27T07:36:52.302 INFO:teuthology.orchestra.run.smithi052.stdout: "smtp_from_name": { 2024-05-27T07:36:52.302 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "smtp_from_name", 2024-05-27T07:36:52.302 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "str", 2024-05-27T07:36:52.302 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-05-27T07:36:52.302 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 1, 2024-05-27T07:36:52.302 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "Ceph", 2024-05-27T07:36:52.302 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-05-27T07:36:52.302 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-05-27T07:36:52.302 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-05-27T07:36:52.302 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "Email From: name", 2024-05-27T07:36:52.303 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-05-27T07:36:52.303 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-05-27T07:36:52.303 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-05-27T07:36:52.303 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-05-27T07:36:52.303 INFO:teuthology.orchestra.run.smithi052.stdout: "smtp_host": { 2024-05-27T07:36:52.303 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "smtp_host", 2024-05-27T07:36:52.303 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "str", 2024-05-27T07:36:52.303 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-05-27T07:36:52.303 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 1, 2024-05-27T07:36:52.303 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "", 2024-05-27T07:36:52.303 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-05-27T07:36:52.303 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-05-27T07:36:52.303 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-05-27T07:36:52.304 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "SMTP server", 2024-05-27T07:36:52.304 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-05-27T07:36:52.304 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-05-27T07:36:52.304 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-05-27T07:36:52.304 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-05-27T07:36:52.304 INFO:teuthology.orchestra.run.smithi052.stdout: "smtp_password": { 2024-05-27T07:36:52.304 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "smtp_password", 2024-05-27T07:36:52.304 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "str", 2024-05-27T07:36:52.304 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-05-27T07:36:52.304 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 1, 2024-05-27T07:36:52.304 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "", 2024-05-27T07:36:52.304 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-05-27T07:36:52.305 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-05-27T07:36:52.305 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-05-27T07:36:52.305 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "Password to authenticate with", 2024-05-27T07:36:52.305 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-05-27T07:36:52.305 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-05-27T07:36:52.305 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-05-27T07:36:52.305 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-05-27T07:36:52.305 INFO:teuthology.orchestra.run.smithi052.stdout: "smtp_port": { 2024-05-27T07:36:52.305 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "smtp_port", 2024-05-27T07:36:52.305 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "int", 2024-05-27T07:36:52.305 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-05-27T07:36:52.305 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 1, 2024-05-27T07:36:52.306 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "465", 2024-05-27T07:36:52.306 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-05-27T07:36:52.306 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-05-27T07:36:52.306 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-05-27T07:36:52.306 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "SMTP port", 2024-05-27T07:36:52.306 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-05-27T07:36:52.306 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-05-27T07:36:52.306 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-05-27T07:36:52.306 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-05-27T07:36:52.306 INFO:teuthology.orchestra.run.smithi052.stdout: "smtp_sender": { 2024-05-27T07:36:52.306 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "smtp_sender", 2024-05-27T07:36:52.306 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "str", 2024-05-27T07:36:52.306 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-05-27T07:36:52.307 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 1, 2024-05-27T07:36:52.307 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "", 2024-05-27T07:36:52.307 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-05-27T07:36:52.307 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-05-27T07:36:52.307 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-05-27T07:36:52.307 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "SMTP envelope sender", 2024-05-27T07:36:52.307 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-05-27T07:36:52.307 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-05-27T07:36:52.307 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-05-27T07:36:52.307 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-05-27T07:36:52.307 INFO:teuthology.orchestra.run.smithi052.stdout: "smtp_ssl": { 2024-05-27T07:36:52.307 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "smtp_ssl", 2024-05-27T07:36:52.307 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "bool", 2024-05-27T07:36:52.308 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-05-27T07:36:52.308 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 1, 2024-05-27T07:36:52.308 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "True", 2024-05-27T07:36:52.308 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-05-27T07:36:52.308 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-05-27T07:36:52.308 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-05-27T07:36:52.308 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "Use SSL to connect to SMTP server", 2024-05-27T07:36:52.308 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-05-27T07:36:52.308 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-05-27T07:36:52.308 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-05-27T07:36:52.308 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-05-27T07:36:52.308 INFO:teuthology.orchestra.run.smithi052.stdout: "smtp_user": { 2024-05-27T07:36:52.308 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "smtp_user", 2024-05-27T07:36:52.309 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "str", 2024-05-27T07:36:52.309 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-05-27T07:36:52.309 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 1, 2024-05-27T07:36:52.309 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "", 2024-05-27T07:36:52.309 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-05-27T07:36:52.309 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-05-27T07:36:52.309 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-05-27T07:36:52.309 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "User to authenticate as", 2024-05-27T07:36:52.309 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-05-27T07:36:52.309 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-05-27T07:36:52.309 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-05-27T07:36:52.309 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-05-27T07:36:52.310 INFO:teuthology.orchestra.run.smithi052.stdout: "sqlite3_killpoint": { 2024-05-27T07:36:52.310 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "sqlite3_killpoint", 2024-05-27T07:36:52.310 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "int", 2024-05-27T07:36:52.310 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "dev", 2024-05-27T07:36:52.310 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 1, 2024-05-27T07:36:52.310 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "0", 2024-05-27T07:36:52.310 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-05-27T07:36:52.310 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-05-27T07:36:52.310 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-05-27T07:36:52.310 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-05-27T07:36:52.310 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-05-27T07:36:52.310 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-05-27T07:36:52.310 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-05-27T07:36:52.311 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-05-27T07:36:52.311 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-05-27T07:36:52.311 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-05-27T07:36:52.311 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-05-27T07:36:52.311 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "balancer", 2024-05-27T07:36:52.311 INFO:teuthology.orchestra.run.smithi052.stdout: "can_run": true, 2024-05-27T07:36:52.311 INFO:teuthology.orchestra.run.smithi052.stdout: "error_string": "", 2024-05-27T07:36:52.311 INFO:teuthology.orchestra.run.smithi052.stdout: "module_options": { 2024-05-27T07:36:52.311 INFO:teuthology.orchestra.run.smithi052.stdout: "active": { 2024-05-27T07:36:52.311 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "active", 2024-05-27T07:36:52.311 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "bool", 2024-05-27T07:36:52.311 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-05-27T07:36:52.311 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 1, 2024-05-27T07:36:52.312 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "True", 2024-05-27T07:36:52.312 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-05-27T07:36:52.312 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-05-27T07:36:52.312 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-05-27T07:36:52.312 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "automatically balance PGs across cluster", 2024-05-27T07:36:52.312 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-05-27T07:36:52.312 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-05-27T07:36:52.312 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-05-27T07:36:52.312 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-05-27T07:36:52.312 INFO:teuthology.orchestra.run.smithi052.stdout: "begin_time": { 2024-05-27T07:36:52.312 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "begin_time", 2024-05-27T07:36:52.312 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "str", 2024-05-27T07:36:52.312 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-05-27T07:36:52.313 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 1, 2024-05-27T07:36:52.313 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "0000", 2024-05-27T07:36:52.313 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-05-27T07:36:52.313 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-05-27T07:36:52.313 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-05-27T07:36:52.313 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "beginning time of day to automatically balance", 2024-05-27T07:36:52.313 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "This is a time of day in the format HHMM.", 2024-05-27T07:36:52.313 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-05-27T07:36:52.313 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-05-27T07:36:52.313 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-05-27T07:36:52.313 INFO:teuthology.orchestra.run.smithi052.stdout: "begin_weekday": { 2024-05-27T07:36:52.313 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "begin_weekday", 2024-05-27T07:36:52.313 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "uint", 2024-05-27T07:36:52.314 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-05-27T07:36:52.314 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 1, 2024-05-27T07:36:52.314 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "0", 2024-05-27T07:36:52.314 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "0", 2024-05-27T07:36:52.314 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "6", 2024-05-27T07:36:52.314 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-05-27T07:36:52.314 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "Restrict automatic balancing to this day of the week or later", 2024-05-27T07:36:52.314 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "0 = Sunday, 1 = Monday, etc.", 2024-05-27T07:36:52.314 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-05-27T07:36:52.314 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-05-27T07:36:52.314 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-05-27T07:36:52.314 INFO:teuthology.orchestra.run.smithi052.stdout: "crush_compat_max_iterations": { 2024-05-27T07:36:52.315 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "crush_compat_max_iterations", 2024-05-27T07:36:52.315 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "uint", 2024-05-27T07:36:52.315 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-05-27T07:36:52.315 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 1, 2024-05-27T07:36:52.315 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "25", 2024-05-27T07:36:52.315 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "1", 2024-05-27T07:36:52.315 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "250", 2024-05-27T07:36:52.315 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-05-27T07:36:52.315 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "maximum number of iterations to attempt optimization", 2024-05-27T07:36:52.315 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-05-27T07:36:52.315 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-05-27T07:36:52.315 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-05-27T07:36:52.315 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-05-27T07:36:52.316 INFO:teuthology.orchestra.run.smithi052.stdout: "crush_compat_metrics": { 2024-05-27T07:36:52.316 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "crush_compat_metrics", 2024-05-27T07:36:52.316 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "str", 2024-05-27T07:36:52.316 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-05-27T07:36:52.316 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 1, 2024-05-27T07:36:52.316 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "pgs,objects,bytes", 2024-05-27T07:36:52.316 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-05-27T07:36:52.316 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-05-27T07:36:52.316 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-05-27T07:36:52.316 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "metrics with which to calculate OSD utilization", 2024-05-27T07:36:52.316 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.", 2024-05-27T07:36:52.316 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-05-27T07:36:52.317 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-05-27T07:36:52.317 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-05-27T07:36:52.317 INFO:teuthology.orchestra.run.smithi052.stdout: "crush_compat_step": { 2024-05-27T07:36:52.317 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "crush_compat_step", 2024-05-27T07:36:52.317 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "float", 2024-05-27T07:36:52.317 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-05-27T07:36:52.317 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 1, 2024-05-27T07:36:52.317 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "0.5", 2024-05-27T07:36:52.317 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "0.001", 2024-05-27T07:36:52.317 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "0.999", 2024-05-27T07:36:52.317 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-05-27T07:36:52.317 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "aggressiveness of optimization", 2024-05-27T07:36:52.317 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": ".99 is very aggressive, .01 is less aggressive", 2024-05-27T07:36:52.318 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-05-27T07:36:52.318 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-05-27T07:36:52.318 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-05-27T07:36:52.318 INFO:teuthology.orchestra.run.smithi052.stdout: "end_time": { 2024-05-27T07:36:52.318 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "end_time", 2024-05-27T07:36:52.318 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "str", 2024-05-27T07:36:52.318 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-05-27T07:36:52.318 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 1, 2024-05-27T07:36:52.318 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "2359", 2024-05-27T07:36:52.318 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-05-27T07:36:52.318 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-05-27T07:36:52.318 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-05-27T07:36:52.319 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "ending time of day to automatically balance", 2024-05-27T07:36:52.319 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "This is a time of day in the format HHMM.", 2024-05-27T07:36:52.319 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-05-27T07:36:52.319 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-05-27T07:36:52.319 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-05-27T07:36:52.319 INFO:teuthology.orchestra.run.smithi052.stdout: "end_weekday": { 2024-05-27T07:36:52.319 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "end_weekday", 2024-05-27T07:36:52.319 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "uint", 2024-05-27T07:36:52.319 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-05-27T07:36:52.319 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 1, 2024-05-27T07:36:52.319 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "0", 2024-05-27T07:36:52.319 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "0", 2024-05-27T07:36:52.320 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "6", 2024-05-27T07:36:52.320 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-05-27T07:36:52.320 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "Restrict automatic balancing to days of the week earlier than this", 2024-05-27T07:36:52.320 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "0 = Sunday, 1 = Monday, etc.", 2024-05-27T07:36:52.320 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-05-27T07:36:52.320 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-05-27T07:36:52.320 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-05-27T07:36:52.320 INFO:teuthology.orchestra.run.smithi052.stdout: "log_level": { 2024-05-27T07:36:52.320 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "log_level", 2024-05-27T07:36:52.320 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "str", 2024-05-27T07:36:52.320 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-05-27T07:36:52.320 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 1, 2024-05-27T07:36:52.321 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "", 2024-05-27T07:36:52.321 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-05-27T07:36:52.321 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-05-27T07:36:52.321 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [ 2024-05-27T07:36:52.321 INFO:teuthology.orchestra.run.smithi052.stdout: "", 2024-05-27T07:36:52.321 INFO:teuthology.orchestra.run.smithi052.stdout: "critical", 2024-05-27T07:36:52.321 INFO:teuthology.orchestra.run.smithi052.stdout: "debug", 2024-05-27T07:36:52.321 INFO:teuthology.orchestra.run.smithi052.stdout: "error", 2024-05-27T07:36:52.321 INFO:teuthology.orchestra.run.smithi052.stdout: "info", 2024-05-27T07:36:52.321 INFO:teuthology.orchestra.run.smithi052.stdout: "warning" 2024-05-27T07:36:52.321 INFO:teuthology.orchestra.run.smithi052.stdout: ], 2024-05-27T07:36:52.321 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-05-27T07:36:52.321 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-05-27T07:36:52.322 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-05-27T07:36:52.322 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-05-27T07:36:52.322 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-05-27T07:36:52.322 INFO:teuthology.orchestra.run.smithi052.stdout: "log_to_cluster": { 2024-05-27T07:36:52.322 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "log_to_cluster", 2024-05-27T07:36:52.322 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "bool", 2024-05-27T07:36:52.322 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-05-27T07:36:52.322 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 1, 2024-05-27T07:36:52.322 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "False", 2024-05-27T07:36:52.322 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-05-27T07:36:52.322 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-05-27T07:36:52.322 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-05-27T07:36:52.323 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-05-27T07:36:52.323 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-05-27T07:36:52.323 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-05-27T07:36:52.323 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-05-27T07:36:52.323 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-05-27T07:36:52.323 INFO:teuthology.orchestra.run.smithi052.stdout: "log_to_cluster_level": { 2024-05-27T07:36:52.323 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "log_to_cluster_level", 2024-05-27T07:36:52.323 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "str", 2024-05-27T07:36:52.323 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-05-27T07:36:52.323 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 1, 2024-05-27T07:36:52.323 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "info", 2024-05-27T07:36:52.323 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-05-27T07:36:52.323 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-05-27T07:36:52.324 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [ 2024-05-27T07:36:52.324 INFO:teuthology.orchestra.run.smithi052.stdout: "", 2024-05-27T07:36:52.324 INFO:teuthology.orchestra.run.smithi052.stdout: "critical", 2024-05-27T07:36:52.324 INFO:teuthology.orchestra.run.smithi052.stdout: "debug", 2024-05-27T07:36:52.324 INFO:teuthology.orchestra.run.smithi052.stdout: "error", 2024-05-27T07:36:52.324 INFO:teuthology.orchestra.run.smithi052.stdout: "info", 2024-05-27T07:36:52.324 INFO:teuthology.orchestra.run.smithi052.stdout: "warning" 2024-05-27T07:36:52.324 INFO:teuthology.orchestra.run.smithi052.stdout: ], 2024-05-27T07:36:52.324 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-05-27T07:36:52.324 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-05-27T07:36:52.324 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-05-27T07:36:52.324 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-05-27T07:36:52.324 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-05-27T07:36:52.325 INFO:teuthology.orchestra.run.smithi052.stdout: "log_to_file": { 2024-05-27T07:36:52.325 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "log_to_file", 2024-05-27T07:36:52.325 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "bool", 2024-05-27T07:36:52.325 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-05-27T07:36:52.325 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 1, 2024-05-27T07:36:52.325 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "False", 2024-05-27T07:36:52.325 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-05-27T07:36:52.325 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-05-27T07:36:52.325 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-05-27T07:36:52.325 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-05-27T07:36:52.325 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-05-27T07:36:52.325 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-05-27T07:36:52.325 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-05-27T07:36:52.326 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-05-27T07:36:52.326 INFO:teuthology.orchestra.run.smithi052.stdout: "min_score": { 2024-05-27T07:36:52.326 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "min_score", 2024-05-27T07:36:52.326 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "float", 2024-05-27T07:36:52.326 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-05-27T07:36:52.326 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 1, 2024-05-27T07:36:52.326 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "0", 2024-05-27T07:36:52.326 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-05-27T07:36:52.326 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-05-27T07:36:52.326 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-05-27T07:36:52.326 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "minimum score, below which no optimization is attempted", 2024-05-27T07:36:52.326 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-05-27T07:36:52.326 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-05-27T07:36:52.327 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-05-27T07:36:52.327 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-05-27T07:36:52.327 INFO:teuthology.orchestra.run.smithi052.stdout: "mode": { 2024-05-27T07:36:52.327 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "mode", 2024-05-27T07:36:52.327 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "str", 2024-05-27T07:36:52.327 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-05-27T07:36:52.327 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 1, 2024-05-27T07:36:52.327 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "upmap", 2024-05-27T07:36:52.327 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-05-27T07:36:52.327 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-05-27T07:36:52.327 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [ 2024-05-27T07:36:52.327 INFO:teuthology.orchestra.run.smithi052.stdout: "crush-compat", 2024-05-27T07:36:52.327 INFO:teuthology.orchestra.run.smithi052.stdout: "none", 2024-05-27T07:36:52.328 INFO:teuthology.orchestra.run.smithi052.stdout: "read", 2024-05-27T07:36:52.328 INFO:teuthology.orchestra.run.smithi052.stdout: "upmap", 2024-05-27T07:36:52.328 INFO:teuthology.orchestra.run.smithi052.stdout: "upmap-read" 2024-05-27T07:36:52.328 INFO:teuthology.orchestra.run.smithi052.stdout: ], 2024-05-27T07:36:52.328 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "Balancer mode", 2024-05-27T07:36:52.328 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-05-27T07:36:52.328 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-05-27T07:36:52.328 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-05-27T07:36:52.328 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-05-27T07:36:52.328 INFO:teuthology.orchestra.run.smithi052.stdout: "pool_ids": { 2024-05-27T07:36:52.328 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "pool_ids", 2024-05-27T07:36:52.328 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "str", 2024-05-27T07:36:52.329 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-05-27T07:36:52.329 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 1, 2024-05-27T07:36:52.329 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "", 2024-05-27T07:36:52.329 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-05-27T07:36:52.329 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-05-27T07:36:52.329 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-05-27T07:36:52.329 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "pools which the automatic balancing will be limited to", 2024-05-27T07:36:52.329 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-05-27T07:36:52.329 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-05-27T07:36:52.329 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-05-27T07:36:52.329 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-05-27T07:36:52.329 INFO:teuthology.orchestra.run.smithi052.stdout: "sleep_interval": { 2024-05-27T07:36:52.329 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "sleep_interval", 2024-05-27T07:36:52.329 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "secs", 2024-05-27T07:36:52.330 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-05-27T07:36:52.330 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 1, 2024-05-27T07:36:52.330 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "60", 2024-05-27T07:36:52.330 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-05-27T07:36:52.330 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-05-27T07:36:52.330 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-05-27T07:36:52.330 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "how frequently to wake up and attempt optimization", 2024-05-27T07:36:52.330 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-05-27T07:36:52.330 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-05-27T07:36:52.330 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-05-27T07:36:52.330 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-05-27T07:36:52.330 INFO:teuthology.orchestra.run.smithi052.stdout: "sqlite3_killpoint": { 2024-05-27T07:36:52.330 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "sqlite3_killpoint", 2024-05-27T07:36:52.331 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "int", 2024-05-27T07:36:52.331 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "dev", 2024-05-27T07:36:52.331 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 1, 2024-05-27T07:36:52.331 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "0", 2024-05-27T07:36:52.331 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-05-27T07:36:52.331 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-05-27T07:36:52.331 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-05-27T07:36:52.331 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-05-27T07:36:52.331 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-05-27T07:36:52.331 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-05-27T07:36:52.331 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-05-27T07:36:52.331 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-05-27T07:36:52.332 INFO:teuthology.orchestra.run.smithi052.stdout: "upmap_max_deviation": { 2024-05-27T07:36:52.332 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "upmap_max_deviation", 2024-05-27T07:36:52.332 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "int", 2024-05-27T07:36:52.332 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-05-27T07:36:52.332 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 1, 2024-05-27T07:36:52.332 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "5", 2024-05-27T07:36:52.332 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "1", 2024-05-27T07:36:52.332 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-05-27T07:36:52.332 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-05-27T07:36:52.332 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "deviation below which no optimization is attempted", 2024-05-27T07:36:52.332 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "If the number of PGs are within this count then no optimization is attempted", 2024-05-27T07:36:52.332 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-05-27T07:36:52.332 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-05-27T07:36:52.333 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-05-27T07:36:52.333 INFO:teuthology.orchestra.run.smithi052.stdout: "upmap_max_optimizations": { 2024-05-27T07:36:52.333 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "upmap_max_optimizations", 2024-05-27T07:36:52.333 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "uint", 2024-05-27T07:36:52.333 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-05-27T07:36:52.333 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 1, 2024-05-27T07:36:52.333 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "10", 2024-05-27T07:36:52.333 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-05-27T07:36:52.333 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-05-27T07:36:52.333 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-05-27T07:36:52.333 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "maximum upmap optimizations to make per attempt", 2024-05-27T07:36:52.333 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-05-27T07:36:52.334 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-05-27T07:36:52.334 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-05-27T07:36:52.334 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-05-27T07:36:52.334 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-05-27T07:36:52.334 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-05-27T07:36:52.334 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-05-27T07:36:52.334 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "cephadm", 2024-05-27T07:36:52.334 INFO:teuthology.orchestra.run.smithi052.stdout: "can_run": true, 2024-05-27T07:36:52.334 INFO:teuthology.orchestra.run.smithi052.stdout: "error_string": "", 2024-05-27T07:36:52.334 INFO:teuthology.orchestra.run.smithi052.stdout: "module_options": { 2024-05-27T07:36:52.334 INFO:teuthology.orchestra.run.smithi052.stdout: "agent_down_multiplier": { 2024-05-27T07:36:52.334 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "agent_down_multiplier", 2024-05-27T07:36:52.334 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "float", 2024-05-27T07:36:52.335 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-05-27T07:36:52.335 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 0, 2024-05-27T07:36:52.335 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "3.0", 2024-05-27T07:36:52.335 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-05-27T07:36:52.335 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-05-27T07:36:52.335 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-05-27T07:36:52.335 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "Multiplied by agent refresh rate to calculate how long agent must not report before being marked down", 2024-05-27T07:36:52.335 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-05-27T07:36:52.335 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-05-27T07:36:52.335 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-05-27T07:36:52.335 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-05-27T07:36:52.335 INFO:teuthology.orchestra.run.smithi052.stdout: "agent_refresh_rate": { 2024-05-27T07:36:52.335 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "agent_refresh_rate", 2024-05-27T07:36:52.336 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "secs", 2024-05-27T07:36:52.336 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-05-27T07:36:52.336 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 0, 2024-05-27T07:36:52.336 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "20", 2024-05-27T07:36:52.336 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-05-27T07:36:52.336 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-05-27T07:36:52.336 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-05-27T07:36:52.336 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "How often agent on each host will try to gather and send metadata", 2024-05-27T07:36:52.336 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-05-27T07:36:52.336 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-05-27T07:36:52.336 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-05-27T07:36:52.336 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-05-27T07:36:52.336 INFO:teuthology.orchestra.run.smithi052.stdout: "agent_starting_port": { 2024-05-27T07:36:52.337 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "agent_starting_port", 2024-05-27T07:36:52.337 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "int", 2024-05-27T07:36:52.337 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-05-27T07:36:52.337 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 0, 2024-05-27T07:36:52.337 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "4721", 2024-05-27T07:36:52.337 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-05-27T07:36:52.337 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-05-27T07:36:52.337 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-05-27T07:36:52.337 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)", 2024-05-27T07:36:52.337 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-05-27T07:36:52.337 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-05-27T07:36:52.337 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-05-27T07:36:52.338 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-05-27T07:36:52.338 INFO:teuthology.orchestra.run.smithi052.stdout: "allow_ptrace": { 2024-05-27T07:36:52.338 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "allow_ptrace", 2024-05-27T07:36:52.338 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "bool", 2024-05-27T07:36:52.338 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-05-27T07:36:52.338 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 0, 2024-05-27T07:36:52.338 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "False", 2024-05-27T07:36:52.338 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-05-27T07:36:52.338 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-05-27T07:36:52.338 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-05-27T07:36:52.338 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "allow SYS_PTRACE capability on ceph containers", 2024-05-27T07:36:52.338 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.", 2024-05-27T07:36:52.339 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-05-27T07:36:52.339 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-05-27T07:36:52.339 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-05-27T07:36:52.339 INFO:teuthology.orchestra.run.smithi052.stdout: "autotune_interval": { 2024-05-27T07:36:52.339 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "autotune_interval", 2024-05-27T07:36:52.339 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "secs", 2024-05-27T07:36:52.339 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-05-27T07:36:52.339 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 0, 2024-05-27T07:36:52.339 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "600", 2024-05-27T07:36:52.339 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-05-27T07:36:52.339 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-05-27T07:36:52.339 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-05-27T07:36:52.340 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "how frequently to autotune daemon memory", 2024-05-27T07:36:52.340 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-05-27T07:36:52.340 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-05-27T07:36:52.340 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-05-27T07:36:52.340 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-05-27T07:36:52.340 INFO:teuthology.orchestra.run.smithi052.stdout: "autotune_memory_target_ratio": { 2024-05-27T07:36:52.340 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "autotune_memory_target_ratio", 2024-05-27T07:36:52.340 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "float", 2024-05-27T07:36:52.340 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-05-27T07:36:52.340 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 0, 2024-05-27T07:36:52.340 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "0.7", 2024-05-27T07:36:52.340 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-05-27T07:36:52.340 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-05-27T07:36:52.341 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-05-27T07:36:52.341 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "ratio of total system memory to divide amongst autotuned daemons", 2024-05-27T07:36:52.341 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-05-27T07:36:52.341 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-05-27T07:36:52.341 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-05-27T07:36:52.341 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-05-27T07:36:52.341 INFO:teuthology.orchestra.run.smithi052.stdout: "cephadm_log_destination": { 2024-05-27T07:36:52.341 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "cephadm_log_destination", 2024-05-27T07:36:52.341 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "str", 2024-05-27T07:36:52.341 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-05-27T07:36:52.341 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 0, 2024-05-27T07:36:52.341 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "", 2024-05-27T07:36:52.341 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-05-27T07:36:52.342 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-05-27T07:36:52.342 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [ 2024-05-27T07:36:52.342 INFO:teuthology.orchestra.run.smithi052.stdout: "file", 2024-05-27T07:36:52.342 INFO:teuthology.orchestra.run.smithi052.stdout: "file,syslog", 2024-05-27T07:36:52.342 INFO:teuthology.orchestra.run.smithi052.stdout: "syslog" 2024-05-27T07:36:52.342 INFO:teuthology.orchestra.run.smithi052.stdout: ], 2024-05-27T07:36:52.342 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "Destination for cephadm command's persistent logging", 2024-05-27T07:36:52.342 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-05-27T07:36:52.342 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-05-27T07:36:52.342 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-05-27T07:36:52.342 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-05-27T07:36:52.342 INFO:teuthology.orchestra.run.smithi052.stdout: "cgroups_split": { 2024-05-27T07:36:52.343 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "cgroups_split", 2024-05-27T07:36:52.343 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "bool", 2024-05-27T07:36:52.343 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-05-27T07:36:52.343 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 0, 2024-05-27T07:36:52.343 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "True", 2024-05-27T07:36:52.343 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-05-27T07:36:52.343 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-05-27T07:36:52.343 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-05-27T07:36:52.343 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "Pass --cgroups=split when cephadm creates containers (currently podman only)", 2024-05-27T07:36:52.343 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-05-27T07:36:52.343 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-05-27T07:36:52.343 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-05-27T07:36:52.343 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-05-27T07:36:52.344 INFO:teuthology.orchestra.run.smithi052.stdout: "config_checks_enabled": { 2024-05-27T07:36:52.344 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "config_checks_enabled", 2024-05-27T07:36:52.344 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "bool", 2024-05-27T07:36:52.344 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-05-27T07:36:52.344 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 0, 2024-05-27T07:36:52.344 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "False", 2024-05-27T07:36:52.344 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-05-27T07:36:52.344 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-05-27T07:36:52.344 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-05-27T07:36:52.344 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "Enable or disable the cephadm configuration analysis", 2024-05-27T07:36:52.344 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-05-27T07:36:52.344 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-05-27T07:36:52.344 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-05-27T07:36:52.345 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-05-27T07:36:52.345 INFO:teuthology.orchestra.run.smithi052.stdout: "config_dashboard": { 2024-05-27T07:36:52.345 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "config_dashboard", 2024-05-27T07:36:52.345 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "bool", 2024-05-27T07:36:52.345 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-05-27T07:36:52.345 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 0, 2024-05-27T07:36:52.345 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "True", 2024-05-27T07:36:52.345 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-05-27T07:36:52.345 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-05-27T07:36:52.345 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-05-27T07:36:52.345 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "manage configs like API endpoints in Dashboard.", 2024-05-27T07:36:52.345 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-05-27T07:36:52.345 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-05-27T07:36:52.345 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-05-27T07:36:52.346 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-05-27T07:36:52.346 INFO:teuthology.orchestra.run.smithi052.stdout: "container_image_alertmanager": { 2024-05-27T07:36:52.346 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "container_image_alertmanager", 2024-05-27T07:36:52.346 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "str", 2024-05-27T07:36:52.346 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-05-27T07:36:52.346 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 0, 2024-05-27T07:36:52.346 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "quay.io/prometheus/alertmanager:v0.25.0", 2024-05-27T07:36:52.346 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-05-27T07:36:52.346 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-05-27T07:36:52.346 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-05-27T07:36:52.346 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "Prometheus container image", 2024-05-27T07:36:52.347 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-05-27T07:36:52.347 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-05-27T07:36:52.347 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-05-27T07:36:52.347 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-05-27T07:36:52.347 INFO:teuthology.orchestra.run.smithi052.stdout: "container_image_base": { 2024-05-27T07:36:52.347 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "container_image_base", 2024-05-27T07:36:52.347 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "str", 2024-05-27T07:36:52.347 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-05-27T07:36:52.347 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 1, 2024-05-27T07:36:52.347 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "quay.io/ceph/ceph", 2024-05-27T07:36:52.347 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-05-27T07:36:52.347 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-05-27T07:36:52.347 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-05-27T07:36:52.348 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "Container image name, without the tag", 2024-05-27T07:36:52.348 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-05-27T07:36:52.348 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-05-27T07:36:52.348 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-05-27T07:36:52.348 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-05-27T07:36:52.348 INFO:teuthology.orchestra.run.smithi052.stdout: "container_image_elasticsearch": { 2024-05-27T07:36:52.348 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "container_image_elasticsearch", 2024-05-27T07:36:52.348 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "str", 2024-05-27T07:36:52.348 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-05-27T07:36:52.348 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 0, 2024-05-27T07:36:52.348 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "quay.io/omrizeneva/elasticsearch:6.8.23", 2024-05-27T07:36:52.348 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-05-27T07:36:52.349 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-05-27T07:36:52.349 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-05-27T07:36:52.349 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "elasticsearch container image", 2024-05-27T07:36:52.349 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-05-27T07:36:52.349 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-05-27T07:36:52.349 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-05-27T07:36:52.349 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-05-27T07:36:52.349 INFO:teuthology.orchestra.run.smithi052.stdout: "container_image_grafana": { 2024-05-27T07:36:52.349 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "container_image_grafana", 2024-05-27T07:36:52.349 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "str", 2024-05-27T07:36:52.349 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-05-27T07:36:52.349 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 0, 2024-05-27T07:36:52.350 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "quay.io/ceph/grafana:9.4.12", 2024-05-27T07:36:52.350 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-05-27T07:36:52.350 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-05-27T07:36:52.350 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-05-27T07:36:52.350 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "Prometheus container image", 2024-05-27T07:36:52.350 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-05-27T07:36:52.350 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-05-27T07:36:52.350 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-05-27T07:36:52.350 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-05-27T07:36:52.350 INFO:teuthology.orchestra.run.smithi052.stdout: "container_image_haproxy": { 2024-05-27T07:36:52.350 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "container_image_haproxy", 2024-05-27T07:36:52.350 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "str", 2024-05-27T07:36:52.351 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-05-27T07:36:52.351 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 0, 2024-05-27T07:36:52.351 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "quay.io/ceph/haproxy:2.3", 2024-05-27T07:36:52.351 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-05-27T07:36:52.351 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-05-27T07:36:52.351 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-05-27T07:36:52.351 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "HAproxy container image", 2024-05-27T07:36:52.351 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-05-27T07:36:52.351 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-05-27T07:36:52.351 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-05-27T07:36:52.351 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-05-27T07:36:52.352 INFO:teuthology.orchestra.run.smithi052.stdout: "container_image_jaeger_agent": { 2024-05-27T07:36:52.352 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "container_image_jaeger_agent", 2024-05-27T07:36:52.352 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "str", 2024-05-27T07:36:52.352 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-05-27T07:36:52.352 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 0, 2024-05-27T07:36:52.352 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "quay.io/jaegertracing/jaeger-agent:1.29", 2024-05-27T07:36:52.352 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-05-27T07:36:52.352 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-05-27T07:36:52.352 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-05-27T07:36:52.352 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "Jaeger agent container image", 2024-05-27T07:36:52.352 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-05-27T07:36:52.352 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-05-27T07:36:52.353 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-05-27T07:36:52.353 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-05-27T07:36:52.353 INFO:teuthology.orchestra.run.smithi052.stdout: "container_image_jaeger_collector": { 2024-05-27T07:36:52.353 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "container_image_jaeger_collector", 2024-05-27T07:36:52.353 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "str", 2024-05-27T07:36:52.353 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-05-27T07:36:52.353 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 0, 2024-05-27T07:36:52.353 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "quay.io/jaegertracing/jaeger-collector:1.29", 2024-05-27T07:36:52.353 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-05-27T07:36:52.353 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-05-27T07:36:52.353 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-05-27T07:36:52.353 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "Jaeger collector container image", 2024-05-27T07:36:52.353 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-05-27T07:36:52.354 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-05-27T07:36:52.354 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-05-27T07:36:52.354 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-05-27T07:36:52.354 INFO:teuthology.orchestra.run.smithi052.stdout: "container_image_jaeger_query": { 2024-05-27T07:36:52.354 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "container_image_jaeger_query", 2024-05-27T07:36:52.354 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "str", 2024-05-27T07:36:52.354 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-05-27T07:36:52.354 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 0, 2024-05-27T07:36:52.354 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "quay.io/jaegertracing/jaeger-query:1.29", 2024-05-27T07:36:52.354 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-05-27T07:36:52.354 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-05-27T07:36:52.354 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-05-27T07:36:52.354 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "Jaeger query container image", 2024-05-27T07:36:52.355 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-05-27T07:36:52.355 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-05-27T07:36:52.355 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-05-27T07:36:52.355 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-05-27T07:36:52.355 INFO:teuthology.orchestra.run.smithi052.stdout: "container_image_keepalived": { 2024-05-27T07:36:52.355 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "container_image_keepalived", 2024-05-27T07:36:52.355 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "str", 2024-05-27T07:36:52.355 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-05-27T07:36:52.355 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 0, 2024-05-27T07:36:52.355 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "quay.io/ceph/keepalived:2.2.4", 2024-05-27T07:36:52.355 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-05-27T07:36:52.355 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-05-27T07:36:52.355 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-05-27T07:36:52.356 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "Keepalived container image", 2024-05-27T07:36:52.356 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-05-27T07:36:52.356 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-05-27T07:36:52.356 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-05-27T07:36:52.356 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-05-27T07:36:52.356 INFO:teuthology.orchestra.run.smithi052.stdout: "container_image_loki": { 2024-05-27T07:36:52.356 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "container_image_loki", 2024-05-27T07:36:52.356 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "str", 2024-05-27T07:36:52.356 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-05-27T07:36:52.356 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 0, 2024-05-27T07:36:52.356 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "docker.io/grafana/loki:3.0.0", 2024-05-27T07:36:52.356 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-05-27T07:36:52.356 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-05-27T07:36:52.357 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-05-27T07:36:52.357 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "Loki container image", 2024-05-27T07:36:52.357 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-05-27T07:36:52.357 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-05-27T07:36:52.357 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-05-27T07:36:52.357 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-05-27T07:36:52.357 INFO:teuthology.orchestra.run.smithi052.stdout: "container_image_node_exporter": { 2024-05-27T07:36:52.357 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "container_image_node_exporter", 2024-05-27T07:36:52.357 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "str", 2024-05-27T07:36:52.357 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-05-27T07:36:52.357 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 0, 2024-05-27T07:36:52.357 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "quay.io/prometheus/node-exporter:v1.5.0", 2024-05-27T07:36:52.357 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-05-27T07:36:52.358 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-05-27T07:36:52.358 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-05-27T07:36:52.358 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "Prometheus container image", 2024-05-27T07:36:52.358 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-05-27T07:36:52.358 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-05-27T07:36:52.358 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-05-27T07:36:52.358 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-05-27T07:36:52.358 INFO:teuthology.orchestra.run.smithi052.stdout: "container_image_nvmeof": { 2024-05-27T07:36:52.358 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "container_image_nvmeof", 2024-05-27T07:36:52.358 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "str", 2024-05-27T07:36:52.358 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-05-27T07:36:52.358 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 0, 2024-05-27T07:36:52.358 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "quay.io/ceph/nvmeof:1.2.5", 2024-05-27T07:36:52.359 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-05-27T07:36:52.359 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-05-27T07:36:52.359 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-05-27T07:36:52.359 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "Nvme-of container image", 2024-05-27T07:36:52.359 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-05-27T07:36:52.359 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-05-27T07:36:52.359 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-05-27T07:36:52.359 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-05-27T07:36:52.359 INFO:teuthology.orchestra.run.smithi052.stdout: "container_image_prometheus": { 2024-05-27T07:36:52.359 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "container_image_prometheus", 2024-05-27T07:36:52.359 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "str", 2024-05-27T07:36:52.359 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-05-27T07:36:52.360 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 0, 2024-05-27T07:36:52.360 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "quay.io/prometheus/prometheus:v2.43.0", 2024-05-27T07:36:52.360 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-05-27T07:36:52.360 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-05-27T07:36:52.360 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-05-27T07:36:52.360 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "Prometheus container image", 2024-05-27T07:36:52.360 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-05-27T07:36:52.360 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-05-27T07:36:52.360 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-05-27T07:36:52.360 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-05-27T07:36:52.360 INFO:teuthology.orchestra.run.smithi052.stdout: "container_image_promtail": { 2024-05-27T07:36:52.360 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "container_image_promtail", 2024-05-27T07:36:52.360 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "str", 2024-05-27T07:36:52.361 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-05-27T07:36:52.361 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 0, 2024-05-27T07:36:52.361 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "docker.io/grafana/promtail:3.0.0", 2024-05-27T07:36:52.361 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-05-27T07:36:52.361 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-05-27T07:36:52.361 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-05-27T07:36:52.361 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "Promtail container image", 2024-05-27T07:36:52.361 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-05-27T07:36:52.361 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-05-27T07:36:52.361 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-05-27T07:36:52.361 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-05-27T07:36:52.361 INFO:teuthology.orchestra.run.smithi052.stdout: "container_image_samba": { 2024-05-27T07:36:52.361 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "container_image_samba", 2024-05-27T07:36:52.362 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "str", 2024-05-27T07:36:52.362 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-05-27T07:36:52.362 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 0, 2024-05-27T07:36:52.362 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "quay.io/samba.org/samba-server:devbuilds-centos-amd64", 2024-05-27T07:36:52.362 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-05-27T07:36:52.362 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-05-27T07:36:52.362 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-05-27T07:36:52.362 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "Samba/SMB container image", 2024-05-27T07:36:52.362 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-05-27T07:36:52.362 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-05-27T07:36:52.362 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-05-27T07:36:52.362 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-05-27T07:36:52.363 INFO:teuthology.orchestra.run.smithi052.stdout: "container_image_snmp_gateway": { 2024-05-27T07:36:52.363 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "container_image_snmp_gateway", 2024-05-27T07:36:52.363 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "str", 2024-05-27T07:36:52.363 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-05-27T07:36:52.363 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 0, 2024-05-27T07:36:52.363 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "docker.io/maxwo/snmp-notifier:v1.2.1", 2024-05-27T07:36:52.363 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-05-27T07:36:52.363 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-05-27T07:36:52.363 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-05-27T07:36:52.363 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "SNMP Gateway container image", 2024-05-27T07:36:52.363 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-05-27T07:36:52.363 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-05-27T07:36:52.364 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-05-27T07:36:52.364 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-05-27T07:36:52.364 INFO:teuthology.orchestra.run.smithi052.stdout: "container_init": { 2024-05-27T07:36:52.364 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "container_init", 2024-05-27T07:36:52.364 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "bool", 2024-05-27T07:36:52.364 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-05-27T07:36:52.364 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 0, 2024-05-27T07:36:52.364 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "True", 2024-05-27T07:36:52.364 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-05-27T07:36:52.364 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-05-27T07:36:52.364 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-05-27T07:36:52.364 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "Run podman/docker with `--init`", 2024-05-27T07:36:52.364 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-05-27T07:36:52.365 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-05-27T07:36:52.365 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-05-27T07:36:52.365 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-05-27T07:36:52.365 INFO:teuthology.orchestra.run.smithi052.stdout: "daemon_cache_timeout": { 2024-05-27T07:36:52.365 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "daemon_cache_timeout", 2024-05-27T07:36:52.365 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "secs", 2024-05-27T07:36:52.365 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-05-27T07:36:52.365 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 0, 2024-05-27T07:36:52.365 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "600", 2024-05-27T07:36:52.365 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-05-27T07:36:52.365 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-05-27T07:36:52.365 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-05-27T07:36:52.365 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "seconds to cache service (daemon) inventory", 2024-05-27T07:36:52.366 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-05-27T07:36:52.366 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-05-27T07:36:52.366 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-05-27T07:36:52.366 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-05-27T07:36:52.366 INFO:teuthology.orchestra.run.smithi052.stdout: "default_cephadm_command_timeout": { 2024-05-27T07:36:52.366 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "default_cephadm_command_timeout", 2024-05-27T07:36:52.366 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "int", 2024-05-27T07:36:52.369 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-05-27T07:36:52.369 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 0, 2024-05-27T07:36:52.369 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "900", 2024-05-27T07:36:52.369 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-05-27T07:36:52.369 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-05-27T07:36:52.370 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-05-27T07:36:52.370 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "Default timeout applied to cephadm commands run directly on the host (in seconds)", 2024-05-27T07:36:52.370 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-05-27T07:36:52.370 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-05-27T07:36:52.370 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-05-27T07:36:52.370 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-05-27T07:36:52.370 INFO:teuthology.orchestra.run.smithi052.stdout: "default_registry": { 2024-05-27T07:36:52.370 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "default_registry", 2024-05-27T07:36:52.370 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "str", 2024-05-27T07:36:52.370 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-05-27T07:36:52.370 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 0, 2024-05-27T07:36:52.370 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "docker.io", 2024-05-27T07:36:52.370 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-05-27T07:36:52.371 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-05-27T07:36:52.371 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-05-27T07:36:52.371 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "Search-registry to which we should normalize unqualified image names. This is not the default registry", 2024-05-27T07:36:52.371 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-05-27T07:36:52.371 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-05-27T07:36:52.371 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-05-27T07:36:52.371 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-05-27T07:36:52.371 INFO:teuthology.orchestra.run.smithi052.stdout: "device_cache_timeout": { 2024-05-27T07:36:52.371 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "device_cache_timeout", 2024-05-27T07:36:52.371 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "secs", 2024-05-27T07:36:52.371 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-05-27T07:36:52.371 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 0, 2024-05-27T07:36:52.372 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "1800", 2024-05-27T07:36:52.372 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-05-27T07:36:52.372 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-05-27T07:36:52.372 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-05-27T07:36:52.372 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "seconds to cache device inventory", 2024-05-27T07:36:52.372 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-05-27T07:36:52.372 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-05-27T07:36:52.372 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-05-27T07:36:52.372 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-05-27T07:36:52.372 INFO:teuthology.orchestra.run.smithi052.stdout: "device_enhanced_scan": { 2024-05-27T07:36:52.372 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "device_enhanced_scan", 2024-05-27T07:36:52.372 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "bool", 2024-05-27T07:36:52.372 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-05-27T07:36:52.373 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 0, 2024-05-27T07:36:52.373 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "False", 2024-05-27T07:36:52.373 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-05-27T07:36:52.373 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-05-27T07:36:52.373 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-05-27T07:36:52.373 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "Use libstoragemgmt during device scans", 2024-05-27T07:36:52.373 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-05-27T07:36:52.373 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-05-27T07:36:52.373 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-05-27T07:36:52.373 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-05-27T07:36:52.373 INFO:teuthology.orchestra.run.smithi052.stdout: "facts_cache_timeout": { 2024-05-27T07:36:52.373 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "facts_cache_timeout", 2024-05-27T07:36:52.373 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "secs", 2024-05-27T07:36:52.374 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-05-27T07:36:52.374 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 0, 2024-05-27T07:36:52.374 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "60", 2024-05-27T07:36:52.374 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-05-27T07:36:52.374 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-05-27T07:36:52.374 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-05-27T07:36:52.374 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "seconds to cache host facts data", 2024-05-27T07:36:52.374 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-05-27T07:36:52.374 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-05-27T07:36:52.374 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-05-27T07:36:52.374 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-05-27T07:36:52.374 INFO:teuthology.orchestra.run.smithi052.stdout: "grafana_dashboards_path": { 2024-05-27T07:36:52.375 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "grafana_dashboards_path", 2024-05-27T07:36:52.375 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "str", 2024-05-27T07:36:52.375 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-05-27T07:36:52.375 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 0, 2024-05-27T07:36:52.375 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "/etc/grafana/dashboards/ceph-dashboard/", 2024-05-27T07:36:52.375 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-05-27T07:36:52.375 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-05-27T07:36:52.375 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-05-27T07:36:52.375 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "location of dashboards to include in grafana deployments", 2024-05-27T07:36:52.375 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-05-27T07:36:52.375 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-05-27T07:36:52.375 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-05-27T07:36:52.375 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-05-27T07:36:52.376 INFO:teuthology.orchestra.run.smithi052.stdout: "host_check_interval": { 2024-05-27T07:36:52.376 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "host_check_interval", 2024-05-27T07:36:52.376 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "secs", 2024-05-27T07:36:52.376 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-05-27T07:36:52.376 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 0, 2024-05-27T07:36:52.376 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "600", 2024-05-27T07:36:52.376 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-05-27T07:36:52.376 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-05-27T07:36:52.376 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-05-27T07:36:52.376 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "how frequently to perform a host check", 2024-05-27T07:36:52.376 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-05-27T07:36:52.377 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-05-27T07:36:52.377 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-05-27T07:36:52.377 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-05-27T07:36:52.377 INFO:teuthology.orchestra.run.smithi052.stdout: "hw_monitoring": { 2024-05-27T07:36:52.377 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "hw_monitoring", 2024-05-27T07:36:52.377 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "bool", 2024-05-27T07:36:52.377 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-05-27T07:36:52.377 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 0, 2024-05-27T07:36:52.377 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "False", 2024-05-27T07:36:52.377 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-05-27T07:36:52.377 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-05-27T07:36:52.377 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-05-27T07:36:52.377 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "Deploy hw monitoring daemon on every host.", 2024-05-27T07:36:52.378 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-05-27T07:36:52.378 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-05-27T07:36:52.378 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-05-27T07:36:52.378 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-05-27T07:36:52.378 INFO:teuthology.orchestra.run.smithi052.stdout: "inventory_list_all": { 2024-05-27T07:36:52.378 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "inventory_list_all", 2024-05-27T07:36:52.378 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "bool", 2024-05-27T07:36:52.378 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-05-27T07:36:52.378 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 0, 2024-05-27T07:36:52.378 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "False", 2024-05-27T07:36:52.378 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-05-27T07:36:52.378 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-05-27T07:36:52.379 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-05-27T07:36:52.379 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)", 2024-05-27T07:36:52.379 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-05-27T07:36:52.379 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-05-27T07:36:52.379 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-05-27T07:36:52.379 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-05-27T07:36:52.379 INFO:teuthology.orchestra.run.smithi052.stdout: "log_level": { 2024-05-27T07:36:52.379 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "log_level", 2024-05-27T07:36:52.379 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "str", 2024-05-27T07:36:52.379 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-05-27T07:36:52.379 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 1, 2024-05-27T07:36:52.379 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "", 2024-05-27T07:36:52.380 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-05-27T07:36:52.380 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-05-27T07:36:52.380 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [ 2024-05-27T07:36:52.380 INFO:teuthology.orchestra.run.smithi052.stdout: "", 2024-05-27T07:36:52.380 INFO:teuthology.orchestra.run.smithi052.stdout: "critical", 2024-05-27T07:36:52.380 INFO:teuthology.orchestra.run.smithi052.stdout: "debug", 2024-05-27T07:36:52.380 INFO:teuthology.orchestra.run.smithi052.stdout: "error", 2024-05-27T07:36:52.380 INFO:teuthology.orchestra.run.smithi052.stdout: "info", 2024-05-27T07:36:52.380 INFO:teuthology.orchestra.run.smithi052.stdout: "warning" 2024-05-27T07:36:52.380 INFO:teuthology.orchestra.run.smithi052.stdout: ], 2024-05-27T07:36:52.380 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-05-27T07:36:52.380 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-05-27T07:36:52.380 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-05-27T07:36:52.381 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-05-27T07:36:52.381 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-05-27T07:36:52.381 INFO:teuthology.orchestra.run.smithi052.stdout: "log_refresh_metadata": { 2024-05-27T07:36:52.381 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "log_refresh_metadata", 2024-05-27T07:36:52.381 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "bool", 2024-05-27T07:36:52.381 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-05-27T07:36:52.381 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 0, 2024-05-27T07:36:52.381 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "False", 2024-05-27T07:36:52.381 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-05-27T07:36:52.381 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-05-27T07:36:52.381 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-05-27T07:36:52.381 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level", 2024-05-27T07:36:52.381 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-05-27T07:36:52.382 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-05-27T07:36:52.382 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-05-27T07:36:52.382 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-05-27T07:36:52.382 INFO:teuthology.orchestra.run.smithi052.stdout: "log_to_cluster": { 2024-05-27T07:36:52.382 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "log_to_cluster", 2024-05-27T07:36:52.382 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "bool", 2024-05-27T07:36:52.382 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-05-27T07:36:52.382 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 0, 2024-05-27T07:36:52.382 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "True", 2024-05-27T07:36:52.382 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-05-27T07:36:52.382 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-05-27T07:36:52.382 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-05-27T07:36:52.383 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "log to the \"cephadm\" cluster log channel\"", 2024-05-27T07:36:52.383 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-05-27T07:36:52.383 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-05-27T07:36:52.383 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-05-27T07:36:52.383 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-05-27T07:36:52.383 INFO:teuthology.orchestra.run.smithi052.stdout: "log_to_cluster_level": { 2024-05-27T07:36:52.383 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "log_to_cluster_level", 2024-05-27T07:36:52.383 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "str", 2024-05-27T07:36:52.383 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-05-27T07:36:52.383 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 1, 2024-05-27T07:36:52.383 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "info", 2024-05-27T07:36:52.383 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-05-27T07:36:52.383 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-05-27T07:36:52.384 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [ 2024-05-27T07:36:52.384 INFO:teuthology.orchestra.run.smithi052.stdout: "", 2024-05-27T07:36:52.384 INFO:teuthology.orchestra.run.smithi052.stdout: "critical", 2024-05-27T07:36:52.384 INFO:teuthology.orchestra.run.smithi052.stdout: "debug", 2024-05-27T07:36:52.384 INFO:teuthology.orchestra.run.smithi052.stdout: "error", 2024-05-27T07:36:52.384 INFO:teuthology.orchestra.run.smithi052.stdout: "info", 2024-05-27T07:36:52.384 INFO:teuthology.orchestra.run.smithi052.stdout: "warning" 2024-05-27T07:36:52.384 INFO:teuthology.orchestra.run.smithi052.stdout: ], 2024-05-27T07:36:52.384 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-05-27T07:36:52.384 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-05-27T07:36:52.384 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-05-27T07:36:52.384 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-05-27T07:36:52.384 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-05-27T07:36:52.384 INFO:teuthology.orchestra.run.smithi052.stdout: "log_to_file": { 2024-05-27T07:36:52.385 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "log_to_file", 2024-05-27T07:36:52.385 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "bool", 2024-05-27T07:36:52.385 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-05-27T07:36:52.385 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 1, 2024-05-27T07:36:52.385 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "False", 2024-05-27T07:36:52.385 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-05-27T07:36:52.385 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-05-27T07:36:52.385 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-05-27T07:36:52.385 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-05-27T07:36:52.385 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-05-27T07:36:52.385 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-05-27T07:36:52.385 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-05-27T07:36:52.385 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-05-27T07:36:52.386 INFO:teuthology.orchestra.run.smithi052.stdout: "manage_etc_ceph_ceph_conf": { 2024-05-27T07:36:52.386 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "manage_etc_ceph_ceph_conf", 2024-05-27T07:36:52.386 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "bool", 2024-05-27T07:36:52.386 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-05-27T07:36:52.386 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 0, 2024-05-27T07:36:52.386 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "False", 2024-05-27T07:36:52.386 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-05-27T07:36:52.386 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-05-27T07:36:52.386 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-05-27T07:36:52.386 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "Manage and own /etc/ceph/ceph.conf on the hosts.", 2024-05-27T07:36:52.386 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-05-27T07:36:52.386 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-05-27T07:36:52.387 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-05-27T07:36:52.387 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-05-27T07:36:52.387 INFO:teuthology.orchestra.run.smithi052.stdout: "manage_etc_ceph_ceph_conf_hosts": { 2024-05-27T07:36:52.387 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "manage_etc_ceph_ceph_conf_hosts", 2024-05-27T07:36:52.387 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "str", 2024-05-27T07:36:52.387 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-05-27T07:36:52.387 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 0, 2024-05-27T07:36:52.387 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "*", 2024-05-27T07:36:52.387 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-05-27T07:36:52.387 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-05-27T07:36:52.387 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-05-27T07:36:52.387 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf", 2024-05-27T07:36:52.387 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-05-27T07:36:52.388 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-05-27T07:36:52.388 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-05-27T07:36:52.388 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-05-27T07:36:52.388 INFO:teuthology.orchestra.run.smithi052.stdout: "max_count_per_host": { 2024-05-27T07:36:52.388 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "max_count_per_host", 2024-05-27T07:36:52.388 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "int", 2024-05-27T07:36:52.388 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-05-27T07:36:52.388 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 0, 2024-05-27T07:36:52.388 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "10", 2024-05-27T07:36:52.388 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-05-27T07:36:52.388 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-05-27T07:36:52.388 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-05-27T07:36:52.388 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "max number of daemons per service per host", 2024-05-27T07:36:52.388 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-05-27T07:36:52.389 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-05-27T07:36:52.389 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-05-27T07:36:52.389 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-05-27T07:36:52.389 INFO:teuthology.orchestra.run.smithi052.stdout: "max_osd_draining_count": { 2024-05-27T07:36:52.389 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "max_osd_draining_count", 2024-05-27T07:36:52.389 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "int", 2024-05-27T07:36:52.389 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-05-27T07:36:52.389 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 0, 2024-05-27T07:36:52.389 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "10", 2024-05-27T07:36:52.389 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-05-27T07:36:52.389 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-05-27T07:36:52.389 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-05-27T07:36:52.389 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "max number of osds that will be drained simultaneously when osds are removed", 2024-05-27T07:36:52.389 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-05-27T07:36:52.390 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-05-27T07:36:52.390 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-05-27T07:36:52.390 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-05-27T07:36:52.390 INFO:teuthology.orchestra.run.smithi052.stdout: "migration_current": { 2024-05-27T07:36:52.390 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "migration_current", 2024-05-27T07:36:52.390 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "int", 2024-05-27T07:36:52.390 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-05-27T07:36:52.390 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 0, 2024-05-27T07:36:52.390 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "", 2024-05-27T07:36:52.390 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-05-27T07:36:52.390 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-05-27T07:36:52.390 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-05-27T07:36:52.391 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "internal - do not modify", 2024-05-27T07:36:52.391 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-05-27T07:36:52.391 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-05-27T07:36:52.391 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-05-27T07:36:52.391 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-05-27T07:36:52.391 INFO:teuthology.orchestra.run.smithi052.stdout: "mode": { 2024-05-27T07:36:52.391 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "mode", 2024-05-27T07:36:52.391 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "str", 2024-05-27T07:36:52.391 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-05-27T07:36:52.391 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 0, 2024-05-27T07:36:52.391 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "root", 2024-05-27T07:36:52.391 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-05-27T07:36:52.391 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-05-27T07:36:52.392 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [ 2024-05-27T07:36:52.392 INFO:teuthology.orchestra.run.smithi052.stdout: "cephadm-package", 2024-05-27T07:36:52.392 INFO:teuthology.orchestra.run.smithi052.stdout: "root" 2024-05-27T07:36:52.392 INFO:teuthology.orchestra.run.smithi052.stdout: ], 2024-05-27T07:36:52.392 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "mode for remote execution of cephadm", 2024-05-27T07:36:52.392 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-05-27T07:36:52.392 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-05-27T07:36:52.392 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-05-27T07:36:52.392 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-05-27T07:36:52.392 INFO:teuthology.orchestra.run.smithi052.stdout: "oob_default_addr": { 2024-05-27T07:36:52.392 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "oob_default_addr", 2024-05-27T07:36:52.392 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "str", 2024-05-27T07:36:52.392 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-05-27T07:36:52.393 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 0, 2024-05-27T07:36:52.393 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "169.254.1.1", 2024-05-27T07:36:52.393 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-05-27T07:36:52.393 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-05-27T07:36:52.393 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-05-27T07:36:52.393 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "Default address for RedFish API (oob management).", 2024-05-27T07:36:52.393 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-05-27T07:36:52.393 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-05-27T07:36:52.393 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-05-27T07:36:52.393 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-05-27T07:36:52.393 INFO:teuthology.orchestra.run.smithi052.stdout: "prometheus_alerts_path": { 2024-05-27T07:36:52.393 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "prometheus_alerts_path", 2024-05-27T07:36:52.393 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "str", 2024-05-27T07:36:52.393 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-05-27T07:36:52.394 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 0, 2024-05-27T07:36:52.394 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "/etc/prometheus/ceph/ceph_default_alerts.yml", 2024-05-27T07:36:52.394 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-05-27T07:36:52.394 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-05-27T07:36:52.394 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-05-27T07:36:52.394 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "location of alerts to include in prometheus deployments", 2024-05-27T07:36:52.394 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-05-27T07:36:52.394 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-05-27T07:36:52.394 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-05-27T07:36:52.394 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-05-27T07:36:52.394 INFO:teuthology.orchestra.run.smithi052.stdout: "registry_insecure": { 2024-05-27T07:36:52.394 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "registry_insecure", 2024-05-27T07:36:52.395 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "bool", 2024-05-27T07:36:52.395 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-05-27T07:36:52.395 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 0, 2024-05-27T07:36:52.395 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "False", 2024-05-27T07:36:52.395 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-05-27T07:36:52.395 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-05-27T07:36:52.395 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-05-27T07:36:52.395 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "Registry is to be considered insecure (no TLS available). Only for development purposes.", 2024-05-27T07:36:52.395 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-05-27T07:36:52.395 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-05-27T07:36:52.395 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-05-27T07:36:52.395 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-05-27T07:36:52.395 INFO:teuthology.orchestra.run.smithi052.stdout: "registry_password": { 2024-05-27T07:36:52.396 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "registry_password", 2024-05-27T07:36:52.396 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "str", 2024-05-27T07:36:52.396 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-05-27T07:36:52.396 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 0, 2024-05-27T07:36:52.396 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "", 2024-05-27T07:36:52.396 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-05-27T07:36:52.396 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-05-27T07:36:52.396 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-05-27T07:36:52.396 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "Custom repository password. Only used for logging into a registry.", 2024-05-27T07:36:52.396 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-05-27T07:36:52.396 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-05-27T07:36:52.396 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-05-27T07:36:52.397 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-05-27T07:36:52.397 INFO:teuthology.orchestra.run.smithi052.stdout: "registry_url": { 2024-05-27T07:36:52.397 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "registry_url", 2024-05-27T07:36:52.397 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "str", 2024-05-27T07:36:52.397 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-05-27T07:36:52.397 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 0, 2024-05-27T07:36:52.397 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "", 2024-05-27T07:36:52.397 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-05-27T07:36:52.397 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-05-27T07:36:52.397 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-05-27T07:36:52.397 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "Registry url for login purposes. This is not the default registry", 2024-05-27T07:36:52.397 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-05-27T07:36:52.397 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-05-27T07:36:52.398 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-05-27T07:36:52.398 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-05-27T07:36:52.398 INFO:teuthology.orchestra.run.smithi052.stdout: "registry_username": { 2024-05-27T07:36:52.398 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "registry_username", 2024-05-27T07:36:52.398 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "str", 2024-05-27T07:36:52.398 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-05-27T07:36:52.398 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 0, 2024-05-27T07:36:52.398 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "", 2024-05-27T07:36:52.398 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-05-27T07:36:52.398 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-05-27T07:36:52.398 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-05-27T07:36:52.398 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "Custom repository username. Only used for logging into a registry.", 2024-05-27T07:36:52.398 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-05-27T07:36:52.399 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-05-27T07:36:52.399 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-05-27T07:36:52.399 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-05-27T07:36:52.399 INFO:teuthology.orchestra.run.smithi052.stdout: "secure_monitoring_stack": { 2024-05-27T07:36:52.399 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "secure_monitoring_stack", 2024-05-27T07:36:52.399 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "bool", 2024-05-27T07:36:52.399 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-05-27T07:36:52.399 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 0, 2024-05-27T07:36:52.399 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "False", 2024-05-27T07:36:52.399 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-05-27T07:36:52.399 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-05-27T07:36:52.399 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-05-27T07:36:52.399 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "Enable TLS security for all the monitoring stack daemons", 2024-05-27T07:36:52.400 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-05-27T07:36:52.400 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-05-27T07:36:52.400 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-05-27T07:36:52.400 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-05-27T07:36:52.400 INFO:teuthology.orchestra.run.smithi052.stdout: "service_discovery_port": { 2024-05-27T07:36:52.400 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "service_discovery_port", 2024-05-27T07:36:52.400 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "int", 2024-05-27T07:36:52.400 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-05-27T07:36:52.400 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 0, 2024-05-27T07:36:52.400 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "8765", 2024-05-27T07:36:52.400 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-05-27T07:36:52.400 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-05-27T07:36:52.400 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-05-27T07:36:52.401 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "cephadm service discovery port", 2024-05-27T07:36:52.401 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-05-27T07:36:52.401 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-05-27T07:36:52.401 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-05-27T07:36:52.401 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-05-27T07:36:52.401 INFO:teuthology.orchestra.run.smithi052.stdout: "sqlite3_killpoint": { 2024-05-27T07:36:52.401 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "sqlite3_killpoint", 2024-05-27T07:36:52.401 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "int", 2024-05-27T07:36:52.401 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "dev", 2024-05-27T07:36:52.401 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 1, 2024-05-27T07:36:52.401 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "0", 2024-05-27T07:36:52.401 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-05-27T07:36:52.401 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-05-27T07:36:52.402 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-05-27T07:36:52.402 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-05-27T07:36:52.402 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-05-27T07:36:52.402 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-05-27T07:36:52.402 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-05-27T07:36:52.402 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-05-27T07:36:52.402 INFO:teuthology.orchestra.run.smithi052.stdout: "ssh_config_file": { 2024-05-27T07:36:52.402 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "ssh_config_file", 2024-05-27T07:36:52.402 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "str", 2024-05-27T07:36:52.402 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-05-27T07:36:52.402 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 0, 2024-05-27T07:36:52.402 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "", 2024-05-27T07:36:52.403 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-05-27T07:36:52.403 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-05-27T07:36:52.403 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-05-27T07:36:52.403 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "customized SSH config file to connect to managed hosts", 2024-05-27T07:36:52.403 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-05-27T07:36:52.403 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-05-27T07:36:52.403 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-05-27T07:36:52.403 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-05-27T07:36:52.403 INFO:teuthology.orchestra.run.smithi052.stdout: "use_agent": { 2024-05-27T07:36:52.403 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "use_agent", 2024-05-27T07:36:52.403 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "bool", 2024-05-27T07:36:52.403 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-05-27T07:36:52.403 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 0, 2024-05-27T07:36:52.404 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "False", 2024-05-27T07:36:52.404 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-05-27T07:36:52.404 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-05-27T07:36:52.404 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-05-27T07:36:52.404 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "Use cephadm agent on each host to gather and send metadata", 2024-05-27T07:36:52.404 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-05-27T07:36:52.404 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-05-27T07:36:52.404 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-05-27T07:36:52.404 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-05-27T07:36:52.404 INFO:teuthology.orchestra.run.smithi052.stdout: "use_repo_digest": { 2024-05-27T07:36:52.404 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "use_repo_digest", 2024-05-27T07:36:52.404 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "bool", 2024-05-27T07:36:52.404 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-05-27T07:36:52.405 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 0, 2024-05-27T07:36:52.405 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "True", 2024-05-27T07:36:52.405 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-05-27T07:36:52.405 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-05-27T07:36:52.405 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-05-27T07:36:52.405 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "Automatically convert image tags to image digest. Make sure all daemons use the same image", 2024-05-27T07:36:52.405 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-05-27T07:36:52.405 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-05-27T07:36:52.405 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-05-27T07:36:52.405 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-05-27T07:36:52.405 INFO:teuthology.orchestra.run.smithi052.stdout: "warn_on_failed_host_check": { 2024-05-27T07:36:52.405 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "warn_on_failed_host_check", 2024-05-27T07:36:52.405 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "bool", 2024-05-27T07:36:52.406 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-05-27T07:36:52.406 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 0, 2024-05-27T07:36:52.406 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "True", 2024-05-27T07:36:52.406 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-05-27T07:36:52.406 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-05-27T07:36:52.406 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-05-27T07:36:52.406 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "raise a health warning if the host check fails", 2024-05-27T07:36:52.406 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-05-27T07:36:52.406 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-05-27T07:36:52.406 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-05-27T07:36:52.406 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-05-27T07:36:52.406 INFO:teuthology.orchestra.run.smithi052.stdout: "warn_on_stray_daemons": { 2024-05-27T07:36:52.407 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "warn_on_stray_daemons", 2024-05-27T07:36:52.407 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "bool", 2024-05-27T07:36:52.407 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-05-27T07:36:52.407 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 0, 2024-05-27T07:36:52.407 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "True", 2024-05-27T07:36:52.407 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-05-27T07:36:52.407 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-05-27T07:36:52.407 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-05-27T07:36:52.407 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "raise a health warning if daemons are detected that are not managed by cephadm", 2024-05-27T07:36:52.407 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-05-27T07:36:52.407 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-05-27T07:36:52.407 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-05-27T07:36:52.407 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-05-27T07:36:52.407 INFO:teuthology.orchestra.run.smithi052.stdout: "warn_on_stray_hosts": { 2024-05-27T07:36:52.408 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "warn_on_stray_hosts", 2024-05-27T07:36:52.408 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "bool", 2024-05-27T07:36:52.408 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-05-27T07:36:52.408 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 0, 2024-05-27T07:36:52.408 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "True", 2024-05-27T07:36:52.408 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-05-27T07:36:52.408 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-05-27T07:36:52.408 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-05-27T07:36:52.408 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "raise a health warning if daemons are detected on a host that is not managed by cephadm", 2024-05-27T07:36:52.408 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-05-27T07:36:52.408 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-05-27T07:36:52.408 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-05-27T07:36:52.409 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-05-27T07:36:52.409 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-05-27T07:36:52.409 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-05-27T07:36:52.409 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-05-27T07:36:52.409 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "crash", 2024-05-27T07:36:52.409 INFO:teuthology.orchestra.run.smithi052.stdout: "can_run": true, 2024-05-27T07:36:52.409 INFO:teuthology.orchestra.run.smithi052.stdout: "error_string": "", 2024-05-27T07:36:52.409 INFO:teuthology.orchestra.run.smithi052.stdout: "module_options": { 2024-05-27T07:36:52.409 INFO:teuthology.orchestra.run.smithi052.stdout: "log_level": { 2024-05-27T07:36:52.409 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "log_level", 2024-05-27T07:36:52.409 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "str", 2024-05-27T07:36:52.409 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-05-27T07:36:52.409 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 1, 2024-05-27T07:36:52.410 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "", 2024-05-27T07:36:52.410 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-05-27T07:36:52.410 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-05-27T07:36:52.410 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [ 2024-05-27T07:36:52.410 INFO:teuthology.orchestra.run.smithi052.stdout: "", 2024-05-27T07:36:52.410 INFO:teuthology.orchestra.run.smithi052.stdout: "critical", 2024-05-27T07:36:52.410 INFO:teuthology.orchestra.run.smithi052.stdout: "debug", 2024-05-27T07:36:52.410 INFO:teuthology.orchestra.run.smithi052.stdout: "error", 2024-05-27T07:36:52.410 INFO:teuthology.orchestra.run.smithi052.stdout: "info", 2024-05-27T07:36:52.410 INFO:teuthology.orchestra.run.smithi052.stdout: "warning" 2024-05-27T07:36:52.410 INFO:teuthology.orchestra.run.smithi052.stdout: ], 2024-05-27T07:36:52.410 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-05-27T07:36:52.410 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-05-27T07:36:52.411 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-05-27T07:36:52.411 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-05-27T07:36:52.411 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-05-27T07:36:52.411 INFO:teuthology.orchestra.run.smithi052.stdout: "log_to_cluster": { 2024-05-27T07:36:52.411 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "log_to_cluster", 2024-05-27T07:36:52.411 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "bool", 2024-05-27T07:36:52.411 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-05-27T07:36:52.411 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 1, 2024-05-27T07:36:52.411 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "False", 2024-05-27T07:36:52.411 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-05-27T07:36:52.411 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-05-27T07:36:52.411 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-05-27T07:36:52.411 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-05-27T07:36:52.412 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-05-27T07:36:52.412 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-05-27T07:36:52.412 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-05-27T07:36:52.412 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-05-27T07:36:52.412 INFO:teuthology.orchestra.run.smithi052.stdout: "log_to_cluster_level": { 2024-05-27T07:36:52.412 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "log_to_cluster_level", 2024-05-27T07:36:52.412 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "str", 2024-05-27T07:36:52.412 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-05-27T07:36:52.412 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 1, 2024-05-27T07:36:52.412 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "info", 2024-05-27T07:36:52.412 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-05-27T07:36:52.412 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-05-27T07:36:52.412 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [ 2024-05-27T07:36:52.412 INFO:teuthology.orchestra.run.smithi052.stdout: "", 2024-05-27T07:36:52.413 INFO:teuthology.orchestra.run.smithi052.stdout: "critical", 2024-05-27T07:36:52.413 INFO:teuthology.orchestra.run.smithi052.stdout: "debug", 2024-05-27T07:36:52.413 INFO:teuthology.orchestra.run.smithi052.stdout: "error", 2024-05-27T07:36:52.413 INFO:teuthology.orchestra.run.smithi052.stdout: "info", 2024-05-27T07:36:52.413 INFO:teuthology.orchestra.run.smithi052.stdout: "warning" 2024-05-27T07:36:52.413 INFO:teuthology.orchestra.run.smithi052.stdout: ], 2024-05-27T07:36:52.413 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-05-27T07:36:52.413 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-05-27T07:36:52.413 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-05-27T07:36:52.413 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-05-27T07:36:52.413 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-05-27T07:36:52.413 INFO:teuthology.orchestra.run.smithi052.stdout: "log_to_file": { 2024-05-27T07:36:52.414 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "log_to_file", 2024-05-27T07:36:52.414 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "bool", 2024-05-27T07:36:52.414 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-05-27T07:36:52.414 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 1, 2024-05-27T07:36:52.414 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "False", 2024-05-27T07:36:52.414 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-05-27T07:36:52.414 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-05-27T07:36:52.414 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-05-27T07:36:52.414 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-05-27T07:36:52.414 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-05-27T07:36:52.414 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-05-27T07:36:52.414 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-05-27T07:36:52.414 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-05-27T07:36:52.415 INFO:teuthology.orchestra.run.smithi052.stdout: "retain_interval": { 2024-05-27T07:36:52.415 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "retain_interval", 2024-05-27T07:36:52.415 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "secs", 2024-05-27T07:36:52.415 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-05-27T07:36:52.415 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 1, 2024-05-27T07:36:52.415 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "31536000", 2024-05-27T07:36:52.415 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-05-27T07:36:52.415 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-05-27T07:36:52.415 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-05-27T07:36:52.415 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "how long to retain crashes before pruning them", 2024-05-27T07:36:52.415 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-05-27T07:36:52.415 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-05-27T07:36:52.416 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-05-27T07:36:52.416 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-05-27T07:36:52.416 INFO:teuthology.orchestra.run.smithi052.stdout: "sqlite3_killpoint": { 2024-05-27T07:36:52.416 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "sqlite3_killpoint", 2024-05-27T07:36:52.416 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "int", 2024-05-27T07:36:52.416 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "dev", 2024-05-27T07:36:52.416 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 1, 2024-05-27T07:36:52.416 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "0", 2024-05-27T07:36:52.416 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-05-27T07:36:52.416 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-05-27T07:36:52.416 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-05-27T07:36:52.416 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-05-27T07:36:52.416 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-05-27T07:36:52.417 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-05-27T07:36:52.417 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-05-27T07:36:52.417 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-05-27T07:36:52.417 INFO:teuthology.orchestra.run.smithi052.stdout: "warn_recent_interval": { 2024-05-27T07:36:52.417 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "warn_recent_interval", 2024-05-27T07:36:52.417 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "secs", 2024-05-27T07:36:52.417 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-05-27T07:36:52.417 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 1, 2024-05-27T07:36:52.417 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "1209600", 2024-05-27T07:36:52.417 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-05-27T07:36:52.417 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-05-27T07:36:52.417 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-05-27T07:36:52.418 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "time interval in which to warn about recent crashes", 2024-05-27T07:36:52.418 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-05-27T07:36:52.418 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-05-27T07:36:52.418 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-05-27T07:36:52.418 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-05-27T07:36:52.418 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-05-27T07:36:52.418 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-05-27T07:36:52.418 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-05-27T07:36:52.418 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "dashboard", 2024-05-27T07:36:52.418 INFO:teuthology.orchestra.run.smithi052.stdout: "can_run": true, 2024-05-27T07:36:52.418 INFO:teuthology.orchestra.run.smithi052.stdout: "error_string": "", 2024-05-27T07:36:52.419 INFO:teuthology.orchestra.run.smithi052.stdout: "module_options": { 2024-05-27T07:36:52.419 INFO:teuthology.orchestra.run.smithi052.stdout: "ACCOUNT_LOCKOUT_ATTEMPTS": { 2024-05-27T07:36:52.419 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "ACCOUNT_LOCKOUT_ATTEMPTS", 2024-05-27T07:36:52.419 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "int", 2024-05-27T07:36:52.419 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-05-27T07:36:52.419 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 0, 2024-05-27T07:36:52.419 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "10", 2024-05-27T07:36:52.419 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-05-27T07:36:52.419 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-05-27T07:36:52.419 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-05-27T07:36:52.419 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-05-27T07:36:52.419 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-05-27T07:36:52.419 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-05-27T07:36:52.420 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-05-27T07:36:52.420 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-05-27T07:36:52.420 INFO:teuthology.orchestra.run.smithi052.stdout: "ALERTMANAGER_API_HOST": { 2024-05-27T07:36:52.420 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "ALERTMANAGER_API_HOST", 2024-05-27T07:36:52.420 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "str", 2024-05-27T07:36:52.420 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-05-27T07:36:52.420 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 0, 2024-05-27T07:36:52.420 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "", 2024-05-27T07:36:52.420 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-05-27T07:36:52.420 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-05-27T07:36:52.420 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-05-27T07:36:52.420 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-05-27T07:36:52.420 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-05-27T07:36:52.421 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-05-27T07:36:52.421 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-05-27T07:36:52.421 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-05-27T07:36:52.421 INFO:teuthology.orchestra.run.smithi052.stdout: "ALERTMANAGER_API_SSL_VERIFY": { 2024-05-27T07:36:52.421 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "ALERTMANAGER_API_SSL_VERIFY", 2024-05-27T07:36:52.421 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "bool", 2024-05-27T07:36:52.421 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-05-27T07:36:52.421 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 0, 2024-05-27T07:36:52.421 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "True", 2024-05-27T07:36:52.421 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-05-27T07:36:52.421 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-05-27T07:36:52.421 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-05-27T07:36:52.422 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-05-27T07:36:52.422 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-05-27T07:36:52.422 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-05-27T07:36:52.422 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-05-27T07:36:52.422 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-05-27T07:36:52.422 INFO:teuthology.orchestra.run.smithi052.stdout: "AUDIT_API_ENABLED": { 2024-05-27T07:36:52.422 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "AUDIT_API_ENABLED", 2024-05-27T07:36:52.422 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "bool", 2024-05-27T07:36:52.422 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-05-27T07:36:52.422 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 0, 2024-05-27T07:36:52.422 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "False", 2024-05-27T07:36:52.422 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-05-27T07:36:52.422 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-05-27T07:36:52.423 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-05-27T07:36:52.423 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-05-27T07:36:52.423 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-05-27T07:36:52.423 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-05-27T07:36:52.423 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-05-27T07:36:52.423 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-05-27T07:36:52.423 INFO:teuthology.orchestra.run.smithi052.stdout: "AUDIT_API_LOG_PAYLOAD": { 2024-05-27T07:36:52.423 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "AUDIT_API_LOG_PAYLOAD", 2024-05-27T07:36:52.423 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "bool", 2024-05-27T07:36:52.423 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-05-27T07:36:52.423 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 0, 2024-05-27T07:36:52.423 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "True", 2024-05-27T07:36:52.423 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-05-27T07:36:52.424 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-05-27T07:36:52.424 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-05-27T07:36:52.424 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-05-27T07:36:52.424 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-05-27T07:36:52.424 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-05-27T07:36:52.424 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-05-27T07:36:52.424 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-05-27T07:36:52.424 INFO:teuthology.orchestra.run.smithi052.stdout: "ENABLE_BROWSABLE_API": { 2024-05-27T07:36:52.424 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "ENABLE_BROWSABLE_API", 2024-05-27T07:36:52.424 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "bool", 2024-05-27T07:36:52.424 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-05-27T07:36:52.424 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 0, 2024-05-27T07:36:52.425 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "True", 2024-05-27T07:36:52.425 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-05-27T07:36:52.425 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-05-27T07:36:52.425 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-05-27T07:36:52.425 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-05-27T07:36:52.425 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-05-27T07:36:52.425 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-05-27T07:36:52.425 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-05-27T07:36:52.425 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-05-27T07:36:52.425 INFO:teuthology.orchestra.run.smithi052.stdout: "FEATURE_TOGGLE_CEPHFS": { 2024-05-27T07:36:52.425 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "FEATURE_TOGGLE_CEPHFS", 2024-05-27T07:36:52.425 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "bool", 2024-05-27T07:36:52.425 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-05-27T07:36:52.426 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 0, 2024-05-27T07:36:52.426 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "True", 2024-05-27T07:36:52.426 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-05-27T07:36:52.426 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-05-27T07:36:52.426 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-05-27T07:36:52.426 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-05-27T07:36:52.426 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-05-27T07:36:52.426 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-05-27T07:36:52.426 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-05-27T07:36:52.426 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-05-27T07:36:52.426 INFO:teuthology.orchestra.run.smithi052.stdout: "FEATURE_TOGGLE_DASHBOARD": { 2024-05-27T07:36:52.426 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "FEATURE_TOGGLE_DASHBOARD", 2024-05-27T07:36:52.427 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "bool", 2024-05-27T07:36:52.427 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-05-27T07:36:52.427 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 0, 2024-05-27T07:36:52.427 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "True", 2024-05-27T07:36:52.427 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-05-27T07:36:52.427 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-05-27T07:36:52.427 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-05-27T07:36:52.427 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-05-27T07:36:52.427 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-05-27T07:36:52.427 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-05-27T07:36:52.427 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-05-27T07:36:52.427 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-05-27T07:36:52.427 INFO:teuthology.orchestra.run.smithi052.stdout: "FEATURE_TOGGLE_ISCSI": { 2024-05-27T07:36:52.428 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "FEATURE_TOGGLE_ISCSI", 2024-05-27T07:36:52.428 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "bool", 2024-05-27T07:36:52.428 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-05-27T07:36:52.428 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 0, 2024-05-27T07:36:52.428 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "True", 2024-05-27T07:36:52.428 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-05-27T07:36:52.428 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-05-27T07:36:52.428 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-05-27T07:36:52.428 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-05-27T07:36:52.428 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-05-27T07:36:52.428 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-05-27T07:36:52.428 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-05-27T07:36:52.428 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-05-27T07:36:52.429 INFO:teuthology.orchestra.run.smithi052.stdout: "FEATURE_TOGGLE_MIRRORING": { 2024-05-27T07:36:52.429 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "FEATURE_TOGGLE_MIRRORING", 2024-05-27T07:36:52.429 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "bool", 2024-05-27T07:36:52.429 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-05-27T07:36:52.429 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 0, 2024-05-27T07:36:52.429 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "True", 2024-05-27T07:36:52.429 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-05-27T07:36:52.429 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-05-27T07:36:52.429 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-05-27T07:36:52.429 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-05-27T07:36:52.429 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-05-27T07:36:52.429 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-05-27T07:36:52.429 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-05-27T07:36:52.430 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-05-27T07:36:52.430 INFO:teuthology.orchestra.run.smithi052.stdout: "FEATURE_TOGGLE_NFS": { 2024-05-27T07:36:52.430 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "FEATURE_TOGGLE_NFS", 2024-05-27T07:36:52.430 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "bool", 2024-05-27T07:36:52.430 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-05-27T07:36:52.430 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 0, 2024-05-27T07:36:52.430 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "True", 2024-05-27T07:36:52.430 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-05-27T07:36:52.430 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-05-27T07:36:52.430 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-05-27T07:36:52.430 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-05-27T07:36:52.430 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-05-27T07:36:52.430 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-05-27T07:36:52.431 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-05-27T07:36:52.431 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-05-27T07:36:52.431 INFO:teuthology.orchestra.run.smithi052.stdout: "FEATURE_TOGGLE_RBD": { 2024-05-27T07:36:52.431 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "FEATURE_TOGGLE_RBD", 2024-05-27T07:36:52.431 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "bool", 2024-05-27T07:36:52.431 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-05-27T07:36:52.431 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 0, 2024-05-27T07:36:52.431 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "True", 2024-05-27T07:36:52.431 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-05-27T07:36:52.431 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-05-27T07:36:52.431 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-05-27T07:36:52.431 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-05-27T07:36:52.431 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-05-27T07:36:52.432 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-05-27T07:36:52.432 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-05-27T07:36:52.432 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-05-27T07:36:52.432 INFO:teuthology.orchestra.run.smithi052.stdout: "FEATURE_TOGGLE_RGW": { 2024-05-27T07:36:52.432 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "FEATURE_TOGGLE_RGW", 2024-05-27T07:36:52.432 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "bool", 2024-05-27T07:36:52.432 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-05-27T07:36:52.432 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 0, 2024-05-27T07:36:52.432 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "True", 2024-05-27T07:36:52.432 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-05-27T07:36:52.432 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-05-27T07:36:52.432 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-05-27T07:36:52.433 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-05-27T07:36:52.433 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-05-27T07:36:52.433 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-05-27T07:36:52.433 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-05-27T07:36:52.433 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-05-27T07:36:52.433 INFO:teuthology.orchestra.run.smithi052.stdout: "GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE": { 2024-05-27T07:36:52.433 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE", 2024-05-27T07:36:52.433 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "str", 2024-05-27T07:36:52.433 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-05-27T07:36:52.433 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 0, 2024-05-27T07:36:52.433 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "", 2024-05-27T07:36:52.433 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-05-27T07:36:52.433 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-05-27T07:36:52.434 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-05-27T07:36:52.434 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-05-27T07:36:52.434 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-05-27T07:36:52.434 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-05-27T07:36:52.434 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-05-27T07:36:52.434 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-05-27T07:36:52.434 INFO:teuthology.orchestra.run.smithi052.stdout: "GRAFANA_API_PASSWORD": { 2024-05-27T07:36:52.434 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "GRAFANA_API_PASSWORD", 2024-05-27T07:36:52.434 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "str", 2024-05-27T07:36:52.434 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-05-27T07:36:52.434 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 0, 2024-05-27T07:36:52.435 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "admin", 2024-05-27T07:36:52.435 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-05-27T07:36:52.435 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-05-27T07:36:52.435 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-05-27T07:36:52.435 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-05-27T07:36:52.435 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-05-27T07:36:52.435 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-05-27T07:36:52.435 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-05-27T07:36:52.435 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-05-27T07:36:52.435 INFO:teuthology.orchestra.run.smithi052.stdout: "GRAFANA_API_SSL_VERIFY": { 2024-05-27T07:36:52.435 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "GRAFANA_API_SSL_VERIFY", 2024-05-27T07:36:52.435 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "bool", 2024-05-27T07:36:52.435 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-05-27T07:36:52.436 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 0, 2024-05-27T07:36:52.436 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "True", 2024-05-27T07:36:52.436 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-05-27T07:36:52.436 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-05-27T07:36:52.436 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-05-27T07:36:52.436 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-05-27T07:36:52.436 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-05-27T07:36:52.436 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-05-27T07:36:52.436 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-05-27T07:36:52.436 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-05-27T07:36:52.436 INFO:teuthology.orchestra.run.smithi052.stdout: "GRAFANA_API_URL": { 2024-05-27T07:36:52.436 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "GRAFANA_API_URL", 2024-05-27T07:36:52.436 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "str", 2024-05-27T07:36:52.437 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-05-27T07:36:52.437 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 0, 2024-05-27T07:36:52.437 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "", 2024-05-27T07:36:52.437 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-05-27T07:36:52.437 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-05-27T07:36:52.437 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-05-27T07:36:52.437 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-05-27T07:36:52.437 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-05-27T07:36:52.437 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-05-27T07:36:52.437 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-05-27T07:36:52.437 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-05-27T07:36:52.437 INFO:teuthology.orchestra.run.smithi052.stdout: "GRAFANA_API_USERNAME": { 2024-05-27T07:36:52.437 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "GRAFANA_API_USERNAME", 2024-05-27T07:36:52.438 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "str", 2024-05-27T07:36:52.438 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-05-27T07:36:52.438 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 0, 2024-05-27T07:36:52.438 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "admin", 2024-05-27T07:36:52.438 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-05-27T07:36:52.438 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-05-27T07:36:52.438 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-05-27T07:36:52.438 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-05-27T07:36:52.438 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-05-27T07:36:52.438 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-05-27T07:36:52.438 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-05-27T07:36:52.438 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-05-27T07:36:52.439 INFO:teuthology.orchestra.run.smithi052.stdout: "GRAFANA_FRONTEND_API_URL": { 2024-05-27T07:36:52.439 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "GRAFANA_FRONTEND_API_URL", 2024-05-27T07:36:52.439 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "str", 2024-05-27T07:36:52.439 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-05-27T07:36:52.439 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 0, 2024-05-27T07:36:52.439 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "", 2024-05-27T07:36:52.439 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-05-27T07:36:52.439 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-05-27T07:36:52.439 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-05-27T07:36:52.439 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-05-27T07:36:52.439 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-05-27T07:36:52.439 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-05-27T07:36:52.439 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-05-27T07:36:52.440 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-05-27T07:36:52.440 INFO:teuthology.orchestra.run.smithi052.stdout: "GRAFANA_UPDATE_DASHBOARDS": { 2024-05-27T07:36:52.440 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "GRAFANA_UPDATE_DASHBOARDS", 2024-05-27T07:36:52.440 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "bool", 2024-05-27T07:36:52.440 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-05-27T07:36:52.440 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 0, 2024-05-27T07:36:52.440 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "False", 2024-05-27T07:36:52.440 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-05-27T07:36:52.440 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-05-27T07:36:52.440 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-05-27T07:36:52.440 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-05-27T07:36:52.440 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-05-27T07:36:52.440 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-05-27T07:36:52.441 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-05-27T07:36:52.441 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-05-27T07:36:52.441 INFO:teuthology.orchestra.run.smithi052.stdout: "ISCSI_API_SSL_VERIFICATION": { 2024-05-27T07:36:52.441 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "ISCSI_API_SSL_VERIFICATION", 2024-05-27T07:36:52.441 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "bool", 2024-05-27T07:36:52.441 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-05-27T07:36:52.441 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 0, 2024-05-27T07:36:52.441 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "True", 2024-05-27T07:36:52.441 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-05-27T07:36:52.441 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-05-27T07:36:52.441 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-05-27T07:36:52.441 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-05-27T07:36:52.441 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-05-27T07:36:52.442 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-05-27T07:36:52.442 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-05-27T07:36:52.442 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-05-27T07:36:52.442 INFO:teuthology.orchestra.run.smithi052.stdout: "ISSUE_TRACKER_API_KEY": { 2024-05-27T07:36:52.442 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "ISSUE_TRACKER_API_KEY", 2024-05-27T07:36:52.442 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "str", 2024-05-27T07:36:52.442 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-05-27T07:36:52.442 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 0, 2024-05-27T07:36:52.442 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "", 2024-05-27T07:36:52.442 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-05-27T07:36:52.442 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-05-27T07:36:52.442 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-05-27T07:36:52.443 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-05-27T07:36:52.443 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-05-27T07:36:52.443 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-05-27T07:36:52.443 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-05-27T07:36:52.443 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-05-27T07:36:52.443 INFO:teuthology.orchestra.run.smithi052.stdout: "MANAGED_BY_CLUSTERS": { 2024-05-27T07:36:52.443 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "MANAGED_BY_CLUSTERS", 2024-05-27T07:36:52.443 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "str", 2024-05-27T07:36:52.443 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-05-27T07:36:52.443 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 0, 2024-05-27T07:36:52.443 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "[]", 2024-05-27T07:36:52.443 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-05-27T07:36:52.443 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-05-27T07:36:52.444 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-05-27T07:36:52.444 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-05-27T07:36:52.444 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-05-27T07:36:52.444 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-05-27T07:36:52.444 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-05-27T07:36:52.444 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-05-27T07:36:52.444 INFO:teuthology.orchestra.run.smithi052.stdout: "MULTICLUSTER_CONFIG": { 2024-05-27T07:36:52.444 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "MULTICLUSTER_CONFIG", 2024-05-27T07:36:52.444 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "str", 2024-05-27T07:36:52.444 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-05-27T07:36:52.444 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 0, 2024-05-27T07:36:52.444 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "{}", 2024-05-27T07:36:52.444 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-05-27T07:36:52.445 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-05-27T07:36:52.445 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-05-27T07:36:52.445 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-05-27T07:36:52.445 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-05-27T07:36:52.445 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-05-27T07:36:52.445 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-05-27T07:36:52.445 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-05-27T07:36:52.445 INFO:teuthology.orchestra.run.smithi052.stdout: "PROMETHEUS_API_HOST": { 2024-05-27T07:36:52.445 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "PROMETHEUS_API_HOST", 2024-05-27T07:36:52.445 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "str", 2024-05-27T07:36:52.445 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-05-27T07:36:52.445 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 0, 2024-05-27T07:36:52.445 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "", 2024-05-27T07:36:52.446 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-05-27T07:36:52.446 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-05-27T07:36:52.446 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-05-27T07:36:52.446 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-05-27T07:36:52.446 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-05-27T07:36:52.446 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-05-27T07:36:52.446 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-05-27T07:36:52.446 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-05-27T07:36:52.446 INFO:teuthology.orchestra.run.smithi052.stdout: "PROMETHEUS_API_SSL_VERIFY": { 2024-05-27T07:36:52.446 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "PROMETHEUS_API_SSL_VERIFY", 2024-05-27T07:36:52.446 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "bool", 2024-05-27T07:36:52.446 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-05-27T07:36:52.446 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 0, 2024-05-27T07:36:52.447 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "True", 2024-05-27T07:36:52.447 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-05-27T07:36:52.447 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-05-27T07:36:52.447 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-05-27T07:36:52.447 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-05-27T07:36:52.447 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-05-27T07:36:52.447 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-05-27T07:36:52.447 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-05-27T07:36:52.447 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-05-27T07:36:52.447 INFO:teuthology.orchestra.run.smithi052.stdout: "PWD_POLICY_CHECK_COMPLEXITY_ENABLED": { 2024-05-27T07:36:52.447 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "PWD_POLICY_CHECK_COMPLEXITY_ENABLED", 2024-05-27T07:36:52.447 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "bool", 2024-05-27T07:36:52.447 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-05-27T07:36:52.448 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 0, 2024-05-27T07:36:52.448 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "False", 2024-05-27T07:36:52.448 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-05-27T07:36:52.448 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-05-27T07:36:52.448 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-05-27T07:36:52.448 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-05-27T07:36:52.448 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-05-27T07:36:52.448 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-05-27T07:36:52.448 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-05-27T07:36:52.448 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-05-27T07:36:52.448 INFO:teuthology.orchestra.run.smithi052.stdout: "PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED": { 2024-05-27T07:36:52.448 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED", 2024-05-27T07:36:52.449 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "bool", 2024-05-27T07:36:52.449 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-05-27T07:36:52.449 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 0, 2024-05-27T07:36:52.449 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "False", 2024-05-27T07:36:52.449 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-05-27T07:36:52.449 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-05-27T07:36:52.449 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-05-27T07:36:52.449 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-05-27T07:36:52.449 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-05-27T07:36:52.449 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-05-27T07:36:52.449 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-05-27T07:36:52.449 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-05-27T07:36:52.449 INFO:teuthology.orchestra.run.smithi052.stdout: "PWD_POLICY_CHECK_LENGTH_ENABLED": { 2024-05-27T07:36:52.450 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "PWD_POLICY_CHECK_LENGTH_ENABLED", 2024-05-27T07:36:52.450 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "bool", 2024-05-27T07:36:52.450 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-05-27T07:36:52.450 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 0, 2024-05-27T07:36:52.450 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "True", 2024-05-27T07:36:52.450 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-05-27T07:36:52.450 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-05-27T07:36:52.450 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-05-27T07:36:52.450 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-05-27T07:36:52.450 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-05-27T07:36:52.450 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-05-27T07:36:52.450 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-05-27T07:36:52.451 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-05-27T07:36:52.451 INFO:teuthology.orchestra.run.smithi052.stdout: "PWD_POLICY_CHECK_OLDPWD_ENABLED": { 2024-05-27T07:36:52.451 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "PWD_POLICY_CHECK_OLDPWD_ENABLED", 2024-05-27T07:36:52.451 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "bool", 2024-05-27T07:36:52.451 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-05-27T07:36:52.451 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 0, 2024-05-27T07:36:52.451 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "True", 2024-05-27T07:36:52.451 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-05-27T07:36:52.451 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-05-27T07:36:52.451 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-05-27T07:36:52.451 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-05-27T07:36:52.451 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-05-27T07:36:52.451 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-05-27T07:36:52.452 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-05-27T07:36:52.452 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-05-27T07:36:52.452 INFO:teuthology.orchestra.run.smithi052.stdout: "PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED": { 2024-05-27T07:36:52.452 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED", 2024-05-27T07:36:52.452 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "bool", 2024-05-27T07:36:52.452 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-05-27T07:36:52.452 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 0, 2024-05-27T07:36:52.452 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "False", 2024-05-27T07:36:52.452 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-05-27T07:36:52.452 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-05-27T07:36:52.452 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-05-27T07:36:52.452 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-05-27T07:36:52.453 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-05-27T07:36:52.453 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-05-27T07:36:52.453 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-05-27T07:36:52.453 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-05-27T07:36:52.453 INFO:teuthology.orchestra.run.smithi052.stdout: "PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED": { 2024-05-27T07:36:52.453 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED", 2024-05-27T07:36:52.453 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "bool", 2024-05-27T07:36:52.453 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-05-27T07:36:52.453 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 0, 2024-05-27T07:36:52.453 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "False", 2024-05-27T07:36:52.453 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-05-27T07:36:52.453 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-05-27T07:36:52.453 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-05-27T07:36:52.454 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-05-27T07:36:52.454 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-05-27T07:36:52.454 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-05-27T07:36:52.454 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-05-27T07:36:52.454 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-05-27T07:36:52.454 INFO:teuthology.orchestra.run.smithi052.stdout: "PWD_POLICY_CHECK_USERNAME_ENABLED": { 2024-05-27T07:36:52.454 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "PWD_POLICY_CHECK_USERNAME_ENABLED", 2024-05-27T07:36:52.454 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "bool", 2024-05-27T07:36:52.454 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-05-27T07:36:52.454 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 0, 2024-05-27T07:36:52.454 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "False", 2024-05-27T07:36:52.454 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-05-27T07:36:52.455 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-05-27T07:36:52.455 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-05-27T07:36:52.455 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-05-27T07:36:52.455 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-05-27T07:36:52.455 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-05-27T07:36:52.455 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-05-27T07:36:52.455 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-05-27T07:36:52.455 INFO:teuthology.orchestra.run.smithi052.stdout: "PWD_POLICY_ENABLED": { 2024-05-27T07:36:52.455 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "PWD_POLICY_ENABLED", 2024-05-27T07:36:52.455 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "bool", 2024-05-27T07:36:52.455 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-05-27T07:36:52.455 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 0, 2024-05-27T07:36:52.456 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "True", 2024-05-27T07:36:52.456 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-05-27T07:36:52.456 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-05-27T07:36:52.456 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-05-27T07:36:52.456 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-05-27T07:36:52.456 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-05-27T07:36:52.456 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-05-27T07:36:52.456 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-05-27T07:36:52.456 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-05-27T07:36:52.456 INFO:teuthology.orchestra.run.smithi052.stdout: "PWD_POLICY_EXCLUSION_LIST": { 2024-05-27T07:36:52.456 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "PWD_POLICY_EXCLUSION_LIST", 2024-05-27T07:36:52.456 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "str", 2024-05-27T07:36:52.457 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-05-27T07:36:52.457 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 0, 2024-05-27T07:36:52.457 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps", 2024-05-27T07:36:52.457 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-05-27T07:36:52.457 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-05-27T07:36:52.457 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-05-27T07:36:52.457 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-05-27T07:36:52.457 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-05-27T07:36:52.457 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-05-27T07:36:52.457 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-05-27T07:36:52.457 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-05-27T07:36:52.457 INFO:teuthology.orchestra.run.smithi052.stdout: "PWD_POLICY_MIN_COMPLEXITY": { 2024-05-27T07:36:52.457 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "PWD_POLICY_MIN_COMPLEXITY", 2024-05-27T07:36:52.458 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "int", 2024-05-27T07:36:52.458 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-05-27T07:36:52.458 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 0, 2024-05-27T07:36:52.458 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "10", 2024-05-27T07:36:52.458 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-05-27T07:36:52.458 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-05-27T07:36:52.458 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-05-27T07:36:52.458 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-05-27T07:36:52.458 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-05-27T07:36:52.458 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-05-27T07:36:52.458 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-05-27T07:36:52.458 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-05-27T07:36:52.458 INFO:teuthology.orchestra.run.smithi052.stdout: "PWD_POLICY_MIN_LENGTH": { 2024-05-27T07:36:52.459 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "PWD_POLICY_MIN_LENGTH", 2024-05-27T07:36:52.459 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "int", 2024-05-27T07:36:52.459 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-05-27T07:36:52.459 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 0, 2024-05-27T07:36:52.459 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "8", 2024-05-27T07:36:52.459 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-05-27T07:36:52.459 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-05-27T07:36:52.459 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-05-27T07:36:52.459 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-05-27T07:36:52.459 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-05-27T07:36:52.459 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-05-27T07:36:52.459 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-05-27T07:36:52.459 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-05-27T07:36:52.460 INFO:teuthology.orchestra.run.smithi052.stdout: "REST_REQUESTS_TIMEOUT": { 2024-05-27T07:36:52.460 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "REST_REQUESTS_TIMEOUT", 2024-05-27T07:36:52.460 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "int", 2024-05-27T07:36:52.460 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-05-27T07:36:52.460 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 0, 2024-05-27T07:36:52.460 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "45", 2024-05-27T07:36:52.460 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-05-27T07:36:52.460 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-05-27T07:36:52.460 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-05-27T07:36:52.460 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-05-27T07:36:52.460 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-05-27T07:36:52.460 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-05-27T07:36:52.460 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-05-27T07:36:52.461 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-05-27T07:36:52.461 INFO:teuthology.orchestra.run.smithi052.stdout: "RGW_API_ACCESS_KEY": { 2024-05-27T07:36:52.461 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "RGW_API_ACCESS_KEY", 2024-05-27T07:36:52.461 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "str", 2024-05-27T07:36:52.461 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-05-27T07:36:52.461 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 0, 2024-05-27T07:36:52.461 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "", 2024-05-27T07:36:52.461 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-05-27T07:36:52.461 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-05-27T07:36:52.461 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-05-27T07:36:52.461 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-05-27T07:36:52.461 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-05-27T07:36:52.462 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-05-27T07:36:52.462 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-05-27T07:36:52.462 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-05-27T07:36:52.462 INFO:teuthology.orchestra.run.smithi052.stdout: "RGW_API_ADMIN_RESOURCE": { 2024-05-27T07:36:52.462 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "RGW_API_ADMIN_RESOURCE", 2024-05-27T07:36:52.462 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "str", 2024-05-27T07:36:52.462 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-05-27T07:36:52.462 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 0, 2024-05-27T07:36:52.462 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "admin", 2024-05-27T07:36:52.462 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-05-27T07:36:52.462 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-05-27T07:36:52.462 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-05-27T07:36:52.462 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-05-27T07:36:52.463 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-05-27T07:36:52.463 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-05-27T07:36:52.463 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-05-27T07:36:52.463 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-05-27T07:36:52.463 INFO:teuthology.orchestra.run.smithi052.stdout: "RGW_API_SECRET_KEY": { 2024-05-27T07:36:52.463 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "RGW_API_SECRET_KEY", 2024-05-27T07:36:52.463 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "str", 2024-05-27T07:36:52.463 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-05-27T07:36:52.463 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 0, 2024-05-27T07:36:52.463 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "", 2024-05-27T07:36:52.463 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-05-27T07:36:52.463 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-05-27T07:36:52.463 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-05-27T07:36:52.464 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-05-27T07:36:52.464 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-05-27T07:36:52.464 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-05-27T07:36:52.464 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-05-27T07:36:52.464 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-05-27T07:36:52.464 INFO:teuthology.orchestra.run.smithi052.stdout: "RGW_API_SSL_VERIFY": { 2024-05-27T07:36:52.464 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "RGW_API_SSL_VERIFY", 2024-05-27T07:36:52.464 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "bool", 2024-05-27T07:36:52.464 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-05-27T07:36:52.464 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 0, 2024-05-27T07:36:52.464 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "True", 2024-05-27T07:36:52.464 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-05-27T07:36:52.464 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-05-27T07:36:52.464 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-05-27T07:36:52.465 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-05-27T07:36:52.465 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-05-27T07:36:52.465 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-05-27T07:36:52.465 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-05-27T07:36:52.465 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-05-27T07:36:52.465 INFO:teuthology.orchestra.run.smithi052.stdout: "UNSAFE_TLS_v1_2": { 2024-05-27T07:36:52.465 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "UNSAFE_TLS_v1_2", 2024-05-27T07:36:52.465 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "bool", 2024-05-27T07:36:52.465 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-05-27T07:36:52.465 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 0, 2024-05-27T07:36:52.465 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "False", 2024-05-27T07:36:52.465 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-05-27T07:36:52.465 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-05-27T07:36:52.466 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-05-27T07:36:52.466 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-05-27T07:36:52.466 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-05-27T07:36:52.466 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-05-27T07:36:52.466 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-05-27T07:36:52.466 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-05-27T07:36:52.466 INFO:teuthology.orchestra.run.smithi052.stdout: "USER_PWD_EXPIRATION_SPAN": { 2024-05-27T07:36:52.466 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "USER_PWD_EXPIRATION_SPAN", 2024-05-27T07:36:52.466 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "int", 2024-05-27T07:36:52.466 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-05-27T07:36:52.466 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 0, 2024-05-27T07:36:52.466 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "0", 2024-05-27T07:36:52.467 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-05-27T07:36:52.467 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-05-27T07:36:52.467 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-05-27T07:36:52.467 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-05-27T07:36:52.467 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-05-27T07:36:52.467 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-05-27T07:36:52.467 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-05-27T07:36:52.467 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-05-27T07:36:52.467 INFO:teuthology.orchestra.run.smithi052.stdout: "USER_PWD_EXPIRATION_WARNING_1": { 2024-05-27T07:36:52.467 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "USER_PWD_EXPIRATION_WARNING_1", 2024-05-27T07:36:52.467 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "int", 2024-05-27T07:36:52.467 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-05-27T07:36:52.467 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 0, 2024-05-27T07:36:52.468 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "10", 2024-05-27T07:36:52.468 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-05-27T07:36:52.468 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-05-27T07:36:52.468 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-05-27T07:36:52.468 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-05-27T07:36:52.468 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-05-27T07:36:52.468 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-05-27T07:36:52.468 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-05-27T07:36:52.468 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-05-27T07:36:52.468 INFO:teuthology.orchestra.run.smithi052.stdout: "USER_PWD_EXPIRATION_WARNING_2": { 2024-05-27T07:36:52.468 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "USER_PWD_EXPIRATION_WARNING_2", 2024-05-27T07:36:52.468 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "int", 2024-05-27T07:36:52.468 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-05-27T07:36:52.469 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 0, 2024-05-27T07:36:52.469 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "5", 2024-05-27T07:36:52.469 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-05-27T07:36:52.469 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-05-27T07:36:52.469 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-05-27T07:36:52.469 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-05-27T07:36:52.469 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-05-27T07:36:52.469 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-05-27T07:36:52.469 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-05-27T07:36:52.469 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-05-27T07:36:52.469 INFO:teuthology.orchestra.run.smithi052.stdout: "cross_origin_url": { 2024-05-27T07:36:52.469 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "cross_origin_url", 2024-05-27T07:36:52.469 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "str", 2024-05-27T07:36:52.469 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-05-27T07:36:52.470 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 0, 2024-05-27T07:36:52.470 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "", 2024-05-27T07:36:52.470 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-05-27T07:36:52.470 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-05-27T07:36:52.470 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-05-27T07:36:52.470 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-05-27T07:36:52.470 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-05-27T07:36:52.470 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-05-27T07:36:52.470 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-05-27T07:36:52.470 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-05-27T07:36:52.470 INFO:teuthology.orchestra.run.smithi052.stdout: "crt_file": { 2024-05-27T07:36:52.470 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "crt_file", 2024-05-27T07:36:52.471 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "str", 2024-05-27T07:36:52.471 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-05-27T07:36:52.471 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 0, 2024-05-27T07:36:52.471 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "", 2024-05-27T07:36:52.471 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-05-27T07:36:52.471 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-05-27T07:36:52.471 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-05-27T07:36:52.471 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-05-27T07:36:52.471 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-05-27T07:36:52.471 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-05-27T07:36:52.471 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-05-27T07:36:52.471 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-05-27T07:36:52.471 INFO:teuthology.orchestra.run.smithi052.stdout: "debug": { 2024-05-27T07:36:52.472 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "debug", 2024-05-27T07:36:52.472 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "bool", 2024-05-27T07:36:52.472 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-05-27T07:36:52.472 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 0, 2024-05-27T07:36:52.472 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "False", 2024-05-27T07:36:52.472 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-05-27T07:36:52.472 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-05-27T07:36:52.472 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-05-27T07:36:52.472 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "Enable/disable debug options", 2024-05-27T07:36:52.472 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-05-27T07:36:52.472 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-05-27T07:36:52.472 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-05-27T07:36:52.472 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-05-27T07:36:52.472 INFO:teuthology.orchestra.run.smithi052.stdout: "jwt_token_ttl": { 2024-05-27T07:36:52.473 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "jwt_token_ttl", 2024-05-27T07:36:52.473 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "int", 2024-05-27T07:36:52.473 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-05-27T07:36:52.473 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 0, 2024-05-27T07:36:52.473 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "28800", 2024-05-27T07:36:52.473 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-05-27T07:36:52.473 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-05-27T07:36:52.473 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-05-27T07:36:52.473 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-05-27T07:36:52.473 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-05-27T07:36:52.473 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-05-27T07:36:52.473 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-05-27T07:36:52.473 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-05-27T07:36:52.474 INFO:teuthology.orchestra.run.smithi052.stdout: "key_file": { 2024-05-27T07:36:52.474 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "key_file", 2024-05-27T07:36:52.474 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "str", 2024-05-27T07:36:52.474 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-05-27T07:36:52.474 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 0, 2024-05-27T07:36:52.474 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "", 2024-05-27T07:36:52.474 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-05-27T07:36:52.474 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-05-27T07:36:52.474 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-05-27T07:36:52.474 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-05-27T07:36:52.474 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-05-27T07:36:52.474 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-05-27T07:36:52.474 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-05-27T07:36:52.475 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-05-27T07:36:52.475 INFO:teuthology.orchestra.run.smithi052.stdout: "log_level": { 2024-05-27T07:36:52.475 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "log_level", 2024-05-27T07:36:52.475 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "str", 2024-05-27T07:36:52.475 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-05-27T07:36:52.475 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 1, 2024-05-27T07:36:52.475 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "", 2024-05-27T07:36:52.475 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-05-27T07:36:52.475 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-05-27T07:36:52.475 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [ 2024-05-27T07:36:52.475 INFO:teuthology.orchestra.run.smithi052.stdout: "", 2024-05-27T07:36:52.475 INFO:teuthology.orchestra.run.smithi052.stdout: "critical", 2024-05-27T07:36:52.475 INFO:teuthology.orchestra.run.smithi052.stdout: "debug", 2024-05-27T07:36:52.476 INFO:teuthology.orchestra.run.smithi052.stdout: "error", 2024-05-27T07:36:52.476 INFO:teuthology.orchestra.run.smithi052.stdout: "info", 2024-05-27T07:36:52.476 INFO:teuthology.orchestra.run.smithi052.stdout: "warning" 2024-05-27T07:36:52.476 INFO:teuthology.orchestra.run.smithi052.stdout: ], 2024-05-27T07:36:52.476 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-05-27T07:36:52.476 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-05-27T07:36:52.476 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-05-27T07:36:52.476 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-05-27T07:36:52.476 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-05-27T07:36:52.476 INFO:teuthology.orchestra.run.smithi052.stdout: "log_to_cluster": { 2024-05-27T07:36:52.476 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "log_to_cluster", 2024-05-27T07:36:52.476 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "bool", 2024-05-27T07:36:52.476 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-05-27T07:36:52.477 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 1, 2024-05-27T07:36:52.477 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "False", 2024-05-27T07:36:52.477 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-05-27T07:36:52.477 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-05-27T07:36:52.477 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-05-27T07:36:52.477 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-05-27T07:36:52.477 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-05-27T07:36:52.477 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-05-27T07:36:52.477 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-05-27T07:36:52.477 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-05-27T07:36:52.477 INFO:teuthology.orchestra.run.smithi052.stdout: "log_to_cluster_level": { 2024-05-27T07:36:52.477 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "log_to_cluster_level", 2024-05-27T07:36:52.478 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "str", 2024-05-27T07:36:52.478 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-05-27T07:36:52.478 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 1, 2024-05-27T07:36:52.478 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "info", 2024-05-27T07:36:52.478 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-05-27T07:36:52.478 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-05-27T07:36:52.478 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [ 2024-05-27T07:36:52.478 INFO:teuthology.orchestra.run.smithi052.stdout: "", 2024-05-27T07:36:52.478 INFO:teuthology.orchestra.run.smithi052.stdout: "critical", 2024-05-27T07:36:52.478 INFO:teuthology.orchestra.run.smithi052.stdout: "debug", 2024-05-27T07:36:52.478 INFO:teuthology.orchestra.run.smithi052.stdout: "error", 2024-05-27T07:36:52.478 INFO:teuthology.orchestra.run.smithi052.stdout: "info", 2024-05-27T07:36:52.478 INFO:teuthology.orchestra.run.smithi052.stdout: "warning" 2024-05-27T07:36:52.479 INFO:teuthology.orchestra.run.smithi052.stdout: ], 2024-05-27T07:36:52.479 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-05-27T07:36:52.479 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-05-27T07:36:52.479 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-05-27T07:36:52.479 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-05-27T07:36:52.479 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-05-27T07:36:52.479 INFO:teuthology.orchestra.run.smithi052.stdout: "log_to_file": { 2024-05-27T07:36:52.479 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "log_to_file", 2024-05-27T07:36:52.479 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "bool", 2024-05-27T07:36:52.481 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-05-27T07:36:52.481 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 1, 2024-05-27T07:36:52.481 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "False", 2024-05-27T07:36:52.481 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-05-27T07:36:52.482 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-05-27T07:36:52.482 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-05-27T07:36:52.482 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-05-27T07:36:52.482 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-05-27T07:36:52.482 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-05-27T07:36:52.482 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-05-27T07:36:52.482 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-05-27T07:36:52.482 INFO:teuthology.orchestra.run.smithi052.stdout: "motd": { 2024-05-27T07:36:52.482 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "motd", 2024-05-27T07:36:52.482 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "str", 2024-05-27T07:36:52.482 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-05-27T07:36:52.482 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 0, 2024-05-27T07:36:52.482 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "", 2024-05-27T07:36:52.483 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-05-27T07:36:52.483 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-05-27T07:36:52.483 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-05-27T07:36:52.483 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "The message of the day", 2024-05-27T07:36:52.483 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-05-27T07:36:52.483 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-05-27T07:36:52.483 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-05-27T07:36:52.483 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-05-27T07:36:52.483 INFO:teuthology.orchestra.run.smithi052.stdout: "redirect_resolve_ip_addr": { 2024-05-27T07:36:52.483 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "redirect_resolve_ip_addr", 2024-05-27T07:36:52.483 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "bool", 2024-05-27T07:36:52.483 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-05-27T07:36:52.484 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 0, 2024-05-27T07:36:52.484 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "False", 2024-05-27T07:36:52.484 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-05-27T07:36:52.484 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-05-27T07:36:52.484 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-05-27T07:36:52.484 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-05-27T07:36:52.484 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-05-27T07:36:52.484 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-05-27T07:36:52.484 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-05-27T07:36:52.484 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-05-27T07:36:52.484 INFO:teuthology.orchestra.run.smithi052.stdout: "server_addr": { 2024-05-27T07:36:52.484 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "server_addr", 2024-05-27T07:36:52.484 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "str", 2024-05-27T07:36:52.485 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-05-27T07:36:52.485 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 0, 2024-05-27T07:36:52.485 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "::", 2024-05-27T07:36:52.485 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-05-27T07:36:52.485 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-05-27T07:36:52.485 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-05-27T07:36:52.485 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-05-27T07:36:52.485 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-05-27T07:36:52.485 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-05-27T07:36:52.485 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-05-27T07:36:52.485 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-05-27T07:36:52.485 INFO:teuthology.orchestra.run.smithi052.stdout: "server_port": { 2024-05-27T07:36:52.486 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "server_port", 2024-05-27T07:36:52.486 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "int", 2024-05-27T07:36:52.486 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-05-27T07:36:52.486 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 0, 2024-05-27T07:36:52.486 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "8080", 2024-05-27T07:36:52.486 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-05-27T07:36:52.486 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-05-27T07:36:52.486 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-05-27T07:36:52.486 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-05-27T07:36:52.486 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-05-27T07:36:52.486 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-05-27T07:36:52.486 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-05-27T07:36:52.486 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-05-27T07:36:52.487 INFO:teuthology.orchestra.run.smithi052.stdout: "sqlite3_killpoint": { 2024-05-27T07:36:52.487 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "sqlite3_killpoint", 2024-05-27T07:36:52.487 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "int", 2024-05-27T07:36:52.487 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "dev", 2024-05-27T07:36:52.487 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 1, 2024-05-27T07:36:52.487 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "0", 2024-05-27T07:36:52.487 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-05-27T07:36:52.487 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-05-27T07:36:52.487 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-05-27T07:36:52.487 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-05-27T07:36:52.487 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-05-27T07:36:52.487 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-05-27T07:36:52.487 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-05-27T07:36:52.488 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-05-27T07:36:52.488 INFO:teuthology.orchestra.run.smithi052.stdout: "ssl": { 2024-05-27T07:36:52.488 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "ssl", 2024-05-27T07:36:52.488 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "bool", 2024-05-27T07:36:52.488 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-05-27T07:36:52.488 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 0, 2024-05-27T07:36:52.488 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "True", 2024-05-27T07:36:52.488 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-05-27T07:36:52.488 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-05-27T07:36:52.488 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-05-27T07:36:52.488 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-05-27T07:36:52.488 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-05-27T07:36:52.488 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-05-27T07:36:52.489 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-05-27T07:36:52.489 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-05-27T07:36:52.489 INFO:teuthology.orchestra.run.smithi052.stdout: "ssl_server_port": { 2024-05-27T07:36:52.489 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "ssl_server_port", 2024-05-27T07:36:52.489 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "int", 2024-05-27T07:36:52.489 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-05-27T07:36:52.489 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 0, 2024-05-27T07:36:52.489 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "8443", 2024-05-27T07:36:52.489 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-05-27T07:36:52.489 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-05-27T07:36:52.489 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-05-27T07:36:52.489 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-05-27T07:36:52.489 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-05-27T07:36:52.490 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-05-27T07:36:52.490 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-05-27T07:36:52.490 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-05-27T07:36:52.490 INFO:teuthology.orchestra.run.smithi052.stdout: "standby_behaviour": { 2024-05-27T07:36:52.490 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "standby_behaviour", 2024-05-27T07:36:52.490 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "str", 2024-05-27T07:36:52.490 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-05-27T07:36:52.490 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 0, 2024-05-27T07:36:52.490 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "redirect", 2024-05-27T07:36:52.490 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-05-27T07:36:52.490 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-05-27T07:36:52.490 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [ 2024-05-27T07:36:52.490 INFO:teuthology.orchestra.run.smithi052.stdout: "error", 2024-05-27T07:36:52.491 INFO:teuthology.orchestra.run.smithi052.stdout: "redirect" 2024-05-27T07:36:52.491 INFO:teuthology.orchestra.run.smithi052.stdout: ], 2024-05-27T07:36:52.491 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-05-27T07:36:52.491 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-05-27T07:36:52.491 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-05-27T07:36:52.491 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-05-27T07:36:52.491 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-05-27T07:36:52.491 INFO:teuthology.orchestra.run.smithi052.stdout: "standby_error_status_code": { 2024-05-27T07:36:52.491 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "standby_error_status_code", 2024-05-27T07:36:52.491 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "int", 2024-05-27T07:36:52.491 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-05-27T07:36:52.491 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 0, 2024-05-27T07:36:52.491 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "500", 2024-05-27T07:36:52.491 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "400", 2024-05-27T07:36:52.492 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "599", 2024-05-27T07:36:52.492 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-05-27T07:36:52.492 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-05-27T07:36:52.492 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-05-27T07:36:52.492 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-05-27T07:36:52.492 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-05-27T07:36:52.492 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-05-27T07:36:52.492 INFO:teuthology.orchestra.run.smithi052.stdout: "url_prefix": { 2024-05-27T07:36:52.492 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "url_prefix", 2024-05-27T07:36:52.492 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "str", 2024-05-27T07:36:52.492 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-05-27T07:36:52.492 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 0, 2024-05-27T07:36:52.493 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "", 2024-05-27T07:36:52.493 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-05-27T07:36:52.493 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-05-27T07:36:52.493 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-05-27T07:36:52.493 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-05-27T07:36:52.493 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-05-27T07:36:52.493 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-05-27T07:36:52.493 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-05-27T07:36:52.493 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-05-27T07:36:52.493 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-05-27T07:36:52.493 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-05-27T07:36:52.493 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-05-27T07:36:52.493 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "devicehealth", 2024-05-27T07:36:52.494 INFO:teuthology.orchestra.run.smithi052.stdout: "can_run": true, 2024-05-27T07:36:52.494 INFO:teuthology.orchestra.run.smithi052.stdout: "error_string": "", 2024-05-27T07:36:52.494 INFO:teuthology.orchestra.run.smithi052.stdout: "module_options": { 2024-05-27T07:36:52.494 INFO:teuthology.orchestra.run.smithi052.stdout: "enable_monitoring": { 2024-05-27T07:36:52.494 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "enable_monitoring", 2024-05-27T07:36:52.494 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "bool", 2024-05-27T07:36:52.494 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-05-27T07:36:52.494 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 1, 2024-05-27T07:36:52.494 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "True", 2024-05-27T07:36:52.494 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-05-27T07:36:52.494 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-05-27T07:36:52.494 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-05-27T07:36:52.495 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "monitor device health metrics", 2024-05-27T07:36:52.495 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-05-27T07:36:52.495 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-05-27T07:36:52.495 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-05-27T07:36:52.495 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-05-27T07:36:52.495 INFO:teuthology.orchestra.run.smithi052.stdout: "log_level": { 2024-05-27T07:36:52.495 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "log_level", 2024-05-27T07:36:52.495 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "str", 2024-05-27T07:36:52.495 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-05-27T07:36:52.495 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 1, 2024-05-27T07:36:52.495 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "", 2024-05-27T07:36:52.495 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-05-27T07:36:52.495 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-05-27T07:36:52.496 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [ 2024-05-27T07:36:52.496 INFO:teuthology.orchestra.run.smithi052.stdout: "", 2024-05-27T07:36:52.496 INFO:teuthology.orchestra.run.smithi052.stdout: "critical", 2024-05-27T07:36:52.496 INFO:teuthology.orchestra.run.smithi052.stdout: "debug", 2024-05-27T07:36:52.496 INFO:teuthology.orchestra.run.smithi052.stdout: "error", 2024-05-27T07:36:52.496 INFO:teuthology.orchestra.run.smithi052.stdout: "info", 2024-05-27T07:36:52.496 INFO:teuthology.orchestra.run.smithi052.stdout: "warning" 2024-05-27T07:36:52.496 INFO:teuthology.orchestra.run.smithi052.stdout: ], 2024-05-27T07:36:52.496 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-05-27T07:36:52.496 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-05-27T07:36:52.496 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-05-27T07:36:52.496 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-05-27T07:36:52.496 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-05-27T07:36:52.497 INFO:teuthology.orchestra.run.smithi052.stdout: "log_to_cluster": { 2024-05-27T07:36:52.497 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "log_to_cluster", 2024-05-27T07:36:52.497 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "bool", 2024-05-27T07:36:52.497 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-05-27T07:36:52.497 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 1, 2024-05-27T07:36:52.497 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "False", 2024-05-27T07:36:52.497 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-05-27T07:36:52.497 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-05-27T07:36:52.497 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-05-27T07:36:52.497 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-05-27T07:36:52.497 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-05-27T07:36:52.497 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-05-27T07:36:52.497 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-05-27T07:36:52.498 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-05-27T07:36:52.498 INFO:teuthology.orchestra.run.smithi052.stdout: "log_to_cluster_level": { 2024-05-27T07:36:52.498 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "log_to_cluster_level", 2024-05-27T07:36:52.498 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "str", 2024-05-27T07:36:52.498 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-05-27T07:36:52.498 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 1, 2024-05-27T07:36:52.498 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "info", 2024-05-27T07:36:52.498 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-05-27T07:36:52.498 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-05-27T07:36:52.498 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [ 2024-05-27T07:36:52.498 INFO:teuthology.orchestra.run.smithi052.stdout: "", 2024-05-27T07:36:52.498 INFO:teuthology.orchestra.run.smithi052.stdout: "critical", 2024-05-27T07:36:52.499 INFO:teuthology.orchestra.run.smithi052.stdout: "debug", 2024-05-27T07:36:52.499 INFO:teuthology.orchestra.run.smithi052.stdout: "error", 2024-05-27T07:36:52.499 INFO:teuthology.orchestra.run.smithi052.stdout: "info", 2024-05-27T07:36:52.499 INFO:teuthology.orchestra.run.smithi052.stdout: "warning" 2024-05-27T07:36:52.499 INFO:teuthology.orchestra.run.smithi052.stdout: ], 2024-05-27T07:36:52.499 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-05-27T07:36:52.499 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-05-27T07:36:52.499 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-05-27T07:36:52.499 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-05-27T07:36:52.499 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-05-27T07:36:52.499 INFO:teuthology.orchestra.run.smithi052.stdout: "log_to_file": { 2024-05-27T07:36:52.499 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "log_to_file", 2024-05-27T07:36:52.499 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "bool", 2024-05-27T07:36:52.500 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-05-27T07:36:52.500 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 1, 2024-05-27T07:36:52.500 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "False", 2024-05-27T07:36:52.500 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-05-27T07:36:52.500 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-05-27T07:36:52.500 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-05-27T07:36:52.500 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-05-27T07:36:52.500 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-05-27T07:36:52.500 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-05-27T07:36:52.500 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-05-27T07:36:52.500 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-05-27T07:36:52.500 INFO:teuthology.orchestra.run.smithi052.stdout: "mark_out_threshold": { 2024-05-27T07:36:52.500 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "mark_out_threshold", 2024-05-27T07:36:52.501 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "secs", 2024-05-27T07:36:52.501 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-05-27T07:36:52.501 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 1, 2024-05-27T07:36:52.501 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "2419200", 2024-05-27T07:36:52.501 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-05-27T07:36:52.501 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-05-27T07:36:52.501 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-05-27T07:36:52.501 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "automatically mark OSD if it may fail before this long", 2024-05-27T07:36:52.501 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-05-27T07:36:52.501 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-05-27T07:36:52.501 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-05-27T07:36:52.501 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-05-27T07:36:52.501 INFO:teuthology.orchestra.run.smithi052.stdout: "pool_name": { 2024-05-27T07:36:52.502 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "pool_name", 2024-05-27T07:36:52.502 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "str", 2024-05-27T07:36:52.502 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-05-27T07:36:52.502 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 1, 2024-05-27T07:36:52.502 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "device_health_metrics", 2024-05-27T07:36:52.502 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-05-27T07:36:52.502 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-05-27T07:36:52.502 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-05-27T07:36:52.502 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "name of pool in which to store device health metrics", 2024-05-27T07:36:52.502 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-05-27T07:36:52.502 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-05-27T07:36:52.502 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-05-27T07:36:52.502 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-05-27T07:36:52.503 INFO:teuthology.orchestra.run.smithi052.stdout: "retention_period": { 2024-05-27T07:36:52.503 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "retention_period", 2024-05-27T07:36:52.503 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "secs", 2024-05-27T07:36:52.503 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-05-27T07:36:52.503 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 1, 2024-05-27T07:36:52.503 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "15552000", 2024-05-27T07:36:52.503 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-05-27T07:36:52.503 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-05-27T07:36:52.503 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-05-27T07:36:52.503 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "how long to retain device health metrics", 2024-05-27T07:36:52.503 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-05-27T07:36:52.503 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-05-27T07:36:52.503 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-05-27T07:36:52.504 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-05-27T07:36:52.504 INFO:teuthology.orchestra.run.smithi052.stdout: "scrape_frequency": { 2024-05-27T07:36:52.504 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "scrape_frequency", 2024-05-27T07:36:52.504 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "secs", 2024-05-27T07:36:52.504 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-05-27T07:36:52.504 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 1, 2024-05-27T07:36:52.504 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "86400", 2024-05-27T07:36:52.504 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-05-27T07:36:52.504 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-05-27T07:36:52.504 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-05-27T07:36:52.504 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "how frequently to scrape device health metrics", 2024-05-27T07:36:52.504 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-05-27T07:36:52.504 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-05-27T07:36:52.505 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-05-27T07:36:52.505 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-05-27T07:36:52.505 INFO:teuthology.orchestra.run.smithi052.stdout: "self_heal": { 2024-05-27T07:36:52.505 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "self_heal", 2024-05-27T07:36:52.505 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "bool", 2024-05-27T07:36:52.505 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-05-27T07:36:52.505 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 1, 2024-05-27T07:36:52.505 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "True", 2024-05-27T07:36:52.505 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-05-27T07:36:52.505 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-05-27T07:36:52.505 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-05-27T07:36:52.505 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "preemptively heal cluster around devices that may fail", 2024-05-27T07:36:52.505 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-05-27T07:36:52.506 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-05-27T07:36:52.506 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-05-27T07:36:52.506 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-05-27T07:36:52.506 INFO:teuthology.orchestra.run.smithi052.stdout: "sleep_interval": { 2024-05-27T07:36:52.506 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "sleep_interval", 2024-05-27T07:36:52.506 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "secs", 2024-05-27T07:36:52.506 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-05-27T07:36:52.506 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 1, 2024-05-27T07:36:52.506 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "600", 2024-05-27T07:36:52.506 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-05-27T07:36:52.506 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-05-27T07:36:52.506 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-05-27T07:36:52.506 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "how frequently to wake up and check device health", 2024-05-27T07:36:52.507 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-05-27T07:36:52.507 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-05-27T07:36:52.507 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-05-27T07:36:52.507 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-05-27T07:36:52.507 INFO:teuthology.orchestra.run.smithi052.stdout: "sqlite3_killpoint": { 2024-05-27T07:36:52.507 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "sqlite3_killpoint", 2024-05-27T07:36:52.507 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "int", 2024-05-27T07:36:52.507 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "dev", 2024-05-27T07:36:52.507 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 1, 2024-05-27T07:36:52.508 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "0", 2024-05-27T07:36:52.508 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-05-27T07:36:52.508 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-05-27T07:36:52.508 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-05-27T07:36:52.508 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-05-27T07:36:52.508 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-05-27T07:36:52.508 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-05-27T07:36:52.508 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-05-27T07:36:52.508 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-05-27T07:36:52.508 INFO:teuthology.orchestra.run.smithi052.stdout: "warn_threshold": { 2024-05-27T07:36:52.508 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "warn_threshold", 2024-05-27T07:36:52.508 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "secs", 2024-05-27T07:36:52.508 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-05-27T07:36:52.509 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 1, 2024-05-27T07:36:52.509 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "7257600", 2024-05-27T07:36:52.509 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-05-27T07:36:52.509 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-05-27T07:36:52.509 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-05-27T07:36:52.509 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "raise health warning if OSD may fail before this long", 2024-05-27T07:36:52.509 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-05-27T07:36:52.509 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-05-27T07:36:52.509 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-05-27T07:36:52.509 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-05-27T07:36:52.509 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-05-27T07:36:52.509 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-05-27T07:36:52.509 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-05-27T07:36:52.510 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "diskprediction_local", 2024-05-27T07:36:52.510 INFO:teuthology.orchestra.run.smithi052.stdout: "can_run": true, 2024-05-27T07:36:52.510 INFO:teuthology.orchestra.run.smithi052.stdout: "error_string": "", 2024-05-27T07:36:52.510 INFO:teuthology.orchestra.run.smithi052.stdout: "module_options": { 2024-05-27T07:36:52.510 INFO:teuthology.orchestra.run.smithi052.stdout: "log_level": { 2024-05-27T07:36:52.510 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "log_level", 2024-05-27T07:36:52.510 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "str", 2024-05-27T07:36:52.510 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-05-27T07:36:52.510 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 1, 2024-05-27T07:36:52.510 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "", 2024-05-27T07:36:52.510 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-05-27T07:36:52.510 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-05-27T07:36:52.511 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [ 2024-05-27T07:36:52.511 INFO:teuthology.orchestra.run.smithi052.stdout: "", 2024-05-27T07:36:52.511 INFO:teuthology.orchestra.run.smithi052.stdout: "critical", 2024-05-27T07:36:52.511 INFO:teuthology.orchestra.run.smithi052.stdout: "debug", 2024-05-27T07:36:52.511 INFO:teuthology.orchestra.run.smithi052.stdout: "error", 2024-05-27T07:36:52.511 INFO:teuthology.orchestra.run.smithi052.stdout: "info", 2024-05-27T07:36:52.511 INFO:teuthology.orchestra.run.smithi052.stdout: "warning" 2024-05-27T07:36:52.511 INFO:teuthology.orchestra.run.smithi052.stdout: ], 2024-05-27T07:36:52.511 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-05-27T07:36:52.511 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-05-27T07:36:52.511 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-05-27T07:36:52.511 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-05-27T07:36:52.511 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-05-27T07:36:52.512 INFO:teuthology.orchestra.run.smithi052.stdout: "log_to_cluster": { 2024-05-27T07:36:52.512 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "log_to_cluster", 2024-05-27T07:36:52.512 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "bool", 2024-05-27T07:36:52.512 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-05-27T07:36:52.512 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 1, 2024-05-27T07:36:52.512 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "False", 2024-05-27T07:36:52.512 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-05-27T07:36:52.512 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-05-27T07:36:52.512 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-05-27T07:36:52.512 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-05-27T07:36:52.512 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-05-27T07:36:52.512 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-05-27T07:36:52.512 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-05-27T07:36:52.513 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-05-27T07:36:52.513 INFO:teuthology.orchestra.run.smithi052.stdout: "log_to_cluster_level": { 2024-05-27T07:36:52.513 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "log_to_cluster_level", 2024-05-27T07:36:52.513 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "str", 2024-05-27T07:36:52.513 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-05-27T07:36:52.513 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 1, 2024-05-27T07:36:52.513 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "info", 2024-05-27T07:36:52.513 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-05-27T07:36:52.513 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-05-27T07:36:52.513 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [ 2024-05-27T07:36:52.513 INFO:teuthology.orchestra.run.smithi052.stdout: "", 2024-05-27T07:36:52.513 INFO:teuthology.orchestra.run.smithi052.stdout: "critical", 2024-05-27T07:36:52.514 INFO:teuthology.orchestra.run.smithi052.stdout: "debug", 2024-05-27T07:36:52.514 INFO:teuthology.orchestra.run.smithi052.stdout: "error", 2024-05-27T07:36:52.514 INFO:teuthology.orchestra.run.smithi052.stdout: "info", 2024-05-27T07:36:52.514 INFO:teuthology.orchestra.run.smithi052.stdout: "warning" 2024-05-27T07:36:52.514 INFO:teuthology.orchestra.run.smithi052.stdout: ], 2024-05-27T07:36:52.514 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-05-27T07:36:52.514 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-05-27T07:36:52.514 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-05-27T07:36:52.514 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-05-27T07:36:52.514 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-05-27T07:36:52.514 INFO:teuthology.orchestra.run.smithi052.stdout: "log_to_file": { 2024-05-27T07:36:52.514 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "log_to_file", 2024-05-27T07:36:52.515 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "bool", 2024-05-27T07:36:52.515 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-05-27T07:36:52.515 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 1, 2024-05-27T07:36:52.515 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "False", 2024-05-27T07:36:52.515 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-05-27T07:36:52.515 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-05-27T07:36:52.515 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-05-27T07:36:52.515 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-05-27T07:36:52.515 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-05-27T07:36:52.515 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-05-27T07:36:52.515 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-05-27T07:36:52.515 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-05-27T07:36:52.515 INFO:teuthology.orchestra.run.smithi052.stdout: "predict_interval": { 2024-05-27T07:36:52.516 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "predict_interval", 2024-05-27T07:36:52.516 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "str", 2024-05-27T07:36:52.516 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-05-27T07:36:52.516 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 0, 2024-05-27T07:36:52.516 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "86400", 2024-05-27T07:36:52.516 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-05-27T07:36:52.516 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-05-27T07:36:52.516 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-05-27T07:36:52.516 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-05-27T07:36:52.516 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-05-27T07:36:52.516 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-05-27T07:36:52.516 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-05-27T07:36:52.516 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-05-27T07:36:52.517 INFO:teuthology.orchestra.run.smithi052.stdout: "predictor_model": { 2024-05-27T07:36:52.517 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "predictor_model", 2024-05-27T07:36:52.517 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "str", 2024-05-27T07:36:52.517 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-05-27T07:36:52.517 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 0, 2024-05-27T07:36:52.517 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "prophetstor", 2024-05-27T07:36:52.517 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-05-27T07:36:52.517 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-05-27T07:36:52.517 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-05-27T07:36:52.517 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-05-27T07:36:52.517 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-05-27T07:36:52.517 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-05-27T07:36:52.517 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-05-27T07:36:52.518 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-05-27T07:36:52.518 INFO:teuthology.orchestra.run.smithi052.stdout: "sleep_interval": { 2024-05-27T07:36:52.518 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "sleep_interval", 2024-05-27T07:36:52.518 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "str", 2024-05-27T07:36:52.518 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-05-27T07:36:52.518 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 0, 2024-05-27T07:36:52.518 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "600", 2024-05-27T07:36:52.518 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-05-27T07:36:52.518 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-05-27T07:36:52.518 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-05-27T07:36:52.518 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-05-27T07:36:52.518 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-05-27T07:36:52.518 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-05-27T07:36:52.519 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-05-27T07:36:52.519 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-05-27T07:36:52.519 INFO:teuthology.orchestra.run.smithi052.stdout: "sqlite3_killpoint": { 2024-05-27T07:36:52.519 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "sqlite3_killpoint", 2024-05-27T07:36:52.519 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "int", 2024-05-27T07:36:52.519 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "dev", 2024-05-27T07:36:52.519 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 1, 2024-05-27T07:36:52.519 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "0", 2024-05-27T07:36:52.519 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-05-27T07:36:52.519 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-05-27T07:36:52.519 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-05-27T07:36:52.519 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-05-27T07:36:52.519 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-05-27T07:36:52.520 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-05-27T07:36:52.520 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-05-27T07:36:52.520 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-05-27T07:36:52.520 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-05-27T07:36:52.520 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-05-27T07:36:52.520 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-05-27T07:36:52.520 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "influx", 2024-05-27T07:36:52.520 INFO:teuthology.orchestra.run.smithi052.stdout: "can_run": false, 2024-05-27T07:36:52.520 INFO:teuthology.orchestra.run.smithi052.stdout: "error_string": "influxdb python module not found", 2024-05-27T07:36:52.520 INFO:teuthology.orchestra.run.smithi052.stdout: "module_options": { 2024-05-27T07:36:52.520 INFO:teuthology.orchestra.run.smithi052.stdout: "batch_size": { 2024-05-27T07:36:52.520 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "batch_size", 2024-05-27T07:36:52.521 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "int", 2024-05-27T07:36:52.521 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-05-27T07:36:52.521 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 0, 2024-05-27T07:36:52.521 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "5000", 2024-05-27T07:36:52.521 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-05-27T07:36:52.521 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-05-27T07:36:52.521 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-05-27T07:36:52.521 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "How big batches of data points should be when sending to InfluxDB.", 2024-05-27T07:36:52.521 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-05-27T07:36:52.521 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-05-27T07:36:52.521 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-05-27T07:36:52.521 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-05-27T07:36:52.522 INFO:teuthology.orchestra.run.smithi052.stdout: "database": { 2024-05-27T07:36:52.522 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "database", 2024-05-27T07:36:52.522 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "str", 2024-05-27T07:36:52.522 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-05-27T07:36:52.522 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 0, 2024-05-27T07:36:52.522 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "ceph", 2024-05-27T07:36:52.522 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-05-27T07:36:52.522 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-05-27T07:36:52.522 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-05-27T07:36:52.522 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.", 2024-05-27T07:36:52.522 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-05-27T07:36:52.522 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-05-27T07:36:52.523 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-05-27T07:36:52.523 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-05-27T07:36:52.523 INFO:teuthology.orchestra.run.smithi052.stdout: "hostname": { 2024-05-27T07:36:52.523 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "hostname", 2024-05-27T07:36:52.523 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "str", 2024-05-27T07:36:52.523 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-05-27T07:36:52.523 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 0, 2024-05-27T07:36:52.523 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "", 2024-05-27T07:36:52.523 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-05-27T07:36:52.523 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-05-27T07:36:52.523 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-05-27T07:36:52.523 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "InfluxDB server hostname", 2024-05-27T07:36:52.524 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-05-27T07:36:52.524 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-05-27T07:36:52.524 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-05-27T07:36:52.524 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-05-27T07:36:52.524 INFO:teuthology.orchestra.run.smithi052.stdout: "interval": { 2024-05-27T07:36:52.524 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "interval", 2024-05-27T07:36:52.524 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "secs", 2024-05-27T07:36:52.524 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-05-27T07:36:52.524 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 0, 2024-05-27T07:36:52.524 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "30", 2024-05-27T07:36:52.524 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "5", 2024-05-27T07:36:52.524 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-05-27T07:36:52.524 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-05-27T07:36:52.525 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "Time between reports to InfluxDB. Default 30 seconds.", 2024-05-27T07:36:52.525 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-05-27T07:36:52.525 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-05-27T07:36:52.525 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-05-27T07:36:52.525 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-05-27T07:36:52.525 INFO:teuthology.orchestra.run.smithi052.stdout: "log_level": { 2024-05-27T07:36:52.525 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "log_level", 2024-05-27T07:36:52.525 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "str", 2024-05-27T07:36:52.525 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-05-27T07:36:52.525 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 1, 2024-05-27T07:36:52.525 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "", 2024-05-27T07:36:52.525 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-05-27T07:36:52.525 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-05-27T07:36:52.526 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [ 2024-05-27T07:36:52.526 INFO:teuthology.orchestra.run.smithi052.stdout: "", 2024-05-27T07:36:52.526 INFO:teuthology.orchestra.run.smithi052.stdout: "critical", 2024-05-27T07:36:52.526 INFO:teuthology.orchestra.run.smithi052.stdout: "debug", 2024-05-27T07:36:52.526 INFO:teuthology.orchestra.run.smithi052.stdout: "error", 2024-05-27T07:36:52.526 INFO:teuthology.orchestra.run.smithi052.stdout: "info", 2024-05-27T07:36:52.526 INFO:teuthology.orchestra.run.smithi052.stdout: "warning" 2024-05-27T07:36:52.526 INFO:teuthology.orchestra.run.smithi052.stdout: ], 2024-05-27T07:36:52.526 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-05-27T07:36:52.526 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-05-27T07:36:52.526 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-05-27T07:36:52.526 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-05-27T07:36:52.526 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-05-27T07:36:52.527 INFO:teuthology.orchestra.run.smithi052.stdout: "log_to_cluster": { 2024-05-27T07:36:52.527 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "log_to_cluster", 2024-05-27T07:36:52.527 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "bool", 2024-05-27T07:36:52.527 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-05-27T07:36:52.527 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 1, 2024-05-27T07:36:52.527 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "False", 2024-05-27T07:36:52.527 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-05-27T07:36:52.527 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-05-27T07:36:52.527 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-05-27T07:36:52.527 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-05-27T07:36:52.527 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-05-27T07:36:52.527 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-05-27T07:36:52.528 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-05-27T07:36:52.528 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-05-27T07:36:52.528 INFO:teuthology.orchestra.run.smithi052.stdout: "log_to_cluster_level": { 2024-05-27T07:36:52.528 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "log_to_cluster_level", 2024-05-27T07:36:52.528 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "str", 2024-05-27T07:36:52.528 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-05-27T07:36:52.528 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 1, 2024-05-27T07:36:52.528 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "info", 2024-05-27T07:36:52.528 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-05-27T07:36:52.528 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-05-27T07:36:52.528 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [ 2024-05-27T07:36:52.528 INFO:teuthology.orchestra.run.smithi052.stdout: "", 2024-05-27T07:36:52.528 INFO:teuthology.orchestra.run.smithi052.stdout: "critical", 2024-05-27T07:36:52.528 INFO:teuthology.orchestra.run.smithi052.stdout: "debug", 2024-05-27T07:36:52.529 INFO:teuthology.orchestra.run.smithi052.stdout: "error", 2024-05-27T07:36:52.529 INFO:teuthology.orchestra.run.smithi052.stdout: "info", 2024-05-27T07:36:52.529 INFO:teuthology.orchestra.run.smithi052.stdout: "warning" 2024-05-27T07:36:52.529 INFO:teuthology.orchestra.run.smithi052.stdout: ], 2024-05-27T07:36:52.529 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-05-27T07:36:52.529 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-05-27T07:36:52.529 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-05-27T07:36:52.529 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-05-27T07:36:52.529 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-05-27T07:36:52.529 INFO:teuthology.orchestra.run.smithi052.stdout: "log_to_file": { 2024-05-27T07:36:52.529 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "log_to_file", 2024-05-27T07:36:52.529 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "bool", 2024-05-27T07:36:52.530 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-05-27T07:36:52.530 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 1, 2024-05-27T07:36:52.530 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "False", 2024-05-27T07:36:52.530 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-05-27T07:36:52.530 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-05-27T07:36:52.530 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-05-27T07:36:52.530 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-05-27T07:36:52.530 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-05-27T07:36:52.530 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-05-27T07:36:52.530 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-05-27T07:36:52.530 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-05-27T07:36:52.530 INFO:teuthology.orchestra.run.smithi052.stdout: "password": { 2024-05-27T07:36:52.530 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "password", 2024-05-27T07:36:52.531 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "str", 2024-05-27T07:36:52.531 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-05-27T07:36:52.531 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 0, 2024-05-27T07:36:52.531 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "", 2024-05-27T07:36:52.531 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-05-27T07:36:52.531 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-05-27T07:36:52.531 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-05-27T07:36:52.531 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "password of InfluxDB server user", 2024-05-27T07:36:52.531 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-05-27T07:36:52.531 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-05-27T07:36:52.531 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-05-27T07:36:52.531 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-05-27T07:36:52.531 INFO:teuthology.orchestra.run.smithi052.stdout: "port": { 2024-05-27T07:36:52.532 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "port", 2024-05-27T07:36:52.532 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "int", 2024-05-27T07:36:52.532 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-05-27T07:36:52.532 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 0, 2024-05-27T07:36:52.532 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "8086", 2024-05-27T07:36:52.532 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-05-27T07:36:52.532 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-05-27T07:36:52.532 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-05-27T07:36:52.532 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "InfluxDB server port", 2024-05-27T07:36:52.532 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-05-27T07:36:52.532 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-05-27T07:36:52.532 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-05-27T07:36:52.533 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-05-27T07:36:52.533 INFO:teuthology.orchestra.run.smithi052.stdout: "sqlite3_killpoint": { 2024-05-27T07:36:52.533 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "sqlite3_killpoint", 2024-05-27T07:36:52.533 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "int", 2024-05-27T07:36:52.533 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "dev", 2024-05-27T07:36:52.533 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 1, 2024-05-27T07:36:52.533 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "0", 2024-05-27T07:36:52.533 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-05-27T07:36:52.533 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-05-27T07:36:52.533 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-05-27T07:36:52.533 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-05-27T07:36:52.533 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-05-27T07:36:52.533 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-05-27T07:36:52.534 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-05-27T07:36:52.534 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-05-27T07:36:52.534 INFO:teuthology.orchestra.run.smithi052.stdout: "ssl": { 2024-05-27T07:36:52.534 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "ssl", 2024-05-27T07:36:52.534 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "str", 2024-05-27T07:36:52.534 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-05-27T07:36:52.534 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 0, 2024-05-27T07:36:52.534 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "false", 2024-05-27T07:36:52.534 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-05-27T07:36:52.534 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-05-27T07:36:52.534 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-05-27T07:36:52.534 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "Use https connection for InfluxDB server. Use \"true\" or \"false\".", 2024-05-27T07:36:52.534 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-05-27T07:36:52.535 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-05-27T07:36:52.535 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-05-27T07:36:52.535 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-05-27T07:36:52.535 INFO:teuthology.orchestra.run.smithi052.stdout: "threads": { 2024-05-27T07:36:52.535 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "threads", 2024-05-27T07:36:52.535 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "int", 2024-05-27T07:36:52.535 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-05-27T07:36:52.535 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 0, 2024-05-27T07:36:52.535 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "5", 2024-05-27T07:36:52.535 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "1", 2024-05-27T07:36:52.535 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "32", 2024-05-27T07:36:52.536 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-05-27T07:36:52.536 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "How many worker threads should be spawned for sending data to InfluxDB.", 2024-05-27T07:36:52.536 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-05-27T07:36:52.536 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-05-27T07:36:52.536 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-05-27T07:36:52.536 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-05-27T07:36:52.536 INFO:teuthology.orchestra.run.smithi052.stdout: "username": { 2024-05-27T07:36:52.536 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "username", 2024-05-27T07:36:52.536 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "str", 2024-05-27T07:36:52.536 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-05-27T07:36:52.536 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 0, 2024-05-27T07:36:52.536 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "", 2024-05-27T07:36:52.536 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-05-27T07:36:52.537 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-05-27T07:36:52.537 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-05-27T07:36:52.537 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "username of InfluxDB server user", 2024-05-27T07:36:52.537 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-05-27T07:36:52.537 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-05-27T07:36:52.537 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-05-27T07:36:52.537 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-05-27T07:36:52.537 INFO:teuthology.orchestra.run.smithi052.stdout: "verify_ssl": { 2024-05-27T07:36:52.537 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "verify_ssl", 2024-05-27T07:36:52.537 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "str", 2024-05-27T07:36:52.537 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-05-27T07:36:52.537 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 0, 2024-05-27T07:36:52.538 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "true", 2024-05-27T07:36:52.538 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-05-27T07:36:52.538 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-05-27T07:36:52.538 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-05-27T07:36:52.538 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "Verify https cert for InfluxDB server. Use \"true\" or \"false\".", 2024-05-27T07:36:52.538 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-05-27T07:36:52.538 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-05-27T07:36:52.538 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-05-27T07:36:52.538 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-05-27T07:36:52.538 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-05-27T07:36:52.538 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-05-27T07:36:52.538 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-05-27T07:36:52.538 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "insights", 2024-05-27T07:36:52.539 INFO:teuthology.orchestra.run.smithi052.stdout: "can_run": true, 2024-05-27T07:36:52.539 INFO:teuthology.orchestra.run.smithi052.stdout: "error_string": "", 2024-05-27T07:36:52.539 INFO:teuthology.orchestra.run.smithi052.stdout: "module_options": { 2024-05-27T07:36:52.539 INFO:teuthology.orchestra.run.smithi052.stdout: "log_level": { 2024-05-27T07:36:52.539 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "log_level", 2024-05-27T07:36:52.539 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "str", 2024-05-27T07:36:52.539 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-05-27T07:36:52.539 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 1, 2024-05-27T07:36:52.539 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "", 2024-05-27T07:36:52.539 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-05-27T07:36:52.539 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-05-27T07:36:52.539 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [ 2024-05-27T07:36:52.540 INFO:teuthology.orchestra.run.smithi052.stdout: "", 2024-05-27T07:36:52.540 INFO:teuthology.orchestra.run.smithi052.stdout: "critical", 2024-05-27T07:36:52.540 INFO:teuthology.orchestra.run.smithi052.stdout: "debug", 2024-05-27T07:36:52.540 INFO:teuthology.orchestra.run.smithi052.stdout: "error", 2024-05-27T07:36:52.540 INFO:teuthology.orchestra.run.smithi052.stdout: "info", 2024-05-27T07:36:52.540 INFO:teuthology.orchestra.run.smithi052.stdout: "warning" 2024-05-27T07:36:52.540 INFO:teuthology.orchestra.run.smithi052.stdout: ], 2024-05-27T07:36:52.540 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-05-27T07:36:52.540 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-05-27T07:36:52.540 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-05-27T07:36:52.540 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-05-27T07:36:52.540 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-05-27T07:36:52.541 INFO:teuthology.orchestra.run.smithi052.stdout: "log_to_cluster": { 2024-05-27T07:36:52.541 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "log_to_cluster", 2024-05-27T07:36:52.541 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "bool", 2024-05-27T07:36:52.541 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-05-27T07:36:52.541 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 1, 2024-05-27T07:36:52.541 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "False", 2024-05-27T07:36:52.541 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-05-27T07:36:52.541 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-05-27T07:36:52.541 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-05-27T07:36:52.541 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-05-27T07:36:52.541 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-05-27T07:36:52.541 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-05-27T07:36:52.541 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-05-27T07:36:52.542 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-05-27T07:36:52.542 INFO:teuthology.orchestra.run.smithi052.stdout: "log_to_cluster_level": { 2024-05-27T07:36:52.542 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "log_to_cluster_level", 2024-05-27T07:36:52.542 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "str", 2024-05-27T07:36:52.542 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-05-27T07:36:52.542 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 1, 2024-05-27T07:36:52.542 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "info", 2024-05-27T07:36:52.542 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-05-27T07:36:52.542 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-05-27T07:36:52.542 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [ 2024-05-27T07:36:52.542 INFO:teuthology.orchestra.run.smithi052.stdout: "", 2024-05-27T07:36:52.542 INFO:teuthology.orchestra.run.smithi052.stdout: "critical", 2024-05-27T07:36:52.543 INFO:teuthology.orchestra.run.smithi052.stdout: "debug", 2024-05-27T07:36:52.543 INFO:teuthology.orchestra.run.smithi052.stdout: "error", 2024-05-27T07:36:52.543 INFO:teuthology.orchestra.run.smithi052.stdout: "info", 2024-05-27T07:36:52.543 INFO:teuthology.orchestra.run.smithi052.stdout: "warning" 2024-05-27T07:36:52.543 INFO:teuthology.orchestra.run.smithi052.stdout: ], 2024-05-27T07:36:52.543 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-05-27T07:36:52.543 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-05-27T07:36:52.543 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-05-27T07:36:52.543 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-05-27T07:36:52.543 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-05-27T07:36:52.543 INFO:teuthology.orchestra.run.smithi052.stdout: "log_to_file": { 2024-05-27T07:36:52.543 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "log_to_file", 2024-05-27T07:36:52.543 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "bool", 2024-05-27T07:36:52.544 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-05-27T07:36:52.544 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 1, 2024-05-27T07:36:52.544 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "False", 2024-05-27T07:36:52.544 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-05-27T07:36:52.544 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-05-27T07:36:52.544 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-05-27T07:36:52.544 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-05-27T07:36:52.544 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-05-27T07:36:52.544 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-05-27T07:36:52.544 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-05-27T07:36:52.544 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-05-27T07:36:52.544 INFO:teuthology.orchestra.run.smithi052.stdout: "sqlite3_killpoint": { 2024-05-27T07:36:52.544 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "sqlite3_killpoint", 2024-05-27T07:36:52.545 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "int", 2024-05-27T07:36:52.545 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "dev", 2024-05-27T07:36:52.545 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 1, 2024-05-27T07:36:52.545 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "0", 2024-05-27T07:36:52.545 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-05-27T07:36:52.545 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-05-27T07:36:52.545 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-05-27T07:36:52.545 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-05-27T07:36:52.545 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-05-27T07:36:52.545 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-05-27T07:36:52.545 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-05-27T07:36:52.545 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-05-27T07:36:52.545 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-05-27T07:36:52.545 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-05-27T07:36:52.546 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-05-27T07:36:52.546 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "iostat", 2024-05-27T07:36:52.546 INFO:teuthology.orchestra.run.smithi052.stdout: "can_run": true, 2024-05-27T07:36:52.546 INFO:teuthology.orchestra.run.smithi052.stdout: "error_string": "", 2024-05-27T07:36:52.546 INFO:teuthology.orchestra.run.smithi052.stdout: "module_options": { 2024-05-27T07:36:52.546 INFO:teuthology.orchestra.run.smithi052.stdout: "log_level": { 2024-05-27T07:36:52.546 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "log_level", 2024-05-27T07:36:52.546 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "str", 2024-05-27T07:36:52.546 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-05-27T07:36:52.546 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 1, 2024-05-27T07:36:52.546 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "", 2024-05-27T07:36:52.546 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-05-27T07:36:52.547 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-05-27T07:36:52.547 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [ 2024-05-27T07:36:52.547 INFO:teuthology.orchestra.run.smithi052.stdout: "", 2024-05-27T07:36:52.547 INFO:teuthology.orchestra.run.smithi052.stdout: "critical", 2024-05-27T07:36:52.547 INFO:teuthology.orchestra.run.smithi052.stdout: "debug", 2024-05-27T07:36:52.547 INFO:teuthology.orchestra.run.smithi052.stdout: "error", 2024-05-27T07:36:52.547 INFO:teuthology.orchestra.run.smithi052.stdout: "info", 2024-05-27T07:36:52.547 INFO:teuthology.orchestra.run.smithi052.stdout: "warning" 2024-05-27T07:36:52.547 INFO:teuthology.orchestra.run.smithi052.stdout: ], 2024-05-27T07:36:52.547 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-05-27T07:36:52.547 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-05-27T07:36:52.547 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-05-27T07:36:52.547 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-05-27T07:36:52.548 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-05-27T07:36:52.548 INFO:teuthology.orchestra.run.smithi052.stdout: "log_to_cluster": { 2024-05-27T07:36:52.548 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "log_to_cluster", 2024-05-27T07:36:52.548 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "bool", 2024-05-27T07:36:52.548 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-05-27T07:36:52.548 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 1, 2024-05-27T07:36:52.548 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "False", 2024-05-27T07:36:52.548 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-05-27T07:36:52.548 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-05-27T07:36:52.548 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-05-27T07:36:52.548 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-05-27T07:36:52.548 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-05-27T07:36:52.549 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-05-27T07:36:52.549 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-05-27T07:36:52.549 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-05-27T07:36:52.549 INFO:teuthology.orchestra.run.smithi052.stdout: "log_to_cluster_level": { 2024-05-27T07:36:52.549 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "log_to_cluster_level", 2024-05-27T07:36:52.549 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "str", 2024-05-27T07:36:52.549 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-05-27T07:36:52.549 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 1, 2024-05-27T07:36:52.549 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "info", 2024-05-27T07:36:52.549 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-05-27T07:36:52.549 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-05-27T07:36:52.549 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [ 2024-05-27T07:36:52.549 INFO:teuthology.orchestra.run.smithi052.stdout: "", 2024-05-27T07:36:52.550 INFO:teuthology.orchestra.run.smithi052.stdout: "critical", 2024-05-27T07:36:52.550 INFO:teuthology.orchestra.run.smithi052.stdout: "debug", 2024-05-27T07:36:52.550 INFO:teuthology.orchestra.run.smithi052.stdout: "error", 2024-05-27T07:36:52.550 INFO:teuthology.orchestra.run.smithi052.stdout: "info", 2024-05-27T07:36:52.550 INFO:teuthology.orchestra.run.smithi052.stdout: "warning" 2024-05-27T07:36:52.550 INFO:teuthology.orchestra.run.smithi052.stdout: ], 2024-05-27T07:36:52.550 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-05-27T07:36:52.550 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-05-27T07:36:52.550 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-05-27T07:36:52.550 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-05-27T07:36:52.550 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-05-27T07:36:52.550 INFO:teuthology.orchestra.run.smithi052.stdout: "log_to_file": { 2024-05-27T07:36:52.550 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "log_to_file", 2024-05-27T07:36:52.551 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "bool", 2024-05-27T07:36:52.551 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-05-27T07:36:52.551 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 1, 2024-05-27T07:36:52.551 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "False", 2024-05-27T07:36:52.551 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-05-27T07:36:52.551 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-05-27T07:36:52.551 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-05-27T07:36:52.551 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-05-27T07:36:52.551 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-05-27T07:36:52.551 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-05-27T07:36:52.551 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-05-27T07:36:52.551 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-05-27T07:36:52.551 INFO:teuthology.orchestra.run.smithi052.stdout: "sqlite3_killpoint": { 2024-05-27T07:36:52.552 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "sqlite3_killpoint", 2024-05-27T07:36:52.552 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "int", 2024-05-27T07:36:52.552 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "dev", 2024-05-27T07:36:52.552 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 1, 2024-05-27T07:36:52.552 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "0", 2024-05-27T07:36:52.552 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-05-27T07:36:52.552 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-05-27T07:36:52.552 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-05-27T07:36:52.552 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-05-27T07:36:52.552 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-05-27T07:36:52.552 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-05-27T07:36:52.552 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-05-27T07:36:52.553 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-05-27T07:36:52.553 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-05-27T07:36:52.553 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-05-27T07:36:52.553 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-05-27T07:36:52.553 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "k8sevents", 2024-05-27T07:36:52.553 INFO:teuthology.orchestra.run.smithi052.stdout: "can_run": true, 2024-05-27T07:36:52.553 INFO:teuthology.orchestra.run.smithi052.stdout: "error_string": "", 2024-05-27T07:36:52.553 INFO:teuthology.orchestra.run.smithi052.stdout: "module_options": { 2024-05-27T07:36:52.553 INFO:teuthology.orchestra.run.smithi052.stdout: "ceph_event_retention_days": { 2024-05-27T07:36:52.553 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "ceph_event_retention_days", 2024-05-27T07:36:52.553 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "int", 2024-05-27T07:36:52.553 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-05-27T07:36:52.553 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 0, 2024-05-27T07:36:52.554 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "7", 2024-05-27T07:36:52.554 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-05-27T07:36:52.554 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-05-27T07:36:52.554 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-05-27T07:36:52.554 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "Days to hold ceph event information within local cache", 2024-05-27T07:36:52.554 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-05-27T07:36:52.554 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-05-27T07:36:52.554 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-05-27T07:36:52.554 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-05-27T07:36:52.554 INFO:teuthology.orchestra.run.smithi052.stdout: "config_check_secs": { 2024-05-27T07:36:52.554 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "config_check_secs", 2024-05-27T07:36:52.554 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "int", 2024-05-27T07:36:52.554 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-05-27T07:36:52.555 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 0, 2024-05-27T07:36:52.555 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "10", 2024-05-27T07:36:52.555 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "10", 2024-05-27T07:36:52.555 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-05-27T07:36:52.555 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-05-27T07:36:52.555 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "interval (secs) to check for cluster configuration changes", 2024-05-27T07:36:52.555 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-05-27T07:36:52.555 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-05-27T07:36:52.555 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-05-27T07:36:52.555 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-05-27T07:36:52.555 INFO:teuthology.orchestra.run.smithi052.stdout: "log_level": { 2024-05-27T07:36:52.555 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "log_level", 2024-05-27T07:36:52.556 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "str", 2024-05-27T07:36:52.556 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-05-27T07:36:52.556 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 1, 2024-05-27T07:36:52.556 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "", 2024-05-27T07:36:52.556 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-05-27T07:36:52.556 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-05-27T07:36:52.556 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [ 2024-05-27T07:36:52.556 INFO:teuthology.orchestra.run.smithi052.stdout: "", 2024-05-27T07:36:52.556 INFO:teuthology.orchestra.run.smithi052.stdout: "critical", 2024-05-27T07:36:52.556 INFO:teuthology.orchestra.run.smithi052.stdout: "debug", 2024-05-27T07:36:52.556 INFO:teuthology.orchestra.run.smithi052.stdout: "error", 2024-05-27T07:36:52.556 INFO:teuthology.orchestra.run.smithi052.stdout: "info", 2024-05-27T07:36:52.556 INFO:teuthology.orchestra.run.smithi052.stdout: "warning" 2024-05-27T07:36:52.557 INFO:teuthology.orchestra.run.smithi052.stdout: ], 2024-05-27T07:36:52.557 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-05-27T07:36:52.557 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-05-27T07:36:52.557 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-05-27T07:36:52.557 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-05-27T07:36:52.557 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-05-27T07:36:52.557 INFO:teuthology.orchestra.run.smithi052.stdout: "log_to_cluster": { 2024-05-27T07:36:52.557 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "log_to_cluster", 2024-05-27T07:36:52.557 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "bool", 2024-05-27T07:36:52.557 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-05-27T07:36:52.557 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 1, 2024-05-27T07:36:52.557 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "False", 2024-05-27T07:36:52.557 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-05-27T07:36:52.558 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-05-27T07:36:52.558 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-05-27T07:36:52.558 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-05-27T07:36:52.558 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-05-27T07:36:52.558 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-05-27T07:36:52.558 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-05-27T07:36:52.558 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-05-27T07:36:52.558 INFO:teuthology.orchestra.run.smithi052.stdout: "log_to_cluster_level": { 2024-05-27T07:36:52.558 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "log_to_cluster_level", 2024-05-27T07:36:52.558 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "str", 2024-05-27T07:36:52.558 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-05-27T07:36:52.559 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 1, 2024-05-27T07:36:52.559 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "info", 2024-05-27T07:36:52.559 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-05-27T07:36:52.559 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-05-27T07:36:52.559 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [ 2024-05-27T07:36:52.559 INFO:teuthology.orchestra.run.smithi052.stdout: "", 2024-05-27T07:36:52.559 INFO:teuthology.orchestra.run.smithi052.stdout: "critical", 2024-05-27T07:36:52.559 INFO:teuthology.orchestra.run.smithi052.stdout: "debug", 2024-05-27T07:36:52.559 INFO:teuthology.orchestra.run.smithi052.stdout: "error", 2024-05-27T07:36:52.559 INFO:teuthology.orchestra.run.smithi052.stdout: "info", 2024-05-27T07:36:52.559 INFO:teuthology.orchestra.run.smithi052.stdout: "warning" 2024-05-27T07:36:52.559 INFO:teuthology.orchestra.run.smithi052.stdout: ], 2024-05-27T07:36:52.559 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-05-27T07:36:52.560 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-05-27T07:36:52.560 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-05-27T07:36:52.560 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-05-27T07:36:52.560 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-05-27T07:36:52.560 INFO:teuthology.orchestra.run.smithi052.stdout: "log_to_file": { 2024-05-27T07:36:52.560 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "log_to_file", 2024-05-27T07:36:52.560 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "bool", 2024-05-27T07:36:52.560 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-05-27T07:36:52.560 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 1, 2024-05-27T07:36:52.560 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "False", 2024-05-27T07:36:52.560 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-05-27T07:36:52.560 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-05-27T07:36:52.560 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-05-27T07:36:52.561 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-05-27T07:36:52.561 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-05-27T07:36:52.561 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-05-27T07:36:52.561 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-05-27T07:36:52.561 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-05-27T07:36:52.561 INFO:teuthology.orchestra.run.smithi052.stdout: "sqlite3_killpoint": { 2024-05-27T07:36:52.561 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "sqlite3_killpoint", 2024-05-27T07:36:52.561 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "int", 2024-05-27T07:36:52.561 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "dev", 2024-05-27T07:36:52.561 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 1, 2024-05-27T07:36:52.561 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "0", 2024-05-27T07:36:52.561 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-05-27T07:36:52.561 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-05-27T07:36:52.562 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-05-27T07:36:52.562 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-05-27T07:36:52.562 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-05-27T07:36:52.562 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-05-27T07:36:52.562 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-05-27T07:36:52.562 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-05-27T07:36:52.562 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-05-27T07:36:52.562 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-05-27T07:36:52.562 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-05-27T07:36:52.562 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "localpool", 2024-05-27T07:36:52.562 INFO:teuthology.orchestra.run.smithi052.stdout: "can_run": true, 2024-05-27T07:36:52.562 INFO:teuthology.orchestra.run.smithi052.stdout: "error_string": "", 2024-05-27T07:36:52.563 INFO:teuthology.orchestra.run.smithi052.stdout: "module_options": { 2024-05-27T07:36:52.563 INFO:teuthology.orchestra.run.smithi052.stdout: "failure_domain": { 2024-05-27T07:36:52.563 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "failure_domain", 2024-05-27T07:36:52.563 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "str", 2024-05-27T07:36:52.563 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-05-27T07:36:52.563 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 1, 2024-05-27T07:36:52.563 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "host", 2024-05-27T07:36:52.563 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-05-27T07:36:52.563 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-05-27T07:36:52.563 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-05-27T07:36:52.563 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "failure domain for any created local pool", 2024-05-27T07:36:52.563 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "what failure domain we should separate data replicas across.", 2024-05-27T07:36:52.563 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-05-27T07:36:52.564 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-05-27T07:36:52.564 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-05-27T07:36:52.564 INFO:teuthology.orchestra.run.smithi052.stdout: "log_level": { 2024-05-27T07:36:52.564 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "log_level", 2024-05-27T07:36:52.564 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "str", 2024-05-27T07:36:52.564 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-05-27T07:36:52.564 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 1, 2024-05-27T07:36:52.564 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "", 2024-05-27T07:36:52.564 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-05-27T07:36:52.564 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-05-27T07:36:52.564 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [ 2024-05-27T07:36:52.564 INFO:teuthology.orchestra.run.smithi052.stdout: "", 2024-05-27T07:36:52.564 INFO:teuthology.orchestra.run.smithi052.stdout: "critical", 2024-05-27T07:36:52.565 INFO:teuthology.orchestra.run.smithi052.stdout: "debug", 2024-05-27T07:36:52.565 INFO:teuthology.orchestra.run.smithi052.stdout: "error", 2024-05-27T07:36:52.565 INFO:teuthology.orchestra.run.smithi052.stdout: "info", 2024-05-27T07:36:52.565 INFO:teuthology.orchestra.run.smithi052.stdout: "warning" 2024-05-27T07:36:52.565 INFO:teuthology.orchestra.run.smithi052.stdout: ], 2024-05-27T07:36:52.565 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-05-27T07:36:52.565 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-05-27T07:36:52.565 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-05-27T07:36:52.565 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-05-27T07:36:52.565 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-05-27T07:36:52.565 INFO:teuthology.orchestra.run.smithi052.stdout: "log_to_cluster": { 2024-05-27T07:36:52.565 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "log_to_cluster", 2024-05-27T07:36:52.565 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "bool", 2024-05-27T07:36:52.565 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-05-27T07:36:52.566 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 1, 2024-05-27T07:36:52.566 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "False", 2024-05-27T07:36:52.566 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-05-27T07:36:52.566 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-05-27T07:36:52.566 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-05-27T07:36:52.566 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-05-27T07:36:52.566 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-05-27T07:36:52.566 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-05-27T07:36:52.566 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-05-27T07:36:52.566 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-05-27T07:36:52.566 INFO:teuthology.orchestra.run.smithi052.stdout: "log_to_cluster_level": { 2024-05-27T07:36:52.566 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "log_to_cluster_level", 2024-05-27T07:36:52.567 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "str", 2024-05-27T07:36:52.567 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-05-27T07:36:52.567 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 1, 2024-05-27T07:36:52.567 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "info", 2024-05-27T07:36:52.567 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-05-27T07:36:52.567 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-05-27T07:36:52.567 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [ 2024-05-27T07:36:52.567 INFO:teuthology.orchestra.run.smithi052.stdout: "", 2024-05-27T07:36:52.567 INFO:teuthology.orchestra.run.smithi052.stdout: "critical", 2024-05-27T07:36:52.567 INFO:teuthology.orchestra.run.smithi052.stdout: "debug", 2024-05-27T07:36:52.567 INFO:teuthology.orchestra.run.smithi052.stdout: "error", 2024-05-27T07:36:52.567 INFO:teuthology.orchestra.run.smithi052.stdout: "info", 2024-05-27T07:36:52.568 INFO:teuthology.orchestra.run.smithi052.stdout: "warning" 2024-05-27T07:36:52.568 INFO:teuthology.orchestra.run.smithi052.stdout: ], 2024-05-27T07:36:52.568 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-05-27T07:36:52.568 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-05-27T07:36:52.568 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-05-27T07:36:52.568 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-05-27T07:36:52.568 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-05-27T07:36:52.568 INFO:teuthology.orchestra.run.smithi052.stdout: "log_to_file": { 2024-05-27T07:36:52.568 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "log_to_file", 2024-05-27T07:36:52.568 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "bool", 2024-05-27T07:36:52.568 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-05-27T07:36:52.568 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 1, 2024-05-27T07:36:52.568 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "False", 2024-05-27T07:36:52.569 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-05-27T07:36:52.569 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-05-27T07:36:52.569 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-05-27T07:36:52.569 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-05-27T07:36:52.569 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-05-27T07:36:52.569 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-05-27T07:36:52.569 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-05-27T07:36:52.569 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-05-27T07:36:52.569 INFO:teuthology.orchestra.run.smithi052.stdout: "min_size": { 2024-05-27T07:36:52.569 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "min_size", 2024-05-27T07:36:52.569 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "int", 2024-05-27T07:36:52.569 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-05-27T07:36:52.569 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 1, 2024-05-27T07:36:52.570 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "", 2024-05-27T07:36:52.570 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-05-27T07:36:52.570 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-05-27T07:36:52.570 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-05-27T07:36:52.570 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "default min_size for any created local pool", 2024-05-27T07:36:52.570 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "value to set min_size to (unchanged from Ceph's default if this option is not set)", 2024-05-27T07:36:52.570 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-05-27T07:36:52.570 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-05-27T07:36:52.570 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-05-27T07:36:52.570 INFO:teuthology.orchestra.run.smithi052.stdout: "num_rep": { 2024-05-27T07:36:52.570 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "num_rep", 2024-05-27T07:36:52.570 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "int", 2024-05-27T07:36:52.571 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-05-27T07:36:52.571 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 1, 2024-05-27T07:36:52.571 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "3", 2024-05-27T07:36:52.571 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-05-27T07:36:52.571 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-05-27T07:36:52.571 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-05-27T07:36:52.571 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "default replica count for any created local pool", 2024-05-27T07:36:52.571 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-05-27T07:36:52.571 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-05-27T07:36:52.571 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-05-27T07:36:52.571 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-05-27T07:36:52.571 INFO:teuthology.orchestra.run.smithi052.stdout: "pg_num": { 2024-05-27T07:36:52.571 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "pg_num", 2024-05-27T07:36:52.572 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "int", 2024-05-27T07:36:52.572 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-05-27T07:36:52.572 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 1, 2024-05-27T07:36:52.572 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "128", 2024-05-27T07:36:52.572 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-05-27T07:36:52.572 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-05-27T07:36:52.572 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-05-27T07:36:52.572 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "default pg_num for any created local pool", 2024-05-27T07:36:52.572 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-05-27T07:36:52.572 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-05-27T07:36:52.572 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-05-27T07:36:52.572 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-05-27T07:36:52.572 INFO:teuthology.orchestra.run.smithi052.stdout: "prefix": { 2024-05-27T07:36:52.572 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "prefix", 2024-05-27T07:36:52.573 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "str", 2024-05-27T07:36:52.573 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-05-27T07:36:52.573 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 1, 2024-05-27T07:36:52.573 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "", 2024-05-27T07:36:52.573 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-05-27T07:36:52.573 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-05-27T07:36:52.573 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-05-27T07:36:52.573 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "name prefix for any created local pool", 2024-05-27T07:36:52.573 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-05-27T07:36:52.573 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-05-27T07:36:52.573 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-05-27T07:36:52.573 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-05-27T07:36:52.573 INFO:teuthology.orchestra.run.smithi052.stdout: "sqlite3_killpoint": { 2024-05-27T07:36:52.574 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "sqlite3_killpoint", 2024-05-27T07:36:52.574 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "int", 2024-05-27T07:36:52.574 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "dev", 2024-05-27T07:36:52.574 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 1, 2024-05-27T07:36:52.574 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "0", 2024-05-27T07:36:52.574 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-05-27T07:36:52.574 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-05-27T07:36:52.574 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-05-27T07:36:52.574 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-05-27T07:36:52.574 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-05-27T07:36:52.574 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-05-27T07:36:52.574 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-05-27T07:36:52.574 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-05-27T07:36:52.575 INFO:teuthology.orchestra.run.smithi052.stdout: "subtree": { 2024-05-27T07:36:52.575 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "subtree", 2024-05-27T07:36:52.575 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "str", 2024-05-27T07:36:52.575 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-05-27T07:36:52.575 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 1, 2024-05-27T07:36:52.575 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "rack", 2024-05-27T07:36:52.575 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-05-27T07:36:52.575 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-05-27T07:36:52.575 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-05-27T07:36:52.575 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "CRUSH level for which to create a local pool", 2024-05-27T07:36:52.575 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "which CRUSH subtree type the module should create a pool for.", 2024-05-27T07:36:52.575 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-05-27T07:36:52.575 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-05-27T07:36:52.576 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-05-27T07:36:52.576 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-05-27T07:36:52.576 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-05-27T07:36:52.576 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-05-27T07:36:52.576 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "mds_autoscaler", 2024-05-27T07:36:52.576 INFO:teuthology.orchestra.run.smithi052.stdout: "can_run": true, 2024-05-27T07:36:52.576 INFO:teuthology.orchestra.run.smithi052.stdout: "error_string": "", 2024-05-27T07:36:52.576 INFO:teuthology.orchestra.run.smithi052.stdout: "module_options": { 2024-05-27T07:36:52.576 INFO:teuthology.orchestra.run.smithi052.stdout: "log_level": { 2024-05-27T07:36:52.576 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "log_level", 2024-05-27T07:36:52.576 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "str", 2024-05-27T07:36:52.576 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-05-27T07:36:52.576 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 1, 2024-05-27T07:36:52.577 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "", 2024-05-27T07:36:52.577 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-05-27T07:36:52.577 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-05-27T07:36:52.577 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [ 2024-05-27T07:36:52.577 INFO:teuthology.orchestra.run.smithi052.stdout: "", 2024-05-27T07:36:52.577 INFO:teuthology.orchestra.run.smithi052.stdout: "critical", 2024-05-27T07:36:52.577 INFO:teuthology.orchestra.run.smithi052.stdout: "debug", 2024-05-27T07:36:52.577 INFO:teuthology.orchestra.run.smithi052.stdout: "error", 2024-05-27T07:36:52.577 INFO:teuthology.orchestra.run.smithi052.stdout: "info", 2024-05-27T07:36:52.577 INFO:teuthology.orchestra.run.smithi052.stdout: "warning" 2024-05-27T07:36:52.577 INFO:teuthology.orchestra.run.smithi052.stdout: ], 2024-05-27T07:36:52.577 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-05-27T07:36:52.577 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-05-27T07:36:52.577 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-05-27T07:36:52.578 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-05-27T07:36:52.578 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-05-27T07:36:52.578 INFO:teuthology.orchestra.run.smithi052.stdout: "log_to_cluster": { 2024-05-27T07:36:52.578 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "log_to_cluster", 2024-05-27T07:36:52.578 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "bool", 2024-05-27T07:36:52.578 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-05-27T07:36:52.578 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 1, 2024-05-27T07:36:52.578 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "False", 2024-05-27T07:36:52.578 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-05-27T07:36:52.578 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-05-27T07:36:52.578 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-05-27T07:36:52.578 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-05-27T07:36:52.578 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-05-27T07:36:52.579 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-05-27T07:36:52.579 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-05-27T07:36:52.579 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-05-27T07:36:52.579 INFO:teuthology.orchestra.run.smithi052.stdout: "log_to_cluster_level": { 2024-05-27T07:36:52.579 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "log_to_cluster_level", 2024-05-27T07:36:52.579 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "str", 2024-05-27T07:36:52.579 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-05-27T07:36:52.579 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 1, 2024-05-27T07:36:52.579 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "info", 2024-05-27T07:36:52.579 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-05-27T07:36:52.579 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-05-27T07:36:52.579 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [ 2024-05-27T07:36:52.579 INFO:teuthology.orchestra.run.smithi052.stdout: "", 2024-05-27T07:36:52.580 INFO:teuthology.orchestra.run.smithi052.stdout: "critical", 2024-05-27T07:36:52.580 INFO:teuthology.orchestra.run.smithi052.stdout: "debug", 2024-05-27T07:36:52.580 INFO:teuthology.orchestra.run.smithi052.stdout: "error", 2024-05-27T07:36:52.580 INFO:teuthology.orchestra.run.smithi052.stdout: "info", 2024-05-27T07:36:52.580 INFO:teuthology.orchestra.run.smithi052.stdout: "warning" 2024-05-27T07:36:52.580 INFO:teuthology.orchestra.run.smithi052.stdout: ], 2024-05-27T07:36:52.580 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-05-27T07:36:52.580 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-05-27T07:36:52.580 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-05-27T07:36:52.580 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-05-27T07:36:52.580 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-05-27T07:36:52.580 INFO:teuthology.orchestra.run.smithi052.stdout: "log_to_file": { 2024-05-27T07:36:52.580 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "log_to_file", 2024-05-27T07:36:52.580 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "bool", 2024-05-27T07:36:52.581 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-05-27T07:36:52.581 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 1, 2024-05-27T07:36:52.581 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "False", 2024-05-27T07:36:52.581 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-05-27T07:36:52.581 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-05-27T07:36:52.581 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-05-27T07:36:52.581 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-05-27T07:36:52.581 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-05-27T07:36:52.581 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-05-27T07:36:52.581 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-05-27T07:36:52.581 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-05-27T07:36:52.581 INFO:teuthology.orchestra.run.smithi052.stdout: "sqlite3_killpoint": { 2024-05-27T07:36:52.581 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "sqlite3_killpoint", 2024-05-27T07:36:52.581 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "int", 2024-05-27T07:36:52.582 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "dev", 2024-05-27T07:36:52.582 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 1, 2024-05-27T07:36:52.582 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "0", 2024-05-27T07:36:52.582 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-05-27T07:36:52.582 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-05-27T07:36:52.582 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-05-27T07:36:52.582 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-05-27T07:36:52.582 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-05-27T07:36:52.582 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-05-27T07:36:52.582 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-05-27T07:36:52.582 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-05-27T07:36:52.582 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-05-27T07:36:52.583 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-05-27T07:36:52.583 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-05-27T07:36:52.583 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "mirroring", 2024-05-27T07:36:52.583 INFO:teuthology.orchestra.run.smithi052.stdout: "can_run": true, 2024-05-27T07:36:52.583 INFO:teuthology.orchestra.run.smithi052.stdout: "error_string": "", 2024-05-27T07:36:52.583 INFO:teuthology.orchestra.run.smithi052.stdout: "module_options": { 2024-05-27T07:36:52.583 INFO:teuthology.orchestra.run.smithi052.stdout: "log_level": { 2024-05-27T07:36:52.583 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "log_level", 2024-05-27T07:36:52.583 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "str", 2024-05-27T07:36:52.583 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-05-27T07:36:52.583 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 1, 2024-05-27T07:36:52.583 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "", 2024-05-27T07:36:52.583 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-05-27T07:36:52.583 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-05-27T07:36:52.584 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [ 2024-05-27T07:36:52.584 INFO:teuthology.orchestra.run.smithi052.stdout: "", 2024-05-27T07:36:52.584 INFO:teuthology.orchestra.run.smithi052.stdout: "critical", 2024-05-27T07:36:52.584 INFO:teuthology.orchestra.run.smithi052.stdout: "debug", 2024-05-27T07:36:52.584 INFO:teuthology.orchestra.run.smithi052.stdout: "error", 2024-05-27T07:36:52.584 INFO:teuthology.orchestra.run.smithi052.stdout: "info", 2024-05-27T07:36:52.584 INFO:teuthology.orchestra.run.smithi052.stdout: "warning" 2024-05-27T07:36:52.584 INFO:teuthology.orchestra.run.smithi052.stdout: ], 2024-05-27T07:36:52.584 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-05-27T07:36:52.584 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-05-27T07:36:52.584 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-05-27T07:36:52.584 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-05-27T07:36:52.584 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-05-27T07:36:52.585 INFO:teuthology.orchestra.run.smithi052.stdout: "log_to_cluster": { 2024-05-27T07:36:52.585 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "log_to_cluster", 2024-05-27T07:36:52.585 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "bool", 2024-05-27T07:36:52.585 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-05-27T07:36:52.585 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 1, 2024-05-27T07:36:52.585 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "False", 2024-05-27T07:36:52.585 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-05-27T07:36:52.585 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-05-27T07:36:52.585 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-05-27T07:36:52.585 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-05-27T07:36:52.585 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-05-27T07:36:52.585 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-05-27T07:36:52.585 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-05-27T07:36:52.586 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-05-27T07:36:52.586 INFO:teuthology.orchestra.run.smithi052.stdout: "log_to_cluster_level": { 2024-05-27T07:36:52.586 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "log_to_cluster_level", 2024-05-27T07:36:52.586 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "str", 2024-05-27T07:36:52.586 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-05-27T07:36:52.586 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 1, 2024-05-27T07:36:52.586 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "info", 2024-05-27T07:36:52.586 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-05-27T07:36:52.586 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-05-27T07:36:52.586 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [ 2024-05-27T07:36:52.586 INFO:teuthology.orchestra.run.smithi052.stdout: "", 2024-05-27T07:36:52.586 INFO:teuthology.orchestra.run.smithi052.stdout: "critical", 2024-05-27T07:36:52.587 INFO:teuthology.orchestra.run.smithi052.stdout: "debug", 2024-05-27T07:36:52.587 INFO:teuthology.orchestra.run.smithi052.stdout: "error", 2024-05-27T07:36:52.587 INFO:teuthology.orchestra.run.smithi052.stdout: "info", 2024-05-27T07:36:52.587 INFO:teuthology.orchestra.run.smithi052.stdout: "warning" 2024-05-27T07:36:52.587 INFO:teuthology.orchestra.run.smithi052.stdout: ], 2024-05-27T07:36:52.587 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-05-27T07:36:52.587 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-05-27T07:36:52.587 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-05-27T07:36:52.587 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-05-27T07:36:52.587 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-05-27T07:36:52.587 INFO:teuthology.orchestra.run.smithi052.stdout: "log_to_file": { 2024-05-27T07:36:52.587 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "log_to_file", 2024-05-27T07:36:52.588 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "bool", 2024-05-27T07:36:52.588 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-05-27T07:36:52.588 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 1, 2024-05-27T07:36:52.588 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "False", 2024-05-27T07:36:52.588 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-05-27T07:36:52.588 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-05-27T07:36:52.588 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-05-27T07:36:52.588 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-05-27T07:36:52.588 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-05-27T07:36:52.588 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-05-27T07:36:52.588 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-05-27T07:36:52.588 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-05-27T07:36:52.588 INFO:teuthology.orchestra.run.smithi052.stdout: "sqlite3_killpoint": { 2024-05-27T07:36:52.589 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "sqlite3_killpoint", 2024-05-27T07:36:52.589 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "int", 2024-05-27T07:36:52.589 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "dev", 2024-05-27T07:36:52.589 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 1, 2024-05-27T07:36:52.589 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "0", 2024-05-27T07:36:52.589 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-05-27T07:36:52.589 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-05-27T07:36:52.589 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-05-27T07:36:52.589 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-05-27T07:36:52.589 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-05-27T07:36:52.589 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-05-27T07:36:52.589 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-05-27T07:36:52.589 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-05-27T07:36:52.590 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-05-27T07:36:52.590 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-05-27T07:36:52.590 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-05-27T07:36:52.590 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "nfs", 2024-05-27T07:36:52.590 INFO:teuthology.orchestra.run.smithi052.stdout: "can_run": true, 2024-05-27T07:36:52.590 INFO:teuthology.orchestra.run.smithi052.stdout: "error_string": "", 2024-05-27T07:36:52.590 INFO:teuthology.orchestra.run.smithi052.stdout: "module_options": { 2024-05-27T07:36:52.590 INFO:teuthology.orchestra.run.smithi052.stdout: "log_level": { 2024-05-27T07:36:52.590 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "log_level", 2024-05-27T07:36:52.590 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "str", 2024-05-27T07:36:52.590 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-05-27T07:36:52.590 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 1, 2024-05-27T07:36:52.591 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "", 2024-05-27T07:36:52.591 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-05-27T07:36:52.591 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-05-27T07:36:52.591 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [ 2024-05-27T07:36:52.591 INFO:teuthology.orchestra.run.smithi052.stdout: "", 2024-05-27T07:36:52.591 INFO:teuthology.orchestra.run.smithi052.stdout: "critical", 2024-05-27T07:36:52.591 INFO:teuthology.orchestra.run.smithi052.stdout: "debug", 2024-05-27T07:36:52.591 INFO:teuthology.orchestra.run.smithi052.stdout: "error", 2024-05-27T07:36:52.591 INFO:teuthology.orchestra.run.smithi052.stdout: "info", 2024-05-27T07:36:52.591 INFO:teuthology.orchestra.run.smithi052.stdout: "warning" 2024-05-27T07:36:52.591 INFO:teuthology.orchestra.run.smithi052.stdout: ], 2024-05-27T07:36:52.591 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-05-27T07:36:52.591 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-05-27T07:36:52.592 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-05-27T07:36:52.592 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-05-27T07:36:52.592 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-05-27T07:36:52.592 INFO:teuthology.orchestra.run.smithi052.stdout: "log_to_cluster": { 2024-05-27T07:36:52.592 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "log_to_cluster", 2024-05-27T07:36:52.592 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "bool", 2024-05-27T07:36:52.592 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-05-27T07:36:52.592 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 1, 2024-05-27T07:36:52.592 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "False", 2024-05-27T07:36:52.592 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-05-27T07:36:52.592 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-05-27T07:36:52.592 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-05-27T07:36:52.592 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-05-27T07:36:52.593 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-05-27T07:36:52.593 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-05-27T07:36:52.593 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-05-27T07:36:52.593 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-05-27T07:36:52.593 INFO:teuthology.orchestra.run.smithi052.stdout: "log_to_cluster_level": { 2024-05-27T07:36:52.593 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "log_to_cluster_level", 2024-05-27T07:36:52.593 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "str", 2024-05-27T07:36:52.593 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-05-27T07:36:52.593 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 1, 2024-05-27T07:36:52.593 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "info", 2024-05-27T07:36:52.593 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-05-27T07:36:52.593 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-05-27T07:36:52.594 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [ 2024-05-27T07:36:52.594 INFO:teuthology.orchestra.run.smithi052.stdout: "", 2024-05-27T07:36:52.594 INFO:teuthology.orchestra.run.smithi052.stdout: "critical", 2024-05-27T07:36:52.594 INFO:teuthology.orchestra.run.smithi052.stdout: "debug", 2024-05-27T07:36:52.594 INFO:teuthology.orchestra.run.smithi052.stdout: "error", 2024-05-27T07:36:52.594 INFO:teuthology.orchestra.run.smithi052.stdout: "info", 2024-05-27T07:36:52.594 INFO:teuthology.orchestra.run.smithi052.stdout: "warning" 2024-05-27T07:36:52.594 INFO:teuthology.orchestra.run.smithi052.stdout: ], 2024-05-27T07:36:52.594 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-05-27T07:36:52.594 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-05-27T07:36:52.594 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-05-27T07:36:52.594 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-05-27T07:36:52.594 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-05-27T07:36:52.595 INFO:teuthology.orchestra.run.smithi052.stdout: "log_to_file": { 2024-05-27T07:36:52.595 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "log_to_file", 2024-05-27T07:36:52.595 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "bool", 2024-05-27T07:36:52.595 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-05-27T07:36:52.595 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 1, 2024-05-27T07:36:52.595 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "False", 2024-05-27T07:36:52.595 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-05-27T07:36:52.595 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-05-27T07:36:52.595 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-05-27T07:36:52.595 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-05-27T07:36:52.595 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-05-27T07:36:52.595 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-05-27T07:36:52.595 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-05-27T07:36:52.596 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-05-27T07:36:52.596 INFO:teuthology.orchestra.run.smithi052.stdout: "sqlite3_killpoint": { 2024-05-27T07:36:52.596 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "sqlite3_killpoint", 2024-05-27T07:36:52.596 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "int", 2024-05-27T07:36:52.596 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "dev", 2024-05-27T07:36:52.596 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 1, 2024-05-27T07:36:52.596 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "0", 2024-05-27T07:36:52.596 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-05-27T07:36:52.596 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-05-27T07:36:52.596 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-05-27T07:36:52.596 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-05-27T07:36:52.596 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-05-27T07:36:52.596 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-05-27T07:36:52.597 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-05-27T07:36:52.597 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-05-27T07:36:52.597 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-05-27T07:36:52.597 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-05-27T07:36:52.597 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-05-27T07:36:52.597 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "orchestrator", 2024-05-27T07:36:52.597 INFO:teuthology.orchestra.run.smithi052.stdout: "can_run": true, 2024-05-27T07:36:52.597 INFO:teuthology.orchestra.run.smithi052.stdout: "error_string": "", 2024-05-27T07:36:52.597 INFO:teuthology.orchestra.run.smithi052.stdout: "module_options": { 2024-05-27T07:36:52.597 INFO:teuthology.orchestra.run.smithi052.stdout: "fail_fs": { 2024-05-27T07:36:52.597 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "fail_fs", 2024-05-27T07:36:52.597 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "bool", 2024-05-27T07:36:52.598 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-05-27T07:36:52.598 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 0, 2024-05-27T07:36:52.598 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "False", 2024-05-27T07:36:52.598 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-05-27T07:36:52.598 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-05-27T07:36:52.598 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-05-27T07:36:52.598 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "Fail filesystem for rapid multi-rank mds upgrade", 2024-05-27T07:36:52.598 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-05-27T07:36:52.598 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-05-27T07:36:52.598 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-05-27T07:36:52.598 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-05-27T07:36:52.598 INFO:teuthology.orchestra.run.smithi052.stdout: "log_level": { 2024-05-27T07:36:52.599 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "log_level", 2024-05-27T07:36:52.599 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "str", 2024-05-27T07:36:52.599 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-05-27T07:36:52.599 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 1, 2024-05-27T07:36:52.599 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "", 2024-05-27T07:36:52.599 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-05-27T07:36:52.599 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-05-27T07:36:52.599 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [ 2024-05-27T07:36:52.599 INFO:teuthology.orchestra.run.smithi052.stdout: "", 2024-05-27T07:36:52.599 INFO:teuthology.orchestra.run.smithi052.stdout: "critical", 2024-05-27T07:36:52.599 INFO:teuthology.orchestra.run.smithi052.stdout: "debug", 2024-05-27T07:36:52.599 INFO:teuthology.orchestra.run.smithi052.stdout: "error", 2024-05-27T07:36:52.599 INFO:teuthology.orchestra.run.smithi052.stdout: "info", 2024-05-27T07:36:52.600 INFO:teuthology.orchestra.run.smithi052.stdout: "warning" 2024-05-27T07:36:52.600 INFO:teuthology.orchestra.run.smithi052.stdout: ], 2024-05-27T07:36:52.600 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-05-27T07:36:52.600 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-05-27T07:36:52.600 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-05-27T07:36:52.600 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-05-27T07:36:52.600 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-05-27T07:36:52.600 INFO:teuthology.orchestra.run.smithi052.stdout: "log_to_cluster": { 2024-05-27T07:36:52.600 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "log_to_cluster", 2024-05-27T07:36:52.600 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "bool", 2024-05-27T07:36:52.600 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-05-27T07:36:52.600 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 1, 2024-05-27T07:36:52.600 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "False", 2024-05-27T07:36:52.601 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-05-27T07:36:52.601 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-05-27T07:36:52.601 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-05-27T07:36:52.601 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-05-27T07:36:52.601 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-05-27T07:36:52.601 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-05-27T07:36:52.601 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-05-27T07:36:52.601 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-05-27T07:36:52.601 INFO:teuthology.orchestra.run.smithi052.stdout: "log_to_cluster_level": { 2024-05-27T07:36:52.601 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "log_to_cluster_level", 2024-05-27T07:36:52.601 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "str", 2024-05-27T07:36:52.601 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-05-27T07:36:52.602 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 1, 2024-05-27T07:36:52.602 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "info", 2024-05-27T07:36:52.602 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-05-27T07:36:52.602 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-05-27T07:36:52.602 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [ 2024-05-27T07:36:52.602 INFO:teuthology.orchestra.run.smithi052.stdout: "", 2024-05-27T07:36:52.602 INFO:teuthology.orchestra.run.smithi052.stdout: "critical", 2024-05-27T07:36:52.602 INFO:teuthology.orchestra.run.smithi052.stdout: "debug", 2024-05-27T07:36:52.602 INFO:teuthology.orchestra.run.smithi052.stdout: "error", 2024-05-27T07:36:52.602 INFO:teuthology.orchestra.run.smithi052.stdout: "info", 2024-05-27T07:36:52.602 INFO:teuthology.orchestra.run.smithi052.stdout: "warning" 2024-05-27T07:36:52.602 INFO:teuthology.orchestra.run.smithi052.stdout: ], 2024-05-27T07:36:52.602 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-05-27T07:36:52.603 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-05-27T07:36:52.603 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-05-27T07:36:52.603 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-05-27T07:36:52.603 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-05-27T07:36:52.603 INFO:teuthology.orchestra.run.smithi052.stdout: "log_to_file": { 2024-05-27T07:36:52.603 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "log_to_file", 2024-05-27T07:36:52.603 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "bool", 2024-05-27T07:36:52.603 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-05-27T07:36:52.603 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 1, 2024-05-27T07:36:52.603 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "False", 2024-05-27T07:36:52.603 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-05-27T07:36:52.603 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-05-27T07:36:52.603 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-05-27T07:36:52.604 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-05-27T07:36:52.604 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-05-27T07:36:52.604 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-05-27T07:36:52.604 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-05-27T07:36:52.604 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-05-27T07:36:52.604 INFO:teuthology.orchestra.run.smithi052.stdout: "orchestrator": { 2024-05-27T07:36:52.604 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "orchestrator", 2024-05-27T07:36:52.604 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "str", 2024-05-27T07:36:52.604 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-05-27T07:36:52.604 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 1, 2024-05-27T07:36:52.604 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "", 2024-05-27T07:36:52.604 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-05-27T07:36:52.604 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-05-27T07:36:52.605 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [ 2024-05-27T07:36:52.605 INFO:teuthology.orchestra.run.smithi052.stdout: "cephadm", 2024-05-27T07:36:52.605 INFO:teuthology.orchestra.run.smithi052.stdout: "rook", 2024-05-27T07:36:52.605 INFO:teuthology.orchestra.run.smithi052.stdout: "test_orchestrator" 2024-05-27T07:36:52.605 INFO:teuthology.orchestra.run.smithi052.stdout: ], 2024-05-27T07:36:52.605 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "Orchestrator backend", 2024-05-27T07:36:52.605 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-05-27T07:36:52.605 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-05-27T07:36:52.605 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-05-27T07:36:52.605 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-05-27T07:36:52.605 INFO:teuthology.orchestra.run.smithi052.stdout: "sqlite3_killpoint": { 2024-05-27T07:36:52.605 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "sqlite3_killpoint", 2024-05-27T07:36:52.605 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "int", 2024-05-27T07:36:52.606 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "dev", 2024-05-27T07:36:52.606 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 1, 2024-05-27T07:36:52.606 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "0", 2024-05-27T07:36:52.606 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-05-27T07:36:52.606 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-05-27T07:36:52.606 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-05-27T07:36:52.606 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-05-27T07:36:52.606 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-05-27T07:36:52.606 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-05-27T07:36:52.606 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-05-27T07:36:52.606 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-05-27T07:36:52.606 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-05-27T07:36:52.607 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-05-27T07:36:52.607 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-05-27T07:36:52.607 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "osd_perf_query", 2024-05-27T07:36:52.607 INFO:teuthology.orchestra.run.smithi052.stdout: "can_run": true, 2024-05-27T07:36:52.607 INFO:teuthology.orchestra.run.smithi052.stdout: "error_string": "", 2024-05-27T07:36:52.607 INFO:teuthology.orchestra.run.smithi052.stdout: "module_options": { 2024-05-27T07:36:52.607 INFO:teuthology.orchestra.run.smithi052.stdout: "log_level": { 2024-05-27T07:36:52.607 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "log_level", 2024-05-27T07:36:52.607 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "str", 2024-05-27T07:36:52.607 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-05-27T07:36:52.607 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 1, 2024-05-27T07:36:52.607 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "", 2024-05-27T07:36:52.607 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-05-27T07:36:52.608 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-05-27T07:36:52.608 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [ 2024-05-27T07:36:52.608 INFO:teuthology.orchestra.run.smithi052.stdout: "", 2024-05-27T07:36:52.608 INFO:teuthology.orchestra.run.smithi052.stdout: "critical", 2024-05-27T07:36:52.608 INFO:teuthology.orchestra.run.smithi052.stdout: "debug", 2024-05-27T07:36:52.608 INFO:teuthology.orchestra.run.smithi052.stdout: "error", 2024-05-27T07:36:52.608 INFO:teuthology.orchestra.run.smithi052.stdout: "info", 2024-05-27T07:36:52.608 INFO:teuthology.orchestra.run.smithi052.stdout: "warning" 2024-05-27T07:36:52.608 INFO:teuthology.orchestra.run.smithi052.stdout: ], 2024-05-27T07:36:52.608 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-05-27T07:36:52.608 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-05-27T07:36:52.608 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-05-27T07:36:52.608 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-05-27T07:36:52.609 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-05-27T07:36:52.609 INFO:teuthology.orchestra.run.smithi052.stdout: "log_to_cluster": { 2024-05-27T07:36:52.609 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "log_to_cluster", 2024-05-27T07:36:52.609 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "bool", 2024-05-27T07:36:52.609 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-05-27T07:36:52.609 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 1, 2024-05-27T07:36:52.609 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "False", 2024-05-27T07:36:52.609 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-05-27T07:36:52.609 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-05-27T07:36:52.609 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-05-27T07:36:52.609 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-05-27T07:36:52.609 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-05-27T07:36:52.609 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-05-27T07:36:52.610 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-05-27T07:36:52.610 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-05-27T07:36:52.610 INFO:teuthology.orchestra.run.smithi052.stdout: "log_to_cluster_level": { 2024-05-27T07:36:52.610 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "log_to_cluster_level", 2024-05-27T07:36:52.610 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "str", 2024-05-27T07:36:52.610 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-05-27T07:36:52.610 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 1, 2024-05-27T07:36:52.610 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "info", 2024-05-27T07:36:52.610 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-05-27T07:36:52.610 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-05-27T07:36:52.610 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [ 2024-05-27T07:36:52.610 INFO:teuthology.orchestra.run.smithi052.stdout: "", 2024-05-27T07:36:52.610 INFO:teuthology.orchestra.run.smithi052.stdout: "critical", 2024-05-27T07:36:52.611 INFO:teuthology.orchestra.run.smithi052.stdout: "debug", 2024-05-27T07:36:52.611 INFO:teuthology.orchestra.run.smithi052.stdout: "error", 2024-05-27T07:36:52.611 INFO:teuthology.orchestra.run.smithi052.stdout: "info", 2024-05-27T07:36:52.611 INFO:teuthology.orchestra.run.smithi052.stdout: "warning" 2024-05-27T07:36:52.611 INFO:teuthology.orchestra.run.smithi052.stdout: ], 2024-05-27T07:36:52.611 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-05-27T07:36:52.611 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-05-27T07:36:52.611 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-05-27T07:36:52.611 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-05-27T07:36:52.611 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-05-27T07:36:52.611 INFO:teuthology.orchestra.run.smithi052.stdout: "log_to_file": { 2024-05-27T07:36:52.611 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "log_to_file", 2024-05-27T07:36:52.611 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "bool", 2024-05-27T07:36:52.612 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-05-27T07:36:52.612 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 1, 2024-05-27T07:36:52.612 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "False", 2024-05-27T07:36:52.612 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-05-27T07:36:52.612 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-05-27T07:36:52.612 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-05-27T07:36:52.612 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-05-27T07:36:52.612 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-05-27T07:36:52.612 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-05-27T07:36:52.612 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-05-27T07:36:52.612 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-05-27T07:36:52.612 INFO:teuthology.orchestra.run.smithi052.stdout: "sqlite3_killpoint": { 2024-05-27T07:36:52.612 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "sqlite3_killpoint", 2024-05-27T07:36:52.613 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "int", 2024-05-27T07:36:52.613 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "dev", 2024-05-27T07:36:52.613 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 1, 2024-05-27T07:36:52.613 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "0", 2024-05-27T07:36:52.613 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-05-27T07:36:52.613 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-05-27T07:36:52.613 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-05-27T07:36:52.613 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-05-27T07:36:52.613 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-05-27T07:36:52.613 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-05-27T07:36:52.613 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-05-27T07:36:52.613 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-05-27T07:36:52.614 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-05-27T07:36:52.614 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-05-27T07:36:52.614 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-05-27T07:36:52.614 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "osd_support", 2024-05-27T07:36:52.614 INFO:teuthology.orchestra.run.smithi052.stdout: "can_run": true, 2024-05-27T07:36:52.614 INFO:teuthology.orchestra.run.smithi052.stdout: "error_string": "", 2024-05-27T07:36:52.614 INFO:teuthology.orchestra.run.smithi052.stdout: "module_options": { 2024-05-27T07:36:52.614 INFO:teuthology.orchestra.run.smithi052.stdout: "log_level": { 2024-05-27T07:36:52.614 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "log_level", 2024-05-27T07:36:52.614 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "str", 2024-05-27T07:36:52.614 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-05-27T07:36:52.614 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 1, 2024-05-27T07:36:52.614 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "", 2024-05-27T07:36:52.615 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-05-27T07:36:52.615 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-05-27T07:36:52.615 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [ 2024-05-27T07:36:52.615 INFO:teuthology.orchestra.run.smithi052.stdout: "", 2024-05-27T07:36:52.615 INFO:teuthology.orchestra.run.smithi052.stdout: "critical", 2024-05-27T07:36:52.615 INFO:teuthology.orchestra.run.smithi052.stdout: "debug", 2024-05-27T07:36:52.615 INFO:teuthology.orchestra.run.smithi052.stdout: "error", 2024-05-27T07:36:52.615 INFO:teuthology.orchestra.run.smithi052.stdout: "info", 2024-05-27T07:36:52.615 INFO:teuthology.orchestra.run.smithi052.stdout: "warning" 2024-05-27T07:36:52.615 INFO:teuthology.orchestra.run.smithi052.stdout: ], 2024-05-27T07:36:52.615 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-05-27T07:36:52.616 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-05-27T07:36:52.616 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-05-27T07:36:52.616 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-05-27T07:36:52.616 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-05-27T07:36:52.616 INFO:teuthology.orchestra.run.smithi052.stdout: "log_to_cluster": { 2024-05-27T07:36:52.616 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "log_to_cluster", 2024-05-27T07:36:52.616 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "bool", 2024-05-27T07:36:52.616 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-05-27T07:36:52.616 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 1, 2024-05-27T07:36:52.616 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "False", 2024-05-27T07:36:52.616 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-05-27T07:36:52.616 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-05-27T07:36:52.616 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-05-27T07:36:52.617 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-05-27T07:36:52.617 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-05-27T07:36:52.617 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-05-27T07:36:52.617 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-05-27T07:36:52.617 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-05-27T07:36:52.617 INFO:teuthology.orchestra.run.smithi052.stdout: "log_to_cluster_level": { 2024-05-27T07:36:52.617 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "log_to_cluster_level", 2024-05-27T07:36:52.617 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "str", 2024-05-27T07:36:52.617 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-05-27T07:36:52.617 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 1, 2024-05-27T07:36:52.617 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "info", 2024-05-27T07:36:52.617 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-05-27T07:36:52.618 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-05-27T07:36:52.618 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [ 2024-05-27T07:36:52.618 INFO:teuthology.orchestra.run.smithi052.stdout: "", 2024-05-27T07:36:52.618 INFO:teuthology.orchestra.run.smithi052.stdout: "critical", 2024-05-27T07:36:52.618 INFO:teuthology.orchestra.run.smithi052.stdout: "debug", 2024-05-27T07:36:52.618 INFO:teuthology.orchestra.run.smithi052.stdout: "error", 2024-05-27T07:36:52.618 INFO:teuthology.orchestra.run.smithi052.stdout: "info", 2024-05-27T07:36:52.618 INFO:teuthology.orchestra.run.smithi052.stdout: "warning" 2024-05-27T07:36:52.618 INFO:teuthology.orchestra.run.smithi052.stdout: ], 2024-05-27T07:36:52.618 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-05-27T07:36:52.618 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-05-27T07:36:52.618 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-05-27T07:36:52.619 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-05-27T07:36:52.619 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-05-27T07:36:52.619 INFO:teuthology.orchestra.run.smithi052.stdout: "log_to_file": { 2024-05-27T07:36:52.619 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "log_to_file", 2024-05-27T07:36:52.619 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "bool", 2024-05-27T07:36:52.619 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-05-27T07:36:52.619 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 1, 2024-05-27T07:36:52.619 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "False", 2024-05-27T07:36:52.619 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-05-27T07:36:52.619 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-05-27T07:36:52.619 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-05-27T07:36:52.619 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-05-27T07:36:52.619 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-05-27T07:36:52.620 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-05-27T07:36:52.620 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-05-27T07:36:52.620 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-05-27T07:36:52.620 INFO:teuthology.orchestra.run.smithi052.stdout: "sqlite3_killpoint": { 2024-05-27T07:36:52.620 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "sqlite3_killpoint", 2024-05-27T07:36:52.620 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "int", 2024-05-27T07:36:52.620 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "dev", 2024-05-27T07:36:52.620 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 1, 2024-05-27T07:36:52.620 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "0", 2024-05-27T07:36:52.620 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-05-27T07:36:52.620 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-05-27T07:36:52.620 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-05-27T07:36:52.620 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-05-27T07:36:52.621 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-05-27T07:36:52.621 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-05-27T07:36:52.621 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-05-27T07:36:52.621 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-05-27T07:36:52.621 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-05-27T07:36:52.621 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-05-27T07:36:52.621 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-05-27T07:36:52.621 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "pg_autoscaler", 2024-05-27T07:36:52.621 INFO:teuthology.orchestra.run.smithi052.stdout: "can_run": true, 2024-05-27T07:36:52.621 INFO:teuthology.orchestra.run.smithi052.stdout: "error_string": "", 2024-05-27T07:36:52.621 INFO:teuthology.orchestra.run.smithi052.stdout: "module_options": { 2024-05-27T07:36:52.621 INFO:teuthology.orchestra.run.smithi052.stdout: "log_level": { 2024-05-27T07:36:52.621 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "log_level", 2024-05-27T07:36:52.622 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "str", 2024-05-27T07:36:52.622 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-05-27T07:36:52.622 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 1, 2024-05-27T07:36:52.622 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "", 2024-05-27T07:36:52.622 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-05-27T07:36:52.622 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-05-27T07:36:52.622 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [ 2024-05-27T07:36:52.622 INFO:teuthology.orchestra.run.smithi052.stdout: "", 2024-05-27T07:36:52.622 INFO:teuthology.orchestra.run.smithi052.stdout: "critical", 2024-05-27T07:36:52.622 INFO:teuthology.orchestra.run.smithi052.stdout: "debug", 2024-05-27T07:36:52.622 INFO:teuthology.orchestra.run.smithi052.stdout: "error", 2024-05-27T07:36:52.622 INFO:teuthology.orchestra.run.smithi052.stdout: "info", 2024-05-27T07:36:52.622 INFO:teuthology.orchestra.run.smithi052.stdout: "warning" 2024-05-27T07:36:52.623 INFO:teuthology.orchestra.run.smithi052.stdout: ], 2024-05-27T07:36:52.623 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-05-27T07:36:52.623 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-05-27T07:36:52.623 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-05-27T07:36:52.623 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-05-27T07:36:52.623 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-05-27T07:36:52.623 INFO:teuthology.orchestra.run.smithi052.stdout: "log_to_cluster": { 2024-05-27T07:36:52.623 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "log_to_cluster", 2024-05-27T07:36:52.623 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "bool", 2024-05-27T07:36:52.623 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-05-27T07:36:52.623 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 1, 2024-05-27T07:36:52.623 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "False", 2024-05-27T07:36:52.624 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-05-27T07:36:52.624 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-05-27T07:36:52.624 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-05-27T07:36:52.624 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-05-27T07:36:52.624 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-05-27T07:36:52.624 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-05-27T07:36:52.624 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-05-27T07:36:52.624 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-05-27T07:36:52.624 INFO:teuthology.orchestra.run.smithi052.stdout: "log_to_cluster_level": { 2024-05-27T07:36:52.624 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "log_to_cluster_level", 2024-05-27T07:36:52.624 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "str", 2024-05-27T07:36:52.624 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-05-27T07:36:52.624 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 1, 2024-05-27T07:36:52.625 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "info", 2024-05-27T07:36:52.625 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-05-27T07:36:52.625 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-05-27T07:36:52.625 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [ 2024-05-27T07:36:52.625 INFO:teuthology.orchestra.run.smithi052.stdout: "", 2024-05-27T07:36:52.625 INFO:teuthology.orchestra.run.smithi052.stdout: "critical", 2024-05-27T07:36:52.625 INFO:teuthology.orchestra.run.smithi052.stdout: "debug", 2024-05-27T07:36:52.625 INFO:teuthology.orchestra.run.smithi052.stdout: "error", 2024-05-27T07:36:52.625 INFO:teuthology.orchestra.run.smithi052.stdout: "info", 2024-05-27T07:36:52.625 INFO:teuthology.orchestra.run.smithi052.stdout: "warning" 2024-05-27T07:36:52.625 INFO:teuthology.orchestra.run.smithi052.stdout: ], 2024-05-27T07:36:52.625 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-05-27T07:36:52.625 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-05-27T07:36:52.626 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-05-27T07:36:52.626 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-05-27T07:36:52.626 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-05-27T07:36:52.626 INFO:teuthology.orchestra.run.smithi052.stdout: "log_to_file": { 2024-05-27T07:36:52.626 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "log_to_file", 2024-05-27T07:36:52.626 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "bool", 2024-05-27T07:36:52.626 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-05-27T07:36:52.626 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 1, 2024-05-27T07:36:52.626 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "False", 2024-05-27T07:36:52.626 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-05-27T07:36:52.626 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-05-27T07:36:52.626 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-05-27T07:36:52.627 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-05-27T07:36:52.627 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-05-27T07:36:52.627 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-05-27T07:36:52.627 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-05-27T07:36:52.627 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-05-27T07:36:52.627 INFO:teuthology.orchestra.run.smithi052.stdout: "sleep_interval": { 2024-05-27T07:36:52.627 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "sleep_interval", 2024-05-27T07:36:52.627 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "secs", 2024-05-27T07:36:52.627 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-05-27T07:36:52.627 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 0, 2024-05-27T07:36:52.627 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "60", 2024-05-27T07:36:52.627 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-05-27T07:36:52.627 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-05-27T07:36:52.628 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-05-27T07:36:52.628 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-05-27T07:36:52.628 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-05-27T07:36:52.628 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-05-27T07:36:52.628 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-05-27T07:36:52.628 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-05-27T07:36:52.628 INFO:teuthology.orchestra.run.smithi052.stdout: "sqlite3_killpoint": { 2024-05-27T07:36:52.628 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "sqlite3_killpoint", 2024-05-27T07:36:52.628 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "int", 2024-05-27T07:36:52.628 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "dev", 2024-05-27T07:36:52.628 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 1, 2024-05-27T07:36:52.628 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "0", 2024-05-27T07:36:52.628 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-05-27T07:36:52.629 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-05-27T07:36:52.629 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-05-27T07:36:52.629 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-05-27T07:36:52.629 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-05-27T07:36:52.629 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-05-27T07:36:52.629 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-05-27T07:36:52.629 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-05-27T07:36:52.629 INFO:teuthology.orchestra.run.smithi052.stdout: "threshold": { 2024-05-27T07:36:52.629 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "threshold", 2024-05-27T07:36:52.629 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "float", 2024-05-27T07:36:52.629 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-05-27T07:36:52.629 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 0, 2024-05-27T07:36:52.630 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "3.0", 2024-05-27T07:36:52.630 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "1.0", 2024-05-27T07:36:52.630 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-05-27T07:36:52.630 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-05-27T07:36:52.630 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "scaling threshold", 2024-05-27T07:36:52.630 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0", 2024-05-27T07:36:52.630 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-05-27T07:36:52.630 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-05-27T07:36:52.630 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-05-27T07:36:52.630 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-05-27T07:36:52.630 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-05-27T07:36:52.630 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-05-27T07:36:52.630 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "progress", 2024-05-27T07:36:52.631 INFO:teuthology.orchestra.run.smithi052.stdout: "can_run": true, 2024-05-27T07:36:52.631 INFO:teuthology.orchestra.run.smithi052.stdout: "error_string": "", 2024-05-27T07:36:52.631 INFO:teuthology.orchestra.run.smithi052.stdout: "module_options": { 2024-05-27T07:36:52.631 INFO:teuthology.orchestra.run.smithi052.stdout: "allow_pg_recovery_event": { 2024-05-27T07:36:52.631 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "allow_pg_recovery_event", 2024-05-27T07:36:52.631 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "bool", 2024-05-27T07:36:52.631 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-05-27T07:36:52.631 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 1, 2024-05-27T07:36:52.631 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "False", 2024-05-27T07:36:52.631 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-05-27T07:36:52.631 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-05-27T07:36:52.631 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-05-27T07:36:52.631 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "allow the module to show pg recovery progress", 2024-05-27T07:36:52.632 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-05-27T07:36:52.632 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-05-27T07:36:52.632 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-05-27T07:36:52.632 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-05-27T07:36:52.632 INFO:teuthology.orchestra.run.smithi052.stdout: "enabled": { 2024-05-27T07:36:52.632 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "enabled", 2024-05-27T07:36:52.632 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "bool", 2024-05-27T07:36:52.632 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-05-27T07:36:52.632 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 0, 2024-05-27T07:36:52.632 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "True", 2024-05-27T07:36:52.632 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-05-27T07:36:52.632 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-05-27T07:36:52.632 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-05-27T07:36:52.632 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-05-27T07:36:52.633 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-05-27T07:36:52.633 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-05-27T07:36:52.633 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-05-27T07:36:52.633 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-05-27T07:36:52.633 INFO:teuthology.orchestra.run.smithi052.stdout: "log_level": { 2024-05-27T07:36:52.633 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "log_level", 2024-05-27T07:36:52.633 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "str", 2024-05-27T07:36:52.633 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-05-27T07:36:52.633 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 1, 2024-05-27T07:36:52.633 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "", 2024-05-27T07:36:52.633 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-05-27T07:36:52.633 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-05-27T07:36:52.633 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [ 2024-05-27T07:36:52.634 INFO:teuthology.orchestra.run.smithi052.stdout: "", 2024-05-27T07:36:52.634 INFO:teuthology.orchestra.run.smithi052.stdout: "critical", 2024-05-27T07:36:52.634 INFO:teuthology.orchestra.run.smithi052.stdout: "debug", 2024-05-27T07:36:52.634 INFO:teuthology.orchestra.run.smithi052.stdout: "error", 2024-05-27T07:36:52.634 INFO:teuthology.orchestra.run.smithi052.stdout: "info", 2024-05-27T07:36:52.634 INFO:teuthology.orchestra.run.smithi052.stdout: "warning" 2024-05-27T07:36:52.634 INFO:teuthology.orchestra.run.smithi052.stdout: ], 2024-05-27T07:36:52.634 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-05-27T07:36:52.634 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-05-27T07:36:52.634 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-05-27T07:36:52.634 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-05-27T07:36:52.634 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-05-27T07:36:52.635 INFO:teuthology.orchestra.run.smithi052.stdout: "log_to_cluster": { 2024-05-27T07:36:52.635 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "log_to_cluster", 2024-05-27T07:36:52.635 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "bool", 2024-05-27T07:36:52.635 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-05-27T07:36:52.635 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 1, 2024-05-27T07:36:52.635 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "False", 2024-05-27T07:36:52.635 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-05-27T07:36:52.635 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-05-27T07:36:52.635 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-05-27T07:36:52.635 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-05-27T07:36:52.635 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-05-27T07:36:52.635 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-05-27T07:36:52.635 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-05-27T07:36:52.635 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-05-27T07:36:52.636 INFO:teuthology.orchestra.run.smithi052.stdout: "log_to_cluster_level": { 2024-05-27T07:36:52.636 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "log_to_cluster_level", 2024-05-27T07:36:52.636 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "str", 2024-05-27T07:36:52.636 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-05-27T07:36:52.636 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 1, 2024-05-27T07:36:52.636 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "info", 2024-05-27T07:36:52.636 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-05-27T07:36:52.636 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-05-27T07:36:52.636 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [ 2024-05-27T07:36:52.636 INFO:teuthology.orchestra.run.smithi052.stdout: "", 2024-05-27T07:36:52.636 INFO:teuthology.orchestra.run.smithi052.stdout: "critical", 2024-05-27T07:36:52.636 INFO:teuthology.orchestra.run.smithi052.stdout: "debug", 2024-05-27T07:36:52.636 INFO:teuthology.orchestra.run.smithi052.stdout: "error", 2024-05-27T07:36:52.637 INFO:teuthology.orchestra.run.smithi052.stdout: "info", 2024-05-27T07:36:52.637 INFO:teuthology.orchestra.run.smithi052.stdout: "warning" 2024-05-27T07:36:52.637 INFO:teuthology.orchestra.run.smithi052.stdout: ], 2024-05-27T07:36:52.637 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-05-27T07:36:52.637 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-05-27T07:36:52.637 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-05-27T07:36:52.637 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-05-27T07:36:52.637 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-05-27T07:36:52.637 INFO:teuthology.orchestra.run.smithi052.stdout: "log_to_file": { 2024-05-27T07:36:52.637 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "log_to_file", 2024-05-27T07:36:52.637 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "bool", 2024-05-27T07:36:52.637 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-05-27T07:36:52.637 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 1, 2024-05-27T07:36:52.638 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "False", 2024-05-27T07:36:52.638 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-05-27T07:36:52.638 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-05-27T07:36:52.638 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-05-27T07:36:52.638 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-05-27T07:36:52.638 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-05-27T07:36:52.638 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-05-27T07:36:52.638 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-05-27T07:36:52.638 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-05-27T07:36:52.638 INFO:teuthology.orchestra.run.smithi052.stdout: "max_completed_events": { 2024-05-27T07:36:52.638 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "max_completed_events", 2024-05-27T07:36:52.638 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "int", 2024-05-27T07:36:52.639 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-05-27T07:36:52.639 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 1, 2024-05-27T07:36:52.639 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "50", 2024-05-27T07:36:52.639 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-05-27T07:36:52.639 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-05-27T07:36:52.639 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-05-27T07:36:52.639 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "number of past completed events to remember", 2024-05-27T07:36:52.639 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-05-27T07:36:52.639 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-05-27T07:36:52.639 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-05-27T07:36:52.639 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-05-27T07:36:52.639 INFO:teuthology.orchestra.run.smithi052.stdout: "sleep_interval": { 2024-05-27T07:36:52.639 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "sleep_interval", 2024-05-27T07:36:52.640 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "secs", 2024-05-27T07:36:52.640 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-05-27T07:36:52.640 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 1, 2024-05-27T07:36:52.640 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "5", 2024-05-27T07:36:52.640 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-05-27T07:36:52.640 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-05-27T07:36:52.640 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-05-27T07:36:52.640 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "how long the module is going to sleep", 2024-05-27T07:36:52.640 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-05-27T07:36:52.640 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-05-27T07:36:52.640 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-05-27T07:36:52.640 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-05-27T07:36:52.641 INFO:teuthology.orchestra.run.smithi052.stdout: "sqlite3_killpoint": { 2024-05-27T07:36:52.641 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "sqlite3_killpoint", 2024-05-27T07:36:52.641 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "int", 2024-05-27T07:36:52.641 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "dev", 2024-05-27T07:36:52.641 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 1, 2024-05-27T07:36:52.641 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "0", 2024-05-27T07:36:52.641 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-05-27T07:36:52.641 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-05-27T07:36:52.641 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-05-27T07:36:52.641 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-05-27T07:36:52.641 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-05-27T07:36:52.641 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-05-27T07:36:52.641 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-05-27T07:36:52.641 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-05-27T07:36:52.642 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-05-27T07:36:52.642 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-05-27T07:36:52.642 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-05-27T07:36:52.642 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "prometheus", 2024-05-27T07:36:52.642 INFO:teuthology.orchestra.run.smithi052.stdout: "can_run": true, 2024-05-27T07:36:52.642 INFO:teuthology.orchestra.run.smithi052.stdout: "error_string": "", 2024-05-27T07:36:52.642 INFO:teuthology.orchestra.run.smithi052.stdout: "module_options": { 2024-05-27T07:36:52.642 INFO:teuthology.orchestra.run.smithi052.stdout: "cache": { 2024-05-27T07:36:52.642 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "cache", 2024-05-27T07:36:52.642 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "bool", 2024-05-27T07:36:52.642 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-05-27T07:36:52.642 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 0, 2024-05-27T07:36:52.643 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "True", 2024-05-27T07:36:52.643 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-05-27T07:36:52.643 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-05-27T07:36:52.643 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-05-27T07:36:52.643 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-05-27T07:36:52.643 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-05-27T07:36:52.643 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-05-27T07:36:52.643 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-05-27T07:36:52.643 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-05-27T07:36:52.643 INFO:teuthology.orchestra.run.smithi052.stdout: "exclude_perf_counters": { 2024-05-27T07:36:52.643 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "exclude_perf_counters", 2024-05-27T07:36:52.643 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "bool", 2024-05-27T07:36:52.643 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-05-27T07:36:52.644 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 1, 2024-05-27T07:36:52.644 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "True", 2024-05-27T07:36:52.644 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-05-27T07:36:52.644 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-05-27T07:36:52.644 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-05-27T07:36:52.644 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "Do not include perf-counters in the metrics output", 2024-05-27T07:36:52.644 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.", 2024-05-27T07:36:52.644 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-05-27T07:36:52.644 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-05-27T07:36:52.644 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-05-27T07:36:52.644 INFO:teuthology.orchestra.run.smithi052.stdout: "log_level": { 2024-05-27T07:36:52.644 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "log_level", 2024-05-27T07:36:52.644 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "str", 2024-05-27T07:36:52.645 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-05-27T07:36:52.645 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 1, 2024-05-27T07:36:52.645 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "", 2024-05-27T07:36:52.645 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-05-27T07:36:52.645 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-05-27T07:36:52.645 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [ 2024-05-27T07:36:52.645 INFO:teuthology.orchestra.run.smithi052.stdout: "", 2024-05-27T07:36:52.645 INFO:teuthology.orchestra.run.smithi052.stdout: "critical", 2024-05-27T07:36:52.645 INFO:teuthology.orchestra.run.smithi052.stdout: "debug", 2024-05-27T07:36:52.645 INFO:teuthology.orchestra.run.smithi052.stdout: "error", 2024-05-27T07:36:52.645 INFO:teuthology.orchestra.run.smithi052.stdout: "info", 2024-05-27T07:36:52.645 INFO:teuthology.orchestra.run.smithi052.stdout: "warning" 2024-05-27T07:36:52.646 INFO:teuthology.orchestra.run.smithi052.stdout: ], 2024-05-27T07:36:52.646 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-05-27T07:36:52.646 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-05-27T07:36:52.646 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-05-27T07:36:52.646 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-05-27T07:36:52.646 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-05-27T07:36:52.646 INFO:teuthology.orchestra.run.smithi052.stdout: "log_to_cluster": { 2024-05-27T07:36:52.646 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "log_to_cluster", 2024-05-27T07:36:52.646 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "bool", 2024-05-27T07:36:52.646 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-05-27T07:36:52.646 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 1, 2024-05-27T07:36:52.646 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "False", 2024-05-27T07:36:52.647 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-05-27T07:36:52.647 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-05-27T07:36:52.647 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-05-27T07:36:52.647 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-05-27T07:36:52.647 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-05-27T07:36:52.647 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-05-27T07:36:52.647 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-05-27T07:36:52.647 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-05-27T07:36:52.647 INFO:teuthology.orchestra.run.smithi052.stdout: "log_to_cluster_level": { 2024-05-27T07:36:52.647 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "log_to_cluster_level", 2024-05-27T07:36:52.647 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "str", 2024-05-27T07:36:52.647 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-05-27T07:36:52.647 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 1, 2024-05-27T07:36:52.648 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "info", 2024-05-27T07:36:52.648 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-05-27T07:36:52.648 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-05-27T07:36:52.648 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [ 2024-05-27T07:36:52.648 INFO:teuthology.orchestra.run.smithi052.stdout: "", 2024-05-27T07:36:52.648 INFO:teuthology.orchestra.run.smithi052.stdout: "critical", 2024-05-27T07:36:52.648 INFO:teuthology.orchestra.run.smithi052.stdout: "debug", 2024-05-27T07:36:52.648 INFO:teuthology.orchestra.run.smithi052.stdout: "error", 2024-05-27T07:36:52.648 INFO:teuthology.orchestra.run.smithi052.stdout: "info", 2024-05-27T07:36:52.648 INFO:teuthology.orchestra.run.smithi052.stdout: "warning" 2024-05-27T07:36:52.648 INFO:teuthology.orchestra.run.smithi052.stdout: ], 2024-05-27T07:36:52.648 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-05-27T07:36:52.648 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-05-27T07:36:52.649 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-05-27T07:36:52.649 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-05-27T07:36:52.649 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-05-27T07:36:52.649 INFO:teuthology.orchestra.run.smithi052.stdout: "log_to_file": { 2024-05-27T07:36:52.649 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "log_to_file", 2024-05-27T07:36:52.649 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "bool", 2024-05-27T07:36:52.649 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-05-27T07:36:52.649 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 1, 2024-05-27T07:36:52.649 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "False", 2024-05-27T07:36:52.649 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-05-27T07:36:52.649 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-05-27T07:36:52.649 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-05-27T07:36:52.649 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-05-27T07:36:52.649 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-05-27T07:36:52.650 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-05-27T07:36:52.650 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-05-27T07:36:52.650 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-05-27T07:36:52.650 INFO:teuthology.orchestra.run.smithi052.stdout: "rbd_stats_pools": { 2024-05-27T07:36:52.650 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "rbd_stats_pools", 2024-05-27T07:36:52.650 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "str", 2024-05-27T07:36:52.650 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-05-27T07:36:52.650 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 0, 2024-05-27T07:36:52.650 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "", 2024-05-27T07:36:52.650 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-05-27T07:36:52.650 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-05-27T07:36:52.650 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-05-27T07:36:52.650 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-05-27T07:36:52.651 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-05-27T07:36:52.651 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-05-27T07:36:52.651 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-05-27T07:36:52.651 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-05-27T07:36:52.651 INFO:teuthology.orchestra.run.smithi052.stdout: "rbd_stats_pools_refresh_interval": { 2024-05-27T07:36:52.651 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "rbd_stats_pools_refresh_interval", 2024-05-27T07:36:52.651 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "int", 2024-05-27T07:36:52.651 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-05-27T07:36:52.651 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 0, 2024-05-27T07:36:52.651 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "300", 2024-05-27T07:36:52.651 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-05-27T07:36:52.651 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-05-27T07:36:52.652 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-05-27T07:36:52.652 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-05-27T07:36:52.652 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-05-27T07:36:52.652 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-05-27T07:36:52.652 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-05-27T07:36:52.652 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-05-27T07:36:52.652 INFO:teuthology.orchestra.run.smithi052.stdout: "scrape_interval": { 2024-05-27T07:36:52.652 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "scrape_interval", 2024-05-27T07:36:52.652 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "float", 2024-05-27T07:36:52.652 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-05-27T07:36:52.652 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 0, 2024-05-27T07:36:52.652 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "15.0", 2024-05-27T07:36:52.652 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-05-27T07:36:52.653 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-05-27T07:36:52.653 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-05-27T07:36:52.653 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-05-27T07:36:52.653 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-05-27T07:36:52.653 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-05-27T07:36:52.653 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-05-27T07:36:52.653 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-05-27T07:36:52.653 INFO:teuthology.orchestra.run.smithi052.stdout: "server_addr": { 2024-05-27T07:36:52.653 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "server_addr", 2024-05-27T07:36:52.653 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "str", 2024-05-27T07:36:52.653 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-05-27T07:36:52.653 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 0, 2024-05-27T07:36:52.653 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "::", 2024-05-27T07:36:52.653 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-05-27T07:36:52.654 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-05-27T07:36:52.654 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-05-27T07:36:52.654 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "the IPv4 or IPv6 address on which the module listens for HTTP requests", 2024-05-27T07:36:52.654 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-05-27T07:36:52.654 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-05-27T07:36:52.654 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-05-27T07:36:52.654 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-05-27T07:36:52.654 INFO:teuthology.orchestra.run.smithi052.stdout: "server_port": { 2024-05-27T07:36:52.654 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "server_port", 2024-05-27T07:36:52.654 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "int", 2024-05-27T07:36:52.654 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-05-27T07:36:52.654 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 1, 2024-05-27T07:36:52.654 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "9283", 2024-05-27T07:36:52.655 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-05-27T07:36:52.655 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-05-27T07:36:52.655 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-05-27T07:36:52.655 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "the port on which the module listens for HTTP requests", 2024-05-27T07:36:52.655 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-05-27T07:36:52.655 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-05-27T07:36:52.655 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-05-27T07:36:52.655 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-05-27T07:36:52.655 INFO:teuthology.orchestra.run.smithi052.stdout: "sqlite3_killpoint": { 2024-05-27T07:36:52.655 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "sqlite3_killpoint", 2024-05-27T07:36:52.655 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "int", 2024-05-27T07:36:52.655 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "dev", 2024-05-27T07:36:52.655 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 1, 2024-05-27T07:36:52.655 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "0", 2024-05-27T07:36:52.656 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-05-27T07:36:52.656 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-05-27T07:36:52.656 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-05-27T07:36:52.656 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-05-27T07:36:52.656 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-05-27T07:36:52.656 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-05-27T07:36:52.656 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-05-27T07:36:52.656 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-05-27T07:36:52.656 INFO:teuthology.orchestra.run.smithi052.stdout: "stale_cache_strategy": { 2024-05-27T07:36:52.656 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "stale_cache_strategy", 2024-05-27T07:36:52.656 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "str", 2024-05-27T07:36:52.656 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-05-27T07:36:52.656 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 0, 2024-05-27T07:36:52.657 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "log", 2024-05-27T07:36:52.657 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-05-27T07:36:52.657 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-05-27T07:36:52.657 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-05-27T07:36:52.657 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-05-27T07:36:52.657 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-05-27T07:36:52.657 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-05-27T07:36:52.657 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-05-27T07:36:52.657 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-05-27T07:36:52.657 INFO:teuthology.orchestra.run.smithi052.stdout: "standby_behaviour": { 2024-05-27T07:36:52.657 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "standby_behaviour", 2024-05-27T07:36:52.657 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "str", 2024-05-27T07:36:52.657 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-05-27T07:36:52.658 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 1, 2024-05-27T07:36:52.658 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "default", 2024-05-27T07:36:52.658 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-05-27T07:36:52.658 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-05-27T07:36:52.658 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [ 2024-05-27T07:36:52.658 INFO:teuthology.orchestra.run.smithi052.stdout: "default", 2024-05-27T07:36:52.658 INFO:teuthology.orchestra.run.smithi052.stdout: "error" 2024-05-27T07:36:52.658 INFO:teuthology.orchestra.run.smithi052.stdout: ], 2024-05-27T07:36:52.658 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-05-27T07:36:52.658 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-05-27T07:36:52.658 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-05-27T07:36:52.658 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-05-27T07:36:52.658 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-05-27T07:36:52.659 INFO:teuthology.orchestra.run.smithi052.stdout: "standby_error_status_code": { 2024-05-27T07:36:52.659 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "standby_error_status_code", 2024-05-27T07:36:52.659 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "int", 2024-05-27T07:36:52.659 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-05-27T07:36:52.659 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 1, 2024-05-27T07:36:52.659 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "500", 2024-05-27T07:36:52.659 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "400", 2024-05-27T07:36:52.659 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "599", 2024-05-27T07:36:52.659 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-05-27T07:36:52.659 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-05-27T07:36:52.659 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-05-27T07:36:52.659 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-05-27T07:36:52.659 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-05-27T07:36:52.660 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-05-27T07:36:52.660 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-05-27T07:36:52.660 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-05-27T07:36:52.660 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-05-27T07:36:52.660 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "rbd_support", 2024-05-27T07:36:52.660 INFO:teuthology.orchestra.run.smithi052.stdout: "can_run": true, 2024-05-27T07:36:52.660 INFO:teuthology.orchestra.run.smithi052.stdout: "error_string": "", 2024-05-27T07:36:52.660 INFO:teuthology.orchestra.run.smithi052.stdout: "module_options": { 2024-05-27T07:36:52.660 INFO:teuthology.orchestra.run.smithi052.stdout: "log_level": { 2024-05-27T07:36:52.660 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "log_level", 2024-05-27T07:36:52.660 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "str", 2024-05-27T07:36:52.660 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-05-27T07:36:52.660 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 1, 2024-05-27T07:36:52.661 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "", 2024-05-27T07:36:52.661 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-05-27T07:36:52.661 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-05-27T07:36:52.661 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [ 2024-05-27T07:36:52.661 INFO:teuthology.orchestra.run.smithi052.stdout: "", 2024-05-27T07:36:52.661 INFO:teuthology.orchestra.run.smithi052.stdout: "critical", 2024-05-27T07:36:52.661 INFO:teuthology.orchestra.run.smithi052.stdout: "debug", 2024-05-27T07:36:52.661 INFO:teuthology.orchestra.run.smithi052.stdout: "error", 2024-05-27T07:36:52.661 INFO:teuthology.orchestra.run.smithi052.stdout: "info", 2024-05-27T07:36:52.661 INFO:teuthology.orchestra.run.smithi052.stdout: "warning" 2024-05-27T07:36:52.661 INFO:teuthology.orchestra.run.smithi052.stdout: ], 2024-05-27T07:36:52.661 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-05-27T07:36:52.662 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-05-27T07:36:52.662 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-05-27T07:36:52.662 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-05-27T07:36:52.662 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-05-27T07:36:52.662 INFO:teuthology.orchestra.run.smithi052.stdout: "log_to_cluster": { 2024-05-27T07:36:52.662 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "log_to_cluster", 2024-05-27T07:36:52.662 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "bool", 2024-05-27T07:36:52.662 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-05-27T07:36:52.662 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 1, 2024-05-27T07:36:52.662 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "False", 2024-05-27T07:36:52.662 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-05-27T07:36:52.662 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-05-27T07:36:52.662 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-05-27T07:36:52.663 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-05-27T07:36:52.663 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-05-27T07:36:52.663 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-05-27T07:36:52.663 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-05-27T07:36:52.663 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-05-27T07:36:52.663 INFO:teuthology.orchestra.run.smithi052.stdout: "log_to_cluster_level": { 2024-05-27T07:36:52.663 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "log_to_cluster_level", 2024-05-27T07:36:52.663 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "str", 2024-05-27T07:36:52.663 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-05-27T07:36:52.663 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 1, 2024-05-27T07:36:52.663 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "info", 2024-05-27T07:36:52.663 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-05-27T07:36:52.664 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-05-27T07:36:52.664 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [ 2024-05-27T07:36:52.664 INFO:teuthology.orchestra.run.smithi052.stdout: "", 2024-05-27T07:36:52.664 INFO:teuthology.orchestra.run.smithi052.stdout: "critical", 2024-05-27T07:36:52.664 INFO:teuthology.orchestra.run.smithi052.stdout: "debug", 2024-05-27T07:36:52.664 INFO:teuthology.orchestra.run.smithi052.stdout: "error", 2024-05-27T07:36:52.664 INFO:teuthology.orchestra.run.smithi052.stdout: "info", 2024-05-27T07:36:52.664 INFO:teuthology.orchestra.run.smithi052.stdout: "warning" 2024-05-27T07:36:52.664 INFO:teuthology.orchestra.run.smithi052.stdout: ], 2024-05-27T07:36:52.664 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-05-27T07:36:52.664 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-05-27T07:36:52.664 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-05-27T07:36:52.664 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-05-27T07:36:52.665 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-05-27T07:36:52.665 INFO:teuthology.orchestra.run.smithi052.stdout: "log_to_file": { 2024-05-27T07:36:52.665 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "log_to_file", 2024-05-27T07:36:52.665 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "bool", 2024-05-27T07:36:52.665 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-05-27T07:36:52.665 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 1, 2024-05-27T07:36:52.665 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "False", 2024-05-27T07:36:52.665 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-05-27T07:36:52.665 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-05-27T07:36:52.665 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-05-27T07:36:52.665 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-05-27T07:36:52.665 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-05-27T07:36:52.666 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-05-27T07:36:52.666 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-05-27T07:36:52.666 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-05-27T07:36:52.666 INFO:teuthology.orchestra.run.smithi052.stdout: "max_concurrent_snap_create": { 2024-05-27T07:36:52.666 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "max_concurrent_snap_create", 2024-05-27T07:36:52.666 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "int", 2024-05-27T07:36:52.666 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-05-27T07:36:52.666 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 0, 2024-05-27T07:36:52.666 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "10", 2024-05-27T07:36:52.666 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-05-27T07:36:52.666 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-05-27T07:36:52.666 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-05-27T07:36:52.666 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-05-27T07:36:52.667 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-05-27T07:36:52.667 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-05-27T07:36:52.667 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-05-27T07:36:52.667 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-05-27T07:36:52.667 INFO:teuthology.orchestra.run.smithi052.stdout: "mirror_snapshot_schedule": { 2024-05-27T07:36:52.667 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "mirror_snapshot_schedule", 2024-05-27T07:36:52.667 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "str", 2024-05-27T07:36:52.667 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-05-27T07:36:52.667 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 0, 2024-05-27T07:36:52.667 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "", 2024-05-27T07:36:52.667 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-05-27T07:36:52.667 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-05-27T07:36:52.668 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-05-27T07:36:52.668 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-05-27T07:36:52.668 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-05-27T07:36:52.668 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-05-27T07:36:52.668 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-05-27T07:36:52.668 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-05-27T07:36:52.668 INFO:teuthology.orchestra.run.smithi052.stdout: "sqlite3_killpoint": { 2024-05-27T07:36:52.668 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "sqlite3_killpoint", 2024-05-27T07:36:52.668 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "int", 2024-05-27T07:36:52.668 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "dev", 2024-05-27T07:36:52.668 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 1, 2024-05-27T07:36:52.668 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "0", 2024-05-27T07:36:52.668 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-05-27T07:36:52.668 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-05-27T07:36:52.669 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-05-27T07:36:52.669 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-05-27T07:36:52.669 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-05-27T07:36:52.669 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-05-27T07:36:52.669 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-05-27T07:36:52.669 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-05-27T07:36:52.669 INFO:teuthology.orchestra.run.smithi052.stdout: "trash_purge_schedule": { 2024-05-27T07:36:52.669 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "trash_purge_schedule", 2024-05-27T07:36:52.669 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "str", 2024-05-27T07:36:52.669 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-05-27T07:36:52.669 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 0, 2024-05-27T07:36:52.669 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "", 2024-05-27T07:36:52.669 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-05-27T07:36:52.670 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-05-27T07:36:52.670 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-05-27T07:36:52.670 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-05-27T07:36:52.670 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-05-27T07:36:52.670 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-05-27T07:36:52.670 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-05-27T07:36:52.670 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-05-27T07:36:52.670 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-05-27T07:36:52.670 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-05-27T07:36:52.670 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-05-27T07:36:52.670 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "restful", 2024-05-27T07:36:52.670 INFO:teuthology.orchestra.run.smithi052.stdout: "can_run": true, 2024-05-27T07:36:52.670 INFO:teuthology.orchestra.run.smithi052.stdout: "error_string": "", 2024-05-27T07:36:52.671 INFO:teuthology.orchestra.run.smithi052.stdout: "module_options": { 2024-05-27T07:36:52.671 INFO:teuthology.orchestra.run.smithi052.stdout: "enable_auth": { 2024-05-27T07:36:52.671 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "enable_auth", 2024-05-27T07:36:52.671 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "bool", 2024-05-27T07:36:52.671 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-05-27T07:36:52.671 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 0, 2024-05-27T07:36:52.671 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "True", 2024-05-27T07:36:52.671 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-05-27T07:36:52.671 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-05-27T07:36:52.671 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-05-27T07:36:52.671 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-05-27T07:36:52.671 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-05-27T07:36:52.671 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-05-27T07:36:52.672 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-05-27T07:36:52.672 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-05-27T07:36:52.672 INFO:teuthology.orchestra.run.smithi052.stdout: "key_file": { 2024-05-27T07:36:52.672 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "key_file", 2024-05-27T07:36:52.672 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "str", 2024-05-27T07:36:52.672 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-05-27T07:36:52.672 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 0, 2024-05-27T07:36:52.672 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "", 2024-05-27T07:36:52.672 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-05-27T07:36:52.672 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-05-27T07:36:52.672 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-05-27T07:36:52.672 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-05-27T07:36:52.672 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-05-27T07:36:52.673 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-05-27T07:36:52.673 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-05-27T07:36:52.673 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-05-27T07:36:52.673 INFO:teuthology.orchestra.run.smithi052.stdout: "log_level": { 2024-05-27T07:36:52.673 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "log_level", 2024-05-27T07:36:52.673 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "str", 2024-05-27T07:36:52.673 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-05-27T07:36:52.673 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 1, 2024-05-27T07:36:52.673 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "", 2024-05-27T07:36:52.673 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-05-27T07:36:52.673 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-05-27T07:36:52.673 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [ 2024-05-27T07:36:52.673 INFO:teuthology.orchestra.run.smithi052.stdout: "", 2024-05-27T07:36:52.674 INFO:teuthology.orchestra.run.smithi052.stdout: "critical", 2024-05-27T07:36:52.674 INFO:teuthology.orchestra.run.smithi052.stdout: "debug", 2024-05-27T07:36:52.674 INFO:teuthology.orchestra.run.smithi052.stdout: "error", 2024-05-27T07:36:52.674 INFO:teuthology.orchestra.run.smithi052.stdout: "info", 2024-05-27T07:36:52.674 INFO:teuthology.orchestra.run.smithi052.stdout: "warning" 2024-05-27T07:36:52.674 INFO:teuthology.orchestra.run.smithi052.stdout: ], 2024-05-27T07:36:52.674 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-05-27T07:36:52.674 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-05-27T07:36:52.674 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-05-27T07:36:52.674 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-05-27T07:36:52.674 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-05-27T07:36:52.674 INFO:teuthology.orchestra.run.smithi052.stdout: "log_to_cluster": { 2024-05-27T07:36:52.674 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "log_to_cluster", 2024-05-27T07:36:52.675 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "bool", 2024-05-27T07:36:52.675 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-05-27T07:36:52.675 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 1, 2024-05-27T07:36:52.675 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "False", 2024-05-27T07:36:52.675 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-05-27T07:36:52.675 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-05-27T07:36:52.675 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-05-27T07:36:52.675 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-05-27T07:36:52.675 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-05-27T07:36:52.675 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-05-27T07:36:52.675 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-05-27T07:36:52.675 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-05-27T07:36:52.675 INFO:teuthology.orchestra.run.smithi052.stdout: "log_to_cluster_level": { 2024-05-27T07:36:52.676 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "log_to_cluster_level", 2024-05-27T07:36:52.676 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "str", 2024-05-27T07:36:52.676 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-05-27T07:36:52.676 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 1, 2024-05-27T07:36:52.676 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "info", 2024-05-27T07:36:52.676 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-05-27T07:36:52.676 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-05-27T07:36:52.676 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [ 2024-05-27T07:36:52.676 INFO:teuthology.orchestra.run.smithi052.stdout: "", 2024-05-27T07:36:52.676 INFO:teuthology.orchestra.run.smithi052.stdout: "critical", 2024-05-27T07:36:52.676 INFO:teuthology.orchestra.run.smithi052.stdout: "debug", 2024-05-27T07:36:52.676 INFO:teuthology.orchestra.run.smithi052.stdout: "error", 2024-05-27T07:36:52.677 INFO:teuthology.orchestra.run.smithi052.stdout: "info", 2024-05-27T07:36:52.677 INFO:teuthology.orchestra.run.smithi052.stdout: "warning" 2024-05-27T07:36:52.677 INFO:teuthology.orchestra.run.smithi052.stdout: ], 2024-05-27T07:36:52.677 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-05-27T07:36:52.677 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-05-27T07:36:52.677 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-05-27T07:36:52.677 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-05-27T07:36:52.677 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-05-27T07:36:52.677 INFO:teuthology.orchestra.run.smithi052.stdout: "log_to_file": { 2024-05-27T07:36:52.677 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "log_to_file", 2024-05-27T07:36:52.677 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "bool", 2024-05-27T07:36:52.677 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-05-27T07:36:52.677 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 1, 2024-05-27T07:36:52.678 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "False", 2024-05-27T07:36:52.678 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-05-27T07:36:52.678 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-05-27T07:36:52.678 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-05-27T07:36:52.678 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-05-27T07:36:52.678 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-05-27T07:36:52.678 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-05-27T07:36:52.678 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-05-27T07:36:52.678 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-05-27T07:36:52.678 INFO:teuthology.orchestra.run.smithi052.stdout: "server_addr": { 2024-05-27T07:36:52.678 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "server_addr", 2024-05-27T07:36:52.678 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "str", 2024-05-27T07:36:52.678 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-05-27T07:36:52.679 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 0, 2024-05-27T07:36:52.679 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "", 2024-05-27T07:36:52.679 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-05-27T07:36:52.679 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-05-27T07:36:52.679 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-05-27T07:36:52.679 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-05-27T07:36:52.679 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-05-27T07:36:52.679 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-05-27T07:36:52.679 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-05-27T07:36:52.679 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-05-27T07:36:52.679 INFO:teuthology.orchestra.run.smithi052.stdout: "server_port": { 2024-05-27T07:36:52.679 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "server_port", 2024-05-27T07:36:52.679 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "str", 2024-05-27T07:36:52.680 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-05-27T07:36:52.680 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 0, 2024-05-27T07:36:52.680 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "", 2024-05-27T07:36:52.680 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-05-27T07:36:52.680 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-05-27T07:36:52.680 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-05-27T07:36:52.680 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-05-27T07:36:52.680 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-05-27T07:36:52.680 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-05-27T07:36:52.680 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-05-27T07:36:52.680 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-05-27T07:36:52.680 INFO:teuthology.orchestra.run.smithi052.stdout: "sqlite3_killpoint": { 2024-05-27T07:36:52.681 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "sqlite3_killpoint", 2024-05-27T07:36:52.681 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "int", 2024-05-27T07:36:52.681 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "dev", 2024-05-27T07:36:52.681 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 1, 2024-05-27T07:36:52.681 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "0", 2024-05-27T07:36:52.681 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-05-27T07:36:52.681 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-05-27T07:36:52.681 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-05-27T07:36:52.681 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-05-27T07:36:52.681 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-05-27T07:36:52.681 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-05-27T07:36:52.681 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-05-27T07:36:52.681 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-05-27T07:36:52.682 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-05-27T07:36:52.682 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-05-27T07:36:52.682 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-05-27T07:36:52.682 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "rgw", 2024-05-27T07:36:52.682 INFO:teuthology.orchestra.run.smithi052.stdout: "can_run": true, 2024-05-27T07:36:52.682 INFO:teuthology.orchestra.run.smithi052.stdout: "error_string": "", 2024-05-27T07:36:52.682 INFO:teuthology.orchestra.run.smithi052.stdout: "module_options": { 2024-05-27T07:36:52.682 INFO:teuthology.orchestra.run.smithi052.stdout: "log_level": { 2024-05-27T07:36:52.682 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "log_level", 2024-05-27T07:36:52.682 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "str", 2024-05-27T07:36:52.682 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-05-27T07:36:52.682 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 1, 2024-05-27T07:36:52.682 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "", 2024-05-27T07:36:52.683 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-05-27T07:36:52.683 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-05-27T07:36:52.683 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [ 2024-05-27T07:36:52.683 INFO:teuthology.orchestra.run.smithi052.stdout: "", 2024-05-27T07:36:52.683 INFO:teuthology.orchestra.run.smithi052.stdout: "critical", 2024-05-27T07:36:52.683 INFO:teuthology.orchestra.run.smithi052.stdout: "debug", 2024-05-27T07:36:52.683 INFO:teuthology.orchestra.run.smithi052.stdout: "error", 2024-05-27T07:36:52.683 INFO:teuthology.orchestra.run.smithi052.stdout: "info", 2024-05-27T07:36:52.683 INFO:teuthology.orchestra.run.smithi052.stdout: "warning" 2024-05-27T07:36:52.683 INFO:teuthology.orchestra.run.smithi052.stdout: ], 2024-05-27T07:36:52.683 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-05-27T07:36:52.683 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-05-27T07:36:52.683 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-05-27T07:36:52.683 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-05-27T07:36:52.684 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-05-27T07:36:52.684 INFO:teuthology.orchestra.run.smithi052.stdout: "log_to_cluster": { 2024-05-27T07:36:52.684 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "log_to_cluster", 2024-05-27T07:36:52.684 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "bool", 2024-05-27T07:36:52.684 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-05-27T07:36:52.684 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 1, 2024-05-27T07:36:52.684 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "False", 2024-05-27T07:36:52.684 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-05-27T07:36:52.684 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-05-27T07:36:52.684 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-05-27T07:36:52.684 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-05-27T07:36:52.684 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-05-27T07:36:52.684 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-05-27T07:36:52.685 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-05-27T07:36:52.685 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-05-27T07:36:52.685 INFO:teuthology.orchestra.run.smithi052.stdout: "log_to_cluster_level": { 2024-05-27T07:36:52.685 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "log_to_cluster_level", 2024-05-27T07:36:52.685 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "str", 2024-05-27T07:36:52.685 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-05-27T07:36:52.685 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 1, 2024-05-27T07:36:52.685 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "info", 2024-05-27T07:36:52.685 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-05-27T07:36:52.685 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-05-27T07:36:52.685 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [ 2024-05-27T07:36:52.685 INFO:teuthology.orchestra.run.smithi052.stdout: "", 2024-05-27T07:36:52.685 INFO:teuthology.orchestra.run.smithi052.stdout: "critical", 2024-05-27T07:36:52.686 INFO:teuthology.orchestra.run.smithi052.stdout: "debug", 2024-05-27T07:36:52.686 INFO:teuthology.orchestra.run.smithi052.stdout: "error", 2024-05-27T07:36:52.686 INFO:teuthology.orchestra.run.smithi052.stdout: "info", 2024-05-27T07:36:52.686 INFO:teuthology.orchestra.run.smithi052.stdout: "warning" 2024-05-27T07:36:52.686 INFO:teuthology.orchestra.run.smithi052.stdout: ], 2024-05-27T07:36:52.686 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-05-27T07:36:52.686 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-05-27T07:36:52.686 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-05-27T07:36:52.686 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-05-27T07:36:52.686 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-05-27T07:36:52.686 INFO:teuthology.orchestra.run.smithi052.stdout: "log_to_file": { 2024-05-27T07:36:52.686 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "log_to_file", 2024-05-27T07:36:52.686 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "bool", 2024-05-27T07:36:52.687 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-05-27T07:36:52.687 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 1, 2024-05-27T07:36:52.687 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "False", 2024-05-27T07:36:52.687 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-05-27T07:36:52.687 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-05-27T07:36:52.687 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-05-27T07:36:52.687 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-05-27T07:36:52.687 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-05-27T07:36:52.687 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-05-27T07:36:52.687 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-05-27T07:36:52.687 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-05-27T07:36:52.687 INFO:teuthology.orchestra.run.smithi052.stdout: "sqlite3_killpoint": { 2024-05-27T07:36:52.687 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "sqlite3_killpoint", 2024-05-27T07:36:52.688 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "int", 2024-05-27T07:36:52.688 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "dev", 2024-05-27T07:36:52.688 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 1, 2024-05-27T07:36:52.688 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "0", 2024-05-27T07:36:52.688 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-05-27T07:36:52.688 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-05-27T07:36:52.688 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-05-27T07:36:52.688 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-05-27T07:36:52.688 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-05-27T07:36:52.688 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-05-27T07:36:52.688 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-05-27T07:36:52.688 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-05-27T07:36:52.688 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-05-27T07:36:52.689 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-05-27T07:36:52.689 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-05-27T07:36:52.689 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "rook", 2024-05-27T07:36:52.689 INFO:teuthology.orchestra.run.smithi052.stdout: "can_run": true, 2024-05-27T07:36:52.689 INFO:teuthology.orchestra.run.smithi052.stdout: "error_string": "", 2024-05-27T07:36:52.689 INFO:teuthology.orchestra.run.smithi052.stdout: "module_options": { 2024-05-27T07:36:52.689 INFO:teuthology.orchestra.run.smithi052.stdout: "log_level": { 2024-05-27T07:36:52.689 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "log_level", 2024-05-27T07:36:52.689 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "str", 2024-05-27T07:36:52.689 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-05-27T07:36:52.689 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 1, 2024-05-27T07:36:52.689 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "", 2024-05-27T07:36:52.689 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-05-27T07:36:52.690 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-05-27T07:36:52.690 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [ 2024-05-27T07:36:52.690 INFO:teuthology.orchestra.run.smithi052.stdout: "", 2024-05-27T07:36:52.690 INFO:teuthology.orchestra.run.smithi052.stdout: "critical", 2024-05-27T07:36:52.690 INFO:teuthology.orchestra.run.smithi052.stdout: "debug", 2024-05-27T07:36:52.690 INFO:teuthology.orchestra.run.smithi052.stdout: "error", 2024-05-27T07:36:52.690 INFO:teuthology.orchestra.run.smithi052.stdout: "info", 2024-05-27T07:36:52.690 INFO:teuthology.orchestra.run.smithi052.stdout: "warning" 2024-05-27T07:36:52.690 INFO:teuthology.orchestra.run.smithi052.stdout: ], 2024-05-27T07:36:52.690 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-05-27T07:36:52.690 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-05-27T07:36:52.690 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-05-27T07:36:52.691 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-05-27T07:36:52.691 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-05-27T07:36:52.691 INFO:teuthology.orchestra.run.smithi052.stdout: "log_to_cluster": { 2024-05-27T07:36:52.691 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "log_to_cluster", 2024-05-27T07:36:52.691 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "bool", 2024-05-27T07:36:52.691 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-05-27T07:36:52.691 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 1, 2024-05-27T07:36:52.691 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "False", 2024-05-27T07:36:52.691 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-05-27T07:36:52.691 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-05-27T07:36:52.691 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-05-27T07:36:52.691 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-05-27T07:36:52.691 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-05-27T07:36:52.692 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-05-27T07:36:52.692 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-05-27T07:36:52.692 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-05-27T07:36:52.692 INFO:teuthology.orchestra.run.smithi052.stdout: "log_to_cluster_level": { 2024-05-27T07:36:52.692 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "log_to_cluster_level", 2024-05-27T07:36:52.692 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "str", 2024-05-27T07:36:52.692 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-05-27T07:36:52.692 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 1, 2024-05-27T07:36:52.692 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "info", 2024-05-27T07:36:52.692 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-05-27T07:36:52.692 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-05-27T07:36:52.692 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [ 2024-05-27T07:36:52.692 INFO:teuthology.orchestra.run.smithi052.stdout: "", 2024-05-27T07:36:52.693 INFO:teuthology.orchestra.run.smithi052.stdout: "critical", 2024-05-27T07:36:52.693 INFO:teuthology.orchestra.run.smithi052.stdout: "debug", 2024-05-27T07:36:52.693 INFO:teuthology.orchestra.run.smithi052.stdout: "error", 2024-05-27T07:36:52.693 INFO:teuthology.orchestra.run.smithi052.stdout: "info", 2024-05-27T07:36:52.693 INFO:teuthology.orchestra.run.smithi052.stdout: "warning" 2024-05-27T07:36:52.693 INFO:teuthology.orchestra.run.smithi052.stdout: ], 2024-05-27T07:36:52.693 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-05-27T07:36:52.693 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-05-27T07:36:52.693 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-05-27T07:36:52.693 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-05-27T07:36:52.693 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-05-27T07:36:52.693 INFO:teuthology.orchestra.run.smithi052.stdout: "log_to_file": { 2024-05-27T07:36:52.693 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "log_to_file", 2024-05-27T07:36:52.694 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "bool", 2024-05-27T07:36:52.694 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-05-27T07:36:52.694 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 1, 2024-05-27T07:36:52.694 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "False", 2024-05-27T07:36:52.694 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-05-27T07:36:52.694 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-05-27T07:36:52.694 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-05-27T07:36:52.694 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-05-27T07:36:52.694 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-05-27T07:36:52.694 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-05-27T07:36:52.694 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-05-27T07:36:52.694 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-05-27T07:36:52.694 INFO:teuthology.orchestra.run.smithi052.stdout: "sqlite3_killpoint": { 2024-05-27T07:36:52.695 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "sqlite3_killpoint", 2024-05-27T07:36:52.695 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "int", 2024-05-27T07:36:52.695 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "dev", 2024-05-27T07:36:52.695 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 1, 2024-05-27T07:36:52.695 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "0", 2024-05-27T07:36:52.695 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-05-27T07:36:52.695 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-05-27T07:36:52.695 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-05-27T07:36:52.695 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-05-27T07:36:52.695 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-05-27T07:36:52.695 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-05-27T07:36:52.695 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-05-27T07:36:52.695 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-05-27T07:36:52.696 INFO:teuthology.orchestra.run.smithi052.stdout: "storage_class": { 2024-05-27T07:36:52.696 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "storage_class", 2024-05-27T07:36:52.696 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "str", 2024-05-27T07:36:52.696 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-05-27T07:36:52.696 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 0, 2024-05-27T07:36:52.696 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "local", 2024-05-27T07:36:52.696 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-05-27T07:36:52.696 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-05-27T07:36:52.696 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-05-27T07:36:52.696 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "storage class name for LSO-discovered PVs", 2024-05-27T07:36:52.696 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-05-27T07:36:52.696 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-05-27T07:36:52.696 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-05-27T07:36:52.697 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-05-27T07:36:52.697 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-05-27T07:36:52.697 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-05-27T07:36:52.697 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-05-27T07:36:52.697 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "selftest", 2024-05-27T07:36:52.697 INFO:teuthology.orchestra.run.smithi052.stdout: "can_run": true, 2024-05-27T07:36:52.697 INFO:teuthology.orchestra.run.smithi052.stdout: "error_string": "", 2024-05-27T07:36:52.697 INFO:teuthology.orchestra.run.smithi052.stdout: "module_options": { 2024-05-27T07:36:52.697 INFO:teuthology.orchestra.run.smithi052.stdout: "log_level": { 2024-05-27T07:36:52.697 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "log_level", 2024-05-27T07:36:52.697 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "str", 2024-05-27T07:36:52.697 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-05-27T07:36:52.697 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 1, 2024-05-27T07:36:52.697 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "", 2024-05-27T07:36:52.698 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-05-27T07:36:52.698 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-05-27T07:36:52.698 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [ 2024-05-27T07:36:52.698 INFO:teuthology.orchestra.run.smithi052.stdout: "", 2024-05-27T07:36:52.698 INFO:teuthology.orchestra.run.smithi052.stdout: "critical", 2024-05-27T07:36:52.698 INFO:teuthology.orchestra.run.smithi052.stdout: "debug", 2024-05-27T07:36:52.698 INFO:teuthology.orchestra.run.smithi052.stdout: "error", 2024-05-27T07:36:52.698 INFO:teuthology.orchestra.run.smithi052.stdout: "info", 2024-05-27T07:36:52.698 INFO:teuthology.orchestra.run.smithi052.stdout: "warning" 2024-05-27T07:36:52.698 INFO:teuthology.orchestra.run.smithi052.stdout: ], 2024-05-27T07:36:52.698 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-05-27T07:36:52.698 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-05-27T07:36:52.698 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-05-27T07:36:52.699 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-05-27T07:36:52.699 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-05-27T07:36:52.699 INFO:teuthology.orchestra.run.smithi052.stdout: "log_to_cluster": { 2024-05-27T07:36:52.699 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "log_to_cluster", 2024-05-27T07:36:52.699 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "bool", 2024-05-27T07:36:52.699 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-05-27T07:36:52.699 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 1, 2024-05-27T07:36:52.699 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "False", 2024-05-27T07:36:52.699 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-05-27T07:36:52.699 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-05-27T07:36:52.699 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-05-27T07:36:52.699 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-05-27T07:36:52.699 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-05-27T07:36:52.699 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-05-27T07:36:52.700 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-05-27T07:36:52.700 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-05-27T07:36:52.700 INFO:teuthology.orchestra.run.smithi052.stdout: "log_to_cluster_level": { 2024-05-27T07:36:52.700 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "log_to_cluster_level", 2024-05-27T07:36:52.700 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "str", 2024-05-27T07:36:52.700 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-05-27T07:36:52.700 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 1, 2024-05-27T07:36:52.700 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "info", 2024-05-27T07:36:52.700 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-05-27T07:36:52.700 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-05-27T07:36:52.700 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [ 2024-05-27T07:36:52.700 INFO:teuthology.orchestra.run.smithi052.stdout: "", 2024-05-27T07:36:52.700 INFO:teuthology.orchestra.run.smithi052.stdout: "critical", 2024-05-27T07:36:52.701 INFO:teuthology.orchestra.run.smithi052.stdout: "debug", 2024-05-27T07:36:52.701 INFO:teuthology.orchestra.run.smithi052.stdout: "error", 2024-05-27T07:36:52.701 INFO:teuthology.orchestra.run.smithi052.stdout: "info", 2024-05-27T07:36:52.701 INFO:teuthology.orchestra.run.smithi052.stdout: "warning" 2024-05-27T07:36:52.701 INFO:teuthology.orchestra.run.smithi052.stdout: ], 2024-05-27T07:36:52.701 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-05-27T07:36:52.701 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-05-27T07:36:52.701 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-05-27T07:36:52.701 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-05-27T07:36:52.701 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-05-27T07:36:52.701 INFO:teuthology.orchestra.run.smithi052.stdout: "log_to_file": { 2024-05-27T07:36:52.701 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "log_to_file", 2024-05-27T07:36:52.701 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "bool", 2024-05-27T07:36:52.701 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-05-27T07:36:52.702 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 1, 2024-05-27T07:36:52.702 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "False", 2024-05-27T07:36:52.702 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-05-27T07:36:52.702 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-05-27T07:36:52.702 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-05-27T07:36:52.702 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-05-27T07:36:52.702 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-05-27T07:36:52.702 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-05-27T07:36:52.702 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-05-27T07:36:52.702 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-05-27T07:36:52.702 INFO:teuthology.orchestra.run.smithi052.stdout: "roption1": { 2024-05-27T07:36:52.702 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "roption1", 2024-05-27T07:36:52.703 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "str", 2024-05-27T07:36:52.703 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-05-27T07:36:52.703 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 0, 2024-05-27T07:36:52.703 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "", 2024-05-27T07:36:52.703 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-05-27T07:36:52.703 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-05-27T07:36:52.703 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-05-27T07:36:52.703 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-05-27T07:36:52.703 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-05-27T07:36:52.703 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-05-27T07:36:52.703 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-05-27T07:36:52.703 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-05-27T07:36:52.703 INFO:teuthology.orchestra.run.smithi052.stdout: "roption2": { 2024-05-27T07:36:52.703 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "roption2", 2024-05-27T07:36:52.704 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "str", 2024-05-27T07:36:52.704 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-05-27T07:36:52.704 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 0, 2024-05-27T07:36:52.704 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "xyz", 2024-05-27T07:36:52.704 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-05-27T07:36:52.704 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-05-27T07:36:52.704 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-05-27T07:36:52.704 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-05-27T07:36:52.704 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-05-27T07:36:52.704 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-05-27T07:36:52.704 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-05-27T07:36:52.704 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-05-27T07:36:52.704 INFO:teuthology.orchestra.run.smithi052.stdout: "rwoption1": { 2024-05-27T07:36:52.705 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "rwoption1", 2024-05-27T07:36:52.705 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "str", 2024-05-27T07:36:52.705 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-05-27T07:36:52.705 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 0, 2024-05-27T07:36:52.705 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "", 2024-05-27T07:36:52.705 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-05-27T07:36:52.705 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-05-27T07:36:52.705 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-05-27T07:36:52.705 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-05-27T07:36:52.705 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-05-27T07:36:52.705 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-05-27T07:36:52.705 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-05-27T07:36:52.705 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-05-27T07:36:52.706 INFO:teuthology.orchestra.run.smithi052.stdout: "rwoption2": { 2024-05-27T07:36:52.706 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "rwoption2", 2024-05-27T07:36:52.706 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "int", 2024-05-27T07:36:52.706 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-05-27T07:36:52.706 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 0, 2024-05-27T07:36:52.706 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "", 2024-05-27T07:36:52.706 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-05-27T07:36:52.706 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-05-27T07:36:52.706 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-05-27T07:36:52.706 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-05-27T07:36:52.706 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-05-27T07:36:52.706 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-05-27T07:36:52.706 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-05-27T07:36:52.707 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-05-27T07:36:52.707 INFO:teuthology.orchestra.run.smithi052.stdout: "rwoption3": { 2024-05-27T07:36:52.707 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "rwoption3", 2024-05-27T07:36:52.707 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "float", 2024-05-27T07:36:52.707 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-05-27T07:36:52.707 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 0, 2024-05-27T07:36:52.707 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "", 2024-05-27T07:36:52.707 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-05-27T07:36:52.707 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-05-27T07:36:52.707 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-05-27T07:36:52.707 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-05-27T07:36:52.707 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-05-27T07:36:52.707 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-05-27T07:36:52.707 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-05-27T07:36:52.708 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-05-27T07:36:52.708 INFO:teuthology.orchestra.run.smithi052.stdout: "rwoption4": { 2024-05-27T07:36:52.708 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "rwoption4", 2024-05-27T07:36:52.708 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "str", 2024-05-27T07:36:52.708 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-05-27T07:36:52.708 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 0, 2024-05-27T07:36:52.708 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "", 2024-05-27T07:36:52.708 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-05-27T07:36:52.708 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-05-27T07:36:52.708 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-05-27T07:36:52.708 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-05-27T07:36:52.708 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-05-27T07:36:52.708 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-05-27T07:36:52.709 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-05-27T07:36:52.709 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-05-27T07:36:52.709 INFO:teuthology.orchestra.run.smithi052.stdout: "rwoption5": { 2024-05-27T07:36:52.709 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "rwoption5", 2024-05-27T07:36:52.709 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "bool", 2024-05-27T07:36:52.709 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-05-27T07:36:52.709 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 0, 2024-05-27T07:36:52.709 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "", 2024-05-27T07:36:52.709 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-05-27T07:36:52.709 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-05-27T07:36:52.709 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-05-27T07:36:52.709 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-05-27T07:36:52.709 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-05-27T07:36:52.710 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-05-27T07:36:52.710 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-05-27T07:36:52.710 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-05-27T07:36:52.710 INFO:teuthology.orchestra.run.smithi052.stdout: "rwoption6": { 2024-05-27T07:36:52.710 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "rwoption6", 2024-05-27T07:36:52.710 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "bool", 2024-05-27T07:36:52.710 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-05-27T07:36:52.710 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 0, 2024-05-27T07:36:52.710 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "True", 2024-05-27T07:36:52.710 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-05-27T07:36:52.710 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-05-27T07:36:52.710 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-05-27T07:36:52.710 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-05-27T07:36:52.710 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-05-27T07:36:52.711 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-05-27T07:36:52.711 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-05-27T07:36:52.711 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-05-27T07:36:52.711 INFO:teuthology.orchestra.run.smithi052.stdout: "rwoption7": { 2024-05-27T07:36:52.711 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "rwoption7", 2024-05-27T07:36:52.711 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "int", 2024-05-27T07:36:52.711 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-05-27T07:36:52.711 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 0, 2024-05-27T07:36:52.711 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "", 2024-05-27T07:36:52.711 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "1", 2024-05-27T07:36:52.711 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "42", 2024-05-27T07:36:52.711 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-05-27T07:36:52.711 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-05-27T07:36:52.712 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-05-27T07:36:52.712 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-05-27T07:36:52.712 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-05-27T07:36:52.712 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-05-27T07:36:52.712 INFO:teuthology.orchestra.run.smithi052.stdout: "sqlite3_killpoint": { 2024-05-27T07:36:52.712 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "sqlite3_killpoint", 2024-05-27T07:36:52.712 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "int", 2024-05-27T07:36:52.712 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "dev", 2024-05-27T07:36:52.712 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 1, 2024-05-27T07:36:52.712 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "0", 2024-05-27T07:36:52.712 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-05-27T07:36:52.712 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-05-27T07:36:52.712 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-05-27T07:36:52.713 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-05-27T07:36:52.713 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-05-27T07:36:52.713 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-05-27T07:36:52.713 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-05-27T07:36:52.713 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-05-27T07:36:52.713 INFO:teuthology.orchestra.run.smithi052.stdout: "testkey": { 2024-05-27T07:36:52.713 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "testkey", 2024-05-27T07:36:52.713 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "str", 2024-05-27T07:36:52.713 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-05-27T07:36:52.713 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 0, 2024-05-27T07:36:52.713 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "", 2024-05-27T07:36:52.713 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-05-27T07:36:52.713 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-05-27T07:36:52.714 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-05-27T07:36:52.714 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-05-27T07:36:52.714 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-05-27T07:36:52.714 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-05-27T07:36:52.714 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-05-27T07:36:52.714 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-05-27T07:36:52.714 INFO:teuthology.orchestra.run.smithi052.stdout: "testlkey": { 2024-05-27T07:36:52.714 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "testlkey", 2024-05-27T07:36:52.714 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "str", 2024-05-27T07:36:52.714 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-05-27T07:36:52.714 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 0, 2024-05-27T07:36:52.714 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "", 2024-05-27T07:36:52.714 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-05-27T07:36:52.715 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-05-27T07:36:52.715 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-05-27T07:36:52.715 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-05-27T07:36:52.715 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-05-27T07:36:52.715 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-05-27T07:36:52.715 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-05-27T07:36:52.715 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-05-27T07:36:52.715 INFO:teuthology.orchestra.run.smithi052.stdout: "testnewline": { 2024-05-27T07:36:52.715 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "testnewline", 2024-05-27T07:36:52.715 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "str", 2024-05-27T07:36:52.715 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-05-27T07:36:52.715 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 0, 2024-05-27T07:36:52.715 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "", 2024-05-27T07:36:52.715 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-05-27T07:36:52.716 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-05-27T07:36:52.716 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-05-27T07:36:52.716 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-05-27T07:36:52.716 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-05-27T07:36:52.716 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-05-27T07:36:52.716 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-05-27T07:36:52.716 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-05-27T07:36:52.716 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-05-27T07:36:52.716 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-05-27T07:36:52.716 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-05-27T07:36:52.716 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "smb", 2024-05-27T07:36:52.716 INFO:teuthology.orchestra.run.smithi052.stdout: "can_run": true, 2024-05-27T07:36:52.716 INFO:teuthology.orchestra.run.smithi052.stdout: "error_string": "", 2024-05-27T07:36:52.717 INFO:teuthology.orchestra.run.smithi052.stdout: "module_options": { 2024-05-27T07:36:52.717 INFO:teuthology.orchestra.run.smithi052.stdout: "log_level": { 2024-05-27T07:36:52.717 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "log_level", 2024-05-27T07:36:52.717 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "str", 2024-05-27T07:36:52.717 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-05-27T07:36:52.717 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 1, 2024-05-27T07:36:52.717 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "", 2024-05-27T07:36:52.717 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-05-27T07:36:52.717 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-05-27T07:36:52.717 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [ 2024-05-27T07:36:52.717 INFO:teuthology.orchestra.run.smithi052.stdout: "", 2024-05-27T07:36:52.717 INFO:teuthology.orchestra.run.smithi052.stdout: "critical", 2024-05-27T07:36:52.718 INFO:teuthology.orchestra.run.smithi052.stdout: "debug", 2024-05-27T07:36:52.718 INFO:teuthology.orchestra.run.smithi052.stdout: "error", 2024-05-27T07:36:52.718 INFO:teuthology.orchestra.run.smithi052.stdout: "info", 2024-05-27T07:36:52.718 INFO:teuthology.orchestra.run.smithi052.stdout: "warning" 2024-05-27T07:36:52.718 INFO:teuthology.orchestra.run.smithi052.stdout: ], 2024-05-27T07:36:52.718 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-05-27T07:36:52.718 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-05-27T07:36:52.718 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-05-27T07:36:52.718 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-05-27T07:36:52.718 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-05-27T07:36:52.718 INFO:teuthology.orchestra.run.smithi052.stdout: "log_to_cluster": { 2024-05-27T07:36:52.718 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "log_to_cluster", 2024-05-27T07:36:52.718 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "bool", 2024-05-27T07:36:52.719 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-05-27T07:36:52.719 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 1, 2024-05-27T07:36:52.719 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "False", 2024-05-27T07:36:52.719 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-05-27T07:36:52.719 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-05-27T07:36:52.719 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-05-27T07:36:52.719 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-05-27T07:36:52.719 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-05-27T07:36:52.719 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-05-27T07:36:52.719 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-05-27T07:36:52.719 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-05-27T07:36:52.719 INFO:teuthology.orchestra.run.smithi052.stdout: "log_to_cluster_level": { 2024-05-27T07:36:52.720 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "log_to_cluster_level", 2024-05-27T07:36:52.720 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "str", 2024-05-27T07:36:52.720 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-05-27T07:36:52.720 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 1, 2024-05-27T07:36:52.720 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "info", 2024-05-27T07:36:52.720 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-05-27T07:36:52.720 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-05-27T07:36:52.720 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [ 2024-05-27T07:36:52.720 INFO:teuthology.orchestra.run.smithi052.stdout: "", 2024-05-27T07:36:52.720 INFO:teuthology.orchestra.run.smithi052.stdout: "critical", 2024-05-27T07:36:52.720 INFO:teuthology.orchestra.run.smithi052.stdout: "debug", 2024-05-27T07:36:52.720 INFO:teuthology.orchestra.run.smithi052.stdout: "error", 2024-05-27T07:36:52.720 INFO:teuthology.orchestra.run.smithi052.stdout: "info", 2024-05-27T07:36:52.721 INFO:teuthology.orchestra.run.smithi052.stdout: "warning" 2024-05-27T07:36:52.721 INFO:teuthology.orchestra.run.smithi052.stdout: ], 2024-05-27T07:36:52.721 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-05-27T07:36:52.721 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-05-27T07:36:52.721 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-05-27T07:36:52.721 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-05-27T07:36:52.721 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-05-27T07:36:52.721 INFO:teuthology.orchestra.run.smithi052.stdout: "log_to_file": { 2024-05-27T07:36:52.721 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "log_to_file", 2024-05-27T07:36:52.721 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "bool", 2024-05-27T07:36:52.721 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-05-27T07:36:52.721 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 1, 2024-05-27T07:36:52.721 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "False", 2024-05-27T07:36:52.722 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-05-27T07:36:52.722 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-05-27T07:36:52.722 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-05-27T07:36:52.722 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-05-27T07:36:52.722 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-05-27T07:36:52.722 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-05-27T07:36:52.722 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-05-27T07:36:52.722 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-05-27T07:36:52.722 INFO:teuthology.orchestra.run.smithi052.stdout: "sqlite3_killpoint": { 2024-05-27T07:36:52.722 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "sqlite3_killpoint", 2024-05-27T07:36:52.722 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "int", 2024-05-27T07:36:52.722 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "dev", 2024-05-27T07:36:52.723 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 1, 2024-05-27T07:36:52.723 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "0", 2024-05-27T07:36:52.723 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-05-27T07:36:52.723 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-05-27T07:36:52.723 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-05-27T07:36:52.723 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-05-27T07:36:52.723 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-05-27T07:36:52.723 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-05-27T07:36:52.723 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-05-27T07:36:52.723 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-05-27T07:36:52.723 INFO:teuthology.orchestra.run.smithi052.stdout: "update_orchestration": { 2024-05-27T07:36:52.723 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "update_orchestration", 2024-05-27T07:36:52.723 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "bool", 2024-05-27T07:36:52.724 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-05-27T07:36:52.724 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 0, 2024-05-27T07:36:52.724 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "True", 2024-05-27T07:36:52.724 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-05-27T07:36:52.724 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-05-27T07:36:52.724 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-05-27T07:36:52.724 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "automatically update orchestration when smb resources are changed", 2024-05-27T07:36:52.724 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-05-27T07:36:52.724 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-05-27T07:36:52.724 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-05-27T07:36:52.724 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-05-27T07:36:52.724 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-05-27T07:36:52.724 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-05-27T07:36:52.725 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-05-27T07:36:52.725 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "snap_schedule", 2024-05-27T07:36:52.725 INFO:teuthology.orchestra.run.smithi052.stdout: "can_run": true, 2024-05-27T07:36:52.725 INFO:teuthology.orchestra.run.smithi052.stdout: "error_string": "", 2024-05-27T07:36:52.725 INFO:teuthology.orchestra.run.smithi052.stdout: "module_options": { 2024-05-27T07:36:52.725 INFO:teuthology.orchestra.run.smithi052.stdout: "allow_m_granularity": { 2024-05-27T07:36:52.725 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "allow_m_granularity", 2024-05-27T07:36:52.725 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "bool", 2024-05-27T07:36:52.725 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-05-27T07:36:52.725 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 1, 2024-05-27T07:36:52.725 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "False", 2024-05-27T07:36:52.725 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-05-27T07:36:52.725 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-05-27T07:36:52.725 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-05-27T07:36:52.726 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "allow minute scheduled snapshots", 2024-05-27T07:36:52.726 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-05-27T07:36:52.726 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-05-27T07:36:52.726 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-05-27T07:36:52.726 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-05-27T07:36:52.726 INFO:teuthology.orchestra.run.smithi052.stdout: "dump_on_update": { 2024-05-27T07:36:52.726 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "dump_on_update", 2024-05-27T07:36:52.726 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "bool", 2024-05-27T07:36:52.726 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-05-27T07:36:52.726 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 1, 2024-05-27T07:36:52.726 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "False", 2024-05-27T07:36:52.726 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-05-27T07:36:52.727 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-05-27T07:36:52.727 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-05-27T07:36:52.727 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "dump database to debug log on update", 2024-05-27T07:36:52.727 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-05-27T07:36:52.727 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-05-27T07:36:52.727 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-05-27T07:36:52.727 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-05-27T07:36:52.727 INFO:teuthology.orchestra.run.smithi052.stdout: "log_level": { 2024-05-27T07:36:52.727 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "log_level", 2024-05-27T07:36:52.727 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "str", 2024-05-27T07:36:52.727 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-05-27T07:36:52.728 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 1, 2024-05-27T07:36:52.728 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "", 2024-05-27T07:36:52.728 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-05-27T07:36:52.728 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-05-27T07:36:52.728 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [ 2024-05-27T07:36:52.728 INFO:teuthology.orchestra.run.smithi052.stdout: "", 2024-05-27T07:36:52.728 INFO:teuthology.orchestra.run.smithi052.stdout: "critical", 2024-05-27T07:36:52.728 INFO:teuthology.orchestra.run.smithi052.stdout: "debug", 2024-05-27T07:36:52.728 INFO:teuthology.orchestra.run.smithi052.stdout: "error", 2024-05-27T07:36:52.728 INFO:teuthology.orchestra.run.smithi052.stdout: "info", 2024-05-27T07:36:52.728 INFO:teuthology.orchestra.run.smithi052.stdout: "warning" 2024-05-27T07:36:52.728 INFO:teuthology.orchestra.run.smithi052.stdout: ], 2024-05-27T07:36:52.728 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-05-27T07:36:52.728 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-05-27T07:36:52.729 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-05-27T07:36:52.729 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-05-27T07:36:52.729 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-05-27T07:36:52.729 INFO:teuthology.orchestra.run.smithi052.stdout: "log_to_cluster": { 2024-05-27T07:36:52.729 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "log_to_cluster", 2024-05-27T07:36:52.729 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "bool", 2024-05-27T07:36:52.729 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-05-27T07:36:52.729 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 1, 2024-05-27T07:36:52.729 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "False", 2024-05-27T07:36:52.729 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-05-27T07:36:52.729 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-05-27T07:36:52.729 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-05-27T07:36:52.729 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-05-27T07:36:52.730 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-05-27T07:36:52.730 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-05-27T07:36:52.730 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-05-27T07:36:52.730 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-05-27T07:36:52.730 INFO:teuthology.orchestra.run.smithi052.stdout: "log_to_cluster_level": { 2024-05-27T07:36:52.730 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "log_to_cluster_level", 2024-05-27T07:36:52.730 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "str", 2024-05-27T07:36:52.730 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-05-27T07:36:52.730 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 1, 2024-05-27T07:36:52.730 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "info", 2024-05-27T07:36:52.730 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-05-27T07:36:52.730 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-05-27T07:36:52.730 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [ 2024-05-27T07:36:52.731 INFO:teuthology.orchestra.run.smithi052.stdout: "", 2024-05-27T07:36:52.731 INFO:teuthology.orchestra.run.smithi052.stdout: "critical", 2024-05-27T07:36:52.731 INFO:teuthology.orchestra.run.smithi052.stdout: "debug", 2024-05-27T07:36:52.731 INFO:teuthology.orchestra.run.smithi052.stdout: "error", 2024-05-27T07:36:52.731 INFO:teuthology.orchestra.run.smithi052.stdout: "info", 2024-05-27T07:36:52.731 INFO:teuthology.orchestra.run.smithi052.stdout: "warning" 2024-05-27T07:36:52.731 INFO:teuthology.orchestra.run.smithi052.stdout: ], 2024-05-27T07:36:52.731 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-05-27T07:36:52.731 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-05-27T07:36:52.731 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-05-27T07:36:52.731 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-05-27T07:36:52.731 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-05-27T07:36:52.732 INFO:teuthology.orchestra.run.smithi052.stdout: "log_to_file": { 2024-05-27T07:36:52.732 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "log_to_file", 2024-05-27T07:36:52.732 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "bool", 2024-05-27T07:36:52.732 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-05-27T07:36:52.732 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 1, 2024-05-27T07:36:52.732 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "False", 2024-05-27T07:36:52.732 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-05-27T07:36:52.732 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-05-27T07:36:52.732 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-05-27T07:36:52.732 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-05-27T07:36:52.732 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-05-27T07:36:52.732 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-05-27T07:36:52.733 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-05-27T07:36:52.733 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-05-27T07:36:52.733 INFO:teuthology.orchestra.run.smithi052.stdout: "sqlite3_killpoint": { 2024-05-27T07:36:52.733 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "sqlite3_killpoint", 2024-05-27T07:36:52.733 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "int", 2024-05-27T07:36:52.733 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "dev", 2024-05-27T07:36:52.733 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 1, 2024-05-27T07:36:52.733 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "0", 2024-05-27T07:36:52.733 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-05-27T07:36:52.733 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-05-27T07:36:52.733 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-05-27T07:36:52.733 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-05-27T07:36:52.733 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-05-27T07:36:52.734 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-05-27T07:36:52.734 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-05-27T07:36:52.734 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-05-27T07:36:52.734 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-05-27T07:36:52.734 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-05-27T07:36:52.734 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-05-27T07:36:52.734 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "stats", 2024-05-27T07:36:52.734 INFO:teuthology.orchestra.run.smithi052.stdout: "can_run": true, 2024-05-27T07:36:52.734 INFO:teuthology.orchestra.run.smithi052.stdout: "error_string": "", 2024-05-27T07:36:52.734 INFO:teuthology.orchestra.run.smithi052.stdout: "module_options": { 2024-05-27T07:36:52.734 INFO:teuthology.orchestra.run.smithi052.stdout: "log_level": { 2024-05-27T07:36:52.734 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "log_level", 2024-05-27T07:36:52.735 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "str", 2024-05-27T07:36:52.735 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-05-27T07:36:52.735 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 1, 2024-05-27T07:36:52.735 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "", 2024-05-27T07:36:52.735 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-05-27T07:36:52.735 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-05-27T07:36:52.735 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [ 2024-05-27T07:36:52.735 INFO:teuthology.orchestra.run.smithi052.stdout: "", 2024-05-27T07:36:52.735 INFO:teuthology.orchestra.run.smithi052.stdout: "critical", 2024-05-27T07:36:52.735 INFO:teuthology.orchestra.run.smithi052.stdout: "debug", 2024-05-27T07:36:52.735 INFO:teuthology.orchestra.run.smithi052.stdout: "error", 2024-05-27T07:36:52.735 INFO:teuthology.orchestra.run.smithi052.stdout: "info", 2024-05-27T07:36:52.735 INFO:teuthology.orchestra.run.smithi052.stdout: "warning" 2024-05-27T07:36:52.736 INFO:teuthology.orchestra.run.smithi052.stdout: ], 2024-05-27T07:36:52.736 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-05-27T07:36:52.736 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-05-27T07:36:52.736 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-05-27T07:36:52.736 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-05-27T07:36:52.736 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-05-27T07:36:52.736 INFO:teuthology.orchestra.run.smithi052.stdout: "log_to_cluster": { 2024-05-27T07:36:52.736 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "log_to_cluster", 2024-05-27T07:36:52.736 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "bool", 2024-05-27T07:36:52.736 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-05-27T07:36:52.736 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 1, 2024-05-27T07:36:52.736 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "False", 2024-05-27T07:36:52.736 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-05-27T07:36:52.737 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-05-27T07:36:52.737 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-05-27T07:36:52.737 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-05-27T07:36:52.737 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-05-27T07:36:52.737 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-05-27T07:36:52.737 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-05-27T07:36:52.737 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-05-27T07:36:52.737 INFO:teuthology.orchestra.run.smithi052.stdout: "log_to_cluster_level": { 2024-05-27T07:36:52.737 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "log_to_cluster_level", 2024-05-27T07:36:52.737 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "str", 2024-05-27T07:36:52.737 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-05-27T07:36:52.737 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 1, 2024-05-27T07:36:52.737 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "info", 2024-05-27T07:36:52.738 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-05-27T07:36:52.738 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-05-27T07:36:52.738 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [ 2024-05-27T07:36:52.738 INFO:teuthology.orchestra.run.smithi052.stdout: "", 2024-05-27T07:36:52.738 INFO:teuthology.orchestra.run.smithi052.stdout: "critical", 2024-05-27T07:36:52.738 INFO:teuthology.orchestra.run.smithi052.stdout: "debug", 2024-05-27T07:36:52.738 INFO:teuthology.orchestra.run.smithi052.stdout: "error", 2024-05-27T07:36:52.738 INFO:teuthology.orchestra.run.smithi052.stdout: "info", 2024-05-27T07:36:52.738 INFO:teuthology.orchestra.run.smithi052.stdout: "warning" 2024-05-27T07:36:52.738 INFO:teuthology.orchestra.run.smithi052.stdout: ], 2024-05-27T07:36:52.738 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-05-27T07:36:52.738 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-05-27T07:36:52.739 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-05-27T07:36:52.739 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-05-27T07:36:52.739 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-05-27T07:36:52.739 INFO:teuthology.orchestra.run.smithi052.stdout: "log_to_file": { 2024-05-27T07:36:52.739 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "log_to_file", 2024-05-27T07:36:52.739 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "bool", 2024-05-27T07:36:52.739 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-05-27T07:36:52.739 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 1, 2024-05-27T07:36:52.739 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "False", 2024-05-27T07:36:52.739 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-05-27T07:36:52.739 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-05-27T07:36:52.739 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-05-27T07:36:52.739 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-05-27T07:36:52.740 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-05-27T07:36:52.740 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-05-27T07:36:52.740 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-05-27T07:36:52.740 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-05-27T07:36:52.740 INFO:teuthology.orchestra.run.smithi052.stdout: "sqlite3_killpoint": { 2024-05-27T07:36:52.740 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "sqlite3_killpoint", 2024-05-27T07:36:52.740 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "int", 2024-05-27T07:36:52.740 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "dev", 2024-05-27T07:36:52.740 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 1, 2024-05-27T07:36:52.740 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "0", 2024-05-27T07:36:52.740 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-05-27T07:36:52.740 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-05-27T07:36:52.740 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-05-27T07:36:52.741 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-05-27T07:36:52.741 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-05-27T07:36:52.741 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-05-27T07:36:52.741 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-05-27T07:36:52.741 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-05-27T07:36:52.741 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-05-27T07:36:52.741 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-05-27T07:36:52.741 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-05-27T07:36:52.741 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "status", 2024-05-27T07:36:52.741 INFO:teuthology.orchestra.run.smithi052.stdout: "can_run": true, 2024-05-27T07:36:52.741 INFO:teuthology.orchestra.run.smithi052.stdout: "error_string": "", 2024-05-27T07:36:52.741 INFO:teuthology.orchestra.run.smithi052.stdout: "module_options": { 2024-05-27T07:36:52.741 INFO:teuthology.orchestra.run.smithi052.stdout: "log_level": { 2024-05-27T07:36:52.741 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "log_level", 2024-05-27T07:36:52.742 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "str", 2024-05-27T07:36:52.742 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-05-27T07:36:52.742 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 1, 2024-05-27T07:36:52.742 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "", 2024-05-27T07:36:52.742 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-05-27T07:36:52.742 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-05-27T07:36:52.742 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [ 2024-05-27T07:36:52.742 INFO:teuthology.orchestra.run.smithi052.stdout: "", 2024-05-27T07:36:52.742 INFO:teuthology.orchestra.run.smithi052.stdout: "critical", 2024-05-27T07:36:52.742 INFO:teuthology.orchestra.run.smithi052.stdout: "debug", 2024-05-27T07:36:52.742 INFO:teuthology.orchestra.run.smithi052.stdout: "error", 2024-05-27T07:36:52.742 INFO:teuthology.orchestra.run.smithi052.stdout: "info", 2024-05-27T07:36:52.742 INFO:teuthology.orchestra.run.smithi052.stdout: "warning" 2024-05-27T07:36:52.743 INFO:teuthology.orchestra.run.smithi052.stdout: ], 2024-05-27T07:36:52.743 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-05-27T07:36:52.743 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-05-27T07:36:52.743 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-05-27T07:36:52.743 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-05-27T07:36:52.743 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-05-27T07:36:52.743 INFO:teuthology.orchestra.run.smithi052.stdout: "log_to_cluster": { 2024-05-27T07:36:52.743 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "log_to_cluster", 2024-05-27T07:36:52.743 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "bool", 2024-05-27T07:36:52.743 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-05-27T07:36:52.743 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 1, 2024-05-27T07:36:52.743 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "False", 2024-05-27T07:36:52.743 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-05-27T07:36:52.744 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-05-27T07:36:52.744 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-05-27T07:36:52.744 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-05-27T07:36:52.744 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-05-27T07:36:52.744 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-05-27T07:36:52.744 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-05-27T07:36:52.744 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-05-27T07:36:52.744 INFO:teuthology.orchestra.run.smithi052.stdout: "log_to_cluster_level": { 2024-05-27T07:36:52.744 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "log_to_cluster_level", 2024-05-27T07:36:52.744 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "str", 2024-05-27T07:36:52.744 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-05-27T07:36:52.744 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 1, 2024-05-27T07:36:52.744 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "info", 2024-05-27T07:36:52.745 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-05-27T07:36:52.745 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-05-27T07:36:52.745 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [ 2024-05-27T07:36:52.745 INFO:teuthology.orchestra.run.smithi052.stdout: "", 2024-05-27T07:36:52.745 INFO:teuthology.orchestra.run.smithi052.stdout: "critical", 2024-05-27T07:36:52.745 INFO:teuthology.orchestra.run.smithi052.stdout: "debug", 2024-05-27T07:36:52.745 INFO:teuthology.orchestra.run.smithi052.stdout: "error", 2024-05-27T07:36:52.745 INFO:teuthology.orchestra.run.smithi052.stdout: "info", 2024-05-27T07:36:52.745 INFO:teuthology.orchestra.run.smithi052.stdout: "warning" 2024-05-27T07:36:52.745 INFO:teuthology.orchestra.run.smithi052.stdout: ], 2024-05-27T07:36:52.745 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-05-27T07:36:52.745 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-05-27T07:36:52.745 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-05-27T07:36:52.746 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-05-27T07:36:52.746 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-05-27T07:36:52.746 INFO:teuthology.orchestra.run.smithi052.stdout: "log_to_file": { 2024-05-27T07:36:52.746 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "log_to_file", 2024-05-27T07:36:52.746 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "bool", 2024-05-27T07:36:52.746 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-05-27T07:36:52.746 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 1, 2024-05-27T07:36:52.746 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "False", 2024-05-27T07:36:52.746 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-05-27T07:36:52.746 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-05-27T07:36:52.746 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-05-27T07:36:52.746 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-05-27T07:36:52.746 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-05-27T07:36:52.747 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-05-27T07:36:52.747 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-05-27T07:36:52.747 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-05-27T07:36:52.747 INFO:teuthology.orchestra.run.smithi052.stdout: "sqlite3_killpoint": { 2024-05-27T07:36:52.747 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "sqlite3_killpoint", 2024-05-27T07:36:52.747 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "int", 2024-05-27T07:36:52.747 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "dev", 2024-05-27T07:36:52.747 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 1, 2024-05-27T07:36:52.747 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "0", 2024-05-27T07:36:52.747 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-05-27T07:36:52.747 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-05-27T07:36:52.747 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-05-27T07:36:52.747 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-05-27T07:36:52.748 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-05-27T07:36:52.748 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-05-27T07:36:52.748 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-05-27T07:36:52.748 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-05-27T07:36:52.748 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-05-27T07:36:52.748 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-05-27T07:36:52.748 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-05-27T07:36:52.748 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "telegraf", 2024-05-27T07:36:52.748 INFO:teuthology.orchestra.run.smithi052.stdout: "can_run": true, 2024-05-27T07:36:52.748 INFO:teuthology.orchestra.run.smithi052.stdout: "error_string": "", 2024-05-27T07:36:52.748 INFO:teuthology.orchestra.run.smithi052.stdout: "module_options": { 2024-05-27T07:36:52.748 INFO:teuthology.orchestra.run.smithi052.stdout: "address": { 2024-05-27T07:36:52.748 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "address", 2024-05-27T07:36:52.749 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "str", 2024-05-27T07:36:52.749 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-05-27T07:36:52.749 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 0, 2024-05-27T07:36:52.749 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "unixgram:///tmp/telegraf.sock", 2024-05-27T07:36:52.749 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-05-27T07:36:52.749 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-05-27T07:36:52.749 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-05-27T07:36:52.749 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-05-27T07:36:52.749 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-05-27T07:36:52.749 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-05-27T07:36:52.749 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-05-27T07:36:52.749 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-05-27T07:36:52.749 INFO:teuthology.orchestra.run.smithi052.stdout: "interval": { 2024-05-27T07:36:52.750 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "interval", 2024-05-27T07:36:52.750 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "secs", 2024-05-27T07:36:52.750 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-05-27T07:36:52.750 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 0, 2024-05-27T07:36:52.750 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "15", 2024-05-27T07:36:52.750 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-05-27T07:36:52.750 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-05-27T07:36:52.750 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-05-27T07:36:52.750 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-05-27T07:36:52.750 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-05-27T07:36:52.750 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-05-27T07:36:52.750 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-05-27T07:36:52.751 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-05-27T07:36:52.751 INFO:teuthology.orchestra.run.smithi052.stdout: "log_level": { 2024-05-27T07:36:52.751 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "log_level", 2024-05-27T07:36:52.751 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "str", 2024-05-27T07:36:52.751 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-05-27T07:36:52.751 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 1, 2024-05-27T07:36:52.751 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "", 2024-05-27T07:36:52.751 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-05-27T07:36:52.751 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-05-27T07:36:52.751 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [ 2024-05-27T07:36:52.751 INFO:teuthology.orchestra.run.smithi052.stdout: "", 2024-05-27T07:36:52.751 INFO:teuthology.orchestra.run.smithi052.stdout: "critical", 2024-05-27T07:36:52.751 INFO:teuthology.orchestra.run.smithi052.stdout: "debug", 2024-05-27T07:36:52.752 INFO:teuthology.orchestra.run.smithi052.stdout: "error", 2024-05-27T07:36:52.752 INFO:teuthology.orchestra.run.smithi052.stdout: "info", 2024-05-27T07:36:52.752 INFO:teuthology.orchestra.run.smithi052.stdout: "warning" 2024-05-27T07:36:52.752 INFO:teuthology.orchestra.run.smithi052.stdout: ], 2024-05-27T07:36:52.752 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-05-27T07:36:52.752 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-05-27T07:36:52.752 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-05-27T07:36:52.752 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-05-27T07:36:52.752 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-05-27T07:36:52.752 INFO:teuthology.orchestra.run.smithi052.stdout: "log_to_cluster": { 2024-05-27T07:36:52.752 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "log_to_cluster", 2024-05-27T07:36:52.752 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "bool", 2024-05-27T07:36:52.752 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-05-27T07:36:52.753 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 1, 2024-05-27T07:36:52.753 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "False", 2024-05-27T07:36:52.753 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-05-27T07:36:52.753 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-05-27T07:36:52.753 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-05-27T07:36:52.753 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-05-27T07:36:52.753 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-05-27T07:36:52.753 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-05-27T07:36:52.753 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-05-27T07:36:52.753 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-05-27T07:36:52.753 INFO:teuthology.orchestra.run.smithi052.stdout: "log_to_cluster_level": { 2024-05-27T07:36:52.753 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "log_to_cluster_level", 2024-05-27T07:36:52.753 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "str", 2024-05-27T07:36:52.754 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-05-27T07:36:52.754 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 1, 2024-05-27T07:36:52.754 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "info", 2024-05-27T07:36:52.754 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-05-27T07:36:52.754 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-05-27T07:36:52.754 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [ 2024-05-27T07:36:52.754 INFO:teuthology.orchestra.run.smithi052.stdout: "", 2024-05-27T07:36:52.754 INFO:teuthology.orchestra.run.smithi052.stdout: "critical", 2024-05-27T07:36:52.754 INFO:teuthology.orchestra.run.smithi052.stdout: "debug", 2024-05-27T07:36:52.754 INFO:teuthology.orchestra.run.smithi052.stdout: "error", 2024-05-27T07:36:52.754 INFO:teuthology.orchestra.run.smithi052.stdout: "info", 2024-05-27T07:36:52.754 INFO:teuthology.orchestra.run.smithi052.stdout: "warning" 2024-05-27T07:36:52.755 INFO:teuthology.orchestra.run.smithi052.stdout: ], 2024-05-27T07:36:52.755 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-05-27T07:36:52.755 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-05-27T07:36:52.755 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-05-27T07:36:52.755 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-05-27T07:36:52.755 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-05-27T07:36:52.755 INFO:teuthology.orchestra.run.smithi052.stdout: "log_to_file": { 2024-05-27T07:36:52.755 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "log_to_file", 2024-05-27T07:36:52.755 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "bool", 2024-05-27T07:36:52.755 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-05-27T07:36:52.755 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 1, 2024-05-27T07:36:52.755 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "False", 2024-05-27T07:36:52.755 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-05-27T07:36:52.755 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-05-27T07:36:52.756 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-05-27T07:36:52.756 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-05-27T07:36:52.756 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-05-27T07:36:52.756 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-05-27T07:36:52.756 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-05-27T07:36:52.756 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-05-27T07:36:52.756 INFO:teuthology.orchestra.run.smithi052.stdout: "sqlite3_killpoint": { 2024-05-27T07:36:52.756 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "sqlite3_killpoint", 2024-05-27T07:36:52.756 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "int", 2024-05-27T07:36:52.756 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "dev", 2024-05-27T07:36:52.756 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 1, 2024-05-27T07:36:52.756 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "0", 2024-05-27T07:36:52.756 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-05-27T07:36:52.757 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-05-27T07:36:52.757 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-05-27T07:36:52.757 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-05-27T07:36:52.757 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-05-27T07:36:52.757 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-05-27T07:36:52.757 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-05-27T07:36:52.757 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-05-27T07:36:52.757 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-05-27T07:36:52.757 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-05-27T07:36:52.757 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-05-27T07:36:52.757 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "telemetry", 2024-05-27T07:36:52.757 INFO:teuthology.orchestra.run.smithi052.stdout: "can_run": true, 2024-05-27T07:36:52.757 INFO:teuthology.orchestra.run.smithi052.stdout: "error_string": "", 2024-05-27T07:36:52.758 INFO:teuthology.orchestra.run.smithi052.stdout: "module_options": { 2024-05-27T07:36:52.758 INFO:teuthology.orchestra.run.smithi052.stdout: "channel_basic": { 2024-05-27T07:36:52.758 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "channel_basic", 2024-05-27T07:36:52.758 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "bool", 2024-05-27T07:36:52.758 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-05-27T07:36:52.758 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 0, 2024-05-27T07:36:52.758 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "True", 2024-05-27T07:36:52.758 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-05-27T07:36:52.758 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-05-27T07:36:52.758 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-05-27T07:36:52.758 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "Share basic cluster information (size, version)", 2024-05-27T07:36:52.758 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-05-27T07:36:52.759 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-05-27T07:36:52.759 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-05-27T07:36:52.759 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-05-27T07:36:52.759 INFO:teuthology.orchestra.run.smithi052.stdout: "channel_crash": { 2024-05-27T07:36:52.759 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "channel_crash", 2024-05-27T07:36:52.759 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "bool", 2024-05-27T07:36:52.759 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-05-27T07:36:52.759 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 0, 2024-05-27T07:36:52.759 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "True", 2024-05-27T07:36:52.759 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-05-27T07:36:52.759 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-05-27T07:36:52.759 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-05-27T07:36:52.759 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "Share metadata about Ceph daemon crashes (version, stack straces, etc)", 2024-05-27T07:36:52.760 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-05-27T07:36:52.760 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-05-27T07:36:52.760 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-05-27T07:36:52.760 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-05-27T07:36:52.760 INFO:teuthology.orchestra.run.smithi052.stdout: "channel_device": { 2024-05-27T07:36:52.760 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "channel_device", 2024-05-27T07:36:52.760 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "bool", 2024-05-27T07:36:52.760 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-05-27T07:36:52.760 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 0, 2024-05-27T07:36:52.760 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "True", 2024-05-27T07:36:52.760 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-05-27T07:36:52.760 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-05-27T07:36:52.760 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-05-27T07:36:52.760 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)", 2024-05-27T07:36:52.761 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-05-27T07:36:52.761 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-05-27T07:36:52.761 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-05-27T07:36:52.761 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-05-27T07:36:52.761 INFO:teuthology.orchestra.run.smithi052.stdout: "channel_ident": { 2024-05-27T07:36:52.761 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "channel_ident", 2024-05-27T07:36:52.761 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "bool", 2024-05-27T07:36:52.761 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-05-27T07:36:52.761 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 0, 2024-05-27T07:36:52.761 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "False", 2024-05-27T07:36:52.761 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-05-27T07:36:52.761 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-05-27T07:36:52.761 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-05-27T07:36:52.762 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "Share a user-provided description and/or contact email for the cluster", 2024-05-27T07:36:52.762 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-05-27T07:36:52.762 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-05-27T07:36:52.762 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-05-27T07:36:52.762 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-05-27T07:36:52.762 INFO:teuthology.orchestra.run.smithi052.stdout: "channel_perf": { 2024-05-27T07:36:52.762 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "channel_perf", 2024-05-27T07:36:52.762 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "bool", 2024-05-27T07:36:52.762 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-05-27T07:36:52.762 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 0, 2024-05-27T07:36:52.762 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "False", 2024-05-27T07:36:52.762 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-05-27T07:36:52.763 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-05-27T07:36:52.763 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-05-27T07:36:52.763 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "Share various performance metrics of a cluster", 2024-05-27T07:36:52.763 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-05-27T07:36:52.763 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-05-27T07:36:52.763 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-05-27T07:36:52.763 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-05-27T07:36:52.763 INFO:teuthology.orchestra.run.smithi052.stdout: "contact": { 2024-05-27T07:36:52.763 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "contact", 2024-05-27T07:36:52.763 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "str", 2024-05-27T07:36:52.763 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-05-27T07:36:52.763 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 0, 2024-05-27T07:36:52.763 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "", 2024-05-27T07:36:52.764 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-05-27T07:36:52.764 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-05-27T07:36:52.764 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-05-27T07:36:52.764 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-05-27T07:36:52.764 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-05-27T07:36:52.764 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-05-27T07:36:52.764 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-05-27T07:36:52.764 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-05-27T07:36:52.764 INFO:teuthology.orchestra.run.smithi052.stdout: "description": { 2024-05-27T07:36:52.764 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "description", 2024-05-27T07:36:52.764 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "str", 2024-05-27T07:36:52.764 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-05-27T07:36:52.764 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 0, 2024-05-27T07:36:52.765 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "", 2024-05-27T07:36:52.765 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-05-27T07:36:52.765 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-05-27T07:36:52.765 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-05-27T07:36:52.765 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-05-27T07:36:52.765 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-05-27T07:36:52.765 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-05-27T07:36:52.765 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-05-27T07:36:52.765 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-05-27T07:36:52.765 INFO:teuthology.orchestra.run.smithi052.stdout: "device_url": { 2024-05-27T07:36:52.765 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "device_url", 2024-05-27T07:36:52.765 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "str", 2024-05-27T07:36:52.765 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-05-27T07:36:52.766 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 0, 2024-05-27T07:36:52.766 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "https://telemetry.ceph.com/device", 2024-05-27T07:36:52.766 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-05-27T07:36:52.766 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-05-27T07:36:52.766 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-05-27T07:36:52.766 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-05-27T07:36:52.766 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-05-27T07:36:52.766 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-05-27T07:36:52.766 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-05-27T07:36:52.766 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-05-27T07:36:52.766 INFO:teuthology.orchestra.run.smithi052.stdout: "enabled": { 2024-05-27T07:36:52.766 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "enabled", 2024-05-27T07:36:52.766 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "bool", 2024-05-27T07:36:52.767 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-05-27T07:36:52.767 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 0, 2024-05-27T07:36:52.767 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "False", 2024-05-27T07:36:52.767 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-05-27T07:36:52.767 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-05-27T07:36:52.767 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-05-27T07:36:52.767 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-05-27T07:36:52.767 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-05-27T07:36:52.767 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-05-27T07:36:52.767 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-05-27T07:36:52.767 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-05-27T07:36:52.767 INFO:teuthology.orchestra.run.smithi052.stdout: "interval": { 2024-05-27T07:36:52.768 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "interval", 2024-05-27T07:36:52.768 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "int", 2024-05-27T07:36:52.768 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-05-27T07:36:52.768 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 0, 2024-05-27T07:36:52.768 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "24", 2024-05-27T07:36:52.768 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "8", 2024-05-27T07:36:52.768 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-05-27T07:36:52.768 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-05-27T07:36:52.768 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-05-27T07:36:52.768 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-05-27T07:36:52.768 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-05-27T07:36:52.768 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-05-27T07:36:52.768 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-05-27T07:36:52.769 INFO:teuthology.orchestra.run.smithi052.stdout: "last_opt_revision": { 2024-05-27T07:36:52.769 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "last_opt_revision", 2024-05-27T07:36:52.769 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "int", 2024-05-27T07:36:52.769 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-05-27T07:36:52.769 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 0, 2024-05-27T07:36:52.769 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "1", 2024-05-27T07:36:52.769 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-05-27T07:36:52.769 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-05-27T07:36:52.769 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-05-27T07:36:52.769 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-05-27T07:36:52.769 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-05-27T07:36:52.769 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-05-27T07:36:52.769 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-05-27T07:36:52.770 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-05-27T07:36:52.770 INFO:teuthology.orchestra.run.smithi052.stdout: "leaderboard": { 2024-05-27T07:36:52.770 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "leaderboard", 2024-05-27T07:36:52.770 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "bool", 2024-05-27T07:36:52.770 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-05-27T07:36:52.770 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 0, 2024-05-27T07:36:52.770 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "False", 2024-05-27T07:36:52.770 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-05-27T07:36:52.770 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-05-27T07:36:52.770 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-05-27T07:36:52.770 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-05-27T07:36:52.770 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-05-27T07:36:52.770 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-05-27T07:36:52.771 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-05-27T07:36:52.771 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-05-27T07:36:52.771 INFO:teuthology.orchestra.run.smithi052.stdout: "leaderboard_description": { 2024-05-27T07:36:52.771 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "leaderboard_description", 2024-05-27T07:36:52.771 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "str", 2024-05-27T07:36:52.771 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-05-27T07:36:52.771 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 0, 2024-05-27T07:36:52.771 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "", 2024-05-27T07:36:52.771 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-05-27T07:36:52.771 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-05-27T07:36:52.771 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-05-27T07:36:52.771 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-05-27T07:36:52.772 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-05-27T07:36:52.772 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-05-27T07:36:52.772 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-05-27T07:36:52.772 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-05-27T07:36:52.772 INFO:teuthology.orchestra.run.smithi052.stdout: "log_level": { 2024-05-27T07:36:52.772 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "log_level", 2024-05-27T07:36:52.772 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "str", 2024-05-27T07:36:52.772 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-05-27T07:36:52.772 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 1, 2024-05-27T07:36:52.772 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "", 2024-05-27T07:36:52.772 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-05-27T07:36:52.772 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-05-27T07:36:52.772 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [ 2024-05-27T07:36:52.772 INFO:teuthology.orchestra.run.smithi052.stdout: "", 2024-05-27T07:36:52.773 INFO:teuthology.orchestra.run.smithi052.stdout: "critical", 2024-05-27T07:36:52.773 INFO:teuthology.orchestra.run.smithi052.stdout: "debug", 2024-05-27T07:36:52.773 INFO:teuthology.orchestra.run.smithi052.stdout: "error", 2024-05-27T07:36:52.773 INFO:teuthology.orchestra.run.smithi052.stdout: "info", 2024-05-27T07:36:52.773 INFO:teuthology.orchestra.run.smithi052.stdout: "warning" 2024-05-27T07:36:52.773 INFO:teuthology.orchestra.run.smithi052.stdout: ], 2024-05-27T07:36:52.773 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-05-27T07:36:52.773 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-05-27T07:36:52.773 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-05-27T07:36:52.773 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-05-27T07:36:52.773 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-05-27T07:36:52.773 INFO:teuthology.orchestra.run.smithi052.stdout: "log_to_cluster": { 2024-05-27T07:36:52.773 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "log_to_cluster", 2024-05-27T07:36:52.774 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "bool", 2024-05-27T07:36:52.774 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-05-27T07:36:52.774 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 1, 2024-05-27T07:36:52.774 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "False", 2024-05-27T07:36:52.774 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-05-27T07:36:52.774 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-05-27T07:36:52.774 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-05-27T07:36:52.774 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-05-27T07:36:52.774 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-05-27T07:36:52.774 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-05-27T07:36:52.774 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-05-27T07:36:52.774 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-05-27T07:36:52.774 INFO:teuthology.orchestra.run.smithi052.stdout: "log_to_cluster_level": { 2024-05-27T07:36:52.775 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "log_to_cluster_level", 2024-05-27T07:36:52.775 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "str", 2024-05-27T07:36:52.775 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-05-27T07:36:52.775 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 1, 2024-05-27T07:36:52.775 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "info", 2024-05-27T07:36:52.775 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-05-27T07:36:52.775 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-05-27T07:36:52.775 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [ 2024-05-27T07:36:52.775 INFO:teuthology.orchestra.run.smithi052.stdout: "", 2024-05-27T07:36:52.775 INFO:teuthology.orchestra.run.smithi052.stdout: "critical", 2024-05-27T07:36:52.775 INFO:teuthology.orchestra.run.smithi052.stdout: "debug", 2024-05-27T07:36:52.775 INFO:teuthology.orchestra.run.smithi052.stdout: "error", 2024-05-27T07:36:52.775 INFO:teuthology.orchestra.run.smithi052.stdout: "info", 2024-05-27T07:36:52.776 INFO:teuthology.orchestra.run.smithi052.stdout: "warning" 2024-05-27T07:36:52.776 INFO:teuthology.orchestra.run.smithi052.stdout: ], 2024-05-27T07:36:52.776 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-05-27T07:36:52.776 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-05-27T07:36:52.776 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-05-27T07:36:52.776 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-05-27T07:36:52.776 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-05-27T07:36:52.776 INFO:teuthology.orchestra.run.smithi052.stdout: "log_to_file": { 2024-05-27T07:36:52.776 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "log_to_file", 2024-05-27T07:36:52.776 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "bool", 2024-05-27T07:36:52.776 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-05-27T07:36:52.776 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 1, 2024-05-27T07:36:52.776 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "False", 2024-05-27T07:36:52.777 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-05-27T07:36:52.777 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-05-27T07:36:52.777 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-05-27T07:36:52.777 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-05-27T07:36:52.777 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-05-27T07:36:52.777 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-05-27T07:36:52.777 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-05-27T07:36:52.777 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-05-27T07:36:52.777 INFO:teuthology.orchestra.run.smithi052.stdout: "organization": { 2024-05-27T07:36:52.777 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "organization", 2024-05-27T07:36:52.777 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "str", 2024-05-27T07:36:52.777 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-05-27T07:36:52.777 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 0, 2024-05-27T07:36:52.778 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "", 2024-05-27T07:36:52.778 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-05-27T07:36:52.778 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-05-27T07:36:52.778 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-05-27T07:36:52.778 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-05-27T07:36:52.778 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-05-27T07:36:52.778 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-05-27T07:36:52.778 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-05-27T07:36:52.778 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-05-27T07:36:52.778 INFO:teuthology.orchestra.run.smithi052.stdout: "proxy": { 2024-05-27T07:36:52.778 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "proxy", 2024-05-27T07:36:52.778 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "str", 2024-05-27T07:36:52.779 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-05-27T07:36:52.779 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 0, 2024-05-27T07:36:52.779 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "", 2024-05-27T07:36:52.779 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-05-27T07:36:52.779 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-05-27T07:36:52.779 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-05-27T07:36:52.779 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-05-27T07:36:52.779 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-05-27T07:36:52.779 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-05-27T07:36:52.779 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-05-27T07:36:52.779 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-05-27T07:36:52.779 INFO:teuthology.orchestra.run.smithi052.stdout: "sqlite3_killpoint": { 2024-05-27T07:36:52.779 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "sqlite3_killpoint", 2024-05-27T07:36:52.780 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "int", 2024-05-27T07:36:52.780 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "dev", 2024-05-27T07:36:52.780 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 1, 2024-05-27T07:36:52.780 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "0", 2024-05-27T07:36:52.780 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-05-27T07:36:52.780 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-05-27T07:36:52.780 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-05-27T07:36:52.780 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-05-27T07:36:52.780 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-05-27T07:36:52.780 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-05-27T07:36:52.780 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-05-27T07:36:52.780 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-05-27T07:36:52.780 INFO:teuthology.orchestra.run.smithi052.stdout: "url": { 2024-05-27T07:36:52.781 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "url", 2024-05-27T07:36:52.781 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "str", 2024-05-27T07:36:52.781 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-05-27T07:36:52.781 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 0, 2024-05-27T07:36:52.781 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "https://telemetry.ceph.com/report", 2024-05-27T07:36:52.781 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-05-27T07:36:52.781 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-05-27T07:36:52.781 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-05-27T07:36:52.781 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-05-27T07:36:52.781 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-05-27T07:36:52.781 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-05-27T07:36:52.781 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-05-27T07:36:52.781 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-05-27T07:36:52.782 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-05-27T07:36:52.782 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-05-27T07:36:52.782 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-05-27T07:36:52.782 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "test_orchestrator", 2024-05-27T07:36:52.782 INFO:teuthology.orchestra.run.smithi052.stdout: "can_run": true, 2024-05-27T07:36:52.782 INFO:teuthology.orchestra.run.smithi052.stdout: "error_string": "", 2024-05-27T07:36:52.782 INFO:teuthology.orchestra.run.smithi052.stdout: "module_options": { 2024-05-27T07:36:52.782 INFO:teuthology.orchestra.run.smithi052.stdout: "log_level": { 2024-05-27T07:36:52.782 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "log_level", 2024-05-27T07:36:52.782 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "str", 2024-05-27T07:36:52.782 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-05-27T07:36:52.782 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 1, 2024-05-27T07:36:52.782 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "", 2024-05-27T07:36:52.783 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-05-27T07:36:52.783 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-05-27T07:36:52.783 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [ 2024-05-27T07:36:52.783 INFO:teuthology.orchestra.run.smithi052.stdout: "", 2024-05-27T07:36:52.783 INFO:teuthology.orchestra.run.smithi052.stdout: "critical", 2024-05-27T07:36:52.783 INFO:teuthology.orchestra.run.smithi052.stdout: "debug", 2024-05-27T07:36:52.783 INFO:teuthology.orchestra.run.smithi052.stdout: "error", 2024-05-27T07:36:52.783 INFO:teuthology.orchestra.run.smithi052.stdout: "info", 2024-05-27T07:36:52.783 INFO:teuthology.orchestra.run.smithi052.stdout: "warning" 2024-05-27T07:36:52.783 INFO:teuthology.orchestra.run.smithi052.stdout: ], 2024-05-27T07:36:52.783 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-05-27T07:36:52.783 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-05-27T07:36:52.783 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-05-27T07:36:52.784 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-05-27T07:36:52.784 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-05-27T07:36:52.784 INFO:teuthology.orchestra.run.smithi052.stdout: "log_to_cluster": { 2024-05-27T07:36:52.784 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "log_to_cluster", 2024-05-27T07:36:52.784 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "bool", 2024-05-27T07:36:52.784 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-05-27T07:36:52.784 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 1, 2024-05-27T07:36:52.784 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "False", 2024-05-27T07:36:52.784 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-05-27T07:36:52.784 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-05-27T07:36:52.784 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-05-27T07:36:52.784 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-05-27T07:36:52.784 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-05-27T07:36:52.785 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-05-27T07:36:52.785 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-05-27T07:36:52.785 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-05-27T07:36:52.785 INFO:teuthology.orchestra.run.smithi052.stdout: "log_to_cluster_level": { 2024-05-27T07:36:52.785 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "log_to_cluster_level", 2024-05-27T07:36:52.785 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "str", 2024-05-27T07:36:52.785 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-05-27T07:36:52.785 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 1, 2024-05-27T07:36:52.785 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "info", 2024-05-27T07:36:52.785 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-05-27T07:36:52.785 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-05-27T07:36:52.785 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [ 2024-05-27T07:36:52.785 INFO:teuthology.orchestra.run.smithi052.stdout: "", 2024-05-27T07:36:52.786 INFO:teuthology.orchestra.run.smithi052.stdout: "critical", 2024-05-27T07:36:52.786 INFO:teuthology.orchestra.run.smithi052.stdout: "debug", 2024-05-27T07:36:52.786 INFO:teuthology.orchestra.run.smithi052.stdout: "error", 2024-05-27T07:36:52.786 INFO:teuthology.orchestra.run.smithi052.stdout: "info", 2024-05-27T07:36:52.786 INFO:teuthology.orchestra.run.smithi052.stdout: "warning" 2024-05-27T07:36:52.786 INFO:teuthology.orchestra.run.smithi052.stdout: ], 2024-05-27T07:36:52.786 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-05-27T07:36:52.786 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-05-27T07:36:52.786 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-05-27T07:36:52.786 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-05-27T07:36:52.786 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-05-27T07:36:52.786 INFO:teuthology.orchestra.run.smithi052.stdout: "log_to_file": { 2024-05-27T07:36:52.787 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "log_to_file", 2024-05-27T07:36:52.787 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "bool", 2024-05-27T07:36:52.787 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-05-27T07:36:52.787 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 1, 2024-05-27T07:36:52.787 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "False", 2024-05-27T07:36:52.787 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-05-27T07:36:52.787 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-05-27T07:36:52.787 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-05-27T07:36:52.787 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-05-27T07:36:52.787 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-05-27T07:36:52.787 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-05-27T07:36:52.787 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-05-27T07:36:52.787 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-05-27T07:36:52.787 INFO:teuthology.orchestra.run.smithi052.stdout: "sqlite3_killpoint": { 2024-05-27T07:36:52.788 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "sqlite3_killpoint", 2024-05-27T07:36:52.788 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "int", 2024-05-27T07:36:52.788 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "dev", 2024-05-27T07:36:52.788 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 1, 2024-05-27T07:36:52.788 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "0", 2024-05-27T07:36:52.788 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-05-27T07:36:52.788 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-05-27T07:36:52.788 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-05-27T07:36:52.788 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-05-27T07:36:52.788 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-05-27T07:36:52.788 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-05-27T07:36:52.788 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-05-27T07:36:52.789 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-05-27T07:36:52.789 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-05-27T07:36:52.789 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-05-27T07:36:52.789 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-05-27T07:36:52.789 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "volumes", 2024-05-27T07:36:52.789 INFO:teuthology.orchestra.run.smithi052.stdout: "can_run": true, 2024-05-27T07:36:52.789 INFO:teuthology.orchestra.run.smithi052.stdout: "error_string": "", 2024-05-27T07:36:52.789 INFO:teuthology.orchestra.run.smithi052.stdout: "module_options": { 2024-05-27T07:36:52.789 INFO:teuthology.orchestra.run.smithi052.stdout: "log_level": { 2024-05-27T07:36:52.789 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "log_level", 2024-05-27T07:36:52.789 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "str", 2024-05-27T07:36:52.789 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-05-27T07:36:52.789 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 1, 2024-05-27T07:36:52.790 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "", 2024-05-27T07:36:52.790 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-05-27T07:36:52.790 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-05-27T07:36:52.790 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [ 2024-05-27T07:36:52.790 INFO:teuthology.orchestra.run.smithi052.stdout: "", 2024-05-27T07:36:52.790 INFO:teuthology.orchestra.run.smithi052.stdout: "critical", 2024-05-27T07:36:52.790 INFO:teuthology.orchestra.run.smithi052.stdout: "debug", 2024-05-27T07:36:52.790 INFO:teuthology.orchestra.run.smithi052.stdout: "error", 2024-05-27T07:36:52.790 INFO:teuthology.orchestra.run.smithi052.stdout: "info", 2024-05-27T07:36:52.790 INFO:teuthology.orchestra.run.smithi052.stdout: "warning" 2024-05-27T07:36:52.790 INFO:teuthology.orchestra.run.smithi052.stdout: ], 2024-05-27T07:36:52.790 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-05-27T07:36:52.790 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-05-27T07:36:52.791 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-05-27T07:36:52.791 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-05-27T07:36:52.791 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-05-27T07:36:52.791 INFO:teuthology.orchestra.run.smithi052.stdout: "log_to_cluster": { 2024-05-27T07:36:52.791 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "log_to_cluster", 2024-05-27T07:36:52.791 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "bool", 2024-05-27T07:36:52.791 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-05-27T07:36:52.791 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 1, 2024-05-27T07:36:52.791 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "False", 2024-05-27T07:36:52.791 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-05-27T07:36:52.791 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-05-27T07:36:52.791 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-05-27T07:36:52.791 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-05-27T07:36:52.792 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-05-27T07:36:52.792 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-05-27T07:36:52.792 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-05-27T07:36:52.792 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-05-27T07:36:52.792 INFO:teuthology.orchestra.run.smithi052.stdout: "log_to_cluster_level": { 2024-05-27T07:36:52.792 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "log_to_cluster_level", 2024-05-27T07:36:52.792 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "str", 2024-05-27T07:36:52.792 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-05-27T07:36:52.792 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 1, 2024-05-27T07:36:52.792 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "info", 2024-05-27T07:36:52.792 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-05-27T07:36:52.792 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-05-27T07:36:52.792 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [ 2024-05-27T07:36:52.793 INFO:teuthology.orchestra.run.smithi052.stdout: "", 2024-05-27T07:36:52.793 INFO:teuthology.orchestra.run.smithi052.stdout: "critical", 2024-05-27T07:36:52.793 INFO:teuthology.orchestra.run.smithi052.stdout: "debug", 2024-05-27T07:36:52.793 INFO:teuthology.orchestra.run.smithi052.stdout: "error", 2024-05-27T07:36:52.793 INFO:teuthology.orchestra.run.smithi052.stdout: "info", 2024-05-27T07:36:52.793 INFO:teuthology.orchestra.run.smithi052.stdout: "warning" 2024-05-27T07:36:52.793 INFO:teuthology.orchestra.run.smithi052.stdout: ], 2024-05-27T07:36:52.793 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-05-27T07:36:52.793 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-05-27T07:36:52.793 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-05-27T07:36:52.793 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-05-27T07:36:52.793 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-05-27T07:36:52.794 INFO:teuthology.orchestra.run.smithi052.stdout: "log_to_file": { 2024-05-27T07:36:52.794 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "log_to_file", 2024-05-27T07:36:52.794 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "bool", 2024-05-27T07:36:52.794 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-05-27T07:36:52.794 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 1, 2024-05-27T07:36:52.794 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "False", 2024-05-27T07:36:52.794 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-05-27T07:36:52.794 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-05-27T07:36:52.794 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-05-27T07:36:52.794 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-05-27T07:36:52.794 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-05-27T07:36:52.794 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-05-27T07:36:52.794 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-05-27T07:36:52.795 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-05-27T07:36:52.795 INFO:teuthology.orchestra.run.smithi052.stdout: "max_concurrent_clones": { 2024-05-27T07:36:52.795 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "max_concurrent_clones", 2024-05-27T07:36:52.795 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "int", 2024-05-27T07:36:52.795 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-05-27T07:36:52.795 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 0, 2024-05-27T07:36:52.795 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "4", 2024-05-27T07:36:52.795 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-05-27T07:36:52.795 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-05-27T07:36:52.795 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-05-27T07:36:52.795 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "Number of asynchronous cloner threads", 2024-05-27T07:36:52.795 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-05-27T07:36:52.795 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-05-27T07:36:52.796 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-05-27T07:36:52.796 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-05-27T07:36:52.796 INFO:teuthology.orchestra.run.smithi052.stdout: "periodic_async_work": { 2024-05-27T07:36:52.796 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "periodic_async_work", 2024-05-27T07:36:52.796 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "bool", 2024-05-27T07:36:52.796 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-05-27T07:36:52.796 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 0, 2024-05-27T07:36:52.796 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "False", 2024-05-27T07:36:52.796 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-05-27T07:36:52.796 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-05-27T07:36:52.796 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-05-27T07:36:52.796 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "Periodically check for async work", 2024-05-27T07:36:52.797 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-05-27T07:36:52.797 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-05-27T07:36:52.797 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-05-27T07:36:52.797 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-05-27T07:36:52.797 INFO:teuthology.orchestra.run.smithi052.stdout: "snapshot_clone_delay": { 2024-05-27T07:36:52.797 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "snapshot_clone_delay", 2024-05-27T07:36:52.797 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "int", 2024-05-27T07:36:52.797 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-05-27T07:36:52.797 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 0, 2024-05-27T07:36:52.797 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "0", 2024-05-27T07:36:52.797 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-05-27T07:36:52.797 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-05-27T07:36:52.797 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-05-27T07:36:52.798 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "Delay clone begin operation by snapshot_clone_delay seconds", 2024-05-27T07:36:52.798 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-05-27T07:36:52.798 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-05-27T07:36:52.798 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-05-27T07:36:52.798 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-05-27T07:36:52.798 INFO:teuthology.orchestra.run.smithi052.stdout: "snapshot_clone_no_wait": { 2024-05-27T07:36:52.798 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "snapshot_clone_no_wait", 2024-05-27T07:36:52.798 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "bool", 2024-05-27T07:36:52.798 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-05-27T07:36:52.798 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 0, 2024-05-27T07:36:52.798 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "True", 2024-05-27T07:36:52.798 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-05-27T07:36:52.798 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-05-27T07:36:52.799 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-05-27T07:36:52.799 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "Reject subvolume clone request when cloner threads are busy", 2024-05-27T07:36:52.799 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-05-27T07:36:52.799 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-05-27T07:36:52.799 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-05-27T07:36:52.799 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-05-27T07:36:52.799 INFO:teuthology.orchestra.run.smithi052.stdout: "sqlite3_killpoint": { 2024-05-27T07:36:52.799 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "sqlite3_killpoint", 2024-05-27T07:36:52.799 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "int", 2024-05-27T07:36:52.799 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "dev", 2024-05-27T07:36:52.799 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 1, 2024-05-27T07:36:52.799 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "0", 2024-05-27T07:36:52.800 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-05-27T07:36:52.800 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-05-27T07:36:52.800 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-05-27T07:36:52.800 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-05-27T07:36:52.800 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-05-27T07:36:52.800 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-05-27T07:36:52.800 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-05-27T07:36:52.800 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-05-27T07:36:52.800 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-05-27T07:36:52.800 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-05-27T07:36:52.800 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-05-27T07:36:52.800 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "zabbix", 2024-05-27T07:36:52.800 INFO:teuthology.orchestra.run.smithi052.stdout: "can_run": true, 2024-05-27T07:36:52.800 INFO:teuthology.orchestra.run.smithi052.stdout: "error_string": "", 2024-05-27T07:36:52.801 INFO:teuthology.orchestra.run.smithi052.stdout: "module_options": { 2024-05-27T07:36:52.801 INFO:teuthology.orchestra.run.smithi052.stdout: "discovery_interval": { 2024-05-27T07:36:52.801 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "discovery_interval", 2024-05-27T07:36:52.801 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "uint", 2024-05-27T07:36:52.801 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-05-27T07:36:52.801 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 0, 2024-05-27T07:36:52.801 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "100", 2024-05-27T07:36:52.801 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-05-27T07:36:52.801 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-05-27T07:36:52.801 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-05-27T07:36:52.801 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-05-27T07:36:52.801 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-05-27T07:36:52.801 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-05-27T07:36:52.802 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-05-27T07:36:52.802 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-05-27T07:36:52.802 INFO:teuthology.orchestra.run.smithi052.stdout: "identifier": { 2024-05-27T07:36:52.802 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "identifier", 2024-05-27T07:36:52.802 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "str", 2024-05-27T07:36:52.802 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-05-27T07:36:52.802 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 0, 2024-05-27T07:36:52.802 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "", 2024-05-27T07:36:52.802 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-05-27T07:36:52.802 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-05-27T07:36:52.802 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-05-27T07:36:52.802 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-05-27T07:36:52.802 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-05-27T07:36:52.803 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-05-27T07:36:52.803 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-05-27T07:36:52.803 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-05-27T07:36:52.803 INFO:teuthology.orchestra.run.smithi052.stdout: "interval": { 2024-05-27T07:36:52.803 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "interval", 2024-05-27T07:36:52.803 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "secs", 2024-05-27T07:36:52.803 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-05-27T07:36:52.803 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 0, 2024-05-27T07:36:52.803 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "60", 2024-05-27T07:36:52.803 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-05-27T07:36:52.803 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-05-27T07:36:52.803 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-05-27T07:36:52.803 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-05-27T07:36:52.804 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-05-27T07:36:52.804 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-05-27T07:36:52.804 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-05-27T07:36:52.804 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-05-27T07:36:52.804 INFO:teuthology.orchestra.run.smithi052.stdout: "log_level": { 2024-05-27T07:36:52.804 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "log_level", 2024-05-27T07:36:52.804 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "str", 2024-05-27T07:36:52.804 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-05-27T07:36:52.804 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 1, 2024-05-27T07:36:52.804 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "", 2024-05-27T07:36:52.804 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-05-27T07:36:52.804 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-05-27T07:36:52.804 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [ 2024-05-27T07:36:52.805 INFO:teuthology.orchestra.run.smithi052.stdout: "", 2024-05-27T07:36:52.805 INFO:teuthology.orchestra.run.smithi052.stdout: "critical", 2024-05-27T07:36:52.805 INFO:teuthology.orchestra.run.smithi052.stdout: "debug", 2024-05-27T07:36:52.805 INFO:teuthology.orchestra.run.smithi052.stdout: "error", 2024-05-27T07:36:52.805 INFO:teuthology.orchestra.run.smithi052.stdout: "info", 2024-05-27T07:36:52.805 INFO:teuthology.orchestra.run.smithi052.stdout: "warning" 2024-05-27T07:36:52.805 INFO:teuthology.orchestra.run.smithi052.stdout: ], 2024-05-27T07:36:52.805 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-05-27T07:36:52.805 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-05-27T07:36:52.805 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-05-27T07:36:52.805 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-05-27T07:36:52.805 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-05-27T07:36:52.805 INFO:teuthology.orchestra.run.smithi052.stdout: "log_to_cluster": { 2024-05-27T07:36:52.805 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "log_to_cluster", 2024-05-27T07:36:52.806 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "bool", 2024-05-27T07:36:52.806 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-05-27T07:36:52.806 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 1, 2024-05-27T07:36:52.806 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "False", 2024-05-27T07:36:52.806 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-05-27T07:36:52.806 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-05-27T07:36:52.806 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-05-27T07:36:52.806 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-05-27T07:36:52.806 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-05-27T07:36:52.806 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-05-27T07:36:52.806 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-05-27T07:36:52.806 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-05-27T07:36:52.807 INFO:teuthology.orchestra.run.smithi052.stdout: "log_to_cluster_level": { 2024-05-27T07:36:52.807 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "log_to_cluster_level", 2024-05-27T07:36:52.807 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "str", 2024-05-27T07:36:52.807 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-05-27T07:36:52.807 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 1, 2024-05-27T07:36:52.807 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "info", 2024-05-27T07:36:52.807 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-05-27T07:36:52.807 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-05-27T07:36:52.807 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [ 2024-05-27T07:36:52.807 INFO:teuthology.orchestra.run.smithi052.stdout: "", 2024-05-27T07:36:52.807 INFO:teuthology.orchestra.run.smithi052.stdout: "critical", 2024-05-27T07:36:52.807 INFO:teuthology.orchestra.run.smithi052.stdout: "debug", 2024-05-27T07:36:52.807 INFO:teuthology.orchestra.run.smithi052.stdout: "error", 2024-05-27T07:36:52.808 INFO:teuthology.orchestra.run.smithi052.stdout: "info", 2024-05-27T07:36:52.808 INFO:teuthology.orchestra.run.smithi052.stdout: "warning" 2024-05-27T07:36:52.808 INFO:teuthology.orchestra.run.smithi052.stdout: ], 2024-05-27T07:36:52.808 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-05-27T07:36:52.808 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-05-27T07:36:52.808 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-05-27T07:36:52.808 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-05-27T07:36:52.808 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-05-27T07:36:52.808 INFO:teuthology.orchestra.run.smithi052.stdout: "log_to_file": { 2024-05-27T07:36:52.808 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "log_to_file", 2024-05-27T07:36:52.808 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "bool", 2024-05-27T07:36:52.808 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-05-27T07:36:52.808 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 1, 2024-05-27T07:36:52.809 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "False", 2024-05-27T07:36:52.809 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-05-27T07:36:52.809 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-05-27T07:36:52.809 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-05-27T07:36:52.809 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-05-27T07:36:52.809 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-05-27T07:36:52.809 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-05-27T07:36:52.809 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-05-27T07:36:52.809 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-05-27T07:36:52.809 INFO:teuthology.orchestra.run.smithi052.stdout: "sqlite3_killpoint": { 2024-05-27T07:36:52.809 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "sqlite3_killpoint", 2024-05-27T07:36:52.809 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "int", 2024-05-27T07:36:52.810 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "dev", 2024-05-27T07:36:52.810 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 1, 2024-05-27T07:36:52.810 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "0", 2024-05-27T07:36:52.810 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-05-27T07:36:52.810 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-05-27T07:36:52.810 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-05-27T07:36:52.810 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-05-27T07:36:52.810 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-05-27T07:36:52.810 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-05-27T07:36:52.810 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-05-27T07:36:52.810 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-05-27T07:36:52.810 INFO:teuthology.orchestra.run.smithi052.stdout: "zabbix_host": { 2024-05-27T07:36:52.811 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "zabbix_host", 2024-05-27T07:36:52.811 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "str", 2024-05-27T07:36:52.811 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-05-27T07:36:52.811 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 0, 2024-05-27T07:36:52.811 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "", 2024-05-27T07:36:52.811 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-05-27T07:36:52.811 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-05-27T07:36:52.811 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-05-27T07:36:52.811 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-05-27T07:36:52.811 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-05-27T07:36:52.811 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-05-27T07:36:52.811 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-05-27T07:36:52.811 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-05-27T07:36:52.812 INFO:teuthology.orchestra.run.smithi052.stdout: "zabbix_port": { 2024-05-27T07:36:52.812 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "zabbix_port", 2024-05-27T07:36:52.812 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "int", 2024-05-27T07:36:52.812 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-05-27T07:36:52.812 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 0, 2024-05-27T07:36:52.812 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "10051", 2024-05-27T07:36:52.812 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-05-27T07:36:52.812 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-05-27T07:36:52.812 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-05-27T07:36:52.812 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-05-27T07:36:52.812 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-05-27T07:36:52.812 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-05-27T07:36:52.812 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-05-27T07:36:52.813 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-05-27T07:36:52.813 INFO:teuthology.orchestra.run.smithi052.stdout: "zabbix_sender": { 2024-05-27T07:36:52.813 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "zabbix_sender", 2024-05-27T07:36:52.813 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "str", 2024-05-27T07:36:52.813 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-05-27T07:36:52.813 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 0, 2024-05-27T07:36:52.813 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "/usr/bin/zabbix_sender", 2024-05-27T07:36:52.813 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-05-27T07:36:52.813 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-05-27T07:36:52.813 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-05-27T07:36:52.813 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-05-27T07:36:52.813 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-05-27T07:36:52.813 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-05-27T07:36:52.814 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-05-27T07:36:52.814 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-05-27T07:36:52.814 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-05-27T07:36:52.814 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-05-27T07:36:52.814 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2024-05-27T07:36:52.814 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-05-27T07:36:52.814 INFO:teuthology.orchestra.run.smithi052.stdout: ], 2024-05-27T07:36:52.814 INFO:teuthology.orchestra.run.smithi052.stdout: "modules": [ 2024-05-27T07:36:52.814 INFO:teuthology.orchestra.run.smithi052.stdout: "cephadm", 2024-05-27T07:36:52.814 INFO:teuthology.orchestra.run.smithi052.stdout: "dashboard", 2024-05-27T07:36:52.814 INFO:teuthology.orchestra.run.smithi052.stdout: "iostat", 2024-05-27T07:36:52.814 INFO:teuthology.orchestra.run.smithi052.stdout: "nfs", 2024-05-27T07:36:52.814 INFO:teuthology.orchestra.run.smithi052.stdout: "restful", 2024-05-27T07:36:52.815 INFO:teuthology.orchestra.run.smithi052.stdout: "snap_schedule" 2024-05-27T07:36:52.815 INFO:teuthology.orchestra.run.smithi052.stdout: ], 2024-05-27T07:36:52.815 INFO:teuthology.orchestra.run.smithi052.stdout: "available_modules": [ 2024-05-27T07:36:52.815 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-05-27T07:36:52.815 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "alerts", 2024-05-27T07:36:52.815 INFO:teuthology.orchestra.run.smithi052.stdout: "can_run": true, 2024-05-27T07:36:52.815 INFO:teuthology.orchestra.run.smithi052.stdout: "error_string": "", 2024-05-27T07:36:52.815 INFO:teuthology.orchestra.run.smithi052.stdout: "module_options": { 2024-05-27T07:36:52.815 INFO:teuthology.orchestra.run.smithi052.stdout: "interval": { 2024-05-27T07:36:52.815 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "interval", 2024-05-27T07:36:52.815 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "secs", 2024-05-27T07:36:52.815 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-05-27T07:36:52.815 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 1, 2024-05-27T07:36:52.816 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "60", 2024-05-27T07:36:52.816 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-05-27T07:36:52.816 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-05-27T07:36:52.816 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-05-27T07:36:52.816 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "How frequently to reexamine health status", 2024-05-27T07:36:52.816 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-05-27T07:36:52.816 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-05-27T07:36:52.816 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-05-27T07:36:52.816 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-05-27T07:36:52.816 INFO:teuthology.orchestra.run.smithi052.stdout: "log_level": { 2024-05-27T07:36:52.816 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "log_level", 2024-05-27T07:36:52.816 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "str", 2024-05-27T07:36:52.816 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-05-27T07:36:52.816 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 1, 2024-05-27T07:36:52.817 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "", 2024-05-27T07:36:52.817 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-05-27T07:36:52.817 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-05-27T07:36:52.817 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [ 2024-05-27T07:36:52.817 INFO:teuthology.orchestra.run.smithi052.stdout: "", 2024-05-27T07:36:52.817 INFO:teuthology.orchestra.run.smithi052.stdout: "critical", 2024-05-27T07:36:52.817 INFO:teuthology.orchestra.run.smithi052.stdout: "debug", 2024-05-27T07:36:52.817 INFO:teuthology.orchestra.run.smithi052.stdout: "error", 2024-05-27T07:36:52.817 INFO:teuthology.orchestra.run.smithi052.stdout: "info", 2024-05-27T07:36:52.817 INFO:teuthology.orchestra.run.smithi052.stdout: "warning" 2024-05-27T07:36:52.817 INFO:teuthology.orchestra.run.smithi052.stdout: ], 2024-05-27T07:36:52.817 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-05-27T07:36:52.817 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-05-27T07:36:52.818 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-05-27T07:36:52.818 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-05-27T07:36:52.818 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-05-27T07:36:52.818 INFO:teuthology.orchestra.run.smithi052.stdout: "log_to_cluster": { 2024-05-27T07:36:52.818 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "log_to_cluster", 2024-05-27T07:36:52.818 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "bool", 2024-05-27T07:36:52.818 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-05-27T07:36:52.818 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 1, 2024-05-27T07:36:52.818 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "False", 2024-05-27T07:36:52.818 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-05-27T07:36:52.818 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-05-27T07:36:52.818 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-05-27T07:36:52.818 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-05-27T07:36:52.819 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-05-27T07:36:52.819 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-05-27T07:36:52.819 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-05-27T07:36:52.819 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-05-27T07:36:52.819 INFO:teuthology.orchestra.run.smithi052.stdout: "log_to_cluster_level": { 2024-05-27T07:36:52.819 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "log_to_cluster_level", 2024-05-27T07:36:52.819 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "str", 2024-05-27T07:36:52.819 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-05-27T07:36:52.819 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 1, 2024-05-27T07:36:52.819 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "info", 2024-05-27T07:36:52.819 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-05-27T07:36:52.819 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-05-27T07:36:52.819 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [ 2024-05-27T07:36:52.820 INFO:teuthology.orchestra.run.smithi052.stdout: "", 2024-05-27T07:36:52.820 INFO:teuthology.orchestra.run.smithi052.stdout: "critical", 2024-05-27T07:36:52.820 INFO:teuthology.orchestra.run.smithi052.stdout: "debug", 2024-05-27T07:36:52.820 INFO:teuthology.orchestra.run.smithi052.stdout: "error", 2024-05-27T07:36:52.820 INFO:teuthology.orchestra.run.smithi052.stdout: "info", 2024-05-27T07:36:52.820 INFO:teuthology.orchestra.run.smithi052.stdout: "warning" 2024-05-27T07:36:52.820 INFO:teuthology.orchestra.run.smithi052.stdout: ], 2024-05-27T07:36:52.820 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-05-27T07:36:52.820 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-05-27T07:36:52.820 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-05-27T07:36:52.820 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-05-27T07:36:52.820 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-05-27T07:36:52.820 INFO:teuthology.orchestra.run.smithi052.stdout: "log_to_file": { 2024-05-27T07:36:52.821 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "log_to_file", 2024-05-27T07:36:52.821 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "bool", 2024-05-27T07:36:52.821 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-05-27T07:36:52.821 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 1, 2024-05-27T07:36:52.821 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "False", 2024-05-27T07:36:52.821 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-05-27T07:36:52.821 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-05-27T07:36:52.821 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-05-27T07:36:52.821 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-05-27T07:36:52.821 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-05-27T07:36:52.821 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-05-27T07:36:52.821 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-05-27T07:36:52.821 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-05-27T07:36:52.821 INFO:teuthology.orchestra.run.smithi052.stdout: "smtp_destination": { 2024-05-27T07:36:52.822 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "smtp_destination", 2024-05-27T07:36:52.822 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "str", 2024-05-27T07:36:52.822 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-05-27T07:36:52.822 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 1, 2024-05-27T07:36:52.822 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "", 2024-05-27T07:36:52.822 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-05-27T07:36:52.822 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-05-27T07:36:52.822 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-05-27T07:36:52.822 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "Email address to send alerts to", 2024-05-27T07:36:52.822 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-05-27T07:36:52.822 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-05-27T07:36:52.822 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-05-27T07:36:52.823 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-05-27T07:36:52.823 INFO:teuthology.orchestra.run.smithi052.stdout: "smtp_from_name": { 2024-05-27T07:36:52.823 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "smtp_from_name", 2024-05-27T07:36:52.823 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "str", 2024-05-27T07:36:52.823 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-05-27T07:36:52.823 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 1, 2024-05-27T07:36:52.823 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "Ceph", 2024-05-27T07:36:52.823 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-05-27T07:36:52.823 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-05-27T07:36:52.823 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-05-27T07:36:52.823 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "Email From: name", 2024-05-27T07:36:52.823 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-05-27T07:36:52.823 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-05-27T07:36:52.823 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-05-27T07:36:52.824 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-05-27T07:36:52.824 INFO:teuthology.orchestra.run.smithi052.stdout: "smtp_host": { 2024-05-27T07:36:52.824 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "smtp_host", 2024-05-27T07:36:52.824 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "str", 2024-05-27T07:36:52.824 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-05-27T07:36:52.824 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 1, 2024-05-27T07:36:52.824 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "", 2024-05-27T07:36:52.824 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-05-27T07:36:52.824 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-05-27T07:36:52.824 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-05-27T07:36:52.824 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "SMTP server", 2024-05-27T07:36:52.824 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-05-27T07:36:52.824 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-05-27T07:36:52.825 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-05-27T07:36:52.825 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-05-27T07:36:52.825 INFO:teuthology.orchestra.run.smithi052.stdout: "smtp_password": { 2024-05-27T07:36:52.825 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "smtp_password", 2024-05-27T07:36:52.825 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "str", 2024-05-27T07:36:52.825 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-05-27T07:36:52.825 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 1, 2024-05-27T07:36:52.825 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "", 2024-05-27T07:36:52.825 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-05-27T07:36:52.825 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-05-27T07:36:52.825 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-05-27T07:36:52.825 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "Password to authenticate with", 2024-05-27T07:36:52.825 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-05-27T07:36:52.826 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-05-27T07:36:52.826 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-05-27T07:36:52.826 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-05-27T07:36:52.826 INFO:teuthology.orchestra.run.smithi052.stdout: "smtp_port": { 2024-05-27T07:36:52.826 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "smtp_port", 2024-05-27T07:36:52.826 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "int", 2024-05-27T07:36:52.826 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-05-27T07:36:52.826 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 1, 2024-05-27T07:36:52.826 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "465", 2024-05-27T07:36:52.826 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-05-27T07:36:52.826 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-05-27T07:36:52.827 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-05-27T07:36:52.827 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "SMTP port", 2024-05-27T07:36:52.827 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-05-27T07:36:52.827 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-05-27T07:36:52.827 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-05-27T07:36:52.827 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-05-27T07:36:52.827 INFO:teuthology.orchestra.run.smithi052.stdout: "smtp_sender": { 2024-05-27T07:36:52.827 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "smtp_sender", 2024-05-27T07:36:52.827 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "str", 2024-05-27T07:36:52.827 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-05-27T07:36:52.827 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 1, 2024-05-27T07:36:52.827 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "", 2024-05-27T07:36:52.827 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-05-27T07:36:52.828 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-05-27T07:36:52.828 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-05-27T07:36:52.828 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "SMTP envelope sender", 2024-05-27T07:36:52.828 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-05-27T07:36:52.828 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-05-27T07:36:52.828 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-05-27T07:36:52.828 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-05-27T07:36:52.828 INFO:teuthology.orchestra.run.smithi052.stdout: "smtp_ssl": { 2024-05-27T07:36:52.828 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "smtp_ssl", 2024-05-27T07:36:52.828 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "bool", 2024-05-27T07:36:52.828 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-05-27T07:36:52.828 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 1, 2024-05-27T07:36:52.829 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "True", 2024-05-27T07:36:52.829 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-05-27T07:36:52.829 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-05-27T07:36:52.829 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-05-27T07:36:52.829 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "Use SSL to connect to SMTP server", 2024-05-27T07:36:52.829 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-05-27T07:36:52.829 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-05-27T07:36:52.829 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-05-27T07:36:52.829 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-05-27T07:36:52.829 INFO:teuthology.orchestra.run.smithi052.stdout: "smtp_user": { 2024-05-27T07:36:52.829 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "smtp_user", 2024-05-27T07:36:52.829 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "str", 2024-05-27T07:36:52.830 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-05-27T07:36:52.830 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 1, 2024-05-27T07:36:52.830 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "", 2024-05-27T07:36:52.830 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-05-27T07:36:52.830 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-05-27T07:36:52.830 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-05-27T07:36:52.830 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "User to authenticate as", 2024-05-27T07:36:52.830 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-05-27T07:36:52.830 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-05-27T07:36:52.830 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-05-27T07:36:52.830 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-05-27T07:36:52.830 INFO:teuthology.orchestra.run.smithi052.stdout: "sqlite3_killpoint": { 2024-05-27T07:36:52.830 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "sqlite3_killpoint", 2024-05-27T07:36:52.831 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "int", 2024-05-27T07:36:52.831 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "dev", 2024-05-27T07:36:52.831 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 1, 2024-05-27T07:36:52.831 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "0", 2024-05-27T07:36:52.831 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-05-27T07:36:52.831 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-05-27T07:36:52.831 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-05-27T07:36:52.831 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-05-27T07:36:52.831 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-05-27T07:36:52.831 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-05-27T07:36:52.831 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-05-27T07:36:52.831 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-05-27T07:36:52.831 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-05-27T07:36:52.832 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-05-27T07:36:52.832 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-05-27T07:36:52.832 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "balancer", 2024-05-27T07:36:52.832 INFO:teuthology.orchestra.run.smithi052.stdout: "can_run": true, 2024-05-27T07:36:52.832 INFO:teuthology.orchestra.run.smithi052.stdout: "error_string": "", 2024-05-27T07:36:52.832 INFO:teuthology.orchestra.run.smithi052.stdout: "module_options": { 2024-05-27T07:36:52.832 INFO:teuthology.orchestra.run.smithi052.stdout: "active": { 2024-05-27T07:36:52.832 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "active", 2024-05-27T07:36:52.832 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "bool", 2024-05-27T07:36:52.832 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-05-27T07:36:52.832 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 1, 2024-05-27T07:36:52.832 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "True", 2024-05-27T07:36:52.832 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-05-27T07:36:52.833 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-05-27T07:36:52.833 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-05-27T07:36:52.833 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "automatically balance PGs across cluster", 2024-05-27T07:36:52.833 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-05-27T07:36:52.833 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-05-27T07:36:52.833 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-05-27T07:36:52.833 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-05-27T07:36:52.833 INFO:teuthology.orchestra.run.smithi052.stdout: "begin_time": { 2024-05-27T07:36:52.833 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "begin_time", 2024-05-27T07:36:52.833 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "str", 2024-05-27T07:36:52.833 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-05-27T07:36:52.833 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 1, 2024-05-27T07:36:52.834 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "0000", 2024-05-27T07:36:52.834 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-05-27T07:36:52.834 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-05-27T07:36:52.834 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-05-27T07:36:52.834 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "beginning time of day to automatically balance", 2024-05-27T07:36:52.834 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "This is a time of day in the format HHMM.", 2024-05-27T07:36:52.834 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-05-27T07:36:52.834 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-05-27T07:36:52.834 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-05-27T07:36:52.834 INFO:teuthology.orchestra.run.smithi052.stdout: "begin_weekday": { 2024-05-27T07:36:52.834 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "begin_weekday", 2024-05-27T07:36:52.834 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "uint", 2024-05-27T07:36:52.835 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-05-27T07:36:52.835 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 1, 2024-05-27T07:36:52.835 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "0", 2024-05-27T07:36:52.835 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "0", 2024-05-27T07:36:52.835 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "6", 2024-05-27T07:36:52.835 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-05-27T07:36:52.835 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "Restrict automatic balancing to this day of the week or later", 2024-05-27T07:36:52.835 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "0 = Sunday, 1 = Monday, etc.", 2024-05-27T07:36:52.835 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-05-27T07:36:52.835 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-05-27T07:36:52.835 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-05-27T07:36:52.835 INFO:teuthology.orchestra.run.smithi052.stdout: "crush_compat_max_iterations": { 2024-05-27T07:36:52.836 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "crush_compat_max_iterations", 2024-05-27T07:36:52.836 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "uint", 2024-05-27T07:36:52.836 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-05-27T07:36:52.836 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 1, 2024-05-27T07:36:52.836 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "25", 2024-05-27T07:36:52.836 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "1", 2024-05-27T07:36:52.836 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "250", 2024-05-27T07:36:52.836 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-05-27T07:36:52.836 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "maximum number of iterations to attempt optimization", 2024-05-27T07:36:52.836 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-05-27T07:36:52.836 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-05-27T07:36:52.836 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-05-27T07:36:52.836 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-05-27T07:36:52.837 INFO:teuthology.orchestra.run.smithi052.stdout: "crush_compat_metrics": { 2024-05-27T07:36:52.837 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "crush_compat_metrics", 2024-05-27T07:36:52.837 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "str", 2024-05-27T07:36:52.837 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-05-27T07:36:52.837 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 1, 2024-05-27T07:36:52.837 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "pgs,objects,bytes", 2024-05-27T07:36:52.837 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-05-27T07:36:52.837 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-05-27T07:36:52.837 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-05-27T07:36:52.837 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "metrics with which to calculate OSD utilization", 2024-05-27T07:36:52.837 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.", 2024-05-27T07:36:52.837 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-05-27T07:36:52.837 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-05-27T07:36:52.838 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-05-27T07:36:52.838 INFO:teuthology.orchestra.run.smithi052.stdout: "crush_compat_step": { 2024-05-27T07:36:52.838 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "crush_compat_step", 2024-05-27T07:36:52.838 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "float", 2024-05-27T07:36:52.838 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-05-27T07:36:52.838 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 1, 2024-05-27T07:36:52.838 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "0.5", 2024-05-27T07:36:52.838 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "0.001", 2024-05-27T07:36:52.838 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "0.999", 2024-05-27T07:36:52.838 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-05-27T07:36:52.838 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "aggressiveness of optimization", 2024-05-27T07:36:52.838 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": ".99 is very aggressive, .01 is less aggressive", 2024-05-27T07:36:52.838 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-05-27T07:36:52.839 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-05-27T07:36:52.839 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-05-27T07:36:52.839 INFO:teuthology.orchestra.run.smithi052.stdout: "end_time": { 2024-05-27T07:36:52.839 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "end_time", 2024-05-27T07:36:52.839 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "str", 2024-05-27T07:36:52.839 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-05-27T07:36:52.839 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 1, 2024-05-27T07:36:52.839 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "2359", 2024-05-27T07:36:52.839 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-05-27T07:36:52.839 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-05-27T07:36:52.839 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-05-27T07:36:52.839 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "ending time of day to automatically balance", 2024-05-27T07:36:52.839 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "This is a time of day in the format HHMM.", 2024-05-27T07:36:52.840 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-05-27T07:36:52.840 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-05-27T07:36:52.840 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-05-27T07:36:52.840 INFO:teuthology.orchestra.run.smithi052.stdout: "end_weekday": { 2024-05-27T07:36:52.840 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "end_weekday", 2024-05-27T07:36:52.840 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "uint", 2024-05-27T07:36:52.840 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-05-27T07:36:52.840 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 1, 2024-05-27T07:36:52.840 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "0", 2024-05-27T07:36:52.840 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "0", 2024-05-27T07:36:52.840 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "6", 2024-05-27T07:36:52.840 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-05-27T07:36:52.840 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "Restrict automatic balancing to days of the week earlier than this", 2024-05-27T07:36:52.841 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "0 = Sunday, 1 = Monday, etc.", 2024-05-27T07:36:52.841 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-05-27T07:36:52.841 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-05-27T07:36:52.841 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-05-27T07:36:52.841 INFO:teuthology.orchestra.run.smithi052.stdout: "log_level": { 2024-05-27T07:36:52.841 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "log_level", 2024-05-27T07:36:52.841 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "str", 2024-05-27T07:36:52.841 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-05-27T07:36:52.841 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 1, 2024-05-27T07:36:52.841 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "", 2024-05-27T07:36:52.841 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-05-27T07:36:52.841 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-05-27T07:36:52.841 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [ 2024-05-27T07:36:52.841 INFO:teuthology.orchestra.run.smithi052.stdout: "", 2024-05-27T07:36:52.842 INFO:teuthology.orchestra.run.smithi052.stdout: "critical", 2024-05-27T07:36:52.842 INFO:teuthology.orchestra.run.smithi052.stdout: "debug", 2024-05-27T07:36:52.842 INFO:teuthology.orchestra.run.smithi052.stdout: "error", 2024-05-27T07:36:52.842 INFO:teuthology.orchestra.run.smithi052.stdout: "info", 2024-05-27T07:36:52.842 INFO:teuthology.orchestra.run.smithi052.stdout: "warning" 2024-05-27T07:36:52.842 INFO:teuthology.orchestra.run.smithi052.stdout: ], 2024-05-27T07:36:52.842 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-05-27T07:36:52.842 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-05-27T07:36:52.842 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-05-27T07:36:52.842 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-05-27T07:36:52.842 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-05-27T07:36:52.842 INFO:teuthology.orchestra.run.smithi052.stdout: "log_to_cluster": { 2024-05-27T07:36:52.843 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "log_to_cluster", 2024-05-27T07:36:52.843 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "bool", 2024-05-27T07:36:52.843 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-05-27T07:36:52.843 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 1, 2024-05-27T07:36:52.843 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "False", 2024-05-27T07:36:52.843 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-05-27T07:36:52.843 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-05-27T07:36:52.843 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-05-27T07:36:52.843 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-05-27T07:36:52.843 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-05-27T07:36:52.843 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-05-27T07:36:52.843 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-05-27T07:36:52.843 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-05-27T07:36:52.844 INFO:teuthology.orchestra.run.smithi052.stdout: "log_to_cluster_level": { 2024-05-27T07:36:52.844 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "log_to_cluster_level", 2024-05-27T07:36:52.844 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "str", 2024-05-27T07:36:52.844 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-05-27T07:36:52.844 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 1, 2024-05-27T07:36:52.844 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "info", 2024-05-27T07:36:52.844 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-05-27T07:36:52.844 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-05-27T07:36:52.844 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [ 2024-05-27T07:36:52.844 INFO:teuthology.orchestra.run.smithi052.stdout: "", 2024-05-27T07:36:52.844 INFO:teuthology.orchestra.run.smithi052.stdout: "critical", 2024-05-27T07:36:52.844 INFO:teuthology.orchestra.run.smithi052.stdout: "debug", 2024-05-27T07:36:52.845 INFO:teuthology.orchestra.run.smithi052.stdout: "error", 2024-05-27T07:36:52.845 INFO:teuthology.orchestra.run.smithi052.stdout: "info", 2024-05-27T07:36:52.845 INFO:teuthology.orchestra.run.smithi052.stdout: "warning" 2024-05-27T07:36:52.845 INFO:teuthology.orchestra.run.smithi052.stdout: ], 2024-05-27T07:36:52.845 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-05-27T07:36:52.845 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-05-27T07:36:52.845 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-05-27T07:36:52.845 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-05-27T07:36:52.845 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-05-27T07:36:52.845 INFO:teuthology.orchestra.run.smithi052.stdout: "log_to_file": { 2024-05-27T07:36:52.845 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "log_to_file", 2024-05-27T07:36:52.845 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "bool", 2024-05-27T07:36:52.845 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-05-27T07:36:52.846 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 1, 2024-05-27T07:36:52.846 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "False", 2024-05-27T07:36:52.846 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-05-27T07:36:52.846 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-05-27T07:36:52.846 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-05-27T07:36:52.846 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-05-27T07:36:52.846 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-05-27T07:36:52.846 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-05-27T07:36:52.846 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-05-27T07:36:52.846 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-05-27T07:36:52.846 INFO:teuthology.orchestra.run.smithi052.stdout: "min_score": { 2024-05-27T07:36:52.847 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "min_score", 2024-05-27T07:36:52.847 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "float", 2024-05-27T07:36:52.847 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-05-27T07:36:52.847 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 1, 2024-05-27T07:36:52.847 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "0", 2024-05-27T07:36:52.847 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-05-27T07:36:52.847 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-05-27T07:36:52.847 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-05-27T07:36:52.847 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "minimum score, below which no optimization is attempted", 2024-05-27T07:36:52.847 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-05-27T07:36:52.847 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-05-27T07:36:52.847 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-05-27T07:36:52.847 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-05-27T07:36:52.848 INFO:teuthology.orchestra.run.smithi052.stdout: "mode": { 2024-05-27T07:36:52.848 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "mode", 2024-05-27T07:36:52.848 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "str", 2024-05-27T07:36:52.848 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-05-27T07:36:52.848 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 1, 2024-05-27T07:36:52.848 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "upmap", 2024-05-27T07:36:52.848 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-05-27T07:36:52.848 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-05-27T07:36:52.848 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [ 2024-05-27T07:36:52.848 INFO:teuthology.orchestra.run.smithi052.stdout: "crush-compat", 2024-05-27T07:36:52.848 INFO:teuthology.orchestra.run.smithi052.stdout: "none", 2024-05-27T07:36:52.848 INFO:teuthology.orchestra.run.smithi052.stdout: "read", 2024-05-27T07:36:52.848 INFO:teuthology.orchestra.run.smithi052.stdout: "upmap", 2024-05-27T07:36:52.849 INFO:teuthology.orchestra.run.smithi052.stdout: "upmap-read" 2024-05-27T07:36:52.849 INFO:teuthology.orchestra.run.smithi052.stdout: ], 2024-05-27T07:36:52.849 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "Balancer mode", 2024-05-27T07:36:52.849 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-05-27T07:36:52.849 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-05-27T07:36:52.849 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-05-27T07:36:52.849 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-05-27T07:36:52.849 INFO:teuthology.orchestra.run.smithi052.stdout: "pool_ids": { 2024-05-27T07:36:52.849 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "pool_ids", 2024-05-27T07:36:52.849 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "str", 2024-05-27T07:36:52.849 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-05-27T07:36:52.849 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 1, 2024-05-27T07:36:52.850 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "", 2024-05-27T07:36:52.850 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-05-27T07:36:52.850 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-05-27T07:36:52.850 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-05-27T07:36:52.850 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "pools which the automatic balancing will be limited to", 2024-05-27T07:36:52.850 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-05-27T07:36:52.850 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-05-27T07:36:52.850 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-05-27T07:36:52.850 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-05-27T07:36:52.850 INFO:teuthology.orchestra.run.smithi052.stdout: "sleep_interval": { 2024-05-27T07:36:52.850 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "sleep_interval", 2024-05-27T07:36:52.850 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "secs", 2024-05-27T07:36:52.850 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-05-27T07:36:52.851 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 1, 2024-05-27T07:36:52.851 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "60", 2024-05-27T07:36:52.851 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-05-27T07:36:52.851 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-05-27T07:36:52.851 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-05-27T07:36:52.851 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "how frequently to wake up and attempt optimization", 2024-05-27T07:36:52.851 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-05-27T07:36:52.851 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-05-27T07:36:52.851 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-05-27T07:36:52.851 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-05-27T07:36:52.851 INFO:teuthology.orchestra.run.smithi052.stdout: "sqlite3_killpoint": { 2024-05-27T07:36:52.851 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "sqlite3_killpoint", 2024-05-27T07:36:52.851 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "int", 2024-05-27T07:36:52.852 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "dev", 2024-05-27T07:36:52.852 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 1, 2024-05-27T07:36:52.852 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "0", 2024-05-27T07:36:52.852 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-05-27T07:36:52.852 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-05-27T07:36:52.852 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-05-27T07:36:52.852 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-05-27T07:36:52.852 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-05-27T07:36:52.852 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-05-27T07:36:52.852 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-05-27T07:36:52.852 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-05-27T07:36:52.852 INFO:teuthology.orchestra.run.smithi052.stdout: "upmap_max_deviation": { 2024-05-27T07:36:52.852 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "upmap_max_deviation", 2024-05-27T07:36:52.853 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "int", 2024-05-27T07:36:52.853 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-05-27T07:36:52.853 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 1, 2024-05-27T07:36:52.853 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "5", 2024-05-27T07:36:52.853 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "1", 2024-05-27T07:36:52.853 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-05-27T07:36:52.853 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-05-27T07:36:52.853 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "deviation below which no optimization is attempted", 2024-05-27T07:36:52.853 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "If the number of PGs are within this count then no optimization is attempted", 2024-05-27T07:36:52.853 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-05-27T07:36:52.853 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-05-27T07:36:52.853 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-05-27T07:36:52.853 INFO:teuthology.orchestra.run.smithi052.stdout: "upmap_max_optimizations": { 2024-05-27T07:36:52.854 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "upmap_max_optimizations", 2024-05-27T07:36:52.854 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "uint", 2024-05-27T07:36:52.854 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-05-27T07:36:52.854 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 1, 2024-05-27T07:36:52.854 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "10", 2024-05-27T07:36:52.854 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-05-27T07:36:52.854 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-05-27T07:36:52.854 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-05-27T07:36:52.854 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "maximum upmap optimizations to make per attempt", 2024-05-27T07:36:52.854 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-05-27T07:36:52.854 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-05-27T07:36:52.854 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-05-27T07:36:52.854 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-05-27T07:36:52.855 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-05-27T07:36:52.855 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-05-27T07:36:52.855 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-05-27T07:36:52.855 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "cephadm", 2024-05-27T07:36:52.855 INFO:teuthology.orchestra.run.smithi052.stdout: "can_run": true, 2024-05-27T07:36:52.855 INFO:teuthology.orchestra.run.smithi052.stdout: "error_string": "", 2024-05-27T07:36:52.855 INFO:teuthology.orchestra.run.smithi052.stdout: "module_options": { 2024-05-27T07:36:52.855 INFO:teuthology.orchestra.run.smithi052.stdout: "agent_down_multiplier": { 2024-05-27T07:36:52.855 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "agent_down_multiplier", 2024-05-27T07:36:52.855 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "float", 2024-05-27T07:36:52.855 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-05-27T07:36:52.855 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 0, 2024-05-27T07:36:52.855 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "3.0", 2024-05-27T07:36:52.856 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-05-27T07:36:52.856 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-05-27T07:36:52.856 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-05-27T07:36:52.856 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "Multiplied by agent refresh rate to calculate how long agent must not report before being marked down", 2024-05-27T07:36:52.856 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-05-27T07:36:52.856 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-05-27T07:36:52.856 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-05-27T07:36:52.856 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-05-27T07:36:52.856 INFO:teuthology.orchestra.run.smithi052.stdout: "agent_refresh_rate": { 2024-05-27T07:36:52.856 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "agent_refresh_rate", 2024-05-27T07:36:52.856 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "secs", 2024-05-27T07:36:52.856 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-05-27T07:36:52.856 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 0, 2024-05-27T07:36:52.856 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "20", 2024-05-27T07:36:52.857 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-05-27T07:36:52.857 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-05-27T07:36:52.857 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-05-27T07:36:52.857 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "How often agent on each host will try to gather and send metadata", 2024-05-27T07:36:52.857 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-05-27T07:36:52.857 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-05-27T07:36:52.857 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-05-27T07:36:52.857 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-05-27T07:36:52.857 INFO:teuthology.orchestra.run.smithi052.stdout: "agent_starting_port": { 2024-05-27T07:36:52.857 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "agent_starting_port", 2024-05-27T07:36:52.857 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "int", 2024-05-27T07:36:52.857 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-05-27T07:36:52.858 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 0, 2024-05-27T07:36:52.858 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "4721", 2024-05-27T07:36:52.858 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-05-27T07:36:52.858 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-05-27T07:36:52.858 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-05-27T07:36:52.858 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)", 2024-05-27T07:36:52.858 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-05-27T07:36:52.858 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-05-27T07:36:52.858 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-05-27T07:36:52.858 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-05-27T07:36:52.858 INFO:teuthology.orchestra.run.smithi052.stdout: "allow_ptrace": { 2024-05-27T07:36:52.858 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "allow_ptrace", 2024-05-27T07:36:52.859 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "bool", 2024-05-27T07:36:52.859 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-05-27T07:36:52.859 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 0, 2024-05-27T07:36:52.859 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "False", 2024-05-27T07:36:52.859 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-05-27T07:36:52.859 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-05-27T07:36:52.859 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-05-27T07:36:52.859 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "allow SYS_PTRACE capability on ceph containers", 2024-05-27T07:36:52.859 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.", 2024-05-27T07:36:52.859 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-05-27T07:36:52.859 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-05-27T07:36:52.859 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-05-27T07:36:52.859 INFO:teuthology.orchestra.run.smithi052.stdout: "autotune_interval": { 2024-05-27T07:36:52.860 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "autotune_interval", 2024-05-27T07:36:52.860 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "secs", 2024-05-27T07:36:52.860 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-05-27T07:36:52.860 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 0, 2024-05-27T07:36:52.860 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "600", 2024-05-27T07:36:52.860 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-05-27T07:36:52.860 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-05-27T07:36:52.860 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-05-27T07:36:52.860 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "how frequently to autotune daemon memory", 2024-05-27T07:36:52.860 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-05-27T07:36:52.860 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-05-27T07:36:52.860 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-05-27T07:36:52.861 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-05-27T07:36:52.861 INFO:teuthology.orchestra.run.smithi052.stdout: "autotune_memory_target_ratio": { 2024-05-27T07:36:52.861 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "autotune_memory_target_ratio", 2024-05-27T07:36:52.861 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "float", 2024-05-27T07:36:52.861 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-05-27T07:36:52.861 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 0, 2024-05-27T07:36:52.861 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "0.7", 2024-05-27T07:36:52.861 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-05-27T07:36:52.861 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-05-27T07:36:52.861 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-05-27T07:36:52.861 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "ratio of total system memory to divide amongst autotuned daemons", 2024-05-27T07:36:52.861 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-05-27T07:36:52.861 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-05-27T07:36:52.862 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-05-27T07:36:52.862 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-05-27T07:36:52.862 INFO:teuthology.orchestra.run.smithi052.stdout: "cephadm_log_destination": { 2024-05-27T07:36:52.862 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "cephadm_log_destination", 2024-05-27T07:36:52.862 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "str", 2024-05-27T07:36:52.862 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-05-27T07:36:52.862 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 0, 2024-05-27T07:36:52.862 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "", 2024-05-27T07:36:52.862 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-05-27T07:36:52.862 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-05-27T07:36:52.862 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [ 2024-05-27T07:36:52.862 INFO:teuthology.orchestra.run.smithi052.stdout: "file", 2024-05-27T07:36:52.863 INFO:teuthology.orchestra.run.smithi052.stdout: "file,syslog", 2024-05-27T07:36:52.863 INFO:teuthology.orchestra.run.smithi052.stdout: "syslog" 2024-05-27T07:36:52.863 INFO:teuthology.orchestra.run.smithi052.stdout: ], 2024-05-27T07:36:52.863 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "Destination for cephadm command's persistent logging", 2024-05-27T07:36:52.863 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-05-27T07:36:52.863 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-05-27T07:36:52.863 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-05-27T07:36:52.863 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-05-27T07:36:52.863 INFO:teuthology.orchestra.run.smithi052.stdout: "cgroups_split": { 2024-05-27T07:36:52.863 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "cgroups_split", 2024-05-27T07:36:52.863 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "bool", 2024-05-27T07:36:52.863 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-05-27T07:36:52.863 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 0, 2024-05-27T07:36:52.864 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "True", 2024-05-27T07:36:52.864 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-05-27T07:36:52.864 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-05-27T07:36:52.864 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-05-27T07:36:52.864 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "Pass --cgroups=split when cephadm creates containers (currently podman only)", 2024-05-27T07:36:52.864 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-05-27T07:36:52.864 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-05-27T07:36:52.864 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-05-27T07:36:52.864 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-05-27T07:36:52.864 INFO:teuthology.orchestra.run.smithi052.stdout: "config_checks_enabled": { 2024-05-27T07:36:52.864 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "config_checks_enabled", 2024-05-27T07:36:52.864 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "bool", 2024-05-27T07:36:52.865 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-05-27T07:36:52.865 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 0, 2024-05-27T07:36:52.865 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "False", 2024-05-27T07:36:52.865 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-05-27T07:36:52.865 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-05-27T07:36:52.865 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-05-27T07:36:52.865 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "Enable or disable the cephadm configuration analysis", 2024-05-27T07:36:52.865 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-05-27T07:36:52.865 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-05-27T07:36:52.865 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-05-27T07:36:52.865 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-05-27T07:36:52.865 INFO:teuthology.orchestra.run.smithi052.stdout: "config_dashboard": { 2024-05-27T07:36:52.865 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "config_dashboard", 2024-05-27T07:36:52.866 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "bool", 2024-05-27T07:36:52.866 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-05-27T07:36:52.866 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 0, 2024-05-27T07:36:52.866 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "True", 2024-05-27T07:36:52.866 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-05-27T07:36:52.866 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-05-27T07:36:52.866 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-05-27T07:36:52.866 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "manage configs like API endpoints in Dashboard.", 2024-05-27T07:36:52.866 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-05-27T07:36:52.866 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-05-27T07:36:52.866 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-05-27T07:36:52.866 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-05-27T07:36:52.866 INFO:teuthology.orchestra.run.smithi052.stdout: "container_image_alertmanager": { 2024-05-27T07:36:52.867 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "container_image_alertmanager", 2024-05-27T07:36:52.867 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "str", 2024-05-27T07:36:52.867 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-05-27T07:36:52.867 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 0, 2024-05-27T07:36:52.867 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "quay.io/prometheus/alertmanager:v0.25.0", 2024-05-27T07:36:52.867 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-05-27T07:36:52.867 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-05-27T07:36:52.867 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-05-27T07:36:52.867 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "Prometheus container image", 2024-05-27T07:36:52.867 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-05-27T07:36:52.867 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-05-27T07:36:52.867 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-05-27T07:36:52.867 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-05-27T07:36:52.868 INFO:teuthology.orchestra.run.smithi052.stdout: "container_image_base": { 2024-05-27T07:36:52.868 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "container_image_base", 2024-05-27T07:36:52.868 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "str", 2024-05-27T07:36:52.868 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-05-27T07:36:52.868 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 1, 2024-05-27T07:36:52.868 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "quay.io/ceph/ceph", 2024-05-27T07:36:52.868 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-05-27T07:36:52.868 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-05-27T07:36:52.868 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-05-27T07:36:52.868 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "Container image name, without the tag", 2024-05-27T07:36:52.868 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-05-27T07:36:52.868 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-05-27T07:36:52.868 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-05-27T07:36:52.869 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-05-27T07:36:52.869 INFO:teuthology.orchestra.run.smithi052.stdout: "container_image_elasticsearch": { 2024-05-27T07:36:52.869 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "container_image_elasticsearch", 2024-05-27T07:36:52.869 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "str", 2024-05-27T07:36:52.869 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-05-27T07:36:52.869 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 0, 2024-05-27T07:36:52.869 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "quay.io/omrizeneva/elasticsearch:6.8.23", 2024-05-27T07:36:52.869 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-05-27T07:36:52.869 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-05-27T07:36:52.869 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-05-27T07:36:52.869 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "elasticsearch container image", 2024-05-27T07:36:52.869 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-05-27T07:36:52.869 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-05-27T07:36:52.870 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-05-27T07:36:52.870 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-05-27T07:36:52.870 INFO:teuthology.orchestra.run.smithi052.stdout: "container_image_grafana": { 2024-05-27T07:36:52.870 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "container_image_grafana", 2024-05-27T07:36:52.870 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "str", 2024-05-27T07:36:52.870 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-05-27T07:36:52.870 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 0, 2024-05-27T07:36:52.870 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "quay.io/ceph/grafana:9.4.12", 2024-05-27T07:36:52.870 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-05-27T07:36:52.870 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-05-27T07:36:52.870 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-05-27T07:36:52.870 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "Prometheus container image", 2024-05-27T07:36:52.870 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-05-27T07:36:52.871 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-05-27T07:36:52.871 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-05-27T07:36:52.871 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-05-27T07:36:52.871 INFO:teuthology.orchestra.run.smithi052.stdout: "container_image_haproxy": { 2024-05-27T07:36:52.871 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "container_image_haproxy", 2024-05-27T07:36:52.871 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "str", 2024-05-27T07:36:52.871 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-05-27T07:36:52.871 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 0, 2024-05-27T07:36:52.871 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "quay.io/ceph/haproxy:2.3", 2024-05-27T07:36:52.871 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-05-27T07:36:52.871 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-05-27T07:36:52.871 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-05-27T07:36:52.871 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "HAproxy container image", 2024-05-27T07:36:52.871 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-05-27T07:36:52.872 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-05-27T07:36:52.872 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-05-27T07:36:52.872 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-05-27T07:36:52.872 INFO:teuthology.orchestra.run.smithi052.stdout: "container_image_jaeger_agent": { 2024-05-27T07:36:52.872 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "container_image_jaeger_agent", 2024-05-27T07:36:52.872 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "str", 2024-05-27T07:36:52.872 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-05-27T07:36:52.872 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 0, 2024-05-27T07:36:52.872 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "quay.io/jaegertracing/jaeger-agent:1.29", 2024-05-27T07:36:52.872 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-05-27T07:36:52.872 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-05-27T07:36:52.872 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-05-27T07:36:52.872 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "Jaeger agent container image", 2024-05-27T07:36:52.873 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-05-27T07:36:52.873 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-05-27T07:36:52.873 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-05-27T07:36:52.873 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-05-27T07:36:52.873 INFO:teuthology.orchestra.run.smithi052.stdout: "container_image_jaeger_collector": { 2024-05-27T07:36:52.873 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "container_image_jaeger_collector", 2024-05-27T07:36:52.873 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "str", 2024-05-27T07:36:52.873 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-05-27T07:36:52.873 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 0, 2024-05-27T07:36:52.873 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "quay.io/jaegertracing/jaeger-collector:1.29", 2024-05-27T07:36:52.873 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-05-27T07:36:52.873 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-05-27T07:36:52.874 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-05-27T07:36:52.874 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "Jaeger collector container image", 2024-05-27T07:36:52.874 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-05-27T07:36:52.874 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-05-27T07:36:52.874 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-05-27T07:36:52.874 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-05-27T07:36:52.874 INFO:teuthology.orchestra.run.smithi052.stdout: "container_image_jaeger_query": { 2024-05-27T07:36:52.874 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "container_image_jaeger_query", 2024-05-27T07:36:52.874 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "str", 2024-05-27T07:36:52.874 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-05-27T07:36:52.874 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 0, 2024-05-27T07:36:52.874 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "quay.io/jaegertracing/jaeger-query:1.29", 2024-05-27T07:36:52.874 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-05-27T07:36:52.875 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-05-27T07:36:52.875 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-05-27T07:36:52.875 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "Jaeger query container image", 2024-05-27T07:36:52.875 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-05-27T07:36:52.875 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-05-27T07:36:52.875 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-05-27T07:36:52.875 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-05-27T07:36:52.875 INFO:teuthology.orchestra.run.smithi052.stdout: "container_image_keepalived": { 2024-05-27T07:36:52.875 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "container_image_keepalived", 2024-05-27T07:36:52.875 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "str", 2024-05-27T07:36:52.875 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-05-27T07:36:52.875 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 0, 2024-05-27T07:36:52.875 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "quay.io/ceph/keepalived:2.2.4", 2024-05-27T07:36:52.876 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-05-27T07:36:52.876 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-05-27T07:36:52.876 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-05-27T07:36:52.876 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "Keepalived container image", 2024-05-27T07:36:52.876 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-05-27T07:36:52.876 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-05-27T07:36:52.876 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-05-27T07:36:52.876 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-05-27T07:36:52.876 INFO:teuthology.orchestra.run.smithi052.stdout: "container_image_loki": { 2024-05-27T07:36:52.876 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "container_image_loki", 2024-05-27T07:36:52.876 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "str", 2024-05-27T07:36:52.876 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-05-27T07:36:52.876 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 0, 2024-05-27T07:36:52.877 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "docker.io/grafana/loki:3.0.0", 2024-05-27T07:36:52.877 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-05-27T07:36:52.877 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-05-27T07:36:52.877 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-05-27T07:36:52.877 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "Loki container image", 2024-05-27T07:36:52.877 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-05-27T07:36:52.877 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-05-27T07:36:52.877 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-05-27T07:36:52.877 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-05-27T07:36:52.877 INFO:teuthology.orchestra.run.smithi052.stdout: "container_image_node_exporter": { 2024-05-27T07:36:52.877 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "container_image_node_exporter", 2024-05-27T07:36:52.877 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "str", 2024-05-27T07:36:52.877 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-05-27T07:36:52.878 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 0, 2024-05-27T07:36:52.878 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "quay.io/prometheus/node-exporter:v1.5.0", 2024-05-27T07:36:52.878 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-05-27T07:36:52.878 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-05-27T07:36:52.878 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-05-27T07:36:52.878 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "Prometheus container image", 2024-05-27T07:36:52.878 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-05-27T07:36:52.878 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-05-27T07:36:52.878 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-05-27T07:36:52.878 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-05-27T07:36:52.878 INFO:teuthology.orchestra.run.smithi052.stdout: "container_image_nvmeof": { 2024-05-27T07:36:52.878 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "container_image_nvmeof", 2024-05-27T07:36:52.878 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "str", 2024-05-27T07:36:52.879 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-05-27T07:36:52.879 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 0, 2024-05-27T07:36:52.879 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "quay.io/ceph/nvmeof:1.2.5", 2024-05-27T07:36:52.879 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-05-27T07:36:52.879 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-05-27T07:36:52.879 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-05-27T07:36:52.879 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "Nvme-of container image", 2024-05-27T07:36:52.879 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-05-27T07:36:52.879 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-05-27T07:36:52.879 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-05-27T07:36:52.879 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-05-27T07:36:52.879 INFO:teuthology.orchestra.run.smithi052.stdout: "container_image_prometheus": { 2024-05-27T07:36:52.879 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "container_image_prometheus", 2024-05-27T07:36:52.880 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "str", 2024-05-27T07:36:52.880 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-05-27T07:36:52.880 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 0, 2024-05-27T07:36:52.880 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "quay.io/prometheus/prometheus:v2.43.0", 2024-05-27T07:36:52.880 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-05-27T07:36:52.880 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-05-27T07:36:52.880 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-05-27T07:36:52.880 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "Prometheus container image", 2024-05-27T07:36:52.880 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-05-27T07:36:52.880 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-05-27T07:36:52.880 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-05-27T07:36:52.880 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-05-27T07:36:52.880 INFO:teuthology.orchestra.run.smithi052.stdout: "container_image_promtail": { 2024-05-27T07:36:52.881 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "container_image_promtail", 2024-05-27T07:36:52.881 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "str", 2024-05-27T07:36:52.881 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-05-27T07:36:52.881 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 0, 2024-05-27T07:36:52.881 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "docker.io/grafana/promtail:3.0.0", 2024-05-27T07:36:52.881 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-05-27T07:36:52.881 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-05-27T07:36:52.881 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-05-27T07:36:52.881 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "Promtail container image", 2024-05-27T07:36:52.881 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-05-27T07:36:52.881 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-05-27T07:36:52.881 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-05-27T07:36:52.882 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-05-27T07:36:52.882 INFO:teuthology.orchestra.run.smithi052.stdout: "container_image_samba": { 2024-05-27T07:36:52.882 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "container_image_samba", 2024-05-27T07:36:52.882 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "str", 2024-05-27T07:36:52.882 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-05-27T07:36:52.882 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 0, 2024-05-27T07:36:52.882 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "quay.io/samba.org/samba-server:devbuilds-centos-amd64", 2024-05-27T07:36:52.882 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-05-27T07:36:52.882 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-05-27T07:36:52.882 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-05-27T07:36:52.882 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "Samba/SMB container image", 2024-05-27T07:36:52.882 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-05-27T07:36:52.883 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-05-27T07:36:52.883 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-05-27T07:36:52.883 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-05-27T07:36:52.883 INFO:teuthology.orchestra.run.smithi052.stdout: "container_image_snmp_gateway": { 2024-05-27T07:36:52.883 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "container_image_snmp_gateway", 2024-05-27T07:36:52.883 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "str", 2024-05-27T07:36:52.883 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-05-27T07:36:52.883 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 0, 2024-05-27T07:36:52.883 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "docker.io/maxwo/snmp-notifier:v1.2.1", 2024-05-27T07:36:52.883 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-05-27T07:36:52.883 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-05-27T07:36:52.883 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-05-27T07:36:52.883 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "SNMP Gateway container image", 2024-05-27T07:36:52.884 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-05-27T07:36:52.884 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-05-27T07:36:52.884 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-05-27T07:36:52.884 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-05-27T07:36:52.884 INFO:teuthology.orchestra.run.smithi052.stdout: "container_init": { 2024-05-27T07:36:52.884 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "container_init", 2024-05-27T07:36:52.884 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "bool", 2024-05-27T07:36:52.884 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-05-27T07:36:52.884 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 0, 2024-05-27T07:36:52.884 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "True", 2024-05-27T07:36:52.884 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-05-27T07:36:52.884 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-05-27T07:36:52.884 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-05-27T07:36:52.884 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "Run podman/docker with `--init`", 2024-05-27T07:36:52.885 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-05-27T07:36:52.885 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-05-27T07:36:52.885 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-05-27T07:36:52.885 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-05-27T07:36:52.885 INFO:teuthology.orchestra.run.smithi052.stdout: "daemon_cache_timeout": { 2024-05-27T07:36:52.885 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "daemon_cache_timeout", 2024-05-27T07:36:52.885 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "secs", 2024-05-27T07:36:52.885 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-05-27T07:36:52.885 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 0, 2024-05-27T07:36:52.885 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "600", 2024-05-27T07:36:52.885 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-05-27T07:36:52.885 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-05-27T07:36:52.885 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-05-27T07:36:52.886 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "seconds to cache service (daemon) inventory", 2024-05-27T07:36:52.886 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-05-27T07:36:52.886 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-05-27T07:36:52.886 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-05-27T07:36:52.886 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-05-27T07:36:52.886 INFO:teuthology.orchestra.run.smithi052.stdout: "default_cephadm_command_timeout": { 2024-05-27T07:36:52.886 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "default_cephadm_command_timeout", 2024-05-27T07:36:52.886 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "int", 2024-05-27T07:36:52.886 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-05-27T07:36:52.886 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 0, 2024-05-27T07:36:52.886 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "900", 2024-05-27T07:36:52.886 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-05-27T07:36:52.886 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-05-27T07:36:52.887 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-05-27T07:36:52.887 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "Default timeout applied to cephadm commands run directly on the host (in seconds)", 2024-05-27T07:36:52.887 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-05-27T07:36:52.887 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-05-27T07:36:52.887 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-05-27T07:36:52.887 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-05-27T07:36:52.887 INFO:teuthology.orchestra.run.smithi052.stdout: "default_registry": { 2024-05-27T07:36:52.887 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "default_registry", 2024-05-27T07:36:52.887 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "str", 2024-05-27T07:36:52.887 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-05-27T07:36:52.887 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 0, 2024-05-27T07:36:52.887 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "docker.io", 2024-05-27T07:36:52.887 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-05-27T07:36:52.887 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-05-27T07:36:52.888 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-05-27T07:36:52.888 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "Search-registry to which we should normalize unqualified image names. This is not the default registry", 2024-05-27T07:36:52.888 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-05-27T07:36:52.888 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-05-27T07:36:52.888 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-05-27T07:36:52.888 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-05-27T07:36:52.888 INFO:teuthology.orchestra.run.smithi052.stdout: "device_cache_timeout": { 2024-05-27T07:36:52.888 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "device_cache_timeout", 2024-05-27T07:36:52.888 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "secs", 2024-05-27T07:36:52.888 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-05-27T07:36:52.888 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 0, 2024-05-27T07:36:52.888 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "1800", 2024-05-27T07:36:52.888 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-05-27T07:36:52.889 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-05-27T07:36:52.889 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-05-27T07:36:52.889 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "seconds to cache device inventory", 2024-05-27T07:36:52.889 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-05-27T07:36:52.889 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-05-27T07:36:52.889 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-05-27T07:36:52.889 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-05-27T07:36:52.889 INFO:teuthology.orchestra.run.smithi052.stdout: "device_enhanced_scan": { 2024-05-27T07:36:52.889 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "device_enhanced_scan", 2024-05-27T07:36:52.889 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "bool", 2024-05-27T07:36:52.889 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-05-27T07:36:52.889 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 0, 2024-05-27T07:36:52.889 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "False", 2024-05-27T07:36:52.890 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-05-27T07:36:52.890 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-05-27T07:36:52.890 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-05-27T07:36:52.890 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "Use libstoragemgmt during device scans", 2024-05-27T07:36:52.890 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-05-27T07:36:52.890 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-05-27T07:36:52.890 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-05-27T07:36:52.890 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-05-27T07:36:52.890 INFO:teuthology.orchestra.run.smithi052.stdout: "facts_cache_timeout": { 2024-05-27T07:36:52.890 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "facts_cache_timeout", 2024-05-27T07:36:52.890 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "secs", 2024-05-27T07:36:52.890 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-05-27T07:36:52.891 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 0, 2024-05-27T07:36:52.891 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "60", 2024-05-27T07:36:52.891 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-05-27T07:36:52.891 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-05-27T07:36:52.891 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-05-27T07:36:52.891 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "seconds to cache host facts data", 2024-05-27T07:36:52.891 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-05-27T07:36:52.891 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-05-27T07:36:52.891 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-05-27T07:36:52.891 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-05-27T07:36:52.891 INFO:teuthology.orchestra.run.smithi052.stdout: "grafana_dashboards_path": { 2024-05-27T07:36:52.891 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "grafana_dashboards_path", 2024-05-27T07:36:52.891 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "str", 2024-05-27T07:36:52.892 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-05-27T07:36:52.892 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 0, 2024-05-27T07:36:52.892 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "/etc/grafana/dashboards/ceph-dashboard/", 2024-05-27T07:36:52.892 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-05-27T07:36:52.892 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-05-27T07:36:52.892 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-05-27T07:36:52.892 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "location of dashboards to include in grafana deployments", 2024-05-27T07:36:52.892 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-05-27T07:36:52.892 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-05-27T07:36:52.892 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-05-27T07:36:52.892 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-05-27T07:36:52.892 INFO:teuthology.orchestra.run.smithi052.stdout: "host_check_interval": { 2024-05-27T07:36:52.892 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "host_check_interval", 2024-05-27T07:36:52.892 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "secs", 2024-05-27T07:36:52.893 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-05-27T07:36:52.893 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 0, 2024-05-27T07:36:52.893 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "600", 2024-05-27T07:36:52.893 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-05-27T07:36:52.893 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-05-27T07:36:52.893 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-05-27T07:36:52.893 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "how frequently to perform a host check", 2024-05-27T07:36:52.893 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-05-27T07:36:52.893 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-05-27T07:36:52.893 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-05-27T07:36:52.893 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-05-27T07:36:52.893 INFO:teuthology.orchestra.run.smithi052.stdout: "hw_monitoring": { 2024-05-27T07:36:52.893 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "hw_monitoring", 2024-05-27T07:36:52.894 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "bool", 2024-05-27T07:36:52.894 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-05-27T07:36:52.894 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 0, 2024-05-27T07:36:52.894 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "False", 2024-05-27T07:36:52.894 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-05-27T07:36:52.894 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-05-27T07:36:52.894 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-05-27T07:36:52.894 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "Deploy hw monitoring daemon on every host.", 2024-05-27T07:36:52.894 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-05-27T07:36:52.894 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-05-27T07:36:52.894 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-05-27T07:36:52.894 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-05-27T07:36:52.895 INFO:teuthology.orchestra.run.smithi052.stdout: "inventory_list_all": { 2024-05-27T07:36:52.895 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "inventory_list_all", 2024-05-27T07:36:52.895 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "bool", 2024-05-27T07:36:52.895 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-05-27T07:36:52.895 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 0, 2024-05-27T07:36:52.895 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "False", 2024-05-27T07:36:52.895 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-05-27T07:36:52.895 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-05-27T07:36:52.895 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-05-27T07:36:52.895 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)", 2024-05-27T07:36:52.895 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-05-27T07:36:52.895 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-05-27T07:36:52.895 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-05-27T07:36:52.895 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-05-27T07:36:52.896 INFO:teuthology.orchestra.run.smithi052.stdout: "log_level": { 2024-05-27T07:36:52.896 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "log_level", 2024-05-27T07:36:52.896 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "str", 2024-05-27T07:36:52.896 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-05-27T07:36:52.896 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 1, 2024-05-27T07:36:52.896 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "", 2024-05-27T07:36:52.896 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-05-27T07:36:52.896 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-05-27T07:36:52.896 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [ 2024-05-27T07:36:52.896 INFO:teuthology.orchestra.run.smithi052.stdout: "", 2024-05-27T07:36:52.896 INFO:teuthology.orchestra.run.smithi052.stdout: "critical", 2024-05-27T07:36:52.896 INFO:teuthology.orchestra.run.smithi052.stdout: "debug", 2024-05-27T07:36:52.896 INFO:teuthology.orchestra.run.smithi052.stdout: "error", 2024-05-27T07:36:52.896 INFO:teuthology.orchestra.run.smithi052.stdout: "info", 2024-05-27T07:36:52.897 INFO:teuthology.orchestra.run.smithi052.stdout: "warning" 2024-05-27T07:36:52.897 INFO:teuthology.orchestra.run.smithi052.stdout: ], 2024-05-27T07:36:52.897 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-05-27T07:36:52.897 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-05-27T07:36:52.897 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-05-27T07:36:52.897 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-05-27T07:36:52.897 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-05-27T07:36:52.897 INFO:teuthology.orchestra.run.smithi052.stdout: "log_refresh_metadata": { 2024-05-27T07:36:52.897 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "log_refresh_metadata", 2024-05-27T07:36:52.897 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "bool", 2024-05-27T07:36:52.897 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-05-27T07:36:52.897 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 0, 2024-05-27T07:36:52.897 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "False", 2024-05-27T07:36:52.897 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-05-27T07:36:52.898 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-05-27T07:36:52.898 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-05-27T07:36:52.898 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level", 2024-05-27T07:36:52.898 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-05-27T07:36:52.898 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-05-27T07:36:52.898 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-05-27T07:36:52.898 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-05-27T07:36:52.898 INFO:teuthology.orchestra.run.smithi052.stdout: "log_to_cluster": { 2024-05-27T07:36:52.898 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "log_to_cluster", 2024-05-27T07:36:52.898 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "bool", 2024-05-27T07:36:52.898 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-05-27T07:36:52.898 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 0, 2024-05-27T07:36:52.898 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "True", 2024-05-27T07:36:52.899 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-05-27T07:36:52.899 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-05-27T07:36:52.899 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-05-27T07:36:52.899 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "log to the \"cephadm\" cluster log channel\"", 2024-05-27T07:36:52.899 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-05-27T07:36:52.899 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-05-27T07:36:52.899 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-05-27T07:36:52.899 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-05-27T07:36:52.899 INFO:teuthology.orchestra.run.smithi052.stdout: "log_to_cluster_level": { 2024-05-27T07:36:52.899 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "log_to_cluster_level", 2024-05-27T07:36:52.899 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "str", 2024-05-27T07:36:52.899 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-05-27T07:36:52.899 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 1, 2024-05-27T07:36:52.900 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "info", 2024-05-27T07:36:52.900 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-05-27T07:36:52.900 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-05-27T07:36:52.900 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [ 2024-05-27T07:36:52.900 INFO:teuthology.orchestra.run.smithi052.stdout: "", 2024-05-27T07:36:52.900 INFO:teuthology.orchestra.run.smithi052.stdout: "critical", 2024-05-27T07:36:52.900 INFO:teuthology.orchestra.run.smithi052.stdout: "debug", 2024-05-27T07:36:52.900 INFO:teuthology.orchestra.run.smithi052.stdout: "error", 2024-05-27T07:36:52.900 INFO:teuthology.orchestra.run.smithi052.stdout: "info", 2024-05-27T07:36:52.900 INFO:teuthology.orchestra.run.smithi052.stdout: "warning" 2024-05-27T07:36:52.900 INFO:teuthology.orchestra.run.smithi052.stdout: ], 2024-05-27T07:36:52.900 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-05-27T07:36:52.901 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-05-27T07:36:52.901 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-05-27T07:36:52.901 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-05-27T07:36:52.901 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-05-27T07:36:52.901 INFO:teuthology.orchestra.run.smithi052.stdout: "log_to_file": { 2024-05-27T07:36:52.901 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "log_to_file", 2024-05-27T07:36:52.901 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "bool", 2024-05-27T07:36:52.901 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-05-27T07:36:52.901 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 1, 2024-05-27T07:36:52.901 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "False", 2024-05-27T07:36:52.901 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-05-27T07:36:52.901 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-05-27T07:36:52.901 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-05-27T07:36:52.901 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-05-27T07:36:52.902 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-05-27T07:36:52.902 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-05-27T07:36:52.902 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-05-27T07:36:52.902 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-05-27T07:36:52.902 INFO:teuthology.orchestra.run.smithi052.stdout: "manage_etc_ceph_ceph_conf": { 2024-05-27T07:36:52.902 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "manage_etc_ceph_ceph_conf", 2024-05-27T07:36:52.902 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "bool", 2024-05-27T07:36:52.902 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-05-27T07:36:52.902 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 0, 2024-05-27T07:36:52.902 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "False", 2024-05-27T07:36:52.902 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-05-27T07:36:52.902 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-05-27T07:36:52.903 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-05-27T07:36:52.903 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "Manage and own /etc/ceph/ceph.conf on the hosts.", 2024-05-27T07:36:52.903 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-05-27T07:36:52.903 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-05-27T07:36:52.903 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-05-27T07:36:52.903 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-05-27T07:36:52.903 INFO:teuthology.orchestra.run.smithi052.stdout: "manage_etc_ceph_ceph_conf_hosts": { 2024-05-27T07:36:52.903 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "manage_etc_ceph_ceph_conf_hosts", 2024-05-27T07:36:52.903 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "str", 2024-05-27T07:36:52.903 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-05-27T07:36:52.903 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 0, 2024-05-27T07:36:52.903 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "*", 2024-05-27T07:36:52.903 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-05-27T07:36:52.904 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-05-27T07:36:52.904 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-05-27T07:36:52.904 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf", 2024-05-27T07:36:52.904 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-05-27T07:36:52.904 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-05-27T07:36:52.904 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-05-27T07:36:52.904 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-05-27T07:36:52.904 INFO:teuthology.orchestra.run.smithi052.stdout: "max_count_per_host": { 2024-05-27T07:36:52.904 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "max_count_per_host", 2024-05-27T07:36:52.904 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "int", 2024-05-27T07:36:52.904 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-05-27T07:36:52.904 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 0, 2024-05-27T07:36:52.904 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "10", 2024-05-27T07:36:52.904 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-05-27T07:36:52.905 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-05-27T07:36:52.905 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-05-27T07:36:52.905 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "max number of daemons per service per host", 2024-05-27T07:36:52.905 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-05-27T07:36:52.905 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-05-27T07:36:52.905 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-05-27T07:36:52.905 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-05-27T07:36:52.905 INFO:teuthology.orchestra.run.smithi052.stdout: "max_osd_draining_count": { 2024-05-27T07:36:52.905 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "max_osd_draining_count", 2024-05-27T07:36:52.905 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "int", 2024-05-27T07:36:52.906 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-05-27T07:36:52.906 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 0, 2024-05-27T07:36:52.906 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "10", 2024-05-27T07:36:52.906 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-05-27T07:36:52.906 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-05-27T07:36:52.906 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-05-27T07:36:52.906 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "max number of osds that will be drained simultaneously when osds are removed", 2024-05-27T07:36:52.906 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-05-27T07:36:52.906 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-05-27T07:36:52.906 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-05-27T07:36:52.906 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-05-27T07:36:52.906 INFO:teuthology.orchestra.run.smithi052.stdout: "migration_current": { 2024-05-27T07:36:52.907 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "migration_current", 2024-05-27T07:36:52.907 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "int", 2024-05-27T07:36:52.907 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-05-27T07:36:52.907 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 0, 2024-05-27T07:36:52.907 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "", 2024-05-27T07:36:52.907 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-05-27T07:36:52.907 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-05-27T07:36:52.907 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-05-27T07:36:52.907 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "internal - do not modify", 2024-05-27T07:36:52.907 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-05-27T07:36:52.907 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-05-27T07:36:52.907 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-05-27T07:36:52.907 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-05-27T07:36:52.908 INFO:teuthology.orchestra.run.smithi052.stdout: "mode": { 2024-05-27T07:36:52.908 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "mode", 2024-05-27T07:36:52.908 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "str", 2024-05-27T07:36:52.908 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-05-27T07:36:52.908 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 0, 2024-05-27T07:36:52.908 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "root", 2024-05-27T07:36:52.908 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-05-27T07:36:52.908 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-05-27T07:36:52.908 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [ 2024-05-27T07:36:52.908 INFO:teuthology.orchestra.run.smithi052.stdout: "cephadm-package", 2024-05-27T07:36:52.908 INFO:teuthology.orchestra.run.smithi052.stdout: "root" 2024-05-27T07:36:52.908 INFO:teuthology.orchestra.run.smithi052.stdout: ], 2024-05-27T07:36:52.908 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "mode for remote execution of cephadm", 2024-05-27T07:36:52.908 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-05-27T07:36:52.909 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-05-27T07:36:52.909 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-05-27T07:36:52.909 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-05-27T07:36:52.909 INFO:teuthology.orchestra.run.smithi052.stdout: "oob_default_addr": { 2024-05-27T07:36:52.909 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "oob_default_addr", 2024-05-27T07:36:52.909 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "str", 2024-05-27T07:36:52.909 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-05-27T07:36:52.909 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 0, 2024-05-27T07:36:52.909 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "169.254.1.1", 2024-05-27T07:36:52.909 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-05-27T07:36:52.909 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-05-27T07:36:52.909 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-05-27T07:36:52.909 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "Default address for RedFish API (oob management).", 2024-05-27T07:36:52.910 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-05-27T07:36:52.910 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-05-27T07:36:52.910 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-05-27T07:36:52.910 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-05-27T07:36:52.910 INFO:teuthology.orchestra.run.smithi052.stdout: "prometheus_alerts_path": { 2024-05-27T07:36:52.910 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "prometheus_alerts_path", 2024-05-27T07:36:52.910 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "str", 2024-05-27T07:36:52.910 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-05-27T07:36:52.910 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 0, 2024-05-27T07:36:52.910 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "/etc/prometheus/ceph/ceph_default_alerts.yml", 2024-05-27T07:36:52.910 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-05-27T07:36:52.910 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-05-27T07:36:52.911 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-05-27T07:36:52.911 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "location of alerts to include in prometheus deployments", 2024-05-27T07:36:52.911 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-05-27T07:36:52.911 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-05-27T07:36:52.911 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-05-27T07:36:52.911 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-05-27T07:36:52.911 INFO:teuthology.orchestra.run.smithi052.stdout: "registry_insecure": { 2024-05-27T07:36:52.911 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "registry_insecure", 2024-05-27T07:36:52.911 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "bool", 2024-05-27T07:36:52.911 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-05-27T07:36:52.911 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 0, 2024-05-27T07:36:52.911 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "False", 2024-05-27T07:36:52.911 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-05-27T07:36:52.912 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-05-27T07:36:52.912 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-05-27T07:36:52.912 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "Registry is to be considered insecure (no TLS available). Only for development purposes.", 2024-05-27T07:36:52.912 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-05-27T07:36:52.912 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-05-27T07:36:52.912 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-05-27T07:36:52.912 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-05-27T07:36:52.912 INFO:teuthology.orchestra.run.smithi052.stdout: "registry_password": { 2024-05-27T07:36:52.912 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "registry_password", 2024-05-27T07:36:52.912 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "str", 2024-05-27T07:36:52.912 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-05-27T07:36:52.912 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 0, 2024-05-27T07:36:52.913 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "", 2024-05-27T07:36:52.913 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-05-27T07:36:52.913 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-05-27T07:36:52.913 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-05-27T07:36:52.913 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "Custom repository password. Only used for logging into a registry.", 2024-05-27T07:36:52.913 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-05-27T07:36:52.913 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-05-27T07:36:52.913 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-05-27T07:36:52.913 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-05-27T07:36:52.913 INFO:teuthology.orchestra.run.smithi052.stdout: "registry_url": { 2024-05-27T07:36:52.913 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "registry_url", 2024-05-27T07:36:52.913 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "str", 2024-05-27T07:36:52.913 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-05-27T07:36:52.914 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 0, 2024-05-27T07:36:52.914 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "", 2024-05-27T07:36:52.914 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-05-27T07:36:52.914 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-05-27T07:36:52.914 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-05-27T07:36:52.914 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "Registry url for login purposes. This is not the default registry", 2024-05-27T07:36:52.914 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-05-27T07:36:52.914 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-05-27T07:36:52.914 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-05-27T07:36:52.914 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-05-27T07:36:52.914 INFO:teuthology.orchestra.run.smithi052.stdout: "registry_username": { 2024-05-27T07:36:52.914 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "registry_username", 2024-05-27T07:36:52.914 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "str", 2024-05-27T07:36:52.915 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-05-27T07:36:52.915 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 0, 2024-05-27T07:36:52.915 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "", 2024-05-27T07:36:52.915 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-05-27T07:36:52.915 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-05-27T07:36:52.915 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-05-27T07:36:52.915 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "Custom repository username. Only used for logging into a registry.", 2024-05-27T07:36:52.915 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-05-27T07:36:52.915 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-05-27T07:36:52.915 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-05-27T07:36:52.915 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-05-27T07:36:52.915 INFO:teuthology.orchestra.run.smithi052.stdout: "secure_monitoring_stack": { 2024-05-27T07:36:52.915 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "secure_monitoring_stack", 2024-05-27T07:36:52.916 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "bool", 2024-05-27T07:36:52.916 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-05-27T07:36:52.916 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 0, 2024-05-27T07:36:52.916 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "False", 2024-05-27T07:36:52.916 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-05-27T07:36:52.916 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-05-27T07:36:52.916 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-05-27T07:36:52.916 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "Enable TLS security for all the monitoring stack daemons", 2024-05-27T07:36:52.916 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-05-27T07:36:52.916 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-05-27T07:36:52.916 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-05-27T07:36:52.916 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-05-27T07:36:52.916 INFO:teuthology.orchestra.run.smithi052.stdout: "service_discovery_port": { 2024-05-27T07:36:52.917 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "service_discovery_port", 2024-05-27T07:36:52.917 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "int", 2024-05-27T07:36:52.917 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-05-27T07:36:52.917 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 0, 2024-05-27T07:36:52.917 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "8765", 2024-05-27T07:36:52.917 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-05-27T07:36:52.917 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-05-27T07:36:52.917 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-05-27T07:36:52.917 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "cephadm service discovery port", 2024-05-27T07:36:52.917 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-05-27T07:36:52.917 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-05-27T07:36:52.917 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-05-27T07:36:52.917 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-05-27T07:36:52.918 INFO:teuthology.orchestra.run.smithi052.stdout: "sqlite3_killpoint": { 2024-05-27T07:36:52.918 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "sqlite3_killpoint", 2024-05-27T07:36:52.918 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "int", 2024-05-27T07:36:52.918 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "dev", 2024-05-27T07:36:52.918 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 1, 2024-05-27T07:36:52.918 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "0", 2024-05-27T07:36:52.918 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-05-27T07:36:52.918 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-05-27T07:36:52.918 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-05-27T07:36:52.918 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-05-27T07:36:52.918 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-05-27T07:36:52.918 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-05-27T07:36:52.918 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-05-27T07:36:52.919 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-05-27T07:36:52.919 INFO:teuthology.orchestra.run.smithi052.stdout: "ssh_config_file": { 2024-05-27T07:36:52.919 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "ssh_config_file", 2024-05-27T07:36:52.919 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "str", 2024-05-27T07:36:52.919 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-05-27T07:36:52.919 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 0, 2024-05-27T07:36:52.919 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "", 2024-05-27T07:36:52.919 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-05-27T07:36:52.919 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-05-27T07:36:52.919 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-05-27T07:36:52.919 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "customized SSH config file to connect to managed hosts", 2024-05-27T07:36:52.919 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-05-27T07:36:52.919 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-05-27T07:36:52.920 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-05-27T07:36:52.920 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-05-27T07:36:52.920 INFO:teuthology.orchestra.run.smithi052.stdout: "use_agent": { 2024-05-27T07:36:52.920 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "use_agent", 2024-05-27T07:36:52.920 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "bool", 2024-05-27T07:36:52.920 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-05-27T07:36:52.920 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 0, 2024-05-27T07:36:52.920 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "False", 2024-05-27T07:36:52.920 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-05-27T07:36:52.920 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-05-27T07:36:52.920 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-05-27T07:36:52.920 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "Use cephadm agent on each host to gather and send metadata", 2024-05-27T07:36:52.920 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-05-27T07:36:52.921 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-05-27T07:36:52.921 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-05-27T07:36:52.921 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-05-27T07:36:52.921 INFO:teuthology.orchestra.run.smithi052.stdout: "use_repo_digest": { 2024-05-27T07:36:52.921 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "use_repo_digest", 2024-05-27T07:36:52.921 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "bool", 2024-05-27T07:36:52.921 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-05-27T07:36:52.921 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 0, 2024-05-27T07:36:52.921 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "True", 2024-05-27T07:36:52.921 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-05-27T07:36:52.921 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-05-27T07:36:52.921 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-05-27T07:36:52.921 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "Automatically convert image tags to image digest. Make sure all daemons use the same image", 2024-05-27T07:36:52.922 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-05-27T07:36:52.922 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-05-27T07:36:52.922 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-05-27T07:36:52.922 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-05-27T07:36:52.922 INFO:teuthology.orchestra.run.smithi052.stdout: "warn_on_failed_host_check": { 2024-05-27T07:36:52.922 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "warn_on_failed_host_check", 2024-05-27T07:36:52.922 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "bool", 2024-05-27T07:36:52.922 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-05-27T07:36:52.922 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 0, 2024-05-27T07:36:52.922 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "True", 2024-05-27T07:36:52.922 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-05-27T07:36:52.922 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-05-27T07:36:52.923 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-05-27T07:36:52.923 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "raise a health warning if the host check fails", 2024-05-27T07:36:52.923 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-05-27T07:36:52.923 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-05-27T07:36:52.923 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-05-27T07:36:52.923 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-05-27T07:36:52.923 INFO:teuthology.orchestra.run.smithi052.stdout: "warn_on_stray_daemons": { 2024-05-27T07:36:52.923 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "warn_on_stray_daemons", 2024-05-27T07:36:52.923 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "bool", 2024-05-27T07:36:52.923 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-05-27T07:36:52.923 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 0, 2024-05-27T07:36:52.923 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "True", 2024-05-27T07:36:52.923 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-05-27T07:36:52.924 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-05-27T07:36:52.924 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-05-27T07:36:52.924 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "raise a health warning if daemons are detected that are not managed by cephadm", 2024-05-27T07:36:52.924 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-05-27T07:36:52.924 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-05-27T07:36:52.924 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-05-27T07:36:52.924 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-05-27T07:36:52.924 INFO:teuthology.orchestra.run.smithi052.stdout: "warn_on_stray_hosts": { 2024-05-27T07:36:52.924 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "warn_on_stray_hosts", 2024-05-27T07:36:52.924 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "bool", 2024-05-27T07:36:52.924 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-05-27T07:36:52.924 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 0, 2024-05-27T07:36:52.925 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "True", 2024-05-27T07:36:52.925 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-05-27T07:36:52.925 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-05-27T07:36:52.925 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-05-27T07:36:52.925 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "raise a health warning if daemons are detected on a host that is not managed by cephadm", 2024-05-27T07:36:52.925 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-05-27T07:36:52.925 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-05-27T07:36:52.925 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-05-27T07:36:52.925 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-05-27T07:36:52.925 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-05-27T07:36:52.925 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-05-27T07:36:52.925 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-05-27T07:36:52.925 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "crash", 2024-05-27T07:36:52.926 INFO:teuthology.orchestra.run.smithi052.stdout: "can_run": true, 2024-05-27T07:36:52.926 INFO:teuthology.orchestra.run.smithi052.stdout: "error_string": "", 2024-05-27T07:36:52.926 INFO:teuthology.orchestra.run.smithi052.stdout: "module_options": { 2024-05-27T07:36:52.926 INFO:teuthology.orchestra.run.smithi052.stdout: "log_level": { 2024-05-27T07:36:52.926 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "log_level", 2024-05-27T07:36:52.926 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "str", 2024-05-27T07:36:52.926 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-05-27T07:36:52.926 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 1, 2024-05-27T07:36:52.926 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "", 2024-05-27T07:36:52.926 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-05-27T07:36:52.926 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-05-27T07:36:52.926 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [ 2024-05-27T07:36:52.927 INFO:teuthology.orchestra.run.smithi052.stdout: "", 2024-05-27T07:36:52.927 INFO:teuthology.orchestra.run.smithi052.stdout: "critical", 2024-05-27T07:36:52.927 INFO:teuthology.orchestra.run.smithi052.stdout: "debug", 2024-05-27T07:36:52.927 INFO:teuthology.orchestra.run.smithi052.stdout: "error", 2024-05-27T07:36:52.927 INFO:teuthology.orchestra.run.smithi052.stdout: "info", 2024-05-27T07:36:52.927 INFO:teuthology.orchestra.run.smithi052.stdout: "warning" 2024-05-27T07:36:52.927 INFO:teuthology.orchestra.run.smithi052.stdout: ], 2024-05-27T07:36:52.927 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-05-27T07:36:52.927 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-05-27T07:36:52.927 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-05-27T07:36:52.927 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-05-27T07:36:52.927 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-05-27T07:36:52.927 INFO:teuthology.orchestra.run.smithi052.stdout: "log_to_cluster": { 2024-05-27T07:36:52.928 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "log_to_cluster", 2024-05-27T07:36:52.928 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "bool", 2024-05-27T07:36:52.928 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-05-27T07:36:52.928 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 1, 2024-05-27T07:36:52.928 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "False", 2024-05-27T07:36:52.928 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-05-27T07:36:52.928 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-05-27T07:36:52.928 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-05-27T07:36:52.928 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-05-27T07:36:52.928 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-05-27T07:36:52.928 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-05-27T07:36:52.928 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-05-27T07:36:52.928 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-05-27T07:36:52.929 INFO:teuthology.orchestra.run.smithi052.stdout: "log_to_cluster_level": { 2024-05-27T07:36:52.929 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "log_to_cluster_level", 2024-05-27T07:36:52.929 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "str", 2024-05-27T07:36:52.929 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-05-27T07:36:52.929 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 1, 2024-05-27T07:36:52.929 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "info", 2024-05-27T07:36:52.929 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-05-27T07:36:52.929 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-05-27T07:36:52.929 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [ 2024-05-27T07:36:52.929 INFO:teuthology.orchestra.run.smithi052.stdout: "", 2024-05-27T07:36:52.929 INFO:teuthology.orchestra.run.smithi052.stdout: "critical", 2024-05-27T07:36:52.929 INFO:teuthology.orchestra.run.smithi052.stdout: "debug", 2024-05-27T07:36:52.929 INFO:teuthology.orchestra.run.smithi052.stdout: "error", 2024-05-27T07:36:52.930 INFO:teuthology.orchestra.run.smithi052.stdout: "info", 2024-05-27T07:36:52.930 INFO:teuthology.orchestra.run.smithi052.stdout: "warning" 2024-05-27T07:36:52.930 INFO:teuthology.orchestra.run.smithi052.stdout: ], 2024-05-27T07:36:52.930 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-05-27T07:36:52.930 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-05-27T07:36:52.930 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-05-27T07:36:52.930 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-05-27T07:36:52.930 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-05-27T07:36:52.930 INFO:teuthology.orchestra.run.smithi052.stdout: "log_to_file": { 2024-05-27T07:36:52.930 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "log_to_file", 2024-05-27T07:36:52.930 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "bool", 2024-05-27T07:36:52.930 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-05-27T07:36:52.930 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 1, 2024-05-27T07:36:52.931 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "False", 2024-05-27T07:36:52.931 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-05-27T07:36:52.931 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-05-27T07:36:52.931 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-05-27T07:36:52.931 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-05-27T07:36:52.931 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-05-27T07:36:52.931 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-05-27T07:36:52.931 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-05-27T07:36:52.931 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-05-27T07:36:52.931 INFO:teuthology.orchestra.run.smithi052.stdout: "retain_interval": { 2024-05-27T07:36:52.931 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "retain_interval", 2024-05-27T07:36:52.931 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "secs", 2024-05-27T07:36:52.931 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-05-27T07:36:52.932 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 1, 2024-05-27T07:36:52.932 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "31536000", 2024-05-27T07:36:52.932 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-05-27T07:36:52.932 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-05-27T07:36:52.932 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-05-27T07:36:52.932 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "how long to retain crashes before pruning them", 2024-05-27T07:36:52.932 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-05-27T07:36:52.932 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-05-27T07:36:52.932 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-05-27T07:36:52.932 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-05-27T07:36:52.932 INFO:teuthology.orchestra.run.smithi052.stdout: "sqlite3_killpoint": { 2024-05-27T07:36:52.932 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "sqlite3_killpoint", 2024-05-27T07:36:52.932 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "int", 2024-05-27T07:36:52.933 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "dev", 2024-05-27T07:36:52.933 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 1, 2024-05-27T07:36:52.933 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "0", 2024-05-27T07:36:52.933 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-05-27T07:36:52.933 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-05-27T07:36:52.933 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-05-27T07:36:52.933 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-05-27T07:36:52.933 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-05-27T07:36:52.933 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-05-27T07:36:52.933 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-05-27T07:36:52.933 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-05-27T07:36:52.933 INFO:teuthology.orchestra.run.smithi052.stdout: "warn_recent_interval": { 2024-05-27T07:36:52.934 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "warn_recent_interval", 2024-05-27T07:36:52.934 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "secs", 2024-05-27T07:36:52.934 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-05-27T07:36:52.934 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 1, 2024-05-27T07:36:52.934 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "1209600", 2024-05-27T07:36:52.934 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-05-27T07:36:52.934 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-05-27T07:36:52.934 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-05-27T07:36:52.934 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "time interval in which to warn about recent crashes", 2024-05-27T07:36:52.934 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-05-27T07:36:52.934 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-05-27T07:36:52.934 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-05-27T07:36:52.934 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-05-27T07:36:52.935 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-05-27T07:36:52.935 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-05-27T07:36:52.935 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-05-27T07:36:52.935 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "dashboard", 2024-05-27T07:36:52.935 INFO:teuthology.orchestra.run.smithi052.stdout: "can_run": true, 2024-05-27T07:36:52.935 INFO:teuthology.orchestra.run.smithi052.stdout: "error_string": "", 2024-05-27T07:36:52.935 INFO:teuthology.orchestra.run.smithi052.stdout: "module_options": { 2024-05-27T07:36:52.935 INFO:teuthology.orchestra.run.smithi052.stdout: "ACCOUNT_LOCKOUT_ATTEMPTS": { 2024-05-27T07:36:52.935 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "ACCOUNT_LOCKOUT_ATTEMPTS", 2024-05-27T07:36:52.935 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "int", 2024-05-27T07:36:52.935 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-05-27T07:36:52.935 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 0, 2024-05-27T07:36:52.935 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "10", 2024-05-27T07:36:52.935 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-05-27T07:36:52.936 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-05-27T07:36:52.936 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-05-27T07:36:52.936 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-05-27T07:36:52.936 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-05-27T07:36:52.936 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-05-27T07:36:52.936 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-05-27T07:36:52.936 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-05-27T07:36:52.936 INFO:teuthology.orchestra.run.smithi052.stdout: "ALERTMANAGER_API_HOST": { 2024-05-27T07:36:52.936 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "ALERTMANAGER_API_HOST", 2024-05-27T07:36:52.936 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "str", 2024-05-27T07:36:52.936 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-05-27T07:36:52.936 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 0, 2024-05-27T07:36:52.936 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "", 2024-05-27T07:36:52.937 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-05-27T07:36:52.937 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-05-27T07:36:52.937 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-05-27T07:36:52.937 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-05-27T07:36:52.937 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-05-27T07:36:52.937 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-05-27T07:36:52.937 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-05-27T07:36:52.937 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-05-27T07:36:52.937 INFO:teuthology.orchestra.run.smithi052.stdout: "ALERTMANAGER_API_SSL_VERIFY": { 2024-05-27T07:36:52.937 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "ALERTMANAGER_API_SSL_VERIFY", 2024-05-27T07:36:52.937 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "bool", 2024-05-27T07:36:52.937 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-05-27T07:36:52.937 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 0, 2024-05-27T07:36:52.938 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "True", 2024-05-27T07:36:52.938 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-05-27T07:36:52.938 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-05-27T07:36:52.938 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-05-27T07:36:52.938 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-05-27T07:36:52.938 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-05-27T07:36:52.938 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-05-27T07:36:52.938 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-05-27T07:36:52.938 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-05-27T07:36:52.938 INFO:teuthology.orchestra.run.smithi052.stdout: "AUDIT_API_ENABLED": { 2024-05-27T07:36:52.938 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "AUDIT_API_ENABLED", 2024-05-27T07:36:52.938 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "bool", 2024-05-27T07:36:52.938 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-05-27T07:36:52.939 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 0, 2024-05-27T07:36:52.939 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "False", 2024-05-27T07:36:52.939 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-05-27T07:36:52.939 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-05-27T07:36:52.939 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-05-27T07:36:52.939 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-05-27T07:36:52.939 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-05-27T07:36:52.939 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-05-27T07:36:52.939 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-05-27T07:36:52.939 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-05-27T07:36:52.939 INFO:teuthology.orchestra.run.smithi052.stdout: "AUDIT_API_LOG_PAYLOAD": { 2024-05-27T07:36:52.939 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "AUDIT_API_LOG_PAYLOAD", 2024-05-27T07:36:52.940 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "bool", 2024-05-27T07:36:52.940 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-05-27T07:36:52.940 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 0, 2024-05-27T07:36:52.940 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "True", 2024-05-27T07:36:52.940 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-05-27T07:36:52.940 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-05-27T07:36:52.940 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-05-27T07:36:52.940 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-05-27T07:36:52.940 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-05-27T07:36:52.940 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-05-27T07:36:52.940 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-05-27T07:36:52.940 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-05-27T07:36:52.940 INFO:teuthology.orchestra.run.smithi052.stdout: "ENABLE_BROWSABLE_API": { 2024-05-27T07:36:52.940 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "ENABLE_BROWSABLE_API", 2024-05-27T07:36:52.941 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "bool", 2024-05-27T07:36:52.941 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-05-27T07:36:52.941 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 0, 2024-05-27T07:36:52.941 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "True", 2024-05-27T07:36:52.941 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-05-27T07:36:52.941 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-05-27T07:36:52.941 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-05-27T07:36:52.941 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-05-27T07:36:52.941 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-05-27T07:36:52.941 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-05-27T07:36:52.941 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-05-27T07:36:52.941 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-05-27T07:36:52.942 INFO:teuthology.orchestra.run.smithi052.stdout: "FEATURE_TOGGLE_CEPHFS": { 2024-05-27T07:36:52.942 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "FEATURE_TOGGLE_CEPHFS", 2024-05-27T07:36:52.942 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "bool", 2024-05-27T07:36:52.942 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-05-27T07:36:52.942 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 0, 2024-05-27T07:36:52.942 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "True", 2024-05-27T07:36:52.942 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-05-27T07:36:52.942 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-05-27T07:36:52.942 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-05-27T07:36:52.942 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-05-27T07:36:52.942 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-05-27T07:36:52.942 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-05-27T07:36:52.942 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-05-27T07:36:52.943 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-05-27T07:36:52.943 INFO:teuthology.orchestra.run.smithi052.stdout: "FEATURE_TOGGLE_DASHBOARD": { 2024-05-27T07:36:52.943 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "FEATURE_TOGGLE_DASHBOARD", 2024-05-27T07:36:52.943 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "bool", 2024-05-27T07:36:52.943 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-05-27T07:36:52.943 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 0, 2024-05-27T07:36:52.943 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "True", 2024-05-27T07:36:52.943 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-05-27T07:36:52.943 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-05-27T07:36:52.943 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-05-27T07:36:52.943 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-05-27T07:36:52.943 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-05-27T07:36:52.943 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-05-27T07:36:52.944 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-05-27T07:36:52.944 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-05-27T07:36:52.944 INFO:teuthology.orchestra.run.smithi052.stdout: "FEATURE_TOGGLE_ISCSI": { 2024-05-27T07:36:52.944 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "FEATURE_TOGGLE_ISCSI", 2024-05-27T07:36:52.944 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "bool", 2024-05-27T07:36:52.944 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-05-27T07:36:52.944 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 0, 2024-05-27T07:36:52.944 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "True", 2024-05-27T07:36:52.944 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-05-27T07:36:52.944 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-05-27T07:36:52.944 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-05-27T07:36:52.944 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-05-27T07:36:52.945 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-05-27T07:36:52.945 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-05-27T07:36:52.945 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-05-27T07:36:52.945 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-05-27T07:36:52.945 INFO:teuthology.orchestra.run.smithi052.stdout: "FEATURE_TOGGLE_MIRRORING": { 2024-05-27T07:36:52.945 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "FEATURE_TOGGLE_MIRRORING", 2024-05-27T07:36:52.945 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "bool", 2024-05-27T07:36:52.945 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-05-27T07:36:52.945 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 0, 2024-05-27T07:36:52.945 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "True", 2024-05-27T07:36:52.945 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-05-27T07:36:52.945 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-05-27T07:36:52.945 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-05-27T07:36:52.946 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-05-27T07:36:52.946 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-05-27T07:36:52.946 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-05-27T07:36:52.946 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-05-27T07:36:52.946 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-05-27T07:36:52.946 INFO:teuthology.orchestra.run.smithi052.stdout: "FEATURE_TOGGLE_NFS": { 2024-05-27T07:36:52.946 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "FEATURE_TOGGLE_NFS", 2024-05-27T07:36:52.946 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "bool", 2024-05-27T07:36:52.946 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-05-27T07:36:52.946 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 0, 2024-05-27T07:36:52.946 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "True", 2024-05-27T07:36:52.946 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-05-27T07:36:52.947 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-05-27T07:36:52.947 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-05-27T07:36:52.947 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-05-27T07:36:52.947 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-05-27T07:36:52.947 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-05-27T07:36:52.947 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-05-27T07:36:52.947 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-05-27T07:36:52.947 INFO:teuthology.orchestra.run.smithi052.stdout: "FEATURE_TOGGLE_RBD": { 2024-05-27T07:36:52.947 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "FEATURE_TOGGLE_RBD", 2024-05-27T07:36:52.947 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "bool", 2024-05-27T07:36:52.947 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-05-27T07:36:52.947 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 0, 2024-05-27T07:36:52.947 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "True", 2024-05-27T07:36:52.948 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-05-27T07:36:52.948 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-05-27T07:36:52.948 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-05-27T07:36:52.948 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-05-27T07:36:52.948 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-05-27T07:36:52.948 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-05-27T07:36:52.948 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-05-27T07:36:52.948 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-05-27T07:36:52.948 INFO:teuthology.orchestra.run.smithi052.stdout: "FEATURE_TOGGLE_RGW": { 2024-05-27T07:36:52.948 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "FEATURE_TOGGLE_RGW", 2024-05-27T07:36:52.948 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "bool", 2024-05-27T07:36:52.948 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-05-27T07:36:52.948 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 0, 2024-05-27T07:36:52.949 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "True", 2024-05-27T07:36:52.949 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-05-27T07:36:52.949 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-05-27T07:36:52.949 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-05-27T07:36:52.949 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-05-27T07:36:52.949 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-05-27T07:36:52.949 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-05-27T07:36:52.949 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-05-27T07:36:52.949 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-05-27T07:36:52.949 INFO:teuthology.orchestra.run.smithi052.stdout: "GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE": { 2024-05-27T07:36:52.949 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE", 2024-05-27T07:36:52.949 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "str", 2024-05-27T07:36:52.950 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-05-27T07:36:52.950 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 0, 2024-05-27T07:36:52.950 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "", 2024-05-27T07:36:52.950 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-05-27T07:36:52.950 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-05-27T07:36:52.950 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-05-27T07:36:52.950 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-05-27T07:36:52.950 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-05-27T07:36:52.950 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-05-27T07:36:52.950 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-05-27T07:36:52.950 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-05-27T07:36:52.950 INFO:teuthology.orchestra.run.smithi052.stdout: "GRAFANA_API_PASSWORD": { 2024-05-27T07:36:52.950 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "GRAFANA_API_PASSWORD", 2024-05-27T07:36:52.951 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "str", 2024-05-27T07:36:52.951 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-05-27T07:36:52.951 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 0, 2024-05-27T07:36:52.951 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "admin", 2024-05-27T07:36:52.951 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-05-27T07:36:52.951 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-05-27T07:36:52.951 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-05-27T07:36:52.951 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-05-27T07:36:52.951 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-05-27T07:36:52.951 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-05-27T07:36:52.951 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-05-27T07:36:52.951 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-05-27T07:36:52.951 INFO:teuthology.orchestra.run.smithi052.stdout: "GRAFANA_API_SSL_VERIFY": { 2024-05-27T07:36:52.951 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "GRAFANA_API_SSL_VERIFY", 2024-05-27T07:36:52.952 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "bool", 2024-05-27T07:36:52.952 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-05-27T07:36:52.952 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 0, 2024-05-27T07:36:52.952 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "True", 2024-05-27T07:36:52.952 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-05-27T07:36:52.952 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-05-27T07:36:52.952 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-05-27T07:36:52.952 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-05-27T07:36:52.952 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-05-27T07:36:52.952 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-05-27T07:36:52.952 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-05-27T07:36:52.952 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-05-27T07:36:52.952 INFO:teuthology.orchestra.run.smithi052.stdout: "GRAFANA_API_URL": { 2024-05-27T07:36:52.953 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "GRAFANA_API_URL", 2024-05-27T07:36:52.953 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "str", 2024-05-27T07:36:52.953 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-05-27T07:36:52.953 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 0, 2024-05-27T07:36:52.953 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "", 2024-05-27T07:36:52.953 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-05-27T07:36:52.953 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-05-27T07:36:52.953 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-05-27T07:36:52.953 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-05-27T07:36:52.953 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-05-27T07:36:52.953 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-05-27T07:36:52.953 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-05-27T07:36:52.953 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-05-27T07:36:52.954 INFO:teuthology.orchestra.run.smithi052.stdout: "GRAFANA_API_USERNAME": { 2024-05-27T07:36:52.954 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "GRAFANA_API_USERNAME", 2024-05-27T07:36:52.954 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "str", 2024-05-27T07:36:52.954 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-05-27T07:36:52.954 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 0, 2024-05-27T07:36:52.954 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "admin", 2024-05-27T07:36:52.954 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-05-27T07:36:52.954 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-05-27T07:36:52.954 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-05-27T07:36:52.954 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-05-27T07:36:52.954 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-05-27T07:36:52.954 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-05-27T07:36:52.954 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-05-27T07:36:52.955 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-05-27T07:36:52.955 INFO:teuthology.orchestra.run.smithi052.stdout: "GRAFANA_FRONTEND_API_URL": { 2024-05-27T07:36:52.955 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "GRAFANA_FRONTEND_API_URL", 2024-05-27T07:36:52.955 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "str", 2024-05-27T07:36:52.955 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-05-27T07:36:52.955 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 0, 2024-05-27T07:36:52.955 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "", 2024-05-27T07:36:52.955 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-05-27T07:36:52.955 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-05-27T07:36:52.955 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-05-27T07:36:52.955 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-05-27T07:36:52.955 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-05-27T07:36:52.955 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-05-27T07:36:52.956 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-05-27T07:36:52.956 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-05-27T07:36:52.956 INFO:teuthology.orchestra.run.smithi052.stdout: "GRAFANA_UPDATE_DASHBOARDS": { 2024-05-27T07:36:52.956 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "GRAFANA_UPDATE_DASHBOARDS", 2024-05-27T07:36:52.956 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "bool", 2024-05-27T07:36:52.956 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-05-27T07:36:52.956 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 0, 2024-05-27T07:36:52.956 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "False", 2024-05-27T07:36:52.956 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-05-27T07:36:52.956 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-05-27T07:36:52.956 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-05-27T07:36:52.956 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-05-27T07:36:52.956 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-05-27T07:36:52.957 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-05-27T07:36:52.957 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-05-27T07:36:52.957 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-05-27T07:36:52.957 INFO:teuthology.orchestra.run.smithi052.stdout: "ISCSI_API_SSL_VERIFICATION": { 2024-05-27T07:36:52.957 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "ISCSI_API_SSL_VERIFICATION", 2024-05-27T07:36:52.957 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "bool", 2024-05-27T07:36:52.957 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-05-27T07:36:52.957 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 0, 2024-05-27T07:36:52.957 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "True", 2024-05-27T07:36:52.957 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-05-27T07:36:52.957 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-05-27T07:36:52.957 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-05-27T07:36:52.958 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-05-27T07:36:52.958 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-05-27T07:36:52.958 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-05-27T07:36:52.958 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-05-27T07:36:52.958 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-05-27T07:36:52.958 INFO:teuthology.orchestra.run.smithi052.stdout: "ISSUE_TRACKER_API_KEY": { 2024-05-27T07:36:52.958 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "ISSUE_TRACKER_API_KEY", 2024-05-27T07:36:52.958 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "str", 2024-05-27T07:36:52.958 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-05-27T07:36:52.958 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 0, 2024-05-27T07:36:52.958 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "", 2024-05-27T07:36:52.958 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-05-27T07:36:52.959 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-05-27T07:36:52.959 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-05-27T07:36:52.959 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-05-27T07:36:52.959 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-05-27T07:36:52.959 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-05-27T07:36:52.959 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-05-27T07:36:52.959 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-05-27T07:36:52.959 INFO:teuthology.orchestra.run.smithi052.stdout: "MANAGED_BY_CLUSTERS": { 2024-05-27T07:36:52.959 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "MANAGED_BY_CLUSTERS", 2024-05-27T07:36:52.959 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "str", 2024-05-27T07:36:52.959 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-05-27T07:36:52.959 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 0, 2024-05-27T07:36:52.959 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "[]", 2024-05-27T07:36:52.960 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-05-27T07:36:52.960 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-05-27T07:36:52.960 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-05-27T07:36:52.960 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-05-27T07:36:52.960 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-05-27T07:36:52.960 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-05-27T07:36:52.960 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-05-27T07:36:52.960 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-05-27T07:36:52.960 INFO:teuthology.orchestra.run.smithi052.stdout: "MULTICLUSTER_CONFIG": { 2024-05-27T07:36:52.960 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "MULTICLUSTER_CONFIG", 2024-05-27T07:36:52.960 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "str", 2024-05-27T07:36:52.960 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-05-27T07:36:52.960 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 0, 2024-05-27T07:36:52.961 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "{}", 2024-05-27T07:36:52.961 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-05-27T07:36:52.961 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-05-27T07:36:52.961 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-05-27T07:36:52.961 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-05-27T07:36:52.961 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-05-27T07:36:52.961 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-05-27T07:36:52.961 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-05-27T07:36:52.961 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-05-27T07:36:52.961 INFO:teuthology.orchestra.run.smithi052.stdout: "PROMETHEUS_API_HOST": { 2024-05-27T07:36:52.961 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "PROMETHEUS_API_HOST", 2024-05-27T07:36:52.961 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "str", 2024-05-27T07:36:52.961 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-05-27T07:36:52.962 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 0, 2024-05-27T07:36:52.962 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "", 2024-05-27T07:36:52.962 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-05-27T07:36:52.962 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-05-27T07:36:52.962 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-05-27T07:36:52.962 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-05-27T07:36:52.962 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-05-27T07:36:52.962 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-05-27T07:36:52.962 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-05-27T07:36:52.962 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-05-27T07:36:52.962 INFO:teuthology.orchestra.run.smithi052.stdout: "PROMETHEUS_API_SSL_VERIFY": { 2024-05-27T07:36:52.962 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "PROMETHEUS_API_SSL_VERIFY", 2024-05-27T07:36:52.963 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "bool", 2024-05-27T07:36:52.963 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-05-27T07:36:52.963 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 0, 2024-05-27T07:36:52.963 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "True", 2024-05-27T07:36:52.963 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-05-27T07:36:52.963 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-05-27T07:36:52.963 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-05-27T07:36:52.963 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-05-27T07:36:52.963 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-05-27T07:36:52.963 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-05-27T07:36:52.963 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-05-27T07:36:52.963 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-05-27T07:36:52.963 INFO:teuthology.orchestra.run.smithi052.stdout: "PWD_POLICY_CHECK_COMPLEXITY_ENABLED": { 2024-05-27T07:36:52.964 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "PWD_POLICY_CHECK_COMPLEXITY_ENABLED", 2024-05-27T07:36:52.964 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "bool", 2024-05-27T07:36:52.964 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-05-27T07:36:52.964 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 0, 2024-05-27T07:36:52.964 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "False", 2024-05-27T07:36:52.964 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-05-27T07:36:52.964 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-05-27T07:36:52.964 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-05-27T07:36:52.964 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-05-27T07:36:52.964 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-05-27T07:36:52.964 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-05-27T07:36:52.964 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-05-27T07:36:52.964 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-05-27T07:36:52.965 INFO:teuthology.orchestra.run.smithi052.stdout: "PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED": { 2024-05-27T07:36:52.965 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED", 2024-05-27T07:36:52.965 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "bool", 2024-05-27T07:36:52.965 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-05-27T07:36:52.965 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 0, 2024-05-27T07:36:52.965 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "False", 2024-05-27T07:36:52.965 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-05-27T07:36:52.965 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-05-27T07:36:52.965 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-05-27T07:36:52.965 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-05-27T07:36:52.965 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-05-27T07:36:52.965 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-05-27T07:36:52.965 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-05-27T07:36:52.966 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-05-27T07:36:52.966 INFO:teuthology.orchestra.run.smithi052.stdout: "PWD_POLICY_CHECK_LENGTH_ENABLED": { 2024-05-27T07:36:52.966 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "PWD_POLICY_CHECK_LENGTH_ENABLED", 2024-05-27T07:36:52.966 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "bool", 2024-05-27T07:36:52.966 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-05-27T07:36:52.966 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 0, 2024-05-27T07:36:52.966 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "True", 2024-05-27T07:36:52.966 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-05-27T07:36:52.966 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-05-27T07:36:52.966 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-05-27T07:36:52.966 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-05-27T07:36:52.966 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-05-27T07:36:52.966 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-05-27T07:36:52.967 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-05-27T07:36:52.967 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-05-27T07:36:52.967 INFO:teuthology.orchestra.run.smithi052.stdout: "PWD_POLICY_CHECK_OLDPWD_ENABLED": { 2024-05-27T07:36:52.967 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "PWD_POLICY_CHECK_OLDPWD_ENABLED", 2024-05-27T07:36:52.967 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "bool", 2024-05-27T07:36:52.967 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-05-27T07:36:52.967 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 0, 2024-05-27T07:36:52.967 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "True", 2024-05-27T07:36:52.967 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-05-27T07:36:52.967 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-05-27T07:36:52.967 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-05-27T07:36:52.967 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-05-27T07:36:52.967 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-05-27T07:36:52.968 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-05-27T07:36:52.968 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-05-27T07:36:52.968 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-05-27T07:36:52.968 INFO:teuthology.orchestra.run.smithi052.stdout: "PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED": { 2024-05-27T07:36:52.968 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED", 2024-05-27T07:36:52.968 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "bool", 2024-05-27T07:36:52.968 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-05-27T07:36:52.968 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 0, 2024-05-27T07:36:52.968 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "False", 2024-05-27T07:36:52.968 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-05-27T07:36:52.968 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-05-27T07:36:52.968 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-05-27T07:36:52.968 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-05-27T07:36:52.969 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-05-27T07:36:52.969 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-05-27T07:36:52.969 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-05-27T07:36:52.969 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-05-27T07:36:52.969 INFO:teuthology.orchestra.run.smithi052.stdout: "PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED": { 2024-05-27T07:36:52.969 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED", 2024-05-27T07:36:52.969 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "bool", 2024-05-27T07:36:52.969 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-05-27T07:36:52.969 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 0, 2024-05-27T07:36:52.969 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "False", 2024-05-27T07:36:52.969 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-05-27T07:36:52.969 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-05-27T07:36:52.969 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-05-27T07:36:52.969 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-05-27T07:36:52.970 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-05-27T07:36:52.970 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-05-27T07:36:52.970 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-05-27T07:36:52.970 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-05-27T07:36:52.970 INFO:teuthology.orchestra.run.smithi052.stdout: "PWD_POLICY_CHECK_USERNAME_ENABLED": { 2024-05-27T07:36:52.970 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "PWD_POLICY_CHECK_USERNAME_ENABLED", 2024-05-27T07:36:52.970 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "bool", 2024-05-27T07:36:52.970 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-05-27T07:36:52.970 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 0, 2024-05-27T07:36:52.970 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "False", 2024-05-27T07:36:52.970 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-05-27T07:36:52.970 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-05-27T07:36:52.970 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-05-27T07:36:52.971 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-05-27T07:36:52.971 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-05-27T07:36:52.971 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-05-27T07:36:52.971 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-05-27T07:36:52.971 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-05-27T07:36:52.971 INFO:teuthology.orchestra.run.smithi052.stdout: "PWD_POLICY_ENABLED": { 2024-05-27T07:36:52.971 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "PWD_POLICY_ENABLED", 2024-05-27T07:36:52.971 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "bool", 2024-05-27T07:36:52.971 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-05-27T07:36:52.971 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 0, 2024-05-27T07:36:52.971 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "True", 2024-05-27T07:36:52.971 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-05-27T07:36:52.971 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-05-27T07:36:52.972 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-05-27T07:36:52.972 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-05-27T07:36:52.972 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-05-27T07:36:52.972 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-05-27T07:36:52.972 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-05-27T07:36:52.972 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-05-27T07:36:52.972 INFO:teuthology.orchestra.run.smithi052.stdout: "PWD_POLICY_EXCLUSION_LIST": { 2024-05-27T07:36:52.972 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "PWD_POLICY_EXCLUSION_LIST", 2024-05-27T07:36:52.972 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "str", 2024-05-27T07:36:52.972 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-05-27T07:36:52.972 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 0, 2024-05-27T07:36:52.972 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps", 2024-05-27T07:36:52.972 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-05-27T07:36:52.972 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-05-27T07:36:52.973 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-05-27T07:36:52.973 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-05-27T07:36:52.973 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-05-27T07:36:52.973 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-05-27T07:36:52.973 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-05-27T07:36:52.973 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-05-27T07:36:52.973 INFO:teuthology.orchestra.run.smithi052.stdout: "PWD_POLICY_MIN_COMPLEXITY": { 2024-05-27T07:36:52.973 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "PWD_POLICY_MIN_COMPLEXITY", 2024-05-27T07:36:52.973 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "int", 2024-05-27T07:36:52.973 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-05-27T07:36:52.973 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 0, 2024-05-27T07:36:52.973 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "10", 2024-05-27T07:36:52.973 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-05-27T07:36:52.973 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-05-27T07:36:52.974 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-05-27T07:36:52.974 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-05-27T07:36:52.974 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-05-27T07:36:52.974 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-05-27T07:36:52.974 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-05-27T07:36:52.974 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-05-27T07:36:52.974 INFO:teuthology.orchestra.run.smithi052.stdout: "PWD_POLICY_MIN_LENGTH": { 2024-05-27T07:36:52.974 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "PWD_POLICY_MIN_LENGTH", 2024-05-27T07:36:52.974 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "int", 2024-05-27T07:36:52.974 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-05-27T07:36:52.974 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 0, 2024-05-27T07:36:52.974 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "8", 2024-05-27T07:36:52.975 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-05-27T07:36:52.975 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-05-27T07:36:52.975 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-05-27T07:36:52.975 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-05-27T07:36:52.975 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-05-27T07:36:52.975 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-05-27T07:36:52.975 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-05-27T07:36:52.975 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-05-27T07:36:52.975 INFO:teuthology.orchestra.run.smithi052.stdout: "REST_REQUESTS_TIMEOUT": { 2024-05-27T07:36:52.975 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "REST_REQUESTS_TIMEOUT", 2024-05-27T07:36:52.975 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "int", 2024-05-27T07:36:52.975 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-05-27T07:36:52.975 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 0, 2024-05-27T07:36:52.976 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "45", 2024-05-27T07:36:52.976 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-05-27T07:36:52.976 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-05-27T07:36:52.976 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-05-27T07:36:52.976 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-05-27T07:36:52.976 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-05-27T07:36:52.976 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-05-27T07:36:52.976 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-05-27T07:36:52.976 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-05-27T07:36:52.976 INFO:teuthology.orchestra.run.smithi052.stdout: "RGW_API_ACCESS_KEY": { 2024-05-27T07:36:52.976 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "RGW_API_ACCESS_KEY", 2024-05-27T07:36:52.976 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "str", 2024-05-27T07:36:52.976 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-05-27T07:36:52.977 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 0, 2024-05-27T07:36:52.977 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "", 2024-05-27T07:36:52.977 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-05-27T07:36:52.977 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-05-27T07:36:52.977 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-05-27T07:36:52.977 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-05-27T07:36:52.977 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-05-27T07:36:52.977 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-05-27T07:36:52.977 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-05-27T07:36:52.977 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-05-27T07:36:52.977 INFO:teuthology.orchestra.run.smithi052.stdout: "RGW_API_ADMIN_RESOURCE": { 2024-05-27T07:36:52.977 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "RGW_API_ADMIN_RESOURCE", 2024-05-27T07:36:52.977 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "str", 2024-05-27T07:36:52.978 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-05-27T07:36:52.978 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 0, 2024-05-27T07:36:52.978 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "admin", 2024-05-27T07:36:52.978 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-05-27T07:36:52.978 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-05-27T07:36:52.978 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-05-27T07:36:52.978 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-05-27T07:36:52.978 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-05-27T07:36:52.978 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-05-27T07:36:52.978 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-05-27T07:36:52.978 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-05-27T07:36:52.978 INFO:teuthology.orchestra.run.smithi052.stdout: "RGW_API_SECRET_KEY": { 2024-05-27T07:36:52.979 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "RGW_API_SECRET_KEY", 2024-05-27T07:36:52.979 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "str", 2024-05-27T07:36:52.979 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-05-27T07:36:52.979 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 0, 2024-05-27T07:36:52.979 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "", 2024-05-27T07:36:52.979 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-05-27T07:36:52.979 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-05-27T07:36:52.979 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-05-27T07:36:52.979 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-05-27T07:36:52.979 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-05-27T07:36:52.979 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-05-27T07:36:52.979 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-05-27T07:36:52.979 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-05-27T07:36:52.980 INFO:teuthology.orchestra.run.smithi052.stdout: "RGW_API_SSL_VERIFY": { 2024-05-27T07:36:52.980 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "RGW_API_SSL_VERIFY", 2024-05-27T07:36:52.980 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "bool", 2024-05-27T07:36:52.980 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-05-27T07:36:52.980 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 0, 2024-05-27T07:36:52.980 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "True", 2024-05-27T07:36:52.980 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-05-27T07:36:52.980 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-05-27T07:36:52.980 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-05-27T07:36:52.980 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-05-27T07:36:52.980 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-05-27T07:36:52.980 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-05-27T07:36:52.980 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-05-27T07:36:52.981 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-05-27T07:36:52.981 INFO:teuthology.orchestra.run.smithi052.stdout: "UNSAFE_TLS_v1_2": { 2024-05-27T07:36:52.981 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "UNSAFE_TLS_v1_2", 2024-05-27T07:36:52.981 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "bool", 2024-05-27T07:36:52.981 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-05-27T07:36:52.981 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 0, 2024-05-27T07:36:52.981 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "False", 2024-05-27T07:36:52.981 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-05-27T07:36:52.981 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-05-27T07:36:52.981 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-05-27T07:36:52.981 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-05-27T07:36:52.981 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-05-27T07:36:52.981 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-05-27T07:36:52.982 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-05-27T07:36:52.982 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-05-27T07:36:52.982 INFO:teuthology.orchestra.run.smithi052.stdout: "USER_PWD_EXPIRATION_SPAN": { 2024-05-27T07:36:52.982 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "USER_PWD_EXPIRATION_SPAN", 2024-05-27T07:36:52.982 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "int", 2024-05-27T07:36:52.982 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-05-27T07:36:52.982 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 0, 2024-05-27T07:36:52.982 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "0", 2024-05-27T07:36:52.982 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-05-27T07:36:52.982 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-05-27T07:36:52.982 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-05-27T07:36:52.982 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-05-27T07:36:52.983 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-05-27T07:36:52.983 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-05-27T07:36:52.983 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-05-27T07:36:52.983 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-05-27T07:36:52.983 INFO:teuthology.orchestra.run.smithi052.stdout: "USER_PWD_EXPIRATION_WARNING_1": { 2024-05-27T07:36:52.983 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "USER_PWD_EXPIRATION_WARNING_1", 2024-05-27T07:36:52.983 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "int", 2024-05-27T07:36:52.983 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-05-27T07:36:52.983 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 0, 2024-05-27T07:36:52.983 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "10", 2024-05-27T07:36:52.983 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-05-27T07:36:52.983 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-05-27T07:36:52.983 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-05-27T07:36:52.984 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-05-27T07:36:52.984 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-05-27T07:36:52.984 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-05-27T07:36:52.984 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-05-27T07:36:52.984 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-05-27T07:36:52.984 INFO:teuthology.orchestra.run.smithi052.stdout: "USER_PWD_EXPIRATION_WARNING_2": { 2024-05-27T07:36:52.984 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "USER_PWD_EXPIRATION_WARNING_2", 2024-05-27T07:36:52.984 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "int", 2024-05-27T07:36:52.984 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-05-27T07:36:52.984 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 0, 2024-05-27T07:36:52.984 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "5", 2024-05-27T07:36:52.984 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-05-27T07:36:52.984 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-05-27T07:36:52.985 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-05-27T07:36:52.985 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-05-27T07:36:52.985 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-05-27T07:36:52.985 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-05-27T07:36:52.985 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-05-27T07:36:52.985 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-05-27T07:36:52.985 INFO:teuthology.orchestra.run.smithi052.stdout: "cross_origin_url": { 2024-05-27T07:36:52.985 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "cross_origin_url", 2024-05-27T07:36:52.985 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "str", 2024-05-27T07:36:52.985 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-05-27T07:36:52.985 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 0, 2024-05-27T07:36:52.985 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "", 2024-05-27T07:36:52.985 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-05-27T07:36:52.986 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-05-27T07:36:52.986 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-05-27T07:36:52.986 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-05-27T07:36:52.986 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-05-27T07:36:52.986 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-05-27T07:36:52.986 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-05-27T07:36:52.986 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-05-27T07:36:52.986 INFO:teuthology.orchestra.run.smithi052.stdout: "crt_file": { 2024-05-27T07:36:52.986 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "crt_file", 2024-05-27T07:36:52.986 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "str", 2024-05-27T07:36:52.986 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-05-27T07:36:52.986 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 0, 2024-05-27T07:36:52.986 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "", 2024-05-27T07:36:52.987 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-05-27T07:36:52.987 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-05-27T07:36:52.987 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-05-27T07:36:52.987 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-05-27T07:36:52.987 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-05-27T07:36:52.987 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-05-27T07:36:52.987 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-05-27T07:36:52.987 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-05-27T07:36:52.987 INFO:teuthology.orchestra.run.smithi052.stdout: "debug": { 2024-05-27T07:36:52.987 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "debug", 2024-05-27T07:36:52.987 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "bool", 2024-05-27T07:36:52.987 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-05-27T07:36:52.987 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 0, 2024-05-27T07:36:52.988 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "False", 2024-05-27T07:36:52.988 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-05-27T07:36:52.988 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-05-27T07:36:52.988 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-05-27T07:36:52.988 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "Enable/disable debug options", 2024-05-27T07:36:52.988 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-05-27T07:36:52.988 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-05-27T07:36:52.988 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-05-27T07:36:52.988 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-05-27T07:36:52.988 INFO:teuthology.orchestra.run.smithi052.stdout: "jwt_token_ttl": { 2024-05-27T07:36:52.988 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "jwt_token_ttl", 2024-05-27T07:36:52.988 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "int", 2024-05-27T07:36:52.988 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-05-27T07:36:52.989 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 0, 2024-05-27T07:36:52.989 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "28800", 2024-05-27T07:36:52.989 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-05-27T07:36:52.989 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-05-27T07:36:52.989 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-05-27T07:36:52.989 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-05-27T07:36:52.989 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-05-27T07:36:52.989 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-05-27T07:36:52.989 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-05-27T07:36:52.989 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-05-27T07:36:52.989 INFO:teuthology.orchestra.run.smithi052.stdout: "key_file": { 2024-05-27T07:36:52.989 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "key_file", 2024-05-27T07:36:52.989 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "str", 2024-05-27T07:36:52.990 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-05-27T07:36:52.990 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 0, 2024-05-27T07:36:52.990 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "", 2024-05-27T07:36:52.990 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-05-27T07:36:52.990 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-05-27T07:36:52.990 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-05-27T07:36:52.990 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-05-27T07:36:52.990 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-05-27T07:36:52.990 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-05-27T07:36:52.990 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-05-27T07:36:52.990 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-05-27T07:36:52.990 INFO:teuthology.orchestra.run.smithi052.stdout: "log_level": { 2024-05-27T07:36:52.990 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "log_level", 2024-05-27T07:36:52.991 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "str", 2024-05-27T07:36:52.991 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-05-27T07:36:52.991 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 1, 2024-05-27T07:36:52.991 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "", 2024-05-27T07:36:52.991 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-05-27T07:36:52.991 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-05-27T07:36:52.991 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [ 2024-05-27T07:36:52.991 INFO:teuthology.orchestra.run.smithi052.stdout: "", 2024-05-27T07:36:52.991 INFO:teuthology.orchestra.run.smithi052.stdout: "critical", 2024-05-27T07:36:52.991 INFO:teuthology.orchestra.run.smithi052.stdout: "debug", 2024-05-27T07:36:52.991 INFO:teuthology.orchestra.run.smithi052.stdout: "error", 2024-05-27T07:36:52.991 INFO:teuthology.orchestra.run.smithi052.stdout: "info", 2024-05-27T07:36:52.991 INFO:teuthology.orchestra.run.smithi052.stdout: "warning" 2024-05-27T07:36:52.992 INFO:teuthology.orchestra.run.smithi052.stdout: ], 2024-05-27T07:36:52.992 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-05-27T07:36:52.992 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-05-27T07:36:52.992 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-05-27T07:36:52.992 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-05-27T07:36:52.992 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-05-27T07:36:52.992 INFO:teuthology.orchestra.run.smithi052.stdout: "log_to_cluster": { 2024-05-27T07:36:52.992 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "log_to_cluster", 2024-05-27T07:36:52.992 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "bool", 2024-05-27T07:36:52.992 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-05-27T07:36:52.992 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 1, 2024-05-27T07:36:52.992 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "False", 2024-05-27T07:36:52.992 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-05-27T07:36:52.993 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-05-27T07:36:52.993 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-05-27T07:36:52.993 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-05-27T07:36:52.993 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-05-27T07:36:52.993 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-05-27T07:36:52.993 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-05-27T07:36:52.993 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-05-27T07:36:52.993 INFO:teuthology.orchestra.run.smithi052.stdout: "log_to_cluster_level": { 2024-05-27T07:36:52.993 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "log_to_cluster_level", 2024-05-27T07:36:52.993 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "str", 2024-05-27T07:36:52.993 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-05-27T07:36:52.993 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 1, 2024-05-27T07:36:52.993 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "info", 2024-05-27T07:36:52.994 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-05-27T07:36:52.994 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-05-27T07:36:52.994 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [ 2024-05-27T07:36:52.994 INFO:teuthology.orchestra.run.smithi052.stdout: "", 2024-05-27T07:36:52.994 INFO:teuthology.orchestra.run.smithi052.stdout: "critical", 2024-05-27T07:36:52.994 INFO:teuthology.orchestra.run.smithi052.stdout: "debug", 2024-05-27T07:36:52.994 INFO:teuthology.orchestra.run.smithi052.stdout: "error", 2024-05-27T07:36:52.994 INFO:teuthology.orchestra.run.smithi052.stdout: "info", 2024-05-27T07:36:52.994 INFO:teuthology.orchestra.run.smithi052.stdout: "warning" 2024-05-27T07:36:52.994 INFO:teuthology.orchestra.run.smithi052.stdout: ], 2024-05-27T07:36:52.994 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-05-27T07:36:52.994 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-05-27T07:36:52.995 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-05-27T07:36:52.995 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-05-27T07:36:52.995 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-05-27T07:36:52.995 INFO:teuthology.orchestra.run.smithi052.stdout: "log_to_file": { 2024-05-27T07:36:52.995 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "log_to_file", 2024-05-27T07:36:52.995 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "bool", 2024-05-27T07:36:52.995 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-05-27T07:36:52.995 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 1, 2024-05-27T07:36:52.995 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "False", 2024-05-27T07:36:52.995 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-05-27T07:36:52.995 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-05-27T07:36:52.995 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-05-27T07:36:52.995 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-05-27T07:36:52.995 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-05-27T07:36:52.996 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-05-27T07:36:52.996 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-05-27T07:36:52.996 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-05-27T07:36:52.996 INFO:teuthology.orchestra.run.smithi052.stdout: "motd": { 2024-05-27T07:36:52.996 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "motd", 2024-05-27T07:36:52.996 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "str", 2024-05-27T07:36:52.996 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-05-27T07:36:52.996 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 0, 2024-05-27T07:36:52.996 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "", 2024-05-27T07:36:52.996 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-05-27T07:36:52.996 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-05-27T07:36:52.996 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-05-27T07:36:52.996 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "The message of the day", 2024-05-27T07:36:52.997 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-05-27T07:36:52.997 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-05-27T07:36:52.997 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-05-27T07:36:52.997 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-05-27T07:36:52.997 INFO:teuthology.orchestra.run.smithi052.stdout: "redirect_resolve_ip_addr": { 2024-05-27T07:36:52.997 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "redirect_resolve_ip_addr", 2024-05-27T07:36:52.997 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "bool", 2024-05-27T07:36:52.997 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-05-27T07:36:52.997 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 0, 2024-05-27T07:36:52.997 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "False", 2024-05-27T07:36:52.997 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-05-27T07:36:52.997 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-05-27T07:36:52.997 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-05-27T07:36:52.998 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-05-27T07:36:52.998 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-05-27T07:36:52.998 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-05-27T07:36:52.998 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-05-27T07:36:52.998 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-05-27T07:36:52.998 INFO:teuthology.orchestra.run.smithi052.stdout: "server_addr": { 2024-05-27T07:36:52.998 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "server_addr", 2024-05-27T07:36:52.998 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "str", 2024-05-27T07:36:52.998 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-05-27T07:36:52.998 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 0, 2024-05-27T07:36:52.998 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "::", 2024-05-27T07:36:52.998 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-05-27T07:36:52.998 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-05-27T07:36:52.999 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-05-27T07:36:52.999 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-05-27T07:36:52.999 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-05-27T07:36:52.999 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-05-27T07:36:52.999 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-05-27T07:36:52.999 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-05-27T07:36:52.999 INFO:teuthology.orchestra.run.smithi052.stdout: "server_port": { 2024-05-27T07:36:52.999 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "server_port", 2024-05-27T07:36:52.999 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "int", 2024-05-27T07:36:52.999 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-05-27T07:36:52.999 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 0, 2024-05-27T07:36:52.999 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "8080", 2024-05-27T07:36:53.000 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-05-27T07:36:53.000 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-05-27T07:36:53.000 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-05-27T07:36:53.000 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-05-27T07:36:53.000 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-05-27T07:36:53.000 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-05-27T07:36:53.000 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-05-27T07:36:53.000 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-05-27T07:36:53.000 INFO:teuthology.orchestra.run.smithi052.stdout: "sqlite3_killpoint": { 2024-05-27T07:36:53.000 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "sqlite3_killpoint", 2024-05-27T07:36:53.000 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "int", 2024-05-27T07:36:53.000 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "dev", 2024-05-27T07:36:53.000 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 1, 2024-05-27T07:36:53.000 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "0", 2024-05-27T07:36:53.001 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-05-27T07:36:53.001 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-05-27T07:36:53.001 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-05-27T07:36:53.001 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-05-27T07:36:53.001 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-05-27T07:36:53.001 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-05-27T07:36:53.001 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-05-27T07:36:53.001 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-05-27T07:36:53.001 INFO:teuthology.orchestra.run.smithi052.stdout: "ssl": { 2024-05-27T07:36:53.001 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "ssl", 2024-05-27T07:36:53.001 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "bool", 2024-05-27T07:36:53.001 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-05-27T07:36:53.001 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 0, 2024-05-27T07:36:53.001 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "True", 2024-05-27T07:36:53.002 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-05-27T07:36:53.002 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-05-27T07:36:53.002 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-05-27T07:36:53.002 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-05-27T07:36:53.002 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-05-27T07:36:53.002 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-05-27T07:36:53.002 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-05-27T07:36:53.002 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-05-27T07:36:53.002 INFO:teuthology.orchestra.run.smithi052.stdout: "ssl_server_port": { 2024-05-27T07:36:53.002 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "ssl_server_port", 2024-05-27T07:36:53.002 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "int", 2024-05-27T07:36:53.002 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-05-27T07:36:53.002 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 0, 2024-05-27T07:36:53.003 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "8443", 2024-05-27T07:36:53.003 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-05-27T07:36:53.003 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-05-27T07:36:53.003 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-05-27T07:36:53.003 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-05-27T07:36:53.003 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-05-27T07:36:53.003 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-05-27T07:36:53.003 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-05-27T07:36:53.003 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-05-27T07:36:53.003 INFO:teuthology.orchestra.run.smithi052.stdout: "standby_behaviour": { 2024-05-27T07:36:53.003 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "standby_behaviour", 2024-05-27T07:36:53.003 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "str", 2024-05-27T07:36:53.003 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-05-27T07:36:53.004 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 0, 2024-05-27T07:36:53.004 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "redirect", 2024-05-27T07:36:53.004 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-05-27T07:36:53.004 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-05-27T07:36:53.004 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [ 2024-05-27T07:36:53.004 INFO:teuthology.orchestra.run.smithi052.stdout: "error", 2024-05-27T07:36:53.004 INFO:teuthology.orchestra.run.smithi052.stdout: "redirect" 2024-05-27T07:36:53.004 INFO:teuthology.orchestra.run.smithi052.stdout: ], 2024-05-27T07:36:53.004 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-05-27T07:36:53.004 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-05-27T07:36:53.004 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-05-27T07:36:53.004 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-05-27T07:36:53.004 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-05-27T07:36:53.004 INFO:teuthology.orchestra.run.smithi052.stdout: "standby_error_status_code": { 2024-05-27T07:36:53.005 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "standby_error_status_code", 2024-05-27T07:36:53.005 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "int", 2024-05-27T07:36:53.005 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-05-27T07:36:53.005 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 0, 2024-05-27T07:36:53.005 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "500", 2024-05-27T07:36:53.005 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "400", 2024-05-27T07:36:53.005 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "599", 2024-05-27T07:36:53.005 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-05-27T07:36:53.005 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-05-27T07:36:53.005 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-05-27T07:36:53.005 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-05-27T07:36:53.005 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-05-27T07:36:53.005 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-05-27T07:36:53.006 INFO:teuthology.orchestra.run.smithi052.stdout: "url_prefix": { 2024-05-27T07:36:53.006 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "url_prefix", 2024-05-27T07:36:53.006 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "str", 2024-05-27T07:36:53.006 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-05-27T07:36:53.006 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 0, 2024-05-27T07:36:53.006 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "", 2024-05-27T07:36:53.006 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-05-27T07:36:53.006 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-05-27T07:36:53.006 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-05-27T07:36:53.006 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-05-27T07:36:53.006 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-05-27T07:36:53.006 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-05-27T07:36:53.006 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-05-27T07:36:53.007 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-05-27T07:36:53.007 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-05-27T07:36:53.007 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-05-27T07:36:53.007 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-05-27T07:36:53.007 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "devicehealth", 2024-05-27T07:36:53.007 INFO:teuthology.orchestra.run.smithi052.stdout: "can_run": true, 2024-05-27T07:36:53.007 INFO:teuthology.orchestra.run.smithi052.stdout: "error_string": "", 2024-05-27T07:36:53.007 INFO:teuthology.orchestra.run.smithi052.stdout: "module_options": { 2024-05-27T07:36:53.007 INFO:teuthology.orchestra.run.smithi052.stdout: "enable_monitoring": { 2024-05-27T07:36:53.007 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "enable_monitoring", 2024-05-27T07:36:53.007 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "bool", 2024-05-27T07:36:53.007 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-05-27T07:36:53.007 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 1, 2024-05-27T07:36:53.008 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "True", 2024-05-27T07:36:53.008 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-05-27T07:36:53.008 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-05-27T07:36:53.008 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-05-27T07:36:53.008 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "monitor device health metrics", 2024-05-27T07:36:53.008 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-05-27T07:36:53.008 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-05-27T07:36:53.008 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-05-27T07:36:53.008 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-05-27T07:36:53.008 INFO:teuthology.orchestra.run.smithi052.stdout: "log_level": { 2024-05-27T07:36:53.008 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "log_level", 2024-05-27T07:36:53.008 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "str", 2024-05-27T07:36:53.008 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-05-27T07:36:53.009 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 1, 2024-05-27T07:36:53.009 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "", 2024-05-27T07:36:53.009 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-05-27T07:36:53.009 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-05-27T07:36:53.009 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [ 2024-05-27T07:36:53.009 INFO:teuthology.orchestra.run.smithi052.stdout: "", 2024-05-27T07:36:53.009 INFO:teuthology.orchestra.run.smithi052.stdout: "critical", 2024-05-27T07:36:53.009 INFO:teuthology.orchestra.run.smithi052.stdout: "debug", 2024-05-27T07:36:53.009 INFO:teuthology.orchestra.run.smithi052.stdout: "error", 2024-05-27T07:36:53.009 INFO:teuthology.orchestra.run.smithi052.stdout: "info", 2024-05-27T07:36:53.009 INFO:teuthology.orchestra.run.smithi052.stdout: "warning" 2024-05-27T07:36:53.009 INFO:teuthology.orchestra.run.smithi052.stdout: ], 2024-05-27T07:36:53.009 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-05-27T07:36:53.010 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-05-27T07:36:53.010 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-05-27T07:36:53.010 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-05-27T07:36:53.010 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-05-27T07:36:53.010 INFO:teuthology.orchestra.run.smithi052.stdout: "log_to_cluster": { 2024-05-27T07:36:53.010 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "log_to_cluster", 2024-05-27T07:36:53.010 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "bool", 2024-05-27T07:36:53.010 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-05-27T07:36:53.010 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 1, 2024-05-27T07:36:53.010 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "False", 2024-05-27T07:36:53.010 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-05-27T07:36:53.010 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-05-27T07:36:53.011 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-05-27T07:36:53.011 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-05-27T07:36:53.011 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-05-27T07:36:53.011 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-05-27T07:36:53.011 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-05-27T07:36:53.011 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-05-27T07:36:53.011 INFO:teuthology.orchestra.run.smithi052.stdout: "log_to_cluster_level": { 2024-05-27T07:36:53.011 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "log_to_cluster_level", 2024-05-27T07:36:53.011 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "str", 2024-05-27T07:36:53.011 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-05-27T07:36:53.011 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 1, 2024-05-27T07:36:53.011 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "info", 2024-05-27T07:36:53.011 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-05-27T07:36:53.012 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-05-27T07:36:53.012 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [ 2024-05-27T07:36:53.012 INFO:teuthology.orchestra.run.smithi052.stdout: "", 2024-05-27T07:36:53.012 INFO:teuthology.orchestra.run.smithi052.stdout: "critical", 2024-05-27T07:36:53.012 INFO:teuthology.orchestra.run.smithi052.stdout: "debug", 2024-05-27T07:36:53.012 INFO:teuthology.orchestra.run.smithi052.stdout: "error", 2024-05-27T07:36:53.012 INFO:teuthology.orchestra.run.smithi052.stdout: "info", 2024-05-27T07:36:53.012 INFO:teuthology.orchestra.run.smithi052.stdout: "warning" 2024-05-27T07:36:53.012 INFO:teuthology.orchestra.run.smithi052.stdout: ], 2024-05-27T07:36:53.012 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-05-27T07:36:53.012 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-05-27T07:36:53.012 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-05-27T07:36:53.013 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-05-27T07:36:53.013 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-05-27T07:36:53.013 INFO:teuthology.orchestra.run.smithi052.stdout: "log_to_file": { 2024-05-27T07:36:53.013 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "log_to_file", 2024-05-27T07:36:53.013 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "bool", 2024-05-27T07:36:53.013 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-05-27T07:36:53.013 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 1, 2024-05-27T07:36:53.013 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "False", 2024-05-27T07:36:53.013 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-05-27T07:36:53.013 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-05-27T07:36:53.013 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-05-27T07:36:53.013 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-05-27T07:36:53.013 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-05-27T07:36:53.014 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-05-27T07:36:53.014 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-05-27T07:36:53.014 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-05-27T07:36:53.014 INFO:teuthology.orchestra.run.smithi052.stdout: "mark_out_threshold": { 2024-05-27T07:36:53.014 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "mark_out_threshold", 2024-05-27T07:36:53.014 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "secs", 2024-05-27T07:36:53.014 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-05-27T07:36:53.014 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 1, 2024-05-27T07:36:53.014 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "2419200", 2024-05-27T07:36:53.014 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-05-27T07:36:53.014 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-05-27T07:36:53.014 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-05-27T07:36:53.014 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "automatically mark OSD if it may fail before this long", 2024-05-27T07:36:53.015 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-05-27T07:36:53.015 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-05-27T07:36:53.015 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-05-27T07:36:53.015 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-05-27T07:36:53.015 INFO:teuthology.orchestra.run.smithi052.stdout: "pool_name": { 2024-05-27T07:36:53.015 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "pool_name", 2024-05-27T07:36:53.015 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "str", 2024-05-27T07:36:53.015 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-05-27T07:36:53.015 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 1, 2024-05-27T07:36:53.015 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "device_health_metrics", 2024-05-27T07:36:53.015 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-05-27T07:36:53.015 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-05-27T07:36:53.015 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-05-27T07:36:53.016 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "name of pool in which to store device health metrics", 2024-05-27T07:36:53.016 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-05-27T07:36:53.016 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-05-27T07:36:53.016 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-05-27T07:36:53.016 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-05-27T07:36:53.016 INFO:teuthology.orchestra.run.smithi052.stdout: "retention_period": { 2024-05-27T07:36:53.016 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "retention_period", 2024-05-27T07:36:53.016 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "secs", 2024-05-27T07:36:53.016 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-05-27T07:36:53.016 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 1, 2024-05-27T07:36:53.016 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "15552000", 2024-05-27T07:36:53.016 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-05-27T07:36:53.016 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-05-27T07:36:53.017 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-05-27T07:36:53.017 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "how long to retain device health metrics", 2024-05-27T07:36:53.017 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-05-27T07:36:53.017 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-05-27T07:36:53.017 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-05-27T07:36:53.017 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-05-27T07:36:53.017 INFO:teuthology.orchestra.run.smithi052.stdout: "scrape_frequency": { 2024-05-27T07:36:53.017 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "scrape_frequency", 2024-05-27T07:36:53.017 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "secs", 2024-05-27T07:36:53.017 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-05-27T07:36:53.017 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 1, 2024-05-27T07:36:53.017 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "86400", 2024-05-27T07:36:53.017 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-05-27T07:36:53.018 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-05-27T07:36:53.018 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-05-27T07:36:53.018 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "how frequently to scrape device health metrics", 2024-05-27T07:36:53.018 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-05-27T07:36:53.018 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-05-27T07:36:53.018 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-05-27T07:36:53.018 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-05-27T07:36:53.018 INFO:teuthology.orchestra.run.smithi052.stdout: "self_heal": { 2024-05-27T07:36:53.018 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "self_heal", 2024-05-27T07:36:53.018 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "bool", 2024-05-27T07:36:53.018 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-05-27T07:36:53.018 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 1, 2024-05-27T07:36:53.019 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "True", 2024-05-27T07:36:53.019 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-05-27T07:36:53.019 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-05-27T07:36:53.019 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-05-27T07:36:53.019 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "preemptively heal cluster around devices that may fail", 2024-05-27T07:36:53.019 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-05-27T07:36:53.019 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-05-27T07:36:53.019 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-05-27T07:36:53.019 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-05-27T07:36:53.019 INFO:teuthology.orchestra.run.smithi052.stdout: "sleep_interval": { 2024-05-27T07:36:53.019 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "sleep_interval", 2024-05-27T07:36:53.019 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "secs", 2024-05-27T07:36:53.020 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-05-27T07:36:53.020 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 1, 2024-05-27T07:36:53.020 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "600", 2024-05-27T07:36:53.020 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-05-27T07:36:53.020 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-05-27T07:36:53.020 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-05-27T07:36:53.020 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "how frequently to wake up and check device health", 2024-05-27T07:36:53.020 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-05-27T07:36:53.020 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-05-27T07:36:53.020 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-05-27T07:36:53.020 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-05-27T07:36:53.020 INFO:teuthology.orchestra.run.smithi052.stdout: "sqlite3_killpoint": { 2024-05-27T07:36:53.020 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "sqlite3_killpoint", 2024-05-27T07:36:53.021 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "int", 2024-05-27T07:36:53.021 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "dev", 2024-05-27T07:36:53.021 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 1, 2024-05-27T07:36:53.021 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "0", 2024-05-27T07:36:53.021 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-05-27T07:36:53.021 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-05-27T07:36:53.021 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-05-27T07:36:53.021 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-05-27T07:36:53.021 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-05-27T07:36:53.021 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-05-27T07:36:53.021 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-05-27T07:36:53.021 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-05-27T07:36:53.021 INFO:teuthology.orchestra.run.smithi052.stdout: "warn_threshold": { 2024-05-27T07:36:53.022 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "warn_threshold", 2024-05-27T07:36:53.022 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "secs", 2024-05-27T07:36:53.022 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-05-27T07:36:53.022 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 1, 2024-05-27T07:36:53.022 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "7257600", 2024-05-27T07:36:53.022 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-05-27T07:36:53.022 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-05-27T07:36:53.022 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-05-27T07:36:53.022 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "raise health warning if OSD may fail before this long", 2024-05-27T07:36:53.022 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-05-27T07:36:53.022 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-05-27T07:36:53.022 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-05-27T07:36:53.023 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-05-27T07:36:53.023 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-05-27T07:36:53.023 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-05-27T07:36:53.023 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-05-27T07:36:53.023 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "diskprediction_local", 2024-05-27T07:36:53.023 INFO:teuthology.orchestra.run.smithi052.stdout: "can_run": true, 2024-05-27T07:36:53.023 INFO:teuthology.orchestra.run.smithi052.stdout: "error_string": "", 2024-05-27T07:36:53.023 INFO:teuthology.orchestra.run.smithi052.stdout: "module_options": { 2024-05-27T07:36:53.023 INFO:teuthology.orchestra.run.smithi052.stdout: "log_level": { 2024-05-27T07:36:53.023 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "log_level", 2024-05-27T07:36:53.023 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "str", 2024-05-27T07:36:53.023 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-05-27T07:36:53.024 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 1, 2024-05-27T07:36:53.024 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "", 2024-05-27T07:36:53.024 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-05-27T07:36:53.024 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-05-27T07:36:53.024 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [ 2024-05-27T07:36:53.024 INFO:teuthology.orchestra.run.smithi052.stdout: "", 2024-05-27T07:36:53.024 INFO:teuthology.orchestra.run.smithi052.stdout: "critical", 2024-05-27T07:36:53.024 INFO:teuthology.orchestra.run.smithi052.stdout: "debug", 2024-05-27T07:36:53.024 INFO:teuthology.orchestra.run.smithi052.stdout: "error", 2024-05-27T07:36:53.024 INFO:teuthology.orchestra.run.smithi052.stdout: "info", 2024-05-27T07:36:53.024 INFO:teuthology.orchestra.run.smithi052.stdout: "warning" 2024-05-27T07:36:53.024 INFO:teuthology.orchestra.run.smithi052.stdout: ], 2024-05-27T07:36:53.024 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-05-27T07:36:53.025 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-05-27T07:36:53.025 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-05-27T07:36:53.025 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-05-27T07:36:53.025 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-05-27T07:36:53.025 INFO:teuthology.orchestra.run.smithi052.stdout: "log_to_cluster": { 2024-05-27T07:36:53.025 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "log_to_cluster", 2024-05-27T07:36:53.025 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "bool", 2024-05-27T07:36:53.025 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-05-27T07:36:53.025 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 1, 2024-05-27T07:36:53.025 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "False", 2024-05-27T07:36:53.025 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-05-27T07:36:53.025 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-05-27T07:36:53.026 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-05-27T07:36:53.026 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-05-27T07:36:53.026 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-05-27T07:36:53.026 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-05-27T07:36:53.026 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-05-27T07:36:53.026 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-05-27T07:36:53.026 INFO:teuthology.orchestra.run.smithi052.stdout: "log_to_cluster_level": { 2024-05-27T07:36:53.026 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "log_to_cluster_level", 2024-05-27T07:36:53.026 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "str", 2024-05-27T07:36:53.026 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-05-27T07:36:53.026 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 1, 2024-05-27T07:36:53.026 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "info", 2024-05-27T07:36:53.026 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-05-27T07:36:53.027 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-05-27T07:36:53.027 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [ 2024-05-27T07:36:53.027 INFO:teuthology.orchestra.run.smithi052.stdout: "", 2024-05-27T07:36:53.027 INFO:teuthology.orchestra.run.smithi052.stdout: "critical", 2024-05-27T07:36:53.027 INFO:teuthology.orchestra.run.smithi052.stdout: "debug", 2024-05-27T07:36:53.027 INFO:teuthology.orchestra.run.smithi052.stdout: "error", 2024-05-27T07:36:53.027 INFO:teuthology.orchestra.run.smithi052.stdout: "info", 2024-05-27T07:36:53.027 INFO:teuthology.orchestra.run.smithi052.stdout: "warning" 2024-05-27T07:36:53.027 INFO:teuthology.orchestra.run.smithi052.stdout: ], 2024-05-27T07:36:53.027 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-05-27T07:36:53.027 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-05-27T07:36:53.027 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-05-27T07:36:53.027 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-05-27T07:36:53.028 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-05-27T07:36:53.028 INFO:teuthology.orchestra.run.smithi052.stdout: "log_to_file": { 2024-05-27T07:36:53.028 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "log_to_file", 2024-05-27T07:36:53.028 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "bool", 2024-05-27T07:36:53.028 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-05-27T07:36:53.028 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 1, 2024-05-27T07:36:53.028 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "False", 2024-05-27T07:36:53.028 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-05-27T07:36:53.028 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-05-27T07:36:53.028 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-05-27T07:36:53.028 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-05-27T07:36:53.028 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-05-27T07:36:53.029 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-05-27T07:36:53.029 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-05-27T07:36:53.029 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-05-27T07:36:53.029 INFO:teuthology.orchestra.run.smithi052.stdout: "predict_interval": { 2024-05-27T07:36:53.029 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "predict_interval", 2024-05-27T07:36:53.029 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "str", 2024-05-27T07:36:53.029 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-05-27T07:36:53.029 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 0, 2024-05-27T07:36:53.029 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "86400", 2024-05-27T07:36:53.029 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-05-27T07:36:53.029 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-05-27T07:36:53.029 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-05-27T07:36:53.029 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-05-27T07:36:53.030 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-05-27T07:36:53.030 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-05-27T07:36:53.030 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-05-27T07:36:53.030 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-05-27T07:36:53.030 INFO:teuthology.orchestra.run.smithi052.stdout: "predictor_model": { 2024-05-27T07:36:53.030 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "predictor_model", 2024-05-27T07:36:53.030 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "str", 2024-05-27T07:36:53.030 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-05-27T07:36:53.030 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 0, 2024-05-27T07:36:53.030 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "prophetstor", 2024-05-27T07:36:53.030 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-05-27T07:36:53.030 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-05-27T07:36:53.030 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-05-27T07:36:53.031 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-05-27T07:36:53.031 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-05-27T07:36:53.031 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-05-27T07:36:53.031 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-05-27T07:36:53.031 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-05-27T07:36:53.031 INFO:teuthology.orchestra.run.smithi052.stdout: "sleep_interval": { 2024-05-27T07:36:53.031 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "sleep_interval", 2024-05-27T07:36:53.031 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "str", 2024-05-27T07:36:53.031 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-05-27T07:36:53.031 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 0, 2024-05-27T07:36:53.031 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "600", 2024-05-27T07:36:53.031 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-05-27T07:36:53.032 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-05-27T07:36:53.032 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-05-27T07:36:53.032 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-05-27T07:36:53.032 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-05-27T07:36:53.032 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-05-27T07:36:53.032 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-05-27T07:36:53.032 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-05-27T07:36:53.032 INFO:teuthology.orchestra.run.smithi052.stdout: "sqlite3_killpoint": { 2024-05-27T07:36:53.032 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "sqlite3_killpoint", 2024-05-27T07:36:53.032 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "int", 2024-05-27T07:36:53.032 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "dev", 2024-05-27T07:36:53.032 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 1, 2024-05-27T07:36:53.032 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "0", 2024-05-27T07:36:53.033 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-05-27T07:36:53.033 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-05-27T07:36:53.033 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-05-27T07:36:53.033 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-05-27T07:36:53.033 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-05-27T07:36:53.033 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-05-27T07:36:53.033 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-05-27T07:36:53.033 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-05-27T07:36:53.033 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-05-27T07:36:53.033 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-05-27T07:36:53.033 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-05-27T07:36:53.033 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "influx", 2024-05-27T07:36:53.033 INFO:teuthology.orchestra.run.smithi052.stdout: "can_run": false, 2024-05-27T07:36:53.033 INFO:teuthology.orchestra.run.smithi052.stdout: "error_string": "influxdb python module not found", 2024-05-27T07:36:53.034 INFO:teuthology.orchestra.run.smithi052.stdout: "module_options": { 2024-05-27T07:36:53.034 INFO:teuthology.orchestra.run.smithi052.stdout: "batch_size": { 2024-05-27T07:36:53.034 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "batch_size", 2024-05-27T07:36:53.034 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "int", 2024-05-27T07:36:53.034 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-05-27T07:36:53.034 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 0, 2024-05-27T07:36:53.034 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "5000", 2024-05-27T07:36:53.034 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-05-27T07:36:53.034 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-05-27T07:36:53.034 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-05-27T07:36:53.034 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "How big batches of data points should be when sending to InfluxDB.", 2024-05-27T07:36:53.034 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-05-27T07:36:53.035 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-05-27T07:36:53.035 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-05-27T07:36:53.035 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-05-27T07:36:53.035 INFO:teuthology.orchestra.run.smithi052.stdout: "database": { 2024-05-27T07:36:53.035 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "database", 2024-05-27T07:36:53.035 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "str", 2024-05-27T07:36:53.035 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-05-27T07:36:53.035 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 0, 2024-05-27T07:36:53.035 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "ceph", 2024-05-27T07:36:53.035 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-05-27T07:36:53.035 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-05-27T07:36:53.035 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-05-27T07:36:53.035 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.", 2024-05-27T07:36:53.036 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-05-27T07:36:53.036 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-05-27T07:36:53.036 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-05-27T07:36:53.036 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-05-27T07:36:53.036 INFO:teuthology.orchestra.run.smithi052.stdout: "hostname": { 2024-05-27T07:36:53.036 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "hostname", 2024-05-27T07:36:53.036 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "str", 2024-05-27T07:36:53.036 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-05-27T07:36:53.036 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 0, 2024-05-27T07:36:53.036 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "", 2024-05-27T07:36:53.036 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-05-27T07:36:53.036 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-05-27T07:36:53.036 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-05-27T07:36:53.037 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "InfluxDB server hostname", 2024-05-27T07:36:53.037 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-05-27T07:36:53.037 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-05-27T07:36:53.037 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-05-27T07:36:53.037 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-05-27T07:36:53.037 INFO:teuthology.orchestra.run.smithi052.stdout: "interval": { 2024-05-27T07:36:53.037 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "interval", 2024-05-27T07:36:53.037 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "secs", 2024-05-27T07:36:53.037 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-05-27T07:36:53.037 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 0, 2024-05-27T07:36:53.037 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "30", 2024-05-27T07:36:53.037 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "5", 2024-05-27T07:36:53.037 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-05-27T07:36:53.038 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-05-27T07:36:53.038 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "Time between reports to InfluxDB. Default 30 seconds.", 2024-05-27T07:36:53.038 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-05-27T07:36:53.038 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-05-27T07:36:53.038 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-05-27T07:36:53.038 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-05-27T07:36:53.038 INFO:teuthology.orchestra.run.smithi052.stdout: "log_level": { 2024-05-27T07:36:53.038 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "log_level", 2024-05-27T07:36:53.038 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "str", 2024-05-27T07:36:53.038 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-05-27T07:36:53.038 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 1, 2024-05-27T07:36:53.038 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "", 2024-05-27T07:36:53.038 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-05-27T07:36:53.039 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-05-27T07:36:53.039 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [ 2024-05-27T07:36:53.039 INFO:teuthology.orchestra.run.smithi052.stdout: "", 2024-05-27T07:36:53.039 INFO:teuthology.orchestra.run.smithi052.stdout: "critical", 2024-05-27T07:36:53.039 INFO:teuthology.orchestra.run.smithi052.stdout: "debug", 2024-05-27T07:36:53.039 INFO:teuthology.orchestra.run.smithi052.stdout: "error", 2024-05-27T07:36:53.039 INFO:teuthology.orchestra.run.smithi052.stdout: "info", 2024-05-27T07:36:53.039 INFO:teuthology.orchestra.run.smithi052.stdout: "warning" 2024-05-27T07:36:53.039 INFO:teuthology.orchestra.run.smithi052.stdout: ], 2024-05-27T07:36:53.039 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-05-27T07:36:53.039 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-05-27T07:36:53.039 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-05-27T07:36:53.039 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-05-27T07:36:53.039 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-05-27T07:36:53.040 INFO:teuthology.orchestra.run.smithi052.stdout: "log_to_cluster": { 2024-05-27T07:36:53.040 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "log_to_cluster", 2024-05-27T07:36:53.040 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "bool", 2024-05-27T07:36:53.040 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-05-27T07:36:53.040 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 1, 2024-05-27T07:36:53.040 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "False", 2024-05-27T07:36:53.040 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-05-27T07:36:53.040 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-05-27T07:36:53.040 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-05-27T07:36:53.040 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-05-27T07:36:53.040 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-05-27T07:36:53.040 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-05-27T07:36:53.040 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-05-27T07:36:53.040 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-05-27T07:36:53.041 INFO:teuthology.orchestra.run.smithi052.stdout: "log_to_cluster_level": { 2024-05-27T07:36:53.041 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "log_to_cluster_level", 2024-05-27T07:36:53.041 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "str", 2024-05-27T07:36:53.041 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-05-27T07:36:53.041 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 1, 2024-05-27T07:36:53.041 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "info", 2024-05-27T07:36:53.041 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-05-27T07:36:53.041 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-05-27T07:36:53.041 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [ 2024-05-27T07:36:53.041 INFO:teuthology.orchestra.run.smithi052.stdout: "", 2024-05-27T07:36:53.041 INFO:teuthology.orchestra.run.smithi052.stdout: "critical", 2024-05-27T07:36:53.041 INFO:teuthology.orchestra.run.smithi052.stdout: "debug", 2024-05-27T07:36:53.041 INFO:teuthology.orchestra.run.smithi052.stdout: "error", 2024-05-27T07:36:53.041 INFO:teuthology.orchestra.run.smithi052.stdout: "info", 2024-05-27T07:36:53.042 INFO:teuthology.orchestra.run.smithi052.stdout: "warning" 2024-05-27T07:36:53.042 INFO:teuthology.orchestra.run.smithi052.stdout: ], 2024-05-27T07:36:53.042 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-05-27T07:36:53.042 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-05-27T07:36:53.042 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-05-27T07:36:53.042 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-05-27T07:36:53.042 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-05-27T07:36:53.042 INFO:teuthology.orchestra.run.smithi052.stdout: "log_to_file": { 2024-05-27T07:36:53.042 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "log_to_file", 2024-05-27T07:36:53.042 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "bool", 2024-05-27T07:36:53.042 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-05-27T07:36:53.042 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 1, 2024-05-27T07:36:53.043 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "False", 2024-05-27T07:36:53.043 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-05-27T07:36:53.043 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-05-27T07:36:53.043 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-05-27T07:36:53.043 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-05-27T07:36:53.043 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-05-27T07:36:53.043 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-05-27T07:36:53.043 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-05-27T07:36:53.043 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-05-27T07:36:53.043 INFO:teuthology.orchestra.run.smithi052.stdout: "password": { 2024-05-27T07:36:53.043 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "password", 2024-05-27T07:36:53.043 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "str", 2024-05-27T07:36:53.043 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-05-27T07:36:53.043 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 0, 2024-05-27T07:36:53.044 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "", 2024-05-27T07:36:53.044 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-05-27T07:36:53.044 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-05-27T07:36:53.044 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-05-27T07:36:53.044 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "password of InfluxDB server user", 2024-05-27T07:36:53.044 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-05-27T07:36:53.044 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-05-27T07:36:53.044 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-05-27T07:36:53.044 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-05-27T07:36:53.044 INFO:teuthology.orchestra.run.smithi052.stdout: "port": { 2024-05-27T07:36:53.044 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "port", 2024-05-27T07:36:53.044 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "int", 2024-05-27T07:36:53.044 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-05-27T07:36:53.045 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 0, 2024-05-27T07:36:53.045 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "8086", 2024-05-27T07:36:53.045 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-05-27T07:36:53.045 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-05-27T07:36:53.045 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-05-27T07:36:53.045 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "InfluxDB server port", 2024-05-27T07:36:53.045 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-05-27T07:36:53.045 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-05-27T07:36:53.045 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-05-27T07:36:53.045 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-05-27T07:36:53.045 INFO:teuthology.orchestra.run.smithi052.stdout: "sqlite3_killpoint": { 2024-05-27T07:36:53.045 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "sqlite3_killpoint", 2024-05-27T07:36:53.045 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "int", 2024-05-27T07:36:53.046 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "dev", 2024-05-27T07:36:53.046 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 1, 2024-05-27T07:36:53.046 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "0", 2024-05-27T07:36:53.046 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-05-27T07:36:53.046 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-05-27T07:36:53.046 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-05-27T07:36:53.046 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-05-27T07:36:53.046 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-05-27T07:36:53.046 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-05-27T07:36:53.046 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-05-27T07:36:53.046 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-05-27T07:36:53.046 INFO:teuthology.orchestra.run.smithi052.stdout: "ssl": { 2024-05-27T07:36:53.046 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "ssl", 2024-05-27T07:36:53.047 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "str", 2024-05-27T07:36:53.047 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-05-27T07:36:53.047 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 0, 2024-05-27T07:36:53.047 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "false", 2024-05-27T07:36:53.047 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-05-27T07:36:53.047 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-05-27T07:36:53.047 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-05-27T07:36:53.047 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "Use https connection for InfluxDB server. Use \"true\" or \"false\".", 2024-05-27T07:36:53.047 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-05-27T07:36:53.047 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-05-27T07:36:53.047 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-05-27T07:36:53.047 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-05-27T07:36:53.047 INFO:teuthology.orchestra.run.smithi052.stdout: "threads": { 2024-05-27T07:36:53.048 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "threads", 2024-05-27T07:36:53.048 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "int", 2024-05-27T07:36:53.048 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-05-27T07:36:53.048 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 0, 2024-05-27T07:36:53.048 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "5", 2024-05-27T07:36:53.048 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "1", 2024-05-27T07:36:53.048 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "32", 2024-05-27T07:36:53.048 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-05-27T07:36:53.048 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "How many worker threads should be spawned for sending data to InfluxDB.", 2024-05-27T07:36:53.048 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-05-27T07:36:53.048 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-05-27T07:36:53.048 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-05-27T07:36:53.048 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-05-27T07:36:53.049 INFO:teuthology.orchestra.run.smithi052.stdout: "username": { 2024-05-27T07:36:53.049 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "username", 2024-05-27T07:36:53.049 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "str", 2024-05-27T07:36:53.049 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-05-27T07:36:53.049 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 0, 2024-05-27T07:36:53.049 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "", 2024-05-27T07:36:53.049 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-05-27T07:36:53.049 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-05-27T07:36:53.049 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-05-27T07:36:53.049 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "username of InfluxDB server user", 2024-05-27T07:36:53.049 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-05-27T07:36:53.049 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-05-27T07:36:53.049 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-05-27T07:36:53.050 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-05-27T07:36:53.050 INFO:teuthology.orchestra.run.smithi052.stdout: "verify_ssl": { 2024-05-27T07:36:53.050 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "verify_ssl", 2024-05-27T07:36:53.050 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "str", 2024-05-27T07:36:53.050 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-05-27T07:36:53.050 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 0, 2024-05-27T07:36:53.050 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "true", 2024-05-27T07:36:53.050 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-05-27T07:36:53.050 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-05-27T07:36:53.050 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-05-27T07:36:53.050 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "Verify https cert for InfluxDB server. Use \"true\" or \"false\".", 2024-05-27T07:36:53.050 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-05-27T07:36:53.050 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-05-27T07:36:53.051 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-05-27T07:36:53.051 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-05-27T07:36:53.051 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-05-27T07:36:53.051 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-05-27T07:36:53.051 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-05-27T07:36:53.051 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "insights", 2024-05-27T07:36:53.051 INFO:teuthology.orchestra.run.smithi052.stdout: "can_run": true, 2024-05-27T07:36:53.051 INFO:teuthology.orchestra.run.smithi052.stdout: "error_string": "", 2024-05-27T07:36:53.051 INFO:teuthology.orchestra.run.smithi052.stdout: "module_options": { 2024-05-27T07:36:53.051 INFO:teuthology.orchestra.run.smithi052.stdout: "log_level": { 2024-05-27T07:36:53.051 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "log_level", 2024-05-27T07:36:53.051 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "str", 2024-05-27T07:36:53.051 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-05-27T07:36:53.052 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 1, 2024-05-27T07:36:53.052 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "", 2024-05-27T07:36:53.052 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-05-27T07:36:53.052 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-05-27T07:36:53.052 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [ 2024-05-27T07:36:53.052 INFO:teuthology.orchestra.run.smithi052.stdout: "", 2024-05-27T07:36:53.052 INFO:teuthology.orchestra.run.smithi052.stdout: "critical", 2024-05-27T07:36:53.052 INFO:teuthology.orchestra.run.smithi052.stdout: "debug", 2024-05-27T07:36:53.052 INFO:teuthology.orchestra.run.smithi052.stdout: "error", 2024-05-27T07:36:53.052 INFO:teuthology.orchestra.run.smithi052.stdout: "info", 2024-05-27T07:36:53.052 INFO:teuthology.orchestra.run.smithi052.stdout: "warning" 2024-05-27T07:36:53.052 INFO:teuthology.orchestra.run.smithi052.stdout: ], 2024-05-27T07:36:53.053 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-05-27T07:36:53.053 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-05-27T07:36:53.053 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-05-27T07:36:53.053 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-05-27T07:36:53.053 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-05-27T07:36:53.053 INFO:teuthology.orchestra.run.smithi052.stdout: "log_to_cluster": { 2024-05-27T07:36:53.053 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "log_to_cluster", 2024-05-27T07:36:53.053 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "bool", 2024-05-27T07:36:53.053 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-05-27T07:36:53.053 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 1, 2024-05-27T07:36:53.053 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "False", 2024-05-27T07:36:53.053 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-05-27T07:36:53.053 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-05-27T07:36:53.054 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-05-27T07:36:53.054 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-05-27T07:36:53.054 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-05-27T07:36:53.054 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-05-27T07:36:53.054 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-05-27T07:36:53.054 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-05-27T07:36:53.054 INFO:teuthology.orchestra.run.smithi052.stdout: "log_to_cluster_level": { 2024-05-27T07:36:53.054 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "log_to_cluster_level", 2024-05-27T07:36:53.054 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "str", 2024-05-27T07:36:53.054 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-05-27T07:36:53.054 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 1, 2024-05-27T07:36:53.054 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "info", 2024-05-27T07:36:53.054 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-05-27T07:36:53.055 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-05-27T07:36:53.055 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [ 2024-05-27T07:36:53.055 INFO:teuthology.orchestra.run.smithi052.stdout: "", 2024-05-27T07:36:53.055 INFO:teuthology.orchestra.run.smithi052.stdout: "critical", 2024-05-27T07:36:53.055 INFO:teuthology.orchestra.run.smithi052.stdout: "debug", 2024-05-27T07:36:53.055 INFO:teuthology.orchestra.run.smithi052.stdout: "error", 2024-05-27T07:36:53.055 INFO:teuthology.orchestra.run.smithi052.stdout: "info", 2024-05-27T07:36:53.055 INFO:teuthology.orchestra.run.smithi052.stdout: "warning" 2024-05-27T07:36:53.055 INFO:teuthology.orchestra.run.smithi052.stdout: ], 2024-05-27T07:36:53.055 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-05-27T07:36:53.055 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-05-27T07:36:53.055 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-05-27T07:36:53.056 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-05-27T07:36:53.056 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-05-27T07:36:53.056 INFO:teuthology.orchestra.run.smithi052.stdout: "log_to_file": { 2024-05-27T07:36:53.056 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "log_to_file", 2024-05-27T07:36:53.056 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "bool", 2024-05-27T07:36:53.056 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-05-27T07:36:53.056 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 1, 2024-05-27T07:36:53.056 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "False", 2024-05-27T07:36:53.056 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-05-27T07:36:53.056 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-05-27T07:36:53.056 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-05-27T07:36:53.056 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-05-27T07:36:53.056 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-05-27T07:36:53.056 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-05-27T07:36:53.057 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-05-27T07:36:53.057 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-05-27T07:36:53.057 INFO:teuthology.orchestra.run.smithi052.stdout: "sqlite3_killpoint": { 2024-05-27T07:36:53.057 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "sqlite3_killpoint", 2024-05-27T07:36:53.057 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "int", 2024-05-27T07:36:53.057 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "dev", 2024-05-27T07:36:53.057 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 1, 2024-05-27T07:36:53.057 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "0", 2024-05-27T07:36:53.057 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-05-27T07:36:53.057 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-05-27T07:36:53.057 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-05-27T07:36:53.057 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-05-27T07:36:53.057 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-05-27T07:36:53.058 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-05-27T07:36:53.058 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-05-27T07:36:53.058 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-05-27T07:36:53.058 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-05-27T07:36:53.058 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-05-27T07:36:53.058 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-05-27T07:36:53.058 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "iostat", 2024-05-27T07:36:53.058 INFO:teuthology.orchestra.run.smithi052.stdout: "can_run": true, 2024-05-27T07:36:53.058 INFO:teuthology.orchestra.run.smithi052.stdout: "error_string": "", 2024-05-27T07:36:53.058 INFO:teuthology.orchestra.run.smithi052.stdout: "module_options": { 2024-05-27T07:36:53.058 INFO:teuthology.orchestra.run.smithi052.stdout: "log_level": { 2024-05-27T07:36:53.058 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "log_level", 2024-05-27T07:36:53.058 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "str", 2024-05-27T07:36:53.059 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-05-27T07:36:53.059 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 1, 2024-05-27T07:36:53.059 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "", 2024-05-27T07:36:53.059 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-05-27T07:36:53.059 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-05-27T07:36:53.059 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [ 2024-05-27T07:36:53.059 INFO:teuthology.orchestra.run.smithi052.stdout: "", 2024-05-27T07:36:53.059 INFO:teuthology.orchestra.run.smithi052.stdout: "critical", 2024-05-27T07:36:53.059 INFO:teuthology.orchestra.run.smithi052.stdout: "debug", 2024-05-27T07:36:53.059 INFO:teuthology.orchestra.run.smithi052.stdout: "error", 2024-05-27T07:36:53.059 INFO:teuthology.orchestra.run.smithi052.stdout: "info", 2024-05-27T07:36:53.059 INFO:teuthology.orchestra.run.smithi052.stdout: "warning" 2024-05-27T07:36:53.059 INFO:teuthology.orchestra.run.smithi052.stdout: ], 2024-05-27T07:36:53.060 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-05-27T07:36:53.060 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-05-27T07:36:53.060 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-05-27T07:36:53.060 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-05-27T07:36:53.060 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-05-27T07:36:53.060 INFO:teuthology.orchestra.run.smithi052.stdout: "log_to_cluster": { 2024-05-27T07:36:53.060 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "log_to_cluster", 2024-05-27T07:36:53.060 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "bool", 2024-05-27T07:36:53.060 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-05-27T07:36:53.060 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 1, 2024-05-27T07:36:53.060 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "False", 2024-05-27T07:36:53.060 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-05-27T07:36:53.060 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-05-27T07:36:53.061 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-05-27T07:36:53.061 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-05-27T07:36:53.061 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-05-27T07:36:53.061 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-05-27T07:36:53.061 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-05-27T07:36:53.061 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-05-27T07:36:53.061 INFO:teuthology.orchestra.run.smithi052.stdout: "log_to_cluster_level": { 2024-05-27T07:36:53.061 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "log_to_cluster_level", 2024-05-27T07:36:53.061 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "str", 2024-05-27T07:36:53.061 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-05-27T07:36:53.061 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 1, 2024-05-27T07:36:53.061 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "info", 2024-05-27T07:36:53.061 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-05-27T07:36:53.062 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-05-27T07:36:53.062 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [ 2024-05-27T07:36:53.062 INFO:teuthology.orchestra.run.smithi052.stdout: "", 2024-05-27T07:36:53.062 INFO:teuthology.orchestra.run.smithi052.stdout: "critical", 2024-05-27T07:36:53.062 INFO:teuthology.orchestra.run.smithi052.stdout: "debug", 2024-05-27T07:36:53.062 INFO:teuthology.orchestra.run.smithi052.stdout: "error", 2024-05-27T07:36:53.062 INFO:teuthology.orchestra.run.smithi052.stdout: "info", 2024-05-27T07:36:53.062 INFO:teuthology.orchestra.run.smithi052.stdout: "warning" 2024-05-27T07:36:53.062 INFO:teuthology.orchestra.run.smithi052.stdout: ], 2024-05-27T07:36:53.062 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-05-27T07:36:53.062 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-05-27T07:36:53.062 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-05-27T07:36:53.062 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-05-27T07:36:53.063 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-05-27T07:36:53.063 INFO:teuthology.orchestra.run.smithi052.stdout: "log_to_file": { 2024-05-27T07:36:53.063 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "log_to_file", 2024-05-27T07:36:53.063 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "bool", 2024-05-27T07:36:53.063 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-05-27T07:36:53.063 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 1, 2024-05-27T07:36:53.063 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "False", 2024-05-27T07:36:53.063 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-05-27T07:36:53.063 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-05-27T07:36:53.063 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-05-27T07:36:53.063 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-05-27T07:36:53.063 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-05-27T07:36:53.063 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-05-27T07:36:53.064 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-05-27T07:36:53.064 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-05-27T07:36:53.064 INFO:teuthology.orchestra.run.smithi052.stdout: "sqlite3_killpoint": { 2024-05-27T07:36:53.064 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "sqlite3_killpoint", 2024-05-27T07:36:53.064 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "int", 2024-05-27T07:36:53.064 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "dev", 2024-05-27T07:36:53.064 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 1, 2024-05-27T07:36:53.064 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "0", 2024-05-27T07:36:53.064 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-05-27T07:36:53.064 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-05-27T07:36:53.064 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-05-27T07:36:53.064 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-05-27T07:36:53.065 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-05-27T07:36:53.065 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-05-27T07:36:53.065 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-05-27T07:36:53.065 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-05-27T07:36:53.065 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-05-27T07:36:53.065 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-05-27T07:36:53.065 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-05-27T07:36:53.065 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "k8sevents", 2024-05-27T07:36:53.065 INFO:teuthology.orchestra.run.smithi052.stdout: "can_run": true, 2024-05-27T07:36:53.065 INFO:teuthology.orchestra.run.smithi052.stdout: "error_string": "", 2024-05-27T07:36:53.065 INFO:teuthology.orchestra.run.smithi052.stdout: "module_options": { 2024-05-27T07:36:53.065 INFO:teuthology.orchestra.run.smithi052.stdout: "ceph_event_retention_days": { 2024-05-27T07:36:53.065 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "ceph_event_retention_days", 2024-05-27T07:36:53.066 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "int", 2024-05-27T07:36:53.066 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-05-27T07:36:53.066 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 0, 2024-05-27T07:36:53.066 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "7", 2024-05-27T07:36:53.066 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-05-27T07:36:53.066 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-05-27T07:36:53.066 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-05-27T07:36:53.066 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "Days to hold ceph event information within local cache", 2024-05-27T07:36:53.066 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-05-27T07:36:53.066 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-05-27T07:36:53.066 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-05-27T07:36:53.066 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-05-27T07:36:53.066 INFO:teuthology.orchestra.run.smithi052.stdout: "config_check_secs": { 2024-05-27T07:36:53.067 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "config_check_secs", 2024-05-27T07:36:53.067 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "int", 2024-05-27T07:36:53.067 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-05-27T07:36:53.067 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 0, 2024-05-27T07:36:53.067 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "10", 2024-05-27T07:36:53.067 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "10", 2024-05-27T07:36:53.067 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-05-27T07:36:53.067 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-05-27T07:36:53.067 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "interval (secs) to check for cluster configuration changes", 2024-05-27T07:36:53.067 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-05-27T07:36:53.067 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-05-27T07:36:53.067 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-05-27T07:36:53.067 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-05-27T07:36:53.068 INFO:teuthology.orchestra.run.smithi052.stdout: "log_level": { 2024-05-27T07:36:53.068 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "log_level", 2024-05-27T07:36:53.068 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "str", 2024-05-27T07:36:53.068 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-05-27T07:36:53.068 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 1, 2024-05-27T07:36:53.068 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "", 2024-05-27T07:36:53.068 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-05-27T07:36:53.068 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-05-27T07:36:53.068 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [ 2024-05-27T07:36:53.068 INFO:teuthology.orchestra.run.smithi052.stdout: "", 2024-05-27T07:36:53.068 INFO:teuthology.orchestra.run.smithi052.stdout: "critical", 2024-05-27T07:36:53.068 INFO:teuthology.orchestra.run.smithi052.stdout: "debug", 2024-05-27T07:36:53.068 INFO:teuthology.orchestra.run.smithi052.stdout: "error", 2024-05-27T07:36:53.069 INFO:teuthology.orchestra.run.smithi052.stdout: "info", 2024-05-27T07:36:53.069 INFO:teuthology.orchestra.run.smithi052.stdout: "warning" 2024-05-27T07:36:53.069 INFO:teuthology.orchestra.run.smithi052.stdout: ], 2024-05-27T07:36:53.069 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-05-27T07:36:53.069 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-05-27T07:36:53.069 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-05-27T07:36:53.069 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-05-27T07:36:53.069 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-05-27T07:36:53.069 INFO:teuthology.orchestra.run.smithi052.stdout: "log_to_cluster": { 2024-05-27T07:36:53.069 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "log_to_cluster", 2024-05-27T07:36:53.069 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "bool", 2024-05-27T07:36:53.069 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-05-27T07:36:53.069 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 1, 2024-05-27T07:36:53.070 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "False", 2024-05-27T07:36:53.070 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-05-27T07:36:53.070 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-05-27T07:36:53.070 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-05-27T07:36:53.070 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-05-27T07:36:53.070 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-05-27T07:36:53.070 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-05-27T07:36:53.070 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-05-27T07:36:53.070 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-05-27T07:36:53.070 INFO:teuthology.orchestra.run.smithi052.stdout: "log_to_cluster_level": { 2024-05-27T07:36:53.070 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "log_to_cluster_level", 2024-05-27T07:36:53.070 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "str", 2024-05-27T07:36:53.070 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-05-27T07:36:53.071 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 1, 2024-05-27T07:36:53.071 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "info", 2024-05-27T07:36:53.071 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-05-27T07:36:53.071 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-05-27T07:36:53.071 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [ 2024-05-27T07:36:53.071 INFO:teuthology.orchestra.run.smithi052.stdout: "", 2024-05-27T07:36:53.071 INFO:teuthology.orchestra.run.smithi052.stdout: "critical", 2024-05-27T07:36:53.071 INFO:teuthology.orchestra.run.smithi052.stdout: "debug", 2024-05-27T07:36:53.071 INFO:teuthology.orchestra.run.smithi052.stdout: "error", 2024-05-27T07:36:53.071 INFO:teuthology.orchestra.run.smithi052.stdout: "info", 2024-05-27T07:36:53.071 INFO:teuthology.orchestra.run.smithi052.stdout: "warning" 2024-05-27T07:36:53.071 INFO:teuthology.orchestra.run.smithi052.stdout: ], 2024-05-27T07:36:53.071 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-05-27T07:36:53.071 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-05-27T07:36:53.072 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-05-27T07:36:53.072 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-05-27T07:36:53.072 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-05-27T07:36:53.072 INFO:teuthology.orchestra.run.smithi052.stdout: "log_to_file": { 2024-05-27T07:36:53.072 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "log_to_file", 2024-05-27T07:36:53.072 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "bool", 2024-05-27T07:36:53.072 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-05-27T07:36:53.072 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 1, 2024-05-27T07:36:53.072 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "False", 2024-05-27T07:36:53.072 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-05-27T07:36:53.072 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-05-27T07:36:53.072 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-05-27T07:36:53.072 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-05-27T07:36:53.072 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-05-27T07:36:53.073 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-05-27T07:36:53.073 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-05-27T07:36:53.073 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-05-27T07:36:53.073 INFO:teuthology.orchestra.run.smithi052.stdout: "sqlite3_killpoint": { 2024-05-27T07:36:53.073 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "sqlite3_killpoint", 2024-05-27T07:36:53.073 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "int", 2024-05-27T07:36:53.073 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "dev", 2024-05-27T07:36:53.073 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 1, 2024-05-27T07:36:53.073 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "0", 2024-05-27T07:36:53.073 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-05-27T07:36:53.073 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-05-27T07:36:53.073 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-05-27T07:36:53.074 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-05-27T07:36:53.074 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-05-27T07:36:53.074 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-05-27T07:36:53.074 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-05-27T07:36:53.074 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-05-27T07:36:53.074 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-05-27T07:36:53.074 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-05-27T07:36:53.074 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-05-27T07:36:53.074 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "localpool", 2024-05-27T07:36:53.074 INFO:teuthology.orchestra.run.smithi052.stdout: "can_run": true, 2024-05-27T07:36:53.074 INFO:teuthology.orchestra.run.smithi052.stdout: "error_string": "", 2024-05-27T07:36:53.074 INFO:teuthology.orchestra.run.smithi052.stdout: "module_options": { 2024-05-27T07:36:53.075 INFO:teuthology.orchestra.run.smithi052.stdout: "failure_domain": { 2024-05-27T07:36:53.075 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "failure_domain", 2024-05-27T07:36:53.075 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "str", 2024-05-27T07:36:53.075 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-05-27T07:36:53.075 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 1, 2024-05-27T07:36:53.075 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "host", 2024-05-27T07:36:53.075 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-05-27T07:36:53.075 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-05-27T07:36:53.075 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-05-27T07:36:53.075 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "failure domain for any created local pool", 2024-05-27T07:36:53.075 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "what failure domain we should separate data replicas across.", 2024-05-27T07:36:53.075 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-05-27T07:36:53.075 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-05-27T07:36:53.076 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-05-27T07:36:53.076 INFO:teuthology.orchestra.run.smithi052.stdout: "log_level": { 2024-05-27T07:36:53.076 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "log_level", 2024-05-27T07:36:53.076 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "str", 2024-05-27T07:36:53.076 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-05-27T07:36:53.076 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 1, 2024-05-27T07:36:53.076 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "", 2024-05-27T07:36:53.076 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-05-27T07:36:53.076 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-05-27T07:36:53.076 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [ 2024-05-27T07:36:53.076 INFO:teuthology.orchestra.run.smithi052.stdout: "", 2024-05-27T07:36:53.076 INFO:teuthology.orchestra.run.smithi052.stdout: "critical", 2024-05-27T07:36:53.077 INFO:teuthology.orchestra.run.smithi052.stdout: "debug", 2024-05-27T07:36:53.077 INFO:teuthology.orchestra.run.smithi052.stdout: "error", 2024-05-27T07:36:53.077 INFO:teuthology.orchestra.run.smithi052.stdout: "info", 2024-05-27T07:36:53.077 INFO:teuthology.orchestra.run.smithi052.stdout: "warning" 2024-05-27T07:36:53.077 INFO:teuthology.orchestra.run.smithi052.stdout: ], 2024-05-27T07:36:53.077 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-05-27T07:36:53.077 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-05-27T07:36:53.077 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-05-27T07:36:53.077 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-05-27T07:36:53.077 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-05-27T07:36:53.077 INFO:teuthology.orchestra.run.smithi052.stdout: "log_to_cluster": { 2024-05-27T07:36:53.077 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "log_to_cluster", 2024-05-27T07:36:53.077 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "bool", 2024-05-27T07:36:53.077 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-05-27T07:36:53.078 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 1, 2024-05-27T07:36:53.078 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "False", 2024-05-27T07:36:53.078 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-05-27T07:36:53.078 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-05-27T07:36:53.078 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-05-27T07:36:53.078 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-05-27T07:36:53.078 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-05-27T07:36:53.078 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-05-27T07:36:53.078 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-05-27T07:36:53.078 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-05-27T07:36:53.078 INFO:teuthology.orchestra.run.smithi052.stdout: "log_to_cluster_level": { 2024-05-27T07:36:53.078 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "log_to_cluster_level", 2024-05-27T07:36:53.079 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "str", 2024-05-27T07:36:53.079 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-05-27T07:36:53.079 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 1, 2024-05-27T07:36:53.079 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "info", 2024-05-27T07:36:53.079 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-05-27T07:36:53.079 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-05-27T07:36:53.079 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [ 2024-05-27T07:36:53.079 INFO:teuthology.orchestra.run.smithi052.stdout: "", 2024-05-27T07:36:53.079 INFO:teuthology.orchestra.run.smithi052.stdout: "critical", 2024-05-27T07:36:53.079 INFO:teuthology.orchestra.run.smithi052.stdout: "debug", 2024-05-27T07:36:53.079 INFO:teuthology.orchestra.run.smithi052.stdout: "error", 2024-05-27T07:36:53.079 INFO:teuthology.orchestra.run.smithi052.stdout: "info", 2024-05-27T07:36:53.079 INFO:teuthology.orchestra.run.smithi052.stdout: "warning" 2024-05-27T07:36:53.080 INFO:teuthology.orchestra.run.smithi052.stdout: ], 2024-05-27T07:36:53.080 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-05-27T07:36:53.080 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-05-27T07:36:53.080 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-05-27T07:36:53.080 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-05-27T07:36:53.080 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-05-27T07:36:53.080 INFO:teuthology.orchestra.run.smithi052.stdout: "log_to_file": { 2024-05-27T07:36:53.080 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "log_to_file", 2024-05-27T07:36:53.080 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "bool", 2024-05-27T07:36:53.080 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-05-27T07:36:53.080 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 1, 2024-05-27T07:36:53.080 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "False", 2024-05-27T07:36:53.080 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-05-27T07:36:53.080 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-05-27T07:36:53.081 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-05-27T07:36:53.081 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-05-27T07:36:53.081 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-05-27T07:36:53.081 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-05-27T07:36:53.081 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-05-27T07:36:53.081 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-05-27T07:36:53.081 INFO:teuthology.orchestra.run.smithi052.stdout: "min_size": { 2024-05-27T07:36:53.081 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "min_size", 2024-05-27T07:36:53.081 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "int", 2024-05-27T07:36:53.081 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-05-27T07:36:53.081 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 1, 2024-05-27T07:36:53.081 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "", 2024-05-27T07:36:53.082 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-05-27T07:36:53.082 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-05-27T07:36:53.082 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-05-27T07:36:53.082 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "default min_size for any created local pool", 2024-05-27T07:36:53.082 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "value to set min_size to (unchanged from Ceph's default if this option is not set)", 2024-05-27T07:36:53.082 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-05-27T07:36:53.082 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-05-27T07:36:53.082 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-05-27T07:36:53.082 INFO:teuthology.orchestra.run.smithi052.stdout: "num_rep": { 2024-05-27T07:36:53.082 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "num_rep", 2024-05-27T07:36:53.082 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "int", 2024-05-27T07:36:53.082 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-05-27T07:36:53.082 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 1, 2024-05-27T07:36:53.083 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "3", 2024-05-27T07:36:53.083 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-05-27T07:36:53.083 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-05-27T07:36:53.083 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-05-27T07:36:53.083 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "default replica count for any created local pool", 2024-05-27T07:36:53.083 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-05-27T07:36:53.083 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-05-27T07:36:53.083 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-05-27T07:36:53.083 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-05-27T07:36:53.083 INFO:teuthology.orchestra.run.smithi052.stdout: "pg_num": { 2024-05-27T07:36:53.083 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "pg_num", 2024-05-27T07:36:53.083 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "int", 2024-05-27T07:36:53.084 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-05-27T07:36:53.084 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 1, 2024-05-27T07:36:53.084 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "128", 2024-05-27T07:36:53.084 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-05-27T07:36:53.084 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-05-27T07:36:53.084 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-05-27T07:36:53.084 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "default pg_num for any created local pool", 2024-05-27T07:36:53.084 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-05-27T07:36:53.084 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-05-27T07:36:53.084 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-05-27T07:36:53.084 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-05-27T07:36:53.084 INFO:teuthology.orchestra.run.smithi052.stdout: "prefix": { 2024-05-27T07:36:53.084 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "prefix", 2024-05-27T07:36:53.085 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "str", 2024-05-27T07:36:53.085 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-05-27T07:36:53.085 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 1, 2024-05-27T07:36:53.085 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "", 2024-05-27T07:36:53.085 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-05-27T07:36:53.085 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-05-27T07:36:53.085 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-05-27T07:36:53.085 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "name prefix for any created local pool", 2024-05-27T07:36:53.085 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-05-27T07:36:53.085 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-05-27T07:36:53.085 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-05-27T07:36:53.085 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-05-27T07:36:53.085 INFO:teuthology.orchestra.run.smithi052.stdout: "sqlite3_killpoint": { 2024-05-27T07:36:53.086 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "sqlite3_killpoint", 2024-05-27T07:36:53.086 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "int", 2024-05-27T07:36:53.086 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "dev", 2024-05-27T07:36:53.086 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 1, 2024-05-27T07:36:53.086 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "0", 2024-05-27T07:36:53.086 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-05-27T07:36:53.086 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-05-27T07:36:53.086 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-05-27T07:36:53.086 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-05-27T07:36:53.086 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-05-27T07:36:53.086 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-05-27T07:36:53.086 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-05-27T07:36:53.086 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-05-27T07:36:53.087 INFO:teuthology.orchestra.run.smithi052.stdout: "subtree": { 2024-05-27T07:36:53.087 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "subtree", 2024-05-27T07:36:53.087 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "str", 2024-05-27T07:36:53.087 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-05-27T07:36:53.087 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 1, 2024-05-27T07:36:53.087 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "rack", 2024-05-27T07:36:53.087 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-05-27T07:36:53.087 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-05-27T07:36:53.087 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-05-27T07:36:53.087 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "CRUSH level for which to create a local pool", 2024-05-27T07:36:53.087 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "which CRUSH subtree type the module should create a pool for.", 2024-05-27T07:36:53.087 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-05-27T07:36:53.088 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-05-27T07:36:53.088 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-05-27T07:36:53.088 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-05-27T07:36:53.088 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-05-27T07:36:53.088 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-05-27T07:36:53.088 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "mds_autoscaler", 2024-05-27T07:36:53.088 INFO:teuthology.orchestra.run.smithi052.stdout: "can_run": true, 2024-05-27T07:36:53.088 INFO:teuthology.orchestra.run.smithi052.stdout: "error_string": "", 2024-05-27T07:36:53.088 INFO:teuthology.orchestra.run.smithi052.stdout: "module_options": { 2024-05-27T07:36:53.088 INFO:teuthology.orchestra.run.smithi052.stdout: "log_level": { 2024-05-27T07:36:53.088 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "log_level", 2024-05-27T07:36:53.088 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "str", 2024-05-27T07:36:53.088 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-05-27T07:36:53.089 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 1, 2024-05-27T07:36:53.089 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "", 2024-05-27T07:36:53.089 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-05-27T07:36:53.089 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-05-27T07:36:53.089 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [ 2024-05-27T07:36:53.089 INFO:teuthology.orchestra.run.smithi052.stdout: "", 2024-05-27T07:36:53.089 INFO:teuthology.orchestra.run.smithi052.stdout: "critical", 2024-05-27T07:36:53.089 INFO:teuthology.orchestra.run.smithi052.stdout: "debug", 2024-05-27T07:36:53.089 INFO:teuthology.orchestra.run.smithi052.stdout: "error", 2024-05-27T07:36:53.089 INFO:teuthology.orchestra.run.smithi052.stdout: "info", 2024-05-27T07:36:53.089 INFO:teuthology.orchestra.run.smithi052.stdout: "warning" 2024-05-27T07:36:53.089 INFO:teuthology.orchestra.run.smithi052.stdout: ], 2024-05-27T07:36:53.089 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-05-27T07:36:53.090 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-05-27T07:36:53.090 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-05-27T07:36:53.090 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-05-27T07:36:53.090 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-05-27T07:36:53.090 INFO:teuthology.orchestra.run.smithi052.stdout: "log_to_cluster": { 2024-05-27T07:36:53.090 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "log_to_cluster", 2024-05-27T07:36:53.090 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "bool", 2024-05-27T07:36:53.090 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-05-27T07:36:53.090 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 1, 2024-05-27T07:36:53.090 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "False", 2024-05-27T07:36:53.090 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-05-27T07:36:53.090 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-05-27T07:36:53.091 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-05-27T07:36:53.091 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-05-27T07:36:53.091 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-05-27T07:36:53.091 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-05-27T07:36:53.091 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-05-27T07:36:53.091 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-05-27T07:36:53.091 INFO:teuthology.orchestra.run.smithi052.stdout: "log_to_cluster_level": { 2024-05-27T07:36:53.091 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "log_to_cluster_level", 2024-05-27T07:36:53.091 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "str", 2024-05-27T07:36:53.091 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-05-27T07:36:53.091 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 1, 2024-05-27T07:36:53.091 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "info", 2024-05-27T07:36:53.091 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-05-27T07:36:53.091 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-05-27T07:36:53.092 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [ 2024-05-27T07:36:53.092 INFO:teuthology.orchestra.run.smithi052.stdout: "", 2024-05-27T07:36:53.092 INFO:teuthology.orchestra.run.smithi052.stdout: "critical", 2024-05-27T07:36:53.092 INFO:teuthology.orchestra.run.smithi052.stdout: "debug", 2024-05-27T07:36:53.092 INFO:teuthology.orchestra.run.smithi052.stdout: "error", 2024-05-27T07:36:53.092 INFO:teuthology.orchestra.run.smithi052.stdout: "info", 2024-05-27T07:36:53.092 INFO:teuthology.orchestra.run.smithi052.stdout: "warning" 2024-05-27T07:36:53.092 INFO:teuthology.orchestra.run.smithi052.stdout: ], 2024-05-27T07:36:53.092 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-05-27T07:36:53.092 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-05-27T07:36:53.092 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-05-27T07:36:53.092 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-05-27T07:36:53.092 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-05-27T07:36:53.093 INFO:teuthology.orchestra.run.smithi052.stdout: "log_to_file": { 2024-05-27T07:36:53.093 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "log_to_file", 2024-05-27T07:36:53.093 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "bool", 2024-05-27T07:36:53.093 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-05-27T07:36:53.093 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 1, 2024-05-27T07:36:53.093 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "False", 2024-05-27T07:36:53.093 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-05-27T07:36:53.093 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-05-27T07:36:53.093 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-05-27T07:36:53.093 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-05-27T07:36:53.093 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-05-27T07:36:53.093 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-05-27T07:36:53.093 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-05-27T07:36:53.094 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-05-27T07:36:53.094 INFO:teuthology.orchestra.run.smithi052.stdout: "sqlite3_killpoint": { 2024-05-27T07:36:53.094 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "sqlite3_killpoint", 2024-05-27T07:36:53.094 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "int", 2024-05-27T07:36:53.094 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "dev", 2024-05-27T07:36:53.094 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 1, 2024-05-27T07:36:53.094 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "0", 2024-05-27T07:36:53.094 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-05-27T07:36:53.094 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-05-27T07:36:53.094 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-05-27T07:36:53.094 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-05-27T07:36:53.094 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-05-27T07:36:53.094 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-05-27T07:36:53.095 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-05-27T07:36:53.095 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-05-27T07:36:53.095 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-05-27T07:36:53.095 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-05-27T07:36:53.095 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-05-27T07:36:53.095 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "mirroring", 2024-05-27T07:36:53.095 INFO:teuthology.orchestra.run.smithi052.stdout: "can_run": true, 2024-05-27T07:36:53.095 INFO:teuthology.orchestra.run.smithi052.stdout: "error_string": "", 2024-05-27T07:36:53.095 INFO:teuthology.orchestra.run.smithi052.stdout: "module_options": { 2024-05-27T07:36:53.095 INFO:teuthology.orchestra.run.smithi052.stdout: "log_level": { 2024-05-27T07:36:53.095 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "log_level", 2024-05-27T07:36:53.095 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "str", 2024-05-27T07:36:53.095 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-05-27T07:36:53.096 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 1, 2024-05-27T07:36:53.096 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "", 2024-05-27T07:36:53.096 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-05-27T07:36:53.096 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-05-27T07:36:53.096 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [ 2024-05-27T07:36:53.096 INFO:teuthology.orchestra.run.smithi052.stdout: "", 2024-05-27T07:36:53.096 INFO:teuthology.orchestra.run.smithi052.stdout: "critical", 2024-05-27T07:36:53.096 INFO:teuthology.orchestra.run.smithi052.stdout: "debug", 2024-05-27T07:36:53.096 INFO:teuthology.orchestra.run.smithi052.stdout: "error", 2024-05-27T07:36:53.096 INFO:teuthology.orchestra.run.smithi052.stdout: "info", 2024-05-27T07:36:53.096 INFO:teuthology.orchestra.run.smithi052.stdout: "warning" 2024-05-27T07:36:53.096 INFO:teuthology.orchestra.run.smithi052.stdout: ], 2024-05-27T07:36:53.096 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-05-27T07:36:53.097 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-05-27T07:36:53.097 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-05-27T07:36:53.097 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-05-27T07:36:53.097 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-05-27T07:36:53.097 INFO:teuthology.orchestra.run.smithi052.stdout: "log_to_cluster": { 2024-05-27T07:36:53.097 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "log_to_cluster", 2024-05-27T07:36:53.097 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "bool", 2024-05-27T07:36:53.097 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-05-27T07:36:53.097 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 1, 2024-05-27T07:36:53.097 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "False", 2024-05-27T07:36:53.097 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-05-27T07:36:53.097 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-05-27T07:36:53.097 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-05-27T07:36:53.097 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-05-27T07:36:53.098 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-05-27T07:36:53.098 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-05-27T07:36:53.098 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-05-27T07:36:53.098 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-05-27T07:36:53.098 INFO:teuthology.orchestra.run.smithi052.stdout: "log_to_cluster_level": { 2024-05-27T07:36:53.098 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "log_to_cluster_level", 2024-05-27T07:36:53.098 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "str", 2024-05-27T07:36:53.098 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-05-27T07:36:53.098 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 1, 2024-05-27T07:36:53.098 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "info", 2024-05-27T07:36:53.098 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-05-27T07:36:53.098 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-05-27T07:36:53.098 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [ 2024-05-27T07:36:53.099 INFO:teuthology.orchestra.run.smithi052.stdout: "", 2024-05-27T07:36:53.099 INFO:teuthology.orchestra.run.smithi052.stdout: "critical", 2024-05-27T07:36:53.099 INFO:teuthology.orchestra.run.smithi052.stdout: "debug", 2024-05-27T07:36:53.099 INFO:teuthology.orchestra.run.smithi052.stdout: "error", 2024-05-27T07:36:53.099 INFO:teuthology.orchestra.run.smithi052.stdout: "info", 2024-05-27T07:36:53.099 INFO:teuthology.orchestra.run.smithi052.stdout: "warning" 2024-05-27T07:36:53.099 INFO:teuthology.orchestra.run.smithi052.stdout: ], 2024-05-27T07:36:53.099 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-05-27T07:36:53.099 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-05-27T07:36:53.099 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-05-27T07:36:53.099 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-05-27T07:36:53.099 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-05-27T07:36:53.099 INFO:teuthology.orchestra.run.smithi052.stdout: "log_to_file": { 2024-05-27T07:36:53.100 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "log_to_file", 2024-05-27T07:36:53.100 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "bool", 2024-05-27T07:36:53.100 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-05-27T07:36:53.100 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 1, 2024-05-27T07:36:53.100 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "False", 2024-05-27T07:36:53.100 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-05-27T07:36:53.100 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-05-27T07:36:53.100 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-05-27T07:36:53.100 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-05-27T07:36:53.100 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-05-27T07:36:53.100 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-05-27T07:36:53.100 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-05-27T07:36:53.100 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-05-27T07:36:53.101 INFO:teuthology.orchestra.run.smithi052.stdout: "sqlite3_killpoint": { 2024-05-27T07:36:53.101 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "sqlite3_killpoint", 2024-05-27T07:36:53.101 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "int", 2024-05-27T07:36:53.101 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "dev", 2024-05-27T07:36:53.101 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 1, 2024-05-27T07:36:53.101 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "0", 2024-05-27T07:36:53.101 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-05-27T07:36:53.101 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-05-27T07:36:53.101 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-05-27T07:36:53.101 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-05-27T07:36:53.101 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-05-27T07:36:53.101 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-05-27T07:36:53.101 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-05-27T07:36:53.102 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-05-27T07:36:53.102 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-05-27T07:36:53.102 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-05-27T07:36:53.102 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-05-27T07:36:53.102 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "nfs", 2024-05-27T07:36:53.102 INFO:teuthology.orchestra.run.smithi052.stdout: "can_run": true, 2024-05-27T07:36:53.102 INFO:teuthology.orchestra.run.smithi052.stdout: "error_string": "", 2024-05-27T07:36:53.102 INFO:teuthology.orchestra.run.smithi052.stdout: "module_options": { 2024-05-27T07:36:53.102 INFO:teuthology.orchestra.run.smithi052.stdout: "log_level": { 2024-05-27T07:36:53.102 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "log_level", 2024-05-27T07:36:53.102 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "str", 2024-05-27T07:36:53.102 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-05-27T07:36:53.102 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 1, 2024-05-27T07:36:53.103 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "", 2024-05-27T07:36:53.103 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-05-27T07:36:53.103 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-05-27T07:36:53.103 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [ 2024-05-27T07:36:53.103 INFO:teuthology.orchestra.run.smithi052.stdout: "", 2024-05-27T07:36:53.103 INFO:teuthology.orchestra.run.smithi052.stdout: "critical", 2024-05-27T07:36:53.103 INFO:teuthology.orchestra.run.smithi052.stdout: "debug", 2024-05-27T07:36:53.103 INFO:teuthology.orchestra.run.smithi052.stdout: "error", 2024-05-27T07:36:53.103 INFO:teuthology.orchestra.run.smithi052.stdout: "info", 2024-05-27T07:36:53.103 INFO:teuthology.orchestra.run.smithi052.stdout: "warning" 2024-05-27T07:36:53.103 INFO:teuthology.orchestra.run.smithi052.stdout: ], 2024-05-27T07:36:53.103 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-05-27T07:36:53.103 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-05-27T07:36:53.104 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-05-27T07:36:53.104 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-05-27T07:36:53.104 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-05-27T07:36:53.104 INFO:teuthology.orchestra.run.smithi052.stdout: "log_to_cluster": { 2024-05-27T07:36:53.104 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "log_to_cluster", 2024-05-27T07:36:53.104 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "bool", 2024-05-27T07:36:53.104 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-05-27T07:36:53.104 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 1, 2024-05-27T07:36:53.104 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "False", 2024-05-27T07:36:53.104 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-05-27T07:36:53.104 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-05-27T07:36:53.104 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-05-27T07:36:53.104 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-05-27T07:36:53.104 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-05-27T07:36:53.105 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-05-27T07:36:53.105 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-05-27T07:36:53.105 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-05-27T07:36:53.105 INFO:teuthology.orchestra.run.smithi052.stdout: "log_to_cluster_level": { 2024-05-27T07:36:53.105 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "log_to_cluster_level", 2024-05-27T07:36:53.105 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "str", 2024-05-27T07:36:53.105 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-05-27T07:36:53.105 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 1, 2024-05-27T07:36:53.105 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "info", 2024-05-27T07:36:53.105 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-05-27T07:36:53.105 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-05-27T07:36:53.105 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [ 2024-05-27T07:36:53.105 INFO:teuthology.orchestra.run.smithi052.stdout: "", 2024-05-27T07:36:53.106 INFO:teuthology.orchestra.run.smithi052.stdout: "critical", 2024-05-27T07:36:53.106 INFO:teuthology.orchestra.run.smithi052.stdout: "debug", 2024-05-27T07:36:53.106 INFO:teuthology.orchestra.run.smithi052.stdout: "error", 2024-05-27T07:36:53.106 INFO:teuthology.orchestra.run.smithi052.stdout: "info", 2024-05-27T07:36:53.106 INFO:teuthology.orchestra.run.smithi052.stdout: "warning" 2024-05-27T07:36:53.106 INFO:teuthology.orchestra.run.smithi052.stdout: ], 2024-05-27T07:36:53.106 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-05-27T07:36:53.106 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-05-27T07:36:53.106 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-05-27T07:36:53.106 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-05-27T07:36:53.106 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-05-27T07:36:53.106 INFO:teuthology.orchestra.run.smithi052.stdout: "log_to_file": { 2024-05-27T07:36:53.106 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "log_to_file", 2024-05-27T07:36:53.107 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "bool", 2024-05-27T07:36:53.107 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-05-27T07:36:53.107 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 1, 2024-05-27T07:36:53.107 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "False", 2024-05-27T07:36:53.107 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-05-27T07:36:53.107 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-05-27T07:36:53.107 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-05-27T07:36:53.107 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-05-27T07:36:53.107 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-05-27T07:36:53.107 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-05-27T07:36:53.107 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-05-27T07:36:53.107 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-05-27T07:36:53.107 INFO:teuthology.orchestra.run.smithi052.stdout: "sqlite3_killpoint": { 2024-05-27T07:36:53.108 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "sqlite3_killpoint", 2024-05-27T07:36:53.108 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "int", 2024-05-27T07:36:53.108 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "dev", 2024-05-27T07:36:53.108 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 1, 2024-05-27T07:36:53.108 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "0", 2024-05-27T07:36:53.108 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-05-27T07:36:53.108 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-05-27T07:36:53.108 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-05-27T07:36:53.108 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-05-27T07:36:53.108 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-05-27T07:36:53.108 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-05-27T07:36:53.108 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-05-27T07:36:53.108 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-05-27T07:36:53.109 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-05-27T07:36:53.109 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-05-27T07:36:53.109 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-05-27T07:36:53.109 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "orchestrator", 2024-05-27T07:36:53.109 INFO:teuthology.orchestra.run.smithi052.stdout: "can_run": true, 2024-05-27T07:36:53.109 INFO:teuthology.orchestra.run.smithi052.stdout: "error_string": "", 2024-05-27T07:36:53.109 INFO:teuthology.orchestra.run.smithi052.stdout: "module_options": { 2024-05-27T07:36:53.109 INFO:teuthology.orchestra.run.smithi052.stdout: "fail_fs": { 2024-05-27T07:36:53.109 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "fail_fs", 2024-05-27T07:36:53.109 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "bool", 2024-05-27T07:36:53.109 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-05-27T07:36:53.109 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 0, 2024-05-27T07:36:53.109 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "False", 2024-05-27T07:36:53.110 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-05-27T07:36:53.110 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-05-27T07:36:53.110 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-05-27T07:36:53.110 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "Fail filesystem for rapid multi-rank mds upgrade", 2024-05-27T07:36:53.110 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-05-27T07:36:53.110 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-05-27T07:36:53.110 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-05-27T07:36:53.110 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-05-27T07:36:53.110 INFO:teuthology.orchestra.run.smithi052.stdout: "log_level": { 2024-05-27T07:36:53.110 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "log_level", 2024-05-27T07:36:53.110 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "str", 2024-05-27T07:36:53.110 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-05-27T07:36:53.111 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 1, 2024-05-27T07:36:53.111 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "", 2024-05-27T07:36:53.111 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-05-27T07:36:53.111 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-05-27T07:36:53.111 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [ 2024-05-27T07:36:53.111 INFO:teuthology.orchestra.run.smithi052.stdout: "", 2024-05-27T07:36:53.111 INFO:teuthology.orchestra.run.smithi052.stdout: "critical", 2024-05-27T07:36:53.111 INFO:teuthology.orchestra.run.smithi052.stdout: "debug", 2024-05-27T07:36:53.111 INFO:teuthology.orchestra.run.smithi052.stdout: "error", 2024-05-27T07:36:53.111 INFO:teuthology.orchestra.run.smithi052.stdout: "info", 2024-05-27T07:36:53.111 INFO:teuthology.orchestra.run.smithi052.stdout: "warning" 2024-05-27T07:36:53.111 INFO:teuthology.orchestra.run.smithi052.stdout: ], 2024-05-27T07:36:53.111 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-05-27T07:36:53.111 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-05-27T07:36:53.112 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-05-27T07:36:53.112 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-05-27T07:36:53.112 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-05-27T07:36:53.112 INFO:teuthology.orchestra.run.smithi052.stdout: "log_to_cluster": { 2024-05-27T07:36:53.112 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "log_to_cluster", 2024-05-27T07:36:53.112 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "bool", 2024-05-27T07:36:53.112 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-05-27T07:36:53.112 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 1, 2024-05-27T07:36:53.112 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "False", 2024-05-27T07:36:53.112 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-05-27T07:36:53.112 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-05-27T07:36:53.112 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-05-27T07:36:53.112 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-05-27T07:36:53.113 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-05-27T07:36:53.113 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-05-27T07:36:53.113 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-05-27T07:36:53.113 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-05-27T07:36:53.113 INFO:teuthology.orchestra.run.smithi052.stdout: "log_to_cluster_level": { 2024-05-27T07:36:53.113 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "log_to_cluster_level", 2024-05-27T07:36:53.113 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "str", 2024-05-27T07:36:53.113 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-05-27T07:36:53.113 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 1, 2024-05-27T07:36:53.113 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "info", 2024-05-27T07:36:53.113 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-05-27T07:36:53.113 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-05-27T07:36:53.113 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [ 2024-05-27T07:36:53.113 INFO:teuthology.orchestra.run.smithi052.stdout: "", 2024-05-27T07:36:53.114 INFO:teuthology.orchestra.run.smithi052.stdout: "critical", 2024-05-27T07:36:53.114 INFO:teuthology.orchestra.run.smithi052.stdout: "debug", 2024-05-27T07:36:53.114 INFO:teuthology.orchestra.run.smithi052.stdout: "error", 2024-05-27T07:36:53.114 INFO:teuthology.orchestra.run.smithi052.stdout: "info", 2024-05-27T07:36:53.114 INFO:teuthology.orchestra.run.smithi052.stdout: "warning" 2024-05-27T07:36:53.114 INFO:teuthology.orchestra.run.smithi052.stdout: ], 2024-05-27T07:36:53.114 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-05-27T07:36:53.114 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-05-27T07:36:53.114 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-05-27T07:36:53.114 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-05-27T07:36:53.114 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-05-27T07:36:53.114 INFO:teuthology.orchestra.run.smithi052.stdout: "log_to_file": { 2024-05-27T07:36:53.115 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "log_to_file", 2024-05-27T07:36:53.115 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "bool", 2024-05-27T07:36:53.115 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-05-27T07:36:53.115 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 1, 2024-05-27T07:36:53.115 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "False", 2024-05-27T07:36:53.115 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-05-27T07:36:53.115 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-05-27T07:36:53.115 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-05-27T07:36:53.115 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-05-27T07:36:53.115 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-05-27T07:36:53.115 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-05-27T07:36:53.115 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-05-27T07:36:53.115 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-05-27T07:36:53.115 INFO:teuthology.orchestra.run.smithi052.stdout: "orchestrator": { 2024-05-27T07:36:53.116 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "orchestrator", 2024-05-27T07:36:53.116 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "str", 2024-05-27T07:36:53.116 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-05-27T07:36:53.116 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 1, 2024-05-27T07:36:53.116 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "", 2024-05-27T07:36:53.116 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-05-27T07:36:53.116 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-05-27T07:36:53.116 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [ 2024-05-27T07:36:53.116 INFO:teuthology.orchestra.run.smithi052.stdout: "cephadm", 2024-05-27T07:36:53.116 INFO:teuthology.orchestra.run.smithi052.stdout: "rook", 2024-05-27T07:36:53.116 INFO:teuthology.orchestra.run.smithi052.stdout: "test_orchestrator" 2024-05-27T07:36:53.116 INFO:teuthology.orchestra.run.smithi052.stdout: ], 2024-05-27T07:36:53.116 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "Orchestrator backend", 2024-05-27T07:36:53.117 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-05-27T07:36:53.117 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-05-27T07:36:53.117 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-05-27T07:36:53.117 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-05-27T07:36:53.117 INFO:teuthology.orchestra.run.smithi052.stdout: "sqlite3_killpoint": { 2024-05-27T07:36:53.117 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "sqlite3_killpoint", 2024-05-27T07:36:53.117 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "int", 2024-05-27T07:36:53.117 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "dev", 2024-05-27T07:36:53.117 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 1, 2024-05-27T07:36:53.117 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "0", 2024-05-27T07:36:53.117 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-05-27T07:36:53.117 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-05-27T07:36:53.117 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-05-27T07:36:53.118 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-05-27T07:36:53.118 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-05-27T07:36:53.118 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-05-27T07:36:53.118 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-05-27T07:36:53.118 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-05-27T07:36:53.118 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-05-27T07:36:53.118 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-05-27T07:36:53.118 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-05-27T07:36:53.118 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "osd_perf_query", 2024-05-27T07:36:53.118 INFO:teuthology.orchestra.run.smithi052.stdout: "can_run": true, 2024-05-27T07:36:53.118 INFO:teuthology.orchestra.run.smithi052.stdout: "error_string": "", 2024-05-27T07:36:53.118 INFO:teuthology.orchestra.run.smithi052.stdout: "module_options": { 2024-05-27T07:36:53.118 INFO:teuthology.orchestra.run.smithi052.stdout: "log_level": { 2024-05-27T07:36:53.119 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "log_level", 2024-05-27T07:36:53.119 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "str", 2024-05-27T07:36:53.119 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-05-27T07:36:53.119 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 1, 2024-05-27T07:36:53.119 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "", 2024-05-27T07:36:53.119 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-05-27T07:36:53.119 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-05-27T07:36:53.119 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [ 2024-05-27T07:36:53.119 INFO:teuthology.orchestra.run.smithi052.stdout: "", 2024-05-27T07:36:53.119 INFO:teuthology.orchestra.run.smithi052.stdout: "critical", 2024-05-27T07:36:53.119 INFO:teuthology.orchestra.run.smithi052.stdout: "debug", 2024-05-27T07:36:53.119 INFO:teuthology.orchestra.run.smithi052.stdout: "error", 2024-05-27T07:36:53.119 INFO:teuthology.orchestra.run.smithi052.stdout: "info", 2024-05-27T07:36:53.120 INFO:teuthology.orchestra.run.smithi052.stdout: "warning" 2024-05-27T07:36:53.120 INFO:teuthology.orchestra.run.smithi052.stdout: ], 2024-05-27T07:36:53.120 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-05-27T07:36:53.120 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-05-27T07:36:53.120 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-05-27T07:36:53.120 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-05-27T07:36:53.120 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-05-27T07:36:53.120 INFO:teuthology.orchestra.run.smithi052.stdout: "log_to_cluster": { 2024-05-27T07:36:53.120 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "log_to_cluster", 2024-05-27T07:36:53.120 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "bool", 2024-05-27T07:36:53.120 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-05-27T07:36:53.120 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 1, 2024-05-27T07:36:53.120 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "False", 2024-05-27T07:36:53.121 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-05-27T07:36:53.121 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-05-27T07:36:53.121 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-05-27T07:36:53.121 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-05-27T07:36:53.121 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-05-27T07:36:53.121 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-05-27T07:36:53.121 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-05-27T07:36:53.121 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-05-27T07:36:53.121 INFO:teuthology.orchestra.run.smithi052.stdout: "log_to_cluster_level": { 2024-05-27T07:36:53.121 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "log_to_cluster_level", 2024-05-27T07:36:53.121 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "str", 2024-05-27T07:36:53.121 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-05-27T07:36:53.121 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 1, 2024-05-27T07:36:53.122 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "info", 2024-05-27T07:36:53.122 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-05-27T07:36:53.122 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-05-27T07:36:53.122 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [ 2024-05-27T07:36:53.122 INFO:teuthology.orchestra.run.smithi052.stdout: "", 2024-05-27T07:36:53.122 INFO:teuthology.orchestra.run.smithi052.stdout: "critical", 2024-05-27T07:36:53.122 INFO:teuthology.orchestra.run.smithi052.stdout: "debug", 2024-05-27T07:36:53.122 INFO:teuthology.orchestra.run.smithi052.stdout: "error", 2024-05-27T07:36:53.122 INFO:teuthology.orchestra.run.smithi052.stdout: "info", 2024-05-27T07:36:53.122 INFO:teuthology.orchestra.run.smithi052.stdout: "warning" 2024-05-27T07:36:53.122 INFO:teuthology.orchestra.run.smithi052.stdout: ], 2024-05-27T07:36:53.122 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-05-27T07:36:53.123 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-05-27T07:36:53.123 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-05-27T07:36:53.123 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-05-27T07:36:53.123 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-05-27T07:36:53.123 INFO:teuthology.orchestra.run.smithi052.stdout: "log_to_file": { 2024-05-27T07:36:53.123 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "log_to_file", 2024-05-27T07:36:53.123 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "bool", 2024-05-27T07:36:53.123 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-05-27T07:36:53.123 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 1, 2024-05-27T07:36:53.123 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "False", 2024-05-27T07:36:53.123 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-05-27T07:36:53.123 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-05-27T07:36:53.123 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-05-27T07:36:53.124 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-05-27T07:36:53.124 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-05-27T07:36:53.124 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-05-27T07:36:53.124 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-05-27T07:36:53.124 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-05-27T07:36:53.124 INFO:teuthology.orchestra.run.smithi052.stdout: "sqlite3_killpoint": { 2024-05-27T07:36:53.124 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "sqlite3_killpoint", 2024-05-27T07:36:53.124 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "int", 2024-05-27T07:36:53.124 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "dev", 2024-05-27T07:36:53.124 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 1, 2024-05-27T07:36:53.124 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "0", 2024-05-27T07:36:53.124 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-05-27T07:36:53.124 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-05-27T07:36:53.125 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-05-27T07:36:53.125 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-05-27T07:36:53.125 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-05-27T07:36:53.125 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-05-27T07:36:53.125 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-05-27T07:36:53.125 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-05-27T07:36:53.125 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-05-27T07:36:53.125 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-05-27T07:36:53.125 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-05-27T07:36:53.125 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "osd_support", 2024-05-27T07:36:53.125 INFO:teuthology.orchestra.run.smithi052.stdout: "can_run": true, 2024-05-27T07:36:53.125 INFO:teuthology.orchestra.run.smithi052.stdout: "error_string": "", 2024-05-27T07:36:53.125 INFO:teuthology.orchestra.run.smithi052.stdout: "module_options": { 2024-05-27T07:36:53.125 INFO:teuthology.orchestra.run.smithi052.stdout: "log_level": { 2024-05-27T07:36:53.126 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "log_level", 2024-05-27T07:36:53.126 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "str", 2024-05-27T07:36:53.126 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-05-27T07:36:53.126 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 1, 2024-05-27T07:36:53.126 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "", 2024-05-27T07:36:53.126 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-05-27T07:36:53.126 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-05-27T07:36:53.126 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [ 2024-05-27T07:36:53.126 INFO:teuthology.orchestra.run.smithi052.stdout: "", 2024-05-27T07:36:53.126 INFO:teuthology.orchestra.run.smithi052.stdout: "critical", 2024-05-27T07:36:53.126 INFO:teuthology.orchestra.run.smithi052.stdout: "debug", 2024-05-27T07:36:53.126 INFO:teuthology.orchestra.run.smithi052.stdout: "error", 2024-05-27T07:36:53.126 INFO:teuthology.orchestra.run.smithi052.stdout: "info", 2024-05-27T07:36:53.127 INFO:teuthology.orchestra.run.smithi052.stdout: "warning" 2024-05-27T07:36:53.127 INFO:teuthology.orchestra.run.smithi052.stdout: ], 2024-05-27T07:36:53.127 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-05-27T07:36:53.127 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-05-27T07:36:53.127 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-05-27T07:36:53.127 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-05-27T07:36:53.127 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-05-27T07:36:53.127 INFO:teuthology.orchestra.run.smithi052.stdout: "log_to_cluster": { 2024-05-27T07:36:53.127 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "log_to_cluster", 2024-05-27T07:36:53.127 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "bool", 2024-05-27T07:36:53.127 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-05-27T07:36:53.127 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 1, 2024-05-27T07:36:53.127 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "False", 2024-05-27T07:36:53.128 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-05-27T07:36:53.128 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-05-27T07:36:53.128 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-05-27T07:36:53.128 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-05-27T07:36:53.128 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-05-27T07:36:53.128 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-05-27T07:36:53.128 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-05-27T07:36:53.128 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-05-27T07:36:53.128 INFO:teuthology.orchestra.run.smithi052.stdout: "log_to_cluster_level": { 2024-05-27T07:36:53.128 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "log_to_cluster_level", 2024-05-27T07:36:53.128 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "str", 2024-05-27T07:36:53.128 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-05-27T07:36:53.128 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 1, 2024-05-27T07:36:53.128 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "info", 2024-05-27T07:36:53.129 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-05-27T07:36:53.129 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-05-27T07:36:53.129 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [ 2024-05-27T07:36:53.129 INFO:teuthology.orchestra.run.smithi052.stdout: "", 2024-05-27T07:36:53.129 INFO:teuthology.orchestra.run.smithi052.stdout: "critical", 2024-05-27T07:36:53.129 INFO:teuthology.orchestra.run.smithi052.stdout: "debug", 2024-05-27T07:36:53.129 INFO:teuthology.orchestra.run.smithi052.stdout: "error", 2024-05-27T07:36:53.129 INFO:teuthology.orchestra.run.smithi052.stdout: "info", 2024-05-27T07:36:53.129 INFO:teuthology.orchestra.run.smithi052.stdout: "warning" 2024-05-27T07:36:53.129 INFO:teuthology.orchestra.run.smithi052.stdout: ], 2024-05-27T07:36:53.129 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-05-27T07:36:53.129 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-05-27T07:36:53.129 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-05-27T07:36:53.130 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-05-27T07:36:53.130 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-05-27T07:36:53.130 INFO:teuthology.orchestra.run.smithi052.stdout: "log_to_file": { 2024-05-27T07:36:53.130 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "log_to_file", 2024-05-27T07:36:53.130 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "bool", 2024-05-27T07:36:53.130 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-05-27T07:36:53.130 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 1, 2024-05-27T07:36:53.130 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "False", 2024-05-27T07:36:53.130 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-05-27T07:36:53.130 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-05-27T07:36:53.130 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-05-27T07:36:53.130 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-05-27T07:36:53.130 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-05-27T07:36:53.131 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-05-27T07:36:53.131 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-05-27T07:36:53.131 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-05-27T07:36:53.131 INFO:teuthology.orchestra.run.smithi052.stdout: "sqlite3_killpoint": { 2024-05-27T07:36:53.131 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "sqlite3_killpoint", 2024-05-27T07:36:53.131 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "int", 2024-05-27T07:36:53.131 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "dev", 2024-05-27T07:36:53.131 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 1, 2024-05-27T07:36:53.131 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "0", 2024-05-27T07:36:53.131 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-05-27T07:36:53.131 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-05-27T07:36:53.131 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-05-27T07:36:53.131 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-05-27T07:36:53.131 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-05-27T07:36:53.132 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-05-27T07:36:53.132 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-05-27T07:36:53.132 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-05-27T07:36:53.132 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-05-27T07:36:53.132 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-05-27T07:36:53.132 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-05-27T07:36:53.132 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "pg_autoscaler", 2024-05-27T07:36:53.132 INFO:teuthology.orchestra.run.smithi052.stdout: "can_run": true, 2024-05-27T07:36:53.132 INFO:teuthology.orchestra.run.smithi052.stdout: "error_string": "", 2024-05-27T07:36:53.132 INFO:teuthology.orchestra.run.smithi052.stdout: "module_options": { 2024-05-27T07:36:53.132 INFO:teuthology.orchestra.run.smithi052.stdout: "log_level": { 2024-05-27T07:36:53.132 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "log_level", 2024-05-27T07:36:53.132 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "str", 2024-05-27T07:36:53.133 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-05-27T07:36:53.133 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 1, 2024-05-27T07:36:53.133 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "", 2024-05-27T07:36:53.133 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-05-27T07:36:53.133 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-05-27T07:36:53.133 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [ 2024-05-27T07:36:53.133 INFO:teuthology.orchestra.run.smithi052.stdout: "", 2024-05-27T07:36:53.133 INFO:teuthology.orchestra.run.smithi052.stdout: "critical", 2024-05-27T07:36:53.133 INFO:teuthology.orchestra.run.smithi052.stdout: "debug", 2024-05-27T07:36:53.133 INFO:teuthology.orchestra.run.smithi052.stdout: "error", 2024-05-27T07:36:53.133 INFO:teuthology.orchestra.run.smithi052.stdout: "info", 2024-05-27T07:36:53.133 INFO:teuthology.orchestra.run.smithi052.stdout: "warning" 2024-05-27T07:36:53.133 INFO:teuthology.orchestra.run.smithi052.stdout: ], 2024-05-27T07:36:53.133 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-05-27T07:36:53.134 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-05-27T07:36:53.134 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-05-27T07:36:53.134 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-05-27T07:36:53.134 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-05-27T07:36:53.134 INFO:teuthology.orchestra.run.smithi052.stdout: "log_to_cluster": { 2024-05-27T07:36:53.134 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "log_to_cluster", 2024-05-27T07:36:53.134 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "bool", 2024-05-27T07:36:53.134 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-05-27T07:36:53.134 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 1, 2024-05-27T07:36:53.134 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "False", 2024-05-27T07:36:53.134 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-05-27T07:36:53.134 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-05-27T07:36:53.134 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-05-27T07:36:53.135 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-05-27T07:36:53.135 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-05-27T07:36:53.135 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-05-27T07:36:53.135 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-05-27T07:36:53.135 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-05-27T07:36:53.135 INFO:teuthology.orchestra.run.smithi052.stdout: "log_to_cluster_level": { 2024-05-27T07:36:53.135 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "log_to_cluster_level", 2024-05-27T07:36:53.135 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "str", 2024-05-27T07:36:53.135 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-05-27T07:36:53.135 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 1, 2024-05-27T07:36:53.135 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "info", 2024-05-27T07:36:53.135 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-05-27T07:36:53.135 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-05-27T07:36:53.136 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [ 2024-05-27T07:36:53.136 INFO:teuthology.orchestra.run.smithi052.stdout: "", 2024-05-27T07:36:53.136 INFO:teuthology.orchestra.run.smithi052.stdout: "critical", 2024-05-27T07:36:53.136 INFO:teuthology.orchestra.run.smithi052.stdout: "debug", 2024-05-27T07:36:53.136 INFO:teuthology.orchestra.run.smithi052.stdout: "error", 2024-05-27T07:36:53.136 INFO:teuthology.orchestra.run.smithi052.stdout: "info", 2024-05-27T07:36:53.136 INFO:teuthology.orchestra.run.smithi052.stdout: "warning" 2024-05-27T07:36:53.136 INFO:teuthology.orchestra.run.smithi052.stdout: ], 2024-05-27T07:36:53.136 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-05-27T07:36:53.136 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-05-27T07:36:53.136 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-05-27T07:36:53.136 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-05-27T07:36:53.136 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-05-27T07:36:53.137 INFO:teuthology.orchestra.run.smithi052.stdout: "log_to_file": { 2024-05-27T07:36:53.137 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "log_to_file", 2024-05-27T07:36:53.137 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "bool", 2024-05-27T07:36:53.137 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-05-27T07:36:53.137 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 1, 2024-05-27T07:36:53.137 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "False", 2024-05-27T07:36:53.137 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-05-27T07:36:53.137 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-05-27T07:36:53.137 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-05-27T07:36:53.137 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-05-27T07:36:53.137 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-05-27T07:36:53.137 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-05-27T07:36:53.137 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-05-27T07:36:53.138 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-05-27T07:36:53.138 INFO:teuthology.orchestra.run.smithi052.stdout: "sleep_interval": { 2024-05-27T07:36:53.138 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "sleep_interval", 2024-05-27T07:36:53.138 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "secs", 2024-05-27T07:36:53.138 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-05-27T07:36:53.138 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 0, 2024-05-27T07:36:53.138 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "60", 2024-05-27T07:36:53.138 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-05-27T07:36:53.138 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-05-27T07:36:53.138 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-05-27T07:36:53.138 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-05-27T07:36:53.138 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-05-27T07:36:53.138 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-05-27T07:36:53.139 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-05-27T07:36:53.139 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-05-27T07:36:53.139 INFO:teuthology.orchestra.run.smithi052.stdout: "sqlite3_killpoint": { 2024-05-27T07:36:53.139 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "sqlite3_killpoint", 2024-05-27T07:36:53.139 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "int", 2024-05-27T07:36:53.139 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "dev", 2024-05-27T07:36:53.139 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 1, 2024-05-27T07:36:53.139 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "0", 2024-05-27T07:36:53.139 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-05-27T07:36:53.139 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-05-27T07:36:53.139 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-05-27T07:36:53.139 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-05-27T07:36:53.139 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-05-27T07:36:53.140 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-05-27T07:36:53.140 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-05-27T07:36:53.140 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-05-27T07:36:53.140 INFO:teuthology.orchestra.run.smithi052.stdout: "threshold": { 2024-05-27T07:36:53.140 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "threshold", 2024-05-27T07:36:53.140 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "float", 2024-05-27T07:36:53.140 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-05-27T07:36:53.140 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 0, 2024-05-27T07:36:53.140 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "3.0", 2024-05-27T07:36:53.140 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "1.0", 2024-05-27T07:36:53.140 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-05-27T07:36:53.140 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-05-27T07:36:53.140 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "scaling threshold", 2024-05-27T07:36:53.141 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0", 2024-05-27T07:36:53.141 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-05-27T07:36:53.141 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-05-27T07:36:53.141 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-05-27T07:36:53.141 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-05-27T07:36:53.141 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-05-27T07:36:53.141 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-05-27T07:36:53.141 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "progress", 2024-05-27T07:36:53.141 INFO:teuthology.orchestra.run.smithi052.stdout: "can_run": true, 2024-05-27T07:36:53.141 INFO:teuthology.orchestra.run.smithi052.stdout: "error_string": "", 2024-05-27T07:36:53.141 INFO:teuthology.orchestra.run.smithi052.stdout: "module_options": { 2024-05-27T07:36:53.141 INFO:teuthology.orchestra.run.smithi052.stdout: "allow_pg_recovery_event": { 2024-05-27T07:36:53.141 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "allow_pg_recovery_event", 2024-05-27T07:36:53.141 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "bool", 2024-05-27T07:36:53.142 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-05-27T07:36:53.142 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 1, 2024-05-27T07:36:53.142 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "False", 2024-05-27T07:36:53.142 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-05-27T07:36:53.142 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-05-27T07:36:53.142 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-05-27T07:36:53.142 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "allow the module to show pg recovery progress", 2024-05-27T07:36:53.142 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-05-27T07:36:53.142 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-05-27T07:36:53.142 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-05-27T07:36:53.142 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-05-27T07:36:53.142 INFO:teuthology.orchestra.run.smithi052.stdout: "enabled": { 2024-05-27T07:36:53.142 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "enabled", 2024-05-27T07:36:53.143 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "bool", 2024-05-27T07:36:53.143 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-05-27T07:36:53.143 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 0, 2024-05-27T07:36:53.143 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "True", 2024-05-27T07:36:53.143 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-05-27T07:36:53.143 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-05-27T07:36:53.143 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-05-27T07:36:53.143 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-05-27T07:36:53.143 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-05-27T07:36:53.143 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-05-27T07:36:53.143 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-05-27T07:36:53.143 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-05-27T07:36:53.143 INFO:teuthology.orchestra.run.smithi052.stdout: "log_level": { 2024-05-27T07:36:53.144 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "log_level", 2024-05-27T07:36:53.144 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "str", 2024-05-27T07:36:53.144 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-05-27T07:36:53.144 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 1, 2024-05-27T07:36:53.144 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "", 2024-05-27T07:36:53.144 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-05-27T07:36:53.144 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-05-27T07:36:53.144 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [ 2024-05-27T07:36:53.144 INFO:teuthology.orchestra.run.smithi052.stdout: "", 2024-05-27T07:36:53.144 INFO:teuthology.orchestra.run.smithi052.stdout: "critical", 2024-05-27T07:36:53.144 INFO:teuthology.orchestra.run.smithi052.stdout: "debug", 2024-05-27T07:36:53.144 INFO:teuthology.orchestra.run.smithi052.stdout: "error", 2024-05-27T07:36:53.144 INFO:teuthology.orchestra.run.smithi052.stdout: "info", 2024-05-27T07:36:53.145 INFO:teuthology.orchestra.run.smithi052.stdout: "warning" 2024-05-27T07:36:53.145 INFO:teuthology.orchestra.run.smithi052.stdout: ], 2024-05-27T07:36:53.145 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-05-27T07:36:53.145 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-05-27T07:36:53.145 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-05-27T07:36:53.145 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-05-27T07:36:53.145 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-05-27T07:36:53.145 INFO:teuthology.orchestra.run.smithi052.stdout: "log_to_cluster": { 2024-05-27T07:36:53.145 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "log_to_cluster", 2024-05-27T07:36:53.145 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "bool", 2024-05-27T07:36:53.145 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-05-27T07:36:53.145 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 1, 2024-05-27T07:36:53.145 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "False", 2024-05-27T07:36:53.145 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-05-27T07:36:53.146 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-05-27T07:36:53.146 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-05-27T07:36:53.146 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-05-27T07:36:53.146 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-05-27T07:36:53.146 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-05-27T07:36:53.146 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-05-27T07:36:53.146 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-05-27T07:36:53.146 INFO:teuthology.orchestra.run.smithi052.stdout: "log_to_cluster_level": { 2024-05-27T07:36:53.146 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "log_to_cluster_level", 2024-05-27T07:36:53.146 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "str", 2024-05-27T07:36:53.146 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-05-27T07:36:53.146 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 1, 2024-05-27T07:36:53.146 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "info", 2024-05-27T07:36:53.147 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-05-27T07:36:53.147 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-05-27T07:36:53.147 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [ 2024-05-27T07:36:53.147 INFO:teuthology.orchestra.run.smithi052.stdout: "", 2024-05-27T07:36:53.147 INFO:teuthology.orchestra.run.smithi052.stdout: "critical", 2024-05-27T07:36:53.147 INFO:teuthology.orchestra.run.smithi052.stdout: "debug", 2024-05-27T07:36:53.147 INFO:teuthology.orchestra.run.smithi052.stdout: "error", 2024-05-27T07:36:53.147 INFO:teuthology.orchestra.run.smithi052.stdout: "info", 2024-05-27T07:36:53.147 INFO:teuthology.orchestra.run.smithi052.stdout: "warning" 2024-05-27T07:36:53.147 INFO:teuthology.orchestra.run.smithi052.stdout: ], 2024-05-27T07:36:53.147 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-05-27T07:36:53.147 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-05-27T07:36:53.147 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-05-27T07:36:53.148 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-05-27T07:36:53.148 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-05-27T07:36:53.148 INFO:teuthology.orchestra.run.smithi052.stdout: "log_to_file": { 2024-05-27T07:36:53.148 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "log_to_file", 2024-05-27T07:36:53.148 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "bool", 2024-05-27T07:36:53.148 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-05-27T07:36:53.148 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 1, 2024-05-27T07:36:53.148 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "False", 2024-05-27T07:36:53.148 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-05-27T07:36:53.148 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-05-27T07:36:53.148 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-05-27T07:36:53.148 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-05-27T07:36:53.148 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-05-27T07:36:53.148 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-05-27T07:36:53.149 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-05-27T07:36:53.149 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-05-27T07:36:53.149 INFO:teuthology.orchestra.run.smithi052.stdout: "max_completed_events": { 2024-05-27T07:36:53.149 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "max_completed_events", 2024-05-27T07:36:53.149 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "int", 2024-05-27T07:36:53.149 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-05-27T07:36:53.149 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 1, 2024-05-27T07:36:53.149 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "50", 2024-05-27T07:36:53.149 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-05-27T07:36:53.149 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-05-27T07:36:53.149 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-05-27T07:36:53.149 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "number of past completed events to remember", 2024-05-27T07:36:53.149 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-05-27T07:36:53.150 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-05-27T07:36:53.150 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-05-27T07:36:53.150 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-05-27T07:36:53.150 INFO:teuthology.orchestra.run.smithi052.stdout: "sleep_interval": { 2024-05-27T07:36:53.150 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "sleep_interval", 2024-05-27T07:36:53.150 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "secs", 2024-05-27T07:36:53.150 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-05-27T07:36:53.150 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 1, 2024-05-27T07:36:53.150 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "5", 2024-05-27T07:36:53.150 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-05-27T07:36:53.150 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-05-27T07:36:53.150 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-05-27T07:36:53.150 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "how long the module is going to sleep", 2024-05-27T07:36:53.151 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-05-27T07:36:53.151 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-05-27T07:36:53.151 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-05-27T07:36:53.151 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-05-27T07:36:53.151 INFO:teuthology.orchestra.run.smithi052.stdout: "sqlite3_killpoint": { 2024-05-27T07:36:53.151 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "sqlite3_killpoint", 2024-05-27T07:36:53.151 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "int", 2024-05-27T07:36:53.151 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "dev", 2024-05-27T07:36:53.151 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 1, 2024-05-27T07:36:53.151 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "0", 2024-05-27T07:36:53.151 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-05-27T07:36:53.151 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-05-27T07:36:53.151 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-05-27T07:36:53.151 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-05-27T07:36:53.152 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-05-27T07:36:53.152 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-05-27T07:36:53.152 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-05-27T07:36:53.152 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-05-27T07:36:53.152 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-05-27T07:36:53.152 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-05-27T07:36:53.152 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-05-27T07:36:53.152 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "prometheus", 2024-05-27T07:36:53.152 INFO:teuthology.orchestra.run.smithi052.stdout: "can_run": true, 2024-05-27T07:36:53.152 INFO:teuthology.orchestra.run.smithi052.stdout: "error_string": "", 2024-05-27T07:36:53.152 INFO:teuthology.orchestra.run.smithi052.stdout: "module_options": { 2024-05-27T07:36:53.152 INFO:teuthology.orchestra.run.smithi052.stdout: "cache": { 2024-05-27T07:36:53.152 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "cache", 2024-05-27T07:36:53.152 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "bool", 2024-05-27T07:36:53.153 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-05-27T07:36:53.153 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 0, 2024-05-27T07:36:53.153 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "True", 2024-05-27T07:36:53.153 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-05-27T07:36:53.153 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-05-27T07:36:53.153 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-05-27T07:36:53.153 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-05-27T07:36:53.153 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-05-27T07:36:53.153 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-05-27T07:36:53.153 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-05-27T07:36:53.153 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-05-27T07:36:53.153 INFO:teuthology.orchestra.run.smithi052.stdout: "exclude_perf_counters": { 2024-05-27T07:36:53.153 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "exclude_perf_counters", 2024-05-27T07:36:53.154 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "bool", 2024-05-27T07:36:53.154 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-05-27T07:36:53.154 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 1, 2024-05-27T07:36:53.154 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "True", 2024-05-27T07:36:53.154 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-05-27T07:36:53.154 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-05-27T07:36:53.154 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-05-27T07:36:53.154 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "Do not include perf-counters in the metrics output", 2024-05-27T07:36:53.154 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.", 2024-05-27T07:36:53.154 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-05-27T07:36:53.154 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-05-27T07:36:53.154 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-05-27T07:36:53.155 INFO:teuthology.orchestra.run.smithi052.stdout: "log_level": { 2024-05-27T07:36:53.155 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "log_level", 2024-05-27T07:36:53.155 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "str", 2024-05-27T07:36:53.155 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-05-27T07:36:53.155 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 1, 2024-05-27T07:36:53.155 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "", 2024-05-27T07:36:53.155 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-05-27T07:36:53.155 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-05-27T07:36:53.155 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [ 2024-05-27T07:36:53.155 INFO:teuthology.orchestra.run.smithi052.stdout: "", 2024-05-27T07:36:53.155 INFO:teuthology.orchestra.run.smithi052.stdout: "critical", 2024-05-27T07:36:53.155 INFO:teuthology.orchestra.run.smithi052.stdout: "debug", 2024-05-27T07:36:53.155 INFO:teuthology.orchestra.run.smithi052.stdout: "error", 2024-05-27T07:36:53.156 INFO:teuthology.orchestra.run.smithi052.stdout: "info", 2024-05-27T07:36:53.156 INFO:teuthology.orchestra.run.smithi052.stdout: "warning" 2024-05-27T07:36:53.156 INFO:teuthology.orchestra.run.smithi052.stdout: ], 2024-05-27T07:36:53.156 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-05-27T07:36:53.156 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-05-27T07:36:53.156 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-05-27T07:36:53.156 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-05-27T07:36:53.156 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-05-27T07:36:53.156 INFO:teuthology.orchestra.run.smithi052.stdout: "log_to_cluster": { 2024-05-27T07:36:53.156 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "log_to_cluster", 2024-05-27T07:36:53.156 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "bool", 2024-05-27T07:36:53.156 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-05-27T07:36:53.156 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 1, 2024-05-27T07:36:53.157 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "False", 2024-05-27T07:36:53.157 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-05-27T07:36:53.157 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-05-27T07:36:53.157 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-05-27T07:36:53.157 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-05-27T07:36:53.157 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-05-27T07:36:53.157 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-05-27T07:36:53.157 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-05-27T07:36:53.157 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-05-27T07:36:53.157 INFO:teuthology.orchestra.run.smithi052.stdout: "log_to_cluster_level": { 2024-05-27T07:36:53.157 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "log_to_cluster_level", 2024-05-27T07:36:53.157 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "str", 2024-05-27T07:36:53.157 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-05-27T07:36:53.158 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 1, 2024-05-27T07:36:53.158 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "info", 2024-05-27T07:36:53.158 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-05-27T07:36:53.158 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-05-27T07:36:53.158 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [ 2024-05-27T07:36:53.158 INFO:teuthology.orchestra.run.smithi052.stdout: "", 2024-05-27T07:36:53.158 INFO:teuthology.orchestra.run.smithi052.stdout: "critical", 2024-05-27T07:36:53.158 INFO:teuthology.orchestra.run.smithi052.stdout: "debug", 2024-05-27T07:36:53.158 INFO:teuthology.orchestra.run.smithi052.stdout: "error", 2024-05-27T07:36:53.158 INFO:teuthology.orchestra.run.smithi052.stdout: "info", 2024-05-27T07:36:53.158 INFO:teuthology.orchestra.run.smithi052.stdout: "warning" 2024-05-27T07:36:53.158 INFO:teuthology.orchestra.run.smithi052.stdout: ], 2024-05-27T07:36:53.158 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-05-27T07:36:53.159 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-05-27T07:36:53.159 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-05-27T07:36:53.159 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-05-27T07:36:53.159 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-05-27T07:36:53.159 INFO:teuthology.orchestra.run.smithi052.stdout: "log_to_file": { 2024-05-27T07:36:53.159 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "log_to_file", 2024-05-27T07:36:53.159 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "bool", 2024-05-27T07:36:53.159 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-05-27T07:36:53.159 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 1, 2024-05-27T07:36:53.159 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "False", 2024-05-27T07:36:53.159 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-05-27T07:36:53.159 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-05-27T07:36:53.159 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-05-27T07:36:53.160 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-05-27T07:36:53.160 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-05-27T07:36:53.160 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-05-27T07:36:53.160 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-05-27T07:36:53.160 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-05-27T07:36:53.160 INFO:teuthology.orchestra.run.smithi052.stdout: "rbd_stats_pools": { 2024-05-27T07:36:53.160 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "rbd_stats_pools", 2024-05-27T07:36:53.160 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "str", 2024-05-27T07:36:53.160 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-05-27T07:36:53.160 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 0, 2024-05-27T07:36:53.160 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "", 2024-05-27T07:36:53.160 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-05-27T07:36:53.160 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-05-27T07:36:53.161 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-05-27T07:36:53.161 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-05-27T07:36:53.161 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-05-27T07:36:53.161 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-05-27T07:36:53.161 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-05-27T07:36:53.161 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-05-27T07:36:53.161 INFO:teuthology.orchestra.run.smithi052.stdout: "rbd_stats_pools_refresh_interval": { 2024-05-27T07:36:53.161 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "rbd_stats_pools_refresh_interval", 2024-05-27T07:36:53.161 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "int", 2024-05-27T07:36:53.161 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-05-27T07:36:53.161 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 0, 2024-05-27T07:36:53.161 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "300", 2024-05-27T07:36:53.161 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-05-27T07:36:53.162 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-05-27T07:36:53.162 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-05-27T07:36:53.162 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-05-27T07:36:53.162 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-05-27T07:36:53.162 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-05-27T07:36:53.162 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-05-27T07:36:53.162 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-05-27T07:36:53.162 INFO:teuthology.orchestra.run.smithi052.stdout: "scrape_interval": { 2024-05-27T07:36:53.162 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "scrape_interval", 2024-05-27T07:36:53.162 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "float", 2024-05-27T07:36:53.162 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-05-27T07:36:53.162 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 0, 2024-05-27T07:36:53.162 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "15.0", 2024-05-27T07:36:53.163 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-05-27T07:36:53.163 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-05-27T07:36:53.163 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-05-27T07:36:53.163 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-05-27T07:36:53.163 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-05-27T07:36:53.163 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-05-27T07:36:53.163 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-05-27T07:36:53.163 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-05-27T07:36:53.163 INFO:teuthology.orchestra.run.smithi052.stdout: "server_addr": { 2024-05-27T07:36:53.163 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "server_addr", 2024-05-27T07:36:53.163 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "str", 2024-05-27T07:36:53.163 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-05-27T07:36:53.163 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 0, 2024-05-27T07:36:53.163 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "::", 2024-05-27T07:36:53.164 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-05-27T07:36:53.164 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-05-27T07:36:53.164 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-05-27T07:36:53.164 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "the IPv4 or IPv6 address on which the module listens for HTTP requests", 2024-05-27T07:36:53.164 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-05-27T07:36:53.164 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-05-27T07:36:53.164 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-05-27T07:36:53.164 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-05-27T07:36:53.164 INFO:teuthology.orchestra.run.smithi052.stdout: "server_port": { 2024-05-27T07:36:53.164 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "server_port", 2024-05-27T07:36:53.164 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "int", 2024-05-27T07:36:53.164 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-05-27T07:36:53.164 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 1, 2024-05-27T07:36:53.165 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "9283", 2024-05-27T07:36:53.165 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-05-27T07:36:53.165 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-05-27T07:36:53.165 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-05-27T07:36:53.165 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "the port on which the module listens for HTTP requests", 2024-05-27T07:36:53.165 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-05-27T07:36:53.165 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-05-27T07:36:53.165 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-05-27T07:36:53.165 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-05-27T07:36:53.165 INFO:teuthology.orchestra.run.smithi052.stdout: "sqlite3_killpoint": { 2024-05-27T07:36:53.165 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "sqlite3_killpoint", 2024-05-27T07:36:53.165 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "int", 2024-05-27T07:36:53.165 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "dev", 2024-05-27T07:36:53.166 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 1, 2024-05-27T07:36:53.166 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "0", 2024-05-27T07:36:53.166 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-05-27T07:36:53.166 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-05-27T07:36:53.166 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-05-27T07:36:53.166 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-05-27T07:36:53.166 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-05-27T07:36:53.166 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-05-27T07:36:53.166 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-05-27T07:36:53.166 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-05-27T07:36:53.166 INFO:teuthology.orchestra.run.smithi052.stdout: "stale_cache_strategy": { 2024-05-27T07:36:53.166 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "stale_cache_strategy", 2024-05-27T07:36:53.166 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "str", 2024-05-27T07:36:53.166 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-05-27T07:36:53.167 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 0, 2024-05-27T07:36:53.167 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "log", 2024-05-27T07:36:53.167 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-05-27T07:36:53.167 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-05-27T07:36:53.167 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-05-27T07:36:53.167 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-05-27T07:36:53.167 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-05-27T07:36:53.167 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-05-27T07:36:53.167 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-05-27T07:36:53.167 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-05-27T07:36:53.167 INFO:teuthology.orchestra.run.smithi052.stdout: "standby_behaviour": { 2024-05-27T07:36:53.167 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "standby_behaviour", 2024-05-27T07:36:53.167 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "str", 2024-05-27T07:36:53.168 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-05-27T07:36:53.168 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 1, 2024-05-27T07:36:53.168 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "default", 2024-05-27T07:36:53.168 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-05-27T07:36:53.168 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-05-27T07:36:53.168 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [ 2024-05-27T07:36:53.168 INFO:teuthology.orchestra.run.smithi052.stdout: "default", 2024-05-27T07:36:53.168 INFO:teuthology.orchestra.run.smithi052.stdout: "error" 2024-05-27T07:36:53.168 INFO:teuthology.orchestra.run.smithi052.stdout: ], 2024-05-27T07:36:53.168 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-05-27T07:36:53.168 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-05-27T07:36:53.168 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-05-27T07:36:53.168 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-05-27T07:36:53.168 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-05-27T07:36:53.169 INFO:teuthology.orchestra.run.smithi052.stdout: "standby_error_status_code": { 2024-05-27T07:36:53.169 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "standby_error_status_code", 2024-05-27T07:36:53.169 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "int", 2024-05-27T07:36:53.169 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-05-27T07:36:53.169 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 1, 2024-05-27T07:36:53.169 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "500", 2024-05-27T07:36:53.169 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "400", 2024-05-27T07:36:53.169 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "599", 2024-05-27T07:36:53.169 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-05-27T07:36:53.169 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-05-27T07:36:53.169 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-05-27T07:36:53.169 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-05-27T07:36:53.169 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-05-27T07:36:53.170 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-05-27T07:36:53.170 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-05-27T07:36:53.170 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-05-27T07:36:53.170 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-05-27T07:36:53.170 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "rbd_support", 2024-05-27T07:36:53.170 INFO:teuthology.orchestra.run.smithi052.stdout: "can_run": true, 2024-05-27T07:36:53.170 INFO:teuthology.orchestra.run.smithi052.stdout: "error_string": "", 2024-05-27T07:36:53.170 INFO:teuthology.orchestra.run.smithi052.stdout: "module_options": { 2024-05-27T07:36:53.170 INFO:teuthology.orchestra.run.smithi052.stdout: "log_level": { 2024-05-27T07:36:53.170 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "log_level", 2024-05-27T07:36:53.170 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "str", 2024-05-27T07:36:53.170 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-05-27T07:36:53.170 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 1, 2024-05-27T07:36:53.171 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "", 2024-05-27T07:36:53.171 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-05-27T07:36:53.171 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-05-27T07:36:53.171 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [ 2024-05-27T07:36:53.171 INFO:teuthology.orchestra.run.smithi052.stdout: "", 2024-05-27T07:36:53.171 INFO:teuthology.orchestra.run.smithi052.stdout: "critical", 2024-05-27T07:36:53.171 INFO:teuthology.orchestra.run.smithi052.stdout: "debug", 2024-05-27T07:36:53.171 INFO:teuthology.orchestra.run.smithi052.stdout: "error", 2024-05-27T07:36:53.171 INFO:teuthology.orchestra.run.smithi052.stdout: "info", 2024-05-27T07:36:53.171 INFO:teuthology.orchestra.run.smithi052.stdout: "warning" 2024-05-27T07:36:53.171 INFO:teuthology.orchestra.run.smithi052.stdout: ], 2024-05-27T07:36:53.171 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-05-27T07:36:53.171 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-05-27T07:36:53.172 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-05-27T07:36:53.172 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-05-27T07:36:53.172 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-05-27T07:36:53.172 INFO:teuthology.orchestra.run.smithi052.stdout: "log_to_cluster": { 2024-05-27T07:36:53.172 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "log_to_cluster", 2024-05-27T07:36:53.172 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "bool", 2024-05-27T07:36:53.172 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-05-27T07:36:53.172 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 1, 2024-05-27T07:36:53.172 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "False", 2024-05-27T07:36:53.172 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-05-27T07:36:53.172 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-05-27T07:36:53.172 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-05-27T07:36:53.172 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-05-27T07:36:53.172 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-05-27T07:36:53.173 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-05-27T07:36:53.173 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-05-27T07:36:53.173 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-05-27T07:36:53.173 INFO:teuthology.orchestra.run.smithi052.stdout: "log_to_cluster_level": { 2024-05-27T07:36:53.173 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "log_to_cluster_level", 2024-05-27T07:36:53.173 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "str", 2024-05-27T07:36:53.173 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-05-27T07:36:53.173 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 1, 2024-05-27T07:36:53.173 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "info", 2024-05-27T07:36:53.173 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-05-27T07:36:53.173 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-05-27T07:36:53.173 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [ 2024-05-27T07:36:53.173 INFO:teuthology.orchestra.run.smithi052.stdout: "", 2024-05-27T07:36:53.174 INFO:teuthology.orchestra.run.smithi052.stdout: "critical", 2024-05-27T07:36:53.174 INFO:teuthology.orchestra.run.smithi052.stdout: "debug", 2024-05-27T07:36:53.174 INFO:teuthology.orchestra.run.smithi052.stdout: "error", 2024-05-27T07:36:53.174 INFO:teuthology.orchestra.run.smithi052.stdout: "info", 2024-05-27T07:36:53.174 INFO:teuthology.orchestra.run.smithi052.stdout: "warning" 2024-05-27T07:36:53.174 INFO:teuthology.orchestra.run.smithi052.stdout: ], 2024-05-27T07:36:53.174 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-05-27T07:36:53.174 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-05-27T07:36:53.174 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-05-27T07:36:53.174 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-05-27T07:36:53.174 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-05-27T07:36:53.174 INFO:teuthology.orchestra.run.smithi052.stdout: "log_to_file": { 2024-05-27T07:36:53.174 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "log_to_file", 2024-05-27T07:36:53.175 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "bool", 2024-05-27T07:36:53.175 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-05-27T07:36:53.175 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 1, 2024-05-27T07:36:53.175 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "False", 2024-05-27T07:36:53.175 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-05-27T07:36:53.175 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-05-27T07:36:53.175 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-05-27T07:36:53.175 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-05-27T07:36:53.175 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-05-27T07:36:53.175 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-05-27T07:36:53.175 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-05-27T07:36:53.175 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-05-27T07:36:53.175 INFO:teuthology.orchestra.run.smithi052.stdout: "max_concurrent_snap_create": { 2024-05-27T07:36:53.175 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "max_concurrent_snap_create", 2024-05-27T07:36:53.176 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "int", 2024-05-27T07:36:53.176 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-05-27T07:36:53.176 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 0, 2024-05-27T07:36:53.176 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "10", 2024-05-27T07:36:53.176 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-05-27T07:36:53.176 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-05-27T07:36:53.176 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-05-27T07:36:53.176 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-05-27T07:36:53.176 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-05-27T07:36:53.176 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-05-27T07:36:53.176 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-05-27T07:36:53.176 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-05-27T07:36:53.176 INFO:teuthology.orchestra.run.smithi052.stdout: "mirror_snapshot_schedule": { 2024-05-27T07:36:53.177 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "mirror_snapshot_schedule", 2024-05-27T07:36:53.177 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "str", 2024-05-27T07:36:53.177 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-05-27T07:36:53.177 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 0, 2024-05-27T07:36:53.177 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "", 2024-05-27T07:36:53.177 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-05-27T07:36:53.177 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-05-27T07:36:53.177 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-05-27T07:36:53.177 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-05-27T07:36:53.177 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-05-27T07:36:53.177 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-05-27T07:36:53.177 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-05-27T07:36:53.177 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-05-27T07:36:53.178 INFO:teuthology.orchestra.run.smithi052.stdout: "sqlite3_killpoint": { 2024-05-27T07:36:53.178 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "sqlite3_killpoint", 2024-05-27T07:36:53.178 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "int", 2024-05-27T07:36:53.178 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "dev", 2024-05-27T07:36:53.178 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 1, 2024-05-27T07:36:53.178 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "0", 2024-05-27T07:36:53.178 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-05-27T07:36:53.178 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-05-27T07:36:53.178 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-05-27T07:36:53.178 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-05-27T07:36:53.178 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-05-27T07:36:53.178 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-05-27T07:36:53.178 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-05-27T07:36:53.179 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-05-27T07:36:53.179 INFO:teuthology.orchestra.run.smithi052.stdout: "trash_purge_schedule": { 2024-05-27T07:36:53.179 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "trash_purge_schedule", 2024-05-27T07:36:53.179 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "str", 2024-05-27T07:36:53.179 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-05-27T07:36:53.179 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 0, 2024-05-27T07:36:53.179 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "", 2024-05-27T07:36:53.179 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-05-27T07:36:53.179 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-05-27T07:36:53.179 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-05-27T07:36:53.179 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-05-27T07:36:53.179 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-05-27T07:36:53.179 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-05-27T07:36:53.180 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-05-27T07:36:53.180 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-05-27T07:36:53.180 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-05-27T07:36:53.180 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-05-27T07:36:53.180 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-05-27T07:36:53.180 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "restful", 2024-05-27T07:36:53.180 INFO:teuthology.orchestra.run.smithi052.stdout: "can_run": true, 2024-05-27T07:36:53.180 INFO:teuthology.orchestra.run.smithi052.stdout: "error_string": "", 2024-05-27T07:36:53.180 INFO:teuthology.orchestra.run.smithi052.stdout: "module_options": { 2024-05-27T07:36:53.180 INFO:teuthology.orchestra.run.smithi052.stdout: "enable_auth": { 2024-05-27T07:36:53.180 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "enable_auth", 2024-05-27T07:36:53.180 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "bool", 2024-05-27T07:36:53.180 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-05-27T07:36:53.180 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 0, 2024-05-27T07:36:53.181 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "True", 2024-05-27T07:36:53.181 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-05-27T07:36:53.181 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-05-27T07:36:53.181 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-05-27T07:36:53.181 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-05-27T07:36:53.181 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-05-27T07:36:53.181 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-05-27T07:36:53.181 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-05-27T07:36:53.181 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-05-27T07:36:53.181 INFO:teuthology.orchestra.run.smithi052.stdout: "key_file": { 2024-05-27T07:36:53.181 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "key_file", 2024-05-27T07:36:53.181 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "str", 2024-05-27T07:36:53.181 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-05-27T07:36:53.182 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 0, 2024-05-27T07:36:53.182 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "", 2024-05-27T07:36:53.182 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-05-27T07:36:53.182 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-05-27T07:36:53.182 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-05-27T07:36:53.182 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-05-27T07:36:53.182 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-05-27T07:36:53.182 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-05-27T07:36:53.182 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-05-27T07:36:53.182 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-05-27T07:36:53.182 INFO:teuthology.orchestra.run.smithi052.stdout: "log_level": { 2024-05-27T07:36:53.182 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "log_level", 2024-05-27T07:36:53.182 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "str", 2024-05-27T07:36:53.182 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-05-27T07:36:53.183 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 1, 2024-05-27T07:36:53.183 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "", 2024-05-27T07:36:53.183 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-05-27T07:36:53.183 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-05-27T07:36:53.183 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [ 2024-05-27T07:36:53.183 INFO:teuthology.orchestra.run.smithi052.stdout: "", 2024-05-27T07:36:53.183 INFO:teuthology.orchestra.run.smithi052.stdout: "critical", 2024-05-27T07:36:53.183 INFO:teuthology.orchestra.run.smithi052.stdout: "debug", 2024-05-27T07:36:53.183 INFO:teuthology.orchestra.run.smithi052.stdout: "error", 2024-05-27T07:36:53.183 INFO:teuthology.orchestra.run.smithi052.stdout: "info", 2024-05-27T07:36:53.183 INFO:teuthology.orchestra.run.smithi052.stdout: "warning" 2024-05-27T07:36:53.183 INFO:teuthology.orchestra.run.smithi052.stdout: ], 2024-05-27T07:36:53.183 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-05-27T07:36:53.184 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-05-27T07:36:53.184 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-05-27T07:36:53.184 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-05-27T07:36:53.184 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-05-27T07:36:53.184 INFO:teuthology.orchestra.run.smithi052.stdout: "log_to_cluster": { 2024-05-27T07:36:53.184 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "log_to_cluster", 2024-05-27T07:36:53.185 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "bool", 2024-05-27T07:36:53.185 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-05-27T07:36:53.186 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 1, 2024-05-27T07:36:53.186 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "False", 2024-05-27T07:36:53.186 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-05-27T07:36:53.186 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-05-27T07:36:53.186 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-05-27T07:36:53.186 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-05-27T07:36:53.186 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-05-27T07:36:53.186 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-05-27T07:36:53.186 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-05-27T07:36:53.186 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-05-27T07:36:53.186 INFO:teuthology.orchestra.run.smithi052.stdout: "log_to_cluster_level": { 2024-05-27T07:36:53.186 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "log_to_cluster_level", 2024-05-27T07:36:53.186 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "str", 2024-05-27T07:36:53.186 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-05-27T07:36:53.187 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 1, 2024-05-27T07:36:53.187 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "info", 2024-05-27T07:36:53.187 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-05-27T07:36:53.187 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-05-27T07:36:53.187 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [ 2024-05-27T07:36:53.187 INFO:teuthology.orchestra.run.smithi052.stdout: "", 2024-05-27T07:36:53.187 INFO:teuthology.orchestra.run.smithi052.stdout: "critical", 2024-05-27T07:36:53.187 INFO:teuthology.orchestra.run.smithi052.stdout: "debug", 2024-05-27T07:36:53.187 INFO:teuthology.orchestra.run.smithi052.stdout: "error", 2024-05-27T07:36:53.187 INFO:teuthology.orchestra.run.smithi052.stdout: "info", 2024-05-27T07:36:53.187 INFO:teuthology.orchestra.run.smithi052.stdout: "warning" 2024-05-27T07:36:53.187 INFO:teuthology.orchestra.run.smithi052.stdout: ], 2024-05-27T07:36:53.187 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-05-27T07:36:53.188 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-05-27T07:36:53.188 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-05-27T07:36:53.188 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-05-27T07:36:53.188 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-05-27T07:36:53.188 INFO:teuthology.orchestra.run.smithi052.stdout: "log_to_file": { 2024-05-27T07:36:53.188 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "log_to_file", 2024-05-27T07:36:53.188 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "bool", 2024-05-27T07:36:53.188 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-05-27T07:36:53.188 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 1, 2024-05-27T07:36:53.188 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "False", 2024-05-27T07:36:53.188 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-05-27T07:36:53.188 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-05-27T07:36:53.188 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-05-27T07:36:53.188 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-05-27T07:36:53.189 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-05-27T07:36:53.189 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-05-27T07:36:53.189 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-05-27T07:36:53.189 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-05-27T07:36:53.189 INFO:teuthology.orchestra.run.smithi052.stdout: "server_addr": { 2024-05-27T07:36:53.189 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "server_addr", 2024-05-27T07:36:53.189 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "str", 2024-05-27T07:36:53.189 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-05-27T07:36:53.189 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 0, 2024-05-27T07:36:53.189 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "", 2024-05-27T07:36:53.189 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-05-27T07:36:53.189 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-05-27T07:36:53.189 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-05-27T07:36:53.190 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-05-27T07:36:53.190 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-05-27T07:36:53.190 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-05-27T07:36:53.190 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-05-27T07:36:53.190 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-05-27T07:36:53.190 INFO:teuthology.orchestra.run.smithi052.stdout: "server_port": { 2024-05-27T07:36:53.190 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "server_port", 2024-05-27T07:36:53.190 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "str", 2024-05-27T07:36:53.190 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-05-27T07:36:53.190 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 0, 2024-05-27T07:36:53.190 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "", 2024-05-27T07:36:53.190 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-05-27T07:36:53.190 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-05-27T07:36:53.191 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-05-27T07:36:53.191 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-05-27T07:36:53.191 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-05-27T07:36:53.191 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-05-27T07:36:53.191 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-05-27T07:36:53.191 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-05-27T07:36:53.191 INFO:teuthology.orchestra.run.smithi052.stdout: "sqlite3_killpoint": { 2024-05-27T07:36:53.191 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "sqlite3_killpoint", 2024-05-27T07:36:53.191 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "int", 2024-05-27T07:36:53.191 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "dev", 2024-05-27T07:36:53.191 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 1, 2024-05-27T07:36:53.191 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "0", 2024-05-27T07:36:53.192 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-05-27T07:36:53.192 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-05-27T07:36:53.192 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-05-27T07:36:53.192 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-05-27T07:36:53.192 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-05-27T07:36:53.192 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-05-27T07:36:53.192 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-05-27T07:36:53.192 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-05-27T07:36:53.192 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-05-27T07:36:53.192 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-05-27T07:36:53.192 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-05-27T07:36:53.192 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "rgw", 2024-05-27T07:36:53.192 INFO:teuthology.orchestra.run.smithi052.stdout: "can_run": true, 2024-05-27T07:36:53.193 INFO:teuthology.orchestra.run.smithi052.stdout: "error_string": "", 2024-05-27T07:36:53.193 INFO:teuthology.orchestra.run.smithi052.stdout: "module_options": { 2024-05-27T07:36:53.193 INFO:teuthology.orchestra.run.smithi052.stdout: "log_level": { 2024-05-27T07:36:53.193 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "log_level", 2024-05-27T07:36:53.193 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "str", 2024-05-27T07:36:53.193 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-05-27T07:36:53.193 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 1, 2024-05-27T07:36:53.193 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "", 2024-05-27T07:36:53.193 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-05-27T07:36:53.193 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-05-27T07:36:53.193 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [ 2024-05-27T07:36:53.193 INFO:teuthology.orchestra.run.smithi052.stdout: "", 2024-05-27T07:36:53.193 INFO:teuthology.orchestra.run.smithi052.stdout: "critical", 2024-05-27T07:36:53.193 INFO:teuthology.orchestra.run.smithi052.stdout: "debug", 2024-05-27T07:36:53.194 INFO:teuthology.orchestra.run.smithi052.stdout: "error", 2024-05-27T07:36:53.194 INFO:teuthology.orchestra.run.smithi052.stdout: "info", 2024-05-27T07:36:53.194 INFO:teuthology.orchestra.run.smithi052.stdout: "warning" 2024-05-27T07:36:53.194 INFO:teuthology.orchestra.run.smithi052.stdout: ], 2024-05-27T07:36:53.194 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-05-27T07:36:53.194 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-05-27T07:36:53.194 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-05-27T07:36:53.194 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-05-27T07:36:53.194 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-05-27T07:36:53.194 INFO:teuthology.orchestra.run.smithi052.stdout: "log_to_cluster": { 2024-05-27T07:36:53.194 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "log_to_cluster", 2024-05-27T07:36:53.194 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "bool", 2024-05-27T07:36:53.195 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-05-27T07:36:53.195 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 1, 2024-05-27T07:36:53.195 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "False", 2024-05-27T07:36:53.195 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-05-27T07:36:53.195 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-05-27T07:36:53.195 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-05-27T07:36:53.195 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-05-27T07:36:53.195 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-05-27T07:36:53.195 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-05-27T07:36:53.195 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-05-27T07:36:53.195 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-05-27T07:36:53.195 INFO:teuthology.orchestra.run.smithi052.stdout: "log_to_cluster_level": { 2024-05-27T07:36:53.195 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "log_to_cluster_level", 2024-05-27T07:36:53.195 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "str", 2024-05-27T07:36:53.196 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-05-27T07:36:53.196 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 1, 2024-05-27T07:36:53.196 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "info", 2024-05-27T07:36:53.196 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-05-27T07:36:53.196 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-05-27T07:36:53.196 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [ 2024-05-27T07:36:53.196 INFO:teuthology.orchestra.run.smithi052.stdout: "", 2024-05-27T07:36:53.196 INFO:teuthology.orchestra.run.smithi052.stdout: "critical", 2024-05-27T07:36:53.196 INFO:teuthology.orchestra.run.smithi052.stdout: "debug", 2024-05-27T07:36:53.196 INFO:teuthology.orchestra.run.smithi052.stdout: "error", 2024-05-27T07:36:53.196 INFO:teuthology.orchestra.run.smithi052.stdout: "info", 2024-05-27T07:36:53.196 INFO:teuthology.orchestra.run.smithi052.stdout: "warning" 2024-05-27T07:36:53.196 INFO:teuthology.orchestra.run.smithi052.stdout: ], 2024-05-27T07:36:53.197 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-05-27T07:36:53.197 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-05-27T07:36:53.197 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-05-27T07:36:53.197 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-05-27T07:36:53.197 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-05-27T07:36:53.197 INFO:teuthology.orchestra.run.smithi052.stdout: "log_to_file": { 2024-05-27T07:36:53.197 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "log_to_file", 2024-05-27T07:36:53.197 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "bool", 2024-05-27T07:36:53.197 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-05-27T07:36:53.197 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 1, 2024-05-27T07:36:53.197 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "False", 2024-05-27T07:36:53.197 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-05-27T07:36:53.197 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-05-27T07:36:53.198 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-05-27T07:36:53.198 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-05-27T07:36:53.198 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-05-27T07:36:53.198 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-05-27T07:36:53.198 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-05-27T07:36:53.198 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-05-27T07:36:53.198 INFO:teuthology.orchestra.run.smithi052.stdout: "sqlite3_killpoint": { 2024-05-27T07:36:53.198 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "sqlite3_killpoint", 2024-05-27T07:36:53.198 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "int", 2024-05-27T07:36:53.198 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "dev", 2024-05-27T07:36:53.198 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 1, 2024-05-27T07:36:53.198 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "0", 2024-05-27T07:36:53.198 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-05-27T07:36:53.199 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-05-27T07:36:53.199 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-05-27T07:36:53.199 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-05-27T07:36:53.199 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-05-27T07:36:53.199 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-05-27T07:36:53.199 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-05-27T07:36:53.199 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-05-27T07:36:53.199 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-05-27T07:36:53.199 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-05-27T07:36:53.199 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-05-27T07:36:53.199 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "rook", 2024-05-27T07:36:53.199 INFO:teuthology.orchestra.run.smithi052.stdout: "can_run": true, 2024-05-27T07:36:53.199 INFO:teuthology.orchestra.run.smithi052.stdout: "error_string": "", 2024-05-27T07:36:53.199 INFO:teuthology.orchestra.run.smithi052.stdout: "module_options": { 2024-05-27T07:36:53.200 INFO:teuthology.orchestra.run.smithi052.stdout: "log_level": { 2024-05-27T07:36:53.200 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "log_level", 2024-05-27T07:36:53.200 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "str", 2024-05-27T07:36:53.200 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-05-27T07:36:53.200 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 1, 2024-05-27T07:36:53.200 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "", 2024-05-27T07:36:53.200 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-05-27T07:36:53.200 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-05-27T07:36:53.200 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [ 2024-05-27T07:36:53.200 INFO:teuthology.orchestra.run.smithi052.stdout: "", 2024-05-27T07:36:53.200 INFO:teuthology.orchestra.run.smithi052.stdout: "critical", 2024-05-27T07:36:53.200 INFO:teuthology.orchestra.run.smithi052.stdout: "debug", 2024-05-27T07:36:53.200 INFO:teuthology.orchestra.run.smithi052.stdout: "error", 2024-05-27T07:36:53.201 INFO:teuthology.orchestra.run.smithi052.stdout: "info", 2024-05-27T07:36:53.201 INFO:teuthology.orchestra.run.smithi052.stdout: "warning" 2024-05-27T07:36:53.201 INFO:teuthology.orchestra.run.smithi052.stdout: ], 2024-05-27T07:36:53.201 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-05-27T07:36:53.201 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-05-27T07:36:53.201 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-05-27T07:36:53.201 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-05-27T07:36:53.201 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-05-27T07:36:53.201 INFO:teuthology.orchestra.run.smithi052.stdout: "log_to_cluster": { 2024-05-27T07:36:53.201 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "log_to_cluster", 2024-05-27T07:36:53.201 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "bool", 2024-05-27T07:36:53.201 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-05-27T07:36:53.201 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 1, 2024-05-27T07:36:53.201 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "False", 2024-05-27T07:36:53.202 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-05-27T07:36:53.202 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-05-27T07:36:53.202 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-05-27T07:36:53.202 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-05-27T07:36:53.202 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-05-27T07:36:53.202 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-05-27T07:36:53.202 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-05-27T07:36:53.202 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-05-27T07:36:53.202 INFO:teuthology.orchestra.run.smithi052.stdout: "log_to_cluster_level": { 2024-05-27T07:36:53.202 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "log_to_cluster_level", 2024-05-27T07:36:53.202 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "str", 2024-05-27T07:36:53.202 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-05-27T07:36:53.202 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 1, 2024-05-27T07:36:53.203 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "info", 2024-05-27T07:36:53.203 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-05-27T07:36:53.203 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-05-27T07:36:53.203 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [ 2024-05-27T07:36:53.203 INFO:teuthology.orchestra.run.smithi052.stdout: "", 2024-05-27T07:36:53.203 INFO:teuthology.orchestra.run.smithi052.stdout: "critical", 2024-05-27T07:36:53.203 INFO:teuthology.orchestra.run.smithi052.stdout: "debug", 2024-05-27T07:36:53.203 INFO:teuthology.orchestra.run.smithi052.stdout: "error", 2024-05-27T07:36:53.203 INFO:teuthology.orchestra.run.smithi052.stdout: "info", 2024-05-27T07:36:53.203 INFO:teuthology.orchestra.run.smithi052.stdout: "warning" 2024-05-27T07:36:53.203 INFO:teuthology.orchestra.run.smithi052.stdout: ], 2024-05-27T07:36:53.203 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-05-27T07:36:53.203 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-05-27T07:36:53.204 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-05-27T07:36:53.204 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-05-27T07:36:53.204 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-05-27T07:36:53.204 INFO:teuthology.orchestra.run.smithi052.stdout: "log_to_file": { 2024-05-27T07:36:53.204 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "log_to_file", 2024-05-27T07:36:53.204 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "bool", 2024-05-27T07:36:53.204 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-05-27T07:36:53.204 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 1, 2024-05-27T07:36:53.204 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "False", 2024-05-27T07:36:53.204 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-05-27T07:36:53.204 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-05-27T07:36:53.204 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-05-27T07:36:53.204 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-05-27T07:36:53.204 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-05-27T07:36:53.205 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-05-27T07:36:53.205 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-05-27T07:36:53.205 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-05-27T07:36:53.205 INFO:teuthology.orchestra.run.smithi052.stdout: "sqlite3_killpoint": { 2024-05-27T07:36:53.205 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "sqlite3_killpoint", 2024-05-27T07:36:53.205 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "int", 2024-05-27T07:36:53.205 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "dev", 2024-05-27T07:36:53.205 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 1, 2024-05-27T07:36:53.205 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "0", 2024-05-27T07:36:53.205 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-05-27T07:36:53.205 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-05-27T07:36:53.205 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-05-27T07:36:53.205 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-05-27T07:36:53.206 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-05-27T07:36:53.206 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-05-27T07:36:53.206 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-05-27T07:36:53.206 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-05-27T07:36:53.206 INFO:teuthology.orchestra.run.smithi052.stdout: "storage_class": { 2024-05-27T07:36:53.206 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "storage_class", 2024-05-27T07:36:53.206 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "str", 2024-05-27T07:36:53.206 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-05-27T07:36:53.206 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 0, 2024-05-27T07:36:53.206 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "local", 2024-05-27T07:36:53.206 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-05-27T07:36:53.206 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-05-27T07:36:53.206 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-05-27T07:36:53.206 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "storage class name for LSO-discovered PVs", 2024-05-27T07:36:53.207 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-05-27T07:36:53.207 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-05-27T07:36:53.207 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-05-27T07:36:53.207 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-05-27T07:36:53.207 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-05-27T07:36:53.207 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-05-27T07:36:53.207 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-05-27T07:36:53.207 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "selftest", 2024-05-27T07:36:53.207 INFO:teuthology.orchestra.run.smithi052.stdout: "can_run": true, 2024-05-27T07:36:53.207 INFO:teuthology.orchestra.run.smithi052.stdout: "error_string": "", 2024-05-27T07:36:53.207 INFO:teuthology.orchestra.run.smithi052.stdout: "module_options": { 2024-05-27T07:36:53.207 INFO:teuthology.orchestra.run.smithi052.stdout: "log_level": { 2024-05-27T07:36:53.207 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "log_level", 2024-05-27T07:36:53.208 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "str", 2024-05-27T07:36:53.208 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-05-27T07:36:53.208 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 1, 2024-05-27T07:36:53.208 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "", 2024-05-27T07:36:53.208 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-05-27T07:36:53.208 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-05-27T07:36:53.208 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [ 2024-05-27T07:36:53.208 INFO:teuthology.orchestra.run.smithi052.stdout: "", 2024-05-27T07:36:53.208 INFO:teuthology.orchestra.run.smithi052.stdout: "critical", 2024-05-27T07:36:53.208 INFO:teuthology.orchestra.run.smithi052.stdout: "debug", 2024-05-27T07:36:53.208 INFO:teuthology.orchestra.run.smithi052.stdout: "error", 2024-05-27T07:36:53.208 INFO:teuthology.orchestra.run.smithi052.stdout: "info", 2024-05-27T07:36:53.208 INFO:teuthology.orchestra.run.smithi052.stdout: "warning" 2024-05-27T07:36:53.208 INFO:teuthology.orchestra.run.smithi052.stdout: ], 2024-05-27T07:36:53.209 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-05-27T07:36:53.209 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-05-27T07:36:53.209 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-05-27T07:36:53.209 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-05-27T07:36:53.209 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-05-27T07:36:53.209 INFO:teuthology.orchestra.run.smithi052.stdout: "log_to_cluster": { 2024-05-27T07:36:53.209 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "log_to_cluster", 2024-05-27T07:36:53.209 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "bool", 2024-05-27T07:36:53.209 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-05-27T07:36:53.209 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 1, 2024-05-27T07:36:53.209 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "False", 2024-05-27T07:36:53.209 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-05-27T07:36:53.209 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-05-27T07:36:53.210 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-05-27T07:36:53.210 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-05-27T07:36:53.210 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-05-27T07:36:53.210 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-05-27T07:36:53.210 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-05-27T07:36:53.210 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-05-27T07:36:53.210 INFO:teuthology.orchestra.run.smithi052.stdout: "log_to_cluster_level": { 2024-05-27T07:36:53.210 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "log_to_cluster_level", 2024-05-27T07:36:53.210 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "str", 2024-05-27T07:36:53.210 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-05-27T07:36:53.210 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 1, 2024-05-27T07:36:53.210 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "info", 2024-05-27T07:36:53.210 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-05-27T07:36:53.211 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-05-27T07:36:53.211 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [ 2024-05-27T07:36:53.211 INFO:teuthology.orchestra.run.smithi052.stdout: "", 2024-05-27T07:36:53.211 INFO:teuthology.orchestra.run.smithi052.stdout: "critical", 2024-05-27T07:36:53.211 INFO:teuthology.orchestra.run.smithi052.stdout: "debug", 2024-05-27T07:36:53.211 INFO:teuthology.orchestra.run.smithi052.stdout: "error", 2024-05-27T07:36:53.211 INFO:teuthology.orchestra.run.smithi052.stdout: "info", 2024-05-27T07:36:53.211 INFO:teuthology.orchestra.run.smithi052.stdout: "warning" 2024-05-27T07:36:53.211 INFO:teuthology.orchestra.run.smithi052.stdout: ], 2024-05-27T07:36:53.211 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-05-27T07:36:53.211 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-05-27T07:36:53.211 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-05-27T07:36:53.211 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-05-27T07:36:53.212 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-05-27T07:36:53.212 INFO:teuthology.orchestra.run.smithi052.stdout: "log_to_file": { 2024-05-27T07:36:53.212 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "log_to_file", 2024-05-27T07:36:53.212 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "bool", 2024-05-27T07:36:53.212 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-05-27T07:36:53.212 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 1, 2024-05-27T07:36:53.212 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "False", 2024-05-27T07:36:53.212 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-05-27T07:36:53.212 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-05-27T07:36:53.212 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-05-27T07:36:53.212 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-05-27T07:36:53.212 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-05-27T07:36:53.212 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-05-27T07:36:53.213 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-05-27T07:36:53.213 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-05-27T07:36:53.213 INFO:teuthology.orchestra.run.smithi052.stdout: "roption1": { 2024-05-27T07:36:53.213 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "roption1", 2024-05-27T07:36:53.213 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "str", 2024-05-27T07:36:53.213 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-05-27T07:36:53.213 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 0, 2024-05-27T07:36:53.213 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "", 2024-05-27T07:36:53.213 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-05-27T07:36:53.213 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-05-27T07:36:53.213 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-05-27T07:36:53.213 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-05-27T07:36:53.213 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-05-27T07:36:53.214 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-05-27T07:36:53.214 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-05-27T07:36:53.214 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-05-27T07:36:53.214 INFO:teuthology.orchestra.run.smithi052.stdout: "roption2": { 2024-05-27T07:36:53.214 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "roption2", 2024-05-27T07:36:53.214 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "str", 2024-05-27T07:36:53.214 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-05-27T07:36:53.214 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 0, 2024-05-27T07:36:53.214 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "xyz", 2024-05-27T07:36:53.214 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-05-27T07:36:53.214 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-05-27T07:36:53.214 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-05-27T07:36:53.214 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-05-27T07:36:53.215 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-05-27T07:36:53.215 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-05-27T07:36:53.215 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-05-27T07:36:53.215 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-05-27T07:36:53.215 INFO:teuthology.orchestra.run.smithi052.stdout: "rwoption1": { 2024-05-27T07:36:53.215 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "rwoption1", 2024-05-27T07:36:53.215 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "str", 2024-05-27T07:36:53.215 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-05-27T07:36:53.215 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 0, 2024-05-27T07:36:53.215 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "", 2024-05-27T07:36:53.215 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-05-27T07:36:53.215 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-05-27T07:36:53.215 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-05-27T07:36:53.215 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-05-27T07:36:53.216 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-05-27T07:36:53.216 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-05-27T07:36:53.216 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-05-27T07:36:53.216 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-05-27T07:36:53.216 INFO:teuthology.orchestra.run.smithi052.stdout: "rwoption2": { 2024-05-27T07:36:53.216 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "rwoption2", 2024-05-27T07:36:53.216 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "int", 2024-05-27T07:36:53.216 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-05-27T07:36:53.216 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 0, 2024-05-27T07:36:53.216 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "", 2024-05-27T07:36:53.216 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-05-27T07:36:53.216 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-05-27T07:36:53.216 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-05-27T07:36:53.217 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-05-27T07:36:53.217 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-05-27T07:36:53.217 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-05-27T07:36:53.217 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-05-27T07:36:53.217 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-05-27T07:36:53.217 INFO:teuthology.orchestra.run.smithi052.stdout: "rwoption3": { 2024-05-27T07:36:53.217 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "rwoption3", 2024-05-27T07:36:53.217 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "float", 2024-05-27T07:36:53.217 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-05-27T07:36:53.217 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 0, 2024-05-27T07:36:53.217 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "", 2024-05-27T07:36:53.217 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-05-27T07:36:53.217 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-05-27T07:36:53.218 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-05-27T07:36:53.218 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-05-27T07:36:53.218 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-05-27T07:36:53.218 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-05-27T07:36:53.218 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-05-27T07:36:53.218 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-05-27T07:36:53.218 INFO:teuthology.orchestra.run.smithi052.stdout: "rwoption4": { 2024-05-27T07:36:53.218 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "rwoption4", 2024-05-27T07:36:53.218 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "str", 2024-05-27T07:36:53.218 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-05-27T07:36:53.218 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 0, 2024-05-27T07:36:53.218 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "", 2024-05-27T07:36:53.218 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-05-27T07:36:53.219 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-05-27T07:36:53.219 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-05-27T07:36:53.219 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-05-27T07:36:53.219 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-05-27T07:36:53.219 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-05-27T07:36:53.219 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-05-27T07:36:53.219 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-05-27T07:36:53.219 INFO:teuthology.orchestra.run.smithi052.stdout: "rwoption5": { 2024-05-27T07:36:53.219 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "rwoption5", 2024-05-27T07:36:53.219 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "bool", 2024-05-27T07:36:53.219 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-05-27T07:36:53.219 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 0, 2024-05-27T07:36:53.219 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "", 2024-05-27T07:36:53.219 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-05-27T07:36:53.220 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-05-27T07:36:53.220 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-05-27T07:36:53.220 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-05-27T07:36:53.220 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-05-27T07:36:53.220 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-05-27T07:36:53.220 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-05-27T07:36:53.220 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-05-27T07:36:53.220 INFO:teuthology.orchestra.run.smithi052.stdout: "rwoption6": { 2024-05-27T07:36:53.220 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "rwoption6", 2024-05-27T07:36:53.220 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "bool", 2024-05-27T07:36:53.220 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-05-27T07:36:53.220 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 0, 2024-05-27T07:36:53.220 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "True", 2024-05-27T07:36:53.220 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-05-27T07:36:53.221 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-05-27T07:36:53.221 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-05-27T07:36:53.221 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-05-27T07:36:53.221 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-05-27T07:36:53.221 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-05-27T07:36:53.221 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-05-27T07:36:53.221 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-05-27T07:36:53.221 INFO:teuthology.orchestra.run.smithi052.stdout: "rwoption7": { 2024-05-27T07:36:53.221 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "rwoption7", 2024-05-27T07:36:53.221 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "int", 2024-05-27T07:36:53.221 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-05-27T07:36:53.221 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 0, 2024-05-27T07:36:53.221 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "", 2024-05-27T07:36:53.222 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "1", 2024-05-27T07:36:53.222 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "42", 2024-05-27T07:36:53.222 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-05-27T07:36:53.222 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-05-27T07:36:53.222 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-05-27T07:36:53.222 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-05-27T07:36:53.222 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-05-27T07:36:53.222 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-05-27T07:36:53.222 INFO:teuthology.orchestra.run.smithi052.stdout: "sqlite3_killpoint": { 2024-05-27T07:36:53.222 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "sqlite3_killpoint", 2024-05-27T07:36:53.222 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "int", 2024-05-27T07:36:53.222 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "dev", 2024-05-27T07:36:53.222 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 1, 2024-05-27T07:36:53.223 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "0", 2024-05-27T07:36:53.223 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-05-27T07:36:53.223 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-05-27T07:36:53.223 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-05-27T07:36:53.223 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-05-27T07:36:53.223 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-05-27T07:36:53.223 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-05-27T07:36:53.223 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-05-27T07:36:53.223 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-05-27T07:36:53.223 INFO:teuthology.orchestra.run.smithi052.stdout: "testkey": { 2024-05-27T07:36:53.223 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "testkey", 2024-05-27T07:36:53.223 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "str", 2024-05-27T07:36:53.223 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-05-27T07:36:53.224 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 0, 2024-05-27T07:36:53.224 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "", 2024-05-27T07:36:53.224 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-05-27T07:36:53.224 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-05-27T07:36:53.224 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-05-27T07:36:53.224 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-05-27T07:36:53.224 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-05-27T07:36:53.224 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-05-27T07:36:53.224 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-05-27T07:36:53.224 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-05-27T07:36:53.224 INFO:teuthology.orchestra.run.smithi052.stdout: "testlkey": { 2024-05-27T07:36:53.224 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "testlkey", 2024-05-27T07:36:53.224 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "str", 2024-05-27T07:36:53.224 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-05-27T07:36:53.225 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 0, 2024-05-27T07:36:53.225 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "", 2024-05-27T07:36:53.225 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-05-27T07:36:53.225 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-05-27T07:36:53.225 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-05-27T07:36:53.225 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-05-27T07:36:53.225 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-05-27T07:36:53.225 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-05-27T07:36:53.225 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-05-27T07:36:53.225 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-05-27T07:36:53.225 INFO:teuthology.orchestra.run.smithi052.stdout: "testnewline": { 2024-05-27T07:36:53.225 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "testnewline", 2024-05-27T07:36:53.225 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "str", 2024-05-27T07:36:53.226 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-05-27T07:36:53.226 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 0, 2024-05-27T07:36:53.226 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "", 2024-05-27T07:36:53.226 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-05-27T07:36:53.226 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-05-27T07:36:53.226 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-05-27T07:36:53.226 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-05-27T07:36:53.226 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-05-27T07:36:53.226 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-05-27T07:36:53.226 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-05-27T07:36:53.226 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-05-27T07:36:53.226 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-05-27T07:36:53.226 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-05-27T07:36:53.226 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-05-27T07:36:53.227 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "smb", 2024-05-27T07:36:53.227 INFO:teuthology.orchestra.run.smithi052.stdout: "can_run": true, 2024-05-27T07:36:53.227 INFO:teuthology.orchestra.run.smithi052.stdout: "error_string": "", 2024-05-27T07:36:53.227 INFO:teuthology.orchestra.run.smithi052.stdout: "module_options": { 2024-05-27T07:36:53.227 INFO:teuthology.orchestra.run.smithi052.stdout: "log_level": { 2024-05-27T07:36:53.227 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "log_level", 2024-05-27T07:36:53.227 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "str", 2024-05-27T07:36:53.227 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-05-27T07:36:53.227 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 1, 2024-05-27T07:36:53.227 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "", 2024-05-27T07:36:53.227 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-05-27T07:36:53.227 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-05-27T07:36:53.227 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [ 2024-05-27T07:36:53.228 INFO:teuthology.orchestra.run.smithi052.stdout: "", 2024-05-27T07:36:53.228 INFO:teuthology.orchestra.run.smithi052.stdout: "critical", 2024-05-27T07:36:53.228 INFO:teuthology.orchestra.run.smithi052.stdout: "debug", 2024-05-27T07:36:53.228 INFO:teuthology.orchestra.run.smithi052.stdout: "error", 2024-05-27T07:36:53.228 INFO:teuthology.orchestra.run.smithi052.stdout: "info", 2024-05-27T07:36:53.228 INFO:teuthology.orchestra.run.smithi052.stdout: "warning" 2024-05-27T07:36:53.228 INFO:teuthology.orchestra.run.smithi052.stdout: ], 2024-05-27T07:36:53.228 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-05-27T07:36:53.228 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-05-27T07:36:53.228 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-05-27T07:36:53.228 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-05-27T07:36:53.228 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-05-27T07:36:53.228 INFO:teuthology.orchestra.run.smithi052.stdout: "log_to_cluster": { 2024-05-27T07:36:53.229 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "log_to_cluster", 2024-05-27T07:36:53.229 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "bool", 2024-05-27T07:36:53.229 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-05-27T07:36:53.229 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 1, 2024-05-27T07:36:53.229 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "False", 2024-05-27T07:36:53.229 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-05-27T07:36:53.229 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-05-27T07:36:53.229 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-05-27T07:36:53.229 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-05-27T07:36:53.229 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-05-27T07:36:53.229 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-05-27T07:36:53.229 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-05-27T07:36:53.229 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-05-27T07:36:53.230 INFO:teuthology.orchestra.run.smithi052.stdout: "log_to_cluster_level": { 2024-05-27T07:36:53.230 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "log_to_cluster_level", 2024-05-27T07:36:53.230 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "str", 2024-05-27T07:36:53.230 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-05-27T07:36:53.230 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 1, 2024-05-27T07:36:53.230 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "info", 2024-05-27T07:36:53.230 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-05-27T07:36:53.230 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-05-27T07:36:53.230 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [ 2024-05-27T07:36:53.230 INFO:teuthology.orchestra.run.smithi052.stdout: "", 2024-05-27T07:36:53.230 INFO:teuthology.orchestra.run.smithi052.stdout: "critical", 2024-05-27T07:36:53.230 INFO:teuthology.orchestra.run.smithi052.stdout: "debug", 2024-05-27T07:36:53.230 INFO:teuthology.orchestra.run.smithi052.stdout: "error", 2024-05-27T07:36:53.231 INFO:teuthology.orchestra.run.smithi052.stdout: "info", 2024-05-27T07:36:53.231 INFO:teuthology.orchestra.run.smithi052.stdout: "warning" 2024-05-27T07:36:53.231 INFO:teuthology.orchestra.run.smithi052.stdout: ], 2024-05-27T07:36:53.231 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-05-27T07:36:53.231 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-05-27T07:36:53.231 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-05-27T07:36:53.231 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-05-27T07:36:53.231 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-05-27T07:36:53.231 INFO:teuthology.orchestra.run.smithi052.stdout: "log_to_file": { 2024-05-27T07:36:53.231 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "log_to_file", 2024-05-27T07:36:53.231 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "bool", 2024-05-27T07:36:53.231 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-05-27T07:36:53.231 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 1, 2024-05-27T07:36:53.231 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "False", 2024-05-27T07:36:53.232 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-05-27T07:36:53.232 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-05-27T07:36:53.232 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-05-27T07:36:53.232 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-05-27T07:36:53.232 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-05-27T07:36:53.232 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-05-27T07:36:53.232 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-05-27T07:36:53.232 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-05-27T07:36:53.232 INFO:teuthology.orchestra.run.smithi052.stdout: "sqlite3_killpoint": { 2024-05-27T07:36:53.232 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "sqlite3_killpoint", 2024-05-27T07:36:53.232 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "int", 2024-05-27T07:36:53.232 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "dev", 2024-05-27T07:36:53.232 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 1, 2024-05-27T07:36:53.233 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "0", 2024-05-27T07:36:53.233 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-05-27T07:36:53.233 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-05-27T07:36:53.233 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-05-27T07:36:53.233 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-05-27T07:36:53.233 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-05-27T07:36:53.233 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-05-27T07:36:53.233 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-05-27T07:36:53.233 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-05-27T07:36:53.233 INFO:teuthology.orchestra.run.smithi052.stdout: "update_orchestration": { 2024-05-27T07:36:53.233 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "update_orchestration", 2024-05-27T07:36:53.233 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "bool", 2024-05-27T07:36:53.233 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-05-27T07:36:53.234 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 0, 2024-05-27T07:36:53.234 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "True", 2024-05-27T07:36:53.234 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-05-27T07:36:53.234 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-05-27T07:36:53.234 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-05-27T07:36:53.234 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "automatically update orchestration when smb resources are changed", 2024-05-27T07:36:53.234 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-05-27T07:36:53.234 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-05-27T07:36:53.234 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-05-27T07:36:53.234 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-05-27T07:36:53.234 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-05-27T07:36:53.234 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-05-27T07:36:53.234 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-05-27T07:36:53.235 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "snap_schedule", 2024-05-27T07:36:53.235 INFO:teuthology.orchestra.run.smithi052.stdout: "can_run": true, 2024-05-27T07:36:53.235 INFO:teuthology.orchestra.run.smithi052.stdout: "error_string": "", 2024-05-27T07:36:53.235 INFO:teuthology.orchestra.run.smithi052.stdout: "module_options": { 2024-05-27T07:36:53.235 INFO:teuthology.orchestra.run.smithi052.stdout: "allow_m_granularity": { 2024-05-27T07:36:53.235 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "allow_m_granularity", 2024-05-27T07:36:53.235 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "bool", 2024-05-27T07:36:53.235 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-05-27T07:36:53.235 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 1, 2024-05-27T07:36:53.235 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "False", 2024-05-27T07:36:53.235 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-05-27T07:36:53.235 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-05-27T07:36:53.235 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-05-27T07:36:53.236 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "allow minute scheduled snapshots", 2024-05-27T07:36:53.236 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-05-27T07:36:53.236 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-05-27T07:36:53.236 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-05-27T07:36:53.236 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-05-27T07:36:53.236 INFO:teuthology.orchestra.run.smithi052.stdout: "dump_on_update": { 2024-05-27T07:36:53.236 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "dump_on_update", 2024-05-27T07:36:53.236 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "bool", 2024-05-27T07:36:53.236 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-05-27T07:36:53.236 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 1, 2024-05-27T07:36:53.236 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "False", 2024-05-27T07:36:53.236 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-05-27T07:36:53.236 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-05-27T07:36:53.236 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-05-27T07:36:53.237 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "dump database to debug log on update", 2024-05-27T07:36:53.237 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-05-27T07:36:53.237 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-05-27T07:36:53.237 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-05-27T07:36:53.237 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-05-27T07:36:53.237 INFO:teuthology.orchestra.run.smithi052.stdout: "log_level": { 2024-05-27T07:36:53.237 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "log_level", 2024-05-27T07:36:53.237 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "str", 2024-05-27T07:36:53.237 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-05-27T07:36:53.237 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 1, 2024-05-27T07:36:53.237 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "", 2024-05-27T07:36:53.237 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-05-27T07:36:53.237 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-05-27T07:36:53.237 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [ 2024-05-27T07:36:53.238 INFO:teuthology.orchestra.run.smithi052.stdout: "", 2024-05-27T07:36:53.238 INFO:teuthology.orchestra.run.smithi052.stdout: "critical", 2024-05-27T07:36:53.238 INFO:teuthology.orchestra.run.smithi052.stdout: "debug", 2024-05-27T07:36:53.238 INFO:teuthology.orchestra.run.smithi052.stdout: "error", 2024-05-27T07:36:53.238 INFO:teuthology.orchestra.run.smithi052.stdout: "info", 2024-05-27T07:36:53.238 INFO:teuthology.orchestra.run.smithi052.stdout: "warning" 2024-05-27T07:36:53.238 INFO:teuthology.orchestra.run.smithi052.stdout: ], 2024-05-27T07:36:53.238 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-05-27T07:36:53.238 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-05-27T07:36:53.238 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-05-27T07:36:53.238 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-05-27T07:36:53.238 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-05-27T07:36:53.238 INFO:teuthology.orchestra.run.smithi052.stdout: "log_to_cluster": { 2024-05-27T07:36:53.239 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "log_to_cluster", 2024-05-27T07:36:53.239 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "bool", 2024-05-27T07:36:53.239 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-05-27T07:36:53.239 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 1, 2024-05-27T07:36:53.239 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "False", 2024-05-27T07:36:53.239 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-05-27T07:36:53.239 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-05-27T07:36:53.239 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-05-27T07:36:53.239 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-05-27T07:36:53.239 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-05-27T07:36:53.239 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-05-27T07:36:53.239 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-05-27T07:36:53.239 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-05-27T07:36:53.240 INFO:teuthology.orchestra.run.smithi052.stdout: "log_to_cluster_level": { 2024-05-27T07:36:53.240 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "log_to_cluster_level", 2024-05-27T07:36:53.240 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "str", 2024-05-27T07:36:53.240 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-05-27T07:36:53.240 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 1, 2024-05-27T07:36:53.240 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "info", 2024-05-27T07:36:53.240 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-05-27T07:36:53.240 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-05-27T07:36:53.240 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [ 2024-05-27T07:36:53.240 INFO:teuthology.orchestra.run.smithi052.stdout: "", 2024-05-27T07:36:53.240 INFO:teuthology.orchestra.run.smithi052.stdout: "critical", 2024-05-27T07:36:53.240 INFO:teuthology.orchestra.run.smithi052.stdout: "debug", 2024-05-27T07:36:53.240 INFO:teuthology.orchestra.run.smithi052.stdout: "error", 2024-05-27T07:36:53.241 INFO:teuthology.orchestra.run.smithi052.stdout: "info", 2024-05-27T07:36:53.241 INFO:teuthology.orchestra.run.smithi052.stdout: "warning" 2024-05-27T07:36:53.241 INFO:teuthology.orchestra.run.smithi052.stdout: ], 2024-05-27T07:36:53.241 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-05-27T07:36:53.241 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-05-27T07:36:53.241 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-05-27T07:36:53.241 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-05-27T07:36:53.241 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-05-27T07:36:53.241 INFO:teuthology.orchestra.run.smithi052.stdout: "log_to_file": { 2024-05-27T07:36:53.241 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "log_to_file", 2024-05-27T07:36:53.241 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "bool", 2024-05-27T07:36:53.241 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-05-27T07:36:53.241 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 1, 2024-05-27T07:36:53.242 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "False", 2024-05-27T07:36:53.242 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-05-27T07:36:53.242 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-05-27T07:36:53.242 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-05-27T07:36:53.242 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-05-27T07:36:53.242 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-05-27T07:36:53.242 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-05-27T07:36:53.242 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-05-27T07:36:53.242 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-05-27T07:36:53.242 INFO:teuthology.orchestra.run.smithi052.stdout: "sqlite3_killpoint": { 2024-05-27T07:36:53.242 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "sqlite3_killpoint", 2024-05-27T07:36:53.242 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "int", 2024-05-27T07:36:53.242 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "dev", 2024-05-27T07:36:53.242 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 1, 2024-05-27T07:36:53.243 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "0", 2024-05-27T07:36:53.243 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-05-27T07:36:53.243 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-05-27T07:36:53.243 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-05-27T07:36:53.243 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-05-27T07:36:53.243 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-05-27T07:36:53.243 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-05-27T07:36:53.243 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-05-27T07:36:53.243 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-05-27T07:36:53.243 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-05-27T07:36:53.243 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-05-27T07:36:53.243 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-05-27T07:36:53.243 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "stats", 2024-05-27T07:36:53.244 INFO:teuthology.orchestra.run.smithi052.stdout: "can_run": true, 2024-05-27T07:36:53.244 INFO:teuthology.orchestra.run.smithi052.stdout: "error_string": "", 2024-05-27T07:36:53.244 INFO:teuthology.orchestra.run.smithi052.stdout: "module_options": { 2024-05-27T07:36:53.244 INFO:teuthology.orchestra.run.smithi052.stdout: "log_level": { 2024-05-27T07:36:53.244 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "log_level", 2024-05-27T07:36:53.244 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "str", 2024-05-27T07:36:53.244 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-05-27T07:36:53.244 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 1, 2024-05-27T07:36:53.244 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "", 2024-05-27T07:36:53.244 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-05-27T07:36:53.244 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-05-27T07:36:53.244 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [ 2024-05-27T07:36:53.244 INFO:teuthology.orchestra.run.smithi052.stdout: "", 2024-05-27T07:36:53.244 INFO:teuthology.orchestra.run.smithi052.stdout: "critical", 2024-05-27T07:36:53.245 INFO:teuthology.orchestra.run.smithi052.stdout: "debug", 2024-05-27T07:36:53.245 INFO:teuthology.orchestra.run.smithi052.stdout: "error", 2024-05-27T07:36:53.245 INFO:teuthology.orchestra.run.smithi052.stdout: "info", 2024-05-27T07:36:53.245 INFO:teuthology.orchestra.run.smithi052.stdout: "warning" 2024-05-27T07:36:53.245 INFO:teuthology.orchestra.run.smithi052.stdout: ], 2024-05-27T07:36:53.245 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-05-27T07:36:53.245 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-05-27T07:36:53.245 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-05-27T07:36:53.245 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-05-27T07:36:53.245 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-05-27T07:36:53.245 INFO:teuthology.orchestra.run.smithi052.stdout: "log_to_cluster": { 2024-05-27T07:36:53.245 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "log_to_cluster", 2024-05-27T07:36:53.245 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "bool", 2024-05-27T07:36:53.246 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-05-27T07:36:53.246 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 1, 2024-05-27T07:36:53.246 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "False", 2024-05-27T07:36:53.246 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-05-27T07:36:53.246 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-05-27T07:36:53.246 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-05-27T07:36:53.246 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-05-27T07:36:53.246 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-05-27T07:36:53.246 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-05-27T07:36:53.246 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-05-27T07:36:53.246 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-05-27T07:36:53.247 INFO:teuthology.orchestra.run.smithi052.stdout: "log_to_cluster_level": { 2024-05-27T07:36:53.247 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "log_to_cluster_level", 2024-05-27T07:36:53.247 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "str", 2024-05-27T07:36:53.247 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-05-27T07:36:53.247 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 1, 2024-05-27T07:36:53.247 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "info", 2024-05-27T07:36:53.247 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-05-27T07:36:53.247 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-05-27T07:36:53.247 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [ 2024-05-27T07:36:53.247 INFO:teuthology.orchestra.run.smithi052.stdout: "", 2024-05-27T07:36:53.247 INFO:teuthology.orchestra.run.smithi052.stdout: "critical", 2024-05-27T07:36:53.247 INFO:teuthology.orchestra.run.smithi052.stdout: "debug", 2024-05-27T07:36:53.247 INFO:teuthology.orchestra.run.smithi052.stdout: "error", 2024-05-27T07:36:53.248 INFO:teuthology.orchestra.run.smithi052.stdout: "info", 2024-05-27T07:36:53.248 INFO:teuthology.orchestra.run.smithi052.stdout: "warning" 2024-05-27T07:36:53.248 INFO:teuthology.orchestra.run.smithi052.stdout: ], 2024-05-27T07:36:53.248 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-05-27T07:36:53.248 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-05-27T07:36:53.248 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-05-27T07:36:53.248 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-05-27T07:36:53.248 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-05-27T07:36:53.248 INFO:teuthology.orchestra.run.smithi052.stdout: "log_to_file": { 2024-05-27T07:36:53.248 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "log_to_file", 2024-05-27T07:36:53.248 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "bool", 2024-05-27T07:36:53.248 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-05-27T07:36:53.248 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 1, 2024-05-27T07:36:53.248 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "False", 2024-05-27T07:36:53.249 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-05-27T07:36:53.249 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-05-27T07:36:53.249 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-05-27T07:36:53.249 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-05-27T07:36:53.249 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-05-27T07:36:53.249 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-05-27T07:36:53.249 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-05-27T07:36:53.249 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-05-27T07:36:53.249 INFO:teuthology.orchestra.run.smithi052.stdout: "sqlite3_killpoint": { 2024-05-27T07:36:53.249 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "sqlite3_killpoint", 2024-05-27T07:36:53.249 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "int", 2024-05-27T07:36:53.249 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "dev", 2024-05-27T07:36:53.249 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 1, 2024-05-27T07:36:53.250 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "0", 2024-05-27T07:36:53.250 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-05-27T07:36:53.250 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-05-27T07:36:53.250 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-05-27T07:36:53.250 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-05-27T07:36:53.250 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-05-27T07:36:53.250 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-05-27T07:36:53.250 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-05-27T07:36:53.250 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-05-27T07:36:53.250 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-05-27T07:36:53.250 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-05-27T07:36:53.250 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-05-27T07:36:53.250 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "status", 2024-05-27T07:36:53.251 INFO:teuthology.orchestra.run.smithi052.stdout: "can_run": true, 2024-05-27T07:36:53.251 INFO:teuthology.orchestra.run.smithi052.stdout: "error_string": "", 2024-05-27T07:36:53.251 INFO:teuthology.orchestra.run.smithi052.stdout: "module_options": { 2024-05-27T07:36:53.251 INFO:teuthology.orchestra.run.smithi052.stdout: "log_level": { 2024-05-27T07:36:53.251 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "log_level", 2024-05-27T07:36:53.251 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "str", 2024-05-27T07:36:53.251 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-05-27T07:36:53.251 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 1, 2024-05-27T07:36:53.251 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "", 2024-05-27T07:36:53.251 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-05-27T07:36:53.251 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-05-27T07:36:53.251 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [ 2024-05-27T07:36:53.251 INFO:teuthology.orchestra.run.smithi052.stdout: "", 2024-05-27T07:36:53.252 INFO:teuthology.orchestra.run.smithi052.stdout: "critical", 2024-05-27T07:36:53.252 INFO:teuthology.orchestra.run.smithi052.stdout: "debug", 2024-05-27T07:36:53.252 INFO:teuthology.orchestra.run.smithi052.stdout: "error", 2024-05-27T07:36:53.252 INFO:teuthology.orchestra.run.smithi052.stdout: "info", 2024-05-27T07:36:53.252 INFO:teuthology.orchestra.run.smithi052.stdout: "warning" 2024-05-27T07:36:53.252 INFO:teuthology.orchestra.run.smithi052.stdout: ], 2024-05-27T07:36:53.252 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-05-27T07:36:53.252 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-05-27T07:36:53.252 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-05-27T07:36:53.252 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-05-27T07:36:53.252 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-05-27T07:36:53.252 INFO:teuthology.orchestra.run.smithi052.stdout: "log_to_cluster": { 2024-05-27T07:36:53.252 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "log_to_cluster", 2024-05-27T07:36:53.253 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "bool", 2024-05-27T07:36:53.253 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-05-27T07:36:53.253 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 1, 2024-05-27T07:36:53.253 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "False", 2024-05-27T07:36:53.253 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-05-27T07:36:53.253 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-05-27T07:36:53.253 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-05-27T07:36:53.253 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-05-27T07:36:53.253 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-05-27T07:36:53.253 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-05-27T07:36:53.253 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-05-27T07:36:53.253 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-05-27T07:36:53.253 INFO:teuthology.orchestra.run.smithi052.stdout: "log_to_cluster_level": { 2024-05-27T07:36:53.253 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "log_to_cluster_level", 2024-05-27T07:36:53.254 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "str", 2024-05-27T07:36:53.254 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-05-27T07:36:53.254 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 1, 2024-05-27T07:36:53.254 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "info", 2024-05-27T07:36:53.254 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-05-27T07:36:53.254 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-05-27T07:36:53.254 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [ 2024-05-27T07:36:53.254 INFO:teuthology.orchestra.run.smithi052.stdout: "", 2024-05-27T07:36:53.254 INFO:teuthology.orchestra.run.smithi052.stdout: "critical", 2024-05-27T07:36:53.254 INFO:teuthology.orchestra.run.smithi052.stdout: "debug", 2024-05-27T07:36:53.254 INFO:teuthology.orchestra.run.smithi052.stdout: "error", 2024-05-27T07:36:53.254 INFO:teuthology.orchestra.run.smithi052.stdout: "info", 2024-05-27T07:36:53.255 INFO:teuthology.orchestra.run.smithi052.stdout: "warning" 2024-05-27T07:36:53.255 INFO:teuthology.orchestra.run.smithi052.stdout: ], 2024-05-27T07:36:53.255 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-05-27T07:36:53.255 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-05-27T07:36:53.255 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-05-27T07:36:53.255 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-05-27T07:36:53.255 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-05-27T07:36:53.255 INFO:teuthology.orchestra.run.smithi052.stdout: "log_to_file": { 2024-05-27T07:36:53.255 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "log_to_file", 2024-05-27T07:36:53.255 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "bool", 2024-05-27T07:36:53.255 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-05-27T07:36:53.255 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 1, 2024-05-27T07:36:53.255 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "False", 2024-05-27T07:36:53.256 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-05-27T07:36:53.256 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-05-27T07:36:53.256 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-05-27T07:36:53.256 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-05-27T07:36:53.256 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-05-27T07:36:53.256 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-05-27T07:36:53.256 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-05-27T07:36:53.256 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-05-27T07:36:53.256 INFO:teuthology.orchestra.run.smithi052.stdout: "sqlite3_killpoint": { 2024-05-27T07:36:53.256 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "sqlite3_killpoint", 2024-05-27T07:36:53.256 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "int", 2024-05-27T07:36:53.256 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "dev", 2024-05-27T07:36:53.256 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 1, 2024-05-27T07:36:53.257 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "0", 2024-05-27T07:36:53.257 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-05-27T07:36:53.257 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-05-27T07:36:53.257 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-05-27T07:36:53.257 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-05-27T07:36:53.257 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-05-27T07:36:53.257 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-05-27T07:36:53.257 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-05-27T07:36:53.257 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-05-27T07:36:53.257 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-05-27T07:36:53.257 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-05-27T07:36:53.257 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-05-27T07:36:53.257 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "telegraf", 2024-05-27T07:36:53.257 INFO:teuthology.orchestra.run.smithi052.stdout: "can_run": true, 2024-05-27T07:36:53.258 INFO:teuthology.orchestra.run.smithi052.stdout: "error_string": "", 2024-05-27T07:36:53.258 INFO:teuthology.orchestra.run.smithi052.stdout: "module_options": { 2024-05-27T07:36:53.258 INFO:teuthology.orchestra.run.smithi052.stdout: "address": { 2024-05-27T07:36:53.258 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "address", 2024-05-27T07:36:53.258 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "str", 2024-05-27T07:36:53.258 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-05-27T07:36:53.258 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 0, 2024-05-27T07:36:53.258 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "unixgram:///tmp/telegraf.sock", 2024-05-27T07:36:53.258 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-05-27T07:36:53.258 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-05-27T07:36:53.258 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-05-27T07:36:53.258 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-05-27T07:36:53.258 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-05-27T07:36:53.259 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-05-27T07:36:53.259 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-05-27T07:36:53.259 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-05-27T07:36:53.259 INFO:teuthology.orchestra.run.smithi052.stdout: "interval": { 2024-05-27T07:36:53.259 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "interval", 2024-05-27T07:36:53.259 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "secs", 2024-05-27T07:36:53.259 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-05-27T07:36:53.259 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 0, 2024-05-27T07:36:53.259 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "15", 2024-05-27T07:36:53.259 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-05-27T07:36:53.259 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-05-27T07:36:53.259 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-05-27T07:36:53.259 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-05-27T07:36:53.260 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-05-27T07:36:53.260 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-05-27T07:36:53.260 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-05-27T07:36:53.260 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-05-27T07:36:53.260 INFO:teuthology.orchestra.run.smithi052.stdout: "log_level": { 2024-05-27T07:36:53.260 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "log_level", 2024-05-27T07:36:53.260 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "str", 2024-05-27T07:36:53.260 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-05-27T07:36:53.260 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 1, 2024-05-27T07:36:53.260 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "", 2024-05-27T07:36:53.260 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-05-27T07:36:53.260 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-05-27T07:36:53.260 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [ 2024-05-27T07:36:53.261 INFO:teuthology.orchestra.run.smithi052.stdout: "", 2024-05-27T07:36:53.261 INFO:teuthology.orchestra.run.smithi052.stdout: "critical", 2024-05-27T07:36:53.261 INFO:teuthology.orchestra.run.smithi052.stdout: "debug", 2024-05-27T07:36:53.261 INFO:teuthology.orchestra.run.smithi052.stdout: "error", 2024-05-27T07:36:53.261 INFO:teuthology.orchestra.run.smithi052.stdout: "info", 2024-05-27T07:36:53.261 INFO:teuthology.orchestra.run.smithi052.stdout: "warning" 2024-05-27T07:36:53.261 INFO:teuthology.orchestra.run.smithi052.stdout: ], 2024-05-27T07:36:53.261 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-05-27T07:36:53.261 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-05-27T07:36:53.261 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-05-27T07:36:53.261 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-05-27T07:36:53.261 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-05-27T07:36:53.261 INFO:teuthology.orchestra.run.smithi052.stdout: "log_to_cluster": { 2024-05-27T07:36:53.261 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "log_to_cluster", 2024-05-27T07:36:53.262 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "bool", 2024-05-27T07:36:53.262 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-05-27T07:36:53.262 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 1, 2024-05-27T07:36:53.262 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "False", 2024-05-27T07:36:53.262 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-05-27T07:36:53.262 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-05-27T07:36:53.262 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-05-27T07:36:53.262 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-05-27T07:36:53.262 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-05-27T07:36:53.262 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-05-27T07:36:53.262 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-05-27T07:36:53.262 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-05-27T07:36:53.262 INFO:teuthology.orchestra.run.smithi052.stdout: "log_to_cluster_level": { 2024-05-27T07:36:53.263 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "log_to_cluster_level", 2024-05-27T07:36:53.263 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "str", 2024-05-27T07:36:53.263 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-05-27T07:36:53.263 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 1, 2024-05-27T07:36:53.263 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "info", 2024-05-27T07:36:53.263 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-05-27T07:36:53.263 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-05-27T07:36:53.263 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [ 2024-05-27T07:36:53.263 INFO:teuthology.orchestra.run.smithi052.stdout: "", 2024-05-27T07:36:53.263 INFO:teuthology.orchestra.run.smithi052.stdout: "critical", 2024-05-27T07:36:53.263 INFO:teuthology.orchestra.run.smithi052.stdout: "debug", 2024-05-27T07:36:53.263 INFO:teuthology.orchestra.run.smithi052.stdout: "error", 2024-05-27T07:36:53.263 INFO:teuthology.orchestra.run.smithi052.stdout: "info", 2024-05-27T07:36:53.264 INFO:teuthology.orchestra.run.smithi052.stdout: "warning" 2024-05-27T07:36:53.264 INFO:teuthology.orchestra.run.smithi052.stdout: ], 2024-05-27T07:36:53.264 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-05-27T07:36:53.264 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-05-27T07:36:53.264 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-05-27T07:36:53.264 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-05-27T07:36:53.264 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-05-27T07:36:53.264 INFO:teuthology.orchestra.run.smithi052.stdout: "log_to_file": { 2024-05-27T07:36:53.264 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "log_to_file", 2024-05-27T07:36:53.264 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "bool", 2024-05-27T07:36:53.264 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-05-27T07:36:53.264 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 1, 2024-05-27T07:36:53.264 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "False", 2024-05-27T07:36:53.265 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-05-27T07:36:53.265 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-05-27T07:36:53.265 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-05-27T07:36:53.265 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-05-27T07:36:53.265 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-05-27T07:36:53.265 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-05-27T07:36:53.265 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-05-27T07:36:53.265 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-05-27T07:36:53.265 INFO:teuthology.orchestra.run.smithi052.stdout: "sqlite3_killpoint": { 2024-05-27T07:36:53.265 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "sqlite3_killpoint", 2024-05-27T07:36:53.265 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "int", 2024-05-27T07:36:53.265 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "dev", 2024-05-27T07:36:53.265 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 1, 2024-05-27T07:36:53.266 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "0", 2024-05-27T07:36:53.266 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-05-27T07:36:53.266 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-05-27T07:36:53.266 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-05-27T07:36:53.266 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-05-27T07:36:53.266 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-05-27T07:36:53.266 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-05-27T07:36:53.266 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-05-27T07:36:53.266 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-05-27T07:36:53.266 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-05-27T07:36:53.266 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-05-27T07:36:53.266 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-05-27T07:36:53.266 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "telemetry", 2024-05-27T07:36:53.267 INFO:teuthology.orchestra.run.smithi052.stdout: "can_run": true, 2024-05-27T07:36:53.267 INFO:teuthology.orchestra.run.smithi052.stdout: "error_string": "", 2024-05-27T07:36:53.267 INFO:teuthology.orchestra.run.smithi052.stdout: "module_options": { 2024-05-27T07:36:53.267 INFO:teuthology.orchestra.run.smithi052.stdout: "channel_basic": { 2024-05-27T07:36:53.267 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "channel_basic", 2024-05-27T07:36:53.267 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "bool", 2024-05-27T07:36:53.267 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-05-27T07:36:53.267 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 0, 2024-05-27T07:36:53.267 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "True", 2024-05-27T07:36:53.267 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-05-27T07:36:53.267 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-05-27T07:36:53.267 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-05-27T07:36:53.267 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "Share basic cluster information (size, version)", 2024-05-27T07:36:53.268 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-05-27T07:36:53.268 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-05-27T07:36:53.268 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-05-27T07:36:53.268 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-05-27T07:36:53.268 INFO:teuthology.orchestra.run.smithi052.stdout: "channel_crash": { 2024-05-27T07:36:53.268 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "channel_crash", 2024-05-27T07:36:53.268 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "bool", 2024-05-27T07:36:53.268 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-05-27T07:36:53.268 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 0, 2024-05-27T07:36:53.268 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "True", 2024-05-27T07:36:53.268 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-05-27T07:36:53.268 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-05-27T07:36:53.268 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-05-27T07:36:53.269 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "Share metadata about Ceph daemon crashes (version, stack straces, etc)", 2024-05-27T07:36:53.269 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-05-27T07:36:53.269 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-05-27T07:36:53.269 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-05-27T07:36:53.269 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-05-27T07:36:53.269 INFO:teuthology.orchestra.run.smithi052.stdout: "channel_device": { 2024-05-27T07:36:53.269 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "channel_device", 2024-05-27T07:36:53.269 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "bool", 2024-05-27T07:36:53.269 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-05-27T07:36:53.269 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 0, 2024-05-27T07:36:53.269 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "True", 2024-05-27T07:36:53.269 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-05-27T07:36:53.269 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-05-27T07:36:53.270 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-05-27T07:36:53.270 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)", 2024-05-27T07:36:53.270 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-05-27T07:36:53.270 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-05-27T07:36:53.270 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-05-27T07:36:53.270 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-05-27T07:36:53.270 INFO:teuthology.orchestra.run.smithi052.stdout: "channel_ident": { 2024-05-27T07:36:53.270 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "channel_ident", 2024-05-27T07:36:53.270 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "bool", 2024-05-27T07:36:53.270 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-05-27T07:36:53.270 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 0, 2024-05-27T07:36:53.270 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "False", 2024-05-27T07:36:53.270 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-05-27T07:36:53.271 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-05-27T07:36:53.271 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-05-27T07:36:53.271 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "Share a user-provided description and/or contact email for the cluster", 2024-05-27T07:36:53.271 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-05-27T07:36:53.271 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-05-27T07:36:53.271 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-05-27T07:36:53.271 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-05-27T07:36:53.271 INFO:teuthology.orchestra.run.smithi052.stdout: "channel_perf": { 2024-05-27T07:36:53.271 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "channel_perf", 2024-05-27T07:36:53.271 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "bool", 2024-05-27T07:36:53.271 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-05-27T07:36:53.271 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 0, 2024-05-27T07:36:53.272 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "False", 2024-05-27T07:36:53.272 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-05-27T07:36:53.272 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-05-27T07:36:53.272 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-05-27T07:36:53.272 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "Share various performance metrics of a cluster", 2024-05-27T07:36:53.272 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-05-27T07:36:53.272 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-05-27T07:36:53.272 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-05-27T07:36:53.272 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-05-27T07:36:53.272 INFO:teuthology.orchestra.run.smithi052.stdout: "contact": { 2024-05-27T07:36:53.272 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "contact", 2024-05-27T07:36:53.272 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "str", 2024-05-27T07:36:53.272 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-05-27T07:36:53.273 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 0, 2024-05-27T07:36:53.273 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "", 2024-05-27T07:36:53.273 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-05-27T07:36:53.273 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-05-27T07:36:53.273 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-05-27T07:36:53.273 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-05-27T07:36:53.273 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-05-27T07:36:53.273 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-05-27T07:36:53.273 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-05-27T07:36:53.273 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-05-27T07:36:53.273 INFO:teuthology.orchestra.run.smithi052.stdout: "description": { 2024-05-27T07:36:53.273 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "description", 2024-05-27T07:36:53.273 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "str", 2024-05-27T07:36:53.274 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-05-27T07:36:53.274 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 0, 2024-05-27T07:36:53.274 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "", 2024-05-27T07:36:53.274 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-05-27T07:36:53.274 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-05-27T07:36:53.274 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-05-27T07:36:53.274 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-05-27T07:36:53.274 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-05-27T07:36:53.274 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-05-27T07:36:53.274 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-05-27T07:36:53.274 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-05-27T07:36:53.274 INFO:teuthology.orchestra.run.smithi052.stdout: "device_url": { 2024-05-27T07:36:53.274 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "device_url", 2024-05-27T07:36:53.275 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "str", 2024-05-27T07:36:53.275 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-05-27T07:36:53.275 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 0, 2024-05-27T07:36:53.275 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "https://telemetry.ceph.com/device", 2024-05-27T07:36:53.275 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-05-27T07:36:53.275 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-05-27T07:36:53.275 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-05-27T07:36:53.275 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-05-27T07:36:53.275 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-05-27T07:36:53.275 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-05-27T07:36:53.275 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-05-27T07:36:53.275 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-05-27T07:36:53.275 INFO:teuthology.orchestra.run.smithi052.stdout: "enabled": { 2024-05-27T07:36:53.276 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "enabled", 2024-05-27T07:36:53.276 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "bool", 2024-05-27T07:36:53.276 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-05-27T07:36:53.276 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 0, 2024-05-27T07:36:53.276 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "False", 2024-05-27T07:36:53.276 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-05-27T07:36:53.276 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-05-27T07:36:53.276 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-05-27T07:36:53.276 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-05-27T07:36:53.276 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-05-27T07:36:53.276 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-05-27T07:36:53.276 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-05-27T07:36:53.276 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-05-27T07:36:53.276 INFO:teuthology.orchestra.run.smithi052.stdout: "interval": { 2024-05-27T07:36:53.277 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "interval", 2024-05-27T07:36:53.277 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "int", 2024-05-27T07:36:53.277 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-05-27T07:36:53.277 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 0, 2024-05-27T07:36:53.277 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "24", 2024-05-27T07:36:53.277 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "8", 2024-05-27T07:36:53.277 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-05-27T07:36:53.277 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-05-27T07:36:53.277 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-05-27T07:36:53.277 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-05-27T07:36:53.277 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-05-27T07:36:53.277 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-05-27T07:36:53.277 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-05-27T07:36:53.278 INFO:teuthology.orchestra.run.smithi052.stdout: "last_opt_revision": { 2024-05-27T07:36:53.278 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "last_opt_revision", 2024-05-27T07:36:53.278 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "int", 2024-05-27T07:36:53.278 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-05-27T07:36:53.278 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 0, 2024-05-27T07:36:53.278 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "1", 2024-05-27T07:36:53.278 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-05-27T07:36:53.278 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-05-27T07:36:53.278 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-05-27T07:36:53.278 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-05-27T07:36:53.278 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-05-27T07:36:53.278 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-05-27T07:36:53.279 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-05-27T07:36:53.279 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-05-27T07:36:53.279 INFO:teuthology.orchestra.run.smithi052.stdout: "leaderboard": { 2024-05-27T07:36:53.279 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "leaderboard", 2024-05-27T07:36:53.279 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "bool", 2024-05-27T07:36:53.279 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-05-27T07:36:53.279 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 0, 2024-05-27T07:36:53.279 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "False", 2024-05-27T07:36:53.279 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-05-27T07:36:53.279 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-05-27T07:36:53.279 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-05-27T07:36:53.279 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-05-27T07:36:53.279 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-05-27T07:36:53.279 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-05-27T07:36:53.280 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-05-27T07:36:53.280 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-05-27T07:36:53.280 INFO:teuthology.orchestra.run.smithi052.stdout: "leaderboard_description": { 2024-05-27T07:36:53.280 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "leaderboard_description", 2024-05-27T07:36:53.280 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "str", 2024-05-27T07:36:53.280 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-05-27T07:36:53.280 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 0, 2024-05-27T07:36:53.280 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "", 2024-05-27T07:36:53.280 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-05-27T07:36:53.280 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-05-27T07:36:53.280 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-05-27T07:36:53.280 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-05-27T07:36:53.280 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-05-27T07:36:53.280 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-05-27T07:36:53.281 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-05-27T07:36:53.281 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-05-27T07:36:53.281 INFO:teuthology.orchestra.run.smithi052.stdout: "log_level": { 2024-05-27T07:36:53.281 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "log_level", 2024-05-27T07:36:53.281 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "str", 2024-05-27T07:36:53.281 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-05-27T07:36:53.281 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 1, 2024-05-27T07:36:53.281 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "", 2024-05-27T07:36:53.281 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-05-27T07:36:53.281 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-05-27T07:36:53.281 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [ 2024-05-27T07:36:53.281 INFO:teuthology.orchestra.run.smithi052.stdout: "", 2024-05-27T07:36:53.281 INFO:teuthology.orchestra.run.smithi052.stdout: "critical", 2024-05-27T07:36:53.282 INFO:teuthology.orchestra.run.smithi052.stdout: "debug", 2024-05-27T07:36:53.282 INFO:teuthology.orchestra.run.smithi052.stdout: "error", 2024-05-27T07:36:53.282 INFO:teuthology.orchestra.run.smithi052.stdout: "info", 2024-05-27T07:36:53.282 INFO:teuthology.orchestra.run.smithi052.stdout: "warning" 2024-05-27T07:36:53.282 INFO:teuthology.orchestra.run.smithi052.stdout: ], 2024-05-27T07:36:53.282 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-05-27T07:36:53.282 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-05-27T07:36:53.282 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-05-27T07:36:53.282 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-05-27T07:36:53.282 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-05-27T07:36:53.282 INFO:teuthology.orchestra.run.smithi052.stdout: "log_to_cluster": { 2024-05-27T07:36:53.282 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "log_to_cluster", 2024-05-27T07:36:53.282 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "bool", 2024-05-27T07:36:53.283 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-05-27T07:36:53.283 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 1, 2024-05-27T07:36:53.283 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "False", 2024-05-27T07:36:53.283 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-05-27T07:36:53.283 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-05-27T07:36:53.283 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-05-27T07:36:53.283 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-05-27T07:36:53.283 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-05-27T07:36:53.283 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-05-27T07:36:53.283 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-05-27T07:36:53.283 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-05-27T07:36:53.283 INFO:teuthology.orchestra.run.smithi052.stdout: "log_to_cluster_level": { 2024-05-27T07:36:53.283 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "log_to_cluster_level", 2024-05-27T07:36:53.284 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "str", 2024-05-27T07:36:53.284 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-05-27T07:36:53.284 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 1, 2024-05-27T07:36:53.284 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "info", 2024-05-27T07:36:53.284 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-05-27T07:36:53.284 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-05-27T07:36:53.284 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [ 2024-05-27T07:36:53.284 INFO:teuthology.orchestra.run.smithi052.stdout: "", 2024-05-27T07:36:53.284 INFO:teuthology.orchestra.run.smithi052.stdout: "critical", 2024-05-27T07:36:53.284 INFO:teuthology.orchestra.run.smithi052.stdout: "debug", 2024-05-27T07:36:53.284 INFO:teuthology.orchestra.run.smithi052.stdout: "error", 2024-05-27T07:36:53.284 INFO:teuthology.orchestra.run.smithi052.stdout: "info", 2024-05-27T07:36:53.284 INFO:teuthology.orchestra.run.smithi052.stdout: "warning" 2024-05-27T07:36:53.285 INFO:teuthology.orchestra.run.smithi052.stdout: ], 2024-05-27T07:36:53.285 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-05-27T07:36:53.285 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-05-27T07:36:53.285 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-05-27T07:36:53.285 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-05-27T07:36:53.285 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-05-27T07:36:53.285 INFO:teuthology.orchestra.run.smithi052.stdout: "log_to_file": { 2024-05-27T07:36:53.285 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "log_to_file", 2024-05-27T07:36:53.285 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "bool", 2024-05-27T07:36:53.285 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-05-27T07:36:53.285 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 1, 2024-05-27T07:36:53.285 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "False", 2024-05-27T07:36:53.285 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-05-27T07:36:53.286 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-05-27T07:36:53.286 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-05-27T07:36:53.286 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-05-27T07:36:53.286 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-05-27T07:36:53.286 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-05-27T07:36:53.286 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-05-27T07:36:53.286 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-05-27T07:36:53.286 INFO:teuthology.orchestra.run.smithi052.stdout: "organization": { 2024-05-27T07:36:53.286 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "organization", 2024-05-27T07:36:53.286 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "str", 2024-05-27T07:36:53.286 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-05-27T07:36:53.286 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 0, 2024-05-27T07:36:53.287 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "", 2024-05-27T07:36:53.287 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-05-27T07:36:53.287 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-05-27T07:36:53.287 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-05-27T07:36:53.287 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-05-27T07:36:53.287 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-05-27T07:36:53.287 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-05-27T07:36:53.287 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-05-27T07:36:53.287 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-05-27T07:36:53.287 INFO:teuthology.orchestra.run.smithi052.stdout: "proxy": { 2024-05-27T07:36:53.287 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "proxy", 2024-05-27T07:36:53.287 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "str", 2024-05-27T07:36:53.287 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-05-27T07:36:53.288 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 0, 2024-05-27T07:36:53.288 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "", 2024-05-27T07:36:53.288 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-05-27T07:36:53.288 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-05-27T07:36:53.288 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-05-27T07:36:53.288 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-05-27T07:36:53.288 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-05-27T07:36:53.288 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-05-27T07:36:53.288 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-05-27T07:36:53.288 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-05-27T07:36:53.288 INFO:teuthology.orchestra.run.smithi052.stdout: "sqlite3_killpoint": { 2024-05-27T07:36:53.288 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "sqlite3_killpoint", 2024-05-27T07:36:53.288 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "int", 2024-05-27T07:36:53.289 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "dev", 2024-05-27T07:36:53.289 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 1, 2024-05-27T07:36:53.289 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "0", 2024-05-27T07:36:53.289 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-05-27T07:36:53.289 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-05-27T07:36:53.289 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-05-27T07:36:53.289 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-05-27T07:36:53.289 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-05-27T07:36:53.289 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-05-27T07:36:53.289 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-05-27T07:36:53.289 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-05-27T07:36:53.289 INFO:teuthology.orchestra.run.smithi052.stdout: "url": { 2024-05-27T07:36:53.289 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "url", 2024-05-27T07:36:53.290 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "str", 2024-05-27T07:36:53.290 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-05-27T07:36:53.290 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 0, 2024-05-27T07:36:53.290 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "https://telemetry.ceph.com/report", 2024-05-27T07:36:53.290 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-05-27T07:36:53.290 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-05-27T07:36:53.290 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-05-27T07:36:53.290 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-05-27T07:36:53.290 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-05-27T07:36:53.290 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-05-27T07:36:53.290 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-05-27T07:36:53.290 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-05-27T07:36:53.290 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-05-27T07:36:53.291 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-05-27T07:36:53.291 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-05-27T07:36:53.291 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "test_orchestrator", 2024-05-27T07:36:53.291 INFO:teuthology.orchestra.run.smithi052.stdout: "can_run": true, 2024-05-27T07:36:53.291 INFO:teuthology.orchestra.run.smithi052.stdout: "error_string": "", 2024-05-27T07:36:53.291 INFO:teuthology.orchestra.run.smithi052.stdout: "module_options": { 2024-05-27T07:36:53.291 INFO:teuthology.orchestra.run.smithi052.stdout: "log_level": { 2024-05-27T07:36:53.291 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "log_level", 2024-05-27T07:36:53.291 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "str", 2024-05-27T07:36:53.291 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-05-27T07:36:53.291 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 1, 2024-05-27T07:36:53.291 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "", 2024-05-27T07:36:53.291 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-05-27T07:36:53.291 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-05-27T07:36:53.292 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [ 2024-05-27T07:36:53.292 INFO:teuthology.orchestra.run.smithi052.stdout: "", 2024-05-27T07:36:53.292 INFO:teuthology.orchestra.run.smithi052.stdout: "critical", 2024-05-27T07:36:53.292 INFO:teuthology.orchestra.run.smithi052.stdout: "debug", 2024-05-27T07:36:53.292 INFO:teuthology.orchestra.run.smithi052.stdout: "error", 2024-05-27T07:36:53.292 INFO:teuthology.orchestra.run.smithi052.stdout: "info", 2024-05-27T07:36:53.292 INFO:teuthology.orchestra.run.smithi052.stdout: "warning" 2024-05-27T07:36:53.292 INFO:teuthology.orchestra.run.smithi052.stdout: ], 2024-05-27T07:36:53.292 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-05-27T07:36:53.292 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-05-27T07:36:53.292 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-05-27T07:36:53.292 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-05-27T07:36:53.292 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-05-27T07:36:53.293 INFO:teuthology.orchestra.run.smithi052.stdout: "log_to_cluster": { 2024-05-27T07:36:53.293 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "log_to_cluster", 2024-05-27T07:36:53.293 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "bool", 2024-05-27T07:36:53.293 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-05-27T07:36:53.293 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 1, 2024-05-27T07:36:53.293 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "False", 2024-05-27T07:36:53.293 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-05-27T07:36:53.293 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-05-27T07:36:53.293 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-05-27T07:36:53.293 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-05-27T07:36:53.293 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-05-27T07:36:53.293 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-05-27T07:36:53.293 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-05-27T07:36:53.293 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-05-27T07:36:53.294 INFO:teuthology.orchestra.run.smithi052.stdout: "log_to_cluster_level": { 2024-05-27T07:36:53.294 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "log_to_cluster_level", 2024-05-27T07:36:53.294 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "str", 2024-05-27T07:36:53.294 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-05-27T07:36:53.294 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 1, 2024-05-27T07:36:53.294 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "info", 2024-05-27T07:36:53.294 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-05-27T07:36:53.294 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-05-27T07:36:53.294 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [ 2024-05-27T07:36:53.294 INFO:teuthology.orchestra.run.smithi052.stdout: "", 2024-05-27T07:36:53.294 INFO:teuthology.orchestra.run.smithi052.stdout: "critical", 2024-05-27T07:36:53.294 INFO:teuthology.orchestra.run.smithi052.stdout: "debug", 2024-05-27T07:36:53.294 INFO:teuthology.orchestra.run.smithi052.stdout: "error", 2024-05-27T07:36:53.295 INFO:teuthology.orchestra.run.smithi052.stdout: "info", 2024-05-27T07:36:53.295 INFO:teuthology.orchestra.run.smithi052.stdout: "warning" 2024-05-27T07:36:53.295 INFO:teuthology.orchestra.run.smithi052.stdout: ], 2024-05-27T07:36:53.295 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-05-27T07:36:53.295 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-05-27T07:36:53.295 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-05-27T07:36:53.295 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-05-27T07:36:53.295 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-05-27T07:36:53.295 INFO:teuthology.orchestra.run.smithi052.stdout: "log_to_file": { 2024-05-27T07:36:53.295 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "log_to_file", 2024-05-27T07:36:53.295 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "bool", 2024-05-27T07:36:53.295 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-05-27T07:36:53.295 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 1, 2024-05-27T07:36:53.296 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "False", 2024-05-27T07:36:53.296 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-05-27T07:36:53.296 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-05-27T07:36:53.296 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-05-27T07:36:53.296 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-05-27T07:36:53.296 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-05-27T07:36:53.296 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-05-27T07:36:53.296 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-05-27T07:36:53.296 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-05-27T07:36:53.296 INFO:teuthology.orchestra.run.smithi052.stdout: "sqlite3_killpoint": { 2024-05-27T07:36:53.296 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "sqlite3_killpoint", 2024-05-27T07:36:53.296 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "int", 2024-05-27T07:36:53.296 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "dev", 2024-05-27T07:36:53.296 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 1, 2024-05-27T07:36:53.297 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "0", 2024-05-27T07:36:53.297 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-05-27T07:36:53.297 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-05-27T07:36:53.297 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-05-27T07:36:53.297 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-05-27T07:36:53.297 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-05-27T07:36:53.297 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-05-27T07:36:53.297 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-05-27T07:36:53.297 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-05-27T07:36:53.297 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-05-27T07:36:53.297 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-05-27T07:36:53.297 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-05-27T07:36:53.297 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "volumes", 2024-05-27T07:36:53.298 INFO:teuthology.orchestra.run.smithi052.stdout: "can_run": true, 2024-05-27T07:36:53.298 INFO:teuthology.orchestra.run.smithi052.stdout: "error_string": "", 2024-05-27T07:36:53.298 INFO:teuthology.orchestra.run.smithi052.stdout: "module_options": { 2024-05-27T07:36:53.298 INFO:teuthology.orchestra.run.smithi052.stdout: "log_level": { 2024-05-27T07:36:53.298 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "log_level", 2024-05-27T07:36:53.298 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "str", 2024-05-27T07:36:53.298 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-05-27T07:36:53.298 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 1, 2024-05-27T07:36:53.298 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "", 2024-05-27T07:36:53.298 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-05-27T07:36:53.298 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-05-27T07:36:53.298 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [ 2024-05-27T07:36:53.298 INFO:teuthology.orchestra.run.smithi052.stdout: "", 2024-05-27T07:36:53.299 INFO:teuthology.orchestra.run.smithi052.stdout: "critical", 2024-05-27T07:36:53.299 INFO:teuthology.orchestra.run.smithi052.stdout: "debug", 2024-05-27T07:36:53.299 INFO:teuthology.orchestra.run.smithi052.stdout: "error", 2024-05-27T07:36:53.299 INFO:teuthology.orchestra.run.smithi052.stdout: "info", 2024-05-27T07:36:53.299 INFO:teuthology.orchestra.run.smithi052.stdout: "warning" 2024-05-27T07:36:53.299 INFO:teuthology.orchestra.run.smithi052.stdout: ], 2024-05-27T07:36:53.299 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-05-27T07:36:53.299 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-05-27T07:36:53.299 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-05-27T07:36:53.299 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-05-27T07:36:53.299 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-05-27T07:36:53.299 INFO:teuthology.orchestra.run.smithi052.stdout: "log_to_cluster": { 2024-05-27T07:36:53.299 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "log_to_cluster", 2024-05-27T07:36:53.300 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "bool", 2024-05-27T07:36:53.300 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-05-27T07:36:53.300 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 1, 2024-05-27T07:36:53.300 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "False", 2024-05-27T07:36:53.300 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-05-27T07:36:53.300 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-05-27T07:36:53.300 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-05-27T07:36:53.300 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-05-27T07:36:53.300 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-05-27T07:36:53.300 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-05-27T07:36:53.300 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-05-27T07:36:53.300 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-05-27T07:36:53.300 INFO:teuthology.orchestra.run.smithi052.stdout: "log_to_cluster_level": { 2024-05-27T07:36:53.301 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "log_to_cluster_level", 2024-05-27T07:36:53.301 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "str", 2024-05-27T07:36:53.301 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-05-27T07:36:53.301 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 1, 2024-05-27T07:36:53.301 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "info", 2024-05-27T07:36:53.301 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-05-27T07:36:53.301 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-05-27T07:36:53.301 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [ 2024-05-27T07:36:53.301 INFO:teuthology.orchestra.run.smithi052.stdout: "", 2024-05-27T07:36:53.301 INFO:teuthology.orchestra.run.smithi052.stdout: "critical", 2024-05-27T07:36:53.301 INFO:teuthology.orchestra.run.smithi052.stdout: "debug", 2024-05-27T07:36:53.301 INFO:teuthology.orchestra.run.smithi052.stdout: "error", 2024-05-27T07:36:53.302 INFO:teuthology.orchestra.run.smithi052.stdout: "info", 2024-05-27T07:36:53.302 INFO:teuthology.orchestra.run.smithi052.stdout: "warning" 2024-05-27T07:36:53.302 INFO:teuthology.orchestra.run.smithi052.stdout: ], 2024-05-27T07:36:53.302 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-05-27T07:36:53.302 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-05-27T07:36:53.302 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-05-27T07:36:53.302 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-05-27T07:36:53.302 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-05-27T07:36:53.302 INFO:teuthology.orchestra.run.smithi052.stdout: "log_to_file": { 2024-05-27T07:36:53.302 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "log_to_file", 2024-05-27T07:36:53.302 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "bool", 2024-05-27T07:36:53.302 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-05-27T07:36:53.302 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 1, 2024-05-27T07:36:53.303 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "False", 2024-05-27T07:36:53.303 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-05-27T07:36:53.303 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-05-27T07:36:53.303 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-05-27T07:36:53.303 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-05-27T07:36:53.303 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-05-27T07:36:53.303 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-05-27T07:36:53.303 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-05-27T07:36:53.303 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-05-27T07:36:53.303 INFO:teuthology.orchestra.run.smithi052.stdout: "max_concurrent_clones": { 2024-05-27T07:36:53.303 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "max_concurrent_clones", 2024-05-27T07:36:53.303 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "int", 2024-05-27T07:36:53.303 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-05-27T07:36:53.304 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 0, 2024-05-27T07:36:53.304 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "4", 2024-05-27T07:36:53.304 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-05-27T07:36:53.304 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-05-27T07:36:53.304 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-05-27T07:36:53.304 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "Number of asynchronous cloner threads", 2024-05-27T07:36:53.304 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-05-27T07:36:53.304 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-05-27T07:36:53.304 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-05-27T07:36:53.304 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-05-27T07:36:53.304 INFO:teuthology.orchestra.run.smithi052.stdout: "periodic_async_work": { 2024-05-27T07:36:53.304 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "periodic_async_work", 2024-05-27T07:36:53.304 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "bool", 2024-05-27T07:36:53.305 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-05-27T07:36:53.305 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 0, 2024-05-27T07:36:53.305 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "False", 2024-05-27T07:36:53.305 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-05-27T07:36:53.305 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-05-27T07:36:53.305 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-05-27T07:36:53.305 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "Periodically check for async work", 2024-05-27T07:36:53.305 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-05-27T07:36:53.305 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-05-27T07:36:53.305 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-05-27T07:36:53.305 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-05-27T07:36:53.305 INFO:teuthology.orchestra.run.smithi052.stdout: "snapshot_clone_delay": { 2024-05-27T07:36:53.305 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "snapshot_clone_delay", 2024-05-27T07:36:53.306 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "int", 2024-05-27T07:36:53.306 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-05-27T07:36:53.306 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 0, 2024-05-27T07:36:53.306 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "0", 2024-05-27T07:36:53.306 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-05-27T07:36:53.306 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-05-27T07:36:53.306 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-05-27T07:36:53.306 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "Delay clone begin operation by snapshot_clone_delay seconds", 2024-05-27T07:36:53.306 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-05-27T07:36:53.306 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-05-27T07:36:53.306 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-05-27T07:36:53.306 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-05-27T07:36:53.306 INFO:teuthology.orchestra.run.smithi052.stdout: "snapshot_clone_no_wait": { 2024-05-27T07:36:53.307 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "snapshot_clone_no_wait", 2024-05-27T07:36:53.307 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "bool", 2024-05-27T07:36:53.307 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-05-27T07:36:53.307 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 0, 2024-05-27T07:36:53.307 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "True", 2024-05-27T07:36:53.307 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-05-27T07:36:53.307 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-05-27T07:36:53.307 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-05-27T07:36:53.307 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "Reject subvolume clone request when cloner threads are busy", 2024-05-27T07:36:53.307 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-05-27T07:36:53.307 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-05-27T07:36:53.307 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-05-27T07:36:53.307 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-05-27T07:36:53.307 INFO:teuthology.orchestra.run.smithi052.stdout: "sqlite3_killpoint": { 2024-05-27T07:36:53.308 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "sqlite3_killpoint", 2024-05-27T07:36:53.308 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "int", 2024-05-27T07:36:53.308 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "dev", 2024-05-27T07:36:53.308 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 1, 2024-05-27T07:36:53.308 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "0", 2024-05-27T07:36:53.308 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-05-27T07:36:53.308 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-05-27T07:36:53.308 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-05-27T07:36:53.308 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-05-27T07:36:53.308 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-05-27T07:36:53.308 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-05-27T07:36:53.308 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-05-27T07:36:53.309 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-05-27T07:36:53.309 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-05-27T07:36:53.309 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-05-27T07:36:53.309 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-05-27T07:36:53.309 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "zabbix", 2024-05-27T07:36:53.309 INFO:teuthology.orchestra.run.smithi052.stdout: "can_run": true, 2024-05-27T07:36:53.309 INFO:teuthology.orchestra.run.smithi052.stdout: "error_string": "", 2024-05-27T07:36:53.309 INFO:teuthology.orchestra.run.smithi052.stdout: "module_options": { 2024-05-27T07:36:53.309 INFO:teuthology.orchestra.run.smithi052.stdout: "discovery_interval": { 2024-05-27T07:36:53.309 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "discovery_interval", 2024-05-27T07:36:53.309 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "uint", 2024-05-27T07:36:53.309 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-05-27T07:36:53.309 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 0, 2024-05-27T07:36:53.309 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "100", 2024-05-27T07:36:53.310 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-05-27T07:36:53.310 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-05-27T07:36:53.310 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-05-27T07:36:53.310 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-05-27T07:36:53.310 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-05-27T07:36:53.310 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-05-27T07:36:53.310 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-05-27T07:36:53.310 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-05-27T07:36:53.310 INFO:teuthology.orchestra.run.smithi052.stdout: "identifier": { 2024-05-27T07:36:53.310 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "identifier", 2024-05-27T07:36:53.310 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "str", 2024-05-27T07:36:53.310 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-05-27T07:36:53.310 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 0, 2024-05-27T07:36:53.311 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "", 2024-05-27T07:36:53.311 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-05-27T07:36:53.311 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-05-27T07:36:53.311 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-05-27T07:36:53.311 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-05-27T07:36:53.311 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-05-27T07:36:53.311 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-05-27T07:36:53.311 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-05-27T07:36:53.311 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-05-27T07:36:53.311 INFO:teuthology.orchestra.run.smithi052.stdout: "interval": { 2024-05-27T07:36:53.311 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "interval", 2024-05-27T07:36:53.311 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "secs", 2024-05-27T07:36:53.311 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-05-27T07:36:53.312 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 0, 2024-05-27T07:36:53.312 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "60", 2024-05-27T07:36:53.312 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-05-27T07:36:53.312 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-05-27T07:36:53.312 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-05-27T07:36:53.312 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-05-27T07:36:53.312 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-05-27T07:36:53.312 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-05-27T07:36:53.312 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-05-27T07:36:53.312 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-05-27T07:36:53.312 INFO:teuthology.orchestra.run.smithi052.stdout: "log_level": { 2024-05-27T07:36:53.312 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "log_level", 2024-05-27T07:36:53.312 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "str", 2024-05-27T07:36:53.312 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-05-27T07:36:53.313 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 1, 2024-05-27T07:36:53.313 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "", 2024-05-27T07:36:53.313 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-05-27T07:36:53.313 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-05-27T07:36:53.313 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [ 2024-05-27T07:36:53.313 INFO:teuthology.orchestra.run.smithi052.stdout: "", 2024-05-27T07:36:53.313 INFO:teuthology.orchestra.run.smithi052.stdout: "critical", 2024-05-27T07:36:53.313 INFO:teuthology.orchestra.run.smithi052.stdout: "debug", 2024-05-27T07:36:53.313 INFO:teuthology.orchestra.run.smithi052.stdout: "error", 2024-05-27T07:36:53.313 INFO:teuthology.orchestra.run.smithi052.stdout: "info", 2024-05-27T07:36:53.313 INFO:teuthology.orchestra.run.smithi052.stdout: "warning" 2024-05-27T07:36:53.313 INFO:teuthology.orchestra.run.smithi052.stdout: ], 2024-05-27T07:36:53.313 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-05-27T07:36:53.314 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-05-27T07:36:53.314 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-05-27T07:36:53.314 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-05-27T07:36:53.314 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-05-27T07:36:53.314 INFO:teuthology.orchestra.run.smithi052.stdout: "log_to_cluster": { 2024-05-27T07:36:53.314 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "log_to_cluster", 2024-05-27T07:36:53.314 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "bool", 2024-05-27T07:36:53.314 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-05-27T07:36:53.314 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 1, 2024-05-27T07:36:53.314 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "False", 2024-05-27T07:36:53.314 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-05-27T07:36:53.314 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-05-27T07:36:53.314 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-05-27T07:36:53.314 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-05-27T07:36:53.315 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-05-27T07:36:53.315 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-05-27T07:36:53.315 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-05-27T07:36:53.315 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-05-27T07:36:53.315 INFO:teuthology.orchestra.run.smithi052.stdout: "log_to_cluster_level": { 2024-05-27T07:36:53.315 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "log_to_cluster_level", 2024-05-27T07:36:53.315 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "str", 2024-05-27T07:36:53.315 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-05-27T07:36:53.315 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 1, 2024-05-27T07:36:53.315 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "info", 2024-05-27T07:36:53.315 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-05-27T07:36:53.315 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-05-27T07:36:53.315 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [ 2024-05-27T07:36:53.316 INFO:teuthology.orchestra.run.smithi052.stdout: "", 2024-05-27T07:36:53.316 INFO:teuthology.orchestra.run.smithi052.stdout: "critical", 2024-05-27T07:36:53.316 INFO:teuthology.orchestra.run.smithi052.stdout: "debug", 2024-05-27T07:36:53.316 INFO:teuthology.orchestra.run.smithi052.stdout: "error", 2024-05-27T07:36:53.316 INFO:teuthology.orchestra.run.smithi052.stdout: "info", 2024-05-27T07:36:53.316 INFO:teuthology.orchestra.run.smithi052.stdout: "warning" 2024-05-27T07:36:53.316 INFO:teuthology.orchestra.run.smithi052.stdout: ], 2024-05-27T07:36:53.316 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-05-27T07:36:53.316 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-05-27T07:36:53.316 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-05-27T07:36:53.316 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-05-27T07:36:53.316 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-05-27T07:36:53.316 INFO:teuthology.orchestra.run.smithi052.stdout: "log_to_file": { 2024-05-27T07:36:53.316 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "log_to_file", 2024-05-27T07:36:53.317 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "bool", 2024-05-27T07:36:53.317 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-05-27T07:36:53.317 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 1, 2024-05-27T07:36:53.317 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "False", 2024-05-27T07:36:53.317 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-05-27T07:36:53.317 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-05-27T07:36:53.317 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-05-27T07:36:53.317 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-05-27T07:36:53.317 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-05-27T07:36:53.317 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-05-27T07:36:53.317 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-05-27T07:36:53.317 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-05-27T07:36:53.317 INFO:teuthology.orchestra.run.smithi052.stdout: "sqlite3_killpoint": { 2024-05-27T07:36:53.318 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "sqlite3_killpoint", 2024-05-27T07:36:53.318 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "int", 2024-05-27T07:36:53.318 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "dev", 2024-05-27T07:36:53.318 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 1, 2024-05-27T07:36:53.318 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "0", 2024-05-27T07:36:53.318 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-05-27T07:36:53.318 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-05-27T07:36:53.318 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-05-27T07:36:53.318 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-05-27T07:36:53.318 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-05-27T07:36:53.318 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-05-27T07:36:53.318 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-05-27T07:36:53.318 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-05-27T07:36:53.319 INFO:teuthology.orchestra.run.smithi052.stdout: "zabbix_host": { 2024-05-27T07:36:53.319 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "zabbix_host", 2024-05-27T07:36:53.319 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "str", 2024-05-27T07:36:53.319 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-05-27T07:36:53.319 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 0, 2024-05-27T07:36:53.319 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "", 2024-05-27T07:36:53.319 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-05-27T07:36:53.319 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-05-27T07:36:53.319 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-05-27T07:36:53.319 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-05-27T07:36:53.319 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-05-27T07:36:53.319 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-05-27T07:36:53.319 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-05-27T07:36:53.320 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-05-27T07:36:53.320 INFO:teuthology.orchestra.run.smithi052.stdout: "zabbix_port": { 2024-05-27T07:36:53.320 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "zabbix_port", 2024-05-27T07:36:53.320 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "int", 2024-05-27T07:36:53.320 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-05-27T07:36:53.320 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 0, 2024-05-27T07:36:53.320 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "10051", 2024-05-27T07:36:53.320 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-05-27T07:36:53.320 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-05-27T07:36:53.320 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-05-27T07:36:53.320 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-05-27T07:36:53.320 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-05-27T07:36:53.320 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-05-27T07:36:53.320 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-05-27T07:36:53.321 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-05-27T07:36:53.321 INFO:teuthology.orchestra.run.smithi052.stdout: "zabbix_sender": { 2024-05-27T07:36:53.321 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "zabbix_sender", 2024-05-27T07:36:53.321 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "str", 2024-05-27T07:36:53.321 INFO:teuthology.orchestra.run.smithi052.stdout: "level": "advanced", 2024-05-27T07:36:53.321 INFO:teuthology.orchestra.run.smithi052.stdout: "flags": 0, 2024-05-27T07:36:53.321 INFO:teuthology.orchestra.run.smithi052.stdout: "default_value": "/usr/bin/zabbix_sender", 2024-05-27T07:36:53.321 INFO:teuthology.orchestra.run.smithi052.stdout: "min": "", 2024-05-27T07:36:53.321 INFO:teuthology.orchestra.run.smithi052.stdout: "max": "", 2024-05-27T07:36:53.321 INFO:teuthology.orchestra.run.smithi052.stdout: "enum_allowed": [], 2024-05-27T07:36:53.321 INFO:teuthology.orchestra.run.smithi052.stdout: "desc": "", 2024-05-27T07:36:53.321 INFO:teuthology.orchestra.run.smithi052.stdout: "long_desc": "", 2024-05-27T07:36:53.321 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2024-05-27T07:36:53.322 INFO:teuthology.orchestra.run.smithi052.stdout: "see_also": [] 2024-05-27T07:36:53.322 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-05-27T07:36:53.322 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-05-27T07:36:53.322 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-05-27T07:36:53.322 INFO:teuthology.orchestra.run.smithi052.stdout: ], 2024-05-27T07:36:53.322 INFO:teuthology.orchestra.run.smithi052.stdout: "services": { 2024-05-27T07:36:53.322 INFO:teuthology.orchestra.run.smithi052.stdout: "dashboard": "https://172.21.15.52:8443/" 2024-05-27T07:36:53.322 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-05-27T07:36:53.322 INFO:teuthology.orchestra.run.smithi052.stdout: "always_on_modules": { 2024-05-27T07:36:53.322 INFO:teuthology.orchestra.run.smithi052.stdout: "octopus": [ 2024-05-27T07:36:53.322 INFO:teuthology.orchestra.run.smithi052.stdout: "balancer", 2024-05-27T07:36:53.322 INFO:teuthology.orchestra.run.smithi052.stdout: "crash", 2024-05-27T07:36:53.322 INFO:teuthology.orchestra.run.smithi052.stdout: "devicehealth", 2024-05-27T07:36:53.322 INFO:teuthology.orchestra.run.smithi052.stdout: "orchestrator", 2024-05-27T07:36:53.323 INFO:teuthology.orchestra.run.smithi052.stdout: "pg_autoscaler", 2024-05-27T07:36:53.323 INFO:teuthology.orchestra.run.smithi052.stdout: "progress", 2024-05-27T07:36:53.323 INFO:teuthology.orchestra.run.smithi052.stdout: "rbd_support", 2024-05-27T07:36:53.323 INFO:teuthology.orchestra.run.smithi052.stdout: "status", 2024-05-27T07:36:53.323 INFO:teuthology.orchestra.run.smithi052.stdout: "telemetry", 2024-05-27T07:36:53.323 INFO:teuthology.orchestra.run.smithi052.stdout: "volumes" 2024-05-27T07:36:53.323 INFO:teuthology.orchestra.run.smithi052.stdout: ], 2024-05-27T07:36:53.323 INFO:teuthology.orchestra.run.smithi052.stdout: "pacific": [ 2024-05-27T07:36:53.323 INFO:teuthology.orchestra.run.smithi052.stdout: "balancer", 2024-05-27T07:36:53.323 INFO:teuthology.orchestra.run.smithi052.stdout: "crash", 2024-05-27T07:36:53.323 INFO:teuthology.orchestra.run.smithi052.stdout: "devicehealth", 2024-05-27T07:36:53.323 INFO:teuthology.orchestra.run.smithi052.stdout: "orchestrator", 2024-05-27T07:36:53.323 INFO:teuthology.orchestra.run.smithi052.stdout: "pg_autoscaler", 2024-05-27T07:36:53.324 INFO:teuthology.orchestra.run.smithi052.stdout: "progress", 2024-05-27T07:36:53.324 INFO:teuthology.orchestra.run.smithi052.stdout: "rbd_support", 2024-05-27T07:36:53.324 INFO:teuthology.orchestra.run.smithi052.stdout: "status", 2024-05-27T07:36:53.324 INFO:teuthology.orchestra.run.smithi052.stdout: "telemetry", 2024-05-27T07:36:53.324 INFO:teuthology.orchestra.run.smithi052.stdout: "volumes" 2024-05-27T07:36:53.324 INFO:teuthology.orchestra.run.smithi052.stdout: ], 2024-05-27T07:36:53.324 INFO:teuthology.orchestra.run.smithi052.stdout: "quincy": [ 2024-05-27T07:36:53.324 INFO:teuthology.orchestra.run.smithi052.stdout: "balancer", 2024-05-27T07:36:53.324 INFO:teuthology.orchestra.run.smithi052.stdout: "crash", 2024-05-27T07:36:53.324 INFO:teuthology.orchestra.run.smithi052.stdout: "devicehealth", 2024-05-27T07:36:53.324 INFO:teuthology.orchestra.run.smithi052.stdout: "orchestrator", 2024-05-27T07:36:53.324 INFO:teuthology.orchestra.run.smithi052.stdout: "pg_autoscaler", 2024-05-27T07:36:53.324 INFO:teuthology.orchestra.run.smithi052.stdout: "progress", 2024-05-27T07:36:53.324 INFO:teuthology.orchestra.run.smithi052.stdout: "rbd_support", 2024-05-27T07:36:53.325 INFO:teuthology.orchestra.run.smithi052.stdout: "status", 2024-05-27T07:36:53.325 INFO:teuthology.orchestra.run.smithi052.stdout: "telemetry", 2024-05-27T07:36:53.325 INFO:teuthology.orchestra.run.smithi052.stdout: "volumes" 2024-05-27T07:36:53.325 INFO:teuthology.orchestra.run.smithi052.stdout: ], 2024-05-27T07:36:53.325 INFO:teuthology.orchestra.run.smithi052.stdout: "reef": [ 2024-05-27T07:36:53.325 INFO:teuthology.orchestra.run.smithi052.stdout: "balancer", 2024-05-27T07:36:53.325 INFO:teuthology.orchestra.run.smithi052.stdout: "crash", 2024-05-27T07:36:53.325 INFO:teuthology.orchestra.run.smithi052.stdout: "devicehealth", 2024-05-27T07:36:53.325 INFO:teuthology.orchestra.run.smithi052.stdout: "orchestrator", 2024-05-27T07:36:53.325 INFO:teuthology.orchestra.run.smithi052.stdout: "pg_autoscaler", 2024-05-27T07:36:53.325 INFO:teuthology.orchestra.run.smithi052.stdout: "progress", 2024-05-27T07:36:53.325 INFO:teuthology.orchestra.run.smithi052.stdout: "rbd_support", 2024-05-27T07:36:53.325 INFO:teuthology.orchestra.run.smithi052.stdout: "status", 2024-05-27T07:36:53.325 INFO:teuthology.orchestra.run.smithi052.stdout: "telemetry", 2024-05-27T07:36:53.326 INFO:teuthology.orchestra.run.smithi052.stdout: "volumes" 2024-05-27T07:36:53.326 INFO:teuthology.orchestra.run.smithi052.stdout: ], 2024-05-27T07:36:53.326 INFO:teuthology.orchestra.run.smithi052.stdout: "squid": [ 2024-05-27T07:36:53.326 INFO:teuthology.orchestra.run.smithi052.stdout: "balancer", 2024-05-27T07:36:53.326 INFO:teuthology.orchestra.run.smithi052.stdout: "crash", 2024-05-27T07:36:53.326 INFO:teuthology.orchestra.run.smithi052.stdout: "devicehealth", 2024-05-27T07:36:53.326 INFO:teuthology.orchestra.run.smithi052.stdout: "orchestrator", 2024-05-27T07:36:53.326 INFO:teuthology.orchestra.run.smithi052.stdout: "pg_autoscaler", 2024-05-27T07:36:53.326 INFO:teuthology.orchestra.run.smithi052.stdout: "progress", 2024-05-27T07:36:53.326 INFO:teuthology.orchestra.run.smithi052.stdout: "rbd_support", 2024-05-27T07:36:53.326 INFO:teuthology.orchestra.run.smithi052.stdout: "status", 2024-05-27T07:36:53.326 INFO:teuthology.orchestra.run.smithi052.stdout: "telemetry", 2024-05-27T07:36:53.326 INFO:teuthology.orchestra.run.smithi052.stdout: "volumes" 2024-05-27T07:36:53.327 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2024-05-27T07:36:53.327 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-05-27T07:36:53.327 INFO:teuthology.orchestra.run.smithi052.stdout: "last_failure_osd_epoch": 91, 2024-05-27T07:36:53.327 INFO:teuthology.orchestra.run.smithi052.stdout: "active_clients": [ 2024-05-27T07:36:53.327 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-05-27T07:36:53.327 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "devicehealth", 2024-05-27T07:36:53.327 INFO:teuthology.orchestra.run.smithi052.stdout: "addrvec": [ 2024-05-27T07:36:53.327 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-05-27T07:36:53.327 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "v2", 2024-05-27T07:36:53.327 INFO:teuthology.orchestra.run.smithi052.stdout: "addr": "172.21.15.52:0", 2024-05-27T07:36:53.327 INFO:teuthology.orchestra.run.smithi052.stdout: "nonce": 2303544924 2024-05-27T07:36:53.327 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-05-27T07:36:53.327 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2024-05-27T07:36:53.328 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-05-27T07:36:53.328 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-05-27T07:36:53.328 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "libcephsqlite", 2024-05-27T07:36:53.328 INFO:teuthology.orchestra.run.smithi052.stdout: "addrvec": [ 2024-05-27T07:36:53.328 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-05-27T07:36:53.328 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "v2", 2024-05-27T07:36:53.328 INFO:teuthology.orchestra.run.smithi052.stdout: "addr": "172.21.15.52:0", 2024-05-27T07:36:53.328 INFO:teuthology.orchestra.run.smithi052.stdout: "nonce": 862791578 2024-05-27T07:36:53.328 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-05-27T07:36:53.328 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2024-05-27T07:36:53.328 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-05-27T07:36:53.328 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-05-27T07:36:53.328 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "rbd_support", 2024-05-27T07:36:53.328 INFO:teuthology.orchestra.run.smithi052.stdout: "addrvec": [ 2024-05-27T07:36:53.329 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-05-27T07:36:53.329 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "v2", 2024-05-27T07:36:53.329 INFO:teuthology.orchestra.run.smithi052.stdout: "addr": "172.21.15.52:0", 2024-05-27T07:36:53.329 INFO:teuthology.orchestra.run.smithi052.stdout: "nonce": 392269284 2024-05-27T07:36:53.329 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-05-27T07:36:53.329 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2024-05-27T07:36:53.329 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-05-27T07:36:53.329 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-05-27T07:36:53.329 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "snap_schedule", 2024-05-27T07:36:53.329 INFO:teuthology.orchestra.run.smithi052.stdout: "addrvec": [ 2024-05-27T07:36:53.329 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-05-27T07:36:53.329 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "v2", 2024-05-27T07:36:53.329 INFO:teuthology.orchestra.run.smithi052.stdout: "addr": "172.21.15.52:0", 2024-05-27T07:36:53.329 INFO:teuthology.orchestra.run.smithi052.stdout: "nonce": 2002286852 2024-05-27T07:36:53.330 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-05-27T07:36:53.330 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2024-05-27T07:36:53.330 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-05-27T07:36:53.330 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-05-27T07:36:53.330 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "snap_schedule", 2024-05-27T07:36:53.330 INFO:teuthology.orchestra.run.smithi052.stdout: "addrvec": [ 2024-05-27T07:36:53.330 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-05-27T07:36:53.330 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "v2", 2024-05-27T07:36:53.330 INFO:teuthology.orchestra.run.smithi052.stdout: "addr": "172.21.15.52:0", 2024-05-27T07:36:53.330 INFO:teuthology.orchestra.run.smithi052.stdout: "nonce": 3248787016 2024-05-27T07:36:53.330 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-05-27T07:36:53.330 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2024-05-27T07:36:53.330 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2024-05-27T07:36:53.331 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-05-27T07:36:53.331 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "volumes", 2024-05-27T07:36:53.331 INFO:teuthology.orchestra.run.smithi052.stdout: "addrvec": [ 2024-05-27T07:36:53.331 INFO:teuthology.orchestra.run.smithi052.stdout: { 2024-05-27T07:36:53.331 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "v2", 2024-05-27T07:36:53.331 INFO:teuthology.orchestra.run.smithi052.stdout: "addr": "172.21.15.52:0", 2024-05-27T07:36:53.331 INFO:teuthology.orchestra.run.smithi052.stdout: "nonce": 3941799655 2024-05-27T07:36:53.331 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-05-27T07:36:53.331 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2024-05-27T07:36:53.331 INFO:teuthology.orchestra.run.smithi052.stdout: } 2024-05-27T07:36:53.331 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2024-05-27T07:36:53.331 INFO:teuthology.orchestra.run.smithi052.stdout:} 2024-05-27T07:36:53.333 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:36:51 smithi105 ceph-mon[35977]: from='client.? 172.21.15.52:0/3851470066' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-05-27T07:36:53.333 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:36:51 smithi105 ceph-mon[35977]: pgmap v366: 129 pgs: 8 active+clean+snaptrim, 19 active+clean+snaptrim_wait, 102 active+clean; 622 MiB data, 7.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-05-27T07:36:53.333 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:36:51 smithi105 ceph-mon[35977]: from='mgr.15333 172.21.15.52:0/416389082' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-05-27T07:36:53.333 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:36:51 smithi105 ceph-mon[35977]: from='mgr.15333 172.21.15.52:0/416389082' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-05-27T07:36:53.333 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:36:51 smithi105 ceph-mon[35977]: from='mgr.15333 172.21.15.52:0/416389082' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-05-27T07:36:53.333 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:36:51 smithi105 ceph-mon[35977]: from='mgr.15333 172.21.15.52:0/416389082' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-05-27T07:36:53.333 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:36:51 smithi105 ceph-mon[35977]: Adjusting osd_memory_target on smithi052 to 177.2M 2024-05-27T07:36:53.333 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:36:51 smithi105 ceph-mon[35977]: Unable to set osd_memory_target on smithi052 to 185879142: error parsing value: Value '185879142' is below minimum 939524096 2024-05-27T07:36:53.333 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:36:51 smithi105 ceph-mon[35977]: from='mgr.15333 172.21.15.52:0/416389082' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T07:36:53.333 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:36:51 smithi105 ceph-mon[35977]: from='mgr.15333 172.21.15.52:0/416389082' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-27T07:36:53.333 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:36:51 smithi105 ceph-mon[35977]: pgmap v367: 129 pgs: 8 active+clean+snaptrim, 19 active+clean+snaptrim_wait, 102 active+clean; 622 MiB data, 7.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-05-27T07:36:53.333 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:36:51 smithi105 ceph-mon[35977]: from='mgr.15333 172.21.15.52:0/416389082' entity='mgr.x' 2024-05-27T07:36:53.334 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:36:52 smithi105 ceph-mon[35977]: Health check failed: 7 failed cephadm daemon(s) (CEPHADM_FAILED_DAEMON) 2024-05-27T07:36:53.334 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:36:52 smithi105 ceph-mon[35977]: from='client.? 172.21.15.52:0/923522457' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json-pretty"}]: dispatch 2024-05-27T07:36:53.334 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:36:51 smithi173 ceph-mon[37000]: from='client.? 172.21.15.52:0/3851470066' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-05-27T07:36:53.334 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:36:51 smithi173 ceph-mon[37000]: pgmap v366: 129 pgs: 8 active+clean+snaptrim, 19 active+clean+snaptrim_wait, 102 active+clean; 622 MiB data, 7.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-05-27T07:36:53.334 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:36:51 smithi173 ceph-mon[37000]: from='mgr.15333 172.21.15.52:0/416389082' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-05-27T07:36:53.334 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:36:51 smithi173 ceph-mon[37000]: from='mgr.15333 172.21.15.52:0/416389082' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-05-27T07:36:53.334 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:36:51 smithi173 ceph-mon[37000]: from='mgr.15333 172.21.15.52:0/416389082' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-05-27T07:36:53.334 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:36:51 smithi173 ceph-mon[37000]: from='mgr.15333 172.21.15.52:0/416389082' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-05-27T07:36:53.334 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:36:51 smithi173 ceph-mon[37000]: Adjusting osd_memory_target on smithi052 to 177.2M 2024-05-27T07:36:53.334 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:36:51 smithi173 ceph-mon[37000]: Unable to set osd_memory_target on smithi052 to 185879142: error parsing value: Value '185879142' is below minimum 939524096 2024-05-27T07:36:53.334 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:36:51 smithi173 ceph-mon[37000]: from='mgr.15333 172.21.15.52:0/416389082' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T07:36:53.334 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:36:51 smithi173 ceph-mon[37000]: from='mgr.15333 172.21.15.52:0/416389082' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-27T07:36:53.335 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:36:51 smithi173 ceph-mon[37000]: pgmap v367: 129 pgs: 8 active+clean+snaptrim, 19 active+clean+snaptrim_wait, 102 active+clean; 622 MiB data, 7.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-05-27T07:36:53.335 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:36:51 smithi173 ceph-mon[37000]: from='mgr.15333 172.21.15.52:0/416389082' entity='mgr.x' 2024-05-27T07:36:53.335 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:36:52 smithi173 ceph-mon[37000]: Health check failed: 7 failed cephadm daemon(s) (CEPHADM_FAILED_DAEMON) 2024-05-27T07:36:53.335 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:36:52 smithi173 ceph-mon[37000]: from='client.? 172.21.15.52:0/923522457' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json-pretty"}]: dispatch 2024-05-27T07:36:53.336 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:51.771+0000 7fee5759e640 1 -- 172.21.15.52:0/923522457 >> [v2:172.21.15.52:6800/1822318384,v1:172.21.15.52:6801/1822318384] conn(0x7fee400b2970 msgr2=0x7fee400b4e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:36:53.336 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:51.771+0000 7fee5759e640 1 --2- 172.21.15.52:0/923522457 >> [v2:172.21.15.52:6800/1822318384,v1:172.21.15.52:6801/1822318384] conn(0x7fee400b2970 0x7fee400b4e80 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7fee58114b70 tx=0x7fee50007310 comp rx=0 tx=0).stop 2024-05-27T07:36:53.336 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:51.771+0000 7fee5759e640 1 -- 172.21.15.52:0/923522457 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fee5811d150 msgr2=0x7fee58113980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:36:53.336 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:51.771+0000 7fee5759e640 1 --2- 172.21.15.52:0/923522457 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fee5811d150 0x7fee58113980 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7fee4c005ab0 tx=0x7fee4c003820 comp rx=0 tx=0).stop 2024-05-27T07:36:53.336 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:51.771+0000 7fee5759e640 1 -- 172.21.15.52:0/923522457 shutdown_connections 2024-05-27T07:36:53.336 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:51.771+0000 7fee5759e640 1 --2- 172.21.15.52:0/923522457 >> [v2:172.21.15.52:6800/1822318384,v1:172.21.15.52:6801/1822318384] conn(0x7fee400b2970 0x7fee400b4e80 unknown :-1 s=CLOSED pgs=205 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:36:53.336 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:51.771+0000 7fee5759e640 1 --2- 172.21.15.52:0/923522457 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fee5811d150 0x7fee58113980 unknown :-1 s=CLOSED pgs=236 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:36:53.336 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:51.771+0000 7fee5759e640 1 --2- 172.21.15.52:0/923522457 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fee5811a7f0 0x7fee58113320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:36:53.336 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:51.771+0000 7fee5759e640 1 --2- 172.21.15.52:0/923522457 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fee5806ecf0 0x7fee5811a200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:36:53.336 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:51.771+0000 7fee5759e640 1 -- 172.21.15.52:0/923522457 >> 172.21.15.52:0/923522457 conn(0x7fee58109ea0 msgr2=0x7fee5811b330 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:36:53.336 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:51.771+0000 7fee5759e640 1 -- 172.21.15.52:0/923522457 shutdown_connections 2024-05-27T07:36:53.337 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:51.771+0000 7fee5759e640 1 -- 172.21.15.52:0/923522457 wait complete. 2024-05-27T07:36:53.343 DEBUG:tasks.check_counter:Getting stats from a 2024-05-27T07:36:53.344 DEBUG:teuthology.orchestra.run.smithi052:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:38f9d900c83c63799fbdbe61acc9a11b0d3554a6 shell --fsid 73df2636-1bf8-11ef-bc9b-c7b262605968 -- ceph daemon mds.a perf dump 2024-05-27T07:36:53.344 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:36:52 smithi052 ceph-mon[31955]: Health check failed: 7 failed cephadm daemon(s) (CEPHADM_FAILED_DAEMON) 2024-05-27T07:36:53.344 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:36:52 smithi052 ceph-mon[31955]: from='client.? 172.21.15.52:0/923522457' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json-pretty"}]: dispatch 2024-05-27T07:36:53.738 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:36:53 smithi052 ceph-mon[31955]: pgmap v368: 129 pgs: 8 active+clean+snaptrim, 19 active+clean+snaptrim_wait, 102 active+clean; 621 MiB data, 7.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-05-27T07:36:53.754 INFO:teuthology.orchestra.run.smithi052.stderr:Inferring config /var/lib/ceph/73df2636-1bf8-11ef-bc9b-c7b262605968/mon.a/config 2024-05-27T07:36:53.858 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:36:53 smithi173 ceph-mon[37000]: pgmap v368: 129 pgs: 8 active+clean+snaptrim, 19 active+clean+snaptrim_wait, 102 active+clean; 621 MiB data, 7.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-05-27T07:36:53.946 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:36:53 smithi105 ceph-mon[35977]: pgmap v368: 129 pgs: 8 active+clean+snaptrim, 19 active+clean+snaptrim_wait, 102 active+clean; 621 MiB data, 7.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-05-27T07:36:56.216 INFO:teuthology.orchestra.run.smithi052.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2024-05-27T07:36:56.486 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:36:56 smithi052 ceph-mon[31955]: pgmap v369: 129 pgs: 6 active+clean+snaptrim, 12 active+clean+snaptrim_wait, 111 active+clean; 621 MiB data, 7.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-05-27T07:36:56.696 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:36:56 smithi105 ceph-mon[35977]: pgmap v369: 129 pgs: 6 active+clean+snaptrim, 12 active+clean+snaptrim_wait, 111 active+clean; 621 MiB data, 7.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-05-27T07:36:56.858 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:36:56 smithi173 ceph-mon[37000]: pgmap v369: 129 pgs: 6 active+clean+snaptrim, 12 active+clean+snaptrim_wait, 111 active+clean; 621 MiB data, 7.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-05-27T07:36:57.035 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-05-27T07:36:57.036 ERROR:teuthology.run_tasks:Manager failed: check-counter Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_e7ae7580c417d6ab27fda98fecadca5d9f85d63e/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/home/teuthworker/src/git.ceph.com_teuthology_e7ae7580c417d6ab27fda98fecadca5d9f85d63e/teuthology/task/__init__.py", line 132, in __exit__ self.end() File "/home/teuthworker/src/git.ceph.com_ceph-c_38f9d900c83c63799fbdbe61acc9a11b0d3554a6/qa/tasks/check_counter.py", line 84, in end proc = manager.admin_socket(daemon_type, daemon_id, ["perf", "dump"]) File "/home/teuthworker/src/git.ceph.com_ceph-c_38f9d900c83c63799fbdbe61acc9a11b0d3554a6/qa/tasks/ceph_manager.py", line 1893, in admin_socket return shell( File "/home/teuthworker/src/git.ceph.com_ceph-c_38f9d900c83c63799fbdbe61acc9a11b0d3554a6/qa/tasks/ceph_manager.py", line 43, in shell return remote.run( File "/home/teuthworker/src/git.ceph.com_teuthology_e7ae7580c417d6ab27fda98fecadca5d9f85d63e/teuthology/orchestra/remote.py", line 523, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_e7ae7580c417d6ab27fda98fecadca5d9f85d63e/teuthology/orchestra/run.py", line 455, in run r.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_e7ae7580c417d6ab27fda98fecadca5d9f85d63e/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_e7ae7580c417d6ab27fda98fecadca5d9f85d63e/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi052 with status 22: 'sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:38f9d900c83c63799fbdbe61acc9a11b0d3554a6 shell --fsid 73df2636-1bf8-11ef-bc9b-c7b262605968 -- ceph daemon mds.a perf dump' 2024-05-27T07:36:57.038 DEBUG:teuthology.run_tasks:Unwinding manager kclient 2024-05-27T07:36:57.051 INFO:tasks.kclient:Unmounting kernel clients... 2024-05-27T07:36:57.051 DEBUG:teuthology.orchestra.run.smithi052:> set -ex 2024-05-27T07:36:57.051 DEBUG:teuthology.orchestra.run.smithi052:> dd if=/proc/self/mounts of=/dev/stdout 2024-05-27T07:36:57.071 DEBUG:teuthology.orchestra.run.smithi052:> set -ex 2024-05-27T07:36:57.071 DEBUG:teuthology.orchestra.run.smithi052:> dd if=/proc/self/mounts of=/dev/stdout 2024-05-27T07:36:57.129 DEBUG:teuthology.orchestra.run.smithi052:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:38f9d900c83c63799fbdbe61acc9a11b0d3554a6 shell --fsid 73df2636-1bf8-11ef-bc9b-c7b262605968 -- ceph osd blocklist ls 2024-05-27T07:36:57.442 INFO:teuthology.orchestra.run.smithi052.stderr:Inferring config /var/lib/ceph/73df2636-1bf8-11ef-bc9b-c7b262605968/mon.a/config 2024-05-27T07:36:58.597 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:36:58 smithi052 ceph-mon[31955]: pgmap v370: 129 pgs: 4 active+clean+snaptrim, 8 active+clean+snaptrim_wait, 117 active+clean; 620 MiB data, 7.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-05-27T07:36:58.696 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:36:58 smithi105 ceph-mon[35977]: pgmap v370: 129 pgs: 4 active+clean+snaptrim, 8 active+clean+snaptrim_wait, 117 active+clean; 620 MiB data, 7.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-05-27T07:36:58.858 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:36:58 smithi173 ceph-mon[37000]: pgmap v370: 129 pgs: 4 active+clean+snaptrim, 8 active+clean+snaptrim_wait, 117 active+clean; 620 MiB data, 7.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-05-27T07:37:00.192 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:59.956+0000 7f9727c2c700 1 -- 172.21.15.52:0/3490874263 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f97201034b0 msgr2=0x7f97201038c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:37:00.192 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:59.956+0000 7f9727c2c700 1 --2- 172.21.15.52:0/3490874263 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f97201034b0 0x7f97201038c0 secure :-1 s=READY pgs=767 cs=0 l=1 rev1=1 crypto rx=0x7f971000b0d0 tx=0x7f971000b3e0 comp rx=0 tx=0).stop 2024-05-27T07:37:00.192 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:59.956+0000 7f9727c2c700 1 -- 172.21.15.52:0/3490874263 shutdown_connections 2024-05-27T07:37:00.192 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:59.956+0000 7f9727c2c700 1 --2- 172.21.15.52:0/3490874263 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f9720108730 0x7f9720110090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:37:00.192 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:59.956+0000 7f9727c2c700 1 --2- 172.21.15.52:0/3490874263 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f97201034b0 0x7f97201038c0 unknown :-1 s=CLOSED pgs=767 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:37:00.192 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:59.956+0000 7f9727c2c700 1 --2- 172.21.15.52:0/3490874263 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f9720107de0 0x7f97201081f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:37:00.192 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:59.956+0000 7f9727c2c700 1 -- 172.21.15.52:0/3490874263 >> 172.21.15.52:0/3490874263 conn(0x7f97200fedb0 msgr2=0x7f97201011d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:37:00.193 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:59.956+0000 7f9727c2c700 1 -- 172.21.15.52:0/3490874263 shutdown_connections 2024-05-27T07:37:00.193 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:59.956+0000 7f9727c2c700 1 -- 172.21.15.52:0/3490874263 wait complete. 2024-05-27T07:37:00.193 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:59.957+0000 7f9727c2c700 1 Processor -- start 2024-05-27T07:37:00.193 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:59.957+0000 7f9727c2c700 1 -- start start 2024-05-27T07:37:00.193 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:59.957+0000 7f9727c2c700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f97201034b0 0x7f9720199d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:37:00.193 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:59.957+0000 7f9727c2c700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f9720107de0 0x7f972019a2b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:37:00.193 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:59.957+0000 7f9727c2c700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f9720108730 0x7f972019e540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:37:00.193 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:59.957+0000 7f9727c2c700 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x7f9720113360 con 0x7f97201034b0 2024-05-27T07:37:00.193 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:59.957+0000 7f9727c2c700 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f97201131e0 con 0x7f9720108730 2024-05-27T07:37:00.193 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:59.957+0000 7f9727c2c700 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f97201134e0 con 0x7f9720107de0 2024-05-27T07:37:00.193 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:59.957+0000 7f97259c8700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f97201034b0 0x7f9720199d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:37:00.193 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:59.957+0000 7f97259c8700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f97201034b0 0x7f9720199d10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.52:39266/0 (socket says 172.21.15.52:39266) 2024-05-27T07:37:00.194 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:59.957+0000 7f97259c8700 1 -- 172.21.15.52:0/436257540 learned_addr learned my addr 172.21.15.52:0/436257540 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-05-27T07:37:00.194 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:59.957+0000 7f97261c9700 1 --2- 172.21.15.52:0/436257540 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f9720108730 0x7f972019e540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:37:00.194 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:59.957+0000 7f97251c7700 1 --2- 172.21.15.52:0/436257540 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f9720107de0 0x7f972019a2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:37:00.194 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:59.958+0000 7f97259c8700 1 -- 172.21.15.52:0/436257540 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f9720107de0 msgr2=0x7f972019a2b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:37:00.194 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:59.958+0000 7f97259c8700 1 --2- 172.21.15.52:0/436257540 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f9720107de0 0x7f972019a2b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:37:00.194 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:59.958+0000 7f97259c8700 1 -- 172.21.15.52:0/436257540 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f9720108730 msgr2=0x7f972019e540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:37:00.194 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:59.958+0000 7f97259c8700 1 --2- 172.21.15.52:0/436257540 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f9720108730 0x7f972019e540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:37:00.194 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:59.958+0000 7f97259c8700 1 -- 172.21.15.52:0/436257540 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f972019ebe0 con 0x7f97201034b0 2024-05-27T07:37:00.194 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:59.958+0000 7f97259c8700 1 --2- 172.21.15.52:0/436257540 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f97201034b0 0x7f9720199d10 secure :-1 s=READY pgs=768 cs=0 l=1 rev1=1 crypto rx=0x7f9710007a30 tx=0x7f9710007a60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:37:00.194 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:59.958+0000 7f971effd700 1 -- 172.21.15.52:0/436257540 <== mon.0 v2:172.21.15.52:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f971001f070 con 0x7f97201034b0 2024-05-27T07:37:00.194 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:59.958+0000 7f9727c2c700 1 -- 172.21.15.52:0/436257540 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f972019ee70 con 0x7f97201034b0 2024-05-27T07:37:00.194 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:59.958+0000 7f971effd700 1 -- 172.21.15.52:0/436257540 <== mon.0 v2:172.21.15.52:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f9710004c50 con 0x7f97201034b0 2024-05-27T07:37:00.194 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:59.958+0000 7f971effd700 1 -- 172.21.15.52:0/436257540 <== mon.0 v2:172.21.15.52:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f971001b690 con 0x7f97201034b0 2024-05-27T07:37:00.195 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:59.958+0000 7f9727c2c700 1 -- 172.21.15.52:0/436257540 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f972019f350 con 0x7f97201034b0 2024-05-27T07:37:00.195 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:59.959+0000 7f9727c2c700 1 -- 172.21.15.52:0/436257540 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f97200512b0 con 0x7f97201034b0 2024-05-27T07:37:00.195 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:59.962+0000 7f971effd700 1 -- 172.21.15.52:0/436257540 <== mon.0 v2:172.21.15.52:3300/0 4 ==== mgrmap(e 26) ==== 149783+0+0 (secure 0 0 0) 0x7f9710022070 con 0x7f97201034b0 2024-05-27T07:37:00.195 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:59.962+0000 7f971effd700 1 --2- 172.21.15.52:0/436257540 >> [v2:172.21.15.52:6800/1822318384,v1:172.21.15.52:6801/1822318384] conn(0x7f97080b29f0 0x7f97080b4f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:37:00.195 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:59.963+0000 7f97251c7700 1 --2- 172.21.15.52:0/436257540 >> [v2:172.21.15.52:6800/1822318384,v1:172.21.15.52:6801/1822318384] conn(0x7f97080b29f0 0x7f97080b4f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:37:00.195 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:59.963+0000 7f971effd700 1 -- 172.21.15.52:0/436257540 <== mon.0 v2:172.21.15.52:3300/0 5 ==== osd_map(127..127 src has 1..127) ==== 9555+0+0 (secure 0 0 0) 0x7f971000f020 con 0x7f97201034b0 2024-05-27T07:37:00.195 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:59.963+0000 7f97251c7700 1 --2- 172.21.15.52:0/436257540 >> [v2:172.21.15.52:6800/1822318384,v1:172.21.15.52:6801/1822318384] conn(0x7f97080b29f0 0x7f97080b4f00 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7f9720107d60 tx=0x7f9714009260 comp rx=0 tx=0).ready entity=mgr.15333 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:37:00.195 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:36:59.966+0000 7f971effd700 1 -- 172.21.15.52:0/436257540 <== mon.0 v2:172.21.15.52:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191963 (secure 0 0 0) 0x7f9710029020 con 0x7f97201034b0 2024-05-27T07:37:00.202 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:37:00.206+0000 7f9727c2c700 1 -- 172.21.15.52:0/436257540 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) -- 0x7f972002e5e0 con 0x7f97201034b0 2024-05-27T07:37:00.202 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:37:00.206+0000 7f971effd700 1 -- 172.21.15.52:0/436257540 <== mon.0 v2:172.21.15.52:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 16 entries v128) ==== 81+0+941 (secure 0 0 0) 0x7f97100ab490 con 0x7f97201034b0 2024-05-27T07:37:00.203 INFO:teuthology.orchestra.run.smithi052.stdout:172.21.15.52:0/3430810618 2024-05-28T07:25:56.925056+0000 2024-05-27T07:37:00.203 INFO:teuthology.orchestra.run.smithi052.stdout:172.21.15.52:0/2723769517 2024-05-28T07:25:56.925056+0000 2024-05-27T07:37:00.203 INFO:teuthology.orchestra.run.smithi052.stdout:172.21.15.52:0/262308791 2024-05-28T07:25:56.925056+0000 2024-05-27T07:37:00.203 INFO:teuthology.orchestra.run.smithi052.stdout:172.21.15.52:6801/1631683163 2024-05-28T07:14:29.544633+0000 2024-05-27T07:37:00.203 INFO:teuthology.orchestra.run.smithi052.stdout:172.21.15.52:6801/3513075805 2024-05-28T07:25:56.925056+0000 2024-05-27T07:37:00.203 INFO:teuthology.orchestra.run.smithi052.stdout:172.21.15.52:0/2776144659 2024-05-28T07:14:29.544633+0000 2024-05-27T07:37:00.203 INFO:teuthology.orchestra.run.smithi052.stdout:172.21.15.52:0/3577886285 2024-05-28T07:25:56.925056+0000 2024-05-27T07:37:00.203 INFO:teuthology.orchestra.run.smithi052.stdout:172.21.15.52:6800/1631683163 2024-05-28T07:14:29.544633+0000 2024-05-27T07:37:00.203 INFO:teuthology.orchestra.run.smithi052.stdout:172.21.15.52:0/3675529097 2024-05-28T07:14:01.960550+0000 2024-05-27T07:37:00.203 INFO:teuthology.orchestra.run.smithi052.stdout:172.21.15.52:0/727808107 2024-05-28T07:14:01.960550+0000 2024-05-27T07:37:00.203 INFO:teuthology.orchestra.run.smithi052.stdout:172.21.15.52:0/3839436833 2024-05-28T07:14:29.544633+0000 2024-05-27T07:37:00.204 INFO:teuthology.orchestra.run.smithi052.stdout:172.21.15.52:6800/587662512 2024-05-28T07:14:01.960550+0000 2024-05-27T07:37:00.204 INFO:teuthology.orchestra.run.smithi052.stdout:172.21.15.52:6800/3513075805 2024-05-28T07:25:56.925056+0000 2024-05-27T07:37:00.204 INFO:teuthology.orchestra.run.smithi052.stdout:172.21.15.52:6801/587662512 2024-05-28T07:14:01.960550+0000 2024-05-27T07:37:00.204 INFO:teuthology.orchestra.run.smithi052.stdout:172.21.15.52:0/853867465 2024-05-28T07:14:01.960550+0000 2024-05-27T07:37:00.204 INFO:teuthology.orchestra.run.smithi052.stdout:172.21.15.52:0/3480918279 2024-05-28T07:14:29.544633+0000 2024-05-27T07:37:00.205 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:37:00.209+0000 7f9727c2c700 1 -- 172.21.15.52:0/436257540 >> [v2:172.21.15.52:6800/1822318384,v1:172.21.15.52:6801/1822318384] conn(0x7f97080b29f0 msgr2=0x7f97080b4f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:37:00.205 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:37:00.209+0000 7f9727c2c700 1 --2- 172.21.15.52:0/436257540 >> [v2:172.21.15.52:6800/1822318384,v1:172.21.15.52:6801/1822318384] conn(0x7f97080b29f0 0x7f97080b4f00 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7f9720107d60 tx=0x7f9714009260 comp rx=0 tx=0).stop 2024-05-27T07:37:00.205 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:37:00.209+0000 7f9727c2c700 1 -- 172.21.15.52:0/436257540 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f97201034b0 msgr2=0x7f9720199d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:37:00.205 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:37:00.209+0000 7f9727c2c700 1 --2- 172.21.15.52:0/436257540 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f97201034b0 0x7f9720199d10 secure :-1 s=READY pgs=768 cs=0 l=1 rev1=1 crypto rx=0x7f9710007a30 tx=0x7f9710007a60 comp rx=0 tx=0).stop 2024-05-27T07:37:00.205 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:37:00.209+0000 7f9727c2c700 1 -- 172.21.15.52:0/436257540 shutdown_connections 2024-05-27T07:37:00.205 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:37:00.209+0000 7f9727c2c700 1 --2- 172.21.15.52:0/436257540 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f9720108730 0x7f972019e540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:37:00.205 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:37:00.209+0000 7f9727c2c700 1 --2- 172.21.15.52:0/436257540 >> [v2:172.21.15.52:6800/1822318384,v1:172.21.15.52:6801/1822318384] conn(0x7f97080b29f0 0x7f97080b4f00 unknown :-1 s=CLOSED pgs=206 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:37:00.205 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:37:00.209+0000 7f9727c2c700 1 --2- 172.21.15.52:0/436257540 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f97201034b0 0x7f9720199d10 unknown :-1 s=CLOSED pgs=768 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:37:00.205 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:37:00.209+0000 7f9727c2c700 1 --2- 172.21.15.52:0/436257540 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f9720107de0 0x7f972019a2b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:37:00.206 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:37:00.209+0000 7f9727c2c700 1 -- 172.21.15.52:0/436257540 >> 172.21.15.52:0/436257540 conn(0x7f97200fedb0 msgr2=0x7f97201011d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:37:00.206 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:37:00.209+0000 7f9727c2c700 1 -- 172.21.15.52:0/436257540 shutdown_connections 2024-05-27T07:37:00.206 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:37:00.209+0000 7f9727c2c700 1 -- 172.21.15.52:0/436257540 wait complete. 2024-05-27T07:37:00.206 INFO:teuthology.orchestra.run.smithi052.stderr:listed 16 entries 2024-05-27T07:37:00.485 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:37:00 smithi052 ceph-mon[31955]: pgmap v371: 129 pgs: 2 active+clean+snaptrim, 2 active+clean+snaptrim_wait, 125 active+clean; 620 MiB data, 7.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-05-27T07:37:00.485 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:37:00 smithi052 ceph-mon[31955]: osdmap e128: 12 total, 12 up, 12 in 2024-05-27T07:37:00.485 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:37:00 smithi052 ceph-mon[31955]: from='client.? 172.21.15.52:0/436257540' entity='client.admin' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-05-27T07:37:00.696 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:37:00 smithi105 ceph-mon[35977]: pgmap v371: 129 pgs: 2 active+clean+snaptrim, 2 active+clean+snaptrim_wait, 125 active+clean; 620 MiB data, 7.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-05-27T07:37:00.696 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:37:00 smithi105 ceph-mon[35977]: osdmap e128: 12 total, 12 up, 12 in 2024-05-27T07:37:00.696 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:37:00 smithi105 ceph-mon[35977]: from='client.? 172.21.15.52:0/436257540' entity='client.admin' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-05-27T07:37:00.858 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:37:00 smithi173 ceph-mon[37000]: pgmap v371: 129 pgs: 2 active+clean+snaptrim, 2 active+clean+snaptrim_wait, 125 active+clean; 620 MiB data, 7.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-05-27T07:37:00.858 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:37:00 smithi173 ceph-mon[37000]: osdmap e128: 12 total, 12 up, 12 in 2024-05-27T07:37:00.858 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:37:00 smithi173 ceph-mon[37000]: from='client.? 172.21.15.52:0/436257540' entity='client.admin' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-05-27T07:37:01.275 DEBUG:tasks.cephfs.kernel_mount:Unmounting client client.0... 2024-05-27T07:37:01.275 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-27T07:37:01.275 DEBUG:teuthology.orchestra.run.smithi052:> sudo umount /home/ubuntu/cephtest/mnt.0 2024-05-27T07:37:01.467 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi052.front.sepia.ceph.com 2024-05-27T07:37:01.467 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-27T07:37:01.467 DEBUG:teuthology.orchestra.run.smithi052:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2024-05-27T07:37:01.514 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-27T07:37:01.514 DEBUG:teuthology.orchestra.run.smithi052:> ip netns list 2024-05-27T07:37:01.561 INFO:teuthology.orchestra.run.smithi052.stdout:ceph-ns--home-ubuntu-cephtest-mnt.0 (id: 0) 2024-05-27T07:37:01.561 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-27T07:37:01.562 DEBUG:teuthology.orchestra.run.smithi052:> sudo ip netns delete ceph-ns--home-ubuntu-cephtest-mnt.0 2024-05-27T07:37:01.673 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-27T07:37:01.673 DEBUG:teuthology.orchestra.run.smithi052:> sudo ip link delete ceph-brx 2024-05-27T07:37:01.771 INFO:tasks.kclient:Unmounting kernel clients... 2024-05-27T07:37:01.771 DEBUG:teuthology.orchestra.run.smithi052:> set -ex 2024-05-27T07:37:01.771 DEBUG:teuthology.orchestra.run.smithi052:> dd if=/proc/self/mounts of=/dev/stdout 2024-05-27T07:37:01.809 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4057103,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6498912k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=322 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249452k,nr_inodes=812363,mode=700,uid=1000,gid=1316,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/sda1 /var/lib/containers/storage/overlay ext4 rw,seclabel,relatime 0 0 overlay /var/lib/containers/storage/overlay/339b609a774606614a4357911c7efdf1485294f41086dc57156ce1506c859e82/merged overlay rw,context="system_u:object_r:container_file_t:s0:c1022,c1023",relatime,lowerdir=/var/lib/containers/storage/overlay/l/HWOJAUINDDYF4PFSJO5NBJIDVG:/var/lib/containers/storage/overlay/l/XBXPWWNV7WWGJBRVLHAY7MWYU7,upperdir=/var/lib/containers/storage/overlay/339b609a774606614a4357911c7efdf1485294f41086dc57156ce1506c859e82/diff,workdir=/var/lib/containers/storage/overlay/339b609a774606614a4357911c7efdf1485294f41086dc57156ce1506c859e82/work,metacopy=on,volatile 0 0 overlay /var/lib/containers/storage/overlay/156afeab7561c4315f8c0269174cc6d7fc51ff200896f8ecdd8453475afa51f5/merged overlay rw,context="system_u:object_r:container_file_t:s0:c1022,c1023",nodev,relatime,lowerdir=/var/lib/containers/storage/overlay/l/HWOJAUINDDYF4PFSJO5NBJIDVG:/var/lib/containers/storage/overlay/l/XBXPWWNV7WWGJBRVLHAY7MWYU7,upperdir=/var/lib/containers/storage/overlay/156afeab7561c4315f8c0269174cc6d7fc51ff200896f8ecdd8453475afa51f5/diff,workdir=/var/lib/containers/storage/overlay/156afeab7561c4315f8c0269174cc6d7fc51ff200896f8ecdd8453475afa51f5/work,metacopy=on,volatile 0 0 tmpfs /run/user/0 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249452k,nr_inodes=812363,mode=700,inode64 0 0 binfmt_misc /proc/sys/fs/binfmt_misc binfmt_misc rw,nosuid,nodev,noexec,relatime 0 0 overlay /var/lib/containers/storage/overlay/57343f17079f73a0c532f85d4aa22f6ba25f238945cd9874c632b74b3232ec25/merged overlay rw,context="system_u:object_r:container_file_t:s0:c1022,c1023",relatime,lowerdir=/var/lib/containers/storage/overlay/l/HWOJAUINDDYF4PFSJO5NBJIDVG:/var/lib/containers/storage/overlay/l/XBXPWWNV7WWGJBRVLHAY7MWYU7,upperdir=/var/lib/containers/storage/overlay/57343f17079f73a0c532f85d4aa22f6ba25f238945cd9874c632b74b3232ec25/diff,workdir=/var/lib/containers/storage/overlay/57343f17079f73a0c532f85d4aa22f6ba25f238945cd9874c632b74b3232ec25/work,metacopy=on,volatile 0 0 overlay /var/lib/containers/storage/overlay/8032bf106ed313012e44c3d322c9802484683cfca36223e8fd3f0691555a7f4d/merged overlay rw,context="system_u:object_r:container_file_t:s0:c1022,c1023",relatime,lowerdir=/var/lib/containers/storage/overlay/l/HWOJAUINDDYF4PFSJO5NBJIDVG:/var/lib/containers/storage/overlay/l/XBXPWWNV7WWGJBRVLHAY7MWYU7,upperdir=/var/lib/containers/storage/overlay/8032bf106ed313012e44c3d322c9802484683cfca36223e8fd3f0691555a7f4d/diff,workdir=/var/lib/containers/storage/overlay/8032bf106ed313012e44c3d322c9802484683cfca36223e8fd3f0691555a7f4d/work,metacopy=on,volatile 0 0 overlay /var/lib/containers/storage/overlay/0df7e3d814ba0a1e066e632896b4b3d19bc0a7b31de4a6efbe598517d3d4b3ee/merged overlay rw,context="system_u:object_r:container_file_t:s0:c1022,c1023",relatime,lowerdir=/var/lib/containers/storage/overlay/l/HWOJAUINDDYF4PFSJO5NBJIDVG:/var/lib/containers/storage/overlay/l/XBXPWWNV7WWGJBRVLHAY7MWYU7,upperdir=/var/lib/containers/storage/overlay/0df7e3d814ba0a1e066e632896b4b3d19bc0a7b31de4a6efbe598517d3d4b3ee/diff,workdir=/var/lib/containers/storage/overlay/0df7e3d814ba0a1e066e632896b4b3d19bc0a7b31de4a6efbe598517d3d4b3ee/work,metacopy=on,volatile 0 0 overlay /var/lib/containers/storage/overlay/eb022041ddf436dc94838d9f6739734727769c7ddd3193bcc904b0f0d41551fd/merged overlay rw,context="system_u:object_r:container_file_t:s0:c1022,c1023",relatime,lowerdir=/var/lib/containers/storage/overlay/l/HWOJAUINDDYF4PFSJO5NBJIDVG:/var/lib/containers/storage/overlay/l/XBXPWWNV7WWGJBRVLHAY7MWYU7,upperdir=/var/lib/containers/storage/overlay/eb022041ddf436dc94838d9f6739734727769c7ddd3193bcc904b0f0d41551fd/diff,workdir=/var/lib/containers/storage/overlay/eb022041ddf436dc94838d9f6739734727769c7ddd3193bcc904b0f0d41551fd/work,metacopy=on,volatile 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6498912k,nr_inodes=819200,mode=755,inode64 0 0 2024-05-27T07:37:01.810 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-27T07:37:01.810 DEBUG:teuthology.orchestra.run.smithi052:> ip netns list 2024-05-27T07:37:01.894 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-27T07:37:01.894 DEBUG:teuthology.orchestra.run.smithi052:> sudo ip link delete ceph-brx 2024-05-27T07:37:01.985 INFO:teuthology.orchestra.run.smithi052.stderr:Cannot find device "ceph-brx" 2024-05-27T07:37:01.987 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-05-27T07:37:01.987 DEBUG:teuthology.run_tasks:Unwinding manager cephadm 2024-05-27T07:37:01.998 INFO:tasks.cephadm:Teardown begin 2024-05-27T07:37:01.999 INFO:tasks.cephfs.filesystem:Destroying file system cephfs and related pools 2024-05-27T07:37:01.999 DEBUG:teuthology.orchestra.run.smithi052:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:38f9d900c83c63799fbdbe61acc9a11b0d3554a6 shell --fsid 73df2636-1bf8-11ef-bc9b-c7b262605968 -- ceph fs dump --format=json 2024-05-27T07:37:02.259 INFO:teuthology.orchestra.run.smithi052.stderr:Inferring config /var/lib/ceph/73df2636-1bf8-11ef-bc9b-c7b262605968/mon.a/config 2024-05-27T07:37:02.638 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:37:02 smithi052 ceph-mon[31955]: pgmap v373: 129 pgs: 1 active+clean+snaptrim, 2 active+clean+snaptrim_wait, 126 active+clean; 620 MiB data, 7.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 27 KiB/s wr, 2 op/s 2024-05-27T07:37:02.696 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:37:02 smithi105 ceph-mon[35977]: pgmap v373: 129 pgs: 1 active+clean+snaptrim, 2 active+clean+snaptrim_wait, 126 active+clean; 620 MiB data, 7.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 27 KiB/s wr, 2 op/s 2024-05-27T07:37:02.858 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:37:02 smithi173 ceph-mon[37000]: pgmap v373: 129 pgs: 1 active+clean+snaptrim, 2 active+clean+snaptrim_wait, 126 active+clean; 620 MiB data, 7.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 27 KiB/s wr, 2 op/s 2024-05-27T07:37:03.696 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:37:03 smithi105 ceph-mon[35977]: osdmap e129: 12 total, 12 up, 12 in 2024-05-27T07:37:03.696 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:37:03 smithi105 ceph-mon[35977]: pgmap v375: 129 pgs: 17 active+clean+snaptrim_wait, 6 active+clean+snaptrim, 106 active+clean; 619 MiB data, 7.4 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 68 KiB/s wr, 26 op/s 2024-05-27T07:37:03.736 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:37:03 smithi052 ceph-mon[31955]: osdmap e129: 12 total, 12 up, 12 in 2024-05-27T07:37:03.736 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:37:03 smithi052 ceph-mon[31955]: pgmap v375: 129 pgs: 17 active+clean+snaptrim_wait, 6 active+clean+snaptrim, 106 active+clean; 619 MiB data, 7.4 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 68 KiB/s wr, 26 op/s 2024-05-27T07:37:03.858 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:37:03 smithi173 ceph-mon[37000]: osdmap e129: 12 total, 12 up, 12 in 2024-05-27T07:37:03.858 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:37:03 smithi173 ceph-mon[37000]: pgmap v375: 129 pgs: 17 active+clean+snaptrim_wait, 6 active+clean+snaptrim, 106 active+clean; 619 MiB data, 7.4 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 68 KiB/s wr, 26 op/s 2024-05-27T07:37:04.231 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:37:04.234+0000 7fb8650ef700 1 -- 172.21.15.52:0/742914752 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fb860110650 msgr2=0x7fb860110a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:37:04.231 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:37:04.234+0000 7fb8650ef700 1 --2- 172.21.15.52:0/742914752 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fb860110650 0x7fb860110a60 secure :-1 s=READY pgs=769 cs=0 l=1 rev1=1 crypto rx=0x7fb8500099c0 tx=0x7fb850009cd0 comp rx=0 tx=0).stop 2024-05-27T07:37:04.231 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:37:04.234+0000 7fb8650ef700 1 -- 172.21.15.52:0/742914752 shutdown_connections 2024-05-27T07:37:04.232 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:37:04.234+0000 7fb8650ef700 1 --2- 172.21.15.52:0/742914752 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fb860110fa0 0x7fb8601177d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:37:04.232 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:37:04.234+0000 7fb8650ef700 1 --2- 172.21.15.52:0/742914752 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fb86010b890 0x7fb86010bca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:37:04.232 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:37:04.234+0000 7fb8650ef700 1 --2- 172.21.15.52:0/742914752 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fb860110650 0x7fb860110a60 secure :-1 s=CLOSED pgs=769 cs=0 l=1 rev1=1 crypto rx=0x7fb8500099c0 tx=0x7fb850009cd0 comp rx=0 tx=0).stop 2024-05-27T07:37:04.232 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:37:04.234+0000 7fb8650ef700 1 -- 172.21.15.52:0/742914752 >> 172.21.15.52:0/742914752 conn(0x7fb86007a010 msgr2=0x7fb86007c430 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:37:04.232 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:37:04.236+0000 7fb8650ef700 1 -- 172.21.15.52:0/742914752 shutdown_connections 2024-05-27T07:37:04.233 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:37:04.236+0000 7fb8650ef700 1 -- 172.21.15.52:0/742914752 wait complete. 2024-05-27T07:37:04.233 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:37:04.237+0000 7fb8650ef700 1 Processor -- start 2024-05-27T07:37:04.233 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:37:04.237+0000 7fb8650ef700 1 -- start start 2024-05-27T07:37:04.234 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:37:04.238+0000 7fb8650ef700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fb86010b890 0x7fb86006e540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:37:04.234 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:37:04.238+0000 7fb8650ef700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fb860110fa0 0x7fb86006ea80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:37:04.234 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:37:04.238+0000 7fb8650ef700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fb86006efc0 0x7fb860072f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:37:04.234 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:37:04.238+0000 7fb85e59c700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fb860110fa0 0x7fb86006ea80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:37:04.235 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:37:04.238+0000 7fb8650ef700 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x7fb86011abe0 con 0x7fb860110fa0 2024-05-27T07:37:04.235 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:37:04.238+0000 7fb85e59c700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fb860110fa0 0x7fb86006ea80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.52:55566/0 (socket says 172.21.15.52:55566) 2024-05-27T07:37:04.235 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:37:04.238+0000 7fb85e59c700 1 -- 172.21.15.52:0/508405300 learned_addr learned my addr 172.21.15.52:0/508405300 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-05-27T07:37:04.235 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:37:04.238+0000 7fb8650ef700 1 -- 172.21.15.52:0/508405300 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7fb86011aa60 con 0x7fb86006efc0 2024-05-27T07:37:04.235 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:37:04.238+0000 7fb8650ef700 1 -- 172.21.15.52:0/508405300 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7fb86011ad60 con 0x7fb86010b890 2024-05-27T07:37:04.235 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:37:04.238+0000 7fb85f59e700 1 --2- 172.21.15.52:0/508405300 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fb86006efc0 0x7fb860072f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:37:04.236 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:37:04.239+0000 7fb85e59c700 1 -- 172.21.15.52:0/508405300 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fb86010b890 msgr2=0x7fb86006e540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:37:04.236 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:37:04.239+0000 7fb85e59c700 1 --2- 172.21.15.52:0/508405300 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fb86010b890 0x7fb86006e540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:37:04.236 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:37:04.239+0000 7fb85e59c700 1 -- 172.21.15.52:0/508405300 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fb86006efc0 msgr2=0x7fb860072f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:37:04.236 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:37:04.239+0000 7fb85e59c700 1 --2- 172.21.15.52:0/508405300 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fb86006efc0 0x7fb860072f40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:37:04.236 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:37:04.239+0000 7fb85e59c700 1 -- 172.21.15.52:0/508405300 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb860073480 con 0x7fb860110fa0 2024-05-27T07:37:04.236 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:37:04.239+0000 7fb85e59c700 1 --2- 172.21.15.52:0/508405300 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fb860110fa0 0x7fb86006ea80 secure :-1 s=READY pgs=770 cs=0 l=1 rev1=1 crypto rx=0x7fb85000b580 tx=0x7fb8500055f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:37:04.236 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:37:04.240+0000 7fb847fff700 1 -- 172.21.15.52:0/508405300 <== mon.0 v2:172.21.15.52:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb850009e60 con 0x7fb860110fa0 2024-05-27T07:37:04.236 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:37:04.240+0000 7fb8650ef700 1 -- 172.21.15.52:0/508405300 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb860073710 con 0x7fb860110fa0 2024-05-27T07:37:04.237 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:37:04.240+0000 7fb8650ef700 1 -- 172.21.15.52:0/508405300 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb860073ae0 con 0x7fb860110fa0 2024-05-27T07:37:04.237 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:37:04.240+0000 7fb847fff700 1 -- 172.21.15.52:0/508405300 <== mon.0 v2:172.21.15.52:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fb85000f460 con 0x7fb860110fa0 2024-05-27T07:37:04.237 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:37:04.240+0000 7fb847fff700 1 -- 172.21.15.52:0/508405300 <== mon.0 v2:172.21.15.52:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb850017650 con 0x7fb860110fa0 2024-05-27T07:37:04.241 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:37:04.245+0000 7fb847fff700 1 -- 172.21.15.52:0/508405300 <== mon.0 v2:172.21.15.52:3300/0 4 ==== mgrmap(e 26) ==== 149783+0+0 (secure 0 0 0) 0x7fb8500177f0 con 0x7fb860110fa0 2024-05-27T07:37:04.243 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:37:04.246+0000 7fb8650ef700 1 -- 172.21.15.52:0/508405300 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb86007eac0 con 0x7fb860110fa0 2024-05-27T07:37:04.243 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:37:04.246+0000 7fb847fff700 1 --2- 172.21.15.52:0/508405300 >> [v2:172.21.15.52:6800/1822318384,v1:172.21.15.52:6801/1822318384] conn(0x7fb8480b2c00 0x7fb8480b5110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:37:04.243 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:37:04.247+0000 7fb847fff700 1 -- 172.21.15.52:0/508405300 <== mon.0 v2:172.21.15.52:3300/0 5 ==== osd_map(129..129 src has 1..129) ==== 9571+0+0 (secure 0 0 0) 0x7fb8500e45b0 con 0x7fb860110fa0 2024-05-27T07:37:04.243 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:37:04.247+0000 7fb85ed9d700 1 --2- 172.21.15.52:0/508405300 >> [v2:172.21.15.52:6800/1822318384,v1:172.21.15.52:6801/1822318384] conn(0x7fb8480b2c00 0x7fb8480b5110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:37:04.244 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:37:04.248+0000 7fb85ed9d700 1 --2- 172.21.15.52:0/508405300 >> [v2:172.21.15.52:6800/1822318384,v1:172.21.15.52:6801/1822318384] conn(0x7fb8480b2c00 0x7fb8480b5110 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7fb854005950 tx=0x7fb8540058e0 comp rx=0 tx=0).ready entity=mgr.15333 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:37:04.254 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:37:04.256+0000 7fb847fff700 1 -- 172.21.15.52:0/508405300 <== mon.0 v2:172.21.15.52:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191963 (secure 0 0 0) 0x7fb850013070 con 0x7fb860110fa0 2024-05-27T07:37:04.727 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:37:04.731+0000 7fb8650ef700 1 -- 172.21.15.52:0/508405300 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fb8600512b0 con 0x7fb860110fa0 2024-05-27T07:37:04.730 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:37:04.734+0000 7fb847fff700 1 -- 172.21.15.52:0/508405300 <== mon.0 v2:172.21.15.52:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 108 v108) ==== 95+0+6124 (secure 0 0 0) 0x7fb8500abba0 con 0x7fb860110fa0 2024-05-27T07:37:04.731 INFO:teuthology.orchestra.run.smithi052.stdout: 2024-05-27T07:37:04.731 INFO:teuthology.orchestra.run.smithi052.stdout:{"epoch":108,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":108,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-05-27T07:21:50.521783+0000","modified":"2024-05-27T07:35:15.151893+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24527,"mds_1":24497,"mds_2":24475,"mds_3":24473,"mds_4":24466},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24466":{"gid":24466,"name":"c","rank":4,"incarnation":26,"state":"up:active","state_seq":25,"addr":"172.21.15.173:6837/1396363827","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6836","nonce":1396363827},{"type":"v1","addr":"172.21.15.173:6837","nonce":1396363827}]},"join_fscid":-1,"export_targets":[0],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24473":{"gid":24473,"name":"e","rank":3,"incarnation":24,"state":"up:active","state_seq":26,"addr":"172.21.15.105:6835/3298498384","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6834","nonce":3298498384},{"type":"v1","addr":"172.21.15.105:6835","nonce":3298498384}]},"join_fscid":-1,"export_targets":[0],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24475":{"gid":24475,"name":"i","rank":2,"incarnation":22,"state":"up:active","state_seq":23,"addr":"172.21.15.173:6839/564778293","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6838","nonce":564778293},{"type":"v1","addr":"172.21.15.173:6839","nonce":564778293}]},"join_fscid":-1,"export_targets":[0],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24497":{"gid":24497,"name":"h","rank":1,"incarnation":20,"state":"up:active","state_seq":24,"addr":"172.21.15.105:6837/191393149","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6836","nonce":191393149},{"type":"v1","addr":"172.21.15.105:6837","nonce":191393149}]},"join_fscid":-1,"export_targets":[0],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24527":{"gid":24527,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":5,"addr":"172.21.15.105:6839/500348151","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6838","nonce":500348151},{"type":"v1","addr":"172.21.15.105:6839","nonce":500348151}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24527,"qdb_cluster":[24497,24527,24475,24473,24466]},"id":1}]} 2024-05-27T07:37:04.735 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:37:04.738+0000 7fb8650ef700 1 -- 172.21.15.52:0/508405300 >> [v2:172.21.15.52:6800/1822318384,v1:172.21.15.52:6801/1822318384] conn(0x7fb8480b2c00 msgr2=0x7fb8480b5110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:37:04.735 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:37:04.738+0000 7fb8650ef700 1 --2- 172.21.15.52:0/508405300 >> [v2:172.21.15.52:6800/1822318384,v1:172.21.15.52:6801/1822318384] conn(0x7fb8480b2c00 0x7fb8480b5110 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7fb854005950 tx=0x7fb8540058e0 comp rx=0 tx=0).stop 2024-05-27T07:37:04.735 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:37:04.738+0000 7fb8650ef700 1 -- 172.21.15.52:0/508405300 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fb860110fa0 msgr2=0x7fb86006ea80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:37:04.735 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:37:04.738+0000 7fb8650ef700 1 --2- 172.21.15.52:0/508405300 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fb860110fa0 0x7fb86006ea80 secure :-1 s=READY pgs=770 cs=0 l=1 rev1=1 crypto rx=0x7fb85000b580 tx=0x7fb8500055f0 comp rx=0 tx=0).stop 2024-05-27T07:37:04.735 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:37:04.738+0000 7fb8650ef700 1 -- 172.21.15.52:0/508405300 shutdown_connections 2024-05-27T07:37:04.735 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:37:04.738+0000 7fb8650ef700 1 --2- 172.21.15.52:0/508405300 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fb86006efc0 0x7fb860072f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:37:04.735 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:37:04.738+0000 7fb8650ef700 1 --2- 172.21.15.52:0/508405300 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fb86010b890 0x7fb86006e540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:37:04.735 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:37:04.738+0000 7fb8650ef700 1 --2- 172.21.15.52:0/508405300 >> [v2:172.21.15.52:6800/1822318384,v1:172.21.15.52:6801/1822318384] conn(0x7fb8480b2c00 0x7fb8480b5110 secure :-1 s=CLOSED pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7fb854005950 tx=0x7fb8540058e0 comp rx=0 tx=0).stop 2024-05-27T07:37:04.735 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:37:04.738+0000 7fb8650ef700 1 --2- 172.21.15.52:0/508405300 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fb860110fa0 0x7fb86006ea80 unknown :-1 s=CLOSED pgs=770 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:37:04.735 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:37:04.739+0000 7fb8650ef700 1 -- 172.21.15.52:0/508405300 >> 172.21.15.52:0/508405300 conn(0x7fb86007a010 msgr2=0x7fb86007c8b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:37:04.737 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:37:04.741+0000 7fb8650ef700 1 -- 172.21.15.52:0/508405300 shutdown_connections 2024-05-27T07:37:04.738 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:37:04.742+0000 7fb8650ef700 1 -- 172.21.15.52:0/508405300 wait complete. 2024-05-27T07:37:04.741 INFO:teuthology.orchestra.run.smithi052.stderr:dumped fsmap epoch 108 2024-05-27T07:37:04.985 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:37:04 smithi052 ceph-mon[31955]: from='client.? 172.21.15.52:0/508405300' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-27T07:37:05.108 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:37:04 smithi173 ceph-mon[37000]: from='client.? 172.21.15.52:0/508405300' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-27T07:37:05.196 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:37:04 smithi105 ceph-mon[35977]: from='client.? 172.21.15.52:0/508405300' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-27T07:37:05.974 DEBUG:teuthology.orchestra.run.smithi052:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:38f9d900c83c63799fbdbe61acc9a11b0d3554a6 shell --fsid 73df2636-1bf8-11ef-bc9b-c7b262605968 -- ceph fs dump --format=json 2024-05-27T07:37:06.235 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:37:06 smithi052 ceph-mon[31955]: osdmap e130: 12 total, 12 up, 12 in 2024-05-27T07:37:06.236 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:37:06 smithi052 ceph-mon[31955]: pgmap v377: 129 pgs: 52 active+clean+snaptrim_wait, 18 active+clean+snaptrim, 59 active+clean; 619 MiB data, 7.4 GiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 92 KiB/s wr, 36 op/s 2024-05-27T07:37:06.423 INFO:teuthology.orchestra.run.smithi052.stderr:Inferring config /var/lib/ceph/73df2636-1bf8-11ef-bc9b-c7b262605968/mon.a/config 2024-05-27T07:37:06.446 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:37:06 smithi105 ceph-mon[35977]: osdmap e130: 12 total, 12 up, 12 in 2024-05-27T07:37:06.446 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:37:06 smithi105 ceph-mon[35977]: pgmap v377: 129 pgs: 52 active+clean+snaptrim_wait, 18 active+clean+snaptrim, 59 active+clean; 619 MiB data, 7.4 GiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 92 KiB/s wr, 36 op/s 2024-05-27T07:37:06.608 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:37:06 smithi173 ceph-mon[37000]: osdmap e130: 12 total, 12 up, 12 in 2024-05-27T07:37:06.608 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:37:06 smithi173 ceph-mon[37000]: pgmap v377: 129 pgs: 52 active+clean+snaptrim_wait, 18 active+clean+snaptrim, 59 active+clean; 619 MiB data, 7.4 GiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 92 KiB/s wr, 36 op/s 2024-05-27T07:37:08.579 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:37:08 smithi052 ceph-mon[31955]: pgmap v378: 129 pgs: 64 active+clean+snaptrim_wait, 22 active+clean+snaptrim, 43 active+clean; 619 MiB data, 7.5 GiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 76 KiB/s wr, 30 op/s 2024-05-27T07:37:08.696 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:37:08 smithi105 ceph-mon[35977]: pgmap v378: 129 pgs: 64 active+clean+snaptrim_wait, 22 active+clean+snaptrim, 43 active+clean; 619 MiB data, 7.5 GiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 76 KiB/s wr, 30 op/s 2024-05-27T07:37:08.858 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:37:08 smithi173 ceph-mon[37000]: pgmap v378: 129 pgs: 64 active+clean+snaptrim_wait, 22 active+clean+snaptrim, 43 active+clean; 619 MiB data, 7.5 GiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 76 KiB/s wr, 30 op/s 2024-05-27T07:37:09.094 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:37:09.090+0000 7f3168d7b700 1 -- 172.21.15.52:0/745251768 <== mon.2 v2:172.21.15.105:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f315c012650 con 0x7f316411d920 2024-05-27T07:37:09.094 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:37:09.091+0000 7f316bfe1700 1 -- 172.21.15.52:0/745251768 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f316411d920 msgr2=0x7f316411fd40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:37:09.094 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:37:09.091+0000 7f316bfe1700 1 --2- 172.21.15.52:0/745251768 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f316411d920 0x7f316411fd40 secure :-1 s=READY pgs=283 cs=0 l=1 rev1=1 crypto rx=0x7f315c009f90 tx=0x7f315c00b330 comp rx=0 tx=0).stop 2024-05-27T07:37:09.101 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:37:09.097+0000 7f316bfe1700 1 -- 172.21.15.52:0/745251768 shutdown_connections 2024-05-27T07:37:09.101 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:37:09.097+0000 7f316bfe1700 1 --2- 172.21.15.52:0/745251768 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f316411d920 0x7f316411fd40 unknown :-1 s=CLOSED pgs=283 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:37:09.101 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:37:09.097+0000 7f316bfe1700 1 --2- 172.21.15.52:0/745251768 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f316410a920 0x7f316410ad30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:37:09.101 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:37:09.097+0000 7f316bfe1700 1 --2- 172.21.15.52:0/745251768 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f316410b270 0x7f316411d3e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:37:09.101 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:37:09.097+0000 7f316bfe1700 1 -- 172.21.15.52:0/745251768 >> 172.21.15.52:0/745251768 conn(0x7f316406f4c0 msgr2=0x7f316406f900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:37:09.101 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:37:09.100+0000 7f316bfe1700 1 -- 172.21.15.52:0/745251768 shutdown_connections 2024-05-27T07:37:09.101 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:37:09.100+0000 7f316bfe1700 1 -- 172.21.15.52:0/745251768 wait complete. 2024-05-27T07:37:09.101 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:37:09.101+0000 7f316bfe1700 1 Processor -- start 2024-05-27T07:37:09.101 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:37:09.104+0000 7f316bfe1700 1 -- start start 2024-05-27T07:37:09.101 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:37:09.104+0000 7f316bfe1700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f316410a920 0x7f316411cf90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:37:09.101 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:37:09.104+0000 7f316bfe1700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f316410b270 0x7f3164117100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:37:09.102 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:37:09.104+0000 7f316bfe1700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f316411d920 0x7f3164117640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:37:09.102 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:37:09.104+0000 7f316bfe1700 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x7f3164113090 con 0x7f316410b270 2024-05-27T07:37:09.102 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:37:09.104+0000 7f316bfe1700 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f3164112f10 con 0x7f316410a920 2024-05-27T07:37:09.102 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:37:09.104+0000 7f316bfe1700 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f3164113210 con 0x7f316411d920 2024-05-27T07:37:09.106 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:37:09.105+0000 7f316957c700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f316410b270 0x7f3164117100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:37:09.106 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:37:09.105+0000 7f316957c700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f316410b270 0x7f3164117100 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.52:55586/0 (socket says 172.21.15.52:55586) 2024-05-27T07:37:09.106 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:37:09.105+0000 7f316957c700 1 -- 172.21.15.52:0/2119181744 learned_addr learned my addr 172.21.15.52:0/2119181744 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-05-27T07:37:09.106 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:37:09.106+0000 7f3169d7d700 1 --2- 172.21.15.52:0/2119181744 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f316410a920 0x7f316411cf90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:37:09.106 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:37:09.106+0000 7f316a57e700 1 --2- 172.21.15.52:0/2119181744 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f316411d920 0x7f3164117640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:37:09.107 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:37:09.106+0000 7f316957c700 1 -- 172.21.15.52:0/2119181744 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f316411d920 msgr2=0x7f3164117640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:37:09.107 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:37:09.106+0000 7f316957c700 1 --2- 172.21.15.52:0/2119181744 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f316411d920 0x7f3164117640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:37:09.107 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:37:09.106+0000 7f316957c700 1 -- 172.21.15.52:0/2119181744 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f316410a920 msgr2=0x7f316411cf90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:37:09.107 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:37:09.106+0000 7f316957c700 1 --2- 172.21.15.52:0/2119181744 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f316410a920 0x7f316411cf90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:37:09.107 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:37:09.106+0000 7f316957c700 1 -- 172.21.15.52:0/2119181744 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3164117c00 con 0x7f316410b270 2024-05-27T07:37:09.107 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:37:09.108+0000 7f316957c700 1 --2- 172.21.15.52:0/2119181744 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f316410b270 0x7f3164117100 secure :-1 s=READY pgs=771 cs=0 l=1 rev1=1 crypto rx=0x7f315400e9f0 tx=0x7f315400edb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:37:09.109 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:37:09.108+0000 7f315affd700 1 -- 172.21.15.52:0/2119181744 <== mon.0 v2:172.21.15.52:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f315400cbd0 con 0x7f316410b270 2024-05-27T07:37:09.110 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:37:09.109+0000 7f316bfe1700 1 -- 172.21.15.52:0/2119181744 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3164117ef0 con 0x7f316410b270 2024-05-27T07:37:09.110 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:37:09.109+0000 7f316bfe1700 1 -- 172.21.15.52:0/2119181744 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f31641beec0 con 0x7f316410b270 2024-05-27T07:37:09.110 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:37:09.109+0000 7f315affd700 1 -- 172.21.15.52:0/2119181744 <== mon.0 v2:172.21.15.52:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f315400cd70 con 0x7f316410b270 2024-05-27T07:37:09.110 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:37:09.109+0000 7f315affd700 1 -- 172.21.15.52:0/2119181744 <== mon.0 v2:172.21.15.52:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f31540056b0 con 0x7f316410b270 2024-05-27T07:37:09.110 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:37:09.112+0000 7f316bfe1700 1 -- 172.21.15.52:0/2119181744 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3148005330 con 0x7f316410b270 2024-05-27T07:37:09.114 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:37:09.115+0000 7f315affd700 1 -- 172.21.15.52:0/2119181744 <== mon.0 v2:172.21.15.52:3300/0 4 ==== mgrmap(e 26) ==== 149783+0+0 (secure 0 0 0) 0x7f3154004750 con 0x7f316410b270 2024-05-27T07:37:09.115 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:37:09.116+0000 7f315affd700 1 --2- 172.21.15.52:0/2119181744 >> [v2:172.21.15.52:6800/1822318384,v1:172.21.15.52:6801/1822318384] conn(0x7f31500b2c00 0x7f31500b5110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:37:09.116 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:37:09.116+0000 7f315affd700 1 -- 172.21.15.52:0/2119181744 <== mon.0 v2:172.21.15.52:3300/0 5 ==== osd_map(130..130 src has 1..130) ==== 9511+0+0 (secure 0 0 0) 0x7f31540e2ac0 con 0x7f316410b270 2024-05-27T07:37:09.116 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:37:09.118+0000 7f3169d7d700 1 --2- 172.21.15.52:0/2119181744 >> [v2:172.21.15.52:6800/1822318384,v1:172.21.15.52:6801/1822318384] conn(0x7f31500b2c00 0x7f31500b5110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:37:09.123 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:37:09.122+0000 7f3169d7d700 1 --2- 172.21.15.52:0/2119181744 >> [v2:172.21.15.52:6800/1822318384,v1:172.21.15.52:6801/1822318384] conn(0x7f31500b2c00 0x7f31500b5110 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7f3160009bf0 tx=0x7f3160009260 comp rx=0 tx=0).ready entity=mgr.15333 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:37:09.128 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:37:09.126+0000 7f315affd700 1 -- 172.21.15.52:0/2119181744 <== mon.0 v2:172.21.15.52:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191963 (secure 0 0 0) 0x7f31540aa060 con 0x7f316410b270 2024-05-27T07:37:09.624 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:37:09.627+0000 7f316bfe1700 1 -- 172.21.15.52:0/2119181744 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f31480058f0 con 0x7f316410b270 2024-05-27T07:37:09.626 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:37:09.628+0000 7f315affd700 1 -- 172.21.15.52:0/2119181744 <== mon.0 v2:172.21.15.52:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 108 v108) ==== 95+0+6124 (secure 0 0 0) 0x7f31540a9960 con 0x7f316410b270 2024-05-27T07:37:09.627 INFO:teuthology.orchestra.run.smithi052.stdout: 2024-05-27T07:37:09.627 INFO:teuthology.orchestra.run.smithi052.stdout:{"epoch":108,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":108,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-05-27T07:21:50.521783+0000","modified":"2024-05-27T07:35:15.151893+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24527,"mds_1":24497,"mds_2":24475,"mds_3":24473,"mds_4":24466},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24466":{"gid":24466,"name":"c","rank":4,"incarnation":26,"state":"up:active","state_seq":25,"addr":"172.21.15.173:6837/1396363827","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6836","nonce":1396363827},{"type":"v1","addr":"172.21.15.173:6837","nonce":1396363827}]},"join_fscid":-1,"export_targets":[0],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24473":{"gid":24473,"name":"e","rank":3,"incarnation":24,"state":"up:active","state_seq":26,"addr":"172.21.15.105:6835/3298498384","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6834","nonce":3298498384},{"type":"v1","addr":"172.21.15.105:6835","nonce":3298498384}]},"join_fscid":-1,"export_targets":[0],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24475":{"gid":24475,"name":"i","rank":2,"incarnation":22,"state":"up:active","state_seq":23,"addr":"172.21.15.173:6839/564778293","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6838","nonce":564778293},{"type":"v1","addr":"172.21.15.173:6839","nonce":564778293}]},"join_fscid":-1,"export_targets":[0],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24497":{"gid":24497,"name":"h","rank":1,"incarnation":20,"state":"up:active","state_seq":24,"addr":"172.21.15.105:6837/191393149","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6836","nonce":191393149},{"type":"v1","addr":"172.21.15.105:6837","nonce":191393149}]},"join_fscid":-1,"export_targets":[0],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24527":{"gid":24527,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":5,"addr":"172.21.15.105:6839/500348151","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6838","nonce":500348151},{"type":"v1","addr":"172.21.15.105:6839","nonce":500348151}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24527,"qdb_cluster":[24497,24527,24475,24473,24466]},"id":1}]} 2024-05-27T07:37:09.632 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:37:09.634+0000 7f316bfe1700 1 -- 172.21.15.52:0/2119181744 >> [v2:172.21.15.52:6800/1822318384,v1:172.21.15.52:6801/1822318384] conn(0x7f31500b2c00 msgr2=0x7f31500b5110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:37:09.632 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:37:09.634+0000 7f316bfe1700 1 --2- 172.21.15.52:0/2119181744 >> [v2:172.21.15.52:6800/1822318384,v1:172.21.15.52:6801/1822318384] conn(0x7f31500b2c00 0x7f31500b5110 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7f3160009bf0 tx=0x7f3160009260 comp rx=0 tx=0).stop 2024-05-27T07:37:09.632 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:37:09.634+0000 7f316bfe1700 1 -- 172.21.15.52:0/2119181744 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f316410b270 msgr2=0x7f3164117100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:37:09.632 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:37:09.634+0000 7f316bfe1700 1 --2- 172.21.15.52:0/2119181744 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f316410b270 0x7f3164117100 secure :-1 s=READY pgs=771 cs=0 l=1 rev1=1 crypto rx=0x7f315400e9f0 tx=0x7f315400edb0 comp rx=0 tx=0).stop 2024-05-27T07:37:09.632 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:37:09.635+0000 7f316bfe1700 1 -- 172.21.15.52:0/2119181744 shutdown_connections 2024-05-27T07:37:09.633 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:37:09.635+0000 7f316bfe1700 1 --2- 172.21.15.52:0/2119181744 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f316411d920 0x7f3164117640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:37:09.633 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:37:09.635+0000 7f316bfe1700 1 --2- 172.21.15.52:0/2119181744 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f316410a920 0x7f316411cf90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:37:09.633 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:37:09.635+0000 7f316bfe1700 1 --2- 172.21.15.52:0/2119181744 >> [v2:172.21.15.52:6800/1822318384,v1:172.21.15.52:6801/1822318384] conn(0x7f31500b2c00 0x7f31500b5110 unknown :-1 s=CLOSED pgs=208 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:37:09.633 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:37:09.635+0000 7f316bfe1700 1 --2- 172.21.15.52:0/2119181744 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f316410b270 0x7f3164117100 unknown :-1 s=CLOSED pgs=771 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:37:09.633 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:37:09.635+0000 7f316bfe1700 1 -- 172.21.15.52:0/2119181744 >> 172.21.15.52:0/2119181744 conn(0x7f316406f4c0 msgr2=0x7f316411dfb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:37:09.633 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:37:09.635+0000 7f316bfe1700 1 -- 172.21.15.52:0/2119181744 shutdown_connections 2024-05-27T07:37:09.633 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:37:09.635+0000 7f316bfe1700 1 -- 172.21.15.52:0/2119181744 wait complete. 2024-05-27T07:37:09.634 INFO:teuthology.orchestra.run.smithi052.stderr:dumped fsmap epoch 108 2024-05-27T07:37:09.858 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:37:09 smithi173 ceph-mon[37000]: pgmap v379: 129 pgs: 72 active+clean+snaptrim_wait, 24 active+clean+snaptrim, 33 active+clean; 617 MiB data, 7.5 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 36 KiB/s wr, 24 op/s 2024-05-27T07:37:09.946 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:37:09 smithi105 ceph-mon[35977]: pgmap v379: 129 pgs: 72 active+clean+snaptrim_wait, 24 active+clean+snaptrim, 33 active+clean; 617 MiB data, 7.5 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 36 KiB/s wr, 24 op/s 2024-05-27T07:37:09.985 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:37:09 smithi052 ceph-mon[31955]: pgmap v379: 129 pgs: 72 active+clean+snaptrim_wait, 24 active+clean+snaptrim, 33 active+clean; 617 MiB data, 7.5 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 36 KiB/s wr, 24 op/s 2024-05-27T07:37:10.858 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:37:10 smithi173 ceph-mon[37000]: from='client.? 172.21.15.52:0/2119181744' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-27T07:37:10.946 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:37:10 smithi105 ceph-mon[35977]: from='client.? 172.21.15.52:0/2119181744' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-27T07:37:10.985 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:37:10 smithi052 ceph-mon[31955]: from='client.? 172.21.15.52:0/2119181744' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-27T07:37:11.039 DEBUG:teuthology.orchestra.run.smithi052:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:38f9d900c83c63799fbdbe61acc9a11b0d3554a6 shell --fsid 73df2636-1bf8-11ef-bc9b-c7b262605968 -- ceph osd dump --format=json 2024-05-27T07:37:11.500 INFO:teuthology.orchestra.run.smithi052.stderr:Inferring config /var/lib/ceph/73df2636-1bf8-11ef-bc9b-c7b262605968/mon.a/config 2024-05-27T07:37:11.788 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:37:11 smithi052 ceph-mon[31955]: pgmap v380: 129 pgs: 72 active+clean+snaptrim_wait, 24 active+clean+snaptrim, 33 active+clean; 617 MiB data, 7.5 GiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 32 KiB/s wr, 21 op/s 2024-05-27T07:37:11.858 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:37:11 smithi173 ceph-mon[37000]: pgmap v380: 129 pgs: 72 active+clean+snaptrim_wait, 24 active+clean+snaptrim, 33 active+clean; 617 MiB data, 7.5 GiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 32 KiB/s wr, 21 op/s 2024-05-27T07:37:11.946 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:37:11 smithi105 ceph-mon[35977]: pgmap v380: 129 pgs: 72 active+clean+snaptrim_wait, 24 active+clean+snaptrim, 33 active+clean; 617 MiB data, 7.5 GiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 32 KiB/s wr, 21 op/s 2024-05-27T07:37:13.456 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:37:13.458+0000 7ff742766700 1 -- 172.21.15.52:0/1650927271 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff73c10a920 msgr2=0x7ff73c10ad30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:37:13.456 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:37:13.458+0000 7ff742766700 1 --2- 172.21.15.52:0/1650927271 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff73c10a920 0x7ff73c10ad30 secure :-1 s=READY pgs=772 cs=0 l=1 rev1=1 crypto rx=0x7ff73001a9a0 tx=0x7ff73001acb0 comp rx=0 tx=0).stop 2024-05-27T07:37:13.462 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:37:13.461+0000 7ff742766700 1 -- 172.21.15.52:0/1650927271 shutdown_connections 2024-05-27T07:37:13.462 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:37:13.461+0000 7ff742766700 1 --2- 172.21.15.52:0/1650927271 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7ff73c11d920 0x7ff73c11fd40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:37:13.462 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:37:13.461+0000 7ff742766700 1 --2- 172.21.15.52:0/1650927271 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff73c10a920 0x7ff73c10ad30 unknown :-1 s=CLOSED pgs=772 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:37:13.462 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:37:13.461+0000 7ff742766700 1 --2- 172.21.15.52:0/1650927271 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7ff73c10b270 0x7ff73c11d3e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:37:13.462 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:37:13.461+0000 7ff742766700 1 -- 172.21.15.52:0/1650927271 >> 172.21.15.52:0/1650927271 conn(0x7ff73c06f4c0 msgr2=0x7ff73c06f900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:37:13.470 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:37:13.468+0000 7ff742766700 1 -- 172.21.15.52:0/1650927271 shutdown_connections 2024-05-27T07:37:13.470 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:37:13.469+0000 7ff742766700 1 -- 172.21.15.52:0/1650927271 wait complete. 2024-05-27T07:37:13.470 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:37:13.470+0000 7ff742766700 1 Processor -- start 2024-05-27T07:37:13.470 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:37:13.470+0000 7ff742766700 1 -- start start 2024-05-27T07:37:13.470 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:37:13.471+0000 7ff742766700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7ff73c10a920 0x7ff73c11d220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:37:13.470 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:37:13.471+0000 7ff742766700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7ff73c10b270 0x7ff73c119360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:37:13.470 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:37:13.471+0000 7ff742766700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff73c11d920 0x7ff73c1198a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:37:13.470 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:37:13.471+0000 7ff742766700 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x7ff73c1153e0 con 0x7ff73c11d920 2024-05-27T07:37:13.470 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:37:13.471+0000 7ff742766700 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7ff73c115260 con 0x7ff73c10b270 2024-05-27T07:37:13.470 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:37:13.471+0000 7ff742766700 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7ff73c115560 con 0x7ff73c10a920 2024-05-27T07:37:13.470 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:37:13.471+0000 7ff73b7fe700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7ff73c10b270 0x7ff73c119360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:37:13.471 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:37:13.471+0000 7ff73b7fe700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7ff73c10b270 0x7ff73c119360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.52:55182/0 (socket says 172.21.15.52:55182) 2024-05-27T07:37:13.471 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:37:13.471+0000 7ff73b7fe700 1 -- 172.21.15.52:0/505270008 learned_addr learned my addr 172.21.15.52:0/505270008 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-05-27T07:37:13.473 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:37:13.475+0000 7ff73bfff700 1 --2- 172.21.15.52:0/505270008 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7ff73c10a920 0x7ff73c11d220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:37:13.474 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:37:13.475+0000 7ff740d03700 1 --2- 172.21.15.52:0/505270008 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff73c11d920 0x7ff73c1198a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:37:13.474 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:37:13.475+0000 7ff73b7fe700 1 -- 172.21.15.52:0/505270008 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7ff73c10a920 msgr2=0x7ff73c11d220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:37:13.474 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:37:13.475+0000 7ff73b7fe700 1 --2- 172.21.15.52:0/505270008 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7ff73c10a920 0x7ff73c11d220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:37:13.474 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:37:13.475+0000 7ff73b7fe700 1 -- 172.21.15.52:0/505270008 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff73c11d920 msgr2=0x7ff73c1198a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:37:13.474 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:37:13.475+0000 7ff73b7fe700 1 --2- 172.21.15.52:0/505270008 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff73c11d920 0x7ff73c1198a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:37:13.474 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:37:13.475+0000 7ff73b7fe700 1 -- 172.21.15.52:0/505270008 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff73c11a080 con 0x7ff73c10b270 2024-05-27T07:37:13.476 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:37:13.479+0000 7ff73b7fe700 1 --2- 172.21.15.52:0/505270008 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7ff73c10b270 0x7ff73c119360 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 crypto rx=0x7ff72c00b530 tx=0x7ff72c00b8f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:37:13.480 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:37:13.481+0000 7ff7397fa700 1 -- 172.21.15.52:0/505270008 <== mon.2 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff72c015020 con 0x7ff73c10b270 2024-05-27T07:37:13.480 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:37:13.481+0000 7ff7397fa700 1 -- 172.21.15.52:0/505270008 <== mon.2 v2:172.21.15.105:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ff72c011ce0 con 0x7ff73c10b270 2024-05-27T07:37:13.480 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:37:13.481+0000 7ff7397fa700 1 -- 172.21.15.52:0/505270008 <== mon.2 v2:172.21.15.105:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff72c010d20 con 0x7ff73c10b270 2024-05-27T07:37:13.480 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:37:13.483+0000 7ff742766700 1 -- 172.21.15.52:0/505270008 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff73c07ba10 con 0x7ff73c10b270 2024-05-27T07:37:13.480 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:37:13.483+0000 7ff742766700 1 -- 172.21.15.52:0/505270008 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff73c07bf00 con 0x7ff73c10b270 2024-05-27T07:37:13.488 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:37:13.487+0000 7ff742766700 1 -- 172.21.15.52:0/505270008 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff728005360 con 0x7ff73c10b270 2024-05-27T07:37:13.488 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:37:13.491+0000 7ff7397fa700 1 -- 172.21.15.52:0/505270008 <== mon.2 v2:172.21.15.105:3300/0 4 ==== mgrmap(e 26) ==== 149783+0+0 (secure 0 0 0) 0x7ff72c018020 con 0x7ff73c10b270 2024-05-27T07:37:13.490 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:37:13.492+0000 7ff7397fa700 1 --2- 172.21.15.52:0/505270008 >> [v2:172.21.15.52:6800/1822318384,v1:172.21.15.52:6801/1822318384] conn(0x7ff7240b2bc0 0x7ff7240b50d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:37:13.490 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:37:13.492+0000 7ff7397fa700 1 -- 172.21.15.52:0/505270008 <== mon.2 v2:172.21.15.105:3300/0 5 ==== osd_map(130..130 src has 1..130) ==== 9511+0+0 (secure 0 0 0) 0x7ff72c0e1fc0 con 0x7ff73c10b270 2024-05-27T07:37:13.493 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:37:13.496+0000 7ff73bfff700 1 --2- 172.21.15.52:0/505270008 >> [v2:172.21.15.52:6800/1822318384,v1:172.21.15.52:6801/1822318384] conn(0x7ff7240b2bc0 0x7ff7240b50d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:37:13.497 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:37:13.500+0000 7ff7397fa700 1 -- 172.21.15.52:0/505270008 <== mon.2 v2:172.21.15.105:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191963 (secure 0 0 0) 0x7ff72c01f020 con 0x7ff73c10b270 2024-05-27T07:37:13.502 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:37:13.502+0000 7ff73bfff700 1 --2- 172.21.15.52:0/505270008 >> [v2:172.21.15.52:6800/1822318384,v1:172.21.15.52:6801/1822318384] conn(0x7ff7240b2bc0 0x7ff7240b50d0 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7ff730000c00 tx=0x7ff730005960 comp rx=0 tx=0).ready entity=mgr.15333 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:37:13.915 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:37:13.917+0000 7ff742766700 1 -- 172.21.15.52:0/505270008 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7ff728005920 con 0x7ff73c10b270 2024-05-27T07:37:13.918 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:37:13.921+0000 7ff7397fa700 1 -- 172.21.15.52:0/505270008 <== mon.2 v2:172.21.15.105:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v130) ==== 74+0+25331 (secure 0 0 0) 0x7ff72c0a9670 con 0x7ff73c10b270 2024-05-27T07:37:13.919 INFO:teuthology.orchestra.run.smithi052.stdout: 2024-05-27T07:37:13.919 INFO:teuthology.orchestra.run.smithi052.stdout:{"epoch":130,"fsid":"73df2636-1bf8-11ef-bc9b-c7b262605968","created":"2024-05-27T07:13:34.738032+0000","modified":"2024-05-27T07:37:05.128554+0000","last_up_change":"2024-05-27T07:20:45.777597+0000","last_in_change":"2024-05-27T07:20:25.314823+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-05-27T07:16:57.644034+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-05-27T07:21:43.917748+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"79","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-05-27T07:21:47.341876+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"129","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":17,"snap_epoch":129,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-05-27T07:21:57.707611+0000","flags":8197,"flags_names":"hashpspool,ec_overwrites,selfmanaged_snaps","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"129","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":17,"snap_epoch":129,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"347b8a89-9dbe-4848-83b0-fd0e9374f33d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":82,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.52:6802","nonce":1626508899},{"type":"v1","addr":"172.21.15.52:6803","nonce":1626508899}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.52:6804","nonce":1626508899},{"type":"v1","addr":"172.21.15.52:6805","nonce":1626508899}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.52:6808","nonce":1626508899},{"type":"v1","addr":"172.21.15.52:6809","nonce":1626508899}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.52:6806","nonce":1626508899},{"type":"v1","addr":"172.21.15.52:6807","nonce":1626508899}]},"public_addr":"172.21.15.52:6803/1626508899","cluster_addr":"172.21.15.52:6805/1626508899","heartbeat_back_addr":"172.21.15.52:6809/1626508899","heartbeat_front_addr":"172.21.15.52:6807/1626508899","state":["exists","up"]},{"osd":1,"uuid":"7a3623d4-e087-40cb-9336-d4a5c52203d4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6800","nonce":2900093195},{"type":"v1","addr":"172.21.15.105:6801","nonce":2900093195}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6802","nonce":2900093195},{"type":"v1","addr":"172.21.15.105:6803","nonce":2900093195}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6806","nonce":2900093195},{"type":"v1","addr":"172.21.15.105:6807","nonce":2900093195}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6804","nonce":2900093195},{"type":"v1","addr":"172.21.15.105:6805","nonce":2900093195}]},"public_addr":"172.21.15.105:6801/2900093195","cluster_addr":"172.21.15.105:6803/2900093195","heartbeat_back_addr":"172.21.15.105:6807/2900093195","heartbeat_front_addr":"172.21.15.105:6805/2900093195","state":["exists","up"]},{"osd":2,"uuid":"41a9df5b-882a-47da-bb26-7f4d26ccf6c1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6800","nonce":2251189314},{"type":"v1","addr":"172.21.15.173:6801","nonce":2251189314}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6802","nonce":2251189314},{"type":"v1","addr":"172.21.15.173:6803","nonce":2251189314}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6806","nonce":2251189314},{"type":"v1","addr":"172.21.15.173:6807","nonce":2251189314}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6804","nonce":2251189314},{"type":"v1","addr":"172.21.15.173:6805","nonce":2251189314}]},"public_addr":"172.21.15.173:6801/2251189314","cluster_addr":"172.21.15.173:6803/2251189314","heartbeat_back_addr":"172.21.15.173:6807/2251189314","heartbeat_front_addr":"172.21.15.173:6805/2251189314","state":["exists","up"]},{"osd":3,"uuid":"f1245439-42a5-46c5-a998-e0a78f9df3fb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":26,"up_thru":82,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.52:6810","nonce":57920794},{"type":"v1","addr":"172.21.15.52:6811","nonce":57920794}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.52:6812","nonce":57920794},{"type":"v1","addr":"172.21.15.52:6813","nonce":57920794}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.52:6816","nonce":57920794},{"type":"v1","addr":"172.21.15.52:6817","nonce":57920794}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.52:6814","nonce":57920794},{"type":"v1","addr":"172.21.15.52:6815","nonce":57920794}]},"public_addr":"172.21.15.52:6811/57920794","cluster_addr":"172.21.15.52:6813/57920794","heartbeat_back_addr":"172.21.15.52:6817/57920794","heartbeat_front_addr":"172.21.15.52:6815/57920794","state":["exists","up"]},{"osd":4,"uuid":"2592d1cc-1726-4359-83d8-a2e6314b54ec","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":32,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6808","nonce":3441134818},{"type":"v1","addr":"172.21.15.105:6809","nonce":3441134818}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6810","nonce":3441134818},{"type":"v1","addr":"172.21.15.105:6811","nonce":3441134818}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6814","nonce":3441134818},{"type":"v1","addr":"172.21.15.105:6815","nonce":3441134818}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6812","nonce":3441134818},{"type":"v1","addr":"172.21.15.105:6813","nonce":3441134818}]},"public_addr":"172.21.15.105:6809/3441134818","cluster_addr":"172.21.15.105:6811/3441134818","heartbeat_back_addr":"172.21.15.105:6815/3441134818","heartbeat_front_addr":"172.21.15.105:6813/3441134818","state":["exists","up"]},{"osd":5,"uuid":"27e98ee2-4a92-4972-b2a3-959d0ed90981","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":37,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6808","nonce":2989239729},{"type":"v1","addr":"172.21.15.173:6809","nonce":2989239729}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6810","nonce":2989239729},{"type":"v1","addr":"172.21.15.173:6811","nonce":2989239729}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6814","nonce":2989239729},{"type":"v1","addr":"172.21.15.173:6815","nonce":2989239729}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6812","nonce":2989239729},{"type":"v1","addr":"172.21.15.173:6813","nonce":2989239729}]},"public_addr":"172.21.15.173:6809/2989239729","cluster_addr":"172.21.15.173:6811/2989239729","heartbeat_back_addr":"172.21.15.173:6815/2989239729","heartbeat_front_addr":"172.21.15.173:6813/2989239729","state":["exists","up"]},{"osd":6,"uuid":"2b0ec0cb-f438-4b2c-8642-0778ed374e9a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":43,"up_thru":82,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.52:6818","nonce":4244423065},{"type":"v1","addr":"172.21.15.52:6819","nonce":4244423065}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.52:6820","nonce":4244423065},{"type":"v1","addr":"172.21.15.52:6821","nonce":4244423065}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.52:6824","nonce":4244423065},{"type":"v1","addr":"172.21.15.52:6825","nonce":4244423065}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.52:6822","nonce":4244423065},{"type":"v1","addr":"172.21.15.52:6823","nonce":4244423065}]},"public_addr":"172.21.15.52:6819/4244423065","cluster_addr":"172.21.15.52:6821/4244423065","heartbeat_back_addr":"172.21.15.52:6825/4244423065","heartbeat_front_addr":"172.21.15.52:6823/4244423065","state":["exists","up"]},{"osd":7,"uuid":"cb2a7a6b-8015-43ea-865a-be144e45c51a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":48,"up_thru":82,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6816","nonce":2063611552},{"type":"v1","addr":"172.21.15.105:6817","nonce":2063611552}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6818","nonce":2063611552},{"type":"v1","addr":"172.21.15.105:6819","nonce":2063611552}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6822","nonce":2063611552},{"type":"v1","addr":"172.21.15.105:6823","nonce":2063611552}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6820","nonce":2063611552},{"type":"v1","addr":"172.21.15.105:6821","nonce":2063611552}]},"public_addr":"172.21.15.105:6817/2063611552","cluster_addr":"172.21.15.105:6819/2063611552","heartbeat_back_addr":"172.21.15.105:6823/2063611552","heartbeat_front_addr":"172.21.15.105:6821/2063611552","state":["exists","up"]},{"osd":8,"uuid":"c2fc565d-a0b9-4396-a209-bd884cddcfe3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":54,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6816","nonce":4131399852},{"type":"v1","addr":"172.21.15.173:6817","nonce":4131399852}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6818","nonce":4131399852},{"type":"v1","addr":"172.21.15.173:6819","nonce":4131399852}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6822","nonce":4131399852},{"type":"v1","addr":"172.21.15.173:6823","nonce":4131399852}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6820","nonce":4131399852},{"type":"v1","addr":"172.21.15.173:6821","nonce":4131399852}]},"public_addr":"172.21.15.173:6817/4131399852","cluster_addr":"172.21.15.173:6819/4131399852","heartbeat_back_addr":"172.21.15.173:6823/4131399852","heartbeat_front_addr":"172.21.15.173:6821/4131399852","state":["exists","up"]},{"osd":9,"uuid":"78fa7f17-1dad-4262-be40-31fa82e75520","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":60,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.52:6826","nonce":479119778},{"type":"v1","addr":"172.21.15.52:6827","nonce":479119778}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.52:6828","nonce":479119778},{"type":"v1","addr":"172.21.15.52:6829","nonce":479119778}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.52:6832","nonce":479119778},{"type":"v1","addr":"172.21.15.52:6833","nonce":479119778}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.52:6830","nonce":479119778},{"type":"v1","addr":"172.21.15.52:6831","nonce":479119778}]},"public_addr":"172.21.15.52:6827/479119778","cluster_addr":"172.21.15.52:6829/479119778","heartbeat_back_addr":"172.21.15.52:6833/479119778","heartbeat_front_addr":"172.21.15.52:6831/479119778","state":["exists","up"]},{"osd":10,"uuid":"e6f8e753-e15c-45b5-8ee9-d50f384559f9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":65,"up_thru":76,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6824","nonce":4032386628},{"type":"v1","addr":"172.21.15.105:6825","nonce":4032386628}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6826","nonce":4032386628},{"type":"v1","addr":"172.21.15.105:6827","nonce":4032386628}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6830","nonce":4032386628},{"type":"v1","addr":"172.21.15.105:6831","nonce":4032386628}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6828","nonce":4032386628},{"type":"v1","addr":"172.21.15.105:6829","nonce":4032386628}]},"public_addr":"172.21.15.105:6825/4032386628","cluster_addr":"172.21.15.105:6827/4032386628","heartbeat_back_addr":"172.21.15.105:6831/4032386628","heartbeat_front_addr":"172.21.15.105:6829/4032386628","state":["exists","up"]},{"osd":11,"uuid":"164572bf-67ed-4e8e-b32d-d7a2e8da607b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":71,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6824","nonce":1816308401},{"type":"v1","addr":"172.21.15.173:6825","nonce":1816308401}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6826","nonce":1816308401},{"type":"v1","addr":"172.21.15.173:6827","nonce":1816308401}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6830","nonce":1816308401},{"type":"v1","addr":"172.21.15.173:6831","nonce":1816308401}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6828","nonce":1816308401},{"type":"v1","addr":"172.21.15.173:6829","nonce":1816308401}]},"public_addr":"172.21.15.173:6825/1816308401","cluster_addr":"172.21.15.173:6827/1816308401","heartbeat_back_addr":"172.21.15.173:6831/1816308401","heartbeat_front_addr":"172.21.15.173:6829/1816308401","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-27T07:16:06.279220+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-27T07:16:29.668167+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-27T07:16:53.903808+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-27T07:17:18.787372+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-27T07:17:43.480784+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-27T07:18:08.816030+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-27T07:18:34.076148+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-27T07:19:00.283468+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-27T07:19:25.403023+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-27T07:19:52.139173+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-27T07:20:17.530304+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-27T07:20:43.665632+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.8","mappings":[{"from":2,"to":11}]},{"pgid":"2.f","mappings":[{"from":8,"to":11}]},{"pgid":"2.10","mappings":[{"from":2,"to":4}]},{"pgid":"3.0","mappings":[{"from":5,"to":9}]},{"pgid":"3.1b","mappings":[{"from":4,"to":8}]},{"pgid":"3.2c","mappings":[{"from":4,"to":9}]},{"pgid":"3.3c","mappings":[{"from":4,"to":9}]},{"pgid":"4.e","mappings":[{"from":9,"to":11}]},{"pgid":"4.12","mappings":[{"from":5,"to":10}]},{"pgid":"4.19","mappings":[{"from":9,"to":8}]},{"pgid":"4.1f","mappings":[{"from":9,"to":8}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.52:0/3430810618":"2024-05-28T07:25:56.925056+0000","172.21.15.52:0/2723769517":"2024-05-28T07:25:56.925056+0000","172.21.15.52:0/262308791":"2024-05-28T07:25:56.925056+0000","172.21.15.52:6801/1631683163":"2024-05-28T07:14:29.544633+0000","172.21.15.52:6801/3513075805":"2024-05-28T07:25:56.925056+0000","172.21.15.52:0/2776144659":"2024-05-28T07:14:29.544633+0000","172.21.15.52:0/3577886285":"2024-05-28T07:25:56.925056+0000","172.21.15.52:6800/1631683163":"2024-05-28T07:14:29.544633+0000","172.21.15.52:0/3675529097":"2024-05-28T07:14:01.960550+0000","172.21.15.52:0/727808107":"2024-05-28T07:14:01.960550+0000","172.21.15.52:0/3839436833":"2024-05-28T07:14:29.544633+0000","172.21.15.52:6800/587662512":"2024-05-28T07:14:01.960550+0000","172.21.15.52:6800/3513075805":"2024-05-28T07:25:56.925056+0000","172.21.15.52:6801/587662512":"2024-05-28T07:14:01.960550+0000","172.21.15.52:0/853867465":"2024-05-28T07:14:01.960550+0000","172.21.15.52:0/3480918279":"2024-05-28T07:14:29.544633+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":6,"length":1},{"begin":17,"length":1}]},{"pool":4,"snaps":[{"begin":6,"length":1},{"begin":17,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[{"pool":4,"snaps":[{"begin":5,"length":1},{"begin":15,"length":1}]}],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-05-27T07:37:13.927 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:37:13.929+0000 7ff722ffd700 1 -- 172.21.15.52:0/505270008 >> [v2:172.21.15.52:6800/1822318384,v1:172.21.15.52:6801/1822318384] conn(0x7ff7240b2bc0 msgr2=0x7ff7240b50d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:37:13.927 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:37:13.929+0000 7ff722ffd700 1 --2- 172.21.15.52:0/505270008 >> [v2:172.21.15.52:6800/1822318384,v1:172.21.15.52:6801/1822318384] conn(0x7ff7240b2bc0 0x7ff7240b50d0 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7ff730000c00 tx=0x7ff730005960 comp rx=0 tx=0).stop 2024-05-27T07:37:13.927 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:37:13.929+0000 7ff722ffd700 1 -- 172.21.15.52:0/505270008 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7ff73c10b270 msgr2=0x7ff73c119360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:37:13.927 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:37:13.929+0000 7ff722ffd700 1 --2- 172.21.15.52:0/505270008 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7ff73c10b270 0x7ff73c119360 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 crypto rx=0x7ff72c00b530 tx=0x7ff72c00b8f0 comp rx=0 tx=0).stop 2024-05-27T07:37:13.928 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:37:13.930+0000 7ff722ffd700 1 -- 172.21.15.52:0/505270008 shutdown_connections 2024-05-27T07:37:13.928 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:37:13.930+0000 7ff722ffd700 1 --2- 172.21.15.52:0/505270008 >> [v2:172.21.15.52:6800/1822318384,v1:172.21.15.52:6801/1822318384] conn(0x7ff7240b2bc0 0x7ff7240b50d0 unknown :-1 s=CLOSED pgs=209 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:37:13.928 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:37:13.930+0000 7ff722ffd700 1 --2- 172.21.15.52:0/505270008 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff73c11d920 0x7ff73c1198a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:37:13.928 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:37:13.930+0000 7ff722ffd700 1 --2- 172.21.15.52:0/505270008 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7ff73c10a920 0x7ff73c11d220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:37:13.928 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:37:13.930+0000 7ff722ffd700 1 --2- 172.21.15.52:0/505270008 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7ff73c10b270 0x7ff73c119360 unknown :-1 s=CLOSED pgs=284 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:37:13.928 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:37:13.930+0000 7ff722ffd700 1 -- 172.21.15.52:0/505270008 >> 172.21.15.52:0/505270008 conn(0x7ff73c06f4c0 msgr2=0x7ff73c11df10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:37:13.932 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:37:13.935+0000 7ff722ffd700 1 -- 172.21.15.52:0/505270008 shutdown_connections 2024-05-27T07:37:13.933 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:37:13.936+0000 7ff722ffd700 1 -- 172.21.15.52:0/505270008 wait complete. 2024-05-27T07:37:14.612 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:37:14 smithi052 ceph-mon[31955]: pgmap v381: 129 pgs: 72 active+clean+snaptrim_wait, 24 active+clean+snaptrim, 33 active+clean; 616 MiB data, 7.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s wr, 1 op/s 2024-05-27T07:37:14.612 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:37:14 smithi052 ceph-mon[31955]: from='client.? 172.21.15.52:0/505270008' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-05-27T07:37:14.664 DEBUG:teuthology.orchestra.run.smithi052:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:38f9d900c83c63799fbdbe61acc9a11b0d3554a6 shell --fsid 73df2636-1bf8-11ef-bc9b-c7b262605968 -- ceph fs fail cephfs 2024-05-27T07:37:14.696 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:37:14 smithi105 ceph-mon[35977]: pgmap v381: 129 pgs: 72 active+clean+snaptrim_wait, 24 active+clean+snaptrim, 33 active+clean; 616 MiB data, 7.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s wr, 1 op/s 2024-05-27T07:37:14.696 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:37:14 smithi105 ceph-mon[35977]: from='client.? 172.21.15.52:0/505270008' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-05-27T07:37:14.860 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:37:14 smithi173 ceph-mon[37000]: pgmap v381: 129 pgs: 72 active+clean+snaptrim_wait, 24 active+clean+snaptrim, 33 active+clean; 616 MiB data, 7.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s wr, 1 op/s 2024-05-27T07:37:14.860 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:37:14 smithi173 ceph-mon[37000]: from='client.? 172.21.15.52:0/505270008' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-05-27T07:37:15.127 INFO:teuthology.orchestra.run.smithi052.stderr:Inferring config /var/lib/ceph/73df2636-1bf8-11ef-bc9b-c7b262605968/mon.a/config 2024-05-27T07:37:15.598 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:37:15 smithi052 ceph-mon[31955]: pgmap v382: 129 pgs: 70 active+clean+snaptrim_wait, 23 active+clean+snaptrim, 36 active+clean; 615 MiB data, 7.6 GiB used, 1.0 TiB / 1.0 TiB avail; 499 B/s wr, 0 op/s 2024-05-27T07:37:15.858 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:37:15 smithi173 ceph-mon[37000]: pgmap v382: 129 pgs: 70 active+clean+snaptrim_wait, 23 active+clean+snaptrim, 36 active+clean; 615 MiB data, 7.6 GiB used, 1.0 TiB / 1.0 TiB avail; 499 B/s wr, 0 op/s 2024-05-27T07:37:15.946 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:37:15 smithi105 ceph-mon[35977]: pgmap v382: 129 pgs: 70 active+clean+snaptrim_wait, 23 active+clean+snaptrim, 36 active+clean; 615 MiB data, 7.6 GiB used, 1.0 TiB / 1.0 TiB avail; 499 B/s wr, 0 op/s 2024-05-27T07:37:18.697 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:37:18 smithi105 ceph-mon[35977]: pgmap v383: 129 pgs: 70 active+clean+snaptrim_wait, 23 active+clean+snaptrim, 36 active+clean; 615 MiB data, 7.7 GiB used, 1.0 TiB / 1.0 TiB avail; 255 B/s wr, 0 op/s 2024-05-27T07:37:18.711 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:37:18 smithi052 ceph-mon[31955]: pgmap v383: 129 pgs: 70 active+clean+snaptrim_wait, 23 active+clean+snaptrim, 36 active+clean; 615 MiB data, 7.7 GiB used, 1.0 TiB / 1.0 TiB avail; 255 B/s wr, 0 op/s 2024-05-27T07:37:18.858 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:37:18 smithi173 ceph-mon[37000]: pgmap v383: 129 pgs: 70 active+clean+snaptrim_wait, 23 active+clean+snaptrim, 36 active+clean; 615 MiB data, 7.7 GiB used, 1.0 TiB / 1.0 TiB avail; 255 B/s wr, 0 op/s 2024-05-27T07:37:18.918 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:37:18.920+0000 7f34e359e700 1 -- 172.21.15.52:0/1372156272 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f34e4103980 msgr2=0x7f34e4107eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:37:18.918 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:37:18.920+0000 7f34e359e700 1 --2- 172.21.15.52:0/1372156272 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f34e4103980 0x7f34e4107eb0 secure :-1 s=READY pgs=285 cs=0 l=1 rev1=1 crypto rx=0x7f34d8009990 tx=0x7f34d8009ca0 comp rx=0 tx=0).stop 2024-05-27T07:37:18.920 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:37:18.923+0000 7f34e359e700 1 -- 172.21.15.52:0/1372156272 shutdown_connections 2024-05-27T07:37:18.920 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:37:18.923+0000 7f34e359e700 1 --2- 172.21.15.52:0/1372156272 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f34e4108480 0x7f34e4108950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:37:18.920 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:37:18.923+0000 7f34e359e700 1 --2- 172.21.15.52:0/1372156272 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f34e4103030 0x7f34e4103440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:37:18.921 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:37:18.923+0000 7f34e359e700 1 --2- 172.21.15.52:0/1372156272 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f34e4103980 0x7f34e4107eb0 unknown :-1 s=CLOSED pgs=285 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:37:18.921 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:37:18.923+0000 7f34e359e700 1 -- 172.21.15.52:0/1372156272 >> 172.21.15.52:0/1372156272 conn(0x7f34e4078aa0 msgr2=0x7f34e4078f00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:37:18.922 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:37:18.925+0000 7f34e359e700 1 -- 172.21.15.52:0/1372156272 shutdown_connections 2024-05-27T07:37:18.925 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:37:18.927+0000 7f34e359e700 1 -- 172.21.15.52:0/1372156272 wait complete. 2024-05-27T07:37:18.926 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:37:18.928+0000 7f34e359e700 1 Processor -- start 2024-05-27T07:37:18.926 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:37:18.928+0000 7f34e359e700 1 -- start start 2024-05-27T07:37:18.926 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:37:18.929+0000 7f34e359e700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f34e4103030 0x7f34e41176d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:37:18.926 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:37:18.929+0000 7f34e359e700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f34e4103980 0x7f34e4110850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:37:18.926 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:37:18.929+0000 7f34e359e700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f34e4108480 0x7f34e4110ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:37:18.927 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:37:18.930+0000 7f34e359e700 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x7f34e411a9c0 con 0x7f34e4103980 2024-05-27T07:37:18.928 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:37:18.930+0000 7f34e359e700 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f34e411a840 con 0x7f34e4103030 2024-05-27T07:37:18.928 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:37:18.930+0000 7f34e2d9d700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f34e4108480 0x7f34e4110ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:37:18.928 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:37:18.930+0000 7f34e2d9d700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f34e4108480 0x7f34e4110ef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.173:3300/0 says I am v2:172.21.15.52:48646/0 (socket says 172.21.15.52:48646) 2024-05-27T07:37:18.928 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:37:18.930+0000 7f34e2d9d700 1 -- 172.21.15.52:0/3211100101 learned_addr learned my addr 172.21.15.52:0/3211100101 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-05-27T07:37:18.928 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:37:18.930+0000 7f34e359e700 1 -- 172.21.15.52:0/3211100101 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f34e411ab40 con 0x7f34e4108480 2024-05-27T07:37:18.928 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:37:18.930+0000 7f34e259c700 1 --2- 172.21.15.52:0/3211100101 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f34e4103030 0x7f34e41176d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:37:18.928 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:37:18.931+0000 7f34e259c700 1 -- 172.21.15.52:0/3211100101 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f34e4108480 msgr2=0x7f34e4110ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:37:18.928 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:37:18.931+0000 7f34e259c700 1 --2- 172.21.15.52:0/3211100101 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f34e4108480 0x7f34e4110ef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:37:18.929 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:37:18.931+0000 7f34e259c700 1 -- 172.21.15.52:0/3211100101 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f34e4103980 msgr2=0x7f34e4110850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:37:18.929 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:37:18.931+0000 7f34e259c700 1 --2- 172.21.15.52:0/3211100101 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f34e4103980 0x7f34e4110850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:37:18.929 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:37:18.931+0000 7f34e259c700 1 -- 172.21.15.52:0/3211100101 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f34e4115200 con 0x7f34e4103030 2024-05-27T07:37:18.929 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:37:18.931+0000 7f34e259c700 1 --2- 172.21.15.52:0/3211100101 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f34e4103030 0x7f34e41176d0 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 crypto rx=0x7f34d400b450 tx=0x7f34d400b810 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:37:18.932 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:37:18.933+0000 7f34d37fe700 1 -- 172.21.15.52:0/3211100101 <== mon.2 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f34d4015020 con 0x7f34e4103030 2024-05-27T07:37:18.932 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:37:18.933+0000 7f34e359e700 1 -- 172.21.15.52:0/3211100101 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f34e41154f0 con 0x7f34e4103030 2024-05-27T07:37:18.932 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:37:18.933+0000 7f34e359e700 1 -- 172.21.15.52:0/3211100101 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f34e4115a30 con 0x7f34e4103030 2024-05-27T07:37:18.932 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:37:18.933+0000 7f34d37fe700 1 -- 172.21.15.52:0/3211100101 <== mon.2 v2:172.21.15.105:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f34d4011ce0 con 0x7f34e4103030 2024-05-27T07:37:18.932 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:37:18.933+0000 7f34d37fe700 1 -- 172.21.15.52:0/3211100101 <== mon.2 v2:172.21.15.105:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f34d4010dd0 con 0x7f34e4103030 2024-05-27T07:37:18.936 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:37:18.939+0000 7f34e359e700 1 -- 172.21.15.52:0/3211100101 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f34c4005330 con 0x7f34e4103030 2024-05-27T07:37:18.939 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:37:18.941+0000 7f34d37fe700 1 -- 172.21.15.52:0/3211100101 <== mon.2 v2:172.21.15.105:3300/0 4 ==== mgrmap(e 26) ==== 149783+0+0 (secure 0 0 0) 0x7f34d4018020 con 0x7f34e4103030 2024-05-27T07:37:18.940 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:37:18.943+0000 7f34d37fe700 1 --2- 172.21.15.52:0/3211100101 >> [v2:172.21.15.52:6800/1822318384,v1:172.21.15.52:6801/1822318384] conn(0x7f34cc0b2b80 0x7f34cc0b5090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:37:18.940 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:37:18.943+0000 7f34d37fe700 1 -- 172.21.15.52:0/3211100101 <== mon.2 v2:172.21.15.105:3300/0 5 ==== osd_map(130..130 src has 1..130) ==== 9511+0+0 (secure 0 0 0) 0x7f34d40e2310 con 0x7f34e4103030 2024-05-27T07:37:18.947 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:37:18.947+0000 7f34e1d9b700 1 --2- 172.21.15.52:0/3211100101 >> [v2:172.21.15.52:6800/1822318384,v1:172.21.15.52:6801/1822318384] conn(0x7f34cc0b2b80 0x7f34cc0b5090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:37:18.947 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:37:18.948+0000 7f34e1d9b700 1 --2- 172.21.15.52:0/3211100101 >> [v2:172.21.15.52:6800/1822318384,v1:172.21.15.52:6801/1822318384] conn(0x7f34cc0b2b80 0x7f34cc0b5090 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7f34d8009750 tx=0x7f34d8005fb0 comp rx=0 tx=0).ready entity=mgr.15333 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:37:18.947 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:37:18.949+0000 7f34d37fe700 1 -- 172.21.15.52:0/3211100101 <== mon.2 v2:172.21.15.105:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191963 (secure 0 0 0) 0x7f34d40a9aa0 con 0x7f34e4103030 2024-05-27T07:37:19.455 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:37:19.458+0000 7f34e359e700 1 -- 172.21.15.52:0/3211100101 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_command({"prefix": "fs fail", "fs_name": "cephfs"} v 0) -- 0x7f34c40058f0 con 0x7f34e4103030 2024-05-27T07:37:19.476 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:37:19.479+0000 7f34d37fe700 1 -- 172.21.15.52:0/3211100101 <== mon.2 v2:172.21.15.105:3300/0 7 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "cephfs"}]=0 cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v109) ==== 161+0+0 (secure 0 0 0) 0x7f34d40a93a0 con 0x7f34e4103030 2024-05-27T07:37:19.481 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:37:19.483+0000 7f34d17fa700 1 -- 172.21.15.52:0/3211100101 >> [v2:172.21.15.52:6800/1822318384,v1:172.21.15.52:6801/1822318384] conn(0x7f34cc0b2b80 msgr2=0x7f34cc0b5090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:37:19.481 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:37:19.483+0000 7f34d17fa700 1 --2- 172.21.15.52:0/3211100101 >> [v2:172.21.15.52:6800/1822318384,v1:172.21.15.52:6801/1822318384] conn(0x7f34cc0b2b80 0x7f34cc0b5090 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7f34d8009750 tx=0x7f34d8005fb0 comp rx=0 tx=0).stop 2024-05-27T07:37:19.481 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:37:19.483+0000 7f34d17fa700 1 -- 172.21.15.52:0/3211100101 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f34e4103030 msgr2=0x7f34e41176d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:37:19.481 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:37:19.483+0000 7f34d17fa700 1 --2- 172.21.15.52:0/3211100101 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f34e4103030 0x7f34e41176d0 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 crypto rx=0x7f34d400b450 tx=0x7f34d400b810 comp rx=0 tx=0).stop 2024-05-27T07:37:19.481 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:37:19.483+0000 7f34d17fa700 1 -- 172.21.15.52:0/3211100101 shutdown_connections 2024-05-27T07:37:19.481 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:37:19.483+0000 7f34d17fa700 1 --2- 172.21.15.52:0/3211100101 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f34e4108480 0x7f34e4110ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:37:19.481 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:37:19.483+0000 7f34d17fa700 1 --2- 172.21.15.52:0/3211100101 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f34e4103030 0x7f34e41176d0 secure :-1 s=CLOSED pgs=286 cs=0 l=1 rev1=1 crypto rx=0x7f34d400b450 tx=0x7f34d400b810 comp rx=0 tx=0).stop 2024-05-27T07:37:19.481 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:37:19.483+0000 7f34d17fa700 1 --2- 172.21.15.52:0/3211100101 >> [v2:172.21.15.52:6800/1822318384,v1:172.21.15.52:6801/1822318384] conn(0x7f34cc0b2b80 0x7f34cc0b5090 unknown :-1 s=CLOSED pgs=210 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:37:19.482 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:37:19.483+0000 7f34d17fa700 1 --2- 172.21.15.52:0/3211100101 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f34e4103980 0x7f34e4110850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:37:19.482 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:37:19.483+0000 7f34d17fa700 1 -- 172.21.15.52:0/3211100101 >> 172.21.15.52:0/3211100101 conn(0x7f34e4078aa0 msgr2=0x7f34e4105e80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:37:19.482 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:37:19.483+0000 7f34d17fa700 1 -- 172.21.15.52:0/3211100101 shutdown_connections 2024-05-27T07:37:19.482 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:37:19.483+0000 7f34d17fa700 1 -- 172.21.15.52:0/3211100101 wait complete. 2024-05-27T07:37:19.484 INFO:teuthology.orchestra.run.smithi052.stderr:cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2024-05-27T07:37:19.737 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:37:19 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mon-a[31950]: 2024-05-27T07:37:19.460+0000 7f4a9da17700 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-05-27T07:37:19.737 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:37:19 smithi052 ceph-mon[31955]: pgmap v384: 129 pgs: 67 active+clean+snaptrim_wait, 21 active+clean+snaptrim, 41 active+clean; 613 MiB data, 7.7 GiB used, 1.0 TiB / 1.0 TiB avail; 85 B/s wr, 0 op/s 2024-05-27T07:37:19.737 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:37:19 smithi052 ceph-mon[31955]: from='client.? 172.21.15.52:0/3211100101' entity='client.admin' cmd=[{"prefix": "fs fail", "fs_name": "cephfs"}]: dispatch 2024-05-27T07:37:19.738 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:37:19 smithi052 ceph-mon[31955]: from='client.? ' entity='client.admin' cmd=[{"prefix": "fs fail", "fs_name": "cephfs"}]: dispatch 2024-05-27T07:37:19.738 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:37:19 smithi052 ceph-mon[31955]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-05-27T07:37:19.738 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:37:19 smithi052 ceph-mon[31955]: Health check failed: 1 filesystem has a failed mds daemon (FS_WITH_FAILED_MDS) 2024-05-27T07:37:19.738 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:37:19 smithi052 ceph-mon[31955]: Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-05-27T07:37:19.738 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:37:19 smithi052 ceph-mon[31955]: from='client.? ' entity='client.admin' cmd='[{"prefix": "fs fail", "fs_name": "cephfs"}]': finished 2024-05-27T07:37:19.738 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:37:19 smithi052 ceph-mon[31955]: osdmap e131: 12 total, 12 up, 12 in 2024-05-27T07:37:19.738 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:37:19 smithi052 ceph-mon[31955]: fsmap cephfs:0/5, 5 failed 2024-05-27T07:37:19.859 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:37:19 smithi173 ceph-mon[37000]: pgmap v384: 129 pgs: 67 active+clean+snaptrim_wait, 21 active+clean+snaptrim, 41 active+clean; 613 MiB data, 7.7 GiB used, 1.0 TiB / 1.0 TiB avail; 85 B/s wr, 0 op/s 2024-05-27T07:37:19.859 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:37:19 smithi173 ceph-mon[37000]: from='client.? 172.21.15.52:0/3211100101' entity='client.admin' cmd=[{"prefix": "fs fail", "fs_name": "cephfs"}]: dispatch 2024-05-27T07:37:19.859 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:37:19 smithi173 ceph-mon[37000]: from='client.? ' entity='client.admin' cmd=[{"prefix": "fs fail", "fs_name": "cephfs"}]: dispatch 2024-05-27T07:37:19.859 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:37:19 smithi173 ceph-mon[37000]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-05-27T07:37:19.859 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:37:19 smithi173 ceph-mon[37000]: Health check failed: 1 filesystem has a failed mds daemon (FS_WITH_FAILED_MDS) 2024-05-27T07:37:19.859 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:37:19 smithi173 ceph-mon[37000]: Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-05-27T07:37:19.859 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:37:19 smithi173 ceph-mon[37000]: from='client.? ' entity='client.admin' cmd='[{"prefix": "fs fail", "fs_name": "cephfs"}]': finished 2024-05-27T07:37:19.859 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:37:19 smithi173 ceph-mon[37000]: osdmap e131: 12 total, 12 up, 12 in 2024-05-27T07:37:19.859 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:37:19 smithi173 ceph-mon[37000]: fsmap cephfs:0/5, 5 failed 2024-05-27T07:37:19.860 INFO:journalctl@ceph.mds.c.smithi173.stdout:May 27 07:37:19 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-c[68050]: ignoring --setuser ceph since I am not root 2024-05-27T07:37:19.860 INFO:journalctl@ceph.mds.c.smithi173.stdout:May 27 07:37:19 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-c[68050]: ignoring --setgroup ceph since I am not root 2024-05-27T07:37:19.860 INFO:journalctl@ceph.mds.c.smithi173.stdout:May 27 07:37:19 smithi173 ceph-mds[68054]: -- 172.21.15.173:0/1489484272 <== mon.1 v2:172.21.15.173:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x5577625984e0 con 0x5577625e3600 2024-05-27T07:37:19.860 INFO:journalctl@ceph.mds.c.smithi173.stdout:May 27 07:37:19 smithi173 ceph-mds[68054]: -- 172.21.15.173:0/1489484272 <== mon.1 v2:172.21.15.173:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55776259ba40 con 0x5577625e3600 2024-05-27T07:37:19.860 INFO:journalctl@ceph.mds.c.smithi173.stdout:May 27 07:37:19 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-c[68050]: starting mds.c at 2024-05-27T07:37:19.860 INFO:journalctl@ceph.mds.i.smithi173.stdout:May 27 07:37:19 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-i[68353]: ignoring --setuser ceph since I am not root 2024-05-27T07:37:19.860 INFO:journalctl@ceph.mds.i.smithi173.stdout:May 27 07:37:19 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-i[68353]: ignoring --setgroup ceph since I am not root 2024-05-27T07:37:19.860 INFO:journalctl@ceph.mds.i.smithi173.stdout:May 27 07:37:19 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-i[68353]: starting mds.i at 2024-05-27T07:37:19.946 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:37:19 smithi105 ceph-mon[35977]: pgmap v384: 129 pgs: 67 active+clean+snaptrim_wait, 21 active+clean+snaptrim, 41 active+clean; 613 MiB data, 7.7 GiB used, 1.0 TiB / 1.0 TiB avail; 85 B/s wr, 0 op/s 2024-05-27T07:37:19.946 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:37:19 smithi105 ceph-mon[35977]: from='client.? 172.21.15.52:0/3211100101' entity='client.admin' cmd=[{"prefix": "fs fail", "fs_name": "cephfs"}]: dispatch 2024-05-27T07:37:19.946 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:37:19 smithi105 ceph-mon[35977]: from='client.? ' entity='client.admin' cmd=[{"prefix": "fs fail", "fs_name": "cephfs"}]: dispatch 2024-05-27T07:37:19.947 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:37:19 smithi105 ceph-mon[35977]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-05-27T07:37:19.947 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:37:19 smithi105 ceph-mon[35977]: Health check failed: 1 filesystem has a failed mds daemon (FS_WITH_FAILED_MDS) 2024-05-27T07:37:19.947 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:37:19 smithi105 ceph-mon[35977]: Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-05-27T07:37:19.947 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:37:19 smithi105 ceph-mon[35977]: from='client.? ' entity='client.admin' cmd='[{"prefix": "fs fail", "fs_name": "cephfs"}]': finished 2024-05-27T07:37:19.947 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:37:19 smithi105 ceph-mon[35977]: osdmap e131: 12 total, 12 up, 12 in 2024-05-27T07:37:19.947 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:37:19 smithi105 ceph-mon[35977]: fsmap cephfs:0/5, 5 failed 2024-05-27T07:37:19.947 INFO:journalctl@ceph.mds.e.smithi105.stdout:May 27 07:37:19 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-e[64932]: ignoring --setuser ceph since I am not root 2024-05-27T07:37:19.947 INFO:journalctl@ceph.mds.e.smithi105.stdout:May 27 07:37:19 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-e[64932]: ignoring --setgroup ceph since I am not root 2024-05-27T07:37:19.947 INFO:journalctl@ceph.mds.e.smithi105.stdout:May 27 07:37:19 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-e[64932]: starting mds.e at 2024-05-27T07:37:19.948 INFO:journalctl@ceph.mds.h.smithi105.stdout:May 27 07:37:19 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-h[65240]: ignoring --setuser ceph since I am not root 2024-05-27T07:37:19.948 INFO:journalctl@ceph.mds.h.smithi105.stdout:May 27 07:37:19 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-h[65240]: ignoring --setgroup ceph since I am not root 2024-05-27T07:37:19.948 INFO:journalctl@ceph.mds.h.smithi105.stdout:May 27 07:37:19 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-h[65240]: starting mds.h at 2024-05-27T07:37:19.948 INFO:journalctl@ceph.mds.b.smithi105.stdout:May 27 07:37:19 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-b[65543]: ignoring --setuser ceph since I am not root 2024-05-27T07:37:19.948 INFO:journalctl@ceph.mds.b.smithi105.stdout:May 27 07:37:19 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-b[65543]: ignoring --setgroup ceph since I am not root 2024-05-27T07:37:19.948 INFO:journalctl@ceph.mds.b.smithi105.stdout:May 27 07:37:19 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-b[65543]: starting mds.b at 2024-05-27T07:37:20.373 DEBUG:teuthology.orchestra.run.smithi052:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:38f9d900c83c63799fbdbe61acc9a11b0d3554a6 shell --fsid 73df2636-1bf8-11ef-bc9b-c7b262605968 -- ceph fs rm cephfs --yes-i-really-mean-it 2024-05-27T07:37:20.736 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:37:20 smithi052 ceph-mon[31955]: Health check cleared: FS_WITH_FAILED_MDS (was: 1 filesystem has a failed mds daemon) 2024-05-27T07:37:20.736 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:37:20 smithi052 ceph-mon[31955]: mds.? [v2:172.21.15.173:6832/3562445654,v1:172.21.15.173:6833/3562445654] up:boot 2024-05-27T07:37:20.736 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:37:20 smithi052 ceph-mon[31955]: mds.? [v2:172.21.15.173:6834/1584378798,v1:172.21.15.173:6835/1584378798] up:boot 2024-05-27T07:37:20.736 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:37:20 smithi052 ceph-mon[31955]: mds.? [v2:172.21.15.105:6832/1010990298,v1:172.21.15.105:6833/1010990298] up:boot 2024-05-27T07:37:20.736 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:37:20 smithi052 ceph-mon[31955]: mds.? [v2:172.21.15.105:6834/4261809036,v1:172.21.15.105:6835/4261809036] up:boot 2024-05-27T07:37:20.736 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:37:20 smithi052 ceph-mon[31955]: mds.? [v2:172.21.15.105:6836/1399567009,v1:172.21.15.105:6837/1399567009] up:boot 2024-05-27T07:37:20.737 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:37:20 smithi052 ceph-mon[31955]: fsmap cephfs:0/5 5 up:standby, 5 failed 2024-05-27T07:37:20.737 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:37:20 smithi052 ceph-mon[31955]: from='mgr.15333 172.21.15.52:0/416389082' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "b"}]: dispatch 2024-05-27T07:37:20.737 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:37:20 smithi052 ceph-mon[31955]: from='mgr.15333 172.21.15.52:0/416389082' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "i"}]: dispatch 2024-05-27T07:37:20.737 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:37:20 smithi052 ceph-mon[31955]: from='mgr.15333 172.21.15.52:0/416389082' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "e"}]: dispatch 2024-05-27T07:37:20.737 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:37:20 smithi052 ceph-mon[31955]: from='mgr.15333 172.21.15.52:0/416389082' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2024-05-27T07:37:20.737 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:37:20 smithi052 ceph-mon[31955]: from='mgr.15333 172.21.15.52:0/416389082' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2024-05-27T07:37:20.858 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:37:20 smithi173 ceph-mon[37000]: Health check cleared: FS_WITH_FAILED_MDS (was: 1 filesystem has a failed mds daemon) 2024-05-27T07:37:20.859 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:37:20 smithi173 ceph-mon[37000]: mds.? [v2:172.21.15.173:6832/3562445654,v1:172.21.15.173:6833/3562445654] up:boot 2024-05-27T07:37:20.859 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:37:20 smithi173 ceph-mon[37000]: mds.? [v2:172.21.15.173:6834/1584378798,v1:172.21.15.173:6835/1584378798] up:boot 2024-05-27T07:37:20.859 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:37:20 smithi173 ceph-mon[37000]: mds.? [v2:172.21.15.105:6832/1010990298,v1:172.21.15.105:6833/1010990298] up:boot 2024-05-27T07:37:20.859 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:37:20 smithi173 ceph-mon[37000]: mds.? [v2:172.21.15.105:6834/4261809036,v1:172.21.15.105:6835/4261809036] up:boot 2024-05-27T07:37:20.859 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:37:20 smithi173 ceph-mon[37000]: mds.? [v2:172.21.15.105:6836/1399567009,v1:172.21.15.105:6837/1399567009] up:boot 2024-05-27T07:37:20.859 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:37:20 smithi173 ceph-mon[37000]: fsmap cephfs:0/5 5 up:standby, 5 failed 2024-05-27T07:37:20.859 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:37:20 smithi173 ceph-mon[37000]: from='mgr.15333 172.21.15.52:0/416389082' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "b"}]: dispatch 2024-05-27T07:37:20.859 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:37:20 smithi173 ceph-mon[37000]: from='mgr.15333 172.21.15.52:0/416389082' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "i"}]: dispatch 2024-05-27T07:37:20.859 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:37:20 smithi173 ceph-mon[37000]: from='mgr.15333 172.21.15.52:0/416389082' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "e"}]: dispatch 2024-05-27T07:37:20.859 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:37:20 smithi173 ceph-mon[37000]: from='mgr.15333 172.21.15.52:0/416389082' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2024-05-27T07:37:20.859 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:37:20 smithi173 ceph-mon[37000]: from='mgr.15333 172.21.15.52:0/416389082' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2024-05-27T07:37:20.862 INFO:teuthology.orchestra.run.smithi052.stderr:Inferring config /var/lib/ceph/73df2636-1bf8-11ef-bc9b-c7b262605968/mon.a/config 2024-05-27T07:37:20.946 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:37:20 smithi105 ceph-mon[35977]: Health check cleared: FS_WITH_FAILED_MDS (was: 1 filesystem has a failed mds daemon) 2024-05-27T07:37:20.946 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:37:20 smithi105 ceph-mon[35977]: mds.? [v2:172.21.15.173:6832/3562445654,v1:172.21.15.173:6833/3562445654] up:boot 2024-05-27T07:37:20.947 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:37:20 smithi105 ceph-mon[35977]: mds.? [v2:172.21.15.173:6834/1584378798,v1:172.21.15.173:6835/1584378798] up:boot 2024-05-27T07:37:20.947 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:37:20 smithi105 ceph-mon[35977]: mds.? [v2:172.21.15.105:6832/1010990298,v1:172.21.15.105:6833/1010990298] up:boot 2024-05-27T07:37:20.947 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:37:20 smithi105 ceph-mon[35977]: mds.? [v2:172.21.15.105:6834/4261809036,v1:172.21.15.105:6835/4261809036] up:boot 2024-05-27T07:37:20.947 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:37:20 smithi105 ceph-mon[35977]: mds.? [v2:172.21.15.105:6836/1399567009,v1:172.21.15.105:6837/1399567009] up:boot 2024-05-27T07:37:20.947 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:37:20 smithi105 ceph-mon[35977]: fsmap cephfs:0/5 5 up:standby, 5 failed 2024-05-27T07:37:20.947 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:37:20 smithi105 ceph-mon[35977]: from='mgr.15333 172.21.15.52:0/416389082' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "b"}]: dispatch 2024-05-27T07:37:20.947 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:37:20 smithi105 ceph-mon[35977]: from='mgr.15333 172.21.15.52:0/416389082' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "i"}]: dispatch 2024-05-27T07:37:20.947 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:37:20 smithi105 ceph-mon[35977]: from='mgr.15333 172.21.15.52:0/416389082' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "e"}]: dispatch 2024-05-27T07:37:20.947 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:37:20 smithi105 ceph-mon[35977]: from='mgr.15333 172.21.15.52:0/416389082' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2024-05-27T07:37:20.947 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:37:20 smithi105 ceph-mon[35977]: from='mgr.15333 172.21.15.52:0/416389082' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2024-05-27T07:37:21.614 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:37:21 smithi052 ceph-mon[31955]: pgmap v386: 129 pgs: 67 active+clean+snaptrim_wait, 21 active+clean+snaptrim, 41 active+clean; 612 MiB data, 7.7 GiB used, 1.0 TiB / 1.0 TiB avail 2024-05-27T07:37:21.858 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:37:21 smithi173 ceph-mon[37000]: pgmap v386: 129 pgs: 67 active+clean+snaptrim_wait, 21 active+clean+snaptrim, 41 active+clean; 612 MiB data, 7.7 GiB used, 1.0 TiB / 1.0 TiB avail 2024-05-27T07:37:21.946 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:37:21 smithi105 ceph-mon[35977]: pgmap v386: 129 pgs: 67 active+clean+snaptrim_wait, 21 active+clean+snaptrim, 41 active+clean; 612 MiB data, 7.7 GiB used, 1.0 TiB / 1.0 TiB avail 2024-05-27T07:37:23.407 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:37:23.408+0000 7ff4fa2b0700 1 -- 172.21.15.52:0/3677539639 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7ff4f410a7b0 msgr2=0x7ff4f410abc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:37:23.407 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:37:23.408+0000 7ff4fa2b0700 1 --2- 172.21.15.52:0/3677539639 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7ff4f410a7b0 0x7ff4f410abc0 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7ff4e4009990 tx=0x7ff4e4009ca0 comp rx=0 tx=0).stop 2024-05-27T07:37:23.407 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:37:23.408+0000 7ff4fa2b0700 1 -- 172.21.15.52:0/3677539639 shutdown_connections 2024-05-27T07:37:23.407 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:37:23.408+0000 7ff4fa2b0700 1 --2- 172.21.15.52:0/3677539639 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7ff4f411d920 0x7ff4f411fd40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:37:23.407 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:37:23.408+0000 7ff4fa2b0700 1 --2- 172.21.15.52:0/3677539639 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7ff4f410a7b0 0x7ff4f410abc0 secure :-1 s=CLOSED pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7ff4e4009990 tx=0x7ff4e4009ca0 comp rx=0 tx=0).stop 2024-05-27T07:37:23.407 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:37:23.408+0000 7ff4fa2b0700 1 --2- 172.21.15.52:0/3677539639 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff4f410b100 0x7ff4f411d3e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:37:23.407 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:37:23.408+0000 7ff4fa2b0700 1 -- 172.21.15.52:0/3677539639 >> 172.21.15.52:0/3677539639 conn(0x7ff4f406f280 msgr2=0x7ff4f406f6c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:37:23.409 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:37:23.412+0000 7ff4fa2b0700 1 -- 172.21.15.52:0/3677539639 shutdown_connections 2024-05-27T07:37:23.409 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:37:23.412+0000 7ff4fa2b0700 1 -- 172.21.15.52:0/3677539639 wait complete. 2024-05-27T07:37:23.414 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:37:23.413+0000 7ff4fa2b0700 1 Processor -- start 2024-05-27T07:37:23.414 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:37:23.413+0000 7ff4fa2b0700 1 -- start start 2024-05-27T07:37:23.414 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:37:23.413+0000 7ff4fa2b0700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7ff4f410b100 0x7ff4f411ad00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:37:23.415 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:37:23.413+0000 7ff4fa2b0700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff4f411d920 0x7ff4f4112cb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:37:23.415 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:37:23.413+0000 7ff4fa2b0700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7ff4f41131f0 0x7ff4f4113600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:37:23.415 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:37:23.413+0000 7ff4fa2b0700 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x7ff4f41230e0 con 0x7ff4f411d920 2024-05-27T07:37:23.415 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:37:23.413+0000 7ff4fa2b0700 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7ff4f4122f60 con 0x7ff4f410b100 2024-05-27T07:37:23.415 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:37:23.413+0000 7ff4fa2b0700 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7ff4f4123260 con 0x7ff4f41131f0 2024-05-27T07:37:23.415 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:37:23.414+0000 7ff4f9aaf700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7ff4f41131f0 0x7ff4f4113600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:37:23.415 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:37:23.414+0000 7ff4f9aaf700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7ff4f41131f0 0x7ff4f4113600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.173:3300/0 says I am v2:172.21.15.52:37446/0 (socket says 172.21.15.52:37446) 2024-05-27T07:37:23.415 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:37:23.414+0000 7ff4f9aaf700 1 -- 172.21.15.52:0/1182338448 learned_addr learned my addr 172.21.15.52:0/1182338448 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-05-27T07:37:23.415 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:37:23.414+0000 7ff4f9aaf700 1 -- 172.21.15.52:0/1182338448 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7ff4f410b100 msgr2=0x7ff4f411ad00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:37:23.415 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:37:23.414+0000 7ff4f9aaf700 1 --2- 172.21.15.52:0/1182338448 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7ff4f410b100 0x7ff4f411ad00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:37:23.415 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:37:23.414+0000 7ff4f9aaf700 1 -- 172.21.15.52:0/1182338448 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff4f411d920 msgr2=0x7ff4f4112cb0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-05-27T07:37:23.415 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:37:23.414+0000 7ff4f9aaf700 1 --2- 172.21.15.52:0/1182338448 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff4f411d920 0x7ff4f4112cb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:37:23.416 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:37:23.414+0000 7ff4f9aaf700 1 -- 172.21.15.52:0/1182338448 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff4f41be9f0 con 0x7ff4f41131f0 2024-05-27T07:37:23.416 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:37:23.415+0000 7ff4f9aaf700 1 --2- 172.21.15.52:0/1182338448 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7ff4f41131f0 0x7ff4f4113600 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7ff4ec030950 tx=0x7ff4ec030d10 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:37:23.417 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:37:23.418+0000 7ff4ea7fc700 1 -- 172.21.15.52:0/1182338448 <== mon.1 v2:172.21.15.173:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff4ec02ebe0 con 0x7ff4f41131f0 2024-05-27T07:37:23.417 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:37:23.418+0000 7ff4ea7fc700 1 -- 172.21.15.52:0/1182338448 <== mon.1 v2:172.21.15.173:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ff4ec02ed80 con 0x7ff4f41131f0 2024-05-27T07:37:23.417 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:37:23.418+0000 7ff4fa2b0700 1 -- 172.21.15.52:0/1182338448 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff4f41a98f0 con 0x7ff4f41131f0 2024-05-27T07:37:23.417 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:37:23.418+0000 7ff4fa2b0700 1 -- 172.21.15.52:0/1182338448 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff4f41a9e30 con 0x7ff4f41131f0 2024-05-27T07:37:23.417 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:37:23.418+0000 7ff4ea7fc700 1 -- 172.21.15.52:0/1182338448 <== mon.1 v2:172.21.15.173:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff4ec032440 con 0x7ff4f41131f0 2024-05-27T07:37:23.418 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:37:23.421+0000 7ff4fa2b0700 1 -- 172.21.15.52:0/1182338448 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff4d8005330 con 0x7ff4f41131f0 2024-05-27T07:37:23.422 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:37:23.425+0000 7ff4ea7fc700 1 -- 172.21.15.52:0/1182338448 <== mon.1 v2:172.21.15.173:3300/0 4 ==== mgrmap(e 26) ==== 149783+0+0 (secure 0 0 0) 0x7ff4ec004750 con 0x7ff4f41131f0 2024-05-27T07:37:23.423 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:37:23.426+0000 7ff4ea7fc700 1 --2- 172.21.15.52:0/1182338448 >> [v2:172.21.15.52:6800/1822318384,v1:172.21.15.52:6801/1822318384] conn(0x7ff4e00b2c00 0x7ff4e00b5110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:37:23.424 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:37:23.426+0000 7ff4f92ae700 1 --2- 172.21.15.52:0/1182338448 >> [v2:172.21.15.52:6800/1822318384,v1:172.21.15.52:6801/1822318384] conn(0x7ff4e00b2c00 0x7ff4e00b5110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:37:23.424 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:37:23.427+0000 7ff4ea7fc700 1 -- 172.21.15.52:0/1182338448 <== mon.1 v2:172.21.15.173:3300/0 5 ==== osd_map(131..131 src has 1..131) ==== 9897+0+0 (secure 0 0 0) 0x7ff4ec104db0 con 0x7ff4f41131f0 2024-05-27T07:37:23.427 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:37:23.430+0000 7ff4f92ae700 1 --2- 172.21.15.52:0/1182338448 >> [v2:172.21.15.52:6800/1822318384,v1:172.21.15.52:6801/1822318384] conn(0x7ff4e00b2c00 0x7ff4e00b5110 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7ff4e40098e0 tx=0x7ff4e4005fb0 comp rx=0 tx=0).ready entity=mgr.15333 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:37:23.430 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:37:23.433+0000 7ff4ea7fc700 1 -- 172.21.15.52:0/1182338448 <== mon.1 v2:172.21.15.173:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191963 (secure 0 0 0) 0x7ff4ec0cc1d0 con 0x7ff4f41131f0 2024-05-27T07:37:23.939 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:37:23.943+0000 7ff4fa2b0700 1 -- 172.21.15.52:0/1182338448 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_command({"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true} v 0) -- 0x7ff4d80058f0 con 0x7ff4f41131f0 2024-05-27T07:37:24.457 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:37:24.461+0000 7ff4ea7fc700 1 -- 172.21.15.52:0/1182338448 <== mon.1 v2:172.21.15.173:3300/0 7 ==== mon_command_ack([{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]=0 v111) ==== 104+0+0 (secure 0 0 0) 0x7ff4ec0cbad0 con 0x7ff4f41131f0 2024-05-27T07:37:24.461 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:37:24.465+0000 7ff4fa2b0700 1 -- 172.21.15.52:0/1182338448 >> [v2:172.21.15.52:6800/1822318384,v1:172.21.15.52:6801/1822318384] conn(0x7ff4e00b2c00 msgr2=0x7ff4e00b5110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:37:24.461 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:37:24.465+0000 7ff4fa2b0700 1 --2- 172.21.15.52:0/1182338448 >> [v2:172.21.15.52:6800/1822318384,v1:172.21.15.52:6801/1822318384] conn(0x7ff4e00b2c00 0x7ff4e00b5110 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7ff4e40098e0 tx=0x7ff4e4005fb0 comp rx=0 tx=0).stop 2024-05-27T07:37:24.461 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:37:24.465+0000 7ff4fa2b0700 1 -- 172.21.15.52:0/1182338448 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7ff4f41131f0 msgr2=0x7ff4f4113600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:37:24.461 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:37:24.465+0000 7ff4fa2b0700 1 --2- 172.21.15.52:0/1182338448 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7ff4f41131f0 0x7ff4f4113600 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7ff4ec030950 tx=0x7ff4ec030d10 comp rx=0 tx=0).stop 2024-05-27T07:37:24.462 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:37:24.466+0000 7ff4fa2b0700 1 -- 172.21.15.52:0/1182338448 shutdown_connections 2024-05-27T07:37:24.462 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:37:24.466+0000 7ff4fa2b0700 1 --2- 172.21.15.52:0/1182338448 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7ff4f41131f0 0x7ff4f4113600 unknown :-1 s=CLOSED pgs=242 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:37:24.462 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:37:24.466+0000 7ff4fa2b0700 1 --2- 172.21.15.52:0/1182338448 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7ff4f410b100 0x7ff4f411ad00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:37:24.462 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:37:24.466+0000 7ff4fa2b0700 1 --2- 172.21.15.52:0/1182338448 >> [v2:172.21.15.52:6800/1822318384,v1:172.21.15.52:6801/1822318384] conn(0x7ff4e00b2c00 0x7ff4e00b5110 unknown :-1 s=CLOSED pgs=216 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:37:24.462 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:37:24.466+0000 7ff4fa2b0700 1 --2- 172.21.15.52:0/1182338448 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff4f411d920 0x7ff4f4112cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:37:24.463 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:37:24.466+0000 7ff4fa2b0700 1 -- 172.21.15.52:0/1182338448 >> 172.21.15.52:0/1182338448 conn(0x7ff4f406f280 msgr2=0x7ff4f411b3d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:37:24.465 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:37:24.469+0000 7ff4fa2b0700 1 -- 172.21.15.52:0/1182338448 shutdown_connections 2024-05-27T07:37:24.465 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:37:24.469+0000 7ff4fa2b0700 1 -- 172.21.15.52:0/1182338448 wait complete. 2024-05-27T07:37:24.696 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:37:24 smithi105 ceph-mon[35977]: pgmap v387: 129 pgs: 67 active+clean+snaptrim_wait, 21 active+clean+snaptrim, 41 active+clean; 611 MiB data, 7.8 GiB used, 1.0 TiB / 1.0 TiB avail 2024-05-27T07:37:24.696 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:37:24 smithi105 ceph-mon[35977]: from='client.? 172.21.15.52:0/1182338448' entity='client.admin' cmd=[{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]: dispatch 2024-05-27T07:37:24.696 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:37:24 smithi105 ceph-mon[35977]: from='client.? ' entity='client.admin' cmd=[{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]: dispatch 2024-05-27T07:37:24.741 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:37:24 smithi052 ceph-mon[31955]: pgmap v387: 129 pgs: 67 active+clean+snaptrim_wait, 21 active+clean+snaptrim, 41 active+clean; 611 MiB data, 7.8 GiB used, 1.0 TiB / 1.0 TiB avail 2024-05-27T07:37:24.741 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:37:24 smithi052 ceph-mon[31955]: from='client.? 172.21.15.52:0/1182338448' entity='client.admin' cmd=[{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]: dispatch 2024-05-27T07:37:24.741 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:37:24 smithi052 ceph-mon[31955]: from='client.? ' entity='client.admin' cmd=[{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]: dispatch 2024-05-27T07:37:24.858 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:37:24 smithi173 ceph-mon[37000]: pgmap v387: 129 pgs: 67 active+clean+snaptrim_wait, 21 active+clean+snaptrim, 41 active+clean; 611 MiB data, 7.8 GiB used, 1.0 TiB / 1.0 TiB avail 2024-05-27T07:37:24.859 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:37:24 smithi173 ceph-mon[37000]: from='client.? 172.21.15.52:0/1182338448' entity='client.admin' cmd=[{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]: dispatch 2024-05-27T07:37:24.859 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:37:24 smithi173 ceph-mon[37000]: from='client.? ' entity='client.admin' cmd=[{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]: dispatch 2024-05-27T07:37:25.201 DEBUG:teuthology.orchestra.run.smithi052:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:38f9d900c83c63799fbdbe61acc9a11b0d3554a6 shell --fsid 73df2636-1bf8-11ef-bc9b-c7b262605968 -- ceph osd pool rm cephfs_metadata cephfs_metadata --yes-i-really-really-mean-it 2024-05-27T07:37:25.696 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:37:25 smithi105 ceph-mon[35977]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-05-27T07:37:25.697 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:37:25 smithi105 ceph-mon[35977]: Health check cleared: MDS_ALL_DOWN (was: 1 filesystem is offline) 2024-05-27T07:37:25.697 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:37:25 smithi105 ceph-mon[35977]: from='client.? ' entity='client.admin' cmd='[{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]': finished 2024-05-27T07:37:25.697 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:37:25 smithi105 ceph-mon[35977]: fsmap 5 up:standby 2024-05-27T07:37:25.697 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:37:25 smithi105 ceph-mon[35977]: pgmap v388: 129 pgs: 62 active+clean+snaptrim_wait, 20 active+clean+snaptrim, 47 active+clean; 610 MiB data, 7.8 GiB used, 1.0 TiB / 1.0 TiB avail 2024-05-27T07:37:25.728 INFO:teuthology.orchestra.run.smithi052.stderr:Inferring config /var/lib/ceph/73df2636-1bf8-11ef-bc9b-c7b262605968/mon.a/config 2024-05-27T07:37:25.736 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:37:25 smithi052 ceph-mon[31955]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-05-27T07:37:25.736 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:37:25 smithi052 ceph-mon[31955]: Health check cleared: MDS_ALL_DOWN (was: 1 filesystem is offline) 2024-05-27T07:37:25.736 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:37:25 smithi052 ceph-mon[31955]: from='client.? ' entity='client.admin' cmd='[{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]': finished 2024-05-27T07:37:25.736 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:37:25 smithi052 ceph-mon[31955]: fsmap 5 up:standby 2024-05-27T07:37:25.736 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:37:25 smithi052 ceph-mon[31955]: pgmap v388: 129 pgs: 62 active+clean+snaptrim_wait, 20 active+clean+snaptrim, 47 active+clean; 610 MiB data, 7.8 GiB used, 1.0 TiB / 1.0 TiB avail 2024-05-27T07:37:25.858 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:37:25 smithi173 ceph-mon[37000]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-05-27T07:37:25.858 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:37:25 smithi173 ceph-mon[37000]: Health check cleared: MDS_ALL_DOWN (was: 1 filesystem is offline) 2024-05-27T07:37:25.858 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:37:25 smithi173 ceph-mon[37000]: from='client.? ' entity='client.admin' cmd='[{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]': finished 2024-05-27T07:37:25.859 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:37:25 smithi173 ceph-mon[37000]: fsmap 5 up:standby 2024-05-27T07:37:25.859 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:37:25 smithi173 ceph-mon[37000]: pgmap v388: 129 pgs: 62 active+clean+snaptrim_wait, 20 active+clean+snaptrim, 47 active+clean; 610 MiB data, 7.8 GiB used, 1.0 TiB / 1.0 TiB avail 2024-05-27T07:37:28.696 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:37:28 smithi105 ceph-mon[35977]: pgmap v389: 129 pgs: 56 active+clean+snaptrim_wait, 20 active+clean+snaptrim, 53 active+clean; 610 MiB data, 7.8 GiB used, 1.0 TiB / 1.0 TiB avail 2024-05-27T07:37:28.736 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:37:28 smithi052 ceph-mon[31955]: pgmap v389: 129 pgs: 56 active+clean+snaptrim_wait, 20 active+clean+snaptrim, 53 active+clean; 610 MiB data, 7.8 GiB used, 1.0 TiB / 1.0 TiB avail 2024-05-27T07:37:28.858 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:37:28 smithi173 ceph-mon[37000]: pgmap v389: 129 pgs: 56 active+clean+snaptrim_wait, 20 active+clean+snaptrim, 53 active+clean; 610 MiB data, 7.8 GiB used, 1.0 TiB / 1.0 TiB avail 2024-05-27T07:37:29.689 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:37:29.692+0000 7fa3a325f700 1 -- 172.21.15.52:0/464892583 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fa39c1045e0 msgr2=0x7fa39c1049f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:37:29.689 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:37:29.692+0000 7fa3a325f700 1 --2- 172.21.15.52:0/464892583 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fa39c1045e0 0x7fa39c1049f0 secure :-1 s=READY pgs=290 cs=0 l=1 rev1=1 crypto rx=0x7fa38c009990 tx=0x7fa38c009ca0 comp rx=0 tx=0).stop 2024-05-27T07:37:29.692 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:37:29.696+0000 7fa3a325f700 1 -- 172.21.15.52:0/464892583 shutdown_connections 2024-05-27T07:37:29.692 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:37:29.696+0000 7fa3a325f700 1 --2- 172.21.15.52:0/464892583 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fa39c109860 0x7fa39c1111c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:37:29.692 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:37:29.696+0000 7fa3a325f700 1 --2- 172.21.15.52:0/464892583 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fa39c1045e0 0x7fa39c1049f0 unknown :-1 s=CLOSED pgs=290 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:37:29.692 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:37:29.696+0000 7fa3a325f700 1 --2- 172.21.15.52:0/464892583 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fa39c108f10 0x7fa39c109320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:37:29.692 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:37:29.696+0000 7fa3a325f700 1 -- 172.21.15.52:0/464892583 >> 172.21.15.52:0/464892583 conn(0x7fa39c100030 msgr2=0x7fa39c102450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:37:29.692 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:37:29.696+0000 7fa3a325f700 1 -- 172.21.15.52:0/464892583 shutdown_connections 2024-05-27T07:37:29.694 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:37:29.696+0000 7fa3a325f700 1 -- 172.21.15.52:0/464892583 wait complete. 2024-05-27T07:37:29.694 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:37:29.697+0000 7fa3a325f700 1 Processor -- start 2024-05-27T07:37:29.694 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:37:29.697+0000 7fa3a325f700 1 -- start start 2024-05-27T07:37:29.694 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:37:29.698+0000 7fa3a325f700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fa39c1045e0 0x7fa39c075310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:37:29.694 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:37:29.698+0000 7fa3a325f700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fa39c108f10 0x7fa39c06e440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:37:29.694 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:37:29.698+0000 7fa3a325f700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fa39c109860 0x7fa39c06e9e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:37:29.696 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:37:29.698+0000 7fa3a325f700 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x7fa39c1167e0 con 0x7fa39c108f10 2024-05-27T07:37:29.696 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:37:29.698+0000 7fa3a325f700 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7fa39c116660 con 0x7fa39c109860 2024-05-27T07:37:29.696 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:37:29.698+0000 7fa3a325f700 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7fa39c116960 con 0x7fa39c1045e0 2024-05-27T07:37:29.696 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:37:29.699+0000 7fa3a0ffb700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fa39c1045e0 0x7fa39c075310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:37:29.696 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:37:29.699+0000 7fa3a17fc700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fa39c109860 0x7fa39c06e9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:37:29.696 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:37:29.699+0000 7fa3a17fc700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fa39c109860 0x7fa39c06e9e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.52:39408/0 (socket says 172.21.15.52:39408) 2024-05-27T07:37:29.696 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:37:29.699+0000 7fa3a17fc700 1 -- 172.21.15.52:0/350787862 learned_addr learned my addr 172.21.15.52:0/350787862 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-05-27T07:37:29.696 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:37:29.699+0000 7fa39bfff700 1 --2- 172.21.15.52:0/350787862 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fa39c108f10 0x7fa39c06e440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:37:29.696 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:37:29.699+0000 7fa3a17fc700 1 -- 172.21.15.52:0/350787862 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fa39c1045e0 msgr2=0x7fa39c075310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:37:29.697 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:37:29.699+0000 7fa3a17fc700 1 --2- 172.21.15.52:0/350787862 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fa39c1045e0 0x7fa39c075310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:37:29.697 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:37:29.699+0000 7fa3a17fc700 1 -- 172.21.15.52:0/350787862 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fa39c108f10 msgr2=0x7fa39c06e440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:37:29.697 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:37:29.699+0000 7fa3a17fc700 1 --2- 172.21.15.52:0/350787862 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fa39c108f10 0x7fa39c06e440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:37:29.697 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:37:29.699+0000 7fa3a17fc700 1 -- 172.21.15.52:0/350787862 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa39c06f240 con 0x7fa39c109860 2024-05-27T07:37:29.698 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:37:29.700+0000 7fa3a17fc700 1 --2- 172.21.15.52:0/350787862 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fa39c109860 0x7fa39c06e9e0 secure :-1 s=READY pgs=291 cs=0 l=1 rev1=1 crypto rx=0x7fa39400b970 tx=0x7fa39400bc80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:37:29.698 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:37:29.701+0000 7fa399ffb700 1 -- 172.21.15.52:0/350787862 <== mon.2 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa394019070 con 0x7fa39c109860 2024-05-27T07:37:29.698 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:37:29.701+0000 7fa399ffb700 1 -- 172.21.15.52:0/350787862 <== mon.2 v2:172.21.15.105:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fa39400cde0 con 0x7fa39c109860 2024-05-27T07:37:29.698 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:37:29.701+0000 7fa399ffb700 1 -- 172.21.15.52:0/350787862 <== mon.2 v2:172.21.15.105:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa394007370 con 0x7fa39c109860 2024-05-27T07:37:29.698 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:37:29.701+0000 7fa3a325f700 1 -- 172.21.15.52:0/350787862 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa39c072f60 con 0x7fa39c109860 2024-05-27T07:37:29.698 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:37:29.701+0000 7fa3a325f700 1 -- 172.21.15.52:0/350787862 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa39c0733f0 con 0x7fa39c109860 2024-05-27T07:37:29.699 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:37:29.703+0000 7fa3a0ffb700 1 --2- 172.21.15.52:0/350787862 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fa39c1045e0 0x7fa39c075310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-27T07:37:29.700 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:37:29.704+0000 7fa3a325f700 1 -- 172.21.15.52:0/350787862 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa39c105a90 con 0x7fa39c109860 2024-05-27T07:37:29.705 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:37:29.709+0000 7fa399ffb700 1 -- 172.21.15.52:0/350787862 <== mon.2 v2:172.21.15.105:3300/0 4 ==== mgrmap(e 26) ==== 149783+0+0 (secure 0 0 0) 0x7fa39401c070 con 0x7fa39c109860 2024-05-27T07:37:29.707 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:37:29.711+0000 7fa399ffb700 1 --2- 172.21.15.52:0/350787862 >> [v2:172.21.15.52:6800/1822318384,v1:172.21.15.52:6801/1822318384] conn(0x7fa3840b2c00 0x7fa3840b5110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:37:29.707 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:37:29.711+0000 7fa399ffb700 1 -- 172.21.15.52:0/350787862 <== mon.2 v2:172.21.15.105:3300/0 5 ==== osd_map(131..131 src has 1..131) ==== 9897+0+0 (secure 0 0 0) 0x7fa3940e2e80 con 0x7fa39c109860 2024-05-27T07:37:29.708 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:37:29.712+0000 7fa3a0ffb700 1 --2- 172.21.15.52:0/350787862 >> [v2:172.21.15.52:6800/1822318384,v1:172.21.15.52:6801/1822318384] conn(0x7fa3840b2c00 0x7fa3840b5110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:37:29.712 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:37:29.716+0000 7fa3a0ffb700 1 --2- 172.21.15.52:0/350787862 >> [v2:172.21.15.52:6800/1822318384,v1:172.21.15.52:6801/1822318384] conn(0x7fa3840b2c00 0x7fa3840b5110 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7fa38c009750 tx=0x7fa38c009f60 comp rx=0 tx=0).ready entity=mgr.15333 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:37:29.713 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:37:29.717+0000 7fa399ffb700 1 -- 172.21.15.52:0/350787862 <== mon.2 v2:172.21.15.105:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191963 (secure 0 0 0) 0x7fa394051df0 con 0x7fa39c109860 2024-05-27T07:37:30.135 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:37:30.132+0000 7fa3a325f700 1 -- 172.21.15.52:0/350787862 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true} v 0) -- 0x7fa39c10c330 con 0x7fa39c109860 2024-05-27T07:37:30.481 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:37:30.461+0000 7fa399ffb700 1 -- 172.21.15.52:0/350787862 <== mon.2 v2:172.21.15.105:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_metadata' removed v132) ==== 181+0+0 (secure 0 0 0) 0x7fa3940aa3b0 con 0x7fa39c109860 2024-05-27T07:37:30.482 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:37:30.465+0000 7fa3a325f700 1 -- 172.21.15.52:0/350787862 >> [v2:172.21.15.52:6800/1822318384,v1:172.21.15.52:6801/1822318384] conn(0x7fa3840b2c00 msgr2=0x7fa3840b5110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:37:30.482 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:37:30.465+0000 7fa3a325f700 1 --2- 172.21.15.52:0/350787862 >> [v2:172.21.15.52:6800/1822318384,v1:172.21.15.52:6801/1822318384] conn(0x7fa3840b2c00 0x7fa3840b5110 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7fa38c009750 tx=0x7fa38c009f60 comp rx=0 tx=0).stop 2024-05-27T07:37:30.482 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:37:30.466+0000 7fa3a325f700 1 -- 172.21.15.52:0/350787862 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fa39c109860 msgr2=0x7fa39c06e9e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:37:30.482 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:37:30.466+0000 7fa3a325f700 1 --2- 172.21.15.52:0/350787862 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fa39c109860 0x7fa39c06e9e0 secure :-1 s=READY pgs=291 cs=0 l=1 rev1=1 crypto rx=0x7fa39400b970 tx=0x7fa39400bc80 comp rx=0 tx=0).stop 2024-05-27T07:37:30.482 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:37:30.466+0000 7fa3a325f700 1 -- 172.21.15.52:0/350787862 shutdown_connections 2024-05-27T07:37:30.482 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:37:30.466+0000 7fa3a325f700 1 --2- 172.21.15.52:0/350787862 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fa39c109860 0x7fa39c06e9e0 unknown :-1 s=CLOSED pgs=291 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:37:30.482 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:37:30.466+0000 7fa3a325f700 1 --2- 172.21.15.52:0/350787862 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fa39c1045e0 0x7fa39c075310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:37:30.482 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:37:30.466+0000 7fa3a325f700 1 --2- 172.21.15.52:0/350787862 >> [v2:172.21.15.52:6800/1822318384,v1:172.21.15.52:6801/1822318384] conn(0x7fa3840b2c00 0x7fa3840b5110 unknown :-1 s=CLOSED pgs=217 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:37:30.482 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:37:30.466+0000 7fa3a325f700 1 --2- 172.21.15.52:0/350787862 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fa39c108f10 0x7fa39c06e440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:37:30.482 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:37:30.466+0000 7fa3a325f700 1 -- 172.21.15.52:0/350787862 >> 172.21.15.52:0/350787862 conn(0x7fa39c100030 msgr2=0x7fa39c108680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:37:30.482 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:37:30.468+0000 7fa3a325f700 1 -- 172.21.15.52:0/350787862 shutdown_connections 2024-05-27T07:37:30.483 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:37:30.468+0000 7fa3a325f700 1 -- 172.21.15.52:0/350787862 wait complete. 2024-05-27T07:37:30.483 INFO:teuthology.orchestra.run.smithi052.stderr:pool 'cephfs_metadata' removed 2024-05-27T07:37:30.697 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:37:30 smithi105 ceph-mon[35977]: pgmap v390: 129 pgs: 56 active+clean+snaptrim_wait, 20 active+clean+snaptrim, 53 active+clean; 609 MiB data, 7.9 GiB used, 1.0 TiB / 1.0 TiB avail 2024-05-27T07:37:30.697 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:37:30 smithi105 ceph-mon[35977]: from='client.? 172.21.15.52:0/350787862' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]: dispatch 2024-05-27T07:37:30.697 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:37:30 smithi105 ceph-mon[35977]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]: dispatch 2024-05-27T07:37:30.739 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:37:30 smithi052 ceph-mon[31955]: pgmap v390: 129 pgs: 56 active+clean+snaptrim_wait, 20 active+clean+snaptrim, 53 active+clean; 609 MiB data, 7.9 GiB used, 1.0 TiB / 1.0 TiB avail 2024-05-27T07:37:30.739 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:37:30 smithi052 ceph-mon[31955]: from='client.? 172.21.15.52:0/350787862' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]: dispatch 2024-05-27T07:37:30.739 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:37:30 smithi052 ceph-mon[31955]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]: dispatch 2024-05-27T07:37:30.858 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:37:30 smithi173 ceph-mon[37000]: pgmap v390: 129 pgs: 56 active+clean+snaptrim_wait, 20 active+clean+snaptrim, 53 active+clean; 609 MiB data, 7.9 GiB used, 1.0 TiB / 1.0 TiB avail 2024-05-27T07:37:30.858 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:37:30 smithi173 ceph-mon[37000]: from='client.? 172.21.15.52:0/350787862' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]: dispatch 2024-05-27T07:37:30.859 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:37:30 smithi173 ceph-mon[37000]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]: dispatch 2024-05-27T07:37:31.292 DEBUG:teuthology.orchestra.run.smithi052:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:38f9d900c83c63799fbdbe61acc9a11b0d3554a6 shell --fsid 73df2636-1bf8-11ef-bc9b-c7b262605968 -- ceph osd pool rm cephfs_data cephfs_data --yes-i-really-really-mean-it 2024-05-27T07:37:31.736 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:37:31 smithi052 ceph-mon[31955]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]': finished 2024-05-27T07:37:31.736 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:37:31 smithi052 ceph-mon[31955]: osdmap e132: 12 total, 12 up, 12 in 2024-05-27T07:37:31.736 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:37:31 smithi052 ceph-mon[31955]: pgmap v392: 97 pgs: 55 active+clean+snaptrim_wait, 19 active+clean+snaptrim, 23 active+clean; 561 MiB data, 7.9 GiB used, 1.0 TiB / 1.0 TiB avail 2024-05-27T07:37:31.804 INFO:teuthology.orchestra.run.smithi052.stderr:Inferring config /var/lib/ceph/73df2636-1bf8-11ef-bc9b-c7b262605968/mon.a/config 2024-05-27T07:37:31.858 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:37:31 smithi173 ceph-mon[37000]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]': finished 2024-05-27T07:37:31.858 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:37:31 smithi173 ceph-mon[37000]: osdmap e132: 12 total, 12 up, 12 in 2024-05-27T07:37:31.859 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:37:31 smithi173 ceph-mon[37000]: pgmap v392: 97 pgs: 55 active+clean+snaptrim_wait, 19 active+clean+snaptrim, 23 active+clean; 561 MiB data, 7.9 GiB used, 1.0 TiB / 1.0 TiB avail 2024-05-27T07:37:31.946 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:37:31 smithi105 ceph-mon[35977]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]': finished 2024-05-27T07:37:31.946 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:37:31 smithi105 ceph-mon[35977]: osdmap e132: 12 total, 12 up, 12 in 2024-05-27T07:37:31.946 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:37:31 smithi105 ceph-mon[35977]: pgmap v392: 97 pgs: 55 active+clean+snaptrim_wait, 19 active+clean+snaptrim, 23 active+clean; 561 MiB data, 7.9 GiB used, 1.0 TiB / 1.0 TiB avail 2024-05-27T07:37:33.567 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:37:33 smithi052 ceph-mon[31955]: pgmap v393: 97 pgs: 55 active+clean+snaptrim_wait, 18 active+clean+snaptrim, 24 active+clean; 561 MiB data, 7.9 GiB used, 1.0 TiB / 1.0 TiB avail 2024-05-27T07:37:33.776 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:37:33.777+0000 7fe3fc6cf700 1 -- 172.21.15.52:0/2596148987 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fe3f4115be0 msgr2=0x7fe3f4118000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:37:33.777 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:37:33.777+0000 7fe3fc6cf700 1 --2- 172.21.15.52:0/2596148987 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fe3f4115be0 0x7fe3f4118000 secure :-1 s=READY pgs=292 cs=0 l=1 rev1=1 crypto rx=0x7fe3ec009f90 tx=0x7fe3ec00b330 comp rx=0 tx=0).stop 2024-05-27T07:37:33.777 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:37:33.780+0000 7fe3fc6cf700 1 -- 172.21.15.52:0/2596148987 shutdown_connections 2024-05-27T07:37:33.777 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:37:33.780+0000 7fe3fc6cf700 1 --2- 172.21.15.52:0/2596148987 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fe3f4115be0 0x7fe3f4118000 unknown :-1 s=CLOSED pgs=292 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:37:33.777 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:37:33.780+0000 7fe3fc6cf700 1 --2- 172.21.15.52:0/2596148987 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fe3f410b610 0x7fe3f410f940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:37:33.777 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:37:33.780+0000 7fe3fc6cf700 1 --2- 172.21.15.52:0/2596148987 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fe3f410fe80 0x7fe3f4110290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:37:33.777 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:37:33.780+0000 7fe3fc6cf700 1 -- 172.21.15.52:0/2596148987 >> 172.21.15.52:0/2596148987 conn(0x7fe3f406f980 msgr2=0x7fe3f406fd90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:37:33.777 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:37:33.781+0000 7fe3fc6cf700 1 -- 172.21.15.52:0/2596148987 shutdown_connections 2024-05-27T07:37:33.778 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:37:33.781+0000 7fe3fc6cf700 1 -- 172.21.15.52:0/2596148987 wait complete. 2024-05-27T07:37:33.779 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:37:33.782+0000 7fe3fc6cf700 1 Processor -- start 2024-05-27T07:37:33.779 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:37:33.782+0000 7fe3fc6cf700 1 -- start start 2024-05-27T07:37:33.779 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:37:33.782+0000 7fe3fc6cf700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fe3f410b610 0x7fe3f41a6080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:37:33.779 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:37:33.782+0000 7fe3fc6cf700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fe3f410fe80 0x7fe3f41a65e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:37:33.780 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:37:33.782+0000 7fe3fc6cf700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fe3f4115be0 0x7fe3f41abda0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:37:33.780 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:37:33.782+0000 7fe3fc6cf700 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x7fe3f411aba0 con 0x7fe3f410fe80 2024-05-27T07:37:33.780 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:37:33.782+0000 7fe3fc6cf700 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7fe3f411aa20 con 0x7fe3f4115be0 2024-05-27T07:37:33.780 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:37:33.782+0000 7fe3fc6cf700 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7fe3f411ad20 con 0x7fe3f410b610 2024-05-27T07:37:33.780 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:37:33.783+0000 7fe3fa46b700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fe3f410b610 0x7fe3f41a6080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:37:33.780 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:37:33.783+0000 7fe3fa46b700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fe3f410b610 0x7fe3f41a6080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.173:3300/0 says I am v2:172.21.15.52:55038/0 (socket says 172.21.15.52:55038) 2024-05-27T07:37:33.780 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:37:33.783+0000 7fe3fa46b700 1 -- 172.21.15.52:0/3267671599 learned_addr learned my addr 172.21.15.52:0/3267671599 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-05-27T07:37:33.781 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:37:33.783+0000 7fe3fa46b700 1 -- 172.21.15.52:0/3267671599 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fe3f4115be0 msgr2=0x7fe3f41abda0 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-05-27T07:37:33.781 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:37:33.783+0000 7fe3fa46b700 1 --2- 172.21.15.52:0/3267671599 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fe3f4115be0 0x7fe3f41abda0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:37:33.781 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:37:33.784+0000 7fe3fa46b700 1 -- 172.21.15.52:0/3267671599 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fe3f410fe80 msgr2=0x7fe3f41a65e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:37:33.781 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:37:33.784+0000 7fe3fa46b700 1 --2- 172.21.15.52:0/3267671599 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fe3f410fe80 0x7fe3f41a65e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:37:33.781 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:37:33.784+0000 7fe3fa46b700 1 -- 172.21.15.52:0/3267671599 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe3f41ac5d0 con 0x7fe3f410b610 2024-05-27T07:37:33.781 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:37:33.784+0000 7fe3fa46b700 1 --2- 172.21.15.52:0/3267671599 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fe3f410b610 0x7fe3f41a6080 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7fe3f000b550 tx=0x7fe3f000b910 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:37:33.782 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:37:33.786+0000 7fe3eb7fe700 1 -- 172.21.15.52:0/3267671599 <== mon.1 v2:172.21.15.173:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe3f0015020 con 0x7fe3f410b610 2024-05-27T07:37:33.782 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:37:33.786+0000 7fe3fc6cf700 1 -- 172.21.15.52:0/3267671599 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe3f41ac8c0 con 0x7fe3f410b610 2024-05-27T07:37:33.784 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:37:33.786+0000 7fe3fc6cf700 1 -- 172.21.15.52:0/3267671599 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe3f41a1410 con 0x7fe3f410b610 2024-05-27T07:37:33.784 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:37:33.786+0000 7fe3eb7fe700 1 -- 172.21.15.52:0/3267671599 <== mon.1 v2:172.21.15.173:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fe3f0011ce0 con 0x7fe3f410b610 2024-05-27T07:37:33.784 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:37:33.786+0000 7fe3eb7fe700 1 -- 172.21.15.52:0/3267671599 <== mon.1 v2:172.21.15.173:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe3f0010de0 con 0x7fe3f410b610 2024-05-27T07:37:33.786 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:37:33.790+0000 7fe3fc6cf700 1 -- 172.21.15.52:0/3267671599 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe3f40512b0 con 0x7fe3f410b610 2024-05-27T07:37:33.789 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:37:33.793+0000 7fe3eb7fe700 1 -- 172.21.15.52:0/3267671599 <== mon.1 v2:172.21.15.173:3300/0 4 ==== mgrmap(e 26) ==== 149783+0+0 (secure 0 0 0) 0x7fe3f0018020 con 0x7fe3f410b610 2024-05-27T07:37:33.790 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:37:33.794+0000 7fe3eb7fe700 1 --2- 172.21.15.52:0/3267671599 >> [v2:172.21.15.52:6800/1822318384,v1:172.21.15.52:6801/1822318384] conn(0x7fe3e00b2bf0 0x7fe3e00b5100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:37:33.791 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:37:33.794+0000 7fe3eb7fe700 1 -- 172.21.15.52:0/3267671599 <== mon.1 v2:172.21.15.173:3300/0 5 ==== osd_map(132..132 src has 1..132) ==== 9372+0+0 (secure 0 0 0) 0x7fe3f00e1620 con 0x7fe3f410b610 2024-05-27T07:37:33.791 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:37:33.795+0000 7fe3f9c6a700 1 --2- 172.21.15.52:0/3267671599 >> [v2:172.21.15.52:6800/1822318384,v1:172.21.15.52:6801/1822318384] conn(0x7fe3e00b2bf0 0x7fe3e00b5100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:37:33.795 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:37:33.796+0000 7fe3f9c6a700 1 --2- 172.21.15.52:0/3267671599 >> [v2:172.21.15.52:6800/1822318384,v1:172.21.15.52:6801/1822318384] conn(0x7fe3e00b2bf0 0x7fe3e00b5100 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7fe3e4005950 tx=0x7fe3e40058e0 comp rx=0 tx=0).ready entity=mgr.15333 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:37:33.798 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:37:33.800+0000 7fe3eb7fe700 1 -- 172.21.15.52:0/3267671599 <== mon.1 v2:172.21.15.173:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191963 (secure 0 0 0) 0x7fe3f00a8d60 con 0x7fe3f410b610 2024-05-27T07:37:33.858 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:37:33 smithi173 ceph-mon[37000]: pgmap v393: 97 pgs: 55 active+clean+snaptrim_wait, 18 active+clean+snaptrim, 24 active+clean; 561 MiB data, 7.9 GiB used, 1.0 TiB / 1.0 TiB avail 2024-05-27T07:37:33.946 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:37:33 smithi105 ceph-mon[35977]: pgmap v393: 97 pgs: 55 active+clean+snaptrim_wait, 18 active+clean+snaptrim, 24 active+clean; 561 MiB data, 7.9 GiB used, 1.0 TiB / 1.0 TiB avail 2024-05-27T07:37:34.211 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:37:34.214+0000 7fe3fc6cf700 1 -- 172.21.15.52:0/3267671599 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true} v 0) -- 0x7fe3f41a16c0 con 0x7fe3f410b610 2024-05-27T07:37:34.507 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:37:34.510+0000 7fe3eb7fe700 1 -- 172.21.15.52:0/3267671599 <== mon.1 v2:172.21.15.173:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_data' removed v133) ==== 169+0+0 (secure 0 0 0) 0x7fe3f00a8660 con 0x7fe3f410b610 2024-05-27T07:37:34.512 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:37:34.514+0000 7fe3fc6cf700 1 -- 172.21.15.52:0/3267671599 >> [v2:172.21.15.52:6800/1822318384,v1:172.21.15.52:6801/1822318384] conn(0x7fe3e00b2bf0 msgr2=0x7fe3e00b5100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:37:34.512 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:37:34.514+0000 7fe3fc6cf700 1 --2- 172.21.15.52:0/3267671599 >> [v2:172.21.15.52:6800/1822318384,v1:172.21.15.52:6801/1822318384] conn(0x7fe3e00b2bf0 0x7fe3e00b5100 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7fe3e4005950 tx=0x7fe3e40058e0 comp rx=0 tx=0).stop 2024-05-27T07:37:34.512 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:37:34.514+0000 7fe3fc6cf700 1 -- 172.21.15.52:0/3267671599 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fe3f410b610 msgr2=0x7fe3f41a6080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:37:34.512 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:37:34.514+0000 7fe3fc6cf700 1 --2- 172.21.15.52:0/3267671599 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fe3f410b610 0x7fe3f41a6080 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7fe3f000b550 tx=0x7fe3f000b910 comp rx=0 tx=0).stop 2024-05-27T07:37:34.512 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:37:34.514+0000 7fe3fc6cf700 1 -- 172.21.15.52:0/3267671599 shutdown_connections 2024-05-27T07:37:34.512 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:37:34.514+0000 7fe3fc6cf700 1 --2- 172.21.15.52:0/3267671599 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fe3f4115be0 0x7fe3f41abda0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:37:34.512 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:37:34.514+0000 7fe3fc6cf700 1 --2- 172.21.15.52:0/3267671599 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fe3f410b610 0x7fe3f41a6080 secure :-1 s=CLOSED pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7fe3f000b550 tx=0x7fe3f000b910 comp rx=0 tx=0).stop 2024-05-27T07:37:34.512 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:37:34.514+0000 7fe3fc6cf700 1 --2- 172.21.15.52:0/3267671599 >> [v2:172.21.15.52:6800/1822318384,v1:172.21.15.52:6801/1822318384] conn(0x7fe3e00b2bf0 0x7fe3e00b5100 secure :-1 s=CLOSED pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7fe3e4005950 tx=0x7fe3e40058e0 comp rx=0 tx=0).stop 2024-05-27T07:37:34.512 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:37:34.514+0000 7fe3fc6cf700 1 --2- 172.21.15.52:0/3267671599 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fe3f410fe80 0x7fe3f41a65e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:37:34.513 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:37:34.514+0000 7fe3fc6cf700 1 -- 172.21.15.52:0/3267671599 >> 172.21.15.52:0/3267671599 conn(0x7fe3f406f980 msgr2=0x7fe3f410e920 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:37:34.513 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:37:34.515+0000 7fe3fc6cf700 1 -- 172.21.15.52:0/3267671599 shutdown_connections 2024-05-27T07:37:34.513 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:37:34.515+0000 7fe3fc6cf700 1 -- 172.21.15.52:0/3267671599 wait complete. 2024-05-27T07:37:34.526 INFO:teuthology.orchestra.run.smithi052.stderr:pool 'cephfs_data' removed 2024-05-27T07:37:34.858 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:37:34 smithi173 ceph-mon[37000]: from='client.? 172.21.15.52:0/3267671599' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]: dispatch 2024-05-27T07:37:34.858 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:37:34 smithi173 ceph-mon[37000]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]: dispatch 2024-05-27T07:37:34.946 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:37:34 smithi105 ceph-mon[35977]: from='client.? 172.21.15.52:0/3267671599' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]: dispatch 2024-05-27T07:37:34.946 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:37:34 smithi105 ceph-mon[35977]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]: dispatch 2024-05-27T07:37:34.985 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:37:34 smithi052 ceph-mon[31955]: from='client.? 172.21.15.52:0/3267671599' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]: dispatch 2024-05-27T07:37:34.986 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:37:34 smithi052 ceph-mon[31955]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]: dispatch 2024-05-27T07:37:35.549 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:37:35 smithi052 ceph-mon[31955]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]': finished 2024-05-27T07:37:35.549 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:37:35 smithi052 ceph-mon[31955]: osdmap e133: 12 total, 12 up, 12 in 2024-05-27T07:37:35.549 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:37:35 smithi052 ceph-mon[31955]: pgmap v395: 33 pgs: 17 active+clean+snaptrim_wait, 14 active+clean+snaptrim, 2 active+clean; 560 MiB data, 7.9 GiB used, 1.0 TiB / 1.0 TiB avail 2024-05-27T07:37:35.602 DEBUG:teuthology.orchestra.run.smithi052:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:38f9d900c83c63799fbdbe61acc9a11b0d3554a6 shell --fsid 73df2636-1bf8-11ef-bc9b-c7b262605968 -- ceph osd pool rm cephfs_data_ec cephfs_data_ec --yes-i-really-really-mean-it 2024-05-27T07:37:35.863 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:37:35 smithi173 ceph-mon[37000]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]': finished 2024-05-27T07:37:35.863 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:37:35 smithi173 ceph-mon[37000]: osdmap e133: 12 total, 12 up, 12 in 2024-05-27T07:37:35.863 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:37:35 smithi173 ceph-mon[37000]: pgmap v395: 33 pgs: 17 active+clean+snaptrim_wait, 14 active+clean+snaptrim, 2 active+clean; 560 MiB data, 7.9 GiB used, 1.0 TiB / 1.0 TiB avail 2024-05-27T07:37:35.946 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:37:35 smithi105 ceph-mon[35977]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]': finished 2024-05-27T07:37:35.946 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:37:35 smithi105 ceph-mon[35977]: osdmap e133: 12 total, 12 up, 12 in 2024-05-27T07:37:35.946 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:37:35 smithi105 ceph-mon[35977]: pgmap v395: 33 pgs: 17 active+clean+snaptrim_wait, 14 active+clean+snaptrim, 2 active+clean; 560 MiB data, 7.9 GiB used, 1.0 TiB / 1.0 TiB avail 2024-05-27T07:37:36.076 INFO:teuthology.orchestra.run.smithi052.stderr:Inferring config /var/lib/ceph/73df2636-1bf8-11ef-bc9b-c7b262605968/mon.a/config 2024-05-27T07:37:37.510 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:37:37 smithi052 ceph-mon[31955]: pgmap v396: 33 pgs: 17 active+clean+snaptrim_wait, 13 active+clean+snaptrim, 3 active+clean; 559 MiB data, 8.0 GiB used, 1.0 TiB / 1.0 TiB avail 2024-05-27T07:37:37.858 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:37:37 smithi173 ceph-mon[37000]: pgmap v396: 33 pgs: 17 active+clean+snaptrim_wait, 13 active+clean+snaptrim, 3 active+clean; 559 MiB data, 8.0 GiB used, 1.0 TiB / 1.0 TiB avail 2024-05-27T07:37:37.946 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:37:37 smithi105 ceph-mon[35977]: pgmap v396: 33 pgs: 17 active+clean+snaptrim_wait, 13 active+clean+snaptrim, 3 active+clean; 559 MiB data, 8.0 GiB used, 1.0 TiB / 1.0 TiB avail 2024-05-27T07:37:38.041 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:37:38.043+0000 7f59b4d8c700 1 -- 172.21.15.52:0/1116063476 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f59b010b100 msgr2=0x7f59b011d3e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:37:38.042 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:37:38.043+0000 7f59b4d8c700 1 --2- 172.21.15.52:0/1116063476 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f59b010b100 0x7f59b011d3e0 secure :-1 s=READY pgs=776 cs=0 l=1 rev1=1 crypto rx=0x7f5998009990 tx=0x7f5998009ca0 comp rx=0 tx=0).stop 2024-05-27T07:37:38.047 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:37:38.050+0000 7f59b4d8c700 1 -- 172.21.15.52:0/1116063476 shutdown_connections 2024-05-27T07:37:38.048 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:37:38.050+0000 7f59b4d8c700 1 --2- 172.21.15.52:0/1116063476 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f59b011d920 0x7f59b011fd40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:37:38.048 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:37:38.050+0000 7f59b4d8c700 1 --2- 172.21.15.52:0/1116063476 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f59b010a7b0 0x7f59b010abc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:37:38.048 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:37:38.050+0000 7f59b4d8c700 1 --2- 172.21.15.52:0/1116063476 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f59b010b100 0x7f59b011d3e0 unknown :-1 s=CLOSED pgs=776 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:37:38.048 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:37:38.050+0000 7f59b4d8c700 1 -- 172.21.15.52:0/1116063476 >> 172.21.15.52:0/1116063476 conn(0x7f59b006f280 msgr2=0x7f59b006f6c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:37:38.050 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:37:38.051+0000 7f59b4d8c700 1 -- 172.21.15.52:0/1116063476 shutdown_connections 2024-05-27T07:37:38.051 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:37:38.054+0000 7f59b4d8c700 1 -- 172.21.15.52:0/1116063476 wait complete. 2024-05-27T07:37:38.051 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:37:38.055+0000 7f59b4d8c700 1 Processor -- start 2024-05-27T07:37:38.054 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:37:38.056+0000 7f59b4d8c700 1 -- start start 2024-05-27T07:37:38.054 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:37:38.056+0000 7f59b4d8c700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f59b010a7b0 0x7f59b01af730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:37:38.054 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:37:38.056+0000 7f59b4d8c700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f59b010b100 0x7f59b01afc70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:37:38.054 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:37:38.056+0000 7f59b4d8c700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f59b011d920 0x7f59b01a9950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:37:38.055 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:37:38.056+0000 7f59b4d8c700 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 -- 0x7f59b0123150 con 0x7f59b011d920 2024-05-27T07:37:38.055 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:37:38.056+0000 7f59b4d8c700 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f59b0122fd0 con 0x7f59b010a7b0 2024-05-27T07:37:38.055 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:37:38.056+0000 7f59b4d8c700 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f59b01232d0 con 0x7f59b010b100 2024-05-27T07:37:38.055 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:37:38.057+0000 7f59affff700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f59b011d920 0x7f59b01a9950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:37:38.055 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:37:38.057+0000 7f59affff700 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f59b011d920 0x7f59b01a9950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.52:49962/0 (socket says 172.21.15.52:49962) 2024-05-27T07:37:38.055 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:37:38.057+0000 7f59affff700 1 -- 172.21.15.52:0/3853904699 learned_addr learned my addr 172.21.15.52:0/3853904699 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-05-27T07:37:38.055 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:37:38.057+0000 7f59af7fe700 1 --2- 172.21.15.52:0/3853904699 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f59b010a7b0 0x7f59b01af730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:37:38.055 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:37:38.057+0000 7f59affff700 1 -- 172.21.15.52:0/3853904699 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f59b010b100 msgr2=0x7f59b01afc70 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-05-27T07:37:38.055 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:37:38.057+0000 7f59affff700 1 --2- 172.21.15.52:0/3853904699 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f59b010b100 0x7f59b01afc70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:37:38.055 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:37:38.057+0000 7f59affff700 1 -- 172.21.15.52:0/3853904699 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f59b010a7b0 msgr2=0x7f59b01af730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:37:38.055 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:37:38.057+0000 7f59affff700 1 --2- 172.21.15.52:0/3853904699 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f59b010a7b0 0x7f59b01af730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:37:38.055 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:37:38.057+0000 7f59affff700 1 -- 172.21.15.52:0/3853904699 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f59b01aa020 con 0x7f59b011d920 2024-05-27T07:37:38.056 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:37:38.057+0000 7f59af7fe700 1 --2- 172.21.15.52:0/3853904699 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f59b010a7b0 0x7f59b01af730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-27T07:37:38.056 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:37:38.058+0000 7f59affff700 1 --2- 172.21.15.52:0/3853904699 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f59b011d920 0x7f59b01a9950 secure :-1 s=READY pgs=777 cs=0 l=1 rev1=1 crypto rx=0x7f59a800ada0 tx=0x7f59a800c330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:37:38.056 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:37:38.059+0000 7f59acff9700 1 -- 172.21.15.52:0/3853904699 <== mon.0 v2:172.21.15.52:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f59a8017020 con 0x7f59b011d920 2024-05-27T07:37:38.056 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:37:38.060+0000 7f59b4d8c700 1 -- 172.21.15.52:0/3853904699 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f59b01aa310 con 0x7f59b011d920 2024-05-27T07:37:38.056 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:37:38.060+0000 7f59b4d8c700 1 -- 172.21.15.52:0/3853904699 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f59b0112bc0 con 0x7f59b011d920 2024-05-27T07:37:38.057 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:37:38.060+0000 7f59acff9700 1 -- 172.21.15.52:0/3853904699 <== mon.0 v2:172.21.15.52:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f59a800ce30 con 0x7f59b011d920 2024-05-27T07:37:38.057 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:37:38.060+0000 7f59acff9700 1 -- 172.21.15.52:0/3853904699 <== mon.0 v2:172.21.15.52:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f59a8005350 con 0x7f59b011d920 2024-05-27T07:37:38.059 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:37:38.063+0000 7f59b4d8c700 1 -- 172.21.15.52:0/3853904699 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5990005330 con 0x7f59b011d920 2024-05-27T07:37:38.066 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:37:38.068+0000 7f59acff9700 1 -- 172.21.15.52:0/3853904699 <== mon.0 v2:172.21.15.52:3300/0 4 ==== mgrmap(e 26) ==== 149783+0+0 (secure 0 0 0) 0x7f59a801a070 con 0x7f59b011d920 2024-05-27T07:37:38.066 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:37:38.069+0000 7f59acff9700 1 --2- 172.21.15.52:0/3853904699 >> [v2:172.21.15.52:6800/1822318384,v1:172.21.15.52:6801/1822318384] conn(0x7f599c0b2aa0 0x7f599c0b4fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:37:38.066 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:37:38.070+0000 7f59af7fe700 1 --2- 172.21.15.52:0/3853904699 >> [v2:172.21.15.52:6800/1822318384,v1:172.21.15.52:6801/1822318384] conn(0x7f599c0b2aa0 0x7f599c0b4fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:37:38.066 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:37:38.070+0000 7f59acff9700 1 -- 172.21.15.52:0/3853904699 <== mon.0 v2:172.21.15.52:3300/0 5 ==== osd_map(133..133 src has 1..133) ==== 8842+0+0 (secure 0 0 0) 0x7f59a80a4f80 con 0x7f59b011d920 2024-05-27T07:37:38.077 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:37:38.080+0000 7f59acff9700 1 -- 172.21.15.52:0/3853904699 <== mon.0 v2:172.21.15.52:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191963 (secure 0 0 0) 0x7f59a804fdf0 con 0x7f59b011d920 2024-05-27T07:37:38.084 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:37:38.087+0000 7f59af7fe700 1 --2- 172.21.15.52:0/3853904699 >> [v2:172.21.15.52:6800/1822318384,v1:172.21.15.52:6801/1822318384] conn(0x7f599c0b2aa0 0x7f599c0b4fb0 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7f59b01aaa90 tx=0x7f59a0009190 comp rx=0 tx=0).ready entity=mgr.15333 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:37:38.494 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:37:38.496+0000 7f59b4d8c700 1 -- 172.21.15.52:0/3853904699 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_data_ec", "pool2": "cephfs_data_ec", "yes_i_really_really_mean_it": true} v 0) -- 0x7f5990005620 con 0x7f59b011d920 2024-05-27T07:37:38.547 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:37:38.551+0000 7f59acff9700 1 -- 172.21.15.52:0/3853904699 <== mon.0 v2:172.21.15.52:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_data_ec", "pool2": "cephfs_data_ec", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_data_ec' removed v134) ==== 178+0+0 (secure 0 0 0) 0x7f59a804fdf0 con 0x7f59b011d920 2024-05-27T07:37:38.553 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:37:38.556+0000 7f59b4d8c700 1 -- 172.21.15.52:0/3853904699 >> [v2:172.21.15.52:6800/1822318384,v1:172.21.15.52:6801/1822318384] conn(0x7f599c0b2aa0 msgr2=0x7f599c0b4fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:37:38.553 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:37:38.556+0000 7f59b4d8c700 1 --2- 172.21.15.52:0/3853904699 >> [v2:172.21.15.52:6800/1822318384,v1:172.21.15.52:6801/1822318384] conn(0x7f599c0b2aa0 0x7f599c0b4fb0 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7f59b01aaa90 tx=0x7f59a0009190 comp rx=0 tx=0).stop 2024-05-27T07:37:38.553 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:37:38.556+0000 7f59b4d8c700 1 -- 172.21.15.52:0/3853904699 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f59b011d920 msgr2=0x7f59b01a9950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:37:38.553 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:37:38.556+0000 7f59b4d8c700 1 --2- 172.21.15.52:0/3853904699 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f59b011d920 0x7f59b01a9950 secure :-1 s=READY pgs=777 cs=0 l=1 rev1=1 crypto rx=0x7f59a800ada0 tx=0x7f59a800c330 comp rx=0 tx=0).stop 2024-05-27T07:37:38.553 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:37:38.557+0000 7f59b4d8c700 1 -- 172.21.15.52:0/3853904699 shutdown_connections 2024-05-27T07:37:38.554 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:37:38.557+0000 7f59b4d8c700 1 --2- 172.21.15.52:0/3853904699 >> [v2:172.21.15.52:6800/1822318384,v1:172.21.15.52:6801/1822318384] conn(0x7f599c0b2aa0 0x7f599c0b4fb0 unknown :-1 s=CLOSED pgs=219 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:37:38.554 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:37:38.557+0000 7f59b4d8c700 1 --2- 172.21.15.52:0/3853904699 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f59b011d920 0x7f59b01a9950 unknown :-1 s=CLOSED pgs=777 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:37:38.554 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:37:38.557+0000 7f59b4d8c700 1 --2- 172.21.15.52:0/3853904699 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f59b010a7b0 0x7f59b01af730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:37:38.554 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:37:38.557+0000 7f59b4d8c700 1 --2- 172.21.15.52:0/3853904699 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f59b010b100 0x7f59b01afc70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:37:38.554 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:37:38.557+0000 7f59b4d8c700 1 -- 172.21.15.52:0/3853904699 >> 172.21.15.52:0/3853904699 conn(0x7f59b006f280 msgr2=0x7f59b011b3d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:37:38.554 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:37:38.557+0000 7f59b4d8c700 1 -- 172.21.15.52:0/3853904699 shutdown_connections 2024-05-27T07:37:38.554 INFO:teuthology.orchestra.run.smithi052.stderr:2024-05-27T07:37:38.558+0000 7f59b4d8c700 1 -- 172.21.15.52:0/3853904699 wait complete. 2024-05-27T07:37:38.558 INFO:teuthology.orchestra.run.smithi052.stderr:pool 'cephfs_data_ec' removed 2024-05-27T07:37:38.858 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:37:38 smithi173 ceph-mon[37000]: from='client.? 172.21.15.52:0/3853904699' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "cephfs_data_ec", "pool2": "cephfs_data_ec", "yes_i_really_really_mean_it": true}]: dispatch 2024-05-27T07:37:38.946 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:37:38 smithi105 ceph-mon[35977]: from='client.? 172.21.15.52:0/3853904699' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "cephfs_data_ec", "pool2": "cephfs_data_ec", "yes_i_really_really_mean_it": true}]: dispatch 2024-05-27T07:37:38.986 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:37:38 smithi052 ceph-mon[31955]: from='client.? 172.21.15.52:0/3853904699' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "cephfs_data_ec", "pool2": "cephfs_data_ec", "yes_i_really_really_mean_it": true}]: dispatch 2024-05-27T07:37:39.858 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:37:39 smithi173 ceph-mon[37000]: from='client.? 172.21.15.52:0/3853904699' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "cephfs_data_ec", "pool2": "cephfs_data_ec", "yes_i_really_really_mean_it": true}]': finished 2024-05-27T07:37:39.859 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:37:39 smithi173 ceph-mon[37000]: osdmap e134: 12 total, 12 up, 12 in 2024-05-27T07:37:39.859 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:37:39 smithi173 ceph-mon[37000]: pgmap v398: 1 pgs: 1 active+clean; 577 KiB data, 8.0 GiB used, 1.0 TiB / 1.0 TiB avail 2024-05-27T07:37:39.946 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:37:39 smithi105 ceph-mon[35977]: from='client.? 172.21.15.52:0/3853904699' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "cephfs_data_ec", "pool2": "cephfs_data_ec", "yes_i_really_really_mean_it": true}]': finished 2024-05-27T07:37:39.947 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:37:39 smithi105 ceph-mon[35977]: osdmap e134: 12 total, 12 up, 12 in 2024-05-27T07:37:39.947 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:37:39 smithi105 ceph-mon[35977]: pgmap v398: 1 pgs: 1 active+clean; 577 KiB data, 8.0 GiB used, 1.0 TiB / 1.0 TiB avail 2024-05-27T07:37:39.977 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:37:39 smithi052 ceph-mon[31955]: from='client.? 172.21.15.52:0/3853904699' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "cephfs_data_ec", "pool2": "cephfs_data_ec", "yes_i_really_really_mean_it": true}]': finished 2024-05-27T07:37:39.977 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:37:39 smithi052 ceph-mon[31955]: osdmap e134: 12 total, 12 up, 12 in 2024-05-27T07:37:39.977 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:37:39 smithi052 ceph-mon[31955]: pgmap v398: 1 pgs: 1 active+clean; 577 KiB data, 8.0 GiB used, 1.0 TiB / 1.0 TiB avail 2024-05-27T07:37:40.020 DEBUG:teuthology.orchestra.run.smithi052:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-05-27T07:37:40.061 DEBUG:teuthology.orchestra.run.smithi105:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-05-27T07:37:40.161 DEBUG:teuthology.orchestra.run.smithi173:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-05-27T07:37:40.188 INFO:tasks.cephadm:Cleaning up testdir ceph.* files... 2024-05-27T07:37:40.188 DEBUG:teuthology.orchestra.run.smithi052:> rm -f /home/ubuntu/cephtest/seed.ceph.conf /home/ubuntu/cephtest/ceph.pub 2024-05-27T07:37:40.215 DEBUG:teuthology.orchestra.run.smithi105:> rm -f /home/ubuntu/cephtest/seed.ceph.conf /home/ubuntu/cephtest/ceph.pub 2024-05-27T07:37:40.230 DEBUG:teuthology.orchestra.run.smithi173:> rm -f /home/ubuntu/cephtest/seed.ceph.conf /home/ubuntu/cephtest/ceph.pub 2024-05-27T07:37:40.245 INFO:tasks.cephadm:Stopping all daemons... 2024-05-27T07:37:40.245 INFO:tasks.cephadm.mon.a:Stopping mon.a... 2024-05-27T07:37:40.245 DEBUG:teuthology.orchestra.run.smithi052:> sudo systemctl stop ceph-73df2636-1bf8-11ef-bc9b-c7b262605968@mon.a 2024-05-27T07:37:40.486 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:37:40 smithi052 systemd[1]: Stopping Ceph mon.a for 73df2636-1bf8-11ef-bc9b-c7b262605968... 2024-05-27T07:37:40.986 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:37:40 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mon-a[31950]: 2024-05-27T07:37:40.677+0000 7f4aa3222700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.a -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-05-27T07:37:40.986 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:37:40 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mon-a[31950]: 2024-05-27T07:37:40.677+0000 7f4aa3222700 -1 mon.a@0(leader) e3 *** Got Signal Terminated *** 2024-05-27T07:37:40.986 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:37:40 smithi052 podman[118777]: 2024-05-27 07:37:40.738750229 +0000 UTC m=+0.190686740 container died 2dc34a76e3c22634d5163db18519ee0c400f5934e46c39548ccebaef4627c41c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:38f9d900c83c63799fbdbe61acc9a11b0d3554a6, name=ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mon-a, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, ceph=True, GIT_CLEAN=True, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240521) 2024-05-27T07:37:41.408 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:37:41 smithi052 podman[118777]: 2024-05-27 07:37:41.110409466 +0000 UTC m=+0.562345955 container cleanup 2dc34a76e3c22634d5163db18519ee0c400f5934e46c39548ccebaef4627c41c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:38f9d900c83c63799fbdbe61acc9a11b0d3554a6, name=ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mon-a, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240521, ceph=True, maintainer=Guillaume Abrioux , RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image) 2024-05-27T07:37:41.408 INFO:journalctl@ceph.mon.a.smithi052.stdout:May 27 07:37:41 smithi052 bash[118777]: ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mon-a 2024-05-27T07:37:41.592 DEBUG:teuthology.orchestra.run.smithi052:> sudo pkill -f 'journalctl -f -n 0 -u ceph-73df2636-1bf8-11ef-bc9b-c7b262605968@mon.a.service' 2024-05-27T07:37:41.642 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-05-27T07:37:41.642 INFO:tasks.cephadm.mon.a:Stopped mon.a 2024-05-27T07:37:41.642 INFO:tasks.cephadm.mon.c:Stopping mon.b... 2024-05-27T07:37:41.643 DEBUG:teuthology.orchestra.run.smithi105:> sudo systemctl stop ceph-73df2636-1bf8-11ef-bc9b-c7b262605968@mon.b 2024-05-27T07:37:41.946 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:37:41 smithi105 systemd[1]: Stopping Ceph mon.b for 73df2636-1bf8-11ef-bc9b-c7b262605968... 2024-05-27T07:37:42.446 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:37:41 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mon-b[35973]: 2024-05-27T07:37:41.974+0000 7f919ed1a700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.b -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-05-27T07:37:42.447 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:37:41 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mon-b[35973]: 2024-05-27T07:37:41.974+0000 7f919ed1a700 -1 mon.b@2(peon) e3 *** Got Signal Terminated *** 2024-05-27T07:37:42.946 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:37:42 smithi105 podman[71807]: 2024-05-27 07:37:42.373177359 +0000 UTC m=+0.538161761 container died ac140bd844088e241552e0adb16cd5f33c4259c1c6d8721e0edfbb1ae48164bf (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mon-b, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, ceph=True, org.label-schema.build-date=20240521, CEPH_POINT_RELEASE=) 2024-05-27T07:37:45.440 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:37:45 smithi105 podman[71807]: 2024-05-27 07:37:45.119459925 +0000 UTC m=+3.284444326 container cleanup ac140bd844088e241552e0adb16cd5f33c4259c1c6d8721e0edfbb1ae48164bf (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mon-b, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , ceph=True, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240521, org.label-schema.license=GPLv2) 2024-05-27T07:37:45.440 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:37:45 smithi105 bash[71807]: ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mon-b 2024-05-27T07:37:45.440 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 27 07:37:45 smithi105 podman[71819]: 2024-05-27 07:37:45.329461414 +0000 UTC m=+2.874785899 container remove ac140bd844088e241552e0adb16cd5f33c4259c1c6d8721e0edfbb1ae48164bf (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mon-b, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, org.label-schema.license=GPLv2, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, org.label-schema.build-date=20240521, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.vendor=CentOS) 2024-05-27T07:37:45.457 DEBUG:teuthology.orchestra.run.smithi105:> sudo pkill -f 'journalctl -f -n 0 -u ceph-73df2636-1bf8-11ef-bc9b-c7b262605968@mon.b.service' 2024-05-27T07:37:45.500 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-05-27T07:37:45.501 INFO:tasks.cephadm.mon.c:Stopped mon.b 2024-05-27T07:37:45.501 INFO:tasks.cephadm.mon.c:Stopping mon.c... 2024-05-27T07:37:45.501 DEBUG:teuthology.orchestra.run.smithi173:> sudo systemctl stop ceph-73df2636-1bf8-11ef-bc9b-c7b262605968@mon.c 2024-05-27T07:37:45.859 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:37:45 smithi173 systemd[1]: Stopping Ceph mon.c for 73df2636-1bf8-11ef-bc9b-c7b262605968... 2024-05-27T07:37:46.824 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:37:46 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mon-c[36996]: 2024-05-27T07:37:46.443+0000 7fb7755ec700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.c -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-05-27T07:37:46.824 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:37:46 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mon-c[36996]: 2024-05-27T07:37:46.443+0000 7fb7755ec700 -1 mon.c@1(peon) e3 *** Got Signal Terminated *** 2024-05-27T07:37:46.825 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:37:46 smithi173 podman[75410]: 2024-05-27 07:37:46.525440243 +0000 UTC m=+0.817569893 container died de9961ba152193f89fc37e84f5f5e9e9c83b488327cbf64a307dcc95c6bd88d9 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mon-c, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, org.label-schema.build-date=20240521, org.label-schema.schema-version=1.0, ceph=True, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, org.label-schema.license=GPLv2) 2024-05-27T07:37:47.583 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:37:47 smithi173 podman[75410]: 2024-05-27 07:37:47.277517472 +0000 UTC m=+1.569647122 container cleanup de9961ba152193f89fc37e84f5f5e9e9c83b488327cbf64a307dcc95c6bd88d9 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mon-c, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, maintainer=Guillaume Abrioux , RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240521) 2024-05-27T07:37:47.584 INFO:journalctl@ceph.mon.c.smithi173.stdout:May 27 07:37:47 smithi173 bash[75410]: ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mon-c 2024-05-27T07:37:47.775 DEBUG:teuthology.orchestra.run.smithi173:> sudo pkill -f 'journalctl -f -n 0 -u ceph-73df2636-1bf8-11ef-bc9b-c7b262605968@mon.c.service' 2024-05-27T07:37:47.819 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-05-27T07:37:47.820 INFO:tasks.cephadm.mon.c:Stopped mon.c 2024-05-27T07:37:47.820 INFO:tasks.cephadm.mgr.x:Stopping mgr.x... 2024-05-27T07:37:47.820 DEBUG:teuthology.orchestra.run.smithi052:> sudo systemctl stop ceph-73df2636-1bf8-11ef-bc9b-c7b262605968@mgr.x 2024-05-27T07:37:48.185 INFO:journalctl@ceph.mgr.x.smithi052.stdout:May 27 07:37:47 smithi052 systemd[1]: Stopping Ceph mgr.x for 73df2636-1bf8-11ef-bc9b-c7b262605968... 2024-05-27T07:37:48.486 INFO:journalctl@ceph.mgr.x.smithi052.stdout:May 27 07:37:48 smithi052 podman[118897]: 2024-05-27 07:37:48.198163325 +0000 UTC m=+0.175166433 container died 706e4038639624b167ca28ab0f525583a0c585bfd1afd594ada1c8c97e04d88a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:38f9d900c83c63799fbdbe61acc9a11b0d3554a6, name=ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mgr-x, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, GIT_BRANCH=HEAD, GIT_CLEAN=True, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, io.buildah.version=1.33.5, org.label-schema.build-date=20240521) 2024-05-27T07:37:48.986 INFO:journalctl@ceph.mgr.x.smithi052.stdout:May 27 07:37:48 smithi052 podman[118897]: 2024-05-27 07:37:48.745593787 +0000 UTC m=+0.722596884 container cleanup 706e4038639624b167ca28ab0f525583a0c585bfd1afd594ada1c8c97e04d88a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:38f9d900c83c63799fbdbe61acc9a11b0d3554a6, name=ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mgr-x, GIT_CLEAN=True, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, org.label-schema.build-date=20240521, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90) 2024-05-27T07:37:48.986 INFO:journalctl@ceph.mgr.x.smithi052.stdout:May 27 07:37:48 smithi052 bash[118897]: ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mgr-x 2024-05-27T07:37:49.222 DEBUG:teuthology.orchestra.run.smithi052:> sudo pkill -f 'journalctl -f -n 0 -u ceph-73df2636-1bf8-11ef-bc9b-c7b262605968@mgr.x.service' 2024-05-27T07:37:49.258 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-05-27T07:37:49.258 INFO:tasks.cephadm.mgr.x:Stopped mgr.x 2024-05-27T07:37:49.258 INFO:tasks.cephadm.mgr.y:Stopping mgr.y... 2024-05-27T07:37:49.258 DEBUG:teuthology.orchestra.run.smithi105:> sudo systemctl stop ceph-73df2636-1bf8-11ef-bc9b-c7b262605968@mgr.y 2024-05-27T07:37:49.650 INFO:journalctl@ceph.mgr.y.smithi105.stdout:May 27 07:37:49 smithi105 systemd[1]: Stopping Ceph mgr.y for 73df2636-1bf8-11ef-bc9b-c7b262605968... 2024-05-27T07:37:49.946 INFO:journalctl@ceph.mgr.y.smithi105.stdout:May 27 07:37:49 smithi105 podman[71931]: 2024-05-27 07:37:49.654571528 +0000 UTC m=+0.211190769 container died 09be672cee56cee4e1118bf17007a09c0cd0ef66c3f1e03c903f912c131fe716 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mgr-y, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, ceph=True, GIT_CLEAN=True, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, CEPH_POINT_RELEASE=, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, org.label-schema.build-date=20240521, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux ) 2024-05-27T07:37:50.446 INFO:journalctl@ceph.mgr.y.smithi105.stdout:May 27 07:37:50 smithi105 podman[71931]: 2024-05-27 07:37:50.141669519 +0000 UTC m=+0.698288761 container cleanup 09be672cee56cee4e1118bf17007a09c0cd0ef66c3f1e03c903f912c131fe716 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mgr-y, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, GIT_CLEAN=True, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240521, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD) 2024-05-27T07:37:50.447 INFO:journalctl@ceph.mgr.y.smithi105.stdout:May 27 07:37:50 smithi105 bash[71931]: ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mgr-y 2024-05-27T07:37:51.440 DEBUG:teuthology.orchestra.run.smithi105:> sudo pkill -f 'journalctl -f -n 0 -u ceph-73df2636-1bf8-11ef-bc9b-c7b262605968@mgr.y.service' 2024-05-27T07:37:51.475 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-05-27T07:37:51.475 INFO:tasks.cephadm.mgr.y:Stopped mgr.y 2024-05-27T07:37:51.475 INFO:tasks.cephadm.mgr.z:Stopping mgr.z... 2024-05-27T07:37:51.475 DEBUG:teuthology.orchestra.run.smithi173:> sudo systemctl stop ceph-73df2636-1bf8-11ef-bc9b-c7b262605968@mgr.z 2024-05-27T07:37:51.858 INFO:journalctl@ceph.mgr.z.smithi173.stdout:May 27 07:37:51 smithi173 systemd[1]: Stopping Ceph mgr.z for 73df2636-1bf8-11ef-bc9b-c7b262605968... 2024-05-27T07:37:53.109 INFO:journalctl@ceph.mgr.z.smithi173.stdout:May 27 07:37:52 smithi173 podman[75546]: 2024-05-27 07:37:52.681542339 +0000 UTC m=+0.985057038 container died db6f11578c162527f37009af50bca17a42cbf6043df42591b22f1f62ea94ee02 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mgr-z, CEPH_POINT_RELEASE=, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, org.label-schema.vendor=CentOS, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, org.label-schema.build-date=20240521, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5) 2024-05-27T07:37:53.840 INFO:journalctl@ceph.mgr.z.smithi173.stdout:May 27 07:37:53 smithi173 podman[75546]: 2024-05-27 07:37:53.592697586 +0000 UTC m=+1.896212280 container cleanup db6f11578c162527f37009af50bca17a42cbf6043df42591b22f1f62ea94ee02 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mgr-z, GIT_BRANCH=HEAD, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, org.label-schema.build-date=20240521, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, ceph=True, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2) 2024-05-27T07:37:53.840 INFO:journalctl@ceph.mgr.z.smithi173.stdout:May 27 07:37:53 smithi173 bash[75546]: ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mgr-z 2024-05-27T07:37:53.981 DEBUG:teuthology.orchestra.run.smithi173:> sudo pkill -f 'journalctl -f -n 0 -u ceph-73df2636-1bf8-11ef-bc9b-c7b262605968@mgr.z.service' 2024-05-27T07:37:54.015 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-05-27T07:37:54.015 INFO:tasks.cephadm.mgr.z:Stopped mgr.z 2024-05-27T07:37:54.015 INFO:tasks.cephadm.osd.0:Stopping osd.0... 2024-05-27T07:37:54.016 DEBUG:teuthology.orchestra.run.smithi052:> sudo systemctl stop ceph-73df2636-1bf8-11ef-bc9b-c7b262605968@osd.0 2024-05-27T07:37:54.326 INFO:journalctl@ceph.osd.0.smithi052.stdout:May 27 07:37:54 smithi052 systemd[1]: Stopping Ceph osd.0 for 73df2636-1bf8-11ef-bc9b-c7b262605968... 2024-05-27T07:37:54.736 INFO:journalctl@ceph.osd.0.smithi052.stdout:May 27 07:37:54 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-osd-0[47137]: 2024-05-27T07:37:54.329+0000 7f6c9d69e700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.0 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-05-27T07:37:54.736 INFO:journalctl@ceph.osd.0.smithi052.stdout:May 27 07:37:54 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-osd-0[47137]: 2024-05-27T07:37:54.329+0000 7f6c9d69e700 -1 osd.0 134 *** Got signal Terminated *** 2024-05-27T07:37:54.736 INFO:journalctl@ceph.osd.0.smithi052.stdout:May 27 07:37:54 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-osd-0[47137]: 2024-05-27T07:37:54.329+0000 7f6c9d69e700 -1 osd.0 134 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-05-27T07:37:59.986 INFO:journalctl@ceph.osd.0.smithi052.stdout:May 27 07:37:59 smithi052 podman[119018]: 2024-05-27 07:37:59.798312206 +0000 UTC m=+5.581835104 container died c5b7c154820231caa641deccbbdc7149731b32833afe5f6cdde8604f9cbbdb3f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-osd-0, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, org.label-schema.build-date=20240521, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, ceph=True, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0) 2024-05-27T07:38:00.486 INFO:journalctl@ceph.osd.0.smithi052.stdout:May 27 07:38:00 smithi052 podman[119018]: 2024-05-27 07:38:00.213177697 +0000 UTC m=+5.996700585 container cleanup c5b7c154820231caa641deccbbdc7149731b32833afe5f6cdde8604f9cbbdb3f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-osd-0, maintainer=Guillaume Abrioux , GIT_CLEAN=True, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, ceph=True, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240521, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2) 2024-05-27T07:38:00.486 INFO:journalctl@ceph.osd.0.smithi052.stdout:May 27 07:38:00 smithi052 bash[119018]: ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-osd-0 2024-05-27T07:38:00.986 INFO:journalctl@ceph.osd.0.smithi052.stdout:May 27 07:38:00 smithi052 podman[119048]: 2024-05-27 07:38:00.570298104 +0000 UTC m=+0.763468438 container remove c5b7c154820231caa641deccbbdc7149731b32833afe5f6cdde8604f9cbbdb3f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-osd-0, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, maintainer=Guillaume Abrioux , GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240521) 2024-05-27T07:38:01.265 INFO:journalctl@ceph.osd.0.smithi052.stdout:May 27 07:38:00 smithi052 podman[119137]: 2024-05-27 07:38:00.900353918 +0000 UTC m=+0.017754414 image pull e36ea4fc8a5807e8d2701370e350a0de3856538797f7fba64cfce7e344bd6799 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6 2024-05-27T07:38:01.265 INFO:journalctl@ceph.osd.0.smithi052.stdout:May 27 07:38:01 smithi052 podman[119137]: 2024-05-27 07:38:01.024432062 +0000 UTC m=+0.141832562 container create 0187357a004219755ad1ce1d207a9704a9122bc509fcb27c924f9d4126cf0706 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-osd-0-deactivate, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240521, GIT_CLEAN=True, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, org.label-schema.schema-version=1.0, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image) 2024-05-27T07:38:01.265 INFO:journalctl@ceph.osd.0.smithi052.stdout:May 27 07:38:01 smithi052 podman[119137]: 2024-05-27 07:38:01.268930008 +0000 UTC m=+0.386330489 container init 0187357a004219755ad1ce1d207a9704a9122bc509fcb27c924f9d4126cf0706 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-osd-0-deactivate, ceph=True, GIT_CLEAN=True, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.build-date=20240521) 2024-05-27T07:38:01.575 INFO:journalctl@ceph.osd.0.smithi052.stdout:May 27 07:38:01 smithi052 podman[119137]: 2024-05-27 07:38:01.272444477 +0000 UTC m=+0.389844962 container start 0187357a004219755ad1ce1d207a9704a9122bc509fcb27c924f9d4126cf0706 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-osd-0-deactivate, GIT_BRANCH=HEAD, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, maintainer=Guillaume Abrioux , ceph=True, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240521, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90) 2024-05-27T07:38:01.575 INFO:journalctl@ceph.osd.0.smithi052.stdout:May 27 07:38:01 smithi052 podman[119137]: 2024-05-27 07:38:01.377246622 +0000 UTC m=+0.494647108 container attach 0187357a004219755ad1ce1d207a9704a9122bc509fcb27c924f9d4126cf0706 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-osd-0-deactivate, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, org.label-schema.schema-version=1.0, ceph=True, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, io.buildah.version=1.33.5, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240521, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image) 2024-05-27T07:38:01.926 INFO:journalctl@ceph.osd.0.smithi052.stdout:May 27 07:38:01 smithi052 podman[119137]: 2024-05-27 07:38:01.579704164 +0000 UTC m=+0.697104651 container died 0187357a004219755ad1ce1d207a9704a9122bc509fcb27c924f9d4126cf0706 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-osd-0-deactivate, ceph=True, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, GIT_CLEAN=True, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.build-date=20240521) 2024-05-27T07:38:02.225 DEBUG:teuthology.orchestra.run.smithi052:> sudo pkill -f 'journalctl -f -n 0 -u ceph-73df2636-1bf8-11ef-bc9b-c7b262605968@osd.0.service' 2024-05-27T07:38:02.266 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-05-27T07:38:02.266 INFO:tasks.cephadm.osd.0:Stopped osd.0 2024-05-27T07:38:02.266 INFO:tasks.cephadm.osd.1:Stopping osd.1... 2024-05-27T07:38:02.266 DEBUG:teuthology.orchestra.run.smithi105:> sudo systemctl stop ceph-73df2636-1bf8-11ef-bc9b-c7b262605968@osd.1 2024-05-27T07:38:02.602 INFO:journalctl@ceph.osd.1.smithi105.stdout:May 27 07:38:02 smithi105 systemd[1]: Stopping Ceph osd.1 for 73df2636-1bf8-11ef-bc9b-c7b262605968... 2024-05-27T07:38:02.946 INFO:journalctl@ceph.osd.1.smithi105.stdout:May 27 07:38:02 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-osd-1[40586]: 2024-05-27T07:38:02.605+0000 7f854bd09700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.1 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-05-27T07:38:02.947 INFO:journalctl@ceph.osd.1.smithi105.stdout:May 27 07:38:02 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-osd-1[40586]: 2024-05-27T07:38:02.605+0000 7f854bd09700 -1 osd.1 134 *** Got signal Terminated *** 2024-05-27T07:38:02.947 INFO:journalctl@ceph.osd.1.smithi105.stdout:May 27 07:38:02 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-osd-1[40586]: 2024-05-27T07:38:02.605+0000 7f854bd09700 -1 osd.1 134 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-05-27T07:38:08.307 INFO:journalctl@ceph.osd.1.smithi105.stdout:May 27 07:38:08 smithi105 podman[72068]: 2024-05-27 07:38:08.008188936 +0000 UTC m=+5.532078709 container died 1f9c3cf032b501c7c94b88f4c47e92845812cd9b1e02fc4b57d2a9885358cb21 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-osd-1, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240521, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, ceph=True, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, org.label-schema.license=GPLv2, GIT_CLEAN=True, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image) 2024-05-27T07:38:08.604 INFO:journalctl@ceph.osd.1.smithi105.stdout:May 27 07:38:08 smithi105 podman[72068]: 2024-05-27 07:38:08.414475085 +0000 UTC m=+5.938364849 container cleanup 1f9c3cf032b501c7c94b88f4c47e92845812cd9b1e02fc4b57d2a9885358cb21 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-osd-1, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240521, org.label-schema.vendor=CentOS, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , ceph=True, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, GIT_CLEAN=True) 2024-05-27T07:38:08.604 INFO:journalctl@ceph.osd.1.smithi105.stdout:May 27 07:38:08 smithi105 bash[72068]: ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-osd-1 2024-05-27T07:38:08.946 INFO:journalctl@ceph.osd.1.smithi105.stdout:May 27 07:38:08 smithi105 podman[72082]: 2024-05-27 07:38:08.607640659 +0000 UTC m=+0.592403019 container remove 1f9c3cf032b501c7c94b88f4c47e92845812cd9b1e02fc4b57d2a9885358cb21 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-osd-1, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, ceph=True, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, org.label-schema.build-date=20240521, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, GIT_CLEAN=True, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS) 2024-05-27T07:38:09.334 INFO:journalctl@ceph.osd.1.smithi105.stdout:May 27 07:38:09 smithi105 podman[72176]: 2024-05-27 07:38:08.951205274 +0000 UTC m=+0.017150221 image pull e36ea4fc8a5807e8d2701370e350a0de3856538797f7fba64cfce7e344bd6799 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6 2024-05-27T07:38:09.334 INFO:journalctl@ceph.osd.1.smithi105.stdout:May 27 07:38:09 smithi105 podman[72176]: 2024-05-27 07:38:09.091887415 +0000 UTC m=+0.157832352 container create 028f21892fb2551ec376801a0bf0e8d809c4c2042dc24f304dbd894828f10b53 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-osd-1-deactivate, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, GIT_BRANCH=HEAD, org.label-schema.build-date=20240521, GIT_CLEAN=True, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-05-27T07:38:09.334 INFO:journalctl@ceph.osd.1.smithi105.stdout:May 27 07:38:09 smithi105 podman[72176]: 2024-05-27 07:38:09.337750413 +0000 UTC m=+0.403695360 container init 028f21892fb2551ec376801a0bf0e8d809c4c2042dc24f304dbd894828f10b53 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-osd-1-deactivate, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240521, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, maintainer=Guillaume Abrioux , GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, ceph=True, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90) 2024-05-27T07:38:09.697 INFO:journalctl@ceph.osd.1.smithi105.stdout:May 27 07:38:09 smithi105 podman[72176]: 2024-05-27 07:38:09.341118558 +0000 UTC m=+0.407063493 container start 028f21892fb2551ec376801a0bf0e8d809c4c2042dc24f304dbd894828f10b53 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-osd-1-deactivate, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, ceph=True, GIT_BRANCH=HEAD, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, org.label-schema.build-date=20240521, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0) 2024-05-27T07:38:09.697 INFO:journalctl@ceph.osd.1.smithi105.stdout:May 27 07:38:09 smithi105 podman[72176]: 2024-05-27 07:38:09.444355446 +0000 UTC m=+0.510300393 container attach 028f21892fb2551ec376801a0bf0e8d809c4c2042dc24f304dbd894828f10b53 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-osd-1-deactivate, org.label-schema.build-date=20240521, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, GIT_CLEAN=True, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, maintainer=Guillaume Abrioux , GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, ceph=True) 2024-05-27T07:38:10.194 INFO:journalctl@ceph.osd.1.smithi105.stdout:May 27 07:38:09 smithi105 podman[72176]: 2024-05-27 07:38:09.886685949 +0000 UTC m=+0.952630886 container died 028f21892fb2551ec376801a0bf0e8d809c4c2042dc24f304dbd894828f10b53 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-osd-1-deactivate, GIT_CLEAN=True, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, org.label-schema.build-date=20240521, GIT_BRANCH=HEAD, io.buildah.version=1.33.5) 2024-05-27T07:38:10.870 DEBUG:teuthology.orchestra.run.smithi105:> sudo pkill -f 'journalctl -f -n 0 -u ceph-73df2636-1bf8-11ef-bc9b-c7b262605968@osd.1.service' 2024-05-27T07:38:10.915 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-05-27T07:38:10.915 INFO:tasks.cephadm.osd.1:Stopped osd.1 2024-05-27T07:38:10.915 INFO:tasks.cephadm.osd.2:Stopping osd.2... 2024-05-27T07:38:10.916 DEBUG:teuthology.orchestra.run.smithi173:> sudo systemctl stop ceph-73df2636-1bf8-11ef-bc9b-c7b262605968@osd.2 2024-05-27T07:38:11.359 INFO:journalctl@ceph.osd.2.smithi173.stdout:May 27 07:38:10 smithi173 systemd[1]: Stopping Ceph osd.2 for 73df2636-1bf8-11ef-bc9b-c7b262605968... 2024-05-27T07:38:11.859 INFO:journalctl@ceph.osd.2.smithi173.stdout:May 27 07:38:11 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-osd-2[42253]: 2024-05-27T07:38:11.454+0000 7fe7d50fb700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.2 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-05-27T07:38:11.859 INFO:journalctl@ceph.osd.2.smithi173.stdout:May 27 07:38:11 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-osd-2[42253]: 2024-05-27T07:38:11.454+0000 7fe7d50fb700 -1 osd.2 134 *** Got signal Terminated *** 2024-05-27T07:38:11.859 INFO:journalctl@ceph.osd.2.smithi173.stdout:May 27 07:38:11 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-osd-2[42253]: 2024-05-27T07:38:11.454+0000 7fe7d50fb700 -1 osd.2 134 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-05-27T07:38:17.328 INFO:journalctl@ceph.osd.2.smithi173.stdout:May 27 07:38:17 smithi173 podman[75675]: 2024-05-27 07:38:17.061148372 +0000 UTC m=+5.942892313 container died d7fbbe1e5b24410165c73604accd46c2b3b0bfab8b587711b3dcef841c803b08 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-osd-2, ceph=True, org.label-schema.build-date=20240521, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90) 2024-05-27T07:38:17.603 INFO:journalctl@ceph.osd.2.smithi173.stdout:May 27 07:38:17 smithi173 podman[75675]: 2024-05-27 07:38:17.396805193 +0000 UTC m=+6.278549123 container cleanup d7fbbe1e5b24410165c73604accd46c2b3b0bfab8b587711b3dcef841c803b08 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-osd-2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240521, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, GIT_BRANCH=HEAD, ceph=True, org.label-schema.vendor=CentOS, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image) 2024-05-27T07:38:17.603 INFO:journalctl@ceph.osd.2.smithi173.stdout:May 27 07:38:17 smithi173 bash[75675]: ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-osd-2 2024-05-27T07:38:17.859 INFO:journalctl@ceph.osd.2.smithi173.stdout:May 27 07:38:17 smithi173 podman[75691]: 2024-05-27 07:38:17.606749704 +0000 UTC m=+0.543777262 container remove d7fbbe1e5b24410165c73604accd46c2b3b0bfab8b587711b3dcef841c803b08 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-osd-2, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240521, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux ) 2024-05-27T07:38:18.274 INFO:journalctl@ceph.osd.2.smithi173.stdout:May 27 07:38:17 smithi173 podman[75778]: 2024-05-27 07:38:17.999183568 +0000 UTC m=+0.099744854 container create 9231e83dd7f492ed219245e1d2616e3d53c0d5ff09cf6acc39b5bc9af5eb3eea (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-osd-2-deactivate, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, maintainer=Guillaume Abrioux , RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, GIT_CLEAN=True, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, ceph=True, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.build-date=20240521, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-05-27T07:38:18.274 INFO:journalctl@ceph.osd.2.smithi173.stdout:May 27 07:38:18 smithi173 podman[75778]: 2024-05-27 07:38:17.917026657 +0000 UTC m=+0.017582763 image pull e36ea4fc8a5807e8d2701370e350a0de3856538797f7fba64cfce7e344bd6799 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6 2024-05-27T07:38:18.274 INFO:journalctl@ceph.osd.2.smithi173.stdout:May 27 07:38:18 smithi173 podman[75778]: 2024-05-27 07:38:18.276834421 +0000 UTC m=+0.377390517 container init 9231e83dd7f492ed219245e1d2616e3d53c0d5ff09cf6acc39b5bc9af5eb3eea (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-osd-2-deactivate, GIT_CLEAN=True, io.buildah.version=1.33.5, ceph=True, org.label-schema.license=GPLv2, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, org.label-schema.build-date=20240521, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-05-27T07:38:18.567 INFO:journalctl@ceph.osd.2.smithi173.stdout:May 27 07:38:18 smithi173 podman[75778]: 2024-05-27 07:38:18.279993678 +0000 UTC m=+0.380549775 container start 9231e83dd7f492ed219245e1d2616e3d53c0d5ff09cf6acc39b5bc9af5eb3eea (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-osd-2-deactivate, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, ceph=True, org.label-schema.build-date=20240521, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0) 2024-05-27T07:38:18.568 INFO:journalctl@ceph.osd.2.smithi173.stdout:May 27 07:38:18 smithi173 podman[75778]: 2024-05-27 07:38:18.368689341 +0000 UTC m=+0.469245437 container attach 9231e83dd7f492ed219245e1d2616e3d53c0d5ff09cf6acc39b5bc9af5eb3eea (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-osd-2-deactivate, org.label-schema.schema-version=1.0, GIT_CLEAN=True, org.label-schema.license=GPLv2, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.build-date=20240521, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, io.buildah.version=1.33.5) 2024-05-27T07:38:18.859 INFO:journalctl@ceph.osd.2.smithi173.stdout:May 27 07:38:18 smithi173 podman[75778]: 2024-05-27 07:38:18.571196695 +0000 UTC m=+0.671752801 container died 9231e83dd7f492ed219245e1d2616e3d53c0d5ff09cf6acc39b5bc9af5eb3eea (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-osd-2-deactivate, ceph=True, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, GIT_CLEAN=True, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, org.label-schema.build-date=20240521) 2024-05-27T07:38:19.251 DEBUG:teuthology.orchestra.run.smithi173:> sudo pkill -f 'journalctl -f -n 0 -u ceph-73df2636-1bf8-11ef-bc9b-c7b262605968@osd.2.service' 2024-05-27T07:38:19.289 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-05-27T07:38:19.289 INFO:tasks.cephadm.osd.2:Stopped osd.2 2024-05-27T07:38:19.290 INFO:tasks.cephadm.osd.3:Stopping osd.3... 2024-05-27T07:38:19.290 DEBUG:teuthology.orchestra.run.smithi052:> sudo systemctl stop ceph-73df2636-1bf8-11ef-bc9b-c7b262605968@osd.3 2024-05-27T07:38:19.614 INFO:journalctl@ceph.osd.3.smithi052.stdout:May 27 07:38:19 smithi052 systemd[1]: Stopping Ceph osd.3 for 73df2636-1bf8-11ef-bc9b-c7b262605968... 2024-05-27T07:38:19.986 INFO:journalctl@ceph.osd.3.smithi052.stdout:May 27 07:38:19 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-osd-3[53606]: 2024-05-27T07:38:19.616+0000 7f532a5c8700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.3 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-05-27T07:38:19.987 INFO:journalctl@ceph.osd.3.smithi052.stdout:May 27 07:38:19 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-osd-3[53606]: 2024-05-27T07:38:19.616+0000 7f532a5c8700 -1 osd.3 134 *** Got signal Terminated *** 2024-05-27T07:38:19.987 INFO:journalctl@ceph.osd.3.smithi052.stdout:May 27 07:38:19 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-osd-3[53606]: 2024-05-27T07:38:19.616+0000 7f532a5c8700 -1 osd.3 134 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-05-27T07:38:22.447 INFO:journalctl@ceph.osd.10.smithi105.stdout:May 27 07:38:22 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-osd-10[60334]: 2024-05-27T07:38:22.113+0000 7f162fb8f700 -1 osd.10 134 heartbeat_check: no reply from 172.21.15.52:6806 osd.0 since back 2024-05-27T07:37:56.754985+0000 front 2024-05-27T07:37:56.755070+0000 (oldest deadline 2024-05-27T07:38:22.054911+0000) 2024-05-27T07:38:22.859 INFO:journalctl@ceph.osd.11.smithi173.stdout:May 27 07:38:22 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-osd-11[63021]: 2024-05-27T07:38:22.584+0000 7f9d44ac8700 -1 osd.11 134 heartbeat_check: no reply from 172.21.15.52:6806 osd.0 since back 2024-05-27T07:37:57.927796+0000 front 2024-05-27T07:37:57.927759+0000 (oldest deadline 2024-05-27T07:38:22.028025+0000) 2024-05-27T07:38:23.236 INFO:journalctl@ceph.osd.9.smithi052.stdout:May 27 07:38:22 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-osd-9[67134]: 2024-05-27T07:38:22.797+0000 7f1e3e2ca700 -1 osd.9 134 heartbeat_check: no reply from 172.21.15.52:6806 osd.0 since back 2024-05-27T07:37:58.861291+0000 front 2024-05-27T07:37:58.861258+0000 (oldest deadline 2024-05-27T07:38:22.361571+0000) 2024-05-27T07:38:23.446 INFO:journalctl@ceph.osd.10.smithi105.stdout:May 27 07:38:23 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-osd-10[60334]: 2024-05-27T07:38:23.101+0000 7f162fb8f700 -1 osd.10 134 heartbeat_check: no reply from 172.21.15.52:6806 osd.0 since back 2024-05-27T07:37:56.754985+0000 front 2024-05-27T07:37:56.755070+0000 (oldest deadline 2024-05-27T07:38:22.054911+0000) 2024-05-27T07:38:23.859 INFO:journalctl@ceph.osd.11.smithi173.stdout:May 27 07:38:23 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-osd-11[63021]: 2024-05-27T07:38:23.535+0000 7f9d44ac8700 -1 osd.11 134 heartbeat_check: no reply from 172.21.15.52:6806 osd.0 since back 2024-05-27T07:37:57.927796+0000 front 2024-05-27T07:37:57.927759+0000 (oldest deadline 2024-05-27T07:38:22.028025+0000) 2024-05-27T07:38:23.859 INFO:journalctl@ceph.osd.5.smithi173.stdout:May 27 07:38:23 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-osd-5[49009]: 2024-05-27T07:38:23.630+0000 7f331c3f3700 -1 osd.5 134 heartbeat_check: no reply from 172.21.15.52:6806 osd.0 since back 2024-05-27T07:37:57.916860+0000 front 2024-05-27T07:37:57.916876+0000 (oldest deadline 2024-05-27T07:38:22.616672+0000) 2024-05-27T07:38:24.236 INFO:journalctl@ceph.osd.9.smithi052.stdout:May 27 07:38:23 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-osd-9[67134]: 2024-05-27T07:38:23.823+0000 7f1e3e2ca700 -1 osd.9 134 heartbeat_check: no reply from 172.21.15.52:6806 osd.0 since back 2024-05-27T07:37:58.861291+0000 front 2024-05-27T07:37:58.861258+0000 (oldest deadline 2024-05-27T07:38:22.361571+0000) 2024-05-27T07:38:24.236 INFO:journalctl@ceph.osd.6.smithi052.stdout:May 27 07:38:23 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-osd-6[60312]: 2024-05-27T07:38:23.828+0000 7f362964d700 -1 osd.6 134 heartbeat_check: no reply from 172.21.15.52:6806 osd.0 since back 2024-05-27T07:37:58.099066+0000 front 2024-05-27T07:37:58.099034+0000 (oldest deadline 2024-05-27T07:38:23.399228+0000) 2024-05-27T07:38:24.447 INFO:journalctl@ceph.osd.10.smithi105.stdout:May 27 07:38:24 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-osd-10[60334]: 2024-05-27T07:38:24.105+0000 7f162fb8f700 -1 osd.10 134 heartbeat_check: no reply from 172.21.15.52:6806 osd.0 since back 2024-05-27T07:37:56.754985+0000 front 2024-05-27T07:37:56.755070+0000 (oldest deadline 2024-05-27T07:38:22.054911+0000) 2024-05-27T07:38:24.859 INFO:journalctl@ceph.osd.5.smithi173.stdout:May 27 07:38:24 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-osd-5[49009]: 2024-05-27T07:38:24.662+0000 7f331c3f3700 -1 osd.5 134 heartbeat_check: no reply from 172.21.15.52:6806 osd.0 since back 2024-05-27T07:37:57.916860+0000 front 2024-05-27T07:37:57.916876+0000 (oldest deadline 2024-05-27T07:38:22.616672+0000) 2024-05-27T07:38:24.859 INFO:journalctl@ceph.osd.11.smithi173.stdout:May 27 07:38:24 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-osd-11[63021]: 2024-05-27T07:38:24.521+0000 7f9d44ac8700 -1 osd.11 134 heartbeat_check: no reply from 172.21.15.52:6806 osd.0 since back 2024-05-27T07:37:57.927796+0000 front 2024-05-27T07:37:57.927759+0000 (oldest deadline 2024-05-27T07:38:22.028025+0000) 2024-05-27T07:38:25.127 INFO:journalctl@ceph.osd.6.smithi052.stdout:May 27 07:38:24 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-osd-6[60312]: 2024-05-27T07:38:24.793+0000 7f362964d700 -1 osd.6 134 heartbeat_check: no reply from 172.21.15.52:6806 osd.0 since back 2024-05-27T07:37:58.099066+0000 front 2024-05-27T07:37:58.099034+0000 (oldest deadline 2024-05-27T07:38:23.399228+0000) 2024-05-27T07:38:25.128 INFO:journalctl@ceph.osd.9.smithi052.stdout:May 27 07:38:24 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-osd-9[67134]: 2024-05-27T07:38:24.813+0000 7f1e3e2ca700 -1 osd.9 134 heartbeat_check: no reply from 172.21.15.52:6806 osd.0 since back 2024-05-27T07:37:58.861291+0000 front 2024-05-27T07:37:58.861258+0000 (oldest deadline 2024-05-27T07:38:22.361571+0000) 2024-05-27T07:38:25.425 INFO:journalctl@ceph.osd.3.smithi052.stdout:May 27 07:38:25 smithi052 podman[119243]: 2024-05-27 07:38:25.131467827 +0000 UTC m=+5.636308794 container died 04dfdd33a347a2d7a399fe17043b648fae2ee8334d3a30460662399ef05ea40a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-osd-3, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, org.label-schema.build-date=20240521, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, ceph=True, org.label-schema.vendor=CentOS, GIT_CLEAN=True, maintainer=Guillaume Abrioux , RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0) 2024-05-27T07:38:25.446 INFO:journalctl@ceph.osd.10.smithi105.stdout:May 27 07:38:25 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-osd-10[60334]: 2024-05-27T07:38:25.137+0000 7f162fb8f700 -1 osd.10 134 heartbeat_check: no reply from 172.21.15.52:6806 osd.0 since back 2024-05-27T07:37:56.754985+0000 front 2024-05-27T07:37:56.755070+0000 (oldest deadline 2024-05-27T07:38:22.054911+0000) 2024-05-27T07:38:25.736 INFO:journalctl@ceph.osd.3.smithi052.stdout:May 27 07:38:25 smithi052 podman[119243]: 2024-05-27 07:38:25.63368713 +0000 UTC m=+6.138528119 container cleanup 04dfdd33a347a2d7a399fe17043b648fae2ee8334d3a30460662399ef05ea40a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-osd-3, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, org.label-schema.build-date=20240521, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, GIT_CLEAN=True, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, org.label-schema.license=GPLv2, ceph=True, io.buildah.version=1.33.5) 2024-05-27T07:38:25.736 INFO:journalctl@ceph.osd.3.smithi052.stdout:May 27 07:38:25 smithi052 bash[119243]: ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-osd-3 2024-05-27T07:38:25.859 INFO:journalctl@ceph.osd.5.smithi173.stdout:May 27 07:38:25 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-osd-5[49009]: 2024-05-27T07:38:25.618+0000 7f331c3f3700 -1 osd.5 134 heartbeat_check: no reply from 172.21.15.52:6806 osd.0 since back 2024-05-27T07:37:57.916860+0000 front 2024-05-27T07:37:57.916876+0000 (oldest deadline 2024-05-27T07:38:22.616672+0000) 2024-05-27T07:38:25.859 INFO:journalctl@ceph.osd.11.smithi173.stdout:May 27 07:38:25 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-osd-11[63021]: 2024-05-27T07:38:25.510+0000 7f9d44ac8700 -1 osd.11 134 heartbeat_check: no reply from 172.21.15.52:6806 osd.0 since back 2024-05-27T07:37:57.927796+0000 front 2024-05-27T07:37:57.927759+0000 (oldest deadline 2024-05-27T07:38:22.028025+0000) 2024-05-27T07:38:26.236 INFO:journalctl@ceph.osd.3.smithi052.stdout:May 27 07:38:25 smithi052 podman[119259]: 2024-05-27 07:38:25.902111449 +0000 UTC m=+0.765254579 container remove 04dfdd33a347a2d7a399fe17043b648fae2ee8334d3a30460662399ef05ea40a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-osd-3, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, GIT_CLEAN=True, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, ceph=True, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, org.label-schema.build-date=20240521, GIT_BRANCH=HEAD) 2024-05-27T07:38:26.236 INFO:journalctl@ceph.osd.6.smithi052.stdout:May 27 07:38:25 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-osd-6[60312]: 2024-05-27T07:38:25.779+0000 7f362964d700 -1 osd.6 134 heartbeat_check: no reply from 172.21.15.52:6806 osd.0 since back 2024-05-27T07:37:58.099066+0000 front 2024-05-27T07:37:58.099034+0000 (oldest deadline 2024-05-27T07:38:23.399228+0000) 2024-05-27T07:38:26.237 INFO:journalctl@ceph.osd.9.smithi052.stdout:May 27 07:38:25 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-osd-9[67134]: 2024-05-27T07:38:25.831+0000 7f1e3e2ca700 -1 osd.9 134 heartbeat_check: no reply from 172.21.15.52:6806 osd.0 since back 2024-05-27T07:37:58.861291+0000 front 2024-05-27T07:37:58.861258+0000 (oldest deadline 2024-05-27T07:38:22.361571+0000) 2024-05-27T07:38:26.447 INFO:journalctl@ceph.osd.10.smithi105.stdout:May 27 07:38:26 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-osd-10[60334]: 2024-05-27T07:38:26.114+0000 7f162fb8f700 -1 osd.10 134 heartbeat_check: no reply from 172.21.15.52:6806 osd.0 since back 2024-05-27T07:37:56.754985+0000 front 2024-05-27T07:37:56.755070+0000 (oldest deadline 2024-05-27T07:38:22.054911+0000) 2024-05-27T07:38:26.576 INFO:journalctl@ceph.osd.3.smithi052.stdout:May 27 07:38:26 smithi052 podman[119347]: 2024-05-27 07:38:26.2613391 +0000 UTC m=+0.113707447 container create 8388f787020251c0f26244f23b26926643a7a069cdb16db9a4161031bd3ce7d6 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-osd-3-deactivate, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, maintainer=Guillaume Abrioux , ceph=True, GIT_CLEAN=True, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240521, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2) 2024-05-27T07:38:26.577 INFO:journalctl@ceph.osd.3.smithi052.stdout:May 27 07:38:26 smithi052 podman[119347]: 2024-05-27 07:38:26.165012488 +0000 UTC m=+0.017380846 image pull e36ea4fc8a5807e8d2701370e350a0de3856538797f7fba64cfce7e344bd6799 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6 2024-05-27T07:38:26.577 INFO:journalctl@ceph.osd.3.smithi052.stdout:May 27 07:38:26 smithi052 podman[119347]: 2024-05-27 07:38:26.480386358 +0000 UTC m=+0.332754715 container init 8388f787020251c0f26244f23b26926643a7a069cdb16db9a4161031bd3ce7d6 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-osd-3-deactivate, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, org.label-schema.build-date=20240521, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, maintainer=Guillaume Abrioux ) 2024-05-27T07:38:26.577 INFO:journalctl@ceph.osd.3.smithi052.stdout:May 27 07:38:26 smithi052 podman[119347]: 2024-05-27 07:38:26.483668159 +0000 UTC m=+0.336036505 container start 8388f787020251c0f26244f23b26926643a7a069cdb16db9a4161031bd3ce7d6 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-osd-3-deactivate, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, GIT_CLEAN=True, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240521, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, ceph=True, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, CEPH_POINT_RELEASE=) 2024-05-27T07:38:26.577 INFO:journalctl@ceph.osd.3.smithi052.stdout:May 27 07:38:26 smithi052 podman[119347]: 2024-05-27 07:38:26.580529618 +0000 UTC m=+0.432897972 container attach 8388f787020251c0f26244f23b26926643a7a069cdb16db9a4161031bd3ce7d6 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-osd-3-deactivate, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240521, org.label-schema.vendor=CentOS, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, GIT_CLEAN=True, org.label-schema.schema-version=1.0, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0) 2024-05-27T07:38:26.859 INFO:journalctl@ceph.osd.11.smithi173.stdout:May 27 07:38:26 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-osd-11[63021]: 2024-05-27T07:38:26.493+0000 7f9d44ac8700 -1 osd.11 134 heartbeat_check: no reply from 172.21.15.52:6806 osd.0 since back 2024-05-27T07:37:57.927796+0000 front 2024-05-27T07:37:57.927759+0000 (oldest deadline 2024-05-27T07:38:22.028025+0000) 2024-05-27T07:38:26.859 INFO:journalctl@ceph.osd.5.smithi173.stdout:May 27 07:38:26 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-osd-5[49009]: 2024-05-27T07:38:26.663+0000 7f331c3f3700 -1 osd.5 134 heartbeat_check: no reply from 172.21.15.52:6806 osd.0 since back 2024-05-27T07:37:57.916860+0000 front 2024-05-27T07:37:57.916876+0000 (oldest deadline 2024-05-27T07:38:22.616672+0000) 2024-05-27T07:38:26.986 INFO:journalctl@ceph.osd.3.smithi052.stdout:May 27 07:38:26 smithi052 podman[119347]: 2024-05-27 07:38:26.788234393 +0000 UTC m=+0.640602751 container died 8388f787020251c0f26244f23b26926643a7a069cdb16db9a4161031bd3ce7d6 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-osd-3-deactivate, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, org.label-schema.build-date=20240521, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, ceph=True) 2024-05-27T07:38:26.986 INFO:journalctl@ceph.osd.6.smithi052.stdout:May 27 07:38:26 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-osd-6[60312]: 2024-05-27T07:38:26.753+0000 7f362964d700 -1 osd.6 134 heartbeat_check: no reply from 172.21.15.52:6806 osd.0 since back 2024-05-27T07:37:58.099066+0000 front 2024-05-27T07:37:58.099034+0000 (oldest deadline 2024-05-27T07:38:23.399228+0000) 2024-05-27T07:38:26.987 INFO:journalctl@ceph.osd.9.smithi052.stdout:May 27 07:38:26 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-osd-9[67134]: 2024-05-27T07:38:26.829+0000 7f1e3e2ca700 -1 osd.9 134 heartbeat_check: no reply from 172.21.15.52:6806 osd.0 since back 2024-05-27T07:37:58.861291+0000 front 2024-05-27T07:37:58.861258+0000 (oldest deadline 2024-05-27T07:38:22.361571+0000) 2024-05-27T07:38:27.447 INFO:journalctl@ceph.osd.10.smithi105.stdout:May 27 07:38:27 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-osd-10[60334]: 2024-05-27T07:38:27.112+0000 7f162fb8f700 -1 osd.10 134 heartbeat_check: no reply from 172.21.15.52:6806 osd.0 since back 2024-05-27T07:37:56.754985+0000 front 2024-05-27T07:37:56.755070+0000 (oldest deadline 2024-05-27T07:38:22.054911+0000) 2024-05-27T07:38:27.535 DEBUG:teuthology.orchestra.run.smithi052:> sudo pkill -f 'journalctl -f -n 0 -u ceph-73df2636-1bf8-11ef-bc9b-c7b262605968@osd.3.service' 2024-05-27T07:38:27.581 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-05-27T07:38:27.581 INFO:tasks.cephadm.osd.3:Stopped osd.3 2024-05-27T07:38:27.582 INFO:tasks.cephadm.osd.4:Stopping osd.4... 2024-05-27T07:38:27.582 DEBUG:teuthology.orchestra.run.smithi105:> sudo systemctl stop ceph-73df2636-1bf8-11ef-bc9b-c7b262605968@osd.4 2024-05-27T07:38:27.834 INFO:journalctl@ceph.osd.6.smithi052.stdout:May 27 07:38:27 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-osd-6[60312]: 2024-05-27T07:38:27.705+0000 7f362964d700 -1 osd.6 134 heartbeat_check: no reply from 172.21.15.52:6806 osd.0 since back 2024-05-27T07:37:58.099066+0000 front 2024-05-27T07:37:58.099034+0000 (oldest deadline 2024-05-27T07:38:23.399228+0000) 2024-05-27T07:38:27.859 INFO:journalctl@ceph.osd.11.smithi173.stdout:May 27 07:38:27 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-osd-11[63021]: 2024-05-27T07:38:27.471+0000 7f9d44ac8700 -1 osd.11 134 heartbeat_check: no reply from 172.21.15.52:6806 osd.0 since back 2024-05-27T07:37:57.927796+0000 front 2024-05-27T07:37:57.927759+0000 (oldest deadline 2024-05-27T07:38:22.028025+0000) 2024-05-27T07:38:27.859 INFO:journalctl@ceph.osd.5.smithi173.stdout:May 27 07:38:27 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-osd-5[49009]: 2024-05-27T07:38:27.695+0000 7f331c3f3700 -1 osd.5 134 heartbeat_check: no reply from 172.21.15.52:6806 osd.0 since back 2024-05-27T07:37:57.916860+0000 front 2024-05-27T07:37:57.916876+0000 (oldest deadline 2024-05-27T07:38:22.616672+0000) 2024-05-27T07:38:27.890 INFO:journalctl@ceph.osd.4.smithi105.stdout:May 27 07:38:27 smithi105 systemd[1]: Stopping Ceph osd.4 for 73df2636-1bf8-11ef-bc9b-c7b262605968... 2024-05-27T07:38:28.196 INFO:journalctl@ceph.osd.10.smithi105.stdout:May 27 07:38:28 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-osd-10[60334]: 2024-05-27T07:38:28.139+0000 7f162fb8f700 -1 osd.10 134 heartbeat_check: no reply from 172.21.15.52:6806 osd.0 since back 2024-05-27T07:37:56.754985+0000 front 2024-05-27T07:37:56.755070+0000 (oldest deadline 2024-05-27T07:38:22.054911+0000) 2024-05-27T07:38:28.197 INFO:journalctl@ceph.osd.4.smithi105.stdout:May 27 07:38:27 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-osd-4[47069]: 2024-05-27T07:38:27.893+0000 7f8a422e8700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.4 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-05-27T07:38:28.197 INFO:journalctl@ceph.osd.4.smithi105.stdout:May 27 07:38:27 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-osd-4[47069]: 2024-05-27T07:38:27.893+0000 7f8a422e8700 -1 osd.4 134 *** Got signal Terminated *** 2024-05-27T07:38:28.197 INFO:journalctl@ceph.osd.4.smithi105.stdout:May 27 07:38:27 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-osd-4[47069]: 2024-05-27T07:38:27.893+0000 7f8a422e8700 -1 osd.4 134 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-05-27T07:38:28.236 INFO:journalctl@ceph.osd.9.smithi052.stdout:May 27 07:38:27 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-osd-9[67134]: 2024-05-27T07:38:27.837+0000 7f1e3e2ca700 -1 osd.9 134 heartbeat_check: no reply from 172.21.15.52:6806 osd.0 since back 2024-05-27T07:37:58.861291+0000 front 2024-05-27T07:37:58.861258+0000 (oldest deadline 2024-05-27T07:38:22.361571+0000) 2024-05-27T07:38:28.859 INFO:journalctl@ceph.osd.5.smithi173.stdout:May 27 07:38:28 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-osd-5[49009]: 2024-05-27T07:38:28.729+0000 7f331c3f3700 -1 osd.5 134 heartbeat_check: no reply from 172.21.15.52:6806 osd.0 since back 2024-05-27T07:37:57.916860+0000 front 2024-05-27T07:37:57.916876+0000 (oldest deadline 2024-05-27T07:38:22.616672+0000) 2024-05-27T07:38:28.859 INFO:journalctl@ceph.osd.11.smithi173.stdout:May 27 07:38:28 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-osd-11[63021]: 2024-05-27T07:38:28.518+0000 7f9d44ac8700 -1 osd.11 134 heartbeat_check: no reply from 172.21.15.52:6806 osd.0 since back 2024-05-27T07:37:57.927796+0000 front 2024-05-27T07:37:57.927759+0000 (oldest deadline 2024-05-27T07:38:22.028025+0000) 2024-05-27T07:38:28.986 INFO:journalctl@ceph.osd.6.smithi052.stdout:May 27 07:38:28 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-osd-6[60312]: 2024-05-27T07:38:28.696+0000 7f362964d700 -1 osd.6 134 heartbeat_check: no reply from 172.21.15.52:6806 osd.0 since back 2024-05-27T07:37:58.099066+0000 front 2024-05-27T07:37:58.099034+0000 (oldest deadline 2024-05-27T07:38:23.399228+0000) 2024-05-27T07:38:28.987 INFO:journalctl@ceph.osd.6.smithi052.stdout:May 27 07:38:28 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-osd-6[60312]: 2024-05-27T07:38:28.696+0000 7f362964d700 -1 osd.6 134 heartbeat_check: no reply from 172.21.15.105:6804 osd.1 since back 2024-05-27T07:38:03.400081+0000 front 2024-05-27T07:38:03.400066+0000 (oldest deadline 2024-05-27T07:38:28.099925+0000) 2024-05-27T07:38:28.987 INFO:journalctl@ceph.osd.9.smithi052.stdout:May 27 07:38:28 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-osd-9[67134]: 2024-05-27T07:38:28.802+0000 7f1e3e2ca700 -1 osd.9 134 heartbeat_check: no reply from 172.21.15.52:6806 osd.0 since back 2024-05-27T07:37:58.861291+0000 front 2024-05-27T07:37:58.861258+0000 (oldest deadline 2024-05-27T07:38:22.361571+0000) 2024-05-27T07:38:29.446 INFO:journalctl@ceph.osd.10.smithi105.stdout:May 27 07:38:29 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-osd-10[60334]: 2024-05-27T07:38:29.140+0000 7f162fb8f700 -1 osd.10 134 heartbeat_check: no reply from 172.21.15.52:6806 osd.0 since back 2024-05-27T07:37:56.754985+0000 front 2024-05-27T07:37:56.755070+0000 (oldest deadline 2024-05-27T07:38:22.054911+0000) 2024-05-27T07:38:29.757 INFO:journalctl@ceph.osd.11.smithi173.stdout:May 27 07:38:29 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-osd-11[63021]: 2024-05-27T07:38:29.477+0000 7f9d44ac8700 -1 osd.11 134 heartbeat_check: no reply from 172.21.15.52:6806 osd.0 since back 2024-05-27T07:37:57.927796+0000 front 2024-05-27T07:37:57.927759+0000 (oldest deadline 2024-05-27T07:38:22.028025+0000) 2024-05-27T07:38:29.757 INFO:journalctl@ceph.osd.11.smithi173.stdout:May 27 07:38:29 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-osd-11[63021]: 2024-05-27T07:38:29.477+0000 7f9d44ac8700 -1 osd.11 134 heartbeat_check: no reply from 172.21.15.105:6804 osd.1 since back 2024-05-27T07:38:03.129257+0000 front 2024-05-27T07:38:03.129120+0000 (oldest deadline 2024-05-27T07:38:28.929543+0000) 2024-05-27T07:38:30.109 INFO:journalctl@ceph.osd.5.smithi173.stdout:May 27 07:38:29 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-osd-5[49009]: 2024-05-27T07:38:29.760+0000 7f331c3f3700 -1 osd.5 134 heartbeat_check: no reply from 172.21.15.52:6806 osd.0 since back 2024-05-27T07:37:57.916860+0000 front 2024-05-27T07:37:57.916876+0000 (oldest deadline 2024-05-27T07:38:22.616672+0000) 2024-05-27T07:38:30.236 INFO:journalctl@ceph.osd.9.smithi052.stdout:May 27 07:38:29 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-osd-9[67134]: 2024-05-27T07:38:29.851+0000 7f1e3e2ca700 -1 osd.9 134 heartbeat_check: no reply from 172.21.15.52:6806 osd.0 since back 2024-05-27T07:37:58.861291+0000 front 2024-05-27T07:37:58.861258+0000 (oldest deadline 2024-05-27T07:38:22.361571+0000) 2024-05-27T07:38:30.236 INFO:journalctl@ceph.osd.6.smithi052.stdout:May 27 07:38:29 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-osd-6[60312]: 2024-05-27T07:38:29.742+0000 7f362964d700 -1 osd.6 134 heartbeat_check: no reply from 172.21.15.52:6806 osd.0 since back 2024-05-27T07:37:58.099066+0000 front 2024-05-27T07:37:58.099034+0000 (oldest deadline 2024-05-27T07:38:23.399228+0000) 2024-05-27T07:38:30.236 INFO:journalctl@ceph.osd.6.smithi052.stdout:May 27 07:38:29 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-osd-6[60312]: 2024-05-27T07:38:29.742+0000 7f362964d700 -1 osd.6 134 heartbeat_check: no reply from 172.21.15.105:6804 osd.1 since back 2024-05-27T07:38:03.400081+0000 front 2024-05-27T07:38:03.400066+0000 (oldest deadline 2024-05-27T07:38:28.099925+0000) 2024-05-27T07:38:30.447 INFO:journalctl@ceph.osd.10.smithi105.stdout:May 27 07:38:30 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-osd-10[60334]: 2024-05-27T07:38:30.189+0000 7f162fb8f700 -1 osd.10 134 heartbeat_check: no reply from 172.21.15.52:6806 osd.0 since back 2024-05-27T07:37:56.754985+0000 front 2024-05-27T07:37:56.755070+0000 (oldest deadline 2024-05-27T07:38:22.054911+0000) 2024-05-27T07:38:30.749 INFO:journalctl@ceph.osd.11.smithi173.stdout:May 27 07:38:30 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-osd-11[63021]: 2024-05-27T07:38:30.491+0000 7f9d44ac8700 -1 osd.11 134 heartbeat_check: no reply from 172.21.15.52:6806 osd.0 since back 2024-05-27T07:37:57.927796+0000 front 2024-05-27T07:37:57.927759+0000 (oldest deadline 2024-05-27T07:38:22.028025+0000) 2024-05-27T07:38:30.749 INFO:journalctl@ceph.osd.11.smithi173.stdout:May 27 07:38:30 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-osd-11[63021]: 2024-05-27T07:38:30.491+0000 7f9d44ac8700 -1 osd.11 134 heartbeat_check: no reply from 172.21.15.105:6804 osd.1 since back 2024-05-27T07:38:03.129257+0000 front 2024-05-27T07:38:03.129120+0000 (oldest deadline 2024-05-27T07:38:28.929543+0000) 2024-05-27T07:38:31.109 INFO:journalctl@ceph.osd.5.smithi173.stdout:May 27 07:38:30 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-osd-5[49009]: 2024-05-27T07:38:30.752+0000 7f331c3f3700 -1 osd.5 134 heartbeat_check: no reply from 172.21.15.52:6806 osd.0 since back 2024-05-27T07:37:57.916860+0000 front 2024-05-27T07:37:57.916876+0000 (oldest deadline 2024-05-27T07:38:22.616672+0000) 2024-05-27T07:38:31.109 INFO:journalctl@ceph.osd.8.smithi173.stdout:May 27 07:38:30 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-osd-8[55969]: 2024-05-27T07:38:30.932+0000 7f425b2ea700 -1 osd.8 134 heartbeat_check: no reply from 172.21.15.105:6804 osd.1 since back 2024-05-27T07:38:06.957078+0000 front 2024-05-27T07:38:06.957280+0000 (oldest deadline 2024-05-27T07:38:30.457072+0000) 2024-05-27T07:38:31.137 INFO:journalctl@ceph.osd.4.smithi105.stdout:May 27 07:38:30 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-osd-4[47069]: 2024-05-27T07:38:30.751+0000 7f8a3e89e700 -1 osd.4 134 heartbeat_check: no reply from 172.21.15.105:6804 osd.1 since back 2024-05-27T07:38:07.296438+0000 front 2024-05-27T07:38:07.296414+0000 (oldest deadline 2024-05-27T07:38:30.196646+0000) 2024-05-27T07:38:31.236 INFO:journalctl@ceph.osd.9.smithi052.stdout:May 27 07:38:30 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-osd-9[67134]: 2024-05-27T07:38:30.824+0000 7f1e3e2ca700 -1 osd.9 134 heartbeat_check: no reply from 172.21.15.52:6806 osd.0 since back 2024-05-27T07:37:58.861291+0000 front 2024-05-27T07:37:58.861258+0000 (oldest deadline 2024-05-27T07:38:22.361571+0000) 2024-05-27T07:38:31.236 INFO:journalctl@ceph.osd.9.smithi052.stdout:May 27 07:38:30 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-osd-9[67134]: 2024-05-27T07:38:30.824+0000 7f1e3e2ca700 -1 osd.9 134 heartbeat_check: no reply from 172.21.15.105:6804 osd.1 since back 2024-05-27T07:38:05.862960+0000 front 2024-05-27T07:38:05.862645+0000 (oldest deadline 2024-05-27T07:38:30.562694+0000) 2024-05-27T07:38:31.237 INFO:journalctl@ceph.osd.6.smithi052.stdout:May 27 07:38:30 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-osd-6[60312]: 2024-05-27T07:38:30.759+0000 7f362964d700 -1 osd.6 134 heartbeat_check: no reply from 172.21.15.52:6806 osd.0 since back 2024-05-27T07:37:58.099066+0000 front 2024-05-27T07:37:58.099034+0000 (oldest deadline 2024-05-27T07:38:23.399228+0000) 2024-05-27T07:38:31.237 INFO:journalctl@ceph.osd.6.smithi052.stdout:May 27 07:38:30 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-osd-6[60312]: 2024-05-27T07:38:30.759+0000 7f362964d700 -1 osd.6 134 heartbeat_check: no reply from 172.21.15.105:6804 osd.1 since back 2024-05-27T07:38:03.400081+0000 front 2024-05-27T07:38:03.400066+0000 (oldest deadline 2024-05-27T07:38:28.099925+0000) 2024-05-27T07:38:31.446 INFO:journalctl@ceph.osd.10.smithi105.stdout:May 27 07:38:31 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-osd-10[60334]: 2024-05-27T07:38:31.140+0000 7f162fb8f700 -1 osd.10 134 heartbeat_check: no reply from 172.21.15.52:6806 osd.0 since back 2024-05-27T07:37:56.754985+0000 front 2024-05-27T07:37:56.755070+0000 (oldest deadline 2024-05-27T07:38:22.054911+0000) 2024-05-27T07:38:31.750 INFO:journalctl@ceph.osd.11.smithi173.stdout:May 27 07:38:31 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-osd-11[63021]: 2024-05-27T07:38:31.499+0000 7f9d44ac8700 -1 osd.11 134 heartbeat_check: no reply from 172.21.15.52:6806 osd.0 since back 2024-05-27T07:37:57.927796+0000 front 2024-05-27T07:37:57.927759+0000 (oldest deadline 2024-05-27T07:38:22.028025+0000) 2024-05-27T07:38:31.750 INFO:journalctl@ceph.osd.11.smithi173.stdout:May 27 07:38:31 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-osd-11[63021]: 2024-05-27T07:38:31.499+0000 7f9d44ac8700 -1 osd.11 134 heartbeat_check: no reply from 172.21.15.105:6804 osd.1 since back 2024-05-27T07:38:03.129257+0000 front 2024-05-27T07:38:03.129120+0000 (oldest deadline 2024-05-27T07:38:28.929543+0000) 2024-05-27T07:38:32.109 INFO:journalctl@ceph.osd.8.smithi173.stdout:May 27 07:38:31 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-osd-8[55969]: 2024-05-27T07:38:31.977+0000 7f425b2ea700 -1 osd.8 134 heartbeat_check: no reply from 172.21.15.105:6804 osd.1 since back 2024-05-27T07:38:06.957078+0000 front 2024-05-27T07:38:06.957280+0000 (oldest deadline 2024-05-27T07:38:30.457072+0000) 2024-05-27T07:38:32.109 INFO:journalctl@ceph.osd.5.smithi173.stdout:May 27 07:38:31 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-osd-5[49009]: 2024-05-27T07:38:31.753+0000 7f331c3f3700 -1 osd.5 134 heartbeat_check: no reply from 172.21.15.52:6806 osd.0 since back 2024-05-27T07:37:57.916860+0000 front 2024-05-27T07:37:57.916876+0000 (oldest deadline 2024-05-27T07:38:22.616672+0000) 2024-05-27T07:38:32.188 INFO:journalctl@ceph.osd.4.smithi105.stdout:May 27 07:38:31 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-osd-4[47069]: 2024-05-27T07:38:31.702+0000 7f8a3e89e700 -1 osd.4 134 heartbeat_check: no reply from 172.21.15.105:6804 osd.1 since back 2024-05-27T07:38:07.296438+0000 front 2024-05-27T07:38:07.296414+0000 (oldest deadline 2024-05-27T07:38:30.196646+0000) 2024-05-27T07:38:32.236 INFO:journalctl@ceph.osd.6.smithi052.stdout:May 27 07:38:31 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-osd-6[60312]: 2024-05-27T07:38:31.764+0000 7f362964d700 -1 osd.6 134 heartbeat_check: no reply from 172.21.15.52:6806 osd.0 since back 2024-05-27T07:37:58.099066+0000 front 2024-05-27T07:37:58.099034+0000 (oldest deadline 2024-05-27T07:38:23.399228+0000) 2024-05-27T07:38:32.236 INFO:journalctl@ceph.osd.6.smithi052.stdout:May 27 07:38:31 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-osd-6[60312]: 2024-05-27T07:38:31.764+0000 7f362964d700 -1 osd.6 134 heartbeat_check: no reply from 172.21.15.105:6804 osd.1 since back 2024-05-27T07:38:03.400081+0000 front 2024-05-27T07:38:03.400066+0000 (oldest deadline 2024-05-27T07:38:28.099925+0000) 2024-05-27T07:38:32.237 INFO:journalctl@ceph.osd.9.smithi052.stdout:May 27 07:38:31 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-osd-9[67134]: 2024-05-27T07:38:31.814+0000 7f1e3e2ca700 -1 osd.9 134 heartbeat_check: no reply from 172.21.15.52:6806 osd.0 since back 2024-05-27T07:37:58.861291+0000 front 2024-05-27T07:37:58.861258+0000 (oldest deadline 2024-05-27T07:38:22.361571+0000) 2024-05-27T07:38:32.237 INFO:journalctl@ceph.osd.9.smithi052.stdout:May 27 07:38:31 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-osd-9[67134]: 2024-05-27T07:38:31.814+0000 7f1e3e2ca700 -1 osd.9 134 heartbeat_check: no reply from 172.21.15.105:6804 osd.1 since back 2024-05-27T07:38:05.862960+0000 front 2024-05-27T07:38:05.862645+0000 (oldest deadline 2024-05-27T07:38:30.562694+0000) 2024-05-27T07:38:32.447 INFO:journalctl@ceph.osd.10.smithi105.stdout:May 27 07:38:32 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-osd-10[60334]: 2024-05-27T07:38:32.190+0000 7f162fb8f700 -1 osd.10 134 heartbeat_check: no reply from 172.21.15.52:6806 osd.0 since back 2024-05-27T07:37:56.754985+0000 front 2024-05-27T07:37:56.755070+0000 (oldest deadline 2024-05-27T07:38:22.054911+0000) 2024-05-27T07:38:32.447 INFO:journalctl@ceph.osd.10.smithi105.stdout:May 27 07:38:32 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-osd-10[60334]: 2024-05-27T07:38:32.190+0000 7f162fb8f700 -1 osd.10 134 heartbeat_check: no reply from 172.21.15.105:6804 osd.1 since back 2024-05-27T07:38:05.556137+0000 front 2024-05-27T07:38:05.556112+0000 (oldest deadline 2024-05-27T07:38:31.456213+0000) 2024-05-27T07:38:32.859 INFO:journalctl@ceph.osd.5.smithi173.stdout:May 27 07:38:32 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-osd-5[49009]: 2024-05-27T07:38:32.744+0000 7f331c3f3700 -1 osd.5 134 heartbeat_check: no reply from 172.21.15.52:6806 osd.0 since back 2024-05-27T07:37:57.916860+0000 front 2024-05-27T07:37:57.916876+0000 (oldest deadline 2024-05-27T07:38:22.616672+0000) 2024-05-27T07:38:32.859 INFO:journalctl@ceph.osd.11.smithi173.stdout:May 27 07:38:32 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-osd-11[63021]: 2024-05-27T07:38:32.519+0000 7f9d44ac8700 -1 osd.11 134 heartbeat_check: no reply from 172.21.15.52:6806 osd.0 since back 2024-05-27T07:37:57.927796+0000 front 2024-05-27T07:37:57.927759+0000 (oldest deadline 2024-05-27T07:38:22.028025+0000) 2024-05-27T07:38:32.859 INFO:journalctl@ceph.osd.11.smithi173.stdout:May 27 07:38:32 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-osd-11[63021]: 2024-05-27T07:38:32.519+0000 7f9d44ac8700 -1 osd.11 134 heartbeat_check: no reply from 172.21.15.105:6804 osd.1 since back 2024-05-27T07:38:03.129257+0000 front 2024-05-27T07:38:03.129120+0000 (oldest deadline 2024-05-27T07:38:28.929543+0000) 2024-05-27T07:38:33.143 INFO:journalctl@ceph.osd.7.smithi105.stdout:May 27 07:38:32 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-osd-7[53596]: 2024-05-27T07:38:32.819+0000 7f5eefdf1700 -1 osd.7 134 heartbeat_check: no reply from 172.21.15.105:6804 osd.1 since back 2024-05-27T07:38:06.798385+0000 front 2024-05-27T07:38:06.798353+0000 (oldest deadline 2024-05-27T07:38:32.098493+0000) 2024-05-27T07:38:33.144 INFO:journalctl@ceph.osd.4.smithi105.stdout:May 27 07:38:32 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-osd-4[47069]: 2024-05-27T07:38:32.708+0000 7f8a3e89e700 -1 osd.4 134 heartbeat_check: no reply from 172.21.15.105:6804 osd.1 since back 2024-05-27T07:38:07.296438+0000 front 2024-05-27T07:38:07.296414+0000 (oldest deadline 2024-05-27T07:38:30.196646+0000) 2024-05-27T07:38:33.236 INFO:journalctl@ceph.osd.6.smithi052.stdout:May 27 07:38:32 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-osd-6[60312]: 2024-05-27T07:38:32.811+0000 7f362964d700 -1 osd.6 134 heartbeat_check: no reply from 172.21.15.52:6806 osd.0 since back 2024-05-27T07:37:58.099066+0000 front 2024-05-27T07:37:58.099034+0000 (oldest deadline 2024-05-27T07:38:23.399228+0000) 2024-05-27T07:38:33.236 INFO:journalctl@ceph.osd.6.smithi052.stdout:May 27 07:38:32 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-osd-6[60312]: 2024-05-27T07:38:32.811+0000 7f362964d700 -1 osd.6 134 heartbeat_check: no reply from 172.21.15.105:6804 osd.1 since back 2024-05-27T07:38:03.400081+0000 front 2024-05-27T07:38:03.400066+0000 (oldest deadline 2024-05-27T07:38:28.099925+0000) 2024-05-27T07:38:33.237 INFO:journalctl@ceph.osd.9.smithi052.stdout:May 27 07:38:32 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-osd-9[67134]: 2024-05-27T07:38:32.849+0000 7f1e3e2ca700 -1 osd.9 134 heartbeat_check: no reply from 172.21.15.52:6806 osd.0 since back 2024-05-27T07:37:58.861291+0000 front 2024-05-27T07:37:58.861258+0000 (oldest deadline 2024-05-27T07:38:22.361571+0000) 2024-05-27T07:38:33.237 INFO:journalctl@ceph.osd.9.smithi052.stdout:May 27 07:38:32 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-osd-9[67134]: 2024-05-27T07:38:32.849+0000 7f1e3e2ca700 -1 osd.9 134 heartbeat_check: no reply from 172.21.15.105:6804 osd.1 since back 2024-05-27T07:38:05.862960+0000 front 2024-05-27T07:38:05.862645+0000 (oldest deadline 2024-05-27T07:38:30.562694+0000) 2024-05-27T07:38:33.359 INFO:journalctl@ceph.osd.8.smithi173.stdout:May 27 07:38:32 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-osd-8[55969]: 2024-05-27T07:38:32.992+0000 7f425b2ea700 -1 osd.8 134 heartbeat_check: no reply from 172.21.15.105:6804 osd.1 since back 2024-05-27T07:38:06.957078+0000 front 2024-05-27T07:38:06.957280+0000 (oldest deadline 2024-05-27T07:38:30.457072+0000) 2024-05-27T07:38:33.446 INFO:journalctl@ceph.osd.10.smithi105.stdout:May 27 07:38:33 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-osd-10[60334]: 2024-05-27T07:38:33.146+0000 7f162fb8f700 -1 osd.10 134 heartbeat_check: no reply from 172.21.15.52:6806 osd.0 since back 2024-05-27T07:37:56.754985+0000 front 2024-05-27T07:37:56.755070+0000 (oldest deadline 2024-05-27T07:38:22.054911+0000) 2024-05-27T07:38:33.447 INFO:journalctl@ceph.osd.10.smithi105.stdout:May 27 07:38:33 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-osd-10[60334]: 2024-05-27T07:38:33.146+0000 7f162fb8f700 -1 osd.10 134 heartbeat_check: no reply from 172.21.15.105:6804 osd.1 since back 2024-05-27T07:38:05.556137+0000 front 2024-05-27T07:38:05.556112+0000 (oldest deadline 2024-05-27T07:38:31.456213+0000) 2024-05-27T07:38:33.736 INFO:journalctl@ceph.osd.11.smithi173.stdout:May 27 07:38:33 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-osd-11[63021]: 2024-05-27T07:38:33.478+0000 7f9d44ac8700 -1 osd.11 134 heartbeat_check: no reply from 172.21.15.52:6806 osd.0 since back 2024-05-27T07:37:57.927796+0000 front 2024-05-27T07:37:57.927759+0000 (oldest deadline 2024-05-27T07:38:22.028025+0000) 2024-05-27T07:38:33.736 INFO:journalctl@ceph.osd.11.smithi173.stdout:May 27 07:38:33 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-osd-11[63021]: 2024-05-27T07:38:33.478+0000 7f9d44ac8700 -1 osd.11 134 heartbeat_check: no reply from 172.21.15.105:6804 osd.1 since back 2024-05-27T07:38:03.129257+0000 front 2024-05-27T07:38:03.129120+0000 (oldest deadline 2024-05-27T07:38:28.929543+0000) 2024-05-27T07:38:33.837 INFO:journalctl@ceph.osd.7.smithi105.stdout:May 27 07:38:33 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-osd-7[53596]: 2024-05-27T07:38:33.788+0000 7f5eefdf1700 -1 osd.7 134 heartbeat_check: no reply from 172.21.15.105:6804 osd.1 since back 2024-05-27T07:38:06.798385+0000 front 2024-05-27T07:38:06.798353+0000 (oldest deadline 2024-05-27T07:38:32.098493+0000) 2024-05-27T07:38:33.837 INFO:journalctl@ceph.osd.4.smithi105.stdout:May 27 07:38:33 smithi105 podman[72299]: 2024-05-27 07:38:33.550121524 +0000 UTC m=+5.769915126 container died 69b940abbf36b9c70a305c8258c0c975c7307acef1b537f3d4507b70cc67aae7 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-osd-4, maintainer=Guillaume Abrioux , ceph=True, GIT_BRANCH=HEAD, org.label-schema.build-date=20240521, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-05-27T07:38:34.109 INFO:journalctl@ceph.osd.8.smithi173.stdout:May 27 07:38:33 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-osd-8[55969]: 2024-05-27T07:38:33.952+0000 7f425b2ea700 -1 osd.8 134 heartbeat_check: no reply from 172.21.15.105:6804 osd.1 since back 2024-05-27T07:38:06.957078+0000 front 2024-05-27T07:38:06.957280+0000 (oldest deadline 2024-05-27T07:38:30.457072+0000) 2024-05-27T07:38:34.109 INFO:journalctl@ceph.osd.5.smithi173.stdout:May 27 07:38:33 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-osd-5[49009]: 2024-05-27T07:38:33.739+0000 7f331c3f3700 -1 osd.5 134 heartbeat_check: no reply from 172.21.15.52:6806 osd.0 since back 2024-05-27T07:37:57.916860+0000 front 2024-05-27T07:37:57.916876+0000 (oldest deadline 2024-05-27T07:38:22.616672+0000) 2024-05-27T07:38:34.109 INFO:journalctl@ceph.osd.5.smithi173.stdout:May 27 07:38:33 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-osd-5[49009]: 2024-05-27T07:38:33.739+0000 7f331c3f3700 -1 osd.5 134 heartbeat_check: no reply from 172.21.15.105:6804 osd.1 since back 2024-05-27T07:38:06.117796+0000 front 2024-05-27T07:38:06.117770+0000 (oldest deadline 2024-05-27T07:38:33.118474+0000) 2024-05-27T07:38:34.180 INFO:journalctl@ceph.osd.4.smithi105.stdout:May 27 07:38:33 smithi105 podman[72299]: 2024-05-27 07:38:33.944092179 +0000 UTC m=+6.163885790 container cleanup 69b940abbf36b9c70a305c8258c0c975c7307acef1b537f3d4507b70cc67aae7 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-osd-4, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, GIT_CLEAN=True, io.buildah.version=1.33.5, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.license=GPLv2, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, org.label-schema.build-date=20240521, org.label-schema.name=CentOS Stream 8 Base Image) 2024-05-27T07:38:34.181 INFO:journalctl@ceph.osd.4.smithi105.stdout:May 27 07:38:33 smithi105 bash[72299]: ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-osd-4 2024-05-27T07:38:34.236 INFO:journalctl@ceph.osd.6.smithi052.stdout:May 27 07:38:33 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-osd-6[60312]: 2024-05-27T07:38:33.803+0000 7f362964d700 -1 osd.6 134 heartbeat_check: no reply from 172.21.15.52:6806 osd.0 since back 2024-05-27T07:37:58.099066+0000 front 2024-05-27T07:37:58.099034+0000 (oldest deadline 2024-05-27T07:38:23.399228+0000) 2024-05-27T07:38:34.237 INFO:journalctl@ceph.osd.6.smithi052.stdout:May 27 07:38:33 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-osd-6[60312]: 2024-05-27T07:38:33.803+0000 7f362964d700 -1 osd.6 134 heartbeat_check: no reply from 172.21.15.105:6804 osd.1 since back 2024-05-27T07:38:03.400081+0000 front 2024-05-27T07:38:03.400066+0000 (oldest deadline 2024-05-27T07:38:28.099925+0000) 2024-05-27T07:38:34.237 INFO:journalctl@ceph.osd.9.smithi052.stdout:May 27 07:38:33 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-osd-9[67134]: 2024-05-27T07:38:33.847+0000 7f1e3e2ca700 -1 osd.9 134 heartbeat_check: no reply from 172.21.15.52:6806 osd.0 since back 2024-05-27T07:37:58.861291+0000 front 2024-05-27T07:37:58.861258+0000 (oldest deadline 2024-05-27T07:38:22.361571+0000) 2024-05-27T07:38:34.237 INFO:journalctl@ceph.osd.9.smithi052.stdout:May 27 07:38:33 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-osd-9[67134]: 2024-05-27T07:38:33.847+0000 7f1e3e2ca700 -1 osd.9 134 heartbeat_check: no reply from 172.21.15.105:6804 osd.1 since back 2024-05-27T07:38:05.862960+0000 front 2024-05-27T07:38:05.862645+0000 (oldest deadline 2024-05-27T07:38:30.562694+0000) 2024-05-27T07:38:34.447 INFO:journalctl@ceph.osd.4.smithi105.stdout:May 27 07:38:34 smithi105 podman[72314]: 2024-05-27 07:38:34.204078008 +0000 UTC m=+0.649921781 container remove 69b940abbf36b9c70a305c8258c0c975c7307acef1b537f3d4507b70cc67aae7 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-osd-4, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240521, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , ceph=True, io.buildah.version=1.33.5) 2024-05-27T07:38:34.447 INFO:journalctl@ceph.osd.10.smithi105.stdout:May 27 07:38:34 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-osd-10[60334]: 2024-05-27T07:38:34.184+0000 7f162fb8f700 -1 osd.10 134 heartbeat_check: no reply from 172.21.15.52:6806 osd.0 since back 2024-05-27T07:37:56.754985+0000 front 2024-05-27T07:37:56.755070+0000 (oldest deadline 2024-05-27T07:38:22.054911+0000) 2024-05-27T07:38:34.447 INFO:journalctl@ceph.osd.10.smithi105.stdout:May 27 07:38:34 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-osd-10[60334]: 2024-05-27T07:38:34.184+0000 7f162fb8f700 -1 osd.10 134 heartbeat_check: no reply from 172.21.15.105:6804 osd.1 since back 2024-05-27T07:38:05.556137+0000 front 2024-05-27T07:38:05.556112+0000 (oldest deadline 2024-05-27T07:38:31.456213+0000) 2024-05-27T07:38:34.701 INFO:journalctl@ceph.osd.11.smithi173.stdout:May 27 07:38:34 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-osd-11[63021]: 2024-05-27T07:38:34.447+0000 7f9d44ac8700 -1 osd.11 134 heartbeat_check: no reply from 172.21.15.52:6806 osd.0 since back 2024-05-27T07:37:57.927796+0000 front 2024-05-27T07:37:57.927759+0000 (oldest deadline 2024-05-27T07:38:22.028025+0000) 2024-05-27T07:38:34.701 INFO:journalctl@ceph.osd.11.smithi173.stdout:May 27 07:38:34 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-osd-11[63021]: 2024-05-27T07:38:34.447+0000 7f9d44ac8700 -1 osd.11 134 heartbeat_check: no reply from 172.21.15.105:6804 osd.1 since back 2024-05-27T07:38:03.129257+0000 front 2024-05-27T07:38:03.129120+0000 (oldest deadline 2024-05-27T07:38:28.929543+0000) 2024-05-27T07:38:34.780 INFO:journalctl@ceph.osd.4.smithi105.stdout:May 27 07:38:34 smithi105 podman[72406]: 2024-05-27 07:38:34.417823127 +0000 UTC m=+0.017060318 image pull e36ea4fc8a5807e8d2701370e350a0de3856538797f7fba64cfce7e344bd6799 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6 2024-05-27T07:38:34.780 INFO:journalctl@ceph.osd.4.smithi105.stdout:May 27 07:38:34 smithi105 podman[72406]: 2024-05-27 07:38:34.546534114 +0000 UTC m=+0.145771285 container create f759d4f312ea2c03449a25316acf7f937b55bb4140d5c646735809012371d2b9 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-osd-4-deactivate, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, GIT_CLEAN=True, org.label-schema.schema-version=1.0, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, org.label-schema.build-date=20240521, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, ceph=True, CEPH_POINT_RELEASE=) 2024-05-27T07:38:35.096 INFO:journalctl@ceph.osd.4.smithi105.stdout:May 27 07:38:34 smithi105 podman[72406]: 2024-05-27 07:38:34.807355072 +0000 UTC m=+0.406592240 container init f759d4f312ea2c03449a25316acf7f937b55bb4140d5c646735809012371d2b9 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-osd-4-deactivate, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, org.label-schema.license=GPLv2, org.label-schema.build-date=20240521, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image) 2024-05-27T07:38:35.096 INFO:journalctl@ceph.osd.4.smithi105.stdout:May 27 07:38:34 smithi105 podman[72406]: 2024-05-27 07:38:34.810851118 +0000 UTC m=+0.410088301 container start f759d4f312ea2c03449a25316acf7f937b55bb4140d5c646735809012371d2b9 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-osd-4-deactivate, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, GIT_CLEAN=True, org.label-schema.build-date=20240521, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, GIT_BRANCH=HEAD) 2024-05-27T07:38:35.096 INFO:journalctl@ceph.osd.4.smithi105.stdout:May 27 07:38:34 smithi105 podman[72406]: 2024-05-27 07:38:34.907434356 +0000 UTC m=+0.506671542 container attach f759d4f312ea2c03449a25316acf7f937b55bb4140d5c646735809012371d2b9 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-osd-4-deactivate, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_CLEAN=True, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, org.label-schema.build-date=20240521, ceph=True, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, maintainer=Guillaume Abrioux ) 2024-05-27T07:38:35.096 INFO:journalctl@ceph.osd.7.smithi105.stdout:May 27 07:38:34 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-osd-7[53596]: 2024-05-27T07:38:34.783+0000 7f5eefdf1700 -1 osd.7 134 heartbeat_check: no reply from 172.21.15.105:6804 osd.1 since back 2024-05-27T07:38:06.798385+0000 front 2024-05-27T07:38:06.798353+0000 (oldest deadline 2024-05-27T07:38:32.098493+0000) 2024-05-27T07:38:35.109 INFO:journalctl@ceph.osd.8.smithi173.stdout:May 27 07:38:34 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-osd-8[55969]: 2024-05-27T07:38:34.911+0000 7f425b2ea700 -1 osd.8 134 heartbeat_check: no reply from 172.21.15.105:6804 osd.1 since back 2024-05-27T07:38:06.957078+0000 front 2024-05-27T07:38:06.957280+0000 (oldest deadline 2024-05-27T07:38:30.457072+0000) 2024-05-27T07:38:35.109 INFO:journalctl@ceph.osd.5.smithi173.stdout:May 27 07:38:34 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-osd-5[49009]: 2024-05-27T07:38:34.704+0000 7f331c3f3700 -1 osd.5 134 heartbeat_check: no reply from 172.21.15.52:6806 osd.0 since back 2024-05-27T07:37:57.916860+0000 front 2024-05-27T07:37:57.916876+0000 (oldest deadline 2024-05-27T07:38:22.616672+0000) 2024-05-27T07:38:35.109 INFO:journalctl@ceph.osd.5.smithi173.stdout:May 27 07:38:34 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-osd-5[49009]: 2024-05-27T07:38:34.704+0000 7f331c3f3700 -1 osd.5 134 heartbeat_check: no reply from 172.21.15.105:6804 osd.1 since back 2024-05-27T07:38:06.117796+0000 front 2024-05-27T07:38:06.117770+0000 (oldest deadline 2024-05-27T07:38:33.118474+0000) 2024-05-27T07:38:35.236 INFO:journalctl@ceph.osd.6.smithi052.stdout:May 27 07:38:34 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-osd-6[60312]: 2024-05-27T07:38:34.794+0000 7f362964d700 -1 osd.6 134 heartbeat_check: no reply from 172.21.15.52:6806 osd.0 since back 2024-05-27T07:37:58.099066+0000 front 2024-05-27T07:37:58.099034+0000 (oldest deadline 2024-05-27T07:38:23.399228+0000) 2024-05-27T07:38:35.236 INFO:journalctl@ceph.osd.6.smithi052.stdout:May 27 07:38:34 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-osd-6[60312]: 2024-05-27T07:38:34.794+0000 7f362964d700 -1 osd.6 134 heartbeat_check: no reply from 172.21.15.105:6804 osd.1 since back 2024-05-27T07:38:03.400081+0000 front 2024-05-27T07:38:03.400066+0000 (oldest deadline 2024-05-27T07:38:28.099925+0000) 2024-05-27T07:38:35.237 INFO:journalctl@ceph.osd.9.smithi052.stdout:May 27 07:38:34 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-osd-9[67134]: 2024-05-27T07:38:34.858+0000 7f1e3e2ca700 -1 osd.9 134 heartbeat_check: no reply from 172.21.15.52:6806 osd.0 since back 2024-05-27T07:37:58.861291+0000 front 2024-05-27T07:37:58.861258+0000 (oldest deadline 2024-05-27T07:38:22.361571+0000) 2024-05-27T07:38:35.237 INFO:journalctl@ceph.osd.9.smithi052.stdout:May 27 07:38:34 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-osd-9[67134]: 2024-05-27T07:38:34.858+0000 7f1e3e2ca700 -1 osd.9 134 heartbeat_check: no reply from 172.21.15.105:6804 osd.1 since back 2024-05-27T07:38:05.862960+0000 front 2024-05-27T07:38:05.862645+0000 (oldest deadline 2024-05-27T07:38:30.562694+0000) 2024-05-27T07:38:35.406 INFO:journalctl@ceph.osd.4.smithi105.stdout:May 27 07:38:35 smithi105 podman[72406]: 2024-05-27 07:38:35.099861112 +0000 UTC m=+0.699098294 container died f759d4f312ea2c03449a25316acf7f937b55bb4140d5c646735809012371d2b9 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-osd-4-deactivate, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, ceph=True, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240521, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, GIT_BRANCH=HEAD) 2024-05-27T07:38:35.406 INFO:journalctl@ceph.osd.10.smithi105.stdout:May 27 07:38:35 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-osd-10[60334]: 2024-05-27T07:38:35.206+0000 7f162fb8f700 -1 osd.10 134 heartbeat_check: no reply from 172.21.15.52:6806 osd.0 since back 2024-05-27T07:37:56.754985+0000 front 2024-05-27T07:37:56.755070+0000 (oldest deadline 2024-05-27T07:38:22.054911+0000) 2024-05-27T07:38:35.407 INFO:journalctl@ceph.osd.10.smithi105.stdout:May 27 07:38:35 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-osd-10[60334]: 2024-05-27T07:38:35.206+0000 7f162fb8f700 -1 osd.10 134 heartbeat_check: no reply from 172.21.15.105:6804 osd.1 since back 2024-05-27T07:38:05.556137+0000 front 2024-05-27T07:38:05.556112+0000 (oldest deadline 2024-05-27T07:38:31.456213+0000) 2024-05-27T07:38:35.742 INFO:journalctl@ceph.osd.11.smithi173.stdout:May 27 07:38:35 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-osd-11[63021]: 2024-05-27T07:38:35.443+0000 7f9d44ac8700 -1 osd.11 134 heartbeat_check: no reply from 172.21.15.52:6806 osd.0 since back 2024-05-27T07:37:57.927796+0000 front 2024-05-27T07:37:57.927759+0000 (oldest deadline 2024-05-27T07:38:22.028025+0000) 2024-05-27T07:38:35.743 INFO:journalctl@ceph.osd.11.smithi173.stdout:May 27 07:38:35 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-osd-11[63021]: 2024-05-27T07:38:35.443+0000 7f9d44ac8700 -1 osd.11 134 heartbeat_check: no reply from 172.21.15.105:6804 osd.1 since back 2024-05-27T07:38:03.129257+0000 front 2024-05-27T07:38:03.129120+0000 (oldest deadline 2024-05-27T07:38:28.929543+0000) 2024-05-27T07:38:35.743 DEBUG:teuthology.orchestra.run.smithi105:> sudo pkill -f 'journalctl -f -n 0 -u ceph-73df2636-1bf8-11ef-bc9b-c7b262605968@osd.4.service' 2024-05-27T07:38:35.782 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-05-27T07:38:35.783 INFO:tasks.cephadm.osd.4:Stopped osd.4 2024-05-27T07:38:35.783 INFO:tasks.cephadm.osd.5:Stopping osd.5... 2024-05-27T07:38:35.783 DEBUG:teuthology.orchestra.run.smithi173:> sudo systemctl stop ceph-73df2636-1bf8-11ef-bc9b-c7b262605968@osd.5 2024-05-27T07:38:36.105 INFO:journalctl@ceph.osd.8.smithi173.stdout:May 27 07:38:35 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-osd-8[55969]: 2024-05-27T07:38:35.897+0000 7f425b2ea700 -1 osd.8 134 heartbeat_check: no reply from 172.21.15.105:6804 osd.1 since back 2024-05-27T07:38:06.957078+0000 front 2024-05-27T07:38:06.957280+0000 (oldest deadline 2024-05-27T07:38:30.457072+0000) 2024-05-27T07:38:36.106 INFO:journalctl@ceph.osd.5.smithi173.stdout:May 27 07:38:35 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-osd-5[49009]: 2024-05-27T07:38:35.746+0000 7f331c3f3700 -1 osd.5 134 heartbeat_check: no reply from 172.21.15.52:6806 osd.0 since back 2024-05-27T07:37:57.916860+0000 front 2024-05-27T07:37:57.916876+0000 (oldest deadline 2024-05-27T07:38:22.616672+0000) 2024-05-27T07:38:36.106 INFO:journalctl@ceph.osd.5.smithi173.stdout:May 27 07:38:35 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-osd-5[49009]: 2024-05-27T07:38:35.746+0000 7f331c3f3700 -1 osd.5 134 heartbeat_check: no reply from 172.21.15.105:6804 osd.1 since back 2024-05-27T07:38:06.117796+0000 front 2024-05-27T07:38:06.117770+0000 (oldest deadline 2024-05-27T07:38:33.118474+0000) 2024-05-27T07:38:36.106 INFO:journalctl@ceph.osd.5.smithi173.stdout:May 27 07:38:35 smithi173 systemd[1]: Stopping Ceph osd.5 for 73df2636-1bf8-11ef-bc9b-c7b262605968... 2024-05-27T07:38:36.197 INFO:journalctl@ceph.osd.7.smithi105.stdout:May 27 07:38:35 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-osd-7[53596]: 2024-05-27T07:38:35.771+0000 7f5eefdf1700 -1 osd.7 134 heartbeat_check: no reply from 172.21.15.105:6804 osd.1 since back 2024-05-27T07:38:06.798385+0000 front 2024-05-27T07:38:06.798353+0000 (oldest deadline 2024-05-27T07:38:32.098493+0000) 2024-05-27T07:38:36.237 INFO:journalctl@ceph.osd.6.smithi052.stdout:May 27 07:38:35 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-osd-6[60312]: 2024-05-27T07:38:35.822+0000 7f362964d700 -1 osd.6 134 heartbeat_check: no reply from 172.21.15.52:6806 osd.0 since back 2024-05-27T07:37:58.099066+0000 front 2024-05-27T07:37:58.099034+0000 (oldest deadline 2024-05-27T07:38:23.399228+0000) 2024-05-27T07:38:36.237 INFO:journalctl@ceph.osd.6.smithi052.stdout:May 27 07:38:35 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-osd-6[60312]: 2024-05-27T07:38:35.822+0000 7f362964d700 -1 osd.6 134 heartbeat_check: no reply from 172.21.15.105:6804 osd.1 since back 2024-05-27T07:38:03.400081+0000 front 2024-05-27T07:38:03.400066+0000 (oldest deadline 2024-05-27T07:38:28.099925+0000) 2024-05-27T07:38:36.237 INFO:journalctl@ceph.osd.9.smithi052.stdout:May 27 07:38:35 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-osd-9[67134]: 2024-05-27T07:38:35.842+0000 7f1e3e2ca700 -1 osd.9 134 heartbeat_check: no reply from 172.21.15.52:6806 osd.0 since back 2024-05-27T07:37:58.861291+0000 front 2024-05-27T07:37:58.861258+0000 (oldest deadline 2024-05-27T07:38:22.361571+0000) 2024-05-27T07:38:36.238 INFO:journalctl@ceph.osd.9.smithi052.stdout:May 27 07:38:35 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-osd-9[67134]: 2024-05-27T07:38:35.842+0000 7f1e3e2ca700 -1 osd.9 134 heartbeat_check: no reply from 172.21.15.105:6804 osd.1 since back 2024-05-27T07:38:05.862960+0000 front 2024-05-27T07:38:05.862645+0000 (oldest deadline 2024-05-27T07:38:30.562694+0000) 2024-05-27T07:38:36.359 INFO:journalctl@ceph.osd.5.smithi173.stdout:May 27 07:38:36 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-osd-5[49009]: 2024-05-27T07:38:36.108+0000 7f331fe3d700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.5 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-05-27T07:38:36.359 INFO:journalctl@ceph.osd.5.smithi173.stdout:May 27 07:38:36 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-osd-5[49009]: 2024-05-27T07:38:36.108+0000 7f331fe3d700 -1 osd.5 134 *** Got signal Terminated *** 2024-05-27T07:38:36.359 INFO:journalctl@ceph.osd.5.smithi173.stdout:May 27 07:38:36 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-osd-5[49009]: 2024-05-27T07:38:36.108+0000 7f331fe3d700 -1 osd.5 134 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-05-27T07:38:36.697 INFO:journalctl@ceph.osd.10.smithi105.stdout:May 27 07:38:36 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-osd-10[60334]: 2024-05-27T07:38:36.203+0000 7f162fb8f700 -1 osd.10 134 heartbeat_check: no reply from 172.21.15.52:6806 osd.0 since back 2024-05-27T07:37:56.754985+0000 front 2024-05-27T07:37:56.755070+0000 (oldest deadline 2024-05-27T07:38:22.054911+0000) 2024-05-27T07:38:36.697 INFO:journalctl@ceph.osd.10.smithi105.stdout:May 27 07:38:36 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-osd-10[60334]: 2024-05-27T07:38:36.203+0000 7f162fb8f700 -1 osd.10 134 heartbeat_check: no reply from 172.21.15.105:6804 osd.1 since back 2024-05-27T07:38:05.556137+0000 front 2024-05-27T07:38:05.556112+0000 (oldest deadline 2024-05-27T07:38:31.456213+0000) 2024-05-27T07:38:36.740 INFO:journalctl@ceph.osd.11.smithi173.stdout:May 27 07:38:36 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-osd-11[63021]: 2024-05-27T07:38:36.458+0000 7f9d44ac8700 -1 osd.11 134 heartbeat_check: no reply from 172.21.15.52:6806 osd.0 since back 2024-05-27T07:37:57.927796+0000 front 2024-05-27T07:37:57.927759+0000 (oldest deadline 2024-05-27T07:38:22.028025+0000) 2024-05-27T07:38:36.741 INFO:journalctl@ceph.osd.11.smithi173.stdout:May 27 07:38:36 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-osd-11[63021]: 2024-05-27T07:38:36.458+0000 7f9d44ac8700 -1 osd.11 134 heartbeat_check: no reply from 172.21.15.105:6804 osd.1 since back 2024-05-27T07:38:03.129257+0000 front 2024-05-27T07:38:03.129120+0000 (oldest deadline 2024-05-27T07:38:28.929543+0000) 2024-05-27T07:38:37.109 INFO:journalctl@ceph.osd.8.smithi173.stdout:May 27 07:38:36 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-osd-8[55969]: 2024-05-27T07:38:36.908+0000 7f425b2ea700 -1 osd.8 134 heartbeat_check: no reply from 172.21.15.105:6804 osd.1 since back 2024-05-27T07:38:06.957078+0000 front 2024-05-27T07:38:06.957280+0000 (oldest deadline 2024-05-27T07:38:30.457072+0000) 2024-05-27T07:38:37.109 INFO:journalctl@ceph.osd.5.smithi173.stdout:May 27 07:38:36 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-osd-5[49009]: 2024-05-27T07:38:36.744+0000 7f331c3f3700 -1 osd.5 134 heartbeat_check: no reply from 172.21.15.52:6806 osd.0 since back 2024-05-27T07:37:57.916860+0000 front 2024-05-27T07:37:57.916876+0000 (oldest deadline 2024-05-27T07:38:22.616672+0000) 2024-05-27T07:38:37.109 INFO:journalctl@ceph.osd.5.smithi173.stdout:May 27 07:38:36 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-osd-5[49009]: 2024-05-27T07:38:36.744+0000 7f331c3f3700 -1 osd.5 134 heartbeat_check: no reply from 172.21.15.105:6804 osd.1 since back 2024-05-27T07:38:06.117796+0000 front 2024-05-27T07:38:06.117770+0000 (oldest deadline 2024-05-27T07:38:33.118474+0000) 2024-05-27T07:38:37.197 INFO:journalctl@ceph.osd.7.smithi105.stdout:May 27 07:38:36 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-osd-7[53596]: 2024-05-27T07:38:36.790+0000 7f5eefdf1700 -1 osd.7 134 heartbeat_check: no reply from 172.21.15.105:6804 osd.1 since back 2024-05-27T07:38:06.798385+0000 front 2024-05-27T07:38:06.798353+0000 (oldest deadline 2024-05-27T07:38:32.098493+0000) 2024-05-27T07:38:37.236 INFO:journalctl@ceph.osd.6.smithi052.stdout:May 27 07:38:36 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-osd-6[60312]: 2024-05-27T07:38:36.856+0000 7f362964d700 -1 osd.6 134 heartbeat_check: no reply from 172.21.15.52:6806 osd.0 since back 2024-05-27T07:37:58.099066+0000 front 2024-05-27T07:37:58.099034+0000 (oldest deadline 2024-05-27T07:38:23.399228+0000) 2024-05-27T07:38:37.237 INFO:journalctl@ceph.osd.6.smithi052.stdout:May 27 07:38:36 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-osd-6[60312]: 2024-05-27T07:38:36.856+0000 7f362964d700 -1 osd.6 134 heartbeat_check: no reply from 172.21.15.105:6804 osd.1 since back 2024-05-27T07:38:03.400081+0000 front 2024-05-27T07:38:03.400066+0000 (oldest deadline 2024-05-27T07:38:28.099925+0000) 2024-05-27T07:38:37.237 INFO:journalctl@ceph.osd.9.smithi052.stdout:May 27 07:38:36 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-osd-9[67134]: 2024-05-27T07:38:36.797+0000 7f1e3e2ca700 -1 osd.9 134 heartbeat_check: no reply from 172.21.15.52:6806 osd.0 since back 2024-05-27T07:37:58.861291+0000 front 2024-05-27T07:37:58.861258+0000 (oldest deadline 2024-05-27T07:38:22.361571+0000) 2024-05-27T07:38:37.237 INFO:journalctl@ceph.osd.9.smithi052.stdout:May 27 07:38:36 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-osd-9[67134]: 2024-05-27T07:38:36.797+0000 7f1e3e2ca700 -1 osd.9 134 heartbeat_check: no reply from 172.21.15.105:6804 osd.1 since back 2024-05-27T07:38:05.862960+0000 front 2024-05-27T07:38:05.862645+0000 (oldest deadline 2024-05-27T07:38:30.562694+0000) 2024-05-27T07:38:37.697 INFO:journalctl@ceph.osd.10.smithi105.stdout:May 27 07:38:37 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-osd-10[60334]: 2024-05-27T07:38:37.211+0000 7f162fb8f700 -1 osd.10 134 heartbeat_check: no reply from 172.21.15.52:6806 osd.0 since back 2024-05-27T07:37:56.754985+0000 front 2024-05-27T07:37:56.755070+0000 (oldest deadline 2024-05-27T07:38:22.054911+0000) 2024-05-27T07:38:37.697 INFO:journalctl@ceph.osd.10.smithi105.stdout:May 27 07:38:37 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-osd-10[60334]: 2024-05-27T07:38:37.211+0000 7f162fb8f700 -1 osd.10 134 heartbeat_check: no reply from 172.21.15.105:6804 osd.1 since back 2024-05-27T07:38:05.556137+0000 front 2024-05-27T07:38:05.556112+0000 (oldest deadline 2024-05-27T07:38:31.456213+0000) 2024-05-27T07:38:37.749 INFO:journalctl@ceph.osd.11.smithi173.stdout:May 27 07:38:37 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-osd-11[63021]: 2024-05-27T07:38:37.495+0000 7f9d44ac8700 -1 osd.11 134 heartbeat_check: no reply from 172.21.15.52:6806 osd.0 since back 2024-05-27T07:37:57.927796+0000 front 2024-05-27T07:37:57.927759+0000 (oldest deadline 2024-05-27T07:38:22.028025+0000) 2024-05-27T07:38:37.750 INFO:journalctl@ceph.osd.11.smithi173.stdout:May 27 07:38:37 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-osd-11[63021]: 2024-05-27T07:38:37.495+0000 7f9d44ac8700 -1 osd.11 134 heartbeat_check: no reply from 172.21.15.105:6804 osd.1 since back 2024-05-27T07:38:03.129257+0000 front 2024-05-27T07:38:03.129120+0000 (oldest deadline 2024-05-27T07:38:28.929543+0000) 2024-05-27T07:38:38.112 INFO:journalctl@ceph.osd.8.smithi173.stdout:May 27 07:38:37 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-osd-8[55969]: 2024-05-27T07:38:37.952+0000 7f425b2ea700 -1 osd.8 134 heartbeat_check: no reply from 172.21.15.105:6804 osd.1 since back 2024-05-27T07:38:06.957078+0000 front 2024-05-27T07:38:06.957280+0000 (oldest deadline 2024-05-27T07:38:30.457072+0000) 2024-05-27T07:38:38.112 INFO:journalctl@ceph.osd.5.smithi173.stdout:May 27 07:38:37 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-osd-5[49009]: 2024-05-27T07:38:37.752+0000 7f331c3f3700 -1 osd.5 134 heartbeat_check: no reply from 172.21.15.52:6806 osd.0 since back 2024-05-27T07:37:57.916860+0000 front 2024-05-27T07:37:57.916876+0000 (oldest deadline 2024-05-27T07:38:22.616672+0000) 2024-05-27T07:38:38.112 INFO:journalctl@ceph.osd.5.smithi173.stdout:May 27 07:38:37 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-osd-5[49009]: 2024-05-27T07:38:37.752+0000 7f331c3f3700 -1 osd.5 134 heartbeat_check: no reply from 172.21.15.105:6804 osd.1 since back 2024-05-27T07:38:06.117796+0000 front 2024-05-27T07:38:06.117770+0000 (oldest deadline 2024-05-27T07:38:33.118474+0000) 2024-05-27T07:38:38.201 INFO:journalctl@ceph.osd.7.smithi105.stdout:May 27 07:38:37 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-osd-7[53596]: 2024-05-27T07:38:37.802+0000 7f5eefdf1700 -1 osd.7 134 heartbeat_check: no reply from 172.21.15.105:6804 osd.1 since back 2024-05-27T07:38:06.798385+0000 front 2024-05-27T07:38:06.798353+0000 (oldest deadline 2024-05-27T07:38:32.098493+0000) 2024-05-27T07:38:38.237 INFO:journalctl@ceph.osd.6.smithi052.stdout:May 27 07:38:37 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-osd-6[60312]: 2024-05-27T07:38:37.849+0000 7f362964d700 -1 osd.6 134 heartbeat_check: no reply from 172.21.15.52:6806 osd.0 since back 2024-05-27T07:37:58.099066+0000 front 2024-05-27T07:37:58.099034+0000 (oldest deadline 2024-05-27T07:38:23.399228+0000) 2024-05-27T07:38:38.237 INFO:journalctl@ceph.osd.6.smithi052.stdout:May 27 07:38:37 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-osd-6[60312]: 2024-05-27T07:38:37.849+0000 7f362964d700 -1 osd.6 134 heartbeat_check: no reply from 172.21.15.105:6804 osd.1 since back 2024-05-27T07:38:03.400081+0000 front 2024-05-27T07:38:03.400066+0000 (oldest deadline 2024-05-27T07:38:28.099925+0000) 2024-05-27T07:38:38.237 INFO:journalctl@ceph.osd.9.smithi052.stdout:May 27 07:38:37 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-osd-9[67134]: 2024-05-27T07:38:37.767+0000 7f1e3e2ca700 -1 osd.9 134 heartbeat_check: no reply from 172.21.15.52:6806 osd.0 since back 2024-05-27T07:37:58.861291+0000 front 2024-05-27T07:37:58.861258+0000 (oldest deadline 2024-05-27T07:38:22.361571+0000) 2024-05-27T07:38:38.237 INFO:journalctl@ceph.osd.9.smithi052.stdout:May 27 07:38:37 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-osd-9[67134]: 2024-05-27T07:38:37.767+0000 7f1e3e2ca700 -1 osd.9 134 heartbeat_check: no reply from 172.21.15.105:6804 osd.1 since back 2024-05-27T07:38:05.862960+0000 front 2024-05-27T07:38:05.862645+0000 (oldest deadline 2024-05-27T07:38:30.562694+0000) 2024-05-27T07:38:38.697 INFO:journalctl@ceph.osd.10.smithi105.stdout:May 27 07:38:38 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-osd-10[60334]: 2024-05-27T07:38:38.216+0000 7f162fb8f700 -1 osd.10 134 heartbeat_check: no reply from 172.21.15.52:6806 osd.0 since back 2024-05-27T07:37:56.754985+0000 front 2024-05-27T07:37:56.755070+0000 (oldest deadline 2024-05-27T07:38:22.054911+0000) 2024-05-27T07:38:38.697 INFO:journalctl@ceph.osd.10.smithi105.stdout:May 27 07:38:38 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-osd-10[60334]: 2024-05-27T07:38:38.216+0000 7f162fb8f700 -1 osd.10 134 heartbeat_check: no reply from 172.21.15.105:6804 osd.1 since back 2024-05-27T07:38:05.556137+0000 front 2024-05-27T07:38:05.556112+0000 (oldest deadline 2024-05-27T07:38:31.456213+0000) 2024-05-27T07:38:38.760 INFO:journalctl@ceph.osd.11.smithi173.stdout:May 27 07:38:38 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-osd-11[63021]: 2024-05-27T07:38:38.485+0000 7f9d44ac8700 -1 osd.11 134 heartbeat_check: no reply from 172.21.15.52:6806 osd.0 since back 2024-05-27T07:37:57.927796+0000 front 2024-05-27T07:37:57.927759+0000 (oldest deadline 2024-05-27T07:38:22.028025+0000) 2024-05-27T07:38:38.760 INFO:journalctl@ceph.osd.11.smithi173.stdout:May 27 07:38:38 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-osd-11[63021]: 2024-05-27T07:38:38.485+0000 7f9d44ac8700 -1 osd.11 134 heartbeat_check: no reply from 172.21.15.105:6804 osd.1 since back 2024-05-27T07:38:03.129257+0000 front 2024-05-27T07:38:03.129120+0000 (oldest deadline 2024-05-27T07:38:28.929543+0000) 2024-05-27T07:38:39.109 INFO:journalctl@ceph.osd.8.smithi173.stdout:May 27 07:38:38 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-osd-8[55969]: 2024-05-27T07:38:38.926+0000 7f425b2ea700 -1 osd.8 134 heartbeat_check: no reply from 172.21.15.105:6804 osd.1 since back 2024-05-27T07:38:06.957078+0000 front 2024-05-27T07:38:06.957280+0000 (oldest deadline 2024-05-27T07:38:30.457072+0000) 2024-05-27T07:38:39.109 INFO:journalctl@ceph.osd.5.smithi173.stdout:May 27 07:38:38 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-osd-5[49009]: 2024-05-27T07:38:38.762+0000 7f331c3f3700 -1 osd.5 134 heartbeat_check: no reply from 172.21.15.52:6806 osd.0 since back 2024-05-27T07:37:57.916860+0000 front 2024-05-27T07:37:57.916876+0000 (oldest deadline 2024-05-27T07:38:22.616672+0000) 2024-05-27T07:38:39.109 INFO:journalctl@ceph.osd.5.smithi173.stdout:May 27 07:38:38 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-osd-5[49009]: 2024-05-27T07:38:38.762+0000 7f331c3f3700 -1 osd.5 134 heartbeat_check: no reply from 172.21.15.105:6804 osd.1 since back 2024-05-27T07:38:06.117796+0000 front 2024-05-27T07:38:06.117770+0000 (oldest deadline 2024-05-27T07:38:33.118474+0000) 2024-05-27T07:38:39.197 INFO:journalctl@ceph.osd.7.smithi105.stdout:May 27 07:38:38 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-osd-7[53596]: 2024-05-27T07:38:38.802+0000 7f5eefdf1700 -1 osd.7 134 heartbeat_check: no reply from 172.21.15.105:6804 osd.1 since back 2024-05-27T07:38:06.798385+0000 front 2024-05-27T07:38:06.798353+0000 (oldest deadline 2024-05-27T07:38:32.098493+0000) 2024-05-27T07:38:39.197 INFO:journalctl@ceph.osd.7.smithi105.stdout:May 27 07:38:38 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-osd-7[53596]: 2024-05-27T07:38:38.802+0000 7f5eefdf1700 -1 osd.7 134 heartbeat_check: no reply from 172.21.15.173:6804 osd.2 since back 2024-05-27T07:38:13.199843+0000 front 2024-05-27T07:38:13.199692+0000 (oldest deadline 2024-05-27T07:38:37.899787+0000) 2024-05-27T07:38:39.237 INFO:journalctl@ceph.osd.6.smithi052.stdout:May 27 07:38:38 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-osd-6[60312]: 2024-05-27T07:38:38.848+0000 7f362964d700 -1 osd.6 134 heartbeat_check: no reply from 172.21.15.52:6806 osd.0 since back 2024-05-27T07:37:58.099066+0000 front 2024-05-27T07:37:58.099034+0000 (oldest deadline 2024-05-27T07:38:23.399228+0000) 2024-05-27T07:38:39.237 INFO:journalctl@ceph.osd.6.smithi052.stdout:May 27 07:38:38 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-osd-6[60312]: 2024-05-27T07:38:38.848+0000 7f362964d700 -1 osd.6 134 heartbeat_check: no reply from 172.21.15.105:6804 osd.1 since back 2024-05-27T07:38:03.400081+0000 front 2024-05-27T07:38:03.400066+0000 (oldest deadline 2024-05-27T07:38:28.099925+0000) 2024-05-27T07:38:39.237 INFO:journalctl@ceph.osd.9.smithi052.stdout:May 27 07:38:38 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-osd-9[67134]: 2024-05-27T07:38:38.799+0000 7f1e3e2ca700 -1 osd.9 134 heartbeat_check: no reply from 172.21.15.52:6806 osd.0 since back 2024-05-27T07:37:58.861291+0000 front 2024-05-27T07:37:58.861258+0000 (oldest deadline 2024-05-27T07:38:22.361571+0000) 2024-05-27T07:38:39.237 INFO:journalctl@ceph.osd.9.smithi052.stdout:May 27 07:38:38 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-osd-9[67134]: 2024-05-27T07:38:38.799+0000 7f1e3e2ca700 -1 osd.9 134 heartbeat_check: no reply from 172.21.15.105:6804 osd.1 since back 2024-05-27T07:38:05.862960+0000 front 2024-05-27T07:38:05.862645+0000 (oldest deadline 2024-05-27T07:38:30.562694+0000) 2024-05-27T07:38:39.237 INFO:journalctl@ceph.osd.9.smithi052.stdout:May 27 07:38:38 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-osd-9[67134]: 2024-05-27T07:38:38.799+0000 7f1e3e2ca700 -1 osd.9 134 heartbeat_check: no reply from 172.21.15.173:6804 osd.2 since back 2024-05-27T07:38:15.566595+0000 front 2024-05-27T07:38:15.566403+0000 (oldest deadline 2024-05-27T07:38:37.866552+0000) 2024-05-27T07:38:39.697 INFO:journalctl@ceph.osd.10.smithi105.stdout:May 27 07:38:39 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-osd-10[60334]: 2024-05-27T07:38:39.254+0000 7f162fb8f700 -1 osd.10 134 heartbeat_check: no reply from 172.21.15.52:6806 osd.0 since back 2024-05-27T07:37:56.754985+0000 front 2024-05-27T07:37:56.755070+0000 (oldest deadline 2024-05-27T07:38:22.054911+0000) 2024-05-27T07:38:39.697 INFO:journalctl@ceph.osd.10.smithi105.stdout:May 27 07:38:39 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-osd-10[60334]: 2024-05-27T07:38:39.254+0000 7f162fb8f700 -1 osd.10 134 heartbeat_check: no reply from 172.21.15.105:6804 osd.1 since back 2024-05-27T07:38:05.556137+0000 front 2024-05-27T07:38:05.556112+0000 (oldest deadline 2024-05-27T07:38:31.456213+0000) 2024-05-27T07:38:39.767 INFO:journalctl@ceph.osd.11.smithi173.stdout:May 27 07:38:39 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-osd-11[63021]: 2024-05-27T07:38:39.476+0000 7f9d44ac8700 -1 osd.11 134 heartbeat_check: no reply from 172.21.15.52:6806 osd.0 since back 2024-05-27T07:37:57.927796+0000 front 2024-05-27T07:37:57.927759+0000 (oldest deadline 2024-05-27T07:38:22.028025+0000) 2024-05-27T07:38:39.767 INFO:journalctl@ceph.osd.11.smithi173.stdout:May 27 07:38:39 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-osd-11[63021]: 2024-05-27T07:38:39.476+0000 7f9d44ac8700 -1 osd.11 134 heartbeat_check: no reply from 172.21.15.105:6804 osd.1 since back 2024-05-27T07:38:03.129257+0000 front 2024-05-27T07:38:03.129120+0000 (oldest deadline 2024-05-27T07:38:28.929543+0000) 2024-05-27T07:38:40.109 INFO:journalctl@ceph.osd.8.smithi173.stdout:May 27 07:38:39 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-osd-8[55969]: 2024-05-27T07:38:39.911+0000 7f425b2ea700 -1 osd.8 134 heartbeat_check: no reply from 172.21.15.105:6804 osd.1 since back 2024-05-27T07:38:06.957078+0000 front 2024-05-27T07:38:06.957280+0000 (oldest deadline 2024-05-27T07:38:30.457072+0000) 2024-05-27T07:38:40.109 INFO:journalctl@ceph.osd.5.smithi173.stdout:May 27 07:38:39 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-osd-5[49009]: 2024-05-27T07:38:39.770+0000 7f331c3f3700 -1 osd.5 134 heartbeat_check: no reply from 172.21.15.52:6806 osd.0 since back 2024-05-27T07:37:57.916860+0000 front 2024-05-27T07:37:57.916876+0000 (oldest deadline 2024-05-27T07:38:22.616672+0000) 2024-05-27T07:38:40.109 INFO:journalctl@ceph.osd.5.smithi173.stdout:May 27 07:38:39 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-osd-5[49009]: 2024-05-27T07:38:39.770+0000 7f331c3f3700 -1 osd.5 134 heartbeat_check: no reply from 172.21.15.105:6804 osd.1 since back 2024-05-27T07:38:06.117796+0000 front 2024-05-27T07:38:06.117770+0000 (oldest deadline 2024-05-27T07:38:33.118474+0000) 2024-05-27T07:38:40.109 INFO:journalctl@ceph.osd.5.smithi173.stdout:May 27 07:38:39 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-osd-5[49009]: 2024-05-27T07:38:39.770+0000 7f331c3f3700 -1 osd.5 134 heartbeat_check: no reply from 172.21.15.173:6804 osd.2 since back 2024-05-27T07:38:13.619458+0000 front 2024-05-27T07:38:13.619490+0000 (oldest deadline 2024-05-27T07:38:38.919616+0000) 2024-05-27T07:38:40.197 INFO:journalctl@ceph.osd.7.smithi105.stdout:May 27 07:38:39 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-osd-7[53596]: 2024-05-27T07:38:39.814+0000 7f5eefdf1700 -1 osd.7 134 heartbeat_check: no reply from 172.21.15.105:6804 osd.1 since back 2024-05-27T07:38:06.798385+0000 front 2024-05-27T07:38:06.798353+0000 (oldest deadline 2024-05-27T07:38:32.098493+0000) 2024-05-27T07:38:40.197 INFO:journalctl@ceph.osd.7.smithi105.stdout:May 27 07:38:39 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-osd-7[53596]: 2024-05-27T07:38:39.814+0000 7f5eefdf1700 -1 osd.7 134 heartbeat_check: no reply from 172.21.15.173:6804 osd.2 since back 2024-05-27T07:38:13.199843+0000 front 2024-05-27T07:38:13.199692+0000 (oldest deadline 2024-05-27T07:38:37.899787+0000) 2024-05-27T07:38:40.236 INFO:journalctl@ceph.osd.6.smithi052.stdout:May 27 07:38:39 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-osd-6[60312]: 2024-05-27T07:38:39.874+0000 7f362964d700 -1 osd.6 134 heartbeat_check: no reply from 172.21.15.52:6806 osd.0 since back 2024-05-27T07:37:58.099066+0000 front 2024-05-27T07:37:58.099034+0000 (oldest deadline 2024-05-27T07:38:23.399228+0000) 2024-05-27T07:38:40.236 INFO:journalctl@ceph.osd.6.smithi052.stdout:May 27 07:38:39 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-osd-6[60312]: 2024-05-27T07:38:39.874+0000 7f362964d700 -1 osd.6 134 heartbeat_check: no reply from 172.21.15.105:6804 osd.1 since back 2024-05-27T07:38:03.400081+0000 front 2024-05-27T07:38:03.400066+0000 (oldest deadline 2024-05-27T07:38:28.099925+0000) 2024-05-27T07:38:40.236 INFO:journalctl@ceph.osd.9.smithi052.stdout:May 27 07:38:39 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-osd-9[67134]: 2024-05-27T07:38:39.757+0000 7f1e3e2ca700 -1 osd.9 134 heartbeat_check: no reply from 172.21.15.52:6806 osd.0 since back 2024-05-27T07:37:58.861291+0000 front 2024-05-27T07:37:58.861258+0000 (oldest deadline 2024-05-27T07:38:22.361571+0000) 2024-05-27T07:38:40.237 INFO:journalctl@ceph.osd.9.smithi052.stdout:May 27 07:38:39 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-osd-9[67134]: 2024-05-27T07:38:39.757+0000 7f1e3e2ca700 -1 osd.9 134 heartbeat_check: no reply from 172.21.15.105:6804 osd.1 since back 2024-05-27T07:38:05.862960+0000 front 2024-05-27T07:38:05.862645+0000 (oldest deadline 2024-05-27T07:38:30.562694+0000) 2024-05-27T07:38:40.237 INFO:journalctl@ceph.osd.9.smithi052.stdout:May 27 07:38:39 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-osd-9[67134]: 2024-05-27T07:38:39.757+0000 7f1e3e2ca700 -1 osd.9 134 heartbeat_check: no reply from 172.21.15.173:6804 osd.2 since back 2024-05-27T07:38:15.566595+0000 front 2024-05-27T07:38:15.566403+0000 (oldest deadline 2024-05-27T07:38:37.866552+0000) 2024-05-27T07:38:40.697 INFO:journalctl@ceph.osd.10.smithi105.stdout:May 27 07:38:40 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-osd-10[60334]: 2024-05-27T07:38:40.276+0000 7f162fb8f700 -1 osd.10 134 heartbeat_check: no reply from 172.21.15.52:6806 osd.0 since back 2024-05-27T07:37:56.754985+0000 front 2024-05-27T07:37:56.755070+0000 (oldest deadline 2024-05-27T07:38:22.054911+0000) 2024-05-27T07:38:40.697 INFO:journalctl@ceph.osd.10.smithi105.stdout:May 27 07:38:40 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-osd-10[60334]: 2024-05-27T07:38:40.276+0000 7f162fb8f700 -1 osd.10 134 heartbeat_check: no reply from 172.21.15.105:6804 osd.1 since back 2024-05-27T07:38:05.556137+0000 front 2024-05-27T07:38:05.556112+0000 (oldest deadline 2024-05-27T07:38:31.456213+0000) 2024-05-27T07:38:40.772 INFO:journalctl@ceph.osd.11.smithi173.stdout:May 27 07:38:40 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-osd-11[63021]: 2024-05-27T07:38:40.497+0000 7f9d44ac8700 -1 osd.11 134 heartbeat_check: no reply from 172.21.15.52:6806 osd.0 since back 2024-05-27T07:37:57.927796+0000 front 2024-05-27T07:37:57.927759+0000 (oldest deadline 2024-05-27T07:38:22.028025+0000) 2024-05-27T07:38:40.772 INFO:journalctl@ceph.osd.11.smithi173.stdout:May 27 07:38:40 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-osd-11[63021]: 2024-05-27T07:38:40.497+0000 7f9d44ac8700 -1 osd.11 134 heartbeat_check: no reply from 172.21.15.105:6804 osd.1 since back 2024-05-27T07:38:03.129257+0000 front 2024-05-27T07:38:03.129120+0000 (oldest deadline 2024-05-27T07:38:28.929543+0000) 2024-05-27T07:38:40.986 INFO:journalctl@ceph.osd.6.smithi052.stdout:May 27 07:38:40 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-osd-6[60312]: 2024-05-27T07:38:40.912+0000 7f362964d700 -1 osd.6 134 heartbeat_check: no reply from 172.21.15.52:6806 osd.0 since back 2024-05-27T07:37:58.099066+0000 front 2024-05-27T07:37:58.099034+0000 (oldest deadline 2024-05-27T07:38:23.399228+0000) 2024-05-27T07:38:40.986 INFO:journalctl@ceph.osd.6.smithi052.stdout:May 27 07:38:40 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-osd-6[60312]: 2024-05-27T07:38:40.912+0000 7f362964d700 -1 osd.6 134 heartbeat_check: no reply from 172.21.15.105:6804 osd.1 since back 2024-05-27T07:38:03.400081+0000 front 2024-05-27T07:38:03.400066+0000 (oldest deadline 2024-05-27T07:38:28.099925+0000) 2024-05-27T07:38:40.987 INFO:journalctl@ceph.osd.6.smithi052.stdout:May 27 07:38:40 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-osd-6[60312]: 2024-05-27T07:38:40.912+0000 7f362964d700 -1 osd.6 134 heartbeat_check: no reply from 172.21.15.173:6804 osd.2 since back 2024-05-27T07:38:15.702116+0000 front 2024-05-27T07:38:15.702087+0000 (oldest deadline 2024-05-27T07:38:40.402041+0000) 2024-05-27T07:38:40.987 INFO:journalctl@ceph.osd.9.smithi052.stdout:May 27 07:38:40 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-osd-9[67134]: 2024-05-27T07:38:40.729+0000 7f1e3e2ca700 -1 osd.9 134 heartbeat_check: no reply from 172.21.15.52:6806 osd.0 since back 2024-05-27T07:37:58.861291+0000 front 2024-05-27T07:37:58.861258+0000 (oldest deadline 2024-05-27T07:38:22.361571+0000) 2024-05-27T07:38:40.987 INFO:journalctl@ceph.osd.9.smithi052.stdout:May 27 07:38:40 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-osd-9[67134]: 2024-05-27T07:38:40.729+0000 7f1e3e2ca700 -1 osd.9 134 heartbeat_check: no reply from 172.21.15.105:6804 osd.1 since back 2024-05-27T07:38:05.862960+0000 front 2024-05-27T07:38:05.862645+0000 (oldest deadline 2024-05-27T07:38:30.562694+0000) 2024-05-27T07:38:40.987 INFO:journalctl@ceph.osd.9.smithi052.stdout:May 27 07:38:40 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-osd-9[67134]: 2024-05-27T07:38:40.729+0000 7f1e3e2ca700 -1 osd.9 134 heartbeat_check: no reply from 172.21.15.173:6804 osd.2 since back 2024-05-27T07:38:15.566595+0000 front 2024-05-27T07:38:15.566403+0000 (oldest deadline 2024-05-27T07:38:37.866552+0000) 2024-05-27T07:38:41.109 INFO:journalctl@ceph.osd.8.smithi173.stdout:May 27 07:38:40 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-osd-8[55969]: 2024-05-27T07:38:40.884+0000 7f425b2ea700 -1 osd.8 134 heartbeat_check: no reply from 172.21.15.105:6804 osd.1 since back 2024-05-27T07:38:06.957078+0000 front 2024-05-27T07:38:06.957280+0000 (oldest deadline 2024-05-27T07:38:30.457072+0000) 2024-05-27T07:38:41.109 INFO:journalctl@ceph.osd.5.smithi173.stdout:May 27 07:38:40 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-osd-5[49009]: 2024-05-27T07:38:40.775+0000 7f331c3f3700 -1 osd.5 134 heartbeat_check: no reply from 172.21.15.52:6806 osd.0 since back 2024-05-27T07:37:57.916860+0000 front 2024-05-27T07:37:57.916876+0000 (oldest deadline 2024-05-27T07:38:22.616672+0000) 2024-05-27T07:38:41.109 INFO:journalctl@ceph.osd.5.smithi173.stdout:May 27 07:38:40 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-osd-5[49009]: 2024-05-27T07:38:40.775+0000 7f331c3f3700 -1 osd.5 134 heartbeat_check: no reply from 172.21.15.105:6804 osd.1 since back 2024-05-27T07:38:06.117796+0000 front 2024-05-27T07:38:06.117770+0000 (oldest deadline 2024-05-27T07:38:33.118474+0000) 2024-05-27T07:38:41.109 INFO:journalctl@ceph.osd.5.smithi173.stdout:May 27 07:38:40 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-osd-5[49009]: 2024-05-27T07:38:40.775+0000 7f331c3f3700 -1 osd.5 134 heartbeat_check: no reply from 172.21.15.173:6804 osd.2 since back 2024-05-27T07:38:13.619458+0000 front 2024-05-27T07:38:13.619490+0000 (oldest deadline 2024-05-27T07:38:38.919616+0000) 2024-05-27T07:38:41.197 INFO:journalctl@ceph.osd.7.smithi105.stdout:May 27 07:38:40 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-osd-7[53596]: 2024-05-27T07:38:40.795+0000 7f5eefdf1700 -1 osd.7 134 heartbeat_check: no reply from 172.21.15.105:6804 osd.1 since back 2024-05-27T07:38:06.798385+0000 front 2024-05-27T07:38:06.798353+0000 (oldest deadline 2024-05-27T07:38:32.098493+0000) 2024-05-27T07:38:41.197 INFO:journalctl@ceph.osd.7.smithi105.stdout:May 27 07:38:40 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-osd-7[53596]: 2024-05-27T07:38:40.795+0000 7f5eefdf1700 -1 osd.7 134 heartbeat_check: no reply from 172.21.15.173:6804 osd.2 since back 2024-05-27T07:38:13.199843+0000 front 2024-05-27T07:38:13.199692+0000 (oldest deadline 2024-05-27T07:38:37.899787+0000) 2024-05-27T07:38:41.697 INFO:journalctl@ceph.osd.10.smithi105.stdout:May 27 07:38:41 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-osd-10[60334]: 2024-05-27T07:38:41.252+0000 7f162fb8f700 -1 osd.10 134 heartbeat_check: no reply from 172.21.15.52:6806 osd.0 since back 2024-05-27T07:37:56.754985+0000 front 2024-05-27T07:37:56.755070+0000 (oldest deadline 2024-05-27T07:38:22.054911+0000) 2024-05-27T07:38:41.697 INFO:journalctl@ceph.osd.10.smithi105.stdout:May 27 07:38:41 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-osd-10[60334]: 2024-05-27T07:38:41.252+0000 7f162fb8f700 -1 osd.10 134 heartbeat_check: no reply from 172.21.15.105:6804 osd.1 since back 2024-05-27T07:38:05.556137+0000 front 2024-05-27T07:38:05.556112+0000 (oldest deadline 2024-05-27T07:38:31.456213+0000) 2024-05-27T07:38:41.697 INFO:journalctl@ceph.osd.10.smithi105.stdout:May 27 07:38:41 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-osd-10[60334]: 2024-05-27T07:38:41.252+0000 7f162fb8f700 -1 osd.10 134 heartbeat_check: no reply from 172.21.15.173:6804 osd.2 since back 2024-05-27T07:38:15.557554+0000 front 2024-05-27T07:38:15.557409+0000 (oldest deadline 2024-05-27T07:38:40.857509+0000) 2024-05-27T07:38:41.859 INFO:journalctl@ceph.osd.5.smithi173.stdout:May 27 07:38:41 smithi173 podman[75896]: 2024-05-27 07:38:41.694416896 +0000 UTC m=+5.705247854 container died 1e1290f6e23aa9d41142f1ce7f0503a10851f073e6e044fb700007b92de1520a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-osd-5, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240521, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, GIT_CLEAN=True, ceph=True, GIT_BRANCH=HEAD, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0) 2024-05-27T07:38:41.859 INFO:journalctl@ceph.osd.11.smithi173.stdout:May 27 07:38:41 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-osd-11[63021]: 2024-05-27T07:38:41.516+0000 7f9d44ac8700 -1 osd.11 134 heartbeat_check: no reply from 172.21.15.52:6806 osd.0 since back 2024-05-27T07:37:57.927796+0000 front 2024-05-27T07:37:57.927759+0000 (oldest deadline 2024-05-27T07:38:22.028025+0000) 2024-05-27T07:38:41.859 INFO:journalctl@ceph.osd.11.smithi173.stdout:May 27 07:38:41 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-osd-11[63021]: 2024-05-27T07:38:41.516+0000 7f9d44ac8700 -1 osd.11 134 heartbeat_check: no reply from 172.21.15.105:6804 osd.1 since back 2024-05-27T07:38:03.129257+0000 front 2024-05-27T07:38:03.129120+0000 (oldest deadline 2024-05-27T07:38:28.929543+0000) 2024-05-27T07:38:42.197 INFO:journalctl@ceph.osd.7.smithi105.stdout:May 27 07:38:41 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-osd-7[53596]: 2024-05-27T07:38:41.791+0000 7f5eefdf1700 -1 osd.7 134 heartbeat_check: no reply from 172.21.15.105:6804 osd.1 since back 2024-05-27T07:38:06.798385+0000 front 2024-05-27T07:38:06.798353+0000 (oldest deadline 2024-05-27T07:38:32.098493+0000) 2024-05-27T07:38:42.197 INFO:journalctl@ceph.osd.7.smithi105.stdout:May 27 07:38:41 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-osd-7[53596]: 2024-05-27T07:38:41.791+0000 7f5eefdf1700 -1 osd.7 134 heartbeat_check: no reply from 172.21.15.173:6804 osd.2 since back 2024-05-27T07:38:13.199843+0000 front 2024-05-27T07:38:13.199692+0000 (oldest deadline 2024-05-27T07:38:37.899787+0000) 2024-05-27T07:38:42.236 INFO:journalctl@ceph.osd.6.smithi052.stdout:May 27 07:38:41 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-osd-6[60312]: 2024-05-27T07:38:41.882+0000 7f362964d700 -1 osd.6 134 heartbeat_check: no reply from 172.21.15.52:6806 osd.0 since back 2024-05-27T07:37:58.099066+0000 front 2024-05-27T07:37:58.099034+0000 (oldest deadline 2024-05-27T07:38:23.399228+0000) 2024-05-27T07:38:42.236 INFO:journalctl@ceph.osd.6.smithi052.stdout:May 27 07:38:41 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-osd-6[60312]: 2024-05-27T07:38:41.882+0000 7f362964d700 -1 osd.6 134 heartbeat_check: no reply from 172.21.15.105:6804 osd.1 since back 2024-05-27T07:38:03.400081+0000 front 2024-05-27T07:38:03.400066+0000 (oldest deadline 2024-05-27T07:38:28.099925+0000) 2024-05-27T07:38:42.236 INFO:journalctl@ceph.osd.6.smithi052.stdout:May 27 07:38:41 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-osd-6[60312]: 2024-05-27T07:38:41.882+0000 7f362964d700 -1 osd.6 134 heartbeat_check: no reply from 172.21.15.173:6804 osd.2 since back 2024-05-27T07:38:15.702116+0000 front 2024-05-27T07:38:15.702087+0000 (oldest deadline 2024-05-27T07:38:40.402041+0000) 2024-05-27T07:38:42.237 INFO:journalctl@ceph.osd.9.smithi052.stdout:May 27 07:38:41 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-osd-9[67134]: 2024-05-27T07:38:41.755+0000 7f1e3e2ca700 -1 osd.9 134 heartbeat_check: no reply from 172.21.15.52:6806 osd.0 since back 2024-05-27T07:37:58.861291+0000 front 2024-05-27T07:37:58.861258+0000 (oldest deadline 2024-05-27T07:38:22.361571+0000) 2024-05-27T07:38:42.237 INFO:journalctl@ceph.osd.9.smithi052.stdout:May 27 07:38:41 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-osd-9[67134]: 2024-05-27T07:38:41.755+0000 7f1e3e2ca700 -1 osd.9 134 heartbeat_check: no reply from 172.21.15.105:6804 osd.1 since back 2024-05-27T07:38:05.862960+0000 front 2024-05-27T07:38:05.862645+0000 (oldest deadline 2024-05-27T07:38:30.562694+0000) 2024-05-27T07:38:42.237 INFO:journalctl@ceph.osd.9.smithi052.stdout:May 27 07:38:41 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-osd-9[67134]: 2024-05-27T07:38:41.755+0000 7f1e3e2ca700 -1 osd.9 134 heartbeat_check: no reply from 172.21.15.173:6804 osd.2 since back 2024-05-27T07:38:15.566595+0000 front 2024-05-27T07:38:15.566403+0000 (oldest deadline 2024-05-27T07:38:37.866552+0000) 2024-05-27T07:38:42.323 INFO:journalctl@ceph.osd.8.smithi173.stdout:May 27 07:38:41 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-osd-8[55969]: 2024-05-27T07:38:41.873+0000 7f425b2ea700 -1 osd.8 134 heartbeat_check: no reply from 172.21.15.105:6804 osd.1 since back 2024-05-27T07:38:06.957078+0000 front 2024-05-27T07:38:06.957280+0000 (oldest deadline 2024-05-27T07:38:30.457072+0000) 2024-05-27T07:38:42.324 INFO:journalctl@ceph.osd.8.smithi173.stdout:May 27 07:38:41 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-osd-8[55969]: 2024-05-27T07:38:41.873+0000 7f425b2ea700 -1 osd.8 134 heartbeat_check: no reply from 172.21.15.173:6804 osd.2 since back 2024-05-27T07:38:16.258968+0000 front 2024-05-27T07:38:16.258998+0000 (oldest deadline 2024-05-27T07:38:41.559145+0000) 2024-05-27T07:38:42.324 INFO:journalctl@ceph.osd.5.smithi173.stdout:May 27 07:38:42 smithi173 podman[75896]: 2024-05-27 07:38:42.075872701 +0000 UTC m=+6.086703664 container cleanup 1e1290f6e23aa9d41142f1ce7f0503a10851f073e6e044fb700007b92de1520a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-osd-5, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, org.label-schema.build-date=20240521, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, ceph=True, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0) 2024-05-27T07:38:42.324 INFO:journalctl@ceph.osd.5.smithi173.stdout:May 27 07:38:42 smithi173 bash[75896]: ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-osd-5 2024-05-27T07:38:42.609 INFO:journalctl@ceph.osd.5.smithi173.stdout:May 27 07:38:42 smithi173 podman[75914]: 2024-05-27 07:38:42.327358141 +0000 UTC m=+0.629818075 container remove 1e1290f6e23aa9d41142f1ce7f0503a10851f073e6e044fb700007b92de1520a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-osd-5, GIT_BRANCH=HEAD, GIT_CLEAN=True, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, org.label-schema.build-date=20240521, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5) 2024-05-27T07:38:42.609 INFO:journalctl@ceph.osd.11.smithi173.stdout:May 27 07:38:42 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-osd-11[63021]: 2024-05-27T07:38:42.496+0000 7f9d44ac8700 -1 osd.11 134 heartbeat_check: no reply from 172.21.15.52:6806 osd.0 since back 2024-05-27T07:37:57.927796+0000 front 2024-05-27T07:37:57.927759+0000 (oldest deadline 2024-05-27T07:38:22.028025+0000) 2024-05-27T07:38:42.609 INFO:journalctl@ceph.osd.11.smithi173.stdout:May 27 07:38:42 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-osd-11[63021]: 2024-05-27T07:38:42.496+0000 7f9d44ac8700 -1 osd.11 134 heartbeat_check: no reply from 172.21.15.105:6804 osd.1 since back 2024-05-27T07:38:03.129257+0000 front 2024-05-27T07:38:03.129120+0000 (oldest deadline 2024-05-27T07:38:28.929543+0000) 2024-05-27T07:38:42.609 INFO:journalctl@ceph.osd.11.smithi173.stdout:May 27 07:38:42 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-osd-11[63021]: 2024-05-27T07:38:42.496+0000 7f9d44ac8700 -1 osd.11 134 heartbeat_check: no reply from 172.21.15.173:6804 osd.2 since back 2024-05-27T07:38:16.431856+0000 front 2024-05-27T07:38:16.431878+0000 (oldest deadline 2024-05-27T07:38:42.332069+0000) 2024-05-27T07:38:42.697 INFO:journalctl@ceph.osd.10.smithi105.stdout:May 27 07:38:42 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-osd-10[60334]: 2024-05-27T07:38:42.296+0000 7f162fb8f700 -1 osd.10 134 heartbeat_check: no reply from 172.21.15.52:6806 osd.0 since back 2024-05-27T07:37:56.754985+0000 front 2024-05-27T07:37:56.755070+0000 (oldest deadline 2024-05-27T07:38:22.054911+0000) 2024-05-27T07:38:42.697 INFO:journalctl@ceph.osd.10.smithi105.stdout:May 27 07:38:42 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-osd-10[60334]: 2024-05-27T07:38:42.296+0000 7f162fb8f700 -1 osd.10 134 heartbeat_check: no reply from 172.21.15.105:6804 osd.1 since back 2024-05-27T07:38:05.556137+0000 front 2024-05-27T07:38:05.556112+0000 (oldest deadline 2024-05-27T07:38:31.456213+0000) 2024-05-27T07:38:42.697 INFO:journalctl@ceph.osd.10.smithi105.stdout:May 27 07:38:42 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-osd-10[60334]: 2024-05-27T07:38:42.296+0000 7f162fb8f700 -1 osd.10 134 heartbeat_check: no reply from 172.21.15.173:6804 osd.2 since back 2024-05-27T07:38:15.557554+0000 front 2024-05-27T07:38:15.557409+0000 (oldest deadline 2024-05-27T07:38:40.857509+0000) 2024-05-27T07:38:42.961 INFO:journalctl@ceph.osd.8.smithi173.stdout:May 27 07:38:42 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-osd-8[55969]: 2024-05-27T07:38:42.869+0000 7f425b2ea700 -1 osd.8 134 heartbeat_check: no reply from 172.21.15.105:6804 osd.1 since back 2024-05-27T07:38:06.957078+0000 front 2024-05-27T07:38:06.957280+0000 (oldest deadline 2024-05-27T07:38:30.457072+0000) 2024-05-27T07:38:42.961 INFO:journalctl@ceph.osd.8.smithi173.stdout:May 27 07:38:42 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-osd-8[55969]: 2024-05-27T07:38:42.869+0000 7f425b2ea700 -1 osd.8 134 heartbeat_check: no reply from 172.21.15.173:6804 osd.2 since back 2024-05-27T07:38:16.258968+0000 front 2024-05-27T07:38:16.258998+0000 (oldest deadline 2024-05-27T07:38:41.559145+0000) 2024-05-27T07:38:42.961 INFO:journalctl@ceph.osd.5.smithi173.stdout:May 27 07:38:42 smithi173 podman[76004]: 2024-05-27 07:38:42.63670012 +0000 UTC m=+0.110979142 container create e0625a6fda3323481344002d7984bd4a48e2f193e18caa4f14b0bb11f89a6173 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-osd-5-deactivate, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240521, org.label-schema.license=GPLv2, ceph=True, GIT_CLEAN=True, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0) 2024-05-27T07:38:42.961 INFO:journalctl@ceph.osd.5.smithi173.stdout:May 27 07:38:42 smithi173 podman[76004]: 2024-05-27 07:38:42.542402046 +0000 UTC m=+0.016681079 image pull e36ea4fc8a5807e8d2701370e350a0de3856538797f7fba64cfce7e344bd6799 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6 2024-05-27T07:38:42.961 INFO:journalctl@ceph.osd.5.smithi173.stdout:May 27 07:38:42 smithi173 podman[76004]: 2024-05-27 07:38:42.872633004 +0000 UTC m=+0.346912040 container init e0625a6fda3323481344002d7984bd4a48e2f193e18caa4f14b0bb11f89a6173 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-osd-5-deactivate, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, GIT_CLEAN=True, GIT_BRANCH=HEAD, org.label-schema.build-date=20240521, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, io.buildah.version=1.33.5, ceph=True, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, org.label-schema.license=GPLv2) 2024-05-27T07:38:42.961 INFO:journalctl@ceph.osd.5.smithi173.stdout:May 27 07:38:42 smithi173 podman[76004]: 2024-05-27 07:38:42.875756298 +0000 UTC m=+0.350035332 container start e0625a6fda3323481344002d7984bd4a48e2f193e18caa4f14b0bb11f89a6173 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-osd-5-deactivate, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , GIT_CLEAN=True, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240521, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=) 2024-05-27T07:38:42.962 INFO:journalctl@ceph.osd.5.smithi173.stdout:May 27 07:38:42 smithi173 podman[76004]: 2024-05-27 07:38:42.964359448 +0000 UTC m=+0.438638490 container attach e0625a6fda3323481344002d7984bd4a48e2f193e18caa4f14b0bb11f89a6173 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-osd-5-deactivate, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , GIT_CLEAN=True, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, ceph=True, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240521, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=) 2024-05-27T07:38:43.197 INFO:journalctl@ceph.osd.7.smithi105.stdout:May 27 07:38:42 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-osd-7[53596]: 2024-05-27T07:38:42.743+0000 7f5eefdf1700 -1 osd.7 134 heartbeat_check: no reply from 172.21.15.105:6804 osd.1 since back 2024-05-27T07:38:06.798385+0000 front 2024-05-27T07:38:06.798353+0000 (oldest deadline 2024-05-27T07:38:32.098493+0000) 2024-05-27T07:38:43.197 INFO:journalctl@ceph.osd.7.smithi105.stdout:May 27 07:38:42 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-osd-7[53596]: 2024-05-27T07:38:42.743+0000 7f5eefdf1700 -1 osd.7 134 heartbeat_check: no reply from 172.21.15.173:6804 osd.2 since back 2024-05-27T07:38:13.199843+0000 front 2024-05-27T07:38:13.199692+0000 (oldest deadline 2024-05-27T07:38:37.899787+0000) 2024-05-27T07:38:43.236 INFO:journalctl@ceph.osd.6.smithi052.stdout:May 27 07:38:42 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-osd-6[60312]: 2024-05-27T07:38:42.875+0000 7f362964d700 -1 osd.6 134 heartbeat_check: no reply from 172.21.15.52:6806 osd.0 since back 2024-05-27T07:37:58.099066+0000 front 2024-05-27T07:37:58.099034+0000 (oldest deadline 2024-05-27T07:38:23.399228+0000) 2024-05-27T07:38:43.237 INFO:journalctl@ceph.osd.6.smithi052.stdout:May 27 07:38:42 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-osd-6[60312]: 2024-05-27T07:38:42.875+0000 7f362964d700 -1 osd.6 134 heartbeat_check: no reply from 172.21.15.105:6804 osd.1 since back 2024-05-27T07:38:03.400081+0000 front 2024-05-27T07:38:03.400066+0000 (oldest deadline 2024-05-27T07:38:28.099925+0000) 2024-05-27T07:38:43.237 INFO:journalctl@ceph.osd.6.smithi052.stdout:May 27 07:38:42 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-osd-6[60312]: 2024-05-27T07:38:42.875+0000 7f362964d700 -1 osd.6 134 heartbeat_check: no reply from 172.21.15.173:6804 osd.2 since back 2024-05-27T07:38:15.702116+0000 front 2024-05-27T07:38:15.702087+0000 (oldest deadline 2024-05-27T07:38:40.402041+0000) 2024-05-27T07:38:43.237 INFO:journalctl@ceph.osd.9.smithi052.stdout:May 27 07:38:42 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-osd-9[67134]: 2024-05-27T07:38:42.760+0000 7f1e3e2ca700 -1 osd.9 134 heartbeat_check: no reply from 172.21.15.52:6806 osd.0 since back 2024-05-27T07:37:58.861291+0000 front 2024-05-27T07:37:58.861258+0000 (oldest deadline 2024-05-27T07:38:22.361571+0000) 2024-05-27T07:38:43.237 INFO:journalctl@ceph.osd.9.smithi052.stdout:May 27 07:38:42 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-osd-9[67134]: 2024-05-27T07:38:42.760+0000 7f1e3e2ca700 -1 osd.9 134 heartbeat_check: no reply from 172.21.15.105:6804 osd.1 since back 2024-05-27T07:38:05.862960+0000 front 2024-05-27T07:38:05.862645+0000 (oldest deadline 2024-05-27T07:38:30.562694+0000) 2024-05-27T07:38:43.237 INFO:journalctl@ceph.osd.9.smithi052.stdout:May 27 07:38:42 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-osd-9[67134]: 2024-05-27T07:38:42.760+0000 7f1e3e2ca700 -1 osd.9 134 heartbeat_check: no reply from 172.21.15.173:6804 osd.2 since back 2024-05-27T07:38:15.566595+0000 front 2024-05-27T07:38:15.566403+0000 (oldest deadline 2024-05-27T07:38:37.866552+0000) 2024-05-27T07:38:43.361 INFO:journalctl@ceph.osd.5.smithi173.stdout:May 27 07:38:43 smithi173 podman[76004]: 2024-05-27 07:38:43.170363632 +0000 UTC m=+0.644642663 container died e0625a6fda3323481344002d7984bd4a48e2f193e18caa4f14b0bb11f89a6173 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-osd-5-deactivate, io.buildah.version=1.33.5, ceph=True, org.label-schema.schema-version=1.0, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240521, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image) 2024-05-27T07:38:43.697 INFO:journalctl@ceph.osd.10.smithi105.stdout:May 27 07:38:43 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-osd-10[60334]: 2024-05-27T07:38:43.316+0000 7f162fb8f700 -1 osd.10 134 heartbeat_check: no reply from 172.21.15.52:6806 osd.0 since back 2024-05-27T07:37:56.754985+0000 front 2024-05-27T07:37:56.755070+0000 (oldest deadline 2024-05-27T07:38:22.054911+0000) 2024-05-27T07:38:43.697 INFO:journalctl@ceph.osd.10.smithi105.stdout:May 27 07:38:43 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-osd-10[60334]: 2024-05-27T07:38:43.316+0000 7f162fb8f700 -1 osd.10 134 heartbeat_check: no reply from 172.21.15.105:6804 osd.1 since back 2024-05-27T07:38:05.556137+0000 front 2024-05-27T07:38:05.556112+0000 (oldest deadline 2024-05-27T07:38:31.456213+0000) 2024-05-27T07:38:43.697 INFO:journalctl@ceph.osd.10.smithi105.stdout:May 27 07:38:43 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-osd-10[60334]: 2024-05-27T07:38:43.316+0000 7f162fb8f700 -1 osd.10 134 heartbeat_check: no reply from 172.21.15.173:6804 osd.2 since back 2024-05-27T07:38:15.557554+0000 front 2024-05-27T07:38:15.557409+0000 (oldest deadline 2024-05-27T07:38:40.857509+0000) 2024-05-27T07:38:43.807 INFO:journalctl@ceph.osd.11.smithi173.stdout:May 27 07:38:43 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-osd-11[63021]: 2024-05-27T07:38:43.473+0000 7f9d44ac8700 -1 osd.11 134 heartbeat_check: no reply from 172.21.15.52:6806 osd.0 since back 2024-05-27T07:37:57.927796+0000 front 2024-05-27T07:37:57.927759+0000 (oldest deadline 2024-05-27T07:38:22.028025+0000) 2024-05-27T07:38:43.807 INFO:journalctl@ceph.osd.11.smithi173.stdout:May 27 07:38:43 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-osd-11[63021]: 2024-05-27T07:38:43.473+0000 7f9d44ac8700 -1 osd.11 134 heartbeat_check: no reply from 172.21.15.105:6804 osd.1 since back 2024-05-27T07:38:03.129257+0000 front 2024-05-27T07:38:03.129120+0000 (oldest deadline 2024-05-27T07:38:28.929543+0000) 2024-05-27T07:38:43.807 INFO:journalctl@ceph.osd.11.smithi173.stdout:May 27 07:38:43 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-osd-11[63021]: 2024-05-27T07:38:43.473+0000 7f9d44ac8700 -1 osd.11 134 heartbeat_check: no reply from 172.21.15.173:6804 osd.2 since back 2024-05-27T07:38:16.431856+0000 front 2024-05-27T07:38:16.431878+0000 (oldest deadline 2024-05-27T07:38:42.332069+0000) 2024-05-27T07:38:43.825 DEBUG:teuthology.orchestra.run.smithi173:> sudo pkill -f 'journalctl -f -n 0 -u ceph-73df2636-1bf8-11ef-bc9b-c7b262605968@osd.5.service' 2024-05-27T07:38:43.864 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-05-27T07:38:43.865 INFO:tasks.cephadm.osd.5:Stopped osd.5 2024-05-27T07:38:43.865 INFO:tasks.cephadm.osd.6:Stopping osd.6... 2024-05-27T07:38:43.865 DEBUG:teuthology.orchestra.run.smithi052:> sudo systemctl stop ceph-73df2636-1bf8-11ef-bc9b-c7b262605968@osd.6 2024-05-27T07:38:43.986 INFO:journalctl@ceph.osd.9.smithi052.stdout:May 27 07:38:43 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-osd-9[67134]: 2024-05-27T07:38:43.715+0000 7f1e3e2ca700 -1 osd.9 134 heartbeat_check: no reply from 172.21.15.52:6806 osd.0 since back 2024-05-27T07:37:58.861291+0000 front 2024-05-27T07:37:58.861258+0000 (oldest deadline 2024-05-27T07:38:22.361571+0000) 2024-05-27T07:38:43.986 INFO:journalctl@ceph.osd.9.smithi052.stdout:May 27 07:38:43 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-osd-9[67134]: 2024-05-27T07:38:43.715+0000 7f1e3e2ca700 -1 osd.9 134 heartbeat_check: no reply from 172.21.15.105:6804 osd.1 since back 2024-05-27T07:38:05.862960+0000 front 2024-05-27T07:38:05.862645+0000 (oldest deadline 2024-05-27T07:38:30.562694+0000) 2024-05-27T07:38:43.987 INFO:journalctl@ceph.osd.9.smithi052.stdout:May 27 07:38:43 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-osd-9[67134]: 2024-05-27T07:38:43.715+0000 7f1e3e2ca700 -1 osd.9 134 heartbeat_check: no reply from 172.21.15.173:6804 osd.2 since back 2024-05-27T07:38:15.566595+0000 front 2024-05-27T07:38:15.566403+0000 (oldest deadline 2024-05-27T07:38:37.866552+0000) 2024-05-27T07:38:43.987 INFO:journalctl@ceph.osd.6.smithi052.stdout:May 27 07:38:43 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-osd-6[60312]: 2024-05-27T07:38:43.863+0000 7f362964d700 -1 osd.6 134 heartbeat_check: no reply from 172.21.15.52:6806 osd.0 since back 2024-05-27T07:37:58.099066+0000 front 2024-05-27T07:37:58.099034+0000 (oldest deadline 2024-05-27T07:38:23.399228+0000) 2024-05-27T07:38:43.987 INFO:journalctl@ceph.osd.6.smithi052.stdout:May 27 07:38:43 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-osd-6[60312]: 2024-05-27T07:38:43.863+0000 7f362964d700 -1 osd.6 134 heartbeat_check: no reply from 172.21.15.105:6804 osd.1 since back 2024-05-27T07:38:03.400081+0000 front 2024-05-27T07:38:03.400066+0000 (oldest deadline 2024-05-27T07:38:28.099925+0000) 2024-05-27T07:38:43.987 INFO:journalctl@ceph.osd.6.smithi052.stdout:May 27 07:38:43 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-osd-6[60312]: 2024-05-27T07:38:43.863+0000 7f362964d700 -1 osd.6 134 heartbeat_check: no reply from 172.21.15.173:6804 osd.2 since back 2024-05-27T07:38:15.702116+0000 front 2024-05-27T07:38:15.702087+0000 (oldest deadline 2024-05-27T07:38:40.402041+0000) 2024-05-27T07:38:43.987 INFO:journalctl@ceph.osd.6.smithi052.stdout:May 27 07:38:43 smithi052 systemd[1]: Stopping Ceph osd.6 for 73df2636-1bf8-11ef-bc9b-c7b262605968... 2024-05-27T07:38:44.109 INFO:journalctl@ceph.osd.8.smithi173.stdout:May 27 07:38:43 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-osd-8[55969]: 2024-05-27T07:38:43.886+0000 7f425b2ea700 -1 osd.8 134 heartbeat_check: no reply from 172.21.15.105:6804 osd.1 since back 2024-05-27T07:38:06.957078+0000 front 2024-05-27T07:38:06.957280+0000 (oldest deadline 2024-05-27T07:38:30.457072+0000) 2024-05-27T07:38:44.109 INFO:journalctl@ceph.osd.8.smithi173.stdout:May 27 07:38:43 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-osd-8[55969]: 2024-05-27T07:38:43.886+0000 7f425b2ea700 -1 osd.8 134 heartbeat_check: no reply from 172.21.15.173:6804 osd.2 since back 2024-05-27T07:38:16.258968+0000 front 2024-05-27T07:38:16.258998+0000 (oldest deadline 2024-05-27T07:38:41.559145+0000) 2024-05-27T07:38:44.197 INFO:journalctl@ceph.osd.7.smithi105.stdout:May 27 07:38:43 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-osd-7[53596]: 2024-05-27T07:38:43.781+0000 7f5eefdf1700 -1 osd.7 134 heartbeat_check: no reply from 172.21.15.105:6804 osd.1 since back 2024-05-27T07:38:06.798385+0000 front 2024-05-27T07:38:06.798353+0000 (oldest deadline 2024-05-27T07:38:32.098493+0000) 2024-05-27T07:38:44.197 INFO:journalctl@ceph.osd.7.smithi105.stdout:May 27 07:38:43 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-osd-7[53596]: 2024-05-27T07:38:43.781+0000 7f5eefdf1700 -1 osd.7 134 heartbeat_check: no reply from 172.21.15.173:6804 osd.2 since back 2024-05-27T07:38:13.199843+0000 front 2024-05-27T07:38:13.199692+0000 (oldest deadline 2024-05-27T07:38:37.899787+0000) 2024-05-27T07:38:44.486 INFO:journalctl@ceph.osd.6.smithi052.stdout:May 27 07:38:44 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-osd-6[60312]: 2024-05-27T07:38:44.177+0000 7f362d097700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.6 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-05-27T07:38:44.487 INFO:journalctl@ceph.osd.6.smithi052.stdout:May 27 07:38:44 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-osd-6[60312]: 2024-05-27T07:38:44.177+0000 7f362d097700 -1 osd.6 134 *** Got signal Terminated *** 2024-05-27T07:38:44.487 INFO:journalctl@ceph.osd.6.smithi052.stdout:May 27 07:38:44 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-osd-6[60312]: 2024-05-27T07:38:44.177+0000 7f362d097700 -1 osd.6 134 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-05-27T07:38:44.697 INFO:journalctl@ceph.osd.10.smithi105.stdout:May 27 07:38:44 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-osd-10[60334]: 2024-05-27T07:38:44.304+0000 7f162fb8f700 -1 osd.10 134 heartbeat_check: no reply from 172.21.15.52:6806 osd.0 since back 2024-05-27T07:37:56.754985+0000 front 2024-05-27T07:37:56.755070+0000 (oldest deadline 2024-05-27T07:38:22.054911+0000) 2024-05-27T07:38:44.697 INFO:journalctl@ceph.osd.10.smithi105.stdout:May 27 07:38:44 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-osd-10[60334]: 2024-05-27T07:38:44.304+0000 7f162fb8f700 -1 osd.10 134 heartbeat_check: no reply from 172.21.15.105:6804 osd.1 since back 2024-05-27T07:38:05.556137+0000 front 2024-05-27T07:38:05.556112+0000 (oldest deadline 2024-05-27T07:38:31.456213+0000) 2024-05-27T07:38:44.697 INFO:journalctl@ceph.osd.10.smithi105.stdout:May 27 07:38:44 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-osd-10[60334]: 2024-05-27T07:38:44.304+0000 7f162fb8f700 -1 osd.10 134 heartbeat_check: no reply from 172.21.15.173:6804 osd.2 since back 2024-05-27T07:38:15.557554+0000 front 2024-05-27T07:38:15.557409+0000 (oldest deadline 2024-05-27T07:38:40.857509+0000) 2024-05-27T07:38:44.859 INFO:journalctl@ceph.osd.11.smithi173.stdout:May 27 07:38:44 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-osd-11[63021]: 2024-05-27T07:38:44.475+0000 7f9d44ac8700 -1 osd.11 134 heartbeat_check: no reply from 172.21.15.52:6806 osd.0 since back 2024-05-27T07:37:57.927796+0000 front 2024-05-27T07:37:57.927759+0000 (oldest deadline 2024-05-27T07:38:22.028025+0000) 2024-05-27T07:38:44.859 INFO:journalctl@ceph.osd.11.smithi173.stdout:May 27 07:38:44 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-osd-11[63021]: 2024-05-27T07:38:44.475+0000 7f9d44ac8700 -1 osd.11 134 heartbeat_check: no reply from 172.21.15.105:6804 osd.1 since back 2024-05-27T07:38:03.129257+0000 front 2024-05-27T07:38:03.129120+0000 (oldest deadline 2024-05-27T07:38:28.929543+0000) 2024-05-27T07:38:44.860 INFO:journalctl@ceph.osd.11.smithi173.stdout:May 27 07:38:44 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-osd-11[63021]: 2024-05-27T07:38:44.475+0000 7f9d44ac8700 -1 osd.11 134 heartbeat_check: no reply from 172.21.15.173:6804 osd.2 since back 2024-05-27T07:38:16.431856+0000 front 2024-05-27T07:38:16.431878+0000 (oldest deadline 2024-05-27T07:38:42.332069+0000) 2024-05-27T07:38:45.197 INFO:journalctl@ceph.osd.7.smithi105.stdout:May 27 07:38:44 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-osd-7[53596]: 2024-05-27T07:38:44.813+0000 7f5eefdf1700 -1 osd.7 134 heartbeat_check: no reply from 172.21.15.105:6804 osd.1 since back 2024-05-27T07:38:06.798385+0000 front 2024-05-27T07:38:06.798353+0000 (oldest deadline 2024-05-27T07:38:32.098493+0000) 2024-05-27T07:38:45.197 INFO:journalctl@ceph.osd.7.smithi105.stdout:May 27 07:38:44 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-osd-7[53596]: 2024-05-27T07:38:44.813+0000 7f5eefdf1700 -1 osd.7 134 heartbeat_check: no reply from 172.21.15.173:6804 osd.2 since back 2024-05-27T07:38:13.199843+0000 front 2024-05-27T07:38:13.199692+0000 (oldest deadline 2024-05-27T07:38:37.899787+0000) 2024-05-27T07:38:45.236 INFO:journalctl@ceph.osd.6.smithi052.stdout:May 27 07:38:44 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-osd-6[60312]: 2024-05-27T07:38:44.869+0000 7f362964d700 -1 osd.6 134 heartbeat_check: no reply from 172.21.15.52:6806 osd.0 since back 2024-05-27T07:37:58.099066+0000 front 2024-05-27T07:37:58.099034+0000 (oldest deadline 2024-05-27T07:38:23.399228+0000) 2024-05-27T07:38:45.236 INFO:journalctl@ceph.osd.6.smithi052.stdout:May 27 07:38:44 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-osd-6[60312]: 2024-05-27T07:38:44.869+0000 7f362964d700 -1 osd.6 134 heartbeat_check: no reply from 172.21.15.105:6804 osd.1 since back 2024-05-27T07:38:03.400081+0000 front 2024-05-27T07:38:03.400066+0000 (oldest deadline 2024-05-27T07:38:28.099925+0000) 2024-05-27T07:38:45.236 INFO:journalctl@ceph.osd.6.smithi052.stdout:May 27 07:38:44 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-osd-6[60312]: 2024-05-27T07:38:44.869+0000 7f362964d700 -1 osd.6 134 heartbeat_check: no reply from 172.21.15.173:6804 osd.2 since back 2024-05-27T07:38:15.702116+0000 front 2024-05-27T07:38:15.702087+0000 (oldest deadline 2024-05-27T07:38:40.402041+0000) 2024-05-27T07:38:45.237 INFO:journalctl@ceph.osd.9.smithi052.stdout:May 27 07:38:44 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-osd-9[67134]: 2024-05-27T07:38:44.748+0000 7f1e3e2ca700 -1 osd.9 134 heartbeat_check: no reply from 172.21.15.52:6806 osd.0 since back 2024-05-27T07:37:58.861291+0000 front 2024-05-27T07:37:58.861258+0000 (oldest deadline 2024-05-27T07:38:22.361571+0000) 2024-05-27T07:38:45.237 INFO:journalctl@ceph.osd.9.smithi052.stdout:May 27 07:38:44 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-osd-9[67134]: 2024-05-27T07:38:44.748+0000 7f1e3e2ca700 -1 osd.9 134 heartbeat_check: no reply from 172.21.15.105:6804 osd.1 since back 2024-05-27T07:38:05.862960+0000 front 2024-05-27T07:38:05.862645+0000 (oldest deadline 2024-05-27T07:38:30.562694+0000) 2024-05-27T07:38:45.237 INFO:journalctl@ceph.osd.9.smithi052.stdout:May 27 07:38:44 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-osd-9[67134]: 2024-05-27T07:38:44.748+0000 7f1e3e2ca700 -1 osd.9 134 heartbeat_check: no reply from 172.21.15.173:6804 osd.2 since back 2024-05-27T07:38:15.566595+0000 front 2024-05-27T07:38:15.566403+0000 (oldest deadline 2024-05-27T07:38:37.866552+0000) 2024-05-27T07:38:45.359 INFO:journalctl@ceph.osd.8.smithi173.stdout:May 27 07:38:44 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-osd-8[55969]: 2024-05-27T07:38:44.914+0000 7f425b2ea700 -1 osd.8 134 heartbeat_check: no reply from 172.21.15.105:6804 osd.1 since back 2024-05-27T07:38:06.957078+0000 front 2024-05-27T07:38:06.957280+0000 (oldest deadline 2024-05-27T07:38:30.457072+0000) 2024-05-27T07:38:45.359 INFO:journalctl@ceph.osd.8.smithi173.stdout:May 27 07:38:44 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-osd-8[55969]: 2024-05-27T07:38:44.914+0000 7f425b2ea700 -1 osd.8 134 heartbeat_check: no reply from 172.21.15.173:6804 osd.2 since back 2024-05-27T07:38:16.258968+0000 front 2024-05-27T07:38:16.258998+0000 (oldest deadline 2024-05-27T07:38:41.559145+0000) 2024-05-27T07:38:45.697 INFO:journalctl@ceph.osd.10.smithi105.stdout:May 27 07:38:45 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-osd-10[60334]: 2024-05-27T07:38:45.320+0000 7f162fb8f700 -1 osd.10 134 heartbeat_check: no reply from 172.21.15.52:6806 osd.0 since back 2024-05-27T07:37:56.754985+0000 front 2024-05-27T07:37:56.755070+0000 (oldest deadline 2024-05-27T07:38:22.054911+0000) 2024-05-27T07:38:45.697 INFO:journalctl@ceph.osd.10.smithi105.stdout:May 27 07:38:45 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-osd-10[60334]: 2024-05-27T07:38:45.320+0000 7f162fb8f700 -1 osd.10 134 heartbeat_check: no reply from 172.21.15.105:6804 osd.1 since back 2024-05-27T07:38:05.556137+0000 front 2024-05-27T07:38:05.556112+0000 (oldest deadline 2024-05-27T07:38:31.456213+0000) 2024-05-27T07:38:45.697 INFO:journalctl@ceph.osd.10.smithi105.stdout:May 27 07:38:45 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-osd-10[60334]: 2024-05-27T07:38:45.320+0000 7f162fb8f700 -1 osd.10 134 heartbeat_check: no reply from 172.21.15.173:6804 osd.2 since back 2024-05-27T07:38:15.557554+0000 front 2024-05-27T07:38:15.557409+0000 (oldest deadline 2024-05-27T07:38:40.857509+0000) 2024-05-27T07:38:45.840 INFO:journalctl@ceph.osd.7.smithi105.stdout:May 27 07:38:45 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-osd-7[53596]: 2024-05-27T07:38:45.841+0000 7f5eefdf1700 -1 osd.7 134 heartbeat_check: no reply from 172.21.15.105:6804 osd.1 since back 2024-05-27T07:38:06.798385+0000 front 2024-05-27T07:38:06.798353+0000 (oldest deadline 2024-05-27T07:38:32.098493+0000) 2024-05-27T07:38:45.840 INFO:journalctl@ceph.osd.7.smithi105.stdout:May 27 07:38:45 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-osd-7[53596]: 2024-05-27T07:38:45.841+0000 7f5eefdf1700 -1 osd.7 134 heartbeat_check: no reply from 172.21.15.173:6804 osd.2 since back 2024-05-27T07:38:13.199843+0000 front 2024-05-27T07:38:13.199692+0000 (oldest deadline 2024-05-27T07:38:37.899787+0000) 2024-05-27T07:38:45.859 INFO:journalctl@ceph.osd.11.smithi173.stdout:May 27 07:38:45 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-osd-11[63021]: 2024-05-27T07:38:45.439+0000 7f9d44ac8700 -1 osd.11 134 heartbeat_check: no reply from 172.21.15.52:6806 osd.0 since back 2024-05-27T07:37:57.927796+0000 front 2024-05-27T07:37:57.927759+0000 (oldest deadline 2024-05-27T07:38:22.028025+0000) 2024-05-27T07:38:45.859 INFO:journalctl@ceph.osd.11.smithi173.stdout:May 27 07:38:45 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-osd-11[63021]: 2024-05-27T07:38:45.439+0000 7f9d44ac8700 -1 osd.11 134 heartbeat_check: no reply from 172.21.15.105:6804 osd.1 since back 2024-05-27T07:38:03.129257+0000 front 2024-05-27T07:38:03.129120+0000 (oldest deadline 2024-05-27T07:38:28.929543+0000) 2024-05-27T07:38:45.859 INFO:journalctl@ceph.osd.11.smithi173.stdout:May 27 07:38:45 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-osd-11[63021]: 2024-05-27T07:38:45.439+0000 7f9d44ac8700 -1 osd.11 134 heartbeat_check: no reply from 172.21.15.173:6804 osd.2 since back 2024-05-27T07:38:16.431856+0000 front 2024-05-27T07:38:16.431878+0000 (oldest deadline 2024-05-27T07:38:42.332069+0000) 2024-05-27T07:38:45.987 INFO:journalctl@ceph.osd.6.smithi052.stdout:May 27 07:38:45 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-osd-6[60312]: 2024-05-27T07:38:45.876+0000 7f362964d700 -1 osd.6 134 heartbeat_check: no reply from 172.21.15.52:6806 osd.0 since back 2024-05-27T07:37:58.099066+0000 front 2024-05-27T07:37:58.099034+0000 (oldest deadline 2024-05-27T07:38:23.399228+0000) 2024-05-27T07:38:45.987 INFO:journalctl@ceph.osd.6.smithi052.stdout:May 27 07:38:45 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-osd-6[60312]: 2024-05-27T07:38:45.876+0000 7f362964d700 -1 osd.6 134 heartbeat_check: no reply from 172.21.15.105:6804 osd.1 since back 2024-05-27T07:38:03.400081+0000 front 2024-05-27T07:38:03.400066+0000 (oldest deadline 2024-05-27T07:38:28.099925+0000) 2024-05-27T07:38:45.987 INFO:journalctl@ceph.osd.6.smithi052.stdout:May 27 07:38:45 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-osd-6[60312]: 2024-05-27T07:38:45.876+0000 7f362964d700 -1 osd.6 134 heartbeat_check: no reply from 172.21.15.173:6804 osd.2 since back 2024-05-27T07:38:15.702116+0000 front 2024-05-27T07:38:15.702087+0000 (oldest deadline 2024-05-27T07:38:40.402041+0000) 2024-05-27T07:38:45.987 INFO:journalctl@ceph.osd.6.smithi052.stdout:May 27 07:38:45 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-osd-6[60312]: 2024-05-27T07:38:45.876+0000 7f362964d700 -1 osd.6 134 heartbeat_check: no reply from 172.21.15.52:6814 osd.3 since back 2024-05-27T07:38:22.503976+0000 front 2024-05-27T07:38:22.504017+0000 (oldest deadline 2024-05-27T07:38:45.404131+0000) 2024-05-27T07:38:45.988 INFO:journalctl@ceph.osd.9.smithi052.stdout:May 27 07:38:45 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-osd-9[67134]: 2024-05-27T07:38:45.716+0000 7f1e3e2ca700 -1 osd.9 134 heartbeat_check: no reply from 172.21.15.52:6806 osd.0 since back 2024-05-27T07:37:58.861291+0000 front 2024-05-27T07:37:58.861258+0000 (oldest deadline 2024-05-27T07:38:22.361571+0000) 2024-05-27T07:38:45.988 INFO:journalctl@ceph.osd.9.smithi052.stdout:May 27 07:38:45 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-osd-9[67134]: 2024-05-27T07:38:45.716+0000 7f1e3e2ca700 -1 osd.9 134 heartbeat_check: no reply from 172.21.15.105:6804 osd.1 since back 2024-05-27T07:38:05.862960+0000 front 2024-05-27T07:38:05.862645+0000 (oldest deadline 2024-05-27T07:38:30.562694+0000) 2024-05-27T07:38:45.988 INFO:journalctl@ceph.osd.9.smithi052.stdout:May 27 07:38:45 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-osd-9[67134]: 2024-05-27T07:38:45.716+0000 7f1e3e2ca700 -1 osd.9 134 heartbeat_check: no reply from 172.21.15.173:6804 osd.2 since back 2024-05-27T07:38:15.566595+0000 front 2024-05-27T07:38:15.566403+0000 (oldest deadline 2024-05-27T07:38:37.866552+0000) 2024-05-27T07:38:46.359 INFO:journalctl@ceph.osd.8.smithi173.stdout:May 27 07:38:45 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-osd-8[55969]: 2024-05-27T07:38:45.871+0000 7f425b2ea700 -1 osd.8 134 heartbeat_check: no reply from 172.21.15.105:6804 osd.1 since back 2024-05-27T07:38:06.957078+0000 front 2024-05-27T07:38:06.957280+0000 (oldest deadline 2024-05-27T07:38:30.457072+0000) 2024-05-27T07:38:46.359 INFO:journalctl@ceph.osd.8.smithi173.stdout:May 27 07:38:45 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-osd-8[55969]: 2024-05-27T07:38:45.871+0000 7f425b2ea700 -1 osd.8 134 heartbeat_check: no reply from 172.21.15.173:6804 osd.2 since back 2024-05-27T07:38:16.258968+0000 front 2024-05-27T07:38:16.258998+0000 (oldest deadline 2024-05-27T07:38:41.559145+0000) 2024-05-27T07:38:46.697 INFO:journalctl@ceph.osd.10.smithi105.stdout:May 27 07:38:46 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-osd-10[60334]: 2024-05-27T07:38:46.366+0000 7f162fb8f700 -1 osd.10 134 heartbeat_check: no reply from 172.21.15.52:6806 osd.0 since back 2024-05-27T07:37:56.754985+0000 front 2024-05-27T07:37:56.755070+0000 (oldest deadline 2024-05-27T07:38:22.054911+0000) 2024-05-27T07:38:46.697 INFO:journalctl@ceph.osd.10.smithi105.stdout:May 27 07:38:46 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-osd-10[60334]: 2024-05-27T07:38:46.366+0000 7f162fb8f700 -1 osd.10 134 heartbeat_check: no reply from 172.21.15.105:6804 osd.1 since back 2024-05-27T07:38:05.556137+0000 front 2024-05-27T07:38:05.556112+0000 (oldest deadline 2024-05-27T07:38:31.456213+0000) 2024-05-27T07:38:46.697 INFO:journalctl@ceph.osd.10.smithi105.stdout:May 27 07:38:46 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-osd-10[60334]: 2024-05-27T07:38:46.366+0000 7f162fb8f700 -1 osd.10 134 heartbeat_check: no reply from 172.21.15.173:6804 osd.2 since back 2024-05-27T07:38:15.557554+0000 front 2024-05-27T07:38:15.557409+0000 (oldest deadline 2024-05-27T07:38:40.857509+0000) 2024-05-27T07:38:46.859 INFO:journalctl@ceph.osd.11.smithi173.stdout:May 27 07:38:46 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-osd-11[63021]: 2024-05-27T07:38:46.450+0000 7f9d44ac8700 -1 osd.11 134 heartbeat_check: no reply from 172.21.15.52:6806 osd.0 since back 2024-05-27T07:37:57.927796+0000 front 2024-05-27T07:37:57.927759+0000 (oldest deadline 2024-05-27T07:38:22.028025+0000) 2024-05-27T07:38:46.859 INFO:journalctl@ceph.osd.11.smithi173.stdout:May 27 07:38:46 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-osd-11[63021]: 2024-05-27T07:38:46.450+0000 7f9d44ac8700 -1 osd.11 134 heartbeat_check: no reply from 172.21.15.105:6804 osd.1 since back 2024-05-27T07:38:03.129257+0000 front 2024-05-27T07:38:03.129120+0000 (oldest deadline 2024-05-27T07:38:28.929543+0000) 2024-05-27T07:38:46.859 INFO:journalctl@ceph.osd.11.smithi173.stdout:May 27 07:38:46 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-osd-11[63021]: 2024-05-27T07:38:46.450+0000 7f9d44ac8700 -1 osd.11 134 heartbeat_check: no reply from 172.21.15.173:6804 osd.2 since back 2024-05-27T07:38:16.431856+0000 front 2024-05-27T07:38:16.431878+0000 (oldest deadline 2024-05-27T07:38:42.332069+0000) 2024-05-27T07:38:46.986 INFO:journalctl@ceph.osd.6.smithi052.stdout:May 27 07:38:46 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-osd-6[60312]: 2024-05-27T07:38:46.827+0000 7f362964d700 -1 osd.6 134 heartbeat_check: no reply from 172.21.15.52:6806 osd.0 since back 2024-05-27T07:37:58.099066+0000 front 2024-05-27T07:37:58.099034+0000 (oldest deadline 2024-05-27T07:38:23.399228+0000) 2024-05-27T07:38:46.987 INFO:journalctl@ceph.osd.6.smithi052.stdout:May 27 07:38:46 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-osd-6[60312]: 2024-05-27T07:38:46.827+0000 7f362964d700 -1 osd.6 134 heartbeat_check: no reply from 172.21.15.105:6804 osd.1 since back 2024-05-27T07:38:03.400081+0000 front 2024-05-27T07:38:03.400066+0000 (oldest deadline 2024-05-27T07:38:28.099925+0000) 2024-05-27T07:38:46.987 INFO:journalctl@ceph.osd.6.smithi052.stdout:May 27 07:38:46 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-osd-6[60312]: 2024-05-27T07:38:46.827+0000 7f362964d700 -1 osd.6 134 heartbeat_check: no reply from 172.21.15.173:6804 osd.2 since back 2024-05-27T07:38:15.702116+0000 front 2024-05-27T07:38:15.702087+0000 (oldest deadline 2024-05-27T07:38:40.402041+0000) 2024-05-27T07:38:46.987 INFO:journalctl@ceph.osd.6.smithi052.stdout:May 27 07:38:46 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-osd-6[60312]: 2024-05-27T07:38:46.827+0000 7f362964d700 -1 osd.6 134 heartbeat_check: no reply from 172.21.15.52:6814 osd.3 since back 2024-05-27T07:38:22.503976+0000 front 2024-05-27T07:38:22.504017+0000 (oldest deadline 2024-05-27T07:38:45.404131+0000) 2024-05-27T07:38:46.987 INFO:journalctl@ceph.osd.9.smithi052.stdout:May 27 07:38:46 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-osd-9[67134]: 2024-05-27T07:38:46.691+0000 7f1e3e2ca700 -1 osd.9 134 heartbeat_check: no reply from 172.21.15.52:6806 osd.0 since back 2024-05-27T07:37:58.861291+0000 front 2024-05-27T07:37:58.861258+0000 (oldest deadline 2024-05-27T07:38:22.361571+0000) 2024-05-27T07:38:46.987 INFO:journalctl@ceph.osd.9.smithi052.stdout:May 27 07:38:46 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-osd-9[67134]: 2024-05-27T07:38:46.691+0000 7f1e3e2ca700 -1 osd.9 134 heartbeat_check: no reply from 172.21.15.105:6804 osd.1 since back 2024-05-27T07:38:05.862960+0000 front 2024-05-27T07:38:05.862645+0000 (oldest deadline 2024-05-27T07:38:30.562694+0000) 2024-05-27T07:38:46.987 INFO:journalctl@ceph.osd.9.smithi052.stdout:May 27 07:38:46 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-osd-9[67134]: 2024-05-27T07:38:46.691+0000 7f1e3e2ca700 -1 osd.9 134 heartbeat_check: no reply from 172.21.15.173:6804 osd.2 since back 2024-05-27T07:38:15.566595+0000 front 2024-05-27T07:38:15.566403+0000 (oldest deadline 2024-05-27T07:38:37.866552+0000) 2024-05-27T07:38:47.197 INFO:journalctl@ceph.osd.7.smithi105.stdout:May 27 07:38:46 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-osd-7[53596]: 2024-05-27T07:38:46.835+0000 7f5eefdf1700 -1 osd.7 134 heartbeat_check: no reply from 172.21.15.105:6804 osd.1 since back 2024-05-27T07:38:06.798385+0000 front 2024-05-27T07:38:06.798353+0000 (oldest deadline 2024-05-27T07:38:32.098493+0000) 2024-05-27T07:38:47.197 INFO:journalctl@ceph.osd.7.smithi105.stdout:May 27 07:38:46 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-osd-7[53596]: 2024-05-27T07:38:46.835+0000 7f5eefdf1700 -1 osd.7 134 heartbeat_check: no reply from 172.21.15.173:6804 osd.2 since back 2024-05-27T07:38:13.199843+0000 front 2024-05-27T07:38:13.199692+0000 (oldest deadline 2024-05-27T07:38:37.899787+0000) 2024-05-27T07:38:47.359 INFO:journalctl@ceph.osd.8.smithi173.stdout:May 27 07:38:46 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-osd-8[55969]: 2024-05-27T07:38:46.875+0000 7f425b2ea700 -1 osd.8 134 heartbeat_check: no reply from 172.21.15.105:6804 osd.1 since back 2024-05-27T07:38:06.957078+0000 front 2024-05-27T07:38:06.957280+0000 (oldest deadline 2024-05-27T07:38:30.457072+0000) 2024-05-27T07:38:47.359 INFO:journalctl@ceph.osd.8.smithi173.stdout:May 27 07:38:46 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-osd-8[55969]: 2024-05-27T07:38:46.875+0000 7f425b2ea700 -1 osd.8 134 heartbeat_check: no reply from 172.21.15.173:6804 osd.2 since back 2024-05-27T07:38:16.258968+0000 front 2024-05-27T07:38:16.258998+0000 (oldest deadline 2024-05-27T07:38:41.559145+0000) 2024-05-27T07:38:47.697 INFO:journalctl@ceph.osd.10.smithi105.stdout:May 27 07:38:47 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-osd-10[60334]: 2024-05-27T07:38:47.374+0000 7f162fb8f700 -1 osd.10 134 heartbeat_check: no reply from 172.21.15.52:6806 osd.0 since back 2024-05-27T07:37:56.754985+0000 front 2024-05-27T07:37:56.755070+0000 (oldest deadline 2024-05-27T07:38:22.054911+0000) 2024-05-27T07:38:47.697 INFO:journalctl@ceph.osd.10.smithi105.stdout:May 27 07:38:47 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-osd-10[60334]: 2024-05-27T07:38:47.374+0000 7f162fb8f700 -1 osd.10 134 heartbeat_check: no reply from 172.21.15.105:6804 osd.1 since back 2024-05-27T07:38:05.556137+0000 front 2024-05-27T07:38:05.556112+0000 (oldest deadline 2024-05-27T07:38:31.456213+0000) 2024-05-27T07:38:47.697 INFO:journalctl@ceph.osd.10.smithi105.stdout:May 27 07:38:47 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-osd-10[60334]: 2024-05-27T07:38:47.374+0000 7f162fb8f700 -1 osd.10 134 heartbeat_check: no reply from 172.21.15.173:6804 osd.2 since back 2024-05-27T07:38:15.557554+0000 front 2024-05-27T07:38:15.557409+0000 (oldest deadline 2024-05-27T07:38:40.857509+0000) 2024-05-27T07:38:47.859 INFO:journalctl@ceph.osd.11.smithi173.stdout:May 27 07:38:47 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-osd-11[63021]: 2024-05-27T07:38:47.427+0000 7f9d44ac8700 -1 osd.11 134 heartbeat_check: no reply from 172.21.15.52:6806 osd.0 since back 2024-05-27T07:37:57.927796+0000 front 2024-05-27T07:37:57.927759+0000 (oldest deadline 2024-05-27T07:38:22.028025+0000) 2024-05-27T07:38:47.859 INFO:journalctl@ceph.osd.11.smithi173.stdout:May 27 07:38:47 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-osd-11[63021]: 2024-05-27T07:38:47.427+0000 7f9d44ac8700 -1 osd.11 134 heartbeat_check: no reply from 172.21.15.105:6804 osd.1 since back 2024-05-27T07:38:03.129257+0000 front 2024-05-27T07:38:03.129120+0000 (oldest deadline 2024-05-27T07:38:28.929543+0000) 2024-05-27T07:38:47.859 INFO:journalctl@ceph.osd.11.smithi173.stdout:May 27 07:38:47 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-osd-11[63021]: 2024-05-27T07:38:47.427+0000 7f9d44ac8700 -1 osd.11 134 heartbeat_check: no reply from 172.21.15.173:6804 osd.2 since back 2024-05-27T07:38:16.431856+0000 front 2024-05-27T07:38:16.431878+0000 (oldest deadline 2024-05-27T07:38:42.332069+0000) 2024-05-27T07:38:47.986 INFO:journalctl@ceph.osd.9.smithi052.stdout:May 27 07:38:47 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-osd-9[67134]: 2024-05-27T07:38:47.700+0000 7f1e3e2ca700 -1 osd.9 134 heartbeat_check: no reply from 172.21.15.52:6806 osd.0 since back 2024-05-27T07:37:58.861291+0000 front 2024-05-27T07:37:58.861258+0000 (oldest deadline 2024-05-27T07:38:22.361571+0000) 2024-05-27T07:38:47.987 INFO:journalctl@ceph.osd.9.smithi052.stdout:May 27 07:38:47 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-osd-9[67134]: 2024-05-27T07:38:47.700+0000 7f1e3e2ca700 -1 osd.9 134 heartbeat_check: no reply from 172.21.15.105:6804 osd.1 since back 2024-05-27T07:38:05.862960+0000 front 2024-05-27T07:38:05.862645+0000 (oldest deadline 2024-05-27T07:38:30.562694+0000) 2024-05-27T07:38:47.987 INFO:journalctl@ceph.osd.9.smithi052.stdout:May 27 07:38:47 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-osd-9[67134]: 2024-05-27T07:38:47.700+0000 7f1e3e2ca700 -1 osd.9 134 heartbeat_check: no reply from 172.21.15.173:6804 osd.2 since back 2024-05-27T07:38:15.566595+0000 front 2024-05-27T07:38:15.566403+0000 (oldest deadline 2024-05-27T07:38:37.866552+0000) 2024-05-27T07:38:47.987 INFO:journalctl@ceph.osd.6.smithi052.stdout:May 27 07:38:47 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-osd-6[60312]: 2024-05-27T07:38:47.876+0000 7f362964d700 -1 osd.6 134 heartbeat_check: no reply from 172.21.15.52:6806 osd.0 since back 2024-05-27T07:37:58.099066+0000 front 2024-05-27T07:37:58.099034+0000 (oldest deadline 2024-05-27T07:38:23.399228+0000) 2024-05-27T07:38:47.987 INFO:journalctl@ceph.osd.6.smithi052.stdout:May 27 07:38:47 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-osd-6[60312]: 2024-05-27T07:38:47.876+0000 7f362964d700 -1 osd.6 134 heartbeat_check: no reply from 172.21.15.105:6804 osd.1 since back 2024-05-27T07:38:03.400081+0000 front 2024-05-27T07:38:03.400066+0000 (oldest deadline 2024-05-27T07:38:28.099925+0000) 2024-05-27T07:38:47.987 INFO:journalctl@ceph.osd.6.smithi052.stdout:May 27 07:38:47 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-osd-6[60312]: 2024-05-27T07:38:47.876+0000 7f362964d700 -1 osd.6 134 heartbeat_check: no reply from 172.21.15.173:6804 osd.2 since back 2024-05-27T07:38:15.702116+0000 front 2024-05-27T07:38:15.702087+0000 (oldest deadline 2024-05-27T07:38:40.402041+0000) 2024-05-27T07:38:47.987 INFO:journalctl@ceph.osd.6.smithi052.stdout:May 27 07:38:47 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-osd-6[60312]: 2024-05-27T07:38:47.876+0000 7f362964d700 -1 osd.6 134 heartbeat_check: no reply from 172.21.15.52:6814 osd.3 since back 2024-05-27T07:38:22.503976+0000 front 2024-05-27T07:38:22.504017+0000 (oldest deadline 2024-05-27T07:38:45.404131+0000) 2024-05-27T07:38:48.197 INFO:journalctl@ceph.osd.7.smithi105.stdout:May 27 07:38:47 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-osd-7[53596]: 2024-05-27T07:38:47.846+0000 7f5eefdf1700 -1 osd.7 134 heartbeat_check: no reply from 172.21.15.105:6804 osd.1 since back 2024-05-27T07:38:06.798385+0000 front 2024-05-27T07:38:06.798353+0000 (oldest deadline 2024-05-27T07:38:32.098493+0000) 2024-05-27T07:38:48.197 INFO:journalctl@ceph.osd.7.smithi105.stdout:May 27 07:38:47 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-osd-7[53596]: 2024-05-27T07:38:47.846+0000 7f5eefdf1700 -1 osd.7 134 heartbeat_check: no reply from 172.21.15.173:6804 osd.2 since back 2024-05-27T07:38:13.199843+0000 front 2024-05-27T07:38:13.199692+0000 (oldest deadline 2024-05-27T07:38:37.899787+0000) 2024-05-27T07:38:48.197 INFO:journalctl@ceph.osd.7.smithi105.stdout:May 27 07:38:47 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-osd-7[53596]: 2024-05-27T07:38:47.846+0000 7f5eefdf1700 -1 osd.7 134 heartbeat_check: no reply from 172.21.15.52:6814 osd.3 since back 2024-05-27T07:38:23.002094+0000 front 2024-05-27T07:38:23.002188+0000 (oldest deadline 2024-05-27T07:38:47.702083+0000) 2024-05-27T07:38:48.359 INFO:journalctl@ceph.osd.8.smithi173.stdout:May 27 07:38:47 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-osd-8[55969]: 2024-05-27T07:38:47.893+0000 7f425b2ea700 -1 osd.8 134 heartbeat_check: no reply from 172.21.15.105:6804 osd.1 since back 2024-05-27T07:38:06.957078+0000 front 2024-05-27T07:38:06.957280+0000 (oldest deadline 2024-05-27T07:38:30.457072+0000) 2024-05-27T07:38:48.359 INFO:journalctl@ceph.osd.8.smithi173.stdout:May 27 07:38:47 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-osd-8[55969]: 2024-05-27T07:38:47.893+0000 7f425b2ea700 -1 osd.8 134 heartbeat_check: no reply from 172.21.15.173:6804 osd.2 since back 2024-05-27T07:38:16.258968+0000 front 2024-05-27T07:38:16.258998+0000 (oldest deadline 2024-05-27T07:38:41.559145+0000) 2024-05-27T07:38:48.631 INFO:journalctl@ceph.osd.11.smithi173.stdout:May 27 07:38:48 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-osd-11[63021]: 2024-05-27T07:38:48.389+0000 7f9d44ac8700 -1 osd.11 134 heartbeat_check: no reply from 172.21.15.52:6806 osd.0 since back 2024-05-27T07:37:57.927796+0000 front 2024-05-27T07:37:57.927759+0000 (oldest deadline 2024-05-27T07:38:22.028025+0000) 2024-05-27T07:38:48.632 INFO:journalctl@ceph.osd.11.smithi173.stdout:May 27 07:38:48 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-osd-11[63021]: 2024-05-27T07:38:48.389+0000 7f9d44ac8700 -1 osd.11 134 heartbeat_check: no reply from 172.21.15.105:6804 osd.1 since back 2024-05-27T07:38:03.129257+0000 front 2024-05-27T07:38:03.129120+0000 (oldest deadline 2024-05-27T07:38:28.929543+0000) 2024-05-27T07:38:48.632 INFO:journalctl@ceph.osd.11.smithi173.stdout:May 27 07:38:48 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-osd-11[63021]: 2024-05-27T07:38:48.389+0000 7f9d44ac8700 -1 osd.11 134 heartbeat_check: no reply from 172.21.15.173:6804 osd.2 since back 2024-05-27T07:38:16.431856+0000 front 2024-05-27T07:38:16.431878+0000 (oldest deadline 2024-05-27T07:38:42.332069+0000) 2024-05-27T07:38:48.697 INFO:journalctl@ceph.osd.10.smithi105.stdout:May 27 07:38:48 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-osd-10[60334]: 2024-05-27T07:38:48.386+0000 7f162fb8f700 -1 osd.10 134 heartbeat_check: no reply from 172.21.15.52:6806 osd.0 since back 2024-05-27T07:37:56.754985+0000 front 2024-05-27T07:37:56.755070+0000 (oldest deadline 2024-05-27T07:38:22.054911+0000) 2024-05-27T07:38:48.697 INFO:journalctl@ceph.osd.10.smithi105.stdout:May 27 07:38:48 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-osd-10[60334]: 2024-05-27T07:38:48.386+0000 7f162fb8f700 -1 osd.10 134 heartbeat_check: no reply from 172.21.15.105:6804 osd.1 since back 2024-05-27T07:38:05.556137+0000 front 2024-05-27T07:38:05.556112+0000 (oldest deadline 2024-05-27T07:38:31.456213+0000) 2024-05-27T07:38:48.697 INFO:journalctl@ceph.osd.10.smithi105.stdout:May 27 07:38:48 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-osd-10[60334]: 2024-05-27T07:38:48.386+0000 7f162fb8f700 -1 osd.10 134 heartbeat_check: no reply from 172.21.15.173:6804 osd.2 since back 2024-05-27T07:38:15.557554+0000 front 2024-05-27T07:38:15.557409+0000 (oldest deadline 2024-05-27T07:38:40.857509+0000) 2024-05-27T07:38:48.697 INFO:journalctl@ceph.osd.10.smithi105.stdout:May 27 07:38:48 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-osd-10[60334]: 2024-05-27T07:38:48.386+0000 7f162fb8f700 -1 osd.10 134 heartbeat_check: no reply from 172.21.15.52:6814 osd.3 since back 2024-05-27T07:38:21.958641+0000 front 2024-05-27T07:38:21.958614+0000 (oldest deadline 2024-05-27T07:38:47.858600+0000) 2024-05-27T07:38:49.197 INFO:journalctl@ceph.osd.7.smithi105.stdout:May 27 07:38:48 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-osd-7[53596]: 2024-05-27T07:38:48.804+0000 7f5eefdf1700 -1 osd.7 134 heartbeat_check: no reply from 172.21.15.105:6804 osd.1 since back 2024-05-27T07:38:06.798385+0000 front 2024-05-27T07:38:06.798353+0000 (oldest deadline 2024-05-27T07:38:32.098493+0000) 2024-05-27T07:38:49.197 INFO:journalctl@ceph.osd.7.smithi105.stdout:May 27 07:38:48 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-osd-7[53596]: 2024-05-27T07:38:48.804+0000 7f5eefdf1700 -1 osd.7 134 heartbeat_check: no reply from 172.21.15.173:6804 osd.2 since back 2024-05-27T07:38:13.199843+0000 front 2024-05-27T07:38:13.199692+0000 (oldest deadline 2024-05-27T07:38:37.899787+0000) 2024-05-27T07:38:49.197 INFO:journalctl@ceph.osd.7.smithi105.stdout:May 27 07:38:48 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-osd-7[53596]: 2024-05-27T07:38:48.804+0000 7f5eefdf1700 -1 osd.7 134 heartbeat_check: no reply from 172.21.15.52:6814 osd.3 since back 2024-05-27T07:38:23.002094+0000 front 2024-05-27T07:38:23.002188+0000 (oldest deadline 2024-05-27T07:38:47.702083+0000) 2024-05-27T07:38:49.236 INFO:journalctl@ceph.osd.6.smithi052.stdout:May 27 07:38:48 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-osd-6[60312]: 2024-05-27T07:38:48.919+0000 7f362964d700 -1 osd.6 134 heartbeat_check: no reply from 172.21.15.52:6806 osd.0 since back 2024-05-27T07:37:58.099066+0000 front 2024-05-27T07:37:58.099034+0000 (oldest deadline 2024-05-27T07:38:23.399228+0000) 2024-05-27T07:38:49.236 INFO:journalctl@ceph.osd.6.smithi052.stdout:May 27 07:38:48 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-osd-6[60312]: 2024-05-27T07:38:48.919+0000 7f362964d700 -1 osd.6 134 heartbeat_check: no reply from 172.21.15.105:6804 osd.1 since back 2024-05-27T07:38:03.400081+0000 front 2024-05-27T07:38:03.400066+0000 (oldest deadline 2024-05-27T07:38:28.099925+0000) 2024-05-27T07:38:49.236 INFO:journalctl@ceph.osd.6.smithi052.stdout:May 27 07:38:48 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-osd-6[60312]: 2024-05-27T07:38:48.919+0000 7f362964d700 -1 osd.6 134 heartbeat_check: no reply from 172.21.15.173:6804 osd.2 since back 2024-05-27T07:38:15.702116+0000 front 2024-05-27T07:38:15.702087+0000 (oldest deadline 2024-05-27T07:38:40.402041+0000) 2024-05-27T07:38:49.237 INFO:journalctl@ceph.osd.6.smithi052.stdout:May 27 07:38:48 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-osd-6[60312]: 2024-05-27T07:38:48.919+0000 7f362964d700 -1 osd.6 134 heartbeat_check: no reply from 172.21.15.52:6814 osd.3 since back 2024-05-27T07:38:22.503976+0000 front 2024-05-27T07:38:22.504017+0000 (oldest deadline 2024-05-27T07:38:45.404131+0000) 2024-05-27T07:38:49.237 INFO:journalctl@ceph.osd.9.smithi052.stdout:May 27 07:38:48 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-osd-9[67134]: 2024-05-27T07:38:48.747+0000 7f1e3e2ca700 -1 osd.9 134 heartbeat_check: no reply from 172.21.15.52:6806 osd.0 since back 2024-05-27T07:37:58.861291+0000 front 2024-05-27T07:37:58.861258+0000 (oldest deadline 2024-05-27T07:38:22.361571+0000) 2024-05-27T07:38:49.237 INFO:journalctl@ceph.osd.9.smithi052.stdout:May 27 07:38:48 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-osd-9[67134]: 2024-05-27T07:38:48.747+0000 7f1e3e2ca700 -1 osd.9 134 heartbeat_check: no reply from 172.21.15.105:6804 osd.1 since back 2024-05-27T07:38:05.862960+0000 front 2024-05-27T07:38:05.862645+0000 (oldest deadline 2024-05-27T07:38:30.562694+0000) 2024-05-27T07:38:49.237 INFO:journalctl@ceph.osd.9.smithi052.stdout:May 27 07:38:48 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-osd-9[67134]: 2024-05-27T07:38:48.747+0000 7f1e3e2ca700 -1 osd.9 134 heartbeat_check: no reply from 172.21.15.173:6804 osd.2 since back 2024-05-27T07:38:15.566595+0000 front 2024-05-27T07:38:15.566403+0000 (oldest deadline 2024-05-27T07:38:37.866552+0000) 2024-05-27T07:38:49.237 INFO:journalctl@ceph.osd.9.smithi052.stdout:May 27 07:38:48 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-osd-9[67134]: 2024-05-27T07:38:48.747+0000 7f1e3e2ca700 -1 osd.9 134 heartbeat_check: no reply from 172.21.15.52:6814 osd.3 since back 2024-05-27T07:38:22.468720+0000 front 2024-05-27T07:38:22.468751+0000 (oldest deadline 2024-05-27T07:38:48.269806+0000) 2024-05-27T07:38:49.359 INFO:journalctl@ceph.osd.8.smithi173.stdout:May 27 07:38:48 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-osd-8[55969]: 2024-05-27T07:38:48.885+0000 7f425b2ea700 -1 osd.8 134 heartbeat_check: no reply from 172.21.15.105:6804 osd.1 since back 2024-05-27T07:38:06.957078+0000 front 2024-05-27T07:38:06.957280+0000 (oldest deadline 2024-05-27T07:38:30.457072+0000) 2024-05-27T07:38:49.360 INFO:journalctl@ceph.osd.8.smithi173.stdout:May 27 07:38:48 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-osd-8[55969]: 2024-05-27T07:38:48.885+0000 7f425b2ea700 -1 osd.8 134 heartbeat_check: no reply from 172.21.15.173:6804 osd.2 since back 2024-05-27T07:38:16.258968+0000 front 2024-05-27T07:38:16.258998+0000 (oldest deadline 2024-05-27T07:38:41.559145+0000) 2024-05-27T07:38:49.697 INFO:journalctl@ceph.osd.10.smithi105.stdout:May 27 07:38:49 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-osd-10[60334]: 2024-05-27T07:38:49.420+0000 7f162fb8f700 -1 osd.10 134 heartbeat_check: no reply from 172.21.15.52:6806 osd.0 since back 2024-05-27T07:37:56.754985+0000 front 2024-05-27T07:37:56.755070+0000 (oldest deadline 2024-05-27T07:38:22.054911+0000) 2024-05-27T07:38:49.697 INFO:journalctl@ceph.osd.10.smithi105.stdout:May 27 07:38:49 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-osd-10[60334]: 2024-05-27T07:38:49.420+0000 7f162fb8f700 -1 osd.10 134 heartbeat_check: no reply from 172.21.15.105:6804 osd.1 since back 2024-05-27T07:38:05.556137+0000 front 2024-05-27T07:38:05.556112+0000 (oldest deadline 2024-05-27T07:38:31.456213+0000) 2024-05-27T07:38:49.697 INFO:journalctl@ceph.osd.10.smithi105.stdout:May 27 07:38:49 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-osd-10[60334]: 2024-05-27T07:38:49.420+0000 7f162fb8f700 -1 osd.10 134 heartbeat_check: no reply from 172.21.15.173:6804 osd.2 since back 2024-05-27T07:38:15.557554+0000 front 2024-05-27T07:38:15.557409+0000 (oldest deadline 2024-05-27T07:38:40.857509+0000) 2024-05-27T07:38:49.697 INFO:journalctl@ceph.osd.10.smithi105.stdout:May 27 07:38:49 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-osd-10[60334]: 2024-05-27T07:38:49.420+0000 7f162fb8f700 -1 osd.10 134 heartbeat_check: no reply from 172.21.15.52:6814 osd.3 since back 2024-05-27T07:38:21.958641+0000 front 2024-05-27T07:38:21.958614+0000 (oldest deadline 2024-05-27T07:38:47.858600+0000) 2024-05-27T07:38:49.859 INFO:journalctl@ceph.osd.11.smithi173.stdout:May 27 07:38:49 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-osd-11[63021]: 2024-05-27T07:38:49.392+0000 7f9d44ac8700 -1 osd.11 134 heartbeat_check: no reply from 172.21.15.52:6806 osd.0 since back 2024-05-27T07:37:57.927796+0000 front 2024-05-27T07:37:57.927759+0000 (oldest deadline 2024-05-27T07:38:22.028025+0000) 2024-05-27T07:38:49.859 INFO:journalctl@ceph.osd.11.smithi173.stdout:May 27 07:38:49 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-osd-11[63021]: 2024-05-27T07:38:49.392+0000 7f9d44ac8700 -1 osd.11 134 heartbeat_check: no reply from 172.21.15.105:6804 osd.1 since back 2024-05-27T07:38:03.129257+0000 front 2024-05-27T07:38:03.129120+0000 (oldest deadline 2024-05-27T07:38:28.929543+0000) 2024-05-27T07:38:49.859 INFO:journalctl@ceph.osd.11.smithi173.stdout:May 27 07:38:49 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-osd-11[63021]: 2024-05-27T07:38:49.392+0000 7f9d44ac8700 -1 osd.11 134 heartbeat_check: no reply from 172.21.15.173:6804 osd.2 since back 2024-05-27T07:38:16.431856+0000 front 2024-05-27T07:38:16.431878+0000 (oldest deadline 2024-05-27T07:38:42.332069+0000) 2024-05-27T07:38:49.986 INFO:journalctl@ceph.osd.6.smithi052.stdout:May 27 07:38:49 smithi052 podman[119465]: 2024-05-27 07:38:49.821150071 +0000 UTC m=+5.751322927 container died 23fe920dff3bdd4c0ed1c509a0c122f10650b3806b0070d364496f70a54c8add (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-osd-6, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, org.label-schema.build-date=20240521, org.label-schema.vendor=CentOS, ceph=True, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.license=GPLv2, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image) 2024-05-27T07:38:49.987 INFO:journalctl@ceph.osd.9.smithi052.stdout:May 27 07:38:49 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-osd-9[67134]: 2024-05-27T07:38:49.721+0000 7f1e3e2ca700 -1 osd.9 134 heartbeat_check: no reply from 172.21.15.52:6806 osd.0 since back 2024-05-27T07:37:58.861291+0000 front 2024-05-27T07:37:58.861258+0000 (oldest deadline 2024-05-27T07:38:22.361571+0000) 2024-05-27T07:38:49.987 INFO:journalctl@ceph.osd.9.smithi052.stdout:May 27 07:38:49 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-osd-9[67134]: 2024-05-27T07:38:49.721+0000 7f1e3e2ca700 -1 osd.9 134 heartbeat_check: no reply from 172.21.15.105:6804 osd.1 since back 2024-05-27T07:38:05.862960+0000 front 2024-05-27T07:38:05.862645+0000 (oldest deadline 2024-05-27T07:38:30.562694+0000) 2024-05-27T07:38:49.987 INFO:journalctl@ceph.osd.9.smithi052.stdout:May 27 07:38:49 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-osd-9[67134]: 2024-05-27T07:38:49.721+0000 7f1e3e2ca700 -1 osd.9 134 heartbeat_check: no reply from 172.21.15.173:6804 osd.2 since back 2024-05-27T07:38:15.566595+0000 front 2024-05-27T07:38:15.566403+0000 (oldest deadline 2024-05-27T07:38:37.866552+0000) 2024-05-27T07:38:49.987 INFO:journalctl@ceph.osd.9.smithi052.stdout:May 27 07:38:49 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-osd-9[67134]: 2024-05-27T07:38:49.721+0000 7f1e3e2ca700 -1 osd.9 134 heartbeat_check: no reply from 172.21.15.52:6814 osd.3 since back 2024-05-27T07:38:22.468720+0000 front 2024-05-27T07:38:22.468751+0000 (oldest deadline 2024-05-27T07:38:48.269806+0000) 2024-05-27T07:38:50.197 INFO:journalctl@ceph.osd.7.smithi105.stdout:May 27 07:38:49 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-osd-7[53596]: 2024-05-27T07:38:49.822+0000 7f5eefdf1700 -1 osd.7 134 heartbeat_check: no reply from 172.21.15.105:6804 osd.1 since back 2024-05-27T07:38:06.798385+0000 front 2024-05-27T07:38:06.798353+0000 (oldest deadline 2024-05-27T07:38:32.098493+0000) 2024-05-27T07:38:50.197 INFO:journalctl@ceph.osd.7.smithi105.stdout:May 27 07:38:49 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-osd-7[53596]: 2024-05-27T07:38:49.822+0000 7f5eefdf1700 -1 osd.7 134 heartbeat_check: no reply from 172.21.15.173:6804 osd.2 since back 2024-05-27T07:38:13.199843+0000 front 2024-05-27T07:38:13.199692+0000 (oldest deadline 2024-05-27T07:38:37.899787+0000) 2024-05-27T07:38:50.197 INFO:journalctl@ceph.osd.7.smithi105.stdout:May 27 07:38:49 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-osd-7[53596]: 2024-05-27T07:38:49.822+0000 7f5eefdf1700 -1 osd.7 134 heartbeat_check: no reply from 172.21.15.52:6814 osd.3 since back 2024-05-27T07:38:23.002094+0000 front 2024-05-27T07:38:23.002188+0000 (oldest deadline 2024-05-27T07:38:47.702083+0000) 2024-05-27T07:38:50.359 INFO:journalctl@ceph.osd.8.smithi173.stdout:May 27 07:38:49 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-osd-8[55969]: 2024-05-27T07:38:49.930+0000 7f425b2ea700 -1 osd.8 134 heartbeat_check: no reply from 172.21.15.105:6804 osd.1 since back 2024-05-27T07:38:06.957078+0000 front 2024-05-27T07:38:06.957280+0000 (oldest deadline 2024-05-27T07:38:30.457072+0000) 2024-05-27T07:38:50.359 INFO:journalctl@ceph.osd.8.smithi173.stdout:May 27 07:38:49 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-osd-8[55969]: 2024-05-27T07:38:49.930+0000 7f425b2ea700 -1 osd.8 134 heartbeat_check: no reply from 172.21.15.173:6804 osd.2 since back 2024-05-27T07:38:16.258968+0000 front 2024-05-27T07:38:16.258998+0000 (oldest deadline 2024-05-27T07:38:41.559145+0000) 2024-05-27T07:38:50.359 INFO:journalctl@ceph.osd.8.smithi173.stdout:May 27 07:38:49 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-osd-8[55969]: 2024-05-27T07:38:49.930+0000 7f425b2ea700 -1 osd.8 134 heartbeat_check: no reply from 172.21.15.52:6814 osd.3 since back 2024-05-27T07:38:23.860441+0000 front 2024-05-27T07:38:23.860421+0000 (oldest deadline 2024-05-27T07:38:49.760323+0000) 2024-05-27T07:38:50.477 INFO:journalctl@ceph.osd.6.smithi052.stdout:May 27 07:38:50 smithi052 podman[119465]: 2024-05-27 07:38:50.237171981 +0000 UTC m=+6.167344881 container cleanup 23fe920dff3bdd4c0ed1c509a0c122f10650b3806b0070d364496f70a54c8add (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-osd-6, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, org.label-schema.build-date=20240521, ceph=True, maintainer=Guillaume Abrioux , GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS) 2024-05-27T07:38:50.477 INFO:journalctl@ceph.osd.6.smithi052.stdout:May 27 07:38:50 smithi052 bash[119465]: ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-osd-6 2024-05-27T07:38:50.697 INFO:journalctl@ceph.osd.10.smithi105.stdout:May 27 07:38:50 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-osd-10[60334]: 2024-05-27T07:38:50.399+0000 7f162fb8f700 -1 osd.10 134 heartbeat_check: no reply from 172.21.15.52:6806 osd.0 since back 2024-05-27T07:37:56.754985+0000 front 2024-05-27T07:37:56.755070+0000 (oldest deadline 2024-05-27T07:38:22.054911+0000) 2024-05-27T07:38:50.697 INFO:journalctl@ceph.osd.10.smithi105.stdout:May 27 07:38:50 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-osd-10[60334]: 2024-05-27T07:38:50.399+0000 7f162fb8f700 -1 osd.10 134 heartbeat_check: no reply from 172.21.15.105:6804 osd.1 since back 2024-05-27T07:38:05.556137+0000 front 2024-05-27T07:38:05.556112+0000 (oldest deadline 2024-05-27T07:38:31.456213+0000) 2024-05-27T07:38:50.697 INFO:journalctl@ceph.osd.10.smithi105.stdout:May 27 07:38:50 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-osd-10[60334]: 2024-05-27T07:38:50.399+0000 7f162fb8f700 -1 osd.10 134 heartbeat_check: no reply from 172.21.15.173:6804 osd.2 since back 2024-05-27T07:38:15.557554+0000 front 2024-05-27T07:38:15.557409+0000 (oldest deadline 2024-05-27T07:38:40.857509+0000) 2024-05-27T07:38:50.697 INFO:journalctl@ceph.osd.10.smithi105.stdout:May 27 07:38:50 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-osd-10[60334]: 2024-05-27T07:38:50.399+0000 7f162fb8f700 -1 osd.10 134 heartbeat_check: no reply from 172.21.15.52:6814 osd.3 since back 2024-05-27T07:38:21.958641+0000 front 2024-05-27T07:38:21.958614+0000 (oldest deadline 2024-05-27T07:38:47.858600+0000) 2024-05-27T07:38:50.736 INFO:journalctl@ceph.osd.6.smithi052.stdout:May 27 07:38:50 smithi052 podman[119480]: 2024-05-27 07:38:50.480285539 +0000 UTC m=+0.661305722 container remove 23fe920dff3bdd4c0ed1c509a0c122f10650b3806b0070d364496f70a54c8add (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-osd-6, org.label-schema.build-date=20240521, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, GIT_CLEAN=True, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, ceph=True, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image) 2024-05-27T07:38:50.737 INFO:journalctl@ceph.osd.9.smithi052.stdout:May 27 07:38:50 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-osd-9[67134]: 2024-05-27T07:38:50.697+0000 7f1e3e2ca700 -1 osd.9 134 heartbeat_check: no reply from 172.21.15.52:6806 osd.0 since back 2024-05-27T07:37:58.861291+0000 front 2024-05-27T07:37:58.861258+0000 (oldest deadline 2024-05-27T07:38:22.361571+0000) 2024-05-27T07:38:50.737 INFO:journalctl@ceph.osd.9.smithi052.stdout:May 27 07:38:50 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-osd-9[67134]: 2024-05-27T07:38:50.697+0000 7f1e3e2ca700 -1 osd.9 134 heartbeat_check: no reply from 172.21.15.105:6804 osd.1 since back 2024-05-27T07:38:05.862960+0000 front 2024-05-27T07:38:05.862645+0000 (oldest deadline 2024-05-27T07:38:30.562694+0000) 2024-05-27T07:38:50.737 INFO:journalctl@ceph.osd.9.smithi052.stdout:May 27 07:38:50 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-osd-9[67134]: 2024-05-27T07:38:50.697+0000 7f1e3e2ca700 -1 osd.9 134 heartbeat_check: no reply from 172.21.15.173:6804 osd.2 since back 2024-05-27T07:38:15.566595+0000 front 2024-05-27T07:38:15.566403+0000 (oldest deadline 2024-05-27T07:38:37.866552+0000) 2024-05-27T07:38:50.737 INFO:journalctl@ceph.osd.9.smithi052.stdout:May 27 07:38:50 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-osd-9[67134]: 2024-05-27T07:38:50.697+0000 7f1e3e2ca700 -1 osd.9 134 heartbeat_check: no reply from 172.21.15.52:6814 osd.3 since back 2024-05-27T07:38:22.468720+0000 front 2024-05-27T07:38:22.468751+0000 (oldest deadline 2024-05-27T07:38:48.269806+0000) 2024-05-27T07:38:50.859 INFO:journalctl@ceph.osd.11.smithi173.stdout:May 27 07:38:50 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-osd-11[63021]: 2024-05-27T07:38:50.400+0000 7f9d44ac8700 -1 osd.11 134 heartbeat_check: no reply from 172.21.15.52:6806 osd.0 since back 2024-05-27T07:37:57.927796+0000 front 2024-05-27T07:37:57.927759+0000 (oldest deadline 2024-05-27T07:38:22.028025+0000) 2024-05-27T07:38:50.859 INFO:journalctl@ceph.osd.11.smithi173.stdout:May 27 07:38:50 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-osd-11[63021]: 2024-05-27T07:38:50.400+0000 7f9d44ac8700 -1 osd.11 134 heartbeat_check: no reply from 172.21.15.105:6804 osd.1 since back 2024-05-27T07:38:03.129257+0000 front 2024-05-27T07:38:03.129120+0000 (oldest deadline 2024-05-27T07:38:28.929543+0000) 2024-05-27T07:38:50.859 INFO:journalctl@ceph.osd.11.smithi173.stdout:May 27 07:38:50 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-osd-11[63021]: 2024-05-27T07:38:50.400+0000 7f9d44ac8700 -1 osd.11 134 heartbeat_check: no reply from 172.21.15.173:6804 osd.2 since back 2024-05-27T07:38:16.431856+0000 front 2024-05-27T07:38:16.431878+0000 (oldest deadline 2024-05-27T07:38:42.332069+0000) 2024-05-27T07:38:51.163 INFO:journalctl@ceph.osd.6.smithi052.stdout:May 27 07:38:50 smithi052 podman[119568]: 2024-05-27 07:38:50.818415539 +0000 UTC m=+0.018049365 image pull e36ea4fc8a5807e8d2701370e350a0de3856538797f7fba64cfce7e344bd6799 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6 2024-05-27T07:38:51.163 INFO:journalctl@ceph.osd.6.smithi052.stdout:May 27 07:38:50 smithi052 podman[119568]: 2024-05-27 07:38:50.931026463 +0000 UTC m=+0.130660266 container create 95b9967329bca7b87c215f07ec45330e8a6a7a26dfb6bd85368c3c0cf4e42da6 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-osd-6-deactivate, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240521, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, GIT_CLEAN=True, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, org.label-schema.schema-version=1.0) 2024-05-27T07:38:51.163 INFO:journalctl@ceph.osd.6.smithi052.stdout:May 27 07:38:51 smithi052 podman[119568]: 2024-05-27 07:38:51.166948441 +0000 UTC m=+0.366582264 container init 95b9967329bca7b87c215f07ec45330e8a6a7a26dfb6bd85368c3c0cf4e42da6 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-osd-6-deactivate, CEPH_POINT_RELEASE=, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240521, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD) 2024-05-27T07:38:51.197 INFO:journalctl@ceph.osd.7.smithi105.stdout:May 27 07:38:50 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-osd-7[53596]: 2024-05-27T07:38:50.778+0000 7f5eefdf1700 -1 osd.7 134 heartbeat_check: no reply from 172.21.15.105:6804 osd.1 since back 2024-05-27T07:38:06.798385+0000 front 2024-05-27T07:38:06.798353+0000 (oldest deadline 2024-05-27T07:38:32.098493+0000) 2024-05-27T07:38:51.197 INFO:journalctl@ceph.osd.7.smithi105.stdout:May 27 07:38:50 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-osd-7[53596]: 2024-05-27T07:38:50.778+0000 7f5eefdf1700 -1 osd.7 134 heartbeat_check: no reply from 172.21.15.173:6804 osd.2 since back 2024-05-27T07:38:13.199843+0000 front 2024-05-27T07:38:13.199692+0000 (oldest deadline 2024-05-27T07:38:37.899787+0000) 2024-05-27T07:38:51.197 INFO:journalctl@ceph.osd.7.smithi105.stdout:May 27 07:38:50 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-osd-7[53596]: 2024-05-27T07:38:50.778+0000 7f5eefdf1700 -1 osd.7 134 heartbeat_check: no reply from 172.21.15.52:6814 osd.3 since back 2024-05-27T07:38:23.002094+0000 front 2024-05-27T07:38:23.002188+0000 (oldest deadline 2024-05-27T07:38:47.702083+0000) 2024-05-27T07:38:51.359 INFO:journalctl@ceph.osd.8.smithi173.stdout:May 27 07:38:50 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-osd-8[55969]: 2024-05-27T07:38:50.926+0000 7f425b2ea700 -1 osd.8 134 heartbeat_check: no reply from 172.21.15.105:6804 osd.1 since back 2024-05-27T07:38:06.957078+0000 front 2024-05-27T07:38:06.957280+0000 (oldest deadline 2024-05-27T07:38:30.457072+0000) 2024-05-27T07:38:51.359 INFO:journalctl@ceph.osd.8.smithi173.stdout:May 27 07:38:50 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-osd-8[55969]: 2024-05-27T07:38:50.926+0000 7f425b2ea700 -1 osd.8 134 heartbeat_check: no reply from 172.21.15.173:6804 osd.2 since back 2024-05-27T07:38:16.258968+0000 front 2024-05-27T07:38:16.258998+0000 (oldest deadline 2024-05-27T07:38:41.559145+0000) 2024-05-27T07:38:51.359 INFO:journalctl@ceph.osd.8.smithi173.stdout:May 27 07:38:50 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-osd-8[55969]: 2024-05-27T07:38:50.926+0000 7f425b2ea700 -1 osd.8 134 heartbeat_check: no reply from 172.21.15.52:6814 osd.3 since back 2024-05-27T07:38:23.860441+0000 front 2024-05-27T07:38:23.860421+0000 (oldest deadline 2024-05-27T07:38:49.760323+0000) 2024-05-27T07:38:51.471 INFO:journalctl@ceph.osd.10.smithi105.stdout:May 27 07:38:51 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-osd-10[60334]: 2024-05-27T07:38:51.377+0000 7f162fb8f700 -1 osd.10 134 heartbeat_check: no reply from 172.21.15.52:6806 osd.0 since back 2024-05-27T07:37:56.754985+0000 front 2024-05-27T07:37:56.755070+0000 (oldest deadline 2024-05-27T07:38:22.054911+0000) 2024-05-27T07:38:51.471 INFO:journalctl@ceph.osd.10.smithi105.stdout:May 27 07:38:51 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-osd-10[60334]: 2024-05-27T07:38:51.377+0000 7f162fb8f700 -1 osd.10 134 heartbeat_check: no reply from 172.21.15.105:6804 osd.1 since back 2024-05-27T07:38:05.556137+0000 front 2024-05-27T07:38:05.556112+0000 (oldest deadline 2024-05-27T07:38:31.456213+0000) 2024-05-27T07:38:51.471 INFO:journalctl@ceph.osd.10.smithi105.stdout:May 27 07:38:51 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-osd-10[60334]: 2024-05-27T07:38:51.377+0000 7f162fb8f700 -1 osd.10 134 heartbeat_check: no reply from 172.21.15.173:6804 osd.2 since back 2024-05-27T07:38:15.557554+0000 front 2024-05-27T07:38:15.557409+0000 (oldest deadline 2024-05-27T07:38:40.857509+0000) 2024-05-27T07:38:51.471 INFO:journalctl@ceph.osd.10.smithi105.stdout:May 27 07:38:51 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-osd-10[60334]: 2024-05-27T07:38:51.377+0000 7f162fb8f700 -1 osd.10 134 heartbeat_check: no reply from 172.21.15.52:6814 osd.3 since back 2024-05-27T07:38:21.958641+0000 front 2024-05-27T07:38:21.958614+0000 (oldest deadline 2024-05-27T07:38:47.858600+0000) 2024-05-27T07:38:51.475 INFO:journalctl@ceph.osd.6.smithi052.stdout:May 27 07:38:51 smithi052 podman[119568]: 2024-05-27 07:38:51.170901722 +0000 UTC m=+0.370535539 container start 95b9967329bca7b87c215f07ec45330e8a6a7a26dfb6bd85368c3c0cf4e42da6 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-osd-6-deactivate, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, CEPH_POINT_RELEASE=, GIT_CLEAN=True, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.build-date=20240521, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD) 2024-05-27T07:38:51.475 INFO:journalctl@ceph.osd.6.smithi052.stdout:May 27 07:38:51 smithi052 podman[119568]: 2024-05-27 07:38:51.258529876 +0000 UTC m=+0.458163686 container attach 95b9967329bca7b87c215f07ec45330e8a6a7a26dfb6bd85368c3c0cf4e42da6 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-osd-6-deactivate, ceph=True, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240521, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-05-27T07:38:51.737 INFO:journalctl@ceph.osd.6.smithi052.stdout:May 27 07:38:51 smithi052 podman[119568]: 2024-05-27 07:38:51.479174062 +0000 UTC m=+0.678807888 container died 95b9967329bca7b87c215f07ec45330e8a6a7a26dfb6bd85368c3c0cf4e42da6 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-osd-6-deactivate, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240521, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image) 2024-05-27T07:38:51.737 INFO:journalctl@ceph.osd.9.smithi052.stdout:May 27 07:38:51 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-osd-9[67134]: 2024-05-27T07:38:51.660+0000 7f1e3e2ca700 -1 osd.9 134 heartbeat_check: no reply from 172.21.15.52:6806 osd.0 since back 2024-05-27T07:37:58.861291+0000 front 2024-05-27T07:37:58.861258+0000 (oldest deadline 2024-05-27T07:38:22.361571+0000) 2024-05-27T07:38:51.737 INFO:journalctl@ceph.osd.9.smithi052.stdout:May 27 07:38:51 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-osd-9[67134]: 2024-05-27T07:38:51.660+0000 7f1e3e2ca700 -1 osd.9 134 heartbeat_check: no reply from 172.21.15.105:6804 osd.1 since back 2024-05-27T07:38:05.862960+0000 front 2024-05-27T07:38:05.862645+0000 (oldest deadline 2024-05-27T07:38:30.562694+0000) 2024-05-27T07:38:51.737 INFO:journalctl@ceph.osd.9.smithi052.stdout:May 27 07:38:51 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-osd-9[67134]: 2024-05-27T07:38:51.660+0000 7f1e3e2ca700 -1 osd.9 134 heartbeat_check: no reply from 172.21.15.173:6804 osd.2 since back 2024-05-27T07:38:15.566595+0000 front 2024-05-27T07:38:15.566403+0000 (oldest deadline 2024-05-27T07:38:37.866552+0000) 2024-05-27T07:38:51.737 INFO:journalctl@ceph.osd.9.smithi052.stdout:May 27 07:38:51 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-osd-9[67134]: 2024-05-27T07:38:51.660+0000 7f1e3e2ca700 -1 osd.9 134 heartbeat_check: no reply from 172.21.15.52:6814 osd.3 since back 2024-05-27T07:38:22.468720+0000 front 2024-05-27T07:38:22.468751+0000 (oldest deadline 2024-05-27T07:38:48.269806+0000) 2024-05-27T07:38:51.859 INFO:journalctl@ceph.osd.11.smithi173.stdout:May 27 07:38:51 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-osd-11[63021]: 2024-05-27T07:38:51.376+0000 7f9d44ac8700 -1 osd.11 134 heartbeat_check: no reply from 172.21.15.52:6806 osd.0 since back 2024-05-27T07:37:57.927796+0000 front 2024-05-27T07:37:57.927759+0000 (oldest deadline 2024-05-27T07:38:22.028025+0000) 2024-05-27T07:38:51.859 INFO:journalctl@ceph.osd.11.smithi173.stdout:May 27 07:38:51 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-osd-11[63021]: 2024-05-27T07:38:51.376+0000 7f9d44ac8700 -1 osd.11 134 heartbeat_check: no reply from 172.21.15.105:6804 osd.1 since back 2024-05-27T07:38:03.129257+0000 front 2024-05-27T07:38:03.129120+0000 (oldest deadline 2024-05-27T07:38:28.929543+0000) 2024-05-27T07:38:51.859 INFO:journalctl@ceph.osd.11.smithi173.stdout:May 27 07:38:51 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-osd-11[63021]: 2024-05-27T07:38:51.376+0000 7f9d44ac8700 -1 osd.11 134 heartbeat_check: no reply from 172.21.15.173:6804 osd.2 since back 2024-05-27T07:38:16.431856+0000 front 2024-05-27T07:38:16.431878+0000 (oldest deadline 2024-05-27T07:38:42.332069+0000) 2024-05-27T07:38:52.113 DEBUG:teuthology.orchestra.run.smithi052:> sudo pkill -f 'journalctl -f -n 0 -u ceph-73df2636-1bf8-11ef-bc9b-c7b262605968@osd.6.service' 2024-05-27T07:38:52.159 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-05-27T07:38:52.160 INFO:tasks.cephadm.osd.6:Stopped osd.6 2024-05-27T07:38:52.160 INFO:tasks.cephadm.osd.7:Stopping osd.7... 2024-05-27T07:38:52.160 DEBUG:teuthology.orchestra.run.smithi105:> sudo systemctl stop ceph-73df2636-1bf8-11ef-bc9b-c7b262605968@osd.7 2024-05-27T07:38:52.182 INFO:journalctl@ceph.osd.7.smithi105.stdout:May 27 07:38:51 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-osd-7[53596]: 2024-05-27T07:38:51.782+0000 7f5eefdf1700 -1 osd.7 134 heartbeat_check: no reply from 172.21.15.105:6804 osd.1 since back 2024-05-27T07:38:06.798385+0000 front 2024-05-27T07:38:06.798353+0000 (oldest deadline 2024-05-27T07:38:32.098493+0000) 2024-05-27T07:38:52.182 INFO:journalctl@ceph.osd.7.smithi105.stdout:May 27 07:38:51 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-osd-7[53596]: 2024-05-27T07:38:51.782+0000 7f5eefdf1700 -1 osd.7 134 heartbeat_check: no reply from 172.21.15.173:6804 osd.2 since back 2024-05-27T07:38:13.199843+0000 front 2024-05-27T07:38:13.199692+0000 (oldest deadline 2024-05-27T07:38:37.899787+0000) 2024-05-27T07:38:52.182 INFO:journalctl@ceph.osd.7.smithi105.stdout:May 27 07:38:51 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-osd-7[53596]: 2024-05-27T07:38:51.782+0000 7f5eefdf1700 -1 osd.7 134 heartbeat_check: no reply from 172.21.15.52:6814 osd.3 since back 2024-05-27T07:38:23.002094+0000 front 2024-05-27T07:38:23.002188+0000 (oldest deadline 2024-05-27T07:38:47.702083+0000) 2024-05-27T07:38:52.359 INFO:journalctl@ceph.osd.8.smithi173.stdout:May 27 07:38:51 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-osd-8[55969]: 2024-05-27T07:38:51.882+0000 7f425b2ea700 -1 osd.8 134 heartbeat_check: no reply from 172.21.15.105:6804 osd.1 since back 2024-05-27T07:38:06.957078+0000 front 2024-05-27T07:38:06.957280+0000 (oldest deadline 2024-05-27T07:38:30.457072+0000) 2024-05-27T07:38:52.359 INFO:journalctl@ceph.osd.8.smithi173.stdout:May 27 07:38:51 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-osd-8[55969]: 2024-05-27T07:38:51.882+0000 7f425b2ea700 -1 osd.8 134 heartbeat_check: no reply from 172.21.15.173:6804 osd.2 since back 2024-05-27T07:38:16.258968+0000 front 2024-05-27T07:38:16.258998+0000 (oldest deadline 2024-05-27T07:38:41.559145+0000) 2024-05-27T07:38:52.359 INFO:journalctl@ceph.osd.8.smithi173.stdout:May 27 07:38:51 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-osd-8[55969]: 2024-05-27T07:38:51.882+0000 7f425b2ea700 -1 osd.8 134 heartbeat_check: no reply from 172.21.15.52:6814 osd.3 since back 2024-05-27T07:38:23.860441+0000 front 2024-05-27T07:38:23.860421+0000 (oldest deadline 2024-05-27T07:38:49.760323+0000) 2024-05-27T07:38:52.441 INFO:journalctl@ceph.osd.7.smithi105.stdout:May 27 07:38:52 smithi105 systemd[1]: Stopping Ceph osd.7 for 73df2636-1bf8-11ef-bc9b-c7b262605968... 2024-05-27T07:38:52.441 INFO:journalctl@ceph.osd.10.smithi105.stdout:May 27 07:38:52 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-osd-10[60334]: 2024-05-27T07:38:52.329+0000 7f162fb8f700 -1 osd.10 134 heartbeat_check: no reply from 172.21.15.52:6806 osd.0 since back 2024-05-27T07:37:56.754985+0000 front 2024-05-27T07:37:56.755070+0000 (oldest deadline 2024-05-27T07:38:22.054911+0000) 2024-05-27T07:38:52.441 INFO:journalctl@ceph.osd.10.smithi105.stdout:May 27 07:38:52 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-osd-10[60334]: 2024-05-27T07:38:52.329+0000 7f162fb8f700 -1 osd.10 134 heartbeat_check: no reply from 172.21.15.105:6804 osd.1 since back 2024-05-27T07:38:05.556137+0000 front 2024-05-27T07:38:05.556112+0000 (oldest deadline 2024-05-27T07:38:31.456213+0000) 2024-05-27T07:38:52.441 INFO:journalctl@ceph.osd.10.smithi105.stdout:May 27 07:38:52 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-osd-10[60334]: 2024-05-27T07:38:52.329+0000 7f162fb8f700 -1 osd.10 134 heartbeat_check: no reply from 172.21.15.173:6804 osd.2 since back 2024-05-27T07:38:15.557554+0000 front 2024-05-27T07:38:15.557409+0000 (oldest deadline 2024-05-27T07:38:40.857509+0000) 2024-05-27T07:38:52.442 INFO:journalctl@ceph.osd.10.smithi105.stdout:May 27 07:38:52 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-osd-10[60334]: 2024-05-27T07:38:52.329+0000 7f162fb8f700 -1 osd.10 134 heartbeat_check: no reply from 172.21.15.52:6814 osd.3 since back 2024-05-27T07:38:21.958641+0000 front 2024-05-27T07:38:21.958614+0000 (oldest deadline 2024-05-27T07:38:47.858600+0000) 2024-05-27T07:38:52.697 INFO:journalctl@ceph.osd.7.smithi105.stdout:May 27 07:38:52 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-osd-7[53596]: 2024-05-27T07:38:52.497+0000 7f5ef7843700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.7 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-05-27T07:38:52.697 INFO:journalctl@ceph.osd.7.smithi105.stdout:May 27 07:38:52 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-osd-7[53596]: 2024-05-27T07:38:52.497+0000 7f5ef7843700 -1 osd.7 134 *** Got signal Terminated *** 2024-05-27T07:38:52.697 INFO:journalctl@ceph.osd.7.smithi105.stdout:May 27 07:38:52 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-osd-7[53596]: 2024-05-27T07:38:52.497+0000 7f5ef7843700 -1 osd.7 134 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-05-27T07:38:52.859 INFO:journalctl@ceph.osd.11.smithi173.stdout:May 27 07:38:52 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-osd-11[63021]: 2024-05-27T07:38:52.392+0000 7f9d44ac8700 -1 osd.11 134 heartbeat_check: no reply from 172.21.15.52:6806 osd.0 since back 2024-05-27T07:37:57.927796+0000 front 2024-05-27T07:37:57.927759+0000 (oldest deadline 2024-05-27T07:38:22.028025+0000) 2024-05-27T07:38:52.859 INFO:journalctl@ceph.osd.11.smithi173.stdout:May 27 07:38:52 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-osd-11[63021]: 2024-05-27T07:38:52.392+0000 7f9d44ac8700 -1 osd.11 134 heartbeat_check: no reply from 172.21.15.105:6804 osd.1 since back 2024-05-27T07:38:03.129257+0000 front 2024-05-27T07:38:03.129120+0000 (oldest deadline 2024-05-27T07:38:28.929543+0000) 2024-05-27T07:38:52.859 INFO:journalctl@ceph.osd.11.smithi173.stdout:May 27 07:38:52 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-osd-11[63021]: 2024-05-27T07:38:52.392+0000 7f9d44ac8700 -1 osd.11 134 heartbeat_check: no reply from 172.21.15.173:6804 osd.2 since back 2024-05-27T07:38:16.431856+0000 front 2024-05-27T07:38:16.431878+0000 (oldest deadline 2024-05-27T07:38:42.332069+0000) 2024-05-27T07:38:52.986 INFO:journalctl@ceph.osd.9.smithi052.stdout:May 27 07:38:52 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-osd-9[67134]: 2024-05-27T07:38:52.688+0000 7f1e3e2ca700 -1 osd.9 134 heartbeat_check: no reply from 172.21.15.52:6806 osd.0 since back 2024-05-27T07:37:58.861291+0000 front 2024-05-27T07:37:58.861258+0000 (oldest deadline 2024-05-27T07:38:22.361571+0000) 2024-05-27T07:38:52.986 INFO:journalctl@ceph.osd.9.smithi052.stdout:May 27 07:38:52 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-osd-9[67134]: 2024-05-27T07:38:52.688+0000 7f1e3e2ca700 -1 osd.9 134 heartbeat_check: no reply from 172.21.15.105:6804 osd.1 since back 2024-05-27T07:38:05.862960+0000 front 2024-05-27T07:38:05.862645+0000 (oldest deadline 2024-05-27T07:38:30.562694+0000) 2024-05-27T07:38:52.986 INFO:journalctl@ceph.osd.9.smithi052.stdout:May 27 07:38:52 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-osd-9[67134]: 2024-05-27T07:38:52.688+0000 7f1e3e2ca700 -1 osd.9 134 heartbeat_check: no reply from 172.21.15.173:6804 osd.2 since back 2024-05-27T07:38:15.566595+0000 front 2024-05-27T07:38:15.566403+0000 (oldest deadline 2024-05-27T07:38:37.866552+0000) 2024-05-27T07:38:52.987 INFO:journalctl@ceph.osd.9.smithi052.stdout:May 27 07:38:52 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-osd-9[67134]: 2024-05-27T07:38:52.688+0000 7f1e3e2ca700 -1 osd.9 134 heartbeat_check: no reply from 172.21.15.52:6814 osd.3 since back 2024-05-27T07:38:22.468720+0000 front 2024-05-27T07:38:22.468751+0000 (oldest deadline 2024-05-27T07:38:48.269806+0000) 2024-05-27T07:38:53.197 INFO:journalctl@ceph.osd.7.smithi105.stdout:May 27 07:38:52 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-osd-7[53596]: 2024-05-27T07:38:52.787+0000 7f5eefdf1700 -1 osd.7 134 heartbeat_check: no reply from 172.21.15.105:6804 osd.1 since back 2024-05-27T07:38:06.798385+0000 front 2024-05-27T07:38:06.798353+0000 (oldest deadline 2024-05-27T07:38:32.098493+0000) 2024-05-27T07:38:53.197 INFO:journalctl@ceph.osd.7.smithi105.stdout:May 27 07:38:52 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-osd-7[53596]: 2024-05-27T07:38:52.787+0000 7f5eefdf1700 -1 osd.7 134 heartbeat_check: no reply from 172.21.15.173:6804 osd.2 since back 2024-05-27T07:38:13.199843+0000 front 2024-05-27T07:38:13.199692+0000 (oldest deadline 2024-05-27T07:38:37.899787+0000) 2024-05-27T07:38:53.197 INFO:journalctl@ceph.osd.7.smithi105.stdout:May 27 07:38:52 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-osd-7[53596]: 2024-05-27T07:38:52.787+0000 7f5eefdf1700 -1 osd.7 134 heartbeat_check: no reply from 172.21.15.52:6814 osd.3 since back 2024-05-27T07:38:23.002094+0000 front 2024-05-27T07:38:23.002188+0000 (oldest deadline 2024-05-27T07:38:47.702083+0000) 2024-05-27T07:38:53.359 INFO:journalctl@ceph.osd.8.smithi173.stdout:May 27 07:38:52 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-osd-8[55969]: 2024-05-27T07:38:52.868+0000 7f425b2ea700 -1 osd.8 134 heartbeat_check: no reply from 172.21.15.105:6804 osd.1 since back 2024-05-27T07:38:06.957078+0000 front 2024-05-27T07:38:06.957280+0000 (oldest deadline 2024-05-27T07:38:30.457072+0000) 2024-05-27T07:38:53.359 INFO:journalctl@ceph.osd.8.smithi173.stdout:May 27 07:38:52 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-osd-8[55969]: 2024-05-27T07:38:52.868+0000 7f425b2ea700 -1 osd.8 134 heartbeat_check: no reply from 172.21.15.173:6804 osd.2 since back 2024-05-27T07:38:16.258968+0000 front 2024-05-27T07:38:16.258998+0000 (oldest deadline 2024-05-27T07:38:41.559145+0000) 2024-05-27T07:38:53.359 INFO:journalctl@ceph.osd.8.smithi173.stdout:May 27 07:38:52 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-osd-8[55969]: 2024-05-27T07:38:52.868+0000 7f425b2ea700 -1 osd.8 134 heartbeat_check: no reply from 172.21.15.52:6814 osd.3 since back 2024-05-27T07:38:23.860441+0000 front 2024-05-27T07:38:23.860421+0000 (oldest deadline 2024-05-27T07:38:49.760323+0000) 2024-05-27T07:38:53.697 INFO:journalctl@ceph.osd.10.smithi105.stdout:May 27 07:38:53 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-osd-10[60334]: 2024-05-27T07:38:53.367+0000 7f162fb8f700 -1 osd.10 134 heartbeat_check: no reply from 172.21.15.52:6806 osd.0 since back 2024-05-27T07:37:56.754985+0000 front 2024-05-27T07:37:56.755070+0000 (oldest deadline 2024-05-27T07:38:22.054911+0000) 2024-05-27T07:38:53.697 INFO:journalctl@ceph.osd.10.smithi105.stdout:May 27 07:38:53 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-osd-10[60334]: 2024-05-27T07:38:53.367+0000 7f162fb8f700 -1 osd.10 134 heartbeat_check: no reply from 172.21.15.105:6804 osd.1 since back 2024-05-27T07:38:05.556137+0000 front 2024-05-27T07:38:05.556112+0000 (oldest deadline 2024-05-27T07:38:31.456213+0000) 2024-05-27T07:38:53.697 INFO:journalctl@ceph.osd.10.smithi105.stdout:May 27 07:38:53 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-osd-10[60334]: 2024-05-27T07:38:53.367+0000 7f162fb8f700 -1 osd.10 134 heartbeat_check: no reply from 172.21.15.173:6804 osd.2 since back 2024-05-27T07:38:15.557554+0000 front 2024-05-27T07:38:15.557409+0000 (oldest deadline 2024-05-27T07:38:40.857509+0000) 2024-05-27T07:38:53.697 INFO:journalctl@ceph.osd.10.smithi105.stdout:May 27 07:38:53 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-osd-10[60334]: 2024-05-27T07:38:53.367+0000 7f162fb8f700 -1 osd.10 134 heartbeat_check: no reply from 172.21.15.52:6814 osd.3 since back 2024-05-27T07:38:21.958641+0000 front 2024-05-27T07:38:21.958614+0000 (oldest deadline 2024-05-27T07:38:47.858600+0000) 2024-05-27T07:38:53.859 INFO:journalctl@ceph.osd.11.smithi173.stdout:May 27 07:38:53 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-osd-11[63021]: 2024-05-27T07:38:53.369+0000 7f9d44ac8700 -1 osd.11 134 heartbeat_check: no reply from 172.21.15.52:6806 osd.0 since back 2024-05-27T07:37:57.927796+0000 front 2024-05-27T07:37:57.927759+0000 (oldest deadline 2024-05-27T07:38:22.028025+0000) 2024-05-27T07:38:53.859 INFO:journalctl@ceph.osd.11.smithi173.stdout:May 27 07:38:53 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-osd-11[63021]: 2024-05-27T07:38:53.369+0000 7f9d44ac8700 -1 osd.11 134 heartbeat_check: no reply from 172.21.15.105:6804 osd.1 since back 2024-05-27T07:38:03.129257+0000 front 2024-05-27T07:38:03.129120+0000 (oldest deadline 2024-05-27T07:38:28.929543+0000) 2024-05-27T07:38:53.859 INFO:journalctl@ceph.osd.11.smithi173.stdout:May 27 07:38:53 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-osd-11[63021]: 2024-05-27T07:38:53.369+0000 7f9d44ac8700 -1 osd.11 134 heartbeat_check: no reply from 172.21.15.173:6804 osd.2 since back 2024-05-27T07:38:16.431856+0000 front 2024-05-27T07:38:16.431878+0000 (oldest deadline 2024-05-27T07:38:42.332069+0000) 2024-05-27T07:38:53.986 INFO:journalctl@ceph.osd.9.smithi052.stdout:May 27 07:38:53 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-osd-9[67134]: 2024-05-27T07:38:53.722+0000 7f1e3e2ca700 -1 osd.9 134 heartbeat_check: no reply from 172.21.15.52:6806 osd.0 since back 2024-05-27T07:37:58.861291+0000 front 2024-05-27T07:37:58.861258+0000 (oldest deadline 2024-05-27T07:38:22.361571+0000) 2024-05-27T07:38:53.986 INFO:journalctl@ceph.osd.9.smithi052.stdout:May 27 07:38:53 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-osd-9[67134]: 2024-05-27T07:38:53.722+0000 7f1e3e2ca700 -1 osd.9 134 heartbeat_check: no reply from 172.21.15.105:6804 osd.1 since back 2024-05-27T07:38:05.862960+0000 front 2024-05-27T07:38:05.862645+0000 (oldest deadline 2024-05-27T07:38:30.562694+0000) 2024-05-27T07:38:53.986 INFO:journalctl@ceph.osd.9.smithi052.stdout:May 27 07:38:53 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-osd-9[67134]: 2024-05-27T07:38:53.722+0000 7f1e3e2ca700 -1 osd.9 134 heartbeat_check: no reply from 172.21.15.173:6804 osd.2 since back 2024-05-27T07:38:15.566595+0000 front 2024-05-27T07:38:15.566403+0000 (oldest deadline 2024-05-27T07:38:37.866552+0000) 2024-05-27T07:38:53.987 INFO:journalctl@ceph.osd.9.smithi052.stdout:May 27 07:38:53 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-osd-9[67134]: 2024-05-27T07:38:53.722+0000 7f1e3e2ca700 -1 osd.9 134 heartbeat_check: no reply from 172.21.15.52:6814 osd.3 since back 2024-05-27T07:38:22.468720+0000 front 2024-05-27T07:38:22.468751+0000 (oldest deadline 2024-05-27T07:38:48.269806+0000) 2024-05-27T07:38:54.197 INFO:journalctl@ceph.osd.7.smithi105.stdout:May 27 07:38:53 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-osd-7[53596]: 2024-05-27T07:38:53.792+0000 7f5eefdf1700 -1 osd.7 134 heartbeat_check: no reply from 172.21.15.105:6804 osd.1 since back 2024-05-27T07:38:06.798385+0000 front 2024-05-27T07:38:06.798353+0000 (oldest deadline 2024-05-27T07:38:32.098493+0000) 2024-05-27T07:38:54.197 INFO:journalctl@ceph.osd.7.smithi105.stdout:May 27 07:38:53 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-osd-7[53596]: 2024-05-27T07:38:53.792+0000 7f5eefdf1700 -1 osd.7 134 heartbeat_check: no reply from 172.21.15.173:6804 osd.2 since back 2024-05-27T07:38:13.199843+0000 front 2024-05-27T07:38:13.199692+0000 (oldest deadline 2024-05-27T07:38:37.899787+0000) 2024-05-27T07:38:54.197 INFO:journalctl@ceph.osd.7.smithi105.stdout:May 27 07:38:53 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-osd-7[53596]: 2024-05-27T07:38:53.792+0000 7f5eefdf1700 -1 osd.7 134 heartbeat_check: no reply from 172.21.15.52:6814 osd.3 since back 2024-05-27T07:38:23.002094+0000 front 2024-05-27T07:38:23.002188+0000 (oldest deadline 2024-05-27T07:38:47.702083+0000) 2024-05-27T07:38:54.359 INFO:journalctl@ceph.osd.8.smithi173.stdout:May 27 07:38:53 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-osd-8[55969]: 2024-05-27T07:38:53.888+0000 7f425b2ea700 -1 osd.8 134 heartbeat_check: no reply from 172.21.15.105:6804 osd.1 since back 2024-05-27T07:38:06.957078+0000 front 2024-05-27T07:38:06.957280+0000 (oldest deadline 2024-05-27T07:38:30.457072+0000) 2024-05-27T07:38:54.359 INFO:journalctl@ceph.osd.8.smithi173.stdout:May 27 07:38:53 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-osd-8[55969]: 2024-05-27T07:38:53.888+0000 7f425b2ea700 -1 osd.8 134 heartbeat_check: no reply from 172.21.15.173:6804 osd.2 since back 2024-05-27T07:38:16.258968+0000 front 2024-05-27T07:38:16.258998+0000 (oldest deadline 2024-05-27T07:38:41.559145+0000) 2024-05-27T07:38:54.359 INFO:journalctl@ceph.osd.8.smithi173.stdout:May 27 07:38:53 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-osd-8[55969]: 2024-05-27T07:38:53.888+0000 7f425b2ea700 -1 osd.8 134 heartbeat_check: no reply from 172.21.15.52:6814 osd.3 since back 2024-05-27T07:38:23.860441+0000 front 2024-05-27T07:38:23.860421+0000 (oldest deadline 2024-05-27T07:38:49.760323+0000) 2024-05-27T07:38:54.359 INFO:journalctl@ceph.osd.8.smithi173.stdout:May 27 07:38:53 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-osd-8[55969]: 2024-05-27T07:38:53.888+0000 7f425b2ea700 -1 osd.8 134 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-05-27T07:38:32.562246+0000 front 2024-05-27T07:38:32.562420+0000 (oldest deadline 2024-05-27T07:38:53.662280+0000) 2024-05-27T07:38:54.697 INFO:journalctl@ceph.osd.10.smithi105.stdout:May 27 07:38:54 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-osd-10[60334]: 2024-05-27T07:38:54.352+0000 7f162fb8f700 -1 osd.10 134 heartbeat_check: no reply from 172.21.15.52:6806 osd.0 since back 2024-05-27T07:37:56.754985+0000 front 2024-05-27T07:37:56.755070+0000 (oldest deadline 2024-05-27T07:38:22.054911+0000) 2024-05-27T07:38:54.697 INFO:journalctl@ceph.osd.10.smithi105.stdout:May 27 07:38:54 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-osd-10[60334]: 2024-05-27T07:38:54.352+0000 7f162fb8f700 -1 osd.10 134 heartbeat_check: no reply from 172.21.15.105:6804 osd.1 since back 2024-05-27T07:38:05.556137+0000 front 2024-05-27T07:38:05.556112+0000 (oldest deadline 2024-05-27T07:38:31.456213+0000) 2024-05-27T07:38:54.697 INFO:journalctl@ceph.osd.10.smithi105.stdout:May 27 07:38:54 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-osd-10[60334]: 2024-05-27T07:38:54.352+0000 7f162fb8f700 -1 osd.10 134 heartbeat_check: no reply from 172.21.15.173:6804 osd.2 since back 2024-05-27T07:38:15.557554+0000 front 2024-05-27T07:38:15.557409+0000 (oldest deadline 2024-05-27T07:38:40.857509+0000) 2024-05-27T07:38:54.697 INFO:journalctl@ceph.osd.10.smithi105.stdout:May 27 07:38:54 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-osd-10[60334]: 2024-05-27T07:38:54.352+0000 7f162fb8f700 -1 osd.10 134 heartbeat_check: no reply from 172.21.15.52:6814 osd.3 since back 2024-05-27T07:38:21.958641+0000 front 2024-05-27T07:38:21.958614+0000 (oldest deadline 2024-05-27T07:38:47.858600+0000) 2024-05-27T07:38:54.859 INFO:journalctl@ceph.osd.11.smithi173.stdout:May 27 07:38:54 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-osd-11[63021]: 2024-05-27T07:38:54.367+0000 7f9d44ac8700 -1 osd.11 134 heartbeat_check: no reply from 172.21.15.52:6806 osd.0 since back 2024-05-27T07:37:57.927796+0000 front 2024-05-27T07:37:57.927759+0000 (oldest deadline 2024-05-27T07:38:22.028025+0000) 2024-05-27T07:38:54.859 INFO:journalctl@ceph.osd.11.smithi173.stdout:May 27 07:38:54 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-osd-11[63021]: 2024-05-27T07:38:54.367+0000 7f9d44ac8700 -1 osd.11 134 heartbeat_check: no reply from 172.21.15.105:6804 osd.1 since back 2024-05-27T07:38:03.129257+0000 front 2024-05-27T07:38:03.129120+0000 (oldest deadline 2024-05-27T07:38:28.929543+0000) 2024-05-27T07:38:54.859 INFO:journalctl@ceph.osd.11.smithi173.stdout:May 27 07:38:54 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-osd-11[63021]: 2024-05-27T07:38:54.367+0000 7f9d44ac8700 -1 osd.11 134 heartbeat_check: no reply from 172.21.15.173:6804 osd.2 since back 2024-05-27T07:38:16.431856+0000 front 2024-05-27T07:38:16.431878+0000 (oldest deadline 2024-05-27T07:38:42.332069+0000) 2024-05-27T07:38:55.197 INFO:journalctl@ceph.osd.7.smithi105.stdout:May 27 07:38:54 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-osd-7[53596]: 2024-05-27T07:38:54.828+0000 7f5eefdf1700 -1 osd.7 134 heartbeat_check: no reply from 172.21.15.105:6804 osd.1 since back 2024-05-27T07:38:06.798385+0000 front 2024-05-27T07:38:06.798353+0000 (oldest deadline 2024-05-27T07:38:32.098493+0000) 2024-05-27T07:38:55.405 INFO:journalctl@ceph.osd.7.smithi105.stdout:May 27 07:38:54 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-osd-7[53596]: 2024-05-27T07:38:54.828+0000 7f5eefdf1700 -1 osd.7 134 heartbeat_check: no reply from 172.21.15.173:6804 osd.2 since back 2024-05-27T07:38:13.199843+0000 front 2024-05-27T07:38:13.199692+0000 (oldest deadline 2024-05-27T07:38:37.899787+0000) 2024-05-27T07:38:55.405 INFO:journalctl@ceph.osd.7.smithi105.stdout:May 27 07:38:54 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-osd-7[53596]: 2024-05-27T07:38:54.828+0000 7f5eefdf1700 -1 osd.7 134 heartbeat_check: no reply from 172.21.15.52:6814 osd.3 since back 2024-05-27T07:38:23.002094+0000 front 2024-05-27T07:38:23.002188+0000 (oldest deadline 2024-05-27T07:38:47.702083+0000) 2024-05-27T07:38:55.406 INFO:journalctl@ceph.osd.8.smithi173.stdout:May 27 07:38:54 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-osd-8[55969]: 2024-05-27T07:38:54.862+0000 7f425b2ea700 -1 osd.8 134 heartbeat_check: no reply from 172.21.15.105:6804 osd.1 since back 2024-05-27T07:38:06.957078+0000 front 2024-05-27T07:38:06.957280+0000 (oldest deadline 2024-05-27T07:38:30.457072+0000) 2024-05-27T07:38:55.406 INFO:journalctl@ceph.osd.8.smithi173.stdout:May 27 07:38:54 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-osd-8[55969]: 2024-05-27T07:38:54.862+0000 7f425b2ea700 -1 osd.8 134 heartbeat_check: no reply from 172.21.15.173:6804 osd.2 since back 2024-05-27T07:38:16.258968+0000 front 2024-05-27T07:38:16.258998+0000 (oldest deadline 2024-05-27T07:38:41.559145+0000) 2024-05-27T07:38:55.406 INFO:journalctl@ceph.osd.8.smithi173.stdout:May 27 07:38:54 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-osd-8[55969]: 2024-05-27T07:38:54.862+0000 7f425b2ea700 -1 osd.8 134 heartbeat_check: no reply from 172.21.15.52:6814 osd.3 since back 2024-05-27T07:38:23.860441+0000 front 2024-05-27T07:38:23.860421+0000 (oldest deadline 2024-05-27T07:38:49.760323+0000) 2024-05-27T07:38:55.406 INFO:journalctl@ceph.osd.8.smithi173.stdout:May 27 07:38:54 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-osd-8[55969]: 2024-05-27T07:38:54.862+0000 7f425b2ea700 -1 osd.8 134 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-05-27T07:38:32.562246+0000 front 2024-05-27T07:38:32.562420+0000 (oldest deadline 2024-05-27T07:38:53.662280+0000) 2024-05-27T07:38:55.406 INFO:journalctl@ceph.osd.9.smithi052.stdout:May 27 07:38:54 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-osd-9[67134]: 2024-05-27T07:38:54.742+0000 7f1e3e2ca700 -1 osd.9 134 heartbeat_check: no reply from 172.21.15.52:6806 osd.0 since back 2024-05-27T07:37:58.861291+0000 front 2024-05-27T07:37:58.861258+0000 (oldest deadline 2024-05-27T07:38:22.361571+0000) 2024-05-27T07:38:55.406 INFO:journalctl@ceph.osd.9.smithi052.stdout:May 27 07:38:54 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-osd-9[67134]: 2024-05-27T07:38:54.742+0000 7f1e3e2ca700 -1 osd.9 134 heartbeat_check: no reply from 172.21.15.105:6804 osd.1 since back 2024-05-27T07:38:05.862960+0000 front 2024-05-27T07:38:05.862645+0000 (oldest deadline 2024-05-27T07:38:30.562694+0000) 2024-05-27T07:38:55.407 INFO:journalctl@ceph.osd.9.smithi052.stdout:May 27 07:38:54 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-osd-9[67134]: 2024-05-27T07:38:54.742+0000 7f1e3e2ca700 -1 osd.9 134 heartbeat_check: no reply from 172.21.15.173:6804 osd.2 since back 2024-05-27T07:38:15.566595+0000 front 2024-05-27T07:38:15.566403+0000 (oldest deadline 2024-05-27T07:38:37.866552+0000) 2024-05-27T07:38:55.407 INFO:journalctl@ceph.osd.9.smithi052.stdout:May 27 07:38:54 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-osd-9[67134]: 2024-05-27T07:38:54.742+0000 7f1e3e2ca700 -1 osd.9 134 heartbeat_check: no reply from 172.21.15.52:6814 osd.3 since back 2024-05-27T07:38:22.468720+0000 front 2024-05-27T07:38:22.468751+0000 (oldest deadline 2024-05-27T07:38:48.269806+0000) 2024-05-27T07:38:55.610 INFO:journalctl@ceph.osd.11.smithi173.stdout:May 27 07:38:55 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-osd-11[63021]: 2024-05-27T07:38:55.325+0000 7f9d44ac8700 -1 osd.11 134 heartbeat_check: no reply from 172.21.15.52:6806 osd.0 since back 2024-05-27T07:37:57.927796+0000 front 2024-05-27T07:37:57.927759+0000 (oldest deadline 2024-05-27T07:38:22.028025+0000) 2024-05-27T07:38:55.610 INFO:journalctl@ceph.osd.11.smithi173.stdout:May 27 07:38:55 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-osd-11[63021]: 2024-05-27T07:38:55.325+0000 7f9d44ac8700 -1 osd.11 134 heartbeat_check: no reply from 172.21.15.105:6804 osd.1 since back 2024-05-27T07:38:03.129257+0000 front 2024-05-27T07:38:03.129120+0000 (oldest deadline 2024-05-27T07:38:28.929543+0000) 2024-05-27T07:38:55.610 INFO:journalctl@ceph.osd.11.smithi173.stdout:May 27 07:38:55 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-osd-11[63021]: 2024-05-27T07:38:55.325+0000 7f9d44ac8700 -1 osd.11 134 heartbeat_check: no reply from 172.21.15.173:6804 osd.2 since back 2024-05-27T07:38:16.431856+0000 front 2024-05-27T07:38:16.431878+0000 (oldest deadline 2024-05-27T07:38:42.332069+0000) 2024-05-27T07:38:55.697 INFO:journalctl@ceph.osd.10.smithi105.stdout:May 27 07:38:55 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-osd-10[60334]: 2024-05-27T07:38:55.396+0000 7f162fb8f700 -1 osd.10 134 heartbeat_check: no reply from 172.21.15.52:6806 osd.0 since back 2024-05-27T07:37:56.754985+0000 front 2024-05-27T07:37:56.755070+0000 (oldest deadline 2024-05-27T07:38:22.054911+0000) 2024-05-27T07:38:55.697 INFO:journalctl@ceph.osd.10.smithi105.stdout:May 27 07:38:55 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-osd-10[60334]: 2024-05-27T07:38:55.396+0000 7f162fb8f700 -1 osd.10 134 heartbeat_check: no reply from 172.21.15.105:6804 osd.1 since back 2024-05-27T07:38:05.556137+0000 front 2024-05-27T07:38:05.556112+0000 (oldest deadline 2024-05-27T07:38:31.456213+0000) 2024-05-27T07:38:55.697 INFO:journalctl@ceph.osd.10.smithi105.stdout:May 27 07:38:55 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-osd-10[60334]: 2024-05-27T07:38:55.396+0000 7f162fb8f700 -1 osd.10 134 heartbeat_check: no reply from 172.21.15.173:6804 osd.2 since back 2024-05-27T07:38:15.557554+0000 front 2024-05-27T07:38:15.557409+0000 (oldest deadline 2024-05-27T07:38:40.857509+0000) 2024-05-27T07:38:55.697 INFO:journalctl@ceph.osd.10.smithi105.stdout:May 27 07:38:55 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-osd-10[60334]: 2024-05-27T07:38:55.396+0000 7f162fb8f700 -1 osd.10 134 heartbeat_check: no reply from 172.21.15.52:6814 osd.3 since back 2024-05-27T07:38:21.958641+0000 front 2024-05-27T07:38:21.958614+0000 (oldest deadline 2024-05-27T07:38:47.858600+0000) 2024-05-27T07:38:55.697 INFO:journalctl@ceph.osd.10.smithi105.stdout:May 27 07:38:55 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-osd-10[60334]: 2024-05-27T07:38:55.396+0000 7f162fb8f700 -1 osd.10 134 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-05-27T07:38:30.560765+0000 front 2024-05-27T07:38:30.560788+0000 (oldest deadline 2024-05-27T07:38:55.260887+0000) 2024-05-27T07:38:56.109 INFO:journalctl@ceph.osd.8.smithi173.stdout:May 27 07:38:55 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-osd-8[55969]: 2024-05-27T07:38:55.857+0000 7f425b2ea700 -1 osd.8 134 heartbeat_check: no reply from 172.21.15.105:6804 osd.1 since back 2024-05-27T07:38:06.957078+0000 front 2024-05-27T07:38:06.957280+0000 (oldest deadline 2024-05-27T07:38:30.457072+0000) 2024-05-27T07:38:56.109 INFO:journalctl@ceph.osd.8.smithi173.stdout:May 27 07:38:55 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-osd-8[55969]: 2024-05-27T07:38:55.857+0000 7f425b2ea700 -1 osd.8 134 heartbeat_check: no reply from 172.21.15.173:6804 osd.2 since back 2024-05-27T07:38:16.258968+0000 front 2024-05-27T07:38:16.258998+0000 (oldest deadline 2024-05-27T07:38:41.559145+0000) 2024-05-27T07:38:56.109 INFO:journalctl@ceph.osd.8.smithi173.stdout:May 27 07:38:55 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-osd-8[55969]: 2024-05-27T07:38:55.857+0000 7f425b2ea700 -1 osd.8 134 heartbeat_check: no reply from 172.21.15.52:6814 osd.3 since back 2024-05-27T07:38:23.860441+0000 front 2024-05-27T07:38:23.860421+0000 (oldest deadline 2024-05-27T07:38:49.760323+0000) 2024-05-27T07:38:56.109 INFO:journalctl@ceph.osd.8.smithi173.stdout:May 27 07:38:55 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-osd-8[55969]: 2024-05-27T07:38:55.857+0000 7f425b2ea700 -1 osd.8 134 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-05-27T07:38:32.562246+0000 front 2024-05-27T07:38:32.562420+0000 (oldest deadline 2024-05-27T07:38:53.662280+0000) 2024-05-27T07:38:56.197 INFO:journalctl@ceph.osd.7.smithi105.stdout:May 27 07:38:55 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-osd-7[53596]: 2024-05-27T07:38:55.860+0000 7f5eefdf1700 -1 osd.7 134 heartbeat_check: no reply from 172.21.15.105:6804 osd.1 since back 2024-05-27T07:38:06.798385+0000 front 2024-05-27T07:38:06.798353+0000 (oldest deadline 2024-05-27T07:38:32.098493+0000) 2024-05-27T07:38:56.197 INFO:journalctl@ceph.osd.7.smithi105.stdout:May 27 07:38:55 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-osd-7[53596]: 2024-05-27T07:38:55.860+0000 7f5eefdf1700 -1 osd.7 134 heartbeat_check: no reply from 172.21.15.173:6804 osd.2 since back 2024-05-27T07:38:13.199843+0000 front 2024-05-27T07:38:13.199692+0000 (oldest deadline 2024-05-27T07:38:37.899787+0000) 2024-05-27T07:38:56.197 INFO:journalctl@ceph.osd.7.smithi105.stdout:May 27 07:38:55 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-osd-7[53596]: 2024-05-27T07:38:55.860+0000 7f5eefdf1700 -1 osd.7 134 heartbeat_check: no reply from 172.21.15.52:6814 osd.3 since back 2024-05-27T07:38:23.002094+0000 front 2024-05-27T07:38:23.002188+0000 (oldest deadline 2024-05-27T07:38:47.702083+0000) 2024-05-27T07:38:56.236 INFO:journalctl@ceph.osd.9.smithi052.stdout:May 27 07:38:55 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-osd-9[67134]: 2024-05-27T07:38:55.747+0000 7f1e3e2ca700 -1 osd.9 134 heartbeat_check: no reply from 172.21.15.52:6806 osd.0 since back 2024-05-27T07:37:58.861291+0000 front 2024-05-27T07:37:58.861258+0000 (oldest deadline 2024-05-27T07:38:22.361571+0000) 2024-05-27T07:38:56.236 INFO:journalctl@ceph.osd.9.smithi052.stdout:May 27 07:38:55 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-osd-9[67134]: 2024-05-27T07:38:55.747+0000 7f1e3e2ca700 -1 osd.9 134 heartbeat_check: no reply from 172.21.15.105:6804 osd.1 since back 2024-05-27T07:38:05.862960+0000 front 2024-05-27T07:38:05.862645+0000 (oldest deadline 2024-05-27T07:38:30.562694+0000) 2024-05-27T07:38:56.236 INFO:journalctl@ceph.osd.9.smithi052.stdout:May 27 07:38:55 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-osd-9[67134]: 2024-05-27T07:38:55.747+0000 7f1e3e2ca700 -1 osd.9 134 heartbeat_check: no reply from 172.21.15.173:6804 osd.2 since back 2024-05-27T07:38:15.566595+0000 front 2024-05-27T07:38:15.566403+0000 (oldest deadline 2024-05-27T07:38:37.866552+0000) 2024-05-27T07:38:56.236 INFO:journalctl@ceph.osd.9.smithi052.stdout:May 27 07:38:55 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-osd-9[67134]: 2024-05-27T07:38:55.747+0000 7f1e3e2ca700 -1 osd.9 134 heartbeat_check: no reply from 172.21.15.52:6814 osd.3 since back 2024-05-27T07:38:22.468720+0000 front 2024-05-27T07:38:22.468751+0000 (oldest deadline 2024-05-27T07:38:48.269806+0000) 2024-05-27T07:38:56.609 INFO:journalctl@ceph.osd.11.smithi173.stdout:May 27 07:38:56 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-osd-11[63021]: 2024-05-27T07:38:56.343+0000 7f9d44ac8700 -1 osd.11 134 heartbeat_check: no reply from 172.21.15.52:6806 osd.0 since back 2024-05-27T07:37:57.927796+0000 front 2024-05-27T07:37:57.927759+0000 (oldest deadline 2024-05-27T07:38:22.028025+0000) 2024-05-27T07:38:56.609 INFO:journalctl@ceph.osd.11.smithi173.stdout:May 27 07:38:56 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-osd-11[63021]: 2024-05-27T07:38:56.343+0000 7f9d44ac8700 -1 osd.11 134 heartbeat_check: no reply from 172.21.15.105:6804 osd.1 since back 2024-05-27T07:38:03.129257+0000 front 2024-05-27T07:38:03.129120+0000 (oldest deadline 2024-05-27T07:38:28.929543+0000) 2024-05-27T07:38:56.609 INFO:journalctl@ceph.osd.11.smithi173.stdout:May 27 07:38:56 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-osd-11[63021]: 2024-05-27T07:38:56.343+0000 7f9d44ac8700 -1 osd.11 134 heartbeat_check: no reply from 172.21.15.173:6804 osd.2 since back 2024-05-27T07:38:16.431856+0000 front 2024-05-27T07:38:16.431878+0000 (oldest deadline 2024-05-27T07:38:42.332069+0000) 2024-05-27T07:38:56.609 INFO:journalctl@ceph.osd.11.smithi173.stdout:May 27 07:38:56 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-osd-11[63021]: 2024-05-27T07:38:56.343+0000 7f9d44ac8700 -1 osd.11 134 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-05-27T07:38:28.133691+0000 front 2024-05-27T07:38:28.134117+0000 (oldest deadline 2024-05-27T07:38:55.734329+0000) 2024-05-27T07:38:56.697 INFO:journalctl@ceph.osd.10.smithi105.stdout:May 27 07:38:56 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-osd-10[60334]: 2024-05-27T07:38:56.408+0000 7f162fb8f700 -1 osd.10 134 heartbeat_check: no reply from 172.21.15.52:6806 osd.0 since back 2024-05-27T07:37:56.754985+0000 front 2024-05-27T07:37:56.755070+0000 (oldest deadline 2024-05-27T07:38:22.054911+0000) 2024-05-27T07:38:56.697 INFO:journalctl@ceph.osd.10.smithi105.stdout:May 27 07:38:56 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-osd-10[60334]: 2024-05-27T07:38:56.408+0000 7f162fb8f700 -1 osd.10 134 heartbeat_check: no reply from 172.21.15.105:6804 osd.1 since back 2024-05-27T07:38:05.556137+0000 front 2024-05-27T07:38:05.556112+0000 (oldest deadline 2024-05-27T07:38:31.456213+0000) 2024-05-27T07:38:56.697 INFO:journalctl@ceph.osd.10.smithi105.stdout:May 27 07:38:56 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-osd-10[60334]: 2024-05-27T07:38:56.408+0000 7f162fb8f700 -1 osd.10 134 heartbeat_check: no reply from 172.21.15.173:6804 osd.2 since back 2024-05-27T07:38:15.557554+0000 front 2024-05-27T07:38:15.557409+0000 (oldest deadline 2024-05-27T07:38:40.857509+0000) 2024-05-27T07:38:56.697 INFO:journalctl@ceph.osd.10.smithi105.stdout:May 27 07:38:56 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-osd-10[60334]: 2024-05-27T07:38:56.408+0000 7f162fb8f700 -1 osd.10 134 heartbeat_check: no reply from 172.21.15.52:6814 osd.3 since back 2024-05-27T07:38:21.958641+0000 front 2024-05-27T07:38:21.958614+0000 (oldest deadline 2024-05-27T07:38:47.858600+0000) 2024-05-27T07:38:56.697 INFO:journalctl@ceph.osd.10.smithi105.stdout:May 27 07:38:56 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-osd-10[60334]: 2024-05-27T07:38:56.408+0000 7f162fb8f700 -1 osd.10 134 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-05-27T07:38:30.560765+0000 front 2024-05-27T07:38:30.560788+0000 (oldest deadline 2024-05-27T07:38:55.260887+0000) 2024-05-27T07:38:56.986 INFO:journalctl@ceph.osd.9.smithi052.stdout:May 27 07:38:56 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-osd-9[67134]: 2024-05-27T07:38:56.713+0000 7f1e3e2ca700 -1 osd.9 134 heartbeat_check: no reply from 172.21.15.52:6806 osd.0 since back 2024-05-27T07:37:58.861291+0000 front 2024-05-27T07:37:58.861258+0000 (oldest deadline 2024-05-27T07:38:22.361571+0000) 2024-05-27T07:38:56.986 INFO:journalctl@ceph.osd.9.smithi052.stdout:May 27 07:38:56 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-osd-9[67134]: 2024-05-27T07:38:56.713+0000 7f1e3e2ca700 -1 osd.9 134 heartbeat_check: no reply from 172.21.15.105:6804 osd.1 since back 2024-05-27T07:38:05.862960+0000 front 2024-05-27T07:38:05.862645+0000 (oldest deadline 2024-05-27T07:38:30.562694+0000) 2024-05-27T07:38:56.987 INFO:journalctl@ceph.osd.9.smithi052.stdout:May 27 07:38:56 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-osd-9[67134]: 2024-05-27T07:38:56.713+0000 7f1e3e2ca700 -1 osd.9 134 heartbeat_check: no reply from 172.21.15.173:6804 osd.2 since back 2024-05-27T07:38:15.566595+0000 front 2024-05-27T07:38:15.566403+0000 (oldest deadline 2024-05-27T07:38:37.866552+0000) 2024-05-27T07:38:56.987 INFO:journalctl@ceph.osd.9.smithi052.stdout:May 27 07:38:56 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-osd-9[67134]: 2024-05-27T07:38:56.713+0000 7f1e3e2ca700 -1 osd.9 134 heartbeat_check: no reply from 172.21.15.52:6814 osd.3 since back 2024-05-27T07:38:22.468720+0000 front 2024-05-27T07:38:22.468751+0000 (oldest deadline 2024-05-27T07:38:48.269806+0000) 2024-05-27T07:38:57.109 INFO:journalctl@ceph.osd.8.smithi173.stdout:May 27 07:38:56 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-osd-8[55969]: 2024-05-27T07:38:56.847+0000 7f425b2ea700 -1 osd.8 134 heartbeat_check: no reply from 172.21.15.105:6804 osd.1 since back 2024-05-27T07:38:06.957078+0000 front 2024-05-27T07:38:06.957280+0000 (oldest deadline 2024-05-27T07:38:30.457072+0000) 2024-05-27T07:38:57.109 INFO:journalctl@ceph.osd.8.smithi173.stdout:May 27 07:38:56 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-osd-8[55969]: 2024-05-27T07:38:56.847+0000 7f425b2ea700 -1 osd.8 134 heartbeat_check: no reply from 172.21.15.173:6804 osd.2 since back 2024-05-27T07:38:16.258968+0000 front 2024-05-27T07:38:16.258998+0000 (oldest deadline 2024-05-27T07:38:41.559145+0000) 2024-05-27T07:38:57.109 INFO:journalctl@ceph.osd.8.smithi173.stdout:May 27 07:38:56 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-osd-8[55969]: 2024-05-27T07:38:56.847+0000 7f425b2ea700 -1 osd.8 134 heartbeat_check: no reply from 172.21.15.52:6814 osd.3 since back 2024-05-27T07:38:23.860441+0000 front 2024-05-27T07:38:23.860421+0000 (oldest deadline 2024-05-27T07:38:49.760323+0000) 2024-05-27T07:38:57.109 INFO:journalctl@ceph.osd.8.smithi173.stdout:May 27 07:38:56 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-osd-8[55969]: 2024-05-27T07:38:56.847+0000 7f425b2ea700 -1 osd.8 134 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-05-27T07:38:32.562246+0000 front 2024-05-27T07:38:32.562420+0000 (oldest deadline 2024-05-27T07:38:53.662280+0000) 2024-05-27T07:38:57.197 INFO:journalctl@ceph.osd.7.smithi105.stdout:May 27 07:38:56 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-osd-7[53596]: 2024-05-27T07:38:56.823+0000 7f5eefdf1700 -1 osd.7 134 heartbeat_check: no reply from 172.21.15.105:6804 osd.1 since back 2024-05-27T07:38:06.798385+0000 front 2024-05-27T07:38:06.798353+0000 (oldest deadline 2024-05-27T07:38:32.098493+0000) 2024-05-27T07:38:57.197 INFO:journalctl@ceph.osd.7.smithi105.stdout:May 27 07:38:56 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-osd-7[53596]: 2024-05-27T07:38:56.823+0000 7f5eefdf1700 -1 osd.7 134 heartbeat_check: no reply from 172.21.15.173:6804 osd.2 since back 2024-05-27T07:38:13.199843+0000 front 2024-05-27T07:38:13.199692+0000 (oldest deadline 2024-05-27T07:38:37.899787+0000) 2024-05-27T07:38:57.197 INFO:journalctl@ceph.osd.7.smithi105.stdout:May 27 07:38:56 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-osd-7[53596]: 2024-05-27T07:38:56.823+0000 7f5eefdf1700 -1 osd.7 134 heartbeat_check: no reply from 172.21.15.52:6814 osd.3 since back 2024-05-27T07:38:23.002094+0000 front 2024-05-27T07:38:23.002188+0000 (oldest deadline 2024-05-27T07:38:47.702083+0000) 2024-05-27T07:38:57.609 INFO:journalctl@ceph.osd.11.smithi173.stdout:May 27 07:38:57 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-osd-11[63021]: 2024-05-27T07:38:57.333+0000 7f9d44ac8700 -1 osd.11 134 heartbeat_check: no reply from 172.21.15.52:6806 osd.0 since back 2024-05-27T07:37:57.927796+0000 front 2024-05-27T07:37:57.927759+0000 (oldest deadline 2024-05-27T07:38:22.028025+0000) 2024-05-27T07:38:57.609 INFO:journalctl@ceph.osd.11.smithi173.stdout:May 27 07:38:57 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-osd-11[63021]: 2024-05-27T07:38:57.333+0000 7f9d44ac8700 -1 osd.11 134 heartbeat_check: no reply from 172.21.15.105:6804 osd.1 since back 2024-05-27T07:38:03.129257+0000 front 2024-05-27T07:38:03.129120+0000 (oldest deadline 2024-05-27T07:38:28.929543+0000) 2024-05-27T07:38:57.610 INFO:journalctl@ceph.osd.11.smithi173.stdout:May 27 07:38:57 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-osd-11[63021]: 2024-05-27T07:38:57.333+0000 7f9d44ac8700 -1 osd.11 134 heartbeat_check: no reply from 172.21.15.173:6804 osd.2 since back 2024-05-27T07:38:16.431856+0000 front 2024-05-27T07:38:16.431878+0000 (oldest deadline 2024-05-27T07:38:42.332069+0000) 2024-05-27T07:38:57.610 INFO:journalctl@ceph.osd.11.smithi173.stdout:May 27 07:38:57 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-osd-11[63021]: 2024-05-27T07:38:57.333+0000 7f9d44ac8700 -1 osd.11 134 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-05-27T07:38:28.133691+0000 front 2024-05-27T07:38:28.134117+0000 (oldest deadline 2024-05-27T07:38:55.734329+0000) 2024-05-27T07:38:57.697 INFO:journalctl@ceph.osd.10.smithi105.stdout:May 27 07:38:57 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-osd-10[60334]: 2024-05-27T07:38:57.394+0000 7f162fb8f700 -1 osd.10 134 heartbeat_check: no reply from 172.21.15.52:6806 osd.0 since back 2024-05-27T07:37:56.754985+0000 front 2024-05-27T07:37:56.755070+0000 (oldest deadline 2024-05-27T07:38:22.054911+0000) 2024-05-27T07:38:57.697 INFO:journalctl@ceph.osd.10.smithi105.stdout:May 27 07:38:57 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-osd-10[60334]: 2024-05-27T07:38:57.394+0000 7f162fb8f700 -1 osd.10 134 heartbeat_check: no reply from 172.21.15.105:6804 osd.1 since back 2024-05-27T07:38:05.556137+0000 front 2024-05-27T07:38:05.556112+0000 (oldest deadline 2024-05-27T07:38:31.456213+0000) 2024-05-27T07:38:57.697 INFO:journalctl@ceph.osd.10.smithi105.stdout:May 27 07:38:57 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-osd-10[60334]: 2024-05-27T07:38:57.394+0000 7f162fb8f700 -1 osd.10 134 heartbeat_check: no reply from 172.21.15.173:6804 osd.2 since back 2024-05-27T07:38:15.557554+0000 front 2024-05-27T07:38:15.557409+0000 (oldest deadline 2024-05-27T07:38:40.857509+0000) 2024-05-27T07:38:57.697 INFO:journalctl@ceph.osd.10.smithi105.stdout:May 27 07:38:57 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-osd-10[60334]: 2024-05-27T07:38:57.394+0000 7f162fb8f700 -1 osd.10 134 heartbeat_check: no reply from 172.21.15.52:6814 osd.3 since back 2024-05-27T07:38:21.958641+0000 front 2024-05-27T07:38:21.958614+0000 (oldest deadline 2024-05-27T07:38:47.858600+0000) 2024-05-27T07:38:57.697 INFO:journalctl@ceph.osd.10.smithi105.stdout:May 27 07:38:57 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-osd-10[60334]: 2024-05-27T07:38:57.394+0000 7f162fb8f700 -1 osd.10 134 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-05-27T07:38:30.560765+0000 front 2024-05-27T07:38:30.560788+0000 (oldest deadline 2024-05-27T07:38:55.260887+0000) 2024-05-27T07:38:58.109 INFO:journalctl@ceph.osd.8.smithi173.stdout:May 27 07:38:57 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-osd-8[55969]: 2024-05-27T07:38:57.811+0000 7f425b2ea700 -1 osd.8 134 heartbeat_check: no reply from 172.21.15.105:6804 osd.1 since back 2024-05-27T07:38:06.957078+0000 front 2024-05-27T07:38:06.957280+0000 (oldest deadline 2024-05-27T07:38:30.457072+0000) 2024-05-27T07:38:58.109 INFO:journalctl@ceph.osd.8.smithi173.stdout:May 27 07:38:57 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-osd-8[55969]: 2024-05-27T07:38:57.811+0000 7f425b2ea700 -1 osd.8 134 heartbeat_check: no reply from 172.21.15.173:6804 osd.2 since back 2024-05-27T07:38:16.258968+0000 front 2024-05-27T07:38:16.258998+0000 (oldest deadline 2024-05-27T07:38:41.559145+0000) 2024-05-27T07:38:58.109 INFO:journalctl@ceph.osd.8.smithi173.stdout:May 27 07:38:57 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-osd-8[55969]: 2024-05-27T07:38:57.811+0000 7f425b2ea700 -1 osd.8 134 heartbeat_check: no reply from 172.21.15.52:6814 osd.3 since back 2024-05-27T07:38:23.860441+0000 front 2024-05-27T07:38:23.860421+0000 (oldest deadline 2024-05-27T07:38:49.760323+0000) 2024-05-27T07:38:58.109 INFO:journalctl@ceph.osd.8.smithi173.stdout:May 27 07:38:57 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-osd-8[55969]: 2024-05-27T07:38:57.811+0000 7f425b2ea700 -1 osd.8 134 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-05-27T07:38:32.562246+0000 front 2024-05-27T07:38:32.562420+0000 (oldest deadline 2024-05-27T07:38:53.662280+0000) 2024-05-27T07:38:58.236 INFO:journalctl@ceph.osd.9.smithi052.stdout:May 27 07:38:57 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-osd-9[67134]: 2024-05-27T07:38:57.747+0000 7f1e3e2ca700 -1 osd.9 134 heartbeat_check: no reply from 172.21.15.52:6806 osd.0 since back 2024-05-27T07:37:58.861291+0000 front 2024-05-27T07:37:58.861258+0000 (oldest deadline 2024-05-27T07:38:22.361571+0000) 2024-05-27T07:38:58.237 INFO:journalctl@ceph.osd.9.smithi052.stdout:May 27 07:38:57 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-osd-9[67134]: 2024-05-27T07:38:57.747+0000 7f1e3e2ca700 -1 osd.9 134 heartbeat_check: no reply from 172.21.15.105:6804 osd.1 since back 2024-05-27T07:38:05.862960+0000 front 2024-05-27T07:38:05.862645+0000 (oldest deadline 2024-05-27T07:38:30.562694+0000) 2024-05-27T07:38:58.237 INFO:journalctl@ceph.osd.9.smithi052.stdout:May 27 07:38:57 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-osd-9[67134]: 2024-05-27T07:38:57.747+0000 7f1e3e2ca700 -1 osd.9 134 heartbeat_check: no reply from 172.21.15.173:6804 osd.2 since back 2024-05-27T07:38:15.566595+0000 front 2024-05-27T07:38:15.566403+0000 (oldest deadline 2024-05-27T07:38:37.866552+0000) 2024-05-27T07:38:58.237 INFO:journalctl@ceph.osd.9.smithi052.stdout:May 27 07:38:57 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-osd-9[67134]: 2024-05-27T07:38:57.747+0000 7f1e3e2ca700 -1 osd.9 134 heartbeat_check: no reply from 172.21.15.52:6814 osd.3 since back 2024-05-27T07:38:22.468720+0000 front 2024-05-27T07:38:22.468751+0000 (oldest deadline 2024-05-27T07:38:48.269806+0000) 2024-05-27T07:38:58.237 INFO:journalctl@ceph.osd.9.smithi052.stdout:May 27 07:38:57 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-osd-9[67134]: 2024-05-27T07:38:57.747+0000 7f1e3e2ca700 -1 osd.9 134 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-05-27T07:38:31.171291+0000 front 2024-05-27T07:38:31.171372+0000 (oldest deadline 2024-05-27T07:38:56.971580+0000) 2024-05-27T07:38:58.367 INFO:journalctl@ceph.osd.7.smithi105.stdout:May 27 07:38:58 smithi105 podman[72522]: 2024-05-27 07:38:58.09947972 +0000 UTC m=+5.731303303 container died ae6d7314f1dc2397a8039676c5fd7b431957f17653cd77532ee2037894a6fbcd (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-osd-7, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, CEPH_POINT_RELEASE=, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, org.label-schema.build-date=20240521, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True) 2024-05-27T07:38:58.609 INFO:journalctl@ceph.osd.11.smithi173.stdout:May 27 07:38:58 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-osd-11[63021]: 2024-05-27T07:38:58.306+0000 7f9d44ac8700 -1 osd.11 134 heartbeat_check: no reply from 172.21.15.52:6806 osd.0 since back 2024-05-27T07:37:57.927796+0000 front 2024-05-27T07:37:57.927759+0000 (oldest deadline 2024-05-27T07:38:22.028025+0000) 2024-05-27T07:38:58.609 INFO:journalctl@ceph.osd.11.smithi173.stdout:May 27 07:38:58 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-osd-11[63021]: 2024-05-27T07:38:58.306+0000 7f9d44ac8700 -1 osd.11 134 heartbeat_check: no reply from 172.21.15.105:6804 osd.1 since back 2024-05-27T07:38:03.129257+0000 front 2024-05-27T07:38:03.129120+0000 (oldest deadline 2024-05-27T07:38:28.929543+0000) 2024-05-27T07:38:58.609 INFO:journalctl@ceph.osd.11.smithi173.stdout:May 27 07:38:58 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-osd-11[63021]: 2024-05-27T07:38:58.306+0000 7f9d44ac8700 -1 osd.11 134 heartbeat_check: no reply from 172.21.15.173:6804 osd.2 since back 2024-05-27T07:38:16.431856+0000 front 2024-05-27T07:38:16.431878+0000 (oldest deadline 2024-05-27T07:38:42.332069+0000) 2024-05-27T07:38:58.609 INFO:journalctl@ceph.osd.11.smithi173.stdout:May 27 07:38:58 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-osd-11[63021]: 2024-05-27T07:38:58.306+0000 7f9d44ac8700 -1 osd.11 134 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-05-27T07:38:28.133691+0000 front 2024-05-27T07:38:28.134117+0000 (oldest deadline 2024-05-27T07:38:55.734329+0000) 2024-05-27T07:38:58.697 INFO:journalctl@ceph.osd.7.smithi105.stdout:May 27 07:38:58 smithi105 podman[72522]: 2024-05-27 07:38:58.489368471 +0000 UTC m=+6.121192096 container cleanup ae6d7314f1dc2397a8039676c5fd7b431957f17653cd77532ee2037894a6fbcd (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-osd-7, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240521, GIT_BRANCH=HEAD, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, ceph=True) 2024-05-27T07:38:58.697 INFO:journalctl@ceph.osd.7.smithi105.stdout:May 27 07:38:58 smithi105 bash[72522]: ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-osd-7 2024-05-27T07:38:58.698 INFO:journalctl@ceph.osd.10.smithi105.stdout:May 27 07:38:58 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-osd-10[60334]: 2024-05-27T07:38:58.370+0000 7f162fb8f700 -1 osd.10 134 heartbeat_check: no reply from 172.21.15.52:6806 osd.0 since back 2024-05-27T07:37:56.754985+0000 front 2024-05-27T07:37:56.755070+0000 (oldest deadline 2024-05-27T07:38:22.054911+0000) 2024-05-27T07:38:58.698 INFO:journalctl@ceph.osd.10.smithi105.stdout:May 27 07:38:58 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-osd-10[60334]: 2024-05-27T07:38:58.370+0000 7f162fb8f700 -1 osd.10 134 heartbeat_check: no reply from 172.21.15.105:6804 osd.1 since back 2024-05-27T07:38:05.556137+0000 front 2024-05-27T07:38:05.556112+0000 (oldest deadline 2024-05-27T07:38:31.456213+0000) 2024-05-27T07:38:58.698 INFO:journalctl@ceph.osd.10.smithi105.stdout:May 27 07:38:58 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-osd-10[60334]: 2024-05-27T07:38:58.370+0000 7f162fb8f700 -1 osd.10 134 heartbeat_check: no reply from 172.21.15.173:6804 osd.2 since back 2024-05-27T07:38:15.557554+0000 front 2024-05-27T07:38:15.557409+0000 (oldest deadline 2024-05-27T07:38:40.857509+0000) 2024-05-27T07:38:58.698 INFO:journalctl@ceph.osd.10.smithi105.stdout:May 27 07:38:58 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-osd-10[60334]: 2024-05-27T07:38:58.370+0000 7f162fb8f700 -1 osd.10 134 heartbeat_check: no reply from 172.21.15.52:6814 osd.3 since back 2024-05-27T07:38:21.958641+0000 front 2024-05-27T07:38:21.958614+0000 (oldest deadline 2024-05-27T07:38:47.858600+0000) 2024-05-27T07:38:58.698 INFO:journalctl@ceph.osd.10.smithi105.stdout:May 27 07:38:58 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-osd-10[60334]: 2024-05-27T07:38:58.370+0000 7f162fb8f700 -1 osd.10 134 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-05-27T07:38:30.560765+0000 front 2024-05-27T07:38:30.560788+0000 (oldest deadline 2024-05-27T07:38:55.260887+0000) 2024-05-27T07:38:59.075 INFO:journalctl@ceph.osd.8.smithi173.stdout:May 27 07:38:58 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-osd-8[55969]: 2024-05-27T07:38:58.790+0000 7f425b2ea700 -1 osd.8 134 heartbeat_check: no reply from 172.21.15.105:6804 osd.1 since back 2024-05-27T07:38:06.957078+0000 front 2024-05-27T07:38:06.957280+0000 (oldest deadline 2024-05-27T07:38:30.457072+0000) 2024-05-27T07:38:59.075 INFO:journalctl@ceph.osd.8.smithi173.stdout:May 27 07:38:58 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-osd-8[55969]: 2024-05-27T07:38:58.790+0000 7f425b2ea700 -1 osd.8 134 heartbeat_check: no reply from 172.21.15.173:6804 osd.2 since back 2024-05-27T07:38:16.258968+0000 front 2024-05-27T07:38:16.258998+0000 (oldest deadline 2024-05-27T07:38:41.559145+0000) 2024-05-27T07:38:59.076 INFO:journalctl@ceph.osd.8.smithi173.stdout:May 27 07:38:58 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-osd-8[55969]: 2024-05-27T07:38:58.790+0000 7f425b2ea700 -1 osd.8 134 heartbeat_check: no reply from 172.21.15.52:6814 osd.3 since back 2024-05-27T07:38:23.860441+0000 front 2024-05-27T07:38:23.860421+0000 (oldest deadline 2024-05-27T07:38:49.760323+0000) 2024-05-27T07:38:59.076 INFO:journalctl@ceph.osd.8.smithi173.stdout:May 27 07:38:58 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-osd-8[55969]: 2024-05-27T07:38:58.790+0000 7f425b2ea700 -1 osd.8 134 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-05-27T07:38:32.562246+0000 front 2024-05-27T07:38:32.562420+0000 (oldest deadline 2024-05-27T07:38:53.662280+0000) 2024-05-27T07:38:59.087 INFO:journalctl@ceph.osd.7.smithi105.stdout:May 27 07:38:58 smithi105 podman[72540]: 2024-05-27 07:38:58.782589307 +0000 UTC m=+0.679046466 container remove ae6d7314f1dc2397a8039676c5fd7b431957f17653cd77532ee2037894a6fbcd (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-osd-7, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , ceph=True, GIT_CLEAN=True, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240521, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0) 2024-05-27T07:38:59.138 INFO:journalctl@ceph.osd.9.smithi052.stdout:May 27 07:38:58 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-osd-9[67134]: 2024-05-27T07:38:58.793+0000 7f1e3e2ca700 -1 osd.9 134 heartbeat_check: no reply from 172.21.15.52:6806 osd.0 since back 2024-05-27T07:37:58.861291+0000 front 2024-05-27T07:37:58.861258+0000 (oldest deadline 2024-05-27T07:38:22.361571+0000) 2024-05-27T07:38:59.138 INFO:journalctl@ceph.osd.9.smithi052.stdout:May 27 07:38:58 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-osd-9[67134]: 2024-05-27T07:38:58.793+0000 7f1e3e2ca700 -1 osd.9 134 heartbeat_check: no reply from 172.21.15.105:6804 osd.1 since back 2024-05-27T07:38:05.862960+0000 front 2024-05-27T07:38:05.862645+0000 (oldest deadline 2024-05-27T07:38:30.562694+0000) 2024-05-27T07:38:59.138 INFO:journalctl@ceph.osd.9.smithi052.stdout:May 27 07:38:58 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-osd-9[67134]: 2024-05-27T07:38:58.793+0000 7f1e3e2ca700 -1 osd.9 134 heartbeat_check: no reply from 172.21.15.173:6804 osd.2 since back 2024-05-27T07:38:15.566595+0000 front 2024-05-27T07:38:15.566403+0000 (oldest deadline 2024-05-27T07:38:37.866552+0000) 2024-05-27T07:38:59.138 INFO:journalctl@ceph.osd.9.smithi052.stdout:May 27 07:38:58 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-osd-9[67134]: 2024-05-27T07:38:58.793+0000 7f1e3e2ca700 -1 osd.9 134 heartbeat_check: no reply from 172.21.15.52:6814 osd.3 since back 2024-05-27T07:38:22.468720+0000 front 2024-05-27T07:38:22.468751+0000 (oldest deadline 2024-05-27T07:38:48.269806+0000) 2024-05-27T07:38:59.138 INFO:journalctl@ceph.osd.9.smithi052.stdout:May 27 07:38:58 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-osd-9[67134]: 2024-05-27T07:38:58.793+0000 7f1e3e2ca700 -1 osd.9 134 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-05-27T07:38:31.171291+0000 front 2024-05-27T07:38:31.171372+0000 (oldest deadline 2024-05-27T07:38:56.971580+0000) 2024-05-27T07:38:59.359 INFO:journalctl@ceph.osd.11.smithi173.stdout:May 27 07:38:59 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-osd-11[63021]: 2024-05-27T07:38:59.268+0000 7f9d44ac8700 -1 osd.11 134 heartbeat_check: no reply from 172.21.15.52:6806 osd.0 since back 2024-05-27T07:37:57.927796+0000 front 2024-05-27T07:37:57.927759+0000 (oldest deadline 2024-05-27T07:38:22.028025+0000) 2024-05-27T07:38:59.359 INFO:journalctl@ceph.osd.11.smithi173.stdout:May 27 07:38:59 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-osd-11[63021]: 2024-05-27T07:38:59.268+0000 7f9d44ac8700 -1 osd.11 134 heartbeat_check: no reply from 172.21.15.105:6804 osd.1 since back 2024-05-27T07:38:03.129257+0000 front 2024-05-27T07:38:03.129120+0000 (oldest deadline 2024-05-27T07:38:28.929543+0000) 2024-05-27T07:38:59.359 INFO:journalctl@ceph.osd.11.smithi173.stdout:May 27 07:38:59 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-osd-11[63021]: 2024-05-27T07:38:59.268+0000 7f9d44ac8700 -1 osd.11 134 heartbeat_check: no reply from 172.21.15.173:6804 osd.2 since back 2024-05-27T07:38:16.431856+0000 front 2024-05-27T07:38:16.431878+0000 (oldest deadline 2024-05-27T07:38:42.332069+0000) 2024-05-27T07:38:59.359 INFO:journalctl@ceph.osd.11.smithi173.stdout:May 27 07:38:59 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-osd-11[63021]: 2024-05-27T07:38:59.268+0000 7f9d44ac8700 -1 osd.11 134 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-05-27T07:38:28.133691+0000 front 2024-05-27T07:38:28.134117+0000 (oldest deadline 2024-05-27T07:38:55.734329+0000) 2024-05-27T07:38:59.361 INFO:journalctl@ceph.osd.7.smithi105.stdout:May 27 07:38:59 smithi105 podman[72629]: 2024-05-27 07:38:59.091606786 +0000 UTC m=+0.104836837 container create a54dd05b062e3ef98ff73beafc3b180f0ea8b3b6ab7615d49d44c80af305c31b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-osd-7-deactivate, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, GIT_CLEAN=True, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, org.label-schema.build-date=20240521, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-05-27T07:38:59.362 INFO:journalctl@ceph.osd.7.smithi105.stdout:May 27 07:38:59 smithi105 podman[72629]: 2024-05-27 07:38:59.003667457 +0000 UTC m=+0.016897525 image pull e36ea4fc8a5807e8d2701370e350a0de3856538797f7fba64cfce7e344bd6799 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6 2024-05-27T07:38:59.362 INFO:journalctl@ceph.osd.7.smithi105.stdout:May 27 07:38:59 smithi105 podman[72629]: 2024-05-27 07:38:59.302653933 +0000 UTC m=+0.315883985 container init a54dd05b062e3ef98ff73beafc3b180f0ea8b3b6ab7615d49d44c80af305c31b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-osd-7-deactivate, io.buildah.version=1.33.5, GIT_CLEAN=True, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240521, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD) 2024-05-27T07:38:59.362 INFO:journalctl@ceph.osd.7.smithi105.stdout:May 27 07:38:59 smithi105 podman[72629]: 2024-05-27 07:38:59.305611899 +0000 UTC m=+0.318841966 container start a54dd05b062e3ef98ff73beafc3b180f0ea8b3b6ab7615d49d44c80af305c31b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-osd-7-deactivate, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, org.label-schema.build-date=20240521, ceph=True, GIT_CLEAN=True, org.label-schema.vendor=CentOS, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux ) 2024-05-27T07:38:59.697 INFO:journalctl@ceph.osd.7.smithi105.stdout:May 27 07:38:59 smithi105 podman[72629]: 2024-05-27 07:38:59.419345345 +0000 UTC m=+0.432575406 container attach a54dd05b062e3ef98ff73beafc3b180f0ea8b3b6ab7615d49d44c80af305c31b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-osd-7-deactivate, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240521, ceph=True, maintainer=Guillaume Abrioux , GIT_CLEAN=True, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, org.label-schema.vendor=CentOS, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2) 2024-05-27T07:38:59.698 INFO:journalctl@ceph.osd.7.smithi105.stdout:May 27 07:38:59 smithi105 podman[72629]: 2024-05-27 07:38:59.606886061 +0000 UTC m=+0.620116119 container died a54dd05b062e3ef98ff73beafc3b180f0ea8b3b6ab7615d49d44c80af305c31b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-osd-7-deactivate, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.license=GPLv2, org.label-schema.build-date=20240521, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, GIT_CLEAN=True, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image) 2024-05-27T07:38:59.698 INFO:journalctl@ceph.osd.10.smithi105.stdout:May 27 07:38:59 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-osd-10[60334]: 2024-05-27T07:38:59.363+0000 7f162fb8f700 -1 osd.10 134 heartbeat_check: no reply from 172.21.15.52:6806 osd.0 since back 2024-05-27T07:37:56.754985+0000 front 2024-05-27T07:37:56.755070+0000 (oldest deadline 2024-05-27T07:38:22.054911+0000) 2024-05-27T07:38:59.698 INFO:journalctl@ceph.osd.10.smithi105.stdout:May 27 07:38:59 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-osd-10[60334]: 2024-05-27T07:38:59.363+0000 7f162fb8f700 -1 osd.10 134 heartbeat_check: no reply from 172.21.15.105:6804 osd.1 since back 2024-05-27T07:38:05.556137+0000 front 2024-05-27T07:38:05.556112+0000 (oldest deadline 2024-05-27T07:38:31.456213+0000) 2024-05-27T07:38:59.698 INFO:journalctl@ceph.osd.10.smithi105.stdout:May 27 07:38:59 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-osd-10[60334]: 2024-05-27T07:38:59.363+0000 7f162fb8f700 -1 osd.10 134 heartbeat_check: no reply from 172.21.15.173:6804 osd.2 since back 2024-05-27T07:38:15.557554+0000 front 2024-05-27T07:38:15.557409+0000 (oldest deadline 2024-05-27T07:38:40.857509+0000) 2024-05-27T07:38:59.698 INFO:journalctl@ceph.osd.10.smithi105.stdout:May 27 07:38:59 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-osd-10[60334]: 2024-05-27T07:38:59.363+0000 7f162fb8f700 -1 osd.10 134 heartbeat_check: no reply from 172.21.15.52:6814 osd.3 since back 2024-05-27T07:38:21.958641+0000 front 2024-05-27T07:38:21.958614+0000 (oldest deadline 2024-05-27T07:38:47.858600+0000) 2024-05-27T07:38:59.698 INFO:journalctl@ceph.osd.10.smithi105.stdout:May 27 07:38:59 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-osd-10[60334]: 2024-05-27T07:38:59.363+0000 7f162fb8f700 -1 osd.10 134 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-05-27T07:38:30.560765+0000 front 2024-05-27T07:38:30.560788+0000 (oldest deadline 2024-05-27T07:38:55.260887+0000) 2024-05-27T07:39:00.109 INFO:journalctl@ceph.osd.8.smithi173.stdout:May 27 07:38:59 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-osd-8[55969]: 2024-05-27T07:38:59.754+0000 7f425b2ea700 -1 osd.8 134 heartbeat_check: no reply from 172.21.15.105:6804 osd.1 since back 2024-05-27T07:38:06.957078+0000 front 2024-05-27T07:38:06.957280+0000 (oldest deadline 2024-05-27T07:38:30.457072+0000) 2024-05-27T07:39:00.109 INFO:journalctl@ceph.osd.8.smithi173.stdout:May 27 07:38:59 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-osd-8[55969]: 2024-05-27T07:38:59.754+0000 7f425b2ea700 -1 osd.8 134 heartbeat_check: no reply from 172.21.15.173:6804 osd.2 since back 2024-05-27T07:38:16.258968+0000 front 2024-05-27T07:38:16.258998+0000 (oldest deadline 2024-05-27T07:38:41.559145+0000) 2024-05-27T07:39:00.109 INFO:journalctl@ceph.osd.8.smithi173.stdout:May 27 07:38:59 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-osd-8[55969]: 2024-05-27T07:38:59.754+0000 7f425b2ea700 -1 osd.8 134 heartbeat_check: no reply from 172.21.15.52:6814 osd.3 since back 2024-05-27T07:38:23.860441+0000 front 2024-05-27T07:38:23.860421+0000 (oldest deadline 2024-05-27T07:38:49.760323+0000) 2024-05-27T07:39:00.109 INFO:journalctl@ceph.osd.8.smithi173.stdout:May 27 07:38:59 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-osd-8[55969]: 2024-05-27T07:38:59.754+0000 7f425b2ea700 -1 osd.8 134 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-05-27T07:38:32.562246+0000 front 2024-05-27T07:38:32.562420+0000 (oldest deadline 2024-05-27T07:38:53.662280+0000) 2024-05-27T07:39:00.236 INFO:journalctl@ceph.osd.9.smithi052.stdout:May 27 07:38:59 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-osd-9[67134]: 2024-05-27T07:38:59.806+0000 7f1e3e2ca700 -1 osd.9 134 heartbeat_check: no reply from 172.21.15.52:6806 osd.0 since back 2024-05-27T07:37:58.861291+0000 front 2024-05-27T07:37:58.861258+0000 (oldest deadline 2024-05-27T07:38:22.361571+0000) 2024-05-27T07:39:00.236 INFO:journalctl@ceph.osd.9.smithi052.stdout:May 27 07:38:59 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-osd-9[67134]: 2024-05-27T07:38:59.806+0000 7f1e3e2ca700 -1 osd.9 134 heartbeat_check: no reply from 172.21.15.105:6804 osd.1 since back 2024-05-27T07:38:05.862960+0000 front 2024-05-27T07:38:05.862645+0000 (oldest deadline 2024-05-27T07:38:30.562694+0000) 2024-05-27T07:39:00.237 INFO:journalctl@ceph.osd.9.smithi052.stdout:May 27 07:38:59 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-osd-9[67134]: 2024-05-27T07:38:59.806+0000 7f1e3e2ca700 -1 osd.9 134 heartbeat_check: no reply from 172.21.15.173:6804 osd.2 since back 2024-05-27T07:38:15.566595+0000 front 2024-05-27T07:38:15.566403+0000 (oldest deadline 2024-05-27T07:38:37.866552+0000) 2024-05-27T07:39:00.237 INFO:journalctl@ceph.osd.9.smithi052.stdout:May 27 07:38:59 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-osd-9[67134]: 2024-05-27T07:38:59.806+0000 7f1e3e2ca700 -1 osd.9 134 heartbeat_check: no reply from 172.21.15.52:6814 osd.3 since back 2024-05-27T07:38:22.468720+0000 front 2024-05-27T07:38:22.468751+0000 (oldest deadline 2024-05-27T07:38:48.269806+0000) 2024-05-27T07:39:00.237 INFO:journalctl@ceph.osd.9.smithi052.stdout:May 27 07:38:59 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-osd-9[67134]: 2024-05-27T07:38:59.806+0000 7f1e3e2ca700 -1 osd.9 134 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-05-27T07:38:31.171291+0000 front 2024-05-27T07:38:31.171372+0000 (oldest deadline 2024-05-27T07:38:56.971580+0000) 2024-05-27T07:39:00.279 DEBUG:teuthology.orchestra.run.smithi105:> sudo pkill -f 'journalctl -f -n 0 -u ceph-73df2636-1bf8-11ef-bc9b-c7b262605968@osd.7.service' 2024-05-27T07:39:00.357 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-05-27T07:39:00.357 INFO:tasks.cephadm.osd.7:Stopped osd.7 2024-05-27T07:39:00.357 INFO:tasks.cephadm.osd.8:Stopping osd.8... 2024-05-27T07:39:00.357 DEBUG:teuthology.orchestra.run.smithi173:> sudo systemctl stop ceph-73df2636-1bf8-11ef-bc9b-c7b262605968@osd.8 2024-05-27T07:39:00.609 INFO:journalctl@ceph.osd.8.smithi173.stdout:May 27 07:39:00 smithi173 systemd[1]: Stopping Ceph osd.8 for 73df2636-1bf8-11ef-bc9b-c7b262605968... 2024-05-27T07:39:00.609 INFO:journalctl@ceph.osd.11.smithi173.stdout:May 27 07:39:00 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-osd-11[63021]: 2024-05-27T07:39:00.283+0000 7f9d44ac8700 -1 osd.11 134 heartbeat_check: no reply from 172.21.15.52:6806 osd.0 since back 2024-05-27T07:37:57.927796+0000 front 2024-05-27T07:37:57.927759+0000 (oldest deadline 2024-05-27T07:38:22.028025+0000) 2024-05-27T07:39:00.610 INFO:journalctl@ceph.osd.11.smithi173.stdout:May 27 07:39:00 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-osd-11[63021]: 2024-05-27T07:39:00.283+0000 7f9d44ac8700 -1 osd.11 134 heartbeat_check: no reply from 172.21.15.105:6804 osd.1 since back 2024-05-27T07:38:03.129257+0000 front 2024-05-27T07:38:03.129120+0000 (oldest deadline 2024-05-27T07:38:28.929543+0000) 2024-05-27T07:39:00.610 INFO:journalctl@ceph.osd.11.smithi173.stdout:May 27 07:39:00 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-osd-11[63021]: 2024-05-27T07:39:00.283+0000 7f9d44ac8700 -1 osd.11 134 heartbeat_check: no reply from 172.21.15.173:6804 osd.2 since back 2024-05-27T07:38:16.431856+0000 front 2024-05-27T07:38:16.431878+0000 (oldest deadline 2024-05-27T07:38:42.332069+0000) 2024-05-27T07:39:00.610 INFO:journalctl@ceph.osd.11.smithi173.stdout:May 27 07:39:00 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-osd-11[63021]: 2024-05-27T07:39:00.283+0000 7f9d44ac8700 -1 osd.11 134 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-05-27T07:38:28.133691+0000 front 2024-05-27T07:38:28.134117+0000 (oldest deadline 2024-05-27T07:38:55.734329+0000) 2024-05-27T07:39:00.697 INFO:journalctl@ceph.osd.10.smithi105.stdout:May 27 07:39:00 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-osd-10[60334]: 2024-05-27T07:39:00.400+0000 7f162fb8f700 -1 osd.10 134 heartbeat_check: no reply from 172.21.15.52:6806 osd.0 since back 2024-05-27T07:37:56.754985+0000 front 2024-05-27T07:37:56.755070+0000 (oldest deadline 2024-05-27T07:38:22.054911+0000) 2024-05-27T07:39:00.697 INFO:journalctl@ceph.osd.10.smithi105.stdout:May 27 07:39:00 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-osd-10[60334]: 2024-05-27T07:39:00.400+0000 7f162fb8f700 -1 osd.10 134 heartbeat_check: no reply from 172.21.15.105:6804 osd.1 since back 2024-05-27T07:38:05.556137+0000 front 2024-05-27T07:38:05.556112+0000 (oldest deadline 2024-05-27T07:38:31.456213+0000) 2024-05-27T07:39:00.697 INFO:journalctl@ceph.osd.10.smithi105.stdout:May 27 07:39:00 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-osd-10[60334]: 2024-05-27T07:39:00.400+0000 7f162fb8f700 -1 osd.10 134 heartbeat_check: no reply from 172.21.15.173:6804 osd.2 since back 2024-05-27T07:38:15.557554+0000 front 2024-05-27T07:38:15.557409+0000 (oldest deadline 2024-05-27T07:38:40.857509+0000) 2024-05-27T07:39:00.697 INFO:journalctl@ceph.osd.10.smithi105.stdout:May 27 07:39:00 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-osd-10[60334]: 2024-05-27T07:39:00.400+0000 7f162fb8f700 -1 osd.10 134 heartbeat_check: no reply from 172.21.15.52:6814 osd.3 since back 2024-05-27T07:38:21.958641+0000 front 2024-05-27T07:38:21.958614+0000 (oldest deadline 2024-05-27T07:38:47.858600+0000) 2024-05-27T07:39:00.697 INFO:journalctl@ceph.osd.10.smithi105.stdout:May 27 07:39:00 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-osd-10[60334]: 2024-05-27T07:39:00.400+0000 7f162fb8f700 -1 osd.10 134 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-05-27T07:38:30.560765+0000 front 2024-05-27T07:38:30.560788+0000 (oldest deadline 2024-05-27T07:38:55.260887+0000) 2024-05-27T07:39:01.109 INFO:journalctl@ceph.osd.8.smithi173.stdout:May 27 07:39:00 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-osd-8[55969]: 2024-05-27T07:39:00.728+0000 7f425ed34700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.8 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-05-27T07:39:01.109 INFO:journalctl@ceph.osd.8.smithi173.stdout:May 27 07:39:00 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-osd-8[55969]: 2024-05-27T07:39:00.728+0000 7f425ed34700 -1 osd.8 134 *** Got signal Terminated *** 2024-05-27T07:39:01.109 INFO:journalctl@ceph.osd.8.smithi173.stdout:May 27 07:39:00 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-osd-8[55969]: 2024-05-27T07:39:00.728+0000 7f425ed34700 -1 osd.8 134 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-05-27T07:39:01.109 INFO:journalctl@ceph.osd.8.smithi173.stdout:May 27 07:39:00 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-osd-8[55969]: 2024-05-27T07:39:00.772+0000 7f425b2ea700 -1 osd.8 134 heartbeat_check: no reply from 172.21.15.105:6804 osd.1 since back 2024-05-27T07:38:06.957078+0000 front 2024-05-27T07:38:06.957280+0000 (oldest deadline 2024-05-27T07:38:30.457072+0000) 2024-05-27T07:39:01.110 INFO:journalctl@ceph.osd.8.smithi173.stdout:May 27 07:39:00 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-osd-8[55969]: 2024-05-27T07:39:00.772+0000 7f425b2ea700 -1 osd.8 134 heartbeat_check: no reply from 172.21.15.173:6804 osd.2 since back 2024-05-27T07:38:16.258968+0000 front 2024-05-27T07:38:16.258998+0000 (oldest deadline 2024-05-27T07:38:41.559145+0000) 2024-05-27T07:39:01.110 INFO:journalctl@ceph.osd.8.smithi173.stdout:May 27 07:39:00 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-osd-8[55969]: 2024-05-27T07:39:00.772+0000 7f425b2ea700 -1 osd.8 134 heartbeat_check: no reply from 172.21.15.52:6814 osd.3 since back 2024-05-27T07:38:23.860441+0000 front 2024-05-27T07:38:23.860421+0000 (oldest deadline 2024-05-27T07:38:49.760323+0000) 2024-05-27T07:39:01.110 INFO:journalctl@ceph.osd.8.smithi173.stdout:May 27 07:39:00 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-osd-8[55969]: 2024-05-27T07:39:00.772+0000 7f425b2ea700 -1 osd.8 134 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-05-27T07:38:32.562246+0000 front 2024-05-27T07:38:32.562420+0000 (oldest deadline 2024-05-27T07:38:53.662280+0000) 2024-05-27T07:39:01.237 INFO:journalctl@ceph.osd.9.smithi052.stdout:May 27 07:39:00 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-osd-9[67134]: 2024-05-27T07:39:00.793+0000 7f1e3e2ca700 -1 osd.9 134 heartbeat_check: no reply from 172.21.15.52:6806 osd.0 since back 2024-05-27T07:37:58.861291+0000 front 2024-05-27T07:37:58.861258+0000 (oldest deadline 2024-05-27T07:38:22.361571+0000) 2024-05-27T07:39:01.237 INFO:journalctl@ceph.osd.9.smithi052.stdout:May 27 07:39:00 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-osd-9[67134]: 2024-05-27T07:39:00.793+0000 7f1e3e2ca700 -1 osd.9 134 heartbeat_check: no reply from 172.21.15.105:6804 osd.1 since back 2024-05-27T07:38:05.862960+0000 front 2024-05-27T07:38:05.862645+0000 (oldest deadline 2024-05-27T07:38:30.562694+0000) 2024-05-27T07:39:01.237 INFO:journalctl@ceph.osd.9.smithi052.stdout:May 27 07:39:00 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-osd-9[67134]: 2024-05-27T07:39:00.793+0000 7f1e3e2ca700 -1 osd.9 134 heartbeat_check: no reply from 172.21.15.173:6804 osd.2 since back 2024-05-27T07:38:15.566595+0000 front 2024-05-27T07:38:15.566403+0000 (oldest deadline 2024-05-27T07:38:37.866552+0000) 2024-05-27T07:39:01.237 INFO:journalctl@ceph.osd.9.smithi052.stdout:May 27 07:39:00 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-osd-9[67134]: 2024-05-27T07:39:00.793+0000 7f1e3e2ca700 -1 osd.9 134 heartbeat_check: no reply from 172.21.15.52:6814 osd.3 since back 2024-05-27T07:38:22.468720+0000 front 2024-05-27T07:38:22.468751+0000 (oldest deadline 2024-05-27T07:38:48.269806+0000) 2024-05-27T07:39:01.237 INFO:journalctl@ceph.osd.9.smithi052.stdout:May 27 07:39:00 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-osd-9[67134]: 2024-05-27T07:39:00.793+0000 7f1e3e2ca700 -1 osd.9 134 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-05-27T07:38:31.171291+0000 front 2024-05-27T07:38:31.171372+0000 (oldest deadline 2024-05-27T07:38:56.971580+0000) 2024-05-27T07:39:01.609 INFO:journalctl@ceph.osd.11.smithi173.stdout:May 27 07:39:01 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-osd-11[63021]: 2024-05-27T07:39:01.324+0000 7f9d44ac8700 -1 osd.11 134 heartbeat_check: no reply from 172.21.15.52:6806 osd.0 since back 2024-05-27T07:37:57.927796+0000 front 2024-05-27T07:37:57.927759+0000 (oldest deadline 2024-05-27T07:38:22.028025+0000) 2024-05-27T07:39:01.609 INFO:journalctl@ceph.osd.11.smithi173.stdout:May 27 07:39:01 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-osd-11[63021]: 2024-05-27T07:39:01.324+0000 7f9d44ac8700 -1 osd.11 134 heartbeat_check: no reply from 172.21.15.105:6804 osd.1 since back 2024-05-27T07:38:03.129257+0000 front 2024-05-27T07:38:03.129120+0000 (oldest deadline 2024-05-27T07:38:28.929543+0000) 2024-05-27T07:39:01.609 INFO:journalctl@ceph.osd.11.smithi173.stdout:May 27 07:39:01 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-osd-11[63021]: 2024-05-27T07:39:01.324+0000 7f9d44ac8700 -1 osd.11 134 heartbeat_check: no reply from 172.21.15.173:6804 osd.2 since back 2024-05-27T07:38:16.431856+0000 front 2024-05-27T07:38:16.431878+0000 (oldest deadline 2024-05-27T07:38:42.332069+0000) 2024-05-27T07:39:01.609 INFO:journalctl@ceph.osd.11.smithi173.stdout:May 27 07:39:01 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-osd-11[63021]: 2024-05-27T07:39:01.324+0000 7f9d44ac8700 -1 osd.11 134 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-05-27T07:38:28.133691+0000 front 2024-05-27T07:38:28.134117+0000 (oldest deadline 2024-05-27T07:38:55.734329+0000) 2024-05-27T07:39:01.697 INFO:journalctl@ceph.osd.10.smithi105.stdout:May 27 07:39:01 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-osd-10[60334]: 2024-05-27T07:39:01.399+0000 7f162fb8f700 -1 osd.10 134 heartbeat_check: no reply from 172.21.15.52:6806 osd.0 since back 2024-05-27T07:37:56.754985+0000 front 2024-05-27T07:37:56.755070+0000 (oldest deadline 2024-05-27T07:38:22.054911+0000) 2024-05-27T07:39:01.697 INFO:journalctl@ceph.osd.10.smithi105.stdout:May 27 07:39:01 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-osd-10[60334]: 2024-05-27T07:39:01.399+0000 7f162fb8f700 -1 osd.10 134 heartbeat_check: no reply from 172.21.15.105:6804 osd.1 since back 2024-05-27T07:38:05.556137+0000 front 2024-05-27T07:38:05.556112+0000 (oldest deadline 2024-05-27T07:38:31.456213+0000) 2024-05-27T07:39:01.697 INFO:journalctl@ceph.osd.10.smithi105.stdout:May 27 07:39:01 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-osd-10[60334]: 2024-05-27T07:39:01.399+0000 7f162fb8f700 -1 osd.10 134 heartbeat_check: no reply from 172.21.15.173:6804 osd.2 since back 2024-05-27T07:38:15.557554+0000 front 2024-05-27T07:38:15.557409+0000 (oldest deadline 2024-05-27T07:38:40.857509+0000) 2024-05-27T07:39:01.697 INFO:journalctl@ceph.osd.10.smithi105.stdout:May 27 07:39:01 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-osd-10[60334]: 2024-05-27T07:39:01.399+0000 7f162fb8f700 -1 osd.10 134 heartbeat_check: no reply from 172.21.15.52:6814 osd.3 since back 2024-05-27T07:38:21.958641+0000 front 2024-05-27T07:38:21.958614+0000 (oldest deadline 2024-05-27T07:38:47.858600+0000) 2024-05-27T07:39:01.697 INFO:journalctl@ceph.osd.10.smithi105.stdout:May 27 07:39:01 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-osd-10[60334]: 2024-05-27T07:39:01.399+0000 7f162fb8f700 -1 osd.10 134 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-05-27T07:38:30.560765+0000 front 2024-05-27T07:38:30.560788+0000 (oldest deadline 2024-05-27T07:38:55.260887+0000) 2024-05-27T07:39:02.109 INFO:journalctl@ceph.osd.8.smithi173.stdout:May 27 07:39:01 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-osd-8[55969]: 2024-05-27T07:39:01.809+0000 7f425b2ea700 -1 osd.8 134 heartbeat_check: no reply from 172.21.15.105:6804 osd.1 since back 2024-05-27T07:38:06.957078+0000 front 2024-05-27T07:38:06.957280+0000 (oldest deadline 2024-05-27T07:38:30.457072+0000) 2024-05-27T07:39:02.109 INFO:journalctl@ceph.osd.8.smithi173.stdout:May 27 07:39:01 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-osd-8[55969]: 2024-05-27T07:39:01.809+0000 7f425b2ea700 -1 osd.8 134 heartbeat_check: no reply from 172.21.15.173:6804 osd.2 since back 2024-05-27T07:38:16.258968+0000 front 2024-05-27T07:38:16.258998+0000 (oldest deadline 2024-05-27T07:38:41.559145+0000) 2024-05-27T07:39:02.109 INFO:journalctl@ceph.osd.8.smithi173.stdout:May 27 07:39:01 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-osd-8[55969]: 2024-05-27T07:39:01.809+0000 7f425b2ea700 -1 osd.8 134 heartbeat_check: no reply from 172.21.15.52:6814 osd.3 since back 2024-05-27T07:38:23.860441+0000 front 2024-05-27T07:38:23.860421+0000 (oldest deadline 2024-05-27T07:38:49.760323+0000) 2024-05-27T07:39:02.109 INFO:journalctl@ceph.osd.8.smithi173.stdout:May 27 07:39:01 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-osd-8[55969]: 2024-05-27T07:39:01.809+0000 7f425b2ea700 -1 osd.8 134 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-05-27T07:38:32.562246+0000 front 2024-05-27T07:38:32.562420+0000 (oldest deadline 2024-05-27T07:38:53.662280+0000) 2024-05-27T07:39:02.236 INFO:journalctl@ceph.osd.9.smithi052.stdout:May 27 07:39:01 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-osd-9[67134]: 2024-05-27T07:39:01.765+0000 7f1e3e2ca700 -1 osd.9 134 heartbeat_check: no reply from 172.21.15.52:6806 osd.0 since back 2024-05-27T07:37:58.861291+0000 front 2024-05-27T07:37:58.861258+0000 (oldest deadline 2024-05-27T07:38:22.361571+0000) 2024-05-27T07:39:02.237 INFO:journalctl@ceph.osd.9.smithi052.stdout:May 27 07:39:01 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-osd-9[67134]: 2024-05-27T07:39:01.765+0000 7f1e3e2ca700 -1 osd.9 134 heartbeat_check: no reply from 172.21.15.105:6804 osd.1 since back 2024-05-27T07:38:05.862960+0000 front 2024-05-27T07:38:05.862645+0000 (oldest deadline 2024-05-27T07:38:30.562694+0000) 2024-05-27T07:39:02.237 INFO:journalctl@ceph.osd.9.smithi052.stdout:May 27 07:39:01 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-osd-9[67134]: 2024-05-27T07:39:01.765+0000 7f1e3e2ca700 -1 osd.9 134 heartbeat_check: no reply from 172.21.15.173:6804 osd.2 since back 2024-05-27T07:38:15.566595+0000 front 2024-05-27T07:38:15.566403+0000 (oldest deadline 2024-05-27T07:38:37.866552+0000) 2024-05-27T07:39:02.237 INFO:journalctl@ceph.osd.9.smithi052.stdout:May 27 07:39:01 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-osd-9[67134]: 2024-05-27T07:39:01.765+0000 7f1e3e2ca700 -1 osd.9 134 heartbeat_check: no reply from 172.21.15.52:6814 osd.3 since back 2024-05-27T07:38:22.468720+0000 front 2024-05-27T07:38:22.468751+0000 (oldest deadline 2024-05-27T07:38:48.269806+0000) 2024-05-27T07:39:02.237 INFO:journalctl@ceph.osd.9.smithi052.stdout:May 27 07:39:01 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-osd-9[67134]: 2024-05-27T07:39:01.765+0000 7f1e3e2ca700 -1 osd.9 134 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-05-27T07:38:31.171291+0000 front 2024-05-27T07:38:31.171372+0000 (oldest deadline 2024-05-27T07:38:56.971580+0000) 2024-05-27T07:39:02.609 INFO:journalctl@ceph.osd.11.smithi173.stdout:May 27 07:39:02 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-osd-11[63021]: 2024-05-27T07:39:02.295+0000 7f9d44ac8700 -1 osd.11 134 heartbeat_check: no reply from 172.21.15.52:6806 osd.0 since back 2024-05-27T07:37:57.927796+0000 front 2024-05-27T07:37:57.927759+0000 (oldest deadline 2024-05-27T07:38:22.028025+0000) 2024-05-27T07:39:02.609 INFO:journalctl@ceph.osd.11.smithi173.stdout:May 27 07:39:02 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-osd-11[63021]: 2024-05-27T07:39:02.295+0000 7f9d44ac8700 -1 osd.11 134 heartbeat_check: no reply from 172.21.15.105:6804 osd.1 since back 2024-05-27T07:38:03.129257+0000 front 2024-05-27T07:38:03.129120+0000 (oldest deadline 2024-05-27T07:38:28.929543+0000) 2024-05-27T07:39:02.609 INFO:journalctl@ceph.osd.11.smithi173.stdout:May 27 07:39:02 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-osd-11[63021]: 2024-05-27T07:39:02.295+0000 7f9d44ac8700 -1 osd.11 134 heartbeat_check: no reply from 172.21.15.173:6804 osd.2 since back 2024-05-27T07:38:16.431856+0000 front 2024-05-27T07:38:16.431878+0000 (oldest deadline 2024-05-27T07:38:42.332069+0000) 2024-05-27T07:39:02.609 INFO:journalctl@ceph.osd.11.smithi173.stdout:May 27 07:39:02 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-osd-11[63021]: 2024-05-27T07:39:02.295+0000 7f9d44ac8700 -1 osd.11 134 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-05-27T07:38:28.133691+0000 front 2024-05-27T07:38:28.134117+0000 (oldest deadline 2024-05-27T07:38:55.734329+0000) 2024-05-27T07:39:02.609 INFO:journalctl@ceph.osd.11.smithi173.stdout:May 27 07:39:02 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-osd-11[63021]: 2024-05-27T07:39:02.295+0000 7f9d44ac8700 -1 osd.11 134 heartbeat_check: no reply from 172.21.15.173:6812 osd.5 since back 2024-05-27T07:38:41.035302+0000 front 2024-05-27T07:38:41.035361+0000 (oldest deadline 2024-05-27T07:39:02.036045+0000) 2024-05-27T07:39:02.697 INFO:journalctl@ceph.osd.10.smithi105.stdout:May 27 07:39:02 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-osd-10[60334]: 2024-05-27T07:39:02.439+0000 7f162fb8f700 -1 osd.10 134 heartbeat_check: no reply from 172.21.15.52:6806 osd.0 since back 2024-05-27T07:37:56.754985+0000 front 2024-05-27T07:37:56.755070+0000 (oldest deadline 2024-05-27T07:38:22.054911+0000) 2024-05-27T07:39:02.697 INFO:journalctl@ceph.osd.10.smithi105.stdout:May 27 07:39:02 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-osd-10[60334]: 2024-05-27T07:39:02.439+0000 7f162fb8f700 -1 osd.10 134 heartbeat_check: no reply from 172.21.15.105:6804 osd.1 since back 2024-05-27T07:38:05.556137+0000 front 2024-05-27T07:38:05.556112+0000 (oldest deadline 2024-05-27T07:38:31.456213+0000) 2024-05-27T07:39:02.697 INFO:journalctl@ceph.osd.10.smithi105.stdout:May 27 07:39:02 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-osd-10[60334]: 2024-05-27T07:39:02.439+0000 7f162fb8f700 -1 osd.10 134 heartbeat_check: no reply from 172.21.15.173:6804 osd.2 since back 2024-05-27T07:38:15.557554+0000 front 2024-05-27T07:38:15.557409+0000 (oldest deadline 2024-05-27T07:38:40.857509+0000) 2024-05-27T07:39:02.698 INFO:journalctl@ceph.osd.10.smithi105.stdout:May 27 07:39:02 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-osd-10[60334]: 2024-05-27T07:39:02.439+0000 7f162fb8f700 -1 osd.10 134 heartbeat_check: no reply from 172.21.15.52:6814 osd.3 since back 2024-05-27T07:38:21.958641+0000 front 2024-05-27T07:38:21.958614+0000 (oldest deadline 2024-05-27T07:38:47.858600+0000) 2024-05-27T07:39:02.698 INFO:journalctl@ceph.osd.10.smithi105.stdout:May 27 07:39:02 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-osd-10[60334]: 2024-05-27T07:39:02.439+0000 7f162fb8f700 -1 osd.10 134 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-05-27T07:38:30.560765+0000 front 2024-05-27T07:38:30.560788+0000 (oldest deadline 2024-05-27T07:38:55.260887+0000) 2024-05-27T07:39:03.109 INFO:journalctl@ceph.osd.8.smithi173.stdout:May 27 07:39:02 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-osd-8[55969]: 2024-05-27T07:39:02.787+0000 7f425b2ea700 -1 osd.8 134 heartbeat_check: no reply from 172.21.15.105:6804 osd.1 since back 2024-05-27T07:38:06.957078+0000 front 2024-05-27T07:38:06.957280+0000 (oldest deadline 2024-05-27T07:38:30.457072+0000) 2024-05-27T07:39:03.110 INFO:journalctl@ceph.osd.8.smithi173.stdout:May 27 07:39:02 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-osd-8[55969]: 2024-05-27T07:39:02.787+0000 7f425b2ea700 -1 osd.8 134 heartbeat_check: no reply from 172.21.15.173:6804 osd.2 since back 2024-05-27T07:38:16.258968+0000 front 2024-05-27T07:38:16.258998+0000 (oldest deadline 2024-05-27T07:38:41.559145+0000) 2024-05-27T07:39:03.110 INFO:journalctl@ceph.osd.8.smithi173.stdout:May 27 07:39:02 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-osd-8[55969]: 2024-05-27T07:39:02.787+0000 7f425b2ea700 -1 osd.8 134 heartbeat_check: no reply from 172.21.15.52:6814 osd.3 since back 2024-05-27T07:38:23.860441+0000 front 2024-05-27T07:38:23.860421+0000 (oldest deadline 2024-05-27T07:38:49.760323+0000) 2024-05-27T07:39:03.110 INFO:journalctl@ceph.osd.8.smithi173.stdout:May 27 07:39:02 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-osd-8[55969]: 2024-05-27T07:39:02.787+0000 7f425b2ea700 -1 osd.8 134 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-05-27T07:38:32.562246+0000 front 2024-05-27T07:38:32.562420+0000 (oldest deadline 2024-05-27T07:38:53.662280+0000) 2024-05-27T07:39:03.110 INFO:journalctl@ceph.osd.8.smithi173.stdout:May 27 07:39:02 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-osd-8[55969]: 2024-05-27T07:39:02.787+0000 7f425b2ea700 -1 osd.8 134 heartbeat_check: no reply from 172.21.15.173:6812 osd.5 since back 2024-05-27T07:38:37.163477+0000 front 2024-05-27T07:38:37.163568+0000 (oldest deadline 2024-05-27T07:39:02.364041+0000) 2024-05-27T07:39:03.238 INFO:journalctl@ceph.osd.9.smithi052.stdout:May 27 07:39:02 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-osd-9[67134]: 2024-05-27T07:39:02.794+0000 7f1e3e2ca700 -1 osd.9 134 heartbeat_check: no reply from 172.21.15.52:6806 osd.0 since back 2024-05-27T07:37:58.861291+0000 front 2024-05-27T07:37:58.861258+0000 (oldest deadline 2024-05-27T07:38:22.361571+0000) 2024-05-27T07:39:03.238 INFO:journalctl@ceph.osd.9.smithi052.stdout:May 27 07:39:02 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-osd-9[67134]: 2024-05-27T07:39:02.794+0000 7f1e3e2ca700 -1 osd.9 134 heartbeat_check: no reply from 172.21.15.105:6804 osd.1 since back 2024-05-27T07:38:05.862960+0000 front 2024-05-27T07:38:05.862645+0000 (oldest deadline 2024-05-27T07:38:30.562694+0000) 2024-05-27T07:39:03.239 INFO:journalctl@ceph.osd.9.smithi052.stdout:May 27 07:39:02 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-osd-9[67134]: 2024-05-27T07:39:02.794+0000 7f1e3e2ca700 -1 osd.9 134 heartbeat_check: no reply from 172.21.15.173:6804 osd.2 since back 2024-05-27T07:38:15.566595+0000 front 2024-05-27T07:38:15.566403+0000 (oldest deadline 2024-05-27T07:38:37.866552+0000) 2024-05-27T07:39:03.239 INFO:journalctl@ceph.osd.9.smithi052.stdout:May 27 07:39:02 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-osd-9[67134]: 2024-05-27T07:39:02.794+0000 7f1e3e2ca700 -1 osd.9 134 heartbeat_check: no reply from 172.21.15.52:6814 osd.3 since back 2024-05-27T07:38:22.468720+0000 front 2024-05-27T07:38:22.468751+0000 (oldest deadline 2024-05-27T07:38:48.269806+0000) 2024-05-27T07:39:03.239 INFO:journalctl@ceph.osd.9.smithi052.stdout:May 27 07:39:02 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-osd-9[67134]: 2024-05-27T07:39:02.794+0000 7f1e3e2ca700 -1 osd.9 134 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-05-27T07:38:31.171291+0000 front 2024-05-27T07:38:31.171372+0000 (oldest deadline 2024-05-27T07:38:56.971580+0000) 2024-05-27T07:39:03.609 INFO:journalctl@ceph.osd.11.smithi173.stdout:May 27 07:39:03 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-osd-11[63021]: 2024-05-27T07:39:03.298+0000 7f9d44ac8700 -1 osd.11 134 heartbeat_check: no reply from 172.21.15.52:6806 osd.0 since back 2024-05-27T07:37:57.927796+0000 front 2024-05-27T07:37:57.927759+0000 (oldest deadline 2024-05-27T07:38:22.028025+0000) 2024-05-27T07:39:03.609 INFO:journalctl@ceph.osd.11.smithi173.stdout:May 27 07:39:03 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-osd-11[63021]: 2024-05-27T07:39:03.298+0000 7f9d44ac8700 -1 osd.11 134 heartbeat_check: no reply from 172.21.15.105:6804 osd.1 since back 2024-05-27T07:38:03.129257+0000 front 2024-05-27T07:38:03.129120+0000 (oldest deadline 2024-05-27T07:38:28.929543+0000) 2024-05-27T07:39:03.609 INFO:journalctl@ceph.osd.11.smithi173.stdout:May 27 07:39:03 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-osd-11[63021]: 2024-05-27T07:39:03.298+0000 7f9d44ac8700 -1 osd.11 134 heartbeat_check: no reply from 172.21.15.173:6804 osd.2 since back 2024-05-27T07:38:16.431856+0000 front 2024-05-27T07:38:16.431878+0000 (oldest deadline 2024-05-27T07:38:42.332069+0000) 2024-05-27T07:39:03.609 INFO:journalctl@ceph.osd.11.smithi173.stdout:May 27 07:39:03 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-osd-11[63021]: 2024-05-27T07:39:03.298+0000 7f9d44ac8700 -1 osd.11 134 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-05-27T07:38:28.133691+0000 front 2024-05-27T07:38:28.134117+0000 (oldest deadline 2024-05-27T07:38:55.734329+0000) 2024-05-27T07:39:03.609 INFO:journalctl@ceph.osd.11.smithi173.stdout:May 27 07:39:03 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-osd-11[63021]: 2024-05-27T07:39:03.298+0000 7f9d44ac8700 -1 osd.11 134 heartbeat_check: no reply from 172.21.15.173:6812 osd.5 since back 2024-05-27T07:38:41.035302+0000 front 2024-05-27T07:38:41.035361+0000 (oldest deadline 2024-05-27T07:39:02.036045+0000) 2024-05-27T07:39:03.697 INFO:journalctl@ceph.osd.10.smithi105.stdout:May 27 07:39:03 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-osd-10[60334]: 2024-05-27T07:39:03.448+0000 7f162fb8f700 -1 osd.10 134 heartbeat_check: no reply from 172.21.15.52:6806 osd.0 since back 2024-05-27T07:37:56.754985+0000 front 2024-05-27T07:37:56.755070+0000 (oldest deadline 2024-05-27T07:38:22.054911+0000) 2024-05-27T07:39:03.697 INFO:journalctl@ceph.osd.10.smithi105.stdout:May 27 07:39:03 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-osd-10[60334]: 2024-05-27T07:39:03.448+0000 7f162fb8f700 -1 osd.10 134 heartbeat_check: no reply from 172.21.15.105:6804 osd.1 since back 2024-05-27T07:38:05.556137+0000 front 2024-05-27T07:38:05.556112+0000 (oldest deadline 2024-05-27T07:38:31.456213+0000) 2024-05-27T07:39:03.698 INFO:journalctl@ceph.osd.10.smithi105.stdout:May 27 07:39:03 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-osd-10[60334]: 2024-05-27T07:39:03.448+0000 7f162fb8f700 -1 osd.10 134 heartbeat_check: no reply from 172.21.15.173:6804 osd.2 since back 2024-05-27T07:38:15.557554+0000 front 2024-05-27T07:38:15.557409+0000 (oldest deadline 2024-05-27T07:38:40.857509+0000) 2024-05-27T07:39:03.698 INFO:journalctl@ceph.osd.10.smithi105.stdout:May 27 07:39:03 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-osd-10[60334]: 2024-05-27T07:39:03.448+0000 7f162fb8f700 -1 osd.10 134 heartbeat_check: no reply from 172.21.15.52:6814 osd.3 since back 2024-05-27T07:38:21.958641+0000 front 2024-05-27T07:38:21.958614+0000 (oldest deadline 2024-05-27T07:38:47.858600+0000) 2024-05-27T07:39:03.698 INFO:journalctl@ceph.osd.10.smithi105.stdout:May 27 07:39:03 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-osd-10[60334]: 2024-05-27T07:39:03.448+0000 7f162fb8f700 -1 osd.10 134 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-05-27T07:38:30.560765+0000 front 2024-05-27T07:38:30.560788+0000 (oldest deadline 2024-05-27T07:38:55.260887+0000) 2024-05-27T07:39:04.109 INFO:journalctl@ceph.osd.8.smithi173.stdout:May 27 07:39:03 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-osd-8[55969]: 2024-05-27T07:39:03.787+0000 7f425b2ea700 -1 osd.8 134 heartbeat_check: no reply from 172.21.15.105:6804 osd.1 since back 2024-05-27T07:38:06.957078+0000 front 2024-05-27T07:38:06.957280+0000 (oldest deadline 2024-05-27T07:38:30.457072+0000) 2024-05-27T07:39:04.109 INFO:journalctl@ceph.osd.8.smithi173.stdout:May 27 07:39:03 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-osd-8[55969]: 2024-05-27T07:39:03.787+0000 7f425b2ea700 -1 osd.8 134 heartbeat_check: no reply from 172.21.15.173:6804 osd.2 since back 2024-05-27T07:38:16.258968+0000 front 2024-05-27T07:38:16.258998+0000 (oldest deadline 2024-05-27T07:38:41.559145+0000) 2024-05-27T07:39:04.109 INFO:journalctl@ceph.osd.8.smithi173.stdout:May 27 07:39:03 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-osd-8[55969]: 2024-05-27T07:39:03.787+0000 7f425b2ea700 -1 osd.8 134 heartbeat_check: no reply from 172.21.15.52:6814 osd.3 since back 2024-05-27T07:38:23.860441+0000 front 2024-05-27T07:38:23.860421+0000 (oldest deadline 2024-05-27T07:38:49.760323+0000) 2024-05-27T07:39:04.109 INFO:journalctl@ceph.osd.8.smithi173.stdout:May 27 07:39:03 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-osd-8[55969]: 2024-05-27T07:39:03.787+0000 7f425b2ea700 -1 osd.8 134 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-05-27T07:38:32.562246+0000 front 2024-05-27T07:38:32.562420+0000 (oldest deadline 2024-05-27T07:38:53.662280+0000) 2024-05-27T07:39:04.110 INFO:journalctl@ceph.osd.8.smithi173.stdout:May 27 07:39:03 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-osd-8[55969]: 2024-05-27T07:39:03.787+0000 7f425b2ea700 -1 osd.8 134 heartbeat_check: no reply from 172.21.15.173:6812 osd.5 since back 2024-05-27T07:38:37.163477+0000 front 2024-05-27T07:38:37.163568+0000 (oldest deadline 2024-05-27T07:39:02.364041+0000) 2024-05-27T07:39:04.236 INFO:journalctl@ceph.osd.9.smithi052.stdout:May 27 07:39:03 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-osd-9[67134]: 2024-05-27T07:39:03.757+0000 7f1e3e2ca700 -1 osd.9 134 heartbeat_check: no reply from 172.21.15.52:6806 osd.0 since back 2024-05-27T07:37:58.861291+0000 front 2024-05-27T07:37:58.861258+0000 (oldest deadline 2024-05-27T07:38:22.361571+0000) 2024-05-27T07:39:04.237 INFO:journalctl@ceph.osd.9.smithi052.stdout:May 27 07:39:03 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-osd-9[67134]: 2024-05-27T07:39:03.757+0000 7f1e3e2ca700 -1 osd.9 134 heartbeat_check: no reply from 172.21.15.105:6804 osd.1 since back 2024-05-27T07:38:05.862960+0000 front 2024-05-27T07:38:05.862645+0000 (oldest deadline 2024-05-27T07:38:30.562694+0000) 2024-05-27T07:39:04.237 INFO:journalctl@ceph.osd.9.smithi052.stdout:May 27 07:39:03 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-osd-9[67134]: 2024-05-27T07:39:03.757+0000 7f1e3e2ca700 -1 osd.9 134 heartbeat_check: no reply from 172.21.15.173:6804 osd.2 since back 2024-05-27T07:38:15.566595+0000 front 2024-05-27T07:38:15.566403+0000 (oldest deadline 2024-05-27T07:38:37.866552+0000) 2024-05-27T07:39:04.237 INFO:journalctl@ceph.osd.9.smithi052.stdout:May 27 07:39:03 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-osd-9[67134]: 2024-05-27T07:39:03.757+0000 7f1e3e2ca700 -1 osd.9 134 heartbeat_check: no reply from 172.21.15.52:6814 osd.3 since back 2024-05-27T07:38:22.468720+0000 front 2024-05-27T07:38:22.468751+0000 (oldest deadline 2024-05-27T07:38:48.269806+0000) 2024-05-27T07:39:04.237 INFO:journalctl@ceph.osd.9.smithi052.stdout:May 27 07:39:03 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-osd-9[67134]: 2024-05-27T07:39:03.757+0000 7f1e3e2ca700 -1 osd.9 134 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-05-27T07:38:31.171291+0000 front 2024-05-27T07:38:31.171372+0000 (oldest deadline 2024-05-27T07:38:56.971580+0000) 2024-05-27T07:39:04.237 INFO:journalctl@ceph.osd.9.smithi052.stdout:May 27 07:39:03 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-osd-9[67134]: 2024-05-27T07:39:03.757+0000 7f1e3e2ca700 -1 osd.9 134 heartbeat_check: no reply from 172.21.15.173:6812 osd.5 since back 2024-05-27T07:38:36.972252+0000 front 2024-05-27T07:38:36.972432+0000 (oldest deadline 2024-05-27T07:39:02.872102+0000) 2024-05-27T07:39:04.609 INFO:journalctl@ceph.osd.11.smithi173.stdout:May 27 07:39:04 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-osd-11[63021]: 2024-05-27T07:39:04.298+0000 7f9d44ac8700 -1 osd.11 134 heartbeat_check: no reply from 172.21.15.52:6806 osd.0 since back 2024-05-27T07:37:57.927796+0000 front 2024-05-27T07:37:57.927759+0000 (oldest deadline 2024-05-27T07:38:22.028025+0000) 2024-05-27T07:39:04.609 INFO:journalctl@ceph.osd.11.smithi173.stdout:May 27 07:39:04 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-osd-11[63021]: 2024-05-27T07:39:04.298+0000 7f9d44ac8700 -1 osd.11 134 heartbeat_check: no reply from 172.21.15.105:6804 osd.1 since back 2024-05-27T07:38:03.129257+0000 front 2024-05-27T07:38:03.129120+0000 (oldest deadline 2024-05-27T07:38:28.929543+0000) 2024-05-27T07:39:04.609 INFO:journalctl@ceph.osd.11.smithi173.stdout:May 27 07:39:04 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-osd-11[63021]: 2024-05-27T07:39:04.298+0000 7f9d44ac8700 -1 osd.11 134 heartbeat_check: no reply from 172.21.15.173:6804 osd.2 since back 2024-05-27T07:38:16.431856+0000 front 2024-05-27T07:38:16.431878+0000 (oldest deadline 2024-05-27T07:38:42.332069+0000) 2024-05-27T07:39:04.609 INFO:journalctl@ceph.osd.11.smithi173.stdout:May 27 07:39:04 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-osd-11[63021]: 2024-05-27T07:39:04.298+0000 7f9d44ac8700 -1 osd.11 134 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-05-27T07:38:28.133691+0000 front 2024-05-27T07:38:28.134117+0000 (oldest deadline 2024-05-27T07:38:55.734329+0000) 2024-05-27T07:39:04.610 INFO:journalctl@ceph.osd.11.smithi173.stdout:May 27 07:39:04 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-osd-11[63021]: 2024-05-27T07:39:04.298+0000 7f9d44ac8700 -1 osd.11 134 heartbeat_check: no reply from 172.21.15.173:6812 osd.5 since back 2024-05-27T07:38:41.035302+0000 front 2024-05-27T07:38:41.035361+0000 (oldest deadline 2024-05-27T07:39:02.036045+0000) 2024-05-27T07:39:04.947 INFO:journalctl@ceph.osd.10.smithi105.stdout:May 27 07:39:04 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-osd-10[60334]: 2024-05-27T07:39:04.497+0000 7f162fb8f700 -1 osd.10 134 heartbeat_check: no reply from 172.21.15.52:6806 osd.0 since back 2024-05-27T07:37:56.754985+0000 front 2024-05-27T07:37:56.755070+0000 (oldest deadline 2024-05-27T07:38:22.054911+0000) 2024-05-27T07:39:04.947 INFO:journalctl@ceph.osd.10.smithi105.stdout:May 27 07:39:04 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-osd-10[60334]: 2024-05-27T07:39:04.497+0000 7f162fb8f700 -1 osd.10 134 heartbeat_check: no reply from 172.21.15.105:6804 osd.1 since back 2024-05-27T07:38:05.556137+0000 front 2024-05-27T07:38:05.556112+0000 (oldest deadline 2024-05-27T07:38:31.456213+0000) 2024-05-27T07:39:04.947 INFO:journalctl@ceph.osd.10.smithi105.stdout:May 27 07:39:04 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-osd-10[60334]: 2024-05-27T07:39:04.497+0000 7f162fb8f700 -1 osd.10 134 heartbeat_check: no reply from 172.21.15.173:6804 osd.2 since back 2024-05-27T07:38:15.557554+0000 front 2024-05-27T07:38:15.557409+0000 (oldest deadline 2024-05-27T07:38:40.857509+0000) 2024-05-27T07:39:04.947 INFO:journalctl@ceph.osd.10.smithi105.stdout:May 27 07:39:04 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-osd-10[60334]: 2024-05-27T07:39:04.497+0000 7f162fb8f700 -1 osd.10 134 heartbeat_check: no reply from 172.21.15.52:6814 osd.3 since back 2024-05-27T07:38:21.958641+0000 front 2024-05-27T07:38:21.958614+0000 (oldest deadline 2024-05-27T07:38:47.858600+0000) 2024-05-27T07:39:04.947 INFO:journalctl@ceph.osd.10.smithi105.stdout:May 27 07:39:04 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-osd-10[60334]: 2024-05-27T07:39:04.497+0000 7f162fb8f700 -1 osd.10 134 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-05-27T07:38:30.560765+0000 front 2024-05-27T07:38:30.560788+0000 (oldest deadline 2024-05-27T07:38:55.260887+0000) 2024-05-27T07:39:04.986 INFO:journalctl@ceph.osd.9.smithi052.stdout:May 27 07:39:04 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-osd-9[67134]: 2024-05-27T07:39:04.736+0000 7f1e3e2ca700 -1 osd.9 134 heartbeat_check: no reply from 172.21.15.52:6806 osd.0 since back 2024-05-27T07:37:58.861291+0000 front 2024-05-27T07:37:58.861258+0000 (oldest deadline 2024-05-27T07:38:22.361571+0000) 2024-05-27T07:39:04.987 INFO:journalctl@ceph.osd.9.smithi052.stdout:May 27 07:39:04 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-osd-9[67134]: 2024-05-27T07:39:04.736+0000 7f1e3e2ca700 -1 osd.9 134 heartbeat_check: no reply from 172.21.15.105:6804 osd.1 since back 2024-05-27T07:38:05.862960+0000 front 2024-05-27T07:38:05.862645+0000 (oldest deadline 2024-05-27T07:38:30.562694+0000) 2024-05-27T07:39:04.987 INFO:journalctl@ceph.osd.9.smithi052.stdout:May 27 07:39:04 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-osd-9[67134]: 2024-05-27T07:39:04.736+0000 7f1e3e2ca700 -1 osd.9 134 heartbeat_check: no reply from 172.21.15.173:6804 osd.2 since back 2024-05-27T07:38:15.566595+0000 front 2024-05-27T07:38:15.566403+0000 (oldest deadline 2024-05-27T07:38:37.866552+0000) 2024-05-27T07:39:04.987 INFO:journalctl@ceph.osd.9.smithi052.stdout:May 27 07:39:04 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-osd-9[67134]: 2024-05-27T07:39:04.736+0000 7f1e3e2ca700 -1 osd.9 134 heartbeat_check: no reply from 172.21.15.52:6814 osd.3 since back 2024-05-27T07:38:22.468720+0000 front 2024-05-27T07:38:22.468751+0000 (oldest deadline 2024-05-27T07:38:48.269806+0000) 2024-05-27T07:39:04.987 INFO:journalctl@ceph.osd.9.smithi052.stdout:May 27 07:39:04 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-osd-9[67134]: 2024-05-27T07:39:04.736+0000 7f1e3e2ca700 -1 osd.9 134 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-05-27T07:38:31.171291+0000 front 2024-05-27T07:38:31.171372+0000 (oldest deadline 2024-05-27T07:38:56.971580+0000) 2024-05-27T07:39:04.987 INFO:journalctl@ceph.osd.9.smithi052.stdout:May 27 07:39:04 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-osd-9[67134]: 2024-05-27T07:39:04.736+0000 7f1e3e2ca700 -1 osd.9 134 heartbeat_check: no reply from 172.21.15.173:6812 osd.5 since back 2024-05-27T07:38:36.972252+0000 front 2024-05-27T07:38:36.972432+0000 (oldest deadline 2024-05-27T07:39:02.872102+0000) 2024-05-27T07:39:05.109 INFO:journalctl@ceph.osd.8.smithi173.stdout:May 27 07:39:04 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-osd-8[55969]: 2024-05-27T07:39:04.788+0000 7f425b2ea700 -1 osd.8 134 heartbeat_check: no reply from 172.21.15.105:6804 osd.1 since back 2024-05-27T07:38:06.957078+0000 front 2024-05-27T07:38:06.957280+0000 (oldest deadline 2024-05-27T07:38:30.457072+0000) 2024-05-27T07:39:05.109 INFO:journalctl@ceph.osd.8.smithi173.stdout:May 27 07:39:04 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-osd-8[55969]: 2024-05-27T07:39:04.788+0000 7f425b2ea700 -1 osd.8 134 heartbeat_check: no reply from 172.21.15.173:6804 osd.2 since back 2024-05-27T07:38:16.258968+0000 front 2024-05-27T07:38:16.258998+0000 (oldest deadline 2024-05-27T07:38:41.559145+0000) 2024-05-27T07:39:05.110 INFO:journalctl@ceph.osd.8.smithi173.stdout:May 27 07:39:04 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-osd-8[55969]: 2024-05-27T07:39:04.788+0000 7f425b2ea700 -1 osd.8 134 heartbeat_check: no reply from 172.21.15.52:6814 osd.3 since back 2024-05-27T07:38:23.860441+0000 front 2024-05-27T07:38:23.860421+0000 (oldest deadline 2024-05-27T07:38:49.760323+0000) 2024-05-27T07:39:05.110 INFO:journalctl@ceph.osd.8.smithi173.stdout:May 27 07:39:04 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-osd-8[55969]: 2024-05-27T07:39:04.788+0000 7f425b2ea700 -1 osd.8 134 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-05-27T07:38:32.562246+0000 front 2024-05-27T07:38:32.562420+0000 (oldest deadline 2024-05-27T07:38:53.662280+0000) 2024-05-27T07:39:05.110 INFO:journalctl@ceph.osd.8.smithi173.stdout:May 27 07:39:04 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-osd-8[55969]: 2024-05-27T07:39:04.788+0000 7f425b2ea700 -1 osd.8 134 heartbeat_check: no reply from 172.21.15.173:6812 osd.5 since back 2024-05-27T07:38:37.163477+0000 front 2024-05-27T07:38:37.163568+0000 (oldest deadline 2024-05-27T07:39:02.364041+0000) 2024-05-27T07:39:05.609 INFO:journalctl@ceph.osd.11.smithi173.stdout:May 27 07:39:05 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-osd-11[63021]: 2024-05-27T07:39:05.249+0000 7f9d44ac8700 -1 osd.11 134 heartbeat_check: no reply from 172.21.15.52:6806 osd.0 since back 2024-05-27T07:37:57.927796+0000 front 2024-05-27T07:37:57.927759+0000 (oldest deadline 2024-05-27T07:38:22.028025+0000) 2024-05-27T07:39:05.609 INFO:journalctl@ceph.osd.11.smithi173.stdout:May 27 07:39:05 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-osd-11[63021]: 2024-05-27T07:39:05.249+0000 7f9d44ac8700 -1 osd.11 134 heartbeat_check: no reply from 172.21.15.105:6804 osd.1 since back 2024-05-27T07:38:03.129257+0000 front 2024-05-27T07:38:03.129120+0000 (oldest deadline 2024-05-27T07:38:28.929543+0000) 2024-05-27T07:39:05.610 INFO:journalctl@ceph.osd.11.smithi173.stdout:May 27 07:39:05 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-osd-11[63021]: 2024-05-27T07:39:05.249+0000 7f9d44ac8700 -1 osd.11 134 heartbeat_check: no reply from 172.21.15.173:6804 osd.2 since back 2024-05-27T07:38:16.431856+0000 front 2024-05-27T07:38:16.431878+0000 (oldest deadline 2024-05-27T07:38:42.332069+0000) 2024-05-27T07:39:05.610 INFO:journalctl@ceph.osd.11.smithi173.stdout:May 27 07:39:05 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-osd-11[63021]: 2024-05-27T07:39:05.249+0000 7f9d44ac8700 -1 osd.11 134 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-05-27T07:38:28.133691+0000 front 2024-05-27T07:38:28.134117+0000 (oldest deadline 2024-05-27T07:38:55.734329+0000) 2024-05-27T07:39:05.610 INFO:journalctl@ceph.osd.11.smithi173.stdout:May 27 07:39:05 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-osd-11[63021]: 2024-05-27T07:39:05.249+0000 7f9d44ac8700 -1 osd.11 134 heartbeat_check: no reply from 172.21.15.173:6812 osd.5 since back 2024-05-27T07:38:41.035302+0000 front 2024-05-27T07:38:41.035361+0000 (oldest deadline 2024-05-27T07:39:02.036045+0000) 2024-05-27T07:39:05.947 INFO:journalctl@ceph.osd.10.smithi105.stdout:May 27 07:39:05 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-osd-10[60334]: 2024-05-27T07:39:05.512+0000 7f162fb8f700 -1 osd.10 134 heartbeat_check: no reply from 172.21.15.52:6806 osd.0 since back 2024-05-27T07:37:56.754985+0000 front 2024-05-27T07:37:56.755070+0000 (oldest deadline 2024-05-27T07:38:22.054911+0000) 2024-05-27T07:39:05.947 INFO:journalctl@ceph.osd.10.smithi105.stdout:May 27 07:39:05 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-osd-10[60334]: 2024-05-27T07:39:05.512+0000 7f162fb8f700 -1 osd.10 134 heartbeat_check: no reply from 172.21.15.105:6804 osd.1 since back 2024-05-27T07:38:05.556137+0000 front 2024-05-27T07:38:05.556112+0000 (oldest deadline 2024-05-27T07:38:31.456213+0000) 2024-05-27T07:39:05.947 INFO:journalctl@ceph.osd.10.smithi105.stdout:May 27 07:39:05 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-osd-10[60334]: 2024-05-27T07:39:05.512+0000 7f162fb8f700 -1 osd.10 134 heartbeat_check: no reply from 172.21.15.173:6804 osd.2 since back 2024-05-27T07:38:15.557554+0000 front 2024-05-27T07:38:15.557409+0000 (oldest deadline 2024-05-27T07:38:40.857509+0000) 2024-05-27T07:39:05.947 INFO:journalctl@ceph.osd.10.smithi105.stdout:May 27 07:39:05 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-osd-10[60334]: 2024-05-27T07:39:05.512+0000 7f162fb8f700 -1 osd.10 134 heartbeat_check: no reply from 172.21.15.52:6814 osd.3 since back 2024-05-27T07:38:21.958641+0000 front 2024-05-27T07:38:21.958614+0000 (oldest deadline 2024-05-27T07:38:47.858600+0000) 2024-05-27T07:39:05.947 INFO:journalctl@ceph.osd.10.smithi105.stdout:May 27 07:39:05 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-osd-10[60334]: 2024-05-27T07:39:05.512+0000 7f162fb8f700 -1 osd.10 134 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-05-27T07:38:30.560765+0000 front 2024-05-27T07:38:30.560788+0000 (oldest deadline 2024-05-27T07:38:55.260887+0000) 2024-05-27T07:39:05.947 INFO:journalctl@ceph.osd.10.smithi105.stdout:May 27 07:39:05 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-osd-10[60334]: 2024-05-27T07:39:05.512+0000 7f162fb8f700 -1 osd.10 134 heartbeat_check: no reply from 172.21.15.173:6812 osd.5 since back 2024-05-27T07:38:36.962107+0000 front 2024-05-27T07:38:36.962140+0000 (oldest deadline 2024-05-27T07:39:04.562331+0000) 2024-05-27T07:39:05.986 INFO:journalctl@ceph.osd.9.smithi052.stdout:May 27 07:39:05 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-osd-9[67134]: 2024-05-27T07:39:05.724+0000 7f1e3e2ca700 -1 osd.9 134 heartbeat_check: no reply from 172.21.15.52:6806 osd.0 since back 2024-05-27T07:37:58.861291+0000 front 2024-05-27T07:37:58.861258+0000 (oldest deadline 2024-05-27T07:38:22.361571+0000) 2024-05-27T07:39:05.986 INFO:journalctl@ceph.osd.9.smithi052.stdout:May 27 07:39:05 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-osd-9[67134]: 2024-05-27T07:39:05.724+0000 7f1e3e2ca700 -1 osd.9 134 heartbeat_check: no reply from 172.21.15.105:6804 osd.1 since back 2024-05-27T07:38:05.862960+0000 front 2024-05-27T07:38:05.862645+0000 (oldest deadline 2024-05-27T07:38:30.562694+0000) 2024-05-27T07:39:05.986 INFO:journalctl@ceph.osd.9.smithi052.stdout:May 27 07:39:05 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-osd-9[67134]: 2024-05-27T07:39:05.724+0000 7f1e3e2ca700 -1 osd.9 134 heartbeat_check: no reply from 172.21.15.173:6804 osd.2 since back 2024-05-27T07:38:15.566595+0000 front 2024-05-27T07:38:15.566403+0000 (oldest deadline 2024-05-27T07:38:37.866552+0000) 2024-05-27T07:39:05.987 INFO:journalctl@ceph.osd.9.smithi052.stdout:May 27 07:39:05 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-osd-9[67134]: 2024-05-27T07:39:05.724+0000 7f1e3e2ca700 -1 osd.9 134 heartbeat_check: no reply from 172.21.15.52:6814 osd.3 since back 2024-05-27T07:38:22.468720+0000 front 2024-05-27T07:38:22.468751+0000 (oldest deadline 2024-05-27T07:38:48.269806+0000) 2024-05-27T07:39:05.987 INFO:journalctl@ceph.osd.9.smithi052.stdout:May 27 07:39:05 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-osd-9[67134]: 2024-05-27T07:39:05.724+0000 7f1e3e2ca700 -1 osd.9 134 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-05-27T07:38:31.171291+0000 front 2024-05-27T07:38:31.171372+0000 (oldest deadline 2024-05-27T07:38:56.971580+0000) 2024-05-27T07:39:05.987 INFO:journalctl@ceph.osd.9.smithi052.stdout:May 27 07:39:05 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-osd-9[67134]: 2024-05-27T07:39:05.724+0000 7f1e3e2ca700 -1 osd.9 134 heartbeat_check: no reply from 172.21.15.173:6812 osd.5 since back 2024-05-27T07:38:36.972252+0000 front 2024-05-27T07:38:36.972432+0000 (oldest deadline 2024-05-27T07:39:02.872102+0000) 2024-05-27T07:39:06.594 INFO:journalctl@ceph.osd.8.smithi173.stdout:May 27 07:39:06 smithi173 podman[76123]: 2024-05-27 07:39:06.30728226 +0000 UTC m=+5.746966118 container died a2a560a11b938ba80396db04f39ddde228c39b7b76f020d98fe521628ff27033 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-osd-8, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , GIT_CLEAN=True, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, ceph=True, org.label-schema.build-date=20240521, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0) 2024-05-27T07:39:06.595 INFO:journalctl@ceph.osd.11.smithi173.stdout:May 27 07:39:06 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-osd-11[63021]: 2024-05-27T07:39:06.248+0000 7f9d44ac8700 -1 osd.11 134 heartbeat_check: no reply from 172.21.15.52:6806 osd.0 since back 2024-05-27T07:37:57.927796+0000 front 2024-05-27T07:37:57.927759+0000 (oldest deadline 2024-05-27T07:38:22.028025+0000) 2024-05-27T07:39:06.595 INFO:journalctl@ceph.osd.11.smithi173.stdout:May 27 07:39:06 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-osd-11[63021]: 2024-05-27T07:39:06.248+0000 7f9d44ac8700 -1 osd.11 134 heartbeat_check: no reply from 172.21.15.105:6804 osd.1 since back 2024-05-27T07:38:03.129257+0000 front 2024-05-27T07:38:03.129120+0000 (oldest deadline 2024-05-27T07:38:28.929543+0000) 2024-05-27T07:39:06.595 INFO:journalctl@ceph.osd.11.smithi173.stdout:May 27 07:39:06 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-osd-11[63021]: 2024-05-27T07:39:06.248+0000 7f9d44ac8700 -1 osd.11 134 heartbeat_check: no reply from 172.21.15.173:6804 osd.2 since back 2024-05-27T07:38:16.431856+0000 front 2024-05-27T07:38:16.431878+0000 (oldest deadline 2024-05-27T07:38:42.332069+0000) 2024-05-27T07:39:06.595 INFO:journalctl@ceph.osd.11.smithi173.stdout:May 27 07:39:06 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-osd-11[63021]: 2024-05-27T07:39:06.248+0000 7f9d44ac8700 -1 osd.11 134 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-05-27T07:38:28.133691+0000 front 2024-05-27T07:38:28.134117+0000 (oldest deadline 2024-05-27T07:38:55.734329+0000) 2024-05-27T07:39:06.595 INFO:journalctl@ceph.osd.11.smithi173.stdout:May 27 07:39:06 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-osd-11[63021]: 2024-05-27T07:39:06.248+0000 7f9d44ac8700 -1 osd.11 134 heartbeat_check: no reply from 172.21.15.173:6812 osd.5 since back 2024-05-27T07:38:41.035302+0000 front 2024-05-27T07:38:41.035361+0000 (oldest deadline 2024-05-27T07:39:02.036045+0000) 2024-05-27T07:39:06.859 INFO:journalctl@ceph.osd.8.smithi173.stdout:May 27 07:39:06 smithi173 podman[76123]: 2024-05-27 07:39:06.69613393 +0000 UTC m=+6.135817797 container cleanup a2a560a11b938ba80396db04f39ddde228c39b7b76f020d98fe521628ff27033 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-osd-8, org.label-schema.schema-version=1.0, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, org.label-schema.build-date=20240521, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, ceph=True, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, GIT_CLEAN=True) 2024-05-27T07:39:06.859 INFO:journalctl@ceph.osd.8.smithi173.stdout:May 27 07:39:06 smithi173 bash[76123]: ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-osd-8 2024-05-27T07:39:06.947 INFO:journalctl@ceph.osd.10.smithi105.stdout:May 27 07:39:06 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-osd-10[60334]: 2024-05-27T07:39:06.488+0000 7f162fb8f700 -1 osd.10 134 heartbeat_check: no reply from 172.21.15.52:6806 osd.0 since back 2024-05-27T07:37:56.754985+0000 front 2024-05-27T07:37:56.755070+0000 (oldest deadline 2024-05-27T07:38:22.054911+0000) 2024-05-27T07:39:06.947 INFO:journalctl@ceph.osd.10.smithi105.stdout:May 27 07:39:06 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-osd-10[60334]: 2024-05-27T07:39:06.488+0000 7f162fb8f700 -1 osd.10 134 heartbeat_check: no reply from 172.21.15.105:6804 osd.1 since back 2024-05-27T07:38:05.556137+0000 front 2024-05-27T07:38:05.556112+0000 (oldest deadline 2024-05-27T07:38:31.456213+0000) 2024-05-27T07:39:06.947 INFO:journalctl@ceph.osd.10.smithi105.stdout:May 27 07:39:06 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-osd-10[60334]: 2024-05-27T07:39:06.488+0000 7f162fb8f700 -1 osd.10 134 heartbeat_check: no reply from 172.21.15.173:6804 osd.2 since back 2024-05-27T07:38:15.557554+0000 front 2024-05-27T07:38:15.557409+0000 (oldest deadline 2024-05-27T07:38:40.857509+0000) 2024-05-27T07:39:06.947 INFO:journalctl@ceph.osd.10.smithi105.stdout:May 27 07:39:06 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-osd-10[60334]: 2024-05-27T07:39:06.488+0000 7f162fb8f700 -1 osd.10 134 heartbeat_check: no reply from 172.21.15.52:6814 osd.3 since back 2024-05-27T07:38:21.958641+0000 front 2024-05-27T07:38:21.958614+0000 (oldest deadline 2024-05-27T07:38:47.858600+0000) 2024-05-27T07:39:06.947 INFO:journalctl@ceph.osd.10.smithi105.stdout:May 27 07:39:06 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-osd-10[60334]: 2024-05-27T07:39:06.488+0000 7f162fb8f700 -1 osd.10 134 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-05-27T07:38:30.560765+0000 front 2024-05-27T07:38:30.560788+0000 (oldest deadline 2024-05-27T07:38:55.260887+0000) 2024-05-27T07:39:06.947 INFO:journalctl@ceph.osd.10.smithi105.stdout:May 27 07:39:06 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-osd-10[60334]: 2024-05-27T07:39:06.488+0000 7f162fb8f700 -1 osd.10 134 heartbeat_check: no reply from 172.21.15.173:6812 osd.5 since back 2024-05-27T07:38:36.962107+0000 front 2024-05-27T07:38:36.962140+0000 (oldest deadline 2024-05-27T07:39:04.562331+0000) 2024-05-27T07:39:07.236 INFO:journalctl@ceph.osd.9.smithi052.stdout:May 27 07:39:06 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-osd-9[67134]: 2024-05-27T07:39:06.767+0000 7f1e3e2ca700 -1 osd.9 134 heartbeat_check: no reply from 172.21.15.52:6806 osd.0 since back 2024-05-27T07:37:58.861291+0000 front 2024-05-27T07:37:58.861258+0000 (oldest deadline 2024-05-27T07:38:22.361571+0000) 2024-05-27T07:39:07.237 INFO:journalctl@ceph.osd.9.smithi052.stdout:May 27 07:39:06 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-osd-9[67134]: 2024-05-27T07:39:06.767+0000 7f1e3e2ca700 -1 osd.9 134 heartbeat_check: no reply from 172.21.15.105:6804 osd.1 since back 2024-05-27T07:38:05.862960+0000 front 2024-05-27T07:38:05.862645+0000 (oldest deadline 2024-05-27T07:38:30.562694+0000) 2024-05-27T07:39:07.237 INFO:journalctl@ceph.osd.9.smithi052.stdout:May 27 07:39:06 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-osd-9[67134]: 2024-05-27T07:39:06.767+0000 7f1e3e2ca700 -1 osd.9 134 heartbeat_check: no reply from 172.21.15.173:6804 osd.2 since back 2024-05-27T07:38:15.566595+0000 front 2024-05-27T07:38:15.566403+0000 (oldest deadline 2024-05-27T07:38:37.866552+0000) 2024-05-27T07:39:07.237 INFO:journalctl@ceph.osd.9.smithi052.stdout:May 27 07:39:06 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-osd-9[67134]: 2024-05-27T07:39:06.767+0000 7f1e3e2ca700 -1 osd.9 134 heartbeat_check: no reply from 172.21.15.52:6814 osd.3 since back 2024-05-27T07:38:22.468720+0000 front 2024-05-27T07:38:22.468751+0000 (oldest deadline 2024-05-27T07:38:48.269806+0000) 2024-05-27T07:39:07.237 INFO:journalctl@ceph.osd.9.smithi052.stdout:May 27 07:39:06 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-osd-9[67134]: 2024-05-27T07:39:06.767+0000 7f1e3e2ca700 -1 osd.9 134 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-05-27T07:38:31.171291+0000 front 2024-05-27T07:38:31.171372+0000 (oldest deadline 2024-05-27T07:38:56.971580+0000) 2024-05-27T07:39:07.237 INFO:journalctl@ceph.osd.9.smithi052.stdout:May 27 07:39:06 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-osd-9[67134]: 2024-05-27T07:39:06.767+0000 7f1e3e2ca700 -1 osd.9 134 heartbeat_check: no reply from 172.21.15.173:6812 osd.5 since back 2024-05-27T07:38:36.972252+0000 front 2024-05-27T07:38:36.972432+0000 (oldest deadline 2024-05-27T07:39:02.872102+0000) 2024-05-27T07:39:07.356 INFO:journalctl@ceph.osd.8.smithi173.stdout:May 27 07:39:07 smithi173 podman[76138]: 2024-05-27 07:39:07.000169172 +0000 UTC m=+0.692811268 container remove a2a560a11b938ba80396db04f39ddde228c39b7b76f020d98fe521628ff27033 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-osd-8, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, org.label-schema.build-date=20240521, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, ceph=True) 2024-05-27T07:39:07.357 INFO:journalctl@ceph.osd.11.smithi173.stdout:May 27 07:39:07 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-osd-11[63021]: 2024-05-27T07:39:07.230+0000 7f9d44ac8700 -1 osd.11 134 heartbeat_check: no reply from 172.21.15.52:6806 osd.0 since back 2024-05-27T07:37:57.927796+0000 front 2024-05-27T07:37:57.927759+0000 (oldest deadline 2024-05-27T07:38:22.028025+0000) 2024-05-27T07:39:07.357 INFO:journalctl@ceph.osd.11.smithi173.stdout:May 27 07:39:07 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-osd-11[63021]: 2024-05-27T07:39:07.230+0000 7f9d44ac8700 -1 osd.11 134 heartbeat_check: no reply from 172.21.15.105:6804 osd.1 since back 2024-05-27T07:38:03.129257+0000 front 2024-05-27T07:38:03.129120+0000 (oldest deadline 2024-05-27T07:38:28.929543+0000) 2024-05-27T07:39:07.357 INFO:journalctl@ceph.osd.11.smithi173.stdout:May 27 07:39:07 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-osd-11[63021]: 2024-05-27T07:39:07.230+0000 7f9d44ac8700 -1 osd.11 134 heartbeat_check: no reply from 172.21.15.173:6804 osd.2 since back 2024-05-27T07:38:16.431856+0000 front 2024-05-27T07:38:16.431878+0000 (oldest deadline 2024-05-27T07:38:42.332069+0000) 2024-05-27T07:39:07.357 INFO:journalctl@ceph.osd.11.smithi173.stdout:May 27 07:39:07 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-osd-11[63021]: 2024-05-27T07:39:07.230+0000 7f9d44ac8700 -1 osd.11 134 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-05-27T07:38:28.133691+0000 front 2024-05-27T07:38:28.134117+0000 (oldest deadline 2024-05-27T07:38:55.734329+0000) 2024-05-27T07:39:07.357 INFO:journalctl@ceph.osd.11.smithi173.stdout:May 27 07:39:07 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-osd-11[63021]: 2024-05-27T07:39:07.230+0000 7f9d44ac8700 -1 osd.11 134 heartbeat_check: no reply from 172.21.15.173:6812 osd.5 since back 2024-05-27T07:38:41.035302+0000 front 2024-05-27T07:38:41.035361+0000 (oldest deadline 2024-05-27T07:39:02.036045+0000) 2024-05-27T07:39:07.609 INFO:journalctl@ceph.osd.8.smithi173.stdout:May 27 07:39:07 smithi173 podman[76229]: 2024-05-27 07:39:07.262857936 +0000 UTC m=+0.016965003 image pull e36ea4fc8a5807e8d2701370e350a0de3856538797f7fba64cfce7e344bd6799 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6 2024-05-27T07:39:07.609 INFO:journalctl@ceph.osd.8.smithi173.stdout:May 27 07:39:07 smithi173 podman[76229]: 2024-05-27 07:39:07.367827061 +0000 UTC m=+0.121934113 container create 0147d51559f6650b685a1a5c0e0c9531fcdc043cc0e0de1262a524d2eadc50db (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-osd-8-deactivate, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, org.label-schema.build-date=20240521, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, ceph=True, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, maintainer=Guillaume Abrioux ) 2024-05-27T07:39:07.609 INFO:journalctl@ceph.osd.8.smithi173.stdout:May 27 07:39:07 smithi173 podman[76229]: 2024-05-27 07:39:07.570120286 +0000 UTC m=+0.324227343 container init 0147d51559f6650b685a1a5c0e0c9531fcdc043cc0e0de1262a524d2eadc50db (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-osd-8-deactivate, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240521, GIT_CLEAN=True, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS) 2024-05-27T07:39:07.609 INFO:journalctl@ceph.osd.8.smithi173.stdout:May 27 07:39:07 smithi173 podman[76229]: 2024-05-27 07:39:07.573305461 +0000 UTC m=+0.327412522 container start 0147d51559f6650b685a1a5c0e0c9531fcdc043cc0e0de1262a524d2eadc50db (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-osd-8-deactivate, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, GIT_CLEAN=True, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, ceph=True, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, org.label-schema.build-date=20240521, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90) 2024-05-27T07:39:07.862 INFO:journalctl@ceph.osd.8.smithi173.stdout:May 27 07:39:07 smithi173 podman[76229]: 2024-05-27 07:39:07.670318723 +0000 UTC m=+0.424425778 container attach 0147d51559f6650b685a1a5c0e0c9531fcdc043cc0e0de1262a524d2eadc50db (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-osd-8-deactivate, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240521, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, GIT_CLEAN=True, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, ceph=True) 2024-05-27T07:39:07.948 INFO:journalctl@ceph.osd.10.smithi105.stdout:May 27 07:39:07 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-osd-10[60334]: 2024-05-27T07:39:07.526+0000 7f162fb8f700 -1 osd.10 134 heartbeat_check: no reply from 172.21.15.52:6806 osd.0 since back 2024-05-27T07:37:56.754985+0000 front 2024-05-27T07:37:56.755070+0000 (oldest deadline 2024-05-27T07:38:22.054911+0000) 2024-05-27T07:39:07.948 INFO:journalctl@ceph.osd.10.smithi105.stdout:May 27 07:39:07 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-osd-10[60334]: 2024-05-27T07:39:07.526+0000 7f162fb8f700 -1 osd.10 134 heartbeat_check: no reply from 172.21.15.105:6804 osd.1 since back 2024-05-27T07:38:05.556137+0000 front 2024-05-27T07:38:05.556112+0000 (oldest deadline 2024-05-27T07:38:31.456213+0000) 2024-05-27T07:39:07.948 INFO:journalctl@ceph.osd.10.smithi105.stdout:May 27 07:39:07 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-osd-10[60334]: 2024-05-27T07:39:07.526+0000 7f162fb8f700 -1 osd.10 134 heartbeat_check: no reply from 172.21.15.173:6804 osd.2 since back 2024-05-27T07:38:15.557554+0000 front 2024-05-27T07:38:15.557409+0000 (oldest deadline 2024-05-27T07:38:40.857509+0000) 2024-05-27T07:39:07.948 INFO:journalctl@ceph.osd.10.smithi105.stdout:May 27 07:39:07 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-osd-10[60334]: 2024-05-27T07:39:07.526+0000 7f162fb8f700 -1 osd.10 134 heartbeat_check: no reply from 172.21.15.52:6814 osd.3 since back 2024-05-27T07:38:21.958641+0000 front 2024-05-27T07:38:21.958614+0000 (oldest deadline 2024-05-27T07:38:47.858600+0000) 2024-05-27T07:39:07.948 INFO:journalctl@ceph.osd.10.smithi105.stdout:May 27 07:39:07 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-osd-10[60334]: 2024-05-27T07:39:07.526+0000 7f162fb8f700 -1 osd.10 134 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-05-27T07:38:30.560765+0000 front 2024-05-27T07:38:30.560788+0000 (oldest deadline 2024-05-27T07:38:55.260887+0000) 2024-05-27T07:39:07.948 INFO:journalctl@ceph.osd.10.smithi105.stdout:May 27 07:39:07 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-osd-10[60334]: 2024-05-27T07:39:07.526+0000 7f162fb8f700 -1 osd.10 134 heartbeat_check: no reply from 172.21.15.173:6812 osd.5 since back 2024-05-27T07:38:36.962107+0000 front 2024-05-27T07:38:36.962140+0000 (oldest deadline 2024-05-27T07:39:04.562331+0000) 2024-05-27T07:39:08.161 INFO:journalctl@ceph.osd.8.smithi173.stdout:May 27 07:39:07 smithi173 podman[76229]: 2024-05-27 07:39:07.86570014 +0000 UTC m=+0.619807195 container died 0147d51559f6650b685a1a5c0e0c9531fcdc043cc0e0de1262a524d2eadc50db (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-osd-8-deactivate, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, ceph=True, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240521, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, GIT_BRANCH=HEAD, GIT_CLEAN=True, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, CEPH_POINT_RELEASE=) 2024-05-27T07:39:08.236 INFO:journalctl@ceph.osd.9.smithi052.stdout:May 27 07:39:07 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-osd-9[67134]: 2024-05-27T07:39:07.775+0000 7f1e3e2ca700 -1 osd.9 134 heartbeat_check: no reply from 172.21.15.52:6806 osd.0 since back 2024-05-27T07:37:58.861291+0000 front 2024-05-27T07:37:58.861258+0000 (oldest deadline 2024-05-27T07:38:22.361571+0000) 2024-05-27T07:39:08.236 INFO:journalctl@ceph.osd.9.smithi052.stdout:May 27 07:39:07 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-osd-9[67134]: 2024-05-27T07:39:07.775+0000 7f1e3e2ca700 -1 osd.9 134 heartbeat_check: no reply from 172.21.15.105:6804 osd.1 since back 2024-05-27T07:38:05.862960+0000 front 2024-05-27T07:38:05.862645+0000 (oldest deadline 2024-05-27T07:38:30.562694+0000) 2024-05-27T07:39:08.237 INFO:journalctl@ceph.osd.9.smithi052.stdout:May 27 07:39:07 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-osd-9[67134]: 2024-05-27T07:39:07.775+0000 7f1e3e2ca700 -1 osd.9 134 heartbeat_check: no reply from 172.21.15.173:6804 osd.2 since back 2024-05-27T07:38:15.566595+0000 front 2024-05-27T07:38:15.566403+0000 (oldest deadline 2024-05-27T07:38:37.866552+0000) 2024-05-27T07:39:08.237 INFO:journalctl@ceph.osd.9.smithi052.stdout:May 27 07:39:07 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-osd-9[67134]: 2024-05-27T07:39:07.775+0000 7f1e3e2ca700 -1 osd.9 134 heartbeat_check: no reply from 172.21.15.52:6814 osd.3 since back 2024-05-27T07:38:22.468720+0000 front 2024-05-27T07:38:22.468751+0000 (oldest deadline 2024-05-27T07:38:48.269806+0000) 2024-05-27T07:39:08.237 INFO:journalctl@ceph.osd.9.smithi052.stdout:May 27 07:39:07 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-osd-9[67134]: 2024-05-27T07:39:07.775+0000 7f1e3e2ca700 -1 osd.9 134 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-05-27T07:38:31.171291+0000 front 2024-05-27T07:38:31.171372+0000 (oldest deadline 2024-05-27T07:38:56.971580+0000) 2024-05-27T07:39:08.237 INFO:journalctl@ceph.osd.9.smithi052.stdout:May 27 07:39:07 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-osd-9[67134]: 2024-05-27T07:39:07.775+0000 7f1e3e2ca700 -1 osd.9 134 heartbeat_check: no reply from 172.21.15.173:6812 osd.5 since back 2024-05-27T07:38:36.972252+0000 front 2024-05-27T07:38:36.972432+0000 (oldest deadline 2024-05-27T07:39:02.872102+0000) 2024-05-27T07:39:08.438 INFO:journalctl@ceph.osd.11.smithi173.stdout:May 27 07:39:08 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-osd-11[63021]: 2024-05-27T07:39:08.199+0000 7f9d44ac8700 -1 osd.11 134 heartbeat_check: no reply from 172.21.15.52:6806 osd.0 since back 2024-05-27T07:37:57.927796+0000 front 2024-05-27T07:37:57.927759+0000 (oldest deadline 2024-05-27T07:38:22.028025+0000) 2024-05-27T07:39:08.438 INFO:journalctl@ceph.osd.11.smithi173.stdout:May 27 07:39:08 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-osd-11[63021]: 2024-05-27T07:39:08.199+0000 7f9d44ac8700 -1 osd.11 134 heartbeat_check: no reply from 172.21.15.105:6804 osd.1 since back 2024-05-27T07:38:03.129257+0000 front 2024-05-27T07:38:03.129120+0000 (oldest deadline 2024-05-27T07:38:28.929543+0000) 2024-05-27T07:39:08.438 INFO:journalctl@ceph.osd.11.smithi173.stdout:May 27 07:39:08 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-osd-11[63021]: 2024-05-27T07:39:08.199+0000 7f9d44ac8700 -1 osd.11 134 heartbeat_check: no reply from 172.21.15.173:6804 osd.2 since back 2024-05-27T07:38:16.431856+0000 front 2024-05-27T07:38:16.431878+0000 (oldest deadline 2024-05-27T07:38:42.332069+0000) 2024-05-27T07:39:08.438 INFO:journalctl@ceph.osd.11.smithi173.stdout:May 27 07:39:08 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-osd-11[63021]: 2024-05-27T07:39:08.199+0000 7f9d44ac8700 -1 osd.11 134 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-05-27T07:38:28.133691+0000 front 2024-05-27T07:38:28.134117+0000 (oldest deadline 2024-05-27T07:38:55.734329+0000) 2024-05-27T07:39:08.438 INFO:journalctl@ceph.osd.11.smithi173.stdout:May 27 07:39:08 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-osd-11[63021]: 2024-05-27T07:39:08.199+0000 7f9d44ac8700 -1 osd.11 134 heartbeat_check: no reply from 172.21.15.173:6812 osd.5 since back 2024-05-27T07:38:41.035302+0000 front 2024-05-27T07:38:41.035361+0000 (oldest deadline 2024-05-27T07:39:02.036045+0000) 2024-05-27T07:39:08.460 DEBUG:teuthology.orchestra.run.smithi173:> sudo pkill -f 'journalctl -f -n 0 -u ceph-73df2636-1bf8-11ef-bc9b-c7b262605968@osd.8.service' 2024-05-27T07:39:08.497 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-05-27T07:39:08.497 INFO:tasks.cephadm.osd.8:Stopped osd.8 2024-05-27T07:39:08.498 INFO:tasks.cephadm.osd.9:Stopping osd.9... 2024-05-27T07:39:08.498 DEBUG:teuthology.orchestra.run.smithi052:> sudo systemctl stop ceph-73df2636-1bf8-11ef-bc9b-c7b262605968@osd.9 2024-05-27T07:39:08.823 INFO:journalctl@ceph.osd.9.smithi052.stdout:May 27 07:39:08 smithi052 systemd[1]: Stopping Ceph osd.9 for 73df2636-1bf8-11ef-bc9b-c7b262605968... 2024-05-27T07:39:08.823 INFO:journalctl@ceph.osd.9.smithi052.stdout:May 27 07:39:08 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-osd-9[67134]: 2024-05-27T07:39:08.738+0000 7f1e3e2ca700 -1 osd.9 134 heartbeat_check: no reply from 172.21.15.52:6806 osd.0 since back 2024-05-27T07:37:58.861291+0000 front 2024-05-27T07:37:58.861258+0000 (oldest deadline 2024-05-27T07:38:22.361571+0000) 2024-05-27T07:39:08.824 INFO:journalctl@ceph.osd.9.smithi052.stdout:May 27 07:39:08 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-osd-9[67134]: 2024-05-27T07:39:08.738+0000 7f1e3e2ca700 -1 osd.9 134 heartbeat_check: no reply from 172.21.15.105:6804 osd.1 since back 2024-05-27T07:38:05.862960+0000 front 2024-05-27T07:38:05.862645+0000 (oldest deadline 2024-05-27T07:38:30.562694+0000) 2024-05-27T07:39:08.824 INFO:journalctl@ceph.osd.9.smithi052.stdout:May 27 07:39:08 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-osd-9[67134]: 2024-05-27T07:39:08.738+0000 7f1e3e2ca700 -1 osd.9 134 heartbeat_check: no reply from 172.21.15.173:6804 osd.2 since back 2024-05-27T07:38:15.566595+0000 front 2024-05-27T07:38:15.566403+0000 (oldest deadline 2024-05-27T07:38:37.866552+0000) 2024-05-27T07:39:08.824 INFO:journalctl@ceph.osd.9.smithi052.stdout:May 27 07:39:08 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-osd-9[67134]: 2024-05-27T07:39:08.738+0000 7f1e3e2ca700 -1 osd.9 134 heartbeat_check: no reply from 172.21.15.52:6814 osd.3 since back 2024-05-27T07:38:22.468720+0000 front 2024-05-27T07:38:22.468751+0000 (oldest deadline 2024-05-27T07:38:48.269806+0000) 2024-05-27T07:39:08.824 INFO:journalctl@ceph.osd.9.smithi052.stdout:May 27 07:39:08 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-osd-9[67134]: 2024-05-27T07:39:08.738+0000 7f1e3e2ca700 -1 osd.9 134 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-05-27T07:38:31.171291+0000 front 2024-05-27T07:38:31.171372+0000 (oldest deadline 2024-05-27T07:38:56.971580+0000) 2024-05-27T07:39:08.824 INFO:journalctl@ceph.osd.9.smithi052.stdout:May 27 07:39:08 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-osd-9[67134]: 2024-05-27T07:39:08.738+0000 7f1e3e2ca700 -1 osd.9 134 heartbeat_check: no reply from 172.21.15.173:6812 osd.5 since back 2024-05-27T07:38:36.972252+0000 front 2024-05-27T07:38:36.972432+0000 (oldest deadline 2024-05-27T07:39:02.872102+0000) 2024-05-27T07:39:08.947 INFO:journalctl@ceph.osd.10.smithi105.stdout:May 27 07:39:08 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-osd-10[60334]: 2024-05-27T07:39:08.534+0000 7f162fb8f700 -1 osd.10 134 heartbeat_check: no reply from 172.21.15.52:6806 osd.0 since back 2024-05-27T07:37:56.754985+0000 front 2024-05-27T07:37:56.755070+0000 (oldest deadline 2024-05-27T07:38:22.054911+0000) 2024-05-27T07:39:08.947 INFO:journalctl@ceph.osd.10.smithi105.stdout:May 27 07:39:08 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-osd-10[60334]: 2024-05-27T07:39:08.534+0000 7f162fb8f700 -1 osd.10 134 heartbeat_check: no reply from 172.21.15.105:6804 osd.1 since back 2024-05-27T07:38:05.556137+0000 front 2024-05-27T07:38:05.556112+0000 (oldest deadline 2024-05-27T07:38:31.456213+0000) 2024-05-27T07:39:08.947 INFO:journalctl@ceph.osd.10.smithi105.stdout:May 27 07:39:08 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-osd-10[60334]: 2024-05-27T07:39:08.534+0000 7f162fb8f700 -1 osd.10 134 heartbeat_check: no reply from 172.21.15.173:6804 osd.2 since back 2024-05-27T07:38:15.557554+0000 front 2024-05-27T07:38:15.557409+0000 (oldest deadline 2024-05-27T07:38:40.857509+0000) 2024-05-27T07:39:08.947 INFO:journalctl@ceph.osd.10.smithi105.stdout:May 27 07:39:08 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-osd-10[60334]: 2024-05-27T07:39:08.534+0000 7f162fb8f700 -1 osd.10 134 heartbeat_check: no reply from 172.21.15.52:6814 osd.3 since back 2024-05-27T07:38:21.958641+0000 front 2024-05-27T07:38:21.958614+0000 (oldest deadline 2024-05-27T07:38:47.858600+0000) 2024-05-27T07:39:08.947 INFO:journalctl@ceph.osd.10.smithi105.stdout:May 27 07:39:08 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-osd-10[60334]: 2024-05-27T07:39:08.534+0000 7f162fb8f700 -1 osd.10 134 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-05-27T07:38:30.560765+0000 front 2024-05-27T07:38:30.560788+0000 (oldest deadline 2024-05-27T07:38:55.260887+0000) 2024-05-27T07:39:08.948 INFO:journalctl@ceph.osd.10.smithi105.stdout:May 27 07:39:08 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-osd-10[60334]: 2024-05-27T07:39:08.534+0000 7f162fb8f700 -1 osd.10 134 heartbeat_check: no reply from 172.21.15.173:6812 osd.5 since back 2024-05-27T07:38:36.962107+0000 front 2024-05-27T07:38:36.962140+0000 (oldest deadline 2024-05-27T07:39:04.562331+0000) 2024-05-27T07:39:09.236 INFO:journalctl@ceph.osd.9.smithi052.stdout:May 27 07:39:08 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-osd-9[67134]: 2024-05-27T07:39:08.826+0000 7f1e45d1c700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.9 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-05-27T07:39:09.236 INFO:journalctl@ceph.osd.9.smithi052.stdout:May 27 07:39:08 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-osd-9[67134]: 2024-05-27T07:39:08.826+0000 7f1e45d1c700 -1 osd.9 134 *** Got signal Terminated *** 2024-05-27T07:39:09.236 INFO:journalctl@ceph.osd.9.smithi052.stdout:May 27 07:39:08 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-osd-9[67134]: 2024-05-27T07:39:08.826+0000 7f1e45d1c700 -1 osd.9 134 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-05-27T07:39:09.609 INFO:journalctl@ceph.osd.11.smithi173.stdout:May 27 07:39:09 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-osd-11[63021]: 2024-05-27T07:39:09.237+0000 7f9d44ac8700 -1 osd.11 134 heartbeat_check: no reply from 172.21.15.52:6806 osd.0 since back 2024-05-27T07:37:57.927796+0000 front 2024-05-27T07:37:57.927759+0000 (oldest deadline 2024-05-27T07:38:22.028025+0000) 2024-05-27T07:39:09.609 INFO:journalctl@ceph.osd.11.smithi173.stdout:May 27 07:39:09 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-osd-11[63021]: 2024-05-27T07:39:09.237+0000 7f9d44ac8700 -1 osd.11 134 heartbeat_check: no reply from 172.21.15.105:6804 osd.1 since back 2024-05-27T07:38:03.129257+0000 front 2024-05-27T07:38:03.129120+0000 (oldest deadline 2024-05-27T07:38:28.929543+0000) 2024-05-27T07:39:09.609 INFO:journalctl@ceph.osd.11.smithi173.stdout:May 27 07:39:09 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-osd-11[63021]: 2024-05-27T07:39:09.237+0000 7f9d44ac8700 -1 osd.11 134 heartbeat_check: no reply from 172.21.15.173:6804 osd.2 since back 2024-05-27T07:38:16.431856+0000 front 2024-05-27T07:38:16.431878+0000 (oldest deadline 2024-05-27T07:38:42.332069+0000) 2024-05-27T07:39:09.609 INFO:journalctl@ceph.osd.11.smithi173.stdout:May 27 07:39:09 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-osd-11[63021]: 2024-05-27T07:39:09.237+0000 7f9d44ac8700 -1 osd.11 134 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-05-27T07:38:28.133691+0000 front 2024-05-27T07:38:28.134117+0000 (oldest deadline 2024-05-27T07:38:55.734329+0000) 2024-05-27T07:39:09.609 INFO:journalctl@ceph.osd.11.smithi173.stdout:May 27 07:39:09 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-osd-11[63021]: 2024-05-27T07:39:09.237+0000 7f9d44ac8700 -1 osd.11 134 heartbeat_check: no reply from 172.21.15.173:6812 osd.5 since back 2024-05-27T07:38:41.035302+0000 front 2024-05-27T07:38:41.035361+0000 (oldest deadline 2024-05-27T07:39:02.036045+0000) 2024-05-27T07:39:09.947 INFO:journalctl@ceph.osd.10.smithi105.stdout:May 27 07:39:09 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-osd-10[60334]: 2024-05-27T07:39:09.575+0000 7f162fb8f700 -1 osd.10 134 heartbeat_check: no reply from 172.21.15.52:6806 osd.0 since back 2024-05-27T07:37:56.754985+0000 front 2024-05-27T07:37:56.755070+0000 (oldest deadline 2024-05-27T07:38:22.054911+0000) 2024-05-27T07:39:09.947 INFO:journalctl@ceph.osd.10.smithi105.stdout:May 27 07:39:09 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-osd-10[60334]: 2024-05-27T07:39:09.575+0000 7f162fb8f700 -1 osd.10 134 heartbeat_check: no reply from 172.21.15.105:6804 osd.1 since back 2024-05-27T07:38:05.556137+0000 front 2024-05-27T07:38:05.556112+0000 (oldest deadline 2024-05-27T07:38:31.456213+0000) 2024-05-27T07:39:09.947 INFO:journalctl@ceph.osd.10.smithi105.stdout:May 27 07:39:09 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-osd-10[60334]: 2024-05-27T07:39:09.575+0000 7f162fb8f700 -1 osd.10 134 heartbeat_check: no reply from 172.21.15.173:6804 osd.2 since back 2024-05-27T07:38:15.557554+0000 front 2024-05-27T07:38:15.557409+0000 (oldest deadline 2024-05-27T07:38:40.857509+0000) 2024-05-27T07:39:09.947 INFO:journalctl@ceph.osd.10.smithi105.stdout:May 27 07:39:09 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-osd-10[60334]: 2024-05-27T07:39:09.575+0000 7f162fb8f700 -1 osd.10 134 heartbeat_check: no reply from 172.21.15.52:6814 osd.3 since back 2024-05-27T07:38:21.958641+0000 front 2024-05-27T07:38:21.958614+0000 (oldest deadline 2024-05-27T07:38:47.858600+0000) 2024-05-27T07:39:09.947 INFO:journalctl@ceph.osd.10.smithi105.stdout:May 27 07:39:09 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-osd-10[60334]: 2024-05-27T07:39:09.575+0000 7f162fb8f700 -1 osd.10 134 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-05-27T07:38:30.560765+0000 front 2024-05-27T07:38:30.560788+0000 (oldest deadline 2024-05-27T07:38:55.260887+0000) 2024-05-27T07:39:09.947 INFO:journalctl@ceph.osd.10.smithi105.stdout:May 27 07:39:09 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-osd-10[60334]: 2024-05-27T07:39:09.575+0000 7f162fb8f700 -1 osd.10 134 heartbeat_check: no reply from 172.21.15.173:6812 osd.5 since back 2024-05-27T07:38:36.962107+0000 front 2024-05-27T07:38:36.962140+0000 (oldest deadline 2024-05-27T07:39:04.562331+0000) 2024-05-27T07:39:10.236 INFO:journalctl@ceph.osd.9.smithi052.stdout:May 27 07:39:09 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-osd-9[67134]: 2024-05-27T07:39:09.765+0000 7f1e3e2ca700 -1 osd.9 134 heartbeat_check: no reply from 172.21.15.52:6806 osd.0 since back 2024-05-27T07:37:58.861291+0000 front 2024-05-27T07:37:58.861258+0000 (oldest deadline 2024-05-27T07:38:22.361571+0000) 2024-05-27T07:39:10.237 INFO:journalctl@ceph.osd.9.smithi052.stdout:May 27 07:39:09 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-osd-9[67134]: 2024-05-27T07:39:09.765+0000 7f1e3e2ca700 -1 osd.9 134 heartbeat_check: no reply from 172.21.15.105:6804 osd.1 since back 2024-05-27T07:38:05.862960+0000 front 2024-05-27T07:38:05.862645+0000 (oldest deadline 2024-05-27T07:38:30.562694+0000) 2024-05-27T07:39:10.237 INFO:journalctl@ceph.osd.9.smithi052.stdout:May 27 07:39:09 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-osd-9[67134]: 2024-05-27T07:39:09.765+0000 7f1e3e2ca700 -1 osd.9 134 heartbeat_check: no reply from 172.21.15.173:6804 osd.2 since back 2024-05-27T07:38:15.566595+0000 front 2024-05-27T07:38:15.566403+0000 (oldest deadline 2024-05-27T07:38:37.866552+0000) 2024-05-27T07:39:10.237 INFO:journalctl@ceph.osd.9.smithi052.stdout:May 27 07:39:09 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-osd-9[67134]: 2024-05-27T07:39:09.765+0000 7f1e3e2ca700 -1 osd.9 134 heartbeat_check: no reply from 172.21.15.52:6814 osd.3 since back 2024-05-27T07:38:22.468720+0000 front 2024-05-27T07:38:22.468751+0000 (oldest deadline 2024-05-27T07:38:48.269806+0000) 2024-05-27T07:39:10.237 INFO:journalctl@ceph.osd.9.smithi052.stdout:May 27 07:39:09 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-osd-9[67134]: 2024-05-27T07:39:09.765+0000 7f1e3e2ca700 -1 osd.9 134 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-05-27T07:38:31.171291+0000 front 2024-05-27T07:38:31.171372+0000 (oldest deadline 2024-05-27T07:38:56.971580+0000) 2024-05-27T07:39:10.237 INFO:journalctl@ceph.osd.9.smithi052.stdout:May 27 07:39:09 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-osd-9[67134]: 2024-05-27T07:39:09.765+0000 7f1e3e2ca700 -1 osd.9 134 heartbeat_check: no reply from 172.21.15.173:6812 osd.5 since back 2024-05-27T07:38:36.972252+0000 front 2024-05-27T07:38:36.972432+0000 (oldest deadline 2024-05-27T07:39:02.872102+0000) 2024-05-27T07:39:10.610 INFO:journalctl@ceph.osd.11.smithi173.stdout:May 27 07:39:10 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-osd-11[63021]: 2024-05-27T07:39:10.246+0000 7f9d44ac8700 -1 osd.11 134 heartbeat_check: no reply from 172.21.15.52:6806 osd.0 since back 2024-05-27T07:37:57.927796+0000 front 2024-05-27T07:37:57.927759+0000 (oldest deadline 2024-05-27T07:38:22.028025+0000) 2024-05-27T07:39:10.610 INFO:journalctl@ceph.osd.11.smithi173.stdout:May 27 07:39:10 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-osd-11[63021]: 2024-05-27T07:39:10.246+0000 7f9d44ac8700 -1 osd.11 134 heartbeat_check: no reply from 172.21.15.105:6804 osd.1 since back 2024-05-27T07:38:03.129257+0000 front 2024-05-27T07:38:03.129120+0000 (oldest deadline 2024-05-27T07:38:28.929543+0000) 2024-05-27T07:39:10.610 INFO:journalctl@ceph.osd.11.smithi173.stdout:May 27 07:39:10 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-osd-11[63021]: 2024-05-27T07:39:10.246+0000 7f9d44ac8700 -1 osd.11 134 heartbeat_check: no reply from 172.21.15.173:6804 osd.2 since back 2024-05-27T07:38:16.431856+0000 front 2024-05-27T07:38:16.431878+0000 (oldest deadline 2024-05-27T07:38:42.332069+0000) 2024-05-27T07:39:10.610 INFO:journalctl@ceph.osd.11.smithi173.stdout:May 27 07:39:10 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-osd-11[63021]: 2024-05-27T07:39:10.246+0000 7f9d44ac8700 -1 osd.11 134 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-05-27T07:38:28.133691+0000 front 2024-05-27T07:38:28.134117+0000 (oldest deadline 2024-05-27T07:38:55.734329+0000) 2024-05-27T07:39:10.610 INFO:journalctl@ceph.osd.11.smithi173.stdout:May 27 07:39:10 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-osd-11[63021]: 2024-05-27T07:39:10.246+0000 7f9d44ac8700 -1 osd.11 134 heartbeat_check: no reply from 172.21.15.173:6812 osd.5 since back 2024-05-27T07:38:41.035302+0000 front 2024-05-27T07:38:41.035361+0000 (oldest deadline 2024-05-27T07:39:02.036045+0000) 2024-05-27T07:39:10.610 INFO:journalctl@ceph.osd.11.smithi173.stdout:May 27 07:39:10 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-osd-11[63021]: 2024-05-27T07:39:10.246+0000 7f9d44ac8700 -1 osd.11 134 heartbeat_check: no reply from 172.21.15.52:6822 osd.6 since back 2024-05-27T07:38:47.237679+0000 front 2024-05-27T07:38:47.237750+0000 (oldest deadline 2024-05-27T07:39:10.137647+0000) 2024-05-27T07:39:10.697 INFO:journalctl@ceph.osd.10.smithi105.stdout:May 27 07:39:10 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-osd-10[60334]: 2024-05-27T07:39:10.577+0000 7f162fb8f700 -1 osd.10 134 heartbeat_check: no reply from 172.21.15.52:6806 osd.0 since back 2024-05-27T07:37:56.754985+0000 front 2024-05-27T07:37:56.755070+0000 (oldest deadline 2024-05-27T07:38:22.054911+0000) 2024-05-27T07:39:10.697 INFO:journalctl@ceph.osd.10.smithi105.stdout:May 27 07:39:10 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-osd-10[60334]: 2024-05-27T07:39:10.577+0000 7f162fb8f700 -1 osd.10 134 heartbeat_check: no reply from 172.21.15.105:6804 osd.1 since back 2024-05-27T07:38:05.556137+0000 front 2024-05-27T07:38:05.556112+0000 (oldest deadline 2024-05-27T07:38:31.456213+0000) 2024-05-27T07:39:10.697 INFO:journalctl@ceph.osd.10.smithi105.stdout:May 27 07:39:10 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-osd-10[60334]: 2024-05-27T07:39:10.577+0000 7f162fb8f700 -1 osd.10 134 heartbeat_check: no reply from 172.21.15.173:6804 osd.2 since back 2024-05-27T07:38:15.557554+0000 front 2024-05-27T07:38:15.557409+0000 (oldest deadline 2024-05-27T07:38:40.857509+0000) 2024-05-27T07:39:10.697 INFO:journalctl@ceph.osd.10.smithi105.stdout:May 27 07:39:10 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-osd-10[60334]: 2024-05-27T07:39:10.577+0000 7f162fb8f700 -1 osd.10 134 heartbeat_check: no reply from 172.21.15.52:6814 osd.3 since back 2024-05-27T07:38:21.958641+0000 front 2024-05-27T07:38:21.958614+0000 (oldest deadline 2024-05-27T07:38:47.858600+0000) 2024-05-27T07:39:10.697 INFO:journalctl@ceph.osd.10.smithi105.stdout:May 27 07:39:10 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-osd-10[60334]: 2024-05-27T07:39:10.577+0000 7f162fb8f700 -1 osd.10 134 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-05-27T07:38:30.560765+0000 front 2024-05-27T07:38:30.560788+0000 (oldest deadline 2024-05-27T07:38:55.260887+0000) 2024-05-27T07:39:10.697 INFO:journalctl@ceph.osd.10.smithi105.stdout:May 27 07:39:10 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-osd-10[60334]: 2024-05-27T07:39:10.577+0000 7f162fb8f700 -1 osd.10 134 heartbeat_check: no reply from 172.21.15.173:6812 osd.5 since back 2024-05-27T07:38:36.962107+0000 front 2024-05-27T07:38:36.962140+0000 (oldest deadline 2024-05-27T07:39:04.562331+0000) 2024-05-27T07:39:11.237 INFO:journalctl@ceph.osd.9.smithi052.stdout:May 27 07:39:10 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-osd-9[67134]: 2024-05-27T07:39:10.814+0000 7f1e3e2ca700 -1 osd.9 134 heartbeat_check: no reply from 172.21.15.52:6806 osd.0 since back 2024-05-27T07:37:58.861291+0000 front 2024-05-27T07:37:58.861258+0000 (oldest deadline 2024-05-27T07:38:22.361571+0000) 2024-05-27T07:39:11.237 INFO:journalctl@ceph.osd.9.smithi052.stdout:May 27 07:39:10 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-osd-9[67134]: 2024-05-27T07:39:10.814+0000 7f1e3e2ca700 -1 osd.9 134 heartbeat_check: no reply from 172.21.15.105:6804 osd.1 since back 2024-05-27T07:38:05.862960+0000 front 2024-05-27T07:38:05.862645+0000 (oldest deadline 2024-05-27T07:38:30.562694+0000) 2024-05-27T07:39:11.237 INFO:journalctl@ceph.osd.9.smithi052.stdout:May 27 07:39:10 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-osd-9[67134]: 2024-05-27T07:39:10.814+0000 7f1e3e2ca700 -1 osd.9 134 heartbeat_check: no reply from 172.21.15.173:6804 osd.2 since back 2024-05-27T07:38:15.566595+0000 front 2024-05-27T07:38:15.566403+0000 (oldest deadline 2024-05-27T07:38:37.866552+0000) 2024-05-27T07:39:11.237 INFO:journalctl@ceph.osd.9.smithi052.stdout:May 27 07:39:10 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-osd-9[67134]: 2024-05-27T07:39:10.814+0000 7f1e3e2ca700 -1 osd.9 134 heartbeat_check: no reply from 172.21.15.52:6814 osd.3 since back 2024-05-27T07:38:22.468720+0000 front 2024-05-27T07:38:22.468751+0000 (oldest deadline 2024-05-27T07:38:48.269806+0000) 2024-05-27T07:39:11.237 INFO:journalctl@ceph.osd.9.smithi052.stdout:May 27 07:39:10 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-osd-9[67134]: 2024-05-27T07:39:10.814+0000 7f1e3e2ca700 -1 osd.9 134 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-05-27T07:38:31.171291+0000 front 2024-05-27T07:38:31.171372+0000 (oldest deadline 2024-05-27T07:38:56.971580+0000) 2024-05-27T07:39:11.237 INFO:journalctl@ceph.osd.9.smithi052.stdout:May 27 07:39:10 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-osd-9[67134]: 2024-05-27T07:39:10.814+0000 7f1e3e2ca700 -1 osd.9 134 heartbeat_check: no reply from 172.21.15.173:6812 osd.5 since back 2024-05-27T07:38:36.972252+0000 front 2024-05-27T07:38:36.972432+0000 (oldest deadline 2024-05-27T07:39:02.872102+0000) 2024-05-27T07:39:11.609 INFO:journalctl@ceph.osd.11.smithi173.stdout:May 27 07:39:11 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-osd-11[63021]: 2024-05-27T07:39:11.249+0000 7f9d44ac8700 -1 osd.11 134 heartbeat_check: no reply from 172.21.15.52:6806 osd.0 since back 2024-05-27T07:37:57.927796+0000 front 2024-05-27T07:37:57.927759+0000 (oldest deadline 2024-05-27T07:38:22.028025+0000) 2024-05-27T07:39:11.609 INFO:journalctl@ceph.osd.11.smithi173.stdout:May 27 07:39:11 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-osd-11[63021]: 2024-05-27T07:39:11.249+0000 7f9d44ac8700 -1 osd.11 134 heartbeat_check: no reply from 172.21.15.105:6804 osd.1 since back 2024-05-27T07:38:03.129257+0000 front 2024-05-27T07:38:03.129120+0000 (oldest deadline 2024-05-27T07:38:28.929543+0000) 2024-05-27T07:39:11.609 INFO:journalctl@ceph.osd.11.smithi173.stdout:May 27 07:39:11 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-osd-11[63021]: 2024-05-27T07:39:11.249+0000 7f9d44ac8700 -1 osd.11 134 heartbeat_check: no reply from 172.21.15.173:6804 osd.2 since back 2024-05-27T07:38:16.431856+0000 front 2024-05-27T07:38:16.431878+0000 (oldest deadline 2024-05-27T07:38:42.332069+0000) 2024-05-27T07:39:11.610 INFO:journalctl@ceph.osd.11.smithi173.stdout:May 27 07:39:11 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-osd-11[63021]: 2024-05-27T07:39:11.249+0000 7f9d44ac8700 -1 osd.11 134 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-05-27T07:38:28.133691+0000 front 2024-05-27T07:38:28.134117+0000 (oldest deadline 2024-05-27T07:38:55.734329+0000) 2024-05-27T07:39:11.610 INFO:journalctl@ceph.osd.11.smithi173.stdout:May 27 07:39:11 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-osd-11[63021]: 2024-05-27T07:39:11.249+0000 7f9d44ac8700 -1 osd.11 134 heartbeat_check: no reply from 172.21.15.173:6812 osd.5 since back 2024-05-27T07:38:41.035302+0000 front 2024-05-27T07:38:41.035361+0000 (oldest deadline 2024-05-27T07:39:02.036045+0000) 2024-05-27T07:39:11.610 INFO:journalctl@ceph.osd.11.smithi173.stdout:May 27 07:39:11 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-osd-11[63021]: 2024-05-27T07:39:11.249+0000 7f9d44ac8700 -1 osd.11 134 heartbeat_check: no reply from 172.21.15.52:6822 osd.6 since back 2024-05-27T07:38:47.237679+0000 front 2024-05-27T07:38:47.237750+0000 (oldest deadline 2024-05-27T07:39:10.137647+0000) 2024-05-27T07:39:11.947 INFO:journalctl@ceph.osd.10.smithi105.stdout:May 27 07:39:11 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-osd-10[60334]: 2024-05-27T07:39:11.581+0000 7f162fb8f700 -1 osd.10 134 heartbeat_check: no reply from 172.21.15.52:6806 osd.0 since back 2024-05-27T07:37:56.754985+0000 front 2024-05-27T07:37:56.755070+0000 (oldest deadline 2024-05-27T07:38:22.054911+0000) 2024-05-27T07:39:11.947 INFO:journalctl@ceph.osd.10.smithi105.stdout:May 27 07:39:11 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-osd-10[60334]: 2024-05-27T07:39:11.581+0000 7f162fb8f700 -1 osd.10 134 heartbeat_check: no reply from 172.21.15.105:6804 osd.1 since back 2024-05-27T07:38:05.556137+0000 front 2024-05-27T07:38:05.556112+0000 (oldest deadline 2024-05-27T07:38:31.456213+0000) 2024-05-27T07:39:11.947 INFO:journalctl@ceph.osd.10.smithi105.stdout:May 27 07:39:11 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-osd-10[60334]: 2024-05-27T07:39:11.581+0000 7f162fb8f700 -1 osd.10 134 heartbeat_check: no reply from 172.21.15.173:6804 osd.2 since back 2024-05-27T07:38:15.557554+0000 front 2024-05-27T07:38:15.557409+0000 (oldest deadline 2024-05-27T07:38:40.857509+0000) 2024-05-27T07:39:11.947 INFO:journalctl@ceph.osd.10.smithi105.stdout:May 27 07:39:11 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-osd-10[60334]: 2024-05-27T07:39:11.581+0000 7f162fb8f700 -1 osd.10 134 heartbeat_check: no reply from 172.21.15.52:6814 osd.3 since back 2024-05-27T07:38:21.958641+0000 front 2024-05-27T07:38:21.958614+0000 (oldest deadline 2024-05-27T07:38:47.858600+0000) 2024-05-27T07:39:11.947 INFO:journalctl@ceph.osd.10.smithi105.stdout:May 27 07:39:11 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-osd-10[60334]: 2024-05-27T07:39:11.581+0000 7f162fb8f700 -1 osd.10 134 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-05-27T07:38:30.560765+0000 front 2024-05-27T07:38:30.560788+0000 (oldest deadline 2024-05-27T07:38:55.260887+0000) 2024-05-27T07:39:11.948 INFO:journalctl@ceph.osd.10.smithi105.stdout:May 27 07:39:11 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-osd-10[60334]: 2024-05-27T07:39:11.581+0000 7f162fb8f700 -1 osd.10 134 heartbeat_check: no reply from 172.21.15.173:6812 osd.5 since back 2024-05-27T07:38:36.962107+0000 front 2024-05-27T07:38:36.962140+0000 (oldest deadline 2024-05-27T07:39:04.562331+0000) 2024-05-27T07:39:11.948 INFO:journalctl@ceph.osd.10.smithi105.stdout:May 27 07:39:11 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-osd-10[60334]: 2024-05-27T07:39:11.581+0000 7f162fb8f700 -1 osd.10 134 heartbeat_check: no reply from 172.21.15.52:6822 osd.6 since back 2024-05-27T07:38:45.063573+0000 front 2024-05-27T07:38:45.063539+0000 (oldest deadline 2024-05-27T07:39:10.963318+0000) 2024-05-27T07:39:12.238 INFO:journalctl@ceph.osd.9.smithi052.stdout:May 27 07:39:11 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-osd-9[67134]: 2024-05-27T07:39:11.827+0000 7f1e3e2ca700 -1 osd.9 134 heartbeat_check: no reply from 172.21.15.52:6806 osd.0 since back 2024-05-27T07:37:58.861291+0000 front 2024-05-27T07:37:58.861258+0000 (oldest deadline 2024-05-27T07:38:22.361571+0000) 2024-05-27T07:39:12.238 INFO:journalctl@ceph.osd.9.smithi052.stdout:May 27 07:39:11 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-osd-9[67134]: 2024-05-27T07:39:11.827+0000 7f1e3e2ca700 -1 osd.9 134 heartbeat_check: no reply from 172.21.15.105:6804 osd.1 since back 2024-05-27T07:38:05.862960+0000 front 2024-05-27T07:38:05.862645+0000 (oldest deadline 2024-05-27T07:38:30.562694+0000) 2024-05-27T07:39:12.238 INFO:journalctl@ceph.osd.9.smithi052.stdout:May 27 07:39:11 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-osd-9[67134]: 2024-05-27T07:39:11.827+0000 7f1e3e2ca700 -1 osd.9 134 heartbeat_check: no reply from 172.21.15.173:6804 osd.2 since back 2024-05-27T07:38:15.566595+0000 front 2024-05-27T07:38:15.566403+0000 (oldest deadline 2024-05-27T07:38:37.866552+0000) 2024-05-27T07:39:12.238 INFO:journalctl@ceph.osd.9.smithi052.stdout:May 27 07:39:11 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-osd-9[67134]: 2024-05-27T07:39:11.827+0000 7f1e3e2ca700 -1 osd.9 134 heartbeat_check: no reply from 172.21.15.52:6814 osd.3 since back 2024-05-27T07:38:22.468720+0000 front 2024-05-27T07:38:22.468751+0000 (oldest deadline 2024-05-27T07:38:48.269806+0000) 2024-05-27T07:39:12.238 INFO:journalctl@ceph.osd.9.smithi052.stdout:May 27 07:39:11 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-osd-9[67134]: 2024-05-27T07:39:11.827+0000 7f1e3e2ca700 -1 osd.9 134 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-05-27T07:38:31.171291+0000 front 2024-05-27T07:38:31.171372+0000 (oldest deadline 2024-05-27T07:38:56.971580+0000) 2024-05-27T07:39:12.238 INFO:journalctl@ceph.osd.9.smithi052.stdout:May 27 07:39:11 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-osd-9[67134]: 2024-05-27T07:39:11.827+0000 7f1e3e2ca700 -1 osd.9 134 heartbeat_check: no reply from 172.21.15.173:6812 osd.5 since back 2024-05-27T07:38:36.972252+0000 front 2024-05-27T07:38:36.972432+0000 (oldest deadline 2024-05-27T07:39:02.872102+0000) 2024-05-27T07:39:12.609 INFO:journalctl@ceph.osd.11.smithi173.stdout:May 27 07:39:12 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-osd-11[63021]: 2024-05-27T07:39:12.256+0000 7f9d44ac8700 -1 osd.11 134 heartbeat_check: no reply from 172.21.15.52:6806 osd.0 since back 2024-05-27T07:37:57.927796+0000 front 2024-05-27T07:37:57.927759+0000 (oldest deadline 2024-05-27T07:38:22.028025+0000) 2024-05-27T07:39:12.610 INFO:journalctl@ceph.osd.11.smithi173.stdout:May 27 07:39:12 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-osd-11[63021]: 2024-05-27T07:39:12.256+0000 7f9d44ac8700 -1 osd.11 134 heartbeat_check: no reply from 172.21.15.105:6804 osd.1 since back 2024-05-27T07:38:03.129257+0000 front 2024-05-27T07:38:03.129120+0000 (oldest deadline 2024-05-27T07:38:28.929543+0000) 2024-05-27T07:39:12.610 INFO:journalctl@ceph.osd.11.smithi173.stdout:May 27 07:39:12 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-osd-11[63021]: 2024-05-27T07:39:12.256+0000 7f9d44ac8700 -1 osd.11 134 heartbeat_check: no reply from 172.21.15.173:6804 osd.2 since back 2024-05-27T07:38:16.431856+0000 front 2024-05-27T07:38:16.431878+0000 (oldest deadline 2024-05-27T07:38:42.332069+0000) 2024-05-27T07:39:12.610 INFO:journalctl@ceph.osd.11.smithi173.stdout:May 27 07:39:12 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-osd-11[63021]: 2024-05-27T07:39:12.256+0000 7f9d44ac8700 -1 osd.11 134 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-05-27T07:38:28.133691+0000 front 2024-05-27T07:38:28.134117+0000 (oldest deadline 2024-05-27T07:38:55.734329+0000) 2024-05-27T07:39:12.610 INFO:journalctl@ceph.osd.11.smithi173.stdout:May 27 07:39:12 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-osd-11[63021]: 2024-05-27T07:39:12.256+0000 7f9d44ac8700 -1 osd.11 134 heartbeat_check: no reply from 172.21.15.173:6812 osd.5 since back 2024-05-27T07:38:41.035302+0000 front 2024-05-27T07:38:41.035361+0000 (oldest deadline 2024-05-27T07:39:02.036045+0000) 2024-05-27T07:39:12.610 INFO:journalctl@ceph.osd.11.smithi173.stdout:May 27 07:39:12 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-osd-11[63021]: 2024-05-27T07:39:12.256+0000 7f9d44ac8700 -1 osd.11 134 heartbeat_check: no reply from 172.21.15.52:6822 osd.6 since back 2024-05-27T07:38:47.237679+0000 front 2024-05-27T07:38:47.237750+0000 (oldest deadline 2024-05-27T07:39:10.137647+0000) 2024-05-27T07:39:12.947 INFO:journalctl@ceph.osd.10.smithi105.stdout:May 27 07:39:12 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-osd-10[60334]: 2024-05-27T07:39:12.605+0000 7f162fb8f700 -1 osd.10 134 heartbeat_check: no reply from 172.21.15.52:6806 osd.0 since back 2024-05-27T07:37:56.754985+0000 front 2024-05-27T07:37:56.755070+0000 (oldest deadline 2024-05-27T07:38:22.054911+0000) 2024-05-27T07:39:12.947 INFO:journalctl@ceph.osd.10.smithi105.stdout:May 27 07:39:12 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-osd-10[60334]: 2024-05-27T07:39:12.605+0000 7f162fb8f700 -1 osd.10 134 heartbeat_check: no reply from 172.21.15.105:6804 osd.1 since back 2024-05-27T07:38:05.556137+0000 front 2024-05-27T07:38:05.556112+0000 (oldest deadline 2024-05-27T07:38:31.456213+0000) 2024-05-27T07:39:12.947 INFO:journalctl@ceph.osd.10.smithi105.stdout:May 27 07:39:12 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-osd-10[60334]: 2024-05-27T07:39:12.605+0000 7f162fb8f700 -1 osd.10 134 heartbeat_check: no reply from 172.21.15.173:6804 osd.2 since back 2024-05-27T07:38:15.557554+0000 front 2024-05-27T07:38:15.557409+0000 (oldest deadline 2024-05-27T07:38:40.857509+0000) 2024-05-27T07:39:12.947 INFO:journalctl@ceph.osd.10.smithi105.stdout:May 27 07:39:12 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-osd-10[60334]: 2024-05-27T07:39:12.605+0000 7f162fb8f700 -1 osd.10 134 heartbeat_check: no reply from 172.21.15.52:6814 osd.3 since back 2024-05-27T07:38:21.958641+0000 front 2024-05-27T07:38:21.958614+0000 (oldest deadline 2024-05-27T07:38:47.858600+0000) 2024-05-27T07:39:12.947 INFO:journalctl@ceph.osd.10.smithi105.stdout:May 27 07:39:12 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-osd-10[60334]: 2024-05-27T07:39:12.605+0000 7f162fb8f700 -1 osd.10 134 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-05-27T07:38:30.560765+0000 front 2024-05-27T07:38:30.560788+0000 (oldest deadline 2024-05-27T07:38:55.260887+0000) 2024-05-27T07:39:12.947 INFO:journalctl@ceph.osd.10.smithi105.stdout:May 27 07:39:12 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-osd-10[60334]: 2024-05-27T07:39:12.605+0000 7f162fb8f700 -1 osd.10 134 heartbeat_check: no reply from 172.21.15.173:6812 osd.5 since back 2024-05-27T07:38:36.962107+0000 front 2024-05-27T07:38:36.962140+0000 (oldest deadline 2024-05-27T07:39:04.562331+0000) 2024-05-27T07:39:12.948 INFO:journalctl@ceph.osd.10.smithi105.stdout:May 27 07:39:12 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-osd-10[60334]: 2024-05-27T07:39:12.605+0000 7f162fb8f700 -1 osd.10 134 heartbeat_check: no reply from 172.21.15.52:6822 osd.6 since back 2024-05-27T07:38:45.063573+0000 front 2024-05-27T07:38:45.063539+0000 (oldest deadline 2024-05-27T07:39:10.963318+0000) 2024-05-27T07:39:13.236 INFO:journalctl@ceph.osd.9.smithi052.stdout:May 27 07:39:12 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-osd-9[67134]: 2024-05-27T07:39:12.812+0000 7f1e3e2ca700 -1 osd.9 134 heartbeat_check: no reply from 172.21.15.52:6806 osd.0 since back 2024-05-27T07:37:58.861291+0000 front 2024-05-27T07:37:58.861258+0000 (oldest deadline 2024-05-27T07:38:22.361571+0000) 2024-05-27T07:39:13.236 INFO:journalctl@ceph.osd.9.smithi052.stdout:May 27 07:39:12 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-osd-9[67134]: 2024-05-27T07:39:12.812+0000 7f1e3e2ca700 -1 osd.9 134 heartbeat_check: no reply from 172.21.15.105:6804 osd.1 since back 2024-05-27T07:38:05.862960+0000 front 2024-05-27T07:38:05.862645+0000 (oldest deadline 2024-05-27T07:38:30.562694+0000) 2024-05-27T07:39:13.237 INFO:journalctl@ceph.osd.9.smithi052.stdout:May 27 07:39:12 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-osd-9[67134]: 2024-05-27T07:39:12.812+0000 7f1e3e2ca700 -1 osd.9 134 heartbeat_check: no reply from 172.21.15.173:6804 osd.2 since back 2024-05-27T07:38:15.566595+0000 front 2024-05-27T07:38:15.566403+0000 (oldest deadline 2024-05-27T07:38:37.866552+0000) 2024-05-27T07:39:13.237 INFO:journalctl@ceph.osd.9.smithi052.stdout:May 27 07:39:12 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-osd-9[67134]: 2024-05-27T07:39:12.812+0000 7f1e3e2ca700 -1 osd.9 134 heartbeat_check: no reply from 172.21.15.52:6814 osd.3 since back 2024-05-27T07:38:22.468720+0000 front 2024-05-27T07:38:22.468751+0000 (oldest deadline 2024-05-27T07:38:48.269806+0000) 2024-05-27T07:39:13.237 INFO:journalctl@ceph.osd.9.smithi052.stdout:May 27 07:39:12 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-osd-9[67134]: 2024-05-27T07:39:12.812+0000 7f1e3e2ca700 -1 osd.9 134 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-05-27T07:38:31.171291+0000 front 2024-05-27T07:38:31.171372+0000 (oldest deadline 2024-05-27T07:38:56.971580+0000) 2024-05-27T07:39:13.237 INFO:journalctl@ceph.osd.9.smithi052.stdout:May 27 07:39:12 smithi052 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-osd-9[67134]: 2024-05-27T07:39:12.812+0000 7f1e3e2ca700 -1 osd.9 134 heartbeat_check: no reply from 172.21.15.173:6812 osd.5 since back 2024-05-27T07:38:36.972252+0000 front 2024-05-27T07:38:36.972432+0000 (oldest deadline 2024-05-27T07:39:02.872102+0000) 2024-05-27T07:39:13.609 INFO:journalctl@ceph.osd.11.smithi173.stdout:May 27 07:39:13 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-osd-11[63021]: 2024-05-27T07:39:13.277+0000 7f9d44ac8700 -1 osd.11 134 heartbeat_check: no reply from 172.21.15.52:6806 osd.0 since back 2024-05-27T07:37:57.927796+0000 front 2024-05-27T07:37:57.927759+0000 (oldest deadline 2024-05-27T07:38:22.028025+0000) 2024-05-27T07:39:13.610 INFO:journalctl@ceph.osd.11.smithi173.stdout:May 27 07:39:13 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-osd-11[63021]: 2024-05-27T07:39:13.277+0000 7f9d44ac8700 -1 osd.11 134 heartbeat_check: no reply from 172.21.15.105:6804 osd.1 since back 2024-05-27T07:38:03.129257+0000 front 2024-05-27T07:38:03.129120+0000 (oldest deadline 2024-05-27T07:38:28.929543+0000) 2024-05-27T07:39:13.610 INFO:journalctl@ceph.osd.11.smithi173.stdout:May 27 07:39:13 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-osd-11[63021]: 2024-05-27T07:39:13.277+0000 7f9d44ac8700 -1 osd.11 134 heartbeat_check: no reply from 172.21.15.173:6804 osd.2 since back 2024-05-27T07:38:16.431856+0000 front 2024-05-27T07:38:16.431878+0000 (oldest deadline 2024-05-27T07:38:42.332069+0000) 2024-05-27T07:39:13.610 INFO:journalctl@ceph.osd.11.smithi173.stdout:May 27 07:39:13 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-osd-11[63021]: 2024-05-27T07:39:13.277+0000 7f9d44ac8700 -1 osd.11 134 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-05-27T07:38:28.133691+0000 front 2024-05-27T07:38:28.134117+0000 (oldest deadline 2024-05-27T07:38:55.734329+0000) 2024-05-27T07:39:13.610 INFO:journalctl@ceph.osd.11.smithi173.stdout:May 27 07:39:13 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-osd-11[63021]: 2024-05-27T07:39:13.277+0000 7f9d44ac8700 -1 osd.11 134 heartbeat_check: no reply from 172.21.15.173:6812 osd.5 since back 2024-05-27T07:38:41.035302+0000 front 2024-05-27T07:38:41.035361+0000 (oldest deadline 2024-05-27T07:39:02.036045+0000) 2024-05-27T07:39:13.610 INFO:journalctl@ceph.osd.11.smithi173.stdout:May 27 07:39:13 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-osd-11[63021]: 2024-05-27T07:39:13.277+0000 7f9d44ac8700 -1 osd.11 134 heartbeat_check: no reply from 172.21.15.52:6822 osd.6 since back 2024-05-27T07:38:47.237679+0000 front 2024-05-27T07:38:47.237750+0000 (oldest deadline 2024-05-27T07:39:10.137647+0000) 2024-05-27T07:39:13.947 INFO:journalctl@ceph.osd.10.smithi105.stdout:May 27 07:39:13 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-osd-10[60334]: 2024-05-27T07:39:13.626+0000 7f162fb8f700 -1 osd.10 134 heartbeat_check: no reply from 172.21.15.52:6806 osd.0 since back 2024-05-27T07:37:56.754985+0000 front 2024-05-27T07:37:56.755070+0000 (oldest deadline 2024-05-27T07:38:22.054911+0000) 2024-05-27T07:39:13.947 INFO:journalctl@ceph.osd.10.smithi105.stdout:May 27 07:39:13 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-osd-10[60334]: 2024-05-27T07:39:13.626+0000 7f162fb8f700 -1 osd.10 134 heartbeat_check: no reply from 172.21.15.105:6804 osd.1 since back 2024-05-27T07:38:05.556137+0000 front 2024-05-27T07:38:05.556112+0000 (oldest deadline 2024-05-27T07:38:31.456213+0000) 2024-05-27T07:39:13.947 INFO:journalctl@ceph.osd.10.smithi105.stdout:May 27 07:39:13 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-osd-10[60334]: 2024-05-27T07:39:13.626+0000 7f162fb8f700 -1 osd.10 134 heartbeat_check: no reply from 172.21.15.173:6804 osd.2 since back 2024-05-27T07:38:15.557554+0000 front 2024-05-27T07:38:15.557409+0000 (oldest deadline 2024-05-27T07:38:40.857509+0000) 2024-05-27T07:39:13.948 INFO:journalctl@ceph.osd.10.smithi105.stdout:May 27 07:39:13 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-osd-10[60334]: 2024-05-27T07:39:13.626+0000 7f162fb8f700 -1 osd.10 134 heartbeat_check: no reply from 172.21.15.52:6814 osd.3 since back 2024-05-27T07:38:21.958641+0000 front 2024-05-27T07:38:21.958614+0000 (oldest deadline 2024-05-27T07:38:47.858600+0000) 2024-05-27T07:39:13.948 INFO:journalctl@ceph.osd.10.smithi105.stdout:May 27 07:39:13 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-osd-10[60334]: 2024-05-27T07:39:13.626+0000 7f162fb8f700 -1 osd.10 134 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-05-27T07:38:30.560765+0000 front 2024-05-27T07:38:30.560788+0000 (oldest deadline 2024-05-27T07:38:55.260887+0000) 2024-05-27T07:39:13.948 INFO:journalctl@ceph.osd.10.smithi105.stdout:May 27 07:39:13 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-osd-10[60334]: 2024-05-27T07:39:13.626+0000 7f162fb8f700 -1 osd.10 134 heartbeat_check: no reply from 172.21.15.173:6812 osd.5 since back 2024-05-27T07:38:36.962107+0000 front 2024-05-27T07:38:36.962140+0000 (oldest deadline 2024-05-27T07:39:04.562331+0000) 2024-05-27T07:39:13.948 INFO:journalctl@ceph.osd.10.smithi105.stdout:May 27 07:39:13 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-osd-10[60334]: 2024-05-27T07:39:13.626+0000 7f162fb8f700 -1 osd.10 134 heartbeat_check: no reply from 172.21.15.52:6822 osd.6 since back 2024-05-27T07:38:45.063573+0000 front 2024-05-27T07:38:45.063539+0000 (oldest deadline 2024-05-27T07:39:10.963318+0000) 2024-05-27T07:39:14.573 INFO:journalctl@ceph.osd.9.smithi052.stdout:May 27 07:39:14 smithi052 podman[119685]: 2024-05-27 07:39:14.316300009 +0000 UTC m=+5.612565424 container died ac0d99cb0d4c9e4d92c94ccc32f94db32300e92917e928d79c3d8bb0c500bd49 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-osd-9, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, org.label-schema.build-date=20240521, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, ceph=True, org.label-schema.vendor=CentOS) 2024-05-27T07:39:14.609 INFO:journalctl@ceph.osd.11.smithi173.stdout:May 27 07:39:14 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-osd-11[63021]: 2024-05-27T07:39:14.276+0000 7f9d44ac8700 -1 osd.11 134 heartbeat_check: no reply from 172.21.15.52:6806 osd.0 since back 2024-05-27T07:37:57.927796+0000 front 2024-05-27T07:37:57.927759+0000 (oldest deadline 2024-05-27T07:38:22.028025+0000) 2024-05-27T07:39:14.609 INFO:journalctl@ceph.osd.11.smithi173.stdout:May 27 07:39:14 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-osd-11[63021]: 2024-05-27T07:39:14.276+0000 7f9d44ac8700 -1 osd.11 134 heartbeat_check: no reply from 172.21.15.105:6804 osd.1 since back 2024-05-27T07:38:03.129257+0000 front 2024-05-27T07:38:03.129120+0000 (oldest deadline 2024-05-27T07:38:28.929543+0000) 2024-05-27T07:39:14.609 INFO:journalctl@ceph.osd.11.smithi173.stdout:May 27 07:39:14 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-osd-11[63021]: 2024-05-27T07:39:14.276+0000 7f9d44ac8700 -1 osd.11 134 heartbeat_check: no reply from 172.21.15.173:6804 osd.2 since back 2024-05-27T07:38:16.431856+0000 front 2024-05-27T07:38:16.431878+0000 (oldest deadline 2024-05-27T07:38:42.332069+0000) 2024-05-27T07:39:14.609 INFO:journalctl@ceph.osd.11.smithi173.stdout:May 27 07:39:14 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-osd-11[63021]: 2024-05-27T07:39:14.276+0000 7f9d44ac8700 -1 osd.11 134 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-05-27T07:38:28.133691+0000 front 2024-05-27T07:38:28.134117+0000 (oldest deadline 2024-05-27T07:38:55.734329+0000) 2024-05-27T07:39:14.610 INFO:journalctl@ceph.osd.11.smithi173.stdout:May 27 07:39:14 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-osd-11[63021]: 2024-05-27T07:39:14.276+0000 7f9d44ac8700 -1 osd.11 134 heartbeat_check: no reply from 172.21.15.173:6812 osd.5 since back 2024-05-27T07:38:41.035302+0000 front 2024-05-27T07:38:41.035361+0000 (oldest deadline 2024-05-27T07:39:02.036045+0000) 2024-05-27T07:39:14.610 INFO:journalctl@ceph.osd.11.smithi173.stdout:May 27 07:39:14 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-osd-11[63021]: 2024-05-27T07:39:14.276+0000 7f9d44ac8700 -1 osd.11 134 heartbeat_check: no reply from 172.21.15.52:6822 osd.6 since back 2024-05-27T07:38:47.237679+0000 front 2024-05-27T07:38:47.237750+0000 (oldest deadline 2024-05-27T07:39:10.137647+0000) 2024-05-27T07:39:14.948 INFO:journalctl@ceph.osd.10.smithi105.stdout:May 27 07:39:14 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-osd-10[60334]: 2024-05-27T07:39:14.615+0000 7f162fb8f700 -1 osd.10 134 heartbeat_check: no reply from 172.21.15.52:6806 osd.0 since back 2024-05-27T07:37:56.754985+0000 front 2024-05-27T07:37:56.755070+0000 (oldest deadline 2024-05-27T07:38:22.054911+0000) 2024-05-27T07:39:14.948 INFO:journalctl@ceph.osd.10.smithi105.stdout:May 27 07:39:14 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-osd-10[60334]: 2024-05-27T07:39:14.615+0000 7f162fb8f700 -1 osd.10 134 heartbeat_check: no reply from 172.21.15.105:6804 osd.1 since back 2024-05-27T07:38:05.556137+0000 front 2024-05-27T07:38:05.556112+0000 (oldest deadline 2024-05-27T07:38:31.456213+0000) 2024-05-27T07:39:14.948 INFO:journalctl@ceph.osd.10.smithi105.stdout:May 27 07:39:14 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-osd-10[60334]: 2024-05-27T07:39:14.615+0000 7f162fb8f700 -1 osd.10 134 heartbeat_check: no reply from 172.21.15.173:6804 osd.2 since back 2024-05-27T07:38:15.557554+0000 front 2024-05-27T07:38:15.557409+0000 (oldest deadline 2024-05-27T07:38:40.857509+0000) 2024-05-27T07:39:14.948 INFO:journalctl@ceph.osd.10.smithi105.stdout:May 27 07:39:14 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-osd-10[60334]: 2024-05-27T07:39:14.615+0000 7f162fb8f700 -1 osd.10 134 heartbeat_check: no reply from 172.21.15.52:6814 osd.3 since back 2024-05-27T07:38:21.958641+0000 front 2024-05-27T07:38:21.958614+0000 (oldest deadline 2024-05-27T07:38:47.858600+0000) 2024-05-27T07:39:14.948 INFO:journalctl@ceph.osd.10.smithi105.stdout:May 27 07:39:14 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-osd-10[60334]: 2024-05-27T07:39:14.615+0000 7f162fb8f700 -1 osd.10 134 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-05-27T07:38:30.560765+0000 front 2024-05-27T07:38:30.560788+0000 (oldest deadline 2024-05-27T07:38:55.260887+0000) 2024-05-27T07:39:14.949 INFO:journalctl@ceph.osd.10.smithi105.stdout:May 27 07:39:14 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-osd-10[60334]: 2024-05-27T07:39:14.615+0000 7f162fb8f700 -1 osd.10 134 heartbeat_check: no reply from 172.21.15.173:6812 osd.5 since back 2024-05-27T07:38:36.962107+0000 front 2024-05-27T07:38:36.962140+0000 (oldest deadline 2024-05-27T07:39:04.562331+0000) 2024-05-27T07:39:14.949 INFO:journalctl@ceph.osd.10.smithi105.stdout:May 27 07:39:14 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-osd-10[60334]: 2024-05-27T07:39:14.615+0000 7f162fb8f700 -1 osd.10 134 heartbeat_check: no reply from 172.21.15.52:6822 osd.6 since back 2024-05-27T07:38:45.063573+0000 front 2024-05-27T07:38:45.063539+0000 (oldest deadline 2024-05-27T07:39:10.963318+0000) 2024-05-27T07:39:14.987 INFO:journalctl@ceph.osd.9.smithi052.stdout:May 27 07:39:14 smithi052 podman[119685]: 2024-05-27 07:39:14.767606642 +0000 UTC m=+6.063872057 container cleanup ac0d99cb0d4c9e4d92c94ccc32f94db32300e92917e928d79c3d8bb0c500bd49 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-osd-9, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, GIT_CLEAN=True, ceph=True, io.buildah.version=1.33.5, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240521) 2024-05-27T07:39:14.987 INFO:journalctl@ceph.osd.9.smithi052.stdout:May 27 07:39:14 smithi052 bash[119685]: ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-osd-9 2024-05-27T07:39:15.335 INFO:journalctl@ceph.osd.9.smithi052.stdout:May 27 07:39:15 smithi052 podman[119700]: 2024-05-27 07:39:15.018605501 +0000 UTC m=+0.705482677 container remove ac0d99cb0d4c9e4d92c94ccc32f94db32300e92917e928d79c3d8bb0c500bd49 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-osd-9, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, GIT_CLEAN=True, ceph=True, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, org.label-schema.build-date=20240521, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS) 2024-05-27T07:39:15.609 INFO:journalctl@ceph.osd.11.smithi173.stdout:May 27 07:39:15 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-osd-11[63021]: 2024-05-27T07:39:15.234+0000 7f9d44ac8700 -1 osd.11 134 heartbeat_check: no reply from 172.21.15.52:6806 osd.0 since back 2024-05-27T07:37:57.927796+0000 front 2024-05-27T07:37:57.927759+0000 (oldest deadline 2024-05-27T07:38:22.028025+0000) 2024-05-27T07:39:15.610 INFO:journalctl@ceph.osd.11.smithi173.stdout:May 27 07:39:15 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-osd-11[63021]: 2024-05-27T07:39:15.234+0000 7f9d44ac8700 -1 osd.11 134 heartbeat_check: no reply from 172.21.15.105:6804 osd.1 since back 2024-05-27T07:38:03.129257+0000 front 2024-05-27T07:38:03.129120+0000 (oldest deadline 2024-05-27T07:38:28.929543+0000) 2024-05-27T07:39:15.610 INFO:journalctl@ceph.osd.11.smithi173.stdout:May 27 07:39:15 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-osd-11[63021]: 2024-05-27T07:39:15.234+0000 7f9d44ac8700 -1 osd.11 134 heartbeat_check: no reply from 172.21.15.173:6804 osd.2 since back 2024-05-27T07:38:16.431856+0000 front 2024-05-27T07:38:16.431878+0000 (oldest deadline 2024-05-27T07:38:42.332069+0000) 2024-05-27T07:39:15.610 INFO:journalctl@ceph.osd.11.smithi173.stdout:May 27 07:39:15 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-osd-11[63021]: 2024-05-27T07:39:15.234+0000 7f9d44ac8700 -1 osd.11 134 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-05-27T07:38:28.133691+0000 front 2024-05-27T07:38:28.134117+0000 (oldest deadline 2024-05-27T07:38:55.734329+0000) 2024-05-27T07:39:15.610 INFO:journalctl@ceph.osd.11.smithi173.stdout:May 27 07:39:15 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-osd-11[63021]: 2024-05-27T07:39:15.234+0000 7f9d44ac8700 -1 osd.11 134 heartbeat_check: no reply from 172.21.15.173:6812 osd.5 since back 2024-05-27T07:38:41.035302+0000 front 2024-05-27T07:38:41.035361+0000 (oldest deadline 2024-05-27T07:39:02.036045+0000) 2024-05-27T07:39:15.610 INFO:journalctl@ceph.osd.11.smithi173.stdout:May 27 07:39:15 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-osd-11[63021]: 2024-05-27T07:39:15.234+0000 7f9d44ac8700 -1 osd.11 134 heartbeat_check: no reply from 172.21.15.52:6822 osd.6 since back 2024-05-27T07:38:47.237679+0000 front 2024-05-27T07:38:47.237750+0000 (oldest deadline 2024-05-27T07:39:10.137647+0000) 2024-05-27T07:39:15.644 INFO:journalctl@ceph.osd.9.smithi052.stdout:May 27 07:39:15 smithi052 podman[119791]: 2024-05-27 07:39:15.241421817 +0000 UTC m=+0.017403074 image pull e36ea4fc8a5807e8d2701370e350a0de3856538797f7fba64cfce7e344bd6799 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6 2024-05-27T07:39:15.644 INFO:journalctl@ceph.osd.9.smithi052.stdout:May 27 07:39:15 smithi052 podman[119791]: 2024-05-27 07:39:15.344884216 +0000 UTC m=+0.120865484 container create 4ed344270c955c8f78c58f459907af3897d4412d8a4201968575b276b80e3bd7 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-osd-9-deactivate, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.build-date=20240521, org.label-schema.license=GPLv2, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, GIT_CLEAN=True, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, ceph=True) 2024-05-27T07:39:15.644 INFO:journalctl@ceph.osd.9.smithi052.stdout:May 27 07:39:15 smithi052 podman[119791]: 2024-05-27 07:39:15.572940418 +0000 UTC m=+0.348921716 container init 4ed344270c955c8f78c58f459907af3897d4412d8a4201968575b276b80e3bd7 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-osd-9-deactivate, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240521, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , GIT_CLEAN=True) 2024-05-27T07:39:15.644 INFO:journalctl@ceph.osd.9.smithi052.stdout:May 27 07:39:15 smithi052 podman[119791]: 2024-05-27 07:39:15.576628728 +0000 UTC m=+0.352609983 container start 4ed344270c955c8f78c58f459907af3897d4412d8a4201968575b276b80e3bd7 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-osd-9-deactivate, org.label-schema.vendor=CentOS, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, org.label-schema.license=GPLv2, ceph=True, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, org.label-schema.build-date=20240521, org.label-schema.name=CentOS Stream 8 Base Image) 2024-05-27T07:39:15.644 INFO:journalctl@ceph.osd.9.smithi052.stdout:May 27 07:39:15 smithi052 podman[119791]: 2024-05-27 07:39:15.647745089 +0000 UTC m=+0.423726358 container attach 4ed344270c955c8f78c58f459907af3897d4412d8a4201968575b276b80e3bd7 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-osd-9-deactivate, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, org.label-schema.build-date=20240521, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , GIT_CLEAN=True, ceph=True, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, org.label-schema.vendor=CentOS) 2024-05-27T07:39:15.947 INFO:journalctl@ceph.osd.10.smithi105.stdout:May 27 07:39:15 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-osd-10[60334]: 2024-05-27T07:39:15.617+0000 7f162fb8f700 -1 osd.10 134 heartbeat_check: no reply from 172.21.15.52:6806 osd.0 since back 2024-05-27T07:37:56.754985+0000 front 2024-05-27T07:37:56.755070+0000 (oldest deadline 2024-05-27T07:38:22.054911+0000) 2024-05-27T07:39:15.947 INFO:journalctl@ceph.osd.10.smithi105.stdout:May 27 07:39:15 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-osd-10[60334]: 2024-05-27T07:39:15.617+0000 7f162fb8f700 -1 osd.10 134 heartbeat_check: no reply from 172.21.15.105:6804 osd.1 since back 2024-05-27T07:38:05.556137+0000 front 2024-05-27T07:38:05.556112+0000 (oldest deadline 2024-05-27T07:38:31.456213+0000) 2024-05-27T07:39:15.947 INFO:journalctl@ceph.osd.10.smithi105.stdout:May 27 07:39:15 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-osd-10[60334]: 2024-05-27T07:39:15.617+0000 7f162fb8f700 -1 osd.10 134 heartbeat_check: no reply from 172.21.15.173:6804 osd.2 since back 2024-05-27T07:38:15.557554+0000 front 2024-05-27T07:38:15.557409+0000 (oldest deadline 2024-05-27T07:38:40.857509+0000) 2024-05-27T07:39:15.947 INFO:journalctl@ceph.osd.10.smithi105.stdout:May 27 07:39:15 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-osd-10[60334]: 2024-05-27T07:39:15.617+0000 7f162fb8f700 -1 osd.10 134 heartbeat_check: no reply from 172.21.15.52:6814 osd.3 since back 2024-05-27T07:38:21.958641+0000 front 2024-05-27T07:38:21.958614+0000 (oldest deadline 2024-05-27T07:38:47.858600+0000) 2024-05-27T07:39:15.947 INFO:journalctl@ceph.osd.10.smithi105.stdout:May 27 07:39:15 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-osd-10[60334]: 2024-05-27T07:39:15.617+0000 7f162fb8f700 -1 osd.10 134 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-05-27T07:38:30.560765+0000 front 2024-05-27T07:38:30.560788+0000 (oldest deadline 2024-05-27T07:38:55.260887+0000) 2024-05-27T07:39:15.947 INFO:journalctl@ceph.osd.10.smithi105.stdout:May 27 07:39:15 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-osd-10[60334]: 2024-05-27T07:39:15.617+0000 7f162fb8f700 -1 osd.10 134 heartbeat_check: no reply from 172.21.15.173:6812 osd.5 since back 2024-05-27T07:38:36.962107+0000 front 2024-05-27T07:38:36.962140+0000 (oldest deadline 2024-05-27T07:39:04.562331+0000) 2024-05-27T07:39:15.948 INFO:journalctl@ceph.osd.10.smithi105.stdout:May 27 07:39:15 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-osd-10[60334]: 2024-05-27T07:39:15.617+0000 7f162fb8f700 -1 osd.10 134 heartbeat_check: no reply from 172.21.15.52:6822 osd.6 since back 2024-05-27T07:38:45.063573+0000 front 2024-05-27T07:38:45.063539+0000 (oldest deadline 2024-05-27T07:39:10.963318+0000) 2024-05-27T07:39:15.986 INFO:journalctl@ceph.osd.9.smithi052.stdout:May 27 07:39:15 smithi052 podman[119791]: 2024-05-27 07:39:15.878818358 +0000 UTC m=+0.654799618 container died 4ed344270c955c8f78c58f459907af3897d4412d8a4201968575b276b80e3bd7 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-osd-9-deactivate, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, GIT_CLEAN=True, org.label-schema.build-date=20240521, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, CEPH_POINT_RELEASE=, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, org.label-schema.schema-version=1.0, ceph=True, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image) 2024-05-27T07:39:16.412 INFO:journalctl@ceph.osd.9.smithi052.stdout:May 27 07:39:16 smithi052 podman[119817]: 2024-05-27 07:39:16.401085735 +0000 UTC m=+0.516545552 container remove 4ed344270c955c8f78c58f459907af3897d4412d8a4201968575b276b80e3bd7 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-osd-9-deactivate, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, ceph=True, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, GIT_CLEAN=True, CEPH_POINT_RELEASE=, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240521, org.label-schema.license=GPLv2) 2024-05-27T07:39:16.417 DEBUG:teuthology.orchestra.run.smithi052:> sudo pkill -f 'journalctl -f -n 0 -u ceph-73df2636-1bf8-11ef-bc9b-c7b262605968@osd.9.service' 2024-05-27T07:39:16.458 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-05-27T07:39:16.458 INFO:tasks.cephadm.osd.9:Stopped osd.9 2024-05-27T07:39:16.458 INFO:tasks.cephadm.osd.10:Stopping osd.10... 2024-05-27T07:39:16.458 DEBUG:teuthology.orchestra.run.smithi105:> sudo systemctl stop ceph-73df2636-1bf8-11ef-bc9b-c7b262605968@osd.10 2024-05-27T07:39:16.609 INFO:journalctl@ceph.osd.11.smithi173.stdout:May 27 07:39:16 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-osd-11[63021]: 2024-05-27T07:39:16.274+0000 7f9d44ac8700 -1 osd.11 134 heartbeat_check: no reply from 172.21.15.52:6806 osd.0 since back 2024-05-27T07:37:57.927796+0000 front 2024-05-27T07:37:57.927759+0000 (oldest deadline 2024-05-27T07:38:22.028025+0000) 2024-05-27T07:39:16.609 INFO:journalctl@ceph.osd.11.smithi173.stdout:May 27 07:39:16 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-osd-11[63021]: 2024-05-27T07:39:16.274+0000 7f9d44ac8700 -1 osd.11 134 heartbeat_check: no reply from 172.21.15.105:6804 osd.1 since back 2024-05-27T07:38:03.129257+0000 front 2024-05-27T07:38:03.129120+0000 (oldest deadline 2024-05-27T07:38:28.929543+0000) 2024-05-27T07:39:16.610 INFO:journalctl@ceph.osd.11.smithi173.stdout:May 27 07:39:16 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-osd-11[63021]: 2024-05-27T07:39:16.274+0000 7f9d44ac8700 -1 osd.11 134 heartbeat_check: no reply from 172.21.15.173:6804 osd.2 since back 2024-05-27T07:38:16.431856+0000 front 2024-05-27T07:38:16.431878+0000 (oldest deadline 2024-05-27T07:38:42.332069+0000) 2024-05-27T07:39:16.610 INFO:journalctl@ceph.osd.11.smithi173.stdout:May 27 07:39:16 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-osd-11[63021]: 2024-05-27T07:39:16.274+0000 7f9d44ac8700 -1 osd.11 134 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-05-27T07:38:28.133691+0000 front 2024-05-27T07:38:28.134117+0000 (oldest deadline 2024-05-27T07:38:55.734329+0000) 2024-05-27T07:39:16.610 INFO:journalctl@ceph.osd.11.smithi173.stdout:May 27 07:39:16 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-osd-11[63021]: 2024-05-27T07:39:16.274+0000 7f9d44ac8700 -1 osd.11 134 heartbeat_check: no reply from 172.21.15.173:6812 osd.5 since back 2024-05-27T07:38:41.035302+0000 front 2024-05-27T07:38:41.035361+0000 (oldest deadline 2024-05-27T07:39:02.036045+0000) 2024-05-27T07:39:16.610 INFO:journalctl@ceph.osd.11.smithi173.stdout:May 27 07:39:16 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-osd-11[63021]: 2024-05-27T07:39:16.274+0000 7f9d44ac8700 -1 osd.11 134 heartbeat_check: no reply from 172.21.15.52:6822 osd.6 since back 2024-05-27T07:38:47.237679+0000 front 2024-05-27T07:38:47.237750+0000 (oldest deadline 2024-05-27T07:39:10.137647+0000) 2024-05-27T07:39:16.806 INFO:journalctl@ceph.osd.10.smithi105.stdout:May 27 07:39:16 smithi105 systemd[1]: Stopping Ceph osd.10 for 73df2636-1bf8-11ef-bc9b-c7b262605968... 2024-05-27T07:39:16.806 INFO:journalctl@ceph.osd.10.smithi105.stdout:May 27 07:39:16 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-osd-10[60334]: 2024-05-27T07:39:16.601+0000 7f162fb8f700 -1 osd.10 134 heartbeat_check: no reply from 172.21.15.52:6806 osd.0 since back 2024-05-27T07:37:56.754985+0000 front 2024-05-27T07:37:56.755070+0000 (oldest deadline 2024-05-27T07:38:22.054911+0000) 2024-05-27T07:39:16.806 INFO:journalctl@ceph.osd.10.smithi105.stdout:May 27 07:39:16 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-osd-10[60334]: 2024-05-27T07:39:16.601+0000 7f162fb8f700 -1 osd.10 134 heartbeat_check: no reply from 172.21.15.105:6804 osd.1 since back 2024-05-27T07:38:05.556137+0000 front 2024-05-27T07:38:05.556112+0000 (oldest deadline 2024-05-27T07:38:31.456213+0000) 2024-05-27T07:39:16.806 INFO:journalctl@ceph.osd.10.smithi105.stdout:May 27 07:39:16 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-osd-10[60334]: 2024-05-27T07:39:16.601+0000 7f162fb8f700 -1 osd.10 134 heartbeat_check: no reply from 172.21.15.173:6804 osd.2 since back 2024-05-27T07:38:15.557554+0000 front 2024-05-27T07:38:15.557409+0000 (oldest deadline 2024-05-27T07:38:40.857509+0000) 2024-05-27T07:39:16.806 INFO:journalctl@ceph.osd.10.smithi105.stdout:May 27 07:39:16 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-osd-10[60334]: 2024-05-27T07:39:16.601+0000 7f162fb8f700 -1 osd.10 134 heartbeat_check: no reply from 172.21.15.52:6814 osd.3 since back 2024-05-27T07:38:21.958641+0000 front 2024-05-27T07:38:21.958614+0000 (oldest deadline 2024-05-27T07:38:47.858600+0000) 2024-05-27T07:39:16.806 INFO:journalctl@ceph.osd.10.smithi105.stdout:May 27 07:39:16 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-osd-10[60334]: 2024-05-27T07:39:16.601+0000 7f162fb8f700 -1 osd.10 134 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-05-27T07:38:30.560765+0000 front 2024-05-27T07:38:30.560788+0000 (oldest deadline 2024-05-27T07:38:55.260887+0000) 2024-05-27T07:39:16.806 INFO:journalctl@ceph.osd.10.smithi105.stdout:May 27 07:39:16 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-osd-10[60334]: 2024-05-27T07:39:16.601+0000 7f162fb8f700 -1 osd.10 134 heartbeat_check: no reply from 172.21.15.173:6812 osd.5 since back 2024-05-27T07:38:36.962107+0000 front 2024-05-27T07:38:36.962140+0000 (oldest deadline 2024-05-27T07:39:04.562331+0000) 2024-05-27T07:39:16.806 INFO:journalctl@ceph.osd.10.smithi105.stdout:May 27 07:39:16 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-osd-10[60334]: 2024-05-27T07:39:16.601+0000 7f162fb8f700 -1 osd.10 134 heartbeat_check: no reply from 172.21.15.52:6822 osd.6 since back 2024-05-27T07:38:45.063573+0000 front 2024-05-27T07:38:45.063539+0000 (oldest deadline 2024-05-27T07:39:10.963318+0000) 2024-05-27T07:39:17.197 INFO:journalctl@ceph.osd.10.smithi105.stdout:May 27 07:39:16 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-osd-10[60334]: 2024-05-27T07:39:16.808+0000 7f16335d9700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.10 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-05-27T07:39:17.197 INFO:journalctl@ceph.osd.10.smithi105.stdout:May 27 07:39:16 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-osd-10[60334]: 2024-05-27T07:39:16.808+0000 7f16335d9700 -1 osd.10 134 *** Got signal Terminated *** 2024-05-27T07:39:17.197 INFO:journalctl@ceph.osd.10.smithi105.stdout:May 27 07:39:16 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-osd-10[60334]: 2024-05-27T07:39:16.808+0000 7f16335d9700 -1 osd.10 134 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-05-27T07:39:17.609 INFO:journalctl@ceph.osd.11.smithi173.stdout:May 27 07:39:17 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-osd-11[63021]: 2024-05-27T07:39:17.268+0000 7f9d44ac8700 -1 osd.11 134 heartbeat_check: no reply from 172.21.15.52:6806 osd.0 since back 2024-05-27T07:37:57.927796+0000 front 2024-05-27T07:37:57.927759+0000 (oldest deadline 2024-05-27T07:38:22.028025+0000) 2024-05-27T07:39:17.609 INFO:journalctl@ceph.osd.11.smithi173.stdout:May 27 07:39:17 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-osd-11[63021]: 2024-05-27T07:39:17.268+0000 7f9d44ac8700 -1 osd.11 134 heartbeat_check: no reply from 172.21.15.105:6804 osd.1 since back 2024-05-27T07:38:03.129257+0000 front 2024-05-27T07:38:03.129120+0000 (oldest deadline 2024-05-27T07:38:28.929543+0000) 2024-05-27T07:39:17.610 INFO:journalctl@ceph.osd.11.smithi173.stdout:May 27 07:39:17 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-osd-11[63021]: 2024-05-27T07:39:17.268+0000 7f9d44ac8700 -1 osd.11 134 heartbeat_check: no reply from 172.21.15.173:6804 osd.2 since back 2024-05-27T07:38:16.431856+0000 front 2024-05-27T07:38:16.431878+0000 (oldest deadline 2024-05-27T07:38:42.332069+0000) 2024-05-27T07:39:17.610 INFO:journalctl@ceph.osd.11.smithi173.stdout:May 27 07:39:17 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-osd-11[63021]: 2024-05-27T07:39:17.268+0000 7f9d44ac8700 -1 osd.11 134 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-05-27T07:38:28.133691+0000 front 2024-05-27T07:38:28.134117+0000 (oldest deadline 2024-05-27T07:38:55.734329+0000) 2024-05-27T07:39:17.610 INFO:journalctl@ceph.osd.11.smithi173.stdout:May 27 07:39:17 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-osd-11[63021]: 2024-05-27T07:39:17.268+0000 7f9d44ac8700 -1 osd.11 134 heartbeat_check: no reply from 172.21.15.173:6812 osd.5 since back 2024-05-27T07:38:41.035302+0000 front 2024-05-27T07:38:41.035361+0000 (oldest deadline 2024-05-27T07:39:02.036045+0000) 2024-05-27T07:39:17.610 INFO:journalctl@ceph.osd.11.smithi173.stdout:May 27 07:39:17 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-osd-11[63021]: 2024-05-27T07:39:17.268+0000 7f9d44ac8700 -1 osd.11 134 heartbeat_check: no reply from 172.21.15.52:6822 osd.6 since back 2024-05-27T07:38:47.237679+0000 front 2024-05-27T07:38:47.237750+0000 (oldest deadline 2024-05-27T07:39:10.137647+0000) 2024-05-27T07:39:17.947 INFO:journalctl@ceph.osd.10.smithi105.stdout:May 27 07:39:17 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-osd-10[60334]: 2024-05-27T07:39:17.576+0000 7f162fb8f700 -1 osd.10 134 heartbeat_check: no reply from 172.21.15.52:6806 osd.0 since back 2024-05-27T07:37:56.754985+0000 front 2024-05-27T07:37:56.755070+0000 (oldest deadline 2024-05-27T07:38:22.054911+0000) 2024-05-27T07:39:17.947 INFO:journalctl@ceph.osd.10.smithi105.stdout:May 27 07:39:17 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-osd-10[60334]: 2024-05-27T07:39:17.576+0000 7f162fb8f700 -1 osd.10 134 heartbeat_check: no reply from 172.21.15.105:6804 osd.1 since back 2024-05-27T07:38:05.556137+0000 front 2024-05-27T07:38:05.556112+0000 (oldest deadline 2024-05-27T07:38:31.456213+0000) 2024-05-27T07:39:17.947 INFO:journalctl@ceph.osd.10.smithi105.stdout:May 27 07:39:17 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-osd-10[60334]: 2024-05-27T07:39:17.576+0000 7f162fb8f700 -1 osd.10 134 heartbeat_check: no reply from 172.21.15.173:6804 osd.2 since back 2024-05-27T07:38:15.557554+0000 front 2024-05-27T07:38:15.557409+0000 (oldest deadline 2024-05-27T07:38:40.857509+0000) 2024-05-27T07:39:17.947 INFO:journalctl@ceph.osd.10.smithi105.stdout:May 27 07:39:17 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-osd-10[60334]: 2024-05-27T07:39:17.576+0000 7f162fb8f700 -1 osd.10 134 heartbeat_check: no reply from 172.21.15.52:6814 osd.3 since back 2024-05-27T07:38:21.958641+0000 front 2024-05-27T07:38:21.958614+0000 (oldest deadline 2024-05-27T07:38:47.858600+0000) 2024-05-27T07:39:17.947 INFO:journalctl@ceph.osd.10.smithi105.stdout:May 27 07:39:17 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-osd-10[60334]: 2024-05-27T07:39:17.576+0000 7f162fb8f700 -1 osd.10 134 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-05-27T07:38:30.560765+0000 front 2024-05-27T07:38:30.560788+0000 (oldest deadline 2024-05-27T07:38:55.260887+0000) 2024-05-27T07:39:17.948 INFO:journalctl@ceph.osd.10.smithi105.stdout:May 27 07:39:17 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-osd-10[60334]: 2024-05-27T07:39:17.576+0000 7f162fb8f700 -1 osd.10 134 heartbeat_check: no reply from 172.21.15.173:6812 osd.5 since back 2024-05-27T07:38:36.962107+0000 front 2024-05-27T07:38:36.962140+0000 (oldest deadline 2024-05-27T07:39:04.562331+0000) 2024-05-27T07:39:17.948 INFO:journalctl@ceph.osd.10.smithi105.stdout:May 27 07:39:17 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-osd-10[60334]: 2024-05-27T07:39:17.576+0000 7f162fb8f700 -1 osd.10 134 heartbeat_check: no reply from 172.21.15.52:6822 osd.6 since back 2024-05-27T07:38:45.063573+0000 front 2024-05-27T07:38:45.063539+0000 (oldest deadline 2024-05-27T07:39:10.963318+0000) 2024-05-27T07:39:18.511 INFO:journalctl@ceph.osd.11.smithi173.stdout:May 27 07:39:18 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-osd-11[63021]: 2024-05-27T07:39:18.245+0000 7f9d44ac8700 -1 osd.11 134 heartbeat_check: no reply from 172.21.15.52:6806 osd.0 since back 2024-05-27T07:37:57.927796+0000 front 2024-05-27T07:37:57.927759+0000 (oldest deadline 2024-05-27T07:38:22.028025+0000) 2024-05-27T07:39:18.511 INFO:journalctl@ceph.osd.11.smithi173.stdout:May 27 07:39:18 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-osd-11[63021]: 2024-05-27T07:39:18.245+0000 7f9d44ac8700 -1 osd.11 134 heartbeat_check: no reply from 172.21.15.105:6804 osd.1 since back 2024-05-27T07:38:03.129257+0000 front 2024-05-27T07:38:03.129120+0000 (oldest deadline 2024-05-27T07:38:28.929543+0000) 2024-05-27T07:39:18.511 INFO:journalctl@ceph.osd.11.smithi173.stdout:May 27 07:39:18 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-osd-11[63021]: 2024-05-27T07:39:18.245+0000 7f9d44ac8700 -1 osd.11 134 heartbeat_check: no reply from 172.21.15.173:6804 osd.2 since back 2024-05-27T07:38:16.431856+0000 front 2024-05-27T07:38:16.431878+0000 (oldest deadline 2024-05-27T07:38:42.332069+0000) 2024-05-27T07:39:18.511 INFO:journalctl@ceph.osd.11.smithi173.stdout:May 27 07:39:18 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-osd-11[63021]: 2024-05-27T07:39:18.245+0000 7f9d44ac8700 -1 osd.11 134 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-05-27T07:38:28.133691+0000 front 2024-05-27T07:38:28.134117+0000 (oldest deadline 2024-05-27T07:38:55.734329+0000) 2024-05-27T07:39:18.511 INFO:journalctl@ceph.osd.11.smithi173.stdout:May 27 07:39:18 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-osd-11[63021]: 2024-05-27T07:39:18.245+0000 7f9d44ac8700 -1 osd.11 134 heartbeat_check: no reply from 172.21.15.173:6812 osd.5 since back 2024-05-27T07:38:41.035302+0000 front 2024-05-27T07:38:41.035361+0000 (oldest deadline 2024-05-27T07:39:02.036045+0000) 2024-05-27T07:39:18.511 INFO:journalctl@ceph.osd.11.smithi173.stdout:May 27 07:39:18 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-osd-11[63021]: 2024-05-27T07:39:18.245+0000 7f9d44ac8700 -1 osd.11 134 heartbeat_check: no reply from 172.21.15.52:6822 osd.6 since back 2024-05-27T07:38:47.237679+0000 front 2024-05-27T07:38:47.237750+0000 (oldest deadline 2024-05-27T07:39:10.137647+0000) 2024-05-27T07:39:18.947 INFO:journalctl@ceph.osd.10.smithi105.stdout:May 27 07:39:18 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-osd-10[60334]: 2024-05-27T07:39:18.542+0000 7f162fb8f700 -1 osd.10 134 heartbeat_check: no reply from 172.21.15.52:6806 osd.0 since back 2024-05-27T07:37:56.754985+0000 front 2024-05-27T07:37:56.755070+0000 (oldest deadline 2024-05-27T07:38:22.054911+0000) 2024-05-27T07:39:18.947 INFO:journalctl@ceph.osd.10.smithi105.stdout:May 27 07:39:18 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-osd-10[60334]: 2024-05-27T07:39:18.542+0000 7f162fb8f700 -1 osd.10 134 heartbeat_check: no reply from 172.21.15.105:6804 osd.1 since back 2024-05-27T07:38:05.556137+0000 front 2024-05-27T07:38:05.556112+0000 (oldest deadline 2024-05-27T07:38:31.456213+0000) 2024-05-27T07:39:18.947 INFO:journalctl@ceph.osd.10.smithi105.stdout:May 27 07:39:18 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-osd-10[60334]: 2024-05-27T07:39:18.542+0000 7f162fb8f700 -1 osd.10 134 heartbeat_check: no reply from 172.21.15.173:6804 osd.2 since back 2024-05-27T07:38:15.557554+0000 front 2024-05-27T07:38:15.557409+0000 (oldest deadline 2024-05-27T07:38:40.857509+0000) 2024-05-27T07:39:18.947 INFO:journalctl@ceph.osd.10.smithi105.stdout:May 27 07:39:18 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-osd-10[60334]: 2024-05-27T07:39:18.542+0000 7f162fb8f700 -1 osd.10 134 heartbeat_check: no reply from 172.21.15.52:6814 osd.3 since back 2024-05-27T07:38:21.958641+0000 front 2024-05-27T07:38:21.958614+0000 (oldest deadline 2024-05-27T07:38:47.858600+0000) 2024-05-27T07:39:18.947 INFO:journalctl@ceph.osd.10.smithi105.stdout:May 27 07:39:18 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-osd-10[60334]: 2024-05-27T07:39:18.542+0000 7f162fb8f700 -1 osd.10 134 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-05-27T07:38:30.560765+0000 front 2024-05-27T07:38:30.560788+0000 (oldest deadline 2024-05-27T07:38:55.260887+0000) 2024-05-27T07:39:18.947 INFO:journalctl@ceph.osd.10.smithi105.stdout:May 27 07:39:18 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-osd-10[60334]: 2024-05-27T07:39:18.542+0000 7f162fb8f700 -1 osd.10 134 heartbeat_check: no reply from 172.21.15.173:6812 osd.5 since back 2024-05-27T07:38:36.962107+0000 front 2024-05-27T07:38:36.962140+0000 (oldest deadline 2024-05-27T07:39:04.562331+0000) 2024-05-27T07:39:18.948 INFO:journalctl@ceph.osd.10.smithi105.stdout:May 27 07:39:18 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-osd-10[60334]: 2024-05-27T07:39:18.542+0000 7f162fb8f700 -1 osd.10 134 heartbeat_check: no reply from 172.21.15.52:6822 osd.6 since back 2024-05-27T07:38:45.063573+0000 front 2024-05-27T07:38:45.063539+0000 (oldest deadline 2024-05-27T07:39:10.963318+0000) 2024-05-27T07:39:19.609 INFO:journalctl@ceph.osd.11.smithi173.stdout:May 27 07:39:19 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-osd-11[63021]: 2024-05-27T07:39:19.273+0000 7f9d44ac8700 -1 osd.11 134 heartbeat_check: no reply from 172.21.15.52:6806 osd.0 since back 2024-05-27T07:37:57.927796+0000 front 2024-05-27T07:37:57.927759+0000 (oldest deadline 2024-05-27T07:38:22.028025+0000) 2024-05-27T07:39:19.610 INFO:journalctl@ceph.osd.11.smithi173.stdout:May 27 07:39:19 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-osd-11[63021]: 2024-05-27T07:39:19.273+0000 7f9d44ac8700 -1 osd.11 134 heartbeat_check: no reply from 172.21.15.105:6804 osd.1 since back 2024-05-27T07:38:03.129257+0000 front 2024-05-27T07:38:03.129120+0000 (oldest deadline 2024-05-27T07:38:28.929543+0000) 2024-05-27T07:39:19.610 INFO:journalctl@ceph.osd.11.smithi173.stdout:May 27 07:39:19 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-osd-11[63021]: 2024-05-27T07:39:19.273+0000 7f9d44ac8700 -1 osd.11 134 heartbeat_check: no reply from 172.21.15.173:6804 osd.2 since back 2024-05-27T07:38:16.431856+0000 front 2024-05-27T07:38:16.431878+0000 (oldest deadline 2024-05-27T07:38:42.332069+0000) 2024-05-27T07:39:19.610 INFO:journalctl@ceph.osd.11.smithi173.stdout:May 27 07:39:19 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-osd-11[63021]: 2024-05-27T07:39:19.273+0000 7f9d44ac8700 -1 osd.11 134 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-05-27T07:38:28.133691+0000 front 2024-05-27T07:38:28.134117+0000 (oldest deadline 2024-05-27T07:38:55.734329+0000) 2024-05-27T07:39:19.610 INFO:journalctl@ceph.osd.11.smithi173.stdout:May 27 07:39:19 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-osd-11[63021]: 2024-05-27T07:39:19.273+0000 7f9d44ac8700 -1 osd.11 134 heartbeat_check: no reply from 172.21.15.173:6812 osd.5 since back 2024-05-27T07:38:41.035302+0000 front 2024-05-27T07:38:41.035361+0000 (oldest deadline 2024-05-27T07:39:02.036045+0000) 2024-05-27T07:39:19.610 INFO:journalctl@ceph.osd.11.smithi173.stdout:May 27 07:39:19 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-osd-11[63021]: 2024-05-27T07:39:19.273+0000 7f9d44ac8700 -1 osd.11 134 heartbeat_check: no reply from 172.21.15.52:6822 osd.6 since back 2024-05-27T07:38:47.237679+0000 front 2024-05-27T07:38:47.237750+0000 (oldest deadline 2024-05-27T07:39:10.137647+0000) 2024-05-27T07:39:19.947 INFO:journalctl@ceph.osd.10.smithi105.stdout:May 27 07:39:19 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-osd-10[60334]: 2024-05-27T07:39:19.539+0000 7f162fb8f700 -1 osd.10 134 heartbeat_check: no reply from 172.21.15.52:6806 osd.0 since back 2024-05-27T07:37:56.754985+0000 front 2024-05-27T07:37:56.755070+0000 (oldest deadline 2024-05-27T07:38:22.054911+0000) 2024-05-27T07:39:19.947 INFO:journalctl@ceph.osd.10.smithi105.stdout:May 27 07:39:19 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-osd-10[60334]: 2024-05-27T07:39:19.539+0000 7f162fb8f700 -1 osd.10 134 heartbeat_check: no reply from 172.21.15.105:6804 osd.1 since back 2024-05-27T07:38:05.556137+0000 front 2024-05-27T07:38:05.556112+0000 (oldest deadline 2024-05-27T07:38:31.456213+0000) 2024-05-27T07:39:19.947 INFO:journalctl@ceph.osd.10.smithi105.stdout:May 27 07:39:19 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-osd-10[60334]: 2024-05-27T07:39:19.539+0000 7f162fb8f700 -1 osd.10 134 heartbeat_check: no reply from 172.21.15.173:6804 osd.2 since back 2024-05-27T07:38:15.557554+0000 front 2024-05-27T07:38:15.557409+0000 (oldest deadline 2024-05-27T07:38:40.857509+0000) 2024-05-27T07:39:19.947 INFO:journalctl@ceph.osd.10.smithi105.stdout:May 27 07:39:19 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-osd-10[60334]: 2024-05-27T07:39:19.539+0000 7f162fb8f700 -1 osd.10 134 heartbeat_check: no reply from 172.21.15.52:6814 osd.3 since back 2024-05-27T07:38:21.958641+0000 front 2024-05-27T07:38:21.958614+0000 (oldest deadline 2024-05-27T07:38:47.858600+0000) 2024-05-27T07:39:19.947 INFO:journalctl@ceph.osd.10.smithi105.stdout:May 27 07:39:19 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-osd-10[60334]: 2024-05-27T07:39:19.539+0000 7f162fb8f700 -1 osd.10 134 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-05-27T07:38:30.560765+0000 front 2024-05-27T07:38:30.560788+0000 (oldest deadline 2024-05-27T07:38:55.260887+0000) 2024-05-27T07:39:19.948 INFO:journalctl@ceph.osd.10.smithi105.stdout:May 27 07:39:19 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-osd-10[60334]: 2024-05-27T07:39:19.539+0000 7f162fb8f700 -1 osd.10 134 heartbeat_check: no reply from 172.21.15.173:6812 osd.5 since back 2024-05-27T07:38:36.962107+0000 front 2024-05-27T07:38:36.962140+0000 (oldest deadline 2024-05-27T07:39:04.562331+0000) 2024-05-27T07:39:19.948 INFO:journalctl@ceph.osd.10.smithi105.stdout:May 27 07:39:19 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-osd-10[60334]: 2024-05-27T07:39:19.539+0000 7f162fb8f700 -1 osd.10 134 heartbeat_check: no reply from 172.21.15.52:6822 osd.6 since back 2024-05-27T07:38:45.063573+0000 front 2024-05-27T07:38:45.063539+0000 (oldest deadline 2024-05-27T07:39:10.963318+0000) 2024-05-27T07:39:20.610 INFO:journalctl@ceph.osd.11.smithi173.stdout:May 27 07:39:20 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-osd-11[63021]: 2024-05-27T07:39:20.252+0000 7f9d44ac8700 -1 osd.11 134 heartbeat_check: no reply from 172.21.15.52:6806 osd.0 since back 2024-05-27T07:37:57.927796+0000 front 2024-05-27T07:37:57.927759+0000 (oldest deadline 2024-05-27T07:38:22.028025+0000) 2024-05-27T07:39:20.610 INFO:journalctl@ceph.osd.11.smithi173.stdout:May 27 07:39:20 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-osd-11[63021]: 2024-05-27T07:39:20.252+0000 7f9d44ac8700 -1 osd.11 134 heartbeat_check: no reply from 172.21.15.105:6804 osd.1 since back 2024-05-27T07:38:03.129257+0000 front 2024-05-27T07:38:03.129120+0000 (oldest deadline 2024-05-27T07:38:28.929543+0000) 2024-05-27T07:39:20.610 INFO:journalctl@ceph.osd.11.smithi173.stdout:May 27 07:39:20 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-osd-11[63021]: 2024-05-27T07:39:20.252+0000 7f9d44ac8700 -1 osd.11 134 heartbeat_check: no reply from 172.21.15.173:6804 osd.2 since back 2024-05-27T07:38:16.431856+0000 front 2024-05-27T07:38:16.431878+0000 (oldest deadline 2024-05-27T07:38:42.332069+0000) 2024-05-27T07:39:20.610 INFO:journalctl@ceph.osd.11.smithi173.stdout:May 27 07:39:20 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-osd-11[63021]: 2024-05-27T07:39:20.252+0000 7f9d44ac8700 -1 osd.11 134 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-05-27T07:38:28.133691+0000 front 2024-05-27T07:38:28.134117+0000 (oldest deadline 2024-05-27T07:38:55.734329+0000) 2024-05-27T07:39:20.610 INFO:journalctl@ceph.osd.11.smithi173.stdout:May 27 07:39:20 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-osd-11[63021]: 2024-05-27T07:39:20.252+0000 7f9d44ac8700 -1 osd.11 134 heartbeat_check: no reply from 172.21.15.173:6812 osd.5 since back 2024-05-27T07:38:41.035302+0000 front 2024-05-27T07:38:41.035361+0000 (oldest deadline 2024-05-27T07:39:02.036045+0000) 2024-05-27T07:39:20.610 INFO:journalctl@ceph.osd.11.smithi173.stdout:May 27 07:39:20 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-osd-11[63021]: 2024-05-27T07:39:20.252+0000 7f9d44ac8700 -1 osd.11 134 heartbeat_check: no reply from 172.21.15.52:6822 osd.6 since back 2024-05-27T07:38:47.237679+0000 front 2024-05-27T07:38:47.237750+0000 (oldest deadline 2024-05-27T07:39:10.137647+0000) 2024-05-27T07:39:20.947 INFO:journalctl@ceph.osd.10.smithi105.stdout:May 27 07:39:20 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-osd-10[60334]: 2024-05-27T07:39:20.563+0000 7f162fb8f700 -1 osd.10 134 heartbeat_check: no reply from 172.21.15.52:6806 osd.0 since back 2024-05-27T07:37:56.754985+0000 front 2024-05-27T07:37:56.755070+0000 (oldest deadline 2024-05-27T07:38:22.054911+0000) 2024-05-27T07:39:20.947 INFO:journalctl@ceph.osd.10.smithi105.stdout:May 27 07:39:20 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-osd-10[60334]: 2024-05-27T07:39:20.563+0000 7f162fb8f700 -1 osd.10 134 heartbeat_check: no reply from 172.21.15.105:6804 osd.1 since back 2024-05-27T07:38:05.556137+0000 front 2024-05-27T07:38:05.556112+0000 (oldest deadline 2024-05-27T07:38:31.456213+0000) 2024-05-27T07:39:20.947 INFO:journalctl@ceph.osd.10.smithi105.stdout:May 27 07:39:20 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-osd-10[60334]: 2024-05-27T07:39:20.563+0000 7f162fb8f700 -1 osd.10 134 heartbeat_check: no reply from 172.21.15.173:6804 osd.2 since back 2024-05-27T07:38:15.557554+0000 front 2024-05-27T07:38:15.557409+0000 (oldest deadline 2024-05-27T07:38:40.857509+0000) 2024-05-27T07:39:20.948 INFO:journalctl@ceph.osd.10.smithi105.stdout:May 27 07:39:20 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-osd-10[60334]: 2024-05-27T07:39:20.563+0000 7f162fb8f700 -1 osd.10 134 heartbeat_check: no reply from 172.21.15.52:6814 osd.3 since back 2024-05-27T07:38:21.958641+0000 front 2024-05-27T07:38:21.958614+0000 (oldest deadline 2024-05-27T07:38:47.858600+0000) 2024-05-27T07:39:20.948 INFO:journalctl@ceph.osd.10.smithi105.stdout:May 27 07:39:20 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-osd-10[60334]: 2024-05-27T07:39:20.563+0000 7f162fb8f700 -1 osd.10 134 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-05-27T07:38:30.560765+0000 front 2024-05-27T07:38:30.560788+0000 (oldest deadline 2024-05-27T07:38:55.260887+0000) 2024-05-27T07:39:20.948 INFO:journalctl@ceph.osd.10.smithi105.stdout:May 27 07:39:20 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-osd-10[60334]: 2024-05-27T07:39:20.563+0000 7f162fb8f700 -1 osd.10 134 heartbeat_check: no reply from 172.21.15.173:6812 osd.5 since back 2024-05-27T07:38:36.962107+0000 front 2024-05-27T07:38:36.962140+0000 (oldest deadline 2024-05-27T07:39:04.562331+0000) 2024-05-27T07:39:20.948 INFO:journalctl@ceph.osd.10.smithi105.stdout:May 27 07:39:20 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-osd-10[60334]: 2024-05-27T07:39:20.563+0000 7f162fb8f700 -1 osd.10 134 heartbeat_check: no reply from 172.21.15.52:6822 osd.6 since back 2024-05-27T07:38:45.063573+0000 front 2024-05-27T07:38:45.063539+0000 (oldest deadline 2024-05-27T07:39:10.963318+0000) 2024-05-27T07:39:21.609 INFO:journalctl@ceph.osd.11.smithi173.stdout:May 27 07:39:21 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-osd-11[63021]: 2024-05-27T07:39:21.285+0000 7f9d44ac8700 -1 osd.11 134 heartbeat_check: no reply from 172.21.15.52:6806 osd.0 since back 2024-05-27T07:37:57.927796+0000 front 2024-05-27T07:37:57.927759+0000 (oldest deadline 2024-05-27T07:38:22.028025+0000) 2024-05-27T07:39:21.610 INFO:journalctl@ceph.osd.11.smithi173.stdout:May 27 07:39:21 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-osd-11[63021]: 2024-05-27T07:39:21.285+0000 7f9d44ac8700 -1 osd.11 134 heartbeat_check: no reply from 172.21.15.105:6804 osd.1 since back 2024-05-27T07:38:03.129257+0000 front 2024-05-27T07:38:03.129120+0000 (oldest deadline 2024-05-27T07:38:28.929543+0000) 2024-05-27T07:39:21.610 INFO:journalctl@ceph.osd.11.smithi173.stdout:May 27 07:39:21 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-osd-11[63021]: 2024-05-27T07:39:21.285+0000 7f9d44ac8700 -1 osd.11 134 heartbeat_check: no reply from 172.21.15.173:6804 osd.2 since back 2024-05-27T07:38:16.431856+0000 front 2024-05-27T07:38:16.431878+0000 (oldest deadline 2024-05-27T07:38:42.332069+0000) 2024-05-27T07:39:21.610 INFO:journalctl@ceph.osd.11.smithi173.stdout:May 27 07:39:21 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-osd-11[63021]: 2024-05-27T07:39:21.285+0000 7f9d44ac8700 -1 osd.11 134 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-05-27T07:38:28.133691+0000 front 2024-05-27T07:38:28.134117+0000 (oldest deadline 2024-05-27T07:38:55.734329+0000) 2024-05-27T07:39:21.610 INFO:journalctl@ceph.osd.11.smithi173.stdout:May 27 07:39:21 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-osd-11[63021]: 2024-05-27T07:39:21.285+0000 7f9d44ac8700 -1 osd.11 134 heartbeat_check: no reply from 172.21.15.173:6812 osd.5 since back 2024-05-27T07:38:41.035302+0000 front 2024-05-27T07:38:41.035361+0000 (oldest deadline 2024-05-27T07:39:02.036045+0000) 2024-05-27T07:39:21.610 INFO:journalctl@ceph.osd.11.smithi173.stdout:May 27 07:39:21 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-osd-11[63021]: 2024-05-27T07:39:21.285+0000 7f9d44ac8700 -1 osd.11 134 heartbeat_check: no reply from 172.21.15.52:6822 osd.6 since back 2024-05-27T07:38:47.237679+0000 front 2024-05-27T07:38:47.237750+0000 (oldest deadline 2024-05-27T07:39:10.137647+0000) 2024-05-27T07:39:21.849 INFO:journalctl@ceph.osd.10.smithi105.stdout:May 27 07:39:21 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-osd-10[60334]: 2024-05-27T07:39:21.577+0000 7f162fb8f700 -1 osd.10 134 heartbeat_check: no reply from 172.21.15.52:6806 osd.0 since back 2024-05-27T07:37:56.754985+0000 front 2024-05-27T07:37:56.755070+0000 (oldest deadline 2024-05-27T07:38:22.054911+0000) 2024-05-27T07:39:21.849 INFO:journalctl@ceph.osd.10.smithi105.stdout:May 27 07:39:21 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-osd-10[60334]: 2024-05-27T07:39:21.577+0000 7f162fb8f700 -1 osd.10 134 heartbeat_check: no reply from 172.21.15.105:6804 osd.1 since back 2024-05-27T07:38:05.556137+0000 front 2024-05-27T07:38:05.556112+0000 (oldest deadline 2024-05-27T07:38:31.456213+0000) 2024-05-27T07:39:21.849 INFO:journalctl@ceph.osd.10.smithi105.stdout:May 27 07:39:21 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-osd-10[60334]: 2024-05-27T07:39:21.577+0000 7f162fb8f700 -1 osd.10 134 heartbeat_check: no reply from 172.21.15.173:6804 osd.2 since back 2024-05-27T07:38:15.557554+0000 front 2024-05-27T07:38:15.557409+0000 (oldest deadline 2024-05-27T07:38:40.857509+0000) 2024-05-27T07:39:21.849 INFO:journalctl@ceph.osd.10.smithi105.stdout:May 27 07:39:21 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-osd-10[60334]: 2024-05-27T07:39:21.577+0000 7f162fb8f700 -1 osd.10 134 heartbeat_check: no reply from 172.21.15.52:6814 osd.3 since back 2024-05-27T07:38:21.958641+0000 front 2024-05-27T07:38:21.958614+0000 (oldest deadline 2024-05-27T07:38:47.858600+0000) 2024-05-27T07:39:21.849 INFO:journalctl@ceph.osd.10.smithi105.stdout:May 27 07:39:21 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-osd-10[60334]: 2024-05-27T07:39:21.577+0000 7f162fb8f700 -1 osd.10 134 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-05-27T07:38:30.560765+0000 front 2024-05-27T07:38:30.560788+0000 (oldest deadline 2024-05-27T07:38:55.260887+0000) 2024-05-27T07:39:21.849 INFO:journalctl@ceph.osd.10.smithi105.stdout:May 27 07:39:21 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-osd-10[60334]: 2024-05-27T07:39:21.577+0000 7f162fb8f700 -1 osd.10 134 heartbeat_check: no reply from 172.21.15.173:6812 osd.5 since back 2024-05-27T07:38:36.962107+0000 front 2024-05-27T07:38:36.962140+0000 (oldest deadline 2024-05-27T07:39:04.562331+0000) 2024-05-27T07:39:21.849 INFO:journalctl@ceph.osd.10.smithi105.stdout:May 27 07:39:21 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-osd-10[60334]: 2024-05-27T07:39:21.577+0000 7f162fb8f700 -1 osd.10 134 heartbeat_check: no reply from 172.21.15.52:6822 osd.6 since back 2024-05-27T07:38:45.063573+0000 front 2024-05-27T07:38:45.063539+0000 (oldest deadline 2024-05-27T07:39:10.963318+0000) 2024-05-27T07:39:22.447 INFO:journalctl@ceph.osd.10.smithi105.stdout:May 27 07:39:22 smithi105 podman[72737]: 2024-05-27 07:39:22.227426249 +0000 UTC m=+5.566239193 container died 6f53fe5b482685bffa08af7060196d7db7f39ff00bc9d97f9d1ae20c568e67ae (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-osd-10, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, GIT_CLEAN=True, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240521, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True) 2024-05-27T07:39:22.609 INFO:journalctl@ceph.osd.11.smithi173.stdout:May 27 07:39:22 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-osd-11[63021]: 2024-05-27T07:39:22.309+0000 7f9d44ac8700 -1 osd.11 134 heartbeat_check: no reply from 172.21.15.52:6806 osd.0 since back 2024-05-27T07:37:57.927796+0000 front 2024-05-27T07:37:57.927759+0000 (oldest deadline 2024-05-27T07:38:22.028025+0000) 2024-05-27T07:39:22.610 INFO:journalctl@ceph.osd.11.smithi173.stdout:May 27 07:39:22 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-osd-11[63021]: 2024-05-27T07:39:22.309+0000 7f9d44ac8700 -1 osd.11 134 heartbeat_check: no reply from 172.21.15.105:6804 osd.1 since back 2024-05-27T07:38:03.129257+0000 front 2024-05-27T07:38:03.129120+0000 (oldest deadline 2024-05-27T07:38:28.929543+0000) 2024-05-27T07:39:22.610 INFO:journalctl@ceph.osd.11.smithi173.stdout:May 27 07:39:22 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-osd-11[63021]: 2024-05-27T07:39:22.309+0000 7f9d44ac8700 -1 osd.11 134 heartbeat_check: no reply from 172.21.15.173:6804 osd.2 since back 2024-05-27T07:38:16.431856+0000 front 2024-05-27T07:38:16.431878+0000 (oldest deadline 2024-05-27T07:38:42.332069+0000) 2024-05-27T07:39:22.610 INFO:journalctl@ceph.osd.11.smithi173.stdout:May 27 07:39:22 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-osd-11[63021]: 2024-05-27T07:39:22.309+0000 7f9d44ac8700 -1 osd.11 134 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-05-27T07:38:28.133691+0000 front 2024-05-27T07:38:28.134117+0000 (oldest deadline 2024-05-27T07:38:55.734329+0000) 2024-05-27T07:39:22.610 INFO:journalctl@ceph.osd.11.smithi173.stdout:May 27 07:39:22 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-osd-11[63021]: 2024-05-27T07:39:22.309+0000 7f9d44ac8700 -1 osd.11 134 heartbeat_check: no reply from 172.21.15.173:6812 osd.5 since back 2024-05-27T07:38:41.035302+0000 front 2024-05-27T07:38:41.035361+0000 (oldest deadline 2024-05-27T07:39:02.036045+0000) 2024-05-27T07:39:22.610 INFO:journalctl@ceph.osd.11.smithi173.stdout:May 27 07:39:22 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-osd-11[63021]: 2024-05-27T07:39:22.309+0000 7f9d44ac8700 -1 osd.11 134 heartbeat_check: no reply from 172.21.15.52:6822 osd.6 since back 2024-05-27T07:38:47.237679+0000 front 2024-05-27T07:38:47.237750+0000 (oldest deadline 2024-05-27T07:39:10.137647+0000) 2024-05-27T07:39:22.764 INFO:journalctl@ceph.osd.10.smithi105.stdout:May 27 07:39:22 smithi105 podman[72737]: 2024-05-27 07:39:22.566912333 +0000 UTC m=+5.905725277 container cleanup 6f53fe5b482685bffa08af7060196d7db7f39ff00bc9d97f9d1ae20c568e67ae (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-osd-10, org.label-schema.build-date=20240521, ceph=True, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, GIT_CLEAN=True, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=) 2024-05-27T07:39:22.765 INFO:journalctl@ceph.osd.10.smithi105.stdout:May 27 07:39:22 smithi105 bash[72737]: ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-osd-10 2024-05-27T07:39:23.092 INFO:journalctl@ceph.osd.10.smithi105.stdout:May 27 07:39:22 smithi105 podman[72756]: 2024-05-27 07:39:22.768033411 +0000 UTC m=+0.538262080 container remove 6f53fe5b482685bffa08af7060196d7db7f39ff00bc9d97f9d1ae20c568e67ae (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-osd-10, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, ceph=True, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240521, org.label-schema.schema-version=1.0, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image) 2024-05-27T07:39:23.351 INFO:journalctl@ceph.osd.10.smithi105.stdout:May 27 07:39:23 smithi105 podman[72852]: 2024-05-27 07:39:23.185631431 +0000 UTC m=+0.112084896 container create 03ed0a2f3e7290c3bf4c3f79eab228207d23550254712f9ce29b0e8f89cf6a51 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-osd-10-deactivate, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240521, GIT_CLEAN=True, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, ceph=True, io.buildah.version=1.33.5) 2024-05-27T07:39:23.351 INFO:journalctl@ceph.osd.10.smithi105.stdout:May 27 07:39:23 smithi105 podman[72852]: 2024-05-27 07:39:23.090624295 +0000 UTC m=+0.017077781 image pull e36ea4fc8a5807e8d2701370e350a0de3856538797f7fba64cfce7e344bd6799 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6 2024-05-27T07:39:23.351 INFO:journalctl@ceph.osd.10.smithi105.stdout:May 27 07:39:23 smithi105 podman[72852]: 2024-05-27 07:39:23.354492087 +0000 UTC m=+0.280945558 container init 03ed0a2f3e7290c3bf4c3f79eab228207d23550254712f9ce29b0e8f89cf6a51 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-osd-10-deactivate, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240521, io.buildah.version=1.33.5, GIT_CLEAN=True, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, ceph=True, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0) 2024-05-27T07:39:23.609 INFO:journalctl@ceph.osd.11.smithi173.stdout:May 27 07:39:23 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-osd-11[63021]: 2024-05-27T07:39:23.351+0000 7f9d44ac8700 -1 osd.11 134 heartbeat_check: no reply from 172.21.15.52:6806 osd.0 since back 2024-05-27T07:37:57.927796+0000 front 2024-05-27T07:37:57.927759+0000 (oldest deadline 2024-05-27T07:38:22.028025+0000) 2024-05-27T07:39:23.609 INFO:journalctl@ceph.osd.11.smithi173.stdout:May 27 07:39:23 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-osd-11[63021]: 2024-05-27T07:39:23.351+0000 7f9d44ac8700 -1 osd.11 134 heartbeat_check: no reply from 172.21.15.105:6804 osd.1 since back 2024-05-27T07:38:03.129257+0000 front 2024-05-27T07:38:03.129120+0000 (oldest deadline 2024-05-27T07:38:28.929543+0000) 2024-05-27T07:39:23.609 INFO:journalctl@ceph.osd.11.smithi173.stdout:May 27 07:39:23 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-osd-11[63021]: 2024-05-27T07:39:23.351+0000 7f9d44ac8700 -1 osd.11 134 heartbeat_check: no reply from 172.21.15.173:6804 osd.2 since back 2024-05-27T07:38:16.431856+0000 front 2024-05-27T07:38:16.431878+0000 (oldest deadline 2024-05-27T07:38:42.332069+0000) 2024-05-27T07:39:23.610 INFO:journalctl@ceph.osd.11.smithi173.stdout:May 27 07:39:23 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-osd-11[63021]: 2024-05-27T07:39:23.351+0000 7f9d44ac8700 -1 osd.11 134 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-05-27T07:38:28.133691+0000 front 2024-05-27T07:38:28.134117+0000 (oldest deadline 2024-05-27T07:38:55.734329+0000) 2024-05-27T07:39:23.610 INFO:journalctl@ceph.osd.11.smithi173.stdout:May 27 07:39:23 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-osd-11[63021]: 2024-05-27T07:39:23.351+0000 7f9d44ac8700 -1 osd.11 134 heartbeat_check: no reply from 172.21.15.173:6812 osd.5 since back 2024-05-27T07:38:41.035302+0000 front 2024-05-27T07:38:41.035361+0000 (oldest deadline 2024-05-27T07:39:02.036045+0000) 2024-05-27T07:39:23.610 INFO:journalctl@ceph.osd.11.smithi173.stdout:May 27 07:39:23 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-osd-11[63021]: 2024-05-27T07:39:23.351+0000 7f9d44ac8700 -1 osd.11 134 heartbeat_check: no reply from 172.21.15.52:6822 osd.6 since back 2024-05-27T07:38:47.237679+0000 front 2024-05-27T07:38:47.237750+0000 (oldest deadline 2024-05-27T07:39:10.137647+0000) 2024-05-27T07:39:23.610 INFO:journalctl@ceph.osd.11.smithi173.stdout:May 27 07:39:23 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-osd-11[63021]: 2024-05-27T07:39:23.351+0000 7f9d44ac8700 -1 osd.11 134 heartbeat_check: no reply from 172.21.15.105:6820 osd.7 since back 2024-05-27T07:38:56.539259+0000 front 2024-05-27T07:38:56.539235+0000 (oldest deadline 2024-05-27T07:39:22.438948+0000) 2024-05-27T07:39:23.649 INFO:journalctl@ceph.osd.10.smithi105.stdout:May 27 07:39:23 smithi105 podman[72852]: 2024-05-27 07:39:23.357551407 +0000 UTC m=+0.284004883 container start 03ed0a2f3e7290c3bf4c3f79eab228207d23550254712f9ce29b0e8f89cf6a51 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-osd-10-deactivate, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240521, ceph=True, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, GIT_BRANCH=HEAD, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0) 2024-05-27T07:39:23.649 INFO:journalctl@ceph.osd.10.smithi105.stdout:May 27 07:39:23 smithi105 podman[72852]: 2024-05-27 07:39:23.44615457 +0000 UTC m=+0.372608052 container attach 03ed0a2f3e7290c3bf4c3f79eab228207d23550254712f9ce29b0e8f89cf6a51 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-osd-10-deactivate, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, GIT_CLEAN=True, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, GIT_BRANCH=HEAD, org.label-schema.build-date=20240521, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS) 2024-05-27T07:39:23.937 INFO:journalctl@ceph.osd.10.smithi105.stdout:May 27 07:39:23 smithi105 podman[72852]: 2024-05-27 07:39:23.653161252 +0000 UTC m=+0.579614731 container died 03ed0a2f3e7290c3bf4c3f79eab228207d23550254712f9ce29b0e8f89cf6a51 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-osd-10-deactivate, GIT_BRANCH=HEAD, ceph=True, GIT_CLEAN=True, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240521, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5) 2024-05-27T07:39:24.248 DEBUG:teuthology.orchestra.run.smithi105:> sudo pkill -f 'journalctl -f -n 0 -u ceph-73df2636-1bf8-11ef-bc9b-c7b262605968@osd.10.service' 2024-05-27T07:39:24.285 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-05-27T07:39:24.285 INFO:tasks.cephadm.osd.10:Stopped osd.10 2024-05-27T07:39:24.285 INFO:tasks.cephadm.osd.11:Stopping osd.11... 2024-05-27T07:39:24.286 DEBUG:teuthology.orchestra.run.smithi173:> sudo systemctl stop ceph-73df2636-1bf8-11ef-bc9b-c7b262605968@osd.11 2024-05-27T07:39:24.606 INFO:journalctl@ceph.osd.11.smithi173.stdout:May 27 07:39:24 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-osd-11[63021]: 2024-05-27T07:39:24.327+0000 7f9d44ac8700 -1 osd.11 134 heartbeat_check: no reply from 172.21.15.52:6806 osd.0 since back 2024-05-27T07:37:57.927796+0000 front 2024-05-27T07:37:57.927759+0000 (oldest deadline 2024-05-27T07:38:22.028025+0000) 2024-05-27T07:39:24.606 INFO:journalctl@ceph.osd.11.smithi173.stdout:May 27 07:39:24 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-osd-11[63021]: 2024-05-27T07:39:24.327+0000 7f9d44ac8700 -1 osd.11 134 heartbeat_check: no reply from 172.21.15.105:6804 osd.1 since back 2024-05-27T07:38:03.129257+0000 front 2024-05-27T07:38:03.129120+0000 (oldest deadline 2024-05-27T07:38:28.929543+0000) 2024-05-27T07:39:24.606 INFO:journalctl@ceph.osd.11.smithi173.stdout:May 27 07:39:24 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-osd-11[63021]: 2024-05-27T07:39:24.327+0000 7f9d44ac8700 -1 osd.11 134 heartbeat_check: no reply from 172.21.15.173:6804 osd.2 since back 2024-05-27T07:38:16.431856+0000 front 2024-05-27T07:38:16.431878+0000 (oldest deadline 2024-05-27T07:38:42.332069+0000) 2024-05-27T07:39:24.606 INFO:journalctl@ceph.osd.11.smithi173.stdout:May 27 07:39:24 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-osd-11[63021]: 2024-05-27T07:39:24.327+0000 7f9d44ac8700 -1 osd.11 134 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-05-27T07:38:28.133691+0000 front 2024-05-27T07:38:28.134117+0000 (oldest deadline 2024-05-27T07:38:55.734329+0000) 2024-05-27T07:39:24.606 INFO:journalctl@ceph.osd.11.smithi173.stdout:May 27 07:39:24 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-osd-11[63021]: 2024-05-27T07:39:24.327+0000 7f9d44ac8700 -1 osd.11 134 heartbeat_check: no reply from 172.21.15.173:6812 osd.5 since back 2024-05-27T07:38:41.035302+0000 front 2024-05-27T07:38:41.035361+0000 (oldest deadline 2024-05-27T07:39:02.036045+0000) 2024-05-27T07:39:24.606 INFO:journalctl@ceph.osd.11.smithi173.stdout:May 27 07:39:24 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-osd-11[63021]: 2024-05-27T07:39:24.327+0000 7f9d44ac8700 -1 osd.11 134 heartbeat_check: no reply from 172.21.15.52:6822 osd.6 since back 2024-05-27T07:38:47.237679+0000 front 2024-05-27T07:38:47.237750+0000 (oldest deadline 2024-05-27T07:39:10.137647+0000) 2024-05-27T07:39:24.606 INFO:journalctl@ceph.osd.11.smithi173.stdout:May 27 07:39:24 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-osd-11[63021]: 2024-05-27T07:39:24.327+0000 7f9d44ac8700 -1 osd.11 134 heartbeat_check: no reply from 172.21.15.105:6820 osd.7 since back 2024-05-27T07:38:56.539259+0000 front 2024-05-27T07:38:56.539235+0000 (oldest deadline 2024-05-27T07:39:22.438948+0000) 2024-05-27T07:39:24.606 INFO:journalctl@ceph.osd.11.smithi173.stdout:May 27 07:39:24 smithi173 systemd[1]: Stopping Ceph osd.11 for 73df2636-1bf8-11ef-bc9b-c7b262605968... 2024-05-27T07:39:24.607 INFO:journalctl@ceph.osd.11.smithi173.stdout:May 27 07:39:24 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-osd-11[63021]: 2024-05-27T07:39:24.608+0000 7f9d4c51a700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.11 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-05-27T07:39:24.607 INFO:journalctl@ceph.osd.11.smithi173.stdout:May 27 07:39:24 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-osd-11[63021]: 2024-05-27T07:39:24.608+0000 7f9d4c51a700 -1 osd.11 134 *** Got signal Terminated *** 2024-05-27T07:39:24.607 INFO:journalctl@ceph.osd.11.smithi173.stdout:May 27 07:39:24 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-osd-11[63021]: 2024-05-27T07:39:24.608+0000 7f9d4c51a700 -1 osd.11 134 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-05-27T07:39:25.609 INFO:journalctl@ceph.osd.11.smithi173.stdout:May 27 07:39:25 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-osd-11[63021]: 2024-05-27T07:39:25.333+0000 7f9d44ac8700 -1 osd.11 134 heartbeat_check: no reply from 172.21.15.52:6806 osd.0 since back 2024-05-27T07:37:57.927796+0000 front 2024-05-27T07:37:57.927759+0000 (oldest deadline 2024-05-27T07:38:22.028025+0000) 2024-05-27T07:39:25.610 INFO:journalctl@ceph.osd.11.smithi173.stdout:May 27 07:39:25 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-osd-11[63021]: 2024-05-27T07:39:25.333+0000 7f9d44ac8700 -1 osd.11 134 heartbeat_check: no reply from 172.21.15.105:6804 osd.1 since back 2024-05-27T07:38:03.129257+0000 front 2024-05-27T07:38:03.129120+0000 (oldest deadline 2024-05-27T07:38:28.929543+0000) 2024-05-27T07:39:25.610 INFO:journalctl@ceph.osd.11.smithi173.stdout:May 27 07:39:25 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-osd-11[63021]: 2024-05-27T07:39:25.333+0000 7f9d44ac8700 -1 osd.11 134 heartbeat_check: no reply from 172.21.15.173:6804 osd.2 since back 2024-05-27T07:38:16.431856+0000 front 2024-05-27T07:38:16.431878+0000 (oldest deadline 2024-05-27T07:38:42.332069+0000) 2024-05-27T07:39:25.610 INFO:journalctl@ceph.osd.11.smithi173.stdout:May 27 07:39:25 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-osd-11[63021]: 2024-05-27T07:39:25.333+0000 7f9d44ac8700 -1 osd.11 134 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-05-27T07:38:28.133691+0000 front 2024-05-27T07:38:28.134117+0000 (oldest deadline 2024-05-27T07:38:55.734329+0000) 2024-05-27T07:39:25.610 INFO:journalctl@ceph.osd.11.smithi173.stdout:May 27 07:39:25 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-osd-11[63021]: 2024-05-27T07:39:25.333+0000 7f9d44ac8700 -1 osd.11 134 heartbeat_check: no reply from 172.21.15.173:6812 osd.5 since back 2024-05-27T07:38:41.035302+0000 front 2024-05-27T07:38:41.035361+0000 (oldest deadline 2024-05-27T07:39:02.036045+0000) 2024-05-27T07:39:25.610 INFO:journalctl@ceph.osd.11.smithi173.stdout:May 27 07:39:25 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-osd-11[63021]: 2024-05-27T07:39:25.333+0000 7f9d44ac8700 -1 osd.11 134 heartbeat_check: no reply from 172.21.15.52:6822 osd.6 since back 2024-05-27T07:38:47.237679+0000 front 2024-05-27T07:38:47.237750+0000 (oldest deadline 2024-05-27T07:39:10.137647+0000) 2024-05-27T07:39:25.610 INFO:journalctl@ceph.osd.11.smithi173.stdout:May 27 07:39:25 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-osd-11[63021]: 2024-05-27T07:39:25.333+0000 7f9d44ac8700 -1 osd.11 134 heartbeat_check: no reply from 172.21.15.105:6820 osd.7 since back 2024-05-27T07:38:56.539259+0000 front 2024-05-27T07:38:56.539235+0000 (oldest deadline 2024-05-27T07:39:22.438948+0000) 2024-05-27T07:39:26.859 INFO:journalctl@ceph.osd.11.smithi173.stdout:May 27 07:39:26 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-osd-11[63021]: 2024-05-27T07:39:26.382+0000 7f9d44ac8700 -1 osd.11 134 heartbeat_check: no reply from 172.21.15.52:6806 osd.0 since back 2024-05-27T07:37:57.927796+0000 front 2024-05-27T07:37:57.927759+0000 (oldest deadline 2024-05-27T07:38:22.028025+0000) 2024-05-27T07:39:26.860 INFO:journalctl@ceph.osd.11.smithi173.stdout:May 27 07:39:26 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-osd-11[63021]: 2024-05-27T07:39:26.382+0000 7f9d44ac8700 -1 osd.11 134 heartbeat_check: no reply from 172.21.15.105:6804 osd.1 since back 2024-05-27T07:38:03.129257+0000 front 2024-05-27T07:38:03.129120+0000 (oldest deadline 2024-05-27T07:38:28.929543+0000) 2024-05-27T07:39:26.860 INFO:journalctl@ceph.osd.11.smithi173.stdout:May 27 07:39:26 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-osd-11[63021]: 2024-05-27T07:39:26.382+0000 7f9d44ac8700 -1 osd.11 134 heartbeat_check: no reply from 172.21.15.173:6804 osd.2 since back 2024-05-27T07:38:16.431856+0000 front 2024-05-27T07:38:16.431878+0000 (oldest deadline 2024-05-27T07:38:42.332069+0000) 2024-05-27T07:39:26.860 INFO:journalctl@ceph.osd.11.smithi173.stdout:May 27 07:39:26 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-osd-11[63021]: 2024-05-27T07:39:26.382+0000 7f9d44ac8700 -1 osd.11 134 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-05-27T07:38:28.133691+0000 front 2024-05-27T07:38:28.134117+0000 (oldest deadline 2024-05-27T07:38:55.734329+0000) 2024-05-27T07:39:26.860 INFO:journalctl@ceph.osd.11.smithi173.stdout:May 27 07:39:26 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-osd-11[63021]: 2024-05-27T07:39:26.382+0000 7f9d44ac8700 -1 osd.11 134 heartbeat_check: no reply from 172.21.15.173:6812 osd.5 since back 2024-05-27T07:38:41.035302+0000 front 2024-05-27T07:38:41.035361+0000 (oldest deadline 2024-05-27T07:39:02.036045+0000) 2024-05-27T07:39:26.860 INFO:journalctl@ceph.osd.11.smithi173.stdout:May 27 07:39:26 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-osd-11[63021]: 2024-05-27T07:39:26.382+0000 7f9d44ac8700 -1 osd.11 134 heartbeat_check: no reply from 172.21.15.52:6822 osd.6 since back 2024-05-27T07:38:47.237679+0000 front 2024-05-27T07:38:47.237750+0000 (oldest deadline 2024-05-27T07:39:10.137647+0000) 2024-05-27T07:39:26.860 INFO:journalctl@ceph.osd.11.smithi173.stdout:May 27 07:39:26 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-osd-11[63021]: 2024-05-27T07:39:26.382+0000 7f9d44ac8700 -1 osd.11 134 heartbeat_check: no reply from 172.21.15.105:6820 osd.7 since back 2024-05-27T07:38:56.539259+0000 front 2024-05-27T07:38:56.539235+0000 (oldest deadline 2024-05-27T07:39:22.438948+0000) 2024-05-27T07:39:27.859 INFO:journalctl@ceph.osd.11.smithi173.stdout:May 27 07:39:27 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-osd-11[63021]: 2024-05-27T07:39:27.373+0000 7f9d44ac8700 -1 osd.11 134 heartbeat_check: no reply from 172.21.15.52:6806 osd.0 since back 2024-05-27T07:37:57.927796+0000 front 2024-05-27T07:37:57.927759+0000 (oldest deadline 2024-05-27T07:38:22.028025+0000) 2024-05-27T07:39:27.860 INFO:journalctl@ceph.osd.11.smithi173.stdout:May 27 07:39:27 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-osd-11[63021]: 2024-05-27T07:39:27.373+0000 7f9d44ac8700 -1 osd.11 134 heartbeat_check: no reply from 172.21.15.105:6804 osd.1 since back 2024-05-27T07:38:03.129257+0000 front 2024-05-27T07:38:03.129120+0000 (oldest deadline 2024-05-27T07:38:28.929543+0000) 2024-05-27T07:39:27.860 INFO:journalctl@ceph.osd.11.smithi173.stdout:May 27 07:39:27 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-osd-11[63021]: 2024-05-27T07:39:27.373+0000 7f9d44ac8700 -1 osd.11 134 heartbeat_check: no reply from 172.21.15.173:6804 osd.2 since back 2024-05-27T07:38:16.431856+0000 front 2024-05-27T07:38:16.431878+0000 (oldest deadline 2024-05-27T07:38:42.332069+0000) 2024-05-27T07:39:27.860 INFO:journalctl@ceph.osd.11.smithi173.stdout:May 27 07:39:27 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-osd-11[63021]: 2024-05-27T07:39:27.373+0000 7f9d44ac8700 -1 osd.11 134 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-05-27T07:38:28.133691+0000 front 2024-05-27T07:38:28.134117+0000 (oldest deadline 2024-05-27T07:38:55.734329+0000) 2024-05-27T07:39:27.860 INFO:journalctl@ceph.osd.11.smithi173.stdout:May 27 07:39:27 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-osd-11[63021]: 2024-05-27T07:39:27.373+0000 7f9d44ac8700 -1 osd.11 134 heartbeat_check: no reply from 172.21.15.173:6812 osd.5 since back 2024-05-27T07:38:41.035302+0000 front 2024-05-27T07:38:41.035361+0000 (oldest deadline 2024-05-27T07:39:02.036045+0000) 2024-05-27T07:39:27.860 INFO:journalctl@ceph.osd.11.smithi173.stdout:May 27 07:39:27 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-osd-11[63021]: 2024-05-27T07:39:27.373+0000 7f9d44ac8700 -1 osd.11 134 heartbeat_check: no reply from 172.21.15.52:6822 osd.6 since back 2024-05-27T07:38:47.237679+0000 front 2024-05-27T07:38:47.237750+0000 (oldest deadline 2024-05-27T07:39:10.137647+0000) 2024-05-27T07:39:27.860 INFO:journalctl@ceph.osd.11.smithi173.stdout:May 27 07:39:27 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-osd-11[63021]: 2024-05-27T07:39:27.373+0000 7f9d44ac8700 -1 osd.11 134 heartbeat_check: no reply from 172.21.15.105:6820 osd.7 since back 2024-05-27T07:38:56.539259+0000 front 2024-05-27T07:38:56.539235+0000 (oldest deadline 2024-05-27T07:39:22.438948+0000) 2024-05-27T07:39:27.860 INFO:journalctl@ceph.osd.11.smithi173.stdout:May 27 07:39:27 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-osd-11[63021]: 2024-05-27T07:39:27.373+0000 7f9d44ac8700 -1 osd.11 134 heartbeat_check: no reply from 172.21.15.173:6820 osd.8 since back 2024-05-27T07:39:02.439479+0000 front 2024-05-27T07:39:02.439525+0000 (oldest deadline 2024-05-27T07:39:26.539479+0000) 2024-05-27T07:39:28.859 INFO:journalctl@ceph.osd.11.smithi173.stdout:May 27 07:39:28 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-osd-11[63021]: 2024-05-27T07:39:28.412+0000 7f9d44ac8700 -1 osd.11 134 heartbeat_check: no reply from 172.21.15.52:6806 osd.0 since back 2024-05-27T07:37:57.927796+0000 front 2024-05-27T07:37:57.927759+0000 (oldest deadline 2024-05-27T07:38:22.028025+0000) 2024-05-27T07:39:28.859 INFO:journalctl@ceph.osd.11.smithi173.stdout:May 27 07:39:28 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-osd-11[63021]: 2024-05-27T07:39:28.412+0000 7f9d44ac8700 -1 osd.11 134 heartbeat_check: no reply from 172.21.15.105:6804 osd.1 since back 2024-05-27T07:38:03.129257+0000 front 2024-05-27T07:38:03.129120+0000 (oldest deadline 2024-05-27T07:38:28.929543+0000) 2024-05-27T07:39:28.859 INFO:journalctl@ceph.osd.11.smithi173.stdout:May 27 07:39:28 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-osd-11[63021]: 2024-05-27T07:39:28.412+0000 7f9d44ac8700 -1 osd.11 134 heartbeat_check: no reply from 172.21.15.173:6804 osd.2 since back 2024-05-27T07:38:16.431856+0000 front 2024-05-27T07:38:16.431878+0000 (oldest deadline 2024-05-27T07:38:42.332069+0000) 2024-05-27T07:39:28.860 INFO:journalctl@ceph.osd.11.smithi173.stdout:May 27 07:39:28 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-osd-11[63021]: 2024-05-27T07:39:28.412+0000 7f9d44ac8700 -1 osd.11 134 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-05-27T07:38:28.133691+0000 front 2024-05-27T07:38:28.134117+0000 (oldest deadline 2024-05-27T07:38:55.734329+0000) 2024-05-27T07:39:28.860 INFO:journalctl@ceph.osd.11.smithi173.stdout:May 27 07:39:28 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-osd-11[63021]: 2024-05-27T07:39:28.412+0000 7f9d44ac8700 -1 osd.11 134 heartbeat_check: no reply from 172.21.15.173:6812 osd.5 since back 2024-05-27T07:38:41.035302+0000 front 2024-05-27T07:38:41.035361+0000 (oldest deadline 2024-05-27T07:39:02.036045+0000) 2024-05-27T07:39:28.860 INFO:journalctl@ceph.osd.11.smithi173.stdout:May 27 07:39:28 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-osd-11[63021]: 2024-05-27T07:39:28.412+0000 7f9d44ac8700 -1 osd.11 134 heartbeat_check: no reply from 172.21.15.52:6822 osd.6 since back 2024-05-27T07:38:47.237679+0000 front 2024-05-27T07:38:47.237750+0000 (oldest deadline 2024-05-27T07:39:10.137647+0000) 2024-05-27T07:39:28.860 INFO:journalctl@ceph.osd.11.smithi173.stdout:May 27 07:39:28 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-osd-11[63021]: 2024-05-27T07:39:28.412+0000 7f9d44ac8700 -1 osd.11 134 heartbeat_check: no reply from 172.21.15.105:6820 osd.7 since back 2024-05-27T07:38:56.539259+0000 front 2024-05-27T07:38:56.539235+0000 (oldest deadline 2024-05-27T07:39:22.438948+0000) 2024-05-27T07:39:28.860 INFO:journalctl@ceph.osd.11.smithi173.stdout:May 27 07:39:28 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-osd-11[63021]: 2024-05-27T07:39:28.412+0000 7f9d44ac8700 -1 osd.11 134 heartbeat_check: no reply from 172.21.15.173:6820 osd.8 since back 2024-05-27T07:39:02.439479+0000 front 2024-05-27T07:39:02.439525+0000 (oldest deadline 2024-05-27T07:39:26.539479+0000) 2024-05-27T07:39:29.860 INFO:journalctl@ceph.osd.11.smithi173.stdout:May 27 07:39:29 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-osd-11[63021]: 2024-05-27T07:39:29.402+0000 7f9d44ac8700 -1 osd.11 134 heartbeat_check: no reply from 172.21.15.52:6806 osd.0 since back 2024-05-27T07:37:57.927796+0000 front 2024-05-27T07:37:57.927759+0000 (oldest deadline 2024-05-27T07:38:22.028025+0000) 2024-05-27T07:39:29.860 INFO:journalctl@ceph.osd.11.smithi173.stdout:May 27 07:39:29 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-osd-11[63021]: 2024-05-27T07:39:29.402+0000 7f9d44ac8700 -1 osd.11 134 heartbeat_check: no reply from 172.21.15.105:6804 osd.1 since back 2024-05-27T07:38:03.129257+0000 front 2024-05-27T07:38:03.129120+0000 (oldest deadline 2024-05-27T07:38:28.929543+0000) 2024-05-27T07:39:29.860 INFO:journalctl@ceph.osd.11.smithi173.stdout:May 27 07:39:29 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-osd-11[63021]: 2024-05-27T07:39:29.402+0000 7f9d44ac8700 -1 osd.11 134 heartbeat_check: no reply from 172.21.15.173:6804 osd.2 since back 2024-05-27T07:38:16.431856+0000 front 2024-05-27T07:38:16.431878+0000 (oldest deadline 2024-05-27T07:38:42.332069+0000) 2024-05-27T07:39:29.860 INFO:journalctl@ceph.osd.11.smithi173.stdout:May 27 07:39:29 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-osd-11[63021]: 2024-05-27T07:39:29.402+0000 7f9d44ac8700 -1 osd.11 134 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-05-27T07:38:28.133691+0000 front 2024-05-27T07:38:28.134117+0000 (oldest deadline 2024-05-27T07:38:55.734329+0000) 2024-05-27T07:39:29.860 INFO:journalctl@ceph.osd.11.smithi173.stdout:May 27 07:39:29 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-osd-11[63021]: 2024-05-27T07:39:29.402+0000 7f9d44ac8700 -1 osd.11 134 heartbeat_check: no reply from 172.21.15.173:6812 osd.5 since back 2024-05-27T07:38:41.035302+0000 front 2024-05-27T07:38:41.035361+0000 (oldest deadline 2024-05-27T07:39:02.036045+0000) 2024-05-27T07:39:29.860 INFO:journalctl@ceph.osd.11.smithi173.stdout:May 27 07:39:29 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-osd-11[63021]: 2024-05-27T07:39:29.402+0000 7f9d44ac8700 -1 osd.11 134 heartbeat_check: no reply from 172.21.15.52:6822 osd.6 since back 2024-05-27T07:38:47.237679+0000 front 2024-05-27T07:38:47.237750+0000 (oldest deadline 2024-05-27T07:39:10.137647+0000) 2024-05-27T07:39:29.860 INFO:journalctl@ceph.osd.11.smithi173.stdout:May 27 07:39:29 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-osd-11[63021]: 2024-05-27T07:39:29.402+0000 7f9d44ac8700 -1 osd.11 134 heartbeat_check: no reply from 172.21.15.105:6820 osd.7 since back 2024-05-27T07:38:56.539259+0000 front 2024-05-27T07:38:56.539235+0000 (oldest deadline 2024-05-27T07:39:22.438948+0000) 2024-05-27T07:39:29.861 INFO:journalctl@ceph.osd.11.smithi173.stdout:May 27 07:39:29 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-osd-11[63021]: 2024-05-27T07:39:29.402+0000 7f9d44ac8700 -1 osd.11 134 heartbeat_check: no reply from 172.21.15.173:6820 osd.8 since back 2024-05-27T07:39:02.439479+0000 front 2024-05-27T07:39:02.439525+0000 (oldest deadline 2024-05-27T07:39:26.539479+0000) 2024-05-27T07:39:30.489 INFO:journalctl@ceph.osd.11.smithi173.stdout:May 27 07:39:30 smithi173 podman[76346]: 2024-05-27 07:39:30.204699469 +0000 UTC m=+5.721361092 container died 559955021bb4bce79c5b449bd917b56fee751b8c81337cd371e14e109bc73925 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-osd-11, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, ceph=True, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.build-date=20240521, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0) 2024-05-27T07:39:30.489 INFO:journalctl@ceph.osd.11.smithi173.stdout:May 27 07:39:30 smithi173 podman[76346]: 2024-05-27 07:39:30.491537724 +0000 UTC m=+6.008199345 container cleanup 559955021bb4bce79c5b449bd917b56fee751b8c81337cd371e14e109bc73925 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-osd-11, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, GIT_BRANCH=HEAD, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, org.label-schema.build-date=20240521, io.buildah.version=1.33.5, ceph=True, maintainer=Guillaume Abrioux , RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0) 2024-05-27T07:39:30.859 INFO:journalctl@ceph.osd.11.smithi173.stdout:May 27 07:39:30 smithi173 bash[76346]: ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-osd-11 2024-05-27T07:39:30.859 INFO:journalctl@ceph.osd.11.smithi173.stdout:May 27 07:39:30 smithi173 podman[76363]: 2024-05-27 07:39:30.734327291 +0000 UTC m=+0.525094387 container remove 559955021bb4bce79c5b449bd917b56fee751b8c81337cd371e14e109bc73925 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-osd-11, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240521, io.buildah.version=1.33.5, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, ceph=True, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, GIT_CLEAN=True, GIT_BRANCH=HEAD) 2024-05-27T07:39:31.359 INFO:journalctl@ceph.osd.11.smithi173.stdout:May 27 07:39:31 smithi173 podman[76449]: 2024-05-27 07:39:31.00245049 +0000 UTC m=+0.017670427 image pull e36ea4fc8a5807e8d2701370e350a0de3856538797f7fba64cfce7e344bd6799 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6 2024-05-27T07:39:31.360 INFO:journalctl@ceph.osd.11.smithi173.stdout:May 27 07:39:31 smithi173 podman[76449]: 2024-05-27 07:39:31.110441154 +0000 UTC m=+0.125661116 container create 27439fc66a11fdf8ecaf61c23fc435e9df2e09c649d15427b24c3e3d3a54d155 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-osd-11-deactivate, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, org.label-schema.build-date=20240521, maintainer=Guillaume Abrioux , RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, io.buildah.version=1.33.5) 2024-05-27T07:39:31.360 INFO:journalctl@ceph.osd.11.smithi173.stdout:May 27 07:39:31 smithi173 podman[76449]: 2024-05-27 07:39:31.312726864 +0000 UTC m=+0.327946771 container init 27439fc66a11fdf8ecaf61c23fc435e9df2e09c649d15427b24c3e3d3a54d155 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-osd-11-deactivate, org.label-schema.schema-version=1.0, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, org.label-schema.build-date=20240521, ceph=True, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, CEPH_POINT_RELEASE=, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2) 2024-05-27T07:39:31.360 INFO:journalctl@ceph.osd.11.smithi173.stdout:May 27 07:39:31 smithi173 podman[76449]: 2024-05-27 07:39:31.315749618 +0000 UTC m=+0.330969539 container start 27439fc66a11fdf8ecaf61c23fc435e9df2e09c649d15427b24c3e3d3a54d155 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-osd-11-deactivate, maintainer=Guillaume Abrioux , GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, GIT_CLEAN=True, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240521, GIT_BRANCH=HEAD, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2) 2024-05-27T07:39:31.859 INFO:journalctl@ceph.osd.11.smithi173.stdout:May 27 07:39:31 smithi173 podman[76449]: 2024-05-27 07:39:31.395969513 +0000 UTC m=+0.411189434 container attach 27439fc66a11fdf8ecaf61c23fc435e9df2e09c649d15427b24c3e3d3a54d155 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-osd-11-deactivate, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240521, org.label-schema.schema-version=1.0, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, ceph=True, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux ) 2024-05-27T07:39:31.859 INFO:journalctl@ceph.osd.11.smithi173.stdout:May 27 07:39:31 smithi173 podman[76449]: 2024-05-27 07:39:31.608176698 +0000 UTC m=+0.623396619 container died 27439fc66a11fdf8ecaf61c23fc435e9df2e09c649d15427b24c3e3d3a54d155 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-osd-11-deactivate, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, org.label-schema.build-date=20240521, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, ceph=True, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, GIT_CLEAN=True, maintainer=Guillaume Abrioux , RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=) 2024-05-27T07:39:32.162 INFO:journalctl@ceph.osd.11.smithi173.stdout:May 27 07:39:32 smithi173 podman[76475]: 2024-05-27 07:39:32.140785001 +0000 UTC m=+0.522543774 container remove 27439fc66a11fdf8ecaf61c23fc435e9df2e09c649d15427b24c3e3d3a54d155 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-osd-11-deactivate, ceph=True, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.build-date=20240521, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, maintainer=Guillaume Abrioux ) 2024-05-27T07:39:32.163 INFO:journalctl@ceph.osd.11.smithi173.stdout:May 27 07:39:32 smithi173 systemd[1]: ceph-73df2636-1bf8-11ef-bc9b-c7b262605968@osd.11.service: Deactivated successfully. 2024-05-27T07:39:32.163 INFO:journalctl@ceph.osd.11.smithi173.stdout:May 27 07:39:32 smithi173 systemd[1]: Stopped Ceph osd.11 for 73df2636-1bf8-11ef-bc9b-c7b262605968. 2024-05-27T07:39:32.163 INFO:journalctl@ceph.osd.11.smithi173.stdout:May 27 07:39:32 smithi173 systemd[1]: ceph-73df2636-1bf8-11ef-bc9b-c7b262605968@osd.11.service: Consumed 7min 54.653s CPU time. 2024-05-27T07:39:32.165 DEBUG:teuthology.orchestra.run.smithi173:> sudo pkill -f 'journalctl -f -n 0 -u ceph-73df2636-1bf8-11ef-bc9b-c7b262605968@osd.11.service' 2024-05-27T07:39:32.201 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-05-27T07:39:32.201 INFO:tasks.cephadm.osd.11:Stopped osd.11 2024-05-27T07:39:32.201 INFO:tasks.cephadm.mds.a:Stopping mds.a... 2024-05-27T07:39:32.201 DEBUG:teuthology.orchestra.run.smithi052:> sudo systemctl stop ceph-73df2636-1bf8-11ef-bc9b-c7b262605968@mds.a 2024-05-27T07:39:32.240 DEBUG:teuthology.orchestra.run.smithi052:> sudo pkill -f 'journalctl -f -n 0 -u ceph-73df2636-1bf8-11ef-bc9b-c7b262605968@mds.a.service' 2024-05-27T07:39:32.319 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-05-27T07:39:32.320 INFO:tasks.cephadm.mds.a:Stopped mds.a 2024-05-27T07:39:32.320 INFO:tasks.cephadm.mds.d:Stopping mds.d... 2024-05-27T07:39:32.320 DEBUG:teuthology.orchestra.run.smithi052:> sudo systemctl stop ceph-73df2636-1bf8-11ef-bc9b-c7b262605968@mds.d 2024-05-27T07:39:32.393 DEBUG:teuthology.orchestra.run.smithi052:> sudo pkill -f 'journalctl -f -n 0 -u ceph-73df2636-1bf8-11ef-bc9b-c7b262605968@mds.d.service' 2024-05-27T07:39:32.468 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-05-27T07:39:32.469 INFO:tasks.cephadm.mds.d:Stopped mds.d 2024-05-27T07:39:32.469 INFO:tasks.cephadm.mds.g:Stopping mds.g... 2024-05-27T07:39:32.469 DEBUG:teuthology.orchestra.run.smithi052:> sudo systemctl stop ceph-73df2636-1bf8-11ef-bc9b-c7b262605968@mds.g 2024-05-27T07:39:32.543 DEBUG:teuthology.orchestra.run.smithi052:> sudo pkill -f 'journalctl -f -n 0 -u ceph-73df2636-1bf8-11ef-bc9b-c7b262605968@mds.g.service' 2024-05-27T07:39:32.618 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-05-27T07:39:32.619 INFO:tasks.cephadm.mds.g:Stopped mds.g 2024-05-27T07:39:32.619 INFO:tasks.cephadm.mds.j:Stopping mds.j... 2024-05-27T07:39:32.619 DEBUG:teuthology.orchestra.run.smithi052:> sudo systemctl stop ceph-73df2636-1bf8-11ef-bc9b-c7b262605968@mds.j 2024-05-27T07:39:32.689 DEBUG:teuthology.orchestra.run.smithi052:> sudo pkill -f 'journalctl -f -n 0 -u ceph-73df2636-1bf8-11ef-bc9b-c7b262605968@mds.j.service' 2024-05-27T07:39:32.764 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-05-27T07:39:32.764 INFO:tasks.cephadm.mds.j:Stopped mds.j 2024-05-27T07:39:32.764 INFO:tasks.cephadm.mds.b:Stopping mds.b... 2024-05-27T07:39:32.764 DEBUG:teuthology.orchestra.run.smithi105:> sudo systemctl stop ceph-73df2636-1bf8-11ef-bc9b-c7b262605968@mds.b 2024-05-27T07:39:33.155 INFO:journalctl@ceph.mds.b.smithi105.stdout:May 27 07:39:32 smithi105 systemd[1]: Stopping Ceph mds.b for 73df2636-1bf8-11ef-bc9b-c7b262605968... 2024-05-27T07:39:33.156 INFO:journalctl@ceph.mds.b.smithi105.stdout:May 27 07:39:33 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-b[65543]: 2024-05-27T07:39:33.051+0000 7fc1f8856700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.b -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-05-27T07:39:33.156 INFO:journalctl@ceph.mds.b.smithi105.stdout:May 27 07:39:33 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-b[65543]: 2024-05-27T07:39:33.051+0000 7fc1f8856700 -1 mds.b *** got signal Terminated *** 2024-05-27T07:39:35.973 INFO:journalctl@ceph.mds.b.smithi105.stdout:May 27 07:39:35 smithi105 podman[72961]: 2024-05-27 07:39:35.703789085 +0000 UTC m=+2.743140643 container died 94f6b4e4fe5671bd78bfffb952250e5a090ae24c024db4e59a43aaa435caab4d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-b, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240521, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, ceph=True, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, org.label-schema.license=GPLv2) 2024-05-27T07:39:35.973 INFO:journalctl@ceph.mds.b.smithi105.stdout:May 27 07:39:35 smithi105 podman[72961]: 2024-05-27 07:39:35.976582093 +0000 UTC m=+3.015933651 container cleanup 94f6b4e4fe5671bd78bfffb952250e5a090ae24c024db4e59a43aaa435caab4d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-b, org.label-schema.build-date=20240521, CEPH_POINT_RELEASE=, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, ceph=True, io.buildah.version=1.33.5, org.label-schema.license=GPLv2) 2024-05-27T07:39:36.232 INFO:journalctl@ceph.mds.b.smithi105.stdout:May 27 07:39:35 smithi105 bash[72961]: ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-b 2024-05-27T07:39:36.395 DEBUG:teuthology.orchestra.run.smithi105:> sudo pkill -f 'journalctl -f -n 0 -u ceph-73df2636-1bf8-11ef-bc9b-c7b262605968@mds.b.service' 2024-05-27T07:39:36.428 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-05-27T07:39:36.428 INFO:tasks.cephadm.mds.b:Stopped mds.b 2024-05-27T07:39:36.428 INFO:tasks.cephadm.mds.e:Stopping mds.e... 2024-05-27T07:39:36.428 DEBUG:teuthology.orchestra.run.smithi105:> sudo systemctl stop ceph-73df2636-1bf8-11ef-bc9b-c7b262605968@mds.e 2024-05-27T07:39:36.743 INFO:journalctl@ceph.mds.e.smithi105.stdout:May 27 07:39:36 smithi105 systemd[1]: Stopping Ceph mds.e for 73df2636-1bf8-11ef-bc9b-c7b262605968... 2024-05-27T07:39:37.197 INFO:journalctl@ceph.mds.e.smithi105.stdout:May 27 07:39:36 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-e[64932]: 2024-05-27T07:39:36.746+0000 7f70fb37e700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.e -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-05-27T07:39:37.197 INFO:journalctl@ceph.mds.e.smithi105.stdout:May 27 07:39:36 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-e[64932]: 2024-05-27T07:39:36.746+0000 7f70fb37e700 -1 mds.e *** got signal Terminated *** 2024-05-27T07:39:39.947 INFO:journalctl@ceph.mds.e.smithi105.stdout:May 27 07:39:39 smithi105 podman[73083]: 2024-05-27 07:39:39.689567884 +0000 UTC m=+3.044894821 container died 04d00bde09b404f47e49ec06cf03c8343ec4e1627082b900a48012676de9f248 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-e, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, GIT_CLEAN=True, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, ceph=True, org.label-schema.build-date=20240521) 2024-05-27T07:39:40.220 INFO:journalctl@ceph.mds.e.smithi105.stdout:May 27 07:39:40 smithi105 podman[73083]: 2024-05-27 07:39:40.030354852 +0000 UTC m=+3.385681789 container cleanup 04d00bde09b404f47e49ec06cf03c8343ec4e1627082b900a48012676de9f248 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-e, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, GIT_CLEAN=True, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240521, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-05-27T07:39:40.220 INFO:journalctl@ceph.mds.e.smithi105.stdout:May 27 07:39:40 smithi105 bash[73083]: ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-e 2024-05-27T07:39:40.456 DEBUG:teuthology.orchestra.run.smithi105:> sudo pkill -f 'journalctl -f -n 0 -u ceph-73df2636-1bf8-11ef-bc9b-c7b262605968@mds.e.service' 2024-05-27T07:39:40.476 INFO:journalctl@ceph.mds.e.smithi105.stdout:May 27 07:39:40 smithi105 podman[73097]: 2024-05-27 07:39:40.22318247 +0000 UTC m=+0.529604776 container remove 04d00bde09b404f47e49ec06cf03c8343ec4e1627082b900a48012676de9f248 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-e, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, ceph=True, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240521, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, GIT_CLEAN=True) 2024-05-27T07:39:40.477 INFO:journalctl@ceph.mds.e.smithi105.stdout:May 27 07:39:40 smithi105 systemd[1]: ceph-73df2636-1bf8-11ef-bc9b-c7b262605968@mds.e.service: Deactivated successfully. 2024-05-27T07:39:40.477 INFO:journalctl@ceph.mds.e.smithi105.stdout:May 27 07:39:40 smithi105 systemd[1]: Stopped Ceph mds.e for 73df2636-1bf8-11ef-bc9b-c7b262605968. 2024-05-27T07:39:40.477 INFO:journalctl@ceph.mds.e.smithi105.stdout:May 27 07:39:40 smithi105 systemd[1]: ceph-73df2636-1bf8-11ef-bc9b-c7b262605968@mds.e.service: Consumed 19.845s CPU time. 2024-05-27T07:39:40.487 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-05-27T07:39:40.488 INFO:tasks.cephadm.mds.e:Stopped mds.e 2024-05-27T07:39:40.488 INFO:tasks.cephadm.mds.h:Stopping mds.h... 2024-05-27T07:39:40.488 DEBUG:teuthology.orchestra.run.smithi105:> sudo systemctl stop ceph-73df2636-1bf8-11ef-bc9b-c7b262605968@mds.h 2024-05-27T07:39:40.814 INFO:journalctl@ceph.mds.h.smithi105.stdout:May 27 07:39:40 smithi105 systemd[1]: Stopping Ceph mds.h for 73df2636-1bf8-11ef-bc9b-c7b262605968... 2024-05-27T07:39:41.197 INFO:journalctl@ceph.mds.h.smithi105.stdout:May 27 07:39:40 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-h[65240]: 2024-05-27T07:39:40.816+0000 7f4abf1e5700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.h -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-05-27T07:39:41.197 INFO:journalctl@ceph.mds.h.smithi105.stdout:May 27 07:39:40 smithi105 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-h[65240]: 2024-05-27T07:39:40.816+0000 7f4abf1e5700 -1 mds.h *** got signal Terminated *** 2024-05-27T07:39:44.056 INFO:journalctl@ceph.mds.h.smithi105.stdout:May 27 07:39:43 smithi105 podman[73204]: 2024-05-27 07:39:43.703830011 +0000 UTC m=+2.996604352 container died 15a481459439bc75b8aae597fe9e32292b6991f3dfbc26fee069c6c5d1a9fd7c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-h, org.label-schema.build-date=20240521, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, ceph=True, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.vendor=CentOS) 2024-05-27T07:39:44.315 INFO:journalctl@ceph.mds.h.smithi105.stdout:May 27 07:39:44 smithi105 podman[73204]: 2024-05-27 07:39:44.133809561 +0000 UTC m=+3.426583908 container cleanup 15a481459439bc75b8aae597fe9e32292b6991f3dfbc26fee069c6c5d1a9fd7c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-h, maintainer=Guillaume Abrioux , ceph=True, CEPH_POINT_RELEASE=, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, org.label-schema.build-date=20240521, GIT_BRANCH=HEAD, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, GIT_CLEAN=True, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5) 2024-05-27T07:39:44.315 INFO:journalctl@ceph.mds.h.smithi105.stdout:May 27 07:39:44 smithi105 bash[73204]: ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-h 2024-05-27T07:39:44.528 DEBUG:teuthology.orchestra.run.smithi105:> sudo pkill -f 'journalctl -f -n 0 -u ceph-73df2636-1bf8-11ef-bc9b-c7b262605968@mds.h.service' 2024-05-27T07:39:44.557 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-05-27T07:39:44.557 INFO:tasks.cephadm.mds.h:Stopped mds.h 2024-05-27T07:39:44.557 INFO:tasks.cephadm.mds.k:Stopping mds.k... 2024-05-27T07:39:44.557 DEBUG:teuthology.orchestra.run.smithi105:> sudo systemctl stop ceph-73df2636-1bf8-11ef-bc9b-c7b262605968@mds.k 2024-05-27T07:39:44.626 DEBUG:teuthology.orchestra.run.smithi105:> sudo pkill -f 'journalctl -f -n 0 -u ceph-73df2636-1bf8-11ef-bc9b-c7b262605968@mds.k.service' 2024-05-27T07:39:44.696 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-05-27T07:39:44.696 INFO:tasks.cephadm.mds.k:Stopped mds.k 2024-05-27T07:39:44.696 INFO:tasks.cephadm.mds.c:Stopping mds.c... 2024-05-27T07:39:44.697 DEBUG:teuthology.orchestra.run.smithi173:> sudo systemctl stop ceph-73df2636-1bf8-11ef-bc9b-c7b262605968@mds.c 2024-05-27T07:39:45.109 INFO:journalctl@ceph.mds.c.smithi173.stdout:May 27 07:39:44 smithi173 systemd[1]: Stopping Ceph mds.c for 73df2636-1bf8-11ef-bc9b-c7b262605968... 2024-05-27T07:39:45.610 INFO:journalctl@ceph.mds.c.smithi173.stdout:May 27 07:39:45 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-c[68050]: 2024-05-27T07:39:45.140+0000 7ff553415700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.c -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-05-27T07:39:45.656 INFO:journalctl@ceph.mds.c.smithi173.stdout:May 27 07:39:45 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-c[68050]: 2024-05-27T07:39:45.140+0000 7ff553415700 -1 mds.c *** got signal Terminated *** 2024-05-27T07:39:47.907 INFO:journalctl@ceph.mds.c.smithi173.stdout:May 27 07:39:47 smithi173 podman[76554]: 2024-05-27 07:39:47.616192981 +0000 UTC m=+2.724927601 container died 2c67bd7fd298e151c6edf858968c857239929a098508aee6f757811727548049 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-c, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240521, GIT_CLEAN=True) 2024-05-27T07:39:48.240 INFO:journalctl@ceph.mds.c.smithi173.stdout:May 27 07:39:48 smithi173 podman[76554]: 2024-05-27 07:39:48.009140144 +0000 UTC m=+3.117874757 container cleanup 2c67bd7fd298e151c6edf858968c857239929a098508aee6f757811727548049 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-c, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, ceph=True, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, org.label-schema.license=GPLv2, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240521, GIT_BRANCH=HEAD) 2024-05-27T07:39:48.240 INFO:journalctl@ceph.mds.c.smithi173.stdout:May 27 07:39:48 smithi173 bash[76554]: ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-c 2024-05-27T07:39:48.394 DEBUG:teuthology.orchestra.run.smithi173:> sudo pkill -f 'journalctl -f -n 0 -u ceph-73df2636-1bf8-11ef-bc9b-c7b262605968@mds.c.service' 2024-05-27T07:39:48.426 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-05-27T07:39:48.426 INFO:tasks.cephadm.mds.c:Stopped mds.c 2024-05-27T07:39:48.426 INFO:tasks.cephadm.mds.f:Stopping mds.f... 2024-05-27T07:39:48.427 DEBUG:teuthology.orchestra.run.smithi173:> sudo systemctl stop ceph-73df2636-1bf8-11ef-bc9b-c7b262605968@mds.f 2024-05-27T07:39:48.495 DEBUG:teuthology.orchestra.run.smithi173:> sudo pkill -f 'journalctl -f -n 0 -u ceph-73df2636-1bf8-11ef-bc9b-c7b262605968@mds.f.service' 2024-05-27T07:39:48.565 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-05-27T07:39:48.566 INFO:tasks.cephadm.mds.f:Stopped mds.f 2024-05-27T07:39:48.566 INFO:tasks.cephadm.mds.i:Stopping mds.i... 2024-05-27T07:39:48.566 DEBUG:teuthology.orchestra.run.smithi173:> sudo systemctl stop ceph-73df2636-1bf8-11ef-bc9b-c7b262605968@mds.i 2024-05-27T07:39:48.859 INFO:journalctl@ceph.mds.i.smithi173.stdout:May 27 07:39:48 smithi173 systemd[1]: Stopping Ceph mds.i for 73df2636-1bf8-11ef-bc9b-c7b262605968... 2024-05-27T07:39:49.360 INFO:journalctl@ceph.mds.i.smithi173.stdout:May 27 07:39:49 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-i[68353]: 2024-05-27T07:39:49.037+0000 7f8ac0fc9700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.i -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-05-27T07:39:49.360 INFO:journalctl@ceph.mds.i.smithi173.stdout:May 27 07:39:49 smithi173 ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-i[68353]: 2024-05-27T07:39:49.037+0000 7f8ac0fc9700 -1 mds.i *** got signal Terminated *** 2024-05-27T07:39:51.859 INFO:journalctl@ceph.mds.i.smithi173.stdout:May 27 07:39:51 smithi173 podman[76710]: 2024-05-27 07:39:51.59919829 +0000 UTC m=+2.808481676 container died 6c1bde6811ef6919ac682726a379a32a04e27699f36ca1ade9e3d442c7d97415 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-i, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, maintainer=Guillaume Abrioux , GIT_CLEAN=True, ceph=True, org.label-schema.build-date=20240521, org.label-schema.vendor=CentOS) 2024-05-27T07:39:52.144 INFO:journalctl@ceph.mds.i.smithi173.stdout:May 27 07:39:51 smithi173 podman[76710]: 2024-05-27 07:39:51.8626766 +0000 UTC m=+3.071959980 container cleanup 6c1bde6811ef6919ac682726a379a32a04e27699f36ca1ade9e3d442c7d97415 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-i, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , GIT_CLEAN=True, ceph=True, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, org.label-schema.build-date=20240521) 2024-05-27T07:39:52.144 INFO:journalctl@ceph.mds.i.smithi173.stdout:May 27 07:39:51 smithi173 bash[76710]: ceph-73df2636-1bf8-11ef-bc9b-c7b262605968-mds-i 2024-05-27T07:39:52.284 DEBUG:teuthology.orchestra.run.smithi173:> sudo pkill -f 'journalctl -f -n 0 -u ceph-73df2636-1bf8-11ef-bc9b-c7b262605968@mds.i.service' 2024-05-27T07:39:52.317 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-05-27T07:39:52.318 INFO:tasks.cephadm.mds.i:Stopped mds.i 2024-05-27T07:39:52.318 INFO:tasks.cephadm.mds.l:Stopping mds.l... 2024-05-27T07:39:52.318 DEBUG:teuthology.orchestra.run.smithi173:> sudo systemctl stop ceph-73df2636-1bf8-11ef-bc9b-c7b262605968@mds.l 2024-05-27T07:39:52.386 DEBUG:teuthology.orchestra.run.smithi173:> sudo pkill -f 'journalctl -f -n 0 -u ceph-73df2636-1bf8-11ef-bc9b-c7b262605968@mds.l.service' 2024-05-27T07:39:52.457 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-05-27T07:39:52.458 INFO:tasks.cephadm.mds.l:Stopped mds.l 2024-05-27T07:39:52.458 DEBUG:teuthology.orchestra.run.smithi052:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid 73df2636-1bf8-11ef-bc9b-c7b262605968 --force --keep-logs 2024-05-27T07:39:52.660 INFO:teuthology.orchestra.run.smithi052.stdout:Deleting cluster with fsid: 73df2636-1bf8-11ef-bc9b-c7b262605968 2024-05-27T07:39:55.193 DEBUG:teuthology.orchestra.run.smithi105:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid 73df2636-1bf8-11ef-bc9b-c7b262605968 --force --keep-logs 2024-05-27T07:39:55.386 INFO:teuthology.orchestra.run.smithi105.stdout:Deleting cluster with fsid: 73df2636-1bf8-11ef-bc9b-c7b262605968 2024-05-27T07:39:57.993 DEBUG:teuthology.orchestra.run.smithi173:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid 73df2636-1bf8-11ef-bc9b-c7b262605968 --force --keep-logs 2024-05-27T07:39:58.186 INFO:teuthology.orchestra.run.smithi173.stdout:Deleting cluster with fsid: 73df2636-1bf8-11ef-bc9b-c7b262605968 2024-05-27T07:40:00.507 DEBUG:teuthology.orchestra.run.smithi052:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-05-27T07:40:00.541 DEBUG:teuthology.orchestra.run.smithi105:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-05-27T07:40:00.573 DEBUG:teuthology.orchestra.run.smithi173:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-05-27T07:40:00.599 INFO:tasks.cephadm:Archiving crash dumps... 2024-05-27T07:40:00.601 DEBUG:teuthology.misc:Transferring archived files from smithi052:/var/lib/ceph/73df2636-1bf8-11ef-bc9b-c7b262605968/crash to /home/teuthworker/archive/vshankar-2024-05-27_05:52:49-fs-wip-vshankar-testing-20240521.063558-debug-testing-default-smithi/7727648/remote/smithi052/crash 2024-05-27T07:40:00.602 DEBUG:teuthology.orchestra.run.smithi052:> sudo tar c -f - -C /var/lib/ceph/73df2636-1bf8-11ef-bc9b-c7b262605968/crash -- . 2024-05-27T07:40:00.632 INFO:teuthology.orchestra.run.smithi052.stderr:tar: /var/lib/ceph/73df2636-1bf8-11ef-bc9b-c7b262605968/crash: Cannot open: No such file or directory 2024-05-27T07:40:00.632 INFO:teuthology.orchestra.run.smithi052.stderr:tar: Error is not recoverable: exiting now 2024-05-27T07:40:00.634 DEBUG:teuthology.misc:Transferring archived files from smithi105:/var/lib/ceph/73df2636-1bf8-11ef-bc9b-c7b262605968/crash to /home/teuthworker/archive/vshankar-2024-05-27_05:52:49-fs-wip-vshankar-testing-20240521.063558-debug-testing-default-smithi/7727648/remote/smithi105/crash 2024-05-27T07:40:00.635 DEBUG:teuthology.orchestra.run.smithi105:> sudo tar c -f - -C /var/lib/ceph/73df2636-1bf8-11ef-bc9b-c7b262605968/crash -- . 2024-05-27T07:40:00.675 INFO:teuthology.orchestra.run.smithi105.stderr:tar: /var/lib/ceph/73df2636-1bf8-11ef-bc9b-c7b262605968/crash: Cannot open: No such file or directory 2024-05-27T07:40:00.675 INFO:teuthology.orchestra.run.smithi105.stderr:tar: Error is not recoverable: exiting now 2024-05-27T07:40:00.677 DEBUG:teuthology.misc:Transferring archived files from smithi173:/var/lib/ceph/73df2636-1bf8-11ef-bc9b-c7b262605968/crash to /home/teuthworker/archive/vshankar-2024-05-27_05:52:49-fs-wip-vshankar-testing-20240521.063558-debug-testing-default-smithi/7727648/remote/smithi173/crash 2024-05-27T07:40:00.678 DEBUG:teuthology.orchestra.run.smithi173:> sudo tar c -f - -C /var/lib/ceph/73df2636-1bf8-11ef-bc9b-c7b262605968/crash -- . 2024-05-27T07:40:00.728 INFO:teuthology.orchestra.run.smithi173.stderr:tar: /var/lib/ceph/73df2636-1bf8-11ef-bc9b-c7b262605968/crash: Cannot open: No such file or directory 2024-05-27T07:40:00.728 INFO:teuthology.orchestra.run.smithi173.stderr:tar: Error is not recoverable: exiting now 2024-05-27T07:40:00.729 INFO:tasks.cephadm:Checking cluster log for badness... 2024-05-27T07:40:00.729 DEBUG:teuthology.orchestra.run.smithi052:> sudo egrep '\[ERR\]|\[WRN\]|\[SEC\]' /var/log/ceph/73df2636-1bf8-11ef-bc9b-c7b262605968/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v FS_DEGRADED | egrep -v FS_INLINE_DATA_DEPRECATED | egrep -v FS_WITH_FAILED_MDS | egrep -v MDS_ALL_DOWN | egrep -v MDS_DAMAGE | egrep -v MDS_DEGRADED | egrep -v MDS_FAILED | egrep -v MDS_INSUFFICIENT_STANDBY | egrep -v MDS_UP_LESS_THAN_MAX | egrep -v 'filesystem is online with fewer MDS than max_mds' | egrep -v POOL_APP_NOT_ENABLED | egrep -v 'do not have an application enabled' | egrep -v 'overall HEALTH_' | egrep -v 'Replacing daemon' | egrep -v 'deprecated feature inline_data' | egrep -v 'overall HEALTH_' | egrep -v '\(OSD_DOWN\)' | egrep -v '\(OSD_' | egrep -v 'but it is still running' | egrep -v 'is not responding' | egrep -v PG_AVAILABILITY | egrep -v PG_DEGRADED | egrep -v 'Reduced data availability' | egrep -v 'Degraded data redundancy' | egrep -v 'slow metadata IO' | egrep -v SLOW_OPS | egrep -v 'slow request' | head -n 1 2024-05-27T07:40:00.773 INFO:teuthology.orchestra.run.smithi052.stdout:2024-05-27T07:30:12.723703+0000 mds.b (mds.0) 24 : cluster [WRN] Scrub error on inode 0x10000003fad (/volumes/qa/sv_1/95e9ac73-825f-48e3-998b-133d94d96559/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-58) see mds.b log and `damage ls` output for details 2024-05-27T07:40:00.773 WARNING:tasks.cephadm:Found errors (ERR|WRN|SEC) in cluster log 2024-05-27T07:40:00.774 INFO:tasks.cephadm:Compressing logs... 2024-05-27T07:40:00.774 DEBUG:teuthology.orchestra.run.smithi052:> time sudo find /var/log/ceph /var/log/rbd-target-api -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-05-27T07:40:00.816 DEBUG:teuthology.orchestra.run.smithi105:> time sudo find /var/log/ceph /var/log/rbd-target-api -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-05-27T07:40:00.818 DEBUG:teuthology.orchestra.run.smithi173:> time sudo find /var/log/ceph /var/log/rbd-target-api -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-05-27T07:40:00.842 INFO:teuthology.orchestra.run.smithi173.stderr:find: ‘/var/log/rbd-target-api’gzip -5 --verbose -- /var/log/ceph/73df2636-1bf8-11ef-bc9b-c7b262605968/ceph.log 2024-05-27T07:40:00.842 INFO:teuthology.orchestra.run.smithi173.stderr:: No such file or directory 2024-05-27T07:40:00.842 INFO:teuthology.orchestra.run.smithi173.stderr:gzip -5 --verbose -- /var/log/ceph/73df2636-1bf8-11ef-bc9b-c7b262605968/ceph-osd.2.log 2024-05-27T07:40:00.842 INFO:teuthology.orchestra.run.smithi173.stderr:/var/log/ceph/73df2636-1bf8-11ef-bc9b-c7b262605968/ceph.log: gzip -5 --verbose -- /var/log/ceph/73df2636-1bf8-11ef-bc9b-c7b262605968/ceph-mds.f.log 2024-05-27T07:40:00.842 INFO:teuthology.orchestra.run.smithi173.stderr:/var/log/ceph/73df2636-1bf8-11ef-bc9b-c7b262605968/ceph-osd.2.log: gzip -5 --verbose -- /var/log/ceph/73df2636-1bf8-11ef-bc9b-c7b262605968/ceph.audit.log 2024-05-27T07:40:00.846 INFO:teuthology.orchestra.run.smithi173.stderr:/var/log/ceph/73df2636-1bf8-11ef-bc9b-c7b262605968/ceph-mds.f.log: gzip -5 --verbose -- /var/log/ceph/73df2636-1bf8-11ef-bc9b-c7b262605968/ceph-osd.5.log 2024-05-27T07:40:00.847 INFO:teuthology.orchestra.run.smithi173.stderr:/var/log/ceph/73df2636-1bf8-11ef-bc9b-c7b262605968/ceph.audit.log: gzip -5 --verbose -- /var/log/ceph/73df2636-1bf8-11ef-bc9b-c7b262605968/ceph-mds.c.log 2024-05-27T07:40:00.847 INFO:teuthology.orchestra.run.smithi173.stderr:/var/log/ceph/73df2636-1bf8-11ef-bc9b-c7b262605968/ceph-osd.5.log: gzip -5 --verbose -- /var/log/ceph/73df2636-1bf8-11ef-bc9b-c7b262605968/ceph-mon.c.log 2024-05-27T07:40:00.847 INFO:teuthology.orchestra.run.smithi173.stderr:/var/log/ceph/73df2636-1bf8-11ef-bc9b-c7b262605968/ceph-mds.c.log: gzip -5 --verbose -- /var/log/ceph/73df2636-1bf8-11ef-bc9b-c7b262605968/ceph-mgr.z.log 2024-05-27T07:40:00.847 INFO:teuthology.orchestra.run.smithi173.stderr:/var/log/ceph/73df2636-1bf8-11ef-bc9b-c7b262605968/ceph-mon.c.log: 86.8% 90.4% -- replaced with /var/log/ceph/73df2636-1bf8-11ef-bc9b-c7b262605968/ceph.audit.log.gz 2024-05-27T07:40:00.847 INFO:teuthology.orchestra.run.smithi173.stderr: -- replaced with /var/log/ceph/73df2636-1bf8-11ef-bc9b-c7b262605968/ceph.log.gz 2024-05-27T07:40:00.849 INFO:teuthology.orchestra.run.smithi052.stderr:find: ‘/var/log/rbd-target-api’: No such file or directory 2024-05-27T07:40:00.849 INFO:teuthology.orchestra.run.smithi052.stderr:gzip -5 --verbose -- /var/log/ceph/73df2636-1bf8-11ef-bc9b-c7b262605968/ceph.log 2024-05-27T07:40:00.849 INFO:teuthology.orchestra.run.smithi052.stderr:gzip -5 --verbose -- /var/log/ceph/73df2636-1bf8-11ef-bc9b-c7b262605968/ceph.audit.log 2024-05-27T07:40:00.849 INFO:teuthology.orchestra.run.smithi052.stderr:gzip -5 --verbose -- /var/log/ceph/73df2636-1bf8-11ef-bc9b-c7b262605968/ceph-mds.g.log 2024-05-27T07:40:00.849 INFO:teuthology.orchestra.run.smithi052.stderr:/var/log/ceph/73df2636-1bf8-11ef-bc9b-c7b262605968/ceph.log: /var/log/ceph/73df2636-1bf8-11ef-bc9b-c7b262605968/ceph.audit.log: gzip -5 --verbose -- /var/log/ceph/73df2636-1bf8-11ef-bc9b-c7b262605968/ceph-mds.j.log 2024-05-27T07:40:00.849 INFO:teuthology.orchestra.run.smithi052.stderr:/var/log/ceph/73df2636-1bf8-11ef-bc9b-c7b262605968/ceph-mds.g.log: gzip -5 --verbose -- /var/log/ceph/73df2636-1bf8-11ef-bc9b-c7b262605968/ceph-osd.3.log 2024-05-27T07:40:00.849 INFO:teuthology.orchestra.run.smithi052.stderr:/var/log/ceph/73df2636-1bf8-11ef-bc9b-c7b262605968/ceph-mds.j.log: gzip -5 --verbose -- /var/log/ceph/73df2636-1bf8-11ef-bc9b-c7b262605968/ceph-osd.0.log 2024-05-27T07:40:00.850 INFO:teuthology.orchestra.run.smithi052.stderr:/var/log/ceph/73df2636-1bf8-11ef-bc9b-c7b262605968/ceph-osd.3.log: gzip -5 --verbose -- /var/log/ceph/73df2636-1bf8-11ef-bc9b-c7b262605968/ceph-mgr.x.log 2024-05-27T07:40:00.850 INFO:teuthology.orchestra.run.smithi052.stderr:/var/log/ceph/73df2636-1bf8-11ef-bc9b-c7b262605968/ceph-osd.0.log: gzip -5 --verbose -- /var/log/ceph/73df2636-1bf8-11ef-bc9b-c7b262605968/ceph-mon.a.log 2024-05-27T07:40:00.850 INFO:teuthology.orchestra.run.smithi052.stderr:/var/log/ceph/73df2636-1bf8-11ef-bc9b-c7b262605968/ceph-mgr.x.log: gzip -5 --verbose -- /var/log/ceph/73df2636-1bf8-11ef-bc9b-c7b262605968/ceph-osd.6.log 2024-05-27T07:40:00.850 INFO:teuthology.orchestra.run.smithi052.stderr:/var/log/ceph/73df2636-1bf8-11ef-bc9b-c7b262605968/ceph-mon.a.log: 90.3% -- replaced with /var/log/ceph/73df2636-1bf8-11ef-bc9b-c7b262605968/ceph.audit.log.gz 2024-05-27T07:40:00.850 INFO:teuthology.orchestra.run.smithi052.stderr: 86.8% -- replaced with /var/log/ceph/73df2636-1bf8-11ef-bc9b-c7b262605968/ceph.log.gz 2024-05-27T07:40:00.850 INFO:teuthology.orchestra.run.smithi105.stderr:find: ‘/var/log/rbd-target-api’: No such file or directory 2024-05-27T07:40:00.851 INFO:teuthology.orchestra.run.smithi105.stderr:gzip -5 --verbose -- /var/log/ceph/73df2636-1bf8-11ef-bc9b-c7b262605968/ceph.log 2024-05-27T07:40:00.851 INFO:teuthology.orchestra.run.smithi105.stderr:gzip -5 --verbose -- /var/log/ceph/73df2636-1bf8-11ef-bc9b-c7b262605968/ceph.audit.log 2024-05-27T07:40:00.851 INFO:teuthology.orchestra.run.smithi105.stderr:/var/log/ceph/73df2636-1bf8-11ef-bc9b-c7b262605968/ceph.log: gzip -5 --verbose -- /var/log/ceph/73df2636-1bf8-11ef-bc9b-c7b262605968/ceph-osd.7.log 2024-05-27T07:40:00.851 INFO:teuthology.orchestra.run.smithi105.stderr:/var/log/ceph/73df2636-1bf8-11ef-bc9b-c7b262605968/ceph.audit.log: gzip -5 --verbose -- /var/log/ceph/73df2636-1bf8-11ef-bc9b-c7b262605968/ceph-mds.b.log 2024-05-27T07:40:00.854 INFO:teuthology.orchestra.run.smithi173.stderr:gzip -5 --verbose -- /var/log/ceph/73df2636-1bf8-11ef-bc9b-c7b262605968/ceph-mds.i.log 2024-05-27T07:40:00.854 INFO:teuthology.orchestra.run.smithi173.stderr:/var/log/ceph/73df2636-1bf8-11ef-bc9b-c7b262605968/ceph-mgr.z.log: gzip -5 --verbose -- /var/log/ceph/73df2636-1bf8-11ef-bc9b-c7b262605968/ceph.cephadm.log 2024-05-27T07:40:00.856 INFO:teuthology.orchestra.run.smithi105.stderr:/var/log/ceph/73df2636-1bf8-11ef-bc9b-c7b262605968/ceph-osd.7.log: gzip -5 --verbose -- /var/log/ceph/73df2636-1bf8-11ef-bc9b-c7b262605968/ceph-osd.4.log 2024-05-27T07:40:00.856 INFO:teuthology.orchestra.run.smithi105.stderr:/var/log/ceph/73df2636-1bf8-11ef-bc9b-c7b262605968/ceph-mds.b.log: gzip -5 --verbose -- /var/log/ceph/73df2636-1bf8-11ef-bc9b-c7b262605968/ceph-osd.1.log 2024-05-27T07:40:00.856 INFO:teuthology.orchestra.run.smithi105.stderr:/var/log/ceph/73df2636-1bf8-11ef-bc9b-c7b262605968/ceph-osd.4.log: gzip -5 --verbose -- /var/log/ceph/73df2636-1bf8-11ef-bc9b-c7b262605968/ceph-mgr.y.log 2024-05-27T07:40:00.856 INFO:teuthology.orchestra.run.smithi105.stderr:/var/log/ceph/73df2636-1bf8-11ef-bc9b-c7b262605968/ceph-osd.1.log: gzip -5 --verbose -- /var/log/ceph/73df2636-1bf8-11ef-bc9b-c7b262605968/ceph-mds.h.log 2024-05-27T07:40:00.856 INFO:teuthology.orchestra.run.smithi105.stderr:/var/log/ceph/73df2636-1bf8-11ef-bc9b-c7b262605968/ceph-mgr.y.log: gzip -5 --verbose -- /var/log/ceph/73df2636-1bf8-11ef-bc9b-c7b262605968/ceph-osd.10.log 2024-05-27T07:40:00.856 INFO:teuthology.orchestra.run.smithi105.stderr:/var/log/ceph/73df2636-1bf8-11ef-bc9b-c7b262605968/ceph-mds.h.log: gzip -5 --verbose -- /var/log/ceph/73df2636-1bf8-11ef-bc9b-c7b262605968/ceph.cephadm.log 2024-05-27T07:40:00.856 INFO:teuthology.orchestra.run.smithi105.stderr:/var/log/ceph/73df2636-1bf8-11ef-bc9b-c7b262605968/ceph-osd.10.log: gzip -5 --verbose -- /var/log/ceph/73df2636-1bf8-11ef-bc9b-c7b262605968/ceph-mds.e.log 2024-05-27T07:40:00.857 INFO:teuthology.orchestra.run.smithi105.stderr:/var/log/ceph/73df2636-1bf8-11ef-bc9b-c7b262605968/ceph.cephadm.log: gzip -5 --verbose -- /var/log/ceph/73df2636-1bf8-11ef-bc9b-c7b262605968/ceph-volume.log 2024-05-27T07:40:00.857 INFO:teuthology.orchestra.run.smithi105.stderr:/var/log/ceph/73df2636-1bf8-11ef-bc9b-c7b262605968/ceph-mds.e.log: gzip -5 --verbose -- /var/log/ceph/73df2636-1bf8-11ef-bc9b-c7b262605968/ceph-mon.b.log 2024-05-27T07:40:00.857 INFO:teuthology.orchestra.run.smithi105.stderr:/var/log/ceph/73df2636-1bf8-11ef-bc9b-c7b262605968/ceph-volume.log: gzip -5 --verbose -- /var/log/ceph/73df2636-1bf8-11ef-bc9b-c7b262605968/ceph-mds.k.log 2024-05-27T07:40:00.857 INFO:teuthology.orchestra.run.smithi105.stderr:/var/log/ceph/73df2636-1bf8-11ef-bc9b-c7b262605968/ceph-mon.b.log: gzip -5 --verbose -- /var/log/ceph/cephadm.log 2024-05-27T07:40:00.859 INFO:teuthology.orchestra.run.smithi052.stderr:gzip -5 --verbose -- /var/log/ceph/73df2636-1bf8-11ef-bc9b-c7b262605968/ceph.cephadm.log 2024-05-27T07:40:00.859 INFO:teuthology.orchestra.run.smithi052.stderr:/var/log/ceph/73df2636-1bf8-11ef-bc9b-c7b262605968/ceph-osd.6.log: gzip -5 --verbose -- /var/log/ceph/73df2636-1bf8-11ef-bc9b-c7b262605968/ceph-mds.a.log 2024-05-27T07:40:00.860 INFO:teuthology.orchestra.run.smithi052.stderr:/var/log/ceph/73df2636-1bf8-11ef-bc9b-c7b262605968/ceph.cephadm.log: 84.3% -- replaced with /var/log/ceph/73df2636-1bf8-11ef-bc9b-c7b262605968/ceph.cephadm.log.gz 2024-05-27T07:40:00.861 INFO:teuthology.orchestra.run.smithi173.stderr:/var/log/ceph/73df2636-1bf8-11ef-bc9b-c7b262605968/ceph-mds.i.log: 92.1% -- replaced with /var/log/ceph/73df2636-1bf8-11ef-bc9b-c7b262605968/ceph-mgr.z.log.gz 2024-05-27T07:40:00.862 INFO:teuthology.orchestra.run.smithi173.stderr:gzip -5 --verbose -- /var/log/ceph/73df2636-1bf8-11ef-bc9b-c7b262605968/ceph-osd.11.log 2024-05-27T07:40:00.863 INFO:teuthology.orchestra.run.smithi173.stderr:/var/log/ceph/73df2636-1bf8-11ef-bc9b-c7b262605968/ceph.cephadm.log: 83.9% -- replaced with /var/log/ceph/73df2636-1bf8-11ef-bc9b-c7b262605968/ceph.cephadm.log.gz 2024-05-27T07:40:00.863 INFO:teuthology.orchestra.run.smithi173.stderr:gzip -5 --verbose -- /var/log/ceph/73df2636-1bf8-11ef-bc9b-c7b262605968/ceph-osd.8.log 2024-05-27T07:40:00.873 INFO:teuthology.orchestra.run.smithi052.stderr:gzip -5 --verbose -- /var/log/ceph/73df2636-1bf8-11ef-bc9b-c7b262605968/ceph-volume.log 2024-05-27T07:40:00.876 INFO:teuthology.orchestra.run.smithi173.stderr:/var/log/ceph/73df2636-1bf8-11ef-bc9b-c7b262605968/ceph-osd.11.log: gzip -5 --verbose -- /var/log/ceph/73df2636-1bf8-11ef-bc9b-c7b262605968/ceph-volume.log 2024-05-27T07:40:00.885 INFO:teuthology.orchestra.run.smithi052.stderr:/var/log/ceph/73df2636-1bf8-11ef-bc9b-c7b262605968/ceph-mds.a.log: gzip -5 --verbose -- /var/log/ceph/73df2636-1bf8-11ef-bc9b-c7b262605968/ceph-mds.d.log 2024-05-27T07:40:00.890 INFO:teuthology.orchestra.run.smithi173.stderr:/var/log/ceph/73df2636-1bf8-11ef-bc9b-c7b262605968/ceph-osd.8.log: gzip -5 --verbose -- /var/log/ceph/73df2636-1bf8-11ef-bc9b-c7b262605968/ceph-mds.l.log 2024-05-27T07:40:00.894 INFO:teuthology.orchestra.run.smithi052.stderr:/var/log/ceph/73df2636-1bf8-11ef-bc9b-c7b262605968/ceph-volume.log: gzip -5 --verbose -- /var/log/ceph/73df2636-1bf8-11ef-bc9b-c7b262605968/ceph-osd.9.log 2024-05-27T07:40:00.898 INFO:teuthology.orchestra.run.smithi173.stderr:/var/log/ceph/73df2636-1bf8-11ef-bc9b-c7b262605968/ceph-volume.log: gzip -5 --verbose -- /var/log/ceph/cephadm.log 2024-05-27T07:40:00.900 INFO:teuthology.orchestra.run.smithi052.stderr:/var/log/ceph/73df2636-1bf8-11ef-bc9b-c7b262605968/ceph-mds.d.log: gzip -5 --verbose -- /var/log/ceph/cephadm.log 2024-05-27T07:40:00.901 INFO:teuthology.orchestra.run.smithi173.stderr:/var/log/ceph/73df2636-1bf8-11ef-bc9b-c7b262605968/ceph-mds.l.log: /var/log/ceph/cephadm.log: 92.2% -- replaced with /var/log/ceph/cephadm.log.gz 2024-05-27T07:40:00.907 INFO:teuthology.orchestra.run.smithi052.stderr:/var/log/ceph/73df2636-1bf8-11ef-bc9b-c7b262605968/ceph-osd.9.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.log 2024-05-27T07:40:00.915 INFO:teuthology.orchestra.run.smithi052.stderr:/var/log/ceph/cephadm.log: /var/log/ceph/ceph-client.admin.log: 91.1% -- replaced with /var/log/ceph/73df2636-1bf8-11ef-bc9b-c7b262605968/ceph-volume.log.gz 2024-05-27T07:40:00.916 INFO:teuthology.orchestra.run.smithi173.stderr: 91.0% -- replaced with /var/log/ceph/73df2636-1bf8-11ef-bc9b-c7b262605968/ceph-volume.log.gz 2024-05-27T07:40:00.920 INFO:teuthology.orchestra.run.smithi052.stderr: 92.7% -- replaced with /var/log/ceph/ceph-client.admin.log.gz 2024-05-27T07:40:00.931 INFO:teuthology.orchestra.run.smithi052.stderr: 94.2% -- replaced with /var/log/ceph/cephadm.log.gz 2024-05-27T07:40:00.958 INFO:teuthology.orchestra.run.smithi105.stderr:/var/log/ceph/73df2636-1bf8-11ef-bc9b-c7b262605968/ceph-mds.k.log: /var/log/ceph/cephadm.log: 91.0% -- replaced with /var/log/ceph/73df2636-1bf8-11ef-bc9b-c7b262605968/ceph-volume.log.gz 2024-05-27T07:40:01.023 INFO:teuthology.orchestra.run.smithi105.stderr: 83.9% -- replaced with /var/log/ceph/73df2636-1bf8-11ef-bc9b-c7b262605968/ceph.cephadm.log.gz 2024-05-27T07:40:01.024 INFO:teuthology.orchestra.run.smithi105.stderr: 90.4% -- replaced with /var/log/ceph/73df2636-1bf8-11ef-bc9b-c7b262605968/ceph.audit.log.gz 2024-05-27T07:40:01.087 INFO:teuthology.orchestra.run.smithi105.stderr: 91.8% -- replaced with /var/log/ceph/cephadm.log.gz 2024-05-27T07:40:01.156 INFO:teuthology.orchestra.run.smithi052.stderr: 94.7% -- replaced with /var/log/ceph/73df2636-1bf8-11ef-bc9b-c7b262605968/ceph-mds.g.log.gz 2024-05-27T07:40:01.185 INFO:teuthology.orchestra.run.smithi052.stderr: 94.8% -- replaced with /var/log/ceph/73df2636-1bf8-11ef-bc9b-c7b262605968/ceph-mds.j.log.gz 2024-05-27T07:40:01.185 INFO:teuthology.orchestra.run.smithi105.stderr: 92.1% -- replaced with /var/log/ceph/73df2636-1bf8-11ef-bc9b-c7b262605968/ceph-mgr.y.log.gz 2024-05-27T07:40:01.190 INFO:teuthology.orchestra.run.smithi105.stderr: 86.8% -- replaced with /var/log/ceph/73df2636-1bf8-11ef-bc9b-c7b262605968/ceph.log.gz 2024-05-27T07:40:01.200 INFO:teuthology.orchestra.run.smithi052.stderr: 94.4% -- replaced with /var/log/ceph/73df2636-1bf8-11ef-bc9b-c7b262605968/ceph-mds.d.log.gz 2024-05-27T07:40:01.303 INFO:teuthology.orchestra.run.smithi052.stderr: 90.3% -- replaced with /var/log/ceph/73df2636-1bf8-11ef-bc9b-c7b262605968/ceph-mgr.x.log.gz 2024-05-27T07:40:01.315 INFO:teuthology.orchestra.run.smithi173.stderr: 91.5% -- replaced with /var/log/ceph/73df2636-1bf8-11ef-bc9b-c7b262605968/ceph-mon.c.log.gz 2024-05-27T07:40:01.367 INFO:teuthology.orchestra.run.smithi173.stderr: 95.3% -- replaced with /var/log/ceph/73df2636-1bf8-11ef-bc9b-c7b262605968/ceph-mds.l.log.gz 2024-05-27T07:40:01.388 INFO:teuthology.orchestra.run.smithi052.stderr: 94.9% -- replaced with /var/log/ceph/73df2636-1bf8-11ef-bc9b-c7b262605968/ceph-mds.a.log.gz 2024-05-27T07:40:01.793 INFO:teuthology.orchestra.run.smithi173.stderr: 95.3% -- replaced with /var/log/ceph/73df2636-1bf8-11ef-bc9b-c7b262605968/ceph-mds.f.log.gz 2024-05-27T07:40:02.283 INFO:teuthology.orchestra.run.smithi052.stderr: 90.2% -- replaced with /var/log/ceph/73df2636-1bf8-11ef-bc9b-c7b262605968/ceph-mon.a.log.gz 2024-05-27T07:40:04.772 INFO:teuthology.orchestra.run.smithi105.stderr: 91.5% -- replaced with /var/log/ceph/73df2636-1bf8-11ef-bc9b-c7b262605968/ceph-mon.b.log.gz 2024-05-27T07:40:06.317 INFO:teuthology.orchestra.run.smithi173.stderr: 94.1% -- replaced with /var/log/ceph/73df2636-1bf8-11ef-bc9b-c7b262605968/ceph-mds.c.log.gz 2024-05-27T07:40:09.874 INFO:teuthology.orchestra.run.smithi105.stderr: 94.5% -- replaced with /var/log/ceph/73df2636-1bf8-11ef-bc9b-c7b262605968/ceph-mds.k.log.gz 2024-05-27T07:40:10.884 INFO:teuthology.orchestra.run.smithi173.stderr: 94.4% -- replaced with /var/log/ceph/73df2636-1bf8-11ef-bc9b-c7b262605968/ceph-mds.i.log.gz 2024-05-27T07:40:33.408 INFO:teuthology.orchestra.run.smithi173.stderr: 91.6% -- replaced with /var/log/ceph/73df2636-1bf8-11ef-bc9b-c7b262605968/ceph-osd.8.log.gz 2024-05-27T07:40:41.561 INFO:teuthology.orchestra.run.smithi052.stderr: 91.2% -- replaced with /var/log/ceph/73df2636-1bf8-11ef-bc9b-c7b262605968/ceph-osd.0.log.gz 2024-05-27T07:40:44.682 INFO:teuthology.orchestra.run.smithi173.stderr: 91.4% -- replaced with /var/log/ceph/73df2636-1bf8-11ef-bc9b-c7b262605968/ceph-osd.11.log.gz 2024-05-27T07:40:47.900 INFO:teuthology.orchestra.run.smithi052.stderr: 91.2% -- replaced with /var/log/ceph/73df2636-1bf8-11ef-bc9b-c7b262605968/ceph-osd.6.log.gz 2024-05-27T07:40:48.702 INFO:teuthology.orchestra.run.smithi052.stderr: 91.3% -- replaced with /var/log/ceph/73df2636-1bf8-11ef-bc9b-c7b262605968/ceph-osd.3.log.gz 2024-05-27T07:40:48.919 INFO:teuthology.orchestra.run.smithi173.stderr: 91.5% -- replaced with /var/log/ceph/73df2636-1bf8-11ef-bc9b-c7b262605968/ceph-osd.2.log.gz 2024-05-27T07:40:57.555 INFO:teuthology.orchestra.run.smithi052.stderr: 91.3% -- replaced with /var/log/ceph/73df2636-1bf8-11ef-bc9b-c7b262605968/ceph-osd.9.log.gz 2024-05-27T07:40:57.557 INFO:teuthology.orchestra.run.smithi052.stderr: 2024-05-27T07:40:57.558 INFO:teuthology.orchestra.run.smithi052.stderr:real 0m56.727s 2024-05-27T07:40:57.558 INFO:teuthology.orchestra.run.smithi052.stderr:user 3m9.208s 2024-05-27T07:40:57.558 INFO:teuthology.orchestra.run.smithi052.stderr:sys 0m5.504s 2024-05-27T07:40:59.174 INFO:teuthology.orchestra.run.smithi173.stderr: 91.3% -- replaced with /var/log/ceph/73df2636-1bf8-11ef-bc9b-c7b262605968/ceph-osd.5.log.gz 2024-05-27T07:40:59.176 INFO:teuthology.orchestra.run.smithi173.stderr: 2024-05-27T07:40:59.176 INFO:teuthology.orchestra.run.smithi173.stderr:real 0m58.344s 2024-05-27T07:40:59.176 INFO:teuthology.orchestra.run.smithi173.stderr:user 3m13.162s 2024-05-27T07:40:59.176 INFO:teuthology.orchestra.run.smithi173.stderr:sys 0m5.990s 2024-05-27T07:41:42.116 INFO:teuthology.orchestra.run.smithi105.stderr: 94.6% -- replaced with /var/log/ceph/73df2636-1bf8-11ef-bc9b-c7b262605968/ceph-mds.e.log.gz 2024-05-27T07:42:53.380 INFO:teuthology.orchestra.run.smithi105.stderr: 94.3% -- replaced with /var/log/ceph/73df2636-1bf8-11ef-bc9b-c7b262605968/ceph-mds.h.log.gz 2024-05-27T07:46:02.646 INFO:teuthology.orchestra.run.smithi105.stderr: 91.5% -- replaced with /var/log/ceph/73df2636-1bf8-11ef-bc9b-c7b262605968/ceph-osd.1.log.gz 2024-05-27T07:46:21.971 INFO:teuthology.orchestra.run.smithi105.stderr: 91.6% -- replaced with /var/log/ceph/73df2636-1bf8-11ef-bc9b-c7b262605968/ceph-osd.10.log.gz 2024-05-27T07:48:40.247 INFO:teuthology.orchestra.run.smithi105.stderr: 91.3% -- replaced with /var/log/ceph/73df2636-1bf8-11ef-bc9b-c7b262605968/ceph-osd.7.log.gz 2024-05-27T07:48:57.822 INFO:teuthology.orchestra.run.smithi105.stderr: 91.5% -- replaced with /var/log/ceph/73df2636-1bf8-11ef-bc9b-c7b262605968/ceph-osd.4.log.gz 2024-05-27T07:51:58.169 INFO:teuthology.orchestra.run.smithi105.stderr: 92.9% -- replaced with /var/log/ceph/73df2636-1bf8-11ef-bc9b-c7b262605968/ceph-mds.b.log.gz 2024-05-27T07:51:58.171 INFO:teuthology.orchestra.run.smithi105.stderr: 2024-05-27T07:51:58.171 INFO:teuthology.orchestra.run.smithi105.stderr:real 11m57.337s 2024-05-27T07:51:58.171 INFO:teuthology.orchestra.run.smithi105.stderr:user 6m25.208s 2024-05-27T07:51:58.171 INFO:teuthology.orchestra.run.smithi105.stderr:sys 0m16.265s 2024-05-27T07:51:58.171 INFO:tasks.cephadm:Archiving logs... 2024-05-27T07:51:58.172 DEBUG:teuthology.misc:Transferring archived files from smithi052:/var/log/ceph to /home/teuthworker/archive/vshankar-2024-05-27_05:52:49-fs-wip-vshankar-testing-20240521.063558-debug-testing-default-smithi/7727648/remote/smithi052/log 2024-05-27T07:51:58.173 DEBUG:teuthology.orchestra.run.smithi052:> sudo tar c -f - -C /var/log/ceph -- . 2024-05-27T07:52:29.797 DEBUG:teuthology.misc:Transferring archived files from smithi105:/var/log/ceph to /home/teuthworker/archive/vshankar-2024-05-27_05:52:49-fs-wip-vshankar-testing-20240521.063558-debug-testing-default-smithi/7727648/remote/smithi105/log 2024-05-27T07:52:29.798 DEBUG:teuthology.orchestra.run.smithi105:> sudo tar c -f - -C /var/log/ceph -- . 2024-05-27T07:53:32.028 DEBUG:teuthology.misc:Transferring archived files from smithi173:/var/log/ceph to /home/teuthworker/archive/vshankar-2024-05-27_05:52:49-fs-wip-vshankar-testing-20240521.063558-debug-testing-default-smithi/7727648/remote/smithi173/log 2024-05-27T07:53:32.029 DEBUG:teuthology.orchestra.run.smithi173:> sudo tar c -f - -C /var/log/ceph -- . 2024-05-27T07:54:02.688 INFO:tasks.cephadm:Removing cluster... 2024-05-27T07:54:02.689 DEBUG:teuthology.orchestra.run.smithi052:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid 73df2636-1bf8-11ef-bc9b-c7b262605968 --force 2024-05-27T07:54:02.892 INFO:teuthology.orchestra.run.smithi052.stdout:Deleting cluster with fsid: 73df2636-1bf8-11ef-bc9b-c7b262605968 2024-05-27T07:54:03.479 DEBUG:teuthology.orchestra.run.smithi105:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid 73df2636-1bf8-11ef-bc9b-c7b262605968 --force 2024-05-27T07:54:03.967 INFO:teuthology.orchestra.run.smithi105.stdout:Deleting cluster with fsid: 73df2636-1bf8-11ef-bc9b-c7b262605968 2024-05-27T07:54:04.692 DEBUG:teuthology.orchestra.run.smithi173:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid 73df2636-1bf8-11ef-bc9b-c7b262605968 --force 2024-05-27T07:54:04.884 INFO:teuthology.orchestra.run.smithi173.stdout:Deleting cluster with fsid: 73df2636-1bf8-11ef-bc9b-c7b262605968 2024-05-27T07:54:05.440 INFO:tasks.cephadm:Removing cephadm ... 2024-05-27T07:54:05.440 DEBUG:teuthology.orchestra.run.smithi052:> rm -rf /home/ubuntu/cephtest/cephadm 2024-05-27T07:54:05.460 DEBUG:teuthology.orchestra.run.smithi105:> rm -rf /home/ubuntu/cephtest/cephadm 2024-05-27T07:54:05.477 DEBUG:teuthology.orchestra.run.smithi173:> rm -rf /home/ubuntu/cephtest/cephadm 2024-05-27T07:54:05.492 INFO:tasks.cephadm:Teardown complete 2024-05-27T07:54:05.492 DEBUG:teuthology.run_tasks:Unwinding manager install 2024-05-27T07:54:05.505 INFO:teuthology.task.install.util:Removing shipped files: /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer... 2024-05-27T07:54:05.505 DEBUG:teuthology.orchestra.run.smithi052:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-05-27T07:54:05.508 DEBUG:teuthology.orchestra.run.smithi105:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-05-27T07:54:05.519 DEBUG:teuthology.orchestra.run.smithi173:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-05-27T07:54:05.647 DEBUG:teuthology.run_tasks:Unwinding manager clock 2024-05-27T07:54:05.658 INFO:teuthology.task.clock:Checking final clock skew... 2024-05-27T07:54:05.659 DEBUG:teuthology.orchestra.run.smithi052:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-05-27T07:54:05.661 DEBUG:teuthology.orchestra.run.smithi105:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-05-27T07:54:05.664 DEBUG:teuthology.orchestra.run.smithi173:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-05-27T07:54:05.676 INFO:teuthology.orchestra.run.smithi105.stderr:bash: line 1: ntpq: command not found 2024-05-27T07:54:05.676 INFO:teuthology.orchestra.run.smithi052.stderr:bash: line 1: ntpq: command not found 2024-05-27T07:54:05.677 INFO:teuthology.orchestra.run.smithi173.stderr:bash: line 1: ntpq: command not found 2024-05-27T07:54:05.682 INFO:teuthology.orchestra.run.smithi052.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-05-27T07:54:05.682 INFO:teuthology.orchestra.run.smithi052.stdout:=============================================================================== 2024-05-27T07:54:05.682 INFO:teuthology.orchestra.run.smithi052.stdout:^+ hv01.front.sepia.ceph.com 3 6 377 31 +995us[ +995us] +/- 50ms 2024-05-27T07:54:05.682 INFO:teuthology.orchestra.run.smithi052.stdout:^* hv02.front.sepia.ceph.com 3 6 377 33 -390us[ -379us] +/- 72ms 2024-05-27T07:54:05.682 INFO:teuthology.orchestra.run.smithi052.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 31 -470us[ -470us] +/- 63ms 2024-05-27T07:54:05.682 INFO:teuthology.orchestra.run.smithi052.stdout:^? hv04.front.sepia.ceph.com 0 9 0 - +0ns[ +0ns] +/- 0ns 2024-05-27T07:54:05.684 INFO:teuthology.orchestra.run.smithi173.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-05-27T07:54:05.684 INFO:teuthology.orchestra.run.smithi173.stdout:=============================================================================== 2024-05-27T07:54:05.684 INFO:teuthology.orchestra.run.smithi173.stdout:^+ hv01.front.sepia.ceph.com 3 6 377 30 +877us[+1014us] +/- 50ms 2024-05-27T07:54:05.684 INFO:teuthology.orchestra.run.smithi173.stdout:^* hv02.front.sepia.ceph.com 3 6 377 28 -528us[ -391us] +/- 72ms 2024-05-27T07:54:05.684 INFO:teuthology.orchestra.run.smithi173.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 32 -595us[ -458us] +/- 63ms 2024-05-27T07:54:05.684 INFO:teuthology.orchestra.run.smithi173.stdout:^? hv04.front.sepia.ceph.com 0 9 0 - +0ns[ +0ns] +/- 0ns 2024-05-27T07:54:05.706 INFO:teuthology.orchestra.run.smithi105.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-05-27T07:54:05.707 INFO:teuthology.orchestra.run.smithi105.stdout:=============================================================================== 2024-05-27T07:54:05.707 INFO:teuthology.orchestra.run.smithi105.stdout:^+ hv01.front.sepia.ceph.com 3 6 377 31 +849us[ +959us] +/- 50ms 2024-05-27T07:54:05.707 INFO:teuthology.orchestra.run.smithi105.stdout:^* hv02.front.sepia.ceph.com 3 6 377 31 -517us[ -407us] +/- 72ms 2024-05-27T07:54:05.707 INFO:teuthology.orchestra.run.smithi105.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 31 -590us[ -480us] +/- 63ms 2024-05-27T07:54:05.707 INFO:teuthology.orchestra.run.smithi105.stdout:^? hv04.front.sepia.ceph.com 0 9 0 - +0ns[ +0ns] +/- 0ns 2024-05-27T07:54:05.708 DEBUG:teuthology.run_tasks:Unwinding manager ansible.cephlab 2024-05-27T07:54:05.718 INFO:teuthology.task.ansible:Skipping ansible cleanup... 2024-05-27T07:54:05.719 DEBUG:teuthology.run_tasks:Unwinding manager selinux 2024-05-27T07:54:05.753 DEBUG:teuthology.orchestra.run.smithi052:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-05-27T07:54:05.846 DEBUG:teuthology.orchestra.run.smithi105:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-05-27T07:54:05.955 DEBUG:teuthology.orchestra.run.smithi173:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-05-27T07:54:06.021 DEBUG:teuthology.orchestra.run.smithi052:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e scontext=system_u:system_r:logrotate_t:s0 -e scontext=system_u:system_r:getty_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-05-27T07:54:06.054 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-05-27T07:54:06.054 DEBUG:teuthology.orchestra.run.smithi105:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e scontext=system_u:system_r:logrotate_t:s0 -e scontext=system_u:system_r:getty_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-05-27T07:54:06.081 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-05-27T07:54:06.082 DEBUG:teuthology.orchestra.run.smithi173:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e scontext=system_u:system_r:logrotate_t:s0 -e scontext=system_u:system_r:getty_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-05-27T07:54:06.108 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-05-27T07:54:06.109 DEBUG:teuthology.run_tasks:Unwinding manager pcp 2024-05-27T07:54:06.121 DEBUG:teuthology.run_tasks:Unwinding manager internal.timer 2024-05-27T07:54:06.132 INFO:teuthology.task.internal:Duration was 2921.606556 seconds 2024-05-27T07:54:06.133 DEBUG:teuthology.run_tasks:Unwinding manager internal.syslog 2024-05-27T07:54:06.143 INFO:teuthology.task.internal.syslog:Shutting down syslog monitoring... 2024-05-27T07:54:06.143 DEBUG:teuthology.orchestra.run.smithi052:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-05-27T07:54:06.146 DEBUG:teuthology.orchestra.run.smithi105:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-05-27T07:54:06.148 DEBUG:teuthology.orchestra.run.smithi173:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-05-27T07:54:06.185 INFO:teuthology.orchestra.run.smithi173.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-05-27T07:54:06.187 INFO:teuthology.orchestra.run.smithi052.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-05-27T07:54:06.200 INFO:teuthology.orchestra.run.smithi105.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-05-27T07:54:06.531 INFO:teuthology.task.internal.syslog:Checking logs for errors... 2024-05-27T07:54:06.531 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi052.front.sepia.ceph.com 2024-05-27T07:54:06.531 DEBUG:teuthology.orchestra.run.smithi052:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-05-27T07:54:06.595 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi105.front.sepia.ceph.com 2024-05-27T07:54:06.595 DEBUG:teuthology.orchestra.run.smithi105:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-05-27T07:54:06.626 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi173.front.sepia.ceph.com 2024-05-27T07:54:06.626 DEBUG:teuthology.orchestra.run.smithi173:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-05-27T07:54:06.649 INFO:teuthology.task.internal.syslog:Compressing syslogs... 2024-05-27T07:54:06.649 DEBUG:teuthology.orchestra.run.smithi052:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-05-27T07:54:06.651 DEBUG:teuthology.orchestra.run.smithi105:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-05-27T07:54:06.669 DEBUG:teuthology.orchestra.run.smithi173:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-05-27T07:54:06.715 INFO:teuthology.task.internal.syslog:Gathering journactl -b0... 2024-05-27T07:54:06.715 DEBUG:teuthology.orchestra.run.smithi052:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-05-27T07:54:06.722 DEBUG:teuthology.orchestra.run.smithi105:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-05-27T07:54:06.734 DEBUG:teuthology.orchestra.run.smithi173:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-05-27T07:54:07.766 DEBUG:teuthology.run_tasks:Unwinding manager internal.sudo 2024-05-27T07:54:07.777 INFO:teuthology.task.internal:Restoring /etc/sudoers... 2024-05-27T07:54:07.778 DEBUG:teuthology.orchestra.run.smithi052:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-05-27T07:54:07.804 DEBUG:teuthology.orchestra.run.smithi105:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-05-27T07:54:07.850 DEBUG:teuthology.orchestra.run.smithi173:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-05-27T07:54:07.876 DEBUG:teuthology.run_tasks:Unwinding manager internal.coredump 2024-05-27T07:54:07.888 DEBUG:teuthology.orchestra.run.smithi052:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-05-27T07:54:07.890 DEBUG:teuthology.orchestra.run.smithi105:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-05-27T07:54:07.893 DEBUG:teuthology.orchestra.run.smithi173:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-05-27T07:54:07.912 INFO:teuthology.orchestra.run.smithi052.stdout:kernel.core_pattern = core 2024-05-27T07:54:07.931 INFO:teuthology.orchestra.run.smithi105.stdout:kernel.core_pattern = core 2024-05-27T07:54:07.941 INFO:teuthology.orchestra.run.smithi173.stdout:kernel.core_pattern = core 2024-05-27T07:54:07.977 DEBUG:teuthology.orchestra.run.smithi052:> test -e /home/ubuntu/cephtest/archive/coredump 2024-05-27T07:54:07.991 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-05-27T07:54:07.992 DEBUG:teuthology.orchestra.run.smithi105:> test -e /home/ubuntu/cephtest/archive/coredump 2024-05-27T07:54:08.008 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-05-27T07:54:08.008 DEBUG:teuthology.orchestra.run.smithi173:> test -e /home/ubuntu/cephtest/archive/coredump 2024-05-27T07:54:08.032 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-05-27T07:54:08.033 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive 2024-05-27T07:54:08.044 INFO:teuthology.task.internal:Transferring archived files... 2024-05-27T07:54:08.045 DEBUG:teuthology.misc:Transferring archived files from smithi052:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/vshankar-2024-05-27_05:52:49-fs-wip-vshankar-testing-20240521.063558-debug-testing-default-smithi/7727648/remote/smithi052 2024-05-27T07:54:08.045 DEBUG:teuthology.orchestra.run.smithi052:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-05-27T07:54:08.104 DEBUG:teuthology.misc:Transferring archived files from smithi105:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/vshankar-2024-05-27_05:52:49-fs-wip-vshankar-testing-20240521.063558-debug-testing-default-smithi/7727648/remote/smithi105 2024-05-27T07:54:08.104 DEBUG:teuthology.orchestra.run.smithi105:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-05-27T07:54:08.143 DEBUG:teuthology.misc:Transferring archived files from smithi173:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/vshankar-2024-05-27_05:52:49-fs-wip-vshankar-testing-20240521.063558-debug-testing-default-smithi/7727648/remote/smithi173 2024-05-27T07:54:08.143 DEBUG:teuthology.orchestra.run.smithi173:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-05-27T07:54:08.179 INFO:teuthology.task.internal:Removing archive directory... 2024-05-27T07:54:08.179 DEBUG:teuthology.orchestra.run.smithi052:> rm -rf -- /home/ubuntu/cephtest/archive 2024-05-27T07:54:08.181 DEBUG:teuthology.orchestra.run.smithi105:> rm -rf -- /home/ubuntu/cephtest/archive 2024-05-27T07:54:08.184 DEBUG:teuthology.orchestra.run.smithi173:> rm -rf -- /home/ubuntu/cephtest/archive 2024-05-27T07:54:08.229 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive_upload 2024-05-27T07:54:08.243 INFO:teuthology.task.internal:Not uploading archives. 2024-05-27T07:54:08.243 DEBUG:teuthology.run_tasks:Unwinding manager internal.base 2024-05-27T07:54:08.254 INFO:teuthology.task.internal:Tidying up after the test... 2024-05-27T07:54:08.254 DEBUG:teuthology.orchestra.run.smithi052:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-05-27T07:54:08.256 DEBUG:teuthology.orchestra.run.smithi105:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-05-27T07:54:08.259 DEBUG:teuthology.orchestra.run.smithi173:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-05-27T07:54:08.270 INFO:teuthology.orchestra.run.smithi052.stdout: 265810 4 drwxr-xr-x 2 ubuntu ubuntu 4096 May 27 07:54 /home/ubuntu/cephtest 2024-05-27T07:54:08.273 INFO:teuthology.orchestra.run.smithi105.stdout: 262161 4 drwxr-xr-x 2 ubuntu ubuntu 4096 May 27 07:54 /home/ubuntu/cephtest 2024-05-27T07:54:08.284 INFO:teuthology.orchestra.run.smithi173.stdout: 262161 4 drwxr-xr-x 2 ubuntu ubuntu 4096 May 27 07:54 /home/ubuntu/cephtest 2024-05-27T07:54:08.286 DEBUG:teuthology.run_tasks:Unwinding manager kernel 2024-05-27T07:54:08.339 DEBUG:teuthology.run_tasks:Unwinding manager console_log 2024-05-27T07:54:08.367 INFO:teuthology.run:Summary data: description: fs/workload/{0-centos_9.stream begin/{0-install 1-cephadm 2-logrotate 3-modules} clusters/1a11s-mds-1c-client-3node conf/{client mds mgr mon osd} mount/kclient/{base/{mount-syntax/{v1} mount overrides/{distro/stock/{centos_9.stream k-stock} ms-die-on-skipped}} ms_mode/legacy wsync/yes} objectstore-ec/bluestore-comp-ec-root omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts pg_health session_timeout} ranks/multi/{balancer/random export-check n/5 replication/default} standby-replay tasks/{0-subvolume/{with-no-extra-options} 1-check-counter 2-scrub/yes 3-snaps/yes 4-flush/yes 5-quiesce/no 6-workunit/suites/blogbench}} duration: 2921.6065561771393 failure_reason: 'Command failed on smithi052 with status 22: ''sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:38f9d900c83c63799fbdbe61acc9a11b0d3554a6 shell --fsid 73df2636-1bf8-11ef-bc9b-c7b262605968 -- ceph daemon mds.a perf dump''' flavor: default owner: scheduled_vshankar@teuthology status: fail success: false 2024-05-27T07:54:08.367 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-05-27T07:54:08.447 INFO:teuthology.run:FAIL